Minimize accidental Dev/Prod configuration drift

[IzBrain67]

Context

The Web application (gui/) currently has substantial differences between the Dev and Prod runtime configurations, which makes "works on Dev, breaks on Prod" a recurring class of bug. PR #41 (fix-dev-prod-compose-split) split the configurations into docker-compose.yml (Dev) and docker-compose.prod.yml (Prod overlay), which was a good first step — but a number of accidental differences (not required by performance, security, or operational concerns) remain, mixed in with the legitimate environmental differences.

This issue proposes a focused cleanup whose goal is to minimize the differences between Dev and Prod so that what passes in Dev is much more likely to also work in Prod. It does not propose introducing CI/CD automation, release tagging, or release-procedure documentation — those are valuable but out of scope here.

The plan also assumes that we may eventually run a staging environment or an additional Prod instance alongside snnbuilder.riken.jp, so domain-specific values must not be hardcoded.

Classification of current differences

Essential differences (keep, but document)

These are intentional and driven by performance or security. They should remain, and we should explicitly document them in gui/DOCKER_SETUP.md:

Area	Dev	Prod	Reason
Backend process	runserver	gunicorn --workers ${GUNICORN_WORKERS:-3}	production-grade WSGI server
Frontend	Vite dev server with HMR	nginx + multi-stage built static assets	production performance
Keycloak	start-dev	start	production mode
Cookies	Secure=false, SameSite=Lax	Secure=true, SameSite=None	HTTPS-only in Prod
Django	DEBUG=true, ALLOWED_HOSTS=*	DEBUG=false, ALLOWED_HOSTS_ALL=false + extras	safety
JupyterHub Authenticator	Dummy	FirstUse	password-based login

Accidental differences (eliminate — this issue's scope)

#	Drift	Files	Proposed fix
1	mcp_client.py still falls back to MCP_PROXY_URL even though both compose files have already standardized on MCP_SERVER_URL	gui/workflow_backend/django-project/app/chat/services/mcp_client.py:8-11	Remove the fallback; standardize on MCP_SERVER_URL only
2	JupyterHub subpath is configured in two places that must be kept in sync manually (JUPYTERHUB_API_URL on the backend side, JUPYTERHUB_BASE_URL on the hub side)	gui/docker-compose.yml:34, gui/docker-compose.prod.yml:22,30, gui/workflow_backend/django-project/app/workflow/jupyter_execution_service.py:13,41,104-106,158-166	Use JUPYTERHUB_BASE_URL as the single source of truth and compose JUPYTERHUB_API_URL from ${JUPYTERHUB_INTERNAL_HOST}${JUPYTERHUB_BASE_URL%/} (or have the backend build it)
3	Frontend uses an absolute VITE_JUPYTER_BASE_URL=http://localhost:8000 in Dev but a relative /jupyter in Prod; Vite proxy only forwards /api so other paths cannot be made relative on Dev as-is	gui/workflow_frontend/env.template:4, gui/workflow_frontend/vite.config.ts:28-34	Make Dev use relative paths too; extend the Vite proxy to cover /jupyter, /mcp, /auth so the browser always talks to its own origin
4	API timeout is hardcoded per Vite MODE to production=10s / staging=15s / development=30s	gui/workflow_frontend/src/api/config.ts:7-30	Replace with import.meta.env.VITE_API_TIMEOUT ?? 30000 so it is tunable per deployment
5	docker-compose.prod.yml has snnbuilder.riken.jp hardcoded as the default for ALLOWED_HOSTS_EXTRA, CORS_ALLOWED_ORIGINS_EXTRA, CSRF_TRUSTED_ORIGINS_EXTRA, JUPYTERHUB_FRAME_ORIGIN, and KC_HOSTNAME (lines 18, 19, 20, 33, 68)	gui/docker-compose.prod.yml	Make those variables required at startup (${VAR:?required}); add a gui/.env.prod.example file enumerating them; change the documented startup command to use --env-file gui/.env.prod
6	No documented way to run a "Prod-like" configuration locally	gui/DOCKER_SETUP.md	Add a section explaining how to run the Prod overlay locally, with HTTP-only override examples (SESSION_COOKIE_SECURE=false, JUPYTERHUB_COOKIE_SECURE=false, ALLOWED_HOSTS_EXTRA=localhost)

Proposed tasks (in priority order)

High priority

T1. Remove the MCP_PROXY_URL fallback in mcp_client.py

• File: gui/workflow_backend/django-project/app/chat/services/mcp_client.py:8-11
• Change: Replace the dual lookup with os.environ.get("MCP_SERVER_URL", "http://mcp:8001")
• Verify: grep -rn "MCP_PROXY_URL" gui/ src/ returns 0 matches
• Risk: A developer still running a local .env with MCP_PROXY_URL set will need to rename it. Mention this in the release notes.

T2. Make JupyterHub subpath a single-source value

• Design: Make JUPYTERHUB_BASE_URL the single source of truth. Either build JUPYTERHUB_API_URL from it in the compose files (e.g. JUPYTERHUB_API_URL=http://jupyterhub:8000${JUPYTERHUB_BASE_URL%/}), or have the backend assemble the URL from a separate JUPYTERHUB_INTERNAL_HOST + JUPYTERHUB_BASE_URL pair.
• Files:
  • gui/workflow_backend/django-project/app/workflow/jupyter_execution_service.py:13,41,104-106,158-166
  • gui/docker-compose.yml:34
  • gui/docker-compose.prod.yml:22,30
• Verify: On both Dev and Prod overlay startups, the backend can hit ${hub_url}/hub/api/users/<user> (HTTP) and ${ws_url}/user/<user>/api/kernels/<id>/channels (WebSocket).
• Risk: The WebSocket URL is currently built by string-splitting on ://. Confirm it still works when the host portion contains a subpath. Prod already works, but a smoke test is required.

T3. Unify the Dev/Prod frontend URL structure

• Changes:
  • In gui/workflow_frontend/env.template:4, change VITE_JUPYTER_BASE_URL=http://localhost:8000 to /jupyter so it matches Prod's relative form
  • In gui/workflow_frontend/vite.config.ts:28-34, add proxy entries for /jupyter, /mcp, and /auth (targets driven by env vars: VITE_PROXY_BACKEND, VITE_PROXY_JUPYTER, VITE_PROXY_MCP, VITE_PROXY_KEYCLOAK) so the browser always speaks to its own origin
  • Document the migration in gui/DOCKER_SETUP.md
• Verify: After pnpm run dev, the browser Network tab shows every request (/api/*, /jupyter/*, /mcp/*, /auth/*) going to the dev origin (e.g. http://localhost:5173) — none of them should be absolute cross-origin calls.
• Risk: Any developer's local .env that hardcodes VITE_JUPYTER_BASE_URL=http://localhost:8000 will need to be updated. Add a note to the README.

T4. Make API timeout environment-driven instead of mode-driven

• File: gui/workflow_frontend/src/api/config.ts:7-30
• Change: Replace the three mode-specific timeouts with import.meta.env.VITE_API_TIMEOUT ?? 30000. Keep the internalSecret default fallback as-is.
• Verify: Existing API calls behave identically with the default; smoke-test that long-running calls aren't being silently truncated in Prod.
• Risk: If any deployment relied on the 10s Prod timeout to catch hung backends, exposing it as VITE_API_TIMEOUT=10000 recovers the old behavior.

Medium priority

T5. Remove domain hardcoding, ship .env.prod.example

• Files:
  • gui/docker-compose.prod.yml — drop the :-snnbuilder.riken.jp and :-https://snnbuilder.riken.jp defaults on lines 18, 19, 20, 33, 68 and switch to the ${VAR:?…} form so the container refuses to start if a required value is missing
  • New file gui/.env.prod.example listing every required variable (ALLOWED_HOSTS_EXTRA, CORS_ALLOWED_ORIGINS_EXTRA, CSRF_TRUSTED_ORIGINS_EXTRA, JUPYTERHUB_FRAME_ORIGIN, KEYCLOAK_HOSTNAME, VITE_KEYCLOAK_REALM, VITE_KEYCLOAK_CLIENT_ID, etc.)
  • The documented startup command becomes:

    docker compose --env-file gui/.env.prod \
      -f gui/docker-compose.yml -f gui/docker-compose.prod.yml up -d

• Verify: Starting without .env.prod exits with a clear error; starting with ALLOWED_HOSTS_EXTRA=staging.example.com reflects that hostname in the Django logs.
• Risk: Operational impact — any existing production workflow that runs docker compose -f … -f … (without --env-file) will break the next time this lands. Roll this out together with deploying .env.prod on the production host and updating any operator scripts.

T6. Document Prod-like local startup in gui/DOCKER_SETUP.md

• File: gui/DOCKER_SETUP.md
• Add: A "Production-like local run" section, e.g.:

  cp gui/.env.prod.example gui/.env.prod
  # Edit .env.prod, for example:
  #   ALLOWED_HOSTS_EXTRA=localhost
  #   CORS_ALLOWED_ORIGINS_EXTRA=http://localhost:5173
  #   CSRF_TRUSTED_ORIGINS_EXTRA=http://localhost:5173
  #   SESSION_COOKIE_SECURE=false
  #   CSRF_COOKIE_SECURE=false
  #   JUPYTERHUB_COOKIE_SECURE=false
  docker compose --env-file gui/.env.prod \
    -f gui/docker-compose.yml -f gui/docker-compose.prod.yml up --build

• Verify: A new developer can follow the section as-is and reach a logged-in app.

Low priority

T7. Document the differences we intentionally keep

• File: gui/DOCKER_SETUP.md (appended) or a new gui/PROD_DIFFERENCES.md
• Content: The "Essential differences" table above

End-to-end verification checklist

Before merging the final PR in the series:

1. Static checks
   • grep -rn "MCP_PROXY_URL" gui/ src/ → 0 matches
   • grep -rn "snnbuilder.riken.jp" gui/docker-compose.prod.yml → 0 matches
2. Dev smoke
   • cd gui && docker compose up --build
   • Open http://localhost:5173 in a browser, log in
   • Create a workflow → generate code → execute via Jupyter end-to-end
   • In the browser Network tab, confirm that every request (/api, /jupyter, /mcp, /auth) is going through the dev origin
3. Prod-like local smoke
   • cp gui/.env.prod.example gui/.env.prod, override the cookie/secure flags for local HTTP
   • docker compose --env-file gui/.env.prod -f gui/docker-compose.yml -f gui/docker-compose.prod.yml up --build -d
   • Run the same workflow scenario and confirm parity with Dev
4. Pre-production deploy
   • Release notes must call out: (a) the startup command change, (b) the requirement to deploy .env.prod
   • Coordinate with the operator of snnbuilder.riken.jp to stage .env.prod and switch the startup command in a single change window

Staged PR plan

To keep each step easy to review and roll back:

1. PR-A — T1 (MCP fallback removal) + T2 (JupyterHub URL single-source)
2. PR-B — T3 (frontend Dev/Prod structural parity) + T4 (API timeout env var)
3. PR-C — T5 (drop hardcoded domain, add .env.prod.example). Ship last, because it changes the production startup command.
4. PR-D — T6 + T7 (documentation)

Out of scope (future work)

• GitHub Actions for CI/CD (automatic test + build + tag-driven deploy)
• Version management (pyproject.toml version, git tags)
• Runtime env substitution for the built frontend image (so VITE_KEYCLOAK_* can be changed without rebuilding)
• End-to-end test suite (Playwright/Cypress)

[carlosengutierrez]

@IzBrain67 Thanks for the thorough write-up this is exactly the kind of analysis we need to keep Dev and Prod aligned. The classification into "essential vs. accidental" differences is the right framing and I agree with the proposed split.