Add Keycloak-gated catalog search (mdb Stage 2)

deployment/DEPLOY_MDB_CATALOG.md

@@ -0,0 +1,156 @@
+# Deploy NeuroWorkflow catalog (mdb Stage 2)
+
+How to reproduce the Catalog tab from `main` after this PR. Secrets stay in
+gitignored `.env` files — never commit tokens, keys, or `.env`.
+
+Implementation: `docs/CATALOG_SEARCH.md`. HTTP contract:
+`deployment/MDB_CLIENT_CONTRACT.md`.
+
+mdb itself lives in `oist/bm_mindsdb` (separate compose project). This repo
+only talks to it over HTTP.
+
+---
+
+## What you are wiring
+
+```
+Keycloak user → nginx → NeuroWorkflow /api/catalog/* → mdb-mindsdb:8004
+```
+
+- Public ports stay SSH / 80 / 443. mdb binds **`127.0.0.1:8004`** only.
+- **Do not** add nginx `/mdb`, iframe the mdb UI, or publish `0.0.0.0:8004`.
+- **Do not** set `MDB_ADMIN_TOKEN` in NeuroWorkflow.
+- **Do not** put `MDB_*` in `VITE_*`, `gui/.env`, or the frontend image.
+
+---
+
+## 1. Pin the Docker network (this repo)
+
+`gui/docker-compose.yml` names the backend network:
+
+```yaml
+networks:
+  workflow:
+    name: neuro-workflow_workflow
+```
+
+After `docker compose up` in `gui/`, `docker network ls` should show
+`neuro-workflow_workflow`. Backend hostname on that network is `backend`.
+
+---
+
+## 2. Attach mdb (bm_mindsdb repo, not this one)
+
+On the mdb compose overlay, join the **existing** NeuroWorkflow network. Example
+(hostnames and file names may differ slightly in `oist/bm_mindsdb`):
+
+```yaml
+services:
+  mdb-mindsdb:
+    networks:
+      - default
+      - neuro-workflow_workflow
+
+networks:
+  neuro-workflow_workflow:
+    external: true
+    name: neuro-workflow_workflow
+```
+
+Recreate **mdb-mindsdb only** (keep its volume). Do not recreate the
+NeuroWorkflow backend (ssh-agent). Bind stays `127.0.0.1:8004`.
+
+After attach, from the NeuroWorkflow **backend container**:
+
+```bash
+curl -sS -o /dev/null -w '%{http_code}\n' http://mdb-mindsdb:8004/
+# expect 200
+```
+
+Host check (operators / SSH tunnel):
+
+```bash
+curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8004/
+# expect 200
+```
+
+If mdb was recreated, start in-container MindsDB again with the **mdb admin**
+token (SSH / mdb UI). NeuroWorkflow does not start or stop MindsDB.
+
+---
+
+## 3. NeuroWorkflow backend env (gitignored)
+
+Copy placeholders from `gui/workflow_backend/env.template`. In
+**`gui/workflow_backend/.env` only** (never print or commit the token):
+
+```
+MDB_BASE_URL=http://mdb-mindsdb:8004
+MDB_API_TOKEN=<same search token as in the mdb .env>
+# optional:
+# MDB_TIMEOUT=15
+```
+
+Copy `MDB_API_TOKEN` from the mdb environment file on the host. Do not put
+`MDB_ADMIN_TOKEN` here.
+
+`get_mdb_config()` reads process env first, then the bind-mounted `.env`
+(`override=False`), so a later token add can be picked up with gunicorn SIGHUP
+without recreating the backend.
+
+---
+
+## 4. Load code without dropping Slurm ssh-agent
+
+Backend code is bind-mounted in this compose. Prefer:
+
+```bash
+# inside neuro-workflow-backend-1: SIGHUP the gunicorn master (container PID 1)
+python3 -c "import os,signal; os.kill(1, signal.SIGHUP)"
+```
+
+Recreate gunicorn/backend only if the new env vars are not visible after
+SIGHUP. Recreate drops ssh-agent; you would need `ssh-add` again.
+
+Frontend in production is a **built nginx image** (no src mount). Rebuild and
+recreate **frontend only**:
+
+```bash
+cd gui
+docker compose -f docker-compose.yml -f docker-compose.prod.yml build frontend
+docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-deps --no-build frontend
+```
+
+MCP: `workflow_mcp.py` is bind-mounted; `docker compose restart mcp` (or
+`--no-deps`) is enough for the four catalog tools.
+
+Do not `compose down`, do not recreate Hub / db / Keycloak for this feature.
+
+---
+
+## 5. Smoke
+
+Unauthenticated API (route exists, Keycloak required):
+
+```bash
+curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/api/catalog/statistics/
+# 401
+```
+
+Logged-in user in the browser: **Catalog** in the header → `/catalog` →
+statistics chips load → keyword search (e.g. `mouse`) returns rows → open a
+record.
+
+Failures to report to the mdb side: HTTP status and whether it was
+statistics, search, or lookup. Typical causes: 401/403 from mdb (wrong/missing
+search token) or timeout (wrong `MDB_BASE_URL`).
+
+---
+
+## Rollback
+
+- Remove `MDB_BASE_URL` / `MDB_API_TOKEN` from `gui/workflow_backend/.env` and
+  SIGHUP gunicorn (Catalog API returns `catalog_unconfigured`).
+- Revert the frontend image to the previous build if you need the header link
+  gone.
+- Leave mdb running; it does not depend on NeuroWorkflow.

deployment/MDB_CLIENT_CONTRACT.md

@@ -0,0 +1,165 @@
+# mdb client contract — NeuroWorkflow catalog
+
+This is the HTTP contract NeuroWorkflow’s **backend** should call. The browser
+must never talk to port 8004 and must never see mdb tokens.
+
+Companion (NeuroWorkflow implementation + UI): `docs/CATALOG_SEARCH.md`.  
+Companion (how to wire the two stacks): `deployment/DEPLOY_MDB_CATALOG.md`.
+
+mdb is a separate repo (`oist/bm_mindsdb`). Do not put mdb source or secrets in
+this repository.
+
+---
+
+## Architecture (fixed)
+
+```
+User browser  →  NeuroWorkflow (Keycloak, nginx 80/443)
+                      ↓  same-origin /api/...  (no mdb token)
+              NeuroWorkflow backend
+                      ↓  Authorization: Bearer <MDB_API_TOKEN>
+              mdb-mindsdb:8004   (Docker network neuro-workflow_workflow)
+```
+
+Only SSH / HTTP / HTTPS should be public; Docker `ports:` must bind
+`127.0.0.1`. **Do not** add nginx `/mdb`, iframe the mdb dashboard, put tokens
+in `VITE_*`, or publish `0.0.0.0:8004`.
+
+`MDB_BASE_URL=http://172.17.0.1:8004` **does not work** while 8004 is
+loopback-only, and it is **not** a public bind. Use a shared Docker network and
+`http://mdb-mindsdb:8004`. Keep host `127.0.0.1:8004` for operators (SSH tunnel).
+
+mdb has no Keycloak. **Catalog admin for NeuroWorkflow: none.** Sync stays
+SSH-only (`MDB_ADMIN_TOKEN` is not used by NW). Daily harvest is
+`MDB_AUTO_SYNC_INTERVAL` inside mdb.
+
+---
+
+## Tokens (backend `.env` only)
+
+| Variable | Who may send it | Used for |
+|---|---|---|
+| `MDB_API_TOKEN` | NW backend only (`gui/workflow_backend/.env`). Never `VITE_*`, never frontend env, never `gui/.env`. Never commit the value. | Stage 2 search / statistics / lookup |
+| `MDB_ADMIN_TOKEN` | Operators (SSH mdb UI / curl). **Not** used by NeuroWorkflow. Never commit. | Sync, ingest, NWB, MindsDB process |
+
+Header: `Authorization: Bearer <token>`.  
+Mutating POST/PUT/PATCH also need `Content-Type: application/json` (415
+otherwise). Empty body → send `{}`.
+
+**Always send `MDB_API_TOKEN` from the proxy**, even on GETs that mdb currently
+leaves ungated (`/api/api_statistics`, keyword `GET /api/catalog_search`). Do
+not depend on ungated holes.
+
+---
+
+## HTTP errors from mdb
+
+| Code | When |
+|---|---|
+| **401** | Token required but missing, or token unset on mdb |
+| **403** | Wrong token (search token on an admin route, or vice versa) |
+| **400** | Bad input (`query`/`id` missing, unsupported `mode`/`table`) |
+| **404** | Dataset / conversation not found |
+| **410** | Retired: `/api/execute_sql`, `/api/predict_*`, … |
+| **415** | Mutating POST without `Content-Type: application/json` |
+| **500** | Generic `{ "status": "error", "error": "Request failed" }` |
+| **503** | Orchestrator not ready, or agent search “not ready” (MindsDB down) |
+
+Auth error JSON: `{ "status": "error", "error": "<message>" }`.
+
+---
+
+## Stage 2 — proxy these (search / catalog UI)
+
+From the NW backend container, `$MDB_BASE_URL` is `http://mdb-mindsdb:8004`.
+Operators on the host use `http://127.0.0.1:8004`.
+
+Catalog sources (internal keys): `dandi`, `cbs`, `brainminds`, `bmb_human`, plus
+local `aws` (display `SRPBS_TS`). Records include `source` and `source_display`.
+
+### Health / counts
+
+```bash
+curl -sS -H "Authorization: Bearer $MDB_API_TOKEN" \
+  "$MDB_BASE_URL/api/api_statistics"
+```
+
+Optional: `GET /api/status` (counts + whether MindsDB is up).
+
+### Keyword search (no MindsDB)
+
+```bash
+curl -sS -X POST "$MDB_BASE_URL/api/catalog_search" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $MDB_API_TOKEN" \
+  -d '{"query":"mouse","mode":"keyword","limit":20}'
+```
+
+Optional `source` (e.g. `"dandi"`). `limit` clamped 1–200 (default 50).
+
+Also: `GET $MDB_BASE_URL/api/catalog_search?q=mouse&mode=keyword&limit=20`.
+
+Browse without a query: `GET /api/api_datasets?source=dandi&limit=20`.
+
+### Record lookup
+
+```bash
+curl -sS -H "Authorization: Bearer $MDB_API_TOKEN" \
+  "$MDB_BASE_URL/api/catalog_lookup?table=api_datasets&source=dandi&id=000015"
+```
+
+NeuroWorkflow Stage 2 allows **`table=api_datasets` only**.
+
+---
+
+## Present on mdb, not proxied in Stage 2
+
+Agent / intelligent search and catalog agent chat exist on mdb (`mode=agent`,
+`mode=intelligent`, `/api/mindsdb_agent/chat`). They need MindsDB + OpenAI on
+the mdb side. NeuroWorkflow does **not** expose them in this PR.
+
+Admin routes require `MDB_ADMIN_TOKEN` (search token → 403). Do not proxy these
+to ordinary Keycloak users:
+
+| Method | Path |
+|---|---|
+| POST | `/api/sync_apis` |
+| POST | `/api/ingest_local_catalog` |
+| POST | `/api/process_nwb_file` |
+| POST | `/api/start_mindsdb_server` |
+| POST | `/api/stop_mindsdb_server` |
+| POST | `/api/connect_mindsdb_datasource` |
+| POST | `/api/mindsdb_agent/setup` |
+| POST | `/api/mindsdb_agent/drop` |
+
+Never proxy `/api/execute_sql` or retired predict routes (410). Never call
+MindsDB on 47334 (unpublished, in-container only).
+
+---
+
+## Frozen Stage 2 URL layout
+
+Keycloak-gated. Forward JSON as-is. Do not copy the catalog into Postgres.
+
+| NW (Keycloak) | mdb |
+|---|---|
+| `GET  /api/catalog/statistics/` | `GET /api/api_statistics` |
+| `POST /api/catalog/search/` | `POST /api/catalog_search` (**`mode=keyword` only**) |
+| `GET  /api/catalog/lookup/` | `GET /api/catalog_lookup` |
+| `GET  /api/catalog/datasets/` | `GET /api/api_datasets` |
+
+**Not in this PR:** `/api/catalog/chat`, `/api/catalog/sync`, agent/intelligent
+modes, `GET /api/local_catalog/…`.
+
+---
+
+## Frozen answers
+
+1. **URL layout** — table above.
+2. **Catalog admin** — **none**. No Sync button in NeuroWorkflow.
+3. **Docker network** — mdb joins existing **`neuro-workflow_workflow`** as
+   hostname `mdb-mindsdb`. `MDB_BASE_URL=http://mdb-mindsdb:8004`. Keep host
+   `127.0.0.1:8004`. Attach **mdb**, not the NW backend (avoids gunicorn
+   recreate / Slurm ssh-agent drop).
+4. **Stage 2 scope** — keyword + statistics + lookup + browse only.
+5. **`MDB_*`** — `gui/workflow_backend/.env` only; values are never committed.