Let users edit run.sbatch and copy Slurm .out/.err back

deployment/compute_server/README.md

@@ -1,12 +1,45 @@
 # Remote Slurm execution - operator runbook (RIKEN compute server)
 
 This enables the Django backend to submit a workflow as a **Slurm batch job** on
-the RIKEN compute server over SSH, poll its status, and read back results.
+the RIKEN compute server over SSH, poll its status, and copy logs and results
+back into the NeuroWorkflow project tree.
 
 It is **additive and OFF by default**: the live JupyterHub run path is untouched,
 and this path runs only when a run is submitted with `backend=slurm`. The
 ssh-agent in the backend container holds no key until an admin unlocks it.
 
+## User flow (GUI)
+
+1. **Prepare (draft).** Opening **Run on Compute Cluster** creates a `draft`
+   `WorkflowRun` and writes `run.sbatch` to
+   `codes/projects/<project_id>/batch/<run_id>/run.sbatch` (the same tree
+   Jupyter mounts). No SSH and no `sbatch` yet.
+2. **Edit.** The Cluster Run modal shows the script. The user can edit it
+   there or **Edit in Jupyter** (then **Reload from project**). Changing
+   partition/CPU/memory/time regenerates the script unless the textarea is
+   dirty (confirm before overwrite).
+3. **Submit.** `POST /api/workflow/<id>/runs/submit/` with `backend: "slurm"`,
+   the draft `run_id`, and the (possibly edited) `sbatch` text. The executor
+   pins `#SBATCH --chdir` / `--output` / `--error` to the remote run dir, rsyncs
+   the batch dir, and runs `sbatch run.sbatch`.
+4. **Copy-back.** On the first poll that sees COMPLETED, FAILED, or CANCELLED,
+   the backend rsyncs `slurm-*.out`, `slurm-*.err`, `stdout.log`, `stderr.log`,
+   `exit_code.txt`, and `manifest.json` into
+   `codes/projects/<project_id>/batch/<run_id>/logs/`, and `results/` into
+   `.../results/` when that directory exists. The Runs panel lists **Logs** and
+   **Results** for download. Failed jobs are copied too (that is when `.err`
+   matters).
+
+Closing the modal without submit leaves the draft in the Runs panel (delete
+removes it). **Edit script & resubmit** copies the previous `run.sbatch` into a
+new draft.
+
+The full UI/API path is `POST /api/workflow/<id>/runs/submit/` with
+`{"backend": "slurm"}` (not `/run-submit/`). Drafts use
+`POST /api/workflow/<id>/runs/prepare/` and
+`GET`/`PUT /api/workflow/<id>/runs/<run_id>/sbatch/`.
+
+
 ## Facts baked into the implementation (from RIKEN)
 
 - Login node: `digitalbrain.brainminds.jp`, user `neuro-workflow`. Used only for
@@ -114,8 +147,10 @@ PY
 ```
 
 Success = status `COMPLETED`, exit code `0`, and stdout containing the test
-summary. The full UI/API path (`POST /api/workflow/<id>/run-submit/` with
-`{"backend": "slurm"}`) uses exactly this executor.
+summary. The full UI/API path (`POST /api/workflow/<id>/runs/submit/` with
+`{"backend": "slurm"}`) uses exactly this executor. After a terminal status,
+`get_status(..., project_id=...)` also copies `slurm-*.out/.err` into
+`codes/projects/<project_id>/batch/<run_id>/logs/`.
 
 ---
 

deployment/pr-cluster-sbatch-logs-progress.log

@@ -0,0 +1,154 @@
+================================================================================
+PR cluster-sbatch-logs — progress log (append-only)
+Worktree: /home/nw-kirill/neuro-workflow-cluster-sbatch
+Branch:   feat/cluster-sbatch-logs  (from origin/main e1218777)
+Base:     origin/main  (independent of #88–#95)
+================================================================================
+
+## Verdict
+
+Cluster submit on main already generates run.sbatch from a template and rsyncs
+only results/ on COMPLETED. Native slurm-%j.out / .err stay on the compute
+server; failed jobs are not copied back; there is no editor. This PR completes
+cluster-job I/O: draft + editable run.sbatch (modal + Jupyter) and copy-back
+of Slurm/Python logs into codes/projects/<id>/batch/<run_id>/logs/.
+
+Kirill’s choices
+----------------
+- Study/product: Open Composer pattern inside NWF (form → edit script → submit).
+  Modal textarea + file on disk under Jupyter’s project tree. No OOD install.
+- Copy-back into the same batch dir (logs/ + existing results/). COMPLETED and
+  FAILED (and CANCELLED if files exist). Pin --chdir/--output/--error after edits.
+- No new dependencies, no Dockerfile.nest, no live compose/deploy/merge.
+
+Non-goals
+---------
+sudo; write /data; nest on #88–#95; force-push; Open OnDemand; Keycloak/nginx;
+expand GET .../files/ into a recursive browser; call unused get_logs() from HTTP.
+
+Integrity vs other PRs
+-----------------------
+Independent branch from origin/main. Surgical edits: executor, run views,
+ClusterRunModal, Runs panel, run API client, compute-server README.
+Adding WorkflowRun.Status.DRAFT is a Python choice; no DB migration.
+DetailView polls only pending/running — copy-back must run on the get_status
+that first sees COMPLETED/FAILED. Draft must not call sacct.
+
+Live UI baseline
+----------------
+https://neuro-workflow.dbrain.jp/ will not show this until deploy. After
+implementation, confirm the gap is still there (no sbatch editor, no .out/.err
+in Runs). Prove the engine with pytest in this worktree.
+
+================================================================================
+
+## Step 1 — Worktree and progress log
+
+Done. Created worktree from origin/main (e1218777, Merge pull request #87).
+Branch feat/cluster-sbatch-logs. This file created.
+
+Verify: git status is feat/cluster-sbatch-logs; HEAD is e1218777; log exists.
+
+================================================================================
+
+## Step 2 — Draft + editable run.sbatch API
+
+Done. Added WorkflowRun.Status.DRAFT (no migration). normalize_sbatch pins
+--chdir/--output/--error. POST /runs/prepare/ writes run.sbatch with no SSH.
+GET/PUT /runs/<id>/sbatch/. Submit accepts optional run_id (draft) + sbatch.
+from_run_id copies a previous script into a new draft. PUT with
+resource_requests re-renders when the modal form changes.
+
+Verify: prepare does not call sbatch (pytest boom on _ssh); PUT without
+--chdir still has pinned directives on disk.
+
+## Step 3 — Copy .out/.err back
+
+Done. get_status on COMPLETED/FAILED/CANCELLED rsyncs slurm-*.out/.err,
+stdout.log, stderr.log, exit_code.txt, manifest.json into logs/, and still
+fetches results/ when present (including FAILED). artifacts JSON has files +
+logs. Artifact view serves logs/, results/, run.sbatch; rejects nodes/ and
+.. . DB stdout/stderr capped at 512 KiB. get_logs() still unused by HTTP.
+
+Verify: mocked FAILED copy-back writes logs/slurm-1.err; download 200;
+path=../x is 400; nodes/foo.py is 400.
+
+## Step 4 — Cluster Run modal
+
+Done. Prepare on open; monospace textarea; Reload from project; Edit in Jupyter
+(user1 tree path, same as today); dirty flag; confirm before regenerate;
+submit sends run_id + sbatch after generate-code.
+
+## Step 5 — Runs panel
+
+Done. draft badge (purple). Logs + Results download lists. Open run folder in
+Jupyter. Edit script & resubmit. Cancel hidden for draft (treated as terminal
+for polling).
+
+## Step 6 — Tests
+
+Added django-project/tests/test_cluster_sbatch_logs.py (8 tests). No vitest
+files in the frontend, so no new frontend runner. SQLite cannot apply this
+repo’s Postgres migrations; pytest uses an ephemeral postgres:16 container
+on docker network nw-sbatch-test (not live compose, not live DB).
+
+Verify: 8 passed.
+
+## Step 7 — Docs
+
+Updated deployment/compute_server/README.md: draft → edit → sbatch; copy-back
+on success and failure; Jupyter path; fixed stale /run-submit/ →
+/runs/submit/. deployment/DEPLOY_COMPUTE_SERVER.md is not in this git tree
+(control/docs only); the in-repo operator README is the document that shipped.
+
+No TODOs/placeholders in the new engine paths except existing executor
+NotImplemented elsewhere.
+
+================================================================================
+
+## Step 8 — Isolated verification
+
+pytest django-project/tests/test_cluster_sbatch_logs.py against ephemeral
+postgres:16 (docker network nw-sbatch-test, not live DB): 8 passed.
+
+black --check --fast --line-length 88 and isort --check --profile black on
+touched Python: pass.
+
+Live https://neuro-workflow.dbrain.jp/ (logged in as kirill; did not submit a
+cluster job, did not upload data, did not change settings):
+- Toolbar still has “Run on Compute Cluster”.
+- JS bundle /assets/index-CbQzE8nY.js has no run.sbatch, /runs/prepare,
+  “Edit in Jupyter”, or “Open run folder in Jupyter”.
+- No NW_Optimization (old bundle). This is the expected baseline, not a
+  failed test of this PR.
+
+Ephemeral Postgres container nw-sbatch-pg removed after tests.
+
+================================================================================
+
+## Step 9 — Commit, push, PR, close-out
+
+Done. No merge, no deploy, no force-push.
+
+Commit:  b70cc6b7 Let users edit run.sbatch and copy Slurm logs back into the project
+Pushed:  origin/feat/cluster-sbatch-logs
+PR:      https://github.com/oist/neuro-workflow/pull/96  (base: main)
+
+Verify commands
+---------------
+  python -m pytest django-project/tests/test_cluster_sbatch_logs.py -q
+  → 8 passed (ephemeral postgres:16, not live DB)
+  black --check --fast --line-length 88 (touched Python)
+  isort --check --profile black (same paths)
+
+Live dbrain.jp: old bundle; Cluster Run exists but no sbatch editor / log copy-back.
+
+Remaining risk
+--------------
+- Not deployed; ssh-agent still required for a real sbatch.
+- Jupyter URL still uses the existing user1 path.
+- Study/product: drafts left in Runs if the modal is closed without submit
+  (delete works).
+- Independent of #88–#95; later merge with those may touch views.py.
+
+================================================================================

gui/workflow_backend/django-project/app/workflow/execution/__init__.py

@@ -1,11 +1,17 @@
-from .base import ExecutionBackend, ExecutionStatus, ExecutionResult
+from .base import ExecutionBackend, ExecutionResult, ExecutionStatus
 from .local_executor import LocalExecutor
-from .remote_slurm_executor import RemoteSlurmExecutor
+from .remote_slurm_executor import (
+    RemoteSlurmExecutor,
+    jupyter_sbatch_path,
+    normalize_sbatch,
+)
 
 __all__ = [
     "ExecutionBackend",
     "ExecutionStatus",
     "ExecutionResult",
     "LocalExecutor",
     "RemoteSlurmExecutor",
+    "jupyter_sbatch_path",
+    "normalize_sbatch",
 ]

gui/workflow_backend/django-project/app/workflow/execution/base.py

@@ -46,12 +46,15 @@ def submit(
         *,
         run_id: Optional[str] = None,
         resource_requests: Optional[dict] = None,
+        sbatch_text: Optional[str] = None,
     ) -> ExecutionResult:
         """Submit a workflow run. Returns immediately with a pending result.
 
         ``run_id`` lets the caller pin the run identifier (e.g. the DB
         WorkflowRun id) so staging dirs, remote dirs and later status polls all
         line up. If omitted, a fresh UUID is generated.
+        ``sbatch_text`` is used by the Slurm backend when the user edited the
+        batch script; local backends ignore it.
         """
         ...
 

gui/workflow_backend/django-project/app/workflow/execution/local_executor.py

@@ -34,6 +34,7 @@ def submit(
         *,
         run_id: Optional[str] = None,
         resource_requests: Optional[dict] = None,
+        sbatch_text: Optional[str] = None,
     ) -> ExecutionResult:
         result = ExecutionResult(
             status=ExecutionStatus.PENDING,