Add serialize_allocations to chain Slurm allocations

[daniel-thom]

Problem

A workflow whose sequential work outlives any single allocation has no way to keep going. Each allocation runs what fits in its walltime and exits, and nothing brings up the next one. For a 500-job chain at 2-4 hours each — weeks of serial work against a 12-hour partition limit — every existing option is poor:

• torc watch --auto-schedule needs a process alive on the login node for the entire run. Most sites reap long-running login-node processes well before six weeks.
• on_worker_complete + schedule_nodes self-schedules from the compute node, but needs persistent = true to chain more than once — and then never stops. execute_action calls schedule_slurm_nodes unconditionally, so after the last job the exiting worker still submits a successor, which finds no work, exits, and submits another.
• --dependency=singleton via the scheduler's extra gets the flag but not the shared job name that makes it mean anything. Slurm scopes singleton to (user, job name), and torc's generated name embeds the submitting PID and an allocation counter, so no two allocations ever land in the same singleton set.

Change

Adds serialize_allocations to SlurmSchedulerSpec/SlurmSchedulerModel. When set, every allocation for that scheduler submits under one derived name with --dependency=singleton, so Slurm runs them strictly one at a time. Submit N up front and they chain themselves, with nothing running on the login node.

slurm_schedulers:
  - name: chain
    account: my_account
    walltime: "12:00:00"
    serialize_allocations: true

torc slurm schedule-nodes <workflow_id> -n 167
torc slurm create <wf> -n chain -a acct -W 12:00:00 --serialize-allocations
torc slurm update <sched_id> --serialize-allocations true

A queued successor also accrues priority while its predecessor runs — unlike a replacement submitted at shutdown, which enters the queue with no age and pays full wait on every link of the chain.

Design notes

Both halves ride existing extension points, so the diff stays small. The dependency goes through config_map, which already emits #SBATCH --{key}={value}; it is inserted after slurm_defaults so a workflow-level dependency default cannot silently break the chain. The name is decoupled by passing create_submission_script a different first argument than the script path — the unique PID-bearing name stays on the .sh file so concurrent submissions do not overwrite each other. No signature changed, so none of the existing call sites moved.

serialized_slurm_job_name() keys on (workflow_id, scheduler_id) — stable across the separate processes that may submit into one chain, and distinct per scheduler so unrelated allocations do not serialize against each other. The torc- prefix keeps the singleton set clear of the user's other Slurm jobs.

On the scheduler, not the action. Four paths reach schedule_slurm_nodes(): the schedule_nodes action, the CLI, watch --auto-schedule, and recover. Putting the field in action_config would mean someone launching a chain with torc slurm schedule-nodes -n 167 silently gets 167 concurrent allocations. On the scheduler, all four inherit it, so allocations added later join the chain instead of racing it. Auto-generated schedulers (slurm generate/regenerate, scheduler_plan) default to unset — serializing is an opt-in on a user-authored scheduler.

No change needed to allocation cleanup. cancel_unneeded_pending_allocations runs on head-node exit and cancels queued allocations only when no ready/pending/running job remains. The sequential handoff clears that bar comfortably: the background unblock task runs on a 5s interval while an idle worker waits the full 90s before exiting, so the next job is ready long before the check. Surplus allocations are therefore cheap — round the count up and the extras never start.

Testing

• test_serialized_slurm_job_name_is_stable / _scopes_to_workflow_and_scheduler / _honors_job_prefix — the chain only forms if the name never varies with anything process-local.
• test_create_submission_script_with_singleton_dependency — asserts both the --dependency=singleton directive and the shared --job-name land in the script.
• test_slurm_scheduler_serialize_allocations_kdl_roundtrip — this caught a real bug during development: the first emitter wrote a bare true, but the codebase is on KDL v2 (kdl = "6.5") where booleans are #true/#false, so a KDL spec with the field set emitted a file that failed to reparse. Also asserts an omitted value stays None rather than round-tripping to Some(false).

Full suite green (1787 tests), plus cargo fmt --check, clippy --all --all-targets --all-features -D warnings, dprint check, and mdbook build.

Note for reviewers

One thing I could not verify without a live cluster: whether a later duplicate #SBATCH --job-name overrides an earlier one. Nothing in this change depends on it — torc now emits exactly one --job-name, the shared one. It matters only for anyone combining extra: "--job-name=..." with this feature. The docs note that a --dependency set in extra is emitted last and will override the generated one, breaking the chain.

🤖 Generated with Claude Code

[Copilot]

🟡 Not ready to approve

The serialized Slurm job name currently incorporates the user-provided --job-prefix, which can prevent later allocations from different scheduling paths from joining the same singleton chain and also contradicts the new documentation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds an opt-in serialize_allocations flag to Slurm schedulers so multiple Slurm allocations can be submitted up front and reliably run one-at-a-time via --dependency=singleton, enabling long sequential workflows to continue across allocation walltime limits.

Changes:

• Add serialize_allocations to the Slurm scheduler DB model, server API, OpenAPI spec, and generated clients.
• Implement allocation chaining by emitting a shared Slurm --job-name and --dependency=singleton in generated submission scripts when enabled.
• Add tests and documentation covering KDL/YAML round-trips and submission script behavior.

File summaries

File	Description
torc-server/migrations/20260801000000_add_serialize_allocations.up.sql	Adds serialize_allocations column to slurm_scheduler.
torc-server/migrations/20260801000000_add_serialize_allocations.down.sql	Drops the serialize_allocations column.
src/server/api/schedulers.rs	Persists/reads/updates the new scheduler field via SQLx queries.
src/openapi_spec.rs	Updates OpenAPI parity checks for scheduler properties.
src/models.rs	Extends SlurmSchedulerModel with serialize_allocations and docs.
src/client/workflow_spec.rs	Adds serialize_allocations to spec + KDL parse/emit support.
src/client/scheduler_plan.rs	Ensures auto-planned schedulers don’t opt into serialization.
src/client/commands/slurm.rs	Adds CLI flags and implements singleton chaining/job-name behavior.
api/openapi.yaml	Documents the new serialize_allocations field in the API schema.
api/openapi.codegen.yaml	Keeps codegen spec in sync with the checked-in OpenAPI.
.sqlx/query-84c60a7116c68da5863d814db5336cfc175f1d41b19f69905ddee407ebbcf2fc.json	Updates SQLx offline query metadata for the new INSERT parameter.
tests/test_workflow_spec.rs	Updates spec serialization tests + adds KDL round-trip test for the new field.
tests/test_slurm_regenerate.rs	Updates scheduler construction to include serialize_allocations.
tests/test_slurm_commands.rs	Adds a test asserting --dependency=singleton + shared job-name emission.
tests/test_scheduled_compute_nodes.rs	Updates scheduler construction to include serialize_allocations.
tests/test_recover.rs	Updates scheduler construction to include serialize_allocations.
tests/test_orphaned_jobs.rs	Updates scheduler construction to include serialize_allocations.
tests/test_hpc.rs	Updates scheduler generation tests to include serialize_allocations.
tests/test_auto_schedule.rs	Updates scheduler creation tests to include serialize_allocations.
python_client/src/torc/openapi_client/models/slurm_scheduler_model.py	Regenerates Python model to include serialize_allocations.
julia_client/Torc/src/api/models/model_SlurmSchedulerModel.jl	Regenerates Julia model to include serialize_allocations.
julia_client/julia_client/docs/SlurmSchedulerModel.md	Updates Julia docs to include serialize_allocations.
docs/src/SUMMARY.md	Adds the new “Chained Allocations” doc page to the TOC.
docs/src/specialized/hpc/chained-allocations.md	New documentation page explaining allocation chaining and usage.
docs/src/core/reference/workflow-spec.md	Adds serialize_allocations to the workflow spec reference table.

Review details

• Files reviewed: 25/25 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The server’s create Slurm scheduler response can return serialize_allocations: null even though the stored column is NOT NULL, creating an API inconsistency that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

src/server/api/schedulers.rs:319

• create_slurm_scheduler() stores serialize_allocations as NOT NULL in the database, but if the request omits the field the response will still return serialize_allocations: None (since body is echoed back). This makes create responses inconsistent with get/list responses, which always return a boolean value based on the stored column.

        // Stored NOT NULL; an omitted value means "not serialized".
        let serialize_allocations = body.serialize_allocations.unwrap_or(false);

docs/src/specialized/hpc/chained-allocations.md:6

• Grammar: “partitions typically cap allocations at hours” is missing a quantifier and reads incorrectly.

sequential jobs at 2-4 hours each is weeks of serial work, but partitions typically cap allocations
at hours. You need one allocation to run as many jobs as fit, exit, and the next to pick up where it
left off.

• Files reviewed: 26/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.