Add serialize_allocations to chain Slurm allocations

api/openapi.codegen.yaml

@@ -7736,6 +7736,18 @@ components:
           type:
           - string
           - 'null'
+        serialize_allocations:
+          type:
+          - boolean
+          - 'null'
+          description: |-
+            Run this scheduler's allocations strictly one at a time.
+
+            When set, every allocation submitted for this scheduler shares one Slurm job
+            name and carries `--dependency=singleton`, so Slurm serializes them. Submit N
+            allocations up front and they chain: each runs until its walltime can no longer
+            fit a ready job, exits, and the next starts. Used for long sequential workflows
+            that outlive any single allocation.
         tmp:
           type:
           - string

api/openapi.yaml

@@ -7736,6 +7736,18 @@ components:
           type:
           - string
           - 'null'
+        serialize_allocations:
+          type:
+          - boolean
+          - 'null'
+          description: |-
+            Run this scheduler's allocations strictly one at a time.
+
+            When set, every allocation submitted for this scheduler shares one Slurm job
+            name and carries `--dependency=singleton`, so Slurm serializes them. Submit N
+            allocations up front and they chain: each runs until its walltime can no longer
+            fit a ready job, exits, and the next starts. Used for long sequential workflows
+            that outlive any single allocation.
         tmp:
           type:
           - string

docs/src/SUMMARY.md

@@ -78,6 +78,7 @@
   - [Slurm Overview](./specialized/hpc/slurm-workflows.md)
   - [Submitting Slurm Workflows](./specialized/hpc/submit-slurm-workflow.md)
   - [Multi-Node Jobs](./specialized/hpc/multi-node-jobs.md)
+  - [Chained Allocations](./specialized/hpc/chained-allocations.md)
   - [Advanced Slurm Configuration](./specialized/hpc/slurm.md)
   - [Slurm Exit Codes](./specialized/hpc/slurm-exit-codes.md)
   - [Debugging Slurm Workflows](./specialized/hpc/debugging-slurm.md)

docs/src/core/reference/workflow-spec.md

@@ -266,19 +266,20 @@ A single rule within a failure handler for handling specific exit codes.
 
 Defines a Slurm HPC job scheduler configuration.
 
-| Name              | Type    | Default      | Description                                  |
-| ----------------- | ------- | ------------ | -------------------------------------------- |
-| `name`            | string  | none         | Name of the scheduler (used for referencing) |
-| `account`         | string  | _required_   | Slurm account                                |
-| `partition`       | string  | none         | Slurm partition name                         |
-| `nodes`           | integer | `1`          | Number of nodes to allocate                  |
-| `walltime`        | string  | `"01:00:00"` | Wall time limit                              |
-| `mem`             | string  | none         | Memory specification                         |
-| `gres`            | string  | none         | Generic resources (e.g., GPUs)               |
-| `qos`             | string  | none         | Quality of service                           |
-| `ntasks_per_node` | integer | none         | Number of tasks per node                     |
-| `tmp`             | string  | none         | Temporary storage specification              |
-| `extra`           | string  | none         | Additional Slurm parameters                  |
+| Name                    | Type    | Default      | Description                                                                                                             |
+| ----------------------- | ------- | ------------ | ----------------------------------------------------------------------------------------------------------------------- |
+| `name`                  | string  | none         | Name of the scheduler (used for referencing)                                                                            |
+| `account`               | string  | _required_   | Slurm account                                                                                                           |
+| `partition`             | string  | none         | Slurm partition name                                                                                                    |
+| `nodes`                 | integer | `1`          | Number of nodes to allocate                                                                                             |
+| `walltime`              | string  | `"01:00:00"` | Wall time limit                                                                                                         |
+| `mem`                   | string  | none         | Memory specification                                                                                                    |
+| `gres`                  | string  | none         | Generic resources (e.g., GPUs)                                                                                          |
+| `qos`                   | string  | none         | Quality of service                                                                                                      |
+| `ntasks_per_node`       | integer | none         | Number of tasks per node                                                                                                |
+| `tmp`                   | string  | none         | Temporary storage specification                                                                                         |
+| `extra`                 | string  | none         | Additional Slurm parameters                                                                                             |
+| `serialize_allocations` | boolean | `false`      | Run this scheduler's allocations one at a time. See [Chained Allocations](../../specialized/hpc/chained-allocations.md) |
 
 ## ExecutionConfig
 

docs/src/specialized/hpc/chained-allocations.md

@@ -0,0 +1,102 @@
+# Chained Allocations
+
+Some workflows need more wall time than any single Slurm allocation provides. A chain of 500
+sequential jobs at 2-4 hours each is weeks of serial work, but partitions typically cap allocations
+at hours. You need one allocation to run as many jobs as fit, exit, and the next to pick up where it
+left off.
+
+Set `serialize_allocations` on a Slurm scheduler and torc submits every allocation for that
+scheduler under one shared Slurm job name with `--dependency=singleton`. Slurm then runs them
+strictly one at a time. Submit them all up front and they chain themselves, with no long-running
+process on the login node.
+
+## Configuring a Scheduler
+
+```yaml
+slurm_schedulers:
+  - name: chain
+    account: my_account
+    walltime: "12:00:00"
+    nodes: 1
+    serialize_allocations: true
+
+resource_requirements:
+  - name: serial
+    num_cpus: 104
+    memory: "200g"
+    runtime: "PT4H"
+```
+
+A complete runnable spec is at `examples/yaml/chained_allocations.yaml`.
+
+Or on an existing scheduler:
+
+```bash
+torc slurm create <workflow_id> -n chain -a my_account -W 12:00:00 --serialize-allocations
+torc slurm update <scheduler_id> --serialize-allocations true
+```
+
+Then submit the whole chain at once:
+
+```bash
+torc slurm schedule-nodes <workflow_id> -n 167
+```
+
+All 167 allocations enter the queue immediately. Slurm starts one, holds the rest, and releases the
+next each time the current one ends.
+
+## How Many Allocations to Submit
+
+Divide the total work by what one allocation can absorb. A worker claims jobs until its remaining
+wall time can no longer fit the next one, so with a 12-hour walltime and a declared `runtime` of
+`PT4H`, each allocation completes at least three jobs:
+
+```
+allocations = ceil(total_jobs / floor(walltime / runtime))
+            = ceil(500 / floor(12 / 4)) = 167
+```
+
+Round up. Over-submitting is cheap: once the workflow has no runnable jobs left, the finishing
+worker cancels every allocation still queued for the workflow, so the surplus never starts.
+
+## Why the Chain Beats Scheduling on Shutdown
+
+A worker could submit its own replacement as it exits, but that pays Slurm's full
+submit-and-schedule latency at every link — 167 times over — and makes each link's submission depend
+on its predecessor exiting cleanly. A chained allocation is already sitting in the queue when its
+predecessor ends, so it is typically released the moment the slot frees. On clusters that accrue age
+priority for dependency-held jobs (`PriorityFlags=ACCRUE_ALWAYS` in the Slurm config), each link
+also builds priority while its predecessor runs; by default Slurm starts the age clock only when the
+dependency clears.
+
+## Wall Time and Job Runtime
+
+The server only hands a worker jobs whose declared `runtime` fits in the allocation's remaining wall
+time. This is what makes the handoff clean: an allocation stops claiming once the next job no longer
+fits, idles briefly, and exits, releasing the slot early rather than sitting idle until walltime.
+
+Declare `runtime` at or above the worst case you expect. If a job overruns its declared runtime it
+is terminated when the walltime expires, and jobs downstream of it are left blocked with nothing to
+unblock them — which ends the chain, since the remaining queued allocations have no runnable work.
+
+## Scope of the Chain
+
+The shared job name is derived from the workflow ID and the scheduler ID, so:
+
+- Two schedulers in one workflow chain independently.
+- The same workflow submitted twice chains independently per scheduler.
+- Allocations added later — by a second `schedule-nodes` call, or by a `schedule_nodes` action
+  firing from a compute node — join the existing chain rather than running alongside it.
+
+Slurm scopes `singleton` to a job name **per user**, so the name is prefixed with `torc-` to keep
+the chain from serializing against your unrelated Slurm jobs.
+
+Because the chain depends on every allocation sharing one fixed name,
+`torc slurm schedule-nodes --job-prefix` is rejected for a serialized scheduler: a per-invocation
+prefix would change the name and fork the chain.
+
+## Interaction with `extra`
+
+`extra` is emitted after torc's own `#SBATCH` directives, so a `--dependency` set there overrides
+the generated `--dependency=singleton` and breaks the chain. Use `extra` for unrelated flags
+(`--reservation`, `--constraint`) when serializing allocations.

examples/yaml/chained_allocations.yaml

@@ -0,0 +1,58 @@
+name: "Chained Allocations"
+description: "Sequential jobs spanning multiple Slurm allocations chained with serialize_allocations"
+
+# A sequence of jobs whose total runtime exceeds any single allocation's walltime.
+# The scheduler below sets serialize_allocations, so every allocation is submitted
+# under one shared Slurm job name with --dependency=singleton: Slurm runs them one
+# at a time, and each picks up where its predecessor left off.
+#
+# See docs: specialized/hpc/chained-allocations.md
+
+jobs:
+  - name: "step1"
+    command: "echo 'Running step 1' && sleep 10"
+    resource_requirements: "serial"
+
+  - name: "step2"
+    command: "echo 'Running step 2' && sleep 10"
+    depends_on: ["step1"]
+    resource_requirements: "serial"
+
+  - name: "step3"
+    command: "echo 'Running step 3' && sleep 10"
+    depends_on: ["step2"]
+    resource_requirements: "serial"
+
+  - name: "step4"
+    command: "echo 'Running step 4' && sleep 10"
+    depends_on: ["step3"]
+    resource_requirements: "serial"
+
+resource_requirements:
+  # Declare runtime at or above the worst case: the server only hands a worker a job
+  # whose declared runtime fits the allocation's remaining walltime, which is what
+  # makes each allocation exit cleanly when the next job no longer fits.
+  - name: "serial"
+    num_cpus: 4
+    num_nodes: 1
+    memory: "8g"
+    runtime: "PT2H"
+
+slurm_schedulers:
+  - name: "chain"
+    account: "demo_project"
+    nodes: 1
+    walltime: "04:00:00"
+    serialize_allocations: true
+
+actions:
+  # Submit the whole chain up front. With a 4-hour walltime and 2-hour job runtime,
+  # each allocation completes floor(4 / 2) = 2 jobs, so 4 jobs need
+  # ceil(4 / 2) = 2 allocations. Over-submitting is safe: once no runnable jobs
+  # remain, the finishing worker cancels the surplus still sitting in the queue.
+  - trigger_type: "on_jobs_ready"
+    action_type: "schedule_nodes"
+    jobs: ["step1"]
+    scheduler: "chain"
+    scheduler_type: "slurm"
+    num_allocations: 2

julia_client/Torc/src/api/models/model_SlurmSchedulerModel.jl

@@ -15,6 +15,7 @@
         ntasks_per_node=nothing,
         partition=nothing,
         qos=nothing,
+        serialize_allocations=nothing,
         tmp=nothing,
         walltime=nothing,
         workflow_id=nothing,
@@ -30,6 +31,7 @@
     - ntasks_per_node::Int64
     - partition::String
     - qos::String
+    - serialize_allocations::Bool : Run this scheduler's allocations strictly one at a time.  When set, every allocation submitted for this scheduler shares one Slurm job name and carries `--dependency=singleton`, so Slurm serializes them. Submit N allocations up front and they chain: each runs until its walltime can no longer fit a ready job, exits, and the next starts. Used for long sequential workflows that outlive any single allocation.
     - tmp::String
     - walltime::String
     - workflow_id::Int64
@@ -45,18 +47,19 @@ Base.@kwdef mutable struct SlurmSchedulerModel <: OpenAPI.APIModel
     ntasks_per_node::Union{Nothing, Int64} = nothing
     partition::Union{Nothing, String} = nothing
     qos::Union{Nothing, String} = nothing
+    serialize_allocations::Union{Nothing, Bool} = nothing
     tmp::Union{Nothing, String} = nothing
     walltime::Union{Nothing, String} = nothing
     workflow_id::Union{Nothing, Int64} = nothing
 
-    function SlurmSchedulerModel(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, tmp, walltime, workflow_id, )
-        o = new(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, tmp, walltime, workflow_id, )
+    function SlurmSchedulerModel(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, serialize_allocations, tmp, walltime, workflow_id, )
+        o = new(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, serialize_allocations, tmp, walltime, workflow_id, )
         OpenAPI.validate_properties(o)
         return o
     end
 end # type SlurmSchedulerModel
 
-const _property_types_SlurmSchedulerModel = Dict{Symbol,String}(Symbol("account")=>"String", Symbol("extra")=>"String", Symbol("gres")=>"String", Symbol("id")=>"Int64", Symbol("mem")=>"String", Symbol("name")=>"String", Symbol("nodes")=>"Int64", Symbol("ntasks_per_node")=>"Int64", Symbol("partition")=>"String", Symbol("qos")=>"String", Symbol("tmp")=>"String", Symbol("walltime")=>"String", Symbol("workflow_id")=>"Int64", )
+const _property_types_SlurmSchedulerModel = Dict{Symbol,String}(Symbol("account")=>"String", Symbol("extra")=>"String", Symbol("gres")=>"String", Symbol("id")=>"Int64", Symbol("mem")=>"String", Symbol("name")=>"String", Symbol("nodes")=>"Int64", Symbol("ntasks_per_node")=>"Int64", Symbol("partition")=>"String", Symbol("qos")=>"String", Symbol("serialize_allocations")=>"Bool", Symbol("tmp")=>"String", Symbol("walltime")=>"String", Symbol("workflow_id")=>"Int64", )
 OpenAPI.property_type(::Type{ SlurmSchedulerModel }, name::Symbol) = Union{Nothing,eval(Base.Meta.parse(_property_types_SlurmSchedulerModel[name]))}
 
 function OpenAPI.check_required(o::SlurmSchedulerModel)
@@ -78,6 +81,7 @@ function OpenAPI.validate_properties(o::SlurmSchedulerModel)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("ntasks_per_node"), o.ntasks_per_node)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("partition"), o.partition)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("qos"), o.qos)
+    OpenAPI.validate_property(SlurmSchedulerModel, Symbol("serialize_allocations"), o.serialize_allocations)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("tmp"), o.tmp)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("walltime"), o.walltime)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("workflow_id"), o.workflow_id)
@@ -106,6 +110,7 @@ function OpenAPI.validate_property(::Type{ SlurmSchedulerModel }, name::Symbol,
 
 
 
+
     if name === Symbol("workflow_id")
         OpenAPI.validate_param(name, "SlurmSchedulerModel", :format, val, "int64")
     end

julia_client/julia_client/docs/SlurmSchedulerModel.md

@@ -14,6 +14,7 @@ Name | Type | Description | Notes
 **ntasks_per_node** | **Int64** |  | [optional] [default to nothing]
 **partition** | **String** |  | [optional] [default to nothing]
 **qos** | **String** |  | [optional] [default to nothing]
+**serialize_allocations** | **Bool** | Run this scheduler's allocations strictly one at a time.  When set, every allocation submitted for this scheduler shares one Slurm job name and carries `--dependency=singleton`, so Slurm serializes them. Submit N allocations up front and they chain: each runs until its walltime can no longer fit a ready job, exits, and the next starts. Used for long sequential workflows that outlive any single allocation. | [optional] [default to nothing]
 **tmp** | **String** |  | [optional] [default to nothing]
 **walltime** | **String** |  | [default to nothing]
 **workflow_id** | **Int64** |  | [default to nothing]