Add serialize_allocations to chain Slurm allocations

api/openapi.codegen.yaml

@@ -7736,6 +7736,18 @@ components:
           type:
           - string
           - 'null'
+        serialize_allocations:
+          type:
+          - boolean
+          - 'null'
+          description: |-
+            Run this scheduler's allocations strictly one at a time.
+
+            When set, every allocation submitted for this scheduler shares one Slurm job
+            name and carries `--dependency=singleton`, so Slurm serializes them. Submit N
+            allocations up front and they chain: each runs until its walltime can no longer
+            fit a ready job, exits, and the next starts. Used for long sequential workflows
+            that outlive any single allocation.
         tmp:
           type:
           - string

api/openapi.yaml

@@ -7736,6 +7736,18 @@ components:
           type:
           - string
           - 'null'
+        serialize_allocations:
+          type:
+          - boolean
+          - 'null'
+          description: |-
+            Run this scheduler's allocations strictly one at a time.
+
+            When set, every allocation submitted for this scheduler shares one Slurm job
+            name and carries `--dependency=singleton`, so Slurm serializes them. Submit N
+            allocations up front and they chain: each runs until its walltime can no longer
+            fit a ready job, exits, and the next starts. Used for long sequential workflows
+            that outlive any single allocation.
         tmp:
           type:
           - string

docs/src/SUMMARY.md

@@ -78,6 +78,7 @@
   - [Slurm Overview](./specialized/hpc/slurm-workflows.md)
   - [Submitting Slurm Workflows](./specialized/hpc/submit-slurm-workflow.md)
   - [Multi-Node Jobs](./specialized/hpc/multi-node-jobs.md)
+  - [Chained Allocations](./specialized/hpc/chained-allocations.md)
   - [Advanced Slurm Configuration](./specialized/hpc/slurm.md)
   - [Slurm Exit Codes](./specialized/hpc/slurm-exit-codes.md)
   - [Debugging Slurm Workflows](./specialized/hpc/debugging-slurm.md)

docs/src/core/reference/workflow-spec.md

@@ -266,19 +266,20 @@ A single rule within a failure handler for handling specific exit codes.
 
 Defines a Slurm HPC job scheduler configuration.
 
-| Name              | Type    | Default      | Description                                  |
-| ----------------- | ------- | ------------ | -------------------------------------------- |
-| `name`            | string  | none         | Name of the scheduler (used for referencing) |
-| `account`         | string  | _required_   | Slurm account                                |
-| `partition`       | string  | none         | Slurm partition name                         |
-| `nodes`           | integer | `1`          | Number of nodes to allocate                  |
-| `walltime`        | string  | `"01:00:00"` | Wall time limit                              |
-| `mem`             | string  | none         | Memory specification                         |
-| `gres`            | string  | none         | Generic resources (e.g., GPUs)               |
-| `qos`             | string  | none         | Quality of service                           |
-| `ntasks_per_node` | integer | none         | Number of tasks per node                     |
-| `tmp`             | string  | none         | Temporary storage specification              |
-| `extra`           | string  | none         | Additional Slurm parameters                  |
+| Name                    | Type    | Default      | Description                                                                                                             |
+| ----------------------- | ------- | ------------ | ----------------------------------------------------------------------------------------------------------------------- |
+| `name`                  | string  | none         | Name of the scheduler (used for referencing)                                                                            |
+| `account`               | string  | _required_   | Slurm account                                                                                                           |
+| `partition`             | string  | none         | Slurm partition name                                                                                                    |
+| `nodes`                 | integer | `1`          | Number of nodes to allocate                                                                                             |
+| `walltime`              | string  | `"01:00:00"` | Wall time limit                                                                                                         |
+| `mem`                   | string  | none         | Memory specification                                                                                                    |
+| `gres`                  | string  | none         | Generic resources (e.g., GPUs)                                                                                          |
+| `qos`                   | string  | none         | Quality of service                                                                                                      |
+| `ntasks_per_node`       | integer | none         | Number of tasks per node                                                                                                |
+| `tmp`                   | string  | none         | Temporary storage specification                                                                                         |
+| `extra`                 | string  | none         | Additional Slurm parameters                                                                                             |
+| `serialize_allocations` | boolean | `false`      | Run this scheduler's allocations one at a time. See [Chained Allocations](../../specialized/hpc/chained-allocations.md) |
 
 ## ExecutionConfig
 

docs/src/specialized/hpc/chained-allocations.md

@@ -0,0 +1,93 @@
+# Chained Allocations
+
+Some workflows need more wall time than any single Slurm allocation provides. A chain of 500
+sequential jobs at 2-4 hours each is weeks of serial work, but partitions typically cap allocations
+at hours. You need one allocation to run as many jobs as fit, exit, and the next to pick up where it
+left off.
+
+Set `serialize_allocations` on a Slurm scheduler and torc submits every allocation for that
+scheduler under one shared Slurm job name with `--dependency=singleton`. Slurm then runs them
+strictly one at a time. Submit them all up front and they chain themselves, with no long-running
+process on the login node.
+
+## Configuring a Scheduler
+
+```yaml
+slurm_schedulers:
+  - name: chain
+    account: my_account
+    walltime: "12:00:00"
+    nodes: 1
+    serialize_allocations: true
+
+resource_requirements:
+  - name: serial
+    num_cpus: 104
+    memory: "200g"
+    runtime: "PT4H"
+```
+
+Or on an existing scheduler:
+
+```bash
+torc slurm create <workflow_id> -n chain -a my_account -W 12:00:00 --serialize-allocations
+torc slurm update <scheduler_id> --serialize-allocations true
+```
+
+Then submit the whole chain at once:
+
+```bash
+torc slurm schedule-nodes <workflow_id> -n 167
+```
+
+All 167 allocations enter the queue immediately. Slurm starts one, holds the rest, and releases the
+next each time the current one ends.
+
+## How Many Allocations to Submit
+
+Divide the total work by what one allocation can absorb. A worker claims jobs until its remaining
+wall time can no longer fit the next one, so with a 12-hour walltime and a declared `runtime` of
+`PT4H`, each allocation completes at least three jobs:
+
+```
+allocations = ceil(total_jobs / floor(walltime / runtime))
+            = ceil(500 / floor(12 / 4)) = 167
+```
+
+Round up. Over-submitting is cheap: once the workflow has no runnable jobs left, the finishing
+worker cancels every allocation still queued for the workflow, so the surplus never starts.
+
+## Why the Chain Beats Scheduling on Shutdown
+
+A worker could submit its own replacement as it exits, but each replacement would enter the queue
+with no accrued age and pay full queue wait — 167 times over. A chained allocation is queued from
+the start and accrues priority while its predecessor runs, so it is typically ready to start the
+moment the slot frees.
+
+## Wall Time and Job Runtime
+
+The server only hands a worker jobs whose declared `runtime` fits in the allocation's remaining wall
+time. This is what makes the handoff clean: an allocation stops claiming once the next job no longer
+fits, idles briefly, and exits, releasing the slot early rather than sitting idle until walltime.
+
+Declare `runtime` at or above the worst case you expect. If a job overruns its declared runtime it
+is terminated when the walltime expires, and jobs downstream of it are left blocked with nothing to
+unblock them — which ends the chain, since the remaining queued allocations have no runnable work.
+
+## Scope of the Chain
+
+The shared job name is derived from the workflow ID and the scheduler ID, so:
+
+- Two schedulers in one workflow chain independently.
+- The same workflow submitted twice chains independently per scheduler.
+- Allocations added later — by a second `schedule-nodes` call, or by a `schedule_nodes` action
+  firing from a compute node — join the existing chain rather than running alongside it.
+
+Slurm scopes `singleton` to a job name **per user**, so the name is prefixed with `torc-` to keep
+the chain from serializing against your unrelated Slurm jobs.
+
+## Interaction with `extra`
+
+`extra` is emitted after torc's own `#SBATCH` directives, so a `--dependency` set there overrides
+the generated `--dependency=singleton` and breaks the chain. Use `extra` for unrelated flags
+(`--reservation`, `--constraint`) when serializing allocations.

julia_client/Torc/src/api/models/model_SlurmSchedulerModel.jl

@@ -15,6 +15,7 @@
         ntasks_per_node=nothing,
         partition=nothing,
         qos=nothing,
+        serialize_allocations=nothing,
         tmp=nothing,
         walltime=nothing,
         workflow_id=nothing,
@@ -30,6 +31,7 @@
     - ntasks_per_node::Int64
     - partition::String
     - qos::String
+    - serialize_allocations::Bool : Run this scheduler's allocations strictly one at a time.  When set, every allocation submitted for this scheduler shares one Slurm job name and carries `--dependency=singleton`, so Slurm serializes them. Submit N allocations up front and they chain: each runs until its walltime can no longer fit a ready job, exits, and the next starts. Used for long sequential workflows that outlive any single allocation.
     - tmp::String
     - walltime::String
     - workflow_id::Int64
@@ -45,18 +47,19 @@ Base.@kwdef mutable struct SlurmSchedulerModel <: OpenAPI.APIModel
     ntasks_per_node::Union{Nothing, Int64} = nothing
     partition::Union{Nothing, String} = nothing
     qos::Union{Nothing, String} = nothing
+    serialize_allocations::Union{Nothing, Bool} = nothing
     tmp::Union{Nothing, String} = nothing
     walltime::Union{Nothing, String} = nothing
     workflow_id::Union{Nothing, Int64} = nothing
 
-    function SlurmSchedulerModel(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, tmp, walltime, workflow_id, )
-        o = new(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, tmp, walltime, workflow_id, )
+    function SlurmSchedulerModel(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, serialize_allocations, tmp, walltime, workflow_id, )
+        o = new(account, extra, gres, id, mem, name, nodes, ntasks_per_node, partition, qos, serialize_allocations, tmp, walltime, workflow_id, )
         OpenAPI.validate_properties(o)
         return o
     end
 end # type SlurmSchedulerModel
 
-const _property_types_SlurmSchedulerModel = Dict{Symbol,String}(Symbol("account")=>"String", Symbol("extra")=>"String", Symbol("gres")=>"String", Symbol("id")=>"Int64", Symbol("mem")=>"String", Symbol("name")=>"String", Symbol("nodes")=>"Int64", Symbol("ntasks_per_node")=>"Int64", Symbol("partition")=>"String", Symbol("qos")=>"String", Symbol("tmp")=>"String", Symbol("walltime")=>"String", Symbol("workflow_id")=>"Int64", )
+const _property_types_SlurmSchedulerModel = Dict{Symbol,String}(Symbol("account")=>"String", Symbol("extra")=>"String", Symbol("gres")=>"String", Symbol("id")=>"Int64", Symbol("mem")=>"String", Symbol("name")=>"String", Symbol("nodes")=>"Int64", Symbol("ntasks_per_node")=>"Int64", Symbol("partition")=>"String", Symbol("qos")=>"String", Symbol("serialize_allocations")=>"Bool", Symbol("tmp")=>"String", Symbol("walltime")=>"String", Symbol("workflow_id")=>"Int64", )
 OpenAPI.property_type(::Type{ SlurmSchedulerModel }, name::Symbol) = Union{Nothing,eval(Base.Meta.parse(_property_types_SlurmSchedulerModel[name]))}
 
 function OpenAPI.check_required(o::SlurmSchedulerModel)
@@ -78,6 +81,7 @@ function OpenAPI.validate_properties(o::SlurmSchedulerModel)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("ntasks_per_node"), o.ntasks_per_node)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("partition"), o.partition)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("qos"), o.qos)
+    OpenAPI.validate_property(SlurmSchedulerModel, Symbol("serialize_allocations"), o.serialize_allocations)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("tmp"), o.tmp)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("walltime"), o.walltime)
     OpenAPI.validate_property(SlurmSchedulerModel, Symbol("workflow_id"), o.workflow_id)
@@ -106,6 +110,7 @@ function OpenAPI.validate_property(::Type{ SlurmSchedulerModel }, name::Symbol,
 
 
 
+
     if name === Symbol("workflow_id")
         OpenAPI.validate_param(name, "SlurmSchedulerModel", :format, val, "int64")
     end

julia_client/julia_client/docs/SlurmSchedulerModel.md

@@ -14,6 +14,7 @@ Name | Type | Description | Notes
 **ntasks_per_node** | **Int64** |  | [optional] [default to nothing]
 **partition** | **String** |  | [optional] [default to nothing]
 **qos** | **String** |  | [optional] [default to nothing]
+**serialize_allocations** | **Bool** | Run this scheduler's allocations strictly one at a time.  When set, every allocation submitted for this scheduler shares one Slurm job name and carries `--dependency=singleton`, so Slurm serializes them. Submit N allocations up front and they chain: each runs until its walltime can no longer fit a ready job, exits, and the next starts. Used for long sequential workflows that outlive any single allocation. | [optional] [default to nothing]
 **tmp** | **String** |  | [optional] [default to nothing]
 **walltime** | **String** |  | [default to nothing]
 **workflow_id** | **Int64** |  | [default to nothing]

python_client/src/torc/openapi_client/models/slurm_scheduler_model.py

@@ -17,7 +17,7 @@
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictInt, StrictStr
+from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictInt, StrictStr
 from typing import Any, ClassVar, Dict, List, Optional
 from typing import Optional, Set
 from typing_extensions import Self
@@ -36,10 +36,11 @@ class SlurmSchedulerModel(BaseModel):
     ntasks_per_node: Optional[StrictInt] = None
     partition: Optional[StrictStr] = None
     qos: Optional[StrictStr] = None
+    serialize_allocations: Optional[StrictBool] = Field(default=None, description="Run this scheduler's allocations strictly one at a time.  When set, every allocation submitted for this scheduler shares one Slurm job name and carries `--dependency=singleton`, so Slurm serializes them. Submit N allocations up front and they chain: each runs until its walltime can no longer fit a ready job, exits, and the next starts. Used for long sequential workflows that outlive any single allocation.")
     tmp: Optional[StrictStr] = None
     walltime: StrictStr
     workflow_id: StrictInt
-    __properties: ClassVar[List[str]] = ["account", "extra", "gres", "id", "mem", "name", "nodes", "ntasks_per_node", "partition", "qos", "tmp", "walltime", "workflow_id"]
+    __properties: ClassVar[List[str]] = ["account", "extra", "gres", "id", "mem", "name", "nodes", "ntasks_per_node", "partition", "qos", "serialize_allocations", "tmp", "walltime", "workflow_id"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -120,6 +121,11 @@ def to_dict(self) -> Dict[str, Any]:
         if self.qos is None and "qos" in self.model_fields_set:
             _dict['qos'] = None
 
+        # set to None if serialize_allocations (nullable) is None
+        # and model_fields_set contains the field
+        if self.serialize_allocations is None and "serialize_allocations" in self.model_fields_set:
+            _dict['serialize_allocations'] = None
+
         # set to None if tmp (nullable) is None
         # and model_fields_set contains the field
         if self.tmp is None and "tmp" in self.model_fields_set:
@@ -147,6 +153,7 @@ def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
             "ntasks_per_node": obj.get("ntasks_per_node"),
             "partition": obj.get("partition"),
             "qos": obj.get("qos"),
+            "serialize_allocations": obj.get("serialize_allocations"),
             "tmp": obj.get("tmp"),
             "walltime": obj.get("walltime"),
             "workflow_id": obj.get("workflow_id")