feat: add task chunking

CHANGELOG.md

@@ -1,3 +1,17 @@
+## 0.18.17 (Unreleased)
+
+
+### Features
+* feat: Add task chunking support to Nuke submitter. Frames are now rendered using
+  the OpenJD TASK_CHUNKING extension with contiguous chunks. Chunk size defaults to 1
+  (one frame per task, same as before). Increase chunk size to group frames and reduce
+  per-task overhead. Optional target chunk duration enables dynamic chunk sizing.
+
+### Important
+* This version requires a worker agent that supports the TASK_CHUNKING extension.
+  Service-managed fleets always use a compatible version. If you use customer-managed
+  fleets, ensure your worker agents are updated before submitting jobs.
+
 ## 0.18.16 (2026-01-20)
 
 Added support for training CopyCat nodes. Users are now able to select to submit a CopyCat training job. A CopyCat training job will perform CopyCat training on the render farm, rather than on the local system.

README.md

@@ -26,6 +26,10 @@ This library requires:
 
 This package provides a Nuke plugin that creates jobs for AWS Deadline Cloud using the [AWS Deadline Cloud client library][deadline-cloud-client]. Based on the loaded comp it determines the files required, allows the user to specify render options, and builds an [OpenJD template][openjd] that defines the workflow.
 
+The submitter supports [task chunking][task-chunking], which groups multiple frames into contiguous chunks to reduce per-task overhead. When combined with the adaptor's sticky rendering, this provides optimal performance by eliminating both repeated application startup and scene loading time.
+
+[task-chunking]: https://docs.aws.amazon.com/deadline-cloud/latest/developerguide/build-job-bundle-chunking.html
+
 ## Adaptor
 
 The Nuke Adaptor implements the [OpenJD][openjd-adaptor-runtime] interface that allows render workloads to launch Nuke and feed it commands. This gives the following benefits:

docs/user_guide/using-submitter/render-submissions.md

@@ -25,6 +25,8 @@ The **Job-specific settings** tab has options specific to jobs created in Nuke.
   - *Override frame range* - Select this to render a different frame or frame range than is set in Nuke. Frame ranges follow the [Open Job Description](https://github.com/OpenJobDescription/openjd-specifications/wiki/2023-09-Template-Schemas#34111-intrangeexpr) pattern.
   - *Use proxy mode* - Manages whether to use [proxy mode](https://learn.foundry.com/nuke/9.0/content/getting_started/managing_scripts/proxy_mode.html) in the submitted job.
   - *Continue on error* - If set, try to continue rendering if Nuke encounters an error. If false, in the case of an error the task is failed.
+  - *Chunk size* - Number of frames to group into each chunk (1-150). Use 1 for one frame per task (default). Higher values group frames into contiguous chunks to reduce per-task overhead. For more information, see [Task chunking for job templates](https://docs.aws.amazon.com/deadline-cloud/latest/developerguide/build-job-bundle-chunking.html).
+  - *Target chunk duration (seconds)* - When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks, aiming for this duration per chunk. Leave at 0 to use a fixed chunk size for all chunks.
   - *Use timeouts* - Whether or not to use user configured timeouts.
   - *Render task timeout* - Maximum duration of each action which performs a render. Default is 6 days.
   - *Setup timeout* - Maximum duration of each action which sets up the job for rendering, such as scene load. Default is 1 day.

job_bundle_output_tests/cwd-path/expected_job_bundle/parameter_values.yaml

@@ -7,6 +7,10 @@ parameterValues:
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount

job_bundle_output_tests/cwd-path/expected_job_bundle/template.yaml

@@ -1,4 +1,6 @@
 specificationVersion: jobtemplate-2023-09
+extensions:
+- TASK_CHUNKING
 name: cwd-path.nk
 parameterDefinitions:
 - name: NukeScriptFile
@@ -20,6 +22,24 @@ parameterDefinitions:
   type: STRING
   description: The frames to render. E.g. 1-3,8,11-15
   minLength: 1
+- name: ChunkSize
+  type: INT
+  default: 1
+  minValue: 1
+  description: |
+    Number of frames per chunk. Use 1 for one frame per task (default). Higher values group frames into chunks to reduce per-task overhead.
+  userInterface:
+    control: SPIN_BOX
+    label: Chunk Size
+- name: TargetChunkDuration
+  type: INT
+  default: 0
+  minValue: 0
+  description: |
+    Optional target duration in seconds per chunk. When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks. Set to 0 to use a fixed chunk size for all chunks.
+  userInterface:
+    control: SPIN_BOX
+    label: Target Chunk Duration (Seconds)
 - name: WriteNode
   type: STRING
   userInterface:
@@ -64,8 +84,12 @@ steps:
   parameterSpace:
     taskParameterDefinitions:
     - name: Frame
-      type: INT
+      type: CHUNK[INT]
       range: '{{Param.Frames}}'
+      chunks:
+        defaultTaskCount: '{{Param.ChunkSize}}'
+        targetRuntimeSeconds: '{{Param.TargetChunkDuration}}'
+        rangeConstraint: CONTIGUOUS
   stepEnvironments:
   - name: Nuke
     description: Runs Nuke in the background with a script file loaded.
@@ -125,4 +149,4 @@ steps:
         - file://{{ Task.File.runData }}
         cancelation:
           mode: NOTIFY_THEN_TERMINATE
-        timeout: 518400
+        timeout: 518400

job_bundle_output_tests/frame/expected_job_bundle/parameter_values.yaml

@@ -9,6 +9,10 @@ parameterValues:
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount

job_bundle_output_tests/frame/expected_job_bundle/template.yaml

@@ -1,4 +1,6 @@
 specificationVersion: jobtemplate-2023-09
+extensions:
+- TASK_CHUNKING
 name: frame.nk
 parameterDefinitions:
 - name: NukeScriptFile
@@ -20,6 +22,24 @@ parameterDefinitions:
   type: STRING
   description: The frames to render. E.g. 1-3,8,11-15
   minLength: 1
+- name: ChunkSize
+  type: INT
+  default: 1
+  minValue: 1
+  description: |
+    Number of frames per chunk. Use 1 for one frame per task (default). Higher values group frames into chunks to reduce per-task overhead.
+  userInterface:
+    control: SPIN_BOX
+    label: Chunk Size
+- name: TargetChunkDuration
+  type: INT
+  default: 0
+  minValue: 0
+  description: |
+    Optional target duration in seconds per chunk. When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks. Set to 0 to use a fixed chunk size for all chunks.
+  userInterface:
+    control: SPIN_BOX
+    label: Target Chunk Duration (Seconds)
 - name: WriteNode
   type: STRING
   userInterface:
@@ -64,8 +84,12 @@ steps:
   parameterSpace:
     taskParameterDefinitions:
     - name: Frame
-      type: INT
+      type: CHUNK[INT]
       range: '{{Param.Frames}}'
+      chunks:
+        defaultTaskCount: '{{Param.ChunkSize}}'
+        targetRuntimeSeconds: '{{Param.TargetChunkDuration}}'
+        rangeConstraint: CONTIGUOUS
   stepEnvironments:
   - name: Nuke
     description: Runs Nuke in the background with a script file loaded.

job_bundle_output_tests/group-read/expected_job_bundle/asset_references.yaml

@@ -2,8 +2,8 @@ assetReferences:
   inputs:
     directories: []
     filenames:
-    - /normalized/cwd/colorbars.png
     - /normalized/job/bundle/dir/group-read.nk
+    - /normalized/cwd/colorbars.png
   outputs:
     directories: []
   referencedPaths: []

job_bundle_output_tests/group-read/expected_job_bundle/parameter_values.yaml

@@ -7,6 +7,10 @@ parameterValues:
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount

job_bundle_output_tests/group-read/expected_job_bundle/template.yaml

@@ -1,4 +1,6 @@
 specificationVersion: jobtemplate-2023-09
+extensions:
+- TASK_CHUNKING
 name: group-read.nk
 parameterDefinitions:
 - name: NukeScriptFile
@@ -20,6 +22,24 @@ parameterDefinitions:
   type: STRING
   description: The frames to render. E.g. 1-3,8,11-15
   minLength: 1
+- name: ChunkSize
+  type: INT
+  default: 1
+  minValue: 1
+  description: |
+    Number of frames per chunk. Use 1 for one frame per task (default). Higher values group frames into chunks to reduce per-task overhead.
+  userInterface:
+    control: SPIN_BOX
+    label: Chunk Size
+- name: TargetChunkDuration
+  type: INT
+  default: 0
+  minValue: 0
+  description: |
+    Optional target duration in seconds per chunk. When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks. Set to 0 to use a fixed chunk size for all chunks.
+  userInterface:
+    control: SPIN_BOX
+    label: Target Chunk Duration (Seconds)
 - name: WriteNode
   type: STRING
   userInterface:
@@ -63,8 +83,12 @@ steps:
   parameterSpace:
     taskParameterDefinitions:
     - name: Frame
-      type: INT
+      type: CHUNK[INT]
       range: '{{Param.Frames}}'
+      chunks:
+        defaultTaskCount: '{{Param.ChunkSize}}'
+        targetRuntimeSeconds: '{{Param.TargetChunkDuration}}'
+        rangeConstraint: CONTIGUOUS
   stepEnvironments:
   - name: Nuke
     description: Runs Nuke in the background with a script file loaded.

job_bundle_output_tests/multi-load-save/expected_job_bundle/parameter_values.yaml

@@ -7,6 +7,10 @@ parameterValues:
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount

job_bundle_output_tests/multi-load-save/expected_job_bundle/template.yaml

@@ -1,4 +1,6 @@
 specificationVersion: jobtemplate-2023-09
+extensions:
+- TASK_CHUNKING
 name: multi-load-save.nk
 parameterDefinitions:
 - name: NukeScriptFile
@@ -20,6 +22,24 @@ parameterDefinitions:
   type: STRING
   description: The frames to render. E.g. 1-3,8,11-15
   minLength: 1
+- name: ChunkSize
+  type: INT
+  default: 1
+  minValue: 1
+  description: |
+    Number of frames per chunk. Use 1 for one frame per task (default). Higher values group frames into chunks to reduce per-task overhead.
+  userInterface:
+    control: SPIN_BOX
+    label: Chunk Size
+- name: TargetChunkDuration
+  type: INT
+  default: 0
+  minValue: 0
+  description: |
+    Optional target duration in seconds per chunk. When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks. Set to 0 to use a fixed chunk size for all chunks.
+  userInterface:
+    control: SPIN_BOX
+    label: Target Chunk Duration (Seconds)
 - name: WriteNode
   type: STRING
   userInterface:
@@ -66,8 +86,12 @@ steps:
   parameterSpace:
     taskParameterDefinitions:
     - name: Frame
-      type: INT
+      type: CHUNK[INT]
       range: '{{Param.Frames}}'
+      chunks:
+        defaultTaskCount: '{{Param.ChunkSize}}'
+        targetRuntimeSeconds: '{{Param.TargetChunkDuration}}'
+        rangeConstraint: CONTIGUOUS
   stepEnvironments:
   - name: Nuke
     description: Runs Nuke in the background with a script file loaded.

job_bundle_output_tests/noise-saver/expected_job_bundle/parameter_values.yaml

@@ -7,6 +7,10 @@ parameterValues:
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount

job_bundle_output_tests/noise-saver/expected_job_bundle/template.yaml

@@ -1,4 +1,6 @@
 specificationVersion: jobtemplate-2023-09
+extensions:
+- TASK_CHUNKING
 name: noise-saver.nk
 parameterDefinitions:
 - name: NukeScriptFile
@@ -20,6 +22,24 @@ parameterDefinitions:
   type: STRING
   description: The frames to render. E.g. 1-3,8,11-15
   minLength: 1
+- name: ChunkSize
+  type: INT
+  default: 1
+  minValue: 1
+  description: |
+    Number of frames per chunk. Use 1 for one frame per task (default). Higher values group frames into chunks to reduce per-task overhead.
+  userInterface:
+    control: SPIN_BOX
+    label: Chunk Size
+- name: TargetChunkDuration
+  type: INT
+  default: 0
+  minValue: 0
+  description: |
+    Optional target duration in seconds per chunk. When set, the scheduler dynamically adjusts chunk sizes based on observed runtimes of completed chunks. Set to 0 to use a fixed chunk size for all chunks.
+  userInterface:
+    control: SPIN_BOX
+    label: Target Chunk Duration (Seconds)
 - name: WriteNode
   type: STRING
   userInterface:
@@ -64,8 +84,12 @@ steps:
   parameterSpace:
     taskParameterDefinitions:
     - name: Frame
-      type: INT
+      type: CHUNK[INT]
       range: '{{Param.Frames}}'
+      chunks:
+        defaultTaskCount: '{{Param.ChunkSize}}'
+        targetRuntimeSeconds: '{{Param.TargetChunkDuration}}'
+        rangeConstraint: CONTIGUOUS
   stepEnvironments:
   - name: Nuke
     description: Runs Nuke in the background with a script file loaded.

job_bundle_output_tests/ocio/expected_job_bundle/asset_references.yaml

@@ -1,8 +1,8 @@
 assetReferences:
   inputs:
     directories:
-    - /normalized/job/bundle/dir/luts
     - /tmp/luts
+    - /normalized/job/bundle/dir/luts
     filenames:
     - /normalized/job/bundle/dir/config.ocio
     - /normalized/job/bundle/dir/ocio.nk

job_bundle_output_tests/ocio/expected_job_bundle/parameter_values.yaml

@@ -3,12 +3,16 @@ parameterValues:
   value: 1-100
 - name: NukeScriptFile
   value: /normalized/job/bundle/dir/ocio.nk
-- name: OCIOConfigPath
-  value: /normalized/job/bundle/dir/config.ocio
 - name: ProxyMode
   value: 'false'
 - name: ContinueOnError
   value: 'false'
+- name: ChunkSize
+  value: 1
+- name: TargetChunkDuration
+  value: 0
+- name: OCIOConfigPath
+  value: /normalized/job/bundle/dir/config.ocio
 - name: deadline:targetTaskRunStatus
   value: READY
 - name: deadline:maxFailedTasksCount