Skip to content

feat: Add task chunking support to Nuke submitter#293

Closed
rickrams wants to merge 1 commit intoaws-deadline:mainlinefrom
rickrams:feat/task-chunking
Closed

feat: Add task chunking support to Nuke submitter#293
rickrams wants to merge 1 commit intoaws-deadline:mainlinefrom
rickrams:feat/task-chunking

Conversation

@rickrams
Copy link
Copy Markdown

@rickrams rickrams commented Feb 25, 2026

Add opt-in task chunking using the OpenJD TASK_CHUNKING extension. When enabled, frames are grouped into contiguous chunks instead of being dispatched individually, reducing per-task overhead. A future update could add non-contiguous chunking but for now this seems useful as is.

Changes:

  • New chunked job template with CHUNK[INT] Frame parameter, ChunkSize, and TargetRuntimeSeconds job parameters
  • RenderSettings: use_chunking, chunk_size, target_runtime_seconds fields (sticky, default off)
  • Submitter selects chunked template when use_chunking is enabled
  • UI: Enable task chunking checkbox with chunk size and target chunk duration controls

No adaptor changes needed - the existing NukeHandler.start_render() already handles the contiguous chunk frame range format (e.g. 1-10).

How was this change tested? Local testing of some 500 frame scenes in Nuke 15.

Please run the integration tests and paste the results below

Don't have this setup so no.

If installer/ was modified or a file was added/removed from src/, then update the installer tests and post the test results below

No modified

Did you run the "Job Bundle Output Tests"? If not, why not? If so, paste the test results here.

Don't have this setup so no.

Required: paste the contents of job_bundle_output_tests/test-job-bundle-results.txt here

Was this change documented?

Yes

Is this a breaking change?

No

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@rickrams rickrams requested a review from a team as a code owner February 25, 2026 18:41
@github-actions github-actions bot added the waiting-on-maintainers Waiting on the maintainers to review. label Feb 25, 2026
npmacl
npmacl reviewed Mar 3, 2026
View reviewed changes
src/deadline/nuke_submitter/default_nuke_chunked_job_template.yaml Outdated
@@ -0,0 +1,162 @@
specificationVersion: jobtemplate-2023-09
Copy link
Copy Markdown
Contributor

@npmacl npmacl Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't see a need for creating a new job template. I think it would be better to update the existing job template to support chunking. when running without chunking we could just use a chunk size of 1 with targetRuntimeSeconds set to 0.

Copy link
Copy Markdown
Author

@rickrams rickrams Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea - deleted the separate chunked template. Updated default_nuke_job_template.yaml in-place with TASK_CHUNKING extension and CHUNK[INT] Frame parameter. ChunkSize defaults to 1 (identical to old per-frame behavior).

src/deadline/nuke_submitter/ui/components/scene_settings_tab.py Outdated
Comment on lines +127 to +128
"Useful when frame render times vary. Leave at 0 to use a fixed\n"
"chunk size for all chunks."
Copy link
Copy Markdown
Contributor

@npmacl npmacl Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Useful when frame render times vary" seems doesn't seem like the right use-case for dynamic chunk sizing. this feature relies on the scheduler estimating how long each task will take based on the duration of previous tasks.

Copy link
Copy Markdown
Author

@rickrams rickrams Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good feedback, updated

src/deadline/nuke_submitter/ui/components/scene_settings_tab.py Outdated
Comment on lines +103 to +107
self.use_chunking_check = QCheckBox("Enable task chunking", self)
self.use_chunking_check.setToolTip(
"Group multiple frames into chunks to reduce application startup overhead per task"
)
lyt.addWidget(self.use_chunking_check, 5, 0)
Copy link
Copy Markdown
Contributor

@npmacl npmacl Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than adding a check box for choosing whether to enable chunking, I think we are better just using the spinner for chunk size. customers who want no chunking can just use chunkSize of 1 with dynamic chunking disabled.

@rickrams
feat: Add task chunking support to Nuke submitter
d893a6c 
Add task chunking using the OpenJD TASK_CHUNKING extension. The existing
job template now uses CHUNK[INT] for the Frame parameter with contiguous
chunks. Chunk size defaults to 1 (one frame per task, same as before).
Users increase chunk size to group frames and reduce per-task overhead.

Changes:
- Updated default job template with TASK_CHUNKING extension, CHUNK[INT]
  Frame parameter, ChunkSize (default 1), and TargetChunkDuration params
- RenderSettings: chunk_size and target_chunk_duration fields (sticky)
- UI: Chunk size and target chunk duration spinners in job settings
- Docs: Updated user guide, README, and CHANGELOG

No adaptor changes needed - the existing NukeHandler.start_render()
already handles the contiguous chunk frame range format (e.g. 1-10).

Note: Requires a worker agent that supports the TASK_CHUNKING extension.
Service-managed fleets always use a compatible version.

Signed-off-by: Rick Ramsay <49293857+rickrams@users.noreply.github.com>
@rickrams rickrams force-pushed the feat/task-chunking branch from 8bb2443 to d893a6c Compare March 3, 2026 22:08
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Mar 3, 2026

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@npmacl
Copy link
Copy Markdown
Contributor

npmacl commented Mar 19, 2026

tested running integration tests, they passed

Required test coverage of 69.0% reached. Total coverage: 70.83%
=================================================== slowest 5 durations ===================================================
2.68s call     test/unit/deadline_adaptor_for_nuke/CopyCatAdaptor/test_copycat_adaptor.py::test_execution_with_stubbed_nuke_fail
2.47s call     test/unit/deadline_adaptor_for_nuke/CopyCatAdaptor/test_copycat_adaptor.py::test_execution_with_stubbed_nuke_success
0.24s call     test/unit/deadline_adaptor_for_nuke/NukeAdaptor/test_adaptor.py::TestNukeAdaptor_on_cleanup::test_on_cleanup
0.24s call     test/unit/deadline_adaptor_for_nuke/NukeAdaptor/test_adaptor.py::TestNukeAdaptor_on_run::test_on_run
0.22s setup    test/unit/deadline_submitter_for_nuke/test_chunking.py::TestJobTemplate::test_template_has_task_chunking_extension
============================================= 155 passed, 8 skipped in 29.94s =============================================

@npmacl
Copy link
Copy Markdown
Contributor

npmacl commented Mar 19, 2026

did some manual testing, seems to work as expected.

the job bundle output tests fail though. Its doesn't seem to be a regression, the test code just has to be updated. I'll send you a zip with the results on slack, it should be pretty simple to update the tests accordingly.

@npmacl npmacl mentioned this pull request Mar 26, 2026
Open
@npmacl
Copy link
Copy Markdown
Contributor

npmacl commented Mar 26, 2026

We decided to move this to a different PR that includes a fix for the job output tests: #294.

@npmacl npmacl closed this Mar 26, 2026
@npmacl npmacl removed the waiting-on-maintainers Waiting on the maintainers to review. label Mar 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@npmacl npmacl npmacl left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rickrams @npmacl