Refactor Deadline Submission JobInfo #49

kalisp · 2024-10-18T14:01:28Z

Changelog Description

This PR tries to make DL submissions easier, mostly handling of JobInfo segment. Collection of values is moved out of submission phase much earlier where values could be filled from default values in Settings or overriden by selected list of field by artist in Publisher UI.

Move to universal collector should help with potential validation of filled values.
Profiles in Settings should allow more control of values (like priority, limit_groups) based on task_types or hosts etc.

Additional info

Should be used with ynput/ayon-core#958 !!

All DCC need to be tested:

Testing notes:

do regular DL publishing
test that nothing breaks
update ayon+settings://deadline/publish/CollectJobInfo with new defaults and check if they propagete to Publisher
update values in Publisher if they propagate to DL

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

It is handling EnvironmentKey* type of fields

BigRoy

I like where this is going! :)

client/ayon_deadline/lib.py

client/ayon_deadline/abstract_submit_deadline.py

BigRoy · 2024-10-18T15:44:58Z

server/settings/publish_plugins.py

+    chunk_size: int = SettingsField(999, title="Frames per Task")
+    priority: int = SettingsField(50, title="Priority")
+    group: str = SettingsField("", title="Group")
+    limit_groups: list[LimitGroupsSubmodel] = SettingsField(
+        default_factory=list,
+        title="Limit Groups",
+    )
+    concurrent_tasks: int = SettingsField(
+        1, title="Number of concurrent tasks")
+    department: str = SettingsField("", title="Department")
+    use_gpu: bool = SettingsField("", title="Use GPU")
+    job_delay: str = SettingsField(
+        "", title="Delay job",
+        placeholder="dd:hh:mm:ss"
+    )
+    use_published: bool = SettingsField(True, title="Use Published scene")
+    asset_dependencies: bool = SettingsField(True, title="Use Asset dependencies")
+    workfile_dependency: bool = SettingsField(True, title="Workfile Dependency")
+    multiprocess: bool = SettingsField(False, title="Multiprocess")


Would be very nice to get description argument set for each of these.

BigRoy · 2024-10-18T15:50:23Z

server/settings/publish_plugins.py

    enabled: bool = SettingsField(True)
    optional: bool = SettingsField(title="Optional")
    active: bool = SettingsField(title="Active")


With all the other attributes removed from these models we can now make one EnableStateModel class instead that we use for all of these making this file much smaller.

The question is if these jobs should be actually optional at all as if they would be disabled, submit publish job will fail.

The question is if these jobs should be actually optional at all as if they would be disabled, submit publish job will fail.

I guess the idea would have been that it maybe would allow to disable just the farm publishing support for an instance - but there creators definitely are not configured that way. So I agree - may make no sense to be optional, or even be disabled, currently?

BigRoy · 2024-10-18T15:52:56Z

client/ayon_deadline/plugins/publish/aftereffects/submit_aftereffects_deadline.py

-        dln_job_info.add_render_job_env_var()
+
+        # already collected explicit values for rendered Frames
+        if not dln_job_info.Frames:


I actually think this is a place where we should not be setting this in a Deadline specific way but have it accessible on the instance even outside of the job info. So Frames would be build up from instance.data["frames"] instead so that it could turn out to be non-Deadline specific. However, not critical for now and we can do that later. Might be worth a TODO in code or an issue after this PR.

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

client/ayon_deadline/plugins/publish/global/submit_publish_job.py

server/settings/publish_plugins.py

Not yet working on AOVs.

Deadline portion must be there for ValidateExpectedFiles

client/ayon_deadline/lib.py

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

Co-authored-by: Jakub Trllo <[email protected]>

That way FARM_FAMILIES could be purged only 'deadline' after all creators would add it.

client/ayon_deadline/lib.py

Some DCC might not collect that in time Co-authored-by: Mustafa Jafar <[email protected]>

MustafaJafar

It works pretty well in Houdini.
Publishing cache doesn't error any more. Cache jobs don't error any more.
Publishing renders works fine with different targets. Render jobs work fine.
(there's still one bug mentioned in #49 (comment) where export job attributes override render job attributes. I'll try to debug it on my side.)

client/ayon_deadline/abstract_submit_deadline.py

Co-authored-by: Mustafa Jafar <[email protected]>

MustafaJafar · 2024-11-15T11:44:57Z

Tested Nuke. it works.
maybe we can update the label of the jobs to differentiate between bake and render but that's a separate issue I guess.

moonyuet

Tested with 3dsMax. Single Camera submission is working but not the multiple camera one.

DEBUG: Using Redshift...published scene wont be used..
INFO: Using D:\sh_boss_proj\interior_lenny\lenny_bedroom\asset\publish\workfile\workfileGeneric\v042\il_asset_workfileGeneric_v042.max for render/export.
DEBUG: Submitting 3dsMax render..
DEBUG: Submitting jobs for multiple cameras..
Traceback (most recent call last):
  File "C:\Users\Kayla\AppData\Local\Ynput\AYON\dependency_packages\ayon_2402141620_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process
    runner(*args)
  File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\abstract_submit_deadline.py", line 116, in process
    job_id = self.process_submission()
  File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\plugins\publish\max\submit_max_deadline.py", line 96, in process_submission
    payload = self._use_published_name_for_multiples(
  File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\plugins\publish\max\submit_max_deadline.py", line 286, in _use_published_name_for_multiples
    job_info = self.get_job_info_through_camera(cam)
  File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\plugins\publish\max\submit_max_deadline.py", line 197, in get_job_info_through_camera
    job_info.OutputDirectory += os.path.dirname(filepath)
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

moonyuet · 2024-11-15T15:20:01Z

client/ayon_deadline/plugins/publish/max/submit_max_deadline.py

+        # do not add expected files for multiCamera
+        if instance.data.get("multiCamera"):
+            job_info.OutputDirectory = None
+            job_info.OutputFilename = None


Suggested change

# do not add expected files for multiCamera

if instance.data.get("multiCamera"):

job_info.OutputDirectory = None

job_info.OutputFilename = None

# do not add expected files for multiCamera

# if instance.data.get("multiCamera"):

# job_info.OutputDirectory = None

# job_info.OutputFilename = None

As 3dsMax supports multi camera submission by scene in current workflow, so maybe we can comment out this one first and we can come back to this when we refactor the multi-camera submission by cameras in the future.

This will submit the renders correctly after commenting out

Or maybe resetting it only to ''? It seems to have issue with concatenating None + str

Or maybe resetting it only to ''? It seems to have issue with concatenating None + str

still not working.
It would instead raise AttributeError: 'str' object has no attribute 'serialize' in submit_max_deadline.py

DEBUG: Using Redshift...published scene wont be used.. INFO: Using D:\sh_boss_proj\interior_lenny\lenny_bedroom\asset\publish\workfile\workfileGeneric\v045\il_asset_workfileGeneric_v045.max for render/export. DEBUG: Submitting 3dsMax render.. DEBUG: Submitting jobs for multiple cameras.. DEBUG: Executing [GET] settings?project_name=interior_lenny&variant=test_thumbnail-prod-04&bundle_name=test_thumbnail-prod-04&site_id=lumpy-fervent-newt DEBUG: http://localhost:5000 "GET /api/settings?project_name=interior_lenny&variant=test_thumbnail-prod-04&bundle_name=test_thumbnail-prod-04&site_id=lumpy-fervent-newt HTTP/1.1" 200 113377 DEBUG: Response <RestApiResponse [200]> Traceback (most recent call last): File "C:\Users\Kayla\AppData\Local\Ynput\AYON\dependency_packages\ayon_2402141620_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process runner(*args) File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\abstract_submit_deadline.py", line 116, in process job_id = self.process_submission() File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\plugins\publish\max\submit_max_deadline.py", line 101, in process_submission self.assemble_payload(job_info, plugin_info), File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\abstract_submit_deadline.py", line 301, in assemble_payload "JobInfo": job.serialize(), File "D:\ayon-addon_template\ayon-deadline\client\ayon_deadline\lib.py", line 505, in serialize serialized.update(attribute.serialize()) AttributeError: 'str' object has no attribute 'serialize'

Yes - makes sense. You can't assign directly to it like that because you'd be replacing the attribute completely.

You should be doing:

Suggested change

# do not add expected files for multiCamera

if instance.data.get("multiCamera"):

job_info.OutputDirectory = None

job_info.OutputFilename = None

# do not add expected files for multiCamera

if instance.data.get("multiCamera"):

job_info.OutputDirectory.clear()

job_info.OutputFilename.clear()

To just clear the dict.

Yes - makes sense. You can't assign directly to it like that because you'd be replacing the attribute completely.

You should be doing:

To just clear the dict.

Tested with this suggestion, it can successfully render the multi-camera accordingly and also cleans up duplicated the job info in terms of OutputDirectory and OutputFIlename.

kalisp added 13 commits October 16, 2024 18:50

New Settings for CollectJobInfoItem

d1a883b

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

Matched implemented artist overrides definitions

f72e87f

Added collected jobInfo object

2b29403

Use DeadlineJobInfo as collector object

cb0d24d

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Removed six dependency

37c0a0d

Implemented new generic method applicable for all DCC

3358838

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Removed unneeded imports

60da8b9

Do not pass empty string to DeadlineJobInfo

ae3c739

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

Removed unneded imports

1c8990a

Reimplemented get_job_info, moved to generic impl

57860ce

Removed storign deadline info to instance

10f566f

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

Updates to DeadlineJobInfo to match previous implementation

bbb12b7

It is handling EnvironmentKey* type of fields

Added Frames field for explicit rendering

c26c997

kalisp requested review from BigRoy, antirotor and iLLiCiTiT October 18, 2024 14:01

kalisp self-assigned this Oct 18, 2024

BigRoy reviewed Oct 18, 2024

View reviewed changes

kalisp added 8 commits October 18, 2024 18:15

Run CollectJobInfo only locally

e1eba7f

Implemented explicit frames filtering on simple files

8062462

Not yet working on AOVs.

Reworked removal of deadline in submit job

ec908e0

Deadline portion must be there for ValidateExpectedFiles

Removed empty line

64b22ff

Added empty line at the end

0d3125d

Formatting

c206b24

Removed forgotten dev code

803c25c

Refactor batchname

201c5b8

BigRoy added the type: enhancement Improvement of existing functionality or minor addition label Oct 21, 2024

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/lib.py Show resolved Hide resolved

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/lib.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Nov 13, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py Show resolved Hide resolved

kalisp and others added 13 commits November 13, 2024 14:07

Fix wrong access to job_delay default value

9732866

Use list comprehension for pool enum

42148a9

Co-authored-by: Jakub Trllo <[email protected]>

Use list comprehension for machines enum

c9d1c4a

Co-authored-by: Jakub Trllo <[email protected]>

Use list comprehension

d58ff4d

Co-authored-by: Jakub Trllo <[email protected]>

Fix condition

005a5fb

Co-authored-by: Jakub Trllo <[email protected]>

Fix copy to deepcopy

deb8aa8

Co-authored-by: Jakub Trllo <[email protected]>

Fix formatting

ba26549

Use list comprehension for groups

faa1cba

Use list comprehension for limit groups

ea9b2d6

Use list comprehension for machine_limit defaults

a307520

Fix wrong assignment

7e0ad18

Fix deepcopy from import

df7a85e

Added deadline to instance families

5c4ebd0

That way FARM_FAMILIES could be purged only 'deadline' after all creators would add it.

MustafaJafar reviewed Nov 13, 2024

View reviewed changes

client/ayon_deadline/lib.py Outdated Show resolved Hide resolved

Make frames optional

c05ddc2

Some DCC might not collect that in time Co-authored-by: Mustafa Jafar <[email protected]>

kalisp requested review from LiborBatek and MilaKudr November 14, 2024 11:00

MustafaJafar requested review from MustafaJafar, iLLiCiTiT and BigRoy November 14, 2024 17:12

MustafaJafar approved these changes Nov 14, 2024

View reviewed changes

MustafaJafar reviewed Nov 14, 2024

View reviewed changes

client/ayon_deadline/abstract_submit_deadline.py Outdated Show resolved Hide resolved

kalisp and others added 3 commits November 15, 2024 11:40

Fix updates of job_info between jobs

d3cbe0b

Co-authored-by: Mustafa Jafar <[email protected]>

add missing import

269b943

remove unused import

fa9327c

moonyuet requested changes Nov 15, 2024

View reviewed changes

moonyuet reviewed Nov 15, 2024

View reviewed changes

Fix propagation updated job_info

3aa671c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Deadline Submission JobInfo #49

Refactor Deadline Submission JobInfo #49

kalisp commented Oct 18, 2024 •

edited by moonyuet

Loading

BigRoy left a comment

BigRoy Oct 18, 2024

BigRoy Oct 18, 2024

kalisp Nov 11, 2024 •

edited

Loading

BigRoy Nov 11, 2024

BigRoy Oct 18, 2024

MustafaJafar left a comment

MustafaJafar commented Nov 15, 2024

moonyuet left a comment

moonyuet Nov 15, 2024 •

edited

Loading

kalisp Nov 15, 2024

moonyuet Nov 16, 2024 •

edited

Loading

BigRoy Nov 16, 2024

moonyuet Nov 18, 2024

Refactor Deadline Submission JobInfo #49

Are you sure you want to change the base?

Refactor Deadline Submission JobInfo #49

Conversation

kalisp commented Oct 18, 2024 • edited by moonyuet Loading

Changelog Description

Additional info

Testing notes:

BigRoy left a comment

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

kalisp Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

BigRoy Nov 11, 2024

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

MustafaJafar left a comment

Choose a reason for hiding this comment

MustafaJafar commented Nov 15, 2024

moonyuet left a comment

Choose a reason for hiding this comment

moonyuet Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

kalisp Nov 15, 2024

Choose a reason for hiding this comment

moonyuet Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

BigRoy Nov 16, 2024

Choose a reason for hiding this comment

moonyuet Nov 18, 2024

Choose a reason for hiding this comment

kalisp commented Oct 18, 2024 •

edited by moonyuet

Loading

kalisp Nov 11, 2024 •

edited

Loading

moonyuet Nov 15, 2024 •

edited

Loading

moonyuet Nov 16, 2024 •

edited

Loading