Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix various bugs in dag/steps decorator #1221

Merged
merged 17 commits into from
Oct 3, 2024
Merged

Fix various bugs in dag/steps decorator #1221

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d7dbe1b
Allow no arg for dag decorator function
elliotgunton Sep 30, 2024
aafa3d5
Replace underscores in step/dag tasks
elliotgunton Sep 30, 2024
ccb7d4b
Raise error if Input in kwargs
elliotgunton Sep 30, 2024
24d6ff6
Add error for new script decorator incorrect use
elliotgunton Sep 30, 2024
7a79a66
Add ArtifactLoaders to new_dag_decorator_artifacts
elliotgunton Sep 30, 2024
19be598
Don't add path for Steps/DAG artifact inputs
elliotgunton Sep 30, 2024
df6819b
Ignore func kwargs when running locally
elliotgunton Oct 1, 2024
3f53b63
Fix output hoisting losing values for DAG/Steps
elliotgunton Oct 1, 2024
a2cf05b
fixup! Allow no arg for dag decorator function
elliotgunton Oct 1, 2024
250a274
fixup! Replace underscores in step/dag tasks
elliotgunton Oct 1, 2024
372c29a
fixup! Fix output hoisting losing values for DAG/Steps
elliotgunton Oct 1, 2024
615045e
fixup! Don't add path for Steps/DAG artifact inputs
elliotgunton Oct 1, 2024
06bc5ae
Fix bad returns from dag/steps
elliotgunton Oct 1, 2024
5b56891
Raise error if `result` or `exit_code` is set in dag
elliotgunton Oct 1, 2024
20fd421
Allow template set as a context in add_sub_node
elliotgunton Oct 1, 2024
1d80e4b
Fix `name=None` case, improve dag return code error checking
elliotgunton Oct 2, 2024
9cd73bb
Update user guide using wrong syntax
elliotgunton Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions docs/examples/workflows/experimental/new_dag_decorator_artifacts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from typing_extensions import Annotated

from hera.shared import global_config
from hera.workflows import Artifact, Input, Output, Workflow
from hera.workflows import Artifact, ArtifactLoader, Input, Output, Workflow

global_config.experimental_features["decorator_syntax"] = True

Expand All @@ -24,8 +24,8 @@


class ConcatInput(Input):
word_a: Annotated[str, Artifact(name="word_a")]
word_b: Annotated[str, Artifact(name="word_b")]
word_a: Annotated[str, Artifact(name="word_a", loader=ArtifactLoader.json)]
word_b: Annotated[str, Artifact(name="word_b", loader=ArtifactLoader.json)]


@w.script()
Expand Down Expand Up @@ -105,23 +105,21 @@
name: word_a
- from: '{{inputs.artifacts.artifact_b}}'
name: word_b
name: concat_1
name: concat-1
template: concat
- arguments:
artifacts:
- from: '{{tasks.concat_1.outputs.artifacts.an-artifact}}'
- from: '{{tasks.concat-1.outputs.artifacts.an-artifact}}'
name: word_a
- from: '{{tasks.concat_1.outputs.artifacts.an-artifact}}'
- from: '{{tasks.concat-1.outputs.artifacts.an-artifact}}'
name: word_b
depends: concat_1
depends: concat-1
name: concat-2-custom-name
template: concat
inputs:
artifacts:
- name: artifact_a
path: /tmp/hera-inputs/artifacts/artifact_a
- name: artifact_b
path: /tmp/hera-inputs/artifacts/artifact_b
name: worker
outputs:
artifacts:
Expand Down
38 changes: 19 additions & 19 deletions docs/examples/workflows/experimental/new_dag_decorator_inner_dag.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,34 +123,34 @@
source: '{{inputs.parameters}}'
- dag:
tasks:
- name: setup_task
- name: setup-task
template: setup
- arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_a}}'
- name: word_b
value: '{{tasks.setup_task.outputs.parameters.environment_parameter}}'
depends: setup_task
name: task_a
value: '{{tasks.setup-task.outputs.parameters.environment_parameter}}'
depends: setup-task
name: task-a
template: concat
- arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_b}}'
- name: word_b
value: '{{tasks.setup_task.outputs.result}}'
depends: setup_task
name: task_b
value: '{{tasks.setup-task.outputs.result}}'
depends: setup-task
name: task-b
template: concat
- arguments:
parameters:
- name: word_a
value: '{{tasks.task_a.outputs.result}}'
value: '{{tasks.task-a.outputs.result}}'
- name: word_b
value: '{{tasks.task_b.outputs.result}}'
depends: task_a && task_b
name: final_task
value: '{{tasks.task-b.outputs.result}}'
depends: task-a && task-b
name: final-task
template: concat
inputs:
parameters:
Expand All @@ -161,7 +161,7 @@
parameters:
- name: value
valueFrom:
parameter: '{{tasks.final_task.outputs.result}}'
parameter: '{{tasks.final-task.outputs.result}}'
- dag:
tasks:
- arguments:
Expand All @@ -170,24 +170,24 @@
value: dag_a
- name: value_b
value: '{{inputs.parameters.value_a}}'
name: sub_dag_a
name: sub-dag-a
template: worker
- arguments:
parameters:
- name: value_a
value: dag_b
- name: value_b
value: '{{inputs.parameters.value_b}}'
name: sub_dag_b
name: sub-dag-b
template: worker
- arguments:
parameters:
- name: value_a
value: '{{tasks.sub_dag_a.outputs.parameters.value}}'
value: '{{tasks.sub-dag-a.outputs.parameters.value}}'
- name: value_b
value: '{{tasks.sub_dag_b.outputs.parameters.value}}'
depends: sub_dag_a && sub_dag_b
name: sub_dag_c
value: '{{tasks.sub-dag-b.outputs.parameters.value}}'
depends: sub-dag-a && sub-dag-b
name: sub-dag-c
template: worker
inputs:
parameters:
Expand All @@ -198,6 +198,6 @@
parameters:
- name: value
valueFrom:
parameter: '{{tasks.sub_dag_c.outputs.parameters.value}}'
parameter: '{{tasks.sub-dag-c.outputs.parameters.value}}'
```

40 changes: 22 additions & 18 deletions docs/examples/workflows/experimental/new_dag_decorator_params.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@

class SetupOutput(Output):
environment_parameter: str
an_annotated_parameter: Annotated[int, Parameter(name="dummy-param")] # use an annotated non-str
an_annotated_parameter: Annotated[int, Parameter()] # use an annotated non-str, infer name from field
setup_config: Annotated[SetupConfig, Parameter(name="setup-config")] # use a pydantic BaseModel


Expand Down Expand Up @@ -71,7 +71,8 @@


class WorkerOutput(Output):
value: str
result_value: str
another_value: str


@w.set_entrypoint
Expand All @@ -87,7 +88,7 @@
task_b = concat(ConcatInput(word_a=worker_input.value_b, word_b=setup_task.result))
final_task = concat(ConcatInput(word_a=task_a.result, word_b=task_b.result))

return WorkerOutput(value=final_task.result)
return WorkerOutput(result_value=final_task.result, another_value=setup_task.an_annotated_parameter)
```

=== "YAML"
Expand All @@ -106,9 +107,9 @@
- name: environment_parameter
valueFrom:
path: /tmp/hera-outputs/parameters/environment_parameter
- name: dummy-param
- name: an_annotated_parameter
valueFrom:
path: /tmp/hera-outputs/parameters/dummy-param
path: /tmp/hera-outputs/parameters/an_annotated_parameter
- name: setup-config
valueFrom:
path: /tmp/hera-outputs/parameters/setup-config
Expand Down Expand Up @@ -156,40 +157,40 @@
source: '{{inputs.parameters}}'
- dag:
tasks:
- name: setup_task
- name: setup-task
template: setup
- arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_a}}'
- name: word_b
value: '{{tasks.setup_task.outputs.parameters.environment_parameter}}{{tasks.setup_task.outputs.parameters.dummy-param}}'
value: '{{tasks.setup-task.outputs.parameters.environment_parameter}}{{tasks.setup-task.outputs.parameters.an_annotated_parameter}}'
- name: concat_config
value: '{"reverse": false}'
depends: setup_task
name: task_a
depends: setup-task
name: task-a
template: concat
- arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_b}}'
- name: word_b
value: '{{tasks.setup_task.outputs.result}}'
value: '{{tasks.setup-task.outputs.result}}'
- name: concat_config
value: '{"reverse": false}'
depends: setup_task
name: task_b
depends: setup-task
name: task-b
template: concat
- arguments:
parameters:
- name: word_a
value: '{{tasks.task_a.outputs.result}}'
value: '{{tasks.task-a.outputs.result}}'
- name: word_b
value: '{{tasks.task_b.outputs.result}}'
value: '{{tasks.task-b.outputs.result}}'
- name: concat_config
value: '{"reverse": false}'
depends: task_a && task_b
name: final_task
depends: task-a && task-b
name: final-task
template: concat
inputs:
parameters:
Expand All @@ -203,8 +204,11 @@
name: worker
outputs:
parameters:
- name: value
- name: result_value
valueFrom:
parameter: '{{tasks.final_task.outputs.result}}'
parameter: '{{tasks.final-task.outputs.result}}'
- name: another_value
valueFrom:
parameter: '{{tasks.setup-task.outputs.parameters.an_annotated_parameter}}'
```

24 changes: 12 additions & 12 deletions docs/examples/workflows/experimental/new_decorators_auto_template_refs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@
clusterScope: true
name: my-cluster-workflow-template
template: run-setup-dag
- name: setup_task
- name: setup-task
templateRef:
clusterScope: true
name: my-cluster-workflow-template
Expand All @@ -126,11 +126,11 @@
- name: word_a
value: '{{inputs.parameters.value_a}}'
- name: word_b
value: '{{tasks.setup_task.outputs.parameters.environment_parameter}}{{tasks.setup_task.outputs.parameters.dummy-param}}'
value: '{{tasks.setup-task.outputs.parameters.environment_parameter}}{{tasks.setup-task.outputs.parameters.dummy-param}}'
- name: concat_config
value: '{"reverse": false}'
depends: setup_task
name: task_a
depends: setup-task
name: task-a
templateRef:
name: my-workflow-template
template: concat
Expand All @@ -139,24 +139,24 @@
- name: word_a
value: '{{inputs.parameters.value_b}}'
- name: word_b
value: '{{tasks.setup_task.outputs.result}}'
value: '{{tasks.setup-task.outputs.result}}'
- name: concat_config
value: '{"reverse": false}'
depends: setup_task
name: task_b
depends: setup-task
name: task-b
templateRef:
name: my-workflow-template
template: concat
- arguments:
parameters:
- name: word_a
value: '{{tasks.task_a.outputs.result}}'
value: '{{tasks.task-a.outputs.result}}'
- name: word_b
value: '{{tasks.task_b.outputs.result}}'
value: '{{tasks.task-b.outputs.result}}'
- name: concat_config
value: '{"reverse": false}'
depends: task_a && task_b
name: final_task
depends: task-a && task-b
name: final-task
templateRef:
name: my-workflow-template
template: concat
Expand All @@ -174,6 +174,6 @@
parameters:
- name: value
valueFrom:
parameter: '{{tasks.final_task.outputs.result}}'
parameter: '{{tasks.final-task.outputs.result}}'
```

18 changes: 9 additions & 9 deletions docs/examples/workflows/experimental/new_steps_decorator_with_parallel_steps.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,33 +138,33 @@
parameters:
- name: value
valueFrom:
parameter: '{{steps.final_step.outputs.result}}'
parameter: '{{steps.final-step.outputs.result}}'
steps:
- - name: setup_step
- - name: setup-step
template: setup
- - arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_a}}'
- name: word_b
value: '{{steps.setup_step.outputs.parameters.environment_parameter}}{{steps.setup_step.outputs.parameters.dummy-param}}'
name: step_a
value: '{{steps.setup-step.outputs.parameters.environment_parameter}}{{steps.setup-step.outputs.parameters.dummy-param}}'
name: step-a
template: concat
- arguments:
parameters:
- name: word_a
value: '{{inputs.parameters.value_b}}'
- name: word_b
value: '{{steps.setup_step.outputs.result}}'
name: step_b
value: '{{steps.setup-step.outputs.result}}'
name: step-b
template: concat
- - arguments:
parameters:
- name: word_a
value: '{{steps.step_a.outputs.result}}'
value: '{{steps.step-a.outputs.result}}'
- name: word_b
value: '{{steps.step_b.outputs.result}}'
name: final_step
value: '{{steps.step-b.outputs.result}}'
name: final-step
template: concat
```

13 changes: 13 additions & 0 deletions docs/examples/workflows/experimental/template_sets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@
return Output(result="Setting things up")


@templates.dag()
def my_dag():
setup(name="task-a")
setup(name="task-b")


w.add_template_set(templates)
```

Expand Down Expand Up @@ -52,5 +58,12 @@
value: ''
image: python:3.9
source: '{{inputs.parameters}}'
- dag:
tasks:
- name: task-a
template: setup
- name: task-b
template: setup
name: my-dag
```

Loading
Loading