Skip to content

Commit 0418f2b

Browse files
elliotguntonelliotgunton
authored
Add example using with_items for step/dag decorators (#1490)
Received a question on how to perform fan outs using the decorator syntax. It is already possible, so I added an example and expanded the user guide. Added multiple warnins as to what's keeping decorator syntax as "experimental", as I've received a few questions around that too - essentially we just want more feedback and to improve the UX as much as possible before graduating the feature. --------- Signed-off-by: Elliot Gunton <[email protected]>
1 parent f5cdc64 commit 0418f2b

File tree

4 files changed

+310
-2
lines changed

4 files changed

+310
-2
lines changed

‎docs/examples/workflows/experimental/new_decorators_fanout_loop.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# New Decorators Fanout Loop
2+
3+
4+
5+
This example shows how to pass extra Step/Task kwargs when calling a function.
6+
7+
This lets you perform fanouts using `with_items`, set conditions using `when` and more.
8+
9+
!!! warning
10+
11+
These features are not compatible with local-running of dags and steps, please see
12+
<https://github.com/argoproj-labs/hera/issues/1492>.
13+
14+
15+
=== "Hera"
16+
17+
```python linenums="1"
18+
from hera.shared import global_config
19+
from hera.workflows import Input, Workflow
20+
21+
global_config.experimental_features["decorator_syntax"] = True
22+
23+
24+
w = Workflow(
25+
generate_name="fanout-workflow-",
26+
)
27+
28+
29+
class PrintMessageInput(Input):
30+
message: str = ""
31+
an_int: int = 42
32+
33+
34+
@w.script()
35+
def print_message(inputs: PrintMessageInput):
36+
print(inputs.message)
37+
38+
39+
@w.script()
40+
def print_int(inputs: PrintMessageInput):
41+
print(inputs.an_int)
42+
43+
44+
@w.set_entrypoint
45+
@w.steps()
46+
def loop_example():
47+
print_message(
48+
PrintMessageInput(message="{{item}}"),
49+
name="print-str-message-loop-with-items",
50+
with_items=["hello world", "goodbye world"],
51+
)
52+
# For general use of loops in decorator functions, you will need to
53+
# use `.construct` to pass the `"{{item}}"` string.
54+
print_int(
55+
PrintMessageInput.construct(an_int="{{item}}"),
56+
name="print-int-loop-with-items",
57+
with_items=[42, 123, 321],
58+
)
59+
```
60+
61+
=== "YAML"
62+
63+
```yaml linenums="1"
64+
apiVersion: argoproj.io/v1alpha1
65+
kind: Workflow
66+
metadata:
67+
generateName: fanout-workflow-
68+
spec:
69+
entrypoint: loop-example
70+
templates:
71+
- name: print-message
72+
inputs:
73+
parameters:
74+
- name: message
75+
default: ''
76+
- name: an_int
77+
default: '42'
78+
script:
79+
image: python:3.9
80+
source: '{{inputs.parameters}}'
81+
args:
82+
- -m
83+
- hera.workflows.runner
84+
- -e
85+
- examples.workflows.experimental.new_decorators_fanout_loop:print_message
86+
command:
87+
- python
88+
env:
89+
- name: hera__script_pydantic_io
90+
value: ''
91+
- name: print-int
92+
inputs:
93+
parameters:
94+
- name: message
95+
default: ''
96+
- name: an_int
97+
default: '42'
98+
script:
99+
image: python:3.9
100+
source: '{{inputs.parameters}}'
101+
args:
102+
- -m
103+
- hera.workflows.runner
104+
- -e
105+
- examples.workflows.experimental.new_decorators_fanout_loop:print_int
106+
command:
107+
- python
108+
env:
109+
- name: hera__script_pydantic_io
110+
value: ''
111+
- name: loop-example
112+
steps:
113+
- - name: print-str-message-loop-with-items
114+
template: print-message
115+
withItems:
116+
- hello world
117+
- goodbye world
118+
arguments:
119+
parameters:
120+
- name: message
121+
value: '{{item}}'
122+
- name: an_int
123+
value: '42'
124+
- - name: print-int-loop-with-items
125+
template: print-int
126+
withItems:
127+
- 42
128+
- 123
129+
- 321
130+
arguments:
131+
parameters:
132+
- name: message
133+
value: ''
134+
- name: an_int
135+
value: '{{item}}'
136+
```
137+

‎docs/user-guides/decorators.md‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ tasks. It does this by tracking the parameters and artifacts passed between task
3636
must be written in a natural running order (which is also why it can be run locally). Note that only a basic "depends"
3737
relationship is deduced, more complex dependencies are not yet supported.
3838

39-
The `steps` decorator allows you to use the `parallel` function from `hera.workflows`, which is used to open a context under which all the steps will run in parallel.
39+
The `steps` decorator allows you to use the `parallel` function from `hera.workflows`, which is used to open a context
40+
under which all the steps will run in parallel.
4041

4142
The choice between `dag` and `steps` to arrange your templates mostly comes down to personal preference. If you want to
4243
run as many _dependent_ templates in parallel as possible then use a DAG, which will figure out which tasks can run
@@ -168,5 +169,52 @@ def my_steps() -> None:
168169

169170
We can simply call the script templates, passing the input objects in.
170171

171-
For more complex examples, including use of a dag, see
172+
173+
## Passing extra `kwargs`
174+
175+
In order to access normal `Step` and `Task` functionality, such as loops through `with_items`, you can pass extra kwargs
176+
when calling the function in the steps or dag function.
177+
178+
!!! warning
179+
180+
Running this particular `steps` function locally will not loop over the items in `with_items`. In general, the extra
181+
kwargs will not work with local-running `dag` and `steps` functions in the way you might expect, as there is no
182+
local-runtime for interpreting these values. Please see issue
183+
[#1492](https://github.com/argoproj-labs/hera/issues/1492) for updates on Hera's local runtime feature for
184+
decorators.
185+
186+
```py
187+
w = Workflow(
188+
generate_name="fanout-workflow-",
189+
)
190+
191+
192+
class PrintMessageInput(Input):
193+
message: str
194+
195+
196+
@w.script()
197+
def print_message(inputs: PrintMessageInput):
198+
print(inputs.message)
199+
200+
201+
@w.set_entrypoint
202+
@w.steps()
203+
def loop_example():
204+
print_message(
205+
PrintMessageInput(message="{{item}}"),
206+
name="print-message-loop-with-items",
207+
with_items=["hello world", "goodbye world"],
208+
)
209+
```
210+
211+
!!! warning
212+
213+
Your IDE/linter may complain about this function call, as it will not recognise the magic that Hera does at compile
214+
time. For that reason, until we can improve on this user experience and confirm it is the right way forward,
215+
decorators will remain experimental for the foreseeable future. Your input is appreciated in the
216+
[Hera Slack channel](https://cloud-native.slack.com/archives/C03NRMD9KPY) and
217+
[GitHub discussions](https://github.com/argoproj-labs/hera/discussions)!
218+
219+
For more examples, including use of a dag, see
172220
[the "experimental" examples](../examples/workflows/experimental/new_dag_decorator_params.md).

‎examples/workflows/experimental/new-decorators-fanout-loop.yaml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: Workflow
3+
metadata:
4+
generateName: fanout-workflow-
5+
spec:
6+
entrypoint: loop-example
7+
templates:
8+
- name: print-message
9+
inputs:
10+
parameters:
11+
- name: message
12+
default: ''
13+
- name: an_int
14+
default: '42'
15+
script:
16+
image: python:3.9
17+
source: '{{inputs.parameters}}'
18+
args:
19+
- -m
20+
- hera.workflows.runner
21+
- -e
22+
- examples.workflows.experimental.new_decorators_fanout_loop:print_message
23+
command:
24+
- python
25+
env:
26+
- name: hera__script_pydantic_io
27+
value: ''
28+
- name: print-int
29+
inputs:
30+
parameters:
31+
- name: message
32+
default: ''
33+
- name: an_int
34+
default: '42'
35+
script:
36+
image: python:3.9
37+
source: '{{inputs.parameters}}'
38+
args:
39+
- -m
40+
- hera.workflows.runner
41+
- -e
42+
- examples.workflows.experimental.new_decorators_fanout_loop:print_int
43+
command:
44+
- python
45+
env:
46+
- name: hera__script_pydantic_io
47+
value: ''
48+
- name: loop-example
49+
steps:
50+
- - name: print-str-message-loop-with-items
51+
template: print-message
52+
withItems:
53+
- hello world
54+
- goodbye world
55+
arguments:
56+
parameters:
57+
- name: message
58+
value: '{{item}}'
59+
- name: an_int
60+
value: '42'
61+
- - name: print-int-loop-with-items
62+
template: print-int
63+
withItems:
64+
- 42
65+
- 123
66+
- 321
67+
arguments:
68+
parameters:
69+
- name: message
70+
value: ''
71+
- name: an_int
72+
value: '{{item}}'

‎examples/workflows/experimental/new_decorators_fanout_loop.py‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
"""This example shows how to pass extra Step/Task kwargs when calling a function.
2+
3+
This lets you perform fanouts using `with_items`, set conditions using `when` and more.
4+
5+
!!! warning
6+
7+
These features are not compatible with local-running of dags and steps, please see
8+
<https://github.com/argoproj-labs/hera/issues/1492>.
9+
"""
10+
11+
from hera.shared import global_config
12+
from hera.workflows import Input, Workflow
13+
14+
global_config.experimental_features["decorator_syntax"] = True
15+
16+
17+
w = Workflow(
18+
generate_name="fanout-workflow-",
19+
)
20+
21+
22+
class PrintMessageInput(Input):
23+
message: str = ""
24+
an_int: int = 42
25+
26+
27+
@w.script()
28+
def print_message(inputs: PrintMessageInput):
29+
print(inputs.message)
30+
31+
32+
@w.script()
33+
def print_int(inputs: PrintMessageInput):
34+
print(inputs.an_int)
35+
36+
37+
@w.set_entrypoint
38+
@w.steps()
39+
def loop_example():
40+
print_message(
41+
PrintMessageInput(message="{{item}}"),
42+
name="print-str-message-loop-with-items",
43+
with_items=["hello world", "goodbye world"],
44+
)
45+
# For general use of loops in decorator functions, you will need to
46+
# use `.construct` to pass the `"{{item}}"` string.
47+
print_int(
48+
PrintMessageInput.construct(an_int="{{item}}"),
49+
name="print-int-loop-with-items",
50+
with_items=[42, 123, 321],
51+
)

0 commit comments

Comments
 (0)