Skip to content

Commit 0f9cda6

Browse files
elliotguntonelliotgunton
authored
Add loops walkthrough (with_item/with_param, fanout/fanin) (#660)
**Pull Request Checklist** - [x] Part of #627 - [x] ~Tests added~ Docs only - [x] Documentation/examples added - [x] [Good commit messages](https://cbea.ms/git-commit/) and/or PR title --------- Signed-off-by: Elliot Gunton <[email protected]>
1 parent 7a584be commit 0f9cda6

File tree

2 files changed

+320
-0
lines changed

2 files changed

+320
-0
lines changed

‎docs/getting-started/walk-through/loops.md‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
# Loops
2+
3+
When writing Workflows, you may want to reuse a single template over a set of inputs. Argo exposes two mechanisms for
4+
this looping, which are "with items" and "with param". These mechanisms function in exactly the same way, but as the
5+
name suggests, "with param" lets you use a parameter to loop over, while "with items" is generally for a hard-coded list
6+
of items. When using loops in Argo, the template will run in parallel for all the items; the items will be launched
7+
sequentially but the running times may overlap. If you do not want to loop over the items in parallel, you should use a
8+
[Synchronization](https://argoproj.github.io/argo-workflows/synchronization/) mechanism.
9+
10+
## Loops in Hera
11+
12+
In pure Argo YAML specification, `withItems` and `withParam` are single values or JSON objects. In Hera, we can pass any
13+
`Parameter` or serializable object, plus, `with_items` and `with_params` work exactly the same for hard-coded values.
14+
15+
## A Simple `with_items` Example
16+
17+
Consider the [Hello World](hello-world.md) example:
18+
19+
```py
20+
@script()
21+
def echo(message: str):
22+
print(message)
23+
24+
25+
with Workflow(
26+
generate_name="hello-world-",
27+
entrypoint="steps",
28+
) as w:
29+
with Steps(name="steps"):
30+
echo(arguments={"message": "Hello world!"})
31+
```
32+
33+
We can easily convert this to call `echo` for multiple strings; the only changes we need to make are in the function
34+
call. First, specify the list of items you want to echo in the `with_items` kwarg:
35+
36+
```py
37+
echo(
38+
arguments={"message": "Hello world!"},
39+
with_items=["Hello world!", "I'm looping!", "Goodbye world!"],
40+
)
41+
```
42+
43+
Now, we need to replace the value of the `message` argument. In Argo, you would use the `"{{item}}"` expression syntax,
44+
which is also what we use in Hera:
45+
46+
```py
47+
echo(
48+
arguments={"message": "{{item}}"},
49+
with_items=["Hello world!", "I'm looping!", "Goodbye world!"],
50+
)
51+
```
52+
53+
When running this Workflow, each value of `with_items` is passed to the `{{item}}` expression and runs in an independent
54+
instance of the `echo` script container. Your Workflow logs should look something like:
55+
56+
```console
57+
hello-world-9cf9j-echo-3186990983: Hello world!
58+
hello-world-9cf9j-echo-4182774221: I'm looping!
59+
hello-world-9cf9j-echo-1812072106: Goodbye world!
60+
```
61+
62+
## `with_items` Using a Dictionary
63+
64+
We mentioned we can use any serializable object, so let's see how dictionaries are handled.
65+
66+
The `{{item}}` syntax represents the whole "item" passed in to the argument, so in the "hello world" example above, that
67+
translates to just a string. For dictionaries, this `{{item}}` would translate to the whole dictionary. If instead we
68+
want to pass values from the item dictionary to the function arguments, we provide them with Argo's key access syntax:
69+
`{{item.key}}`.
70+
71+
Let's create a workflow to process everyone's favorite bubble tea orders!
72+
73+
First, a function that takes the customer's name, the drink flavor, ice level and sugar level:
74+
75+
```py
76+
@script()
77+
def make_bubble_tea(
78+
name: str,
79+
flavor: str,
80+
ice_level: float,
81+
sugar_level: float,
82+
):
83+
print(
84+
f"Making {name}'s {flavor} bubble tea with {ice_level:.0%} ice and {sugar_level:.0%} sugar."
85+
)
86+
87+
```
88+
89+
Now, a Workflow with a `Steps` context:
90+
91+
```py
92+
with Workflow(
93+
generate_name="make-drinks-",
94+
entrypoint="steps",
95+
) as w:
96+
with Steps(name="steps"):
97+
make_bubble_tea(...)
98+
```
99+
100+
And now for each argument of `make_bubble_tea`, we can let Hera infer from the values in `with_item`! We just need to
101+
pass a list of dictionaries, with the keys matching the `make_bubble_tea` arguments: "name", "flavor", "ice_level" and
102+
"sugar_level":
103+
104+
```py
105+
with Workflow(
106+
generate_name="make-drinks-",
107+
entrypoint="steps",
108+
) as w:
109+
with Steps(name="steps"):
110+
make_bubble_tea(
111+
with_items=[
112+
{
113+
"name": "Elliot",
114+
"flavor": "Taro Milk Tea",
115+
"ice_level": 0.25,
116+
"sugar_level": 0.75,
117+
},
118+
{
119+
"name": "Flaviu",
120+
"flavor": "Brown Sugar Milk Tea",
121+
"ice_level": 1.00,
122+
"sugar_level": 0.5,
123+
},
124+
{
125+
"name": "Sambhav",
126+
"flavor": "Green Tea",
127+
"ice_level": 0.5,
128+
"sugar_level": 0.25,
129+
},
130+
],
131+
)
132+
```
133+
134+
Running this Workflow, in the UI we'll see a fanout of three nodes, and the logs for "All" containers will show:
135+
136+
```console
137+
make-drinks-h2qgq-make-bubble-tea-3759662853: Making Elliot's Taro Milk Tea bubble tea with 25% ice and 75% sugar.
138+
make-drinks-h2qgq-make-bubble-tea-470512305: Making Sambhav's Green Tea bubble tea with 50% ice and 25% sugar.
139+
make-drinks-h2qgq-make-bubble-tea-615962639: Making Flaviu's Brown Sugar Milk Tea bubble tea with 100% ice and 50% sugar.
140+
```
141+
142+
Remember in the above example, we could swap out `with_item` for `with_param` and get the same output. `with_param` is
143+
useful for passing dynamically generated lists and fanning out to process the list which we'll learn in the next section.
144+
145+
## Dynamic Fanout Using `with_param`
146+
147+
Let's improve our bubble tea maker by generating a dynamic list of orders! To do this, we'll need a new `create_orders`
148+
function. We're going to make use of the script's `result` output parameter by dumping the orders to stdout.
149+
150+
Let's make our order randomizer:
151+
152+
```py
153+
@script()
154+
def create_orders():
155+
import json
156+
import random
157+
158+
names = ["Elliot", "Flaviu", "Sambhav"]
159+
flavors = ["Brown Sugar Milk Tea", "Green Tea", "Taro Milk Tea"]
160+
levels = [0, 0.25, 0.5, 0.75, 1.0]
161+
162+
orders = []
163+
for _ in range(random.randint(4, 7)):
164+
orders.append(
165+
{
166+
"name": random.choice(names),
167+
"flavor": random.choice(flavors),
168+
"ice_level": random.choice(levels),
169+
"sugar_level": random.choice(levels),
170+
}
171+
)
172+
173+
print(json.dumps(orders, indent=4)) # indent is just used here for nice human-readable logs
174+
```
175+
176+
> **Note:** we must import any modules used within the function itself, as Hera currently only passes the source lines
177+
> of the function to Argo. If you need to import modules not in the standard Python image, use a custom image as
178+
> described in [the `script` decorator](hello-world.md#the-script-decorator) section, or see the **experimental**
179+
> [callable script](../../examples/workflows/callable_script.md) example.
180+
181+
Now we can construct a Workflow that calls `create_orders`, and passes its `result` to `make_bubble_tea`. We'll need to
182+
hold onto the `Step` returned from the `create_orders` call, and change `with_items` to `with_param` to use `.result`.
183+
We'll keep the `arguments` the same, as the `.result` will be a json-encoded *list* of *dictionaries*!
184+
185+
```py
186+
with Workflow(
187+
generate_name="make-drinks-",
188+
entrypoint="steps",
189+
) as w:
190+
with Steps(name="steps"):
191+
orders = create_orders()
192+
make_bubble_tea(with_param=orders.result)
193+
```
194+
195+
<details> <summary>Click to expand for logs. A Workflow run will look <i>something</i> like this. Remember, it's all
196+
random!</summary>
197+
198+
```console
199+
make-drinks-t49mm-create-orders-628494701: [
200+
make-drinks-t49mm-create-orders-628494701: {
201+
make-drinks-t49mm-create-orders-628494701: "name": "Flaviu",
202+
make-drinks-t49mm-create-orders-628494701: "flavor": "Brown Sugar Milk Tea",
203+
make-drinks-t49mm-create-orders-628494701: "ice_level": 1.0,
204+
make-drinks-t49mm-create-orders-628494701: "sugar_level": 1.0
205+
make-drinks-t49mm-create-orders-628494701: },
206+
make-drinks-t49mm-create-orders-628494701: {
207+
make-drinks-t49mm-create-orders-628494701: "name": "Elliot",
208+
make-drinks-t49mm-create-orders-628494701: "flavor": "Green Tea",
209+
make-drinks-t49mm-create-orders-628494701: "ice_level": 0,
210+
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.5
211+
make-drinks-t49mm-create-orders-628494701: },
212+
make-drinks-t49mm-create-orders-628494701: {
213+
make-drinks-t49mm-create-orders-628494701: "name": "Sambhav",
214+
make-drinks-t49mm-create-orders-628494701: "flavor": "Green Tea",
215+
make-drinks-t49mm-create-orders-628494701: "ice_level": 0,
216+
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.25
217+
make-drinks-t49mm-create-orders-628494701: },
218+
make-drinks-t49mm-create-orders-628494701: {
219+
make-drinks-t49mm-create-orders-628494701: "name": "Flaviu",
220+
make-drinks-t49mm-create-orders-628494701: "flavor": "Taro Milk Tea",
221+
make-drinks-t49mm-create-orders-628494701: "ice_level": 0.5,
222+
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.5
223+
make-drinks-t49mm-create-orders-628494701: }
224+
make-drinks-t49mm-create-orders-628494701: ]
225+
make-drinks-t49mm-make-bubble-tea-3020754075: Making Flaviu's Brown Sugar Milk Tea bubble tea with 100% ice and 100% sugar.
226+
make-drinks-t49mm-make-bubble-tea-2627605331: Making Elliot's Green Tea bubble tea with 0% ice and 50% sugar.
227+
make-drinks-t49mm-make-bubble-tea-3584623812: Making Sambhav's Green Tea bubble tea with 0% ice and 25% sugar.
228+
make-drinks-t49mm-make-bubble-tea-1040507004: Making Flaviu's Taro Milk Tea bubble tea with 50% ice and 50% sugar.
229+
```
230+
</details>
231+
232+
## Aggregating Fan Out Results (Fan In)
233+
234+
Okay, we've made all these drinks, now we need to serve them up together!
235+
236+
For this, we can again use the `result` output parameter, but as we will use it on the `make_bubble_tea` step, it
237+
expects JSON objects to aggregate them together.
238+
239+
Let's edit our `make_bubble_tea` function to dump a JSON object:
240+
241+
```py
242+
@script()
243+
def make_bubble_tea(
244+
name: str,
245+
flavor: str,
246+
ice_level: float,
247+
sugar_level: float,
248+
):
249+
import json
250+
251+
print(json.dumps({"name": name, "status": "Completed"}))
252+
```
253+
254+
And now let's write a function to call out "Serving N orders" and the names attached to the orders:
255+
256+
```py
257+
@script()
258+
def serve_orders(orders: List[Dict[str, str]]):
259+
names = list(set([order["name"] for order in orders]))
260+
print(f"Serving {len(orders)} orders for {', '.join(names[:-1])} and {names[-1]}!")
261+
```
262+
263+
In our Workflow, we can now link these scripts together with each Step's `result`:
264+
265+
```py
266+
with Workflow(
267+
generate_name="make-drinks-",
268+
entrypoint="steps",
269+
) as w:
270+
with Steps(name="steps"):
271+
orders = create_orders()
272+
teas = make_bubble_tea(with_param=orders.result)
273+
serve_orders(arguments={"orders": teas.result})
274+
```
275+
276+
277+
<details> <summary>The logs will look something like this (click to expand).</summary>
278+
279+
```console
280+
make-drinks-xk4hm-create-orders-2830038274: [
281+
make-drinks-xk4hm-create-orders-2830038274: {
282+
make-drinks-xk4hm-create-orders-2830038274: "name": "Elliot",
283+
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
284+
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
285+
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.5
286+
make-drinks-xk4hm-create-orders-2830038274: },
287+
make-drinks-xk4hm-create-orders-2830038274: {
288+
make-drinks-xk4hm-create-orders-2830038274: "name": "Elliot",
289+
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Taro Milk Tea",
290+
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.5,
291+
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 1.0
292+
make-drinks-xk4hm-create-orders-2830038274: },
293+
make-drinks-xk4hm-create-orders-2830038274: {
294+
make-drinks-xk4hm-create-orders-2830038274: "name": "Sambhav",
295+
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
296+
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
297+
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 1.0
298+
make-drinks-xk4hm-create-orders-2830038274: },
299+
make-drinks-xk4hm-create-orders-2830038274: {
300+
make-drinks-xk4hm-create-orders-2830038274: "name": "Sambhav",
301+
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Taro Milk Tea",
302+
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.5,
303+
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.75
304+
make-drinks-xk4hm-create-orders-2830038274: },
305+
make-drinks-xk4hm-create-orders-2830038274: {
306+
make-drinks-xk4hm-create-orders-2830038274: "name": "Flaviu",
307+
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
308+
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
309+
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.75
310+
make-drinks-xk4hm-create-orders-2830038274: }
311+
make-drinks-xk4hm-create-orders-2830038274: ]
312+
make-drinks-xk4hm-make-bubble-tea-2143417526: {"name": "Elliot", "status": "Completed"}
313+
make-drinks-xk4hm-make-bubble-tea-2058639815: {"name": "Elliot", "status": "Completed"}
314+
make-drinks-xk4hm-make-bubble-tea-316598325: {"name": "Sambhav", "status": "Completed"}
315+
make-drinks-xk4hm-make-bubble-tea-4190830807: {"name": "Sambhav", "status": "Completed"}
316+
make-drinks-xk4hm-make-bubble-tea-3301714217: {"name": "Flaviu", "status": "Completed"}
317+
make-drinks-xk4hm-serve-orders-974975305: Serving 5 orders for Sambhav, Elliot and Flaviu!
318+
```
319+
</details>

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ nav:
1515
- getting-started/walk-through/steps.md
1616
- getting-started/walk-through/dag.md
1717
- getting-started/walk-through/artifacts.md
18+
- getting-started/walk-through/loops.md
1819
- Hera expr transpiler: expr.md
1920
- Examples:
2021
- Workflows:

0 commit comments

Comments
 (0)