Skip to content

Commit bc916d3

Browse files
authored
Wrap up getting started docs (#664)
**Pull Request Checklist** - [x] Fixes #627 (i.e. this is the last PR) - [x] ~Tests added~ Docs only - [x] Documentation/examples added - [x] [Good commit messages](https://cbea.ms/git-commit/) and/or PR title --------- Signed-off-by: Elliot Gunton <[email protected]>
1 parent 637c9e9 commit bc916d3

File tree

10 files changed

+393
-14
lines changed

10 files changed

+393
-14
lines changed

docs/expr.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ There maybe times that you want to get a struct field whose name conflicts with
6262

6363
Map elements can be accessed used the `[]` syntax.
6464

65-
* `g.var['element']` transpiles to `var.['element']`
65+
* `g.var['element']` transpiles to `var['element']`
6666

6767
### Reference
6868

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Advanced Hera Features
2+
3+
This section is used to publicize Hera's features beyond the essentials covered in the walk through. Note that these
4+
features do not exist in Argo as they are specific to the `hera` module.
5+
6+
## Pre-Build Hooks
7+
8+
Hera offers a pre-build hook feature through `hera.shared.register_pre_build_hook` with huge flexibility to do pre-build
9+
processing on any type of `template` or `Workflow`. For example, it can be used to conditionally set the `image` of a
10+
`Script`, or set which cluster to submit a `Workflow` to.
11+
12+
To use this feature, you can write a function that takes an object of type `template` or `Workflow`, does some
13+
processing on the object, then returns it.
14+
15+
For a simple example, we'll write a function that adds an annotation with key "hera", value "This workflow was submitted
16+
through Hera!"
17+
18+
```py
19+
from hera.shared import register_pre_build_hook
20+
from hera.workflows import Workflow
21+
22+
@register_pre_build_hook
23+
def set_workflow_default_labels(workflow: Workflow) -> Workflow:
24+
if workflow.annotations is None:
25+
workflow.annotations = {}
26+
27+
workflow.annotations["hera-annotation"] = "This workflow was submitted through Hera!"
28+
return workflow
29+
30+
```
31+
32+
Now, any time `build` is called on the Workflow (e.g. to submit it or dump it to yaml), it will add in the annotation!
33+
34+
## Experimental Features
35+
36+
From time to time, Hera will release a new feature under the "experimental feature" flag while we develop the feature
37+
and ensure stability. Currently this is used for the `RunnerScriptConstructor` seen in the
38+
[runner script example](../../examples/workflows/callable_script.md).
39+
40+
To enable experimental features you must set the feature by name to `True` in the `global_config.experimental_features`
41+
dictionary before using the feature:
42+
43+
```py
44+
global_config.experimental_features["script_runner"] = True
45+
```
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Advanced Template Features
2+
3+
This section exemplifies `template` features found in Argo, but are beyond the scope of the Walk Through.
4+
5+
## Secrets
6+
7+
To access secrets stored in your Kubernetes/Argo cluster, you should use the `env` or `env_from` members of
8+
`TemplateMixin` (i.e. any standard template).
9+
10+
```py
11+
from hera.workflows import (
12+
ConfigMapEnv,
13+
ConfigMapEnvFrom,
14+
Container,
15+
Env,
16+
ResourceEnv,
17+
SecretEnv,
18+
SecretEnvFrom,
19+
Workflow,
20+
)
21+
22+
with Workflow(generate_name="secret-env-from-", entrypoint="whalesay") as w:
23+
whalesay = Container(
24+
image="docker/whalesay:latest",
25+
command=["cowsay"],
26+
env_from=[
27+
SecretEnvFrom(prefix="abc", name="secret", optional=False),
28+
ConfigMapEnvFrom(prefix="abc", name="configmap", optional=False),
29+
],
30+
env=[
31+
Env(name="test", value="1"),
32+
SecretEnv(name="s1", secret_key="s1", secret_name="abc"),
33+
ResourceEnv(name="r1", resource="abc"),
34+
ConfigMapEnv(name="c1", config_map_key="c1", config_map_name="abc"),
35+
],
36+
)
37+
```
38+
39+
## Retrying and Timeouts
40+
41+
You can easily set retries and timeouts on templates through the `retry_strategy` and `timeout` members of `TemplateMixin`.
42+
43+
See the example below for a combination of these features, which retries with a backoff up to 10 times or until the
44+
timeout, set at 5 minutes.
45+
46+
```py
47+
from hera.workflows import (
48+
Container,
49+
RetryStrategy,
50+
Workflow,
51+
models as m,
52+
)
53+
54+
with Workflow(
55+
generate_name="retry-backoff-til-timeout-",
56+
entrypoint="retry-backoff-til-timeout",
57+
) as w:
58+
retry_backoff_til_timeout = Container(
59+
name="retry-backoff-til-timeout",
60+
image="python:alpine3.6",
61+
command=["python", "-c"],
62+
args=["import random; import sys; exit_code = random.choice([0, 0, 0, 1]); sys.exit(exit_code)"],
63+
retry_strategy=RetryStrategy(
64+
limit=10,
65+
backoff=m.Backoff(
66+
duration="1",
67+
factor="2",
68+
max_duration="1m",
69+
),
70+
),
71+
timeout="5m",
72+
)
73+
```
74+
75+
## Recursion
76+
77+
Individual Steps and Tasks can specify the `Steps` or `DAG` template that it is a part of, allowing recursive Template
78+
Invocators. You must ensure there is a break condition, and you should *NOT* make the first `Step` or `Task` the parent
79+
`Steps` or `DAG` without a condition, otherwise the Argo controller will expand the definition indefinitely until it
80+
crashes.
81+
82+
See the [recursive coinflip](../../examples/workflows/upstream/coinflip-recursive.md) example for a recursive `Steps`
83+
template.
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Advanced Workflow Features
2+
3+
This section exemplifies Workflow features found in Argo, but are beyond the scope of the Walk Through.
4+
5+
## Exit Handlers
6+
7+
Exit handlers are templates that always execute (regardless of success or failure) at the end of a workflow.
8+
9+
Some use cases listed from [Argo Workflows](https://argoproj.github.io/argo-workflows/walk-through/exit-handlers/)
10+
include:
11+
12+
* cleaning up after a workflow runs
13+
* sending notifications of workflow status (e.g. e-mail/Slack)
14+
* posting the pass/fail status to a web-hook result (e.g. GitHub build result)
15+
* resubmitting or submitting another workflow
16+
17+
To use an exit handler on a Workflow in Hera, you can either define the template to use within the workflow, then set
18+
the `on_exit` member which will take the name from the template itself, or specify the template to use as a string when
19+
initializing the Workflow object. See both methods below.
20+
21+
```py
22+
with Workflow(
23+
generate_name="exit-handler-workflow-",
24+
) as w1:
25+
cleanup_exit_handler = Container(
26+
name="cleanup",
27+
image="docker/whalesay",
28+
command=["cowsay"],
29+
args=["cleanup!"],
30+
)
31+
w1.on_exit = cleanup_exit_handler
32+
...
33+
34+
with Workflow(
35+
generate_name="exit-handler-workflow-",
36+
on_exit="cleanup",
37+
) as w2:
38+
cleanup_exit_handler = Container(
39+
name="cleanup",
40+
image="docker/whalesay",
41+
command=["cowsay"],
42+
args=["cleanup!"],
43+
)
44+
...
45+
```
46+
47+
## Volume Claim Templates
48+
49+
Volume Claim Templates can be used to copy a large amount data from one step to another in a Workflow.
50+
51+
See the [Empty Volume](../../examples/workflows/upstream/volumes_emptydir.md) example for creating a volume dynamically for the Workflow, or the [Existing Volumes](../../examples/workflows/upstream/volumes_existing.md) example for using an existing volume.
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Conditionals
2+
3+
Conditional execution of steps and tasks is available through the [basic `when` clause](#basic-when-clauses), along with
4+
more complex [expressions](#complex-when-clauses) and [`Task` functions](#improved-task-conditionals).
5+
6+
## Basic `when` Clauses
7+
8+
Argo uses [`govaluate`](https://github.com/Knetic/govaluate) in its `when` expressions - Hera is compatible with these
9+
`when` expression if you write them in literal strings, however, without prior Argo knowledge, you may not know the
10+
chain of keys to access steps and their parameters.
11+
12+
You can instead use `Parameters` and the special `result` parameter in an f-string to make your code more readable:
13+
14+
```py
15+
run_script(
16+
...,
17+
when=f'{previous_step.get_parameter("some-parameter")} == "some-value"'
18+
)
19+
```
20+
21+
is equivalent to
22+
23+
```py
24+
run_script(
25+
...,
26+
when='{{steps.previous-step.outputs.parameter.some-parameter}} == "some-value"'
27+
)
28+
```
29+
30+
And
31+
32+
```py
33+
run_script(
34+
...,
35+
when=f'{previous_step.result} == "some-value"'
36+
)
37+
```
38+
39+
is equivalent to
40+
41+
```py
42+
run_script(
43+
...,
44+
when='{{steps.previous-step.outputs.result}} == "some-value"'
45+
)
46+
```
47+
48+
## Complex `when` Clauses
49+
50+
If you want to use more complex `when` clauses than simple comparisons, involving Argo expression functions such as
51+
`filter` or `map` or even [sprig functions](http://masterminds.github.io/sprig/), you should check out Hera's
52+
[`expr` module documentation](../../expr.md).
53+
54+
## Improved Task Conditionals
55+
56+
When using DAGs, there are many convenience functions we can use:
57+
58+
* `on_success`
59+
* `on_failure`
60+
* `on_error`
61+
* `on_other_result`
62+
* `when_any_succeeded`
63+
* `when_all_failed`
64+
* `on_workflow_status`
65+
66+
These functions (except `on_workflow_status`) are essentially aliases to `Task`'s `next` function, with the `on`
67+
parameter populated with the corrected `TaskResult`, so they can be used to set stricter dependencies than with the
68+
rshift (`>>`) operator.

docs/getting-started/walk-through/loops.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ call. First, specify the list of items you want to echo in the `with_items` kwar
4040
)
4141
```
4242

43-
Now, we need to replace the value of the `message` argument. In Argo, you would use the `"{{item}}"` expression syntax,
44-
which is also what we use in Hera:
43+
Now, we need to replace the value of the `message` argument. In Argo, you would use the `{{item}}` expression syntax,
44+
which is also what we use in Hera (within a string):
4545

4646
```py
4747
echo(
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Integrated Pydantic Support
2+
3+
<!-- TODO -->

docs/getting-started/walk-through/steps.md

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -60,16 +60,14 @@ with Workflow(
6060
w.create()
6161
```
6262

63-
Remember any parallel steps will run indeterminately within the context, so `parallel-1`, `parallel-2` and `parallel-3` could
64-
run in any order, but `pre-parallel` will always run before the parallel steps and `post-parallel` will run after *all* the
65-
parallel steps have completed.
66-
63+
Remember any parallel steps will run indeterminately within the context, so `parallel-1`, `parallel-2` and `parallel-3`
64+
could run in any order, but `pre-parallel` will always run before the parallel steps and `post-parallel` will run after
65+
*all* the parallel steps have completed.
6766

6867
## `when` Clauses
6968

70-
Examples of `when` clauses can be found throughout the examples, such as
71-
[the Argo coinflip example](../../examples/workflows/upstream/coinflip.md). They specify conditions under which the step
72-
will run.
69+
A `when` clause specifies the conditions under which the step or task will run. Examples of `when` clauses can be found
70+
throughout the examples, such as [the Argo coinflip example](../../examples/workflows/upstream/coinflip.md).
7371

7472
If we consider features offered by Hera along with what we've learned about parameters and parallel steps, we
7573
can form a Workflow with identical behaviour to the upstream coinflip, but using only Python scripts and syntactic sugar
@@ -89,14 +87,25 @@ def flip():
8987

9088

9189
@script()
92-
def it_was(result):
93-
print(f"it was {result}")
90+
def it_was(coin_result):
91+
print(f"it was {coin_result}")
9492

9593

9694
with Workflow(generate_name="coinflip-", entrypoint="steps") as w:
9795
with Steps(name="steps") as s:
9896
f = flip()
9997
with s.parallel():
100-
it_was(name="heads", arguments={"result": "heads"}).on_other_result(f, "heads")
101-
it_was(name="tails", arguments={"result": "tails"}).on_other_result(f, "tails")
98+
it_was(name="heads", arguments={"coin_result": "heads"}, when=f'{f.result} == "heads"')
99+
it_was(name="tails", arguments={"coin_result": "tails"}, when=f'{f.result} == "tails"')
100+
```
101+
102+
<details><summary>Click to see an example Workflow log</summary>
103+
104+
```console
105+
coinflip-gfrws-flip-1899249874: heads
106+
coinflip-gfrws-it-was-2809981541: it was heads
102107
```
108+
109+
</details>
110+
111+
For more about `when` clauses, see the [Conditionals](conditionals.md) page!

0 commit comments

Comments
 (0)