Confusion around context handler behavior

[daturkel]

I'm a bit confused about the behavior of context managers in Hera and exactly how they behave under the hood.

If I have a python prelude like this:

from hera.workflows import Container, Steps, Workflow

def get_container():
    my_container = Container(
        image="my-image",
        command=["bash", "-c"],
        args=["echo hello"],
    )
    return my_container

There are a couple ways I might be inclined to use this container factory function.

I could declare the Container in global scope, which works:

container = get_container()
with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    with Steps(name="entrypoint") as my_steps:
        container(name="foo")

I could declare the Container in the Workflow scope, which also works, though I'm not sure what the advantage is:

with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    container = get_container()
    with Steps(name="entrypoint") as my_steps:
        container(name="foo")

Lastly, I could declare the container in the Steps scope, which does not work:

with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    with Steps(name="entrypoint") as my_steps:
        container = get_container()
        container(name="foo")

# hera.workflows.exceptions.InvalidType: <class 'hera.workflows.container.Container'>

I would love some information on the advantages of the Workflow and Steps context managers and when to use them, and why the third snippet here doesn't work. There doesn't seem to be a ton in the Hera docs on this topic (though maybe I've missed it) and I wouldn't be against contributing some documentation, but first I'll have to understand the behavior better!

There's some relevant discussion on documentation for idiomatic workflows in this issue: #440

[flaviuvadan]

Hey Dan! Apologies for the delayed response. This happens because of the order in which Hera registers templates and usages of those templates. I’ll walk through your examples 1 by 1

container = get_container()
with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    with Steps(name="entrypoint") as my_steps:
        container(name="foo")

works because Hera adds the template associated with your container call to your workflow when you invoke the callable container (source)

with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    container = get_container()
    with Steps(name="entrypoint") as my_steps:
        container(name="foo")

works because Hera intercepts the creation of the container inside the context and adds the template to the workflow (source, ends up in container via inheritance, template added here)

with Workflow(name="my-workflow", entrypoint="entrypoint") as my_workflow:
    with Steps(name="entrypoint") as my_steps:
        container = get_container()
        container(name="foo")

doesn’t work because instead of adding the template associated with your container first to the workflow context Hera checks that you have a callable invocation (what you get when you execute container(…)). I think this is something we can improve on Hera’s side.

CC: @elliotgunton @alicederyn and @jeongukjae for thoughts!
OP: https://cloud-native.slack.com/archives/C03NRMD9KPY/p1729875572482339

[daturkel]

Thank you for the thorough response, Flaviu! This is a really helpful breakdown, and I appreciate the links to the relevant bits of code.

I don't know that this behavior necessarily needs to change, but I do think there's an opportunity for improvement to the documentation on the correct or idiomatic way to use the context managers, or even the advantages of using the context handlers versus the .add methods.

[flaviuvadan]

I agree! I'll make an issue from this