Add a script constructor that allows for flexible script construction and add a robust runner mode (#543)

Signed-off-by: Sambhav Kothari <[email protected]>

This PR introduces a new mode for script called callable mode, which
runs the function using the hera runner.

hera runner is a robust version of the current script mode which can
accomodate large files, imports and calling functions with kwargs. It is
highly compatible/tied to pydantic and makes writing argo commands a
breeze using pydantic + hera.

It automatically serializes/deserializes inputs/outputs using pydantic
and also updates the script callable to only return a task when under an
active hera context.

This mode is only possible if the user source code is installed in the
output image along with hera.


---

After conversations with @flaviuvadan, updated the implementation to
allow for the following -

- Added a new ScriptConstructor class that allows users to define their
own ScriptConstructor that can inspect the callable source and generate
the output source. Provided two default implementations - inline and
runner. The default value is `inline` right now, but there are
placeholders in the code to allow for the default value to be detected
automatically. Additionally, users can provide their own
ScriptConstructor if they wish to and set it as a default.
- Added the ability to set class defaults for any hera class using
global_config. This allows users to introduce default values of hera
classes without needing us to implement it for every single use case.
This also removes any circular dependencies between hera global config
and any hera specific classes.

---------

Signed-off-by: Sambhav Kothari <[email protected]>

Author: sambhav

docs/examples/workflows/callable_coinflip.md

@@ -0,0 +1,107 @@
+# Callable Coinflip
+
+
+
+
+
+
+=== "Hera"
+
+    ```python linenums="1"
+    import random
+
+    from hera.shared import global_config
+    from hera.workflows import DAG, Script, Workflow, script
+
+    # Note, setting constructor to runner is only possible if the source code is available
+    # along with dependencies include hera in the image.
+    # Callable is a robust mode that allows you to run any python function
+    # and is compatible with pydantic. It automatically parses the input
+    # and serializes the output.
+    global_config.image = "my-image-with-python-source-code-and-dependencies"
+    global_config.set_class_defaults(Script, constructor="runner")
+    # Runner script constructor is still and experimental feature and we need to explicitly opt in to it
+    # Note that experimental features are subject to breaking changes in future releases of the same major version
+    global_config.experimental_features["script_runner"] = True
+
+
+    @script()
+    def flip():
+        return "heads" if random.randint(0, 1) == 0 else "tails"
+
+
+    @script()
+    def heads():
+        return "it was heads"
+
+
+    @script()
+    def tails():
+        return "it was tails"
+
+
+    with Workflow(generate_name="coinflip-", entrypoint="d") as w:
+        with DAG(name="d") as s:
+            f = flip()
+            heads().on_other_result(f, "heads")
+            tails().on_other_result(f, "tails")
+    ```
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    apiVersion: argoproj.io/v1alpha1
+    kind: Workflow
+    metadata:
+      generateName: coinflip-
+    spec:
+      entrypoint: d
+      templates:
+      - dag:
+          tasks:
+          - name: flip
+            template: flip
+          - depends: flip
+            name: heads
+            template: heads
+            when: '{{tasks.flip.outputs.result}} == heads'
+          - depends: flip
+            name: tails
+            template: tails
+            when: '{{tasks.flip.outputs.result}} == tails'
+        name: d
+      - name: flip
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_coinflip:flip
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+      - name: heads
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_coinflip:heads
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+      - name: tails
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_coinflip:tails
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+    ```
+

docs/examples/workflows/callable_script.md

@@ -0,0 +1,153 @@
+# Callable Script
+
+
+
+
+
+
+=== "Hera"
+
+    ```python linenums="1"
+    from typing import List
+
+    from pydantic import BaseModel
+
+    from hera.shared import global_config
+    from hera.workflows import Script, Steps, Workflow, script
+
+    # Note, setting constructor to runner is only possible if the source code is available
+    # along with dependencies include hera in the image.
+    # Callable is a robust mode that allows you to run any python function
+    # and is compatible with pydantic. It automatically parses the input
+    # and serializes the output.
+    global_config.image = "my-image-with-python-source-code-and-dependencies"
+    global_config.set_class_defaults(Script, constructor="runner")
+    # Runner script constructor is still and experimental feature and we need to explicitly opt in to it
+    # Note that experimental features are subject to breaking changes in future releases of the same major version
+    global_config.experimental_features["script_runner"] = True
+
+
+    # An optional pydantic input type
+    # hera can automatically de-serialize argo
+    # arguments into types denoted by your function's signature
+    # as long as they are de-serializable by pydantic
+    # This provides auto-magic input parsing with validation
+    # provided by pydantic.
+    class Input(BaseModel):
+        a: int
+        b: str = "foo"
+
+
+    # An optional pydantic output type
+    # hera can automatically serialize the output
+    # of your function into a json string
+    # as long as they are serializable by pydantic or json serializable
+    # This provides auto-magic output serialization with validation
+    # provided by pydantic.
+    class Output(BaseModel):
+        output: List[Input]
+
+
+    @script()
+    def my_function(input: Input) -> Output:
+        return Output(output=[input])
+
+
+    # Note that the input type is a list of Input
+    # hera can also automatically de-serialize
+    # composite types like lists and dicts
+    @script()
+    def another_function(inputs: List[Input]) -> Output:
+        return Output(output=inputs)
+
+
+    # it also works with raw json strings
+    # but those must be explicitly marked as
+    # a string type
+    @script()
+    def str_function(input: str) -> Output:
+        # Example function to ensure we are not json parsing
+        # string types before passing it to the function
+        return Output(output=[Input.parse_raw(input)])
+
+
+    with Workflow(name="my-workflow") as w:
+        with Steps(name="my-steps") as s:
+            my_function(arguments={"input": Input(a=2, b="bar")})
+            str_function(arguments={"input": Input(a=2, b="bar").json()})
+            another_function(arguments={"inputs": [Input(a=2, b="bar"), Input(a=2, b="bar")]})
+    ```
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    apiVersion: argoproj.io/v1alpha1
+    kind: Workflow
+    metadata:
+      name: my-workflow
+    spec:
+      templates:
+      - name: my-steps
+        steps:
+        - - arguments:
+              parameters:
+              - name: input
+                value: '{"a": 2, "b": "bar"}'
+            name: my-function
+            template: my-function
+        - - arguments:
+              parameters:
+              - name: input
+                value: '{"a": 2, "b": "bar"}'
+            name: str-function
+            template: str-function
+        - - arguments:
+              parameters:
+              - name: inputs
+                value: '[{"a": 2, "b": "bar"}, {"a": 2, "b": "bar"}]'
+            name: another-function
+            template: another-function
+      - inputs:
+          parameters:
+          - name: input
+        name: my-function
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_script:my_function
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+      - inputs:
+          parameters:
+          - name: input
+        name: str-function
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_script:str_function
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+      - inputs:
+          parameters:
+          - name: inputs
+        name: another-function
+        script:
+          args:
+          - -m
+          - hera.workflows.runner
+          - -e
+          - examples.workflows.callable_script:another_function
+          command:
+          - python
+          image: my-image-with-python-source-code-and-dependencies
+          source: '{{inputs.parameters}}'
+    ```
+

docs/examples/workflows/coinflip.md

@@ -63,7 +63,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys
@@ -82,7 +82,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys
@@ -96,7 +96,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys

docs/examples/workflows/complex_deps.md

@@ -75,7 +75,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: "import os\nimport sys\nsys.path.append(os.getcwd())\nimport json\n\
             try: p = json.loads(r'''{{inputs.parameters.p}}''')\nexcept: p = r'''{{inputs.parameters.p}}'''\n\
             \nif p < 0.5:\n    raise Exception(p)\nprint(42)\n"

docs/examples/workflows/dag_with_script_param_passing.md

@@ -54,7 +54,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys
@@ -71,7 +71,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys

docs/examples/workflows/dynamic_volumes.md

@@ -42,7 +42,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys

docs/examples/workflows/global_config.md

@@ -16,6 +16,7 @@
     global_config.service_account_name = "argo-account"
     global_config.image = "image-say"
     global_config.script_command = ["python3"]
+    global_config.set_class_defaults(Container, active_deadline_seconds=100, command=["cowsay"])
 
 
     @script()
@@ -24,7 +25,7 @@
 
 
     with Workflow(generate_name="global-config-", entrypoint="whalesay") as w:
-        whalesay = Container(image="docker/whalesay:latest", command=["cowsay"])
+        whalesay = Container(image="docker/whalesay:latest")
         say()
     ```
 
@@ -40,7 +41,8 @@
       entrypoint: whalesay
       serviceAccountName: argo-account
       templates:
-      - container:
+      - activeDeadlineSeconds: '100'
+        container:
           command:
           - cowsay
           image: docker/whalesay:latest

docs/examples/workflows/multi_env.md

@@ -52,7 +52,7 @@
             value: '2'
           - name: c
             value: '3'
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys

docs/examples/workflows/script_auto_infer.md

@@ -66,7 +66,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: "import os\nimport sys\nsys.path.append(os.getcwd())\nimport pickle\n\
             \nresult = \"foo testing\"\nwith open(\"/tmp/result\", \"wb\") as f:\n   \
             \ pickle.dump(result, f)\n"
@@ -78,7 +78,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: "import os\nimport sys\nsys.path.append(os.getcwd())\nimport json\n\n\
             import pickle\n\nwith open(\"/tmp/i\", \"rb\") as f:\n    i = pickle.load(f)\n\
             print(i)\n"

docs/examples/workflows/script_with_default_params.md

@@ -80,7 +80,7 @@
         script:
           command:
           - python
-          image: python:3.7
+          image: python:3.8
           source: 'import os
 
             import sys