fix: docs and removing pytorch

* fix: mock executor fix

* fix: remove pytorch, we will revisit later

* docs: improved docs and examples

* test: writting more unit tests

Author: vijayvammi

docs/concepts/job_intro.md

@@ -0,0 +1,87 @@
+Jobs are isolated unit of work which can be python functions, jupyter notebooks or shell scripts.
+
+
+
+Considering a simple function:
+
+```python
+def add_numbers(x: int, y: int):
+    # save some data in data.csv
+    return x + y
+```
+
+The runnable representation of it is:
+
+```python
+from functions import add_numbers
+from runnable import PythonJob, Catalog
+
+write_catalog = Catalog(put=["data.csv"])
+job = PythonJob(function=add_numbers,
+                returns["sum_of_numbers"],
+                catalog=write_catalog,
+            )
+
+```
+
+```PythonJob``` requires a function to call. The input parameters are passed in
+ from the parameters provided at the time of execution.
+
+The return parameters are stored for future reference. Any data object generated in the
+process can be saved to the catalog.
+
+<hr style="border:2px dotted orange">
+
+
+## Python functions
+
+You can use Python functions as jobs in a pipeline, enabling flexible encapsulation of logic, parameter passing, result capturing, and cataloging of outputs.
+
+=== "Basic Python Function as a Job"
+    ```python
+    --8<-- "examples/11-jobs/python_tasks.py"
+    ```
+
+    The stdout (e.g., "Hello World!") and logs are captured and stored in the catalog for traceability.
+
+=== "Writing Data to the Catalog"
+    ```python
+    --8<-- "examples/11-jobs/catalog.py"
+    ```
+
+    The `Catalog` object specifies which files or data should be saved after job execution.
+
+=== "Passing and Returning Parameters"
+
+    ```python
+    --8<-- "examples/11-jobs/passing_parameters_python.py"
+    ```
+
+    Parameters can be passed at execution time, and returned values can be automatically handled, serialized, and tracked as metrics.
+
+---
+
+## Notebooks
+
+You can also use Jupyter notebooks as jobs in your pipeline. This allows you to encapsulate notebook logic, capture outputs, and integrate notebooks seamlessly into your workflow.
+
+=== "Notebook as a Job"
+    ```python
+    --8<-- "examples/11-jobs/notebooks.py"
+    ```
+    The output of the notebook will be captured as execution log
+    along with the actual notebook and stored in the catalog for traceability.
+
+---
+
+## Shell script
+
+You can also use shell scripts or commands as jobs in your pipeline. This allows you to execute any shell command, capture its output, and integrate it into your workflow.
+
+=== "Shell Script"
+    ```python
+    --8<-- "examples/11-jobs/scripts.py"
+    ```
+    The stdout and stderr of the shell command are captured as execution log and stored in the catalog for traceability.
+
+For more advanced examples, see the files in `examples/11-jobs/`.

docs/concepts/nesting.md

@@ -64,7 +64,10 @@ The step ```Train Models``` is a parallel step that has the ```branches``` as th
 
     def main():
         train_models = Parallel(name="train models",
-                        branches={'baseline': get_baseline_pipeline, 'cnn': get_cnn_pipeline()},
+                        branches={
+                            'baseline': get_baseline_pipeline,
+                            'cnn': get_cnn_pipeline()
+                        },
                         terminate_with_success=True)
         pipeline = Pipeline(steps=[train_models])
 
@@ -123,12 +126,12 @@ The parallel step is considered successful only if all the branches of the step
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="53-57"
+    ```python linenums="1""
     --8<-- "examples/06-parallel/parallel.py"
     ```
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="40-45"
+    ```yaml linenums="1""
     --8<-- "examples/06-parallel/parallel.yaml"
     ```

docs/concepts/parallel.md

@@ -108,9 +108,10 @@ This behavior can be over-ridden to follow a different path based on expected fa
 
 
 ```step 1``` fails as the function raises an exception.
-```step 4``` is an alternate node to a successful execution.
 
-```step 4``` is the step to execution in case of the failure.
+```step 4``` is a single node pipeline to execute if ```step1``` fails. The failure
+pipeline can have as many steps as needed.
+
 
 === "pseudo code"
 
@@ -126,7 +127,7 @@ This behavior can be over-ridden to follow a different path based on expected fa
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="24 29 34 31"
+    ```python linenums="1""
     --8<-- "examples/02-sequential/on_failure_succeed.py"
     ```
 
@@ -135,17 +136,18 @@ This behavior can be over-ridden to follow a different path based on expected fa
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="23 25 32-34"
+    ```yaml linenums="1""
     --8<-- "examples/02-sequential/on_failure_succeed.yaml"
     ```
 
 
 ### On failure fail
 
 ```step 1``` fails as the function raises an exception.
-```step 4``` is an alternate node to a successful execution.
 
-```step 4``` is the step to execution in case of the failure.
+```step 4``` is a single node pipeline to execute if ```step1``` fails. The failure
+pipeline can have as many steps as needed.
+
 
 === "pseudo code"
 
@@ -162,15 +164,13 @@ This behavior can be over-ridden to follow a different path based on expected fa
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="24 29 34 31"
+    ```python linenums="1""
     --8<-- "examples/02-sequential/on_failure_fail.py"
     ```
 
-    1. ```terminate_with_failure``` is ```true``` traverses to fail node.
-
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="23 25 32-34"
+    ```yaml linenums="1""
     --8<-- "examples/02-sequential/on_failure_fail.yaml"
     ```

docs/concepts/pipeline.md

@@ -1,3 +1,5 @@
+### Pipeline
+
 Without any orchestrator, the simplest pipeline could be the below functions:
 
 
@@ -47,8 +49,8 @@ pipeline.execute()
 - Tasks can [access and return](parameters.md/#access_returns) parameters.
 - Tasks can also share files between them using [catalog](catalog.md).
 - Tasks are stitched together as [pipeline](pipeline.md)
-- The execution environment is configured via # todo
-
+- The execution environment is configured via
+# TODO: figure this link
 
 ## Examples
 

docs/concepts/index.md renamed to docs/concepts/pipeline_intro.md

@@ -26,7 +26,7 @@ Uses python functions as tasks.
         than the pipeline definition, if you are using Python SDK.
 
 
-    ```python linenums="1" hl_lines="29-33"
+    ```python linenums="1""
     --8<-- "examples/01-tasks/python_tasks.py"
     ```
 
@@ -54,7 +54,7 @@ Uses python functions as tasks.
 
             ```
 
-    ```yaml linenums="1" hl_lines="20-23"
+    ```yaml linenums="1""
     --8<-- "examples/01-tasks/python_tasks.yaml"
     ```
 
@@ -77,13 +77,13 @@ the name of the notebook and is also saved in the ```catalog``` for logging and
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="29-33"
+    ```python linenums="1""
     --8<-- "examples/01-tasks/notebook.py"
     ```
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="27-31"
+    ```yaml linenums="1""
     --8<-- "examples/01-tasks/notebook.yaml"
     ```
 
@@ -102,13 +102,13 @@ ecosystem while shell provides a interface to non-python executables.
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="22-26"
+    ```python linenums="1""
     --8<-- "examples/01-tasks/scripts.py"
     ```
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="16-23"
+    ```yaml linenums="1""
     --8<-- "examples/01-tasks/scripts.yaml"
     ```
 
@@ -135,12 +135,12 @@ Stub nodes can take arbitrary number of parameters and is always a success.
 
 === "sdk"
 
-    ```python linenums="1" hl_lines="23 28 30"
+    ```python linenums="1""
     --8<-- "examples/01-tasks/stub.py"
     ```
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="19-29"
+    ```yaml linenums="1""
     --8<-- "examples/01-tasks/stub.yaml"
     ```

docs/concepts/task.md

@@ -20,7 +20,6 @@ They can be installed by ```"pip install runnable[<extra>]"```
 - ```notebook``` : enables notebooks as tasks/jobs
 - ```k8s``` : enables running jobs in kubernetes or minikube clusters
 - ```s3``` : enables using ```s3``` buckets for ```run log store``` and ```catalog```
-- ```torch``` : enables to run pytorch jobs or as tasks in pipeline
 
 
 ## Usage
@@ -53,26 +52,7 @@ runnable execute
 
 ### Execute a job
 
-Jobs defined in **runnable** can be either via [python sdk](reference.md) or ```yaml``` based definitions.
-
-The options are detailed below:
-
-```shell
-Usage: runnable submit-job [OPTIONS] JOB_DEFINITION_FILE
-
-╭─ Arguments ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *    job_definition_file      TEXT  The yaml file containing the job definition [default: None] [required]                                         │
-╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --config      -c      TEXT                              The configuration file specifying the services                                             │
-│ --parameters  -p      TEXT                              Parameters, in yaml,  accessible by the application                                        │
-│ --log-level           [INFO|DEBUG|WARNING|ERROR|FATAL]  The log level [default: WARNING]                                                           │
-│ --tag                 TEXT                              A tag attached to the run                                                                  │
-│ --run-id              TEXT                              An optional run_id, one would be generated if its not provided                             │
-│ --help                                                  Show this message and exit.                                                                │
-╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-```
-
+Jobs defined in **runnable** can be via [python sdk](reference.md)
 
 <hr style="border:2px dotted orange">
 

docs/usage.md

@@ -13,6 +13,10 @@
 
     You can run this pipeline by:
        python examples/01-tasks/stub.py
+
+You can execute this pipeline by:
+
+    python examples/01-tasks/stub.py
 """
 
 from runnable import Pipeline, Stub

examples/01-tasks/stub.py

@@ -10,8 +10,15 @@ def when_tails_function():
 
 
 def toss_function():
+    import os
     import random
 
+    if "FIX_RANDOM_TOSS" in os.environ:
+        # Use the fixed value for testing
+        toss = os.environ["FIX_RANDOM_TOSS"]
+        print(f"Using fixed toss result: {toss}")
+        return toss
+
     # Simulate a coin toss
     toss = random.choice(["heads", "tails"])
     print(f"Toss result: {toss}")
@@ -52,3 +59,4 @@ def main():
 
 if __name__ == "__main__":
     main()
+    main()

examples/02-sequential/conditional.py

@@ -3,12 +3,11 @@
 
     python examples/02-sequential/traversal.py
 
-    A pipeline can have any "tasks" as part of it. In the
-    below example, we have a mix of stub, python, shell and notebook tasks.
-
-    As with simpler tasks, the stdout and stderr of each task are captured
-    and stored in the catalog.
+A pipeline can have any "tasks" as part of it. In the
+below example, we have a mix of stub, python, shell and notebook tasks.
 
+As with simpler tasks, the stdout and stderr of each task are captured
+and stored in the catalog.
 """
 
 from examples.common.functions import hello