workflow_manager

Workflow manager

Concept

The main aim here is a separation of concerns between managing dependencies and running model processes.

A scheduler can run sets of jobs - either with or without knowledge of dependencies between jobs. A simple scheduler might have no knowledge of dependencies and would run all jobs requested. A more sophisticated scheduler might consume directed acyclic graphs, which can be iteratively produced by a smif model run as it steps through decision iterations.

We might refactor the smif ModelRunner/SosModel classes so that:

• a workflow scheduler accepts directed acyclic graphs (i.e. we need to implement a local version of the DAFNI workflow consuming job scheduler)
• ModelRunner works out how to divide up the decision and model iterations into DAGs
• ModelRunner then passes DAG fragments to the job scheduler

           +--------------+
           |  smif-app    |
           +-------|------+
                   | http-api
                   |
                   |
           +-------|------+
  (cli)----| smif-manager |-----------------+
           +-------|------+                 |
                   |                        |
                   |                        |
                   | http-api               | http-api
           +-------|------+         +-------|------+
   cli ----| smif-runner  |         |     dafni    |
           +--------------+         +--------------+

smif-app

• Configure a system-of-systems
• Start / stop Modelruns
• Visualise results

smif-manager

• Manage dependencies between models
• Manage decision iterations
• Coordinate jobs and expose workflows to workers-pool

smif-runner

• Runs jobs (individual timestep/iteration/model units) and sends results back to smif-manager

 Worker-pool
+-----------------------------------------------------------------------+
|                                                                       |
|   local-machine     ouce-cluster      dafni                           |
|  +-------------+   +-------------+   +-------------+ +-------------+  |
|  | smif-runner |   | smif-runner |   | smif-runner | | smif-runner |  |
|  +-------------+   +-------------+   +-------------+ +-------------+  |
|                    +-------------+   +-------------+ +-------------+  |
|                    | smif-runner |   | smif-runner | | smif-runner |  |
|                    +-------------+   +-------------+ +-------------+  |
|                                                                       |
+-----------------------------------------------------------------------+

Detailed Design

smif-manager

smif-runner register <ip> <number_of_jobs> register a runner to the smif-manager

smif-runner

A local service that duplicates the Dafni scheduler API

smif-runner run run the runner service

smif-runner status get status of runner