Django Worker Pool Example

Quick Start

Create a Python virtual environment at the root of the project:

python3.10 -m venv --prompt venv .venv

To enable the virtual environment:

source .venv/bin/activate

Upgrade pip and install dependencies:

pip install -U pip
pip install -r requirements.txt

Navigate to the Django project and ensure the database is initialized:

cd server
python manage.py migrate
python manage.py loaddata example

Run the development server:

python manage.py runserver

Run a default worker pool:

python manage.py runworkers

Run a worker pool with a broker and 8 workers:

python manage.py runworkers -b -w8

Example Graph

For demonstration purposes, the database is initialized with an example of a very simple directed acyclic graph (DAG) of connected nodes representing processors and their parameters:

(in_0) ──> A ──> B ──> D ───────> (out_0)
           |           ^
           └───> C ────┴──> E ──> (out_1)

To dump new example data from the database to example.yaml:

python manage.py dumpdata --format yaml -o procs/fixtures/example.yaml procs