Implement ORB AWS EC2 Worker Adapter

.github/actions/run-test/action.yml

@@ -55,7 +55,13 @@ runs:
       run: |
         uv pip install --system -r examples/applications/requirements_applications.txt
         uv pip install --system -r examples/ray_compat/requirements.txt
+        readarray -t skip_examples < examples/skip_examples.txt
         for example in "./examples"/*.py; do
+          filename=$(basename "$example")
+          if [[ " ${skip_examples[*]} " =~ [[:space:]]${filename}[[:space:]] ]]; then
+            echo "Skipping $example"
+            continue
+          fi
           echo "Running $example"
           python $example
         done

.gitignore

@@ -36,6 +36,9 @@ CMakeFiles/
 src/scaler/protocol/capnp/*.c++
 src/scaler/protocol/capnp/*.h
 
+orb/logs/
+orb/metrics/
+
 # AWS HPC test-generated files
 .scaler_aws_batch_config.json
 .scaler_aws_hpc.env

README.md

@@ -279,6 +279,7 @@ The following table maps each Scaler command to its corresponding section name i
 | `scaler_worker_manager symphony`        | `[[worker_manager]]` + `type = "symphony"`          |
 | `scaler_worker_manager aws_raw_ecs`     | `[[worker_manager]]` + `type = "aws_raw_ecs"`       |
 | `scaler_worker_manager aws_hpc`         | `[[worker_manager]]` + `type = "aws_hpc"`           |
+| `scaler_worker_manager orb_aws_ec2`     | `[[worker_manager]]` + `type = "orb_aws_ec2"`       |
 
 ### Practical Scenarios & Examples
 
@@ -507,6 +508,40 @@ where `deepest_nesting_level` is the deepest nesting level a task has in your wo
 workload that has
 a base task that calls a nested task that calls another nested task, then the deepest nesting level is 2.
 
+## ORB AWS EC2 integration
+
+A Scaler scheduler can interface with ORB (Open Resource Broker) to dynamically provision and manage workers on AWS EC2 instances.
+
+```bash
+$ scaler_worker_manager orb_aws_ec2 tcp://127.0.0.1:2345 --image-id ami-0528819f94f4f5fa5
+```
+
+This will start an ORB AWS EC2 worker adapter that connects to the Scaler scheduler at `tcp://127.0.0.1:2345`. The scheduler can then request new workers from this adapter, which will be launched as EC2 instances.
+
+The ORB AWS EC2 worker manager can also be included in a `scaler` all-in-one TOML config:
+
+```toml
+[scheduler]
+scheduler_address = "tcp://127.0.0.1:2345"
+
+[[worker_manager]]
+type = "orb_aws_ec2"
+scheduler_address = "tcp://127.0.0.1:2345"
+image_id = "ami-0528819f94f4f5fa5"
+instance_type = "t3.medium"
+aws_region = "us-east-1"
+```
+
+### Configuration
+
+The ORB AWS EC2 adapter requires `orb-py` and `boto3` to be installed. You can install them with:
+
+```bash
+$ pip install "opengris-scaler[orb_aws_ec2]"
+```
+
+For more details on configuring ORB AWS EC2, including AWS credentials and instance templates, please refer to the [ORB AWS EC2 Worker Adapter documentation](https://finos.github.io/opengris-scaler/tutorials/worker_manager_adapter/orb_aws_ec2.html).
+
 ## Worker Manager usage
 
 > **Note**: This feature is experimental and may change in future releases.

docs/source/tutorials/commands.rst

@@ -14,7 +14,7 @@ After installing ``opengris-scaler``, the following CLI commands are available f
    * - :ref:`scaler_scheduler <cmd-scaler-scheduler>`
      - Start only the scheduler process (and auto-start object storage when needed).
    * - :ref:`scaler_worker_manager <cmd-scaler-worker-manager>`
-     - Start one worker manager using a subcommand (``baremetal_native``, ``symphony``, ``aws_raw_ecs``, ``aws_hpc``).
+     - Start one worker manager using a subcommand (``baremetal_native``, ``symphony``, ``aws_raw_ecs``, ``aws_hpc``, ``orb_aws_ec2``).
    * - :ref:`scaler_object_storage_server <cmd-scaler-object-storage-server>`
      - Start only the object storage server.
    * - :ref:`scaler_top <cmd-scaler-top>`
@@ -53,6 +53,8 @@ All commands support ``--config``/``-c``. In practice, most deployments use TOML
      - ``[[worker_manager]]`` + ``type = "aws_raw_ecs"``
    * - ``scaler_worker_manager aws_hpc``
      - ``[[worker_manager]]`` + ``type = "aws_hpc"``
+   * - ``scaler_worker_manager orb_aws_ec2``
+     - ``[[worker_manager]]`` + ``type = "orb_aws_ec2"``
 
 
 .. _cmd-scaler:
@@ -352,6 +354,7 @@ Available subcommands:
 - ``symphony``
 - ``aws_raw_ecs``
 - ``aws_hpc``
+- ``orb_aws_ec2``
 
 When ``--config``/``-c`` is supplied, ``scaler_worker_manager`` reads the ``[[worker_manager]]``
 array from the TOML file and picks the entry whose ``type`` field matches the subcommand.
@@ -753,6 +756,79 @@ AWS Batch worker manager.
      - ``60``
      - Timeout for each submitted job.
 
+Subcommand: ``orb_aws_ec2``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ORB (Open Resource Broker) worker manager — dynamically provisions workers on AWS EC2 instances.
+
+.. code-block:: bash
+
+    $ scaler_worker_manager orb_aws_ec2 [options] <scheduler_address>
+
+.. tabs::
+
+    .. group-tab:: command line
+
+        .. code-block:: bash
+
+            $ scaler_worker_manager orb_aws_ec2 tcp://127.0.0.1:6378 \
+                --object-storage-address tcp://127.0.0.1:6379 \
+                --image-id ami-0528819f94f4f5fa5 \
+                --instance-type t3.medium \
+                --aws-region us-east-1
+
+    .. group-tab:: config.toml
+
+        .. code-block:: toml
+
+            [[worker_manager]]
+            type = "orb_aws_ec2"
+            scheduler_address = "tcp://127.0.0.1:6378"
+            object_storage_address = "tcp://127.0.0.1:6379"
+            image_id = "ami-0528819f94f4f5fa5"
+            instance_type = "t3.medium"
+            aws_region = "us-east-1"
+
+        Run command:
+
+        .. code-block:: bash
+
+            $ scaler config.toml
+
+.. list-table::
+   :header-rows: 1
+
+   * - Argument
+     - Required
+     - Default
+     - Description
+   * - ``--image-id``
+     - Yes
+     - -
+     - AMI ID for the worker EC2 instances.
+   * - ``--instance-type``
+     - No
+     - ``t2.micro``
+     - EC2 instance type.
+   * - ``--aws-region``
+     - No
+     - ``us-east-1``
+     - AWS region.
+   * - ``--key-name``
+     - No
+     - ``None``
+     - AWS key pair name. A temporary key pair is created if omitted.
+   * - ``--subnet-id``
+     - No
+     - ``None``
+     - AWS subnet ID. Defaults to the default subnet in the default VPC.
+   * - ``--security-group-ids``
+     - No
+     - ``[]``
+     - Comma-separated AWS security group IDs. A temporary group is created if omitted.
+
+For full details, see :doc:`worker_managers/orb_aws_ec2`.
+
 
 .. _cmd-scaler-object-storage-server:
 

docs/source/tutorials/compatibility/ray.rst

@@ -6,7 +6,7 @@ Ray
 Scaler is a lightweight distributed computation engine similar to Ray. Scaler supports many of the same concepts as Ray including
 remote functions (known as tasks in Scaler), futures, cluster object storage, labels (known as capabilities in Scaler), and it comes with comparable monitoring tools.
 
-Unlike Ray, Scaler supports both local clusters and also easily integrates with multiple cloud providers out of the box, including AWS EC2 and IBM Symphony,
+Unlike Ray, Scaler supports both local clusters and also easily integrates with multiple cloud providers out of the box, including ORB (AWS EC2) and IBM Symphony,
 with more providers planned for the future. You can view our `roadmap on GitHub <https://github.com/finos/opengris-scaler/discussions/333>`_
 for details on upcoming cloud integrations.
 

docs/source/tutorials/worker_managers/index.rst

@@ -54,6 +54,10 @@ Worker Managers Overview
      - Offloads tasks to IBM Spectrum Symphony via the SOAM API.
      - Concurrency-limited
      - IBM Symphony
+   * - :doc:`ORB AWS EC2 <orb_aws_ec2>`
+     - Dynamically provisions workers on AWS EC2 instances using the ORB system.
+     - Dynamic (scheduler-driven)
+     - AWS EC2
 
 Although worker managers target different infrastructures, many configuration options are shared.
 See :doc:`Common Worker Manager Parameters <common_parameters>` for these shared settings.
@@ -72,4 +76,5 @@ The :ref:`scaler <cmd-scaler>` command boots the full stack from a single TOML c
     aws_hpc_batch
     aws_raw_ecs
     symphony
+    orb_aws_ec2
     common_parameters

docs/source/tutorials/worker_managers/orb_aws_ec2.rst

@@ -0,0 +1,145 @@
+ORB AWS EC2 Worker Adapter
+==========================
+
+The ORB AWS EC2 worker adapter allows Scaler to dynamically provision workers on AWS EC2 instances using the ORB (Open Resource Broker) system. This is particularly useful for scaling workloads that require significant compute resources or specialized hardware available in the cloud.
+
+This tutorial describes the steps required to get up and running with the ORB AWS EC2 adapter.
+
+Requirements
+------------
+
+Before using the ORB AWS EC2 worker adapter, ensure the following requirements are met on the machine that will run the adapter:
+
+1.  **orb-py and boto3**: The ``orb-py`` and ``boto3`` packages must be installed. These can be installed using the ``orb_aws_ec2`` optional dependency of Scaler:
+
+    .. code-block:: bash
+
+        pip install "opengris-scaler[orb_aws_ec2]"
+
+2.  **AWS CLI**: The AWS Command Line Interface must be installed and configured with a default profile that has permissions to launch, describe, and terminate EC2 instances.
+
+3.  **Network Connectivity**: The adapter must be able to communicate with AWS APIs and the Scaler scheduler.
+
+Getting Started
+---------------
+
+To start the ORB AWS EC2 worker adapter, use the ``scaler_worker_manager orb_aws_ec2`` subcommand:
+
+.. code-block:: bash
+
+    scaler_worker_manager orb_aws_ec2 tcp://<SCHEDULER_EXTERNAL_IP>:8516 \
+        --object-storage-address tcp://<OSS_EXTERNAL_IP>:8517 \
+        --image-id ami-0528819f94f4f5fa5 \
+        --instance-type t3.medium \
+        --aws-region us-east-1 \
+        --logging-level INFO \
+        --task-timeout-seconds 60
+
+Equivalent configuration using a TOML file with ``scaler``:
+
+.. code-block:: toml
+
+    # stack.toml
+
+    [scheduler]
+    scheduler_address = "tcp://<SCHEDULER_EXTERNAL_IP>:8516"
+
+    [[worker_manager]]
+    type = "orb_aws_ec2"
+    scheduler_address = "tcp://<SCHEDULER_EXTERNAL_IP>:8516"
+    object_storage_address = "tcp://<OSS_EXTERNAL_IP>:8517"
+    image_id = "ami-0528819f94f4f5fa5"
+    instance_type = "t3.medium"
+    aws_region = "us-east-1"
+    logging_level = "INFO"
+    task_timeout_seconds = 60
+
+.. code-block:: bash
+
+    scaler stack.toml
+
+*   ``tcp://<SCHEDULER_EXTERNAL_IP>:8516`` is the address workers will use to connect to the scheduler.
+*   ``tcp://<OSS_EXTERNAL_IP>:8517`` is the address workers will use to connect to the object storage server.
+*   New workers will be launched using the specified AMI and instance type.
+
+Networking Configuration
+------------------------
+
+Workers launched by the ORB AWS EC2 adapter are EC2 instances and require an externally-reachable IP address for the scheduler.
+
+*   **Internal Communication**: If the machine running the scheduler is another EC2 instance in the same VPC, you can use EC2 private IP addresses.
+*   **Public Internet**: If communicating over the public internet, it is highly recommended to set up robust security rules and/or a VPN to protect the cluster.
+
+Publicly Available AMIs
+-----------------------
+
+We regularly publish publicly available Amazon Machine Images (AMIs) with Python and ``opengris-scaler`` pre-installed.
+
+.. list-table:: Available Public AMIs
+   :widths: 15 15 20 20 30
+   :header-rows: 1
+
+   * - Scaler Version
+     - Python Version
+     - Amazon Linux 2023 Version
+     - Date (MM/DD/YYYY)
+     - AMI ID (us-east-1)
+   * - 1.14.2
+     - 3.13
+     - 2023.10.20260120
+     - 01/30/2026
+     - ``ami-0528819f94f4f5fa5``
+   * - 1.15.0
+     - 3.13
+     - 2023.10.20260302.1
+     - 03/16/2026
+     - ``ami-044265172bea55d51``
+   * - 1.26.4
+     - 3.13
+     - 2023.10.20260302.1
+     - 03/26/2026
+     - ``ami-0b76605999d8f5d2b``
+
+New AMIs will be added to this list as they become available.
+
+Supported Parameters
+--------------------
+
+.. note::
+    For more details on how to configure Scaler, see the :doc:`../configuration` section.
+
+The ORB AWS EC2 worker adapter supports ORB-specific configuration parameters as well as common worker adapter parameters.
+
+ORB AWS EC2 Template Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*   ``--image-id`` (Required): AMI ID for the worker instances.
+*   ``--instance-type``: EC2 instance type (default: ``t2.micro``).
+*   ``--aws-region``: AWS region (default: ``us-east-1``).
+*   ``--key-name``: AWS key pair name for the instances. If not provided, a temporary key pair will be created and deleted on cleanup.
+*   ``--subnet-id``: AWS subnet ID where the instances will be launched. If not provided, it attempts to discover the default subnet in the default VPC.
+*   ``--security-group-ids``: Comma-separated list of AWS security group IDs.
+*   ``--allowed-ip``: IP address to allow in the security group (if created automatically). Defaults to the adapter's external IP.
+*   ``--orb-config-path``: Path to the ORB root directory (default: ``src/scaler/drivers/orb``).
+
+Common Parameters
+~~~~~~~~~~~~~~~~~
+
+For a full list of common parameters including networking, worker configuration, and logging, see :doc:`common_parameters`.
+
+Cleanup
+-------
+
+The ORB AWS EC2 worker adapter is designed to be self-cleaning, but it is important to be aware of the resources it manages:
+
+*   **Key Pairs**: If a ``--key-name`` is not provided, the adapter creates a temporary AWS key pair.
+*   **Security Groups**: If ``--security-group-ids`` are not provided, the adapter creates a temporary security group to allow communication.
+*   **Launch Templates**: ORB may additionally create EC2 Launch Templates as part of the machine provisioning process.
+
+The adapter attempts to delete these temporary resources and terminate all launched EC2 instances when it shuts down gracefully. However, in the event of an ungraceful crash or network failure, some resources may persist in your AWS account.
+
+.. tip::
+    It is recommended to periodically check your AWS console for any orphaned resources (instances, security groups, key pairs, or launch templates) and clean them up manually if necessary to avoid unexpected costs.
+
+.. warning::
+    **Subnet and Security Groups**: Currently, specifying ``--subnet-id`` or ``--security-group-ids`` via configuration might not have the intended effect as the adapter is designed to auto-discover or create these resources. Specifically, the adapter may still attempt to use default subnets or create its own temporary security groups regardless of these parameters.

pyproject.toml

@@ -50,10 +50,15 @@ graphblas = [
 aws = [
     "boto3",
 ]
+orb_aws_ec2 = [
+    "orb-py~=1.5.1; python_version >= '3.10'",
+    "boto3; python_version >= '3.10'",
+]
 all = [
     "opengris-scaler[aws]",
     "opengris-scaler[graphblas]",
     "opengris-scaler[gui]",
+    "opengris-scaler[orb_aws_ec2]",
     "opengris-scaler[uvloop]",
 ]
 

src/scaler/config/defaults.py

@@ -56,7 +56,7 @@
 # WORKER SPECIFIC OPTIONS
 
 # number of workers, echo worker use 1 process
-DEFAULT_MAX_TASK_CONCURRENCY = os.cpu_count() - 1
+DEFAULT_MAX_TASK_CONCURRENCY = os.cpu_count()
 
 # number of seconds that worker agent send heartbeat to scheduler
 DEFAULT_HEARTBEAT_INTERVAL_SECONDS = 2