atomic-gang-provisioner

Atomic, N-or-0 node provisioning for gang (distributed) jobs on Amazon EKS, wired into Kueue as a custom AdmissionCheck.

atomic-gang-provisioner is an annotation-driven, in-cluster Kubernetes controller that provisions a fixed set of N EKS nodes for a gang job all-or-nothing, and only lets the job's pods run once the entire node set exists. You annotate a Job with the gang's shape (count, instance-type, region, …); Kueue reserves quota and raises a pending AdmissionCheck; this controller atomically acquires exactly N EC2 instances (or zero); the nodes self-join via nodeadm; the controller flips the check to Ready; Kueue admits the workload and the pods schedule onto those nodes. When the job finishes, the controller releases the nodes (after an optional keep-warm window).

It does not fork, patch, or replace Kueue. Kueue remains the scheduler and quota manager; this controller is a separate process that Kueue calls out to through the standard AdmissionCheck extension point. Kueue and EKS are prerequisites — the chart installs neither.

Reality check on pure on-demand capacity. Large GPU gangs (for example, 4× p4d.24xlarge) requested purely on-demand frequently come back InsufficientInstanceCapacity from EC2. This controller's behaviour on a shortfall is deliberate: it holds the workload pending at $0 and retries the whole gang, rather than stranding a partial, idle-but-billing node set. For any large or latency-sensitive gang you should reserve capacity (On-Demand Capacity Reservation or EC2 Capacity Block) and pass its id via atomic-gang.k8s.aws/capacity-reservation-id.

Problem statement

On AWS, no mainstream node autoscaler provisions atomically:

Tool	Behaviour for a gang of N
Karpenter	Adds nodes incrementally / best-effort; can settle on a partial set.
Cluster Autoscaler	Scales node groups up incrementally; no all-or-nothing semantics.
Cluster-API-Provider-AWS	Reconciles machines individually; no N-or-0 primitive.

All of them can leave you with some of the nodes: the gang can't start (it needs all N), yet the nodes that did come up are running and billing. For GPU instances that is expensive idle burn.

The only true "N nodes or 0" primitives on EC2 are:

RunInstances with MinCount == MaxCount == N (single-AZ). EC2 admission-checks the entire request before anything launches: either all N enter pending, or the call fails and nothing is billed. Strongest atomicity, zero billing window.
CreateFleet with Type=instant + MinTargetCapacity=N (multi-AZ, one call). EC2 fails the instant fleet if it cannot satisfy N. The controller also validates the returned ID count and rolls back any malformed result as defense in depth.

This controller wires those two primitives into EKS node bootstrap (nodeadm) and into Kueue admission, so a gang job is admitted only when its full node set physically exists.

How it works

                       ┌──────────────────────── Kueue (scheduler / quota) ──────────────────────┐
                       │                                                                          │
  kubectl apply Job    │   Workload created ──▶ quota reserved ──▶ AdmissionCheck: PENDING        │
  (queue-name label +  │                                              │                           │
   atomic-gang         │                                              │ (Kueue is the CALLER)     │
   annotations)  ──────┼──────────────────────────────────────────▶  │                           │
                       └──────────────────────────────────────────────┼──────────────────────────┘
                                                                       │
                                              atomic-gang-provisioner  ▼  (the CALLEE)
                       ┌──────────────────────────────────────────────────────────────────────────┐
                       │  Workload reconciler: upsert GangNodePool.spec (shape + claim)             │
                       │            │                                                               │
                       │            ▼                                                               │
                       │  GangNodePool reconciler (serialized per pool):                            │
                       │     RunInstances(Min==Max==N)  ── or ──  CreateFleet(instant, MinTarget=N) │
                       │            │                                                               │
                       │      all N?├── no ─▶ retry-while-PENDING at $0  (InsufficientCapacity)     │
                       │            │                              malformed response ▶ rollback   │
                       │            ▼ yes                                                           │
                       │     N instances launched ──▶ nodeadm self-join ──▶ N Ready nodes           │
                       │            │  (join watchdog bounds the wait)                              │
                       │            ▼                                                               │
                       │     Phase=Satisfied ──▶ flip AdmissionCheckState: PENDING ▶ READY          │
                       └──────────────────────────────────────────────────────────────────────────┘
                                                                       │
                       ┌───────────────────────────────────────────────┼──────────────────────────┐
                       │  Kueue admits the Workload ──▶ pods schedule onto the gang nodes           │
                       │  (ResourceFlavor nodeSelector + toleration match the gang taint/labels)    │
                       │                                                                            │
                       │  Job completes ──▶ claim removed ──▶ keep-warm TTL ──▶ nodes released       │
                       └────────────────────────────────────────────────────────────────────────────┘

The AdmissionCheck integration (who calls whom)

Kueue owns admission. When a Workload's ClusterQueue carries an AdmissionCheck whose spec.controllerName matches ours, Kueue creates an AdmissionCheckState and waits for us to resolve it. We are the callee: we never mutate Kueue's scheduling logic; we only report Pending → Ready (or Retry) on the check for that workload. This is the sanctioned extension point, so you can upgrade Kueue independently.

Admission integration

admissioncheck — A Job opts in with the standard kueue.x-k8s.io/queue-name label routing it to a LocalQueue whose ClusterQueue carries our AdmissionCheck. This is the only supported mode. Direct Job watching cannot prevent pod creation before capacity exists, so the binary rejects it.

Architecture

The manager runs three controller units plus a periodic reaper. The design enforces strict single-writer ownership of each object so there is never a write race.

Unit	Watches	Writes	Role
AdmissionCheck reconciler	`AdmissionCheck` (filtered to our `controllerName`)	`AdmissionCheckActive=True` condition	Advertises liveness so Kueue knows the controller is present and healthy. Always registered.
Workload reconciler	Kueue `Workload`	sole writer of `GangNodePool.spec`; flips the per-Workload `AdmissionCheckState`	Parses/validates annotations, manages the pool's sole structured claim, and moves the check `Pending → Ready`/`Retry`.
GangNodePool reconciler	`GangNodePool`	sole writer of `GangNodePool.status` + all EC2 lifecycle	Owns acquisition: `RunInstances`/`CreateFleet`, join tracking, release. Serialized per pool (reconcile key == pool name) so a pool can never double-acquire.
Reaper (periodic `Runnable`)	— (lists EC2 by tag)	terminates `managed=true` instances	Leak backstop: sweeps for max-lifetime / expired keep-warm / orphaned instances every `reapIntervalSeconds`.

The `GangNodePool` CRD is controller-owned

GangNodePool (group atomic-gang.k8s.aws, v1alpha1, cluster-scoped, short name gnp) is an internal record — not user configuration. Users never author one; they annotate Jobs and the controller upserts the pool. spec is the desired node set (a snapshot of the first claimant's shape plus one namespaced Workload reference); status records the active acquisition group, exact EC2 instance IDs, Ready count, keep-warm expiry, and crash-recovery bookkeeping.

Safety properties

Real atomicity (N-or-0). RunInstances Min==Max==N and instant CreateFleet MinTargetCapacity=N acquire N or fail. Returned IDs are still validated and any malformed set is terminated.
No pods before exact readiness. Ready requires exactly N live EC2 IDs and N Ready Nodes with matching provider IDs and active-generation labels. Kueue PodSetUpdates add that pool and group as node selectors.
One active Workload per pool. A second claimant receives Retry, releases its quota, and requeues. Deleted, evicted, rejected, quota-less, and finished Workloads release their structured claims.
Crash idempotency via ClientToken. Every launch carries a ClientToken seeded from status.acquireGeneration. If the controller dies between "EC2 returned" and "instance ids persisted", the next reconcile reissues the same token and EC2 hands back the existing reservation instead of double-launching. The generation is bumped only after a terminal result (confirmed success or capacity shortfall).
Fleet defense in depth. Although MinTargetCapacity=N makes a shortfall atomic, any unexpected non-N response is terminated before retry.
Join watchdog. If the N launched instances don't become Ready nodes within joinDeadlineSeconds (nodeadm failure, AMI/userdata issue), the acquisition is considered failed and rolled forward — no silently-stuck half-joined pool.
Keep-warm TTL with reset-on-new-job. After a pool goes idle its nodes are retained for keepWarmTTL so the next claiming job reuses them (skipping cold acquisition + join). A new claim clears the window. KeepWarmUntil is mirrored to an EC2 tag so the independent reaper can enforce it after a controller restart.
Max-lifetime reaper. A hard per-pool instance age cap (maxLifetime) is enforced by the reaper regardless of claims, so nothing runs forever. A finalizer guarantees instances are terminated before a GangNodePool is deleted — a kubectl delete can never orphan a billing set.
VPC CNI ENI cleanup. Before terminating nodes, the controller records secondary ENIs whose attachments have DeleteOnTermination=false. It waits for detachment and deletes them, and can recover a detached VPC CNI ENI on retry from its exact aws-K8S-<instance-id> description. This prevents leaked ENIs from retaining IPs or blocking security-group deletion.

Prerequisites

The chart deploys the controller only. You must already have:

Amazon EKS >= 1.30 with a working worker-node join path (cluster security group, node IAM role, subnets). Gang nodes launched by the controller join this cluster.
EKS Access Entries for the node IAM role. The nodeadm-launched instances must be authorized to register.

aws-auth ConfigMap-only clusters need migration to access entries (or a matching aws-auth mapping for the node role). Access entries are the supported, API-driven path and are strongly recommended.
Kueue v0.18.3 or newer installed, exposing the kueue.x-k8s.io/v1beta2 API, with a waitForPodsReady configuration so Kueue enforces pod-set readiness as a gang.
A node IAM role + instance profile, pre-provisioned out-of-band. Profile names use atomic-gang-instance-profile-<cluster>-<role-name> (hash-truncated at the IAM limit). By default the controller verifies the profile and expected role.
Controller IAM identity via EKS Pod Identity (recommended) or IRSA, with the least-privilege actions in the IAM appendix.
(Optional) cert-manager, if you use webhook.certMode=certManager (the default) for the validating webhook.
(Optional) Prometheus Operator CRDs, if you enable metrics.serviceMonitor.enabled.

Install from source

The repository ships the controller source and Helm chart together. A fresh install does not depend on a pre-published controller image or OCI chart.

Set the target environment:

export AWS_REGION=us-east-1
export CLUSTER_NAME=my-eks-cluster
export NODE_ROLE_ARN=arn:aws:iam::123456789012:role/eks-gang-node-role
export CONTROLLER_ROLE_ARN=arn:aws:iam::123456789012:role/atomic-gang-controller-role
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export IMAGE_REPOSITORY="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/atomic-gang-provisioner"
export IMAGE_TAG="$(git rev-parse --short=12 HEAD)"

Create the ECR repository once, authenticate, and publish a multi-architecture image:

aws ecr describe-repositories \
  --region "${AWS_REGION}" \
  --repository-names atomic-gang-provisioner

aws ecr get-login-password --region "${AWS_REGION}" |
  docker login \
    --username AWS \
    --password-stdin "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag "${IMAGE_REPOSITORY}:${IMAGE_TAG}" \
  --push .

If describe-repositories reports RepositoryNotFoundException, create it with:

aws ecr create-repository \
  --region "${AWS_REGION}" \
  --repository-name atomic-gang-provisioner

Before installing the chart:

Install Kueue v0.18.3+ and configure its managerConfig with waitForPodsReady (see the Kueue installation guide and pod-readiness guide).
Attach the controller policy from the IAM appendix to CONTROLLER_ROLE_ARN.
Create the convention-named instance profile containing NODE_ROLE_ARN, and create an EKS access entry for that role with type EC2_LINUX. The optional nodeBootstrap chart hook can perform this, but the default and least-privilege path is to provision both out of band.
Install cert-manager, or explicitly choose another supported webhook certificate mode. Do not disable the validating webhook in production.

Create the controller identity and EKS Pod Identity association:

kubectl create namespace atomic-gang-system \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create serviceaccount atomic-gang-controller \
  --namespace atomic-gang-system \
  --dry-run=client -o yaml | kubectl apply -f -

aws eks create-pod-identity-association \
  --region "${AWS_REGION}" \
  --cluster-name "${CLUSTER_NAME}" \
  --namespace atomic-gang-system \
  --service-account atomic-gang-controller \
  --role-arn "${CONTROLLER_ROLE_ARN}"

Install the chart from the checkout:

helm upgrade --install atomic-gang ./charts/atomic-gang-provisioner \
  --namespace atomic-gang-system --create-namespace \
  --set serviceAccount.create=false \
  --set-string clusterName="${CLUSTER_NAME}" \
  --set-string defaults.region="${AWS_REGION}" \
  --set-string defaults.nodeRoleArn="${NODE_ROLE_ARN}" \
  --set-string image.repository="${IMAGE_REPOSITORY}" \
  --set-string image.tag="${IMAGE_TAG}" \
  --set-string kueue.clusterQueue.nominalQuota.cpu=256 \
  --set-string kueue.clusterQueue.nominalQuota.memory=1Ti \
  --wait --timeout 5m

clusterName, defaults.region, and non-zero ClusterQueue CPU/memory quotas are required. defaults.nodeRoleArn may be omitted only when every Job supplies one.

Pod Identity association. With auth.mode=podIdentity (the default), associate the controller's ServiceAccount (serviceAccount.name, default atomic-gang-controller) with its IAM role out of band. The Helm chart cannot create this AWS API resource. If an association already exists, inspect or update it rather than creating a duplicate:

aws eks create-pod-identity-association \
  --cluster-name my-eks-cluster \
  --namespace atomic-gang-system \
  --service-account atomic-gang-controller \
  --role-arn arn:aws:iam::123456789012:role/atomic-gang-controller-role

For IRSA instead, set auth.mode=irsa and serviceAccount.roleArn (rendered as the eks.amazonaws.com/role-arn annotation).

Usage

A complete, annotated example Job

apiVersion: batch/v1
kind: Job
metadata:
  name: allreduce-pretrain
  namespace: default
  labels:
    # (1) Opt in via Kueue: routes the Workload to a LocalQueue whose ClusterQueue
    #     carries our AdmissionCheck. THIS is the opt-in in admissioncheck mode.
    kueue.x-k8s.io/queue-name: atomic-gang-queue
    # (2) Pool key: keep-warm/reuse bucket AND the GangNodePool name. Jobs sharing a
    #     pool reuse one node set and reset each other's keep-warm TTL.
    atomic-gang.k8s.aws/pool: pretrain-p4d
  annotations:
    atomic-gang.k8s.aws/count: "4"                 # N — the all-or-nothing gang size
    atomic-gang.k8s.aws/instance-type: p4d.24xlarge
    atomic-gang.k8s.aws/mode: runinstances         # runinstances (single-AZ) | fleet (multi-AZ)
    atomic-gang.k8s.aws/region: us-east-1
    atomic-gang.k8s.aws/availability-zones: us-east-1a  # single-AZ for runinstances
    atomic-gang.k8s.aws/node-role-arn: arn:aws:iam::123456789012:role/eks-gang-node-role
    atomic-gang.k8s.aws/keep-warm-ttl: 15m         # retain idle nodes for reuse
    atomic-gang.k8s.aws/max-lifetime: 6h           # hard age cap enforced by the reaper
    # Optional: reserve capacity for large GPU gangs to avoid InsufficientInstanceCapacity
    # atomic-gang.k8s.aws/capacity-reservation-id: cr-0123456789abcdef0
spec:
  parallelism: 4          # MUST equal count (Kueue gang-admits the whole pod set)
  completions: 4
  completionMode: Indexed
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: trainer
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/trainer:latest
          resources:
            limits:
              nvidia.com/gpu: "8"

count must equal the Job's total pod count. The controller rejects a mismatch (CountMismatch) so Kueue's quota sizing and waitForPodsReady gang-admit exactly the gang.

Annotation & label reference

All keys live under the atomic-gang.k8s.aws/ prefix (except the standard Kueue label). Values are resolved and validated by internal/annotations; unset annotations fall back to the controller defaults (chart defaults.*).

Key	Required	Default	Description
`kueue.x-k8s.io/queue-name` (label)	Yes	—	Kueue queue-name. Routes the Workload to a LocalQueue carrying our AdmissionCheck. This is the opt-in.
`atomic-gang.k8s.aws/pool` (label)	Yes	—	Keep-warm/reuse pool key and the `GangNodePool` name. Must be a `<=63`-char RFC1123 label. Jobs sharing a pool reuse one node set.
`atomic-gang.k8s.aws/count`	Yes	—	Gang size N (min==max==N). Positive integer, `<= limits.maxGangSize`, and must equal the Job's total pod count.
`atomic-gang.k8s.aws/instance-type`	Yes	—	EC2 instance type. Must match `allowedInstanceTypes` (exact or trailing-`*` glob) when that allowlist is set.
`atomic-gang.k8s.aws/mode`	No	`defaults.mode` (`runinstances`)	Acquisition strategy: `runinstances` (single-AZ, N-or-0, $0 shortfall) or `fleet` (multi-AZ, rollback partials).
`atomic-gang.k8s.aws/region`	No	`defaults.region`	AWS region. Must equal the EKS cluster region in `defaults.region`.
`atomic-gang.k8s.aws/availability-zones`	No	discover from cluster subnets	Comma-separated AZ list. For `runinstances` it's an ordered sweep list; for `fleet` an override set. Each AZ must be in `region`. Mutually exclusive with `subnets`.
`atomic-gang.k8s.aws/subnets`	No	—	Comma-separated explicit subnet ids (`subnet-…`). Highest precedence; overrides AZ discovery. Mutually exclusive with `availability-zones`.
`atomic-gang.k8s.aws/node-role-arn`	Conditional	`defaults.nodeRoleArn`	Node IAM role passed to instances. Required if no default is configured. Must be in `allowedNodeRoleArns` (the controller's `iam:PassRole` is scoped to it).
`atomic-gang.k8s.aws/keep-warm-ttl`	No	`defaults.keepWarmTTL` (`10m`)	Go duration; how long idle nodes are retained for reuse. Non-negative, `<= limits.maxKeepWarmTTL`.
`atomic-gang.k8s.aws/max-lifetime`	No	`defaults.maxLifetime` (`4h`)	Go duration; hard instance age cap enforced by the reaper. Positive.
`atomic-gang.k8s.aws/ami-variant`	No	derived from instance family	EKS AL2023 image line: `standard`, `nvidia` (`p`/`g`), or `neuron` (`inf`/`trn`).
`atomic-gang.k8s.aws/ami-id`	No	SSM resolution	Explicit AMI id (`ami-…`); overrides SSM lookup.
`atomic-gang.k8s.aws/capacity-reservation-id`	No	—	ODCR / Capacity Block target (`cr-…`). Empty ⇒ on-demand. Recommended for large GPU gangs.

Chart values reference (key knobs)

Value	Default	Description
`clusterName`	`""` (required)	EKS cluster the gang nodes join; drives ownership tag + `nodeadm` join params.
`defaults.region`	`""` (required)	EKS cluster region and the only supported launch region.
`defaults.nodeRoleArn`	`""` (required unless every Job sets its own)	Default node IAM role ARN passed to launched instances.
`defaults.mode`	`runinstances`	Default acquisition mode.
`defaults.keepWarmTTL`	`10m`	Default idle-node retention window.
`defaults.maxLifetime`	`4h`	Default hard instance age cap.
`controllerName`	`atomic-gang.k8s.aws/ec2-atomic`	`AdmissionCheck.spec.controllerName` we own. Treat as fixed — changing it orphans in-flight Workloads.
`integrationMode`	`admissioncheck`	Only supported integration mode.
`replicaCount`	`2`	Controller replicas; leader election keeps one active (warm standby).
`auth.mode`	`podIdentity`	`podIdentity` \| `irsa` \| `static`/`node`.
`limits.maxGangSize`	`64`	Max N any Job may request.
`limits.maxKeepWarmTTL`	`24h`	Max keep-warm TTL any Job may request.
`allowedInstanceTypes`	`[]` (any)	Allowlist; exact or trailing-`*` glob.
`allowedNodeRoleArns`	`[]` (only `defaults.nodeRoleArn`)	Node-role allowlist; the controller's `iam:PassRole` is scoped to this set.
`reapIntervalSeconds`	`120`	Reaper sweep interval.
`joinDeadlineSeconds`	`600`	Node-join watchdog deadline.
`acquireAttemptsBeforeRetry`	`20`	Consecutive shortfalls tolerated at `Pending` before flipping the check to `Retry`.
`maxConcurrentPoolReconciles`	`5`	Cross-pool parallelism (a single pool is always serialized).
`leaderElection.enabled`	`true`	Single-active-replica election; keep on when `replicaCount > 1`.
`webhook.enabled` / `webhook.certMode` / `webhook.failurePolicy`	`true` / `certManager` / `Fail`	Validating Job webhook (the fail-closed trust boundary).
`nodeBootstrap.enabled`	`false`	Run the elevated install hook that creates node access entries and instance profiles.
`iam.selfManageInstanceProfile`	`false`	When `true`, the controller creates/attaches the instance profile (needs extra IAM).
`kueue.create.*`	`true`	Create the AdmissionCheck / ResourceFlavor / ClusterQueue / LocalQueue against the pre-existing Kueue install.
`kueue.clusterQueue.nominalQuota`	required, no default	Non-zero Kueue accounting limits required before acquisition can begin.
`kueue.localQueueNamespaces`	`[default]`	Namespaces that get a LocalQueue.
`metrics.serviceMonitor.enabled`	`false`	Create a Prometheus Operator ServiceMonitor for `/metrics`.
`podDisruptionBudget.enabled`	`true`	PDB to keep the leader available during voluntary disruptions.

Verifying

# 1. The AdmissionCheck should report Active=True (controller is present + healthy).
kubectl get admissioncheck
# NAME                      AGE
# atomic-gang-check         3m
kubectl get admissioncheck atomic-gang-check -o jsonpath='{.status.conditions}'

# 2. Inspect the internal pool records (controller-owned; short name: gnp).
kubectl get gangnodepools
# NAME            PHASE       READY   COUNT   TYPE            MODE           AGE
# pretrain-p4d    Satisfied   4       4       p4d.24xlarge    runinstances   6m

# 3. Watch the Workload's admission move Pending -> Ready as nodes join.
kubectl get workloads
kubectl describe workload <workload-name>   # see the AdmissionCheckState transitions

The GangNodePool phase walks Idle → Acquiring → Joining → Satisfied on the way up, and Satisfied → KeepWarm → Releasing on the way down.

Modes, concurrency & pools

Sequential pool reuse. The atomic-gang.k8s.aws/pool label is the reuse bucket. One Workload owns it at a time; later claimants receive Retry. Once released, the next Workload can reuse the same warm node set without cold acquisition or join.
Keep-warm. When the last claim drops, nodes are retained until KeepWarmUntil (keepWarmTTL after idle). The expiry is tagged on EC2 so the independent reaper can enforce it after a controller restart. A deleted pool is treated as an orphan and terminated.
Shape-conflict rejection. A GangNodePool's shape is a snapshot of the first claimant. A later claimant whose shape differs (count, instance-type, mode, region, role, AMI, AZs/subnets, or max lifetime — see GangSpec.ShapeEquals) is Rejected rather than silently mutating a live pool. Keep-warm TTL may vary between sequential claimants. Use distinct pool keys for distinct shapes.
Per-pool serialization. The GangNodePool reconcile key is the pool name, so all work for a pool is serialized on the elected leader — the anti-double-acquire guarantee. maxConcurrentPoolReconciles only bounds parallelism across pools; replicaCount never widens per-pool concurrency (leader election keeps one active).

IAM appendix

Controller role (least privilege)

Attach to the role assumed by the controller ServiceAccount (Pod Identity / IRSA). Tag conditions scope every mutating call to instances this controller owns.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LaunchGang",
      "Effect": "Allow",
      "Action": ["ec2:RunInstances", "ec2:CreateFleet"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/atomic-gang.k8s.aws/managed": "true"
        }
      }
    },
    {
      "Sid": "ReleaseGang",
      "Effect": "Allow",
      "Action": ["ec2:TerminateInstances", "ec2:DeleteFleets"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/atomic-gang.k8s.aws/managed": "true"
        }
      }
    },
    {
      "Sid": "DescribeForReconcile",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeFleets",
        "ec2:DescribeFleetInstances",
        "ec2:DescribeSubnets",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:CreateTags"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DeleteDetachedVPCCNIInterfaces",
      "Effect": "Allow",
      "Action": "ec2:DeleteNetworkInterface",
      "Resource": "arn:aws:ec2:us-east-1:123456789012:network-interface/*"
    },
    {
      "Sid": "ManageGangLaunchTemplates",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateLaunchTemplate",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:ModifyLaunchTemplate",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeleteLaunchTemplateVersions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VerifyNodeInstanceProfile",
      "Effect": "Allow",
      "Action": "iam:GetInstanceProfile",
      "Resource": "*"
    },
    {
      "Sid": "PassNodeRoleToEC2Only",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::123456789012:role/eks-gang-node-role"
      ],
      "Condition": {
        "StringEquals": { "iam:PassedToService": "ec2.amazonaws.com" }
      }
    },
    {
      "Sid": "ResolveClusterAndAMI",
      "Effect": "Allow",
      "Action": ["eks:DescribeCluster", "eks:DescribeAccessEntry", "ssm:GetParameter"],
      "Resource": "*"
    }
  ]
}

ec2:RunInstances / ec2:CreateFleet — tag-conditioned on aws:RequestTag/atomic-gang.k8s.aws/managed=true, so the controller can only launch instances it stamps as managed.
ec2:TerminateInstances / ec2:DeleteFleets — conditioned on the resource tag atomic-gang.k8s.aws/managed=true, so it can only tear down its own instances.
ec2:DescribeNetworkInterfaces / ec2:DeleteNetworkInterface — find VPC CNI secondary ENIs before node termination and delete them after detachment. Scope the delete ARN to the cluster region/account. These ENIs do not inherit the instance's managed tag, so the controller also bounds deletion at runtime to ENIs attached to the target instances or carrying an exact aws-K8S-<instance-id> description.
Other Describe* / CreateTags actions read state and stamp ownership/keep-warm tags.
iam:PassRole — scoped to exactly allowedNodeRoleArns (list every allowed node role ARN in Resource) and iam:PassedToService=ec2.amazonaws.com.
eks:DescribeCluster / eks:DescribeAccessEntry — fetch join parameters and verify the node role is authorized before launch.
ssm:GetParameter — resolve the EKS-optimized AL2023 AMI id.
If iam.selfManageInstanceProfile=true, additionally grant iam:CreateInstanceProfile, iam:AddRoleToInstanceProfile, iam:GetInstanceProfile (scoped to the profile you let it manage).

Node role (the role passed to gang instances)

The defaults.nodeRoleArn (and any entry in allowedNodeRoleArns) is a standard EKS worker node role. Attach the EKS worker managed policies:

AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy (or attach it to the VPC CNI's own role if using IRSA for the CNI)
AmazonEC2ContainerRegistryReadOnly
AmazonSSMManagedInstanceCore (optional, for SSM access to nodes)

Register this role with the cluster via an EKS Access Entry (recommended) so nodeadm-launched instances can join.

Limitations & notes

Pure on-demand capacity is not guaranteed. Large GPU gangs (e.g. multiple p4d, p5) commonly return InsufficientInstanceCapacity on-demand. The controller does the right thing — retry the whole gang while holding the workload pending at $0, and after acquireAttemptsBeforeRetry consecutive shortfalls it flips the check to Retry so Kueue can release the quota reservation and reconsider (another flavor / backoff). For reliable placement, reserve capacity (ODCR / Capacity Block) and pass capacity-reservation-id.
Both modes fail atomically on a capacity shortfall. RunInstances Min==Max==N is single-AZ; instant CreateFleet MinTargetCapacity=N can use a multi-AZ capacity net. The controller retains rollback code only as defense against an unexpected malformed response.
Kueue is pinned to v1beta2. The controller builds against kueue.x-k8s.io/v1beta2 (and controller-runtime v0.24.1 / k8s.io v0.36.2). Bump the module deps in lockstep with the Kueue release you target.
aws-auth-only clusters must migrate to access entries (or add a matching node mapping) for gang nodes to register.
The GangNodePool CRD is internal. Don't hand-edit it; drive everything through Job annotations.

Build & develop

Requires Go 1.26+. controller-gen (v0.17.2) is needed only to regenerate API artifacts; a normal build uses the checked-in CRD and DeepCopy code.

make build          # go build -o bin/manager ./cmd   (runs fmt + vet)
make test           # unit tests with coverage
make run            # run the controller against your kubeconfig's cluster

make manifests      # regenerate CRDs/RBAC/webhook (needs controller-gen)
make generate       # regenerate zz_generated.deepcopy.go (needs controller-gen)
make helm-sync      # copy config/crd + config/rbac into the chart
make helm-lint      # helm lint the chart
make helm-test      # positive and fail-closed chart renders
make helm-package   # package the chart into ./bin

make docker-build IMG=<repo>:<tag>
make docker-push  IMG=<repo>:<tag>

After changing api/v1alpha1 types, run make generate && make manifests && make helm-sync and commit the regenerated artifacts.

License

Apache License 2.0.

Name		Name	Last commit message	Last commit date
api/v1alpha1		api/v1alpha1
charts/atomic-gang-provisioner		charts/atomic-gang-provisioner
cmd		cmd
config/crd/bases		config/crd/bases
hack		hack
internal		internal
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum

Navigation Menu

Repository navigation

About

Repository files navigation

atomic-gang-provisioner

Problem statement

How it works

The AdmissionCheck integration (who calls whom)

Admission integration

Architecture

The `GangNodePool` CRD is controller-owned

Safety properties

Prerequisites

Install from source

Usage

A complete, annotated example Job

Annotation & label reference

Chart values reference (key knobs)

Verifying

Modes, concurrency & pools

IAM appendix

Controller role (least privilege)

Node role (the role passed to gang instances)

Limitations & notes

Build & develop

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

atomic-gang-provisioner

Problem statement

How it works

The AdmissionCheck integration (who calls whom)

Admission integration

Architecture

The GangNodePool CRD is controller-owned

Safety properties

Prerequisites

Install from source

Usage

A complete, annotated example Job

Annotation & label reference

Chart values reference (key knobs)

Verifying

Modes, concurrency & pools

IAM appendix

Controller role (least privilege)

Node role (the role passed to gang instances)

Limitations & notes

Build & develop

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

The `GangNodePool` CRD is controller-owned