K8s GPU Scheduler Lab

A local-first, hands-on lab for learning how Kubernetes discovers, schedules, shares, and recovers GPU capacity—without requiring a physical GPU.

K8s GPU Scheduler Lab runs a real Kubernetes control plane, a project-built Go kubelet device plugin, and real Kueue controllers against simulated accelerator devices. It is designed for infrastructure engineers who want to learn GPU platform mechanics on an Apple Silicon Mac before paying for GPU infrastructure.

Start with the hospital story · Run the first experiment · Browse experiments · Review validation boundaries · Read the plan

Why this project exists

GPU infrastructure is expensive and difficult to access, but many important platform-engineering problems happen before a CUDA process ever reaches the hardware. Kubernetes still has to discover devices, account for finite capacity, place Pods, react to health changes, admit team workloads, enforce quota, and resolve competition between tenants.

This repository turns those control-plane behaviors into small, reproducible experiments. Each experiment asks one question, records observable evidence, and states exactly where simulation ends. A result is never presented as hardware validation unless physical GPU evidence exists.

Start here: you are the hospital administrator

You have been asked to operate a hospital with four MRI machines shared by multiple departments. Demand is higher than capacity, machines can become unhealthy, some patients require a specific imaging wing, and each department has a limited budget.

Your job is to answer two different questions:

1. Should this scan be allowed to start now? The booking policy considers departmental quota, borrowing, priority, and historical usage.
2. If it may start, which machine can perform it? The assignment system considers machine inventory, health, capacity, destination, and access rules.

flowchart LR
    A[Department A<br/>waiting patients]
    B[Department B<br/>waiting patients]
    Booking[Shared booking desk<br/>Should this scan start now?]
    Wait[Remain in department<br/>No machine assignment yet]
    Approved[Scan approved]
    Assignment[MRI assignment desk<br/>Which machine can run it?]
    Registrar[Equipment registrar<br/>machine identity and health]

    subgraph machines[Hospital MRI area]
        M0[MRI 0]
        M1[MRI 1]
        M2[MRI 2]
        M3[MRI 3]
    end

    A --> Booking
    B --> Booking
    Booking -->|Not admitted| Wait
    Booking -->|Admitted| Approved --> Assignment
    Registrar --> Assignment
    Assignment --> M0
    Assignment --> M1
    Assignment --> M2
    Assignment --> M3

Loading

The project builds the GPU-platform equivalent of this hospital system:

Hospital concept	GPU platform concept
MRI machine	GPU device
Equipment registrar	kubelet device plugin
Registered versus assignable machines	Node capacity versus allocatable
Patient requesting one scan	Pod requesting one GPU
MRI assignment desk	Kubernetes scheduler and kubelet
Requested imaging wing	nodeSelector
Restricted-wing access pass	toleration for a node taint
Department waiting list	Kueue LocalQueue
Shared funding plan	Kueue ClusterQueue quota
Booking record	Kueue Workload
Approved booking	Kueue admission and quota reservation

The analogy is deliberately limited: it explains control-plane decisions, not medical safety or physical GPU execution.

Beginner learning path

Begin with the experiment guide, then follow these two topics in order.

Your first hands-on result

After installing the prerequisites listed below, reproduce finite capacity:

make cluster-up
make simulated-gpu-up
make capacity-exhaustion-run
make capacity-exhaustion-status

Predict the result before checking status: five patients request four MRI machines. The expected Kubernetes evidence is four Running Pods and one Pending Pod with an Insufficient simulator.accelerator-fleet.io/gpu event.

Clean up when finished:

make capacity-exhaustion-cleanup
make simulated-gpu-down
make cluster-down

Topic 1: Discover and assign GPU devices

First read the hospital recap for MRI inventory and assignment. Then work through:

1. Capacity exhaustion: why four devices cannot satisfy five whole-device requests.
2. Capacity loss and recovery: why reduced capacity affects future placement without undoing existing bindings.
3. Device-plugin allocation: how kubelet learns device identity and chooses a concrete device.
4. Device health recovery: why health changes assignable supply and how waiting work recovers.
5. Placement controls: why free capacity is insufficient when selector or toleration checks fail.

By the end, you should be able to explain how a Pod travels from a GPU request to a node binding and a concrete device allocation.

Topic 2: Share GPU access between teams

Next read the hospital recap for departments sharing MRI reservations. Then work through:

1. Admission versus capacity: why a Job can wait without a Pod even when GPUs are physically free.
2. Borrowing and reclaim: how one team borrows idle quota and how its owner gets it back.
3. Priority-aware reclaim: when policy permits interrupting borrowed work.
4. Admission fair sharing: why a newer request from a less-served team may receive the next free slot.

By the end, you should be able to distinguish a Kueue-pending Workload from a scheduler-Pending Pod and explain quota, borrowing, preemption, and fairness.

Skills demonstrated

• GPU capacity accounting & scheduler enforcement — driving a real Kubernetes scheduler to enforce finite accelerator capacity, including exhaustion and dynamic capacity loss/recovery. (01, 02)
• Kubelet device-plugin protocol — a Go implementation of the kubelet device-plugin gRPC API over a Unix socket: registration, ListAndWatch, and Allocate. (03)
• Failure injection & health-driven recovery — an HTTP control surface that flips simulated device health and observes kubelet/scheduler reactions in real time. (04)
• Placement policy — node selectors and taints/tolerations as independent scheduling filters, distinct from raw capacity. (05)
• Workload admission and GPU quota — Kueue ResourceFlavor, ClusterQueue, LocalQueue, and Workload behavior before Kubernetes creates and schedules Pods, including cohort borrowing and reclaim-driven preemption. (06, 07, 08, 09)
• Production-minded container hardening — scratch runtime image, pinned builder digest, read-only filesystem, dropped Linux capabilities, no service-account token, minimal host mounts.
• Reproducible infrastructure as code — every experiment is a scripted, repeatable procedure with unit, lifecycle, cancellation, socket-safety, HTTP-contract, and race tests, not a one-off manual run.

Experiment catalogue

Browse by technical area:

• Kubernetes device scheduling — capacity, allocation, health, and placement;
• Kueue multi-tenancy — admission, quota, borrowing, priority, and fairness.

Each group includes a hospital/MRI analogy that recaps its complete control flow.

Experiment	Mechanism	Main finding	Validation
01: Capacity exhaustion	Versioned Node status patch	Four one-GPU requests fit; the fifth remained Pending	Control-plane validated
02: Capacity loss and recovery	Versioned Node status patches	Lower capacity did not evict bound Pods; expansion unblocked pending work	Control-plane validated
03: Device-plugin allocation	Real kubelet device-plugin API	Kubelet selected sim-gpu-1 and Allocate injected the ID	Control-plane validated
04: Device health recovery	HTTP failure injection plus ListAndWatch	Health changed allocatable, not capacity; existing allocation survived	Control-plane validated
05: GPU node placement controls	Node selector plus NoSchedule taint	Capacity remained available while selector and toleration failures independently blocked placement	Control-plane validated
06: Kueue admission versus GPU capacity	Shared two-GPU ClusterQueue in front of four-GPU node supply	Quota held team B before Pod creation; releasing team A admitted the existing Workload	Control-plane validated
07: Kueue borrowing and reclaim	Two ClusterQueues in one cohort with bounded borrowing and reclaimWithinCohort: Any	Team A borrowed one GPU; team B reclaimed it by preempting one Workload; the same Workload was later readmitted	Control-plane validated
08: Kueue priority-aware reclaim	WorkloadPriorityClass plus reclaimWithinCohort: LowerPriority	Equal priority protected borrowed work; promotion from 100 to 1000 enabled one auditable preemption	Control-plane validated
09: Kueue admission fair sharing	Usage-based fair admission across two equal-weight LocalQueues	A newer Workload from the less-served team was admitted before three older Workloads from the higher-usage team	Control-plane validated

Project status

Area	Evidence	Status
Kubernetes device scheduling	Experiments 01–05	Control-plane validated
Kueue multi-tenancy	Experiments 06–09	Control-plane validated
HAMi scheduling and sharing	Planned next	Not yet validated
CUDA and physical GPU behavior	Deferred hardware experiments	Not yet validated

The completed results were produced by a real scheduler, kubelet, device-plugin API, or Kueue controller while accelerator hardware was simulated. Planning progress is tracked separately in PLAN.md.

What is implemented

• A single-node kind cluster with repeatable lifecycle commands.
• Kubernetes extended-resource experiments using simulator.accelerator-fleet.io/gpu.
• A Go kubelet device plugin that implements registration, ListAndWatch, and Allocate for four simulated devices named sim-gpu-0 through sim-gpu-3.
• A concurrency-safe inventory with idempotent Healthy and Unhealthy transitions.
• A simulator-only HTTP control API for failure injection.
• Nine documented experiments with setup, observation, evidence, cleanup, and limitations recorded as infrastructure as code.
• A checksum-pinned Kueue v0.19.0 controller with repeatable installation and teardown automation.

Key findings

• Node allocatable is total schedulable supply, not a live "free GPU" counter. The scheduler subtracts requests from already-bound Pods when testing a new Pod.
• Register associates a qualified resource name with a plugin endpoint; ListAndWatch supplies device identity and health; kubelet publishes Node status.
• Kubelet chooses the concrete device ID. The scheduler reasons about integer resource quantities, not sim-gpu-* identities.
• Marking an allocated device unhealthy reduced allocatable capacity from four to three but did not revoke the allocation or evict its running Pod.
• Restoring health returned allocatable capacity to four and caused the scheduler to reuse the existing Pending Pod without workload resubmission.
• Available GPU capacity is necessary but not sufficient for scheduling: selector and toleration filters can independently make the node ineligible.
• Kueue quota exhaustion can hold a Job in suspend=true before any Pod exists, even while Kubernetes node capacity remains available.
• Releasing a Kueue reservation does not change Node allocatable, and logical quota release can precede complete termination of the old Pod.
• A Kueue cohort let one tenant borrow idle nominal quota. With reclaimWithinCohort: Any, the lender's return preempted exactly the borrowed one-GPU Workload; Kueue later readmitted the same Workload and the Job created a replacement Pod.
• reclaimWithinCohort: LowerPriority protected equal-priority borrowed work. Updating the same pending Workload from priority 100 to 1000 authorized one cohort-reclamation preemption, without changing Pod priority.
• With equal LocalQueue weights and one quota slot, Admission Fair Sharing chose team B's newer Workload over team A's three older Workloads after team A had accumulated greater historical GPU use. No preemption was involved.

Roadmap

1. Evaluate HAMi scheduling and fragmentation with explicit simulation limits.
2. Build a local Slurm cluster with fake GPU GRES resources.
3. Compare equivalent Kubernetes and Slurm workload behavior.
4. Add fake DCGM-style telemetry, alerts, and incident replay.
5. Validate selected findings on an ephemeral real GPU only after local gates pass.

---

Technical detail

For readers who want to verify the mechanics behind the findings above.

Architecture

flowchart LR
    Operator[Make targets and experiment scripts]
    API[HTTP control API<br/>PUT device health]
    Inventory[Shared device inventory<br/>IDs and health]
    Plugin[Device-plugin gRPC server<br/>ListAndWatch and Allocate]
    Kubelet[kubelet device manager]
    Node[Node capacity and allocatable]
    Scheduler[Kubernetes scheduler]
    Pods[GPU-requesting Pods]

    Operator -->|kubectl port-forward + PUT| API
    API -->|SetHealth| Inventory
    Inventory -->|snapshot notification| Plugin
    Plugin <-->|Unix socket| Kubelet
    Kubelet -->|publishes| Node
    Node --> Scheduler
    Scheduler -->|binds or leaves Pending| Pods
    Kubelet -->|selected ID via Allocate| Pods

Loading

One *device.Inventory is shared by the HTTP control server and the device-plugin gRPC server. A real health change closes the current inventory notification channel. ListAndWatch wakes, sends a complete new snapshot to kubelet, and waits on the next notification. Reapplying the current health succeeds without publishing unnecessary state.

The plugin never patches Node status. Kubelet derives capacity and allocatable resources from the registered inventory and publishes them to the Kubernetes API.

Prerequisites

• macOS on Apple Silicon or another Docker-capable development machine;
• Docker Desktop, OrbStack, Colima, or an equivalent runtime;
• kind, kubectl, curl, make, and Go 1.26 for local development.

Reproduce the device-health experiment

Create the cluster and deploy the plugin:

make cluster-up
make device-plugin-up

Start with three Pods consuming three of four simulated GPUs:

make device-health-run

Mark sim-gpu-0 unhealthy and request a fourth Pod:

make device-health-degrade

Expected degraded state:

capacity=4
allocatable=3
Deployment=3/4 Ready
3 Running, 1 Pending
FailedScheduling: Insufficient simulator.accelerator-fleet.io/gpu

Restore the device and observe the existing Pending Pod schedule:

make device-health-recover

Inspect Node accounting, Pod allocations, plugin logs, and scheduler events:

make device-health-status

Clean up the workload and leave the plugin healthy:

make device-health-cleanup
make device-plugin-down
make cluster-down

Simulator control API

The API is an internal failure-injection surface, not part of Kubernetes' device-plugin protocol. It binds to 127.0.0.1:8080 inside the plugin Pod and is accessed through kubectl port-forward; no Service exposes it.

PUT /devices/{id}/health
Content-Type: application/json

{"health":"Unhealthy"}

Accepted health values are exactly Healthy and Unhealthy.

Response	Meaning
204	State accepted, including an idempotent no-op
400	Malformed JSON or invalid health value
404	Unknown device ID
405	Unsupported HTTP method

Use the recorded wrapper instead of a manual call:

make device-plugin-health DEVICE_ID=sim-gpu-0 HEALTH=Unhealthy

Repository map

components/simulated-device-plugin/          Shared Go device-plugin component
  cmd/fake-device-plugin/                    Process configuration and orchestration
  internal/device/                           Inventory, health, and notifications
  internal/plugin/                           Kubelet gRPC lifecycle and RPCs
  internal/control/                          HTTP health-control API and lifecycle
kubernetes/kind/                             Local cluster configuration
kubernetes/fake-gpu/                         Device-plugin deployment manifests
experiments/
  kubernetes-device-scheduling/              Capacity, allocation, health, placement
  kueue-multitenancy/                        Admission, quota, priority, fairness
scripts/                                     Lifecycle and experiment orchestration
Makefile                                     Stable high-level operator interface

Validation boundaries

What this project validates locally:

• Kubernetes scheduling and extended-resource accounting;
• kubelet device-plugin registration, inventory, health, and allocation calls;
• capacity exhaustion, Pending workloads, health loss, and recovery;
• label, selector, taint, and toleration placement behavior;
• controller, event, logging, automation, and cleanup behavior.

What it does not validate without physical GPUs:

• CUDA execution or compute performance;
• physical memory isolation or sharing enforcement;
• NVML or real DCGM hardware collection;
• thermal throttling, XID faults, GPU reset, or hardware recovery;
• NCCL, NVLink, GPUDirect RDMA, or physical topology.

Results use these labels:

• Simulated: generated device, health, or telemetry behavior.
• Control-plane validated: observed through a real scheduler, kubelet, API, or controller while accelerator hardware was simulated.
• Hardware validated: observed later on documented physical GPU infrastructure.
• Not yet validated: planned without acceptable evidence yet.

Documentation

• Learning and delivery plan
• Experiment guide
• Kubernetes device-scheduling recap
• Kueue multi-tenancy recap
• Experiment 01: capacity exhaustion
• Experiment 02: capacity loss and recovery
• Experiment 03: device-plugin allocation
• Experiment 04: device health loss and recovery
• Experiment 05: GPU node placement controls
• Experiment 06: Kueue admission versus GPU capacity
• Experiment 07: Kueue borrowing and reclaim
• Experiment 08: Kueue priority-aware reclaim
• Experiment 09: Kueue admission fair sharing

Licensed under the MIT License.