Skip to content

README.md

Latest commit

 

History

History
313 lines (251 loc) · 32.8 KB

README.md

File metadata and controls

313 lines (251 loc) · 32.8 KB
Raw

GPUStack Operator Documentation

Start with the project README to install the operator. Choose a reading path by task, or find a capability in the module map.

The capability guides are Heterogeneous Devices, RDMA Networking, Topology, KV Cache, Model Delivery, Model Deployment and GPU Instances.

Reading paths

For a first workload, follow Usage, then use Accelerator Requests to choose its resource keys. The Heterogeneous Devices Walkthrough records a run on a four-node cluster. Workloads that need RDMA also need the RDMA Operations.

For cluster operations, read Architecture and Installation Modes first. The settings, vendor prerequisites and preflight pages cover installation decisions.

Use the topology, model store, high availability and RDMA runbooks as those features are enabled. Hardware partitioning has separate runbooks for NVIDIA, T-Head and Hygon.

For complete examples, use the Walkthroughs entry point. For replica autoscaling, follow the Scaling Walkthrough and read the measured Scaling Benchmark.

Start an upgrade with the relevant page under migration.

For code changes, read Architecture, then use the module map to find the relevant guide. Internals covers startup constraints, and Development lists the build and validation commands. The gpustack-operator-docs skill routes documentation changes.

Agents should load gpustack-operator-overview, Architecture, and only the deep page needed for the task.

Module map

Each entry routes one module or shared task to its guides, owning specs, code entry points, related modules and skills. Links here are navigation, not proof: a spec records what was decided and measured when it was written, so what the current checkout implements needs the current code. A skill link names a procedure; the entry's other fields carry shared facts.

Heterogeneous Devices

RDMA Networking

Topology

KV Cache

Model Delivery

Model Deployment

GPU Instances

API Changes

Installation

Development

All pages

Getting Started

Page Description
Architecture See the operator’s four-stage path.
Vendor Prerequisites Prepare each manufacturer’s driver.

Walkthroughs

Page Description
Walkthroughs Choose a complete example by module.
Heterogeneous Devices Walkthrough Follow a recorded accelerator scheduling run.
RDMA Networking Walkthrough Request and verify RDMA endpoints and NUMA alignment.
Hierarchical Placement Walkthrough Place serving replicas across zones and physical racks.
Shared Cache Walkthrough Create a shared cache from backend to workload.
Model Prefetch Walkthrough Cache weights and verify captured model CR output.
Model Deployment Walkthroughs Choose a serving configuration and its hardware, model and InstanceType setup.
Prefill and Decode Walkthrough Deploy H100 prefill and decode roles with tested TCP transfer settings.
Encode, Prefill and Decode Walkthrough Deploy encode, prefill and decode roles and verify a multimodal request.
Model Deployment Scaling Walkthrough Configure Shadow and Active scaling, choose algorithms and observe replica retirement.
Elastic EP Walkthrough Expand an H100 collective from four to eight GPUs and verify effective width and inference.
External DP Walkthrough Deploy four fixed H100 ranks and verify captured endpoint and Router coverage.
Multi-Host Serving Walkthrough Scale models across multiple GPU nodes.

Heterogeneous Devices

Page Description
Heterogeneous Devices Choose an accelerator request.
Preflight Checks Check a node before installation.
Device Discovery Trace hardware detection and allocation.
Accelerator Requests Look up resource keys and valid requests.
Scheduling Chain Follow capacity from labels to queues.
Admission Understand the five request checks.
NVIDIA MIG Operations Enable and recover NVIDIA partitions.
Hygon MIG Operations Enable and recover Hygon partitions.
T-Head MIG Operations Enable and recover T-Head partitions.

RDMA Networking

Page Description
RDMA Networking Start with RDMA requests.
Network Topology See how links and devices are discovered.
RDMA Operations Request and check RDMA endpoints.

Topology

Page Description
Topology Start with placement by domain.
Topology-Aware Scheduling Follow topology data into scheduling.
Topology-Aware Scheduling Operations Enable and diagnose topology placement.

KV Cache

Page Description
KV Cache Start with a shared inference cache.
KV Cache Backend Understand the store and its capacity.
KV Cache Pools and Bindings Grant and limit cache use.
KV Cache Injection Reference Attach a Pod to a pool.
KV Cache Election and Failover Understand leader election and health.
KV Cache Local Disk Tier Configure local disk storage.
KV Cache on Disk-Heavy Nodes Size memory on disk-heavy nodes.

Model Delivery

Page Description
Model Delivery Choose how weights reach workloads.
Model Artifact Resolve and verify model weights.
Model Artifact API Read artifact and node cache status.
Model Image Source Package weights in an image.
Model Prefetch Warm weights before a workload starts.
Node Model Store Understand node cache state and collection.
Node-to-Node Sync Move cached weights between nodes.
Model Store Operations Operate the node model cache.

Model Deployment

Page Description
Model Deployment Start with managed model serving.
Model Deployment Configuration Configure serving roles and overrides.
Engine Compatibility Check engine releases, client and store requirements, and transport support.
Prefill and Decode Pair serving roles.
Prefill and Decode Benchmark Compare equal-GPU H100 aggregated and disaggregated serving across fixed shapes and mixed traffic.
Encode, Prefill and Decode Disaggregate visual encoding, prefill and decode roles.
Encode, Prefill and Decode Benchmark Compare EPD, prefill/decode and aggregated on measured latency and throughput.
Elastic EP Resize one serving instance's collective in place.
Elastic EP Benchmark Compare four- and eight-GPU H100 collectives and measure requests during live expansion.
External DP Run one fixed data-parallel group with an HTTP endpoint on every rank.
External DP Benchmark Compare External DP ranks with a monolithic wide tensor-parallel group.
Model Deployment Scaling Plan a deployment's replica count from the signals it publishes.
Model Deployment Scaling Simulation Compare the three scaler algorithms on identical synthetic traffic with measured adjustment dynamics.
Model Deployment Scaling Benchmark Compare H100 and H200 scaling configurations and capacity priors with measured latency, throughput and lifecycle limits.
Model Deployment Routing Choose a routing policy, confirm a Router serves.
Model Deployment Metrics Read serving metrics.
Model Deployment Status Diagnose deployment conditions, follow a retirement, read the scaler annotations.
Model Deployment Shutdown Understand replica retirement, draining, adoption and unadmitted removal.

GPU Instances

Page Description
GPU Instances Start an accelerator-backed workspace.

Operations

Page Description
Installation Modes Choose chart or image installation.
High Availability Operations Set replica counts for control-plane parts.
Migrating from v0.5.x Upgrade across the queue refactor.
Migrating to Bundled Subcharts Transfer chart ownership.
Upgrading to an Enforced Binding Dtype Check dtype before upgrading.
Migration Troubleshooting Recover a stuck upgrade.

Reference

Page Description
Command Reference Look up binary commands and flags.
Settings & Environment Variables Look up operator configuration.
Instance Type Unit Resources Reference Look up CPU and memory presets.
Instance Metrics Reference Read an Instance’s resource use.

Contribution

Page Description
Development Build, generate and lint the project.
Local Cluster Setup Install kind and provide a cluster for development tests.
Internals Review startup and naming constraints.

Conventions

Every page in this directory (this index excepted) carries:

  • a short introduction that explains the subject or task;
  • a ## Contents list mirroring its ## headings, in order;
  • a footer — **See also** for sideways links and **Next** → for the next page on the path.

Each label above matches its page's H1 exactly. The page table follows the site navigation order. Deep rationale is demoted into a > **Why** note, so the rule stays skimmable. Length is not the defect, verbosity is.

Adding or moving a page means updating its row above. Run make lint docs to check links, Contents, paragraph length, index labels, page coverage and paths named in skills. Documentation changes do not require Go tests. The former Go tests that read Markdown pages were removed; compare code-owned values with their reference pages when either changes.