[thxCode]

What type of PR is this?

/kind enhancement
/kind api-change
/kind documentation
/area worker

What this PR does / why we need it:

ModelDeploymentScaler previously required every non-encode role using TokenThroughput or Predictive to supply explicit manual token capacities (tokensPerSecondPerReplica, prefillTokensPerSecondPerReplica, decodeTokensPerSecondPerReplica), or rely on cold-cluster benchmarking before enabling autoscaling. This PR adds offline capacity-prior catalog resolution, a typed ModelDeploymentScalerCapacityPriorCatalog schema, a deterministic controller simulation harness, and H200 hardware qualification benchmarks:

1. Typed Capacity Prior API & Catalog Schema (api/worker/v1alpha1/model_deployment_scaler.go, pkg/worker/scaling/priors/, gen/priors/) — adds spec.algorithm.capacityPrior (catalogRef, headroom defaulting to "0.75") on ModelDeploymentScaler, plus the ModelDeploymentScalerCapacityPriorCatalog (apiVersion: worker.gpustack.ai/v1alpha1, kind: ModelDeploymentScalerCapacityPriorCatalog) non-CRD runtime.Object schema supporting arbitrary p<N> latency quantile keys in (0, 100] (p25, p50, p70, p90, p95, p99, etc.), embedded pkg/worker/scaling/priors/priors.yaml (qualified from SemiAnalysis InferenceX and local H200 sweeps), and make generate priors (gen/priors) for offline import, merge, and retirement.
2. Fail-closed Prior Resolution & Identity Inference (pkg/worker/controllers/worker/model_deployment_scaler_priors.go, pkg/worker/webhooks/worker/model_deployment_scaler.go) — resolves role capacity priors from the embedded catalog or a same-namespace ConfigMap (catalogRef), matching canonical (role, model, quantization, engine, hardware, tp, dp, ep) identity inferred from ModelArtifact (spec.source.huggingFace.repo or spec.source.modelScope.repo via spec.model.artifactRef), role engine flags (--quantization, -tp, -dp, --enable-expert-parallel), and InstanceType / node hardware (nvidia-h100, nvidia-h200 140Gi/141Gi, nvidia-b200, amd-mi300x, amd-mi325x, amd-mi355x). Evaluates the highest-throughput operating point satisfying configured TTFT/ITL latency targets with exact or two-bracket linear quantile interpolation, scales by headroom, preserves explicit capacity overrides per field, and surfaces CapacityPriorResolved (CatalogResolved, ConfigMapNotFound, CatalogInvalid, RoleIdentityUnresolved, ProfileNotFound, NoPointWithinLatencyTarget).
3. Deterministic Local Controller Simulation & Frozen Capacity Estimator Evaluation (pkg/worker/controllers/worker/model_deployment_scaler_sim_test.go, pkg/worker/scaling/priors/estimator_eval_test.go) — exercises the real sampler, algorithms (Proportional, TokenThroughput, Predictive), and planner across 9 synthetic traffic scenarios (idle, step, ramp, burst-fault-reset, holdout-cache-shift, noise, quota-ceiling, quota-shortfall, length-shift), and evaluates five frozen capacity estimation candidates on disjoint holdout splits.
4. Reproducible Benchmark Harness, Documentation, and Skills (docs/modules/model-deployment/scaling.md, scaling-benchmark.md, scaling-simulation.md, docs/walkthroughs/model-deployment/scaling.md, .agents/skills/gpustack-operator-priors/, .agents/skills/gpustack-operator-benchmark/) — adds H200 4-node hardware qualification results (Suites A–E), Mermaid xychart-beta trend charts with explicit color legends, algorithm selection and tuning guidance, the scaling-simulation.md reference, and the gpustack-operator-priors maintenance skill.

Which issue(s) this PR links to:

Addresses #756

Special notes for your reviewer:

• Spec and verification record: specs/2026-10-26-inferencex-model-deployment-scaling.md.
• ModelDeploymentScalerCapacityPriorCatalog is annotated with +protobuf=false and +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object so it generates deepcopy and OpenAPI definitions without producing a separate CRD or protobuf type.

Does this PR introduce a user-facing change?

Add `spec.algorithm.capacityPrior` (`catalogRef`, `headroom`) and `ModelDeploymentScalerCapacityPriorCatalog` (`worker.gpustack.ai/v1alpha1`) to `ModelDeploymentScaler` for offline capacity-prior resolution from embedded InferenceX/H200 catalogs or a same-namespace `ConfigMap`, with arbitrary `p<N>` latency quantile interpolation and the `CapacityPriorResolved` status condition.

[gpustack-code-review]

🔍 OpenCodeReview found 1 issue(s) in this PR.

• ✅ Successfully posted inline: 0 comment(s)
• 📋 Routed to summary by policy: 1 comment(s)

---

📄 pkg/worker/webhooks/worker/model_deployment_scaler.go (L1516-L1518)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category test)

Reminder (not a claim — these files are excluded from this review): this is a behavior-tightening bug fix in admission validation. The same PR must ship a regression test covering the newly rejected cases (e.g. server-role scaler with ITLTargetMilliseconds set but TTFTTargetMilliseconds nil on an unresolved target, and the mirrored prefill/decode cases), and the regenerated zz_generated.webhooks.go output from make generate must be included if this hook signature/output contract appears there.

Reviewed b724758..9536857 only; earlier commits in this PR were reviewed in a previous run.

[thxCode]

Review Round 1 Summary

Inline Review Threads (10)

Thread / Comment	Location	Outcome
4396639535	.agents/skills/gpustack-operator-benchmark/scripts/autoscaling_qualification.py:740-741	Changed (b724758d) — Redirected stderr in run_load_phase to Path(out_dir) / f"{phase}.log" and read its tail on failure so verbose warnings cannot block an undrained PIPE during process.poll().
4396639551	pkg/worker/controllers/worker/model_deployment_scaler_prior_identity.go:877-884	Changed (0820c1e8) — Extended modelDeploymentScalerRedactEnvValues to also redact raw substring occurrences for secret values with len(value) >= 4 (excluding substrings of [redacted]), with a regression case in TestModelDeploymentScalerRedactEnvValues.
4396639554	pkg/worker/controllers/worker/model_deployment_scaler_prior_lifecycle.go:34-39	Kept as-is (thread left open) — StabilizationHistory records replica counts computed under EffectiveTokensPerSecond (headroom * seed); retaining them across seed/SLO/headroom changes would hold the role to replica counts computed under a stale denominator (pinned by TestModelDeploymentScalerCapacityPriorFingerprint and TestModelDeploymentScalerCapacityPriorPrewriteRefresh).
4396639557	pkg/worker/controllers/worker/model_deployment_scaler_priors.go:128-132	Changed (e1b7ae14, 0820c1e8) — Cached priors.LoadEmbedded() with sync.OnceValues and added a content-addressed SHA-256 cache (parseCachedConfigMapCatalog, capped at 32 entries) for ConfigMap catalog parsing.
4396639562	pkg/worker/controllers/worker/model_deployment_scaler_priors.go:250	Changed (0820c1e8) — Checked resolution.GPUsPerReplica > math.MaxInt32 || resolution.MeasuredReplicas > math.MaxInt32 on the int fields before casting to int32.
4396639567	pkg/worker/scaling/priors/catalog_yaml.go:263-264	Changed (b724758d) — Documented in toProfile why flat ModelDeploymentScalerCapacityPriorCatalog entries set CacheBehavior: CacheBehaviorAllMisses (flat catalogs only carry profiles qualified under all-miss prompt token accounting, required by ValidateCatalog).
4396639577	pkg/worker/scaling/priors/catalog_yaml.go:281	Changed (b724758d) — Added Note: "highest concurrency point of the measured sweep" on &Saturation{Measured: true} and documented the conservative upper-bound sweep cap contract.
4396639581	pkg/worker/webhooks/worker/model_deployment_scaler.go:235-237	Changed (0820c1e8, b724758d) — Added validateModelDeploymentScalerUnresolvedTargetSLOs when md == nil so creating a scaler before its target ModelDeployment exists still refuses capacityPrior configurations with unstated capacities and missing explicit latency targets.
4396639585	pkg/worker/webhooks/worker/model_deployment_scaler.go:238-242	Kept as-is (thread left open) — Propagating non-NotFound APIReader errors from Default is intentional fail-closed admission behavior so transient apiserver read errors do not mutate the spec with unwanted default latency targets (pinned by TestModelDeploymentScalerDefault_CapacityPrior).
4396639590	pkg/worker/webhooks/worker/model_deployment_scaler.go:1438-1440	Kept as-is (thread left open) — Returning field.InternalError on non-NotFound APIReader errors in ValidateCreate/ValidateUpdate while softening typed *priors.RefusalError into warnings on update is intentional fail-closed validation behavior (pinned by TestModelDeploymentScalerValidate_CapacityPrior).

Summary-Routed Findings (IC_kwDOSVaals6w-Rb8)

Item	Location	Outcome
1	pkg/worker/scaling/priors/validate.go:180-197	Changed (b724758d) — Sorted parallel topology keys before iterating profile.Identity.Parallel so validation error ordering is deterministic.
2	pkg/worker/scaling/sim/plant.go:362-368	Changed (a9896dfd) — Cloned values via slices.Clone before sorting in Quantile, with a regression assertion in TestQuantile.
3	pkg/worker/webhooks/worker/model_deployment_scaler.go:238-242	Changed (0820c1e8) — Read the target ModelDeployment once in validateModelDeploymentScaler via readTargetModelDeployment and passed md into shouldSuppressDefaultSLOs and ResolveModelDeploymentScalerCapacityPriorsWithTarget.
4	.agents/skills/gpustack-operator-benchmark/scripts/autoscaling_qualification.py:1109-1110	Changed (b724758d) — Updated the run_benchmark_suite docstring to accurately describe the serial execution of TokenThroughput and Predictive on the shared 4-GPU node.
5	.agents/skills/gpustack-operator-benchmark/scripts/autoscaling_report.py:482	Changed (819018d6) — Added "reason": None to the non-null branch of metric() so the schema matches the null branch.
6	pkg/nodefeature/unit_resources_preset.go:191-197	Changed (e1b7ae14) — Added isKnownVendorToken so resolveAcceleratorFamily returns empty immediately when tokens[0] is a known manufacturer (nvidia, amd, hygon, ascend, cambricon, metax, iluvatar, mthreads, t-head) and resolveVendorTokens(tokens[1:]) fails, with a regression case in TestResolveAcceleratorFamily.
7	pkg/worker/controllers/worker/model_deployment_scaler_priors.go:374-377	Changed (a9896dfd, e1b7ae14, 0820c1e8, b724758d) — Added regression unit tests covering Quantile non-mutation, resolveAcceleratorFamily unknown vendor models, env secret substring redaction, and unresolved-target admission validation.

[thxCode]

Review Round 2 Summary

Inline Review Threads (6)

Thread / Comment	Location	Outcome
4396769532	docs/walkthroughs/model-deployment/scaling.md:270-271	Changed (9536857c) — Updated the walkthrough to reference worker.gpustack.ai/v1alpha1 ModelDeploymentScalerCapacityPriorCatalog.
4396769553	pkg/worker/controllers/worker/model_deployment_scaler_prior_lifecycle.go:19-24	Partially changed (a1d44b45, thread left open) — Eliminated the duplicate ResolveModelDeploymentScalerCapacityPriors invocation during binding acquisition via planWithPriors. Cross-tick memoization keyed only by scaler/target Generation is intentionally avoided because external catalog ConfigMap / ModelArtifact / InstanceType changes do not bump ModelDeployment generation (pinned by TestModelDeploymentScalerCapacityPriorResetsRestoredState).
4396769559	pkg/worker/controllers/worker/model_deployment_scaler.go:57	Changed (a1d44b45) — Added modelDeploymentScalerCatalogPredicate() to the ConfigMap watch so empty ConfigMap creates and metadata/resourceVersion-only updates are filtered before mapCapacityPriorCatalog, with unit test assertions in TestModelDeploymentScalerCapacityPriorCatalogWatch.
4396769563	pkg/worker/scaling/priors/catalog_yaml.go:286-289	Kept as-is (thread left open) — Flat ModelDeploymentScalerCapacityPriorCatalog documents do not expose a per-point saturation field; capping at the highest measured concurrency point when all measured points satisfy a relaxed latency target is an intentional conservative bound pinned by TestFlatCatalogCapsConservativelyAtHighestConcurrency.
4396769567	pkg/worker/scaling/priors/refusal.go:123-128	Changed (7f291781) — Changed the model.name check in identityMismatch from ModelIdentityMatches to ModelNameMatches(want.Model.Name, got.Model.Name) so unstated deployment quantizations are accepted and revision/tokenizer mismatches report their own field names (TestResolveAcceptsUnstatedDeploymentQuantization).
4396769580	pkg/worker/webhooks/worker/model_deployment_scaler.go:1516-1523	Changed (a1d44b45) — Simplified validateModelDeploymentScalerUnresolvedTargetSLOs to check needsTTFT := needsServer || needsPrefill and needsITL := needsServer || needsDecode independently, with a regression case in TestModelDeploymentScalerValidate_CapacityPrior.

Summary-Routed Findings (IC_kwDOSVaals8AAAABgQwVGg)

Item	Location	Outcome
1	.agents/skills/gpustack-operator-benchmark/scripts/autoscaling_qualification.py:1195-1196	Changed (14522326) — Truncated timeline.jsonl (timeline_path.write_text("")) at the start of each simulation case before appending samples.
2	pkg/worker/scaling/priors/validate.go:233-235	Changed (7f291781) — Added "load axis is incoherent: " context prefix when wrapping loadAxisIncoherence(profile).
3	pkg/worker/scaling/sim/traffic.go:49-55	Changed (a14c1897) — Cloned and sorted config.Phases by UntilSeconds in NewTraffic, with a regression assertion in TestPlantPartialServiceQuantum.
4	pkg/worker/controllers/worker/model_deployment_scaler_priors.go:149-153	Changed (a1d44b45) — Evicted a single entry when configMapCatalogCache reaches maxCachedConfigMapCatalogs instead of clearing the entire map.
5	api/worker/v1alpha1/model_deployment_scaler.go:694-703	Verified — Generated deepcopy, protobuf, and CRD manifests are up to date (API/verify-generated passes in CI).
6	pkg/worker/webhooks/worker/model_deployment_scaler.go:1452-1455	Changed (a1d44b45) — Added regression coverage in TestModelDeploymentScalerValidate_CapacityPrior.
7	pkg/worker/scaling/sim/plant.go:363-368	Changed (a14c1897) — Clamped the nearest-rank index in Quantile to [0, len(sorted)-1] so q <= 0 and q > 1 cannot panic, with regression assertions in TestPlantPartialServiceQuantum.

[thxCode]

Review Round 3 Summary (ocr-summary-run:40632320724-1)

• Inline Review Threads: 0 new threads (upstream review converged).
• Summary-Routed Findings (1):
  • pkg/worker/webhooks/worker/model_deployment_scaler.go:1516-1518 (test-exclusion reminder) — Already covered (a1d44b45) by TestModelDeploymentScalerValidate_CapacityPrior in pkg/worker/webhooks/worker/model_deployment_scaler_prior_test.go, and no webhook marker or signature changed (API/verify-generated passes in CI).