Skip to content

[Minor Improvements] Vllmruntime Autoscaling in Operator#918

Merged
aeon-x merged 3 commits into
vllm-project:mainfrom
aeon-x:improve_autoscaling_in_operator
Apr 21, 2026
Merged

[Minor Improvements] Vllmruntime Autoscaling in Operator#918
aeon-x merged 3 commits into
vllm-project:mainfrom
aeon-x:improve_autoscaling_in_operator

Conversation

@aeon-x

@aeon-x aeon-x commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Three improvements to VLLMRuntime KEDA autoscaling:

  1. Fix: resolve --served-model-name override in Prometheus queries — The keepalive trigger queries Prometheus with model="". Previously it always used ModelURL (e.g., meta-llama/Llama-3.1-8B-Instruct), ignoring --served-model-name in ExtraArgs. If a user overrides the served model name, the query returns no data and KEDA may incorrectly scale to zero while traffic is active.
  2. Widen keepalive rate window from [1m] to [2m] — A 1-minute window is tight; delayed Prometheus scrapes or brief metric gaps can return 0 and trigger a false scale-to-zero. A 2-minute window is more resilient.
  3. Add AvailableReplicas, UpdatedReplicas, UnavailableReplicas to VLLMRuntimeStatus — Exposes deployment replica status directly on the CR, so users can observe scaling progress via kubectl get vr without inspecting the underlying Deployment.

Changes:

  • operator/api/v1alpha1/vllmruntime_types.go — Add 3 status fields
  • operator/internal/controller/vllmruntime_controller.go — Served-model-name parsing, 2m keepalive window, populate new status fields
  • operator/api/v1alpha1/zz_generated.deepcopy.go — Regenerated
  • operator/config/crd/bases/production-stack.vllm.ai_vllmruntimes.yaml — Regenerated

Tests Done:

  • make generate manifests — regenerated without errors
  • make test — all VLLMRuntime and autoscaling tests pass (11/11)
  • Deploy with a VLLMRuntime CR using --served-model-name override and verify KEDA ScaledObject uses the correct model name in Prometheus queries
  • Verify kubectl get vr shows the new replica status fields

Signed-off-by: aeon-x <talexcao@gmail.com>
@aeon-x aeon-x changed the title init [Operator] Improve autoscaling Apr 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds AvailableReplicas, UpdatedReplicas, and UnavailableReplicas to the VLLMRuntime status and updates the controller to populate these from the Deployment. It also introduces logic to detect --served-model-name overrides in ExtraArgs and increases the Prometheus query interval to 2m. Feedback indicates that the argument parsing logic should be expanded to handle space-separated values to ensure the model name is correctly identified in all common Kubernetes configurations.

Comment thread operator/internal/controller/vllmruntime_controller.go
@aeon-x aeon-x changed the title [Operator] Improve autoscaling [Minor Improvements] Vllmruntime Autoscaling in Operator Apr 16, 2026
@aeon-x aeon-x marked this pull request as ready for review April 16, 2026 05:55

@ruizhang0101 ruizhang0101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aeon-x aeon-x enabled auto-merge (squash) April 20, 2026 23:55
@aeon-x aeon-x merged commit b66c80c into vllm-project:main Apr 21, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants