Repository navigation
fix(worker): let the ConfigMap owner watch enqueue elastic state records - #722
Conversation
The ModelDeployment controller watches ConfigMaps through Owns(), but the ownership predicate only accepted systemmeta resource notes. Elastic operation records carry gpustack.ai/elastic-operation=true and observation records carried no marker at all, so after an operator restart, persisted elastic state changes could not enqueue the owning ModelDeployment and a resize in progress froze (#721). Label observation records with gpustack.ai/elastic-observation=true and accept both elastic state labels in the predicate. Signed-off-by: thxCode <thxcode0824@gmail.com>
|
🔍 OpenCodeReview found 1 issue(s) in this PR.
📄
|
What type of PR is this?
/kind bug
/area worker
What this PR does / why we need it:
After an operator restart, a native elastic DP resize in progress froze: membership stayed pinned and a follow-up scale-down patch produced no terminations for 30+ minutes (#721, measured in the W09 8-GPU evidence).
Root cause: the controller watches ConfigMaps through
Owns(&core.ConfigMap{}), but the ownership predicatemodelDeploymentOwnedResourceonly acceptedsystemmetaresource notes. Elastic operation records carrygpustack.ai/elastic-operation=trueand no resource note; observation records carried only the owner reference. Neither passed the predicate, so persisted elastic state changes after a restart could not enqueue the owning ModelDeployment.Fix: label new observation records with
gpustack.ai/elastic-observation=trueand accept both elastic state labels in the predicate. No API or CRD shape changes.Regression coverage: the ownership predicate test now requires both elastic state objects to be accepted (RED on the unmodified predicate, GREEN with the fix; full
pkg/worker/controllers/workerpackage suite passes).Scope note: the same evidence window also logged intermittent embedded extension API handler timeouts. Those are temporally correlated but not proven to share this cause; this PR fixes the proven reconcile wake-up defect only.
Which issue(s) this PR links to:
Fixes #721
Special notes for your reviewer:
Does this PR introduce a user-facing change?