Summary
With more than one CubeProxy replica, the first request to a sandbox after it auto-pauses intermittently returns 504 (~3s) and fails to trigger auto-resume; the sandbox stays paused. The failure probability is ~(N-1)/N for N replicas.
Environment
- CubeSandbox version / commit:
master (verified on current HEAD; the relevant code paths are unchanged in release-0.5.x)
- Host OS and kernel version: N/A — this is a control-plane request-routing bug, independent of host/KVM
- KVM info (
modinfo kvm): N/A
- Deployment mode: cluster, CubeProxy with replicas ≥ 2 behind a load balancer / Ingress that spreads traffic across pod endpoints (not sandbox-sticky)
- Relevant component: CubeProxy (lua gate) + cube-lifecycle-manager (auto-pause/resume sidecar)
Steps to Reproduce
- Deploy CubeProxy with
replicas: 2 (each pod runs its own co-located cube-lifecycle-manager), fronted by a load balancer that round-robins across the pod endpoints.
- Create a sandbox with auto-pause + auto-resume enabled; start a listener inside it and note its proxied host.
- Leave it idle until it auto-pauses (confirm the sandbox reaches
paused).
- Send the first request to the sandbox host.
To make it deterministic, bypass the LB and send the same first request directly to each replica's dataplane listener:
- Request to the replica that consumed this sandbox's lifecycle events →
200, sandbox resumes.
- Request to any other replica →
504 after ~3s, and the sandbox stays paused (no resume attempted).
Expected Behavior
The first request after auto-pause transparently resumes the sandbox and returns 200, regardless of replica count or which replica serves the request.
Actual Behavior
With N CubeProxy replicas, the first post-pause request has a ~(N-1)/N chance of returning 504 after ~3s (nginx proxy_connect_timeout) and leaving the sandbox paused. Subsequent requests may succeed once some request happens to land on the owning replica and resumes it.
first request -> non-owning replica: HTTP 504 in 3.01s, sandbox still "paused"
first request -> owning replica: HTTP 200 in 0.10s, sandbox -> "running"
Root Cause
The auto-pause/resume state is per-replica and is fed by a competing-consumer partition of the lifecycle event stream, so only one replica ever learns about a given sandbox:
- CubeMaster publishes sandbox lifecycle
create/delete events to a Redis Stream (cube:v1:shared:sandbox:lifecycle:events) plus a meta HSet snapshot (…:lifecycle:meta).
- Every
cube-lifecycle-manager consumes that stream with a fixed shared consumer group "cube-proxy-sidecar" (cube-lifecycle-manager/internal/config/config.go:101) and a per-host consumer name. Redis Stream consumer-group semantics = competing consumers: each event is delivered to exactly one replica (cube-lifecycle-manager/internal/redisstream/stream.go:88, XReadGroup Streams=[key, ">"]).
- That single replica applies the event only to (a) its in-memory registry and (b) its co-located nginx admin dict over loopback (
127.0.0.1:8082). There is no cross-replica propagation, so the other replicas' cube_sandbox_state dicts never learn the sandbox exists.
- The lua gate reads the per-replica shared dict on the hot path and treats a dict MISS as "running → pass through" (
CubeProxy/lua/sandbox_state.lua:47, if not state or state == "running" then return end). So a first post-pause request routed to a non-owning replica bypasses the resume gate; nginx proxy_pass dials the vCPU-frozen guest, the SYN is black-holed, and proxy_connect_timeout 3s fires → 504. It also never fires /_sidecar_resume, so that request does not wake the sandbox.
- Even forcing a miss through
/_sidecar_resume on a non-owning replica does not help as-is: the resumer requires the sandbox in the local in-memory registry (cube-lifecycle-manager/internal/resumer/resumer.go, doResume → "sandbox not in registry"), which the non-owning replica also lacks.
The intermittency (~(N-1)/N) is purely a function of which replica the load balancer picks for the first request.
Proposed Fixes
Workaround (zero code): run CubeProxy with replicas: 1. Eliminates cross-replica divergence. Cost: no CubeProxy HA.
Real fix — fan out lifecycle events so every replica holds full state. Cross-replica pause/resume is already serialized by the shared Redis SETNX state lock (AcquireState on …:lifecycle:state:<id>), so having every replica consume every event is safe. Two ways:
- (a) per-replica unique consumer group — every replica receives all events. Downside: orphan consumer groups accumulate in Redis on pod churn and need periodic
XGROUP DESTROY cleanup.
- (b) groupless
XREAD (preferred) — each replica reads the whole stream from its own in-memory cursor; no consumer groups, nothing to clean up. This fits the existing design: the meta HSet is the authoritative snapshot (already loaded via HGETALL bootstrap at startup) and the stream is a delta feed, so the group's at-least-once/PEL/redelivery guarantees are unnecessary. (To avoid a startup gap, record the stream's current last-ID before the bootstrap HGETALL, then XREAD from that ID.)
Alternative (keeps competing consumption, serving-side authority): on a local dict MISS, have the gate consult the shared Redis state key instead of passing through, and have the resumer fall back to the shared meta HSet when the sandbox is absent from the local registry. More moving parts (a Redis client inside OpenResty, plus Redis credentials in the nginx config).
Additional Context
Related but distinct: #790 (auto-resume recovery after node failure) and #793 (autopause activity signals). This report is specifically about per-replica state visibility under a shared consumer group, which is orthogonal to those.
Happy to submit a PR for the groupless XREAD fix (option b) if the maintainers agree with the direction.
Summary
With more than one CubeProxy replica, the first request to a sandbox after it auto-pauses intermittently returns 504 (~3s) and fails to trigger auto-resume; the sandbox stays paused. The failure probability is ~
(N-1)/NforNreplicas.Environment
master(verified on current HEAD; the relevant code paths are unchanged inrelease-0.5.x)modinfo kvm): N/ASteps to Reproduce
replicas: 2(each pod runs its own co-locatedcube-lifecycle-manager), fronted by a load balancer that round-robins across the pod endpoints.paused).To make it deterministic, bypass the LB and send the same first request directly to each replica's dataplane listener:
200, sandbox resumes.504after ~3s, and the sandbox stays paused (no resume attempted).Expected Behavior
The first request after auto-pause transparently resumes the sandbox and returns
200, regardless of replica count or which replica serves the request.Actual Behavior
With
NCubeProxy replicas, the first post-pause request has a~(N-1)/Nchance of returning504after ~3s (nginxproxy_connect_timeout) and leaving the sandbox paused. Subsequent requests may succeed once some request happens to land on the owning replica and resumes it.Root Cause
The auto-pause/resume state is per-replica and is fed by a competing-consumer partition of the lifecycle event stream, so only one replica ever learns about a given sandbox:
create/deleteevents to a Redis Stream (cube:v1:shared:sandbox:lifecycle:events) plus a metaHSetsnapshot (…:lifecycle:meta).cube-lifecycle-managerconsumes that stream with a fixed shared consumer group"cube-proxy-sidecar"(cube-lifecycle-manager/internal/config/config.go:101) and a per-host consumer name. Redis Stream consumer-group semantics = competing consumers: each event is delivered to exactly one replica (cube-lifecycle-manager/internal/redisstream/stream.go:88,XReadGroup Streams=[key, ">"]).127.0.0.1:8082). There is no cross-replica propagation, so the other replicas'cube_sandbox_statedicts never learn the sandbox exists.CubeProxy/lua/sandbox_state.lua:47,if not state or state == "running" then return end). So a first post-pause request routed to a non-owning replica bypasses the resume gate; nginxproxy_passdials the vCPU-frozen guest, the SYN is black-holed, andproxy_connect_timeout 3sfires → 504. It also never fires/_sidecar_resume, so that request does not wake the sandbox./_sidecar_resumeon a non-owning replica does not help as-is: the resumer requires the sandbox in the local in-memory registry (cube-lifecycle-manager/internal/resumer/resumer.go,doResume→"sandbox not in registry"), which the non-owning replica also lacks.The intermittency (~
(N-1)/N) is purely a function of which replica the load balancer picks for the first request.Proposed Fixes
Workaround (zero code): run CubeProxy with
replicas: 1. Eliminates cross-replica divergence. Cost: no CubeProxy HA.Real fix — fan out lifecycle events so every replica holds full state. Cross-replica pause/resume is already serialized by the shared Redis
SETNXstate lock (AcquireStateon…:lifecycle:state:<id>), so having every replica consume every event is safe. Two ways:XGROUP DESTROYcleanup.XREAD(preferred) — each replica reads the whole stream from its own in-memory cursor; no consumer groups, nothing to clean up. This fits the existing design: the metaHSetis the authoritative snapshot (already loaded viaHGETALLbootstrap at startup) and the stream is a delta feed, so the group's at-least-once/PEL/redelivery guarantees are unnecessary. (To avoid a startup gap, record the stream's current last-ID before the bootstrapHGETALL, thenXREADfrom that ID.)Alternative (keeps competing consumption, serving-side authority): on a local dict MISS, have the gate consult the shared Redis state key instead of passing through, and have the resumer fall back to the shared meta
HSetwhen the sandbox is absent from the local registry. More moving parts (a Redis client inside OpenResty, plus Redis credentials in the nginx config).Additional Context
Related but distinct: #790 (auto-resume recovery after node failure) and #793 (autopause activity signals). This report is specifically about per-replica state visibility under a shared consumer group, which is orthogonal to those.
Happy to submit a PR for the groupless
XREADfix (option b) if the maintainers agree with the direction.