Stop stage sweeps on server congestion

[tlrmchlsmth]

Summary

• add an until_congested() Starlark helper for finite stage sweeps
• stop before the next stage when sustained queue depth + TTFT or KV usage + preemptions cross configured thresholds
• query aggregate deployments or both prefill and decode roles in disaggregated deployments
• add stage-completion control flow, validation, detailed logging, tests, and README documentation

How congestion is determined

Checks run after each measured stage. Threshold comparisons are inclusive. The first 20% and final 5% of a stage are excluded as ramp/boundary time; short stages use their full duration.

Signal	Calculation	Trigger
Sustained queueing	Sum waiting requests across pods per role, then temporal p50; server TTFT p99 over the same steady-state window	waiting_requests_p50 and ttft_p99 are both reached; these may come from different PD roles because they represent end-to-end latency
KV-cache pressure	Maximum KV usage in the steady-state window plus preemption counter increase over the full stage	kv_cache_usage and preemptions are both reached on the same role

Missing series evaluate as zero. Query failures are logged and fail open. The congested stage remains in the results; only later stages are skipped.

Starlark configuration

waiting_requests_p50 and ttft_p99 are required and deployment-specific. kv_cache_usage defaults to 0.95 (fraction) and preemptions defaults to 1 per stage.

Balanced example

scenario(
    stages = until_congested(
        [stage("2m", concurrency=c) for c in range(64, 2049, 64)],
        waiting_requests_p50=32,
        ttft_p99="2s",
        kv_cache_usage=0.95,
        preemptions=1,
    ),
)

Latency-sensitive example

stages = until_congested(
    [stage("2m", concurrency=c) for c in range(16, 257, 16)],
    waiting_requests_p50=8,
    ttft_p99="750ms",
    kv_cache_usage=0.90,
    preemptions=1,
)

Throughput-oriented example

stages = until_congested(
    [stage("5m", concurrency=c) for c in range(128, 4097, 128)],
    waiting_requests_p50=64,
    ttft_p99="5s",
    kv_cache_usage=0.98,
    preemptions=5,
)

Run with:

nyann-bench generate --config scenario.star \
  --prometheus-url http://prometheus:9090 \
  --deploy-name my-model-decode

Testing

• go test ./... -count=1