fix(dashboard): SGLang panels show No data when exporter negotiates escaping=underscores

[attack204]

Problem

Every panel in verl_tainer_v1_with_sglang_engine renders No data against a live SGLang rollout, even though Prometheus is scraping the SGLang targets successfully.

The dashboard queries metrics as sglang:e2e_request_latency_seconds_bucket, but the series are stored as sglang_e2e_request_latency_seconds_bucket.

Root cause

SGLang 0.5.16 answers /metrics with:

content-type: application/openmetrics-text; version=1.0.0; charset=utf-8; escaping=underscores

The payload itself uses colon-prefixed names (sglang:cache_hit_rate, ...), but the negotiated escaping=underscores scheme only permits [a-zA-Z0-9_]. Colons are outside that set, so Prometheus escapes them to underscores before the sample is stored.

This is observable as a mismatch between what Prometheus parsed and what it stored, on the same target:

# parsed names keep the colon
$ curl -sG localhost:9090/api/v1/targets/metadata \
    --data-urlencode 'match_target={job="sglang"}' | jq -r '.data[].metric' | grep e2e
sglang:e2e_request_latency_seconds

# stored names do not — 0 of 194 metric names contain a colon
$ curl -s localhost:9090/api/v1/label/__name__/values | jq '[.data[] | select(contains(":"))] | length'
0
$ curl -s localhost:9090/api/v1/label/__name__/values | jq -r '.data[]' | grep e2e
sglang_e2e_request_latency_seconds_bucket

Environment: SGLang 0.5.16, Prometheus 2.54.1 (the version installed by rl-insight server install), rl-insight 0.3.0, verl with rollout.name=sglang.

Fix

Select through __name__ with a character class that accepts both spellings:

rate({__name__=~"sglang[:_]e2e_request_latency_seconds_bucket", model_name=~"$sglang_model_name"}[$__rate_interval])

I deliberately avoided rewriting sglang: to sglang_, because that would break deployments whose exporter or scrape negotiation preserves the colon form. The character class works in both.

The four label_values() template queries need the same change. Without them $sglang_model_name and $sglang_replica resolve to nothing and every panel stays empty even after the expressions are corrected.

Verification

Against a live 4-replica SGLang rollout, after the change:

• $sglang_model_name resolves to ["DeepSeek-V4-Flash-FP8"], $sglang_replica to ["0","1","2","3"]
• the latency ratio panel returns one series per replica
• histogram_quantile(0.95, ...) over the bucket selector returns a value

Only query strings changed; no panel layout, title, or datasource was touched. The file remains valid JSON.

• After fix