Skip to content

PromQL quantile(φ, …) emits one-arg ES|QL PERCENTILE() — compiles & lints clean, fails at runtime with 400 #213

Description

@shmsr

Summary

quantile(φ, expr) (PromQL aggregation operator) translates to a one-argument ES|QL PERCENTILE(...), dropping the φ percentile. ES|QL PERCENTILE requires two arguments (PERCENTILE(field, percentile)), so the panel compiles and lints clean but fails at query time with HTTP 400. This is a runtime-only defect — kb-dashboard-cli compile / kb-dashboard-lint do not catch it; only executing the emitted ES|QL against Elasticsearch does.

Found by executing every emitted panel query against a live Serverless cluster (the new live-oracle path from #211 / PR #212). 3 instances surfaced across panel types, all one root cause.

Reproduce

Source PromQL (e.g. on a timeseries/gauge/stat panel):

quantile(0.9, rate(http_requests_total[5m]))

Emitted ES|QL:

TS metrics-*
| WHERE http_requests_total IS NOT NULL
| STATS http_requests_total = PERCENTILE(RATE(http_requests_total, 5m)) BY time_bucket = TBUCKET(5 minute), service, status, method
| SORT time_bucket ASC

Elasticsearch POST /_query response:

{"error":{"type":"parsing_exception",
  "reason":"line 3:31: error building [percentile]: expects exactly two arguments",
  "caused_by":{"type":"ql_illegal_argument_exception","reason":"expects exactly two arguments"}},
 "status":400}

Root cause

The φ value is parsed and stored but never emitted:

  • observability_migration/adapters/source/grafana/promql.py:1921 and :1794 set frag.extra["quantile_phi"].
  • OUTER_AGG_MAP maps quantile -> PERCENTILE (promql.py:347).
  • The emission sites apply the outer agg as a single-arg wrapper and never append the percentile:
    • promql.py:3079 (simple_agg): stats_expr = f"{outer}({inner_expr})"
    • promql.py:3128 (range_agg): stats_expr = f"{esql_outer}({esql_inner}({metric_field}, {frag.range_window}))"
    • (_build_stats_call at promql.py:2433-2436 has the same shape for any path that routes through it.)

The code comment at promql.py:1907 already documents the intended form — PERCENTILE(expr, phi*100) — so the captured quantile_phi simply isn't threaded into emission. Note PromQL φ is in [0,1]; ES|QL wants [0,100], so the second arg must be φ*100 (0.9 -> 90, 0.95 -> 95).

Suggested fix

Centralize outer-agg wrapping so PERCENTILE always receives the percentile, e.g.:

def _wrap_outer_agg(esql_outer: str, inner: str, frag) -> str:
    if esql_outer == "PERCENTILE":
        phi = frag.extra.get("quantile_phi")
        if phi is None:
            return ""  # caller marks not_feasible (parse already guards invalid phi)
        return f"PERCENTILE({inner}, {_format_scalar_value(float(phi) * 100)})"
    return f"{esql_outer}({inner})"

Apply it at the simple_agg (3079) and range_agg (3128) emission sites (and any path through _build_stats_call). Add snapshot/translation coverage asserting quantile(0.9, …) -> PERCENTILE(..., 90).

Why existing gates missed it

  • kb-dashboard-lint esql-sql-syntax is heuristic and accepts a one-arg PERCENTILE(...).
  • kb-dashboard-cli compile produces valid NDJSON regardless.
  • Only live POST /_query execution rejects it. This is the value of the live-oracle / live-query layer in PR feat(verifier): dashboard fidelity verification framework #212.

Scope

  • 3 occurrences in the synthetic hard-panel matrix; affects any dashboard using quantile(φ, …).
  • histogram_quantile(...) uses a different path (PERCENTILE_OVER_TIME) and is not affected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions