[Blocked] Benchmark/migration perf: parallelize compile_all (dominant cost) and the two benchmark legs

[stefans-elastic]

Summary

The dashboard benchmark (and real bulk migrations) spend most of their wall-clock in the --compile step, which runs kb-dashboard-cli once per dashboard, serially. Profiling shows compile — not translation and not the co-occurrence probes — is the dominant cost, in both the Grafana and Datadog legs. There are two independent levers worth ~2–3× total speedup; both leave migration output unchanged.

Not urgent (filed post-investigation for July, after the June 29 release). Captured here with full data so it can be picked up cold.

Baseline (G:500 D:671, against the local lab)

Component	Time	% of total
Total	2060.8s (34.3 min)	100%
Grafana leg	1108.2s	53.8%
Datadog leg	952.5s	46.2%
└ co-occurrence probes (grafana)	43.2s	2.1%

• 1171 dashboards / 32,551 panels.
• Grafana per-dashboard translate: median 0.03s, p95 0.59s, p99 2.01s, max 3.17s → translation is negligible.
• The two legs run serially and additively (1108 + 953 ≈ 2061).
• (This run included one-time downloads of uncached dashboards; compare cached-to-cached when measuring fixes.)

Where the time goes (G:50 D:50 phase profile, timestamped stream)

Total 202.9s:

Phase	Time	% of total
Datadog leg	75.7s	37%
Grafana — compile (kb-dashboard-cli / uvx, per-dashboard)	68.8s	34%
Grafana — migrate/translate (50 dashboards)	57.6s	28%
Grafana — schema discovery	0.7s	<1%
└ of which co-occurrence probes	5.0s	2.5%

Per-dashboard translate is tiny; the grafana leg is dominated by compile. The Datadog leg also runs --compile, so it is compile-heavy too.

Root cause

compile_all compiles each YAML serially, one kb-dashboard-cli subprocess per dashboard:

# observability_migration/targets/kibana/compile.py:48
def compile_all(yaml_dir, compiled_dir):
    Path(compiled_dir).mkdir(parents=True, exist_ok=True)
    results = []
    for yaml_file in sorted(Path(yaml_dir).glob("*.yaml")):   # <-- serial
        out_dir = Path(compiled_dir) / yaml_file.stem
        out_dir.mkdir(parents=True, exist_ok=True)
        success, output = compile_yaml(yaml_file, out_dir)     # <-- 1 subprocess each
        results.append((yaml_file.name, success, output))
    return results

For G:500 D:671 that's 1171 sequential subprocess starts (each kb-dashboard-cli compile --input-file <one>). It's embarrassingly parallel — each file is independent and results are collected by name.

Proposed work (ranked)

Lever A — parallelize compile_all (engine, compile.py). Thread-pool the per-file compiles (bounded worker pool, e.g. min(os.cpu_count(), 8)), preserve deterministic result collection by name. Expected: compile 69s → ~10–15s on a multi-core box (~25–30% of total). Low risk (independent subprocesses; identical output), and it speeds up real migrations too, not just the benchmark. Recommended first.

Lever B — run the two benchmark legs in parallel (harness, tools repo server.py). The Grafana and Datadog legs are independent but serial. Running them concurrently → max(1108, 953) ≈ 1108s ≈ ~46% off. Caveat: the legs currently share one INPUT_DIR/OUTPUT_DIR, so this needs per-leg dirs threaded through _build_migrate_cmd / _parse_report. Medium effort/risk; benchmark-only win.

Lever C — parallelize per-dashboard migrate/translate (engine). 28% of the grafana leg at G:50. Bigger engine change; lower priority.

Levers A and B compound but both consume cores, so combined < additive. On a core-starved CI box the parallel speedup is smaller.

Secondary finding — wasted co-occurrence probes

Probes are only ~2% of wall-clock on the local lab (~6ms each), but the count is very wasteful: at G:500, 6766 probe queries, 6202 errored (91.7%), 1527 error-fallbacks (batched probe 400s → per-candidate re-probe fan-out, issue #182/#188). On a remote/Serverless target each probe is network-bound (10s timeout on errors), so this could flip from negligible to dominant. Worth a follow-up: skip/avoid probing fields that will 400, and/or cache negative results more aggressively. Tracked-adjacent to #182/#188.

Tooling already available (to support this work)

Instrumentation built during the investigation (on branches, not yet merged/pushed):

• Engine branch perf/188-probe-instrumentation:
  • SchemaResolver.probe_stats() + a [probe-stats] {json} line gated on OBS_MIGRATE_PROBE_STATS=1 (queries/seconds/errors/fallbacks/candidates).
  • scripts/bench_probe_ab.py — interleaved A/B of migrate --es-url across two commits, with a request-layer probe counter that works on un-instrumented commits.
• Tools branch feat/probe-stats-instrumentation (elastic/observability-migration-platform-tools):
  • Captures probe_stats per run into benchmark_history.json + renders it (Insights → Speed card, Duration tooltip).
  • OBS_BENCH_ES_URL override so the benchmark can target a non-Cloud ES (e.g. the open local lab) creds-free.

How to reproduce

# Local lab (open ES, security off):
bash scripts/full_local_demo.sh --recreate-lab          # ES :19200, Kibana :15601, seeded metrics-*

# Headless phase profile (timestamped stream → compile vs migrate split):
OBS_MIGRATE_PROBE_STATS=1 obs-migrate migrate --source grafana --input-mode files \
  --input-dir infra/grafana/dashboards --output-dir /tmp/out --compile \
  --es-url http://localhost:19200 --esql-index 'metrics-*'

# Full benchmark via the harness (tools repo):
OBS_MIG_DIR=<engine> OBS_BENCH_ES_URL=http://localhost:19200 python3 server.py
# → Dashboards Benchmark, Schema discovery on

Measurement plan for a fix

1. Headless G:50/D:50 with timestamped stream before/after → isolate the compile delta (baseline compile = 68.8s).
2. One cached G:500/D:671 end-to-end before/after for the total (baseline = 34.3 min, but re-baseline cached to remove download noise).

Notes / caveats

• All numbers above are against the local lab, where probes are cheap (~6ms). The manager's benchmark historically ran against a remote/Serverless cluster, where probe latency would reshuffle the ranking (see secondary finding).
• These optimizations change speed only, not migration output (compile produces the same per-dashboard NDJSON; legs are independent).

[miguel-sanchez-elastic]

Deprioritizing this for now — I don't think migration performance is an actual problem at this point.

Revisit when perf becomes a real constraint.