Skip to content

benchmarks.md

Latest commit

 

History

History
189 lines (133 loc) · 8.05 KB

benchmarks.md

File metadata and controls

189 lines (133 loc) · 8.05 KB
Raw

Benchmarks

Engram is benchmarked on MuSiQue (multi-hop QA) — the harder cousin of HotpotQA where the answer chain typically spans 2-4 paragraphs without shared entity surface forms. Source: dgslibisey/MuSiQue on HuggingFace.

How to run

python -m benchmarks.musique \
  --question-ids-file benchmarks/fixtures/musique_n200_seed1_ids.json \
  --mode baseline \
  --rerank \
  --ircot \
  --output predictions.jsonl

Full CLI in docs/configuration.md. Code lives in benchmarks/musique.py.

Fixtures (pinned question IDs)

benchmarks/fixtures/ holds canonical question-ID JSON files. These exist so re-runs are reproducible against the same questions — crucial for noise-aware comparison.

Fixture Questions Use case
musique_n100_seed1_ids.json 100 Cheap ablations (~$0.30-1.50 per run)
musique_n200_seed1_ids.json 200 Canonical evaluation (~$1.50-4 per run)

Hop distribution at n=200: 97 × 2-hop, 65 × 3-hop, 38 × 4-hop (103 multi-hop ≥ 3).

When --question-ids-file is passed, --n / --seed / --no-shuffle are ignored.

Reference numbers (n=200 baseline mode, synth_off index)

Production v1 config (--mode baseline --rerank --ircot) on the gpt-4o-mini reader:

Config F1 EM
Plain hybrid (no rerank) ~0.40 —
+ Cohere Rerank 0.46 0.32
+ IRCoT (production v1) 0.54 0.40

Run-to-run reader noise floor at n=200: ±0.02-0.03 F1 (we measured this with paired parallel re-runs of identical pipelines).

Reference numbers (KG mode comparison, n=200)

With and without IRCoT, with and without synthesis at ingest:

Without IRCoT

Mode synth_on (F1 / EM) synth_off (F1 / EM)
Baseline (no KG) 0.450 / 0.300 0.446 / 0.300
KG-hybrid 0.478 / 0.350 0.466 / 0.340
Δ KG-Baseline +0.028 / +0.05 +0.020 / +0.04

With IRCoT

Mode synth_on (F1 / EM) synth_off (F1 / EM)
Baseline (no KG) 0.543 / 0.400 0.537 / 0.365
KG-hybrid 0.514 / 0.360 0.534 / 0.390
Δ KG-Baseline -0.029 / -0.04 -0.003 / +0.025

The takeaway: without IRCoT, KG adds +0.02-0.03 F1. With IRCoT on, KG is flat-to-slightly-negative on aggregate (IRCoT does the same multi-hop reasoning more cheaply). Use KG mode for capabilities (graph queries, bi-temporal supersession, contradiction surfacing), not for benchmark F1.

Negative ablations (what we tried that didn't work)

All measured at n=100 or n=200, baseline mode, synth_off index, on top of --ircot:

Added on top of IRCoT Effect Suspected mechanism
Multi-query expansion (--multi-query) F1 -0.08 Alternative phrasings diversify retrieval at the cost of guidance; RRF fuses noise
Question decomposition (--decompose) F1 -0.08 Sub-questions retrieve their own chunks; bridge facts get displaced
Sufficiency judge F1 -0.04 Over-prunes IRCoT round 2 on hard questions
CRAG-style chunk filter F1 -0.13 (combined with sufficiency judge) Double-pruning + Cohere rerank already filters relevance well

Detail in research/query-side-ablation-results.md (internal — gitignored).

Field comparison at gpt-4o-mini reader

System F1 EM
KAG 0.460 0.338
HippoRAG 2 (gpt-4o-mini estimate) 0.486 ~0.32
IRCoT + GFM-RAG 0.492 0.366
G-reasoner SOTA 0.525 0.385
Engram (production v1: hybrid + Cohere + IRCoT) 0.54 0.40

Caveat: our 200-question sample isn't directly comparable to their full-dev-set numbers. Sample variance at n=200 is ±0.02-0.03 F1. The position is "at SOTA tier" not "above SOTA."

Cost profile (gpt-4o-mini + text-embedding-3-small + Cohere Rerank 3.5)

Ingest

Mode $/1K chunks Wall time/1K chunks
Baseline (chunks only) ~$0.008 ~15 sec
+ cold path, synth ON ~$0.46 ~12 min
+ cold path, synth OFF (--disable-synthesis) ~$0.38 ~8.5 min

Cold path breakdown:

  • Synthesis hot path: ~$0.30 / 1K + ~4 min/1K (only fires when enable_synthesis=True)
  • Entity + fact extraction (batched 25 chunks/call): ~$0.62 / 4K chunks
  • Embeddings (chunks + fact triples): negligible (~$0.025 / 4K)

Query

Mode $/query Latency
Plain (no rerank, no IRCoT) ~$0.0005 ~1 sec
+ Cohere Rerank ("fast mode") ~$0.0015 ~1.5-2 sec
+ IRCoT (production v1) ~$0.003 ~3-5 sec
+ KG retrieval ~$0.004 ~4-6 sec

Per-query cost breakdown for production v1 (IRCoT):

  • IRCoT round 1 reader: ~$0.0004 (~2000 in + 150 out gpt-4o-mini tokens)
  • IRCoT round 2 reader: ~$0.0004
  • 2× Cohere Rerank (one per round): ~$0.002
  • Query embeddings: negligible

Reproducibility tips

Reusing existing indices (no re-ingest)

If you have a populated --data-dir, omit --reingest. The runner reuses the LMDB env and only rebuilds the BM25 in-memory index (~30 ms). This is critical for cheap query-side ablations — see the next section.

Replaying queries against an existing index

python -m benchmarks.musique \
  --question-ids-file benchmarks/fixtures/musique_n100_seed1_ids.json \
  --mode baseline \
  --rerank --ircot \
  --data-dir ~/.engram-bench/my_existing_index/ \
  --output replay_predictions.jsonl

This is how we ran the ablation matrix without paying ingest costs every time. ~$0.30-1.50 per query-only replay at n=100-200.

Per-question diagnostic

Predictions are emitted as JSONL, one record per question. Compare runs by question_id:

import json, re

def norm(s):
    return re.sub(r'\s+', ' ', re.sub(r'[^\w\s]', '', s.lower())).strip()

def em(pred, golds):
    p = norm(pred)
    return any(p == norm(g) for g in golds)

# Load both runs and find questions where one flipped right ↔ wrong
prior = {json.loads(line)['question_id']: json.loads(line) for line in open('run_a.jsonl')}
curr = {json.loads(line)['question_id']: json.loads(line) for line in open('run_b.jsonl')}

regressed = [qid for qid in prior if em(prior[qid]['prediction'], prior[qid]['gold_answers'])
                                  and not em(curr[qid]['prediction'], curr[qid]['gold_answers'])]
gained = [qid for qid in prior if not em(prior[qid]['prediction'], prior[qid]['gold_answers'])
                              and em(curr[qid]['prediction'], curr[qid]['gold_answers'])]
print(f"regressed: {len(regressed)}  gained: {len(gained)}")

Rate control during runs

The benchmark uses Engram's TokenBucket + AdaptiveConcurrency (src/engram/llm/rate_control.py) — a port of Vrin's Gradient2-style limiter. Defaults:

Flag Default What it controls
--rps 20.0 Sustained requests per second cap
--burst 25 Token-bucket burst capacity
--initial-concurrency 4 Where AdaptiveConcurrency starts
--max-concurrency-limit 12 Hard ceiling for AdaptiveConcurrency

These prevent rate-limit failures on the OpenAI / Anthropic / Bedrock APIs without flat-rate-limiting (Gradient2 adjusts based on observed RTT).

Scoring

benchmarks/scoring.py implements SQuAD-style normalization (lowercase, strip articles/punct, collapse whitespace) for both EM and F1. Same metric used by every paper we compare against.

Cost-conscious ablation discipline

Best practices we landed on after many ablations:

  1. Reuse indices. Re-ingesting 4K chunks costs ~$2 and takes 25-40 min. For query-side ablations, point --data-dir at an existing index and skip --reingest.
  2. Start at n=100. Half the cost, similar signal. Promote to n=200 only when n=100 shows meaningful direction.
  3. Run a single configuration per ablation. Multi-flag stacking creates ambiguity about which flag did what.
  4. Always have a noise-floor measurement. Re-run the same config twice on the same index to estimate run-to-run noise before claiming small lifts are real.
  5. Use the predictions_*.jsonl outputs for per-question diagnostics. Aggregate F1 hides where the wins and losses actually happen.