Skip to content

perf: compact repeated request history prefixes#288

Open
liaohch3 wants to merge 1 commit into
mainfrom
feat/sqlite-prefix-compaction
Open

perf: compact repeated request history prefixes#288
liaohch3 wants to merge 1 commit into
mainfrom
feat/sqlite-prefix-compaction

Conversation

@liaohch3

@liaohch3 liaohch3 commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add SQLite schema v5 prefix-delta compaction for repeated request.body.input and request.body.messages histories.
  • Keep existing v3/v4 SQLite data readable; v4 -> v5 only adds the new schema version marker and does not rewrite existing rows.
  • Add anchor-aware pagination, boundary reads, JSONL/compact export materialization, and an append-time last-record cache so large sessions stay fast.

Validation

  • uv run ruff check .
  • uv run ruff format --check .
  • uv run pytest tests/ -x --timeout=60 -> 625 passed, 25 skipped
  • uv run pytest tests/test_trace_store_compaction.py tests/test_history.py -q -> 24 passed
  • Real-shape storage validation: seeded from .traces/codex-tool-mechanisms-20260509-more/2026-05-09/trace_083536.jsonl, wrote 60 Codex-style growing-history records into SQLite v5, verified random pagination, boundary reads, JSONL export, compact export, and materialization. Result: 57 prefix-delta rows, 16,359,129 raw JSONL bytes vs 916,272 compact bytes, storage ratio 0.056.
  • Legacy SQLite validation: copied .traces/pr269-bedrock-evidence/traces.sqlite3 and .traces/pr269-newapi-bedrock-claude/traces.sqlite3, loaded/exported sessions successfully, and confirmed schema v5 migration.
  • Evidence screenshot validation: uv run python scripts/check_screenshots.py .agents/evidence/pr/pr-sqlite-prefix-compaction-viewer.png

Evidence

Privacy

  • This PR does not commit raw trace JSONL, generated HTML viewers, logs, API keys, or auth tokens.

Co-authored-by: Talon <talon@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants