Commit d371b63
feat: 3-tier compression (LSM tree architecture) (#200)
* feat(state): add CompressionTier type, tier field, and getTierTokenUsage
Foundation for multi-tier compression: CompressionTier (1|2|3) type, tier field on CompressionBlock, lastTierNudgeTokens in Nudges, getTierTokenUsage() utility, and tier parsing in loadPruneMessagesState.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat(compress): tier auto-detection in applyCompressionState
When compressing blocks (b prefix), auto-detect output tier from consumed blocks: max(consumed.tier) + 1, capped at 3. Pipeline phantom-check carve-out for tier escalation (consumedBlockIds >= 2).
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat(compress): hide consumed compress calls from visible context
New hideConsumedCompressCalls module removes tool-call parts from compress calls whose blocks have been consumed by tier 2+ compression. Prevents double-counting: consumed block summaries no longer appear both in the tier-1 compress call AND the tier-2 compress call.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat(inject): independent tier 2/3 trigger in nudge system
Each tier triggers INDEPENDENTLY when its input summaries reach nudgeGrowthTokens. T2 fires when tier1Tokens >= threshold, T3 fires when tier2Tokens >= threshold. Compression ratio naturally controls frequency: T1 every few turns, T2 every few dozen, T3 every few hundred. Includes growth-floor cadence gate and clears lastTierNudgeTokens on compress.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat(status): tier labels and effective token display in acp_status
acp_status now shows tier labels (T1/T2/T3), effective compressed tokens (recursive sum through consumed blocks), and tier breakdown line. System prompt gains MULTI-TIER COMPRESSION section explaining the 3-tier mechanism to the model.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* test: tier compression E2E tests
9 E2E tests covering: tier auto-detection (T1 from raw, T2 from T1 blocks, T3 from T2 blocks, cap at 3), tier token usage counting, and trigger threshold logic.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* chore: pin context-compress-algorithms to 1.2.0
Pin exact version instead of ^1.0.0 to prevent incompatible versions. cc-alg 1.2.0 adds TIER2_DISTILL_RULES and TIER3_CONDENSE_RULES required by the tier compression system.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* chore: add tier lifecycle simulation script
scripts/simulate-tier-lifecycle.ts: 5-year simulation of tier compression lifecycle from empty session. Shows T1/T2/T3 trigger frequency, compression ratios, and net context growth rate.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* docs: tier compression devlog
REQ + WORKLOG for 3-tier compression (LSM tree architecture): problem analysis, design decisions, implementation phases, Oracle review fixes, trigger redesign, cc-alg release.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix: address dual-agent review findings on tier compression
Fixes from PR #200 dual-agent review:
- Remove unsound (cbTier+1) as CompressionTier cast; use Math.min(3, maxConsumedTier+1)
- Fix enoughCandidates dead code: candidates.length >= 2 (not tautological token check)
- Remove unnecessary structural cast in hide-consumed.ts (p as {tool?:string}).tool
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* test: fix makeAssistantMessage 3-arg call + add tier persistence round-trip tests
Fixes from PR #200 dual-agent review:
- makeAssistantMessage now accepts optional extraParts (3rd arg was silently discarded before)
- Add 3 persistence round-trip tests: tier 1/2/3 preserved, missing tier defaults to undefined, invalid tier (>3) rejected
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix(compress): prevent cross-tier contamination and track effectiveCompressedTokens
Two deferred known limitations from dual-agent review:
1. Cross-tier contamination: when a tier escalation nudge suggests
'compress(b5, b20)', search.ts resolves the range by anchor position
and consumes ALL active blocks in range — including non-target-tier
blocks. This causes unintended tier escalation (e.g., T2 trigger
consuming a T2 block → output T3 instead of T2).
Fix: nudge sorts candidates by blockId and narrows the suggested range
to exclude non-target active blocks (inject.ts). Safety net in
applyCompressionState uses minConsumedTier for output tier and skips
deactivation of non-target-tier consumed blocks (state.ts).
2. compressedTokens=0 for T2+ blocks: tier escalation blocks have no
direct messages, so compressedTokens=0. This causes notification
'0→2K', stats undercount, and misleading display.
Fix: new effectiveCompressedTokens field on CompressionBlock
(types.ts). Computed at creation time as compressedTokens + sum of
consumed blocks' effectiveCompressedTokens (state.ts). Persisted
through loadPruneMessagesState (utils.ts). Stats use effective tokens.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix(ui): use effectiveCompressedTokens in notification and display surfaces
notification.ts: log compression uses effective tokens instead of raw
compressedTokens (was 0 for T2+ blocks).
status.ts: getEffectiveCompressedTokens prefers stored field, falls back
to recursive computation for old state files.
compression-targets.ts: sum uses effectiveCompressedTokens ?? compressedTokens
so decompress/recompress display labels show full coverage.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* test: add cross-tier safety and effectiveCompressedTokens tests
4 new E2E tests:
- T2 trigger narrows range when non-target (T2) block between T1 candidates
- applyCompressionState mixed-tier consumption produces minTier+1
- T2 block gets effectiveCompressedTokens = consumed T1 tokens
- T1 block gets effectiveCompressedTokens = compressedTokens
2 new persistence tests:
- effectiveCompressedTokens round-trip through loadPruneMessagesState
- missing effectiveCompressedTokens defaults to undefined
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* docs: update devlog with deferred limitation fixes
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* refactor: remove dead code from cross-tier fix (maxConsumedTier, included var)
Cleanup from dual-agent review:
- Remove unused maxConsumedTier variable (output tier uses minConsumedTier)
- Remove unused included variable (includedBlockIds uses consumed.filter directly)
- Fix indentation on includedBlockIds/consumedBlockIds fields
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* chore: pin context-compress-algorithms to 1.2.1
TIER2/TIER3 prompt fixes: remove Memory reference, fix SIZE TARGET
contradictions, add priority guidance, add cross-block synthesis.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* refactor(state): split lastTierNudgeTokens into independent per-tier counters
T2 and T3 triggers now have independent cadence: lastTier2NudgeTokens and lastTier3NudgeTokens. Old lastTierNudgeTokens kept as @deprecated persisted field, migrated to lastTier2NudgeTokens on load (same tier level).
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* refactor(inject): unified tier trigger loop with T1 priority
Replace separate T2/T3 trigger block with a unified loop over tierChecks array. T1 (normal compression) gets priority via !shouldInject guard — if T1 fires, tier escalation is skipped this turn. Each tier has independent cadence counter, so T2 firing doesn't block T3. First match in priority order wins, then break.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* test: large-scale E2E simulation for tier compression strategy (8 tests)
SIM 1-8 verify: T1 fires on context limit, T2 escalation from T1 summaries, T3 escalation from T2 summaries, T1 priority over T2, independent per-tier cadence, cross-tier safety narrowing, no-crash steady state over 30 turns.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* docs(readme): document three-tier compression architecture (T1/T2/T3)
Replace old single-tier compress/decompress diagram with three-tier
LSM-tree lifecycle: T1 (capture, ~45x) -> T2 (distill, ~10x) ->
T3 (condense, ~5x). Add tier trigger explanation (independent cadence
counters, T1 priority guard), 5-year lifecycle projection table,
and rename 'Compression strategy' to 'Compression strategy (Tier 1)'.
Update GC safety net section to note 179 tok/day net growth.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* docs(readme): add token estimates to lifecycle projection table
Show summary token counts at year 5 (T1=87K, T2=9K, total~96K=9.6% of 1M)
and at 15-year fill point (~150K). Makes the projection concrete.
* docs(readme): fix lifecycle projection with actual simulation data
Year 5: visible context 327K (32.7%), not 96K. Summary overhead 64K.
Net growth 179 tok/day. Context fills 1M in ~15 years.
Simulation source: scripts/simulate-tier-lifecycle.ts
* docs: add cumulative token savings table to lifecycle projection
Track cumulative token consumption (billing metric) alongside
instantaneous context size. Key finding: without ACP cumulative
grows O(n²), with ACP O(n). 1-year session saves 96%, 5-year
saves 97%.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* docs: replace lifecycle projection with real-calibrated session capacity data
Old projection used 7.3K/day growth (wrong). New simulation calibrated
from real sessions (500 calls/day, 9.6K/call). Two context limits:
- 1M model: 68.9B tokens over 259 days
- 400K model: 10.3B tokens over 89 days (500 calls/day)
- 400K model: 9.5B tokens over 212 days (200 calls/day)
Key insight: ACP enables 1000x+ more total work per session.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* feat(decompress): tier-aware decompress — one level up by default
Decompressing a T2+ block now restores the PREVIOUS tier's summaries
(e.g., decompress T2 → T1 summaries visible) instead of jumping all
the way to original messages. This prevents catastrophic context
explosion when decompressing high-tier blocks.
New option: full:true restores all content to original messages
(legacy behavior). Useful when exact original content is needed.
Changes:
- deactivateCompressionTarget: consumed blocks only marked
deactivatedByUser when full:true (was always)
- decompress tool: added full parameter to schema
- system prompt: updated decompress description for tier awareness
- Tests: 2 new unit tests + 2 new E2E tests (915 total pass)
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix: address dual-agent review findings (5 MEDIUM)
1. full:true decompress now recursively walks consumed tree (T3+ fix)
2. stats use compressedTokens not effectiveCompressedTokens (no double-count)
3. Fix broken SIM 1 assertion + remove as any cast
4. minConsumedTier default 3→1 (safe fallback for missing blocks)
5. Remove @ts-expect-error on optional tier field
6. Replace (p: any) with proper type guards in test helpers
915 tests pass, typecheck clean.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* test: add E2E round-trip tests for compress→decompress→recompress
4 new E2E tests (919 total pass):
1. Round-trip content identity: compress→decompress→content identical
2. Recompress no redundancy: decompress→recompress→no duplicate blocks
3. T3 default decompress: T3→T2 summaries (one level up)
4. T3 full:true decompress: recursive to raw (all descendants deactivated)
Also reverts minConsumedTier default 3→1 (M4 fix was incorrect —
min logic requires high initial value; starting at 1 breaks T3 detection
when consuming T2 blocks).
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix: blocks survive anchor message removal (sync.ts)
Removed anchor-missing deactivation in syncCompressionBlocks. Previously,
blocks were deactivated when their anchorMessageId (the compress tool
call) was removed from the message list. This caused 1137 blocks across
21 sessions to be incorrectly deactivated when opencode compaction
removed old compress tool calls.
The block's existence IS proof that compression happened — same logic
as Bug 3 (compressMessageId check, already removed).
Updated 4 tests: blocks now stay active when anchor is gone.
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
* fix(state): tier detection + effectiveMessageIds filter (M1+M2)
M1: minConsumedTier uses undefined sentinel instead of magic init=3. If
consumed IDs don't resolve in blocksById, falls back to T1 (safe default).
M2: effectiveMessageIds now filtered by targetTierForConsumption. Non-target-
tier consumed blocks no longer pollute the effective set.
Also: includedBlockIds stores ALL consumed (unfiltered) for display, while
consumedBlockIds remains tier-filtered for sync deactivation.
* fix(decompress): deactivatedByUserDeep for full:true recursive (M4+L1)
M4: full:true decompress now sets deactivatedByUserDeep (not deactivatedByUser)
on consumed blocks. Recompress walks consumedBlockIds recursively to clear it.
This prevents consumed blocks from being permanently flagged after full:true.
L1: Removed missingOriginBlockIds dead code from sync.ts (declared, never
populated, only logged).
Also: sync.ts and hide-consumed.ts check both deactivatedByUser and
deactivatedByUserDeep to keep blocks inactive.
* fix(pipeline+status): phantom tier check + includedBlockIds display (M3+L2)
M3: Phantom carve-out now requires homogeneous tier (tiers.size === 1).
Mixed-tier consumed blocks (cross-tier contamination) no longer bypass
phantom rejection.
L2: Status display uses includedBlockIds (all consumed) instead of
consumedBlockIds (target-tier only) for nested= rendering.
* test: fix review findings M5/M6 + update assertions for M4
M5: Renamed misleading test 'fires even when T1 nudge would also fire' to
'fires when T1 summaries exceed threshold even with large context'. Code
has T1 priority via !shouldInject guard, not independent firing.
M6: Fixed simulation factory parentBlockIds from [...consumedBlockIds]
(reversed semantics) to [] (blocks aren't consumed by anything in test).
Updated assertions for M4: consumed blocks now check deactivatedByUserDeep
instead of deactivatedByUser.
* docs: update devlog REQ + WORKLOG with all phases (7-10)
---------
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>1 parent df83bc8 commit d371b63
32 files changed
Lines changed: 3265 additions & 102 deletions
File tree
- devlog/2026-07-25_tier-compression
- lib
- commands
- compress
- messages
- inject
- prompts
- state
- ui
- scripts
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
94 | | - | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
95 | 97 | | |
96 | | - | |
| 98 | + | |
97 | 99 | | |
98 | | - | |
99 | | - | |
100 | | - | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
104 | | - | |
105 | | - | |
106 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
107 | 113 | | |
108 | 114 | | |
109 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
110 | 163 | | |
111 | 164 | | |
112 | 165 | | |
| |||
132 | 185 | | |
133 | 186 | | |
134 | 187 | | |
135 | | - | |
| 188 | + | |
136 | 189 | | |
137 | 190 | | |
138 | 191 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | | - | |
| 79 | + | |
80 | 80 | | |
81 | | - | |
| 81 | + | |
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
86 | | - | |
87 | | - | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
88 | 92 | | |
89 | 93 | | |
90 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
91 | 130 | | |
92 | 131 | | |
93 | 132 | | |
| |||
107 | 146 | | |
108 | 147 | | |
109 | 148 | | |
110 | | - | |
| 149 | + | |
111 | 150 | | |
112 | 151 | | |
113 | 152 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
0 commit comments