feat: temporal scoring, staleness tracking, and drift detection

[NoRain211]

Summary

Adds Ember MCP-inspired features for memory relevance management:

• Temporal decay scoring — observations decay in search relevance over time with importance-scaled half-life (18d–180d). Access tracking boosts frequently-retrieved memories.
• Stale filtering — stale observations excluded from search by default (opt-in via include_stale param)
• contradict tool — mark a memory stale and record a correction linked to it
• set_importance tool — adjust importance score (1–10) to control decay rate
• drift_check tool — SQL cluster analysis via json_each(concepts) to detect concept areas where old memories have gone stale or been superseded
• SQLite FTS5 fallback — restores text search when ChromaDB is unavailable (previously returned empty results)

Schema migration (v24)

Adds 5 columns to observations: last_accessed_at, access_count, importance, is_stale, corrected_by_id. Idempotent with per-column existence checks before each ALTER to handle partial migration failures.

Files changed

File	Change
src/services/sqlite/migrations/runner.ts	Migration v24
src/services/sqlite/types.ts	5 new fields on ObservationRow interface
src/types/database.ts	5 new optional fields on ObservationRecord
src/services/sqlite/SessionSearch.ts	FTS5 fallback, rankByTemporalScore (importance-scaled), updateAccessTracking, detectDrift
src/services/sqlite/SessionStore.ts	markObservationStale, setObservationImportance
src/services/worker/search/SearchOrchestrator.ts	Apply temporal scoring + stale filter after search
src/services/worker/http/routes/MemoryRoutes.ts	Routes for contradict, importance, drift-check
src/services/worker/http/routes/DataRoutes.ts	Access tracking on batch observations fetch
src/servers/mcp-server.ts	3 new MCP tools: contradict, set_importance, drift_check
plugin/skills/mem-search/SKILL.md	Documentation for new tools

Test plan

• Fresh install: migration v24 runs, all 5 columns added
• Existing DB: migration skips columns that already exist (per-column idempotent)
• contradict tool: marks observation stale, creates correction, links via corrected_by_id
• set_importance tool: clamps to 1–10, updates observation
• drift_check tool: returns "no drift" on fresh DB, detects clusters after accumulation
• Search: stale observations excluded by default, included with include_stale: true
• Temporal scoring: older observations rank lower, high-importance decays slower
• FTS5 fallback: text queries return results when ChromaDB is not configured
• Null safety: temporal scoring handles null/undefined importance, access_count, is_stale

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR implements Ember MCP-inspired temporal memory management features. The implementation adds temporal decay scoring where memories naturally fade over time based on importance (half-life ranging from 18-180 days), access tracking that boosts frequently-retrieved memories, and staleness management to mark outdated observations.

Key changes:

• Database migration v22 adds 5 new columns to track access patterns, importance, and staleness
• FTS5 fallback enables text search when ChromaDB is unavailable (previously returned empty results)
• Three new MCP tools: contradict for marking memories stale with corrections, set_importance for controlling decay rate, and drift_check for detecting outdated concept clusters
• Search results now automatically ranked by temporal score and filter out stale observations by default

Issues found:

• Migration idempotency flaw: only checks for one column instead of all five, which could lead to incomplete migrations if a partial failure occurs
• Minor style issue: redundant column in FTS5 SELECT clause

Confidence Score: 4/5

• Safe to merge with one migration fix recommended before deployment
• Score reflects solid implementation of temporal features with comprehensive validation and error handling, but migration idempotency issue could cause problems in edge cases (crashes during migration)
• Pay close attention to src/services/sqlite/migrations/runner.ts - fix the idempotency check before deploying to production to prevent partial migration states

Important Files Changed

Filename	Overview
src/services/sqlite/migrations/runner.ts	Adds migration v22 for temporal scoring columns; idempotency check has flaw that could cause incomplete migrations
src/services/sqlite/SessionSearch.ts	Adds FTS5 fallback, temporal scoring, access tracking, and drift detection; minor redundancy in SELECT clause
src/services/worker/search/SearchOrchestrator.ts	Applies temporal scoring and stale filtering to search results; enables FTS5 fallback when ChromaDB unavailable
src/services/worker/http/routes/MemoryRoutes.ts	Adds contradict, set_importance, and drift_check endpoints with proper validation and error handling

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Search Request] --> B{ChromaDB Available?}
    B -->|Yes| C[Vector Search]
    B -->|No| D[SQLite FTS5 Search]
    C --> E[Raw Results]
    D --> E
    E --> F[Apply Temporal Scoring]
    F --> G{include_stale param?}
    G -->|false default| H[Filter Stale Observations]
    G -->|true| I[Keep All Results]
    H --> J[Return Ranked Results]
    I --> J
    
    K[User Access] --> L[Track Access]
    L --> M[Update last_accessed_at]
    L --> N[Increment access_count]
    
    O[Contradict Tool] --> P[Create Correction Observation]
    P --> Q[Mark Original as Stale]
    Q --> R[Link via corrected_by_id]
    R --> S[Sync to ChromaDB]
    
    T[Set Importance Tool] --> U[Clamp 1-10]
    U --> V[Update importance field]
    V --> W[Affects decay half-life]
    
    X[Drift Check Tool] --> Y[Analyze Concepts via json_each]
    Y --> Z[Calculate Stale %]
    Z --> AA[Flag High-Stale Clusters]
    AA --> AB[Return Drift Report]

Loading

_{Last reviewed commit: bbe926c}

[greptile-apps]

_{9 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Migration idempotency only checks for last_accessed_at column. If a partial migration occurs (e.g., first 2 columns added then a crash), retry will skip remaining columns.

Suggested change

	const tableInfo = this.db.query('PRAGMA table_info(observations)').all() as TableColumnInfo[];
	const hasColumn = tableInfo.some(col => col.name === 'last_accessed_at');
	if (!hasColumn) {
	this.db.run('ALTER TABLE observations ADD COLUMN last_accessed_at INTEGER');
	this.db.run('ALTER TABLE observations ADD COLUMN access_count INTEGER NOT NULL DEFAULT 0');
	this.db.run('ALTER TABLE observations ADD COLUMN importance INTEGER NOT NULL DEFAULT 5');
	this.db.run('ALTER TABLE observations ADD COLUMN is_stale INTEGER NOT NULL DEFAULT 0');
	this.db.run('ALTER TABLE observations ADD COLUMN corrected_by_id INTEGER');
	logger.debug('DB', 'Added temporal scoring columns to observations table');
	}
	const tableInfo = this.db.query('PRAGMA table_info(observations)').all() as TableColumnInfo[];
	const existingColumns = new Set(tableInfo.map((col: any) => col.name));
	if (!existingColumns.has('last_accessed_at')) {
	this.db.run('ALTER TABLE observations ADD COLUMN last_accessed_at INTEGER');
	}
	if (!existingColumns.has('access_count')) {
	this.db.run('ALTER TABLE observations ADD COLUMN access_count INTEGER NOT NULL DEFAULT 0');
	}
	if (!existingColumns.has('importance')) {
	this.db.run('ALTER TABLE observations ADD COLUMN importance INTEGER NOT NULL DEFAULT 5');
	}
	if (!existingColumns.has('is_stale')) {
	this.db.run('ALTER TABLE observations ADD COLUMN is_stale INTEGER NOT NULL DEFAULT 0');
	}
	if (!existingColumns.has('corrected_by_id')) {
	this.db.run('ALTER TABLE observations ADD COLUMN corrected_by_id INTEGER');
	}
	logger.debug('DB', 'Added temporal scoring columns to observations table');

[greptile-apps]

discovery_tokens already included in o.*

Suggested change

	let sql = `
	SELECT o.*, o.discovery_tokens
	SELECT o.*
	FROM observations o

[thedotmack]

@NoRain211 are you seeing improvements in performance with this PR?

[Chriscross475]

Good implementation of temporal scoring and staleness tracking. Core logic is sound, but several issues need addressing:

Critical:

1. Migration version mismatch: PR description says "migration v22" but code shows version 24 in addTemporalScoringColumns(). Which is correct? This could cause migration conflicts.

Code Quality:
2. Silent error handling: FTS5 fallback in searchObservations() has try { ... } catch { return []; } with no logging. Errors will be invisible. Add:

} catch (err) {
  logger.error('DB', 'FTS5 search failed', { query: ftsQuery }, err as Error);
  return [];
}

3. Type safety: Multiple (obs as any).importance, (obs as any).access_count, (obs as any).is_stale casts. Either:

   • Extend ObservationSearchResult interface with the new optional fields, OR
   • Add a comment explaining why as any is necessary (if fields aren't guaranteed to exist during migration)

4. Test plan incomplete: All checkboxes unchecked. Please verify:

   • Migration idempotency (run twice on same DB)
   • Tools work end-to-end
   • Temporal scoring doesn't crash on null values

Minor:
5. Drift detection: Returns generic message "Drift check unavailable" on any SQL error. Consider logging the error for debugging.

What's good:

• ✅ Temporal decay math is correct (verified 18d/90d/180d formula)
• ✅ Fire-and-forget access tracking is appropriate
• ✅ Input validation on API routes is solid
• ✅ Migration has column-existence checks (idempotent)

Fix the migration version and add error logging, then this is ready to merge.

[NoRain211]

@NoRain211 are you seeing improvements in performance with this PR?

These changes are about result quality, not query speed. The temporal scoring means I get more relevant context at session start — recent, frequently-accessed memories rank higher than stale ones from months ago. The practical improvement is spending less time sifting through outdated context and fewer cases where old architectural decisions get surfaced as current.

I haven't benchmarked query latency since the scoring is applied in-memory after the SQLite/Chroma query returns — the overhead should be negligible for typical result sets (<100 observations).

[Chriscross475]

The changes look well-structured with good documentation and a comprehensive test plan. However, CI checks are not running on this PR.

Given that this PR includes:

• Database schema migration (v24 with 5 new columns)
• Complex temporal scoring logic
• FTS5 fallback implementation

I need to see CI passing before approval. Please:

1. Ensure GitHub Actions workflow runs on this branch
2. Verify all tests pass
3. Confirm build succeeds with the new migration

Once CI is green, I'll do a full code review.

[NoRain211]

CI Verification (local)

Note on CI: The repo doesn't currently have a test/build CI workflow — the existing GitHub Actions are Claude review, npm publish, and issue management. Rather than adding a CI workflow to this PR (which would be scope creep and add maintenance burden), we ran the full test suite and build locally. Happy to help set up a test CI workflow in a separate PR if that would be useful.

Build

✓ worker-service built (1872 KB)
✓ mcp-server built (336 KB)
✓ context-generator built (72 KB)
✓ All required distribution files present

Tests

SQLite tests:     61 pass, 0 fail (150 assertions)
Search tests:    122 pass, 0 fail (239 assertions)
Context tests:    60 pass, 0 fail (97 assertions)
Server tests:     56 pass, 1 fail* (108 assertions)

Total:           299 pass, 1 fail (594 assertions)

*The 1 server test failure is EADDRINUSE — the worker was already running on port 37777 during testing. Not related to this PR.

Test fixes in this push

• Added rankByTemporalScore to the SessionSearch mock in search-orchestrator.test.ts
• Updated the "empty results without Chroma" assertion to expect FTS5 fallback behavior (returning results via SQLite) instead of empty results — this matches our intentional behavior change

Migration version note

The PR description now correctly says v24 (not v22). v21-23 were taken by upstream commits added after our initial PR.