diff --git a/.gitignore b/.gitignore index 73c1687..a1230d7 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ node_modules/ # Build / cache .wrangler/ +.recall-update/ dist/ .cache/ *.tsbuildinfo diff --git a/CHANGELOG.md b/CHANGELOG.md index 166189c..10cda7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,31 @@ All notable changes to Recall are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [2.2.0] — 2026-06-22 + +Trustworthy-memory and Hermes-integration release. Adds provenance/lifecycle/supersession workflows and an optional conservative updater path without changing the config ownership model. + +### Added + +- **Provenance and lifecycle metadata** for trustworthy agent memory: source type/URL/path/title/hash, confidence, verification, expiration, status, and supersession fields. +- **Lifecycle MCP tools**: `mark_memory_status`, `verify_memory`, and `supersede_memory`, bringing the expected tool count to 10. +- **Optional updater script** (`recall-update.sh`) with safe default commands: `--check`, `--doctor`, `--apply`, `--rollback`, and `--install-cron weekly`. +- **Updating documentation** (`docs/UPDATING.md`) covering the config preservation contract, migration behavior, rollback, cron opt-in, and GitHub Actions notes. +- **Hermes integration guide** (`docs/HERMES_INTEGRATION.md`) documenting one-instance namespace routing, Discord provenance patterns, wrapper-script MCP config, and cron examples. +- **Release checklist** (`docs/RELEASE_CHECKLIST.md`) and an example weekly GitHub Actions workflow (`examples/github-actions-update.yml`). + +### Changed + +- README now describes Recall as trustworthy memory with provenance/lifecycle/supersession, links the new docs, and updates Docker quickstart `tools/list` expectation to 10 tools. +- Consolidation and lifecycle guidance emphasizes preserving audit trails and superseding stale facts instead of silently overwriting history. + +### Upgrade notes + +- Run `npm ci`, `npm run typecheck`, and `npm test` before deploying. +- Apply numbered D1 migrations in order, or use `./recall-update.sh --check` to inspect pending migrations and `./recall-update.sh --apply` only when you are comfortable with its conservative deploy flow. +- The updater intentionally refuses to overwrite user-owned config/secrets (`wrangler.toml`, `.dev.vars`, `.recall-api-key`, MCP configs, D1/Vectorize/routes/env blocks). If a release changes those areas, merge them manually. +- Rollback redeploys previous worker code only; D1 schema rollback remains manual. + ## [1.1.2] — 2026-04-12 Three bug fixes discovered while running the v1.1.1 wizard end-to-end against a fresh project (the AgentBoard repo). All three would have hit real users; two were silently destructive. @@ -127,6 +152,7 @@ First public release. Foundational architecture and a usable end-to-end setup wi **Note on v1.0.0 tagging history:** Between the initial v1.0.0 tag and the release of v1.1.0, the v1.0.0 tag was force-pushed several times during iterative hardening. This was a mistake — tags should be immutable. With v1.1.0, the tag discipline resets: v1.0.0 is now permanently pinned to its original commit, v1.1.0 is the new pinned default, and future versions will ship as new tags (no force-pushes). See the repository's release notes on GitHub for the canonical per-version state. +[2.2.0]: https://github.com/cashcon57/recall/releases/tag/v2.2.0 [1.1.2]: https://github.com/cashcon57/recall/releases/tag/v1.1.2 [1.1.1]: https://github.com/cashcon57/recall/releases/tag/v1.1.1 [1.1.0]: https://github.com/cashcon57/recall/releases/tag/v1.1.0 diff --git a/README.md b/README.md index 12e9621..e6ea928 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ [![Backends](https://img.shields.io/badge/Backends-3-brightgreen)](#deployment-options) [![Ko-fi](https://img.shields.io/badge/Ko--fi-Support-FF5E5B?logo=kofi&logoColor=white)](https://ko-fi.com/cash508287) -**A self-hosted MCP memory server with hybrid semantic + keyword search. Deploy to Cloudflare Workers, run locally without any cloud dependencies, or self-host with Docker.** +**A self-hosted MCP memory server for trustworthy agent memory: hybrid semantic + keyword search, provenance, lifecycle controls, supersession, and durable recall. Deploy to Cloudflare Workers, run locally without any cloud dependencies, or self-host with Docker.** > Not affiliated with Microsoft Windows Recall. This is an open-source memory server for AI coding assistants (Claude Code, Cursor, Windsurf, Cline, Claude Desktop, anything speaking MCP). @@ -69,11 +69,19 @@ Three backends. Same tool surface. Choose what fits your infra: > **Memory portability between backends:** All three backends use the bge-m3 model (1024D embeddings). Local and Docker backends run the [Xenova ONNX export](https://huggingface.co/Xenova/bge-m3) with CLS pooling to match the reference recipe. Vector compatibility across backends is close in practice — semantic search works cross-backend — but exact bit-for-bit portability is not yet verified with an end-to-end smoke test. If strict portability matters, test your workload before migrating. +## Docs + +- [Updating Recall](./docs/UPDATING.md) — optional updater script, config preservation contract, migrations, rollback, cron, and GitHub Actions notes. +- [Hermes integration](./docs/HERMES_INTEGRATION.md) — recommended namespaces, Discord provenance, MCP wrapper config, routing rules, and cron examples for Hermes Agent. +- [Release checklist](./docs/RELEASE_CHECKLIST.md) — validation checklist for typecheck/test/deploy, MCP smoke tests, lifecycle tools, consolidation, updater, and Hermes docs. + ## Why Recall? Most MCP memory servers do one of two things: dump text into SQLite with cosine similarity, or call a hosted vector DB. Recall does both — at the same time — and reranks the combined results with a cross-encoder. - **Hybrid search** — Vector similarity (bge-m3, 1024D) + BM25 full-text search, fused via Reciprocal Rank Fusion. Catches both semantic paraphrases and exact keyword matches. +- **Trustworthy memory metadata** — Optional provenance fields (`source_type`, URL/path/title/hash), confidence, verification time, expiration, and status make retrieved memories auditable instead of context-free snippets. +- **Lifecycle + supersession** — Mark memories active/stale/superseded/deprecated, verify them, or supersede older keys without losing the historical trail that explains past behavior. - **Cross-encoder reranking** — Final candidates run through bge-reranker-base for precision. Content is truncated before reranking to keep AI token usage low. - **Tiered recency decay** — Three memory tiers with biological half-lives: `episodic` (7d, session events), `semantic` (69d, facts/concepts), `procedural` (693d, stable rules/credentials). Episodic context ages out fast; architecture decisions stay relevant for years. Tier design inspired by [NornicDB](https://github.com/orneryd/NornicDB). - **Auto-relationship graph** — Memories with embedding similarity > 0.82 are automatically linked on store. Traverse with `get_related_memories` to find contextually adjacent knowledge without re-querying. @@ -209,7 +217,7 @@ curl -s -X POST http://localhost:8788 \ -H "Authorization: Bearer your-secret-key" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq '.result.tools | length' -# Expected: 7 +# Expected: 10 # 4. Tear down (keeps data in pgdata volume) docker compose down @@ -355,6 +363,9 @@ All tools are callable via the standard MCP `tools/call` method. | `retrieve_memory` | Hybrid search (vector + BM25 → RRF → rerank → recency decay → importance). Returns top-N with combined scores. Optional `namespace` filter. | | `list_memories` | Browse with pagination + filters (tag, author, `namespace`, limit, offset). Returns metadata only. | | `get_related_memories` | Traverse the auto-relationship graph from a given key. Returns related memories ranked by edge strength. | +| `mark_memory_status` | Change lifecycle status (`active`, `stale`, `superseded`, `deprecated`) without deleting historical context. | +| `verify_memory` | Mark a memory verified with optional confidence/source updates so agents can prefer trustworthy facts. | +| `supersede_memory` | Link an old key to a replacement key, preserving the audit trail while steering retrieval to the newer fact. | | `delete_memory` | Remove a memory by key from the store, FTS, and the vector index. | | `clear_memories` | Wipe everything. **Default-disabled** — requires both `confirm: true` AND the `ALLOW_DESTRUCTIVE_TOOLS=true` secret on the worker. See [Security](#security) for why. | | `consolidate_memories` | Read-only analysis: flags similar memory pairs and stale entries. Returns a markdown report. | @@ -695,7 +706,7 @@ recall/ ├── src/ │ ├── index.ts # Worker fetch + scheduled handler, rate limit, auth │ ├── mcp.ts # JSON-RPC 2.0 / MCP protocol dispatcher -│ ├── tools.ts # 6 tool implementations + search pipeline + consolidation +│ ├── tools.ts # MCP tool implementations + search, lifecycle, and consolidation │ ├── auth.ts # Constant-time HMAC-SHA256 API key verify │ └── types.ts # Env bindings + domain + JSON-RPC types ├── schema.sql # D1 table + indexes + FTS5 virtual table diff --git a/docker/migrations/0004_provenance_lifecycle.sql b/docker/migrations/0004_provenance_lifecycle.sql new file mode 100644 index 0000000..c862e95 --- /dev/null +++ b/docker/migrations/0004_provenance_lifecycle.sql @@ -0,0 +1,20 @@ +-- Migration 0004: Provenance and lifecycle metadata (Postgres) + +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_type TEXT NOT NULL DEFAULT 'manual'; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_url TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_path TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_line_start INTEGER; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_line_end INTEGER; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_title TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_hash TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active'; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS confidence REAL NOT NULL DEFAULT 0.75; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS verified_at TIMESTAMPTZ; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS supersedes_key TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS superseded_by_key TEXT; + +CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status); +CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type); +CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at); +CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key); diff --git a/docker/setup.sql b/docker/setup.sql index 18fd954..f18eff8 100644 --- a/docker/setup.sql +++ b/docker/setup.sql @@ -1,5 +1,19 @@ CREATE EXTENSION IF NOT EXISTS vector; +CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + applied_at TEXT NOT NULL +); + +-- Fresh installs use this full current schema, so mark historical migrations +-- applied. Older installs still apply numbered migration files via the updater. +INSERT INTO schema_migrations(version, applied_at) VALUES + ('0000', NOW()), + ('0002', NOW()), + ('0003', NOW()), + ('0004', NOW()) +ON CONFLICT (version) DO NOTHING; + CREATE TABLE IF NOT EXISTS memories ( id TEXT PRIMARY KEY, key TEXT UNIQUE NOT NULL, @@ -12,7 +26,20 @@ CREATE TABLE IF NOT EXISTS memories ( created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), accessed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - access_count INTEGER NOT NULL DEFAULT 0 + access_count INTEGER NOT NULL DEFAULT 0, + source_type TEXT NOT NULL DEFAULT 'manual', + source_url TEXT, + source_path TEXT, + source_line_start INTEGER, + source_line_end INTEGER, + source_title TEXT, + source_hash TEXT, + status TEXT NOT NULL DEFAULT 'active', + confidence REAL NOT NULL DEFAULT 0.75, + verified_at TIMESTAMPTZ, + expires_at TIMESTAMPTZ, + supersedes_key TEXT, + superseded_by_key TEXT ); CREATE INDEX IF NOT EXISTS idx_memories_key ON memories(key); @@ -20,6 +47,10 @@ CREATE INDEX IF NOT EXISTS idx_memories_author ON memories(author); CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance); CREATE INDEX IF NOT EXISTS idx_memories_accessed_at ON memories(accessed_at); CREATE INDEX IF NOT EXISTS idx_memories_namespace ON memories(namespace); +CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status); +CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type); +CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at); +CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key); CREATE INDEX IF NOT EXISTS idx_memories_fts ON memories USING GIN( to_tsvector('english', content || ' ' || key) diff --git a/docs/HERMES_INTEGRATION.md b/docs/HERMES_INTEGRATION.md new file mode 100644 index 0000000..f6f7863 --- /dev/null +++ b/docs/HERMES_INTEGRATION.md @@ -0,0 +1,102 @@ +# Hermes Integration Guide + +This guide describes a recommended pattern for using one Recall instance as Hermes Agent's durable memory backend across projects. + +## One Recall instance for Hermes + +For a solo operator or trusted personal automation setup, run one Recall instance and use namespaces to separate scopes instead of deploying a worker per project. + +Recommended namespaces: + +- `global` — durable facts and preferences that apply everywhere. +- `hermes` — Hermes Agent configuration, skills, profiles, workflows, and agent behavior notes. +- `switchr` — Switchr-specific product, infra, and codebase memories. +- `corkscrew` — Corkscrew-specific product, infra, and codebase memories. +- `oss-dashboard` — OSS dashboard project memories. +- `retro` — retrospectives, mistakes, release notes, and lessons learned. +- `research` — reusable research notes, vendor comparisons, and investigation results. + +Namespaces are a retrieval filter, not an access-control boundary. Anyone with the Recall API key can read/write any namespace. + +## Provenance for Discord-originated memory + +Hermes can store memories that came from Discord, but the memory should carry enough provenance for future agents to understand where it came from. + +Suggested source/provenance pattern: + +```json +{ + "source_type": "chat", + "source_url": "https://discord.com/channels///", + "source_title": "#channel-name / thread title", + "tags": ["discord", "decision"], + "namespace": "hermes" +} +``` + +For thread discussions, include both the parent channel and thread name in `source_title`, and include a short quote or summary in the memory content. Avoid storing private message content unless you actually want it in long-term agent memory. + +### Limitation: no implicit Discord history reconstruction + +Recall only knows what was explicitly stored. Hermes cannot reconstruct a Discord channel or thread history from Recall unless messages, decisions, or summaries were saved into Recall at the time (or imported later). Store durable summaries, not raw chat logs, unless raw logs are explicitly needed. + +## MCP config without putting the API key in config.yaml + +Prefer a tiny wrapper script that reads the API key from a local file or secret manager and launches the MCP bridge. This keeps `config.yaml` generic and avoids committing secrets. + +Example wrapper (`~/bin/recall-mcp`): + +```bash +#!/usr/bin/env bash +set -euo pipefail +RECALL_URL="${RECALL_URL:-https://your-worker.workers.dev/mcp}" +KEY_FILE="${RECALL_KEY_FILE:-$HOME/.config/recall/api-key}" +if [[ -z "${RECALL_API_KEY:-}" ]]; then + RECALL_API_KEY="$(tr -d '\r\n' < "$KEY_FILE")" +fi +exec npx -y mcp-remote "$RECALL_URL" \ + --header "Authorization: Bearer ${RECALL_API_KEY}" +``` + +Hermes MCP config pattern: + +```yaml +mcp_servers: + recall: + command: /home/you/bin/recall-mcp + args: [] +``` + +Do not paste the real API key into shared config. If you need to show examples in docs, use `Bearer ` or `Bearer your-api-key-here`. + +## Memory routing rules + +Use these conventions when Hermes decides where to store or retrieve memory: + +1. Store cross-project personal preferences in `global`. +2. Store Hermes setup, MCP, skill, plugin, cron, and profile behavior in `hermes`. +3. Store project-specific code/infra/product facts in that project's namespace. +4. Store postmortems, mistakes, and durable lessons in `retro`, with project tags when relevant. +5. Store reusable vendor/library/API research in `research`. +6. Never store API keys, passwords, private keys, session cookies, or recovery codes. +7. Prefer memories with provenance: source type, URL/path, title, line range, or Discord channel/thread reference. +8. Mark decisions with a `decision` tag and write the rationale in the content. +9. Supersede stale memories rather than silently overwriting when the old fact may still explain past behavior. +10. Use `confidence`, `verified_at`, and expiration fields for facts that may go stale. + +## Cron examples + +Weekly Recall update check (non-mutating): + +```cron +0 4 * * 1 cd /path/to/recall && ./recall-update.sh --check >> .recall-update/cron.log 2>&1 +``` + +Weekly consolidation report through the deployed worker is configured in `wrangler.toml`: + +```toml +[triggers] +crons = ["0 3 * * SUN"] +``` + +A Hermes-side reminder can retrieve the latest `_system.consolidation-report`, review stale/similar candidates, and ask before deleting or superseding anything. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md new file mode 100644 index 0000000..2d891dc --- /dev/null +++ b/docs/RELEASE_CHECKLIST.md @@ -0,0 +1,51 @@ +# Release Checklist + +Use this checklist before tagging or deploying a Recall release. + +## Code quality + +- [ ] `npm ci` +- [ ] `npm run typecheck` +- [ ] `npm test` +- [ ] Review `git diff` for accidental secrets, local `wrangler.toml` values, or `.recall-api-key` content. +- [ ] Confirm migrations are numbered, documented, and covered by tests when applicable. + +## Cloudflare deploy smoke + +- [ ] Apply required D1 migrations to a staging or test instance. +- [ ] `npx wrangler deploy` +- [ ] `curl https:///health` returns `{ "status": "ok" }` or equivalent minimal health JSON. +- [ ] Authenticated MCP `tools/list` returns the expected tool set. +- [ ] JSON-RPC retrieval smoke: `retrieve_memory` returns valid JSON for an empty or seeded query. +- [ ] JSON-RPC storage smoke: `store_memory` can store a test memory with namespace/provenance fields. +- [ ] Delete the test memory with `delete_memory`. + +## Tool surface + +- [ ] Core tools: `store_memory`, `retrieve_memory`, `list_memories`, `get_related_memories`, `delete_memory`, `clear_memories`, `consolidate_memories`. +- [ ] Lifecycle/provenance tools from the current release are present and documented. +- [ ] Destructive `clear_memories` remains gated by `ALLOW_DESTRUCTIVE_TOOLS=true` and `confirm: true`. +- [ ] Lifecycle validation rejects invalid statuses, confidence, dates, or supersession cycles. +- [ ] Consolidation remains read-only and does not delete or mutate memories automatically. + +## Updater checks + +- [ ] `bash -n recall-update.sh` +- [ ] `./recall-update.sh --check` runs without destructive mutation. +- [ ] `./recall-update.sh --doctor` does not print secrets. +- [ ] `./recall-update.sh --install-cron weekly` prints a cron line but does not install without `--yes`. +- [ ] `docs/UPDATING.md` describes migration and rollback behavior for this release. + +## Hermes integration checks + +- [ ] `docs/HERMES_INTEGRATION.md` namespace guidance is still current. +- [ ] Wrapper-script MCP config examples do not include a real API key. +- [ ] Discord provenance guidance reflects current source fields. +- [ ] README links to updating, Hermes integration, and release checklist docs. + +## Release notes + +- [ ] `CHANGELOG.md` has a top entry with Added/Changed/Fixed/Security/Upgrade notes as appropriate. +- [ ] README tool count and examples match the actual MCP tool surface. +- [ ] Tag name follows semver (`vX.Y.Z`). +- [ ] GitHub release notes include exact upgrade steps and migration commands. diff --git a/docs/UPDATING.md b/docs/UPDATING.md new file mode 100644 index 0000000..22641e8 --- /dev/null +++ b/docs/UPDATING.md @@ -0,0 +1,110 @@ +# Updating Recall + +Recall updates are intentionally conservative. The repository includes an optional helper, `recall-update.sh`, but you can always update manually with Git, Wrangler, and the release notes. + +## Configuration preservation contract + +Update tooling must preserve user-owned configuration and secrets. It must not regenerate, replace, or normalize: + +- `wrangler.toml` +- `.dev.vars` +- `.recall-api-key` +- MCP client config files such as `.mcp.json` or `claude_desktop_config.json` +- D1 database bindings, Vectorize bindings, route/custom-domain blocks, env blocks, or Wrangler secrets + +If an upstream release changes a protected config file, the updater refuses to apply it automatically. Review the diff manually and copy only the safe changes into your local config. + +## Optional updater script + +From the repository root: + +```bash +./recall-update.sh --check +./recall-update.sh --doctor +./recall-update.sh --apply +./recall-update.sh --rollback +./recall-update.sh --install-cron weekly +``` + +### `--check` + +Non-mutating. Fetches tags when possible, prints the current git revision/package version, reports the latest tag, and compares `migrations/*.sql` against the remote D1 `schema_migrations` table when Wrangler can query it. + +If D1 is unavailable, `--check` warns and continues. It does not deploy, install packages, write config, or apply migrations. + +### `--doctor` + +Best-effort diagnostics. Checks for Node, npm, Wrangler, `node_modules`, `wrangler.toml`, D1 database name, `MEMORY_API_KEY` secret presence by name only, and `/health` if a worker URL is detectable. + +`--doctor` does not intentionally mutate. If dependencies are missing, it tells you to run `npm ci` instead of installing them for you. + +### `--apply` + +Mutating and intentionally strict: + +1. Requires a clean git worktree. +2. Fetches tags. +3. Refuses to proceed if the target revision changes protected config/secrets files. +4. Records the current revision in `.recall-update/previous-revision`. +5. Runs `npm ci`. +6. Runs `npm run typecheck` and `npm test`. +7. Confirms `schema_migrations` is reachable, bootstrapping it for older installs when possible, then applies pending D1 migrations. +8. Runs `wrangler deploy`. +9. Requires smoke tests: `/health` plus authenticated MCP `tools/list`. Set `RECALL_WORKER_URL` if your worker URL cannot be detected from `wrangler.toml`; provide `MEMORY_API_KEY` or `.recall-api-key`. + +The updater reads `.recall-api-key` only for the authenticated smoke test and never prints the value. Logs redact authorization examples. + +### `--rollback` + +Checks out the revision recorded in `.recall-update/previous-revision`, redeploys the worker, and smoke-tests when possible. + +Rollback only affects worker code. D1 schema rollback is manual because SQLite/D1 migrations may be destructive or non-reversible. Before applying a risky release, export D1 if you need point-in-time database recovery: + +```bash +npx wrangler d1 export --remote --output recall-backup.sql +``` + +## Migration behavior + +Migrations live in `migrations/` and are tracked by the D1 table `schema_migrations`. + +- `--check` lists pending migrations when it can query D1. +- `--apply` runs pending numbered SQL files in order and refuses to continue unless `schema_migrations` is reachable. +- For older installs that predate migration tracking, `--apply` first creates `schema_migrations` and records already-applied historical migrations only when the live schema proves they exist. +- If D1 cannot be queried, fix D1 access or run/record migrations manually before rerunning `--apply`. +- Existing Docker/Postgres and local SQLite installs may have backend-specific migration behavior; check each migration comment header and release notes. + +Manual D1 migration example: + +```bash +npx wrangler d1 execute --remote --file=migrations/0004_provenance_lifecycle.sql +npx wrangler d1 execute --remote \ + --command "INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES ('0004', datetime('now'));" +``` + +## Auto-update opt-in + +Auto-update is not enabled by default. To install a weekly non-mutating check: + +```bash +./recall-update.sh --install-cron weekly +``` + +This prints the crontab line only. To actually add it to your user crontab: + +```bash +./recall-update.sh --install-cron weekly --yes +``` + +The cron job runs `--check`, not `--apply`, so it reports availability but does not deploy or migrate. + +## GitHub Actions notes + +An optional workflow skeleton lives at [`examples/github-actions-update.yml`](../examples/github-actions-update.yml). It is intentionally not installed under `.github/workflows/`. + +If you adapt it: + +- Store Cloudflare credentials as GitHub Actions secrets (`CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`). +- Do not put `MEMORY_API_KEY` in the repository. +- Keep update jobs manual or scheduled only after you are comfortable with your migration/rollback process. +- Prefer running `--check`, `npm run typecheck`, and `npm test` before any deploy step. diff --git a/examples/github-actions-update.yml b/examples/github-actions-update.yml new file mode 100644 index 0000000..5a06c85 --- /dev/null +++ b/examples/github-actions-update.yml @@ -0,0 +1,53 @@ +# Example only. Copy to .github/workflows/recall-update.yml if you want it. +# Requires repository secrets: +# CLOUDFLARE_API_TOKEN +# CLOUDFLARE_ACCOUNT_ID +# Do not store MEMORY_API_KEY in this workflow unless you need authenticated MCP smoke tests. + +name: Recall weekly update check + +on: + workflow_dispatch: + schedule: + - cron: "0 10 * * 1" + +jobs: + check-test-and-optional-apply: + runs-on: ubuntu-latest + permissions: + contents: read + env: + CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} + CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} + # Optional for smoke tests if your worker URL is not detectable from wrangler.toml. + # RECALL_WORKER_URL: https://your-worker.workers.dev + steps: + - name: Check out repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: 20 + cache: npm + + - name: Install dependencies + run: npm ci + + - name: Updater check + run: ./recall-update.sh --check + + - name: Typecheck + run: npm run typecheck + + - name: Test + run: npm test + + # Opt-in only. Uncomment after you are comfortable with automated deploys + # and after verifying your migrations/rollback process. + # + # - name: Apply update and deploy + # if: github.event_name == 'workflow_dispatch' + # run: ./recall-update.sh --apply --yes diff --git a/local/setup.sql b/local/setup.sql index 5e28d41..8d8d7f8 100644 --- a/local/setup.sql +++ b/local/setup.sql @@ -1,3 +1,16 @@ +CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + applied_at TEXT NOT NULL +); + +-- Fresh installs use this full current schema, so mark historical migrations +-- applied. Older installs still apply numbered migration files via the updater. +INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES + ('0000', datetime('now')), + ('0002', datetime('now')), + ('0003', datetime('now')), + ('0004', datetime('now')); + CREATE TABLE IF NOT EXISTS memories ( id TEXT PRIMARY KEY, key TEXT UNIQUE NOT NULL, @@ -10,7 +23,20 @@ CREATE TABLE IF NOT EXISTS memories ( created_at TEXT NOT NULL, updated_at TEXT NOT NULL, accessed_at TEXT NOT NULL, - access_count INTEGER NOT NULL DEFAULT 0 + access_count INTEGER NOT NULL DEFAULT 0, + source_type TEXT NOT NULL DEFAULT 'manual', + source_url TEXT, + source_path TEXT, + source_line_start INTEGER, + source_line_end INTEGER, + source_title TEXT, + source_hash TEXT, + status TEXT NOT NULL DEFAULT 'active', + confidence REAL NOT NULL DEFAULT 0.75, + verified_at TEXT, + expires_at TEXT, + supersedes_key TEXT, + superseded_by_key TEXT ); CREATE INDEX IF NOT EXISTS idx_memories_key ON memories(key); @@ -18,6 +44,10 @@ CREATE INDEX IF NOT EXISTS idx_memories_author ON memories(author); CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance); CREATE INDEX IF NOT EXISTS idx_memories_accessed_at ON memories(accessed_at); CREATE INDEX IF NOT EXISTS idx_memories_namespace ON memories(namespace); +CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status); +CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type); +CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at); +CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key); CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5( key, diff --git a/migrations/0000_schema_migrations.sql b/migrations/0000_schema_migrations.sql new file mode 100644 index 0000000..29cdd75 --- /dev/null +++ b/migrations/0000_schema_migrations.sql @@ -0,0 +1,4 @@ +CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + applied_at TEXT NOT NULL +); diff --git a/migrations/0004_provenance_lifecycle.sql b/migrations/0004_provenance_lifecycle.sql new file mode 100644 index 0000000..93dd45c --- /dev/null +++ b/migrations/0004_provenance_lifecycle.sql @@ -0,0 +1,30 @@ +-- Migration 0004: Provenance and lifecycle metadata +-- +-- Adds source/provenance fields plus lifecycle, verification, expiration, +-- and supersession metadata to memories. +-- +-- Run once against your live D1 database: +-- wrangler d1 execute recall --remote --file=migrations/0004_provenance_lifecycle.sql +-- +-- SQLite does not support ALTER TABLE ... ADD COLUMN IF NOT EXISTS. +-- Index creation is idempotent; ALTER TABLE will error if a column already +-- exists. Safe to run once on any database that has not had this migration applied. + +ALTER TABLE memories ADD COLUMN source_type TEXT NOT NULL DEFAULT 'manual'; +ALTER TABLE memories ADD COLUMN source_url TEXT; +ALTER TABLE memories ADD COLUMN source_path TEXT; +ALTER TABLE memories ADD COLUMN source_line_start INTEGER; +ALTER TABLE memories ADD COLUMN source_line_end INTEGER; +ALTER TABLE memories ADD COLUMN source_title TEXT; +ALTER TABLE memories ADD COLUMN source_hash TEXT; +ALTER TABLE memories ADD COLUMN status TEXT NOT NULL DEFAULT 'active'; +ALTER TABLE memories ADD COLUMN confidence REAL NOT NULL DEFAULT 0.75; +ALTER TABLE memories ADD COLUMN verified_at TEXT; +ALTER TABLE memories ADD COLUMN expires_at TEXT; +ALTER TABLE memories ADD COLUMN supersedes_key TEXT; +ALTER TABLE memories ADD COLUMN superseded_by_key TEXT; + +CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status); +CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type); +CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at); +CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key); diff --git a/package-lock.json b/package-lock.json index 658c1b7..48e504e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "recall-mcp", - "version": "2.0.0", + "version": "2.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "recall-mcp", - "version": "2.0.0", + "version": "2.2.0", "license": "MIT", "devDependencies": { "@cloudflare/workers-types": "^4.20260412.1", diff --git a/package.json b/package.json index 4696789..5d60684 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "recall-mcp", - "version": "2.1.0", + "version": "2.2.0", "description": "Self-hosted MCP memory server with hybrid semantic + keyword search, running on Cloudflare Workers", "license": "MIT", "author": "Cash Conway", diff --git a/recall-update.sh b/recall-update.sh new file mode 100755 index 0000000..d8cf1fc --- /dev/null +++ b/recall-update.sh @@ -0,0 +1,543 @@ +#!/usr/bin/env bash +set -Eeuo pipefail +IFS=$'\n\t' + +SCRIPT_NAME="$(basename "$0")" +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$ROOT_DIR" + +APPLY=0 +CHECK=0 +DOCTOR=0 +ROLLBACK=0 +INSTALL_CRON="" +YES=0 + +BACKUP_DIR=".recall-update" +SECURE_TMP_DIR="$(mktemp -d -t recall-update.XXXXXX)" +trap 'rm -rf "$SECURE_TMP_DIR"' EXIT +PREVIOUS_REVISION_FILE="$BACKUP_DIR/previous-revision" +PROTECTED_PATHS=( + "wrangler.toml" + ".dev.vars" + ".recall-api-key" + ".mcp.json" + "mcp.json" + "claude_desktop_config.json" +) + +log() { printf '%s\n' "$*"; } +warn() { printf 'warning: %s\n' "$*" >&2; } +err() { printf 'error: %s\n' "$*" >&2; } + +usage() { + cat <<'USAGE' +Recall optional updater (safe by default) + +Usage: + ./recall-update.sh --check + ./recall-update.sh --doctor + ./recall-update.sh --apply [--yes] + ./recall-update.sh --rollback + ./recall-update.sh --install-cron weekly [--yes] + +Safety contract: + - --check and --doctor do not intentionally mutate files, deployments, secrets, or databases. + - --apply refuses a dirty worktree, records the previous git revision, runs tests, applies only pending tracked D1 migrations, deploys, and smoke-tests when enough local config exists. + - --rollback checks out the recorded previous revision and redeploys. D1 schema rollback is manual. + - User-owned config/secrets are never regenerated or overwritten by this script: wrangler.toml, .dev.vars, .recall-api-key, MCP configs, D1/Vectorize/secrets/routes/env blocks. +USAGE +} + +require_cmd() { + if ! command -v "$1" >/dev/null 2>&1; then + err "missing required command: $1" + return 1 + fi +} + +run() { + log "+ $*" + "$@" +} + +npx_wrangler() { + if [[ -x node_modules/.bin/wrangler ]]; then + node_modules/.bin/wrangler "$@" + else + npx wrangler "$@" + fi +} + +current_revision() { + git rev-parse --short HEAD 2>/dev/null || printf 'unknown' +} + +current_ref() { + git describe --tags --always --dirty 2>/dev/null || current_revision +} + +fetch_tags_best_effort() { + if git remote get-url origin >/dev/null 2>&1; then + git fetch --tags origin >/dev/null 2>&1 || warn "could not fetch tags from origin; continuing with local tags" + else + warn "no git remote named origin; using local tags only" + fi +} + +latest_tag() { + git tag --sort=-v:refname 2>/dev/null | head -n1 || true +} + +latest_remote_target() { + local tag + tag="$(latest_tag)" + if [[ -n "$tag" ]]; then + printf '%s' "$tag" + return 0 + fi + if git rev-parse --verify origin/main >/dev/null 2>&1; then + printf '%s' "origin/main" + return 0 + fi + printf '%s' "HEAD" +} + +package_version() { + node -e "try{console.log(require('./package.json').version)}catch(e){process.exit(1)}" 2>/dev/null || printf 'unknown' +} + +parse_d1_db_name() { + [[ -f wrangler.toml ]] || return 1 + node <<'NODE' +const fs = require('fs'); +const text = fs.readFileSync('wrangler.toml', 'utf8'); +const dbBlock = text.match(/\[\[d1_databases\]\]([\s\S]*?)(?=\n\s*\[|\n\s*\[\[|$)/); +if (!dbBlock) process.exit(1); +const name = dbBlock[1].match(/^\s*database_name\s*=\s*["']([^"']+)["']/m); +if (!name) process.exit(1); +console.log(name[1]); +NODE +} + +parse_worker_url() { + if [[ -n "${RECALL_WORKER_URL:-}" ]]; then + printf '%s' "${RECALL_WORKER_URL%/}" + return 0 + fi + [[ -f wrangler.toml ]] || return 1 + node <<'NODE' +const fs = require('fs'); +const text = fs.readFileSync('wrangler.toml', 'utf8'); +const route = text.match(/^\s*pattern\s*=\s*["']([^"']+)["']/m); +if (route) { + let p = route[1].replace(/\/\*$/, ''); + if (!p.startsWith('http')) p = 'https://' + p; + console.log(p); + process.exit(0); +} +process.exit(1); +NODE +} + +migration_files() { + if [[ -d migrations ]]; then + find migrations -maxdepth 1 -type f -name '[0-9][0-9][0-9][0-9]_*.sql' | sort + fi +} + +query_applied_migrations() { + local db_name="$1" + npx_wrangler d1 execute "$db_name" --remote --command "SELECT version FROM schema_migrations ORDER BY version;" 2>"$SECURE_TMP_DIR/d1.err" \ + | sed -nE 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/p' +} + +pending_migrations() { + local db_name applied tmp file base version + db_name="${1:-}" + tmp="$(mktemp "$SECURE_TMP_DIR/pending.XXXXXX")" + if [[ -n "$db_name" ]] && query_applied_migrations "$db_name" >"$tmp"; then + : + else + rm -f "$tmp" + return 2 + fi + while IFS= read -r file; do + base="$(basename "$file")" + version="${base%%_*}" + if ! grep -qxF "$version" "$tmp"; then + printf '%s\n' "$file" + fi + done < <(migration_files) + rm -f "$tmp" +} + +remote_query_has_row() { + local db_name="$1" + local command="$2" + npx_wrangler d1 execute "$db_name" --remote --command "$command" 2>/dev/null | grep -q '"found"[[:space:]]*:[[:space:]]*1' +} + +bootstrap_schema_migrations() { + local db_name="$1" + if [[ ! -f migrations/0000_schema_migrations.sql ]]; then + err "missing migrations/0000_schema_migrations.sql; cannot bootstrap migration tracking" + return 1 + fi + warn "schema_migrations is not reachable; bootstrapping migration tracking" + run npx_wrangler d1 execute "$db_name" --remote --file=migrations/0000_schema_migrations.sql + run npx_wrangler d1 execute "$db_name" --remote --command "INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES ('0000', datetime('now'));" + + # Existing installs before migration tracking may already have 0002/0003 applied. + # Record only what the live schema proves, then let pending_migrations apply the rest. + if remote_query_has_row "$db_name" "SELECT 1 AS found FROM pragma_table_info('memories') WHERE name = 'memory_type' LIMIT 1;" && \ + remote_query_has_row "$db_name" "SELECT 1 AS found FROM sqlite_master WHERE type = 'table' AND name = 'memory_relationships' LIMIT 1;"; then + run npx_wrangler d1 execute "$db_name" --remote --command "INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES ('0002', datetime('now'));" + fi + if remote_query_has_row "$db_name" "SELECT 1 AS found FROM pragma_table_info('memories') WHERE name = 'namespace' LIMIT 1;"; then + run npx_wrangler d1 execute "$db_name" --remote --command "INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES ('0003', datetime('now'));" + fi +} + +print_migration_status() { + local db_name pending_status + db_name="$(parse_d1_db_name 2>/dev/null || true)" + if [[ -z "$db_name" ]]; then + warn "could not detect D1 database_name from wrangler.toml; skipping remote migration status" + log "Migration files present:" + migration_files | sed 's/^/ /' || true + return 0 + fi + log "D1 database: $db_name" + log "Pending migrations:" + set +e + local pending + pending="$(pending_migrations "$db_name")" + pending_status=$? + set -e + if [[ $pending_status -eq 0 ]]; then + if [[ -n "$pending" ]]; then + printf '%s\n' "$pending" | sed 's/^/ /' + else + log " none" + fi + else + warn "could not query D1 schema_migrations; continuing without remote migration status" + if [[ -s "$SECURE_TMP_DIR/d1.err" ]]; then + sed 's/^/ wrangler: /' "$SECURE_TMP_DIR/d1.err" >&2 || true + fi + log "Migration files present:" + migration_files | sed 's/^/ /' || true + fi +} + +check_clean_worktree() { + if [[ -n "$(git status --porcelain=v1)" ]]; then + err "worktree is dirty; commit/stash changes before --apply/--rollback" + git status --short + return 1 + fi +} + +ensure_target_does_not_change_protected_paths() { + local target="$1" p + for p in "${PROTECTED_PATHS[@]}"; do + if git cat-file -e "HEAD:$p" 2>/dev/null || git cat-file -e "$target:$p" 2>/dev/null; then + if ! git diff --quiet "HEAD" "$target" -- "$p" 2>/dev/null; then + err "refusing update: target $target changes protected user-owned config path '$p'" + err "update manually and preserve local D1/Vectorize/secrets/routes/env settings" + return 1 + fi + fi + done +} + +backup_state() { + mkdir -p "$BACKUP_DIR/protected" + git rev-parse HEAD >"$PREVIOUS_REVISION_FILE" + local p dest + for p in "${PROTECTED_PATHS[@]}"; do + if [[ -e "$p" ]]; then + dest="$BACKUP_DIR/protected/$p" + mkdir -p "$(dirname "$dest")" + cp -p "$p" "$dest" + fi + done + chmod 700 "$BACKUP_DIR" || true + log "Recorded previous revision in $PREVIOUS_REVISION_FILE" +} + +run_check() { + require_cmd git + require_cmd node || true + log "Recall updater check (no mutations)" + log "Repository: $ROOT_DIR" + log "Current git ref: $(current_ref) ($(current_revision))" + log "Package version: $(package_version)" + fetch_tags_best_effort + local tag + tag="$(latest_tag)" + if [[ -n "$tag" ]]; then + log "Latest local/fetched tag: $tag" + else + warn "no git tags found" + fi + print_migration_status +} + +run_doctor() { + log "Recall updater doctor (best-effort, no intentional mutations)" + require_cmd git || true + require_cmd node || true + if command -v npm >/dev/null 2>&1; then + log "npm: $(npm --version)" + else + warn "npm not found" + fi + if [[ -x node_modules/.bin/wrangler ]]; then + log "wrangler: $(node_modules/.bin/wrangler --version 2>/dev/null || true)" + elif command -v npx >/dev/null 2>&1; then + log "wrangler: using npx wrangler (install dependencies first if this prompts)" + else + warn "wrangler unavailable: no node_modules/.bin/wrangler and no npx" + fi + if [[ -d node_modules ]]; then + log "node_modules: present" + else + warn "node_modules missing; run 'npm ci' before typecheck/test/deploy" + fi + if [[ -f package-lock.json ]]; then + log "package-lock.json: present" + else + warn "package-lock.json missing; npm ci will not work" + fi + if [[ -f wrangler.toml ]]; then + log "wrangler.toml: present" + local db + db="$(parse_d1_db_name 2>/dev/null || true)" + [[ -n "$db" ]] && log "D1 database_name detected: $db" || warn "could not detect D1 database_name" + else + warn "wrangler.toml missing; copy wrangler.toml.example and fill bindings before deploy" + fi + if [[ -f .dev.vars ]]; then + log ".dev.vars: present (not read)" + fi + if [[ -f .recall-api-key ]]; then + log ".recall-api-key: present (value not printed)" + else + warn ".recall-api-key missing; MCP smoke tests will be skipped unless MEMORY_API_KEY is exported" + fi + if [[ -f wrangler.toml ]] && ( [[ -x node_modules/.bin/wrangler ]] || command -v npx >/dev/null 2>&1 ); then + if npx_wrangler secret list 2>"$SECURE_TMP_DIR/secret-list.err" | grep -q 'MEMORY_API_KEY'; then + log "Cloudflare secret MEMORY_API_KEY: present (name only)" + else + warn "could not confirm MEMORY_API_KEY via wrangler secret list (not logged in, no permissions, or secret absent)" + fi + fi + local url + url="$(parse_worker_url 2>/dev/null || true)" + if [[ -n "$url" ]]; then + log "Health URL detected: ${url%/}/health" + if command -v curl >/dev/null 2>&1; then + curl -fsS --max-time 10 "${url%/}/health" >"$SECURE_TMP_DIR/health.json" \ + && log "/health: reachable" \ + || warn "/health check failed" + rm -f "$SECURE_TMP_DIR/health.json" + fi + else + warn "worker URL not detectable; set RECALL_WORKER_URL=https://your-worker.example.com for health checks" + fi +} + +apply_pending_migrations() { + local db_name pending file base version + db_name="$(parse_d1_db_name 2>/dev/null || true)" + if [[ -z "$db_name" ]]; then + err "no D1 database_name detected; cannot apply required migrations" + return 1 + fi + set +e + pending="$(pending_migrations "$db_name")" + local status=$? + set -e + if [[ $status -ne 0 ]]; then + bootstrap_schema_migrations "$db_name" + set +e + pending="$(pending_migrations "$db_name")" + status=$? + set -e + if [[ $status -ne 0 ]]; then + err "schema_migrations is not reachable after bootstrap; refusing --apply because migrations are required to be checked/applied" + err "Run migrations manually or fix D1 access, then rerun --apply." + return 1 + fi + fi + if [[ -z "$pending" ]]; then + log "No pending D1 migrations" + return 0 + fi + while IFS= read -r file; do + [[ -n "$file" ]] || continue + base="$(basename "$file")" + version="${base%%_*}" + run npx_wrangler d1 execute "$db_name" --remote --file="$file" + run npx_wrangler d1 execute "$db_name" --remote --command "INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES ('$version', datetime('now'));" + done <<<"$pending" +} + +smoke_test() { + local mode="${1:-required}" + local url key auth_url + url="$(parse_worker_url 2>/dev/null || true)" + if [[ -z "$url" ]]; then + if [[ "$mode" == "optional" ]]; then + warn "worker URL not detectable; skipping optional smoke tests (set RECALL_WORKER_URL)" + return 0 + fi + err "worker URL not detectable; set RECALL_WORKER_URL=https://your-worker.example.com so --apply can run required smoke tests" + return 1 + fi + if [[ "$mode" == "optional" ]]; then + run curl -fsS --max-time 10 "${url%/}/health" >/dev/null || { warn "optional health check failed"; return 0; } + else + run curl -fsS --max-time 10 "${url%/}/health" >/dev/null + fi + if [[ -n "${MEMORY_API_KEY:-}" ]]; then + key="$MEMORY_API_KEY" + elif [[ -f .recall-api-key ]]; then + key="$(tr -d '\r\n' < .recall-api-key)" + else + if [[ "$mode" == "optional" ]]; then + warn "no MEMORY_API_KEY env or .recall-api-key; skipping optional authenticated MCP tools/list smoke test" + return 0 + fi + err "no MEMORY_API_KEY env or .recall-api-key; authenticated MCP tools/list smoke test is required for --apply" + return 1 + fi + auth_url="${url%/}/mcp" + log "+ curl -fsS --max-time 15 -H 'Authorization: Bearer ' -H 'Content-Type: application/json' -d '{...tools/list...}' '$auth_url'" + curl -fsS --max-time 15 \ + -H "Authorization: Bearer ${key}" \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \ + "$auth_url" >"$SECURE_TMP_DIR/tools.json" + RECALL_UPDATE_TOOLS_JSON="$SECURE_TMP_DIR/tools.json" node -e "const fs=require('fs'); const j=JSON.parse(fs.readFileSync(process.env.RECALL_UPDATE_TOOLS_JSON,'utf8')); if (j.error) throw new Error(JSON.stringify(j.error)); const n=j.result?.tools?.length; if (!n) throw new Error('tools/list returned no tools'); console.log('tools/list:', n, 'tools');" + rm -f "$SECURE_TMP_DIR/tools.json" +} +run_apply() { + require_cmd git + require_cmd node + require_cmd npm + check_clean_worktree + fetch_tags_best_effort + local target + target="$(latest_remote_target)" + log "Apply target: $target" + if [[ "$target" == "HEAD" ]]; then + warn "no newer tag or origin/main available; staying on current revision" + fi + ensure_target_does_not_change_protected_paths "$target" + backup_state + if [[ "$target" != "HEAD" ]]; then + run git checkout "$target" + fi + run npm ci + run npm run typecheck + run npm test + apply_pending_migrations + run npx_wrangler deploy + smoke_test + log "Update apply completed. Protected config files were not regenerated or overwritten." +} + +run_rollback() { + require_cmd git + require_cmd node + require_cmd npm || true + if [[ ! -f "$PREVIOUS_REVISION_FILE" ]]; then + err "no previous revision recorded at $PREVIOUS_REVISION_FILE" + return 1 + fi + check_clean_worktree + local prev + prev="$(cat "$PREVIOUS_REVISION_FILE")" + [[ -n "$prev" ]] || { err "previous revision file is empty"; return 1; } + warn "D1 schema rollback is manual; this only rolls back worker code." + ensure_target_does_not_change_protected_paths "$prev" + run git checkout "$prev" + if [[ -d node_modules ]]; then + run npm run typecheck + run npm test + else + warn "node_modules missing; skipping typecheck/test during rollback" + fi + run npx_wrangler deploy + smoke_test optional + log "Rollback deploy completed. Review database schema manually if migrations were applied." +} + +install_cron_weekly() { + local line escaped existing tmp + line="0 4 * * 1 cd '$ROOT_DIR' && ./recall-update.sh --check >> '$ROOT_DIR/$BACKUP_DIR/cron.log' 2>&1 # recall-update weekly check" + log "Suggested weekly cron line (non-mutating --check):" + log "$line" + if [[ $YES -ne 1 ]]; then + warn "not installing cron without --yes" + return 0 + fi + require_cmd crontab + mkdir -p "$BACKUP_DIR" + tmp="$(mktemp "$SECURE_TMP_DIR/pending.XXXXXX")" + crontab -l >"$tmp" 2>/dev/null || true + if grep -Fq "# recall-update weekly check" "$tmp"; then + warn "recall-update weekly cron line already present; leaving crontab unchanged" + rm -f "$tmp" + return 0 + fi + printf '%s\n' "$line" >>"$tmp" + crontab "$tmp" + rm -f "$tmp" + log "Installed weekly recall-update --check cron line." +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --check) CHECK=1 ;; + --doctor) DOCTOR=1 ;; + --apply) APPLY=1 ;; + --rollback) ROLLBACK=1 ;; + --install-cron) + shift + INSTALL_CRON="${1:-}" + if [[ "$INSTALL_CRON" != "weekly" ]]; then + err "--install-cron currently supports only: weekly" + exit 2 + fi + ;; + --yes|-y) YES=1 ;; + --help|-h) usage; exit 0 ;; + *) err "unknown argument: $1"; usage; exit 2 ;; + esac + shift +done + +selected=$((CHECK + DOCTOR + APPLY + ROLLBACK)) +if [[ -n "$INSTALL_CRON" ]]; then + selected=$((selected + 1)) +fi +if [[ $selected -ne 1 ]]; then + err "choose exactly one command" + usage + exit 2 +fi + +case 1 in + $CHECK) run_check ;; + $DOCTOR) run_doctor ;; + $APPLY) run_apply ;; + $ROLLBACK) run_rollback ;; +esac + +if [[ -n "$INSTALL_CRON" ]]; then + install_cron_weekly +fi diff --git a/schema.sql b/schema.sql index d951e4e..ea23b7e 100644 --- a/schema.sql +++ b/schema.sql @@ -1,5 +1,18 @@ -- Recall: D1 schema for memory storage + FTS5 keyword search +CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + applied_at TEXT NOT NULL +); + +-- Fresh installs use this full current schema, so mark historical migrations +-- applied. Older installs still apply numbered migration files via the updater. +INSERT OR IGNORE INTO schema_migrations(version, applied_at) VALUES + ('0000', datetime('now')), + ('0002', datetime('now')), + ('0003', datetime('now')), + ('0004', datetime('now')); + CREATE TABLE IF NOT EXISTS memories ( id TEXT PRIMARY KEY, key TEXT UNIQUE NOT NULL, @@ -12,7 +25,20 @@ CREATE TABLE IF NOT EXISTS memories ( created_at TEXT NOT NULL, updated_at TEXT NOT NULL, accessed_at TEXT NOT NULL, - access_count INTEGER NOT NULL DEFAULT 0 + access_count INTEGER NOT NULL DEFAULT 0, + source_type TEXT NOT NULL DEFAULT 'manual', + source_url TEXT, + source_path TEXT, + source_line_start INTEGER, + source_line_end INTEGER, + source_title TEXT, + source_hash TEXT, + status TEXT NOT NULL DEFAULT 'active', + confidence REAL NOT NULL DEFAULT 0.75, + verified_at TEXT, + expires_at TEXT, + supersedes_key TEXT, + superseded_by_key TEXT ); CREATE INDEX IF NOT EXISTS idx_memories_key ON memories(key); @@ -20,6 +46,10 @@ CREATE INDEX IF NOT EXISTS idx_memories_author ON memories(author); CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance); CREATE INDEX IF NOT EXISTS idx_memories_accessed_at ON memories(accessed_at); CREATE INDEX IF NOT EXISTS idx_memories_namespace ON memories(namespace); +CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status); +CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type); +CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at); +CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key); -- FTS5 virtual table for keyword/BM25 search (hybrid search with Vectorize) CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts USING fts5( diff --git a/src/tools.ts b/src/tools.ts index 04052ab..165093e 100644 --- a/src/tools.ts +++ b/src/tools.ts @@ -2,9 +2,15 @@ import type { Memory, MemoryRow, MemoryType, + MemorySourceType, + MemoryStatus, + RetrieveFormat, McpToolDefinition, McpToolResult, StoreMemoryInput, + MarkMemoryStatusInput, + VerifyMemoryInput, + SupersedeMemoryInput, RetrieveMemoryInput, ListMemoriesInput, DeleteMemoryInput, @@ -62,10 +68,103 @@ export const TOOL_DEFINITIONS: McpToolDefinition[] = [ description: 'Optional project/scope identifier. Memories with a namespace are only returned when retrieve_memory/list_memories filter by the same namespace. Use to isolate memories across projects. Format: alphanumeric, hyphens, underscores, dots.', }, + source_type: { + type: 'string', + enum: ['manual', 'chat', 'doc', 'code', 'issue', 'pull_request', 'log', 'web', 'inferred'], + description: 'Provenance source type for this memory (default: manual).', + }, + source_url: { type: 'string', description: 'Optional source URL for provenance.' }, + source_path: { type: 'string', description: 'Optional source file/path for provenance.' }, + source_line_start: { type: 'integer', minimum: 1, description: 'Optional starting line number in the source.' }, + source_line_end: { type: 'integer', minimum: 1, description: 'Optional ending line number in the source.' }, + source_title: { type: 'string', description: 'Optional human-readable source title.' }, + source_hash: { type: 'string', description: 'Optional content/source hash for provenance.' }, + status: { + type: 'string', + enum: ['active', 'stale', 'superseded', 'deprecated'], + description: 'Lifecycle status for this memory (default: active).', + }, + confidence: { + type: 'number', + minimum: 0, + maximum: 1, + description: 'Confidence score for this memory, 0.0–1.0 (default: 0.75).', + }, + verified_at: { type: 'string', description: 'Optional ISO 8601 timestamp when this memory was verified.' }, + expires_at: { type: 'string', description: 'Optional ISO 8601 timestamp when this memory should expire.' }, + supersedes: { type: 'string', description: 'Optional key of a memory this new memory supersedes.' }, }, required: ['key', 'content', 'author'], }, }, + { + name: 'mark_memory_status', + description: 'Update the lifecycle status of an existing memory without deleting its search indexes. D1 is canonical; vector metadata may remain stale until reindex.', + inputSchema: { + type: 'object', + properties: { + key: { type: 'string', description: 'Memory key to update' }, + status: { + type: 'string', + enum: ['active', 'stale', 'superseded', 'deprecated'], + description: 'New lifecycle status for the memory.', + }, + reason: { type: 'string', description: 'Optional human-readable reason for the status change.' }, + }, + required: ['key', 'status'], + }, + }, + { + name: 'verify_memory', + description: 'Verify an existing memory and update provided provenance, confidence, expiration, and lifecycle fields while preserving unprovided fields.', + inputSchema: { + type: 'object', + properties: { + key: { type: 'string', description: 'Memory key to verify' }, + confidence: { type: 'number', minimum: 0, maximum: 1, description: 'Optional confidence score, 0.0–1.0.' }, + source_type: { type: 'string', enum: ['manual', 'chat', 'doc', 'code', 'issue', 'pull_request', 'log', 'web', 'inferred'] }, + source_url: { type: 'string' }, + source_path: { type: 'string' }, + source_line_start: { type: 'integer', minimum: 1 }, + source_line_end: { type: 'integer', minimum: 1 }, + source_title: { type: 'string' }, + source_hash: { type: 'string' }, + verified_at: { type: 'string', description: 'ISO timestamp, or "auto-now" / omitted to use the current time.' }, + expires_at: { type: 'string', description: 'Optional ISO expiration timestamp.' }, + status: { type: 'string', enum: ['active', 'stale', 'superseded', 'deprecated'], description: 'Lifecycle status (default active).' }, + }, + required: ['key'], + }, + }, + { + name: 'supersede_memory', + description: 'Create a replacement memory, mark the old memory superseded, and add supersession relationship edges.', + inputSchema: { + type: 'object', + properties: { + old_key: { type: 'string', description: 'Existing memory key being replaced.' }, + new_key: { type: 'string', description: 'New memory key to create.' }, + content: { type: 'string', description: 'Replacement memory content.' }, + author: { type: 'string', description: 'Who created the replacement memory.' }, + reason: { type: 'string', description: 'Optional reason for supersession.' }, + tags: { type: 'array', items: { type: 'string' } }, + importance: { type: 'number', minimum: 0, maximum: 1 }, + memory_type: { type: 'string', enum: ['episodic', 'semantic', 'procedural'] }, + namespace: { type: 'string' }, + source_type: { type: 'string', enum: ['manual', 'chat', 'doc', 'code', 'issue', 'pull_request', 'log', 'web', 'inferred'] }, + source_url: { type: 'string' }, + source_path: { type: 'string' }, + source_line_start: { type: 'integer', minimum: 1 }, + source_line_end: { type: 'integer', minimum: 1 }, + source_title: { type: 'string' }, + source_hash: { type: 'string' }, + confidence: { type: 'number', minimum: 0, maximum: 1 }, + verified_at: { type: 'string' }, + expires_at: { type: 'string' }, + }, + required: ['old_key', 'new_key', 'content', 'author'], + }, + }, { name: 'retrieve_memory', description: @@ -99,6 +198,20 @@ export const TOOL_DEFINITIONS: McpToolDefinition[] = [ description: 'Filter to memories stored in this namespace. Omit to search across all memories (including unnamespaced ones). When set, unnamespaced memories are NOT returned.', }, + include_statuses: { + type: 'array', + items: { type: 'string', enum: ['active', 'stale', 'superseded', 'deprecated'] }, + description: 'Lifecycle statuses to include in retrieval (default: active and stale).', + }, + include_provenance: { + type: 'boolean', + description: 'Whether to include provenance metadata in retrieval text output (default: true).', + }, + format: { + type: 'string', + enum: ['text', 'json'], + description: 'Retrieval output format (default: text).', + }, }, required: ['query'], }, @@ -127,6 +240,16 @@ export const TOOL_DEFINITIONS: McpToolDefinition[] = [ minimum: 0, description: 'Number of results to skip (default 0)', }, + status: { + type: 'string', + enum: ['active', 'stale', 'superseded', 'deprecated'], + description: 'Filter by lifecycle status.', + }, + source_type: { + type: 'string', + enum: ['manual', 'chat', 'doc', 'code', 'issue', 'pull_request', 'log', 'web', 'inferred'], + description: 'Filter by provenance source type.', + }, }, }, }, @@ -159,7 +282,7 @@ export const TOOL_DEFINITIONS: McpToolDefinition[] = [ { name: 'consolidate_memories', description: - 'Analyze the memory store for consolidation opportunities. Finds similar memory pairs that may be candidates for merging and stale memories that are never accessed. This is a READ-ONLY operation — no memories are modified. Returns a report with recommendations that you can act on using store_memory and delete_memory.', + 'Analyze the memory store for consolidation and lifecycle maintenance opportunities. Finds similar pairs, stale/expired/unverified/low-confidence memories, provenance gaps, superseded-but-accessed entries, and deprecated deletion candidates. This is a READ-ONLY operation — no memories are modified. Returns a report with recommendations that you can act on using mark_memory_status, verify_memory, supersede_memory, store_memory, and delete_memory.', inputSchema: { type: 'object', properties: { @@ -225,8 +348,39 @@ const MAX_TAGS = 20; const MAX_QUERY_LEN = 1000; const VALID_MEMORY_TYPES = new Set(['episodic', 'semantic', 'procedural']); +export const VALID_SOURCE_TYPES = new Set([ + 'manual', + 'chat', + 'doc', + 'code', + 'issue', + 'pull_request', + 'log', + 'web', + 'inferred', +]); +export const VALID_MEMORY_STATUSES = new Set([ + 'active', + 'stale', + 'superseded', + 'deprecated', +]); +const VALID_RETRIEVE_FORMATS = new Set(['text', 'json']); const MAX_NAMESPACE_LEN = 128; +const MAX_SOURCE_FIELD_LEN = 2048; +const MAX_SOURCE_TITLE_LEN = 512; +const MAX_REASON_LEN = 2048; + +function validateRequiredKeyRef(raw: unknown, field: string): string { + const key = validateOptionalKeyRef(raw, field); + if (key === undefined) throw new Error(`${field} must be a non-empty string (max ${MAX_KEY_LEN} chars)`); + return key; +} + +function hasOwn(args: Record, key: string): boolean { + return Object.prototype.hasOwnProperty.call(args, key); +} function validateNamespace(raw: unknown): string | null { if (raw === undefined || raw === null || raw === '') return null; @@ -241,7 +395,81 @@ function validateNamespace(raw: unknown): string | null { return trimmed; } -function validateStoreInput(args: Record): StoreMemoryInput { +export function validateSourceType(raw: unknown, field = 'source_type'): MemorySourceType { + if (typeof raw !== 'string' || !VALID_SOURCE_TYPES.has(raw as MemorySourceType)) { + throw new Error(`${field} must be one of: ${[...VALID_SOURCE_TYPES].join(', ')}`); + } + return raw as MemorySourceType; +} + +export function validateMemoryStatus(raw: unknown, field = 'status'): MemoryStatus { + if (typeof raw !== 'string' || !VALID_MEMORY_STATUSES.has(raw as MemoryStatus)) { + throw new Error(`${field} must be one of: ${[...VALID_MEMORY_STATUSES].join(', ')}`); + } + return raw as MemoryStatus; +} + +export function validateConfidence(raw: unknown): number { + if (typeof raw !== 'number' || !Number.isFinite(raw) || raw < 0 || raw > 1) { + throw new Error('confidence must be a number between 0.0 and 1.0'); + } + return Math.round(raw * 100) / 100; +} + +export function validateIsoDateString(raw: unknown, field: string): string | null { + if (raw === undefined || raw === null || raw === '') return null; + if (typeof raw !== 'string') throw new Error(`${field} must be an ISO 8601 date string`); + const trimmed = raw.trim(); + const parsed = Date.parse(trimmed); + if (!trimmed || Number.isNaN(parsed)) throw new Error(`${field} must be an ISO 8601 date string`); + return trimmed; +} + +function validateNullableString(raw: unknown, field: string, maxLen = MAX_SOURCE_FIELD_LEN): string | null { + if (raw === undefined || raw === null || raw === '') return null; + if (typeof raw !== 'string' || raw.length > maxLen) { + throw new Error(`${field} must be a string (max ${maxLen} chars)`); + } + const trimmed = raw.trim(); + return trimmed || null; +} + +function validateOptionalLine(raw: unknown, field: string): number | null { + if (raw === undefined || raw === null || raw === '') return null; + if (typeof raw !== 'number' || !Number.isInteger(raw) || raw < 1) { + throw new Error(`${field} must be a positive integer`); + } + return raw; +} + +export function validateOptionalLineRange(args: Record): { + source_line_start: number | null; + source_line_end: number | null; +} { + const source_line_start = validateOptionalLine(args.source_line_start, 'source_line_start'); + const source_line_end = validateOptionalLine(args.source_line_end, 'source_line_end'); + if (source_line_start === null && source_line_end !== null) { + throw new Error('source_line_start must be provided if source_line_end is specified'); + } + if (source_line_start !== null && source_line_end !== null && source_line_start > source_line_end) { + throw new Error('source_line_start must be less than or equal to source_line_end'); + } + return { source_line_start, source_line_end }; +} + +export function validateOptionalKeyRef(raw: unknown, field: string): string | undefined { + if (raw === undefined || raw === null || raw === '') return undefined; + if (typeof raw !== 'string' || raw.length === 0 || raw.length > MAX_KEY_LEN) { + throw new Error(`${field} must be a non-empty string (max ${MAX_KEY_LEN} chars)`); + } + const trimmed = raw.trim(); + if (!KEY_PATTERN.test(trimmed)) { + throw new Error(`${field} must contain only alphanumeric characters, hyphens, underscores, and dots`); + } + return trimmed; +} + +export function validateStoreInput(args: Record): StoreMemoryInput { const { key, content, tags, importance, author, memory_type, namespace } = args; if (typeof key !== 'string' || key.length === 0 || key.length > MAX_KEY_LEN) { @@ -287,7 +515,10 @@ function validateStoreInput(args: Record): StoreMemoryInput { validatedMemoryType = memory_type as MemoryType; } - return { + const lineRange = validateOptionalLineRange(args); + const supersedes = validateOptionalKeyRef(args.supersedes, 'supersedes'); + + const result: StoreMemoryInput = { key: key.trim(), content: content.trim(), tags: validatedTags, @@ -295,10 +526,73 @@ function validateStoreInput(args: Record): StoreMemoryInput { author: author.trim(), memory_type: validatedMemoryType, namespace: validateNamespace(namespace), + source_type: args.source_type === undefined ? 'manual' : validateSourceType(args.source_type), + source_url: validateNullableString(args.source_url, 'source_url'), + source_path: validateNullableString(args.source_path, 'source_path'), + source_line_start: lineRange.source_line_start, + source_line_end: lineRange.source_line_end, + source_title: validateNullableString(args.source_title, 'source_title', MAX_SOURCE_TITLE_LEN), + source_hash: validateNullableString(args.source_hash, 'source_hash'), + status: args.status === undefined ? 'active' : validateMemoryStatus(args.status), + confidence: args.confidence === undefined ? 0.75 : validateConfidence(args.confidence), + verified_at: validateIsoDateString(args.verified_at, 'verified_at'), + expires_at: validateIsoDateString(args.expires_at, 'expires_at'), }; + + if (supersedes !== undefined) result.supersedes = supersedes; + return result; } -function validateRetrieveInput(args: Record): RetrieveMemoryInput { +export function validateMarkMemoryStatusInput(args: Record): MarkMemoryStatusInput { + const result: MarkMemoryStatusInput = { + key: validateRequiredKeyRef(args.key, 'key'), + status: validateMemoryStatus(args.status), + }; + if (args.reason !== undefined) { + result.reason = validateNullableString(args.reason, 'reason', MAX_REASON_LEN) ?? undefined; + } + return result; +} + +export function validateVerifyMemoryInput(args: Record): VerifyMemoryInput { + const result: VerifyMemoryInput = { + key: validateRequiredKeyRef(args.key, 'key'), + verified_at: new Date().toISOString(), + }; + + if (args.verified_at !== undefined && args.verified_at !== 'auto-now') { + const verified = validateIsoDateString(args.verified_at, 'verified_at'); + if (verified === null) throw new Error('verified_at must be an ISO 8601 date string or "auto-now"'); + result.verified_at = verified; + } + if (hasOwn(args, 'confidence')) result.confidence = validateConfidence(args.confidence); + if (hasOwn(args, 'source_type')) result.source_type = validateSourceType(args.source_type); + if (hasOwn(args, 'source_url')) result.source_url = validateNullableString(args.source_url, 'source_url'); + if (hasOwn(args, 'source_path')) result.source_path = validateNullableString(args.source_path, 'source_path'); + const lineRange = validateOptionalLineRange(args); + if (hasOwn(args, 'source_line_start')) result.source_line_start = lineRange.source_line_start; + if (hasOwn(args, 'source_line_end')) result.source_line_end = lineRange.source_line_end; + if (hasOwn(args, 'source_title')) result.source_title = validateNullableString(args.source_title, 'source_title', MAX_SOURCE_TITLE_LEN); + if (hasOwn(args, 'source_hash')) result.source_hash = validateNullableString(args.source_hash, 'source_hash'); + if (hasOwn(args, 'expires_at')) result.expires_at = validateIsoDateString(args.expires_at, 'expires_at'); + if (hasOwn(args, 'status')) result.status = validateMemoryStatus(args.status); + + return result; +} + +export function validateSupersedeMemoryInput(args: Record): SupersedeMemoryInput { + const old_key = validateRequiredKeyRef(args.old_key, 'old_key'); + const new_key = validateRequiredKeyRef(args.new_key, 'new_key'); + if (old_key === new_key) throw new Error('new_key must differ from old_key'); + const storeInput = validateStoreInput({ ...args, key: new_key, supersedes: old_key, status: 'active' }); + const result: SupersedeMemoryInput = { ...storeInput, old_key, new_key }; + if (args.reason !== undefined) { + result.reason = validateNullableString(args.reason, 'reason', MAX_REASON_LEN) ?? undefined; + } + return result; +} + +export function validateRetrieveInput(args: Record): RetrieveMemoryInput { const { query, limit, min_importance, tags, namespace } = args; if (typeof query !== 'string' || query.length === 0 || query.length > MAX_QUERY_LEN) { @@ -327,13 +621,32 @@ function validateRetrieveInput(args: Record): RetrieveMemoryInp }); } + if (args.include_statuses !== undefined) { + if (!Array.isArray(args.include_statuses)) { + throw new Error('include_statuses must be an array of memory statuses'); + } + result.include_statuses = args.include_statuses.map((s) => validateMemoryStatus(s, 'include_statuses')); + } + if (args.include_provenance !== undefined) { + if (typeof args.include_provenance !== 'boolean') { + throw new Error('include_provenance must be a boolean'); + } + result.include_provenance = args.include_provenance; + } + if (args.format !== undefined) { + if (typeof args.format !== 'string' || !VALID_RETRIEVE_FORMATS.has(args.format as RetrieveFormat)) { + throw new Error('format must be "text" or "json"'); + } + result.format = args.format as RetrieveFormat; + } + const ns = validateNamespace(namespace); if (ns) result.namespace = ns; return result; } -function validateListInput(args: Record): ListMemoriesInput { +export function validateListInput(args: Record): ListMemoriesInput { const result: ListMemoriesInput = {}; if (args.tag !== undefined) { if (typeof args.tag !== 'string') throw new Error('tag must be a string'); @@ -355,6 +668,12 @@ function validateListInput(args: Record): ListMemoriesInput { } result.offset = args.offset; } + if (args.status !== undefined) { + result.status = validateMemoryStatus(args.status); + } + if (args.source_type !== undefined) { + result.source_type = validateSourceType(args.source_type); + } const ns = validateNamespace(args.namespace); if (ns) result.namespace = ns; return result; @@ -509,8 +828,116 @@ function rrfMerge( return scores; } -function rowToMemory(row: MemoryRow): Memory { - return { ...row, tags: JSON.parse(row.tags) as string[] }; +function rowToMemory(row: MemoryRow | (Partial & Pick)): Memory { + return { + id: row.id, + key: row.key, + content: row.content, + tags: JSON.parse(row.tags) as string[], + importance: row.importance, + author: row.author, + memory_type: row.memory_type, + namespace: row.namespace ?? null, + source_type: row.source_type ?? 'manual', + source_url: row.source_url ?? null, + source_path: row.source_path ?? null, + source_line_start: row.source_line_start ?? null, + source_line_end: row.source_line_end ?? null, + source_title: row.source_title ?? null, + source_hash: row.source_hash ?? null, + status: row.status ?? 'active', + confidence: row.confidence ?? 0.75, + verified_at: row.verified_at ?? null, + expires_at: row.expires_at ?? null, + supersedes_key: row.supersedes_key ?? null, + superseded_by_key: row.superseded_by_key ?? null, + created_at: row.created_at ?? new Date(0).toISOString(), + updated_at: row.updated_at ?? new Date(0).toISOString(), + accessed_at: row.accessed_at ?? row.updated_at ?? row.created_at ?? new Date(0).toISOString(), + access_count: row.access_count ?? 0, + }; +} + +type RetrievalResult = { memory: Memory; combinedScore: number; rerankerScore: number }; + +function memoryProvenance(memory: Memory) { + return { + source_type: memory.source_type, + source_url: memory.source_url, + source_path: memory.source_path, + source_line_start: memory.source_line_start, + source_line_end: memory.source_line_end, + source_title: memory.source_title, + source_hash: memory.source_hash, + }; +} + +function memoryLifecycle(memory: Memory) { + return { + status: memory.status, + confidence: memory.confidence, + verified_at: memory.verified_at, + expires_at: memory.expires_at, + }; +} + +function memorySupersession(memory: Memory) { + return { + supersedes_key: memory.supersedes_key, + superseded_by_key: memory.superseded_by_key, + }; +} + +function memoryWarnings(memory: Memory): string[] { + const warnings: string[] = []; + if (memory.status === 'stale') warnings.push('Warning: stale memory; verify before relying on it.'); + if (memory.status === 'superseded' && memory.superseded_by_key) { + warnings.push(`Warning: superseded by ${memory.superseded_by_key}; prefer the replacement.`); + } + if (memory.confidence < 0.5) warnings.push(`Warning: low confidence (${memory.confidence}).`); + return warnings; +} + +function provenanceLine(memory: Memory): string | null { + const hasSpecificSource = Boolean(memory.source_url || memory.source_path || memory.source_title || memory.source_hash || memory.source_line_start || memory.source_line_end); + if (memory.source_type === 'manual' && !hasSpecificSource) return null; + + const parts = [`type: ${memory.source_type}`]; + if (memory.source_title) parts.push(`title: ${memory.source_title}`); + if (memory.source_url) parts.push(`url: ${memory.source_url}`); + if (memory.source_path) { + let path = memory.source_path; + if (memory.source_line_start !== null && memory.source_line_end !== null) path += `:${memory.source_line_start}-${memory.source_line_end}`; + else if (memory.source_line_start !== null) path += `:${memory.source_line_start}`; + parts.push(`path: ${path}`); + } else if (memory.source_line_start !== null || memory.source_line_end !== null) { + const range = memory.source_line_start !== null && memory.source_line_end !== null + ? `${memory.source_line_start}-${memory.source_line_end}` + : String(memory.source_line_start ?? memory.source_line_end); + parts.push(`lines: ${range}`); + } + if (memory.source_hash) parts.push(`hash: ${memory.source_hash}`); + return ` Source: ${parts.join(' | ')}`; +} + +function formatJsonResults(results: RetrievalResult[]): string { + return JSON.stringify({ + results: results.map((r) => ({ + key: r.memory.key, + content: r.memory.content, + score: r.combinedScore, + reranker_score: r.rerankerScore, + importance: r.memory.importance, + status: r.memory.status, + confidence: r.memory.confidence, + tags: r.memory.tags, + namespace: r.memory.namespace, + provenance: memoryProvenance(r.memory), + lifecycle: memoryLifecycle(r.memory), + supersession: memorySupersession(r.memory), + warnings: memoryWarnings(r.memory), + })), + }, null, 2); } function textResult(text: string, isError = false): McpToolResult { @@ -528,6 +955,12 @@ export async function executeTool( switch (name) { case 'store_memory': return await storeMemory(validateStoreInput(args), adapter); + case 'mark_memory_status': + return await markMemoryStatus(validateMarkMemoryStatusInput(args), adapter); + case 'verify_memory': + return await verifyMemory(validateVerifyMemoryInput(args), adapter); + case 'supersede_memory': + return await supersedeMemory(validateSupersedeMemoryInput(args), adapter); case 'retrieve_memory': return await retrieveMemory(validateRetrieveInput(args), adapter); case 'list_memories': @@ -548,8 +981,6 @@ export async function executeTool( } } -// ─── Tool implementations ─────────────────────────────────────────── - async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Promise { const id = crypto.randomUUID(); const now = new Date().toISOString(); @@ -562,7 +993,7 @@ async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Pro if (dupeMatches.length) { const top = dupeMatches[0]; - if (top.score > 0.92 && top.id !== input.key) { + if (top.score > 0.92 && top.id !== input.key && top.id !== input.supersedes) { return textResult( `Similar memory already exists at key "${top.id}" (similarity: ${top.score.toFixed(3)}). ` + `Consider updating that memory instead, or use the same key to overwrite. Memory was NOT stored.`, @@ -590,8 +1021,14 @@ async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Pro // 1. D1 — source of truth. If this fails, nothing is stored anywhere. try { await adapter.query( - `INSERT INTO memories (id, key, content, tags, importance, author, memory_type, namespace, created_at, updated_at, accessed_at, access_count) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0) + `INSERT INTO memories ( + id, key, content, tags, importance, author, memory_type, namespace, + source_type, source_url, source_path, source_line_start, source_line_end, + source_title, source_hash, status, confidence, verified_at, expires_at, + supersedes_key, superseded_by_key, + created_at, updated_at, accessed_at, access_count + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, ?, ?, ?, 0) ON CONFLICT (key) DO UPDATE SET content = excluded.content, tags = excluded.tags, @@ -599,8 +1036,45 @@ async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Pro author = excluded.author, memory_type = excluded.memory_type, namespace = excluded.namespace, + source_type = excluded.source_type, + source_url = excluded.source_url, + source_path = excluded.source_path, + source_line_start = excluded.source_line_start, + source_line_end = excluded.source_line_end, + source_title = excluded.source_title, + source_hash = excluded.source_hash, + status = excluded.status, + confidence = excluded.confidence, + verified_at = excluded.verified_at, + expires_at = excluded.expires_at, + supersedes_key = excluded.supersedes_key, + superseded_by_key = NULL, updated_at = excluded.updated_at`, - [id, input.key, input.content, JSON.stringify(input.tags), input.importance, input.author, input.memory_type, input.namespace, now, now, now], + [ + id, + input.key, + input.content, + JSON.stringify(input.tags), + input.importance, + input.author, + input.memory_type, + input.namespace, + input.source_type, + input.source_url, + input.source_path, + input.source_line_start, + input.source_line_end, + input.source_title, + input.source_hash, + input.status, + input.confidence, + input.verified_at, + input.expires_at, + input.supersedes ?? null, + now, + now, + now, + ], ); } catch (err) { console.error('[storeMemory] D1 upsert failed:', err instanceof Error ? err.message : err, { key: input.key }); @@ -630,6 +1104,10 @@ async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Pro tags: input.tags.join(','), importance: input.importance, author: input.author, + status: input.status, + source_type: input.source_type, + confidence: input.confidence, + namespace: input.namespace, }); } catch (err) { vecOk = false; @@ -709,6 +1187,92 @@ async function storeMemory(input: StoreMemoryInput, adapter: RecallAdapter): Pro ); } +async function markMemoryStatus(input: MarkMemoryStatusInput, adapter: RecallAdapter): Promise { + const existing = await adapter.query('SELECT id FROM memories WHERE key = ?', [input.key]); + if (!existing.length) return textResult(`Memory "${input.key}" not found.`, true); + + const now = new Date().toISOString(); + await adapter.query('UPDATE memories SET status = ?, updated_at = ? WHERE key = ?', [input.status, now, input.key]); + + const reason = input.reason ? ` Reason: ${input.reason}` : ''; + return textResult(`Marked memory "${input.key}" as ${input.status}.${reason}`); +} + +async function verifyMemory(input: VerifyMemoryInput, adapter: RecallAdapter): Promise { + const existing = await adapter.query('SELECT id FROM memories WHERE key = ?', [input.key]); + if (!existing.length) return textResult(`Memory "${input.key}" not found.`, true); + + const assignments: string[] = ['verified_at = ?', 'updated_at = ?']; + const now = new Date().toISOString(); + const params: unknown[] = [input.verified_at, now]; + + if (input.status !== undefined) { + assignments.push('status = ?'); + params.push(input.status); + } + + const optionalFields: Array = [ + 'confidence', + 'source_type', + 'source_url', + 'source_path', + 'source_line_start', + 'source_line_end', + 'source_title', + 'source_hash', + 'expires_at', + ]; + for (const field of optionalFields) { + if (Object.prototype.hasOwnProperty.call(input, field)) { + assignments.push(`${field} = ?`); + params.push(input[field]); + } + } + params.push(input.key); + + await adapter.query(`UPDATE memories SET ${assignments.join(', ')} WHERE key = ?`, params); + const statusNote = input.status === undefined ? 'existing status preserved' : `status ${input.status}`; + return textResult(`Verified memory "${input.key}" at ${input.verified_at} with ${statusNote}.`); +} + +async function supersedeMemory(input: SupersedeMemoryInput, adapter: RecallAdapter): Promise { + const oldExists = await adapter.query('SELECT id FROM memories WHERE key = ?', [input.old_key]); + if (!oldExists.length) return textResult(`Memory "${input.old_key}" not found.`, true); + + const newExistsBefore = await adapter.query('SELECT id FROM memories WHERE key = ?', [input.new_key]); + if (newExistsBefore.length) return textResult(`Replacement memory "${input.new_key}" already exists.`, true); + + const storeResult = await storeMemory(input, adapter); + const newExistsAfter = await adapter.query('SELECT id FROM memories WHERE key = ?', [input.new_key]); + if (!newExistsAfter.length) { + return storeResult.isError ? storeResult : textResult(`Replacement memory "${input.new_key}" was not stored.`, true); + } + + const now = new Date().toISOString(); + await adapter.batch([ + { + sql: 'UPDATE memories SET status = ?, superseded_by_key = ?, updated_at = ? WHERE key = ?', + params: ['superseded', input.new_key, now, input.old_key], + }, + { + sql: `INSERT INTO memory_relationships (from_key, to_key, relationship_type, strength, created_at) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT (from_key, to_key, relationship_type) DO UPDATE SET strength = excluded.strength`, + params: [input.new_key, input.old_key, 'supersedes', 1.0, now], + }, + { + sql: `INSERT INTO memory_relationships (from_key, to_key, relationship_type, strength, created_at) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT (from_key, to_key, relationship_type) DO UPDATE SET strength = excluded.strength`, + params: [input.old_key, input.new_key, 'superseded_by', 1.0, now], + }, + ]); + + const reason = input.reason ? ` Reason: ${input.reason}` : ''; + const warning = storeResult.isError ? ` Store warning: ${storeResult.content[0]?.text ?? 'index sync warning'}` : ''; + return textResult(`Superseded memory "${input.old_key}" with "${input.new_key}".${reason}${warning}`); +} + async function retrieveMemory(input: RetrieveMemoryInput, adapter: RecallAdapter): Promise { const limit = input.limit ?? 5; const candidateCount = 20; @@ -763,6 +1327,10 @@ async function retrieveMemory(input: RetrieveMemoryInput, adapter: RecallAdapter memories = memories.filter((m) => m.namespace === input.namespace); } + // Lifecycle filter: active/stale by default; superseded/deprecated only when explicitly requested. + const includeStatuses = input.include_statuses ?? ['active', 'stale']; + memories = memories.filter((m) => includeStatuses.includes(m.status)); + if (!memories.length) { return textResult('No memories found matching your query and filters.'); } @@ -778,8 +1346,8 @@ async function retrieveMemory(input: RetrieveMemoryInput, adapter: RecallAdapter (now - new Date(item.memory.accessed_at).getTime()) / (1000 * 60 * 60); const halfLife = HALF_LIFE_HOURS[item.memory.memory_type] ?? HALF_LIFE_HOURS.semantic; const recencyScore = Math.pow(2, -Math.max(hoursSinceAccess, 0) / halfLife); - const combinedScore = - 0.5 * item.rerankerScore + 0.3 * recencyScore + 0.2 * item.memory.importance; + const baseScore = 0.5 * item.rerankerScore + 0.3 * recencyScore + 0.2 * item.memory.importance; + const combinedScore = item.memory.status === 'stale' ? baseScore - 0.05 : baseScore; return { memory: item.memory, combinedScore, rerankerScore: item.rerankerScore }; }); @@ -805,13 +1373,21 @@ async function retrieveMemory(input: RetrieveMemoryInput, adapter: RecallAdapter await adapter.batch(accessUpdates); } + if (input.format === 'json') { + return textResult(formatJsonResults(topResults)); + } + const lines = topResults.map((r, i) => { const tagStr = r.memory.tags.length ? r.memory.tags.join(', ') : 'none'; - return [ - `${i + 1}. **${r.memory.key}** (score: ${r.combinedScore.toFixed(3)}, importance: ${r.memory.importance})`, + const details = [ + `${i + 1}. **${r.memory.key}** (score: ${r.combinedScore.toFixed(3)}, importance: ${r.memory.importance}, status: ${r.memory.status}, confidence: ${r.memory.confidence})`, ` Tags: ${tagStr} | Author: ${r.memory.author}`, - ` ${r.memory.content}`, - ].join('\n'); + ]; + const source = input.include_provenance === false ? null : provenanceLine(r.memory); + if (source) details.push(source); + details.push(...memoryWarnings(r.memory).map((w) => ` ${w}`)); + details.push(` ${r.memory.content}`); + return details.join('\n'); }); return textResult(`Found ${topResults.length} relevant memories:\n\n${lines.join('\n\n')}`); @@ -821,7 +1397,7 @@ async function listMemories(input: ListMemoriesInput, adapter: RecallAdapter): P const limit = input.limit ?? 50; const offset = input.offset ?? 0; - let query = 'SELECT key, tags, importance, author, memory_type, namespace, created_at, updated_at, accessed_at, access_count FROM memories'; + let query = 'SELECT key, tags, importance, author, memory_type, namespace, status, confidence, source_type, supersedes_key, superseded_by_key, created_at, updated_at, accessed_at, access_count FROM memories'; let countQuery = 'SELECT COUNT(*) as total FROM memories'; const conditions: string[] = []; const bindings: unknown[] = []; @@ -836,6 +1412,16 @@ async function listMemories(input: ListMemoriesInput, adapter: RecallAdapter): P bindings.push(input.namespace); } + if (input.status) { + conditions.push('status = ?'); + bindings.push(input.status); + } + + if (input.source_type) { + conditions.push('source_type = ?'); + bindings.push(input.source_type); + } + if (conditions.length) { const where = ' WHERE ' + conditions.join(' AND '); query += where; @@ -865,6 +1451,11 @@ async function listMemories(input: ListMemoriesInput, adapter: RecallAdapter): P author: r.author, memory_type: r.memory_type, namespace: r.namespace, + status: r.status ?? 'active', + confidence: r.confidence ?? 0.75, + source_type: r.source_type ?? 'manual', + supersedes_key: r.supersedes_key ?? null, + superseded_by_key: r.superseded_by_key ?? null, updated_at: r.updated_at, access_count: r.access_count, })); @@ -880,7 +1471,7 @@ async function listMemories(input: ListMemoriesInput, adapter: RecallAdapter): P const lines = items.map((m) => { const nsPart = m.namespace ? ` [ns: ${m.namespace}]` : ''; - return `- **${m.key}**${nsPart} [${m.importance}] ${m.memory_type} by ${m.author} — tags: ${m.tags.join(', ') || 'none'} (updated: ${m.updated_at}, accessed: ${m.access_count}x)`; + return `- **${m.key}**${nsPart} [${m.importance}] ${m.memory_type} by ${m.author} — status: ${m.status}, confidence: ${m.confidence}, source: ${m.source_type}, tags: ${m.tags.join(', ') || 'none'} (updated: ${m.updated_at}, accessed: ${m.access_count}x)`; }); const pageInfo = `Showing ${offset + 1}–${offset + items.length} of ${total}`; @@ -1084,27 +1675,69 @@ export async function runConsolidationReport( ); if (!allRows.length) { - return '## Memory Consolidation Report\n\nNo memories to analyze.'; + return '## Memory Consolidation Report\n\n**READ-ONLY: this report did not modify, delete, verify, supersede, or mark any memories.**\n\nNo memories to analyze.'; } const memories = allRows.map(rowToMemory); - // 2. Find stale memories (cheap — D1 only, no AI calls) + // 2. Find lifecycle maintenance candidates (cheap — D1 only, no AI calls) const now = Date.now(); const staleThresholdMs = staleDays * 24 * 60 * 60 * 1000; + const activeOrStale = new Set(['active', 'stale']); + const daysSince = (iso: string): number => Math.floor((now - new Date(iso).getTime()) / (24 * 60 * 60 * 1000)); + const hasNoProvenance = (m: Memory): boolean => + m.source_type === 'manual' && + !m.source_url && + !m.source_path && + !m.source_title && + !m.source_hash; + const staleMemories = memories .filter((m) => { const age = now - new Date(m.updated_at).getTime(); - return m.access_count === 0 && age > staleThresholdMs; + return activeOrStale.has(m.status) && m.access_count === 0 && age > staleThresholdMs; }) .map((m) => ({ key: m.key, - age_days: Math.floor( - (now - new Date(m.updated_at).getTime()) / (24 * 60 * 60 * 1000), - ), + status: m.status, + age_days: daysSince(m.updated_at), importance: m.importance, })); + const expiredMemories = memories + .filter((m) => activeOrStale.has(m.status) && m.expires_at && Date.parse(m.expires_at) < now) + .map((m) => ({ key: m.key, status: m.status, expired_days: daysSince(m.expires_at as string), expires_at: m.expires_at })); + + const oldUnverifiedMemories = memories + .filter((m) => { + if (!activeOrStale.has(m.status)) return false; + if (!m.verified_at) return true; + return now - Date.parse(m.verified_at) > staleThresholdMs; + }) + .map((m) => ({ + key: m.key, + status: m.status, + verified_at: m.verified_at, + unverified_days: m.verified_at ? daysSince(m.verified_at) : null, + confidence: m.confidence, + })); + + const supersededAccessedMemories = memories + .filter((m) => m.status === 'superseded' && m.access_count > 0) + .map((m) => ({ key: m.key, access_count: m.access_count, superseded_by_key: m.superseded_by_key })); + + const lowConfidenceMemories = memories + .filter((m) => m.confidence < 0.5) + .map((m) => ({ key: m.key, status: m.status, confidence: m.confidence })); + + const noProvenanceMemories = memories + .filter(hasNoProvenance) + .map((m) => ({ key: m.key, status: m.status, confidence: m.confidence })); + + const deprecatedMemories = memories + .filter((m) => m.status === 'deprecated') + .map((m) => ({ key: m.key, importance: m.importance, access_count: m.access_count })); + // 3. Generate embeddings sequentially for similarity analysis const embeddings: number[][] = []; for (const m of memories) { @@ -1148,14 +1781,19 @@ export async function runConsolidationReport( report.push( `Scanned ${memories.length} memories at ${new Date().toISOString()}`, ); + report.push('**READ-ONLY: this report did not modify, delete, verify, supersede, or mark any memories.**'); report.push(''); + const pushEmpty = (title: string, empty: string): void => { + report.push(`### ${title}`, empty, ''); + }; + if (similarPairs.length > 0) { report.push( - `### Similar Memory Pairs (threshold: ${similarityThreshold})`, + `### Similar Memory Pairs That May Need Merge/Supersession (threshold: ${similarityThreshold})`, ); report.push( - `Found ${similarPairs.length} pair(s) that may be candidates for merging:`, + `Found ${similarPairs.length} pair(s) that may be candidates for merge or supersession. Suggested lifecycle tools: \`supersede_memory\` when one memory replaces another, or \`store_memory\` followed by \`delete_memory\` when manually merging into a new key.`, ); report.push(''); for (const pair of similarPairs.slice(0, 25)) { @@ -1168,11 +1806,7 @@ export async function runConsolidationReport( } report.push(''); } else { - report.push( - '### Similar Memory Pairs', - 'No similar pairs found above threshold.', - '', - ); + pushEmpty('Similar Memory Pairs That May Need Merge/Supersession', 'No similar pairs found above threshold.'); } if (staleMemories.length > 0) { @@ -1185,24 +1819,97 @@ export async function runConsolidationReport( report.push(''); for (const mem of staleMemories) { report.push( - `- **"${mem.key}"** — ${mem.age_days} days old, importance: ${mem.importance}`, + `- **"${mem.key}"** — status: ${mem.status}, ${mem.age_days} days old, importance: ${mem.importance}. Suggested: \`mark_memory_status\`, \`verify_memory\`, or \`delete_memory\` after review.`, ); } report.push(''); } else { - report.push('### Stale Memories', 'No stale memories found.', ''); + pushEmpty('Stale Memories', 'No stale memories found.'); + } + + if (expiredMemories.length > 0) { + report.push('### Expired Active/Stale Memories'); + report.push('Active or stale memories whose `expires_at` is in the past. Suggested: `verify_memory` to refresh, `mark_memory_status` to stale/deprecated, or `delete_memory` if no longer useful.'); + report.push(''); + for (const mem of expiredMemories) { + report.push(`- **"${mem.key}"** — status: ${mem.status}, expired ${mem.expired_days} days ago (expires_at: ${mem.expires_at})`); + } + report.push(''); + } else { + pushEmpty('Expired Active/Stale Memories', 'No expired active/stale memories found.'); + } + + if (oldUnverifiedMemories.length > 0) { + report.push(`### Old Unverified Active/Stale Memories (stale_days: ${staleDays})`); + report.push('Active or stale memories with `verified_at` missing or older than the configured stale_days. Suggested: `verify_memory` with provenance/confidence, or `mark_memory_status` if trust is uncertain.'); + report.push(''); + for (const mem of oldUnverifiedMemories) { + const age = mem.unverified_days === null ? 'never verified' : `verified ${mem.unverified_days} days ago`; + report.push(`- **"${mem.key}"** — status: ${mem.status}, ${age}, confidence: ${mem.confidence}`); + } + report.push(''); + } else { + pushEmpty('Old Unverified Active/Stale Memories', 'No old unverified active/stale memories found.'); + } + + if (supersededAccessedMemories.length > 0) { + report.push('### Superseded Memories Still Highly Accessed'); + report.push('Superseded memories with access_count > 0 may still be retrieved or referenced. Suggested: inspect callers, `verify_memory`/`mark_memory_status`, or update retrieval habits before `delete_memory`.'); + report.push(''); + for (const mem of supersededAccessedMemories) { + report.push(`- **"${mem.key}"** — access_count: ${mem.access_count}${mem.superseded_by_key ? `, superseded_by: ${mem.superseded_by_key}` : ''}`); + } + report.push(''); + } else { + pushEmpty('Superseded Memories Still Highly Accessed', 'No superseded memories with access_count > 0 found.'); + } + + if (lowConfidenceMemories.length > 0) { + report.push('### Low-Confidence Memories (confidence < 0.5)'); + report.push('Suggested: `verify_memory` with stronger provenance/confidence, `mark_memory_status` as stale/deprecated, or `delete_memory` if incorrect.'); + report.push(''); + for (const mem of lowConfidenceMemories) { + report.push(`- **"${mem.key}"** — status: ${mem.status}, confidence: ${mem.confidence}`); + } + report.push(''); + } else { + pushEmpty('Low-Confidence Memories', 'No memories with confidence < 0.5 found.'); + } + + if (noProvenanceMemories.length > 0) { + report.push('### Memories With No Provenance'); + report.push('Manual memories without source_url/source_path/source_title/source_hash. Suggested: `verify_memory` to attach provenance or confidence, or `mark_memory_status` if trust cannot be established.'); + report.push(''); + for (const mem of noProvenanceMemories) { + report.push(`- **"${mem.key}"** — status: ${mem.status}, confidence: ${mem.confidence}`); + } + report.push(''); + } else { + pushEmpty('Memories With No Provenance', 'No manual memories without provenance found.'); + } + + if (deprecatedMemories.length > 0) { + report.push('### Deprecated Memories That Can Be Deleted'); + report.push('Deprecated memories are deletion candidates after final review. Suggested: `delete_memory` for obsolete entries, or `mark_memory_status` if they should be retained.'); + report.push(''); + for (const mem of deprecatedMemories) { + report.push(`- **"${mem.key}"** — importance: ${mem.importance}, access_count: ${mem.access_count}`); + } + report.push(''); + } else { + pushEmpty('Deprecated Memories That Can Be Deleted', 'No deprecated deletion candidates found.'); } report.push('### Recommendations'); - if (similarPairs.length > 0 || staleMemories.length > 0) { - report.push( - 'Review the items above and use `store_memory` (to merge content under one key) or `delete_memory` (to remove stale entries) as needed.', - ); + const issueCount = similarPairs.length + staleMemories.length + expiredMemories.length + oldUnverifiedMemories.length + supersededAccessedMemories.length + lowConfidenceMemories.length + noProvenanceMemories.length + deprecatedMemories.length; + if (issueCount > 0) { report.push( - '**This report is read-only — no memories were modified.**', + 'Review the sections above and apply lifecycle tools deliberately: `verify_memory` to refresh trust/provenance, `mark_memory_status` to active/stale/deprecated/superseded, `supersede_memory` when replacing stale content, `store_memory` for manual merges, and `delete_memory` only after review.', ); + report.push('**This report is read-only — no memories were modified.**'); } else { - report.push('Memory store looks clean. No action needed.'); + report.push('Memory store looks clean. No lifecycle action needed.'); + report.push('**This report is read-only — no memories were modified.**'); } return report.join('\n'); diff --git a/src/types.ts b/src/types.ts index 84ef999..3132c15 100644 --- a/src/types.ts +++ b/src/types.ts @@ -16,6 +16,9 @@ export interface Env { // --- Domain types --- export type MemoryType = 'episodic' | 'semantic' | 'procedural'; +export type MemorySourceType = 'manual' | 'chat' | 'doc' | 'code' | 'issue' | 'pull_request' | 'log' | 'web' | 'inferred'; +export type MemoryStatus = 'active' | 'stale' | 'superseded' | 'deprecated'; +export type RetrieveFormat = 'text' | 'json'; export interface Memory { id: string; @@ -26,6 +29,19 @@ export interface Memory { author: string; memory_type: MemoryType; namespace: string | null; + source_type: MemorySourceType; + source_url: string | null; + source_path: string | null; + source_line_start: number | null; + source_line_end: number | null; + source_title: string | null; + source_hash: string | null; + status: MemoryStatus; + confidence: number; + verified_at: string | null; + expires_at: string | null; + supersedes_key: string | null; + superseded_by_key: string | null; created_at: string; updated_at: string; accessed_at: string; @@ -42,6 +58,19 @@ export interface MemoryRow { author: string; memory_type: MemoryType; namespace: string | null; + source_type: MemorySourceType; + source_url: string | null; + source_path: string | null; + source_line_start: number | null; + source_line_end: number | null; + source_title: string | null; + source_hash: string | null; + status: MemoryStatus; + confidence: number; + verified_at: string | null; + expires_at: string | null; + supersedes_key: string | null; + superseded_by_key: string | null; created_at: string; updated_at: string; accessed_at: string; @@ -101,6 +130,45 @@ export interface StoreMemoryInput { author: string; memory_type: MemoryType; namespace: string | null; + source_type: MemorySourceType; + source_url: string | null; + source_path: string | null; + source_line_start: number | null; + source_line_end: number | null; + source_title: string | null; + source_hash: string | null; + status: MemoryStatus; + confidence: number; + verified_at: string | null; + expires_at: string | null; + supersedes?: string; +} + +export interface MarkMemoryStatusInput { + key: string; + status: MemoryStatus; + reason?: string; +} + +export interface VerifyMemoryInput { + key: string; + confidence?: number; + source_type?: MemorySourceType; + source_url?: string | null; + source_path?: string | null; + source_line_start?: number | null; + source_line_end?: number | null; + source_title?: string | null; + source_hash?: string | null; + verified_at: string; + expires_at?: string | null; + status?: MemoryStatus; +} + +export interface SupersedeMemoryInput extends StoreMemoryInput { + old_key: string; + new_key: string; + reason?: string; } export interface GetRelatedMemoriesInput { @@ -115,6 +183,9 @@ export interface RetrieveMemoryInput { min_importance?: number; tags?: string[]; namespace?: string; + include_statuses?: MemoryStatus[]; + include_provenance?: boolean; + format?: RetrieveFormat; } export interface ListMemoriesInput { @@ -123,6 +194,8 @@ export interface ListMemoriesInput { namespace?: string; limit?: number; offset?: number; + status?: MemoryStatus; + source_type?: MemorySourceType; } export interface DeleteMemoryInput { diff --git a/test/consolidation-lifecycle.test.ts b/test/consolidation-lifecycle.test.ts new file mode 100644 index 0000000..e6e6698 --- /dev/null +++ b/test/consolidation-lifecycle.test.ts @@ -0,0 +1,115 @@ +import { beforeEach, describe, expect, test } from 'vitest'; +import { executeTool } from '../src/tools'; +import { MockAdapter } from './mock-adapter'; + +describe('v2.2 phase 5A lifecycle-aware consolidation report', () => { + let adapter: MockAdapter; + + beforeEach(() => { + adapter = new MockAdapter(); + }); + + async function store(args: Record) { + const key = String(args.key ?? 'memory'); + return executeTool('store_memory', { content: `phase five consolidation lifecycle memory ${key}`, author: 'test', ...args }, adapter); + } + + async function report(args: Record = {}) { + return (await executeTool('consolidate_memories', { max_memories: 50, stale_days: 1, ...args }, adapter)).content[0].text; + } + + test('report includes expired active/stale memories', async () => { + await store({ + key: 'expired-active', + status: 'active', + expires_at: '2000-01-01T00:00:00.000Z', + source_type: 'doc', + source_url: 'https://example.com/expired-active', + }); + await store({ + key: 'expired-stale', + status: 'stale', + expires_at: '2000-01-01T00:00:00.000Z', + source_type: 'doc', + source_url: 'https://example.com/expired-stale', + }); + + const text = await report(); + + expect(text).toContain('### Expired Active/Stale Memories'); + expect(text).toContain('expired-active'); + expect(text).toContain('status: active'); + expect(text).toContain('expired-stale'); + expect(text).toContain('status: stale'); + }); + + test('report includes low-confidence and no-provenance memories', async () => { + await store({ + key: 'low-confidence', + confidence: 0.2, + source_type: 'doc', + source_url: 'https://example.com/low-confidence', + }); + await store({ + key: 'no-provenance', + confidence: 0.8, + }); + + const text = await report(); + + expect(text).toContain('### Low-Confidence Memories'); + expect(text).toContain('low-confidence'); + expect(text).toContain('confidence: 0.2'); + expect(text).toContain('### Memories With No Provenance'); + expect(text).toContain('no-provenance'); + }); + + test('report includes deprecated deletion candidates', async () => { + await store({ + key: 'deprecated-delete-me', + status: 'deprecated', + source_type: 'doc', + source_url: 'https://example.com/deprecated', + }); + + const text = await report(); + + expect(text).toContain('### Deprecated Memories That Can Be Deleted'); + expect(text).toContain('deprecated-delete-me'); + expect(text).toContain('delete_memory'); + }); + + test('report includes superseded but accessed memories', async () => { + await store({ + key: 'superseded-accessed', + status: 'superseded', + source_type: 'doc', + source_url: 'https://example.com/superseded', + }); + const row = adapter.memories.get('superseded-accessed'); + if (row) { + row.access_count = 3; + row.superseded_by_key = 'replacement-memory'; + } + + const text = await report(); + + expect(text).toContain('### Superseded Memories Still Highly Accessed'); + expect(text).toContain('superseded-accessed'); + expect(text).toContain('access_count: 3'); + expect(text).toContain('replacement-memory'); + }); + + test('report includes suggested lifecycle tool names and read-only statement', async () => { + await store({ key: 'needs-review', confidence: 0.1 }); + + const text = await report(); + + expect(text).toContain('READ-ONLY'); + expect(text).toContain('This report is read-only'); + expect(text).toContain('mark_memory_status'); + expect(text).toContain('verify_memory'); + expect(text).toContain('supersede_memory'); + expect(text).toContain('delete_memory'); + }); +}); diff --git a/test/lifecycle-tools.test.ts b/test/lifecycle-tools.test.ts new file mode 100644 index 0000000..242b701 --- /dev/null +++ b/test/lifecycle-tools.test.ts @@ -0,0 +1,185 @@ +import { beforeEach, describe, expect, test } from 'vitest'; +import { executeTool } from '../src/tools'; +import { MockAdapter } from './mock-adapter'; + +describe('v2.2 phase 4 lifecycle tools', () => { + let adapter: MockAdapter; + + beforeEach(() => { + adapter = new MockAdapter(); + }); + + const text = async (tool: string, args: Record) => + (await executeTool(tool, args, adapter)).content[0].text; + + const result = (tool: string, args: Record) => executeTool(tool, args, adapter); + + async function store(args: Record) { + return executeTool('store_memory', args, adapter); + } + + test('mark_memory_status marks stale and retrieval warning appears', async () => { + await store({ key: 'phase4-stale', content: 'phase four stale warning lookup', author: 'test' }); + + const mark = await result('mark_memory_status', { key: 'phase4-stale', status: 'stale', reason: 'needs review' }); + expect(mark.isError).toBeFalsy(); + expect(mark.content[0].text).toContain('needs review'); + expect(adapter.memories.get('phase4-stale')).toMatchObject({ status: 'stale' }); + + const retrieved = await text('retrieve_memory', { query: 'phase four stale warning lookup' }); + expect(retrieved).toContain('phase4-stale'); + expect(retrieved).toContain('status: stale'); + expect(retrieved).toContain('Warning: stale memory; verify before relying on it.'); + }); + + test('verify_memory defaults verified_at and preserves status while updating provided provenance/confidence', async () => { + await store({ key: 'phase4-verify', content: 'phase four verify defaults', author: 'test', status: 'stale' }); + + const before = Date.now(); + const verify = await result('verify_memory', { + key: 'phase4-verify', + confidence: 0.91, + source_type: 'code', + source_path: 'src/tools.ts', + source_line_start: 12, + source_line_end: 20, + source_title: 'Lifecycle tools', + source_hash: 'sha256:phase4', + }); + const after = Date.now(); + + expect(verify.isError).toBeFalsy(); + const row = adapter.memories.get('phase4-verify'); + expect(row).toMatchObject({ + status: 'stale', + confidence: 0.91, + source_type: 'code', + source_path: 'src/tools.ts', + source_line_start: 12, + source_line_end: 20, + source_title: 'Lifecycle tools', + source_hash: 'sha256:phase4', + }); + const verifiedAt = Date.parse(row?.verified_at as string); + expect(verifiedAt).toBeGreaterThanOrEqual(before - 1000); + expect(verifiedAt).toBeLessThanOrEqual(after + 1000); + }); + + test('verify_memory preserves unprovided provenance fields', async () => { + await store({ + key: 'phase4-preserve', + content: 'phase four verify preserve', + author: 'test', + source_type: 'doc', + source_url: 'https://example.com/original', + source_path: 'docs/original.md', + source_line_start: 3, + source_line_end: 7, + source_title: 'Original doc', + source_hash: 'sha256:original', + confidence: 0.4, + status: 'stale', + }); + + await result('verify_memory', { key: 'phase4-preserve', confidence: 0.8, verified_at: '2026-06-22T12:00:00.000Z' }); + + expect(adapter.memories.get('phase4-preserve')).toMatchObject({ + source_type: 'doc', + source_url: 'https://example.com/original', + source_path: 'docs/original.md', + source_line_start: 3, + source_line_end: 7, + source_title: 'Original doc', + source_hash: 'sha256:original', + confidence: 0.8, + verified_at: '2026-06-22T12:00:00.000Z', + status: 'stale', + }); + }); + + test('verify_memory can explicitly reactivate when status is provided', async () => { + await store({ key: 'phase4-reactivate', content: 'phase four verify explicit active', author: 'test', status: 'stale' }); + + const verify = await result('verify_memory', { key: 'phase4-reactivate', status: 'active' }); + + expect(verify.isError).toBeFalsy(); + expect(verify.content[0].text).toContain('status active'); + expect(adapter.memories.get('phase4-reactivate')).toMatchObject({ status: 'active' }); + }); + + test('supersede_memory creates new row, marks old, and retrieval lifecycle behavior works', async () => { + await store({ key: 'phase4-old', content: 'phase four old replacement topic', author: 'test', tags: ['old'] }); + + const supersede = await result('supersede_memory', { + old_key: 'phase4-old', + new_key: 'phase4-new', + content: 'phase four new replacement topic', + author: 'test', + tags: ['new'], + importance: 0.9, + reason: 'newer source', + }); + + expect(supersede.isError).toBeFalsy(); + expect(adapter.memories.get('phase4-new')).toMatchObject({ + key: 'phase4-new', + status: 'active', + supersedes_key: 'phase4-old', + }); + expect(adapter.memories.get('phase4-old')).toMatchObject({ + status: 'superseded', + superseded_by_key: 'phase4-new', + }); + + const defaultRetrieved = await text('retrieve_memory', { query: 'phase four replacement topic', limit: 10 }); + expect(defaultRetrieved).toContain('phase4-new'); + expect(defaultRetrieved).not.toContain('phase4-old'); + + const oldRetrieved = await text('retrieve_memory', { query: 'phase four old replacement topic', include_statuses: ['superseded'] }); + expect(oldRetrieved).toContain('phase4-old'); + expect(oldRetrieved).toContain('Warning: superseded by phase4-new; prefer the replacement.'); + }); + + test('get_related_memories with relationship_type supersedes returns old', async () => { + await store({ key: 'phase4-related-old', content: 'phase four relation old', author: 'test' }); + await result('supersede_memory', { + old_key: 'phase4-related-old', + new_key: 'phase4-related-new', + content: 'phase four relation new', + author: 'test', + }); + + const related = await text('get_related_memories', { key: 'phase4-related-new', relationship_type: 'supersedes' }); + expect(related).toContain('phase4-related-old'); + expect(related).toContain('supersedes'); + }); + + test('supersede_memory bypasses duplicate guard for the memory being replaced', async () => { + await store({ key: 'phase4-dupe-old', content: 'identical replacement content', author: 'test' }); + + const supersede = await result('supersede_memory', { + old_key: 'phase4-dupe-old', + new_key: 'phase4-dupe-new', + content: 'identical replacement content', + author: 'test', + }); + + expect(supersede.isError).toBeFalsy(); + expect(adapter.memories.get('phase4-dupe-new')).toMatchObject({ + key: 'phase4-dupe-new', + supersedes_key: 'phase4-dupe-old', + status: 'active', + }); + expect(adapter.memories.get('phase4-dupe-old')).toMatchObject({ + status: 'superseded', + superseded_by_key: 'phase4-dupe-new', + }); + }); + + test('invalid/missing keys error', async () => { + expect((await result('mark_memory_status', { key: 'missing', status: 'stale' })).isError).toBe(true); + expect((await result('verify_memory', { key: 'missing' })).isError).toBe(true); + expect((await result('supersede_memory', { old_key: 'missing', new_key: 'new', content: 'replacement', author: 'test' })).isError).toBe(true); + expect((await result('supersede_memory', { old_key: 'same', new_key: 'same', content: 'replacement', author: 'test' })).isError).toBe(true); + }); +}); diff --git a/test/lifecycle-validation.test.ts b/test/lifecycle-validation.test.ts new file mode 100644 index 0000000..0992bb8 --- /dev/null +++ b/test/lifecycle-validation.test.ts @@ -0,0 +1,119 @@ +import { describe, expect, test } from 'vitest'; +import { + validateListInput, + validateRetrieveInput, + validateStoreInput, +} from '../src/tools'; + +describe('v2.2 lifecycle/provenance validation', () => { + const baseStore = { + key: 'phase2-note', + content: 'validated lifecycle provenance defaults', + author: 'test', + }; + + test('store_memory applies valid lifecycle/provenance defaults', () => { + expect(validateStoreInput(baseStore)).toMatchObject({ + key: 'phase2-note', + content: 'validated lifecycle provenance defaults', + author: 'test', + source_type: 'manual', + status: 'active', + confidence: 0.75, + source_url: null, + source_path: null, + source_line_start: null, + source_line_end: null, + source_title: null, + source_hash: null, + verified_at: null, + expires_at: null, + }); + }); + + test('store_memory accepts valid lifecycle/provenance fields', () => { + const input = validateStoreInput({ + ...baseStore, + source_type: 'code', + source_url: 'https://example.com/repo/file.ts', + source_path: 'src/file.ts', + source_line_start: 10, + source_line_end: 12, + source_title: 'file.ts', + source_hash: 'sha256:abc123', + status: 'stale', + confidence: 0.9, + verified_at: '2026-06-22T00:00:00.000Z', + expires_at: '2026-07-22T00:00:00.000Z', + supersedes: 'old-note', + }); + + expect(input).toMatchObject({ + source_type: 'code', + source_line_start: 10, + source_line_end: 12, + status: 'stale', + confidence: 0.9, + verified_at: '2026-06-22T00:00:00.000Z', + expires_at: '2026-07-22T00:00:00.000Z', + supersedes: 'old-note', + }); + }); + + test('store_memory rejects invalid status', () => { + expect(() => validateStoreInput({ ...baseStore, status: 'archived' })).toThrow(/status/i); + }); + + test('store_memory rejects invalid source_type', () => { + expect(() => validateStoreInput({ ...baseStore, source_type: 'email' })).toThrow(/source_type/i); + }); + + test('store_memory rejects invalid confidence', () => { + expect(() => validateStoreInput({ ...baseStore, confidence: 1.1 })).toThrow(/confidence/i); + expect(() => validateStoreInput({ ...baseStore, confidence: -0.1 })).toThrow(/confidence/i); + }); + + test('store_memory rejects invalid line range', () => { + expect(() => validateStoreInput({ + ...baseStore, + source_line_start: 42, + source_line_end: 41, + })).toThrow(/source_line_start/i); + + expect(() => validateStoreInput({ + ...baseStore, + source_line_end: 41, + })).toThrow(/source_line_start/i); + }); + + test('store_memory rejects invalid date', () => { + expect(() => validateStoreInput({ ...baseStore, verified_at: 'not-a-date' })).toThrow(/verified_at/i); + expect(() => validateStoreInput({ ...baseStore, expires_at: 'not-a-date' })).toThrow(/expires_at/i); + }); + + test('retrieve_memory validates include_statuses and format', () => { + expect(validateRetrieveInput({ + query: 'phase 2', + include_statuses: ['active', 'deprecated'], + include_provenance: true, + format: 'json', + })).toMatchObject({ + include_statuses: ['active', 'deprecated'], + include_provenance: true, + format: 'json', + }); + + expect(() => validateRetrieveInput({ query: 'phase 2', include_statuses: ['archived'] })).toThrow(/include_statuses/i); + expect(() => validateRetrieveInput({ query: 'phase 2', format: 'xml' })).toThrow(/format/i); + }); + + test('list_memories validates status and source_type filters', () => { + expect(validateListInput({ status: 'superseded', source_type: 'issue' })).toMatchObject({ + status: 'superseded', + source_type: 'issue', + }); + + expect(() => validateListInput({ status: 'archived' })).toThrow(/status/i); + expect(() => validateListInput({ source_type: 'email' })).toThrow(/source_type/i); + }); +}); diff --git a/test/migrations.test.ts b/test/migrations.test.ts new file mode 100644 index 0000000..2ef795e --- /dev/null +++ b/test/migrations.test.ts @@ -0,0 +1,111 @@ +import { describe, expect, test } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; + +const root = process.cwd(); +const sql = (path: string) => readFileSync(join(root, path), 'utf8'); + +const provenanceColumns = [ + { name: 'source_type', sqlite: /source_type\s+TEXT\s+NOT\s+NULL\s+DEFAULT\s+'manual'/i, pg: /source_type\s+TEXT\s+NOT\s+NULL\s+DEFAULT\s+'manual'/i }, + { name: 'source_url', sqlite: /source_url\s+TEXT/i, pg: /source_url\s+TEXT/i }, + { name: 'source_path', sqlite: /source_path\s+TEXT/i, pg: /source_path\s+TEXT/i }, + { name: 'source_line_start', sqlite: /source_line_start\s+INTEGER/i, pg: /source_line_start\s+INTEGER/i }, + { name: 'source_line_end', sqlite: /source_line_end\s+INTEGER/i, pg: /source_line_end\s+INTEGER/i }, + { name: 'source_title', sqlite: /source_title\s+TEXT/i, pg: /source_title\s+TEXT/i }, + { name: 'source_hash', sqlite: /source_hash\s+TEXT/i, pg: /source_hash\s+TEXT/i }, + { name: 'status', sqlite: /status\s+TEXT\s+NOT\s+NULL\s+DEFAULT\s+'active'/i, pg: /status\s+TEXT\s+NOT\s+NULL\s+DEFAULT\s+'active'/i }, + { name: 'confidence', sqlite: /confidence\s+REAL\s+NOT\s+NULL\s+DEFAULT\s+0\.75/i, pg: /confidence\s+REAL\s+NOT\s+NULL\s+DEFAULT\s+0\.75/i }, + { name: 'verified_at', sqlite: /verified_at\s+TEXT/i, pg: /verified_at\s+TIMESTAMPTZ/i }, + { name: 'expires_at', sqlite: /expires_at\s+TEXT/i, pg: /expires_at\s+TIMESTAMPTZ/i }, + { name: 'supersedes_key', sqlite: /supersedes_key\s+TEXT/i, pg: /supersedes_key\s+TEXT/i }, + { name: 'superseded_by_key', sqlite: /superseded_by_key\s+TEXT/i, pg: /superseded_by_key\s+TEXT/i }, +]; + +const provenanceIndexes = [ + 'idx_memories_status', + 'idx_memories_source_type', + 'idx_memories_verified_at', + 'idx_memories_superseded_by', +]; + +function expectSchemaMigrationsTable(contents: string) { + expect(contents).toMatch(/CREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\s+schema_migrations/i); + expect(contents).toMatch(/version\s+TEXT\s+PRIMARY\s+KEY/i); + expect(contents).toMatch(/applied_at\s+TEXT\s+NOT\s+NULL/i); +} + +function expectFreshSchemaSeedsCurrentMigrations(contents: string, dialect: 'sqlite' | 'pg') { + if (dialect === 'sqlite') { + expect(contents).toMatch(/INSERT\s+OR\s+IGNORE\s+INTO\s+schema_migrations/i); + } else { + expect(contents).toMatch(/INSERT\s+INTO\s+schema_migrations/i); + expect(contents).toMatch(/ON\s+CONFLICT\s*\(version\)\s+DO\s+NOTHING/i); + } + for (const version of ['0000', '0002', '0003', '0004']) { + expect(contents).toContain(`'${version}'`); + } +} + +function expectProvenanceColumns(contents: string, dialect: 'sqlite' | 'pg') { + for (const column of provenanceColumns) { + expect(contents, `missing column ${column.name}`).toMatch(column[dialect]); + } +} + +function expectProvenanceIndexes(contents: string) { + for (const index of provenanceIndexes) { + expect(contents, `missing index ${index}`).toMatch(new RegExp(`CREATE\\s+INDEX\\s+IF\\s+NOT\\s+EXISTS\\s+${index}`, 'i')); + } +} + +describe('schema migrations', () => { + test('0000 creates schema_migrations tracking table', () => { + expectSchemaMigrationsTable(sql('migrations/0000_schema_migrations.sql')); + }); + + test('fresh install schemas include migration tracking table', () => { + for (const path of ['schema.sql', 'local/setup.sql', 'docker/setup.sql']) { + expectSchemaMigrationsTable(sql(path)); + } + }); + + test('fresh install schemas mark bundled historical migrations as applied', () => { + expectFreshSchemaSeedsCurrentMigrations(sql('schema.sql'), 'sqlite'); + expectFreshSchemaSeedsCurrentMigrations(sql('local/setup.sql'), 'sqlite'); + expectFreshSchemaSeedsCurrentMigrations(sql('docker/setup.sql'), 'pg'); + }); + + test('fresh SQLite/D1 schemas include provenance/lifecycle columns and indexes', () => { + for (const path of ['schema.sql', 'local/setup.sql']) { + const contents = sql(path); + expectProvenanceColumns(contents, 'sqlite'); + expectProvenanceIndexes(contents); + } + }); + + test('fresh Postgres schema includes provenance/lifecycle columns and indexes', () => { + const contents = sql('docker/setup.sql'); + expectProvenanceColumns(contents, 'pg'); + expectProvenanceIndexes(contents); + }); + + test('SQLite/D1 0004 migration adds provenance/lifecycle columns and indexes', () => { + const contents = sql('migrations/0004_provenance_lifecycle.sql'); + for (const column of provenanceColumns) { + expect(contents, `missing ALTER for ${column.name}`).toMatch(new RegExp(`ALTER\\s+TABLE\\s+memories\\s+ADD\\s+COLUMN\\s+${column.name}`, 'i')); + } + expectProvenanceColumns(contents, 'sqlite'); + expectProvenanceIndexes(contents); + }); + + test('Postgres 0004 migration uses idempotent column/index creation', () => { + const contents = sql('docker/migrations/0004_provenance_lifecycle.sql'); + for (const column of provenanceColumns) { + expect(contents, `missing idempotent ALTER for ${column.name}`).toMatch( + new RegExp(`ALTER\\s+TABLE\\s+memories\\s+ADD\\s+COLUMN\\s+IF\\s+NOT\\s+EXISTS\\s+${column.name}`, 'i'), + ); + } + expectProvenanceColumns(contents, 'pg'); + expectProvenanceIndexes(contents); + }); +}); diff --git a/test/mock-adapter.ts b/test/mock-adapter.ts index 8d21e43..81af829 100644 --- a/test/mock-adapter.ts +++ b/test/mock-adapter.ts @@ -17,7 +17,7 @@ export class MockAdapter implements RecallAdapter { await this.execBatchStatement(sql, params); return [] as T[]; } - if (s.startsWith('update memories set accessed_at')) { + if (s.startsWith('update memories set ')) { await this.execBatchStatement(sql, params); return [] as T[]; } @@ -107,6 +107,14 @@ export class MockAdapter implements RecallAdapter { rows = rows.filter(r => r.namespace === params[pIdx]); pIdx++; } + if (s.includes('status = ?')) { + rows = rows.filter(r => r.status === params[pIdx]); + pIdx++; + } + if (s.includes('source_type = ?')) { + rows = rows.filter(r => r.source_type === params[pIdx]); + pIdx++; + } return rows; } @@ -120,9 +128,85 @@ export class MockAdapter implements RecallAdapter { const s = sql.trim().toLowerCase(); if (s.startsWith('insert into memories ') || s.startsWith('insert into memories(')) { - // Param order: id, key, content, tags, importance, author, memory_type, namespace, created_at, updated_at, accessed_at - const [id, key, content, tags, importance, author, memory_type, namespace, created_at, updated_at, accessed_at] = - params as [string, string, string, string, number, string, string, string | null | undefined, string, string, string]; + let id: string; + let key: string; + let content: string; + let tags: string; + let importance: number; + let author: string; + let memory_type: string; + let namespace: string | null | undefined; + let source_type: string | null | undefined = 'manual'; + let source_url: string | null | undefined = null; + let source_path: string | null | undefined = null; + let source_line_start: number | null | undefined = null; + let source_line_end: number | null | undefined = null; + let source_title: string | null | undefined = null; + let source_hash: string | null | undefined = null; + let status: string | null | undefined = 'active'; + let confidence: number | null | undefined = 0.75; + let verified_at: string | null | undefined = null; + let expires_at: string | null | undefined = null; + let supersedes_key: string | null | undefined = null; + let created_at: string; + let updated_at: string; + let accessed_at: string; + + if (params.length >= 23) { + [ + id, + key, + content, + tags, + importance, + author, + memory_type, + namespace, + source_type, + source_url, + source_path, + source_line_start, + source_line_end, + source_title, + source_hash, + status, + confidence, + verified_at, + expires_at, + supersedes_key, + created_at, + updated_at, + accessed_at, + ] = params as [ + string, + string, + string, + string, + number, + string, + string, + string | null | undefined, + string | null | undefined, + string | null | undefined, + string | null | undefined, + number | null | undefined, + number | null | undefined, + string | null | undefined, + string | null | undefined, + string | null | undefined, + number | null | undefined, + string | null | undefined, + string | null | undefined, + string | null | undefined, + string, + string, + string, + ]; + } else { + [id, key, content, tags, importance, author, memory_type, namespace, created_at, updated_at, accessed_at] = + params as [string, string, string, string, number, string, string, string | null | undefined, string, string, string]; + } + const existing = this.memories.get(key); this.memories.set(key, { id: existing?.id ?? id, @@ -133,6 +217,19 @@ export class MockAdapter implements RecallAdapter { author, memory_type, namespace: namespace ?? null, + source_type: source_type ?? 'manual', + source_url: source_url ?? null, + source_path: source_path ?? null, + source_line_start: source_line_start ?? null, + source_line_end: source_line_end ?? null, + source_title: source_title ?? null, + source_hash: source_hash ?? null, + status: status ?? 'active', + confidence: confidence ?? 0.75, + verified_at: verified_at ?? null, + expires_at: expires_at ?? null, + supersedes_key: supersedes_key ?? null, + superseded_by_key: null, created_at: existing?.created_at ?? created_at, updated_at, accessed_at: existing?.accessed_at ?? accessed_at, @@ -151,6 +248,41 @@ export class MockAdapter implements RecallAdapter { return; } + if (s.startsWith('update memories set status = ?, updated_at = ? where key = ?')) { + const [status, updatedAt, key] = params as [string, string, string]; + const m = this.memories.get(key); + if (m) { + m.status = status; + m.updated_at = updatedAt; + } + return; + } + + if (s.startsWith('update memories set status = ?, superseded_by_key = ?, updated_at = ? where key = ?')) { + const [status, supersededByKey, updatedAt, key] = params as [string, string, string, string]; + const m = this.memories.get(key); + if (m) { + m.status = status; + m.superseded_by_key = supersededByKey; + m.updated_at = updatedAt; + } + return; + } + + if (s.startsWith('update memories set ') && s.includes(' where key = ?')) { + const key = params[params.length - 1] as string; + const m = this.memories.get(key); + if (m) { + const lowerSql = sql.toLowerCase(); + const setPart = sql.slice(lowerSql.indexOf('set ') + 4, lowerSql.lastIndexOf(' where ')); + const columns = setPart.split(',').map(part => part.trim().split(/\s*=\s*\?/)[0]); + for (let i = 0; i < columns.length; i++) { + m[columns[i]] = params[i]; + } + } + return; + } + if (s.startsWith('delete from memories where key = ?')) { this.memories.delete(params[0] as string); return; @@ -185,10 +317,13 @@ export class MockAdapter implements RecallAdapter { } if (s.startsWith('insert into memory_relationships')) { - const [from_key, to_key, strength, created_at] = params as [string, string, number, string]; - const existing = this.relationships.find(r => r.from_key === from_key && r.to_key === to_key && r.relationship_type === 'similar'); + const [from_key, to_key] = params as [string, string]; + const relationship_type = typeof params[2] === 'string' ? params[2] as string : 'similar'; + const strength = (typeof params[2] === 'string' ? params[3] : params[2]) as number; + const created_at = (typeof params[2] === 'string' ? params[4] : params[3]) as string; + const existing = this.relationships.find(r => r.from_key === from_key && r.to_key === to_key && r.relationship_type === relationship_type); if (existing) existing.strength = strength; - else this.relationships.push({ from_key, to_key, relationship_type: 'similar', strength, created_at }); + else this.relationships.push({ from_key, to_key, relationship_type, strength, created_at }); return; } } diff --git a/test/retrieve-lifecycle.test.ts b/test/retrieve-lifecycle.test.ts new file mode 100644 index 0000000..e30d17d --- /dev/null +++ b/test/retrieve-lifecycle.test.ts @@ -0,0 +1,158 @@ +import { beforeEach, describe, expect, test } from 'vitest'; +import { executeTool } from '../src/tools'; +import { MockAdapter } from './mock-adapter'; + +describe('v2.2 lifecycle/provenance retrieval and listing', () => { + let adapter: MockAdapter; + + beforeEach(() => { + adapter = new MockAdapter(); + }); + + async function store(args: Record) { + return executeTool('store_memory', args, adapter); + } + + async function retrieve(args: Record) { + return executeTool('retrieve_memory', args, adapter); + } + + async function list(args: Record = {}) { + return executeTool('list_memories', args, adapter); + } + + test('store_memory persists lifecycle/provenance fields in MockAdapter row and vector metadata', async () => { + await store({ + key: 'phase3-new', + content: 'phase three persisted provenance lifecycle', + author: 'test', + tags: ['Recall'], + namespace: 'recall-v22', + source_type: 'code', + source_url: 'https://example.com/repo/blob/src/tools.ts', + source_path: 'src/tools.ts', + source_line_start: 10, + source_line_end: 20, + source_title: 'tools.ts', + source_hash: 'sha256:abc123', + status: 'stale', + confidence: 0.42, + verified_at: '2026-06-22T00:00:00.000Z', + expires_at: '2026-07-22T00:00:00.000Z', + supersedes: 'phase3-old', + }); + + const row = adapter.memories.get('phase3-new'); + expect(row).toMatchObject({ + source_type: 'code', + source_url: 'https://example.com/repo/blob/src/tools.ts', + source_path: 'src/tools.ts', + source_line_start: 10, + source_line_end: 20, + source_title: 'tools.ts', + source_hash: 'sha256:abc123', + status: 'stale', + confidence: 0.42, + verified_at: '2026-06-22T00:00:00.000Z', + expires_at: '2026-07-22T00:00:00.000Z', + supersedes_key: 'phase3-old', + superseded_by_key: null, + }); + expect(adapter.vectors.get('phase3-new')?.metadata).toMatchObject({ + status: 'stale', + source_type: 'code', + confidence: 0.42, + namespace: 'recall-v22', + }); + }); + + test('default retrieve excludes superseded/deprecated and includes active/stale', async () => { + await store({ key: 'active-note', content: 'lifecycle query active result', author: 'test', status: 'active' }); + await store({ key: 'stale-note', content: 'lifecycle query stale result', author: 'test', status: 'stale' }); + await store({ key: 'superseded-note', content: 'lifecycle query superseded result', author: 'test', status: 'superseded' }); + await store({ key: 'deprecated-note', content: 'lifecycle query deprecated result', author: 'test', status: 'deprecated' }); + + const text = (await retrieve({ query: 'lifecycle query', limit: 10 })).content[0].text; + expect(text).toContain('active-note'); + expect(text).toContain('stale-note'); + expect(text).not.toContain('superseded-note'); + expect(text).not.toContain('deprecated-note'); + }); + + test('explicit include_statuses can retrieve superseded', async () => { + await store({ key: 'superseded-only', content: 'superseded explicit lookup', author: 'test', status: 'superseded' }); + const row = adapter.memories.get('superseded-only'); + if (row) row.superseded_by_key = 'replacement-note'; + + const text = (await retrieve({ query: 'superseded explicit', include_statuses: ['superseded'] })).content[0].text; + expect(text).toContain('superseded-only'); + expect(text).toContain('Warning: superseded by replacement-note; prefer the replacement.'); + }); + + test('stale result has warning and visible status/score', async () => { + await store({ key: 'stale-warning', content: 'stale penalty visible', author: 'test', status: 'stale', confidence: 0.49 }); + + const text = (await retrieve({ query: 'stale penalty visible' })).content[0].text; + expect(text).toContain('stale-warning'); + expect(text).toMatch(/score: \d+\.\d{3}/); + expect(text).toContain('status: stale'); + expect(text).toContain('confidence: 0.49'); + expect(text).toContain('Warning: stale memory; verify before relying on it.'); + expect(text).toContain('Warning: low confidence (0.49).'); + }); + + test('JSON format parses and contains provenance/lifecycle/supersession/warnings', async () => { + await store({ + key: 'json-lifecycle', + content: 'json lifecycle output', + author: 'test', + source_type: 'doc', + source_url: 'https://example.com/docs', + status: 'stale', + confidence: 0.4, + supersedes: 'json-old', + }); + + const text = (await retrieve({ query: 'json lifecycle output', format: 'json' })).content[0].text; + const parsed = JSON.parse(text); + expect(parsed.results[0]).toMatchObject({ + key: 'json-lifecycle', + provenance: { source_type: 'doc', source_url: 'https://example.com/docs' }, + lifecycle: { status: 'stale', confidence: 0.4 }, + supersession: { supersedes_key: 'json-old', superseded_by_key: null }, + }); + expect(parsed.results[0].warnings).toContain('Warning: stale memory; verify before relying on it.'); + expect(parsed.results[0].warnings).toContain('Warning: low confidence (0.4).'); + }); + + test('include_provenance false omits Source line', async () => { + await store({ + key: 'no-source-line', + content: 'hide source line output', + author: 'test', + source_type: 'code', + source_path: 'src/tools.ts', + source_line_start: 1, + source_line_end: 2, + }); + + const withSource = (await retrieve({ query: 'hide source line output' })).content[0].text; + const withoutSource = (await retrieve({ query: 'hide source line output', include_provenance: false })).content[0].text; + expect(withSource).toContain('Source:'); + expect(withoutSource).not.toContain('Source:'); + }); + + test('list_memories status/source_type filters work and output includes lifecycle fields', async () => { + await store({ key: 'code-stale', content: 'code stale listed', author: 'test', status: 'stale', source_type: 'code', confidence: 0.66 }); + await store({ key: 'doc-stale', content: 'doc stale listed', author: 'test', status: 'stale', source_type: 'doc' }); + await store({ key: 'code-active', content: 'code active listed', author: 'test', status: 'active', source_type: 'code' }); + + const text = (await list({ status: 'stale', source_type: 'code' })).content[0].text; + expect(text).toContain('code-stale'); + expect(text).not.toContain('doc-stale'); + expect(text).not.toContain('code-active'); + expect(text).toContain('status: stale'); + expect(text).toContain('confidence: 0.66'); + expect(text).toContain('source: code'); + }); +});