Recall v2.2: trustworthy memory lifecycle, provenance, and updater

.gitignore

@@ -3,6 +3,7 @@ node_modules/
 
 # Build / cache
 .wrangler/
+.recall-update/
 dist/
 .cache/
 *.tsbuildinfo

CHANGELOG.md

@@ -4,6 +4,31 @@
 
 All notable changes to Recall are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.0] — 2026-06-22
+
+Trustworthy-memory and Hermes-integration release. Adds provenance/lifecycle/supersession workflows and an optional conservative updater path without changing the config ownership model.
+
+### Added
+
+- **Provenance and lifecycle metadata** for trustworthy agent memory: source type/URL/path/title/hash, confidence, verification, expiration, status, and supersession fields.
+- **Lifecycle MCP tools**: `mark_memory_status`, `verify_memory`, and `supersede_memory`, bringing the expected tool count to 10.
+- **Optional updater script** (`recall-update.sh`) with safe default commands: `--check`, `--doctor`, `--apply`, `--rollback`, and `--install-cron weekly`.
+- **Updating documentation** (`docs/UPDATING.md`) covering the config preservation contract, migration behavior, rollback, cron opt-in, and GitHub Actions notes.
+- **Hermes integration guide** (`docs/HERMES_INTEGRATION.md`) documenting one-instance namespace routing, Discord provenance patterns, wrapper-script MCP config, and cron examples.
+- **Release checklist** (`docs/RELEASE_CHECKLIST.md`) and an example weekly GitHub Actions workflow (`examples/github-actions-update.yml`).
+
+### Changed
+
+- README now describes Recall as trustworthy memory with provenance/lifecycle/supersession, links the new docs, and updates Docker quickstart `tools/list` expectation to 10 tools.
+- Consolidation and lifecycle guidance emphasizes preserving audit trails and superseding stale facts instead of silently overwriting history.
+
+### Upgrade notes
+
+- Run `npm ci`, `npm run typecheck`, and `npm test` before deploying.
+- Apply numbered D1 migrations in order, or use `./recall-update.sh --check` to inspect pending migrations and `./recall-update.sh --apply` only when you are comfortable with its conservative deploy flow.
+- The updater intentionally refuses to overwrite user-owned config/secrets (`wrangler.toml`, `.dev.vars`, `.recall-api-key`, MCP configs, D1/Vectorize/routes/env blocks). If a release changes those areas, merge them manually.
+- Rollback redeploys previous worker code only; D1 schema rollback remains manual.
+
 ## [1.1.2] — 2026-04-12
 
 Three bug fixes discovered while running the v1.1.1 wizard end-to-end against a fresh project (the AgentBoard repo). All three would have hit real users; two were silently destructive.
@@ -127,6 +152,7 @@ First public release. Foundational architecture and a usable end-to-end setup wi
 
 **Note on v1.0.0 tagging history:** Between the initial v1.0.0 tag and the release of v1.1.0, the v1.0.0 tag was force-pushed several times during iterative hardening. This was a mistake — tags should be immutable. With v1.1.0, the tag discipline resets: v1.0.0 is now permanently pinned to its original commit, v1.1.0 is the new pinned default, and future versions will ship as new tags (no force-pushes). See the repository's release notes on GitHub for the canonical per-version state.
 
+[2.2.0]: https://github.com/cashcon57/recall/releases/tag/v2.2.0
 [1.1.2]: https://github.com/cashcon57/recall/releases/tag/v1.1.2
 [1.1.1]: https://github.com/cashcon57/recall/releases/tag/v1.1.1
 [1.1.0]: https://github.com/cashcon57/recall/releases/tag/v1.1.0

README.md

@@ -9,7 +9,7 @@
 [![Backends](https://img.shields.io/badge/Backends-3-brightgreen)](#deployment-options)
 [![Ko-fi](https://img.shields.io/badge/Ko--fi-Support-FF5E5B?logo=kofi&logoColor=white)](https://ko-fi.com/cash508287)
 
-**A self-hosted MCP memory server with hybrid semantic + keyword search. Deploy to Cloudflare Workers, run locally without any cloud dependencies, or self-host with Docker.**
+**A self-hosted MCP memory server for trustworthy agent memory: hybrid semantic + keyword search, provenance, lifecycle controls, supersession, and durable recall. Deploy to Cloudflare Workers, run locally without any cloud dependencies, or self-host with Docker.**
 
 > Not affiliated with Microsoft Windows Recall. This is an open-source memory server for AI coding assistants (Claude Code, Cursor, Windsurf, Cline, Claude Desktop, anything speaking MCP).
 
@@ -69,11 +69,19 @@ Three backends. Same tool surface. Choose what fits your infra:
 
 > **Memory portability between backends:** All three backends use the bge-m3 model (1024D embeddings). Local and Docker backends run the [Xenova ONNX export](https://huggingface.co/Xenova/bge-m3) with CLS pooling to match the reference recipe. Vector compatibility across backends is close in practice — semantic search works cross-backend — but exact bit-for-bit portability is not yet verified with an end-to-end smoke test. If strict portability matters, test your workload before migrating.
 
+## Docs
+
+- [Updating Recall](./docs/UPDATING.md) — optional updater script, config preservation contract, migrations, rollback, cron, and GitHub Actions notes.
+- [Hermes integration](./docs/HERMES_INTEGRATION.md) — recommended namespaces, Discord provenance, MCP wrapper config, routing rules, and cron examples for Hermes Agent.
+- [Release checklist](./docs/RELEASE_CHECKLIST.md) — validation checklist for typecheck/test/deploy, MCP smoke tests, lifecycle tools, consolidation, updater, and Hermes docs.
+
 ## Why Recall?
 
 Most MCP memory servers do one of two things: dump text into SQLite with cosine similarity, or call a hosted vector DB. Recall does both — at the same time — and reranks the combined results with a cross-encoder.
 
 - **Hybrid search** — Vector similarity (bge-m3, 1024D) + BM25 full-text search, fused via Reciprocal Rank Fusion. Catches both semantic paraphrases and exact keyword matches.
+- **Trustworthy memory metadata** — Optional provenance fields (`source_type`, URL/path/title/hash), confidence, verification time, expiration, and status make retrieved memories auditable instead of context-free snippets.
+- **Lifecycle + supersession** — Mark memories active/stale/superseded/deprecated, verify them, or supersede older keys without losing the historical trail that explains past behavior.
 - **Cross-encoder reranking** — Final candidates run through bge-reranker-base for precision. Content is truncated before reranking to keep AI token usage low.
 - **Tiered recency decay** — Three memory tiers with biological half-lives: `episodic` (7d, session events), `semantic` (69d, facts/concepts), `procedural` (693d, stable rules/credentials). Episodic context ages out fast; architecture decisions stay relevant for years. Tier design inspired by [NornicDB](https://github.com/orneryd/NornicDB).
 - **Auto-relationship graph** — Memories with embedding similarity > 0.82 are automatically linked on store. Traverse with `get_related_memories` to find contextually adjacent knowledge without re-querying.
@@ -209,7 +217,7 @@ curl -s -X POST http://localhost:8788 \
   -H "Authorization: Bearer your-secret-key" \
   -H "Content-Type: application/json" \
   -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq '.result.tools | length'
-# Expected: 7
+# Expected: 10
 
 # 4. Tear down (keeps data in pgdata volume)
 docker compose down
@@ -355,6 +363,9 @@ All tools are callable via the standard MCP `tools/call` method.
 | `retrieve_memory`      | Hybrid search (vector + BM25 → RRF → rerank → recency decay → importance). Returns top-N with combined scores. Optional `namespace` filter. |
 | `list_memories`        | Browse with pagination + filters (tag, author, `namespace`, limit, offset). Returns metadata only. |
 | `get_related_memories` | Traverse the auto-relationship graph from a given key. Returns related memories ranked by edge strength. |
+| `mark_memory_status`   | Change lifecycle status (`active`, `stale`, `superseded`, `deprecated`) without deleting historical context. |
+| `verify_memory`        | Mark a memory verified with optional confidence/source updates so agents can prefer trustworthy facts. |
+| `supersede_memory`     | Link an old key to a replacement key, preserving the audit trail while steering retrieval to the newer fact. |
 | `delete_memory`        | Remove a memory by key from the store, FTS, and the vector index. |
 | `clear_memories`       | Wipe everything. **Default-disabled** — requires both `confirm: true` AND the `ALLOW_DESTRUCTIVE_TOOLS=true` secret on the worker. See [Security](#security) for why. |
 | `consolidate_memories` | Read-only analysis: flags similar memory pairs and stale entries. Returns a markdown report. |
@@ -695,7 +706,7 @@ recall/
 ├── src/
 │   ├── index.ts      # Worker fetch + scheduled handler, rate limit, auth
 │   ├── mcp.ts        # JSON-RPC 2.0 / MCP protocol dispatcher
-│   ├── tools.ts      # 6 tool implementations + search pipeline + consolidation
+│   ├── tools.ts      # MCP tool implementations + search, lifecycle, and consolidation
 │   ├── auth.ts       # Constant-time HMAC-SHA256 API key verify
 │   └── types.ts      # Env bindings + domain + JSON-RPC types
 ├── schema.sql        # D1 table + indexes + FTS5 virtual table

docker/migrations/0004_provenance_lifecycle.sql

@@ -0,0 +1,20 @@
+-- Migration 0004: Provenance and lifecycle metadata (Postgres)
+
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_type TEXT NOT NULL DEFAULT 'manual';
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_url TEXT;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_path TEXT;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_line_start INTEGER;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_line_end INTEGER;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_title TEXT;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS source_hash TEXT;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'active';
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS confidence REAL NOT NULL DEFAULT 0.75;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS verified_at TIMESTAMPTZ;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS expires_at TIMESTAMPTZ;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS supersedes_key TEXT;
+ALTER TABLE memories ADD COLUMN IF NOT EXISTS superseded_by_key TEXT;
+
+CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
+CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type);
+CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at);
+CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key);

docker/setup.sql

@@ -1,5 +1,10 @@
 CREATE EXTENSION IF NOT EXISTS vector;
 
+CREATE TABLE IF NOT EXISTS schema_migrations (
+  version TEXT PRIMARY KEY,
+  applied_at TEXT NOT NULL
+);
+
 CREATE TABLE IF NOT EXISTS memories (
   id TEXT PRIMARY KEY,
   key TEXT UNIQUE NOT NULL,
@@ -12,14 +17,31 @@ CREATE TABLE IF NOT EXISTS memories (
   created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
   updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
   accessed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  access_count INTEGER NOT NULL DEFAULT 0
+  access_count INTEGER NOT NULL DEFAULT 0,
+  source_type TEXT NOT NULL DEFAULT 'manual',
+  source_url TEXT,
+  source_path TEXT,
+  source_line_start INTEGER,
+  source_line_end INTEGER,
+  source_title TEXT,
+  source_hash TEXT,
+  status TEXT NOT NULL DEFAULT 'active',
+  confidence REAL NOT NULL DEFAULT 0.75,
+  verified_at TIMESTAMPTZ,
+  expires_at TIMESTAMPTZ,
+  supersedes_key TEXT,
+  superseded_by_key TEXT
 );
 
 CREATE INDEX IF NOT EXISTS idx_memories_key ON memories(key);
 CREATE INDEX IF NOT EXISTS idx_memories_author ON memories(author);
 CREATE INDEX IF NOT EXISTS idx_memories_importance ON memories(importance);
 CREATE INDEX IF NOT EXISTS idx_memories_accessed_at ON memories(accessed_at);
 CREATE INDEX IF NOT EXISTS idx_memories_namespace ON memories(namespace);
+CREATE INDEX IF NOT EXISTS idx_memories_status ON memories(status);
+CREATE INDEX IF NOT EXISTS idx_memories_source_type ON memories(source_type);
+CREATE INDEX IF NOT EXISTS idx_memories_verified_at ON memories(verified_at);
+CREATE INDEX IF NOT EXISTS idx_memories_superseded_by ON memories(superseded_by_key);
 
 CREATE INDEX IF NOT EXISTS idx_memories_fts ON memories USING GIN(
   to_tsvector('english', content || ' ' || key)

docs/HERMES_INTEGRATION.md

@@ -0,0 +1,102 @@
+# Hermes Integration Guide
+
+This guide describes a recommended pattern for using one Recall instance as Hermes Agent's durable memory backend across projects.
+
+## One Recall instance for Hermes
+
+For a solo operator or trusted personal automation setup, run one Recall instance and use namespaces to separate scopes instead of deploying a worker per project.
+
+Recommended namespaces:
+
+- `global` — durable facts and preferences that apply everywhere.
+- `hermes` — Hermes Agent configuration, skills, profiles, workflows, and agent behavior notes.
+- `switchr` — Switchr-specific product, infra, and codebase memories.
+- `corkscrew` — Corkscrew-specific product, infra, and codebase memories.
+- `oss-dashboard` — OSS dashboard project memories.
+- `retro` — retrospectives, mistakes, release notes, and lessons learned.
+- `research` — reusable research notes, vendor comparisons, and investigation results.
+
+Namespaces are a retrieval filter, not an access-control boundary. Anyone with the Recall API key can read/write any namespace.
+
+## Provenance for Discord-originated memory
+
+Hermes can store memories that came from Discord, but the memory should carry enough provenance for future agents to understand where it came from.
+
+Suggested source/provenance pattern:
+
+```json
+{
+  "source_type": "chat",
+  "source_url": "https://discord.com/channels/<guild-id>/<channel-id>/<message-id>",
+  "source_title": "#channel-name / thread title",
+  "tags": ["discord", "decision"],
+  "namespace": "hermes"
+}
+```
+
+For thread discussions, include both the parent channel and thread name in `source_title`, and include a short quote or summary in the memory content. Avoid storing private message content unless you actually want it in long-term agent memory.
+
+### Limitation: no implicit Discord history reconstruction
+
+Recall only knows what was explicitly stored. Hermes cannot reconstruct a Discord channel or thread history from Recall unless messages, decisions, or summaries were saved into Recall at the time (or imported later). Store durable summaries, not raw chat logs, unless raw logs are explicitly needed.
+
+## MCP config without putting the API key in config.yaml
+
+Prefer a tiny wrapper script that reads the API key from a local file or secret manager and launches the MCP bridge. This keeps `config.yaml` generic and avoids committing secrets.
+
+Example wrapper (`~/bin/recall-mcp`):
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+RECALL_URL="${RECALL_URL:-https://your-worker.workers.dev/mcp}"
+KEY_FILE="${RECALL_KEY_FILE:-$HOME/.config/recall/api-key}"
+if [[ -z "${RECALL_API_KEY:-}" ]]; then
+  RECALL_API_KEY="$(tr -d '\r\n' < "$KEY_FILE")"
+fi
+exec npx -y mcp-remote "$RECALL_URL" \
+  --header "Authorization: Bearer ${RECALL_API_KEY}"
+```
+
+Hermes MCP config pattern:
+
+```yaml
+mcp_servers:
+  recall:
+    command: /home/you/bin/recall-mcp
+    args: []
+```
+
+Do not paste the real API key into shared config. If you need to show examples in docs, use `Bearer <redacted>` or `Bearer your-api-key-here`.
+
+## Memory routing rules
+
+Use these conventions when Hermes decides where to store or retrieve memory:
+
+1. Store cross-project personal preferences in `global`.
+2. Store Hermes setup, MCP, skill, plugin, cron, and profile behavior in `hermes`.
+3. Store project-specific code/infra/product facts in that project's namespace.
+4. Store postmortems, mistakes, and durable lessons in `retro`, with project tags when relevant.
+5. Store reusable vendor/library/API research in `research`.
+6. Never store API keys, passwords, private keys, session cookies, or recovery codes.
+7. Prefer memories with provenance: source type, URL/path, title, line range, or Discord channel/thread reference.
+8. Mark decisions with a `decision` tag and write the rationale in the content.
+9. Supersede stale memories rather than silently overwriting when the old fact may still explain past behavior.
+10. Use `confidence`, `verified_at`, and expiration fields for facts that may go stale.
+
+## Cron examples
+
+Weekly Recall update check (non-mutating):
+
+```cron
+0 4 * * 1 cd /path/to/recall && ./recall-update.sh --check >> .recall-update/cron.log 2>&1
+```
+
+Weekly consolidation report through the deployed worker is configured in `wrangler.toml`:
+
+```toml
+[triggers]
+crons = ["0 3 * * SUN"]
+```
+
+A Hermes-side reminder can retrieve the latest `_system.consolidation-report`, review stale/similar candidates, and ask before deleting or superseding anything.

docs/RELEASE_CHECKLIST.md

@@ -0,0 +1,51 @@
+# Release Checklist
+
+Use this checklist before tagging or deploying a Recall release.
+
+## Code quality
+
+- [ ] `npm ci`
+- [ ] `npm run typecheck`
+- [ ] `npm test`
+- [ ] Review `git diff` for accidental secrets, local `wrangler.toml` values, or `.recall-api-key` content.
+- [ ] Confirm migrations are numbered, documented, and covered by tests when applicable.
+
+## Cloudflare deploy smoke
+
+- [ ] Apply required D1 migrations to a staging or test instance.
+- [ ] `npx wrangler deploy`
+- [ ] `curl https://<worker>/health` returns `{ "status": "ok" }` or equivalent minimal health JSON.
+- [ ] Authenticated MCP `tools/list` returns the expected tool set.
+- [ ] JSON-RPC retrieval smoke: `retrieve_memory` returns valid JSON for an empty or seeded query.
+- [ ] JSON-RPC storage smoke: `store_memory` can store a test memory with namespace/provenance fields.
+- [ ] Delete the test memory with `delete_memory`.
+
+## Tool surface
+
+- [ ] Core tools: `store_memory`, `retrieve_memory`, `list_memories`, `get_related_memories`, `delete_memory`, `clear_memories`, `consolidate_memories`.
+- [ ] Lifecycle/provenance tools from the current release are present and documented.
+- [ ] Destructive `clear_memories` remains gated by `ALLOW_DESTRUCTIVE_TOOLS=true` and `confirm: true`.
+- [ ] Lifecycle validation rejects invalid statuses, confidence, dates, or supersession cycles.
+- [ ] Consolidation remains read-only and does not delete or mutate memories automatically.
+
+## Updater checks
+
+- [ ] `bash -n recall-update.sh`
+- [ ] `./recall-update.sh --check` runs without destructive mutation.
+- [ ] `./recall-update.sh --doctor` does not print secrets.
+- [ ] `./recall-update.sh --install-cron weekly` prints a cron line but does not install without `--yes`.
+- [ ] `docs/UPDATING.md` describes migration and rollback behavior for this release.
+
+## Hermes integration checks
+
+- [ ] `docs/HERMES_INTEGRATION.md` namespace guidance is still current.
+- [ ] Wrapper-script MCP config examples do not include a real API key.
+- [ ] Discord provenance guidance reflects current source fields.
+- [ ] README links to updating, Hermes integration, and release checklist docs.
+
+## Release notes
+
+- [ ] `CHANGELOG.md` has a top entry with Added/Changed/Fixed/Security/Upgrade notes as appropriate.
+- [ ] README tool count and examples match the actual MCP tool surface.
+- [ ] Tag name follows semver (`vX.Y.Z`).
+- [ ] GitHub release notes include exact upgrade steps and migration commands.