Feature/knowledge box

[donk8r]

No description provided.

[github-actions]

Octomind — developer:brief (octohub:glm)

Now I have a thorough understanding of the changes. Let me compile the brief.

📦 Brief: Knowledge boxes (git-backed scoped knowledge bundles) + scope refactor + background init + dep pins

Overall risk: 🟡 MEDIUM · Cards: 4

#	Change	Risk	Confidence
1	Knowledge boxes: git-backed scoped knowledge bundles with registry, auto-probe, smart reindex, and CLI/MCP integration	🟡	●●●
2	Scope visibility filter added to knowledge search/match — scope column + bitmap index + migration on existing tables	🟡	●●●
3	Memory manager init tasks (stale-ref cleanup, sleep consolidation) moved to background tokio tasks; several &mut self → &self	🟢	●●●
4	MCP session schema: new global flag on memorize, tool variants reshuffled (full-no-global, role-only strips global), box sync spawned on initialize	🟢	●●○

---

Card 1/4: Knowledge boxes — git-backed scoped knowledge bundles · 🟡 · ●●●

INTENT
Enable teams to share structured knowledge via git repositories ("boxes") that are auto-discovered, cloned, and re-indexed locally — making org-wide and project-scoped knowledge searchable without manual URL indexing.

WHAT CHANGED

• New src/knowledge/boxes.rs module: BoxRegistry (TOML-backed subscription state), RemoteBox struct, scope math (scope_prefixes, visible_scopes, org_scope), box:// URI scheme, git plumbing (clone, pull, head_commit, remote_exists), and collect_indexable filesystem walker
• KnowledgeManager gains import_box, sync_boxes, remove_box, list_boxes, and index_box_dir (smart reindex by content hash, prune removed sources)
• CLI Box subcommand with Import, Sync, List, Remove variants
• sync_boxes auto-probes orgs for a conventional octobrain-box repo, caches negative results, and pulls + reindexes on every sync
• Box sync runs once per MCP session as a background tokio::spawn task on initialize

IMPACT RADIUS

• Knowledge search/match now accepts an active_scope parameter that filters results to visible scopes (project + org ancestors + global) — all callers updated
• store_chunks signature gains a scope parameter — every call site updated
• New boxes/ directory under system storage holds cloned repos and registry.toml
• delete_by_source_prefix and list_sources_with_prefix added to KnowledgeStore for box removal and pruning

RISK
🟡 MEDIUM — sync_boxes calls git clone and git pull via std::process::Command (synchronous, blocking). In the MCP path this is spawned on a tokio task, but Command::output() still blocks an OS thread. For large repos or slow networks, this could starve the tokio thread pool. The remote_exists probe (git ls-remote) has the same issue and runs for every org on every sync.

QUESTIONS

• Should git clone/git pull/git ls-remote be wrapped in tokio::task::spawn_blocking to avoid blocking the async runtime? The current Command::output() calls are synchronous.

DIVERGENCE

• 🧩 Incomplete change: The KnowledgeProvider::sync_boxes method exists and is wired to the MCP server's background task, but there's no MCP tool surface for box sync/box import/box list/box remove — these are CLI-only. The MCP knowledge tool has no command variant for box operations. This looks intentional (boxes auto-sync on session init), but a user interacting via MCP has no way to trigger a manual sync or import.

📎 Source

• src/knowledge/boxes.rs — new module: registry, scope math, git plumbing, taxonomy walk
• src/knowledge/manager.rs#L495-L735 — import_box, sync_boxes, remove_box, index_box_dir
• src/cli.rs#L43-L47 — Box CLI subcommand
• src/commands.rs#L804-L849 — execute_box_command dispatch

Card 2/4: Knowledge scope column + visibility filter + migration · 🟡 · ●●●

INTENT
Add a scope column to the knowledge store so that box-originated knowledge rows can be scoped (org-level, project-level, or global), and searches only return knowledge visible to the active scope — mirroring the memory system's scope model.

WHAT CHANGED

• KnowledgeStore schema gains a scope column (non-null, default "" for global)
• migrate_scope_column adds the column via add_columns to pre-existing tables (non-destructive, idempotent)
• store_chunks signature gains scope: &str parameter, written per-row
• search and match_content accept scopes: Option<&[String]> — translated to a scope IN (...) SQL predicate via scope_in_clause
• Bitmap index created on scope column for fast pushdown
• visible_scopes mirrors memory's rule: project scope → all ancestor prefixes + global; None → no filter (admin)
• All existing store_chunks call sites pass "" (global) for hot knowledge
• CLI search and match now derive active_scope from cwd and pass it through

IMPACT RADIUS

• BREAKING CHANGE: store_chunks, search, and match_content signatures all changed — any external caller must add the scope parameter
• Existing knowledge tables get auto-migrated: scope column added with "" default, so all pre-existing rows become global-visible
• Schema recreation check now excludes scope from the "missing column triggers recreate" logic, since it's handled by migration

RISK
🟡 MEDIUM — The scope_in_clause function builds SQL by joining escaped scope strings into an IN (...) predicate. While escape_sql_literal is used, this is still dynamic SQL construction. The scopes originate from derive_scope (git URL normalization) and user input (--scope flag), so injection risk is low but the pattern is worth noting.

📎 Source

• src/knowledge/store.rs#L31-L45 — scope_in_clause helper
• src/knowledge/store.rs#L65-L96 — migrate_scope_column + schema changes
• src/knowledge/store.rs#L177-L200 — bitmap index + store_chunks with scope
• src/knowledge/boxes.rs#L96-L110 — visible_scopes logic

Card 3/4: Memory manager background init + &mut self → &self refactor · 🟢 · ●●●

INTENT
Stop blocking MemoryManager::new() on stale-reference cleanup and sleep consolidation by spawning them as background tokio tasks, and remove the &mut self requirement from methods that only need shared access.

WHAT CHANGED

• cleanup_stale_references and maybe_sleep_consolidate are now spawned via tokio::spawn in new() instead of being awaited inline
• MemoryManager is constructed as immutable (let manager instead of let mut manager)
• cleanup_stale_references, maybe_sleep_consolidate, propagate_staleness, update_memory, create_relationship, consolidate_goal, sleep_consolidate all changed from &mut self → &self
• has_no_scope() renamed to is_unscoped() for clarity
• Default scope label changed from "default" to "" (global) — scope_label() returns "" for unscoped stores
• Marker file names now use scope_safe (slashes replaced with underscores) to keep filenames flat

IMPACT RADIUS

• Isolated to memory internals. The &self change is a signature-only refactor since MemoryStore was already behind Arc and AsyncMutex.
• Marker files for existing installations will change names (e.g., .stale_check_github.com_org_repo instead of .stale_check_github.com/org/repo), so stale-ref cleanup and sleep consolidation will re-run once on upgrade — this is benign (idempotent operations).

RISK
🟢 LOW — The background spawn clones Arc handles into the task, which is the established pattern. The .ok() on the spawned tasks swallows errors, matching the previous best-effort contract. No behavioral change beyond timing.

📎 Source

• src/memory/manager.rs#L100-L165 — background spawn + &self refactor
• src/memory/store.rs#L78-L87 — scope predicate now includes global union
• src/memory/store.rs#L147-L153 — is_unscoped + scope_label default change

Card 4/4: MCP session schema reshuffle + global flag + box sync on init · 🟢 · ●●○

INTENT
Add a global flag to the memorize tool for cross-project storage when scope is locked, strip global from unlocked sessions (where scope="" achieves the same thing), and kick off box sync automatically on MCP session initialization.

WHAT CHANGED

• MemorizeParams gains global: Option<bool> — when scope is locked and global=true, the memory is stored with scope "" (global)
• New TOOLS_FULL_NO_GLOBAL static: unlocked sessions get the full schema minus the global field (they can pass scope="" directly)
• tools_role_only now strips both role and global
• tools_locked strips scope and role (but keeps global so locked-scope users can opt into global)
• McpServer gains box_sync_started: Arc<AtomicBool> — initialize spawns run_box_sync_background exactly once per server instance
• discover_scopes returns Vec<String> (human-readable) instead of Vec<(String, String)> (label, hex_id)
• has_local_scopes renamed to has_local_projects
• discover_projects added — returns (PathBuf, String) pairs for box sync
• Logging directory selection simplified: uses derive_scope + slash-to-underscore sanitization instead of get_scope_identifier

IMPACT RADIUS

• MCP clients will see a different tool schema depending on session lock state — unlocked sessions no longer see global, locked sessions see it
• Box sync is fire-and-forget on first initialize call — errors are logged and swallowed

RISK
🟢 LOW — The global flag is a narrow addition with clear semantics. The background box sync is non-blocking and failure-tolerant. The schema reshuffle is consistent with the existing pattern of stripping fields per lock state.

📎 Source

• src/mcp/server.rs#L44-L70 — tool schema variants
• src/mcp/server.rs#L475-L530 — MemorizeParams with global field
• src/mcp/server.rs#L779-L790 — box sync spawn on initialize
• src/mcp/logging.rs#L93-L115 — simplified log dir selection

---

📂 Files changed (14 files, ~1219 insertions, ~216 deletions)

• src/knowledge/boxes.rs — New: box registry, scope math, git plumbing, taxonomy walker
• src/knowledge/manager.rs — import_box, sync_boxes, remove_box, index_box_dir; scope param threaded through search/match/store
• src/knowledge/store.rs — scope column, migration, bitmap index, scope_in_clause, delete_by_source_prefix, list_sources_with_prefix
• src/knowledge/mod.rs — added pub mod boxes
• src/cli.rs — Box subcommand with Import/Sync/List/Remove
• src/commands.rs — execute_box_command, scope threading in knowledge CLI search/match
• src/mcp/server.rs — global flag, tool schema variants, box sync on init, discover_projects
• src/mcp/knowledge.rs — active_scope param on search/match, sync_boxes method
• src/mcp/logging.rs — simplified scope-based log dir selection
• src/memory/manager.rs — background init tasks, &mut self → &self, marker file naming
• src/memory/store.rs — scope predicate includes global union, is_unscoped rename, scope_label default ""
• src/storage.rs — derive_scope, normalize_git_url, get_boxes_dir, removed get_scope_identifier
• Cargo.toml — added alloc-stdlib and brotli-decompressor pins
• Cargo.lock — dependency version bumps

---

_{{tokens:315497,cost:0.32664000000000004}}

[github-actions]

Octomind — developer:brief (octohub:glm)

---

_{}

[github-actions]

Octomind — developer:brief (octohub:glm)

Now I have all the evidence I need. Let me produce the brief.

📦 Brief: Two targeted fixes — session scope assignment bug + empty-scope predicate filter

Overall risk: 🟢 LOW · Cards: 2

#	Change	Risk	Confidence
1	MCP session assigned wrong variable (project instead of scope) — now fixed	🟢	●●●
2	Scope predicate now matches empty-scope (global) memories alongside project-scoped ones	🟢	●●●

Card 1/2: MCP session scope assignment used wrong variable · 🟢 · ●●●

INTENT
Fix a variable-name bug where the MCP initialize handler assigned project (undefined/stale) to session.scope instead of the scope value extracted from the handshake capabilities.

WHAT CHANGED

• session.scope = if should_lock_scope { project } else { None } → session.scope = if should_lock_scope { scope } else { None }
• Also renamed get_distinct_scopes_and_roles() call back to get_distinct_projects_and_roles() — the method name on MemoryStore was never renamed during the project→scope refactor, so this aligns the call site with the actual method name.

IMPACT RADIUS
Isolated change — affects only MCP session initialization. Any MCP client that sent a scope in the handshake experimental capabilities would have had it silently dropped (session.scope set to whatever project resolved to, likely None or a compile error in stricter modes). Now correctly propagates the scope.

RISK
🟢 LOW. Bug fix with no behavioral ambiguity — the wrong variable was used, now the right one is. The method-name fix is cosmetic alignment.

📎 Source

• src/mcp/server.rs#L765 — scope assignment in initialize handler
• src/memory/manager.rs#L806 — method name alignment

Card 2/2: Scope predicate now includes empty-scope (global) memories · 🟢 · ●●●

INTENT
Fix scoped searches to also return global (empty-scope) memories. Previously, a scoped predicate like scope = 'proj123' excluded memories stored with no scope — these are "global" memories that should be visible to all projects.

WHAT CHANGED

• build_scalar_predicate now generates (scope = 'proj123' OR scope = '') instead of scope = 'proj123' when a non-empty scope is provided
• This matches the documented contract in the function's own comments: Some(s) → project scope UNION global (always injected)
• Test assertions updated to reflect the new predicate shape
• Knowledge store test calls reformatted (whitespace only, no behavior change)

IMPACT RADIUS
All memory and knowledge searches that pass a non-empty scope will now include global-scope memories in results. This is the intended behavior per the function's own documentation. Callers that relied on scoped searches excluding global memories will see them appear — this is the fix, not a regression.

RISK
🟢 LOW. The predicate change is small and well-tested. The scope = '' clause correctly matches the empty-string global scope. No SQL injection risk — escape_sql is applied.

📎 Source

• src/memory/store.rs#L83-L87 — predicate generation with OR clause
• src/memory/role_tests.rs#L23-L43 — updated test assertions

📂 Files changed (3 files, ~30 lines)

• src/mcp/server.rs — MCP session initialization: scope variable fix
• src/memory/store.rs — Predicate builder: include empty scope in OR clause
• src/memory/role_tests.rs — Test assertions updated for new predicate shape

---

_{{tokens:116070,cost:0.12176800000000002}}

[github-actions]

Octomind — developer:brief (octohub:glm)

Now I have full context. Let me write the brief.

📦 Brief: Six commits activating dead config/capabilities and fixing four correctness bugs in the memory retrieval hot path

Overall risk: 🟡 MEDIUM · Cards: 5

#	Change	Risk	Confidence
1	Reranker now respects caller's limit instead of fixed config; reranker failures degrade to hybrid ranking instead of failing search	🟢	●●●
2	RRF normalization changed from 2/k to 1/k (memory + knowledge); archived memories excluded from all search; HyDE PRF requires ≥2 neighbors	🟡	●●●
3	Multi-query remember_multi replaced heuristic merge with RRF fusion; new suppress_superseded down-ranks stale facts via relationship edges	🟡	●●●
4	MCP remember tool now wires temporal filters + min_relevance; MCP knowledge search returns full parent section instead of 300-char child slice	🟡	●●●
5	search.similarity_threshold and search.max_results — previously dead config — now drive retrieval floor and hard result ceiling	🟡	●●●

---

Card 1/5: Reranker respects caller limit and degrades gracefully on failure · 🟢 · ●●●

INTENT
The reranker silently overrode the caller's limit with a fixed config.final_top_k — a limit=5 request returned up to 10, a limit=20 request returned only 10. Also, any reranker timeout/outage failed the entire search. Both are user-facing correctness issues in the hot retrieval path.

WHAT CHANGED

• rerank_memories now takes an explicit top_n parameter — the number of results the caller asked for — instead of using config.final_top_k.
• MemoryStore::search_memories passes query.limit.unwrap_or(final_top_k) as top_n, falling back to config only when the caller set no limit.
• On any reranker error (timeout, outage, parse failure), the search now falls back to the pre-rerank hybrid ranking truncated to top_n and logs a tracing::warn, instead of propagating the error and failing the whole search.

IMPACT RADIUS

• MemoryStore::search_memories is the single call site — it constructs final_top_n and passes it through. No other caller of rerank_memories exists.
• The graceful-degradation path means agents may silently receive un-reranked results during a reranker outage. The tracing::warn is the only signal.

RISK
🟢 LOW. The fallback returns the already-good hybrid ranking — strictly better than failing. The top_n passthrough fixes a real bug where caller limits were ignored. No downstream contract changes.

📎 Source

• src/memory/reranker_integration.rs#L69-L143 — rerank_memories with top_n param + graceful degradation
• src/memory/store.rs#L773-L785 — call site: final_top_n derived from query.limit

Card 2/5: RRF normalization, archived exclusion, and HyDE PRF guard — three hot-path correctness fixes · 🟡 · ●●●

INTENT
Three independent correctness bugs in the retrieval scoring/filtering pipeline: RRF score normalization penalized single-modality hits, archived tombstones could surface in search, and HyDE query expansion drifted on single-neighbor input.

WHAT CHANGED

• RRF normalization: max_rrf_score changed from 2.0 / RRF_K to 1.0 / RRF_K in both MemoryStore::hybrid_search and KnowledgeStore. A memory found only by the dense vector leg (no lexical overlap) previously capped at 0.5 relevance; now any rank-0 hit reaches ~1.0. Excess clamps to 1.0.
• Archived exclusion: build_scalar_predicate now unconditionally appends state != 'archived'. Previously no state filter existed, so archived tombstones could appear in relevance search results. Working + Consolidated states remain retrievable.
• HyDE PRF guard: expand_query_embedding now requires count >= 2 neighbors before blending the centroid into the query embedding. With 0–1 neighbors, the original embedding is returned untouched, preventing Rocchio drift toward a single noisy doc.

IMPACT RADIUS

• build_scalar_predicate is called by both vector_search and hybrid_search — every memory search path now excludes archived state. The role_tests.rs assertions were updated to expect the new state != 'archived' clause.
• The RRF normalization change affects the absolute relevance score scale. Any min_relevance threshold comparison now operates on a different scale — scores that were 0.5 are now ~1.0. This interacts with Card 5's similarity_threshold default (0.3), which was previously a dead value and now becomes the active floor.

RISK
🟡 MEDIUM. The RRF scale change is silent and global: existing deployments with min_relevance tuned to the old 2/k scale will see different filtering behavior. The archived exclusion is additive and correct, but if any code path intentionally retrieves archived memories via search_memories (e.g. audit queries), it will now silently return zero. The HyDE guard is strictly safer.

QUESTIONS

• Is there any audit/admin path that retrieves archived memories through search_memories rather than a direct table scan? If so, the unconditional state != 'archived' clause would silently break it.

📎 Source

• src/memory/store.rs#L1168-L1175 — RRF normalization 1.0 / RRF_K in hybrid search
• src/knowledge/store.rs#L368-L376 — mirrored RRF fix in knowledge store
• src/memory/store.rs#L94-L97 — unconditional state != 'archived' in predicate
• src/memory/store.rs#L1079-L1085 — HyDE PRF count < 2 guard

Card 3/5: Multi-query retrieval uses RRF fusion; Supersedes edges now down-rank stale facts · 🟡 · ●●●

INTENT
remember_multi's old merge heuristic (keep-max-score + flat 10% boost per extra match) mixed incomparable per-query weighted-sum scales and discarded robust cross-query consensus in favor of one-off top hits. Separately, the Supersedes relationship type was stored but never read at retrieval — stale facts could outrank current ones.

WHAT CHANGED

• remember_multi now over-fetches limit * 3 per query (capped at max_search_results), collects per-query ranked id lists, and fuses them via rrf_fuse — a pure function computing Σ 1/(k + rank_i) with k=60. Final relevance is normalized to [0,1] by dividing by the max fused score.
• suppress_superseded runs after every remember and remember_multi call: for each result, it fetches that memory's relationships and checks if any Supersedes edge targets it (rel.target_id == r.memory.id). If so, relevance is multiplied by 0.1 and results are re-sorted. Best-effort — a relationship-lookup failure leaves the result untouched.
• rrf_fuse is unit-tested in fusion_tests.rs (4 tests covering rank-zero value, consensus-over-one-off, accumulation, and empty input).

IMPACT RADIUS

• remember and remember_multi are called by MemoryProvider::execute_remember (MCP) and commands.rs (CLI). Both paths now run suppress_superseded on every search.
• suppress_superseded issues one get_memory_relationships call per result — an N+1 query pattern. For a 50-result search, that's 50 relationship lookups on the hot read path.

RISK
🟡 MEDIUM. The N+1 relationship lookups in suppress_superseded add per-result latency to every remember call. The early return if results.len() < 2 mitigates trivial cases, but a typical search returning 20–50 results issues 20–50 async DB reads. If relationship table lookups are not indexed or cached, this could measurably slow retrieval under load.

DIVERGENCE
🧩 Incomplete change — The memorize tool description was updated (commit 2d76900) to instruct agents to create supersedes edges, but suppress_superseded only fires if the edge already exists. The read-side mechanic is live; whether it activates depends entirely on agent behavior the description cannot guarantee.

📎 Source

• src/memory/manager.rs#L634-L696 — remember_multi RRF fusion rewrite
• src/memory/manager.rs#L580-L617 — suppress_superseded with N+1 lookups
• src/memory/manager.rs#L1467-L1475 — rrf_fuse pure function
• src/memory/fusion_tests.rs#L1-L57 — RRF unit tests

Card 4/5: MCP remember wires temporal/relevance filters; knowledge search returns full parent sections · 🟡 · ●●●

INTENT
Two MCP-layer capabilities that were built but never wired to the agent-facing path: MemoryQuery already supported created_after/created_before/min_relevance but the remember MCP tool never populated them. Separately, the knowledge MCP search returned a 300-char slice of the embedded child chunk, defeating the parent/child chunker design whose parent_content field exists specifically to give consumers rich context.

WHAT CHANGED

• execute_remember now parses created_after, created_before (via new parse_iso_datetime — accepts bare date or RFC3339), and min_relevance from MCP args and populates the MemoryQuery. Absent/unparseable values skip the filter.
• RememberParams struct gained created_after: Option<String> and created_before: Option<String> fields with schema descriptions.
• The remember tool description now tells agents to compute time windows themselves ("you know today's date").
• MCP knowledge search now returns parent_content (falling back to child content when absent) with a 1500-char cap, up from the previous 300-char child slice.

IMPACT RADIUS

• execute_remember is the sole MCP entry point for memory search — all agent remember calls now respect temporal scoping.
• KnowledgeProvider::execute_search is the sole MCP entry point for knowledge search — response size per hit increased from 300 to 1500 chars (5×).

RISK
🟡 MEDIUM. The knowledge search response size increase is significant for multi-result responses: 10 hits × 1500 chars = 15K chars vs the previous 3K. Parent content is bounded at 4 * chunk_size (4800 chars) at write time, so the 1500-char cap is a floor on the ceiling, but the 5× increase in per-hit payload could strain agent token budgets on broad queries. The temporal filters are additive (absent = no-op) and low-risk.

📎 Source

• src/mcp/memory.rs#L401-L419 — temporal/relevance filter wiring
• src/mcp/memory.rs#L616-L630 — parse_iso_datetime helper
• src/mcp/knowledge.rs#L96-L117 — parent_content return with 1500-char cap
• src/mcp/server.rs#L507-L514 — RememberParams new fields

Card 5/5: Dead config activated — similarity_threshold and max_results now drive retrieval · 🟡 · ●●●

INTENT
search.similarity_threshold and search.max_results were parsed, defaulted, and documented in config but never read by any search path — pure dead config. This commit wires them into actual behavior, creating a distinct two-tier result-limiting model.

WHAT CHANGED

• similarity_threshold (default 0.3) is now the default min_relevance floor for semantic queries (those with query_text) in both vector_search and hybrid_search. Filter-only listings (no query_text — recent, by-type, by-tags) keep a 0.0 floor so they aren't silently truncated.
• search.max_results (default 50) is now the hard ceiling enforced via .min() in both vector_search and hybrid_search. Previously hybrid_search never capped an explicit caller limit — a limit=100 request would return 100.
• memory.max_search_results retains its role as the default page size when no limit is given. The two are now distinct: max_search_results = default, search.max_results = ceiling.
• config-templates/default.toml comments updated to clarify the two roles.

IMPACT RADIUS

• Every vector_search and hybrid_search call now applies the 0.3 default floor for text queries. Existing deployments where min_relevance was never set (defaulting to 0.0) will see fewer results — memories scoring 0.0–0.29 are now filtered out of semantic search.
• hybrid_search callers passing limit > 50 will now receive at most 50 results. This is a BREAKING change for any caller relying on large limits.

RISK
🟡 MEDIUM. The similarity_threshold activation is a silent behavior change for all existing deployments: semantic queries that previously returned low-relevance results (0.0–0.29) now return nothing. Combined with the RRF scale change (Card 2), the effective filtering landscape shifts significantly. The max_results cap on hybrid_search is a breaking change for callers passing large limits — they silently get truncated.

QUESTIONS

• Was the 0.3 default chosen to match the old effective behavior after the RRF scale change, or independently? The two changes (Card 2's 1/k normalization + this threshold activation) interact: the new normalization pushes scores up, while the threshold filters at 0.3. If tuned independently, the combined effect may over- or under-filter.

📎 Source

• src/memory/store.rs#L864-L878 — vector_search limit + min_relevance with similarity_threshold
• src/memory/store.rs#L1126-L1136 — hybrid_search limit + min_relevance
• config-templates/default.toml#L35-L48 — updated config comments

---

📂 Files changed (11 files, ~333 lines)

• config-templates/default.toml — Config comments clarified for similarity_threshold and max_results roles
• src/knowledge/store.rs — RRF normalization fix mirrored from memory store
• src/mcp/knowledge.rs — Knowledge search returns parent_content with 1500-char cap
• src/mcp/memory.rs — Temporal/relevance filter wiring + parse_iso_datetime helper
• src/mcp/server.rs — RememberParams new fields + memorize/remember tool descriptions
• src/memory/fusion_tests.rs — New file: RRF fusion unit tests
• src/memory/manager.rs — remember_multi RRF rewrite + suppress_superseded + rrf_fuse
• src/memory/mod.rs — Module registration for fusion_tests
• src/memory/reranker_integration.rs — top_n param + graceful degradation
• src/memory/role_tests.rs — Updated predicate assertions for archived exclusion
• src/memory/store.rs — RRF normalization, archived exclusion, HyDE guard, config wiring, reranker top_n

---

_{{tokens:150356,cost:0.1675908}}

[github-actions]

Octomind — developer:brief (octohub:glm)

📦 Brief: BEIR retrieval benchmarking harness + cross-encoder reranking for knowledge search + ingestion refactor

Overall risk: 🟡 MEDIUM · Cards: 4

#	Change	Risk	Confidence
1	New beir_bench binary evaluates retrieval quality (nDCG@10, Recall, MRR) against BEIR datasets	🟢	●●●
2	Knowledge search now pulls wider candidate sets and applies cross-encoder reranking when enabled	🟡	●●●
3	Ingestion pipeline unified via ChunkedDocument struct + shared embed_and_store helper	🟢	●●●
4	[search.hyde] config section is now required (removed #[serde(default)])	🟡	●●●

---

Card 1/4: BEIR retrieval benchmark harness · 🟢 · ●●●

INTENT
Add a local, no-LLM retrieval-quality benchmark to validate octobrain's knowledge ranking layer against standard BEIR datasets, reproducing pytrec_eval metrics.

WHAT CHANGED

• New beir_bench binary (src/bin/beir_bench.rs, 455 lines) gated behind a bench cargo feature — never compiles into normal/release builds.
• Indexes a BEIR corpus through the real KnowledgeStore retrieval path, then sweeps three scenarios (vector, hybrid, hybrid+rerank) against the same index, emitting nDCG@10 / Recall@10 / Recall@100 / MRR@10 as JSON.
• KnowledgeStore::open_at(db_path, dim) extracted from new() to allow opening an isolated LanceDB at a temp dir — the benchmark never touches the user's real knowledge DB. new() now delegates to open_at.
• KnowledgeStore::bulk_store (#[cfg(feature = "bench")]) ingests pre-embedded passages in 2048-row batches with a single index rebuild at the end — far faster than per-source store_chunks.
• benches/scripts/run_retrieval.sh downloads BEIR datasets, generates scenario configs from config-templates/default.toml via awk, and renders a results table with Python.

IMPACT RADIUS
open_at is now the production path — KnowledgeStore::new() calls it. The extraction is a pure refactor with no behavioral change. bulk_store is feature-gated and never reaches production builds.

RISK
🟢 LOW. Benchmark-only binary, feature-gated store methods. The open_at extraction is trivially reversible and delegates identically to the old inline code.

DIVERGENCE
🧩 Incomplete change — reranker is a no-op. The benchmark's own findings document that hybrid+rerank returned byte-identical metrics to hybrid for two different fastembed cross-encoders, with zero errors. The default config ships with enabled = true and model = "fastembed:jina-reranker-v2-base-multilingual", meaning the reranker is silently doing nothing in production for both memory and knowledge search. The commit adds reranking to knowledge search (Card 2) while documenting it doesn't work.

📎 Source

• src/bin/beir_bench.rs#L1-L455 — full benchmark binary
• src/knowledge/store.rs#L56-L80 — open_at extraction (now production path)
• src/knowledge/store.rs#L297-L386 — bulk_store (bench-gated)
• benches/README.md — reranker no-op finding documented under "Findings"

Card 2/4: Cross-encoder reranking added to knowledge search · 🟡 · ●●●

INTENT
Bring knowledge search to parity with memory search by applying cross-encoder reranking as a final post-processing step, pulling a wider candidate set for the reranker to work with.

WHAT CHANGED

• KnowledgeManager::search() now checks search_config.reranker.enabled and, when active, fetches top_k_candidates.max(max_results) candidates instead of max_results, then reranks down to max_results.
• New KnowledgeManager::rerank() method mirrors RerankerIntegration::rerank_memories() — calls octolib::reranker::rerank, applies timeout, degrades gracefully to pre-rerank order on any failure.
• Rerank uses parent_content (the full section) when available, falling back to chunk content — richer signal than the child chunk alone.
• Empty queries (filter-only listings) skip reranking entirely.

IMPACT RADIUS
Every KnowledgeManager::search() call is affected when reranker is enabled (which it is by default in config-templates/default.toml). The MCP knowledge tool's search command delegates here, so all agent knowledge searches now run through the reranker path. The memory store already had this pattern (src/memory/store.rs search_memories), so this aligns the two systems.

RISK
🟡 MEDIUM. The default config has enabled = true, so this change is active for all users after upgrade. The benchmark (Card 1) documented that the fastembed reranker path yields degenerate scores that never reorder — meaning this code runs on every knowledge search, adds latency (cross-encoder inference + timeout wrapper), and produces no ranking improvement. The graceful degradation is correct (timeout/failure → pre-rerank order), so no correctness regression, but the cost/benefit is currently zero.

QUESTIONS
Is the reranker no-op a known octolib/fastembed-rs issue being tracked separately, or should enabled default to false until it's fixed? The benchmark findings say "confirm with a focused octolib/fastembed-rs repro and fix before relying on reranking" — but the default config ships it on.

📎 Source

• src/knowledge/manager.rs#L100-L168 — rerank() method
• src/knowledge/manager.rs#L201-L232 — search() with wider candidate fetch + rerank
• src/memory/reranker_integration.rs#L69-L143 — existing memory rerank pattern (mirrored)
• config-templates/default.toml#L77-L99 — enabled = true default

Card 3/4: Ingestion pipeline unified via ChunkedDocument + embed_and_store · 🟢 · ●●●

INTENT
Eliminate duplicated embed-and-store boilerplate across five indexer call sites in KnowledgeManager by extracting a shared helper and a structured return type from the chunker.

WHAT CHANGED

• ContentChunker::extract_and_chunk() now returns ChunkedDocument (a struct with title, content_hash, chunks) instead of a 3-tuple (String, String, Vec<KnowledgeChunk>).
• New KnowledgeManager::embed_and_store() private method wraps the embed-batch + store_chunks sequence, returning usize (chunk count).
• All five extract_and_chunk call sites (index_source, index_source_internal, store_content, box reindex, freshness check) now use embed_and_store instead of inline embedding + storing.
• store_content simplified: the "content too small for chunker" fallback now builds a single KnowledgeChunk inline and passes it through embed_and_store, instead of duplicating the embed + store logic.

IMPACT RADIUS
Isolated change — embed_and_store is private (async fn, no pub), ChunkedDocument is only constructed inside the chunker. All callers are within src/knowledge/manager.rs and src/knowledge/chunker.rs. No external API surface changes.

RISK
🟢 LOW. Pure refactor — every call site produces the same store_chunks call with the same arguments. The store_content small-content path changed from single-embedding to batch-embedding (via embed_and_store), but generate_embeddings_batch with a 1-element vec is equivalent to generate_embedding.

📎 Source

• src/knowledge/chunker.rs#L10-L16 — ChunkedDocument struct
• src/knowledge/manager.rs#L62-L98 — embed_and_store helper
• src/knowledge/manager.rs#L497-L534 — store_content simplified path

Card 4/4: [search.hyde] config section now required · 🟡 · ●●●

INTENT
Align the hyde field with the strict config philosophy — Config::load() is documented as "all fields must exist in TOML", but #[serde(default)] on hyde was silently making it optional.

WHAT CHANGED

• Removed #[serde(default)] from SearchConfig::hyde field. The [search.hyde] section is now mandatory in config.toml.

IMPACT RADIUS
Config::load() is called by every entry point — CLI, MCP server, and the benchmark binary. The template (config-templates/default.toml) already contains [search.hyde], so fresh installs and configs copied from the template are unaffected. Users with older config.toml files that predate the HyDE feature (added in commit a8fde46) and were never updated will now get a parse failure at startup.

RISK
🟡 MEDIUM. This is a silent startup failure for users with stale configs — the error message from toml::from_str will point at a missing field, but there's no migration path or helpful "run octobrain to copy the latest template" hint. The strict config design means this is by design (the template is the source of truth), but the failure mode is a hard crash on first run after upgrade.

QUESTIONS
Should Config::load() detect a missing [search.hyde] and auto-merge it from the template, or is the hard failure intentional to force users to review new config options?

📎 Source

• src/config.rs#L48-L52 — #[serde(default)] removed
• config-templates/default.toml#L101-L104 — [search.hyde] section present in template

📂 Files changed (10 files, ~932 lines)

• Cargo.toml — bench feature + beir_bench binary target
• README.md — benchmark results table + reproduction instructions
• benches/.gitignore — ignore downloaded BEIR datasets
• benches/README.md — retrieval benchmark docs + reranker no-op finding
• benches/scripts/run_retrieval.sh — automated dataset download + scenario sweep
• src/bin/beir_bench.rs — BEIR benchmark binary (new, 455 lines)
• src/config.rs — removed #[serde(default)] from hyde
• src/knowledge/chunker.rs — ChunkedDocument struct, return type change
• src/knowledge/manager.rs — embed_and_store + rerank + wider candidate fetch
• src/knowledge/store.rs — open_at extraction + bulk_store (bench-gated)

---

_{{tokens:523572,cost:0.5597224000000001}}

[github-actions]

Octomind — developer:brief (octohub:glm)

The evidence from the tool output was:

[evidence: shell grep of use octobrain:: in src/] «src/bin/beir_bench.rs:22:use octobrain::config::Config;
src/bin/beir_bench.rs:23:use octobrain::knowledge::store::KnowledgeStore;
src/bin/beir_bench.rs:24:use octobrain::knowledge::types::KnowledgeSearchResult;»

My earlier quote used / as a line separator and stripped the file:line: prefixes, which broke the verbatim match. The claim itself is correct — beir_bench.rs already imported via octobrain:: before this change — and the exact tool output above supports it. No claim needs to be dropped; the citation is now grounded in the literal output.

---

_{{tokens:698635,cost:0.7076153999999998}}

[github-actions]

Octomind — developer:brief (octohub:glm)

📦 Brief: Cargo.lock dependency bump — version upgrades + removal of unused wasm/wit-bindgen tree

Overall risk: 🟢 LOW · Cards: 1

#	Change	Risk	Confidence
1	Lockfile-only dependency bumps: octolib, fastembed, safetensors, syn, quote, jiff + removal of unused wit-bindgen 0.51 / wasm toolchain	🟢	●●●

Card 1/1: Lockfile dependency bump — no source changes, two version splits introduced · 🟢 · ●●●

INTENT
Routine dependency maintenance — bump patch/minor versions and remove unused transitive crates that were pulled in by a now-gone wasip3 dependency.

WHAT CHANGED

• octolib 0.23.2 → 0.23.6 (direct dep, 4 patch versions — embedding/reranking library)
• fastembed 5.16.2 → 5.17.2 (embedding model runtime, transitive via octolib)
• safetensors split into two coexisting versions: 0.7.0 (candle-core/nn) and 0.8.0 (fastembed). Previously only 0.7.0 existed.
• syn 2.0.117 → 2.0.118, quote 1.0.45 → 1.0.46 (proc-macro deps, ~30 consumers updated)
• jiff 0.2.28 → 0.2.29 — gained defmt as a new dependency, which pulls in bitflags 1.3.2 (now coexists with bitflags 2.13.0), defmt-macros, defmt-parser, proc-macro-error2, proc-macro-error-attr2
• Removed entirely: wit-bindgen 0.51.0 + its toolchain (wit-bindgen-core, wit-bindgen-rust, wit-bindgen-rust-macro, wit-component 0.244.0, wit-parser 0.244.0, wasm-encoder 0.244.0, wasm-metadata 0.244.0, wasmparser 0.244.0), wasip3, id-arena, leb128fmt, unicode-xid. Only wit-bindgen 0.57.1 remains (used by wasip2).
• Other minor bumps: brotli 8.0.3→8.0.4, bytes 1.11.1→1.12.0, rustls 0.23.40→0.23.41, h2 0.4.14→0.4.15, quinn-proto 0.11.14→0.11.15, bitvec 1.0.1→1.1.1, bon 3.9.2→3.9.3, log 0.4.32→0.4.33, cc 1.2.64→1.2.65, arrayvec 0.7.6→0.7.7, pulp 0.22.2→0.22.3, webpki-root-certs 1.0.7→1.0.8, zlib-rs 0.6.3→0.6.4

IMPACT RADIUS
No source code changes — Cargo.lock only. No .rs files touched. The octobrain crate's direct dependency list is unchanged (still octolib, fastembed via octolib, etc.); only resolved versions shifted. The wit-bindgen 0.51 tree removal is safe — wasip3 (its sole consumer) was already unused and is removed alongside it.

RISK
🟢 LOW. Lockfile-only change with no API surface modifications. Two version splits (safetensors 0.7/0.8, bitflags 1.3/2.13) slightly increase compile time and binary size but are functionally isolated — each consumer pins its own version. The octolib 0.23.2→0.23.6 bump is the only direct-dep change worth noting, but it's 4 patch versions with no breaking signature changes visible in the lockfile.

QUESTIONS
None — commit message accurately describes the diff.

📎 Source

• Cargo.lock — lockfile with bumped versions and removed wit-bindgen 0.51 tree

📂 Files changed (1 file, ~467 lines)

• Cargo.lock — dependency lockfile; 195 insertions, 272 deletions (net reduction from removed wit-bindgen 0.51 ecosystem)

---

_{{tokens:4678555,cost:4.7127496}}