Merge pull request #114 from tinyhumansai/section-memory-api

Add the section API: conversations, learnings, documents, and recall

Author: senamakel

README.md

@@ -215,6 +215,41 @@ Capabilities are asked **once, at bind time, and cached**: a host filters its RP
 surface and its agent-tool list from the answer, so a set that changed
 afterwards would not be noticed.
 
+## The section surface
+
+Namespaces follow a `<section>:<scope>` convention — `conversation:thread-8f21`,
+`learning:rust-async`, `document:handbook` — so "conversational memory",
+"document memory" and "learnings" mean the same thing to every host and every
+engine. `tinymemory::sections` makes that convention a typed surface instead of
+a string every caller concatenates by hand:
+
+```rust
+use tinymemory::sections::Sections;
+
+let sections = Sections::new(provider.as_ref());
+
+sections.conversations().put("thread-8f21", "turn-1", text, category, None, taint).await?;
+let topics = sections.learnings().scopes().await?;
+let hits = sections.recall().across_section(&MemorySection::Learning, "async", 10, &opts, None).await?;
+```
+
+`conversations()`, `learnings()` and `documents()` are the three sections a host
+writes to routinely; `section()` reaches the other four and `Custom`. Every call
+composes the **mandatory** families only, so the whole surface works on every
+driver — nothing to negotiate, and no capability-absent path. On a driver that
+retains nothing, every call succeeds and returns empty.
+
+`across_section` is a fan-out: one namespace enumeration plus one recall per
+scope, capped, reporting what it searched and whether the cap bit. It is not an
+unfinished optimisation — `OwnedRecallOpts::namespace` is an exact match, and
+leaving it unset means the `global` namespace on the embedded engine but *every*
+namespace on the reference driver, so there is no cross-namespace recall to build
+a single call on. See [`docs/specs/memory-section-api.md`](docs/specs/memory-section-api.md).
+
+Handing the layer a **file** is a different path: `DocumentIntake` sniffs the
+format, converts it, and picks the capability family. The section surface is for
+text you already hold.
+
 ## What lives here, and what deliberately does not
 
 | Here | In the host |

crates/tinymemory/examples/tinycortex.rs

@@ -1,4 +1,5 @@
-//! The embedded engine, end to end: admit, construct, audit, store, recall.
+//! The embedded engine, end to end: admit, construct, audit, store, recall,
+//! and the same store read back through the section surface.
 //!
 //! Run with:
 //!
@@ -19,7 +20,9 @@ use std::sync::Arc;
 use tinymemory::api::provider::{audit_provider, MemoryProvider};
 use tinymemory::api::recall::OwnedRecallOpts;
 use tinymemory::api::types::{MemoryCategory, MemoryTaint};
+use tinymemory::namespace::MemorySection;
 use tinymemory::registry::{ConfigLabels, DriverRegistry, TINYCORTEX_DRIVER_ID};
+use tinymemory::sections::Sections;
 use tinymemory::tinycortex::{provider, InMemoryMemoryStore};
 
 #[tokio::main]
@@ -59,5 +62,44 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
     let hits = provider.recall("hello", 8, &opts, None).await?;
     println!("recall found {} entr(y/ies)", hits.len());
     assert!(!hits.is_empty(), "the stored entry must be recallable");
+
+    // 5. The same engine through the section surface: the caller names a
+    //    scope, never a namespace, and asks the whole section one question.
+    let sections = Sections::new(provider.as_ref());
+    for (scope, note) in [
+        ("rust-async", "pinning is not unpinning"),
+        ("rust-macros", "hygiene is per-expansion"),
+    ] {
+        let namespace = sections
+            .learnings()
+            .put(
+                scope,
+                "note",
+                note,
+                MemoryCategory::Core,
+                None,
+                MemoryTaint::Internal,
+            )
+            .await?;
+        println!("learning stored in '{namespace}'");
+    }
+
+    let found = sections
+        .recall()
+        .across_section(
+            &MemorySection::Learning,
+            "is",
+            8,
+            &OwnedRecallOpts::default(),
+            None,
+        )
+        .await?;
+    println!(
+        "section recall searched {} namespace(s) and found {} hit(s)",
+        found.namespaces_searched,
+        found.hits.len()
+    );
+    assert_eq!(found.namespaces_searched, 2, "both scopes must be searched");
+    assert!(!found.hits.is_empty(), "the section recall must find them");
     Ok(())
 }

crates/tinymemory/src/lib.rs

@@ -19,6 +19,10 @@
 //!   rather than re-deriving the same four subtleties.
 //! - **[`registry`]** — driver admission. Which driver ids exist, what class
 //!   each binds as, and the fail-closed rule for out-of-process drivers.
+//! - **[`sections`]** — typed surfaces for the sections the namespace
+//!   convention names: conversations, learnings, documents, and a
+//!   section-aware recall. Composes the mandatory families only, so it works
+//!   on every driver.
 //! - **Engine adapters** — one crate per engine under `crates/`, each
 //!   implementing [`provider::MemoryProvider`] over a concrete engine, and
 //!   each selected by the feature named after it.
@@ -143,12 +147,18 @@ pub use tinymemory_conformance as conformance;
 
 pub mod registry;
 
+// Typed surfaces for the sections the namespace convention names —
+// conversations, learnings, documents — plus a section-aware recall. Documented
+// by its own `//!` docs; an outer doc comment here as well would merge the two
+// and resolve the module's intra-doc links in this file's scope instead.
+pub mod sections;
+
 // The contract, re-exported wholesale. Listed module by module rather than as a
 // glob so the crate's own surface is visible in one place and rustdoc links
 // resolve — and so adding a module to the contract is a deliberate act here too.
 pub use tinymemory_api::{
-    capabilities, chunks, error, goals, health, null, provider, recall, tool_memory, traits, tree,
-    types,
+    capabilities, chunks, error, goals, health, namespace, null, provider, recall, tool_memory,
+    traits, tree, types,
 };
 pub use tinymemory_api::{is_compatible, CONTRACT_VERSION};
 

crates/tinymemory/src/sections/README.md

@@ -0,0 +1,105 @@
+# `sections`
+
+Typed surfaces over the `<section>:<scope>` namespace convention
+(`crates/tinymemory-bus/src/namespace.rs`): `Sections`, `SectionView`, and
+`SectionRecall`. Nothing here is a new capability — every call composes
+`MemoryCore` and `MemoryRecall`, which every driver implements as supertraits —
+this module only stops a caller from hand-concatenating the `conversation:` /
+`learning:` / `document:` prefix, where a typo silently produces a different,
+valid namespace instead of an error.
+
+## Design
+
+```text
+Sections::new(provider)
+  ├── conversations()  ─┐
+  ├── learnings()       ├─ SectionView  put / get / forget / list
+  ├── documents()       │                scopes / list_section
+  ├── section(custom)  ─┘
+  └── recall()          ── SectionRecall  in_scope / across_section
+```
+
+- `Sections` is the entry point: one named accessor per routine section
+  (`conversations`, `learnings`, `documents`) plus `section(&MemorySection)` for
+  the rest of the vocabulary (`entity:`, `profile:`, `tool:`, `source:`, and
+  `Custom`) and `recall()` for the cross-cutting query surface.
+- `SectionView` addresses one section by scope — `put` / `get` / `forget` /
+  `list` take the bare scope (`"thread-8f21"`), never the prefixed namespace —
+  and enumerates it with `scopes()` / `list_section()`.
+- `SectionRecall` answers two different questions, deliberately kept apart
+  because they cost different amounts: `in_scope` is one provider call;
+  `across_section` fans out to one call per namespace in the section.
+
+Every handle borrows `&dyn MemoryProvider` (see `view.rs`, `recall.rs`): cheap
+to construct, holds no state between calls, and cannot outlive the provider —
+so a caller builds one where it is needed instead of threading it through a
+struct.
+
+`MemorySection` is normalised through `MemorySection::from_prefix` in
+`SectionView::new`, so `Custom("conversation")` and `MemorySection::Conversation`
+are the same view rather than two. Storing the caller's spelling verbatim would
+let a write land under `conversation:` while a `scopes()` call — which compares
+against this normalised field — reported the section as empty.
+
+## Public surface
+
+- `Sections::{new, conversations, learnings, documents, section, recall}`
+- `SectionView::{put, get, forget, list, scopes, list_section}`
+- `SectionRecall::{in_scope, across_section}`
+- `SectionScope`, `SectionHits` — the value types `scopes()` / recall return
+- `MAX_SECTION_NAMESPACES` — the fan-out cap `across_section` enforces
+- `NAMESPACE_FILTER_CONFLICT`, `CROSS_SESSION_SECTION_CONFLICT`,
+  `CROSS_SESSION_FAN_OUT_CONFLICT` — the exact `MemoryError::Invalid` messages
+  the recall refusals carry, exposed so a caller's test can assert against the
+  same string it sees
+
+## Operational constraints
+
+**`across_section` is a fan-out, not a filtered call.** `OwnedRecallOpts::namespace`
+is exact-match, and `namespace: None` means the literal `global` namespace on
+the embedded engine but *every* namespace on the reference driver
+(`crates/tinymemory-conformance/src/reference/mod.rs`). A single unfiltered call
+plus post-filtering would return nothing in production, so `across_section`
+enumerates `scopes()` and issues one exact-namespace recall per scope instead,
+capped at `MAX_SECTION_NAMESPACES` and reported through `SectionHits::truncated`
+when the cap bites. Each namespace is asked for the full `limit`, never a
+share of it — a share would let one scope's best hit lose to another's worst.
+
+**`cross_session` and `session_id` are refused outside the conversation
+section, and refused on `across_section` unconditionally.** The bundled
+`UnifiedMemory` driver's `cross_session` recall option surfaces episodic
+*conversational* rows from other sessions; its `session_id` option
+independently appends that session's episodic rows. Both relabel every such
+row with whichever namespace the call was pinned to, regardless of the
+option's own defaults. Honouring either on a `learning:` or `document:`
+section would therefore return conversational content mislabeled as that
+section's own hits, so `in_scope` rejects both with
+`CROSS_SESSION_SECTION_CONFLICT` — checked against the section's *normalised*
+form, so `Custom("conversation")` counts as `MemorySection::Conversation` —
+unless `section == MemorySection::Conversation`.
+
+`across_section` rejects both unconditionally, with
+`CROSS_SESSION_FAN_OUT_CONFLICT`, including on the conversation section. This
+is not merely the same hazard: the driver's episodic augmentation runs once,
+independent of the pinned namespace, so the fan-out would repeat the exact
+same rows once per scope in the merged result, crowding genuine hits out of
+`limit` — and it is also redundant even where it would not repeat, since
+`across_section` already visits every conversation scope on its own. A caller
+who wants cross-session or session-scoped recall uses `in_scope` instead,
+which issues exactly one call.
+
+**Visit order is by entry count descending, not recency.** `SectionScope::last_updated`
+is optional and no bundled driver currently populates it, so `scopes()` cannot
+order by recency today. This is deliberate and raised as an open question in
+`docs/specs/memory-section-api.md`, not an oversight.
+
+**This is not the document intake path.** `Sections::documents` writes through
+`MemoryCore`, for text a caller already holds. Handing the memory layer a
+*file* — sniffing its format, converting it to markdown, then choosing between
+`MemoryIngest`, `MemoryDocuments`, and `MemoryCore` — is `DocumentIntake`'s job
+in the `documents` module, which is the right entry point for an upload.
+
+**The `namespace: None` divergence between drivers is out of scope here.** The
+embedded engine and the reference driver disagree on what an unfiltered recall
+means, as noted above; fixing that divergence needs its own spec and is
+deliberately not attempted by this module.