diff --git a/hindsight-docs/docs/developer/api/recall.mdx b/hindsight-docs/docs/developer/api/recall.mdx index 00ae6b42d..725665b64 100644 --- a/hindsight-docs/docs/developer/api/recall.mdx +++ b/hindsight-docs/docs/developer/api/recall.mdx @@ -53,7 +53,7 @@ The natural language question or statement to search for. This is the only requi ### types -Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (consolidated knowledge synthesized over time). When omitted, all three types are searched. +Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (deduplicated, evidence-grounded beliefs consolidated from multiple memories). When omitted, all three types are searched. Each type runs the full four-strategy retrieval pipeline independently, so narrowing `types` reduces both the result set and query cost. @@ -79,7 +79,7 @@ Each type runs the full four-strategy retrieval pipeline independently, so narro :::tip About Observations -Observations are consolidated knowledge synthesized from multiple facts over time — patterns, preferences, and learnings the memory bank has built up. They are created automatically in the background after retain operations. +Observations are deduplicated, evidence-grounded beliefs consolidated from multiple facts — preferences, recurring patterns, and durable learnings the memory bank has built up. Each observation references its supporting memories (with exact quotes) and carries a computed freshness trend, and is refined rather than overwritten when new evidence arrives. They are created and maintained automatically in the background after retain operations. ::: ### budget diff --git a/hindsight-docs/docs/developer/configuration.md b/hindsight-docs/docs/developer/configuration.md index fa7740d5a..5b33b988d 100644 --- a/hindsight-docs/docs/developer/configuration.md +++ b/hindsight-docs/docs/developer/configuration.md @@ -985,7 +985,7 @@ For production deployments, use `s3`, `gcs`, or `azure` to avoid storing large b ### Observations (Experimental) {#observations} -Observations are consolidated knowledge synthesized from facts. +Observations are deduplicated, evidence-grounded knowledge consolidated from multiple facts. Each observation tracks its supporting memories, a proof count, and a computed freshness trend, and is refined — not overwritten — when new evidence arrives. | Variable | Description | Default | |----------|-------------|---------| @@ -1008,7 +1008,7 @@ Observations are consolidated knowledge synthesized from facts. **`HINDSIGHT_API_OBSERVATIONS_MISSION` — redefine what this bank synthesises** -By default, observations are durable, specific facts synthesized from memories — the kind of knowledge that stays true over time (preferences, skills, relationships, recurring patterns). Ephemeral state is filtered out. Contradictions are tracked with temporal markers. +By default, observations are durable, specific beliefs consolidated from memories — the kind of knowledge that stays true over time (preferences, skills, relationships, recurring patterns). Each one is grounded in the source memories that support it. Ephemeral state is filtered out. Contradictions are tracked with temporal markers rather than overwriting the prior belief. Set `HINDSIGHT_API_OBSERVATIONS_MISSION` to replace this definition entirely. Write a plain-language description of what observations should be for your use case. The LLM will use this instead of the default rules when deciding what to create or update. Leave it unset to keep the server default. diff --git a/hindsight-docs/docs/developer/index.mdx b/hindsight-docs/docs/developer/index.mdx index f986ce5ab..c247cbde1 100644 --- a/hindsight-docs/docs/developer/index.mdx +++ b/hindsight-docs/docs/developer/index.mdx @@ -96,11 +96,12 @@ graph LR ### Observation Consolidation -After memories are retained, Hindsight automatically consolidates related facts into **observations** — synthesized knowledge representations that capture patterns and learnings: +After memories are retained, Hindsight automatically consolidates related facts into **observations** — deduplicated, evidence-grounded beliefs that the bank has built up across many memories: -- **Automatic synthesis**: New facts are analyzed and consolidated into existing or new observations -- **Evidence tracking**: Each observation tracks which facts support it -- **Continuous refinement**: Observations evolve as new evidence arrives +- **Deduplication**: Overlapping facts are merged into a single durable observation instead of piling up as repeats +- **Evidence tracking**: Each observation references the source memories (with exact quotes) that support it, plus a proof count +- **Continuous refinement**: Observations are updated — not overwritten — when new evidence supports, contradicts, or extends them; history is preserved +- **Freshness trend**: Each observation carries a computed trend (stable / strengthening / weakening / stale) based on when its evidence arrived ### Mission, Directives & Disposition diff --git a/hindsight-docs/docs/developer/observations.mdx b/hindsight-docs/docs/developer/observations.mdx index e845aef93..b9024bdd2 100644 --- a/hindsight-docs/docs/developer/observations.mdx +++ b/hindsight-docs/docs/developer/observations.mdx @@ -8,7 +8,7 @@ import memoryBanksPy from '!!raw-loader!@site/examples/api/memory-banks.py'; # Observations: Knowledge Consolidation -After memories are retained, Hindsight automatically consolidates related facts into **observations** — synthesized knowledge representations that capture patterns and learnings. +After memories are retained, Hindsight automatically consolidates related facts into **observations** — deduplicated, evidence-grounded beliefs the bank has built up from multiple memories. Each observation tracks its supporting evidence (with exact quotes), a proof count, and a computed freshness trend, and is refined rather than overwritten when new evidence arrives. ```mermaid graph LR @@ -24,7 +24,7 @@ graph LR ## What Are Observations? -Observations are **consolidated knowledge** synthesized from multiple facts. Unlike raw facts which are individual pieces of information, observations represent patterns, preferences, and learnings that emerge from accumulated evidence. +Observations are **consolidated knowledge** built from multiple facts. Unlike raw facts — which are individual pieces of information — observations represent deduplicated beliefs, preferences, and learnings grounded in accumulated evidence. They are not summaries the LLM invents on the fly: each observation is backed by specific source memories, carries a proof count, and evolves as new evidence supports, contradicts, or extends it. | Raw Facts | Observation | |-----------|--------------| @@ -33,8 +33,10 @@ Observations are **consolidated knowledge** synthesized from multiple facts. Unl | "Alice recommends type hints" | | Observations provide: -- **Synthesis**: Patterns that emerge from multiple facts -- **Context**: Richer understanding than individual facts +- **Deduplication**: One durable belief instead of many overlapping facts +- **Grounding**: Every observation references the specific memories (with quotes) that support it +- **Evolution**: Refined as evidence strengthens, weakens, or contradicts it — history is preserved +- **Freshness signal**: A computed trend (stable / strengthening / weakening / new / stale) tells you whether the belief still holds - **Efficiency**: Condensed knowledge for faster retrieval --- diff --git a/hindsight-docs/src/pages/best-practices.mdx b/hindsight-docs/src/pages/best-practices.mdx index 0c793e045..903438fb6 100644 --- a/hindsight-docs/src/pages/best-practices.mdx +++ b/hindsight-docs/src/pages/best-practices.mdx @@ -40,7 +40,7 @@ Banks are auto-created on first use. Configure them before ingesting data to ste | **Retain** | Ingests raw content (conversations, documents, notes). The LLM extracts facts, entities, and relationships — raw content is never stored verbatim. | After each conversation turn or session ends | | **Recall** | Retrieves relevant memories using 4 parallel strategies: semantic search, BM25, graph traversal, and temporal ranking. Returns a ranked list of facts. | Before generating a response that benefits from past context | | **Reflect** | Autonomous reasoning loop: searches memory, synthesizes an answer, and returns it directly. Uses mental models and observations hierarchically. | When you want Hindsight to answer a question, not just retrieve facts | -| **Observations** | Auto-synthesized knowledge patterns produced by the consolidation operation, which runs asynchronously after retain completes. Consolidate facts into durable insights (preferences, behavioral patterns, contradictions). | Triggered automatically after retain — not part of the retain call itself | +| **Observations** | Deduplicated, evidence-grounded knowledge consolidated from multiple facts. Each observation tracks its supporting memories with exact quotes, proof counts, and a computed freshness trend (stable/strengthening/weakening/stale). Refined — not overwritten — when new evidence supports, contradicts, or extends them. | Triggered automatically after retain — not part of the retain call itself | | **Mental Models** | Pre-computed reflect responses stored for common queries. Return instantly and consistently. | Create for repeated high-traffic queries or slowly-changing user profiles | --- @@ -53,7 +53,7 @@ Facts extracted during retain are classified into three types: |------|-------------|---------| | `world` | General knowledge, external facts | "The Eiffel Tower is in Paris" | | `experience` | Personal events, user-specific facts | "User moved to Berlin in 2024" | -| `observation` | Consolidated patterns synthesized from facts | "User consistently prefers async communication" | +| `observation` | Consolidated belief grounded in multiple supporting facts; deduplicated and refined over time with tracked evidence and freshness | "User consistently prefers async communication (5 supporting memories, strengthening)" | Use `types` filtering in recall to target specific memory types. diff --git a/skills/hindsight-docs/references/best-practices.md b/skills/hindsight-docs/references/best-practices.md index f41ae7004..5fa76e9c9 100644 --- a/skills/hindsight-docs/references/best-practices.md +++ b/skills/hindsight-docs/references/best-practices.md @@ -34,7 +34,7 @@ Banks are auto-created on first use. Configure them before ingesting data to ste | **Retain** | Ingests raw content (conversations, documents, notes). The LLM extracts facts, entities, and relationships — raw content is never stored verbatim. | After each conversation turn or session ends | | **Recall** | Retrieves relevant memories using 4 parallel strategies: semantic search, BM25, graph traversal, and temporal ranking. Returns a ranked list of facts. | Before generating a response that benefits from past context | | **Reflect** | Autonomous reasoning loop: searches memory, synthesizes an answer, and returns it directly. Uses mental models and observations hierarchically. | When you want Hindsight to answer a question, not just retrieve facts | -| **Observations** | Auto-synthesized knowledge patterns produced by the consolidation operation, which runs asynchronously after retain completes. Consolidate facts into durable insights (preferences, behavioral patterns, contradictions). | Triggered automatically after retain — not part of the retain call itself | +| **Observations** | Deduplicated, evidence-grounded knowledge consolidated from multiple facts. Each observation tracks its supporting memories with exact quotes, proof counts, and a computed freshness trend (stable/strengthening/weakening/stale). Refined — not overwritten — when new evidence supports, contradicts, or extends them. | Triggered automatically after retain — not part of the retain call itself | | **Mental Models** | Pre-computed reflect responses stored for common queries. Return instantly and consistently. | Create for repeated high-traffic queries or slowly-changing user profiles | --- @@ -47,7 +47,7 @@ Facts extracted during retain are classified into three types: |------|-------------|---------| | `world` | General knowledge, external facts | "The Eiffel Tower is in Paris" | | `experience` | Personal events, user-specific facts | "User moved to Berlin in 2024" | -| `observation` | Consolidated patterns synthesized from facts | "User consistently prefers async communication" | +| `observation` | Consolidated belief grounded in multiple supporting facts; deduplicated and refined over time with tracked evidence and freshness | "User consistently prefers async communication (5 supporting memories, strengthening)" | Use `types` filtering in recall to target specific memory types. diff --git a/skills/hindsight-docs/references/developer/api/recall.md b/skills/hindsight-docs/references/developer/api/recall.md index 405ed6232..b7e120fb0 100644 --- a/skills/hindsight-docs/references/developer/api/recall.md +++ b/skills/hindsight-docs/references/developer/api/recall.md @@ -88,7 +88,7 @@ The natural language question or statement to search for. This is the only requi ### types -Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (consolidated knowledge synthesized over time). When omitted, all three types are searched. +Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (deduplicated, evidence-grounded beliefs consolidated from multiple memories). When omitted, all three types are searched. Each type runs the full four-strategy retrieval pipeline independently, so narrowing `types` reduces both the result set and query cost. @@ -151,7 +151,7 @@ hindsight memory recall my-bank "query" --fact-type world,observation > **💡 About Observations** > -Observations are consolidated knowledge synthesized from multiple facts over time — patterns, preferences, and learnings the memory bank has built up. They are created automatically in the background after retain operations. +Observations are deduplicated, evidence-grounded beliefs consolidated from multiple facts — preferences, recurring patterns, and durable learnings the memory bank has built up. Each observation references its supporting memories (with exact quotes) and carries a computed freshness trend, and is refined rather than overwritten when new evidence arrives. They are created and maintained automatically in the background after retain operations. ### budget Controls retrieval depth and breadth. Accepted values are `low`, `mid` (default), and `high`. Use `low` for fast simple lookups, `mid` for balanced everyday queries, and `high` when you need to find indirect connections or exhaustive coverage. diff --git a/skills/hindsight-docs/references/developer/configuration.md b/skills/hindsight-docs/references/developer/configuration.md index 83e869799..a6df2995b 100644 --- a/skills/hindsight-docs/references/developer/configuration.md +++ b/skills/hindsight-docs/references/developer/configuration.md @@ -985,7 +985,7 @@ For production deployments, use `s3`, `gcs`, or `azure` to avoid storing large b ### Observations (Experimental) {#observations} -Observations are consolidated knowledge synthesized from facts. +Observations are deduplicated, evidence-grounded knowledge consolidated from multiple facts. Each observation tracks its supporting memories, a proof count, and a computed freshness trend, and is refined — not overwritten — when new evidence arrives. | Variable | Description | Default | |----------|-------------|---------| @@ -1008,7 +1008,7 @@ Observations are consolidated knowledge synthesized from facts. **`HINDSIGHT_API_OBSERVATIONS_MISSION` — redefine what this bank synthesises** -By default, observations are durable, specific facts synthesized from memories — the kind of knowledge that stays true over time (preferences, skills, relationships, recurring patterns). Ephemeral state is filtered out. Contradictions are tracked with temporal markers. +By default, observations are durable, specific beliefs consolidated from memories — the kind of knowledge that stays true over time (preferences, skills, relationships, recurring patterns). Each one is grounded in the source memories that support it. Ephemeral state is filtered out. Contradictions are tracked with temporal markers rather than overwriting the prior belief. Set `HINDSIGHT_API_OBSERVATIONS_MISSION` to replace this definition entirely. Write a plain-language description of what observations should be for your use case. The LLM will use this instead of the default rules when deciding what to create or update. Leave it unset to keep the server default. diff --git a/skills/hindsight-docs/references/developer/index.md b/skills/hindsight-docs/references/developer/index.md index e84787f80..84cc837ae 100644 --- a/skills/hindsight-docs/references/developer/index.md +++ b/skills/hindsight-docs/references/developer/index.md @@ -91,11 +91,12 @@ graph LR ### Observation Consolidation -After memories are retained, Hindsight automatically consolidates related facts into **observations** — synthesized knowledge representations that capture patterns and learnings: +After memories are retained, Hindsight automatically consolidates related facts into **observations** — deduplicated, evidence-grounded beliefs that the bank has built up across many memories: -- **Automatic synthesis**: New facts are analyzed and consolidated into existing or new observations -- **Evidence tracking**: Each observation tracks which facts support it -- **Continuous refinement**: Observations evolve as new evidence arrives +- **Deduplication**: Overlapping facts are merged into a single durable observation instead of piling up as repeats +- **Evidence tracking**: Each observation references the source memories (with exact quotes) that support it, plus a proof count +- **Continuous refinement**: Observations are updated — not overwritten — when new evidence supports, contradicts, or extends them; history is preserved +- **Freshness trend**: Each observation carries a computed trend (stable / strengthening / weakening / stale) based on when its evidence arrived ### Mission, Directives & Disposition diff --git a/skills/hindsight-docs/references/developer/observations.md b/skills/hindsight-docs/references/developer/observations.md index f3c1a3a15..55fbc31e6 100644 --- a/skills/hindsight-docs/references/developer/observations.md +++ b/skills/hindsight-docs/references/developer/observations.md @@ -2,7 +2,7 @@ # Observations: Knowledge Consolidation -After memories are retained, Hindsight automatically consolidates related facts into **observations** — synthesized knowledge representations that capture patterns and learnings. +After memories are retained, Hindsight automatically consolidates related facts into **observations** — deduplicated, evidence-grounded beliefs the bank has built up from multiple memories. Each observation tracks its supporting evidence (with exact quotes), a proof count, and a computed freshness trend, and is refined rather than overwritten when new evidence arrives. ```mermaid graph LR @@ -18,7 +18,7 @@ graph LR ## What Are Observations? -Observations are **consolidated knowledge** synthesized from multiple facts. Unlike raw facts which are individual pieces of information, observations represent patterns, preferences, and learnings that emerge from accumulated evidence. +Observations are **consolidated knowledge** built from multiple facts. Unlike raw facts — which are individual pieces of information — observations represent deduplicated beliefs, preferences, and learnings grounded in accumulated evidence. They are not summaries the LLM invents on the fly: each observation is backed by specific source memories, carries a proof count, and evolves as new evidence supports, contradicts, or extends it. | Raw Facts | Observation | |-----------|--------------| @@ -27,8 +27,10 @@ Observations are **consolidated knowledge** synthesized from multiple facts. Unl | "Alice recommends type hints" | | Observations provide: -- **Synthesis**: Patterns that emerge from multiple facts -- **Context**: Richer understanding than individual facts +- **Deduplication**: One durable belief instead of many overlapping facts +- **Grounding**: Every observation references the specific memories (with quotes) that support it +- **Evolution**: Refined as evidence strengthens, weakens, or contradicts it — history is preserved +- **Freshness signal**: A computed trend (stable / strengthening / weakening / new / stale) tells you whether the belief still holds - **Efficiency**: Condensed knowledge for faster retrieval ---