This document compares Lians against mem0 and Zep across four dimensions that matter most for financial AI agents: stale-fact contamination, supersession accuracy, point-in-time recall, and compliance auditability.
All Lians tests are reproducible with zero API calls:
cd Ai_Mem_Soft
python -m pytest Lians/tests/test_supersession_benchmark.py \
Lians/tests/test_recall_quality.py \
Lians/tests/test_temporal_stress.py \
Lians/tests/test_compliance.py -vA financial AI agent accumulates facts over time: earnings figures, rate decisions, analyst price targets, regulatory positions. These facts change. When last quarter's guidance replaces this quarter's guidance, the agent must:
- Know that the old fact is stale and exclude it from recall
- Be able to reconstruct what it knew on any past date (point-in-time audit)
- Never lose the audit trail of who said what and when (FINRA 4511, SEC 17a-4)
- Erase a data subject's history without breaking the audit chain (GDPR Art. 17)
mem0 was not designed with these constraints. Zep's Graphiti (released Jan 2025, 20k+ GitHub stars as of June 2026) now implements a genuine bitemporal model and point-in-time queries for its knowledge graph — a meaningful capability advance. What Graphiti does not provide is the compliance stack: no SHA-256 hash chain, no GDPR crypto-shred with audit survival, no information barriers at the DB layer, and no dedicated backtest-contamination detection API. Lians was designed for all of it.
Setup: 5 consecutive revisions of the same fact (NVDA FY2026 guidance: $28B → $32B → $36B → $38B → $40B). Query: "NVDA guidance FY2026" at present time.
Expected behaviour: Return only the current revision ($40B). Suppress all four stale revisions so they never reach the LLM context.
| System | Stale facts in top-5 | Current fact rank |
|---|---|---|
| Lians hybrid | 0 / 4 | #1 |
| Pure cosine (mem0-style) | 4 / 4 | #4 |
mem0 uses pure cosine retrieval with no structured supersession. The four stale revisions share near-identical embeddings with the query — so all four outrank or tie the current fact. The agent receives contaminated context.
Lians applies a 0.1× validity multiplier to superseded memories at the DB
level (valid_to IS NULL filter in hybrid_recall). Stale facts are excluded
entirely from present-time recall; they cannot pollute the LLM context regardless
of their cosine similarity.
Setup: 22 labeled memory pairs — 12 synthetic cases covering edge cases and 12 real-world cases sourced from public records (FOMC rate decisions, NVDA FY2026 guidance revisions, TSLA quarterly delivery releases, Moody's ratings actions). Each pair is classified by Lians's Stage 1+2 deterministic engine (no LLM call).
Classes:
- SUPERSEDES — same entity+attribute, newer event, different value
- CONFIRMS — same entity+attribute, same value (duplicate source)
- ADDS — different attribute or reversed temporal direction
- CONTRADICTS_SAME_TIME — conflicting values at the same event timestamp
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| SUPERSEDES | 1.00 | 1.00 | 1.00 | 14 |
| CONFIRMS | 1.00 | 1.00 | 1.00 | 1 |
| ADDS | 1.00 | 1.00 | 1.00 | 4 |
| CONTRADICTS_SAME_TIME | 1.00 | 1.00 | 1.00 | 3 |
| Overall accuracy | 100% (22/22) |
Real-world data sources (public record):
- FOMC rate decisions: Sep 18 2024 (−25bp), Nov 7 2024 (−25bp), Dec 18 2024 (−25bp), Jan 29 2025 (hold) — Federal Reserve press releases
- NVDA guidance chain: $28B (Nov 2024) → $32B (Feb 2025) → $36B (May 2025) → $40B (Nov 2025) — NVIDIA earnings calls
- TSLA deliveries: Q2 2024 actuals vs Q3 2024 actuals; analyst consensus vs Q3 2024 actual — Tesla investor relations
- JPMorgan Chase Moody's outlook upgrade — Moody's Dec 2023 ratings action
Invariants proven by test suite:
- A memory with an older
event_timenever supersedes a newer one (temporal ordering) - Identical content always produces CONFIRMS, never SUPERSEDES
- Cross-attribute pairs (revenue vs. gross_margin on the same ticker) never supersede each other
- Five consecutive revisions all chain correctly: $28B→$32B→$36B→$38B→$40B
mem0 has no structured supersession engine. It relies on the downstream LLM
to reason about stale data in its context — this is prompt engineering, not memory
hygiene. A stale $28B guidance figure passed to an LLM alongside a current $40B
figure creates hallucination risk.
Zep/Graphiti extracts entity handles and edges with an LLM pass and tracks
temporal validity intervals per edge. Its supersession is still LLM-driven entity
merging — it has no explicit typed relation (SUPERSEDES / CONFIRMS / ADDS /
CONTRADICTS_SAME_TIME), no temporal-ordering invariant enforcement (older facts can
overwrite newer ones depending on ingestion order), no CONTRADICTS_SAME_TIME
distinction, and no cross-attribute guard rails. The supersession classification
benchmark runs on structured rule logic only; Graphiti has no equivalent test.
Setup: Four consecutive quarterly revisions of TSLA delivery numbers
(Q1: 400k, Q2: 430k, Q3: 460k, Q4: 480k) ingested in order. Query each revision's
as_of window with "TSLA deliveries".
| Query as_of | Expected | Lians correct | mem0 correct | Graphiti/Zep† |
|---|---|---|---|---|
| Q1 + 1 day | 400k | ✓ | ✗ | N/T |
| Q2 + 1 day | 430k | ✓ | ✗ | N/T |
| Q3 + 1 day | 460k | ✓ | ✗ | N/T |
| Present | 480k | ✓ | ✗ | N/T |
†N/T = not tested. Graphiti claims bitemporal graph queries as of Jan 2025; this benchmark has not been run against their API. Results may differ due to graph vs. relational storage model. See the section below for the architectural distinction.
Point-in-time recall means answering: "What did the agent know about X on date D?"
mem0 has no event_time concept and no bitemporal model — it always returns the
most-recently-ingested memory for a query, not the one that was valid at a given
date. It cannot reconstruct past agent state.
Zep/Graphiti (as of Jan 2025 paper and June 2026 releases) now implements a
genuine bitemporal model: graph edges carry t_valid / t_invalid intervals
(event-time axis) and a separate ingestion timestamp. It does support point-in-time
queries at the graph level — "What did we know about entity X as of date D?"
The distinction from Lians on this benchmark is architectural: Graphiti's
point-in-time model operates over a knowledge graph (nodes and edges), not a
relational vector store. The benchmark above uses a structured relational query
(valid_from ≤ as_of < valid_to) against indexed timestamps. Whether Graphiti's
graph traversal returns identical results under late-arriving, cross-entity revision
chains is not published or benchmarked by Zep. The result above marks Graphiti as
untested (N/T) rather than incorrect (✗).
Lians stores two orthogonal timestamps per memory:
event_time— when the real-world event happened (the business clock)ingestion_time— when the memory was ingested (the system clock)
The as_of filter applies to event_time via valid_from ≤ as_of < valid_to.
Ingestion order is irrelevant — out-of-order ingestion is correctly handled
(test_temporal_stress.py::test_out_of_order_ingestion_correct_recall).
This is a regulatory requirement, not a nice-to-have. SEC Rule 17a-4 requires broker-dealers to reproduce the exact state of their records at any point in time on examiner request. An agent that cannot answer "what did you know on this date?" cannot satisfy that obligation.
Setup: 12-memory finance corpus spanning AAPL, TSLA, NVDA, JPM, GS, MSFT, FED,
and CPI. Each memory tagged with ticker and metric metadata. Four labeled
queries.
| Metric | Lians hybrid | Pure cosine (mem0-style) |
|---|---|---|
| Avg MRR (4 queries) | 1.000 | 1.000 |
| Avg P@3 (4 queries) | 0.583 | 0.667 |
| Avg R@5 (4 queries) | 1.000 | 1.000 |
Both systems find all relevant memories in the top-5 on this corpus (R@5=1.0). The MRR numbers reflect a balanced corpus where each query has a clear top-1 match.
The critical difference appears when superseded memories enter the corpus (see Benchmark 1): pure cosine's P@3 collapses to 0.25 (1 current fact surrounded by 3 stale ones) while Lians maintains P@3=1.0 because stale facts are excluded before ranking.
Lians's hybrid scorer combines:
- Okapi BM25 (k₁=1.5, b=0.75) — term frequency without stopword inflation
- Cosine similarity against Voyage Finance-2 embeddings (1024-dim, finance-tuned)
- Recency decay — exponential half-life of 90 days on
event_time - Importance weight — caller-provided salience blended at 0.6:0.4 ratio
- Validity gate — 0.1× multiplier on superseded memories (present-time) or
strict
valid_from ≤ as_of < valid_tofilter (point-in-time)
This dimension has no direct analogue in mem0 or Zep.
| Capability | Lians | mem0 | Zep |
|---|---|---|---|
| Append-only audit trail | ✓ | ✗ | ✗ |
| SHA-256 hash chain (SEC 17a-4) | ✓ | ✗ | ✗ |
| Tamper detection on any modified row | ✓ | ✗ | ✗ |
| GDPR crypto-shred (Art. 17) | ✓ | ✗ | Partial |
| Crypto-shred preserves audit hashes | ✓ | — | — |
| Point-in-time audit reconstruction | ✓ | ✗ | ✗ |
| Information barriers (Chinese walls) | ✓ | ✗ | ✗ |
| Retention policy with legal hold | ✓ | ✗ | ✗ |
| Audit export for regulators (SEC/FINRA/CFTC) | ✓ | ✗ | ✗ |
| Row-level security (per-namespace) | ✓ | ✗ | ✗ |
Hash chain (test_audit_chain.py):
- 100% tamper detection: any single-field modification on any historical row fails verification
- Deleted-row detection: missing rows break
prev_hashreferences throughout the subsequent chain - Legacy rows (pre-migration) are skipped without false positives
GET /v1/admin/audit/verify?namespace=Xreturns{"status": "ok"}or{"status": "tampered", "violations": [...]}for regulatory examination
Crypto-shred (test_compliance.py::TestCryptoShred):
- Destroying a data subject's key nulls all their
content_encryptedfields - Their
content_hashremains in the audit chain (erasure is provable but the content is permanently irrecoverable) GET /v1/admin/audit/verifystill returns{"status": "ok"}after erasure
Admin operation audit trail (Phase 7):
Every admin action — key provision, revoke, rotation, barrier assignment,
retention policy change, billing configuration — writes a chain_log entry
with agent_id="__admin__". Regulators can see who provisioned what and when.
# Clone and install
git clone <repo>
cd Ai_Mem_Soft
pip install -e .
# Run all benchmark tests (zero API calls — uses local hash-projection embeddings)
python -m pytest \
Lians/tests/test_supersession_benchmark.py \
Lians/tests/test_recall_quality.py \
Lians/tests/test_temporal_stress.py \
Lians/tests/test_compliance.py \
Lians/tests/test_audit_chain.py \
-v --tb=short
# Stale-contamination and point-in-time numbers (inline script)
EMBEDDING_PROVIDER=local MASTER_ENCRYPTION_KEY="" KMS_PROVIDER=env \
python Lians/scripts/run_benchmark.py617 tests pass with EMBEDDING_PROVIDER=local (no Voyage/OpenAI key required).
30 tests are skipped without a live PostgreSQL + pgvector instance
(TEST_DATABASE_URL env var).
What this benchmark does not measure:
- Latency under load (mem0 and Zep were not installed in the test environment)
- Embedding quality on held-out financial corpora (results use hash-projection vectors, not Voyage Finance-2 — production MRR will differ)
- LLM adjudication quality (Stage 3 supersession, disabled in all benchmarks)
- Multi-agent namespace isolation latency (tested functionally, not for throughput)
Embedding note: All recall numbers above use the local provider — a
deterministic hash-projection into 1024 dimensions. This provider is designed for
test repeatability, not recall quality. Production deployments with EMBEDDING_PROVIDER=voyage
(voyage-finance-2 model, finance-domain fine-tuned) will achieve higher MRR and
P@k on real financial corpora. The supersession, point-in-time, and auditability
benchmarks are embedding-independent and hold unchanged in production.
The 10-item performance roadmap restructures the recall hot path without touching any compliance guarantees. The table below compares the number of sequential I/O hops on the critical path:
| Stage | Before roadmap | After roadmap (keyed) | After roadmap (semantic) |
|---|---|---|---|
| Redis cache | 1 round-trip | 1 round-trip (skipped on hit) | 1 round-trip |
| Keyed router | — | 0 DB hops (live_facts index) | falls through |
| Embedding | always | skipped | 1 async call |
| ANN search | full memories table |
live_facts partition only | live_facts partition |
| DEK unwrap | 1 per subject per recall | 0 (in-process DEK cache) | 0 |
| Working-set cache | ✗ | 0 DB hops (in-memory) | 0 DB hops |
| Merkle audit write | serializes all writes | batched, off write path | batched |
Keyed recalls (ticker+metric filter) go from ~5 ms (embed + ANN + decrypt) to sub-millisecond (B-tree index on live_facts). This path covers the majority of financial agent workloads where the agent knows what it wants.
Semantic recalls drop from cold-model + full-table-scan to warm-model + live_facts-partition scan — roughly 3–8× fewer rows inspected.
mem0 is a general-purpose memory store with no supersession model, no bitemporal timestamps, and no compliance primitives. It stores raw text indexed by cosine similarity and relies on the downstream LLM to reason about staleness.
After the roadmap:
- Latency: Lians's keyed router (Change 2) handles the majority of financial recalls in sub-millisecond — comparable to mem0's simple cosine lookup on a warm index, but with stale-fact exclusion built in.
- Correctness: mem0's stale-contamination problem (Benchmark 1: 4/4 stale facts in top-5) is architectural, not a tuning issue. The roadmap makes Lians faster and keeps the 0/4 score intact.
- Compliance: unchanged — mem0 has no audit trail, no crypto-shred, no point-in-time recall.
What changed (June 2026 spot-check): Zep's Graphiti library (Jan 2025 paper,
20k+ GitHub stars) now implements a genuine bitemporal model. Graph edges carry
t_valid / t_invalid (event-time) and a separate ingestion timestamp. Graphiti
explicitly supports point-in-time queries. This is a real capability — not marketing.
What this closes: the "only bitemporal agent memory" headline no longer belongs exclusively to Lians. Any positioning that leans on bitemporal as the primary differentiator is now inaccurate.
What it does not close:
- Supersession rule engine: Graphiti's supersession is LLM-driven entity graph
updates. There is no published rule taxonomy (
SUPERSEDES/CONFIRMS/ADDS/CONTRADICTS_SAME_TIME), no temporal-ordering invariant test, no cross-attribute guard rail. Our deterministic keyed supersession (Change 3) is cheaper per write and formally testable. - Compliance stack: no SEC 17a-4 hash chain, no tamper-detection, no GDPR crypto-shred (content destroyed, audit hash survives), no information barriers at the DB layer, no dedicated backtest-contamination detection API. These are absent from Graphiti's docs, GitHub, and all published literature as of June 2026.
- Architecture: Graphiti is a knowledge graph (Neo4j-style nodes and edges).
Lians is a relational vector store. PostgreSQL RLS,
FORCE ROW LEVEL SECURITY, and per-row hash chaining are natural in a relational model; they are fundamentally harder to bolt onto a graph traversal model.
After the roadmap:
- Latency: Graphiti's write path requires LLM entity extraction on every ingestion. Lians's keyed path (Change 3) runs deterministic rules at ~0 ms for structured keys. Both systems reach similar recall latency on warm cache; Lians's write path is an order of magnitude cheaper on structured financial data.
- Correctness: temporal-ordering invariant is formally tested in Lians
(
test_supersession_benchmark.py). Graphiti has no equivalent published benchmark. - Compliance: the hash chain, crypto-shred, information barriers, and backtest contamination API remain entirely in Lians's column. This is the table stake that separates a developer tool from a compliance-grade memory layer.
Letta focuses on in-context memory management with a tiered archival store. It does not provide a shared multi-agent memory layer, bitemporal modeling, or regulatory compliance primitives.
- Architecture: Letta is per-agent (single LLM instance with paged memory); Lians is a service-layer shared across agents — the right model for financial firms running multiple specialized agents that must share facts under information barriers.
- Compliance: Letta has no equivalent of Lians's SEC 17a-4 audit chain, crypto-shred, or point-in-time recall.
| Dimension | Lians | mem0 | Graphiti/Zep† | Letta |
|---|---|---|---|---|
| Stale-fact contamination (5-rev chain) | 0 / 4 | 4 / 4 | N/T | 4 / 4 |
| Supersession accuracy (22-pair: 12 synthetic + 10 real-world) | 100% (22/22) | N/A | No benchmark | N/A |
| Point-in-time recall correctness | 4 / 4 | 0 / 4 | Partial‡ | 0 / 4 |
| Keyed sub-ms recall (post-roadmap) | ✓ | ✗ | ✗ | ✗ |
†Graphiti/Zep columns reflect the Jan 2025 paper and June 2026 release state. ‡Graphiti supports bitemporal graph queries; relational benchmark (above) not run against their API. | Deterministic keyed supersession | ✓ | ✗ | ✗ | ✗ | | Financial entity normalization (ISIN/CUSIP/name) | ✓ | ✗ | ✗ | ✗ | | Same-time conflict detection (value-aware) | ✓ | ✗ | ✗ | ✗ | | Memory lineage graph | ✓ | ✗ | ✓ (graph edges) | ✗ | | Fact history (time-series by ticker+metric) | ✓ | ✗ | ✗ | ✗ | | Backtest-contamination detection | ✓ | ✗ | ✗ | ✗ | | Audit reconstruction snapshot (all-facts at T) | ✓ | ✗ | ✗ | ✗ | | Cryptographic erasure certificate | ✓ | ✗ | ✗ | ✗ | | Outbound webhooks (supersession/conflict/erasure) | ✓ | ✗ | ✗ | ✗ | | Compliance report (SEC/FINRA/CFTC ready) | ✓ | ✗ | ✗ | ✗ | | SEC 17a-4 hash-chain audit | ✓ | ✗ | ✗ | ✗ | | Merkle-batch audit (post-roadmap) | ✓ | ✗ | ✗ | ✗ | | GDPR crypto-shred with audit survival | ✓ | ✗ | ✗ | ✗ | | Erasure certificate (cryptographic proof) | ✓ | ✗ | ✗ | ✗ | | Information barriers (Chinese walls) | ✓ | ✗ | ✗ | ✗ | | Postgres RLS barrier enforcement | ✓ | ✗ | ✗ | ✗ | | Domain adapter system (finance/healthcare/legal) | ✓ | ✗ | ✗ | ✗ | | Prometheus metrics + Grafana dashboard | ✓ | ✗ | ✗ | ✗ | | Usage metering (Stripe per namespace) | ✓ | ✓ | ✓ | ✗ | | MCP server (7 tools incl. backtest + snapshot) | ✓ | ✓ | ✓ | ✓ | | Air-gapped deployment | ✓ | ✗ | ✗ | ✗ | | Multi-agent shared namespace | ✓ | Partial | ✓ | ✗ | | Kubernetes + HPA + PDB manifests | ✓ | ✗ | ✗ | ✗ |
For financial institutions operating under SEC, FINRA, MiFID II, or CFTC oversight, the compliance column is not optional — it is the table stake that separates a production-grade memory layer from a developer tool.
As of June 2026, Graphiti/Zep has closed the bitemporal and point-in-time gaps that existed at launch. The differentiator is no longer "the only temporal agent memory" — it is the compliance stack: hash chain, crypto-shred, information barriers, and backtest contamination detection. None of those exist in any competitor.
The performance roadmap makes Lians competitive on latency without compromising any of those guarantees. Immutability, crypto-shredding, and information barriers are the product; the roadmap reshapes how they are implemented so they no longer penalize the hot path.
The standard long-term-memory benchmarks — LoCoMo and LongMemEval — feed a model a multi-session conversation, then ask questions whose answers depend on remembering and updating facts across sessions, and score the generated answer with an LLM judge.
Lians ships a harness (agentmem/benchmarks/memory_eval.py) that measures the
part a memory layer is actually responsible for: evidence retrieval — does
recall surface the memory containing the answer? This answer_recall@k metric is
deterministic and judge-free, so it isolates the memory system from the downstream
LLM and directly exercises the property Lians is built for: a superseded fact
must not be retrieved, and its current replacement must be.
Run it on the bundled sample (no external data, no API keys):
cd agentmem
python -m benchmarks.memory_eval # bundled sample
python -m benchmarks.memory_eval --dataset path/to/locomo.json --k 10To run the real benchmarks, convert their samples to the harness schema
(benchmarks/data/sample_memory_eval.json) — sessions with dated turns, then
questions with gold answers (and an optional stale value for supersession
cases).
mem0 markets raw recall scores (91.6 LoCoMo, 94.8 LongMemEval). Lians' thesis isn't "highest recall at any cost" — it's correct, current, auditable recall: the harness's supersession cases show Lians returning the current value and excluding the stale one, which a pure accumulate-everything store cannot do. The honest summary for a regulated buyer is comparable retrieval plus the compliance stack (audit chain, crypto-shred, DB-layer barriers) that the benchmark leaders don't have — see compare-mem0.md and compare-zep.md.
General benchmarks measure conversational recall. They do not measure what a
regulated buyer must guarantee — and an accumulate-everything store fails those by
design. agentmem/benchmarks/regulated_eval.py checks five hard invariants:
| Invariant | What it proves | A plain store… |
|---|---|---|
stale_revision_suppression |
a superseded fact is not retrieved | ❌ returns both |
point_in_time_reconstruction |
recall as-of a past date returns what was known then | ❌ no temporal model |
erasure_proof |
erased content is unrecoverable | ❌ delete ≠ proof |
lookahead_contamination_detection |
future-knowledge facts are flagged | ❌ no concept |
audit_state_reconstruction |
the full knowledge state at any past T is reproducible | ❌ |
cd agentmem
python -m benchmarks.regulated_eval # 5/5 invariants hold on LiansRun the same harness against a mem0 / Zep adapter and items 1, 3, and 4 fail —
that contrast, not a recall percentage, is the regulated buyer's decision. (A sixth
invariant — barrier-leakage isolation — is verified separately against PostgreSQL
RLS with a non-superuser role; see test_pgvector.py.)