diff --git a/docs/features/F035-observability.md b/docs/features/F035-observability.md new file mode 100644 index 0000000..3b1c2cd --- /dev/null +++ b/docs/features/F035-observability.md @@ -0,0 +1,91 @@ +# F035: Observability — Knowing What the Mind Is Doing + +**Status:** PROPOSED +**Author:** Nous + Tim +**Created:** 2026-04-04 +**Dependencies:** F006 (Event Bus), F034 (Heartbeat), F026 (Execution Ledger) + +--- + +## Problem + +Nous now has 4+ autonomous subsystems that modify state without human initiation: + +- **Heartbeat** (F034) — monitors external services, creates findings, triggers triage +- **Sleep consolidation** — rewrites memory (compaction, reflection, contradiction resolution) +- **Self-tuning** (F034.3) — adjusts heartbeat check intervals based on yield +- **Fact/episode lifecycle** — admission, dedup, pruning all happen automatically + +Each system is individually well-designed. But when something unexpected happens — a fact disappears, a check stops running, behavior shifts — there's no way to answer: **"what chain of autonomous decisions led here?"** + +The event bus exists (F006) and persists events to DB, but there are no processing stats, no causal chains linking autonomous actions, and no trend detection for behavioral drift. Debugging requires reading raw logs and manually reconstructing causality. + +This is the difference between a system that *works* and a system you can *trust*. As Nous becomes more autonomous, observability isn't optional — it's the mechanism for accountable self-modification. + +--- + +## Design Philosophy + +The right mental model isn't "monitoring a server." It's closer to **journaling for a mind** — the system should be able to answer "why did I change my mind about X?" the same way a person with good self-awareness can. + +Minsky's Chapter 6 ("Self-Knowledge is Dangerous") warns that unrestricted self-modification leads to instability. The observability layer is the read-only self-knowledge that makes self-modification safe: you can see what happened and why, but the audit trail itself can't be modified by the systems it monitors. + +--- + +## Architecture — Three Layers + +``` +┌─────────────────────────────────────────────────────────┐ +│ Dashboard / API │ +│ (query any layer, visualize trends) │ +└────────────┬──────────────────┬──────────────────┬───────┘ + │ │ │ + ┌────────▼────────┐ ┌──────▼───────┐ ┌────────▼────────┐ + │ F035.1 │ │ F035.2 │ │ F035.3 │ + │ Event Bus │ │ Causal │ │ Behavioral │ + │ Stats │ │ Chains │ │ Drift │ + │ │ │ │ │ Detection │ + │ "Is the system │ │ "Why did │ │ "Is the system │ + │ healthy now?" │ │ this happen?"│ │ changing?" │ + └─────────────────┘ └──────────────┘ └─────────────────┘ + ▲ ▲ ▲ + │ │ │ + ┌────────┴──────────────────┴──────────────────┴───────┐ + │ Event Bus (F006) │ + │ All events flow through here │ + └──────────────────────────────────────────────────────┘ +``` + +**Layer 1 — Event Bus Stats (F035.1):** Real-time operational health. Event throughput, handler success/fail rates, queue depth. "Is the system healthy right now?" + +**Layer 2 — Causal Chain Tracing (F035.2):** Every autonomous action gets a `caused_by` link back to its trigger. Queryable audit trail. "Why did Nous do X?" + +**Layer 3 — Behavioral Drift Detection (F035.3):** Periodic snapshots of key metrics with trend analysis. Catches slow drift that individual events don't reveal. "Is the system changing in ways nobody intended?" + +--- + +## Sub-Specs + +| Spec | Title | Priority | Depends On | +|------|-------|----------|------------| +| F035.1 | Event Bus Observability | P1 | F006 | +| F035.2 | Autonomous Action Audit Trail | P1 | F035.1 | +| F035.3 | Behavioral Drift Detection | P2 | F035.2 | + +**Sequencing rationale:** F035.1 gives us the infrastructure (stats collection, endpoints). F035.2 adds causal metadata to events. F035.3 builds on both to detect trends. Each is independently useful. + +--- + +## What This Supersedes + +- **006.1 (Event Bus Observability)** — F035.1 absorbs and modernizes this planned spec. The original 006.1 was scoped before heartbeat, sleep consolidation, and self-tuning existed. F035.1 covers the same ground but accounts for the current architecture. + +--- + +## Success Criteria + +1. After any autonomous action, you can trace the full causal chain back to the originating trigger +2. Handler health (success/fail rates) is visible in real-time via API and Telegram +3. Behavioral trends (fact growth rate, censor changes, check frequency drift) are tracked and anomalies flagged +4. The dashboard has an "Autonomous Activity" panel showing recent self-modifications +5. None of this adds measurable latency to the hot path (event processing) diff --git a/docs/features/F035.1-event-bus-observability.md b/docs/features/F035.1-event-bus-observability.md new file mode 100644 index 0000000..221b2ee --- /dev/null +++ b/docs/features/F035.1-event-bus-observability.md @@ -0,0 +1,242 @@ +# F035.1 — Event Bus Observability + +**Status:** PROPOSED +**Priority:** P1 (foundation — F035.2 and F035.3 build on this) +**Depends on:** F006 (Event Bus), F035 (Observability umbrella) +**Supersedes:** 006.1 (Event Bus Observability — planned, never built) +**Author:** Nous + Tim +**Date:** 2026-04-04 + +--- + +## Problem + +The event bus processes all system events but has zero operational visibility. The only insight into event processing is: + +- `bus.pending` — queue depth (single int) +- `logger` calls scattered across handlers +- Raw rows in `nous_system.events` (unindexed, no aggregation) + +You can't answer basic questions: +- How many events processed in the last hour? +- Which handlers are failing? How often? +- Is the queue backing up? +- When did the sleep handler last run successfully? +- Are any handlers silently broken? + +--- + +## Goals + +1. **In-memory stats** — event counts by type, handler success/fail/timing, queue health +2. **Ring buffer** — last N events for quick inspection without DB queries +3. **REST endpoints** — `/events/stats` and `/events/recent` for API/dashboard access +4. **Telegram integration** — event bus health in `/status` output +5. **Handler-level stats** — per-handler invocation counts, error rates, avg latency +6. **Zero hot-path cost** — counter increments only, no allocations on the event path + +--- + +## Design + +### 1. EventBusStats Class + +New class in `nous/events.py`. Purely in-memory — no DB writes on the hot path. + +```python +@dataclass +class HandlerStat: + name: str + invocations: int = 0 + successes: int = 0 + errors: int = 0 + last_invoked: float | None = None # time.monotonic() + last_error: float | None = None + last_error_msg: str | None = None + total_duration_ms: float = 0.0 + +@dataclass +class RecentEvent: + type: str + timestamp: str # ISO format + handlers_invoked: int + handlers_failed: int + duration_ms: float + session_id: str | None = None + +class EventBusStats: + def __init__(self, recent_limit: int = 100): + self._event_counts: dict[str, int] = defaultdict(int) + self._handler_stats: dict[str, HandlerStat] = {} + self._recent: deque[RecentEvent] = deque(maxlen=recent_limit) + self._total_processed: int = 0 + self._total_dropped: int = 0 + self._started_at: float = time.monotonic() +``` + +**Key methods:** +- `record_event(event, handlers_invoked, handlers_failed, duration_ms)` — called after dispatch +- `record_handler_success(handler_name, duration_ms)` — called per handler +- `record_handler_error(handler_name, error_msg)` — called on handler failure +- `record_drop()` — called when queue is full +- `to_dict()` → serializable stats for API +- `recent_events(limit)` → newest-first list from ring buffer + +### 2. EventBus Wiring + +Modify `EventBus` to populate stats: + +- **`__init__`** — create `self.stats = EventBusStats()` +- **`emit()`** — on `QueueFull`, call `self.stats.record_drop()` +- **`_dispatch()`** — time the full dispatch, count handler results, call `record_event()` +- **`_safe_handle()`** — return `bool` (success/fail), call `record_handler_success()` or `record_handler_error()` with timing + +The existing error isolation behavior is unchanged — `_safe_handle` still catches all exceptions. The only addition is recording the outcome. + +### 3. REST Endpoints + +**`GET /events/stats`** +```json +{ + "uptime_seconds": 3600, + "total_processed": 142, + "total_dropped": 0, + "queue_depth": 0, + "event_counts": { + "turn_completed": 85, + "heartbeat_tick": 24, + "episode_completed": 8, + "heartbeat_triage": 3, + "sleep_completed": 1 + }, + "handlers": { + "SessionTimeoutMonitor.on_activity": { + "invocations": 85, + "successes": 85, + "errors": 0, + "error_rate": 0.0, + "last_invoked_ago_s": 120, + "avg_duration_ms": 0.1 + }, + "EpisodeSummarizer.handle": { + "invocations": 12, + "successes": 8, + "errors": 4, + "error_rate": 0.33, + "last_error_ago_s": 600, + "last_error_msg": "LLM call failed: 429", + "avg_duration_ms": 2500.0 + } + } +} +``` + +**`GET /events/recent?limit=20`** +Returns from in-memory ring buffer (fast, no DB). Newest first. + +```json +{ + "events": [ + { + "type": "turn_completed", + "timestamp": "2026-04-04T17:30:00Z", + "handlers_invoked": 2, + "handlers_failed": 0, + "duration_ms": 3.2 + } + ], + "source": "memory", + "count": 20 +} +``` + +### 4. Handler Stats Exposure + +Each major handler gets a `get_stats()` method returning its internal state: + +**SessionTimeoutMonitor:** +- tracked_sessions count + per-session idle time +- sleep_emitted flag +- global idle seconds + +**SleepHandler:** +- total_sleeps, last_sleep_at, last_phases_completed +- currently_sleeping flag, last_interrupted + +**HeartbeatRunner:** +- total_ticks, total_findings, checks_run_by_name +- last_tick_at, currently_running + +These are included in the `/events/stats` response under their own keys. + +### 5. Telegram Integration + +Extend `/status` output with event bus section: + +``` +📡 Event Bus + 142 events processed, 0 dropped + Queue: 0 pending | Uptime: 6h 30m + + Handlers: + ✅ SessionMonitor: 85/85 + ⚠️ EpisodeSummarizer: 8/12 (33% errors) + ✅ FactExtractor: 7/8 + ✅ SleepHandler: 1/1 + ✅ HeartbeatRunner: 24 ticks, 3 findings +``` + +Error flag (⚠️) shown when error_rate > 10%. + +### 6. Dashboard Panel (optional, P2) + +New panel in the Memory Dashboard: "System Health" +- Event throughput chart (events/minute over last hour) +- Handler health table (success rate, avg latency) +- Queue depth gauge +- Reuses `/events/stats` endpoint + +--- + +## Files Changed + +| File | Change | ~Lines | +|------|--------|--------| +| `nous/events.py` | `EventBusStats`, `HandlerStat`, `RecentEvent` + wiring | +100 | +| `nous/api/rest.py` | `GET /events/stats`, `GET /events/recent` | +60 | +| `nous/handlers/session_monitor.py` | `get_stats()` | +15 | +| `nous/handlers/sleep_handler.py` | tracking fields + `get_stats()` | +20 | +| `nous/heartbeat/runner.py` | `get_stats()` | +15 | +| `nous/telegram_bot.py` | event bus section in `/status` | +30 | +| `tests/test_event_bus_observability.py` | **NEW** | +200 | +| **Total** | | ~440 | + +--- + +## Design Decisions + +| # | Decision | Rationale | +|---|----------|-----------| +| D1 | In-memory stats, not DB | Hot-path cost must be zero. Counter increments are O(1). | +| D2 | Ring buffer (deque, maxlen=100) | Fixed memory (~10KB), O(1) append, no unbounded growth. | +| D3 | `time.monotonic()` for timing | Wall clock can shift; monotonic doesn't. "Ago" times are relative. | +| D4 | `_safe_handle` returns bool | Lets `_dispatch` count failures without changing error isolation. | +| D5 | Stats keyed by `__qualname__` | Unique per handler method, human-readable in API output. | +| D6 | Telegram reads from REST API | Single source of truth. `/events/stats` serves both dashboard and bot. | + +--- + +## Acceptance Criteria + +- [ ] `EventBusStats` collects event counts, handler stats, recent events ring buffer +- [ ] `EventBus._dispatch` and `_safe_handle` populate stats on every event +- [ ] Queue-full events increment `total_dropped` +- [ ] `GET /events/stats` returns full stats with handler breakdown +- [ ] `GET /events/recent` returns from ring buffer, newest first +- [ ] `SessionTimeoutMonitor.get_stats()` exposes tracked sessions + idle times +- [ ] `SleepHandler.get_stats()` exposes sleep history + current state +- [ ] `HeartbeatRunner.get_stats()` exposes tick/finding counts +- [ ] Telegram `/status` includes event bus health summary +- [ ] Handler errors flagged with ⚠️ when error_rate > 10% +- [ ] No measurable latency impact on event processing +- [ ] All tests pass (unit + integration) diff --git a/docs/features/F035.2-causal-chain-tracing.md b/docs/features/F035.2-causal-chain-tracing.md new file mode 100644 index 0000000..393423c --- /dev/null +++ b/docs/features/F035.2-causal-chain-tracing.md @@ -0,0 +1,305 @@ +# F035.2 — Autonomous Action Audit Trail (Causal Chain Tracing) + +**Status:** PROPOSED +**Priority:** P1 (critical for debugging self-modification) +**Depends on:** F035.1 (Event Bus Observability), F006 (Event Bus) +**Author:** Nous + Tim +**Date:** 2026-04-04 + +--- + +## Problem + +Nous performs autonomous actions that modify its own state: heartbeat creates findings, sleep consolidation rewrites facts, self-tuning adjusts intervals, contradiction resolution deletes or merges facts. Each action is logged as an individual event, but there's no link between cause and effect. + +When something unexpected happens — a fact vanishes, a check stops running, a censor appears — you have to manually reconstruct the chain: "this fact was deleted by sleep consolidation, which was triggered by a session_ended event, which happened because the user went idle for 30 minutes." + +**The question we need to answer:** Given any state change, trace the full causal chain back to the originating trigger. + +--- + +## Examples of Causal Chains + +**Chain 1: Heartbeat → Finding → Triage → Notification** +``` +heartbeat_tick (scheduled, 15min interval) + └─► check_email executed + └─► finding_created (fingerprint: abc123, urgency: high) + └─► heartbeat_triage (cognitive session opened) + └─► telegram_notification_sent +``` + +**Chain 2: Idle → Sleep → Fact Modification** +``` +session_ended (user idle 30min) + └─► sleep_started + └─► contradiction_found (fact_id: X vs fact_id: Y) + └─► fact_deleted (fact_id: X, reason: "contradicted by newer fact Y") + └─► fact_updated (fact_id: Y, merged content) +``` + +**Chain 3: Self-Tuning Interval Change** +``` +heartbeat_tick (scheduled) + └─► tuning_evaluated (check: health_check, yield: 0/10 recent ticks) + └─► interval_changed (health_check: 60min → 120min, reason: "low yield") +``` + +**Chain 4: Decision Review → Censor Creation** +``` +turn_completed (user conversation) + └─► episode_completed + └─► decision_reviewed (decision_id: D, outcome: failure) + └─► censor_created (trigger: "pattern X", reason: "from failed decision D") +``` + +--- + +## Design + +### 1. Causation Metadata on Events + +Extend the `Event` dataclass with two optional fields: + +```python +@dataclass +class Event: + type: str + agent_id: str + data: dict[str, Any] = field(default_factory=dict) + session_id: str | None = None + timestamp: datetime = field(default_factory=lambda: datetime.now(UTC)) + + # F035.2: Causal chain fields + trace_id: str | None = None # root cause identifier (shared across chain) + caused_by: str | None = None # event_id of the direct parent event + event_id: str = field(default_factory=lambda: uuid4().hex[:12]) +``` + +**`trace_id`** — identifies the root cause. All events in a causal chain share the same trace_id. If an event has no parent (it's a root trigger like `heartbeat_tick` or `message_received`), `trace_id = event_id`. + +**`caused_by`** — the `event_id` of the direct parent. Forms a tree (one parent, multiple children). + +**`event_id`** — unique per event. Short hex (12 chars) for readability. + +### 2. Propagation Rules + +Events propagate trace context through handlers: + +```python +# In a handler that emits child events: +async def handle(self, event: Event) -> None: + # ... do work ... + + # Child event inherits trace_id, points to parent + await self._bus.emit(Event( + type="fact_deleted", + agent_id=event.agent_id, + data={"fact_id": "X", "reason": "contradiction"}, + trace_id=event.trace_id, # inherit root trace + caused_by=event.event_id, # point to parent + )) +``` + +**Root events** (no parent): +- `heartbeat_tick` — trace_id = event_id (heartbeat is the root cause) +- `message_received` — trace_id = event_id (user action is the root cause) +- `session_ended` — trace_id = event_id (timeout is the root cause) +- `daily_tick` / `weekly_tick` — trace_id = event_id + +**Child events** (always have a parent): +- `finding_created`, `heartbeat_triage` — caused by heartbeat_tick +- `sleep_started`, `episode_completed` — caused by session_ended +- `fact_deleted`, `fact_updated` — caused by sleep phase events +- `censor_created` — caused by decision_reviewed +- `interval_changed` — caused by tuning_evaluated + +### 3. DB Schema Extension + +Add columns to the events table: + +```sql +ALTER TABLE nous_system.events + ADD COLUMN event_id VARCHAR(12), + ADD COLUMN trace_id VARCHAR(12), + ADD COLUMN caused_by VARCHAR(12); + +CREATE INDEX idx_events_trace_id ON nous_system.events(trace_id); +CREATE INDEX idx_events_event_id ON nous_system.events(event_id); +``` + +The `trace_id` index is the critical one — it enables "give me the full chain for this trace." + +### 4. Query API + +**`GET /events/trace/{trace_id}`** + +Returns all events in a causal chain, ordered by timestamp: + +```json +{ + "trace_id": "a1b2c3d4e5f6", + "root_event": "heartbeat_tick", + "root_timestamp": "2026-04-04T17:00:00Z", + "events": [ + { + "event_id": "a1b2c3d4e5f6", + "type": "heartbeat_tick", + "caused_by": null, + "timestamp": "2026-04-04T17:00:00Z", + "data": {"checks_run": 3, "findings_count": 1} + }, + { + "event_id": "f7e8d9c0b1a2", + "type": "finding_created", + "caused_by": "a1b2c3d4e5f6", + "timestamp": "2026-04-04T17:00:01Z", + "data": {"fingerprint": "abc123", "urgency": "high"} + }, + { + "event_id": "112233445566", + "type": "heartbeat_triage", + "caused_by": "f7e8d9c0b1a2", + "timestamp": "2026-04-04T17:00:05Z", + "data": {"session_id": "triage-xyz", "action": "notify"} + } + ], + "depth": 3, + "duration_ms": 5000 +} +``` + +**`GET /events/recent-traces?limit=20`** + +Returns recent trace roots with summary (how deep the chain went, what types of events it contains): + +```json +{ + "traces": [ + { + "trace_id": "a1b2c3d4e5f6", + "root_type": "heartbeat_tick", + "timestamp": "2026-04-04T17:00:00Z", + "event_count": 3, + "event_types": ["heartbeat_tick", "finding_created", "heartbeat_triage"], + "has_modifications": true, + "has_errors": false + } + ] +} +``` + +**`GET /events/modifications?hours=24`** + +Returns only traces that resulted in state modifications (fact created/deleted/updated, censor created/modified, interval changed). This is the "what did Nous change about itself?" view. + +### 5. Modification Tagging + +Events that modify agent state get a `modifies` field in their data: + +```python +await self._bus.emit(Event( + type="fact_deleted", + agent_id=event.agent_id, + data={ + "fact_id": "X", + "reason": "contradiction", + "modifies": "fact", # marks this as a state modification + }, + trace_id=event.trace_id, + caused_by=event.event_id, +)) +``` + +Recognized modification targets: `fact`, `censor`, `decision`, `procedure`, `episode`, `check_interval`, `check_config`. + +The `/events/modifications` endpoint filters on `data->>'modifies' IS NOT NULL`. + +### 6. Dashboard Panel: "Autonomous Activity" + +New panel in the Memory Dashboard showing: + +- **Timeline view** — recent traces as collapsible trees. Click to expand full chain. +- **Modification feed** — latest state changes with their root cause. "Fact X deleted ← contradiction resolution ← sleep cycle ← idle timeout" +- **Trace search** — look up any trace_id to see the full chain + +### 7. Telegram Commands + +**`/trace `** — show the full causal chain for a trace +**`/modifications`** — show recent autonomous state changes with causes + +--- + +## Rollout Strategy + +This requires touching every handler that emits events. To avoid a massive PR: + +**Phase A: Infrastructure (non-breaking)** +- Add `event_id`, `trace_id`, `caused_by` to `Event` dataclass (optional, defaults to None) +- Add DB columns (nullable, no migration issues) +- Add trace query endpoints (return empty until events start carrying trace IDs) +- Update `EventBus` DB persister to store new fields + +**Phase B: Root Events** +- Add trace_id to root event emitters: heartbeat_tick, message_received, session_ended, daily_tick +- These are ~5 emit sites. Each just adds `trace_id=event_id`. + +**Phase C: Child Events** +- Update handlers to propagate trace context. Each handler that receives an event and emits new events passes along `trace_id` and sets `caused_by`. +- This is the bulk of the work — ~15 emit sites across handlers. + +**Phase D: Dashboard + Telegram** +- Build the autonomous activity panel +- Add `/trace` and `/modifications` commands + +--- + +## Files Changed + +| File | Change | ~Lines | +|------|--------|--------| +| `nous/events.py` | Add `event_id`, `trace_id`, `caused_by` to Event | +5 | +| `nous/storage/migrator.py` | Add columns + indexes | +15 | +| `nous/brain/brain.py` | Persist new Event fields to DB | +10 | +| `nous/api/rest.py` | `/events/trace/{id}`, `/events/recent-traces`, `/events/modifications` | +90 | +| `nous/heartbeat/runner.py` | Emit trace_id on heartbeat_tick, propagate to children | +15 | +| `nous/handlers/episode_summarizer.py` | Propagate trace context | +5 | +| `nous/handlers/fact_extractor.py` | Propagate trace context + tag modifications | +10 | +| `nous/handlers/sleep_handler.py` | Propagate trace context + tag modifications | +15 | +| `nous/handlers/outcome_detector.py` | Propagate trace context | +5 | +| `nous/handlers/session_monitor.py` | Emit trace_id on session_ended | +5 | +| `nous/cognitive/layer.py` | Emit trace_id on message_received, turn_completed | +10 | +| `nous/telegram_bot.py` | `/trace` and `/modifications` commands | +40 | +| `static/dashboard.js` | Autonomous activity panel | +100 | +| `tests/test_causal_tracing.py` | **NEW** | +200 | +| **Total** | | ~525 | + +--- + +## Design Decisions + +| # | Decision | Rationale | +|---|----------|-----------| +| D1 | Short hex event_id (12 chars) | Readable in logs and Telegram. Collision risk negligible at our volume. | +| D2 | `trace_id` on root, `caused_by` on children | Standard distributed tracing pattern (OpenTelemetry-inspired). Simple tree structure. | +| D3 | `modifies` tag in event data | Avoids a separate table. Queryable via JSON operators. Tags are a flat string, not a complex object. | +| D4 | DB indexes on trace_id and event_id | Trace queries are the primary access pattern. Worth the write overhead. | +| D5 | Phased rollout (infra → roots → children) | Each phase is a small PR. No big-bang migration. Events without trace IDs are handled gracefully (null). | +| D6 | No trace propagation across conversation boundaries | A heartbeat finding that triggers a triage session is one trace. The user conversation that follows is a new trace. Clean separation. | + +--- + +## Acceptance Criteria + +- [ ] `Event` dataclass has `event_id`, `trace_id`, `caused_by` fields +- [ ] DB schema extended with indexed columns +- [ ] Root events (heartbeat_tick, message_received, session_ended) set trace_id +- [ ] All handler-emitted events propagate trace_id and set caused_by +- [ ] State-modifying events tagged with `modifies` in data +- [ ] `GET /events/trace/{trace_id}` returns full causal chain +- [ ] `GET /events/recent-traces` returns trace summaries +- [ ] `GET /events/modifications` returns modification-only traces +- [ ] Telegram `/trace` and `/modifications` commands work +- [ ] Dashboard shows autonomous activity timeline +- [ ] Events without trace IDs (pre-migration) handled gracefully +- [ ] All tests pass diff --git a/docs/features/F035.3-behavioral-drift-detection.md b/docs/features/F035.3-behavioral-drift-detection.md new file mode 100644 index 0000000..b79f357 --- /dev/null +++ b/docs/features/F035.3-behavioral-drift-detection.md @@ -0,0 +1,307 @@ +# F035.3 — Behavioral Drift Detection + +**Status:** PROPOSED +**Priority:** P2 (builds on F035.1 + F035.2) +**Depends on:** F035.1 (Event Bus Stats), F035.2 (Causal Chain Tracing) +**Author:** Nous + Tim +**Date:** 2026-04-04 + +--- + +## Problem + +Individual autonomous actions are reasonable — heartbeat self-tuning adjusts one interval, sleep consolidation prunes one batch of facts, contradiction resolution merges two conflicting facts. But reasonable individual changes can compound into unreasonable drift: + +- Fact count growing 3x faster than normal (over-permissive admission) +- Heartbeat self-tuning a check down to never running (yield was low, but only because the check was running at the wrong time) +- Censors accumulating without retirement (blocking increasingly broad patterns) +- Memory admission rate dropping because dedup threshold is too aggressive +- Sleep consolidation steadily reducing episode detail + +None of these show up as a single alarming event. They're only visible as **trends** — and right now nobody's watching the trends. + +--- + +## Design + +### 1. Metric Snapshots + +Periodically capture a snapshot of key system metrics and store them as a time series. + +**Snapshot frequency:** Every heartbeat cycle (inherits heartbeat interval, typically 15-60 min). This runs as a lightweight heartbeat check — no separate scheduler. + +**Metrics captured per snapshot:** + +```python +@dataclass +class BehaviorSnapshot: + timestamp: datetime + + # Memory metrics + fact_count: int + fact_count_delta: int # change since last snapshot + episode_count: int + episode_count_delta: int + active_censor_count: int + active_censor_delta: int + procedure_count: int + decision_count: int + + # Admission metrics (from event bus stats, reset per snapshot window) + facts_admitted: int # facts learned since last snapshot + facts_rejected_dedup: int # rejected by dedup + facts_rejected_admission: int # rejected by quality gate + admission_rate: float # admitted / (admitted + rejected) + + # Heartbeat metrics + checks_run: int # total checks since last snapshot + findings_created: int + findings_resolved: int + triage_sessions_opened: int + interval_changes: list[dict] # [{check, old, new, reason}] + + # Sleep metrics + sleep_ran: bool + episodes_compacted: int + facts_pruned: int + contradictions_resolved: int + + # Event bus health + events_processed: int + events_dropped: int + handler_error_count: int + handler_error_rate: float + + # Conversation metrics + turns_processed: int + avg_turn_latency_ms: float + tool_calls: int +``` + +### 2. Storage + +Snapshots stored in a dedicated table: + +```sql +CREATE TABLE nous_system.behavior_snapshots ( + id SERIAL PRIMARY KEY, + timestamp TIMESTAMPTZ NOT NULL, + metrics JSONB NOT NULL, + anomalies JSONB DEFAULT '[]', + created_at TIMESTAMPTZ DEFAULT NOW() +); + +CREATE INDEX idx_behavior_snapshots_ts + ON nous_system.behavior_snapshots(timestamp DESC); +``` + +Retention: 30 days of snapshots. Older snapshots are rolled up into daily aggregates (min/max/avg per metric per day) stored in a separate `behavior_daily_rollups` table. This gives trend visibility over months without unbounded growth. + +### 3. Drift Detection + +After each snapshot, compare against the rolling baseline (last 7 days of snapshots). + +**Detection method:** For each metric, compute the rolling mean and standard deviation. Flag when the current value exceeds `mean ± k * stddev` where `k` is configurable per metric (default: 2.0). + +```python +class DriftDetector: + # Per-metric sensitivity configuration + THRESHOLDS = { + "fact_count_delta": {"k": 2.0, "min_samples": 10}, + "admission_rate": {"k": 2.0, "min_samples": 10}, + "active_censor_count": {"k": 2.5, "min_samples": 10}, + "handler_error_rate": {"k": 1.5, "min_samples": 5}, # more sensitive + "events_dropped": {"k": 1.5, "min_samples": 5}, + "facts_pruned": {"k": 2.0, "min_samples": 10}, + "interval_changes": {"k": 2.0, "min_samples": 10}, + } + + def detect(self, current: BehaviorSnapshot, + history: list[BehaviorSnapshot]) -> list[Anomaly]: + anomalies = [] + for metric, config in self.THRESHOLDS.items(): + values = [getattr(s, metric) for s in history] + if len(values) < config["min_samples"]: + continue + mean = statistics.mean(values) + stddev = statistics.stdev(values) + if stddev == 0: + continue + current_val = getattr(current, metric) + z_score = (current_val - mean) / stddev + if abs(z_score) > config["k"]: + anomalies.append(Anomaly( + metric=metric, + current=current_val, + mean=mean, + stddev=stddev, + z_score=z_score, + direction="up" if z_score > 0 else "down", + severity="warning" if abs(z_score) < 3.0 else "alert", + )) + return anomalies +``` + +### 4. Anomaly Response + +When anomalies are detected: + +**Warning (2σ < z < 3σ):** +- Store anomaly in snapshot record +- Log at WARNING level +- Include in next daily digest (if heartbeat digest exists) + +**Alert (z > 3σ):** +- Store anomaly in snapshot record +- Log at ERROR level +- Send Telegram notification immediately: + ``` + ⚠️ Behavioral Drift Detected + + fact_count_delta: 47 (normal: 8.2 ± 4.1) + Direction: ↑ significantly above baseline + + Possible causes: admission threshold change, + high conversation volume, sleep consolidation gap + ``` +- Create a heartbeat finding (so it enters the finding lifecycle and gets triaged) + +### 5. Trend API + +**`GET /behavior/snapshot/latest`** — most recent snapshot + +**`GET /behavior/trends?metric=fact_count_delta&hours=168`** — time series for a metric over N hours + +```json +{ + "metric": "fact_count_delta", + "hours": 168, + "points": [ + {"timestamp": "2026-04-04T17:00:00Z", "value": 12}, + {"timestamp": "2026-04-04T16:00:00Z", "value": 8}, + ... + ], + "stats": { + "mean": 8.2, + "stddev": 4.1, + "min": 0, + "max": 47, + "trend": "stable" + } +} +``` + +**`GET /behavior/anomalies?hours=168`** — all anomalies in the time window + +**`GET /behavior/drift-report`** — human-readable summary of current behavioral state vs. baseline (designed for Telegram or dashboard display) + +### 6. Dashboard Panel: "Behavioral Trends" + +New panel in the Memory Dashboard: + +- **Sparkline charts** for key metrics (fact growth, censor count, admission rate, handler errors) +- **Anomaly markers** on the timeline (red dots for alerts, yellow for warnings) +- **Current vs. baseline** comparison table +- **Drift report** — natural language summary of any detected drift + +### 7. Heartbeat Integration + +Drift detection runs as a heartbeat check: + +```python +class BehaviorDriftCheck(BaseCheck): + name = "behavior_drift" + default_interval_minutes = 60 # hourly + + async def run(self) -> CheckResult: + # 1. Capture snapshot + snapshot = await self._capture_snapshot() + # 2. Store it + await self._store_snapshot(snapshot) + # 3. Load baseline (last 7 days) + baseline = await self._load_baseline(hours=168) + # 4. Detect anomalies + anomalies = self._detector.detect(snapshot, baseline) + # 5. Return findings for any anomalies + findings = [ + Finding( + category="drift", + description=f"{a.metric}: {a.current} ({a.direction} from {a.mean:.1f} ± {a.stddev:.1f})", + urgency="high" if a.severity == "alert" else "normal", + ) + for a in anomalies + ] + return CheckResult(findings=findings) +``` + +This means drift detection benefits from the finding lifecycle (F034.1) — anomalies are deduplicated, escalated if persistent, and resolved when they return to baseline. + +### 8. Self-Tuning Awareness + +The drift detector is aware of self-tuning events. When a metric anomaly coincides with a recent `interval_changed` event (from F035.2 causal tracing), it adds context: + +``` +⚠️ Behavioral Drift Detected + +findings_created: 0 (normal: 3.2 ± 1.8) +Direction: ↓ significantly below baseline + +🔗 Correlated change: health_check interval increased + 60min → 120min (2 hours ago, trace: abc123) + +This may explain the drop — check is running less often. +``` + +This cross-references F035.2 causal chains with F035.3 drift detection. + +--- + +## Files Changed + +| File | Change | ~Lines | +|------|--------|--------| +| `nous/observability/__init__.py` | **NEW** package | +5 | +| `nous/observability/snapshots.py` | `BehaviorSnapshot`, capture logic | +120 | +| `nous/observability/drift.py` | `DriftDetector`, `Anomaly` | +100 | +| `nous/observability/rollups.py` | Daily rollup aggregation | +60 | +| `nous/heartbeat/checks/behavior_drift.py` | **NEW** heartbeat check | +50 | +| `nous/storage/migrator.py` | snapshot + rollup tables | +20 | +| `nous/api/rest.py` | `/behavior/*` endpoints | +80 | +| `nous/telegram_bot.py` | Drift alerts, `/drift` command | +30 | +| `static/dashboard.js` | Behavioral trends panel | +150 | +| `tests/test_drift_detection.py` | **NEW** | +200 | +| **Total** | | ~815 | + +--- + +## Design Decisions + +| # | Decision | Rationale | +|---|----------|-----------| +| D1 | Z-score based detection | Simple, interpretable, no ML dependency. Works well for normally distributed metrics with sufficient history. | +| D2 | Per-metric sensitivity (k-value) | Handler errors are more critical than fact count fluctuation. Different thresholds reflect different risk. | +| D3 | Heartbeat check, not separate scheduler | Reuses existing infrastructure. Benefits from finding lifecycle (dedup, escalation). | +| D4 | 30-day retention + daily rollups | Bounded storage with long-term trend visibility. ~2KB per snapshot × 96/day × 30 = ~5.6MB. | +| D5 | Cross-reference with causal chains | Drift is more useful when you can see *what caused it*. Requires F035.2 but degrades gracefully without it. | +| D6 | Warning (2σ) vs. Alert (3σ) | Two tiers prevent alert fatigue. Warnings go to digest, alerts are immediate. | +| D7 | min_samples guard | Don't flag anomalies until you have enough history. Prevents false positives on fresh systems. | + +--- + +## Acceptance Criteria + +- [ ] `BehaviorSnapshot` captures all specified metrics +- [ ] Snapshots stored in DB with JSONB metrics +- [ ] `DriftDetector` identifies anomalies using z-score thresholds +- [ ] Warning vs. alert severity tiers work correctly +- [ ] `BehaviorDriftCheck` runs as a heartbeat check +- [ ] Anomalies create findings (enter finding lifecycle) +- [ ] Drift alerts sent to Telegram for alert-severity anomalies +- [ ] `GET /behavior/trends` returns time series data +- [ ] `GET /behavior/anomalies` returns detected anomalies +- [ ] `GET /behavior/drift-report` returns human-readable summary +- [ ] 30-day retention with daily rollup aggregation +- [ ] Cross-references causal chain events when available +- [ ] Dashboard shows sparkline trends with anomaly markers +- [ ] All tests pass +- [ ] False positive rate < 5% over first week of operation