Institutional Memory Framework: Adaptive Learning System for AI Runbooks

[dandye]

Institutional Memory Framework: Adaptive Learning System for AI Runbooks

This issue synthesizes and extends proposals from issues #10 and #11, creating a comprehensive institutional memory framework that enables AI agents to learn from analyst feedback and operational experience while maintaining full compatibility with the existing project architecture.

Problem Statement

Currently, our AI runbook system executes static procedures without the ability to learn from operational experience or analyst feedback. This limits the system's effectiveness in adapting to organization-specific contexts, threat landscapes, and operational preferences.

Proposed Solution: Institutional Memory Framework

Core Architecture

The institutional memory system will be implemented as rules_bank/institutional_memory/ with the following structure:

rules_bank/institutional_memory/
├── memories/                    # Individual memory files
│   ├── triage_alerts_tier1.md   # Persona-specific adaptations
│   ├── hunt_apt_threats.md      # Runbook-specific learnings
│   └── ...
├── patterns/                    # Recurring operational patterns
│   ├── false_positive_patterns.md
│   ├── escalation_triggers.md
│   └── ...
├── adaptations/                 # Persona behavioral modifications
│   ├── tier1_soc_analyst.md     # Persona-specific adaptations
│   ├── threat_hunter.md
│   └── ...
├── feedback/                    # Analyst feedback collection
│   ├── feedback_queue.md        # Pending feedback processing
│   ├── processed_feedback.md    # Historical feedback log
│   └── ...
└── MEMORY-THESAURUS.md         # Memory-specific controlled vocabulary

Memory File Format

Each memory file uses YAML frontmatter with standardized fields:

---
runbook: "run_books/triage_alerts.md"
persona: "personas/soc_analyst_tier_1.md"
source_step: "Step 3: IOC Enrichment"
confidence: 0.95
last_updated: "2025-08-23"
feedback_source: "analyst_d_anderson"
validation_count: 5
success_rate: 0.92
related_cases: ["CASE-2024-001", "CASE-2024-045"]
memory_type: "procedure_modification"
---

## Analyst Feedback

"For our organization, when triaging alerts from the 'XYZ' sensor, we must always perform a historical lookup in our internal 'ThreatDB' before proceeding with public enrichment. This is a critical step that is currently missing."

## Context Analysis

**Original Procedure Issue**: Standard enrichment workflow missing critical internal database check
**Organizational Context**: XYZ sensor produces high-volume alerts requiring internal correlation
**Risk Impact**: Without internal check, analysts may miss previously identified threats

## Derived Procedure

1. **Internal Enrichment Check**:
   - **Tool**: `internal_db query`
   - **Parameters**: `database=ThreatDB, query={ioc_value}`
   - **Validation**: Ensure response time < 5 seconds

2. **Conditional Public Enrichment**:
   - **Condition**: If internal enrichment returns no results OR confidence < 0.8
   - **Tool**: `secops-mcp enrich_ioc`
   - **Parameters**: `ioc={ioc_value}`

## Application Log

- **2025-08-23**: Memory created from analyst feedback (D. Anderson)
- **2025-08-24**: Successfully applied to Alert #4512 (reduced analysis time by 40%)
- **2025-08-25**: Applied to Alerts #4520, #4521 (both confirmed beneficial)
- **2025-08-26**: Confidence increased to 0.95 after validation

Integration with Existing Architecture

1. Persona System Integration

• Extend existing personas with memory-aware capabilities
• Add memory query functions to persona workflows
• Maintain backward compatibility with current runbook structure

2. Multi-LLM Support

• Leverage existing symlink architecture (.claude/, .clinerules/, .gemini/)
• Ensure memories propagate across all AI assistants
• Maintain unified access through rules_bank/ master directory

3. Reporting Integration

• Extend existing reporting templates to include memory application tracking
• Add memory effectiveness metrics to operational reports
• Follow established naming conventions: memory_effectiveness_report_YYYYMMDD_HHMM.md

4. Information Architecture Alignment

• Create MEMORY-THESAURUS.md following existing LLMS-THESAURUS format
• Integrate with existing LLMS-SITEMAP structure
• Maintain consistency with established controlled vocabulary

Workflow Integration

Memory Creation Workflow

1. Feedback Collection: Analysts provide natural language feedback during or after runbook execution
2. Processing: Meta-agent processes feedback and identifies procedural improvements
3. Memory Formation: Structured memory file created with confidence scoring
4. Validation: Memory tested against similar scenarios to build confidence
5. Integration: High-confidence memories integrated into active workflows

Memory Application Workflow

1. Pre-execution Check: Agent queries relevant memories before runbook steps
2. Procedure Proposal: Present derived procedures from high-confidence memories
3. Analyst Confirmation: Allow approval/rejection/modification of memory-derived procedures
4. Execution Tracking: Log memory application success/failure for confidence adjustment
5. Continuous Learning: Update confidence scores based on outcomes

New Runbooks Required

Core Memory Management

• rules_bank/run_books/memory_creation.md - Creating new institutional memories
• rules_bank/run_books/memory_validation.md - Validating and adjusting memory confidence
• rules_bank/run_books/memory_retirement.md - Retiring outdated or ineffective memories

Common Steps Extensions

• rules_bank/run_books/common_steps/query_memories.md - Check for relevant memories
• rules_bank/run_books/common_steps/apply_memory_procedure.md - Execute memory-derived procedures
• rules_bank/run_books/common_steps/log_memory_outcome.md - Track memory application results

Compatibility and Migration

• Zero Breaking Changes: Existing runbooks continue to function unchanged
• Gradual Enhancement: Memories enhance rather than replace existing procedures
• Audit Trail: All memory applications logged for compliance and analysis
• Rollback Capability: Ability to disable specific memories or revert to original procedures

Implementation Phases

Phase 1: Foundation (Weeks 1-2)

• Create directory structure and initial templates
• Implement basic memory file format and validation
• Create MEMORY-THESAURUS.md and integration documentation

Phase 2: Core Functionality (Weeks 3-4)

• Implement memory query and application workflows
• Create memory management runbooks
• Extend existing personas with memory capabilities

Phase 3: Integration & Testing (Weeks 5-6)

• Integrate with existing reporting system
• Test multi-LLM compatibility
• Create sample memories for common scenarios

Phase 4: Production Readiness (Weeks 7-8)

• Performance optimization and confidence tuning
• Documentation completion and training materials
• Monitoring and metrics implementation

Success Criteria

• AI agents can learn from analyst feedback and modify behavior accordingly
• Memory system maintains full backward compatibility with existing runbooks
• Multi-LLM support preserved across Claude, Cline, and Gemini CLI
• Memory application tracked and measured for effectiveness
• Zero disruption to current operational workflows
• Clear audit trail for all memory-driven decisions

Related Issues

• Closes memories dir in rules_bank #10 (memories dir in rules_bank)
• Closes Adaptive Learning Framework for AI Runbooks #11 (Adaptive Learning Framework for AI Runbooks)

This comprehensive framework transforms our static runbook system into a dynamic, learning-capable institutional memory that adapts to organizational context while preserving all existing functionality and architectural patterns.