Skip to content

CODEBASE-RAG-IMPLEMENTATION.md

Latest commit

 

History

History
194 lines (149 loc) · 6.09 KB

CODEBASE-RAG-IMPLEMENTATION.md

File metadata and controls

194 lines (149 loc) · 6.09 KB

Codebase RAG Implementation Plan

Architecture Vision

Goal: Enable personas to learn from code through embeddings and self-directed study.

Key Principle: Break down into small, composable commands that can be used independently or combined for emergent behaviors.

Command Hierarchy

Low-Level Commands (Building Blocks)

# 1. Generate embeddings for any text
./jtag ai/embedding/generate --text="code snippet" --model="qwen3-embedding"
# Returns: { embedding: number[], model: string, dimensions: number }

# 2. Store code index entry
./jtag ai/rag/index/create --filePath="..." --content="..." --embedding=[...]
# Returns: { entryId: UUID, indexed: boolean }

# 3. Query code index with embeddings
./jtag ai/rag/index/query --query="PersonaUser" --limit=5
# Returns: { entries: CodeIndexEntry[], relevanceScores: number[] }

Mid-Level Commands (Composed Operations)

# Index a single file (uses: generate + create)
./jtag ai/rag/index/file --filePath="system/user/PersonaUser.ts"
# Internally: Parse → Generate embeddings → Store entries

# Index a directory (uses: index/file repeatedly)
./jtag ai/rag/index/directory --path="system/user" --recursive
# Internally: Find files → Index each file

# Search codebase (uses: generate + query)
./jtag ai/rag/search --query="how does coordination work?" --limit=10
# Internally: Generate query embedding → Search index → Return ranked results

High-Level Commands (User-Facing)

# Index entire codebase
./jtag ai/rag/index-codebase --paths="system,commands" --fileTypes="typescript,markdown"
# Uses: index/directory for each path

# Ask question about codebase
./jtag ai/rag/ask --question="How does PersonaUser handle learning?"
# Uses: search + AI generation with context

Implementation Order

Phase 1: Foundation (Low-Level)

  1. ✅ CodeIndexEntity - Data structure
  2. ✅ Candle embedding generation via Rust IPC
  3. ✅ ai/embedding/generate - Command wrapper for embedding generation
  4. ✅ ai/rag/index/create - Store single code entry
  5. ⬜ ai/rag/query-open - Open similarity search, return handle + first page
  6. ⬜ ai/rag/query-fetch - Fetch results at any offset (bidirectional)
  7. ⬜ ai/rag/query-close - Close query handle and cleanup

Phase 2: Composition (Mid-Level)

  1. ⬜ ai/rag/index/file - Parse and index single file (TypeScript/Markdown)
  2. ⬜ ai/rag/index/directory - Index multiple files
  3. ⬜ ai/rag/search - Semantic code search

Phase 3: User Experience (High-Level)

  1. ⬜ ai/rag/index-codebase - Full codebase indexing
  2. ⬜ ai/rag/ask - Natural language queries

Phase 4: Learning Integration

  1. ⬜ PersonaUser captures conversation embeddings
  2. ⬜ PersonaUser schedules self-training tasks
  3. ⬜ Learning session orchestration

Command Design Pattern

Each command follows this structure:

commands/ai/[category]/[action]/
├── shared/
│   ├── [Action]Types.ts      # Params & Result interfaces
│   └── [Action]Command.ts    # Abstract base class
├── server/
│   └── [Action]ServerCommand.ts  # Implementation
└── browser/
    └── [Action]BrowserCommand.ts # (if needed)

Why This Works

Composability:

  • Low-level commands = Lego blocks
  • Mid-level commands = Pre-built structures
  • High-level commands = Complete solutions
  • PersonaUser can use ANY level for its needs

Emergent Behaviors:

// Simple: Human indexes code manually
./jtag ai/rag/index-codebase --paths="system/user"

// Complex: Persona self-directs learning
PersonaUser.serviceInbox()
  → Detects knowledge gap in "coordination"
  → Creates task: { type: 'study', topic: 'coordination' }
  → Executes: ai/rag/search --query="coordination patterns"
  → Generates quiz from results
  → Answers quiz
  → Captures training data
  → Fine-tunes adapter
  → Returns to normal operation

Reusability:

  • Teacher AI uses ai/rag/search to generate quizzes
  • Helper AI uses ai/rag/search to improve answers
  • CodeReview AI uses ai/rag/search to find patterns
  • Human uses ai/rag/search to explore codebase

Next Steps

  1. Implement ai/embedding/generate command (wrapper for Candle Rust IPC)
  2. Implement ai/rag/index/create command (store single entry)
  3. Implement ai/rag/index/query command (search with embeddings)
  4. Test composition by building ai/rag/index/file from primitives
  5. Continue building upward

Start simple, compose elegantly, unlock emergence.

Universal Iterator/Cursor Pattern

The RAG query commands implement a universal iterator pattern that works for all time-ordered and ranked data.

The Three-Command Pattern

Every iterator follows this structure:

  1. {domain}/query-open - Opens query, returns handle + first page
  2. {domain}/query-fetch - Fetches at any position (bidirectional + random access)
  3. {domain}/query-close - Closes handle and cleanup

Iterator Features

Bidirectional Navigation:

# Move forward through results
./jtag ai/rag/query-fetch --queryHandle="abc" --direction="forward"

# Move backward
./jtag ai/rag/query-fetch --queryHandle="abc" --direction="backward"

Random Access:

# Jump to specific offset
./jtag ai/rag/query-fetch --queryHandle="abc" --offset=30 --limit=10

Position Tracking:

  • Handle maintains current offset
  • Results include hasMore and hasPrevious flags
  • Frontend can implement infinite scroll in both directions

Universal Applications

This pattern applies to:

  • RAG similarity search (relevance-ranked)
  • Chat message history (time-ordered)
  • Game replays (frame-by-frame navigation)
  • Training logs (epoch progression)
  • Database queries (already implemented: data/query-*)
  • Time series data (any chronological sequence)
  • Event streams (real-time or historical)

Future: Formalized Iterator Type

Eventually this will become a generic abstraction:

interface Iterator<T> {
  handle: UUID;
  offset: number;
  limit: number;
  totalCount: number;
  hasMore: boolean;
  hasPrevious: boolean;
}

One mental model, infinite applications.