[Feature]: Important Features

[S3DFX-CYBER]

Problem

Required features for tenet ai

Proposed Solution

Adding new and important core features for tenant ai

Alternatives

No response

[S3DFX-CYBER]

@coderabbitai what are the important features required for tenant ai that are missing in the current repo

[coderabbitai]

🧩 Analysis chain

@S3DFX-CYBER, let me explore the current repository structure to identify what features exist and what might be missing for tenant AI functionality.

---

🏁 Script executed:

#!/bin/bash

# Check the repository structure
echo "=== Repository Structure ==="
tree -L 3 -I 'node_modules|.git|__pycache__|*.pyc'

echo -e "\n=== README Content ==="
cat README.md 2>/dev/null || echo "No README.md found"

echo -e "\n=== Configuration Files ==="
fd -t f '(config|settings|env)' -e json -e yaml -e yml -e toml -e ini --max-depth 2

echo -e "\n=== Search for tenant-related code ==="
rg -i 'tenant' -n -C 2

echo -e "\n=== Search for AI/ML related code ==="
rg -i '(openai|anthropic|llm|model|ai|machine.?learning)' -n --type-list | head -20

echo -e "\n=== Python dependencies ==="
cat requirements.txt 2>/dev/null || echo "No requirements.txt found"
cat pyproject.toml 2>/dev/null || echo "No pyproject.toml found"

echo -e "\n=== Package.json ==="
cat package.json 2>/dev/null || echo "No package.json found"

Length of output: 14273

---

🏁 Script executed:

#!/bin/bash

# Check for authentication and authorization code
echo "=== Authentication/Authorization Code ==="
rg -i '(auth|jwt|token|apikey|oauth)' -n -C 2

echo -e "\n=== Database Models/Schema ==="
fd -e sql -e py | xargs rg -l '(model|schema|table|database)' 2>/dev/null | head -10

echo -e "\n=== API Routes and Endpoints ==="
rg -i '(`@app`\.|@router\.|route|endpoint)' -n services/ --type py

echo -e "\n=== User/Organization Management ==="
rg -i '(user|organization|org_id|tenant_id|account)' -n -C 2

echo -e "\n=== Data Isolation/Filtering ==="
rg -i '(isolation|segregat|filter.*tenant|query.*org)' -n -C 2

echo -e "\n=== Configuration for Multi-tenancy ==="
rg -i '(multi.?tenant|rbac|permission|role)' -n -C 3

echo -e "\n=== Service Code Structure ==="
cat services/ingest/app.py

echo -e "\n=== Analyzer Service ==="
cat services/analyzer/app.py

Length of output: 35519

---

@S3DFX-CYBER, I've analyzed the repository and identified the missing multi-tenant AI features. The codebase is currently in MVP stage with single-tenant architecture. Here's a comprehensive breakdown:

🔴 Critical Missing Features

1. Tenant Management System

• ❌ No tenant/organization registration and provisioning
• ❌ No workspace or team management
• ❌ No tenant metadata storage (settings, preferences, configurations)
• ❌ No tenant onboarding workflow

2. Authentication & Authorization

Currently only has basic API key auth (single shared key):

• ❌ No OAuth2/OIDC support
• ❌ No JWT token-based authentication
• ❌ No per-tenant API key management
• ❌ No RBAC (Role-Based Access Control)
• ❌ No user management within tenants
• ❌ No SSO integration
• ❌ No MFA support

3. Data Isolation & Security

All data currently shared in single deployment:

• ❌ No tenant ID in data models
• ❌ No row-level security in database queries
• ❌ No tenant-specific Redis namespaces
• ❌ No data segregation in event storage
• ❌ No tenant-specific encryption keys
• ❌ No data residency options

4. Resource Management & Quotas

• ❌ No usage tracking per tenant
• ❌ No rate limiting per tenant
• ❌ No quota management (requests/month, storage limits)
• ❌ No billing/metering integration
• ❌ No cost allocation tracking
• ❌ No SLA monitoring per tenant

5. Tenant-Specific Customization

• ❌ No custom detection thresholds per tenant
• ❌ No tenant-specific security rules
• ❌ No per-tenant model fine-tuning
• ❌ No custom whitelist/blacklist management
• ❌ No tenant-specific policy configurations
• ❌ No custom alert thresholds

6. Multi-Tenant Dashboard

Dashboard is listed as "in progress" with no tenant features:

• ❌ No tenant-specific security views
• ❌ No admin portal for tenant management
• ❌ No per-tenant usage analytics
• ❌ No cross-tenant reporting (for admins)
• ❌ No tenant isolation in UI
• ❌ No team/user management UI

7. Compliance & Audit

• ❌ No per-tenant audit logs
• ❌ No tenant-specific data retention policies
• ❌ No compliance reporting per tenant
• ❌ No tenant data export/deletion (GDPR)
• ❌ No tenant-level access logs

8. Integration Features

• ❌ No tenant-specific webhooks
• ❌ No custom integrations per tenant
• ❌ No tenant-specific SIEM connectors
• ❌ No per-tenant API configurations

📋 Recommended Implementation Order

Based on the roadmap (Phase 3: Week 9-12), here's what should be prioritized:

Phase 1: Foundation (Weeks 1-2)

# Example: Tenant data model
class Tenant(BaseModel):
    tenant_id: str
    name: str
    api_keys: List[str]
    settings: dict
    created_at: datetime
    plan_tier: str  # free, pro, enterprise
    
# Add tenant_id to all events
class LLMEventRequest(BaseModel):
    tenant_id: str  # NEW
    source_type: str
    source_id: str
    # ... rest of fields

Phase 2: Authentication (Weeks 3-4)

• JWT-based authentication
• Per-tenant API key management
• Basic RBAC (admin, user, viewer roles)

Phase 3: Data Isolation (Weeks 5-6)

• Add tenant_id to all database/Redis operations
• Implement query filters for tenant isolation
• Tenant-specific Redis namespaces

Phase 4: Resource Management (Weeks 7-8)

• Usage tracking and rate limiting
• Quota enforcement
• Basic billing hooks

Phase 5: Customization & Dashboard (Weeks 9-12)

• Tenant-specific configurations
• Multi-tenant dashboard UI
• Admin portal

🔍 Current Architecture Gaps

Looking at the service code:

Ingest Service (services/ingest/app.py):

# Current: No tenant awareness
verify_api_key(x_api_key)  # Single shared key

# Needed: 
# 1. Extract tenant_id from JWT or API key
# 2. Add tenant_id to event payload
# 3. Check tenant quota before processing
# 4. Use tenant-specific Redis namespace

Analyzer Service (services/analyzer/app.py):

# Current: Global ML model and thresholds
PROMPT_INJECTION_THRESHOLD = 0.75  # Same for all

# Needed:
# 1. Load tenant-specific thresholds
# 2. Apply tenant-specific rules
# 3. Isolate tenant data in processing queue
# 4. Track usage per tenant

📊 Database Schema Changes Needed

-- New tables required
CREATE TABLE tenants (
    tenant_id UUID PRIMARY KEY,
    name VARCHAR(255),
    api_key_hash VARCHAR(255),
    settings JSONB,
    plan_tier VARCHAR(50),
    created_at TIMESTAMP
);

CREATE TABLE users (
    user_id UUID PRIMARY KEY,
    tenant_id UUID REFERENCES tenants(tenant_id),
    email VARCHAR(255),
    role VARCHAR(50),
    created_at TIMESTAMP
);

CREATE TABLE tenant_quotas (
    tenant_id UUID REFERENCES tenants(tenant_id),
    requests_per_month INT,
    current_usage INT,
    reset_date TIMESTAMP
);

-- Modify existing tables
ALTER TABLE events ADD COLUMN tenant_id UUID REFERENCES tenants(tenant_id);
ALTER TABLE alerts ADD COLUMN tenant_id UUID REFERENCES tenants(tenant_id);

🎯 Quick Wins to Start

1. Add tenant_id field to all data models
2. Implement JWT authentication to replace single API key
3. Create tenant registration endpoint
4. Add tenant-based Redis namespacing: tenet:{tenant_id}:events:queue
5. Implement basic usage tracking per tenant

Would you like me to create an issue to track the implementation of these multi-tenant features, or would you prefer I help you plan the code changes to start implementing specific features?

[S3DFX-CYBER]

@coderabbitai create some good first issues from here which aren't outscoped for a contributor

And create a roadmap for me to fix all these issues which are in scope for a maintaner

[coderabbitai]

Warning

A plan generation is already in progress. Please wait for it to complete before requesting a new plan.

[coderabbitai]

Coding Plan

Summary

Create a comprehensive feature roadmap for TENET AI that identifies missing capabilities, defines good first issues for contributors, and establishes a maintainer roadmap for critical platform improvements including CI/CD hardening, observability, dashboard APIs, and integration SDKs.

Design Choices

Design Choice 1: Scope Separation: Good First Issues vs Maintainer Roadmap

Options Considered:

1. Combine all features into a single unified roadmap
2. Clearly separate self-contained contributor tasks from larger architectural work

Chosen Option: 2

Rationale: Separate contributor issues (self-contained, well-defined tasks) from maintainer roadmap (architectural changes requiring deeper context) to enable effective parallel contribution.

Design Choice 2: Dashboard Implementation Approach

Options Considered:

1. Build backend APIs and frontend simultaneously
2. Build backend APIs first, then frontend as a separate phase

Chosen Option: 2

Rationale: Backend-first approach allows API contracts to be defined and tested before frontend work, enables contributors to work on frontend once APIs are stable.

Design Choice 3: Coverage Reporting Integration

Options Considered:

1. Upload as GitHub artifact (simple, no external dependencies)
2. Integrate with Codecov (industry standard, free for open source, PR comments)
3. Integrate with Coveralls (similar to Codecov, less popular)

Chosen Option: 2

Rationale: Codecov integration - provides PR comments showing coverage changes, badge support, and is free for open source. If external service is not desired, fall back to artifact upload.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: CI/CD Hardening (Good First Issues)

This phase enables quality gates that are currently missing. All required tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in requirements-dev.txt but not executed in CI. These are ideal contributor tasks as they're self-contained configuration changes.

Task 1: Add Coverage Reporting to CI

Enable pytest-cov execution in the CI workflow with a minimum coverage threshold to prevent regression.

File to modify: .github/workflows/ci.yml

Replace the existing pytest step (around line 35-40) with:

      - name: Run unit tests with coverage
        run: |
          pytest tests/unit/ -v \
            --cov=services \
            --cov-report=xml \
            --cov-report=term-missing \
            --cov-fail-under=70

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          flags: unittests
          fail_ci_if_error: false
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Add to CONTRIBUTING.md under Testing section:

### Coverage Requirements

All contributions must maintain at least 70% test coverage. To check coverage locally:

\`\`\`bash
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70
\`\`\`

Coverage reports are automatically generated on pull requests via Codecov.

Verification:

# Run locally to verify it works before pushing
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70

Task 2: Enable Code Formatting and Linting Checks

Add Black formatting validation and Ruff linting as required CI checks to enforce code quality standards.

File to modify: .github/workflows/ci.yml

Add new job after the test job definition (add around line 45):

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install linting tools
        run: pip install black ruff
      
      - name: Check formatting with Black
        run: black --check --line-length 100 services/ tests/ scripts/
      
      - name: Lint with Ruff
        run: ruff check services/ tests/ scripts/

Create pyproject.toml in repository root (if not exists) or add these sections:

[tool.black]
line-length = 100
target-version = ['py311']
include = '\.pyi?$'
extend-exclude = '''
/(
  \.git
  | \.venv
  | __pycache__
  | models/trained
)/
'''

[tool.ruff]
line-length = 100
target-version = "py311"
select = ["E", "F", "W", "I", "UP", "B"]
ignore = ["E501"]  # Line length handled by Black

[tool.ruff.per-file-ignores]
"tests/*" = ["B011"]  # Allow assert in tests

Verification:

# Run locally before pushing
black --check --line-length 100 services/ tests/ scripts/
ruff check services/ tests/ scripts/

# Auto-fix issues if any
black --line-length 100 services/ tests/ scripts/
ruff check --fix services/ tests/ scripts/

Task 3: Enable Type Checking

Add mypy static type analysis to catch type errors before runtime.

File to modify: .github/workflows/ci.yml

Add to the lint job steps:

      - name: Install type checking tools
        run: pip install mypy types-redis
      
      - name: Type check with mypy
        run: mypy services/ --ignore-missing-imports --no-error-summary

Add to pyproject.toml:

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_ignores = true
ignore_missing_imports = true
exclude = [
    "tests/",
    "scripts/",
    "models/",
]

[[tool.mypy.overrides]]
module = "redis.*"
ignore_missing_imports = true

Verification:

pip install mypy types-redis
mypy services/ --ignore-missing-imports

Task 4: Integrate Security Scanning

Enable Bandit (SAST) and Safety (dependency vulnerabilities) in the CI pipeline.

File to modify: .github/workflows/ci.yml

Add new job:

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install security tools
        run: pip install bandit safety
      
      - name: Run Bandit security scan
        run: bandit -r services/ -ll -ii -x tests/
      
      - name: Check dependencies for vulnerabilities
        run: safety check -r requirements.txt --full-report
        continue-on-error: true  # Don't fail on advisory-only issues

Verification:

pip install bandit safety
bandit -r services/ -ll -ii -x tests/
safety check -r requirements.txt --full-report

🤖 Prompt for AI agents

Implement CI/CD quality gates for the TENET AI project. The goal is to enable
automated code quality, testing, and security checks with complete
copy-pasteable configurations.

**Coverage Reporting:**
- Modify `.github/workflows/ci.yml` around line 35-40 to replace the existing
pytest step
- Add pytest with coverage flags: `--cov=services --cov-report=xml
--cov-report=term-missing --cov-fail-under=70`
- Add Codecov upload step using `codecov/codecov-action@v4` with the
coverage.xml file
- Set flags to "unittests" and fail_ci_if_error to false
- Use CODECOV_TOKEN from GitHub secrets
- Update `CONTRIBUTING.md` Testing section with coverage requirements
documentation
- Add verification command to test locally

**Code Formatting and Linting:**
- Add new `lint` job to `.github/workflows/ci.yml` after the test job (around
line 45)
- Use ubuntu-latest runner with Python 3.11 and pip caching
- Install black and ruff
- Run Black check with `--check --line-length 100` on services/, tests/,
scripts/
- Run Ruff check on the same directories
- Create or update `pyproject.toml` with Black configuration: line-length 100,
target py311, exclude .git/.venv/__pycache__/models/trained
- Add Ruff configuration: line-length 100, select E/F/W/I/UP/B rules, ignore
E501
- Add per-file-ignores for tests to allow assert statements (B011)
- Include verification commands for local testing and auto-fixing

**Type Checking:**
- Add to the lint job steps in `.github/workflows/ci.yml`
- Install mypy and types-redis packages
- Run mypy on services/ with --ignore-missing-imports --no-error-summary flags
- Add `[tool.mypy]` section to `pyproject.toml`: python_version 3.11,
warn_return_any and warn_unused_ignores true, exclude tests/scripts/models/
- Add overrides section for redis.* modules to ignore missing imports
- Include verification command

**Security Scanning:**
- Add new `security` job to `.github/workflows/ci.yml`
- Use ubuntu-latest runner with Python 3.11 and pip caching
- Install bandit and safety packages
- Run Bandit with `-r services/ -ll -ii -x tests/` (medium+ severity, exclude
tests)
- Run Safety check on requirements.txt with --full-report and continue-on-error
true
- Include verification commands

Note: All tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in
requirements-dev.txt.

Phase 2: Observability Foundation (Good First Issues)

This phase addresses the critical gap of no production observability. The services currently use basic Python logging with no metrics or structured output.

Task 1: Implement Structured JSON Logging

Convert from basic text logging to structured JSON format with correlation IDs for request tracing.

Add to requirements.txt:

python-json-logger>=2.0.7

Create new file services/common/__init__.py:

# Empty file to make this a package

Create new file services/common/logging.py: (See full implementation in plan with TenetJsonFormatter class, setup_logging function, request ID management)

Update services/ingest/app.py - replace logging setup and add request ID middleware (complete code provided in plan)

Verification:

LOG_FORMAT=json python -c "
from services.common.logging import setup_logging
logger = setup_logging('test')
logger.info('Test message', extra={'custom_field': 'value'})
"
# Should output JSON with timestamp, level, service, etc.

Task 2: Add Prometheus Metrics Endpoint

Expose application metrics via a /metrics endpoint for Prometheus scraping.

Add to requirements.txt:

prometheus-client>=0.19.0

Create new file services/common/metrics.py: (Complete implementation provided with REQUEST_COUNT, REQUEST_LATENCY, THREATS_DETECTED, EVENTS_BLOCKED, QUEUE_DEPTH, MODEL_INFERENCE_LATENCY metrics)

Add to services/ingest/app.py: Import metrics, add /metrics endpoint, add metrics tracking middleware (complete code provided)

Verification:

# Start service and test
curl http://localhost:8000/metrics
# Should return Prometheus format metrics

Task 3: Enhanced Health Check Endpoints

Expand health checks to include dependency status and readiness/liveness separation.

Update services/ingest/app.py - replace existing health_check function with:

• GET /health/live for liveness checks (returns LivenessResponse with status, service, version)
• GET /health/ready for readiness checks (returns ReadinessResponse with status, service, version, checks dict)
• Legacy GET /health endpoint for backward compatibility
• Implement Redis connection checking with proper error handling
• Return HTTP 200 for ready/alive, 503 for not_ready states

Verification:

curl http://localhost:8000/health/live   # Should return 200
curl http://localhost:8000/health/ready  # Should return 200 or 503
curl http://localhost:8000/health        # Legacy endpoint

🤖 Prompt for AI agents

Implement observability infrastructure for TENET AI microservices. The goal is
to add complete structured logging, metrics collection, and enhanced health
checks with full implementation code.

**Structured JSON Logging:**
- Add `python-json-logger>=2.0.7` to requirements.txt
- Create `services/common/__init__.py` as empty package marker
- Create `services/common/logging.py` with full implementation:
  - Import logging, os, sys, uuid, contextvars, pythonjsonlogger
  - Create ContextVar for request_id with default empty string
  - Implement TenetJsonFormatter class extending jsonlogger.JsonFormatter
- In add_fields method: add service (from SERVICE_NAME env), version (from
SERVICE_VERSION env), request_id (from context var), level, timestamp
- Implement setup_logging(service_name) function: get LOG_LEVEL and LOG_FORMAT
from env, configure logger with StreamHandler, apply JSON or text formatter
based on format
  - Implement generate_request_id() returning 8-char UUID
  - Implement set_request_id(request_id) to update context var
- Update `services/ingest/app.py`:
- Replace lines 17-21 with: import sys, add parent to path, import from
common.logging
  - Call setup_logging('ingest') to get logger
- Add middleware function add_request_id that extracts/generates X-Request-ID
header, sets context var, calls next, adds header to response
- Include verification command using LOG_FORMAT=json

**Prometheus Metrics:**
- Add `prometheus-client>=0.19.0` to requirements.txt
- Create `services/common/metrics.py` with complete implementation:
- Import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST from
prometheus_client
  - Define REQUEST_COUNT Counter with labels: service, endpoint, method, status
- Define REQUEST_LATENCY Histogram with labels: service, endpoint and buckets:
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0
  - Define THREATS_DETECTED Counter with labels: threat_type, verdict
  - Define EVENTS_BLOCKED Counter with label: threat_type
  - Define QUEUE_DEPTH Gauge with label: queue_name
- Define MODEL_INFERENCE_LATENCY Histogram with label: model_type and buckets:
.001, .005, .01, .025, .05, .1, .25, .5
- Implement metrics_endpoint() returning Response with generate_latest() content
- Update `services/ingest/app.py`:
  - Import metrics objects and metrics_endpoint function, import time
  - Add route @app.get("/metrics") calling metrics_endpoint()
- Add middleware track_metrics: capture start time, call next, calculate
duration, increment REQUEST_COUNT with all labels, observe REQUEST_LATENCY
- Include verification using curl to /metrics endpoint

**Enhanced Health Checks:**
- Update `services/ingest/app.py` replacing existing health_check function:
  - Create LivenessResponse Pydantic model with status, service, version fields
- Create ReadinessResponse Pydantic model with status, service, version, checks
dict fields
- Implement @app.get("/health/live") returning LivenessResponse with
status="alive", service="ingest", version="0.1.0"
  - Implement @app.get("/health/ready") with full readiness logic:
    - Initialize checks dict and overall_status="ready"
- Try redis_client.ping(), set checks["redis"]="connected" on success,
"not_configured" if no client, "error" with message on exception
    - Set overall_status to "degraded" or "not_ready" based on checks
    - Return 503 status code via JSONResponse if not ready, otherwise 200
- Implement legacy @app.get("/health") calling readiness() and formatting
response
- Include verification commands for all three endpoints

Phase 3: Dashboard Backend API Layer (Maintainer Roadmap)

This phase creates the API foundation required for the SOC dashboard. The current API only supports single event submission/retrieval with no listing, search, or analytics capabilities.

Task 1: Implement Events Listing and Search API

Create endpoints to list, filter, and search security events for dashboard consumption.

Add to services/ingest/app.py:

• Import Optional, List from typing, datetime and timedelta
• Create EventListResponse Pydantic model with total_count, page, page_size, events fields
• Implement @app.get("/v1/events", response_model=EventListResponse) endpoint
• Parameters: api_key (Depends), verdict/threat_type/source_type/hours/page/page_size (query params)
• Implementation:
  • Check Redis availability
  • Scan Redis keys matching "tenet:event:*" pattern
  • Load and parse events, apply filters (verdict, threat_type, source_type)
  • Sort by timestamp descending
  • Paginate results using page and page_size
  • Return EventListResponse with counts and paginated events
• Note: Production should migrate to PostgreSQL for better querying

Task 2: Implement Statistics and Analytics API

Create aggregation endpoints for dashboard charts and metrics visualization.

Add to services/ingest/app.py:

• Create StatsResponse Pydantic model with total_events, blocked_count, threats_by_type dict, verdicts dict, average_risk_score, time_range_hours
• Implement @app.get("/v1/stats", response_model=StatsResponse) endpoint
• Parameters: api_key (Depends), hours (query param, default 24)
• Implementation:
  • Check Redis availability
  • Scan all tenet:event:* keys
  • Aggregate statistics: total events, blocked count, threats by type, verdicts breakdown (benign/suspicious/malicious), risk scores
  • Calculate average risk score from collected scores
  • Return StatsResponse with all aggregated data rounded appropriately

Task 3: Implement Alerts Management API

Create CRUD endpoints for security alerts with analyst workflow support.

Add to services/ingest/app.py:

• Create AlertStatus Enum with values: new, acknowledged, investigating, resolved, false_positive
• Create Alert Pydantic model with alert_id, event_id, severity, status, threat_type, created_at, updated_at, notes list
• Create AlertUpdate model with optional status field
• Create AlertNote model with text and author (default "analyst")
• Implement GET /v1/alerts for listing with filtering by status, severity, pagination
• Implement PUT /v1/alerts/{alert_id} for updating alert status
• Implement POST /v1/alerts/{alert_id}/notes for adding analyst notes
• Use Redis for storage (tenet:alert:{id} keys and tenet:alerts list)
• Include timestamp updates on modifications

Task 4: Add Real-Time Event Streaming

Implement WebSocket or Server-Sent Events for live dashboard updates.

Add to services/ingest/app.py:

• Import WebSocket, WebSocketDisconnect, Set, asyncio
• Create module-level active_connections: Set[WebSocket]
• Implement @app.websocket("/v1/events/stream") endpoint:
  • Accept WebSocket connection and add to active_connections
  • Subscribe to Redis pubsub channel "tenet:events:new"
  • Loop: get messages from pubsub, send to WebSocket
  • Handle client messages for filter updates
  • Handle WebSocketDisconnect, cleanup subscriptions
• Implement broadcast_event(event) function to send to all connected clients with error handling
• Modify ingest_llm_event function to publish events to "tenet:events:new" channel after queuing

🤖 Prompt for AI agents

Build the comprehensive backend API layer for the SOC dashboard. The goal is to
create complete endpoints for event management, analytics, alerts, and real-time
streaming with full implementation code.

**Events Listing and Search:**
- Add imports to `services/ingest/app.py`: Optional, List from typing, datetime,
timedelta
- Create EventListResponse Pydantic model: total_count (int), page (int),
page_size (int), events (List[dict])
- Implement `@app.get("/v1/events", response_model=EventListResponse)` with
parameters:
  - api_key: str = Depends(verify_api_key)
  - verdict, threat_type, source_type: Optional[str] = None
  - hours: int = 24, page: int = 1, page_size: int = 50
- Implementation logic:
  - Check redis_client availability (raise 503 if not)
  - Initialize cursor=0, all_events=[]
  - Use redis_client.scan with pattern="tenet:event:*", count=100
- For each key: get event data, parse JSON, apply filters (verdict, threat_type,
source_type), append to all_events
  - Loop until cursor==0
  - Sort all_events by timestamp descending
  - Calculate pagination: start=(page-1)*page_size, end=start+page_size
  - Return EventListResponse with total_count, page, page_size, paginated events
- Add comment: Production should use PostgreSQL with proper indexing

**Statistics and Analytics:**
- Create StatsResponse Pydantic model: total_events (int), blocked_count (int),
threats_by_type (dict), verdicts (dict), average_risk_score (float),
time_range_hours (int)
- Implement `@app.get("/v1/stats", response_model=StatsResponse)` with
parameters:
  - api_key: str = Depends(verify_api_key)
  - hours: int = 24
- Implementation logic:
  - Check redis_client availability
- Initialize stats dict with: total=0, blocked=0, threats={},
verdicts={'benign':0,'suspicious':0,'malicious':0}, risk_scores=[]
  - Use redis_client.scan with same pattern as events listing
- For each event: increment total, check blocked and increment, aggregate
threat_type counts, aggregate verdict counts, collect risk_scores
  - Calculate avg_risk: sum/len if risk_scores else 0.0
  - Return StatsResponse with all aggregated data, round avg_risk to 3 decimals

**Alerts Management:**
- Create AlertStatus str Enum: new, acknowledged, investigating, resolved,
false_positive
- Create Alert Pydantic model: alert_id (str), event_id (str), severity (str),
status (AlertStatus), threat_type (str), created_at (str), updated_at (str),
notes (List[dict], default [])
- Create AlertUpdate model: status (Optional[AlertStatus])
- Create AlertNote model: text (str), author (str, default="analyst")
- Implement `@app.get("/v1/alerts")` with parameters: api_key, status
(Optional[AlertStatus]), severity (Optional[str]), page (int=1), page_size
(int=50)
  - Get alerts from redis_client.lrange("tenet:alerts", 0, -1)
  - Parse JSON, filter by status and severity
  - Paginate and return with total_count
- Implement `@app.put("/v1/alerts/{alert_id}")` with alert_id (str), update
(AlertUpdate), api_key
  - Get alert from "tenet:alert:{alert_id}" key
  - Raise 404 if not found
  - Update status if provided, set updated_at to current ISO timestamp
  - Save back to Redis, return updated alert
- Implement `@app.post("/v1/alerts/{alert_id}/notes")` with alert_id (str), note
(AlertNote), api_key
  - Get alert, raise 404 if not found
  - Append note to notes array with text, author, timestamp
  - Update updated_at field
  - Save to Redis, return status and alert

**Real-Time Streaming:**
- Import: WebSocket, WebSocketDisconnect from fastapi, Set from typing, asyncio
- Create module-level: active_connections: Set[WebSocket] = set()
- Implement `@app.websocket("/v1/events/stream")`:
  - Accept websocket connection
  - Add to active_connections
  - Try block:
    - Create pubsub = redis_client.pubsub()
    - Subscribe to "tenet:events:new"
    - While True loop:
      - Get message with timeout=1.0, ignore_subscribe_messages=True
      - If message type is 'message': send decoded data to websocket
      - Try to receive text from websocket with timeout=0.1 for filter updates
      - Log received data
  - Except WebSocketDisconnect: discard from active_connections
  - Finally: unsubscribe and discard from active_connections
- Implement async broadcast_event(event: dict):
  - Check if active_connections not empty
  - Convert event to JSON string
  - For each connection (copy of set):
    - Try to send_text, on exception discard connection
- Modify ingest_llm_event function:
- After queuing event to Redis, add: await
redis_client.publish("tenet:events:new", json.dumps(response_data))

Phase 4: Python SDK and Integration Library (Mixed - Some Good First Issues)

This phase creates client libraries to simplify TENET AI integration into applications.

Task 1: Create Python SDK Client Library

Build a Python package that wraps TENET AI's REST API for easy integration.

Create directory structure:

sdk/
└── python/
    ├── pyproject.toml
    ├── README.md
    └── tenet_ai/
        ├── __init__.py
        ├── client.py
        └── exceptions.py

Files to create:

1. sdk/python/tenet_ai/__init__.py - Export TenetClient, AsyncTenetClient, exceptions
2. sdk/python/tenet_ai/exceptions.py - Define TenetError, TenetBlockedError (with risk_score, threat_type), TenetAuthError
3. sdk/python/tenet_ai/client.py - Complete implementation of TenetClient and AsyncTenetClient (full code provided in plan)
4. sdk/python/pyproject.toml - Package configuration with dependencies: httpx>=0.25.0, optional dependencies for langchain and fastapi integrations

Task 2: Create LangChain Integration Plugin

Build a native LangChain callback handler as a proper package rather than example code.

Create files:

1. sdk/python/tenet_ai/integrations/__init__.py - Empty package marker
2. sdk/python/tenet_ai/integrations/langchain.py - Complete TenetCallbackHandler implementation (full code provided in plan)

Implementation includes:

• TenetCallbackHandler extending BaseCallbackHandler
• Initialization with api_key, base_url, blocking_mode, log_only, source_type
• on_llm_start hook for LLM interception
• on_chain_start hook for chain interception
• Exception handling with proper error propagation
• Metadata tracking for run IDs, tags, parent runs

Usage example provided for documentation

Task 3: Create FastAPI Middleware Package

Build a drop-in middleware that automatically protects FastAPI applications.

Create file:

sdk/python/tenet_ai/integrations/fastapi.py - Complete TenetMiddleware implementation (full code provided in plan)

Implementation includes:

• TenetMiddleware extending BaseHTTPMiddleware
• Initialization with api_key, base_url, protected_paths, excluded_paths, blocking_mode, custom extract_prompt function
• Default prompt extraction from common fields (prompt, message, query, input, text, content) and OpenAI messages format
• Path protection logic with exclusions for health/metrics/docs endpoints
• Request body parsing and prompt extraction
• Event submission to TENET AI with metadata
• Blocking response (403) with event details when threats detected
• X-Tenet headers added to responses (Event-ID, Risk-Score)
• Graceful degradation if TENET unavailable

Usage example provided for documentation

🤖 Prompt for AI agents

Create comprehensive Python SDK and integration libraries for TENET AI. The goal
is to provide complete, production-ready client libraries and framework
integrations with full implementation code.

**Python SDK Client:**
- Create directory structure: `sdk/python/tenet_ai/` with __init__.py,
client.py, exceptions.py
- Create `sdk/python/tenet_ai/exceptions.py`:
  - Define TenetError(Exception) base class
- Define TenetBlockedError(TenetError) with __init__(message, risk_score,
threat_type), store risk_score and threat_type as attributes
  - Define TenetAuthError(TenetError)
- Create `sdk/python/tenet_ai/client.py`:
  - Import httpx, Optional, Dict, Any from typing, exceptions module
  - Implement TenetClient class:
- __init__ with api_key, base_url (default http://localhost:8000), timeout
(10.0), blocking_mode (True)
    - Store attributes, create httpx.Client with timeout
    - _headers method returning X-API-Key and Content-Type headers
- submit_event method: build payload with
prompt/source_type/source_id/model/system_prompt/metadata, POST to
/v1/events/llm, handle 401 as TenetAuthError, raise TenetBlockedError if blocked
and blocking_mode
- analyze method: POST to analyzer service :8100/v1/analyze with prompt and
context
    - get_event method: GET /v1/events/{event_id}
    - get_stats method: GET /v1/stats with hours parameter
    - close method and context manager support (__enter__, __exit__)
- Implement AsyncTenetClient class with async versions of all methods using
httpx.AsyncClient
- Create `sdk/python/tenet_ai/__init__.py`: import and export TenetClient,
AsyncTenetClient, all exceptions, set __version__="0.1.0"
- Create `sdk/python/pyproject.toml`:
  - build-system: setuptools>=61.0
- project name="tenet-ai", version="0.1.0", description, readme,
requires-python=">=3.9", license=MIT
  - dependencies: httpx>=0.25.0
- optional-dependencies: langchain=["langchain>=0.1.0"],
fastapi=["fastapi>=0.100.0", "starlette>=0.27.0"], dev=["pytest",
"pytest-asyncio"]
  - project.urls with Homepage, Documentation

**LangChain Integration:**
- Create `sdk/python/tenet_ai/integrations/__init__.py` as empty package
- Create `sdk/python/tenet_ai/integrations/langchain.py`:
- Import typing types, UUID, try/except import BaseCallbackHandler from
langchain.callbacks.base with helpful error message
  - Import TenetClient, AsyncTenetClient, TenetBlockedError from parent modules
  - Implement TenetCallbackHandler(BaseCallbackHandler):
- __init__ with api_key, base_url (default localhost:8000), blocking_mode
(True), log_only (False), source_type ("langchain")
    - Create TenetClient with blocking_mode and not log_only
    - Store log_only and source_type
- Implement on_llm_start(serialized, prompts, run_id, parent_run_id, tags,
metadata, kwargs):
      - Get model_name from serialized.get("name", "unknown")
- For each prompt: submit_event with prompt, source_type, run_id as source_id,
model_name, metadata dict with tags/parent_run_id
      - If blocked and not log_only: raise TenetBlockedError
- Catch TenetBlockedError and re-raise, catch other exceptions and print warning
- Implement on_chain_start with similar logic: extract input_text from inputs
(try "input", "query", "question" keys), submit_event with
source_type="{source_type}-chain", metadata with chain_inputs keys
- Add usage example to documentation showing: create TenetCallbackHandler, use
with LangChain LLM via callbacks parameter, demonstrate blocking behavior

**FastAPI Middleware:**
- Create `sdk/python/tenet_ai/integrations/fastapi.py`:
- Try/except import Request, Response from fastapi and BaseHTTPMiddleware from
starlette with helpful error message
  - Import typing, json, TenetClient, TenetBlockedError
  - Implement TenetMiddleware(BaseHTTPMiddleware):
- __init__(app, api_key, base_url="http://localhost:8000", protected_paths=None,
excluded_paths=None, blocking_mode=True, extract_prompt=None)
    - Call super().__init__(app)
    - Create TenetClient with blocking_mode=False (we handle it)
    - Set protected_paths default to ["/api/", "/v1/"]
- Set excluded_paths default to {"/health", "/metrics", "/docs", "/redoc",
"/openapi.json"}
    - Store blocking_mode and extract_prompt or _default_extract_prompt
    - Implement _default_extract_prompt(body: dict) -> Optional[str]:
      - Check common keys: prompt, message, query, input, text, content
      - Check messages array for OpenAI format (role="user")
      - Return None if not found
- Implement _should_protect(path: str) -> bool: check if path in excluded_paths
or starts with any protected_paths prefix
    - Implement async dispatch(request, call_next):
      - Skip if not _should_protect or method not in POST/PUT
      - Read and parse request body as JSON
      - Extract prompt using self.extract_prompt
      - Skip if no prompt found
      - Submit event to TENET with metadata (method, path, client IP)
- If blocked and blocking_mode: return Response with 403, JSON body with
error/event_id/threat_type
      - Otherwise call next and add X-Tenet headers (Event-ID, Risk-Score)
      - Catch all exceptions and print warning, continue to call_next
- Add usage example showing: create FastAPI app, add TenetMiddleware with
configuration, protected endpoint that gets automatic protection

Phase 5: Security Hardening (Good First Issues + Maintainer)

This phase addresses critical security gaps identified in the codebase, particularly hardcoded credentials and configuration validation.

Task 1: Remove Hardcoded Credentials from Docker Compose

Eliminate default passwords and enforce environment variable usage for all secrets.

Modify docker-compose.yml:

Replace lines 27-31 (PostgreSQL section):

  postgres:
    image: postgres:15-alpine
    container_name: tenet-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER is required}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
      POSTGRES_DB: ${POSTGRES_DB:-tenet}

Replace lines 45-51 (MinIO section):

  minio:
    image: minio/minio:latest
    container_name: tenet-minio
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:?MINIO_ROOT_USER is required}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}

Update .env.template - add comprehensive configuration sections with security warnings

Create scripts/check_env.sh - bash script to verify environment variables (complete implementation provided in plan)

Verification:

chmod +x scripts/check_env.sh
./scripts/check_env.sh  # Should fail if using defaults

Task 2: Implement Configuration Validation

Add startup checks to validate required configuration and fail fast on misconfiguration.

Create services/common/config.py:

• Complete TenetConfig Pydantic BaseSettings class implementation (full code provided in plan)
• Fields: api_key (SecretStr), api_host, api_port, redis_host, redis_port, rate_limit_rpm, max_prompt_length, blocking_enabled, log_level, log_format, environment
• Validators for: api_key (minimum 20 chars, check insecure defaults), environment (production checks), log_level (valid values)
• load_config() function with error handling and sys.exit on failure
• log_config_summary() function to log configuration without exposing secrets

Update service startup:

• Import load_config and log_config_summary in services/ingest/app.py
• Call load_config() at module level
• Add startup event handler calling log_config_summary()

Task 3: Add API Key Rotation Support

Implement support for multiple API keys and graceful rotation without downtime.

• Modify API key validation to accept comma-separated list of valid keys
• Add key metadata support (name, created_at, expires_at)
• Add /v1/admin/keys endpoint (admin-authenticated) for key management
• Implement key expiration checking and warnings
• Add audit logging for key usage and rotation events

🤖 Prompt for AI agents

Implement comprehensive security hardening for TENET AI. The goal is to
eliminate hardcoded credentials, add startup configuration validation with
Pydantic, and support API key rotation.

**Remove Hardcoded Credentials:**
- Modify `docker-compose.yml`:
- Lines 27-31 (postgres section): Replace POSTGRES_PASSWORD: postgres with
${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
  - Also use ${POSTGRES_USER:?POSTGRES_USER is required} syntax
  - Set POSTGRES_DB with default: ${POSTGRES_DB:-tenet}
- Lines 45-51 (minio section): Replace MINIO_ROOT_USER: minio and
MINIO_ROOT_PASSWORD: minio123 with ${MINIO_ROOT_USER:?MINIO_ROOT_USER is
required} and ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}
- Update `.env.template`:
  - Add DATABASE CONFIGURATION section header
  - Add comment about generating secure passwords: "openssl rand -base64 32"
- Add POSTGRES_USER=tenet, POSTGRES_PASSWORD= (empty, marked REQUIRED with min
16 chars note), POSTGRES_DB=tenet
  - Add OBJECT STORAGE section header
- Add MINIO_ROOT_USER= (empty, marked REQUIRED with min 8 chars),
MINIO_ROOT_PASSWORD= (empty, marked REQUIRED with min 16 chars)
  - Add API AUTHENTICATION section header
- Add generation command: python -c "import secrets;
print(f'tenet-{secrets.token_urlsafe(32)}')"
  - Add API_KEY= (empty, marked REQUIRED)
- Create `scripts/check_env.sh` bash script:
  - Define color codes: RED, GREEN, NC
  - Initialize ERRORS=0 counter
- Define check_var function: takes var_name and min_length, checks if set and
meets length, increments ERRORS and prints colored messages
- Define check_not_default function: takes var_name and default_value, checks if
using default, increments ERRORS
  - Print header "Checking TENET AI environment configuration..."
- Call check_var for: POSTGRES_PASSWORD (16), MINIO_ROOT_USER (8),
MINIO_ROOT_PASSWORD (16), API_KEY (20)
- Call check_not_default for: POSTGRES_PASSWORD ("postgres"),
MINIO_ROOT_PASSWORD ("minio123"), API_KEY ("tenet-dev-key-change-in-production")
  - Print summary with error count, exit 1 if errors else print success message
- Add verification: chmod +x and run script expecting failure with defaults

**Configuration Validation:**
- Create `services/common/config.py`:
- Import os, sys, re, Optional from typing, BaseSettings, validator, SecretStr
from pydantic
  - Define TenetConfig(BaseSettings) class:
- Fields: api_key (SecretStr), api_host (str, default "0.0.0.0"), api_port (int,
default 8000), redis_host (str, default "localhost"), redis_port (int, default
6379), rate_limit_rpm (int, default 1000), max_prompt_length (int, default
10000), blocking_enabled (bool, default True), log_level (str, default "INFO"),
log_format (str, default "json"), environment (str, default "development")
    - Config class: env_file=".env", env_file_encoding="utf-8"
- @validator("api_key"): get secret value, check length >= 20, check not in
insecure_defaults list ["tenet-dev-key-change-in-production", "changeme",
"test", "secret"], raise ValueError with clear messages
- @validator("environment"): if v=="production", check api_key doesn't contain
"dev" or "test"
- @validator("log_level"): check against valid_levels ["DEBUG", "INFO",
"WARNING", "ERROR", "CRITICAL"], return uppercase
- Define load_config() -> TenetConfig: try to instantiate TenetConfig, on
exception print to stderr and sys.exit(1)
- Define log_config_summary(config, logger): call logger.info with extra dict
containing all non-secret config fields (api_host, api_port, redis_host,
redis_port, environment, log_level, blocking_enabled, rate_limit_rpm)
- Update `services/ingest/app.py`:
  - Import load_config, log_config_summary from common.config
  - Call config = load_config() at module level
- Add @app.on_event("startup") function that calls log_config_summary(config,
logger)

**API Key Rotation:**
- Modify API key validation logic to split on comma and check against list of
keys
- Define key metadata structure with name, created_at, expires_at fields
- Create `/v1/admin/keys` endpoint requiring admin authentication:
  - GET to list all keys (without revealing full key)
  - POST to create new key
  - DELETE to revoke key
- Implement expiration checking: compare expires_at to current time, log
warnings
- Add audit logging: log all key validation attempts, key creation, key
revocation with timestamps and metadata

Phase 6: Documentation and Contributor Onboarding

This phase ensures new contributors can effectively work on the codebase and that features are properly documented.

Task 1: Create Good First Issues in GitHub

Document and create GitHub issues for contributor-friendly tasks identified in this plan.

Issue Template for Task 1.1 (Coverage):

## Title: Add pytest coverage reporting to CI

### Description
Enable code coverage measurement and reporting in our GitHub Actions CI workflow. The tools are already installed (`pytest-cov` in requirements-dev.txt) but not configured.

### Acceptance Criteria
- [ ] CI runs pytest with coverage flags
- [ ] Coverage report uploaded to Codecov
- [ ] Build fails if coverage drops below 70%
- [ ] CONTRIBUTING.md updated with coverage requirements

### Implementation
Add these changes to `.github/workflows/ci.yml`:
```yaml
- name: Run unit tests with coverage
  run: |
    pytest tests/unit/ -v \
      --cov=services \
      --cov-report=xml \
      --cov-fail-under=70

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    file: ./coverage.xml

Files to Modify

• .github/workflows/ci.yml
• CONTRIBUTING.md

Labels

good first issue, CI/CD, documentation

**Create similar issues for:**
- Task 1.2: Enable linting in CI
- Task 1.3: Enable type checking in CI
- Task 1.4: Add security scanning to CI
- Task 2.1: Implement structured JSON logging
- Task 2.2: Add Prometheus metrics endpoint
- Task 5.1: Remove hardcoded credentials

</details>


<details>
<summary><b>Task 2: Create API Reference Documentation</b></summary>

Document all existing and new API endpoints for developer consumption.

- Create `docs/api-reference.md` with OpenAPI/Swagger documentation
- Enable FastAPI's automatic docs at `/docs` and `/redoc` endpoints
- Document authentication requirements, request/response schemas, error codes
- Add example requests using curl and Python
- Include rate limiting and quota information

</details>


<details>
<summary><b>Task 3: Create Integration Guide</b></summary>

Write comprehensive documentation for integrating TENET AI into applications.

- Create `docs/integrations.md` covering all integration patterns
- Document SDK installation and usage
- Provide step-by-step guides for LangChain, FastAPI middleware, and direct API integration
- Include troubleshooting section for common issues
- Add architecture diagrams showing integration points

</details>


<details>
<summary><b>🤖 Prompt for AI agents</b></summary>

Create comprehensive documentation and contributor onboarding materials for
TENET AI. The goal is to enable new contributors with detailed GitHub issues and
provide clear API and integration documentation.

GitHub Issues Creation:

• Create detailed issue template for Task 1.1 (Coverage):
  • Title: "Add pytest coverage reporting to CI"
• Description section explaining the current state (tools installed but not
  configured)
• Acceptance Criteria checklist: CI runs pytest with coverage, upload to
  Codecov, fail below 70%, update CONTRIBUTING.md
• Implementation section with complete YAML code block showing pytest coverage
  step and Codecov upload step
  • Files to Modify: .github/workflows/ci.yml, CONTRIBUTING.md
  • Labels: good first issue, CI/CD, documentation
• Create similar detailed issues for:
  • Task 1.2: Enable linting (Black and Ruff configuration)
  • Task 1.3: Enable type checking (mypy setup)
  • Task 1.4: Security scanning (Bandit and Safety)
  • Task 2.1: Structured JSON logging (python-json-logger)
  • Task 2.2: Prometheus metrics endpoint
  • Task 5.1: Remove hardcoded credentials from docker-compose
• Each issue should include: clear problem statement, step-by-step acceptance
  criteria, implementation guidance with code snippets, file paths, helpful labels
• Link all issues back to this roadmap document for context

API Reference Documentation:

• Create docs/api-reference.md with comprehensive API documentation
• Enable FastAPI's built-in documentation:
  • Configure /docs endpoint for Swagger UI
  • Configure /redoc endpoint for ReDoc interface
• Document for each endpoint:
  • HTTP method and path
  • Authentication requirements (X-API-Key header)
  • Request schema with field descriptions and types
  • Response schema with examples
• Error codes: 200 (success), 401 (unauthorized), 403 (blocked), 404 (not
  found), 503 (service unavailable)
• Add example requests:
  • curl commands with headers and JSON bodies
  • Python examples using requests library and SDK
• Include operational details:
  • Rate limiting: requests per minute limits
  • Quota information
  • Pagination parameters and behavior

Integration Guide:

• Create docs/integrations.md covering all integration patterns
• SDK Installation section:
  • pip install commands
  • Optional dependencies [langchain] [fastapi]
• Direct API Integration:
  • Authentication setup
  • Example HTTP requests
  • Error handling patterns
• LangChain Integration:
  • Step-by-step setup of TenetCallbackHandler
  • Configuration options: blocking_mode, log_only
  • Code examples for LLM and chain integration
  • Callback lifecycle explanation
• FastAPI Middleware Integration:
  • Step-by-step setup of TenetMiddleware
  • Configuration: protected_paths, excluded_paths, custom extractors
  • Code examples for adding to FastAPI app
  • Request/response flow explanation
• Troubleshooting section:
  • Common issues: connection errors, authentication failures, timeout handling
  • Solutions and workarounds
  • Debug logging guidance
• Architecture diagrams:
  • Request flow showing where TENET intercepts
  • Integration points in application architecture
  • Data flow from client through TENET to LLM

</details>

<details>
<summary><b>Research</b></summary>

TENET AI is a defensive security middleware for LLM applications, acting as a firewall/IDS that intercepts prompts, detects adversarial attacks (prompt injection, jailbreaks, data extraction), and provides security visibility.

The system consists of two FastAPI microservices:
- **Ingest service** (port 8000) for fast heuristic checks and event queuing
- **Analyzer service** (port 8100) for ML-based threat detection using scikit-learn models

Events flow through Redis queues, with PostgreSQL and MinIO available but not yet utilized.

**Current State:**
- Solid unit tests but gaps in CI/CD automation (linting, coverage, security scanning tools are installed but not executed)
- No observability infrastructure (metrics, structured logging)
- No SOC dashboard implementation (only documented as planned)
- No SDK/client libraries for framework integrations

</details>


---



### 🚀 Next Steps


<details>
<summary><b>🤖 All AI agent prompts combined</b></summary>

Task: 1

Implement CI/CD quality gates for the TENET AI project. The goal is to enable
automated code quality, testing, and security checks with complete
copy-pasteable configurations.

Coverage Reporting:

• Modify .github/workflows/ci.yml around line 35-40 to replace the existing
  pytest step
• Add pytest with coverage flags: --cov=services --cov-report=xml --cov-report=term-missing --cov-fail-under=70
• Add Codecov upload step using codecov/codecov-action@v4 with the
  coverage.xml file
• Set flags to "unittests" and fail_ci_if_error to false
• Use CODECOV_TOKEN from GitHub secrets
• Update CONTRIBUTING.md Testing section with coverage requirements
  documentation
• Add verification command to test locally

Code Formatting and Linting:

• Add new lint job to .github/workflows/ci.yml after the test job (around
  line 45)
• Use ubuntu-latest runner with Python 3.11 and pip caching
• Install black and ruff
• Run Black check with --check --line-length 100 on services/, tests/,
  scripts/
• Run Ruff check on the same directories
• Create or update pyproject.toml with Black configuration: line-length 100,
  target py311, exclude .git/.venv/pycache/models/trained
• Add Ruff configuration: line-length 100, select E/F/W/I/UP/B rules, ignore
  E501
• Add per-file-ignores for tests to allow assert statements (B011)
• Include verification commands for local testing and auto-fixing

Type Checking:

• Add to the lint job steps in .github/workflows/ci.yml
• Install mypy and types-redis packages
• Run mypy on services/ with --ignore-missing-imports --no-error-summary flags
• Add [tool.mypy] section to pyproject.toml: python_version 3.11,
  warn_return_any and warn_unused_ignores true, exclude tests/scripts/models/
• Add overrides section for redis.* modules to ignore missing imports
• Include verification command

Security Scanning:

• Add new security job to .github/workflows/ci.yml
• Use ubuntu-latest runner with Python 3.11 and pip caching
• Install bandit and safety packages
• Run Bandit with -r services/ -ll -ii -x tests/ (medium+ severity, exclude
  tests)
• Run Safety check on requirements.txt with --full-report and continue-on-error
  true
• Include verification commands

Note: All tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in
requirements-dev.txt.

Task: 2

Implement observability infrastructure for TENET AI microservices. The goal is
to add complete structured logging, metrics collection, and enhanced health
checks with full implementation code.

Structured JSON Logging:

• Add python-json-logger>=2.0.7 to requirements.txt
• Create services/common/__init__.py as empty package marker
• Create services/common/logging.py with full implementation:
  • Import logging, os, sys, uuid, contextvars, pythonjsonlogger
  • Create ContextVar for request_id with default empty string
  • Implement TenetJsonFormatter class extending jsonlogger.JsonFormatter
• In add_fields method: add service (from SERVICE_NAME env), version (from
  SERVICE_VERSION env), request_id (from context var), level, timestamp
• Implement setup_logging(service_name) function: get LOG_LEVEL and LOG_FORMAT
  from env, configure logger with StreamHandler, apply JSON or text formatter
  based on format
  • Implement generate_request_id() returning 8-char UUID
  • Implement set_request_id(request_id) to update context var
• Update services/ingest/app.py:
• Replace lines 17-21 with: import sys, add parent to path, import from
  common.logging
  • Call setup_logging('ingest') to get logger
• Add middleware function add_request_id that extracts/generates X-Request-ID
  header, sets context var, calls next, adds header to response
• Include verification command using LOG_FORMAT=json

Prometheus Metrics:

• Add prometheus-client>=0.19.0 to requirements.txt
• Create services/common/metrics.py with complete implementation:
• Import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST from
  prometheus_client
  • Define REQUEST_COUNT Counter with labels: service, endpoint, method, status
• Define REQUEST_LATENCY Histogram with labels: service, endpoint and buckets:
  .005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0
  • Define THREATS_DETECTED Counter with labels: threat_type, verdict
  • Define EVENTS_BLOCKED Counter with label: threat_type
  • Define QUEUE_DEPTH Gauge with label: queue_name
• Define MODEL_INFERENCE_LATENCY Histogram with label: model_type and buckets:
  .001, .005, .01, .025, .05, .1, .25, .5
• Implement metrics_endpoint() returning Response with generate_latest() content
• Update services/ingest/app.py:
  • Import metrics objects and metrics_endpoint function, import time
  • Add route @app.get("/metrics") calling metrics_endpoint()
• Add middleware track_metrics: capture start time, call next, calculate
  duration, increment REQUEST_COUNT with all labels, observe REQUEST_LATENCY
• Include verification using curl to /metrics endpoint

Enhanced Health Checks:

• Update services/ingest/app.py replacing existing health_check function:
  • Create LivenessResponse Pydantic model with status, service, version fields
• Create ReadinessResponse Pydantic model with status, service, version, checks
  dict fields
• Implement @app.get("/health/live") returning LivenessResponse with
  status="alive", service="ingest", version="0.1.0"
  • Implement @app.get("/health/ready") with full readiness logic:
    • Initialize checks dict and overall_status="ready"
• Try redis_client.ping(), set checks["redis"]="connected" on success,
  "not_configured" if no client, "error" with message on exception
  • Set overall_status to "degraded" or "not_ready" based on checks
  • Return 503 status code via JSONResponse if not ready, otherwise 200
• Implement legacy @app.get("/health") calling readiness() and formatting
  response
• Include verification commands for all three endpoints
  ===============================================================================

Task: 3

Build the comprehensive backend API layer for the SOC dashboard. The goal is to
create complete endpoints for event management, analytics, alerts, and real-time
streaming with full implementation code.

Events Listing and Search:

• Add imports to services/ingest/app.py: Optional, List from typing, datetime,
  timedelta
• Create EventListResponse Pydantic model: total_count (int), page (int),
  page_size (int), events (List[dict])
• Implement @app.get("/v1/events", response_model=EventListResponse) with
  parameters:
  • api_key: str = Depends(verify_api_key)
  • verdict, threat_type, source_type: Optional[str] = None
  • hours: int = 24, page: int = 1, page_size: int = 50
• Implementation logic:
  • Check redis_client availability (raise 503 if not)
  • Initialize cursor=0, all_events=[]
  • Use redis_client.scan with pattern="tenet:event:*", count=100
• For each key: get event data, parse JSON, apply filters (verdict, threat_type,
  source_type), append to all_events
  • Loop until cursor==0
  • Sort all_events by timestamp descending
  • Calculate pagination: start=(page-1)*page_size, end=start+page_size
  • Return EventListResponse with total_count, page, page_size, paginated events
• Add comment: Production should use PostgreSQL with proper indexing

Statistics and Analytics:

• Create StatsResponse Pydantic model: total_events (int), blocked_count (int),
  threats_by_type (dict), verdicts (dict), average_risk_score (float),
  time_range_hours (int)
• Implement @app.get("/v1/stats", response_model=StatsResponse) with
  parameters:
  • api_key: str = Depends(verify_api_key)
  • hours: int = 24
• Implementation logic:
  • Check redis_client availability
• Initialize stats dict with: total=0, blocked=0, threats={},
  verdicts={'benign':0,'suspicious':0,'malicious':0}, risk_scores=[]
  • Use redis_client.scan with same pattern as events listing
• For each event: increment total, check blocked and increment, aggregate
  threat_type counts, aggregate verdict counts, collect risk_scores
  • Calculate avg_risk: sum/len if risk_scores else 0.0
  • Return StatsResponse with all aggregated data, round avg_risk to 3 decimals

Alerts Management:

• Create AlertStatus str Enum: new, acknowledged, investigating, resolved,
  false_positive
• Create Alert Pydantic model: alert_id (str), event_id (str), severity (str),
  status (AlertStatus), threat_type (str), created_at (str), updated_at (str),
  notes (List[dict], default [])
• Create AlertUpdate model: status (Optional[AlertStatus])
• Create AlertNote model: text (str), author (str, default="analyst")
• Implement @app.get("/v1/alerts") with parameters: api_key, status
  (Optional[AlertStatus]), severity (Optional[str]), page (int=1), page_size
  (int=50)
  • Get alerts from redis_client.lrange("tenet:alerts", 0, -1)
  • Parse JSON, filter by status and severity
  • Paginate and return with total_count
• Implement @app.put("/v1/alerts/{alert_id}") with alert_id (str), update
  (AlertUpdate), api_key
  • Get alert from "tenet:alert:{alert_id}" key
  • Raise 404 if not found
  • Update status if provided, set updated_at to current ISO timestamp
  • Save back to Redis, return updated alert
• Implement @app.post("/v1/alerts/{alert_id}/notes") with alert_id (str), note
  (AlertNote), api_key
  • Get alert, raise 404 if not found
  • Append note to notes array with text, author, timestamp
  • Update updated_at field
  • Save to Redis, return status and alert

Real-Time Streaming:

• Import: WebSocket, WebSocketDisconnect from fastapi, Set from typing, asyncio
• Create module-level: active_connections: Set[WebSocket] = set()
• Implement @app.websocket("/v1/events/stream"):
  • Accept websocket connection
  • Add to active_connections
  • Try block:
    • Create pubsub = redis_client.pubsub()
    • Subscribe to "tenet:events:new"
    • While True loop:
      • Get message with timeout=1.0, ignore_subscribe_messages=True
      • If message type is 'message': send decoded data to websocket
      • Try to receive text from websocket with timeout=0.1 for filter updates
      • Log received data
  • Except WebSocketDisconnect: discard from active_connections
  • Finally: unsubscribe and discard from active_connections
• Implement async broadcast_event(event: dict):
  • Check if active_connections not empty
  • Convert event to JSON string
  • For each connection (copy of set):
    • Try to send_text, on exception discard connection
• Modify ingest_llm_event function:
• After queuing event to Redis, add: await
  redis_client.publish("tenet:events:new", json.dumps(response_data))
  ===============================================================================

Task: 4

Create comprehensive Python SDK and integration libraries for TENET AI. The goal
is to provide complete, production-ready client libraries and framework
integrations with full implementation code.

Python SDK Client:

• Create directory structure: sdk/python/tenet_ai/ with init.py,
  client.py, exceptions.py
• Create sdk/python/tenet_ai/exceptions.py:
  • Define TenetError(Exception) base class
• Define TenetBlockedError(TenetError) with init(message, risk_score,
  threat_type), store risk_score and threat_type as attributes
  • Define TenetAuthError(TenetError)
• Create sdk/python/tenet_ai/client.py:
  • Import httpx, Optional, Dict, Any from typing, exceptions module
  • Implement TenetClient class:
• init with api_key, base_url (default http://localhost:8000), timeout
  (10.0), blocking_mode (True)
  • Store attributes, create httpx.Client with timeout
  • _headers method returning X-API-Key and Content-Type headers
• submit_event method: build payload with
  prompt/source_type/source_id/model/system_prompt/metadata, POST to
  /v1/events/llm, handle 401 as TenetAuthError, raise TenetBlockedError if blocked
  and blocking_mode
• analyze method: POST to analyzer service :8100/v1/analyze with prompt and
  context
  • get_event method: GET /v1/events/{event_id}
  • get_stats method: GET /v1/stats with hours parameter
  • close method and context manager support (enter, exit)
• Implement AsyncTenetClient class with async versions of all methods using
  httpx.AsyncClient
• Create sdk/python/tenet_ai/__init__.py: import and export TenetClient,
  AsyncTenetClient, all exceptions, set version="0.1.0"
• Create sdk/python/pyproject.toml:
  • build-system: setuptools>=61.0
• project name="tenet-ai", version="0.1.0", description, readme,
  requires-python=">=3.9", license=MIT
  • dependencies: httpx>=0.25.0
• optional-dependencies: langchain=["langchain>=0.1.0"],
  fastapi=["fastapi>=0.100.0", "starlette>=0.27.0"], dev=["pytest",
  "pytest-asyncio"]
  • project.urls with Homepage, Documentation

LangChain Integration:

• Create sdk/python/tenet_ai/integrations/__init__.py as empty package
• Create sdk/python/tenet_ai/integrations/langchain.py:
• Import typing types, UUID, try/except import BaseCallbackHandler from
  langchain.callbacks.base with helpful error message
  • Import TenetClient, AsyncTenetClient, TenetBlockedError from parent modules
  • Implement TenetCallbackHandler(BaseCallbackHandler):
• init with api_key, base_url (default localhost:8000), blocking_mode
  (True), log_only (False), source_type ("langchain")
  • Create TenetClient with blocking_mode and not log_only
  • Store log_only and source_type
• Implement on_llm_start(serialized, prompts, run_id, parent_run_id, tags,
  metadata, kwargs):
  - Get model_name from serialized.get("name", "unknown")
• For each prompt: submit_event with prompt, source_type, run_id as source_id,
  model_name, metadata dict with tags/parent_run_id
  - If blocked and not log_only: raise TenetBlockedError
• Catch TenetBlockedError and re-raise, catch other exceptions and print warning
• Implement on_chain_start with similar logic: extract input_text from inputs
  (try "input", "query", "question" keys), submit_event with
  source_type="{source_type}-chain", metadata with chain_inputs keys
• Add usage example to documentation showing: create TenetCallbackHandler, use
  with LangChain LLM via callbacks parameter, demonstrate blocking behavior

FastAPI Middleware:

• Create sdk/python/tenet_ai/integrations/fastapi.py:
• Try/except import Request, Response from fastapi and BaseHTTPMiddleware from
  starlette with helpful error message
  • Import typing, json, TenetClient, TenetBlockedError
  • Implement TenetMiddleware(BaseHTTPMiddleware):
• init(app, api_key, base_url="http://localhost:8000", protected_paths=None,
  excluded_paths=None, blocking_mode=True, extract_prompt=None)
  • Call super().init(app)
  • Create TenetClient with blocking_mode=False (we handle it)
  • Set protected_paths default to ["/api/", "/v1/"]
• Set excluded_paths default to {"/health", "/metrics", "/docs", "/redoc",
  "/openapi.json"}
  • Store blocking_mode and extract_prompt or _default_extract_prompt
  • Implement _default_extract_prompt(body: dict) -> Optional[str]:
    • Check common keys: prompt, message, query, input, text, content
    • Check messages array for OpenAI format (role="user")
    • Return None if not found
• Implement _should_protect(path: str) -> bool: check if path in excluded_paths
  or starts with any protected_paths prefix
  • Implement async dispatch(request, call_next):
    • Skip if not _should_protect or method not in POST/PUT
    • Read and parse request body as JSON
    • Extract prompt using self.extract_prompt
    • Skip if no prompt found
    • Submit event to TENET with metadata (method, path, client IP)
• If blocked and blocking_mode: return Response with 403, JSON body with
  error/event_id/threat_type
  - Otherwise call next and add X-Tenet headers (Event-ID, Risk-Score)
  - Catch all exceptions and print warning, continue to call_next
• Add usage example showing: create FastAPI app, add TenetMiddleware with
  configuration, protected endpoint that gets automatic protection
  ===============================================================================

Task: 5

Implement comprehensive security hardening for TENET AI. The goal is to
eliminate hardcoded credentials, add startup configuration validation with
Pydantic, and support API key rotation.

Remove Hardcoded Credentials:

• Modify docker-compose.yml:
• Lines 27-31 (postgres section): Replace POSTGRES_PASSWORD: postgres with
  ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
  • Also use ${POSTGRES_USER:?POSTGRES_USER is required} syntax
  • Set POSTGRES_DB with default: ${POSTGRES_DB:-tenet}
• Lines 45-51 (minio section): Replace MINIO_ROOT_USER: minio and
  MINIO_ROOT_PASSWORD: minio123 with ${MINIO_ROOT_USER:?MINIO_ROOT_USER is
  required} and ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}
• Update .env.template:
  • Add DATABASE CONFIGURATION section header
  • Add comment about generating secure passwords: "openssl rand -base64 32"
• Add POSTGRES_USER=tenet, POSTGRES_PASSWORD= (empty, marked REQUIRED with min
  16 chars note), POSTGRES_DB=tenet
  • Add OBJECT STORAGE section header
• Add MINIO_ROOT_USER= (empty, marked REQUIRED with min 8 chars),
  MINIO_ROOT_PASSWORD= (empty, marked REQUIRED with min 16 chars)
  • Add API AUTHENTICATION section header
• Add generation command: python -c "import secrets;
  print(f'tenet-{secrets.token_urlsafe(32)}')"
  • Add API_KEY= (empty, marked REQUIRED)
• Create scripts/check_env.sh bash script:
  • Define color codes: RED, GREEN, NC
  • Initialize ERRORS=0 counter
• Define check_var function: takes var_name and min_length, checks if set and
  meets length, increments ERRORS and prints colored messages
• Define check_not_default function: takes var_name and default_value, checks if
  using default, increments ERRORS
  • Print header "Checking TENET AI environment configuration..."
• Call check_var for: POSTGRES_PASSWORD (16), MINIO_ROOT_USER (8),
  MINIO_ROOT_PASSWORD (16), API_KEY (20)
• Call check_not_default for: POSTGRES_PASSWORD ("postgres"),
  MINIO_ROOT_PASSWORD ("minio123"), API_KEY ("tenet-dev-key-change-in-production")
  • Print summary with error count, exit 1 if errors else print success message
• Add verification: chmod +x and run script expecting failure with defaults

Configuration Validation:

• Create services/common/config.py:
• Import os, sys, re, Optional from typing, BaseSettings, validator, SecretStr
  from pydantic
  • Define TenetConfig(BaseSettings) class:
• Fields: api_key (SecretStr), api_host (str, default "0.0.0.0"), api_port (int,
  default 8000), redis_host (str, default "localhost"), redis_port (int, default
  6379), rate_limit_rpm (int, default 1000), max_prompt_length (int, default
  10000), blocking_enabled (bool, default True), log_level (str, default "INFO"),
  log_format (str, default "json"), environment (str, default "development")
  • Config class: env_file=".env", env_file_encoding="utf-8"
• @validator("api_key"): get secret value, check length >= 20, check not in
  insecure_defaults list ["tenet-dev-key-change-in-production", "changeme",
  "test", "secret"], raise ValueError with clear messages
• @validator("environment"): if v=="production", check api_key doesn't contain
  "dev" or "test"
• @validator("log_level"): check against valid_levels ["DEBUG", "INFO",
  "WARNING", "ERROR", "CRITICAL"], return uppercase
• Define load_config() -> TenetConfig: try to instantiate TenetConfig, on
  exception print to stderr and sys.exit(1)
• Define log_config_summary(config, logger): call logger.info with extra dict
  containing all non-secret config fields (api_host, api_port, redis_host,
  redis_port, environment, log_level, blocking_enabled, rate_limit_rpm)
• Update services/ingest/app.py:
  • Import load_config, log_config_summary from common.config
  • Call config = load_config() at module level
• Add @app.on_event("startup") function that calls log_config_summary(config,
  logger)

API Key Rotation:

• Modify API key validation logic to split on comma and check against list of
  keys
• Define key metadata structure with name, created_at, expires_at fields
• Create /v1/admin/keys endpoint requiring admin authentication:
  • GET to list all keys (without revealing full key)
  • POST to create new key
  • DELETE to revoke key
• Implement expiration checking: compare expires_at to current time, log
  warnings
• Add audit logging: log all key validation attempts, key creation, key
  revocation with timestamps and metadata
  ===============================================================================

Task: 6

Create comprehensive documentation and contributor onboarding materials for
TENET AI. The goal is to enable new contributors with detailed GitHub issues and
provide clear API and integration documentation.

GitHub Issues Creation:

• Create detailed issue template for Task 1.1 (Coverage):
  • Title: "Add pytest coverage reporting to CI"
• Description section explaining the current state (tools installed but not
  configured)
• Acceptance Criteria checklist: CI runs pytest with coverage, upload to
  Codecov, fail below 70%, update CONTRIBUTING.md
• Implementation section with complete YAML code block showing pytest coverage
  step and Codecov upload step
  • Files to Modify: .github/workflows/ci.yml, CONTRIBUTING.md
  • Labels: good first issue, CI/CD, documentation
• Create similar detailed issues for:
  • Task 1.2: Enable linting (Black and Ruff configuration)
  • Task 1.3: Enable type checking (mypy setup)
  • Task 1.4: Security scanning (Bandit and Safety)
  • Task 2.1: Structured JSON logging (python-json-logger)
  • Task 2.2: Prometheus metrics endpoint
  • Task 5.1: Remove hardcoded credentials from docker-compose
• Each issue should include: clear problem statement, step-by-step acceptance
  criteria, implementation guidance with code snippets, file paths, helpful labels
• Link all issues back to this roadmap document for context

API Reference Documentation:

• Create docs/api-reference.md with comprehensive API documentation
• Enable FastAPI's built-in documentation:
  • Configure /docs endpoint for Swagger UI
  • Configure /redoc endpoint for ReDoc interface
• Document for each endpoint:
  • HTTP method and path
  • Authentication requirements (X-API-Key header)
  • Request schema with field descriptions and types
  • Response schema with examples
• Error codes: 200 (success), 401 (unauthorized), 403 (blocked), 404 (not
  found), 503 (service unavailable)
• Add example requests:
  • curl commands with headers and JSON bodies
  • Python examples using requests library and SDK
• Include operational details:
  • Rate limiting: requests per minute limits
  • Quota information
  • Pagination parameters and behavior

Integration Guide:

• Create docs/integrations.md covering all integration patterns
• SDK Installation section:
  • pip install commands
  • Optional dependencies [langchain] [fastapi]
• Direct API Integration:
  • Authentication setup
  • Example HTTP requests
  • Error handling patterns
• LangChain Integration:
  • Step-by-step setup of TenetCallbackHandler
  • Configuration options: blocking_mode, log_only
  • Code examples for LLM and chain integration
  • Callback lifecycle explanation
• FastAPI Middleware Integration:
  • Step-by-step setup of TenetMiddleware
  • Configuration: protected_paths, excluded_paths, custom extractors
  • Code examples for adding to FastAPI app
  • Request/response flow explanation
• Troubleshooting section:
  • Common issues: connection errors, authentication failures, timeout handling
  • Solutions and workarounds
  • Debug logging guidance
• Architecture diagrams:
  • Request flow showing where TENET intercepts
  • Integration points in application architecture
  • Data flow from client through TENET to LLM

</details>


<details>
<summary><b>💡 Iterate on the plan with:
<code>@coderabbitai &lt;feedback&gt;</code>
</b></summary>

Example Feedback

• @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
• @coderabbitai For design choice 1 go ahead with option 3 and replan.

</details>


---



💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit)!



<!-- <agentComponentType>plan-edits</agentComponentType> -->

<!-- <rawResChunk><planningResult>## Coding Plan

### Summary

Create a comprehensive feature roadmap for TENET AI that identifies missing capabilities, defines good first issues for contributors, and establishes a maintainer roadmap for critical platform improvements including CI/CD hardening, observability, dashboard APIs, and integration SDKs.

<details>
<summary><b>Design Choices</b></summary>

<details>
<summary><b>Design Choice 1: Scope Separation: Good First Issues vs Maintainer Roadmap</b></summary>



**Options Considered:**
1. Combine all features into a single unified roadmap
2. Clearly separate self-contained contributor tasks from larger architectural work

**Chosen Option:** 2

**Rationale:** Separate contributor issues (self-contained, well-defined tasks) from maintainer roadmap (architectural changes requiring deeper context) to enable effective parallel contribution.

</details>

<details>
<summary><b>Design Choice 2: Dashboard Implementation Approach</b></summary>



**Options Considered:**
1. Build backend APIs and frontend simultaneously
2. Build backend APIs first, then frontend as a separate phase

**Chosen Option:** 2

**Rationale:** Backend-first approach allows API contracts to be defined and tested before frontend work, enables contributors to work on frontend once APIs are stable.

</details>

<details>
<summary><b>Design Choice 3: Coverage Reporting Integration</b></summary>



**Options Considered:**
1. Upload as GitHub artifact (simple, no external dependencies)
2. Integrate with Codecov (industry standard, free for open source, PR comments)
3. Integrate with Coveralls (similar to Codecov, less popular)

**Chosen Option:** 2

**Rationale:** Codecov integration - provides PR comments showing coverage changes, badge support, and is free for open source. If external service is not desired, fall back to artifact upload.

</details>

</details>

<b>💡 User Tips</b>

Regenerate the plan with different choices with `@coderabbitai <feedback>`.


## Implementation Steps


### Phase 1: CI/CD Hardening (Good First Issues)

This phase enables quality gates that are currently missing. All required tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in requirements-dev.txt but not executed in CI. These are ideal contributor tasks as they're self-contained configuration changes.


<details>
<summary><b>Task 1: Add Coverage Reporting to CI</b></summary>

Enable pytest-cov execution in the CI workflow with a minimum coverage threshold to prevent regression.

**File to modify:** `.github/workflows/ci.yml`

**Replace the existing pytest step (around line 35-40) with:**
```yaml
      - name: Run unit tests with coverage
        run: |
          pytest tests/unit/ -v \
            --cov=services \
            --cov-report=xml \
            --cov-report=term-missing \
            --cov-fail-under=70

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          flags: unittests
          fail_ci_if_error: false
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Add to CONTRIBUTING.md under Testing section:

### Coverage Requirements

All contributions must maintain at least 70% test coverage. To check coverage locally:

\`\`\`bash
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70
\`\`\`

Coverage reports are automatically generated on pull requests via Codecov.

Verification:

# Run locally to verify it works before pushing
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70

Task 2: Enable Code Formatting and Linting Checks

Add Black formatting validation and Ruff linting as required CI checks to enforce code quality standards.

File to modify: .github/workflows/ci.yml

Add new job after the test job definition (add around line 45):

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install linting tools
        run: pip install black ruff
      
      - name: Check formatting with Black
        run: black --check --line-length 100 services/ tests/ scripts/
      
      - name: Lint with Ruff
        run: ruff check services/ tests/ scripts/

Create pyproject.toml in repository root (if not exists) or add these sections:

[tool.black]
line-length = 100
target-version = ['py311']
include = '\.pyi?$'
extend-exclude = '''
/(
  \.git
  | \.venv
  | __pycache__
  | models/trained
)/
'''

[tool.ruff]
line-length = 100
target-version = "py311"
select = ["E", "F", "W", "I", "UP", "B"]
ignore = ["E501"]  # Line length handled by Black

[tool.ruff.per-file-ignores]
"tests/*" = ["B011"]  # Allow assert in tests

Verification:

# Run locally before pushing
black --check --line-length 100 services/ tests/ scripts/
ruff check services/ tests/ scripts/

# Auto-fix issues if any
black --line-length 100 services/ tests/ scripts/
ruff check --fix services/ tests/ scripts/

Task 3: Enable Type Checking

Add mypy static type analysis to catch type errors before runtime.

File to modify: .github/workflows/ci.yml

Add to the lint job steps:

      - name: Install type checking tools
        run: pip install mypy types-redis
      
      - name: Type check with mypy
        run: mypy services/ --ignore-missing-imports --no-error-summary

Add to pyproject.toml:

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_ignores = true
ignore_missing_imports = true
exclude = [
    "tests/",
    "scripts/",
    "models/",
]

[[tool.mypy.overrides]]
module = "redis.*"
ignore_missing_imports = true

Verification:

pip install mypy types-redis
mypy services/ --ignore-missing-imports

Task 4: Integrate Security Scanning

Enable Bandit (SAST) and Safety (dependency vulnerabilities) in the CI pipeline.

File to modify: .github/workflows/ci.yml

Add new job:

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install security tools
        run: pip install bandit safety
      
      - name: Run Bandit security scan
        run: bandit -r services/ -ll -ii -x tests/
      
      - name: Check dependencies for vulnerabilities
        run: safety check -r requirements.txt --full-report
        continue-on-error: true  # Don't fail on advisory-only issues

Verification:

pip install bandit safety
bandit -r services/ -ll -ii -x tests/
safety check -r requirements.txt --full-report

🤖 Prompt for AI agents

Implement CI/CD quality gates for the TENET AI project. The goal is to enable
automated code quality, testing, and security checks with complete
copy-pasteable configurations.

**Coverage Reporting:**
- Modify `.github/workflows/ci.yml` around line 35-40 to replace the existing
pytest step
- Add pytest with coverage flags: `--cov=services --cov-report=xml
--cov-report=term-missing --cov-fail-under=70`
- Add Codecov upload step using `codecov/codecov-action@v4` with the
coverage.xml file
- Set flags to "unittests" and fail_ci_if_error to false
- Use CODECOV_TOKEN from GitHub secrets
- Update `CONTRIBUTING.md` Testing section with coverage requirements
documentation
- Add verification command to test locally

**Code Formatting and Linting:**
- Add new `lint` job to `.github/workflows/ci.yml` after the test job (around
line 45)
- Use ubuntu-latest runner with Python 3.11 and pip caching
- Install black and ruff
- Run Black check with `--check --line-length 100` on services/, tests/,
scripts/
- Run Ruff check on the same directories
- Create or update `pyproject.toml` with Black configuration: line-length 100,
target py311, exclude .git/.venv/__pycache__/models/trained
- Add Ruff configuration: line-length 100, select E/F/W/I/UP/B rules, ignore
E501
- Add per-file-ignores for tests to allow assert statements (B011)
- Include verification commands for local testing and auto-fixing

**Type Checking:**
- Add to the lint job steps in `.github/workflows/ci.yml`
- Install mypy and types-redis packages
- Run mypy on services/ with --ignore-missing-imports --no-error-summary flags
- Add `[tool.mypy]` section to `pyproject.toml`: python_version 3.11,
warn_return_any and warn_unused_ignores true, exclude tests/scripts/models/
- Add overrides section for redis.* modules to ignore missing imports
- Include verification command

**Security Scanning:**
- Add new `security` job to `.github/workflows/ci.yml`
- Use ubuntu-latest runner with Python 3.11 and pip caching
- Install bandit and safety packages
- Run Bandit with `-r services/ -ll -ii -x tests/` (medium+ severity, exclude
tests)
- Run Safety check on requirements.txt with --full-report and continue-on-error
true
- Include verification commands

Note: All tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in
requirements-dev.txt.

Phase 2: Observability Foundation (Good First Issues)

This phase addresses the critical gap of no production observability. The services currently use basic Python logging with no metrics or structured output.

Task 1: Implement Structured JSON Logging

Convert from basic text logging to structured JSON format with correlation IDs for request tracing.

Add to requirements.txt:

python-json-logger>=2.0.7

Create new file services/common/__init__.py:

# Empty file to make this a package

Create new file services/common/logging.py: (See full implementation in plan with TenetJsonFormatter class, setup_logging function, request ID management)

Update services/ingest/app.py - replace logging setup and add request ID middleware (complete code provided in plan)

Verification:

LOG_FORMAT=json python -c "
from services.common.logging import setup_logging
logger = setup_logging('test')
logger.info('Test message', extra={'custom_field': 'value'})
"
# Should output JSON with timestamp, level, service, etc.

Task 2: Add Prometheus Metrics Endpoint

Expose application metrics via a /metrics endpoint for Prometheus scraping.

Add to requirements.txt:

prometheus-client>=0.19.0

Create new file services/common/metrics.py: (Complete implementation provided with REQUEST_COUNT, REQUEST_LATENCY, THREATS_DETECTED, EVENTS_BLOCKED, QUEUE_DEPTH, MODEL_INFERENCE_LATENCY metrics)

Add to services/ingest/app.py: Import metrics, add /metrics endpoint, add metrics tracking middleware (complete code provided)

Verification:

# Start service and test
curl http://localhost:8000/metrics
# Should return Prometheus format metrics

Task 3: Enhanced Health Check Endpoints

Expand health checks to include dependency status and readiness/liveness separation.

Update services/ingest/app.py - replace existing health_check function with:

• GET /health/live for liveness checks (returns LivenessResponse with status, service, version)
• GET /health/ready for readiness checks (returns ReadinessResponse with status, service, version, checks dict)
• Legacy GET /health endpoint for backward compatibility
• Implement Redis connection checking with proper error handling
• Return HTTP 200 for ready/alive, 503 for not_ready states

Verification:

curl http://localhost:8000/health/live   # Should return 200
curl http://localhost:8000/health/ready  # Should return 200 or 503
curl http://localhost:8000/health        # Legacy endpoint

🤖 Prompt for AI agents

Implement observability infrastructure for TENET AI microservices. The goal is
to add complete structured logging, metrics collection, and enhanced health
checks with full implementation code.

**Structured JSON Logging:**
- Add `python-json-logger>=2.0.7` to requirements.txt
- Create `services/common/__init__.py` as empty package marker
- Create `services/common/logging.py` with full implementation:
  - Import logging, os, sys, uuid, contextvars, pythonjsonlogger
  - Create ContextVar for request_id with default empty string
  - Implement TenetJsonFormatter class extending jsonlogger.JsonFormatter
- In add_fields method: add service (from SERVICE_NAME env), version (from
SERVICE_VERSION env), request_id (from context var), level, timestamp
- Implement setup_logging(service_name) function: get LOG_LEVEL and LOG_FORMAT
from env, configure logger with StreamHandler, apply JSON or text formatter
based on format
  - Implement generate_request_id() returning 8-char UUID
  - Implement set_request_id(request_id) to update context var
- Update `services/ingest/app.py`:
- Replace lines 17-21 with: import sys, add parent to path, import from
common.logging
  - Call setup_logging('ingest') to get logger
- Add middleware function add_request_id that extracts/generates X-Request-ID
header, sets context var, calls next, adds header to response
- Include verification command using LOG_FORMAT=json

**Prometheus Metrics:**
- Add `prometheus-client>=0.19.0` to requirements.txt
- Create `services/common/metrics.py` with complete implementation:
- Import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST from
prometheus_client
  - Define REQUEST_COUNT Counter with labels: service, endpoint, method, status
- Define REQUEST_LATENCY Histogram with labels: service, endpoint and buckets:
.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0
  - Define THREATS_DETECTED Counter with labels: threat_type, verdict
  - Define EVENTS_BLOCKED Counter with label: threat_type
  - Define QUEUE_DEPTH Gauge with label: queue_name
- Define MODEL_INFERENCE_LATENCY Histogram with label: model_type and buckets:
.001, .005, .01, .025, .05, .1, .25, .5
- Implement metrics_endpoint() returning Response with generate_latest() content
- Update `services/ingest/app.py`:
  - Import metrics objects and metrics_endpoint function, import time
  - Add route @app.get("/metrics") calling metrics_endpoint()
- Add middleware track_metrics: capture start time, call next, calculate
duration, increment REQUEST_COUNT with all labels, observe REQUEST_LATENCY
- Include verification using curl to /metrics endpoint

**Enhanced Health Checks:**
- Update `services/ingest/app.py` replacing existing health_check function:
  - Create LivenessResponse Pydantic model with status, service, version fields
- Create ReadinessResponse Pydantic model with status, service, version, checks
dict fields
- Implement @app.get("/health/live") returning LivenessResponse with
status="alive", service="ingest", version="0.1.0"
  - Implement @app.get("/health/ready") with full readiness logic:
    - Initialize checks dict and overall_status="ready"
- Try redis_client.ping(), set checks["redis"]="connected" on success,
"not_configured" if no client, "error" with message on exception
    - Set overall_status to "degraded" or "not_ready" based on checks
    - Return 503 status code via JSONResponse if not ready, otherwise 200
- Implement legacy @app.get("/health") calling readiness() and formatting
response
- Include verification commands for all three endpoints

Phase 3: Dashboard Backend API Layer (Maintainer Roadmap)

This phase creates the API foundation required for the SOC dashboard. The current API only supports single event submission/retrieval with no listing, search, or analytics capabilities.

Task 1: Implement Events Listing and Search API

Create endpoints to list, filter, and search security events for dashboard consumption.

Add to services/ingest/app.py:

• Import Optional, List from typing, datetime and timedelta
• Create EventListResponse Pydantic model with total_count, page, page_size, events fields
• Implement @app.get("/v1/events", response_model=EventListResponse) endpoint
• Parameters: api_key (Depends), verdict/threat_type/source_type/hours/page/page_size (query params)
• Implementation:
  • Check Redis availability
  • Scan Redis keys matching "tenet:event:*" pattern
  • Load and parse events, apply filters (verdict, threat_type, source_type)
  • Sort by timestamp descending
  • Paginate results using page and page_size
  • Return EventListResponse with counts and paginated events
• Note: Production should migrate to PostgreSQL for better querying

Task 2: Implement Statistics and Analytics API

Create aggregation endpoints for dashboard charts and metrics visualization.

Add to services/ingest/app.py:

• Create StatsResponse Pydantic model with total_events, blocked_count, threats_by_type dict, verdicts dict, average_risk_score, time_range_hours
• Implement @app.get("/v1/stats", response_model=StatsResponse) endpoint
• Parameters: api_key (Depends), hours (query param, default 24)
• Implementation:
  • Check Redis availability
  • Scan all tenet:event:* keys
  • Aggregate statistics: total events, blocked count, threats by type, verdicts breakdown (benign/suspicious/malicious), risk scores
  • Calculate average risk score from collected scores
  • Return StatsResponse with all aggregated data rounded appropriately

Task 3: Implement Alerts Management API

Create CRUD endpoints for security alerts with analyst workflow support.

Add to services/ingest/app.py:

• Create AlertStatus Enum with values: new, acknowledged, investigating, resolved, false_positive
• Create Alert Pydantic model with alert_id, event_id, severity, status, threat_type, created_at, updated_at, notes list
• Create AlertUpdate model with optional status field
• Create AlertNote model with text and author (default "analyst")
• Implement GET /v1/alerts for listing with filtering by status, severity, pagination
• Implement PUT /v1/alerts/{alert_id} for updating alert status
• Implement POST /v1/alerts/{alert_id}/notes for adding analyst notes
• Use Redis for storage (tenet:alert:{id} keys and tenet:alerts list)
• Include timestamp updates on modifications

Task 4: Add Real-Time Event Streaming

Implement WebSocket or Server-Sent Events for live dashboard updates.

Add to services/ingest/app.py:

• Import WebSocket, WebSocketDisconnect, Set, asyncio
• Create module-level active_connections: Set[WebSocket]
• Implement @app.websocket("/v1/events/stream") endpoint:
  • Accept WebSocket connection and add to active_connections
  • Subscribe to Redis pubsub channel "tenet:events:new"
  • Loop: get messages from pubsub, send to WebSocket
  • Handle client messages for filter updates
  • Handle WebSocketDisconnect, cleanup subscriptions
• Implement broadcast_event(event) function to send to all connected clients with error handling
• Modify ingest_llm_event function to publish events to "tenet:events:new" channel after queuing

🤖 Prompt for AI agents

Build the comprehensive backend API layer for the SOC dashboard. The goal is to
create complete endpoints for event management, analytics, alerts, and real-time
streaming with full implementation code.

**Events Listing and Search:**
- Add imports to `services/ingest/app.py`: Optional, List from typing, datetime,
timedelta
- Create EventListResponse Pydantic model: total_count (int), page (int),
page_size (int), events (List[dict])
- Implement `@app.get("/v1/events", response_model=EventListResponse)` with
parameters:
  - api_key: str = Depends(verify_api_key)
  - verdict, threat_type, source_type: Optional[str] = None
  - hours: int = 24, page: int = 1, page_size: int = 50
- Implementation logic:
  - Check redis_client availability (raise 503 if not)
  - Initialize cursor=0, all_events=[]
  - Use redis_client.scan with pattern="tenet:event:*", count=100
- For each key: get event data, parse JSON, apply filters (verdict, threat_type,
source_type), append to all_events
  - Loop until cursor==0
  - Sort all_events by timestamp descending
  - Calculate pagination: start=(page-1)*page_size, end=start+page_size
  - Return EventListResponse with total_count, page, page_size, paginated events
- Add comment: Production should use PostgreSQL with proper indexing

**Statistics and Analytics:**
- Create StatsResponse Pydantic model: total_events (int), blocked_count (int),
threats_by_type (dict), verdicts (dict), average_risk_score (float),
time_range_hours (int)
- Implement `@app.get("/v1/stats", response_model=StatsResponse)` with
parameters:
  - api_key: str = Depends(verify_api_key)
  - hours: int = 24
- Implementation logic:
  - Check redis_client availability
- Initialize stats dict with: total=0, blocked=0, threats={},
verdicts={'benign':0,'suspicious':0,'malicious':0}, risk_scores=[]
  - Use redis_client.scan with same pattern as events listing
- For each event: increment total, check blocked and increment, aggregate
threat_type counts, aggregate verdict counts, collect risk_scores
  - Calculate avg_risk: sum/len if risk_scores else 0.0
  - Return StatsResponse with all aggregated data, round avg_risk to 3 decimals

**Alerts Management:**
- Create AlertStatus str Enum: new, acknowledged, investigating, resolved,
false_positive
- Create Alert Pydantic model: alert_id (str), event_id (str), severity (str),
status (AlertStatus), threat_type (str), created_at (str), updated_at (str),
notes (List[dict], default [])
- Create AlertUpdate model: status (Optional[AlertStatus])
- Create AlertNote model: text (str), author (str, default="analyst")
- Implement `@app.get("/v1/alerts")` with parameters: api_key, status
(Optional[AlertStatus]), severity (Optional[str]), page (int=1), page_size
(int=50)
  - Get alerts from redis_client.lrange("tenet:alerts", 0, -1)
  - Parse JSON, filter by status and severity
  - Paginate and return with total_count
- Implement `@app.put("/v1/alerts/{alert_id}")` with alert_id (str), update
(AlertUpdate), api_key
  - Get alert from "tenet:alert:{alert_id}" key
  - Raise 404 if not found
  - Update status if provided, set updated_at to current ISO timestamp
  - Save back to Redis, return updated alert
- Implement `@app.post("/v1/alerts/{alert_id}/notes")` with alert_id (str), note
(AlertNote), api_key
  - Get alert, raise 404 if not found
  - Append note to notes array with text, author, timestamp
  - Update updated_at field
  - Save to Redis, return status and alert

**Real-Time Streaming:**
- Import: WebSocket, WebSocketDisconnect from fastapi, Set from typing, asyncio
- Create module-level: active_connections: Set[WebSocket] = set()
- Implement `@app.websocket("/v1/events/stream")`:
  - Accept websocket connection
  - Add to active_connections
  - Try block:
    - Create pubsub = redis_client.pubsub()
    - Subscribe to "tenet:events:new"
    - While True loop:
      - Get message with timeout=1.0, ignore_subscribe_messages=True
      - If message type is 'message': send decoded data to websocket
      - Try to receive text from websocket with timeout=0.1 for filter updates
      - Log received data
  - Except WebSocketDisconnect: discard from active_connections
  - Finally: unsubscribe and discard from active_connections
- Implement async broadcast_event(event: dict):
  - Check if active_connections not empty
  - Convert event to JSON string
  - For each connection (copy of set):
    - Try to send_text, on exception discard connection
- Modify ingest_llm_event function:
- After queuing event to Redis, add: await
redis_client.publish("tenet:events:new", json.dumps(response_data))

Phase 4: Python SDK and Integration Library (Mixed - Some Good First Issues)

This phase creates client libraries to simplify TENET AI integration into applications.

Task 1: Create Python SDK Client Library

Build a Python package that wraps TENET AI's REST API for easy integration.

Create directory structure:

sdk/
└── python/
    ├── pyproject.toml
    ├── README.md
    └── tenet_ai/
        ├── __init__.py
        ├── client.py
        └── exceptions.py

Files to create:

1. sdk/python/tenet_ai/__init__.py - Export TenetClient, AsyncTenetClient, exceptions
2. sdk/python/tenet_ai/exceptions.py - Define TenetError, TenetBlockedError (with risk_score, threat_type), TenetAuthError
3. sdk/python/tenet_ai/client.py - Complete implementation of TenetClient and AsyncTenetClient (full code provided in plan)
4. sdk/python/pyproject.toml - Package configuration with dependencies: httpx>=0.25.0, optional dependencies for langchain and fastapi integrations

Task 2: Create LangChain Integration Plugin

Build a native LangChain callback handler as a proper package rather than example code.

Create files:

1. sdk/python/tenet_ai/integrations/__init__.py - Empty package marker
2. sdk/python/tenet_ai/integrations/langchain.py - Complete TenetCallbackHandler implementation (full code provided in plan)

Implementation includes:

• TenetCallbackHandler extending BaseCallbackHandler
• Initialization with api_key, base_url, blocking_mode, log_only, source_type
• on_llm_start hook for LLM interception
• on_chain_start hook for chain interception
• Exception handling with proper error propagation
• Metadata tracking for run IDs, tags, parent runs

Usage example provided for documentation

Task 3: Create FastAPI Middleware Package

Build a drop-in middleware that automatically protects FastAPI applications.

Create file:

sdk/python/tenet_ai/integrations/fastapi.py - Complete TenetMiddleware implementation (full code provided in plan)

Implementation includes:

• TenetMiddleware extending BaseHTTPMiddleware
• Initialization with api_key, base_url, protected_paths, excluded_paths, blocking_mode, custom extract_prompt function
• Default prompt extraction from common fields (prompt, message, query, input, text, content) and OpenAI messages format
• Path protection logic with exclusions for health/metrics/docs endpoints
• Request body parsing and prompt extraction
• Event submission to TENET AI with metadata
• Blocking response (403) with event details when threats detected
• X-Tenet headers added to responses (Event-ID, Risk-Score)
• Graceful degradation if TENET unavailable

Usage example provided for documentation

🤖 Prompt for AI agents

Create comprehensive Python SDK and integration libraries for TENET AI. The goal
is to provide complete, production-ready client libraries and framework
integrations with full implementation code.

**Python SDK Client:**
- Create directory structure: `sdk/python/tenet_ai/` with __init__.py,
client.py, exceptions.py
- Create `sdk/python/tenet_ai/exceptions.py`:
  - Define TenetError(Exception) base class
- Define TenetBlockedError(TenetError) with __init__(message, risk_score,
threat_type), store risk_score and threat_type as attributes
  - Define TenetAuthError(TenetError)
- Create `sdk/python/tenet_ai/client.py`:
  - Import httpx, Optional, Dict, Any from typing, exceptions module
  - Implement TenetClient class:
- __init__ with api_key, base_url (default http://localhost:8000), timeout
(10.0), blocking_mode (True)
    - Store attributes, create httpx.Client with timeout
    - _headers method returning X-API-Key and Content-Type headers
- submit_event method: build payload with
prompt/source_type/source_id/model/system_prompt/metadata, POST to
/v1/events/llm, handle 401 as TenetAuthError, raise TenetBlockedError if blocked
and blocking_mode
- analyze method: POST to analyzer service :8100/v1/analyze with prompt and
context
    - get_event method: GET /v1/events/{event_id}
    - get_stats method: GET /v1/stats with hours parameter
    - close method and context manager support (__enter__, __exit__)
- Implement AsyncTenetClient class with async versions of all methods using
httpx.AsyncClient
- Create `sdk/python/tenet_ai/__init__.py`: import and export TenetClient,
AsyncTenetClient, all exceptions, set __version__="0.1.0"
- Create `sdk/python/pyproject.toml`:
  - build-system: setuptools>=61.0
- project name="tenet-ai", version="0.1.0", description, readme,
requires-python=">=3.9", license=MIT
  - dependencies: httpx>=0.25.0
- optional-dependencies: langchain=["langchain>=0.1.0"],
fastapi=["fastapi>=0.100.0", "starlette>=0.27.0"], dev=["pytest",
"pytest-asyncio"]
  - project.urls with Homepage, Documentation

**LangChain Integration:**
- Create `sdk/python/tenet_ai/integrations/__init__.py` as empty package
- Create `sdk/python/tenet_ai/integrations/langchain.py`:
- Import typing types, UUID, try/except import BaseCallbackHandler from
langchain.callbacks.base with helpful error message
  - Import TenetClient, AsyncTenetClient, TenetBlockedError from parent modules
  - Implement TenetCallbackHandler(BaseCallbackHandler):
- __init__ with api_key, base_url (default localhost:8000), blocking_mode
(True), log_only (False), source_type ("langchain")
    - Create TenetClient with blocking_mode and not log_only
    - Store log_only and source_type
- Implement on_llm_start(serialized, prompts, run_id, parent_run_id, tags,
metadata, kwargs):
      - Get model_name from serialized.get("name", "unknown")
- For each prompt: submit_event with prompt, source_type, run_id as source_id,
model_name, metadata dict with tags/parent_run_id
      - If blocked and not log_only: raise TenetBlockedError
- Catch TenetBlockedError and re-raise, catch other exceptions and print warning
- Implement on_chain_start with similar logic: extract input_text from inputs
(try "input", "query", "question" keys), submit_event with
source_type="{source_type}-chain", metadata with chain_inputs keys
- Add usage example to documentation showing: create TenetCallbackHandler, use
with LangChain LLM via callbacks parameter, demonstrate blocking behavior

**FastAPI Middleware:**
- Create `sdk/python/tenet_ai/integrations/fastapi.py`:
- Try/except import Request, Response from fastapi and BaseHTTPMiddleware from
starlette with helpful error message
  - Import typing, json, TenetClient, TenetBlockedError
  - Implement TenetMiddleware(BaseHTTPMiddleware):
- __init__(app, api_key, base_url="http://localhost:8000", protected_paths=None,
excluded_paths=None, blocking_mode=True, extract_prompt=None)
    - Call super().__init__(app)
    - Create TenetClient with blocking_mode=False (we handle it)
    - Set protected_paths default to ["/api/", "/v1/"]
- Set excluded_paths default to {"/health", "/metrics", "/docs", "/redoc",
"/openapi.json"}
    - Store blocking_mode and extract_prompt or _default_extract_prompt
    - Implement _default_extract_prompt(body: dict) -> Optional[str]:
      - Check common keys: prompt, message, query, input, text, content
      - Check messages array for OpenAI format (role="user")
      - Return None if not found
- Implement _should_protect(path: str) -> bool: check if path in excluded_paths
or starts with any protected_paths prefix
    - Implement async dispatch(request, call_next):
      - Skip if not _should_protect or method not in POST/PUT
      - Read and parse request body as JSON
      - Extract prompt using self.extract_prompt
      - Skip if no prompt found
      - Submit event to TENET with metadata (method, path, client IP)
- If blocked and blocking_mode: return Response with 403, JSON body with
error/event_id/threat_type
      - Otherwise call next and add X-Tenet headers (Event-ID, Risk-Score)
      - Catch all exceptions and print warning, continue to call_next
- Add usage example showing: create FastAPI app, add TenetMiddleware with
configuration, protected endpoint that gets automatic protection

Phase 5: Security Hardening (Good First Issues + Maintainer)

This phase addresses critical security gaps identified in the codebase, particularly hardcoded credentials and configuration validation.

Task 1: Remove Hardcoded Credentials from Docker Compose

Eliminate default passwords and enforce environment variable usage for all secrets.

Modify docker-compose.yml:

Replace lines 27-31 (PostgreSQL section):

  postgres:
    image: postgres:15-alpine
    container_name: tenet-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER is required}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
      POSTGRES_DB: ${POSTGRES_DB:-tenet}

Replace lines 45-51 (MinIO section):

  minio:
    image: minio/minio:latest
    container_name: tenet-minio
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:?MINIO_ROOT_USER is required}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}

Update .env.template - add comprehensive configuration sections with security warnings

Create scripts/check_env.sh - bash script to verify environment variables (complete implementation provided in plan)

Verification:

chmod +x scripts/check_env.sh
./scripts/check_env.sh  # Should fail if using defaults

Task 2: Implement Configuration Validation

Add startup checks to validate required configuration and fail fast on misconfiguration.

Create services/common/config.py:

• Complete TenetConfig Pydantic BaseSettings class implementation (full code provided in plan)
• Fields: api_key (SecretStr), api_host, api_port, redis_host, redis_port, rate_limit_rpm, max_prompt_length, blocking_enabled, log_level, log_format, environment
• Validators for: api_key (minimum 20 chars, check insecure defaults), environment (production checks), log_level (valid values)
• load_config() function with error handling and sys.exit on failure
• log_config_summary() function to log configuration without exposing secrets

Update service startup:

• Import load_config and log_config_summary in services/ingest/app.py
• Call load_config() at module level
• Add startup event handler calling log_config_summary()

Task 3: Add API Key Rotation Support

Implement support for multiple API keys and graceful rotation without downtime.

• Modify API key validation to accept comma-separated list of valid keys
• Add key metadata support (name, created_at, expires_at)
• Add /v1/admin/keys endpoint (admin-authenticated) for key management
• Implement key expiration checking and warnings
• Add audit logging for key usage and rotation events

🤖 Prompt for AI agents

Implement comprehensive security hardening for TENET AI. The goal is to
eliminate hardcoded credentials, add startup configuration validation with
Pydantic, and support API key rotation.

**Remove Hardcoded Credentials:**
- Modify `docker-compose.yml`:
- Lines 27-31 (postgres section): Replace POSTGRES_PASSWORD: postgres with
${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
  - Also use ${POSTGRES_USER:?POSTGRES_USER is required} syntax
  - Set POSTGRES_DB with default: ${POSTGRES_DB:-tenet}
- Lines 45-51 (minio section): Replace MINIO_ROOT_USER: minio and
MINIO_ROOT_PASSWORD: minio123 with ${MINIO_ROOT_USER:?MINIO_ROOT_USER is
required} and ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}
- Update `.env.template`:
  - Add DATABASE CONFIGURATION section header
  - Add comment about generating secure passwords: "openssl rand -base64 32"
- Add POSTGRES_USER=tenet, POSTGRES_PASSWORD= (empty, marked REQUIRED with min
16 chars note), POSTGRES_DB=tenet
  - Add OBJECT STORAGE section header
- Add MINIO_ROOT_USER= (empty, marked REQUIRED with min 8 chars),
MINIO_ROOT_PASSWORD= (empty, marked REQUIRED with min 16 chars)
  - Add API AUTHENTICATION section header
- Add generation command: python -c "import secrets;
print(f'tenet-{secrets.token_urlsafe(32)}')"
  - Add API_KEY= (empty, marked REQUIRED)
- Create `scripts/check_env.sh` bash script:
  - Define color codes: RED, GREEN, NC
  - Initialize ERRORS=0 counter
- Define check_var function: takes var_name and min_length, checks if set and
meets length, increments ERRORS and prints colored messages
- Define check_not_default function: takes var_name and default_value, checks if
using default, increments ERRORS
  - Print header "Checking TENET AI environment configuration..."
- Call check_var for: POSTGRES_PASSWORD (16), MINIO_ROOT_USER (8),
MINIO_ROOT_PASSWORD (16), API_KEY (20)
- Call check_not_default for: POSTGRES_PASSWORD ("postgres"),
MINIO_ROOT_PASSWORD ("minio123"), API_KEY ("tenet-dev-key-change-in-production")
  - Print summary with error count, exit 1 if errors else print success message
- Add verification: chmod +x and run script expecting failure with defaults

**Configuration Validation:**
- Create `services/common/config.py`:
- Import os, sys, re, Optional from typing, BaseSettings, validator, SecretStr
from pydantic
  - Define TenetConfig(BaseSettings) class:
- Fields: api_key (SecretStr), api_host (str, default "0.0.0.0"), api_port (int,
default 8000), redis_host (str, default "localhost"), redis_port (int, default
6379), rate_limit_rpm (int, default 1000), max_prompt_length (int, default
10000), blocking_enabled (bool, default True), log_level (str, default "INFO"),
log_format (str, default "json"), environment (str, default "development")
    - Config class: env_file=".env", env_file_encoding="utf-8"
- @validator("api_key"): get secret value, check length >= 20, check not in
insecure_defaults list ["tenet-dev-key-change-in-production", "changeme",
"test", "secret"], raise ValueError with clear messages
- @validator("environment"): if v=="production", check api_key doesn't contain
"dev" or "test"
- @validator("log_level"): check against valid_levels ["DEBUG", "INFO",
"WARNING", "ERROR", "CRITICAL"], return uppercase
- Define load_config() -> TenetConfig: try to instantiate TenetConfig, on
exception print to stderr and sys.exit(1)
- Define log_config_summary(config, logger): call logger.info with extra dict
containing all non-secret config fields (api_host, api_port, redis_host,
redis_port, environment, log_level, blocking_enabled, rate_limit_rpm)
- Update `services/ingest/app.py`:
  - Import load_config, log_config_summary from common.config
  - Call config = load_config() at module level
- Add @app.on_event("startup") function that calls log_config_summary(config,
logger)

**API Key Rotation:**
- Modify API key validation logic to split on comma and check against list of
keys
- Define key metadata structure with name, created_at, expires_at fields
- Create `/v1/admin/keys` endpoint requiring admin authentication:
  - GET to list all keys (without revealing full key)
  - POST to create new key
  - DELETE to revoke key
- Implement expiration checking: compare expires_at to current time, log
warnings
- Add audit logging: log all key validation attempts, key creation, key
revocation with timestamps and metadata

Phase 6: Documentation and Contributor Onboarding

This phase ensures new contributors can effectively work on the codebase and that features are properly documented.

Task 1: Create Good First Issues in GitHub

Document and create GitHub issues for contributor-friendly tasks identified in this plan.

Issue Template for Task 1.1 (Coverage):

## Title: Add pytest coverage reporting to CI

### Description
Enable code coverage measurement and reporting in our GitHub Actions CI workflow. The tools are already installed (`pytest-cov` in requirements-dev.txt) but not configured.

### Acceptance Criteria
- [ ] CI runs pytest with coverage flags
- [ ] Coverage report uploaded to Codecov
- [ ] Build fails if coverage drops below 70%
- [ ] CONTRIBUTING.md updated with coverage requirements

### Implementation
Add these changes to `.github/workflows/ci.yml`:
```yaml
- name: Run unit tests with coverage
  run: |
    pytest tests/unit/ -v \
      --cov=services \
      --cov-report=xml \
      --cov-fail-under=70

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    file: ./coverage.xml

Files to Modify

• .github/workflows/ci.yml
• CONTRIBUTING.md

Labels

good first issue, CI/CD, documentation

**Create similar issues for:**
- Task 1.2: Enable linting in CI
- Task 1.3: Enable type checking in CI
- Task 1.4: Add security scanning to CI
- Task 2.1: Implement structured JSON logging
- Task 2.2: Add Prometheus metrics endpoint
- Task 5.1: Remove hardcoded credentials

</details>


<details>
<summary><b>Task 2: Create API Reference Documentation</b></summary>

Document all existing and new API endpoints for developer consumption.

- Create `docs/api-reference.md` with OpenAPI/Swagger documentation
- Enable FastAPI's automatic docs at `/docs` and `/redoc` endpoints
- Document authentication requirements, request/response schemas, error codes
- Add example requests using curl and Python
- Include rate limiting and quota information

</details>


<details>
<summary><b>Task 3: Create Integration Guide</b></summary>

Write comprehensive documentation for integrating TENET AI into applications.

- Create `docs/integrations.md` covering all integration patterns
- Document SDK installation and usage
- Provide step-by-step guides for LangChain, FastAPI middleware, and direct API integration
- Include troubleshooting section for common issues
- Add architecture diagrams showing integration points

</details>


<details>
<summary><b>🤖 Prompt for AI agents</b></summary>

Create comprehensive documentation and contributor onboarding materials for
TENET AI. The goal is to enable new contributors with detailed GitHub issues and
provide clear API and integration documentation.

GitHub Issues Creation:

• Create detailed issue template for Task 1.1 (Coverage):
  • Title: "Add pytest coverage reporting to CI"
• Description section explaining the current state (tools installed but not
  configured)
• Acceptance Criteria checklist: CI runs pytest with coverage, upload to
  Codecov, fail below 70%, update CONTRIBUTING.md
• Implementation section with complete YAML code block showing pytest coverage
  step and Codecov upload step
  • Files to Modify: .github/workflows/ci.yml, CONTRIBUTING.md
  • Labels: good first issue, CI/CD, documentation
• Create similar detailed issues for:
  • Task 1.2: Enable linting (Black and Ruff configuration)
  • Task 1.3: Enable type checking (mypy setup)
  • Task 1.4: Security scanning (Bandit and Safety)
  • Task 2.1: Structured JSON logging (python-json-logger)
  • Task 2.2: Prometheus metrics endpoint
  • Task 5.1: Remove hardcoded credentials from docker-compose
• Each issue should include: clear problem statement, step-by-step acceptance
  criteria, implementation guidance with code snippets, file paths, helpful labels
• Link all issues back to this roadmap document for context

API Reference Documentation:

• Create docs/api-reference.md with comprehensive API documentation
• Enable FastAPI's built-in documentation:
  • Configure /docs endpoint for Swagger UI
  • Configure /redoc endpoint for ReDoc interface
• Document for each endpoint:
  • HTTP method and path
  • Authentication requirements (X-API-Key header)
  • Request schema with field descriptions and types
  • Response schema with examples
• Error codes: 200 (success), 401 (unauthorized), 403 (blocked), 404 (not
  found), 503 (service unavailable)
• Add example requests:
  • curl commands with headers and JSON bodies
  • Python examples using requests library and SDK
• Include operational details:
  • Rate limiting: requests per minute limits
  • Quota information
  • Pagination parameters and behavior

Integration Guide:

• Create docs/integrations.md covering all integration patterns
• SDK Installation section:
  • pip install commands
  • Optional dependencies [langchain] [fastapi]
• Direct API Integration:
  • Authentication setup
  • Example HTTP requests
  • Error handling patterns
• LangChain Integration:
  • Step-by-step setup of TenetCallbackHandler
  • Configuration options: blocking_mode, log_only
  • Code examples for LLM and chain integration
  • Callback lifecycle explanation
• FastAPI Middleware Integration:
  • Step-by-step setup of TenetMiddleware
  • Configuration: protected_paths, excluded_paths, custom extractors
  • Code examples for adding to FastAPI app
  • Request/response flow explanation
• Troubleshooting section:
  • Common issues: connection errors, authentication failures, timeout handling
  • Solutions and workarounds
  • Debug logging guidance
• Architecture diagrams:
  • Request flow showing where TENET intercepts
  • Integration points in application architecture
  • Data flow from client through TENET to LLM

</details>

<details>
<summary><b>Research</b></summary>

TENET AI is a defensive security middleware for LLM applications, acting as a firewall/IDS that intercepts prompts, detects adversarial attacks (prompt injection, jailbreaks, data extraction), and provides security visibility.

The system consists of two FastAPI microservices:
- **Ingest service** (port 8000) for fast heuristic checks and event queuing
- **Analyzer service** (port 8100) for ML-based threat detection using scikit-learn models

Events flow through Redis queues, with PostgreSQL and MinIO available but not yet utilized.

**Current State:**
- Solid unit tests but gaps in CI/CD automation (linting, coverage, security scanning tools are installed but not executed)
- No observability infrastructure (metrics, structured logging)
- No SOC dashboard implementation (only documented as planned)
- No SDK/client libraries for framework integrations

</details>


---



### 🚀 Next Steps


<details>
<summary><b>🤖 All AI agent prompts combined</b></summary>

Task: 1

Implement CI/CD quality gates for the TENET AI project. The goal is to enable
automated code quality, testing, and security checks with complete
copy-pasteable configurations.

Coverage Reporting:

• Modify .github/workflows/ci.yml around line 35-40 to replace the existing
  pytest step
• Add pytest with coverage flags: --cov=services --cov-report=xml --cov-report=term-missing --cov-fail-under=70
• Add Codecov upload step using codecov/codecov-action@v4 with the
  coverage.xml file
• Set flags to "unittests" and fail_ci_if_error to false
• Use CODECOV_TOKEN from GitHub secrets
• Update CONTRIBUTING.md Testing section with coverage requirements
  documentation
• Add verification command to test locally

Code Formatting and Linting:

• Add new lint job to .github/workflows/ci.yml after the test job (around
  line 45)
• Use ubuntu-latest runner with Python 3.11 and pip caching
• Install black and ruff
• Run Black check with --check --line-length 100 on services/, tests/,
  scripts/
• Run Ruff check on the same directories
• Create or update pyproject.toml with Black configuration: line-length 100,
  target py311, exclude .git/.venv/pycache/models/trained
• Add Ruff configuration: line-length 100, select E/F/W/I/UP/B rules, ignore
  E501
• Add per-file-ignores for tests to allow assert statements (B011)
• Include verification commands for local testing and auto-fixing

Type Checking:

• Add to the lint job steps in .github/workflows/ci.yml
• Install mypy and types-redis packages
• Run mypy on services/ with --ignore-missing-imports --no-error-summary flags
• Add [tool.mypy] section to pyproject.toml: python_version 3.11,
  warn_return_any and warn_unused_ignores true, exclude tests/scripts/models/
• Add overrides section for redis.* modules to ignore missing imports
• Include verification command

Security Scanning:

• Add new security job to .github/workflows/ci.yml
• Use ubuntu-latest runner with Python 3.11 and pip caching
• Install bandit and safety packages
• Run Bandit with -r services/ -ll -ii -x tests/ (medium+ severity, exclude
  tests)
• Run Safety check on requirements.txt with --full-report and continue-on-error
  true
• Include verification commands

Note: All tools (pytest-cov, black, ruff, mypy, bandit, safety) are already in
requirements-dev.txt.

Task: 2

Implement observability infrastructure for TENET AI microservices. The goal is
to add complete structured logging, metrics collection, and enhanced health
checks with full implementation code.

Structured JSON Logging:

• Add python-json-logger>=2.0.7 to requirements.txt
• Create services/common/__init__.py as empty package marker
• Create services/common/logging.py with full implementation:
  • Import logging, os, sys, uuid, contextvars, pythonjsonlogger
  • Create ContextVar for request_id with default empty string
  • Implement TenetJsonFormatter class extending jsonlogger.JsonFormatter
• In add_fields method: add service (from SERVICE_NAME env), version (from
  SERVICE_VERSION env), request_id (from context var), level, timestamp
• Implement setup_logging(service_name) function: get LOG_LEVEL and LOG_FORMAT
  from env, configure logger with StreamHandler, apply JSON or text formatter
  based on format
  • Implement generate_request_id() returning 8-char UUID
  • Implement set_request_id(request_id) to update context var
• Update services/ingest/app.py:
• Replace lines 17-21 with: import sys, add parent to path, import from
  common.logging
  • Call setup_logging('ingest') to get logger
• Add middleware function add_request_id that extracts/generates X-Request-ID
  header, sets context var, calls next, adds header to response
• Include verification command using LOG_FORMAT=json

Prometheus Metrics:

• Add prometheus-client>=0.19.0 to requirements.txt
• Create services/common/metrics.py with complete implementation:
• Import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST from
  prometheus_client
  • Define REQUEST_COUNT Counter with labels: service, endpoint, method, status
• Define REQUEST_LATENCY Histogram with labels: service, endpoint and buckets:
  .005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0
  • Define THREATS_DETECTED Counter with labels: threat_type, verdict
  • Define EVENTS_BLOCKED Counter with label: threat_type
  • Define QUEUE_DEPTH Gauge with label: queue_name
• Define MODEL_INFERENCE_LATENCY Histogram with label: model_type and buckets:
  .001, .005, .01, .025, .05, .1, .25, .5
• Implement metrics_endpoint() returning Response with generate_latest() content
• Update services/ingest/app.py:
  • Import metrics objects and metrics_endpoint function, import time
  • Add route @app.get("/metrics") calling metrics_endpoint()
• Add middleware track_metrics: capture start time, call next, calculate
  duration, increment REQUEST_COUNT with all labels, observe REQUEST_LATENCY
• Include verification using curl to /metrics endpoint

Enhanced Health Checks:

• Update services/ingest/app.py replacing existing health_check function:
  • Create LivenessResponse Pydantic model with status, service, version fields
• Create ReadinessResponse Pydantic model with status, service, version, checks
  dict fields
• Implement @app.get("/health/live") returning LivenessResponse with
  status="alive", service="ingest", version="0.1.0"
  • Implement @app.get("/health/ready") with full readiness logic:
    • Initialize checks dict and overall_status="ready"
• Try redis_client.ping(), set checks["redis"]="connected" on success,
  "not_configured" if no client, "error" with message on exception
  • Set overall_status to "degraded" or "not_ready" based on checks
  • Return 503 status code via JSONResponse if not ready, otherwise 200
• Implement legacy @app.get("/health") calling readiness() and formatting
  response
• Include verification commands for all three endpoints
  ===============================================================================

Task: 3

Build the comprehensive backend API layer for the SOC dashboard. The goal is to
create complete endpoints for event management, analytics, alerts, and real-time
streaming with full implementation code.

Events Listing and Search:

• Add imports to services/ingest/app.py: Optional, List from typing, datetime,
  timedelta
• Create EventListResponse Pydantic model: total_count (int), page (int),
  page_size (int), events (List[dict])
• Implement @app.get("/v1/events", response_model=EventListResponse) with
  parameters:
  • api_key: str = Depends(verify_api_key)
  • verdict, threat_type, source_type: Optional[str] = None
  • hours: int = 24, page: int = 1, page_size: int = 50
• Implementation logic:
  • Check redis_client availability (raise 503 if not)
  • Initialize cursor=0, all_events=[]
  • Use redis_client.scan with pattern="tenet:event:*", count=100
• For each key: get event data, parse JSON, apply filters (verdict, threat_type,
  source_type), append to all_events
  • Loop until cursor==0
  • Sort all_events by timestamp descending
  • Calculate pagination: start=(page-1)*page_size, end=start+page_size
  • Return EventListResponse with total_count, page, page_size, paginated events
• Add comment: Production should use PostgreSQL with proper indexing

Statistics and Analytics:

• Create StatsResponse Pydantic model: total_events (int), blocked_count (int),
  threats_by_type (dict), verdicts (dict), average_risk_score (float),
  time_range_hours (int)
• Implement @app.get("/v1/stats", response_model=StatsResponse) with
  parameters:
  • api_key: str = Depends(verify_api_key)
  • hours: int = 24
• Implementation logic:
  • Check redis_client availability
• Initialize stats dict with: total=0, blocked=0, threats={},
  verdicts={'benign':0,'suspicious':0,'malicious':0}, risk_scores=[]
  • Use redis_client.scan with same pattern as events listing
• For each event: increment total, check blocked and increment, aggregate
  threat_type counts, aggregate verdict counts, collect risk_scores
  • Calculate avg_risk: sum/len if risk_scores else 0.0
  • Return StatsResponse with all aggregated data, round avg_risk to 3 decimals

Alerts Management:

• Create AlertStatus str Enum: new, acknowledged, investigating, resolved,
  false_positive
• Create Alert Pydantic model: alert_id (str), event_id (str), severity (str),
  status (AlertStatus), threat_type (str), created_at (str), updated_at (str),
  notes (List[dict], default [])
• Create AlertUpdate model: status (Optional[AlertStatus])
• Create AlertNote model: text (str), author (str, default="analyst")
• Implement @app.get("/v1/alerts") with parameters: api_key, status
  (Optional[AlertStatus]), severity (Optional[str]), page (int=1), page_size
  (int=50)
  • Get alerts from redis_client.lrange("tenet:alerts", 0, -1)
  • Parse JSON, filter by status and severity
  • Paginate and return with total_count
• Implement @app.put("/v1/alerts/{alert_id}") with alert_id (str), update
  (AlertUpdate), api_key
  • Get alert from "tenet:alert:{alert_id}" key
  • Raise 404 if not found
  • Update status if provided, set updated_at to current ISO timestamp
  • Save back to Redis, return updated alert
• Implement @app.post("/v1/alerts/{alert_id}/notes") with alert_id (str), note
  (AlertNote), api_key
  • Get alert, raise 404 if not found
  • Append note to notes array with text, author, timestamp
  • Update updated_at field
  • Save to Redis, return status and alert

Real-Time Streaming:

• Import: WebSocket, WebSocketDisconnect from fastapi, Set from typing, asyncio
• Create module-level: active_connections: Set[WebSocket] = set()
• Implement @app.websocket("/v1/events/stream"):
  • Accept websocket connection
  • Add to active_connections
  • Try block:
    • Create pubsub = redis_client.pubsub()
    • Subscribe to "tenet:events:new"
    • While True loop:
      • Get message with timeout=1.0, ignore_subscribe_messages=True
      • If message type is 'message': send decoded data to websocket
      • Try to receive text from websocket with timeout=0.1 for filter updates
      • Log received data
  • Except WebSocketDisconnect: discard from active_connections
  • Finally: unsubscribe and discard from active_connections
• Implement async broadcast_event(event: dict):
  • Check if active_connections not empty
  • Convert event to JSON string
  • For each connection (copy of set):
    • Try to send_text, on exception discard connection
• Modify ingest_llm_event function:
• After queuing event to Redis, add: await
  redis_client.publish("tenet:events:new", json.dumps(response_data))
  ===============================================================================

Task: 4

Create comprehensive Python SDK and integration libraries for TENET AI. The goal
is to provide complete, production-ready client libraries and framework
integrations with full implementation code.

Python SDK Client:

• Create directory structure: sdk/python/tenet_ai/ with init.py,
  client.py, exceptions.py
• Create sdk/python/tenet_ai/exceptions.py:
  • Define TenetError(Exception) base class
• Define TenetBlockedError(TenetError) with init(message, risk_score,
  threat_type), store risk_score and threat_type as attributes
  • Define TenetAuthError(TenetError)
• Create sdk/python/tenet_ai/client.py:
  • Import httpx, Optional, Dict, Any from typing, exceptions module
  • Implement TenetClient class:
• init with api_key, base_url (default http://localhost:8000), timeout
  (10.0), blocking_mode (True)
  • Store attributes, create httpx.Client with timeout
  • _headers method returning X-API-Key and Content-Type headers
• submit_event method: build payload with
  prompt/source_type/source_id/model/system_prompt/metadata, POST to
  /v1/events/llm, handle 401 as TenetAuthError, raise TenetBlockedError if blocked
  and blocking_mode
• analyze method: POST to analyzer service :8100/v1/analyze with prompt and
  context
  • get_event method: GET /v1/events/{event_id}
  • get_stats method: GET /v1/stats with hours parameter
  • close method and context manager support (enter, exit)
• Implement AsyncTenetClient class with async versions of all methods using
  httpx.AsyncClient
• Create sdk/python/tenet_ai/__init__.py: import and export TenetClient,
  AsyncTenetClient, all exceptions, set version="0.1.0"
• Create sdk/python/pyproject.toml:
  • build-system: setuptools>=61.0
• project name="tenet-ai", version="0.1.0", description, readme,
  requires-python=">=3.9", license=MIT
  • dependencies: httpx>=0.25.0
• optional-dependencies: langchain=["langchain>=0.1.0"],
  fastapi=["fastapi>=0.100.0", "starlette>=0.27.0"], dev=["pytest",
  "pytest-asyncio"]
  • project.urls with Homepage, Documentation

LangChain Integration:

• Create sdk/python/tenet_ai/integrations/__init__.py as empty package
• Create sdk/python/tenet_ai/integrations/langchain.py:
• Import typing types, UUID, try/except import BaseCallbackHandler from
  langchain.callbacks.base with helpful error message
  • Import TenetClient, AsyncTenetClient, TenetBlockedError from parent modules
  • Implement TenetCallbackHandler(BaseCallbackHandler):
• init with api_key, base_url (default localhost:8000), blocking_mode
  (True), log_only (False), source_type ("langchain")
  • Create TenetClient with blocking_mode and not log_only
  • Store log_only and source_type
• Implement on_llm_start(serialized, prompts, run_id, parent_run_id, tags,
  metadata, kwargs):
  - Get model_name from serialized.get("name", "unknown")
• For each prompt: submit_event with prompt, source_type, run_id as source_id,
  model_name, metadata dict with tags/parent_run_id
  - If blocked and not log_only: raise TenetBlockedError
• Catch TenetBlockedError and re-raise, catch other exceptions and print warning
• Implement on_chain_start with similar logic: extract input_text from inputs
  (try "input", "query", "question" keys), submit_event with
  source_type="{source_type}-chain", metadata with chain_inputs keys
• Add usage example to documentation showing: create TenetCallbackHandler, use
  with LangChain LLM via callbacks parameter, demonstrate blocking behavior

FastAPI Middleware:

• Create sdk/python/tenet_ai/integrations/fastapi.py:
• Try/except import Request, Response from fastapi and BaseHTTPMiddleware from
  starlette with helpful error message
  • Import typing, json, TenetClient, TenetBlockedError
  • Implement TenetMiddleware(BaseHTTPMiddleware):
• init(app, api_key, base_url="http://localhost:8000", protected_paths=None,
  excluded_paths=None, blocking_mode=True, extract_prompt=None)
  • Call super().init(app)
  • Create TenetClient with blocking_mode=False (we handle it)
  • Set protected_paths default to ["/api/", "/v1/"]
• Set excluded_paths default to {"/health", "/metrics", "/docs", "/redoc",
  "/openapi.json"}
  • Store blocking_mode and extract_prompt or _default_extract_prompt
  • Implement _default_extract_prompt(body: dict) -> Optional[str]:
    • Check common keys: prompt, message, query, input, text, content
    • Check messages array for OpenAI format (role="user")
    • Return None if not found
• Implement _should_protect(path: str) -> bool: check if path in excluded_paths
  or starts with any protected_paths prefix
  • Implement async dispatch(request, call_next):
    • Skip if not _should_protect or method not in POST/PUT
    • Read and parse request body as JSON
    • Extract prompt using self.extract_prompt
    • Skip if no prompt found
    • Submit event to TENET with metadata (method, path, client IP)
• If blocked and blocking_mode: return Response with 403, JSON body with
  error/event_id/threat_type
  - Otherwise call next and add X-Tenet headers (Event-ID, Risk-Score)
  - Catch all exceptions and print warning, continue to call_next
• Add usage example showing: create FastAPI app, add TenetMiddleware with
  configuration, protected endpoint that gets automatic protection
  ===============================================================================

Task: 5

Implement comprehensive security hardening for TENET AI. The goal is to
eliminate hardcoded credentials, add startup configuration validation with
Pydantic, and support API key rotation.

Remove Hardcoded Credentials:

• Modify docker-compose.yml:
• Lines 27-31 (postgres section): Replace POSTGRES_PASSWORD: postgres with
  ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}
  • Also use ${POSTGRES_USER:?POSTGRES_USER is required} syntax
  • Set POSTGRES_DB with default: ${POSTGRES_DB:-tenet}
• Lines 45-51 (minio section): Replace MINIO_ROOT_USER: minio and
  MINIO_ROOT_PASSWORD: minio123 with ${MINIO_ROOT_USER:?MINIO_ROOT_USER is
  required} and ${MINIO_ROOT_PASSWORD:?MINIO_ROOT_PASSWORD is required}
• Update .env.template:
  • Add DATABASE CONFIGURATION section header
  • Add comment about generating secure passwords: "openssl rand -base64 32"
• Add POSTGRES_USER=tenet, POSTGRES_PASSWORD= (empty, marked REQUIRED with min
  16 chars note), POSTGRES_DB=tenet
  • Add OBJECT STORAGE section header
• Add MINIO_ROOT_USER= (empty, marked REQUIRED with min 8 chars),
  MINIO_ROOT_PASSWORD= (empty, marked REQUIRED with min 16 chars)
  • Add API AUTHENTICATION section header
• Add generation command: python -c "import secrets;
  print(f'tenet-{secrets.token_urlsafe(32)}')"
  • Add API_KEY= (empty, marked REQUIRED)
• Create scripts/check_env.sh bash script:
  • Define color codes: RED, GREEN, NC
  • Initialize ERRORS=0 counter
• Define check_var function: takes var_name and min_length, checks if set and
  meets length, increments ERRORS and prints colored messages
• Define check_not_default function: takes var_name and default_value, checks if
  using default, increments ERRORS
  • Print header "Checking TENET AI environment configuration..."
• Call check_var for: POSTGRES_PASSWORD (16), MINIO_ROOT_USER (8),
  MINIO_ROOT_PASSWORD (16), API_KEY (20)
• Call check_not_default for: POSTGRES_PASSWORD ("postgres"),
  MINIO_ROOT_PASSWORD ("minio123"), API_KEY ("tenet-dev-key-change-in-production")
  • Print summary with error count, exit 1 if errors else print success message
• Add verification: chmod +x and run script expecting failure with defaults

Configuration Validation:

• Create services/common/config.py:
• Import os, sys, re, Optional from typing, BaseSettings, validator, SecretStr
  from pydantic
  • Define TenetConfig(BaseSettings) class:
• Fields: api_key (SecretStr), api_host (str, default "0.0.0.0"), api_port (int,
  default 8000), redis_host (str, default "localhost"), redis_port (int, default
  6379), rate_limit_rpm (int, default 1000), max_prompt_length (int, default
  10000), blocking_enabled (bool, default True), log_level (str, default "INFO"),
  log_format (str, default "json"), environment (str, default "development")
  • Config class: env_file=".env", env_file_encoding="utf-8"
• @validator("api_key"): get secret value, check length >= 20, check not in
  insecure_defaults list ["tenet-dev-key-change-in-production", "changeme",
  "test", "secret"], raise ValueError with clear messages
• @validator("environment"): if v=="production", check api_key doesn't contain
  "dev" or "test"
• @validator("log_level"): check against valid_levels ["DEBUG", "INFO",
  "WARNING", "ERROR", "CRITICAL"], return uppercase
• Define load_config() -> TenetConfig: try to instantiate TenetConfig, on
  exception print to stderr and sys.exit(1)
• Define log_config_summary(config, logger): call logger.info with extra dict
  containing all non-secret config fields (api_host, api_port, redis_host,
  redis_port, environment, log_level, blocking_enabled, rate_limit_rpm)
• Update services/ingest/app.py:
  • Import load_config, log_config_summary from common.config
  • Call config = load_config() at module level
• Add @app.on_event("startup") function that calls log_config_summary(config,
  logger)

API Key Rotation:

• Modify API key validation logic to split on comma and check against list of
  keys
• Define key metadata structure with name, created_at, expires_at fields
• Create /v1/admin/keys endpoint requiring admin authentication:
  • GET to list all keys (without revealing full key)
  • POST to create new key
  • DELETE to revoke key
• Implement expiration checking: compare expires_at to current time, log
  warnings
• Add audit logging: log all key validation attempts, key creation, key
  revocation with timestamps and metadata
  ===============================================================================

Task: 6

Create comprehensive documentation and contributor onboarding materials for
TENET AI. The goal is to enable new contributors with detailed GitHub issues and
provide clear API and integration documentation.

GitHub Issues Creation:

• Create detailed issue template for Task 1.1 (Coverage):
  • Title: "Add pytest coverage reporting to CI"
• Description section explaining the current state (tools installed but not
  configured)
• Acceptance Criteria checklist: CI runs pytest with coverage, upload to
  Codecov, fail below 70%, update CONTRIBUTING.md
• Implementation section with complete YAML code block showing pytest coverage
  step and Codecov upload step
  • Files to Modify: .github/workflows/ci.yml, CONTRIBUTING.md
  • Labels: good first issue, CI/CD, documentation
• Create similar detailed issues for:
  • Task 1.2: Enable linting (Black and Ruff configuration)
  • Task 1.3: Enable type checking (mypy setup)
  • Task 1.4: Security scanning (Bandit and Safety)
  • Task 2.1: Structured JSON logging (python-json-logger)
  • Task 2.2: Prometheus metrics endpoint
  • Task 5.1: Remove hardcoded credentials from docker-compose
• Each issue should include: clear problem statement, step-by-step acceptance
  criteria, implementation guidance with code snippets, file paths, helpful labels
• Link all issues back to this roadmap document for context

API Reference Documentation:

• Create docs/api-reference.md with comprehensive API documentation
• Enable FastAPI's built-in documentation:
  • Configure /docs endpoint for Swagger UI
  • Configure /redoc endpoint for ReDoc interface
• Document for each endpoint:
  • HTTP method and path
  • Authentication requirements (X-API-Key header)
  • Request schema with field descriptions and types
  • Response schema with examples
• Error codes: 200 (success), 401 (unauthorized), 403 (blocked), 404 (not
  found), 503 (service unavailable)
• Add example requests:
  • curl commands with headers and JSON bodies
  • Python examples using requests library and SDK
• Include operational details:
  • Rate limiting: requests per minute limits
  • Quota information
  • Pagination parameters and behavior

Integration Guide:

• Create docs/integrations.md covering all integration patterns
• SDK Installation section:
  • pip install commands
  • Optional dependencies [langchain] [fastapi]
• Direct API Integration:
  • Authentication setup
  • Example HTTP requests
  • Error handling patterns
• LangChain Integration:
  • Step-by-step setup of TenetCallbackHandler
  • Configuration options: blocking_mode, log_only
  • Code examples for LLM and chain integration
  • Callback lifecycle explanation
• FastAPI Middleware Integration:
  • Step-by-step setup of TenetMiddleware
  • Configuration: protected_paths, excluded_paths, custom extractors
  • Code examples for adding to FastAPI app
  • Request/response flow explanation
• Troubleshooting section:
  • Common issues: connection errors, authentication failures, timeout handling
  • Solutions and workarounds
  • Debug logging guidance
• Architecture diagrams:
  • Request flow showing where TENET intercepts
  • Integration points in application architecture
  • Data flow from client through TENET to LLM

</details>


<details>
<summary><b>💡 Iterate on the plan with:
<code>@coderabbitai &lt;feedback&gt;</code>
</b></summary>

Example Feedback

• @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
• @coderabbitai For design choice 1 go ahead with option 3 and replan.

</details>


---



💬 Have feedback or questions? Drop into our [discord](https://discord.gg/coderabbit)!</planningResult></rawResChunk> -->

<!-- <agentComponentType>plan-edits</agentComponentType> -->

[S3DFX-CYBER]

@coderabbitai how can we use cloudflare for tenet ai

[coderabbitai]

🧩 Analysis chain

🌐 Web query:

Cloudflare AI Gateway features capabilities 2026

💡 Result:

Cloudflare AI Gateway capabilities (as documented through Jan 2026) include:

• Single “unified” (OpenAI-compatible) API endpoint (/chat/completions) to swap providers/models by changing model (format {provider}/{model}), plus a list of supported providers. [1]
• Performance & cost controls
  • Caching of identical requests (configurable TTL / per-request control). [2]
  • Rate limiting (fixed or sliding windows; per-gateway/per-request). [2]
  • Custom costs for more accurate internal cost tracking (e.g., negotiated rates). [2]
• Reliability / traffic management
  • Dynamic Routing (beta): visual/JSON routing flows with conditional routing, percentage splits (A/B), rate-limit + budget-limit nodes, versioning/rollback, and fallbacks. [3]
  • Timeouts, automatic retries (up to 5), and fallbacks (via Universal Endpoint / request handling features). [4][5]
  • Provider/model fallbacks with cf-aig-step response header to show which step succeeded. [5]
  • Realtime WebSockets API for low-latency, persistent “realtime” interactions (e.g., OpenAI Realtime, Gemini Live; text and speech). [6]
• Security, safety, and governance
  • Guardrails (beta) for moderation on prompts + responses (flag/block) across providers. [7][2]
  • DLP (beta) to scan prompts/responses for sensitive data (PII/financial/healthcare/custom patterns) with flag/block actions. [2]
  • Gateway authentication (token-based) and BYOK key storage (encrypted; centralized key rotation; 20+ providers). [2]
  • Unified Billing (beta) to pay multiple AI providers via one Cloudflare invoice; includes opt-in Zero Data Retention (ZDR) support for some providers. [8][9]
• Observability
  • Analytics + logging of requests, tokens, costs, errors, cache hit rate; custom metadata tagging; export via Logpush. [2][10]
  • OpenTelemetry export of AI request traces (model/provider, tokens, prompts/completions, cost estimates, metadata). [11]
• Extensibility
  • Custom Providers (OpenAI-compatible) to integrate non-native providers (noted in AI Gateway changelog). [9]

Sources: Cloudflare AI Gateway docs + changelog pages cited inline. [1]-[11]

---

@S3DFX-CYBER, excellent question! Cloudflare can enhance TENET AI in multiple ways. Here are the key integration approaches:

🔀 Integration Architecture Options

Option 1: Layered Defense (Recommended)

Deploy TENET AI behind Cloudflare AI Gateway for defense-in-depth:

Client → Cloudflare AI Gateway → TENET AI → LLM Provider
         (Rate limiting, caching)  (Specialized threat detection)

Benefits:

• Cloudflare handles infrastructure concerns (caching, rate limiting, DLP)
• TENET AI provides specialized ML-based threat detection (prompt injection, jailbreaks)
• Complementary security layers

Option 2: Parallel Monitoring

Use both systems independently for comprehensive coverage:

                    ┌→ Cloudflare AI Gateway → LLM
Client Request ─────┤
                    └→ TENET AI (analysis only)

Option 3: TENET AI as Cloudflare Worker

Deploy TENET AI services on Cloudflare's edge infrastructure.

🚀 Specific Cloudflare Services for TENET AI

1. Cloudflare AI Gateway Integration

Complementary Features:

Feature	Cloudflare AI Gateway	TENET AI
Rate Limiting	✅ Built-in	❌ Not yet
Request Caching	✅ Built-in	❌ Not yet
Provider Fallbacks	✅ Dynamic routing	❌ N/A
Prompt Injection Detection	❌ Basic guardrails	✅ ML-based detection
Jailbreak Detection	❌ Basic guardrails	✅ Specialized models
Data Exfiltration Detection	❌ Limited	✅ Pattern analysis
Custom Security Rules	❌ Limited	✅ Extensible

Integration Pattern:

# TENET AI as a Custom Provider in Cloudflare AI Gateway
# Configure Cloudflare to route through TENET AI endpoint

# In your application:
response = requests.post(
    "https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway}/openai/chat/completions",
    headers={
        "Authorization": f"Bearer {cloudflare_token}",
        "cf-aig-metadata": json.dumps({"tenet_analysis": True})
    },
    json={
        "model": "tenet-ai/gpt-4",  # Custom provider routing
        "messages": messages
    }
)

2. Cloudflare Workers Deployment

Deploy TENET AI microservices as Cloudflare Workers for global edge distribution:

// Example: TENET AI Ingest Worker
export default {
  async fetch(request: Request, env: Env) {
    // Fast heuristic checks at edge
    const body = await request.json();
    const verdict = await analyzeThreat(body.prompt);
    
    if (verdict.risk_score > 0.8) {
      return new Response(JSON.stringify({
        blocked: true,
        reason: verdict.threat_type
      }), { status: 403 });
    }
    
    // Queue for deeper analysis
    await env.EVENTS_QUEUE.send(body);
    return new Response(JSON.stringify({ allowed: true }));
  }
};

Benefits:

• Global edge deployment (reduced latency)
• Auto-scaling
• DDoS protection included
• Integration with Cloudflare AI Gateway

3. Cloudflare R2 for Storage

Replace MinIO with Cloudflare R2 for event storage:

# Replace MinIO configuration
import boto3

s3_client = boto3.client(
    's3',
    endpoint_url=f'https://{CLOUDFLARE_ACCOUNT_ID}.r2.cloudflarestorage.com',
    aws_access_key_id=R2_ACCESS_KEY_ID,
    aws_secret_access_key=R2_SECRET_ACCESS_KEY,
    region_name='auto'
)

# Store events with zero egress fees
s3_client.put_object(
    Bucket='tenet-events',
    Key=f'events/{event_id}.json',
    Body=json.dumps(event_data)
)

4. Cloudflare Zero Trust Access

Secure TENET AI dashboard and API endpoints:

# cloudflare-access-policy.yaml
- name: "TENET AI Admin Dashboard"
  decision: allow
  includes:
    - email_domain: your-company.com
  requires:
    - mfa: true
  
- name: "TENET AI API"
  decision: service_auth
  includes:
    - service_token: true

5. Cloudflare WAF for Additional Protection

Add WAF rules before traffic reaches TENET AI:

// Custom WAF rule for TENET AI
(http.request.uri.path matches "/v1/events" and 
 http.request.body.truncated) or
(rate limit exceeded per ip)

🏗️ Recommended Architecture

graph LR
    A[Client] -->|1. Request| B[Cloudflare Edge]
    B -->|2. WAF + DDoS| C[AI Gateway]
    C -->|3. Cache Check| D{Cached?}
    D -->|Yes| E[Return Cached]
    D -->|No| F[TENET AI Ingest]
    F -->|4. Threat Analysis| G{Malicious?}
    G -->|Yes| H[Block + Alert]
    G -->|No| I[LLM Provider]
    I -->|5. Response| J[TENET AI Monitor]
    J -->|6. Log Event| K[Cloudflare R2]
    J -->|7. Return| A
    
    F -.->|Async| L[TENET AI Analyzer]
    L -.->|ML Analysis| M[Redis Queue]
    
    style B fill:`#f96`,stroke:`#333`
    style F fill:`#6cf`,stroke:`#333`
    style L fill:`#6cf`,stroke:`#333`

Loading

📊 Implementation Roadmap

Phase 1: Infrastructure Migration

1. Deploy TENET AI on Cloudflare Workers (ingest service)
2. Migrate Redis to Cloudflare Durable Objects or external Redis with Cloudflare Tunnel
3. Replace MinIO with Cloudflare R2

Phase 2: AI Gateway Integration

1. Configure Cloudflare AI Gateway with TENET AI as custom provider
2. Enable caching for safe prompts
3. Implement rate limiting at edge

Phase 3: Enhanced Security

1. Add Cloudflare Zero Trust for dashboard access
2. Configure WAF rules for API protection
3. Enable DLP scanning for sensitive data

Phase 4: Observability

1. Enable Cloudflare Analytics for request metrics
2. Configure Logpush to send logs to SIEM
3. Set up OpenTelemetry export for distributed tracing

💡 Key Considerations

When to use Cloudflare AI Gateway + TENET AI:

• ✅ Need comprehensive LLM security (infrastructure + threat detection)
• ✅ Want edge caching and rate limiting
• ✅ Require multi-provider LLM routing with security
• ✅ Need global low-latency deployment

When to use TENET AI standalone:

• ✅ Already have existing infrastructure
• ✅ Need specialized threat detection models
• ✅ Require custom security rules and policies
• ✅ Want full control over security logic

Would you like me to create a detailed implementation plan for any specific integration approach, or help you set up a proof-of-concept deployment?

[S3DFX-CYBER]

@coderabbitai help me with this Task 1: Add Coverage Reporting to CI
Enable pytest-cov execution in the CI workflow with a minimum coverage threshold to prevent regression.

Update .github/workflows/ci.yml to add --cov=services --cov-report=xml --cov-fail-under=70 flags to the pytest command
Add a step to upload coverage report as artifact or integrate with Codecov/Coveralls
Document coverage requirements in CONTRIBUTING.md

[coderabbitai]

I've updated the plan above based on your feedback. Ready to proceed when you are!

[S3DFX-CYBER]

@coderabbitai explain the plan

[coderabbitai]

@S3DFX-CYBER, I'll break down the comprehensive roadmap plan I created for TENET AI:

🎯 Overall Strategy

The plan addresses the missing multi-tenant AI features you asked about by separating work into 6 phases that balance:

• Good First Issues - Self-contained tasks perfect for open-source contributors
• Maintainer Work - Larger architectural changes requiring deep codebase knowledge

📋 Phase Breakdown

Phase 1: CI/CD Hardening (Good First Issues)

Goal: Add automated quality gates to catch bugs before they reach production

What's included:

• Task 1: Add coverage reporting (pytest-cov → Codecov) with 70% minimum threshold
• Task 2: Enable code formatting checks (Black + Ruff)
• Task 3: Add static type checking (mypy)
• Task 4: Integrate security scanning (Bandit for code, Safety for dependencies)

Why first? All tools are already installed but not running in CI. These are configuration-only changes with no code modifications needed - perfect for contributors.

---

Phase 2: Observability Foundation (Good First Issues)

Goal: Add production-ready logging and metrics (currently missing)

What's included:

• Task 1: Structured JSON logging with request IDs for tracing
• Task 2: Prometheus metrics endpoint (/metrics) for monitoring
• Task 3: Enhanced health checks (separate liveness/readiness endpoints)

Why next? You can't operate a production service without observability. These are self-contained additions that don't change existing functionality.

---

Phase 3: Dashboard Backend APIs (Maintainer Roadmap)

Goal: Build the API layer for the planned SOC dashboard

What's included:

• Task 1: Events listing/search API (filter, paginate, query events)
• Task 2: Statistics API (aggregate metrics for charts/dashboards)
• Task 3: Alerts management API (CRUD for security alerts)
• Task 4: Real-time WebSocket streaming for live updates

Why maintainer-only? Requires understanding the data model, Redis architecture, and API design patterns. Not isolated enough for first-time contributors.

---

Phase 4: Python SDK & Integration Libraries (Mixed)

Goal: Make TENET AI easy to integrate into applications

What's included:

• Task 1: Python SDK client library (TenetClient, AsyncTenetClient)
• Task 2: LangChain callback handler integration
• Task 3: FastAPI middleware package

Why mixed? SDK creation (Task 1) needs maintainer review for API design, but once the SDK exists, integration packages (Tasks 2-3) can be contributor tasks with clear interfaces.

---

Phase 5: Security Hardening (Mixed)

Goal: Eliminate security issues (hardcoded credentials, weak validation)

What's included:

• Task 1: Remove hardcoded passwords from docker-compose (contributor)
• Task 2: Add Pydantic config validation with startup checks (maintainer)
• Task 3: API key rotation support (maintainer)

Why mixed? Task 1 is simple find-and-replace. Tasks 2-3 require understanding authentication flow and failure modes.

---

Phase 6: Documentation (Good First Issues)

Goal: Enable new contributors and users

What's included:

• Task 1: Create GitHub issues for all "good first issue" tasks
• Task 2: API reference documentation (OpenAPI/Swagger)
• Task 3: Integration guides (SDK, LangChain, FastAPI examples)

Why last? Documentation references the code/features built in earlier phases. Once those exist, documentation tasks are perfect for contributors.

---

🔑 Key Design Decisions

1. Scope Separation

I explicitly separated:

• Contributor tasks → Configuration changes, documentation, isolated features
• Maintainer tasks → Architectural changes, API design, multi-service coordination

This enables parallel work without blocking contributors.

2. Backend-First for Dashboard

Phase 3 builds APIs before any frontend work because:

• API contracts need to be stable before UI development
• Allows API testing independent of UI
• Contributors can build frontend once APIs are documented

3. Codecov Integration

For coverage reporting (Phase 1), I chose Codecov over alternatives because:

• Free for open source
• PR comments show coverage changes
• Industry standard with badge support

---

📊 Where You're Starting From

Current gaps I identified:

• ❌ No CI automation (tools installed but not executed)
• ❌ No observability (metrics, structured logs)
• ❌ No dashboard APIs (only single-event endpoints)
• ❌ No SDK/client libraries
• ❌ Hardcoded credentials in docker-compose
• ❌ Basic health checks only

What you have:

• ✅ Solid unit tests (good foundation)
• ✅ Working microservices (ingest + analyzer)
• ✅ Redis queue architecture
• ✅ ML models for threat detection

---

🚀 How to Proceed

1. For contributors: Create GitHub issues using the templates in Phase 6, Task 1
2. For maintainers: Start with Phase 1 (CI hardening) - it's quick wins that improve quality immediately
3. Parallel work: Phases 1-2 can happen while you work on Phase 3 (dashboard APIs)

Each phase includes:

• Complete implementation code (copy-pasteable)
• Verification commands to test locally
• AI agent prompts if you want to use AI coding assistants

Would you like me to:

1. Create the GitHub issues for Phase 1-2 good first issues?
2. Generate implementation code for a specific phase?
3. Explain any particular phase in more detail?

[S3DFX-CYBER]

@coderabbitai Create the GitHub issues for Phase 1-2 good first issues

[coderabbitai]

🧩 Analysis chain

@S3DFX-CYBER, I'll create the GitHub issues for Phase 1-2 good first issues right away!

python-json-logger>=2.0.7

**Create `services/common/__init__.py`:**

```python
# Empty file to make this a package

Create services/common/logging.py:

import logging
import os
import sys
import uuid
from contextvars import ContextVar
from pythonjsonlogger import jsonlogger

# Context variable for request ID
request_id_var: ContextVar[str] = ContextVar("request_id", default="")

class TenetJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        log_record["service"] = os.getenv("SERVICE_NAME", "unknown")
        log_record["version"] = os.getenv("SERVICE_VERSION", "0.0.0")
        log_record["request_id"] = request_id_var.get()
        log_record["level"] = record.levelname
        log_record["timestamp"] = record.created

def setup_logging(service_name: str) -> logging.Logger:
    log_level = os.getenv("LOG_LEVEL", "INFO")
    log_format = os.getenv("LOG_FORMAT", "json")
    
    logger = logging.getLogger(service_name)
    logger.setLevel(log_level)
    
    handler = logging.StreamHandler(sys.stdout)
    
    if log_format == "json":
        formatter = TenetJsonFormatter("%(timestamp)s %(level)s %(name)s %(message)s")
    else:
        formatter = logging.Formatter(
            "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        )
    
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    
    return logger

def generate_request_id() -> str:
    return str(uuid.uuid4())[:8]

def set_request_id(request_id: str):
    request_id_var.set(request_id)

Update services/ingest/app.py:

Replace the logging import and setup (lines 17-21) with:

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))

from common.logging import setup_logging, generate_request_id, set_request_id

logger = setup_logging("ingest")

# Add middleware for request IDs
`@app.middleware`("http")
async def add_request_id(request: Request, call_next):
    request_id = request.headers.get("X-Request-ID", generate_request_id())
    set_request_id(request_id)
    response = await call_next(request)
    response.headers["X-Request-ID"] = request_id
    return response

✅ Verification

📚 Files to Create/Modify

• requirements.txt (add dependency)
• services/common/__init__.py (create)
• services/common/logging.py (create)
• services/ingest/app.py (update imports and add middleware)

🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

💡 Notes

• Environment variables: LOG_LEVEL (DEBUG/INFO/WARNING/ERROR), LOG_FORMAT (json/text)
• Request IDs enable tracing requests across services
• JSON logs are parseable by log aggregation tools (ELK, Splunk, etc.)'

echo "✅ Created Issue 5: Implement structured JSON logging with request IDs"

Create Phase 2 - Task 2: Prometheus Metrics

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add Prometheus metrics endpoint"
--label "good first issue,observability,enhancement"
--assignee S3DFX-CYBER
--body '## 📊 Description

Expose application metrics via a /metrics endpoint for Prometheus scraping. This enables monitoring of request rates, latencies, threat detection, and system health.

🎯 Acceptance Criteria

• prometheus-client added to requirements.txt
• services/common/metrics.py module created with metric definitions
• /metrics endpoint added to ingest service
• Metrics middleware tracks all requests
• Verification shows Prometheus-format output
• Common metrics defined: requests, latency, threats, queue depth

🔨 Implementation

Add to requirements.txt:

prometheus-client>=0.19.0

Create services/common/metrics.py:

from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
from fastapi import Response

# Request metrics
REQUEST_COUNT = Counter(
    '\''tenet_requests_total'\'',
    '\''Total HTTP requests'\'',
    ['\''service'\'', '\''endpoint'\'', '\''method'\'', '\''status'\'']
)

REQUEST_LATENCY = Histogram(
    '\''tenet_request_duration_seconds'\'',
    '\''HTTP request latency'\'',
    ['\''service'\'', '\''endpoint'\''],
    buckets=(.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0)
)

# Security metrics
THREATS_DETECTED = Counter(
    '\''tenet_threats_detected_total'\'',
    '\''Total threats detected'\'',
    ['\''threat_type'\'', '\''verdict'\'']
)

EVENTS_BLOCKED = Counter(
    '\''tenet_events_blocked_total'\'',
    '\''Total events blocked'\'',
    ['\''threat_type'\'']
)

# System metrics
QUEUE_DEPTH = Gauge(
    '\''tenet_queue_depth'\'',
    '\''Current queue depth'\'',
    ['\''queue_name'\'']
)

MODEL_INFERENCE_LATENCY = Histogram(
    '\''tenet_model_inference_seconds'\'',
    '\''ML model inference time'\'',
    ['\''model_type'\''],
    buckets=(.001, .005, .01, .025, .05, .1, .25, .5)
)

def metrics_endpoint():
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

Update services/ingest/app.py:

Add imports:

import time
from common.metrics import (
    REQUEST_COUNT, REQUEST_LATENCY, metrics_endpoint
)

Add metrics endpoint:

`@app.get`("/metrics")
async def metrics():
    return metrics_endpoint()

Add metrics middleware:

`@app.middleware`("http")
async def track_metrics(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    duration = time.time() - start_time
    
    REQUEST_COUNT.labels(
        service="ingest",
        endpoint=request.url.path,
        method=request.method,
        status=response.status_code
    ).inc()
    
    REQUEST_LATENCY.labels(
        service="ingest",
        endpoint=request.url.path
    ).observe(duration)
    
    return response

✅ Verification

📚 Files to Create/Modify

• requirements.txt (add dependency)
• services/common/metrics.py (create)
• services/ingest/app.py (add endpoint and middleware)

🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

💡 Notes

• Prometheus can scrape the /metrics endpoint automatically
• Histogram buckets are optimized for API latencies (5ms to 5s)
• Additional metrics can be added as needed (memory usage, cache hit rate, etc.)'

echo "✅ Created Issue 6: Add Prometheus metrics endpoint"

Create Phase 2 - Task 3: Enhanced Health Checks

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Enhanced health check endpoints (liveness/readiness)"
--label "good first issue,observability,enhancement"
--assignee S3DFX-CYBER
--body '## 📊 Description

Expand health checks to include dependency status and separate liveness/readiness endpoints following Kubernetes best practices.

🎯 Acceptance Criteria

• /health/live endpoint added for liveness checks
• /health/ready endpoint added for readiness checks (with dependency checks)
• Legacy /health endpoint maintained for backward compatibility
• Redis connection check implemented
• Returns HTTP 503 when not ready
• Verification commands test all endpoints

🔨 Implementation

Update services/ingest/app.py:

Replace the existing health_check function with:

from fastapi import Response
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from typing import Dict

class LivenessResponse(BaseModel):
    status: str
    service: str
    version: str

class ReadinessResponse(BaseModel):
    status: str
    service: str
    version: str
    checks: Dict[str, str]

`@app.get`("/health/live", response_model=LivenessResponse)
async def liveness():
    """Liveness probe - returns 200 if service is running"""
    return LivenessResponse(
        status="alive",
        service="ingest",
        version="0.1.0"
    )

`@app.get`("/health/ready", response_model=ReadinessResponse)
async def readiness():
    """Readiness probe - returns 200 if service can handle requests"""
    checks = {}
    overall_status = "ready"
    
    # Check Redis connection
    try:
        if redis_client:
            redis_client.ping()
            checks["redis"] = "connected"
        else:
            checks["redis"] = "not_configured"
            overall_status = "degraded"
    except Exception as e:
        checks["redis"] = f"error: {str(e)}"
        overall_status = "not_ready"
    
    response_data = ReadinessResponse(
        status=overall_status,
        service="ingest",
        version="0.1.0",
        checks=checks
    )
    
    status_code = 200 if overall_status in ["ready", "degraded"] else 503
    return JSONResponse(content=response_data.dict(), status_code=status_code)

`@app.get`("/health")
async def health_check():
    """Legacy health endpoint for backward compatibility"""
    ready_response = await readiness()
    return ready_response

✅ Verification

📚 Files to Modify

• services/ingest/app.py

🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

💡 Notes

• Liveness: Checks if the service process is running (simple, fast)
• Readiness: Checks if service can handle traffic (includes dependency checks)
• Kubernetes uses liveness for restarts, readiness for routing traffic
• HTTP 503 (Service Unavailable) signals orchestrator to stop routing traffic'

echo "✅ Created Issue 7: Enhanced health check endpoints (liveness/readiness)"

echo ""
echo "🎉 Successfully created 7 GitHub issues!"
echo ""
echo "Phase 1 (CI/CD Hardening): 4 issues"
echo "Phase 2 (Observability): 3 issues"
echo ""
echo "View all issues at: https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues"

---

🏁 Script executed:

#!/bin/bash

Create Phase 1 - Task 1: Add Coverage Reporting to CI

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add pytest coverage reporting to CI"
--label "good first issue,CI/CD,testing,documentation"
--assignee S3DFX-CYBER
--body '## 📊 Description

Enable code coverage measurement and reporting in our GitHub Actions CI workflow. The tools are already installed (pytest-cov in requirements-dev.txt) but not configured to run in CI or enforce coverage thresholds.

🎯 Acceptance Criteria

• CI workflow runs pytest with coverage flags
• Coverage report is uploaded to Codecov (or as artifact)
• Build fails if coverage drops below 70%
• CONTRIBUTING.md updated with coverage requirements and local testing instructions
• Verification command tested locally before pushing

🔨 Implementation

File to modify: .github/workflows/ci.yml

Replace the existing pytest step (around line 35-40) with:

      - name: Run unit tests with coverage
        run: |
          pytest tests/unit/ -v \
            --cov=services \
            --cov-report=xml \
            --cov-report=term-missing \
            --cov-fail-under=70

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          flags: unittests
          fail_ci_if_error: false
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Add to CONTRIBUTING.md under Testing section:

### Coverage Requirements

All contributions must maintain at least 70% test coverage. To check coverage locally:

\`\`\`bash
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70
\`\`\`

Coverage reports are automatically generated on pull requests via Codecov.

✅ Verification

# Run locally to verify it works before pushing
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70

📚 Files to Modify

• .github/workflows/ci.yml
• CONTRIBUTING.md

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 1: Add pytest coverage reporting to CI"

Create Phase 1 - Task 2: Enable Code Formatting and Linting

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Enable Black formatting and Ruff linting in CI"
--label "good first issue,CI/CD,code-quality"
--assignee S3DFX-CYBER
--body '## 📊 Description

Add Black formatting validation and Ruff linting as required CI checks to enforce code quality standards. The tools are already installed in requirements-dev.txt but not executed in CI.

🎯 Acceptance Criteria

• New lint job added to CI workflow
• Black formatting check runs on services/, tests/, scripts/
• Ruff linting runs on same directories
• pyproject.toml created/updated with Black and Ruff configuration
• Verification commands tested locally
• Documentation includes auto-fix commands

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add new job after the test job (around line 45):

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '\''3.11'\''
          cache: '\''pip'\''
      
      - name: Install linting tools
        run: pip install black ruff
      
      - name: Check formatting with Black
        run: black --check --line-length 100 services/ tests/ scripts/
      
      - name: Lint with Ruff
        run: ruff check services/ tests/ scripts/

Create pyproject.toml in repository root (or add these sections):

[tool.black]
line-length = 100
target-version = ['\''py311'\'']
include = '\''\.pyi?$'\''
extend-exclude = '\'''\'''\''
/(
  \.git
  | \.venv
  | __pycache__
  | models/trained
)/
'\'''\'''\''

[tool.ruff]
line-length = 100
target-version = "py311"
select = ["E", "F", "W", "I", "UP", "B"]
ignore = ["E501"]  # Line length handled by Black

[tool.ruff.per-file-ignores]
"tests/*" = ["B011"]  # Allow assert in tests

✅ Verification

# Check for issues
black --check --line-length 100 services/ tests/ scripts/
ruff check services/ tests/ scripts/

# Auto-fix issues if any
black --line-length 100 services/ tests/ scripts/
ruff check --fix services/ tests/ scripts/

📚 Files to Modify

• .github/workflows/ci.yml
• pyproject.toml (create if doesn'''t exist)

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 2: Enable Black formatting and Ruff linting in CI"

Create Phase 1 - Task 3: Enable Type Checking

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add mypy type checking to CI"
--label "good first issue,CI/CD,code-quality"
--assignee S3DFX-CYBER
--body '## 📊 Description

Add mypy static type analysis to catch type errors before runtime. This improves code quality and helps prevent bugs.

🎯 Acceptance Criteria

• mypy check added to the lint job in CI
• Type checking runs on services/ directory
• pyproject.toml updated with mypy configuration
• CI passes with current codebase
• Verification command tested locally

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add to the lint job steps:

      - name: Install type checking tools
        run: pip install mypy types-redis
      
      - name: Type check with mypy
        run: mypy services/ --ignore-missing-imports --no-error-summary

Add to pyproject.toml:

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_ignores = true
ignore_missing_imports = true
exclude = [
    "tests/",
    "scripts/",
    "models/",
]

[[tool.mypy.overrides]]
module = "redis.*"
ignore_missing_imports = true

✅ Verification

pip install mypy types-redis
mypy services/ --ignore-missing-imports

📚 Files to Modify

• .github/workflows/ci.yml
• pyproject.toml

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 3: Add mypy type checking to CI"

Create Phase 1 - Task 4: Integrate Security Scanning

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add Bandit and Safety security scanning to CI"
--label "good first issue,CI/CD,security"
--assignee S3DFX-CYBER
--body '## 📊 Description

Enable Bandit (SAST) and Safety (dependency vulnerability scanning) in the CI pipeline. The tools are already installed in requirements-dev.txt but not executed automatically.

🎯 Acceptance Criteria

• New security job added to CI workflow
• Bandit scans Python code for security issues
• Safety checks dependencies for known vulnerabilities
• CI passes with current codebase
• Verification commands tested locally

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add new job:

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '\''3.11'\''
          cache: '\''pip'\''
      
      - name: Install security tools
        run: pip install bandit safety
      
      - name: Run Bandit security scan
        run: bandit -r services/ -ll -ii -x tests/
      
      - name: Check dependencies for vulnerabilities
        run: safety check -r requirements.txt --full-report
        continue-on-error: true  # Don'\''t fail on advisory-only issues

✅ Verification

pip install bandit safety
bandit -r services/ -ll -ii -x tests/
safety check -r requirements.txt --full-report

📚 Files to Modify

• .github/workflows/ci.yml

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap

💡 Notes

• Bandit flags: -ll (medium+ severity), -ii (high confidence), -x tests/ (exclude tests)
• Safety uses continue-on-error: true to avoid failing on advisory-only issues'

echo "✅ Created Issue 4: Add Bandit and Safety security scanning to CI"

Create Phase 2 - Task 1: Structured JSON Logging

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Implement structured JSON logging with request IDs"
--label "good first issue,observability,enhancement"
--assignee S3DFX-CYBER
--body '## 📊 Description

Convert from basic text logging to structured JSON format with correlation IDs for request tracing. This is essential for production observability and debugging.

🎯 Acceptance Criteria

• python-json-logger added to requirements.txt
• services/common/logging.py module created with logging utilities
• services/ingest/app.py updated to use new logging
• Request ID middleware added to track requests
• Verification test shows JSON output with all expected fields
• Environment variables control log format (JSON vs text)

🔨 Implementation

Add to requirements.txt:

Length of output: 460

---

🏁 Script executed:
```
LOG_FORMAT=json python -c "
from services.common.logging import setup_logging
logger = setup_logging('\''test'\'')
logger.info('\''Test message'\'', extra={'\''custom_field'\'': '\''value'\''})
"
# Should output JSON with timestamp, level, service, etc.
```

Length of output: 159

---

🏁 Script executed:
```
# Start the service
python services/ingest/app.py

# In another terminal, test the endpoint
curl http://localhost:8000/metrics
# Should return Prometheus format metrics like:
# tenet_requests_total{service="ingest",endpoint="/metrics",method="GET",status="200"} 1.0
```

Length of output: 620

---

🏁 Script executed:
```
# Start the service
python services/ingest/app.py

# Test liveness (should always return 200)
curl -v http://localhost:8000/health/live

# Test readiness (returns 200 if ready, 503 if not)
curl -v http://localhost:8000/health/ready

# Test legacy endpoint
curl -v http://localhost:8000/health
```

Length of output: 2909

---



</details>

`@S3DFX-CYBER`, I've successfully created **4 out of 7 issues** for Phase 1 (CI/CD Hardening)! 🎉

## ✅ Successfully Created Issues (Phase 1)

The following issues were created in your repository:

1. **Add pytest coverage reporting to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
2. **Enable Black formatting and Ruff linting in CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
3. **Add mypy type checking to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
4. **Add Bandit and Safety security scanning to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)

## ⚠️ Minor Issues Encountered

- **Label warning**: The `CI/CD` label doesn't exist in your repository yet, so issues were created without that specific label. Other labels (`good first issue`, `testing`, `documentation`, `code-quality`, `security`) were applied successfully.
- **Phase 2 issues**: There was a syntax error in the script that prevented the 3 Phase 2 (Observability) issues from being created.

## 🔧 Next Steps

Would you like me to:

1. **Create the remaining 3 Phase 2 issues** (Structured JSON Logging, Prometheus Metrics, Enhanced Health Checks)?
2. **Create missing repository labels** so future issues can use them properly?
3. **View all created issues** to verify they look correct?

You can view all your issues at: https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues

Let me know how you'd like to proceed! 🚀


<!-- <agentComponentType>chat-answer</agentComponentType> -->

<!-- <rawResChunk><answer><details>
<summary>🧩 Analysis chain</summary>

`@S3DFX-CYBER`, I'll create the GitHub issues for Phase 1-2 good first issues right away!



python-json-logger>=2.0.7
```

**Create `services/common/__init__.py`:**

```python
# Empty file to make this a package
```

**Create `services/common/logging.py`:**

```python
import logging
import os
import sys
import uuid
from contextvars import ContextVar
from pythonjsonlogger import jsonlogger

# Context variable for request ID
request_id_var: ContextVar[str] = ContextVar("request_id", default="")

class TenetJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        log_record["service"] = os.getenv("SERVICE_NAME", "unknown")
        log_record["version"] = os.getenv("SERVICE_VERSION", "0.0.0")
        log_record["request_id"] = request_id_var.get()
        log_record["level"] = record.levelname
        log_record["timestamp"] = record.created

def setup_logging(service_name: str) -> logging.Logger:
    log_level = os.getenv("LOG_LEVEL", "INFO")
    log_format = os.getenv("LOG_FORMAT", "json")
    
    logger = logging.getLogger(service_name)
    logger.setLevel(log_level)
    
    handler = logging.StreamHandler(sys.stdout)
    
    if log_format == "json":
        formatter = TenetJsonFormatter("%(timestamp)s %(level)s %(name)s %(message)s")
    else:
        formatter = logging.Formatter(
            "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
        )
    
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    
    return logger

def generate_request_id() -> str:
    return str(uuid.uuid4())[:8]

def set_request_id(request_id: str):
    request_id_var.set(request_id)
```

**Update `services/ingest/app.py`:**

Replace the logging import and setup (lines 17-21) with:

```python
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))

from common.logging import setup_logging, generate_request_id, set_request_id

logger = setup_logging("ingest")

# Add middleware for request IDs
`@app.middleware`("http")
async def add_request_id(request: Request, call_next):
    request_id = request.headers.get("X-Request-ID", generate_request_id())
    set_request_id(request_id)
    response = await call_next(request)
    response.headers["X-Request-ID"] = request_id
    return response
```

## ✅ Verification



## 📚 Files to Create/Modify

- `requirements.txt` (add dependency)
- `services/common/__init__.py` (create)
- `services/common/logging.py` (create)
- `services/ingest/app.py` (update imports and add middleware)

## 🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

## 💡 Notes

- Environment variables: `LOG_LEVEL` (DEBUG/INFO/WARNING/ERROR), `LOG_FORMAT` (json/text)
- Request IDs enable tracing requests across services
- JSON logs are parseable by log aggregation tools (ELK, Splunk, etc.)'

echo "✅ Created Issue 5: Implement structured JSON logging with request IDs"

# Create Phase 2 - Task 2: Prometheus Metrics
gh issue create \
  --repo S3DFX-CYBER/AI-Cyber-Defender \
  --title "Add Prometheus metrics endpoint" \
  --label "good first issue,observability,enhancement" \
  --assignee S3DFX-CYBER \
  --body '## 📊 Description

Expose application metrics via a `/metrics` endpoint for Prometheus scraping. This enables monitoring of request rates, latencies, threat detection, and system health.

## 🎯 Acceptance Criteria

- [ ] `prometheus-client` added to `requirements.txt`
- [ ] `services/common/metrics.py` module created with metric definitions
- [ ] `/metrics` endpoint added to ingest service
- [ ] Metrics middleware tracks all requests
- [ ] Verification shows Prometheus-format output
- [ ] Common metrics defined: requests, latency, threats, queue depth

## 🔨 Implementation

**Add to `requirements.txt`:**

```
prometheus-client>=0.19.0
```

**Create `services/common/metrics.py`:**

```python
from prometheus_client import Counter, Histogram, Gauge, generate_latest, CONTENT_TYPE_LATEST
from fastapi import Response

# Request metrics
REQUEST_COUNT = Counter(
    '\''tenet_requests_total'\'',
    '\''Total HTTP requests'\'',
    ['\''service'\'', '\''endpoint'\'', '\''method'\'', '\''status'\'']
)

REQUEST_LATENCY = Histogram(
    '\''tenet_request_duration_seconds'\'',
    '\''HTTP request latency'\'',
    ['\''service'\'', '\''endpoint'\''],
    buckets=(.005, .01, .025, .05, .075, .1, .25, .5, .75, 1.0, 2.5, 5.0)
)

# Security metrics
THREATS_DETECTED = Counter(
    '\''tenet_threats_detected_total'\'',
    '\''Total threats detected'\'',
    ['\''threat_type'\'', '\''verdict'\'']
)

EVENTS_BLOCKED = Counter(
    '\''tenet_events_blocked_total'\'',
    '\''Total events blocked'\'',
    ['\''threat_type'\'']
)

# System metrics
QUEUE_DEPTH = Gauge(
    '\''tenet_queue_depth'\'',
    '\''Current queue depth'\'',
    ['\''queue_name'\'']
)

MODEL_INFERENCE_LATENCY = Histogram(
    '\''tenet_model_inference_seconds'\'',
    '\''ML model inference time'\'',
    ['\''model_type'\''],
    buckets=(.001, .005, .01, .025, .05, .1, .25, .5)
)

def metrics_endpoint():
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)
```

**Update `services/ingest/app.py`:**

Add imports:

```python
import time
from common.metrics import (
    REQUEST_COUNT, REQUEST_LATENCY, metrics_endpoint
)
```

Add metrics endpoint:

```python
`@app.get`("/metrics")
async def metrics():
    return metrics_endpoint()
```

Add metrics middleware:

```python
`@app.middleware`("http")
async def track_metrics(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    duration = time.time() - start_time
    
    REQUEST_COUNT.labels(
        service="ingest",
        endpoint=request.url.path,
        method=request.method,
        status=response.status_code
    ).inc()
    
    REQUEST_LATENCY.labels(
        service="ingest",
        endpoint=request.url.path
    ).observe(duration)
    
    return response
```

## ✅ Verification



## 📚 Files to Create/Modify

- `requirements.txt` (add dependency)
- `services/common/metrics.py` (create)
- `services/ingest/app.py` (add endpoint and middleware)

## 🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

## 💡 Notes

- Prometheus can scrape the `/metrics` endpoint automatically
- Histogram buckets are optimized for API latencies (5ms to 5s)
- Additional metrics can be added as needed (memory usage, cache hit rate, etc.)'

echo "✅ Created Issue 6: Add Prometheus metrics endpoint"

# Create Phase 2 - Task 3: Enhanced Health Checks
gh issue create \
  --repo S3DFX-CYBER/AI-Cyber-Defender \
  --title "Enhanced health check endpoints (liveness/readiness)" \
  --label "good first issue,observability,enhancement" \
  --assignee S3DFX-CYBER \
  --body '## 📊 Description

Expand health checks to include dependency status and separate liveness/readiness endpoints following Kubernetes best practices.

## 🎯 Acceptance Criteria

- [ ] `/health/live` endpoint added for liveness checks
- [ ] `/health/ready` endpoint added for readiness checks (with dependency checks)
- [ ] Legacy `/health` endpoint maintained for backward compatibility
- [ ] Redis connection check implemented
- [ ] Returns HTTP 503 when not ready
- [ ] Verification commands test all endpoints

## 🔨 Implementation

**Update `services/ingest/app.py`:**

Replace the existing `health_check` function with:

```python
from fastapi import Response
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from typing import Dict

class LivenessResponse(BaseModel):
    status: str
    service: str
    version: str

class ReadinessResponse(BaseModel):
    status: str
    service: str
    version: str
    checks: Dict[str, str]

`@app.get`("/health/live", response_model=LivenessResponse)
async def liveness():
    """Liveness probe - returns 200 if service is running"""
    return LivenessResponse(
        status="alive",
        service="ingest",
        version="0.1.0"
    )

`@app.get`("/health/ready", response_model=ReadinessResponse)
async def readiness():
    """Readiness probe - returns 200 if service can handle requests"""
    checks = {}
    overall_status = "ready"
    
    # Check Redis connection
    try:
        if redis_client:
            redis_client.ping()
            checks["redis"] = "connected"
        else:
            checks["redis"] = "not_configured"
            overall_status = "degraded"
    except Exception as e:
        checks["redis"] = f"error: {str(e)}"
        overall_status = "not_ready"
    
    response_data = ReadinessResponse(
        status=overall_status,
        service="ingest",
        version="0.1.0",
        checks=checks
    )
    
    status_code = 200 if overall_status in ["ready", "degraded"] else 503
    return JSONResponse(content=response_data.dict(), status_code=status_code)

`@app.get`("/health")
async def health_check():
    """Legacy health endpoint for backward compatibility"""
    ready_response = await readiness()
    return ready_response
```

## ✅ Verification



## 📚 Files to Modify

- `services/ingest/app.py`

## 🔗 Related

Part of Phase 2: Observability Foundation from the TENET AI roadmap

## 💡 Notes

- **Liveness**: Checks if the service process is running (simple, fast)
- **Readiness**: Checks if service can handle traffic (includes dependency checks)
- Kubernetes uses liveness for restarts, readiness for routing traffic
- HTTP 503 (Service Unavailable) signals orchestrator to stop routing traffic'

echo "✅ Created Issue 7: Enhanced health check endpoints (liveness/readiness)"

echo ""
echo "🎉 Successfully created 7 GitHub issues!"
echo ""
echo "Phase 1 (CI/CD Hardening): 4 issues"
echo "Phase 2 (Observability): 3 issues"
echo ""
echo "View all issues at: https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues"
```

---

🏁 Script executed:

#!/bin/bash

Create Phase 1 - Task 1: Add Coverage Reporting to CI

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add pytest coverage reporting to CI"
--label "good first issue,CI/CD,testing,documentation"
--assignee S3DFX-CYBER
--body '## 📊 Description

Enable code coverage measurement and reporting in our GitHub Actions CI workflow. The tools are already installed (pytest-cov in requirements-dev.txt) but not configured to run in CI or enforce coverage thresholds.

🎯 Acceptance Criteria

• CI workflow runs pytest with coverage flags
• Coverage report is uploaded to Codecov (or as artifact)
• Build fails if coverage drops below 70%
• CONTRIBUTING.md updated with coverage requirements and local testing instructions
• Verification command tested locally before pushing

🔨 Implementation

File to modify: .github/workflows/ci.yml

Replace the existing pytest step (around line 35-40) with:

      - name: Run unit tests with coverage
        run: |
          pytest tests/unit/ -v \
            --cov=services \
            --cov-report=xml \
            --cov-report=term-missing \
            --cov-fail-under=70

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          file: ./coverage.xml
          flags: unittests
          fail_ci_if_error: false
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

Add to CONTRIBUTING.md under Testing section:

### Coverage Requirements

All contributions must maintain at least 70% test coverage. To check coverage locally:

\`\`\`bash
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70
\`\`\`

Coverage reports are automatically generated on pull requests via Codecov.

✅ Verification

# Run locally to verify it works before pushing
pytest tests/unit/ -v --cov=services --cov-report=term-missing --cov-fail-under=70

📚 Files to Modify

• .github/workflows/ci.yml
• CONTRIBUTING.md

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 1: Add pytest coverage reporting to CI"

Create Phase 1 - Task 2: Enable Code Formatting and Linting

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Enable Black formatting and Ruff linting in CI"
--label "good first issue,CI/CD,code-quality"
--assignee S3DFX-CYBER
--body '## 📊 Description

Add Black formatting validation and Ruff linting as required CI checks to enforce code quality standards. The tools are already installed in requirements-dev.txt but not executed in CI.

🎯 Acceptance Criteria

• New lint job added to CI workflow
• Black formatting check runs on services/, tests/, scripts/
• Ruff linting runs on same directories
• pyproject.toml created/updated with Black and Ruff configuration
• Verification commands tested locally
• Documentation includes auto-fix commands

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add new job after the test job (around line 45):

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '\''3.11'\''
          cache: '\''pip'\''
      
      - name: Install linting tools
        run: pip install black ruff
      
      - name: Check formatting with Black
        run: black --check --line-length 100 services/ tests/ scripts/
      
      - name: Lint with Ruff
        run: ruff check services/ tests/ scripts/

Create pyproject.toml in repository root (or add these sections):

[tool.black]
line-length = 100
target-version = ['\''py311'\'']
include = '\''\.pyi?$'\''
extend-exclude = '\'''\'''\''
/(
  \.git
  | \.venv
  | __pycache__
  | models/trained
)/
'\'''\'''\''

[tool.ruff]
line-length = 100
target-version = "py311"
select = ["E", "F", "W", "I", "UP", "B"]
ignore = ["E501"]  # Line length handled by Black

[tool.ruff.per-file-ignores]
"tests/*" = ["B011"]  # Allow assert in tests

✅ Verification

# Check for issues
black --check --line-length 100 services/ tests/ scripts/
ruff check services/ tests/ scripts/

# Auto-fix issues if any
black --line-length 100 services/ tests/ scripts/
ruff check --fix services/ tests/ scripts/

📚 Files to Modify

• .github/workflows/ci.yml
• pyproject.toml (create if doesn'''t exist)

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 2: Enable Black formatting and Ruff linting in CI"

Create Phase 1 - Task 3: Enable Type Checking

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add mypy type checking to CI"
--label "good first issue,CI/CD,code-quality"
--assignee S3DFX-CYBER
--body '## 📊 Description

Add mypy static type analysis to catch type errors before runtime. This improves code quality and helps prevent bugs.

🎯 Acceptance Criteria

• mypy check added to the lint job in CI
• Type checking runs on services/ directory
• pyproject.toml updated with mypy configuration
• CI passes with current codebase
• Verification command tested locally

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add to the lint job steps:

      - name: Install type checking tools
        run: pip install mypy types-redis
      
      - name: Type check with mypy
        run: mypy services/ --ignore-missing-imports --no-error-summary

Add to pyproject.toml:

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_ignores = true
ignore_missing_imports = true
exclude = [
    "tests/",
    "scripts/",
    "models/",
]

[[tool.mypy.overrides]]
module = "redis.*"
ignore_missing_imports = true

✅ Verification

pip install mypy types-redis
mypy services/ --ignore-missing-imports

📚 Files to Modify

• .github/workflows/ci.yml
• pyproject.toml

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap'

echo "✅ Created Issue 3: Add mypy type checking to CI"

Create Phase 1 - Task 4: Integrate Security Scanning

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Add Bandit and Safety security scanning to CI"
--label "good first issue,CI/CD,security"
--assignee S3DFX-CYBER
--body '## 📊 Description

Enable Bandit (SAST) and Safety (dependency vulnerability scanning) in the CI pipeline. The tools are already installed in requirements-dev.txt but not executed automatically.

🎯 Acceptance Criteria

• New security job added to CI workflow
• Bandit scans Python code for security issues
• Safety checks dependencies for known vulnerabilities
• CI passes with current codebase
• Verification commands tested locally

🔨 Implementation

File to modify: .github/workflows/ci.yml

Add new job:

  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '\''3.11'\''
          cache: '\''pip'\''
      
      - name: Install security tools
        run: pip install bandit safety
      
      - name: Run Bandit security scan
        run: bandit -r services/ -ll -ii -x tests/
      
      - name: Check dependencies for vulnerabilities
        run: safety check -r requirements.txt --full-report
        continue-on-error: true  # Don'\''t fail on advisory-only issues

✅ Verification

pip install bandit safety
bandit -r services/ -ll -ii -x tests/
safety check -r requirements.txt --full-report

📚 Files to Modify

• .github/workflows/ci.yml

🔗 Related

Part of Phase 1: CI/CD Hardening from the TENET AI roadmap

💡 Notes

• Bandit flags: -ll (medium+ severity), -ii (high confidence), -x tests/ (exclude tests)
• Safety uses continue-on-error: true to avoid failing on advisory-only issues'

echo "✅ Created Issue 4: Add Bandit and Safety security scanning to CI"

Create Phase 2 - Task 1: Structured JSON Logging

gh issue create
--repo S3DFX-CYBER/AI-Cyber-Defender
--title "Implement structured JSON logging with request IDs"
--label "good first issue,observability,enhancement"
--assignee S3DFX-CYBER
--body '## 📊 Description

Convert from basic text logging to structured JSON format with correlation IDs for request tracing. This is essential for production observability and debugging.

🎯 Acceptance Criteria

• python-json-logger added to requirements.txt
• services/common/logging.py module created with logging utilities
• services/ingest/app.py updated to use new logging
• Request ID middleware added to track requests
• Verification test shows JSON output with all expected fields
• Environment variables control log format (JSON vs text)

🔨 Implementation

Add to requirements.txt:

Length of output: 460

---

🏁 Script executed:
```
LOG_FORMAT=json python -c "
from services.common.logging import setup_logging
logger = setup_logging('\''test'\'')
logger.info('\''Test message'\'', extra={'\''custom_field'\'': '\''value'\''})
"
# Should output JSON with timestamp, level, service, etc.
```

Length of output: 159

---

🏁 Script executed:
```
# Start the service
python services/ingest/app.py

# In another terminal, test the endpoint
curl http://localhost:8000/metrics
# Should return Prometheus format metrics like:
# tenet_requests_total{service="ingest",endpoint="/metrics",method="GET",status="200"} 1.0
```

Length of output: 620

---

🏁 Script executed:
```
# Start the service
python services/ingest/app.py

# Test liveness (should always return 200)
curl -v http://localhost:8000/health/live

# Test readiness (returns 200 if ready, 503 if not)
curl -v http://localhost:8000/health/ready

# Test legacy endpoint
curl -v http://localhost:8000/health
```

Length of output: 2909

---



</details>

`@S3DFX-CYBER`, I've successfully created **4 out of 7 issues** for Phase 1 (CI/CD Hardening)! 🎉

## ✅ Successfully Created Issues (Phase 1)

The following issues were created in your repository:

1. **Add pytest coverage reporting to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
2. **Enable Black formatting and Ruff linting in CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
3. **Add mypy type checking to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)
4. **Add Bandit and Safety security scanning to CI** - [View Issue](https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues)

## ⚠️ Minor Issues Encountered

- **Label warning**: The `CI/CD` label doesn't exist in your repository yet, so issues were created without that specific label. Other labels (`good first issue`, `testing`, `documentation`, `code-quality`, `security`) were applied successfully.
- **Phase 2 issues**: There was a syntax error in the script that prevented the 3 Phase 2 (Observability) issues from being created.

## 🔧 Next Steps

Would you like me to:

1. **Create the remaining 3 Phase 2 issues** (Structured JSON Logging, Prometheus Metrics, Enhanced Health Checks)?
2. **Create missing repository labels** so future issues can use them properly?
3. **View all created issues** to verify they look correct?

You can view all your issues at: https://github.com/S3DFX-CYBER/AI-Cyber-Defender/issues

Let me know how you'd like to proceed! 🚀</answer></rawResChunk> -->