Feature Request: InfraGPT Skills System - Automated Runbook Execution

[priyanshujain]

🎯 Problem Statement

Support agents and engineers need a way to quickly diagnose and resolve common infrastructure issues without manual runbook navigation. When tagged on Slack, InfraGPT should automatically detect problems and provide structured, actionable debugging information through executable skills.

📋 User Story

As a DevOps Engineer/Support Agent
I want to create executable skills (runbooks) that automatically trigger when issues are detected
So that I can get immediate, detailed debugging information and remediation steps when problems occur

Acceptance Criteria:

• Users can @mention InfraGPT in Slack with a problem description
• InfraGPT analyzes the message and suggests/executes relevant skills
• Skills provide structured diagnostic information and remediation steps
• Results are posted back to the original Slack thread
• Skills are scoped to organizations with proper permissions

🔧 Technical Architecture

System Components

1. Skill Definition System

   • JSON-based skill schema with metadata, triggers, parameters, and execution steps
   • Support for multiple categories: incident-response, monitoring, cost-optimization, security, etc.
   • Versioning and lifecycle management

2. Skill Execution Engine

   • Pattern matching for automatic skill suggestion
   • Parameter extraction from natural language
   • Step-by-step execution with error handling and retries
   • Real-time progress updates to Slack

3. Integration Layer

   • Slack bot integration for triggering and feedback
   • LLM-powered skill recommendation and parameter extraction
   • External service handlers (GCP, AWS, Datadog, PagerDuty, etc.)

4. Management Interface

   • Web-based skill builder with templates
   • Organization-scoped skill enablement
   • Execution history and analytics

Database Schema

-- Core skill storage
CREATE TABLE skills (
    id UUID PRIMARY KEY,
    organization_id UUID NOT NULL,
    name VARCHAR(64) NOT NULL,
    description TEXT NOT NULL,
    definition JSONB NOT NULL,
    status VARCHAR(20) DEFAULT 'draft',
    created_at TIMESTAMP DEFAULT NOW(),
    UNIQUE(organization_id, name)
);

-- Skill executions tracking
CREATE TABLE skill_executions (
    id UUID PRIMARY KEY,
    skill_id UUID NOT NULL,
    conversation_id UUID,
    triggered_by_user_id UUID,
    parameters JSONB,
    status VARCHAR(20) DEFAULT 'running',
    started_at TIMESTAMP DEFAULT NOW(),
    completed_at TIMESTAMP,
    result JSONB,
    execution_trace JSONB
);

-- Additional tables for enablement, permissions, analytics

Skill Definition Schema

{
  "metadata": {
    "name": "gcp-disk-cleanup",
    "display_name": "GCP Disk Space Cleanup",
    "description": "Automated cleanup of disk space on GCP instances",
    "category": "infrastructure",
    "version": "1.0.0"
  },
  "triggers": {
    "manual": {
      "enabled": true,
      "keywords": ["disk space", "storage full", "cleanup"]
    },
    "automatic": {
      "alert_patterns": [
        {
          "source": "monitoring",
          "pattern": "disk.*usage.*critical"
        }
      ]
    }
  },
  "parameters": {
    "required": [
      {
        "name": "instance_id",
        "type": "string",
        "description": "GCP instance identifier"
      }
    ]
  },
  "execution": {
    "steps": [
      {
        "id": "diagnose",
        "type": "diagnostic",
        "action": {
          "handler": "gcp_command",
          "template": "gcloud compute ssh {{instance_id}} --command='df -h'"
        }
      },
      {
        "id": "cleanup",
        "type": "remediation",
        "action": {
          "handler": "gcp_command",
          "template": "gcloud compute ssh {{instance_id}} --command='sudo apt-get clean && sudo rm -rf /tmp/*'"
        }
      }
    ]
  }
}

🚀 User Experience Flow

1. Skill Creation

Engineer creates skill via web UI:
1. Choose template (incident-response, monitoring, etc.)
2. Define triggers (keywords, alert patterns)
3. Set parameters (instance IDs, thresholds, etc.)
4. Build execution steps (diagnostic → remediation)
5. Test skill in sandbox environment
6. Publish to organization

2. Skill Execution

Support agent in Slack:
1. Types: "@infragpt our prod-web-01 server is running out of disk space"
2. InfraGPT analyzes message and suggests: "disk-cleanup" skill
3. Extracts parameters: instance_id="prod-web-01"
4. Executes skill steps with real-time updates:
   - ✅ Checking disk usage: 87% full
   - 🔄 Cleaning temporary files...
   - ✅ Cleanup complete: 87% → 62%
5. Posts formatted results with action buttons

3. Skill Management

Org admin dashboard:
1. View all available skills
2. Enable/disable skills per team
3. Monitor execution history and success rates
4. Configure permissions and approval workflows

📊 API Endpoints

# Skill Management
GET    /api/v1/orgs/{orgId}/skills
POST   /api/v1/orgs/{orgId}/skills
PUT    /api/v1/orgs/{orgId}/skills/{skillId}
DELETE /api/v1/orgs/{orgId}/skills/{skillId}

# Skill Execution
POST   /api/v1/orgs/{orgId}/skills/{skillId}/execute
GET    /api/v1/orgs/{orgId}/executions
GET    /api/v1/orgs/{orgId}/executions/{executionId}

# Skill Discovery
GET    /api/v1/orgs/{orgId}/skills/search
POST   /api/v1/orgs/{orgId}/skills/suggest

🛠️ Implementation Plan

Phase 1: Core Infrastructure (4-6 weeks)

• Database schema implementation
• Skill definition JSON schema and validation
• Basic CRUD operations for skills
• Skill execution engine foundation
• gRPC service definitions

Phase 2: Skill Execution (4-6 weeks)

• Step handler implementations (GCP, AWS, HTTP, etc.)
• Parameter extraction and validation
• Execution orchestration with error handling
• Retry logic and timeout management
• Execution logging and monitoring

Phase 3: Slack Integration (3-4 weeks)

• Message pattern matching for skill triggers
• LLM-powered skill suggestion
• Parameter extraction from natural language
• Real-time execution updates in Slack threads
• Formatted result presentation

Phase 4: Web Interface (4-5 weeks)

• React-based skill builder interface
• Template system for common scenarios
• Execution monitoring dashboard
• Analytics and reporting
• Organization management

Phase 5: Advanced Features (3-4 weeks)

• AI-assisted skill generation from descriptions
• Automated triggers from monitoring alerts
• Approval workflows for high-risk operations
• Skill versioning and rollback
• Performance optimization

🔐 Security Considerations

• Organization-scoped skill isolation
• Role-based access control for skill execution
• Secure credential management for external services
• Audit trail for all skill executions
• Approval workflows for high-risk operations
• Input validation and sanitization

📈 Success Metrics

• Time to resolve common issues (target: 50% reduction)
• Skill adoption rate across organizations
• Execution success rate (target: >95%)
• User satisfaction with automated responses
• Reduction in manual runbook lookups

🧪 Testing Strategy

• Unit tests for skill execution engine
• Integration tests with external services
• End-to-end tests with Slack bot
• Load testing for concurrent executions
• Security testing for permission boundaries

📝 Documentation Requirements

• Skill creation guide with examples
• API documentation for developers
• Integration guide for external services
• Troubleshooting guide for common issues
• Best practices for skill design

🔄 Migration Strategy

• Gradual rollout to select organizations
• Backward compatibility with existing chat flows
• Data migration for existing runbooks
• Training materials for early adopters
• Feedback collection and iteration

🎯 Related Issues

• Connects to existing conversation service architecture
• Builds upon current LLM module integration
• Extends organization-scoped operations model
• Leverages existing integration framework

💡 Future Enhancements

• Visual workflow builder for complex skills
• Skill marketplace for sharing across organizations
• Machine learning for skill optimization
• Integration with ITSM tools (ServiceNow, Jira)
• Mobile app support for skill execution

---

🤖 Generated with Claude Code

[priyanshujain]

Runbook Automation

Overview

An automated support system that analyzes GitHub issues labeled as "support" and executes appropriate runbooks based on the issue content, providing immediate diagnostic responses and remediation steps.

Workflow Architecture

1. Issue Analysis Workflow (Trigger: Issue Creation)

• Trigger: When an issue is opened with "support" label
• Process:
  • Analyzes issue title and body content
  • Uses keyword matching or AI classification to determine issue category
  • Automatically adds appropriate runbook label (e.g., runbook-database-issues, runbook-network-issues)
• Output: Issue gets labeled with specific runbook identifier

2. Runbook Executor Workflow (Trigger: Label Addition)

• Trigger: When any label starting with "runbook-" is added to an issue
• Process:
  • Single workflow file with conditional steps
  • Determines runbook type from label name
  • Executes appropriate diagnostic steps based on runbook type
  • Posts results and recommendations back to the issue as comments
• Output: Automated diagnostic report and action items

Supported Runbook Categories

• Database Issues: Connectivity checks, status verification, performance diagnostics
• Network Issues: Ping tests, port connectivity, DNS resolution, trace routes
• Deployment Issues: Build status, deployment health, rollback procedures
• General Support: Catch-all for unclassified issues

Key Benefits

1. Immediate Response: Support issues get instant automated analysis
2. Consistent Diagnostics: Standardized runbook procedures ensure nothing is missed
3. Clean Action History: Only relevant workflows execute (no spam in Actions tab)
4. Scalable: Easy to add new runbook categories
5. Traceable: Full audit trail of automated actions
6. Interactive: Results posted directly to the issue for team visibility

Technical Implementation

• Two GitHub Action workflows: Analysis trigger + Runbook executor
• Label-based routing: Uses GitHub labels to determine which runbook to execute
• Conditional execution: Single workflow with dynamic step execution
• Issue commenting: Automated posting of diagnostic results and next steps

User Experience

1. User creates issue with "support" label
2. System automatically analyzes and categorizes the issue
3. Appropriate runbook executes within minutes
4. User receives diagnostic results and recommended actions as issue comments
5. Support team can immediately see automated analysis and focus on resolution

This creates a self-service diagnostic system that reduces response time and ensures consistent troubleshooting procedures across all support requests.

[priyanshujain]

We can use custom repository dispatch to trigger workflow

https://docs.github.com/en/actions/reference/events-that-trigger-workflows#repository_dispatch

[priyanshujain]

User: "Deploy the API service to staging"
↓
InfraGPT: Analyzes → Selects skills: ["check-staging-health", "build-api-image", "deploy-to-k8s"]
↓
Triggers each skill via webhook → Individual skill execution (GitHub Actions-like)