🔧 LM Hack Proxy

Intelligent LLM Orchestration Proxy for ECOS CLI Integration

A sophisticated REST API server that provides intelligent model selection, multi-agent orchestration, and performance optimization for large language model queries. Designed specifically to support the advanced AI orchestration capabilities of ECOS CLI v2.1.0+.

🚀 Features

Core Capabilities

• Intelligent Model Routing: Automatic selection of optimal LLM models based on task requirements
• Multi-Agent Orchestration: Support for sequential, parallel, hierarchical, and collaborative agent workflows
• Multi-Provider Support: Native integration with Anthropic, OpenAI, xAI, Meta, Google, and Mistral AI
• Real-time Performance Monitoring: Comprehensive metrics collection and analytics
• Dynamic Configuration: Runtime model registration and policy updates
• Production Ready: Built with FastAPI, async operations, and robust error handling

Supported Models

• Anthropic: Claude 3.5 Sonnet, Claude 3 Opus
• OpenAI: GPT-4, GPT-3.5 Turbo
• xAI: Grok Beta
• Meta: Llama 3.2 70B
• Google: Gemini Pro
• Mistral AI: Mistral 7B Instruct
• Alibaba: Qwen 2.5 72B

📋 API Endpoints

Core Endpoints

GET    /api/models           # List available models with metadata
POST   /api/query            # Single query with intelligent routing
POST   /api/batch-query      # Batch processing capabilities
GET    /api/metrics          # Real-time performance metrics
POST   /api/config/models    # Dynamic model configuration
GET    /api/status           # System health and status
GET    /api/policies/routing # Get routing policies
POST   /api/policies/routing # Update routing policies
POST   /api/agents/orchestrate # Multi-agent orchestration
GET    /health               # Health check

Example Usage

Single Query

curl -X POST "http://localhost:4000/api/query" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Analyze this Python function for potential bugs",
    "task_metadata": {
      "task_type": "code_analysis",
      "complexity": "medium",
      "context_size": 2048,
      "priority": "normal"
    }
  }'

Multi-Agent Orchestration

curl -X POST "http://localhost:4000/api/agents/orchestrate" \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "hierarchical",
    "task": "Design a complete e-commerce system architecture",
    "task_metadata": {
      "task_type": "code_generation",
      "complexity": "complex"
    },
    "agents": [
      {
        "agent_id": "architect",
        "role": "system_architect",
        "expertise": ["system_design", "scalability"]
      },
      {
        "agent_id": "backend_dev",
        "role": "backend_developer",
        "expertise": ["api_design", "database"]
      },
      {
        "agent_id": "frontend_dev",
        "role": "frontend_developer",
        "expertise": ["ui_ux", "react"]
      }
    ]
  }'

🏗️ Architecture

vscode-lm-proxy/
├── server.py              # FastAPI server with all endpoints
├── src/
│   ├── models/
│   │   ├── registry.py    # Model management and metadata
│   │   └── router.py      # Intelligent task routing engine
│   ├── agents/
│   │   └── coordinator.py # Multi-agent orchestration
│   └── utils/
│       ├── providers/
│       │   └── base.py    # Provider abstraction layer
│       └── monitoring/
│           └── metrics.py # Performance monitoring
├── config/
│   ├── models.json        # Model configurations
│   └── policies.yaml      # Routing policies
├── tests/                 # Test suite
├── requirements.txt       # Python dependencies
└── README.md             # This file

🚀 Quick Start

Prerequisites

• Python 3.9+
• API keys for desired LLM providers (optional for mock mode)

Installation

1. Clone and setup:

cd vscode-lm-proxy
pip install -r requirements.txt

2. Configure environment (optional):

# Create .env file for API keys
cp .env.example .env
# Edit .env with your API keys

3. Start the server:

python server.py

The server will start on http://localhost:4000 with automatic API documentation at http://localhost:4000/docs.

Docker Deployment

# Build and run with Docker
docker build -t lm-hack-proxy .
docker run -p 4000:4000 lm-hack-proxy

⚙️ Configuration

Environment Variables

# Server configuration
LM_HACK_PORT=4000
LM_HACK_HOST=0.0.0.0

# Provider API keys (optional)
ANTHROPIC_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
XAI_API_KEY=your_key_here
# ... etc

Model Configuration

Models are automatically configured with sensible defaults. Custom configurations can be added via the /api/config/models endpoint or by editing config/models.json.

Routing Policies

Default routing policies prioritize models based on task types. Policies can be customized via the /api/policies/routing endpoint.

🧪 Testing

Run Tests

# Install test dependencies
pip install -r requirements.txt

# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=html

Mock Mode

The proxy runs in mock mode by default, generating realistic responses without requiring API keys. This is perfect for development and testing.

📊 Monitoring & Metrics

Real-time Metrics

• Request latency and throughput
• Model performance statistics
• Cost tracking per request
• Success/failure rates
• Agent orchestration metrics

Health Checks

curl http://localhost:4000/health
curl http://localhost:4000/api/status

Performance Dashboard

Access metrics via:

curl http://localhost:4000/api/metrics
curl http://localhost:4000/api/metrics?model_name=claude-3-5-sonnet-20241022

🔧 Development

Code Structure

• server.py: Main FastAPI application with all endpoints
• src/models/: Model registry and intelligent routing
• src/agents/: Multi-agent coordination logic
• src/utils/: Provider abstractions and monitoring tools

Adding New Providers

1. Create a new provider class inheriting from BaseProvider
2. Implement the required abstract methods
3. Register the provider in the model registry
4. Add configuration support

Extending Agent Strategies

Agent orchestration strategies are defined in AgentCoordinator. To add new strategies:

1. Implement the strategy logic in a new method
2. Add the strategy name to the main orchestration method
3. Update documentation and tests

🔗 Integration with ECOS CLI

This proxy is specifically designed to work with ECOS CLI v2.1.0+. The integration provides:

• LMHackClient: Direct REST client for single queries
• ModelOrchestrator: Intelligent model selection across providers
• GrokAgent: Specialized agent framework
• AgentMixtures: Complex multi-agent workflows

ECOS CLI Configuration

# In ECOS CLI configuration
lm_hack = {
    'base_url': 'http://localhost:4000',
    'timeout': 300,
    'retry_attempts': 3
}

📈 Performance

Benchmarks (Mock Mode)

• Average Latency: < 2.5 seconds for typical queries
• Concurrent Requests: Supports 100+ simultaneous requests
• Memory Usage: < 200MB base footprint
• Routing Accuracy: > 90% optimal model selection

Production Considerations

• Horizontal scaling with load balancer
• Redis for distributed caching
• Database for persistent metrics storage
• Rate limiting and request queuing

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

Development Guidelines

• Use async/await for all I/O operations
• Add comprehensive type hints
• Include docstrings for all public methods
• Write tests for new features
• Follow PEP 8 style guidelines

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built for the ECOS CLI ecosystem
• Inspired by modern LLM orchestration patterns
• Designed for production AI workflows

---

For more information, visit the API documentation at http://localhost:4000/docs when the server is running.