Model Evaluation Agent

A LangGraph-based coding agent with sandbox code execution, powered by Modal.

Quick Start

1. Clone & Install

git clone https://github.com/your-username/model-eval-agent.git
cd model-eval-agent

2. Install Modal CLI

pip install modal

3. Authenticate Modal

modal token new

This opens a browser for authentication. After login, your token is saved locally.

4. Create Modal Secrets

Go to Modal Secrets Dashboard and create the following secrets:

Secret Name	Key	Value
openai-secret	OPENAI_API_KEY	Your OpenAI API key
google-api-secret	GOOGLE_API_KEY	Your Google AI API key

How to create a secret:

1. Click "Create new secret"
2. Choose "Custom" type
3. Enter the secret name (e.g., openai-secret)
4. Add environment variable: key = OPENAI_API_KEY, value = your key
5. Click "Create"

5. Install Backend Dependencies

cd backend
pip install uv  # if not installed
uv sync

6. Deploy to Modal

modal deploy -m backend.app

After deployment, you'll see output like:

✓ Created web endpoint https://your-username--model-eval-app-web.modal.run

This is your Modal API URL! 🎉

7. Configure Frontend

Create frontend/.env.local:

NEXT_PUBLIC_API_URL=https://your-username--model-eval-app-web.modal.run

8. Run Frontend

cd frontend
npm install
npm run dev

Visit http://localhost:3000 to use the agent.

---

API Endpoints

Endpoint	Method	Description
/ask/stream	POST	Stream agent responses
/health	GET	Health check
/files	GET	List available files

Development

Run Backend Locally (for testing)

modal run -m backend.app --question "Hello"

Debug Mode

modal run -m backend.app --debug --question "Test"

Architecture

┌─────────────────────────────────────────────────┐
│                   Frontend                       │
│              (Next.js + React)                   │
└─────────────────────┬───────────────────────────┘
                      │ HTTP/SSE
                      ▼
┌─────────────────────────────────────────────────┐
│                 Modal App                        │
│  ┌───────────────────────────────────────────┐  │
│  │            LangGraph Agent                │  │
│  │  [generate] → [execute_tools] → [finish]  │  │
│  └───────────────────────────────────────────┘  │
│                      │                           │
│                      ▼                           │
│  ┌───────────────────────────────────────────┐  │
│  │          Modal Sandbox                    │  │
│  │    (Isolated Python Execution)            │  │
│  └───────────────────────────────────────────┘  │
└─────────────────────────────────────────────────┘

Troubleshooting

"Modal token not found"

Run modal token new to authenticate.

"Secret not found"

Create secrets at https://modal.com/secrets with exact names: openai-secret, google-api-secret.

"Deployment failed"

Check Modal logs: modal app logs model-eval-app

Frontend can't connect

Verify NEXT_PUBLIC_API_URL in .env.local matches your deployed Modal URL.