feat: add Claude-driven agentic tuner as an optional extra

[aas008]

Port the TOA hackathon agent into auto-tune-vllm as auto-tune-vllm agent, without changing the Optuna optimizer.

Summary by CodeRabbit

• New Features

  • Added an optional Claude-driven vLLM tuning agent with automated benchmarking, profiling, analysis, and optimization workflows.
  • Added support for SSH and OpenShift execution, including isolated experiment pods.
  • Added trace analysis, kernel mapping, regression detection, cost-efficiency analysis, and vLLM log parsing.
  • Added Markdown and JSON tuning reports with performance, cost, profiling, and usage summaries.
  • Added a new agent CLI command and configurable workload profiles.

• Documentation

  • Added setup, usage, configuration, and experiment-pod guidance for the agentic tuner.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 07bbe4a2-f828-4409-8cfc-55fded1a0cd8

📥 Commits

Reviewing files that changed from the base of the PR and between acff360 and 62d70e6.

📒 Files selected for processing (26)

• .gitignore
• README.md
• auto_tune_vllm/agent/__init__.py
• auto_tune_vllm/agent/__main__.py
• auto_tune_vllm/agent/agentic.py
• auto_tune_vllm/agent/analysis/__init__.py
• auto_tune_vllm/agent/analysis/cost.py
• auto_tune_vllm/agent/analysis/kernel_mapper.py
• auto_tune_vllm/agent/analysis/regression.py
• auto_tune_vllm/agent/analysis/trace_analyzer.py
• auto_tune_vllm/agent/analysis/vllm_log_parser.py
• auto_tune_vllm/agent/llm.py
• auto_tune_vllm/agent/main.py
• auto_tune_vllm/agent/pod_manager.py
• auto_tune_vllm/agent/profiler/profiler_config.yaml
• auto_tune_vllm/agent/profiler/sitecustomize.py
• auto_tune_vllm/agent/reporter.py
• auto_tune_vllm/agent/settings.yaml
• auto_tune_vllm/agent/ssh_client.py
• auto_tune_vllm/agent/tools.py
• auto_tune_vllm/cli/agent_cmd.py
• auto_tune_vllm/cli/main.py
• docs/agentic_tuning.md
• examples/agent/README.md
• examples/agent/experiment-pod.yaml
• pyproject.toml

---

📝 Walkthrough

Walkthrough

The pull request adds an optional Claude-driven vLLM tuning agent. It includes autonomous benchmarking, OpenShift experiment pods, SSH execution, profiling, trace and log analysis, cost and regression analysis, CLI integration, and Markdown/JSON reporting.

Changes

Agentic vLLM tuner

Layer / File(s)	Summary
Package, configuration, and CLI foundation .gitignore, README.md, auto_tune_vllm/agent/*, auto_tune_vllm/cli/*, docs/agentic_tuning.md, examples/agent/*, pyproject.toml	Adds optional dependencies, default tuning settings, package entry points, documentation, a pod template, ignored artifact directories, and the agent CLI command.
Claude client and autonomous loop auto_tune_vllm/agent/llm.py, auto_tune_vllm/agent/agentic.py	Adds Claude and Vertex AI access, token usage tracking, tool-aware conversations, iteration control, decision logs, completion handling, and fallback summaries.
Remote execution and benchmark tools auto_tune_vllm/agent/ssh_client.py, auto_tune_vllm/agent/tools.py	Adds SSH and OpenShift command execution, file operations, GuideLLM benchmarks, log collection, benchmark comparison, preemption checks, and tool dispatch.
Isolated experiment pod lifecycle auto_tune_vllm/agent/pod_manager.py, auto_tune_vllm/agent/tools.py	Creates, monitors, port-forwards, deletes, and cleans up per-experiment vLLM pods.
Benchmark and kernel analysis auto_tune_vllm/agent/analysis/*, auto_tune_vllm/agent/tools.py	Adds trace analysis, kernel mapping, vLLM log parsing, cost calculation, SLO filtering, and regression detection.
Worker profiling hook auto_tune_vllm/agent/profiler/*	Adds configurable PyTorch profiling around vLLM worker execution and exports Chrome traces and profiling statistics.
CLI execution and report generation auto_tune_vllm/agent/main.py, auto_tune_vllm/agent/reporter.py	Adds runtime validation, benchmark result extraction, cleanup handling, and Markdown/JSON reports containing results, analyses, actions, costs, and token usage.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant AgenticRunner
  participant ClaudeClient
  participant AgentTools
  participant PodManager
  participant vLLM
  participant Reporter

  CLI->>AgenticRunner: start tuning run
  AgenticRunner->>ClaudeClient: send baseline and tuning instructions
  ClaudeClient-->>AgenticRunner: return tool call
  AgenticRunner->>AgentTools: dispatch benchmark or tuning action
  AgentTools->>PodManager: create isolated experiment pod
  PodManager->>vLLM: start pod and port-forward endpoint
  AgentTools->>vLLM: run benchmark
  vLLM-->>AgentTools: return throughput and latency metrics
  AgentTools-->>AgenticRunner: return structured result
  AgenticRunner->>ClaudeClient: analyze result and request next action
  AgenticRunner-->>CLI: return AgentState
  CLI->>Reporter: generate Markdown and JSON reports

Loading

Possibly related PRs

• openshift-psap/auto-tuning-vllm#20: Adds benchmark, trial, configuration, execution, and CLI foundations used by this agentic tuning workflow.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/agentic-tuning

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}