Claude Code Evaluation Example (#6)

Closes: elastic/obs-ai-team#642

### Summary
- Added Claude Code evaluation suite 
   - Asking Claude questions about this codebase
   - Sample evaluators for tool use, criteria, latency
- Instrumenting Claude Code with OpenTelemetry and exporting via EDOT
collector
 - Capturing task traces and visualizing using Evals plugin in Kibana 

<img width="1477" height="833" alt="image"
src="https://github.com/user-attachments/assets/dc696ea6-b663-4c8b-ba32-fda7f1cfaa07"
/>

### Testing

```bash
  CONNECTOR_ID="azure-gpt4_1" \
  KIBANA_URL="http://elastic:changeme@host.docker.internal:5601/dev" \
  EDOT_ENDPOINT="http://kibana-edot-collector:4318" \
  ELASTIC_EVALS_TRACING_ENDPOINT="http://kibana-edot-collector:4318/v1/traces" \
  TRACE_ES_URL="http://elastic:changeme@host.docker.internal:9200" \
  uv run elastic-evals run --suite claude-code-eval
```

Author: SrdjanLL

.devcontainer/devcontainer.json

@@ -0,0 +1,12 @@
+{
+  "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
+  "features": {
+    "ghcr.io/devcontainers/features/node:1": {},
+    "ghcr.io/devcontainers/features/python:1": {
+        "version": "3.13"
+    },
+    "ghcr.io/anthropics/devcontainer-features/claude-code:1.0": {},
+    "ghcr.io/jsburckhardt/devcontainer-features/uv:1": {},
+    "ghcr.io/jsburckhardt/devcontainer-features/ruff:1": {}
+  }
+}

examples/claude_code_eval/README.md

@@ -0,0 +1,88 @@
+# Claude Code Eval
+
+Evaluates the Claude Code CLI as a task target. Each dataset example is a prompt sent
+to `claude --print --output-format stream-json`. The eval harness captures:
+
+- **Latency** — wall-clock time for the subprocess to complete
+- **Tool use** — which tools Claude Code actually called vs. expected
+- **Response quality** — optional LLM criteria scoring (requires a Kibana connector)
+
+Claude Code's own telemetry (metrics + logs) is forwarded to a local EDOT collector
+so you can observe both sides — the eval harness and Claude Code itself — in Kibana APM.
+
+## Prerequisites
+
+1. Running Elasticsearch and Kibana (local stack is fine).
+2. EDOT collector listening at `http://localhost:4318` (HTTP/protobuf).
+3. `claude` CLI installed and authenticated (`claude --version`).
+4. A Kibana LLM connector for dataset storage and optional criteria scoring.
+
+## Run the eval
+
+```bash
+CONNECTOR_ID="your-connector-id" \
+KIBANA_URL="http://elastic:changeme@localhost:5601" \
+EDOT_ENDPOINT="http://localhost:4318" \
+uv run elastic-evals run --suite claude-code-eval
+```
+
+Or run directly:
+
+```bash
+  CONNECTOR_ID="azure-gpt4_1" \
+  KIBANA_URL="http://elastic:changeme@host.docker.internal:5601/dev" \
+  EDOT_ENDPOINT="http://kibana-edot-collector:4318" \
+  ELASTIC_EVALS_TRACING_ENDPOINT="http://kibana-edot-collector:4318/v1/traces" \
+  TRACE_ES_URL="http://elastic:changeme@host.docker.internal:9200" \
+  uv run elastic-evals run --suite claude-code-eval
+```
+
+## Telemetry flow
+
+```
+elastic-evals harness
+  │  spans → OTLP/HTTP → localhost:4318 (EDOT)
+  │
+  └─ spawns: claude --print --output-format stream-json
+               │  metrics → OTLP/HTTP → localhost:4318 (EDOT)
+               │  logs    → OTLP/HTTP → localhost:4318 (EDOT)
+               └─ (spans if trace propagation works — see below)
+```
+
+Both services share the same `elastic.evals.run_id` resource attribute, so you can
+filter by run ID in APM / Discover to correlate all signals from a single eval run.
+
+## Trace propagation
+
+The harness injects the current OTel span context as `TRACEPARENT` and `TRACESTATE`
+environment variables before spawning each Claude Code subprocess.
+
+**Current status**: Claude Code's Node.js OTel SDK reads trace context from HTTP headers,
+not from environment variables, so strict parent-child linking does not work out of the box.
+The `TRACEPARENT` env var is set anyway — if a future Claude Code release adds an env-var
+propagator, spans will automatically appear as children in the eval trace.
+
+**What works today**: correlation via `elastic.evals.run_id`. In APM, filter for:
+
+```
+resource.attributes.elastic.evals.run_id: "<your-run-id>"
+```
+
+This surfaces spans from both `elastic-evals` and `claude-code` services for the same run.
+
+## Evaluators
+
+| Name | Kind | What it scores |
+|------|------|----------------|
+| `Latency` | CODE | Wall-clock ms → FAST / GOOD / OK / SLOW / VERY_SLOW |
+| `ToolUse` | CODE | Overlap between expected and actual tool names |
+| `criteria` | LLM | Per-example criteria from dataset metadata (optional) |
+
+## Tuning
+
+| Env var | Default | Purpose |
+|---------|---------|---------|
+| `EDOT_ENDPOINT` | `http://localhost:4318` | OTLP endpoint for both harness and Claude Code |
+| `ELASTIC_EVALS_REPETITIONS` | `1` | Number of times to repeat each example |
+| `ELASTIC_EVALS_CONCURRENCY` | `5` | Parallel task slots |
+| `EVALUATION_CONNECTOR_ID` | — | Separate connector for LLM scoring |

examples/claude_code_eval/__init__.py

@@ -0,0 +1,74 @@
+"""Coding tasks dataset for Claude Code eval."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from elastic_evals.types import EvaluationDataset, Example
+
+CodingTaskExample = Example[dict[str, str], None, dict[str, Any]]
+
+coding_tasks_dataset: EvaluationDataset[CodingTaskExample] = EvaluationDataset(
+    name="claude-code-eval: coding-tasks",
+    description="Tasks that exercise Claude Code's code generation and tool use capabilities.",
+    examples=[
+        Example(
+            input={
+                "prompt": (
+                    "Write a Python function called `validate_email` that checks whether a "
+                    "string is a valid email address using only the standard library. "
+                    "Use type hints and return True if valid, False otherwise."
+                )
+            },
+            output=None,
+            metadata={
+                "expected_tools": [],
+                "task_type": "code_generation",
+                "criteria": [
+                    "The response includes a Python function named validate_email",
+                    "The function uses type hints (str -> bool or similar)",
+                    "The function uses only standard library modules",
+                    "The function handles basic valid and invalid email formats",
+                ],
+            },
+        ),
+        Example(
+            input={
+                "prompt": (
+                    "List the top-level Python files in the current working directory "
+                    "and briefly describe the purpose of each one based on its content."
+                )
+            },
+            output=None,
+            metadata={
+                "expected_tools": ["bash", "read_file"],
+                "task_type": "file_exploration",
+                "criteria": [
+                    "The response lists Python files found in the current directory",
+                    "The response describes the purpose of at least one file",
+                    "The response reads or inspects file contents rather than guessing",
+                ],
+            },
+        ),
+        Example(
+            input={
+                "prompt": (
+                    "Read the pyproject.toml in the current directory and output a "
+                    "Markdown table with two columns — Dependency and Min Version — "
+                    "for all entries under [project.dependencies]."
+                )
+            },
+            output=None,
+            metadata={
+                "expected_tools": ["read_file"],
+                "task_type": "file_reading",
+                "criteria": [
+                    "The response contains a Markdown table",
+                    "The table has columns for Dependency and Min Version",
+                    "The table includes pydantic, opentelemetry entries",
+                    "The data is read from the actual file, not fabricated",
+                ],
+            },
+        ),
+    ],
+)

examples/claude_code_eval/datasets/__init__.py

@@ -0,0 +1,40 @@
+"""Latency evaluator for Claude Code eval."""
+
+from __future__ import annotations
+
+from elastic_evals.evaluators.base import SimpleEvaluator
+from elastic_evals.types import EvaluationResult, Evaluator, EvaluatorParams
+
+# (upper bound ms, score, label)
+_THRESHOLDS: list[tuple[int, float, str]] = [
+    (5_000, 1.0, "FAST"),
+    (15_000, 0.8, "GOOD"),
+    (30_000, 0.6, "OK"),
+    (60_000, 0.4, "SLOW"),
+]
+
+
+def create_latency_evaluator() -> Evaluator:
+    async def evaluate(params: EvaluatorParams) -> EvaluationResult:
+        output = params.output or {}
+        latency_ms: float = output.get("latency_ms", float("inf"))
+
+        score = 0.2
+        label = "VERY_SLOW"
+        for threshold, s, lbl in _THRESHOLDS:
+            if latency_ms <= threshold:
+                score = s
+                label = lbl
+                break
+
+        return EvaluationResult(
+            score=score,
+            label=label,
+            metadata={
+                "latency_ms": latency_ms,
+                "claude_duration_ms": output.get("claude_duration_ms"),
+                "num_turns": output.get("num_turns"),
+            },
+        )
+
+    return SimpleEvaluator(name="Latency", kind="CODE", evaluate=evaluate)

examples/claude_code_eval/datasets/coding_tasks.py

@@ -0,0 +1,67 @@
+"""Tool use evaluator for Claude Code eval."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from elastic_evals.evaluators.base import SimpleEvaluator
+from elastic_evals.types import EvaluationResult, Evaluator, EvaluatorParams
+
+
+def _to_string_list(value: Any) -> list[str]:
+    if not isinstance(value, list):
+        return []
+    return [item for item in value if isinstance(item, str)]
+
+
+def create_tool_use_evaluator() -> Evaluator:
+    """Score whether Claude Code used the expected tools.
+
+    - expected_tools empty  → PASS (1.0) when no tools were used; FAIL otherwise
+    - expected_tools set    → partial credit: overlap / len(expected)
+    """
+
+    async def evaluate(params: EvaluatorParams) -> EvaluationResult:
+        output = params.output or {}
+        metadata = params.metadata or {}
+
+        expected_tools = _to_string_list(metadata.get("expected_tools", []))
+        actual_tools = _to_string_list(output.get("tool_calls", []))
+        actual_set = set(actual_tools)
+
+        if not expected_tools:
+            score = 1.0 if not actual_set else 0.0
+            label = "PASS" if score == 1.0 else "FAIL"
+            return EvaluationResult(
+                score=score,
+                label=label,
+                metadata={
+                    "expected_tools": expected_tools,
+                    "actual_tools": actual_tools,
+                    "tool_call_count": len(actual_tools),
+                },
+            )
+
+        expected_set = set(expected_tools)
+        overlap = len(expected_set & actual_set)
+        score = overlap / len(expected_set)
+        if score >= 1.0:
+            label = "PASS"
+        elif score > 0:
+            label = "PARTIAL"
+        else:
+            label = "FAIL"
+
+        return EvaluationResult(
+            score=score,
+            label=label,
+            metadata={
+                "expected_tools": expected_tools,
+                "actual_tools": actual_tools,
+                "tool_call_count": len(actual_tools),
+                "matched_tools": sorted(expected_set & actual_set),
+                "missing_tools": sorted(expected_set - actual_set),
+            },
+        )
+
+    return SimpleEvaluator(name="ToolUse", kind="CODE", evaluate=evaluate)

examples/claude_code_eval/evaluators/__init__.py

@@ -0,0 +1,90 @@
+"""Run Claude Code eval example."""
+
+from __future__ import annotations
+
+import asyncio
+import os
+from typing import Any
+
+from elastic_evals.config import ElasticEvalsConfig
+from elastic_evals.evaluators.base import SimpleEvaluator
+from elastic_evals.evaluators.criteria import (
+    EvaluationCriterion,
+    create_criteria_evaluator,
+)
+from elastic_evals.executor import ElasticEvalsClient
+from elastic_evals.tracing import init_tracing
+from elastic_evals.types import EvaluationResult, Evaluator, EvaluatorParams
+
+from examples.claude_code_eval.datasets.coding_tasks import coding_tasks_dataset
+from examples.claude_code_eval.evaluators.latency import create_latency_evaluator
+from examples.claude_code_eval.evaluators.tool_use import create_tool_use_evaluator
+from examples.claude_code_eval.tasks.claude_code import claude_code_task
+
+
+def _read_criteria(metadata: dict[str, Any] | None) -> list[EvaluationCriterion]:
+    if not metadata:
+        return []
+    raw = metadata.get("criteria")
+    if not isinstance(raw, list):
+        return []
+    return [c for c in raw if isinstance(c, str)]
+
+
+def create_criteria_evaluator_from_metadata(
+    *, inference_client: Any, log: Any
+) -> Evaluator:
+    async def evaluate(params: EvaluatorParams) -> EvaluationResult:
+        criteria = _read_criteria(params.metadata)
+        evaluator = create_criteria_evaluator(
+            inference_client=inference_client,
+            criteria=criteria,
+            log=log,
+        )
+        return await evaluator.evaluate(params)
+
+    return SimpleEvaluator(name="criteria", kind="LLM", evaluate=evaluate)
+
+
+async def main() -> None:
+    config = ElasticEvalsConfig.from_env()
+    init_tracing(config.tracing)
+
+    client = ElasticEvalsClient(config)
+
+    edot_endpoint = os.environ.get("EDOT_ENDPOINT", "http://localhost:4318")
+
+    print(f"Running evaluation with run_id: {config.run_id}")
+    print(f"Dataset: {coding_tasks_dataset.name}")
+    print(f"Examples: {len(coding_tasks_dataset.examples)}")
+    print(f"Repetitions: {config.repetitions}")
+    print(f"EDOT collector: {edot_endpoint}")
+    print()
+
+    evaluators: list[Evaluator] = [
+        create_latency_evaluator(),
+        create_tool_use_evaluator(),
+    ]
+
+    # LLM criteria evaluator is optional — only added when an evaluator connector is configured.
+    if config.evaluator_connector_id or config.connector_id:
+        inference_client = client.get_inference_client()
+        evaluators.append(
+            create_criteria_evaluator_from_metadata(
+                inference_client=inference_client,
+                log=config.logger,
+            )
+        )
+
+    async def task(example):
+        return await claude_code_task(example, config)
+
+    await client.run_experiment(
+        dataset=coding_tasks_dataset,
+        task=task,
+        evaluators=evaluators,
+    )
+
+
+if __name__ == "__main__":
+    asyncio.run(main())