Merge pull request #7 from ai4curation/add-mcp-eval-comparison-doc

docs: Add MCP evaluation framework comparison

Author: cmungall

docs/comparison.md

@@ -0,0 +1,73 @@
+# MCP Evaluation Frameworks: A Comparison
+
+As AI agents and models increasingly rely on external tools via the Model-Context Protocol (MCP), robust evaluation frameworks have become essential. These frameworks help ensure that MCP servers and the tools they expose are reliable, performant, and provide a good user experience. This document compares three distinct approaches to MCP evaluation: Metacoder's evaluation framework, `mcp-evals`, and Hume AI's role-play-based evaluations.
+
+## 1. Metacoder Evaluation Framework
+
+Metacoder's evaluation component is a powerful, Python-based framework designed for the systematic testing of AI coders and their integration with MCPs.
+
+-   **Core Technology:** Python. It is deeply integrated with the rest of the Metacoder ecosystem.
+-   **Underlying Engine:** It is built on top of [DeepEval](https://github.com/confident-ai/deepeval), a popular open-source LLM evaluation framework. This provides access to over 40 pre-built metrics for correctness, relevance, faithfulness, and more.
+-   **Configuration:** Evaluations are defined in simple and intuitive `YAML` files. This allows users to specify the models, coders, MCP servers, and a list of test cases with their corresponding inputs, expected outputs, and metrics.
+-   **Evaluation Method:** It runs a matrix of tests against every combination of model, coder, and test case. The results are scored using the specified DeepEval metrics, which can range from simple string comparisons to sophisticated LLM-based judgments.
+-   **Focus:** The primary focus is on **correctness, performance, and reproducible benchmarking**. It is designed to be a rigorous tool for developers to catch regressions, compare different models and coders, and validate the functionality of MCPs.
+
+**Key Strengths:**
+-   Leverages a mature evaluation framework (DeepEval).
+-   Extensive library of ready-to-use metrics.
+-   Enables systematic, matrix-based testing for comprehensive comparisons.
+-   Configuration is straightforward and declarative.
+
+## 2. mcp-evals
+
+`mcp-evals` is a lightweight, developer-focused evaluation tool for MCP implementations, delivered as both a Node.js package and a GitHub Action.
+
+-   **Core Technology:** Node.js and TypeScript.
+-   **Configuration:** Supports both `YAML` and TypeScript for defining evaluation cases. This provides flexibility for both simple, declarative tests and more complex, programmatic evaluation logic.
+-   **Evaluation Method:** Uses LLM-based scoring to assess the quality of an MCP server's responses. For each test case, it evaluates the tool's output based on criteria like accuracy, completeness, relevance, and clarity.
+-   **Focus:** The framework is centered on **MCP tool correctness and developer workflow integration**. The GitHub Action automatically runs evaluations on pull requests and posts the results as a comment, making it easy to catch issues before they are merged. It also has built-in support for observability with Prometheus, Grafana, and Jaeger.
+
+**Key Strengths:**
+-   Seamless integration with CI/CD pipelines via GitHub Actions.
+-   Simple to set up and run.
+-   Provides actionable, multi-faceted scores for each evaluation.
+-   Includes built-in monitoring and tracing capabilities.
+
+## 3. Hume AI's Role-Play Evals
+
+Hume AI introduced a novel and sophisticated approach to MCP evaluation that focuses on the complexities of multi-turn, interactive conversations.
+
+-   **Core Technology:** While the implementation is not specified, the concepts are language-agnostic. The examples are shown in a JSX-like format.
+-   **Configuration:** Evals are defined as "scenarios" where a "role-player" LLM is given a persona and a goal.
+-   **Evaluation Method:** This "role-play" approach uses one LLM to act as the user and another to act as the assistant that uses the MCP tools. The entire conversation transcript is then scored (often by a third LLM) against a set of criteria that capture the quality of the interaction.
+-   **Focus:** The primary focus is on evaluating the **overall user experience (UX)** and the agent's ability to handle complex, collaborative tasks over multiple turns. It is designed to uncover subtle issues in conversational flow that traditional, single-turn evals might miss.
+
+**Key Strengths:**
+-   Excels at evaluating complex, multi-turn interactions.
+-   Provides deep insights into the conversational quality of an agent.
+-   Allows for the evaluation of "soft" skills like clarification, suggestion, and collaboration.
+
+## Comparison Summary
+
+| Feature | Metacoder | mcp-evals | Hume AI (Role-Play) |
+| :--- | :--- | :--- | :--- |
+| **Primary Goal** | Correctness & reproducible benchmarks | Tool correctness & CI integration | Conversational user experience |
+| **Technology** | Python | Node.js / TypeScript | Language-agnostic concept |
+| **Configuration**| YAML | YAML, TypeScript | JSX-like scenarios |
+| **Evaluation** | DeepEval metrics (40+) | LLM-based scoring | LLM-based scoring of a role-play |
+| **Interaction** | Primarily single-turn test cases | Primarily single-turn test cases | Multi-turn conversations |
+| **Integration** | `metacoder` CLI | GitHub Action, `npx` CLI | Custom tooling required |
+| **Cost** | Moderate (depends on metrics) | Moderate | High (many LLM calls per eval) |
+| **Predictability**| High | High | Low (emergent interactions) |
+
+## Conclusion: Which Framework Should You Use?
+
+The best choice of framework depends on your specific needs and goals:
+
+-   **Choose Metacoder if:** You need a robust, systematic way to benchmark the performance and correctness of different AI coders and models against a defined set of test cases. Its integration with DeepEval makes it ideal for reproducible research and regression testing.
+
+-   **Choose `mcp-evals` if:** You want a simple, developer-friendly way to ensure your MCP server's tools are working correctly. Its GitHub Action integration makes it a perfect fit for teams that want to automate their evaluation process and embed it into their development workflow.
+
+-   **Choose Hume AI's approach if:** Your primary concern is the quality of the conversational experience. If your agent is expected to handle complex, multi-step tasks that require collaboration with the user, role-play-based evaluations will provide the most valuable and actionable insights.
+
+Ultimately, these frameworks are not mutually exclusive. A comprehensive evaluation strategy might involve using `mcp-evals` for continuous integration checks, Metacoder for periodic, in-depth benchmarking, and role-play evals for designing and refining the user experience of a conversational agent.

mkdocs.yml

@@ -39,6 +39,7 @@ nav:
     - Introduction: evaluations/index.md
     - Examples: evaluations/examples.md
     - Configuration Reference: evaluations/configuration.md
+    - Framework Comparison: evaluations/comparison.md
   - API Reference: api-reference.md
   - Further reading:
     - Motivation: background/report.md