[BUG] Incorrect test properties documented?

[thim81]

Issue

When following the same steps, like mentioned: https://github.com/MCPJam/inspector/blob/main/docs/evals/overview.mdx

with the following test

[
  {
    "title": "Basic tag search - Claude Haiku 3.5",
    "prompt": "Can you list Tag-* tags from my instance? Use a wildcard pattern like TAG* to search.",
    "runs": 1,
    "model": "claude-3-5-haiku-20241022",
    "provider": "anthropic",
    "expectedToolCalls": [
      "search_tags"
    ]
  }
]

We get:

Error: [
  {
    "expected": "string",
    "code": "invalid_type",
    "path": [
      0,
      "query"
    ],
    "message": "Invalid input: expected string, received undefined"
  }
]

When changing prompt to query the error is gone.
It makes me wonder, if we are using an outdated CLI or is the online documentation incorrect?

How to reproduce

1. Follow the documentation about evals: https://docs.mcpjam.com/evals/overview
2. run npx @mcpjam/cli evals run -e environments/local-dev.json -t tests/01-search-tags-claude-haiku.json -l environments/llms.json
3. See error

Expected Behavior

Successfully run the tests.

Running tests
Connected to 1 server: Localhost
Found 15 total tools
Running 1 test

Test 1: Basic tag search - Claude Haiku 3.5
Using anthropic:claude-3-5-haiku-20241022

run 1/1

Platform

• npx

Additional Context

The documentation on https://www.npmjs.com/package/@mcpjam/cli shows different properties (prompt <> query). There is also no mention of the advancedConfig, selectedServers.

[dosubot]

Hi @thim81! I'm Dosu and I’m helping the inspector team.

The CLI currently expects the test input property to be named query, not prompt. Using prompt causes a validation error because the CLI looks for a string under query—this matches your experience and is confirmed by the official documentation for test configs, which uses query in all examples and does not mention prompt as a valid property (source).

The documentation in docs/evals/overview.mdx is outdated: it shows prompt, advancedConfig, and selectedServers in its examples, but these are not supported or referenced in the current CLI or test config schema (source). There are no recent code changes or issues that suggest otherwise.

For now, use query for your test input and omit advancedConfig and selectedServers in your test configs for compatibility with the current CLI. The documentation should be updated to reflect this.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}