Skip to content

Commit b5924fd

Browse files
pcarletonpcarleton
committed
docs: add SDK integration guide and fix spec references
- Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests - Link to guide from README.md - Update spec references to use 2025-11-25 instead of draft URLs
1 parent 5dca74b commit b5924fd

3 files changed

Lines changed: 216 additions & 8 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ A framework for testing MCP (Model Context Protocol) client and server implement
55
> [!WARNING]
66
> This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
77
8+
**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository.
9+
810
## Quick Start
911

1012
### Testing Clients

‎SDK_INTEGRATION.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Using MCP Conformance Tests in SDK Repositories
2+
3+
This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility.
4+
5+
## Quick Start
6+
7+
Install and run conformance tests:
8+
9+
```bash
10+
# Client testing (framework starts a test server, runs your client against it)
11+
npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize
12+
13+
# Server testing (your server must already be running)
14+
npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
15+
```
16+
17+
## Two Testing Modes
18+
19+
### Client Testing
20+
21+
The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument.
22+
23+
```bash
24+
# Run a single scenario
25+
npx @modelcontextprotocol/conformance client \
26+
--command "python tests/conformance/client.py" \
27+
--scenario initialize
28+
29+
# Run a suite of tests
30+
npx @modelcontextprotocol/conformance client \
31+
--command "python tests/conformance/client.py" \
32+
--suite auth
33+
```
34+
35+
**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835`
36+
37+
Your client should:
38+
1. Accept the server URL as its last argument
39+
2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested
40+
3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials)
41+
42+
### Server Testing
43+
44+
Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client.
45+
46+
```bash
47+
# Start your server first
48+
your-server --port 3001 &
49+
50+
# Then run conformance tests
51+
npx @modelcontextprotocol/conformance server \
52+
--url http://localhost:3001/mcp \
53+
--suite active
54+
```
55+
56+
**Available server suites:** `active` (default), `all`, `pending`
57+
58+
**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself.
59+
60+
---
61+
62+
## Expected Failures (Baseline) File
63+
64+
The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when:
65+
- A previously passing test starts failing (regression)
66+
- A previously failing test starts passing (stale baseline - remove the entry)
67+
68+
### File Format
69+
70+
Create a YAML file (e.g., `conformance-baseline.yml`):
71+
72+
```yaml
73+
server:
74+
- tools-call-with-progress
75+
- resources-subscribe
76+
client:
77+
- auth/client-credentials-jwt
78+
```
79+
80+
### Usage
81+
82+
```bash
83+
npx @modelcontextprotocol/conformance server \
84+
--url http://localhost:3000/mcp \
85+
--expected-failures ./conformance-baseline.yml
86+
```
87+
88+
### Exit Code Behavior
89+
90+
| Scenario Result | In Baseline? | Exit Code | Meaning |
91+
|-----------------|--------------|-----------|---------|
92+
| Fails | Yes | 0 | Expected failure |
93+
| Fails | No | 1 | Unexpected regression |
94+
| Passes | Yes | 1 | Stale baseline - remove entry |
95+
| Passes | No | 0 | Normal pass |
96+
97+
---
98+
99+
## GitHub Action
100+
101+
The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution.
102+
103+
### Client Testing Example
104+
105+
```yaml
106+
name: Conformance Tests
107+
on: [push, pull_request]
108+
109+
jobs:
110+
conformance:
111+
runs-on: ubuntu-latest
112+
steps:
113+
- uses: actions/checkout@v4
114+
115+
- name: Set up your SDK
116+
run: |
117+
# Your SDK setup (pip install, npm install, etc.)
118+
pip install -e .
119+
120+
- uses: modelcontextprotocol/conformance@v0.1.11
121+
with:
122+
mode: client
123+
command: 'python tests/conformance/client.py'
124+
suite: auth
125+
expected-failures: ./conformance-baseline.yml
126+
```
127+
128+
### Server Testing Example
129+
130+
```yaml
131+
name: Conformance Tests
132+
on: [push, pull_request]
133+
134+
jobs:
135+
conformance:
136+
runs-on: ubuntu-latest
137+
steps:
138+
- uses: actions/checkout@v4
139+
140+
- name: Set up and start server
141+
run: |
142+
pip install -e .
143+
python -m myserver --port 3001 &
144+
# Wait for server to be ready
145+
timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done'
146+
147+
- uses: modelcontextprotocol/conformance@v0.1.11
148+
with:
149+
mode: server
150+
url: http://localhost:3001/mcp
151+
suite: active
152+
expected-failures: ./conformance-baseline.yml
153+
```
154+
155+
### Action Inputs
156+
157+
| Input | Required | Description |
158+
|-------|----------|-------------|
159+
| `mode` | Yes | `server` or `client` |
160+
| `url` | Server mode | URL of the server to test |
161+
| `command` | Client mode | Command to run the client |
162+
| `expected-failures` | No | Path to YAML baseline file |
163+
| `suite` | No | Test suite to run |
164+
| `scenario` | No | Run a single scenario by name |
165+
| `timeout` | No | Timeout in ms for client tests (default: 30000) |
166+
| `verbose` | No | Show verbose output (default: false) |
167+
| `node-version` | No | Node.js version (default: 20) |
168+
169+
---
170+
171+
## Writing Conformance Clients/Servers
172+
173+
### Example Client Pattern
174+
175+
See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario:
176+
177+
```python
178+
import os
179+
import sys
180+
import json
181+
182+
def main():
183+
server_url = sys.argv[-1] # URL passed as last argument
184+
scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "")
185+
context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}"))
186+
187+
if scenario.startswith("auth/"):
188+
run_auth_scenario(server_url, scenario, context)
189+
else:
190+
run_default_scenario(server_url)
191+
192+
if __name__ == "__main__":
193+
main()
194+
```
195+
196+
### Example Server Pattern
197+
198+
See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios.
199+
200+
---
201+
202+
## Additional Resources
203+
204+
- [Conformance README](./README.md)
205+
- [Design documentation](./src/runner/DESIGN.md)
206+
- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance)

‎src/scenarios/client/auth/spec-references.ts‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,15 +19,15 @@ export const SpecReferences: { [key: string]: SpecReference } = {
1919
},
2020
MCP_PRM_DISCOVERY: {
2121
id: 'MCP-2025-06-18-PRM-discovery',
22-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#protected-resource-metadata-discovery-requirements'
22+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#protected-resource-metadata-discovery-requirements'
2323
},
2424
MCP_AUTH_DISCOVERY: {
2525
id: 'MCP-Authorization-metadata-discovery',
26-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#authorization-server-metadata-discovery'
26+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#authorization-server-metadata-discovery'
2727
},
2828
MCP_DCR: {
2929
id: 'MCP-Dynamic-client-registration',
30-
url: 'https://modelcontextprotocol.io/specification/draft/basic/client#dynamic-client-registration'
30+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/client#dynamic-client-registration'
3131
},
3232
OAUTH_2_1_AUTHORIZATION_ENDPOINT: {
3333
id: 'OAUTH-2.1-authorization-endpoint',
@@ -39,23 +39,23 @@ export const SpecReferences: { [key: string]: SpecReference } = {
3939
},
4040
MCP_ACCESS_TOKEN_USAGE: {
4141
id: 'MCP-Access-token-usage',
42-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#access-token-usage'
42+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#access-token-usage'
4343
},
4444
MCP_SCOPE_SELECTION_STRATEGY: {
4545
id: 'MCP-Scope-selection-strategy',
46-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-selection-strategy'
46+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-selection-strategy'
4747
},
4848
MCP_SCOPE_CHALLENGE_HANDLING: {
4949
id: 'MCP-Scope-challenge-handling',
50-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling'
50+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-challenge-handling'
5151
},
5252
MCP_AUTH_ERROR_HANDLING: {
5353
id: 'MCP-Auth-error-handling',
54-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#error-handling'
54+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling'
5555
},
5656
MCP_CLIENT_ID_METADATA_DOCUMENTS: {
5757
id: 'MCP-Client-ID-Metadata-Documents',
58-
url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#client-id-metadata-documents'
58+
url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#client-id-metadata-documents'
5959
},
6060
IETF_CIMD: {
6161
id: 'IETF-OAuth-Client-ID-Metadata-Document',

0 commit comments

Comments
 (0)