Skip to content

Commit 235fd21

Browse files
authored
test: verify published MCP stdio compatibility in an offline sandbox (#248)
* test: add isolated published MCP compatibility matrix * fix: tighten protocol sandbox outcomes and deadline cleanup
1 parent fb685ca commit 235fd21

15 files changed

Lines changed: 5168 additions & 0 deletions

‎.github/workflows/ci.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,3 +88,28 @@ jobs:
8888
- run: pnpm install --frozen-lockfile
8989

9090
- run: pnpm dogfood:confine
91+
92+
protocol-sandbox:
93+
name: published stdio compatibility (container)
94+
runs-on: ubuntu-latest
95+
timeout-minutes: 10
96+
steps:
97+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
98+
99+
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
100+
with:
101+
node-version: 26.11.1
102+
103+
- name: Compare pinned peers directly and through the published guard
104+
run: |
105+
node --test scripts/protocol-sandbox/launcher-check.mjs
106+
bash scripts/dogfood-protocol.sh "$RUNNER_TEMP/protocol-evidence"
107+
108+
- name: Preserve synthetic compatibility evidence
109+
if: always()
110+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
111+
with:
112+
name: protocol-evidence
113+
path: ${{ runner.temp }}/protocol-evidence
114+
if-no-files-found: error
115+
retention-days: 14

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,14 @@ _Add entries here, never under a stamped version_ — a release commit renames t
88
heading, and a branch that wrote beneath it merges without conflict straight into a
99
published section (it happened to #170).
1010

11+
### Added
12+
13+
- Reproducible offline container validation of the pinned published CLI against
14+
SDK v1/v2 peers and Inspector, with direct/guarded comparisons, bounded
15+
synthetic evidence and explicit unsupported/not-tested results. CI preserves
16+
the compatibility report. This adds test tooling, not runtime behavior or an
17+
installed-client/full-conformance guarantee.
18+
1119
## [0.47.0] - 2026-10-09
1220

1321
MINOR by content: new modern MCP carrier coverage and an additive diagnostic

‎CLAUDE.md‎

Lines changed: 6 additions & 0 deletions
Large diffs are not rendered by default.

‎CONTRIBUTING.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,11 @@ macOS-only end-to-end check for the `--confine` sandbox:
5959
pnpm dogfood:confine # hermetic; must print "✓ confine dogfood PASSED"
6060
```
6161

62+
For offline direct/guarded stdio comparisons using the pinned published release
63+
and official SDK v1/v2 peers, run `bash scripts/dogfood-protocol.sh /tmp/mcpm-protocol-evidence`.
64+
Docker provisions the test dependencies, then runs with no network or host
65+
mounts. See [the matrix, evidence and limits](docs/PROTOCOL-VALIDATION.md).
66+
6267
## Ground rules
6368

6469
- **Tests first, and they must pass.** New behaviour needs a test; bug fixes need a

‎docs/GUARD.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,12 @@ measure adoption or interoperability in installed IDE clients, and do not certif
7676
all 2026-07-28 protocol behavior. `mcpm serve` still uses the existing SDK; no SDK
7777
upgrade or negotiation behavior is included in this change.
7878

79+
The separate [sandbox compatibility harness](PROTOCOL-VALIDATION.md) compares
80+
the pinned published release with official SDK v1/v2 peers and Inspector over
81+
stdio. It records named protocol outcomes, guard errors and persistence under
82+
container isolation. Installed native clients and full conformance remain
83+
unverified; the matrix states the tested operations and exclusions.
84+
7985
### Full message flow
8086

8187
The relay inspects each direction independently — a request (parent → child) and the matching response (child → parent) each run the full pattern + pin + policy pipeline before anything is forwarded:

‎docs/PROTOCOL-VALIDATION.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Sandboxed stdio compatibility validation
2+
3+
Run the pinned **published** CLI against official SDK reference peers, both
4+
directly and through `mcpm guard run --inner`:
5+
6+
```sh
7+
bash scripts/dogfood-protocol.sh /tmp/mcpm-protocol-evidence
8+
```
9+
10+
Requires Docker and a host Node.js to enforce the outer execution deadline.
11+
Provisioning the image needs registry access; the test container has no network.
12+
CI runs this command on Ubuntu and uploads the evidence even when comparisons
13+
fail. This job tests the released baseline, independently of the source tests.
14+
It does **not** test a candidate runtime change or automatically follow `latest`.
15+
The isolated manifest's `npm test` is the container entry point; use the wrapper
16+
above on the host. Host-only launcher regression checks run with
17+
`node --test scripts/protocol-sandbox/launcher-check.mjs`.
18+
19+
## Reproducible inputs and isolation
20+
21+
`scripts/protocol-sandbox/package-lock.json` freezes the complete test-only
22+
dependency tree. The initial inputs are CLI **0.47.0**, SDK v1 **1.32.1**, SDK v2
23+
client/server **2.3.1**, Inspector **2.10.1**, and Node **26.11.1** on Debian
24+
Bookworm. The Docker base is pinned by its multi-platform manifest digest.
25+
Production dependencies and the root lockfile are unchanged. To update a peer
26+
or release baseline, change its exact version in the isolated package manifest,
27+
regenerate that lockfile with npm, rerun the matrix, and update the measured scope
28+
here. Do not infer compatibility from the package's major version alone.
29+
30+
Only the harness and existing public `modern-*.json` fixtures enter the Docker
31+
build context. No host home, client configuration, credentials, SSH agent, Docker
32+
socket, or host directory is mounted. Dependencies install with scripts disabled.
33+
At execution the container uses UID 1000, a read-only root, no network, no Linux
34+
capabilities, no privilege escalation, and limits of 768 MiB, two CPUs and 128
35+
processes. Each comparison has separate synthetic state in a disposable tmpfs.
36+
A fresh anonymous Docker volume holds evidence; cleanup removes both it and the
37+
container after exporting files. The built image remains cached locally.
38+
39+
The harness verifies its UID, effective capabilities, addressed network
40+
interfaces, read-only application directory and absence of host paths/sockets.
41+
The exported container configuration records the actual network/mount/resource
42+
settings. Requests and child shutdowns have deadlines; the outer test run has a
43+
120-second deadline. On timeout the launcher kills the attach process and stops
44+
the container before exporting evidence. Raw wire captures have message/byte
45+
budgets; SDK captures are limited by the tmpfs and checked against a 1 MiB budget
46+
when read. Large raw messages are represented by size/hash instead of full wire
47+
text. Everything is
48+
public or synthetic. These checks are evidence of this container boundary, not a
49+
claim that containers resist a kernel exploit.
50+
51+
## Measured matrix
52+
53+
Initial local run on **2026-10-10**, Docker Desktop Linux/arm64: **40 pass, zero
54+
fail, two expected unsupported combinations**, from **42 direct/guarded
55+
comparisons**, plus four explicit excluded scope categories. Seven harness
56+
self-checks cover comparison failures, recovery contents, unsupported-error
57+
correlation, invalid JSON-RPC envelopes, out-of-order correlation, silent/closed
58+
peers, and unterminated output. A host launcher regression covers successful,
59+
failed and timed-out runs, stop-before-export, cleanup and stale-evidence refusal.
60+
CI separately supplies Linux runner evidence.
61+
62+
| Cases | Observations |
63+
| --- | --- |
64+
| SDK v1 → v1; v2 auto → v1; v1 → v2 dual-era; v2 pinned → v2 | Ordinary tools/resources/prompts and concurrent tool calls; observed legacy `2025-11-25` or modern `2026-07-28`, with matched opening messages and declaration pins. |
65+
| Interactive calls for all four pairs | Deterministic sampling, elicitation and roots callbacks. Modern retry carries four keyed responses, a fresh id, unchanged arguments and exact opaque state. SDK v2's legacy shim is exercised too. |
66+
| Modern decline/cancel, progress, timeout and abort | Expected callback outcomes, progress delivery and successful calls after cancellation/timeouts. |
67+
| Tool-output and server-input attacks | Correlated block errors and subsequent recovery. Legacy input blocks return to the server's request id; modern input blocks return to the client and do not trigger the input callbacks. |
68+
| Legacy client → modern-only server | **Unsupported**: observed `-32022`, preserved through the guard. |
69+
| Modern pinned client → legacy server | **Unsupported**: discovery returns `-32601`; the SDK reports failed era negotiation without fallback. |
70+
| SDK v1 and v2 auto → `mcpm serve` | Legacy initialization/fallback and actual tool listing only; no registry/model API calls. |
71+
| Eleven existing modern public fixtures | Benign payload identity, intended blocks and explicit unsupported-shape warnings. |
72+
| Raw shapes and policies | 512 nested arrays, a 100 KB result, unknown version `-32022`, discovery drift across restart without repinning, coverage-warning ignore/block overrides. |
73+
| Malformed/oversize/closed/silent peers | Bounded rejection; malformed frame block and `frame-too-large` event; silence reaches a deadline. |
74+
| Inspector CLI → SDK v2 | Actual tool listing through an explicit server configuration, directly and guarded. No Inspector web proxy is started. |
75+
76+
SDK/Inspector agreement is not an independent conformance oracle: these tools
77+
share upstream code. Additional assertions check observed version metadata,
78+
result discriminators, error codes, request ids, retry arguments/state, callback
79+
counts, cancellation forwarding, persisted pins/events and benign/recovery
80+
contents. The relevant requirements are
81+
the [versioning contract](https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning)
82+
and [MRTR contract](https://modelcontextprotocol.io/specification/2026-07-28/basic/patterns/mrtr).
83+
84+
## Reading the evidence
85+
86+
`report.json` records package versions, npm artifact URL/integrity, lockfile,
87+
CLI entrypoint and harness hashes, OS/architecture/Node, boundary checks,
88+
per-case outcomes and counts. Each successful comparison links separate direct
89+
and guarded files containing observed protocol, synthetic wire data where
90+
applicable, stderr, pins and events. Assertion failures retain links to observations
91+
already captured. `container.json` and `image.json` identify the executed settings
92+
and image. `execution.json` records the overall run status and exit code, including
93+
deadline failures (124), which may leave a partial `report.json`. Failed assertions
94+
produce `fail` and a nonzero exit; unsupported outcomes require the expected client
95+
error and a correlated protocol rejection. The launcher requires an empty output
96+
directory so older files cannot be mistaken for new evidence. The JSON schema is
97+
local harness output, not a public CLI contract.
98+
99+
## Remaining verification
100+
101+
Installed native clients and their configuration-adapter lifecycles, model/skill
102+
behavior, HTTP/OAuth, and universal protocol conformance remain **not tested**.
103+
This matrix does not cover every method, extension, subscription or repeated
104+
MRTR depth; it does not claim complete schema validation or detection coverage.
105+
It neither upgrades `mcpm serve` nor translates protocols in the guard.
106+
107+
Next, use a disposable macOS VM with synthetic settings to validate VS Code and
108+
Claude Code, including real harmless tool calls and the user-global files mcpm
109+
actually manages. Isolated-profile connection tests alone do not establish
110+
adapter compatibility. Expand to other clients/platforms only with named
111+
versions and observed results; test-account access may be needed for real agent
112+
tool invocation.

‎scripts/dogfood-protocol.sh‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
4+
OUTPUT="${1:?Usage: bash scripts/dogfood-protocol.sh OUTPUT_DIRECTORY}"
5+
mkdir -p "$OUTPUT"
6+
OUTPUT="$(cd "$OUTPUT" && pwd -P)"
7+
node -e 'if(require("node:fs").readdirSync(process.argv[1]).length)throw Error("Evidence directory must be empty")' "$OUTPUT"
8+
CONTEXT="$(mktemp -d)"
9+
CONTAINER=""
10+
cleanup() {
11+
local status=$?
12+
if [ -n "$CONTAINER" ]; then docker rm -fv "$CONTAINER" >/dev/null || status=1; fi
13+
rm -rf "$CONTEXT" || status=1
14+
exit "$status"
15+
}
16+
trap cleanup EXIT
17+
trap 'exit 130' INT
18+
trap 'exit 143' TERM
19+
# Send only the harness and public fixtures to Docker, never the entire checkout.
20+
for file in Dockerfile package.json package-lock.json harness.mjs self-check.mjs peer.mjs run.mjs; do
21+
cp "$ROOT/scripts/protocol-sandbox/$file" "$CONTEXT/$file"
22+
done
23+
mkdir "$CONTEXT/fixtures"
24+
for group in benign warn attacks; do
25+
cp "$ROOT/src/guard/__tests__/fixtures/mcptox/$group"/modern-*.json "$CONTEXT/fixtures/"
26+
done
27+
IMAGE="$(docker build -q "$CONTEXT")"
28+
CONTAINER="$(docker create --network none --read-only --user 1000:1000 --cap-drop ALL \
29+
--security-opt no-new-privileges --pids-limit 128 --memory 768m --cpus 2 --init \
30+
--mount type=volume,destination=/evidence \
31+
--tmpfs /work:rw,noexec,nosuid,size=128m,uid=1000,gid=1000 "$IMAGE")"
32+
set +e
33+
node - "$CONTAINER" "$OUTPUT" <<'NODE'
34+
const r = require('node:child_process').spawnSync('docker', ['start', '-a', process.argv[2]], {
35+
stdio: 'inherit', timeout: 120000, killSignal: 'SIGKILL',
36+
});
37+
const status = r.error?.code === 'ETIMEDOUT' ? 124 : (r.status ?? 1);
38+
require('node:fs').writeFileSync(`${process.argv[3]}/execution.json`, JSON.stringify({
39+
status: status === 0 ? 'pass' : 'fail', exitCode: status, error: r.error?.code, signal: r.signal,
40+
}));
41+
process.exitCode = status;
42+
NODE
43+
RESULT=$?
44+
set -e
45+
# Killing the attach client does not stop its container. Freeze evidence first.
46+
if [ "$(docker inspect --format '{{.State.Running}}' "$CONTAINER")" = true ]; then
47+
docker kill "$CONTAINER" >/dev/null || [ "$(docker inspect --format '{{.State.Running}}' "$CONTAINER")" = false ]
48+
fi
49+
docker cp "$CONTAINER:/evidence/." "$OUTPUT"
50+
docker inspect --format '{{json .HostConfig}}' "$CONTAINER" > "$OUTPUT/container.json"
51+
docker image inspect --format '{{json .}}' "$IMAGE" > "$OUTPUT/image.json"
52+
echo "Protocol evidence: $OUTPUT/report.json"
53+
exit "$RESULT"
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM node:26.11.1-bookworm-slim@sha256:86f07bc9c5dce4578cf37e5a418b7bfc7f817cda25cde66e2b66e95ed86c4567
2+
WORKDIR /app
3+
COPY package.json package-lock.json ./
4+
RUN npm ci --ignore-scripts --no-audit --no-fund && npm cache clean --force
5+
RUN mkdir /evidence && chown 1000:1000 /evidence
6+
COPY harness.mjs self-check.mjs peer.mjs run.mjs ./
7+
COPY fixtures ./fixtures
8+
ENV HOME=/work/home TMPDIR=/work
9+
USER 1000:1000
10+
WORKDIR /work
11+
CMD ["npm", "--prefix", "/app", "test"]
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
import assert from 'node:assert/strict';
2+
import { spawn } from 'node:child_process';
3+
import { createInterface } from 'node:readline';
4+
import { createHash } from 'node:crypto';
5+
6+
export function equivalent(direct, guarded) {
7+
assert.notEqual(direct, undefined, 'missing direct observation');
8+
assert.notEqual(guarded, undefined, 'missing guarded observation');
9+
assert.deepEqual(guarded, direct, 'guard changed benign semantics');
10+
}
11+
12+
export function assertEcho(result, expected) {
13+
assert.ok(!result.isError, 'echo returned a tool error');
14+
assert.deepEqual(result.content, [{ type: 'text', text: JSON.stringify(expected) }]);
15+
}
16+
17+
export function assertUnsupported(observation, method, wireCode, clientCode) {
18+
assert.equal(observation.value.code, clientCode);
19+
const request = observation.wire.find(w => w.direction === 'to-server' && w.message.method === method)?.message;
20+
assert.ok(request, `missing ${method} request`);
21+
const response = observation.wire.find(w => w.direction === 'from-server' && w.message.id === request.id)?.message;
22+
assert.equal(response?.error?.code, wireCode);
23+
}
24+
25+
export function record(wire, direction, message) {
26+
const json = JSON.stringify(message);
27+
assert.ok(wire.length < 256, 'wire message budget exceeded');
28+
wire.push({ direction, ...(json.length <= 32768 ? { message } : {
29+
bytes: Buffer.byteLength(json), sha256: createHash('sha256').update(json).digest('hex'), truncated: true,
30+
}) });
31+
}
32+
33+
export function exchange(command, args, options = {}) {
34+
const child = spawn(command, args, { stdio: ['pipe', 'pipe', 'pipe'], detached: true, ...options });
35+
const wire = [];
36+
const pending = new Map();
37+
let nextId = 0;
38+
let stderr = '';
39+
let closed = false;
40+
let failure;
41+
const rejectAll = error => {
42+
failure = error;
43+
for (const p of pending.values()) { clearTimeout(p.timer); p.reject(error); }
44+
pending.clear();
45+
};
46+
const done = new Promise(resolve => {
47+
child.on('close', () => { closed = true; rejectAll(new Error('peer disconnected')); resolve(); });
48+
child.on('error', rejectAll);
49+
});
50+
child.stdin.on('error', rejectAll);
51+
child.stderr.on('data', b => { stderr = (stderr + b).slice(-32768); });
52+
const reader = createInterface({ input: child.stdout });
53+
let bytes = 0;
54+
child.stdout.on('data', chunk => {
55+
bytes += chunk.length;
56+
if (bytes > 16 * 1024 * 1024) {
57+
rejectAll(new Error('wire byte budget exceeded'));
58+
reader.close(); child.stdout.destroy();
59+
}
60+
});
61+
reader.on('line', line => {
62+
try {
63+
assert.ok(line.length <= 11 * 1024 * 1024, 'wire frame budget exceeded');
64+
const message = JSON.parse(line);
65+
assert.ok(message && typeof message === 'object' && !Array.isArray(message), 'invalid JSON-RPC object');
66+
assert.equal(message.jsonrpc, '2.0', 'invalid JSON-RPC version');
67+
record(wire, 'from-server', message);
68+
if ('method' in message) return;
69+
assert.notEqual('result' in message, 'error' in message, 'response needs exactly one result or error');
70+
const p = pending.get(message.id);
71+
assert.ok(p, `unmatched response ${message.id}`);
72+
clearTimeout(p.timer); pending.delete(message.id); p.resolve(message);
73+
} catch (error) { rejectAll(error); }
74+
});
75+
function send(message) {
76+
if (failure) throw failure;
77+
record(wire, 'to-server', message);
78+
child.stdin.write(JSON.stringify(message) + '\n');
79+
}
80+
return {
81+
child, wire, get stderr() { return stderr; },
82+
send,
83+
request(method, params = {}, timeout = 3000) {
84+
const id = ++nextId;
85+
return new Promise((resolve, reject) => {
86+
const timer = setTimeout(() => { pending.delete(id); reject(new Error(`deadline: ${method}`)); }, timeout);
87+
pending.set(id, { resolve, reject, timer });
88+
try { send({ jsonrpc: '2.0', id, method, params }); }
89+
catch (e) { clearTimeout(timer); pending.delete(id); reject(e); }
90+
});
91+
},
92+
async close() {
93+
child.stdin.end();
94+
const pause = ms => new Promise(resolve => { const t = setTimeout(resolve, ms); done.then(() => { clearTimeout(t); resolve(); }); });
95+
await pause(500);
96+
for (const signal of ['SIGTERM', 'SIGKILL']) {
97+
// The guard owns a child too; tear down this test's entire process group.
98+
try { process.kill(-child.pid, signal); } catch (e) { if (e.code !== 'ESRCH') throw e; }
99+
if (!closed) await pause(1000);
100+
}
101+
reader.close();
102+
assert.ok(closed, 'peer did not terminate');
103+
},
104+
};
105+
}

0 commit comments

Comments
 (0)