|
| 1 | +# Sandboxed stdio compatibility validation |
| 2 | + |
| 3 | +Run the pinned **published** CLI against official SDK reference peers, both |
| 4 | +directly and through `mcpm guard run --inner`: |
| 5 | + |
| 6 | +```sh |
| 7 | +bash scripts/dogfood-protocol.sh /tmp/mcpm-protocol-evidence |
| 8 | +``` |
| 9 | + |
| 10 | +Requires Docker and a host Node.js to enforce the outer execution deadline. |
| 11 | +Provisioning the image needs registry access; the test container has no network. |
| 12 | +CI runs this command on Ubuntu and uploads the evidence even when comparisons |
| 13 | +fail. This job tests the released baseline, independently of the source tests. |
| 14 | +It does **not** test a candidate runtime change or automatically follow `latest`. |
| 15 | +The isolated manifest's `npm test` is the container entry point; use the wrapper |
| 16 | +above on the host. Host-only launcher regression checks run with |
| 17 | +`node --test scripts/protocol-sandbox/launcher-check.mjs`. |
| 18 | + |
| 19 | +## Reproducible inputs and isolation |
| 20 | + |
| 21 | +`scripts/protocol-sandbox/package-lock.json` freezes the complete test-only |
| 22 | +dependency tree. The initial inputs are CLI **0.47.0**, SDK v1 **1.32.1**, SDK v2 |
| 23 | +client/server **2.3.1**, Inspector **2.10.1**, and Node **26.11.1** on Debian |
| 24 | +Bookworm. The Docker base is pinned by its multi-platform manifest digest. |
| 25 | +Production dependencies and the root lockfile are unchanged. To update a peer |
| 26 | +or release baseline, change its exact version in the isolated package manifest, |
| 27 | +regenerate that lockfile with npm, rerun the matrix, and update the measured scope |
| 28 | +here. Do not infer compatibility from the package's major version alone. |
| 29 | + |
| 30 | +Only the harness and existing public `modern-*.json` fixtures enter the Docker |
| 31 | +build context. No host home, client configuration, credentials, SSH agent, Docker |
| 32 | +socket, or host directory is mounted. Dependencies install with scripts disabled. |
| 33 | +At execution the container uses UID 1000, a read-only root, no network, no Linux |
| 34 | +capabilities, no privilege escalation, and limits of 768 MiB, two CPUs and 128 |
| 35 | +processes. Each comparison has separate synthetic state in a disposable tmpfs. |
| 36 | +A fresh anonymous Docker volume holds evidence; cleanup removes both it and the |
| 37 | +container after exporting files. The built image remains cached locally. |
| 38 | + |
| 39 | +The harness verifies its UID, effective capabilities, addressed network |
| 40 | +interfaces, read-only application directory and absence of host paths/sockets. |
| 41 | +The exported container configuration records the actual network/mount/resource |
| 42 | +settings. Requests and child shutdowns have deadlines; the outer test run has a |
| 43 | +120-second deadline. On timeout the launcher kills the attach process and stops |
| 44 | +the container before exporting evidence. Raw wire captures have message/byte |
| 45 | +budgets; SDK captures are limited by the tmpfs and checked against a 1 MiB budget |
| 46 | +when read. Large raw messages are represented by size/hash instead of full wire |
| 47 | +text. Everything is |
| 48 | +public or synthetic. These checks are evidence of this container boundary, not a |
| 49 | +claim that containers resist a kernel exploit. |
| 50 | + |
| 51 | +## Measured matrix |
| 52 | + |
| 53 | +Initial local run on **2026-10-10**, Docker Desktop Linux/arm64: **40 pass, zero |
| 54 | +fail, two expected unsupported combinations**, from **42 direct/guarded |
| 55 | +comparisons**, plus four explicit excluded scope categories. Seven harness |
| 56 | +self-checks cover comparison failures, recovery contents, unsupported-error |
| 57 | +correlation, invalid JSON-RPC envelopes, out-of-order correlation, silent/closed |
| 58 | +peers, and unterminated output. A host launcher regression covers successful, |
| 59 | +failed and timed-out runs, stop-before-export, cleanup and stale-evidence refusal. |
| 60 | +CI separately supplies Linux runner evidence. |
| 61 | + |
| 62 | +| Cases | Observations | |
| 63 | +| --- | --- | |
| 64 | +| SDK v1 → v1; v2 auto → v1; v1 → v2 dual-era; v2 pinned → v2 | Ordinary tools/resources/prompts and concurrent tool calls; observed legacy `2025-11-25` or modern `2026-07-28`, with matched opening messages and declaration pins. | |
| 65 | +| Interactive calls for all four pairs | Deterministic sampling, elicitation and roots callbacks. Modern retry carries four keyed responses, a fresh id, unchanged arguments and exact opaque state. SDK v2's legacy shim is exercised too. | |
| 66 | +| Modern decline/cancel, progress, timeout and abort | Expected callback outcomes, progress delivery and successful calls after cancellation/timeouts. | |
| 67 | +| Tool-output and server-input attacks | Correlated block errors and subsequent recovery. Legacy input blocks return to the server's request id; modern input blocks return to the client and do not trigger the input callbacks. | |
| 68 | +| Legacy client → modern-only server | **Unsupported**: observed `-32022`, preserved through the guard. | |
| 69 | +| Modern pinned client → legacy server | **Unsupported**: discovery returns `-32601`; the SDK reports failed era negotiation without fallback. | |
| 70 | +| SDK v1 and v2 auto → `mcpm serve` | Legacy initialization/fallback and actual tool listing only; no registry/model API calls. | |
| 71 | +| Eleven existing modern public fixtures | Benign payload identity, intended blocks and explicit unsupported-shape warnings. | |
| 72 | +| Raw shapes and policies | 512 nested arrays, a 100 KB result, unknown version `-32022`, discovery drift across restart without repinning, coverage-warning ignore/block overrides. | |
| 73 | +| Malformed/oversize/closed/silent peers | Bounded rejection; malformed frame block and `frame-too-large` event; silence reaches a deadline. | |
| 74 | +| Inspector CLI → SDK v2 | Actual tool listing through an explicit server configuration, directly and guarded. No Inspector web proxy is started. | |
| 75 | + |
| 76 | +SDK/Inspector agreement is not an independent conformance oracle: these tools |
| 77 | +share upstream code. Additional assertions check observed version metadata, |
| 78 | +result discriminators, error codes, request ids, retry arguments/state, callback |
| 79 | +counts, cancellation forwarding, persisted pins/events and benign/recovery |
| 80 | +contents. The relevant requirements are |
| 81 | +the [versioning contract](https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning) |
| 82 | +and [MRTR contract](https://modelcontextprotocol.io/specification/2026-07-28/basic/patterns/mrtr). |
| 83 | + |
| 84 | +## Reading the evidence |
| 85 | + |
| 86 | +`report.json` records package versions, npm artifact URL/integrity, lockfile, |
| 87 | +CLI entrypoint and harness hashes, OS/architecture/Node, boundary checks, |
| 88 | +per-case outcomes and counts. Each successful comparison links separate direct |
| 89 | +and guarded files containing observed protocol, synthetic wire data where |
| 90 | +applicable, stderr, pins and events. Assertion failures retain links to observations |
| 91 | +already captured. `container.json` and `image.json` identify the executed settings |
| 92 | +and image. `execution.json` records the overall run status and exit code, including |
| 93 | +deadline failures (124), which may leave a partial `report.json`. Failed assertions |
| 94 | +produce `fail` and a nonzero exit; unsupported outcomes require the expected client |
| 95 | +error and a correlated protocol rejection. The launcher requires an empty output |
| 96 | +directory so older files cannot be mistaken for new evidence. The JSON schema is |
| 97 | +local harness output, not a public CLI contract. |
| 98 | + |
| 99 | +## Remaining verification |
| 100 | + |
| 101 | +Installed native clients and their configuration-adapter lifecycles, model/skill |
| 102 | +behavior, HTTP/OAuth, and universal protocol conformance remain **not tested**. |
| 103 | +This matrix does not cover every method, extension, subscription or repeated |
| 104 | +MRTR depth; it does not claim complete schema validation or detection coverage. |
| 105 | +It neither upgrades `mcpm serve` nor translates protocols in the guard. |
| 106 | + |
| 107 | +Next, use a disposable macOS VM with synthetic settings to validate VS Code and |
| 108 | +Claude Code, including real harmless tool calls and the user-global files mcpm |
| 109 | +actually manages. Isolated-profile connection tests alone do not establish |
| 110 | +adapter compatibility. Expand to other clients/platforms only with named |
| 111 | +versions and observed results; test-account access may be needed for real agent |
| 112 | +tool invocation. |
0 commit comments