Skip to content

Commit ce2fc2f

Browse files
authored
feat: review explicit skill and plugin snapshots (#327)
1 parent 1b573fc commit ce2fc2f

17 files changed

Lines changed: 1149 additions & 29 deletions

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,12 @@ _Add entries here, never under a stamped version_ — a release commit renames t
88
heading, and a branch that wrote beneath it merges without conflict straight into a
99
published section (it happened to #170).
1010

11+
## [0.51.0] - 2026-11-06
12+
13+
### Added
14+
15+
- Experimental `mcpm review-artifacts` compares two explicit local USTAR skill/plugin snapshots without extraction or execution. Receipts bind file hashes, mode changes, declarations and literal references to retained bytes, with separate byte/declaration coverage and unknown host state.
16+
1117
## [0.50.1] - 2026-11-06
1218

1319
### Fixed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,7 @@ Those 20 points **inform the score but cannot clear a safety floor.** The MCP se
284284
| `mcpm publish check` | Dry-run: show trust score and what would be submitted (`--json` emits the exact request body) |
285285
| `mcpm publish` | Submit to the official MCP registry (requires `GITHUB_TOKEN`/`MCPM_TOKEN`, or `--github-oidc` in GitHub Actions with `id-token: write`) |
286286
| `mcpm scope <project-root>` | Read-only Claude Code source/scope and instruction-file inventory (`--json`); [coverage and unknown states](docs/PROJECT-SCOPE.md) |
287+
| `mcpm review-artifacts <before.tar> <after.tar> --profile <profile>` | Compare explicit local USTAR Agent Skills or Claude plugin snapshots, with byte/mode changes, declarations, literal references and coverage gaps (`--json`); [format and limits](docs/ARTIFACT-REVIEW.md) |
287288
| `mcpm doctor` | Check MCP setup health and report issues (`--json` structured model, `--report` redacted paste-for-bug-reports snapshot) |
288289
| `mcpm init` | Scaffold a starter `mcpm.yaml` stack file in the current directory |
289290
| `mcpm disable <name>` | Disable an MCP server without removing it from config |

‎docs/ARTIFACT-REVIEW-VALIDATION.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Artifact-review candidate evidence — 2026-11-06
2+
3+
This record describes project backlog #156 before merge/publication. Current
4+
published CLI remains v0.50.1; no v0.51.0 publication is claimed here. The
5+
[command/profile contract](ARTIFACT-REVIEW.md) defines the supported archive and
6+
declaration subset. Source checks, historical and final candidate execution, and
7+
the real consumer decision below have separate identities. The 0.51.0 changelog
8+
section is prepared for release; this does not assert publication.
9+
10+
## Reviewed source checkpoint
11+
12+
Implementation base: `1b573fc3568c42dcb1a08149da05eb3c702f32b3`. Sol implemented
13+
the command; Astra independently reproduced and fixed archive/declaration,
14+
reference, pointer and coverage edge cases, then reproduced a Markdown parsing
15+
CPU regression and replaced the backtracking scan with a bounded linear scan.
16+
The latest source receipt is
17+
`/private/tmp/mcpm-156-astra-source-freeze-v2-20261106.json`.
18+
19+
That source checkpoint records Node 24.20.0 typecheck (main/tooling), build,
20+
62 focused tests, whitespace checks and all 163 test files passing: **3,846 tests
21+
passed, three skipped**. It supersedes the first Astra source freeze. These are
22+
source results; fresh packed-candidate execution is recorded separately below. The
23+
placeholder package version remains `0.15.0`; no dependency/lockfile or override
24+
change is part of #156. Documentation additions after the source freeze are
25+
tracked separately and do not change runtime bytes.
26+
27+
## Final latest-source candidate
28+
29+
The fresh candidate archive is
30+
`/private/tmp/mcpm-156-final-candidate-20261106/getmcpm-cli-0.15.0.tgz`, SHA-256
31+
`9a39064a617c76b843e9241b8324a3fc7e09842bc986dce0760b696e7a4805e4`.
32+
Its consumer binding receipt names the latest Astra source-freeze SHA-256
33+
`aa42dbbe48d0e16ee73e29e32f007121e4cfd46a6a33bfc18bc936438f3b8b25`.
34+
The actual isolated image identity is
35+
`sha256:c32866659f49450a1e74f6e7dd13a06670b36a3f2068e01ccbfe5fd3352392a6`.
36+
37+
Independent read-back matched all **66 installed JavaScript files**, the three
38+
input archive hashes below, and every reported member hash, size and mode.
39+
Both newly executed consumer reports matched the previously reviewed reports:
40+
31 old-to-capture and eight tag-to-capture changed/unknown records, with exit 1
41+
and both coverage flags false. The maintainer's retain-source/no-rollback
42+
decision was reaffirmed; no global mutation, native activation or safety
43+
approval followed. The isolated execution used UID 1000, no network, a
44+
read-only root and no host bind mounts. Home/project directories were checked
45+
empty after execution, not traced for absence of transient writes. The temporary
46+
read-back container was removed after its evidence was retained.
47+
48+
| Check | Final candidate/source result | Local receipt |
49+
| --- | --- | --- |
50+
| Clean installation and public CLI smoke | Passed against the exact candidate archive | `/private/tmp/mcpm-156-final-smoke.log` |
51+
| Modern/legacy direct/guarded stdio protocol | 87 passed, zero failures, two expected unsupported combinations, four exclusions; independent actual-image/runtime/input binding passed | `/private/tmp/mcpm-156-final-protocol-20261106/{report,verified-binding}.json` |
52+
| Real macOS confinement dogfood | Passed control access, denied synthetic-secret read, intact relay, tamper refusal and event logging | `/private/tmp/mcpm-156-final-confine.log` |
53+
| Source coverage | Passed: 90.15% statements, 86.64% branches, 87.20% functions, 91.92% lines | Maintainer coverage run |
54+
| Negative coverage-threshold probe | Passed; failing thresholds were rejected | `/private/tmp/mcpm-156-final-negative.log` |
55+
| Real consumer and member/runtime binding | Passed receipt bindings; both product reports intentionally incomplete, retention reaffirmed | `/private/tmp/mcpm-156-final-candidate-20261106/consumer/run-final/verified-binding.json` |
56+
57+
These are candidate/static/protocol/confinement observations for their named
58+
surfaces. They do not prove native artifact loading, default approvals, provider
59+
behavior, universal host coverage or a published package. Published-byte
60+
authentication and companion synchronization remain release gates.
61+
62+
## Historical real-consumer inputs
63+
64+
The maintainer prepared inert uncompressed USTAR snapshots outside the product.
65+
The separate preparation record names the two upstream commits and the explicit
66+
captured source root; the mcpm receipt itself authenticates neither provenance
67+
nor host cache/activation state. Preparation normalized owner/time and retained
68+
member content/modes. Only `.git` metadata was omitted from the captured source;
69+
no vendor or `node_modules` directory was present or omitted. An earlier generic
70+
omission description was corrected while preserving the original record and
71+
unchanged archive bytes.
72+
73+
| Snapshot | Regular files | Archive SHA-256 |
74+
| --- | --- | --- |
75+
| Ponytail tag 5.3.2, upstream commit `16352ac229eb08a9598dcae48fe9f2b342567d86` | 213 | `fead09d7275d23ca5713e92873ec4ab10cf1fa177090b4bb5d18d3e1f929e10f` |
76+
| Ponytail tag 5.3.3, upstream commit `7801d061243d7b753bad0e0b3c057cd250b11755` | 214 | `032b3aa5083a19bec10a8539b9033fdb6f4b5f61a606204073129ae3eb67c544` |
77+
| Explicit local 5.3.3 source capture | 216 | `3ab7882f3a3251aa890c78e095258b52a5dd2715635ed74415855e4a37c35bb2` |
78+
79+
Inputs, preparation script/proof and consumer decision are retained temporarily
80+
under `/private/tmp/mcpm-156-consumer-v2-20261106/`. Both comparisons used
81+
`--profile claude-plugin-2026-11-06 --withhold .git --json`.
82+
83+
## Historical candidate execution and decision
84+
85+
The earlier candidate npm archive SHA-256 was
86+
`0456b014efb655a4037c3127c7579010d4399711b2a911f3b9942df9e26717e5`, bound to the
87+
first Astra source-freeze receipt SHA-256
88+
`ed7d58f54b9549911253b3c0cdd12429f127a2889f0378827dc6ba34757f7c1c`.
89+
This **predates the latest Markdown CPU fix**. The `run-v2` directory name below
90+
identifies a consumer-run revision, not the newer source-freeze v2.
91+
The first `run-v1` consumer attempt failed on input-file permissions before
92+
producing reports and is excluded. Its receipt is retained. The original CPU
93+
regression and source-capture metadata-pack limit failure are also excluded from
94+
passing evidence; no failed capture archive was produced. The final candidate
95+
above was freshly executed after the fixes.
96+
97+
Retained container inspection and binding receipts matched that archive, all
98+
66 installed JavaScript files and the three input hashes. The execution used
99+
UID 1000, no network, read-only root, dropped capabilities, no-new-privileges
100+
and no host bind mounts/credentials. Home/project directories were checked empty
101+
after execution; this is not a syscall-level assertion that no write occurred.
102+
No artifact hooks/scripts, native validators, installers or provider/model calls
103+
were executed.
104+
105+
| Comparison | Result | Evidence used by the maintainer |
106+
| --- | --- | --- |
107+
| 5.3.2 tag → captured 5.3.3 source | Exit 1; byte and declaration coverage incomplete; 31 changed/unknown records | Changed executable/source files, declared shared-hook source/timeouts, literal references and explicit coverage gaps |
108+
| 5.3.3 tag → captured 5.3.3 source | Exit 1; byte and declaration coverage incomplete; eight changed/unknown records | 208 matching known content/mode pairs; five README/translation modifications, two SVG unknown additions, one withheld `.env.example` |
109+
110+
Dynamic `node -e`, opaque Claude-band modules, unsupported manifest fields and
111+
nine other-host manifests remained interpretation gaps. The default-withheld
112+
`.env.example` member and prepackaging `.git` omission remained byte-coverage
113+
gaps. Archive buffering still read packaged member bytes; hashing was not
114+
redaction. Other-host declarations, including Codex, were not evaluated as loaded
115+
components. Configured/cached/loaded/enforced host states remained unknown.
116+
117+
The actual maintainer decision was to **retain the reviewed 5.3.3 source update
118+
without requesting a rollback**. No global changes or native install/activation
119+
approval followed. Manual semantic review separately checked hook timeout
120+
5→15 seconds, activation subprocess timeout 1→3 seconds, Git-list buffer
121+
1→64 MiB with the existing line-count cap, and the opaque band's transcript
122+
lookup across Claude project folders. SVG XML was inspected separately; the
123+
receipt's file-role heuristic did not establish executable status or safety.
124+
125+
The one-consumer gate is met for this bounded experiment: the receipt combined
126+
captured byte/mode identity with component/reference/coverage mapping instead of
127+
requiring that inventory to be assembled manually. Public-tag diff alone did not
128+
bind the captured local bytes. This is not a measured time saving, evidence of
129+
general demand, superiority over native/Git/APM workflows, or a safety verdict.
130+
131+
Historical local receipts:
132+
133+
- `/private/tmp/mcpm-156-consumer-sandbox-20261106/run-v2/{consumer,old-to-cache,tag-to-cache,verified-binding}.json`
134+
- `/private/tmp/mcpm-156-consumer-v2-20261106/consumer-decision.json`
135+
- `/private/tmp/mcpm-156-consumer-v2-20261106/preparation-proof.json`
136+
137+
## Remaining delivery gates
138+
139+
The latest-source repack, consumer bindings, smoke, protocol, confinement and
140+
coverage and negative-threshold checks are complete as described above.
141+
Final artifact/documentation review, hosted checks,
142+
merge, publication, published-byte authentication and companion synchronization
143+
remain pending in this record. The prepared 0.51.0 changelog section is not
144+
publication evidence. The one-consumer result does not authorize #157/#158 or another host
145+
dialect; those retain their own scope and evidence gates.

‎docs/ARTIFACT-REVIEW.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Local artifact change review
2+
3+
`mcpm review-artifacts <before.tar> <after.tar> --profile <profile> [--json]`
4+
compares two explicitly supplied local snapshots. This is a bounded experiment
5+
for a maintainer reviewing a concrete update, alongside manual source review.
6+
It identifies changed bytes/modes, declared components and literal local
7+
references without extracting files, executing code, installing anything,
8+
invoking native validators or reading host configuration/credential stores.
9+
10+
## Input and source identity
11+
12+
Prepare **uncompressed USTAR00** archives outside mcpm, with bundle-relative
13+
member paths (no enclosing checkout directory). One skill snapshot has
14+
`SKILL.md` at its root; one plugin snapshot has `.claude-plugin/plugin.json`.
15+
Directory inputs fail before any child read. No archive member path ever reaches
16+
the filesystem. This avoids the ancestor-symlink race in recursive path readers.
17+
The explicitly opened archive must be a regular file without symlink components.
18+
19+
Fixed limits are 16 MiB per archive, 2 MiB per member, 256 regular files,
20+
512 entries and 16 path levels. Only regular files and zero-size directories
21+
are accepted. Magic/version, unsigned header checksum, octal numbers, paths,
22+
payload bounds, zero padding, two zero end blocks and zero-only aligned trailers
23+
are checked. Links, special files, PAX/GNU/sparse extensions, duplicate normalized
24+
names, file/ancestor collisions, invalid UTF-8 names and truncated/concatenated
25+
archives make inventory incomplete. Unsupported archives yield no partial
26+
definitive member inventory. This strict subset intentionally refuses some
27+
archives accepted by native tar. [Primary format reference](https://www.gnu.org/software/tar/manual/html_node/Standard.html).
28+
29+
Source identity means the retained archive bytes and SHA-256. It does not
30+
authenticate a publisher, commit, tag, cache location or supplied label. Member
31+
hashes bind content from that same buffer; permission modes are separate facts.
32+
Archive order/owner/time differences can change the container hash without
33+
changing member content. Both open handles and named file identities are checked
34+
again after both analyses; observed changes make the result incomplete. This is
35+
not an atomic filesystem snapshot or protection from a privileged writer.
36+
37+
## Pinned interpretation profiles
38+
39+
- `agent-skills-2026-11-06`: required YAML frontmatter, naming/description and
40+
optional standard metadata constraints, root `SKILL.md`, explicit Markdown
41+
links and conventional resource tokens. The root directory name cannot be
42+
verified from a flat archive; nested plugin skill names are checked against
43+
their member directory. Unknown fields remain gaps. [Agent Skills specification](https://agentskills.io/specification).
44+
- `claude-plugin-2026-11-06`: manifest-required static subset of the rolling
45+
Claude reference retrieved on that date. Default/additional skills,
46+
replacement command/agent/style/workflow paths, inline commands, hook paths
47+
and arrays/inline event maps, and path/inline MCP/LSP declarations are
48+
inventoried. Duplicate component names are gaps. Alternate forms outside
49+
this subset, modules/Mods, userConfig, dependencies, settings, experimental
50+
fields, unknown fields and other host manifests remain explicit gaps.
51+
Declared paths are checked as files or directories as appropriate; empty
52+
declared directories are retained. Unsupported hook fields, event names over
53+
128 characters and unsupported or incomplete MCP/LSP entry shapes remain gaps.
54+
A missing manifest is incomplete even though native Claude supports
55+
manifest-free plugins. Plugin version strings are opaque release metadata,
56+
not schema/host versions or byte identity. [Manifest reference](https://code.claude.com/docs/en/plugins-reference),
57+
[loading reference](https://code.claude.com/docs/en/plugins/loading).
58+
59+
Literal local reference candidates come only from Markdown links, conventional
60+
`scripts/`, `references/`, `assets/`, `hooks/` tokens and literal
61+
`${CLAUDE_PLUGIN_ROOT}/...` paths. They are not inferred executions. Dynamic
62+
commands such as `node -e` remain unresolved; JS imports, shell behavior,
63+
dependencies, module semantics and commands in prose are not analyzed. External
64+
URLs are never followed. A named module file is not proof that a host loads it.
65+
Local Markdown URL queries and fragments are omitted from reference targets;
66+
queries retain an interpretation gap. JSON pointers escape member names and
67+
identify the actual singleton, array or wrapped declaration location.
68+
69+
## Receipt, gaps and sensitive input
70+
71+
JSON is `schemaVersion: 1`. Each side contains the explicit source path/state,
72+
archive digest, member paths/states/sizes/modes/digests, components, references
73+
and gaps. Components/references/gaps identify their source member digest where
74+
available. No raw instruction, command, environment, header or credential values
75+
are emitted. Configured/cached/loaded/enforced state and host version are unknown;
76+
archive preparation provenance belongs in a separate maintainer record.
77+
78+
`byteInventoryComplete` and `declarationCoverageComplete` are separate. Neither
79+
means safe, approved or active. A known hash/mode difference on both retained
80+
sides is a `modified` fact even when unrelated members are withheld. Missing
81+
counterparts under incomplete inventory have `unknown` deltas, never definitive
82+
removal/addition. Exit 0 requires both coverage flags; exit 1 retains the receipt
83+
and its gaps. Consumers must inspect gaps and source states, and stop on unknown
84+
receipt versions. There is no universal score, installer or enforcement claim.
85+
86+
Sensitive-looking members (`.env*`, credential/key files and store directories,
87+
`.git`, `node_modules`) are withheld from interpretation and member hashes by
88+
default. `--withhold <paths...>` adds member paths/subtrees, including inputs
89+
omitted during preparation, as coverage gaps. **Archive buffering still reads
90+
packaged member bytes and the container digest includes them. Hashing is not
91+
redaction.** Omit sensitive bytes before packaging when they must not be read;
92+
record omitted paths with `--withhold` and preserve preparation provenance.
93+
94+
## Actual consumer checkpoint — 2026-11-06
95+
96+
An mcpm maintainer used two candidate receipts plus manual source
97+
review to decide whether to retain the reviewed local Ponytail 5.3.3 source
98+
update or request a rollback to 5.3.2. Inputs were inert USTAR snapshots of the
99+
two pinned upstream tags and the explicitly captured 5.3.3 source directory;
100+
source capture provenance was recorded separately from the product reports.
101+
Only `.git` was omitted before packaging; no `node_modules`/vendor directory was
102+
present or omitted. `.git` was declared with `--withhold`, and the packaged
103+
`.env.example` member was withheld from interpretation/member hashing by default.
104+
105+
The old-to-capture receipt exposed 31 changed/unknown member records. The
106+
tag-to-capture comparison had 208 matching known file bytes/modes, five known
107+
README/translation modifications, two unknown SVG additions and one withheld
108+
`.env.example` record. Declared shared hooks and timeout fields focused the code
109+
review; unresolved `node -e`, opaque Claude-band modules, unsupported manifest
110+
fields and nine other-host manifest gaps prevented a complete interpretation.
111+
Both commands exited **1**, with both coverage flags false.
112+
113+
The maintainer retained the reviewed source update without requesting a rollback.
114+
No global configuration, installation or native activation was changed or
115+
approved. Manual review, rather than the report, interpreted subprocess timeout,
116+
buffer-size and transcript-discovery changes; SVG content was checked separately
117+
without treating the file-role heuristic as a safety verdict. The receipt's
118+
observed value was combining actual captured byte/mode identity with component,
119+
reference and missing-coverage mapping. Time savings, general demand and
120+
superiority over native/Git/APM tooling were not measured.
121+
122+
The first reports belonged to an earlier candidate before the Markdown CPU fix.
123+
A fresh archive of the latest reviewed source reran both comparisons with all
124+
66 installed JavaScript files, all three input hashes and every reported member
125+
hash/size/mode checked against the actual isolated image. Both new reports
126+
matched the reviewed earlier reports, and the maintainer reaffirmed retention.
127+
This is new candidate evidence, not an inherited earlier pass or a publication
128+
claim. Exact archive identities, additional smoke/protocol/confinement results,
129+
receipt locations and remaining gates are in the
130+
[candidate validation record](ARTIFACT-REVIEW-VALIDATION.md). Configured, cached,
131+
loaded and enforced host states remain unknown throughout.

0 commit comments

Comments
 (0)