chore - the baker rediscovers Incan's own compiler and SDK on every invocation

[dannymeijer]

Area

Tooling (CLI/formatter/test runner), Runtime / Core crates (stdlib/core/derive)

Summary

The baker treats Incan as an input to discover, not as a constant

Incan's compiler and SDK reach the baker prebuilt and sealed. The SDK publisher writes sdk-inventory.json and the ten .incnlib provider descriptors together, records a sha256: digest for each provider in the inventory, and places the whole thing in a content-addressed directory whose name is that content's digest. Store entries are the same shape: entries/sha256-<digest>.loaf/loaf.json, where the directory name is the digest of the file inside it.

None of that can change without changing its own name. Yet the baker rediscovers all of it on every invocation.

Sealed identity and verification exist for external, user-provided code — code the baker has genuinely never seen and cannot trust. Incan itself is not in that category, and spending the same machinery on it is a category error rather than a slow path.

Measured

On a warm store, an explicit bake of a trivial project that compiles nothing — it reports action: "toolchain_loaf", so every unit it needs is already present:

• 1,454 store-manifest reads of 61 distinct files. Every Loaf manifest parsed, schema-checked and re-digested 26 times in one process.
• 4 complete rebuilds of the SDK provider records — 6 when the bake materializes both profiles, 2 in a plain incan build. Each one re-reads and re-deserializes the whole 6.6 MB provider surface and re-validates all ten descriptors against the inventory that shipped with them.
• Sampling one profile: discover_for_oven 24% of the run, sdk_provider_records 20%, split between validate_sdk_descriptor (15%) and deserializing the api_metadata surface (14%).

Total wall clock for that no-op bake: 14.1 s with both profiles, 7.0 s narrowed to one.

What has already landed

Three changes take the within-process half of this, and together bring the suite's configuration from 14.1 s to 2.26 s:

• chore - narrow compiler-suite bakes to the profile the suite runs (#1037) #1542 — the compiler suite stops materializing a release profile it never executes.
• chore - verify each store manifest once per run instead of once per visit (#1037) #1544 — each store manifest is verified once per run rather than once per visit. 7.0 s → 3.3 s.
• chore - read the sealed SDK once per run instead of once per reading (#1037) #1545 — the sealed SDK is read once per run rather than once per reading. 7.0 s → 5.4 s alone; 2.26 s with chore - verify each store manifest once per run instead of once per visit (#1037) #1544.

All three are process-local memos. They make the rescan cheap. They do not remove it.

What this issue is for

Each bake is a new process that re-establishes the same immutable compiler and SDK from cold. A test file running fifty bakes pays that fifty times, and make with no code changes re-derives it rather than passing straight through.

The shape of the fix is that Incan's own compiler and SDK carry one identity, stamped when they were built, which the baker consumes as given:

• The verification result for a content-addressed entry is a pure function of a name that is already a digest. It should be written down once, beside the store, not re-derived per process.
• validate_sdk_descriptor re-establishes agreement between two files the same publisher wrote at seal time. That agreement belongs in the seal.
• The decoded provider surface is a constant per SDK release. It should be loadable in one read of one pre-decoded artifact rather than ten JSON parses.

The acceptance test is the one stated plainly: with no changes to the compiler, the kernel, or the SDK, an invocation has nothing to establish. Fifty bakes reproduce nothing fifty times.

Scope notes

• External, user-provided code keeps every check it has today. Nothing here weakens admission of anything the baker did not ship.
• The store's lease and capacity model is unaffected; this is about what a reader must re-derive, not about what a writer must prove.
• Concurrency matters: the compiler suite runs many baker processes against one store, so anything persisted has to be written under the existing store locks.

Part of #1037.

Scope

• In scope: the behaviour described above, and the acceptance stated under Done when.
• Out of scope: any change to what a reader must prove about admitted code; this is about work that is repeated, not about checks that are owed.
• Risks: recorded inline above where a measurement or a withdrawal established one.

[dannymeijer]

What is left after the within-process work, profiled

With #1542, #1544, #1545 and #1547 applied, a no-op bake is 1.76 s. Sampling it puts the remainder here:

	share of the bake
bake_oven_project_targets per-profile closure	56%
prepare_oven_project → select_or_bake_generated_project_plan	20%
↳ select_payloads_matching_for_execution → verify_published_entry_manifest	19%
↳ verify_entry_manifest (memo misses — the cold first pass)	13%
↳ artifact_identity_from_manifest → serde_json::to_vec	12%

So the single largest identifiable cost left is re-serializing each of the 61 store manifests to recompute an identity digest — the cold pass the per-process memo cannot remove, paid again by every new baker process.

Why I have not cached it, and what the decision is

The other three seals were free: they reuse a reading this process already made, or a reading whose key is a content digest the SDK publisher recorded. Persisting this one is different in kind.

verify_entry_manifest reads the manifest, recomputes the identity from its content, and compares. Skipping that for a stamp some earlier process recorded means trusting a previous process's integrity check instead of making it. The stamp — path, length, modification time — identifies the bytes well enough for a cache and not well enough for a security boundary, and this is the path Gate 5's intake evidence and Gate 7's provenance blocker both run through.

There is no cheaper honest version. The entry directory name is already sha256-<digest>.loaf, but comparing the path to the recorded identity proves the manifest is filed where it claims, not that its content hashes to that. Only the hash proves the second, and the hash is the 12%.

Three ways to spend it, in rough order of how much they change:

1. Leave it. 1.76 s per bake, of which ~0.2 s is this. The verification stays exactly as strong as it is today.
2. Make the hash cheaper. ArtifactIdentityInput is serialized to JSON and then hashed; a canonical binary encoding, or hashing into the digest without materializing a Vec<u8>, would cut most of the cost with no change to what is proven. This looks like the right answer and it is entirely mechanical.
3. Persist the verdict. Fastest, and it moves the trust boundary from "this process verified these bytes" to "some process in this home verified these bytes". Defensible — the file lives in the same trust domain as the store — but it is a change to what a bake proves, and feature - RFC 119 native Rust facets and direct-rustc planning #1037 is the issue where that matters most.

(2) is worth measuring before anyone considers (3).

Reproduction for any of it: a trivial project, a warm store, INCAN_OVEN_BAKE_PROFILES=debug, and sample on the bake process.

[dannymeijer]

Correcting the previous comment: option (2) is not mechanical, and the real driver is better than that

I said a cheaper encoding would cut the identity cost "with no change to what is proven" and that it was "entirely mechanical". Both halves need fixing. Serializing into the hasher instead of into a Vec is mechanical and digest-preserving, but it saves only the allocation — the profile puts 93 of 94 samples inside the serializer itself, not the final hash. Changing the encoding would cut the serializer cost and change every identity in every store and receipt. Those are two different things and I ran them together.

Measuring what is actually being serialized makes the real answer clearer.

The materialized-file list is the cost

Across the 151 manifests in one warm store:

total materialized_files entries	29,741
largest manifests	~3,900 files each, 1.1 MB of JSON each

ArtifactIdentityInput carries materialized_files: &[OvenArtifactMaterializedFileManifest] in full. So recomputing one Loaf's identity re-serializes a 1.1 MB list of ~3,900 file records, and a bake does that for every manifest it touches. That is the 12%, and it has nothing to do with JSON being slow — it is quadratic-feeling work over a list the identity only needs to cover, not contain.

The option that costs nothing in strength

Fold a digest of the materialized-file list into the identity input instead of the list itself:

materialized_files: &[…]     ->     materialized_files_digest: &str

The list stays fully covered — changing any file record changes its digest, which changes the identity — so the property the check proves is unchanged. The serialized identity input drops from megabytes to a few hundred bytes.

It bumps OVEN_STORE_SCHEMA_VERSION, which invalidates existing entries. That is what the schema version is for, and a store is a regenerable cache; verify_entry_manifest already refuses an unsupported schema with a clear error, so old entries are rejected rather than misread.

Revised ordering

1. Fold the file list into a digest. Removes essentially all of the 12%, proves exactly what it proves today, costs one schema bump.
2. Leave it. 1.76 s per bake with ~0.2 s of this in it.
3. Persist the verdict. Still the only option that moves the trust boundary, and now clearly unnecessary — (1) gets the same win without touching what a bake proves.

I have not implemented (1): a store schema bump on the path Gate 5 and Gate 7 run through is a decision rather than a cleanup, and the measurement above is what it needs to be decided on.

[dannymeijer]

A third option, better than both of the others: verify what you select, not what you enumerate

Looking at the caller rather than the cost changes the answer again.

select_matching_execution_payloads walks the whole entries directory and calls verify_published_entry_manifest on every entry before testing whether it is even wanted:

let path = canonical_published_entry_root(&path)?;
let manifest = verify_published_entry_manifest(&path)?;
if !matches(&manifest) {
    continue;
}

verify_published_entry_manifest parses the manifest and recomputes its identity by re-serializing ArtifactIdentityInput — including the full materialized_files list, which for the big Loaf manifests is ~3,900 records and 1.1 MB. The store measured here holds 151 manifests with 29,741 file records between them; a production store had 519 entries. Every one of them is fully verified on the way to selecting a handful.

Split the two halves and order them by need. Selection needs the manifest parsed — that is what matches reads. It does not need the identity recomputed, because an entry that does not match is never executed. Recompute the identity after the filter, for entries actually selected.

The property that matters is preserved exactly: every payload that gets executed has had its identity verified. A tampered manifest that makes itself match is selected, and therefore verified, and therefore rejected. A tampered manifest that does not match is not used for anything.

Against the other two:

	removes the cost	changes what is proven	invalidates stores
persist the verdict	yes	yes — trusts a previous process	no
fold the file list into a digest	yes	no	yes — schema bump, and everyone re-prewarms
verify only what is selected	yes, for this path	no	no

The third costs nothing and needs no decision about evidence, so it should be tried before either of the others. If a measurable cost remains afterwards, the digest fold is the next one and the persisted verdict should stay off the table.

Why I have not implemented it

It is a contained change — one call site, and a split of verify_published_entry_manifest into parse-and-schema-check and identity-recheck — but it is on the integrity path that Gate 5's intake evidence and Gate 7's provenance blocker both run through, and this environment cannot verify an end-to-end bake of a deriving fixture (a borrowed Loaf leaves incan_derive unlinked, so the failure is the environment rather than the change). Three changes to the store, plan and lockfile paths already went out tonight; a fourth to the same area without end-to-end coverage is where mistakes start compounding.

It wants a session with a working prewarm, and it is the first thing to do on this issue.

[dannymeijer]

Rescoping this issue against the SDK immutability direction.

Most of what I wrote earlier optimises the rescan. That was the right read of the measurements and the wrong read of the destination: once Incan and its SDK are a sealed constant the baker consumes by stamped identity, the SDK half of this issue does not get faster — it disappears. I am not investing further in that half.

What survives the change, and what does not:

	survives?
#1544 — store manifests read once per run	yes — the store keeps mutating as bakes publish into it
#1551 — prove the entry you select, not every entry you walk past	yes — about what verification is owed, not about rescanning the SDK
#1547 first commit — one semantic pass per run	yes — deduplication inside one command
#1547 second commit — that pass kept between runs	no — a cache for a rescan immutability removes
#1545 — memoized SDK provider records	no, and separately unsound; already drafted
folding materialized_files into a digest it covers	partly — a store schema question, independent of the SDK
persisting the verification verdict	no — and it was already the worst of the three

The measurements stay useful as a statement of what the current design spends. A bake that compiles nothing cost 10.51 s, of which: 1,454 store-manifest reads of 61 distinct files, 4 full SDK provider rebuilds re-parsing 6.6 MB, and 2 semantic-identity passes over the same ten providers. That is the size of the prize for treating Incan as a constant, and it argues for the immutability work far better than for any cache in this issue.

Leaving this open for the store-side items only.