ci: ship reproducible zone binaries by Rjected · Pull Request #1264 · tempoxyz/zones

Rjected · 2026-08-24T20:43:12Z

Builds Linux x86_64 node images and release binaries with the canonical reproducible profile. Production publication builds the same tempo-zone image target on Depot and an independent clean GitHub runner, compares their complete image IDs, and stages Depot's candidate only when they match. Companion image tags publish after this check succeeds. A verification-only dispatch exercises the comparison without publishing production tags.

Docker candidate builds have read-only registry access and export archives by artifact ID. Release publication verifies the archived binary against a separate clean rebuild and revalidates its tag. Release tags must point to commits already on main before OIDC-enabled build jobs execute tagged code. Manual SHA and nightly image tags use the verified promotion path.

grandizzy

ty! left couple of comments, please check

grandizzy

looks good, left one comment, think was added for testing?

grandizzy · 2026-08-27T12:02:27Z

cyclops audit

github-actions · 2026-08-27T12:02:37Z

cc @grandizzy

Cyclops audit event published. View workflow run

Config: config: default, iterations: default, hours: default

tempoxyz-bot

👁️ Cyclops Review

PR #1264 adds a reproducible-build and promotion contract for the production Docker image and canonical Linux x86_64 release artifact. The consolidated review found several actionable trust-boundary issues in the workflow/bake wiring; see inline comments.

Reviewer Callouts

⚡ Event-matrix coverage: Add regression coverage that expands Docker metadata/tag outputs for push, schedule, branch dispatch, and tag dispatch, and asserts production aliases are emitted only by the trusted promotion path.
⚡ Build-context determinism: .github/workflows/docker.yml:496-505 downloads .tempo-genesis/genesis.json from a mutable URL before Dockerfile.reproducible performs COPY . .; it is not known to affect the final binary today, but adding it to .dockerignore would remove an unnecessary external input from the reproducible context.
⚡ Shipped binary provenance: Confirm tempo-zone --version, RPC client version, and P2P client string contain the resolved SHA rather than a vergen placeholder when .git is excluded from the Docker context.
⚡ Allocator/runtime asymmetry: JEMALLOC_OVERRIDE=/usr/lib/x86_64-linux-gnu/libjemalloc.a means the canonical Linux artifact links Debian's jemalloc instead of the vendored crate build used elsewhere; confirm this is intentional for runtime and observability expectations.

grandizzy

ty! left couple of comments, pls check

grandizzy · 2026-10-07T17:04:42Z

+1

tempo-voight-kampff

Approving on behalf of @grandizzy, who approved this pull request (head 3fbe460a79a5) with a +1 comment via Voight-Kampff.

GitHub branch protection considers pull request reviews, not +1 comments. Voight-Kampff is recording this approval on the reviewer's behalf so branch protection requirements are met.

tempoxyz-bot · 2026-10-07T17:06:34Z

👁️ Cyclops Security Review

Last reviewed: dab34f2 (latest) · ⚠️ 1 open finding

#	Severity	Finding	Introduced	Status
1	🟢 Low	Companion tags update despite failed release verification	`1894e63`	✅ Fixed in `35561c1`
2	🟡 Medium	Unverified runtime files pass image promotion	`a6d9e7b`	✅ Fixed in `35561c1`
3	⚪ Needs review	Production image freezes its system CA trust store	`1894e63`	🟠 Open
4	⚪ Needs review	Manual release can execute code from an unchecked tag on OIDC-enabled runners	`1894e63`	✅ Fixed in `35561c1`

tempoxyz-bot · 2026-10-07T17:17:37Z

+  publish-companion-images:
+    name: Publish companion Docker images
+    needs: build-and-push
+    if: >-
+      github.repository == 'tempoxyz/zones' &&
+      github.event_name != 'pull_request' && github.event_name != 'merge_group'
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      id-token: write


Low: Companion tags update despite failed release verification

The new publish-companion-images job promotes builder-supplied archives to public companion image tags without waiting for production-reproducible-verify or validating companion contents. A compromised candidate builder can supply altered images that remain published even if node verification fails and blocks tempo-zone promotion.

addressed in 35561c1

Adds manual independent prover EIF measurement verification using the shared workflow from tempoxyz/gh-actions#193. Production and verification share the reproducible-profile compiler, pinned runtime recipe and timestamp-normalizing EIF packaging helper. Production records the source, genesis checksum, binary checksum and toolchain digest needed to reproduce its PCRs. The next production build adopts this recipe and new measurements. The toolchain is resolved from main and cached by recipe hash. Dispatch cannot supply an arbitrary image. Verification requires a genesis checksum, rejects non-main or conflicting verification requests, and uploads both EIFs, PCRs and file hashes for comparison. Stacked on #1264. Dispatch `reproducible_eif_verify=true` from main after merging. Instructions are in `docs/REPRODUCIBLE_BUILDS.md`.

Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>

Adds manual independent prover EIF measurement verification using the shared workflow from tempoxyz/gh-actions#193. Production and verification share the reproducible-profile compiler, pinned runtime recipe and timestamp-normalizing EIF packaging helper. Production records the source, genesis checksum, binary checksum and toolchain digest needed to reproduce its PCRs. The next production build adopts this recipe and new measurements. The toolchain is resolved from main and cached by recipe hash. Dispatch cannot supply an arbitrary image. Verification requires a genesis checksum, rejects non-main or conflicting verification requests, and uploads both EIFs, PCRs and file hashes for comparison. Stacked on #1264. Dispatch `reproducible_eif_verify=true` from main after merging. Instructions are in `docs/REPRODUCIBLE_BUILDS.md`.

tempoxyz-bot · 2026-10-07T17:29:48Z

+          trap cleanup EXIT
+
+          docker load --input "$RUNNER_TEMP/candidate-images/tempo-zone.tar"
+          immutable_image=tempo-zone:candidate
+          container_id="$(docker create "$immutable_image")"
+          docker cp "$container_id:$BINARY_PATH" "$RUNNER_TEMP/tempo-zone"
+          image_sha256="$(sha256sum "$RUNNER_TEMP/tempo-zone" | awk '{print $1}')"
+          [[ "$image_sha256" =~ ^[0-9a-f]{64}$ ]] || {
+            failure="invalid image binary SHA-256"
+            echo "Invalid image binary SHA-256: $image_sha256" >&2
+            exit 1
+          }
+
+          image_runtime_config="$(docker image inspect "$immutable_image" --format '{{json .Config}}' | \
+            jq -cS '{Entrypoint, Cmd, Env, WorkingDir, User, ExposedPorts, Volumes, Healthcheck, StopSignal}')"
+          image_runtime_config_sha256="$(printf '%s' "$image_runtime_config" | sha256sum | awk '{print $1}')"
+          docker cp "$container_id:/etc/ssl/certs/ca-certificates.crt" "$RUNNER_TEMP/ca-certificates.crt"
+          image_ca_bundle_sha256="$(sha256sum "$RUNNER_TEMP/ca-certificates.crt" | awk '{print $1}')"
+
+          [[ "$image_runtime_config_sha256" =~ ^[0-9a-f]{64}$ ]] || {
+            failure="invalid image runtime-config SHA-256"
+            echo "Invalid image runtime-config SHA-256: $image_runtime_config_sha256" >&2
+            exit 1
+          }
+          [[ "$image_ca_bundle_sha256" =~ ^[0-9a-f]{64}$ ]] || {
+            failure="invalid image CA-bundle SHA-256"
+            echo "Invalid image CA-bundle SHA-256: $image_ca_bundle_sha256" >&2
+            exit 1
+          }
+
+          if [[ "$image_sha256" == "$CLEAN_SHA256" && \
+              "$image_runtime_config_sha256" == "$CLEAN_RUNTIME_CONFIG_SHA256" && \
+              "$image_ca_bundle_sha256" == "$CLEAN_CA_BUNDLE_SHA256" ]]; then
+            comparison_result="success"
+            echo "Staged tempo-zone binary and runtime inputs match the clean rebuild"
+          else
+            failure="staged binary or runtime input does not match clean rebuild"
+            echo "::error::Staged production reproducibility verification mismatch"
+            exit 1
+          fi
+
+          # Registry write access exists only on this fresh verification runner.
+          # No candidate executable or build script runs here.
+          tagged_image="$IMAGE_REPOSITORY:$IMAGE_TAG"
+          docker tag "$immutable_image" "$tagged_image"
+          docker push "$tagged_image"


Medium: Unverified runtime files pass image promotion

The production verification gate compares the candidate image’s binary, selected configuration, and CA bundle, but not its remaining filesystem. A compromised candidate builder can supply an image with an unchanged binary and a modified runtime loader or preload file; the gate then promotes that image to production tags.

addressed in 35561c1

tempoxyz-bot · 2026-10-07T18:00:36Z

 target "tempo-zone" {
-  inherits = ["_common", "docker-metadata"]
-  target = "tempo-zone"
+  inherits = ["docker-metadata"]
+  dockerfile = "docker/Dockerfile.reproducible"
+  context = "."
+  target = "tempo-zone-reproducible"
+  args = {
+    SOURCE_DATE_EPOCH = "${SOURCE_DATE_EPOCH}"
+    GIT_SHA = "${GIT_SHA}"
+    VERSION = "${VERSION}"
+  }
+  labels = {
+    "org.opencontainers.image.description" = "Production tempo-zone image; verification covers the tempo-zone binary, runtime execution config, and CA bundle."
+    "org.tempoxyz.reproducible.verification-scope" = "tempo-zone-binary-runtime-config-ca-bundle"
+  }
+  platforms = ["linux/amd64"]
 }


Needs review · unverified: Production image freezes its system CA trust store

Switching the production tempo-zone target to Dockerfile.reproducible makes its runtime install ca-certificates from a fixed May 1, 2026 Debian snapshot instead of the former runtime's updated apt sources. Scheduled clean rebuilds reproduce that same trust store; matching its hash does not establish that the roots remain current. Conditional impact: if an operator uses an HTTPS L1_HTTP_RPC_URL without replacing the image's…

Full finding

Switching the production tempo-zone target to Dockerfile.reproducible makes its runtime install ca-certificates from a fixed May 1, 2026 Debian snapshot instead of the former runtime's updated apt sources. Scheduled clean rebuilds reproduce that same trust store; matching its hash does not establish that the roots remain current. Conditional impact: if an operator uses an HTTPS L1_HTTP_RPC_URL without replacing the image's CA store, and a root trusted by the snapshot is subsequently distrusted, an interceptor able to present a certificate under that root can impersonate the RPC endpoint. L1 storage cache misses use its responses in EVM state reads. The review did not establish that a relevant root has in fact been removed or that a deployment uses this optional HTTPS path. Keep the binary build reproducible while refreshing and independently validating the runtime trust store, or require periodic signed snapshot updates.

First reported at 1894e63; recheck at this head: still present.

tempoxyz-bot · 2026-10-07T18:00:38Z

  workflow_dispatch:
    inputs:
      tag:
-        description: "Tag to release (e.g., v0.1.0)"
+        description: "Existing Git tag to release (e.g., v0.1.0)"
        required: true
        type: string


Needs review · unverified: Manual release can execute code from an unchecked tag on OIDC-enabled runners

A workflow_dispatch run on main can supply any existing syntactically valid tag, including one outside the v*.*.* push pattern. resolve-release checks that the tag resolves consistently, not that its commit is approved or descended from protected main. Both build jobs then check out that commit and execute its scripts/reproducible-build.sh with id-token: write. Setting dry_run=true skips the version check but does…

Full finding

A workflow_dispatch run on main can supply any existing syntactically valid tag, including one outside the v*.*.* push pattern. resolve-release checks that the tag resolves consistently, not that its commit is approved or descended from protected main. Both build jobs then check out that commit and execute its scripts/reproducible-build.sh with id-token: write. Setting dry_run=true skips the version check but does not skip either execution. If an actor can create such a tag and dispatch the workflow, tag-controlled shell code runs in the privileged workflow context; access to external credentials depends on the deployed OIDC trust policy. This manual-dispatch source change is absent from the previous build checkout. Require an approved tag or protected-main ancestry before executing checked-out scripts, and avoid granting OIDC permission to jobs that run untrusted tag contents.

First reported at 1894e63; recheck at this head: still present.

addressed in 35561c1

Compare the full exported runtime filesystem with a clean rebuild, delay companion tags until node verification succeeds, and require release tags to come from main before OIDC-enabled build jobs execute their contents.

grandizzy reviewed Aug 25, 2026

View reviewed changes

Comment thread docker/docker-bake.hcl

Comment thread .github/workflows/docker.yml Outdated

Rjected force-pushed the dan/reproducible-zone-binaries branch from ceada15 to 796c83d Compare August 26, 2026 23:05

Rjected marked this pull request as ready for review August 26, 2026 23:13

grandizzy reviewed Aug 27, 2026

View reviewed changes

Comment thread .github/workflows/release.yml Outdated

tempoxyz-bot reviewed Aug 27, 2026

View reviewed changes

Rjected mentioned this pull request Aug 27, 2026

ci: verify reproducible candidate image #1262

Merged

Rjected force-pushed the dan/reproducible-zone-binaries branch from c17e204 to 67021dd Compare August 28, 2026 23:40

grandizzy self-requested a review September 2, 2026 14:58

decofe mentioned this pull request Sep 2, 2026

fix(prover): harden Nitro enclave bootstrap #1303

Open

6 of 8 tasks

mattsse mentioned this pull request Oct 1, 2026

ci: hard-code the profiling image tags #1630

Open

Rjected force-pushed the dan/reproducible-zone-binaries branch from 67021dd to 5ed989a Compare October 1, 2026 20:09

grandizzy reviewed Oct 2, 2026

View reviewed changes

Comment thread .github/workflows/docker.yml Outdated

Comment thread .github/workflows/docker.yml

Base automatically changed from dan/reproducible-image-verify to main October 2, 2026 15:17

rkrasiuk force-pushed the dan/reproducible-zone-binaries branch from 26c095d to 48a2f12 Compare October 2, 2026 15:17

Rjected mentioned this pull request Oct 5, 2026

ci: verify reproducible prover eif measurements #1474

Merged

tempo-voight-kampff Bot previously approved these changes Oct 7, 2026

View reviewed changes

decofe dismissed tempo-voight-kampff[bot]’s stale review via 1894e63 October 7, 2026 17:06

rkrasiuk previously approved these changes Oct 7, 2026

View reviewed changes

tempoxyz-bot reviewed Oct 7, 2026

View reviewed changes

Rjected dismissed rkrasiuk’s stale review via c858a1b October 7, 2026 17:25

Rjected added 3 commits October 7, 2026 10:26

ci: ship reproducible zone binaries

0978b6e

ci: focus release dry runs on reproducible contract

870780f

ci: stage verified zone image promotion

a21b0ab

Rjected and others added 12 commits October 7, 2026 10:26

fix(ci): restore profiling Docker workflow

a99f8ef

fix(ci): harden reproducible artifact verification

0878c1b

fix(ci): bind release tags and assets to verification

6796616

fix(ci): verify manual and nightly image publication

42903cd

fix(ci): isolate candidate builds from registry writes

ba76f87

fix(ci): disable attestations for Docker archives

9702cdb

fix(ci): distinguish candidate artifact attempts

8e84a5f

fix(ci): rebuild EIF inputs without stale registry cache

4192f6d

fix(ci): avoid expired registry cache sessions

d6177a6

fix(ci): disable mold for macOS releases

6212768

Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>

test(ci): remove Python image publication tests

86971a0

Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>

Rjected force-pushed the dan/reproducible-zone-binaries branch from c858a1b to a6d9e7b Compare October 7, 2026 17:26

rkrasiuk previously approved these changes Oct 7, 2026

View reviewed changes

tempoxyz-bot reviewed Oct 7, 2026

View reviewed changes

fix(ci): gate image publication on trusted verification

92d4f07

Compare the full exported runtime filesystem with a clean rebuild, delay companion tags until node verification succeeds, and require release tags to come from main before OIDC-enabled build jobs execute their contents.

Rjected dismissed rkrasiuk’s stale review via 92d4f07 October 7, 2026 18:12

Rjected added 6 commits October 7, 2026 18:13

docs(ci): describe release tag ancestry requirement

35561c1

refactor(ci): hash runtime filesystem without Python

15e35d2

revert(ci): restore Python filesystem helper

b6271a9

ci: compare complete production images before publishing

8d74a51

ci: allow clean Bake archive export

ac992e2

ci: use pinned BuildKit for clean image export

dab34f2

Conversation

Rjected commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grandizzy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

grandizzy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grandizzy commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tempoxyz-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grandizzy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

grandizzy commented Oct 7, 2026

Uh oh!

tempo-voight-kampff Bot left a comment

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👁️ Cyclops Security Review

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

tempoxyz-bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Rjected commented Aug 24, 2026 •

edited

Loading

github-actions Bot commented Aug 27, 2026 •

edited

Loading

tempoxyz-bot commented Oct 7, 2026 •

edited

Loading