Skip to content

fix(deps): bump TLS trust stores past the emSign intermediate CA compromise - #2754

Merged
carneiro-cw merged 2 commits into
mainfrom
fix/emudhra-trust-store-bump
Oct 27, 2026
Merged

carneiro-cw merged 2 commits into
mainfrom
fix/emudhra-trust-store-bump

Conversation

@ggviolin-cw

@ggviolin-cw ggviolin-cw commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Eight eMudhra (emSign) intermediate CA private keys were compromised — six TLS, two code-signing. Non-browser TLS clients that still trust those intermediates accept forged certificates for any host. Browsers are unaffected (Certificate Transparency enforcement), so the exposure is exactly the trust material this service bundles or loads at runtime.

A trust store is vulnerable under this advisory iff it contains any of four affected roots: emSign ECC Root CA - G3 (cross-signer), emSign Root CS CA - G3 and emSign Root TLS CA - G3 (2024 roots, Microsoft program only), and AAA Certificate Services (Sectigo/Comodo; removed from the Mozilla and Chrome programs, still included in Apple and Microsoft — which is why macOS still carries it). Stores that retain only emSign Root CA - G1 / C1 / C3 are not vulnerable; those removals are unrelated policy churn and are not claimed as part of this fix.

This PR bumps every trust store this repository controls. A binary rebuilt from this commit rejects the compromised intermediates even on a host whose ca-certificates package predates the revocation.

Changes

File Change Why
Cargo.lock rustls-webpki 0.103.4 → 0.103.16 0.103.16 carries a DISTRUSTED_SPKIS blocklist of the eight compromised emSign intermediate SPKIs and rejects any path containing one during chain building — unconditionally, independent of the platform store. Fixes RUSTSEC-2026-0381, RUSTSEC-2026-0382, RUSTSEC-2026-0383.
Cargo.lock webpki-root-certs 1.0.2 → 1.0.10 Regenerated Mozilla root store, used by rustls-platform-verifier when the platform store is unreadable. 1.0.2 contained emSign ECC Root CA - G3; 1.0.10 contains none of the four affected roots. The 0.26.11 wrapper re-exports this same data, so both resolved graphs (reqwest → rustls-platform-verifier 0.7.0, jsonrpsee → rustls-platform-verifier 0.5.3) now carry the fixed roots.
utils/deploy/poetry.lock
utils/deploy/requirements.txt
certifi 2024.8.30 → 2026.10.23 The deploy helper's bundled CA file. 2024.8.30 carried two of the four affected roots; 2026.10.23 carries none. Fixes GHSA-6q48-6c7w-9gw2.
supply-chain/config.toml cargo vet exemptions for the moved versions Required for CI's cargo vet --locked gate. aws-lc-rs, aws-lc-sys, cc, find-msvc-tools and shlex move only because the new webpki pins require them.

Validation

Trust store contents — measured, not inferred. Each artifact was re-downloaded from its upstream registry and verified against the checksum recorded in this repository's own manifests (certifi wheels against utils/deploy/poetry.lock, both webpki-root-certs crates against Cargo.lock at the merge base and at HEAD), then parsed and its root names audited against the four affected names:

Store Affected roots before After
certifi 2024.8.30 → 2026.10.23 emSign ECC Root CA - G3, AAA Certificate Services present absent (151 → 118 roots)
webpki-root-certs 1.0.2 → 1.0.10 emSign ECC Root CA - G3 present absent
rustls-webpki 0.103.4 → 0.103.16 no mechanism to reject a compromised intermediate 8 distrusted intermediate SPKIs enforced on every chain build
  • rustls-webpki 0.103.16 verify_cert.rs defines DISTRUSTED_SPKIS with eight include_bytes!("data/distrusted/*.spki.der") entries (EMDVTLSCAG3B/C, EMEVTLSCAG3B/C, EMOVTLSCAG3B/C, EMOVCSCAG3B/C) and passes the list through ChainOptions into build_chain, where a match returns Error::CertDistrusted. Version 0.103.4 has none of this.

  • Execution evidence for that enforcement: the crate's own verify_cert::tests::distrusted_spki test run against the shipping 0.103.16 source at the release tag → test result: ok. 1 passed; 0 failed. That test asserts a chain through a distrusted SPKI is rejected, that distrusting any element of the path (root, intermediate, cross-signed copy, leaf) rejects the chain, and that distrusting an unused intermediate leaves the valid path usable.

  • The extraction/audit scripts were themselves negative-controlled before their numbers were trusted: the same code that reports 151/118 (certifi) and 144/120 (webpki-root-certs) root names also rejects empty, truncated, and name-stripped inputs through a non-vacuity guard instead of returning an empty "clean" result. Root names are parsed from the data rather than substring-grepped, so a store that changes its emission format cannot yield a silent pass.

Dependency gate: cargo vet --locked → Vetting Succeeded (305 fully audited, 111 partially audited, 319 exempted). The CI Vet Dependencies job passes on this PR.

Deploy helper: utils/deploy resolves from the updated lock (poetry install --dry-run --no-root installs all 5 packages) and requirements.txt installed into a clean Python 3.12 virtualenv; the resulting certifi.where() CA file contains none of the four affected roots.

Test suite: CI on this PR is the authoritative signal and is green — Rust Tests pass, Lint and Style checks pass, Vet Dependencies pass, Check for unused dependencies pass, and the e2e jobs (E2E Tests, E2E Leader & Follower, E2E Leader & Fake Leader, E2E Lint) pass. The developer macOS host cannot link rdkafka-sys against its current Homebrew OpenSSL — a pre-existing environment failure that reproduces identically on unmodified main and is unrelated to this change — so no macOS suite output is quoted here. A parallel local Linux build was started for corroboration and abandoned before completion once CI reported, so it is not claimed as evidence either.

Not exercised, and unchanged by this PR: k8s liveness/readiness probes, secret access, external socket reachability from the runtime userland, and the e2e/Hardhat suites. This PR changes only dependency inputs; it cannot validate runtime connectivity.

What this does not prove

  • CI for this repository dispatches the release build to an external pipeline; the binary is compiled and published there, not by workflows visible in this repo. This PR changes dependency inputs only — the rebuild must run and the artifact must be redeployed for the fix to reach a running host. A host running the currently deployed binary stays unpatched until that rebuild and redeploy happen.
  • The repository contains no Dockerfile and no deployment manifests, so no staging stage exists in-repo. Rollout is: merge → release build on main → deploy.
  • ca-certificates on the runtime host is not addressed here because no base image lives in this repository. If any TLS path relies on the platform store rather than the bundled roots, the host package must also be at a distro build that drops the affected roots; that belongs to the image build, not this repo.

Runtimes already at a fixed version

  • .tool-versions pins nodejs 24.22.1, the fixed Node release for this advisory. Node bundles its own CA store inside the binary, so an OS ca-certificates bump would not have covered it; the pinned version already does. No change needed.
  • No Go toolchain, JDK, or Python service runtime is shipped from this repository — the deploy helper under utils/deploy is a local operator script whose only trust store is certifi, bumped above. There is therefore no ca-certificates or JDK bump to make here: the repository contains no base image or Dockerfile.

Risk and rollback

Low. No application code changes; only dependency pins and the vet exemptions that follow them. rustls-webpki 0.103.16 is the same minor line, and its new rejection path only fires against the eight compromised intermediates.

Rollback: git revert 61583381. Note that reverting restores acceptance of the compromised intermediates.

Reviewed by @cloudwalk/team-ledger per CODEOWNERS. Not merged by its author.

@ggviolin-cw
ggviolin-cw requested a review from a team as a code owner October 26, 2026 14:44
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T14:47:31.408456Z 6158338 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Failed to generate code suggestions for PR

@cloudwalk-review-agent cloudwalk-review-agent Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Lockfile-only security bump addressing the emSign intermediate CA compromise. Reviewed the resolution graph and the changes are internally consistent:

  • rustls-webpki 0.103.4 → 0.103.16: the SPKI blocklist lives in the chain-building layer, so it applies regardless of which root source (platform store or bundled webpki-root-certs) is used — this is the right fix point for non-browser TLS clients.
  • webpki-root-certs 1.0.2 → 1.0.10: both resolved graphs are covered — the rustls-platform-verifier entry (~line 5451) and the 0.26.11 wrapper entry (~line 7359) both moved to 1.0.10, so neither reqwest nor jsonrpsee retains the stale roots. No orphaned webpki-root-certs 1.0.2 reference remains.
  • certifi 2024.8.30 → 2026.10.23: pin updated in both requirements.txt and poetry.lock with matching hashes; the python-versions floor moves >=3.6 → >=3.7, which is fine for any supported Python.
  • supply-chain/config.toml: exemptions are updated for every moved crate, including the two genuinely new ones (find-msvc-tools 0.1.15, shlex 2.0.1), so the cargo vet --locked gate should pass.

Two non-blocking notes for awareness: aws-lc-sys 0.46.0 adds pkg-config as a build dependency — if any build/CI image lacks the system pkg-config tool the build will fail loudly there, so watch the first CI run. And cc 1.2.32 → 1.10.0 is a larger transitive jump than a patch bump, but it's pulled in by the aws-lc-sys requirement and carries no runtime exposure beyond compilation.

No code paths, migrations, or API contracts are touched. Clean change; the PR description's validation (root-by-root subject comparison) is above the usual bar for this class of PR.

…romise

Eight eMudhra (emSign) intermediate CA private keys were compromised
(six TLS, two code-signing). Non-browser TLS clients that still trust
those intermediates accept forged certificates. Browsers are protected
by Certificate Transparency enforcement, so only the trust material
bundled or loaded by this service is affected.

Bump every trust store this repository controls:

- rustls-webpki 0.103.4 -> 0.103.16, which rejects the eight compromised
  intermediate SPKIs unconditionally while building a chain, independent
  of what the operating system trusts (RUSTSEC-2026-0381, -0382, -0383).
- webpki-root-certs 1.0.2 -> 1.0.10, the regenerated Mozilla root store
  used by rustls-platform-verifier when it cannot read the platform
  store. The 0.26.11 wrapper re-exports that same data, so both resolved
  graphs now carry the fixed roots.
- certifi 2024.8.30 -> 2026.10.23 for the deploy helper, which drops the
  affected emSign roots from the bundled CA file (GHSA-6q48-6c7w-9gw2).

aws-lc-rs, aws-lc-sys, cc, find-msvc-tools and shlex move as a
consequence of the webpki pins; supply-chain/config.toml exemptions are
updated so cargo vet still passes.

A rebuilt binary carries the fixed stores, so it rejects the compromised
intermediates even on a host whose ca-certificates package predates the
revocation.
@ggviolin-cw
ggviolin-cw force-pushed the fix/emudhra-trust-store-bump branch from 03f2eaf to fb7de3b Compare October 27, 2026 13:41
@carneiro-cw
carneiro-cw merged commit ec10088 into main Oct 27, 2026
11 checks passed
@carneiro-cw
carneiro-cw deleted the fix/emudhra-trust-store-bump branch October 27, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants