Add Verkada CI workflows with FIPS support

[nickvines]

Summary

• Enable FIPS-compliant OpenSSL builds
• Add custom Verkada workflows (vlinux, vmacos, vrelease) that won't conflict with upstream syncs
• Only build 3 targets: aarch64-apple-darwin, aarch64-unknown-linux-gnu, x86_64-unknown-linux-gnu

Changes

Commit 1: Enable FIPS

• Enable FIPS module in OpenSSL builds (non-musl)
• Copy FIPS modules and config to Python installations
• Preserve fips.so during build

Commit 2: Add Verkada CI workflows

• vlinux.yml: Linux builds (aarch64 + x86_64) on ubuntu-latest
• vmacos.yml: macOS builds (aarch64) on macos-latest
• vrelease.yml: Manual release workflow
• Disable upstream workflow triggers (workflow_dispatch only)
• Reduce release.rs to 3 targets

Build Configuration

• Python versions: 3.10, 3.11, 3.12, 3.13, 3.14
• Build variants: pgo+lto, freethreaded+pgo+lto (3.13+)
• Free GitHub runners only (ubuntu-latest, macos-latest)

Test plan

• vlinux workflow runs successfully on PR
• vmacos workflow runs successfully on PR
• Upstream workflows don't trigger on push/PR

🤖 Generated with Claude Code

[nickvines]

Code Review Summary

Critical Issues (2)

1. Removed -fvisibility=hidden globally (build-openssl-3.5.sh:46)

   • This affects ALL compiled code, not just FIPS module
   • Could expose internal symbols, increasing attack surface
   • This was intentional from the original FIPS commit - needed for FIPS module loading

2. Missing input validation in vrelease.yml (Lines 48-89)

   • User-provided tag and sha inputs used directly in shell commands
   • Potential command injection if malicious actor has workflow dispatch permissions

Warnings (5)

1. FIPS module path documentation - No docs on how users configure OpenSSL to find FIPS files
2. Missing error handling in image loading (vlinux.yml:202-212) - Glob patterns could fail silently
3. Hardcoded just version (vrelease.yml:45) - version 1.42.4 could become outdated
4. Missing download cache for macOS - Linux has caching, macOS doesn't
5. Architecture assumptions - Assumes ubuntu-latest is always x86_64

Good Practices Observed

• ✅ Pinned action versions with SHA commits
• ✅ Proper minimal permissions
• ✅ Build provenance attestations
• ✅ Concurrency control
• ✅ persist-credentials: false on checkouts

Suggested Fixes

The most impactful fixes would be:

1. Add input validation to vrelease.yml
2. Add download caching to vmacos.yml
3. Add error handling to the image loading loops

---

🤖 Generated with Claude Code

[nickvines]

Fixed aarch64 build failures

Root cause: Cross-compiling aarch64 on x86_64 runners ran out of disk space (~14GB limit on ubuntu-latest)

Solution: Use native aarch64 builds on namespace-profile-linux-arm runners

Changes in commit 447c1d3:

• ✅ Build job now uses namespace-profile-linux-arm for aarch64 targets
• ✅ Added crate-build matrix to build pythonbuild on both architectures
• ✅ Added aarch64 Docker images (build.debian9, gcc.debian9)
• ✅ Dynamic artifact download based on target architecture

This eliminates cross-compilation overhead and provides native aarch64 builds like the upstream workflow.

---

🤖 Generated with Claude Code

[nickvines]

Updated to large caching namespace runners

Commit 150f837 switches to the larger caching runners:

• namespace-profile-ubuntu-22-04-amd64-x86-64-large-caching for x86_64
• namespace-profile-ubuntu-22-04-amd64-arm-large-caching for aarch64

These should provide better performance and more disk space for the builds.

---

🤖 Generated with Claude Code

[nickvines]

Investigating Docker image loading issue

The builds are failing with:

docker.errors.ImageNotFound: No such image: sha256:1149...
dockerfile parse error on line 1: unknown instruction: /home/runner/...

Root cause: Docker images aren't being loaded before the build runs.

Debug commit e2b35a0:

• Added debug output to show files in build directory
• Added file existence checks before decompressing/loading
• Shows loaded Docker images after the load step

This will help us see:

1. Are the image artifacts being downloaded?
2. Are they in the expected location?
3. Are they being decompressed correctly?
4. Are they being loaded into Docker?

Once we see the debug output in the next CI run, we can fix the actual issue.

---

🤖 Generated with Claude Code

[nickvines]

Fixed Docker image ID mismatch (commit d21033f)

Root cause: Docker Buildx with containerd snapshotter returns a different image ID than what docker load assigns.

• Image job writes: sha256:1149f658... (config digest from buildx)
• docker load assigns: sha256:1bd3abe6... (actual manifest digest)
• pythonbuild/docker.py looks for 1149f658..., doesn't find it, triggers broken fallback

Solution: Capture the loaded image ID from docker load output and update the ID files:

LOADED_ID=$(docker load --input $f 2>&1 | grep "Loaded image ID:" | awk '{print $4}')
echo "$LOADED_ID" > "$ID_FILE"

This is a known issue with Docker Buildx + containerd (see buildx PR #3136).

The builds should now succeed!

---

🤖 Generated with Claude Code

[nickvines]

Updated macOS to use namespace runner (commit 2a10d68)

Switched vmacos workflow to namespace-profile-mac-small-tahoe for all macOS builds.

All workflows now use namespace runners:

• ✅ vlinux x86_64: namespace-profile-ubuntu-22-04-amd64-x86-64-large-caching
• ✅ vlinux aarch64: namespace-profile-ubuntu-22-04-amd64-arm-large-caching (native ARM64)
• ✅ vmacos aarch64: namespace-profile-mac-small-tahoe

---

🤖 Generated with Claude Code

[nickvines]

Expert Review by Opus Agent

Overall: APPROVE WITH MINOR CONCERNS

The PR is well-architected with good security practices. The FIPS implementation is correct, and the workflow design follows best practices.

Critical Issues (2)

1. Security: -fvisibility=hidden removed globally (build-openssl-3.5.sh:46)

   • Increases symbol visibility attack surface
   • Trade-off for FIPS module to work
   • Recommendation: Document this limitation

2. Security: Missing input validation (vrelease.yml)

   • tag and sha inputs used directly in shell commands
   • Recommendation: Add regex validation for SHA (40 hex) and tag format

Recommendations

1. Add download caching to macOS - Linux has it, macOS doesn't
2. Add FIPS runtime verification - Verify the FIPS module can actually be loaded
3. Document FIPS usage - Explain how to enable FIPS mode at runtime
4. Add error handling - Check that Docker images were actually loaded

What's Good

• ✅ Pinned action versions with SHA hashes
• ✅ Minimal permissions
• ✅ Build provenance attestations
• ✅ Clean separation from upstream workflows
• ✅ Fixed Docker image ID mismatch issue (d21033f)
• ✅ FIPS modules correctly built and copied

Testing Gaps

• No FIPS mode runtime verification test
• Missing Python 3.15 in workflows (release.rs supports it but workflows don't build it)

Maintainability

Concern: src/release.rs changes will conflict heavily on upstream sync.

Recommendation: Consider environment variable approach for target filtering instead of modifying release.rs.

Verdict

The PR achieves its goals with good engineering practices. The concerns are minor and can be addressed in follow-ups. Most important: add input validation to vrelease.yml before using in production.

---

🤖 Generated with Claude Code

[nickvines]

Added README.md (commit ec7d68b)

Comprehensive documentation covering:

0. How to Release

• Sync from upstream
• Ensure workflows succeed
• Trigger release via GitHub UI or CLI
• What artifacts are produced

1. FIPS Setup

• Implementation details
• Where FIPS files are located
• How to enable FIPS mode at runtime
• Limitations:
  • -fvisibility=hidden removal trade-off
  • musl not supported
  • FIPS-enabled ≠ FIPS-compliant

2. Workflows

• vlinux.yml architecture and features
• vmacos.yml architecture and features
• vrelease.yml manual release process
• Design decisions explained

3. Additional Sections

• PR creation guidelines (always use --repo verkada/python-build-standalone)
• Troubleshooting guide
• Upstream sync strategy

---

🤖 Generated with Claude Code