ci(security): scan before push, sign + attest release images (#233, fixes #1213)

[Defilan]

Description

Completes the supply-chain story for LLMKube releases: Trivy now scans every image before it is pushed, and every published image is cosign-signed with an SLSA build provenance attestation and an SBOM attestation, all keyless.

Scan before push (fixes #1213)

Previously goreleaser release --clean built and pushed all 4 multi-arch images atomically, and a separate trivy-scan job scanned the already-published images. A vulnerable image was therefore already in GHCR by the time the scan ran (this is why 0.9.9 shipped with a HIGH grpc CVE and needed the 0.9.10 roll-forward).

Now a hack/scan-images.sh step builds each image for linux/amd64 (the binary built the same CGO_ENABLED=0 way GoReleaser builds it, so the assembled image is representative) and Trivy-scans it locally, before GoReleaser builds the multi-arch manifests and pushes. A CRITICAL/HIGH finding fails the job before anything reaches the registry. The standalone post-push trivy-scan job is deleted. A committed .trivyignore is the documented escape valve so one unpatchable upstream CVE cannot freeze every release.

Sign + attest (keyless)

After push, each of the 4 images (controller, foreman-operator, foreman-agent, router-proxy) is signed and attested by digest:

• cosign sign (keyless, Sigstore/GitHub OIDC)
• actions/attest-build-provenance (SLSA build provenance)
• actions/attest-sbom with a syft SPDX SBOM

The release binary archives (dist/*.tar.gz) also get SLSA provenance. The workflow self-verifies after signing (fails the release if cosign verify / gh attestation verify do not pass). A workflow_dispatch dry-run exercises the whole chain against a throwaway :provenance-test tag without cutting a release. docs/security/verifying-releases.md documents how consumers verify.

Third-party actions added here (setup-trivy, cosign-installer, setup-crane, sbom-action) are pinned to immutable commit SHAs.

.goreleaser.yaml is not touched (digests captured via crane digest in the workflow).

Type of Change

• Bug fix (scan-before-push, [BUG] Release pipeline: trivy-scan gates AFTER publication instead of before #1213) + new feature (signing/provenance/SBOM, Add container image scanning and supply chain security #233)

How Has This Been Tested?

• actionlint clean; YAML parses; shellcheck hack/scan-images.sh clean
• All 4 release binaries compile with the scan build flags (CGO_ENABLED=0 GOOS=linux GOARCH=amd64)
• Pre-merge gate: a workflow_dispatch dry-run must run green on the fork (build -> scan -> push :provenance-test -> sign -> attest -> gh attestation verify), and a deliberately-vulnerable base must fail the scan with nothing pushed. The dry-run derives the GHCR org and cosign identity from the repo context so it works on a fork. This is the real end-to-end proof; code inspection alone does not prove the Sigstore/attestation chain.

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my code
• I have made corresponding changes to the documentation (docs/security/verifying-releases.md, README link)
• My changes generate no new warnings
• New and existing tests pass locally (actionlint / shellcheck / compile checks)
• AI assistance disclosed: implemented via AI subagents under human orchestration; a whole-branch AI review caught two Critical issues (Trivy never installed; cosign identity case) which were fixed and re-reviewed. I directed the design and review and own this PR.

Assisted-by: Claude Code (design, implementation, and multi-stage review under human direction); I reviewed and own the final result.

Fixes #233
Fixes #1213

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!