Image builds: implement the external BuildKit broker (in-sandbox builds are impossible under Kata + PodSecurity baseline)

[sabbour]

Problem

Container image builds (docker build, podman build, BuildKit) cannot be performed inside the
Kata executor sandbox. This was established by measurement during the PR #757 capability validation,
not by assumption, and each step of the chain was observed on the real cluster:

1. buildkitd --oci-worker-no-process-sandbox needs mapped root inside a user namespace.
2. With a single-id map (all we can create), runc fails mounting devpts — gid=5: invalid argument
   — because gid 5 has no representation in the namespace.
3. Mapping an id range requires an effective CAP_SETUID or the setuid helper newuidmap.
   newuidmap is not in the image (and adding a setuid binary to the sandbox image is exactly the
   kind of thing we should not do), and the sidecar cannot hold CAP_SETUID: with runAsNonRoot
   plus allowPrivilegeEscalation=false, added capabilities land in the bounding set only
   (CapPrm/CapEff are 0 — verified in-pod).
4. Running as real root without CAP_SYS_ADMIN fails at the first bind mount with EPERM.
5. CAP_SYS_ADMIN is rejected by the namespace's PodSecurity baseline admission level.

There is no remaining path that does not either break PodSecurity baseline or hand the sandbox
capabilities we have deliberately refused it.

Why not just add an elevated build container to the sandbox pod

Considered and rejected on security grounds, and this is the important part for reviewers. The
executor sidecar and the workspace PVC live in the same Kata VM. A container in that pod running
as root with CAP_SYS_ADMIN could mount the guest's virtiofs share and reach workspace content
belonging to the whole pod, defeating the per-run boundary the sandbox exists to provide. Elevation
must live outside the untrusted run's blast radius, not beside it.

Proposed solution

An external build broker, as already specified in
specs/agent-execution-sandbox/build-images-with-rootless-buildkit.md (spec issue #582, now closed):

• A separate, trusted BuildKit service (its own pod/node pool, its own identity, its own admission
  posture) that the sandbox talks to over a narrow, authenticated API.
• The sandbox sends a build context and receives a result. It never gains build privileges itself.
• The broker enforces its own quotas, registry credentials (never exposed to the run), and
  per-project scoping. Registry credentials staying out of the sandbox is a requirement, not a nicety.

Current behaviour (shipped in #757, so this fails honestly rather than mysteriously)

The sandbox capability contract reports:

image-build  RequiresExternalService
             "Container image builds require an external BuildKit broker; they cannot be
              performed inside the sandbox without capabilities the isolation model refuses."

This is asserted by test (SandboxCapabilityContractTests), so it cannot silently regress into a
false claim of support.

Scope

• Design and implement the broker service and its API.
• Wire the sandbox client so image-build moves from RequiresExternalService to Supported.
• Real AKS integration evidence on the fix(sandbox): unblock AgentHost warm pool on real AKS Kata with a hardened executor sidecar #757 standard: a fixture image built end-to-end and
  inspected, plus negative tests showing the sandbox gained no privileges and never saw registry
  credentials.
• Update docs/deep-dive/sandbox-pod-execution.md capability table.

References

• Spec: specs/agent-execution-sandbox/build-images-with-rootless-buildkit.md
• Closed spec issue: Spec: Enable AgentHost to build Docker/OCI images via rootless BuildKit on Kubernetes #582
• Evidence and rejected alternatives: PR fix(sandbox): unblock AgentHost warm pool on real AKS Kata with a hardened executor sidecar #757

[github-actions]

🔄 Ralph — Auto-Triage

Assigned to: Tank
Reason: Matched backend/API role keywords
Source: role-keyword

Ralph auto-triaged this issue using routing rules.
To reassign, swap the squad:* label.