Skip to content

fix: preserve explicit projected volumes; strip only kubelet auto-mount - #5

Merged
vercel-eddie merged 1 commit into
mainfrom
eddie/preserve-projected-volumes
May 18, 2026
Merged

vercel-eddie merged 1 commit into
mainfrom
eddie/preserve-projected-volumes

Conversation

@vercel-eddie

Copy link
Copy Markdown
Collaborator

Why

StripOrphanedVolumes was dropping every projected volume from the source deployment spec. The original intent (per its doc comment) was to strip the kubelet-injected default ServiceAccount token mount — kube-api-access-<random> — whose name has a per-pod random suffix that would collide if a source spec already declared one with that exact prefix.

The blanket strip also took out legitimately-declared projected volumes that the kubelet does not auto-add back:

  • azure-identity-token (audience: api://AzureADTokenExchange) used for AKS workload identity → Cosmos DB / Azure AD auth.
  • Explicit aws-iam-token declarations.
  • Any other projected ConfigMap / Secret / DownwardAPI volumes a service intentionally declares.

For AWS IRSA the EKS Pod Identity webhook silently re-injects the projected aws-iam-token volume at admission, so AWS-backed services worked by accident. Azure has no such webhook in our staging cluster — the projected volume must be in the spec or the token file never materializes.

Symptom for any @vercel/cosmosdb-using service running through bridge:

AuthenticationRequiredError: ENOENT: no such file or directory,
open '/var/run/secrets/azure/tokens/token'

…and every Cosmos call 500s. Because the projected mount also isn't in the live pod's VolumeMounts after StripOrphanedVolumes runs, it's missing from podVolumeMountPaths, so it's also missing from the devcontainer feature's copyFiles list and never gets pulled into the local container either.

Fix

Narrow the filter from "all projected volumes" to "volumes named kube-api-access-*". The kubelet still auto-mounts a fresh kube-api-access-<random> at pod-create time regardless of what's in the spec, so the original defense is unchanged. Everything else (explicit AKS / IRSA / custom projections) survives intact.

Test plan

Added transforms_test.go covering:

  • Explicit azure-identity-token / aws-iam-token projections survive; their mounts on the container also survive.
  • kube-api-access-* volumes (and their mounts) are still stripped.

Validation against a real service

Reproduces the bug on eddie/devbox-project-image-fallback in vercel/api:

  1. Without this fix: bridge create -s /tmp/rendered/manifests.yaml api-devbox → bridge pod has no azure-identity-token volume → POST /v1/devbox → 500 with the ENOENT above.
  2. With this fix (build locally, bridge remove api-devbox && bridge create -s …): bridge pod gets the projected volume → live pod has the mount → copyFiles includes /var/run/secrets/azure/tokens → local devcontainer has the federated token at the expected path → MSAL exchanges via STS → Cosmos auth succeeds.

I'll attach a second-pass log once I've rebuilt + re-curl'd to confirm.

Follow-ups (separate, not in this PR)

  • Projected SA tokens rotate (10 min TTL for AKS, ~1h for legacy, 24h for IRSA). The current copyFiles is a one-shot at devcontainer-create time. If a bridge session outlives the token TTL, MSAL will fail on refresh. Worth either documenting the limitation or wiring a refresh loop into the devcontainer feature.

StripOrphanedVolumes was dropping every projected volume from the
source deployment spec, but the original intent (per its comment) was
to strip the kubelet-injected default ServiceAccount token mount —
which has a `kube-api-access-<random>` name per pod and would collide
if it ended up duplicated in the bridge pod's spec.

The blanket strip also took out legitimately-declared projected
volumes that the kubelet does NOT auto-add back:

  - `azure-identity-token` (audience: `api://AzureADTokenExchange`)
    used for AKS-style workload identity → Cosmos DB / Azure AD auth
  - explicit `aws-iam-token` declarations
  - any other projected ConfigMap / Secret / DownwardAPI volumes a
    service intentionally declares

For AWS IRSA the EKS Pod Identity webhook silently re-injects the
volume at admission, so AWS-backed services worked anyway. Azure has
no such webhook in venus — the volume must be in the spec or the
projected token file never materializes. Services using
@vercel/cosmosdb hit:

  AuthenticationRequiredError: ENOENT: no such file or directory,
  open '/var/run/secrets/azure/tokens/token'

and every Cosmos call 500s. Bridge then propagates a stripped pod
spec into the devcontainer's `copyFiles` list — because the Azure
mount isn't on the pod at all, it's not in the list and not copied
locally either.

Narrow the filter to match only `kube-api-access-*` volume names so
explicit projections survive. The kubelet still auto-mounts a fresh
`kube-api-access-<random>` at pod-create time, so we keep the
original defense without taking out everything else.

# Test

Added unit tests covering:
- explicit `azure-identity-token` / `aws-iam-token` projections
  survive; their mounts survive
- `kube-api-access-*` volumes are still stripped; their mounts are
  still stripped
@vercel-eddie
vercel-eddie marked this pull request as ready for review May 18, 2026 22:55
@vercel-eddie
vercel-eddie merged commit e352798 into main May 18, 2026
3 of 5 checks passed
@vercel-eddie
vercel-eddie deleted the eddie/preserve-projected-volumes branch May 18, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant