Skip to content

mount trusted CA bundle in runner pods #1258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremyeder merged 3 commits into from
Apr 10, 2026
+248 −2
Merged

mount trusted CA bundle in runner pods #1258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0301b72
mount trusted CA bundle in runner pods
ktdreyer Apr 8, 2026
66d0a3a
Merge branch 'main' into feat/trusted-ca-bundle-runner-pods
Apr 10, 2026
a41b987
fix: set GOTOOLCHAIN=auto in public-api Dockerfile
Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions components/operator/internal/handlers/sessions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
intstr "k8s.io/apimachinery/pkg/util/intstr"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/util/retry"
)

Expand Down Expand Up @@ -1453,6 +1454,9 @@ func handleAgenticSessionEvent(obj *unstructured.Unstructured) error {
}
}

// Mount trusted CA bundle if present in the session namespace (e.g. OpenShift CA injection)
applyTrustedCABundle(config.K8sClient, sessionNamespace, pod)

// NOTE: Google credentials are now fetched at runtime via backend API
// No longer mounting credentials.json as volume
// This ensures tokens are always fresh and automatically refreshed
Expand Down Expand Up @@ -2391,6 +2395,55 @@ func deleteAmbientMlflowObservabilitySecret(ctx context.Context, namespace, excl
return nil
}

// applyTrustedCABundle mounts the cluster's injected CA bundle into the runner
// container so it trusts cluster-internal TLS certificates (e.g. on OpenShift).
// Clusters without the ConfigMap are silently unaffected.
func applyTrustedCABundle(k8sClient kubernetes.Interface, namespace string, pod *corev1.Pod) {
cm, err := k8sClient.CoreV1().ConfigMaps(namespace).Get(
context.TODO(), types.TrustedCABundleConfigMapName, v1.GetOptions{})
if errors.IsNotFound(err) {
return
}
if err != nil {
log.Printf("Warning: failed to check for %s ConfigMap in %s: %v",
types.TrustedCABundleConfigMapName, namespace, err)
return
}
if _, ok := cm.Data["ca-bundle.crt"]; !ok {
if _, ok := cm.BinaryData["ca-bundle.crt"]; !ok {
log.Printf("Warning: %s ConfigMap in %s is missing required key ca-bundle.crt; skipping mount",
types.TrustedCABundleConfigMapName, namespace)
return
}
}
pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
Name: "trusted-ca-bundle",
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: types.TrustedCABundleConfigMapName,
},
},
},
})
for i := range pod.Spec.Containers {
if pod.Spec.Containers[i].Name == "ambient-code-runner" {
pod.Spec.Containers[i].VolumeMounts = append(
pod.Spec.Containers[i].VolumeMounts,
corev1.VolumeMount{
Name: "trusted-ca-bundle",
MountPath: "/etc/pki/tls/certs/ca-bundle.crt",
SubPath: "ca-bundle.crt",
ReadOnly: true,
},
)
log.Printf("Mounted %s ConfigMap to /etc/pki/tls/certs/ca-bundle.crt in runner container",
types.TrustedCABundleConfigMapName)
break
}
Comment on lines +2429 to +2443
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 8, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Mount the bundle into init-hydrate and state-sync too.

This only patches ambient-code-runner. init-hydrate lives in pod.Spec.InitContainers and is configured with repo/S3 inputs (Lines 901-946), and state-sync is configured for S3 (Lines 1230-1248). On private-CA clusters, those containers can still fail TLS before the main runner starts, so private repo hydration/state sync is still broken unless they get the same trust bundle when present.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/operator/internal/handlers/sessions.go` around lines 2199 - 2213,
The current patch only mounts the trusted CA bundle into the ambient-code-runner
container; update the code to also mount the same VolumeMount into init
containers and any other containers that need it (specifically target init
container named "init-hydrate" in pod.Spec.InitContainers and the container
named "state-sync" in pod.Spec.Containers) so they receive the trusted CA before
network operations; reuse the same corev1.VolumeMount (Name "trusted-ca-bundle",
MountPath "/etc/pki/tls/certs/ca-bundle.crt", SubPath "ca-bundle.crt", ReadOnly
true) and log using types.TrustedCABundleConfigMapName, ensuring you iterate
pod.Spec.InitContainers to patch "init-hydrate" and pod.Spec.Containers to patch
both "state-sync" and "ambient-code-runner" (avoid duplicate mounts if already
present).

Copy link
Copy Markdown
Contributor Author

@ktdreyer ktdreyer Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderabbitai how can those require the CA?

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ktdreyer Good challenge — it depends on the deployment:

  • init-hydrate's backend API call (BACKEND_API_URL) uses plain http:// — no TLS at all, so definitely no CA needed there.
  • S3 (s3Endpoint) — if it's internal MinIO over HTTP (the typical default here), no CA needed. But if an operator configures an internal HTTPS S3 endpoint backed by a corporate CA, init-hydrate and state-sync would fail TLS.
  • Git clones in init-hydrate — if repos are hosted on an internal Gitlab/Gitea with a private CA cert, cloning over HTTPS would fail without the bundle.

So the containers can need it, but only when:

  1. The S3 endpoint is HTTPS + corporate CA, or
  2. Git repos are on a private-CA HTTPS host

If your target environments use plain-HTTP MinIO and public-CA git hosts, the runner container is the only one making connections (to Anthropic/Claude APIs, etc.) that realistically need the bundle. In that case you can safely ignore my comment. Your call on whether the edge cases (HTTPS S3, private git) are in scope for this PR.

Copy link
Copy Markdown
Contributor Author

@ktdreyer ktdreyer Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither of those conditions are true. We'll ignore this for now.

}
}

// LEGACY: getBackendAPIURL removed - AG-UI migration
// Workflow and repo changes now call runner's REST endpoints directly

Expand Down
189 changes: 189 additions & 0 deletions components/operator/internal/handlers/sessions_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package handlers

import (
"context"
"fmt"
"testing"

"ambient-code-operator/internal/config"
Expand All @@ -13,6 +14,7 @@ import (
"k8s.io/apimachinery/pkg/runtime"
k8stypes "k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes/fake"
clienttesting "k8s.io/client-go/testing"
)

// setupTestClient initializes a fake Kubernetes client for testing
Expand Down Expand Up @@ -568,6 +570,193 @@ func TestDeleteAmbientVertexSecret_NotFound(t *testing.T) {
}
}

// TestApplyTrustedCABundle_ConfigMapPresent verifies that applyTrustedCABundle adds the volume
// and VolumeMount when the trusted-ca-bundle ConfigMap exists in the session namespace.
func TestApplyTrustedCABundle_ConfigMapPresent(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"ca-bundle.crt": "--- fake CA data ---",
},
}
setupTestClient(cm)

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 1 {
t.Fatalf("expected 1 volume, got %d", len(pod.Spec.Volumes))
}
vol := pod.Spec.Volumes[0]
if vol.Name != "trusted-ca-bundle" {
t.Errorf("expected volume name 'trusted-ca-bundle', got %q", vol.Name)
}
if vol.ConfigMap == nil || vol.ConfigMap.Name != types.TrustedCABundleConfigMapName {
t.Errorf("expected ConfigMap volume sourced from %q", types.TrustedCABundleConfigMapName)
}

mounts := pod.Spec.Containers[0].VolumeMounts
if len(mounts) != 1 {
t.Fatalf("expected 1 VolumeMount, got %d", len(mounts))
}
m := mounts[0]
if m.Name != "trusted-ca-bundle" {
t.Errorf("expected mount name 'trusted-ca-bundle', got %q", m.Name)
}
if m.MountPath != "/etc/pki/tls/certs/ca-bundle.crt" {
t.Errorf("unexpected MountPath: %q", m.MountPath)
}
if m.SubPath != "ca-bundle.crt" {
t.Errorf("expected SubPath 'ca-bundle.crt', got %q", m.SubPath)
}
if !m.ReadOnly {
t.Error("expected ReadOnly=true")
}
}

// TestApplyTrustedCABundle_ConfigMapAbsent verifies that applyTrustedCABundle leaves the pod
// unchanged when the trusted-ca-bundle ConfigMap is not present in the session namespace.
func TestApplyTrustedCABundle_ConfigMapAbsent(t *testing.T) {
setupTestClient() // no ConfigMap

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}

// TestApplyTrustedCABundle_ExistingMountsPreserved verifies that applyTrustedCABundle appends
// to, rather than replacing, existing VolumeMounts on the runner container.
func TestApplyTrustedCABundle_ExistingMountsPreserved(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"ca-bundle.crt": "--- fake CA data ---",
},
}
setupTestClient(cm)

existingMount := corev1.VolumeMount{
Name: "runner-token",
MountPath: "/var/run/secrets/ambient",
ReadOnly: true,
}
pod := &corev1.Pod{
Spec: corev1.PodSpec{
Volumes: []corev1.Volume{
{Name: "runner-token"},
},
Containers: []corev1.Container{
{
Name: "ambient-code-runner",
VolumeMounts: []corev1.VolumeMount{existingMount},
},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 2 {
t.Fatalf("expected 2 volumes (runner-token + trusted-ca-bundle), got %d", len(pod.Spec.Volumes))
}
mounts := pod.Spec.Containers[0].VolumeMounts
if len(mounts) != 2 {
t.Fatalf("expected 2 VolumeMounts, got %d", len(mounts))
}
// Existing mount must still be at index 0
if mounts[0].Name != "runner-token" {
t.Errorf("expected first mount to be 'runner-token', got %q", mounts[0].Name)
}
if mounts[1].Name != "trusted-ca-bundle" {
t.Errorf("expected second mount to be 'trusted-ca-bundle', got %q", mounts[1].Name)
}
}

// TestApplyTrustedCABundle_MissingKey verifies that applyTrustedCABundle leaves the pod
// unchanged when the ConfigMap exists but lacks the ca-bundle.crt key.
func TestApplyTrustedCABundle_MissingKey(t *testing.T) {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: types.TrustedCABundleConfigMapName,
Namespace: "session-ns",
},
Data: map[string]string{
"wrong-key.pem": "--- fake CA data ---",
},
}
setupTestClient(cm)

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes when key is missing, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts when key is missing, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}

// TestApplyTrustedCABundle_APIError verifies that applyTrustedCABundle leaves the pod
// unchanged when the ConfigMap GET returns a non-NotFound error.
func TestApplyTrustedCABundle_APIError(t *testing.T) {
fakeClient := fake.NewSimpleClientset()
fakeClient.PrependReactor("get", "configmaps", func(action clienttesting.Action) (bool, runtime.Object, error) {
return true, nil, fmt.Errorf("connection refused")
})
config.K8sClient = fakeClient

pod := &corev1.Pod{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "ambient-code-runner"},
},
},
}

applyTrustedCABundle(config.K8sClient, "session-ns", pod)

if len(pod.Spec.Volumes) != 0 {
t.Errorf("expected no volumes on API error, got %d", len(pod.Spec.Volumes))
}
if len(pod.Spec.Containers[0].VolumeMounts) != 0 {
t.Errorf("expected no VolumeMounts on API error, got %d", len(pod.Spec.Containers[0].VolumeMounts))
}
}
Comment on lines +573 to +758
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Tests miss required coverage for init-hydrate and state-sync mounts.

These new tests only verify ambient-code-runner. The PR objective requires mounting the trusted CA bundle into init-hydrate, state-sync, and ambient-code-runner; with current assertions, an implementation that skips the first two would still pass.

Proposed test adjustment 
 func TestApplyTrustedCABundle_ConfigMapPresent(t *testing.T) {
@@
 	pod := &corev1.Pod{
 		Spec: corev1.PodSpec{
 			Containers: []corev1.Container{
+				{Name: "init-hydrate"},
+				{Name: "state-sync"},
 				{Name: "ambient-code-runner"},
 			},
 		},
 	}
@@
-	mounts := pod.Spec.Containers[0].VolumeMounts
-	if len(mounts) != 1 {
-		t.Fatalf("expected 1 VolumeMount, got %d", len(mounts))
-	}
-	m := mounts[0]
-	if m.Name != "trusted-ca-bundle" {
-		t.Errorf("expected mount name 'trusted-ca-bundle', got %q", m.Name)
-	}
-	if m.MountPath != "/etc/pki/tls/certs/ca-bundle.crt" {
-		t.Errorf("unexpected MountPath: %q", m.MountPath)
-	}
-	if m.SubPath != "ca-bundle.crt" {
-		t.Errorf("expected SubPath 'ca-bundle.crt', got %q", m.SubPath)
-	}
-	if !m.ReadOnly {
-		t.Error("expected ReadOnly=true")
-	}
+	expected := map[string]bool{
+		"init-hydrate":        false,
+		"state-sync":          false,
+		"ambient-code-runner": false,
+	}
+	for _, c := range pod.Spec.Containers {
+		for _, m := range c.VolumeMounts {
+			if m.Name == "trusted-ca-bundle" &&
+				m.MountPath == "/etc/pki/tls/certs/ca-bundle.crt" &&
+				m.SubPath == "ca-bundle.crt" &&
+				m.ReadOnly {
+				if _, ok := expected[c.Name]; ok {
+					expected[c.Name] = true
+				}
+			}
+		}
+	}
+	for name, ok := range expected {
+		if !ok {
+			t.Fatalf("expected trusted-ca-bundle mount on container %q", name)
+		}
+	}
 }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@components/operator/internal/handlers/sessions_test.go` around lines 573 -
758, The tests for applyTrustedCABundle only assert mounts on the
"ambient-code-runner" container, so they miss regressions where "init-hydrate"
and "state-sync" are not receiving the trusted CA; update the
TestApplyTrustedCABundle_* cases to create pods with three containers named
"init-hydrate", "state-sync", and "ambient-code-runner" (or add those containers
to existing pod fixtures) and assert that the volumes list includes the
trusted-ca-bundle and that each of the three containers has a VolumeMount named
"trusted-ca-bundle" with MountPath "/etc/pki/tls/certs/ca-bundle.crt", SubPath
"ca-bundle.crt" and ReadOnly=true; for the ExistingMountsPreserved test also
ensure pre-existing mounts for each container remain in-place and that the
trusted-ca-bundle mount is appended.


// TestDeleteAmbientVertexSecret_NilAnnotations tests handling of secret with nil annotations
func TestDeleteAmbientVertexSecret_NilAnnotations(t *testing.T) {
secret := &corev1.Secret{
Expand Down
4 changes: 4 additions & 0 deletions components/operator/internal/types/resources.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ const (
// AmbientVertexSecretName is the name of the secret containing Vertex AI credentials
AmbientVertexSecretName = "ambient-vertex"

// TrustedCABundleConfigMapName is the CA bundle ConfigMap injected by OpenShift or provisioned
// manually on private-CA clusters. See issue #1247.
TrustedCABundleConfigMapName = "trusted-ca-bundle"

// CopiedFromAnnotation is the annotation key used to track secrets copied by the operator
CopiedFromAnnotation = "vteam.ambient-code/copied-from"
)
Expand Down
4 changes: 2 additions & 2 deletions components/public-api/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ USER 0

# Copy go mod files first for caching
COPY go.mod go.sum* ./
RUN go mod download
RUN GOTOOLCHAIN=auto go mod download

# Copy source code
COPY . .

# Build the binary
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o public-api .
RUN GOTOOLCHAIN=auto CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o public-api .

# Runtime stage
FROM registry.access.redhat.com/ubi9/ubi-minimal:9.7
Expand Down
Loading