Skip to content

Fix Secrets Mounting - #69

Merged
avasilevskii merged 4 commits into
openshift-psap:mainfrom
avasilevskii:fix-secret-mount
Apr 28, 2026
Merged

Fix Secrets Mounting#69
avasilevskii merged 4 commits into
openshift-psap:mainfrom
avasilevskii:fix-secret-mount

Conversation

@avasilevskii

@avasilevskii avasilevskii commented Apr 28, 2026

Copy link
Copy Markdown

Fix naming pattern for vault secrets, copy kubeconfig to target ns and add FOURNOS_SECRETS envvar

Summary by CodeRabbit

  • New Features

    • Kubeconfig secrets are now copied per job with ownership tracking and idempotent creation.
  • Improvements

    • Secret references accept simplified Vault entry names (no "vault-" prefix); CRD docs updated.
    • Better error handling and clearer logging when secret operations fail.
  • Tests

    • Scheduling and secret-related tests updated to validate job-scoped kubeconfig naming and copied secret contents.

…dd FOURNOS_SECRETS envvar

Signed-off-by: avasilev <avasilev@redhat.com>
@coderabbitai

coderabbitai Bot commented Apr 28, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@avasilevskii has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 49 minutes and 14 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6efba900-ab30-4fe0-9415-97c0ddb69cc0

📥 Commits

Reviewing files that changed from the base of the PR and between 2d81208 and 6c4d18d.

📒 Files selected for processing (1)
  • fournos/core/clusters.py
📝 Walkthrough

Walkthrough

Operator now resolves Vault-entry refs (given without the vault- prefix) to Kubernetes Secret names, copies kubeconfig Secrets into the operator namespace with ownerReferences, updates Tekton task env to expose mounted secrets path, and adjusts tests and CRD docs to the new naming/usage patterns.

Changes

Cohort / File(s) Summary
Tekton Task Configuration
config/forge/workflows/tasks.yaml
Inject FOURNOS_SECRETS=/var/run/secrets/fournos into the Tekton step environment to expose the mounted secrets directory.
Secret Resolution & Copying Logic
fournos/core/clusters.py
Added copy_kubeconfig_secret and _vault_secret_name; _resolve_secret_ref and copy_secret now resolve user refs by prepending the configured vault- pattern, validate Vault-entry label, and handle idempotent copy creation (409).
Execution Handler
fournos/handlers/execution.py
When no PipelineRun exists, kubeconfig is obtained by copying a job-specific kubeconfig Secret with an ownerRef; failures set job to FAILED, update status message/condition, delete Kueue workload, and early-exit.
CRD Schema
manifests/crd.yaml
Clarified spec.secretRefs description: users supply Vault entry names without vault-; operator prefixes vault- to find corresponding Secrets labeled fournos.dev/vault-entry=true.
Mocks & Fixtures
dev/mock-resolve/resolve.sh, dev/mock-secrets.yaml, tests/conftest.py
Updated mock resolve payload placeholder ("placeholder"), changed mock secret managed-by label to fournos-mock, and made PipelineRun kubeconfig param derive from PipelineRun name ({name}-kubeconfig).
Tests — Scheduling
tests/test_scheduling.py
Expect kubeconfig-secret param to be job-derived (<job>-kubeconfig); assert copied Secret exists in operator namespace, contains data.kubeconfig, and has ownerReferences to the FournosJob.
Tests — Secret Refs
tests/test_secret_refs.py
Pre-create noop resolve Job and set spec.secretRefs directly to Vault entry identifiers; adjust assertions to use VAULT_ENTRY and remove resolver-polling helper and unused constants.

Sequence Diagram(s)

sequenceDiagram
    participant User as User
    participant Operator as Operator
    participant K8sAPI as Kubernetes API
    participant Tekton as Tekton PipelineRun
    User->>Operator: Create FournosJob(spec.secretRefs = "my-entry")
    Operator->>K8sAPI: Resolve Secret `vault-my-entry` (get Secret)
    K8sAPI-->>Operator: Return Secret (or 404)
    Operator->>K8sAPI: Create copy in operator namespace with ownerReference
    K8sAPI-->>Operator: 201 Created (or 409 if exists)
    Operator->>Tekton: Create PipelineRun with param kubeconfig-secret=<job>-kubeconfig
    Tekton->>K8sAPI: Pod mounts operator namespace secret at /var/run/secrets/fournos
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Suggested labels

lgtm

Poem

🐰 I hopped through code to find the clue,

vault names trimmed and secrets new.
I copy, label, tuck them tight,
so Tekton finds them in the night.
Hooray — small hops, but everything's in view!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix Secrets Mounting' accurately reflects the main changes: kubeconfig secret copying, Vault secret naming pattern fixes, and adding the FOURNOS_SECRETS environment variable.
Docstring Coverage ✅ Passed Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@kpouget

kpouget commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

looks good at first glance, let's get it tested !

@kpouget

kpouget commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 28, 2026
@kpouget

kpouget commented Apr 28, 2026

Copy link
Copy Markdown
Collaborator

/test fournos-deploy-wip

@avasilevskii

Copy link
Copy Markdown
Author

/test fournos

@avasilevskii

Copy link
Copy Markdown
Author

/test deploy-fournos-wip

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
tests/test_secret_refs.py (1)

150-152: Tighten secret-refs assertion to avoid false positives.

Line 150 uses containment; exact comparison is safer for catching malformed values.

💡 Proposed tightening
-        assert VAULT_ENTRY in refs_param, (
-            f"PipelineRun secret-refs should contain {VAULT_ENTRY!r}, "
+        assert refs_param == [VAULT_ENTRY], (
+            f"PipelineRun secret-refs should equal {[VAULT_ENTRY]!r}, "
             f"got {refs_param!r}"
         )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/test_secret_refs.py` around lines 150 - 152, The current assertion in
tests/test_secret_refs.py uses containment (assert VAULT_ENTRY in refs_param)
which can give false positives; update the assertion to require exact equality
of the secret-refs value by asserting refs_param == VAULT_ENTRY (and keep a
clear error message), locating and replacing the assertion that references
VAULT_ENTRY and refs_param so the test fails on malformed or extra characters
rather than only missing substrings.
tests/conftest.py (1)

467-467: Derive stale kubeconfig secret from the helper input name.

Line 467 hardcodes a specific value inside a generic helper. Prefer f"{name}-kubeconfig" to keep fixture behavior consistent across callers.

♻️ Proposed fix
-                {"name": "kubeconfig-secret", "value": "test-stale-kubeconfig"},
+                {"name": "kubeconfig-secret", "value": f"{name}-kubeconfig"},
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/conftest.py` at line 467, Replace the hardcoded "test-stale-kubeconfig"
value for the "kubeconfig-secret" entry with a value derived from the helper's
input parameter `name` (e.g., use f"{name}-kubeconfig") so the fixture generates
a kubeconfig secret name based on the caller-provided `name`; update the dict
entry that currently contains "kubeconfig-secret" to set its "value" using that
derived string in the helper in tests/conftest.py.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@fournos/core/clusters.py`:
- Around line 143-146: The copy_secret implementation currently calls
self._k8s.read_namespaced_secret(secret_name, settings.secrets_namespace) and
lets ApiException(404) escape, losing the richer ref+namespace context used
elsewhere; update copy_secret to catch k8s.client.exceptions.ApiException (or
ApiException) around the read_namespaced_secret call, translate 404 responses
into a KeyError that includes the original ref and settings.secrets_namespace
(or simply call/reuse self._resolve_secret_ref(ref) to get the resolved name and
raise a KeyError with that info), and re-raise other ApiExceptions unchanged so
diagnostics remain consistent with other code paths (mentioning copy_secret,
_vault_secret_name, _resolve_secret_ref, read_namespaced_secret, ApiException,
KeyError).

---

Nitpick comments:
In `@tests/conftest.py`:
- Line 467: Replace the hardcoded "test-stale-kubeconfig" value for the
"kubeconfig-secret" entry with a value derived from the helper's input parameter
`name` (e.g., use f"{name}-kubeconfig") so the fixture generates a kubeconfig
secret name based on the caller-provided `name`; update the dict entry that
currently contains "kubeconfig-secret" to set its "value" using that derived
string in the helper in tests/conftest.py.

In `@tests/test_secret_refs.py`:
- Around line 150-152: The current assertion in tests/test_secret_refs.py uses
containment (assert VAULT_ENTRY in refs_param) which can give false positives;
update the assertion to require exact equality of the secret-refs value by
asserting refs_param == VAULT_ENTRY (and keep a clear error message), locating
and replacing the assertion that references VAULT_ENTRY and refs_param so the
test fails on malformed or extra characters rather than only missing substrings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 877a26aa-e757-45f1-b524-9818dc8f8ec1

📥 Commits

Reviewing files that changed from the base of the PR and between d6aebd5 and 8f53e62.

📒 Files selected for processing (8)
  • config/forge/workflows/tasks.yaml
  • dev/mock-resolve/resolve.sh
  • fournos/core/clusters.py
  • fournos/handlers/execution.py
  • manifests/crd.yaml
  • tests/conftest.py
  • tests/test_scheduling.py
  • tests/test_secret_refs.py

Comment thread fournos/core/clusters.py Outdated
Comment on lines +143 to +146
secret_name = self._vault_secret_name(ref)
source = self._k8s.read_namespaced_secret(
secret_name, settings.secrets_namespace
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Preserve detailed missing-secret errors in copy_secret.

Line 143-146 now raises raw ApiException on 404, which drops the richer ref + namespace context provided elsewhere. Convert 404 to KeyError here (or reuse _resolve_secret_ref) to keep diagnostics consistent.

🐛 Proposed fix
         secret_name = self._vault_secret_name(ref)
-        source = self._k8s.read_namespaced_secret(
-            secret_name, settings.secrets_namespace
-        )
+        try:
+            source = self._k8s.read_namespaced_secret(
+                secret_name, settings.secrets_namespace
+            )
+        except client.exceptions.ApiException as exc:
+            if exc.status == 404:
+                raise KeyError(
+                    f"Secret {secret_name!r} (ref {ref!r}) not found in "
+                    f"namespace {settings.secrets_namespace}"
+                ) from exc
+            raise
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
secret_name = self._vault_secret_name(ref)
source = self._k8s.read_namespaced_secret(
secret_name, settings.secrets_namespace
)
secret_name = self._vault_secret_name(ref)
try:
source = self._k8s.read_namespaced_secret(
secret_name, settings.secrets_namespace
)
except client.exceptions.ApiException as exc:
if exc.status == 404:
raise KeyError(
f"Secret {secret_name!r} (ref {ref!r}) not found in "
f"namespace {settings.secrets_namespace}"
) from exc
raise
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@fournos/core/clusters.py` around lines 143 - 146, The copy_secret
implementation currently calls self._k8s.read_namespaced_secret(secret_name,
settings.secrets_namespace) and lets ApiException(404) escape, losing the richer
ref+namespace context used elsewhere; update copy_secret to catch
k8s.client.exceptions.ApiException (or ApiException) around the
read_namespaced_secret call, translate 404 responses into a KeyError that
includes the original ref and settings.secrets_namespace (or simply call/reuse
self._resolve_secret_ref(ref) to get the resolved name and raise a KeyError with
that info), and re-raise other ApiExceptions unchanged so diagnostics remain
consistent with other code paths (mentioning copy_secret, _vault_secret_name,
_resolve_secret_ref, read_namespaced_secret, ApiException, KeyError).

@psap-forge-bot

Copy link
Copy Markdown

🟢 Test of 'fournos_deploy --project-source' succeeded after 00 hours 10 minutes 23 seconds 🟢

• Link to the test results.

• No reports index generated...

Test configuration:

/test deploy-fournos-wip

Execution logs

Signed-off-by: avasilev <avasilev@redhat.com>
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Apr 28, 2026
@openshift-ci

openshift-ci Bot commented Apr 28, 2026

Copy link
Copy Markdown

New changes are detected. LGTM label has been removed.

@openshift-ci

openshift-ci Bot commented Apr 28, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from kpouget. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: avasilev <avasilev@redhat.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
fournos/core/clusters.py (1)

143-146: ⚠️ Potential issue | 🟠 Major

Avoid double-read TOCTOU after secret validation.

Line 143 validates via _resolve_secret_ref(ref), then Lines 144-146 read the same Secret again. If the Secret changes/deletes between these calls, copy_secret can fail with a raw ApiException and lose the contextual KeyError path.

🐛 Proposed fix (single-read helper)
+    def _resolve_secret(self, ref: str) -> tuple[str, client.V1Secret]:
+        secret_name = self._vault_secret_name(ref)
+        try:
+            secret = self._k8s.read_namespaced_secret(
+                secret_name, settings.secrets_namespace
+            )
+        except client.exceptions.ApiException as exc:
+            if exc.status == 404:
+                raise KeyError(
+                    f"Secret {secret_name!r} (ref {ref!r}) not found in "
+                    f"namespace {settings.secrets_namespace}"
+                ) from exc
+            raise
+        labels = secret.metadata.labels or {}
+        if labels.get(LABEL_VAULT_ENTRY) != "true":
+            raise KeyError(
+                f"Secret {secret_name!r} exists but is not a Vault-synced secret "
+                f"(missing {LABEL_VAULT_ENTRY}=true label)"
+            )
+        return secret_name, secret
+
     def _resolve_secret_ref(self, ref: str) -> str:
-        secret_name = self._vault_secret_name(ref)
-        try:
-            secret = self._k8s.read_namespaced_secret(
-                secret_name, settings.secrets_namespace
-            )
-        except client.exceptions.ApiException as exc:
-            if exc.status == 404:
-                raise KeyError(
-                    f"Secret {secret_name!r} (ref {ref!r}) not found in "
-                    f"namespace {settings.secrets_namespace}"
-                ) from exc
-            raise
-        labels = secret.metadata.labels or {}
-        if labels.get(LABEL_VAULT_ENTRY) != "true":
-            raise KeyError(
-                f"Secret {secret_name!r} exists but is not a Vault-synced secret "
-                f"(missing {LABEL_VAULT_ENTRY}=true label)"
-            )
+        secret_name, _ = self._resolve_secret(ref)
         logger.debug("Validated secretRef %s -> %s", ref, secret_name)
         return secret_name

     def copy_secret(self, ref: str, fjob_name: str, owner_ref: dict) -> ResolvedSecret:
-        secret_name = self._resolve_secret_ref(ref)
-        source = self._k8s.read_namespaced_secret(
-            secret_name, settings.secrets_namespace
-        )
+        secret_name, source = self._resolve_secret(ref)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@fournos/core/clusters.py` around lines 143 - 146, The code currently calls
_resolve_secret_ref(ref) to validate the secret and then calls
read_namespaced_secret again, risking a TOCTOU race; change to perform a single
read that both validates and returns the secret object (e.g. implement a helper
like _get_validated_secret(ref) or modify _resolve_secret_ref to return the
Secret instead of only the name) and use that single Secret in copy_secret;
ensure the helper catches ApiException and converts a not-found condition into
the existing KeyError path so callers (e.g. copy_secret) preserve the original
contextual error handling instead of exposing raw ApiException.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@fournos/core/clusters.py`:
- Around line 143-146: The code currently calls _resolve_secret_ref(ref) to
validate the secret and then calls read_namespaced_secret again, risking a
TOCTOU race; change to perform a single read that both validates and returns the
secret object (e.g. implement a helper like _get_validated_secret(ref) or modify
_resolve_secret_ref to return the Secret instead of only the name) and use that
single Secret in copy_secret; ensure the helper catches ApiException and
converts a not-found condition into the existing KeyError path so callers (e.g.
copy_secret) preserve the original contextual error handling instead of exposing
raw ApiException.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b56c74f7-3749-40be-8c9e-418d302173db

📥 Commits

Reviewing files that changed from the base of the PR and between 8f53e62 and 2d81208.

📒 Files selected for processing (5)
  • dev/mock-secrets.yaml
  • fournos/core/clusters.py
  • tests/conftest.py
  • tests/test_scheduling.py
  • tests/test_secret_refs.py
✅ Files skipped from review due to trivial changes (1)
  • dev/mock-secrets.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/conftest.py
  • tests/test_scheduling.py
  • tests/test_secret_refs.py

Signed-off-by: avasilev <avasilev@redhat.com>
@avasilevskii
avasilevskii merged commit 103007c into openshift-psap:main Apr 28, 2026
4 of 5 checks passed
@avasilevskii
avasilevskii deleted the fix-secret-mount branch April 28, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants