[carlosthe19916]

Why this is one atomic PR

The pulp_python bump and the attestation config must ship together because the running image and the config have to agree in every environment: pulp_python 3.36.2 reads ATTESTATION_VERIFICATION_KEY (one global key) while 3.38.0 reads ATTESTATION_VERIFICATION_KEYS (a per-domain map). Shipping only the bump (new code, old config) or only the config (new env var, old code) leaves key-signed attestation uploads with no usable key and rejects them for the domains that rely on them. So the bump (PULP-2643) and the per-domain config (PULP-2647) land as a single change.

PULP-2643 — bump + drop patch

• pulp-python pin 3.36.2 -> 3.38.0 (pulp_service/requirements.txt + version table in images/assets/patches/CLAUDE.md).
• Removed patch 0048 (upstreamed in pulp/pulp_python#1327): the .patch, its COPY/RUN in the Dockerfile, and its docs entry. The vendored-key COPY is retained (the map points at the key copied from images/assets/keys/SIGSTORE-redhat-release3.pem to /etc/pki/sigstore/SIGSTORE-redhat-release3).
• Python 3.11 -> 3.12 (Dockerfile ARG PYTHON_VERSION) — 3.38.0 requires >=3.12; cascades via ${PYTHON_VERSION}.

PULP-2647 — per-domain config

Replaces the single PULP_ATTESTATION_VERIFICATION_KEY with the per-domain map in deploy/clowdapp.yaml (api/content/worker containers + template parameter):

- name: PULP_ATTESTATION_VERIFICATION_KEYS
  value: >
    '@json {"public-trusted-libraries": "/etc/pki/sigstore/SIGSTORE-redhat-release3", "lightwell": "/etc/pki/sigstore/SIGSTORE-redhat-release3", "lightwell-stage": "/etc/pki/sigstore/SIGSTORE-redhat-release3"}'

Domains derived from a full prod inventory -> blast radius (stored provenance): only these 3 use the key-signed path (all Pulp User, no cert); the other 22 python domains have 0 attestations.

Validation

• oc process --local renders the map; no PULP_ATTESTATION_VERIFICATION_KEY references remain; clowdapp YAML parses.
• No other patch depends on 0048; vendored key present at images/assets/keys/SIGSTORE-redhat-release3.pem.

Refs: PULP-2643, PULP-2647, epic PULP-2642.

🤖 Generated with Claude Code

Summary by Sourcery

Upgrade pulp_python and migrate attestation verification to per-domain key configuration while aligning the image, deployment, development, and CI environments.

New Features:

• Configure attestation verification with per-domain key maps across deployments, development settings, and CI environments.

Bug Fixes:

• Ensure key-signed attestations are stored and verified correctly with the upgraded pulp_python integration.

Enhancements:

• Upgrade pulp_python to 3.38.0 and Python to 3.12.
• Remove the downstream attestation verification patch now that the functionality is available upstream.

CI:

• Update functional-test environments and Python package paths for the new attestation configuration and Python 3.12 runtime.

Deployment:

• Replace the global attestation verification key deployment parameter with a per-domain key map shared by API, content, and worker containers.

Documentation:

• Update the patch and dependency documentation for the pulp_python upgrade and removal of the obsolete attestation patch.

Tests:

• Update attestation verification tests to validate per-domain configuration and confirm submitted attestations are persisted.

[sourcery-ai]

Reviewer's Guide

This PR upgrades pulp-python to 3.38.0 with Python 3.12, removes the downstream attestation-verification patch that is now upstream, and migrates deployment, development, CI, and functional tests to pulp-python’s per-domain ATTESTATION_VERIFICATION_KEYS configuration while preserving the vendored signing key for the supported production domains.

File-Level Changes

Change	Details	Files
Upgrade the Python plugin and runtime, removing the downstream attestation patch now that support is upstream.	Bump pulp-python from 3.36.2 to 3.38.0 and Python from 3.11 to 3.12. Remove patch 0048 from the image build and patch documentation. Retain the vendored Red Hat signing key for configuration-based verification.	Dockerfile pulp_service/requirements.txt images/assets/patches/0048-Re-enable-attestation-verification-with-vendored-key.patch images/assets/patches/CLAUDE.md
Migrate attestation verification from one global key to a per-domain key map across deployment and local development environments.	Replace PULP_ATTESTATION_VERIFICATION_KEY with PULP_ATTESTATION_VERIFICATION_KEYS in API, content, and worker containers. Configure production domains public-trusted-libraries, lightwell, and lightwell-stage to use the vendored key. Configure development and CI environments with a default-domain test key map.	deploy/clowdapp.yaml dev-container/settings.py .tekton/pulp-deploy-and-test.yaml
Align attestation functional coverage and CI tooling with the new configuration and provenance behavior.	Validate the default-domain entry in the verification key map during end-to-end tests. Update expected provenance publisher assertions for pulp-python 3.38.0 behavior. Update CI package paths and functional test commands from Python 3.11 to 3.12.	pulp_service/pulp_service/tests/functional/test_attestation_verification.py .tekton/pulp-deploy-and-test.yaml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="deploy/clowdapp.yaml" line_range="390" />
<code_context>
             value: ${{PULP_DB_ENCRYPTION_KEY}}
-          - name: PULP_ATTESTATION_VERIFICATION_KEY
-            value: ${{PULP_ATTESTATION_VERIFICATION_KEY}}
+          - name: PULP_ATTESTATION_VERIFICATION_KEYS
+            value: ${{PULP_ATTESTATION_VERIFICATION_KEYS}}
           - name: PULP_CACHE_ENABLED
</code_context>
<issue_to_address>
**Attestation tests reject uploads**

When the pull-request functional tests run with the generated test signing key, the test deployment provides only `PULP_ATTESTATION_VERIFICATION_KEY` and the fixture reads `ATTESTATION_VERIFICATION_KEY`, while `pulp-python` 3.38.0 verifies with the per-domain `ATTESTATION_VERIFICATION_KEYS` map. The generated key is therefore unavailable to verification, so the attestation tests fail.

Update the functional-test deployment and fixture to configure and read the per-domain verification-key map.

Also at `deploy/clowdapp.yaml:391`, `deploy/clowdapp.yaml:752`, `pulp_service/requirements.txt:4`.
</issue_to_address>

### Comment 2
<location path="Dockerfile" line_range="2" />
<code_context>
 FROM registry.access.redhat.com/ubi9/ubi
-ARG PYTHON_VERSION=3.11
+ARG PYTHON_VERSION=3.12
 ENV PYTHONUNBUFFERED=0
 ENV DJANGO_SETTINGS_MODULE=pulpcore.app.settings
</code_context>
<issue_to_address>
**Functional tests cannot import bindings**

When the pull-request Tekton pipeline installs bindings and runs tests against the Python 3.12 image, the Tekton setup still writes namespace-package shims and sets `PYTHONPATH` to Python 3.11 site-packages, but the image uses Python 3.12. The generated bindings are not found and functional-test imports fail.

Update the Tekton shim paths and `PYTHONPATH` to use the Python 3.12 site-packages directory.

Also at `pulp_service/requirements.txt:4`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and if a domain is mapped to an overly trusted verification key, attestations signed by that key could be accepted for artifacts in that domain. Those artifacts and their provenance records can remain available after a revert, so reverting alone would not undo the trust decision's effects.

Blocking findings: deploy/clowdapp.yaml:390, Dockerfile:2

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[carlosthe19916]

@sourcery-ai review

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path=".tekton/pulp-deploy-and-test.yaml" line_range="608" />
<code_context>
                           "name": "pulp-api",
                           "env": [
-                            {"name": "PULP_ATTESTATION_VERIFICATION_KEY", "value": "/etc/pki/attestation/test-key.pem"}
+                            {"name": "PULP_ATTESTATION_VERIFICATION_KEYS", "value": "@json {\"default\": \"/etc/pki/attestation/test-key.pem\"}"}
                           ],
                           "volumeMounts": [
</code_context>
<issue_to_address>
**Attestation tests fail to start**

When tekton applies the deployment patch directly with `oc`, `oc` preserves the `@json` prefix in `PULP_ATTESTATION_VERIFICATION_KEYS`, so the setting is not a dictionary and the fixture's `.get("default")` fails; the attestation functional tests do not run successfully.

Pass a JSON mapping as the environment value without the Clowder-specific `@json` marker.

Also at `.tekton/pulp-deploy-and-test.yaml:638-639`.
</issue_to_address>

### Comment 2
<location path="pulp_service/pulp_service/tests/functional/test_attestation_verification.py" line_range="246" />
<code_context>
     publisher = prov_obj.provenance["attestation_bundles"][0]["publisher"]
-    assert publisher["builder_id"] == "https://konflux-ci.dev/calunga"
-    assert publisher["build_type"] == "https://konflux-ci.dev/PythonWheelBuild@v1"
-    assert publisher["kind"] == "konflux-ci.dev"
+    assert publisher["kind"] == "Pulp User"
</code_context>
<issue_to_address>
**Dropped attestations pass upload test**

When the content-upload handler creates the provenance bundle but omits its attestations, `test_konflux_attestation_via_content_upload` checks only the publisher kind at line 245, so an upload path that creates a Pulp User provenance bundle but drops the submitted attestation still passes.

Assert that the stored provenance contains the submitted attestation, including its envelope.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and the per-domain key map defines which signing key is trusted to verify attestations for the configured domains. If that mapping grants trust to the wrong key or domains, attestations can be accepted under the wrong trust boundary; reverting stops future acceptance but does not undo provenance already recorded.

Blocking findings: .tekton/pulp-deploy-and-test.yaml:608, pulp_service/pulp_service/tests/functional/test_attestation_verification.py:246

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[dkliban]

Let's have a default value here, but we should keep the actual values for stage and prod deployments in app-interface.

[carlosthe19916]

Thanks, good point. I moved it to app-interface as suggested here https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/217576

and then keep here only the default value in this repository