[kdelee]

Summary

• add the shared core_ca_relpath_pattern_idx index through the pulp-service plugin migration
• create it with CONCURRENTLY IF NOT EXISTS and make the migration reversible
• support testing optimized file, index.html, and directory-listing query patterns without installing duplicate equivalent indexes

The index is a B-tree on core_contentartifact.relative_path using text_pattern_ops, which makes LIKE 'prefix%' / Django __startswith indexable under the production collation and also supports equality probes.

This is the shared index for pulpcore #8249 and #8262.

Closes #1641

Validation

• ruff check
• python -m py_compile
• git diff --check
• repository pre-commit checks

Summary by Sourcery

Add a reversible shared relative-path pattern index for efficient content queries.

Enhancements:

• Add a shared, concurrently created relative-path pattern index to support optimized content query patterns without duplicate equivalent indexes.
• Make index creation safe for concurrent migration workers and restrict reversal to indexes owned by this migration.

[sourcery-ai]

Reviewer's Guide

Introduces a non-transactional Django migration that safely creates the shared core_ca_relpath_pattern_idx concurrently with text_pattern_ops, supports optimized relative-path prefix/equality queries, and reverses only indexes explicitly identified as owned by this migration.

Sequence diagram for concurrent relative_path index migration

sequenceDiagram
    participant DjangoMigration
    participant SchemaEditor
    participant PostgreSQL

    DjangoMigration->>SchemaEditor: create_relative_path_index(apps, schema_editor)
    SchemaEditor->>PostgreSQL: _index_info(cursor)
    PostgreSQL-->>SchemaEditor: index metadata
    alt index exists and is valid
        SchemaEditor-->>DjangoMigration: return without creating duplicate
    else index exists but is invalid
        SchemaEditor-->>DjangoMigration: raise RuntimeError
    else index does not exist
        SchemaEditor->>PostgreSQL: cursor.execute(_CREATE_INDEX_SQL)
        PostgreSQL-->>SchemaEditor: concurrent B-tree index created
        SchemaEditor->>PostgreSQL: cursor.execute(_COMMENT_INDEX_SQL)
    end

    DjangoMigration->>SchemaEditor: drop_relative_path_index(apps, schema_editor)
    SchemaEditor->>PostgreSQL: _index_info(cursor)
    PostgreSQL-->>SchemaEditor: index metadata
    alt index comment matches _INDEX_MARKER
        SchemaEditor->>PostgreSQL: cursor.execute(_DROP_INDEX_SQL)
    else index is shared or not owned by migration
        SchemaEditor-->>DjangoMigration: leave index intact
    end

Loading

File-Level Changes

Change	Details	Files
Add a concurrently managed, shared PostgreSQL pattern index for content artifact relative paths.	Create a B-tree text_pattern_ops index to support prefix and equality lookups. Run the migration outside a transaction because concurrent index operations require it. Use IF NOT EXISTS and existing-index checks to avoid duplicate equivalent indexes and preserve indexes owned elsewhere. Refuse to overwrite an existing invalid index rather than dropping it automatically.	pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py
Make index ownership and rollback behavior explicit and safe.	Tag indexes created by this migration with a PostgreSQL comment marker. On reverse migration, drop the index concurrently only when the ownership marker matches. Add the associated issue/change-log entry.	pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py CHANGES/1641.misc

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1641	Provide the shared B-tree index on core_contentartifact.relative_path using text_pattern_ops, with the specified name, to support prefix and equality queries.	✅
#1641	Deploy the index independently through a pulp-service migration, without adding duplicate equivalent indexes for the related query candidates.	✅
#1641	Make the migration non-atomic and create and reverse the index with concurrent operations.	✅

Possibly linked issues

• Add indexes to enable testing optimized query patterns for content file, index.html, and directory-listing requests #1641: The PR adds the requested shared index, using concurrent creation and removal to support testing the optimized query patterns.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="16" />
<code_context>
+    operations = [
+        migrations.RunSQL(
+            sql=(
+                "CREATE INDEX CONCURRENTLY IF NOT EXISTS "
+                '"core_ca_relpath_pattern_idx" ON "core_contentartifact" '
+                'USING btree ("relative_path" text_pattern_ops)'
</code_context>
<issue_to_address>
**Invalid indexes survive retries**

When a concurrent index build fails or is interrupted after leaving an invalid index with this name, `CREATE INDEX CONCURRENTLY IF NOT EXISTS` sees the name, skips rebuilding, and Django records the migration as applied; PostgreSQL cannot use the invalid index, so optimized content queries lose the intended index support.

Remove or rebuild an invalid same-name index before creating the index, rather than letting `IF NOT EXISTS` skip it.
</issue_to_address>

### Comment 2
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="21-22" />
<code_context>
+                'USING btree ("relative_path" text_pattern_ops)'
+            ),
+            reverse_sql=(
+                'DROP INDEX CONCURRENTLY IF EXISTS "core_ca_relpath_pattern_idx"'
+            ),
+        ),
+    ]
</code_context>
<issue_to_address>
**Rollback removes shared index**

When an index with this name exists before the migration is applied, `reverse_sql` drops the named index even though the forward `IF NOT EXISTS` skipped creating it, so rollback removes a pre-existing shared index and callers lose its query acceleration.

Track whether this migration created the index and only drop it on reversal when it did.
</issue_to_address>

Sourcery assessment

Approval pending. 2 findings to address first.

Blocking findings: pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:16, pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:22

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[kdelee]

@sourcery-ai review

[sourcery-ai]

New security issues found

[sourcery-ai]

New security issues found

[sourcery-ai]

Sourcery assessment

Approved.

[sourcery-ai]

Hey - I've found 3 security issues, and 4 other issues

Security issues:

• Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. (link)
• Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. (link)
• Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option. (link)

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="32-34" />
<code_context>
+    """Create the index and mark it so reversal never removes a shared pre-existing index."""
+    with schema_editor.connection.cursor() as cursor:
+        info = _index_info(cursor)
+        if info and not info[0]:
+            # CREATE INDEX ... IF NOT EXISTS skips invalid indexes too, so remove one before retrying.
+            cursor.execute(_DROP_INDEX_SQL)
+            info = None
+
</code_context>
<issue_to_address>
**Concurrent migrations disrupt index builds**

When multiple migration workers run this migration against the same schema, `create_relative_path_index` treats another worker’s in-progress index as stale and drops it; the workers then fail or leave the migration unapplied until a later retry.

Serialize the check, drop, and create sequence so one worker cannot drop another worker’s in-progress build.
</issue_to_address>

### Comment 2
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="31-40" />
<code_context>
+def create_relative_path_index(apps, schema_editor):  # noqa: ARG001
+    """Create the index and mark it so reversal never removes a shared pre-existing index."""
+    with schema_editor.connection.cursor() as cursor:
+        info = _index_info(cursor)
+        if info and not info[0]:
+            # CREATE INDEX ... IF NOT EXISTS skips invalid indexes too, so remove one before retrying.
+            cursor.execute(_DROP_INDEX_SQL)
+            info = None
+
+        if info is None:
+            cursor.execute(_CREATE_INDEX_SQL)
+            # COMMENT provides durable ownership information for the reverse migration.
+            cursor.execute(_COMMENT_INDEX_SQL)
+
+
</code_context>
<issue_to_address>
**Reversal drops a shared index**

When another actor creates the same-named index after `_index_info` checks and before `CREATE INDEX` runs, `CREATE INDEX ... IF NOT EXISTS` skips creation, but the migration writes its ownership marker on the other actor’s index; reversal then drops that shared index.

Make the check and creation race-safe, and write the ownership marker only on an index this migration created.
</issue_to_address>

### Comment 3
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="38-40" />
<code_context>
+            info = None
+
+        if info is None:
+            cursor.execute(_CREATE_INDEX_SQL)
+            # COMMENT provides durable ownership information for the reverse migration.
+            cursor.execute(_COMMENT_INDEX_SQL)
+
+
</code_context>
<issue_to_address>
**Reversal leaves the created index behind**

When index creation succeeds but the process stops or the comment step fails before the ownership marker is written, on retry, `create_relative_path_index` sees a valid index and skips both creation and commenting, so the migration is recorded without its ownership marker and reversal leaves the index installed.

Ensure retries write the ownership marker for an existing valid index created by this migration.
</issue_to_address>

### Comment 4
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="6-8" />
<code_context>
+_INDEX_NAME = "core_ca_relpath_pattern_idx"
+_INDEX_MARKER = "pulp-service:0024:core_ca_relpath_pattern_idx"
+_CREATE_INDEX_SQL = (
+    "CREATE INDEX CONCURRENTLY IF NOT EXISTS "
+    f'"{_INDEX_NAME}" ON "core_contentartifact" '
+    'USING btree ("relative_path" text_pattern_ops)'
+)
+_DROP_INDEX_SQL = f'DROP INDEX CONCURRENTLY IF EXISTS "{_INDEX_NAME}"'
</code_context>
<issue_to_address>
**Equivalent indexes are duplicated**

When an equivalent `relative_path` pattern index already exists under a different name, `create_relative_path_index` checks only `_INDEX_NAME` and creates another index, adding redundant storage and write overhead.

Check for an equivalent index before creating one, regardless of its name.

Also at `pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:38`.
</issue_to_address>

### Comment 5
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="34" />
<code_context>
            cursor.execute(_DROP_INDEX_SQL)
</code_context>
<issue_to_address>
**security (python.sqlalchemy.security.sqlalchemy-execute-raw-query):** Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

*Source: opengrep*
</issue_to_address>

### Comment 6
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="40" />
<code_context>
            cursor.execute(_COMMENT_INDEX_SQL)
</code_context>
<issue_to_address>
**security (python.sqlalchemy.security.sqlalchemy-execute-raw-query):** Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

*Source: opengrep*
</issue_to_address>

### Comment 7
<location path="pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py" line_range="48" />
<code_context>
            cursor.execute(_DROP_INDEX_SQL)
</code_context>
<issue_to_address>
**security (python.sqlalchemy.security.sqlalchemy-execute-raw-query):** Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

*Source: opengrep*
</issue_to_address>

Sourcery assessment

Approval pending. 7 findings to address first.

Blocking findings: pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:34, pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:40, pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:40, pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:8, pulp_service/pulp_service/app/migrations/0024_contentartifact_relative_path_pattern_index.py:34, and 2 more

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[kdelee]

@sourcery-ai review

Sourcery withdrew this approval because the latest commits introduced blocking findings.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Approved.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

Sourcery withdrew this approval because the latest commits introduced blocking findings.

[kdelee]

@sourcery-ai review

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Approved.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[dkliban]

/retest