WIP: Handling of merge-index-image for containerized IIB

[JAVGan]

Work in Progress - This is a draft!

Handling of merge-index-image for containerized IIB

Refers to CLOUDDST-29408

Signed-off-by: Jonathan Gangi [email protected]

Assisted-by: Cursor/Gemini

Summary by Sourcery

Introduce a containerized workflow for merge-index-image requests and refactor bundle-diff logic for reuse.

New Features:

• Add a containerized merge task that orchestrates merging source and target index images via Git-driven FBC and Konflux pipelines.
• Validate bundle pullspecs in parallel before merging and handle missing bundles and deprecations in the containerized flow.

Enhancements:

• Extract reusable logic for computing bundles missing from the source index into a dedicated helper used by both legacy and containerized merge flows.
• Wire the merge-index-image API endpoint to the new containerized merge task and register the task in worker configuration.

[sourcery-ai]

Reviewer's Guide

Refactors merge-index-image bundle-diff logic into a reusable helper and introduces a new containerized merge task that drives an IIB/Konflux-based FBC merge workflow, wiring the new Celery task into the merge API and worker config.

Sequence diagram for the new containerized merge-index-image workflow

sequenceDiagram
    actor User
    participant API as IIB_API
    participant Celery as Celery_Broker
    participant Worker as Celery_Worker
    participant Git as Git_Server
    participant Reg as Container_Registry
    participant Konflux as Konflux_Pipeline
    participant IIBReg as IIB_Registry

    User->>API: POST /api/v1/merge_index_image
    API->>Celery: handle_containerized_merge_request.apply_async(args)
    Celery->>Worker: Dispatch handle_containerized_merge_request

    Worker->>Worker: reset_docker_config()
    Worker->>Worker: set_request_state(preparing_request_for_merge)
    Worker->>Worker: prepare_request_for_build(RequestConfigMerge)
    Worker->>Worker: Opm.set_opm_version(target_index_resolved)
    Worker->>Worker: _update_index_image_build_state()

    Worker->>Git: resolve_git_url(source_from_index)
    Git-->>Worker: index_git_repo
    Worker->>Git: clone_git_repo(index_git_repo, branch)
    Git-->>Worker: local_git_repo_path

    Worker->>Worker: get_hidden_index_database(source_from_index_resolved)
    Worker->>Worker: get_hidden_index_database(target_index_resolved)
    Worker->>Worker: _get_present_bundles(source_index_db)
    Worker->>Worker: _get_present_bundles(target_index_db)

    Worker->>Worker: spawn ValidateBundlesThread for source bundles
    Worker->>Worker: spawn ValidateBundlesThread for target bundles
    Worker->>Reg: skopeo_inspect(bundle_pullspec) * N
    Reg-->>Worker: bundle exists or error
    Worker->>Worker: join all ValidateBundlesThread

    Worker->>Worker: set_request_state(adding_missing_bundles)
    Worker->>Worker: get_missing_bundles_from_target_to_source(...)
    Worker->>Worker: add_max_ocp_version_property(missing_bundle_paths)

    Worker->>Worker: intermediary_db = _get_or_create_temp_index_db_file()
    Worker->>Worker: _opm_registry_add(intermediary_db, missing_bundle_paths)

    Worker->>Worker: get_bundles_from_deprecation_list()
    Worker->>Worker: get_bundles_latest_version()
    loop deprecate in chunks
        Worker->>Worker: opm_registry_deprecatetruncate(intermediary_db, chunk)
    end

    Worker->>Worker: bundles_in_db = get_list_bundles(intermediary_db)
    Worker->>Worker: operators_in_db = [bundle.packageName]

    Worker->>Worker: fbc_dir = opm_migrate(intermediary_db)
    Worker->>Worker: merge_catalogs_dirs(catalog_from_db, git_catalog)
    Worker->>Worker: opm_validate(fbc_dir)
    Worker->>Worker: write_build_metadata(local_git_repo_path,...)

    alt overwrite_target_index_token is not set
        Worker->>Git: create_mr(request_id, local_git_repo_path,...)
        Git-->>Worker: mr_details
    else overwrite_target_index_token is set
        Worker->>Git: commit_and_push(local_git_repo_path,...)
    end

    Worker->>Git: last_commit_sha = get_last_commit_sha()

    Worker->>Konflux: find_pipelinerun(last_commit_sha)
    Konflux-->>Worker: pipelinerun
    Worker->>Konflux: wait_for_pipeline_completion(pipelinerun_name)
    Konflux-->>Worker: run (succeeded)

    Worker->>Konflux: get_pipelinerun_image_url(pipelinerun_name, run)
    Konflux-->>Worker: image_url

    Worker->>Worker: output_pull_specs = get_list_of_output_pullspec(request_id, build_tags)
    loop for each output_pull_spec
        Worker->>IIBReg: _skopeo_copy(docker://image_url, docker://output_pull_spec)
    end

    Worker->>Worker: _update_index_image_pull_spec(..., output_pull_spec,...)

    Worker->>IIBReg: push_index_db_artifact(intermediary_db, operators_in_db)
    IIBReg-->>Worker: original_index_db_digest

    alt MR was created
        Worker->>Git: close_mr(mr_details)
    end

    Worker->>Worker: set_request_state(complete, success_message)

    Celery-->>API: async completion via DB/state
    User-->>API: Poll request status
    API-->>User: Final merged index image pullspec

Loading

Class diagram for new containerized merge task and helpers

classDiagram
    class ValidateBundlesThread {
        +Queue bundles_queue
        +Exception exception
        +__init__(bundles_queue)
        +run() void
    }

    class Thread {
        <<abstract>>
        +start() void
        +join() void
        +run() void
    }

    ValidateBundlesThread --|> Thread

    class BuildContainerizedMergeModule {
        +handle_containerized_merge_request(source_from_index, deprecation_list, request_id, binary_image, target_index, overwrite_target_index, overwrite_target_index_token, distribution_scope, binary_image_config, build_tags, graph_update_mode, ignore_bundle_ocp_version, index_to_gitlab_push_map, parallel_threads) void
    }

    class BuildMergeIndexImageModule {
        +get_missing_bundles_from_target_to_source(source_index_bundles, target_index_bundles, source_from_index, ocp_version, target_index, ignore_bundle_ocp_version) (List~BundleImage~, List~BundleImage~)
    }

    class RequestConfigMerge {
        +_binary_image : str
        +overwrite_target_index_token : str
        +source_from_index : str
        +target_index : str
        +distribution_scope : str
        +binary_image_config : str
    }

    class Opm {
        +opm_version : str
        +set_opm_version(target_index) void
    }

    class GitUtils {
        +resolve_git_url(from_index, index_repo_map) str
        +clone_git_repo(repo_url, branch, token_name, git_token, dest_path) void
        +create_mr(request_id, local_repo_path, repo_url, branch, commit_message) Dict
        +commit_and_push(request_id, local_repo_path, repo_url, branch, commit_message) void
        +close_mr(mr_details, repo_url) void
        +get_last_commit_sha(local_repo_path) str
        +get_git_token(repo_url) (str, str)
    }

    class KonfluxUtils {
        +find_pipelinerun(commit_sha) List
        +wait_for_pipeline_completion(pipelinerun_name) Dict
        +get_pipelinerun_image_url(pipelinerun_name, run) str
    }

    class ContainerizedUtils {
        +write_build_metadata(repo_path, opm_version, ocp_version, distribution_scope, binary_image, request_id) void
        +get_list_of_output_pullspec(request_id, build_tags) List~str~
        +cleanup_on_failure(mr_details, last_commit_sha, index_git_repo, overwrite_from_index, request_id, from_index, index_repo_map, original_index_db_digest, reason) void
        +push_index_db_artifact(request_id, from_index, index_db_path, operators, operators_in_db, overwrite_from_index, request_type) str
    }

    class BuildTasks {
        +_update_index_image_build_state(request_id, prebuild_info) void
        +_get_present_bundles(index_db_path, base_dir) (List~BundleImage~, List~str~)
        +_update_index_image_pull_spec(output_pull_spec, request_id, arches, from_index, overwrite_from_index, overwrite_from_index_token, resolved_prebuild_from_index, add_or_rm, is_image_fbc, index_repo_map) void
        +_skopeo_copy(source, destination, copy_all, exc_msg) void
    }

    class OpMOperations {
        +_opm_registry_add(base_dir, index_db, bundles) void
        +_get_or_create_temp_index_db_file(base_dir, from_index) str
        +opm_registry_deprecatetruncate(base_dir, index_db, bundles) void
        +opm_migrate(index_db, base_dir) (str, str)
        +opm_validate(fbc_dir_path) void
        +get_list_bundles(index_db, base_dir) List
        +get_hidden_index_database(from_index, base_dir) str
    }

    class Utils {
        +prepare_request_for_build(request_id, request_config) Dict
        +request_logger
        +add_max_ocp_version_property(bundle_paths, base_dir) void
        +reset_docker_config() void
        +skopeo_inspect(image, option, return_json) str
        +get_bundles_from_deprecation_list(all_bundles, deprecation_list) List~str~
    }

    class FBCUtils {
        +merge_catalogs_dirs(catalog_from_db, catalog_from_git) void
    }

    class Config {
        +get_worker_config() Config
        +iib_deprecate_bundles_limit : int
    }

    BuildContainerizedMergeModule ..> ValidateBundlesThread : uses
    BuildContainerizedMergeModule ..> BuildMergeIndexImageModule : calls get_missing_bundles_from_target_to_source
    BuildContainerizedMergeModule ..> RequestConfigMerge : constructs
    BuildContainerizedMergeModule ..> Opm : sets opm_version
    BuildContainerizedMergeModule ..> GitUtils : git operations
    BuildContainerizedMergeModule ..> KonfluxUtils : pipeline operations
    BuildContainerizedMergeModule ..> ContainerizedUtils : metadata, artifacts
    BuildContainerizedMergeModule ..> BuildTasks : index image tasks
    BuildContainerizedMergeModule ..> OpMOperations : index.db operations
    BuildContainerizedMergeModule ..> Utils : helpers
    BuildContainerizedMergeModule ..> FBCUtils : merge catalogs
    BuildContainerizedMergeModule ..> Config : deprecate_bundles_limit

    Config ..> ContainerizedUtils
    OpMOperations ..> BuildTasks
    Utils ..> BuildTasks

Loading

File-Level Changes

Change	Details	Files
Extract bundle-diff logic into a reusable helper and adjust existing merge implementation to use it.	Introduced get_missing_bundles_from_target_to_source to compute bundles present in the target index but missing from the source and track bundles with invalid OCP version ranges. Moved logging/state updates and index rebuild responsibilities out of the new helper so it becomes a pure computation/validation function. Refactored _add_bundles_missing_in_source to call the new helper, rebuild missing_bundle_paths from its result, and preserve existing index rebuild behavior.	iib/workers/tasks/build_merge_index_image.py
Wire merge-index-image API to use a new containerized merge task instead of the legacy one.	Changed merge_index_image API to enqueue handle_containerized_merge_request instead of handle_merge_request. Registered the new build_containerized_merge Celery task module in the worker configuration so it is loaded and routable.	iib/web/api_v1.py iib/workers/config.py
Add a new containerized merge worker task that validates bundles, merges index DBs, updates FBC catalogs in Git, triggers a Konflux pipeline, and publishes the resulting index image and index.db artifact.	Implemented handle_containerized_merge_request Celery task to orchestrate merge preparation, including request prebuild resolution, OPM version selection, and Git repo discovery for the source index. Added ValidateBundlesThread to parallelize skopeo_inspect-based validation of bundle pullspecs from both source and target index DBs. Reused existing build helpers to read present bundles from hidden index databases, compute missing and invalid bundles with get_missing_bundles_from_target_to_source, add missing bundles via opm registry add, and process deprecations via opm deprecatetruncate using latest bundle versions. Migrated the merged index.db to FBC with opm_migrate, merged the generated catalog with the Git-hosted catalog using merge_catalogs_dirs, regenerated and validated the final FBC config with opm_validate, and wrote build metadata into the Git repo. Integrated with GitLab by cloning the appropriate repo/branch, either creating a merge request for throw-away builds or committing/pushing directly when overwriting, and used Konflux helpers to find the pipelinerun by commit SHA, wait for completion, and obtain the built image URL. Copied the Konflux-built image to all output pullspecs with skopeo, updated IIB request state and index pull specs via _update_index_image_pull_spec, pushed the updated index.db artifact with push_index_db_artifact when overwriting, and added cleanup_on_failure handling that rolls back Git/MR and index.db state on errors.	iib/workers/tasks/build_containerized_merge.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[lipoja]

In theory we should not use this. Because there is no hidden db anymore.
You need to pull both db files as artifacts.

[lipoja]

The source and target index images are useless - meaning we just take the OCP version from it and that's it. And it is done based on LABEL I believe.
The sqlite database is in registry and catalog is in gitlab repo.