Rework S3 CI cache prefix (HMS-5274) #1130
Open
+228
−57
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While the arguments parser marked the --distro option as optional, in reality it is always passed when calling the script. Moreover, some of the underlying functions would behave weirdly if `distro` would be set to `None` (e.g. create directories named "None"). Signed-off-by: Tomáš Hozza <[email protected]>
Move the osbuild nevra to the beginning of the image build cache S3 prefix. This is a preparation for not using osbuild NEVRA, but osbuild commit ref and used CI runner distro. The goal is to make it easy to determine the S3 prefix corresponding to checked out branch without the need to setup YUM repo and install RPMs on the CI runner. Signed-off-by: Tomáš Hozza <[email protected]>
Rename the function to have "dir_path" in its name to signal that the returned path has a trailing slash to comply with the S3 convention. Signed-off-by: Tomáš Hozza <[email protected]>
Add a helper function to get the host distro, instead of reading data from the os-release and constructing it in every place. Signed-off-by: Tomáš Hozza <[email protected]>
Add the runner distro version to the image build cache S3 prefix. This is a preparation for not using osbuild NEVRA, but osbuild commit ref and used CI runner distro. The goal is to make it easy to determine the S3 prefix corresponding to checked out branch without the need to setup YUM repo and install RPMs on the CI runner. Signed-off-by: Tomáš Hozza <[email protected]>
Replace the osbuild nevra in the image build cache S3 prefix with osbuild ref and prefix it with "osbuild-ref-" to make the meaning of the SHA clear. The goal is to make it easy to determine the S3 prefix corresponding to checked out branch without the need to setup YUM repo and install RPMs on the CI runner. Signed-off-by: Tomáš Hozza <[email protected]>
The function is no longer used by any code. Signed-off-by: Tomáš Hozza <[email protected]>
Let's unify the generation of build info dir path prefix into one function, instead of generating it in various ways in multiple places. This commit introduces a new function to generate the relative dir path prefix, depending on provided arguments. The existing code that generates the prefix on its own will be replaced in the following commits. Signed-off-by: Tomáš Hozza <[email protected]>
Refactor gen_build_info_s3_dir_path() to call gen_build_info_dir_path_prefix() to generate the S3 path, instead of constructing it on its own. Amend the test coverage accordingly. This will allow to use gen_build_info_s3_dir_path() in all places that construct S3 paths. Signed-off-by: Tomáš Hozza <[email protected]>
Use gen_build_info_s3_dir_path() for generating S3 path, instead of constructing it on its own. Signed-off-by: Tomáš Hozza <[email protected]>
Use gen_build_info_dir_path_prefix() for generating build info dir paths, instead of constructing it on its own. Signed-off-by: Tomáš Hozza <[email protected]>
The function is no longer used by any code, delete it. Signed-off-by: Tomáš Hozza <[email protected]>
The function is no longer used by any code, delete it. Signed-off-by: Tomáš Hozza <[email protected]>
Prefix the manifest_id value in the generated path prefix with "manifest-id-" to make it obvious what the value represents. Signed-off-by: Tomáš Hozza <[email protected]>
Modify `gen_build_info_dir_path_prefix()` and `gen_build_info_s3_dir_path()` functions to not require osbuild_ref and runner_distro to be passed explicitly as arguments. If not provided, the runner_distro defaults to the host distro and the osbuild_ref defaults to the pinned version for the runner_distro. This simplifies all the places which call these functions. Signed-off-by: Tomáš Hozza <[email protected]>
schutzbot
changed the title
achilleas-k
approved these changes
Jan 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To replace the manifest-db in osbuild CI with the images repo image build cache, we will need to be able to easily construct the S3 cache prefix corresponding to a specific commit checkout of this repository.
The current S3 prefix for the image build cache looks like this:
e.g.
The most problematic part of constructing the correct prefix without running scripts on the CI runner is the osbuild NEVRA. The specific NEVRA is an RPM build of a particular osbuild commit for a specific distribution (the CI runner distro). The osbuild commit and the default CI runner distro can be determined from a local copy of the
Schutzfile. However, getting the NEVRA requires querying a DNF repository for the CI runner distro.After brainstorming with @achilleas-k, we agreed to rework the image build cache prefix by:
The newly proposed S3 prefix for the image build cache looks like this:
e.g.
/jira-epic COMPOSER-2318
JIRA: HMS-5274