Simplify caching mechanisms for CI and PROD images (canary) #45266

potiuk · 2024-12-29T11:51:53Z

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks:

CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip
The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug)
Caching of pip and uv was not used because it increased size of the image significantly

This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before:

The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs.
The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies.
the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of uv is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast.
we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc.
documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at.

Fixes: #42999
Fixes: #43268

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

.github/actions/checkout_target_commit/action.yml

.github/actions/prepare_all_ci_images/action.yml

.github/workflows/additional-ci-image-checks.yml

.github/workflows/build-images.yml

.github/workflows/ci-image-build.yml

.github/workflows/ci.yml

.github/workflows/k8s-tests.yml

dev/breeze/doc/06_managing_docker_images.rst

dev/breeze/doc/ci/06_diagrams.md

scripts/ci/images/ci_start_arm_instance_and_connect_to_docker.sh

scripts/ci/images/ci_stop_arm_instance.sh

scripts/docker/common.sh

scripts/docker/entrypoint_ci.sh

scripts/docker/install_airflow.sh

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after apache#45266

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after #45266

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in apache#45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in #45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

@assignUser

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks: * CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip * The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug) * Caching of `pip` and `uv` was not used because it increased size of the image significantly This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before: * The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs. * The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies. * the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of `uv` is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast. * we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc. this helps with faster rebuilds of the images for local development environment * documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at. Fixes: apache#42999 Fixes: apache#43268

…#45281) In very simple changes we should skip building ci image altogether. This had been missing condition in one of the earlier refactors. Follow-up after apache#45266

Packaging tools (uv and pip) do not have to use mounted cache as they are always reinstalled when their versions got updated in the Dockerfile, and changing sources of airflow does not invalidate the installation step (COPY . is made after the installation). This will add two more layers being cached in github registry when the image is built. Follow-up after apache#45266

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after apache#45266

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in apache#45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

@assignUser

For a long time we had used a sophisticated mechanism to speed up our CI jobs by building the images in "pull_request_target" workflow and pushing them to GitHub registry. That however had several drawbacks: * CI image was complex when it comes to layer setup (we had to pre- cache installed dependencies by installing them from branch tip * The pull_request_target is a very dangerous workflow, we had a number of security problems with it (and it's difficult to debug) * Caching of `pip` and `uv` was not used because it increased size of the image significantly This PR significantly improves the caching mechanisms for the images building of several advacements that were not possible before: * The upload-artifacts@v4 action and improved stash action developed by @assignUser and published in "apache/infrastructure-actions" allows us to store all images (8GB per run) in artifacts rather than in registry - so we can do the image build once and share it with all the jobs. * The uv speed is "enough" to allow occasional installation of Airlfow locally. This allows to utilize cache-mount and locally build uv cache, rather than rely on "remote" cache when we are building local images for breeze. The first time you build local breeze image it will take 2-5 more minutes (depending on your network speed, but because we can utilise cache mounts, every subsequent build should be very fast - even if all dependencies change. Using uv also allows to "always" reinstall airflow when you build the image even if single source file changed, because with cache it takes sub-seconds to reinstall airflow and all dependencies. * the cache mounts are not included in the image size, and since we can export and import images in CI in artifacts and we do not need to rebuild them, the images shared as compressed artifacts are relatively small (2GB) - cache of `uv` is around 4GB on top of that so sharing image built in the "build image" job with other jobs in the same workflow is fast. * we are still using registry cache for the "non-python" parts of the image - both CI and breeze image build speed benefit from using the image cache for system dependencies, database clients etc. this helps with faster rebuilds of the images for local development environment * documentation has been updated to reflect the new CI setup. The diagrams showing the workflows of ours are no longer needed as the workflows are quite straightforward when they are looked at. Fixes: apache#42999 Fixes: apache#43268

…#45281) In very simple changes we should skip building ci image altogether. This had been missing condition in one of the earlier refactors. Follow-up after apache#45266

Packaging tools (uv and pip) do not have to use mounted cache as they are always reinstalled when their versions got updated in the Dockerfile, and changing sources of airflow does not invalidate the installation step (COPY . is made after the installation). This will add two more layers being cached in github registry when the image is built. Follow-up after apache#45266

We are using various caches in our build and so far - due to the way how "standard" caching works, PRs from forks could not effectively use the cache from main Airflow repository - because caches are not shared with other repositories - so the PRs builds could only use cache effectively when they were rebased and continued running from the same fork. This PR improves caching strategy using "stash" action from the ASF. Unlike `cache` - the action uses artifacts to store cache, and that makes it possible for the stash action to use such cache uploaded from `main` canary builds in PRs coming from the fork. As part of this change all the places where setup-python was used and breeze installed afterwards were reviewed and updated to use only breeze installation action (it already installs python) and this action has been improved to use UV caching effectively. Overall this PR should decrease setup overhead for many jobs across the CI workflow. Follow-up after apache#45266

Prod image build sometimes (quite often) is not needed. When removing `pull_request_target` in apache#45266 `wait-for=prod-images` had the condition that prevented it from running (and the `build-prod-images` step depended on it) - but this condition is gone now. Instead of preventing the whole composite workflow from running, we are adding it to "build-prod-packages" so that the whole workflow can complete as prerequisite to "finalize-tests" which should be executed regardless from `prod-image-build` being executed.

potiuk added full tests needed We need to run full set of tests for this PR to merge canary When set on PR running from apache repo - behave as canary run all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Dec 29, 2024

boring-cyborg bot added area:dev-tools area:production-image Production image improvements and fixes labels Dec 29, 2024

potiuk mentioned this pull request Dec 29, 2024

Simplify caching mechanisms for CI and PROD images (regular PR) #45261

Closed

potiuk force-pushed the redesign-image-caching branch from 4f266c6 to 2682082 Compare December 29, 2024 11:59

potiuk changed the title ~~Simplify caching mechanisms for CI and PROD images~~ Simplify caching mechanisms for CI and PROD images (canary) Dec 29, 2024

potiuk force-pushed the redesign-image-caching branch 4 times, most recently from 1e4fe9b to 374cc5d Compare December 29, 2024 14:29