-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
What is your question?
Our team has been working on implementing the products pipeline flow from the CI Tutorial in the docs.
We do have similar Artifactory repos that correspond to packages
, products
and develop
.
We have the build working pretty well with a re-usable GitHub Actions workflow. We can feed an arbitrary conanfile and a list of target profiles / build systems and have it build everything that is missing in a dynamic way, uploading to products
along the way. What's still work in progress is the promotion workflow.
In the documentated example the entire flow can be summarized as:
- Create lockfile
- Generate build order for every desired configuration
- Merge build orders (with
--reduce
) - Iterate through packages that need to be built, for each:
a) Build missing package
b) Convert build output to list
c) Upload list toproducts
repo (in practice will only be the one package that was just built, everything else will generally already be inproducts
repo because it must have been built earlier per build order). - Combine uploaded lists, preserve this somewhere
- Test the built packages in
products
repo - Later promote uploaded list via
conan download / upload
orconan art:promote
using persisted list fromstep 5
.
This works pretty well, but I'm curious what you recommend for the following scenario:
- Let's assume we have a simple/flat build order of
PkgA/1.0
,PkgB/1.0
andPkgC/1.0
. - Run
steps 1-4
above and say thatPkgC/1.0
fails to build butPkgA/1.0
andPkgB/1.0
have built successfully and have been uploaded toproducts
repo. - We don't continue to
step 5
because of the failure and go resolve the issue withPkgC/1.0
build (could be bad version or a build system issue, whatever). - We try to run
steps 1-4
again and this time the reduced build order is onlyPkgC/1.0
(becausePkgA/1.0
andPkgB/1.0
already exist inproducts
repo from earlier attempt), this completes successfully. - When we get to
step 7
above we only have the list withPkgC/1.0
soPkgA/1.0
andPkgB/1.0
are not promoted.
I see several ways to solve this but was wondering what others think:
- We could clear the
products
repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately. - We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.
- We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using
conan download / upload
promotion (maybe there is a way to filter out packages that are already indevelop
?) but perhaps it is not a big deal withconan art:promote
.
We're kind of leaning toward option 3 here, is that an OK approach? Are we totally missing something?
Thank you in advance!
Have you read the CONTRIBUTING guide?
- I've read the CONTRIBUTING guide