[question] Handling failed builds in products pipeline flow

[akielbas]

What is your question?

Our team has been working on implementing the products pipeline flow from the CI Tutorial in the docs.

We do have similar Artifactory repos that correspond to packages, products and develop.

We have the build working pretty well with a re-usable GitHub Actions workflow. We can feed an arbitrary conanfile and a list of target profiles / build systems and have it build everything that is missing in a dynamic way, uploading to products along the way. What's still work in progress is the promotion workflow.

In the documentated example the entire flow can be summarized as:

1. Create lockfile
2. Generate build order for every desired configuration
3. Merge build orders (with --reduce)
4. Iterate through packages that need to be built, for each:
   a) Build missing package
   b) Convert build output to list
   c) Upload list to products repo (in practice will only be the one package that was just built, everything else will generally already be in products repo because it must have been built earlier per build order).
5. Combine uploaded lists, preserve this somewhere
6. Test the built packages in products repo
7. Later promote uploaded list via conan download / upload or conan art:promote using persisted list from step 5.

This works pretty well, but I'm curious what you recommend for the following scenario:

1. Let's assume we have a simple/flat build order of PkgA/1.0, PkgB/1.0 and PkgC/1.0.
2. Run steps 1-4 above and say that PkgC/1.0 fails to build but PkgA/1.0 and PkgB/1.0 have built successfully and have been uploaded to products repo.
3. We don't continue to step 5 because of the failure and go resolve the issue with PkgC/1.0 build (could be bad version or a build system issue, whatever).
4. We try to run steps 1-4 again and this time the reduced build order is only PkgC/1.0 (because PkgA/1.0 and PkgB/1.0 already exist in products repo from earlier attempt), this completes successfully.
5. When we get to step 7 above we only have the list with PkgC/1.0 so PkgA/1.0 and PkgB/1.0 are not promoted.

I see several ways to solve this but was wondering what others think:

1. We could clear the products repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately.
2. We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.
3. We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using conan download / upload promotion (maybe there is a way to filter out packages that are already in develop?) but perhaps it is not a big deal with conan art:promote.

We're kind of leaning toward option 3 here, is that an OK approach? Are we totally missing something?

Thank you in advance!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[akielbas]

I've implemented option 3 above and it is working quite well, still interested in hearing what's recommended.

We no longer collect and aggregate the built package graphs / pkglists as shown in the tutorial.

Instead after we complete any succesful build process attempt we do the following:

1. compute a new graph info for each configuration
2. turn each build graph info into a package list
3. merge all package lists into one
4. use art:promote to copy from products remote to develop (need to additionally specify --remote for products because usually some packages are already in develop).

[memsharded]

Hi @akielbas

Thanks for your question

This works pretty well, but I'm curious what you recommend for the following scenario:

So this would be the scenario of an interrupted/broken build

From your comment in

We try to run steps 1-4 again and this time the reduced build order is only PkgC/1.0 (because PkgA/1.0 and PkgB/1.0 already exist in products repo from earlier attempt), this completes successfully.

I understand that the breakage doesn't really need a code change in PkgC, but it was a glitch or something, not an API break that requires a code change.

Indeed, PkgA and PkgB don't need to be built, they were already built and uploaded. This is one of the advantages of the approach, it allows to re-use binaries that result from a partial build in further later builds, avoiding unnecessary costs even if the build as a whole failed.

When we get to step 7 above we only have the list with PkgC/1.0 so PkgA/1.0 and PkgB/1.0 are not promoted.

I see, yes, if this is managed as a clean build from scratch, it might fail to promote PkgA and PkgB.

We could clear the products repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately.

I agree, this looks inefficient at best, and potentially problematic

We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.

This doesn't sound that bad, as you had actually preserved the package list in Step 5 above, then, if this is an incremental, combined view of the packages that have been created, it wouldn't be necessary to find all workflows, just the parent/latest one, and pick it from there?

We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using conan download / upload promotion (maybe there is a way to filter out packages that are already in develop?) but perhaps it is not a big deal with conan art:promote.

Yes, and even for the download+upload promotion, the truth is that the checks that the Conan client does to avoid uploading/downloading things that are already in the server or in the cache should save most of the potentially unnecessary costs, so this looks like a pragmatic approach. And it shouldn't be an issue with direct API promotion like the conan art:promote one.

[memsharded]

I've implemented option 3 above and it is working quite well, still interested in hearing what's recommended.

Interesting, thanks for the feedback!

In any case, yes, the approach sounds pragmatic and more robust against re-launched partial builds.
The case explained in the tutorial is focused on the "happy path", trying to be as efficient as possible.

compute a new graph info for each configuration
turn each build graph info into a package list

The information should already be there in the conan graph build-order result, but it is true that this doesn't convert that easily to a package list.

Maybe this is something that we want to revisit, maybe a feature request to create package-lists out of build-orders, that way the extra conan grpah info could be saved.