Skip to content

[question] Handling failed builds in products pipeline flow #18962

@akielbas

Description

@akielbas

What is your question?

Our team has been working on implementing the products pipeline flow from the CI Tutorial in the docs.

We do have similar Artifactory repos that correspond to packages, products and develop.

We have the build working pretty well with a re-usable GitHub Actions workflow. We can feed an arbitrary conanfile and a list of target profiles / build systems and have it build everything that is missing in a dynamic way, uploading to products along the way. What's still work in progress is the promotion workflow.

In the documentated example the entire flow can be summarized as:

  1. Create lockfile
  2. Generate build order for every desired configuration
  3. Merge build orders (with --reduce)
  4. Iterate through packages that need to be built, for each:
    a) Build missing package
    b) Convert build output to list
    c) Upload list to products repo (in practice will only be the one package that was just built, everything else will generally already be in products repo because it must have been built earlier per build order).
  5. Combine uploaded lists, preserve this somewhere
  6. Test the built packages in products repo
  7. Later promote uploaded list via conan download / upload or conan art:promote using persisted list from step 5.

This works pretty well, but I'm curious what you recommend for the following scenario:

  1. Let's assume we have a simple/flat build order of PkgA/1.0, PkgB/1.0 and PkgC/1.0.
  2. Run steps 1-4 above and say that PkgC/1.0 fails to build but PkgA/1.0 and PkgB/1.0 have built successfully and have been uploaded to products repo.
  3. We don't continue to step 5 because of the failure and go resolve the issue with PkgC/1.0 build (could be bad version or a build system issue, whatever).
  4. We try to run steps 1-4 again and this time the reduced build order is only PkgC/1.0 (because PkgA/1.0 and PkgB/1.0 already exist in products repo from earlier attempt), this completes successfully.
  5. When we get to step 7 above we only have the list with PkgC/1.0 so PkgA/1.0 and PkgB/1.0 are not promoted.

I see several ways to solve this but was wondering what others think:

  1. We could clear the products repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately.
  2. We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.
  3. We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using conan download / upload promotion (maybe there is a way to filter out packages that are already in develop?) but perhaps it is not a big deal with conan art:promote.

We're kind of leaning toward option 3 here, is that an OK approach? Are we totally missing something?

Thank you in advance!

Have you read the CONTRIBUTING guide?

  • I've read the CONTRIBUTING guide

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions