vendor branching: a strategy to import external information in a project

[axd1967]

A recurring issue that developers face is how to use external textual information (data, source code, ...). The problem is that long after after importing external information, the external information changes: think of bug fixes and updates to data.

Examples:

• Almagest data (minor fixes, of course - this is essentially frozen data)
• geonames data (https://www.geonames.org/recent-changes.html), https://github.com/Stellarium/stellarium-data/tags
• quasars (https://github.com/Stellarium/stellarium/blob/master/plugins/Quasars/util/quasars.tsv)
• libraries (https://github.com/Stellarium/stellarium/tree/master/src/external)
• HTC imports (src/core/planetsephems/htc20b.cpp, Added support HTC 2.0 theory #1942)

The solution that is usually chosen is the following:

1. copy the external data/code
2. continue with that
3. when problems appear, fix locally

The problem with this approach surfaces when the external information is modified too. This is a subtle problem, because until then, everything works fine. Usually no thought is given to long term maintenance ("this works fine, we don't need updates!", but experience learns that errors appear everywhere, that's bread and butter for the software developer.

So when the external information changes, two options present themselve:

1. ignore the external changes and proceed
2. try to copy the changes locally

The first option will lead to a slow and guaranteed code rot.

The second option can be very difficult to maintain in the long term, and is very prone to bugs. It might sometimes be very difficult to understand what changed in the external information, and manually and flawlessly apply those changes locally.
Also, sometimes there is no record of where the information came from, so there is no possibility to benefit from changes.

In fact, this is an example of the saying "copy-paste is evil".

What is the solution?

The solution is a strategy called "vendor branching".

The idea is quite simple:

1. keep external information alive on a separate branch, called the "vendor branch" (eg vendor/geonames). Every dataset/tool lives on its own vendor branch.
2. store the external information in a separate folder (eg external/<vendor>/<toolname>)
3. ensure that at least a README explains where the external information can be found. A better suggestion is eg VENDOR, as it is possible that the vendor already includes a README.
4. apply a vendor tag that describes which version has been imported (eg vendor/geonames/2021-08-21)
5. merge that branch to wherever it is needed (normally the feature branch, which will eventually be merged to master)
6. if needed, modify the information locally (feature/master branch, but never on the vendor branch)

So the vendor branch becomes a kind of pristine mirror of the external information (except for a added VENDOR text information file).
For prolific vendors, it might be necessary to create subfolders and separate branches per product.

Now it becomes easy to update external information. Whenever the external information changes:

1. checkout the vendor branch
2. update the vendor branch (typically a complete delete of the folder followed by an unzip, then a commit)
3. tag the vendor branch
4. merge the updated vendor branch to master (or via an intermediate feature/bugfix branch, often in order to update local stuff)

The last step might trigger conflicts if the vendor information was modified locally (but never in the vendor branch!). This is usually done in order to provide for local needs or solve small bugs. Note that conflicts are also likely to arise when a locally solved bug is also solved by the vendor, so bug management of vendor information is a critical process that should be coordinated with the vendor whenever possible.

The exception to the rule is the VENDOR text file that might need occasional updates (eg when the URL of the vendor iformation changes).

Sometimes, the external information is managed in a Git repository. In such case, sometimes it might be possible to manage the external information via git submodules or git subtrees. These approaches are not discussed here because they can lead to other problems.

Some information here:

• https://www.roe.ch/Git_Reference
• http://pipe-devnull.com/2012/11/11/vendor-branches-in-git.html
• https://edoras.sdsu.edu/doc/svn-book-html-chunk/svn.advanced.vendorbr.html
• https://www.oreilly.com/library/view/version-control-with/9780596510336/ch04s09s01.html
• https://community.perforce.com/s/article/3374
• https://stackoverflow.com/questions/769786/vendor-branches-in-git
• https://github.com/Stellarium/stellarium/search?q=vendor

[gzotti]

Thanks for the extended explanation. We may have 3-4 such dependencies which may however be updated only 0.2-1x/year: INDI, qxlsx, qtcustomplot (but no more upgrade possible under terms of GPL2). The one library which practically saw no update since the 1990s, TESS, shows most of the tiny nasty glitches nobody was so far willing/able to fix. (Or something is wrong with some calls from our side.)

[axd1967]

Another simple example where this mechanism can be used is when importing user scripts.

Let's take as an example this script: http://www.alienbasecamp.com/Stellarium/sun.htm (after #1088)

I manage a separate git repository where my own Stellarium scripts live. As an example, here I will add the vendor alienbasecamp product sunsetjumps.

1. create+checkout vendor/alienbase/sunsetjumps
2. unzip the initial zip file
3. rename the folder to sunsetjumps (we will store version information in Git metadata)
4. commit the files with message "import sr/ss scripts v5.1" (that is the version of 2021-09-13)
5. create vendor tag vendor/alienbasecamp/sunsetjumps/5.1 (I also append the release date here: 2020-04-02, and this illustrates yet another issue when carelessly using dates: what date is 04/02/2020? )
6. check out master
7. merge the vendor branch to master
8. add VENDOR and write the URL of the vendor in that file
9. commit VENDOR

Whenever one wants to import an update (this is called a "vendor drop")

1. checkout the vendor branch
2. go to the vendor directory
3. empty it
4. unzip the vendor into it
5. if needed, remove version info names from files if you want to detect changes too. Anyway, it is not a good practice to add version info to file names.
6. check in
7. checkout master (or some intermediate branch)
8. merge the vendor branch

This all feels complicated at first: I could simply have unzipped the file and voila, it will run "out of the box".

The vendor approach is a matter of healthy code management, training and habit, and it will pay off the day one wants to modify something without having to worry how to deal with external updates. (By the way, this is fundamentally the same mechanism where you can create a custom Stellarium version - the Stellarium code is is fact a vendor for you).

(Notice that the vendor includes a file that should be renamed from faq - sunset jumps05.txt to faq - sunset jumps.txt - it doesn't make much sense to add version info in file names. I rename such files so that delta's become apparent, assuming that such a file delta does not hide old issues.)

A vendor source code can also be managed in a separate repository. The pattern is the same: store the vendor code in a separate branch, then merge to the master branch. If ever necessary, edit code there. And don;t forget to regularly visit the vendor branch for updates+merging.

Observe that user scripts published under https://stellarium.org/scripts.html could easily be hosted at https://github.com/Stellarium/stellarium-data or even https://github.com/Stellarium/stellarium.github.io. Then, anyonme wishing to use those scripts could clone that repo and enjoy them. But that's yet another discussion.

[axd1967]

(And this probably belongs in a new discussion category "Development")

[axd1967]

I'll prepare an update so that this recommendation can be moved to the Stellarium sourcecode (eg a section in CONTRIBUTING.md or maybe a separate vendor-branches.txt) so that every developer/contributor has access to it.

In general, it is dangerous to keep this kind of information separated from source code, as this tends to become stale and will seldom be read by contributors.

[DarthGandalf]

Thanks for tagging me.

There are multiple ways to manage dependencies, and I agree that copypasting (what's done now) has problems. Vendor branching is not a perfect solution either: it requires strong discipline to never touch the source located there in master branch, even accidentally, or when accepting/reviewing someone's PR, but only update the vendor branch from time to time, and it doesn't help downstream distro maintainers. Note that various distros have policies to not use the bundled libraries: Fedora, Gentoo, Debian.

Other common ways to handle C++ deps are Conan or Vcpkg, but they require some more overhead from the end user who wants to compile the software... and not all libs are available in vcpkg anyway.

In #1295 I had implemented the split between Stellarium code and its deps via CPM, which transparently works inside CMake, which handles these concerns:

• for Stellarium devs, make it easy to update the version of the dependency used
• for end user who wants to compile, they won't notice a difference, and will use cmake/make/ninja/gcc/msvc as usual
• for distro maintainers, CPM provides a way to only use the system library without downloading anything, so they don't have to patch the build system anymore

The main concern there was that CPM requires CMake 3.14+. At this point it's probably only a problem on ancient systems such as Ubuntu Bionic. But in those cases, user can download a newer cmake manually, and use it instead of cmake from the apt repo. WDYT about bumping the required cmake version from the current 2.8.12? I can rebase that PR for the current master.

[axd1967]

WDYT ...

would it be OK to continue that discussion in #1295 rather than here? the concepts explained here are very simple, while the tools (CPM) you are bringing up are quite technical. the concept explained here is generic, so it is applicable to source code as well as data; I'm afraid that bringing that discussion here will totally drown the idea that is dearly needed by contributors that might not master tools such as cmake etc .

[DarthGandalf]

Re technical details of it, sure! Why I brought it here, is because the other points still stand: vendor branching doesn't help distros at all. And if we do it that way, we won't need vendor branches either.

[axd1967]

Let's please keep the vendor branch concept clean for the specific cases that currently exist, and where CPM is far too heavy:

• importing data from the outside (eg Almagest data, geographical names, ...)
• importing source code

Vendor branching IS a valid KISS (Keep It Simple and Stupid) tool for such cases, your statements are killing this simple concept. We are not talking about distros, can we please not enter that discussion here?

But you are absolutely, entirely right; src/external needs a stronger approach! And that is being tackled in #1295.

[axd1967]

GitHub doesn't seem to link from PR to discussion, so here is the related PR: #1906

[axd1967]

The vendor branch approach also guarantees that

• Stellarium can be built off-line (i.e. without the need to depend on external sources or have an Internet connection)
• if necessary, vendor changes that cause havoc (or go missing) can be dealt with locally without ending up with a frozen dependency (as an attempt to revive a library gone AWOL); even more, external code evolutions can continue to be merged in while at the same time, problematic code can be blocked/dealt with.

Because we all know that vendor code can disappear overnight.

• Python: the KIK case
  -- https://www.theregister.com/2016/03/23/npm_left_pad_chaos/
  -- https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code
  -- https://arstechnica.com/information-technology/2016/03/rage-quit-coder-unpublished-17-lines-of-javascript-and-broke-the-internet/
  -- https://www.activestate.com/blog/pythons-atomicwrites-and-software-dependency-availability-issues/
• https://stackoverflow.com/questions/1710027/arguments-for-and-against-including-3rd-party-libraries-in-version-control/1710050
• https://stackoverflow.com/questions/tagged/vendor-branch

[DarthGandalf]

Stellarium can be built off-line (i.e. without the need to depend on external sources or have an Internet connection)

You don't need to be online, if the dependencies are installed already (that's how distros build packages: from their build environment usually there's no internet access), or if you prepopulated CPM_SOURCE_CACHE directory.

external code evolutions can continue to be merged in while at the same time, problematic code can be blocked/dealt with

This is called a fork, and has all the usual upsides and downsides of a fork. But if the vendor does cause havoc, for the cases where the fork is in fact the right solution for this (far from always), let's be explicit about it. That said, there are different ways to do forks, and vendor branches as presented is one of them. However, I'm not going to compare different ways to do forks right now.

Because we all know that vendor code can disappear overnight.

Let's talk when it does disappear. And to remind you, even if it does disappear, it can be recovered from multiple places: starting from developer machines, to mirrors controlled by linux distros, such as Debian or Gentoo, which themselves don't trust upstream to not disappear, that's why they store the tarball with the library source on the mirror (there are more reasons than just this though).

[axd1967]

(related to vendor_branching)
to be labelled with label "dependencies"

[d33psky]

+1 for vendor branches.
Did we consider Brett's git-vendor addition ? https://github.com/brettlangdon/git-vendor
And a nice blog post on it https://medium.com/opsops/git-vendor-295db4bcec3a (free account access)
-- Hans

[axd1967]

interesting command, although the concept is relatively easy to master without extra commands.
(software engineering discipline is what is needed.)

[axd1967]

Also, Stellarium is not complex enough to require sofisticated dependency managers.

The vendor branch approach is simple to apply, considering the level of software engineering skills of the average Stellarium developer/contributor.

[axd1967]

CISA recommendation: axd1967@5f33484

[axd1967]

It might not be the best practice to let software depend on uncertain external sources. One reason is that remote archives come and go. (Re-read CISA #1856 (comment).)

For example... take a look at dfcce4d. Notice that the committer (even if it is a reliable and strong developer) is also the author of that zip file.

Just imagine https://github.com/10110111/CalcMySky/archive/refs/heads/master.zip (see https://github.com/10110111/CalcMySky) contains a bug or a virus. For a short while (until the tag reference was used), there was a potential danger to inject nasty stuff in any system compiling and then running version dfcce4d of Stellarium.

Likewise...

• 7e2d7a9
• 186b06f (more specifically 186b06f#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20aR756)
• 01976bd
• b22283f
• b3d6cee

Now, some packages can be trusted; others might be too large to take in.

On that note, Stellarium is reaching 1G in source code (of which ~50% are language files! but might require 10G when managing branches); maybe the language component could be split off into a separate entity....

[alex-w]

Alex, please stop propose and propose this ugly idea, because if you will really follow your strategy, then you'll never compile stellarium

[10110111]

Just imagine https://github.com/10110111/CalcMySky/archive/refs/heads/master.zip (see https://github.com/10110111/CalcMySky) contains a bug or a virus.

I can generally agree that referring to a moving target such as master is not a good idea for a build system. There's actually been an incident where a developer of a dependency lots of services depended on broke it intentionally.

But this is not a reason to maintain our own versions of external packages. We just should refer to fixed targets.

Likewise...
...

Some of the commits you link to (including the "more specifically ..." part) contain a very precise reference with a hash to check, others don't even refer to anything, so they don't illustrate the point that I agree with.

Now, some packages can be trusted; others might be too large to take in.

Now this is a case of double standards. If the size of a package defines your level of trust in it (since you're apparently OK with not taking it in), and especially if a bigger package is taken as more trustworth, something is wrong with your risk assessment :P

[10110111]

Besides, the use of a local version of a package is completely OK: it's the user who installed it who's responsible for its adequacy, not the build system (the latter should just check that the necessary features are present). Otherwise we must take Qt, libc, and even the kernels of every supported platform into this version branching.

[axd1967]

The art of hyperbole... Just be careful with that "you" word, as it terminates all discussions... @xalioth

Pity we cannot version microcode of all known processors, then we would be totally independent.