Help us flush out the permissible source requirements for the Unified Build project.

[MichaelSimons]

First off, if you are unfamiliar with the Unified Build project please read the Overview.

A result of this work is that Microsoft will be building its version of .NET using the source-build infrastructure. The Microsoft builds will be including windows only components like WPF, winforms etc. which may have some non-OSS licenses. The Microsoft builds will also require some checked in binary files.

Up until this point, the source-build source (VMR and source tarballs) had always been prepped to remove binaries as well as subsections of the product repos that didn't have OSS licenses. A goal is for both the Microsoft and source-build products to build from the same source. How should we think about checked in binary files and non-oss source in this Unified Build context? Would it be acceptable for example to have checked in script that would strip out these non-permissible components for distro-maintainer scenarios that have strict requirements (e.g. no binaries, no non-OSS licences)? The idea is that distro-maintainers would run this script as one of the first steps in their build process.

TIA for your input

cc @dotnet/distro-maintainers, @mmitche, @premun, @ViktorHofer, @jkotas

[asbjornu]

I applaud the goal of having a unified build from the same source for all distributions of .NET – including Microsoft's own. 👏🏼 This is a large step in the right direction and something that is going to improve the source build significantly going forward, as Microsoft will be dogfooding their own developer experience.

I assume that the binaries in question are only required for Microsoft themselves to build the mentioned Microsoft technologies: WPF, Winforms, etc. If so, how about having these in a Git submodule that can just be ignored by every other build system but Microsoft's? The arguments for excluding binaries by default are many; from the size and duration of the Git clone, to the number of distributions not needing it vs. the single one that does (Microsoft's own).

With a Git submodule, the only difference in Microsoft's build would be to clone the submodule. By its existence on disk, the build scripts could identify that "yes, we should build Microsoft components" and pull those binaries in as needed. Otherwise, the binaries could just be ignored. With this setup, the Git submodule could even be behind authentication that would be inaccessible to non-Microsoft distributions.

[mmitche]

The submodule approach is tough because:

1. The files are scattered throughout the repos that make up .NET.
2. Many of the components still build individually, and will need those files there, so you can't easily just factor it all out into one spot.

Also note that all of the files are public. There's no reason to put them behind auth.

[omajid]

Hey! Thanks for bringing this up.

This is a fantastic goal that Red Hat has been hoping for too.

I would love to have everyone (Microsoft, partners, distro owners, packagers and anyone else) build .NET out of the same source tree/archive, as much as possible. I would love to have Microsoft release a single thing (eg release tarball) that they have signed and everyone can consume directly, ensuring that everyone is using the same thing and minimizing chances for a compromised developer to ship a trojan'ed version of .NET.

On the other hand, Fedora's policies state:

Some pre-packaged program binaries or program libraries may be under terms which do not permit redistribution, or be affected by legal scenarios such as patents. In such situations, simply deleting these files [as the first in the build process] is not sufficient, the maintainer will need to make a modified source that does not contain these files

They also say:

When you encounter prebuilt binaries in a package you MUST: Remove all pre-built program binaries and program libraries [as the first step in the build process] prior to the building of the package.

In other words, Fedora (and so RHEL and CentOS Stream) cares more about the licenses of files. If appropriately open-source licensed binaries are included in a source archive, that source archive is acceptable, with the distro maintainer responsible for deleting binaries to make sure everything is built from source. But if non-open source code/binaries are included in a release, I will have no choice but to pre-process them to produce a modified source-archive and lose many benefits that I mentioned above.

[MichaelSimons]

To recap what I read is that non-OSS licensed code in the 9.0 VMR would be a take back/adoption blocker. Having checked in binaries with a script to strip them is permissible.

[omajid]

"non-OSS licensed code" -> "non-OSS licensed code or binaries" (or "non-OSS licensed code or non-OSS licensed binaries")

[tmds]

I would love to have Microsoft release a single thing (eg release tarball) that they have signed and everyone can consume directly

I'm trying to understand what this means. The GitHub releases (like .NET 8.0.1) include PGP signatures for the sources. How does this relate to what you are asking for?

[omajid]

The current situation - where there's a release on github with gpg signatures that we can consume directly is pretty much perfect.

However, as this proposal states, Microsoft's builds require additional content that may not be under an OSS license. If there's content under an non-OSS license, we can't use it. If that non-OSS content is part of the release tarball, we can no longer directly consume that tarball, meaning we lose the advantage of official release tarballs signed by the .NET project.

[tmds]

Ah, thank you for clarifying.

On this topic: for bootstrapping, we use a prebuild SDK tarball, and a prebuild NuGet packages tarball. It would be good if they get the same considerations as the source tarball.

[MichaelSimons]

Friendly ping for source build customers/@dotnet/distro-maintainers to please share your input.

[richlander]

Some folks may want more on the licenses we use.

https://github.com/dotnet/core/blob/main/license-information.md

[mateusrodrigues]

On the Canonical side, we currently do some pre-processing on our end to package every .NET release. From a .NET 8 standpoint, among other things, the pre-process git clones the source code, gathers repo info for later use by the --source-repository and --source-version parameters of build.sh, and removes .git to keep the source package as lightweight as possible.

Adding the execution of a script shipped in the VMR to strip binaries and non-OSS licenses should be easy enough.

However, ideally, as @omajid said, having a signed tarball that everyone can build from not only removes any needs for pre-processing -- which, in turn, removes human error risks, as this is currently a somewhat manual process -- but also minimizes the risk of shipping a dirty build. So, if we end up not having that at first, we'd love to see an effort towards this goal.

[omajid]

I am going on a tangent. Sorry about that in advance.

the pre-process git clones the source code, gathers repo info for later use by the --source-repository and --source-version parameters of build.sh, and removes .git to keep the source package as lightweight as possible

AFAIK, this is all done for you if you use the release tarball and the associated release.json ("Release Manifest") from a release tag like https://github.com/dotnet/dotnet/releases/tag/v8.0.1

[mateusrodrigues]

Yes. However, for security releases, we need to start spinning these builds in our infra several days before the public release, using the bits from the partners AzDO repo. That is, before the release tarball and the release.json are made available in the GitHub releases page.

[omajid]

Oh. Good point. I was told that the partner site should have the same thing: #3717. They will not be bit-by-bit identical to what gets release publicly via github, but the contents of the tarball should be identical, and the signature file will verify the partner tarball. The release.json file should be available there too and if it's not, that should be something to get fixed as well.

[mateusrodrigues]

Yes, you're right. The latest hand-off for February now includes a tarball and a release manifest 🎉

[MichaelSimons]

Cross posting from a separate conversation with @omajid.

Sharing the Debian policy on source-only tarball: https://wiki.debian.org/UpstreamGuide#Source_only_tarball

"If you distribute precompiled binaries with your source code, then the
Debian Maintainer has extra work to strip out these binaries and
repackage your tarball."

[TheJayMann]

It is also worth noting, based on information I've gathered by reading a few posts on the Debian mailing list, that this isn't restricted to just precompiled binary files, but rather any files containing data that is produced by a tool that uses source as an input, and is not meant to be modified outside of modifying the original source and using the tool to generate the new data. In addition to precompiled binaries, this would also include, for example, minifying a Javascript or HTML file, transpiling a SASS file to CSS, saving benchmark data as an HTML file, and generating end user documentation files from tools such as halibut or pandoc.

One reference to this is the discussion of proper packaging of epoch within knot-resolver source package regarding the knot-resover-module-http binary package, as epoch.js and epoch.css are generated from epoch.coffee and epoch.scss and packaged upstream without the original source.

https://lists.debian.org/debian-devel/2016/10/msg00178.html

[jkotas]

removes any needs for pre-processing

I expect we will need to do pre-processing on the content of the full VMR git repo to produce source drop suitable for Linux distros. I think it would be best to completely delete large Windows-specific components like WPF from the source drop for Linux distros, so that they are not a concern.

The question is whether it is better for this pre-processing to execute in Microsoft eng infrastructure and the result published in some form, or whether it is easier for this pre-processing to be done as part of inserting a new .NET build into the Linux distro.

[MichaelSimons]

The question is whether it is better for this pre-processing to execute in Microsoft eng infrastructure and the result published in some form, or whether it is easier for this pre-processing to be done as part of inserting a new .NET build into the Linux distro.

This was already answered - non-OSS licensed code and binaries cannot exist in the source Microsoft hands-off. OSS licensed code and binaries that is not required for a linux source build can be included and stripped as part of the distro maintainer process.

[premun]

We made the decision to leave the windows repositories in the source build branch. Would we cloak the licensed code only or all of the repository contents?

[MichaelSimons]

Thanks for all the input on this topic. #4088 tracks the implementation of what was discussed here.