upload-nexus: flat apt/yum repositories cannot hold packages for more than one distribution release

[shiptux]

Problem

The shared upload-nexus.yml publishes every deb into a single flagos-apt-hosted (one anonymous suite, Codename: flagos-apt-hosted, Architectures: amd64) and every rpm into the root of flagos-yum-hosted. That works while the only content is one package built on one base image, but FEP-0019 Wave 1 (flagos-ai/community#59) is about to publish packages for several distributions into it, and two of them are already incompatible with each other:

package	built on	binary-compatible with
libtriton-jit 0.1.0-1 (on Nexus today)	ubuntu 24.04 / CUDA 12.6 / py3.12	24.04 only (glibc 2.38, libpython3.12)
libtriton_jit#61 output	ubuntu 22.04 / CUDA 12.8 / py3.12 (FEP DEB matrix)	22.04 and 24.04
FlagTree deb (#607/#794)	ubuntu 24.04	24.04 only — 22.04 fails on GLIBC_2.38 (FlagTree#798)
FlagCX / FlagSparse / FlagAttention rpm	fedora 43 and openEuler 24.03 (.fc43 / .oe2403)	one repodata would list both; dnf on openEuler sees fc43 packages with unsatisfiable deps
python3-flag-* (Architecture: all)	any	any deb-based distro

Native packages carry five ABI dimensions at once (glibc, libstdc++ CXX11 ABI, libpython3.X minor, CUDA major, torch version), so a flat repository cannot stay installable once more than one build target is published. For rpm the problem shows up as a single mixed repodata; for deb as one suite that silently mixes glibc floors.

How the same problem is handled elsewhere

Every vendor that ships packages linking distro libraries splits by distribution release: Docker CE (linux/ubuntu/dists/<codename>, linux/<centos|fedora|rocky>/<ver>/<arch>), NVIDIA CUDA (repos/ubuntu2204, debian13, rhel9, fedora43, kylin10, …), Microsoft (ubuntu/22.04/prod, rhel/9/prod), ROCm, PGDG. Only self-contained /opt stacks (Intel oneAPI, Chrome) use one repo for all.

What I'd like to discuss (needs a Nexus administrator — I only have upload rights)

1. apt: one hosted repository per target distribution release, since a Nexus apt-hosted repo has exactly one distribution and apt has no group repos. Something like flagos-apt-ubuntu2204, flagos-apt-ubuntu2404, flagos-apt-debian13, later openKylin / deepin codenames. Architecture: all Python packages would be uploaded to each of them (one line in sources.list per user, no cross-repo pinning).
2. yum: keep one flagos-yum-hosted but set Repodata Depth so that <distro>/<ver>/<arch>/ (e.g. openeuler/24.03/x86_64, fedora/43/x86_64, rocky/9/x86_64) each get their own repodata.
3. How this composes with the existing per-vendor repositories (flagos-apt-nvidia, flagos-apt-mthreads, flagos-pypi-<vendor>): vendor × distro (flagos-apt-nvidia-ubuntu2204), or vendor as a component inside a per-distro repo, or keep vendor repos for SDK mirrors only and put FlagOS packages under the distro axis. I don't have a strong preference; whichever matches how the vendor repos are meant to be used.
4. Once the repositories exist, I'll update upload-nexus.yml to route by build target — the build workflows already encode it in the artifact name / rpm %{?dist} (e.g. rpm-packages-rocky9-cuda1260, ubuntu2204-cuda1280, .oe2403) — and add a per-caller input for the deb suite. Until then, publishing into the flat repos is fine for Architecture: all packages and for one agreed native target, and wrong for anything else.

Related: artifacts in the component repositories expire after 7 days, so a tag whose upload failed cannot be re-published without rebuilding; worth aligning retention-days or uploading straight from the build job.

cc @tengqm — happy to write up whichever option you prefer and prepare the workflow change; the four repositories that will start publishing are wired in FlagAttention#53, FlagSparse#49, FlagTree#1063, FlagCX#561.

[tengqm]

Hi, I'd prefer the "one apt repo per target distro" approach, which is cleaner for things like flagcx, libtriton-jit. As for yum repo, I agree to your suggestion.

I don't have a clear vision regarding the various python packages. That is where I wanted to to hear your inputs. For example, are we attempting to package 'torch_musa==2.9.1' into a deb/rpm package? What are the pros and cons for this effort, if that is what you are striving for?

[shiptux]

Thanks — agreed then: one apt repo per target distro, and yum with per-subpath
repodata. I will align the upload workflow routing with that layout.

On the Python packages, my suggestion is to draw the line by what the package
actually contains, not by language:

1. Pure-Python packages (arch:all / noarch), e.g. flag_gems — worth
packaging as deb/rpm. They are small, have stable dependencies, and native
packages give distro users one consistent install path (apt install the whole
FlagOS stack including the Python glue). We already build flag-gems debs, and
the same source package can be uploaded to every suite repo unchanged.

2. Binary wheels tied to a vendor toolchain, e.g. torch_musa==2.9.1 — I
would not package these as deb/rpm, for four reasons:

• The version matrix does not fit the distro model. torch_musa 2.9.1 is
  only valid against a specific torch build and a specific MUSA SDK release;
  multiply that by Python minor and distro release and you get a matrix that
  deb/rpm naming and dependency syntax cannot express well — pip environment
  markers and index-per-vendor handle this much better.
• Redistribution limits. The MUSA SDK itself is not redistributable, so a
  torch_musa deb/rpm in a public repo could not carry or depend on the
  runtime it actually needs — the resulting package would not be installable
  in a self-contained way.
• Size and churn. Torch-family wheels are in the GB range and release much
  faster than distro packaging can follow; mirroring them into apt/yum repos
  doubles storage and CI cost for no functional gain.
• Precedent. Debian/Fedora do not package GPU-enabled torch either; the
  ecosystem norm (NVIDIA included) is pip indexes and containers for this
  layer.

So the concrete proposal is a division of labor between the two channels:

• deb/rpm repos carry the native libraries (flagcx, libtriton-jit,
  flagtree, ...) plus the pure-Python FlagOS packages — installed with
  apt/dnf.
• The torch-adjacent wheels stay exactly where they are today: the
  existing flagos-pypi-<vendor> hosted repos on Nexus already serve them
  as-is, and users install them with
  pip install --index-url .../flagos-pypi-<vendor>/simple torch_musa.
  Nothing new to build or convert for this layer — for torch_musa==2.9.1
  specifically, my answer is "keep it pip-only, no deb/rpm".

Since deb/rpm dependency fields cannot express a dependency on a pip
package, the two layers are bridged softly rather than by hard Depends:
installation docs point to the pip index after the native packages are
installed (or, if we want something more discoverable, a small meta-package
whose Recommends/description points there). That keeps every deb/rpm we
publish installable on its own, while the fast-moving vendor stack stays
where its tooling works.

Happy to write this up as a short policy doc in this repo if that helps.

[tengqm]

1. Pure-Python packages (arch:all / noarch), e.g. flag_gems — worth packaging as deb/rpm. They are small, have stable dependencies, and native packages give distro users one consistent install path (apt install the whole FlagOS stack including the Python glue). We already build flag-gems debs, and the same source package can be uploaded to every suite repo unchanged.

There are the C++ operators variants to be considered. From code perspective, we have about 8 vendors providing C++ operators, although the maturity varies among them. My current approach is to make the CPP operators a subcomponent of FlagGems. When installing flag_gems, a user can optionally specify that the CPP operators are to be installed too. That is pip install flag_gems[cpp_cuda] or pip install flag_gems[cpp_ix]. Behind the scenes, there are flag_gems-<version>-py3-none-any.whl and flag_gems_cpp_cuda-<version>-cp312-cp312-linux_x86_64.whl colocated on the PyPI server. The latter one is an extra component of the former. The constraint is that <version> has to be an exact match.

This is an ongoing work which I'm now putting aside to focus on application image building. However, it is some background you may want to consider.

2. Binary wheels tied to a vendor toolchain, e.g. torch_musa==2.9.1 — I would not package these as deb/rpm, for four reasons:

* **The version matrix does not fit the distro model.** `torch_musa 2.9.1` is
  only valid against a specific torch build and a specific MUSA SDK release;
  multiply that by Python minor and distro release and you get a matrix that
  deb/rpm naming and dependency syntax cannot express well — pip environment
  markers and index-per-vendor handle this much better.

* **Redistribution limits.** The MUSA SDK itself is not redistributable, so a
  `torch_musa` deb/rpm in a public repo could not carry or depend on the
  runtime it actually needs — the resulting package would not be installable
  in a self-contained way.

* **Size and churn.** Torch-family wheels are in the GB range and release much
  faster than distro packaging can follow; mirroring them into apt/yum repos
  doubles storage and CI cost for no functional gain.

* **Precedent.** Debian/Fedora do not package GPU-enabled torch either; the
  ecosystem norm (NVIDIA included) is pip indexes and containers for this
  layer.

There are usually two "torch" packages for each backend (see configs.yaml). One is the upstream torch, customized by the vendor for bridging or masking out some operation. The package name is usually not changed, i.e. they are called 'torch==2.9.1+musa', for example. The other one is the vendor specific operator implementation package. It has a name like torch_musa, torch_npu, torch_gcu etc. The most annoying thing is about the former package. It cost me more than one month when attempting to teach uv how to differentiate one from another, and I FAILED.

So the concrete proposal is a division of labor between the two channels:

* **deb/rpm repos** carry the native libraries (flagcx, libtriton-jit,
  flagtree, ...) plus the pure-Python FlagOS packages — installed with
  apt/dnf.

* **The torch-adjacent wheels stay exactly where they are today**: the
  existing `flagos-pypi-<vendor>` hosted repos on Nexus already serve them
  as-is, and users install them with
  `pip install --index-url .../flagos-pypi-<vendor>/simple torch_musa`.
  Nothing new to build or convert for this layer — for `torch_musa==2.9.1`
  specifically, my answer is "keep it pip-only, no deb/rpm".

This proposal smells like a mixture of different things. The confusion to me is about the boundary between the two camps. The most confusing ones are the pure-python packages. What are the benefits to wrap them into deb/rpm packages?

Since deb/rpm dependency fields cannot express a dependency on a pip package, the two layers are bridged softly rather than by hard Depends: installation docs point to the pip index after the native packages are installed (or, if we want something more discoverable, a small meta-package whose Recommends/description points there). That keeps every deb/rpm we publish installable on its own, while the fast-moving vendor stack stays where its tooling works.

Happy to write this up as a short policy doc in this repo if that helps.

My not-so-vivid memory about rpm packages was that they don't codify dependencies between packages, they specify dependencies via symbols instead.

Yet another way to "wrap up" all these packages is the Cambricon way. They provide a huge Toolkit deb package for users to install. That Toolkit does three things: 1. self extract the component deb packages into a host path; 2. add a apt source pointing to that host path. 3. invoke a prebuilt script to install the component packages.