wheels: 26.10 `cudf-cu12` wheels depend on `cuda-bindings>=12.9.3` and seem like they shouldn't

[jameslamb]

Description

See NVIDIA/cuml#8613 (comment)

cudf-cu12's latest 26.10 nightlies appear to depend on cuda-bindings>=12.9.3, despite every reference on the release/26.10 branch saying cuda-bindings>=12.9.2 😬

Reproducible Example

$ docker run --rm -it rapidsai/citestwheel:26.10-cuda12.9.2-ubuntu22.04-py3.11 bash

$ pip download --no-deps 'cudf-cu12>=26.10.0a503,==26.10.*'
...
Saved ./cudf_cu12-26.10.0a503.post260908235645-cp311-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

$ pip install pkginfo
$ pkginfo --json ./cudf*.whl | grep cuda
    "cuda-bindings<13.0,>=12.9.3",
    "cupy-cuda12x[ctk]!=14.1.0,>=14.0.1",
    "numba-cuda-mlir[cu12]>=0.3.0",
    "numba-cuda[cu12]>=0.22.2",

Notes

It's almost like wheels are getting built from main but published with version 26.10 or something 😬

[jameslamb]

Aha, I think I see it!!

I think this was a mistake in the release process. It looks like there was a cudf-cu12=26.10.0a503 that was published September 8.

I suspect that was built from main after #21928 merged and got a 26.10.* version accidentally because of some interaction with when the v26.12.00a tag was created (unsure exactly what).

Check this out.

$ pip install 'cudf-cu12>=26.10.0a0,==26.10.*' 'cuda-bindings==12.9.2'
$ pip list | grep -E 'cudf|bindings'
cuda-bindings                    12.9.2
cudf-cu12                        26.10.0a501.post260910193355
libcudf-cu12                     26.10.0a503.post260908235645
pylibcudf-cu12                   26.10.0a501.post260910193355

$ pip install 'cudf-cu12>=26.10.0a0,==26.10.*'
$ pip list | grep -E 'cudf|bindings'
cuda-bindings                    12.9.7
cudf-cu12                        26.10.0a503.post260908235645
libcudf-cu12                     26.10.0a503.post260908235645
pylibcudf-cu12                   26.10.0a503.post260908235645

Notice that those .a503 versions were built on September 8th but the .a501 versions were built on September 10th!

A package with a larger commit distance should always have a later date. The only explanation I see for this is that the .a503 ones were built and then AFTER that, the v26.12.00a tag was stuck somewhere "closer" on main to v26.10.00a.

I think this would be fixed by either of the following:

1. deleting those .a503 nightly wheels
2. just waiting for a few more builds on release/26.10 to get past 503 (most recent one was 501, so just a few more commits)

[bdice]

We definitely need cuda-bindings >=12.9.3 for a bugfix (see discussion on #21928). Feel free to delete the nightlies causing this issue.

[jameslamb]

We definitely need cuda-bindings >=12.9.3 for a bugfix (see discussion on #21928).

Clarifying this from offline discussion:

• Add rapids-doctor basic health checks #21928 changes aren't in cuDF 26.10, just 26.12 onwards
• we don't want to force a cuda-bindings>=12.9.3 onto 26.10

@msarahan and I deleted the offending nightly packages.

Wheels and conda packages with versions like 26.10.0a502 and 26.10.0a503 had changes intended for 26.12 but were published as 26.10.

Now that those are gone, this issue should be resolved.

$ pip install 'cudf-cu12>=26.10.0a0,==26.10.*' 'cuda-bindings==12.9.2'
$ pip list | grep -E 'cudf|bindings'
cuda-bindings                    12.9.2
cudf-cu12                        26.10.0a501.post260910193355
libcudf-cu12                     26.10.0a501.post260910193355
pylibcudf-cu12                   26.10.0a501.post260910193355

That's looking much better.

I think this can be closed.

[jameslamb]

Deleting those problematic versions did help, CI for projects depending on cudf seems to not have this problem any more.

I looked into it more closely, here's exactly what happened.

1. the release/26.10 branch was cut from 70bfef9 (Sep 8, 12:00p CT)
2. the v26.12.00a tag was pushed e44e3cf (Sep 8, 7:50p CT)
3. in the 8ish hours between those, 19 other commits landed on main: 70bfef9...e44e3cf
4. one of those was e518040 (from Add rapids-doctor basic health checks #21928), the one that set cuda-bindings>=12.9.3
5. all those nightly packages from commits produced in that window had 26.10.* versions, despite never being on the relase/26.10 branch.

This could have been avoided by tagging the commit immediately after the release branch was cut as v26.12.00a.

[msarahan]

This could have been avoided by tagging the commit immediately after the release branch was cut as v26.12.00a.

The point of confusion is that I thought the update-version PRs needed to be merged before applying the alpha tag. I will clarify our docs to avoid this next time.