Skip to content

Keep import cudf working after a GPU read - #6

Open
Intron7 wants to merge 1 commit into
mainfrom
fix-nvcomp-import-order
Open

Intron7 wants to merge 1 commit into
mainfrom
fix-nvcomp-import-order

Conversation

@Intron7

@Intron7 Intron7 commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

nvCOMP's __init__ loads its library with import nvidia.libnvcomp, nvidia.libnvcomp.load_library() and then
del nvidia.libnvcomp. That removes the attribute from the nvidia package but leaves the module in sys.modules, so
a later import nvidia.libnvcomp doesn't set it again. libcudf's loader (libcudf/load.py) runs exactly that import
and then nvidia.libnvcomp.load_library(). zarr-kvikio imports nvCOMP at its first GPU decode, so importing cudf, cuML
or rapids-singlecell after a GPU read raised

AttributeError: module 'nvidia' has no attribute 'libnvcomp'

The pipeline now imports nvCOMP through _nvcomp(), which puts the attribute back. It keeps
import nvidia.nvcomp as nv, which (unlike reading nvidia.nvcomp) also works while another thread's first import is
finishing.

Not covered: a libcudf loader that runs in another thread while the process imports nvCOMP for the first time
(~50 ms) can still fail, since the attribute is deleted inside nvCOMP's __init__.

Tested with this patch on an RTX PRO 6000 (CUDA 13, Python 3.14):

  • test_reads_keep_cudf_importable fails without the fix and passes with it.
  • The full suite passes on this branch (423 passed, 5 skipped, 4 xfailed).
  • A GPU decode followed by import rapids_singlecell raised before and works now.

nvCOMP's import deletes the libnvcomp attribute of the nvidia package
but leaves nvidia.libnvcomp in sys.modules. libcudf's loader runs
`import nvidia.libnvcomp` and then `nvidia.libnvcomp.load_library()`,
so importing cudf, cuML or rapids-singlecell after a GPU read raised
AttributeError. The pipeline now imports nvCOMP through one helper
that puts the attribute back.

Signed-off-by: Intron7 <sdicks@nvidia.com>
@codecov-commenter

codecov-commenter commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.77%. Comparing base (1004757) to head (332a78d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   96.76%   96.77%   +0.01%     
==========================================
  Files           6        6              
  Lines        1175     1180       +5     
==========================================
+ Hits         1137     1142       +5     
  Misses         38       38              
Flag Coverage Δ
gpu 96.61% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/zarr_kvikio/_pipeline.py 97.36% <100.00%> (+0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants