[BUG]:  PTX JIT compiler library not found

### Version

1.0.0

### Version

13.1

### Which installation method(s) does this occur on?

Pip

### Describe the bug.

`Failed to launch cuTile kernel: PTX JIT compiler library not found` in CUDA 13.1 container with driver 580.82.09 and visible `libnvidia-ptxjitcompiler.so.*`

## Environment

- **Host GPU**: NVIDIA RTX PRO 6000 Blackwell  
- **Host OS**: Pop!_OS / Ubuntu 22.04–based  
- **NVIDIA driver (host)**: 580.82.09 (`nvidia-smi`)[1][2]
- **CUDA toolkit (in container)**: 13.1.0 (`nvcc --version` reports `Cuda compilation tools, release 13.1, V13.1.80`)[1]
- **Container base image**:  
  - `[nvidia](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html)/cuda:13.1.0-devel-ubuntu22.04`  
- **NVIDIA Container Toolkit**: in use (container launched with `--gpus=all --runtime=nvidia`)[3]
- **Python**: 3.10 (system Python in container)  
- **cuTile Python**: installed via `pip install cuda-tile` (current latest from PyPI)[4]
- **TileIR / PTX compiler tools**: installed via `pip install nvidia-cuda-tileiras`[5]
- **CuPy**: `cupy-cuda13x` wheel[6]
- **PyTorch**: standard CPU/CUDA wheel (no `+cu131` build exists yet; not used in repro)[7]

## Dockerfile (core relevant parts)

```dockerfile
FROM nvidia/cuda:13.1.0-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential git wget curl ca-certificates \
      python3 python3-dev python3-venv python3-pip vim \
    && rm -rf /var/lib/apt/lists/*

ENV CUDA_HOME=/usr/local/cuda
ENV PATH=${CUDA_HOME}/bin:${PATH}
ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}

RUN python3 -m pip install --upgrade pip

# TileIR compiler package
RUN pip install nvidia-cuda-tileiras

# cuTile + CuPy
RUN pip install cuda-tile cupy-cuda13x

WORKDIR /workspace
COPY . /workspace
```

Container is run as:

```bash
docker run --rm -it \
  --gpus=all --ipc=host --ulimit memlock=-1 --runtime=nvidia \
  -v /home/jarvis/Documents/Code/cutile:/workspace \
  cutile_test \
  bash
```

I also tried adding:

```bash
-e LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
-e LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.82.09
```

with no behavioral change.

## Driver / toolkit checks inside container

```bash
nvidia-smi
```

Output (abridged):

- Driver Version: **580.82.09**  
- CUDA Version: **13.1**  
- GPU: **NVIDIA RTX PRO 6000 Blackwell**  

```bash
nvcc --version
```

Output:

- `Cuda compilation tools, release 13.1, V13.1.80`  

## PTX JIT library presence

Inside container:

```bash
ls -l /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so*
```

Shows:

- `/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.82.09`  
- `/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.590.44.01`  
- a `.so.1` symlink pointing at one of these (tried both directions).  

I also tried explicitly:

```bash
cd /usr/lib/x86_64-linux-gnu
rm -f libnvidia-ptxjitcompiler.so libnvidia-ptxjitcompiler.so.1
ln -s libnvidia-ptxjitcompiler.so.580.82.09 libnvidia-ptxjitcompiler.so.1
ln -s libnvidia-ptxjitcompiler.so.1 libnvidia-ptxjitcompiler.so
ldconfig
```

Then re-running the repro.

`LD_LIBRARY_PATH` inside container:

```bash
echo $LD_LIBRARY_PATH
# /usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
# (also tried prefixing /usr/lib/x86_64-linux-gnu)
```

## cuTile installation status

Inside container:

```bash
python3 - << 'EOF'
import cuda.tile as ct
import inspect, os, pkgutil

print("cuda.tile module:", ct)
spec = pkgutil.get_loader("cuda.tile")
print("loader:", spec)
print("file:", inspect.getfile(ct))
print("contents of package dir:")
print(os.listdir(os.path.dirname(inspect.getfile(ct))))
EOF
```

Output (abridged):

- `cuda.tile module: <module 'cuda.tile' from '/usr/local/lib/python3.10/dist-packages/cuda/tile/__init__.py'>`  
- Package dir contains `_cext.cpython-310-x86_64-linux-gnu.so` and the other expected cuTile Python files, so the compiled extension is present and imports successfully.[4]

## Repro 1: official quickstart

From `NVIDIA/cutile-python` repo cloned into `/workspace/cutile-python`:[8][9]

```bash
cd /workspace/cutile-python/samples/quickstart
python3 VectorAdd_quickstart.py
```

Result:

```text
Traceback (most recent call last):
  File "VectorAdd_quickstart.py", line 60, in <module>
    test()
  File "VectorAdd_quickstart.py", line 42, in test
    ct.launch(cp.cuda.get_current_stream(),
RuntimeError: Failed to launch cuTile kernel: PTX JIT compiler library not found
```

This matches the cuTile docs’ recommended launch pattern (`ct.launch(cp.cuda.get_current_stream(), grid, tile_kernel, ...)`).[9][10]

## Repro 2: minimal custom kernel

Minimal script:

```python
import cuda.tile as ct
import cupy as cp

TILE_SIZE = 16

@ct.kernel
def vadd(a, b, c):
    pid = ct.bid(0)
    a_tile = ct.load(a, index=(pid,), shape=(TILE_SIZE,))
    b_tile = ct.load(b, index=(pid,), shape=(TILE_SIZE,))
    ct.store(c, index=(pid,), tile=a_tile + b_tile)

def main():
    n = 1024
    a = cp.arange(n, dtype=cp.float32)
    b = cp.arange(n, dtype=cp.float32)
    c = cp.zeros_like(a)

    grid = (ct.cdiv(n, TILE_SIZE), 1, 1)
    ct.launch(cp.cuda.get_current_stream(), grid, vadd, (a, b, c))
    cp.cuda.get_current_stream().synchronize()
    print("OK", c[:5])

if __name__ == "__main__":
    main()
```

Running `python3 test.py` in the same container yields the **same** error:

```text
RuntimeError: Failed to launch cuTile kernel: PTX JIT compiler library not found
```

## What has already been tried

- Verified driver and toolkit compatibility (CUDA 13.1 + R580 driver, which meets published requirements).[2][11][1]
- Confirmed PTX JIT libraries exist and are accessible in the container (`libnvidia-ptxjitcompiler.so.*` present, with symlinks and `ldconfig`).[12]
- Ensured `/usr/lib/x86_64-linux-gnu` is on `LD_LIBRARY_PATH` inside the container.  
- Tried `LD_PRELOAD` with `libnvidia-ptxjitcompiler.so.580.82.09` when launching the container.  
- Verified that cuTile’s Python extension `_cext.cpython-310-x86_64-linux-gnu.so` is present and imports without error.  
- Reproduced the same error using:
  - The official `VectorAdd_quickstart.py` sample.[9]
  - A minimal custom kernel using the documented `ct.launch` API.[10][13]

Despite all of the above, any attempt to launch a cuTile kernel fails with:

> `RuntimeError: Failed to launch cuTile kernel: PTX JIT compiler library not found`

## Expected vs actual behavior

- **Expected**:  
  With a CUDA‑13.1‑compatible driver, visible PTX JIT libraries, and the documented cuTile Python installation, the quickstart sample and minimal kernels should compile and run successfully in a container environment.

- **Actual**:  
  cuTile kernel launches consistently fail at runtime with `PTX JIT compiler library not found`, even though:
  - The PTX JIT shared objects are present and on the library path.  
  - The GPU and CUDA context are otherwise working.  
  - cuTile’s own compiled extension is installed and importable.

## What would be helpful

- Confirmation whether cuTile Python currently supports CUDA 13.1 + driver 580.82.09 in containerized environments, or whether a newer driver (e.g., 590.x) is required for PTX Compiler API usage.[14][1]
- Guidance on any additional environment variables or configuration settings that cuTile expects to locate the PTX JIT library (beyond standard CUDA/driver setup).[15][16]
- If this is a known issue, a pointer to a `cuda-tile` build / wheel or workaround that is known to work with 13.1 in containers.

[1](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html)
[2](https://docs.nvidia.com/datacenter/tesla/drivers/supported-drivers-and-cuda-toolkit-versions.html)
[3](https://[stackoverflow](https://stackoverflow.com/questions/48786654/nvidia-driver-libraries-in-nvidia-cuda-image).com/questions/48786654/nvidia-driver-libraries-in-nvidia-cuda-image)
[4](https://pypi.org/project/cuda-tile/)
[5](https://www.piwheels.org/project/nvidia-cuda-tileiras/)
[6](https://docs.cupy.dev/en/stable/user_guide/cuda_api.html)
[7](https://pytorch.org/get-started/previous-versions/)
[8](https://github.com/NVIDIA/cutile-python)
[9](https://docs.nvidia.com/cuda/cutile-python/quickstart.html)
[10](https://docs.nvidia.com/cuda/cutile-python/execution.html)
[11](https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html)
[12](https://stackoverflow.com/questions/47258882/theano-gpu-support-ptx-jit-compiler-library-not-found)
[13](https://docs.nvidia.com/cuda/cutile-python/)
[14](https://ssojet.com/news/cuda-toolkit-131-release-notes-and-advanced-technical-insights)
[15](https://docs.nvidia.com/cuda/nvjitlink/index.html)
[16](https://docs.nvidia.com/cuda/ptx-compiler-api/index.html)
[17](https://docs.nvidia.com/cuda/cutile-python/quickstart.html#developer-tools)
[18](https://github.com/rusty1s/[pytorch](https://pytorch.org/get-started/previous-versions/)_sparse/issues/78)
[19](https://github.com/m4rs-mt/ILGPU/issues/724)
[20](https://forums.developer.nvidia.com/t/jit-compilation-ptx-to-machine-code-may-fail-for-certain-gpus/36440)
[21](https://forum.manjaro.org/t/testing-update-2021-11-22-kernels-systemd-249-7-wine-haskell/91683/19)
[22](https://forums.developer.nvidia.com/t/disable-ptx-jit-compilation/264400)
[23](https://discuss.pytorch.org/t/a-ptx-jit-compilation-failed/57927)
[24](https://news.ycombinator.com/item?id=43581584)
[25](https://discourse.nixos.org/t/using-cuda-enabled-packages-on-non-nixos-systems/17788)
[26](https://github.com/NVIDIA/apex/issues/314)
[27](https://github.com/jax-ml/jax/issues/9644)
[28](https://github.com/NVIDIA/numba-cuda/issues/163)
[29](https://gromacs.bioexcel.eu/t/ptx-jit-compilation-failed-on-5070ti-with-gromacs-2025/12350)
[30](https://forums.developer.nvidia.com/t/ptx-jit-spills-registers-in-trivial-programs/279620)
[31](https://www.reddit.com/r/programming/comments/1jv3emt/nvidia_drops_a_gamechanger_native_python_support/)

### Minimum reproducible example

```shell
import cuda.tile as ct
import cupy as cp

TILE_SIZE = 16

@ct.kernel
def vadd(a, b, c):
    pid = ct.bid(0)
    a_tile = ct.load(a, index=(pid,), shape=(TILE_SIZE,))
    b_tile = ct.load(b, index=(pid,), shape=(TILE_SIZE,))
    ct.store(c, index=(pid,), tile=a_tile + b_tile)

def main():
    n = 1024
    a = cp.arange(n, dtype=cp.float32)
    b = cp.arange(n, dtype=cp.float32)
    c = cp.zeros_like(a)

    grid = (ct.cdiv(n, TILE_SIZE), 1, 1)
    ct.launch(cp.cuda.get_current_stream(), grid, vadd, (a, b, c))
    cp.cuda.get_current_stream().synchronize()
    print("OK", c[:5])

if __name__ == "__main__":
    main()
```

### Relevant log output

```shell
Traceback (most recent call last):
  File "VectorAdd_quickstart.py", line 60, in <module>
    test()
  File "VectorAdd_quickstart.py", line 42, in test
    ct.launch(cp.cuda.get_current_stream(),
RuntimeError: Failed to launch cuTile kernel: PTX JIT compiler library not found
```

### Full env printout

```shell
root@263d4f5fc18a:/workspace/cutile-python# ./print_env.sh
<details><summary>Click here to see environment details</summary><pre>

     **git***
     commit 3912ddac97ddee2e4c733fe6d4c972a46deef1ea (HEAD -> main, tag: v1.0.1, origin/main, origin/HEAD)
     Author: Jay Gu <jagu@nvidia.com>
     Date:   Wed Dec 17 14:44:34 2025 -0800

     Add release notes for v1.0.1

     Signed-off-by: Jay Gu <jagu@nvidia.com>
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=22.04
     DISTRIB_CODENAME=jammy
     DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
     PRETTY_NAME="Ubuntu 22.04.5 LTS"
     NAME="Ubuntu"
     VERSION_ID="22.04"
     VERSION="22.04.5 LTS (Jammy Jellyfish)"
     VERSION_CODENAME=jammy
     ID=ubuntu
     ID_LIKE=debian
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     UBUNTU_CODENAME=jammy
     Linux 263d4f5fc18a 6.17.4-76061704-generic #202510191616~1762410050~22.04~898873a SMP PREEMPT_DYNAMIC Thu N x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Sun Dec 28 12:07:44 2025
     +-----------------------------------------------------------------------------------------+
     | NVIDIA-SMI 580.82.09              Driver Version: 580.82.09      CUDA Version: 13.1     |
     +-----------------------------------------+------------------------+----------------------+
     | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
     |                                         |                        |               MIG M. |
     |=========================================+========================+======================|
     |   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:02:00.0 Off |                  Off |
     | 30%   23C    P8             13W /  600W |     714MiB /  97887MiB |      0%      Default |
     |                                         |                        |                  N/A |
     +-----------------------------------------+------------------------+----------------------+

     +-----------------------------------------------------------------------------------------+
     | Processes:                                                                              |
     |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
     |        ID   ID                                                               Usage      |
     |=========================================================================================|
     |  No running processes found                                                             |
     +-----------------------------------------------------------------------------------------+

     ***CPU***
     Architecture:                            x86_64
     CPU op-mode(s):                          32-bit, 64-bit
     Address sizes:                           46 bits physical, 48 bits virtual
     Byte Order:                              Little Endian
     CPU(s):                                  24
     On-line CPU(s) list:                     0-23
     Vendor ID:                               GenuineIntel
     Model name:                              Intel(R) Core(TM) Ultra 9 285K
     CPU family:                              6
     Model:                                   198
     Thread(s) per core:                      1
     Core(s) per socket:                      24
     Socket(s):                               1
     Stepping:                                2
     CPU max MHz:                             5800.0000
     CPU min MHz:                             800.0000
     BogoMIPS:                                7372.80
     Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni lam wbnoinvd dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid bus_lock_detect movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
     Virtualization:                          VT-x
     L1d cache:                               768 KiB (20 instances)
     L1i cache:                               1.3 MiB (20 instances)
     L2 cache:                                40 MiB (12 instances)
     L3 cache:                                36 MiB (1 instance)
     NUMA node(s):                            1
     NUMA node0 CPU(s):                       0-23
     Vulnerability Gather data sampling:      Not affected
     Vulnerability Ghostwrite:                Not affected
     Vulnerability Indirect target selection: Not affected
     Vulnerability Itlb multihit:             Not affected
     Vulnerability L1tf:                      Not affected
     Vulnerability Mds:                       Not affected
     Vulnerability Meltdown:                  Not affected
     Vulnerability Mmio stale data:           Not affected
     Vulnerability Old microcode:             Not affected
     Vulnerability Reg file data sampling:    Not affected
     Vulnerability Retbleed:                  Not affected
     Vulnerability Spec rstack overflow:      Not affected
     Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
     Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS Not affected; BHI BHI_DIS_S
     Vulnerability Srbds:                     Not affected
     Vulnerability Tsa:                       Not affected
     Vulnerability Tsx async abort:           Not affected
     Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

     ***CMake***

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
     Copyright (C) 2021 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


     ***nvcc***
     /usr/local/cuda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2025 NVIDIA Corporation
     Built on Fri_Nov__7_07:23:37_PM_PST_2025
     Cuda compilation tools, release 13.1, V13.1.80
     Build cuda_13.1.r13.1/compiler.36836380_0

     ***Python***

     ***Environment Variables***
     PATH                            : /root/.local/bin:/usr/local/cuda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
     LD_LIBRARY_PATH                 : /usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    :
     PYTHON_PATH                     :

     conda not found
     ***pip packages***
     /usr/local/bin/pip
     Package                  Version
     ------------------------ ---------
     cuda-tile                1.0.1
     cupy-cuda13x             13.6.0
     fastrlock                0.8.3
     filelock                 3.20.1
     fsspec                   2025.12.0
     Jinja2                   3.1.6
     MarkupSafe               3.0.3
     mpmath                   1.3.0
     networkx                 3.4.2
     numpy                    2.2.6
     nvidia-cublas-cu12       12.8.4.1
     nvidia-cuda-cupti-cu12   12.8.90
     nvidia-cuda-nvrtc-cu12   12.8.93
     nvidia-cuda-runtime-cu12 12.8.90
     nvidia-cuda-tileiras     13.1.80
     nvidia-cudnn-cu12        9.10.2.21
     nvidia-cufft-cu12        11.3.3.83
     nvidia-cufile-cu12       1.13.1.3
     nvidia-curand-cu12       10.3.9.90
     nvidia-cusolver-cu12     11.7.3.90
     nvidia-cusparse-cu12     12.5.8.93
     nvidia-cusparselt-cu12   0.7.1
     nvidia-nccl-cu12         2.27.5
     nvidia-nvjitlink-cu12    12.8.93
     nvidia-nvshmem-cu12      3.3.20
     nvidia-nvtx-cu12         12.8.90
     pillow                   12.0.0
     pip                      25.3
     setuptools               59.6.0
     sympy                    1.14.0
     torch                    2.9.1
     torchvision              0.24.1
     triton                   3.5.1
     typing_extensions        4.15.0
     wheel                    0.37.1

</pre></details>
```

### Other/Misc.

_No response_

### Contributing Guidelines

- [x] I agree to follow cuTile Python's contributing guidelines
- [x] I have searched the [open bugs](https://github.com/nvidia/cutile-python/issues?q=is%3Aopen+is%3Aissue+label%3Abug) and have found no duplicates for this bug report

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BUG]: PTX JIT compiler library not found #44

Version

Version

Which installation method(s) does this occur on?

Describe the bug.

Environment

Dockerfile (core relevant parts)

Driver / toolkit checks inside container

PTX JIT library presence

cuTile installation status

Repro 1: official quickstart

Repro 2: minimal custom kernel

What has already been tried

Expected vs actual behavior

What would be helpful

Minimum reproducible example

Relevant log output

Full env printout

Other/Misc.

Contributing Guidelines

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]: PTX JIT compiler library not found #44

Description

Version

Version

Which installation method(s) does this occur on?

Describe the bug.

Environment

Dockerfile (core relevant parts)

Driver / toolkit checks inside container

PTX JIT library presence

cuTile installation status

Repro 1: official quickstart

Repro 2: minimal custom kernel

What has already been tried

Expected vs actual behavior

What would be helpful

Minimum reproducible example

Relevant log output

Full env printout

Other/Misc.

Contributing Guidelines

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions