Cannot train a model due to "Cannot use GPU, CuPy is not installed", even though CuPy is installed #12996

tpanza · 2023-09-21T00:37:05Z

tpanza
Sep 21, 2023

Cam someone please help me understand and resolve this very strange and frustrating error?

OS: Amazon Linux release 2 (Karoo)
uname -r: 4.14.248-189.473.amzn2.x86_64
NVIDIA Driver Version: 450.142.00
CUDA Version: 11.0
Tesla T4
spaCy version: 3.6.1
Platform: Linux-4.14.248-189.473.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.9.18

I am seeing this stack trace, even though CuPy is clearly installed. Verified the installation using pkg_resources.

Traceback (most recent call last):
...
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/spacy/cli/train.py", line 76, in train
    setup_gpu(use_gpu)
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/spacy/cli/_util.py", line 599, in setup_gpu
    require_gpu(use_gpu)
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/thinc/util.py", line 200, in require_gpu
    raise ValueError("Cannot use GPU, CuPy is not installed")
ValueError: Cannot use GPU, CuPy is not installed

I added this small snippet of debug code right before the train, which confirmed that CuPy really is there, but somehow it doesn't get found..??

    env = dict(tuple(str(ws).split()) for ws in pkg_resources.working_set)
    for k, v in env.items():
        logger.report_text(msg=f"{k}: {v}", print_console=True)

Full output of creating the venv:

1695253099538 myhostname:0 DEBUG Looking in indexes: https://nexus.mycompany-internal.com/repository/pypi/simple
Collecting Cython
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/cython/3.0.2/Cython-3.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
Installing collected packages: Cython
Successfully installed Cython-3.0.2
Trying pip install: /home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/requirements.txt
Looking in indexes: https://nexus.mycompany-internal.com/repository/pypi/simple
Collecting numpy==1.25.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/numpy/1.25.1/numpy-1.25.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.7 MB)
Installing collected packages: numpy

1695253104848 myhostname:0 DEBUG Successfully installed numpy-1.25.1
Looking in indexes: https://nexus.mycompany-internal.com/repository/pypi/simple
Collecting blis==0.7.9
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/blis/0.7.9/blis-0.7.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.2 MB)
Collecting boto3==1.28.38
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/boto3/1.28.38/boto3-1.28.38-py3-none-any.whl (135 kB)
Collecting botocore==1.31.38
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/botocore/1.31.38/botocore-1.31.38-py3-none-any.whl (11.2 MB)
Collecting catalogue==2.0.8
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/catalogue/2.0.8/catalogue-2.0.8-py3-none-any.whl (17 kB)
Collecting certifi==2023.5.7
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/certifi/2023.5.7/certifi-2023.5.7-py3-none-any.whl (156 kB)
Collecting charset-normalizer==3.2.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/charset-normalizer/3.2.0/charset_normalizer-3.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (202 kB)
Collecting click==8.1.5
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/click/8.1.5/click-8.1.5-py3-none-any.whl (98 kB)

1695253110338 myhostname:0 DEBUG Collecting cmake==3.26.4
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/cmake/3.26.4/cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB)
Collecting confection==0.1.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/confection/0.1.0/confection-0.1.0-py3-none-any.whl (34 kB)
Collecting cymem==2.0.7
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/cymem/2.0.7/cymem-2.0.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35 kB)
Collecting fastrlock==0.8.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/fastrlock/0.8.1/fastrlock-0.8.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_24_x86_64.whl (49 kB)
Collecting filelock==3.12.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/filelock/3.12.2/filelock-3.12.2-py3-none-any.whl (10 kB)
Collecting flask==2.3.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/flask/2.3.2/Flask-2.3.2-py3-none-any.whl (96 kB)
Collecting fsspec==2023.6.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/fsspec/2023.6.0/fsspec-2023.6.0-py3-none-any.whl (163 kB)
Collecting huggingface-hub==0.16.4
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/huggingface-hub/0.16.4/huggingface_hub-0.16.4-py3-none-any.whl (268 kB)
Collecting idna==3.4
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/idna/3.4/idna-3.4-py3-none-any.whl (61 kB)
Collecting jinja2==3.1.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/jinja2/3.1.2/Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting jmespath==1.0.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/jmespath/1.0.1/jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting langcodes==3.3.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/langcodes/3.3.0/langcodes-3.3.0-py3-none-any.whl (181 kB)
Processing /home/ec2-user/.cache/pip/wheels/99/c2/e3/ed38537e191b4789b6b7ac05a8ee42446a2104e20225feee57/lit-16.0.6-py3-none-any.whl
Collecting markupsafe==2.1.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/markupsafe/2.1.3/MarkupSafe-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting mpmath==1.3.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/mpmath/1.3.0/mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting murmurhash==1.0.9
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/murmurhash/1.0.9/murmurhash-1.0.9-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21 kB)
Collecting networkx==3.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/networkx/3.1/networkx-3.1-py3-none-any.whl (2.1 MB)
Requirement already satisfied: numpy==1.25.1 in /home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages (from -r /tmp/requirements_6lfzgpfu.txt (line 26)) (1.25.1)
Collecting packaging==23.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/packaging/23.1/packaging-23.1-py3-none-any.whl (48 kB)
Collecting pandas==2.0.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pandas/2.0.3/pandas-2.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)

1695253115642 myhostname:0 DEBUG Collecting pathy==0.10.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pathy/0.10.2/pathy-0.10.2-py3-none-any.whl (48 kB)
Collecting preshed==3.0.8
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/preshed/3.0.8/preshed-3.0.8-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (128 kB)
Collecting pydantic==1.10.11
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pydantic/1.10.11/pydantic-1.10.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
Collecting python-dateutil==2.8.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/python-dateutil/2.8.2/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz==2023.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pytz/2023.3/pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting pyyaml==6.0.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pyyaml/6.0.1/PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB)
Collecting regex==2023.6.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/regex/2023.6.3/regex-2023.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)
Collecting requests==2.31.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/requests/2.31.0/requests-2.31.0-py3-none-any.whl (62 kB)
Collecting s3transfer==0.6.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/s3transfer/0.6.2/s3transfer-0.6.2-py3-none-any.whl (79 kB)
Collecting safetensors==0.3.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/safetensors/0.3.1/safetensors-0.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Collecting six==1.16.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/six/1.16.0/six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting smart-open==6.3.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/smart-open/6.3.0/smart_open-6.3.0-py3-none-any.whl (56 kB)
Collecting spacy[cuda-autodetect,lookups,transformers]==3.6.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy/3.6.1/spacy-3.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB)

1695253120937 myhostname:0 DEBUG Collecting spacy-alignments==0.9.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy-alignments/0.9.0/spacy_alignments-0.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting spacy-legacy==3.0.12
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy-legacy/3.0.12/spacy_legacy-3.0.12-py2.py3-none-any.whl (29 kB)
Collecting spacy-loggers==1.0.5
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy-loggers/1.0.5/spacy_loggers-1.0.5-py3-none-any.whl (22 kB)
Collecting srsly==2.4.7
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/srsly/2.4.7/srsly-2.4.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (492 kB)
Collecting sympy==1.12
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/sympy/1.12/sympy-1.12-py3-none-any.whl (5.7 MB)
Collecting thinc==8.1.10
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/thinc/8.1.10/thinc-8.1.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (924 kB)
Collecting tokenizers==0.13.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/tokenizers/0.13.3/tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
Collecting tqdm==4.65.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/tqdm/4.65.0/tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting transformers==4.30.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/transformers/4.30.2/transformers-4.30.2-py3-none-any.whl (7.2 MB)
Collecting triton==2.0.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/triton/2.0.0/triton-2.0.0-1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)

1695253126252 myhostname:0 DEBUG Collecting typer==0.7.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/typer/0.7.0/typer-0.7.0-py3-none-any.whl (38 kB)
Collecting typing-extensions==4.7.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/typing-extensions/4.7.1/typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting tzdata==2023.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/tzdata/2023.3/tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting urllib3==1.25.11
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/urllib3/1.25.11/urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting wasabi==1.1.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/wasabi/1.1.2/wasabi-1.1.2-py3-none-any.whl (27 kB)
Collecting clearml==1.12.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/clearml/1.12.2/clearml-1.12.2-py2.py3-none-any.whl (1.1 MB)
Collecting importlib-metadata>=3.6.0; python_version < "3.10"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/importlib-metadata/6.8.0/importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting Werkzeug>=2.3.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/werkzeug/2.3.7/werkzeug-2.3.7-py3-none-any.whl (242 kB)
Collecting itsdangerous>=2.1.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/itsdangerous/2.1.2/itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting blinker>=1.6.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/blinker/1.6.2/blinker-1.6.2-py3-none-any.whl (13 kB)
Requirement already satisfied: setuptools in /home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages (from spacy[cuda-autodetect,lookups,transformers]==3.6.1->-r /tmp/requirements_6lfzgpfu.txt (line 41)) (68.2.0)
Processing /home/ec2-user/.cache/pip/wheels/c5/80/90/216711ae0fcc9fb1be23535b773d6de1be0d9f00f9f6465bcb/cupy_wheel-12.2.0-py3-none-any.whl
Collecting spacy-lookups-data<1.1.0,>=1.0.3; extra == "lookups"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy-lookups-data/1.0.5/spacy_lookups_data-1.0.5-py2.py3-none-any.whl (98.5 MB)

1695253131570 myhostname:0 DEBUG Collecting spacy-transformers<1.3.0,>=1.1.2; extra == "transformers"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/spacy-transformers/1.2.5/spacy_transformers-1.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (193 kB)
Collecting torch

1695253142158 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/torch/2.0.1/torch-2.0.1-cp39-cp39-manylinux1_x86_64.whl (619.9 MB)

1695253152544 myhostname:0 DEBUG Collecting pathlib2>=2.3.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pathlib2/2.3.7.post1/pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)
Collecting furl>=2.0.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/furl/2.1.3/furl-2.1.3-py2.py3-none-any.whl (20 kB)
Collecting pyparsing>=2.0.3
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pyparsing/3.1.1/pyparsing-3.1.1-py3-none-any.whl (103 kB)
Collecting jsonschema>=2.6.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/jsonschema/4.19.0/jsonschema-4.19.0-py3-none-any.whl (83 kB)
Collecting pyjwt<2.5.0,>=2.4.0; python_version > "3.5"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pyjwt/2.4.0/PyJWT-2.4.0-py3-none-any.whl (18 kB)
Collecting psutil>=3.4.2
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/psutil/5.9.5/psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)

1695253158070 myhostname:0 DEBUG Collecting Pillow>=4.1.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/pillow/10.0.1/Pillow-10.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.5 MB)
Collecting attrs>=18.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/attrs/23.1.0/attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting zipp>=0.5
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/zipp/3.17.0/zipp-3.17.0-py3-none-any.whl (7.4 kB)
Collecting cupy-cuda111==12.2.0
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/cupy-cuda111/12.2.0/cupy_cuda111-12.2.0-cp39-cp39-manylinux2014_x86_64.whl (102.0 MB)
Collecting nvidia-nccl-cu11==2.14.3; platform_system == "Linux" and platform_machine == "x86_64"

1695253163487 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-nccl-cu11/2.14.3/nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)
Collecting nvidia-cuda-runtime-cu11==11.7.99; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cuda-runtime-cu11/11.7.99/nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)

1695253168814 myhostname:0 DEBUG Collecting nvidia-cuda-nvrtc-cu11==11.7.99; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cuda-nvrtc-cu11/11.7.99/nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
Collecting nvidia-cudnn-cu11==8.5.0.96; platform_system == "Linux" and platform_machine == "x86_64"

1695253179243 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cudnn-cu11/8.5.0.96/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)

1695253184681 myhostname:0 DEBUG Collecting nvidia-cublas-cu11==11.10.3.66; platform_system == "Linux" and platform_machine == "x86_64"

1695253195285 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cublas-cu11/11.10.3.66/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)

1695253200674 myhostname:0 DEBUG Collecting nvidia-cuda-cupti-cu11==11.7.101; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cuda-cupti-cu11/11.7.101/nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
Collecting nvidia-nvtx-cu11==11.7.91; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-nvtx-cu11/11.7.91/nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)
Collecting nvidia-cusparse-cu11==11.7.4.91; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cusparse-cu11/11.7.4.91/nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)

1695253206235 myhostname:0 DEBUG Collecting nvidia-curand-cu11==10.2.10.91; platform_system == "Linux" and platform_machine == "x86_64"
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-curand-cu11/10.2.10.91/nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)
Collecting nvidia-cufft-cu11==10.9.0.58; platform_system == "Linux" and platform_machine == "x86_64"

1695253211704 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cufft-cu11/10.9.0.58/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)
Collecting nvidia-cusolver-cu11==11.4.0.1; platform_system == "Linux" and platform_machine == "x86_64"

1695253217067 myhostname:0 DEBUG   Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/nvidia-cusolver-cu11/11.4.0.1/nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)
Collecting orderedmultidict>=1.0.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/orderedmultidict/1.0.1/orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB)
Collecting referencing>=0.28.4
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/referencing/0.30.2/referencing-0.30.2-py3-none-any.whl (25 kB)
Collecting rpds-py>=0.7.1
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/rpds-py/0.10.3/rpds_py-0.10.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting jsonschema-specifications>=2023.03.6
  Using cached https://nexus.mycompany-internal.com/repository/pypi/packages/jsonschema-specifications/2023.7.1/jsonschema_specifications-2023.7.1-py3-none-any.whl (17 kB)
Requirement already satisfied: wheel in /home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages (from nvidia-cuda-runtime-cu11==11.7.99; platform_system == "Linux" and platform_machine == "x86_64"->torch->triton==2.0.0->-r /tmp/requirements_6lfzgpfu.txt (line 51)) (0.41.2)
Installing collected packages: blis, six, python-dateutil, urllib3, jmespath, botocore, s3transfer, boto3, catalogue, certifi, charset-normalizer, click, cmake, srsly, typing-extensions, pydantic, confection, cymem, fastrlock, filelock, zipp, importlib-metadata, markupsafe, Werkzeug, itsdangerous, blinker, jinja2, flask, fsspec, packaging, tqdm, idna, requests, pyyaml, huggingface-hub, langcodes, lit, mpmath, murmurhash, networkx, pytz, tzdata, pandas, smart-open, typer, pathy, preshed, regex, safetensors, spacy-loggers, wasabi, thinc, spacy-legacy, cupy-cuda111, cupy-wheel, spacy-lookups-data, tokenizers, transformers, spacy-alignments, sympy, nvidia-nccl-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, triton, nvidia-cublas-cu11, nvidia-cudnn-cu11, nvidia-cuda-cupti-cu11, nvidia-nvtx-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cusolver-cu11, torch, spacy-transformers, spacy, pathlib2, orderedmultidict, furl, pyparsing, rpds-py, attrs, referencing, jsonschema-specifications, jsonschema, pyjwt, psutil, Pillow, clearml

1695253282928 myhostname:0 DEBUG Successfully installed Pillow-10.0.1 Werkzeug-2.3.7 attrs-23.1.0 blinker-1.6.2 blis-0.7.9 boto3-1.28.38 botocore-1.31.38 catalogue-2.0.8 certifi-2023.5.7 charset-normalizer-3.2.0 clearml-1.12.2 click-8.1.5 cmake-3.26.4 confection-0.1.0 cupy-cuda111-12.2.0 cupy-wheel-12.2.0 cymem-2.0.7 fastrlock-0.8.1 filelock-3.12.2 flask-2.3.2 fsspec-2023.6.0 furl-2.1.3 huggingface-hub-0.16.4 idna-3.4 importlib-metadata-6.8.0 itsdangerous-2.1.2 jinja2-3.1.2 jmespath-1.0.1 jsonschema-4.19.0 jsonschema-specifications-2023.7.1 langcodes-3.3.0 lit-16.0.6 markupsafe-2.1.3 mpmath-1.3.0 murmurhash-1.0.9 networkx-3.1 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 orderedmultidict-1.0.1 packaging-23.1 pandas-2.0.3 pathlib2-2.3.7.post1 pathy-0.10.2 preshed-3.0.8 psutil-5.9.5 pydantic-1.10.11 pyjwt-2.4.0 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3 pyyaml-6.0.1 referencing-0.30.2 regex-2023.6.3 requests-2.31.0 rpds-py-0.10.3 s3transfer-0.6.2 safetensors-0.3.1 six-1.16.0 smart-open-6.3.0 spacy-3.6.1 spacy-alignments-0.9.0 spacy-legacy-3.0.12 spacy-loggers-1.0.5 spacy-lookups-data-1.0.5 spacy-transformers-1.2.5 srsly-2.4.7 sympy-1.12 thinc-8.1.10 tokenizers-0.13.3 torch-2.0.1 tqdm-4.65.0 transformers-4.30.2 triton-2.0.0 typer-0.7.0 typing-extensions-4.7.1 tzdata-2023.3 urllib3-1.25.11 wasabi-1.1.2 zipp-3.17.0
Adding venv into cache: /home/ec2-user/.clearml/venvs-builds/3.9
Running task id [6be6710c02774713bd41192794e3bf0c]:
[.]$ /home/ec2-user/.clearml/venvs-builds/3.9/bin/python -u training/spacy_trainmodel.py
Summary - installed python packages:
pip:
- attrs==23.1.0
- blinker==1.6.2
- blis==0.7.9
- boto3==1.28.38
- botocore==1.31.38
- catalogue==2.0.8
- certifi==2023.5.7
- charset-normalizer==3.2.0
- clearml==1.12.2
- click==8.1.5
- cmake==3.26.4
- confection==0.1.0
- cupy-cuda111==12.2.0
- cupy-wheel==12.2.0
- cymem==2.0.7
- Cython==3.0.2
- fastrlock==0.8.1
- filelock==3.12.2
- Flask==2.3.2
- fsspec==2023.6.0
- furl==2.1.3
- huggingface-hub==0.16.4
- idna==3.4
- importlib-metadata==6.8.0
- itsdangerous==2.1.2
- Jinja2==3.1.2
- jmespath==1.0.1
- jsonschema==4.19.0
- jsonschema-specifications==2023.7.1
- langcodes==3.3.0
- lit==16.0.6
- MarkupSafe==2.1.3
- mpmath==1.3.0
- murmurhash==1.0.9
- networkx==3.1
- numpy==1.25.1
- nvidia-cublas-cu11==11.10.3.66
- nvidia-cuda-cupti-cu11==11.7.101
- nvidia-cuda-nvrtc-cu11==11.7.99
- nvidia-cuda-runtime-cu11==11.7.99
- nvidia-cudnn-cu11==8.5.0.96
- nvidia-cufft-cu11==10.9.0.58
- nvidia-curand-cu11==10.2.10.91
- nvidia-cusolver-cu11==11.4.0.1
- nvidia-cusparse-cu11==11.7.4.91
- nvidia-nccl-cu11==2.14.3
- nvidia-nvtx-cu11==11.7.91
- orderedmultidict==1.0.1
- packaging==23.1
- pandas==2.0.3
- pathlib2==2.3.7.post1
- pathy==0.10.2
- Pillow==10.0.1
- preshed==3.0.8
- psutil==5.9.5
- pydantic==1.10.11
- PyJWT==2.4.0
- pyparsing==3.1.1
- python-dateutil==2.8.2
- pytz==2023.3
- PyYAML==6.0.1
- referencing==0.30.2
- regex==2023.6.3
- requests==2.31.0
- rpds-py==0.10.3
- s3transfer==0.6.2
- safetensors==0.3.1
- six==1.16.0
- smart-open==6.3.0
- spacy==3.6.1
- spacy-alignments==0.9.0
- spacy-legacy==3.0.12
- spacy-loggers==1.0.5
- spacy-lookups-data==1.0.5
- spacy-transformers==1.2.5
- srsly==2.4.7
- sympy==1.12
- thinc==8.1.10
- tokenizers==0.13.3
- torch==2.0.1
- tqdm==4.65.0
- transformers==4.30.2
- triton==2.0.0
- typer==0.7.0
- typing-extensions==4.7.1
- tzdata==2023.3
- urllib3==1.25.11
- wasabi==1.1.2
- werkzeug==2.3.7
- zipp==3.17.0

Environment setup completed successfully

Starting Task Execution:

/home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_trainmodel.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources

1695253288179 myhostname:0 DEBUG ClearML results page: https://clearml-dev.company.internal:8080/projects/a443fe2356294ad0b8f8cff384b19888/experiments/6be6710c02774713bd41192794e3bf0c/output/log
2023-09-20 23:41:23,433 - clearml - INFO - Dataset.get() did not specify alias. Dataset information will not be automatically logged in ClearML Server.
train_loc: /home/ec2-user/.clearml/cache/storage_manager/datasets/ds_b720ef6e83254897b042cf5e2bcf5274/train.spacy
config_path: /home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_config_final.cfg
2023-09-20 23:41:23,970 - clearml.Task - INFO - Using global credentials for 's3://myprojectname-data/myprojectname-models'

1695253293731 myhostname:0 DEBUG 2023-09-20 23:41:33,531 - clearml - WARNING - Could not retrieve remote configuration named 'Spacy Config File'
Using default configuration: /home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_config_final.cfg
Cython: 3.0.2
Flask: 2.3.2
Jinja2: 3.1.2
MarkupSafe: 2.1.3
Pillow: 10.0.1
PyJWT: 2.4.0
PyYAML: 6.0.1
attrs: 23.1.0
blinker: 1.6.2
blis: 0.7.9
boto3: 1.28.38
botocore: 1.31.38
catalogue: 2.0.8
certifi: 2023.5.7
charset-normalizer: 3.2.0
clearml: 1.12.2
click: 8.1.5
cmake: 3.26.4
confection: 0.1.0
cupy-cuda111: 12.2.0
cupy-wheel: 12.2.0
cymem: 2.0.7
fastrlock: 0.8.1
filelock: 3.12.2
fsspec: 2023.6.0
furl: 2.1.3
huggingface-hub: 0.16.4
idna: 3.4
importlib-metadata: 6.8.0
itsdangerous: 2.1.2
jmespath: 1.0.1
jsonschema: 4.19.0
jsonschema-specifications: 2023.7.1
langcodes: 3.3.0
lit: 16.0.6
mpmath: 1.3.0
murmurhash: 1.0.9
networkx: 3.1
numpy: 1.25.1
nvidia-cublas-cu11: 11.10.3.66
nvidia-cuda-cupti-cu11: 11.7.101
nvidia-cuda-nvrtc-cu11: 11.7.99
nvidia-cuda-runtime-cu11: 11.7.99
nvidia-cudnn-cu11: 8.5.0.96
nvidia-cufft-cu11: 10.9.0.58
nvidia-curand-cu11: 10.2.10.91
nvidia-cusolver-cu11: 11.4.0.1
nvidia-cusparse-cu11: 11.7.4.91
nvidia-nccl-cu11: 2.14.3
nvidia-nvtx-cu11: 11.7.91
orderedmultidict: 1.0.1
packaging: 23.1
pandas: 2.0.3
pathlib2: 2.3.7.post1
pathy: 0.10.2
pip: 20.1.1
preshed: 3.0.8
psutil: 5.9.5
pydantic: 1.10.11
pyparsing: 3.1.1
python-dateutil: 2.8.2
pytz: 2023.3
referencing: 0.30.2
regex: 2023.6.3
requests: 2.31.0
rpds-py: 0.10.3
s3transfer: 0.6.2
safetensors: 0.3.1
setuptools: 68.2.0
six: 1.16.0
smart-open: 6.3.0
spacy: 3.6.1
spacy-alignments: 0.9.0
spacy-legacy: 3.0.12
spacy-loggers: 1.0.5
spacy-lookups-data: 1.0.5
spacy-transformers: 1.2.5
srsly: 2.4.7
sympy: 1.12
thinc: 8.1.10
tokenizers: 0.13.3
torch: 2.0.1
tqdm: 4.65.0
transformers: 4.30.2
triton: 2.0.0
typer: 0.7.0
typing-extensions: 4.7.1
tzdata: 2023.3
urllib3: 1.25.11
wasabi: 1.1.2
werkzeug: 2.3.7
wheel: 0.41.2
zipp: 3.17.0
spacy_train(): train_loc: /home/ec2-user/.clearml/cache/storage_manager/datasets/ds_b720ef6e83254897b042cf5e2bcf5274/train.spacy
�[38;5;2m✔ Created output directory: t_trained_model�[0m
�[38;5;4mℹ Saving to output directory: t_trained_model�[0m
�[38;5;4mℹ Using GPU: 0�[0m
Traceback (most recent call last):
  File "/home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_trainmodel.py", line 86, in <module>
    main()
  File "/home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_trainmodel.py", line 81, in main
    spacy_train(config_path, trained_model_path, train_loc, dev_loc, 0)
  File "/home/ec2-user/.clearml/venvs-builds/3.9/task_repository/mygitrepo.git/training/spacy_trainmodel.py", line 26, in spacy_train
    train(config_file, output_directory, use_gpu=int(gpu_id), overrides={
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/spacy/cli/train.py", line 76, in train
    setup_gpu(use_gpu)
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/spacy/cli/_util.py", line 599, in setup_gpu
    require_gpu(use_gpu)
  File "/home/ec2-user/.clearml/venvs-builds/3.9/lib/python3.9/site-packages/thinc/util.py", line 200, in require_gpu
    raise ValueError("Cannot use GPU, CuPy is not installed")
ValueError: Cannot use GPU, CuPy is not installed

1695253309029 myhostname:0 DEBUG Process failed, exit code 1

danieldk · 2023-09-22T11:11:34Z

danieldk
Sep 22, 2023

Could you try if

python -c "import cupy; import cupyx"

runs successfully?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot train a model due to "Cannot use GPU, CuPy is not installed", even though CuPy is installed #12996

{{title}}

Replies: 1 comment

{{title}}

Select a reply

Cannot train a model due to "Cannot use GPU, CuPy is not installed", even though CuPy is installed #12996

tpanza Sep 21, 2023

Replies: 1 comment

danieldk Sep 22, 2023

tpanza
Sep 21, 2023

danieldk
Sep 22, 2023