GPG signing key rotation for CUDA repositories #4

kmittman · 2022-04-28T00:46:28Z

Blog post

Forum post

To best ensure the security and reliability of our RPM and Debian package repositories, NVIDIA is updating and rotating the GPG signing keys used by apt, dnf/yum, and zypper package managers. Failure to update your repository signing keys will result in package management errors when attempting to access or install packages from the CUDA repositories.

The new GPG public keys for the CUDA repository

Debian-based distros: 3bf863cc
RPM-based distros: d42d0685

To ensure continued access to the latest NVIDIA software, please complete the following steps:

For Debian-based distributions, including Ubuntu, replace $distro/$arch in the following commands with appropriate value

amd64	POWER	arm64 server	arm64 cross target
debian10/x86_64
debian11/x86_64
ubuntu1604/x86_64
ubuntu1804/x86_64	ubuntu1804/ppc64el	ubuntu1804/sbsa	ubuntu1804/cross-linux-sbsa
ubuntu2004/x86_64		ubuntu2004/sbsa	ubuntu2004/cross-linux-sbsa
ubuntu2204/x86_64		ubuntu2204/sbsa
wsl-ubuntu/x86_64

For RPM-based distributions, including SUSE, replace $distro/$arch in the following commands with appropriate value

amd64	POWER	arm64 server	arm64 cross target
fedora32/x86_64
fedora33/x86_64
fedora34/x86_64
fedora35/x86_64
opensuse15/x86_64
rhel7/x86_64	rhel7/ppc64le
rhel8/x86_64	rhel8/ppc64le	rhel8/sbsa	rhel8/cross-linux-sbsa
sles15/x86_64		sles15/sbsa	sles15/cross-linux-sbsa

Remove Outdated Signing Key

Debian, Ubuntu, WSL
```
$ sudo apt-key del 7fa2af80
```

Fedora, RHEL, openSUSE, SLES

 $ sudo rpm --erase gpg-pubkey-7fa2af80*

Install New Key (Debian-based distros)

Method 1: `cuda-keyring` package

To avoid the need for manual key installation steps, NVIDIA is providing a new helper package that will automate the installation of new signing keys for the NVIDIA repositories.

$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb

Method 2: `apt-key` (deprecated)

$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/3bf863cc.pub

Method 3: `wget` and `mv`

$ wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro-keyring.gpg
$ sudo mv cuda-$distro-keyring.gpg /usr/share/keyrings/cuda-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/ /" | sudo tee /etc/apt/sources.list.d/cuda-$distro-$arch.list

Install New Key (RPM-based distros)

On fresh installation Fedora, RHEL, openSUSE, or SLES the dnf/yum/zypper package manager will prompt the user to accept new keys when installing packages the first time. Indicate you accept the change when prompted: y

For upgrades existing installations, you must additionally complete the following (package manager specific) step to pick up the new key:

Fedora and RHEL 8

$ sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

RHEL 7

$ sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/$arch/cuda-rhel7.repo

openSUSE and SLES

$ sudo zypper removerepo cuda-$distro-$arch
$ sudo zypper addrepo https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro.repo

Example error messages

apt-get

Reading package lists... Done
W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

dnf and yum

warning: /var/cache/dnf/cuda-fedora32-x86_64-d60aafcddb176bf5/packages/libnvjpeg-11-1-11.3.0.105-1.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d42d0685: NOKEY
cuda-fedora32-x86_64                                                                                  23 kB/s | 1.6 kB     00:00    
Importing GPG key 0x7FA2AF80:
 Userid     : "cudatools <[email protected]>"
 Fingerprint: AE09 FE4B BD22 3A84 B2CC FCE3 F60F 4B3D 7FA2 AF80
 From       : https://developer.download.nvidia.com/compute/cuda/repos/fedora32/x86_64/7fa2af80.pub
Is this ok [y/N]: y
Key imported successfully
Import of key(s) didn't help, wrong key(s)?
Public key for libnvjpeg-11-1-11.3.0.105-1.x86_64.rpm is not installed. Failing package is: libnvjpeg-11-1-11.3.0.105-1.x86_64
 GPG Keys are configured as: https://developer.download.nvidia.com/compute/cuda/repos/fedora32/x86_64/7fa2af80.pub
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

zypper

New repository or package signing key received:

  Repository:       cuda-opensuse15-x86_64
  Key Fingerprint:  610C 7B14 E068 A878 070D A4E9 9CD0 A493 D42D 0685
  Key Name:         cudatools <[email protected]>
  Key Algorithm:    RSA 4096
  Key Created:      Thu Apr 14 22:04:01 2022
  Key Expires:      (does not expire)
  Rpm Name:         gpg-pubkey-d42d0685-62589a51



    Note: Signing data enables the recipient to verify that no modifications occurred after the data
    were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
    and in extreme cases even to a system compromise.

    Note: A GPG pubkey is clearly identified by it's fingerprint. Do not rely the keys name. If you
    are not sure whether the presented key is authentic, ask the repository provider or check his
    web site. Many provider maintain a web page showing the fingerprints of the GPG keys they are
    using.

Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): a

The text was updated successfully, but these errors were encountered:

kmittman · 2022-04-29T05:04:56Z

Adding some additional information

Common errors on Debian-based distros

Duplicate `.list` entries

E: Conflicting values set for option Signed-By regarding source 
   https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /: 
   /usr/share/keyrings/cuda-archive-keyring.gpg != 
E: The list of sources could not be read.

Solution: If previously used add-apt-repository to enable the CUDA repository, then remove the duplicate entry

sudo sed -i '/developer\.download\.nvidia\.com\/compute\/cuda\/repos/d' /etc/apt/sources.list

Also check for and remove cuda*.list files under /etc/apt/sources.list.d/ directory.

New GPG key is not enrolled

Reading package lists...
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease: The following signatures couldn't be verified because the public key is not available: 
NO_PUBKEY A4B46996 3BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64
InRelease' is no longer signed.

Solution:
See above to install cuda-keyring package OR one of the manual enrollment methods for the 3bf863cc public key.

Machine Learning repository

W: An error occurred during the signature verification.
The repository is not updated and the previous index files will be used. 
GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64
Release: The following signatures couldn't be verified because the public key is not available: 
NO_PUBKEY F60F4B3D 7FA2AF80

Solution:
Remove the NVIDIA machine learning repository entry, it is no longer updated. Newer versions of cuDNN, NCCL, and TensorRT are available in the CUDA repository.

File has unexpected size

Packages.gz File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: XXX.XXX.XXX.XXX 443]
Hashes of expected file:
- Filesize:481481 [weak]
- SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
- SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
- MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Solution: Report CDN issue to NVIDIA

…py cuda dlls - CUDA gpg-key change cf. NVIDIA/cuda-repo-management#4

* - nproc * - YANEURAOU_ENGINE_MATERIAL* ビルドテストの一部省略 * - release build material lv1,lv9 * - pacboy -Syuu * - remove base-devel: from actions * - remove base-devel: * - pacman update * - POPCNT, BSF, BSR の実装パターン追加 * - MacOS * - NuGet packages bump up * - CUDA 11.6 Update 2, TensorRT 8.2 GA Update 3, cuda-nvrtc, remove copy cuda dlls - CUDA gpg-key change cf. NVIDIA/cuda-repo-management#4 * - clang-15 * - DNN_Batch_Size max 1024 * - fp16 * - .gitignore * - badge MacOS, WebAssembly * - WASM port cf. https://github.com/arashigaoka/YaneuraOu.wasm cf. https://github.com/niklasf/stockfish.wasm cf. https://github.com/hi-ogawa/Stockfish * - wasm_build * - github workflows: wasm

andresgalaviz · 2022-05-02T23:11:30Z

Method 3 does not work.

There are no entries that match this format:
https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-$distro-keyring.gpg

For example, this doesn't exist:
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804-keyring.gpg

kmittman · 2022-05-03T10:18:23Z

Hi @andresgalaviz
I have uploaded the missing files. Please give method 3 a try again.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804-keyring.gpg
sudo mv cuda-ubuntu1804-keyring.gpg /usr/share/keyrings/cuda-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" | sudo tee /etc/apt/sources.list.d/cuda-ubuntu1804-x86_64.list

sudo apt-get update

(Later on I will add those .list files on the website so then can just be wget and mv to the right location)

mocha-workme · 2022-05-03T12:00:21Z

CUDA: 11.2.1
When is Centos8?

JonnoB · 2022-05-04T08:50:34Z

For people who are not very skilled in Linux (like me). @kmittman 's comment on common problems is very helpful but needs some slight expansion.
On the "Conflifcting values..." problem removing the cuda list can be done using the following commands
sudo rm -f /etc/apt/sources.list.d/cuda*.list
This forces the removal of the cuda lists of which there may be more than one.

k-jeon · 2022-05-13T01:08:51Z

After I followed the instruction, I got the following warnings:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/cudnn-local-repo-ubuntu2004-8.3.2.44  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: file:/var/nccl-local-repo-ubuntu2004-2.11.4-cuda11.5  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Failed to fetch file:/var/cudnn-local-repo-ubuntu2004-8.3.2.44/Release.gpg  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Failed to fetch file:/var/nccl-local-repo-ubuntu2004-2.11.4-cuda11.5/Release.gpg  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: Some index files failed to download. They have been ignored, or old ones used instead.

kmittman · 2022-05-13T01:17:21Z

Hi @k-jeon
The cuDNN local repository installed on your system: /var/cudnn-local-repo-ubuntu2004-8.3.2.44 contains packages and repo metadata signed with the deprecated 7fa2af80 pubkey. Likewise for the NCCL local repository installed on your system: /var/nccl-local-repo/ubuntu2004-2.11.4-cuda11.5 is signed with the deprecated 7fa2af80 pubkey.

I would suggest removing old NVIDIA local repositories from your system. For example,

sudo apt-get remove --purge "cudnn-local-repo*"
NOTE: this does not remove the packages such as libcudnn8 and libcudnn8-dev from the system.
sudo apt-get remove --purge "nccl-local-repo*"
NOTE: this does not remove the packages such as libnccl2 and libnccl-dev from the system.

New local repository installers will no longer use this key. See the CUDA Linux Installation Guide for more details for instructions on enrolling ephemeral keys.

We have no plans to re-release old local repository installers, such as the two enabled on your system. These packages are available, re-signed with a new key in the CUDA network repository.

k-jeon · 2022-05-13T01:21:21Z

This works for me! Thanks a lot for the prompt answer!

would suggest removing old NVIDIA local repositories from your system. For example,

sudo apt-get remove --purge "cudnn-local-repo*"
NOTE: this does not remove the packages such as libcudnn8 and libcudnn8-dev from the system.

sudo apt-get remove --purge "nccl-local-repo*"
NOTE: this does not remove the packages such as libnccl2 and libnccl-dev from the system.

mocha-workme · 2022-05-13T01:26:31Z

@k-jeon

Go here and download the one that matches your OS
https://developer.download.nvidia.com/compute/cuda/repos

For example) https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub

Download and replace the nvidia key in the path below with the one you downloaded
for example)
/etc/pki/rpm-gpg/RPM-GPG-KEY-NVIDIA
And update
For example) rpm --import /etc/pki/rpm-gpg/RPM*

I use nvidia docker, I fixed it temporarily like this

kylevedder · 2022-06-03T20:14:25Z

For this comment, note that the cuda.list file may be inside /etc/apt/sources.list.d/, and not just /etc/apt/sources.d.

kmittman · 2022-06-03T20:36:46Z

Thank you @kylevedder, I have updated the earlier comment. Sorry I was just going from memory.

Angel-Popa · 2022-07-12T03:02:36Z

Hi, I am having problems updating the key, because the file has an unexpected size:

E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Packages.gz File has unexpected size (580845 != 580040). Mirror sync in progress? [IP: XXX.XXX.XX.XXX 443]
Hashes of expected file:
- Filesize:580040 [weak]
- SHA256:dc8abbaf470d3ee626f4f1f4d2871a98d8dc8f770bc592676f0d1f60637e0c2d
- SHA1:109bfa4e5c415731aa44b1c1caae3f19754c5406 [weak]
- MD5Sum:bf8f928cb55b1c9e4c158a69ee52b5d8 [weak]
- Release file created at: Mon, 11 Jul 2022 19:01:54 +0000

kmittman · 2022-07-12T05:07:59Z

Hi everyone, I'm closing this. Please file new issues using the template. For today's incident please see #7

kmittman self-assigned this Apr 28, 2022

kmittman pinned this issue Apr 28, 2022

kmittman mentioned this issue Apr 28, 2022

Receiving error about Packages.gz being the incorrect size #5

Closed

borisfom mentioned this issue Apr 29, 2022

Updating gpg keys; cmake installation streamlined triton-inference-server/server#4307

Closed

mizar added a commit to mizar/YaneuraOu.wasm that referenced this issue Apr 30, 2022

- CUDA 11.6 Update 2, TensorRT 8.2 GA Update 3, cuda-nvrtc, remove co…

4adee75

…py cuda dlls - CUDA gpg-key change cf. NVIDIA/cuda-repo-management#4

mizar added a commit to mizar/YaneuraOu.wasm that referenced this issue Apr 30, 2022

- CUDA 11.6 Update 2, TensorRT 8.2 GA Update 3, cuda-nvrtc, remove co…

386265f

…py cuda dlls - CUDA gpg-key change cf. NVIDIA/cuda-repo-management#4

mizar added a commit to mizar/YaneuraOu.wasm that referenced this issue Apr 30, 2022

- CUDA 11.6 Update 2, TensorRT 8.2 GA Update 3, cuda-nvrtc, remove co…

99311c7

…py cuda dlls - CUDA gpg-key change cf. NVIDIA/cuda-repo-management#4

kmittman closed this as completed Jul 12, 2022

smanders mentioned this issue Sep 17, 2022

centos7-bld: cuda public key not installed externpro/buildpro#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPG signing key rotation for CUDA repositories #4

GPG signing key rotation for CUDA repositories #4

kmittman commented Apr 28, 2022 •

edited

Loading

kmittman commented Apr 29, 2022 •

edited

Loading

andresgalaviz commented May 2, 2022 •

edited

Loading

kmittman commented May 3, 2022

mocha-workme commented May 3, 2022

JonnoB commented May 4, 2022

k-jeon commented May 13, 2022

kmittman commented May 13, 2022

k-jeon commented May 13, 2022

mocha-workme commented May 13, 2022 •

edited

Loading

kylevedder commented Jun 3, 2022

kmittman commented Jun 3, 2022

Angel-Popa commented Jul 12, 2022

kmittman commented Jul 12, 2022

GPG signing key rotation for CUDA repositories #4

GPG signing key rotation for CUDA repositories #4

Comments

kmittman commented Apr 28, 2022 • edited Loading

Blog post

Forum post

The new GPG public keys for the CUDA repository

Remove Outdated Signing Key

Install New Key (Debian-based distros)

Method 1: cuda-keyring package

Method 2: apt-key (deprecated)

Method 3: wget and mv

Install New Key (RPM-based distros)

Fedora and RHEL 8

RHEL 7

openSUSE and SLES

Example error messages

kmittman commented Apr 29, 2022 • edited Loading

Common errors on Debian-based distros

Duplicate .list entries

New GPG key is not enrolled

Machine Learning repository

File has unexpected size

andresgalaviz commented May 2, 2022 • edited Loading

kmittman commented May 3, 2022

mocha-workme commented May 3, 2022

JonnoB commented May 4, 2022

k-jeon commented May 13, 2022

kmittman commented May 13, 2022

k-jeon commented May 13, 2022

mocha-workme commented May 13, 2022 • edited Loading

kylevedder commented Jun 3, 2022

kmittman commented Jun 3, 2022

Angel-Popa commented Jul 12, 2022

kmittman commented Jul 12, 2022

kmittman commented Apr 28, 2022 •

edited

Loading

Method 1: `cuda-keyring` package

Method 2: `apt-key` (deprecated)

Method 3: `wget` and `mv`

kmittman commented Apr 29, 2022 •

edited

Loading

Duplicate `.list` entries

andresgalaviz commented May 2, 2022 •

edited

Loading

mocha-workme commented May 13, 2022 •

edited

Loading