[Closed] GPG key rotation #1

kmittman · 2021-11-07T20:05:16Z

Mega thread for reporting metdata issues with the CUDA repositories

Is the CDN stale? Have you seen something like:

E: Failed to fetch *.deb  Hash Sum mismatch
   Hashes of expected file:
    - SHA512:$sha512, SHA256:$sha256, SHA1:$sha1 [weak], MD5Sum:$md5 [weak], Filesize:$bytes [weak]
   Hashes of received file:
    - SHA512:$sha512, SHA256:$sha256, SHA1:$sha1 [weak], MD5Sum:$md5 [weak], Filesize:$bytes [weak]
   Last modification reported: $(date -R --utc)

or

*.rpm: Downloading successful, but checksum doesn't match. 
Calculated: $sha256(sha256)  
Expected: $sha256(sha256)

Please provide the following information in your comment:

The error message and the last command(s) run.

When was the Release (Debian) or repomd.xml (RPM) file last modified ?

$ curl -I https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/Release
$ curl -I https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/repodata/repomd.xml

The Linux distro and architecture. If cross-compiling or containerized, please mention that.
```
$ cat /etc/os-release
$ uname -a
```
Which NVIDIA repositories do you have enabled ?
Do your .list / .repo files contain URLs using HTTP (port 80) or HTTPS (port 443) ?
Which geographic region is the machine located in ?
Which CDN edge node are you hitting ?
Any other relevant environmental conditions (i.e. a specific Docker container image) ?

The text was updated successfully, but these errors were encountered:

erikwijmans · 2022-04-27T20:10:57Z

Hi, we are getting the following error:

> sudo apt update
Err:1 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC

Details:

See above for command and error
Release was "Last-Modified: Mon, 25 Apr 2022 20:50:31 GMT"

> cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
> uname -a 
Linux droid 4.15.0-66-generic #75~16.04.1-Ubuntu SMP Tue Oct 1 14:01:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

We have the https://developer.download.nvidia.com/compute/cuda/repos/ enabled. Using http (also tried https, same error)
The machine is located in Georgia, USA
I am not sure how to figure this out. Happy to run a command if you give me one tho.
None.

kmittman · 2022-04-27T21:14:20Z

Hi @erikwijmans
We are in the process of rotating our GPG public keys, there will be an announcement on the NVIDIA Developer blog very soon. Currently this has rolled out only for our ubuntu1604/x86_64 and fedora32/x86_64 repos, the rest will follow in a bit.

The new GPG keys for the CUDA repository

Debian-based distros: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub
RPM-based distros: https://developer.download.nvidia.com/compute/cuda/repos/fedora32/x86_64/D42D0685.pub

Please remove the old 7fa2af80 key

Debian-based distros:
sudo apt-key del 7fa2af80
RPM-based distros:
sudo rpm --erase gpg-pubkey-7fa2af80*

And enroll the new signing key (a cuda-keyring package is also provided)

Debian-based distros:

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

RPM-based distros: please update .repo with package manager specific instructions (see blog post)

erikwijmans · 2022-04-27T22:27:34Z

Thank you for the fast reply! That fixes it.

topinfrassi01 · 2022-04-28T18:37:18Z

Hi @kmittman,

I'm facing the same issue using the docker image tensorflow/tensorflow:2.7.0-gpu from Docker Hub. I pulled the image again from the hub but it doesn't seem to work.

From what I see, in the docker image creation, it seems like it tries to get the older signing key (https://hub.docker.com/layers/tensorflow/tensorflow/tensorflow/2.7.0-gpu/images/sha256-fc5eb0604722c7bef7b499bb007b3050c4beec5859c2e0d4409d2cca5c14d442?context=explore)

I was just wondering if I was on the right track and if you knew what could be done about this/where should I post to make the tensorflow team aware of this problem.

Thank you

rgov · 2022-04-28T19:48:10Z

I have updated my Ubuntu 20.04 to use the new key, but am hitting this issue which has happened over the years. This host is in the northeast of the US.

I have tried running sudo apt clean but it doesn't resolve anything.

$ apt update
...
Err:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
  File has unexpected size (631054 != 481481). Mirror sync in progress? [IP: 152.195.19.142 443]
  Hashes of expected file:
   - Filesize:481481 [weak]
   - SHA256:8556d67c6d380c957f05057f448d994584a135d7ed75e5ae6bb25c3fc1070b0b
   - SHA1:c5ea9556407a3b5daec4aac530cd038e9b490441 [weak]
   - MD5Sum:a5513131dbd2d4e50f185422ebb43ac9 [weak]
  Release file created at: Mon, 25 Apr 2022 23:27:19 +0000
...

$ cat /etc/apt/sources.list.d/cuda-ubuntu2004-x86_64.list 
deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /

$ curl -I https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/Release
HTTP/2 200 
accept-ranges: bytes
age: 3661
cache-control: max-age=604800
content-type: application/octet-stream
date: Thu, 28 Apr 2022 19:50:19 GMT
etag: "1944279454"
expires: Thu, 05 May 2022 19:50:19 GMT
last-modified: Tue, 12 Apr 2022 23:01:21 GMT
server: ECAcc (lab/772A)
x-cache: HIT
x-vdms-version: 2.9.4
content-length: 696

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
Linux behemoth 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Edit: Looks like it is resolved 30 minutes later.

kkoehncke · 2022-04-28T20:28:10Z

When trying to update the signing key via the cuda-keyring, getting the follow error:

(base) root@0820a2a1a5d8:/# wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
--2022-04-28 20:29:45--  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 152.195.19.142
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.195.19.142|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4328 (4.2K) [application/x-deb]
Saving to: 'cuda-keyring_1.0-1_all.deb'

cuda-keyring_1.0-1_all.deb               100%[===============================================================================>]   4.23K  --.-KB/s    in 0s

2022-04-28 20:29:45 (99.2 MB/s) - 'cuda-keyring_1.0-1_all.deb' saved [4328/4328]

(base) root@0820a2a1a5d8:/# sudo dpkg -i cuda-keyring_1.0-1_all.deb
bash: sudo: command not found
(base) root@0820a2a1a5d8:/# dpkg -i cuda-keyring_1.0-1_all.deb
Selecting previously unselected package cuda-keyring.
(Reading database ... 24690 files and directories currently installed.)
Preparing to unpack cuda-keyring_1.0-1_all.deb ...
Unpacking cuda-keyring (1.0-1) ...
Setting up cuda-keyring (1.0-1) ...
(base) root@0820a2a1a5d8:/# apt update
E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /: /usr/share/keyrings/cuda-archive-keyring.gpg !=
E: The list of sources could not be read.

When was the Release (Debian) or repomd.xml (RPM) file last modified ?

(base) root@75d6db735dfe:/# curl -I https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Release
HTTP/2 200
accept-ranges: bytes
age: 140
cache-control: max-age=604800
content-type: application/octet-stream
date: Thu, 28 Apr 2022 20:22:02 GMT
etag: "3751684597"
expires: Thu, 05 May 2022 20:22:02 GMT
last-modified: Mon, 25 Apr 2022 23:27:19 GMT
server: ECAcc (lab/772A)
x-cache: HIT
x-vdms-version: 2.9.4
content-length: 690

(base) root@75d6db735dfe:/# curl -I https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/repodata/repomd.xml
HTTP/2 404
cache-control: max-age=604800
content-type: text/html
date: Thu, 28 Apr 2022 20:24:40 GMT
expires: Thu, 05 May 2022 20:24:40 GMT
server: EOS (vny/1B6D)
x-vdms-version: 2.9.4
content-length: 445

The Linux distro and architecture. If cross-compiling or containerized, please mention that.

(base) root@75d6db735dfe:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

(base) root@75d6db735dfe:/# uname -a
Linux 75d6db735dfe 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Which NVIDIA repositories do you have enabled ?

Not sure, definitely CUDA and compute/machine- learning

Do your .list / .repo files contain URLs using HTTP (port 80) or HTTPS (port 443) ?

Both.

Which geographic region is the machine located in ?

Seattle, Washington

Which CDN edge node are you hitting ?

Not sure.

Any other relevant environmental conditions (i.e. a specific Docker container image) ?

Using the nvidia/cuda:11.3.0-base-ubuntu20.04 official image provided by NVIDIA

rgov · 2022-04-28T20:43:27Z

@kkoehncke I think you may have both cuda.list (older) and cuda-ubuntu2004-x86_64.list (newer) in /etc/apt/sources.list.d. Just delete the older one.

@kmittman I would suggest using individual issues instead of using a megathread, since GitHub issues are not threaded and it will be very confusing dealing with several unrelated problems. And everyone who posts a question here will be subscribed to future notifications to unrelated problems which is a little annoying.

kmittman · 2022-04-28T20:58:41Z

Hi @topinfrassi01
I believe that container image is maintained by Google ? Now that our repositories are updated, the next step is to roll out updated CUDA base images, and notify downstream maintainers to re-base.

kmittman · 2022-04-28T20:59:40Z

Hi @kkoehncke please grep for developer.download.nvidia.com in your /etc/apt/sources.list if you had previously use add-apt-repository it gets appended there, rather than a separate .list file in /etc/apt/sources.list.d/.
You'll need to search both locations and remove any existing entries.

kmittman · 2022-04-28T21:00:09Z

Hi @rgov
Yes, we've been hitting CDN issues. Unfortunately it took a very long time (and a few retries for posting and purge) to propagate to the edge nodes.

RE: mega thread, the problem is, unless I assign the issue to myself, I cannot figure out how to configure GitHub to send me email notifications, as technically this repo is owned by the NVIDIA user/group. Is there some way to create an issue template?

rgov · 2022-04-28T21:06:59Z

@kmittman Yep there are issue templates and you can use the "Watch" button (next to "Fork" in the top right) to get notified of all new issues.

pablete · 2022-04-28T23:29:57Z

Lots of nvidia documentation / repositories need to be changed now....

For example this guide on how to use the latest TensorRT
https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html

has the the instruction to add the 7fa2af80.pub explicitly
sudo apt-key add /var/nv-tensorrt-repo-${os}-${tag}/7fa2af80.pub

ericmclachlan · 2022-04-30T18:03:52Z

Hi @erikwijmans We are in the process of rotating our GPG public keys, there will be an announcement on the NVIDIA Developer blog very soon. Currently this has rolled out only for our ubuntu1604/x86_64 and fedora32/x86_64 repos, the rest will follow in a bit.

The new GPG keys for the CUDA repository

Debian-based distros: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

RPM-based distros: https://developer.download.nvidia.com/compute/cuda/repos/fedora32/x86_64/D42D0685.pub

Please remove the old 7fa2af80 key

Debian-based distros:
sudo apt-key del 7fa2af80

RPM-based distros:
sudo rpm --erase gpg-pubkey-7fa2af80*

And enroll the new signing key (a cuda-keyring package is also provided)

Debian-based distros:

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

RPM-based distros: please update .repo with package manager specific instructions (see blog post)

Is there a best practice for applying this change to Nvidia docker images?

For instance, I'm using 11.4.2-cudnn8-runtime-ubuntu20.04 and I have confirmed that I'm now able to run apt update after running the following commands on the container:

apt-key del 7fa2af80
apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

It seems then that running the above commands from the Dockerfile that deploys the image may be a good way of avoiding the NO_PUBKEY GPG error.

RUN apt-key del 7fa2af80
RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/3bf863cc.pub

TL;DR: I'm not sure how often GPG keys are rotated and I was wondering whether the above solution is a good workaround or actually a long-term solution to the problem? And if not, is there a good way to apply this fix to published docker images?

mrgzg1 · 2022-07-15T23:47:56Z

Seeing this happen on the west coast

HTTP/2 404
cache-control: max-age=604800
content-type: text/html
date: Fri, 15 Jul 2022 23:44:04 GMT
expires: Fri, 22 Jul 2022 23:44:04 GMT
server: EOS (vny/0451)
content-length: 445

~~~$ curl -I https://developer.download.nvidia.com/compute/cuda/repos/ubuntu_1804/x86_64/repodata/repomd.xml
HTTP/2 404
cache-control: max-age=604800
content-type: text/html
date: Fri, 15 Jul 2022 23:44:18 GMT
expires: Fri, 22 Jul 2022 23:44:18 GMT
server: EOS (vny/0452)
content-length: 445

~~~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
mgandhi@capacity-ewr-00-station-00-04:/var/lib/dpkg/info$ uname -a
Linux capacity-ewr-00-station-00-04 4.15.0-189-generic #200-Ubuntu SMP Wed Jun 22 19:53:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux|

HTTPS:
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/
US-West
Not sure?

kmittman · 2023-01-13T18:41:35Z

Closing this again. If experience further errors, please file a new issue, thank you.

kmittman self-assigned this Nov 7, 2021

kmittman mentioned this issue Nov 19, 2021

RHEL 8 x86_64 repository wrong package file size #2

Closed

kmittman pinned this issue Jan 19, 2022

samquinan mentioned this issue Apr 28, 2022

Apt-get dependencies in colab broken by NVIDIA signing key updates googlecolab/colabtools#2779

Closed

francesco-giordano mentioned this issue May 2, 2022

Fix gpg key nvidia fabricmanager in amazon linux 2 and centos 7 aws/aws-parallelcluster-cookbook#1426

Merged

4 tasks

0x2b3bfa0 mentioned this issue May 4, 2022

Container test & build workflow broken, again beacuse of NVIDIA iterative/cml#993

Closed

kmittman changed the title ~~Report metadata issues here~~ [Closed] GPG key rotation Jul 18, 2022

kmittman closed this as completed Jan 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Closed] GPG key rotation #1

[Closed] GPG key rotation #1

kmittman commented Nov 7, 2021

erikwijmans commented Apr 27, 2022

kmittman commented Apr 27, 2022

erikwijmans commented Apr 27, 2022

topinfrassi01 commented Apr 28, 2022

rgov commented Apr 28, 2022 •

edited

Loading

kkoehncke commented Apr 28, 2022 •

edited

Loading

rgov commented Apr 28, 2022

kmittman commented Apr 28, 2022

kmittman commented Apr 28, 2022

kmittman commented Apr 28, 2022 •

edited

Loading

rgov commented Apr 28, 2022

pablete commented Apr 28, 2022

ericmclachlan commented Apr 30, 2022 •

edited

Loading

mrgzg1 commented Jul 15, 2022

kmittman commented Jan 13, 2023

[Closed] GPG key rotation #1

[Closed] GPG key rotation #1

Comments

kmittman commented Nov 7, 2021

Mega thread for reporting metdata issues with the CUDA repositories

Please provide the following information in your comment:

erikwijmans commented Apr 27, 2022

kmittman commented Apr 27, 2022

erikwijmans commented Apr 27, 2022

topinfrassi01 commented Apr 28, 2022

rgov commented Apr 28, 2022 • edited Loading

kkoehncke commented Apr 28, 2022 • edited Loading

rgov commented Apr 28, 2022

kmittman commented Apr 28, 2022

kmittman commented Apr 28, 2022

kmittman commented Apr 28, 2022 • edited Loading

rgov commented Apr 28, 2022

pablete commented Apr 28, 2022

ericmclachlan commented Apr 30, 2022 • edited Loading

mrgzg1 commented Jul 15, 2022

kmittman commented Jan 13, 2023

rgov commented Apr 28, 2022 •

edited

Loading

kkoehncke commented Apr 28, 2022 •

edited

Loading

kmittman commented Apr 28, 2022 •

edited

Loading

ericmclachlan commented Apr 30, 2022 •

edited

Loading