Dockerfiles/oc-dev: retry LLVM install on transient apt.llvm.org network failures - #617
Conversation
…ork failures The Linux CLANGPDB CI job intermittently fails during the oc-dev-edk2 image build: llvm.sh's own key-fetch helper (wget to apt.llvm.org/llvm-snapshot.gpg.key) returns exit code 4 (network failure) despite its built-in 3 retries, which aborts the whole Docker build even though the underlying toolchain setup is otherwise fine. Wrap the llvm.sh download+execution in an outer retry loop (5 attempts, 15s backoff) so a short-lived apt.llvm.org outage or rate limit no longer fails CI outright. A persistent failure still fails the build after all attempts are exhausted.
|
Again used Claude. |
|
Is there any proof it actually fails? |
|
Run docker compose run build-duet #2 [internal] load build definition from Dockerfile #3 [internal] load metadata for docker.io/library/ubuntu:24.04 #4 [auth] library/ubuntu:pull token for registry-1.docker.io #3 [internal] load metadata for docker.io/library/ubuntu:24.04 #5 [internal] load .dockerignore #6 [oc-dev-edk2 1/2] FROM docker.io/library/ubuntu:24.04@sha256:33ceb71981b602c1a7443a53469e4dba065f7503eab3078a2d7a57a2ab987517 #7 [oc-dev-edk2 2/2] RUN apt-get update && apt-get install -y lsb-release wget software-properties-common gnupg build-essential nasm uuid-dev libssl-dev libx11-dev libxext-dev iasl curl git zip file && { { [ "$(dpkg --print-architecture)" != "i386" ] && [ "$(dpkg --print-architecture)" != "amd64" ] ; } || apt-get install -y gcc-multilib ; } && { [ "$(dpkg --print-architecture)" == "amd64" ] || { apt-get install -y gcc-x86-64-linux-gnu && export GCC_BIN=x86_64-linux-gnu- ; } ; } && echo "export GCC_BIN=$GCC_BIN" > #7 10.59 Created symlink /etc/systemd/system/multi-user.target.wants/remote-fs.target → /usr/lib/systemd/system/remote-fs.target. #7 10.60 Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service → /usr/lib/systemd/system/systemd-pstore.service. #7 10.60 Initializing machine ID from random generator. #7 10.61 /usr/lib/tmpfiles.d/systemd-network.conf:10: Failed to resolve user 'systemd-network': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd-network.conf:11: Failed to resolve user 'systemd-network': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd-network.conf:12: Failed to resolve user 'systemd-network': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd-network.conf:13: Failed to resolve user 'systemd-network': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd.conf:22: Failed to resolve group 'systemd-journal': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd.conf:23: Failed to resolve group 'systemd-journal': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd.conf:28: Failed to resolve group 'systemd-journal': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd.conf:29: Failed to resolve group 'systemd-journal': No such process #7 10.61 /usr/lib/tmpfiles.d/systemd.conf:30: Failed to resolve group 'systemd-journal': No such process #7 10.62 Creating group 'systemd-journal' with GID 999. #7 10.62 Creating group 'systemd-network' with GID 998. #7 10.62 Creating user 'systemd-network' (systemd Network Management) with UID 998 and GID 998. #7 10.65 Selecting previously unselected package systemd-sysv. #7 17.58 Creating user 'systemd-timesync' (systemd Time Synchronization) with UID 997 and GID 997. #7 17.68 Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /usr/lib/systemd/system/systemd-timesyncd.service. #7 17.68 Created symlink /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service → /usr/lib/systemd/system/systemd-timesyncd.service. #7 17.68 Setting up libatomic1:amd64 (14.2.0-4ubuntu2 #7 20.13 Creating user 'systemd-resolve' (systemd Resolver) with UID 996 and GID 996. #7 20.25 Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → /usr/lib/systemd/system/systemd-resolved.service. #7 20.25 Created symlink /etc/systemd/system/sysinit.target.wants/systemd-resolved.service → /usr/lib/systemd/system/systemd-resolved.service. #7 20.25 Setting up binutils-x86-64-linux-gnu (2.42-4ubuntu2.10) ... #7 20.45 Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-ssh.socket → /usr/lib/systemd/user/gpg-agent-ssh.socket. #7 20.54 Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-extra.socket → /usr/lib/systemd/user/gpg-agent-extra.socket. #7 20.63 Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-browser.socket → /usr/lib/systemd/user/gpg-agent-browser.socket. #7 20.63 Setting up libxcb1-dev:amd64 (1.15-1ubuntu2) ... #7 21.60 Setting up dbus-user-session (1.14.10-4ubuntu4.1) ... #7 21.73 Setting up git (1:2.43.0-1ubuntu7.3) ... #7 23.70 Setting up python3-cryptography (41.0.7-4ubuntu0.4) ... #7 24.07 Setting up python3-oauthlib (3.2.2-1) ... #7 24.51 Setting up python3-software-properties (0.99.49.4) ... #7 24.67 Creating user 'polkitd' (User for polkitd) with UID 995 and GID 995. #7 24.69 invoke-rc.d: could not determine current runlevel #7 24.88 Setting up packagekit-tools (1.2.8-2ubuntu1.5) ...
#7 24.97 Updating certificates in /etc/ssl/certs... #7 25.25 0 added, 0 removed; done. #7 25.25 Running hooks in /etc/ca-certificates/update.d... #7 25.25 done. #7 25.26 Processing triggers for dbus (1.14.10-4ubuntu4.1) ... #7 25.29 Reading package lists... #7 26.03 Building dependency tree... #7 26.18 Reading state information... #7 26.34 The following additional packages will be installed: #7 26.34 gcc-13-multilib lib32asan8 lib32atomic1 lib32gcc-13-dev lib32gcc-s1 #7 26.34 lib32gomp1 lib32itm1 lib32quadmath0 lib32stdc++6 lib32ubsan1 libc6-dev-i386 #7 26.34 libc6-dev-x32 libc6-i386 libc6-x32 libx32asan8 libx32atomic1 #7 26.34 libx32gcc-13-dev libx32gcc-s1 libx32gomp1 libx32itm1 libx32quadmath0 #7 26.34 libx32stdc++6 libx32ubsan1 #7 26.38 The following NEW packages will be installed: #7 26.38 gcc-13-multilib gcc-multilib lib32asan8 lib32atomic1 lib32gcc-13-dev #7 26.38 lib32gcc-s1 lib32gomp1 lib32itm1 lib32quadmath0 lib32stdc++6 lib32ubsan1 #7 26.38 libc6-dev-i386 libc6-dev-x32 libc6-i386 libc6-x32 libx32asan8 libx32atomic1 #7 26.38 libx32gcc-13-dev libx32gcc-s1 libx32gomp1 libx32itm1 libx32quadmath0 #7 26.39 libx32stdc++6 libx32ubsan1 #7 26.73 0 upgraded, 24 newly installed, 0 to remove and 0 not upgraded. #7 26.73 Need to get 24.0 MB of archives. #7 26.73 After this operation, 90.6 MB of additional disk space will be used. #7 26.73 Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-i386 amd64 2.39-0ubuntu8.8 [2787 kB] #7 26.92 Get:2 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-dev-i386 amd64 2.39-0ubuntu8.8 [1446 kB] #7 26.95 Get:3 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-x32 amd64 2.39-0ubuntu8.8 [2919 kB] #7 26.99 Get:4 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libc6-dev-x32 amd64 2.39-0ubuntu8.8 [1637 kB] #7 27.01 Get:5 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32gcc-s1 amd64 14.2.0-4ubuntu2 #7 27.04 Get:7 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32gomp1 amd64 14.2.0-4ubuntu2 #7 27.06 Get:9 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32itm1 amd64 14.2.0-4ubuntu2 #7 27.09 Get:11 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32atomic1 amd64 14.2.0-4ubuntu2 #7 27.12 Get:13 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32asan8 amd64 14.2.0-4ubuntu2 #7 27.18 Get:15 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32stdc++6 amd64 14.2.0-4ubuntu2 #7 27.22 Get:17 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libx32stdc++6 amd64 14.2.0-4ubuntu2 #7 27.26 Get:19 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32quadmath0 amd64 14.2.0-4ubuntu2 #7 27.28 Get:21 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 lib32gcc-13-dev amd64 13.3.0-6ubuntu2 #7 27.33 Get:23 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 gcc-13-multilib amd64 13.3.0-6ubuntu2 #7 27.76 Unpacking lib32gcc-s1 (14.2.0-4ubuntu2 #7 27.78 Unpacking libx32gcc-s1 (14.2.0-4ubuntu2 #7 27.80 Unpacking lib32gomp1 (14.2.0-4ubuntu2 #7 27.81 Unpacking libx32gomp1 (14.2.0-4ubuntu2 #7 27.83 Unpacking lib32itm1 (14.2.0-4ubuntu2 #7 27.85 Unpacking libx32itm1 (14.2.0-4ubuntu2 #7 27.87 Unpacking lib32atomic1 (14.2.0-4ubuntu2 #7 27.88 Unpacking libx32atomic1 (14.2.0-4ubuntu2 #7 27.90 Unpacking lib32asan8 (14.2.0-4ubuntu2 #7 28.00 Unpacking libx32asan8 (14.2.0-4ubuntu2 #7 28.06 Unpacking lib32stdc++6 (14.2.0-4ubuntu2 #7 28.09 Unpacking lib32ubsan1 (14.2.0-4ubuntu2 #7 28.12 Unpacking libx32stdc++6 (14.2.0-4ubuntu2 #7 28.15 Unpacking libx32ubsan1 (14.2.0-4ubuntu2 #7 28.18 Unpacking lib32quadmath0 (14.2.0-4ubuntu2 #7 28.20 Unpacking libx32quadmath0 (14.2.0-4ubuntu2 #7 28.21 Unpacking lib32gcc-13-dev (13.3.0-6ubuntu2 #7 28.26 Unpacking libx32gcc-13-dev (13.3.0-6ubuntu2 #7 28.30 Unpacking gcc-13-multilib (13.3.0-6ubuntu2 #7 28.34 Setting up libc6-i386 (2.39-0ubuntu8.8) ... #7 28.35 Setting up libx32quadmath0 (14.2.0-4ubuntu2 #7 28.35 Setting up libx32atomic1 (14.2.0-4ubuntu2 #7 28.36 Setting up libx32gcc-s1 (14.2.0-4ubuntu2 #7 28.37 Setting up libx32asan8 (14.2.0-4ubuntu2 #7 28.37 Setting up lib32gcc-s1 (14.2.0-4ubuntu2 #7 28.38 Setting up lib32asan8 (14.2.0-4ubuntu2 #7 28.43 Setting up libx32stdc++6 (14.2.0-4ubuntu2 #7 28.43 Setting up lib32ubsan1 (14.2.0-4ubuntu2 #7 28.44 Setting up libx32gcc-13-dev (13.3.0-6ubuntu2 |
2 participants
The Linux CLANGPDB CI job intermittently fails during the oc-dev-edk2
image build: llvm.sh's own key-fetch helper (wget to
apt.llvm.org/llvm-snapshot.gpg.key) returns exit code 4 (network
failure) despite its built-in 3 retries, which aborts the whole Docker
build even though the underlying toolchain setup is otherwise fine.
Wrap the llvm.sh download+execution in an outer retry loop (5 attempts,
15s backoff) so a short-lived apt.llvm.org outage or rate limit no
longer fails CI outright. A persistent failure still fails the build
after all attempts are exhausted.