ubuntu-22.04 / 20240609.1.0 issues with IPv6 DNS resolution inside container

[samjarrett]

Description

Somewhat related to #10061, we're seeing occasional DNS resolution issues of some of our internal domains when trying to resolve inside a container that do not occur on the runner itself since upgrading - tcpdump of the runner shows only AAAA lookups being performed. We use a This seems to be related to the docker version upgrade as well, and when forcfully downgrading docker <26 this is resolved.

This is with IPv6 forcefully disabled on the host prior to the runner agent starting, eg.:

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1
sysctl -w net.ipv6.conf.lo.disable_ipv6=1

tcpdump of the AAAA requests being made:

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240609.1.0

Is it regression?

yes - did not exist in 20240603.1

Expected behavior

As the host does not have an ipv6 interface and the sysctl's explicitly have ipv6 disabled, prior to docker starting, we would've expected this not to happen

Actual behavior

AAAA DNS calls are being made, resulting in DNS resolution failures

Repro steps

inside a container on a host with runner image 20240609.1.0, curl http://some-domain-without-ipv6-results.

[ijunaidm]

@samjarrett - Thank you for bringing this issue to us. We are looking into this issue, we will update you on this issue after investigating the error you have faced.

[samjarrett]

As an update, we tried the following to 100% disable ipv6 in the kernel:

echo 'GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ipv6.disable=1"' > /etc/default/grub.d/10-disable-ipv6.cfg
update-grub

And still get the same issue.

[ijunaidm]

@samjarrett - You can try with the below approach -
you can lower the priority of IPv6 addresses by making changes to the /etc/gai.conf file, there should be enough to add the line "precedence ::ffff:0:0/96 100"

Please let us know once you try it.

[samjarrett]

@ijunaidm in the container? Or on the host? The host has ipv6 disabled at a kernel level - I don't see what this would achieve?

[ijunaidm]

@samjarrett - yes, in the docker container. Try this workaround it may work for you.

[samjarrett]

I don't think reconfiguring networking inside a container is a viable solution to ask of people?

[samjarrett]

@ijunaidm hello - as an update, we added the change to /etc/gai.conf on the machine and it continues to fail.

I've realised that this only seems to be alpine-based container images that have this issue. Alpine does not have a /etc/gai.conf inside the container, and adding one with this config did not help the situation.

As added info while we're also working to better understand it, specifically, we're seeing it most commonly on node:20-alpine inside npm install calls.