netavark dns resolves container fqdn on only one network when multiple networks connected

[ykuksenko]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When a container is connected to multiple networks only one network can resolve containers by fqdn.

Steps to reproduce the issue:

1. create two networks

podman network create --subnet 192.168.55.0/24 network1
podman network create --subnet 192.168.56.0/24 network2

2. start up a container that will be resolved in dns attached only to network1 and have it do nothing

podman run --detach --rm -ti --name container1 --network network1 alpine sleep 9000

3. create a container attached to both networks and run dig to resolve the fqdn of the first container against both network dns servers in resolv.conf

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1.dns.podman @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1.dns.podman @192.168.56.1"

4. repeat number 3 with short names

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1 @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1 @192.168.56.1"

Describe the results you received:
When resolving the fqdn of container1 only one name server responds correctly.

search dns.podman dns.podman
nameserver 192.168.55.1
nameserver 192.168.56.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; ANSWER SECTION:
container1.dns.podman.  86400   IN      A       192.168.55.2

;; Query time: 1 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 78

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 62

When resolving the short name of the container one both name server respond correctly

search dns.podman dns.podman
nameserver 192.168.56.1
nameserver 192.168.55.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 2 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

Describe the results you expected:
Both name servers should respond to both the shortname and fqdn queries.

Additional information you deem important (e.g. issue happens only occasionally):

• /etc/resolv.conf entries show up unsorted so if a user uses fqdns to reference containers they will have intermittent issues in applications. ( see /etc/resolv.conf nameserver order varies when starting a container with multiple networks. podman#14262 )
• It is also notable that /etc/resolv.conf lists the dns suffix dns.podman twice - this probably does not harm anything but should probably be deduplicated by podman.

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 16:15:54 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

# podman info  --debug
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 97.24
    systemPercent: 0.93
    userPercent: 1.83
  cpus: 2
  distribution:
    distribution: fedora
    variant: cloud
    version: "36"
  eventLogger: journald
  hostname: container.redacted
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.5-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 148381696
  memTotal: 6217089024
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 5851836416
  swapTotal: 6217003008
  uptime: 15h 16m 15.62s (Approximately 0.62 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 19
    paused: 0
    running: 19
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 41788899328
  graphRootUsed: 9318744064
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 67
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853754
  BuiltTime: Fri May  6 16:15:54 2022
  GitCommit: ""
  GoVersion: go1.18
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q netavark podman
netavark-1.0.3-3.fc36.x86_64
podman-4.1.0-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Libvirt VM

[mheon]

@flouthoc PTAL

[flouthoc]

@ykuksenko I think container1 is only connected to network1 please connect it to network2 as well in order to get it resolved from nameserver in network2. So podman run --detach --rm -ti --name container1 --network network1,network2 alpine sleep 9000 should be the right thing here.

OTOH you could say that bug is that short name should not get resolved by nameserver if container is not connected to the network of nameserver.

[ykuksenko]

container1 is intentionally connected to only network1 for isolation. For some additional context of where I ran into this issue, I use the following scenario so that the database is more isolated from the load balancer:

Load Balancer  ------------> Application -------------> Database
                (network1)                (network2)

In this scenarios the application container sometimes fails to resolve the database when addressed with FQDN. I prefer using FQDNs for names because it makes it really obvious what the term refers to and that the name is meant to be internally resolved, and not by my main DNS server.

I suppose the biggest thing I would like to see is consistent behavior:

• If shortnames should be resolvable universally, then FQDNs should be too. (preferred solution)
• If FQDNs should not be resolvable universally then shortnames should not be too.

I do not see any reason for shortnames to resolve and FQDNs to fail or vise versa. If there is such a reason could you share it?

I prefer having both FQDNs and shortnames to be resolvable because:

• the autogenerated /etc/resolv.conf file randomly orders podman network name servers which causes unpredictability
• FQDNs avoid name ambiguity if the short name is defined in the host DNS servers. Or if DNS suffixes are utilized (not sure if these are configurable in netavark/aardvark)
• it is convenient
• it avoids having tweaking /etc/resolv.conf to try multiple DNS servers in case the first one cannot respond and response durations. (With the alpine image, if the first server cannot respond pings simply fail saying bad address, same with Keycloak trying to connect to a database)
• Podman container names are unique so any server being able to respond on behalf of any attached network for internal resources seems sound. (This brings up the question of why have more than one server if they act as one anyway but this is not important)
• I am not aware of any down side to this. I am missing one?

I am under the impression that netavark and aardvark-dns should combine DNS results from any network a container is connected to according to @mheon (containers/podman#14262 (comment))

[flouthoc]

@ykuksenko Does the requested feature works on docker cause this seems a bug a too me I think we should fix this otherway around and make short names also not resolvable if they are not part of the network.

[baude]

should this be moved to nv?

[ykuksenko]

Sorry for the delay. I do not usually use docker these days.

Docker works a bit differently in this area but it does work intuitively, at least for me.

1. Docker seems to only allow creating a container with one network attached. After creating you can separately run docker network connect $network_name $container_name to add the second network. I was not able to find another way with a quick look.
2. Docker provides only one dns server in /etc/resolv.conf: even when multiple networks are attached. This is the same across all containers

/ # cat /etc/resolv.conf
search mydomain.com vagrant.mydomain.com
nameserver 127.0.0.11
options edns0 trust-ad ndots:0

3. The original container automatically gets its short name and a long name that is specific to its network. Not a generic name like podman provides. So container1 from this example resolves at container1 and container1.network1.
4. Since container1 is only connected to one network I created both container2 and container3 that are both connected to both network1 and network2 and tested resolving container1 and container2 from container3. As there is only one dns server this is simpler and avoids having the OS figure out which nameserver is responsible for what as well as any timeouts associated with that.

From container3: the following names resolve:
container1 -> 192.168.55.2
container1.network1 -> 192.168.55.2
container1.network2 -> bad address (as it should be as container1 is not connected to network2
container2 -> 192.168.55.3
container2.network1 -> 192.168.55.3
contaner2.network2 -> 192.168.56.2

From container1 things also resolve as expected. Namely only other containers in network1 with both shortnames and fqdns resolve successfully.

5. Every name resolved to one and only one IP. A short name never returned both network1 and network2 names from any container.

Here is the list of commands I ran for setup.

docker network create --subnet 192.168.55.0/24 network1
docker network create --subnet 192.168.56.0/24 network2
docker run --detach --rm -ti --name container1 --network network1 alpine sleep 9000
docker run --rm -ti --name container2 -d --network network1 alpine sleep 9000
docker network connect network2 container2
docker run --rm -ti --name container3 -d --network network1 alpine sleep 9000
docker network connect network2 container3

From there I used the following commands to test:

docker exec -ti container1 apk add bind-tools
docker exec -ti container2 apk add bind-tools
docker exec -ti container3 apk add bind-tools
docker exec -ti container1 cat /etc/resolv.conf     #can see above
docker exec -ti container2 cat /etc/resolv.conf     #can see above
docker exec -ti container3 cat /etc/resolv.conf     #can see above

docker exec -ti container1 dig +short container1             #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network1    #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network2    #-> blank (entry should not exist)
docker exec -ti container1 dig +short container3.network2    #-> blank (container1 is not in network2 so access  is denied)

docker exec -ti container3 dig +short container1             #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network1    #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network2    #-> blank (entry should not exist)
docker exec -ti container3 dig +short container2             #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network1    #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network2    #-> 192.168.56.2
docker exec -ti container3 dig +short container3             #-> 192.168.55.4

Hopefully this covers everything.

edit: my docker version: Docker version 20.10.16, build aa7e414 on fedora 36
edit2: change second sentence to be less presumptuous.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[felixsanz]

Bump. Happening to me and it's very annoying! @flouthoc I can confirm this works as expected in docker, been using this networking stuff for years.

In fact this issue was first reported early 2021 here: containers/podman#9492

[rhatdan]

@flouthoc @Luap99 @felixsanz is @ykuksenko Is this still an issue?

[Luap99]

I move this to aardvark-dns, looks like this is still a problem

[openshift-ci]

@ykuksenko: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to this:

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When a container is connected to multiple networks only one network can resolve containers by fqdn.

Steps to reproduce the issue:

1. create two networks

podman network create --subnet 192.168.55.0/24 network1
podman network create --subnet 192.168.56.0/24 network2

2. start up a container that will be resolved in dns attached only to network1 and have it do nothing

podman run --detach --rm -ti --name container1 --network network1 alpine sleep 9000

3. create a container attached to both networks and run dig to resolve the fqdn of the first container against both network dns servers in resolv.conf

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1.dns.podman @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1.dns.podman @192.168.56.1"

4. repeat number 3 with short names

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1 @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1 @192.168.56.1"

Describe the results you received:
When resolving the fqdn of container1 only one name server responds correctly.

search dns.podman dns.podman
nameserver 192.168.55.1
nameserver 192.168.56.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; ANSWER SECTION:
container1.dns.podman.  86400   IN      A       192.168.55.2

;; Query time: 1 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 78

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 62

When resolving the short name of the container one both name server respond correctly

search dns.podman dns.podman
nameserver 192.168.56.1
nameserver 192.168.55.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 2 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

Describe the results you expected:
Both name servers should respond to both the shortname and fqdn queries.

Additional information you deem important (e.g. issue happens only occasionally):

• /etc/resolv.conf entries show up unsorted so if a user uses fqdns to reference containers they will have intermittent issues in applications. ( see /etc/resolv.conf nameserver order varies when starting a container with multiple networks. podman#14262 )
• It is also notable that /etc/resolv.conf lists the dns suffix dns.podman twice - this probably does not harm anything but should probably be deduplicated by podman.

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 16:15:54 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

# podman info  --debug
host:
 arch: amd64
 buildahVersion: 1.26.1
 cgroupControllers:
 - cpuset
 - cpu
 - io
 - memory
 - hugetlb
 - pids
 - misc
 cgroupManager: systemd
 cgroupVersion: v2
 conmon:
   package: conmon-2.1.0-2.fc36.x86_64
   path: /usr/bin/conmon
   version: 'conmon version 2.1.0, commit: '
 cpuUtilization:
   idlePercent: 97.24
   systemPercent: 0.93
   userPercent: 1.83
 cpus: 2
 distribution:
   distribution: fedora
   variant: cloud
   version: "36"
 eventLogger: journald
 hostname: container.redacted
 idMappings:
   gidmap: null
   uidmap: null
 kernel: 5.17.5-300.fc36.x86_64
 linkmode: dynamic
 logDriver: journald
 memFree: 148381696
 memTotal: 6217089024
 networkBackend: netavark
 ociRuntime:
   name: crun
   package: crun-1.4.4-1.fc36.x86_64
   path: /usr/bin/crun
   version: |-
     crun version 1.4.4
     commit: 6521fcc5806f20f6187eb933f9f45130c86da230
     spec: 1.0.0
     +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
 os: linux
 remoteSocket:
   path: /run/podman/podman.sock
 security:
   apparmorEnabled: false
   capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
   rootless: false
   seccompEnabled: true
   seccompProfilePath: /usr/share/containers/seccomp.json
   selinuxEnabled: true
 serviceIsRemote: false
 slirp4netns:
   executable: /usr/bin/slirp4netns
   package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
   version: |-
     slirp4netns version 1.2.0-beta.0
     commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
     libslirp: 4.6.1
     SLIRP_CONFIG_VERSION_MAX: 3
     libseccomp: 2.5.3
 swapFree: 5851836416
 swapTotal: 6217003008
 uptime: 15h 16m 15.62s (Approximately 0.62 days)
plugins:
 log:
 - k8s-file
 - none
 - passthrough
 - journald
 network:
 - bridge
 - macvlan
 volume:
 - local
registries:
 search:
 - docker.io
store:
 configFile: /usr/share/containers/storage.conf
 containerStore:
   number: 19
   paused: 0
   running: 19
   stopped: 0
 graphDriverName: overlay
 graphOptions:
   overlay.mountopt: nodev,metacopy=on
 graphRoot: /var/lib/containers/storage
 graphRootAllocated: 41788899328
 graphRootUsed: 9318744064
 graphStatus:
   Backing Filesystem: btrfs
   Native Overlay Diff: "false"
   Supports d_type: "true"
   Using metacopy: "true"
 imageCopyTmpDir: /var/tmp
 imageStore:
   number: 67
 runRoot: /run/containers/storage
 volumePath: /var/lib/containers/storage/volumes
version:
 APIVersion: 4.1.0
 Built: 1651853754
 BuiltTime: Fri May  6 16:15:54 2022
 GitCommit: ""
 GoVersion: go1.18
 Os: linux
 OsArch: linux/amd64
 Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q netavark podman
netavark-1.0.3-3.fc36.x86_64
podman-4.1.0-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Libvirt VM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.