unexpected end of input reached #151
Comments
|
Hi @binarious , Thanks for creating the issue. Would it be possible to share a small reproducer. I have tried doing isolated load testing on #!/bin/bash
for (( c=1; c<=100; c++ ))
do
dig -p 5533 @<my-interface-ip> testdbctr
doneI am still trying for a reproducer so will update once i am able to reproduce this issue. |
|
Hi @flouthoc, I'll also do my best to create a small reproduction setup. Thank you for looking into it. |
|
I often see these errors with my Nextcloud instance when I try to open the site after long periods of inactivity. I have 4 machines running Arch Linux, and I see this error on all of them. Might be unrelated, but I also see this error on startup alot of times: |
|
Hello, I'm affected as well by this issue, both with aardvark-dns v1.0.3 and v.1.1.0. In the logs below I'm running a container (vaultwarden) which connects to another container (a Postgres database) through a network bridge named "web-apps-compose_vaultwarden-net" with Podman in rootless mode. Once vaultwarden is started it will sooner or later complain about failed DNS resolutions which are at the same time coupled with logs from aardvark-dns which exactly match the ones reported in this issue. (see more in the logs below) I tried running aardvark-dns with Podman infohost:
arch: amd64
buildahVersion: 1.26.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.3-1
path: /usr/bin/conmon
version: 'conmon version 2.1.3, commit: ab52a597278b20173440140cd810dc9fa8785c93'
cpuUtilization:
idlePercent: 97.48
systemPercent: 1.3
userPercent: 1.23
cpus: 2
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: Botti-VPS
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.18.15-hardened1-1-hardened
linkmode: dynamic
logDriver: journald
memFree: 91676672
memTotal: 2075852800
networkBackend: netavark
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.5-1
path: /usr/bin/crun
version: |-
crun version 1.5
commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1001/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 0
swapTotal: 1037561856
uptime: 2h 51m 22.51s (Approximately 0.08 days)
plugins:
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
volume:
- local
registries: {}
store:
configFile: /home/container/.config/containers/storage.conf
containerStore:
number: 9
paused: 0
running: 9
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/container/.local/share/containers/storage
graphRootAllocated: 42410704896
graphRootUsed: 7771459584
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 9
runRoot: /run/user/1001/containers
volumePath: /home/container/.local/share/containers/storage/volumes
version:
APIVersion: 4.1.1
Built: 1656056036
BuiltTime: Fri Jun 24 09:33:56 2022
GitCommit: f73d8f8875c2be7cd2049094c29aff90b1150241
GoVersion: go1.18.3
Os: linux
OsArch: linux/amd64
Version: 4.1.1
Logs of the container failing to resolve/--------------------------------------------------------------------\ | Starting Vaultwarden | | Version 1.25.2 | |--------------------------------------------------------------------| | This is an *unofficial* Bitwarden implementation, DO NOT use the | | official channels to report bugs/features, regardless of client. | | Send usage/configuration questions or feature requests to: | | https://vaultwarden.discourse.group/ | | Report suspected bugs/issues in the software itself at: | | https://github.com/dani-garcia/vaultwarden/issues/new | \--------------------------------------------------------------------/ Sample of the logs from aardwark-dns running with RUST_LOG=trace (journalctl -b | grep aard)Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: request source address: 10.89.5.3:54633
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: requested record type: AAAA
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: checking if backend has entry for: "postgres-db.dns.podman."
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: server backend.name_mappings: {"email-apps-compose_postfix-mta-sts-resolver-net": {"postfix-mta-sts-resolver": [10.89.7.2], "email-apps-compose_postfix-mta-sts-resolver_1": [10.89.7.2], "54c43a779c7f": [10.89.7.2]}, "web-apps-compose_nextcloud-net": {"web-apps-compose_nextcloud-webapp_1": [10.89.4.4], "web-apps-compose_postgres-db_1": [10.89.4.2], "6897e4e7219e": [10.89.4.2], "833c41410132": [10.89.4.4], "nextcloud-webapp": [10.89.4.4], "web-apps-compose_nextcloud-cron_1": [10.89.4.5], "nextcloud-cron": [10.89.4.5], "nextcloud-redis": [10.89.4.3], "b8607ed4dadb": [10.89.4.3], "ca52f8e2e029": [10.89.4.5], "web-apps-compose_nextcloud-redis_1": [10.89.4.3], "postgres-db": [10.89.4.2]}, "web-apps-compose_trilium-web-net": {"2c27d07a273f": [10.89.3.2], "web-apps-compose_trilium-webapp_1": [10.89.3.2], "trilium-webapp": [10.89.3.2]}, "web-apps-compose_vaultwarden-net": {"e518ae7c468d": [10.89.5.3], "web-apps-compose_postgres-db_1": [10.89.5.2], "6897e4e7219e": [10.89.5.2], "postgres-db": [10.89.5.2], "web-apps-compose_vaultwarden-webapp_1": [10.89.5.3], "vaultwarden-webapp": [10.89.5.3]}}
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: server backend.ip_mappings: {10.89.5.2: ["web-apps-compose_nextcloud-net", "web-apps-compose_vaultwarden-net"], 10.89.3.2: ["web-apps-compose_trilium-web-net"], 10.89.4.3: ["web-apps-compose_nextcloud-net"], 10.89.4.4: ["web-apps-compose_nextcloud-net"], 10.89.7.2: ["email-apps-compose_postfix-mta-sts-resolver-net"], 10.89.4.2: ["web-apps-compose_nextcloud-net", "web-apps-compose_vaultwarden-net"], 10.89.5.3: ["web-apps-compose_vaultwarden-net"], 10.89.4.5: ["web-apps-compose_nextcloud-net"]}
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: server backend kill switch: true
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: No backend lookup found, try resolving in current resolvers entry
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: [136] success reponse
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:48 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: Failed while parsing message: unexpected end of input reached
Aug 02 12:15:50 Botti-VPS aardvark-dns[626]: None received while parsing dns message, this is not expected server will ignore this message
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: parsed message "[19814] parsed message body: postgres-db.dns.podman. A IN edns: false"
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: server name: "web-apps-c"
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: request source address: 10.89.5.3:33697
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: requested record type: A
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: checking if backend has entry for: "postgres-db.dns.podman."
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: server backend.name_mappings: {"email-apps-compose_postfix-mta-sts-resolver-net": {"postfix-mta-sts-resolver": [10.89.7.2], "email-apps-compose_postfix-mta-sts-resolver_1": [10.89.7.2], "54c43a779c7f": [10.89.7.2]}, "web-apps-compose_nextcloud-net": {"web-apps-compose_nextcloud-webapp_1": [10.89.4.4], "web-apps-compose_postgres-db_1": [10.89.4.2], "6897e4e7219e": [10.89.4.2], "833c41410132": [10.89.4.4], "nextcloud-webapp": [10.89.4.4], "web-apps-compose_nextcloud-cron_1": [10.89.4.5], "nextcloud-cron": [10.89.4.5], "nextcloud-redis": [10.89.4.3], "b8607ed4dadb": [10.89.4.3], "ca52f8e2e029": [10.89.4.5], "web-apps-compose_nextcloud-redis_1": [10.89.4.3], "postgres-db": [10.89.4.2]}, "web-apps-compose_trilium-web-net": {"2c27d07a273f": [10.89.3.2], "web-apps-compose_trilium-webapp_1": [10.89.3.2], "trilium-webapp": [10.89.3.2]}, "web-apps-compose_vaultwarden-net": {"e518ae7c468d": [10.89.5.3], "web-apps-compose_postgres-db_1": [10.89.5.2], "6897e4e7219e": [10.89.5.2], "postgres-db": [10.89.5.2], "web-apps-compose_vaultwarden-webapp_1": [10.89.5.3], "vaultwarden-webapp": [10.89.5.3]}}
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: server backend.ip_mappings: {10.89.5.2: ["web-apps-compose_nextcloud-net", "web-apps-compose_vaultwarden-net"], 10.89.3.2: ["web-apps-compose_trilium-web-net"], 10.89.4.3: ["web-apps-compose_nextcloud-net"], 10.89.4.4: ["web-apps-compose_nextcloud-net"], 10.89.7.2: ["email-apps-compose_postfix-mta-sts-resolver-net"], 10.89.4.2: ["web-apps-compose_nextcloud-net", "web-apps-compose_vaultwarden-net"], 10.89.5.3: ["web-apps-compose_vaultwarden-net"], 10.89.4.5: ["web-apps-compose_nextcloud-net"]}
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: server backend kill switch: true
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: No backend lookup found, try resolving in current resolvers entry
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: [19814] success reponse
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: parsed message "[20132] parsed message body: postgres-db.dns.podman. AAAA IN edns: false"
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: server name: "web-apps-c"
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: request source address: 10.89.5.3:33697
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: requested record type: AAAA
Aug 02 12:15:53 Botti-VPS aardvark-dns[626]: checking if backend has entry for: "postgres-db.dns.podman."
aardvark-dns-FULL_LOG.txt (With RUST_LOG=trace; Note that I redacted some parts) |
|
@DynamoFox with version |
Sure, They were updated together and I fully rebooted the machine before experimenting the issue again. |
|
Can you attach strace to the aardvark process and provide the logs when it happens. |
Okay, here they are. Inside there are two different reports:
I tested the report1 configuration only once, when I have more time I'll try it again and report back, so don't mind it too much for now. EDIT: It should be noted (not in these tests) that I tried to manually install dig in one of the running containers affected by this problem, and no matter how much I tried resolutions from it, it never failed... |
|
Can you add |
|
Here's report3, same conditions as report2. |
|
I don't have time to debug this further but it looks like querys for |
|
|
It seems like the issue persists even with the latest update of aardvark to 1.5.0 |
|
has anyone see it in 1.6 or 1.7? |
I've setup a nextcloud deployment with podman in combination with docker-compose. It's working fine mostly and dns is also generally working. But around every 30s I see connection issues between containers. I've created a test script which queries the database every second and I often see connection errors (
SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Try again).journalctlshows the following output at these times:Any ideas?
Deployment Setup
System details
podman
docker-compose.yml
It's
appwhich sometimes cannot connect todbandredis.The text was updated successfully, but these errors were encountered: