dns request failed: request timed out #248
Comments
|
Ah, after turning /etc/resolv.conf again into a symlink of systemd-resolved I don't know how I could let this plugin verbose debug logging into the journal, but I'm glad that it is fixed now. |
|
And it's back It seems to happen if I execute |
|
I detected the same problem on my raspberry pi with Fedora and rootless podman containers. |
|
Having the exact same issue. DNS lookup is extremely unstable, only two thirds of lookups work. This is on Fedora Silverblue 37 using rootless containers and the new networking stack. |
|
Now using a nextcloud container which communicates with a PostgreSQL and a LDAP container. 1/3 of the nextcloud web interface results into a 502 bad gateway, because of the unstable DNS. I hope the next podman release will include the new network stack which hopefully fixes this issue. |
|
It is impossible to help with these issues as reporters did not provide versions for podman, netavark, and aarvark-dns. please provide as much relevant information as possible. |
|
podman: 4.3.1 |
|
would you say your machine/vm is high performent or maybe has slow IO/processor/RAM limitations? I'm trying to understand if you might have a race. |
|
The first bare metal server server with this issue has the HDD connected to SATA with the server. The second bare metal server has a 870 Evo SSD connected via SATA to the server. The second server spams this issue many times. |
|
@flouthoc wdyt? |
|
Now my faster server spammed it at night while the server pods weren't used by clients. |
|
We used to see similar issues in older versions of netavark and aardvark in Podman CI as well but it was fixed in newer versions with #220 but I guess there might be some issue which is not being reproduced in our CI, I'll try to reproduce this locally and see if i can reproduce this. |
|
I'm seeing the same problems on a freshly installed Fedora Server 37 instance as well. The machine is basically completely idling with no load and my journal is still filled with these errors. This issue was not present before 2022, and I've ran similar setups on slower machines without any DNS lookup issues. podman: 4.3.1 |
|
Can you test with v1.5? |
|
I think the timeout is fixed. |
|
Does this cause problems for the container or is just an error that is logged often? |
|
It's logged often with long breaks between. So it appears in groups most of the time. I think the services just repeat the DNS action again until it works. Maybe I don't use it long enough to see long term errors. |
|
Do you have a simple reproducer? What kind of application are you running and how many dns request does it make? |
|
Tested with v1.5 and I'm getting a lot of
It's notable that none of these containers are particularly demanding on the hardware, and my system load average is generally below 0.1 at all times. |
|
It happens without workload. The server just runs
|
|
I'm experiencing the same issue with aardvark-dns reporting lots of It happens with both 1.5.0 and the latest 1.6.0 from podman-next. For me it seems to start after around 3 days of uptime. I've tried changing machines and switching from onboard Realtek to an Intel i350-T2 controller, but both to no avail. Rebooting solves the issue, until uptime reaches 3 days again. |
|
Just ran into this issue as well with Nextcloud + Nginx Proxy Manager. What's funny is that I am using the same docker-compose setup on two different servers and one works fine while the other one doesn't. The only difference is that the one that is breaking isn't publicly accessible on the Internet and is instead setup to respond over a It will work for a bit when I up/down NPM, but then eventually fail after a few hours or even days with 502 bad gateway errors, and My setup
Here are my docker-compose files to set up each of them (rootful btw): version: '3'
services:
db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
restart: always
volumes:
- ./db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=<pw here>
- MARIADB_AUTO_UPGRADE=1
- MARIADB_DISABLE_UPGRADE_BACKUP=1
env_file:
- db.env
networks:
- backend
redis:
image: redis:alpine
restart: always
networks:
- backend
nextcloud:
image: nextcloud:apache
restart: always
volumes:
- ./html:/var/www/html
environment:
- MYSQL_HOST=db
- REDIS_HOST=redis
env_file:
- db.env
depends_on:
- db
- redis
networks:
- nextcloud_frontend
- backend
cron:
image: nextcloud:apache
restart: always
volumes:
- ./html:/var/www/html
entrypoint: /cron.sh
depends_on:
- db
- redis
networks:
- backend
networks:
nextcloud_frontend:
external: true
backend:db.env MYSQL_PASSWORD=<pw here>
MYSQL_DATABASE=nextcloud
MYSQL_USER=nextcloudNginx Proxy Manager docker-compose.yml version: '3.8'
services:
proxy:
image: 'jc21/nginx-proxy-manager:latest'
restart: always
ports:
# These ports are in format <host-port>:<container-port>
- '80:80' # Public HTTP Port
- '443:443' # Public HTTPS Port
- '81:81' # Admin Web Port
# Add any other Stream port you want to expose
# - '21:21' # FTP
# Uncomment the next line if you uncomment anything in the section
# environment:
# Uncomment this if you want to change the location of
# the SQLite DB file within the container
# DB_SQLITE_FILE: "/data/database.sqlite"
# Uncomment this if IPv6 is not enabled on your host
# DISABLE_IPV6: 'true'
healthcheck:
test: ["CMD", "/bin/check-health"]
interval: 30s
timeout: 3s
volumes:
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- nextcloud_frontend
networks:
nextcloud_frontend:
external: true |
|
Can anyone check if they still see this with aardvark-dns v1.12.2? |
|
@Luap99 At least for me it looks fine. |
|
I started getting a fair number of podman version 5.3.0-dev-29eb8ce09 |
|
I encountered the same issue, and it turned out to be self-inflicted. I had added the podman network gateway IP to my |
|
My Over the course of a few hours yesterday (21:31 to 23:00 UTC), a bunch of Also network related, and also odd, is Other than dnf-automatic package installs, not much about my dev machine has changed in the better part of a year. |
I'm using arch linux, so the packages should have the newest version.
I'm using firewallD and rootless podman with netavark and aardvark-dns.
I understand, that rootless podman with netavark won't manage my firewallD, but I would like to know which rules I need to activate to avoid the spam in my journal.
And if the rule need to be in my loopback or network interface. (Also if it is enough to allow communication with the host instead of having an open port in the internet.
My dns resolver is systemd-resolved
$ ls -lha /etc/resolv.conf lrwxrwxrwx 1 root root 39 31. Okt 10:22 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.confMy journal spam:
aardvark-dns[6156]: 21433 dns request failed: request timed outThe rootless container itself can ping to
google.com.I didn't test if they can ping to a container dns name.
The text was updated successfully, but these errors were encountered: