-
Notifications
You must be signed in to change notification settings - Fork 8
Caching DNS requests
DNS is often overlooked part of applications latency.
But let's look to gather real data - from https://jontai.me/blog/2011/11/monitoring-dns-queries-with-tcpdump/
- Get PHP script using:
curl -o parse-tcpdump-udp-port-53.php \
https://gist.githubusercontent.com/jtai/1368338/raw/8bed1a49de49eb2d48e32a9c1bb563a80d7ead70/parse-tcpdump-udp-port-53.php
- Run this command as root:
tcpdump -vvv -s 0 -l -n port 53 -i lo | php parse-tcpdump-udp-port-53.php -f
Why we use loopback. Because of NetworkManager + DNSmasq our
/etc/resolv.conf
looks like:# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN nameserver 127.0.1.1
Warning! If it is NOT your case - than your are not using NetworkManager+DNSMasq and bellow DNSmasq instructions are NOT for you!
- Open your favorite browser on some page - for example here is my Firefox at https://wwww.facebook.com and watch script results, in my case:
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
10:37:09.660751 AAAA detectportal.firefox.com 254.439 ms !!!
10:37:09.660717 A detectportal.firefox.com 271.040 ms !!!
10:37:10.724537 A detectportal.firefox.com 63.309 ms !!
10:37:10.725030 AAAA detectportal.firefox.com 94.826 ms !!
10:37:11.112712 A www.mozilla.org 111.150 ms !!!
10:37:11.112829 AAAA www.mozilla.org 147.084 ms !!!
10:37:13.442631 A addons.mozilla.org 80.087 ms !!
10:37:13.442661 AAAA addons.mozilla.org 80.100 ms !!
10:37:17.622710 A www.facebook.com 119.049 ms !!!
10:37:17.622830 AAAA www.facebook.com 118.971 ms !!!
10:37:17.742210 A scontent-frt3-1.xx.fbcdn.net 141.391 ms !!!
10:37:17.742253 AAAA scontent-frt3-1.xx.fbcdn.net 141.397 ms !!!
10:37:17.941586 A ocsp.digicert.com 34.367 ms !!
10:37:17.941617 AAAA ocsp.digicert.com 58.194 ms !!
10:37:20.170364 A www.facebook.com 237.476 ms !!!
10:37:20.170396 AAAA www.facebook.com 245.351 ms !!!
10:37:27.478426 A www.facebook.com 68.061 ms !!
10:37:27.478457 AAAA www.facebook.com 68.072 ms !!
- Can you see it? Our browser spent hundred of milliseconds just to Query DNS!
These commands has been tested on:
sb_release -ds
Ubuntu 16.04.3 LTS
Which uses Network Manager + non-caching DNS masq.
How to detect, that you have Linux with NetworkManager + dnsmasq?
- If this command returns non empty process list:
ps axw | grep dnsmasq | grep NetworkManager
6779 ? S 0:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --cache-size=0 --conf-file=/dev/null --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
It is your case.
How to tell, that your DNSmasq is not caching.
- Try this:
# This command forces dnsmasq to report status:
ps axw |
grep dnsmasq |
grep NetworkManager |
awk '{print $1}' | xargs kill -USR1
- and look at syslog:
tail /var/log/syslog | fgrep 'cache size'
... dnsmasq[6779]: cache size 0, 0/0 cache insertions re-used unexpired cache entries.
Ooops - cache-size 0 does not look well...
Create file /etc/NetworkManager/dnsmasq.d/cache.conf
with contents:
cache-size=1000
# very important! There is no sensible default,
# because NM runs dnsmasq with --conf-file=/dev/null
min-cache-ttl=3600
Restart dnsmasq with new values:
systemctl restart network-manager
Use again these commands to verify new settings:
ps axw |
grep dnsmasq |
grep NetworkManager |
awk '{print $1}' | xargs kill -USR1
tail /var/log/syslog | fgrep 'cache size'
... dnsmasq[7657]: cache size 1000, 0/11 cache insertions re-used unexpired cache entries.
Okay - cache size 1000 looks good.
- Now try again - close and open your browser on facebook, and you should see (in tcpdump... window):
tcpdump -vvv -s 0 -l -n port 53 -i lo | php parse-tcpdump-udp-port-53.php -f
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:10:49.434832 AAAA detectportal.firefox.com 39.616 ms !!
11:10:49.434795 A detectportal.firefox.com 63.563 ms !!
11:10:51.499680 A www.mozilla.org 46.793 ms !!
11:10:51.499712 AAAA www.mozilla.org 50.748 ms !!
11:10:58.980705 A www.facebook.com 20.359 ms !!
11:10:58.981034 AAAA www.facebook.com 49.408 ms !!
11:10:59.031023 A scontent-frt3-1.xx.fbcdn.net 49.945 ms !!
11:10:59.031054 AAAA scontent-frt3-1.xx.fbcdn.net 49.955 ms !!
11:10:59.117458 A ocsp.digicert.com 24.982 ms !!
11:10:59.117488 AAAA ocsp.digicert.com 56.963 ms !!
11:11:01.201958 A www.facebook.com 0.115 ms
11:11:01.201989 AAAA www.facebook.com 0.133 ms
11:11:01.203442 A www.facebook.com 0.109 ms
11:11:01.203472 AAAA www.facebook.com 0.130 ms
Wow! 0.1 ms looks much better!
NSCD is Name Cache Service Daemon - it communicates with glibc via name socket. It can avoid repeated DNS reqeusts at all.
To install NSCD issue this command:
apt-get install nscd
I recommend following changes in /etc/nscd.conf
(shown changed lines only):
logfile /var/log/nscd.log
debug-level 1
negative-time-to-live hosts 120
Do not forget to restart NSCD after there changes:
systemctl restart nscd
You can now watch cache miss in /var/log/nscd.conf
Try again our browser test (close and open firefox pointing to https://www.facebook.com and look at tcpdump window). You should see in output:
11:23:38.580812 A detectportal.firefox.com 0.114 ms
11:23:38.580842 AAAA detectportal.firefox.com 0.137 ms
11:23:40.058115 A www.mozilla.org 0.095 ms
11:23:40.058144 AAAA www.mozilla.org 0.100 ms
11:23:43.864541 A www.facebook.com 0.108 ms
11:23:43.864571 AAAA www.facebook.com 0.128 ms
11:23:43.865236 A scontent-frt3-1.xx.fbcdn.net 0.088 ms
11:23:43.865261 AAAA scontent-frt3-1.xx.fbcdn.net 0.094 ms
11:23:44.209730 A ocsp.digicert.com 0.100 ms
11:23:44.209759 AAAA ocsp.digicert.com 0.119 ms
11:24:00.949413 A pixel.facebook.com 291.966 ms !!!
11:24:00.949442 AAAA pixel.facebook.com 294.063 ms !!!
What's the difference? There should be no longer repeated DNS queries for same FQDN
- they should be cached by NSCD (you can verify it in
/var/log/nscd.log
.
WARNING! When you change some of NSCD cached files (
/etc/hosts
,/etc/services
,...) you might need to restartnscd
after change - so it reloads its cache. (That need depends on/etc/nscd.conf
configuration).
You can see from TCP dump that there are always two DNS requests, for example:
# request IPv4 address for name
11:24:00.949413 A pixel.facebook.com 291.966 ms !!!
# request IPv6 address for name
11:24:00.949442 AAAA pixel.facebook.com 294.063 ms !!!
However I use IPv4 only - so these requests are waste of time and resources.
If you don't use IPv6 you can disable it following https://askubuntu.com/a/337736 that is:
- Add to your
/etc/default/grub
to variableGRUB_CMDLINE_LINUX_DEFAULT
parameteripv6.disable=1
, in my case:
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"
- Regenerate grub.cfg using:
grub-mkconfig -o /boot/grub/grub.cfg
- Restart your linux using
/sbin/init 6
After restart you should see from tcpdump
that there are no more AAAA
DNS requests.
Copyright © Henryk Paluch. All rights reserved.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License