Caching DNS requests

Caching DNS Requests in Ubuntu

DNS is often overlooked part of applications latency.

But let's look to gather real data - from https://jontai.me/blog/2011/11/monitoring-dns-queries-with-tcpdump/

• Get PHP script using:

curl -o parse-tcpdump-udp-port-53.php \
   https://gist.githubusercontent.com/jtai/1368338/raw/8bed1a49de49eb2d48e32a9c1bb563a80d7ead70/parse-tcpdump-udp-port-53.php

• Run this command as root:

tcpdump -vvv -s 0 -l -n port 53 -i lo | php parse-tcpdump-udp-port-53.php -f

Why we use loopback. Because of NetworkManager + DNSmasq our /etc/resolv.conf looks like:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1

Warning! If it is NOT your case - than your are not using NetworkManager+DNSMasq and bellow DNSmasq instructions are NOT for you!

• Open your favorite browser on some page - for example here is my Firefox at https://wwww.facebook.com and watch script results, in my case:

tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
10:37:09.660751 AAAA detectportal.firefox.com                            254.439 ms !!!
10:37:09.660717 A    detectportal.firefox.com                            271.040 ms !!!
10:37:10.724537 A    detectportal.firefox.com                             63.309 ms !!
10:37:10.725030 AAAA detectportal.firefox.com                             94.826 ms !!
10:37:11.112712 A    www.mozilla.org                                     111.150 ms !!!
10:37:11.112829 AAAA www.mozilla.org                                     147.084 ms !!!
10:37:13.442631 A    addons.mozilla.org                                   80.087 ms !!
10:37:13.442661 AAAA addons.mozilla.org                                   80.100 ms !!
10:37:17.622710 A    www.facebook.com                                    119.049 ms !!!
10:37:17.622830 AAAA www.facebook.com                                    118.971 ms !!!
10:37:17.742210 A    scontent-frt3-1.xx.fbcdn.net                        141.391 ms !!!
10:37:17.742253 AAAA scontent-frt3-1.xx.fbcdn.net                        141.397 ms !!!
10:37:17.941586 A    ocsp.digicert.com                                    34.367 ms !!
10:37:17.941617 AAAA ocsp.digicert.com                                    58.194 ms !!
10:37:20.170364 A    www.facebook.com                                    237.476 ms !!!
10:37:20.170396 AAAA www.facebook.com                                    245.351 ms !!!
10:37:27.478426 A    www.facebook.com                                     68.061 ms !!
10:37:27.478457 AAAA www.facebook.com                                     68.072 ms !!

• Can you see it? Our browser spent hundred of milliseconds just to Query DNS!

Speeding up/avoiding DNS requests

These commands has been tested on:

sb_release -ds
    Ubuntu 16.04.3 LTS

Which uses Network Manager + non-caching DNS masq.

How to detect, that you have Linux with NetworkManager + dnsmasq?

• If this command returns non empty process list:

 ps axw | grep dnsmasq | grep NetworkManager
 6779 ?        S      0:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.1.1 --cache-size=0 --conf-file=/dev/null --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d

It is your case.

How to tell, that your DNSmasq is not caching.

• Try this:

# This command forces dnsmasq to report status:
ps axw | 
  grep dnsmasq |
  grep NetworkManager |
  awk '{print $1}' | xargs kill -USR1

• and look at syslog:

tail /var/log/syslog | fgrep 'cache size'
    ... dnsmasq[6779]: cache size 0, 0/0 cache insertions re-used unexpired cache entries.

Ooops - cache-size 0 does not look well...

Enabling DNSmasq caching

Create file /etc/NetworkManager/dnsmasq.d/cache.conf with contents:

cache-size=1000
# very important! There is no sensible default,
# because NM runs dnsmasq with  --conf-file=/dev/null
min-cache-ttl=3600

Restart dnsmasq with new values:

systemctl restart network-manager

Use again these commands to verify new settings:

ps axw | 
  grep dnsmasq |
  grep NetworkManager |
  awk '{print $1}' | xargs kill -USR1

tail /var/log/syslog | fgrep 'cache size'
    ... dnsmasq[7657]: cache size 1000, 0/11 cache insertions re-used unexpired cache entries.

Okay - cache size 1000 looks good.

• Now try again - close and open your browser on facebook, and you should see (in tcpdump... window):

tcpdump -vvv -s 0 -l -n port 53 -i lo | php parse-tcpdump-udp-port-53.php -f
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:10:49.434832 AAAA detectportal.firefox.com                             39.616 ms !!
11:10:49.434795 A    detectportal.firefox.com                             63.563 ms !!
11:10:51.499680 A    www.mozilla.org                                      46.793 ms !!
11:10:51.499712 AAAA www.mozilla.org                                      50.748 ms !!
11:10:58.980705 A    www.facebook.com                                     20.359 ms !!
11:10:58.981034 AAAA www.facebook.com                                     49.408 ms !!
11:10:59.031023 A    scontent-frt3-1.xx.fbcdn.net                         49.945 ms !!
11:10:59.031054 AAAA scontent-frt3-1.xx.fbcdn.net                         49.955 ms !!
11:10:59.117458 A    ocsp.digicert.com                                    24.982 ms !!
11:10:59.117488 AAAA ocsp.digicert.com                                    56.963 ms !!
11:11:01.201958 A    www.facebook.com                                      0.115 ms
11:11:01.201989 AAAA www.facebook.com                                      0.133 ms
11:11:01.203442 A    www.facebook.com                                      0.109 ms
11:11:01.203472 AAAA www.facebook.com                                      0.130 ms

Wow! 0.1 ms looks much better!

Enabling NSCD

NSCD is Name Cache Service Daemon - it communicates with glibc via name socket. It can avoid repeated DNS reqeusts at all.

To install NSCD issue this command:

apt-get install nscd

I recommend following changes in /etc/nscd.conf (shown changed lines only):

	logfile			/var/log/nscd.log
	debug-level		1
	negative-time-to-live	hosts		120

Do not forget to restart NSCD after there changes:

systemctl restart nscd

You can now watch cache miss in /var/log/nscd.conf

Try again our browser test (close and open firefox pointing to https://www.facebook.com and look at tcpdump window). You should see in output:

11:23:38.580812 A    detectportal.firefox.com                              0.114 ms
11:23:38.580842 AAAA detectportal.firefox.com                              0.137 ms
11:23:40.058115 A    www.mozilla.org                                       0.095 ms
11:23:40.058144 AAAA www.mozilla.org                                       0.100 ms
11:23:43.864541 A    www.facebook.com                                      0.108 ms
11:23:43.864571 AAAA www.facebook.com                                      0.128 ms
11:23:43.865236 A    scontent-frt3-1.xx.fbcdn.net                          0.088 ms
11:23:43.865261 AAAA scontent-frt3-1.xx.fbcdn.net                          0.094 ms
11:23:44.209730 A    ocsp.digicert.com                                     0.100 ms
11:23:44.209759 AAAA ocsp.digicert.com                                     0.119 ms
11:24:00.949413 A    pixel.facebook.com                                  291.966 ms !!!
11:24:00.949442 AAAA pixel.facebook.com                                  294.063 ms !!!

What's the difference? There should be no longer repeated DNS queries for same FQDN

• they should be cached by NSCD (you can verify it in /var/log/nscd.log.

WARNING! When you change some of NSCD cached files (/etc/hosts, /etc/services,...) you might need to restart nscd after change - so it reloads its cache. (That need depends on /etc/nscd.conf configuration).

Getting rid of IPv6

You can see from TCP dump that there are always two DNS requests, for example:

# request IPv4 address for name
11:24:00.949413 A    pixel.facebook.com                                  291.966 ms !!!

# request IPv6 address for name
11:24:00.949442 AAAA pixel.facebook.com                                  294.063 ms !!!

However I use IPv4 only - so these requests are waste of time and resources.

If you don't use IPv6 you can disable it following https://askubuntu.com/a/337736 that is:

• Add to your /etc/default/grub to variable GRUB_CMDLINE_LINUX_DEFAULT parameter ipv6.disable=1, in my case:

GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"

• Regenerate grub.cfg using:

grub-mkconfig -o /boot/grub/grub.cfg

• Restart your linux using

/sbin/init 6

After restart you should see from tcpdump that there are no more AAAA DNS requests.

Resources

• https://jontai.me/blog/2011/11/monitoring-dns-queries-with-tcpdump/