How I reverted to Name Service Switch (NSS) for name resolution — And why

There are problems with the new systemd-resolved resolver and that gives us some good reasons to keep or revert to the NSSwitch resolver.

The Domain Name Services (DNS)¹ system provides the database used in the translation of Internet locations from human readable hostnames, such as www.example.net, to IP Addresses, like 54.204.39.132, so that our Internet connected computers and other devices can access them. Without these name resolver services it would be nearly impossible to surf the web as freely and easily as we do. As humans, we tend to do better with names like opensource.org while computers do much better with numbers like 104.21.84.214. So we need a translation service to convert the names that are easy for us to the IP addresses that are easy for our computers.

Every computer needs its own resolver service so that it can locate hosts on the local network and the Internet. But it’s complicated.

In this article we look at the details of the historical NSS resolver, tools and techniques for managing it, and how to revert to it when there are problems with the newer systemd-resolved resolver that implements mDNS.

resolv.conf

We start with the /etc/resolv.conf file because it is the key to determining how the systemd-resolved.service works. This file is an ASCII plain text file that contains a list of up to three domain name servers that are be used to perform hostname resolution into IP addresses. Under systemd-resolved, it’s actually a link to the /run/systemd/resolve/stub-resolv.conf file.

The resolv.conf file also contains the domain name to search when a fully qualified domain name (FQDN) isn’t appended to the hostname. For example a fully qualified domain name would be host1.example.com. This can be searched without a problem. But suppose I just use a hostname line host1 and not the domain name. In that case the domain name specified for searches is appended to the hostname.

A typical resolv.conf file under systemd-resolved looks like that in Figure 1. In this case it’s a link to the /run/systemd/resolve/stub-resolv.conf file and contains the search domain as well as the IP addresses of the name servers provided by the DHCP server.

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search both.org

Figure 1. A typical resolv.conf file when using systemd-resolved.

The nameserver line of the resolv.conf file is the important one for this article. The IP address 127.0.0.53 is designated as the local host name service resolver. Port 53 on DNS servers is the DNS port and this mDNS resolver only works for the localhost on that IP address. The presence of this line in resolv.conf is one easy way to determine that your host is using mDNS.

Name service strategies

There are currently three strategies available for use in resolving domain names into IP addresses. Each has its own tools, advantages, and best use cases. Two of these tools require work on the part of the SysAdmin. mDNS, though requiring almost no administrative work, is quite chatty and creates a significant amount of network traffic. It also has its share of problems.

We’ll take a quick peek at the /etc/hosts file and Multicast DNS (mDNS). Then we’ll discuss nss-DNS and reverting to it.

The /etc/hosts file

The /etc/hosts² file is an ASCII plain-text file that can list the IP addresses of all hosts on the local network and was the first tool used for local network name resolution. It can also be used for non-local hosts.

In small networks the /etc/hosts file on each host can be used as a simple local name resolver. The SysAdmin can add and manage entries in the hosts file. Maintaining copies of this file on several hosts can become very time-consuming and errors can cause much confusion and wasted time before they’re found. Although the hosts file can have non-local domains such as www.example.net added to it, if the IP addresses can be discovered, it is a labor intensive tool best suited for use in small local networks and limited to internal hosts.

A default hosts file such as the one in Figure 2 is always present but it would normally contain only the lines needed to enable internal services and commands to translate the localhost hostname to IPV4 address 127.0.0.1 and IPV6 address ::1 – this is an explicitly defined standard to enable Linux services and commands to work with the local host.

# Loopback entries; do not change.
# For historical reasons, localhost precedes localhost.localdomain:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# See hosts(5) for proper format and other examples:
# 192.168.1.10 foo.mydomain.org foo
# 192.168.1.13 bar.mydomain.org bar

Figure 2. The default hosts file contains the two standard entries, some additional examples, and a suggestion for obtaining more information.

mDNS

Multicast³ DNS (mDNS⁴) is a relatively new addition to name service resolution. Intended to provide name resolution for local networks that have no internal, central name resolver, mDNS requires no user intervention. In addition to automatic discovery of local hosts, it also uses more traditional name services for access to the Internet. mDNS is now the default name resolution strategy for Fedora and other distributions.

Multicast services like mDNS send out broadcast (multicast) packets that are received and examined by every host on the network. The packet is a request to the computer with the hostname it wants to communicate with, to respond with its IP address so the requesting computer can send further packets directly to that host. Since only one computer (hopefully) has that hostname, only that computer will respond with its IP address and the requesting host enters that hostname/IP address into its local database. Other computers on the network that use mDNS can also add that data to their own mDNS databases. With mDNS each host keeps its own database. Entries in the database have a TTL (Time To Live) so the entries will expire at the end of their TTL. This means that the host must make another mDNS broadcast request to the network in order to obtain that IP address again.

The cost of this level of automation for local host discovery is a significant amount of network traffic from each host that is intended to discover other hosts on the network. This type of chatty protocol sucks up network bandwidth, uses host system resources, and is not fast relative to the more traditional nss-DNS protocols. The speed penalty was why I first noticed this service was less efficient than nss-DNS.

The /etc/resolv.conf and /etc/nsswitch.conf files are symbolic (soft) links to the created files. Figure 3 shows these files. The links keep the same date and time as when they were originally included in the installation package. The /etc/authselect/nsswitch.conf file was created during system installation. The ../run/systemd/resolve/stub-resolv.conf was created during the most recent boot.

# ll resolv.conf nsswitch.conf
lrwxrwxrwx. 1 root root 29 Jan 29 10:16 nsswitch.conf -> /etc/authselect/nsswitch.conf
lrwxrwxrwx. 1 root root 39 Oct 24 10:53 resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

# ll /etc/authselect/nsswitch.conf ../run/systemd/resolve/stub-resolv.conf
-rw-r--r--. 1 root            root            671 Jan 29 10:16 /etc/authselect/nsswitch.conf
-rw-r--r--. 1 systemd-resolve systemd-resolve 927 Jan 30 09:01 ../run/systemd/resolve/stub-resolv.conf

Figure 3. The resolver files used to configure DNS client services.

Actually, two versions of the resolv.conf file are created in the /run/systemd/resolve/ directory each time the host is booted. Figure 4 shows both of these.

root@testvm1:/run/systemd/resolve# cat resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.0.52
nameserver 8.8.8.8
nameserver 8.8.4.4
search both.org


root@testvm1:/run/systemd/resolve# cat stub-resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.cat stub-resolv.conf
#
# This file might be symlinkevd as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search both.org

Figure 4. The contents of the two resolver files in the /run/systemd/resolve directory.

The top file in Figure 4 is a more traditional resolv.conf file but it is not used by default. It defines three external servers for name resolution. This information can be obtained from the DHCP server for the network, or from NetworkManager network connection files⁵ for static configurations.

The bottom file in Figure 4 is the systemd-resolved version, stub-resolv.conf and it is the target of the /etc/resolv.conf symlink. This file defines the local host as the name server for this client host.

nss-DNS

This is the historical name service and resolver combination. The Name Service Switch (NSS)⁶ resolver performs the client tasks of requesting IP addresses from the global Domain Name Service distributed database. It has the attributes of being fast, easy to use, mature, and well known.

Originally developed by Sun Microsystems as part of their Solaris⁷ operating system, NSS was rewritten for the GNU utilities and tools. It’s code is embedded in the GNU glibc⁸ library so is always present on all Linux systems. This makes it easy to revert to it when necessary.

Two ASCII plain text files are used to provide the primary configuration for name services. These files, nsswitch.conf and resolv.conf have historical origins having been around since the earliest versions of name service resolvers.

NSSwitch

As its name implies, the nsswitch – short for Name Service Switch – is used to define the database sources and order in which name-service information is obtained. It’s used for much more than just name services but we’ll ignore those other tasks in this article.

Unicast services like nss-DNS use a single server that maintains the entire database. If a host needs the IP address of another host on the network, it sends a unicast packet only to the name server requesting the IP address from it. The name server responds only to the requesting host with a packet containing that IP address.

The NSS facility is a tool that is used by a number of services that need name resolver data. Using NSS based on the data in the /etc/nsswitch.conf configuration file, it aids them in locating the appropriate configuration and name resolution sources in a specified sequence.

The sequences listed for each service in the nsswitch.conf file can be changed, and can differ between distributions. This file can also be modified to meet local needs. Until relatively recently, I’ve never needed to change anything about this file and it is a good place to start problem determination if there seems to be a problem with name resolution that can’t be otherwise explained.

Figure 5 shows the modern default version of nsswitch.conf. This file is created during Linux startup.

# Generated by authselect
# Do not modify this file manually, use authselect instead. Any user changes will be overwritten.
# You can stop authselect from managing your configuration by calling 'authselect opt-out'.
# See authselect(8) for more details.

# In order of likelihood of use to accelerate lookup.
passwd:     files sss systemd
shadow:     files
group:      files [SUCCESS=merge] sss [SUCCESS=merge] systemd
hosts:      files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns
services:   files sss
netgroup:   files sss
automount:  files sss

aliases:    files
ethers:     files
gshadow:    files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files

Figure 5. The default version of nsswitch.conf.

Look at the hosts database entry in the data stream. The first entry is “files” which means that the resolver is to search first the local database. The database isn’t explicitly specified here but it is the /etc/hosts file that we experimented with above.

If a match is not found, the resolver moves on to the next entry which is “myhostname.” This provides name resolution for the locally configured system hostname as contained in the $HOSTNAME environment variable.

The myhostname entry in this line is new and is intended to provide fail-safe resolution of any local hostnames such as localhost and localhost.localdomain. This is built into the resolver code and is not dependent on the entries in the /etc/hosts file.

Because all of these entries are sequence-sensitive, if an entry is found for a host name in the /etc/hosts database, that takes precedent over any other, later entries.

The nsswitch.conf file in Figure 5 mentions a tool called authselect that’s used to determine the content of the nsswitch.conf file. It also describes the simple command used to opt-out of of using authselect to manage the hosts name services; we’ll use that as we extract ourselves from systemd-resolved.

Fedora name resolution fails when using systemd-resolved

When I first installed Fedora 33, one of the major changes in that release, a switch from the nss resolver to systemd-resolved caused me a significant amount of trouble and borked my entire network.

The change from the venerable nsswitch and NetworkManager to systemd-resolved damaged and slowed name services. The result of this resolver change was apparent in a number of symptoms. Inability to find the addresses of many remote servers resulting in timeouts was the most noticeable. When the connections were made, They were very slow to respond. I didn’t really understand how much slower until after I fixed the problem.

I run my own name server using BIND. I started this soon after I began learning Linux as a way to overcome the horrible name services provided by my series of ISPs. They were very slow and would fail intermittently, always at the most inopportune times for me. It was far less trouble for me to start my own name service and that has been the case — until systemd-resolved forced its way onto my Fedora systems. All of them.

I also found many named errors in the systemd journal. A small sample is shown in Figure 6.

Sep 15 01:41:27 yorktown.both.org named[1464]: SERVFAIL unexpected RCODE resolving 'dns-02.as49870.net/A/IN': 116.203.70.186#53
Sep 15 02:00:14 yorktown.both.org named[1464]: loop detected resolving 'ns6.pinterest.com/A'
Sep 15 02:00:14 yorktown.both.org named[1464]: loop detected resolving 'ns5.pinterest.com/A'
Sep 15 02:24:03 yorktown.both.org named[1464]: REFUSED unexpected RCODE resolving '218.67.58.103.in-addr.arpa/PTR/IN': 103.58.117.2#53
Sep 15 02:34:38 yorktown.both.org named[1464]:   validating in-addr.arpa/SOA: got insecure response; parent indicates it should be secure
Sep 15 02:34:53 yorktown.both.org named[1464]: REFUSED unexpected RCODE resolving '29.140.3.106.in-addr.arpa/PTR/IN': 101.251.253.10#53

Figure 6. A sampling of the name service errors from the systemd journal indicates resolver problems.

Several on-line resources indicate that these errors are caused by configuration issues for the target domain’s name services. The comments on these articles suggest that the domain admins should fix their problems but the commenters recognize that’s unlikely to happen.

We must implement our own changes to fix someone else’s problem. This is not as uncommon as you might hope for SysAdmins. We’re always fixing other people’s problems.

To summarize the root cause of several related resolver problems, much of the progress towards the goal of “Linux on the desktop,” is to make things easier for the end user. This objective has resulted in many changes introduced along with various systemd services that perform a more automatic configuration of the host’s network connection. This has repeatedly introduced new problems into the name resolution process. I’m pretty sure that these problems would cause all but the most technical desktop users to abandon any attempt to use Linux.

Resolving the problem

It takes several steps to resolve this problem. This section describes each step and why it’s needed as part of the complete solution.

1. Stop and disable the Avahi service

The Avahi web site describes Avahi better than I can.

“Avahi is a system which facilitates service discovery on a local network via the mDNS/DNS-SD protocol suite. This enables you to plug your laptop or computer into a network and instantly be able to view other people who you can chat with, find printers to print to or find files being shared. Compatible technology is found in Apple MacOS X (branded “Bonjour” and sometimes ‘Zeroconf’).“

Avahi is the basis for many of the good things that end user simplification can support, however it’s not going to be needed when we disable some of the other services that it supports.

# systemctl disable --now avahi-daemon.service

2. Stop and disable the Avahi daemon

The Avahi daemon socket is a part of the Avahi service. When a program requests Avahi services, it does so through the daemon rather than directly to the service itself. The socket then send the request to the service. Other systemd services also work this way. This won’t be required since we’ve disabled the Avahi service. A socket like this could also cause the service it belongs to to start even though the service is disabled. We don’t want to allow that to happen.

# systemctl disable --now avahi-daemon.socket

3. Stop and disable the systemd-resolved service

The systemd-resolved service is the root cause of the problems we’re having so we disable it. The systemd-resolved man page states its purpose succinctly.

“systemd-resolved is a system service that provides network name resolution to local applications. It implements a caching and validating DNS/DNSSEC stub resolver, as well as an LLMNR and MulticastDNS resolver and responder.“

The man page then proceeds to describe the interfaces it exposes to programs and a high-level statement about how to access it as a resolver. This service is the root cause of the problem and we disable it.

# systemctl disable --now systemd-resolved.service

4. Delete the /etc/resolv.conf link

The NetworkManager service examines /etc/resolv.conf file to determine which servers to use for name resolution. Up to three servers are supported in a simple list format. This file also defines the name of the domain in which to search for host names if a simple hostname is provided, i.e., host, rather than the FQDN (Fully Qualified Domain Name) , i.e., host.example.com.

Only one nameserver is specified in this file; the local host. The systemd-resolved service and Avahi search the local network for other local hosts using systemd-resolved and can configure name resolution so that the hosts can talk amongst themselves. If there is a name server found, such as that provided on wired or wireless routers, it can use that to perform name resolution for external hosts such as www.both.org.

If there’s no locally accessible name server, external name resolution is not possible. This is what happened to me at the beach. The local name server at the hotel was intermittent so no name resolution was possible. I could. however, still ping remote hosts such as www.both.org using the IP address. Yes — this is an edge case. But it clearly does happen.

So we delete the existing /etc/resolve.conf link. We won’t create a new resolv.conf file because once we get the rest of this mess sorted, NetworkManager will create a usable one. The NetworkManager service is responsible for creating the /etc/resolv.conf file at boot time if it doesn’t exist. If systemd-resolved is running, the default link is created, which is not the one we want.

# rm -f /etc/resolv.conf

5. Delete the /etc/nsswitch.conf link

The man page for nsswitch.conf provides a brief description of the uses for this file.

“The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf, is used by the GNU C Library and certain other applications to determine the sources from which to obtain name-service information in a range of categories, and in what order. Each category of information is identified by a database name.“

My testing determined that the /etc/nsswitch.conf file shown at the beginning of this article is directly responsible for the slow resolution speeds I encountered. If you look back at that file, the logic in the hosts line seems to be the cause.

We don’t need — or really want — to delete the actual nsswitch.conf file. We’ll just delete the symbolic link (symlink) in /etc.

# rm -f /etc/nsswitch.conf

5. Create a revised nsswitch.conf

Since we deleted the symlink to this file in the previous step, we need to create a new version of this file, but it won’t be a symlink. After the next step, it won’t be changed or overwritten. I copied the original from /etc/authselect/nsswitch.conf to /etc so that it’s not a symlink. I made my changes to this file. Note that some lines are wrapped. I deleted the first line shown in Figure 5 — since we are going to make changes to this file.

# Do not modify this file manually, use authselect instead. Any user changes will be overwritten.
# You can stop authselect from managing your configuration by calling 'authselect opt-out'.
# See authselect(8) for more details.

# In order of likelihood of use to accelerate lookup.
passwd:     files sss systemd
shadow:     files
group:      files [SUCCESS=merge] sss [SUCCESS=merge] systemd
# hosts:      files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] myhostname dns
hosts:      files myhostname dns
services:   files sss
netgroup:   files sss
automount:  files sss

aliases:    files
ethers:     files
gshadow:    files
networks:   files dns
protocols:  files
publickey:  files
rpc:        files

Figure 7. The revised nsswitch.conf file with the changes shown in bold.

Then I commented out the original hosts line and added a new one with the desired changes. This leaves an easy option for returning to the original configuration.

7. Opt out of authselect

In order to prevent authselect from changing /etc/nsswitch, we opt out.

# authselect opt-out

You can safely ignore the first line of the file and make changes to it manually.

8. Restart NetworkManager

The last step is to restart the NetworkManager service. This will create a new /etc/resolv.conf, and utilize the new nsswitch.conf file we created.

# systemctl restart NetworkManager.service

Every time it’s restarted, whether at Linux startup or a command line restart, NetworkManager creates the new /etc/resolv.conf using the data provided by the DHCP server for the network or from NetworkManager Connection Files. For many stand-alone systems in home and office, this DHCP server is usually the wired/wireless router.

The revised resolv.conf file for my VM is shown in Figure 8 and it contains the information obtained from my DHCP server.

# Generated by NetworkManager
search both.org
nameserver 192.168.0.52
nameserver 8.8.8.8
nameserver 8.8.4.4

Figure 8. The resolv.conf file generated by NetworkManager from the connection files I configured for this interface.

The first name server in the list is my internal name server. The second and third are fallback external name servers. I use Google name servers because I trust them more than my ISP’s whichever ISP I have been using at a given time.

The server at 192.168.0.52 is my internal server. It handles name services for the local network with zone files and uses the top level DNS servers for external network name resolution. If you want to override the network configuration provided by a DHCP server, you can explicitly configure the network interface using NetworkManager Connection Files.

At this point name services are using NSSwitch with a decent and reliable resolv.conf file. I tested this using a few pings to internal and external hosts. I always use the domain example.com for external testing like this, even if it’s only a few pings.

My thoughts about nsswitch

Based on my experimentation, the nsswitch.conf file generated by authselect, and dependence on the Avahi daemon to locate services such as network configuration and other hosts on the local network, slows the entire process to the point of uselessness. I think aiming at Linux on the desktop is an admirable goal. While this may work — once the problems are resolved — for minimally-technical users, its can cause issues for those of us SysAdmins who’ve had things well-configured and working for years.

In previous attempts to fix the resolver problems, I was able to resolve the issues at hand, but after this last round of extreme symptoms, I finally realized the extent of the multiple root causes. Part of the issue is that various systemd name service tools have been added over a period of time. Here we’re able to consider all of the currently known root causes for name service resolution issues related to systemd-resolved and to resolve them.

I hadn’t realized how lengthy the delays in name resolution were until after resolving this problem. Web pages that took minutes to load — and some never did with all the external links they use to load pictures and advertisements — now take only a second or so. Tests using the dig command show name resolution times of around 100 milliseconds (msec) for sites that were not currently in the cache of my name server.

I suggest reading the man pages for each of the files mentioned in this article as there is additional important information about each that can be very helpful.

Summary

Name resolution is a critical service for hosts connected to the Internet, and systemd-resolved is the most current implementation of a resolver for many Linux distributions. Its intended use case is for small networks with no internal name service server, usually because no SysAdmin is available to create a DNS or DHCP server.

While it can work well in that limited use case, it has failed miserably in my use case in which I have both DNS and DHCP services on my internal server. Reverting to the nss-DNS resolver has resolved my name resolution problems.

Don’t misunderstand me. I’m not suddenly saying I hate systemd. That’s not it at all. What I am saying is that the unintended consequences of these decisions can cause SysAdmins pain as they try to determine what’s changed and how to fix it. In this case it’s simply that what’s good for one set of users is not necessarily good for other sets of users. My use case is significantly different from that of non-technical users.

Networks of any size with DHCP or static network configurations are much better suited to nss-DNS name resolution services.

---

1. Both, David, Introduction to Domain Name Services (DNS), https://www.both.org/?p=4759 ↩︎
2. Wikipedia, Hosts file,https://en.wikipedia.org/wiki/Hosts_(file) ↩︎
3. Wikipedia, IP Multicast, https://en.wikipedia.org/wiki/IP_multicast ↩︎
4. Wikipedia, Multicast DNS, https://en.wikipedia.org/wiki/Multicast_DNS ↩︎
5. Both, David, NetworkManager on Linux: #3 — How I migrated to NetworkManager Connection Files for configuration, https://www.both.org/?p=4863 ↩︎
6. GNU.org, System Databases and Name Service Switch, https://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html ↩︎
7. Wikipedia, Oracle Solaris, https://en.wikipedia.org/wiki/Oracle_Solaris ↩︎
8. General C Library. A collection of basic Linux functions that are always present. ↩︎