Keeping time with chrony

“Does anybody really know what time it is? Does anybody really care?”

— Chicago, 1969

Perhaps that rock group didn’t care what time it was but our computers really need to know the exact time. In banking, stock markets, and other financial businesses, transactions must be maintained in the proper order and exact time sequences are critical for that.

NTP is the Network Time Protocol that is used by computers world-wide to synchronize their times with internet standard reference clocks via a hierarchy of NTP servers. Let’s start by exploring NTP and its server hierarchy.

There are two tools that clients can use to access NTP servers to reference and set the system time. This is the operating system time, not hardware time. The older NTP service and the new chrony. Yes, all lowercase. This article is about using chrony as a client.

The chrony project home page describes chrony succinctly.

chrony is a versatile implementation of the Network Time Protocol (NTP). It can synchronise the system clock with NTP servers, reference clocks (e.g. GPS receiver), and manual input using wristwatch and keyboard. It can also operate as an NTPv4 (RFC 5905) server and peer to provide a time service to other computers in the network.

It is designed to perform well in a wide range of conditions, including intermittent network connections, heavily congested networks, changing temperatures (ordinary computer clocks are sensitive to temperature), and systems that do not run continuously, or run on a virtual machine.

Typical accuracy between two machines synchronised over the Internet is within a few milliseconds; on a LAN, accuracy is typically in tens of microseconds. With hardware timestamping, or a hardware reference clock, sub-microsecond accuracy may be possible.

Two programs are included in chrony, chronyd is a daemon that can be started at boot time and chronyc is a command-line interface program which can be used to monitor chronyd’s performance and to change various operating parameters whilst it is running.

— The chrony project home page

NTP client choices

The original NTP daemon, ntpd, has been joined by a newer one, chronyd. Both perform the task of keeping the time of the local host synchronized with the time server. Both services are available and I have seen nothing to indicate that this will change any time soon. In fact, there are use cases that each tool is better suited for than the other.

Chrony has some features which make it the better choice for most environments. Some of the primary advantages of using chrony are shown in this list.

• Chrony can synchronize to the time server much faster than ntp. This is especially good for laptops or desktops that do not run constantly.
• It can compensate for fluctuating clock frequencies such as when a host hibernates, enters a sleep mode, or when the clock speed varies due to frequency stepping that slows clock speeds when loads are low.
• It handles intermittent network connections and bandwidth saturation.
• It adjusts for network delays and latency.
• After the initial time sync is accomplished, Chrony never steps the clock. This ensures stable and consistent time intervals for many system services and applications which require that.
• Chrony can work even without a network connection of any type. In this case the local host or server could be updated manually.

Both the ntp and chrony RPM packages are available from most distribution repositories. It is possible to install both and switch between them but modern releases of Fedora and RHEL have moved from NTP to Chrony as the default time-keeping implementation. I’ve found that Chrony works well, provides a better interface for the SysAdmin, presents much more information, and increases control. I see no reason to use the old NTP service when Chrony is so much better.

The NTP Server Hierarchy

The NTP server hierarchy is built in layers called strata. Each stratum is a layer of NTP servers. The primary servers are at stratum 1 and they are connected directly to various national time services at stratum 0 via satellite, radio, or even modems over phone lines in some cases. Those time services at stratum 0 may be an atomic clock, a radio receiver that is tuned to the signals broadcast by an atomic clock, or a GPS receiver using the highly accurate clock signals broadcast by GPS satellites.

To prevent time requests from time servers lower in the hierarchy, that is with a higher stratum number, from overwhelming the primary reference servers, there are several thousand public NTP stratum 2 and 3 servers that are open and available for all to use. Many users and organizations, myself included, with large numbers of their own hosts that use an NTP server, set up their own time servers so that only one local host actually accesses the stratum 2 time servers. The remaining hosts in our networks are all configured to use the local time server.

So just to make it clear, NTP is a protocol that is implemented with either NTP or Chrony. We will explore only Chrony for both client and server configuration on a Fedora host.

Chrony structure

The Chrony daemon, chronyd, runs in the background and monitors the time and status of the time server specified in the chrony.conf file. If the local time needs to be adjusted, chronyd does so smoothly without the programmatic trauma that would occur if the clock were to be instantly reset to a new time.

Chrony also provides the chronyc tool that allows us to monitor the current status of Chrony and to make changes if necessary. The chronyc utility can be used as a command that accepts sub-commands, or it can be used as an interactive text mode program. We will use it in interactive mode in this article since it’ll save typing. To do so, just enter the chronyc command. Figure 1 shows the chronyc shell. The last line is the chronyc command prompt.

# chronyc
chrony version 4.6.1
Copyright (C) 1997-2003, 2007, 2009-2024 Richard P. Curnow and others
chrony comes with ABSOLUTELY NO WARRANTY.  This is free software, and
you are welcome to redistribute it under certain conditions.  See the
GNU General Public License version 2 for details.

chronyc>

Figure 1. The chronyc shell allows command recall and editing similar to that of the Bash shell.

Client Configuration

The NTP client configuration is simple and requires little or no change for most users. The NTP server can be specified by the SysAdmin after the initial Linux installation or it can be provided by the DHCP server at boot time. The default /etc/chrony.conf file shown in its entirety in Figure 1 requires no alterations to work properly as a client. This file is used for both client and server configuration.

For Fedora, Chrony uses the Fedora NTP pool. CentOS and RHEL also have their own NTP server pools. Like many Red Hat based distributions, the configuration file is well commented.

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (https://www.pool.ntp.org/join.html).
pool 2.fedora.pool.ntp.org iburst

# Use NTP servers from DHCP.
sourcedir /run/chrony-dhcp

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# Allow NTP client access from local network.
#allow 192.168.0.0/16

# Serve time even if not synchronized to a time source.
#local stratum 10

# Require authentication (nts or key option) for all NTP sources.
#authselectmode require

# Specify file containing keys for NTP authentication.
#keyfile /etc/chrony.keys

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Insert/delete leap seconds by slewing instead of stepping.
#leapsecmode slew

# Set the TAI-UTC offset of the system clock.
leapseclist /usr/share/zoneinfo/leap-seconds.list

# Specify directory for log files.
logdir /var/log/chrony

# Select which information is logged.
#log measurements statistics tracking

Figure 2. The default chrony.conf configuration file.

The server pool in Figure 1 contains one meta-server entry for the Fedora NTP pool. There could be up to three meta-servers. Each meta-server, such as 0.fedora.pool.ntp.org, refers to a list of NTP servers whose IP addresses are served in round-robin fashion. Thus, no single server takes the brunt of the incoming requests.

The last line in the chrony.conf file, sourcedir, points to a directory containing one or more files, one for each network interface that has an accessible and active NTP server. Each file is named for the network interface card (NIC) which is configured by DHCP and has an NTP server specified in the DHCP configuration. All files have the .sources extension. The enp0s31f6.sources file shown in Figure 3 is that for my primary workstation which uses DHCP for network configuration for the enp0s31f6 network interface on the motherboard.

server 192.168.0.52 iburst

Figure 3. The contents of the enp0s31f6.sources file defines the NTP server on my network but doesn’t make it a preferred entry.

Exploring with chronyc

Many individual users will never even know that there’s an NTP system on their computers; they’ll probably not even notice that their computer always shows the correct time and will assume it to be the normal state of things.

However, those of us who work with networks, whether modest or complex, will at some point, need to know how to at least observe enough to know whether chrony is working and to determine the source of problems that can occur. Although it’s not usually necessary to intervene in chrony’s normal operation, the chronyc command has several interesting and useful sub-commands that provide information necessary to monitor and manage NTP services.

Figure 3 shows the contents of the enp0s31f6.sources file which defines the NTP server on my network. It contains the correct IP address for my NTP server but it’s not marked as a preferred NTP server. As a result one of the external servers is used, as you can see below. In Figure 4, the sources sub-command displays the list of servers that the NTP client knows about.

chronyc> sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- cooper.nerdfeliz.com          2  10   377   363  +4105us[+4105us] +/-   72ms
^- mail.aptalaska.net            3  10   377   443  -4813us[-4897us] +/-  110ms
^* dutch.arpnet.net              2   8   377   417   +184us[ +100us] +/-   53ms
^+ yorktown.both.org             4   7   377    50   -842us[ -842us] +/-   39ms
chronyc>

Figure 4. The sources command displays the list of NTP servers, informational data, and it marks the currently selected server with a *.

The NTP client selects the server with the shortest response time, in this case, dutch.arpnet.net which is indicated by the asterisk (*). Note that the list of sources and the selected NTP source can change over time. Even when the list of servers doesn’t change, the best time and the selected server can change.

To understand that a little better, Figure 5 is a table with descriptions of the columns in the sources output and their descriptions. This is taken directly from the chronyc manual page. The column identifiers are the first row of the output.

Figure 5. Column descriptions for the sources command.

After observing the results of the sources command over time on several of the hosts in my network, all of them ultimately converge on my local NTP server, yorktown, as their time source. Figure 6 shows this for my primary workstation.

chronyc> sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- 108.61.215.221.vultruser>     3   6    17    19  -2206us[-2206us] +/-   22ms
^- pool-100-34-187-168.phla>     2   6    17    19   -172us[ -172us] +/-   12ms
^+ web02.versadns.com            2   6    17    19  +4141us[+4726us] +/-   17ms
^- 12.167.151.1                  2   6    17    18  -8120us[-8120us] +/-   87ms
^* yorktown.both.org             4   6    17    19  -2102us[-1517us] +/- 8958us
chronyc>

Figure 6. After a period of time, chrony connects with the NTP server with the smallest amount of error.

Note the very last column which is the estimated error for NTP. It’s 8,958us (microseconds) which translates to 8.958ms (milliseconds), thus making it the smallest estimated error of the servers in the list. I’ve watched these types of results over several years when making changes to NTP and chrony configuration. It’s quite clear from my observations that the server with the least amount of error is the one ultimately selected by chrony.

If you forget the meanings of the columns in the sources data, use the -v option with the sources command to see an annotated version of the data.

chronyc> sources -v

 .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
/  .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 108.61.215.221.vultruser>     3   7   377    32  -2452us[-2515us] +/-   22ms
^+ pool-100-34-187-168.phla>     2   7   377    32    -41us[ -104us] +/-   15ms
^+ web02.versadns.com            2   7   377    34  +3997us[+3933us] +/-   18ms
^- 12.167.151.1                  2   7   377    33  -8557us[-8621us] +/-   76ms
^* yorktown.both.org             4   6   377    31  -2293us[-2357us] +/- 9234us
chronyc>

Figure 7. The -v option for the sources command adds short descriptions of the data.

Based on the data in Figure 7, my NTP server, yorktown, has the smallest estimated error, 9.234ms.

Chrony Status

Look at the current status of NTP on your Linux host. This can be done as a regular, non-root user. When used with the tracking sub-command, chronyc provides statistics about that tell us how far off the local system is from the reference server.

chronyc> tracking
Reference ID    : C0A80034 (yorktown.both.org)
Stratum         : 5
Ref time (UTC)  : Thu Feb 20 01:01:35 2025
System time     : 0.000037882 seconds slow of NTP time
Last offset     : +0.000001561 seconds
RMS offset      : 0.000014316 seconds
Frequency       : 3.518 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.013 ppmFigure 11. This command is used to view the systemd journal entries relating to chronyd.
Root delay      : 0.013302401 seconds
Root dispersion : 0.001399564 seconds
Update interval : 64.3 seconds
Leap status     : Normal
chronyc>

Figure 8. The tracking data contains chrony details for the local host.

The Reference ID in the first line of the result is the server to which our host is synchronized. If available, it includes the name of the server, in this case, yorktown is the NTP server on my network. The Stratum for my server is 5, which puts it pretty far down the chain. However, that’s part of the strategy to keep the servers in higher strata, i.e., lower numbers, from bearing the brunt of the load. The number we SysAdmins need to be concerned with is the System time, which shows that the local host is 0.000037882 seconds slow of NTP time.

The rest of these lines are described in the chronyc(1) man page.

Ensure use of local NTP server

Even though the NTP server I’ve set up for my internal network is usually the one selected by chrony because it has the smallest error compared to the pool servers, it sometimes doesn’t, which results in a pool server being the selected source. This can be especially true during the startup of a host, but it can occur any time.

I like to ensure that my own NTP server is always the one used as the source for the rest of the hosts on my network. I also like to have all three of the Fedora NTP server pools as alternates. I modify the top of the /etc/chrony.conf files on my systems to add the two additional pools and the server entry for my own NTP server. All three of those lines are highlighted in Figure 9.

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (https://www.pool.ntp.org/join.html).
pool 1.fedora.pool.ntp.org iburst
pool 2.fedora.pool.ntp.org iburst
pool 3.fedora.pool.ntp.org iburst
server 192.168.0.52 iburst prefer

Figure 9. The modifications made to the /etc/chrony.conf file are highlighted in bold.

After restarting chronyd on the client, the sources data in Figure 10 shows the additional pool servers. The adjusted and measured offsets are also in the nanosecond range.

chronyc> sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^- ovh.maxhost.io                2   6   377    13    +11ms[  +11ms] +/-   66ms
^- tick.chi1.ntfo.org            3   6   377    13  -3707us[-3706us] +/-   96ms
^- time4.lshiy.com               2   6   377    11  +2855us[+2856us] +/-  122ms
^- connected.by.freedominte>     2   6   377    13    +20ms[  +20ms] +/-   70ms
^- clock.xmission.com            1   6   377    12  +3410us[+3411us] +/-   36ms
^- ip229.ip-51-81-226.us         2   6   377    12  +3139us[+3139us] +/-   72ms
^- 74.208.25.46                  2   6   377    12  +1032us[+1032us] +/-   72ms
^- ntpserver-1.walkeritg.org     2   6   175    78  +2900us[+2906us] +/-   17ms
^- 213-88-74-65.gci.net          2   6   377    12   +789us[ +789us] +/-  145ms
^- time.tritan.host              2   6   377    10   -159us[ -159us] +/-   51ms
^- 208.67.75.242                 3   6   377    11  +1289us[+1289us] +/-   92ms
^- ntp1.glypnod.com              2   6   377    12  -3563us[-3563us] +/-   44ms
^* yorktown.both.org             4   6   377    10   -105ns[ +174ns] +/- 8100us
chronyc>

Figure 10. After making the changes to chrony.conf, the local NTP server has the smallest offsets and estimated error.

The tracking command shows significant improvement in the system time difference from NTP time and improved offsets as well. Figure 11 shows these differences when compared to Figure 8.

chronyc> tracking
Reference ID    : C0A80034 (yorktown.both.org)
Stratum         : 5
Ref time (UTC)  : Thu Feb 20 19:27:05 2025
System time     : 0.000000403 seconds fast of NTP time
Last offset     : +0.000000666 seconds
RMS offset      : 0.000011687 seconds
Frequency       : 3.305 ppm slow
Residual freq   : +0.001 ppm
Skew            : 0.019 ppm
Root delay      : 0.013220543 seconds
Root dispersion : 0.001042305 seconds
Update interval : 64.4 seconds
Leap status     : Normal
chronyc>

Figure 11. chronyc tracking data after ensuring that my local NTP server is preferred. Compare to Figure 8.

Why

Here’s the question. Why?

Why does the accuracy of the system time improve so drastically when we use a local NTP server, even if it’s at a lower stratum than a remote NTP server?

The answer is in the way chrony works. The NTP protocol is designed to perform an initial synchronization to a reference clock early in the Linux startup sequence. This initial time sync will be further refined as the system uptime increases. However, switching to different sources can delay that refinement. Dictating that a single source be used ensures that the system clock refinement takes place more quickly and to a greater accuracy.

The command in Figure 12 can be used to view the systemd journal entries relating to chronyd. This will be displayed in seconds since system startup.

# journalctl -b -u chronyd -o short-monotonic

Figure 12. This command is used to view the systemd journal entries relating to chronyd.

On my system, I could see where, prior to making the change to chrony.conf, the source changed multiple times. It did not change after.

Summary

The chrony NTP client is a powerful tool for synchronizing the times of client hosts whether they are all on the local network or scattered around the globe. It is easy to configure because, despite the large number of configuration options available, only a few are required in most circumstances. Even when using a local NTP server, the default settings are fine for most use cases.

Use of a local NTP server can improve system time accuracy which can be important in some time-critical use cases. It also reduces the NTP query load from the public servers. I use a local NTP server to help reduce the load, and I also like having the additional accuracy.

chronyc has more than 50 commands that can be used to manage the system time, although many are for use on the NTP server rather than a client. Most SysAdmins will probably only need a few commands, primarily the ones discussed in this article.

The man pages for chronyd, chronyc, and chrony.conf contain an amazing amount of information that can help you get started or learn about some esoteric configuration options.