Monitoring and Troubleshooting
Carefully Adjusting Your Clock
Except for discussion of iptables/ipchains firewalling, this discussion applies to Unix systems in general as well as Linux in particular. I have worked with ntpd, also known as xntpd, versions 4.1 and 4.2, using Red Hat 7.3, 9.0 and SuSE 9.1.
The documentation for NTP is voluminous and much of it is devoted to complex issues that may concern an expert setting up groups of time servers for other systems to access. Practical information for the average system administrator can be hard to extract from all of this, so a short focused article aimed at practical setup issues seemed like it would be valuable. The material here reflects some of the author's personal experiences in setting up ntpd on various Linux systems and can hopefully serve as a practical "NTP HOWTO".
This article is probably most useful for someone with system administration experience. The setup steps are relatively simple, but it may be hard to resolve a problem without some background. It is aimed at managers of one or a few Linux systems who want to keep their system clocks close enough to correct time for most practical purposes. This does not attempt to help you set up a time server that other systems can synchronize to; only with setting up NTP clients. Feedback or new information are welcomed by the author.
Even on modern Linux systems, a little NTP client setup is needed. In my experience this little setup does not always go right, so I try to help you do a little bit of debugging. The minimum system configuration is a couple of lines in your ntp configuration file, typically /etc/ntp.conf. You will need a line or two or three like this:
server some.timeserver.com server othertime.server.org
Synchronizing with more than one NTP server gives you redundancy in case of server or network problems, and it may improve the accuracy of your time synchronization. IP addresses are also accepted, but I prefer hostnames, since the IP address of a server may change from time to time.
Note: The version of ntpd released with Red Hat 7.3 (4.1.1) could not handle hostnames in the configuration, only IP addresses. The server would send requests and the remote servers would respond, but the local server would ignore the responses. For older versions of ntpd you may need to use IP addresses instead of names.
First you will need to select an NTP server or two. The implementors of NTP itself provide a very useful list of public NTP servers. You might also wish to try pool.ntp.org, which provides access to a number of different time servers all answering to the same hostname.
Before you synchronize with a time server you will want to check how close your system clock is to the correct time and whether it is ahead or behind the correct time. If your clock is within a minute or two of the correct time you may be happy with the standard simple procedure. If your clock is behind the correct time you are probably still OK, but be aware that your clock may jump forward some time within the first hour after you start your local time server. If your clock is ahead, starting the time server may make your system time jump backward while the system is running. This can have effects like making copies of files look older than the original, and can confuse your system in a number of ways. This tends to affect services that deal with data modification times: database servers, network file systems, distributed file synchronization, some cryptographic protocols, etc.. See Carefully Adjusting Your Clock for a procedure to set your clock back carefully.
The Red Hat and SuSE Linux distributions (and most others) come with scripts to start and stop an NTP client, which runs on your system as a service even if it does not accept requests. When you have configured a couple of NTP servers into your NTP config file, locate your system's start/stop scripts, for example /etc/init.d/, and make sure your NTP daemon is freshly started. The start/stop script might be named ntpd or xntpd, and it normally starts at system startup. On systems such as RedHat or SuSE, the command looks something like:
# /etc/init.d/ntpd restart . . . or # /etc/init.d/xntpd restart
You can start using the "ntpq -p" command within a couple of minutes to monitor your time server and its communication with the remote servers, and do check the system log for messages from ntpd. For more information on these see Monitroing and Troubleshooting below. Note that if the difference between your system's clock and the remote server is more than 128ms you should expect your local ntpd to step your clock forward once it is satisfied it knows the appropriate adjustment. After a half hour or so your system should synchronize with your remote servers and the "date" command should give a very accurate time of day. Once you see your time has been synchronized, use a graphical service manager tool or a command-line tool (run as root) such as chkconfig to make your system start ntpd at system boot time.
Your system should now stay synchronized with your time server(s) indefinitely into the future, and should stay synchronized across shutdowns. Your system will also keep your system BIOS clock adjusted, so unless its battery is dead it will probably be very close to the correct time even before ntpd starts up again. Check your system's time again in a day or so to confirm that things are working properly for you.
NTP is built on the UDP protocol rather than TCP. Though these are both part of the Internet Protocol family, UDP raises different issues with firewalls compared with TCP. Most home and small office firewalls will allow NTP to work without special configuration, but if you are behind any kind of firewall, you may want to do some checking in about an hour from startup rather than waiting to see if your system clock stays synchronized. See the Monitoring and Troubleshooting section.
Linux kernel-level firewalling can also interfere with your clock synchronization. A typical problem is that your system's kernel firewall rejects incoming NTP packets. A simple measure to avoid this is to make sure UDP port 123 is open for incoming packets from your selected time servers. If you are using ipchains, a command to allow NTP packets from one of your time servers looks something like this:
# /sbin/ipchains -A input --proto udp -s onetime.server.com ntp --jump ACCEPT
Iptables supports a similar command, but it is a more intelligent facility and probably also supports more sophisticated approaches. Any information on this would be appreciated.
A most useful command for querying any time server's status is "ntpq -p". You can query a remote time server for the same kind of information with "ntpq -p <hostname>".
# ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== LOCAL(0) LOCAL(0) 10 l 22 64 377 0.000 0.000 0.001 *poolxxx.ntp.org .CDMA. 2 u 118 256 377 21.631 -0.401 0.544 +tick.somnet.net 22.214.171.124 2 u 102 256 377 35.372 -0.264 9.730
The first position in the output lines is a quick indicator of the status of the remote server. A "*" indicates the server to which you are currently synchronized. It may take some part of an hour before your server considers itself synchronized. A "+" is a candidate to take the leading role if the current server becomes unsuitable for some reason.
The labeled columns for this are:
You can easily check if packets are going out from your system and coming back to your box using tcpdump. NTP uses port 123, so you might issue a command like this as root:
# tcpdump udp port 123
Allow it to monitor traffic for up to an hour. You should see packets going out to your time servers port 123 (ntp) and replies coming back from each of them. Your local UDP port will probably also be number 123. Kernel firewalling (ipchains or iptables) can prevent synchronization from working even if the servers are replying to your system's requests.
If you see packets coming in from the remote time servers, but ntpq indicates your server is not seeing them, then be sure to look at your ipchains/iptables configuration and to consider the ntpd 4.1.1 issue noted in the configuration section for Red Hat 7.3 and to check the system log for configuration messages from ntpd.
Your ntpd will log various events through your system's syslog facility. This typically means the messages will show up in /var/log/messages, in lines containing the string "ntpd". Configuration problems are reported here. If your local server steps the time ahead or back suddenly, it will report that here also with a message such as: time reset -6.394626 s.
This procedure is especially useful if your system clock is ahead of the actual time. You can use it to prevent ntpd from making large sudden adjustments to your system clock while the system is running, and to make sure the system clock never jumps backward in time. The NTP FAQ suggests doing the following steps if you can take your server down for a little while:
For systems with clocks ahead of real time, plan to leave the system off until real time catches up with its internal time when it was shut down. Since the system clock is initialized from the BIOS clock, this prevents the system clock from jumping back while your system is running. Recall that Linux can be configured to interpret the BIOS time as either local time or GMT (UTC). You can infer the configuration by comparing output from the "date" command with the BIOS clock time.