One of the keys to effective network management is clock synchronization on all network devices.
When something goes wrong, you need to look through your log messages and figure out important things like which device saw the problem first. This automatically tells you where to start looking for the root cause. If your clocks aren’t synchronized, it becomes much more difficult to correlate log messages between devices.
More generally, you want to know if the similar log messages you’re seeing are related to the same incident or if maybe some of them happened much earlier or later. And, related to this, are you seeing the similar problems at around the same time every day?
The other important reason to synchronize the clocks on your network devices is so they can in turn provide time services to other devices at the same site.
How to synchronize clocks on network devices
There are really two ways to synchronize clocks.
The first is to do it manually by logging into every network device and manually setting the clocks against some common source, like your watch.
This is a terrible solution.
The clock in every device will slip a little bit, probably in random directions, so there could be a difference of several seconds between the fastest and the slowest clocks by the end of the day. Then you’ll need to do it again tomorrow. And it’s incredibly time consuming to log into all of those devices and manually set the clocks.
The second way is to use an automatic time synchronization protocol. There are two in common use.
The one we normally use is Network Time Protocol (NTP). It’s a mature protocol that’s been around for decades and is currently in its fourth version. NTP is accurate to about 1ms, although you can get sub-millisecond accuracy with NTPv4 if you design the network carefully and have access to your own Stratum 1 clocks. (I’ll explain what that means in a moment.)
The other common time synchronization protocol is called Precision Time Protocol (PTP), or IEEE 1588. This protocol is able to synchronize clocks to sub-microsecond accuracy. It is, however, more complicated to deploy and not universally available on network equipment.
For most applications, NTP is sufficient.
People tend to use PTP for applications where a microsecond is a long time. This includes things like industrial and financial (stock trading) applications.
Power grids, in particular, need extraordinary accuracy for ensuring the phases of the 60Hz (or 50Hz in many locations) AC currents are correctly matched. If the phase isn’t matched, power is lost. Lost power is heat. So losing a lot of power means possibly creating a lot of heat. In this environment, unsynchronized clocks could result in fires or damaged equipment.
At 60Hz, one full cycle is about 17ms. If you want to match the phase to within 1%, you need the clocks synchronized to 170 microseconds. This is just within the reach of NTPv4, but not previous versions. And there are other industrial applications where even more accuracy is required.
Why syncing clocks is hard
Accurate clock synchronization presents several interesting problems. The first is latency.
If I send you a packet saying to set your clock to 12:00, you need to know very accurately how long it took for that packet to reach you. You can make a pretty good guess at how long it takes for that packet to get to you by sending the server a packet and making careful note of how long it takes to get the response.
This assumes the path to the server is the same as the path from the server, which is usually a fairly reasonable assumption in most modern networks. It also assumes the server is able to turn around the response instantaneously, which is clearly not all that accurate. So the NTP protocol has been refined by including additional timestamps for when the packets are sent and received at each end.
You also need some statistical information on that network latency so you know how much it varies over time when a packet occasionally needs to be buffered because a link is busy.
Then you need to keep track of something called clock skew. This is a quantitative measurement of how accurate your clock is. Is it a little bit fast or a little bit slow? And does it gradually speed up and slow down over time?
All of these parameters can be estimated by keeping careful statistics on how much the clock needs to be adjusted each time there’s new time information from the server.
The more network hops the packets need to cross on their way from the server to the clock being synchronized, the more chances there are for small random queuing delays. Since we’re collecting statistical information on the delays, the protocol is still able to get a pretty good result. But clearly, the more sources of random delays, the less accurate those statistics will be.
And on top of all these issues, we ideally want to keep the total number of time synchronization packets fairly low to avoid causing congestion either on the network or on the NTP server. So, once it has established a reliable synchronization, NTP very cleverly adjusts the timers to back off the time between requests it sends to the server.
Designing NTP into a network
NTP is a hierarchical protocol. The source clock is called Stratum 1. The clocks synchronized from the original source are Stratum 2. Clocks synchronized to the Stratum 2 clocks are called Stratum 3, and so forth.
A typical NTP network design involves using a pair of Stratum 1 time sources. Using multiple sources improves overall accuracy, as well as providing some redundancy in case one of the clocks is temporarily unreachable.
There are several Stratum 1 clocks available on the public internet, free of charge. Refer to http://ntp.org/ for useful lists of load-balanced public NTP servers.
Generally, you’ll want to use the servers closest to you to minimize network hops. To help with this, the NTP.org site includes lists of geographically sorted, publicly available NTP servers.
You’ll generally want to have a small number of servers inside your network that act as the Stratum 2 servers for your organization. I like to use either the Active Directory domain controllers or some network device located close to the Internet edge, like a firewall or a router.
Again, for reliability reasons, you should have a few Stratum 2 servers. Then you can make all other internal devices synchronize their clocks to these Stratum 2 servers, so that nothing is more than Stratum 3.
In organizations with multiple data centers or multiple geographically diverse internet connections, I like to put at least one of my main Stratum 2 servers in each location.
Note that you should not expose your Stratum 2 devices directly to the internet unless they’re firewalls. There could be bugs in the NTP implementation that might compromise your network security.
If you need particularly accurate clock synchronization, it’s worthwhile investigating a standalone Stratum 1 server. These can be purchased as appliances you install in your data center. Your own Stratum 1 server means you’re not dependent on external resources, and you can control the latency between the devices.
There are generally two types of Stratum 1 devices. The less expensive variety have an antenna and take their clock signal from the GPS satellite network. The more expensive variety have their own internal atomic clocks (usually using a Rubidium oscillator) or some other high accuracy clock. These clocks need to take their initial time signal from some external source.
Cisco NTP configuration
Configuring NTP on Cisco devices is quite simple. You just need to point to the server that you’ll be synchronizing to.
! ntp server 192.168.1.1 !
On ASA firewalls, this configuration is slightly different because you need to specify which interface you’ll be using to reach the NTP server.
! ntp server 192.168.1.1 source management !
In addition to these commands, on several router and switch models, you want to let NTP update the hardware clock, rather than just trying to keep track of the time in software. This provides greater accuracy.
! ntp update-calendar !
For edge devices, you’ll probably want to use a DNS name. This requires that your device be configured to use DNS as well. Then you can specify one of the load-balanced groups mentioned on the NTP.org lists.
! ntp server 0.north-america.pool.ntp.org ntp server 1.north-america.pool.ntp.org !
Please carefully review the restrictions and guidelines on the use of these public pools before you use them.
Other time considerations
At the beginning of this article, I mentioned that one of the important reasons to synchronize clocks on network devices is to allow correlation of log messages. So you’ll also need to make sure your devices include a timestamp in each log message.
! logging enable logging timestamp !
I also like to include the date in my timestamps, so I can easily look at a log message and know whether it came today or weeks ago.
! service timestamps debug datetime msec localtime show-timezone service timestamps log datetime msec localtime show-timezone !
And, of course, you’ll need to set the appropriate time zone. The first of these commands sets the timezone on this device to EST, which is 5 hours west of UTC (London, UK). The second command indicates we’re following normal daylight savings time rules.
! clock timezone EST -5 clock summer-time EDT recurring !
There are several different philosophies on what time zone to set your device clocks to. Some people like to set the time zone on the device to the actual location of that device.
Some like to set the time zone on all devices to a common central zone, usually where the network management staff are located.
A third approach, which makes sense in extremely large networks that span the globe, particularly when the network management staff are located in many different places, is to simply leave the clocks at UTC (-0).