Learn Time Synchronization Across Switched Ethernets

Now and then you come across measurement problems that are tightly associated with the notion of synchronicity, meaning that things need to happen simultaneously. The usual things that need such synchronicity are data sampling and motion control. In the case of data sampling, you need to know the value of two different quantities measured at the same time (within a narrow tolerance). If the measurement sources are close together, this is fairly easy to accomplish, but if they are far apart and connected to different measurement nodes, it suddenly gets harder. The usual choices are:

1. Use a special hardware signal on a separate cable between the controller and all nodes that need synchronization. If the nodes are far apart and the tolerances are tight, make sure that all cables that carry the synchronization signal have the same length.
2. Add a local clock to each node and use the present automation network to keep them in synchronization. Tell each node how often the measurement sources should be sampled and require the node to timestamp each measurement.

We shall now take a look at hard synchronization requirements (maximum deviation 1ms) and discuss the possibility of implementing synchronization in a multi-traffic switched Ethernet environment. Common for both solutions is that they adhere to the same standardized time protocol. Such a step would significantly reduce the cabling and transceiver cost, since costly dedicated (separate) links are used for this purpose today.

The Concept of Time Stamping

Let us start at the very beginning – the concept of time stamping:

Timestamping is the association of a data set with a time value. In this context, “time” may also include “date”.

Why would anybody want to time stamp anything? The closest example may be on your own PC – whenever you create a document and save it, the document is automatically assigned a data-and-time value. This value enables you to look for

• Documents created on a certain date (for example last Monday)
• Documents created within a certain time span (for example last half of 1998)
• The order in which a set of documents were created (for example the e-mails in your inbox).

If we just look at the examples above, we see that the accuracy we need is for the time-stamping is about the same as we expect from our trusty old wristwatch. This again means “within a couple of minutes”, but as long as the clock does not stop it does not really matter much how precise it is.

Let Us Synchronize Our Watches!

Now we know about time stamping on our own PC. The next step is to connect the PC to a network, maybe even to the Internet, and start exchanging documents and e-mails. What happens if the clock in your PC (the clock that is used for time stamping) is wrong by a significant amount?

• If you have an e-mail correspondence with someone, a reply (which is time-stamped at the other end) might appear to be written before the question (which is time stamped at your end)
• If you collaborate on some documents, getting the latest version might be problematic.

Therefore, when several PCs are connected together in any sort of network, the PC clocks are still accurate enough, but a new requirement is that they should be synchronized (show the same time at one point in time). Now, we could go around to each PC, look at our wristwatch, and set the PC clock to agree with it. The trouble is that this is a boring and time-consuming job and we should look for a better solution.

One solution is to elect one PC to be the “time reference”, which means that every other PC should get the current time from it at least once a day and set its own clock to agree with that time. This solution works satisfactorily on a local area network (LAN), but all PC clocks will lag the time reference by the time it takes a clock value to travel from the time reference to the synchronizing PC. Except for very unusual cases, this lag is less than one second and thus good enough for office purposes.

Enter the Internet. Suddenly the synchronization problem escalates since two collaborating PCs may be located in different time zones (remember to compensate for that) and a synchronization message may take a long time from one PC to the other. Fortunately, the Internet Network Time Protocol has a solution to both problems. This protocol involves sending a time-stamped time request message to a “timeserver”. This timeserver adds an arrival time stamp and a retransmit time stamp before returning the request message to the requesting PC. The requesting PC time stamps the message when it returns and uses all the timestamps in calculating the correct time. This protocol and its little brother, the Simple Network Time Protocol, are able to synchronize computers across the Internet with precision in the low milliseconds.

Stating the Problem – why Network Synchronization is Difficult

The delays from the time stamping of a time synchronization message in the message source node until it is time-stamped in the message destination node are:

• Message preparation delay
• Communication stack traversal delay (transmission)
• Network access delay
• Network traversal delay
• Communication stack traversal delay (reception)
• Message handling delay

Variations in the delays are due to:

• Real-Time Operating System (RTOS) scheduling unpredictability
• Network access unpredictability
• Network transversal time variations

Time stamping at the lowest stack level helps eliminate the stack delay variations and real-time OS scheduling unpredictability but introduces some complications in the implementation.

An NTP Time Protocol Implementation

The NTP/SNTP algorithm is based on a time client asking a time server for the current time. To do so, the client creates an NTP network packet and inserts its current time in it. The time server logs the time the packet arrives, processes it as fast as possible, and transmits the packet back to the time client, adding a timestamp just before the packet is transmitted. What we now have is a network packet containing three-time stamps:

• t₁: The (client) time the packet was generated in the client asking for the current time
• t₂: The (server) time the packet arrived at the timeserver.
• t₃: The (server) time the packet was updated and put into the transmission queue at the server.

In addition, the calculations require:

• t₄: The (client) time the packet arrived back at the client.

From these four timestamps we can calculate the best estimate for the difference between the Time Server clock and the time client clock: Δt=(t₂+t₃)/2 – (t₁+t₄)/2. We can also calculate an estimate for the time the message uses between the client and the server: τ=(t₄ – t₃ + t₂ – t₁)/2.

Now, t₂ and t₄ can easily be determined down to microsecond (and perhaps even better) using hardware or software timestamps based on packet arrival interrupts. The other two have definite problems, however.

For full accuracy, t₁ and t₃ should have been the time when the packet left the time client or timeserver respectively. The problem is that this timestamp is not available until the packet has really left the time server or client and then it is, of course, too late to incorporate it into the packet. Therefore the time synchronization inaccuracy for an NTP/SNTP setup is the variation in the delay between t₁ and the time the packet leaves the time client plus is the variation in the delay between t₃ and the time the packet leaves the timeserver.

Time client implementation issues

There are several ways of time stamping a network packet. We shall look at three of them and show that only the two first ones are suitable for accurate time synchronization:

1. Hardware time stamping in the Ethernet controller.
2. Software time stamping in an Interrupt Service Routine (ISR) outside the Real-Time Operating System (RTOS). This ISR should be connected to the Ethernet Interrupt Request signal and have a top hardware priority.
3. Software time stamping in an Interrupt Service Routine (ISR) controlled by the RTOS (usually inside the Ethernet driver). This ISR is connected to the Ethernet Interrupt Request signal with a normal hardware priority.

Using any of these low-level time-stamping methods is considered an implementation issue and will not cause any incompatibility between a low-level time-stamping client and a standard high-level time stamping server. In addition to low-level time stamping the time client must consider the following aspects:

• The interval between time updates.
• The specifications of the local time-of-day clock with respect to resolution, accuracy/stability, and the availability of drift and offset correction mechanisms.
• The usage of adaptive filtering and time stamp validation methods in order to remove network delay variations.

A New Time Synchronization Standard – IEEE 1588

In NTP the time client is the active part – it has to decide whether or not to synchronize and send out a synchronization request whenever it deems necessary. In the measurement and automation world, the clients are usually simple devices and are not relied upon to handle such tasks by themselves. The new time synchronization standard IEEE 1588 moves the responsibility for the synchronization to the time master, which is more in line with the traditional automation approach.

How IEEE 1588 Works

When an IEEE 1588 time server wants to synchronize the client clocks on a network, it broadcasts (or multicasts) the current time on the network for every client to pick up. Now each client will have the current time with an error dependent on the time it takes for the time message to pass from the time master across the network to the slave. A more advanced time server will create a timestamp at the actual time the time message has been successfully sent onto the network and then send out a “follow-up” message containing the time stamp in the original message and the new, corrected time stamp. This allows the clients to correct for the time server protocol stack delay and network access delay. It is also possible to ask the time clients to send their current time back to the time server, which allows the time server to calculate the protocol stack delay in the client and inform the client about it.

Some Properties of IEEE Ethernet1588

• The “time server” and “time client” roles in an IEEE 1588 network are not fixed but are a result of a continuous evaluation of the time messages.
• It is possible for an IEEE 1588 device to be physically connected to more than one network. Such a device is designated a “boundary clock”.
• A boundary clock may have different roles on the different network interfaces. A common combination is to be a slave on one network and a master on the other networks.
• Since most routers are unable to handle multicasts, a boundary clock is the standard way for IEEE 1588 time information to pass subnet borders.

Comparing NTP and IEEE1588

• Both NTP and IEEE 1588 attain maximum accuracy on hub-based networks. Both standards have problems with the jitter inherent in switch-based networks. Both standards must implement corrections for this since 100Mbit/s Ethernet is the last version of Ethernet that will be able to use a hub.
• In a switch-based network, NTP measures the network delay in both directions every time, IEEE 1588 does it only when the time server deems it necessary. Thus, NTP has more measurements on which to base corrections.
• The clock synchronization accuracy will be appreciably better for an IEEE 1588 network where “follow-up” messages have been implemented. Without “follow-up” messages, the accuracy will be about the same for both protocols.
• IEEE 1588 mentions NTP as one of the possible standards for external clock synchronization.
• IEEE 1588 and NTP represent time in different ways and use different time origins.
• NTP contains chapters on local clock implementation and local clock adjustment. IEEE 1588 ignores these topics totally.

Ethernet infrastructure implementation issues

In a local measurement or automation network, only one switch should preferably be allowed between a time client and a time server. Having multiple switch levels will impose increased jitter[1] through the infrastructure, which again might call for more complex filtering at the time client-side. The Ethernet switch must also have good switch latency characteristics.

What kind of accuracy do we need?

As usual, the answer is: It depends. What do you need the measurements for and how fast does the measured signal change. If you want to measure temperature and flow rate in a chemical factory, a precision of 10ms will usually be more than adequate. If you want to measure phase information on two 800kV high voltage transmission lines, your measurement stations will usually be far apart (several 100m at least) and the measured voltage will change very rapidly (at 60Hz an 800kV line will change 426V/μs when it passes through 0V).

As our ambitions grow, our measurements need to be more precise. Precision is not only a question of the value of a measurement but also of precisely when the measurement is taken. This is especially important when we want to measure different physical entities and correlate the measurements afterward.

[1] Jitter: variations in the delay

Read my next article https://www.physicsforums.com/insights/administering-tcpip-automation-measurement-networks/