- Use a special hardware signal on a separate cable between the controller and all nodes that need synchronization. If the nodes are far apart and the tolerances are tight, make sure that all cables that carry the synchronization signal have the same length.
- Add a local clock to each node and use the present automation network to keep them in synchronization. Tell each node how often the measurement sources should be sampled and require the node to time stamp each measurement.
We shall now take a look at hard synchronization requirements (maximum deviation 1ms) and discuss the possibility of implementing synchronization in a multi-traffic switched Ethernet environment. Common for both solutions is that they adhere to the same standardized time protocol. Such a step would significantly reduce the cabling and transceiver cost, since costly dedicated (separate) links are used for this purpose today.
The Concept of Time Stamping
Let us start at the very beginning – the concept of time stamping:
Time stamping is the association of a data set with a time value. In this context, “time” may also include “date”.
Why would anybody want to time stamp anything? The closest example may be on your own PC – whenever you create a document and save it, the document is automatically assigned a data-and-time value. This value enables you to look for
- Documents created on a certain date (for example last Monday)
- Documents created within a certain time span (for example last half of 1998)
- The order in which a set of documents were created (for example the e-mails in your inbox).
If we just look at the examples above, we see that the accuracy we need is for the time stamping is about the same as we expect from our trusty old wristwatch. This again means “within a couple of minutes”, but as long as the clock does not stop it does not really matter much how precise it is.
Let Us Synchronize Our Watches!
Now we know about time stamping on our own PC. The next step is to connect the PC to a network, maybe even to the Internet and start exchanging documents and e-mails. What happens if the clock in your PC (the clock that is used for time stamping) is wrong by a significant amount?
- If you have an e-mail correspondence with someone, a reply (which is time stamped at the other end) might appear to be written before the question (which is time stamped at your end)
- If you collaborate on some documents, getting the latest version might be problematical.
Therefore, when several PCs are connected together in any sort of network, the PC clocks are still accurate enough, but a new requirement is that they should be synchronized (show the same time at one point in time). Now, we could go around to each PC, look at our wristwatch and set the PC clock to agree with it. The trouble is that this is a boring and time-consuming job and we should look for a better solution.
One solution is to elect one PC to be the “time reference”, which means that every other PC should get the current time from it at least once a day and set its own clock to agree with that time. This solution works satisfactorily on a local area network (LAN), but all PC clocks will lag the time reference by the time it takes a clock value to travel from the time reference to the synchronizing PC. Except for very unusual cases, this lag is less than one second and thus good enough for office purposes.
Enter the Internet. Suddenly the synchronization problem escalates, since two collaborating PCs may be located in different time zones (remember to compensate for that) and a synchronization message may take a long time from one PC to the other. Fortunately, the Internet Network Time Protocol has a solution to both problems. This protocol involves sending a time stamped time request message to a “timeserver”. This timeserver adds an arrival time stamp and a retransmit time stamp before returning the request message to the requesting PC. The requesting PC time stamps the message when it returns and uses all the time stamps in calculating the correct time. This protocol and its little brother, the Simple Network Time Protocol, are able to synchronize computers across the Internet with a precision in the low milliseconds.
Stating the Problem – why Network Synchronization is Difficult
The delays from the time stamping of a time synchronization message in the message source node until it is time stamped in the message destination node are:
- Message preparation delay
- Communication stack traversal delay (transmission)
- Network access delay
- Network traversal delay
- Communication stack traversal delay (reception)
- Message handling delay
Variations in the delays are due to:
- Real Time Operating System (RTOS) scheduling unpredictability
- Network access unpredictability
- Network transversal time variations
Time stamping at the lowest stack level helps eliminate the stack delay variations and real-time OS scheduling unpredictability but introduces some complications in the implementation.
An NTP Time Protocol Implementation
The NTP/SNTP algorithm is based on a time client asking a time server for the current time. To do so, the client creates an NTP network packet and inserts its current time in it. The time server logs the time the packet arrives, processes it as fast as possible, and transmits the packet back to the time client, adding a time stamp just before the backet is transmitted. What we now have is a network packet containing three time stamps:
- t1: The (client) time the packet was generated in the client asking for the current time
- t2: The (server) time the packet arrived at the timeserver.
- t3: The (server) time the packet was updated and put into the transmission queue at the server.
In addition, the calculations require:
- t4: The (client) time the packet arrived back at the client.
From these four time stamps we can calculate the best estimate for the difference between the Time Server clock and the time client clock: Δt=(t2+t3)/2 – (t1+t4)/2 . We can also calculate an estimate for the time the message uses between the client and the server: τ=(t4 – t3 + t2 – t1)/2.
Now, t2 and t4 can easily be determined down to microsecond (and perhaps even better) using hardware or software time stamps based on packet arrival interrupts. The other two have definite problems, however.
For full accuracy, t1 and t3 should have been the time when the packet left the time client or timeserver respectively. The problem is that this time stamp is not available until the packet has really left the timeserver or client and then it is, of course, too late to incorporate it into the packet. Therefore the time synchronization inaccuracy for an NTP/SNTP setup is the variation in the delay between t1 and the time the packet leaves the time client plus is the variation in the delay between t3 and the time the packet leaves the timeserver.
Time client implementation issues
There are several ways of time stamping a network packet. We shall look at three of them and show that only the two first ones are suitable for accurate time synchronization:
- Hardware time stamping in the Ethernet controller.
- Software time stamping in an Interrupt Service Routine (ISR) outside the Real Time Operating System (RTOS). This ISR should be connected to the Ethernet Interrupt Request signal and have a top hardware priority.
- Software time stamping in an Interrupt Service Routine (ISR) controlled by the RTOS (usually inside the Ethernet driver). This ISR is connected to the Ethernet Interrupt Request signal with a normal hardware priority.
Using any of this low-level time stamping methods is considered an implementation issue and will not cause any incompatibly between a low-level time stamping client and a standard high-level time stamping server. In addition to low-level time stamping the time client must consider the following aspects:
- The interval between time updates.
- The specifications of the local time-of-day clock with respect to resolution, accuracy/stability, and the availability of drift and offset correction mechanisms.
- The usage of adaptive filtering and time stamp validation methods in order to remove network delay variations.
In NTP the time client is the active part – it has to decide whether or not to synchronize and send out a synchronization request whenever it deems necessary. In the measurement and automation world, the clients are usually simple devices and are not relied upon to handle such tasks by themselves. The new time synchronization standard IEEE 1588 moves the responsibility for the synchronization to the time master, which is more in line with the traditional automation approach.
How IEEE 1588 Works
When an IEEE 1588 time server wants to synchronize the client clocks on a network, it broadcasts (or multicasts) the current time on the network for every client to pick up. Now each client will have the current time with an error dependent on the time it takes for the time message to pass from the time master across the network to the slave. A more advanced time server will create a time stamp at the actual time the time message has been successfully sent onto the network and then send out a “follow-up” message containing the time stamp in the original message and the new, corrected time stamp. This allows the clients to correct for the time server protocol stack delay and network access delay. It is also possible to ask the time clients to send their current time back to the time server, which allows the time server to calculate the protocol stack delay in the client and inform the client about it.
Some Properties of IEEE Ethernet1588
- The “time server” and “time client” roles in an IEEE 1588 network are not fixed, but are a result of a continuous evaluation of the time messages.
- It is possible for an IEEE 1588 device to be physically connected to more than one network. Such a device is designated a “boundary clock”.
- A boundary clock may have different roles on the different network interfaces. A common combination is to be a slave on one network and master on the other networks.
- Since most routers are unable to handle multicasts, a boundary clock is the standard way for IEEE 1588 time information to pass subnet borders.
Comparing NTP and IEEE1588
- Both NTP and IEEE 1588 attain maximum accuracy on hub-based networks. Both standards have problems with the jitter inherent in switch-based networks. Both standards must implement corrections for this, since 100Mbit/s Ethernet is the last version of Ethernet that will be able to use a hub.
- In a switch-based network, NTP measures the network delay in both directions every time, IEEE 1588 does it only when the time server deems it necessary. Thus, NTP has more measurements on which to base corrections.
- The clock synchronization accuracy will be appreciably better for an IEEE 1588 network where “follow-up” messages have been implemented. Without “follow-up” messages, the accuracy will be about the same for both protocols.
- IEEE 1588 mentions NTP as one of the possible standards for external clock synchronization.
- IEEE 1588 and NTP represent time in different ways and use different time origins.
- NTP contains chapters on local clock implementation and local clock adjustment. IEEE 1588 ignores these topics totally.
Ethernet infrastructure implementation issues
In a local measurement or automation network only one switch should preferably be allowed between a time client and a time server. Having multiple switch levels will impose increased jitter through the infrastructure, which again might call for more complex filtering at the time client side. The Ethernet switch must also have good switch latency characteristics.
What kind of accuracy do we need?
As usual, the answer is: It depends. What do you need the measurements for and how fast does the measured signal change. If you want to measure temperature and flow rate in a chemical factory, a precision of 10ms will usually be more than adequate. If you want to measure phase information on two 800kV high voltage transmission lines, your measurement stations will usually be far apart (several 100m at least) and the measured voltage will change very rapidly (at 60Hz an 800kV line will change 426V/μs when it passes through 0V).
As our ambitions grow, our measurements need to be more precise. Precision is not only a question of the value of a measurement, but also of precisely when the measurement is taken. This is especially important when we want to measure different physical entities and correlate the measurements afterwards.
 Jitter: variations in the delay