Insights Time Synchronization Across Switched Ethernet - Comments

[Svein]

Svein submitted a new PF Insights post

Time Synchronization Across Switched Ethernet

Continue reading the Original PF Insights Post.

 

[mfb]

Nice post.

Millisecond synchronization? Oh, if we had milliseconds in particle physics... OPERA was off by just 60 nanoseconds, and measurements are often done with a precision of a nanosecond.
For nanosecond synchronization, cable lengths are critical, for high resolution PET scanners even an additional centimeter of cable length will reduce the measurement quality if not accounted for properly.

 

[anorlunda]

Good article. It made fun reading. We didn't have to worry about these issues in the days before we started networking computers.

You mention AC phase-angle measurement. Such phase sensors are now installed continent-wide. It is my understanding that they use GPS to generate time stamps at each sensor.

Your article makes me wonder about a number of similar things outside the scope of your article. For example, the time synchronization requirements and methods of things like very-long-baseline interferometry, or the various devices NASA has deployed all over the solar system.

 

[eltodesukane]

Reminds me of a "Temperature vs Time" graph from a weather monitoring station.
I was puzzled in the morning to see the previous night graph, which somewhat looked like the image attached.

Because of a November "daily saving time" change, time went back one hour.
This complicates data analysis, since the same clock time can now have 2 different temperature associated with it.

 

[Svein]

Because of a November "daily saving time" change, time went back one hour.
This complicates data analysis, since the same clock time can now have 2 different temperature associated with it.

That is why you should time stamp data using UTC.

Some anecdotes regarding time (these are copied from https://en.wikipedia.org/wiki/Year_2038_problem, go there for other examples):

• The latest time that can be represented in Unix's signed 32-bit integer time format is 03:14:07 UTC on Tuesday, 19 January 2038 (2,147,483,647 seconds after 1 January 1970).[2] Times beyond that will "wrap around" and be stored internally as a negative number, which these systems will interpret as having occurred on 13 December 1901 rather than 19 January 2038.
• The Network Time Protocol has a related overflow issue, which manifests itself in 2036, rather than 2038. The 64-bit timestamps used by NTP consist of a 32-bit part for seconds and a 32-bit part for fractional second, giving NTP a time scale that rolls over every 2³² seconds (136 years) and a theoretical resolution of 2⁻³² seconds (233 picoseconds). NTP uses an epoch of 1 January 1900. The first rollover occurs in 2036, prior to the UNIX year 2038 problem.

And, of course, the infamous Y2K bug, stemming from the habit of specifying the year using only two digits in early FORTRAN and COBOL programs.

 

[anorlunda]

That is why you should time stamp data using UTC.

True but it is not always so easy. Many activities depend on human behavior, not physics. Things like rush hour traffic patterns, or peaks and valleys of electric power usage. Imagine the chaos if your doctors office made future appointments on UTC.

In my business, we sold things and kept records by the hour and day. Because of daylight savings time, we had to make the software work with 23, 24, and 25 hour days. It gets harder if your business span time-zone and daylight savings borders, it can become challenging. Many times I wished that time keeping was as simple as UTC.

 

[mfb]

Save it in UTC and (if necessary) in local time?
I think most databases do that. Certainly nearly all web pages.

You'll get weird patterns for two nights (no entries in 1 hour, twice the normal entries in the same hour on a different day), but apart from that everything works nicely.

 

[jim mcnamara]

@anorlunda - part of locale settings, available for almost any extant computer, is the timezone setting. This takes handles UTC -> (local time & date) and also the other way: (local time & date) -> UTC. There is also timezone database (was Olson now housed at IANA). It handles all of the politics involved in daylight time transitions and when and if the time changes. Example - The Knesset (Isreali parliament) determines each year when daylight time starts. Th IANA database keeps track of this kind of thing on an ongoing basis.

@Svein You are correct for 32 bit UNIX OS about the end of UNIX time. 64 bit does not have that problem (from Wikipedia):

Most operating systems designed to run on 64-bit hardware already use signed 64-bit time_t integers. Using a signed 64-bit value introduces a new wraparound date that is over twenty times greater than the estimated age of the universe: approximately 292 billion years from now, at 15:30:08 on Sunday, 4 December 292,277,026,596

FWIW - if you remember Y2K there was concern about 16 bit computing and the century rollover.

 

[mfb]

approximately 292 billion years from now, at 15:30:08 on Sunday, 4 December 292,277,026,596

I wonder how they made a prediction for the future introduction of leap seconds for 292 billion years into the future.

 

[Svein]

You are correct for 32 bit UNIX OS about the end of UNIX time.

I remember doing a calculation: 32bit + 32bit - why divide it exact in the middle? If you divide it 24bit (part of second) + 40bit (seconds) you get a span of 34842 years with a resolution of 59.6ns. The span is certainly long enough, but the resolution (which seemed OK around 1999) is possibly too coarse. Splitting the difference: 28bit + 36 bit gives a span of 2177 years with a resolution of 3.7ns...

 

[mfb]

233 picoseconds is not sufficient for the fastest detectors any more.
Design of a 10 picosecond Time of Flight Detector using Avalanche Photodiodes (2009)
Large-Area Picosecond Photo-Detectors Project (report), aiming for 1 ps to 30 ps, depending on the application (30 ps PET gives 1cm spatial resolution)
Oh, and 5 GHz CPUs have a cycle time of 200 picoseconds.

On the other hand, there is no reason why the year of arrival for those detectors should be relevant and stored in the same way as the precision timing information. Global networks like neutrino detectors would profit from that, but those don't reach such a timing resolution (yet?).

 

[Greg Bernhardt]

Interesting Insight!

 

[JohnM]

Since light takes 130mS to travel around the Earth and time signals in Fibre Optic Cables rather longer you need to think at where the correct time is actually defined. The onine information does not seem to be clear on this - is the correct time defined as a shell around the surface of the Earth where the time changes at the same time even tough they cannot communicate this due to the speed of light ? If this is true it would be possible to send a message through the Earth by a hypothetical (Neutrino) communiication system and have it arrive before it was sent at least according to the time stamps.

Astronomers have a problem with timing the arrival of signals at Earth - it takes 16 minutes for light to cross Earth's orbit so signals would appear 16 minutes earlier in Summer than in Winter ! They solve this by referring the timing to a point near the Sun called the Solar System Barycentre which results in a timescale called TBC. Furthur confusion exists as clocks on the Earth run slow due to the gravitational field.

 

[JohnM]

One thing to add when you get to thinking about how far light can travel in a short period of time is that it travels about the length of a 12" plastic ruler in 1 ns. The speed in cables and fiber optic cables is lower than this of course.

 

[Sherwood Botsford]

The idea of "what time is it" can get murky. In the NTP documentation is is made clear "No clock is right" but some are less wrong. Much of the NTP protocol uses stats and long term checks to gets progressively better agreement over what time it is.

World wide time overall theoretically based on some standard clock. (I think the Washington Naval Observatory is the usual one accepted, with Fort Collins Colorado as secondary) Fort Collins has most of the transmitters for broadcast time.

Even though the Earth is a non-inertial frame, the distance between two points isn't changing. (You, the geologist in the back corner: Stop muttering about continental drift...) So allowing for for this it's theoretically possible to synchronize clocks to some arbitrary tolerance.

The GPS system has an atomic clock in each satellite. Getting a fix requires 4 satellites. 3 for space and one for time in a set of I'm sure are rather messy simultaneous equations in spherical geometry. Anyway, any GPS receiver should be able to determine time to a microsecond.

Your cell phone tower has to do a bunch of time share multiplexing. I recall they use timing off of the GPS system to keep synchronized.

 

[Nik_2213]

When I wrote a 'general procedure' at work that swept up our various timing methods' calibration, precision, accuracy and gotchas, I included a caution about the unknown network latency of PCs. Though this was not noticed by my reviewer, it caused consternation at the next audit.

Given we were only timing stuff to plus or minus a second in half an hour, considering such latency was total overkill.
But, better to have it out in the open...
;-)

 

[Klystron]

Precise synchronization in a local area was an issue before networked digital computers. Take a typical RADAR installation circa 1970's. The transmitter and receiver share wave-guide, antenna, and feed-horn. For detection purposes the transmitter -- a magnetron or klystron -- frequency drifts within operating limits synchronized using a stable local oscillator (STALO). Add a tuned maser a/o parametric amplifier to the receiver to improve signal-to-noise-ratio (SNR).

What if the RADAR transmission was also used as an RF beacon expected at a precise frequency? We were able to compensate for the frequency drift of the transmitter by correcting the STALO referencing the parametric amplifier. I remember being able to improve lobe energy and reduce frequency jitter in the transmitter by tweaking the cavity of the parametric amp in the receiver. Crude methods for modern times but effective. (see "RADAR mile" in Wiki).

Later as a computer scientist helping with data collection at NASA Ames wind tunnels I remember using "official" time signals from the Naval Observatory compared to internal clocks on a VAX computer to improve the average time-stamps associated with data collected on PDP front ends. As previously mentioned the time-stamps themselves are collected along with the actual data of interest along with the NTP output. Still crude compared to particle physics but amenable to analysis.