The Impact of Negative Leap Seconds: Risks and Challenges for IT Systems

[anorlunda]

Conceptually, negative leap seconds are not much different than positive leap seconds. However, many or most IT systems probably have no provision for negative leap seconds and no testing for that event. Add to our list of time/date headaches; this one not Y2K related.

https://www.timeanddate.com/news/as...ne 29, Earth set,1.50 milliseconds on July 26.

If Earth’s fast rotation continues, it could lead to the introduction of the first-ever negative leap second.

What is causing the current downward trend in the length of the shortest day?

It could be related to processes in Earth’s inner or outer layers, oceans, tides, or even climate. Scientists are not sure, and struggle to make predictions about the length of day more than a year ahead. But there are tentative ideas.

At next week’s annual meeting of the Asia Oceania Geosciences Society (presentation SE05_A009), Leonid Zotov—together with his colleagues Christian Bizouard and Nikolay Sidorenkov—will suggest the current decrease in the length of day could have some relation to the ‘Chandler wobble’.

 

[Vanadium 50]

I am not a fan of leap seconds. So astronomical noon isn't the same as chronological noon? This is true for 3599/3600-ths of the planet ever since we set up Time Zones. Let it go for a couple millennia and we'll fix it when the next daylight savings time change comes around.

However, I don't see this as a problem. NTP can handle this, and most of our computers, whether we know it or not, get their time from NTP. They can even be a few seconds - or more - off when first connected to the network, and they quickly settled down.

If it is critically important that a computer keep time to better than a second, somebody is being paid to do so, and that person knows thge difference between UTC and UT1. (I don't, for sure)

 

[jedishrfu]

Time going backward can be a disaster on some computer systems. We had one example while drawing a time vs db level for various frequencies plot which looked similar to a stock price chart. One value had a bad time which caused a zig zag backward and then forward again.

Unable to fix the externally generated data, we tossed it when time went backwards relative to the previous value plotted. It was an intermittent problem so we didn't dwell on it even though we didn't like tossing data.

 

[anorlunda]

However, I don't see this as a problem.

I'm a fan of the school that says software that hasn't been tested doesn't work.

The problem is not figuring out what the time is. It is taking care of the consequences of minutes/hours/days with more or less than the standard number of seconds, and of skipping some intervals and doubling some intervals.

The problem is also dealing with bean counters. They take pride in tracking money down the the last penny. (Would you have it any other way?).

Then there are the impossible to predict logical problems, such as software that does stuff once per second, doing it a second time for the same second. (Whew, three meanings of second in the same sentence. ) Leaps cause exactly the same kinds of problems we feared in Y2K. The size of the leap, seconds or years does not change the kinds of logical errors. If you want an example, think of a software safeguard that sends fraud alerts if it detects double billing. It takes real money and real labor to deal with petty annoyances.

 

[jedishrfu]

Lets not forget the leap rollover when the second counter exceeds the max unsigned value and rolls over to zero.

 

[jedishrfu]

And 2038 is coming up shortly:

https://en.wikipedia.org/wiki/Year_2038_problem

 

[Bystander]

Bookmarked this one; it's got a number of implications for a number of "forbidden" topics.

 

[anorlunda]

Bookmarked this one; it's got a number of implications for a number of "forbidden" topics.

Huh? I don't get that.

 

[Bystander]

Huh? I don't get that.

GW is forbidden, is it not?

 

[anorlunda]

GW is forbidden, is it not?

Sorry, I must be slow tonight. What is GW?

 

[Bystander]

What is GW?

"Global Warming."

 

[Vanadium 50]

I'm a fan of the school that says software that hasn't been tested doesn't work.

It is tested.

When you turn a Llinux - I have no idea what Windows does, other than that is surely confusing, overcomplicated and might not work very well) PC on after being shut off overnight, the real-time clock has drifted a few seconds off the real time. Sometimes ahead, sometimes behind. A few minutes (technically, a few times 64 seconds later) the system time syncs with NTP time. And as I said, sometimes this is forward and sometimes its back.

So this is not really new for "personal" computers.

Big iron? Well, 370's run Linux now, but other computers? Well, if you are running a zillion dollar enterprise and its critical that you don't ever, ever roll back, you probably should be using UT1 or a similar standard anyway. And paying someone who understands all the nuances to make sure this isn't a problem. For the rest of us, POSIX has a monotonic clock (which can take days to catch up if the clock is hours off). If someone needs it, the OS has it, and he doesn't use it, who is it on?

 

[jedishrfu]

Airgapped systems can run into this time drift issue.

 

[anorlunda]

By chance, I just encountered some press reports of real world examples.

https://engineering.fb.com/2022/07/...ts-time-to-leave-the-leap-second-in-the-past/
Back in 2012, Reddit experienced a massive outage because of a leap second; the site was inaccessible for 30 to 40 minutes. This happened when the time change confused the high-resolution timer (hrtimer), sparking hyperactivity on the servers, which locked up the machines’ CPUs.

In 2017, Cloudflare posted a very https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ about the impact of a leap second on the company’s public DNS. The root cause of the bug that affected their DNS service was the belief that time cannot go backward. The code took the upstream time values and fed them to Go’s rand.Int63n() function. The rand.Int63n() function promptly panicked because the argument was negative, which caused the DNS server to fail.

Leap second events have caused issues across the industry and continue to present many risks. As an industry, we bump into problems whenever a leap second is introduced. And because it’s such a rare event, it devastates the community every time it happens. With a growing demand for clock precision across all industries, the leap second is now causing more damage than good, resulting in disturbances and outages.

As engineers at Meta, we are supporting a larger community push to stop the future introduction of leap seconds and remain at the current level of 27, which we believe will be enough for the next millennium.

https://www.wired.com/2012/07/leap-second-glitch-explained/​

The Inside Story of the Extra Second That Crashed the Web​

The "leap second" crash -- which hit several web operations on Saturday evening -- can be traced to a single glitch in the Linux operating system. Here's the inside story on what happened.

 

[pbuk]

By chance, I just encountered some press reports of real world examples.

Why do you think it is by chance?

 

[Vanadium 50]

That was 2012. We've had two leap seconds since then. How did they go'?

 

[pbuk]

Badly: the Cloudflare one was 1 Jan 2017 (they must have got away with it 18 months before).

 

[anorlunda]

Why do you think it is by chance?

Because of this:

http://catless.ncl.ac.uk/Risks/33/35#subj35​

Tech giants, including Meta, Google, and Amazon, want to put an end to leap-seconds (ZDNet)​

Gabe Goldberg <gabe@gabegold.com>
Wed, 27 Jul 2022 20:49:25 -0400

[Body omitted]

Therefore, Meta concludes, we should simply "stop the future introduction of
leap seconds."

https://www.zdnet.com/home-and-office/networking/tech-giants-want-to-put-an-end-to-leap-seconds/

 

[pbuk]

I meant why do you think it is by chance that you have just seen this rather than by observation by some AI of your recent reading interests?

 

[anorlunda]

I meant why do you think it is by chance that you have just seen this rather than by observation by some AI of your recent reading interests?

I'm not sure I understand the question.

• The OP was triggered by the news of the first negative leap second. It showed up on my Google News Feed, which is not AI specific. Google first pointed me to a NY Post popular article.
• I read the ACM Risks Forum as regularly as I read PF. It does not focus on AI, but rather on risks. That post in the risks forum came out August 1.

I see nothing other than a coincidence of timing of those two.

 

[Vanadium 50]

After some more thought:

1. Leap seconds solves a non=problem. Who cares if the place where clock noon is the same as astronomical noon moves by 1500 feet?

2. Negative leap seconds solve even more of a non-problem, since there is an easy solution - wait. Do nothing for a year or three and

3, The present system confuses the difference between time labeling and time keeping. Time keeping is knowing that it is x seconds after a reference time. Time labeling tells us that x corresponds to March 17, 1995 at 5:21 exactly.

3, This suggests this is primarily a failure of analysis, not programming. Of course,m there can be bugs too, There can always be bugs.

Negative time adjustments can and are handled every single day, millions of times, without problems. I can understand a company not wanting to spend money on doing this right. But it's not an impossibility. The fact that this gets rediscovered, is disturbing.

4. Having this happen at any time other than the daylight/summer transitions is just asking for trouble.

 

[pbuk]

1. Leap seconds solves a non=problem. Who cares if the place where clock noon is the same as astronomical noon moves by 1500 feet?

Indeed: I don't know anyone that understands the issues that thinks differently.

2. Negative leap seconds solve even more of a non-problem, since there is an easy solution - wait. Do nothing for a year or three and

Probably, although as we are not sure why the Earth's rotation has speeded up we can't be sure it will slow down again.

3, The present system confuses the difference between time labeling and time keeping. Time keeping is knowing that it is x seconds after a reference time. Time labeling tells us that x corresponds to March 17, 1995 at 5:21 exactly.

Agreed.

3, This suggests this is primarily a failure of analysis, not programming. Of course,m there can be bugs too, There can always be bugs.

Kind of, although if you look at the history you can see how we got to where we are now with incrementally sensible analytical decisions. Or do you mean improper handling of leap seconds by current systems? I think this is a failure at every stage - the analyst should have specified how to handle negative time increments, the programmer should have worked out what to do with them and the tester should have tested it didn't catch fire.

Negative time adjustments can and are handled every single day, millions of times, without problems. I can understand a company not wanting to spend money on doing this right. But it's not an impossibility. The fact that this gets rediscovered, is disturbing.

Agreed: Cloudflare in particular should not have got this wrong.

4. Having this happen at any time other than the daylight/summer transitions is just asking for trouble.

yeah, adjusting UTC at different times depending on the local transition time for daylight saving (if any) would be a lot of fun.

 

[anorlunda]

Who cares if the place where clock noon is the same as astronomical noon moves by 1500 feet?

GPS, among many other things. We are talking about leap seconds applied to Coordinated Universal Time upon which NTP and all other time standards are based. I can't imagine how many long term scientific experiments use UTC, and which may include the unstated assumption that time never moves backward. (Remember when CERN reported FTL neutrinos? That was not an IT error, but it illustrates sensitivity to elapsed times.)

Wikipedia says:

https://en.wikipedia.org/wiki/Coordinated_Universal_Time
The system has been adjusted several times, including a brief period during which the time-coordination radio signals broadcast both UTC and "Stepped Atomic Time (SAT)" before a new UTC was adopted in 1970 and implemented in 1972. This change also adopted leap seconds to simplify future adjustments.

A number of proposals have been made to replace UTC with a new system that would eliminate leap seconds. A decision whether to remove them altogether has been deferred until 2023.

So I guess you are saying that in 2023, you would vote to eliminate leap seconds. That's fine.

But in the meantime, it is very important to adhere to whatever the standard says. Adherence can have negative consequences if not handled correctly.

From an IT standpoint, it costs time and money. I once worked for an organization that billed by the hour. Whenever any change was made to billing software, it had to be tested for 23, 24, and 25 hour day use cases. How many IT shops do you think have tested the negative leap second use case, when they were told the popular view that negative leap seconds can never occur?

In #2, you said that NTP is enough. But have you seen a statement from whoever maintains NTP that it has been tested for the negative leap second case? I just searched and found this:

https://doc.ntp.org/reflib/leap/
# Although the definition also includes the possibility
# of dropping seconds (negative leap seconds), this has
# never been done and is unlikely to be necessary in the
# foreseeable future.

 

[pbuk]

GPS, among many other things.

Actually GPS is one of the few things that is not based on UTC see https://gssc.esa.int/navipedia/index.php/Time_References_in_GNSS#GPS_Time_(GPST)

 

[anorlunda]

Actually GPS is one of the few things that is not based on UTC see https://gssc.esa.int/navipedia/index.php/Time_References_in_GNSS#GPS_Time_(GPST)

That's cool. I didn't know that. However the page linked says.

GLONASS Time (GLONASST) is generated by the GLONASS Central Synchroniser and the difference between the UTC(SU) and GLONASST should not exceed 1 millisecond plus three hours\

So, has it been tested for cases where the difference is greater? The whole point of this thread is software use cases that have never been tested.

 

[pbuk]

So, has it been tested for cases where the difference is greater? The whole point of this thread is software use cases that have never been tested.

Well ISTR that the cumulative leap second offset is stored in each satellite as a 256 bit value: not sure if that is signed (probably not given that nobody thought the day would reduce below 86,400s when leap seconds were introduced).

Plenty of scope for overflow errors there!

 

[pbuk]

So, has it been tested for cases where the difference is greater? The whole point of this thread is software use cases that have never been tested.

GLONASS is the Russian system (hence the 3 hours). Heaven only knows what they have tested it for and we can only hope that there are no missile launch systems that will auto-launch if they think the US has sped up the Earth's rotation.

 

[Vanadium 50]

would be a lot of fun.

OK, you got me. It might be more fun that I thought.

 

[f95toli]

Indeed: I don't know anyone that understands the issues that thinks differently.

I do know quite a few people involved in time keeping; (including people who serve on the relevant committees) and while some of them do indeed want to change the system, some of them don't (for a variety of reasons). This has been an ongoing debate for a long time and I would be extremely surprised if anything changes in 2023; it would require a number of countries to change their current position. which is highly unlikely (it would be common knowledge by now).
Note that this would have to be a unanimous decision; meaning you need all of the governments of the countries that contribute to the UTC to agree. Needless to say, making any form of change takes a lot of time.

 

[jedishrfu]

One small 1.59 millisecond for man, too many leap seconds for mankind.

 

[jedishrfu]

One small 1.59 millisecond for man, too many leap seconds for mankind.

 

[Algr]

Therefore, Meta concludes, we should simply "stop the future introduction of
leap seconds."

I agree as well.

I'm surprised that something like an "Arbitrary Machine Time" (_{Defined as never having leap seconds, and not being connected to the rotation of the Earth.}) doesn't already exist, as it would be a useful tool to keep track of all the leap seconds that have occurred in the past. Right now if you want to know exactly how many seconds have occurred since some arbitrary date decades ago, you can't just use subtraction. You have to look up the history of all those leap seconds and manually adjust.

When a leap second happens, do they transmit to the Voyager probes and tell them to adjust their clocks? I hope not! It seems safer and easier to convert "Voyager time" to familiar units back on Earth as needed.

The leap second was introduced in 1972 and since then 27 leap seconds have been added to UTC. - Wikipedia

If some application needs to know the position of the Earth to this level of accuracy, being half-a-second off probably won't be good enough anyway, and would have to make local adjustments with or without leap seconds.

 

[pbuk]

I agree as well.

I'm surprised that something like an "Arbitrary Machine Time" (_{Defined as never having leap seconds, and not being connected to the rotation of the Earth.}) doesn't already exist, as it would be a useful tool to keep track of all the leap seconds that have occurred in the past.

It does exist, it's called International Atomic Time (TAI). UTC is derived from TAI and it is UTC that must keep track of its cumulative difference from TAI (currently 37 seconds), not the other way round.

 

[anorlunda]

International Atomic Time (TAI). UTC is derived from TAI and it is UTC that must keep track of its cumulative difference from TAI

The complexity mushrooms. We have two foundations for time.

1. The scientifically defined second.
2. The day and year tied to Earth's rotation and orbit. Commerce, school schedules, agriculture, prison sentences, ... will choose to use these units.

Derived units, minutes and hours, can be rooted on either. Your choice. The month and the calendar are already adapted to the year.

The problem occurs when we try to mash the two into a single unified standard. No matter how refined our science and technology, complexity and confusion will never become moot. Yet, I've never heard anyone suggest that we fully divorce the two systems.

Software, more than any other field, is saddled with the complexity of reconciling the two systems, because software serves both scientific and human purposes.

 

[Svein]

For us the implementers, the algorithm is clear:

• If NTP (or GPS or whatever) tells us that we should add a leap second, the last minute befor midnight should contain 61 seconds.
• If we are told that we should subract a leap second (or add a negative leap second), the last minute befor midnight should contain 59 seconds.

Ours is not to decide why, ours is to implement according to the specifications.

 

[Vanadium 50]

If you are using a well-vetted structure, like time_t, and well-vetted routines like difftime, what you suggest will work.

The problem apparently is that people don't use difftime(time_t x, time_t y) for the difference between the times x and y, but rather roll their own.

As you say, NTP handles this fine, and it deals with clocks that are too far ahead millions of times each day with no problems. The difficulty seems to be that some code where time-keeping is missionm critical a) bypass NTP, and b) bypass the well-vetted and built-in functionality described above.

Bad choices produce bad outcomes.

 

[pbuk]

For us the implementers, the algorithm is clear:

• If NTP (or GPS or whatever) tells us that we should add a leap second, the last minute befor midnight should contain 61 seconds.
• If we are told that we should subract a leap second (or add a negative leap second), the last minute befor midnight should contain 59 seconds.

Ours is not to decide why, ours is to implement according to the specifications.

And it is naive views like that that cause the problems.

Many of these problems stem from the fact that most operating systems (and posix systems in particular) do not store time as minutes and seconds, they store time as the notional number of seconds since some arbitrary point in time. For reasons which quickly become obvious, this notional clock is not adjusted for leap seconds so that for instance noon on 1 Jul 2015 is exactly 86,400 seconds after 30 Jun 2015. And 2015-07-01T00:00:01 (stored as 1435708830) is exactly 2 seconds after 2015-06-30T23:59:59.

But because a leap second was declared in that interval, all the clocks on the machine have to fit 3 seconds into that 2 second period - or perhaps in some other period when their NTP daemon catches up. This means (among other things) that an event (such as a file modification) that is recorded as happening at 2015-06-30T23:59:59 may have happened after an event recorded at 2015-07-01T00:00:00.

 

[pbuk]

If you are using a well-vetted structure, like time_t, and well-vetted routines like difftime, what you suggest will work.

Difftime is not going to help, if timestamps are unexpectedly out of sequence it will still report time moving backwards.

The difficulty seems to be that some code where time-keeping is missionm critical a) bypass NTP, and b) bypass the well-vetted and built-in functionality described above.

If time-keeping is mission critical it is absolutely essential that you bypass (any clock synchronised by) NTP and use a clock which measures (fractions of) seconds rather than the movement along an arbitrary non-monotonic scale.

Some discussion can be seen at https://developers.redhat.com/blog/2015/06/01/five-different-ways-handle-leap-seconds-ntp which contains the excellent summary paragraph:

Which method for correcting the system clock on leap second should you choose? It depends on your applications. It they don't require the system time to be monotonic, use ntpd or chronyd in their default configuration. There will be a backward step, but the clock will be off only for one second. If the time must be monotonic and the requirements on its accuracy are not very strict, you can use ntpd with the -x option to slew the clock instead. With chronyd you can set the leapsecmode option set to slew. If there is also a requirement to keep the clocks on multiple systems close to each other, consider using chronyd configured with a smaller maximum slew rate. If that is not an option, you can run your own leap smearing NTP server with chronyd, but be careful to not mix it with other servers.

 

[anorlunda]

Some discussion can be seen at

Thanks for sharing. Who could fail to read and digest all that? [SARCASM]

Who might be tempted to use a simple subtraction to implement t₂-t₁ in code; treating time as different than any other variable?

 

[Svein]

"And it is naive views like that that cause the problems."

Well, I do not agree. And, of course, you never change the real-time clock. What you change is the offset used in reporting the local time, just as you would change the offset when going to and from DST (which, by the way, is a much greater source of confusion than leap seconds).

 

[pbuk]

And, of course, you never change the real-time clock. What you change is the offset used in reporting the local time, just as you would change the offset when going to and from DST

Oh if only that were true, unfortunately it is the opposite of what normally happens in mainstream operating systems which change the "real-time" clock to match UTC. In Linux systems this is generally done by ntpd which by default will rewind the clock by a second without even telling you.

 

[Svein]

The worst nuisance is when files and events are timestampe with the "local clock" instead of UTC. Entering and leaving DST makes the local clock skipping an hour or jumping back an hour respectively.

 

[Melbourne Guy]

Let it go for a couple millennia and we'll fix it when the next daylight savings time change comes around.

Or, for some of us Down Under, much faster than that. Our clocks go back an hour every year, and have been doing so for many years, and non of my electronic devices - digital radios, PVR, phones, Linux / Windows PCs - that use external time servers to sync have ever had a problem.

Is this different to that process?

 

[anorlunda]

https://arstechnica.com/science/202...-be-abandoned-by-2035-for-at-least-a-century/
A near-unanimous vote on Friday in Versailles, France, by parties to the International Bureau of Weights and Measures (BIPM in its native French) on Resolution 4 means that starting in 2035, the leap second, the remarkably complicated way of aligning the earth's inconsistent rotation with atomic-precision timekeeping, will see its use discontinued. Coordinated Universal Time, or UTC, will run without them until 2135. It was unclear whether any leap seconds might occur before then, though it seems unlikely.

Leap seconds R.I.P. But the wait until 2035 sounds strange. Perhaps someone can interpret the following from the same article.

The assumption is that within those 100 years, time-focused scientists (metrologists) will have found a way to synchronize time as measured by humans to time as experienced by our planet orbiting the Sun.

 

[DaveC426913]

"Now that we survived the Y2K bug, we need to be ready for the next one: the Y10K bug. Let's not get caught with our pants down again."
- DaveC426913, 02001

 

[pbuk]

But the wait until 2035 sounds strange.

I gather it is the earliest date that could be agreed - the Russians wanted later (it appears that they believe, mistakenly, that leap seconds give GLONASS an advantage) whereas nearly everyone else wanted it sooner, ideally now. The resolution allows for it to be sooner, it is to be agreed in 2026, so all we need to do is not have any leap seconds before then and common sense will prevail (in BIPM? I'll not place any bets). See the extract below from the published resolutions: https://www.bipm.org/documents/2012...2022.pdf/281f3160-fc56-3e63-dbf7-77b76500990f.

The General Conference on Weights and Measures (CGPM)...

• decides that the maximum value for the difference (UT1-UTC) will be increased in, or before, 2035,
• requests that the CIPM consult with the ITU, and other organizations that may be impacted by this decision in order to
  • propose a new maximum value for the difference (UT1-UTC) that will ensure the continuity of UTC for at least a century,
  • prepare a plan to implement by, or before, 2035 the proposed new maximum value for the difference (UT1-UTC),
  • propose a time period for the review by the CGPM of the new maximum value following its implementation, so that it can maintain control on the applicability and acceptability of the value implemented,
  • draft a resolution including these proposals for agreement at the 28th meeting of the CGPM (2026),

 

[Tom.G]

Leap seconds R.I.P. But the wait until 2035 sounds strange. Perhaps someone can interpret the following from the same article.

The assumption is that within those 100 years, time-focused scientists (metrologists) will have found a way to synchronize time as measured by humans to time as experienced by our planet orbiting the Sun.

It sounds like they don't have a clue about appropriate implementation, and are hoping someone will come up with a "Bright Idea" by then.

 

[pbuk]

It sounds like they don't have a clue about appropriate implementation, and are hoping someone will come up with a "Bright Idea" by then.

I don't know where the journalist got "the assumption is that within those 100 years, time-focused scientists (metrologists) will have found a way to synchronize time as measured by humans to time as experienced by our planet orbiting the Sun" from, and the assumption is unnecessary, because leap seconds serve no useful purpose whatsoever.

It has been suggested that after 100 years UTC may have drifted by up to a minute but

1. this depends on predictions about the rotation of the Earth about which we now realise we don't understand as much as we thought we did
2. IT DOESN'T MATTER. If someone happens to celebrate their 121st birthday by repeating a party they held at dawn on their 21st birthday at the same time and in the same place they are not going to notice that sunrise is a minute later than it was 100 years ago.

 

[anorlunda]

It has been suggested that after 100 years UTC may have drifted by up to a minute but

Ouch. That suggests that they might want a leap minute after a century of programmers being out of practice and having forgotten how to handle a leap.

Oh well, the Technological Singularity should happen before then, so there will be no more human programmers.

 

[Vanadium 50]

That suggests that they might want a leap minute

Well, why I think running code that hasn't been tested in a century is a bad idea, but one could turn it around: what happens if we wait a couple thousand years and use it to adjust daylight savings time?

If astronomical noon doesn't happen at exactly 12:00 in Greenwich, Englan, but instead happens in Westminster, who cares?