Sysadmin

We ran into this bug in a production system a few months back. We had a legacy cluster of windows workbenches which connected to each other using an encrypted communications API on an isolated network. We initially couldn’t determine why the system clocks fell out of sync in a rather cascading fashion. Guess this explains it. We ended up resolving it by bridging them to the internet and forcing a sync with time servers. A few months later, it happened again. At the time we thought it to be a bug in Windows. Go figure it was.

The important bit:

Simen said he believes the STS design is based on a fundamental misinterpretation of the TLS specification. Microsoft’s description of STS acknowledges that some SSL implementations don’t put the current system time of the server in the ServerUnixTime field at all. Instead, these implementations—most notably the widely used OpenSSL code library starting in 2014—populate the field with random values. Microsoft’s description goes on to say, “We have observed that most servers provide a fairly accurate value in this field and the rest provide random values.”

“The false assumption is that most SSL implementations return the server time,” Simen said. “This was probably true in a Microsoft-only ecosystem back when they implemented it, but at that time [when STS was introduced], OpenSSL was already sending random data instead.”

This is the best summary I could come up with:

A few months ago, an engineer in a data center in Norway encountered some perplexing errors that caused a Windows server to suddenly reset its system clock to 55 days in the future.

The engineer relied on the server to maintain a routing table that tracked cell phone numbers in real time as they were being moved from one carrier to the other.

“With these updated routing tables, a lot of people were unable to make calls, as we didn't have a correct state!” the engineer, who asked to be identified only by his first name, Simen, wrote in an email.

Simen had experienced a similar error last August when a machine running Windows Server 2019 reset its clock to January 2023 and then changed it back a short time later.

The mechanism, Microsoft engineers wrote, “helped us to break the cyclical dependency between client system time and security keys, including SSL certificates.”

Simen and Ken, who both asked to be identified only by their first names because they weren’t authorized by their employers to speak on the record, soon found that engineers and administrators had been reporting the same time resets since 2016.

I'm a bot and I'm open source!