Google put atomic clocks in its datacenters so a database could wait on purpose

I went down a rabbit hole on Spanner, Google's globally distributed database, and the part that stuck with me had nothing to do with a clever consensus algorithm. Google hit a problem every distributed database has, and instead of out-thinking it, they bought hardware and then made the database wait. On purpose. A few milliseconds, on every write, across the planet.

Here's the problem, and then the trick.

Google's data center in The Dalles, Oregon. Somewhere inside a building like this is a rack whose only job is to know what time it is. Source: Wikimedia Commons (Visitor7, CC BY-SA 3.0).

Two machines can't agree what time it is

Every computer has a quartz crystal driving its clock, and every crystal drifts. Not by much, tens of microseconds a second, but it adds up, and no two machines drift the same way. NTP, the protocol everything uses to sync clocks over the network, pulls them back into line, but only approximately. It does keep an error estimate of its own, but that's a statistical figure, not a guarantee, and the clock call your code actually makes hands back a bare number with no uncertainty attached.

For most software this doesn't matter. For a database spread across continents it's the whole problem. Say a transaction commits in Oregon and stamps itself 10:00:00.005. A second transaction, which genuinely happened a millisecond afterwards, commits in Belgium and stamps itself 10:00:00.004 because Belgium's clock runs two milliseconds slow. Now anyone reading in timestamp order sees the second write before the first. The database has quietly rewritten history, and nothing crashed, so nobody notices until an accountant does.

The usual escapes all give something up. Logical clocks (Lamport's, vector clocks) give you a consistent order but no relationship to wall time, so you can't ask for "the database as of 10:00." A single coordinator that hands out timestamps gives you order and wall time but becomes the bottleneck for every write on Earth. Or you stop promising anything and call it eventual consistency, which is fine until it isn't.

Google wanted the thing none of those give you: a transaction that commits before another one starts, anywhere in the world, gets the smaller timestamp. Always. The paper calls this external consistency, and claims Spanner as the first system to offer it at global scale.

Stop pretending you know the time

The idea underneath TrueTime is almost embarrassingly simple once you see it. Nearly every system asks the clock "what time is it?" and gets a number. TrueTime refuses to answer that question. It answers a different one: what's the earliest and latest it could possibly be, right now?

TT.now() returns an interval, [earliest, latest], with a guarantee that the true time is inside it. Two helpers fall out of that: TT.after(t) is true only when t is definitely in the past, and TT.before(t) only when t is definitely still in the future. The half-width of the interval, which the paper calls ε, is the machine's honest confession of how much it doesn't know.

How big is ε? In the 2012 paper, on Google's production hardware, it was a sawtooth between roughly 1 and 7 milliseconds, averaging about 4. Every machine re-syncs with a time master every 30 seconds, and between syncs it assumes its own crystal could be drifting at up to 200 microseconds per second, so the uncertainty grows linearly: 30 seconds at 200 µs per second is 6 ms, plus about a millisecond of round trip to the master. Then it syncs, the interval snaps back down to about a millisecond, and the sawtooth starts climbing again.

That interval is not an estimate. It's a bound: the true time is guaranteed to be inside it. The 7 ms is only the top of the normal sawtooth; the paper says ε is generally under 10 ms and shows it spiking when time masters go down for maintenance. And a bound you can trust is worth more than a guess you can't, because you can build on it.

Where the certainty comes from

You can't get a guaranteed bound out of NTP, so Google didn't try. Every datacenter runs a set of time master machines. Most of them have GPS receivers with dedicated antennas. A few have atomic clocks instead, which the paper, with a straight face, calls Armageddon masters. The two kinds fail differently, and that's the point: GPS goes wrong through antenna and receiver faults, radio interference, or spoofing, while an atomic clock has no antenna or satellite to lose, fails in ways uncorrelated with GPS, and over the long run mostly just drifts, so it's very unlikely both fail the same way at once.

A GPS antenna and receiver, this one belonging to NOAA's National Geodetic Survey on a Colorado summit. Spanner's time masters get their time from receivers like this, each with its own antenna, in every datacenter. Source: Wikimedia Commons (Brian Shaw/NOAA, public domain).

The vapor cell of a chip-scale rubidium atomic clock built at NIST, next to a coffee bean for scale. The atomic clocks in Spanner's Armageddon masters are the industrial version of this idea: a clock that keeps its own time and doesn't care what GPS is doing. Source: Wikimedia Commons (Hummon/NIST, public domain).

Every machine in the fleet runs a daemon that polls a spread of these masters, some nearby and some in other datacenters, and runs a variant of Marzullo's algorithm to throw out the ones that are lying. Between polls, the daemon widens its own uncertainty at that conservative worst-case drift rate. The paper mentions a detail I liked: in Google's fleet, a bad CPU was six times more likely than a bad clock. Clocks were the least of their hardware problems. The engineering wasn't about making clocks perfect. It was about knowing exactly how imperfect they were, at every instant, on every machine.

Marzullo's algorithm on three unreliable sources: each reports an interval, and the algorithm finds the smallest interval consistent with the most sources. Here all three agree and it narrows to [11, 12]; a source whose interval missed that overlap would be outvoted. Spanner's time daemons run a variant of this against the masters they poll. Source: Wikimedia Commons (Daniel exb, CC BY-SA 3.0).

Commit-wait: the pause that buys the guarantee

Now the trick.

When a read-write transaction is ready to commit, Spanner picks its commit timestamp s from TrueTime, at least as large as TT.now().latest, so it's no earlier than the true current time. Then it does the thing that looks wrong. It holds the transaction, locks still in place, and waits until TT.after(s) is true: until the timestamp it just assigned is definitely past in true time, so that any interval TrueTime hands out anywhere from then on has its latest edge beyond s. Only then does it release the locks and tell the client "committed."

That wait is at least the width of the interval, 2ε, and the paper notes it runs concurrently with the Paxos round that logs the commit, so only part of it shows up as extra latency. Measured, it added about 5 milliseconds. And it buys external consistency outright. If transaction one commits before transaction two starts, then by the time anyone can see transaction one's write, its timestamp is already in the past, in true time. Transaction two, starting after that, has to pick a timestamp from a later interval. Larger timestamp, guaranteed, with no coordination between the two transactions at all. Belgium can never stamp itself before Oregon, because Oregon didn't let go until true time had definitely passed the mark, and Belgium's interval moves with true time.

What gets me about this is the shape of the solution. The uncertainty didn't go away. Google made it small and known, and then paid for it in the one currency a database has plenty of: a few milliseconds of latency on a write that was already crossing an ocean to replicate.

Why the wait makes reads free

Paying at commit time is what makes everything downstream cheap.

Because every committed write carries a timestamp that's honestly in the past, any replica can answer "what did the database look like at time t?" without taking a single lock. Each replica tracks a safe time, the timestamp up to which it's caught up on both replication and in-flight transactions, and once that reaches t, it can serve a consistent snapshot read at t from wherever it sits. Snapshot reads never block writes, a read at t waits only until the replica has caught up to t, and a read in Singapore against a replica in Singapore sees exactly what a read in Oregon sees for the same t. You pay milliseconds once, at write time, and get lock-free consistent reads from every replica on Earth in return.

TrueTime does one more quiet job. Spanner replicates each shard through Paxos, and every Paxos group has a leader holding a ten-second lease. Two leaders overlapping in time would be a disaster, and the leases are kept disjoint with the same interval reasoning: a replica won't vote for a new leader until TrueTime says its previous vote has definitely expired, and a leader stops using its lease as soon as TrueTime says it might have ended.

What it costs, and where it bends

Nothing here is free, and the paper doesn't pretend otherwise.

Every write carries a latency floor: the commit wait alongside a Paxos round trip that may cross continents. For a system whose first big customer was F1, the backend behind Google's ads, moving off sharded MySQL, that was a trade worth making. For a chat app it might not be.

The guarantee is only as good as the uncertainty stays small. If the time masters degrade and ε grows, the commit wait grows with it. Spanner gets slower rather than wrong, as long as no crystal drifts faster than the 200 µs per second the daemon assumes. A clock caught doing that is evicted rather than trusted, and the paper is frank that one that slipped through would break the assumption, which is why the bad-CPU-versus-bad-clock statistic matters. That's exactly the failure mode you'd want, but you do have to want it.

And you need the hardware. GPS antennas and atomic clocks in every datacenter is a thing Google can do and most companies can't, which is why the open-source systems that borrowed Spanner's design took a different route. CockroachDB, for instance, uses hybrid logical clocks and simply assumes clocks are never more than a configured offset apart (500 ms by default). It gets serializable transactions and per-key linearizability, but not Spanner's global external consistency, and it pays in a different currency: a transaction that reads a value stamped inside its own uncertainty window, its timestamp plus the max offset, may have to restart. Less hardware, more retries. Same problem, different bill.

Why bother

I think the reason this design is worth understanding, even if you'll never run a database across continents, is what it does with an uncomfortable fact. Distributed systems are full of things you can't know precisely: what time it is, whether a message arrived, whether the other node is dead or just slow. The reflex is to pick an answer and hope. Spanner's move is to refuse the guess, measure the doubt, put a hard bound on it, and design around the bound.

Google put atomic clocks in its datacenters, and then wrote a database whose most important instruction is to wait. Not because it couldn't go faster. Because it knew exactly how much it didn't know, and a few milliseconds was the price of never being wrong about the order of things.

Sources: the Spanner paper (Corbett et al., OSDI 2012), and Google Cloud's TrueTime and external consistency page.