How S3 gets eleven nines of durability out of ordinary hard drives

I went looking for how S3 actually stores data, expecting a boring answer about replication, and found something much better. We all use S3. Almost nobody knows what the 99.999999999% on the pricing page physically means, or how it survives on hardware that fails constantly.

What eleven nines means

Store 10,000 objects in S3 and, statistically, you lose one every ten million years. That's the promise. And it's being kept at a scale that's hard to picture: AWS has put the numbers at over 400 trillion objects, 150 million requests a second, more than a petabyte per second of traffic at peak, and tens of millions of hard drives underneath it all.

The question worth asking is how you get that kind of reliability without giving up speed, because the two usually pull against each other.

The hardware S3 is stuck with

At hundreds of exabytes, cost decides the hardware. SSDs would be lovely. They're also unaffordable at that scale, so S3 runs on spinning disks, and spinning disks are limited by physics: a mechanical arm has to move to the right spot on a platter, and that takes around 10 milliseconds for a random read. That number has barely moved in decades.

Inside an opened HDD, the stacked platters and the read/write head riding on its actuator arm. That arm has to physically move, which is where the roughly 10 ms latency comes from. Source: Wikimedia Commons (CC BY-SA).

Set that against a petabyte a second of demand and the gap between what one drive can do and what the system has to deliver is absurd. The only way across it is parallelism, and not the easy kind. Sequential writes parallelize on their own. Reads don't: users ask for any object at any moment, so the data has to be spread in a way that lets any read fan out across many drives at once.

Replication is the obvious answer, and the expensive one

The traditional way to survive disk failures is to keep copies. Want to survive two dead disks? Store three full copies of everything. It works, and it costs 200% extra storage, which at exabyte scale is a number with a lot of zeros on the invoice.

Erasure coding gets the same protection for a fraction of the space. Split an object into K data shards, use some math to generate M extra parity shards, and store all K+M. The property that makes it work: you can rebuild the original from any K of the K+M pieces. With 12 data shards and 4 parity shards, you have 16 pieces, you can lose any four, and the storage overhead is a third instead of 200%.

The same idea in miniature, a RAID 6 stripe: data blocks (A1, A2, A3...) plus independent parity blocks (Ap, Aq) spread across disks, so the stripe survives lost disks. Erasure coding generalizes this to K data plus M parity shards. Source: Wikimedia Commons (CC BY-SA).

AWS doesn't publish its exact schemes. From its own talks, it uses several, including 5+4 (nine pieces total) and 12+4 (sixteen), spread across at least three availability zones, which are separate data centers kilometers apart. So an entire data center can go dark and your data is fine, four simultaneous failures are survivable, and every object has a dozen or more places it can be read from at once. That last part matters as much as the durability, and I'll come back to it.

The math: Reed-Solomon

The encoding S3 uses is Reed-Solomon, the same math inside QR codes and CDs. The core idea only needs school algebra.

You need n points to pin down a polynomial of degree n minus one. Two points fix a line, three fix a parabola, and twelve fix a unique degree-eleven polynomial.

A set of data points and the unique polynomial curve running through them. Reed-Solomon treats each shard as a point like these: any K points pin down the same curve, so the missing ones can always be recovered. Source: Wikimedia Commons (CC BY-SA).

So take the 12 data shards and treat each as a point, at x equals 1 through 12. Find the one polynomial that passes through all of them. Then evaluate that polynomial at x equals 13, 14, 15 and 16. Those four values are the parity shards. Now you hold 16 points that all sit on one curve.

Lose any four of them, and it doesn't matter which four. The 12 you have left are enough to reconstruct the exact same polynomial, and once you have the polynomial you can compute any point you lost. That's the whole trick.

In practice the arithmetic isn't done on ordinary numbers. Binary data doesn't play nicely with floating point, rounding errors would corrupt everything, and the values would grow without bound. So it's done in a Galois field, finite-field arithmetic where the numbers wrap around and stay exact. The picture doesn't change: data lives on a curve, and enough points rebuild the curve.

Where the speed comes from

Erasure coding is usually sold as a durability trick. It's just as much a performance one.

Because every object is already split into 12 to 16 shards on different drives, a read is naturally parallel. One drive alone tops out around 300 MB/s; a dozen working at once don't. Aggregate that over thousands of objects and millions of drives and you get the throughput S3 delivers out of hardware that individually crawls.

Striping in its simplest form: consecutive blocks (A1, A2, A3...) land on different disks, so a single read pulls from both at once. S3's shards do the same thing at enormous scale, every object read fanning out across many drives. Source: Wikimedia Commons (CC BY-SA).

And it stacks. The SDK breaks large files into chunks for upload and download. Those chunks go to different S3 endpoints. Each endpoint stores its data as shards across many storage nodes. Parallelism at every layer, all the way down.

The load balancing is the part I didn't expect. With millions of drives serving billions of requests, S3 can't keep a global picture of which drives are busy; at that scale the picture would be stale before it finished forming. So it doesn't try. When placing shards it samples a few drives at random and takes the least loaded of them. Cheap, and it works. On top of that there's continuous rebalancing: as drives fill and data ages into being read less, S3 shuffles it around to keep both space and I/O spread out.

There's a counterintuitive effect hiding in here. Any single workload is bursty and hard to predict. Add up millions of independent workloads and the bursts average out, so the system as a whole gets smoother and more predictable the bigger it gets. Scale, for once, working in your favor.

Shards also solve the slow-node problem almost for free. If one storage node is dragging, S3 asks other shards for the same data in parallel and takes whichever answers first. Because a shard is a small piece rather than a full copy, the redundant requests cost very little.

What actually happens to your file

On upload, the file is split into 12 data shards, Reed-Solomon generates 4 parity shards, and all 16 are spread across three or more availability zones onto randomly chosen drives. S3 runs an integrity check before it confirms the upload, and from then on it watches for failed drives in the background and regenerates any shard they took with them.

On download, S3 requests shards from several storage nodes in parallel, needs only 12 of the 16, takes whichever 12 arrive first, decodes them, and streams the result to you while the decoding is still running.

What you end up with is an object that survives the loss of a whole availability zone or any four shards at once, reads fast because every read is parallel, and, by AWS's own figure, costs around 1.8x in storage rather than the 3x that three copies would.

What I'm going on

None of this comes from S3's code. AWS doesn't open-source it and doesn't publish its exact erasure schemes; the 5+4 and 12+4 figures, the object counts and the request rates all come from its own talks and posts, and they change as the system does. The walkthrough above is also simplified. Real S3 layers storage classes, caching and hot-data paths on top of this, and none of that is in here. What I'm confident about is the shape: erasure coding for durability, parallelism at every layer for speed, and randomness instead of global state for balance.

Why it stuck with me

The thing I keep coming back to is that S3's durability isn't a triumph of expensive hardware. It's the opposite. The hardware is cheap and slow and dies constantly, and the engineering starts from that fact rather than fighting it: turn an economics constraint into a math problem, then wrap the math in enough systems engineering that the slow parts add up to something fast. The polynomials are a nice trick. Getting them to hold up across tens of millions of failing drives is the actual achievement.