12 weirdest root causes in internet history

An engineer was following an approved, step-by-step playbook to take a few servers offline for debugging. One value in the command was mistyped, and the tooling cheerfully removed a far bigger set of servers than intended — including ones running two core systems that hadn't been fully restarted in years. They needed a long, careful cold start, and the internet got a four-hour nap.

An authorized human. An approved playbook. One wrong keystroke.

Technicians rolled out new trading software to seven of their eight servers. The eighth was missed. Worse, the release reused an old on/off switch that used to activate Power Peg — a retired test routine built years earlier to deliberately buy high and sell low, purely to stress-test the system. It was never deleted. Just left there, sleeping.

The switch flipped on. Seven servers ran the new code. The eighth heard the old signal, woke the zombie, and did precisely what it was designed to do: lose money as fast as possible — with real capital, at market speed.

Late at night, a tired engineer was fighting a lagging database copy. The fix: wipe the backup server's data folder so it could re-sync fresh. He ran the delete command. In the terminal window connected to the main server. He noticed within seconds. Seconds were plenty.

The data came back thanks to a manual snapshot another engineer happened to take six hours earlier — pure luck. The scheduled backups? Failing silently. The cloud copies? Empty folder. The emails warning about all this? Bounced.

Someone ran rm -rf * — the delete-everything-below-this-point command — against the master folder holding the movie. The computer did what computers do: exactly what it was told, without a single follow-up question.

The film survived because technical director Galyn Susman had recently had a baby and was working from home, with a full copy of the movie syncing to a machine in her house. Pixar drove that computer back to the studio wrapped in blankets, buckled in like a passenger. A home desktop was, functionally, the studio's only off-site copy.

Deep inside Azure, a piece of software created security certificates valid for exactly one year — calculated by taking today's date and adding 1 to the year. On February 29, 2012, that math produced February 29, 2013. A date that does not exist. Certificate creation failed, the software mistook its own bug for broken hardware, marked healthy machines as faulty, and the platform's self-healing systems started taking down perfectly good servers at scale.

One line of date math, asleep for four years, waiting for its moment.

A new firewall rule shipped with a text-matching pattern that had a nasty property: on certain inputs, the work it needed exploded instead of finishing. Every processor that touched it got stuck grinding on the same impossible task. And because security rules were built to deploy worldwide in seconds (to react to attacks fast), every machine on the planet received the poisoned rule at once.

One pattern. Written by one person. Reviewed and approved. Down went the network built to keep the internet up — for 27 very long minutes.

An automated system that hands out storage allowances had a bug: it decided Google's central sign-in service was using zero storage, and helpfully shrank its allowance to match. When the grace period ran out, the service that verifies every Google login on Earth had no room left to write. It couldn't record new sessions, so it did the only thing it could: reject everyone.

The oldest failure in computing — disk full — wearing a very modern automation costume.

A routine maintenance command, meant to check spare capacity on Facebook's internal backbone, instead disconnected all of its data centers from each other. A safety mechanism then noticed the disconnect and did what it was designed to do: pull Facebook's addresses off the internet's global map. From the outside, Facebook had simply stopped existing.

The kicker: internal tools, email — even the badge readers on the data center doors — ran on the same downed network. Engineers had to physically drive to the buildings and fight their way in to reach the machines, because the systems they needed to fix were the systems they needed in order to fix them.

The Pakistani government ordered local providers to block YouTube inside the country. One telecom implemented the block by publishing a fake set of directions: "YouTube lives over here now" — pointing at a dead end. Those directions were meant to stay domestic. They leaked to the wider internet, and because the internet's routing system trusts the most specific directions it hears, the entire world obediently rerouted YouTube's traffic into the void.

The internet's routing layer runs on trust between networks. In 2008, nothing stopped one mid-sized provider from accidentally redirecting a global platform.

A server update had quietly rolled the mail software back to an older version — which then read a settings file written for the newer one. Options it didn't understand, it ignored. One of those was the connection timeout, which ended up effectively at zero: a few milliseconds in practice.

A connection could only succeed if the other server answered almost instantly. And light traveling through fiber covers about 3 milliseconds' worth of distance — just over 500 miles. Anything farther away physically couldn't answer in time. The users' impossible bug report was accurate to within a rounding error of the laws of physics.

A 75-year-old woman in neighboring Georgia was digging for scrap copper to sell. Her spade found a buried fiber-optic cable — the main line carrying most of Armenia's connectivity — and cut clean through it. The press nicknamed her "the spade hacker." She reportedly had no idea what the internet was. She remains its most efficient adversary.

A stone marten — a member of the weasel family — climbed onto a 66,000-volt transformer and bit a cable. The official incident log records the cause, in French, as "short circuit caused by fouine." The marten did not survive its contribution to particle physics. Its taxidermied body is now on display at the Rotterdam Natural History Museum, in an exhibit devoted to animals that collided with human infrastructure.

It wasn't even the collider's first wildlife incident. In 2009, a passing bird knocked out part of the cooling system by dropping a baguette on outdoor electrical equipment. The machine that recreates the birth of the universe has now been defeated by both a weasel and French bread.