Normal accidents in the cloud: what Perrow, Cook and Dekker predicted

A bad update that was live for 78 minutes froze about 8.5 million Windows machines in 2024. In October 2025 an empty DNS record took down AWS’s biggest region for 14 hours. This week a Microsoft network change cost much of the world its email for 20 hours. None of it would have surprised a Yale sociologist who never configured a router. In 1984 Charles Perrow argued that in systems that are both complex and tightly coupled, small failures will eventually line up and cascade, however careful everyone is. Richard Cook, a physician, turned that into eighteen blunt points in 1998. Sidney Dekker added the slow version: organisations drift toward the edge one reasonable decision at a time. Dan Geer added the multiplier in 2003: when everyone runs the same thing, everyone fails at once. Here is what they predicted, in their own words, from my library, and how it maps onto those three outages.

Contents · 12 sections

TL;DR #

• Perrow: in complex, tightly coupled systems, accidents are “normal”. You can make them rarer and smaller, not impossible.
• Cook: catastrophe needs several failures, “root cause” is a fiction, and change brings new kinds of failure.
• Woods and Cook warn that economic pressure strips out the buffers, including experienced people, that make recovery possible.
• Dekker: systems drift into failure through normal, locally sensible decisions, and the drift is mostly visible in hindsight.
• Geer: monoculture aggregates risk. CrowdStrike proved it applies to well-meant updates, not just worms.

Perrow: the accident is normal #

My library holds Perrow’s own summary of his theory, the entry “Accidents, Normal” that he wrote for the International Encyclopedia of the Social & Behavioral Sciences. He describes the safeguards, redundancies and alarms engineers build, and then the catch: “But occasionally two or more failures, perhaps quite small ones, can interact in ways that could not be anticipated by designers, procedures, or training.”

If the system is also tightly coupled, the failures cascade. His conclusion gave the book its title: “The accident, then, is in a sense ‘normal’ for the system, even though it may be quite rare, because it is an inescapable part of the system.”

Two ideas matter. Interactive complexity means parts affect each other in ways nobody drew on a diagram. Tight coupling means there is no slack: steps follow each other fast, in a fixed order, with no time to intervene. Perrow also saw why we build systems this way anyway. In the same entry he writes that tight coupling “may be required to ensure the most economical operation and the highest throughput speed”. That is the business case for a global cloud in one sentence.

Cook: there is no root cause #

Cook’s “How Complex Systems Fail” is short enough to read over coffee, and the offline encyclopedia shelf in my library tells his story: an anaesthetist who ended up teaching software operations. Four of his points do most of the work for cloud outages:

• Point 3: “Catastrophe requires multiple failures – single point failures are not enough.”
• Point 7: “Post-accident attribution to a ‘root cause’ is fundamentally wrong.”
• Point 14: “Change introduces new forms of failure.”
• Point 18: “Failure free operations require experience with failure.”

The library also holds Resilience Engineering (edited by Erik Hollnagel, David Woods and Nancy Leveson, 2006), where Woods and Cook take the idea further. Their chapter describes a hospital pharmacy system that failed and was carried by staff working by hand for 24 hours. Quoting Cook’s case study, they conclude that “the resilience of this system resides in its people rather than in its technology”. Then comes the warning I keep returning to: “Economic pressure is reducing the buffers, which operated in the incident response, as ‘inefficiencies.’”

Dekker: drifting into failure #

In the same volume, Sidney Dekker gives the slow-motion version: “‘Drifting into failure’ is a metaphor for the slow, incremental movement of systems operations toward (and eventually across) the boundaries of their safety envelope.” Nobody decides to be unsafe. Each step, a faster deploy pipeline or a skipped canary for “just config”, is locally sensible and worked last time. Dekker puts it plainly: “it is in these normal, day-to-day processes of organizational management and decision-making that we can find the seeds of organizational failure.”

His theory has an uncomfortable corollary: drift is visible mostly in hindsight. That should make us suspicious of confident single-cause stories in both directions, “it was the layoffs” as much as “it was one typo”.

Geer: the monoculture multiplier #

In 2003 Dan Geer and six co-authors, among them Bruce Schneier and Charles Pfleeger, published “CyberInsecurity: The Cost of Monopoly”. Pfleeger’s textbook Security in Computing, which sits in my library, cites the paper. Its core claim is that most of the world’s computers were “vulnerable to the same viruses and worms at the same time”, and that “monocultures create aggregated risk like nothing else”. Geer reportedly lost his job over it.

The paper was about attackers. Twenty-one years later, CrowdStrike showed the logic holds just as well for a well-meaning vendor.

CrowdStrike, July 2024 #

According to CrowdStrike’s own review, a faulty content file for its Falcon sensor was live for 78 minutes, from 04:09 to 05:27 UTC. That was enough to put about 8.5 million Windows machines into boot loops; Microsoft put it at under 1% of all Windows devices. Airlines cancelled thousands of flights, and hospitals and broadcasters went down.

Perrow would see three things that were each fine on their own: a content update, a kernel driver and an unchecked read. They interacted, and the coupling was extreme: the update reached every sensor worldwide within minutes, with no gap in which a human could notice. Geer would see his monoculture made literal: one vendor’s agent on a large share of the world’s corporate machines. Cook’s point 18 showed in the recovery. The fix needed someone at each keyboard, and the organisations that recovered fastest had people who knew how to boot thousands of machines into safe mode.

AWS us-east-1, October 2025 #

AWS’s review names a “latent race condition in the DynamoDB DNS management system” that left the regional endpoint with an empty DNS record. The outage lasted 14 hours 32 minutes.

This is Perrow’s small beginning: a race between two automated processes spread through DynamoDB to EC2 launches, load-balancer health checks, Lambda and sign-in. Each dependency was reasonable on its own. Cook’s point 7 applies too: the race condition is the trigger, not “the cause”. The outage also needed the dependency web, the recovery behaviour and the shared regional control plane. Much of the long tail was recovery itself, as backlogged work and retries hit systems that were coming back. Cook called that kind of abrupt loss of slack “going solid”.

Microsoft 365, 12–13 November 2026 #

Microsoft’s post-incident review is not out yet, so this part uses its preliminary status updates. They describe an earlier network capacity upgrade, servers failing health checks and dropping out, the remaining servers overloaded, a “surge in outbound call volume from routing services to backend servers”, and restored servers getting stuck “in a bad state” again. LSN News, my news engine, counted 4,556 articles from 2,272 outlets in 55 languages on it.

The health checks, a safety mechanism, became part of the failure. Removing unhealthy servers is protective, until it removes so many that the survivors are overwhelmed. Perrow warned that safety devices add complexity of their own. Cook’s point 14 fits the trigger: per Microsoft, it was a capacity upgrade, a change meant to make the system more robust. Microsoft’s own January 2023 WAN outage started the same way, with an engineer adding capacity. And the slow part was not finding the change but coaxing a fleet back to health under retry load, which is exactly where Woods and Cook say buffers and experienced people matter. Geer would note who went quiet at once: intelligence agencies, courts, a stock exchange and GP surgeries in several countries. Monoculture is no longer an operating system. It is a mail and identity platform.

What the theory does not let me say #

It does not tell me that layoffs or AI-written code caused any of these. Cook and Dekker predict that losing experienced staff erodes adaptive capacity, and that faster, cheaper change creates new failure modes. If that is happening, it should show first as longer recoveries, not more triggers. That is testable, and so far the evidence is thin: no published review of these three events mentions staffing or AI, and Microsoft said this week’s issue was “not related to cyber or AI/agents”. I treat that link as an open question, not a conclusion.

Checklist #

Perrow offered two remedies: make systems more linear, or loosen the coupling. In cloud terms:

• Loosen coupling in time. Stage every rollout, config and content included: canary, wait, watch, then widen.
• Loosen coupling in space. Use cells, regional isolation and blast-radius limits.
• Make recovery a design goal. Backoff with jitter, admission control and circuit breakers give a system a way back from “going solid”.
• Keep the buffers. Spare capacity and people who have practised failure look like inefficiency on a spreadsheet until they are the only thing that works.
• Diversify where it is cheap. You probably can’t avoid a hyperscaler, but your incident channel, break-glass identity and status page don’t need to share its fate.

Frequently asked questions #

What is a “normal accident”? #

It is Perrow’s term for a failure that comes from unexpected interactions between several small faults in a complex, tightly coupled system. It is “normal” not because it is frequent but because the system’s design makes it inevitable sooner or later.

Is the cloud a tightly coupled system in Perrow’s sense? #

Its control planes are. One configuration or content file can reach every node worldwide in minutes, and services depend on shared layers like DNS and identity. That is tight coupling plus interactive complexity, the combination Perrow worried about.

Why does Cook say “root cause” is wrong? #

Because a big failure needs several faults at once, and the defences that usually stop them must also have gaps. Naming one trigger as “the cause”, whether a typo, a race condition or a layoff, hides the other conditions that will produce the next outage.

Did layoffs or AI cause these outages? #

There is no published evidence that they did. The theory predicts that losing experience and speeding up change would show first as slower recoveries, so that is what to measure. It is not a conclusion to assume.

What can an ordinary engineering team take from this? #

Stage every change, including config. Limit blast radius. Design clients to back off. Keep someone on the team who has practised recovering from failure. None of it prevents outages, but all of it makes them smaller and shorter.

The lesson #

Perrow’s most uncomfortable insight is that safety and efficiency pull in opposite directions, and we keep choosing efficiency because on most days it wins. The cloud is a magnificent piece of tight coupling, and its failures are the price of that design, not a betrayal of it. We can’t make outages go away. We can make them smaller, shorter and less surprising, and that is the most an honest engineer can promise.