Most engineering failures aren't component failures — they're systems failures

The Challenger disaster's proximate cause was an O-ring that failed in cold temperatures. But that's not really what caused the disaster. The O-ring's cold-temperature behavior was known. The risk had been raised. Launch proceeded because the organizational structure couldn't process the signal. It's a systems failure, not a parts failure.

Charles Perrow's Normal Accident Theory argues that in tightly coupled, complex systems, accidents are not aberrations — they're the expected outcome of how those systems are organized. The interactions between components produce failure modes that no single engineer or manager could anticipate or prevent. The accident is 'normal' in the sense that it was embedded in the system's structure from the start.

This has direct implications for how we design engineered systems and, increasingly, for how we think about AI systems in deployment. A language model fine-tuned on human feedback and deployed in a product with millions of users is a complex sociotechnical system. Its failures will often be interactions between components — the model, the interface, the incentive structure of the platform, the diversity of the user population — that weren't predictable from any component in isolation.

What examples of systems-level engineering failures have you found most instructive? What did they reveal about assumptions embedded in the design process?