Abstract: How we turned "will it hold up?" into testable hypotheses using a deck of chaos experiments (the ChaosEngineeringMasterDeck), and what we uncovered when degrading the latency of a payment microservice in a real-world environment.

Why Break Things on Purpose In distributed systems, partial failure is the normal state, not an exception. With dozens of services, each with 99.9% availability, the composite availability of a request traversing 10 of them in series is approximately $0.999^{10} \approx 99.0\%$. That represents roughly 87 hours of downtime per year that no one anticipated by evaluating services individually. Unit and integration tests verify what code does when everything works; they reveal nothing about system behavior when the network slows down, a node disappears, or a downstream dependency stalls. Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions.…