Kubernetes enables effortless orchestration of workloads with high resiliency, autoscaling, and incremental rollouts that few other systems can replicate. But what do you do when the systems that are supposed to keep your workloads healthy start working against you? And more importantly, what can you do to not only prevent issues like this but level up your architecture and understanding of Kubernetes? In this anecdote from the trenches of Quizlet's infrastructure engineering, I will talk about how our distributed system failed spectacularly and share what we learned and improved because of it.