Our checkout service went down for 47 minutes because the transactional email vendor had an outage. Placing an order does not need an email to be sent at that moment, but the order handler called the email API synchronously, the calls hung until timeout, the worker pool filled up and every other request queued behind them. A dependency that should have caused "confirmation emails are late" instead caused "nobody can buy anything". That incident made us rewrite how we think ab...
Design for failure on AWS: static stability, cells and blast radius
An email provider outage stopped customers from placing orders. Nothing about orders needed email. We rebuilt the service around three ideas: static stability, cell-based isolation and a blast-radius budget per dependency. Here is the worked design.
This is a paid story
Subscribe to read the full content, access all paid stories, and support the authors.
Add a label
Embed this post
<iframe src="https://www.hibug.me/design-for-failure-aws-reliability" width="700" height="420" frameborder="0" loading="lazy"></iframe>
I am a Senior Software Engineer with over a decade of hands-on experience building, scaling, and maintaining complex systems across the full stack—spanning frontend, backend, and DevOps. Over the last 10 years, I have navigated diverse architectures and untangled some of the most frustrating, elusive bugs in production. I believe that every production incident is a massive learning opportunity. Having discovered hibug.me, I am excited to contribute to this knowledge base, breaking down root causes, and sharing verified fixes to help fellow developers debug faster and build more resilient systems.
Related Stories
Route 53 failover not triggering while the primary region was down
Choosing an AWS disaster recovery strategy by RPO and RTO, workload by workload
Comments
0No comments yet. Be the first to comment!
Sign In to post a comment.