The Magic of Error Budgets - Turning Failure into Strategy

One of the usual scenario I have seen with a product team is releasing new features at lightning speed. While that was great for innovation, the incident volume might started rising, & so did customer complaints.
Although they made sure to have dashboards, alerts, & even SLIs/SLOs in place. Still they might miss something crucial: A way to balance velocity & reliability.
That’s when Error Budgets are introduced, & everything changed.

What is Error Budget?

An Error Budget is the amount of unreliability you’re allowed in a system without breaking your SLO.

Let’s say your SLO is 99.9% availability per month. That means: You’re allowed to fail for 0.1% of the time and that's 43 minutes & 12 seconds of downtime per month.

That "allowed failure window" is your error budget given your SLI's caused DownTime in Application is ZERO. Otherwise Error Budget is Available Application DownTime(SLO's Based) - Application Downtime caused by Application SLI's.

And believe it or not, it’s not a bad thing to use it.

Why It's a Game-Changer?

Before using the error budgets, reliability & innovation constantly clashed:

But there was no common language to make decisions. Error budgets gave us that language.

Trade-Offs: Reliability vs Velocity

Error budgets help answer questions like:

Say, if we burned through our monthly error budget in just 7 days due to back-to-back outages. Instead of blame, the conversation need to be changed: Let’s hold off on new releases & invest in fixing alert fatigue & DB failover logic. By the end of the month, not only reliability metrics will recover, but the next sprint will be cleaner & more focused.

Using Error Budgets Effectively

Reliability must be considered as a Product Feature.

Summary and Conclusions

In Site Reliability Engineering(SRE), perfect is the enemy of good.

Aiming for 100% uptime can be wasteful & unsustainable. Instead, agree on what "good enough" looks like & stick to it.

Author

Sagar Mehta is Atgen Software Solutions Founder and a recognised expert in the field of Intelligent Automation, including Robotic Process Automation, Workload Automation, DevOps, SRE and Advanced Analytics. Sagar advocates a pragmatic approach to Automation, encouraging a policy of using ‘the best tool for the job’.

Prior to co-founding Atgen Software Solutions, Sagar worked in Senior Automation roles, architecting and delivering robust, scalable solutions for many of the world’s biggest banks and working with leading Automation vendors. He developed his first automated solution in 2006 and has continued to deliver robust, scalable and sophisticated Automation ever since.

Sagar is a regular guest speaker and panellist at Automation seminars, conferences and user group events.