Reliability: SLOs, error budgets and incidents

Perfect reliability is infinitely expensive and nobody needs it. SRE turns "be reliable" into a number you can budget, alert on and trade against new features.

Prometheus / Sloth (SLO rules)PagerDuty / OpsgenieStatuspageincident.io / FireHydrantblameless postmortem templateschaos engineering tools

SLI, SLO and error budget

SLI (): a measurement of the service as users experience it, such as the fraction of requests that succeed fast enough.
SLO (): the target for an SLI over a window, for example 99.9% of requests succeed within 300 ms over 30 days.
SLA (service level agreement): a contract with consequences, usually looser than the internal SLO.
: the allowed unreliability (100% minus the SLO). While budget remains, ship fast; when it is spent, slow down and invest in reliability.

Why it changes decisions

The budget ends the argument between "ship more" and "be more stable" by making it a shared, measurable trade. It also tells you what to alert on: page when you are burning the budget fast enough to exhaust it early, and file a ticket when the burn is slow.

availability SLI and a fast-burn alertpromql

# SLI: ratio of good requests over 5 minutes
sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# fast burn for a 99.9% SLO: error ratio above 14.4x the allowed rate
(1 - (sum(rate(http_requests_total{status!~"5.."}[1h])) / sum(rate(http_requests_total[1h])))) > (14.4 * 0.001)

Incident response

Declare early and name roles: an who coordinates, someone who investigates, someone who communicates.
Mitigate first, diagnose second: roll back, fail over, turn off the flag.
Communicate on a schedule, even when there is nothing new to say.
Keep a timeline as you go; it is the raw material for the review.
Hold a blameless postmortem: ask how the system allowed the failure, not who made the mistake. End with specific, owned actions.

Reducing risk before it happens

Remove toil: automate repetitive manual operations so people have time for engineering.
Test failure on purpose with game days and fault injection.
Capacity planning and load testing before the traffic arrives.
Runbooks for known failures, kept in step with the system.
Graceful degradation: when a dependency fails, do less, not nothing.

SLIservice level indicatorA measurement of service quality as users experience it, such as the fraction of requests that succeed quickly.see in glossary →SLOservice level objectiveThe target for an SLI over a window. It defines how reliable is reliable enough.see in glossary →Error budgetThe allowed unreliability, 100% minus the SLO. While it remains you can take risks; when spent, reliability work comes first.see in glossary →Incident commanderICThe person who owns the response and makes decisions, while others do the hands-on work. Keeping these roles apart prevents the leader from disappearing into the technical details.see in glossary →

connections on the map

Observability: metrics, logs and tracesfeeds the SLOs ofReliability: SLOs, error budgets and incidents
Release strategies and rollbackspends the budget ofReliability: SLOs, error budgets and incidents

updated on 14/10/2026 · sources to go deeper · report an error

in other domains

SOC SLOs and continuous tuningsecopsSLOs and error budgets, applied to detection

Study before

Observability: metrics, logs and traces

test what you learned

Loading questions…