Site Reliability Engineering treats operations as a software problem: define reliability numerically, spend a fixed budget of allowed failure deliberately, and use incidents to make the system better rather than just restoring it.
SLI — Service Level Indicator
A precisely defined, measured metric of some aspect of the service actually experienced by users. Not a target — just the number.
Common SLIs
Availability — proportion of successful requests. Latency — proportion of requests faster than a threshold. Throughput — requests served per second. Error rate — proportion of requests that failed.
SLO — Service Level Objective
A target value or range for an SLI, over a window, that the team commits to internally.
99.9% of API requests succeed, measured over a rolling 30-day window.Pick SLOs users would actually notice breaching
An SLO tighter than what users can perceive just burns engineering effort with no real benefit. An SLO looser than what users actually need lets you ship an experience that quietly disappoints them while every dashboard stays green. Anchor it to real user impact, not a round number that felt aspirational.
SLA — Service Level Agreement
A contractual commitment to an external party, usually with a financial or credit penalty for missing it. An SLA is typically set looser than the internal SLO it's backed by — you want to breach your own SLO and get a warning long before you're anywhere near breaching a customer-facing SLA.
Error Budget
The allowed amount of unreliability implied by an SLO — 100% minus the SLO.
Error Budget = 100% − SLO99.9% SLO
= 0.1% allowed unavailability
≈ 43 minutes per monthAn error budget reframes reliability as something to spend deliberately, not just protect. Budget remaining → ship features, take reasonable risks, run chaos experiments. Budget exhausted → the team's priority shifts to reliability work until it recovers, by policy, not by argument.
›What happens when an error budget is exhausted?
By policy — agreed before the incident, not negotiated during one — feature work pauses and the team prioritizes reliability fixes until the budget recovers. This is the actual mechanism that makes SLOs meaningful: without a consequence for exhausting the budget, an SLO is just a dashboard number nobody acts on.
MTTR, MTBF, MTTD
MTTD — Mean Time To Detect
How long between a failure starting and someone (or something) noticing it. Driven by monitoring and alerting coverage — a failure with no alert on it has effectively infinite MTTD.
MTTR — Mean Time To Resolve
How long between detection and full resolution. Driven by runbooks, observability depth, and how practiced the team is at the specific failure mode — this is what a good troubleshooting sequence directly shortens.
MTBF — Mean Time Between Failures — the average interval between incidents. Improving MTBF is about prevention (better testing, safer deploys, capacity headroom); improving MTTD/MTTR is about response.
Incident Response
Detect and declare
An alert fires, or someone notices. Declare an incident early rather than investigating quietly first — it pulls in help sooner and starts the timeline that later informs the postmortem.
Assign an incident commander
One person owns coordinating the response — not necessarily the person fixing it. This keeps communication and decision-making from bottlenecking on whoever's heads-down in the code.
Mitigate before you root-cause
Roll back, fail over, or scale up to restore service first. Understanding the exact root cause can wait until users aren't affected — mitigation and diagnosis are different tasks with different urgency.
Communicate on a cadence
Regular, predictable updates (even "still investigating, next update in 15 minutes") to stakeholders reduce the number of people interrupting the responders to ask for status.
Resolve, then write the postmortem
Once mitigated, write a blameless postmortem while details are fresh — timeline, impact, root cause, and concrete follow-up actions with owners and dates.
Postmortems
Blameless, not toothless
Blameless means the postmortem doesn't punish individuals for honest mistakes — psychological safety is what gets people to report the full, accurate timeline instead of a defensive one. It does not mean skipping concrete action items; a postmortem with no follow-through is just a story.
Toil
Toil is manual, repetitive, automatable work tied directly to running a service in production that scales linearly with its size — not all operational work, specifically the kind that provides no lasting engineering value. Google's SRE guidance caps toil at roughly 50% of an SRE's time; the rest goes to engineering work that reduces future toil (automation, better tooling, fixing root causes instead of symptoms).
Capacity Planning
Forecast demand ahead of when it arrives, and provision (or configure auto scaling limits) with enough headroom to absorb normal growth and a reasonable spike — reactive capacity planning is how "unexpectedly high traffic" becomes an incident instead of a graph.
Alerting
Alert on symptoms, not causes
Page on "users are experiencing errors," not "CPU is at 80%" — high CPU alone might be fine. Symptom-based alerting (rooted in SLIs) pages for things that actually matter to users and stays quiet through internal fluctuations that don't.
Reliability Patterns
- Redundancy — no single instance, AZ, or dependency is a single point of failure.
- Graceful degradation — a non-critical dependency failing degrades a feature instead of the whole request.
- Circuit breakers — stop calling a failing dependency for a cooldown period instead of piling up retries against it.
- Retries with backoff and jitter — retry transient failures, but with increasing delay and randomization so a recovering dependency isn't immediately hit by a synchronized wave of retries.
- Bulkheads — isolate resource pools (connections, threads) per dependency so one slow dependency can't exhaust resources needed by every other request path.