Monitoring
Alerts and SLOs
Define an SLO per endpoint and alert when the error budget burns faster than it should. Alerts fire on the condition you set rather than on every error.
Rule types
Add a rule in the Alerts tab. There are three kinds:
| Rule | Fires when |
|---|---|
| Threshold | A search you write matches more than N times in a window |
| New issue | A problem is seen for the first time |
| Regression | An issue you resolved comes back |
A threshold rule takes the same query you'd type in search, so you can get it returning exactly the right thing before you save it as a rule:
level = 'error' AND path LIKE '/v1/checkout%' AND statusCode >= 500The other two need no query and catch what a threshold misses: something new appearing, and something you thought was fixed returning.
Where alerts go
| Destination | |
|---|---|
| Slack | Posts to an incoming webhook |
| Webhook | Sends JSON to any endpoint you own |
Each rule has its own cooldown, which is what stops one flapping endpoint sending 400 Slack messages.
Most alert storms are one bad release. A cooldown that outlives a rollback turns forty messages into two.
Why an SLO rather than a threshold
"More than 10 errors in 5 minutes" fires when a crawler retries a dead link. An error budget asks a better question: at this rate, will we miss our target? A brief spike that a healthy service absorbs never fires.
An SLO sets the target. The error budget is what the target permits you to spend. A burn-rate alert fires when you're spending it fast enough to run out.
Setting one
- Pick the endpoint that matters
Your checkout endpoint has an SLO. Your health check does not.
- Set a target you'd defend
99.9% is not automatically right. Pick the number you'd actually roll back a release over.
- Alert on the burn rate
Fast burn means wake someone. Slow burn means a ticket.
Choosing a target
99% allows about 7 hours of failure a month. 99.9% allows about 43 minutes. 99.99% allows about 4 minutes, which is less time than a deploy takes.
Setting a target you can't hold means an alert that fires constantly and gets muted, which is worse than no alert.
Alerts that get ignored
The failure mode of alerting is not missing an alert. It's having so many that people stop reading them.
Alerts and releases
Most budget burn starts at a deploy. When an alert fires, facet the issue by release first: if it's confined to the newest one, the fix is a rollback, not an investigation.