Learnings
An alert is a request for a decision
- observability
- SLOs
- Prometheus
Start with the user
CPU usage is a diagnostic clue. Failed requests are a user-visible symptom. Page on conditions that require a timely human decision; put explanatory signals on a dashboard.
Use a service-level objective
Define a good event, a total event, and a measurement window. For a request-based service, availability might be successful requests divided by eligible requests. Decide explicitly whether client errors count.
# Illustrative error ratio; adapt selectors to your service.
sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_requests_total{job="api"}[5m])), 1e-9)This is a diagnostic query, not a complete paging rule. A multi-window burn-rate alert combines a sustained budget burn with a shorter confirmation window. Missing telemetry needs its own signal; an absent series is not evidence of good health.
Write the decision into the alert
- What user experience is affected?
- Which dashboard confirms the impact?
- Which runbook contains the first useful step?
- Who owns the service if the first responder cannot recover it?
If there is no useful action, reconsider whether the signal belongs in the paging channel.