Design actionable alerting rules

A page should mean "users are hurt or soon will be, and a human must act now". Pages on causes (CPU at 80%, a pod restarted, a queue non-empty) fire when nothing is wrong and stay silent when something new breaks. Alerts on symptoms users feel (errors, latency, freshness, availability) tied to SLOs catch every cause. Multi-window, multi-burn-rate alerts on the error budget page fast for severe problems and open tickets for slow burns, with few false positives. Everything else is a ticket, a dashboard, or deleted.

Design the alerts for:

Write rules in generic format.

1. State the SLOs you will alert on. If none are given, propose provisional SLIs and targets from the service's purpose (availability as successful requests over valid requests, latency as the share of requests under a threshold, freshness for pipelines), mark them as assumptions, and recommend confirming them.
2. Design burn-rate alerts per SLO. Default for a 30-day window: page when 2% of the budget burns in 1 hour (burn rate 14.4, checked over 1 hour and 5 minutes), page when 5% burns in 6 hours (burn rate 6, over 6 hours and 30 minutes), and open a ticket when 10% burns in 3 days (burn rate 1, over 3 days and 6 hours). Show the arithmetic for this service's target. Adjust if traffic is too low for ratios to be meaningful, and say how (minimum request counts, longer windows, synthetic probes).
3. Add the few cause-based alerts that are worth paging on because they predict imminent user harm with no symptom yet: certificate expiry within days, disk full within hours at the current growth rate, a dead-letter queue growing, a job that has not succeeded within its window. Prefer predictive forms (time to full) over static thresholds.
4. For every alert define: name, expression, for duration, severity (page or ticket), owner, a summary that says what users are experiencing, and a runbook link placeholder.
5. Routing: page versus ticket, quiet hours for non-urgent alerts, grouping and inhibition so one outage produces one page, and dependency-aware suppression.
6. Review the existing rules and page history: list alerts to delete, demote to a ticket or dashboard, or merge, with the reason (fired without action, duplicate, cause not symptom, threshold never meaningful).

• Use only metric names and labels from the input; where you need one that is not there, write it as a placeholder and list it under Gaps.
• Every paging alert must be actionable and have an owner and a runbook placeholder. If you cannot say what the responder would do, it does not page.
• Do not alert on averages for latency; use percentiles or threshold ratios.
• Keep the total number of paging alerts small; justify each one beyond the SLO burn-rate alerts.
• Separate what you verified from what you inferred. Mark inferences as such.
• When you do not know, say "I don't know" once and state what would settle it.

Assumptions

Bullets, including provisional SLOs.

Alert design

A table: alert, type (burn-rate, predictive, cause), severity, why it pages or tickets, what the responder does.

Rules

One fenced block with all rules in the chosen format.

Routing

Bullets or a routing config sketch.

Delete or demote

A table: existing alert, action (delete, demote, merge), reason.

Gaps

Missing metrics or instrumentation needed, or "None".