Incident and operations
Alerts, on-call triage, mitigation, runbooks, observability and postmortems.
Download all 65
Start here 1
The general, start-from-zero entry for a need in this category: it works from a blank start, asks for what it needs and hands off to the specific entries. Example: Plan a trip.
- Triage a production alert
Turns a firing production alert into a severity call, the safest mitigation to try first, ranked hypotheses and the next checks. Use in the first minutes of an incident or page.
Specific 41
One situation, audience, tool or format inside the category. Example: Plan a road trip with children.
- Write an incident status update
Writes a clear status update for an ongoing incident, tuned to customers, internal teams or executives, without speculation or promises the team cannot keep. Use for status pages, Slack and email.
- Write a blameless postmortem
Turns incident notes, chat logs and timelines into a blameless postmortem with impact, timeline, contributing factors and owned action items. Use after an incident is resolved.
- Incident commander
Runs a live incident like an experienced incident commander, assigning roles, keeping a steady comms cadence and driving mitigation before root cause. Use as the coordinating voice during an outage.
- Logging rules
Standing rules for logs an assistant writes, with structured fields, meaningful levels, no secrets or personal data, correlation IDs and errors logged once where they are handled.
- Site reliability engineer
Acts as a site reliability engineer who thinks in SLOs and error budgets, automates toil, designs for failure and writes blameless reviews.
- Build an incident timeline
Builds a timestamped incident timeline from chat logs, alerts and deploy records, marking detection, escalation, mitigation and the gaps between them. Use when preparing a postmortem.
- Design actionable alerting rules
Designs actionable alerts from SLOs and user-facing symptoms, with thresholds, routing, runbook links, and a list of noisy alerts to delete. Use when pages are noisy or real outages go unnoticed.
- Design an on-call rotation
Designs an on-call rotation with schedule, escalation, handoff, alert ownership, compensation norms and health checks. Use when starting on-call or when the current one burns people out.
- Write an operational runbook
Writes a runbook for an alert or routine procedure with symptoms, diagnosis commands, ordered mitigations, verification and escalation. Use so on-call engineers can act without tribal knowledge.
- Plan a game day or chaos exercise
Plans a game day or chaos exercise with failure scenarios, hypotheses, blast-radius limits, abort criteria, roles, an observation checklist and a follow-up review. Use to test resilience.
- Write an on-call handoff
Writes the end-of-shift on-call handoff covering open incidents, alerts that fired and why, silences and their expiry, risky changes in flight and what to watch. Use at every rotation change.
- Instrument a service for observability
Plans and adds logs, metrics and traces using OpenTelemetry conventions, golden signals, useful log fields, cardinality limits and first dashboards. Use when a service is hard to debug in production.
- On-call readiness track
Gets a new service ready for on-call in gated steps, from SLOs on user journeys to symptom alerts, a dashboard, runbooks per alert and an escalation and go-live check.
- Prune noisy alerts
Analyses an alert history export to cut alert fatigue, finding alerts nobody acts on, flapping, duplicates and missing owners, with a delete, tune, route or keep decision per alert.
- Rehearse incident command
Runs a simulated incident with the learner as incident commander, injecting alerts, confused responders and executive pings turn by turn, then debriefs coordination and communication.
- Write observability queries
Writes PromQL, LogQL, TraceQL, SQL or vendor queries for an operational question, explains each part and warns about traps such as rate windows, counter resets and label cardinality.
- Add logs, metrics and traces to a service
Instruments a service in gated steps with structured logs, metrics, traces, correlation ids, business metrics, dashboards as code and a verification run. Use when a service is a black box.
- Audit postmortem action items
Reviews action items across recent postmortems for done, stale and vague items and repeated systemic themes, and rewrites each open item to be specific, owned, dated and verifiable.
- Design a service dashboard
Designs an operational dashboard for one service, with golden signals on top, dependencies, saturation, deploy annotations and drill-down order, plus the query and on-call action for each panel.
- Design synthetic monitoring checks
Designs synthetic checks and uptime probes for a service's key user journeys, with check types, locations, frequency, test accounts, data cleanup and alert rules that avoid single-probe false pages.
- Investigate an error-rate spike
Works a live spike in 5xx responses or exceptions by isolating which endpoints, versions, hosts and dependencies it follows, ending in the safest mitigation to try first.
- Investigate a latency spike
Walks an on-call engineer through a live latency spike one piece of evidence at a time, from percentile and endpoint to deploys, saturation or a slow dependency, and the safest mitigation.
- Quantify an incident's impact
Estimates how many users, requests, orders or revenue an incident affected from logs and metrics, with the method, the queries, a confidence range and stated assumptions.
- Audit a team's runbook library
Audits a team's runbooks for stale commands, missing verification and rollback, dead links and alerts that have no runbook, and returns a prioritised fix list.
- Calculate availability against an SLA
Calculates a period's availability against the SLA text you paste, applying its own downtime definition, exclusions and measurement rules, and shows each step so credits can be checked.
- Decide between rollback and fix forward
Weighs rolling back, disabling a flag or shipping a fix forward during a live incident, checking whether rollback is actually safe after migrations, data writes or dependency changes.
- Facilitate a postmortem review meeting
Prepares the facilitator for a blameless postmortem meeting from the draft, with a timed agenda, questions per timeline gap, phrasing that steers off blame, and how to land owned action items.
- Investigate a sudden cloud cost spike
Investigates a sudden cloud bill jump or cost anomaly alert, finding the runaway resource, loop or abuse and stopping it safely before more hours are billed.
- Respond to a third-party provider outage
Handles an outage at a critical SaaS or API provider - confirms it, degrades gracefully, queues work, keeps users informed and recovers the backlog when it returns.
- Set up a public status page
Sets up a public status page with components that match how users see the product, status definitions, incident templates, update cadence, ownership and monitoring links.
- Stop a runaway notification storm
Stops a bug that is sending duplicate or wrong emails, pushes or SMS to users, with a kill switch, blast-radius count, queue cleanup, deduplication and an apology plan.
- Triage a mobile crash spike
Triages a crash-rate spike after a mobile release, isolating affected versions, devices and OS, server or flag causes, and deciding whether to halt rollout, kill-switch a feature or hotfix.
- Write an incident response plan
Writes an engineering incident response plan - severity levels with criteria, response targets, roles, escalation paths and copy-paste communication templates - as a quick reference for on-call.
- Investigate a database CPU spike
Works a live database CPU spike with the on-call engineer, finding the queries, plan flips, new callers or jobs behind it, then picking a safe mitigation before the permanent fix.
- Fix a simulated broken server
Hands the learner a simulated Linux server with a hidden fault such as a full disk, a dead service or bad permissions, answering their commands until they find and fix it.
- Escalate an incident to a vendor's support
Writes the support case to a cloud or SaaS vendor during an incident, with the right severity, exact identifiers, evidence, what you ruled out and a clear ask, plus follow-up steps.
- Analyse pasted logs from an incident
Reads pasted log excerpts from an incident to find the first real failure, the causal chain and the noise to ignore, and names the next query to run.
- Investigate memory growth in production
Works rising memory or OOM restarts in production with the on-call engineer - sizing time to failure, mitigating first, then capturing heap evidence safely from one instance for the leak hunt.
- Recover from an expired TLS certificate
Restores service after an expired or misconfigured TLS certificate, covering every place the certificate lives, chain errors, pinned clients and expiry monitoring.
- Respond to a full disk in production
Handles a full disk or volume in production by freeing space safely, finding what grew, avoiding deleting open or needed files, and preventing the next one.
- Set up error tracking for an app
Sets up an error tracking tool such as Sentry with release tagging, source maps or symbols, useful grouping, PII scrubbing, sampling and alert rules that do not drown the team.
Advanced 23
Expert or deep work that assumes the user can judge expert output. Example: Plan croissant lamination.
- Define SLOs and burn-rate alerts
Defines SLIs, SLOs and an error-budget policy from a service's user journeys, with multi-window burn-rate alert rules. Use when alerting is noisy or reliability targets are vague.
- Diagnose database replication lag
Diagnoses growing database replication lag during an incident, from long transactions and write bursts to replica resources, and protects reads that depend on fresh data.
- Handle database connection exhaustion
Works a live "too many connections" incident, finding leaking or idle-in-transaction clients, freeing connections safely and capping pools across all replicas.
- Triage an alert storm
Turns dozens or hundreds of alerts firing at once into a few incidents, finding the upstream cause through dependencies and timing, and silencing consequences safely with expiry.
- Automate a manual runbook safely
Turns a manual runbook into automation step by step, keeping human approval on risky actions, with dry runs, logging, idempotent steps and a manual fallback.
- Diagnose a production DNS failure
Diagnoses DNS failures in production, such as a bad record change, cached TTLs, resolver overload in Kubernetes, DNSSEC errors or an expired domain, with safe fixes.
- Execute a region or zone failover
Runs a failover to another region or zone during a provider outage, with go or no-go checks, data-loss assessment, traffic shift, verification and a failback plan.
- Investigate intermittent timeouts between services
Finds the cause of intermittent timeouts between services, such as pool exhaustion, retry amplification, DNS, packet loss, idle-connection resets or one bad host.
- Plan a production maintenance window
Plans a planned production maintenance with a minute-by-minute run sheet, go or no-go checks, a point of no return, a tested rollback, verification and a customer notice schedule.
- Reduce logging, metrics and tracing costs
Finds where observability spend goes, such as noisy logs, high-cardinality metrics, unsampled traces and long retention, and plans cuts ranked by savings without losing what incidents need.
- Run a backup restore drill
Runs a test restore of a production backup into an isolated environment, timing it against the recovery time objective, checking data integrity and recording the gaps found.
- Mitigate an abusive traffic surge at the edge
Mitigates a live surge of bots, scrapers or request floods with targeted blocks, rate limits and challenges at the edge, without locking out real users or partners.
- Run a monthly SLO and error budget review
Runs a recurring SLO and error budget review from the period's data, deciding on reliability work, alert changes and SLO adjustments, each with an owner.
- Recover from a growing queue backlog
Works a live queue or consumer-lag incident, finding whether producers surged, consumers slowed or a poison message is stuck, and plans a safe drain without overloading downstream systems.
- Investigate disk I/O saturation in production
Diagnoses high I/O wait and slow disks on a production host or volume from iostat and cloud volume metrics, including burst credit exhaustion, and picks a safe mitigation.
- Investigate a spike in login failures
Works a live spike in failed logins or users being logged out, checking the identity provider, token signing keys, cookies, clock skew, redirects and recent auth changes.
- Investigate a spike in failed payments
Works a spike in failed payments or checkouts, separating provider outages, issuer declines, 3-D Secure, webhook and configuration problems, while protecting customers from double charges.
- Recover from corrupted production data
Handles production data corrupted by a bug, bad script or migration - stops the writes, scopes the affected rows, and repairs from backups or history with checks before and after.
- Report incident trends for a quarter
Analyses a period's incidents for frequency, severity, detection and recovery times, recurring causes and repeat services, and writes a reliability review with honest caveats and a few priorities.
- Diagnose pods stuck Pending or being evicted across a cluster
Works a Kubernetes capacity incident where many pods sit Pending or get evicted, covering requests, node pressure, autoscaler failures, taints, quotas and preemption.
- Collect evidence for an ongoing incident, read-only
Gathers evidence for a live incident with read-only commands, covering recent deploys, error rates, logs and resource use, and writes a timestamped evidence summary. Use while responders work the fix.
- Prepare a service for a known traffic peak
Prepares a service for a predictable peak such as a launch, sale or broadcast, with a traffic estimate, capacity headroom, pre-scaling, limits, load-shedding, a change freeze and on-call cover.
- Recover from a cache outage or cold-cache stampede
Handles a Redis or Memcached outage or a cold-cache stampede, protecting the database, warming up gradually and adding request coalescing afterwards.
Not: a defect found in development (debugging).