Agent Skills: Prometheus Runbooks

Prometheus Runbooks

When invoked, you read the active alerts from a Prometheus server, then for every firing alert you propose a concrete runbook: the most likely cause, the ordered steps to investigate and remediate, how to verify the fix, and when to escalate. The skill is generic: it works against any Prometheus instance and any alert names, using the alert labels and annotations plus the pattern library below.

Global Context

• User request / scope: $ARGUMENTS — empty, or a Prometheus base URL (for example http://localhost:9090), optionally followed by an alert name to focus on.
• Engine: scripts/fetch_alerts.py (Python standard library only, no third-party dependencies).
• Default Prometheus URL: http://localhost:9090, overridable by the first argument or the PROMETHEUS_URL environment variable.
• Output: a runbook proposal printed to the user. Nothing is written to disk and nothing is changed on the Prometheus server.

Rules

• The alert data comes only from the Prometheus HTTP API (/api/v1/alerts). Never invent alert names, label values, or $value numbers — read them from the script output.
• Read-only against the monitored systems. You only fetch alerts; you never silence, delete, or modify alerts, and you never run remediation commands yourself. You propose steps for a human to run.
• A proposed runbook step is a suggestion, not a guarantee. Always tell the user to confirm the cause before acting on production.
• Do not add comments to any command you run.

Steps

1. Resolve the Prometheus URL. Use, in order: the first word of $ARGUMENTS if it looks like a URL, then $PROMETHEUS_URL, then http://localhost:9090. If a second word is present treat it as an alert name to focus on.

2. Fetch the active alerts:

   python3 scripts/fetch_alerts.py <prometheus-url>
   

   The script prints a JSON array. Each item has name, state (firing or pending), severity, labels, annotations, activeAt, and value.

3. If the script fails to connect, stop and tell the user the URL it tried and that the Prometheus server is unreachable. Do not fabricate alerts.

4. Filter to state == "firing" (mention pending alerts only as a heads-up). If an alert name was given in $ARGUMENTS, keep only that alert. If there are zero firing alerts, say so plainly and stop.

5. For each firing alert, build a runbook proposal:

   • Read the alert annotations. If a runbook_url annotation exists, surface it first — it is the team's own runbook and takes priority over the generic library.
   • Match the alert against the Pattern Library below by alertname, then by severity and by the metric referenced in the summary/description annotations. Matching is fuzzy: HighErrorRate, ErrorBudgetBurn, and 5xxTooHigh all map to the Errors pattern.
   • If nothing matches, fall back to the Unknown Alert procedure.

6. Present the result using the Output Format. Order alerts by severity: critical first, then warning, then everything else.

Pattern Library

Map the alert to one category, then propose the matching steps. Adapt the steps to the alert's labels (use the real instance, job, namespace, pod, etc.).

Availability — target / instance down

Signals: up == 0, InstanceDown, TargetDown, TargetMissing, KubePodNotReady, probe failures. Proposed steps:

1. Confirm the target is really down vs. a scrape problem: open http://<prometheus>/targets and check the endpoint's last error.
2. Reach the instance directly (curl its metrics or health endpoint; ping/telnet the host:port).
3. Check the process/pod state on the host (systemctl status <svc>, podman ps, kubectl get pod <pod> -n <ns>).
4. Check for resource exhaustion that killed it (OOM in dmesg/kubectl describe pod, disk full).
5. Restart the service/pod if the cause is a crash; capture logs first. Verify: the target returns to up == 1 and the alert clears.

Errors — high error / failure rate

Signals: HighErrorRate, 5xx, ErrorBudgetBurn, app_error_rate, error ratios above threshold. Proposed steps:

1. Correlate with recent changes: was there a deploy/config change in the alert's window (activeAt)? If yes, consider rollback.
2. Read the service logs filtered to errors for the affected instance/job.
3. Check the health of downstream dependencies (DB, cache, upstream APIs) — a dependency failure shows as your errors.
4. Check whether errors are uniform or isolated to one instance/region (label breakdown).
5. Mitigate: roll back the suspect deploy, or shed/route traffic away from the bad instance. Verify: error ratio drops below threshold for the alert's for duration.

Saturation (CPU) — high CPU / load

Signals: HighCPU, CPUThrottlingHigh, load average alerts. Proposed steps:

1. Identify the hot process (top/htop, kubectl top pod).
2. Decide runaway vs. genuine load: check request rate alongside CPU.
3. If genuine load: scale out (add replicas) or scale up (bigger instance/limits).
4. If runaway: capture a profile/thread dump, then restart the process. Verify: CPU returns under threshold and latency/error alerts also clear.

Saturation (Memory) — high memory / OOM

Signals: HighMemory, app_memory_usage_ratio, OOMKilled, container memory near limit. Proposed steps:

1. Confirm trend: steady climb (leak) vs. spike (load).
2. For a leak: capture a heap dump if cheap, schedule a restart to restore service, then investigate.
3. For load: raise the memory limit/request or scale out.
4. Check for OOMKilled restarts in pod/container events. Verify: memory usage stabilizes below threshold; no new OOM kills.

Saturation (Disk) — disk filling / full

Signals: DiskWillFillIn, DiskFull, NodeFilesystemAlmostOutOfSpace. Proposed steps:

1. Find the full filesystem and the biggest consumers (df -h, du -xh <mount> | sort -h | tail).
2. Reclaim safely: rotate/compress/ship logs, prune old artifacts, clear caches/temp.
3. If it is data growth, expand the volume.
4. Check whether log spam from another incident is the real cause. Verify: free space recovers and the projected fill time clears the alert.

Latency — slow responses

Signals: HighLatency, HighRequestLatency, p95/p99 latency alerts. Proposed steps:

1. Locate the slow hop: app, DB, cache, or downstream dependency.
2. Check DB slow queries / lock contention and cache hit ratio.
3. Check GC pauses and CPU saturation on the affected instance.
4. Mitigate: scale out, add caching, or roll back a slow change. Verify: p95/p99 returns under the SLO threshold.

Queue / backlog — work piling up

Signals: HighQueueDepth, ConsumerLag, KafkaConsumerLag, backlog alerts. Proposed steps:

1. Confirm producers up and consumers running/healthy.
2. Scale out consumers; check for a poison message blocking a partition.
3. Check the downstream the consumers write to — backpressure pushes the queue up. Verify: queue depth/lag drains back to baseline.

Certificates / expiry

Signals: CertExpiry, SSLCertExpiringSoon, certificate-days-left alerts. Proposed steps:

1. Identify the cert and endpoint from the labels.
2. Renew/rotate the certificate and reload the serving process.
3. Confirm the new expiry date and automate renewal if it is manual. Verify: days-to-expiry returns above threshold.

Unknown Alert procedure

When no pattern matches:

1. Restate what the alert measures from its summary/description annotations and the metric in the expression.
2. Use the labels to scope blast radius (which instance/job/namespace/team).
3. Propose a generic triage: confirm the signal in Prometheus (/graph), read the relevant component's logs, check recent changes, and identify the owning team from labels.
4. Recommend the team add a runbook_url annotation so the next responder has a real runbook.

Output Format

Start with a one-line summary: how many alerts are firing and the highest severity. Then one block per firing alert:

## <alertname>  [<severity>]
- Instance: <instance / job / pod from labels>
- Firing since: <activeAt>
- Signal: <one line from annotations, with the real $value>
- Likely cause: <your best single hypothesis>
- Runbook:
  1. <step>
  2. <step>
  3. <step>
- Verify: <how to confirm it is resolved>
- Escalate if: <condition + owning team from labels, if present>

End with a short "Confirm before acting" note: these are proposed steps; the responder must verify the cause before running anything against production.