How to Write a Runbook

Runbooks are easier to use when they follow a consistent structure. When an alert fires, the responder should already know where to find the impact, procedure, escalation criteria, and supporting information.

This guide walks through a basic runbook template and what belongs in each section.

Make the Runbook Easy to Find

If the observability or alerting platform supports it, link the runbook directly from the alert and include its URL in the alert payload.

annotations:
  summary: "Increased errors on API Gateway {{ $labels.service }}"
  runbook_url: "https://docs.example.com/runbooks/api-gateway-errors"

The link needs to survive the full notification path. If Alertmanager delivers the alert through PagerDuty, Slack, or another system, the responder should still receive it. Test the link when you test the alert.

The Template

Start with one structure and use it consistently. The template below keeps the information needed during a response in predictable places.

Name

Name the runbook after the alert it supports. The name seen in PagerDuty, Slack, or another notification should correspond naturally to the runbook.

Increased Errors on API Gateway {service}

{hostname} Memory Usage Over {threshold}

Overview

The Overview confirms which alert the runbook supports. Include a screenshot of the alert exactly as the responder normally receives it, such as the PagerDuty event or Slack notification.

This runbook is to be used when `Increased Errors on API Gateway {service}` is triggered.

Detailed service or architecture documentation does not belong here. Link to it from Support Notes when it is useful.

Impact

Impact helps the responder determine how the alert should be handled. For a redundant service, the response might look like this:

These values are illustrative, not universal severity definitions. Capture conditions that change the required response here so the responder does not need to page an engineer simply to determine whether the condition is important.

Procedure

Procedure gives the responder an overview of what they are about to do.

This procedure verifies the state of the service on the affected host, checks for recent errors, and performs the approved recovery action when appropriate.

Steps

Write the runbook while someone who understands the system is actually performing the procedure. Capture the commands they use instead of reconstructing them later from memory.

Include representative output when it helps the responder understand what they are looking at or decide what to do next.

● example.service - Example Service
     Loaded: loaded
     Active: active (running)

Do not include output simply because it exists. Make decision points explicit. If restarting an instance is safe only when another instance is healthy, checking the other instance should be a documented step. Include only actions the intended responder is authorized to perform.

Escalation

Explain when the responder should stop following the procedure and what happens next.

If the procedure does not restore the service, the condition meets the High impact criteria above, or the responder encounters a condition not covered by this runbook, stop the procedure and escalate the issue.

The escalation mechanism may differ by impact. A low-impact issue might become a ticket, while an outage may require paging the owner or starting the incident process.

Support Notes

Use Support Notes for information that can help an investigation but does not belong in the main procedure, such as dashboards, previous incidents, or useful service documentation.

## Support Notes

### Dashboards

- API Gateway Service Overview
- API Gateway Error Rates

### Previous Events

- INC-1234: Increased errors caused by an unhealthy backend instance
- INC-5678: API Gateway errors following a deployment

### References

- API Gateway Service Documentation

Keep Support Notes supplemental. A responder should not need to work through old tickets before beginning the normal procedure.

Revision History

Identify who last updated the runbook and when it was last reviewed.

If runbooks are stored in Git, do not require a separate manual version number because the repository already maintains document history. Review runbooks when the service or procedure changes, and correct them when responders discover stale commands, unclear steps, or outdated escalation information.