On-call rotation design

Reference Card

• Origin/Prior Art: IT operations and telecom pagers in the 1990s.
• Related Practices: Alert routing, Runbooks, Escalation policies.
• Key Concepts: Primary/Secondary tiers, Follow-the-sun routing, Compensation.
• Primary Failure Modes: Alert fatigue, Single-person silos, Lack of management support.

Introduction

Software runs 24 hours a day, but humans do not. When a database crashes at 3:00 AM on a Sunday, someone must fix it.

On-call is the system of scheduling engineers to be available outside of standard working hours to respond to critical production incidents. A poorly designed on-call rotation destroys engineering morale, accelerates turnover, and ultimately results in longer outages because exhausted engineers make poor decisions. A well-designed rotation ensures reliable system uptime while aggressively protecting the health and personal lives of the team.

Mental Model

Think of a fire station. Firefighters do not work 24/7/365. They work in strict shifts. When they are on shift, they sleep at the station and are expected to be in the truck within 60 seconds of the alarm sounding. When their shift ends, they go home, turn off their radios, and are completely disconnected from work.

On-call engineering must follow the same model: rigid, predictable shifts of high responsibility, followed by guaranteed periods of complete disconnection.

Operational Pattern

An effective on-call rotation is built on several structural pillars:

1. Primary and Secondary Tiers: The Primary engineer receives the initial alert. If the Primary is deeply asleep, driving, or in an area without cell service, the alert automatically escalates to the Secondary engineer after 5-10 minutes. This provides a safety net for both the system and the humans.
2. Shift Lengths: Rotations are typically scheduled in 1-week blocks (e.g., Tuesday 10:00 AM to the following Tuesday 10:00 AM). Changing shifts mid-week ensures weekends are shared fairly across the year.
3. Escalation Policies: If an alert is not acknowledged by the Primary or the Secondary within 15 minutes, the system must automatically escalate to engineering management. Management must feel the pain of a broken alerting system to ensure it gets fixed.
4. Follow-the-Sun: Global organizations eliminate overnight shifts entirely by routing alerts to whichever engineering office is currently in daytime hours (e.g., routing alerts to the London office while the San Francisco office sleeps).

Failure Modes

The Hero Culture

If a single senior engineer is the only person who understands the deployment pipeline, they end up getting called for every outage, even when they are not officially on call. This “hero” creates a single point of failure and will inevitably burn out. Management must enforce that the team handles the outage. If the on-call engineer doesn’t know how to fix a system, it highlights a failure in documentation and cross-training, not a failure of the engineer.

Unactionable Alerts (Pager Fatigue)

If an engineer is woken up at 3:00 AM because CPU usage spiked to 80% for two minutes and then recovered on its own, they have suffered “pager fatigue.” Sleep was destroyed for an unactionable event. Pages must only fire for SEV-1 or SEV-2 events - situations where a human must actively intervene immediately to stop business impact. Everything else should be routed to a Slack channel or Jira backlog for daytime review.

Security

When designing rotations for security teams (InfoSec), ensure they have a direct escalation path to legal counsel and executive leadership at all hours. Operational engineers restore uptime; security engineers handle active breaches, which often require immediate legal decisions regarding data disclosure and system isolation.

Operational Guidance

On-call work is a massive disruption to personal life. Engineers cannot go to a movie theater, hike in the mountains, or have a few drinks while on call, because they must remain near their laptop and sober.

Organizations must compensate engineers for this disruption. This is typically done through extra pay per on-call shift or explicit time off in lieu (e.g., taking Friday off after a grueling week on call). If an organization treats on-call as “free labor,” engineers will leave for organizations that do not.

Always ensure your alerting tool (e.g., PagerDuty, Opsgenie) is configured to override “Do Not Disturb” modes on mobile phones for high-severity alerts.

Related topics

• Incident severity classification

Sources & further reading

• Google SRE Book: Being On-Call - O'Reilly Media