Disaster recovery and RPO/RTO

Reference Card

• Origin/Prior Art: Enterprise IT continuity planning in the pre-cloud era.
• Related Practices: Multi-region architecture, Automated backups, Chaos engineering.
• Key Concepts: RTO (Recovery Time Objective), RPO (Recovery Point Objective), Active-Passive failover.
• Primary Failure Modes: Backups failing to restore during an actual crisis.

Introduction

A bug in a deployment is a standard incident. A datacenter catching fire, a cloud provider region going completely offline for 24 hours, or a ransomware attack encrypting every database in the cluster is a disaster.

Disaster Recovery (DR) is the set of tools and procedures designed to rebuild the entire application infrastructure from scratch in a secondary, unaffected location. Because rebuilding massive systems is inherently difficult and expensive, organizations define their DR strategies using two critical, negotiated metrics: Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

Mental Model

Think of an office building.

• High Availability (HA) is having two elevators. If one breaks, the other is immediately available. You barely notice the disruption.
• Disaster Recovery (DR) is what happens when the entire building burns down. You cannot rely on the second elevator; you must move the entire company to a new building across town and buy all new desks. DR is about how quickly you can resume operations in the new building (RTO) and how many paper files you permanently lost in the fire (RPO).

Operational Pattern

RPO: Recovery Point Objective (Acceptable Data Loss)

RPO defines how much data the business is willing to lose in a disaster, measured in time.

• If your database backups run every night at midnight, and the datacenter burns down at 11:59 PM, you just lost 24 hours of customer data. Your RPO is 24 hours.
• If you replicate database transactions in real-time to a secondary region, your RPO might be less than 1 second.

RTO: Recovery Time Objective (Acceptable Downtime)

RTO defines how long the business can tolerate being completely offline while the engineers rebuild the system in the secondary location.

• If you have to manually provision servers, install software, and restore a 5TB database backup from cold storage, your RTO might be 48 hours.
• If you use Infrastructure as Code (Terraform) and maintain a “warm standby” database in a secondary region, your RTO might be 15 minutes.

Failure Modes

Schrodinger’s Backup

The most catastrophic failure in DR is realizing your backups are useless during an actual emergency. Perhaps the backup script has been silently failing for six months, or perhaps the encryption key required to read the backup was stored in the datacenter that just burned down. A backup does not exist until you have successfully restored it to a fresh environment.

Over-promising to the Business

Business stakeholders will always demand an RPO of 0 seconds (no data loss) and an RTO of 0 seconds (no downtime). Achieving this requires true Active-Active multi-region architectures, which are exponentially more complex and expensive to build and maintain than standard architectures. Engineering leadership must force the business to align their demands with their budget. “A 0-second RTO will cost $5 million a year in idle infrastructure. A 4-hour RTO will cost $50,000. Which do you prefer?”

Security

During a disaster, engineers often bypass security protocols in a desperate rush to restore service (e.g., granting “Admin” access to everyone on the team, or opening database ports to the public internet). DR procedures must have pre-approved “break glass” security protocols that grant temporary, heavily audited emergency access, ensuring the cure is not worse than the disease.

Operational Guidance

You must test your Disaster Recovery plan annually. This is often called a “Game Day” or “DR Drill.” In a controlled setting, simulate the total destruction of your primary environment. Track exactly how long it takes the team to execute the DR Runbooks and bring the system online in the secondary region. If the drill takes 6 hours, but your SLA promises a 2-hour RTO, you must either rewrite the runbooks, automate the infrastructure provisioning, or renegotiate the SLA.

Related topics

• Incident severity classification
• Runbooks and playbooks

Sources & further reading

• AWS Reliability Pillar: Disaster Recovery - Amazon Web Services