Choosing an AWS disaster recovery strategy by RPO and RTO, workload by workload

When our finance team asked "what happens if the region goes away", the honest answer was that every service had a different, mostly undocumented answer. A few had nightly snapshots. One had a full copy running in a second region that nobody had tested in a year. The rest had nothing.

We fixed it by doing the work in the right order: set recovery targets per workload with the business, map each target to one of the four standard AWS DR strategies, and then prove the numbers with drills. This post walks through that process with the five workloads we actually classified, so you can reuse the method. It covers strategy selection only; designing services to survive failures inside a region and making DNS failover trigger correctly are separate problems.

RPO and RTO, defined precisely

• Recovery Point Objective (RPO): the maximum amount of data, measured in time, you can afford to lose. RPO of 1 hour means after recovery, the last hour of writes may be gone.
• Recovery Time Objective (RTO): the maximum time from the disaster being declared to the service being usable again.

Two details make these numbers honest. RTO starts when the disaster happens, not when someone decides to act, so detection and decision time count. And RPO is set by the replication or backup mechanism, not by intention: a nightly backup has an RPO of up to 24 hours plus however long the copy takes.

We also separate the disaster types, because the right answer differs:

Replication alone fails the second and third rows: a DELETE without a WHERE replicates to the standby in under a second. Every strategy below needs backups as well.

The four strategies

Backup and restore

Data is backed up and copied to another region. Infrastructure is not running there; it is recreated from infrastructure as code when needed.

• Typical RPO: hours (backup frequency) or minutes with continuous backups.
• Typical RTO: hours, dominated by provisioning and restoring large datasets.
• Cost: storage only.
• AWS building blocks: AWS Backup plans with cross-region copy, RDS automated backups, S3 replication, CloudFormation or Terraform to rebuild the stack.

The common failure here is assuming the IaC still works. A template that last ran eighteen months ago will hit deprecated instance types, changed AMIs and quota limits in the recovery region.

Pilot light

Data is replicated continuously to the recovery region and the core data services are live there, but the application tier is switched off or scaled to zero.

• Typical RPO: seconds to minutes (continuous replication).
• Typical RTO: tens of minutes, the time to start and scale the app tier and switch traffic.
• Cost: replicated databases plus storage.
• AWS building blocks: Aurora global database secondary cluster, DynamoDB global tables, S3 cross-region replication, ECS services or Auto Scaling groups at zero capacity.

Warm standby

A complete, working copy of the system runs in the recovery region at reduced capacity. It can serve traffic immediately and is scaled up during a disaster.

• Typical RPO: seconds.
• Typical RTO: minutes.
• Cost: replicated data plus a small but always-on application tier.
• Benefit: because it always runs, you can send synthetic traffic to it continuously and know it works.

Multi-site active-active

Both regions serve production traffic all the time. A region loss means the other absorbs the load.

• Typical RPO: near zero, depending on the data model.
• Typical RTO: near zero for routing, assuming enough capacity in the surviving region.
• Cost: roughly double, plus significant engineering complexity.
• Hard part: writes. With a single-writer database you are really running warm standby for writes. True multi-writer needs a data model that tolerates concurrent writes in two regions, such as DynamoDB global tables with last-writer-wins, or partitioning users by home region.

Comparison at a glance

Worked example: classifying five workloads

We ran a one-hour session with each workload owner and a business stakeholder. The questions were the same every time: what does an hour of downtime cost, what does an hour of lost data cost, and are there contractual or regulatory requirements.

Some decisions that came out of this:

• Nobody got active-active. The only candidate was checkout, and the cost of making order writes safe in two regions outweighed the difference between a 15-minute and a near-zero RTO for our volume.
• Search did not need its data replicated. The index can be rebuilt from the order and product databases, so protecting those covers search. We only needed a tested rebuild job that finishes inside 4 hours.
• Different strategies can share pieces. The order database secondary cluster serves both the pilot light for invoicing and the warm standby for checkout.

Building the warm standby for checkout

The recovery region runs the full checkout stack at 10% of production capacity, which is the smallest size that still handles our synthetic traffic and some real traffic during a drill.

module "checkout_recovery" {
  source = "./modules/checkout"
  providers = { aws = aws.us_west_2 }

  desired_count     = 2      # production runs 20
  min_capacity      = 2
  max_capacity      = 40     # must be within service quotas in this region
  db_cluster_arn    = aws_rds_cluster.orders_secondary.arn
  read_only_mode    = true   # flipped during failover
}

The data tier uses an Aurora global database, which replicates storage-level changes to the secondary region with typical lag under a second. During a real regional failure we run a managed failover to promote the secondary. During a planned drill we use a switchover, which waits for replication to catch up so no data is lost.

aws rds failover-global-cluster \
  --global-cluster-identifier orders-global \
  --target-db-cluster-identifier arn:aws:rds:us-west-2:111122223333:cluster:orders-usw2 \
  --allow-data-loss

The --allow-data-loss flag is the explicit acknowledgement that writes not yet replicated are gone, which is exactly the RPO trade we agreed. For a planned switchover the same command runs with --switchover instead.

Two things are easy to forget:

• Service quotas in the recovery region. Scaling from 2 to 40 tasks fails if the Fargate vCPU quota there is still at the default. We raised quotas in advance and check them in CI.
• Secrets and config. Secrets Manager secrets are replicated to the recovery region, and parameters that hold region-specific values exist in both regions.

Backups underneath every strategy

Regardless of strategy, every workload with data has AWS Backup plans that copy recovery points into a vault in a separate recovery account. That vault uses Vault Lock in compliance mode, so nobody, including an administrator with compromised credentials, can delete backups before their retention ends.

{
  "Rules": [{
    "RuleName": "daily-35d",
    "TargetBackupVaultName": "prod-primary",
    "ScheduleExpression": "cron(0 3 * * ? *)",
    "Lifecycle": { "DeleteAfterDays": 35 },
    "CopyActions": [{
      "DestinationBackupVaultArn": "arn:aws:backup:us-west-2:444455556666:backup-vault:recovery-locked",
      "Lifecycle": { "DeleteAfterDays": 35 }
    }]
  }]
}

For the order database we also keep continuous backups with point-in-time restore, because corruption is far more likely than a regional outage.

Proving the numbers

An RPO and RTO that have never been measured are guesses. Our drill schedule:

The first backup-and-restore drill for the reporting warehouse took 31 hours against a 2-day RTO. That passed, but it revealed that the restore was single-threaded and that one dashboard depended on a resource that existed only in the primary region. Both were invisible on paper.

Key takeaways

• RPO and RTO are business decisions per workload; strategy is the engineering answer to them.
• Replication protects against infrastructure loss only. Corruption and malicious deletion need point-in-time and immutable backups in an isolated account.
• Pilot light and warm standby differ in whether the app tier is already running; that one difference decides whether you can test it continuously.
• Active-active is mainly a data-model problem, not an infrastructure one.
• A strategy is only real after a timed drill; record measured RPO and RTO next to the targets.

References

• AWS whitepaper, Disaster Recovery of Workloads on AWS: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-workloads-on-aws.html
• Aurora global database failover and switchover: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database-disaster-recovery.html
• AWS Backup cross-account and cross-region copy: https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-region-backup.html
• AWS Backup Vault Lock: https://docs.aws.amazon.com/aws-backup/latest/devguide/vault-lock.html
• AWS Resilience Hub: https://docs.aws.amazon.com/resilience-hub/latest/userguide/what-is.html