Blog · 14-min read
Disaster recovery planning for production applications
Skip to content
Last substantive review: August 2026.
A successful backup job is not the same thing as a recoverable production service. Recovery depends on data, identity, networking, secrets, infrastructure, third-party services, operator access, and a sequence of decisions made under pressure. If those pieces have not been restored together and measured, the organization has a hope, not a recovery capability.
This guide is for CTOs, vice presidents of engineering, security and platform leaders, and procurement teams responsible for production applications. It focuses on the engineering work that converts recovery objectives into tested evidence: dependency mapping, protected backups, restores, failover and failback, executable runbooks, exercises, and a sequenced backlog.
The appropriate design is driven by business impact. A payroll workflow, a customer marketplace, and an internal analytics tool may deserve different recovery targets. “Highly available” and “multi-region” are architecture labels, not business requirements.
Start with business workflows, not cloud resources. Identify what customers or operators need to do, how long each workflow can remain unavailable, how much recent data can be reconstructed or lost, and what degraded service is acceptable. Name an executive or business owner who can approve those tradeoffs.
Translate each critical workflow into two targets:
These are objectives, not claims. “RTO: four hours” is not true because it appears in a document. It becomes credible when the recovery path is exercised under realistic conditions, the elapsed time is measured, and gaps are tracked. A workflow may also require manual continuity procedures while systems recover. Document who can invoke them and how data created during that period will be reconciled.
NIST’s Contingency Planning Guide for Federal Information Systems covers business impact analysis, recovery strategies, plan development, testing, training, exercises, and maintenance. Its federal control context will not fit every company, but the planning sequence is a sound reference for connecting impact to recovery work.
Draw the workflow and every service it needs. The map should include DNS, certificates, content delivery, identity, privileged access, secrets and key management, network paths, compute, queues, object storage, databases, search indexes, caches, email or messaging providers, payment services, observability, source repositories, build systems, artifact registries, infrastructure code, and the cloud control plane.
For each dependency, record the owner, region or failure domain, recovery method, configuration source, data source, access path, validation check, and upstream or downstream dependencies. Mark external services that the team cannot restore. Their outage behavior needs a degraded mode, queue, manual process, or a business-approved wait.
Identity and control-plane dependencies deserve special attention. A team may have valid database backups and still be unable to recover because the only administrator account uses the failed identity provider, the encryption key is inaccessible, or the infrastructure repository is in the affected tenant. Recovery credentials, contact information, and essential runbooks need protected access outside the failure they are meant to address.
The map should also state restoration order. Bringing an API online before its identity, database, encryption keys, and event consumers can create errors or inconsistent data. “Restore application” is not one step when the application is a network of stateful and stateless components.
Inventory every source of state. That includes primary databases, object stores, configuration, secrets metadata, queues where loss matters, search or analytical stores that cannot be rebuilt within the objective, customer uploads, infrastructure definitions, and critical third-party exports. For each source, define backup frequency, retention, location, encryption, integrity checks, deletion protection, ownership, and the recovery procedure.
Replication is useful for availability, but it is not necessarily a backup. Accidental deletion, corruption, malicious encryption, or a faulty deployment may propagate to replicas. Recovery design may require immutable or logically isolated copies, an offline copy, separate credentials, or another account or region. The exact pattern should follow the threat model, regulatory duties, and approved RPO.
CISA’s StopRansomware Guide recommends offline, encrypted backups and regular testing of their availability and integrity in a disaster recovery scenario. It also emphasizes critical-system dependencies and exercised response and communications. Apply those principles with the company’s own retention, privacy, and access requirements.
Backup monitoring should answer more than “did the job run?” Record whether the expected assets were included, whether the copy is readable, whether retention and immutability controls are still applied, and whether alerts reach someone who can act. A green dashboard can conceal a backup that excludes a newly created database or cannot be decrypted by the recovery team.
A restore test should create a usable system or data set in a clean target environment. Record the backup identifier, recovery point, operator, start and finish time, commands or automation used, errors, validation results, and cleanup. Validate application-level behavior and data reconciliation, not merely that a database process started.
For a transactional system, validation may include row counts, checksums, referential integrity, a set of representative reads and writes, permission checks, queued-event reconciliation, and confirmation that downstream jobs do not duplicate actions. For an uploaded-file workflow, confirm both metadata and objects. For an encrypted system, prove the keys and permissions are available through the recovery path.
Measure actual recovery time and actual data age against the approved RTO and RPO. Separate delays caused by access, environment creation, data transfer, restore operations, application deployment, validation, and decision-making. The breakdown tells the team which backlog item will reduce exposure. An unmeasured “restore succeeded” gives little planning value.
Do not perform the first meaningful restore during an incident. Use isolated environments and scrubbed or appropriately controlled data. Define how the restored data is destroyed after the exercise and who may inspect it.
Recovery patterns sit on a spectrum. Backup and restore usually has lower steady-state cost and a longer recovery path. A pilot-light or warm-standby design keeps more components ready. Active-passive failover maintains a secondary environment that can take traffic. Active-active operation can reduce some outage windows but adds routing, data-consistency, deployment, testing, and operating complexity.
AWS’s current Well-Architected disaster recovery guidance describes backup and restore, pilot light, warm standby, and multi-site active/active strategies and connects them to recovery objectives. Treat this as cloud design guidance, not a directive that every application must use the most complex pattern.
Compare candidate strategies using business-approved RTO and RPO, measured restore performance, data-consistency requirements, failure domains, security and compliance needs, vendor limitations, engineering capacity, and total operating cost. Include the human burden of keeping the secondary path current. A standby environment that has drifted, lost credentials, or never received the latest schema can be more dangerous than a simpler, rehearsed restore.
Infrastructure as code and automated configuration help recreate environments, but only if the definitions, modules, artifact versions, secrets references, and state are available during the incident. Test from the protected source of truth. Avoid recovery scripts that silently fetch “latest” dependencies or container images.
Failover changes where the service runs. Failback returns it to a normal operating state. Both can lose or duplicate data if the team has not defined authority, fencing, replication direction, write behavior, and validation.
The failover plan should state the trigger, incident role that authorizes it, dependencies checked first, traffic-routing change, expected propagation behavior, data-consistency decision, validation steps, customer communication, and abort conditions. If DNS is involved, document access and expected cache behavior rather than assuming an instantaneous switch. If a queue will buffer work, define its capacity and replay controls.
Before failback, determine which environment contains the authoritative data, reconcile changes made during recovery, confirm the primary is healthy, and plan a controlled traffic transition. Keep the recovery environment available until validation is complete. “Reverse the steps” is not an adequate failback plan.
Third-party outages may require a different mode. The application might accept requests and delay fulfillment, disable one feature, or use a manual workflow. Those behaviors need product decisions, capacity limits, customer messages, and a reconciliation path.
A recovery runbook is an operational interface. It should not depend on the memory of the person who wrote it. Include:
Prefer idempotent, reviewed automation where it reduces operator error. Still document what the automation changes and how to recognize partial failure. A one-click recovery tool that only one employee understands is a concentration of risk.
Keep a minimal protected copy available outside the primary collaboration system if that system is part of the incident. Do not put reusable secrets in the runbook. Point to an approved emergency-access process and test it.
A tabletop exercise walks decision-makers and operators through a scenario without changing production. It is useful for ownership, communications, legal or customer obligations, and discovering missing decisions. It does not prove that data can be restored.
A component exercise restores a database, object set, identity configuration, or infrastructure environment and validates it. A service exercise combines dependencies in a nonproduction recovery environment. A carefully governed production failover exercise tests routing, live controls, and real operating behavior. Each level has different risk and evidence.
Set an exercise schedule from criticality, system change, contractual requirements, and the cost of failure. Retest after material changes to architecture, data stores, identity, encryption, infrastructure tooling, regions, or vendors. Rotate operators so success does not depend on one person. Include nights, weekends, and access constraints when those conditions affect the response plan.
Every exercise should have a scenario, success criteria, safety boundaries, observers, timestamps, and a debrief. Record actual RTO and RPO performance, decisions that stalled, failed steps, manual workarounds, and artifacts that were missing. Convert the findings into owned backlog items with target dates. A recurring exercise that produces the same unresolved findings is evidence of governance failure, not readiness.
A concise recovery packet can support internal risk review, customer security diligence, and procurement without exposing operational secrets. It may include the approved workflow tier and objectives, dependency-map version, backup coverage and retention summary, latest successful restore date, measured recovery time and data age, exercise scope, runbook version, emergency-access review, open exceptions, owner, and next test date.
Distinguish design evidence from operating evidence. An architecture diagram shows intent. A dated restore log with validation results shows performance. A policy says who should act. An exercise record shows whether they could. Do not label a plan “tested” if the exercise stopped at discussion.
Redact credentials, sensitive topology, customer data, and exploit-relevant details before sharing. Procurement can ask for assurance without receiving the exact commands or secrets used to operate the service.
Recovery work competes with customer features, so the backlog needs a defensible order. One practical sequence is:
Within each group, rank by business impact, gap between target and measured performance, likelihood of the failure scenario, dependency breadth, and effort. Link each ticket to the evidence it should produce. “Implement cross-region database” is an architecture task; “demonstrate order recovery within the approved target after regional loss” is a recovery outcome.
Related engineering hygiene still matters. A reproducible AWS CI/CD pipeline can help protect build and deployment paths, while the technical-debt guide explains why undocumented, fragile operational work becomes more expensive when it remains invisible. Neither replaces recovery testing.
Horizon Labs’ product-team lane is $100–$120 per hour for coordinated full-stack delivery, QA, launch, and qualifying work covered by a six-month code warranty under the signed statement of work. It fits when recovery improvements are part of a broader application delivery program and the team owns the relevant product changes.
The $150–$200 per hour senior or specialist lane fits a different need: experienced engineers entering an existing production system to map dependencies, diagnose restore failures, automate infrastructure, close high-risk reliability backlog, or lead a bounded exercise. The rate reflects the required experience and independent judgment; it does not justify an unbounded open-ended engagement.
Start with a bounded recovery assessment or one critical workflow. Agree on access, safety boundaries, evidence, and decision gates. The first deliverable should identify what can be recovered today, what has been measured, what prevents the approved objectives, and which backlog tranche buys the largest reduction in exposure.
If your organization needs senior engineering help to turn recovery assumptions into tested evidence, contact Horizon Labs. Bring the current architecture, latest backup and restore records, incident history, contractual objectives, and the backlog you already have. The work should end with evidence your operators and approvers can inspect.
A backup is a protected copy of data or system state. Disaster recovery is the tested capability to restore an acceptable business service within defined time and data-loss objectives. It also requires dependencies, access, runbooks, decision authority, communications, validation, and failback.
Recovery time objective, or RTO, is the target time for restoring an acceptable level of service after a disruption. Recovery point objective, or RPO, is the maximum targeted age of the data that can be lost. Set them per business workflow and validate them with measured exercises.
There is no useful universal interval. Test according to system criticality, rate of change, contractual duties, and the cost of failure, and retest after material architecture, data, identity, or vendor changes. Use a planned mix of tabletop exercises, component restores, and failover drills.
No. Multi-region designs add cost, operational complexity, and new consistency risks. Choose backup and restore, a standby pattern, or active operation across regions by comparing business-approved RTO and RPO targets with measured recovery performance and total operating burden.
Start with the business workflows that cannot remain unavailable, then map their data and dependencies. Fix missing or untested backups, inaccessible recovery credentials, and undocumented restore order first. Sequence later work by reduction in business exposure, and require evidence from each completed control.
A production AI rollout checklist for evals, risk tiers, guardrails, observability, staged exposure, human override, incidents, and rollback.
Effective strategies and tools for building AI-powered marketplaces, a founder-friendly guide to integrating AI into your product.
A practical map of the identity, availability, transaction, search, trust, operations, and observability seams that make marketplaces hard.
Tell us what you're building. After an intro call with our founders, you'll hear back within 72 business hours: a written scope with an hour range and a hard ceiling, or the questions we need answered first.