DR You Have Not Tested Is a Hypothesis

Most disaster recovery plans are hypotheses dressed as capabilities. They read well, they cite recovery time objectives with a confident second decimal, and they have never once been asked to hold weight. The gap between “the plan says failover completes in four hours” and “failover completed in four hours, we watched it, and we wrote down what broke” is the entire discipline. Until a failover has been exercised end to end, on the actual platform, with the actual data volumes, the number in the plan is an estimate someone typed into a document a year ago.

This matters more, not less, as platforms consolidate onto container orchestration. A single OpenShift cluster now often carries workloads that used to live across many separate servers, each with its own DR arrangement of varying seriousness. Centralising the runtime centralises the failure mode too. If the platform’s cross-site recovery has a hidden dependency — on a certificate that expires, a route that was never re-created, an operator that does not reconcile cleanly against stale state — that dependency now sits underneath everything, not underneath one application.

Why an untested failover is not a plan

A DR plan accumulates untested assumptions the moment it is written, and every change to the platform afterward adds one more. A storage class gets renamed. A namespace gets a new network policy. An image registry moves behind a different pull secret. None of these show up as a DR risk in the change record, because none of them were framed as touching DR — they were ordinary platform maintenance. But the failover path is built out of exactly this kind of ordinary component, and it degrades silently because nothing exercises it between disasters.

The only way to convert a plan into a capability is to run it, on a schedule, against production-representative state, and to treat a test that reveals a gap as a successful test rather than a failed one. A DR test that passes cleanly on the first attempt deserves scrutiny, not celebration — check whether the scope was narrow, the data set small, or the operators already fluent enough to route around the traps a real incident would hit, before crediting the platform.

What a real cross-site test looks like

A credible test on a two-site container platform exercises the full chain, not a sample of it:

• Application state, not empty namespaces. Persistent volumes with production-scale data, replicated through the actual storage mechanism (array-based replication, Ceph mirroring, or application-level replication), not a fresh deployment from GitOps manifests onto an empty secondary.
• Traffic, not just pods. DNS or load-balancer failover, TLS certificates valid at the secondary, and ingress routes that resolve — an application with healthy pods and no reachable route has not recovered.
• Secrets and identity. External secret stores, service accounts, and any federated identity provider reachable from the secondary site with the same trust relationships, not a copy that quietly points back at the primary.
• Sequencing under dependency. Databases before the services that read them, message brokers before consumers, platform operators before the workloads they manage. A restore that races these produces crash loops that look like a platform failure when the platform is fine.
• A stated blast radius and a stated stop condition. What is actually failed over (a namespace, a cluster, a site) and what would cause the test to be aborted and rolled back, decided before anyone starts, not improvised mid-test.

The output of a real test is not “it worked.” It is a timestamped record of each step, the actual duration against the target, and a list of what needs fixing before the next one — because the next one is not optional.

Writing a runbook for the person who is not you

A runbook is only load-bearing if it works for the engineer who is competent, on call, and has never seen this particular failover before. That constraint should be the test for every line in it.

Concretely: no step that says “check if it looks healthy” without the command and the expected output. No step that assumes a mental model of the architecture that lives only in the head of whoever wrote the runbook. No step that says “as usual” or “the normal way.” Every command should be copy-pasteable against the real target, using the real object names, with the expected result stated next to it — not implied.

# step 7: confirm the secondary route resolves before declaring traffic cutover complete
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: app-primary
  namespace: prod-app
spec:
  host: app.example.com  # expected CNAME target: secondary.app.example.com
  to:
    kind: Service
    name: app-primary

A runbook that assumes nothing is longer than one that assumes the reader’s competence. That length is the point — it is the difference between a document that helps at 3 a.m. and one that reads like reference material for someone who already knows the answer.

What “recovered” should mean contractually

Recovery time objective and recovery point objective are frequently written as if they describe the moment data appears at the secondary site. They should describe the moment a user can transact successfully, end to end, with data integrity confirmed — not the moment a database process reports itself as running.

A recovery clause worth signing states, precisely: which services are in scope, what “available” means for each (a health check response, or a successful transaction), the maximum acceptable data loss window measured against a specific replication mechanism, and who verifies recovery before the clock stops — the operating team, not the vendor whose component just failed. Anything vaguer than that resolves in an incident room, under pressure, in favour of whoever has the more persuasive voice that night. Precision written down in calm conditions is the only version worth trusting.