Fix restore test failures

When an engineering team reports that "backups are green," the natural reading is "we are protected." That isn't quite what green means. A successful backup proves a copy was made; it does not prove the copy can be turned back into a working system. The only thing that proves recoverability is a restore test — automatically rebuilding the backup into a live resource and watching it come up healthy. A failed restore test is the system telling you, in advance and at low stakes, that a backup you are counting on may not actually be recoverable when you need it.

This finding flags restore-test jobs that errored out. It is not a cost line item — it is a risk and exposure signal. The exposure is asymmetric: the cost of the failing test is trivial, while the cost of discovering the same failure during a real outage is measured in hours of downtime, lost revenue, customer credits, and reputational damage. The uncomfortable framing for a review meeting is that every failing restore test is a documented gap between your stated disaster-recovery capability and your actual one — a gap that exists today whether or not anyone acts on it.

From a governance and assurance standpoint, this is where DR attestation lives. If the organisation tells auditors, the board, or customers that it can recover critical systems within a stated window, restore-test results are the evidence behind that claim. A backlog of failed restore tests means the attestation is unsupported. The right question at the operational review is not "how much does this cost" but "which of these failing tests sit on systems we have promised we can recover — and what is our actual exposure until they're green?"

Dana is the risk-and-assurance partner who co-owns the operational review with the platform team. The DR dashboard shows three failed restore tests, all on the production database tier that the company's customer contracts and SOC 2 report promise to recover within four hours. Dana doesn't ask which KMS key or which availability zone — those aren't her questions. She asks three: which committed systems do these failures sit on, how long have they been failing, and what is our actual recovery exposure until they're green again.

The answer reframes the meeting. These three failures aren't a cost item to optimise; they are a live gap between what the business has told auditors and customers it can do and what it can currently prove it can do. For the duration the tests are red, the four-hour recovery commitment on that tier is an assumption, not a demonstrated capability. Dana's job is to make that exposure visible and time-boxed: engineering commits to root-cause and re-test within 48 hours for any failure on a committed-RTO system, and the open count goes on the assurance tracker until it's zero.

Two weeks later the same dashboard shows all critical-tier restore tests green with timestamped pass results. That is the artefact Dana needs — not a promise that backups exist, but evidence the backups restore, dated and repeatable. She knows the right floor isn't "zero failures ever" (a transient capacity blip will redden a test occasionally and self-heal) but "zero unresolved failures on committed systems," and that a failure that lingers more than a few days is the real signal that DR ownership is slipping.

There is almost no direct cost story here — a restore test run costs pennies to a few dollars in temporary resource-hours, and a failed one often costs even less because the resource never fully provisioned. Treating this as a cost line misses the point entirely. The material number is the exposure: the expected cost of an unrecoverable backup discovered during a real incident, weighted by the probability that one of your failing tests is sitting on a system you'll actually need to recover. That figure runs from tens of thousands to millions depending on the system, and it's the right way to size the issue at a review.

The exposure is concentrated, not spread evenly. A failed restore test on a quarterly-tested cold-archive system is a low-priority ticket; the same failure on a revenue-bearing production database under a contractual 4-hour RTO is a live business risk. The finance and risk lens adds the weighting engineering doesn't always apply: not all failing tests are equal, and the ones on committed systems should be triaged on a different clock — measured in hours, not added to a backlog.

This is also where DR attestation either holds up or falls apart. If the business represents a recovery capability to auditors, customers, regulators, or the board, restore-test results are the evidence behind that representation. A backlog of failed tests on attested systems means the attestation is, strictly, unsupported — and that's a finding waiting to happen, not a cost to optimise. The assurance question is "can we evidence recovery on every system we've made a commitment about," and a red restore test is a no.

Finally, treat the trend as the metric. A small number of transient failures that self-heal within a run or two is normal and healthy — it means the harness is genuinely exercising the system. A growing count, or any failure on a committed-RTO system that lingers more than a couple of days, is the real signal: it means DR ownership is slipping and the gap between promised and proven recovery is widening unwatched.

Backups being "green" means copies were made. It does not mean those copies can be turned back into running systems. Restore testing is the practice that proves recoverability by actually rebuilding backups on a schedule — and a failed restore test is an early, low-stakes warning that a system we are counting on may not come back in a real disaster. The one question worth asking: can we prove we can restore, or are we assuming it?

This is a business-continuity signal, not a cost line. A backlog of failing restore tests means the gap between our stated recovery capability and our real one is unmeasured — and that gap turns into downtime, lost revenue, and a broken commitment to customers and auditors at exactly the moment it's most visible. The mature outcome isn't "zero failures forever"; it's that failing tests get triaged fast, the critical-tier ones go to zero, and recovery becomes something we can demonstrate on demand rather than hope for.

At one company, the quarterly continuity review used to open with a reassuring slide: "100% of critical systems backed up." Then a real incident required a restore that took eleven hours instead of the promised two, and the board learned the hard way that "backed up" and "recoverable" are different claims. The next review opened with a different metric entirely: restore tests passing on every committed-RTO system, with dates.

Within two quarters the conversation matured. The headline stopped being how many backups existed and became whether the org could prove, on demand, that it could recover its critical systems — and whether any restore-test failure on those systems had been open longer than a couple of days. A failing test was no longer alarming in itself; an unresolved failing test was. The exec sponsor's standing question became simply: "Can we still prove we can restore the things we've promised to restore?"

That's the right outcome state. The goal isn't a permanently green board — transient failures happen and get fixed. The goal is that recovery is demonstrable rather than assumed, and that the gap between the promise and the proof is measured in days, not discovered in an outage.

The dollar cost of restore testing is negligible, so this is never a cost item — it's a continuity-assurance item. Its presence on the report answers one question the board and customers ultimately care about: if a critical system fails, can we actually get it back, and can we prove it before we have to? A clean restore-test record is the only honest "yes" to that question; a backlog of failures is a quiet "we're not sure," dressed up by backup dashboards that only count copies made.

There's a compounding risk too. Forgotten or unresolved restore-test failures don't announce themselves — they sit until an outage converts them from a tracked finding into a headline. They also intersect with compliance and customer trust: the same gap shows up as an audit finding and as a broken SLA in the same incident. Most CFOs care about the financial exposure, most CIOs about the operational and audit exposure; both should care that recovery is something the organisation can demonstrate rather than assume.