Fix cross-region backup copy failures

The organisation decided, at some point, that critical workloads need a backup copy held in a second AWS region so a regional outage can't destroy both the workload and its only backup. That decision usually shows up on the invoice as warm storage in the DR region and a per-copy charge. This finding flags the case where those copies are silently failing: the backup plan reports success, the local backup exists, but the copy to the second region never lands. The intended protection isn't there, even though everyone believes it is.

The exposure is asymmetric and easy to underestimate. Nothing breaks day to day. Reports stay green. The first time the gap becomes visible is during an actual regional disaster — exactly when there is no time left to fix it and the cost of the gap is at its maximum. In risk terms this is a control that is documented, assumed effective, and actually broken: the most dangerous kind, because no one is watching it.

It also has a compliance dimension. Geographic-redundancy requirements in frameworks like SOC 2, ISO 27001, and PCI DSS, and the DR clauses in most enterprise contracts, are satisfied by recovery points that actually exist in a second region — not by a backup plan that was configured to create them. A failing copy job means the control is non-operating, and an auditor who asks to see the destination recovery points will find the gap whether or not the team has. The right framing at the operational review is not "are we paying for DR?" but "can we prove a recent copy actually landed in the DR region?"

Priya is the finance partner embedded with the platform team at a healthtech company. The DR-storage line on the cloud invoice has looked stable for months, so on paper the disaster-recovery posture is funded and working. At the operational review she asks the question that's now standard on the agenda: "For our critical workloads, can we show evidence that a backup copy actually landed in the second region in the last week — not just that the plan is configured to send one?"

The answer that comes back isn't comfortable. The team pulls the copy-job history and finds the copies have been failing; the local backups were green the whole time, so nothing had flagged it. The conversation that follows isn't technical — Priya doesn't ask about KMS keys or vault policies. She asks three things: how long has the copy been failing, which workloads are exposed in the meantime, and what will tell us automatically the next time it breaks. The team commits to fixing the copy, restoring from the destination to prove it, and adding an alert so a failed copy surfaces in minutes.

A month later the review includes a new standing line: "days since last successful DR copy" per critical workload, alongside a note that copy failures now page the on-call. Priya knows the dollar amount on the DR-storage line was never the point — a stable invoice told her nothing about whether the control was working. The metric that matters is evidence of a recent successful copy, and that's now the thing she asks for.

Unlike most findings on a cost dashboard, this one barely touches the invoice. The DR-storage and per-copy line items keep billing at roughly the same rate whether copies succeed or fail — a failing copy might even shave a little off, because nothing new is being stored at the destination. So watching the dollar amount tells you nothing about whether the control works. That's the trap: a stable, healthy-looking invoice can sit on top of a completely broken DR posture.

The real impact is risk exposure that is invisible in steady state and maximal in a disaster. The organisation is funding a disaster-recovery capability and, when copies are failing, simply not receiving it. The exposure is binary — there either is a recent off-region recovery point or there isn't — and it stays hidden until a regional event forces the question. From a risk-quantification standpoint, the expected loss is low-probability but catastrophic-magnitude, which is precisely the profile that justifies a cheap, always-on control rather than periodic spot checks.

There's a commitment-and-budget angle too: this is spend that is failing to buy its intended outcome. Money is leaving the budget for DR storage while the DR outcome — recoverability from a second region — is not being delivered. That's worse than waste, because waste is at least visible; this is paying full price for a capability that silently isn't there. The right budget question is whether the spend is producing verified copies, not whether the spend is stable.

Finally, it's a control-assurance signal. If a DR copy could fail for weeks without anyone noticing, the same blind spot almost certainly applies to other assumed-working controls — encryption, retention, replication. A failing copy that went undetected is a leading indicator that the organisation verifies controls by configuration rather than by evidence, and that pattern predicts other unpleasant audit surprises.

The business agreed that critical data should have a backup held in a second region so a regional outage can't take out both the system and its only copy. This finding means that copy has been silently failing: the primary backup still runs, the dashboards stay green, but the off-region copy the recovery plan depends on isn't being written. The protection the business is paying for and counting on is not actually in place.

The danger is that nothing looks wrong until the day it matters most. This is a continuity and audit issue, not a cost one — the fix is cheap and fast, but only if it's done before a regional event rather than during one. The leadership question is simple: can the team show evidence that a recent backup copy actually landed in the second region, not just that the system was configured to send one?

At one company the quarterly continuity review used to report "backups: green" and move on. Then a near-miss — a copy that had been failing for weeks, caught by luck during an audit — changed the question the exec sponsor asked. Instead of "are backups running?" she started asking "can we prove a recent DR copy actually landed, and how fast would we know if it stopped?"

Within a quarter the reporting changed shape. The headline was no longer a green backup tick; it was "every critical workload has a verified destination copy from the last 24 hours, and a failed copy pages the on-call within minutes." The cost of the DR storage didn't move — it was never the issue — but the confidence behind the number became real. The team had stopped trusting that a configured control was a working one.

That's the right outcome state. The goal isn't "backups configured"; it's "DR copies verified and monitored." A configured-but-unverified copy is the most expensive kind of false comfort, and the leadership move is to make the verification, not the configuration, the thing that gets reported.

The dollar amount here is negligible and beside the point. This is tracked because it represents a continuity control the business is paying for and assuming is effective, that may in fact be silently broken. The risk profile is low-probability and high-consequence — nothing breaks until a regional event, and then the missing off-region copy is the difference between a recovery measured in hours and one measured in days. That asymmetry is exactly why a quietly-failing copy belongs on a leadership report even though it costs almost nothing.

The deeper signal is about how the organisation assures its controls. If a DR copy can fail for weeks unnoticed, the org is trusting configuration over evidence — and the same blind spot likely covers other controls that show up in audits, customer security reviews, and incident post-mortems. Most CFOs care about the audit and contractual exposure; most CIOs care about the recoverability; both should care that an assumed-working control was actually broken and nothing caught it.