Audit alarms that never trigger

CloudWatch standard-resolution alarms cost roughly $0.10 per alarm metric per month. Individually that's noise, but a large AWS account can accumulate thousands of alarms over years of incidents and projects — each one a small, ongoing charge regardless of whether it works. ALH-003 surfaces the subset of that inventory that has never fired: alarms that have sat in OK for months or years with no state transition on record.

From a spend perspective the direct cost is marginal. The FinOps concern is what the inventory represents: every silent alarm is either a legitimate control on a genuinely stable system, or dead weight — a control that would fail to fire even if the condition it was meant to catch actually happened. You can't tell which just from the count. The only way to distinguish a healthy-system alarm from a broken one is to verify it, and that verification process is what ALH-003 is prompting.

For finance, the right frame is governance hygiene. An alarm inventory with a meaningful fraction of never-triggered items is an inventory that hasn't been reviewed. Periodic audits — quarterly is reasonable — convert that ambiguous list into deliberate decisions: keep the verified ones, fix the broken ones, delete the defunct ones. That keeps both the spend and the risk posture honest, and means the team can answer 'do our monitoring controls work?' with evidence rather than assumption.

Dana is the FinOps partner reviewing the Q2 cloud operations summary. The security dashboard shows 31 CloudWatch alarms flagged under ALH-003 — never triggered in 90-plus days. The per-alarm cost is small, but Dana's first question isn't about the cost; it's about what the list implies for the risk register.

She asks the SRE team to split the 31 by business criticality. Of the 31, eight are on production workloads — customer-facing services where an undetected incident would have revenue or SLA impact. The other 23 are on dev, test, and deprecated resources. For the 23, Dana's recommendation is: verify quickly or delete. Paying $0.10/month for an unverified alarm on a retired resource is waste; more importantly it pollutes the inventory and makes the working alarms harder to trust.

For the eight production alarms, Dana asks one question: when were they last proven to fire? The answer — never, for most of them — goes into the risk register as a monitoring-control gap. The cost to fix it is engineering hours, not cloud spend. Her takeaway for the finance pack is: 'Eight production monitoring controls are unverified. The risk isn't the alarm bill — it's that we'd find out about incidents from customers, not pages.'

The direct CloudWatch cost is easy to model: standard-resolution alarms run roughly $0.10 per alarm per month. A fleet of 2,000 alarms is approximately $2,400 per year. A meaningful fraction of that — in many mature accounts, 20–40% — is likely defunct or unverified. That's real but not the primary financial exposure.

The larger cost is incident economics. An unverified alarm on a production workload is a monitoring-control gap with a deferred price tag. When the condition it was supposed to catch actually occurs — a bounce-rate spike, a dead-letter queue backing up, a disk filling — the team finds out from a customer ticket or a manual check rather than a page. Mean-time-to-detect doubles or triples, and the cost of that gap shows up as incident response hours, SLA credits, and reputational exposure. That cost is episodic and hard to budget for, which makes it worse than a predictable line item.

There is also an audit exposure. SOC 2 CC7.2 and ISO 27001 A.12.4 require periodic testing of monitoring controls. An alarm inventory that has never been verified is a gap auditors will find. The cost of a finding at audit — remediation effort plus the possibility of a qualified opinion — typically exceeds the cost of a quarterly verification programme by an order of magnitude.

For finance, the right model is: unverified alarms represent a contingent liability, not just a sunk cost. The quarterly audit converts that liability into a documented, defensible control set, and the small engineering cost to run it is cheap relative to the incidents it prevents.

CloudWatch alarms are the mechanism that tells the team when something is wrong. A 'never-triggered' alarm is one that has sat quietly in an OK state for months or years with no record of ever firing — which could mean the system is healthy, or could mean the alarm is broken and would fail to fire even in an incident. ALH-003 flags the entire ambiguous set.

The leadership question this control surfaces is simple: can you defend the monitoring posture? If a major incident occurs and the alarm that should have caught it didn't fire, the post-incident review will ask when it was last verified. An untested inventory has no good answer. The low-severity rating reflects that a silent alarm isn't actively causing harm today — but the risk materialises at the exact moment you need it most.

The healthy posture is not a zero-finding dashboard — it's a verified, deliberately maintained alarm inventory where every silent alarm has been reviewed, tested, and kept or removed on purpose. That's the difference between 'we have monitoring' and 'our monitoring works.'

At one company the SES bounce-rate alarm had been in the inventory for two years. It was listed in the runbook. It was counted in the monitoring dashboard. Nobody had ever tested it. When the email sending domain was migrated and the metric namespace changed, the alarm silently lost its connection to the underlying metric — but it stayed in OK and nobody noticed.

Six months later a bounce spike hit. The alarm didn't fire. The support team found out from a customer whose emails were bouncing. The post-incident review found the broken alarm within minutes — it had never transitioned state since the migration.

The finding wasn't that the alarm was broken. It was that nobody had a process to know. ALH-003 is that process: it surfaces the alarms that have been silent long enough to be worth verifying. The executive question is whether that verification happens on a schedule, before an incident, or in a post-incident review, after one.

An unverified alarm is a risk that is invisible until it materialises. The team believes the monitoring is in place; the alarm is in the dashboard; the control is on the compliance list. But if it hasn't been tested, all of that is assumption. The impact lands when an incident the alarm was supposed to catch instead arrives as a customer complaint — and the post-incident review finds the alarm has been silently broken for months.

The business impact of missed incidents is well-understood: extended downtime, SLA penalties, customer-facing errors, and the reputational cost of finding out about problems from outside rather than inside. What makes unverified alarms a distinct leadership concern is that the risk is latent and quantifiable — the team can run a quarterly audit and find it — but without that process the organisation carries the exposure invisibly.

The compliance dimension matters too. Regulators and auditors under SOC 2 and ISO 27001 expect monitoring controls to be tested, not just configured. The board question isn't 'do we have alarms?' — it's 'do we have a process that proves they work?' A verified, periodically reviewed alarm inventory answers that question with evidence. An unaudited one does not.