Fix CloudWatch alarms with no actions

A CloudWatch alarm watches a specific metric — CPU, error rate, queue depth, storage space — and transitions into an ALARM state when the metric crosses a threshold. That transition is the system's attempt to tell you something is wrong. An alarm with no actions is one where the system shouts and nobody is on the other end of the line. The alarm state is logged in AWS, but no notification fires, no escalation starts, and no automated response runs.

From a cost-and-risk perspective, this matters because the duration of an incident is the primary driver of its cost. A five-minute outage on a payment service is a minor blip; a four-hour outage — because the alarm sat in ALARM state with no page until someone refreshed the console — is a material loss event. The cost of the average cloud incident scales directly with mean time to detect, and actionless alarms make detection entirely dependent on someone manually checking a dashboard.

The finance framing isn't to pay for elaborate alerting infrastructure everywhere. It's to insist that for any resource whose failure has a measurable business cost, there is at least one human on the other end of the alarm. The question per alarm is: what does an undetected failure of this resource cost per hour, and is the answer large enough to justify wiring a notification? For most production resources the answer is obviously yes, and the remediation cost is a one-time engineer-hour, not a recurring spend.

Dana is the finance partner on the platform account review. The audit surfaces ALM-003: fourteen production alarms with no notification wired. Rather than treating it as a binary engineering fix, she asks the tiering question first: which of these alarms are watching resources that, if they fail silently for four hours, cost the business real money?

She cross-references the alarm names against their resource types and environment tags. Nine are watching production-tier services — checkout CPU, RDS free storage, Lambda error rates. The math is straightforward: an undetected failure on checkout running for three hours costs more than the entire annual monitoring budget for the account. Those nine get wired immediately. The remaining five are development and staging alarms; Dana documents them as intentionally low-priority and records that as a reviewed decision rather than a silent ignore.

Her note for the finance pack is a single line: 'Fourteen actionless production alarms remediated; nine wired to tiered alerting, five dev/staging documented as intentional. Estimated reduction in mean-time-to-detect for production incidents: from hours to under five minutes.' That's the framing that belongs in a financial review — not alarm counts, but incident duration risk reduced.

The financial impact of an actionless alarm is almost entirely driven by incident duration. An alarm that fires and pages the on-call engineer within two minutes results in a short incident. The same alarm with no action means detection depends on someone manually checking the console, which in practice means the incident runs for hours before anyone knows. Incident cost — support tickets, SLA breach credits, engineering time, customer churn — scales roughly linearly with duration.

The billing impact is invisible on the AWS invoice. Actionless alarms cost $0.10 per month, identical to wired alarms. The cost doesn't live in the infrastructure spend line; it lives in the incident response line, the customer success budget, and the recurring cost of a team that doesn't trust its own monitoring. That's harder to quantify but straightforward to frame: for any production resource, model the cost of a three-hour undetected failure versus a ten-minute detected one.

The compliance angle adds a second cost path. SOC 2, ISO 27001, and PCI DSS all require that detection events trigger a documented response. An alarm history showing repeated ALARM transitions with no corresponding incident ticket or response record is audit evidence of a controls gap — one that leads to findings, remediation plans, and the extended re-certification cycles that cost real partner time and occasionally delay customer contracts. The remediation cost of wiring an alarm is measured in engineer-minutes; the cost of a controls gap finding is measured in lawyer-hours.

The right finance intervention is to make sure the per-resource tiering decision is made explicitly. Not every alarm needs to page PagerDuty at 3am. But for every resource whose failure has a cost that exceeds the cost of a single incident response, the question 'is there at least one human who will be notified automatically if this alarm fires?' should have a documented yes.

AWS CloudWatch alarms detect when something goes wrong. But an alarm with no actions is one that fires into a void — the system knows there's a problem, records it, and tells nobody. The incident runs until someone manually checks a dashboard. This is one of the most common monitoring failures in cloud environments precisely because the alarm looks correct when you inspect it; the gap only surfaces when you actually need it.

The leadership question is simple: when a production system fails, is there a guaranteed path from detection to human awareness, or does it depend on someone happening to look? Actionless alarms mean the answer is the latter. The right posture is that every system whose failure has a business consequence has at least one wired notification — not as a technical nicety, but as an organizational commitment that incidents will be detected faster than customers report them.

At one company, the VP of Engineering, Carla, used to learn about production outages from customer support tickets — an hour or two after the problem started. The monitoring dashboard had alarms for every critical service, but most of them had no actions wired. The alarms were decorative.

After a quarterly audit surfaced the pattern, the team spent two days wiring actions to every production alarm and introducing a two-tier SNS topology: a Slack channel for informational alerts, PagerDuty for anything that needed a human awake at 2am. The next time checkout CPU spiked, the on-call engineer got a page within two minutes of the alarm firing.

Carla's question at every engineering review is now: 'Is there any production alarm that can fire without waking someone up?' It's a one-question audit of the monitoring posture. The answer should always be no for anything customer-facing, and every yes should have a written reason behind it.

The practical impact of an actionless alarm is that your first signal of a production incident comes from a customer, not from your own system. By the time a customer files a support ticket or a social media post surfaces, the incident has typically been running for an hour or more. Every minute of additional duration is additional cost — degraded transactions, SLA exposure, support load, and reputational damage. The monitoring infrastructure exists but isn't doing the job.

There's also a governance dimension. Compliance frameworks like SOC 2 and PCI require not just that you detect problems but that detection triggers a response. An alarm history showing hours in ALARM state with no corresponding action is a controls gap — evidence that the detection layer is present on paper and absent in practice. Auditors know what to look for, and this pattern is near the top of the list.

The resolution is straightforward and inexpensive: every production alarm should have at least one action, wired to the right tier of human attention. Exceptions should be deliberate — a non-production alarm intentionally left unwired, with a reason on the record. That's the standard that turns monitoring from a dashboard decoration into an operational guarantee.