Wire OK actions on CloudWatch alarms

A CloudWatch alarm has two critical state transitions: the ALARM state, which fires when a metric breaches a threshold, and the OK state, which fires when the metric recovers. Most alarms in AWS accounts are configured to send a notification — usually a page or a Slack message — only on the ALARM transition. When the metric recovers, nothing is sent. The control ALM-004 flags every alarm where this asymmetry exists.

The unit economics are straightforward. Every incident handled by an on-call engineer requires a manual verification step if there is no OK notification — they must open a dashboard, run a CLI query, or refresh a console view to confirm the system has returned to normal. Studies of on-call patterns put this at roughly 8–12 minutes per incident. At a burdened engineering rate, a team handling several incidents per week can easily spend tens of thousands of dollars annually confirming recoveries that a single SNS publish per alarm would deliver automatically.

The fix costs essentially nothing in cloud spend — wiring an OK action reuses the same SNS topic already attached to the alarm, so there is no incremental per-message cost worth modeling. The control is about labour efficiency: the dollar impact is entirely in the on-call time saved, and that return starts accumulating on the first incident after the fix is applied.

Dana is reviewing the quarterly engineering cost report and notices a line item: on-call support hours are running 15% above budget. She pulls the incident log and sees a pattern — a large share of incidents show a gap of 10–20 minutes between the engineer's mitigation action and the ticket close. The notes are consistent: 'confirmed metric recovered manually.'

She flags the ALM-004 finding count for the same period: 43 alarms with no OK actions across the production account. The math is simple — if each incident costs roughly 10 minutes of senior engineer time for manual recovery confirmation, and the account handles roughly 30 incidents per month touching these alarms, that's 5 hours of burdened engineer cost per month going to a task that a zero-cost configuration change would eliminate.

Dana brings the number to the next engineering review as a concrete ask: fix the 43 alarms, re-run the audit monthly. The cloud cost impact is negligible — SNS publishes are priced in fractions of a cent. The labour saving is immediate and recurring. It shows up in the next quarterly comparison as a reduction in on-call support hours, and the root cause is one line in a bulk fix script.

The most direct line item is on-call labour: every incident that ends without an automated recovery notification requires the on-call engineer to verify recovery manually. At 8–12 minutes per incident and a burdened senior engineer rate, an account with 30 monthly incidents touching alarms with no OK actions can easily accumulate 60–100 hours per year of labour going to a task that a configuration change would eliminate entirely. That's a recurring cost with no corresponding value delivered.

The second cost is mitigation over-run. When on-call lacks a recovery signal, they leave mitigations active longer than necessary — a scale-up running an extra 30–60 minutes waiting for confirmation, a feature flag kept off through peak traffic, a failover left in place until someone manually checks. Each of these has a direct cloud cost or a revenue impact. The compounding effect across a year of incidents adds up to a number that rarely appears on a cost report but is fully attributable to the missing notification.

The third cost is audit exposure. Accurate incident close times require an automated record of when the system recovered. Without OK action timestamps, resolution times are human estimates that don't survive a rigorous SOC 2 or ISO 27001 review. If the organization is working toward or maintaining a certification, retroactively reconstructing resolution times from log data is expensive and often inconclusive. The cost of remediation after an audit finding is typically far higher than the cost of wiring the OK actions before it.

Every CloudWatch alarm in an AWS account can send notifications in two directions: when a problem starts, and when it ends. Most accounts have wired the first and skipped the second. The result is an on-call team that gets paged when something breaks but receives no automated confirmation that it has recovered — they must go looking.

This is a process maturity issue. A monitoring system that only reports the start of an incident forces manual recovery verification on every on-call engineer. That adds latency to the incident close, introduces the risk that mitigations are left active longer than necessary, and creates a gap in the audit trail — the time of recovery becomes an estimate rather than a recorded event. For organizations subject to ISO 27001 or SOC 2, both the detection and the confirmed resolution of an incident are expected to be auditable; missing OK actions create a structural gap in that record.

The finding (ALM-004) is rated MEDIUM, not because the impact is minor, but because the system still self-recovers — the gap is in the notification layer, not the recovery itself. The governance question is whether incident management is operating by policy or by accident: a complete alarm configuration is a deliberate choice, and an incomplete one means the process depends on engineers filling in the gap manually every time.

A VP of Engineering at a mid-size SaaS company was reviewing a post-incident report after a payment processing degradation. The incident had been resolved within 12 minutes of detection — good. But the report's timeline showed that the on-call engineer had spent an additional 14 minutes after mitigation manually verifying the system had recovered before closing the ticket, because no automatic recovery notification came through.

She asked a simple question: 'Why does our monitoring system tell us when things break but not when they're fixed?' The answer — 43 alarms had AlarmActions wired but OKActions empty — took 20 minutes to fix in bulk. After the change, subsequent incident reports showed clean, automated close times. The recovery signal was in the audit trail alongside the alert, no manual verification step, no gap.

The lesson she took wasn't technical — it was organizational. A monitoring system that only reports failures but never confirms recoveries isn't a complete incident management system. Closing that gap is a policy decision as much as a configuration one: symmetric notifications should be the standard, enforced at the point where alarms are created, not patched after the fact when someone notices the trail going cold.

One-way alarms degrade an organization's incident management process in ways that compound over time. The immediate effect is that on-call engineers can't close incidents cleanly — they verify recovery manually, introducing both delay and the risk of a premature close if they're wrong. Over months, the team internalizes the alarm as a one-shot event rather than a state machine, which erodes their ability to reason about system stability from monitoring data alone.

The governance exposure is that incident audit trails become incomplete by construction. ISO 27001 and SOC 2 both treat detection and confirmed resolution as distinct auditable events. An account where OK actions are missing systematically produces audit trails with automated detection timestamps but estimated resolution times — a gap that becomes a finding during certification reviews and, worse, a gap in the post-incident record when an outage becomes a customer escalation.

The organizational signal is whether monitoring is governed by policy or by accident. A team that has reviewed its alarms and deliberately chosen symmetric notifications — ALARM and OK both wired — has made a process decision. A team where OK actions are missing on 40% of its alarms has an inherited default. The difference matters: the first team has a monitoring standard it can point to; the second has a gap it learns about at the worst possible time.