Add CloudWatch alarms to EC2 instances

Every EC2 instance continuously generates free metrics — CPU, status checks, network. Without a CloudWatch alarm attached, those metrics are invisible: the instance can be broken for hours and the only signal is a customer complaint. AWS Trusted Advisor flags this as COV-001. The fix is attaching alarms, and the cost is unusually transparent: each standard-resolution alarm costs $0.10 per month, flat.

The finance framing is a coverage question: four baseline alarms per instance at $0.40/instance/month gives you CPU saturation, hardware fault detection, OS-level failure detection, and disk pressure. On a 100-instance fleet that's $40/month for end-to-end coverage. The alternative — zero alarms — is not a saving, it's an uninsured exposure. An undetected failure on a production instance during business hours typically costs multiples of the entire fleet's annual monitoring budget in incident response and revenue impact.

This control is also a good lens on fleet hygiene. The instances most likely to be unmonitored are the ones nobody owns: contractor-built boxes, cloned AMIs, instances that outlived their original project. A systematic alarm-coverage program surfaces these orphans as a byproduct — and decommissioning a handful of them usually recovers more cost than the alarms themselves add.

Priya is the finance partner for the infrastructure team. At the quarterly review, the Trusted Advisor report flags 47 EC2 instances with COV-001 — zero alarms attached. Her first instinct is not to approve blanket remediation; it's to ask a tiering question: which of these 47 instances are production workloads where an undetected failure actually costs money, and which are dev boxes where the impact is inconvenience at most?

The team pulls the instance list with environment tags. Thirty-one are tagged Environment=prod, including the marketing web server, the API gateway nodes, and several data processing instances. The remaining sixteen are dev, staging, or untagged. Priya agrees the four-alarm baseline on the 31 production instances is straightforward: 31 × $0.40/month = $12.40/month for full production coverage. The dev instances she treats as a separate conversation — some should be decommissioned entirely, which would save more than the monitoring would cost.

Her output for the finance pack is simple: "$12.40/month closes the production monitoring gap. Separately, 16 dev/untagged instances are being reviewed for decommission." The alarm cost is trivially approved; the real value Priya drives is surfacing that 16 instances of uncertain ownership are costing real compute spend with no monitoring and possibly no active use.

The direct financial exposure from an unmonitored instance is the elapsed time between failure and detection multiplied by the business cost of that failure. On a production B2B application, an undetected failure that surfaces six hours later via a customer ticket typically means missed SLOs, support escalation, and engineering incident response — costs that stack up quickly against a $0.10/month alarm that would have paged someone within five minutes.

The self-healing angle has a hard dollar value too. The StatusCheckFailed_System alarm with an ec2:recover action is the cheapest form of business continuity available in AWS: free status-check metric plus a $0.10 alarm triggers automatic hardware migration in about three minutes, with no on-call engineer in the loop. A 3 a.m. hardware fault on a monitored, auto-recovering instance is a non-event. The same fault on an unmonitored instance is an incident with an on-call page, a manual restart, and whatever downstream impact accumulated during the gap.

The cost math scales predictably with fleet size, which makes it easy to model. Four alarms per instance at $0.40/month: a 100-instance production fleet is $40/month, a 500-instance fleet is $200/month, a 1,000-instance fleet is $400/month. These are not material monitoring costs. The crossover point where you'd reconsider blanket per-instance alarms is in the tens of thousands of short-lived nodes — at which point you restructure alarming up the stack rather than eliminating it.

A secondary financial benefit of a coverage program is fleet hygiene. The instances most likely to show up bare in a COV-001 audit are also the most likely to be orphaned compute spend — nobody owns them, nobody decommissions them. Closing the alarm gap often surfaces instances that should be terminated, recovering EC2 spend that dwarfs the monitoring budget.

EC2 instances generate health signals continuously. Without alarms, those signals go nowhere — a server can fail, saturate, or fill its disk and the first anyone knows is a customer complaint. AWS Trusted Advisor tracks this gap as COV-001. Closing it means wiring four baseline alarms per instance at roughly $0.40/instance/month.

The leadership question is straightforward: which instances are running in production without basic failure detection? The answer on most accounts is not zero — it's typically a mix of forgotten boxes, old clones, and standalone instances that escaped the standard provisioning process. This control forces those gaps into the open. The cost of fixing them is predictable and small; the cost of leaving them is periodic operational surprises at the worst possible times.

After a late-night incident where a production web server had been pegged at 100% CPU for six hours before a customer complaint surfaced it, the CTO asked a simple question: why did nobody know? The answer was that there were no alarms — the instance had been launched manually and never wired into the standard monitoring stack.

The team's response was to treat alarm coverage as a policy question rather than a per-instance engineering task. Production instances get the four-alarm baseline automatically on launch via tag-driven Lambda provisioning. Any instance without that baseline shows up on the weekly COV-001 report and is remediated or documented within a week.

Two months later the same CTO asked the same question in a different form: "if a server broke right now, how quickly would we know?" The answer had changed from 'when a customer tells us' to 'within five minutes.' That's the shift this control enables — and it's the one-line confidence signal that belongs on any operational review.

Unmonitored EC2 instances are a governance gap as much as an engineering one. The question isn't whether the team is competent — it's whether the systems that carry business risk have the basic detection infrastructure to give anyone a chance to respond. A production instance with zero alarms means that when it fails, the business is relying on luck, customer complaints, or a stalled batch job to surface it. That's not a posture; it's an accident waiting to happen.

The auto-recovery capability makes this especially stark for leadership. A hardware fault on a properly alarmed instance resolves automatically in three minutes. The same fault on an unmonitored instance becomes a midnight call, a manual restart, and a post-incident review. The people who pay for that incident — in overtime, in customer trust, in SLA credits — are rarely the same people who left the alarm off when the instance was launched.

This control is also a proxy for how mature the provisioning process is. Accounts with high alarm-coverage rates have standardised their instance launch process so monitoring is applied automatically, not manually remembered. Accounts with dozens of COV-001 failures have a process gap upstream: instances are being launched by hand, by contractors, or from old AMIs outside any standard. That upstream gap is worth more executive attention than the alarm count itself.