Manage secrets (rotation and hygiene)

Secrets Manager charges about $0.40 per secret per month plus a few cents per ten thousand API calls, and the AWS-managed rotation Lambdas for RDS and similar databases are pre-built and effectively free to run. EKS secret encryption costs one KMS key, roughly a dollar a month. The remediation across this whole capability is engineering time and rounding-error spend, while the downside it mitigates, a breach traceable to an unrotated or unencrypted credential, routinely runs six to eight figures in incident response, notification and customer trust.

There is a genuine cost angle on the cleanup side. Stale secrets accumulate quietly: a typical multi-account org carries hundreds to thousands of secrets within a couple of years, and the storage line climbs past a few hundred dollars a month for credentials nobody uses. Because each secret has a unit cost, an owner tag and a LastAccessedDate, this is unusually tractable for FinOps: you can produce an aged inventory, attribute waste to teams, and calculate exact savings before deleting anything.

Frame each failing control by the credential behind it. A production payments database master password that is unrotated (SecretsManager.1/.4) or whose rotation is silently failing (SecretsManager.2) is a this-sprint priority with direct PCI exposure; a cancelled-POC secret nobody has read in a year (SecretsManager.3) is a five-minute cleanup that also trims the bill. Prioritise by sensitivity and by what each credential can reach, not by the raw finding count.

Priya is the finance and risk partner for a payments platform. Security Hub fires a batch of secrets findings: several with rotation disabled, one rotation-enabled secret whose last rotation has been silently failing, an EKS cluster without secret encryption, and a long list of secrets nobody has read in over a year. Her first instinct is not to ask what the fixes cost, because the answer is rounding-error spend: Secrets Manager runs about $0.40 per secret per month, the AWS-managed rotation Lambdas are effectively free to run, and EKS secret encryption is one KMS key at roughly a dollar a month. The downside it mitigates, a breach traceable to an unrotated or unencrypted credential, routinely runs six to eight figures.

She reads each failing control by the credential behind it rather than by the finding count. The production payments database master password whose rotation is silently failing (SecretsManager.2) is the one with direct PCI exposure and a this-sprint priority, because the dashboard reports it as managed while the credential ages undetected. The EKS.3 finding on the production cluster is next, because base64 was never a lock and one foothold turns into every secret the cluster holds. The long list of unused secrets (SecretsManager.3) is the rare control that is both a security and a cost win: each secret has a unit cost, an owner tag and a LastAccessedDate, so she produces an aged inventory, attributes the waste per team, and the cleanup trims a storage line that had climbed past a few hundred dollars a month. Her output is a worklist tiered by sensitivity, with the storage line tracked alongside the finding count so a falling line shows the cleanup is sticking.

The cost model is lopsided. Rotation is near-free, EKS secret encryption is about a dollar a month, and unused-secret cleanup actively reduces the bill. The downside is breach-scale: incident response, regulatory notification, customer churn and the cost of rotating every exposed credential, none of which appears on the cloud invoice but all of which lands on the business. There is no plausible cost-benefit case for leaving a high-value credential unrotated or a production cluster's secrets unencrypted.

The cleanup side is the rare control that is a cost win as well as a security one. Each secret has an identifiable unit cost, an attributable owner and a clear staleness signal, so you can produce an aged inventory, attribute waste per team, and recover the spend with a schedule-then-delete pass behind a recovery window. Track the Secrets Manager storage line alongside the finding count: a flat or falling line means the cleanup is sticking; a rising one means lifecycle discipline has slipped.

Tier the work by sensitivity. Rotation findings on PCI- or regulated-scope credentials sit in the must-fix-this-sprint bucket with the audit exposure spelled out; internal-tooling and dev secrets sit lower. The finance contribution is to make the asymmetry explicit so a control this cheap to fix never sits open across a reporting cycle on a credential that matters.

Applications connect to databases and external services using passwords and keys. This group of controls asks whether those credentials are well managed: do they rotate so a leak is only useful for a bounded window, does that rotation actually succeed rather than just being switched on, are unused credentials retired, and are stored secrets genuinely encrypted rather than merely encoded.

The leadership question is simple to state. If a credential leaked today and nobody noticed, how long would an attacker have it? With rotation on a 30-day cadence that actually succeeds, the answer is at most 30 days. Without rotation, or with rotation silently failing, the answer is forever. A rotation that is enabled but failing is the worst case of all, because the dashboard reports it as managed and the org looks elsewhere.

This is a breach-containment discipline, not a spending decision. The fixes are cheap; the value is in treating rotation success as a monitored outcome, retiring credentials when their service retires, and making real encryption a default on every cluster. The defensible end state is that every high-value credential rotates by policy, with documented exceptions for the rare cases that cannot.

After a routine review found that the primary RDS master credential had reported rotation enabled for 400 days while never actually rotating, because the rotation Lambda had no network route to the database and every scheduled run had timed out and rolled back, the CTO asked the security team one question: if a credential leaked today and nobody noticed, how long would an attacker have it. For that secret, the honest answer was forever, and the dashboard had said managed the entire time.

The team stopped treating rotation as a setting to tick and made secret hygiene a monitored outcome. Rotation is now on for every high-value credential at a cadence matched to its threat model, and a CloudWatch alarm on the rotation Lambda's Errors metric pages a human the day a rotation fails rather than at the next audit. Unused secrets are retired when their service retires, behind a recovery window so anything still depending on one fails loudly and can be undone. Every production EKS cluster has KMS envelope encryption on its secrets by default, with the key treated as a critical dependency. Rotation and encryptionConfig are baked into the Terraform and CloudFormation modules so new credentials and clusters are born compliant. The next review answered the CTO differently: every production credential rotates by policy with documented exceptions, and a failed rotation is a same-day alert. That standing posture, monitored as an outcome, is the governance signal.

The core question this capability answers is the one a board asks after any breach: if a credential was stolen today without anyone knowing, how long would an attacker have access? Without rotation, indefinitely. With rotation that actually succeeds, at most the rotation interval. The difference is the gap between a contained incident and a multi-year breach, and it is decided by whether rotation is monitored as an outcome rather than ticked as a setting.

The regulatory and governance stakes are concrete. A rotation-enabled-but-failing secret, a pile of unused credentials, or an EKS cluster with unencrypted secrets are each the kind of finding a regulator or assessor cites directly, and in a post-breach review an unrotated credential the tooling flagged months earlier is a governance failure, not just a technical one. The leadership question is whether every production credential rotates by policy with documented exceptions, and whether a failed rotation pages a human the day it happens.