Alert on Azure Monitor data-ingestion anomalies

Log Analytics is billed largely by the gigabyte ingested, so an ingestion anomaly is a billing event before it is anything else. A verbose flag or a misconfigured diagnostic setting can double a workspace's daily volume, and because the charge accrues quietly across a billing period, the first time anyone sees it is on the invoice unless something was watching. An ingestion alert moves that discovery from the end of the month to the same day, which is the difference between trimming a flag and explaining a variance.

The other side of the same coin is the cost of flying blind. If ingestion collapses because a source stopped sending, or because a daily cap silently shut collection off, you are no longer paying for data you are also no longer receiving the signal that data carried. Mean time to detect a stalled pipeline is the metric that matters here, and an alert on a volume drop or a 'Data collection stopped' event is what keeps it short.

The arithmetic is one-sided. A log search alert rule costs a small, fixed amount per month, documented in Azure Monitor pricing. Against that sits an unbounded ingestion overrun and the unmeasured cost of an outage you could not see. This is the rare control where the spend to fix it is a rounding error and the spend to ignore it is open-ended.

Priya is the finance partner who flagged the Log Analytics variance in the quarterly review. The platform team can tell her the workspace bill rose, but not why or when it started, because nothing was watching ingestion day to day; the variance only became visible once the invoice closed. Her instinct is not to approve a bigger monitoring budget; it is to ask for the early-warning signal that would have caught this in hours rather than weeks. The 'Usage' table query shows the answer plainly: one data type, AppTraces, went from six gigabytes a day to forty-two overnight when a verbose flag shipped, and every day it ran before anyone noticed was billable data the business paid for and did not need.

Priya reframes the work on the cost register. The fix for the spike itself is engineering time, turn the flag off, and costs nothing in Azure spend. The thing worth funding is the alert: a single log search alert rule on billable volume costs a small fixed amount per month and would have surfaced this on day one. The finance pack reads: 'The overrun was an undetected verbose flag running for eleven days; the remediation is free, and the control that stops it recurring is a fixed-price alert that pays for itself the first time it fires. The metric we should be tracking is mean time to detect an ingestion spike, and right now it is the length of a billing cycle.'

The cost model here is asymmetric in the way finance likes. A log search alert rule on ingestion costs a small, fixed, documented amount per month, while the overrun it guards against is unbounded: a verbose flag left running for a full billing cycle can add a large, unplanned line to the Azure Monitor bill, and the only thing standing between the misconfiguration and the invoice is whether anyone was told. Mean time to detect an ingestion spike is the metric that converts directly into avoided spend.

The drop side is harder to put a number on but no less real. When ingestion collapses or a daily cap halts collection, the business stops paying for that data and simultaneously loses the visibility it bought, so an outage runs longer and an audit trail has a hole in it. The finance role is to treat the alert as insurance with a known, trivial premium against two correlated risks, a runaway bill and an undetected blind spot, and to fund it on every workspace rather than debating it per incident.

Monitoring exists so that the business is not surprised. An ingestion anomaly is the moment the monitoring itself becomes the surprise: a logging change quietly trebles the Azure Monitor bill, or a source goes quiet and the dashboards keep looking healthy because nothing is arriving to turn them red. Both are visibility failures, and both are invisible until someone is told.

The leadership question is not 'how much are we spending on logs?' It is 'if the volume of monitoring data changes sharply tomorrow, in either direction, how long before we know?' The defensible end state is that every workspace has an alert on its ingestion volume and on the events that signal a stall, so a spike or a collapse reaches a person within hours rather than at month end.

This is a visibility decision, not a budget one. The alert costs almost nothing. What it buys is a shorter mean time to detect, both for runaway cost and for a monitoring blind spot, which is exactly the protection a monitoring estate is supposed to provide in the first place.

After a Log Analytics overrun showed up in the quarterly numbers with no one able to say when it had started, the CFO asked the obvious question at the next review: if our monitoring bill runs away again, when do we find out? The honest answer at the time was 'when the invoice arrives,' because ingestion volume was something the team looked at after the fact, not something that told them anything in the moment.

The platform team's response was to stop treating ingestion as a number on a bill and start treating it as a signal. They built an alert on each workspace's daily billable volume, added alerts on the 'Ingestion rate' and 'Data collection stopped' events so a dropped-data or capped-collection condition pages someone, and set a sensible threshold from each workspace's own baseline. The next time a diagnostic setting was switched to verbose, the alert fired the same afternoon and the flag was off within the hour. When the CFO asked the question again, the answer had changed from 'at the invoice' to 'within hours, in either direction, whether the volume spikes or a source goes quiet.' That shift, from discovering a monitoring problem at month end to being told the day it begins, is the confidence signal this capability delivers, and it costs almost nothing to reach.

Every monitoring estate exists to make sure the business is not surprised. An ingestion anomaly is the failure where the monitoring surprises you instead: a logging change quietly runs away with the bill, or a source goes quiet and the dashboards stay green because nothing is arriving to turn them red. Both undermine the one job monitoring has, and both are invisible until someone is notified.

The cost of getting this wrong is an overrun discovered at month end or an incident that ran longer because nobody could see it, while the cost of fixing it is a handful of near-free alert rules. This is the rare control where the risk of inaction is open-ended on both cost and visibility, and the cost of action is a rounding error, which is exactly the trade leadership should approve without hesitation.