Filter resource logs with diagnostic settings and DCR transformations

Log Analytics ingestion is billed per gigabyte, and unfiltered diagnostic settings are one of the most common reasons an Azure Monitor bill climbs without anyone deciding it should. The trap for finance is to treat the logging line as fat to be trimmed by simply collecting less, because the same logs are what let the team detect and resolve an incident quickly. Cut the wrong category and you have not saved money, you have traded a predictable monthly charge for a slower, more expensive outage later.

The right framing is that this capability lowers cost and protects visibility at the same time, which is rare. Selecting only the log categories the team investigates with costs nothing in Azure spend and often removes the largest share of the volume. Where a needed category is mostly noise, an ingestion-time transformation drops the noisy rows before they are billed, so you keep the signal and stop paying for the rest. Microsoft documents that Log Analytics commitment tiers can save up to roughly thirty per cent against pay-as-you-go once volume is predictable, so a one-off filtering pass also makes the remaining spend cheaper to commit to.

There is one billing nuance worth holding. For standard Analytics tables, if a transformation filters out more than half of the incoming data, Azure charges a small data-processing fee on the portion dropped above fifty per cent. It is bounded and usually far smaller than the ingestion it removes, and it does not apply at all when Microsoft Sentinel is enabled on the workspace. The headline still holds: filtering is cheaper than ingesting, and the saving is real.

Priya is the finance partner sitting in on the platform review when the Azure Monitor line comes up: it has roughly doubled in two quarters while headcount and the estate grew far less. The instinct is not to demand a flat percentage cut to logging, because logging is what keeps incidents short and a blunt cut could lengthen the next outage. The better question is which of that ingestion is buying anything. So Priya asks the team to break the bill down by table and to mark, next to each, whether on-call has queried it in a real incident this year.

The answer is stark. The top table is an over-broad diagnostic setting pouring verbose categories into Log Analytics that nobody has ever opened. The second is web-server logs dominated by successful health-probe requests, where the rare error rows are the only ones the team has ever used. Together they are the bulk of the increase, and both can be filtered without touching a single log the team relies on. The finance pack reads: 'The monitoring bill grew because collection grew by default, not by decision. The largest cut is free, it removes data nobody queries, and it leaves every log on-call actually uses in place. Once volume is predictable we can commit to a tier and take Microsoft's documented discount on top.'

The cost model here is unusually favourable. The largest cut, narrowing diagnostic settings to the categories the team genuinely uses, costs nothing in Azure spend and removes data that was never queried. The transformation cut is mostly free too, with only a small, bounded data-processing fee where a filter drops more than half of a table's incoming data, and none at all where Sentinel is enabled. Against that, the spend removed is recurring, per gigabyte, and growing on its own. The arithmetic only points one way.

The risk to manage is overcorrection. The same logs that look like pure cost on the bill are what shorten an incident, so the finance role is not to demand the lowest possible ingestion, it is to make sure every cut removes data nobody investigates with and leaves the rest intact. The metric that belongs on the cost review is ingestion gigabytes by table, annotated with whether on-call has actually queried that table in an incident, so the conversation is about removing waste rather than shaving visibility. Once volume is predictable, the right follow-on is a commitment tier, for which Microsoft documents savings of up to roughly thirty per cent against pay-as-you-go.

Monitoring is one of the few costs that grows on its own. Every new resource ships with an unfiltered diagnostic setting copied from the last one, and the Log Analytics bill rises in step with the estate rather than with any decision anyone made. Left alone, a meaningful slice of that spend buys log data nobody ever queries.

The leadership question is not 'can we spend less on logging?' It is 'are we paying to ingest data we will never use, while still keeping the logs that let us detect and resolve an incident fast?' Those are not in tension. Collecting only the categories the team investigates with, and dropping the routine noise before it is billed, lowers the bill and leaves mean-time-to-detect untouched.

The failure mode to avoid is a blunt cost cut that quietly removes visibility, because the cost of flying blind during an incident, longer detection, slower resolution, more customer impact, dwarfs the logging line it saved. The defensible position is filtered logging by design: the right categories on, the noise dropped at the door, and the saving banked without trading away the ability to see what is happening.

After the finance review flagged a monitoring bill that had doubled with no corresponding decision, the CTO asked the question that actually matters: are we buying anything with that, and if we cut it, can we still see what is happening when something breaks? The honest answer at the time was that nobody knew, because diagnostic settings were created by copying the last one and ticking 'allLogs', so the bill reflected a default rather than a choice.

The team's response was to stop treating monitoring as a fixed overhead and start treating it as a portfolio of decisions. They listed ingestion by table, separated the categories on-call genuinely investigates with from the ones nobody had ever queried, narrowed the diagnostic settings to the former, and added a transformation to drop the routine health-probe noise from the one busy category they had to keep. The bill fell by a large fraction, and crucially the next incident review confirmed mean-time-to-detect was unchanged, because not one log the team relied on had been removed. The next time the CTO asked the question, the answer had changed from 'we are not sure what we are paying for' to 'we collect exactly what we investigate with, we drop the rest at the door, and we can prove detection was not affected.' That shift, from logging by default to logging by design, is the confidence signal this capability delivers, and most of it costs nothing in Azure spend.

Most cost-cutting forces a trade: spend less, get less. Filtering logs is the rare exception, because the bulk of the saving comes from deleting data nobody reads, which leaves the visibility that protects mean-time-to-detect entirely intact and often makes the workspace faster to query during an incident. The risk of inaction is a bill that compounds quietly; the risk of action, if it is done by removing noise rather than signal, is close to nothing.

The one failure mode leadership should guard against is a crude version of this work that saves money by switching off whole categories the team needs, trading a small predictable cost for a slow, expensive outage later. The discipline is simple: every cut must remove data the team does not investigate with. Framed that way, this is the cost decision with the best risk-reward in the monitoring budget, a recurring saving with no loss of the visibility it is meant to protect.