When we took over monitoring for a multi-site client, the platform was generating about fourteen thousand alerts a month across roughly eleven hundred devices. The team had done the rational thing: they had created an inbox rule that filed them all away, and they watched a dashboard instead.
That is the real cost of alert noise. It is not annoyance. It is that alerting stops being the mechanism by which anyone learns something is wrong, and the organisation quietly reverts to finding out from users.
Six months later the same estate produced about four thousand one hundred alerts a month, and the number of genuine incidents detected before a user reported them went up. Here is what we changed, in the order we changed it.
1. Separate alerting from recording
Most platforms conflate two different jobs: collecting data, and telling a human to do something. Almost everything should be collected. Very little should page anyone.
We split every check into three tiers. Page means wake someone up. Ticket means create work for the next business day. Record means store it for trending and never notify. Most checks that were paging belonged in the third tier, and moving them there cost nothing in visibility because the data was still being kept.
2. Dependency suppression
The single biggest win. When a branch router goes down, the old configuration produced forty-one alerts: the router, both switches behind it, twenty-two access points, the NVR, the UPS and every server at that site. All of it was one event.
Model the topology so that children are suppressed when a parent is unreachable. One alert, correctly attributed, instead of forty-one that bury the cause. Every serious monitoring platform supports this and almost nobody configures it, because it requires knowing the topology, which brings us to the next point.
3. Fix the inventory before tuning the thresholds
About nine per cent of the devices being monitored no longer existed. Decommissioned kit, replaced switches, a test server from 2022. Each was faithfully alerting as unreachable, forever.
Reconciling the monitoring inventory against reality removed a meaningful slice of the volume before a single threshold was touched, and it also revealed things that were live and not being monitored at all.
4. Thresholds that reflect the workload
A database server at ninety per cent memory is healthy, because that is what a database does. A default template alerts on it every hour.
Tune per class of system, and use sustained-duration conditions rather than instantaneous ones. "CPU above ninety per cent for fifteen minutes" is an event. "CPU touched ninety per cent" is a backup job. Disk alerts should be based on projected time-to-full rather than a fixed percentage, because eighty per cent of a 12 TB volume is not urgent while eighty per cent of a 100 GB volume might be.
5. Maintenance windows that are actually used
Patch night generating four hundred alerts trains everyone to ignore alerts on every other night too. Scheduled maintenance windows, and an easy way for an engineer to suppress a device for two hours from their phone, remove a large and entirely predictable share of the noise.
6. Every alert must have a runbook
This is the discipline that keeps the improvement from decaying. If an alert cannot be answered with "here is what you do about it", it should not page anyone. Applying that rule honestly retires a surprising number of checks that exist only because they came with a template.
The test we apply. For each alert type ask: if this fires at three in the morning, is there an action a human should take right now? If the answer is no, it is a ticket or a record, not a page. Run this over your top twenty alert types by volume and you will remove most of your noise in an afternoon.
What we measure afterwards
| Metric | Before | After six months |
|---|---|---|
| Alerts per month | ~14,000 | ~4,100 |
| Alerts that paged a human | ~2,600 | ~310 |
| Incidents found by monitoring before a user reported | 38% | 81% |
| Median time from event to acknowledgement | 52 min | 6 min |
| Devices monitored | 1,104 | 1,240 |
Note the last row. We ended up monitoring more, not less. Reducing noise is not the same as reducing coverage, and conflating the two is why teams resist tuning.
Keeping it clean
Alert noise regrows. Review the top ten alert sources monthly and ask what changed. Retire checks when systems are decommissioned, on the same change ticket. And treat a repeated alert that nobody actions as a defect in the monitoring, not as a fact of life.
If your team has an inbox rule that files monitoring email away unread, you already know the state of things. That rule is not the problem, it is the symptom, and the fix is a day of tuning rather than a new platform.