How to Reduce Alert Noise in ServiceNow Event Management
Alert noise is usually not caused by one bad monitor. It is caused by raw events being promoted into operational work before they are normalized, filtered, enriched, and grouped.
Start with evidence, not opinions
Export alert and incident counts for the last 30 days. Sort by message key, source, CI, metric, and assignment group. The first tuning pass should target the patterns that repeatedly create incidents without improving response.
Noise review fields:
- source
- node or CI
- resource/component
- metric name
- severity
- assignment group
- incident created? yes/no
- reopened? yes/no
- resolved within 10 minutes? yes/noClassify each noisy pattern
| Pattern | Likely action |
|---|---|
| Same alert every few minutes | Deduplicate by message key and resource |
| Non-production system creating major incidents | Route separately or stop incident creation |
| Unknown CI or “Other” CI | Fix CI binding and ownership fallback |
| Metric clears before operator opens ticket | Add duration or consecutive sample threshold |
| Maintenance window alerts | Fix maintenance suppression or blackout calendar |
Use duration thresholds carefully
A duration threshold prevents one bad sample from creating noise. It should not delay true outage detection. CPU at 95% for one sample is probably not an incident. A synthetic transaction failure for all users should page quickly.
Fix incident creation rules
Incident creation should require an actionable owner. If the support group is missing, use the change group. If both are missing, assign to a central SysOps or monitoring triage queue and flag the CI data issue in the incident description.
Weekly noise review
- Top 10 alert names by count.
- Top 10 CIs by incident count.
- Top 10 rules creating incidents.
- Alerts resolved within 10 minutes.
- Alerts closed as duplicate, no action, known issue, or monitoring error.