Grafana Alert Noise Reduction

Overview

"Too noisy" has several distinct mechanisms, each with a different correct fix. Applying the wrong one (or reaching for unless filters) either leaves the noise or silently disables the rule. Diagnose the mechanism first.

Step 0: noisy ≠ false

Verify the alert is a true positive before tuning. A rule paging about a real, unfixed fault is doing its job — the fix is the fault, and weakening its detection (threshold, severity, deletion) belongs to whoever owns the fault. Repackaging its notifications (grouping, sibling dedup) is fair game either way. Count the actual pages (notification groups per day) and identify which dimension multiplies them before touching anything.

When several mechanisms apply, do the cheapest reversible one first (grouping is a one-line change), re-measure, and only then add query guards.

Mechanism → fix

N pages per episode, one per target/node → collapse grouping

If notification_settings.group_by includes a fine-grained label (probe_target, node), one episode fans out into one page per value. Drop the fine label so one group forms per episode: