How to set anomaly detection thresholds that do not cry wolf every Monday
Good anomaly detection thresholds compare each day with its own normal: the same weekday, the same season, with data that has finished arriving. Set sensitivity by the kind of metric, mark the events you already know about, and send only urgent alerts right away.
Most alert setups fail in the first week. Monday looks like a crash next to Sunday, a promo looks like a spike, and yesterday looks low because the data is still arriving. The team mutes the channel and never turns it back on. These metric alerting best practices are for the person cleaning that up.
Why static percent-change alerts fail on seasonal metrics
A rule like “alert if today is 20% below yesterday” sounds sensible. On real business data it fires for four reasons that are not problems at all:
- Weekly shape. Most businesses have a weekly rhythm. Comparing Monday with Sunday, or Saturday with Friday, measures the calendar, not a change.
- Yearly seasons. A heating company in spring or a tax practice in May is quiet every year. Comparing with last month flags a slump that is normal.
- Small numbers. Going from 4 form fills to 2 is a 50% drop. On a small base, percentages swing wildly and mean little.
- Late data. Search Console usually arrives two to three days behind. Google says standard GA4 reports can take 24 to 48 hours to finish processing. Google Ads keeps adding conversions to the day of the click. Many POS systems send the day in one batch at closing. Judge a day too early and it always looks low.
If you want to reduce false positive alerts in analytics, fix these four before you touch the sensitivity. Most of the noise comes from here.
Choosing a baseline: rolling median, prior-period, or forecast residual
Anomaly detection needs a steady daily series with a known normal range. The baseline is how you define that normal. There are three common choices:
| Baseline | Good for | Watch out for |
|---|---|---|
| Rolling median of the same weekday | Steady daily series with a clear weekly shape. | Misses yearly seasons and holidays. |
| Prior period (same week last year) | Businesses with strong seasons, like heating, tax or retail. | Needs a year of clean history. Moving holidays and calendar shifts throw it off. |
| Forecast residual | Metrics with trend and season together, once you have the history. | More setup, and a bad model creates its own false alarms. |
A good default
Start with the same weekday over the last several weeks. “Calls down 38% against the last four Tuesdays” is a far better alert than “down 38% against yesterday”. Use a median rather than an average, so one odd week does not move the baseline. Then add a yearly check for metrics with strong seasons, so a slow month that is slow every year stays quiet.
Set a range, not a single line
Build the normal range from how much that weekday usually varies, and alert only outside it. Add a minimum size of change in real units, such as orders or calls, so small numbers cannot set it off on their own.
Sensitivity tiers by metric class: revenue, traffic, support volume, deliverability
One sensitivity for every metric guarantees noise. Group metrics by class and set each class on purpose:
| Class | What to watch | How sensitive |
|---|---|---|
| Revenue and payments | Orders per hour, checkout completion, failed payments, disputes. | Most sensitive. A zero where there is normally activity is always an alert. |
| Traffic and rankings | Sessions and conversion rate by landing page, impressions and position by page. | Less sensitive. Judge only days whose data has settled. |
| Support volume | New tickets per hour, by tag. | Watch for spikes, not drops. A spike in one tag right after a change is the signal. |
| Email deliverability | Bounces, unsubscribes and clicks by send. | Judge each send against your own list's history. Ignore opens. |
Two notes on the email row. Apple Mail Privacy Protection loads images ahead of time, so open counts run high and tell you little. Clicks are the steadier signal. And judge traffic and ranking only once the source has caught up, which for Search Console means about two days later.
Suppressing known events: promos, deploys, billing cycles
Many false alarms are events you knew about in advance. Put them on a calendar the alerts can read:
- Promos. Expect the lift during a sale and the dip after it. Widen the range for the promo window and the days after, rather than switching alerts off.
- Billing cycles. Renewal days bring more payments and more failed payments. Compare a renewal day with past renewal days, not with an ordinary Tuesday.
- Holidays. Mark the ones that matter to your business, including the ones that move each year.
- Tracking changes. A new tag or a changed form can create a cliff in the data that is not a change in the business. Note the date.
Do not mute deploys
Deploys and page edits are different. They are a common cause of the alerts that matter most, like a booking form that stops sending after an edit. Record deploy times and page changes next to the metrics instead, so an alert can say “this started right after Tuesday's 2:10 PM edit”.
Alert routing: what belongs in Slack now vs. a Monday digest
The fastest way to lose a team's trust is to send everything right away. Split alerts in two, and keep the urgent channel quiet.
Send now
- A number that is normally active drops to zero: no orders, no form fills, no bookings.
- Failed payments or disputes jump well above their usual range for that day.
- Checkout completion falls sharply right after a deploy or a page edit.
- An ad budget runs out early in the day.
- A support tag spikes in the hour after a change.
Save for the Monday digest
- Traffic and conversion drift on single landing pages.
- Ranking and impression moves, which arrive days late anyway.
- Bounce and unsubscribe trends across sends.
- Cost per click creeping up over the week.
- Every alert from last week, with what caused it, so the thresholds keep improving.
Every urgent alert should say three things: what changed, how serious it is, and what to check first. An alert that only says “sessions down” sends someone digging through five dashboards.
A two-week tuning checklist
Week one: watch without sending
- List each metric, give it a class from the table above, and name who acts on it.
- Set a same-weekday baseline for each, plus a yearly check where the business has seasons.
- Write down how late each source arrives, and only judge days after that delay.
- Put promos, renewal days, holidays and tracking changes on the calendar.
- Run the alerts silently for the week. Log every one that would have fired.
Week two: tune and switch on
- Mark each logged alert: real problem, known event, or noise.
- Widen the range or add a minimum change for every metric that fired on noise.
- Add a calendar entry for every alert that was a known event.
- Move metrics that never produced a real problem to the digest.
- Switch on the urgent channel only for what is left, and review the digest each Monday.
This is how askotter's change detection works. It compares each day with what is normal for your business at that time of year, so a quiet month that is quiet every year does not set off an alert. When a number falls outside that range, you get a text saying what changed, how serious it is and what to check first. It comes with every plan, from $99 a month. Read more about our marketing analytics.