FSD Frontend System Design

Part III — The Speed · Lesson 7.3

Alerting

In one lineAlert on user-visible symptoms with an owner and an action — everything else is noise that trains people to ignore pages.

Where you've seen itThe third 3am page in a week for something nobody can act on. By night four, real alerts get ignored too.

  • Time14 min
  • DiagramMetric with threshold and window + the alert quality test
  • Chapter7 · Logging & Monitoring

Diagram

diagram
THRESHOLD + WINDOW — both are required

  error rate
    ▲
 8% │                          ╭──────╮      ← sustained: PAGE
    │                         ╱        ╲
 5% │─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─╱─ ─ ─ ─ ─ ─╲─ ─ ─  threshold
    │        ╱╲            ╱              ╲
 2% │───────╱──╲──────────╱                ╲───
    │      ▲    ▲
    └──────┼────┼──────────────────────────────► time
           │    │
     single spike crossing the line = NOT an alert
     "above 5% for 5 minutes"       = an alert

  Without a window, every deploy blip becomes a page.
diagram
THE ALERT QUALITY TEST — all four, or delete it

  1. Does it mean a USER is affected?      not "CPU is 80%"
  2. Is there an ACTION to take right now?  else it's a dashboard
  3. Does it have an OWNER?                 else nobody moves
  4. Would you want to be WOKEN for it?     else it's a ticket
diagram
SEVERITY LADDER — route by impact, not by metric

  P1  PAGE        checkout success < 95%          money stopped
                  JS error rate > 5%              app broken
                  site unreachable

  P2  SLACK       p75 LCP > 4s for 15 min         degraded
                  API 5xx > 1% for 10 min

  P3  TICKET      new error type, low volume      investigate later
                  bundle size +10%

  P4  DASHBOARD   everything else                 look when curious

How it works

  1. Alert on symptoms, not causes. "Checkout success rate dropped" is actionable; "memory rose" is a hypothesis.
  2. Set the window from the metric's noise. Volatile metrics need longer windows or they page on nothing.
  3. Use burn-rate alerts on an SLO (e.g. 99.5% of checkouts succeed) instead of raw thresholds — they scale with traffic and ignore quiet-period noise.
  4. Attach a runbook link to every alert: what it means, what to check first, how to mitigate, who to escalate to.
  5. Review alerts monthly. Delete any that fired without action; each one is training people to ignore the rest.

Trade-offs

Tight thresholdsLoose thresholds
Catch issues earlyFewer false pages
Alert fatigue, ignored pagesUsers notice before you do
More on-call burnoutSlower detection

Anomaly vs static: a static threshold is predictable but wrong at 3am when traffic is 5% of daytime. Rate-based metrics (percentage, not count) handle traffic variation far better than absolute counts.

Interview angle

"Design alerting for a checkout flow."

  • One P1 on the business outcome: checkout completion rate below its normal band for five minutes.
  • Supporting P2s on payment API error rate and p75 checkout LCP, so you learn why before the P1 fires.
  • Every alert names an owner and links a runbook; anything without an action becomes a dashboard, not a page.

Recap

  • Symptom, threshold, window, owner, action — an alert missing any of these should be deleted.
  • Rate-based metrics beat absolute counts because traffic varies.
  • Alert fatigue is a bigger outage risk than missing one alert.