Part III β The Speed Β· Lesson 7.4
Fixing
In one lineStop the bleeding first, find the cause second β rollback beats a clever hotfix at 3am.
Where you've seen itTypeError: undefined is not a function at t.n (main.7b2e04.js:1:48211) β useless, until source maps turn it into Cart.tsx:82.
Diagram
INCIDENT TIMELINE β mitigate before you diagnose
00:00 alert fires error rate 8%
00:02 ACKNOWLEDGE someone owns it
00:04 ASSESS how many users? which route?
what shipped in the last hour?
00:06 MITIGATE βββββββββββββββΊ rollback / feature-flag off
βββ users are safe now. The clock stops here.
00:20 DIAGNOSE source maps, breadcrumbs, traces
01:30 FIX + verify test that reproduces it first (4.6)
next POSTMORTEM blameless, action items with owners
β common mistake: debugging for 40 minutes while users keep failing.SOURCE MAPS β the difference between noise and a line number
minified at t.n (main.7b2e04.js:1:48211) β unusable
β
β .map file, uploaded at BUILD time
βΌ
mapped at CartTotal (src/Cart.tsx:82:14) β fixable
Upload maps to the error service, do NOT serve them publicly β
they publish your source (3.5).TRIAGE QUESTIONS, IN ORDER
1. How many users, which routes, which releases? scope
2. Did it start with a deploy? β usually yes
3. One browser/OS/device class only? compatibility
4. One region or CDN edge? infra
5. Third-party script or API involved? not yours
6. Reproducible locally with the same flags? β A/B variant mattersHow it works
- Acknowledge and own it. An unowned alert is an unhandled alert.
- Scope before fixing β percentage of users, routes affected, and whether it correlates with a release.
- Mitigate immediately: roll back, or flip the feature flag off. Both are faster and safer than a rushed patch.
- Diagnose with mapped stacks, breadcrumbs, and traces, reproducing with the same feature flags and experiment variant.
- Write a failing test first, then fix (4.6). Finish with a blameless postmortem whose action items have owners and dates.
Trade-offs
| Mitigation | When it's right |
|---|---|
| Rollback | Regression from a known recent deploy |
| Feature flag off | Feature is isolated and flagged |
| Hotfix forward | Rollback is impossible (migrations) |
| Scale up / rate limit | Load-driven, not code-driven |
Design for mitigation in advance: ship behind flags, keep deploys small and reversible, and make rollback a one-command operation. The quality of your 3am response is decided weeks earlier.
Interview angle
"Production error rate just spiked. Walk me through your response."
- Acknowledge, then scope: how many users, which route, which release β and check what deployed in the last hour.
- Mitigate before diagnosing: roll back or kill the flag, so the investigation isn't racing against user impact.
- Then map the stack, reproduce with the right flags, add a failing test, fix, and run a blameless postmortem.
Recap
- Mitigate first, diagnose second β rollback is a legitimate fix.
- Source maps uploaded at build time turn noise into line numbers; never serve them publicly.
- Your incident response quality is determined by decisions made long before the incident.