Approval-service outage: DB connection pool exhaustion from a config regression

expense-approval-service · Dana Okafor, Platform Eng · 2026-08-26

A config regression in PR #2291 removed the DB pool's acquire_timeout_ms, causing connections to queue instead of fail fast. Under normal load, the pool exhausted within minutes of v2.14.0 reaching 100% traffic with no canary stage, taking down both approval paths for 29 minutes until a rollback to v2.13.2 resolved it.

Both approval-resolution paths (normal and delegate routing) were fully unavailable for all users for 29 minutes with no workaround — requests hung until timeout rather than failing with a retryable error — and roughly 40 approvals were left stuck mid-flight, risking finance's month-end close SLA.

Impact

Affected: expense-approval-service — 100% of approval submissions (both normal and delegate-routing paths); ~40 approvals left stuck mid-queue. The approval workflow — the core of expense processing — was fully down; finance flagged risk to the month-end close SLA on 2026-08-28 if this recurs.

Timeline

1. 14:02
   DETECTIONPagerDuty

   PagerDuty pages on-call for approval-service 5xx rate at 27%, above the 25% threshold.

   [14:02] PagerDuty: 🔴 approval-service 5xx rate 27% (threshold 25%) — paging on-call

2. 14:03
   DIAGNOSISDana Okafor

   On-call confirms approval POSTs are timing out at ~9s median latency vs. a 200ms baseline, affecting both normal and delegate-routing approval paths.

   [14:03] Dana Okafor: approval POSTs timing out, ~9s median latency vs 200ms baseline. both normal and delegate-routing approvals affected, not just one path

3. 14:05
   DIAGNOSISsupport-bot

   Support-bot reports 14 tickets tagged "approval failed" in the last 5 minutes (baseline 1-2/hr), with the earliest ticket timestamped 13:58, before the alert fired.

   [14:05] support-bot: 14 tickets in the last 5 min tagged "approval failed" (baseline is 1-2/hr). earliest one timestamped 13:58, before the page fired

4. 14:11
   DIAGNOSISDana Okafor

   Root symptom identified: the DB connection pool is at 100% utilization, with connections queuing rather than failing fast. No pool-utilization alert existed; found by manually checking the DB dashboard.

   [14:11] Dana Okafor: found it — DB connection pool at 100%. connections are queuing, not failing fast... we don't have a pool-utilization alert, only caught this by pulling up the DB dashboard by hand

5. 14:19
   DIAGNOSISDana Okafor

   Root cause identified: v2.14.0 shipped to 100% traffic at 13:47 with no canary stage, and its diff shows acquire_timeout_ms was dropped from db_pool.yaml in PR #2291, so the pool now blocks indefinitely instead of failing fast after 2s.

   [14:19] Dana Okafor: yeah — v2.14.0 went to 100% traffic at 13:47, no canary stage. diff shows acquire_timeout_ms got dropped from db_pool.yaml in PR #2291

6. 14:24
   MITIGATIONDana Okafor

   Rollback to v2.13.2 initiated.

   [14:24] Dana Okafor: rolling back to v2.13.2 now

7. 14:31
   MITIGATIONDana Okafor

   Rollback complete; pool utilization dropping and error rate falling.

   [14:31] Dana Okafor: rollback complete, pool utilization dropping, error rate falling

8. 14:41
   RESOLUTIONDana Okafor

   Error rate held under 1% for 10 minutes after the rollback; incident marked resolved.

   [14:41] Dana Okafor: held steady for 10 more minutes, marking this resolved

Root Causes

• RC1 [config] acquire_timeout_ms was removed from db_pool.yaml in PR #2291, so the DB connection pool blocks indefinitely for a free connection instead of failing fast after 2s; under the traffic spike from v2.14.0's full rollout, this caused connections to queue until the pool was exhausted.
  [14:19] Dana Okafor: diff shows acquire_timeout_ms got dropped from db_pool.yaml in PR #2291 — the pool now blocks indefinitely for a free connection instead of failing fast after 2s

Contributing Factors

• CF1 [process] No pool-utilization alert existed, so diagnosis relied on manually opening the DB dashboard rather than being paged directly on the actual bottleneck.
  [14:11] Dana Okafor: we don't have a pool-utilization alert, only caught this by pulling up the DB dashboard by hand
• CF2 [process] v2.14.0 shipped directly to 100% traffic with no canary stage, so the config regression hit the entire user base simultaneously instead of a small fraction first.
  [14:19] Dana Okafor: v2.14.0 went to 100% traffic at 13:47, no canary stage

Symptoms

• 5xx error rate crossed the 27% alert threshold.
  [14:02] PagerDuty: 🔴 approval-service 5xx rate 27% (threshold 25%)
• Support ticket volume spiked to 14 tickets in 5 minutes tagged "approval failed", versus a baseline of 1-2/hr.
  [14:05] support-bot: 14 tickets in the last 5 min tagged "approval failed"

What Went Well / Wrong

Went well

• Alerting caught the regression within 4 minutes of impact starting (13:58 to the 14:02 page).
• Root cause was identified and a rollback executed within 22 minutes of detection, with no need for a hotfix under pressure.

Went wrong

• No pool-utilization alert existed, so the actual bottleneck was found by manually checking a dashboard rather than being paged on it directly.
• The config regression shipped to 100% of traffic with no canary stage, so there was no smaller-blast-radius warning before the full outage.

Action Items