Ested SonarQube, Semgrep, and Checkmarx on our payment service. none caught the database race condition that caused duplicate charges.

we run a SaaS platform with about 40k users. payment processing is handled by a Node.js microservice running 3 instances behind a load balancer, using Stripe webhooks and Postgres. last month we had 7

we run a SaaS platform with about 40k users. payment processing is handled by a Node.js microservice running 3 instances behind a load balancer, using Stripe webhooks and Postgres. last month we had 7 cases of duplicate subscription charges over 2 weeks. took us 3 days to find the root cause. our entire static analysis stack - SonarQube, Semgrep, and a $35k/year Checkmarx enterprise license - found nothing. what happened is: // POST /webhooks/stripe async function handlePaymentSuccess(req, res) { const event = req.body; const session = event.data.object; const userId = session.metadata.user_id; const planId = session.metadata.plan_id; // Check if we already processed this session const existing = await db.query( 'SELECT id FROM subscriptions WHERE stripe_session_id = $1', [session.id] ); if (existing.rows.length > 0) { console.log('Session already processed:', session.id);

return res.json({ received: true }); } // Create subscription record await db.query( \INSERT INTO subscriptions (user_id, plan_id, stripe_session_id, status)` VALUES ($1, $2, $3, 'active')\,` [userId, planId, session.id] ); // Update user account await db.query( 'UPDATE users SET plan = $1, status = $2 WHERE id = $3', [planId, 'active', userId] ); res.json({ received: true }); } standard check-then-insert pattern. looks fine. what broke Stripe's documentation states: "Your endpoint must quickly return a successful status code (2xx) prior to any complex logic that could cause a timeout." we had a slow database query (table lock from a migration running in the background). response took about 8 seconds. Stripe timed out and retried the webhook. When Stripe retries an event, they generate a new signature and timestamp for the new delivery attempt, but the event ID remains the same. 10:23:15.120 - Instance A receives webhook (event_abc123) 10:23:15.140 - Instance A: SELECT... WHERE stripe_session_id = 'cs_xyz'

Result 0 rows 10:23:17.200 - Instance B receives retry (same event_abc123) 10:23:17.220 - Instance B: SELECT... WHERE stripe_session_id = 'cs_xyz'

Result 0 rows ← Instance A hasn't committed yet 10:23:23.100 - Instance A: INSERT subscriptions... 10:23:23.110 - Instance A:

returns 200 to Stripe 10:23:23.150 - Instance B: INSERT subscriptions... ← duplicate! 10:23:23.160 - Instance B:

returns 200 to Stripe classic time-of-check-to-time-of-use (TOCTOU) race condition at the database level across distributed service instances. why it happened: * multiple service instances (standard microservice setup) * Stripe webhook retry hits a different instance * Postgres READ COMMITTED isolation level (the default) allows both transactions to read before either commits - both see zero rows. * both proceed to INSERT * no database constraint to prevent duplicates happened 7 times over 2 weeks because it requires specific timing - webhook retry arriving while first request is still processing but hasn't committed. sonarqube 10.4: * code smells (use const, extract strings) * cognitive complexity: * bugs: 0 * quality gate:

PASSED ✓ * missed the race condition completely semgrep 1.50: * suggested helmet middleware * SQL injection false positive (parameterized queries) * caught one missing await in different file * style warnings didn't work - semgrep is syntax-based, can't model concurrent execution checkmarx sast… * "insufficient logging" * "missing input validation" * SQL injection false positives * error handling alert * concurrency issues found: 0 why they all failed: race conditions materialize from timing of requests, pattern-based static analysis can't reason about concurrent execution. static analyzers see: single execution path, syntax patterns they don't see: multiple instances, interleaving queries, transaction timing, network retries literally paying over 50k/year. and cant catch a simple textbook TOCTOU race condition that a single UNIQUE constraint would have prevented.