Building Idempotent Webhook Handlers in Node.js

You reach for this the first time a payment provider double-fires a webhook in production and you discover a customer was charged twice — or a subscription was suspended and reinstated in a flickering loop. Duplicate deliveries are not an edge case; they are the contract. A 200 that takes too long, a deploy that drops a connection, or the provider’s own retry policy all produce repeats of the same invoice.payment_succeeded. Building idempotent webhook handlers in Node.js is about making the second, third, and fiftieth delivery into safe no-ops. This is the concrete Express implementation of the idempotent webhook consumer pattern; for the broader pipeline it plugs into, see Webhook Processing & Backend State Management.

Trade-offs

The core decision is where the idempotency key lives. The options differ sharply on durability, latency, and auditability.

For financial events the answer is Postgres as the authority, optionally fronted by Redis. The database survives restarts and cache flushes and gives you a queryable audit trail — both non-negotiable for PCI-DSS and SOC 2. The detailed reasoning lives in Redis vs Postgres for Webhook Idempotency Keys.

Step-by-Step Implementation

1. Verify the signature on the raw body

Use express.raw() so the bytes you HMAC are exactly what the provider signed — parsing first and re-serializing changes the bytes and breaks verification. Compare with a timing-safe equality check.

const express = require('express');
const crypto = require('crypto');
const { Pool } = require('pg');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

app.use('/webhooks/billing', express.raw({ type: 'application/json' }));

function verifySignature(rawBody, signature) {
  const expected = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET)
    .update(rawBody)
    .digest('hex');
  const a = Buffer.from(expected);
  const b = Buffer.from(signature || '');
  return a.length === b.length && crypto.timingSafeEqual(a, b);   // ✅ constant-time
}

2. Define the idempotency key and the store

The provider event ID is the key. Back it with a unique-constrained table that doubles as the audit log.

CREATE TABLE webhook_events (
  id            VARCHAR(255) PRIMARY KEY,           -- provider event id = idempotency key
  type          VARCHAR(64)  NOT NULL,
  status        VARCHAR(16)  NOT NULL DEFAULT 'processing',
  received_at   TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at  TIMESTAMPTZ
);

3. Claim the key atomically and process in one transaction

INSERT ... ON CONFLICT DO NOTHING makes the claim the lock. If the insert affects zero rows the event is a duplicate, so acknowledge 200 and stop. Otherwise process state and write the outbox row in the same transaction.

async function webhookMiddleware(req, res, next) {
  const signature = req.headers['x-webhook-signature'];
  if (!verifySignature(req.body, signature)) {
    return res.status(401).json({ error: 'Invalid signature' });   // ✗ reject forgery
  }

  const event = JSON.parse(req.body);
  const idempotencyKey = event.id;
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const claim = await client.query(
      `INSERT INTO webhook_events (id, type, status)
       VALUES ($1, $2, 'processing')
       ON CONFLICT (id) DO NOTHING`,
      [idempotencyKey, event.type]
    );

    if (claim.rowCount === 0) {                                     // ⚠️ duplicate delivery
      await client.query('ROLLBACK');
      return res.set('X-Webhook-Status', 'duplicate').status(200).json({ received: true });
    }

    req.event = event;
    req.dbClient = client;          // hand the open tx to the route handler
    req.commit = async () => {
      await client.query(
        `UPDATE webhook_events SET status = 'done', processed_at = now() WHERE id = $1`,
        [idempotencyKey]
      );
      await client.query('COMMIT');
    };
    next();
  } catch (err) {
    await client.query('ROLLBACK');
    client.release();
    res.status(500).json({ error: 'Idempotency check failed' });
  }
}

app.post('/webhooks/billing', webhookMiddleware, async (req, res) => {
  const { event, dbClient } = req;
  try {
    // domain mutation + outbox row, all inside the SAME transaction
    await dbClient.query(
      `UPDATE subscriptions SET state = $1 WHERE subscription_id = $2`,
      [mapState(event), event.data.subscription_id]
    );
    await dbClient.query(
      `INSERT INTO billing_outbox (aggregate_id, event_type, payload) VALUES ($1, $2, $3)`,
      [event.data.subscription_id, event.type, event.data]
    );
    await req.commit();                                            // ✅ atomic commit
    res.status(200).json({ processed: true });
  } catch (err) {
    await dbClient.query('ROLLBACK');                              // ✗ key not retained
    res.status(500).json({ error: 'Processing failed' });
  } finally {
    dbClient.release();
  }
});

The critical property: the idempotency key is only durably retained when the business logic commits. If processing throws, the rollback removes the claim so the provider’s retry can succeed cleanly.

Verification & Testing

Assert exactly-once under concurrency by firing the same event ID through two simultaneous requests and checking the row count:

-- After a concurrent double-delivery, expect exactly one row, status 'done'.
SELECT count(*) FROM webhook_events WHERE id = 'evt_1Mq... ';   -- expect 1
SELECT count(*) FROM billing_outbox WHERE payload->>'event_id' = 'evt_1Mq... '; -- expect 1

In an integration test, post a payload with a tampered body and assert 401 (signature forgery test). Post the same valid event twice sequentially and assert the second returns 200 with X-Webhook-Status: duplicate and that no second outbox row appears. Simulate a processing failure (throw inside the route) and assert the webhook_events row is absent afterward, proving a retry can still proceed.

Gotchas & Production Pitfalls

• Parsing before verifying: if you let express.json() run first, you HMAC re-serialized bytes that no longer match the provider’s signature and every webhook fails 401. Mount express.raw() on the webhook route only.
• Keys stuck in processing: an unhandled promise rejection between claim and commit can leave a row in processing forever, swallowing the event. Always wrap claim plus logic in one transaction and run a sweep that deletes processing rows older than the provider’s max retry window.
• Trusting Redis TTL alone: provider clock skew or a mid-cycle eviction lets a duplicate slip past a Redis-only check. Keep Postgres as the source of truth and treat Redis as a pre-filter.
• Connection-pool exhaustion under retry storms: a provider outage produces a flood of retries that drains the pool. Add ingress rate-limiting and a circuit breaker so the handler sheds load instead of hanging.
• Deadlocks on high-concurrency upserts: concurrent claims for adjacent keys can deadlock. Retry the transaction with exponential backoff on Postgres error code 40P01 rather than returning 500 to the provider.

Related

• Idempotency & Event Deduplication
• Redis vs Postgres for Webhook Idempotency Keys
• Using the Outbox Pattern for Reliable Billing Events
• Syncing Subscription Status Across Microservices