Redis vs Postgres for Webhook Idempotency Keys

The choice surfaces the first time you ask “where do I record that I have already processed event_id?” — and it is load-bearing, because the idempotency store is what makes a duplicate webhook a no-op instead of a double charge. This page is a decision companion under Outbox Pattern & Event Publishing: the outbox guarantees at-least-once delivery, which means duplicates are certain, which means the idempotency store is non-negotiable. The real question is whether to back it with Redis (fast, in-memory) or Postgres (durable, transactional). The honest answer is that atomicity with your ledger usually decides it.

Trade-offs

The decisive row is “atomicity with the ledger.” If recording the event_id and mutating the ledger must succeed or fail together, only Postgres lets you put both in one transaction. Redis forces a two-phase dance (claim in Redis, then write the ledger) where a crash between phases either reprocesses or strands the key. Redis earns its place when the protected effect is itself idempotent or external, and sub-millisecond dedup matters at high volume.

Step-by-Step Implementation

1. Postgres: claim and mutate in one transaction

The unique constraint is the dedup. ON CONFLICT DO NOTHING ... RETURNING tells you in one round trip whether you won the claim, and the ledger write rides the same transaction.

CREATE TABLE consumed_events (
  event_id    UUID PRIMARY KEY,
  consumed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at  TIMESTAMPTZ NOT NULL DEFAULT now() + interval '3 days'
);

def process_with_postgres(event_id: str, payload: dict, db) -> None:
    with db.transaction() as txn:
        won = txn.execute(
            "INSERT INTO consumed_events (event_id) VALUES ($1) "
            "ON CONFLICT DO NOTHING RETURNING event_id", event_id)
        if not won:
            return  # ✅ duplicate, no-op
        apply_ledger_mutation(payload, txn)  # commits atomically with the claim

2. Postgres: expire keys with a cleanup job

There is no native TTL, so age out rows past the provider retry window on a schedule.

DELETE FROM consumed_events WHERE expires_at < now();  -- run every few minutes

3. Redis: atomic claim with SET NX EX

SET NX makes the claim atomic and EX gives you free expiry matched to the retry window. Treat the effect carefully: it happens after the claim, so it must be safe to repeat if the process dies before completing.

import redis
r = redis.Redis()
RETRY_WINDOW = 3 * 24 * 3600  # match provider retry horizon

def process_with_redis(event_id: str, payload: dict) -> None:
    claimed = r.set(f"idem:{event_id}", "1", nx=True, ex=RETRY_WINDOW)
    if not claimed:
        return  # ✅ duplicate within window, skip
    try:
        apply_idempotent_effect(payload)   # must tolerate at-least-once
    except Exception:
        r.delete(f"idem:{event_id}")       # ⚠️ release so a retry can re-attempt
        raise

4. Optional: Redis as a fast gate in front of Postgres

A common hybrid uses Redis to absorb the duplicate flood cheaply, with Postgres as the durable store of record. Reconcile the two so a Redis flush cannot silently permit reprocessing.

def process_hybrid(event_id: str, payload: dict, db) -> None:
    if not r.set(f"idem:{event_id}", "1", nx=True, ex=RETRY_WINDOW):
        return  # fast reject of obvious duplicates
    process_with_postgres(event_id, payload, db)  # durable, authoritative claim

Verification & Testing

Fire the same event_id concurrently from many workers and assert exactly one effect — this catches non-atomic claims. For Postgres, assert that a failed ledger mutation rolls back the consumed_events insert so the event can be retried. For Redis, simulate a crash after SET NX but before the effect and assert the key is released (or the effect is genuinely idempotent). Verify TTL: write a key, advance the clock past the window, and assert a re-delivery is reprocessed (correct, because the provider stopped retrying long ago).

-- Drift check: durable keys missing from the fast store (acceptable post-flush),
-- and any effect applied without a recorded claim (a bug)
SELECT le.event_id
  FROM ledger_entries le
  LEFT JOIN consumed_events ce ON ce.event_id = le.event_id
 WHERE ce.event_id IS NULL
   AND le.created_at > now() - interval '1 day';

Gotchas & Production Pitfalls

• Redis durability defaults can lose the claim. With async AOF/RDB, a node failure can drop seconds of keys, permitting reprocessing. Enable appendfsync everysec at minimum, and never let Redis be the sole record of a financial dedup decision.
• The two-phase Redis dance is not atomic. Claim-then-effect has a crash window. Either make the effect idempotent on its own or use Postgres so the claim and effect share a transaction.
• TTL shorter than the retry window reprocesses real duplicates. If the key expires before the provider stops retrying, a late retry sails through. Set the TTL to at least the full provider retry horizon (often 72 hours).
• Postgres without a cleanup job grows unbounded. The consumed_events table never self-expires. A forgotten cleanup job bloats it and slows the unique-constraint check over time.
• Eviction under memory pressure silently breaks dedup. A Redis maxmemory policy of allkeys-lru will evict idempotency keys to make room, reopening the duplicate window. Use a dedicated instance or noeviction with capacity headroom.

Related

• Outbox Pattern & Event Publishing
• Using the Outbox Pattern for Reliable Billing Events
• Building Idempotent Webhook Handlers in Node.js
• Webhook Retry & Timeout Strategies
• Webhook Processing & Backend State Management