Idempotency & Event Deduplication

Idempotency is the property that processing the same billing event twice produces the same result as processing it once — and it is the single most important guarantee in Webhook Processing & Backend State Management. Payment providers deliver at least once: a network blip, a slow 200, or an aggressive retry policy means the same invoice.paid can land two, five, or fifty times. Without a deduplication layer, each delivery is a fresh charge, a fresh ledger entry, a fresh dunning email. Event deduplication turns that unreliable stream into exactly-once effects so that ledger integrity holds through charge successes, dunning failures, and tax recalculations alike. This page covers the key architecture, the storage choices that enforce it, and how it composes with state machines and the ledger.

Prerequisites

• A deterministic idempotency key derivable from every inbound event
• An idempotency store: a Postgres UNIQUE-constrained table or Redis with SET NX
• The webhook signing secret available to the handler for signature verification
• A versioned subscription table for monotonic out-of-order event handling
• A transactional outbox table for reliable downstream publishing
• All event timestamps normalized to UTC

Architecture & Data Flow

The idempotency gate sits between signature verification and business logic. The inputs are verified, possibly-duplicate events; the processing step claims the key atomically and runs the effect in a transaction; the outputs are exactly-once ledger entries and outbox events. A duplicate never reaches business logic — it is acknowledged and dropped at the gate.

Implementation Walkthrough

1. Derive a deterministic key

Prefer the provider’s event ID. When the provider does not supply one, hash stable fields so retries of the same logical event collide.

import hashlib

def idempotency_key(event: dict) -> str:
    if event.get("id"):
        return event["id"]                                  # provider event id, best case
    raw = f'{event["customer_id"]}:{event["amount_cents"]}:{event["currency"]}:{event["created_utc"]}'
    return hashlib.sha256(raw.encode()).hexdigest()         # deterministic fallback

2. Claim the key with a unique constraint

The insert is the lock. Concurrent duplicates race for the constraint; exactly one wins.

CREATE TABLE webhook_ingest_log (
  idempotency_key    VARCHAR(255) NOT NULL,
  tenant_id          UUID         NOT NULL,
  provider_event_id  VARCHAR(255),
  status             VARCHAR(16)  NOT NULL DEFAULT 'pending',
  created_at         TIMESTAMPTZ  NOT NULL DEFAULT now(),
  CONSTRAINT uq_tenant_idempotency UNIQUE (tenant_id, idempotency_key)
);

-- Claim: zero rows returned ⇒ duplicate, ack and stop.
INSERT INTO webhook_ingest_log (idempotency_key, tenant_id, provider_event_id)
VALUES (:idempotency_key, :tenant_id, :provider_event_id)
ON CONFLICT (tenant_id, idempotency_key) DO NOTHING
RETURNING idempotency_key;

3. Process the effect inside one transaction

The dedup claim and the state change must commit or roll back together; otherwise a crash leaves a claimed key with no effect, permanently swallowing the event.

async function handleEvent(event: { subId: string; sequence: number; delta: Record<string, unknown> }) {
  const version = await db.getSubscriptionVersion(event.subId);
  if (event.sequence <= version) {
    return { action: 'DEDUPLICATE', version };              // ⚠️ stale / replay
  }
  const tx = await db.beginTransaction();
  try {
    await tx.updateSubscription(event.subId, event.delta);
    await tx.insertOutbox(event.subId, 'subscription.updated', event.delta);
    await tx.commit();
    return { action: 'APPLY', version: event.sequence };    // ✅ exactly once
  } catch (err) {
    await tx.rollback();                                    // ✗ key not retained
    throw err;
  }
}

4. Gate concurrent ledger writes

Tax and ledger math must run once per cycle. Use row locking or optimistic concurrency to serialize.

def apply_ledger_entry(account_id: str, amount_cents: int, tax_rate_bps: int, expected_version: int) -> None:
    row = db.query("SELECT balance, version FROM ledger WHERE id = %s FOR UPDATE", account_id)
    if row.version != expected_version:
        raise ValueError("Ledger version mismatch; retry with latest state")  # ⚠️ concurrent write
    # tax in basis points keeps money in integer minor units — never float
    gross = amount_cents + (amount_cents * tax_rate_bps) // 10_000
    db.execute(
        "UPDATE ledger SET balance = %s, version = %s WHERE id = %s AND version = %s",
        row.balance + gross, row.version + 1, account_id, row.version,
    )

Edge Cases & Failure Modes

Performance & Scale

Unique-constraint dedup is O(1) per event and scales to the renewal-day burst as long as the index stays hot — partition webhook_ingest_log by month and drop old partitions per your retention policy. Layer Redis SET NX as a fast pre-check in front of Postgres for high-throughput endpoints, but never as the authority: an eviction during a billing-cycle close must not admit a duplicate, so the database constraint always has the final say. Size key TTLs to exceed the provider’s maximum retry window (commonly 72 hours for Stripe) so a late retry still finds its claim. Keep the locked critical section minimal — claim the key, do the effect, commit — to avoid lock contention during renewal spikes.

Testing Strategy

Write a replay test that submits the same provider_event_id N times concurrently and asserts exactly one ledger entry and one outbox row. Use a mock clock to make TTL-expiry and grace-period boundaries deterministic. Forge a webhook with an invalid signature and assert it is rejected before the idempotency gate is even reached. Inject a sequence of out-of-order events and assert the monotonic version guard discards the stale ones. Finally, simulate a crash between key claim and commit (kill the transaction) and assert the key is not retained, so a retry can still succeed.

Frequently Asked Questions

How do I handle idempotency when the provider does not supply a unique event ID? Build a deterministic key by hashing stable fields — provider timestamp, customer ID, amount in minor units, and currency. Insert that hash into a unique index before processing. Two retries of the same logical event produce the same hash and collide on the constraint, so only the first takes effect.

Does idempotency guarantee exactly-once processing in distributed systems? Idempotency guarantees that repeated identical requests cause the same single state change; it does not stop the network from duplicating deliveries. You reach effective exactly-once by combining at-least-once delivery, a deduplication filter (the key claim), and an outbox so that the publish and the commit are atomic.

How should tax calculation engines interact with idempotent handlers? Make tax math deterministic, or cache the result keyed by the idempotency key. On a retry, return the cached tax figure rather than re-invoking the external tax API — that keeps the amount stable and avoids a second metered API call.

What database isolation level is recommended for ledger updates? Use REPEATABLE READ or SERIALIZABLE to prevent phantom reads during concurrent invoice generation and dunning transitions. Combine it with SELECT ... FOR UPDATE or a version check so conflicting writes fail fast and retry against fresh state.

Related

• Building Idempotent Webhook Handlers in Node.js
• Database Sync & Consistency Patterns
• Outbox Pattern & Event Publishing
• Webhook Retry & Timeout Strategies
• Webhook Processing & Backend State Management