Resolving Out-of-Order Webhook Delivery Issues

You hit this problem the first time an invoice.payment_succeeded lands before the invoice.created that should have preceded it, and your state machine either rejects a payment for an invoice it has never heard of or silently creates a phantom one. Asynchronous payment providers make no ordering guarantee across HTTP deliveries, so any handler that treats receipt order as truth will eventually corrupt entitlements or double-charge. This page is the sequencing companion to Webhook Retry & Timeout Strategies: retries solve duplication, but they do nothing for ordering. The fix is to decouple ingestion from mutation and apply events in provider-defined order, not arrival order.

Trade-offs

Approach Ordering guarantee Added latency Storage cost Complexity Best fit
Trust arrival order None 0 ms None Trivial Never, for money
created_at timestamp sort Weak (clock skew) ~1–5 s buffer Low Low Single provider, coarse ordering
Provider sequence id + buffer Strong ~2–5 s buffer Medium (sorted set) Medium Most billing systems
Version-guarded upsert only Strong on state, lossy on intermediate events 0 ms Low Low Idempotent end-state events
Full event-sourced log Total order replay Higher (replay) High High Audit-critical ledgers

For most SaaS billing, the pragmatic answer is a per-subscription buffer keyed on a monotonic provider sequence, backed by a version-guarded upsert so anything that slips the buffer still cannot regress state. Wall-clock created_at alone is a trap: providers run multiple emitters and clocks drift, so timestamps regress.

Step-by-Step Implementation

1. Diagnose sequence drift before you build

Extract the provider’s sequence id or created_at from the raw payload — never your server’s receipt time. Log every event where created_at < last_processed_created_at and emit it as a sequence_gap metric. This separates true ordering violations from ordinary network jitter and from retries (which reuse the same created_at).

def classify(event: dict, last_processed_created_at: float) -> str:
    if event["created_at"] == last_processed_created_at:
        return "retry"            # same event, dedupe handles it
    if event["created_at"] < last_processed_created_at:
        return "out_of_order"     # ✗ genuine sequencing violation
    return "in_order"             # ✅ normal forward progress

2. Acknowledge fast, then buffer per subscription

Return 200 immediately so the provider does not retry, then park the payload in a Redis sorted set scored by sequence. This gives O(log N) insert and ordered reads, and isolates one subscription’s drift from every other.

import redis, json

r = redis.Redis()
MAX_BACKLOG = 50

def buffer(subscription_id: str, payload: dict, sequence: int) -> None:
    key = f"webhook_buffer:{subscription_id}"
    lua = """
    redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])
    if redis.call('ZCARD', KEYS[1]) > tonumber(ARGV[3]) then
      return -1
    end
    return 1
    """
    if r.eval(lua, 1, key, sequence, json.dumps(payload), MAX_BACKLOG) == -1:
        route_to_dlq(subscription_id, payload, "backlog_overflow")  # ⚠️ reconcile

3. Release only contiguous sequences

A worker pops the lowest-scored event and releases it only when it is exactly one past the last processed sequence. Anything ahead waits for its predecessor; a TTL converts an indefinite stall into a reconciliation trigger.

def drain(subscription_id: str) -> None:
    key = f"webhook_buffer:{subscription_id}"
    while True:
        head = r.zrange(key, 0, 0, withscores=True)
        if not head:
            return
        payload_raw, sequence = head[0][0], int(head[0][1])
        last = get_last_sequence(subscription_id)
        if sequence == last + 1:
            apply_event(json.loads(payload_raw))   # ✅ in order
            r.zrem(key, payload_raw)
        elif sequence <= last:
            r.zrem(key, payload_raw)               # already applied, drop
        else:
            return                                 # ⚠️ gap, wait or reconcile

4. Upsert with a version guard

Even with a buffer, defend the storage layer. A composite unique constraint blocks duplicates and the WHERE sequence_number < EXCLUDED.sequence_number clause rejects any stale write that bypassed the buffer.

INSERT INTO subscription_ledger (
  ledger_entry_id, subscription_id, provider_event_id,
  event_type, sequence_number, amount_cents, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, now())
ON CONFLICT (provider_event_id) DO UPDATE
   SET sequence_number = EXCLUDED.sequence_number,
       amount_cents    = EXCLUDED.amount_cents,
       updated_at      = now()
 WHERE subscription_ledger.sequence_number < EXCLUDED.sequence_number;

Verification & Testing

Assert the core invariant directly: feed invoice.payment_succeeded (sequence 2) before invoice.created (sequence 1) and confirm the payment is held until creation applies, then both land in order. Replay any event twice and assert one ledger row. Push a stale sequence_number through the upsert and assert the row is unchanged. Run a daily reconciliation query that diffs internal state against the provider’s authoritative API.

-- Reconciliation: events we hold that the provider considers superseded
SELECT l.subscription_id, l.provider_event_id, l.sequence_number
  FROM subscription_ledger l
 WHERE l.status <> 'reconciled'
   AND l.updated_at < now() - interval '1 hour'
 ORDER BY l.subscription_id, l.sequence_number;

Integration tests should shuffle a known event stream, inject random latency and duplicates, and assert the final ledger state is identical to the in-order replay.

Gotchas & Production Pitfalls

  • Clock skew makes created_at regress. Two emitters with drifting clocks produce timestamps that go backwards. Prefer a monotonic provider sequence id; fall back to created_at only when no sequence exists.
  • Buffers stall forever on a lost predecessor. If event 2 never arrives, event 3 waits indefinitely. A TTL must escalate to provider-API reconciliation, not block the pipeline.
  • Backlog overflow hides a deeper outage. A buffer growing past its cap usually means the provider dropped an event, not that traffic spiked. Dead-letter and reconcile rather than raising the cap.
  • Version guards silently swallow late events. A rejected stale upsert is correct but invisible. Log and count rejections, or a real bug looks identical to normal late-arrival handling.
  • Multiple gateways share no sequence space. If two providers are active, normalize into one internal schema with namespaced sequence ids before sequencing — never compare sequences across providers.