Webhook Processing & Backend State Management

A subscription billing system is, at its core, a distributed state machine driven by events it does not control. Stripe, Paddle, Adyen and your bank all emit asynchronous webhooks — invoice.paid, customer.subscription.updated, charge.dispute.created — and your backend must turn that unreliable, out-of-order, frequently-duplicated stream into a ledger accurate to the cent. The hard part is not parsing the JSON. The hard part is that webhooks are delivered at least once, arrive out of order, can be replayed by an attacker, and may land while another node is mutating the same subscription. Get any of these wrong and you get phantom renewals, double charges, suspended-but-paying customers, and an audit trail that does not reconcile. This guide covers the four subsystems that make webhook processing survivable at scale: keeping state consistent across services through Database Sync & Consistency Patterns, guaranteeing exactly-once effects with Idempotency & Event Deduplication, surviving provider outages with sound Webhook Retry & Timeout Strategies, and publishing your own internal events atomically with the outbox pattern for billing events.

Architecture Overview

Every inbound event traverses the same pipeline before it is allowed to touch money. Signature verification rejects forgeries and replays at the edge. An idempotency gate collapses duplicate deliveries to a single effect. A finite state machine validates that the transition is legal. Only then does the handler write domain state and an outbox row in one transaction; a poller drains the outbox to downstream consumers and the double-entry ledger.

Webhook processing pipeline Provider webhooks pass through signature verification, an idempotency gate, a state machine, and a transactional outbox before reaching the ledger and downstream consumers. Payment provider Signature verify (HMAC) Idempotency gate State machine (FSM guard) DB + outbox (one tx) Ledger posting Consumers + bus
Signed events are de-duplicated and validated before a single transaction writes both domain state and the outbox row that feeds the ledger.

The components and their responsibilities:

Component Responsibility Source of truth
Edge verifier HMAC-SHA256 signature check, timestamp-drift rejection (replay defence) Provider signing secret
Idempotency store Collapse duplicate deliveries of the same provider_event_id to one effect Postgres unique index / Redis SET NX
Subscription FSM Reject illegal transitions (canceled → active); enforce monotonic versioning subscriptions.version
Transactional outbox Atomically persist domain change + the event to publish billing_outbox table
Double-entry ledger Post balanced debit/credit lines per financial mutation ledger_lines table
Reconciliation jobs Diff local state against provider settlement reports nightly Provider API + ledger

Core Data Model

The schema separates three concerns: the raw event log (audit + dedup), the subscription state itself (versioned for concurrency control), and the outbox that decouples your internal event stream from the inbound webhook.

-- Inbound event log: deduplication AND audit trail in one table.
CREATE TABLE webhook_events (
  event_id            UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  provider_event_id   VARCHAR(255) NOT NULL,
  tenant_id           UUID         NOT NULL,
  event_type          VARCHAR(64)  NOT NULL,
  payload_hash        BYTEA        NOT NULL,            -- sha256 of raw body
  status              VARCHAR(16)  NOT NULL DEFAULT 'received',
  received_at         TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at        TIMESTAMPTZ,
  -- the dedup guarantee: one effect per provider event per tenant
  CONSTRAINT uq_provider_event UNIQUE (tenant_id, provider_event_id)
);

-- Versioned subscription state. version drives optimistic concurrency.
CREATE TABLE subscriptions (
  subscription_id     UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  customer_id         UUID         NOT NULL,
  price_id            VARCHAR(64)  NOT NULL,
  state               VARCHAR(24)  NOT NULL,            -- trialing|active|past_due|canceled|unpaid
  current_period_end  TIMESTAMPTZ  NOT NULL,
  version             BIGINT       NOT NULL DEFAULT 0,  -- monotonic, set from event sequence
  updated_at          TIMESTAMPTZ  NOT NULL DEFAULT now()
);

-- Transactional outbox: written in the SAME tx as the subscription mutation.
CREATE TABLE billing_outbox (
  id                  UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id        UUID         NOT NULL,            -- subscription_id
  event_type          VARCHAR(64)  NOT NULL,
  payload             JSONB        NOT NULL,
  status              VARCHAR(16)  NOT NULL DEFAULT 'pending',
  retry_count         INT          NOT NULL DEFAULT 0,
  created_at          TIMESTAMPTZ  NOT NULL DEFAULT now(),
  published_at        TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending ON billing_outbox (created_at)
  WHERE status = 'pending';

The version column is the linchpin of correctness: it is set from the provider’s event sequence (or a Lamport-style counter) so that a late-arriving older event can be detected and discarded rather than clobbering newer state. Money is never stored as a float — ledger amounts live as BIGINT minor units (cents) elsewhere in the model.

Key Design Patterns

Idempotency keys

Every effect is gated on a unique key derived from the provider event. The insert is the lock: if two concurrent deliveries race, exactly one wins the unique constraint and the other short-circuits to 200 OK. Use this whenever a provider may deliver the same logical event more than once — which is always.

INSERT INTO webhook_events (provider_event_id, tenant_id, event_type, payload_hash)
VALUES ($1, $2, $3, $4)
ON CONFLICT (tenant_id, provider_event_id) DO NOTHING
RETURNING event_id;   -- zero rows returned ⇒ duplicate, ack and stop

Transactional outbox

Writing domain state and also calling the message broker over the network in one operation is not atomic — a crash between them loses or duplicates events. The outbox sidesteps this: both writes hit Postgres in one transaction, and a separate poller publishes. Use it whenever a state change must reliably emit an event. See Outbox Pattern & Event Publishing for the full treatment.

with db.transaction():                       # ✅ both writes commit or neither does
    db.execute("UPDATE subscriptions SET state=%s, version=%s WHERE subscription_id=%s",
               new_state, event_version, sub_id)
    db.execute("INSERT INTO billing_outbox (aggregate_id, event_type, payload) "
               "VALUES (%s, %s, %s)", sub_id, "subscription.activated", payload)
# a poller drains billing_outbox → broker, at-least-once, idempotent consumers

Subscription FSM

A finite state machine rejects illegal jumps and enforces monotonic versioning so out-of-order delivery cannot corrupt state. Use it for any lifecycle with forbidden transitions.

ALLOWED = {
    "trialing": {"active", "canceled"},
    "active":   {"past_due", "canceled"},
    "past_due": {"active", "unpaid", "canceled"},
}

def transition(record, target_state, event_version):
    if event_version <= record.version:          # ⚠️ stale/out-of-order event
        return "IGNORED_OUT_OF_ORDER"
    if target_state not in ALLOWED.get(record.state, set()):
        raise ValueError(f"{record.state} -> {target_state} blocked")   # ✗ illegal jump
    record.state, record.version = target_state, event_version
    return "TRANSITION_APPLIED"

Dead-letter routing

Events that fail validation repeatedly must not poison the queue. Route them to a dead-letter queue (DLQ) after N attempts for manual inspection, keeping the main pipeline flowing. Use it for malformed payloads and unrecognised event types.

Compliance & Regulatory Boundaries

Webhook payloads routinely carry PII and transaction metadata, so the ingestion layer is squarely in scope for PCI-DSS and GDPR. Never persist raw PANs or CVVs — strip them at the edge before anything is logged or stored. Apply field-level encryption (AES-256-GCM) to customer email and address fields before insertion, with keys managed by a KMS. For GDPR erasure requests that collide with financial retention obligations, pseudonymise the PII while preserving the immutable ledger and event log under a legal-hold flag; the audit trail must remain reconstructable for PCI-DSS and SOC 2. Where webhooks carry tax-relevant amounts, the VAT/GST figures you persist become part of your regulatory reporting record, so they must be reconcilable against the double-entry ledger posting and never silently mutated.

Scalability & Failure Modes

At 10k subscriptions a single Postgres primary with a unique-constrained event log comfortably absorbs the renewal-day burst. At 100k, the failure modes shift. Renewal cycles cluster events at period boundaries, so the dedup table and outbox poller become the bottleneck — partition webhook_events by month and keep the partial index on status = 'pending' so the poller’s working set stays small. Cascade failures are the real danger: if the broker is down and you publish synchronously inside the webhook transaction, every webhook blocks, connection pools exhaust, and the provider’s retries amplify the load (a retry storm). The outbox breaks that cascade by making the broker an asynchronous concern. Wrap every provider API call (used during reconciliation) in a circuit breaker so a provider 5xx does not propagate; trip open after a threshold, serve a pending_verification fallback state, and probe half-open on a timer. Honour Retry-After headers and apply exponential backoff so you never become the cause of the provider’s rate limiting.

Operational Runbook

Instrument the pipeline so drift is caught before it reaches a financial statement.

  • Monitoring signals: outbox lag (count of pending rows older than 60s), webhook p99 processing latency, idempotency hit-rate (sudden spikes signal a provider replay), DLQ depth, and circuit-breaker state transitions.
  • Alert thresholds: page when outbox lag exceeds 500 rows or 5 minutes; warn when DLQ depth is non-zero; page when reconciliation variance exceeds one cent per 10k subscriptions.
  • Reconciliation jobs: a nightly job diffs provider settlement reports against ledger_lines, and a 15-minute job re-syncs any subscription whose last_synced_at is older than two hours. Both write provider-truth overrides to an immutable audit table.
  • Stuck-state sweep: a periodic job clears webhook_events left in processing past the maximum retry window, which otherwise leak idempotency keys.

Frequently Asked Questions

How do we prevent duplicate charges when a provider sends the same webhook multiple times? Gate every effect on a unique index over (tenant_id, provider_event_id) and perform the insert with ON CONFLICT DO NOTHING before any business logic. The first delivery wins the constraint and proceeds; later deliveries return zero rows, so the handler acknowledges with 200 OK and stops. This pushes deduplication into the storage layer where it is atomic.

Should I publish internal events directly from the webhook handler or use an outbox? Use an outbox. Publishing to a broker over the network inside the same operation as your database write is not atomic — a crash between the two loses or duplicates events. Writing the event to an outbox table in the same transaction, then draining it with a poller, gives you at-least-once delivery that pairs cleanly with idempotent consumers for exactly-once effects.

How should the subscription state machine handle out-of-order event delivery? Make transitions monotonic and idempotent. Stamp each subscription with a version derived from the provider event sequence, and discard any event whose version is less than or equal to the current one. Combine that with explicit transition guards so an early canceled cannot be overwritten by a late active.

What is the right architecture for surviving a provider outage? A durable queue (SQS, Kafka, or RabbitMQ) with dead-letter routing, exponential backoff, and a circuit breaker around the provider’s API. The outbox keeps your internal state consistent while the provider recovers, and the reconciliation job repairs any drift once the provider is healthy again.

What compliance considerations apply to storing webhook payloads? Never store raw PANs or CVVs; strip them at ingestion. Encrypt PII fields at rest, keep an immutable audit log of every state mutation for PCI-DSS and SOC 2, and resolve GDPR erasure-versus-retention conflicts by pseudonymising PII while preserving the financial record under legal hold.