Outbox Pattern & Event Publishing

The hardest reliability bug in billing is the dual write: your service updates a subscription in Postgres and then publishes an invoice.paid event to a broker, and the two operations are not atomic. A crash between them either loses the event (state advanced, nobody downstream knows) or, with naive retries, publishes it twice against a committed-or-not state. The transactional outbox closes that gap by making the event publish part of the same transaction as the state change. This page sits under Webhook Processing & Backend State Management and shows the table, the relay worker, and the delivery semantics that turn at-least-once into effectively-once. It pairs naturally with Idempotency & Event Deduplication on the consumer side.

Prerequisites

Architecture & Data Flow

The pattern has four moving parts. A business transaction writes the domain row and an outbox row atomically. A relay worker polls pending outbox rows, publishes each to the broker, and marks it sent only after acknowledgment. Because the write is atomic and the publish is retried until acknowledged, every committed change is delivered at least once; an idempotent consumer collapses any duplicate into effectively-once.

Transactional outbox flow A business transaction writes a domain row and an outbox row atomically; a relay polls pending rows, publishes to the broker, then marks them sent. Business txn domain row + outbox row event_outbox status = pending Relay worker (poller) Message broker Idempotent consumer publish mark sent
One atomic write produces the outbox row; the relay publishes and acknowledges, and the idempotent consumer collapses duplicates into effectively-once.

Inputs are domain commands (charge a card, close an invoice). Processing is the atomic write plus the relay loop. Outputs are durable broker messages that downstream services and a database sync layer consume to keep their own state aligned.

Implementation Walkthrough

1. Define the outbox table

Keep the row self-describing: a stable event_id for dedup, the aggregate it belongs to (for partition/ordering), the serialized payload, and a status lifecycle. Index the relay’s scan path.

CREATE TABLE event_outbox (
  outbox_id     UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  event_id      UUID NOT NULL UNIQUE,        -- stable id consumers dedup on
  aggregate_id  UUID NOT NULL,               -- e.g. subscription_id, for ordering
  event_type    TEXT NOT NULL,               -- 'invoice.paid', 'payment.failed'
  payload       JSONB NOT NULL,
  status        TEXT NOT NULL DEFAULT 'pending',  -- pending | sent | failed
  attempts      INT  NOT NULL DEFAULT 0,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  sent_at       TIMESTAMPTZ
);

CREATE INDEX idx_outbox_pending
  ON event_outbox (created_at)
  WHERE status = 'pending';

2. Write domain state and the outbox row atomically

This single transaction is the whole point: if it commits, the event will be delivered; if it rolls back, neither the state change nor the event exists.

BEGIN;
  UPDATE invoices
     SET status = 'paid', paid_at = now()
   WHERE invoice_id = $1;

  INSERT INTO event_outbox (event_id, aggregate_id, event_type, payload)
  VALUES (
    $2,                       -- event_id (deterministic where possible)
    $3,                       -- subscription_id as aggregate_id
    'invoice.paid',
    jsonb_build_object('invoice_id', $1, 'amount_cents', $4)
  );
COMMIT;  -- ✅ state and event are now inseparable

3. Run the relay loop

Poll pending rows with FOR UPDATE SKIP LOCKED so multiple relay instances never grab the same row. Publish, then mark sent only on broker acknowledgment. Order by created_at and, where ordering matters, drain per aggregate_id.

def relay_tick(db, broker) -> None:
    with db.transaction() as txn:
        rows = txn.fetch("""
            SELECT outbox_id, event_id, aggregate_id, event_type, payload
              FROM event_outbox
             WHERE status = 'pending'
             ORDER BY created_at
             LIMIT 100
               FOR UPDATE SKIP LOCKED
        """)
        for row in rows:
            try:
                broker.publish(
                    topic=row["event_type"],
                    key=row["aggregate_id"],      # partition by aggregate -> per-key order
                    value=row["payload"],
                    headers={"event_id": row["event_id"]},
                )
                txn.execute(
                    "UPDATE event_outbox SET status='sent', sent_at=now() "
                    "WHERE outbox_id=$1", row["outbox_id"])  # ✅ delivered
            except BrokerError:
                txn.execute(
                    "UPDATE event_outbox SET attempts = attempts + 1 "
                    "WHERE outbox_id=$1", row["outbox_id"])   # ⚠️ retry next tick

4. Deduplicate on the consumer

At-least-once means duplicates are guaranteed, not hypothetical. The consumer records each event_id in an idempotency store inside the same transaction as its effect, achieving effectively-once.

def consume(event_id: str, payload: dict, db) -> None:
    with db.transaction() as txn:
        if not txn.execute(
            "INSERT INTO consumed_events (event_id) VALUES ($1) "
            "ON CONFLICT DO NOTHING RETURNING event_id", event_id):
            return  # ✅ already processed, skip
        apply_effect(payload, txn)  # state change + dedup commit together

Edge Cases & Failure Modes

Failure scenario Symptom Mitigation
Relay crashes after publish, before mark-sent Event republished next tick Idempotent consumer collapses the duplicate
Two relay instances race Same row published twice FOR UPDATE SKIP LOCKED serializes row ownership
Outbox grows unbounded Table bloat, slow scans Archive/delete sent rows on a retention job; partition by month
Broker down for minutes Pending rows pile up Bounded retry with attempts; alert on pending-age, not row count alone
Poison event (always fails) One row blocks throughput After max attempts, set status='failed' and route to operators
Per-key ordering broken Consumer sees events out of order Partition broker by aggregate_id; drain that key serially

Performance & Scale

Polling is the default and it scales further than people expect: a partial index on WHERE status='pending' keeps the relay scan proportional to the backlog, not the table. Batch 100–500 rows per tick and tune the interval (50–500 ms) against latency budget. Delete or archive sent rows aggressively — an outbox is a transient queue, not a log of record. When polling latency or DB load becomes the bottleneck, switch the relay to change data capture (CDC) reading the Postgres WAL via Debezium: it eliminates the poll, captures inserts in near-real-time, and removes the mark-sent write. CDC trades operational complexity (connector, WAL slots, schema registry) for lower latency and load; polling wins on simplicity until you are publishing thousands of events per second.

Testing Strategy

Drive the failure paths deterministically. Kill the relay between publish and the mark-sent update, restart it, and assert the consumer applied the effect exactly once. Run two relay instances against a seeded outbox and assert no event publishes twice. Inject a broker outage and assert pending rows accumulate, then drain cleanly on recovery with attempts reflecting the retries. Replay the same event_id at the consumer twice and assert one effect. Use a mock clock so retention and retry-age assertions need no real waiting.

def test_crash_after_publish_is_effectively_once(relay, broker, consumer):
    seed_outbox(event_id="evt_invoice_paid_42")
    relay.publish_then_crash()       # broker has it, row still 'pending'
    relay.restart().run_tick()       # republishes the same event_id
    consumer.drain(broker)
    assert consumer.effects("evt_invoice_paid_42") == 1  # ✅ exactly once

Frequently Asked Questions

How do I guarantee exactly-once processing across retries? You cannot guarantee exactly-once delivery, but you can get effectively-once. The outbox gives at-least-once (the atomic write plus retried publish), and an idempotent consumer keyed on event_id collapses every duplicate. The combination is indistinguishable from exactly-once at the effect level.

Should I poll the outbox or use CDC? Start with polling — a partial index plus SKIP LOCKED carries most workloads to thousands of events per second with trivial ops. Move to CDC (Debezium reading the WAL) when poll latency or database load becomes the constraint, accepting the added operational surface.

Does the outbox preserve event order? Globally, no — created_at ordering is best-effort. Per aggregate, yes: partition the broker by aggregate_id and drain each key serially so a subscription’s events stay in order even when the global stream interleaves.

How big can the outbox grow before it hurts? Throughput is bounded by the relay, not the table, as long as you keep a partial index on pending rows and archive sent rows. Treat the outbox as a queue: a retention job that deletes or partitions out delivered rows keeps scans flat.