Designing a High-Throughput Metering Event Pipeline

You design this pipeline the day a single Postgres writer stops keeping up with your meter events — usually somewhere past a few thousand events per second, or when one noisy tenant can stall everyone else’s billing. At that point ingestion, aggregation, and rating can no longer share a thread, and “just insert the row” stops being a correctness strategy. This page builds the scaled version of the system described in Usage-Based Billing Implementation: how to ingest events idempotently at volume, buffer them in a log, count them exactly once under replay, and absorb bursts without dropping a single billable event.

Trade-offs

The core architectural decision is what sits between ingestion and aggregation. Each option trades operational weight against the guarantees and throughput you get.

Kafka (or any partitioned log) is the default for genuinely high throughput because partitioning by customer_id gives you per-customer ordering and lets each consumer own counters with zero cross-shard locking. The rest of this page assumes a partitioned log, but the patterns transfer to Redis Streams and SQS.

Step-by-Step Implementation

1. Deduplicate at ingestion

The ingestion endpoint must be cheap — a deduplication check and an append, nothing more. Heavy work here caps your throughput. Key on the event’s natural identity; this mirrors the idempotent webhook consumer pattern applied to the metering boundary.

import hashlib

def ingest(event: dict, redis, producer) -> dict:
    idem_key = hashlib.sha256(
        f"{event['customer_id']}:{event['meter']}:{event['event_id']}".encode()
    ).hexdigest()

    # Claim the key; 7-day TTL covers any producer retry window
    if not redis.set(f"idem:{idem_key}", "1", nx=True, ex=604800):
        return {"status": "duplicate"}                       # ✅ collapse retry

    producer.send(
        topic="usage.raw",
        key=event["customer_id"].encode(),                   # ✅ per-customer ordering
        value=event,
    )
    return {"status": "accepted"}                            # 202, fast path

Redis is the fast first line; it is not the source of truth. A flushed cache must not let duplicates through, which is what the next stage’s durable guard handles.

2. Append to a partitioned durable log

Partitioning by customer_id is the load-bearing decision. All of one customer’s events land on one partition, so a single consumer sees them in order and can hold that customer’s counters locally. Add partitions to scale; rebalancing moves whole customers, never splitting a counter across consumers.

3. Fold into windows with an exactly-once guard

Each consumer reads its partitions and folds events into per-window counters. Because the log is at-least-once and crashes cause replay, the increment itself must be idempotent. Gate every fold on a durable per-window record of applied event ids — the increment runs only when the guard row is freshly inserted.

-- Atomic: claim the event for this window, and only then increment.
WITH claim AS (
  INSERT INTO applied_usage_events (customer_id, meter, period_start, event_id)
  VALUES (:customer_id, :meter, :period_start, :event_id)
  ON CONFLICT DO NOTHING
  RETURNING event_id
)
INSERT INTO usage_counters (customer_id, meter, period_start, period_end, total_quantity)
SELECT :customer_id, :meter, :period_start, :period_end, :quantity
FROM claim                                   -- ✗ no row if already applied → no double count
ON CONFLICT (customer_id, meter, period_start)
DO UPDATE SET total_quantity = usage_counters.total_quantity + EXCLUDED.total_quantity;

Counters are commutative sums, so out-of-order arrival within a window is harmless — a + b == b + a. Ordering only matters for the watermark, not the math. The consumer commits its log offset only after the fold transaction commits, so a crash replays from the last durable point and the guard makes the replay a no-op.

4. Advance the watermark and route late events

A window closes when the watermark — min(event_time) still in flight, minus a grace allowance — passes its period_end. Events arriving after their window’s watermark are late. Do not drop them and do not retroactively mutate a rated window; route them to a carry-forward adjustment on the next open invoice.

def handle_event(event, window_for, watermark, carry_forward):
    win = window_for(event["customer_id"], event["occurred_at"])
    if event["occurred_at"] < watermark(win):
        carry_forward(event)          # ⚠️ late: adjust next invoice, never the closed one
    else:
        fold(event, win)              # ✅ on-time: count into the open window

5. Publish the rated result through an outbox

When a window is rated, write the ledger entry and the outbound event in one transaction so a consumer crash can never publish twice or lose the event — the outbox pattern keeps the commit and the publish atomic.

BEGIN;
INSERT INTO ledger_entries (ledger_entry_id, customer_id, invoice_id, amount_cents, kind, window_key)
VALUES (gen_random_uuid(), :customer_id, :invoice_id, :amount_cents, 'usage', :window_key)
ON CONFLICT (window_key) DO NOTHING;
INSERT INTO outbox (event_type, payload, status)
VALUES ('usage.rated', :payload, 'PENDING');
COMMIT;

6. Apply backpressure instead of dropping

When ingestion outruns aggregation, the correct response is to lag, not to shed. The durable log is the shock absorber: ingestion keeps appending while consumers fall behind, and consumer lag becomes your alerting signal. If producers outrun the log itself, return 429 from ingestion so clients back off — never drop a billable event silently.

Verification & Testing

Drive the pipeline with a deterministic event generator and a controllable clock. Assert that replaying the entire log from offset zero yields byte-identical counters — full-replay idempotency is the headline guarantee. Kill a consumer mid-batch and assert no double counting after it resumes from its last committed offset. Inject duplicates, out-of-order events, and post-watermark late events in one run and assert: duplicates collapse, out-of-order events land in the right window, late events become carry-forward rows. Load-test ingestion to your target rate and assert p99 latency stays flat while consumer lag (not error rate) rises — proving backpressure works.

def test_full_replay_is_deterministic(pipeline):
    events = generate_events(n=100_000, dup_rate=0.1, out_of_order_rate=0.2)
    for e in events:
        pipeline.ingest(e)
    pipeline.consume_all()
    snapshot = pipeline.counter_snapshot()

    pipeline.reset_consumer_offsets()      # simulate full replay
    pipeline.consume_all()
    assert pipeline.counter_snapshot() == snapshot   # ✅ exactly-once under replay

Gotchas & Production Pitfalls

• Partitioning by event id instead of customer. You lose per-customer ordering and scatter a customer’s counter across consumers, forcing distributed locks. Partition by customer_id.
• Committing the log offset before the fold transaction. A crash in that gap loses the event entirely. Always commit the offset after the DB transaction succeeds, and lean on the applied-events guard to make the inevitable replay safe.
• Treating Redis as the dedup source of truth. A cache flush then admits duplicates. Redis is the fast path; the durable per-window guard (or a unique constraint on raw events) is the real exactly-once boundary.
• Mutating a rated window when a late event arrives. It corrupts an already-invoiced number and breaks the audit trail. Route late events to carry-forward only.
• Dropping events under load. Shedding billable events is silent revenue loss. Absorb bursts in the log and signal with 429 and consumer-lag alerts instead.
• Unbounded applied_usage_events growth. It grows with every event forever. Partition it by period_start and drop partitions once their windows are rated and beyond any dispute horizon.

Related

• Usage-Based Billing Implementation
• Implementing Metered Billing with Stripe vs Custom
• Building Idempotent Webhook Handlers in Node.js
• Using the Outbox Pattern for Reliable Billing Events
• Resolving Out-of-Order Webhook Delivery Issues
• Subscription Billing Architecture & Pricing Models