Usage-Based Billing Implementation

Usage-based billing turns a stream of raw telemetry — API calls, gigabytes transferred, seats active, tokens generated — into a single correct number on an invoice. The hard part is not the pricing formula; it is the pipeline underneath it. Events arrive duplicated, out of order, and sometimes hours late, yet the invoice you charge must be exactly right and reproducible during an audit. This guide is part of Subscription Billing Architecture & Pricing Models, and it focuses on the three subsystems that decide whether your metering is trustworthy: idempotent ingestion, windowed aggregation, and late-event reconciliation. If you are weighing a managed meter against your own, start with implementing metered billing with Stripe vs custom; if you need the pipeline to survive real traffic, see designing a high-throughput metering event pipeline.

Prerequisites

• An append-only usage_events table or topic with a per-event natural key
• An idempotency-key store (Redis with SET NX or a Postgres unique constraint)
• A durable queue or log (Kafka, SQS, Redis Streams) buffering raw events before aggregation
• A billing-cycle calendar per subscription (current_period_start / current_period_end in UTC)
• An outbox table written in the same transaction as ledger postings
• A watermark/grace-window policy for events that arrive after their window opens

Architecture & Data Flow

The pipeline separates four concerns: collection (untrusted, high-volume, may duplicate), ingestion (deduplicate and persist the raw event), aggregation (fold events into per-window counters), and rating (turn a closed counter into money on the ledger). Each stage hands the next a stronger guarantee. Collection promises nothing. Ingestion promises every distinct event is stored exactly once. Aggregation promises a counter that only counts each event once, even across replays. Rating promises one ledger entry per window, written atomically with its outbound event.

Inputs are raw events carrying customer_id, a meter name, a quantity, an event-time occurred_at, and a client-generated event_id. Processing deduplicates on event_id, buffers, then aggregates per (customer_id, meter, window). Outputs are one ledger entry per closed window plus a usage.rated event on the outbox for downstream consumers.

Implementation Walkthrough

1. Ingest each event behind an idempotency key

The collector cannot be trusted to send each event once. Retries, at-least-once queues, and client bugs all duplicate. Derive a deterministic key from the natural identity of the event — never from a server timestamp, which changes on replay.

import hashlib
from datetime import timedelta

def ingest_usage_event(event: dict, redis, db) -> dict:
    # Identity is the client's event_id, not when we received it
    idem_key = hashlib.sha256(
        f"usage:{event['customer_id']}:{event['meter']}:{event['event_id']}".encode()
    ).hexdigest()

    # SET NX: claim the key, 7-day window covers any retry storm
    claimed = redis.set(f"idem:{idem_key}", "1", nx=True, ex=int(timedelta(days=7).total_seconds()))
    if not claimed:
        return {"status": "duplicate", "event_id": event["event_id"]}  # ✅ safe replay

    db.execute(
        """INSERT INTO usage_events
             (event_id, customer_id, meter, quantity, occurred_at, received_at)
           VALUES (%(event_id)s, %(customer_id)s, %(meter)s, %(quantity)s,
                   %(occurred_at)s, now())
           ON CONFLICT (event_id) DO NOTHING""",
        event,
    )
    return {"status": "accepted"}

The ON CONFLICT (event_id) DO NOTHING is the durable backstop: even if Redis is flushed, the unique constraint guarantees the raw event lands once. This is the same guarantee an idempotent webhook consumer pattern gives inbound webhooks — applied to the metering boundary instead.

2. Buffer raw events in a durable log

Synchronous aggregation couples ingestion latency to write contention on counter rows. Put a durable queue between them. The ingestion handler returns 202 Accepted the moment the event is persisted; an aggregation consumer drains the log at its own pace and survives a downstream outage by simply lagging.

def publish_for_aggregation(event: dict, producer) -> None:
    # Partition by customer so all of a customer's events land on one consumer,
    # which lets each consumer keep counters in memory with no cross-shard locking
    producer.send(
        topic="usage.raw",
        key=event["customer_id"].encode(),  # ✅ ordering per customer
        value=event,
    )

3. Aggregate into windows with a watermark

A window is a (customer_id, meter, period_start, period_end) bucket aligned to the subscription’s billing cycle. The aggregator upserts a counter per window. The watermark is the policy that decides when a window is final: an event whose occurred_at falls in window W but arrives after W’s watermark has passed is late.

-- Counter is keyed by the window; UPSERT folds each event in once.
-- The aggregator only writes events whose event_id has not been folded yet,
-- enforced by the per-window applied-events guard below.
INSERT INTO usage_counters (customer_id, meter, period_start, period_end, total_quantity)
VALUES (:customer_id, :meter, :period_start, :period_end, :quantity)
ON CONFLICT (customer_id, meter, period_start)
DO UPDATE SET total_quantity = usage_counters.total_quantity + EXCLUDED.total_quantity,
              updated_at = now();

To keep counting exactly once even if the consumer reprocesses the log after a crash, gate each fold on a per-window record of applied event ids:

-- Returns 0 rows if this event was already folded into this window → skip the increment.
INSERT INTO applied_usage_events (customer_id, meter, period_start, event_id)
VALUES (:customer_id, :meter, :period_start, :event_id)
ON CONFLICT DO NOTHING
RETURNING event_id;

4. Rate the closed window into the ledger atomically

When a window’s watermark passes, rate it. The counter, the pricing tier, and the resulting amount become one ledger entry. Write the ledger row and an outbox row in the same transaction so the “usage was rated” event can never be lost or double-published — the outbox pattern makes the commit and the publish atomic.

BEGIN;
INSERT INTO ledger_entries (ledger_entry_id, customer_id, invoice_id, amount_cents, kind, window_key)
VALUES (gen_random_uuid(), :customer_id, :invoice_id, :amount_cents, 'usage', :window_key)
ON CONFLICT (window_key) DO NOTHING;            -- ✅ one entry per window

INSERT INTO outbox (event_type, payload, status)
VALUES ('usage.rated', :payload, 'PENDING');
COMMIT;

5. Reconcile before the invoice finalizes

Before finalizing, re-derive the window total directly from usage_events and compare it against the usage_counters value the ledger used. A mismatch means the aggregator dropped or double-counted; block finalization and alert rather than charge a wrong number.

SELECT c.total_quantity AS counter_total,
       COALESCE(SUM(e.quantity), 0) AS raw_total
FROM usage_counters c
LEFT JOIN usage_events e
  ON e.customer_id = c.customer_id
 AND e.meter = c.meter
 AND e.occurred_at >= c.period_start
 AND e.occurred_at <  c.period_end
WHERE c.customer_id = :customer_id AND c.period_start = :period_start
GROUP BY c.total_quantity;

Edge Cases & Failure Modes

When a subscription changes tier mid-window, the counter must be split at the change instant so each segment is priced against the correct rate — coordinate this with Proration Logic & Calculations so usage caps and baseline fees stay consistent with the audit trail.

Performance & Scale

Ingestion is the hot path: keep it to a Redis SET NX plus an append. Both are O(1) and horizontally scalable. The aggregator is the contended path — partition the durable log by customer_id so each consumer owns a disjoint key range and never contends on another consumer’s counter rows. Index usage_events on (customer_id, meter, occurred_at) so window reconciliation is a range scan, not a full table scan. Batch counter upserts (e.g. fold 500 events, then one UPDATE) to amortize write amplification. At 100k+ subscriptions, run aggregation as a separate fleet from ingestion so a reconciliation backlog never slows event collection. For the deep version of this — exactly-once counting, backpressure, and dedup at sustained throughput — see designing a high-throughput metering event pipeline.

Testing Strategy

Test the pipeline deterministically by controlling the clock and replaying events. Assert that replaying the same event id twice yields one increment. Assert that an out-of-order event lands in the window matching its occurred_at, not its arrival. Inject a late event after the watermark and assert it produces a carry-forward adjustment, not a silent drop or a duplicate. Replay the entire durable log from offset zero and assert the final counters are identical — the system must be idempotent under full replay.

def test_replay_is_idempotent(pipeline, clock):
    event = {"event_id": "evt_1", "customer_id": "cus_42",
             "meter": "api_calls", "quantity": 10, "occurred_at": clock.now()}
    pipeline.ingest(event)
    pipeline.ingest(event)              # duplicate delivery
    pipeline.aggregate_all()
    pipeline.aggregate_all()            # consumer crash + replay
    assert pipeline.counter("cus_42", "api_calls") == 10   # ✅ counted once

Frequently Asked Questions

How long should the late-event grace window be? Set it to the largest realistic clock skew plus collector retry budget — commonly 24 to 72 hours. Events inside the window adjust the open counter; events past it become a carry-forward adjustment on the next invoice. The window must be a deliberate policy, not an accident of when your cron happens to run.

How do you guarantee exactly-once counting when the queue is at-least-once? Counting once does not require delivery once. Deduplicate on a natural event_id at ingestion, and gate each aggregation fold on a per-window applied_usage_events record. Re-delivery then either hits the Redis key, the usage_events unique constraint, or the applied-events guard — three layers that all collapse a duplicate to a no-op.

Should I bucket events by when they happened or when they arrived? By event time (occurred_at). Arrival time is a property of your network, not the customer’s usage, and bucketing by it produces wrong totals the moment delivery is delayed. Reserve received_at for observability and late-event detection only.

Do I need Kafka, or is Postgres enough? For tens of thousands of events per day, a Postgres usage_events table polled by a worker is plenty and far simpler to operate. Reach for a partitioned log like Kafka when sustained ingestion exceeds what a single Postgres writer comfortably handles, or when multiple independent consumers need the same event stream.

Related

• Implementing Metered Billing with Stripe vs Custom
• Designing a High-Throughput Metering Event Pipeline
• Proration Logic & Calculations
• Building Idempotent Webhook Handlers in Node.js
• Using the Outbox Pattern for Reliable Billing Events
• Subscription Billing Architecture & Pricing Models