Syncing Subscription Status Across Microservices

You hit this problem the moment subscription status lives in more than one place. The billing service marks a subscription past_due from an invoice.payment_failed webhook, but the entitlements service still shows active, so a delinquent customer keeps premium access — or worse, a paying customer is locked out because a canceled event was processed before the invoice.paid that should have preceded it. Syncing subscription status across microservices is fundamentally about turning an unreliable, out-of-order provider event stream into convergent local state without double-applying effects. This is the applied form of the Database Sync & Consistency Patterns that govern the wider pipeline, and it leans on the outbox pattern for billing events plus an idempotent webhook consumer pattern to stay correct under retries.

Trade-offs

The propagation mechanism you pick determines latency, operational burden, and how cleanly you can reason about ordering. Real values below assume a Postgres-backed billing service at ~50k active subscriptions.

The pragmatic default is outbox + poller as the primary path with periodic reconciliation as the safety net. CDC is worth its operational weight only once you have many independent consumers.

Step-by-Step Implementation

1. Capture the change and an outbox row atomically

The provider event mutates local state and enqueues an internal event in the same transaction, so a downstream crash can never lose the event.

CREATE TABLE billing_outbox (
  id                UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id      UUID         NOT NULL,           -- subscription_id
  provider_event_id VARCHAR(255) UNIQUE NOT NULL,
  event_type        VARCHAR(64)  NOT NULL,
  payload           JSONB        NOT NULL,
  status            VARCHAR(16)  NOT NULL DEFAULT 'pending',
  retry_count       INT          NOT NULL DEFAULT 0,
  created_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),
  processed_at      TIMESTAMPTZ
);
CREATE INDEX idx_outbox_pending ON billing_outbox (created_at) WHERE status = 'pending';

2. Drain the outbox to the bus

A poller claims a batch with FOR UPDATE SKIP LOCKED so multiple workers never publish the same row, dispatches to the broker, then marks rows published.

async function drainOutbox(db, bus) {
  await db.transaction(async (trx) => {
    const { rows } = await trx.query(
      `SELECT id, aggregate_id, event_type, payload
         FROM billing_outbox
        WHERE status = 'pending'
        ORDER BY created_at
        LIMIT 100
        FOR UPDATE SKIP LOCKED`            // ✅ no two workers grab the same row
    );
    for (const row of rows) {
      await bus.publish(row.event_type, { subscriptionId: row.aggregate_id, ...row.payload });
      await trx.query(
        `UPDATE billing_outbox SET status = 'published', processed_at = now() WHERE id = $1`,
        [row.id]
      );
    }
  });
}

3. Consume idempotently with a state-machine guard

Each consuming service deduplicates on provider_event_id and rejects illegal transitions, so a replayed or out-of-order event is a no-op rather than corruption.

const ALLOWED = {
  trialing: ['active', 'canceled'],
  active:   ['past_due', 'canceled'],
  past_due: ['active', 'unpaid', 'canceled'],
};

async function consume(evt, db) {
  const dup = await db.query('SELECT 1 FROM processed_events WHERE event_id = $1', [evt.providerEventId]);
  if (dup.rowCount > 0) return { status: 'duplicate' };          // ⚠️ already applied

  const sub = await db.query('SELECT state FROM local_subscriptions WHERE subscription_id = $1', [evt.subscriptionId]);
  if (!(ALLOWED[sub.rows[0].state] || []).includes(evt.targetState)) {
    return { status: 'illegal_transition' };                     // ✗ blocked
  }
  await db.transaction(async (trx) => {
    await trx.query('INSERT INTO processed_events (event_id) VALUES ($1)', [evt.providerEventId]);
    await trx.query('UPDATE local_subscriptions SET state = $1 WHERE subscription_id = $2',
      [evt.targetState, evt.subscriptionId]);
  });
  return { status: 'applied' };
}

4. Reconcile against the provider

A scheduled job re-fetches provider subscriptions and overrides local billing fields, catching anything the event stream dropped. Defer to provider truth for status and cancel_at_period_end; keep local metadata as-is.

def reconcile(tenant_shard: int) -> None:
    stale = db.query(
        "SELECT subscription_id FROM local_subscriptions "
        "WHERE last_synced_at < now() - INTERVAL '2 hours' AND tenant_shard = %s",
        tenant_shard,
    )
    for sub_id in stale:
        remote = provider.get_subscription(sub_id)        # honours Retry-After + circuit breaker
        if remote.status != db.local_status(sub_id):
            db.apply_override(sub_id, remote.status, reconciliation_job_id=JOB_ID)  # ✅ audited

Verification & Testing

Assert convergence under chaos: feed the consumer a scrambled sequence (canceled before past_due before active) and assert the final state equals the highest-sequence event, regardless of arrival order. Replay one provider_event_id twice and assert exactly one row in processed_events and one state change. Run this reconciliation query in staging to prove zero drift after a sync:

SELECT l.subscription_id, l.state AS local, p.status AS provider
FROM   local_subscriptions l
JOIN   provider_snapshot   p USING (subscription_id)
WHERE  l.state <> p.status;   -- expect zero rows post-reconciliation

Integration-test the poller by killing the broker mid-drain and asserting that pending rows are re-claimed and published exactly once on recovery (no duplicates downstream because consumers are idempotent).

Gotchas & Production Pitfalls

• Grace-period drift: delayed webhook delivery can leave a subscription past_due during a legitimate grace window. Enforce a configurable grace period (e.g. 72 hours) before any service suspends access, rather than reacting to the first past_due.
• Double suspension from duplicate failures: repeated invoice.payment_failed events can suspend twice and fire two dunning emails. Deduplicate on provider_event_id and verify the invoice_id before mutating entitlements.
• Timezone skew on current_period_end: services compute renewal windows differently if any one of them works in local time. Store and compare everything in UTC; apply offsets only at the presentation layer.
• Reconciliation rate-limit exhaustion: a naive full-table reconciliation will trip the provider’s API limits and partially sync. Shard by tenant, honour Retry-After, and cache successful reads with a short TTL.
• Outbox poller falling behind: if the poller lags by hours you get split-brain reads. Add a health check that halts downstream mutations when consumer lag exceeds a threshold, and alert on pending rows older than five minutes.

Related

• Database Sync & Consistency Patterns
• Idempotency & Event Deduplication
• Outbox Pattern & Event Publishing
• Resolving Out-of-Order Webhook Delivery Issues