Backend Delivery Architecture & Queue Management

This guide is the architectural reference for the server side of web push: how a trigger event becomes an encrypted, signed, rate-limited HTTP request to a browser push service, and how you track, retry, and expire that request at scale. Everything here builds on the same RFC 8030 transport that the Core Protocols & Browser Implementation guide covers from the client side.

Core Delivery Pipeline Architecture

Modern web push delivery requires strict decoupling of ingestion and dispatch. An API gateway accepts trigger events, normalizes payloads, and routes them to a durable message broker. Broker selection depends on throughput and ordering guarantees. RabbitMQ handles complex routing, Kafka streams high-volume events, and AWS SQS provides managed durability. Decoupling prevents cascading failures during traffic spikes.

Payload normalization strips non-essential metadata before encryption. Implementing Message Batching & Throughput Optimization reduces HTTP round-trips to push providers. Grouping payloads by endpoint and tenant improves dispatch velocity while maintaining strict isolation boundaries.

Secure payload preparation requires VAPID authentication and end-to-end encryption. Every payload is capped at a 4 KB ciphertext limit and must use the aes128gcm content-encoding mandated by RFC 8291 — the mechanics are detailed in the Push API Payload Encryption guide. In practice, use a standards-compliant library such as web-push (Node.js) or pywebpush (Python) that implements this encoding correctly, and sign each request with a VAPID key pair loaded from process.env.VAPID_PRIVATE_KEY — never hardcode keys in source. The following example shows TTL validation, which is a simpler but production-critical concern:

import { z } from 'zod';

// FCM and Web Push Protocol (RFC 8030) cap TTL at 2419200 s (28 days).
// APNs uses apns-expiration (Unix timestamp), but per-message max is also 30 days.
const ProviderLimits = {
  fcm:      { maxTTL: 2419200, defaultTTL: 86400 },
  webpush:  { maxTTL: 2419200, defaultTTL: 86400 },
} as const;

export const validateTTL = (
  provider: keyof typeof ProviderLimits,
  requestedTTL?: number
): number => {
  const limits = ProviderLimits[provider];
  const ttl = requestedTTL ?? limits.defaultTTL;
  if (ttl < 0 || ttl > limits.maxTTL) {
    throw new Error(
      `Invalid TTL for ${provider}: must be 0–${limits.maxTTL} seconds`
    );
  }
  return ttl;
};

Always prune invalid subscription endpoints during validation. Retaining dead endpoints increases dispatch latency and violates data minimization principles under GDPR. The schema that backs that validation — the endpoint record, its key material, its status column, and the pruning cadence that keeps the table honest — is covered end to end in Subscription Storage & Lifecycle.

The ingestion stage should also enforce idempotency at the boundary. Campaign APIs frequently fire the same trigger twice — a retried webhook, a double-clicked send button, a replayed event from an at-least-once stream. Deduplicate at the gateway using a deterministic key derived from the campaign and the recipient, and reject the duplicate before it ever reaches the broker. This keeps the queue clean and prevents user-facing notification fatigue downstream. The broker itself should be treated as the durable system of record between ingestion and dispatch: once an event is acknowledged into the queue, the gateway can return 202 Accepted to the caller, and all subsequent work — encryption, signing, rate limiting, retrying — happens asynchronously against that durable record.

Message Lifecycle & TTL Management

Time-sensitive notifications require strict lifecycle controls. Each message carries a metadata schema defining priority, expiration windows, and compliance tags. Providers enforce TTL limits via the TTL HTTP header (RFC 8030). FCM and the Web Push Protocol both accept up to 2,419,200 seconds (28 days). APNs uses the apns-expiration Unix timestamp header instead of a duration.

Ignoring provider-specific TTL defaults leads to stale message delivery. Implementing TTL & Expiration Handling ensures expired payloads are purged before consuming queue resources. Middleware should validate and clamp TTL values based on provider constraints before dispatch.

TTL is not a single setting but a value that must be mirrored across three layers: the broker eviction policy, the TTL HTTP header, and a final freshness check inside the service worker. When these disagree, you get phantom deliveries — a message the push service would have dropped but the broker kept, or vice versa. The cleanest model assigns a TTL tier per notification class at ingestion (seconds for one-time passcodes, hours for system alerts, a day or more for promotional campaigns) and carries that value through every subsequent stage as immutable metadata. The trade-off between the two extremes of that range is explored in TTL 0 vs TTL 86400 delivery guarantees.

Reliability Patterns & Error Recovery

Network partitions and provider outages are inevitable. Dispatch pipelines must implement circuit breakers and dead-letter queues to isolate failures. Transient errors require automated recovery, but aggressive retries trigger provider bans.

Hardcoded retry intervals cause thundering herd effects. Applying Retry Logic & Backoff Strategies introduces jitter and caps maximum attempts. This stabilizes queue depth and prevents cascading overload during recovery windows.

export function calculateBackoff(
  attempt: number,
  baseDelayMs: number,
  maxDelayMs: number
): number {
  const exponential = Math.min(baseDelayMs * Math.pow(2, attempt), maxDelayMs);
  const jitter = Math.random() * (exponential * 0.3);
  return Math.round(exponential + jitter);
}

Idempotency guarantees are non-negotiable. Every dispatch must carry a unique correlation ID. Duplicate processing during retries causes user fatigue and compliance violations. Always verify message state before re-queuing.

Failure classification is the heart of a reliable retry layer. Not every non-2xx response should be retried: 400, 401, 403, 404, and 410 are terminal and retrying them only wastes compute and risks reputation damage, while 429 and the 5xx family are transient and recoverable. A 410 Gone is special — it is the push service telling you the subscription is permanently dead, which should immediately flip the subscription’s status and halt all further attempts; the cleanup pattern at scale is detailed in Handling 410 Gone responses at scale. Pair classification with per-region circuit breakers so a single degraded push gateway cannot drag the whole pipeline into a retry storm.

Provider Constraints & Flow Control

Push providers enforce strict rate limits and concurrency quotas. FCM restricts concurrent connections, APNs limits HTTP/2 streams, and Web Push endpoints vary by browser vendor. Exceeding thresholds results in HTTP 429 or 503 responses, and the way you absorb a sustained 429 directly determines whether a campaign completes or stalls — see Handling 429 Too Many Requests from push services.

Because each vendor exposes a different quota model, error vocabulary and connection ceiling, the abstraction that hides those differences behind one dispatch interface is itself a design problem: pooling, per-origin limits and vendor-specific response mapping are the subject of Multi-Provider Push Gateway Design.

Unbounded queue growth during campaign bursts requires proactive backpressure. A token-bucket rate limiter maintains dispatch velocity within safe boundaries while adaptive pacing dynamically adjusts worker concurrency based on real-time provider feedback.

export class TokenBucketRateLimiter {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private readonly capacity: number,
    private readonly refillRate: number // tokens per second
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  consume(count: number = 1): boolean {
    this.refill();
    if (this.tokens >= count) {
      this.tokens -= count;
      return true;
    }
    return false;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
    this.lastRefill = now;
  }
}

Monitor queue depth and dispatch latency continuously. When provider quotas tighten, route lower-priority messages to delayed windows or batch them for off-peak delivery.

Observability & Delivery State Tracking

Async delivery requires deterministic state reconciliation. Implement webhook listeners to parse provider responses. Map HTTP status codes and provider-specific error payloads to internal state machines. Track sent, delivered, clicked, and failed states using immutable correlation IDs.

Reconciling asynchronous callbacks with internal databases introduces race conditions. Applying Delivery Tracking & Acknowledgment ensures idempotent state transitions and accurate delivery attribution. Audit logs must capture telemetry without exposing sensitive user identifiers.

import { createClient } from 'redis';

export async function processIdempotentDispatch(
  redis: ReturnType<typeof createClient>,
  messageId: string,
  dispatchFn: () => Promise<void>
): Promise<'processed' | 'duplicate'> {
  const lockKey = `dispatch:lock:${messageId}`;
  const acquired = await redis.set(lockKey, '1', { NX: true, EX: 30 });
  if (!acquired) return 'duplicate';

  try {
    await dispatchFn();
    await redis.set(`dispatch:status:${messageId}`, 'sent', { EX: 604800 });
    return 'processed';
  } catch (error) {
    await redis.del(lockKey);
    throw error;
  }
}

GDPR and CCPA compliance mandate strict data minimization. Never log raw payloads, PII, or full subscription endpoints in DLQs or observability platforms. Hash identifiers and mask sensitive fields before telemetry ingestion.

The distinction worth internalizing here is between acknowledgment and delivery. The synchronous HTTP response from the push service confirms only that the encrypted payload was accepted into its queue — it says nothing about whether the device woke, decrypted the message, or showed it. True display confirmation arrives asynchronously, if at all, through events forwarded from the service worker back to your analytics endpoint. Treating the 201 as proof of receipt is the single most common attribution error in push systems, and it inflates “delivered” numbers that the Delivery analytics instrumentation guide shows how to separate into honest delivery-versus-display rates.

Scaling for Enterprise & Multi-Tenant Workloads

High-volume campaigns demand architectural isolation. Multi-tenant environments require strict data boundaries, sharded queues, and tenant-aware worker pools. Horizontal partitioning by region or tenant ID prevents noisy-neighbor degradation.

Cross-region failover requires active-passive or active-active queue replication. Dynamic worker scaling and cost-aware routing reduce latency and help comply with data residency mandates. Route traffic to the nearest provider edge. For infrastructure-level queue scaling details, see Scaling push queues with Redis or RabbitMQ.

Broker choice shapes the operational character of the whole system. Redis with a sorted-set delay queue is the lightest path to scheduled retries and TTL-bound jobs, and it doubles as the idempotency and rate-limiter store, which keeps the moving-part count low. RabbitMQ earns its weight when you need per-tenant exchanges, complex routing keys, and native dead-lettering without hand-rolling it. Kafka fits when the push event itself is part of a larger replayable stream and you want consumer groups to fan the same events into delivery, analytics, and CRM sync independently. Whatever the broker, the dispatch workers themselves stay stateless: they pull a durable job, load the VAPID keys from the secrets manager at runtime, encrypt and sign, and report the outcome back to the ledger. Statelessness is what makes horizontal scaling and spot-instance reclamation safe.

Concurrency control is the other half of scaling. Because Web Push (RFC 8030) is a per-endpoint protocol — one HTTP POST per subscription — “throughput” is really a question of how many of those POSTs you keep in flight without tripping a provider’s rate limit. A semaphore per worker, HTTP/2 connection reuse per push-service origin, and provider-aware pacing are covered end to end in Message Batching & Throughput Optimization, with the empirical sweet-spot tuning in Optimal batch size for web push throughput.

Monitor cost per delivery and optimize routing logic. Use spot instances for non-critical batch processing. Maintain strict SLAs by isolating transactional alerts from marketing campaigns. Regularly audit queue retention policies and purge stale subscription data to maintain compliance and operational efficiency. The campaign-side counterpart to this infrastructure — segmentation, A/B testing, and engagement analytics — lives in the Notification Engagement & Campaign Optimization guide.

Subscription & Delivery Data Model

The delivery pipeline is only as reliable as the storage model behind it. Two tables anchor the system: a subscriptions table that holds the endpoint plus its p256dh/auth keys, and an append-only delivery_log keyed by correlation ID. Store the endpoint hash, not the raw endpoint, anywhere it joins to analytics.

CREATE TABLE subscriptions (
  id              BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  tenant_id       BIGINT NOT NULL,
  endpoint        TEXT NOT NULL UNIQUE,
  endpoint_hash   BYTEA NOT NULL,           -- HMAC-SHA256(endpoint, rotating_key)
  p256dh          TEXT NOT NULL,
  auth            TEXT NOT NULL,
  vapid_key_id    TEXT NOT NULL,            -- which VAPID key signed the most recent send
  status          TEXT NOT NULL DEFAULT 'active', -- active | gone | expired
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  last_seen_at    TIMESTAMPTZ
);

CREATE TABLE delivery_log (
  correlation_id  UUID PRIMARY KEY,
  subscription_id BIGINT NOT NULL REFERENCES subscriptions(id),
  state           TEXT NOT NULL,            -- queued | sent | retrying | gone | expired | failed
  http_status     SMALLINT,
  ttl_seconds     INTEGER NOT NULL,
  attempt         SMALLINT NOT NULL DEFAULT 0,
  dispatched_at   TIMESTAMPTZ,
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_delivery_state ON delivery_log (state, updated_at);

State transitions in delivery_log must be idempotent — a retried dispatch reuses the same correlation_id rather than inserting a new row, which is the contract enforced by the Delivery Tracking & Acknowledgment layer. A 410 Gone flips subscriptions.status to gone and stops all further retries.

Three properties of that schema are load-bearing rather than cosmetic. First, delivery_log is append-mostly and grows linearly with send volume, so it belongs on a range-partitioned table — monthly partitions on updated_at for a moderate sender, daily partitions once you pass roughly ten million dispatches a month. Partition pruning is what keeps the (state, updated_at) index cheap: a sweeper looking for rows stuck in retrying for more than five minutes touches one live partition instead of a year of history. Second, endpoint_hash is an HMAC rather than a bare digest precisely so a leaked analytics extract cannot be brute-forced back into endpoints; the cost of that choice is that rotating the HMAC key invalidates every existing join, so record which key generation produced each hash and dual-write during the rotation window. Third, nothing in delivery_log references a user directly. Identity is reachable only through subscriptions.id, which means an erasure request is satisfied by deleting one subscription row and letting the ledger’s retention window expire the remainder.

Sizing falls out of those decisions. A delivery_log row costs on the order of 120 bytes of tuple plus index overhead, so a sender dispatching five million notifications a day accrues roughly a gigabyte a day including indexes — unremarkable for a month, expensive for a year. Set the retention window from the audit obligation rather than from instinct: keep full rows for 30 days, then roll each expired partition up into per-campaign, per-status counts and drop it. The aggregates feed reporting; the rows feed incident forensics, and forensic value decays fast.

Error Taxonomy

Push services communicate delivery outcomes through HTTP status codes. Mapping each code to a single deterministic action is the difference between a self-healing pipeline and a retry storm.

Capacity Planning & Operational Alerting

Capacity planning for a push pipeline starts from a single number: the wall-clock window in which a campaign has to finish. Divide the audience by that window to get the required dispatch rate, then divide by the per-worker rate you have actually measured to get worker count. A worker holding twenty in-flight HTTP/2 requests against one push-service origin with a 120 ms median round trip sustains roughly 160 dispatches per second. A two-million-subscriber campaign that must complete inside ten minutes therefore needs about 3,300 dispatches per second — twenty-one workers on paper, thirty in practice once you leave headroom for the retries the first pass will generate. Cryptography is not the constraint: sealing a payload with aes128gcm costs well under a millisecond, so the pipeline is dominated by network round trips, not CPU.

What breaks at ten times that volume is rarely the dispatcher itself, and the order of failure is remarkably consistent across deployments.

Alerting should key on the pipeline’s derivatives rather than its absolute levels. Queue depth spikes legitimately whenever a campaign starts, so depth alone is a noisy page; depth still climbing after the ingestion rate has returned to baseline is a genuine incident. The same logic applies to the error mix — a handful of 410 Gone responses is healthy list hygiene, while a 410 rate that jumps by an order of magnitude in an hour usually means a subscription table was restored from a stale backup.

Rollout ordering matters as much as the thresholds. When you introduce any of this into a pipeline that is already sending, add the ledger writes before you add the retry scheduler, and add the retry scheduler before you tighten concurrency. A retry layer built on top of a ledger that is not yet idempotent will multiply duplicate sends rather than recover failures, and raising concurrency before the breaker exists converts a provider hiccup into a full campaign stall.

Compliance, Transport & Privacy

Web push is HTTPS-only end to end: the service worker that receives the message must be served over a secure context, and your dispatch endpoints must reject any non-HTTPS subscription URL during validation (this also doubles as SSRF protection). Under GDPR and CCPA, the legal basis for sending is explicit opt-in, so log consent — timestamp, version of the prompt copy, and the action taken — at the moment of subscription rather than reconstructing it later. The UX side of capturing and honoring that consent is covered in the Frontend Permission UX & Subscription Flows guide.

Apply data minimization everywhere telemetry flows: never write raw payloads, PII, or full endpoints to dead-letter queues, observability platforms, or logs. Hash endpoints with a rotating HMAC key before they leave the dispatch boundary, and scrub payload bodies from aggregation pipelines once a message passes its TTL. A tight Content-Security-Policy on the pages that register the service worker reduces the blast radius of any injected script that could tamper with subscriptions.

The consent record deserves its own append-only table rather than a boolean column on the user row. Store the subscription id, the timestamp, the prompt variant the user actually saw, the origin that asked, the resulting Notification.permission value, and then the same shape of record again on withdrawal. Regulators ask two questions that a boolean cannot answer: can you demonstrate consent for this specific send, and can you demonstrate that a withdrawal took effect promptly? An append-only ledger answers both without archaeology. Wire the withdrawal path so it flips subscription status synchronously and cancels anything already queued for that endpoint — a notification that lands after an unsubscribe is the single most complaint-generating failure in push, and it is almost always a queue that was never drained rather than a policy that was never written.

Three transport checks belong in code rather than in a runbook. Reject any subscription whose endpoint is not an https: URL. Reject endpoints that resolve into private address space, loopback, or link-local ranges before they reach the queue — a dispatcher is an authenticated outbound HTTP client, which makes it an attractive server-side request forgery pivot, and the check has to happen at subscription time because DNS can change afterwards. And keep an explicit list of push-service hosts you are willing to talk to, routing anything unrecognised through a conservative fallback rather than trusting it by default.

For CSP, the sensitive surface is the page that calls pushManager.subscribe(), because a script injected there can silently re-subscribe a user against an attacker’s application server key. script-src 'self' with nonced or hashed inline scripts, object-src 'none', and base-uri 'self' remove the cheap paths. worker-src 'self' matters specifically for push: without it a third-party script can register a service worker from an origin you do not control, and that worker owns every notification the user sees from your domain. Set Strict-Transport-Security with a long max-age too, since a single downgraded request during subscription is enough to leak the keys the browser hands back.

Retention on the compliance side is not the same window as retention on the operations side. Delivery rows can expire after 30 days because their forensic value is gone; consent records generally need to outlive the subscription itself, because the obligation is to prove the state of consent at the time of a send that may be questioned much later. Keep the two tables on separate retention clocks and audit them separately.

Related

• Delivery Tracking & Acknowledgment — map push service HTTP responses to an idempotent delivery ledger.
• Message Batching & Throughput Optimization — concurrency control, connection pooling, and provider-aware pacing.
• Retry Logic & Backoff Strategies — exponential backoff, jitter, dead-letter queues, and circuit breakers.
• TTL & Expiration Handling — lifecycle control from the TTL header down to service-worker filtering.
• Subscription Storage & Lifecycle — the storage model behind the pipeline: schema, deduplication, and pruning cadence.
• Multi-Provider Push Gateway Design — one dispatch interface over FCM, Mozilla Autopush, and APNs Web Push.

Back to Web Push Notifications

FAQ