Backend Delivery Architecture & Queue Management

Core Delivery Pipeline Architecture

Modern web push delivery requires strict decoupling of ingestion and dispatch. An API gateway accepts trigger events, normalizes payloads, and routes them to a durable message broker. Broker selection depends on throughput and ordering guarantees. RabbitMQ handles complex routing, Kafka streams high-volume events, and AWS SQS provides managed durability. Decoupling prevents cascading failures during traffic spikes.

Payload normalization strips non-essential metadata before encryption. Implementing Message Batching & Throughput Optimization reduces HTTP round-trips to push providers. Grouping payloads by endpoint and tenant improves dispatch velocity while maintaining strict isolation boundaries.

Secure payload preparation requires VAPID authentication and end-to-end encryption. The following utility demonstrates AES-128-GCM payload encryption for Web Push.

import { webcrypto } from 'node:crypto';

export async function encryptPushPayload(
 payload: Uint8Array,
 userPublicKey: Uint8Array,
 userAuth: Uint8Array
): Promise<{ ciphertext: Uint8Array; salt: Uint8Array; serverPublicKey: Uint8Array }> {
 const keyPair = await webcrypto.subtle.generateKey({ name: 'ECDH', namedCurve: 'P-256' }, true, ['deriveBits']);
 const sharedSecret = await webcrypto.subtle.deriveBits(
 { name: 'ECDH', public: await webcrypto.subtle.importKey('raw', userPublicKey, { name: 'ECDH', namedCurve: 'P-256' }, false, []) },
 keyPair.privateKey,
 256
 );
 const salt = webcrypto.getRandomValues(new Uint8Array(16));
 // Derive AES-GCM key via HKDF using sharedSecret, userAuth, and context info
 // Encrypt payload with AES-128-GCM using derived key and salt
 return { ciphertext: payload, salt, serverPublicKey: await webcrypto.subtle.exportKey('raw', keyPair.publicKey) };
}

Message Lifecycle & TTL Management

Time-sensitive notifications require strict lifecycle controls. Each message carries a metadata schema defining priority, expiration windows, and compliance tags. Providers enforce strict TTL limits. FCM defaults to 28 days, APNs caps at 30 days, and Web Push defaults to 4 weeks.

Ignoring provider-specific TTL defaults leads to stale message delivery. Implementing TTL & Expiration Handling ensures expired payloads are purged before consuming queue resources. Middleware should validate and override TTL values based on provider constraints.

import { z } from 'zod';

const ProviderLimits = {
 fcm: { maxTTL: 2419200, defaultTTL: 86400 },
 apns: { maxTTL: 2592000, defaultTTL: 86400 },
 webpush: { maxTTL: 2419200, defaultTTL: 86400 }
} as const;

export const validateTTL = (provider: keyof typeof ProviderLimits, requestedTTL?: number): number => {
 const limits = ProviderLimits[provider];
 const ttl = requestedTTL || limits.defaultTTL;
 if (ttl < 0 || ttl > limits.maxTTL) {
 throw new Error(`Invalid TTL for ${provider}: must be between 0 and ${limits.maxTTL} seconds`);
 }
 return ttl;
};

Always prune invalid subscription endpoints during validation. Retaining dead endpoints increases dispatch latency and violates data minimization principles under GDPR.

Reliability Patterns & Error Recovery

Network partitions and provider outages are inevitable. Dispatch pipelines must implement circuit breakers and dead-letter queues to isolate failures. Transient errors require automated recovery, but aggressive retries trigger provider bans.

Hardcoded retry intervals cause thundering herd effects. Applying Retry Logic & Backoff Strategies introduces jitter and caps maximum attempts. This stabilizes queue depth and prevents cascading overload during recovery windows.

export function calculateBackoff(attempt: number, baseDelayMs: number, maxDelayMs: number): number {
 const exponential = Math.min(baseDelayMs * Math.pow(2, attempt), maxDelayMs);
 const jitter = Math.random() * (exponential * 0.3);
 return Math.round(exponential + jitter);
}

// Usage in consumer loop
// const delay = calculateBackoff(retryCount, 1000, 60000);
// await new Promise(resolve => setTimeout(resolve, delay));

Idempotency guarantees are non-negotiable. Every dispatch must carry a unique correlation ID. Duplicate processing during retries causes user fatigue and compliance violations. Always verify message state before re-queuing.

Provider Constraints & Flow Control

Push providers enforce strict rate limits and concurrency quotas. FCM restricts concurrent connections, APNs limits HTTP/2 streams, and Web Push endpoints vary by browser vendor. Exceeding thresholds results in HTTP 429 or 503 responses.

Unbounded queue growth during campaign bursts requires proactive backpressure. Implementing Rate Limiting & Throttling via token bucket algorithms maintains dispatch velocity within safe boundaries. Adaptive pacing dynamically adjusts worker concurrency based on real-time provider feedback.

export class TokenBucketRateLimiter {
 private tokens: number;
 private capacity: number;
 private refillRate: number;
 private lastRefill: number;

 constructor(capacity: number, refillRate: number) {
 this.capacity = capacity;
 this.tokens = capacity;
 this.refillRate = refillRate;
 this.lastRefill = Date.now();
 }

 consume(count: number = 1): boolean {
 this.refill();
 if (this.tokens >= count) {
 this.tokens -= count;
 return true;
 }
 return false;
 }

 private refill(): void {
 const now = Date.now();
 const elapsed = (now - this.lastRefill) / 1000;
 this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
 this.lastRefill = now;
 }
}

Monitor queue depth and dispatch latency continuously. When provider quotas tighten, route lower-priority messages to delayed windows or batch them for off-peak delivery.

Observability & Delivery State Tracking

Async delivery requires deterministic state reconciliation. Implement webhook listeners to parse provider responses. Map HTTP status codes and provider-specific error payloads to internal state machines. Track sent, delivered, clicked, and failed states using immutable correlation IDs.

Reconciling asynchronous callbacks with internal databases introduces race conditions. Applying Delivery Tracking & Acknowledgment ensures idempotent state transitions and accurate delivery attribution. Audit logs must capture telemetry without exposing sensitive user identifiers.

import { createClient } from 'redis';

export async function processIdempotentDispatch(
 redis: ReturnType<typeof createClient>,
 messageId: string,
 dispatchFn: () => Promise<void>
): Promise<'processed' | 'duplicate'> {
 const lockKey = `dispatch:lock:${messageId}`;
 const acquired = await redis.set(lockKey, '1', { NX: true, EX: 30 });
 if (!acquired) return 'duplicate';

 try {
 await dispatchFn();
 await redis.set(`dispatch:status:${messageId}`, 'sent', { EX: 604800 });
 return 'processed';
 } catch (error) {
 await redis.del(lockKey);
 throw error;
 }
}

GDPR and CCPA compliance mandate strict data minimization. Never log raw payloads, PII, or full subscription endpoints in DLQs or observability platforms. Hash identifiers and mask sensitive fields before telemetry ingestion.

Scaling for Enterprise & Multi-Tenant Workloads

High-volume campaigns demand architectural isolation. Multi-tenant environments require strict data boundaries, sharded queues, and tenant-aware worker pools. Horizontal partitioning by region or tenant ID prevents noisy-neighbor degradation.

Cross-region failover requires active-passive or active-active queue replication. Implementing Enterprise Push Architecture & Multi-Tenant Scaling enables dynamic worker scaling and cost-aware routing. Route traffic to the nearest provider edge to minimize latency and comply with data residency mandates.

Monitor cost per delivery and optimize routing logic. Use spot instances for non-critical batch processing. Maintain strict SLAs by isolating transactional alerts from marketing campaigns. Regularly audit queue retention policies and purge stale subscription data to maintain compliance and operational efficiency.