Multi-Provider Push Gateway Design

A web push subscription is just a URL, and that URL decides which company’s infrastructure your message actually crosses. Chrome hands you https://fcm.googleapis.com/…, Firefox hands you https://updates.push.services.mozilla.com/…, Safari hands you https://web.push.apple.com/…, and legacy Edge still hands out https://…notify.windows.com/…. One dispatcher has to serve all of them, and each one throttles, clamps, and fails differently. This guide is the reference for building that dispatcher as part of the broader Backend Delivery Architecture & Queue Management pipeline.

The design goal is narrow and worth stating up front: the gateway must isolate providers from each other. A degraded Autopush region must not consume the worker pool that Chrome traffic needs, and a rate-limit storm on FCM must not delay a Safari alert. Everything below — routing, adapters, budgets, breakers, error normalisation — exists to enforce that isolation.

Prerequisites

• A broker-backed dispatch queue (Redis, RabbitMQ, or Kafka) already feeding stateless workers
• VAPID keys available at runtime as process.env.VAPID_PUBLIC_KEY / process.env.VAPID_PRIVATE_KEY
• Payloads encrypted to aes128gcm and verified under the 4 KB ciphertext limit before enqueue
• Per-request metrics with a provider label already wired into your metrics backend
• Node.js 18+ (for undici, fetch, and stable HTTP/2 client behaviour)

One Dispatcher, Many Push Services

The gateway is a single logical component with four responsibilities: decide which push service a job belongs to, apply that service’s policy, execute the HTTP request over a pooled connection, and translate whatever came back into one internal vocabulary. Everything downstream — the delivery ledger, the retry scheduler, the dashboards — should never need to know which vendor answered.

Routing happens on the endpoint authority (the host portion of the subscription URL), not on a browser column in your database. User agents change, subscriptions get migrated by pushsubscriptionchange, and a Chromium fork may report an unfamiliar UA while still using FCM. The URL the browser gave you is the only authoritative signal, and it is stable for the life of that subscription.

Provider Capability Reference

Before writing adapter code, pin down what each authority actually accepts. The numbers below reflect the published protocol limits plus behaviour that is stable in production traffic; treat the softer columns as starting points and confirm them against your own telemetry, because push services adjust quotas without announcement.

The two authorities that carry almost all real traffic differ in more ways than this table can hold — the byte-level contrast is in FCM vs Mozilla Autopush endpoint differences.

Routing by Endpoint Authority

Resolution is a host lookup with a suffix fallback, because WNS uses regional subdomains (db5p.notify.windows.com, sin.notify.windows.com, and dozens more) and FCM has historically served both /fcm/send/ and /wp/ path shapes on the same host. Match on the registrable suffix, never on a full-string equality against a single hostname.

Unknown authorities must not be an error. Self-hosted Autopush deployments, Chromium forks with a custom push service, and future vendors all appear as hosts you have never seen. Route them to a conservative default adapter — modest concurrency, standard backoff — and emit a metric so the unknown host shows up in your dashboard rather than in a support ticket.

// providers/registry.js
const PROVIDERS = [
  {
    id: 'fcm',
    label: 'Google FCM',
    match: (host) => host === 'fcm.googleapis.com' || host.endsWith('.googleapis.com'),
    maxTtlSeconds: 2419200,
    requestsPerSecond: 900,
    maxConcurrent: 256,
    breaker: { failureRatio: 0.25, windowMs: 30000, cooldownMs: 60000, probes: 3 }
  },
  {
    id: 'autopush',
    label: 'Mozilla Autopush',
    match: (host) => host.endsWith('push.services.mozilla.com'),
    maxTtlSeconds: 2419200,
    requestsPerSecond: 300,
    maxConcurrent: 96,
    breaker: { failureRatio: 0.25, windowMs: 30000, cooldownMs: 60000, probes: 3 }
  },
  {
    id: 'apple',
    label: 'Apple Web Push',
    match: (host) => host.endsWith('push.apple.com'),
    maxTtlSeconds: 2419200,
    requestsPerSecond: 120,
    maxConcurrent: 48,
    breaker: { failureRatio: 0.20, windowMs: 30000, cooldownMs: 90000, probes: 2 }
  },
  {
    id: 'wns',
    label: 'Microsoft WNS',
    match: (host) => host.endsWith('notify.windows.com'),
    maxTtlSeconds: 259200,
    requestsPerSecond: 40,
    maxConcurrent: 16,
    breaker: { failureRatio: 0.30, windowMs: 60000, cooldownMs: 120000, probes: 2 }
  }
];

const FALLBACK = {
  id: 'unknown',
  label: 'Unrecognised push service',
  match: () => true,
  maxTtlSeconds: 86400,
  requestsPerSecond: 25,
  maxConcurrent: 8,
  breaker: { failureRatio: 0.30, windowMs: 60000, cooldownMs: 120000, probes: 2 }
};

export function resolveProvider(endpoint) {
  let host;
  try {
    const url = new URL(endpoint);
    if (url.protocol !== 'https:') throw new Error('non-https endpoint');
    host = url.host.toLowerCase();
  } catch {
    return { provider: FALLBACK, host: null, valid: false };
  }
  const provider = PROVIDERS.find((p) => p.match(host)) ?? FALLBACK;
  return { provider, host, valid: true };
}

Rejecting non-HTTPS endpoints inside the resolver is doing double duty: RFC 8030 requires a secure transport, and refusing arbitrary schemes here is your first line of defence against a stored-subscription SSRF, where an attacker plants an internal URL in the subscription table and lets your worker fetch it.

The Provider Adapter Interface

An adapter is a thin object with one required method and a handful of policy hooks. Keeping the surface small is what makes adding a fifth provider a fifty-line change instead of a refactor. The dispatcher only ever calls send(); everything vendor-specific — header shaping, TTL clamping, response parsing — hides behind it.

// providers/adapter.js
import { request } from 'undici';
import { buildVapidHeader } from '../auth/vapid.js';
import { classify } from '../errors/taxonomy.js';

export function createAdapter(provider, pool, limiter, breaker) {
  return {
    id: provider.id,

    // Clamp before the push service does, so the value we log is the value applied.
    normaliseTtl(requestedSeconds) {
      const ttl = Number.isInteger(requestedSeconds) ? requestedSeconds : 86400;
      return Math.max(0, Math.min(ttl, provider.maxTtlSeconds));
    },

    buildHeaders({ endpoint, ttl, urgency, topic, bodyLength }) {
      const headers = {
        'content-encoding': 'aes128gcm',
        'content-type': 'application/octet-stream',
        'content-length': String(bodyLength),
        ttl: String(ttl),
        authorization: buildVapidHeader(endpoint)
      };
      if (urgency) headers.urgency = urgency;
      // Topic must be a URL-safe base64 token of 32 characters or fewer (RFC 8030 §5.4).
      if (topic) headers.topic = topic.slice(0, 32);
      return headers;
    },

    async send(job) {
      if (!breaker.allow()) {
        return { code: 'CIRCUIT_OPEN', retryable: true, retryAfterMs: breaker.cooldownRemaining() };
      }
      return limiter.run(async () => {
        const ttl = this.normaliseTtl(job.ttlSeconds);
        const started = process.hrtime.bigint();
        try {
          const res = await pool.request({
            path: new URL(job.endpoint).pathname,
            method: 'POST',
            headers: this.buildHeaders({
              endpoint: job.endpoint,
              ttl,
              urgency: job.urgency,
              topic: job.topic,
              bodyLength: job.ciphertext.length
            }),
            body: job.ciphertext,
            headersTimeout: 10000,
            bodyTimeout: 10000
          });
          const text = await res.body.text();
          const outcome = classify(provider.id, res.statusCode, res.headers, text);
          breaker.record(outcome.retryable && outcome.code !== 'GONE');
          outcome.latencyMs = Number(process.hrtime.bigint() - started) / 1e6;
          outcome.appliedTtl = ttl;
          return outcome;
        } catch (networkError) {
          breaker.record(false);
          return { code: 'TRANSPORT_ERROR', retryable: true, detail: networkError.code ?? 'ECONN' };
        }
      });
    }
  };
}

Note what the adapter does not do: it never decides whether to retry, never writes to the database, and never sleeps. It returns a classified outcome and lets the retry scheduler act on it, which keeps the adapter unit-testable against recorded fixtures and keeps retry and backoff policy in exactly one place.

Shared VAPID Signing Across Providers

There is one VAPID key pair for the whole gateway, not one per provider. The public key is what the browser embedded in the subscription when it called pushManager.subscribe(), so it is fixed for the life of that subscription regardless of which push service issued the endpoint. Rotating keys per provider would silently invalidate every existing subscription on that provider.

What is per-provider is the JWT audience. RFC 8292 requires the aud claim to be the origin of the push service — scheme plus host, no path — so a single signer produces four distinct tokens. Cache them: signing an ES256 JWT costs roughly 0.3 ms, which is irrelevant once but very relevant at 5,000 dispatches per second. Cache by audience, expire a few minutes before the token does, and never let the exp exceed 24 hours.

// auth/vapid.js
import { createSign, createPrivateKey } from 'node:crypto';

const TOKEN_LIFETIME_SEC = 12 * 60 * 60;
const REFRESH_MARGIN_SEC = 15 * 60;
const cache = new Map(); // audience -> { header, expiresAt }

const b64url = (buf) => Buffer.from(buf).toString('base64url');

function signJwt(audience) {
  // NEVER hardcode these. They are injected from the secrets manager at boot.
  const privateKeyPem = process.env.VAPID_PRIVATE_KEY;
  const publicKey = process.env.VAPID_PUBLIC_KEY;
  const subject = process.env.VAPID_SUBJECT; // mailto: or https: URL

  const exp = Math.floor(Date.now() / 1000) + TOKEN_LIFETIME_SEC;
  const header = b64url(JSON.stringify({ typ: 'JWT', alg: 'ES256' }));
  const payload = b64url(JSON.stringify({ aud: audience, exp, sub: subject }));

  const signer = createSign('SHA256');
  signer.update(`${header}.${payload}`);
  const der = signer.sign(createPrivateKey(privateKeyPem));
  const jwt = `${header}.${payload}.${b64url(derToJose(der))}`;

  return { value: `vapid t=${jwt}, k=${publicKey}`, exp };
}

// ES256 JWS wants a raw 64-byte r||s pair, not the DER sequence OpenSSL emits.
function derToJose(der) {
  let offset = 3;
  const rLen = der[offset];
  offset += 1;
  const r = der.subarray(offset, offset + rLen);
  offset += rLen + 2;
  const s = der.subarray(offset);
  const out = Buffer.alloc(64);
  Buffer.from(r).subarray(-32).copy(out, 32 - Math.min(32, r.length));
  Buffer.from(s).subarray(-32).copy(out, 64 - Math.min(32, s.length));
  return out;
}

export function buildVapidHeader(endpoint) {
  const { origin } = new URL(endpoint);
  const hit = cache.get(origin);
  const now = Math.floor(Date.now() / 1000);
  if (hit && hit.exp - now > REFRESH_MARGIN_SEC) return hit.value;
  const fresh = signJwt(origin);
  cache.set(origin, fresh);
  return fresh.value;
}

Two failure modes deserve alarms rather than log lines. A 401 across a single authority almost always means clock drift on that worker or an aud built from the full endpoint URL instead of its origin. A 403 means the key pair no longer matches the one that created the subscription, which is what a botched rotation looks like in production — the safe rotation procedure is in VAPID key generation and rotation.

Rate Limits and Concurrency Budgets

Each adapter owns two independent controls. A token bucket caps requests per second, which is what push services actually meter. A semaphore caps requests in flight, which is what protects your own process from unbounded memory and socket growth when a provider slows down. You need both: a fast provider can exhaust the bucket while holding almost nothing in flight, and a stalled provider can hold hundreds of requests in flight while consuming almost no tokens.

Budget allocation should be proportional to your subscriber mix, not equal across providers. If 72% of your subscriptions are Chromium and 19% Firefox, giving Apple and WNS a quarter of the worker pool each guarantees idle capacity next to a starving FCM queue. Compute the split from a nightly query over the subscription table and load it as configuration, so the budget drifts with your audience instead of with a hardcoded guess.

// limits/budget.js
export function allocateBudgets(totalConcurrency, mixByProvider, floors = {}) {
  const total = Object.values(mixByProvider).reduce((a, b) => a + b, 0) || 1;
  const budgets = {};
  let assigned = 0;
  for (const [id, count] of Object.entries(mixByProvider)) {
    const floor = floors[id] ?? 4;
    const share = Math.round((count / total) * totalConcurrency);
    budgets[id] = Math.max(floor, share);
    assigned += budgets[id];
  }
  // Trim proportionally if the floors pushed us over the global budget.
  if (assigned > totalConcurrency) {
    const scale = totalConcurrency / assigned;
    for (const id of Object.keys(budgets)) {
      budgets[id] = Math.max(floors[id] ?? 4, Math.floor(budgets[id] * scale));
    }
  }
  return budgets;
}

Keep a floor for every provider even when its share rounds to zero. A transactional alert to the three Safari users who happen to be your enterprise pilot customers still has to go out, and a budget of zero turns that into an indefinite queue wait rather than a slow send. Connection-level tuning that turns those budgets into real throughput is covered in HTTP/2 connection pooling for web push at scale.

Circuit Breakers per Authority

A circuit breaker keyed to the endpoint authority is the mechanism that makes provider isolation real. When Autopush starts returning 503 at 40%, the Firefox breaker opens, Firefox jobs fail fast and get requeued with a delay, and the workers those jobs would have occupied go straight back to draining FCM. Without the breaker, every Firefox job burns a ten-second timeout while holding a worker slot, and total pipeline throughput collapses even though three of four providers are perfectly healthy.

The subtlety is what counts as a failure. A 410 Gone is a perfectly healthy response — the push service is working correctly and telling you the subscription is dead. Counting 410s as failures means a routine prune of stale Chrome subscriptions trips the FCM breaker in the middle of a campaign. Count only 5xx, 429, and transport errors; treat 4xx other than 429 as a successful round-trip with a bad outcome. Cleanup for those dead endpoints belongs in the 410 Gone handling flow.

// limits/breaker.js
export function createBreaker({ failureRatio, windowMs, cooldownMs, probes }) {
  let state = 'closed';
  let openedAt = 0;
  let probesLeft = 0;
  const events = []; // { t, ok }

  const prune = (now) => {
    while (events.length && now - events[0].t > windowMs) events.shift();
  };

  return {
    get state() { return state; },
    cooldownRemaining() { return Math.max(0, cooldownMs - (Date.now() - openedAt)); },

    allow() {
      const now = Date.now();
      if (state === 'closed') return true;
      if (state === 'open') {
        if (now - openedAt < cooldownMs) return false;
        state = 'half-open';
        probesLeft = probes;
      }
      if (state === 'half-open' && probesLeft > 0) {
        probesLeft -= 1;
        return true;
      }
      return false;
    },

    record(ok) {
      const now = Date.now();
      if (state === 'half-open') {
        if (!ok) { state = 'open'; openedAt = now; return; }
        if (probesLeft === 0) { state = 'closed'; events.length = 0; }
        return;
      }
      events.push({ t: now, ok });
      prune(now);
      if (events.length < 20) return; // never trip on a thin sample
      const failures = events.filter((e) => !e.ok).length;
      if (failures / events.length >= failureRatio) {
        state = 'open';
        openedAt = now;
      }
    }
  };
}

Normalising Divergent Error Bodies

Every push service reports the same underlying conditions in a different shape. FCM returns a JSON envelope with a error.status enum. Autopush returns a flat JSON object with a numeric errno and a human message. Apple returns a JSON body with a reason string. WNS puts everything in X-WNS-* response headers and sends an empty body. If any of that vocabulary leaks past the adapter, every consumer downstream has to learn four dialects.

The mapping itself is mostly status-code driven, with a small per-provider table for the cases where the status alone is ambiguous — most importantly the FCM 404, which can mean either a genuinely unknown registration or a malformed path, and the Autopush 400 with errno 104, which means the TTL header was unparseable rather than the payload being wrong.

// errors/taxonomy.js
const BY_STATUS = {
  200: 'ACCEPTED', 201: 'ACCEPTED', 202: 'ACCEPTED',
  400: 'INVALID_REQUEST',
  401: 'UNAUTHORIZED',
  403: 'UNAUTHORIZED',
  404: 'GONE',
  410: 'GONE',
  413: 'PAYLOAD_TOO_LARGE',
  429: 'RATE_LIMITED'
};

const RETRYABLE = new Set(['RATE_LIMITED', 'PROVIDER_DOWN', 'TRANSPORT_ERROR', 'CIRCUIT_OPEN']);

function parseRetryAfter(headers) {
  const raw = headers['retry-after'];
  if (!raw) return null;
  const seconds = Number(raw);
  if (Number.isFinite(seconds)) return seconds * 1000;
  const when = Date.parse(raw);
  return Number.isNaN(when) ? null : Math.max(0, when - Date.now());
}

function providerDetail(providerId, body, headers) {
  if (providerId === 'wns') return headers['x-wns-error-description'] ?? null;
  try {
    const json = JSON.parse(body);
    if (providerId === 'fcm') return json?.error?.status ?? json?.error?.message ?? null;
    if (providerId === 'autopush') return json?.errno != null ? `errno ${json.errno}` : json?.message ?? null;
    if (providerId === 'apple') return json?.reason ?? null;
    return json?.message ?? null;
  } catch {
    return body ? body.slice(0, 180) : null;
  }
}

export function classify(providerId, status, headers = {}, body = '') {
  let code = BY_STATUS[status];
  if (!code) code = status >= 500 ? 'PROVIDER_DOWN' : 'INVALID_REQUEST';

  const detail = providerDetail(providerId, body, headers);

  // Autopush errno 104 is an unparseable TTL header, not a bad payload.
  if (providerId === 'autopush' && status === 400 && detail === 'errno 104') {
    code = 'INVALID_TTL';
  }
  // Apple sometimes answers 503 during maintenance with no body at all.
  if (providerId === 'apple' && status === 503) code = 'PROVIDER_DOWN';

  return {
    code,
    status,
    provider: providerId,
    retryable: RETRYABLE.has(code),
    retryAfterMs: parseRetryAfter(headers),
    rawDetail: detail
  };
}

Persist both code and rawDetail on the delivery record. The internal code drives every automated decision; the raw detail is what an engineer reads at 3 a.m. when a provider quietly changes an enum value and your INVALID_REQUEST bucket suddenly doubles.

Observability per Provider

Every metric the gateway emits carries a provider label. Aggregate dashboards hide exactly the failures this architecture exists to contain: a 3% overall error rate looks fine and is catastrophic if all of it lands on one authority. Split the four core signals — dispatch latency, outcome mix, rate-limit rate, breaker state — and alert on the per-provider series, never the total.

The minimum viable dashboard row per authority: request rate, p50/p95/p99 latency, a stacked outcome chart by taxonomy code, 429 share, breaker state as a step function, and in-flight requests against the configured budget. That last pair is the one that catches budget misallocation — a provider pinned at its ceiling for hours is under-provisioned, and a provider never touching half its budget is holding capacity that FCM needs.

Cardinality discipline matters. Label by provider and code, never by endpoint, subscription ID, or full URL. Hashing endpoints before they reach telemetry is both a cardinality control and a GDPR data-minimisation requirement, since a push endpoint is a stable per-device identifier.

Implementation Steps

1. Inventory your authorities. Run a GROUP BY over the host portion of every stored endpoint. The result is the real provider mix, and it usually contains one or two hosts nobody expected.
2. Write the resolver and its fallback. Match on registrable suffix, reject non-HTTPS, and emit a counter for every fallback hit.
3. Extract one adapter from your existing send path. Start with the provider carrying the most traffic, keep the old path behind a flag, and compare outcomes for a day.
4. Give each adapter its own limiter, pool and breaker. Size the budgets from the subscriber mix, with a floor per provider.
5. Move VAPID signing behind a shared, audience-keyed cache. Verify that the aud claim is the origin only, with no path component.
6. Replace every direct status-code check downstream with the taxonomy code. Grep for === 410 and statusCode >= 500 across the codebase; those are the leaks.
7. Add the per-provider dashboard and alerts before the rollout, not after. You need the baseline to tell a regression from normal provider variance.

Configuration Reference

Verification

Prove the routing table before you prove anything else. A one-line query over the subscription store tells you whether the resolver covers your real traffic:

SELECT split_part(split_part(endpoint, '://', 2), '/', 1) AS authority,
       count(*)                                            AS subscriptions
FROM subscriptions
WHERE status = 'active'
GROUP BY 1
ORDER BY 2 DESC;

Then confirm the gateway reaches each authority with a valid signature. A TTL: 0 request with no body is the cheapest live probe — the push service either accepts it and drops it immediately, or tells you why it will not:

curl -i -X POST "$TEST_ENDPOINT" \
  -H "TTL: 0" \
  -H "Authorization: vapid t=$VAPID_JWT, k=$VAPID_PUBLIC_KEY" \
  -H "Content-Length: 0"

Expect 201 Created from FCM and Autopush, 201 or 200 from Apple. A 401 means the JWT audience or the clock is wrong; a 403 means the key pair does not match the subscription; a 410 means the endpoint is dead and the resolver is fine. Finally, force a breaker trip in staging by pointing one adapter at a host that returns 503, and watch the other three providers hold their throughput flat — that flat line is the entire point of the design.

Error and Edge-Case Matrix

Cross-Browser Notes

Chromium browsers all route through FCM, but the path shape is not uniform: older subscriptions use /fcm/send/<token>, newer ones use /wp/<token>. Match on the host and pass the path through untouched rather than reconstructing it.

Firefox on desktop and Android both use Autopush, and Mozilla publishes the server, so self-hosted deployments on arbitrary hostnames are legitimate traffic. Your fallback adapter is what handles them gracefully.

Safari’s authority is web.push.apple.com for both macOS and iOS, but the iOS subscription only exists after the user adds the site to the Home Screen — the practical consequences are covered in the Safari and iOS web push integration guide. Apple is also the strictest about the VAPID sub claim, rejecting anything that is not a mailto: or https: URL.

WNS endpoints belong to EdgeHTML-era Edge and are effectively a decaying population. Keep the adapter, keep the budget tiny, and let attrition handle the rest.

Related

• FCM vs Mozilla Autopush endpoint differences — the byte-level contrast between the two authorities that carry most web push traffic.
• HTTP/2 connection pooling for web push at scale — how to turn per-provider concurrency budgets into actual throughput.
• Message Batching & Throughput Optimization — the semaphore and pacing layer the adapters plug into.
• Retry Logic & Backoff Strategies — what the scheduler does with a retryable taxonomy code.
• Delivery Tracking & Acknowledgment — where normalised outcomes land in the delivery ledger.

Back to Backend Delivery Architecture & Queue Management

FAQ