A/B Testing Web Push Notifications

Most teams “test” push notifications by sending two versions and eyeballing which got more clicks. That is not a test — it is a coin flip dressed up as data. A real A/B test assigns subscribers to variants randomly, holds everything else constant, collects enough samples to overcome noise, and only then declares a winner with a stated confidence level. This guide walks through the full procedure with runnable Node.js and service-worker code.

This is a deep-dive within the push engagement and campaign optimization guide. It depends on one measurement primitive — accurately counting clicks against delivered notifications — which is covered in detail in measuring push notification click-through rate.

Prerequisites

• A working push pipeline that can send to subscribers (VAPID auth configured via process.env.VAPID_PUBLIC_KEY and process.env.VAPID_PRIVATE_KEY).
• An append-only event store recording at least sent, accepted, display, and click per subscriber, campaign, and variant.
• A subscriber table with stable subscriber ids and current delivery_status.
• A service worker that handles notificationclick and can emit a tracking beacon.
• At least a few thousand active, recently-engaged subscribers — small lists cannot detect realistic effect sizes.

How a Push A/B Test Works

An experiment has three moving parts. First, a hypothesis: a falsifiable statement like “an urgency-framed title increases CTR over the neutral control.” Second, assignment: each eligible subscriber is deterministically and randomly placed into exactly one variant. Third, attribution: every click is tied back to the variant the subscriber received, so the comparison is apples-to-apples.

The critical discipline is changing one thing at a time. If variant B has both a new title and a new send time, a win tells you nothing about which change mattered. Keep the payload, icon, timing, and segment identical across variants except for the single element under test. Segment selection itself is upstream of the test and is covered in push personalization and segmentation.

Assignment must be sticky: a subscriber who lands in “control” today should land in “control” on a re-send, otherwise contamination ruins the result. Deterministic hashing gives you stickiness without a database write per subscriber.

Step 1 — Write a Falsifiable Hypothesis

State the change, the metric, the direction, and the minimum effect you care about. “Adding a first-name token to the title raises CTR by at least 1.5 absolute points” is testable; “make the copy better” is not. The minimum detectable effect (MDE) directly drives your sample size, so commit to it before you start.

const experiment = {
  id: 'title-personalization-v1',
  hypothesis: 'First-name title raises CTR by >= 1.5 absolute points',
  control: { title: 'Your weekly summary is ready' },
  variant: { title: '{firstName}, your weekly summary is ready' },
  splitRatio: 0.5,        // 50/50
  mdeAbsolute: 0.015,     // 1.5 percentage points
  baselineCtr: 0.06       // measured, not guessed
};

Step 2 — Compute the Required Sample Size

A two-proportion test needs enough subscribers per arm to detect the MDE at your chosen significance (alpha, typically 0.05) and power (typically 0.80). Undersized tests produce false negatives; you change something, it actually worked, and the noise hides it. Compute the per-arm size before sending.

// Per-arm sample size for a two-proportion z-test
function sampleSizePerArm(p1, mde, alpha = 0.05, power = 0.8) {
  const zA = 1.959963985;  // two-sided z for alpha=0.05
  const zB = 0.841621234;  // z for power=0.80
  const p2 = p1 + mde;
  const pBar = (p1 + p2) / 2;
  const num = Math.pow(
    zA * Math.sqrt(2 * pBar * (1 - pBar)) + zB * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)),
    2
  );
  return Math.ceil(num / Math.pow(mde, 2));
}

// e.g. baseline 6% CTR, detect a 1.5pt lift:
console.log(sampleSizePerArm(0.06, 0.015)); // ~ 4,300 delivered per arm

If your eligible cohort is smaller than two arms’ worth of delivered (not merely sent) notifications, either widen the segment, raise the MDE, or run the test longer across multiple sends. The relationship is steeply non-linear: the sample you need grows roughly with the square of the precision you demand, so halving the effect you are willing to detect roughly quadruples the list you need.

Step 3 — Assign Variants Deterministically

Hash the subscriber id together with the experiment id, take a bucket, and map it to a variant by your split ratio. Seeding with the experiment id keeps concurrent experiments independent, and hashing keeps the same subscriber in the same arm on every send.

const crypto = require('crypto');

function assignVariant(subscriberId, experimentId, splitRatio = 0.5) {
  const h = crypto.createHash('sha256')
    .update(`${experimentId}:${subscriberId}`)
    .digest();
  const r = h.readUInt32BE(0) / 0xffffffff; // uniform 0..1
  return r < splitRatio ? 'control' : 'variant';
}

Step 4 — Send and Record the Assignment

When you send, write a sent event with the variant, then record accepted from the push-service response. The variant must travel inside the encrypted payload so the service worker can attribute the eventual click. Keep that payload under the 4 KB ciphertext limit — it is encrypted with the aes128gcm content encoding per RFC 8291 before transmission — so carry only the variant id and a target url, not full copy.

const webpush = require('web-push');
webpush.setVapidDetails(
  'mailto:ops@example.com',
  process.env.VAPID_PUBLIC_KEY,   // never hardcode these
  process.env.VAPID_PRIVATE_KEY
);

async function sendExperiment(db, sub, exp, ctx) {
  const arm = assignVariant(sub.id, exp.id, exp.splitRatio);
  const title = (arm === 'control' ? exp.control.title : exp.variant.title)
    .replace('{firstName}', ctx.firstName ?? 'there');

  await db.query(
    `INSERT INTO push_events (subscriber_id, campaign_id, variant, event_type)
     VALUES ($1, $2, $3, 'sent')`,
    [sub.id, exp.id, arm]
  );

  try {
    const res = await webpush.sendNotification(
      sub,
      JSON.stringify({ title, url: ctx.url, campaignId: exp.id, variant: arm }),
      { TTL: 86400, urgency: 'normal' }
    );
    await db.query(
      `INSERT INTO push_events (subscriber_id, campaign_id, variant, event_type, status_code)
       VALUES ($1, $2, $3, 'accepted', $4)`,
      [sub.id, exp.id, arm, res.statusCode]
    );
  } catch (err) {
    // 410 Gone -> retire endpoint; 429 -> back off and requeue
    await db.query(
      `INSERT INTO push_events (subscriber_id, campaign_id, variant, event_type, status_code)
       VALUES ($1, $2, $3, 'failed', $4)`,
      [sub.id, exp.id, arm, err.statusCode ?? 0]
    );
  }
}

When a send returns 429 Too Many Requests, do not drop the subscriber from the experiment — requeue it using your retry logic and backoff so the arm sizes stay balanced.

Step 5 — Track Clicks in the Service Worker

The browser fires notificationclick when the user taps the notification. Read the variant out of the notification data, fire a tracking beacon, and focus or open the target. This client event is what populates the numerator of CTR.

// sw.js — attribute the click back to its variant
self.addEventListener('notificationclick', (event) => {
  const d = event.notification.data || {};
  event.notification.close();
  event.waitUntil((async () => {
    await fetch('/t/click', {
      method: 'POST',
      keepalive: true,
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ campaignId: d.campaignId, variant: d.variant, ts: Date.now() })
    }).catch(() => {});
    const url = d.url || '/';
    const all = await clients.matchAll({ type: 'window', includeUncontrolled: true });
    const hit = all.find((c) => c.url.includes(url));
    if (hit) return hit.focus();
    return clients.openWindow(url);
  })());
});

Step 6 — Measure Lift and Significance

Per arm, CTR is clicks / accepted. Lift is the difference between variant and control CTR. Then run a two-proportion z-test: if the resulting p-value is below your alpha, the difference is statistically significant. The exact CTR counting logic — and why you must use accepted, not sent, as the denominator — is detailed in measuring push notification click-through rate, while the interpretation of the p-value itself, confidence intervals, and why repeated peeking manufactures false positives are worked through in statistical significance for push notification tests.

function twoProportionZ(cClicks, cN, vClicks, vN) {
  const p1 = cClicks / cN, p2 = vClicks / vN;
  const pPool = (cClicks + vClicks) / (cN + vN);
  const se = Math.sqrt(pPool * (1 - pPool) * (1 / cN + 1 / vN));
  const z = (p2 - p1) / se;
  // two-sided p-value via normal CDF approximation
  const p = 2 * (1 - normCdf(Math.abs(z)));
  return { liftAbsolute: p2 - p1, z, pValue: p, significant: p < 0.05 };
}

function normCdf(x) {
  return (1 + erf(x / Math.SQRT2)) / 2;
}
function erf(x) {
  const t = 1 / (1 + 0.3275911 * Math.abs(x));
  const y = 1 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-x * x);
  return x >= 0 ? y : -y;
}

A significant p-value is a necessary condition for shipping a variant, not a sufficient one. Three gates have to clear in order: both arms must have reached the per-arm sample size you committed to, the p-value must be below alpha, and the arms must be balanced enough that the comparison is fair. Fail any one of them and the honest answer is “keep running” or “fix the plumbing first”.

Step 7 — Record the Decision and Retire the Experiment

An experiment that ends in a chat message is an experiment you will accidentally run again next quarter. Persist the decision next to the numbers that produced it, then promote the winner to be the new control so the following test measures incremental progress instead of re-litigating settled ground.

CREATE TABLE push_experiments (
  id            TEXT PRIMARY KEY,            -- also the assignment hash seed
  hypothesis    TEXT         NOT NULL,
  baseline_ctr  NUMERIC(6,4) NOT NULL,
  mde_absolute  NUMERIC(6,4) NOT NULL,
  alpha         NUMERIC(4,3) NOT NULL DEFAULT 0.050,
  target_n_arm  INT          NOT NULL,       -- written BEFORE the first send
  started_at    TIMESTAMPTZ  NOT NULL,
  ended_at      TIMESTAMPTZ,
  outcome       TEXT,                        -- 'shipped' | 'no_effect' | 'abandoned'
  observed_lift NUMERIC(6,4),
  p_value       NUMERIC(8,6)
);

Writing target_n_arm before the first send is what makes early stopping auditable afterwards: any row whose ended_at precedes the arms reaching target_n_arm was called early, and a reviewer can see that without taking anyone’s word for it. Recording no_effect outcomes matters at least as much as recording wins. The register of changes that did not move CTR is the most valuable document a growth team owns, and teams without one propose the same title experiment every second quarter.

Two habits keep the register honest. Keep a small permanent holdback — a few percent of subscribers who always receive the original control copy — so cumulative drift stays measurable a year later, when the ninth consecutive “winner” has been layered on top of the eighth and nobody can say whether the compound effect is real. And replicate any surprising win once on a fresh cohort before it becomes part of the template: at alpha 0.05 a programme running forty tests a year should expect roughly two significant results that are pure noise, and a replication costs one more send cycle rather than a strategy built on a coin flip.

Configuration Reference

Verification

Before trusting any result, confirm the plumbing. Open Chrome DevTools, go to the Application panel, and dispatch a test push to your own subscription to confirm both the display and click events reach your tracking endpoint.

# Confirm arm balance and that accepted counts are close to 50/50
curl -s https://example.com/admin/experiment/title-personalization-v1/arms | jq
# Expect: { "control": { "accepted": 4180, "click": 251 },
#           "variant": { "accepted": 4203, "click": 309 } }

If the two arms differ by more than a couple of percent in accepted count, your assignment or delivery is skewed — investigate before reading the lift.

Verify the pipeline end to end

Four things can each be individually plausible and still combine into a wrong answer: assignment, payload, beacon, and aggregation. Check them in that order, because a fault upstream makes every downstream check meaningless.

1. Assignment is uniform and sticky. Run the assignment function over your real subscriber ids offline and count the buckets before a single notification is sent. Uniformity you assume is uniformity you do not have.

node -e '
const { assignVariant } = require("./experiment");
const ids = require("./subscriber-ids.json");        // export of real subscriber uuids
const n = { control: 0, variant: 0 };
for (const id of ids) n[assignVariant(id, "title-personalization-v1", 0.5)]++;
const sticky = ids.every(id =>
  assignVariant(id, "title-personalization-v1", 0.5) ===
  assignVariant(id, "title-personalization-v1", 0.5));
console.log(n, "sticky:", sticky);
'
# => { control: 24974, variant: 25026 } sticky: true

2. The payload actually carries the variant. Log the parsed object the service worker receives for a test send and compare it against the shape you expect. A missing variant key is the single most common cause of a test that reports two arms with suspiciously identical CTR, because every click falls through to the default bucket.

{
  "title": "Dana, your weekly summary is ready",
  "url": "/summary/2026-07-20",
  "campaignId": "title-personalization-v1",
  "variant": "variant"
}

3. Replay the beacon by hand. The tracking endpoint should be callable without a browser, which also makes it testable in CI. Fire the same request twice and assert that the second one changes nothing.

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://example.com/t/click \
  -H 'Content-Type: application/json' \
  -d '{"campaignId":"title-personalization-v1","variant":"variant","subscriberId":"11111111-1111-1111-1111-111111111111","ts":1753000000000}'
# 204 — replay the identical request, then confirm the row count did not move

The replay is the check that matters most. Networks retry, keepalive requests get re-sent as the worker is torn down, and users tap the same notification twice. If a duplicate increments your click count, both arms are inflated by an unknown and unequal amount, and the p-value you compute afterwards is fiction rather than evidence.

4. Aggregation agrees with a hand count. Compute the arm table with your reporting query and again with a deliberately naive one, and require them to match before you read the lift.

SELECT variant,
       COUNT(DISTINCT subscriber_id) FILTER (WHERE event_type = 'accepted') AS accepted,
       COUNT(DISTINCT subscriber_id) FILTER (WHERE event_type = 'click')    AS clickers,
       COUNT(*)                      FILTER (WHERE event_type = 'click')    AS click_events
FROM push_events
WHERE campaign_id = 'title-personalization-v1'
GROUP BY variant;

If click_events exceeds clickers by more than a rounding error, you are counting repeat taps. Both definitions are defensible — unique clickers is the honest answer for a copy test, total taps is the right answer for a traffic forecast — but you must pick one and use it in the numerator of both arms. Mixing them silently is how an arm ends up with a CTR above 100%.

Error & Edge-Case Matrix

Cross-Browser Notes

Chromium, Firefox, and Safari all fire notificationclick, so attribution works everywhere, but display behaviour differs. Safari on iOS and macOS enforces stricter user-gesture and background-delivery rules, so its display-to-accepted ratio is often lower — segment your analysis by browser to avoid one platform’s throttling masking a real copy win. Firefox uses its own push service with a slightly smaller effective payload ceiling, so a payload that fits Chrome may be rejected there. Confirm payload sizing against the maximum payload size limits for Chrome vs Firefox, and review broader divergence in cross-browser notification quirks.

Related

• Back to Push Notification Engagement & Campaign Optimization — the engagement loop this experiment optimizes.
• Measuring push notification click-through rate — the CTR counting primitive every A/B test depends on.
• Statistical significance for push notification tests — read p-values, confidence intervals, and peeking risk correctly.
• Push personalization & segmentation — choose the cohort your experiment runs on.
• Retry logic & backoff strategies — keep arm sizes balanced when sends are rate limited.

FAQ