Statistical Significance for Push Notification Tests

Most push experiments are called far too early, on far too little data, against a metric that moves by fractions of a percentage point. This reference shows how to decide — before you send — how much volume a test needs, and how to judge the result once it arrives.

Quick Answer

A push A/B test has concluded when you have reached the sample size you committed to before the send, and the two-proportion z-test on click-through at that sample size clears your significance threshold. Nothing else counts as a conclusion.

The denominator matters as much as the test statistic. Use accepted notifications (a 201 from the push service), exactly as described in measuring push notification click-through rate, so that dead endpoints and rejected payloads cannot bias one arm relative to the other.

The Two-Proportion Z-Test, Worked

Click-through is a binary outcome per notification: clicked or not. That makes each arm a binomial sample, and the standard comparison is a z-test on the difference of two proportions using a pooled standard error, because the null hypothesis says both arms share one true rate.

Take a real-sized experiment. Control and variant each received 40,000 accepted notifications. Control produced 1,520 clicks (3.80%); variant produced 1,680 clicks (4.20%). That is a relative lift of 10.5%, which sounds decisive and is not obviously so.

Pool the two arms: p = (1520 + 1680) / 80000 = 0.0400. The pooled standard error of the difference is sqrt(p * (1 - p) * (1/n1 + 1/n2)), which is sqrt(0.04 * 0.96 * 0.00005) = 0.001386. The observed difference is 0.0420 - 0.0380 = 0.0040, so z = 0.0040 / 0.001386 = 2.89. Against a two-tailed normal distribution that is a p-value of roughly 0.004 — comfortably past the 0.05 line, and past 0.01 as well.

Here is the same arithmetic as code you can drop into a reporting job. The normal CDF is approximated with an error-function series so the function has no dependencies.

// two-proportion z-test on clicks / accepted
function erf(x) {
  const s = x < 0 ? -1 : 1;
  const a = Math.abs(x);
  const t = 1 / (1 + 0.3275911 * a);
  const y = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t
    - 0.284496736) * t + 0.254829592) * t * Math.exp(-a * a);
  return s * y;
}
const normalCdf = (z) => 0.5 * (1 + erf(z / Math.SQRT2));

export function proportionTest(clicksA, acceptedA, clicksB, acceptedB) {
  const pA = clicksA / acceptedA;
  const pB = clicksB / acceptedB;
  const pooled = (clicksA + clicksB) / (acceptedA + acceptedB);
  const se = Math.sqrt(pooled * (1 - pooled) * (1 / acceptedA + 1 / acceptedB));
  const z = (pB - pA) / se;
  const pValue = 2 * (1 - normalCdf(Math.abs(z)));       // two-tailed
  return { pA, pB, absoluteLift: pB - pA, relativeLift: pB / pA - 1, z, pValue };
}

// proportionTest(1520, 40000, 1680, 40000)
// -> { relativeLift: 0.105, z: 2.887, pValue: 0.0039 }

The z-test’s normal approximation is safe when both arms expect at least ten clicks and ten non-clicks. Push volumes clear that easily; the problem is never the approximation, it is the sample size.

Sample Size and Minimum Detectable Effect

The single most useful number in a push experiment is the minimum detectable effect (MDE): the smallest relative lift the test can reliably find. It is a design input, not an outcome. Fix your baseline click-through rate, your alpha (0.05), and your power (0.80), and the required sample per arm follows:

n = (z_alpha/2 + z_beta)^2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1)^2

With alpha = 0.05 two-tailed and 80% power, (1.96 + 0.8416)^2 = 7.85. Everything else is driven by the baseline rate and the size of the lift you are hunting.

Accepted notifications per arm, at alpha = 0.05 two-tailed and 80% power. Double each figure for the total send.

Read that table honestly and the implication is uncomfortable. A subject-line tweak that moves click-through by 5% relative — the kind of change most copy tests actually produce — needs roughly 154,000 accepted notifications in each arm at a 4% baseline. A list of 60,000 subscribers split 50/50 gives you 30,000 per arm and can only resolve a lift of roughly 12–13% or larger. Anything subtler is invisible to you, no matter how long you stare at the dashboard.

Why Push Tests Are Chronically Under-Powered

Three structural facts conspire against push experiments.

The base rate is low. Variance in a proportion peaks near 50% and shrinks toward the extremes, but the relative precision at a 3% rate is poor: you need a lot of trials before the ratio settles. Email open rates of 20–30% are far cheaper to test than push click-through of 3%.

The addressable population is small. A push list is the subset of visitors who granted permission, which is a fraction of traffic. Worse, it decays: endpoints go stale, and every 410 Gone you correctly prune under the dead-endpoint retirement process shrinks tomorrow’s sample.

You cannot re-run cheaply. Sending the same notification twice to gather more data is not a bigger sample — it is a different experiment, because the second send hits users who already saw the first. Push has no equivalent of “leave the landing-page test running another fortnight”.

The practical response is to stop testing small things. Test the notification’s concept rather than its comma placement: a different offer, a different trigger event, action buttons versus none, an image versus text-only. Those changes plausibly move click-through by 20–50%, which your volume can actually detect. Save the micro-copy iteration for surfaces with orders of magnitude more traffic.

The second response is to pool. Run the same structural variant across several campaigns and analyse the combined result, keeping campaign as a blocking factor. Three sends of 12,000 per arm give you 36,000 per arm, which crosses into detectable territory for a 12% lift.

The Peeking Problem

Checking a running test and stopping as soon as the p-value dips below 0.05 destroys the guarantee that alpha is supposed to give you. Under a true null, a p-value fluctuates; given enough looks, it will wander under 0.05 by chance. Monitoring daily for two weeks and stopping at the first crossing inflates the false-positive rate from 5% to somewhere in the 20–30% range depending on the number of looks. That is the single most common reason a “winning” push variant fails to reproduce.

There are two defensible ways out.

Fixed horizon. Compute the sample size up front, write it into the experiment record, and read the result exactly once when the arm counters reach it. Dashboards during the run should show delivery health, not the p-value.

Sequential testing. If you genuinely need the option to stop early — because a variant might be harmful — use a method designed for repeated looks. An alpha-spending boundary (O’Brien-Fleming style) allocates a small slice of your alpha to each interim analysis, demanding a very extreme result early and relaxing toward the nominal threshold at the planned end. A simpler alternative for a small number of pre-declared looks is a Pocock-style constant boundary: with five equally spaced looks, require roughly p under 0.016 at each one instead of 0.05.

// Pocock-style constant boundary: same threshold at every interim look.
// Approximation adequate for 2-6 equally spaced analyses at overall alpha 0.05.
const POCOCK_ALPHA = { 2: 0.0294, 3: 0.0221, 4: 0.0182, 5: 0.0158, 6: 0.0142 };

export function sequentialDecision({ look, totalLooks, pValue, unsubDeltaPp }) {
  const bound = POCOCK_ALPHA[totalLooks];
  if (unsubDeltaPp > 0.15) return 'stop-for-harm';         // guardrail breach
  if (pValue < bound) return 'stop-for-success';
  if (look === totalLooks) return 'stop-no-effect';
  return 'continue';
}

The cost of sequential testing is real: to preserve overall alpha you either need more total sample than a fixed-horizon test, or you accept lower power. Do not adopt it because it feels rigorous — adopt it because you need an early-stop option.

One-Tailed or Two-Tailed

A one-tailed test asks “is the variant better?” and puts all of alpha in one tail, so it reaches significance with a smaller sample — roughly the difference between needing z of 1.645 rather than 1.96. That is a genuine 20% or so saving in required sample.

Use it only when a negative result would lead to exactly the same action as a null result, and you have committed to that in writing before the send. For most push tests that is false: a variant that reduces click-through by 15% is important information about your audience, and you want the test to be able to say so. Two-tailed is the default. Choosing the tail after seeing the direction of the effect is not a statistical choice; it is halving your p-value by hand.

Unsubscribe Rate as a Guardrail

Click-through is the metric you are optimising. Unsubscribe rate is the metric you are protecting. A push variant can win decisively on clicks — an urgent, alarming subject line will — while quietly accelerating list attrition, and the loss compounds over months while the click win is booked once.

Treat the guardrail differently from the primary metric:

• Different threshold. You are not trying to prove the variant harms retention; you are trying to be reassured it does not. Flag any variant whose unsubscribe rate is directionally worse at p under 0.20, and investigate before shipping.
• Different denominator. Unsubscribes belong over notified subscribers, not over notifications, because one person unsubscribing once is a single event regardless of how many sends they received. The distinctions between explicit opt-out, permission revocation and silent attrition are laid out in push unsubscribe rate benchmarks and churn.
• Longer window. Clicks arrive within hours; opt-outs trickle in for days. Read the guardrail at least 7 days after the last send, even if the primary metric was locked earlier.

A useful decision rule: ship the variant if it wins on clicks at alpha = 0.05 and the guardrail shows no directional deterioration; hold it for a second run if it wins on clicks but the guardrail is directionally worse; reject it outright if the guardrail is significantly worse regardless of the click result. Feed the same guardrail into the engagement and campaign optimization reporting so it is visible next to every experiment, and cross-check it against your delivery analytics instrumentation so a display-rate regression is not misread as a copy effect.

Gotchas & Edge Cases

• Randomise the subscriber, not the send. Assigning variants per notification lets the same person receive both arms, which correlates the observations and breaks the independence the z-test assumes. Hash the subscriber id into a bucket and keep it stable.
• Unequal arm sizes are fine, unequal denominators are not. The formula handles n1 != n2. What it cannot handle is one arm counting sent while the other counts accepted.
• Novelty decays. A variant that introduces action buttons often wins its first send and reverts by the third as the surface stops being new. Re-measure any structural winner on a later campaign before making it the default.
• Multiple variants multiply false positives. Testing four variants against one control at alpha = 0.05 gives roughly an 18% chance that at least one looks significant by luck. Apply a Bonferroni correction (divide alpha by the number of comparisons) or run head-to-head.
• Do not compute significance on ratios of ratios. Lift-of-lift metrics such as “conversion per click” have a denominator that is itself an outcome of the experiment; a shift in click-through changes who is in the denominator and the comparison stops being causal.
• Time-of-day skew. If one arm’s send drains from the queue faster, it lands at a systematically different local hour. Interleave the arms in the send queue rather than sending control first.

Related

• Back to A/B Testing Web Push Notifications — the experiment design and variant-assignment workflow this analysis sits inside.
• Measuring push notification click-through rate — the accepted-based denominator every arm must share.
• Push unsubscribe rate benchmarks and churn — how to compute the guardrail metric properly.
• Segmenting push subscribers by behaviour — build cohorts large enough to test within.

FAQ