Dead-Letter Queues for Undeliverable Push Messages

A push job has burned its retry budget and still has not been accepted. Deleting it loses the evidence; retrying it forever poisons the queue. The dead-letter queue is where it goes — but only if it belongs to one of the two failure classes that actually warrant one.

Quick Answer

A DLQ is for messages whose failure is unresolved, not for messages whose failure is resolved by deletion. Two classes qualify: payloads the push service will never accept (poison), and transient failures that outlived their retry budget. Everything else has a correct terminal action that is not “park it somewhere”.

The 410 Gone rule is the one teams get wrong most often. A 410 is a successful interaction: the push service told you the truth about a subscription that no longer exists. The correct handler deletes the row, as covered in handling 410 Gone responses at scale. Routing it to a DLQ converts a solved problem into a growing backlog that masks real failures — and after a large-scale browser reinstall event, that backlog can be millions of rows deep.

The DLQ Record

Store enough to triage and replay, and nothing more. In particular store a reference to the payload, not the payload: a DLQ that holds plaintext notification bodies is a long-lived copy of personal data with none of the retention controls your primary store has.

CREATE TABLE push_dlq (
  dlq_id            BIGSERIAL PRIMARY KEY,
  message_id        UUID        NOT NULL,          -- joins back to push_dispatch
  subscription_id   BIGINT      NOT NULL,
  endpoint_host     TEXT        NOT NULL,          -- fcm.googleapis.com, updates.push.services.mozilla.com
  payload_ref       TEXT        NOT NULL,          -- object-store key, encrypted at rest
  attempt_count     SMALLINT    NOT NULL,
  first_attempt_at  TIMESTAMPTZ NOT NULL,
  last_attempt_at   TIMESTAMPTZ NOT NULL,
  terminal_status   SMALLINT,                      -- NULL for transport errors
  error_class       TEXT        NOT NULL,          -- 'poison' | 'retry_exhausted'
  error_code        TEXT        NOT NULL,          -- '413' | 'ECONNRESET' | 'jwt_audience'
  error_snippet     TEXT,                          -- first 512 bytes of the response body
  original_ttl      INTEGER     NOT NULL,
  expires_at        TIMESTAMPTZ NOT NULL,          -- first_attempt_at + original_ttl
  replay_state      TEXT        NOT NULL DEFAULT 'parked',  -- parked|claimed|replayed|discarded
  replay_key        UUID        NOT NULL,          -- idempotency token for the re-send
  replay_count      SMALLINT    NOT NULL DEFAULT 0,
  parked_at         TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX push_dlq_triage    ON push_dlq (error_class, error_code, endpoint_host, parked_at DESC);
CREATE INDEX push_dlq_replayable ON push_dlq (replay_state, expires_at) WHERE replay_state = 'parked';
CREATE UNIQUE INDEX push_dlq_once ON push_dlq (message_id, subscription_id);

expires_at is the field that separates a working DLQ from a dangerous one. It is computed once, from the original dispatch time and the original TTL, and it never moves. Nothing else in the replay path is allowed to overwrite it.

Poison-Message Detection

A poison message is one that will fail identically on every attempt. Detect it before the retry budget is spent, because retrying a 413 eleven times costs eleven signed JWTs and eleven round trips to learn nothing.

Classify on the first response, not on attempt count:

• Deterministic status. 400 and 413 are deterministic for a fixed payload. Route straight to the DLQ with error_class = 'poison' and attempt_count = 1. Never apply backoff to them; the exponential backoff path is for faults that can heal on their own.
• Repeated identical error. Two attempts with the same error_code and the same response snippet, separated by a real backoff interval, is a poison signal even for a nominally transient class. Promote to poison at attempt 3 rather than waiting for attempt 10.
• Endpoint-scoped clustering of one error code. If 413 appears for 100% of one campaign and 0% of every other campaign, the payload builder for that campaign is broken. Group by (error_code, campaign_id) in triage, not by subscription.
• Serialisation crashes. A job that kills its worker before any HTTP request is made — an unbounded template expansion, a circular reference during JSON encoding — must be caught by the worker and parked with terminal_status = NULL. Without that catch it is redelivered forever by the broker’s visibility timeout and takes a worker down each time.

Replay With Idempotency and a TTL Gate

Replay is where DLQs cause outages. The failure mode is always the same: an operator drains a two-day-old DLQ and 40,000 users receive “Your code is 448120” for a code that expired on Tuesday.

Four gates, in order, and every replay passes all four.

// replay.js — drains parked DLQ records through four gates.
const REPLAY_RATE = 0.1;          // fraction of live send capacity
const CIRCUIT_BREAK_AFTER = 5;    // consecutive failures before abort

async function replayBatch(db, sender, { limit = 500 }) {
  const rows = await db.claimParked(limit); // sets replay_state='claimed' atomically
  const outcome = { replayed: 0, stale: 0, gone: 0, duplicate: 0, failed: 0 };
  let consecutiveFailures = 0;

  for (const row of rows) {
    // Gate 1 — freshness. expires_at was fixed at first dispatch and never moved.
    if (new Date(row.expires_at) <= new Date()) {
      await db.close(row.dlq_id, 'discarded', 'ttl_expired');
      outcome.stale += 1;
      continue;
    }

    // Gate 2 — the endpoint must still exist. A replay is not a reason to
    // resurrect a subscription that was deleted after a 410.
    const sub = await db.getActiveSubscription(row.subscription_id);
    if (!sub) {
      await db.close(row.dlq_id, 'discarded', 'subscription_gone');
      outcome.gone += 1;
      continue;
    }

    // Gate 3 — idempotency. replay_key is unique per DLQ record; the dispatch
    // table rejects a second insert with the same key.
    const claimed = await db.claimReplayKey(row.replay_key);
    if (!claimed) {
      await db.close(row.dlq_id, 'discarded', 'already_replayed');
      outcome.duplicate += 1;
      continue;
    }

    // Gate 4 — remaining TTL, not original TTL. A message parked for 3 hours
    // of a 6-hour TTL is re-sent with 3 hours left, never with 6.
    const remainingTtl = Math.floor((new Date(row.expires_at) - Date.now()) / 1000);
    try {
      await sender.send(sub, await loadPayload(row.payload_ref), {
        TTL: remainingTtl,
        urgency: 'normal',
        messageId: row.message_id,
        replayKey: row.replay_key,
      });
      await db.close(row.dlq_id, 'replayed', null);
      outcome.replayed += 1;
      consecutiveFailures = 0;
    } catch (err) {
      await db.release(row.dlq_id, err);   // back to 'parked', attempt_count += 1
      outcome.failed += 1;
      if (++consecutiveFailures >= CIRCUIT_BREAK_AFTER) {
        throw new Error(`Replay aborted after ${CIRCUIT_BREAK_AFTER} consecutive failures`);
      }
    }
    await pace(REPLAY_RATE);   // never let a drain starve live traffic
  }
  return outcome;
}

Gate 4 is the subtle one. Re-sending with the original TTL value hands the push service a fresh full-length storage window for a message that is already old — the message can then sit in push-service storage for another full TTL and arrive absurdly late, which is exactly the pathology described in why push messages arrive hours late. Send the remaining TTL and the arithmetic stays honest.

Alerting Thresholds

A DLQ nobody watches is a slow data-loss mechanism. Alert on rate and composition, not on absolute depth — depth is meaningless without a baseline.

Pair the age alert with the TTL policy from the TTL and expiration handling guide: if your median message TTL is 3,600 seconds, a record parked for 12 hours is already unreplayable and the alert should fire long before an operator finds it.

Gotchas and Edge Cases

• A broker DLQ is not an application DLQ. RabbitMQ’s x-dead-letter-exchange and SQS’s redrive policy capture broker-level failure (nack, visibility timeout, max receives). They know nothing about HTTP status classes and will happily dead-letter a 410. Use the broker DLQ for worker crashes and your own table for delivery outcomes.
• Redrive-all is not a replay. Moving every message back to the source queue re-runs the failure that put them there, at full rate, usually during the same incident. Always drain through the gates, always rate-limited, always with a circuit breaker.
• replay_state = 'claimed' needs a lease, not a flag. A worker that dies mid-drain leaves records claimed forever. Store claimed_until and reclaim expired leases, or the DLQ silently shrinks to zero replayable rows.
• Encrypt at rest and set a retention policy. DLQ records reference user subscriptions and often notification context. Ninety days is a common ceiling; expired records should be deleted by the same job that handles pruning expired subscriptions.
• Never let the DLQ writer share a transaction with the sender. If parking a record requires the same database transaction as the dispatch update, a database incident that causes send failures also prevents you from recording them. Write DLQ entries through a separate connection with its own timeout, and drop to a local log if that write fails.

FAQ

Related

• Implementing Exponential Backoff for Failed Push Deliveries — the retry budget whose exhaustion is the only legitimate route into the DLQ for transient faults.
• Handling 429 Too Many Requests from Push Services — why throttling belongs in the main queue with Retry-After, never in the dead-letter path.
• Handling 410 Gone Responses at Scale — the terminal-delete pipeline that keeps churn out of your DLQ.
• Pruning Expired Push Subscriptions from Your Database — the retention job that should also age out closed dead-letter records.

Back to Retry Logic & Backoff Strategies