The double-send postmortem

Tuesday 14:02. Support escalates: a user received the same OTP four times in ninety seconds. Logs show one logical "send login code" call from the app. The SMS provider dashboard shows four delivered messages. Billing shows four charges.

Here is the sequence that produced it.

1. Client POST /v1/messages with the OTP payload.
2. Provider accepts the message, enqueues delivery, begins writing 201 {"id":"msg_9f3a…"}.
3. The TCP connection stalls. Client-side read deadline fires at 10s. The client library surfaces TimeoutError.
4. Retry middleware treats timeout as retryable, fires the same POST again.
5. Provider has no memory of the first attempt from the client's point of view. It accepts a second message. User's phone buzzes again.

The message was sent on attempt one. The client simply never learned the outcome. Naive retry-on-timeout is the bug — not the network, not the provider. Every path where you cannot distinguish "never reached the server" from "server finished, response lost" will double-send unless you give the server a stable identity for the logical operation.

That identity is an idempotency key. Everything below exists because of this failure mode.

Idempotency keys done properly

Generate one UUIDv4 per logical message — once, when the business intent is created — not per HTTP attempt. Send it on every try:

POST /v1/messages HTTP/1.1
Host: api.example-sms.invalid
Content-Type: application/json
Idempotency-Key: 6b8e4f2a-9c1d-4e3f-a7b2-1d5c8e9f0a3b

{"to":"+15551234567","body":"Your code is 482901"}

Server contract you need (and must verify in the provider docs):

• If the key has not been seen: process normally, store the response body and status keyed by the idempotency key, return that response.
• If the key has been seen inside the retention window (typically 24h): return the stored status and body byte-for-byte. Do not send another SMS.
• Concurrent requests with the same key: one wins; the other blocks or receives the same replay once the first completes.

Client generation (do this at intent time, persist the key next to the outbox row — see later):

import uuid
from dataclasses import dataclass

@dataclass(frozen=True)
class SmsIntent:
    to: str
    body: str
    idempotency_key: str

def new_sms_intent(to: str, body: str) -> SmsIntent:
    return SmsIntent(
        to=to,
        body=body,
        idempotency_key=str(uuid.uuid4()),
    )

Key scoping gotcha. The server must bind the key to a hash of the request body (method + path + body). If it only keys on the string you send, a retry that accidentally changes body (template fix, different OTP digit, locale tweak) silently returns the original response while your logs claim you "sent" the new content. I first hit the body-hash scoping issue against an SMS HTTP API where my retry wrapper reused the key after a template change — the server correctly replayed the old response, which is exactly what the spec says should happen and exactly not what I expected.

Rule: one key ↔ one immutable payload. Change the payload → new key. Never mint a fresh key inside the retry loop.

Which failures to retry (and which never)

Retrying the wrong status is how you turn one bad number into five failed API calls and a support ticket that looks like an outage.

import httpx

RETRYABLE_STATUS = frozenset({408, 425, 429, 500, 502, 503, 504})
NON_RETRYABLE_STATUS = frozenset({400, 401, 403, 404, 422})

# Provider-specific permanent failures (example codes from public Twilio error docs)
PERMANENT_PROVIDER_CODES = frozenset({
    21211,  # Invalid 'To' Phone Number
    21214,  # 'To' phone number cannot be reached
    21614,  # 'To' number is not a valid mobile number
})

def should_retry(exc: BaseException | None, response: httpx.Response | None) -> bool:
    if response is None:
        # connect error, DNS failure, or read timeout with no response object
        return isinstance(exc, (
            httpx.ConnectError,
            httpx.ConnectTimeout,
            httpx.ReadTimeout,
            httpx.RemoteProtocolError,
        ))
    if response.status_code in NON_RETRYABLE_STATUS:
        return False
    try:
        code = response.json().get("code")
        if code in PERMANENT_PROVIDER_CODES:
            return False
    except Exception:
        pass
    return response.status_code in RETRYABLE_STATUS

Timeout classification matters: a connect timeout never delivered bytes, so retry without fear of duplication even before keys exist. A read timeout after httpx finished writing the body is only safe when the same Idempotency-Key rides along.

Backoff that survives a real outage

Fixed 200ms retries during a provider blip synchronize every worker onto the same recovery second. Use full jitter, cap the sleep, and let Retry-After win on 429.

import random
import time
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def backoff_seconds(attempt: int, base: float = 0.5, cap: float = 60.0) -> float:
    # Full jitter: sleep ~ U(0, min(cap, base * 2^attempt))
    # AWS Architecture Blog ("Exponential Backoff And Jitter") measured
    # full jitter as lower-tail latency and less herd correlation than
    # equal jitter or decorrelated forms under client thundering.
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)

def retry_after_seconds(response: httpx.Response) -> float | None:
    raw = response.headers.get("Retry-After")
    if raw is None:
        return None
    try:
        return max(0.0, float(raw))
    except ValueError:
        try:
            dt = parsedate_to_datetime(raw)
            if dt.tzinfo is None:
                dt = dt.replace(tzinfo=timezone.utc)
            return max(0.0, (dt - datetime.now(timezone.utc)).total_seconds())
        except (TypeError, ValueError, IndexError):
            return None

def send_with_retries(
    client: httpx.Client,
    intent: SmsIntent,
    max_attempts: int = 8,
) -> httpx.Response:
    last_exc: BaseException | None = None
    for attempt in range(max_attempts):
        try:
            resp = client.post(
                "/v1/messages",
                json={"to": intent.to, "body": intent.body},
                headers={"Idempotency-Key": intent.idempotency_key},
                timeout=httpx.Timeout(connect=5.0, read=15.0, write=10.0, pool=5.0),
            )
            if not should_retry(None, resp):
                return resp
            delay = retry_after_seconds(resp)
            if delay is None:
                delay = backoff_seconds(attempt)
            time.sleep(delay)
        except (httpx.ConnectError, httpx.ConnectTimeout,
                httpx.ReadTimeout, httpx.RemoteProtocolError) as exc:
            last_exc = exc
            if not should_retry(exc, None) or attempt == max_attempts - 1:
                raise
            time.sleep(backoff_seconds(attempt))
    if last_exc:
        raise last_exc
    raise RuntimeError("send_with_retries exhausted without response")

Retry-After on 429 is authoritative over your schedule — the provider is telling you when its limiter opens. Full jitter (U(0, capped_exp)) desynchronizes waiters better than equal jitter (half_exp + U(0, half_exp)); the AWS architecture blog on exponential backoff and jitter is the measurement reference most teams lean on here.

The outbox pattern for SMS

Do not call the SMS HTTP API inside the request handler or inside the DB transaction that creates the user session. A crash between COMMIT and the HTTP response leaves you with no durable record of intent — or worse, a commit and a half-finished retry storm.

Write the send intent into an sms_outbox table in the same transaction as the business change. A worker drains the table. The outbox row ID (or a dedicated UUID column) is the idempotency key.

CREATE TABLE sms_outbox (
    id              UUID PRIMARY KEY,
    to_e164         TEXT NOT NULL,
    body            TEXT NOT NULL,
    status          TEXT NOT NULL DEFAULT 'pending'
                    CHECK (status IN ('pending','in_progress','sent','dead')),
    attempt_count   INT  NOT NULL DEFAULT 0,
    max_attempts    INT  NOT NULL DEFAULT 8,
    next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    last_error      TEXT,
    provider_ref    TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX sms_outbox_drain_idx
    ON sms_outbox (next_attempt_at)
    WHERE status = 'pending';

Business handler (same transaction):

import uuid
import psycopg

def create_login_challenge(conn: psycopg.Connection, user_id: int, e164: str, code: str) -> None:
    body = f"Your code is {code}"
    outbox_id = uuid.uuid4()
    with conn.transaction():
        conn.execute(
            "INSERT INTO login_challenges (user_id, code) VALUES (%s, %s)",
            (user_id, code),
        )
        conn.execute(
            """
            INSERT INTO sms_outbox (id, to_e164, body)
            VALUES (%s, %s, %s)
            """,
            (outbox_id, e164, body),
        )

Worker loop (~30 lines, runnable-shaped):

import os
import time
import httpx
import psycopg
from psycopg.rows import dict_row

DSN = os.environ["DATABASE_URL"]
BASE_URL = os.environ["SMS_API_BASE"]

def claim_batch(conn: psycopg.Connection, limit: int = 32) -> list[dict]:
    with conn.transaction():
        rows = conn.execute(
            """
            UPDATE sms_outbox
            SET status = 'in_progress',
                updated_at = now()
            WHERE id IN (
                SELECT id FROM sms_outbox
                WHERE status = 'pending'
                  AND next_attempt_at <= now()
                ORDER BY next_attempt_at
                FOR UPDATE SKIP LOCKED
                LIMIT %s
            )
            RETURNING *
            """,
            (limit,),
        ).fetchall()
    return rows

def mark_sent(conn: psycopg.Connection, row_id, provider_ref: str) -> None:
    conn.execute(
        """
        UPDATE sms_outbox
        SET status = 'sent', provider_ref = %s, updated_at = now()
        WHERE id = %s
        """,
        (provider_ref, row_id),
    )

def mark_retry_or_dead(conn: psycopg.Connection, row: dict, err: str) -> None:
    attempt = row["attempt_count"] + 1
    if attempt >= row["max_attempts"]:
        conn.execute(
            """
            UPDATE sms_outbox
            SET status = 'dead', attempt_count = %s,
                last_error = %s, updated_at = now()
            WHERE id = %s
            """,
            (attempt, err[:1000], row["id"]),
        )
        return
    delay = backoff_seconds(attempt)
    conn.execute(
        """
        UPDATE sms_outbox
        SET status = 'pending',
            attempt_count = %s,
            last_error = %s,
            next_attempt_at = now() + (%s || ' seconds')::interval,
            updated_at = now()
        WHERE id = %s
        """,
        (attempt, err[:1000], str(delay), row["id"]),
    )

def worker_once(client: httpx.Client) -> int:
    with psycopg.connect(DSN, row_factory=dict_row) as conn:
        batch = claim_batch(conn)
        for row in batch:
            intent = SmsIntent(to=row["to_e164"], body=row["body"],
                               idempotency_key=str(row["id"]))
            try:
                resp = send_with_retries(client, intent)
                if resp.status_code >= 400 and not should_retry(None, resp):
                    mark_retry_or_dead(conn, row, resp.text)
                    conn.commit()
                    continue
                resp.raise_for_status()
                ref = resp.json().get("id", "")
                mark_sent(conn, row["id"], ref)
                conn.commit()
            except Exception as exc:
                mark_retry_or_dead(conn, row, repr(exc))
                conn.commit()
        return len(batch)

def main() -> None:
    with httpx.Client(base_url=BASE_URL) as client:
        while True:
            n = worker_once(client)
            if n == 0:
                time.sleep(0.5)

if __name__ == "__main__":
    main()

Crash between provider accept and mark_sent: row stays in_progress or returns to pending via a sweeper that reclaims stale in_progress rows older than 2× your max HTTP budget. The next attempt reuses id as Idempotency-Key, so the provider replays the original accept. Effect on the user's phone: once.

Poison messages and the DLQ

A permanent failure that keeps cycling (21211, malformed body that slipped validation, destination that started rejecting) burns worker capacity and drowns metrics.

• attempt_count + max_attempts on every row (schema above).
• next_attempt_at advances with the same jittered exponential schedule so one bad row does not hot-loop.
• At attempt_count >= max_attempts, set status = 'dead'. That row is the dead-letter queue; no separate system required on day one.
• Alert on SELECT count(*) FROM sms_outbox WHERE status = 'dead' AND updated_at > now() - interval '1 hour' crossing a threshold — not on each individual failure. Individual failures during a provider incident are expected; DLQ depth growing while the provider is healthy means a code or data bug.

Optional sweeper for stuck claims:

UPDATE sms_outbox
SET status = 'pending',
    next_attempt_at = now(),
    updated_at = now()
WHERE status = 'in_progress'
  AND updated_at < now() - interval '5 minutes';

Verifying it actually works

Patterns without a chaos test are articles. Reproduce the original postmortem under control.

1. Run the provider (or a stub that records deliveries) behind toxiproxy.
2. Configure a toxic: timeout with timeout=0 on the response path after the stub has accepted the body — response never reaches the client, message is "delivered" server-side.
3. Drive one logical send through the outbox worker.
4. Assert: stub's delivery log length == 1 for that idempotency key; outbox row ends as sent; worker attempt_count may be > 1.

# stub_provider.py — records by idempotency key
from fastapi import FastAPI, Header, Request
from fastapi.responses import JSONResponse
import uuid

app = FastAPI()
delivered: dict[str, dict] = {}

@app.post("/v1/messages")
async def send(request: Request, idempotency_key: str = Header(alias="Idempotency-Key")):
    body = await request.json()
    if idempotency_key in delivered:
        prev = delivered[idempotency_key]
        return JSONResponse(prev["body"], status_code=prev["status"])
    msg_id = f"msg_{uuid.uuid4().hex[:12]}"
    payload = {"id": msg_id, "to": body["to"], "status": "queued"}
    delivered[idempotency_key] = {"body": payload, "status": 201}
    return JSONResponse(payload, status_code=201)

# toxiproxy: after stub is up on 9090
toxiproxy-cli create sms --listen 127.0.0.1:8888 --upstream 127.0.0.1:9090
toxiproxy-cli toxic add sms -t timeout -a timeout=0 -d downstream
# point SMS_API_BASE at http://127.0.0.1:8888 and run worker_once twice
# then: curl http://127.0.0.1:9090/admin/delivered | jq 'length'  → 1 per key

If the count is 2, your key is not stable across retries or the stub is not replaying. Fix that before shipping the backoff knobs.

Checklist

• [ ] Idempotency key minted once per logical SMS, stored before any HTTP attempt
• [ ] Key sent on every retry; never regenerated inside the retry loop
• [ ] Provider confirmed to dedupe on the key and bind it to body hash for the retention window
• [ ] Retry only 408/425/429/5xx, connect failures, and read timeouts — never 400/401/403/404 or permanent provider codes like 21211
• [ ] Read-timeout retries gated on idempotency key presence
• [ ] Full jitter backoff with ~60s cap; Retry-After overrides the local schedule on 429
• [ ] SMS intent written to sms_outbox in the same DB transaction as the business write; worker drains with FOR UPDATE SKIP LOCKED
• [ ] max_attempts + DLQ (status = 'dead') with alerts on DLQ depth; chaos test with timeout-after-accept proves one delivery per key