SMS webhooks: retries, idempotency keys and out-of-order events
SMS provider webhooks deliver delivery reports and inbound messages with at-least-once semantics. Handlers must tolerate duplicates, out-of-order states, and retries without double-sending OTPs or corrupting status. This guide covers dedupe keys, idempotent SEND, backoff, authenticity checks, and dead-letter handling.
At-least-once webhooks and deduplication
Delivery report (DLR) callbacks and inbound SMS events are typically pushed with at-least-once delivery. Your endpoint may see the same event more than once because the provider retries on non-success responses, network timeouts, or load-balancer blips. Treat every webhook as potentially duplicated work, not as a single authoritative write.
Deduplicate on a composite of message identifier and state (or event type). For DLRs, store processed pairs such as (provider_message_id, state) where state values include intermediate and terminal statuses the provider emits. For inbound messages, use the provider inbound event id when present; otherwise a stable hash of (from, to, timestamp, body) can serve as a weak key, accepting that true uniqueness varies by provider.
On receipt: verify authenticity, parse, check the dedupe store, and if already processed return HTTP 200 quickly without re-applying side effects. Only after a durable record of the pair should you update application state. Returning non-2xx invites another delivery attempt; returning 200 after partial work without recording the pair invites double application on the next attempt. Prefer a transactional outbox or single-row upsert keyed by the composite so the ack and the business write stay aligned.
Inbound events and DLRs for the same logical conversation can interleave. Keep inbound message storage separate from outbound status machines so a duplicate inbound does not flip outbound lifecycle flags. Log correlation ids from the payload when available so operators can reconstruct timelines without inventing guarantees the transport does not provide.
The SMSRoute throughput planner helps size the worker pool that drains these queues.
Idempotency keys on SEND
Client-side retries on the SEND path are a common source of duplicate SMS, especially OTP and other one-shot codes. If the HTTP client times out after the provider accepted the message but before your process saw HTTP 200, a naive retry creates a second message. Attach an idempotency key on every SEND that must not double-fire.
Generate a key per logical user action (for example one key per OTP challenge id), send it in the header or field your API documents, and persist the mapping from key to provider message id before acknowledging success to your caller. On retry with the same key, return the original result instead of submitting again. Keys should be unique within a window long enough to cover your maximum client retry span; exact retention varies.
OTP flows are particularly sensitive: two messages with different codes or two identical codes both undermine trust and rate limits. Tie the idempotency key to the challenge record and refuse a second SEND until the prior attempt is terminal or explicitly invalidated. Do not reuse keys across different recipients or different message purposes.
Idempotency on SEND complements webhook dedupe; they solve opposite directions. SEND idempotency stops you from creating two provider messages. Webhook dedupe stops you from applying one provider event twice. Both belong in production designs that involve mobile-terminated traffic and asynchronous status.
Retries, backoff, timeouts, and validity
When your handler cannot process a webhook (dependency down, deploy race), fail closed with an appropriate status. Use HTTP 500 for transient internal errors so the provider retries. Use HTTP 429 only when you are intentionally rate-limiting intake and can recover by slowing down. Prefer HTTP 200 only when the event is durably recorded or definitively discarded as invalid according to the auth failure policies you choose.
Provider retry schedules vary. On your outbound calls to the provider and on internal workers that re-drive failed webhook processing, use exponential backoff with jitter. Jitter reduces synchronized retry storms across instances. Cap attempts and total retry duration; unbounded retries turn a brief outage into a self-inflicted flood.
Design timeouts relative to SMS validity periods. A message may expire at the provider or downstream network after a validity window you set on SEND (when the API allows). If your status pipeline is slower than that window, you may observe late DLRs for messages your product already treats as expired. Define product rules explicitly: whether a late DELIVRD after local expiry updates analytics only, or also user-visible state.
Separate connect and read timeouts on HTTP clients. A long validity period does not justify blocking a request thread indefinitely waiting for a DLR; DLRs arrive on the webhook path. Polling, if used at all, should be a backup with conservative intervals and the same idempotent status merge logic as webhooks.
The SMSRoute DLR guide pairs with this article for the state semantics behind these events.
Out-of-order events
Webhook order is not a reliable total order. A terminal state such as DELIVRD can arrive before an intermediate ENROUTE for the same message id. Handlers that naively overwrite status with whatever arrived last will regress state (for example showing enroute after delivered).
Maintain an explicit state machine with monotonic ranks. Map provider states into ordered stages (accepted, routed, delivered, failed, expired, unknown). Apply an update only when the incoming state ranks greater than or equal to the stored rank under your merge rules; equal rank may refresh metadata (timestamp, error code) without demoting. Drop or park transitions that move backward.
Unknown or provider-specific codes should not collapse to a terminal failure without evidence. Park them as unknown, alert if volume spikes, and extend the map deliberately. Partial failures (delivered to handset vs accepted by carrier) differ by route; your product copy should not claim outcomes the state does not support.
For multi-part SMS, segment-level callbacks may exist depending on route and encoding. GSM 7-bit user data can carry 160 characters in a single part or 153 per part when concatenated; UCS-2 carries 70 single or 67 per concatenated part. If you only store one status per logical message, define how segment statuses roll up (all segments delivered versus any failed). E.164 destination addresses are limited to 15 digits; normalize before correlating events so formatting differences do not split one message into two keys.
Authenticity, poison messages, and handler checklist
Verify webhook authenticity before parsing into trusted state. Common patterns include a shared secret in a header, an HMAC over the raw body with a shared key, or mutual TLS. Compare MACs with constant-time equality. Reject failed verification with a non-acceptance response policy you document internally; do not execute side effects on unauthenticated payloads. Rotate secrets with overlapping acceptance windows so in-flight requests survive rotation.
Permanently failing events (schema that never parses, references to unknown tenants after retries exhausted, states you cannot map) belong in a poison or dead-letter queue. Do not block the main consumer on a single bad payload. Capture raw body, headers needed for forensics, failure reason, and attempt count. Provide a replay tool that re-injects only after a code fix or manual mapping, still passing through dedupe so replay does not double-apply fixed events.
Observability matters more than aspirational guarantees. Emit metrics for accepted, duplicate, auth_fail, applied, and dead_lettered. Trace by provider message id and your internal send id. Delivery is inherently uncertain across carrier networks; your system should be honest when status is unknown rather than inventing certainty.
Use the checklist table below as a pre-production gate for DLR and inbound handlers. Each row is a concrete engineering control, not a marketing claim.
SMSRoute's error-code lookup covers every SMPP status your handler will meet.
| Check | What to implement |
|---|---|
| Auth | Shared secret or HMAC on raw body; constant-time compare; secret rotation window |
| Dedupe | Persist (message id, state) or inbound event id before side effects; ack with HTTP 200 only after durable record |
| SEND idempotency | Per logical action key on SEND; especially OTP; return original provider id on retry |
| State merge | Monotonic rank so DELIVRD before ENROUTE cannot demote status |
| Retries | Exponential backoff with jitter on re-drive; cap attempts; HTTP 500 transient, careful HTTP 429 |
| Timeouts vs validity | Align product expiry with SEND validity; define late DLR policy |
| Poison queue | Dead-letter permanent failures with raw payload and replay path through dedupe |
| Normalize addresses | E.164 up to 15 digits; consistent keys for correlation |
| Multipart rollup | Define segment aggregation for 160/153 and 70/67 part sizes |
| Logs and metrics | Correlate ids; count duplicates and auth failures; avoid claiming guaranteed delivery |
Frequently asked
- How do I deduplicate SMS delivery report webhooks?
- Persist a composite key of provider message id and state (or event type) before applying side effects. If the pair already exists, return HTTP 200 without re-processing. This matches at-least-once delivery where duplicates are expected after timeouts or retries.
- Why did I receive DELIVRD before ENROUTE on an SMS webhook?
- Providers do not guarantee total order. Use a monotonic state machine: only accept transitions that rank equal or higher than stored status, and refresh metadata on equal rank. That prevents a late intermediate status from overwriting a terminal one.
- How do idempotency keys prevent duplicate OTP SMS on retry?
- Send a unique idempotency key per logical challenge on SEND and store the key-to-message-id mapping. If the client times out and retries with the same key, return the original result instead of submitting a second message, which is a common double-send path for OTPs.