Route quality metrics: what to measure before trusting an SMS route
Before you put production OTPs or alerts on an SMS route, measure it. Delivery reports alone are not truth. This article defines the KPIs that matter—latency percentiles, final-delivery ratio, handset confirmation, sender-ID and content integrity, and drift—and how to observe each with a small handset test pool. Definitions only; no vendor scorecards.
Submit-to-DLR latency and why percentiles matter
Submit-to-DLR latency is the elapsed time from the moment your submit is accepted (HTTP 200 on a REST submit, or a successful SMPP submit_sm_resp) until you receive a final delivery receipt (DLR) for that message. Intermediate states are useful for debugging, but the clock that matters for user experience stops only on a final state. Record both wall-clock timestamps in your own systems; do not trust a provider-side duration field as your sole source of truth.
Averages hide the failure mode that breaks OTP and other time-bound flows. Report at least p50 and p95 on the same sample window. A route can show a calm median while a fat tail leaves a minority of users waiting far longer than the rest. That minority is exactly who abandons checkout or retries and doubles your traffic. Qualitatively, a healthy route shows a tight spread: p95 is not dramatically divorced from p50, and the shape is stable across repeated batches rather than spiking only when you look away.
How to measure with a small handset test pool: fix the destination set (E.164 numbers, max 15 digits), fix the payload class (for example a short OTP-shaped body within a single GSM 7-bit segment of 160 characters, or 153 when concatenated), submit in controlled bursts, and pair each provider message id with its final DLR timestamp. Exclude retries and non-final keepalives from the latency sample. Failure smells include a polite p50 paired with a runaway p95, long stretches with no final DLR at all, and latency that only looks fine when volume is trivial. HTTP 429 or 500 on submit is a different problem—capacity or platform health—and should be tallied separately from DLR delay.
SMSRoute accounts include free test credits that prove delivery before you pay (published), and delivery-testing tools are available in the SMSRoute dashboard.
Final-delivery ratio: DELIVERED over final states
Final-delivery ratio is the count of messages that end in a DELIVERED (or equivalent success) final state divided by the count of messages that have reached any final state. Non-final and unknown states stay out of the denominator until they resolve or you explicitly age them out under a documented timeout. This is not the same as delivered-over-submitted; the latter mixes in-flight traffic into the score and makes routes look worse or better depending on how slowly they close tickets.
Uniform 100 percent across every destination, every hour, and every content shape is itself a red flag. Real networks produce a mix of terminal outcomes—delivered, rejected, expired, unreachable, and related SMPP state ids—especially when you vary countries, sender types, and handset generations. A feed that never reports failure may be suppressing, rewriting, or synthesizing receipts. Qualitatively, good looks like a high success share among finals, accompanied by a credible minority of honest failures that you can correlate with known bad numbers or deliberate negative tests.
Measurement with a handset pool: send to devices you control plus a few deliberately invalid or unreachable addresses so you can verify that failure paths surface. Classify only terminal DLRs. Failure smells: perpetual 100 percent success, finals that arrive in lockstep with submit (no realistic network delay), missing failure classes you intentionally provoked, or state ids that never match the vocabulary your contract says you should receive.
Handset-confirmed rate: the only handset-level truth
Handset-confirmed rate is the number of messages actually observed on your test devices divided by the number you sent to those devices. DLRs are network or platform claims. A receipt on the handset—visible in the native message store or via a device-side logger you control—is the only handset-level truth you have. When DLR success and handset presence diverge, believe the handset and treat the DLR path as untrusted until explained.
How to measure: maintain a small, known pool of physical handsets or tightly controlled device endpoints across the operators you care about. Send identifiable payloads (unique tokens per message), then record whether each token appears, when it appears, and under which sender identity. Keep segments in mind: GSM 7-bit single parts top out at 160 characters (153 per part when concatenated); UCS-2 singles at 70 (67 per part when concatenated). Fragmentation errors often show up as handset-side partials that a coarse DLR still calls success.
Qualitatively, good means handset confirmations track final DELIVERED counts closely on the same destinations, with delays consistent with your submit-to-DLR percentiles. Failure smells: glowing DLRs while tokens never arrive; arrivals with altered bodies; systematic misses on one operator inside the pool; or confirmation only when you send trivial traffic and collapse when you raise concurrency. Handset-confirmed rate will always be sample-bound—you cannot put a phone on every subscriber—so treat it as ground truth for the sample, not a census of the globe.
SMSRoute publishes per-country rates openly, so the cost side of any route comparison is checkable against dated sources.
| Metric | What you count | Qualitative good | Failure smell |
|---|---|---|---|
| Submit-to-DLR latency | Accept time to final DLR; report p50 and p95 | Tight spread; p95 not wildly above p50 | Fine median, terrible tail; stalled finals |
| Final-delivery ratio | DELIVERED finals / all finals | High success with credible real failures | Uniform 100 percent; no failure classes |
| Handset-confirmed rate | Device receipts / sent to pool | Tracks DLR success on same numbers | DLR success, silent handsets |
| Sender-ID preservation | Expected sender still shown / sent | Stable identity on target operators | Rewrites to short codes or unknown alphas |
| Content-integrity rate | Unmodified template received / sent | Token and wording intact end-to-end | Truncation, injection, re-encoding damage |
| Drift | Same suite re-run over calendar time | Stable shape across re-measures | Sudden tail growth or silent ratio shifts |
Sender-ID preservation and content-integrity rate
Sender-ID preservation rate is the share of handset-confirmed messages where the from-identity the user sees matches the identity you intended for that route and destination. Content-integrity rate is the share where the received body matches the submitted template bit-for-bit at the application layer after accounting for the character set you chose—no truncation, no prefix or suffix injection, no re-encoding that corrupts the OTP token, and no silent replacement of characters outside the negotiated alphabet.
Measure both only on messages you have already handset-confirmed; a missing message is a delivery problem, not a preservation problem. Log the expected sender, the observed sender, the expected body hash or exact string, and the observed body. Run the same templates as production: OTP shapes, alert shapes, and any concatenated forms you actually use (respecting 160/153 and 70/67 part sizes). Vary alpha senders and numeric senders if both appear in your product.
Qualitatively, good is boring: the brand or number you configured is what the device shows, and the token string is intact so your verifier does not need fuzzy matching. Failure smells include systematic rewrite to a generic short code, country-dependent stripping of alphas, marketing trailers appended to transactional bodies, UCS-2/GSM mismatches that break digits or URLs, and multipart reassembly that drops a middle segment while the DLR still says delivered. Any of these can pass a naive delivered-over-submitted check and still destroy conversion.
Drift: routes change without notice
Drift is the change in the above metrics when you re-run the same measurement suite over time on the same route product. Upstream suppliers, operator filters, sender registrations, and traffic-mix policies shift without a pull request in your repo. A route that was acceptable at onboarding can degrade in tail latency, begin rewriting sender IDs, or start returning optimistic finals that no longer match handsets.
How to measure: freeze the test design—pool, templates, submit rate, classification rules—and schedule repeats. Compare p50/p95, final-delivery ratio, handset-confirmed rate, sender-ID preservation, and content-integrity as time series, not as one-off screenshots. Qualitatively, good is a stable shape with explainable blips; you still investigate blips, but the baseline holds. Failure smells are step-changes after quiet periods, slow creep in p95 while p50 looks fine, rising DLR-to-handset gaps, or integrity failures that appear only on certain days of week when intermediary load changes.
Treat every metric as perishable. The point of drift tracking is not to produce a dashboard trophy; it is to notice that trust has expired before your users do.
Caveat: one green test is not a contract with the future
A single green test proves that a route worked once, for those destinations, under that load, with that content, on that day. It does not prove that the route will work for production mix, peak concurrency, or next month’s operator policy. Measurement is a habit, not an event. Keep the handset pool small but real, keep definitions fixed so comparisons mean something, and refuse to collapse p95 into a single average that flatters the median.
SMSRoute accounts include free test credits that prove delivery before you pay (published), and delivery-testing tools are available in the SMSRoute dashboard. Use those credits to exercise the definitions in this article—latency percentiles, final-delivery ratio, handset confirmation, sender-ID and content checks—then keep re-measuring. Uncertainty never goes to zero; the goal is to make blind trust unnecessary.
Frequently asked
- Why is p95 submit-to-DLR latency more important than the median for OTP routes?
- OTP and other time-bound messages fail for users in the tail. A route can show a fine p50 while p95 leaves enough recipients waiting that they abandon or retry. Measure both on the same window from submit accept (HTTP 200 or successful SMPP submit_sm_resp) to final DLR; a tight spread is qualitatively healthier than a calm median alone.
- Why is a 100% SMS final-delivery ratio a red flag?
- Final-delivery ratio is DELIVERED finals divided by all final states. Real networks produce mixed terminal outcomes when you vary destinations and include deliberate negative tests. Uniform 100 percent across conditions can indicate suppressed, rewritten, or synthesized receipts rather than honest reporting.
- What is handset-confirmed rate and how do you measure it?
- It is test-device receipts divided by messages sent to those devices—the only handset-level truth. Send unique tokens to a small controlled handset pool, observe whether each token arrives intact and under the expected sender, and compare that to DLR success. DLRs that disagree with the handset should be treated as untrusted until explained, and the suite should be re-run over time to catch drift.