A/B testing SMS content against filtering: method, not folklore

Carrier filtering on SMS is content-sensitive and opaque. Honest A/B tests isolate one content variable while freezing destination, route, and sender identity. Compare handset-confirmed receipt—not delivery reports—on a small shared handset pool, report binary pass/fail, and re-test when routes change. This article is method, not folklore.

Hold destination, route, and sender constant

Content A/B testing only means something if every non-content factor is identical across arms. Fix the destination handsets, the outbound route, and the sender identity (the same alphanumeric or short code on every arm). If any of those move, you are no longer testing message wording—you are testing a bundle of operational differences you cannot attribute.

Vary exactly one content element per arm. Useful single factors include: presence versus absence of a link; shortened URL versus full URL; discount-style wording versus neutral status wording; sender name repeated in the body versus body-only transactional text; and opt-out phrasing variants. Do not stack two of those changes in the same arm. Stacking produces a result you cannot map back to a cause, which is how folklore spreads.

Send every arm to the same small handset pool. Same SIMs, same devices, same networks you care about. Rotating destinations between arms reintroduces handset and network variance and quietly ruins the comparison. SMSRoute accounts include free test credits that prove delivery before you pay, and delivery-testing tools are available in the SMSRoute dashboard for controlled handset checks.

SMSRoute accounts include free test credits that prove delivery before you pay (published), and delivery-testing tools are available in the SMSRoute dashboard.

Compare handset-confirmed receipt, not DLRs

A delivery report (DLR) is a signal from the messaging path, not proof that a human device displayed the SMS. Paths can emit success-looking DLRs when the handset never received the message, and failure-looking signals when the device actually got it. Treat DLRs as operational telemetry. For filter testing, only handset-confirmed receipt counts: the device shows the message, or it does not.

That is why arms must land on a pool you can inspect—physical devices, instrumented test handsets, or dashboard delivery-testing tools that surface handset-side outcome. If you cannot confirm receipt on the device, you do not have a content-filter observation. Logging a DLR status code and calling it a filter test is how teams invent confidence they did not earn. Prefer binary device outcomes over any derived “delivery rate” built from reports alone.

Sample-size honesty: large effects only

With a handful of test numbers you detect only large effects. Say that plainly in any write-up. A pool of a few handsets can show “arm A arrived on these devices and arm B did not,” or “both arrived,” or “neither did.” It cannot support precise percentages, lift claims, or fine-grained ranking of mild wording tweaks. Report binary pass/fail per arm on the shared pool, not fake percentages calculated from tiny n.

If arm A is received on the pool and arm B is not, you have evidence of a large content-sensitive difference under that route and sender. If both pass or both fail, you have no separable content signal at that scale—stop, do not dress the result as a rate. Expanding the pool helps only if destinations stay matched across arms and you still refuse to over-interpret noise. Uncertainty is part of the method; hiding it is not.

SMSRoute's test-matrix guide covers the destination and sender axes this article holds constant.

Common confounds that fake a content result

Different send times across arms are a classic confound. Carrier and handset behavior can vary by time of day and load; staggered sends turn a content test into a timing test. Send arms as close together as your pipeline allows, under the same route conditions.

Warmed versus cold sender identity is another. A sender that has been steadily used on a route is not interchangeable with a fresh identity even when the text is identical. Keep sender warm-up state constant across arms or you will attribute reputation effects to wording. Mixed encodings are a third trap: introducing a character that forces UCS-2 can change segment counts mid-test, alter parts-and-concat behavior, and change how filters and handsets handle the payload. Keep encoding and segment structure intentional and identical unless encoding itself is the single factor under test.

Any mid-test change to template concatenation, language, or hidden unicode is a second variable. If you discover an encoding shift after the fact, invalidate the run and redo it with encoding locked.

When to re-test

Routes change silently. Downstream suppliers, filtering rules, and sender reputation on a path are not frozen because your template repository is. Re-test when you change route, sender, or traffic pattern; after sustained delivery anomalies on production templates that previously passed; and on a deliberate cadence if the traffic matters enough that silent path drift is unacceptable.

A past pass is not a permanent certificate. Treat each confirmed configuration—content arm, route, sender, destination set—as valid until one of those inputs moves. When it moves, run the same binary handset protocol again rather than assuming last month’s wording still clears.

Worked two-arm example

Setup: one route, one sender identity, same three handsets in the pool, sends back-to-back. Single factor: discount-style wording versus neutral order status. Encoding locked to GSM-safe characters; no URL in either arm; opt-out line identical. Outcome rule: pass only if every handset in the pool shows the message; otherwise fail for that arm. No percentages.

Arm N (neutral) uses order-id style status text. Arm D (discount wording) keeps the same order id and structure but inserts promotional phrasing. After send, inspect devices—not DLR codes. Suppose the pool receives Arm N on all handsets and does not receive Arm D on the pool: record N = pass, D = fail. That is a large-effect content signal under this route and sender, nothing more. If both pass, you learned this particular wording contrast was not a hard filter trigger on this path today. Either way, document route, sender, handset set, timestamps, and encoding so the run can be repeated when the path changes.

Frequently asked

Why ignore DLR success when A/B testing SMS content filters?

DLRs report path signalling, not handset display. Success-looking reports can arrive when the device never showed the message. For filter tests, only handset-confirmed receipt on a shared device pool is valid evidence.

How many content changes should each SMS A/B arm include?

Exactly one. Hold destination, route, and sender fixed, and change a single element such as link presence, URL form, discount versus neutral wording, in-body sender name, or opt-out phrasing. Multiple changes in one arm make the result unattributable.

When should I re-run an SMS content filter test that already passed?

Re-test when route, sender, or traffic pattern changes, after delivery anomalies on templates that previously passed, and on a deliberate cadence if silent path drift would matter. Prior passes are not permanent.

Related

• Designing an SMS test matrix
• Fake DLRs explained
• Deliverability checklist