The attack you find on your invoice

Finance forwarded a carrier invoice triple the usual size. The SMS provider dashboard showed a clean spike starting 02:14 UTC. Application logs told the rest: thousands of POST /v1/auth/otp/send calls, each body a different MSISDN, almost no follow-up POST /v1/auth/otp/verify.

The numbers themselves were the signature. Sorted, they looked like this:

+251911000001
+251911000002
+251911000003
+251911000004
...

One request per number. No retries on the same MSISDN. Zero verify attempts against any of them. That pattern is SMS pumping: a bot feeds premium-route destinations into an unauthenticated (or weakly authenticated) "send code" endpoint. The terminating network shares revenue on those messages; the attacker and the terminating party split the inflated traffic. Your budget is the third party in that arrangement.

If your send endpoint accepts any E.164 string and immediately hands it to the provider, you are the open hose.

Layer 1 — per-destination throttling with a real algorithm

Cap how often any single destination can receive a code. A token bucket works: capacity 1, refill one token every 60 seconds, plus a hard daily ceiling so a patient bot cannot drip forever.

Naive GET then SET is racy under concurrency — two requests read tokens=1, both decrement, both send. Use an atomic path. Redis Lua is the portable one:

import redis
import time

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

# KEYS[1] = bucket key for this MSISDN
# ARGV[1] = now (ms), ARGV[2] = capacity, ARGV[3] = refill_ms
# ARGV[4] = daily key, ARGV[5] = daily_cap, ARGV[6] = day_ttl_sec
TOKEN_BUCKET_LUA = """
local key = KEYS[1]
local now = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local refill_ms = tonumber(ARGV[3])
local day_key = ARGV[4]
local day_cap = tonumber(ARGV[5])
local day_ttl = tonumber(ARGV[6])

local day_count = tonumber(redis.call('GET', day_key) or '0')
if day_count >= day_cap then
  return {0, -1, day_count}
end

local data = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(data[1])
local ts = tonumber(data[2])

if tokens == nil then
  tokens = capacity
  ts = now
end

local elapsed = math.max(0, now - ts)
local refill = math.floor(elapsed / refill_ms)
tokens = math.min(capacity, tokens + refill)
if refill > 0 then
  ts = ts + refill * refill_ms
end

if tokens < 1 then
  local wait = refill_ms - (now - ts)
  return {0, wait, day_count}
end

tokens = tokens - 1
redis.call('HMSET', key, 'tokens', tokens, 'ts', ts)
redis.call('PEXPIRE', key, capacity * refill_ms * 2)

day_count = redis.call('INCR', day_key)
if day_count == 1 then
  redis.call('EXPIRE', day_key, day_ttl)
end

return {1, 0, day_count}
"""

token_bucket = r.register_script(TOKEN_BUCKET_LUA)

def allow_send(msisdn: str) -> tuple[bool, int]:
    now_ms = int(time.time() * 1000)
    day_key = f"otp:day:{time.strftime('%Y%m%d')}:{msisdn}"
    allowed, wait_ms, _ = token_bucket(
        keys=[f"otp:bucket:{msisdn}"],
        args=[now_ms, 1, 60_000, day_key, 5, 90_000],
    )
    return bool(allowed), int(wait_ms)

Call allow_send before you touch the provider. On allowed=False, return 429 and do not enqueue an SMS. The INCR+EXPIRE daily counter and the hash update live in one script evaluation, so concurrent workers cannot double-spend a token.

Layer 2 — resend cooldown with exponential step-up

Per-destination limits stop fan-out. Resend abuse on one number still burns budget and annoys a real user. Track a verification session (server-side id bound to the MSISDN after the first accepted send) and step the cooldown: 30s, then 60s, then 300s.

import math
from flask import Flask, request, jsonify

app = Flask(__name__)

COOLDOWNS = [30, 60, 300]  # seconds

def next_cooldown(attempt_index: int) -> int:
    return COOLDOWNS[min(attempt_index, len(COOLDOWNS) - 1)]

@app.post("/v1/auth/otp/send")
def send_otp():
    body = request.get_json(force=True)
    msisdn = body["msisdn"]
    session_id = body.get("session_id")  # issued after first send

    ok, wait_ms = allow_send(msisdn)
    if not ok:
        resp = jsonify({"error": "rate_limited", "scope": "destination"})
        resp.status_code = 429
        resp.headers["Retry-After"] = str(max(1, math.ceil(wait_ms / 1000)))
        return resp

    if session_id:
        attempt = int(r.get(f"otp:sess:attempts:{session_id}") or "0")
        last = r.get(f"otp:sess:last:{session_id}")
        if last is not None:
            elapsed = time.time() - float(last)
            need = next_cooldown(attempt - 1 if attempt > 0 else 0)
            if elapsed < need:
                retry = int(math.ceil(need - elapsed))
                resp = jsonify({"error": "rate_limited", "scope": "resend"})
                resp.status_code = 429
                resp.headers["Retry-After"] = str(retry)
                return resp
        pipe = r.pipeline()
        pipe.incr(f"otp:sess:attempts:{session_id}")
        pipe.set(f"otp:sess:last:{session_id}", str(time.time()), ex=3600)
        pipe.execute()

    # ... generate code, store hash, dispatch SMS, mint session_id on first send ...
    return jsonify({"session_id": session_id or "new-session-id"}), 200

The disabled "Resend" button is UX. The server enforces the timer and speaks 429 with Retry-After so honest clients back off without spinning.

Layer 3 — cost-centre guards

Pumping clusters in a handful of country prefixes. Per-MSISDN buckets alone will still let a bot walk a prefix at one-message-per-number pace. Add three controls above the destination bucket.

Prefix velocity. Maintain a sliding window counter per E.164 prefix (first 4–6 digits after +). Alarm and auto-throttle when sends/minute for that prefix exceed a baseline multiple of your real user traffic.

Global circuit breaker for unauthenticated send. Fail CLOSED: when tripped, unauthenticated /otp/send returns 503 immediately and never reaches the provider. Authenticated recovery flows can stay open on a separate path if you need them.

Prefix allow/deny policy. If you have zero users in a country, deny that prefix at the edge. Keep the list in config, not hardcoded in the handler.

from enum import Enum

class BreakerState(Enum):
    CLOSED = "closed"       # normal
    OPEN = "open"           # blocking unauthenticated sends
    HALF_OPEN = "half_open" # probe

class SendCircuitBreaker:
    def __init__(self, redis_client, threshold_per_min=120, open_for_sec=300):
        self.r = redis_client
        self.threshold = threshold_per_min
        self.open_for = open_for_sec

    def _state_key(self):
        return "otp:breaker:state"

    def record_send(self) -> None:
        key = f"otp:global:{int(time.time() // 60)}"
        count = self.r.incr(key)
        if count == 1:
            self.r.expire(key, 120)
        if count >= self.threshold:
            self.trip()

    def trip(self) -> None:
        self.r.set(self._state_key(), BreakerState.OPEN.value, ex=self.open_for)

    def allow_unauthenticated(self) -> bool:
        state = self.r.get(self._state_key()) or BreakerState.CLOSED.value
        if state == BreakerState.OPEN.value:
            ttl = self.r.ttl(self._state_key())
            if ttl is not None and ttl <= 0:
                self.r.set(self._state_key(), BreakerState.HALF_OPEN.value, ex=30)
                return True  # single probe window
            return False
        return True

    def probe_success(self) -> None:
        self.r.set(self._state_key(), BreakerState.CLOSED.value)

    def probe_failure(self) -> None:
        self.trip()

DENIED_PREFIXES = ("+2519", "+23480", "+22507")  # example: regions you do not serve

def prefix_allowed(msisdn: str) -> bool:
    return not any(msisdn.startswith(p) for p in DENIED_PREFIXES)

Wire prefix_allowed and allow_unauthenticated before allow_send. Whatever guards you build application-side, mirror the ceiling at the provider: the HTTP SMS API I use lets me cap outbound volume per key, which turned a 3 a.m. pumping attempt into a bounded few-dollar incident instead of an invoice conversation.

Layer 4 — make requests expensive before they reach SMS

Do not put a CAPTCHA on every send. Spend friction only when velocity rules say you are under attack: prefix counter above baseline, breaker half-open, or global 429 rate climbing.

When those signals fire, require an invisible challenge or a small proof-of-work on the request-code form only. Bind the solved challenge to a server session id, and bind that session to at most N distinct MSISDNs (N=1 is fine for login OTP). A bot that solved one challenge then cannot fan the same cookie across ten thousand numbers without solving again.

def session_may_target(session_id: str, msisdn: str, max_numbers: int = 1) -> bool:
    key = f"otp:sess:numbers:{session_id}"
    already = r.sismember(key, msisdn)
    if already:
        return True
    size = r.scard(key)
    if size >= max_numbers:
        return False
    pipe = r.pipeline()
    pipe.sadd(key, msisdn)
    pipe.expire(key, 3600)
    pipe.execute()
    return True

Under calm traffic the path stays one round-trip. Under pumping, compute and session cardinality become the tax, not a permanent UX tax on every real user.

Don't leak account existence while you're at it

The same send endpoint is often an oracle: {"error":"no_account"} for unknown numbers and {"ok":true} for known ones. Enumerators and pumpers both love that. Respond with the same body, same status, and comparable timing whether the number is registered or not. Always run the throttle layers first; only then branch into "create pending registration" versus "login OTP" behind a constant response shape.

If you must avoid sending SMS to unknown numbers for product reasons, still return the identical payload and schedule a no-op delay that matches the provider's p50 latency so timing does not re-open the oracle.

Verify-side limits

A code that is never checked is a pure cost event. Give verify its own bucket.

• 5 attempts per code, then invalidate the code.
• Constant-time comparison (hmac.compare_digest in Python, timingSafeEqual in Node's standard library).
• Single-use: delete or rotate on first success.
• TTL 5–10 minutes (RFC 6238-style windows are a reasonable default for SMS OTP).

import hmac
import hashlib
import secrets

def store_code(session_id: str, code: str, ttl_sec: int = 600) -> None:
    digest = hashlib.sha256(code.encode("utf-8")).hexdigest()
    r.setex(f"otp:code:{session_id}", ttl_sec, digest)
    r.setex(f"otp:code:tries:{session_id}", ttl_sec, 0)

def verify_code(session_id: str, candidate: str) -> bool:
    tries_key = f"otp:code:tries:{session_id}"
    tries = int(r.incr(tries_key))
    if tries == 1:
        r.expire(tries_key, 600)
    if tries > 5:
        r.delete(f"otp:code:{session_id}")
        return False
    expected = r.get(f"otp:code:{session_id}")
    if not expected:
        return False
    got = hashlib.sha256(candidate.encode("utf-8")).hexdigest()
    ok = hmac.compare_digest(got, expected)
    if ok:
        r.delete(f"otp:code:{session_id}", tries_key)
    return ok

Track send_count / verify_count per prefix over a rolling window. A prefix with hundreds of sends and near-zero verifies is your cleanest automated detection signal — page on that ratio before finance pages on the invoice.

Incident runbook

When the spend graph bends up, execute in order. Do not start with root-cause analysis.

1. Flip the global unauthenticated send breaker to OPEN. Confirm /otp/send returns 503 without provider calls.
2. Snapshot the last 60 minutes of send logs (MSISDN, prefix, IP, session id, timestamp) to cold storage before retention rotates them.
3. Identify the top prefixes by send volume; add them to DENIED_PREFIXES or the provider-side block list immediately.
4. Lower the global per-minute threshold to a known-safe floor so HALF_OPEN probes cannot reopen the hose.
5. Invalidate in-flight OTP sessions created during the spike window.
6. Verify provider-side key caps are still enforced; tighten if the application breaker lagged.
7. Only then inspect bot signatures (UA, IP ASN, sequential MSISDN ranges) and write the permanent prefix/velocity rules.
8. Reopen the breaker in HALF_OPEN with a single canary prefix you actually serve; watch send:verify ratio for 15 minutes before CLOSED.

Monitoring dashboard spec

Graph these five series on one screen with 1-minute resolution:

Checklist to ship before the next spike:

• [ ] Atomic per-MSISDN token bucket (Lua or equivalent), daily cap included
• [ ] Server-side resend cooldown with 429 + Retry-After
• [ ] Prefix velocity counters + deny list for regions you do not serve
• [ ] Global fail-CLOSED breaker on unauthenticated send
• [ ] Challenge/PoW only when velocity rules trip; session-to-MSISDN cardinality bind
• [ ] Identical responses (body + timing) for known and unknown numbers
• [ ] Verify bucket: 5 tries, constant-time compare, single-use, 5–10 min TTL
• [ ] send:verify ratio alert per prefix
• [ ] Provider-side volume cap mirroring app limits
• [ ] Runbook steps 1–8 posted where on-call actually looks