The bug report that starts it all

Support pasted this payload from a production OTP template:

Your code is 123456 — don't share it

Length check in the admin UI: 38 characters. Expected cost: 1 segment. Provider invoice: 2 segments per send. Volume was high enough that the discrepancy showed up as a real line item within a week.

The straight-looking string is not GSM-7. It contains U+2014 EM DASH (—) and U+2019 RIGHT SINGLE QUOTATION MARK (' in don't). Neither code point exists in the GSM-7 basic or extension tables. One out-of-table character forces the entire message into UCS-2 (UTF-16BE on the wire). The single-segment cap drops from 160 septets to 70 code units. Thirty-eight UCS-2 characters still fit in one segment — until marketing prepends a brand prefix and a URL, the total crosses 70, and you pay for two.

Byte-level view of the first offending character:

—  U+2014  →  UTF-16BE  20 14     (2 bytes, not a GSM septet)
'  U+2019  →  UTF-16BE  20 19     (2 bytes, not a GSM septet)

If those two glyphs had been ASCII hyphen-minus U+002D and apostrophe U+0027, the message would have stayed GSM-7 at 38 septets, one segment. The rest of this article is the exact math that turns “looks fine in the editor” into “3× billing.”

GSM-7 in 5 minutes: septets, not bytes

GSM-7 packs characters into 7-bit units called septets. A 140-byte SMS user-data field holds:

140 bytes × 8 bits = 1120 bits
1120 / 7 = 160 septets

So the familiar “160 characters” number is 160 septets, not 160 bytes.

Most printable ASCII plus a handful of Latin-1 letters live in the basic table and cost 1 septet each: A–Z, a–z, 0–9, space, !, ?, *, #, ä, ö, ñ, ü, à, Æ, ß, and so on.

A second extension table is reached by the escape septet 0x1B. Each extended character therefore costs 2 septets:

€  [  ]  {  }  \  ~  ^  |  and form feed

Common trap: a price string "Total: 12€". The € alone burns two septets. A template that was sized to 160 basic characters silently overflows when someone localizes the currency symbol.

Bit packing is why you cannot reason about SMS length with Buffer.byteLength(str, 'utf8'). UTF-8 byte length and GSM-7 septet length diverge as soon as an escape or a non-GSM code point appears.

When the encoding flips to UCS-2

The rule is absolute: if any character in the message is outside both GSM-7 tables, the full payload is re-encoded as UCS-2. Every character then occupies 16 bits. Capacity becomes:

140 bytes / 2 = 70 characters per single-segment message

No partial mode exists. One emoji, one smart quote, one en-dash — the whole message pays the UCS-2 tax.

Culprits injected daily by CMSes, Google Docs, and macOS “smart punctuation”:

Detection regex you can drop into a pre-send lint:

// Matches common smart punctuation + anything outside GSM-7-ish ASCII range
// for a quick reject/flag. Full classification is in the counter below.
const SMART_PUNCT = /[\u2018\u2019\u201C\u201D\u2013\u2014\u2026\u00A0]/;

function hasNonGsmHint(text) {
  if (SMART_PUNCT.test(text)) return true;
  // Emoji and most non-Latin scripts fall outside basic Latin + Latin-1 supplement used by GSM
  for (const ch of text) {
    const cp = ch.codePointAt(0);
    if (cp > 0xff) return true;
  }
  return false;
}

hasNonGsmHint is a fast tripwire, not a segment counter. Segment math needs the full tables.

Concatenation and the UDH tax

When a message exceeds one segment, the SMPP/SS7 stack splits it and attaches a User Data Header (UDH) so the handset can reassemble parts. The standard concatenation UDH is 6 bytes (length, IEI, IEL, ref, total, seq). Those 6 bytes steal space from the user-data field:

GSM-7 multipart:  (140 - 6) × 8 / 7 = 153.xx → 153 septets per segment
UCS-2 multipart:  (140 - 6) / 2           → 67 code units per segment

Cliff table (this is where invoices jump):

161 GSM-7 characters → 2 segments. 71 UCS-2 characters → 2 segments. There is no soft landing.

Segment-count formula, including the extension-character double-count:

function septetCount(text, basic, extended) {
  let septets = 0;
  for (const ch of text) {
    if (basic.has(ch)) septets += 1;
    else if (extended.has(ch)) septets += 2;
    else return null; // forces UCS-2 path
  }
  return septets;
}

function segmentCountFromSeptets(septets) {
  if (septets <= 160) return 1;
  return Math.ceil(septets / 153);
}

function segmentCountFromUcs2(chars) {
  if (chars <= 70) return 1;
  return Math.ceil(chars / 67);
}

A correct segment counter you can paste

Full classifier. Paste into Node 18+ and run.

const GSM7_BASIC = new Set([
  '@','£','$','¥','è','é','ù','ì','ò','Ç','\n','Ø','ø','\r',
  'Å','å','Δ','_','Φ','Γ','Λ','Ω','Π','Ψ','Σ','Θ','Ξ',
  'Æ','æ','ß','É',' ','!','"','#','¤','%','&','\'','(',')',
  '*','+',',','-','.','/','0','1','2','3','4','5','6','7',
  '8','9',':',';','<','=','>','?','¡',
  'A','B','C','D','E','F','G','H','I','J','K','L','M','N',
  'O','P','Q','R','S','T','U','V','W','X','Y','Z',
  'Ä','Ö','Ñ','Ü','§','¿',
  'a','b','c','d','e','f','g','h','i','j','k','l','m','n',
  'o','p','q','r','s','t','u','v','w','x','y','z',
  'ä','ö','ñ','ü','à'
]);

// Each of these costs 2 septets (ESC 0x1B + code)
const GSM7_EXTENDED = new Set([
  '\f','^','{','}', '\\','[','~',']','|','€'
]);

/**
 * @param {string} text
 * @returns {{ encoding: 'GSM-7'|'UCS-2', segments: number,
 *             charsRemaining: number, septets: number|null, length: number }}
 */
function smsSegments(text) {
  let septets = 0;
  let gsm = true;

  for (const ch of text) {
    if (GSM7_BASIC.has(ch)) {
      septets += 1;
    } else if (GSM7_EXTENDED.has(ch)) {
      septets += 2;
    } else {
      gsm = false;
      break;
    }
  }

  const length = [...text].length; // code points, not UTF-16 code units

  if (gsm) {
    const segments = septets <= 160 ? 1 : Math.ceil(septets / 153);
    const capacity = segments === 1 ? 160 : segments * 153;
    return {
      encoding: 'GSM-7',
      segments,
      charsRemaining: capacity - septets,
      septets,
      length
    };
  }

  const segments = length <= 70 ? 1 : Math.ceil(length / 67);
  const capacity = segments === 1 ? 70 : segments * 67;
  return {
    encoding: 'UCS-2',
    segments,
    charsRemaining: capacity - length,
    septets: null,
    length
  };
}

// --- edge-case tests ---
function assert(cond, msg) {
  if (!cond) throw new Error(msg);
}

// exactly 160 basic chars → 1 segment
assert(smsSegments('a'.repeat(160)).segments === 1, '160 basic');
assert(smsSegments('a'.repeat(160)).encoding === 'GSM-7', '160 enc');

// 161 basic → 2 segments
assert(smsSegments('a'.repeat(161)).segments === 2, '161 basic');

// € costs 2 septets: 159 basic + € = 161 septets → 2 segments
const euroBoundary = 'a'.repeat(159) + '€';
assert(smsSegments(euroBoundary).septets === 161, '€ septet count');
assert(smsSegments(euroBoundary).segments === 2, '€ segments');

// € at position 152 of an otherwise 160-char attempt:
// 151 basic + €(2) + 8 basic = 161 septets → 2 segments
const euro152 = 'a'.repeat(151) + '€' + 'a'.repeat(8);
assert(smsSegments(euro152).septets === 161, '€@152 septets');
assert(smsSegments(euro152).segments === 2, '€@152 segments');

// one emoji → UCS-2, 1 segment if total code points ≤ 70
assert(smsSegments('hello 😀').encoding === 'UCS-2', 'emoji enc');
assert(smsSegments('hello 😀').segments === 1, 'emoji seg');

// 71 UCS-2 chars → 2 segments
assert(smsSegments('你'.repeat(71)).segments === 2, '71 ucs2');
assert(smsSegments('你'.repeat(70)).segments === 1, '70 ucs2');

// original bug payload
const bug = "Your code is 123456 — don't share it";
assert(smsSegments(bug).encoding === 'UCS-2', 'bug enc');

console.log('all tests passed');
console.log(smsSegments(bug));
console.log(smsSegments(euroBoundary));

Run it:

node sms_segments.js

If any assertion fires, the tables or the UDH constants are wrong — fix before shipping the helper to production.

Sanitizing without mangling

Transliterate only the punctuation that never carries linguistic meaning:

const PUNCT_MAP = {
  '\u2018': "'", '\u2019': "'",
  '\u201C': '"', '\u201D': '"',
  '\u2013': '-', '\u2014': '-',
  '\u2026': '...',
  '\u00A0': ' ' // nbsp → space
};

function sanitizePunct(text) {
  return text.replace(
    /[\u2018\u2019\u201C\u201D\u2013\u2014\u2026\u00A0]/g,
    (ch) => PUNCT_MAP[ch]
  );
}

Apply sanitizePunct before smsSegments. The OTP example becomes:

Your code is 123456 - don't share it

GSM-7, 38 septets, 1 segment.

Do not strip or force-Latin user language. Turkish İ / ı, Greek, Cyrillic, CJK — those characters are intentional. Folding them to ASCII to save a segment corrupts the message. Detect, measure, and either accept UCS-2 cost or ask the product owner for a shorter copy in that locale.

NFKC normalization looks tempting ("fi".normalize('NFKC') === "fi") but is unsafe as a blanket pre-pass. It can change Turkish dotted/dotless I behaviour and will decompose characters your legal copy deck required to stay composed. If you normalize, do it only on the punctuation set you control, not on the full string.

What the API actually tells you

Never trust only your local counter for billing reconciliation. Providers recompute segments against their own table revision and will bill what their SMSC emits. Read the segment count from the send response body, log it next to your local estimate, and diff them.

On the HTTP SMS API I use the send response includes the computed segment count, so I export it as a Prometheus metric and page when a deploy suddenly doubles it.

Practical wiring:

// after a successful send
const { segments } = providerResponse; // field name varies
metrics.smsSegments.observe(segments);
metrics.smsSegmentsPerBody.labels(templateId).observe(segments);

// alert: p95(segments) by template jumps after deploy
// e.g. from 1.0 → 2.0 within 15 minutes of a content change

A p95 jump from 1.0 to 2.0 on a single template ID almost always means smart quotes or an emoji landed in the copy. Pair the alert with the pre-send hasNonGsmHint lint in CI so the deploy never ships.

Cheat sheet

Four numbers

Extension characters (2 septets each)
€ [ ] { } \ ~ ^ | and form feed (\f)

Detection

/[\u2018\u2019\u201C\u201D\u2013\u2014\u2026\u00A0]/

Rules of thumb

• One non-GSM character → whole message UCS-2.
• 161 GSM-7 septets or 71 UCS-2 chars → instant 2× cost.
• € in a 159-character body → 161 septets → 2 segments.
• Sanitize curly quotes and dashes; never transliterate real language text.
• Log provider-reported segments; alert on p95 regression per template.