Chapter 3.3☕ 20 min read

Strings: UTF-16, Ropes & Grapheme Clusters

🙂 ek character hai — phir bhi .length 2 kyun? Aaj pata chalega bhai.

0101 — UTF-16: Why JS Uses 2 Bytes Per Character

Unicode is a mapping of numbers (code points) to characters — U+0041 = 'A', U+03A9 = 'Ω', U+1F642 = '🙂'. The full Unicode range goes from U+0000 to U+10FFFF — that's 1,114,112 possible code points.

The Basic Multilingual Plane (BMP) covers U+0000 to U+FFFF — most common scripts (Latin, Greek, Cyrillic, Arabic, Chinese, Japanese, Korean) all fit here. That's only 65,536 code points, enough for 2 bytes. Supplementary planes (U+10000 to U+10FFFF) contain emoji, rare scripts, ancient languages, and musical symbols.

UTF-16 encoding: BMP characters = 2 bytes (1 code unit). Supplementary characters = 4 bytes (2 code units, called a surrogate pair).

JavaScript chose UTF-16 in 1995 when Unicode was still only the BMP — the 4-byte extension came later. The result: JS strings are sequences of UTF-16 code UNITS (2 bytes each), not characters. String.length counts code UNITS, not characters or code points.

// ASCII / BMP characters — 1 code unit (2 bytes)
console.log('A'.length);           // 1 — U+0041, one code unit
console.log('Ω'.length);           // 1 — U+03A9, still BMP
console.log('中'.length);          // 1 — U+4E2D, Chinese character, BMP

// Supplementary plane — emoji, needs 2 code units (4 bytes)
console.log('🙂'.length);          // 2 — U+1F642, outside BMP!
console.log('👨‍👩‍👧‍👦'.length);    // 11 — family emoji = multiple code points!
console.log('🏴󠁧󠁢󠁳󠁣󠁴󠁿'.length);   // huge — flag emoji with tag characters!

// Code point vs code unit
const smile = '🙂';
console.log(smile.length);         // 2 — two UTF-16 code units
console.log([...smile].length);    // 1 — one actual character (using iterator)
console.log(smile.codePointAt(0)); // 128578 — the actual Unicode code point
console.log(smile.charCodeAt(0));  // 55357 — first surrogate (broken!)
console.log(smile.charCodeAt(1));  // 56898 — second surrogate (broken!)

V8 internally uses two string encodings: Latin-1 (1 byte per char) for pure ASCII strings, and UTF-16 (2 bytes per char) for everything with non-ASCII characters. When you concatenate a Latin-1 string with a UTF-16 string, V8 must upconvert the Latin-1 string to UTF-16 — that's a hidden allocation.

0202 — Surrogate Pairs: The 4-Byte Hack

Supplementary plane code points (U+10000 to U+10FFFF) don't fit in 2 bytes. The Unicode consortium's solution: encode one code point using TWO 2-byte code units — a surrogate pair.

High surrogate: U+D800 to U+DBFF (1024 values). Low surrogate: U+DC00 to U+DFFF (1024 values). Together: 1024 × 1024 = 1,048,576 supplementary code points encodable. The formula: codePoint = (high - 0xD800) × 0x400 + (low - 0xDC00) + 0x10000

A lone surrogate is a high or low surrogate without its pair — a malformed string. JS allows lone surrogates; other languages/specs don't.

// Manually inspect a surrogate pair
const emoji = '🙂'; // U+1F642

// Get the two code units (surrogates)
const high = emoji.charCodeAt(0); // 55357 = 0xD83D (high surrogate)
const low  = emoji.charCodeAt(1); // 56898 = 0xDE42 (low surrogate)

console.log(high.toString(16)); // d83d — high surrogate range D800-DBFF
console.log(low.toString(16));  // de42 — low surrogate range DC00-DFFF

// Reconstruct code point from surrogate pair:
const codePoint = (high - 0xD800) * 0x400 + (low - 0xDC00) + 0x10000;
console.log(codePoint.toString(16)); // 1f642 — correct! U+1F642

// ES6+ correct way to get code points
console.log(emoji.codePointAt(0));       // 128578 (0x1F642)
console.log(String.fromCodePoint(128578));// '🙂' — correct

// Old broken way:
console.log(emoji.charCodeAt(0));        // 55357 — just the high surrogate!
console.log(String.fromCharCode(55357)); // '' — broken lone surrogate!

// Lone surrogate — malformed but JS allows it
const lone = '\uD800'; // high surrogate with no pair
console.log(lone.length); // 1 — but it's technically malformed

📋 Golden Rule:
• Always use codePointAt() and String.fromCodePoint() for emoji and non-BMP characters
• NOT charCodeAt() and String.fromCharCode()
• The charCode versions only understand 2-byte code units and return half a surrogate pair for emoji
• codePoint versions correctly handle the 4-byte case

0303 — Grapheme Clusters: What Users See as One Character

A code point ≠ grapheme cluster. A grapheme cluster is what a user perceives as one character — the smallest unit of text that has meaning to a user.

'é' can be TWO different strings: U+00E9 (single code point, precomposed) OR U+0065 U+0301 (e + combining accent = 2 code points). Both render identically but have different .length and different === result!

Emoji sequences are even more complex: the family emoji = 4 person emoji + 3 zero-width joiners = 7 code points, 11 code units. Flag emoji use "regional indicator" code points for country codes. Skin tone modifiers add another code point but display as one character. Intl.Segmenter (ES2022) is the correct API to split strings into grapheme clusters.

// Two ways to write 'é' — identical visually, different in JS
const e1 = '\u00E9';       // é as single code point (precomposed)
const e2 = 'e\u0301';     // e + combining accent (decomposed)

console.log(e1 === e2);   // false — different code points!
console.log(e1.length);   // 1
console.log(e2.length);   // 2 — e + accent = 2 code units

// Visual comparison:
console.log(e1);           // é
console.log(e2);           // é (looks same!)

// Family emoji — 7 code points, 11 code units!
const family = '👨‍👩‍👧‍👦';
console.log(family.length);          // 11
console.log([...family].length);     // 7 — code points via iterator
// Still wrong for grapheme count — correct:
if (typeof Intl.Segmenter !== 'undefined') {
  const segmenter = new Intl.Segmenter();
  const graphemes = [...segmenter.segment(family)];
  console.log(graphemes.length); // 1 — one grapheme cluster!
}

// Normalize to compare strings with combining characters
console.log(e1.normalize('NFC') === e2.normalize('NFC')); // true ✅
// NFC = Canonical Decomposition, followed by Canonical Composition

String normalization (normalize('NFC') or normalize('NFD')) is critical for user input comparison. If one user types 'é' as a single code point and another composes it with accent, plain === fails. Normalize both strings to the same form before comparing.

0404 — String Iteration: for...of vs Index Access

String[index] / charAt(): returns the code UNIT at that index — may be half a surrogate pair. String iterator (for...of, spread): iterates CODE POINTS — correctly handles surrogate pairs.

const text = 'Hi 🙂!';

// Index access — code units (broken for emoji)
console.log(text[0]);       // 'H' — fine, BMP
console.log(text[3]);       // broken lone high surrogate!
console.log(text[4]);       // broken lone low surrogate!
console.log(text.length);   // 6 (H i space 🙂[2 units] !)

// for...of — code points (correct for emoji)
for (const char of text) {
  console.log(char); // H, i, ' ', 🙂, ! — 5 items, emoji as one ✅
}

// Spread — code points array
const chars = [...text];
console.log(chars);        // ['H', 'i', ' ', '🙂', '!']
console.log(chars.length); // 5

// Regex: with /u flag — Unicode aware
console.log(/./u.test('🙂')); // true — matches full emoji as one char
console.log(/./.test('🙂'));  // true — matches only first code unit
console.log('🙂'.match(/./gu).length); // 1 — one full emoji ✅
console.log('🙂'.match(/./g).length);  // 2 — two half-surrogates ❌

// Reverse a string correctly
const str = 'Hello 🙂';
const wrong = str.split('').reverse().join('');      // breaks emoji!
const right = [...str].reverse().join('');           // correct code points
console.log(wrong); // garbled emoji!
console.log(right); // '🙂 olleH' ✅

📋 Golden Rule:
• Use for...of or [...string] when you need to iterate over visible characters (handles surrogate pairs)
• Use Intl.Segmenter when you truly need grapheme clusters (handles combining chars, emoji sequences)
• Never use string[index] for non-ASCII text

0505 — String Performance: Intern, Compare, Slice

String comparison (===): V8 checks pointer equality first (O(1) for interned strings), then length, then content. This is why interned strings compare lightning-fast.

String interning: source code literals are always interned — 'hello' === 'hello' is pointer comparison (O(1)). Runtime strings (from templates, user input) may NOT be interned — they use content comparison. V8's comparison short-circuits: different lengths are immediately false without scanning characters.

// Interned strings — same literal = same object
const a = 'hello';
const b = 'hello';
// V8 uses pointer comparison — O(1)!
console.log(a === b); // true — same interned object

// Runtime strings — may NOT be interned
const c = 'hel' + 'lo'; // V8 may or may not intern this
const d = ['h','e','l','l','o'].join(''); // probably not interned
console.log(c === d); // true — but uses content comparison O(n)

// Slice is zero-copy — fast!
const longString = 'x'.repeat(100000);
console.time('slice');
const view = longString.slice(0, 1000); // SlicedString — no copy
console.timeEnd('slice');

// substring/slice comparison:
// slice(start, end) — negative indices count from end
// substring(start, end) — negative becomes 0, swaps if start > end
const s = 'Hello World';
console.log(s.slice(-5));        // 'World' — last 5 chars
console.log(s.substring(-5, 5));  // 'Hello' — negative → 0

// String deduplication for large data sets
const strings = Array.from({ length: 10000 }, () => 'same-value');
// Each created by runtime code — may not be interned
// Deduplicate with Map if memory matters:
const intern = new Map();
function dedup(str) {
  if (!intern.has(str)) intern.set(str, str);
  return intern.get(str);
}

V8's string comparison short-circuits: if both strings are the same pointer (interned), comparison is one instruction. Otherwise it checks length first (O(1)) — different lengths are immediately false without scanning characters. Content scan only happens for same-length non-identical strings. Always put the longer/less-likely string second in === for performance.

Lo kar liya — Key Points:

✅ JS strings are sequences of UTF-16 code UNITS (2 bytes each) — not characters or code points
✅ BMP characters (U+0000 to U+FFFF) use 1 code unit. Supplementary (emoji, etc.) use 2 code units (surrogate pair)
✅ .length counts code UNITS — emoji shows length 2 because it uses a surrogate pair
✅ Use for...of or spread [...str] to iterate code POINTS — handles emoji correctly
✅ Use Intl.Segmenter for true grapheme clusters — handles combining characters and emoji sequences
✅ Normalize strings before comparing: e1.normalize('NFC') === e2.normalize('NFC') handles precomposed vs decomposed forms

Previous3.2 — Numbers: IEEE 754 📋 All Chapters Next Chapter3.4 — Boolean, Undefined, Null & NaN →