Codepoint Class

Shared codepoint-table threat catalog and regex compiler — the Unicode bidi-override / C0-control / zero-width / null-byte / Unicode-Tags tables (plus UTS #39 confusable-script detection) that the b.guard* family composes internally, exposed on the public surface so a consumer can build a custom free-text screen without re-rolling the regexes (where the zero-width class is mistyped and the astral Unicode-Tags "ASCII smuggling" block forgotten) or coupling to an internal module path. For a ready-made unconstrained-free-text guard reach for b.guardText; use this catalog when you need the raw detectors, tables, or script classifier to compose your own. Detectors emit attack characters programmatically from numeric codepoint tables (never as source literals), so files that use them stay pure ASCII.

Threat detectors that need to match Unicode bidi overrides, C0 control characters, zero-width / invisible chars, etc. compose regex character classes from numeric codepoint range tables here instead of embedding the attack characters directly in their source files. Centralizing the tables means:

- Source files in lib/guard-* stay pure ASCII (zero irregular-whitespace lint findings, no eslint-disable comments for this category). - Adding / removing a codepoint from the catalog is a single edit; every guard picks up the change. - The detector composes the way an attacker would compose the payload (programmatic codepoint emission, not literal typing).

Surface:

hex4(cp) -> "\\uXXXX" escape for a single codepoint charClass(ranges) -> regex character class body for a range table (e.g. [0x200E, [0x202A,0x202E]]) fromCp(cp) -> String.fromCharCode shorthand ranges() -> { BIDI_RANGES, C0_CTRL_RANGES, ZERO_WIDTH_RANGES } compiled() -> { BIDI_RE, BIDI_RE_G, C0_CTRL_RE, C0_CTRL_RE_G, ZERO_WIDTH_RE, ZW_RE_G, NULL_RE_G, NULL_BYTE, BOM_CHAR }

The compiled() exports are RegExp instances built from the codepoint tables at module load. Consumers grab them once at boot.

Codepoint tables:

BIDI_RANGES — Unicode bidi-override family (CVE-2021-42574 Trojan Source). LRM U+200E / RLM U+200F / ALM U+061C / LRE U+202A / RLE U+202B / PDF U+202C / LRO U+202D / RLO U+202E / LRI U+2066 / RLI U+2067 / FSI U+2068 / PDI U+2069.

C0_CTRL_RANGES — C0 control characters minus tab (U+09) / lf (U+0A) / cr (U+0D) — those are dialect-shaped chars that parsers handle separately. Everything else (U+00, U+01-U+08, U+0B-U+0C, U+0E-U+1F) flagged as control-byte injection.

ZERO_WIDTH_RANGES — invisible-formatting / zero-width chars attackers use to hide payloads: SHY U+00AD ZWSP U+200B ZWNJ U+200C ZWJ U+200D WJ U+2060 BOM U+FEFF

b.codepointClass.hex4(cp) #

Format a codepoint as a 4-digit \uXXXX regex escape (zero-padded, upper case) — the building block charClass uses to compile a range table into a character-class body without embedding the attack character as a literal.

b.codepointClass.hex4(0x202E);   // returns the escape "\\u202E"

b.codepointClass.charClass(rangeList) #

Compile a codepoint range table — numbers and [lo, hi] pairs — into a regex character-class body (the inner text of [...]), so a detector can build its own class from a catalog table without typing the codepoints as literals.

var body = b.codepointClass.charClass([0x200E, [0x202A, 0x202E]]);
var re = new RegExp("[" + body + "]");

b.codepointClass.fromCp(cp) #

String.fromCharCode shorthand — emit the actual character for a codepoint at runtime (e.g. to build a test fixture) instead of typing the attack character as a source literal.

var rlo = b.codepointClass.fromCp(0x202E);   // the U+202E override char

b.codepointClass.scriptFor(cp) #

Return the Unicode script name for a codepoint ("latin", "cyrillic", "greek", "han", ...), or null when the codepoint is script-neutral (digits, punctuation, symbols). The classifier detectMixedScripts uses to spot homograph / confusable mixing (UTS #39).

b.codepointClass.scriptFor("a".charCodeAt(0));   // returns "latin"
b.codepointClass.scriptFor(0x0430);              // returns "cyrillic" (the confusable a)

b.codepointClass.detectMixedScripts(label, allowedScripts) #

UTS #39 confusable detection: return null when label is single-script (or every script it uses is in the optional allowedScripts allowlist), or the full array of script names when it mixes scripts — the homograph attack shape (a Cyrillic confusable letter inside an otherwise-Latin label). Callers decide refuse / audit / strip. Pass allowedScripts to permit legitimate mixing (an ASCII word inside a non-Latin label).

b.codepointClass.detectMixedScripts("paypal");   // null (single-script)
var spoof = "pa" + b.codepointClass.fromCp(0x0443) + "pal";  // Cyrillic u (U+0443)
b.codepointClass.detectMixedScripts(spoof);                       // ["latin", "cyrillic"]
b.codepointClass.detectMixedScripts(spoof, ["latin", "cyrillic"]); // null (allowlisted)

b.codepointClass.detectCharThreats(text, opts, codePrefix, zeroWidthSeverity) #

{
  bidiPolicy:      string,   // non-"allow" -> flag bidi overrides
  nullBytePolicy:  string,   // non-"allow" -> flag null bytes
  controlPolicy:   string,   // non-"allow" -> flag C0 controls
  zeroWidthPolicy: string,   // non-"allow" (+ zeroWidthSeverity) -> flag zero-width
}

Scan text for the character-class threats — bidi override, null byte, C0 control, and (opt-in) zero-width — and return an array of issue objects { kind, severity, ruleId, location, snippet }, at most one per class. Each class is gated by an opts policy that isn't "allow"; ruleId is prefixed with codePrefix. The non-throwing detection pass the b.guard* family shares instead of re-rolling the per-class match-and-push. zeroWidthSeverity opts the zero-width scan in and stamps its severity.

var issues = b.codepointClass.detectCharThreats(
  userText, { bidiPolicy: "reject", nullBytePolicy: "reject" }, "comment");
if (issues.length) refuse(issues[0].ruleId);

b.codepointClass.assertNoCharThreats(text, opts, errorFactory, codePrefix) #

{
  bidiPolicy:     string,   // "reject" -> throw on a bidi override
  nullBytePolicy: string,   // "reject" -> throw on a null byte
  controlPolicy:  string,   // "reject" -> throw on a C0 control
}

Throw — via errorFactory(code, message) — when text contains a character class whose opts policy is "reject" (bidi / null byte / C0 control). The throwing counterpart of detectCharThreats; errorFactory lets the caller raise its own typed error and codePrefix namespaces the rule code. The caller bounds the input length before calling (the regexes are unbounded).

b.codepointClass.assertNoCharThreats(value,
  { bidiPolicy: "reject", nullBytePolicy: "reject" },
  function (code, msg) { return new TypeError(code + ": " + msg); }, "note");

b.codepointClass.applyCharStripPolicies(text, opts) #

{
  bidiPolicy:      string,   // "strip" -> remove bidi overrides
  controlPolicy:   string,   // "strip" -> remove C0 controls
  nullBytePolicy:  string,   // "strip" -> remove null bytes
  zeroWidthPolicy: string,   // "strip" -> remove zero-width / invisible chars
  tagsPolicy:      string,   // "strip" -> remove the Unicode Tags block
}

Strip each character-class threat whose opts policy is "strip" and return the cleaned string — the sanitize counterpart of detectCharThreats, shared by every guard's sanitize path so none re-rolls the same sequence of replace() calls. Removes bidi overrides, C0 controls, null bytes, zero-width chars, and the Unicode-Tags block ("ASCII smuggling") per policy.

var clean = b.codepointClass.applyCharStripPolicies(userText,
  { bidiPolicy: "strip", zeroWidthPolicy: "strip", tagsPolicy: "strip" });

b.codepointClass.escapeRegExp(s) #

Escape every ECMAScript RegExp metacharacter in a string so an operator- or input-supplied token matches literally when spliced into a new RegExp(...) — a token destined for dynamic compilation cannot inject a pattern.

var re = new RegExp(b.codepointClass.escapeRegExp("a.b*c"));
re.test("a.b*c");   // true — the . and * are literal

b.codepointClass.isAsciiAlnum(cc) #

Test whether a char code is an ASCII letter or digit (A-Z / a-z / 0-9) — the alphanumeric range check that recurs across every byte-class parser (URL unreserved, XML name chars, header tokens), centralized so the range literals live once.

b.codepointClass.isAsciiAlnum("Z".charCodeAt(0));   // true
b.codepointClass.isAsciiAlnum("-".charCodeAt(0));   // false

b.codepointClass.isUnreserved(cc) #

Test whether a char code is in the RFC 3986 §2.3 unreserved set — ALPHA / DIGIT / - / . / _ / ~. A percent-escape of an unreserved character is over-encoding the URI spec says SHOULD be decoded (§6.2.2.3).

b.codepointClass.isUnreserved("~".charCodeAt(0));   // true
b.codepointClass.isUnreserved("/".charCodeAt(0));   // false

b.codepointClass.isForbiddenControlChar(code, opts) #

{
  forbidTab: boolean,   // also forbid TAB -> predicate is `code < 0x20 || code === 0x7f`
  allowLf:   boolean,   // permit LF (0x0a)
  allowCr:   boolean,   // permit CR (0x0d)
}

The header-injection / RFC 5322 control-byte predicate every "refuse control bytes in a header / line / value" loop shares. Returns true for DEL (0x7f) and any C0 control (< 0x20) other than TAB (0x09); LF and CR are refused by default but can be permitted per call (a reader that already split on CRLF, or a folding grammar). Distinct from the C0_CTRL_RE scanning table which always exempts LF/CR and never matches DEL.

b.codepointClass.isForbiddenControlChar(0x00);                 // true (NUL)
b.codepointClass.isForbiddenControlChar(0x09, { forbidTab: true }); // true (TAB forbidden)

b.codepointClass.firstControlCharOffset(s, opts) #

{
  forbidTab: boolean,   // also treat TAB as forbidden
  allowLf:   boolean,   // permit LF (0x0a)
  allowCr:   boolean,   // permit CR (0x0d)
}

Return the index of the first forbidden control char in s (under the same opts as isForbiddenControlChar), or -1 when none. Callers wrap it as a boolean (!== -1), throw with the offending code (s.charCodeAt(offset)), or derive a byte offset — replacing the open-coded control-byte scan each parser previously rolled by hand.

b.codepointClass.firstControlCharOffset("ok\x00bad");   // 2 (the NUL)
b.codepointClass.firstControlCharOffset("clean");          // -1

b.codepointClass.decodeNumericEntities(s) #

Decode HTML numeric character references (hex &#x..; and decimal &#..;) just enough to expose a scheme hidden behind entity-encoding. The trailing semicolon is OPTIONAL — a browser decodes javascript: (no semicolon) the same as javascript:, so a semicolon-required decoder lets the no-semicolon form slip a scheme past an allowlist. Shared so the markup guards cannot drift on this.

b.codepointClass.decodeNumericEntities("javascript:");   // "javascript:"
b.codepointClass.decodeNumericEntities("javascript:");    // "javascript:" (no semicolon)

b.codepointClass.decodeMarkupEntities(value) #

Decode the character references a browser resolves inside an attribute value -- numeric (hex/decimal, semicolon OPTIONAL) then the named-entity ASCII subset browsers honor in URL/CSS contexts -- and drop the C0 controls and zero-widths a payload hides behind. The single decoder every content guard routes a scheme / CSS-token danger check through, so a threat cannot slip past the guard that forgot to decode an encoding a sibling strips. Pair with stripUrlSchemeWhitespace for a URL-scheme check.

b.codepointClass.decodeMarkupEntities("expression(");   // "expression("
b.codepointClass.decodeMarkupEntities("behavior:");    // "behavior:"

b.codepointClass.stripUrlSchemeWhitespace(s) #

Fold away exactly the whitespace the WHATWG URL parser removes before it resolves a scheme: ASCII tab / LF / CR from ANYWHERE, plus a leading/trailing C0-control-or-space run. tab/lf/cr are excluded from the C0-control catalog and space is not a control, so a danger check that strips only C0/zero-width still lets javascript: or an entity-encoded leading space ( javascript:) read as scheme-less. Run AFTER entity decoding; every guard that extracts a URL scheme for a denylist routes the decoded value through this.

b.codepointClass.stripUrlSchemeWhitespace("  javascript:x");   // "javascript:x"

Last updated 2026-08-08T16:39:49.652Z by seeder.