Codepoint Class
The Unicode threat-codepoint catalog the b.guard* family screens with — bidi overrides, C0 controls, zero-width and invisible formatting, the null byte, the Unicode Tags block, and UTS #39 confusable-script classification. Each class is a numeric range table plus the scanners that read it, so a consumer building a custom free-text screen composes the same tables the guards do instead of retyping a character class (where the zero-width set comes out one codepoint short and the astral Tags block — the "ASCII smuggling" carrier — is missed entirely). For a ready-made screen over unconstrained free text reach for b.guardText; use this catalog when you need the raw detectors, tables, or the script classifier.
The scanners walk codepoints and carry no regular expression. A character class assembled as a pattern is a second grammar between the table and the text, and it is where this catalog's misses have come from: an astral codepoint written as a four-digit escape re-parses into a range that matches almost everything, and a global pattern reused across calls carries its own cursor and answers differently the second time. A walk over codePointAt has neither failure available to it.
Attack characters are emitted from the numeric tables at runtime rather than typed, so every file in the family stays pure ASCII and a payload cannot hide in a source literal.
Surface:
inRanges(cp, ranges) -> is this codepoint in the table? firstInRanges(s, ranges) -> index of the first member, or -1 stripRanges(s, ranges) -> s with every member removed indexOfAny(s, chars) -> index of the first char from a set, or -1 replaceAny(s, chars, to) -> every char from a set replaced trimChars(s, chars) -> leading + trailing run of a set removed containsFolded(s, needle) -> substring search, ASCII case-insensitive indexOfFolded(s, needle) -> the same search, reporting where matchesAtFolded(s, at, n) -> does n sit exactly at this index? hex4(cp) -> "\\uXXXX" escape for one codepoint charClass(ranges) -> character-class body for a range table, for a caller assembling a pattern of its own (the guards do not) fromCp(cp) -> String.fromCharCode shorthand
Codepoint tables:
BIDI_RANGES — Unicode bidi-override family (CVE-2021-42574 Trojan Source). LRM U+200E / RLM U+200F / ALM U+061C / LRE U+202A / RLE U+202B / PDF U+202C / LRO U+202D / RLO U+202E / LRI U+2066 / RLI U+2067 / FSI U+2068 / PDI U+2069.
C0_CTRL_RANGES — C0 control characters minus tab (U+09) / lf (U+0A) / cr (U+0D) — those are dialect-shaped chars that parsers handle separately. Everything else (U+00, U+01-U+08, U+0B-U+0C, U+0E-U+1F) flagged as control-byte injection.
C1_CTRL_RANGES: the C1 control characters U+0080-U+009F. U+009B is the one-character form of ESC [ and U+0085 NEXT LINE is a line break under Unicode line breaking.
CTRL_RANGES: C0_CTRL_RANGES, DEL (U+007F) and C1_CTRL_RANGES. The text guards read this table for controlPolicy.
ZERO_WIDTH_RANGES — invisible-formatting / zero-width chars attackers use to hide payloads: SHY U+00AD ZWSP U+200B ZWNJ U+200C ZWJ U+200D WJ U+2060 BOM U+FEFF
b.codepointClass.hex4(cp) #
Format a codepoint as a 4-digit \uXXXX regex escape (zero-padded, upper case) — the building block charClass uses to compile a range table into a character-class body without embedding the attack character as a literal.
b.codepointClass.hex4(0x202E); // returns the escape "\\u202E"b.codepointClass.charClass(rangeList) #
Compile a codepoint range table — numbers and [lo, hi] pairs — into a regex character-class body (the inner text of [...]), so a detector can build its own class from a catalog table without typing the codepoints as literals.
A codepoint above U+FFFF is emitted as \u{...}, which the resulting regex needs the u flag to read. The 4-digit form cannot express one, and the failure is not a missed match: 0-F re-parses as , the range 0-, and F — a class matching nearly every character, so a threat matcher built from an astral table would fire on ordinary text. A table of only BMP codepoints compiles as before and does not require the flag.
var body = b.codepointClass.charClass([0x200E, [0x202A, 0x202E]]);
var re = new RegExp("[" + body + "]");
// Astral table — the compiled class requires the `u` flag.
var tags = b.codepointClass.charClass([[0xE0000, 0xE007F]]);
var tagRe = new RegExp("[" + tags + "]", "u");b.codepointClass.inRanges(cp, ranges) #
Whether codepoint cp falls in a range table — the same tables the threat catalog is built from, where each entry is a bare codepoint or a [lo, hi] pair.
b.codepointClass.inRanges(0x202E, b.codepointClass.BIDI_RANGES); // → trueb.codepointClass.trimTrailingChars(text, chars) #
Remove the run of characters at the END of text that are all in chars. Returns text unchanged when it is not a string or nothing at the end matches.
The regex spelling of this, text.replace(/\/+$/, ""), has no start anchor, so the engine retries the run from every position and a long run costs time proportional to the SQUARE of its length: about 20ms for a single call on 8192 characters, which is a CPU amplifier when the text arrives on a request. This walks back from the end once.
b.codepointClass.trimTrailingChars("https://x.example///", "/");
// → "https://x.example"
b.codepointClass.trimTrailingChars("host.example...", ".");
// → "host.example"b.codepointClass.firstDelimited(text, open, close, from?) #
The first non-empty open...close group at or after from, as { body, start, end } where start is the index of open and end the index of close. Returns null when there is none, and when either delimiter is not a single character. An empty group is skipped and the search continues, which is what [^close]+ requiring one character does.
Single characters only, as lastDelimited is. Skipping an empty group advances by one and searches again, so a long delimiter would be re-matched from almost the same place every time and the scan would be quadratic in the delimiter's length.
The regex spelling, /<([^>]+)>/, restarts at every open it finds, so a value of nothing but < costs time proportional to its SQUARE: about 20ms for a single call on 8192 characters. This walks the text once.
b.codepointClass.firstDelimited("Name ", "<", ">").body;
// → "a@b.example" b.codepointClass.lastDelimited(text, open, close) #
The open...close group that ENDS the text, ignoring trailing whitespace, as { body, start, end }. Returns null when the last non-space character is not close, when no open precedes it, or when the group is empty.
The end-anchored regex, /<([^>]+)>\s*$/, still scans for its opening character from every position and costs time proportional to the SQUARE of the text. This walks back from the end once.
b.codepointClass.lastDelimited("List ", "<", ">").body;
// → "a@b.example" b.codepointClass.firstInRanges(text, ranges, from?) #
Index of the first codepoint of text at or after from that falls in ranges, or -1. The index is a UTF-16 code-unit offset, matching a JS string index.
The walk reads whole codepoints and steps over surrogate pairs, which the astral tables require: the Unicode Tags block lives at U+E0000 and up, and a charCodeAt walk would read its two surrogates as unrelated BMP codepoints and never match the block at all.
b.codepointClass.firstInRanges("ok" + String.fromCodePoint(0xE0041),
b.codepointClass.TAG_RANGES); // → 2b.codepointClass.isAuditPolicy(policy) #
Whether a policy value asks for an AUDIT — reporting the finding rather than refusing or repairing. audit-only is the framework's own synonym for audit, and this is the one place that decides so.
It exists because the two spellings disagreed. The disposition mapper treated them as the same thing while the severity calculation recognized only the literal audit, so zeroWidthPolicy: "audit-only" produced a high-severity issue and a failed validation where audit produced a warning and a pass. Two spellings of one setting is how they come to disagree; asking one predicate is how they stop.
b.codepointClass.isAuditPolicy("audit"); // → true
b.codepointClass.isAuditPolicy("audit-only"); // → true
b.codepointClass.isAuditPolicy("reject"); // → falseb.codepointClass.stripRanges(text, ranges) #
text with every codepoint in ranges removed. Returns the original string when nothing matched, so a clean input costs no copy.
b.codepointClass.stripRanges("a" + String.fromCharCode(0x200B) + "b",
b.codepointClass.ZERO_WIDTH_RANGES); // → "ab"b.codepointClass.replaceRanges(text, ranges, replacement) #
text with every codepoint in ranges replaced by replacement. stripRanges is this with an empty replacement.
One replacement per CODEPOINT, so a character above U+FFFF becomes a single replacement rather than two — a filename sanitizer that emits one underscore per surrogate has told the operator a single character was two.
b.codepointClass.replaceRanges("a" + String.fromCharCode(0x202E) + "b",
b.codepointClass.BIDI_RANGES, "_"); // → "a_b"b.codepointClass.indexOfAny(text, chars, from?) #
Index of the first character of text that appears in chars, at or after from, or -1. chars is a plain string used as a set — the characters to find, in any order, with no pattern syntax in it.
The set form is what a guard usually wants. A character class written as a pattern has to escape whatever the class syntax reserves, and the reserved set differs between the inside and the outside of the brackets: a ] or a - in the wrong place ends the class early or opens a range, and the result still compiles.
Both the set and the subject are read as codepoints, so a character above U+FFFF in either is one member rather than two surrogate halves — a set holding one cannot match half of an unrelated pair.
b.codepointClass.indexOfAny("report.csv", "<>:\"/\\|?*"); // → 6 b.codepointClass.replaceAny(text, chars, replacement) #
text with EVERY character that appears in chars replaced by replacement (pass "" to remove them). Returns the original string when nothing matched.
Every one, not the first: a sanitizer that replaces a single occurrence leaves the rest of them in place, and the caller has no way to tell from the return value that it did. One replacement per CODEPOINT, so a character above U+FFFF becomes a single replacement rather than two.
b.codepointClass.replaceAny("ac", "<>", "_"); // → "a_b_c"b.codepointClass.trimChars(text, chars, opts?) #
{
leading: boolean, // default: true — trim the run at the start
trailing: boolean, // default: true — trim the run at the end
}text with a leading and trailing run of characters drawn from chars removed. opts.leading / opts.trailing (both default true) restrict it to one end.
b.codepointClass.trimChars(" report. ", " ."); // → "report"b.codepointClass.trimRanges(text, ranges, opts?) #
{
leading: boolean, // default: true — trim the run at the start
trailing: boolean, // default: true — trim the run at the end
}trimChars over a codepoint range table instead of a literal set — for the classes too large to type out, WHITESPACE_RANGES above all.
var nbsp = String.fromCharCode(0x00A0);
b.codepointClass.trimRanges(nbsp + " report\t",
b.codepointClass.WHITESPACE_RANGES); // → "report"b.codepointClass.isRunOf(text, chars, min?, max?) #
Is every character of text drawn from chars, with a length between min (default 1) and max (default unbounded)? The anchored, length-bounded token shape a protocol grammar is written in — an IMAP tag, a message number, an ESMTP parameter name.
Both the set and the length are read in CODEPOINTS. For the ASCII token grammars this exists for the two are the same count; for anything else, a character above U+FFFF is one member and one unit of length rather than two surrogate halves.
ASCII_DIGITS, ASCII_ALPHA, ASCII_ALNUM and ASCII_HEX ship beside it so a set reads as b.codepointClass.ASCII_ALNUM + "._-" rather than as sixty-two typed characters.
var CP = b.codepointClass;
CP.isRunOf("A001", CP.ASCII_ALNUM + "._-", 1, 64); // → true
CP.isRunOf("", CP.ASCII_DIGITS); // → falseb.codepointClass.isRunOfRanges(text, ranges, min?, max?) #
isRunOf against a RANGE TABLE rather than a spelled-out set — for the grammars whose alphabet is a span rather than a list, where writing out the ninety-five printable ASCII characters would obscure what the rule is.
var CP = b.codepointClass;
CP.isRunOfRanges("hi there", [0x0009, [0x0020, 0x007E]], 0); // → trueb.codepointClass.containsFolded(haystack, needle) #
Does haystack contain needle, comparing ASCII letters without regard to case? Non-ASCII characters compare exactly.
The ASCII-only fold is the point. Lower-casing the whole subject first is the obvious alternative and it moves the text underneath the answer: several codepoints case-map to more than one character, so an index into the folded string no longer refers to the same place in the original — and the caller that then reports an offset, or slices around the hit, is off by however many characters expanded ahead of it.
b.codepointClass.containsFolded("dir/%2E%2E/etc", "%2e%2e"); // → trueb.codepointClass.matchesAtFolded(text, at, needle) #
Does needle appear in text starting exactly at index at, comparing ASCII letters without regard to case? Non-ASCII characters compare exactly.
This is the shape a scanner needs when it has already found its anchor and wants to know what follows it — the question a .startsWith on a slice answers by allocating a copy of the rest of the document at every candidate position, which is how a screen that reads in one pass becomes one that reads in a pass per character.
b.codepointClass.matchesAtFolded("", 0, "b.codepointClass.indexOfFolded(haystack, needle, from?) #
Index of the first occurrence of needle in haystack at or after from (default 0), comparing ASCII letters without regard to case, or -1. The index refers to the ORIGINAL string — see containsFolded for why folding the subject first moves the answer out from under the caller.
b.codepointClass.indexOfFolded("a ", "b.codepointClass.fromCp(cp) #
String.fromCharCode shorthand — emit the actual character for a codepoint at runtime (e.g. to build a test fixture) instead of typing the attack character as a source literal.
var rlo = b.codepointClass.fromCp(0x202E); // the U+202E override charb.codepointClass.scriptFor(cp) #
Return the Unicode script name for a codepoint ("latin", "cyrillic", "greek", "han", ...), or null when the codepoint is script-neutral (digits, punctuation, symbols). The classifier detectMixedScripts uses to spot homograph / confusable mixing (UTS #39).
b.codepointClass.scriptFor("a".charCodeAt(0)); // returns "latin"
b.codepointClass.scriptFor(0x0430); // returns "cyrillic" (the confusable a)b.codepointClass.detectMixedScripts(label, allowedScripts) #
UTS #39 confusable detection: return null when label is single-script (or every script it uses is in the optional allowedScripts allowlist), or the full array of script names when it mixes scripts — the homograph attack shape (a Cyrillic confusable letter inside an otherwise-Latin label). Callers decide refuse / audit / strip. Pass allowedScripts to permit legitimate mixing (an ASCII word inside a non-Latin label).
b.codepointClass.detectMixedScripts("paypal"); // null (single-script)
var spoof = "pa" + b.codepointClass.fromCp(0x0443) + "pal"; // Cyrillic u (U+0443)
b.codepointClass.detectMixedScripts(spoof); // ["latin", "cyrillic"]
b.codepointClass.detectMixedScripts(spoof, ["latin", "cyrillic"]); // null (allowlisted)b.codepointClass.detectCharThreats(text, opts, codePrefix) #
{
bidiPolicy: string, // non-"allow" -> flag bidi overrides
nullBytePolicy: string, // non-"allow" -> flag null bytes
controlPolicy: string, // non-"allow" -> flag control chars (C0 + DEL)
zeroWidthPolicy: string, // non-"allow" -> flag zero-width
}Scan text for the character-class threats — bidi override, null byte, control character, zero-width and Unicode Tags — and return an array of issue objects { kind, severity, ruleId, location, snippet }, at most one per class. Each class is gated by an opts policy that isn't "allow"; ruleId is prefixed with codePrefix. The non-throwing detection pass the b.guard* family shares instead of re-rolling the per-class match-and-push. Unicode Tags follows tagsPolicy, or zeroWidthPolicy when the guard names no policy of its own.
Every class scans on the same terms, and a caller cannot ask for a class to be skipped or stamp a severity of its own. severity is NOT a uniform function of the policy, so do not read an outcome out of it — pass the finding to b.gateContract.charThreatDisposition, which resolves every one of the five from its own policy. What each reports:
- zero-width and unicode-tags follow the policy: reject is critical, audit is warn, a repair policy such as strip is high. - control-char follows it too, except that its refusing severity is high rather than critical. A validator narrowed to ["critical"] does not throw on one; the gate still refuses it. - bidi-override and null-byte are critical whatever the policy says. bidiPolicy: "audit" reports at refusing severity, so a caller filtering on severity refuses what the operator asked only to record. charThreatDisposition reads the policy and does not.
location is a UTF-16 code-unit offset into text, which is what a JS string index is — NOT a UTF-8 byte offset. After a multibyte character the two differ (an emoji occupies one byte offset of 4 and one code-unit offset of 2), so a caller converting the value for a byte-addressed report converts it explicitly.
var issues = b.codepointClass.detectCharThreats(
userText, { bidiPolicy: "reject", nullBytePolicy: "reject" }, "comment");
if (issues.length) refuse(issues[0].ruleId);b.codepointClass.assertNoCharThreats(text, opts, errorFactory, codePrefix) #
{
bidiPolicy: string, // "reject" -> throw on a bidi override
nullBytePolicy: string, // "reject" -> throw on a null byte
controlPolicy: string, // "reject" -> throw on a C0 control
zeroWidthPolicy: string, // "reject" -> throw on a zero-width char
tagsPolicy: string, // "reject" -> throw on a Unicode Tags char
}Throw — via errorFactory(code, message) — when text contains a character class whose opts policy is "reject" (bidi / null byte / C0 control / zero-width / Unicode Tags). The throwing counterpart of detectCharThreats, covering the same classes as applyCharStripPolicies strips so that between the two every class has an enforcement path at every policy value; errorFactory lets the caller raise its own typed error and codePrefix namespaces the rule code.
The scans are unbounded, so a caller handling untrusted input bounds it first — with assertWithinMaxBytes, or with whatever ceiling the guard already applies. This is not done here because several callers reach this point having already refused, truncated or repaired an oversized input under their own rule, and a second ceiling with a different error code would override theirs.
A sanitize path calls this BEFORE applyCharStripPolicies: the strip table removes only what is set to "strip", so without the assert a class set to "reject" would be neither refused nor repaired and the caller would get the threat back verbatim.
b.codepointClass.assertNoCharThreats(value,
{ bidiPolicy: "reject", nullBytePolicy: "reject" },
function (code, msg) { return new TypeError(code + ": " + msg); }, "note");b.codepointClass.assertWithinMaxBytes(text, opts, errorFactory, codePrefix) #
{
maxBytes: number, // UTF-8 byte ceiling; absent means unbounded
}Throw errorFactory(code, message) — when text exceeds opts.maxBytes UTF-8 bytes. A no-op when opts.maxBytes is absent.
This is the ceiling assertNoCharThreats and detectCharThreats expect a caller to have applied: their scans are unbounded, so on untrusted input the size refusal has to come first or the scans run on whatever an attacker sends. It is a separate call rather than part of those functions because a guard that already refuses, truncates or repairs an oversized input under its own rule must keep its own error, not inherit this one.
b.codepointClass.assertWithinMaxBytes(body, { maxBytes: 1048576 },
function (code, msg) { return new TypeError(code + ": " + msg); }, "note");b.codepointClass.scrubCharThreats(text, opts, errorFactory, codePrefix) #
{
maxBytes: number, // UTF-8 byte ceiling; omitted means unbounded
bidiPolicy: string, // "reject" -> throw; "strip" -> remove
nullBytePolicy: string,
controlPolicy: string,
zeroWidthPolicy: string,
tagsPolicy: string, // defaults to zeroWidthPolicy when unset
}Bound the input, refuse every character class set to "reject", strip every class set to "strip", and return the cleaned string — the whole sanitize front end for a content guard in one call.
The steps belong together because the order between them is what makes a policy mean anything. applyCharStripPolicies removes only what is set to "strip", so a guard that calls it alone hands a "reject" class straight back to the caller: not refused, not repaired, and no error to say so. And the scans behind the assert are unbounded, so the ceiling has to precede them. Callers that composed the pieces by hand got both of those wrong.
Throws opts.maxBytes (measured in UTF-8 bytes), then whichever errorFactory(code, message) builds the guard's own typed error.
var clean = b.codepointClass.scrubCharThreats(input,
{ maxBytes: 1048576, bidiPolicy: "reject", zeroWidthPolicy: "strip" },
function (code, msg) { return new TypeError(code + ": " + msg); }, "note");b.codepointClass.applyCharStripPolicies(text, opts) #
{
bidiPolicy: string, // "strip" -> remove bidi overrides
controlPolicy: string, // "strip" -> remove C0 controls
nullBytePolicy: string, // "strip" -> remove null bytes
zeroWidthPolicy: string, // "strip" -> remove zero-width / invisible chars
tagsPolicy: string, // "strip" -> remove the Unicode Tags block
}Strip each character-class threat whose opts policy is "strip" and return the cleaned string — the sanitize counterpart of detectCharThreats, shared by every guard's sanitize path so none re-rolls the same sequence of replace() calls. Removes bidi overrides, C0 controls, null bytes, zero-width chars, and the Unicode-Tags block ("ASCII smuggling") per policy.
var clean = b.codepointClass.applyCharStripPolicies(userText,
{ bidiPolicy: "strip", zeroWidthPolicy: "strip", tagsPolicy: "strip" });b.codepointClass.resolveTagsPolicy(opts) #
{
tagsPolicy: string, // when set, this is the answer
zeroWidthPolicy: string, // inherited only when tagsPolicy is unset
}The Unicode Tags policy actually in force: opts.tagsPolicy when the guard names one, otherwise the zeroWidthPolicy it inherits from. Returns undefined when neither is set.
Exported because the inheritance is a rule, not a convention. A caller that re-derives it by testing zeroWidthPolicy alone silently ignores an explicit tagsPolicy: "allow" — validation then reports nothing while the scrub path removes the character anyway.
if (b.codepointClass.resolveTagsPolicy(opts) === "strip") {
text = b.codepointClass.stripRanges(text, b.codepointClass.TAG_RANGES);
}b.codepointClass.escapeRegExp(s) #
Escape every ECMAScript RegExp metacharacter in a string so an operator- or input-supplied token matches literally when spliced into a new RegExp(...) — a token destined for dynamic compilation cannot inject a pattern.
var re = new RegExp(b.codepointClass.escapeRegExp("a.b*c"));
re.test("a.b*c"); // true — the . and * are literalb.codepointClass.isAsciiAlnum(cc) #
Test whether a char code is an ASCII letter or digit (A-Z / a-z / 0-9) — the alphanumeric range check that recurs across every byte-class parser (URL unreserved, XML name chars, header tokens), centralized so the range literals live once.
b.codepointClass.isAsciiAlnum("Z".charCodeAt(0)); // true
b.codepointClass.isAsciiAlnum("-".charCodeAt(0)); // falseb.codepointClass.isAsciiLetter(cc) #
Is this code unit an ASCII letter, A-Z or a-z?
b.codepointClass.isAsciiLetter("Q".charCodeAt(0)); // → trueb.codepointClass.isAsciiDigit(cc) #
Is this code unit an ASCII digit, 0-9?
b.codepointClass.isAsciiDigit("7".charCodeAt(0)); // → trueb.codepointClass.isAsciiHexDigit(cc) #
Is this code unit a hexadecimal digit in either case? The token every percent-escape, numeric character reference, color literal and binary- encoded identifier is spelled in.
b.codepointClass.isAsciiHexDigit("F".charCodeAt(0)); // → trueb.codepointClass.isAtext(cc) #
Is this code unit RFC 5322 section 3.2.3 atext: an ASCII letter, an ASCII digit, or one of `!#$%&'*+-/=?^_{|}~`? The dot is not atext; it joins atoms, which is what isDotAtomText` reads.
b.codepointClass.isAtext("+".charCodeAt(0)); // → trueb.codepointClass.isDotAtomText(text) #
Is this string RFC 5322 section 3.2.3 dot-atom-text: one or more atoms of atext joined by single dots? An empty string, a leading or trailing dot, a doubled dot, and any character outside atext all read false. This is the shape of an unquoted local part and of the left and right halves of a msg-id.
b.codepointClass.isDotAtomText("first.last"); // → true
b.codepointClass.isDotAtomText("first..last"); // → falseb.codepointClass.isUtf8DotAtomText(text) #
Is this string RFC 6531 section 3.3 dot-atom-text for an internationalized address: the RFC 5322 shape with atext extended by every non-ASCII code unit. The dot rule is unchanged, so a leading, trailing or doubled dot still reads false, as does an ASCII character outside atext such as @, ; or a space.
b.codepointClass.isUtf8DotAtomText("üser"); // → true
b.codepointClass.isUtf8DotAtomText("üser@evil"); // → falseb.codepointClass.isAsciiWhitespace(cc) #
Is this code unit one of the five ASCII whitespace characters: tab, line feed, form feed, carriage return or space? This is the set the HTML tokenizer and the CSS tokenizer separate tokens on. A no-break space or any other Unicode space is not in it.
b.codepointClass.isAsciiWhitespace(0x0C); // → true
b.codepointClass.isAsciiWhitespace(0xA0); // → falseb.codepointClass.isIdentifierChar(cc) #
Is this code unit one a bare identifier is made of — a letter, a digit or an underscore? This is the boundary a token screen tests against: a keyword that runs into one of these is a longer word, so DATABASE is not DATA and #ends is not #end.
var CP = b.codepointClass;
CP.isIdentifierChar("_".charCodeAt(0)); // → true
CP.isIdentifierChar("-".charCodeAt(0)); // → falseb.codepointClass.hasPairWhere(text, first, second, accept) #
Is there a place where first is immediately followed by second, and accept says that place counts? accept(i) is called with the index of first, and returns whether the construct opening there is a real one.
This is the frame every two-character construct screen was writing out by hand: a path traversal opening on .., a regex inline-flag group on (?, a YAML merge key on <<. The literals and the follow-on tests differ, the scan does not, and three copies of a scan are three places for an off-by-one to live. It reads the string one character at a time, which is the property a guard needs: asking the question with a pattern would be a screen running the construct it screens over hostile text.
accept decides alone — the scan does not skip past a hit it rejected, so overlapping openers are all offered. A screen that needs to COUNT non-overlapping occurrences is asking a different question and keeps its own cursor.
var CP = b.codepointClass;
CP.hasPairWhere("a/../b", ".", ".", function () { return true; }); // → true
CP.hasPairWhere("a.b", ".", ".", function () { return true; }); // → falseb.codepointClass.splitLines(text) #
Split on LF, dropping a CR immediately before it — a message's lines, whether it uses CRLF or bare LF. A bare CR does NOT end a line, which is what makes one the signal a line-protocol guard screens for.
b.codepointClass.splitLines("a\r\nb\nc"); // → ["a","b","c"]b.codepointClass.splitLinesAny(text) #
splitLines, but a LONE carriage return ends a line too. This is the rule the wire formats use — a vCard, a delivery status notification, an old-Mac text file — where a producer that emits CR alone still means a new line, and a reader that only knows LF folds the whole document into one.
b.codepointClass.splitLinesAny("a\rb\r\nc\nd"); // → ["a","b","c","d"]b.codepointClass.splitOnWhitespace(text) #
Split on runs of whitespace, dropping the empty pieces — the tokens of a protocol line. Whitespace is the full \s set, so a token separated by a no-break space is separated here too.
b.codepointClass.splitOnWhitespace(" MAIL FROM "); // → ["MAIL","FROM"]b.codepointClass.isUnreserved(cc) #
Test whether a char code is in the RFC 3986 §2.3 unreserved set — ALPHA / DIGIT / - / . / _ / ~. A percent-escape of an unreserved character is over-encoding the URI spec says SHOULD be decoded (§6.2.2.3).
b.codepointClass.isUnreserved("~".charCodeAt(0)); // true
b.codepointClass.isUnreserved("/".charCodeAt(0)); // falseb.codepointClass.isForbiddenControlChar(code, opts) #
{
forbidTab: boolean, // also forbid TAB
allowC1: boolean, // permit the C1 controls 0x80-0x9f
allowLf: boolean, // permit LF (0x0a)
allowCr: boolean, // permit CR (0x0d)
}The header-injection / RFC 5322 control-byte predicate every "refuse control bytes in a header / line / value" loop shares. Returns true for DEL (0x7f), any C0 control (< 0x20) other than TAB (0x09), and the C1 controls (0x80-0x9f); LF and CR are refused by default but can be permitted per call (a reader that already split on CRLF, or a folding grammar). It agrees with the CTRL_RANGES table. C0_CTRL_RANGES is a different table: it always exempts LF/CR and never matches DEL or C1.
allowC1 permits 0x80-0x9f. Pass it only when the string is a latin1 view of bytes, such as an HTTP field value, where those codes are the continuation bytes of a UTF-8 sequence, or when the grammar being parsed allows C1 (TOML 1.0 strings).
b.codepointClass.isForbiddenControlChar(0x00); // true (NUL)
b.codepointClass.isForbiddenControlChar(0x09, { forbidTab: true }); // true (TAB forbidden)
b.codepointClass.isForbiddenControlChar(0x9b); // true (C1 CSI)
b.codepointClass.isForbiddenControlChar(0x85, { allowC1: true }); // false (latin1 byte view)b.codepointClass.firstControlCharOffset(s, opts) #
{
forbidTab: boolean, // also treat TAB as forbidden
allowC1: boolean, // permit the C1 controls 0x80-0x9f (latin1 byte views)
allowLf: boolean, // permit LF (0x0a)
allowCr: boolean, // permit CR (0x0d)
}Return the index of the first forbidden control char in s (under the same opts as isForbiddenControlChar), or -1 when none. Callers wrap it as a boolean (!== -1), throw with the offending code (s.charCodeAt(offset)), or derive a byte offset — replacing the open-coded control-byte scan each parser previously rolled by hand.
b.codepointClass.firstControlCharOffset("ok\x00bad"); // 2 (the NUL)
b.codepointClass.firstControlCharOffset("clean"); // -1b.codepointClass.firstLineInjectionCharOffset(s) #
Return the index of the first CR, LF or NUL in s, or -1 when there is none. These three are the bytes that end a line-oriented protocol record, so a value carrying one splits the record it is written into and the remainder is read as a second, attacker-chosen line: the header-injection class in HTTP and Set-Cookie, and the command-injection class in SMTP, POP3, IMAP and ManageSieve.
Narrower than firstControlCharOffset, deliberately. That one refuses every C0 control and DEL, which is right for text a human wrote; this one answers the specific question a wire-protocol writer asks, so a caller adopting it does not silently start refusing values it used to accept.
b.codepointClass.firstLineInjectionCharOffset("nonce-42"); // -1
b.codepointClass.firstLineInjectionCharOffset("abc\r\nOK"); // 3b.codepointClass.decodeNumericEntities(s) #
Decode HTML numeric character references (hex &#x..; and decimal &#..;) just enough to expose a scheme hidden behind entity-encoding. The trailing semicolon is OPTIONAL — a browser decodes javascript: (no semicolon) the same as javascript:, so a semicolon-required decoder lets the no-semicolon form slip a scheme past an allowlist. Shared so the markup guards cannot drift on this.
b.codepointClass.decodeNumericEntities("javascript:"); // "javascript:"
b.codepointClass.decodeNumericEntities("javascript:"); // "javascript:" (no semicolon)b.codepointClass.decodeMarkupEntities(value) #
Decode the character references a browser resolves inside an attribute value: numeric (hex/decimal, semicolon OPTIONAL) first, then the named-entity ASCII subset browsers honor in URL/CSS contexts. Drop the C0 controls and zero-widths a payload hides behind. The single decoder every content guard routes a scheme / CSS-token danger check through, so a threat cannot slip past the guard that forgot to decode an encoding a sibling strips. Pair with stripUrlSchemeWhitespace for a URL-scheme check.
b.codepointClass.decodeMarkupEntities("expression("); // "expression("
b.codepointClass.decodeMarkupEntities("behavior:"); // "behavior:"b.codepointClass.decodeEntityAt(s, at) #
Decode the one entity reference that begins at index at of s, which must be an ampersand: a numeric reference, or failing that a named one. Returns { text, next }: the decoded text and the index just after the reference. Returns null when no numeric or named reference begins there.
This reads one reference in isolation. decodeMarkupEntities decodes numeric references in one pass and named references in a second pass over the result, so a numeric reference that yields an ampersand can complete a named reference with the text after it, and this does not model that: walking ":" with it gives ":" where the whole-string decoder gives ":". Use decodeReferenceAt for a walk that agrees with decodeMarkupEntities.
b.codepointClass.decodeEntityAt("js", 0); // → { text: "j", next: 6 }
b.codepointClass.decodeEntityAt(" ", 0); // → { text: "\t", next: 5 }
b.codepointClass.decodeEntityAt("&x", 0); // → nullb.codepointClass.isUrlSchemeStrippable(cp) #
Report whether a code point is one the URL-scheme normalization removes: a C0 control or space, tab, line feed, carriage return, or a zero-width character. decodeMarkupEntities and stripUrlSchemeWhitespace together drop these before a scheme is read, so a reader locating a scheme by position skips exactly this set.
b.codepointClass.isUrlSchemeStrippable(0x01); // → true
b.codepointClass.isUrlSchemeStrippable(0x200B); // → true
b.codepointClass.isUrlSchemeStrippable(0x6A); // → falseb.codepointClass.decodeNamedEntityAt(s, at) #
Decode the one named entity reference that begins at index at of s, which must be a literal ampersand, and nothing else: a numeric reference at at returns null, whatever it decodes to. Returns { text, next } or null. decodeMarkupEntities decodes numeric references in one pass and named references in a second pass over the result, so the second pass sees a name whose letters, or whose closing semicolon, were written as numeric references: : is : to it. This reads the name the same way, so a caller walking by position recognizes exactly the references the whole-string decoder does. A numeric reference that produces an ampersand can likewise complete a named reference with the text after it; decodeReferenceAt reads that composition.
b.codepointClass.decodeNamedEntityAt(":x", 0); // → { text: ":", next: 7 }
b.codepointClass.decodeNamedEntityAt("&", 0); // → nullb.codepointClass.decodeReferenceAt(s, at) #
Decode the character reference that begins at index at of s, which must be an ampersand, the way decodeMarkupEntities reads it: a named reference, or a numeric reference, and when the numeric reference produces an ampersand, the named reference that ampersand completes with the text after it. Returns { text, next } or null. decodeNamedEntityAt reads only a literal ampersand; this is the one call that walks by position and still sees : as :.
b.codepointClass.decodeReferenceAt(":x", 0); // → { text: ":", next: 11 }
b.codepointClass.decodeReferenceAt("&x", 0); // → { text: "&", next: 5 }
b.codepointClass.decodeReferenceAt(":", 0); // → { text: ":", next: 7 }b.codepointClass.stripUrlSchemeWhitespace(s) #
Fold away exactly the whitespace the WHATWG URL parser removes before it resolves a scheme: ASCII tab / LF / CR from ANYWHERE, plus a leading/trailing C0-control-or-space run. tab/lf/cr are excluded from the C0-control catalog and space is not a control, so a danger check that strips only C0/zero-width still lets java or an entity-encoded leading space ( javascript:) read as scheme-less. Run AFTER entity decoding; every guard that extracts a URL scheme for a denylist routes the decoded value through this.
b.codepointClass.stripUrlSchemeWhitespace(" javascript:x"); // "javascript:x"b.codepointClass.canonicalizeForCase(codePoint, unicode) #
The character a regular-expression engine compares when i is in force, given a code point and whether the u (or v) flag applies.
It is not "the upper case" and it is not "the lower case". Without u the language folds through upper case, keeps a character whose upper case runs to more than one character, and refuses to fold a non-ASCII character onto an ASCII one — the last of which is why /k/i does not match a Kelvin sign, and why converting both characters and comparing gets the wrong answer.
Under u the rule changes to case FOLDING: the ASCII guard drops, so a long s folds onto an s, and an expanding upper case no longer stops the fold — a Greek eta with a iota subscript and its capital form both upper-case to two characters and are still the same character there.
Two characters are the same under i exactly when this returns the same code point for both.
var same = b.codepointClass.canonicalizeForCase;
same("k".codePointAt(0), false) === same("K".codePointAt(0), false); // → true
same("k".codePointAt(0), false) === same(0x212a, false); // → false (Kelvin)
same("s".codePointAt(0), true) === same(0x17f, true); // → true (long s, under u)b.codepointClass.caseFoldPartners(codePoint, unicode) #
Every OTHER code point a regular expression treats as the same character as this one under i, given whether the u (or v) flag applies.
Use it where the characters to compare against are not a list you can walk — matching a character against a class of ranges, say. Where you hold both characters already, canonicalizeForCase answers directly and this is not needed.
Most partners are the character's own upper and lower forms. Several hundred are not reachable that way — a micro sign and a Greek mu, a final sigma and an ordinary one, the title-case digraphs, under u a Kelvin sign and a k, the Greek letters carrying a iota subscript beside their capital forms, and two ligatures that share only the "ST" they upper-case to — and those come from a table derived from the running platform's own case mappings rather than transcribed from a Unicode revision.
var partners = b.codepointClass.caseFoldPartners;
partners("σ".codePointAt(0), true); // sigma → includes the final sigma
partners("k".codePointAt(0), false); // → [ 0x4b ], "K" and not the Kelvin sign