Mail-store FTS (sealed-token full-text index)

Sealed-token full-text search index for b.mailStore. At appendMessage time the row's plaintext subject + addresses + body are tokenized, each token is hashed with the per-deployment vault salt (the same scheme b.cryptoField uses for derived-hash mirrors on sealed columns), and the resulting space-separated token-hash string is inserted into a SQLite FTS5 virtual table. Search runs the same tokenize → hash transform on the operator's query terms and issues MATCH against the FTS5 table — never against plaintext.

The index is unrecoverable without the vault salt. A database dump leaks zero readable text — the FTS5 rows are byte-for-byte indistinguishable from random hashes. Per-tenant separation rides on the cryptoField namespace prefix (bj-

Limitations of sealed-token FTS — operator-facing constraints:

- Exact-token match only. No SQLite FTS5 stemmer, no porter, no Unicode-fold-then-stem, no NEAR with offsets. The token boundary IS the search granularity. Operators that need linguistic search at the cost of plaintext-at-rest opt in to a separate plaintext-FTS layer on top — not part of this primitive. - No prefix wildcard (MATCH 'kub*'). Token hashes don't preserve substring relationships. The cost of partial-match search is sealed-at-rest; operators get either-or. - Stopword filter is conservative (a / the / of / to / in / for / on / and / or / is / are / be / by). Stopwords land in the unsealed plaintext but never reach the FTS row. - Token length capped at 2..64 unicode codepoints after NFC normalisation. Tokens outside the band are dropped (too short = high-collision noise; too long = file-bomb shape).

Posture cascade. The primitive is on by default for every posture (hipaa / pci-dss / gdpr / soc2) — the token index uses the same vault key already protecting sealed-row storage, so adding the FTS index doesn't widen the cryptographic trust boundary. A future opt-in plaintext-FTS overlay would be gated by a relaxed posture; this module ships sealed-only.

b.mailStore.fts.tokenize(text) #

Split text into a deduplicated, lowercased, NFC-normalised token array. Drops stopwords + tokens outside the 2..64-codepoint band. Splits on every non-letter / non-digit codepoint, including the @ + . boundaries of email addresses so local-part + domain labels become independent tokens.

b.mailStore.fts.tokenize("Hello world from alice@example.com");
// → ["hello", "world", "alice", "example", "com"]

b.mailStore.fts.hashToken(table, field, token) #

Keyed hash of one token under the (table, field) namespace. Routes through b.cryptoField.computeNamespacedHash in hmac-shake256 mode — the same keyed-MAC machinery that protects sealed-column derived hashes — so rotating the vault key invalidates every FTS hash in step with every sealed-column hash. Returns a 16-char hex prefix.

var h = b.mailStore.fts.hashToken("mail_messages", "body", "kubernetes");
/^[0-9a-f]{16}$/.test(h);   // → true

b.mailStore.fts.hashTokens(table, field, tokens) #

Hash an array of tokens → space-separated hash string suitable for direct insertion into an FTS5 column. Empty + duplicate token- hashes drop on the way out.

b.mailStore.fts.hashTokens("t", "subject", ["hello", "world"]);
// → "<16hex> <16hex>"

b.mailStore.fts.hashText(table, field, text) #

Tokenize + hash + join in one step. Convenience wrapper — equivalent to hashTokens(table, field, tokenize(text)).

b.mailStore.fts.hashText("mail_messages", "body", "kubernetes deploy");
// → "<16hex> <16hex>"

b.mailStore.fts.rowFromMessage(table, msg) #

Build the FTS5 row payload { objectid, subject_toks, addr_toks, body_toks } from a { objectid, subject, from, to, body } plaintext message. from + to share addr_toks.

b.mailStore.fts.rowFromMessage("t", { objectid:"o1", subject:"Hi", from:"a@x", to:"b@x", body:"hello" });
// → { objectid:"o1", subject_toks:"", addr_toks:" ", body_toks:"" }

b.mailStore.fts.columnAndFieldFor(filterKey) #

Map a search filter key (subject / body / from / to) to the FTS5 column it indexes into PLUS the namespace pseudo-field the indexer uses when hashing tokens. Used by the search path so the query-side hash transform matches the index-side one byte- for-byte.

b.mailStore.fts.columnAndFieldFor("from");
// → { column: "addr_toks", field: "addr" }

b.mailStore.fts.buildMatchExpression(table, field, term) #

Tokenize + hash an operator's query term and produce the FTS5 MATCH expression that selects rows containing every surviving token. Returns null when no tokens survive the tokenize + stopword filter (caller skips the FTS join in that case).

var expr = b.mailStore.fts.buildMatchExpression("t", "body", "kubernetes deploy");
// → "<16hex> AND <16hex>"

b.mailStore.fts.createSql(qFtsTable) #

Returns the CREATE VIRTUAL TABLE IF NOT EXISTS SQL for the sealed-token FTS5 table. The caller passes the quoted table identifier (e.g. "blamejs_mail_messages_fts").

db.prepare(b.mailStore.fts.createSql('"mail_fts"')).run();

Last updated 2026-08-08T16:39:49.652Z by seeder.