Storage

Filesystem-and-cloud-backed object storage with sealed per-file encryption keys, classification routing, and residency enforcement.

b.storage sits one layer above b.objectStore: the lower primitive abstracts the byte-level adapter (local FS, sigv4-style S3-compatible, GCS, Azure Blob, generic HTTP-PUT); this module adds the framework-shaped policy on top — multi-backend registration, per-call classification → backend dispatch, boot-time residency validation against b.db.getDataResidency(), per-file XChaCha20-Poly1305 encryption with the data key sealed into the framework's vault, and audit-chain emission for every read / write / delete / presign.

Configuration accepts either the legacy single-backend shape ({ backend, uploadDir }) or the multi-backend shape ({ backends: { name: cfg, ... }, defaultClassification, refuseUnclassified }). Both normalize internally to the multi-backend form. refuseUnclassified: true forces every call to declare its classification explicitly, which is the right posture for apps mixing personal / operational / public data across different residency zones.

Encrypted save/get is the default surface (saveFile / getFileBuffer / getFileStream); saveRaw / getRawBuffer skip the per-file encryption envelope for content that is already-public or already-encrypted (e.g. signed image assets, pre-encrypted backup bundles).

b.storage.init(opts) #

{
  backend:                "local" | "sigv4" | "gcs" | "azure-blob" | "http-put",  // single-backend shorthand
  uploadDir:              string,             // local backend root (single-backend shorthand)
  backends:               object,             // multi-backend map: name -> backend cfg
  defaultClassification:  string,             // applied when a call omits { classification }
  refuseUnclassified:     boolean,            // refuse calls without explicit classification
}

Register one or more storage backends and lock the framework into the configured policy. Idempotent — a second call after the first succeeds is a no-op (operators rebuild via _resetForTest only). Validates classification → residency mapping at boot so a misconfigured deployment (US backend serving EU personal data) fails fast instead of leaking on first write.

// Single-backend, local FS — typical small-app shape.
b.storage.init({ backend: "local", uploadDir: "./data/uploads" });

// Multi-backend with classification routing + residency tags.
b.storage.init({
  backends: {
    "eu-private": {
      protocol:        "local",
      rootDir:         "/srv/eu/private",
      classifications: ["personal"],
      residencyTag:    "EU",
    },
    "us-ops": {
      protocol:        "local",
      rootDir:         "/srv/us/ops",
      classifications: ["operational", "public"],
      residencyTag:    "US",
    },
  },
  defaultClassification: "operational",
  refuseUnclassified:    true,
});

b.storage.saveFile(buffer, key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name (still validates classification serve)
}

Encrypt buffer under a fresh XChaCha20-Poly1305 data key, seal the data key into the framework vault, and write the ciphertext to the backend selected by opts.classification (or opts.backend for explicit pinning). Returns the storage path plus the sealed key the caller MUST persist alongside the row that references the blob — without it, the bytes are unrecoverable. Emits a system.storage.write audit event with { backend, classification, residencyTag, sizeBytes }.

var buf = Buffer.from("invoice pdf bytes");
var saved = await b.storage.saveFile(buf, "invoices/2026/001.pdf", {
  classification: "personal",
});
// → { storedPath: "invoices/2026/001.pdf",
//     encryptionKey: "v1:...",   // sealed; persist with the row
//     backend: "eu-private",
//     classification: "personal" }

b.storage.getFileBuffer(key, sealedKey, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
}

Fetch the ciphertext at key from the routed backend, unseal the per-file data key via the framework vault, and return the decrypted plaintext as a Buffer. The AEAD tag is verified before any plaintext is released — a tampered ciphertext throws crypto/decrypt-failed, never returns partial bytes. Emits system.storage.read with { backend, key, sizeBytes }.

// Round-trip a small text payload through saveFile/getFileBuffer.
b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var saved = await b.storage.saveFile(Buffer.from("hello"), "greet.txt");
var roundTrip = await b.storage.getFileBuffer("greet.txt", saved.encryptionKey);
roundTrip.toString("utf8");   // → "hello"

b.storage.getFileStream(key, sealedKey, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
}

Buffer-then-stream variant of getFileBuffer — returns a stream.Readable once the AEAD tag has verified the entire ciphertext. Per-file XChaCha20-Poly1305 needs the whole frame before it can release the first byte; chunked AEAD with per-chunk tags would let us stream end-to-end at the cost of finer-grained tampering windows, so the framework defaults to the safe variant.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var saved = await b.storage.saveFile(Buffer.from("stream-me"), "blob.bin");
var stream = await b.storage.getFileStream("blob.bin", saved.encryptionKey);
var chunks = [];
for await (var chunk of stream) chunks.push(chunk);
Buffer.concat(chunks).toString("utf8");   // → "stream-me"

b.storage.saveRaw(buffer, key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
}

Write buffer to the routed backend as-is, skipping the per-file encryption envelope. Use for content that is already public (signed CDN assets, image thumbnails) or already encrypted (pre-sealed backup bundles); use saveFile for everything else. Audit metadata records raw: true so storage reads in the audit chain can be distinguished from encrypted reads.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var saved = await b.storage.saveRaw(Buffer.from("public-bytes"), "logo.png");
// → { storedPath: "logo.png", backend: "default", versionId: null }

b.storage.getRawBuffer(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
}

Fetch the raw bytes at key from the routed backend. No decryption layer is applied — the caller receives whatever was stored, byte-for-byte. Pair with saveRaw; for encrypted blobs use getFileBuffer instead so the AEAD tag is verified.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
await b.storage.saveRaw(Buffer.from("raw-payload"), "asset.bin");
var bytes = await b.storage.getRawBuffer("asset.bin");
bytes.toString("utf8");   // → "raw-payload"

b.storage.deleteFile(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
  versionId:       string,    // erase a specific object version (S3 Object-Lock)
  bypassGovernanceRetention: boolean, // lift GOVERNANCE retention (not COMPLIANCE)
}

Remove key from the routed backend. Returns true when the object existed and was removed, false when it was already absent. Emits system.storage.delete with { backend, key, existed, versionId } so the audit chain records GDPR right-to-erasure flows. The sealed encryption key the caller persisted alongside the row should be discarded by the caller after a successful delete — without the bytes, the key has no recovery value.

On a versioning-enabled (S3 Object-Lock) backend an unversioned delete only writes a delete-marker — the data version survives. To erase a specific version pass versionId (from saveRaw's return or listVersions); a version under an active retention is refused (the call throws), and bypassGovernanceRetention lifts a GOVERNANCE-mode retention for callers with the permission (COMPLIANCE stays immutable). versionId is S3/sigv4-only and is refused on other backends.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
await b.storage.saveRaw(Buffer.from("doomed"), "tmp/x.bin");
var existed = await b.storage.deleteFile("tmp/x.bin");
// → true
var second = await b.storage.deleteFile("tmp/x.bin");
// → false

b.storage.exists(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
}

HEAD-style existence check — returns true when the routed backend reports the key present, false on NOT_FOUND. Other backend errors propagate so transient outages aren't swallowed as "doesn't exist." Cheaper than a full GET when the caller only needs to gate a downstream operation on presence.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
await b.storage.saveRaw(Buffer.from("here"), "probe.bin");
var present = await b.storage.exists("probe.bin");
// → true
var missing = await b.storage.exists("nope.bin");
// → false

b.storage.listVersions(prefix, opts?) #

{
  classification:   string,   // route to a backend serving this classification
  backend:          string,   // explicit backend by name
  maxResults:       number,   // page size
  keyMarker:        string,   // pagination cursor (from a prior page)
  versionIdMarker:  string,   // pagination cursor (from a prior page)
}

Enumerate every object VERSION and delete-marker under prefix on a versioning-enabled (S3 Object-Lock) backend. Plain reads only see the current version; right-to-erasure / crypto-shred on an Object-Lock bucket must target prior versions by versionId, which only this call surfaces. Each item is { key, versionId, isLatest, deleteMarker, size, lastModified, etag }; deleteMarker: true rows are tombstones with no data. Pair with deleteFile(key, { versionId }) to erase a version.

Versioning is an S3/sigv4 feature — a backend without a version surface (filesystem, and the current Azure/GCS adapters) throws VERSIONS_UNSUPPORTED rather than silently returning the current view, so an erasure workflow can never mistake a single-version backend for a fully-enumerated one.

var page = await b.storage.listVersions("filings/2026/");
for (var v of page.items) {
  if (!v.isLatest) await b.storage.deleteFile(v.key, { versionId: v.versionId });
}

b.storage.listBackends() #

Snapshot every registered backend with { name, protocol, classifications, residencyTag, breakerState }. The breakerState is the live circuit-breaker state from the underlying b.objectStore adapter — handy for ops dashboards surfacing a degraded backend before it cascades.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var info = b.storage.listBackends();
info[0].name;       // → "default"
info[0].protocol;   // → "local"

b.storage.presignedUploadUrl(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
  expiresInSec:    number,    // URL lifetime; backend-defaulted when omitted
  contentType:     string,    // pin the upload Content-Type into the signature
}

Issue a short-lived signed URL the client uses to PUT bytes directly to the object store, bypassing the framework process for the upload bytes. Backend-dependent: sigv4 / gcs / azure-blob support it natively; local / http-put backends throw PRESIGN_NOT_SUPPORTED. Emits system.storage.presign with direction: "upload".

b.storage.init({
  backends: {
    "us-ops": {
      protocol:        "sigv4",
      endpoint:        "https://s3.us-east-1.amazonaws.com",
      region:          "us-east-1",
      bucket:          "uploads",
      accessKeyId:     "AKIAEXAMPLE",
      secretAccessKey: "secret",
      classifications: ["operational"],
      residencyTag:    "US",
    },
  },
});
var presigned = b.storage.presignedUploadUrl("incoming/x.bin", {
  backend:      "us-ops",
  expiresInSec: 300,
});
presigned.method;   // → "PUT"

b.storage.presignedDownloadUrl(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
  expiresInSec:    number,    // URL lifetime; backend-defaulted when omitted
  responseHeaders: {          // S3 response-header overrides (sigv4 backend)
    contentDisposition: string,  // e.g. 'attachment; filename="invoice.pdf"'
    contentType:        string,
    contentLanguage:    string,
    contentEncoding:    string,
    cacheControl:       string,
    expires:            string,
  },
}

Issue a short-lived signed URL the client uses to GET bytes directly from the object store. Same backend-support matrix as the upload variant. Use this only with saveRaw content — encrypted blobs (saveFile) need the per-file sealed key, which the framework does not expose to the client.

b.storage.init({
  backends: {
    "us-ops": {
      protocol:        "sigv4",
      endpoint:        "https://s3.us-east-1.amazonaws.com",
      region:          "us-east-1",
      bucket:          "uploads",
      accessKeyId:     "AKIAEXAMPLE",
      secretAccessKey: "secret",
      classifications: ["public"],
      residencyTag:    "US",
    },
  },
});
var presigned = b.storage.presignedDownloadUrl("public/logo.png", {
  backend:      "us-ops",
  expiresInSec: 60,
  responseHeaders: {
    contentDisposition: 'attachment; filename="logo.png"',
  },
});
presigned.method;   // → "GET"

b.storage.presignedUploadPolicy(key, opts) #

{
  classification:  string,    // route to a backend serving this classification
  backend:         string,    // explicit backend by name
  maxBytes:        number,    // body-size cap (required for size enforcement)
  expiresInSec:    number,    // policy lifetime; backend-defaulted when omitted
  contentType:     string,    // pin the upload Content-Type into the policy
}

Issue a signed POST-form policy (sigv4 / gcs) or vendor-equivalent PUT (azure-blob) that the client uploads against, with the body- size cap baked into the signature so an oversize upload is rejected by the object store, not by the framework process. Use this — not presignedUploadUrl — when the upload size matters and you can't trust the client. result.enforcement indicates whether the cap is server-side ("server") or client-only ("client-only" — Azure SAS, where the operator must HEAD the blob post-upload to reject oversize). local and http-put backends throw PRESIGN_NOT_SUPPORTED.

b.storage.init({
  backends: {
    "us-ops": {
      protocol:        "sigv4",
      endpoint:        "https://s3.us-east-1.amazonaws.com",
      region:          "us-east-1",
      bucket:          "uploads",
      accessKeyId:     "AKIAEXAMPLE",
      secretAccessKey: "secret",
      classifications: ["operational"],
      residencyTag:    "US",
    },
  },
});
var policy = b.storage.presignedUploadPolicy("user/avatar.png", {
  backend:      "us-ops",
  maxBytes:     5 * 1024 * 1024,   // 5 MiB cap, server-enforced
  expiresInSec: 300,
  contentType:  "image/png",
});
policy.enforcement;   // → "server"

b.storage.getBackend(name) #

Return the named backend instance from the underlying b.objectStore adapter, or null when no backend with that name is registered. Most operator code routes through the dispatching primitives (saveFile / getFileBuffer / ...); getBackend is the escape hatch for adapter-specific operations (lifecycle policy ops, vendor-specific HEAD probes) the framework does not abstract.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var backend = b.storage.getBackend("default");
backend.protocol;   // → "local"
var missing = b.storage.getBackend("does-not-exist");
// → null

b.storage.chunkScratch(opts?) #

{
  rootKeyPrefix: string,   // default "chunk-scratch" — namespace under the backend
  backend:       string,   // explicit backend by name (default: framework default)
  maxChunkBytes: number,   // default 16 MiB — per-chunk cap
  staleAfterMs:  number,   // default 24h — assemblies idle longer get GC'd
}

Resumable-chunked-upload primitive. Persists incoming upload chunks during the upload window, then concatenates them in order on completion and returns the assembled bytes for the caller to persist (the primitive does not itself write a final file). Owns per-assembly directory layout, envelope-encrypted chunk persistence, ordered gap-checked assembly, and GC of partial assemblies.

Composes existing primitives: each chunk routes through b.storage.saveFile (same XChaCha20-Poly1305 envelope as the non-chunked surface), assembly reads through getFileBuffer, deletion through deleteFile. No new crypto.

Prior art / wire-protocol references: - tus.io v1.0.0 protocol (Termination + Creation + Concatenation extensions) — operator-facing HTTP shape that ships chunks against a server-side assembly. This primitive is the server-side persistence the tus protocol's upload handler consumes. - RFC 9110 §14.4 Content-Range — the wire-protocol header that PUT/PATCH-based resumable uploads use to declare each chunk's byte-range within the assembly. - draft-ietf-httpbis-resumable-upload-08 — IETF working-draft resumable-upload protocol; this primitive's surface mirrors its server-side state requirements. - AWS S3 Multipart Upload — the cloud-vendor analogue; saveChunk / assemble are the framework's local equivalents of UploadPart / CompleteMultipartUpload.

Threat-model coverage: - Path-traversal in upload paths (CVE-2018-1000656 class) — assemblyId is validated to refuse .., /, \, NUL / C0 controls, DEL, dot-prefix, and oversize. A hostile client can't escape the rootKeyPrefix namespace. - Chunk-out-of-order replay / TOCTOU between saveChunk and assemble — assemble verifies monotonic 0..N-1 indices and refuses on gaps; a chunk inserted out-of-order can't be surfaced as a valid assembly. - Storage exhaustion from abandoned uploads — gc({ olderThanMs }) prunes stale assemblies; operator wires it on a schedule. - AEAD context-binding — each chunk's encryption envelope is keyed independently; an attacker who guesses one chunk's key can't decrypt other chunks in the same assembly (the XChaCha20-Poly1305 keys are framework-vault-derived per-call).

assemblyId shape is validated to refuse path-traversal, control chars, and oversize at every entry point.

b.storage.init({ backend: "local", uploadDir: "./data/uploads" });
var cs = b.storage.chunkScratch({ rootKeyPrefix: "uploads/scratch" });

// During upload — each PUT lands one chunk. saveChunk returns the
// chunk's sealed encryptionKey; collect them in order for assemble.
var keys = [];
keys[0] = (await cs.saveChunk({ assemblyId: "upload-abc", chunkIndex: 0, data: chunk0 })).encryptionKey;
keys[1] = (await cs.saveChunk({ assemblyId: "upload-abc", chunkIndex: 1, data: chunk1 })).encryptionKey;
keys[2] = (await cs.saveChunk({ assemblyId: "upload-abc", chunkIndex: 2, data: chunk2 })).encryptionKey;

// On completion — concat the chunks (in order) into the assembled
// buffer, then clean up. chunkEncryptionKeys is one key per chunk.
var assembled = await cs.assemble({ assemblyId: "upload-abc", expectedTotal: 3, chunkEncryptionKeys: keys });
await cs.removeAssembly("upload-abc");

// Periodic GC of partial uploads abandoned mid-stream
var removed = await cs.gc({ olderThanMs: 86400000 });

Last updated 2026-08-08T16:39:49.652Z by seeder.