AGENTS.md

AGENTS.md — rumcodecs

Overview

rumcodecs is a pnpm workspace providing buffer compression codecs for JavaScript — a Rust/WASM reimplementation of numcodecs.js built for Zarr. Five codec classes are exported from one npm package, @fideus-labs/rumcodecs:

• Blosc — the Blosc meta-compressor via the pure-Rust blusc crate, compiled to WASM (SIMD128).
• LZ4 — lz4_flex via WASM, using numcodecs framing (4-byte little-endian original size + LZ4 block).
• Zstd — the zstd crate via WASM, standard Zstd frames.
• GZip / Zlib — pure JS via fflate (a peer dependency); no WASM involved.

The contract is drop-in compatibility: identical API surface to numcodecs.js and identical byte formats to Python numcodecs, so chunks are interchangeable across ecosystems. Three layers hold that up: test/compat.test.ts asserts the API surface, test/interop.test.ts decodes buffers actually written by Python numcodecs, and the Rust unit tests in lib.rs pin the framing we emit. Note what the per-codec suites do not prove: they only show encode/decode symmetry, which would survive both directions drifting off the format together. The interop fixtures are the ones that fail when that happens.

Modules

Module	Read
TypeScript codec classes	./src/
Shared Codec / CodecConstructor interfaces	./src/types.ts
Rust WASM bindings (blusc, lz4_flex, zstd)	./crates/rumcodecs-wasm/src/lib.rs
Test suites (round-trip, compat, interop)	./test/
Benchmark package (vs numcodecs.js)	./benchmark/
CI / release workflows	./.github/workflows/

Context to load on demand

Task	Read
Add or change a codec class	./src/types.ts, the sibling codec in ./src/, its test in ./test/
numcodecs API-compatibility rules	./test/compat.test.ts
Byte-format / interop questions	./test/interop.test.ts, ./test/fixtures/README.md
Change or add WASM exports	./crates/rumcodecs-wasm/src/lib.rs, the build:wasm* scripts in ./package.json
Run or extend benchmarks	./benchmark/src/run.ts, ./benchmark/src/data.ts, ./benchmark/src/results.ts
CI matrix and gates	./.github/workflows/ci.yml, ./.github/workflows/rust-ci.yml
Release / publish flow	./.github/workflows/release.yml

Conventions

• Conventional Commits with optional scopes: feat: …, fix(blosc): …, ci: …, build: … — match the existing git log style.
• Tooling is vite-plus (vp): pnpm build / pnpm test / pnpm check wrap vp build / vp test / vp check (oxlint + oxfmt + typecheck). Keep pnpm check clean before committing — CI runs it as a gate.
• Every codec class is a drop-in for numcodecs.js: static codecId, static fromConfig(config), the same constructor signature, and the same static members (Blosc.SHUFFLE, LZ4.max_buffer_size, Zstd.MAX_CLEVEL, …). Any new surface must be asserted in ./test/compat.test.ts.
• Byte formats are Python-numcodecs compatible and must never change silently: LZ4 uses a 4-byte little-endian original-size header before the block; Zstd emits standard frames; Blosc emits Blosc1-compatible frames (BLOSC_FORWARD_COMPAT_SPLIT). A new codec or option combination needs a fixture in ./test/fixtures/ — regenerate with its generate.py, and do not refresh existing fixtures to make a failure go away, since that discards the evidence of the break.
• WASM-backed codecs load the module lazily through a cached dynamic import (the getModule() pattern in ./src/blosc.ts); new WASM-backed codecs must reuse it so pure-JS codecs never pay the WASM load.
• GZip and Zlib stay pure-JS on fflate, and fflate stays a peerDependency — do not move it to dependencies or reimplement it in the WASM crate.
• The Rust functions return an empty Vec on failure; the TypeScript layer converts that into a thrown Error. Keep error handling on that boundary — no panics across wasm_bindgen, because a panic traps the WASM instance and every later call fails with it, so one corrupt chunk breaks every subsequent read.
• An empty Vec alone is not a sufficient error signal, because a frame that legitimately holds zero bytes decodes to exactly the same thing. Each wrapper compares the result against the length the frame itself declares: blosc_declared_size, the 4-byte LE header src/lz4.ts reads directly, and zstd_declared_content_size. Only a mismatch is an error. Getting this wrong is silent: with out supplied, a "failed" decode returns the caller's own untouched buffer, which reads as data rather than as an error. Note the limit of this check — it catches truncation and a short decode, but a Blosc1 frame carries no checksum over its payload, so corruption that still parses can decode to the declared length and be returned as valid. That is pinned in ./test/compat.test.ts so it stays a known fact.
• Allocations sized from a header must be fallible. The plausibility guards below bound what a header may claim, not what the allocator can supply; an infallible vec![0; n] turns a shortfall into handle_alloc_error, which aborts the module exactly like a panic. Reserve through try_reserve_exact (see try_zeroed) so it degrades to an ordinary decode failure.
• Sizes read out of a compressed header are untrusted input. Every decoder we wrap trusts them to index or to size an allocation up front — blusc indexes the source with them, lz4_flex zero-fills the declared size before reading a byte, zstd reserves the declared content size — so a corrupt header turns into an out-of-bounds panic or a multi-gigabyte allocation that aborts the module. Validate before handing anything over: blosc_header_is_sane, lz4_declared_size_is_plausible, zstd_declared_size_is_plausible. The expansion bounds come from each format's own ceiling (LZ4 255:1, zstd 32768:1) and are pinned by tests that compress a constant buffer, which sits right on that ceiling — tighten them and real data starts failing to decode.
• decode(data, out?) on the WASM-backed codecs must honor the optional out buffer (write into it and return it) — zarr consumers rely on this. GZip and Zlib ignore out and return fflate's own buffer, matching numcodecs.js exactly; both behaviors are pinned in ./test/compat.test.ts.
• WASM builds require RUSTFLAGS="-C target-feature=+simd128" and build both web and nodejs targets into pkg/ (see the build:wasm* scripts); pkg/ is generated output, never hand-edited.
• A new codec touches five places: src/<codec>.ts, src/index.ts, the export map in package.json (subpath export), the build.lib.entry list in ./vite.config.ts, and a test/<codec>.test.ts — plus the codec table in ./README.md. Miss the vite entry and the advertised subpath resolves to a file that was never built; ./test/index.test.ts checks the three lists agree.
• Node ≥ 18 is supported (engines); CI tests Node 22 and 24.

Key dependencies

Dependency	Version	Notes
blusc	0.0.6	Pure-Rust Blosc (Blosc2 API, Blosc1-compatible output) — the Blosc codec
lz4_flex	~0.12	Pure-Rust LZ4 block compression — the LZ4 codec
zstd	~0.13 (no default features)	Zstd — the Zstd codec
wasm-bindgen / js-sys	0.2 / 0.3	WASM boundary
fflate	^0.8 (peer)	Pure-JS gzip/zlib — the GZip and Zlib codecs
vite-plus (vp)	0.2	Build / test / check driver
vitest	4	Test runner
tinybench	6	Benchmark timing (benchmark package)
numcodecs (npm)	^0.3.2 (dev)	numcodecs.js — the head-to-head baseline in ./benchmark/
numcodecs (PyPI)	0.16.5	Python numcodecs — wrote ./test/fixtures/; never installed, uv fetches it pinned