Skip to content
This repository was archived by the owner on Aug 7, 2026. It is now read-only.

Commit f712a2d

Browse files
thewtexthewtex
authored
Merge pull request #1 from fideus-labs/readme
Harden WASM decoders against corrupt headers, pin numcodecs interop, add docs
2 parents e241048 + 2d90299 commit f712a2d

24 files changed

Lines changed: 1481 additions & 8 deletions

‎AGENTS.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# AGENTS.md — rumcodecs
2+
3+
## Overview
4+
5+
rumcodecs is a pnpm workspace providing **buffer compression codecs for
6+
JavaScript** — a Rust/WASM reimplementation of
7+
[numcodecs.js](https://github.com/manzt/numcodecs.js) built for Zarr. Five
8+
codec classes are exported from one npm package,
9+
`@fideus-labs/rumcodecs`:
10+
11+
- **`Blosc`** — the Blosc meta-compressor via the pure-Rust
12+
[blusc](https://crates.io/crates/blusc) crate, compiled to WASM (SIMD128).
13+
- **`LZ4`** — [lz4_flex](https://crates.io/crates/lz4_flex) via WASM, using
14+
numcodecs framing (4-byte little-endian original size + LZ4 block).
15+
- **`Zstd`** — the [zstd](https://crates.io/crates/zstd) crate via WASM,
16+
standard Zstd frames.
17+
- **`GZip`** / **`Zlib`** — pure JS via
18+
[fflate](https://github.com/101arrowz/fflate) (a peer dependency); no WASM
19+
involved.
20+
21+
The contract is **drop-in compatibility**: identical API surface to
22+
numcodecs.js and identical byte formats to Python numcodecs, so chunks are
23+
interchangeable across ecosystems. Three layers hold that up:
24+
[`test/compat.test.ts`](./test/compat.test.ts) asserts the API surface,
25+
[`test/interop.test.ts`](./test/interop.test.ts) decodes buffers actually
26+
written by Python `numcodecs`, and the Rust unit tests in
27+
[`lib.rs`](./crates/rumcodecs-wasm/src/lib.rs) pin the framing we emit. Note
28+
what the per-codec suites do _not_ prove: they only show encode/decode symmetry,
29+
which would survive both directions drifting off the format together. The
30+
interop fixtures are the ones that fail when that happens.
31+
32+
## Modules
33+
34+
| Module | Read |
35+
| ---------------------------------------------- | ------------------------------------------------------------------------ |
36+
| TypeScript codec classes | [./src/](./src/) |
37+
| Shared `Codec` / `CodecConstructor` interfaces | [./src/types.ts](./src/types.ts) |
38+
| Rust WASM bindings (blusc, lz4_flex, zstd) | [./crates/rumcodecs-wasm/src/lib.rs](./crates/rumcodecs-wasm/src/lib.rs) |
39+
| Test suites (round-trip, compat, interop) | [./test/](./test/) |
40+
| Benchmark package (vs numcodecs.js) | [./benchmark/](./benchmark/) |
41+
| CI / release workflows | [./.github/workflows/](./.github/workflows/) |
42+
43+
## Context to load on demand
44+
45+
| Task | Read |
46+
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
47+
| Add or change a codec class | [./src/types.ts](./src/types.ts), the sibling codec in [./src/](./src/), its test in [./test/](./test/) |
48+
| numcodecs API-compatibility rules | [./test/compat.test.ts](./test/compat.test.ts) |
49+
| Byte-format / interop questions | [./test/interop.test.ts](./test/interop.test.ts), [./test/fixtures/README.md](./test/fixtures/README.md) |
50+
| Change or add WASM exports | [./crates/rumcodecs-wasm/src/lib.rs](./crates/rumcodecs-wasm/src/lib.rs), the `build:wasm*` scripts in [./package.json](./package.json) |
51+
| Run or extend benchmarks | [./benchmark/src/run.ts](./benchmark/src/run.ts), [./benchmark/src/data.ts](./benchmark/src/data.ts), [./benchmark/src/results.ts](./benchmark/src/results.ts) |
52+
| CI matrix and gates | [./.github/workflows/ci.yml](./.github/workflows/ci.yml), [./.github/workflows/rust-ci.yml](./.github/workflows/rust-ci.yml) |
53+
| Release / publish flow | [./.github/workflows/release.yml](./.github/workflows/release.yml) |
54+
55+
## Conventions
56+
57+
- Conventional Commits with optional scopes: `feat: …`, `fix(blosc): …`,
58+
`ci: …`, `build: …` — match the existing `git log` style.
59+
- Tooling is [vite-plus](https://www.npmjs.com/package/vite-plus) (`vp`):
60+
`pnpm build` / `pnpm test` / `pnpm check` wrap `vp build` / `vp test` /
61+
`vp check` (oxlint + oxfmt + typecheck). Keep `pnpm check` clean before
62+
committing — CI runs it as a gate.
63+
- **Every codec class is a drop-in for numcodecs.js**: static `codecId`,
64+
static `fromConfig(config)`, the same constructor signature, and the same
65+
static members (`Blosc.SHUFFLE`, `LZ4.max_buffer_size`,
66+
`Zstd.MAX_CLEVEL`, …). Any new surface must be asserted in
67+
[./test/compat.test.ts](./test/compat.test.ts).
68+
- **Byte formats are Python-numcodecs compatible** and must never change
69+
silently: LZ4 uses a 4-byte little-endian original-size header before the
70+
block; Zstd emits standard frames; Blosc emits Blosc1-compatible frames
71+
(`BLOSC_FORWARD_COMPAT_SPLIT`). A new codec or option combination needs a
72+
fixture in [./test/fixtures/](./test/fixtures/) — regenerate with its
73+
`generate.py`, and do not refresh existing fixtures to make a failure go
74+
away, since that discards the evidence of the break.
75+
- WASM-backed codecs load the module lazily through a cached dynamic import
76+
(the `getModule()` pattern in [./src/blosc.ts](./src/blosc.ts)); new
77+
WASM-backed codecs must reuse it so pure-JS codecs never pay the WASM load.
78+
- `GZip` and `Zlib` stay pure-JS on fflate, and fflate stays a
79+
`peerDependency` — do not move it to `dependencies` or reimplement it in
80+
the WASM crate.
81+
- The Rust functions return an empty `Vec` on failure; the TypeScript layer
82+
converts that into a thrown `Error`. Keep error handling on that boundary —
83+
**no panics across `wasm_bindgen`**, because a panic traps the WASM instance
84+
and every later call fails with it, so one corrupt chunk breaks every
85+
subsequent read.
86+
- **An empty `Vec` alone is not a sufficient error signal**, because a frame
87+
that legitimately holds zero bytes decodes to exactly the same thing. Each
88+
wrapper compares the result against the length the frame itself declares:
89+
`blosc_declared_size`, the 4-byte LE header `src/lz4.ts` reads directly, and
90+
`zstd_declared_content_size`. Only a mismatch is an error. Getting this wrong
91+
is silent: with `out` supplied, a "failed" decode returns the caller's own
92+
untouched buffer, which reads as data rather than as an error. Note the limit
93+
of this check — it catches truncation and a short decode, but a Blosc1 frame
94+
carries no checksum over its payload, so corruption that still parses can
95+
decode to the declared length and be returned as valid. That is pinned in
96+
[./test/compat.test.ts](./test/compat.test.ts) so it stays a known fact.
97+
- **Allocations sized from a header must be fallible.** The plausibility guards
98+
below bound what a header may claim, not what the allocator can supply; an
99+
infallible `vec![0; n]` turns a shortfall into `handle_alloc_error`, which
100+
aborts the module exactly like a panic. Reserve through `try_reserve_exact`
101+
(see `try_zeroed`) so it degrades to an ordinary decode failure.
102+
- **Sizes read out of a compressed header are untrusted input.** Every decoder
103+
we wrap trusts them to index or to size an allocation up front — `blusc`
104+
indexes the source with them, `lz4_flex` zero-fills the declared size before
105+
reading a byte, `zstd` reserves the declared content size — so a corrupt
106+
header turns into an out-of-bounds panic or a multi-gigabyte allocation that
107+
aborts the module. Validate before handing anything over:
108+
`blosc_header_is_sane`, `lz4_declared_size_is_plausible`,
109+
`zstd_declared_size_is_plausible`. The expansion bounds come from each
110+
format's own ceiling (LZ4 255:1, zstd 32768:1) and are pinned by tests that
111+
compress a constant buffer, which sits right on that ceiling — tighten them
112+
and real data starts failing to decode.
113+
- `decode(data, out?)` on the WASM-backed codecs must honor the optional `out`
114+
buffer (write into it and return it) — zarr consumers rely on this. `GZip`
115+
and `Zlib` ignore `out` and return fflate's own buffer, matching numcodecs.js
116+
exactly; both behaviors are pinned in
117+
[./test/compat.test.ts](./test/compat.test.ts).
118+
- WASM builds require `RUSTFLAGS="-C target-feature=+simd128"` and build both
119+
`web` and `nodejs` targets into `pkg/` (see the `build:wasm*` scripts);
120+
`pkg/` is generated output, never hand-edited.
121+
- A new codec touches five places: `src/<codec>.ts`, `src/index.ts`, the export
122+
map in `package.json` (subpath export), the `build.lib.entry` list in
123+
[./vite.config.ts](./vite.config.ts), and a `test/<codec>.test.ts` — plus the
124+
codec table in [./README.md](./README.md). Miss the vite entry and the
125+
advertised subpath resolves to a file that was never built;
126+
[./test/index.test.ts](./test/index.test.ts) checks the three lists agree.
127+
- Node ≥ 18 is supported (`engines`); CI tests Node 22 and 24.
128+
129+
## Key dependencies
130+
131+
| Dependency | Version | Notes |
132+
| ------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------- |
133+
| `blusc` | 0.0.6 | Pure-Rust Blosc (Blosc2 API, Blosc1-compatible output) — the `Blosc` codec |
134+
| `lz4_flex` | ~0.12 | Pure-Rust LZ4 block compression — the `LZ4` codec |
135+
| `zstd` | ~0.13 (no default features) | Zstd — the `Zstd` codec |
136+
| `wasm-bindgen` / `js-sys` | 0.2 / 0.3 | WASM boundary |
137+
| `fflate` | ^0.8 (peer) | Pure-JS gzip/zlib — the `GZip` and `Zlib` codecs |
138+
| `vite-plus` (`vp`) | 0.2 | Build / test / check driver |
139+
| `vitest` | 4 | Test runner |
140+
| `tinybench` | 6 | Benchmark timing (benchmark package) |
141+
| `numcodecs` (npm) | ^0.3.2 (dev) | **numcodecs.js** — the head-to-head baseline in [./benchmark/](./benchmark/) |
142+
| `numcodecs` (PyPI) | 0.16.5 | **Python numcodecs** — wrote [./test/fixtures/](./test/fixtures/); never installed, `uv` fetches it pinned |

‎CODE_OF_CONDUCT.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Builder's Code v1.0
2+
3+
A Code of Conduct for people who build things.
4+
5+
---
6+
7+
## The Rule
8+
9+
**Stay professional. Stay technical.**
10+
11+
## Expected
12+
13+
- Contribute constructively.
14+
- Respect others' time and work.
15+
- Focus on the work and its technical merit.
16+
17+
## Not Welcome
18+
19+
- Harassment, name-calling, or personal attacks.
20+
- Trolling, spamming, or derailing discussions.
21+
- Discussions about contributors rather than their contributions.
22+
23+
## Enforcement
24+
25+
Violations result in:
26+
27+
1. **Warning** - First offense.
28+
2. **Temporary suspension** - Repeated or serious violations.
29+
3. **Permanent ban** - Continued violations.
30+
31+
Maintainers can remove, block, or ban anyone who disrupts the project.
32+
33+
---
34+
35+
## License
36+
37+
This Code of Conduct — the text of this file — is dedicated to the public domain
38+
under [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/), so
39+
other projects may reuse it freely. It does not change the license of the rest of
40+
the repository, which is [MIT](LICENSE.txt).

‎README.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
<p align="center">
2+
<picture>
3+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/rumcodecs-logo-dark.svg" />
4+
<img src="docs/assets/rumcodecs-logo.svg" alt="rumcodecs" width="160" />
5+
</picture>
6+
</p>
7+
8+
<h1 align="center">rumcodecs</h1>
9+
10+
<p align="center">
11+
<a href="https://github.com/fideus-labs/rumcodecs/actions/workflows/ci.yml"><img src="https://github.com/fideus-labs/rumcodecs/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
12+
<a href="https://github.com/fideus-labs/rumcodecs/actions/workflows/rust-ci.yml"><img src="https://github.com/fideus-labs/rumcodecs/actions/workflows/rust-ci.yml/badge.svg" alt="Rust CI" /></a>
13+
<a href="https://www.npmjs.com/package/@fideus-labs/rumcodecs"><img src="https://img.shields.io/npm/v/%40fideus-labs%2Frumcodecs" alt="npm" /></a>
14+
<a href="LICENSE.txt"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License" /></a>
15+
</p>
16+
17+
<p align="center">
18+
<strong>Buffer compression codecs for the browser and Node.js — numcodecs, reimplemented in Rust and WebAssembly.</strong>
19+
</p>
20+
21+
<p align="center">
22+
A drop-in replacement for
23+
<a href="https://github.com/manzt/numcodecs.js">numcodecs.js</a>: the same
24+
codec classes with the same API and the same byte formats, backed by Rust
25+
compiled to WASM (SIMD128) via
26+
<a href="https://crates.io/crates/blusc">blusc</a>. Built for
27+
<a href="https://zarr.dev/">Zarr</a> — chunks written by Python
28+
<code>numcodecs</code> decode in the browser, pinned by fixtures Python
29+
actually wrote.
30+
</p>
31+
32+
## 📦 The codecs
33+
34+
| Codec | Backend | Format |
35+
| ----------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
36+
| **`Blosc`** | [blusc](https://crates.io/crates/blusc) (Rust → WASM) | Blosc meta-compressor: `blosclz` / `lz4` / `lz4hc` / `snappy` / `zlib` / `zstd` with shuffle and bitshuffle filters |
37+
| **`LZ4`** | [lz4_flex](https://crates.io/crates/lz4_flex) (Rust → WASM) | numcodecs framing — 4-byte little-endian original size + LZ4 block |
38+
| **`Zstd`** | [zstd](https://crates.io/crates/zstd) (Rust → WASM) | Standard Zstd frames |
39+
| **`GZip`** | [fflate](https://github.com/101arrowz/fflate) (pure JS) | gzip |
40+
| **`Zlib`** | [fflate](https://github.com/101arrowz/fflate) (pure JS) | zlib |
41+
42+
Every class carries the numcodecs.js surface — static `codecId`, static
43+
`fromConfig`, the same constructor signatures and static members
44+
(`Blosc.SHUFFLE`, `Zstd.MAX_CLEVEL`, …) — and
45+
[`test/compat.test.ts`](test/compat.test.ts) asserts it stays that way.
46+
Compatibility with the _Python_ side is checked against real chunks: the
47+
buffers in [`test/fixtures/`](test/fixtures/) were written by Python
48+
`numcodecs`, and [`test/interop.test.ts`](test/interop.test.ts) decodes every
49+
one of them. That pins the Python → JavaScript direction; the reverse is the
50+
same shared byte format but is not exercised here, since asserting it would
51+
mean running Python in CI. The WASM module loads lazily on first
52+
encode/decode, so the pure-JS codecs never pay for it.
53+
54+
## 🚀 Quick start
55+
56+
```sh
57+
npm install @fideus-labs/rumcodecs
58+
# fflate is a peer dependency, needed only for GZip and Zlib
59+
npm install fflate
60+
```
61+
62+
```ts
63+
import { Blosc } from "@fideus-labs/rumcodecs";
64+
65+
const codec = new Blosc(5, "zstd", Blosc.SHUFFLE);
66+
const compressed = await codec.encode(data);
67+
const restored = await codec.decode(compressed);
68+
```
69+
70+
Each codec is also a subpath export, so bundlers ship only what you use:
71+
72+
```ts
73+
import Blosc from "@fideus-labs/rumcodecs/blosc";
74+
75+
// The numcodecs config shape works unchanged — e.g. straight from Zarr metadata.
76+
const codec = Blosc.fromConfig({ id: "blosc", cname: "lz4", clevel: 5, shuffle: 1 });
77+
```
78+
79+
## 📊 Benchmarks
80+
81+
The [`benchmark/`](benchmark/) workspace package runs rumcodecs head-to-head
82+
against numcodecs.js on the same generated arrays
83+
([tinybench](https://github.com/tinylibs/tinybench), multiple sizes and
84+
dtypes):
85+
86+
```sh
87+
pnpm bench # console table with speedups
88+
pnpm bench:report # write JSON + markdown report
89+
```
90+
91+
## 🛠️ Development
92+
93+
### Prerequisites
94+
95+
- Rust 1.91+ with [wasm-pack](https://rustwasm.github.io/wasm-pack/)
96+
- Node 18+ (CI tests 22 and 24) and [pnpm](https://pnpm.io/)
97+
98+
### Setup
99+
100+
```sh
101+
git clone https://github.com/fideus-labs/rumcodecs
102+
cd rumcodecs
103+
pnpm install # runs prepare: builds WASM (web + node) and the bundle
104+
pnpm test
105+
```
106+
107+
### Repository structure
108+
109+
```text
110+
rumcodecs/
111+
├── src/ # TypeScript codec classes (Blosc, GZip, Zlib, LZ4, Zstd)
112+
├── crates/rumcodecs-wasm/ # Rust WASM bindings: blusc, lz4_flex, zstd
113+
├── pkg/ # wasm-pack output (generated)
114+
├── test/ # vitest suites: round-trip, numcodecs API, interop fixtures
115+
├── benchmark/ # head-to-head benchmark vs numcodecs.js
116+
└── .github/workflows/ # ci.yml, rust-ci.yml, release.yml
117+
```
118+
119+
### Commands
120+
121+
| Command | Purpose |
122+
| ------------------------------------------ | ------------------------------------------------------------------------------------------- |
123+
| `pnpm test` | Run all vitest suites (round-trip, per-codec, compat, interop) |
124+
| `pnpm check` | Lint, format, and typecheck (oxlint / oxfmt / tsc via `vp`) |
125+
| `pnpm build` | Bundle with [vite-plus](https://www.npmjs.com/package/vite-plus) and emit type declarations |
126+
| `pnpm build:wasm` / `pnpm build:wasm:node` | Rebuild the WASM package for web / Node (SIMD128) |
127+
| `pnpm bench` | Benchmark against numcodecs.js |
128+
| `cargo test -- --test-threads=1` | Rust unit tests for the WASM crate |
129+
130+
## 🤝 Contributing
131+
132+
Start with [AGENTS.md](AGENTS.md) for the repository map and conventions, and
133+
the [open issues](https://github.com/fideus-labs/rumcodecs/issues) for what to
134+
pick up next. All participation is governed by our
135+
[Code of Conduct](CODE_OF_CONDUCT.md).
136+
137+
## 📄 License
138+
139+
MIT — see [LICENSE.txt](LICENSE.txt). Copyright (c) Fideus Labs LLC.
140+
141+
rumcodecs reimplements the API of
142+
[numcodecs.js](https://github.com/manzt/numcodecs.js) (MIT, © Trevor Manz) so
143+
existing Zarr tooling works unchanged. Blosc support comes from the pure-Rust
144+
[blusc](https://crates.io/crates/blusc) implementation of the
145+
[Blosc](https://www.blosc.org/) meta-compressor; LZ4 from
146+
[lz4_flex](https://crates.io/crates/lz4_flex); Zstd from the
147+
[zstd](https://crates.io/crates/zstd) crate.

0 commit comments

Comments
 (0)