feat(zql-benchmarks): run the ZQL benchmarks on React Native by arv · Pull Request #6507 · rocicorp/mono

arv · 2026-09-08T12:59:12Z

Runs the existing ZQL benchmarks on a real device, so we can see what the IVM
pipeline actually costs on Hermes rather than on a desktop JIT.

Stacked on the Replicache RN harness that merged in #6503. Two thirds of that
runner had no Replicache in it, so this lifts the reusable half into
tools/rn-bench and has both packages drive it through a config object.

What's here

tools/rn-bench — the extracted harness. benchmark.ts, harness.ts
and format.ts move verbatim; hermes-trace.ts moves with its one
hard-coded global name becoming a parameter. runner.ts is the old
perf-rn.ts behind an RnBenchConfig (root dir, global name, variant flag,
extra routes, benchmark list). replicache-perf/src/perf-rn.ts drops from
915 lines to 52 — its SQLite backends and /tmcw.json route are now config,
not core.
A shim for shared/src/bench.ts — the benchmarks call bench/describe/
use, which wrap mitata and Vitest, neither of which runs on Hermes. The RN
esbuild swaps in a stand-in via onResolve, so the seven .bench.ts files
compile unchanged and keep running under Vitest exactly as before. It
reproduces mitata's generator protocol (yield a body, resume for cleanup)
and honours max_samples, which the push benches rely on to avoid
exhausting the heap.
packages/zql-benchmarks/rn — the Expo app, with its own bundle id so it
doesn't overwrite the Replicache one on the device.
--repeat N — runs the whole queue N times and reports the median plus
the observed spread. A single run on an emulator is worth about ±3%, so a
smaller difference than that can't be resolved from one before/after pair.
This is what made the follow-up perf work measurable.
hermes-eval.mjs — evaluates a file in the running app's Hermes runtime
over the CDP connection the profiler already uses. A micro-benchmark loop
that needs a rebuild, reinstall and relaunch per variant is too slow to think
with; this makes it seconds.

Two fixes worth calling out

The runner only rebuilt the bundle under --profile, so a plain run measured
whatever was left in out/. That reports the previous revision's numbers,
which looks exactly like "the change did nothing" — and a null result is the
one outcome nobody re-checks. It cost me a measurement before I caught it.
build-rn.ts had the same latent mainFields bug fix(replicache-perf): resolve main fields when bundling for React Native #6504 just fixed in
replicache-perf. Nothing in this bundle's graph hits it today, but there's
no reason for the two builds to differ.

What it says

ZQL on Hermes is roughly 10–16x slower than on Node/V8, though that conflates
engine and device — Replicache's path showed about 8x on the same comparison.
Profiling the slowest case pointed straight at row comparison, which is what
the companion PR acts on.

Verification

pnpm --filter replicache-perf run perf and perf:rn still work after the
extraction (this is the part most at risk).
pnpm --filter zql-benchmarks run bench — the Vitest/mitata path is
untouched; the shim is only swapped in by the RN build.
Full runs on the Android emulator, both --repeat 3 and --profile.
check-types, lint, format on all touched packages, plus root lint and
syncpack lint.

Not here

The two end-to-end zero-client cases from the plan. zeroForTest
transitively imports Vitest, so they need a new Zero({cacheURL: null}) built
by hand; the ZQL-level suite is useful on its own and I'd rather not hold it.

The Replicache harness (#6503) could only run Replicache's own benchmarks. The more interesting mobile question is a layer up: how ZQL's materialize, pipeline building and IVM behave on Hermes, which nothing measured. Extracts the generic half of that harness and reuses the existing benchmarks rather than rewriting them. tools/rn-bench (new package) Everything that was never Replicache-specific: benchmark.ts, harness.ts, format.ts, the two result formatters and hermes-trace.ts move verbatim, and the runner is parameterised behind an RnBenchConfig — rootDir, the harness global name, an optional variant axis (Replicache's kv backends), extra control-server routes (its 9.7 MB tmcw fixture) and the benchmark list. replicache-perf/src/perf-rn.ts drops from 915 lines to 52. Running mitata benchmarks on Hermes The existing .bench.ts files call bench/describe/use from shared/src/bench.ts, which wraps mitata and Vitest — neither runs on Hermes. tools/rn-bench/src/ mitata-shim.ts reimplements those three names on the Benchmark shape, including mitata's generator protocol (setup / yield body / cleanup), and the RN build swaps it in with an esbuild onResolve hook. An onLoad hook stamps each module with its filename so benchmarks from different files land in different groups; several declare a suite called `push`. Each rep runs the body max_samples times (100 by default), so a figure means "N iterations" rather than one operation. Both the plain and generator forms loop, so everything in a run is on one scale. packages/zql-benchmarks src/rn.ts registers the seven suites that need no Node — ivm-memory, array-view-relationships, array-view-transaction, cold-boot-hydration, memory-ivm-deopt, debug-row-vended, query-hash — 83 benchmarks covering materialize, the query builder, pipeline building, IVM fetch and IVM push. The Node side reads its list from the built bundle rather than re-importing the benchmarks, so the two can never drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…l engine A micro-benchmark loop that needs a rebuild, reinstall and relaunch per variant is too slow to think with, and V8 results do not transfer — several candidates that looked good on paper were far slower on Hermes. This reuses the CDP connection the profiler already opens to evaluate a file in the running app's runtime and print what it returns, so a candidate can be measured on the real engine in seconds. It is how the compare-utf8 first-code-unit probe was found to be a pessimization on Hermes (rocicorp/compare-utf8#11): four alternatives were measured this way, and the one that looked most promising — bisecting to the first differing index and checking a single position, which is precise rather than conservative — turned out to be three times slower than the regex it was meant to replace. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ort the median A single run on an emulator is worth about +/-3%, so a change smaller than that cannot be resolved from one before/after pair. --repeat runs the whole queue N times and reports the median per benchmark plus the observed spread, which makes the noise floor visible instead of implicit.

The bundle was only rebuilt under --profile, so a plain run measured whatever revision happened to be in out/. That silently reports the previous revision's numbers, which looks exactly like "the change did nothing" -- and a null result is the one outcome nobody thinks to re-check.

Same latent bug #6504 fixed in replicache-perf's build: platform 'neutral' resolves no main fields at all, so a dependency shipping only a legacy entry fails to resolve. Nothing in this bundle's dependency graph hits it today -- they all ship exports maps -- but the sibling build documents the trap and there is no reason for the two to differ.

vercel · 2026-09-08T12:59:20Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
replicache-docs	Ready	Preview	Sep 8, 2026 2:58pm UTC
zbugs	Ready	Preview	Sep 8, 2026 2:58pm UTC

Copilot

🟡 Changes recommended

The shared benchmark stats and mitata shim contain correctness bugs that can skew reported timings/results (mean calculation includes discarded samples; non-generator benches can be under-measured).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the existing ZQL benchmark suite to run on real React Native devices (Hermes) by extracting the reusable React Native benchmark runner/harness into tools/rn-bench, adding an Expo host app for zql-benchmarks, and updating replicache-perf to consume the shared harness code.

Changes:

Extract the React Native benchmark runner, formatting, and harness utilities into a shared workspace tool (tools/rn-bench).
Add a ZQL React Native build (tool/build-rn.ts) that bundles existing .bench.ts files unchanged via a shared/src/bench.ts shim swap, plus a new Expo host app at packages/zql-benchmarks/rn/.
Refactor replicache-perf to reuse the shared harness/format/result types and configure package-specific variants/routes via RnBenchConfig.

File summaries

File	Description
tools/rn-bench/tsconfig.json	TS config for the extracted RN benchmark harness tool.
tools/rn-bench/package.json	New workspace tool package providing shared RN benchmark runner/harness.
tools/rn-bench/hermes-eval.mjs	CLI utility to evaluate JS in a running Hermes runtime via CDP.
tools/rn-bench/src/runner.ts	Shared RN runner: device drivers, control server, repeats, profiling, output formats.
tools/rn-bench/src/mitata-shim.ts	RN stand-in for `shared/src/bench.ts` to run existing `.bench.ts` under the harness.
tools/rn-bench/src/hermes-trace.ts	Generalize Hermes profiling to use a configurable harness global name.
tools/rn-bench/src/harness.ts	Shared harness wrapper around benchmark discovery and `runBenchmark`.
tools/rn-bench/src/github-action-benchmark.ts	Emit GitHub Action Benchmark JSON entries for RN results.
tools/rn-bench/src/format.ts	Shared formatting for “replicache” and “benchmarkJS” style outputs.
tools/rn-bench/src/benchmark.ts	Shared benchmark execution + stats used by both web and RN harnesses.
tools/rn-bench/src/bencher-metric-format.ts	Emit Bencher Metrics Format output for RN benchmark results.
pnpm-lock.yaml	Adds new workspace importer for `tools/rn-bench` and new deps used by zql-benchmarks RN build.
packages/zql-benchmarks/tsconfig.json	Adjusts `rootDir` to accommodate new build/run structure.
packages/zql-benchmarks/tool/build-rn.ts	Esbuild bundler for RN benchmark bundle + bench shim plugin.
packages/zql-benchmarks/src/rn.ts	RN entrypoint importing the subset of ZQL `.bench.ts` runnable on device.
packages/zql-benchmarks/src/perf-rn.ts	Package-specific `perf:rn` entry configuring `runRnBench` for ZQL.
packages/zql-benchmarks/rn/tsconfig.json	Expo app TS config (kept out of workspace).
packages/zql-benchmarks/rn/README.md	Run instructions and rationale for keeping the app out of the workspace.
packages/zql-benchmarks/rn/pnpm-workspace.yaml	Stops `pnpm install` from walking up into the monorepo workspace.
packages/zql-benchmarks/rn/package.json	Expo host app package manifest (isolated deps).
packages/zql-benchmarks/rn/index.ts	Expo root registration for the host app.
packages/zql-benchmarks/rn/App.tsx	Host app that polls the runner, runs benchmarks, posts results, exposes `__zqlPerf`.
packages/zql-benchmarks/rn/app.json	Expo config incl. separate bundle ID + cleartext traffic for local control server.
packages/zql-benchmarks/rn/.gitignore	Ignore native build outputs and copied-in generated benchmark artifacts.
packages/zql-benchmarks/package.json	Adds RN build/run scripts and deps on `esbuild` + `rn-bench`.
packages/replicache-perf/src/runner.ts	Reuses shared formatting/metrics helpers from `tools/rn-bench`.
packages/replicache-perf/src/rn.ts	Switches RN harness assembly to `tools/rn-bench` harness + shared types/formatters.
packages/replicache-perf/src/perf.ts	Switches web harness assembly to shared `makeHarness`.
packages/replicache-perf/src/perf-rn.ts	Collapses RN runner to package-specific `RnBenchConfig` (variants, routes, configure expr).
packages/replicache-perf/src/index.ts	Reuses shared `runBenchmark` and formatting from `tools/rn-bench`.
packages/replicache-perf/src/benchmarks/storage.ts	Updates benchmark type import to shared `tools/rn-bench` Benchmark type.
packages/replicache-perf/src/benchmarks/replicache.ts	Updates benchmark/bencher type imports to shared `tools/rn-bench`.
packages/replicache-perf/src/benchmarks/map-loop.ts	Updates benchmark type import to shared `tools/rn-bench`.
packages/replicache-perf/src/benchmarks/idb.ts	Updates benchmark/bencher type imports to shared `tools/rn-bench`.
packages/replicache-perf/src/benchmarks/hash.ts	Updates benchmark type import to shared `tools/rn-bench`.
packages/replicache-perf/src/benchmarks/compare-utf8.ts	Updates benchmark type import to shared `tools/rn-bench`.
packages/replicache-perf/package.json	Adds workspace dependency on `rn-bench`.
oxlint.base.ts	Excludes the new zql RN app from type-aware oxlint traversal.
.syncpackrc	Ignores zql RN app package for dependency version alignment checks.

Review details

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Suppressed comments (1)

tools/rn-bench/src/runner.ts:88

Repo convention is to spell optional properties as prop?: T | undefined (e.g. packages/zql/src/builder/builder.ts:52-61). The optional callback/record fields here omit the explicit | undefined, making the public config surface inconsistent and potentially surprising under exactOptionalPropertyTypes.

Files reviewed: 33/40 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Three fixes from review. `meanMs` divided the sum of *every* run by the count of runs that survived the two-sample discard, so it was not the mean of anything -- it came out above p95 on every benchmark, which is what `avg=245.86 ms` next to `95%=196.15` was telling us all along. Sum the retained samples instead. `sum` itself stays: it is also the loop's run-time budget. The mitata shim called `fn()` once to tell a generator body from a plain one, before starting the clock. For an async body that is harmless, but a synchronous one had already run to completion, so 17 of the ported benchmarks (16 in cold-boot-hydration, 1 in debug-row-vended) reported `iterations - 1` executions while the generator form reported `iterations`. Let the probe settle outside the timed region and measure a full `iterations`, which also keeps that cold first execution out of the measurement where it belongs. Spell the optional fields of RnBenchConfig and Options as `?: T | undefined`, per AGENTS.md.

Copilot

🔵 Needs a closer look

The new BMF output currently reports latency as “throughput” and the benchmark sample-discard logic skews the computed statistics, so benchmark reporting needs correction before relying on the results.

Review details

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Suppressed comments (1)

tools/rn-bench/src/benchmark.ts:101

The summary stats computed below are skewed because the discard step above does times.sort(...) followed by times.splice(0, 2), which removes the two fastest samples even though the comment says “Remove two slowest”. This biases percentiles/mean upward and contradicts the intended warmup-discard behavior.

Files reviewed: 34/40 changed files
Comments generated: 0 new
Review effort level: Lite

Two more from review, both pre-existing in code this PR moves. The discard step sorted ascending and then spliced from the *front*, so it removed the two fastest samples while the comment said "Remove two slowest. Treat them as JIT warmup." Warmup runs are the slow ones; trimming the low tail can only bias every statistic upward. Trim the tail the comment meant. Bencher Metric Format reported `meanMs` under `throughput`. The value is a time in milliseconds, so lower is better; filed as throughput, Bencher reads the direction backwards and a regression shows as an improvement. No workflow consumes this format yet, so renaming the measure costs no tracked history. (cherry picked from commit c75aa8fa76a62197770fe29eb4d0c593031584e4)

arv · 2026-09-08T15:49:47Z

Both findings from the latest review are fixed in 7f11a21. Neither had an inline thread to reply on, hence this comment.

Discard direction (benchmark.ts:101) — correct, and it was the worse of the two. times is sorted ascending and splice(0, 2) took from the front, so it dropped the two fastest samples while the comment said "Remove two slowest. Treat them as JIT warmup." Warmup runs are the slow ones, so trimming the low tail can only bias every statistic upward. Now splice(-2).

BMF measure — the value is meanMs, a time in milliseconds, so lower is better. Filed under throughput Bencher reads the direction backwards and every regression shows up as an improvement. Renamed to latency; I checked that no workflow consumes this format yet, so it costs no tracked history.

Both are pre-existing bugs in code this PR moves out of replicache-perf rather than anything new here, but they are worth fixing while the file is in flight — and together with the meanMs fix in 4df6862 they change reported numbers for replicache-perf too, so its historical figures are not comparable across this PR.

Worth flagging what the discard bug cost: hydrate: issues only reported avg=245.86 ms next to 95%=196.15 in every run I did, and I read past it repeatedly before this review pointed at the cause.

🤖 Addressed by Claude Code

arv and others added 5 commits September 8, 2026 14:57

Copilot AI balanced review requested due to automatic review settings September 8, 2026 12:59

arv requested review from 0xcadams and aboodman as code owners September 8, 2026 12:59

vercel Bot deployed to Preview – zbugs September 8, 2026 12:59 View deployment

Copilot started reviewing on behalf of arv September 8, 2026 12:59 View session

vercel Bot deployed to Preview – replicache-docs September 8, 2026 13:00 View deployment

arv mentioned this pull request Sep 8, 2026

perf(zql): probe the last view entry before binary-searching #6508

Merged

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread tools/rn-bench/src/mitata-shim.ts

Comment thread tools/rn-bench/src/runner.ts Outdated

arv requested review from tantaman and removed request for 0xcadams and aboodman September 8, 2026 13:05

vercel Bot deployed to Preview – replicache-docs September 8, 2026 13:15 View deployment

vercel Bot deployed to Preview – zbugs September 8, 2026 13:15 View deployment

arv mentioned this pull request Sep 8, 2026

perf(zql): stop MemorySource#fetch being a generator #6509

Draft

arv requested a lite review from Copilot September 8, 2026 14:46

Copilot started reviewing on behalf of arv September 8, 2026 14:46 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

vercel Bot deployed to Preview – zbugs September 8, 2026 14:58 View deployment

vercel Bot deployed to Preview – replicache-docs September 8, 2026 14:58 View deployment

arv added this pull request to the merge queue Sep 8, 2026

Merged via the queue into main with commit e5dfd06 Sep 8, 2026
35 of 36 checks passed

arv deleted the arv/zql-benchmarks-rn branch September 8, 2026 16:15

Navigation Menu

Uh oh!

feat(zql-benchmarks): run the ZQL benchmarks on React Native - #6507

feat(zql-benchmarks): run the ZQL benchmarks on React Native#6507
arv merged 7 commits into
mainfrom
arv/zql-benchmarks-rn

arv commented Sep 8, 2026

Uh oh!

vercel Bot commented Sep 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

arv commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

arv commented Sep 8, 2026

What's here

Two fixes worth calling out

What it says

Verification

Not here

Uh oh!

vercel Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Files not reviewed (1)

Suppressed comments (1)

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Files not reviewed (1)

Suppressed comments (1)

Uh oh!

arv commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Sep 8, 2026 •

edited

Loading