[RNE Rewrite] test(ts): add API tests for the TypeScript surface - #1355
Draft
msluszniak wants to merge 7 commits into
Draft
[RNE Rewrite] test(ts): add API tests for the TypeScript surface#1355msluszniak wants to merge 7 commits into
msluszniak wants to merge 7 commits into
Conversation
Every path through `src/` bottoms out in `__rnexecutorch_jsi__`, so stubbing those calls per test would only ever assert against the stub. Instead of stubbing, implement the native contract in JavaScript: typed-array-backed tensors with the real byte semantics, JS implementations of the math/cv/speech operators, and a `loadModel` that serves a program the test describes. Task pipelines therefore run end to end. Alongside it, an in-memory blob-util mock with a programmable server (status, body, Range support, and a gate to hold a download open) and a worklets mock that runs worklets inline. Native memory is not garbage collected, so the setup file asserts after every test that nothing allocated through the fake was left undisposed; each pipeline suite gets disposal coverage for free.
- `core/`: tensor byte semantics and copy windows, model execution and disposal, `wrapAsync` error propagation, and the spec matcher in depth — symbol binding, variant selection, runtime-constraint matching, and the authoring errors that fire before matching starts. - `fetcher/`: caching, forced re-download, byte-weighted progress, HTTP and transport failures, cancellation, requests shared between concurrent callers, iOS partial-file resume, and the Android DownloadManager backend. Telemetry gets its own suite, including the locale parsing that must not read a language-only tag as a country. - `extensions/`: box and point scaling under both resize modes, box decoding, and the seeded generators.
One suite per pipeline: which model signatures it accepts, that a mismatch is rejected with a message naming the mismatch, the postprocessing that is the pipeline's own work (softmax ordering, NMS suppression, argmax colormaps, sigmoid grayscale masks, coordinate scaling back through letterboxing), every option and per-call override, and that `dispose()` releases everything. The pipelines whose behavior depends on real model weights — Whisper, VAD, SDXS, keypoints, instance segmentation — get schema acceptance, rejection and full disposal instead, including Whisper's nested tokenizer and VAD pipeline. `tasks/constructionFailure.test.ts` records a leak the suites surfaced: a `create<Task>` factory that throws after `loadModel` abandons the native model, and the caller never receives a `dispose` to release it. The tests assert the current behavior so it flips loudly once fixed. Hook suites cover the lifecycle apps depend on: disposal on unmount and on config change, the create-after-unmount race, config identity by value, download cache hits, preventLoad, and errors surfaced through the shared field.
- A snapshot of every export, so an addition, rename or removal shows up in the diff of the pull request that causes it rather than in a user's app. - Registry rules that only fail on a device otherwise: https URLs on the software-mansion org, a pinned revision, the `modelname_backend_precision.pte` naming contract, a folder matching the backend suffix, and a default alias structurally identical to one of its own variants. - Label-array invariants, including the ImageNet duplicates that must stay because the array mirrors the model's output vocabulary. - Source-level conventions, parsed with the TypeScript compiler: the `'worklet'` directive on every JSI wrapper, and the core/extensions and hooks/native import boundaries from the architecture guide.
- `SpecMatch.dim(name)` with no expected kind is typed and documented to return
the raw `ConcreteDim`, but unwrapped it anyway — so `dim('N')` handed back a
number typed as an object, and `dims.any(...)` a tuple of them. Unwrap only
when a kind was asked for. Nothing in `src/` used the no-kind form.
- `getRegisteredBackends` calls straight into the JSI global without the
`'worklet'` directive the architecture guide requires, so it could not be
called from a worklet runtime like every other native wrapper.
- `randomNormal`'s default seed is `Date.now()`, whose millisecond resolution
means two unseeded calls in the same millisecond draw the identical sequence
— the opposite of what its own JSDoc promises. Mix in `Math.random`; an
explicit seed still wins, so reproducibility is unaffected.
Adds an `api-tests` job to CI — no native libraries, no simulator, no `.pte`, so the existing TypeScript-only setup action is all it needs. Adds an `add-api-tests` skill covering the fake runtime, the helpers, what to cover for a new pipeline or hook, and the leak-checking contract; wires it into the skills index, the architecture guide and the maintenance list; and adds the test step to the verify-and-build workflow and checklist. Also refreshes the hook example in `add-task-pipeline`, which still showed the `localPath` shape from before the resource fetcher landed.
Rebasing onto the supertonic TTS pipeline (#1317) surfaced two gaps the suites themselves reported: - The registry walk assumed every task config names a single `modelPath`. Supertonic assembles four `.pte` files under `modelPaths`, so its category looked empty and its variant group went unchecked. Recognize both shapes, and compare variants on the whole set of files they name. - The export snapshots record the new TTS surface. Additions only — no removal or rename.
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 10, 2026 06:29
49728d3 to
4755b35
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds TS API tests:
Also adds necessary skills.
To find details about this approach please follow:
__tests__/README.md.Three source fixes, each a case where the declared contract was violated in a way no test could express otherwise:
SpecMatch.dim(name)with no expected kind is typed to return the rawConcreteDim, but unwrapped it anyway, sodims.any(...)returned numbers typed as objects. Nothing insrc/used the no-kind form.getRegisteredBackendscalls into the JSI global without the'worklet'directive the architecture guide requires.randomNormal's default seed isDate.now(), so two calls in the same millisecond draw the identical sequence, the opposite of its JSDoc.One finding left unfixed and recorded. Every
create<Task>that throws afterloadModelabandons the native model, with nodisposeto release native model.tasks/constructionFailure.test.tsasserts the behavior across factories, andhooks/taskHooks.test.tsrecords the app-level shape. Worth separate PR.Introduces a breaking change?
Type of change
Tested on
Testing instructions
yarn workspace react-native-executorch test yarn typecheck yarn lintExpected: 23 suites, 1398 tests, 3 snapshots, 0 skipped; typecheck and lint clean.
yarn preparestill emits onlysrc/intolib/.Screenshots
N/A
Related issues
Closes #1352.
Checklist
Additional notes
Layout
core/fetcher/tasks/hooks/useModel,useResourceDownload, and the task hooks end to endextensions/api/support/Deliberately not covered: the numerical behavior of the native operators (that is
cpp/tests/, and duplicating it here would only test the fake); the long stateful worklets, whose behavior depends on real weights — Whisper, VAD and SDXS get schema acceptance, rejection and full disposal instead; and the worklet thread hop, since worklets run inline. The'worklet'directive convention that makes that hop possible is enforced by parsingsrc/with the TypeScript compiler.