[RNE Rewrite] feat: add SDXS text-to-image pipeline by msluszniak · Pull Request #1307 · software-mansion/react-native-executorch

msluszniak · 2026-07-14T14:45:36Z

Description

Adds the SDXS-512-DreamShaper text-to-image pipeline to the rewrite CV extension (issue #1244 and #939). SDXS is a distilled single-step diffusion model, so the pipeline is a single multi-method .pte (encode / denoise / decode) driven by a lightweight TypeScript scheduler + CLIP tokenizer.

sdxsTextToImage task (createSdxsTextToImage): tokenize → encode → denoise → exact single-step DEIS scheduler (linear in latents+eps) → TAESD decode → in-memory RGBA ImageBuffer.
useTextToImage hook: two-resource download (combined .pte + tokenizer.json).
models.textToImage.SDXS_512_DREAMSHAPER: XNNPACK_FP32 (default, universal) and COREML_FP16 (iOS).
math.mulberry32 / math.randomNormal: seeded uniform generator + Box–Muller normal draw.
apps/computer-vision Text-to-Image demo screen.

Since the pipeline targets exactly one model, values fixed by the export (shapes, scheduler and decoder scalars) are constants in the task file rather than options — all variants differ only in which .pte they load. The convention is written up in .agents/skills.

Models exported and published to software-mansion/react-native-executorch-sdxs-512-dreamshaper (v0.10.0). The exported pipeline reproduces the reference diffusers output pixel-for-pixel (fp32, mean pixel err ~1.8e-5); CoreML fp16 is ~99.7% delegated at mean pixel err ~0.002.

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Run apps/computer-vision on a physical device.
Open Text to Image and pick a variant (CoreML FP16 is offered on physical iOS devices only).
Enter a prompt and tap Generate; a 512×512 image renders.

Verified on a physical iPhone 17 Pro (COREML_FP16) and a Galaxy S24 Ultra (XNNPACK_FP32).

Screenshots

Related issues

#1244, #939

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

SDXS is highly quantization-sensitive. Quantizing the UNet convolutions produces noise, so an 8da4w variant was dropped (poorer quality, and no speed win over fp32 since both are conv-bound).

No MLX variant. The MLX backend does not support aten.native_group_norm, which a diffusion UNet uses in every ResBlock, so denoise fragments into 28 subgraphs and thrashes the GPU/CPU boundary — measured ~3x slower than CoreML on device, with worse quality. The split is structural, not quantization-related: an unquantized fp32 export fragments identically, so no dtype fixes it. Worth revisiting once MLX supports native_group_norm.

The TS scheduler stays in TypeScript. It is two single passes over 16,384 floats, once per image (~0.34 ms measured, ~2.4 ms at a conservative Hermes penalty) against a UNet forward that dominates by orders of magnitude, so a C++ port is not worth it.

barhanc

Checked only lib implementation for now. In general the implementation is incredibly clean. I mostly marked some small nits. The one more important issue worth discussing is how we should handle configuration options for single-model-specific pipelines, something I also commented about in the VAD PR.

barhanc

I've tested the example app and it works great. The difference compared to our previous example is staggering. Added only some small nits. In general great work on this one.

barhanc · 2026-07-17T12:29:43Z

+ * @param options Distribution parameters.
+ * @param options.mean The mean of the distribution. Defaults to 0.
+ * @param options.std The standard deviation of the distribution. Defaults to 1.
+ * @param options.seed The seed for the underlying generator. Defaults to 0.


I feel like not providing seed should default to random results (so like the Date.now trick).

Done — randomNormal now seeds from Date.now() when no seed is passed, so omitting it gives random results.

barhanc · 2026-07-17T12:37:54Z

+    if (!sync) setIsProcessing(true);
+    setError(null);
+    try {
+      const seed = Date.now() % 0x80000000;


This shouldn't be needed for optional seed, i.e. not passing seed == random results.

Done — the demo calls generate(prompt) with no seed now.

Add the SDXS-512-DreamShaper text-to-image pipeline to the rewrite CV extension: a single-step, no-CFG diffusion runner built on the generic model runtime plus a CLIP tokenizer. - sdxsTextToImage task: encode -> denoise (TS 1-step scheduler) -> decode, returning an in-memory RGBA ImageBuffer (rendered via Skia in apps). - useTextToImage hook: whisper-style multi-resource download (combined .pte program + tokenizer.json). - models.textToImage.SDXS_512_DREAMSHAPER registry entry (xnnpack fp32 + 8da4w) and index exports. - computer-vision demo screen + drawer/menu wiring. Model export (single multi-method .pte) is tracked separately; the scheduler-related opts constants are pinned during export. Refs #1244, #939

Remove the XNNPACK 8da4w SDXS variant (no speed win over fp32 since both are conv-bound, and noticeably poorer quality on this quant-sensitive model). Add iOS backends: CoreML fp16 (quality-preserving) and MLX int4. - models.textToImage.SDXS_512_DREAMSHAPER: XNNPACK_FP32 (default), COREML_FP16, MLX_INT4 - demo screen exposes the CoreML/MLX options (iOS-only) Refs #1244, #939

Tokenizer moved from tokenizer/ subdir to the repo root to match the rewrite model-repo convention (root README.md, config.json, tokenizer.json, tokenizer_config.json; per-backend config.json).

The CoreML and MLX SDXS variants delegate to the ANE/GPU, which the iOS Simulator cannot provide, so only expose them when running on a physical device (expo-device Device.isDevice) instead of any iOS target.

- Rename SdxsModel -> SdxsTextToImageModel - Give noiseCoeff its own doc; split the shared coefficient JSDoc - Remove redundant encodePrompt/toCleanLatents comments; tighten the RNG one

- Move the seeded RNG into extensions/math as mulberry32 (uniform) + randomNormal (Box-Muller), separating the generator from the transform - randomNormal takes mean/std, folding away the initNoiseSigma scaling pass - Inline the mis-named encodePrompt and the single-use toCleanLatents - Use tLatents.numel instead of a recomputed latentNumel - Apply the getData and compact-destructuring suggestions Verified bit-identical noise vs the previous implementation across seeds.

All three shipped variants (XNNPACK/CoreML/MLX) used an identical opts object, so none of these values were ever per-model configuration: the shapes are static in the .pte and the scheduler/decoder scalars are pinned by the export. Exposing them as options implied knobs that only ever have one valid value. - Collapse SdxsOptions into constants in sdxsTextToImage.ts; the model type is now just { modelPath, tokenizerPath } - Drop numInferenceSteps and the denoise loop: the linear DEIS coefficients are only valid for the single distilled step at t=999, so looping them would have been silently incorrect rather than merely slower - Keep validateModelSchema, now asserting against those constants

Captures the convention settled while reviewing the SDXS pipeline, so it outlives this PR and applies to the other single-model pipelines (VAD, supertonic TTS) facing the same question. - add-task-pipeline: new Principle 6 bucketing every param by who varies it (variant -> option, export-pinned -> const, per-call -> argument), plus the identical-options smell test and the fake-generality warning - model-schema-validation: Recipe 4 for single-model static exports, and make explicit that a rigid .pte argues for asserting, not for skipping

The MLX backend has no support for aten.native_group_norm, which a diffusion UNet uses in every ResBlock, so denoise fragments into 28 subgraphs and thrashes the GPU/CPU boundary -- ~3x slower than CoreML on device, with worse quality. The fragmentation is structural rather than quantization-related: an unquantized fp32 export fragments identically, so no dtype avoids it. Ship XNNPACK + CoreML and revisit if MLX gains native_group_norm.

- randomNormal(size, { mean?, std?, seed? }) — numpy-style options object, seeds mulberry32 internally (verified bit-identical to the prior signature) - useTextToImage reports the model download progress directly; the tokenizer is negligible next to the multi-GB .pte - Drop the redundant variants comment from models.ts

- generate/generateWorklet seed is now optional, defaulting to Date.now() (in the worklet global allowlist) so callers don't need Date.now tricks - Inline the decode output scale as { alpha: 255.0 }, matching the other CV tasks, and drop the OUT_ALPHA/OUT_BETA constants - Group TIMESTEP with the DEIS coefficients under the one comment that covers them; drop redundant constant comments - Trim the .pte implementation detail from the public API docs

- randomNormal seeds mulberry32 from Date.now() when no seed is given, so the demo no longer needs to compute one and the task passes seed through directly - Remove the models.ts MLX-omission comment; tracked in #1318 - Dedupe coeff in the cspell wordlist (added by both VAD and this branch)

barhanc

🚀

msluszniak self-assigned this Jul 14, 2026

msluszniak added the refactoring label Jul 14, 2026

msluszniak added this to the v0.10.0 milestone Jul 14, 2026

msluszniak linked an issue Jul 14, 2026 that may be closed by this pull request

[RNE Rewrite] CV - add text-to-image pipeline implementation #1244

Closed

msluszniak commented Jul 14, 2026

View reviewed changes

Comment thread packages/react-native-executorch/src/extensions/cv/tasks/sdxsTextToImage.ts Outdated

msluszniak marked this pull request as ready for review July 14, 2026 16:48

msluszniak requested a review from barhanc July 14, 2026 16:48

msluszniak removed this from the v0.10.0 milestone Jul 14, 2026

barhanc reviewed Jul 15, 2026

View reviewed changes

msluszniak requested a review from barhanc July 15, 2026 13:38

barhanc reviewed Jul 15, 2026

View reviewed changes

msluszniak requested a review from barhanc July 16, 2026 18:15

barhanc reviewed Jul 17, 2026

View reviewed changes

msluszniak added 13 commits July 17, 2026 19:32

[RNE Rewrite] chore(tti): point SDXS tokenizer at HF repo root

bcea413

Tokenizer moved from tokenizer/ subdir to the repo root to match the rewrite model-repo convention (root README.md, config.json, tokenizer.json, tokenizer_config.json; per-backend config.json).

Fix lint warnings: add cspell words and drop bitwise seed op

e3de23b

Gate CoreML/MLX variants to physical iOS devices only

963b79e

The CoreML and MLX SDXS variants delegate to the ANE/GPU, which the iOS Simulator cannot provide, so only expose them when running on a physical device (expo-device Device.isDevice) instead of any iOS target.

Address review: rename type, document noiseCoeff, trim helper comments

dd91214

- Rename SdxsModel -> SdxsTextToImageModel - Give noiseCoeff its own doc; split the shared coefficient JSDoc - Remove redundant encodePrompt/toCleanLatents comments; tighten the RNG one

Remove the constants-rationale comment per review

1f6ce96

msluszniak mentioned this pull request Jul 17, 2026

[RNE Rewrite] Add MLX delegates for models that only have CoreML #1318

Closed

8 tasks

msluszniak force-pushed the @ms/rewrite-text-to-image branch from 726b740 to 3a74142 Compare July 17, 2026 17:39

msluszniak requested a review from barhanc July 17, 2026 17:49

barhanc approved these changes Jul 17, 2026

View reviewed changes

msluszniak merged commit c601289 into rne-rewrite Jul 17, 2026
2 checks passed

msluszniak deleted the @ms/rewrite-text-to-image branch July 17, 2026 17:55

barhanc mentioned this pull request Aug 7, 2026

Export and quantize better text-to-image model #939

Closed

Uh oh!

Conversation

msluszniak commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barhanc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barhanc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barhanc Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

msluszniak Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

barhanc Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msluszniak Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

barhanc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msluszniak commented Jul 14, 2026 •

edited

Loading

barhanc Jul 17, 2026 •

edited

Loading