feat: add PARAPHRASE_MULTILINGUAL_MINILM_L12_V2 text embeddings model - #1115
Conversation
78b5a13 to
f1341d2
Compare
9cd7623 to
27e7204
Compare
Benchmarks on iPhone 17 Pro and OnePlus 12 (~80-token input, 50 forwards) showed XNNPACK 8da4w Pareto-dominates the other three variants on both platforms — different from distiluse, where Core ML won iPhone via ANE. Drop the bare _V2, _COREML and _COREML_FP16 exports; ship only _8DA4W. Adds the corresponding rows to model-size, inference-time, and memory-usage docs (memory column uses a different methodology than the older distiluse rows, see PR body).
27e7204 to
2cfefc9
Compare
|
Would you rather keep 8da4w suffix or you are in favour of dropping it? cc: @chmjkb @NorbertKlockiewicz @barhanc |
barhanc
left a comment
There was a problem hiding this comment.
Model works, just this small comment regarding naming.
| /** | ||
| * @category Models - Text Embeddings | ||
| */ | ||
| export const PARAPHRASE_MULTILINGUAL_MINILM_L12_V2_8DA4W = { |
There was a problem hiding this comment.
I think for now it should be PARAPHRASE_MULTILINGUAL_MINILM_L12_V2_QUANTIZED so that we maintain consistent naming (and we should probably retroactively change the DISTILUSE_BASE_MULTILINGUAL_CASED_V2_8DA4W also). We had some discussion about model registry and naming (#612) so we can change the symbols all in one go when implementing that.
There was a problem hiding this comment.
agreed, I think that answers your question @msluszniak
There was a problem hiding this comment.
Ok, so I'll change the name for this model and for distiluse we will fix this in one holistic change.
…#1115) ## Description Adds the `paraphrase-multilingual-MiniLM-L12-v2` sentence-transformer model — the second multilingual embeddings model after distiluse, completing #945. Ships **only the XNNPACK 8da4w variant** under `MODEL_REGISTRY.ALL_MODELS` (see "Why a single variant" below). 384-d output, max 126 tokens, 50+ languages. Tokenizer is Unigram + Precompiled normalizer + Metaspace decoder — **requires the bumped `pytorch/extension/llm/tokenizers` runtime from #1114**, so this PR blocks on that landing first and should be rebased onto main once #1114 merges. HF repo: [software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2) (`v0.9.0` tag, layout mirrors distiluse). **Why a single variant** — TLDR 8da4w works faster then all and was also one of the smallest, without loss in precision. Longer answer: unlike distiluse, where Core ML fp32 won iPhone thanks to ANE acceleration, benchmarks on iPhone 17 Pro + OnePlus 12 (~80-token input, 50 measured forwards after 3 warmups) showed the XNNPACK 8da4w variant Pareto-dominates the other three on both platforms: faster than XNNPACK fp32, Core ML fp32 *and* Core ML fp16 on iPhone, and ~36% smaller steady-state memory footprint than the next-best variant. Likely cause: paraphrase-multilingual-MiniLM-L12-v2 is a smaller model (~118 M params, 12 layers) where Core ML's runtime doesn't push enough work onto ANE for the precision-conversion overhead to pay off. fp16 being slower than fp32 on Core ML for this model is a tell that the runtime is falling back to slower compute units. Shipping only `_8DA4W` keeps the public surface aligned with the data; if a future Core ML or model update flips the verdict, easy to add the other variants back. **Memory methodology note** — the new paraphrase row in `docs/docs/02-benchmarks/memory-usage.md` reports RSS / `phys_footprint` deltas from a clean app baseline (loaded − idle), captured on-device at the same conceptual point. The existing distiluse rows there (36 / 44 MB) come from an older measurement pass with a different (and not reconstructable from the diff) methodology, so the two rows are not directly comparable. A separate pass to re-measure distiluse and other rows with the same methodology would be a good follow-up. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. `cd apps/text-embeddings && npx expo run:ios` (or `run:android`). 2. Pick **"Multilingual Paraphrase (8da4w)"** in the model picker. 3. Add a sentence in one language, query with an aligned sentence in another (e.g. Polish "Słoneczko" against "It's so sunny outside!"). The cross-lingual pair should top the matches. ### Related issues Closes the paraphrase-multilingual half of #945 (the distiluse half landed in #1098). ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes Blocks on #1114.
…#1115) ## Description Adds the `paraphrase-multilingual-MiniLM-L12-v2` sentence-transformer model — the second multilingual embeddings model after distiluse, completing #945. Ships **only the XNNPACK 8da4w variant** under `MODEL_REGISTRY.ALL_MODELS` (see "Why a single variant" below). 384-d output, max 126 tokens, 50+ languages. Tokenizer is Unigram + Precompiled normalizer + Metaspace decoder — **requires the bumped `pytorch/extension/llm/tokenizers` runtime from #1114**, so this PR blocks on that landing first and should be rebased onto main once #1114 merges. HF repo: [software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2) (`v0.9.0` tag, layout mirrors distiluse). **Why a single variant** — TLDR 8da4w works faster then all and was also one of the smallest, without loss in precision. Longer answer: unlike distiluse, where Core ML fp32 won iPhone thanks to ANE acceleration, benchmarks on iPhone 17 Pro + OnePlus 12 (~80-token input, 50 measured forwards after 3 warmups) showed the XNNPACK 8da4w variant Pareto-dominates the other three on both platforms: faster than XNNPACK fp32, Core ML fp32 *and* Core ML fp16 on iPhone, and ~36% smaller steady-state memory footprint than the next-best variant. Likely cause: paraphrase-multilingual-MiniLM-L12-v2 is a smaller model (~118 M params, 12 layers) where Core ML's runtime doesn't push enough work onto ANE for the precision-conversion overhead to pay off. fp16 being slower than fp32 on Core ML for this model is a tell that the runtime is falling back to slower compute units. Shipping only `_8DA4W` keeps the public surface aligned with the data; if a future Core ML or model update flips the verdict, easy to add the other variants back. **Memory methodology note** — the new paraphrase row in `docs/docs/02-benchmarks/memory-usage.md` reports RSS / `phys_footprint` deltas from a clean app baseline (loaded − idle), captured on-device at the same conceptual point. The existing distiluse rows there (36 / 44 MB) come from an older measurement pass with a different (and not reconstructable from the diff) methodology, so the two rows are not directly comparable. A separate pass to re-measure distiluse and other rows with the same methodology would be a good follow-up. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [x] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [x] iOS - [x] Android ### Testing instructions 1. `cd apps/text-embeddings && npx expo run:ios` (or `run:android`). 2. Pick **"Multilingual Paraphrase (8da4w)"** in the model picker. 3. Add a sentence in one language, query with an aligned sentence in another (e.g. Polish "Słoneczko" against "It's so sunny outside!"). The cross-lingual pair should top the matches. ### Related issues Closes the paraphrase-multilingual half of #945 (the distiluse half landed in #1098). ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes Blocks on #1114.
Description
Adds the
paraphrase-multilingual-MiniLM-L12-v2sentence-transformer model — the second multilingual embeddings model after distiluse, completing #945. Ships only the XNNPACK 8da4w variant underMODEL_REGISTRY.ALL_MODELS(see "Why a single variant" below).384-d output, max 126 tokens, 50+ languages. Tokenizer is Unigram + Precompiled normalizer + Metaspace decoder — requires the bumped
pytorch/extension/llm/tokenizersruntime from #1114, so this PR blocks on that landing first and should be rebased onto main once #1114 merges.HF repo: software-mansion/react-native-executorch-paraphrase-multilingual-MiniLM-L12-v2 (
v0.9.0tag, layout mirrors distiluse).Why a single variant —
TLDR 8da4w works faster then all and was also one of the smallest, without loss in precision.
Longer answer:
unlike distiluse, where Core ML fp32 won iPhone thanks to ANE acceleration, benchmarks on iPhone 17 Pro + OnePlus 12 (~80-token input, 50 measured forwards after 3 warmups) showed the XNNPACK 8da4w variant Pareto-dominates the other three on both platforms: faster than XNNPACK fp32, Core ML fp32 and Core ML fp16 on iPhone, and ~36% smaller steady-state memory footprint than the next-best variant. Likely cause: paraphrase-multilingual-MiniLM-L12-v2 is a smaller model (~118 M params, 12 layers) where Core ML's runtime doesn't push enough work onto ANE for the precision-conversion overhead to pay off. fp16 being slower than fp32 on Core ML for this model is a tell that the runtime is falling back to slower compute units. Shipping only
_8DA4Wkeeps the public surface aligned with the data; if a future Core ML or model update flips the verdict, easy to add the other variants back.Memory methodology note — the new paraphrase row in
docs/docs/02-benchmarks/memory-usage.mdreports RSS /phys_footprintdeltas from a clean app baseline (loaded − idle), captured on-device at the same conceptual point. The existing distiluse rows there (36 / 44 MB) come from an older measurement pass with a different (and not reconstructable from the diff) methodology, so the two rows are not directly comparable. A separate pass to re-measure distiluse and other rows with the same methodology would be a good follow-up.Introduces a breaking change?
Type of change
Tested on
Testing instructions
cd apps/text-embeddings && npx expo run:ios(orrun:android).Related issues
Closes the paraphrase-multilingual half of #945 (the distiluse half landed in #1098).
Checklist
Additional notes
Blocks on #1114.