Commit 0f2fa2a
authored
[RNE Rewrite] Deduplicate tokenizer result-unwrapping with a local
## Description
The five `TokenizerHostObject` methods (`encode`, `decode`, `idToToken`,
`tokenToId`, and the encode path) each hand-rolled the same block around
a `tokenizers::Result`:
```cpp
auto result = self->tokenizer_->encode(...);
if (!result.ok()) {
throw jsi::JSError(rt, std::format("encode: Failed to encode input: {}", toString(result.error())));
}
return conversions::toJsiArray(rt, result.get());
```
Replaced with a local `unwrap` helper in the file's anonymous namespace,
mirroring the one already in `core/model.cpp`. That existing helper is
file-local and typed to `executorch::runtime::Result`, so it cannot be
reused here — the tokenizer returns `tokenizers::Result` carrying a
distinct `tokenizers::Error` (which is why this file already has its own
`toString`). Following the established per-file convention rather than
hoisting a shared helper into core keeps the change scoped and leaves
core untouched.
Behaviour is identical: same error messages, same exceptions.
Split out of #1319 so the `std::span` refactor and this cleanup can be
reviewed independently. Part of the broader tokenizer host-object
cleanup tracked in #1316.
### Introduces a breaking change?
- [ ] Yes
- [x] No
### Type of change
- [ ] Bug fix (change which fixes an issue)
- [ ] New feature (change which adds functionality)
- [ ] Documentation update (improves or adds clarity to existing
documentation)
- [x] Other (chores, tests, code style improvements etc.)
### Tested on
- [ ] iOS
- [ ] Android
### Testing instructions
Behaviour-preserving; verification is compile-only, not yet run on a
device.
`tokenizer.cpp` compiles clean under the project's strict warning set
from `.clangd`:
```
cd packages/react-native-executorch
clang++ -fsyntax-only $(tr '\n' ' ' < compile_flags.txt) \
-Wall -Wextra -Wpedantic -Wconversion -Wsign-conversion cpp/extensions/nlp/tokenizer.cpp
```
To exercise at runtime: the tokenizer screen in `apps/nlp`.
### Related issues
Part of #1316
### Checklist
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have updated the documentation accordingly
- [x] My changes generate no new warningsunwrap helper (#1320)1 parent c601289 commit 0f2fa2a
1 file changed
Lines changed: 20 additions & 20 deletions
Lines changed: 20 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
50 | 58 | | |
51 | 59 | | |
52 | 60 | | |
| |||
83 | 91 | | |
84 | 92 | | |
85 | 93 | | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
| 94 | + | |
| 95 | + | |
90 | 96 | | |
91 | | - | |
| 97 | + | |
92 | 98 | | |
93 | 99 | | |
94 | 100 | | |
| |||
121 | 127 | | |
122 | 128 | | |
123 | 129 | | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
| 130 | + | |
| 131 | + | |
128 | 132 | | |
129 | | - | |
| 133 | + | |
130 | 134 | | |
131 | 135 | | |
132 | 136 | | |
| |||
169 | 173 | | |
170 | 174 | | |
171 | 175 | | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
| 176 | + | |
| 177 | + | |
176 | 178 | | |
177 | | - | |
| 179 | + | |
178 | 180 | | |
179 | 181 | | |
180 | 182 | | |
| |||
196 | 198 | | |
197 | 199 | | |
198 | 200 | | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
| 201 | + | |
| 202 | + | |
203 | 203 | | |
204 | | - | |
| 204 | + | |
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
| |||
0 commit comments