Skip to content

Commit 0f2fa2a

Browse files
authored
[RNE Rewrite] Deduplicate tokenizer result-unwrapping with a local unwrap helper (#1320)
## Description The five `TokenizerHostObject` methods (`encode`, `decode`, `idToToken`, `tokenToId`, and the encode path) each hand-rolled the same block around a `tokenizers::Result`: ```cpp auto result = self->tokenizer_->encode(...); if (!result.ok()) { throw jsi::JSError(rt, std::format("encode: Failed to encode input: {}", toString(result.error()))); } return conversions::toJsiArray(rt, result.get()); ``` Replaced with a local `unwrap` helper in the file's anonymous namespace, mirroring the one already in `core/model.cpp`. That existing helper is file-local and typed to `executorch::runtime::Result`, so it cannot be reused here — the tokenizer returns `tokenizers::Result` carrying a distinct `tokenizers::Error` (which is why this file already has its own `toString`). Following the established per-file convention rather than hoisting a shared helper into core keeps the change scoped and leaves core untouched. Behaviour is identical: same error messages, same exceptions. Split out of #1319 so the `std::span` refactor and this cleanup can be reviewed independently. Part of the broader tokenizer host-object cleanup tracked in #1316. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [ ] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [x] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions Behaviour-preserving; verification is compile-only, not yet run on a device. `tokenizer.cpp` compiles clean under the project's strict warning set from `.clangd`: ``` cd packages/react-native-executorch clang++ -fsyntax-only $(tr '\n' ' ' < compile_flags.txt) \ -Wall -Wextra -Wpedantic -Wconversion -Wsign-conversion cpp/extensions/nlp/tokenizer.cpp ``` To exercise at runtime: the tokenizer screen in `apps/nlp`. ### Related issues Part of #1316 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings
1 parent c601289 commit 0f2fa2a

1 file changed

Lines changed: 20 additions & 20 deletions

File tree

packages/react-native-executorch/cpp/extensions/nlp/tokenizer.cpp

Lines changed: 20 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,14 @@ std::string toString(tokenizers::Error error) {
4747
}
4848
return "Unknown(" + std::to_string(static_cast<int32_t>(error)) + ")";
4949
}
50+
51+
template <typename T>
52+
T unwrap(jsi::Runtime &rt, const std::string &ctx, tokenizers::Result<T> result) {
53+
if (!result.ok()) {
54+
throw jsi::JSError(rt, std::format("{}: {}", ctx, toString(result.error())));
55+
}
56+
return std::move(result.get());
57+
}
5058
} // namespace
5159

5260
TokenizerHostObject::TokenizerHostObject(std::string tokenizerPath)
@@ -83,12 +91,10 @@ jsi::Value TokenizerHostObject::get(jsi::Runtime &rt, const jsi::PropNameID &nam
8391
}
8492

8593
auto text = conversions::asType<std::string>(rt, "encode: text", args[0]);
86-
auto result = self->tokenizer_->encode(text, kNumAddedBosTokens, kNumAddedEosTokens);
87-
if (!result.ok()) {
88-
throw jsi::JSError(rt, std::format("encode: Failed to encode input: {}", toString(result.error())));
89-
}
94+
auto tokens = unwrap(rt, "encode: Failed to encode input",
95+
self->tokenizer_->encode(text, kNumAddedBosTokens, kNumAddedEosTokens));
9096

91-
return conversions::toJsiArray(rt, result.get());
97+
return conversions::toJsiArray(rt, tokens);
9298
};
9399
return jsi::Function::createFromHostFunction(rt, jsi::PropNameID::forAscii(rt, "encode"), 1, fnBody);
94100
}
@@ -121,12 +127,10 @@ jsi::Value TokenizerHostObject::get(jsi::Runtime &rt, const jsi::PropNameID &nam
121127
return jsi::String::createFromUtf8(rt, "");
122128
}
123129

124-
auto result = self->tokenizer_->decode(tokens, skipSpecialTokens);
125-
if (!result.ok()) {
126-
throw jsi::JSError(rt, std::format("decode: Failed to decode tokens: {}", toString(result.error())));
127-
}
130+
auto text = unwrap(rt, "decode: Failed to decode tokens",
131+
self->tokenizer_->decode(tokens, skipSpecialTokens));
128132

129-
return jsi::String::createFromUtf8(rt, result.get());
133+
return jsi::String::createFromUtf8(rt, text);
130134
};
131135
return jsi::Function::createFromHostFunction(rt, jsi::PropNameID::forAscii(rt, "decode"), 1, fnBody);
132136
}
@@ -169,12 +173,10 @@ jsi::Value TokenizerHostObject::get(jsi::Runtime &rt, const jsi::PropNameID &nam
169173
}
170174

171175
auto tokenId = conversions::asType<uint64_t>(rt, "idToToken: id", args[0]);
172-
auto result = self->tokenizer_->id_to_piece(tokenId);
173-
if (!result.ok()) {
174-
throw jsi::JSError(rt, std::format("idToToken: Failed to convert id to token: {}", toString(result.error())));
175-
}
176+
auto token = unwrap(rt, "idToToken: Failed to convert id to token",
177+
self->tokenizer_->id_to_piece(tokenId));
176178

177-
return jsi::String::createFromUtf8(rt, result.get());
179+
return jsi::String::createFromUtf8(rt, token);
178180
};
179181
return jsi::Function::createFromHostFunction(rt, jsi::PropNameID::forAscii(rt, "idToToken"), 1, fnBody);
180182
}
@@ -196,12 +198,10 @@ jsi::Value TokenizerHostObject::get(jsi::Runtime &rt, const jsi::PropNameID &nam
196198
}
197199

198200
auto token = conversions::asType<std::string>(rt, "tokenToId: token", args[0]);
199-
auto result = self->tokenizer_->piece_to_id(token);
200-
if (!result.ok()) {
201-
throw jsi::JSError(rt, std::format("tokenToId: Failed to convert token to id: {}", toString(result.error())));
202-
}
201+
auto tokenId = unwrap(rt, "tokenToId: Failed to convert token to id",
202+
self->tokenizer_->piece_to_id(token));
203203

204-
return static_cast<double>(result.get());
204+
return static_cast<double>(tokenId);
205205
};
206206
return jsi::Function::createFromHostFunction(rt, jsi::PropNameID::forAscii(rt, "tokenToId"), 1, fnBody);
207207
}

0 commit comments

Comments
 (0)