Skip to content

Commit 298fd98

Browse files
committed
docs(model-server): clarify orjson serialization comments
Explain the serialization sequence with short, reader-focused sentences so the helper's performance purpose and response handling are clear. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
1 parent a09e6a8 commit 298fd98

1 file changed

Lines changed: 6 additions & 9 deletions

File tree

nemo_gym/base_responses_api_model.py

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -156,17 +156,14 @@ def _plain(value: Any) -> Any:
156156

157157

158158
def _orjson_dispatch_response(content: Any) -> Any:
159-
"""Serialize a finished non-streaming dispatch result with orjson.
159+
"""Serialize a completed non-streaming model response with orjson.
160160
161-
Returning a bare model or dict makes FastAPI run ``jsonable_encoder`` — a pure-Python
162-
recursive walk with one function call per element — followed by stdlib ``json.dumps``.
163-
On training responses the token-id and logprob arrays make that walk the dominant CPU
164-
cost of the whole server. Building the response bytes here keeps the semantics
165-
(``model_dump(mode="json")`` is exactly what ``jsonable_encoder`` does first for a
166-
model) while FastAPI passes a finished ``Response`` through untouched.
161+
FastAPI serializes a bare dictionary or Pydantic model with ``jsonable_encoder``.
162+
That function recursively visits every token ID and log probability before standard-library ``json.dumps`` runs.
167163
168-
Anything that is already a ``Response`` (a server override that built its own) is
169-
returned as-is.
164+
Convert Pydantic models to JSON-compatible values before encoding the result.
165+
Return the encoded bytes as a ``Response`` so FastAPI does not encode them again.
166+
Preserve ``Response`` objects created by model-server overrides.
170167
"""
171168
if isinstance(content, Response):
172169
return content

0 commit comments

Comments
 (0)