fix: reconstruct reverse-mapped tokens split across SSE delta events

[rafaelreis-r]

Fixes #55.

Problem

reverseMap() only rewrites complete patterns. The streaming SSE response transformer applied it independently per content_block_delta event, so a sanitized identifier split across two delta events (.ocpla then tform) was never reverse-mapped and the sanitized form leaked to the client.

A naive slice-offset attempt makes it worse: it emits the incomplete prefix raw, then when the next event completes the pattern the already-sent bytes cannot be retracted, producing mangled output like .ocplalaw.

Fix

createSseEventTransformer() does a proper streaming replace:

• streamReverse() keeps a raw, un-emitted buffer per content block + field. Each delta holds back the trailing bytes that could still grow into a pattern (maxPatternLen-1), emits only the safe prefix, and pulls the cut back so no complete occurrence straddles it.
• The held tail is flushed as synthetic content_block_delta events on content_block_stop; flushAll() covers streams that end without a stop.
• thinking / redacted_thinking blocks stay byte-identical pass-throughs.
• Exports loadConfig / reverseMap / applySseReverseMapChunks and guards startServer() behind require.main for unit-testing.

Tests

test/sse-reversemap.test.js (node --test, no new deps) asserts exact reconstruction — reconstruct(transform(events)) === reverseMap(wholeInput) — at:

• every two-way split offset of a text_delta
• single-character splitting of a text_delta
• every two-way split offset of an input_json_delta (escaped tool/prop names)
• a token split across raw TCP chunks
• flushAll when no content_block_stop arrives
• thinking / redacted_thinking byte-invariance

All pass.

[zacdcook]

Hey @rafaelreis-r — solid catch on this bug. I dug into your diagnosis and reproduction, ran your test suite, and confirmed: the per-event reverseMap was silently leaking sanitized tokens whenever Anthropic's tokenizer split an identifier across delta events. Your createSseEventTransformer algorithm (per-block carry, maxPatternLen-1 cut, complete-pattern pullback, flushAll on stop) is correct, and the test suite is comprehensive — I'd want it in regardless of implementation.

One concern before merging, mostly architectural:

The proxy has had a deliberate "raw string manipulation only" principle since v1.0 — no JSON.parse/JSON.stringify on the body. The original reason was Anthropic's classifier rejecting re-serialized request bodies (Unicode escapes, whitespace, and number formatting all subtly change), and the codebase has consistently honored that through every transform (processBody, reverseMap, billing block injection, metadata injection) — it's why everything reads as indexOf + slice + split/join rather than parsed objects.

Your fix uses JSON.parse(dataStr) + JSON.stringify(payload) on every delta event. For the response direction it's functionally safe — the consumer is OpenClaw, not Anthropic's classifier — but it does break the architectural pattern and changes the byte format of every event the proxy forwards (key ordering, whitespace, possibly Unicode escape canonicalization).

I drafted a raw-string variant that achieves identical functional behavior. It keeps your streamReverse algorithm verbatim, but extracts the index/field/value via small string-aware helpers (extractSseIntField, findSseStringField) and decodes/re-encodes only the field value via tiny hand-rolled JSON-string codec helpers (jsonStringDecode, jsonStringEncode) — no payload parsing, no key reordering. All 6 of your tests pass unchanged against it. I also ran a handful of streamed requests through it on my own production deployment — both text and tool-call paths round-tripped correctly, with sanitized identifiers (openclaw, lossless-claw) reverse-mapped accurately from cross-delta token splits. Short validation window so far, but representative of normal traffic.

Happy to either:

• Send the variant as a follow-up PR against your branch (you'd own the merge with both your original commit and the refactor on top)
• Drop the diff here for you to pull into this PR if you'd rather keep it as one
• Or you can push back if you think JSON.parse is the right call here — open to that. The raw-string principle is a strong default but not sacred, and you may have a reason I haven't considered.

Either way, this bug needed fixing and your work is what got us there. Just want us to land on the variant we'll both be happy maintaining long-term.

[rafaelreis-r]

Agreed — let's go with the raw-string variant. You're right that "functionally safe on the response direction" is the weakest possible reason to keep JSON.parse: it holds today because the consumer is OpenClaw, but a re-serialized byte format (key reordering, whitespace, Unicode-escape canonicalization on every delta) is exactly the kind of invisible change that breaks something downstream later and is miserable to trace. And the consistency cost is real — the raw-string principle only works if it's uniform; a half-and-half codebase turns it into folklore the next contributor won't honor. The hard part — streamReverse — stays verbatim either way, so there's no algorithmic loss in ceding the field-extraction layer.

One ask before merge, since this is where the raw-string approach trades risk rather than removing it: JSON.parse is a bulletproof parser, and jsonStringDecode/jsonStringEncode are hand-rolled. I'd want unit tests on those two helpers specifically, covering:

• \uXXXX and surrogate pairs (astral-plane chars / emoji) — the usual failure spot for hand-rolled codecs
• \", \\, \/, and control chars (\n \t \b \f \r)
• a field value that embeds a fake key (e.g. a sanitized identifier literally containing "index":) — to confirm findSseStringField is genuinely string-aware and doesn't match inside a value

If those pass, the variant is strictly better than mine and I'm happy to maintain it.

For logistics: the follow-up PR against my branch works best — keeps my original commit (the bug repro + test suite) with your refactor on top, and we land it as one. Send it over whenever.

Thanks for the careful review on this.

[rafaelreis-r]

I love the AI talk, btw.

On a side note, this bug had my agents go berzerk. They developed alternate .ocplatform workspaces and broke all sorts of scripts, automations, and basic functionality. They truly believed openclaw and ocplatform were the same word.

Happy that we're fixing it, this tool rocks.
--Rafael

[sontakey]

Hope this fixes the .ocplatform issue! Been driving me insane.

[0n1cOn3]

On a side note, this bug had my agents go berzerk. They developed alternate .ocplatform workspaces and broke all sorts of scripts, automations, and basic functionality. They truly believed openclaw and ocplatform were the same word.

That absolutely driven me crazy as well. It also has created a new folder .ocplatform and has put the data i'd needed inside there instead of .openclaw. Simple fix was a Symlink from .ocplatform to .openclaw.

I still see sometimes .ocplatform but I know it will land for sure in the proper folder. In case of automations or scripts - I dont face these issues. Prolly because I have a call in my system prompts to use bash instead of it when it fails. Since it has its own VM with regular backups and safety guidelines, it didn't had bricked anything so far.

[rafaelreis-r]

@0n1cOn3 you can temporarily patch your deployment with this PR. I've been running smoothly for a couple of weeks now

@zacdcook this is in the wild. happy to move forward as per your recommendation. Let me know.

[sontakey]

I am using this patch and so far so good!

…

On Mon, May 25, 2026 at 10:18 AM, rafaelreis-r < ***@***.*** > wrote: *rafaelreis-r* left a comment (zacdcook/openclaw-billing-proxy#56) ( #56 (comment) ) @0n1cOn3 ( https://github.com/0n1cOn3 ) you can temporarily patch your deployment with this PR. I've been running smoothly for a couple of weeks now @zacdcook ( https://github.com/zacdcook ) this is in the wild. happy to move forward as per your recommendation. Let me know. — Reply to this email directly, view it on GitHub ( #56 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AACXWJXCKPON62QRDTBNOHL44RP57AVCNFSM6AAAAACZBTFKMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKMZVGMZDGNZWGQ ). You are receiving this because you commented. Message ID: <zacdcook/openclaw-billing-proxy/pull/56/c4535323764 @ github. com>

[0n1cOn3]

@0n1cOn3 you can temporarily patch your deployment with this PR. I've been running smoothly for a couple of weeks now

Aight, applying another PR locally hehe.

[hunandy14]

@rafaelreis-r @zacdcook — opened #59 with the raw-string variant the review here asked for.

It keeps #56's streamReverse verbatim (built on the same commit, so the algorithm credit stays with @rafaelreis-r) and only swaps the per-event JSON.parse/JSON.stringify for four hand-rolled codecs, so the SSE path never parses/re-serializes the body. The existing test/sse-reversemap.test.js passes unchanged, plus a new boundary suite. It's a single cherry-pickable commit on top of #56.