Skip to content

linear: a body sent as br, utf-7 or utf-7-imap is decoded as Linear's body-parser decodes it - #1108

Open
nuriyoo wants to merge 24 commits into
linear-a-refused-root-argument-is-answered-firstfrom
linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
Open

nuriyoo wants to merge 24 commits into
linear-a-refused-root-argument-is-answered-firstfrom
linear-a-utf-7-body-is-read-as-iconv-lite-reads-it

Conversation

@nuriyoo

@nuriyoo nuriyoo commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes #849. A body on Linear's GraphQL route sent with Content-Encoding: br is decompressed as Linear's body-parser 2.x decompresses it, and one declared utf-7 or utf-7-imap is decoded as iconv-lite 0.7 decodes it, on Notion's routes too.

What changed

Brotli (backlot.graphql.body._unbrotli). A new runtime dependency, brotli>=1.2; the lock takes 1.2.1, the first release with wheels for Python 3.15. body_parser_text passes a br body through _unbrotli, whose docstring gives the answers to a stream Brotli cannot read, cut short, followed by more bytes, past the limit, and cut short past the limit with bytes Brotli cannot read after it. Notion's reading passes brotli=False and still answers br with the encoding's 415.

UTF-7 (_utf7, _base64_units). iconv-lite 0.7's utf-7 and utf-7-imap decoders, read off encodings/utf7.js, which is the same file in 0.7.2 and 0.7.3; _utf7's docstring sets out how a run is read. Notion's reading (backlot.notion_request) goes through the same body_parser_text.

Tests. Forty rows in _LINEAR_READS (tests/test_linear.py), each with the bytes api.linear.app answered: fifteen with Content-Encoding: br or BR, the limit among them, and twenty-five with charset=utf-7 or charset=utf-7-imap. Two rows in _REQUEST_ROWS (tests/test_notion.py) hold api.notion.com's answers to a utf-7 body.

Not in this PR. raw-body hands iconv-lite each chunk as it arrives, so a UTF-7 body that reaches Linear in pieces can decode otherwise where a piece ends inside a base64 run. Backlot reads the body whole, and how a request is cut into chunks on the way to Linear is not something a client controls, so there is no answer to model and no issue to file. The 413 in the row below for a cut-short br stream past the limit needs none either: the same body drew the decompression failure in the other sends, so nothing a client sends picks it.

Why this is what the real API does

Measured. Against api.linear.app on 2026-10-27 (UTC) with an API key, each request with its own cache-buster: thirty-seven bodies, fifteen with Content-Encoding: br and twenty-two declared utf-7 or utf-7-imap. On 2026-10-28, five more the rows hold: three that are a run holding two byte-order marks, and two runs of UTF-16 surrogates. On 2026-10-29, br streams past the limit, each cut short and followed by bytes Brotli cannot read, as their row below counts. Some of them:

body                                                           api.linear.app
br of {"query":"{ __typename }"}, as BR, then xx, twice        200
br over the plain query                                        400 Decompression failed
br cut short; br over no bytes                                 400 unexpected end of file
br of 10485760 bytes; of 10485761; of twice that               200; 413; 413
br past the limit, cut short, then bytes Brotli cannot read    400 Decompression failed in 14 of 16, 413 in 2
utf-7  {"query":"{ x+AGE- }"}                                  Cannot query field "xa"
utf-7  a run opening with a byte-order mark (+/v8AYQ-)         Cannot query field "xa"
utf-7  a mark inside a run (+AGEA/v8AYQ-)                      Unexpected character: U+00FE
utf-7  a run the body ends in, a mark past eight characters    200
utf-7  a run of five characters (+AGEAY-)                      Cannot query field "xa"
utf-7  a byte past 0x7f outside a run                          Unexpected character: U+FFFD
utf-7  +/v/+/w- opening the body (two marks)                   200
utf-7  +/v/+/w, the whole body                                 Unexpected token '' (one mark left)
utf-7-imap  &/v/+/w, the whole body                            the same
utf-7  a run of a lone high surrogate (+2D0-)                  Invalid character: U+D83D
utf-7  a surrogate pair in a variable's value (+2D3eAA-)       got invalid value, the pair as 4 UTF-8 bytes
utf-7-imap  {"query":"{ x&AGE,- }"}                            Cannot query field "xa"

Each of the forty rows sends one of these bodies, the ones at the limit padded by padded_query rather than with spaces, and the base branch answers thirty-four of them otherwise.

Notion. Against api.notion.com on 2026-10-28 (UTC), with no cache-buster, since Notion refuses a query parameter it does not know, two bodies declared utf-7 to POST /v1/search:

body                     api.notion.com        the base branch
{"page_size":+ADE-}      200                   400
+/v/+/w                  400 invalid_json      the same

Against iconv-lite. The two decoders gave the same UTF-16 units as iconv-lite 0.7.2, run in Node through getDecoder, write and end as raw-body calls them, for each of 64,644 generated byte strings in both charsets; 0.7.3 gave the same for the 6,001 of them that put a byte-order mark inside a run.

Verification

  • Tests — pytest on uv sync --all-extras --locked plus uv pip install --no-deps "llama-index-readers-hubspot<0.6", as CI installs: 8981 passed, 1 skipped.
  • Optional surfaces that ran rather than skipped — every one but the Atlassian MCP test in tests/test_mcp.py, which needs Docker's Linux containers and found no daemon here.
  • Lint — ruff check . && ruff format --check .: clean.

At 6a201eeb, on linear-a-refused-root-argument-is-answered-first at f39a1314.

uv sync --all-extras --locked
uv pip install --no-deps "llama-index-readers-hubspot<0.6"
uv run --no-sync pytest -rs                     8981 passed, 1 skipped in 57.10s
ruff check . && ruff format --check .           All checks passed! / 195 files already formatted
python scripts/prose_check.py <the base>        exit 0
mutations of each line this branch adds         65 (11 by hand), 61 caught
each row against the mutation written for it    41 of 41 rows, 109 of 293 table keys lose to one; 0 missed

Each line this branch adds to backlot/graphql/body.py was mutated against each test file this branch or its base changes: operators generated from its lines, and by hand the marks write and end return, a run the body ends in, a lone base64 character, a high byte outside a run, the bytes after a Brotli stream and the prefix the search stops at past the limit. The two survivors cannot change an answer: , read as / outside IMAP, where it ends the run, and two disjoint branches swapped. Two more never return.

  • A test fails without this change — thirty-four of the utf-7, utf-7-imap and br rows this pull request adds to test_linear_orders_the_body_and_the_credential, and its utf-7 run in test_notion_reads_a_request_as_real_does.
  • A new endpoint serving corpus content is ACL-scoped, proved for both an admin and a scoped token — not applicable: no endpoint is added; the change is how one route reads its body.
  • Comments state what was measured, not the history of the fix

@nuriyoo nuriyoo added the fidelity The vendor API defines the right answer; closing this needs a measurement against it label Oct 27, 2026
@nuriyoo nuriyoo self-assigned this Oct 27, 2026
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…t what end does; Notion reads utf-7 as Linear does
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…f a surrogate pair echoed in a variable's error
@nuriyoo
nuriyoo marked this pull request as ready for review October 28, 2026 09:39
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
@nuriyoo
nuriyoo requested a review from khj809 October 28, 2026 13:08

@khj809 khj809 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed for behaviour (live calls to api.linear.app and api.notion.com on 2026-10-29, the PR's server under uvicorn) and for prose.

Behaviour: pass

All 41 br, BR, utf-7 and utf-7-imap rows of _LINEAR_READS, plus 32 further malformed or combined bodies (br garbage, a br stream past the limit and cut short, a valid start followed by garbage, a second truncated stream, br around utf-7 and utf-7-imap; utf-7 runs of +AHs- and +AAA-, a high byte or = in a run, a lone low surrogate, gzip around a run, a mark at the end; utf-7-imap runs closed by a space, high bytes in and out of a run, a lone high surrogate, a mark in a run, & at the end; charset names UTF-7, UTF-7-IMAP, utf7) came back from Backlot with the same status and body bytes as Linear, 73 of 73. _utf7 agreed with iconv-lite 0.7.3's decoder on 150,000 generated byte strings in both charsets. Nine utf-7 rows against Notion's POST /v1/search matched too. On the base with the PR's tests, 33 -key cases of test_linear_orders_the_body_and_the_credential and test_notion_reads_a_request_as_real_does fail. Every new input is a row of an existing measured table.

Note: uv.lock:2 — the lock goes from revision = 5 (the base and main) to revision = 3 and rewrites the markers of httpcore2, importlib-metadata and secretstorage. uv lock --check still passes, but regenerating with the uv that wrote the base's lock would leave a diff that only adds brotli.

Prose: block

  • backlot/graphql/body.py:385 — "Python's decoder refuses bytes after the stream, which Node leaves unread: the stream ..." repeats _unbrotli's docstring three lines above ("Node reads one stream and leaves what follows it unread"). Suggested: "# Python's decoder refuses bytes after the stream: the stream ends at the shortest prefix that finishes it, if any does."
  • Note, PR body, Verification — the 683-character paragraph arguing each surviving mutation could be one sentence, e.g. "The six survivors cannot change an answer: , read as / outside IMAP, two disjoint branches swapped, a limit check the later read repeats, unjoined surrogate units, and two that never return.", or a table under the command block.

…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…, the docstring having said what Node leaves unread
…t read, is the decompression failure

The prefix search stops at the shortest prefix that finishes the stream or that Brotli refuses,
no longer at one that has given more than the limit. api.linear.app answered such a body with
the decompression failure in 14 of 16 sends on 2026-10-29 (UTC), over three bodies.
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
@nuriyoo

nuriyoo commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Applied in eaef523e, f9e065a8 and 3e52f876.

  • uv.lock:2, in eaef523e: written from the base's lock by uv 0.13.6, the uv that writes revision = 5. The diff against the base adds brotli 1.2.1 and nothing else.
  • backlot/graphql/body.py:385 (now :386), in f9e065a8: your comment, as you wrote it.
  • PR body, Verification: the survivors are two sentences now, one for the two that cannot change an answer and one for the two that never return. One survivor it listed was not equivalent. Dropping len(out) > limit from the prefix search survived because no row reached it, and a row that does is a stream cut short past the limit, then bytes Brotli cannot read. api.linear.app answered three such bodies on 2026-10-29 (UTC) with the 400 Decompression failed in 14 of 16 sends and the 413 in the other two; Backlot answered the 413. 3e52f876 drops that operand, so the search stops only at a prefix that finishes the stream or that Brotli refuses, and adds the row (past it, br cut short then bytes Brotli cannot read), which fails with the operand put back. The same body drew both answers, so nothing a client sends picks the 413; the body's Not in this PR. says so.

@khj809 khj809 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked at 6a201eeb: every finding of the earlier review is addressed. uv.lock is back at revision = 5 and adds only brotli, and the comment in _unbrotli no longer repeats its docstring.

…s-answered-first' into linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
…linear-a-utf-7-body-is-read-as-iconv-lite-reads-it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fidelity The vendor API defines the right answer; closing this needs a measurement against it

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants