Repository navigation
linear, fireflies: a request error is Apollo Server's, Linear's introspection, 401 and unknown id are Linear's, and each method, content type and Accept is answered by the vendor's layer - #1027
Conversation
Both vendors run Apollo Server 4 over graphql-js. The engine now answers a request the way that pair does: Apollo Server's checks of the body and of a persisted query, then graphql-js's parse, validation, choice of operation and coercion of variables, each refusal with Apollo Server's code and graphql-js's wording (backlot.graphql.js_words). Each router writes the refusals in its vendor's envelope, and Linear answers them up to the choice of operation before it reads the credential. The telemetry classifier reads Fireflies' code beside the message, and the GraphQL message slugs those codes replace are gone.
…at JSON cannot reach A table of documents over a schema of every input kind holds graphql-js 16.14.2's answers under Node, picked so that each change to js_words a row can tell apart has one. The vendor tables gain the rows Linear and Fireflies answered for the rest: an error's place after a line separator, a persisted query's version, the unreachable fragments, and the body checks two at a time. js_words drops the branches no request reaches: a circular value, a variable whose type validation already refused, a scalar's GraphQLError.
…looked up Both vendors answered a hash sent with `query` false or 0 as with no query at all, PersistedQueryNotFound at 200, and hashed an empty string.
…ot Backlot's failure Fireflies has no subscription type, and graphql-core refuses a subscription with the operation as the error's node. That is the request's fault, so the call is counted `client`, as the 200 Fireflies answers with says.
…iteral, and a refusal is read off the field error's own nodes
…ersion it cannot show
…d a hash alone with once its query was served
…or-is-apollos # Conflicts: # backlot/telemetry/errors.py # tests/test_telemetry.py
…ounts as graphql-js counts Fireflies runs graphql-depth-limit (maxDepth 10) as graphql-js starts validating: its depth errors come ahead of graphql-js's own and count toward the hundred validate takes, and it throws, which Apollo Server answers with its 500, on a spread of an undefined fragment, a field-less spread cycle and a 101st report. That replaces the walk for undefined fragments, which answered the 500 under __schema, __type and __typename and for operations a later one replaces. A column counts UTF-16 code units, and each error the run makes, a resolver's too, is located as graphql-js locates it. The refusal after fifty refused variables keeps INTERNAL_SERVER_ERROR, which Linear masks as its Internal server error.
… definition and every path's error
…he 101st across two operations throws
…ring say what they point at
…or-is-apollos # Conflicts: # tests/test_fireflies.py # tests/test_linear.py
… credential goes with every credential
…or-is-apollos # Conflicts: # backlot/routers/fireflies.py
khj809
left a comment
There was a problem hiding this comment.
Reviewed at 4624cbdb, split into behaviour (re-measured live) and prose. Verdict: changes requested. There are two behaviour blocks and two prose blocks, and I've also added a note on how js_words.py is scoped to graphql-js 16.
I sent 55 requests to api.linear.app and api.fireflies.ai on 2026-10-28 (UTC), each with its own cache-buster, and sent the same requests to backlot.serve() at the head. They matched except for the two blocks below and the 401 body this PR already declares (#372 block 3). The refusals Linear now answers without a credential come from resolve_payload over the body and the SDL, and no store read is involved. When the credential check is moved ahead of resolve_payload, test_linear_orders_the_body_and_the_credential fails.
Behaviour
[block] backlot/graphql/engine.py:427 — an empty query whose persisted-query hash matches is refused as having no query.
{"query":"","extensions":{"persistedQuery":{"version":1,"sha256Hash":"e3b0c442…b855"}}} (sha256 of "")
api.linear.app (key, and no credential) 400 Syntax Error: Unexpected <EOF>. line 1 col 1 GRAPHQL_PARSE_FAILED
api.fireflies.ai 400 Syntax Error: Unexpected <EOF>. line 1 col 1 GRAPHQL_PARSE_FAILED
Backlot, Linear 400 GraphQL operations must contain a non-empty `query` or a `persistedQuery` extension. INVALID_INPUT
Backlot, Fireflies 400 same message BAD_REQUEST
Apollo Server only refuses a body for having no query when there is no persistedQuery extension. When there is one, a string query whose hash matches goes on to be parsed. _persisted(query="") in tests/test_graphql.py:953 covers only a hash that does not match. The fix is to apply NO_QUERY only when there is no truthy persistedQuery, and to add the matching-hash row to test_apollo_server_reads_a_body_in_its_order and to both vendors' order tables.
[block] backlot/routers/fireflies.py:228 — an operation named __proto__. graphql-depth-limit keeps operations in a plain {} and walks them with for…in, so on Fireflies every document with an operation named __proto__ gets the 500.
query __proto__ { __typename } api.fireflies.ai 500 {"friendly":false,"message":"Internal server error","code":"INTERNAL_SERVER_ERROR","extensions":{}}
Backlot 200 {"data":{"__typename":"Query"}}
query __proto__ { …12 fields deep… } api.fireflies.ai the same 500
Backlot 400 '__proto__' exceeds maximum operation depth of 10, plus a validation error
query constructor { __typename } 200 on both
The fix is for _depth_limit to return the 500 refusal when any operation is named __proto__, with a row added to test_fireflies_orders_the_body_and_the_credential. This case is narrow, so I'd fix the empty-query one first.
Prose
[block] backlot/graphql/engine.py:111 and the module docstring at engine.py:12-13 — "message in graphql-js's words". Several refusals are not in graphql-js's words. body_refusal and persisted_query_refusal use Apollo Server's own messages (Operation batching disabled., PersistedQueryNotFound, Unsupported persisted query version), and _depth_limit uses graphql-depth-limit's. Suggested wording: "message as Apollo Server writes it, in graphql-js's words where graphql-js made the refusal".
[block] backlot/graphql/engine.py:475 — "graphql-core's ExecutionResult.formatted writes "data" first; …". run no longer calls result.formatted; it builds payload from each error.formatted. That clause only contrasts with the deleted line. The fix is to start the comment at "api.fireflies.ai and api.linear.app open a response carrying a field error with "errors" (2026-10-21), …".
[note] persisted_query_refusal docstring (engine.py:262-269). The docstring describes PersistedQueryNotFound first and the version check later, but the code checks the version first: a hash with no query and version 2 gets Unsupported persisted query version. The docstring should follow the code's order.
[note] backlot/routers/fireflies.py:199-201 — _MAX_ERRORS = 100. This repeats _MAX_VALIDATION_ERRORS = 100 in engine.py:104-106. The router could use the engine's constant and keep only the "a 101st report from a rule throws past" clause.
[note] PR body. Six "What changed" paragraphs run to 476–780 characters. The Fireflies paragraph restates _depth_limit's docstring and the Linear paragraph restates _envelope's. "The credential." and the mutation paragraph under Verification put measurements in prose; each would read better as a small table.
js_words.py is graphql-js 16's words, and its head should say so
I ran graphql-js 16.14.2 and 17.0.2 (npm latest) side by side on 19 cases.
same in 16.14.2 and 17.0.2 syntax errors, " quoting, directive location names (QUERY), unknown-field and enum
suggestions, leaf selection, inspect truncation, the 50-error cap on variables
reworded in 17 every refusal of a variable, and the missing-argument validation message:
var wrong type 16 Variable "$v" got invalid value "abc"; Int cannot represent non-integer value: "abc"
17 Variable "$v" has invalid value: Int cannot represent non-integer value: "abc"
var null 16 Variable "$v" of non-null type "Int!" must not be null.
17 Variable "$v" has invalid value: Expected value of non-null type "Int!" not to be null.
var missing 16 Variable "$v" of required type "Int!" was not provided.
17 Variable "$v" has invalid value: Expected a value of non-null type "Int!" to be provided.
list item path 16 … got invalid value null at "i.b[0]"; …
17 … has invalid value at .b[0]: …
missing argument 16 Field "f" argument "x" of type "Int!" is required, but it was not provided.
17 Argument "Query.f(x:)" of type "Int!" is required, but it was not provided.
So coerce_variable_values is entirely graphql-js 16, and so is requote's rewording of the missing-argument message. If a vendor moves to 17, nothing raises; the answers just diverge. Both vendors answered in 16.14.2's words on 2026-10-25. Upgrading graphql-core to 3.3 (the port of 17) would change requote's input, but pyproject.toml:43 already pins <3.3 and states why.
The docstring names 16.14.2 only partway into a sentence about the measurement date, and its summary line names no version. I'd put the scope at the head and leave the pin's reason in pyproject:
"""graphql-js 16's words for the errors graphql-core 3.2 makes with its own.
Linear and Fireflies answered in graphql-js 16.14.2's words on 2026-10-25 (UTC). graphql-js 17
rewords each refusal of a variable and of a missing argument (17.0.2, run on 2026-10-28), so a
vendor's upgrade shows here as a divergence, not an error.
…mpty, and Fireflies refuses an operation named __proto__ Apollo Server refuses a body for having no query only when it sends no persistedQuery extension; with one, a string query whose hash matches, "" included, goes on to be parsed. Both vendors answered "" with its hash with graphql-js's Syntax Error on 2026-10-28 (UTC). graphql-depth-limit keeps the operations in a plain object, so an operation named __proto__ becomes its prototype and its for...in throws. Fireflies answers any document holding one with its 500. The engine's MAX_VALIDATION_ERRORS is now the one constant the Fireflies depth rule reads. Docstrings say a refusal's message is Apollo Server's, in graphql-js's words where graphql-js made it, and js_words names graphql-js 16 as its scope.
|
Each point of the review, at
A variable named |
|
Rechecked at One correction is left, and it comes from my own review: its table showed four variable cases as examples, but they weren't the whole set. Every 17 variable refusal opens with Apart from that sentence, this is good to go. |
…herwise The five refusals the docstring listed were a sample. Run over documents for each of graphql-js 16.14.2's validation rules, lexer and parser errors and value coercions, 17.0.2 also words otherwise the refusal of an argument's value as the field runs, a null literal for a non-null type, an input object literal of the wrong shape, a custom scalar's literal refusal, and the syntax errors at `.5` and `..`; it refuses a subscription to a schema without one as it validates, adds refusals for a subscription root's @skip/@include and for @defer/@stream, and orders an unused variable's refusal first.
|
The sentence is fixed in On 2026-10-28 (UTC) I ran graphql-js 16.14.2 and 17.0.2 side by side on 162 documents. Each of 16.14.2's 27 validation rules fires on at least one of them, and they also cover its lexer and parser errors and each branch of its value coercion. The schemas match the two here in shape: a field-less The rest answered the same: the other validation rules, every other lexer and parser error, a literal of the wrong built-in scalar or enum, and a custom scalar's literal refusal thrown as a The docstring now names each of these kinds and says what the run covered. |
|
The pull request stacked on this one, which is #1028, and the one stacked on that, which is #1029, are squash-merged into this branch as |
khj809
left a comment
There was a problem hiding this comment.
Rechecked at 1bad7ec9, and approving. The ten examples in your reply match 16.14.2 and 17.0.2 exactly when I run them, and the docstring now lists each kind of refusal that changes and no longer counts them. The #1028 and #1029 squashes carry the heads I approved. Apart from js_words.py and the merges from main, nothing changed in the routers, graphql/ or the tests, and the linear.py conflict resolves to the approved version. CI is green on 3.11–3.15.
This is part of three issues. It holds blocks 2 to 4 of the Linear one, which is #372, with its block 5 at
/graphqland thecharsetof its block 6. It holds blocks 2 and 4 of the Fireflies one, which is #373, with its block 3 at/graphqland thecharsetof its block 5, and blocks 1 to 3 of the one about anullvariable, which is #918. A refusal Apollo Server or graphql-js makes before a field runs comes back in each vendor's envelope with Apollo Server's code and message, worded by graphql-js where graphql-js made it, in the order both vendors check. Linear answers each one up to the choice of operation whether or not the request carries a credential.Without a credential, Linear answers a document that selects only
__schemaand__type, and refuses anything else with Linear's own 401; an unknown id at a root such asissue(id:)meets Linear'sEntity not founderror in Linear's members. A request to/linear/graphqlor/fireflies/graphqlby any method uvicorn passes on, under any content type andAccept, is answered by the layer that answers it at the vendor, as the table under Why orders them, but for the requests under Not in this PR.What changed
The engine (
backlot/graphql/engine.py).Engine.resolve_payloadandEngine.runanswer in the order the module docstring gives. Each refusal is aRefusal, which carries the headers Apollo Server sets on it and whether it is thrown ahead of the pipeline. An empty query sent with its own SHA-256 is parsed, as both vendors parse it.Engine.resolverefuses a GET of a mutation or a subscription withallow: POST, ahead of Linear's credential.Errors made as an operation runs. A
GraphQLErrorgraphql-core raises during the run, such as one for anullvariable at a non-null argument, comes back besidedatawithINTERNAL_SERVER_ERROR, the code Apollo Server sets on an error that carries none._with_rootsadds the root types each vendor names and the SDL leaves out, without fields, somutation { __typename }answersMutation.graphql-js 16's words (
backlot/graphql/js_words.py).requoteandlocationsturn graphql-core's messages and locations into graphql-js 16.14.2's.coerce_variable_valuesrefuses variables as graphql-js 16 does, up to the fifty errorsexecutetakes. The module docstring names the kinds of refusal graphql-js 17.0.2 answers otherwise, and what the comparison ran over.Apollo Server's HTTP layer (
backlot/graphql/apollo_http.py). What Apollo Server answers before its request pipeline, read off its source and the two libraries it asks,negotiator1.0.0 andwhatwg-mimetype4.0.0:A refusal Apollo Server throws ahead of the pipeline takes the media type
Acceptallows, or JSON; any other answer of Apollo Server's is its 406 whereAcceptallows none of its types.Linear (
backlot/routers/linear.py). The route picks the operation before it reads the credential, so each refusal up to that point is the same with the key, with no credential and withnope. A variable's refusal and the run wait for the key._envelopewrites Linear's members on each answer, and a field error's in Linear's order,pathbeforelocations(_in_linears_order)._variables_readreads avariablesstring withJSON.parse.Linear without a credential. A request whose credential does not resolve is served when
_introspection_onlyfinds only__schemaand__typeat the root of the operation it picks. Anything else,__typenameincluded, is the 401._UNAUTHENTICATEDholds Linear's body: the messageAuthentication required, not authenticatedand its sevenextensionsmembers,codeAUTHENTICATION_ERRORamong them.An id a by-id root holds nothing for (
backlot/graphql/linear_resolvers.py). Each by-id root raises_not_found(<type>), Linear'sEntity not found: <type>withcodeINPUT_ERRORand the presentableCould not find referenced <type>.;attachmentraises its own words (_ATTACHMENT). A hidden id draws the same error as an absent one, byte for byte (tests/test_acl.py).Linear's front. The route takes
apollo_http.METHODS. The layers in front of Apollo Server answer in the order of the table under Why:_LOAD_BALANCER_501,_preflight,_api_key_as_bearer, thenbody-parser(_has_body). Each answer carries the media type Express writes (_as_express_sends).Fireflies (
backlot/routers/fireflies.py)._envelopewrites each refusal withfriendly: false, the code beside the message and emptyextensions._depth_limitis graphql-depth-limit with amaxDepthof 10, run as graphql-js starts validating. Its docstring says what the rule crawls, what it reports and what makes it throw, which Fireflies answers with its 500.Fireflies' front. In front of Apollo Server come
@fastify/cors(_preflight) and Fastify, which reads a body by its type (body.fastify_media_type) and answers a method its route does not take with_not_found. Apollo Server answers a browser's GET with its page, and a CSRF suspect with its 400, before it reads the credential.Cloudflare (
backlot/main.py).TRACE,CONNECTandPURGEunder/linearand/firefliesget Cloudflare's own answers (_CLOUDFLARE_METHOD), behind its 400 forAuthorizationor a conditional header sent twice. TheTRACE405 closes the connection.NOTICE.md. Apollo Server's MIT notice, forbacklot/graphql/apollo_landing_page.html, the page Apollo Server 5.5.1 serves a browser.Telemetry (
backlot/telemetry/errors.py,backlot.graphql.engine._raised).error_codealso reads thecodeFireflies writes beside the message. A GraphQL refusal is counted by the code its envelope carries (Apollo Server's, or Linear's ownINVALID_INPUT,INPUT_ERRORandAUTHENTICATION_ERROR), not by a slug read from its message. The slugunsupported_filtercounts the Linear resolvers' own refusals that startunsupported, of a filter, a comparator or a sort key.Fireflies'
auth_failed500, on any method, is counted with thebacklot.error_classauth(_AUTH), not as Backlot's own failure. The failures among the answers that newly reach the classifier, the last seven astest_a_call_is_named_by_the_baselineholds them:Each answer carries its own status (
Result.status). The engine answers no refusal in express-graphql's words (POST body sent invalid JSON.,Must provide query string.), since neither vendor does.Docs.
docs/supported-sources.mdsays each endpoint takes a GET, and when Apollo Server serves one.Tests.
test_linear_reads_a_credential_as_linear_doesandtest_linear_orders_the_body_and_the_credentialhold the 401's bytes for each credential that is not one, and the second, in its_UNKNOWN_IDSrows, the not-found's bytes for each by-id root with the key,issue(id: "NOPE-1")among them.test_linear_answers_each_method_type_and_accept_as_linear_doesandtest_fireflies_answers_each_method_type_and_accept_as_fireflies_doeshold 163 and 175 requests as each vendor answered them, andtest_cloudflare_answers_ahead_of_linear_and_firefliesCloudflare's answers on the other paths. Five tables intests/test_graphql.pyhold what each library answered under Node.Not in this PR. The drafts stacked on this one, #1058 to #1063, hold the rest of the three issues. A variable named
$__proto__, which Linear answers as if it had no value, is #1113. The members of anArgument Validation Error,validationErrorsamong them, are #370's. Which of a by-id root's not-found and a later root field's refused arguments Linear answers is #1067. A request uvicorn's parser refuses gets uvicorn's 400 before the app is called, where each front answers its own, which is #1072.A suggestion graphql-js makes from a type Backlot's SDL does not declare (
Did you mean "Node"?on Linear,"Role"on Fireflies) needs no issue, because the SDL declares the types Backlot serves. Introspection answers that SDL, with or without Linear's key, somutationTypeisnullwhere Linear namesMutation. That needs none either:test_linear_introspection_reports_the_served_schemaandtest_fireflies_declares_no_mutationshold it, so a client generating calls from it is offered no write.Each of four answers that differ needs no issue, for the reason beside it:
Why this is what the real API does
Measured. api.linear.app with a personal API key and api.fireflies.ai with a bearer key on 2026-10-25 (UTC), one request per row with its own cache-buster. The requests sent each of Apollo Server's body checks alone and two at a time, persisted queries, and a refusal at each stage. They also sent errors made as the operation ran, fragments and the extra root types, and every body the tables hold that reaches Apollo Server, with the key, with no credential and with
nope. Among them:Replayed over
backlot.serve()at4624cbdb, comparing the status and the body without its trailing newline:The earlier dates the code keeps, 2026-10-18 and 2026-10-19 for Linear's Bearer 400 and Express, and 2026-10-21 and 2026-10-22 for the members of a field error, are measurements this branch moves rather than makes.
A persisted query and an operation's name. Both vendors on 2026-10-28 (UTC), each request with its own cache-buster, answered as
backlot.serve()does at722d3ce8:Linear answered the first two rows the same with no credential. graphql-depth-limit 1.1.0, run locally under graphql-js 16.14.2, throws on the two
__proto__operation rows and passes the rest.graphql-depth-limit and the locations past U+FFFF. api.fireflies.ai and api.linear.app on 2026-10-26 (UTC), the same way:
Apollo Server and graphql-js, run locally on 2026-10-25.
The code names neither Apollo Server major, as 4.13.0 configured otherwise cannot be ruled out.
The credential. The order of the checks is in
body_refusal's docstring, and every pair the tables hold was sent with both members wrong. Each refusal measured up to the choice of operation, on 2026-10-25 (UTC):graphql-js. Linear's refusal of an array where an input object goes is graphql-js 16.14.2's, where 16.9.0 takes each index of the array for a field's name.
test_a_refusal_is_in_graphql_js_wordsholds 42 rows, 38 of them from the 852 and three from graphql-js 16.14.2 run on 2026-10-26.Introspection, the 401 and an unknown id. api.linear.app on 2026-10-25 (UTC), one request per row with its own cache-buster:
With the key:
Replayed over
backlot.serve()at7c8e02f1, comparing the status and the body without its trailing newline:Methods, content types and Accept. api.linear.app and api.fireflies.ai on 2026-10-25 (UTC), with the key, no credential and
nope, and on Linear an API key underBearer, one request per row with its own cache-buster:Among them, and on 2026-10-26 one connection probe per answer, the empty
Origins and the plain text holding 0xE9:On 2026-10-28 (UTC), each request on a connection of its own with its own cache-buster:
backlot.serve()at270eaf3canswers each with the vendor's status, media type and body. The head holds the Linear, Fireflies and GraphQL code of270eaf3cunchanged but forjs_words' docstring.The layers answered in this order:
Which releases. Fireflies' landing page names
@apollo/server@5.5.1, and its 415 for a GET's content type came with Apollo Server 5.5.0; Linear answers no such 415. Run locally:Replayed over
backlot.serve()at270eaf3c, each request to/graphqlitself with a method uvicorn passes on (the other paths are the next draft's, which is #1058), comparing the status, the media type and the body without its trailing newline:Verification
pytestonuv sync --all-extras --lockedplusuv pip install --no-deps "llama-index-readers-hubspot<0.6", as CI installs: 8708 passed, 1 skipped.tests/test_mcp.py, which needs Docker's Linux containers and found no daemon here.ruff check . && ruff format --check .: clean.At
1bad7ec9, onmainatd0934c22.Each generated mutation changes one decision on a line this branch adds over
main; those by hand cover the depth rule, the cap on a variable's errors, the UTF-16 column, each call of_not_found, the introspection a request with no key is served, the preflights and the body's length check. Of those left, the trims of a preflight's header names (the table under Not in this PR.) and of anAcceptparameter differ only past ASCII, and the rest cannot change an answer. The run is on the tree beforemain's last two commits were merged in, which change GitHub's and Slack's baselines and GitHub's router, none of the files it mutates and none of its test files:test_linear_orders_the_body_and_the_credential,test_fireflies_orders_the_body_and_the_credential,test_linear_answers_each_method_type_and_accept_as_linear_does,test_cloudflare_answers_ahead_of_linear_and_fireflies,test_each_graphql_error_is_counted_by_the_code_backlot_sends.test_linear_by_id_roots_do_not_leak_entities_off_hidden_issues,test_fireflies_transcripts_list_hides_denied_meetings.