Skip to content

Make the agent SQL dialect accept what agents actually write, and say what it does - #1266

Merged
kirill-markin merged 5 commits into
mainfrom
integration-agent-sql-dialect-ergonomics
Aug 7, 2026
Merged

Make the agent SQL dialect accept what agents actually write, and say what it does#1266
kirill-markin merged 5 commits into
mainfrom
integration-agent-sql-dialect-ergonomics

Conversation

@kirill-markin

Copy link
Copy Markdown
Owner

Promotes integration-agent-sql-dialect-ergonomics (5 commits, 4 reviewed items) to main.

Why

Measured from Langfuse over 2026-07-07 to 2026-08-06: 170 dialect rejections across 2711 sql tool calls (6.3%). Turns that hit at least one rejection are 29.7% of turns but account for 62.4% of all SQL tool calls, with a median of 13 calls per turn against 3 for clean turns. The recovery loop, not the single rejection, is where the cost sits.

Two groups covered 68% of rejections:

Class Rejections
Parenthesized AND/OR groups in WHERE 58 (34%)
Columns that do not exist (cards.deck_id 35, decks.description 11, bare tag 4, …) 57 (33.5%)
NOT LIKE, array comparison, other predicate forms 19 (11%)

What changed

Grammar. The flat disjunctive-normal-form predicate list became a recursive boolean expression tree, so explicit parentheses parse and evaluate with AND binding tighter than OR. Added NOT LIKE, ILIKE, and array comparison (tags = ('a','b') exact set equality, tags = () for untagged). UPDATE and DELETE resolve target rows through the same evaluator, so they gain the forms too.

Truthful contract. Decks are saved tag filters; cards have no deck membership; decks have no description. OVERLAP and MATCH — implemented all along, documented nowhere, used once each in 2711 calls — are now advertised, with OVERLAP named as the way to filter writes by tag since UNNEST is SELECT-only. Schema discovery must be its own tool call: 78 of 157 discovery calls were batched with the statements that depended on them, which cannot work because a batch is composed before any statement runs.

Two silent wrong answers converted to clear errors. LIKE against a non-text column matched no rows at all and told the agent nothing existed — LOWER(tags) LIKE '%english%' appeared 9 times in the window, wrong every time. And broad UPDATE/DELETE truncated the match set to 100 rows and reported success, so a DELETE matching 250 rows deleted 100 and returned affectedCount: 100 while every surface promised a rejection past 100.

Review

Three cumulative reviews ran across the feature. They caught four published-contract defects that no per-item review could see, because each lived in one item and contradicted another:

  • an invented end-to-end ceiling of 5000 rows per write batch, false because of the independent MAX_SQL_RESULT_CHARS limit;
  • MATCH described as scanning a single column when it scans every value of the row;
  • negated IN on an array column described as matching every row when it matches none;
  • tags = (...) documented ambiguously as "array comparison" with no documented "all of these tags" form, so an agent wanting both tags would silently get an incomplete answer — the exact failure class this feature exists to remove, on a form the feature itself introduced.

The mutation truncation above was also a cumulative finding: the truncation predated the feature, but the feature made it the recommended path.

CI

GitHub Actions has been in a multi-hour major_outage; the last successful run in this repository was 14:11 UTC. Items 1 and 2 merged with green checks including backend tests and typecheck. Items 3 and 4 have no run — GitHub stopped dispatching workflow runs entirely. Do not merge this PR until required checks are green.

Accepted residuals

  • The new LIKE is not supported for column: <name> message does not name the offending construct when the agent wrote LOWER(tags) = 'english', which compiles to the same predicate. Safe to reword; no smoke script asserts it.
  • The claim that deleted_at and metadata can never appear in ORDER BY is over-broad: the grouped path does not consult sortable, so they are accepted as a GROUP BY key. Over-restrictive, so it produces no rejections.
  • Pre-existing, unchanged: OVERLAP against a non-array column and IS NULL on a non-nullable array column return zero rows silently; column IN (...) on an array column matches nothing (now documented); the LIKE pattern is recompiled per row; an empty top-level WHERE on a mutation matches all rows.
  • The bulk-write arithmetic exists as separate hand-maintained copies across the tool contract, the setup envelope, and the OpenAPI specs. Sharing one constant is deliberate follow-up.

🤖 Generated with Claude Code

kirill-markin and others added 5 commits August 6, 2026 14:53
… the agent SQL dialect (#1257)

Agents most often fail on three WHERE forms the published dialect did not
accept: parenthesized boolean groups, NOT LIKE, and comparing an array column
against a literal list. Measured over 30 days these were 76 of 170 dialect
rejections.

Replace the flat disjunctive-normal-form predicate list with a recursive
boolean expression so explicit parentheses parse and evaluate, keeping SQL
precedence where AND binds tighter than OR. Add NOT LIKE and ILIKE, and add
array comparison so tags = ('a','b') matches an exact tag set and tags = ()
matches cards without tags. UPDATE and DELETE resolve target rows through the
same evaluator, so they gain the forms too.

LIKE against a non-text column previously matched no rows at all and silently
told the agent that nothing existed; it now fails with a clear error instead.

Keep the tool contract and both OpenAPI path descriptions in sync with the
grammar in the same change.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ey never find (#1263)

The published contract left agents to guess three things, and measurement over
30 days shows what they guessed. 57 of 170 dialect rejections ask for columns
that do not exist: cards.deck_id 35 times and decks.description 11, because
agents import the deck model from other flashcard products. Schema discovery
did not correct them, since 78 of 157 discovery calls were batched into the same
sql string as the statements depending on the result, which cannot work because
a batch is composed before any statement runs.

State the real model: decks are saved tag filters, cards carry no deck
membership, and a card belongs to a deck only by matching its tags.

Document OVERLAP and MATCH, which have always been implemented but appear in no
description and were each used once in 2711 calls. OVERLAP is the only way to
filter by tag in UPDATE and DELETE, where UNNEST is unavailable.

Forbid batching schema discovery with dependent statements, and state the
bulk-write split arithmetic instead of letting agents discover the limits by
hitting them.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…does (#1264)

The cumulative review of this feature found the new documentation incomplete or
inaccurate in ways that invite the exact failure it was meant to remove.

tags = ('a','b') is exact set equality, so a card carrying any extra tag does
not match. That was documented only as "array comparison against a literal tag
list", sitting next to OVERLAP described as "at least one of", and no form for
"has all of these tags" was documented anywhere. An agent wanting both tags
would reach for the equality form and get a silently incomplete answer. State
the semantics, and document the AND form the new grammar makes expressible:
tags OVERLAP ('a') AND tags OVERLAP ('b').

Document the LIKE column-type restriction the dialect now enforces instead of
letting agents discover it by hitting it, and state the verified behavior of
LOWER(column) IN and NOT IN on array columns, which do not error and match no
rows.

Stop the "case-insensitive exact string matches" qualifier from covering the
LIKE-family pattern forms, stop advertising deleted_at without noting it cannot
appear in WHERE or ORDER BY, and stop the chat prompt from implying an
exhaustive deck column list that contradicts the tool description sent in the
same request.

Bring the bulk-write arithmetic to the agent discovery and setup surfaces, which
still carried the older sentence without the per-batch statement cap or the
no-mixed-read-write rule, so the discovery envelope and the published specs
agree.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…irst 100 rows (#1265)

Broad mutations resolved their target rows with limit 100, so paginateRows
truncated the match set to exactly 100 and dropped hasMore. The following
assertSqlMutationRecordLimit therefore saw 100 and never fired: a DELETE
matching 250 rows deleted 100, reported affectedCount 100, and returned a
success envelope. Every published surface says at most 100 rows per statement
and tells the agent to split larger work, which is a rejection contract rather
than silent partial application.

This mattered little while filtering a mutation by tag was undocumented and
awkward. The preceding commits made it the recommended path by advertising
DELETE FROM cards WHERE tags OVERLAP ('some-tag'), instructing agents to filter
writes by tag, and allowing parenthesized groups in mutation WHERE clauses, so
the latent truncation became the happy path.

Resolve target rows one row past the limit so a broader match set reaches the
existing assertion and the caller receives the documented rejection.

Also complete the WHERE grammar documentation: state that scalar comparisons and
IS NULL / IS NOT NULL are supported, and that only the LOWER(column) IN forms
are text-only while a plain column IN (...) compares numbers and booleans too.
The negated form exists solely as LOWER(column) NOT IN (...).

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@kirill-markin kirill-markin reopened this Aug 7, 2026
@kirill-markin
kirill-markin merged commit f48628b into main Aug 7, 2026
9 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant