Skip to content

fix(sql): preserve RECURSIVE keyword when rebuilding WITH clause - #12041

Open
bhargaviHQ wants to merge 5 commits into
ibis-project:mainfrom
bhargaviHQ:fix/bigquery-recursive-cte-11949-clean
Open

bhargaviHQ wants to merge 5 commits into
ibis-project:mainfrom
bhargaviHQ:fix/bigquery-recursive-cte-11949-clean

Conversation

@bhargaviHQ

@bhargaviHQ bhargaviHQ commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes

translate() and add_query_to_expr() rebuild the WITH clause from its CTEs and drop the original With node, which lost the RECURSIVE flag. The flag is now passed to with_(recursive=...) when the clause is rebuilt. In add_query_to_expr() it's kept if either the user's query or the ibis expression had it, so a recursive CTE coming from the table passed to Table.sql() also works.

CTE aliases are now built as TableAlias instead of a bare identifier, because sqlglot's Trino/Presto generator reads alias.columns when the clause is recursive. The same change is applied to SingleStoreDB's copy of add_query_to_expr(). On MSSQL this makes CTE projections explicit ([t0].[key] AS [key]), so one snapshot changed, with the same meaning.

Known limitations: BigQuery doesn't allow a nested WITH RECURSIVE, and on Trino/Athena a recursive query over a plain table still renders the CTE alias as "t"("*"), which already failed on main.

Issues closed

  The translate() and add_query_to_expr() methods in SQLGlotCompiler
  discarded the With node's recursive flag when reconstructing CTEs.
  Capture the flag before the pop and restore it after the reduce.

  Fixes ibis-project#11949
@github-actions github-actions Bot added tests Issues or PRs related to tests sql Backends that generate SQL labels Jul 20, 2026
@deepyaman
deepyaman self-requested a review October 10, 2026 06:07
@deepyaman

Copy link
Copy Markdown
Collaborator

@bhargaviHQ Sorry, I know this has been out there for a while, but I'm happy to review it.

deepyaman pushed a commit that referenced this pull request Oct 10, 2026
The translate() and add_query_to_expr() methods in SQLGlotCompiler
discarded the With node's recursive flag when reconstructing CTEs.
Capture the flag before the pop and restore it after the reduce.

Cherry-picked from #12041 (code change only; the tests come from the
previous commit).

Fixes #11949
deepyaman added a commit that referenced this pull request Oct 12, 2026
Restrict pytest to the tests relevant to #12041 so the BigQuery job
finishes within its 60-minute limit and prints its failure summary, and
so the other backends turn around faster. Jobs that select none of
these tests (doctests, core) will exit with "no tests collected".

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

@deepyaman deepyaman left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you take a look at #12153 and see what you think of the slightly updated approach? Happy to discuss if you have any thoughts/concerns. I also ran CI across all backends, including the cloud ones that currently work.

Edit: Just to be clear, I only opened #12153 to validate some ideas/run cloud backends; I don't intend to merge that, but rather you update this PR based on whatever makes sense.

Comment thread ibis/backends/sql/compilers/base.py Outdated
Comment thread ibis/backends/sql/compilers/base.py Outdated
@bhargaviHQ

Copy link
Copy Markdown
Contributor Author

Thanks @deepyaman for running this across all the backends. The updated approach makes sense, I will update this PR with it and the tests from #12153.

@github-actions github-actions Bot added the bigquery The BigQuery backend label Oct 13, 2026
@bhargaviHQ

Copy link
Copy Markdown
Contributor Author

Updated with the changes from #12153.

@bhargaviHQ
bhargaviHQ requested a review from deepyaman October 13, 2026 00:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bigquery The BigQuery backend sql Backends that generate SQL tests Issues or PRs related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Ibis removes mandatory "RECURSIVE" keyword from BigQuery recursive CTE definition

2 participants