Skip to content

feat: support self join elimination #19169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhang2014 merged 10 commits into from
Jan 3, 2026
Merged

feat: support self join elimination #19169

zhang2014 merged 10 commits into from
Jan 3, 2026

Conversation

@SkyFan2002
Copy link
Member

@SkyFan2002 SkyFan2002 commented Dec 26, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Implement RuleEliminateSelfJoin to detect redundant self-join branches in inner-join trees (same table + same group key, strict vs. loose aggregates), remove those branches, and restore required outputs via an EvalScalar column remapping.
  • Use EagerAggregation as a speculative pre-rewrite to expose patterns for self-join elimination; if no self-join is eliminated, revert to the original plan so the speculative rewrite doesn’t persist.
  • Improve mark-join handling by tracking removed mark column indexes in metadata and preventing eager-aggregation extra EvalScalar pushdown from depending on removed mark columns.
  • Refine ANY subquery decorrelation to only apply casts when the types actually differ, avoiding unnecessary casting.

TPC-H Q18:
Without Cache: 100.57s -> 83.69s
With Cache: 73.79s->70.42s

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Dec 26, 2025
@SkyFan2002 SkyFan2002 added the ci-cloud Build docker image for cloud test label Dec 26, 2025
@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 26, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 26, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 26, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@github-actions
Copy link
Contributor

github-actions bot commented Dec 26, 2025

Docker Image for PR

  • tag: pr-19169-ee93522-1766741096

note: this image tag is only available for internal use.

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 29, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 29, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 29, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 29, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 29, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 29, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002 SkyFan2002 added ci-benchmark Benchmark: run all test ci-benchmark-load Benchmark: run data load test and removed ci-benchmark Benchmark: run all test labels Dec 29, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2025

Docker Image for PR

  • tag: pr-19169-aa21842-1767016466

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2025

Docker Image for PR

  • tag: pr-19169-aa21842-1767019738

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2025

ClickBench Report

@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2025

ClickBench Report

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 30, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 30, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 30, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 30, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 30, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 30, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002
Copy link
Member Author

SkyFan2002 commented Dec 30, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 30, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@SkyFan2002 SkyFan2002 marked this pull request as ready for review December 30, 2025 06:05
@SkyFan2002 SkyFan2002 requested a review from zhang2014 December 30, 2025 06:06
@zhang2014 zhang2014 merged commit 10c5ba6 into databendlabs:main Jan 3, 2026
93 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@zhang2014 zhang2014 zhang2014 approved these changes

Assignees

No one assigned

Labels

ci-benchmark-load Benchmark: run data load test ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SkyFan2002 @zhang2014