Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KYLIN-5748 Skip build flat table for index build job if all new created indexes can be built from existed parent index #2162

Open
wants to merge 3 commits into
base: kylin5
Choose a base branch
from

Conversation

zznlime
Copy link
Contributor

@zznlime zznlime commented Dec 26, 2023

For an index build job, if all the new added indexes can be built from their existed parent index. No index was built from new created flat table.
For this case, there won't be any data inconsistency across indexes. And there is no need to build a new flat table to serve the stage of index build by layer.
So I think we can skip stage BUILD DICT and stage GENERATE FLAT TABLE for an index build job in condition that all new added indexes can be built from their existed parent index

@zznlime
Copy link
Contributor Author

zznlime commented Dec 26, 2023

@hit-lacus PTAL, Thanks

@Mrhs121
Copy link
Contributor

Mrhs121 commented Mar 26, 2024

Your proposal will be invalid in this use case: if you add a new bitmap type metric to the model, you can build the new index separately, and the task type is also INDEX_BUIDL. If you skip building the dictionary, an error will occur when generating encoding columns for the flat table later.

@pfzhan
Copy link
Contributor

pfzhan commented Sep 2, 2024

@zznlime Does this pr need to be merged into the kylin code? If necessary, please send pr to @liyang-kylin . I will resolve this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants