-
Notifications
You must be signed in to change notification settings - Fork 68
Replace listing namespaces with HEAD requests when scanning #440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
… enforce it when we find out the schema doesnt exist
…use the callback gets issued AFTER the transaction is destroyed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Looks good, but seems like Polaris is failing? I can imagine it's because they don't support the HEAD request to check if tables exist. Maybe I can take a look using our snowflake catalog
So I think for Polaris you need to check for There seems to be another error though, looking into that |
The other error is because the httpfs submodule pointer on main does not have the code to create a secret with just a bearer token. Working on that. duckdb/duckdb-httpfs#108 needs to merge, then merge v1.4-andium into main, then update the submodule pointer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now.
@Tishj feel free to look at my last commit, otherwise I think we can merge
I added myself as reviewer, but that's not really adding much. This seems sound and great to have |
Awesome to have this in 👍 |
This PR is an attempt to fix performance issues caused by nested namespaces.
As the title says, it replaces the current logic of running "LoadEntries" when first hitting the schema entry set in a transaction, which does a "list namespaces" request.
The replacement being to instead do a HEAD request to check the existence of the namespace in the lookup.
This is done for both
IRCSchemaSet::GetEntry
andIRCTableSet::GetEntry