-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add concept of connections #459
Comments
Documentation of GlareDB-specific statements, rework statements |
This is not to match a spec/standard, this is custom (no standard AFAIK for external table related statements) Sean --> Firebolt and Materialize for inspiration on the structure for connection statements. Sean is going to write up a spec. |
Once we have the concept of connections is the plan to remove the ability to create ad-hoc external tables with all the info provided? Wondering if we should have both, the current syntax where we can provide all credentials needed and the the syntax leveraging saved connections? |
Yeah the idea is to create an external table, a 'connection' of some sort is required. My rationale for that is I would rather have one well defined way of doing this. This does bring up issues like what should we do if there's a public parquet file on S3 that you want to access. The 'connection' in this case wouldn't contain anything useful. So I'm still thinking through that... |
Also we should consider some validation of the connection/credentials some time before querying. Currently I can provide a bad key and we will only find out when querying. Maybe as a follow-on task |
I think that is a fair reason. I will provide a scenario to consider. Say someone is using our product to query from their main datasources. They now need to do some ad-hoc transformations on a sizable number of public data sources. They don't need these persisted. Maybe we consider having a temp schema that we automatically clean up on graceful shutdown as an alternative to having a ad-hoc compatible syntax on |
100% agree.
I think there's two approaches we can take here. DuckDB has a The second would be to allow for creating temp external tables via
Ideally we would eventually have |
2023-01-16 Notes
|
Effectively done with the connections stub + metastore work in prog |
Summary
See https://www.notion.so/glaredb/Database-Objects-a7e52611154e40efabb7ac46bf0b2a76#fc55700503794ee6bdb784a4349f1768
Specifications
Rationale
Removes the need of specifying credentials every time an external table is added for a database. Also makes updating credentials easier (when we have support for that) since only one entry needs to be updated instead of multiple tables.
Impact
Will affect https://github.com/GlareDB/cloud/issues/431 and https://github.com/GlareDB/cloud/issues/473. Instead of executing one query, we'll need to execute two. However, doing these queries separately enables things like easily adding more that one table at a time, or even attaching an entire schema.
Tasks
The text was updated successfully, but these errors were encountered: