Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for Redshift connections #2215

Merged
merged 1 commit into from
Nov 24, 2024

Conversation

owenjonesuob
Copy link
Contributor

@owenjonesuob owenjonesuob commented May 3, 2024

An attempt to add support for Amazon Redshift connections.

Redshift is based on Postgres - which is great, because {dm} already has excellent Postgres support 😁

So for the most part, we can just borrow existing functionality.


There are some differences between Redshift and Postgres - more detains in the AWS docs. One note in particular caught my eye:

Unique, primary key, and foreign key constraints are permitted, but they are informational only. They are not enforced by the system, but they are used by the query planner. - https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-features.html

I'm not sure if this has any impact on whether we can build a dm by learning the database keys.

I also tried, but failed, to write a Redshift-friendly version of postgres_column_constraints since Redshift doesn't support some of the syntax used there (e.g. array_position()). This affects dm_meta_raw() - for now I have added an implementation based on the final else clause.


Testing any Redshift functionality directly might be tricky, since it would require a Redshift warehouse in an AWS organisation - i.e. a lot of additional infrastructure (and cost?!). Would we be happy to rely on the fact that it's "nearly Postgres", and therefore mostly covered by Postgres testing? Or can we think of some way to test directly?

Edit: that is, testing via CI/CD - otherwise we are perhaps relying on ad hoc testing by users who happen to have access to a Redshift warehouse...


Requested indirectly in #1678.

Copy link
Contributor

aviator-app bot commented May 3, 2024

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This PR was merged manually (without Aviator). Merging manually can negatively impact the performance of the queue. Consider using Aviator next time.


See the real-time status of this PR on the Aviator webapp.
Use the Aviator Chrome Extension to see the status of your PR within GitHub.

@krlmlr krlmlr force-pushed the f-redshift-support branch from 1dd6b91 to c8e5aca Compare August 14, 2024 21:11
@krlmlr krlmlr force-pushed the f-redshift-support branch from c8e5aca to c85fecb Compare November 21, 2024 22:02
@krlmlr krlmlr enabled auto-merge November 21, 2024 22:03
@krlmlr
Copy link
Collaborator

krlmlr commented Nov 21, 2024

Thanks! I have no easy way to test this, I trust this works for you.

@krlmlr krlmlr disabled auto-merge November 24, 2024 17:39
@krlmlr krlmlr enabled auto-merge (squash) November 24, 2024 17:39
@krlmlr krlmlr disabled auto-merge November 24, 2024 17:39
@krlmlr krlmlr merged commit 1ed4209 into cynkra:main Nov 24, 2024
20 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants