Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat (sync service): multi tenancy #1886

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from

Conversation

kevin-dp
Copy link
Contributor

@kevin-dp kevin-dp commented Oct 24, 2024

Supersedes #1868. I kept that PR up to consult its diff until all the issues have been resolved here.

This PR fixes #1591.
It enables Electric to work with multiple databases (i.e. tenants).

Architectural overview

multi-tenant-electric

TODO:

  • rebase on top of main
  • review commented out code and either remove it or make sure the place it had been moved to has been updated for multitenancy
  • add a database_id query param to the health endpoint to request the health of a given tenant
  • extract validate_tenant_id, load_tenant and assign_tenant which currently are duplicated in the create shape plug and delete shape plug.
  • Implement a two-tier ETS storage for column info and relation data. In this PR we replace two named tables with anonymous ones, necessitating a roundtrip to a single gen server just to fetch the table handle. This gen server becomes a bottleneck. We should either create another static ETS table to lookup individual ETS table handles in or store all column info and relation info in two static (named) tables, ensuring they don't trump each other via proper key design.
  • unit tests for multi tenancy
  • unit tests for add DB plug
  • unit tests for delete DB plug
  • e2e tests for multi tenancy
    • debug flaky cache tests
  • add a diagram explaining the new architecture
  • update the open API spec (for add DB endpoint and remove DB endpoint)
  • persist tenant information and restore it on startup

@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch from 023c50d to 51f005c Compare October 24, 2024 07:59
@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch 2 times, most recently from b1c6c26 to f5def2e Compare October 24, 2024 10:13
@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch from f5def2e to 835a72f Compare October 24, 2024 10:19
Copy link

netlify bot commented Oct 28, 2024

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 6ad20f5
🔍 Latest deploy log https://app.netlify.com/sites/electric-next/deploys/6720a903a3c7550008fccb6d
😎 Deploy Preview https://deploy-preview-1886--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch from 7e96fdc to bc347db Compare October 28, 2024 11:39
@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch from bc347db to 775190e Compare October 28, 2024 11:43
@kevin-dp kevin-dp force-pushed the kevin/multi-tenancy-continued branch from 9a4bd76 to 2a888bf Compare October 28, 2024 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add multi-database support
3 participants