Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: multithreaded bulk import #1077

Merged
merged 47 commits into from
Dec 19, 2024

Conversation

tamassoltesz
Copy link
Contributor

@tamassoltesz tamassoltesz commented Nov 27, 2024

Summary of change

Core implementation of Bulk User Migration.

Related issues

Test Plan

(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your
changes work. Bonus points for screenshots and videos!)

Documentation changes

(If relevant, please create a PR in our docs repo, or create a checklist here
highlighting the necessary changes)

Checklist for important updates

  • Changelog has been updated
    • If there are any db schema changes, mention those changes clearly
  • coreDriverInterfaceSupported.json file has been updated (if needed)
  • pluginInterfaceSupported.json file has been updated (if needed)
  • Changes to the version if needed
    • In build.gradle
  • If added a new paid feature, edit the getPaidFeatureStats function in FeatureFlag.java file
  • Had installed and ran the pre-commit hook
  • If there are new dependencies that have been added in build.gradle, please make sure to add them
    in implementationDependencies.json.
  • Update function getValidFields in io/supertokens/config/CoreConfig.java if new aliases were added for any core
    config (similar to the access_token_signing_key_update_interval config alias).
  • Issue this PR against the latest non released version branch.
    • To know which one it is, run find the latest released tag (git tag) in the format vX.Y.Z, and then find the
      latest branch (git branch --all) whose X.Y is greater than the latest released tag.
    • If no such branch exists, then create one from the latest released branch.
  • If added a foreign key constraint on app_id_to_user_id table, make sure to delete from this table when deleting
    the user as well if deleteUserIdMappingToo is false.

Remaining TODOs for this PR

  • Item1
  • Item2

@tamassoltesz tamassoltesz changed the title Feat: multithreaded bulk import feat: multithreaded bulk import Nov 27, 2024
@tamassoltesz tamassoltesz marked this pull request as ready for review November 27, 2024 11:02
config.yaml Show resolved Hide resolved
devConfig.yaml Outdated Show resolved Hide resolved
src/main/java/io/supertokens/bulkimport/BulkImport.java Outdated Show resolved Hide resolved
src/main/java/io/supertokens/cronjobs/Cronjobs.java Outdated Show resolved Hide resolved
@sattvikc sattvikc merged commit b135066 into feat/bulk-import-base Dec 19, 2024
2 of 3 checks passed
@sattvikc sattvikc deleted the feat/multithreaded-bulk-import branch December 19, 2024 08:40
sattvikc added a commit that referenced this pull request Dec 19, 2024
* feat: Add BulkImport APIs and cron

* chore: update pull request template

* fix: Use the correct tenant config to create the proxy storage

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: PR changes

* fix: Update version

* fix: PR changes

* fix: PR changes

* fix: Rename DeleteBulkImportUser API path

* fix: disable bulk import for in-memory db

* fix: a bug with createTotpDevices

* fix: PR changes

* feat: Add an api to import user in sync

* feat: Add an api to get count of bulk import users

* fix: PR changes

* fix: Add error codes and plainTextPassword import

* fix: PR changes

* feat: multithreaded bulk import

* fix: changelog update

* fix: add new test

* fix: fixing unreliable mutithreaded bulk import with mysql

* fix: review fixes

* fix: fixing failing tests

* feat: bulkimport flow tests

* feat: bulk import cron starter api

* fix: tweaking params for faster import

* fix: tests

* checkpoint

* fix: remove vacuuming

* fix: minor tweaks

* feat: bulk inserting the bulk migration data

* fix: fast as a lightning

* fix: restoring lost method

* fix: reworked error handling to comform previous approach with messages

* fix: fixing tests

* fix: fixing failing tests, changing version

* chore: update changelog

* fix: fixing issues and failing tests

* fix: review changes

* fix: review fixes, reworking cron start/stop

---------

Co-authored-by: Tamas Soltesz <[email protected]>
Co-authored-by: Ankit Tiwari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants