-
Notifications
You must be signed in to change notification settings - Fork 541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add BulkImport APIs and cron #966
Conversation
src/main/java/io/supertokens/webserver/api/bulkimport/BulkImportAPI.java
Outdated
Show resolved
Hide resolved
src/main/java/io/supertokens/webserver/api/bulkimport/BulkImportAPI.java
Show resolved
Hide resolved
src/main/java/io/supertokens/bulkimport/BulkImportUserUtils.java
Outdated
Show resolved
Hide resolved
src/test/java/io/supertokens/test/bulkimport/apis/AddBulkImportUsersTest.java
Outdated
Show resolved
Hide resolved
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Outdated
Show resolved
Hide resolved
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Outdated
Show resolved
Hide resolved
src/test/java/io/supertokens/test/bulkimport/ProcessBulkImportUsersCronJobTest.java
Outdated
Show resolved
Hide resolved
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Show resolved
Hide resolved
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Show resolved
Hide resolved
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Outdated
Show resolved
Hide resolved
src/main/java/io/supertokens/webserver/api/bulkimport/BulkImportAPI.java
Show resolved
Hide resolved
if (arr.size() == 0) { | ||
throw new ServletException(new WebserverAPI.BadRequestException("Field name 'ids' cannot be an empty array")); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in this case, just return status code 200 ok
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in the right JSON output format ofc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think its better to return an error because calling this API with no ids is not a valid case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it makes it harder for people to call the API. Cause if they have a loop in which they are calling the API, they will be forced to check if the input is empty before calling the API (as opposed to being flexible enough to allow them to check it after and then exit the loop)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense. I have updated it.
src/main/java/io/supertokens/webserver/api/bulkimport/BulkImportAPI.java
Outdated
Show resolved
Hide resolved
this.allUserRoles = allUserRoles; | ||
this.allExternalUserIds = new HashSet<>(); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in the bulkimport user type, we also need to allow plain text password cause it will be used during lazy migration task
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is done.
public class ProcessBulkImportUsers extends CronTask { | ||
|
||
public static final String RESOURCE_KEY = "io.supertokens.ee.cronjobs.ProcessBulkImportUsers"; | ||
private Map<String, SQLStorage> userPoolToStorageMap = new HashMap<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to allow parallelism:
- Allow devs to configure the parallelism. Default is 1.
- The prop can be called
BULK_MIGRATION_PARALLELISM
. This is a SaaS protected prop and can be added toPROTECTED_CONFIGS
in CoreConfig.java. It should have the@NotConflictingInApp
annotation. - You can make a pool of proxy storages here based on the value of this prop, and create those number of threads as well.
src/main/java/io/supertokens/webserver/api/bulkimport/BulkImportAPI.java
Show resolved
Hide resolved
.collect(JsonArray::new, JsonArray::add, JsonArray::addAll); | ||
|
||
JsonObject errorResponseJson = new JsonObject(); | ||
errorResponseJson.add("errors", errors); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is an example of how this output looks like?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{
"errors": [
"Role role1 does not exist.",
"Role role2 does not exist.",
"Invalid tenantId: t1 for emailpassword recipe.",
"Invalid tenantId: t1 for thirdparty recipe.",
"Invalid tenantId: t1 for passwordless recipe."
]
}
} | ||
|
||
JsonObject input = InputParser.parseJsonObjectOrThrowError(req); | ||
JsonObject jsonUser = InputParser.parseJsonObjectOrThrowError(input, "user", false); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i dont think the input should have a user prop. Cause then its like:
{
user: {...}
}
Instead. it should just have all the details in the root of the json input
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
src/main/java/io/supertokens/cronjobs/bulkimport/ProcessBulkImportUsers.java
Outdated
Show resolved
Hide resolved
StorageTransactionLogicException exception = (StorageTransactionLogicException) e; | ||
errorMessage[0] = exception.actualException.getMessage(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
StorageQueryException usually means something like the db is down or something. All errors that are due to semantic issues have a different type (like emailalreadyexistsexception). Please double check error types
Summary of change
(A few sentences about this PR)
Related issues
Test Plan
(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your
changes work. Bonus points for screenshots and videos!)
Documentation changes
(If relevant, please create a PR in our docs repo, or create a checklist here
highlighting the necessary changes)
Checklist for important updates
coreDriverInterfaceSupported.json
file has been updated (if needed)pluginInterfaceSupported.json
file has been updated (if needed)build.gradle
getPaidFeatureStats
function in FeatureFlag.java filebuild.gradle
, please make sure to add themin
implementationDependencies.json
.getValidFields
inio/supertokens/config/CoreConfig.java
if new aliases were added for any core config (similar to theaccess_token_signing_key_update_interval
config alias).git tag
) in the formatvX.Y.Z
, and then find thelatest branch (
git branch --all
) whoseX.Y
is greater than the latest released tag.app_id_to_user_id
table, make sure to delete from this table when deleting the user as well ifdeleteUserIdMappingToo
is false.