Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BatchWriteError Error when dy import #259

Open
YN-CTC opened this issue Jul 22, 2024 · 4 comments
Open

BatchWriteError Error when dy import #259

YN-CTC opened this issue Jul 22, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@YN-CTC
Copy link

YN-CTC commented Jul 22, 2024

I'm encountering an issue while trying to import data into a table using Dynein. The process starts, but eventually fails with a BatchWriteError. It seems like the problem might be related to the aws-sdk-rust. Below are the details of the attempts and the errors received.

$ dy import -t <table_name> -i import_data.json
WARN: For the best performance on import/export, dynein recommends OnDemand mode. However the target table is Provisioned mode now. Proceed anyway? 
[WARN: For the best performance on import/export, dynein recommends OnDemand mode. However the target table is Provisioned mode now. Proceed anyway? yes
8350 items processed (1201.56 items/sec)Error: BatchWriteError(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source:CredentialsNotLoaded(CredentialsNotLoaded { source: Some("no providers in chain provided credentials") }), connection: Unknown } }))

$ dy import -t <table_name> -i import_data.json
1975 items processed (1026.56 items/sec)Error: BatchWriteError(DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source:ProviderError(ProviderError { source: Unexpected(Unexpected { source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s })}) }), connection: Unknown } }))
@StoneDot
Copy link
Contributor

Thank you for using dynein and reporting the error. A credential provider's throttling might cause this issue. The current implementation does not reuse clients. I think that the code change on our side should fix this issue.
https://github.com/awslabs/dynein/blob/main/src/batch.rs#L240

@StoneDot StoneDot added the bug Something isn't working label Jul 22, 2024
@YN-CTC
Copy link
Author

YN-CTC commented Jul 24, 2024

Thank you for your response.
Due to the need for immediate use in business, I modified the context function in app.rs to load the authentication information and instantiate the DynamoDbSdkClient. Additionally, by reusing the instantiated client within the context function, I confirmed that the above-mentioned bug no longer occurs.

@StoneDot
Copy link
Contributor

Thank you for sharing your status. I am working on improving the import/export command, and I expect the new implementation will resolve this issue. Therefore, I do not have plans to address this issue individually. I will update this thread once the work is complete. Is this acceptable to you? Please feel free to let me know if you have any other concerns.

@YN-CTC
Copy link
Author

YN-CTC commented Jul 28, 2024

There's no problem with this. I am looking forward to the improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants