Describe the feature
Currently, transfermanager supports multi-part parallel upload only using file as an input. If an InputStream is provided then multi-part upload becomes sequential where all parts are lined up in sequence and uploads are performed. This is a proposal to provide support to accept multiple InputStreams each emitted for a specific portion of content and to perform concurrent uploads where each stream is responsible for the transfer of a specific part. At the end of the upload, complete upload request is triggered to merge the uploaded parts.
Use Case
There are scenarios where pre-processing of content may be required before its transfer. Let's say that this pre-processing work can be carried out independently for different parts of the content. Now, to carry out both pre-processing and transfer of content, we have two options available. Either content is read, pre-processed and transferred serially using InputStream or content is first pre-processed completely by pre-processing different parts concurrently and then making the transfer as a separate operation. In first approach, we avoid duplicate reads but end up making the whole process latent and in second approach we lose streaming nature of the content and end up reading the content twice.
Proposed Solution
What we need here is the ability to apply the package of all streaming operations on a unit of content vs applying individual operation on the whole content. Multiple InputStreams emitted for different parts of content can solve this problem where each stream applies all streaming operations like pre-processing, transfer, post-processing on a unit of content. This can be achieved by assigning an InputStream to a PutObjectRequest request in the multi-part upload process and carrying out the transfer in parallel.
Other Information
We were able to achieve multi-part parallel upload using streams for async uploads using S3AsyncClient. We were also able to customize v1 transfermanager and achieve the same there as well. We haven't got a chance to accommodate these changes in v2 transfermanager though. Following is a reference PR of async upload :
https://github.com/opensearch-project/OpenSearch/pull/7217/files#diff-a472eda8bf5f051224172440b745603502d3545f8326b28d6014a79d5a833cd5
An iterable pattern is used in the class to create a stream for each part.
Stream stream = streamContext.getStreamProvider().provideStream(partIdx)
Acknowledgements
AWS Java SDK version used
2.20.26
JDK version used
11
Operating System and version
OSx
Describe the feature
Currently, transfermanager supports multi-part parallel upload only using file as an input. If an InputStream is provided then multi-part upload becomes sequential where all parts are lined up in sequence and uploads are performed. This is a proposal to provide support to accept multiple InputStreams each emitted for a specific portion of content and to perform concurrent uploads where each stream is responsible for the transfer of a specific part. At the end of the upload, complete upload request is triggered to merge the uploaded parts.
Use Case
There are scenarios where pre-processing of content may be required before its transfer. Let's say that this pre-processing work can be carried out independently for different parts of the content. Now, to carry out both pre-processing and transfer of content, we have two options available. Either content is read, pre-processed and transferred serially using InputStream or content is first pre-processed completely by pre-processing different parts concurrently and then making the transfer as a separate operation. In first approach, we avoid duplicate reads but end up making the whole process latent and in second approach we lose streaming nature of the content and end up reading the content twice.
Proposed Solution
What we need here is the ability to apply the package of all streaming operations on a unit of content vs applying individual operation on the whole content. Multiple InputStreams emitted for different parts of content can solve this problem where each stream applies all streaming operations like pre-processing, transfer, post-processing on a unit of content. This can be achieved by assigning an InputStream to a
PutObjectRequestrequest in the multi-part upload process and carrying out the transfer in parallel.Other Information
We were able to achieve multi-part parallel upload using streams for async uploads using S3AsyncClient. We were also able to customize v1 transfermanager and achieve the same there as well. We haven't got a chance to accommodate these changes in v2 transfermanager though. Following is a reference PR of async upload :
https://github.com/opensearch-project/OpenSearch/pull/7217/files#diff-a472eda8bf5f051224172440b745603502d3545f8326b28d6014a79d5a833cd5
An iterable pattern is used in the class to create a stream for each part.
Stream stream = streamContext.getStreamProvider().provideStream(partIdx)Acknowledgements
AWS Java SDK version used
2.20.26
JDK version used
11
Operating System and version
OSx