Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse buffer for streamingUpload #34

Open
axman6 opened this issue Jan 6, 2023 · 1 comment
Open

Reuse buffer for streamingUpload #34

axman6 opened this issue Jan 6, 2023 · 1 comment

Comments

@axman6
Copy link
Owner

axman6 commented Jan 6, 2023

In streamingUpload, a new buffer is allocated for every chunk in the call to finaliseS. A better idea might be to allocate one buffer and reuse it - at the moment there shouldn't be any possibility of concurrent access to the buffer between different requests.

Taking this one step further, instead of using the S abstraction, we could write directly into this shared buffer when each ByteString is received. This would mean needing to be very careful about ordering so we don't start writing the next chunk into the buffer before sending it, but I think that should be handled by yield passing the chunk to be sent before the conduit continues.

@endgame
Copy link
Collaborator

endgame commented Jan 6, 2023

I would also be interested in concurrent uploads, even when drawing from a stream. Buffering N×chunksize blocks of memory doesn't seem that bad, but getting concurrent uploads and reused buffers right at the same time could be really hard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants