Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up 15x with parallel transfers? #23

Open
inspire22 opened this issue Jan 7, 2014 · 0 comments
Open

speed up 15x with parallel transfers? #23

inspire22 opened this issue Jan 7, 2014 · 0 comments

Comments

@inspire22
Copy link

Not sure if this has been looked into already, but the speed of transfers seems to be vastly improved by threads/processes running in parallel.

My experience with is is using dreamhost, it's possible the official s3 is different.

I noticed a 5x improvement in speed in calling boto-rsync on each of 5 large directories, rather than on the single parent directory. I assume an internal system for using multiple upload threads/queues would see similar improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant