Tornado Support?

[davidblewett]

I started looking at this package when I was trying to determine how easy/possible it would be to get boto to use Tornado's IOLoop/AsyncHTTPClient for pushing data to S3. I see there are many similar abstractions (Futures, queues, task management, etc), but it wasn't clear to me what would need to be adapted to work in the context of Tornado. It seems like it should be possible with an Executor subclass that has shims to convert between Tornado's coroutines and this library. If you're interested in supporting Tornado, I would appreciate some guidance. Thanks!

[jamesls]

I'm pretty sure this is not possible with the code as it's written today. It makes assumptions that we're operating in a synchronous environment.

The underlying lying library making the S3 API calls, botocore, also doesn't support async.

I think it would be great to support this, but I don't know offhand exactly how we'd tackle this.

@kyleknap feel free to chime in if I've missed something.

[kyleknap]

Yeah I think it would be really awesome to get async support. @jamesls summed it up pretty well.

The current blocker is botocore does not have built-in support for async. I know there are are some botocore/async implementations out there, but it would be great to have first-class support for this as s3transfer heavily relies on botocore's client interface. Once that is implemented, we can start looking to see how we can refactor the internals of s3transfer to remove any of the synchronous assumptions we are making. An audit would need be done to figure those out though.