RuntimeError: cannot schedule new futures after interpreter shutdown

[lphuberdeau]

Reverting to boto3 1.17.53, which is the last version using s3transfer<0.4 solves the issue.

The calls to upload_fileobj and download_fileobj are performed within a thread via asyncio's run_in_executor.

Traceback (most recent call last):
...
    s3.download_fileobj(self.bucket_name, key, fp)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 355, in _submit
    self._submit_download_request(
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 384, in _submit_download_request
    self._transfer_coordinator.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 320, in submit
    future = executor.submit(task, tag=tag)
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Traceback (most recent call last):
...
    s3.upload_fileobj(io.BytesIO(data), self.bucket_name, key)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 536, in upload_fileobj
    future = manager.upload(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 326, in upload
    return self._submit_transfer(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 500, in _submit_transfer
    self._submission_executor.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

[jpl-jengelke]

Actually, reverting to an earlier Boto3 version didn't solve the problem. (It was verified using pip freeze that s3transfer was in the 3.x version range.) It did start working when I disabled threading on the s3 operations using TransferConfig().