Error on s3 cp command when specifying the global s3 endpoint url

[simonrdz]

Describe the bug

The following command fails

aws s3 cp s3://mybucket/mypath/sample.txt  sample.txt --endpoint-url http://s3.amazonaws.com

Expected Behavior

The command should download the file

Current Behavior

The command does not download the file

2024-09-04 10:39:24,540 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2024-09-04 10:39:24,540 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7c42d1aff690>>
2024-09-04 10:39:24,540 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-09-04 10:39:24,540 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7c42d1aff090>>
2024-09-04 10:39:24,540 - MainThread - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2024-09-04 10:39:24,540 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7c42d27253a0>
2024-09-04 10:39:24,541 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7c42d1aff150>>
2024-09-04 10:39:24,541 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: An error occurred (301) when calling the HeadObject operation: Moved Permanently
Traceback (most recent call last):
  File "awscli/customizations/s3/s3handler.py", line 149, in call
  File "awscli/customizations/s3/fileinfobuilder.py", line 31, in call
  File "awscli/customizations/s3/filegenerator.py", line 141, in call
  File "awscli/customizations/s3/filegenerator.py", line 317, in list_objects
  File "awscli/customizations/s3/filegenerator.py", line 354, in _list_single_object
  File "awscli/botocore/client.py", line 360, in _api_call
  File "awscli/botocore/client.py", line 739, in _make_api_call
botocore.exceptions.ClientError: An error occurred (301) when calling the HeadObject operation: Moved Permanently
fatal error: An error occurred (301) when calling the HeadObject operation: Moved Permanently
2024-09-04 10:39:24,542 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Reproduction Steps

• Install aws cli v1 or v2
• try to get a file from an s3 bucket specifying --endpoint-url http://s3.amazonaws.com

Possible Solution

No response

Additional Information/Context

Did not reproduce with an old v1 version -> 1.18.140

CLI version used

aws-cli/1.34.11 Python/3.8.10 Linux/6.8.0-1014-aws botocore/1.35.11

Environment details (OS name and version, etc.)

Ubuntu 24.04

[adev-code]

Hey simonrdz@ thanks for reaching out! I have tested with the same environment (Ubuntu 24.04 and aws-cli/1.34.11) and ran the command:
$ aws s3 cp s3://bucket-name/sample.txt sample.txt --endpoint-url http://s3.amazonaws.com
The file from S3: sample.txt is downloaded locally. I was not able to get any errors or reproduce the same issues.

It would be recommended to use the latest version of AWS CLI also: https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html . Could you please provide the full debug logs of the command? You can get debug logs by adding --debug to your command, and redacting any sensitive information.

Thank you

[adev-code]

Hey simonrdz@, the behavior you are seeing is because the bucket is not in US-EAST-1. Per [1] [2], requests made to the global s3 endpoint: http://s3.amazonaws.com are directed to the buckets in US-EAST-1. In my testing, I was able to reproduce the error where my bucket is in US-WEST-2 Oregon and only had http://s3.amazonaws.com :
$ aws s3 cp s3://s3-bucket-in-oregon/sample.txt sample.txt --endpoint-url http://s3.amazonaws.com

Then resolving the error by adding the region on the request endpoint and I was able to download the object successfully:
$ aws s3 cp s3://s3-bucket-in-oregon/sample.txt sample.txt --endpoint-url http://s3.us-west-2.amazonaws.com

Please let me know if this works for you or not or if you have any follow up questions.
Thank you

[1] https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
[2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

[simonrdz]

Hi @adev-code I appreciate your feedbacks my point is how to handle the region without knowing it in advance.
The behavior I've used before is specifying the endpoint http://s3.amazonaws.com/ for a job that would call any bucket in another account or region without taking care of the target region. This is still possible with an old 1.18.140 cli version but not with the last ones. I just wonder if this old behavior is deprecated or if it is actually a bug. Thanks in advance

[adev-code]

Hi simonrdz@ S3 requests should go to a supported S3 endpoint. You can see the supported endpoints here: https://docs.aws.amazon.com/general/latest/gr/s3.html. If specifying –endpoint-url http://s3.amazonaws.com worked on older versions then that may have been a bug. But we recommend migrating to V2 of the AWS CLI if possible, or at least upgrading to a more recent version of v1, since 1.18.14 is several years old. (The latest version per the CHANGELOG is 1.34.16)

[simonrdz]

Hi thanks for your time. I didn't know the old behavior might have been a bug.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.