Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use aws s3 sync #8329

Closed
h0806449f opened this issue Nov 15, 2023 · 7 comments
Closed

Unable to use aws s3 sync #8329

h0806449f opened this issue Nov 15, 2023 · 7 comments
Assignees
Labels
bug This issue is a bug. p3 This is a minor priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. s3

Comments

@h0806449f
Copy link

Describe the bug

aws s3 sync s3://test-bucket-for-sync-source s3://test-bucket-for-sync-destination

can not proceed without turn on the "Transfer Acceleration"

截圖 2023-11-15 22 14 51

Expected Behavior

copy s3://test-bucket-for-sync-source to s3://test-bucket-for-sync-destination

and all object will from source bucket to destination bucket

Current Behavior

fatal error: An error occurred (InvalidRequest) when calling the ListObjectsV2 operation: S3 Transfer Acceleration is not configured on this bucket

截圖 2023-11-15 22 14 51

Reproduction Steps

aws s3 sync s3://test-bucket-for-sync-source s3://test-bucket-for-sync-destination

Possible Solution

turn on Transfer acceleration on both source bucket and destination bucket will fix the problem

Additional Information/Context

Both my bucket in same AZ, I did not want to use Transfer acceleration.

The bucket created on today (2023/11/15) has this issue, if the source bucket and destination bucket are both created on 2023 April won't have this issue.

CLI version used

aws-cli / 2.13.34

Environment details (OS name and version, etc.)

AWS CloudShell

@h0806449f h0806449f added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 15, 2023
@h0806449f
Copy link
Author

aws s3 ls s3://test-bucket-for-sync-source
will get same error and ask for turn on S3 Transfer Acceleration.

截圖 2023-11-16 12 15 04

@RyanFitzSimmonsAK RyanFitzSimmonsAK self-assigned this Nov 16, 2023
@RyanFitzSimmonsAK RyanFitzSimmonsAK added s3 investigating This issue is being investigated and/or work is in progress to resolve the issue. p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Nov 16, 2023
@RyanFitzSimmonsAK
Copy link
Contributor

RyanFitzSimmonsAK commented Nov 16, 2023

Hi @h0806449f, thanks for reaching out. Could you run aws configure get s3_use_accelerate_endpoint? If this value is true, then the CLI is trying to use S3 Transfer Acceleration endpoints for all S3 requests, even on buckets that don't have it enabled. You can change this value to false with aws configure set s3_use_accelerate_endpoint false. Hope that helps!

Relevant documentation : https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html

@RyanFitzSimmonsAK RyanFitzSimmonsAK added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Nov 16, 2023
@h0806449f
Copy link
Author

Thanks, problems solved.
Some instance behind AWS CloudShell set the .aws/config s3_use_accelerate_endpoint = true, some set this value as false

Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

@h0806449f h0806449f reopened this Nov 17, 2023
@h0806449f
Copy link
Author

h0806449f commented Nov 17, 2023

Maybe the priority of default value and manual setting is wrong ?

cat ./.aws/config
[default]
s3 = use_accelerate_endpoint = true
s3_use_accelerate_endpoint = false

will cause
fatal error: An error occurred (InvalidRequest) when calling the ListObjectsV2 operation: S3 Transfer Acceleration is not configured on this bucket

Have to reset default value by using
aws configure set default.s3.use_accelerate_endpoint false
can fix the fatal error

@RyanFitzSimmonsAK
Copy link
Contributor

Adding default. is specifying that you want to change these settings for the profile named default.

s3 = use_accelerate_endpoint = true is under the s3 key, and can be modified with the syntax aws configure set s3.use_accelerate_endpoint false. More information can be found here.

s3_use_accelerate_endpoint = false is under the name of the profile, and can be modified with the syntax aws configure set s3_use_accelerate_endpoint false.

It looks like the s3 key takes priority if both are set.

Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p3 This is a minor priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. s3
Projects
None yet
Development

No branches or pull requests

2 participants