response 404 from aws s3 for filenames with spaces #79

cpeled · 2022-12-08T06:56:56Z

cpeled
Dec 8, 2022

Hi.
in AWS, when filename with spaces is uploaded, in the generated URL, every space replaced with '+'.
example: i uploaded a filename: "n ginx.conf".
the URL generated by aws: https://.s3.eu-central-1.amazonaws.com/n+ginx.conf

when i try to download the file using s3 gw using path "n+ginx.conf", i get 404.

since the GW replaces '+' with '%2B', i made a try and in s3gateway.ls, in "_encodeURIComponent", i replaced back from '%2B' to '+', but then i got from AWS "SignatureDoesNotMatch" response, and in the details i see that the '+' was changed to '%20' which is the space code.

can you please help me understand the issue?

Best,
Chen

Answered by dekobon

Dec 14, 2022

Hi Chen,

I did the following verifications:

# Upload file with + in it to s3
$ echo "Hello World" > "file_with_+.txt"
$ aws s3 cp "file_with_+.txt" s3://testbucket/
upload: ./file_with_+.txt to s3://testbucket/file_with_+.txt

# Upload file with space in it to s3
$ aws s3 cp "file_with space.txt" s3://testbucket/
upload: ./file_with space.txt to s3://testbucket/file_with space.txt

# Run the container in another terminal
$ docker run --env-file ./test-settings --rm -it --publish 80:80 --name nginx-s3-gateway nginx-s3-gateway

# Download uploaded file with + in it escaped properly
$ curl "http://localhost/file_with_%2B.txt"
Hello World

# Verify that we can download uploaded file with + in…

View full answer

dekobon · 2022-12-13T01:19:55Z

dekobon
Dec 13, 2022
Maintainer

Hi Chen,

I transferred this from an issue to a discussion because I do not believe it is an issue. The gateway is doing the correct thing by escaping plus signs as %2B. I think there is a problem in the service that is uploading files to S3 because spaces should be encoded as %20 and not +.
See this StackOverflow conversation for some more detail.

In the automated tests for the gateway, we explicitly test filenames containing pluses. As noted by the comment, a plus must be encoded as %2B.

I hope this makes sense. I'm afraid that my explanation might be a bit confusing. Anyways, I think the problem is that your files that are uploaded to S3 should have %20 for spaces and not the + sign.

1 reply

cpeled Dec 13, 2022
Author

hi @dekobon and thanks for your response.

i tried to upload the file from:

aws s3 console
aws s3 cli.
aws boto3 python client

all resulted the same - the generated url set by aws was with '+' instead of space.

do you suggest to encode the filename before uploading? i upload it as is, means with ' ' and not '+'.

Chen

dekobon · 2022-12-14T19:02:44Z

dekobon
Dec 14, 2022
Maintainer

Hi Chen,

I did the following verifications:

# Upload file with + in it to s3
$ echo "Hello World" > "file_with_+.txt"
$ aws s3 cp "file_with_+.txt" s3://testbucket/
upload: ./file_with_+.txt to s3://testbucket/file_with_+.txt

# Upload file with space in it to s3
$ aws s3 cp "file_with space.txt" s3://testbucket/
upload: ./file_with space.txt to s3://testbucket/file_with space.txt

# Run the container in another terminal
$ docker run --env-file ./test-settings --rm -it --publish 80:80 --name nginx-s3-gateway nginx-s3-gateway

# Download uploaded file with + in it escaped properly
$ curl "http://localhost/file_with_%2B.txt"
Hello World

# Verify that we can download uploaded file with + in it not escaped properly
$ curl "http://localhost/file_with_+.txt"
Hello World

# Now, let's try it again to make absolutely sure that curl is not escaping the request
$ nc localhost 80
GET /file_with_+.txt HTTP/1.1
Host: localhost
User-Agent: curl/7.68.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 14 Dec 2022 18:50:52 GMT
Content-Type: text/plain
Content-Length: 12
Connection: keep-alive
Last-Modified: Wed, 14 Dec 2022 18:44:36 GMT
ETag: "e59ff97941044f85df5297e1c302d260"
Accept-Ranges: bytes

Hello World

# Download uploaded file with space in it escaped properly
$ curl "http://localhost/file_with%20space.txt"
Hello World

# Verify that we cannot download uploaded file with space in it not escaped properly
$ curl "http://localhost/file_with space.txt"
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

# Let's try plus and see what happens
$ curl "http://localhost/file_with+space.txt"
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

# It appears to have the same behavior - as both are invalid URLs
# Let's see what the s3 client does:
$ aws s3 --debug cp "s3://testbucket/file_with space.txt" /tmp/
# It outputs a lot of debug information, but the important part is:
# 2022-12-14 11:00:17,724 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://testbucket.s3.amazonaws.com:443 "GET /file_with%20space.txt HTTP/1.1" 200 12
# That shows that AWS is using %20 to escape spaces

Does this clarify the behavior for you?

1 reply

cpeled Dec 19, 2022
Author

Hi @dekobon ,
thank you so much for this clarification!

it helped me to find the issue in my system that was indeed in the caller side and not the gw side.

Best,
Chen

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

response 404 from aws s3 for filenames with spaces #79

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

response 404 from aws s3 for filenames with spaces #79

cpeled Dec 8, 2022

Replies: 2 comments · 2 replies

dekobon Dec 13, 2022 Maintainer

cpeled Dec 13, 2022 Author

dekobon Dec 14, 2022 Maintainer

cpeled Dec 19, 2022 Author

cpeled
Dec 8, 2022

Replies: 2 comments 2 replies

dekobon
Dec 13, 2022
Maintainer

cpeled Dec 13, 2022
Author

dekobon
Dec 14, 2022
Maintainer

cpeled Dec 19, 2022
Author