Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R2 does not work with S3 endpoint | aws: s3 write file: failed to create multipart upload: NotImplemented #3094

Open
8times4 opened this issue May 28, 2024 · 0 comments
Labels
bug Something isn't working minor nice to have enhancement

Comments

@8times4
Copy link

8times4 commented May 28, 2024

Describe the bug
Dragonfly throws an error when trying to save to an R2 bucket.

To Reproduce
Steps to reproduce the behavior:

Quick docker reproduction command: (still requires a cloudflare r2 account)

 docker run --rm -p 6379:6379 -e AWS_ACCESS_KEY_ID=<r2 access key> -e AWS_SECRET_ACCESS_KEY=<r2 secret key> -e AWS_REGION=us-east-1 --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly:latest --dir s3://dflydb-prod --logtostderr --requirepass=secretpass --s3_endpoint=<accountid>.r2.cloudflarestorage.com --s3_use_https=true

Try to either just cmd/ctrl+c the container, or try to send a SAVE command to it from a redis-cli

Expected behavior
Expected to save to R2.

Logs

I20240528 13:10:03.215538     1 init.cc:70] dragonfly running in opt mode.
I20240528 13:10:03.215699     1 dfly_main.cc:646] Starting dragonfly df-v1.18.1-6851a4c845625b0b14bb145177322dafbbc9858e
W20240528 13:10:03.215804     1 dfly_main.cc:685] SWAP is enabled. Consider disabling it when running Dragonfly.
I20240528 13:10:03.215816     1 dfly_main.cc:690] maxmemory has not been specified. Deciding myself....
I20240528 13:10:03.215817     1 dfly_main.cc:699] Found 4.32GiB available memory. Setting maxmemory to 3.46GiB
W20240528 13:10:03.215864     1 dfly_main.cc:373] Weird error 1 switching to epoll
I20240528 13:10:03.217207     1 proactor_pool.cc:147] Running 8 io threads
I20240528 13:10:03.222258     1 server_family.cc:721] Host OS: Linux 6.7.12-orbstack-00201-g2ddb8f197a46 aarch64 with 8 threads
I20240528 13:10:03.235483    12 snapshot_storage.cc:185] Creating AWS S3 client; region=us-east-1; https=true; endpoint=<account_id>.r2.cloudflarestorage.com
I20240528 13:10:03.235543    12 credentials_provider_chain.cc:28] aws: disabled EC2 metadata
I20240528 13:10:03.236860    12 credentials_provider_chain.cc:36] aws: loaded credentials; provider=environment
I20240528 13:10:03.240021    13 snapshot_storage.cc:242] Load snapshot: Searching for snapshot in S3 path: s3://dflydb-prod/
W20240528 13:10:03.485853     1 server_family.cc:814] Load snapshot: No snapshot found
I20240528 13:10:03.497128     9 listener_interface.cc:101] sock[19] AcceptServer - listening on port 6379
^CI20240528 13:10:04.933329     8 accept_server.cc:25] Exiting on signal Interrupt
I20240528 13:10:04.935065     9 listener_interface.cc:219] Listener stopped for port 6379
E20240528 13:10:05.040671     9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.125510    15 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.130617     9 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.132378    11 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.139791    14 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:05.144387     8 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.125360    13 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.140058    10 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
E20240528 13:10:06.144856    12 s3_write_file.cc:137] aws: s3 write file: failed to create multipart upload: NotImplemented
W20240528 13:10:06.148021     8 server_family.cc:841] Failed to perform snapshot Input/output error: Failed to open write file

Environment (please complete the following information):

  • OS: [ubuntu 22.04 / MacOs 14.5]
  • Containerized?: [k8s via operator & Docker]
  • Dragonfly Version: [latest]

Additional context
R2's S3 api support: https://developers.cloudflare.com/r2/api/s3/api/

Thank you!

  • 8x4
@8times4 8times4 added the bug Something isn't working label May 28, 2024
@romange romange added the minor nice to have enhancement label Jun 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working minor nice to have enhancement
Projects
None yet
Development

No branches or pull requests

2 participants