Cannot upload backup to alternative S3 provider: "Transfering payloads in multiple chunks using aws-chunked is not supported."

[onigoetz]

What steps did you take and what happened:
In a new install of Velero I could not backup to an S3 provider (Infomaniak Swissbackup).
I get an error about NotImplemented: Transfering payloads in multiple chunks using aws-chunked is not supported.

What did you expect to happen:

Apparently S3 clients have a way to use either aws-chunked or the "classic" multipart upload. I would have expected it to fallback to this or have an option to configure

kubectl logs deployment/velero -n velero

time="2024-12-22T22:36:04Z" level=error msg="backup failed" backuprequest=velero/homepage-backup controller=backup error="rpc error: code = Unknown desc = error putting object velero_bsl/backups/homepage-backup/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 501, RequestID: tx005a7ef891d549638010c-0067689454, HostID: tx005a7ef891d549638010c-0067689454, api error NotImplemented: Transfering payloads in multiple chunks using aws-chunked is not supported." logSource="pkg/controller/backup_controller.go:291"

velero backup describe or kubectl get backup/ -n velero -o yaml

❯ velero backup describe homepage-backup
Name:         homepage-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.28.7
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=28

Phase:  Failed (run `velero backup logs homepage-backup` for more information)


Namespaces:
  Included:  homepage
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          false
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-12-22 23:36:03 +0100 CET
Completed:  <n/a>

Expiration:  2025-01-21 23:36:03 +0100 CET

Total items to be backed up:  17
Items backed up:              17

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots: <none included or not detectable>

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

velero backup logs

An error occurred: file not found

velero restore describe or kubectl get restore/ -n velero -o yaml

N/A

velero restore logs

N/A

Anything else you would like to add:

BackupStorageLocation

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  labels:
    app.kubernetes.io/instance: velero
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: velero
    helm.sh/chart: velero-8.1.0
  name: default
  namespace: velero
spec:
  accessMode: ReadWrite
  config:
    region: us-east-1
    s3ForcePathStyle: "true"
    s3Url: https://s3.___.infomaniak.com
  credential:
    key: cloud
    name: velero
  objectStorage:
    bucket: default
    prefix: velero_bsl
  provider: aws

Environment:

Velero version (use velero version): v1.15.0
Velero features (use velero client config get features): features: <NOT SET>
Kubernetes version (use kubectl version): v1.28.7
Kubernetes installer & version: kubeadm v1.28.15
Cloud provider or hardware configuration: Bare Metal / intel
OS (e.g. from /etc/os-release): Ubuntu 24.04.1 LTS

[kaovilai]

After scouring some issues on GitHub containing error string, seems to be specific to OpenStack Swift and some others.

https://github.com/openstack/swift/blob/5d1dbbccbee04bf7c703538dbab7c903ea146778/swift/common/middleware/s3api/s3request.py#L920-L921

I would have expected it to fallback to this or have an option to configure

which projects is this the case with aws-sdk-go-v2? That would be an example we can try mimick.

[onigoetz]

Ah yes indeed good catch, and they have an issue that's been open for a while; https://bugs.launchpad.net/swift/+bug/1810026

I'm no go expert, I tried to find if there is an option somewhere but could find no fallback.

I've seen it work on the Python aws-cli : https://github.com/aws/aws-cli/blob/2.22.22/awscli/customizations/s3/factory.py#L85-L102

2024-12-23 08:59:22,106 - MainThread - awscli.customizations.s3.factory - DEBUG - S3 CRT client optimized for system: False
2024-12-23 08:59:22,106 - MainThread - awscli.customizations.s3.factory - DEBUG - Auto resolved s3 transfer client to: classic
2024-12-23 08:59:22,106 - MainThread - awscli.customizations.s3.factory - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608

But by looking at the go client source code, I have the impression that this "classic client" feature just doesn't exist there.

[kaovilai]

One thing is for certain, we are not able to use aws-sdk-go v1 after mid 2025 since that's going EOL.

[onigoetz]

Yes I understand that starting to use old software for that is probably not a solution.
Well, In that case I'll switch my S3 provider to another.

At least this discussion will be here as information for the next person encountering the same issue