Support custom endpoint URL for AWS S3 binary caching

[SegaraRai]

This PR adds support for specifying a custom S3 endpoint URL for AWS S3 binary caches.
This will enable us to use S3 compatible storages such as Backblaze B2, Cloudflare R2, MinIO, Storj DCS, Wasabi, etc.

Demo: https://github.com/SegaraRai/vcpkg-aws-s3-custom-endpoint-demo

Implementation

UPDATE: See comments below.

Added new endpoint-url support to x-aws-config, which is currently used to specify --no-sign-request.
Use x-aws-config,endpoint-url,https://s3.ap-northeast-1.wasabisys.com;x-aws,s3://example-bucket/,readwrite to specify endpoint-url.
To use no-sign-request and endpoint-url at the same time, use x-aws-config,endpoint-url,https://s3.ap-northeast-1.wasabisys.com;x-aws-config,no-sign-request;x-aws,s3://example-bucket/,readwrite.

Alternative solutions

• Specify custom endpoint via VCPKG_AWS_S3_ENDPOINT_URL environment variable.
• Specify any extra arguments via VCPKG_AWS_CLI_EXTRA_ARGS environment variable.
• Specify any extra arguments via x-aws-config source, with aliasing no-sign-request to --no-sign-request.

Additional context

In the AWS CLI, the endpoint URL can only be specified with the --endpoint-url argument; the endpoint URL cannot be specified using environment variables or configuration files.
This is a known issue of the AWS CLI: aws/aws-cli#1270.

Related: microsoft/vcpkg#25039

[SegaraRai]

@microsoft-github-policy-service agree

[autoantwort]

Hm but it is not possible to have different endpoint urls right?

[SegaraRai]

Oops, I thought I could only use one source per provider (x-aws,foo;x-aws,bar is equivalent to x-aws,bar), but that is not correct.
Certainly this implementation does not support such a situation.
I will try to fix this by pairing prefixes and endpoints.
I have a feeling that no-signed-request has the similar problem, but I'll leave it as is for now.

[autoantwort]

I have a feeling that no-signed-request has the similar problem, but I'll leave it as is for now.

Yeah it has the same problem.

Maybe change it to x-aws,<prefix>[,<rw>,[endpoint-url=<endpoint-url>]]. This could be then extended to x-aws,<prefix>[,<rw>,[endpoint-url=<endpoint-url>][no-signed-request]]

[SegaraRai]

Updated. I had to add a profile parameter as well in order to authenticate against different endpoints.
The proposed syntax is x-aws,<prefix>[,<rw>[,endpoint-url=<endpoint-url>][,profile=<profile>]] where endpoint-url and profile are interchangeable.

[BillyONeal]

I like this direction. It needs:

1. Unit tests like the other binary cache provider parsers.
2. Documentation submitted to https://github.com/microsoft/vcpkg-docs

I prefer getting the information in here rather than employing other environment variables. We did that for the GitHub Actions provider but there we know there will only ever be one of those providers in use (since the token we are looking for there is assigned by GHA to the machine)

Sorry for the delayed review; been slammed with VS deadlines that are thankfully now past :)

[ras0219-msft]

@SegaraRai is this still being worked on?

[SegaraRai]

No. I'm afraid I've retired from a job that uses vcpkg and lost interest.

[BillyONeal]

No. I'm afraid I've retired from a job that uses vcpkg and lost interest.

I'm sorry to hear that. If that changes feel free to reopen this or open a new PR with similar changes.