mTLS Client Certificates Not Sent to OAuth Token Endpoint: Registry `keypair` client certificates not applied to OAuth token endpoint requests

[constantins2001]

When configuring mTLS (mutual TLS) client certificates via buildkitd.toml using the keypair option, BuildKit successfully uses these certificates for registry API calls (/v2/*) but fails to send them when requesting OAuth tokens from the /service/token endpoint.

This causes builds to fail with tls: certificate required when the registry (e.g., Harbor) requires mTLS authentication.

Environment

• BuildKit version: buildx-stable-1 (moby/buildkit:buildx-stable-1)
• Docker Buildx version: v0.19+
• Platform: darwin/arm64, linux/amd64
• Registry: Harbor with mTLS enabled

Configuration

# /etc/buildkit/buildkitd.toml

[registry."harbor.example.com"]
  ca = ["/etc/buildkit/certs/ca.pem"]
  [[registry."harbor.example.com".keypair]]
    cert = "/etc/buildkit/certs/harbor.example.com/client.cert"
    key = "/etc/buildkit/certs/harbor.example.com/client.key"

Steps to Reproduce

1. Set up a Harbor registry with a Reverse Proxy requiring mTLS enabled (requiring client certificates)
2. Configure BuildKit with the above buildkitd.toml including keypair for client certs
3. Create a buildx builder:

   docker buildx create --name test-builder \
     --driver docker-container \
     --buildkitd-config /path/to/buildkitd.toml \
     --bootstrap

4. Verify certificates are copied into the container:

   docker exec buildx_buildkit_test-builder0 ls -la /etc/buildkit/certs/
   docker exec buildx_buildkit_test-builder0 cat /etc/buildkit/buildkitd.toml

5. Attempt to build an image:

   echo 'FROM harbor.example.com/library/nginx:latest' | \
     docker buildx build --builder test-builder -t test:latest -

Expected Behavior

BuildKit should use the configured keypair client certificates for all HTTPS requests to the registry, including:

• Registry API calls (/v2/*)
• OAuth token requests (/service/token)

Actual Behavior

BuildKit fails with:

ERROR: failed to authorize: failed to fetch oauth token: Post "https://harbor.example.com/service/token": remote error: tls: certificate required

The error tls: certificate required indicates the server is requesting a client certificate during TLS handshake, but BuildKit is not providing one.

Verification That Certificates Work

Testing mTLS manually from inside the BuildKit container succeeds:

# This works - proves the certs are valid
docker exec buildx_buildkit_test-builder0 sh -c '
  echo | openssl s_client -connect harbor.example.com:443 \
    -cert /etc/buildkit/certs/harbor.example.com/client.cert \
    -key /etc/buildkit/certs/harbor.example.com/client.key 2>&1 |
  grep "Verify return"
'
# Output: Verify return code: 0 (ok)

Root Cause Analysis

Looking at the code:

1. util/resolver/resolver.go: The loadTLSConfig function correctly loads keypairs and creates a tls.Config with client certificates.

2. util/resolver/authorizer.go: The dockerAuthorizer and authHandler structs receive an *http.Client as a dependency for making token requests.

3. The Bug: The HTTP client passed to the authorizer may not have the same TLS configuration (including client certificates) that is configured for the registry. When fetchToken calls auth.FetchToken(ctx, ah.client, ...), the client doesn't send the mTLS certificate.

Comparison: Registry Without Token Auth Works

A pull-through cache registry (dockercache) configured with mTLS but without OAuth token authentication works correctly:

# This WORKS - no token auth, just direct mTLS
echo 'FROM dockercache.example.com/library/alpine:latest' | \
  docker buildx build --builder test-builder -t test:latest -

This confirms the keypair configuration works for direct registry API calls, but fails specifically for the token endpoint.

Workarounds

1. Disable mTLS on token endpoint doesn't work: Configure the registry's reverse proxy to not require client certificates for /service/token sadly doesn't work as mTLS happens before HTTP Headers are sent
2. Pre-pull images: Use docker pull (which works) before docker buildx build
3. Use registries without token auth: Pull-through caches that don't use OAuth work correctly

Suggested Fix

Ensure the HTTP client used by authorizer.go for token requests has the same TLS configuration (including client certificates from keypair) as the registry transport.

The fix should propagate the tls.Config with client certificates to:

• auth.FetchToken()
• auth.FetchTokenWithOAuth()