cloudflare provider in k3d does not work - invalid header field value for "Authorization"

[monotek]

What happened:

Pod is crashlooping with:

k logs -f -l app.kubernetes.io/instance=external-dns
{"level":"info","msg":"Instantiating new Kubernetes client","time":"2024-12-22T13:46:36Z"}
{"level":"debug","msg":"apiServerURL: ","time":"2024-12-22T13:46:36Z"}
{"level":"debug","msg":"kubeConfig: ","time":"2024-12-22T13:46:36Z"}
{"level":"info","msg":"Using inCluster-config based on serviceaccount-token","time":"2024-12-22T13:46:36Z"}
{"level":"info","msg":"Created Kubernetes client https://10.43.0.1:443","time":"2024-12-22T13:46:36Z"}
{"level":"debug","msg":"apiServerURL: ","time":"2024-12-22T13:46:36Z"}
{"level":"debug","msg":"kubeConfig: ","time":"2024-12-22T13:46:36Z"}
{"level":"info","msg":"Using inCluster-config based on serviceaccount-token","time":"2024-12-22T13:46:36Z"}
{"level":"info","msg":"Created Dynamic Kubernetes client https://10.43.0.1:443","time":"2024-12-22T13:46:36Z"}
{"level":"debug","msg":"no zoneIDFilter configured, looking at all zones","time":"2024-12-22T13:46:36Z"}
{"level":"fatal","msg":"Failed to do run once: HTTP request failed: Get \"https://api.cloudflare.com/client/v4/zones?per_page=50\": net/http: invalid header field value for \"Authorization\"","time":"2024-12-22T13:46:43Z"}

I've checked the secret multiple times.
If is use an ubuntu container in the external-dns namespace i can query the url successfully with curl, using the secret values as bearer token:

 curl -X GET "https://api.cloudflare.com/client/v4/zones?per_page=50" -H "Authorization: Bearer foo-bar" -H "Content-Type:application/json" | jq

{
  "result": [
    {
      "id": "foo-bar",
      "name": "foo-bar.org",
      "status": "active",
      "paused": false,
      "type": "full",
      "development_mode": 0,
      "name_servers": [
        "chip.ns.cloudflare.com",
        "tia.ns.cloudflare.com"
      ],
      "original_name_servers": [
        "helium.ns.hetzner.de",
        "hydrogen.ns.hetzner.com",
        "oxygen.ns.hetzner.com"
      ],
      "original_registrar": "vautron rechenzentrum ag (id: 1443)",
      "original_dnshost": null,
      "modified_on": "2024-12-21T17:34:04.898640Z",
      "created_on": "2024-12-21T17:16:46.080077Z",
      "activated_on": "2024-12-21T17:34:04.898640Z",
      "meta": {
        "step": 2,
        "custom_certificate_quota": 0,
        "page_rule_quota": 3,
        "phishing_detected": false
      },
      "owner": {
        "id": null,
        "type": "user",
        "email": null
      },
      "account": {
        "id": "foo-bar",
        "name": "[email protected]'s Account"
      },
      "tenant": {
        "id": null,
        "name": null
      },
      "tenant_unit": {
        "id": null
      },
      "permissions": [
        "#dns_records:edit",
        "#dns_records:read",
        "#zone:read"
      ],
      "plan": {
        "id": "0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
        "name": "Free Website",
        "price": 0,
        "currency": "USD",
        "frequency": "",
        "is_subscribed": false,
        "can_subscribe": false,
        "legacy_id": "free",
        "legacy_discount": false,
        "externally_managed": false
      }
    },
    {
      "id": "foo-bar",
      "name": "foo-bar.net",
      "status": "active",
      "paused": false,
      "type": "full",
      "development_mode": 0,
      "name_servers": [
        "chip.ns.cloudflare.com",
        "tia.ns.cloudflare.com"
      ],
      "original_name_servers": [
        "helium.ns.hetzner.de",
        "hydrogen.ns.hetzner.com",
        "oxygen.ns.hetzner.com"
      ],
      "original_registrar": "hetzner online gmbh (id: 828)",
      "original_dnshost": null,
      "modified_on": "2024-12-21T17:41:35.762022Z",
      "created_on": "2024-12-21T17:36:14.477489Z",
      "activated_on": "2024-12-21T17:41:35.762022Z",
      "meta": {
        "step": 2,
        "custom_certificate_quota": 0,
        "page_rule_quota": 3,
        "phishing_detected": false
      },
      "owner": {
        "id": null,
        "type": "user",
        "email": null
      },
      "account": {
        "id": "foo-bar",
        "name": "[email protected]'s Account"
      },
      "tenant": {
        "id": null,
        "name": null
      },
      "tenant_unit": {
        "id": null
      },
      "permissions": [
        "#dns_records:edit",
        "#dns_records:read",
        "#zone:read"
      ],
      "plan": {
        "id": "0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
        "name": "Free Website",
        "price": 0,
        "currency": "USD",
        "frequency": "",
        "is_subscribed": false,
        "can_subscribe": false,
        "legacy_id": "free",
        "legacy_discount": false,
        "externally_managed": false
      }
    }
  ],
  "result_info": {
    "page": 1,
    "per_page": 50,
    "total_pages": 1,
    "count": 2,
    "total_count": 2
  },
  "success": true,
  "errors": [],
  "messages": []
}

What you expected to happen:

DNS updated, even if i would likely need to find a workaround for my loadbalancer service having a private external ip:

 k get svc -n kube-system traefik 
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
traefik   LoadBalancer   10.43.84.213   172.18.0.2    80:30174/TCP,443:30715/TCP   3d19h

How to reproduce it (as minimally and precisely as possible):

helm install with the following values:

env:
  - name: CF_API_EMAIL
    valueFrom:
      secretKeyRef:
        key: email
        name: external-dns
  - name: CF_API_TOKEN
    valueFrom:
      secretKeyRef:
        key: apiToken
        name: external-dns
interval: 2m
logFormat: json
logLevel: debug
provider:
  name: cloudflare
  webhook:
    resources:
      limits:
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 64Mi
sources:
  - crd
  - ingress
  - service
  - traefik-proxy
txtOwnerId: foo-bar

My secret looks like:

apiVersion: v1
data:
  apiToken: foobar
  email: foobar
kind: Secret
metadata:
  name: external-dns
  namespace: external-dns
type: Opaque

Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): helm chart 1.15.0
• DNS provider: cloudflare
• Others: k3d running on a public cluster with haproxy running on port 80 and 443 which routes requests to port 8080 and 8443 on localhost, which are the k3s ingress ports.