Cannot install in GKE Autopilot

[anukul]

Error: INSTALLATION FAILED: 1 error occurred:
	* admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-no-write-mode-hostpath]":["hostPath volume data in container vector is accessed in write mode; disallowed in Autopilot.","hostPath volume var-lib used in container vector uses path /var/lib which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume procfs used in container vector uses path /proc which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume sysfs used in container vector uses path /sys which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}

[anukul]

Related - vectordotdev/vector#16871

[curusarn]

Hi @anukul,

Thank you for reaching out!

I'm passing this on to the team to get this resolved for you. I'll make sure to keep you updated here.

Thanks again for rasing this.

[scotchka]

Encountered the same error with GKE Autopilot:

✗ helm install betterstack-logs betterstack-logs/betterstack-logs -f values.yaml \
  --set metrics-server.enabled=false
W0722 15:44:02.000180   91146 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated DaemonSet default/betterstack-logs-vector: defaulted unspecified 'cpu' resource for containers [vector] (see http://g.co/gke/autopilot-defaults).
Error: INSTALLATION FAILED: 1 error occurred:
	* admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-no-write-mode-hostpath]":["hostPath volume data in container vector is accessed in write mode; disallowed in Autopilot.","hostPath volume var-lib used in container vector uses path /var/lib which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume procfs used in container vector uses path /proc which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume sysfs used in container vector uses path /sys which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}

[PetrHeinz]

Hello @anukul and @scotchka,

Thanks again for reporting this and for your patience 🙏

We've managed to reproduce the issue on a testing GKE cluster and troubleshooted all problems with the current Helm chart setting. In our tests, this is a values.yaml configuration that seems to work without any issues, and sends both logs and metrics from GKE Autopilot cluster to Better Stack:

# config for GKE Autopilot cluster
vector:
  customConfig:
    sinks:
      better_stack_http_sink:
        auth:
          strategy: "bearer"
          token: $SOURCE_TOKEN
      better_stack_http_metrics_sink:
        auth:
          strategy: "bearer"
          token: $SOURCE_TOKEN
    sources:
      better_stack_kubernetes_metrics_nodes:
        endpoint: "https://metrics-server.kube-system/apis/metrics.k8s.io/v1beta1/nodes"
        tls:
          verify_certificate: false
          verify_hostname: false
      better_stack_kubernetes_metrics_pods:
        endpoint: "https://metrics-server.kube-system/apis/metrics.k8s.io/v1beta1/pods"
        tls:
          verify_certificate: false
          verify_hostname: false

  # Disable persistence using a volume mount
  persistence:
    hostPath:
      enabled: false

  # Override default volumes and mounts to ensure only /var/log is mounted
  defaultVolumes:
    - name: var-log
      hostPath:
        path: /var/log
        type: DirectoryOrCreate
  defaultVolumeMounts:
    - name: var-log
      mountPath: /var/log
      readOnly: true

# Metrics server is already installed
metrics-server:
  enabled: false

It ensure that only the allowed volumes are mounted, and uses the already installed metrics server from GKE Autopilot cluster which it accesses without verification of TLS due to use of self-signed certificates.

I've also updated the Better Stack Kubernetes logging docs to include this variant of configuration.

Hope this helps to get all the telemetry data from your cluster safely to Better Stack 🚀

Feel free to reopen the issue or create another one if you encounter any problems connecting your GKE cluster 🙏