Skip to content

WARN kube_runtime::watcher: watch list error with 403 #473

@loskutovanl

Description

@loskutovanl

Authorization errors when deploying vector via helm in minikube and in k3s.

pod error logs:

2025-04-09T13:54:58.916837Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "nodes \"nlloskutova-cmp-ns483\" is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"nodes\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 }))
2025-04-09T13:55:04.773524Z  WARN kube_runtime::watcher: watch list error with 403: Api(ErrorResponse { status: "Failure", message: "namespaces is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 })
2025-04-09T13:55:04.773538Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "namespaces is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 }))
2025-04-09T13:55:15.287548Z  WARN kube_runtime::watcher: watch list error with 403: Api(ErrorResponse { status: "Failure", message: "pods is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"pods\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 })
2025-04-09T13:55:15.287563Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "pods is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"pods\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 }))
2025-04-09T13:55:34.015193Z  WARN kube_runtime::watcher: watch list error with 403: Api(ErrorResponse { status: "Failure", message: "nodes \"nlloskutova-cmp-ns483\" is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"nodes\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 })
2025-04-09T13:55:34.015209Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "nodes \"nlloskutova-cmp-ns483\" is forbidden: User \"system:serviceaccount:vector:vector\" cannot list resource \"nodes\" in API group \"\" at the cluster scope", reason: "Forbidden", code: 403 }))

my values.yaml:

service:
  enabled: false

customConfig:
  data_dir: "/var/lib/vector"

  api:
    enabled: false

  sources:
    kube_logs:
      type: "kubernetes_logs"
      self_node_name: "nlloskutova-cmp-ns483" or "minikube" // node name here hardcoded, otherwise failure on empty self_node_name

  transforms:
    kube_parser:
      inputs:
        - "kube_logs"
      type: "remap"
      source: ". = parse_klog!(.message)"

    kube_sampler:
      inputs:
        - "kube_parser"
      type: "sample"
      rate: 2 # only keep 50% (1/`rate`)

  sinks:
    clickhouse_sink:
      type: clickhouse
      inputs:
        - kube_sampler
      compression: gzip
      database: default
      endpoint: http://localhost:9090
      format: json_each_row
      table: mytable

the way I deploy:

helm repo add vector https://helm.vector.dev
helm repo update
helm install vector vector/vector \
  --namespace vector \
  --create-namespace \
  --values values.yaml

kube verion:

// virtualbox driver
nlloskutova@nlloskutova-Cmp-NS483:~$ minikube version
minikube version: v1.35.0
commit: dd5d320e41b5451cdf3c01891bc4e13d189586ed-dirty

system:

nlloskutova@nlloskutova-Cmp-NS483:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions