Configuring Vector in k8s deployment to source pod logs

[1000p]

Question

I am trying to set up Vector for scraping the logs out of my k8s services.
Deploying vector trough Helm charts.

The difference is that I am using 0.28.1-alpine tag.

The pod is running okay. No apparent logs are present that indicate an issue (even with debug flag on).

I have tried to tap vector tap --outputs-of "db_logs" --url http://0.0.0.0:9001/graphql & also extended the configuration to be able to tap to dropped output. However, nothing appears to be logged. I am pretty much stuck at this point.

Here is the deployment resource:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "21"
    meta.helm.sh/release-name: platform
    meta.helm.sh/release-namespace: platform
  creationTimestamp: "2025-08-18T17:56:44Z"
  generation: 21
  labels:
    app.kubernetes.io/instance: platform
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: supabase
    helm.sh/chart: supabase-0.1.3
    vector.dev/exclude: "true"
  name: platform-supabase-vector
  namespace: platform
  resourceVersion: "16172"
  selfLink: /apis/apps/v1/namespaces/platform/deployments/platform-supabase-vector
  uid: dbe66475-d974-46b8-ab54-8c2f110893b9
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: platform
      app.kubernetes.io/name: supabase-vector
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/config: 92b08016ceeaf52c85358809c1143bb6e0b478bb5b242f447590603e93fb18fb
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: platform
        app.kubernetes.io/name: supabase-vector
    spec:
      containers:
      - args:
        - "--config-yaml"
        - /etc/vector/vector.yml
        env:
        - name: enabled
          value: "true"
        - name: graphql
          value: "true"
        - name: VECTOR_SELF_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: LOGFLARE_PRIVATE_ACCESS_TOKEN
          valueFrom:
            secretKeyRef:
              key: apiPrivateKey
              name: platform-supabase-analytics
        - name: LOGFLARE_PUBLIC_ACCESS_TOKEN
          valueFrom:
            secretKeyRef:
              key: apiPublicKey
              name: platform-supabase-analytics
        image: timberio/vector:0.28.1-alpine
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 9001
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: supabase-vector
        ports:
        - containerPort: 9001
          protocol: TCP
        resources: {}
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/log/pods
          name: pod-logs
        - mountPath: /etc/vector/vector.yml
          name: vector-config
          subPath: vector.yml
        - mountPath: /etc/vector/secret.sh
          name: vector-config
          subPath: secret.sh
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: platform-supabase-vector
      serviceAccountName: platform-supabase-vector
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/log/pods
          type: ""
        name: pod-logs
      - configMap:
          defaultMode: 511
          name: platform-supabase-vector-config
        name: vector-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2025-08-19T16:38:11Z"
    lastUpdateTime: "2025-08-19T16:38:11Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2025-08-18T17:56:44Z"
    lastUpdateTime: "2025-08-20T04:53:32Z"
    message: >-
      ReplicaSet "platform-supabase-vector-74c96644cd" has successfully
      progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 21
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Vector Config

secret:
  credentials:
    type: exec
    command:
      - /etc/vector/secret.sh

api:
  enabled: true
  address: 0.0.0.0:9001

sources:
  kubernetes_host:
    type: kubernetes_logs
    extra_label_selector: app.kubernetes.io/instance=platform,app.kubernetes.io/name!=supabase-vector

transforms:
  project_logs:
    type: remap
    inputs:
      - kubernetes_host
    source: |-
      .project = "default"
      .event_message = del(.message)
      .appname = del(.kubernetes.container_name)
      del(.file)
      del(.kubernetes)
      del(.source_type)
      del(.stream)
  router:
    type: route
    inputs:
      - project_logs
    route:
      kong: '.appname == "supabase-kong"'
      auth: '.appname == "supabase-auth"'
      rest: '.appname == "supabase-rest"'
      realtime: '.appname == "supabase-realtime"'
      storage: '.appname == "supabase-storage"'
      functions: '.appname == "supabase-functions"'
      db: '.appname == "supabase-db"'
  # Ignores non nginx errors since they are related with kong booting up
  kong_logs:
    type: remap
    inputs:
      - router.kong
    source: |-
      req, err = parse_nginx_log(.event_message, "combined")
      if err == null {
          .timestamp = req.timestamp
          .metadata.request.headers.referer = req.referer
          .metadata.request.headers.user_agent = req.agent
          .metadata.request.headers.cf_connecting_ip = req.client
          .metadata.request.method = req.method
          .metadata.request.path = req.path
          .metadata.request.protocol = req.protocol
          .metadata.response.status_code = req.status
      }
      if err != null {
        abort
      }
  # Ignores non nginx errors since they are related with kong booting up
  kong_err:
    type: remap
    inputs:
      - router.kong
    source: |-
      .metadata.request.method = "GET"
      .metadata.response.status_code = 200
      parsed, err = parse_nginx_log(.event_message, "error")
      if err == null {
          .timestamp = parsed.timestamp
          .severity = parsed.severity
          .metadata.request.host = parsed.host
          .metadata.request.headers.cf_connecting_ip = parsed.client
          url, err = split(parsed.request, " ")
          if err == null {
              .metadata.request.method = url[0]
              .metadata.request.path = url[1]
              .metadata.request.protocol = url[2]
          }
      }
      if err != null {
        abort
      }
  # Gotrue logs are structured json strings which frontend parses directly. But we keep metadata for consistency.
  auth_logs:
    type: remap
    inputs:
      - router.auth
    source: |-
      parsed, err = parse_json(.event_message)
      if err == null {
          .metadata.timestamp = parsed.time
          .metadata = merge!(.metadata, parsed)
      }
  # PostgREST logs are structured so we separate timestamp from message using regex
  rest_logs:
    type: remap
    inputs:
      - router.rest
    source: |-
      parsed, err = parse_regex(.event_message, r'^(?P<time>.*): (?P<msg>.*)$')
      if err == null {
          .event_message = parsed.msg
          .timestamp = parse_timestamp!(parsed.time, format: "%e/%b/%Y %R %:z")
          .metadata.host = .project
      }
  # Realtime logs are structured so we parse the severity level using regex (ignore time because it has no date)
  realtime_logs:
    type: remap
    inputs:
      - router.realtime
    source: |-
      .metadata.project = del(.project)
      .metadata.external_id = .metadata.project
      parsed, err = parse_regex(.event_message, r'^(?P<time>\d+:\d+:\d+\.\d+) \[(?P<level>\w+)\] (?P<msg>.*)$')
      if err == null {
          .event_message = parsed.msg
          .metadata.level = parsed.level
      }
  # Storage logs may contain json objects so we parse them for completeness
  storage_logs:
    type: remap
    inputs:
      - router.storage
    source: |-
      .metadata.project = del(.project)
      .metadata.tenantId = .metadata.project
      parsed, err = parse_json(.event_message)
      if err == null {
          .event_message = parsed.msg
          .metadata.level = parsed.level
          .metadata.timestamp = parsed.time
          .metadata.context[0].host = parsed.hostname
          .metadata.context[0].pid = parsed.pid
      }
  # Postgres logs some messages to stderr which we map to warning severity level
  db_logs:
    type: remap
    drop_on_abort: true
    reroute_dropped: true
    inputs:
      - router.db
    source: |-
      .metadata.host = "db-default"
      .metadata.parsed.timestamp = .timestamp
      
      parsed, err = parse_regex(.event_message, r'.*(?P<level>INFO|NOTICE|WARNING|ERROR|LOG|FATAL|PANIC?):.*', numeric_groups: true)

      if err != null || parsed == null {
        .metadata.parsed.error_severity = "info"
      }
      if parsed != null {
      .metadata.parsed.error_severity = parsed.level
      }
      if .metadata.parsed.error_severity == "info" {
          .metadata.parsed.error_severity = "log"
      }
      .metadata.parsed.error_severity = upcase!(.metadata.parsed.error_severity)
sinks:
  logflare_auth:
    type: 'http'
    inputs:
      - auth_logs
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=gotrue.logs.prod'
  logflare_realtime:
    type: 'http'
    inputs:
      - realtime_logs
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=realtime.logs.prod'
  logflare_rest:
    type: 'http'
    inputs:
      - rest_logs
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=postgREST.logs.prod'
  logflare_db:
    type: 'http'
    inputs:
      - db_logs
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    # We must route the sink through kong because ingesting logs before logflare is fully initialised will
    # lead to broken queries from studio. This works by the assumption that containers are started in the
    # following order: vector > db > logflare > kong
    uri: 'http://platform-supabase-kong:8000/analytics/v1/api/logs?source_name=postgres.logs'
  logflare_functions:
    type: 'http'
    inputs:
      - router.functions
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=deno-relay-logs'
  logflare_storage:
    type: 'http'
    inputs:
      - storage_logs
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=storage.logs.prod.2'
  logflare_kong:
    type: 'http'
    inputs:
      - kong_logs
      - kong_err
    encoding:
      codec: 'json'
    method: 'post'
    request:
      retry_max_duration_secs: 10
      headers:
        x-api-key: ${LOGFLARE_PUBLIC_ACCESS_TOKEN}
    uri: 'http://platform-supabase-analytics:4000/api/logs?source_name=cloudflare.logs.prod'

Vector Logs

2025-08-19T16:55:11.803028Z  INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2025-08-19T16:55:11.803329Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2025-08-19T16:55:11.803387Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yml"]
2025-08-19T16:55:11.821338Z  WARN vector::config::loading: Transform "db_logs.dropped" has no consumers
2025-08-19T16:55:11.821354Z  WARN vector::config::loading: Transform "router._unmatched" has no consumers
2025-08-19T16:55:11.821667Z  INFO source{component_kind="source" component_id=kubernetes_host component_type=kubernetes_logs component_name=kubernetes_host}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="lima-rancher-desktop"
2025-08-19T16:55:11.827729Z  INFO source{component_kind="source" component_id=kubernetes_host component_type=kubernetes_logs component_name=kubernetes_host}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2025-08-19T16:55:11.943483Z  INFO vector::topology::running: Running healthchecks.
2025-08-19T16:55:11.944040Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944165Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944253Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944335Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944396Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944474Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.944563Z  INFO vector::topology::builder: Healthcheck passed.
2025-08-19T16:55:11.945683Z  INFO vector: Vector has started. debug="false" version="0.28.1" arch="aarch64" revision="ff15924 2023-03-06"
2025-08-19T16:55:11.949784Z  INFO source{component_kind="source" component_id=kubernetes_host component_type=kubernetes_logs component_name=kubernetes_host}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2025-08-19T16:55:11.953489Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:9001 playground=http://0.0.0.0:9001/playground