@spencergilbert I got the following error when sending logs to OpenSearch v2.6.0:

vector is deployed in agent mode following the method in https://github.com/vectordotdev/helm-charts/tree/develop/charts/vector

helm repo add vector https://helm.vector.dev
helm repo update
helm install -n vector -f values.yaml vector vector/vector

error:

root@benjamin-lab:~/vector# kubectl -n vector logs ks-vector-cp9zs -f
2023-05-10T06:31:47.680831Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-05-10T06:31:47.685117Z  INFO vector::config::watcher: Creating configuration file watcher.
2023-05-10T06:31:47.686085Z  INFO vector::config::watcher: Watching configuration files.
2023-05-10T06:31:47.686390Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-05-10T06:31:47.693439Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="benjamin-lab"
2023-05-10T06:31:47.712322Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2023-05-10T06:31:47.773932Z  WARN http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T06:31:47.777630Z  WARN vector::sinks::elasticsearch::common: Failed to determine Elasticsearch version from `/_cluster/state/version`. Please fix the reported error or set an API version explicitly via `api_version`. assumed_version=8 error=Failed to get Elasticsearch API version: Failed to make HTTP(S) request: connection closed before message completed
2023-05-10T06:31:47.811141Z  INFO vector::topology::running: Running healthchecks.
2023-05-10T06:31:47.811762Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
2023-05-10T06:31:47.817480Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-05-10T06:31:47.818404Z  WARN http: vector::internal_events::http_client: Internal log [HTTP error.] is being rate limited.
2023-05-10T06:31:47.818691Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: connection closed before message completed component_kind="sink" component_type="elasticsearch" component_id=opensearch component_name=opensearch
2023-05-10T06:31:47.818215Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
2023-05-10T06:32:49.319390Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-monitoring-system_notification-manager-operator-85d67fdc46-pc8w6_4f7abd6f-004d-4e09-9fd1-7c12f3dc9164/kube-rbac-proxy/1.log file_position=498

config:

customConfig:
  data_dir: /vector-data-dir
  # Vector's API for introspection
  api:
    enabled: true
    address: "0.0.0.0:8686"
    playground: true

  # Read Kubernetes logs from files
  sources:
    kube_logs:
      type: "kubernetes_logs"
      pod_annotation_fields:
        container_id: ".kubernetes.docker_id"
        container_image_id: ""
        pod_annotations: ""
        pod_ip: ""
        pod_ips: ""
        pod_labels: ""
        pod_owner: ""
        pod_uid: ""
        pod_namespace: ".kubernetes.namespace_name"
        pod_node_name: ".kubernetes.node_name"
      namespace_annotation_fields:
        namespace_labels: ""
      node_annotation_fields:
        node_labels: ""
  # Transform Kube logs and remove unused fields
  transforms:
    kube_logs_remapped:
      type: "remap"
      inputs:
        - kube_logs
      source: |-
        .log = .message
        .time = .timestamp
        del(.file)
        del(.message)
        del(.timestamp_end)
        del(.stream)
        del(.source_type)
  # Forward logs to stdout 
  sinks:
#    stdout:
#      type: console
#      inputs:
#        - kube_logs_remapped
#      target: stdout
#      encoding:
#        codec: json
    opensearch:
      type: elasticsearch
      inputs:
        - kube_logs_remapped
      auth:
        strategy: basic
        user: admin
        password: admin  
      api_version: auto
      compression: none
      # This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
      # doc_type: _doc
      endpoints:
        - http://opensearch-cluster-data.<namespace>.svc:9200
      mode: bulk
      bulk: 
        index: "k8s-logging-%Y.%m.%d"

After changing the api_version from auto to v8 or v7, still got below errors:

root@benjamin-lab:~/vector# kubectl -n vector logs ks-vector-qwg2m -f
2023-05-10T07:04:20.840957Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-05-10T07:04:20.844872Z  INFO vector::config::watcher: Creating configuration file watcher.
2023-05-10T07:04:20.846328Z  INFO vector::config::watcher: Watching configuration files.
2023-05-10T07:04:20.846574Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-05-10T07:04:20.856001Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="benjamin-lab"
2023-05-10T07:04:20.873952Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2023-05-10T07:04:20.926348Z  INFO vector::topology::running: Running healthchecks.
2023-05-10T07:04:20.927123Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
2023-05-10T07:04:20.937496Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
2023-05-10T07:04:20.938250Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-05-10T07:04:20.956226Z  WARN http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T07:04:20.956766Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: connection closed before message completed component_kind="sink" component_type="elasticsearch" component_id=opensearch component_name=opensearch
2023-05-10T07:05:22.440838Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-system_minio-7879c5dd65-jgv8s_21b492b2-fa71-4a8c-9a10-d16f92bfae10/minio/1.log file_position=1023
...
2023-05-10T07:05:22.497413Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-logging-system_opensearch-logging-curator-elasticsearch-curator-28060860-hrtl7_aaa43c9a-c5cc-4898-aa42-936cf41d29dd/elasticsearch-curator/0.log file_position=16079
2023-05-10T07:05:22.883870Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T07:05:22.884067Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: connection closed before message completed internal_log_rate_limit=true
2023-05-10T07:05:23.185165Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}:http: vector::internal_events::http_client: Internal log [HTTP error.] is being rate limited.
2023-05-10T07:05:23.185424Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}: vector::sinks::util::retries: Internal log [Retrying after error.] is being rate limited.
2023-05-10T07:05:24.187024Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=http://opensearch-cluster-data.kubesphere-logging-system.svc:9200

Document and integration test OpenSearch support #16761

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions