Jaeger v2 via opentelemetry-operator connecting to the VictoriaLogs ElasticSearch backend

[djryanj]

When deploying Jaeger v2 via the opentelemetry-operator (e.g., using https://github.com/jaegertracing/jaeger/blob/main/cmd/jaeger/config-elasticsearch.yaml) and attempting to use VictoriaLogs' ElasticSearch backend (e.g., https://docs.victoriametrics.com/victorialogs/data-ingestion/opentelemetry/#elasticsearch), the Jaeger pod fails with:

Error: failed to start extensions: failed to initialize storage 'victoria_logs': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available
2025/02/19 16:04:43 failed to start extensions: failed to initialize storage 'victoria_logs': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available

However, we see in the VictoriaLogs pod:

{"ts":"2025-02-19T16:09:47.640Z","level":"warn","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:419","msg":"remoteAddr: \"10.244.3.43:48262\"; requestURI: /insert/elasticsearch; unsupported path requested: \"/insert/elasticsearch\""}

The path appears to be correct based on the VictoriaLogs documentation, (and given that I have provided that path correctly to other compatible agents, like Vector). In addition, network connectivity is not blocked (there are no NetworkPolicies in the cluster at all) since we're seeing errors in the VL pods from the Jaeger pods without issue.

Note that this appears similar to https://github.com/orgs/jaegertracing/discussions/6024 and https://github.com/orgs/jaegertracing/discussions/6681 but neither of those have resolution; #6681 specifically calls out setting es.log-level: "debug" but I am unsure how to do that with the opentelemetry-operator as documentation merely mentions setting log levels but does not provide any way to actually do that (I have tried setting --log-level=debug in the command line flags to the jaeger pods, I have tried setting it via the config passed to opentelemetry-operator, but neither of those work with jaeger complaining about unsupported flags).

Any assistance here would be appreciated.

[dosubot[bot]]

Hey @djryanj! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

I found a similar unsolved discussion regarding Jaeger deployment issues with the Elasticsearch backend ^[1]. Unfortunately, it doesn't provide a resolution, but it might be worth reviewing for any additional context or troubleshooting steps.

To address your issue with configuring Jaeger v2 via the OpenTelemetry operator to connect to the VictoriaLogs Elasticsearch backend and set the log level to "debug," you can use the following configuration:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
    logs:
      level: debug

extensions:
  healthcheckv2:
    use_v2: true
    http:

  jaeger_query:
    storage:
      traces: some_storage
      traces_archive: another_storage
    ui:
      config_file: ./cmd/jaeger/config-ui.json

  jaeger_storage:
    backends:
      some_storage:
        elasticsearch:
          server_urls:
            - http://<victoria-logs-url>:9200
          indices:
            index_prefix: "jaeger-main"
            spans:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            services:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            dependencies:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
            sampling:
              date_layout: "2006-01-02"
              rollover_frequency: "day"
              shards: 5
              replicas: 1
      another_storage:
        elasticsearch:
          server_urls:
            - http://<victoria-logs-url>:9200
          indices:
            index_prefix: "jaeger-archive"

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage

Replace <victoria-logs-url> with the actual URL of your VictoriaLogs Elasticsearch backend. The log level is set to "debug" in this configuration ^[2].

Ensure that the server_urls in your configuration match the expected VictoriaLogs endpoint. If the path /insert/elasticsearch is unsupported, verify the correct endpoint from the VictoriaLogs documentation and adjust accordingly. If the issue persists, consider checking for any specific configuration options in the OpenTelemetry operator that might allow setting custom HTTP paths or log levels.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[djryanj]

@dosu unfortunately there was no extra information provided by increasing the log level.

[dosubot[bot]]

If increasing the log level didn't provide additional information, it might be helpful to verify a few other aspects of your setup:

1. Endpoint Configuration: Double-check that the server_urls in your Jaeger configuration are correctly pointing to the VictoriaLogs endpoint. Ensure that the path and port are accurate according to the VictoriaLogs documentation.

2. Network Connectivity: Since the VictoriaLogs pod is logging requests, it seems like there is connectivity. However, you might want to use tools like curl or telnet from within the Jaeger pod to ensure it can reach the VictoriaLogs endpoint directly.

3. VictoriaLogs Configuration: Verify that VictoriaLogs is configured to accept requests on the specified path. The error message from VictoriaLogs suggests that the path /insert/elasticsearch might not be supported. Ensure that the path aligns with what VictoriaLogs expects for Elasticsearch compatibility.

4. Jaeger Configuration: Ensure that the Jaeger configuration aligns with the expected format for connecting to an Elasticsearch-compatible backend. This includes checking any specific settings related to health checks or timeouts.

If these steps don't resolve the issue, consider reviewing any recent changes in the VictoriaLogs or Jaeger documentation that might affect compatibility or configuration.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[djryanj]

@dosu Does Jaeger v2 support otlphttp for remote storage?

[dosubot[bot]]

Jaeger v2 does not explicitly list OTLP over HTTP (otlphttp) as a supported remote storage protocol in the available documentation. The supported storage backends for Jaeger v2 include Cassandra, Elasticsearch/OpenSearch, Kafka (only as a buffer), Badger, and Memory ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[djryanj]

I opened an issue on the Jaeger repo here and it appears that VictoriaLogs doesn't respond to a path without a trailing slash.

Is there a way in the config to add a "path" that is independent of the URL that Jaeger will add to all requests, even if that path is just a single slash ("/")?

[yurishkuro]

Does Jaeger v2 support otlphttp for remote storage?

@djryanj that's just an otlphttp exporter in the collector pipeline, doesn't need to be part of "remote storage" ecosystem.

[djryanj]

@yurishkuro yes, I think that's my ignorance regarding all the different components in Jaeger and what each is for.

[djryanj]

It appears that this was a bug in VictoriaLogs that has been fixed. Closing.