non flat fields not displayed in log/table output

[curuvija]

What happened:

Hi, we use opentelemetry to ingest data into AWS opensearch domain and add some additional fields to each log using processors. In opensearch we see clearly those fields ingested. This is one example log bellow in json format taken from opensearch:

{ "_index": "logs-test-cluster-platform-experiment", "_id": "V_x6pI8BYBiqWICg75zT", "_version": 1, "_score": null, "_source": { "traceId": "", "spanId": "", "severityText": "", "flags": 0, "time": "2024-05-23T08:04:30.512773595Z", "severityNumber": 0, "droppedAttributesCount": 0, "serviceName": null, "body": "{\"level\":\"info\",\"ts\":1716451470.5126758,\"msg\":\"TracesExporter\",\"kind\":\"exporter\",\"data_type\":\"traces\",\"name\":\"logging\",\"resource spans\":1,\"spans\":4}", "observedTime": "2024-05-23T08:04:30.534179549Z", "schemaUrl": "", "log.attributes.time": "2024-05-23T08:04:30.512773595Z", "resource.attributes.k8s@namespace@name": "tracing", "resource.attributes.k8s@deployment@name": "otel-tracing-collector-opentelemetry-collector", "resource.attributes.k8s@container@name": "opentelemetry-collector", "log.attributes.logtag": "F", "log.attributes.data_type": "traces", "resource.attributes.k8s_cluster_name": "test-cluster-platform", "resource.attributes.k8s@pod@start_time": "2024-05-22T20:00:30Z", "log.attributes.log@iostream": "stderr", "log.attributes.name": "logging", "resource.attributes.k8s@pod@name": "otel-tracing-collector-opentelemetry-collector-5dd8ccfc56-ksdf8", "log.attributes.log@file@path": "/var/log/pods/tracing_otel-tracing-collector-opentelemetry-collector-5dd8ccfc56-ksdf8_b87c0718-6690-4a31-9953-4aa03c2b4c2f/opentelemetry-collector/0.log", "resource.attributes.k8s@pod@uid": "b87c0718-6690-4a31-9953-4aa03c2b4c2f", "resource.attributes.k8s@node@name": "ip-10-3-15-81.eu-central-1.compute.internal", "log.attributes.spans": 4, "log.attributes.ts": 1716451470.5126758, "log.attributes.resource spans": 1, "log.attributes.msg": "TracesExporter", "log.attributes.kind": "exporter", "resource.attributes.k8s@container@restart_count": "0", "resource.attributes.kubernetes_cluster_name": "test-cluster-platform", "log.attributes.level": "info" }, "fields": { "log.attributes.time": [ "2024-05-23T08:04:30.512Z" ], "resource.attributes.k8s@pod@start_time": [ "2024-05-22T20:00:30.000Z" ], "time": [ "2024-05-23T08:04:30.512Z" ], "observedTime": [ "2024-05-23T08:04:30.534Z" ] }, "sort": [ 1716451470512 ] }

When we use opensearch datasource in Grafana to display logs we don't see non flat fields like "resource.attributes.k8s@namespace@name":

When we inspect the data using query inspector we don't even see those fields pulled from opensearch:

What you expected to happen:

I expect to see all other fields in the log output.

How to reproduce it (as minimally and precisely as possible):

Use otel collector to ingest some data with resource processor.

Anything else we need to know?:

No.

Environment:

• Grafana version: 10.4
• OpenSearch version: 2.11, 2.13
• Plugin version: 2.15.1

[idastambuk]

Hi @curuvija, I'm having trouble reproducing this - if I mock the response with your data I can see all the non-flat fields. To help debugging this, I have a few questions:

1. Is the problem the same if you query from Explore as opposed to a dashboard panel?
2. Can you paste the query itself from the Query Inspector here?
3. The response from the screenshot certainly looks strange - can you paste the entire response when querying from a dashboard panel?
4. Is the time field in datasource config set to time or another field?
5. Is your Logs configuration in datasource config different than below:

Thanks a lot!

[curuvija]

hi @idastambuk

thanks for helping me. Here are the details:

1. yes it is the same, just tried it
2. yes I can but partial since it holds some sensitive information
3. yes I can but partial since it holds some sensitive information (it's the same one for Explore and Dashboard panel)
4. it is set to time
5. it is the same

{ "request": { "url": "api/datasources/proxy/uid/cdn259ci6ipkwa/_opendistro/_ppl", "method": "POST", "data": "{\"query\":\"source=logs-aar-cluster-dev*| wheretime>= timestamp('2024-05-28 12:21:08') andtime <= timestamp('2024-05-28 12:23:08')\"}", "hideFromInspector": false }, "response": { "schema": [ { "name": "traceId", "type": "string" }, { "name": "log", "type": "struct" }, { "name": "resource", "type": "struct" }, { "name": "flags", "type": "long" }, { "name": "severityNumber", "type": "long" }, { "name": "body", "type": "string" }, { "name": "observedTime", "type": "timestamp" }, { "name": "schemaUrl", "type": "string" }, { "name": "spanId", "type": "string" }, { "name": "severityText", "type": "string" }, { "name": "droppedAttributesCount", "type": "long" }, { "name": "time", "type": "timestamp" } ], "datarows": [ [ "", null, null, 0, 0, "DEBUG SMTP: Found extension \"STARTTLS\", arg \"\"", "2024-05-28 12:21:22.568546544", "", "", "", 0, "2024-05-28 12:21:22.532425019" ], [ "", null, null, 0, 0, "I0528 12:22:27.696544 1 filter_out_schedulable.go:63] Filtering out schedulables", "2024-05-28 12:22:27.877169317", "", "", "", 0, "2024-05-28 12:22:27.696614513" ], [ "", null, null, 0, 0, "DEBUG SMTP: Found extension \"Ok\", arg \"\"", "2024-05-28 12:21:22.568577483", "", "", "", 0, "2024-05-28 12:21:22.532428728" ], [ "", null, null, 0, 0, "I0528 12:22:27.698224 1 pre_filtering_processor.go:67] Skipping ip-10-3-5-122.eu-central-1.compute.internal - node group min size reached (current: 2, min: 2)", "2024-05-28 12:22:27.87751072", "", "", "", 0, "2024-05-28 12:22:27.698295568" ], [ "", null, null, 0, 0, "DEBUG SMTP: Attempt to authenticate using mechanisms: LOGIN PLAIN DIGEST-MD5 NTLM XOAUTH2", "2024-05-28 12:21:22.56860927", "", "", "", 0, "2024-05-28 12:21:22.532435767" ], [ "", null, null, 0, 0, "I0528 12:22:27.698247 1 pre_filtering_processor.go:67] Skipping ip-10-3-5-19.eu-central-1.compute.internal - node group min size reached (current: 2, min: 2)", "2024-05-28 12:22:27.877552258", "", "", "", 0, "2024-05-28 12:22:27.698301928" ], [ "", null, null, 0, 0, "2024-05-28 12:21:25.747649+00:00 [error] <0.884591.0> Channel error on connection <0.272201.0> (100.64.1.75:50146 -> 100.64.1.97:5672, vhost: 'aar-dev', user: 'digipass'), channel 1:", "2024-05-28 12:21:25.896559443", "", "", "", 0, "2024-05-28 12:21:25.748049637" ], [ "", null, null, 0, 0, "I0528 12:22:27.698306 1 eligibility.go:104] Scale-down calculation: ignoring 3 nodes unremovable in the last 5m0s", "2024-05-28 12:22:27.877595648", "", "", "", 0, "2024-05-28 12:22:27.698351673" ], [ "", null, null, 0, 0, "100.64.1.66 - - [28/May/2024:12:21:31 +0000] \"GET /actuator HTTP/1.1\" 200 172 \"-\" \"okhttp/4.12.0\"", "2024-05-28 12:21:31.43850958", "", "", "", 0, "2024-05-28 12:21:31.289457665" ], [ "", null, null, 0, 0, "I0528 12:22:27.698365 1 static_autoscaler.go:617] Scale down status: lastScaleUpTime=2024-05-09 01:14:42.639815815 +0000 UTC m=-3589.701702971 lastScaleDownDeleteTime=2024-05-09 01:14:42.639815815 +0000 UTC m=-3589.701702971 lastScaleDownFailTime=2024-05-09 01:14:42.639815815 +0000 UTC m=-3589.701702971 scaleDownForbidden=false scaleDownInCooldown=false", "2024-05-28 12:22:27.877639349", "", "", "", 0, "2024-05-28 12:22:27.698417002" ] ], "total": 200, "size": 200, "$$config": { "url": "api/datasources/proxy/uid/cdn259ci6ipkwa/_opendistro/_ppl", "method": "POST", "data": "{\"query\":\"source=logs-cluster-dev*| wheretime>= timestamp('2024-05-28 12:21:08') andtime<= timestamp('2024-05-28 12:23:08')\"}", "hideFromInspector": false } } }

In the response schema I see "resource" as a field of type struct. Could it be that "resource.attributes.k8s@pod@uid" breaks it since it contains "@" symbol?

Kind regards,
Milos

[idastambuk]

Hi again @curuvija,
the struct fields returned from Open Search are null for all logs in the time span you're passing. Was there data there that was removed before pasting or are they indeed null? If you are able to access Open Search dashboards, can you make the same query and see if you're getting this data?

I've tried to reproduce this, but if I mock the response with this data instead of nulls:

{ "attributes": {"k8s@pod@uid": "test"}}

as the value of the "resource" field, the plugin returns the field to Grafana flattened. It returns as a separateresource.attributes.k8s@pod@uid field, for both PPL and Lucene queries.

[curuvija]

hi @idastambuk,

I've tried it and I got:

It works when I try with fields without @ symbol.

[idastambuk]

If you don't specify the field in the query, do you get the same error and if not, which form do you get the resource data in?

[curuvija]

I get the same as I get in Grafana UI:

[idastambuk]

In that case it doesn't look like a problem with the plugin, rather than those fields not being populated in OpenSearch. I would suggest debugging it in your Open Search instance. Since fields are being flattened in Grafana OpenSearch plugin fine, I will close this ticket.

[curuvija]

thanks a lot @idastambuk for your help.