how to trace/debug the source of mapper_parsing_exception in opensearch output

[nachum234]

I have lots of applications that runs on Kubernetes cluster and write their logs through fluentbit to opensearch.
we have some mapper_parsing_exception errors in fluentbit and I would like to trace the source pod.
What is the best/easiest way to do that?
for example now I see the following log in fluentbit:
"Could not dynamically add mapping for field [service.name]. Existing mapping for [data.service] must be of type object but found [text]., Could not dynamically add mapping for field [service.name]. Existing mapping for [data.service] must be of type object but found [text]."

how do I know which pod sent this log?
I have Kubernetes meta-data on the logs but this error log don't print all the fields

[patrick-stephens]

I think this may be tricky - have you tried log_level debug?

My concern is that dumping all the debug may then cause other problems or affect things.

The best bet may be to use an expect or lua filter to apply the tests on the data there and dump the specific details you need if it finds a failing record. You can even with LUA also do some internal mangling/tweaking of the data to provide a fallback or convert the mapping to a valid value.

[nachum234]

fluetbit handles a high volume of logs, so I wanted to avoid using the debug level.
I did use a Lua filter to apply a 'fix' to the record, but I was hoping for a more efficient solution since this approach is quite tedious.
Thank you for your answer, Patrick

[patrick-stephens]

You can use your LUA filter though to also tell you the records that are failing.
I don't think there's a good generic way to do it for any output - particularly if that logging ends up being high volume as well.