"Status: 500. Message: unknown OpenSearch error response" when trying to do any sort of aggregation in grafana

[danfinn]

What happened:
We have a grafana instance connected to an AWS OpenSearch cluster. We can view metrics with this however whenever we try to create any sort of simple aggregation with the data we get a generic opensearch error. There are no errors being logged on our opensearch cluster and there are no 500s being reported. The error that grafana is logging every time this happens is:

logger=context userId=1 orgId=1 uname=admin t=2024-05-29T13:44:46.453355732Z level=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=192.168.128.115 time_ms=9636 duration=9.636188649s size=192 referer="https://grafana.bcdemas.local/explore?orgId=1&panes=%7B%22Qv8%22%3A%7B%22datasource%22%3A%22e3d933fa-bf77-4cc0-baef-ba3ee6d4f9d1%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22datasource%22%3A%7B%22type%22%3A%22grafana-opensearch-datasource%22%2C%22uid%22%3A%22e3d933fa-bf77-4cc0-baef-ba3ee6d4f9d1%22%7D%2C%22query%22%3A%22%22%2C%22queryType%22%3A%22lucene%22%2C%22alias%22%3A%22%22%2C%22metrics%22%3A%5B%7B%22id%22%3A%221%22%2C%22type%22%3A%22avg%22%2C%22field%22%3A%22system.cpu.iowait.pct%22%7D%5D%2C%22bucketAggs%22%3A%5B%7B%22type%22%3A%22date_histogram%22%2C%22id%22%3A%222%22%2C%22settings%22%3A%7B%22interval%22%3A%22auto%22%7D%2C%22field%22%3A%22%40timestamp%22%7D%5D%2C%22format%22%3A%22table%22%2C%22timeField%22%3A%22%40timestamp%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-6h%22%2C%22to%22%3A%22now%22%7D%7D%7D&schemaVersion=1" handler=/api/ds/query

Initially we were getting the error that too many buckets were being created but I've had AWS increase our bucket limit and now this is what we are seeing.

I am on the latest grafana opensearch plugin, 2.15.3

What you expected to happen:
to be able to create simple aggregations on the data in opensearch

How to reproduce it (as minimally and precisely as possible):
do a Group By on any metric

Anything else we need to know?:

Environment:

• Grafana version: 10.2.3
• OpenSearch version: OpenSearch 2.9
• Plugin version: 2.15.3

[kevinwcyu]

Hi @danfinn,

I've opened a PR to improve the error message. Once it's merged and release, it'll hopefully provide a more helpful message to help pinpoint the issue.

[danfinn]

I'm assuming you think I'm still hitting the too many buckets issue then? Previously I was actually getting this error and I increased the bucket limit and now I'm getting this generic error.

[kevinwcyu]

Hi @danfinn, I actually don't have a guess of what the issue is. There's an error from the OpenSearch response that will likely be in caused_by.reason, but the current handling doesn't handle that so it's just returning the generic error message. I'm hoping that the actual error will be in caused_by.reason and it will be more helpful than the generic error message.

[danfinn]

ok, defiinitely sounds like an improvement. hopefully it will give me some insight

[danfinn]

@kevinwcyu do you know when this might get released?

[kevinwcyu]

@danfinn, I just released it in v2.15.4

[danfinn]

That did help, thanks @kevinwcyu. We are still hitting the same max buckets error. Interesting that it was previously giving this error but then stopped once we initially raised the limit.

[kevinwcyu]

Thanks for the update @danfinn, I'll close this issue off since the cause has been found.

[kevinwcyu]

Effort tracked in #394