[receiver/otlpreceiver] Unable to monitor ResourceExhausted errors

[shariq-anwar]

Component(s)

service/internal/graph

Describe the issue you're reporting

Hi there, we recently enabled internal telemetry due to some apps hitting the max_recv_msg_size_mib grpc limits.

Here is the error:

Aug 21, 2025 8:16:31 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Failed to export metrics. Server responded with gRPC status code 8. Error message: grpc: received message larger than max (12377970 vs. 4194304)

When we look at the internal telemetry metric rpc.server.request.size we notice that its way below the 4mb limit.

What seems to be happening is otel does not accept the message/request because its larger than the 4mb limit set. Due to the message/request being dropped the internal telemetry metric does not show its max limit being hit.

What would be the best way to monitor this? Is there a way for otel to drop the message/request after accepting it so that way the internal telemetry metric shows that its hitting its limits? Its really challenging to monitor applications with this limitation because we never know if the max message size limits are being hit.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• service/internal/graph: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[braydonk]

@open-telemetry/collector-triagers - Could we update some information on this issue? I'm thinking title to [receiver/otlpreceiver] Unable to monitor ResourceExhausted errors and adding the corresponding tag.

---

We are in a bit of a bind in this scenario. Your assumption is correct that the message size is not being recorded when the message is too large resulting in a ResourceExhausted error.

In grpc-go, internal metrics are recorded when the GRPC server code calls into the hook points from the configured stats.Handlers on the server. In this case, the ResourceExhausted will be triggered when this comes back with an error: https://github.com/grpc/grpc-go/blob/3c743c9d3f630d12ec805055c304cc30fa6ea89d/server.go#L1387
Which is before the server calls into the stats handlers with payload length info: https://github.com/grpc/grpc-go/blob/3c743c9d3f630d12ec805055c304cc30fa6ea89d/server.go#L1409

Is there a way for otel to drop the message/request after accepting it so that way the internal telemetry metric shows that its hitting its limits?

No, since we are dependent on the grpc-go implementation which will not do this.

What would be the best way to monitor this?

Theoretically the best way would be to monitor ResourceExhausted error counts. Surprisingly though, this doesn't seem to be possible with grpc-go today. If it is, I can't figure out how. I would have figured some kind of rpc.server.error metric with an error.type attribute would exist, but I am not seeing it anywhere. I'm not sure if there's any way we can add this just in the Collector or opentelemetry-go-contrib somewhere, at the moment I don't see a good way.

At this time I don't have a good recommendation for an alternative. I've sent this to other Collector users who might have a better idea than me on this.