Expand `record_lost_event` to Provide Envelope/Event Data and Integration Data

[wesyao]

Problem Statement

Hello!

Currently record_lost_event is very limited and only provides event_processor as the reason to why an event was dropped. It is also missing data on what type of event depending (specifically on the dedup integration).

What is being dropped and why it is being dropped is important for the consumers of the API and can be used to track their own telemetry.

Solution Brainstorm

Expose the event and what integration acted upon the event

[linear]

PY-1855 Expand `record_lost_event` to Provide Envelope/Event Data and Integration Data

[sentrivana]

Hey @wesyao, thanks for writing in.

record_lost_event is an internal method used for populating client reports. The SDK might drop events for many reasons: in order to ensure the SDK's footprint on the app doesn't blow out of proportion, in case of network errors, because either integration logic (like dedupe) or user logic (before_send) drops the event, because of sampling, etc.

Currently record_lost_event is very limited and only provides event_processor as the reason to why an event was dropped.

You're likely only seeing event_processor because it's the most prevalent reason in your setup, but there are other reasons, see https://develop.sentry.dev/sdk/telemetry/client-reports/ (and this for where this is actually happening in the code).

What do you want to get more visibility into? Might be there are other ways to achieve that. I believe debug=True also logs some info when an event is dropped.

[wesyao]

Hey @sentrivana, thanks for getting back! record_lost_event appears to be an API exposed by the Transport interface, which we implement.

For context, we're migrating our companies error reporting to Sentry and we're trying to collect metrics to show that sentry is reporting as well as our internal exception service. This will help keep us on the platform, as paying customers.

@yuval-sentry mentioned this change was added. However, I don't think this is sufficient. What we ideally need to know is what is the envelope and what severity it is?

I think if we just return the event here, this info will be available in record_lost_event.

What are your thoughts?

[JParisFerrer]

I think if we just return the event here, this info will be available in record_lost_event

Not sure if this will work, since this is how the integration tells the SDK to drop the event. However, up in the SDK around here, it has the event prior to feeding it to the integration (which becomes event_), which could be given to record_lost_event, or converted into some other public type and then given to record_lost_event

Also noticed that the integrations have an identifier string, so maybe the reason could be extended (such as event_processor (dedupe)), or a new argument added for the integration's identifier (so that the server-side data, if any, isn't polluted but custom transports have access)

[sentrivana]

Hey folks, thanks for your thoughts on this.

If the goal is to make sure the SDK captures important events, would it make sense to instead of looking at dropped events, look at events that have been captured? Looking at dropped events might be misleading -- if an event has been dropped, it doesn't mean that it hasn't in fact been captured by another integration, for example. You're bound to see a lot of noise there.

Talking about this internally, there is definitely improvement potential with regards to the reasons why an event was dropped -- we will look into making client reports better so that the category is not just event_processor, but ideally also contains the integration responsible. These more fine-grained categories would then be visible in Settings > Stats & Usage. That being said, we won't be changing record_lost_event.