Add Stack Frames and Symbol Mapping Status to SQS Data Forwarding Payload #83688
Comments
|
Assigning to @getsentry/support for routing ⏲️ |
|
Routing to @getsentry/product-owners-settings-integrations for triage ⏲️ |
getsantry
bot
moved this from Waiting for: Support
to Waiting for: Product Owner
in GitHub Issues with 👀 3
|
@chodges15 Amazon SQS is built on top of our deprecated framework called plugins. Unfortunately we don't provide any support for plugins moving forward and replaced it with integration platform where 3rd parties can build integrations for Sentry. This could be a private one built by you for the org or a public one that can be published by community and used by any Sentry organization. |
|
@sentaur-athena Thanks! This is very helpful and will give us access to the data we need from our errors. I think this issue is now more about the documentation under https://docs.sentry.io/organization/integrations/data-visualization/- while those pages indicate that the plugins are community supported, they do not reference that webhook integrations support data forwarding and that this is the Sentry supported way. |
|
Here's an example of error webhook: https://docs.sentry.io/organization/integrations/integration-platform/webhooks/errors/ Documentation for our current/supported framework is under integration platform |
|
I tested and looks like 
@sentaur-athena are there any rate limits on those webhooks or any other scale concerns? (we're looking at a very high number of events) |
|
@sentaur-athena / @realkosty - any updates on scale limitations for the error webhooks? |
|
Also, with integration platform error webhooks, is there a way to specify which projects to forward errors for? The only per-project settings I could find were for the legacy webhook plugin. |
|
In my quick search I didn't find anything about the limitations for webhooks. There's definitely some capacity limit though so I need to dig more. Approximately how many errors per minute are we talking about? Webhooks don't have per project setting but depending on your use case you can add that business logic to the webhook handler since the project is included in the payload. |
|
In our case across all projects, the event rate would be on the order of 500 events per second. |
|
Adding @markstory since he has more context about infra capacity. 500 events per second does seem high to me so I was wondering if instead you can use the issues webhook or is it necessary for your use case to get notified per event? Using issue webhook you get notified per issue which you can get to event stack trace by calling this API. |
|
The use case here is getting error data forwarded to us that includes stack trace information. The SQS data forwarding doesn't include this information, and without being able to enable webhook error forwarding per Sentry project (only per slug) it massively increases the scale of using webhooks - to the point of being an untenable solution. We need a solution that will allow us to get symbolicated error data into our data warehouse. |
|
Looks like there's a slack thread about this and Yuval is following up about potential fixes. He will reach out to you with any updates or questions. |
… projects (#85502) This enables limiting integration's webhooks to fire only for a subset of projects, which would solve #83688 We make sure to skip early on before payload is serialized. No UI planned at this stage # Discussion https://sentry.slack.com/archives/CD4NYFD34/p1739558637148289?thread_ts=1738876805.029239&cid=CD4NYFD34 # Testing re-tested after rebase ``` export P2_DSN=http://[email protected]:8000/2 export P3_DSN=http://[email protected]:8000/3 export P4_DSN=http://[email protected]:8000/4 python3 http_server.py | grep -o '"project":[[:space:]]*[0-9]\+' ngrok http 8077 open http://127.0.0.1:8000/settings/testorg/developer-settings/actual-test-integration-548f00/ # > plug in ngrok URL as webhook url # > check 'error:created' # > click SAVE scurl 'http://127.0.0.1:8000/api/0/sentry-app-installations/bde8f035-832e-4e09-a0c7-e0980d499cdd/service-hook-projects/' -X DELETE scurl 'http://127.0.0.1:8000/api/0/sentry-app-installations/bde8f035-832e-4e09-a0c7-e0980d499cdd/service-hook-projects/' # > verify ^ returns [] SENTRY_DSN=$P4_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P3_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P2_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 # > Verify http server outputs: # "project":4 # "project":3 # "project":2 # if needed check events are ingested at http://127.0.0.1:8000/organizations/testorg/issues/?project=2&query=&referrer=issue-list&statsPeriod=4h # if ngrok/http_server having issues can also check http://127.0.0.1:8000/settings/testorg/developer-settings/actual-test-integration-548f00/dashboard/ scurl 'http://127.0.0.1:8000/api/0/sentry-app-installations/bde8f035-832e-4e09-a0c7-e0980d499cdd/service-hook-projects/' -X POST -H "Content-Type: application/json" -d '{"projects": ["p2", "p4"]}' scurl 'http://127.0.0.1:8000/api/0/sentry-app-installations/bde8f035-832e-4e09-a0c7-e0980d499cdd/service-hook-projects/' # > verify ^ returns [{..., "project_id": 2}, {..., "project_id": 4}] SENTRY_DSN=$P2_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P3_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P4_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 # > Verify http server outputs: # "project":2 # "project":4 scurl 'http://127.0.0.1:8000/api/0/sentry-app-installations/bde8f035-832e-4e09-a0c7-e0980d499cdd/service-hook-projects/' -X POST -H "Content-Type: application/json" -d '{"projects": ["p4"]}' SENTRY_DSN=$P2_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P3_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 SENTRY_DSN=$P4_DSN sentry-cli send-envelope --raw ~/home/empower-mocks-public/3/5 # > Verify http server outputs: # "project":4 ``` --------- Co-authored-by: Christinarlong <[email protected]> Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com> Co-authored-by: Christinarlong <[email protected]>
Problem Statement
Currently, Sentry's data forwarding to Amazon SQS includes basic error information such as environment, event ID, exception type, and release information.
However, it lacks critical debugging context such as stack frames and the status of symbol/source mapping operations. Both of these show up in the event JSON available in the frontend.
Stack Frames: Without detailed stack trace information, teams:
Symbol Mapping Status: The absence of symbolification success indicators means:
Solution Brainstorm
Include more of the event payload in the forwarded data, but not all of it as that would greatly increase the amount of data to be stored. For example, adding the event payload that contains the state of all threads would be counterproductive.
The main keys from the event JSON from the UI we need are:
"exception"and"debug_meta"Product Area
Settings - Integrations
The text was updated successfully, but these errors were encountered: