Significant Latency in ArgoCD Notification Generation (~10 Minutes) for Synchronization Events

[jwalant527]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
We’ve recently implemented Argo CD notifications to track synchronization and deployment status events, specifically sending these notifications to a Webhook. The webhook in turn sends these notifications to a Kafka topic. However, we’re experiencing significant latency, where there’s an approximate 10-minute+ delay between an actual synchronization event (such as a deployment) and the notification being generated and sent to the Webhook.

To Reproduce

• Set up ArgoCD notifications as shown below in the yaml.
• Configure notifications to be sent to a webhook.
• Create a change in the ArgoCD's state. (Success, failure, syncing etc.)
• This is where the delay appears.

https://gitlab.be-md.ncbi.nlm.nih.gov/pd/do/p2/cd/agents/be-clusters/deployment-notifier/-/tree/feat/CD-6874-test?ref_type=heads

Expected behavior

Notifications should be generated and sent in near real-time (or as close as possible) after a synchronization event is triggered.

Actual Behavior

There is an approximate 10-minute delay between a synchronization event occurring and the Argo CD notification being generated and sent.

Screenshots:
Here is a screenshot of my argocd-notifications-cm.yml:

To be clear, the notifications are coming through, but with a significant delay.