No notifications being sent on slack

[btxbtx]

System

• local kind cluster
• argo installed via 4.9.16 helm chart
• argo version v2.4.7+81630e6

Problem

I have configured slack notifications according to this documentation. I then add the kustomize guestbook app manually via the argocd ui. After that, I add the following labels manually to the argo guestbook application

notifications.argoproj.io/subscribe.on-deployed-out-of-date.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-deployed.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-health-degraded.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-sync-failed.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-sync-running.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-sync-status-unknown.slack: <my slack channel>
notifications.argoproj.io/subscribe.on-sync-succeeded.slack: <my slack channel>

I try performing some operations through the argo ci, like syncing, deleting the pod, etc. Nothing shows up in the notification controller logs (which are pasted below this section).

The docs on troubleshooting the notifications state that I should be able to:

kubectl exec -it argocd-notifications-controller-<pod-hash> \
  /app/argocd-notifications trigger get

However, that path doesn't exist. A binary with the same name is at /usr/local/bin/argocd-notifications but it does not work due to a port conflict with another argo resource:

$ k exec argocd-notifications-controller-74ddd66567-vgwvc -- /usr/local/bin/argocd-notifications trigger get
time="2022-08-25T13:31:38Z" level=info msg="serving metrics on port 9001"
time="2022-08-25T13:31:38Z" level=info msg="loading configuration 9001"
time="2022-08-25T13:31:38Z" level=fatal msg="listen tcp 0.0.0.0:9001: bind: address already in use"
command terminated with exit code 1

Logs

msg="Trigger on-deployed-out-of-date result: []" resource=argocd/guestbook
msg="Trigger on-deployed result: []" resource=argocd/guestbook
msg="Trigger on-health-degraded result: []" resource=argocd/guestbook
msg="Trigger on-sync-failed result: []" resource=argocd/guestbook
msg="Trigger on-sync-running result: []" resource=argocd/guestbook
msg="Trigger on-sync-status-unknown result: []" resource=argocd/guestbook
msg="Trigger on-sync-succeeded result: []" resource=argocd/guestbook
msg="Processing completed" resource=argocd/guestbook
msg="Start processing" resource=argocd/guestbook
msg="Trigger on-deployed-out-of-date result: []" resource=argocd/guestbook
msg="Trigger on-deployed result: []" resource=argocd/guestbook
msg="Trigger on-health-degraded result: []" resource=argocd/guestbook
msg="Trigger on-sync-failed result: []" resource=argocd/guestbook
msg="Trigger on-sync-running result: []" resource=argocd/guestbook
msg="Trigger on-sync-status-unknown result: []" resource=argocd/guestbook
msg="Trigger on-sync-succeeded result: []" resource=argocd/guestboo

Desired Behavior

I have logs sent to my slack channel.
I have a more reliable way to test notifications than my current process.

Configuration

argocd-notifications-cm:

apiVersion: v1
data:
  service.slack: |
    token: $slack-token
  subscriptions: |
    - recipients:
      - slack:argocd-feed
      triggers:
        - on-deployed
        - on-deployed-out-of-date
        - on-health-degraded
        - on-sync-failed
        - on-sync-running
        - on-sync-status-unknown
        - on-sync-succeeded
  template.app-sync-status: |
    message: |
      Application "".app.metadata.name"" sync is ".app.status.sync.status"".
      Application details: ".context.argocdUrl""/applications/".app.metadata.name"".
    slack:
      attachments: |
        [{
          "title": "".app.metadata.name""",
          "title_link": "".context.argocdUrl""/applications/".app.metadata.name""",
          "color": "#18be52",
          "fields": [{
            "title": "Sync Status",
            "value": "".app.status.sync.status""",
            "short": true
          }, {
            "title": "Repository",
            "value": "".app.spec.source.repoURL""",
            "short": true
          }]
        }]
  trigger.on-deployed: "- send: [app-sync-status]\n  when:  app.status.operationState.phase
    in ['Succeeded'] and app.status.health.status == 'Healthy' \n"
  trigger.on-deployed-out-of-date: |
    - send: [app-sync-status]
      when: app.status.sync.status == 'OutOfSync' and app.status.operationState.phase in ['Succeeded']
  trigger.on-health-degraded: |
    - send: [app-sync-status]
      when: app.status.health.status == 'Degraded'
  trigger.on-sync-failed: |
    - send: [app-sync-status]
      when: app.status.operationState.phase in ['Error', 'Failed']
  trigger.on-sync-running: |
    - send: [app-sync-status]
      when: app.status.operationState.phase in ['Running']
  trigger.on-sync-status-unknown: |
    - send: [app-sync-status]
      when: app.status.sync.status == 'Unknown'
  trigger.on-sync-succeeded: |
    - send: [app-sync-status]
      oncePer: app.status.sync.revision
      when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: slack-notifier
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2022-08-25T09:26:17Z"
  labels:
    app.kubernetes.io/instance: slack-notifier
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-slack-notifier
    helm.sh/chart: argocd-slack-notifier-0.1.0
  name: argocd-notifications-cm
  namespace: argocd
  resourceVersion: "194238"
  uid: 50dde672-e30e-4352-b6c2-1f83e0f60a86

argocd-notifications-secret:

apiVersion: v1
data:
  slack-token: <redacted>
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: slack-notifier
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2022-08-25T09:26:17Z"
  labels:
    app.kubernetes.io/instance: slack-notifier
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-slack-notifier
    helm.sh/chart: argocd-slack-notifier-0.1.0
  name: argocd-notifications-secret
  namespace: argocd
  resourceVersion: "194237"
  uid: 5cab9c13-57f4-47db-9007-23edfe801da8
type: Opaque

[btxbtx]

The message template is botched. Too many quotes, and curly braces were removed -- this was due to a templating engine failure.