ArgoCD reports Status Sync / Alertmanager reports Sync Failed

[reefland]

ArgoCD: v2.3.4+ac8b7df

My ArgoCD app applications my root App of Apps, reports this in ArgoCD:

OPERATION: Sync
PHASE: Succeeded
MESSAGE: successfully synced (all tasks run)
STARTED AT: a minute ago (Tue Jun 21 2022 17:02:59 GMT-0400)
DURATION: 01:16 min
FINISHED AT: a few seconds ago (Tue Jun 21 2022 17:04:15 GMT-0400)
REVISION: d2f6359
INITIATED BY: admin

Yet, Prometheus is reporting it failed for the past 2 days.

I have this Alert Defined:

name: ArgoAppSyncFailed
expr: argocd_app_sync_total{phase!="Succeeded"} == 1
for: 1m
labels:
severity: warning
annotations:
summary: '{{ $labels.name }}' Application has sync phase as '{{ $labels.phase }}'

This is the current Status being listed as phase=Failed:

alertname=ArgoAppSyncFailedcontainer=application-controller
dest_server=https://kubernetes.default.svc
endpoint=http-metrics
exported_namespace=argocd
instance=10.42.2.135:8082
job=argocd-application-controller-metrics
name=applications
namespace=argocd
phase=Failed
pod=argocd-application-controller-0
project=default
service=argocd-application-controller-metrics
severity=warning	

It has been in state firing for 2 days now:

State: firing
Active Since: 2022-06-19T21:05:00.509122803Z	1

Prometheus Target is up:

serviceMonitor/monitoring/argocd-applicationset-controller/0 (1/1 up)

Last Scrape: 19.884s ago 
Duration: 3.040ms
Error:  <blank>

The specific metric as shown by Prometheus:

argocd_app_sync_total{container="application-controller", dest_server="https://kubernetes.default.svc", endpoint="http-metrics", exported_namespace="argocd", instance="10.42.2.135:8082", job="argocd-application-controller-metrics", name="applications", namespace="argocd", phase="Failed", pod="argocd-application-controller-0", project="default", service="argocd-application-controller-metrics"}

The argocd-application-controller-0 logs show seem to indicate everything is fine:

time="2022-06-21T20:43:32Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=applications
time="2022-06-21T20:43:32Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2022-06-21T20:43:32Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
time="2022-06-21T20:43:32Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=applications
time="2022-06-21T20:43:32Z" level=info msg="getRepoObjs stats" application=applications build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=101 unmarshal_ms=101 version_ms=0
time="2022-06-21T20:43:32Z" level=info msg="Skipping auto-sync: application status is Synced" application=applications
time="2022-06-21T20:43:32Z" level=info msg="No status changes. Skipping patch" application=applications
time="2022-06-21T20:43:32Z" level=info msg="Reconciliation completed" application=applications dedup_ms=0 dest-name= dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=194 fields.level=1 git_ms=101 health_ms=0 live_ms=1 settings_ms=0 sync_ms=0 time_ms=523

Not sure what to do next. I've forced sync'd and refreshed all day long, that's not working.