Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading to v0.15.0 causing conflicts while updating apps #917

Open
schahal opened this issue Nov 5, 2024 · 3 comments
Open

Upgrading to v0.15.0 causing conflicts while updating apps #917

schahal opened this issue Nov 5, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@schahal
Copy link

schahal commented Nov 5, 2024

Describe the bug

After upgrading to to v0.15.0 (as part of https://github.com/argoproj/argo-helm/releases/tag/argocd-image-updater-0.11.1) from v0.14.0, I started seeing the following messages:

...
2024-11-04 14:48:22.402 | time="2024-11-04T22:48:22Z" level=warning msg="conflict occurred while updating application: foo-app-remote-cluster, retrying... (5/7)
2024-11-04 14:48:21.739 | time="2024-11-04T22:48:21Z" level=warning msg="conflict occurred while updating application: baz-app-remote-cluster, retrying... (3/7)"
...

This would repeat for dozens of apps that get deployed to the remote cluster... and that symptom is likely due to the image updater adding overrides for unrelated apps.

For example, my app foo app spec was originally:

destination:
  namespace: foo
  server: https://<redacted>
project: foo
source:
  helm:
    ignoreMissingValueFiles: true
    parameters:
      - forceString: true
        name: foo.image.repository
        value: <redacted>.com/<redacted>/foo
      - forceString: true
        name: foo.image.tag
        value: latest@sha256:<redacted>
    releaseName: foo-app
    skipCrds: true
    valueFiles:
      - values.yaml
  path: charts/app
  repoURL: https://github.com/<redacted>/foo.git
  targetRevision: HEAD
syncPolicy:
  automated:
    prune: true
    selfHeal: false

But after the upgrade, the following additional parameters from other apps were added:

      - forceString: true
        name: bar.image.repository
        value: <redacted>.com/<redacted>/bar
      - forceString: true
        name: bar.image.tag
        value: latest@sha256:<redacted>
      - forceString: true
        name: baz.image.repository
        value: <redacted>.com/<redacted>/baz
      - forceString: true
        name: baz.image.tag
        value: latest@sha256:<redacted>

This then manifested in ArgoCD unable to Sync due to errors like:

"Role/foo-app is part of applications argocd/bar-app foo-app-"

To Reproduce

  1. Be running image-updater v0.14.0 (good)
  2. Have a few (I have about a couple dozen) Applications with spec like above which syncs to a remote cluster
  3. Upgrade image-updater to v0.15.0 (by upgrading to chart https://github.com/argoproj/argo-helm/releases/tag/argocd-image-updater-0.11.1 from 0.11.0)
  4. Notice behavior above

Expected behavior

To work similar to v0.14.0 (i.e., no warnings/errors in logs and not cross-updating Applications incorrectly)

Additional context

Workaround:

  1. Revert back to v0.14.0
  2. Manually remove the extraneous ArgoCD parameters

Version

v0.15.0

Logs

warning and error logs arose from ~12:00 to 14:45 (when I was on v0.15.0)

Screenshot 2024-11-05 at 3 26 37 PM
Those error/warn messages were all essentially this (Click to view):
2024-11-04 14:45:26.980 time="2024-11-04T22:45:26Z" level=warning msg="conflict occurred while updating application: <redacted>, retrying... (4/7)"
2024-11-04 14:45:26.801 time="2024-11-04T22:45:26Z" level=error msg="Could not update application spec: max retries(7) reached while updating application: <redacted>" application=<redacted>
@schahal schahal added the bug Something isn't working label Nov 5, 2024
@mehdicopter
Copy link

#896

@borgarlie
Copy link

I wanna add to this issue. After upgrading to v0.15.0, we had an app that kept getting the content of other apps. It tried to resync very often, getting an seemingly arbitrary application's content. It was always the last application in our list if we look from root's perspective. If we removed that app completely, the new last app in the list got the same problem. This was consistently an issue while running the image updater. Reverting to v0.14.0 fixed the issue.

@mehdicopter
Copy link

This is a duplicate, please close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants