Helm is Constantly creating new releases for charts even when no changes

[infa-madhanb]

Describe the bug

Hello,

On my EKS cluster each time the flux sync runs, the helm operator tries to do an update and create a new release even without any change in the chart. Same helm operator version and monitoring addon helm release is running on several clusters but seeing issue only on few clusters.

Seeing same issue in Flux v2 as well, aws-auth configmap changes is keep on creating even when there isn't any changes in file.

Cluster Version: v1.20.11(EKS)
Helm Controller Version: helm-controller:v0.14.1
kustomize-controller version: v0.18.2
source-controller version: v0.19.2

To Reproduce

Steps to reproduce the behaviour:
HelmRelease example

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: monitoringaddon
  namespace: lamm
spec:
  resetValues: false
  helmVersion: v3
  releaseName: monitoringaddon
  chart:
    repository: https://prometheus-community.github.io/helm-charts
    name: kube-prometheus-stack
    version: 39.5.0
  valuesFrom:
  - secretKeyRef:
      name: lamm-secrets
      namespace: flux
      key: lamm-secrets.yaml
      optional: false
  values:
    # Installing on GKE/EKS/AKS: Since the controlplane is managed in these solutions, make sure you tell prometheus to not monitor the scheduler or controller-manager #
    nameOverride:
      prometheus-operator
      
    kubeScheduler:
      enabled: false

    kubeControllerManager:
      enabled: false

    kubelet:
      serviceMonitor:
        https: true

    kubeProxy:
      enabled: false

    kubeEtcd:
      enabled: false

    defaultRules:
      rules:
        etcd: false
        kubernetesSystem: false
        kubeScheduler: false
        kubeApiserverAvailability: true
        kubeApiserverBurnrate: true
        kubeApiserverHistogram: true
        kubeApiserverSlos: true
        kubePrometheusGeneral: true
        kubePrometheusNodeRecording: true
        kubeStateMetrics: true
        kubelet: true
        network: true
        node: true
        nodeExporterAlerting: true
        nodeExporterRecording: true
      disabled:
        PrometheusBadConfig: false         
    coreDns:
      service:
        selector:
          k8s-app: coredns
    prometheus-node-exporter:
      image:
        repository: quay.io/prometheus/node-exporter
        # Overrides the image tag whose default is {{ printf "v%s" .Chart.AppVersion }}
        tag: "v1.3.1"
        pullPolicy: IfNotPresent     
      podLabels:
        ## Add the 'node-exporter' label to be used by serviceMonitor to match standard common usage in rules and grafana dashboards
        jobLabel: node-exporter
      extraArgs:
        - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
        - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
      service:
        portName: http-metrics 
      prometheus:
        monitor:
          enabled: true
          jobLabel: jobLabel          
    kube-state-metrics:
      image:
        repository: registry.k8s.io/kube-state-metrics/kube-state-metrics
        tag: v2.5.0
        pullPolicy: IfNotPresent
      releaseLabel: true  
      prometheus:
        monitor:
          enabled: true
          honorLabels: true
      collectors:
        - certificatesigningrequests
        - configmaps
        - cronjobs
        - daemonsets
        - deployments
        - endpoints
        - horizontalpodautoscalers
        - ingresses
        - jobs
        - limitranges
        - mutatingwebhookconfigurations
        - namespaces
        - networkpolicies
        - nodes
        - persistentvolumeclaims
        - persistentvolumes
        - poddisruptionbudgets
        - pods
        - replicasets
        - replicationcontrollers
        - resourcequotas
        - secrets
        - services
        - statefulsets
        - storageclasses
        - validatingwebhookconfigurations
        - volumeattachments               
    prometheusOperator:
      tls:
        enabled: true
        internalPort: 10250
      admissionWebhooks:
        enabled: true
        certManager:
          enabled: false
          rootCert:
            duration: 10y
          admissionCert:
            duration: 10y
        patch:
          enabled: true
          image:
            repository: k8s.gcr.io/ingress-nginx/kube-webhook-certgen
            tag: v1.2.0
            sha: ""
            pullPolicy: IfNotPresent         
      serviceMonitor:
        interval: 30s
      image:
        repository: quay.io/prometheus-operator/prometheus-operator
        tag: v0.58.0
        sha: ""
        pullPolicy: IfNotPresent
      prometheusConfigReloader:
        # image to use for config and rule reloading
        image:
          repository: quay.io/prometheus-operator/prometheus-config-reloader
          tag: v0.58.0
          sha: ""

        # resource config for prometheusConfigReloader
        resources:
          requests:
            cpu: 200m
            memory: 50Mi
          limits:
            cpu: 400m
            memory: 1024Mi
      thanosImage:
        sha: "123"                
    prometheus:
      thanosService:
        enabled: false
      thanosServiceMonitor:
        enabled: false
      thanosServiceExternal:
        enabled: false
      thanosIngress:
        enabled: false
      extraSecret:
        data: false 
      serviceMonitor:
        interval: 30s
      ingress:
        enabled: true
        hosts:
        - ***
        annotations:
          cert-manager.io/acme-challenge-type: dns01
          cert-manager.io/acme-dns01-provider: route53
          #cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          #cert-manager.io/acme-challenge-type: "dns01"
          #cert-manager.io/acme-dns01-provider: route53
          nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
          kubernetes.io/ingress.class: inginx
        tls:
        - hosts:
          - ***
          secretName: prometheus-ssl-cert
      prometheusSpec:
        image:
          repository: quay.io/prometheus/prometheus
          tag: v2.37.0
          sha: ""        
        #additionalScrapeConfigsExternal: true
        additionalScrapeConfigsSecret:
          enabled: true
          key: additional-scrape-configs.yaml
          name: monitoringaddon-prometheus-prometheus-scrape-confg  
        externalUrl: ***
        scrapeInterval: 30s
        evaluationInterval: 30s
        externalLabels:
          cluster_name: ***
          cluster: ***
        secrets:
        - istio.default
        - iics-tls-secrets
        retention: 21d
        storageSpec:
          volumeClaimTemplate:
            spec:
              resources:
                requests:
                  storage: 75Gi
        thanos:
          baseImage: quay.io/thanos/thanos
          version: v0.27.0
          objectStorageConfig:
            key: thanos.yaml
            name: thanos-objstore-config
    thanosRuler:
      enabled: false
      serviceMonitor:
        selfMonitor: false
      serviceAccount:
        create: false
      podDisruptionBudget:
        enabled: false
      ingress:
        enabled: false
      extraSecret:
        data: false
    grafana:
      image:
        repository: grafana/grafana
        tag: "9.0.6"
        sha: ""
        pullPolicy: IfNotPresent  
      sidecar:
        image:
          sha: ""
        dashboards:
          enabled: false
          multicluster:
            global:
              enabled: false
            etcd:
              enabled: false
        datasources:
          enabled: false
      serviceMonitor:
        enabled: true           
      persistence:
        enabled: false
      ingress:
        enabled: true
        hosts:
        - ***
        annotations:
          cert-manager.io/acme-challenge-type: dns01
          cert-manager.io/acme-dns01-provider: route53
          #cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          #cert-manager.io/acme-challenge-type: "dns01"
          #cert-manager.io/acme-dns01-provider: route53
          nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
          kubernetes.io/ingress.class: inginx
        tls:
        - hosts:
          - ***
          secretName: grafana-ssl-cert

    alertmanager:
      extraSecret:
        data: false     
      alertmanagerSpec:
        alertmanagerConfigSelector: {}      
        externalUrl: ***
        retention: 4320h
        replicas: 2
        storage:
          volumeClaimTemplate:
            spec:
              resources:
                requests:
                  storage: 5Gi           
      ingress:
        enabled: true
        hosts:
        - ***
        annotations:
          cert-manager.io/acme-challenge-type: dns01
          cert-manager.io/acme-dns01-provider: route53
          #cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          cert-manager.io/cluster-issuer: certissueraddon-ct-acme-issuer
          #cert-manager.io/acme-challenge-type: "dns01"
          #cert-manager.io/acme-dns01-provider: route53
          nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
          kubernetes.io/ingress.class: inginx
        tls:
        - hosts:
          - ***
          secretName: alertmanager-ssl-cert
      config:
        global:
          smtp_smarthost: ***
          resolve_timeout: 5m
          smtp_from: ***
          smtp_require_tls: false
        inhibit_rules:
        - equal:
          - alertname
          source_match:
            severity: critical
          target_match:
            severity: warning
        route:
          group_by:
          - alertname
          - cluster
          - service
          group_interval: 5m
          group_wait: 30s
          receiver: blackhole
          repeat_interval: 1h
          routes:
          - receiver: Watchdog
            repeat_interval: 1m
            group_interval: 1m
            match:
              alertname: Watchdog

Post the HelmRelease status:

Status:
  Conditions:
    Last Transition Time:   2022-08-11T07:11:03Z
    Last Update Time:       2022-08-11T07:11:03Z
    Message:                Chart fetch was successful for Helm release 'monitoringaddon' in 'lamm'.
    Reason:                 ChartFetched
    Status:                 True
    Type:                   ChartFetched
    Last Transition Time:   2022-08-17T07:22:13Z
    Last Update Time:       2022-08-17T07:22:13Z
    Message:                Installation or upgrade succeeded for Helm release 'monitoringaddon' in 'lamm'.
    Reason:                 Deployed
    Status:                 True
    Type:                   Deployed
    Last Transition Time:   2022-08-11T07:11:32Z
    Last Update Time:       2022-08-17T07:22:13Z
    Message:                Release was successful for Helm release 'monitoringaddon' in 'lamm'.
    Reason:                 Succeeded
    Status:                 True
    Type:                   Released
  Last Attempted Revision:  39.5.0
  Observed Generation:      1
  Phase:                    Succeeded
  Release Name:             monitoringaddon
  Release Status:           deployed
  Revision:                 39.5.0
Events:
  Type    Reason         Age                     From           Message
  ----    ------         ----                    ----           -------
  Normal  ReleaseSynced  6m57s (x419 over 2d7h)  helm-operator  managed release 'monitoringaddon' in namespace 'lamm' synchronized`

Expected behavior

Monitoringaddon should only be deployed when there is any change.

Logs

ts=2022-08-17T07:30:04.359897699Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for Ingress \"monitoringaddon-grafana\"" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:04.37198082Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for Ingress \"monitoringaddon-prometheus-alertmanager\"" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:04.387080699Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for Ingress \"monitoringaddon-prometheus-prometheus\"" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:05.376122429Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ServiceAccount" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:05.384348658Z caller=helm.go:69 component=helm version=v3 info="serviceaccounts \"monitoringaddon-prometheus-admission\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:06.231529351Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:06.243600849Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ClusterRole" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:06.248345509Z caller=helm.go:69 component=helm version=v3 info="clusterroles.rbac.authorization.k8s.io \"monitoringaddon-prometheus-admission\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.079869471Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.095985215Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ClusterRoleBinding" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.100762571Z caller=helm.go:69 component=helm version=v3 info="clusterrolebindings.rbac.authorization.k8s.io \"monitoringaddon-prometheus-admission\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.945807948Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.957947912Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" Role" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:07.965741404Z caller=helm.go:69 component=helm version=v3 info="roles.rbac.authorization.k8s.io \"monitoringaddon-prometheus-admission\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:08.844221082Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:08.858515314Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" RoleBinding" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:08.863244981Z caller=helm.go:69 component=helm version=v3 info="rolebindings.rbac.authorization.k8s.io \"monitoringaddon-prometheus-admission\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:09.678628936Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:09.689869718Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission-patch\" Job" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:09.695901751Z caller=helm.go:69 component=helm version=v3 info="jobs.batch \"monitoringaddon-prometheus-admission-patch\" not found" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.510300093Z caller=helm.go:69 component=helm version=v3 info="creating 1 resource(s)" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.521815485Z caller=helm.go:69 component=helm version=v3 info="Watching for changes to Job monitoringaddon-prometheus-admission-patch with timeout of 5m0s" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.524689836Z caller=helm.go:69 component=helm version=v3 info="Add/Modify event for monitoringaddon-prometheus-admission-patch: ADDED" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.524712699Z caller=helm.go:69 component=helm version=v3 info="monitoringaddon-prometheus-admission-patch: Jobs active: 0, jobs failed: 0, jobs succeeded: 0" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.558093423Z caller=helm.go:69 component=helm version=v3 info="Add/Modify event for monitoringaddon-prometheus-admission-patch: MODIFIED" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:10.558131614Z caller=helm.go:69 component=helm version=v3 info="monitoringaddon-prometheus-admission-patch: Jobs active: 1, jobs failed: 0, jobs succeeded: 0" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.319712637Z caller=helm.go:69 component=helm version=v3 info="Add/Modify event for monitoringaddon-prometheus-admission-patch: MODIFIED" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.322212971Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ServiceAccount" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.335082876Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ClusterRole" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.347867939Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" ClusterRoleBinding" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.357674108Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" Role" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.368376867Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission\" RoleBinding" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:12.378087402Z caller=helm.go:69 component=helm version=v3 info="Starting delete for \"monitoringaddon-prometheus-admission-patch\" Job" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:13.180573638Z caller=helm.go:69 component=helm version=v3 info="updating status for upgraded release for monitoringaddon" targetNamespace=lamm release=monitoringaddon ts=2022-08-17T07:30:14.091783856Z caller=release.go:364 component=release release=monitoringaddon targetNamespace=lamm resource=lamm:helmrelease/monitoringaddon helmVersion=v3 info="upgrade succeeded" revision=39.5.0 phase=upgrade

Additional context

• Helm Operator version: 1.4.2
• Kubernetes version: v1.22.6(EKS)

[kingdonb]

Hello, thanks for the report. We have released a new Helm Operator 1.4.4 this week, it may solve your issue.

We cannot support Helm Operator as the maintenance has been declared EOL for over a year, I recommend migrating to Helm Controller and Flux v2. If there is a bug that has crept in and it was not caught by the E2E, I apologize. If there's a previous release of Helm Operator which does not have this bug, or if the later version does not have this issue, welcome to reopen, but as the releases passed E2E testing, the best advice I can give is to migrate to Flux v2 which has a vibrant support and is in active development.

The Helm Controller receives a lot of love from Flux developers and is receiving an overhaul as well in the present/near future releases. It is already several times more efficient and powerful all around than Helm Operator. I can only recommend it very highly for Helm users all around.

In any case, please let us know if there is anything we can do to make your migration to Flux v2 a smoother one!

Thank you for your report, sorry that you are having this issue.

[kingdonb]

Sorry, I have just closed all reports that were over a year old and this one got caught in the crossfire. (Reopened)

I did also find these similar reports when I was doing that:

• Constantly creating new releases for charts even when no changes #457
• A new release is created when using semver chart versions #468
• A new release is created in every cycle when using semver chart versions #469

[kingdonb]

I've reopened #457 for visibility, since there are fresh reports it should probably remain open until we can resolve it or until the repo is archived.

And since that one has a longer discussion and has had multiple maintainers weigh in there already, I'm going to close this one as a duplicate. Welcome to add follow-up comments here even though it is closed, (that would be more preferable than to alert everyone on the thread in #457 by moving the discussion over there, even though this is a duplicate.)

[kingdonb]

I just noticed that you mentioned, you had the same issue in Helm Controller, but the Helm Controller version you have mentioned is very old:

Helm Controller Version: helm-controller:v0.14.1
Current HC Version: helm-controller:v0.23.1

If you do find this issue is recurring on current versions of Helm Controller, please update in a new issue on that repo.

Issues on this repo are unlikely to get attention from the main Helm controller maintainers. If a fix is still needed or possible, that will be where it lands.

There is also a "major" v0.24 release of Helm Controller due out soon, either the next coming Flux v0.x release, or the one that follows. So if your issue really hasn't been resolved in the current release, there is a decent chance that it will be fixed by the rewrite that's ongoing, which should be landing very soon (we think probably weeks at most)