Helm Operator Upgrades Release Every Single Iteration

[kferrone]

Describe the bug

I launched a Helm Release for aws-efs-csi driver chart and helmOperator will constantly upgrade it. I was up to 400 revisions before I noticed the issue. No other chart will do this. Why would HelmOperator think it needs to update the release?

To Reproduce

Here is the exact verbatim HelmRelease I used:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: aws-efs-csi-driver
  namespace: kube-system
spec:
  releaseName: aws-efs-csi-driver
  chart:
    repository: https://kubernetes-sigs.github.io/aws-efs-csi-driver/
    name: aws-efs-csi-driver
    version: 0.1.0
  values:
    image:
      tag: latest

Expected behavior

Don't upgrade and revise when nothing has changed.

Logs

ts=2020-09-16T19:07:11.238214142Z caller=release.go:309 component=release release=aws-efs-csi-driver targetNamespace=kube-system resource=kube-system:helmrelease/aws-efs-csi-driver helmVersion=v3 info="upgrade succeeded" revision=0.1.0 phase=upgrade
ts=2020-09-16T19:10:01.537750201Z caller=release.go:75 component=release release=aws-efs-csi-driver targetNamespace=kube-system resource=kube-system:helmrelease/aws-efs-csi-driver helmVersion=v3 info="starting sync run"
ts=2020-09-16T19:10:03.664056691Z caller=release.go:289 component=release release=aws-efs-csi-driver targetNamespace=kube-system resource=kube-system:helmrelease/aws-efs-csi-driver helmVersion=v3 info="running upgrade" action=upgrade
ts=2020-09-16T19:10:03.720468409Z caller=helm.go:69 component=helm version=v3 info="preparing upgrade for aws-efs-csi-driver" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:03.741358323Z caller=helm.go:69 component=helm version=v3 info="resetting values to the chart's original version" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:05.330241295Z caller=helm.go:69 component=helm version=v3 info="performing update for aws-efs-csi-driver" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.27330204Z caller=helm.go:69 component=helm version=v3 info="creating upgraded release for aws-efs-csi-driver" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.307604087Z caller=helm.go:69 component=helm version=v3 info="checking 2 resources for changes" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.334882147Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for DaemonSet \"efs-csi-node\"" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.396260229Z caller=helm.go:69 component=helm version=v3 info="Looks like there are no changes for CSIDriver \"efs.csi.aws.com\"" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.430817078Z caller=helm.go:69 component=helm version=v3 info="updating status for upgraded release for aws-efs-csi-driver" targetNamespace=kube-system release=aws-efs-csi-driver
ts=2020-09-16T19:10:06.517290787Z caller=release.go:309 component=release release=aws-efs-csi-driver targetNamespace=kube-system resource=kube-system:helmrelease/aws-efs-csi-driver helmVersion=v3 info="upgrade succeeded" revision=0.1.0 phase=upgrade

Additional context

• Helm Operator version: 1.2.0
• Kubernetes version: 1.17

[mnaser]

I believe this is a duplicate of #457

[kferrone]

Ummm, uh oh. Is there any word on that issue yet? What can be done now? What is the ETA for a fix?

[kferrone]

So I think the issue for me was memory. I saw one recommendation about and gave it a shot. I turned it up to 256Mi and so far so good.

[kferrone]

I still have this issue? Everything is exactly the same as another cluster where this issue never happens???

[TwoStone]

We faced this issue too in our clusters. Can we provide any data to tackle down this issue?

[james-gonzalez]

We are seeing a similar issue as well for a CRD.

helm-operator: 1.2.0
k8s: 1.16

[adrian]

As per the discussion on #457, I believe this problem is caused by helm-operator 1.2.0 working with an older version of the HelmRelease CRD. The CRD was updated in 1.2.0 so the helm operator sees a difference on each iteration of the control loop.

Try updating the CRD on your cluster by applying this file, https://github.com/fluxcd/helm-operator/blob/1.2.0/chart/helm-operator/crds/helmrelease.yaml

[kferrone]

I gave Helm Operator only one process in the values.yaml. It seems to work much better now.

[ltagliamonte-dd]

I'm having this problem on helm-operator 1.1.0:

  flux-helm-operator:
    Container ID:  docker://4e4ff10e9adf67cedad9e046e12da6a8508e5bbf98410ae3a330ea0cd28d368c
    Image:         docker.io/fluxcd/helm-operator:1.1.0
    Image ID:      docker-pullable://fluxcd/helm-operator@sha256:78ac7ba7f9d2e21a751ba20793b1e051e3849b94f3df268fed6d40eec57ebe55

helm history -n infra consul-agent
REVISION	UPDATED                 	STATUS    	CHART       	APP VERSION	DESCRIPTION     
2366    	Tue Jun  8 20:14:34 2021	superseded	consul-1.0.0	           	Upgrade complete
2367    	Tue Jun  8 20:17:16 2021	superseded	consul-1.0.0	           	Upgrade complete
2368    	Tue Jun  8 20:20:26 2021	superseded	consul-1.0.0	           	Upgrade complete
2369    	Tue Jun  8 20:23:27 2021	superseded	consul-1.0.0	           	Upgrade complete
2370    	Tue Jun  8 20:26:34 2021	superseded	consul-1.0.0	           	Upgrade complete
2371    	Tue Jun  8 20:29:27 2021	superseded	consul-1.0.0	           	Upgrade complete
2372    	Tue Jun  8 20:32:31 2021	superseded	consul-1.0.0	           	Upgrade complete
2373    	Tue Jun  8 20:35:28 2021	superseded	consul-1.0.0	           	Upgrade complete
2374    	Tue Jun  8 20:38:17 2021	superseded	consul-1.0.0	           	Upgrade complete
2375    	Tue Jun  8 20:41:23 2021	deployed  	consul-1.0.0	           	Upgrade complete

it doesn't happen with releases that ref a git branch:

helm history -n infra cluster-autoscaler
REVISION	UPDATED                 	STATUS    	CHART                   	APP VERSION	DESCRIPTION     
2       	Mon Jun  8 23:10:10 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
3       	Thu Nov 12 17:17:57 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
4       	Thu Nov 12 18:16:09 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
5       	Thu Nov 12 18:46:56 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
6       	Thu Nov 12 18:58:58 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
7       	Mon Nov 16 19:22:38 2020	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
8       	Thu Mar 25 22:50:25 2021	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
9       	Fri Mar 26 17:11:15 2021	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
10      	Tue Apr  6 16:58:05 2021	superseded	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete
11      	Tue Apr 20 03:49:59 2021	deployed  	cluster-autoscaler-0.1.0	0.1.0      	Upgrade complete

is there an official fix for this?

[kingdonb]

Sorry if your issue remains unresolved. The Helm Operator is in maintenance mode, we recommend everybody upgrades to Flux v2 and Helm Controller.

A new release of Helm Operator is out this week, 1.4.4.

We will continue to support Helm Operator in maintenance mode for an indefinite period of time, and eventually archive this repository.

Please be aware that Flux v2 has a vibrant and active developer community who are actively working through minor releases and delivering new features on the way to General Availability for Flux v2.

In the mean time, this repo will still be monitored, but support is basically limited to migration issues only. I will have to close many issues today without reading them all in detail because of time constraints. If your issue is very important, you are welcome to reopen it, but due to staleness of all issues at this point a new report is more likely to be in order. Please open another issue if you have unresolved problems that prevent your migration in the appropriate Flux v2 repo.

Helm Operator releases will continue as possible for a limited time, as a courtesy for those who still cannot migrate yet, but these are strongly not recommended for ongoing production use as our strict adherence to semver backward compatibility guarantees limit many dependencies and we can only upgrade them so far without breaking compatibility. So there are likely known CVEs that cannot be resolved.

We recommend upgrading to Flux v2 which is actively maintained ASAP.

I am going to go ahead and close every issue at once today,
Thanks for participating in Helm Operator and Flux! 💚 💙