ArgoCD vault Plugin: line 5: syntax error near unexpected token `]'

[jauyzed]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After a recent upgrade of ArgoCD operator and argoCD vault plugin on Openshift Cluster 4.16.15, we are seeing this error. Apps on the argocd are going into unknown state throwing an error as below

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `bash -c "helm template \"$ARGOCD_APP_NAME\" -n \"$ARGOCD_APP_NAMESPACE\" -f $(echo \"$ARGOCD_ENV_HELM_VALUES\") . | argocd-vault-plugin generate -"` failed exit status 2: /usr/local/bin/argocd-vault-plugin: line 5: syntax error near unexpected token `]' /usr/local/bin/argocd-vault-plugin: line 5: ` } ]' 

Here is the cmp-plugin.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: cmp-plugin
  namespace: openshift-gitops
  uid: 218a3b09-3a33-4438-9577-b520f1bd8266
  resourceVersion: '516875537'
  creationTimestamp: '2023-09-29T20:51:05Z'
  labels:
    app.kubernetes.io/instance: argocd-gitops-local-cluster
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"avp.yaml":"apiVersion: argoproj.io/v1alpha1\nkind: ConfigManagementPlugin\nmetadata:\n  name: argocd-vault-plugin\nspec:\n  allowConcurrency: true\n  generate:\n    command: [\"bash\", \"-c\"]\n    args: ['helm template \"$ARGOCD_APP_NAME\" -n \"$ARGOCD_APP_NAMESPACE\" -f $(echo \"$ARGOCD_ENV_HELM_VALUES\") . | argocd-vault-plugin generate -']\n  lockRepo: false\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"argocd-gitops-local-cluster"},"name":"cmp-plugin","namespace":"openshift-gitops"}}
  managedFields:
    - manager: argocd-controller
      operation: Update
      apiVersion: v1
      time: '2023-10-02T15:28:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data': {}
        'f:metadata':
          'f:annotations':
            .: {}
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/instance': {}
    - manager: Mozilla
      operation: Update
      apiVersion: v1
      time: '2024-10-29T22:40:35Z'
      fieldsType: FieldsV1
      fieldsV1:
        'f:data':
          'f:avp.yaml': {}
data:
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      generate:
        command: ["bash", "-c"]
        args: ['helm template "$ARGOCD_APP_NAME" -n "$ARGOCD_APP_NAMESPACE" -f $(echo "$ARGOCD_ENV_HELM_VALUES") . | argocd-vault-plugin generate -']
      lockRepo: false

To Reproduce

Using project "openshift-gitops".
➜  ~ oc get pods
NAME                                                          READY   STATUS    RESTARTS   AGE
cluster-6c979d9f99-mdlhk                                      1/1     Running   0          18h
gitops-plugin-655757d6fb-mq8j2                                1/1     Running   0          18h
kam-7fb888bcb-9zn9t                                           1/1     Running   0          18h
openshift-gitops-application-controller-0                     1/1     Running   0          18h
openshift-gitops-applicationset-controller-69d5f85d5d-t4cbp   1/1     Running   0          4d2h
openshift-gitops-dex-server-694889b655-8s87z                  1/1     Running   0          18h
openshift-gitops-redis-7dbc885cb6-zjr5x                       1/1     Running   0          4d2h
openshift-gitops-repo-server-85554f66b-24xdl                  2/2     Running   0          18h
openshift-gitops-server-576df76b48-gvv5t                      1/1     Running   0          18h
➜  ~ oc rsh -c avp openshift-gitops-repo-server-85554f66b-24xdl
sh-4.4$ touch /tmp/vault.yaml
sh-4.4$ argocd-vault-plugin generate /tmp/vault.yaml
/usr/local/bin/argocd-vault-plugin: line 5: syntax error near unexpected token `]'
/usr/local/bin/argocd-vault-plugin: line 5: `  } ]'

Expected behavior

The apps on the argocd UI should sync normally and shouldn't go into unknown state throwing the above error

Screenshots

Version

argocd: v2.8.3+77556d9
  BuildDate: 2023-09-26T08:35:45Z
  GitCommit: 77556d9e64304c27c718bb0794676713628e435e
  GitTreeState: clean
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
  ExtraBuildInfo: {Vendor Information: Red Hat OpenShift GitOps version: v1.10.0}`

Logs

The error is happening while creating the manifest using the argocd-vault-plugin

`bash -c "helm template \"$ARGOCD_APP_NAME\" -n \"$ARGOCD_APP_NAMESPACE\" -f $(echo \"$ARGOCD_ENV_HELM_VALUES\") . | argocd-vault-plugin generate -"`

It seems the error is happening while executing argocd-vault-plugin.

 failed exit status 2: /usr/local/bin/argocd-vault-plugin: line 5: syntax error near unexpected token `]' /usr/local/bin/argocd-vault-plugin: line 5: `

argocd-vault-plugin version
argocd-vault-plugin v1.18.1 (fc452cdd8d4727b412ce3de61ee0416efd75050d) BuildDate: 2024-06-07T03:17:37Z

[pre]

/usr/local/bin/argocd-vault-plugin: line 5: syntax error near unexpected token `]'

This seems to be a shell script there.

What's there on line 5 if you cat /usr/local/bin/argocd-vault-plugin ?

[jauyzed]

@pre i'm inside the repo-server pod. The interesting part is this was working before update/patching of operator and something in the AVP binary might have changed?

[gdsoumya]

If the issue is specific to vault plugin it might be better to raise it here https://github.com/argoproj-labs/argocd-vault-plugin

[jauyzed]

@gdsoumya I have opened there as well: argoproj-labs/argocd-vault-plugin#675
I'm not sure if its just AVP or combination of Argo and AVP