Could not create directory '/home/argocd/.ssh' when using .repo.GetCommitMetadata in argocd notifications

[lukpep]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
It's related to argocd notifications but the errors are coming from argocd-repo-server

I'm unable to produce notifications using repo calls (like .repo.GetCommitMetadata e.g) - it worked a single time (and once only) and since then every other try ends with an error:

argocd-notifications-controller-7df6c975f5-qlcm7 argocd-notifications-controller time="2021-11-25T19:58:17Z" level=error msg="Failed to notify recipient {slack argocd-all-alerts} defined in resource argocd/argo-config: template: app-deployed:18:18: executing \"app-deployed\" at <call .repo.GetCommitMetadata .app.status.sync.revision>: error calling call: rpc error: code = Internal desc = Failed to fetch 54bbb5a4c68cabd9f5162179a5512edbb241affb: git fetch origin --tags --forcefailed exit status 128: Could not create directory '/home/argocd/.ssh' (Read-only file system).\r\nFailed to add the ECDSA host key for IP address '51.68.x.xx' to the list of known hosts (/home/argocd/.ssh/known_hosts).\r\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." resource=argocd/argo-config

I've added our private gitlab instance ssh certs by:
ssh-keyscan gitlab.xxx.xxx.com | argocd cert add-ssh --batch

and cert are there which I can view by:
argocd cert list --cert-type ssh

Before I did that (add certs) another error was present - about the host being not known. After certs were added I was able to successfully produce notification using .repo calls once (pic) and that's it - every other try ends with the above error.

Error in argocd repo server:

time="2021-11-26T13:28:08Z" level=info msg="revision metadata cache miss: [email protected]:iops/cloud-infrastructure/k8s-clusters-bootstrapping.git/97c4b66b499f975caaba899664f4699b892503a1"
time="2021-11-26T13:28:08Z" level=error msg="`git fetch origin --tags --force` failed exit status 128: Could not create directory '/home/argocd/.ssh' (Read-only file system).\r\nFailed to add the ECDSA host key for IP address '51.68.x.x' to the list of known hosts (/home/argocd/.ssh/known_hosts).\r\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=EUGnD

At the same time application sync is working correctly.

To Reproduce

try to use .repo.GetCommitMetadata call in argocd notifications

Expected behaviour

Ability to use repo calls every time

argocd-server: v2.1.7+a408e29
  BuildDate: 2021-11-17T22:10:27Z
  GitCommit: a408e299ffa743213df3aa9135bf7945644ec936
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.6.0+g7f2df64
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

[lukpep]

not sure is is more connected to argocd or to argocd notifications so I've also created: argoproj-labs/argocd-notifications#366

[thuandt]

Ref: argoproj-labs/argocd-notifications#347

[lukpep]

solved by using 1.2.1 version of argocd-notifications <3