Could not create directory '/home/argocd/.ssh' when using .repo.GetCommitMetadata

[lukpep]

Summary

I'm unable to produce notifications using repo calls (like .repo.GetCommitMetadata e.g) - it worked single time (and once only) and since then every other try ends with error:

argocd-notifications-controller-7df6c975f5-qlcm7 argocd-notifications-controller time="2021-11-25T19:58:17Z" level=error msg="Failed to notify recipient {slack argocd-all-alerts} defined in resource argocd/argo-config: template: app-deployed:18:18: executing \"app-deployed\" at <call .repo.GetCommitMetadata .app.status.sync.revision>: error calling call: rpc error: code = Internal desc = Failed to fetch 54bbb5a4c68cabd9f5162179a5512edbb241affb: git fetch origin --tags --forcefailed exit status 128: Could not create directory '/home/argocd/.ssh' (Read-only file system).\r\nFailed to add the ECDSA host key for IP address '51.68.x.xx' to the list of known hosts (/home/argocd/.ssh/known_hosts).\r\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." resource=argocd/argo-config

I've added our private gitlab instance ssh certs by:
ssh-keyscan gitlab.xxx.xxx.com | argocd cert add-ssh --batch

and cert are there which I can view by:
argocd cert list --cert-type ssh

Before I did that (add certs) other error was present - about host being not know. After certs were added I was able to successfully produce notification using .repo calls once (pic) and that's it - every other try ends with above error.

Diagnostics

What Kubernetes provider are you using?
GCP GKE - v1.21.5-gke.1802

What version of Argo CD and Argo CD Notifications are you running?
Argo CD v2.1.7+a408e2
Argo CD notifications: v.1.2.0

# Paste the logs from the notification controller
already pasted above

---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[thuandt]

Ref: #347

[pasha-codefresh]

Hey @lukpep . Could you please share with me if you have the ARGOCD_SSH_DATA_PATH env variable defined inside your repo server deployment?

[lukpep]

Just checked - it's not set

[pasha-codefresh]

@lukpep i just tried both versions (2.1.7 and 2.2.0 rc ) and it is working for me. Maybe we can have pairing session in zoom so I can help you debug this issue ( at least try ).

[lukpep]

sure - thx - I've just ended a quite long journey and I'm feeling a little under the weather - will get back to You (any preferred method? (CNCF slack maybe?) in the upcoming days.

[pasha-codefresh]

slack is good. I hope I will repro and fix it in the coming days. (Slack is "Pavel Kostohrys" )

[lukpep]

solved by using 1.2.1 version of argocd-notifications <3