Server-side argocd app diff giving unexpected executable argocd-k8s-auth not found

[purajit]

We've been successfully running ArgoCD for our entire deployment pipeline for a year now; we've also been using argocd app diff in our CI to generate the expected diffs. I've been using argocd app list, argocd appset create, and other commands without issues, so our basic setup seems to be fine.

Recently, I tried moving the diffs to --server-side-generate as suggested, and haven't been able to get it working. I haven't found any pertinent discussions around this, except #2347 and the resulting docs, which have already been followed and put into place; also the error seems unrelated to that discussion.

> argocd app diff <appname> --local <helm chart location> --server-side-generate

ERRO[0005] finished streaming call with code Unknown     error="error getting server version: Get \"<>.eks.amazonaws.com/version?timeout=32s\": getting credentials: exec: executable argocd-k8s-auth not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins" grpc.code=Unknown grpc.method=GetManifestsWithFiles grpc.service=application.ApplicationService grpc.start_time="2023-07-06T18:40:19-07:00" grpc.time_ms=7.304 span.kind=server system=grpc
FATA[0005] rpc error: code = Unknown desc = error getting server version: Get "<>.eks.amazonaws.com/version?timeout=32s": getting credentials: exec: executable argocd-k8s-auth not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

The executable certainly exists:

> kubectl get pod \
  -l app.kubernetes.io/name=argocd-server \
  -n argocd \
  -o custom-columns=name:metadata.name --no-headers \
  | xargs -I{} kubectl exec {} -- which argocd-k8s-auth

/usr/local/bin/argocd-k8s-auth
/usr/local/bin/argocd-k8s-auth

There are no additional logs either server- or client-side. What am I missing?

Version

argocd: v2.6.7+5bcd846
  BuildDate: 2023-03-23T15:21:19Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.6.7+5bcd846
  BuildDate: 2023-03-23T15:21:19Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.9.0+g7ceeda6
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

[liranZiv27]

Is there any update here?

[crenshaw-dev]

At a glance, looks like a bug in argocd app diff. I think it's trying to auth to the external cluster from the local CLI instead of from the API server. Can you open a bug with reproduction steps?

[liranZiv27]

It happens for argocd app set as well. I'll write it down

[liranZiv27]

@crenshaw-dev #19640