login denied while connecting vault using ArgoCD Vault Plugin

[azeem225]

While connecting to vault using ArgoCD Vault Plugin getting login denied at F5 APM level

I was trying to connect vault cluster using argocd vault plugin for rendering secrets while deployment which are stored in hashicorp vault, i am getting blocked at F5 APM.

Have followed this below official documentation for configure and setup ArgoCD Vault Plugin.
https://argocd-vault-plugin.readthedocs.io/en/stable/

Error message and detailed steps followed for implementation are being noted down below, I would request please address this if someone has come across similar kind of issue.

Error

Unable to create application: application spec for testdemoapp is invalid: InvalidSpecError: Unable to generate manifests in .: rpc error: code = Unknown desc = plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: argocd-vault-plugin generate ./ failed exit status 1: Error: Error making API request. URL: PUT https://kubernetes.default.svc.cluster.local/v1/auth/kubernates/login Code: 403. Raw Message: Forbidden by f5 APM (login denied)
session reference: 25cb39ba31e3c58d631aa21e28ed7f44 Usage: argocd-vault-plugin generate [flags] Flags: -c, --config-path string path to a file containing Vault configuration (YAML, JSON, envfile) to use -h, --help help for generate -s, --secret-name string name of a Kubernetes Secret in the argocd namespace containing Vault configuration data in the argocd namespace of your ArgoCD host (Only available when used in ArgoCD). The namespace can be overridden by using the format : --verbose-sensitive-output enable verbose mode for detailed info to help with debugging. Includes sensitive data (credentials), logged to stderr

Have followed below steps for implementation:
Environment Variables: have configured below env variables of vault as OpenShift secrets and referred them while deploying argocd instance as documented below.

AVP_K8S_ROLE=<role>
AVP_TYPE=vault
AVP_ENV_VAULT_ADDR=<Vault Address>
AVP_AUTH_TYPE=k8s
AVP_K8S_MOUNT_PATH=<Vault mount path>
VAULT_CACERT=/vault/tls/client/ca.crt
VAULT_CAPATH=/vault/tls/client/
VAULT_CLIENT_CERT=/vault/tls/client/client.crt
VAULT_CLIENT_KEY=/vault/tls/client/client.key
VAULT_NAMESPACE=<vault namespace>

Have created and build argocd custom image with vault plugin and used argocd operator for deploying argocd instance with plugin enabled as sidescar container along with argocd-repo-server shown below

Dockerfile:

FROM argoproj/argocd:v2.6.15
USER root

RUN apt-get clean -y
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
COPY ca.crt client.crt client.key /vault/tls/client/

COPY argocd-vault-plugin_1.17.0_linux_amd64 argocd-vault-plugin 
RUN chmod +x argocd-vault-plugin
RUN mv argocd-vault-plugin /usr/local/bin

USER 999

Operator deployment file

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: todoappargocd
  namespace: testing
spec:
  server:
    autoscale:
      enabled: false
    grpc:
      ingress:
        enabled: false
    ingress:
      enabled: false
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi
    route:
      enabled: true
    service:
      type: LoadBalancer
  grafana:
    enabled: false
    ingress:
      enabled: false
    route:
      enabled: false
  monitoring:
    enabled: true
  notifications:
    enabled: false
  prometheus:
    enabled: false
    ingress:
      enabled: true
    route:
      enabled: true
  initialSSHKnownHosts: {}
  applicationInstanceLabelKey: argocd.argoproj.io/instance
  applicationSet:
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi
    webhookServer:
      ingress:
        enabled: false
      route:
        enabled: false
  repo:
    mountsatoken: true
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi
    serviceaccount: todo-app-sa
    sidecarContainers:
      - command:
          - /var/run/argocd/argocd-cmp-server
        envFrom:
          - secretRef:
              name: argocd-vault-plugin-credentials
        image: 'argocd-vault-plugin:31'
        name: cmp-plugin
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 512Mi
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            name: cmp-plugin
            subPath: avp.yaml
          - mountPath: /vault/tls/client
            name: cert
            readOnly: true
    volumes:
      - configMap:
          name: cmp-plugin
        name: cmp-plugin
      - name: cert
        secret:
          secretName: plugin-client-certs
  ha:
    enabled: false
    resources:
      limits:
        cpu: 250m
        memory: 256Mi
      requests:
        cpu: 150m
        memory: 128Mi
  tls:
    ca: {}
  redis:
    resources:
      limits:
        cpu: 250m
        memory: 256Mi
      requests:
        cpu: 150m
        memory: 128Mi
  controller:
    processors: {}
    resources:
      limits:
        cpu: 512m
        memory: 3Gi
      requests:
        cpu: 250m
        memory: 1536Mi
    sharding: {}

Configmap:

Configmap for generating plugin.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  avp.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."
      generate:
        command:
          - argocd-vault-plugin
          - generate
          - "."
      lockRepo: false
---