Vault agent memory leak when template can't write to file

[sgorbaty]

Describe the bug

Unbounded memory growth when DAC unix permissions prevent file to be written to disk.

Noticeable CPU usage growth:

To Reproduce
Steps to reproduce the behavior:

1. Configure vault agent with

 "template": [
    {
      "source": "/etc/vault_agent/vault.ctmpl",
      "destination": "/etc/vault_agent/certificate.pem",
      "perms": "0640",
      "error_on_missing_key": true
    }
  ],

2. vault.ctmpl is this:

{{- with pkiCert "pki/internal_certs/issue/site-ec" "common_name=***-***.internal.net" -}}
{{ .Cert }}{{ .CA }}
{{ if .Key }}
  {{ .Key | writeToFile "/etc/vault_agent/private_key.pem" "" "" "0600" }}
{{ end }}
{{- end -}}

3. /etc/vault_agent/private_key.pem is actually configured as chmod 0400.

Expected behavior
Failure to write to disk and no memory leak or CPU spike.

Environment:

• Vault Server Version (retrieve with vault status):
  Vault v1.17.5
• Vault CLI Version (retrieve with vault version):
  Vault v1.17.5
• Server Operating System/Architecture:
  unix

Vault server configuration file(s):

# renew/rotate the cert for associated ica
path "pki/${function}/issue/${pkirole}" {
  capabilities = ["create", "update"]
}

Additional context
Error:

execute: template: :4:12: executing \"\" at <writeToFile \"/etc/vault_agent/private_key.pem\" \"\" \"\" \"0600\">: error calling writeToFile: open /etc/vault_agent/private_key.pem: permission denied"

[bosouza]

Hi, I wasn't able to reproduce this memory leak behavior locally on Ubuntu 22.04.5. On my machine I just get the same error reported in the issue (execute: template: :4:12: executing \"\" at <writeToFile \"key.pem\" \"\" \"\" \"0600\">: error calling writeToFile: open key.pem: permission denied") but no memory increase at all (monitored it over 1 h with pidstat -r -p <agent-pid>> 5 > memory_log.txt). Can you share more about your system, like which OS you're using? How is the agent being configured beyond the template field? Do you still see the issue using this same script I used to try to reproduce it? Any chance you have GOGC=off?

#!/bin/bash

vault policy write pki-policy - <<EOF
path "pki/issue/local-site" {
  capabilities = ["update"]
}
EOF
vault auth enable approle
vault write auth/approle/role/agent-role policies="pki-policy"

ROLE_ID=$(vault read auth/approle/role/agent-role/role-id -format=json | jq -r '.data.role_id')
SECRET_ID=$(vault write -f auth/approle/role/agent-role/secret-id -format=json | jq -r '.data.secret_id')

echo ${ROLE_ID?} > ./roleid
echo ${SECRET_ID?} > ./secretid

vault write auth/approle/login role_id=${ROLE_ID?} secret_id=${SECRET_ID?}

vault secrets enable pki
vault write pki/root/generate/internal common_name="example.com" ttl=8760h
vault write pki/config/urls \
    issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
    crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"
vault write pki/roles/local-site allowed_domains="internal.net" allow_subdomains=true max_ttl="72h"

touch key.pem
chmod 0400 key.pem

# run agent with inline config
vault agent -config <(cat <<EOF
auto_auth = {
  method = {
    config = {
      role_id_file_path = "./roleid"
      secret_id_file_path = "./secretid"
    }
    type = "approle"
  }
  sink = {
    config = {
      path = "/tmp/file-foo"
    }
    type = "file"
  }
}
template = {
  contents = <<EOT
{{- with pkiCert "pki/issue/local-site" "common_name=example.internal.net" -}}
{{ .Cert }}{{ .CA }}
{{ if .Key }}
  {{ .Key | writeToFile "key.pem" "" "" "0600" }}
{{ end }}
{{- end -}}
EOT
  destination = "./result.txt"
}
EOF
)