Secret from Vault is double base64 encoded

[devth]

Added log at

secret-manager/pkg/internal/vault/vault.go

Line 150 in 999e7d0

if kvVersion == smv1alpha1.DefaultVaultKVEngineVersion {

	fmt.Printf("readSecret secret data: %+v\n", secretData)

Verified secret is not base64 encoded via logs:

2020-10-12T22:15:30.833546792Z readSecret secret data: map[data:map[another_one:awesome yessssssssss:this is secret yetibot_secret:secret value] metadata:map[created_time:2020-10-09T19:08:57.676679845Z deletion_time: destroyed:false version:3]]

readSecret secret data: map[data:map[embedded-enabled:false observers-enabled:false] metadata:map[created_time:2020-09-21T15:41:58.434072039Z deletion_time: destroyed:false version:1]]

Then looked at the K8S Secret that secret-manager created:

± k get secret yetibot-es -ojson | jq -r '.data'
{
  "another_one": "WVhkbGMyOXRaUT09",
  "embedded-enabled": "Wm1Gc2MyVT0=",
  "observers-enabled": "Wm1Gc2MyVT0=",
  "yessssssssss": "ZEdocGN5QnBjeUJ6WldOeVpYUT0=",
  "yetibot_secret": "YzJWamNtVjBJSFpoYkhWbA=="
}

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]'
WVhkbGMyOXRaUT09

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]' | base64 -d
YXdlc29tZQ==%

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]' | base64 -d | base64
 -d
awesome%

Then I went into controller.go and poked around. Removing base64 encoding like secretDataMap[secretKey] = secretData here fixed my problem:

secret-manager/pkg/controller/externalsecret/controller.go

Lines 179 to 180 in 8ea959e

	base64.StdEncoding.Encode(dstBytes, secretData)
	secretDataMap[secretKey] = dstBytes

I also added logging to verify that the data wasn't already base64 encoded with:

		fmt.Printf("Secret %s = %s\n", secretKey, v)

which prints:

2020-10-12T23:48:31.070304612Z Secret yessssssssss = this is secret
2020-10-12T23:48:31.070308806Z Secret yetibot_secret = secret value
2020-10-12T23:48:31.070313366Z Secret another_one = awesome
2020-10-12T23:48:31.070317958Z Secret embedded-enabled = false
2020-10-12T23:48:31.070322966Z Secret observers-enabled = false

Running off my own docker image based on commit 999e7d0.

Ideas? 🤔

[mcavoyk]

Odd, this section should be the only logic to encode before submitting to the Kubernetes API, but anecdotally I have seen this issue when working on adding GCP SecretManager support, although it appeared that the fix there was that the Google SDK was returning the contents base64'd.

End-to-end testing suite was recently added to this project, but currently only covers AWS SecretManager, we can re-check the AWS secret manager testing is verifying the content is not double base64 encoded and add testing for vault (#20).