move external secret creation into the main pipeline only

[jstrachan]

its more secure if we don't give read/write/admin access to secrets in the secret store to PR pipelines

so we maybe want to only run jx secret populate in the main pipeline when changes merge. This does mean that we always want to generate in the main branch, even if nothings changed - to ensure we can populate any missing Secret values in the external secret store

I wonder if we could have a locked down Pipeline/Job (defined in the main branch) which would lazily populate external secrets which could be triggered from a PR - but thats locked down so that a PR can't do anything unsecure - but would reuse the git operator Service Account (so that it can create secrets)?

Or maybe we can use a separate operator to populate the secrets without the pipeline needing RBAC to be able to read/write secrets?

[rawlingsj]

At first it seemed a little overkill to have a separate operator but the more I think about it the better it feels TBH.

A simple operator takes minimal resources and this would be the most secure approach + it would speed up the boot job slightly + I have a feeling admins of a locked down cluster would prefer no in cluster SA having write access to secret stores. Instead they would add secrets manually themselves and disable the secret-generator-operator thing.

[rawlingsj]

Being more secure by default would be great

[jstrachan]

yeah. it would be nice to be able to reuse the jx boot pipeline for secret / external secret handling with other last mile solutions to do the kubectl apply too - in case folks are using argo, flux or google config sync in production repositories; we could then make sure all the secrets are populated in a secure way, before merging changes to the main branch for the last mile kubectl apply to kick in