- Vault Config Operator
- Authentication Engines
- Policy management
- Secret Engines
- Secret Management
- The common authentication section
- The common connection section
- End to end example
- Contributing a new Vault type
- Initializing the connection to Vault
- Deploying the Operator
- Metrics
- Development
- Building/Pushing the operator image
- Deploy to OLM via bundle
- Integration Test
- Releasing
This operator helps set up Vault Configurations. The main intent is to do so such that subsequently pods can consume the secrets made available. There are two main principles through all of the capabilities of this operator:
- high-fidelity API. The CRD exposed by this operator reflect field by field the Vault APIs. This is because we don't want to make any assumption on the kinds of configuration workflow that user will set up. That being said the Vault API is very extensive and we are starting with enough API coverage to support, we think, some simple and very common configuration workflows.
- attention to security (after all we are integrating with a security tool). To prevent credential leaks we give no permissions to the operator itself against Vault. All APIs exposed by this operator contains enough information to authenticate to Vault using a local service account (local to the namespace where the API exist). In other word for a namespace user to be abel to successfully configure Vault, a service account in that namespace must have been previously given the needed Vault permissions.
Currently this operator covers the following Vault APIs:
- AuthEngineMount Sets up a Vault Authentication Endpoint
- KubernetesAuthEngineConfig Configures a Vault Kubernetes Authentication Endpoint.
- KubernetesAuthEngineRole Configures a Vault Kubernetes Authentication Role
- LDAPAuthEngineConfig Configures a Vault LDAP Authentication Endpoint.
- LDAPAuthEngineGroup Creates or updates Vault LDAP Authentication Engine Group policies.
- JWTOIDCAuthEngineConfig Configures a Vault JWT/OIDC Authentication Endpoint
- JWTOIDCAuthEngineRole Register a role in an Authentication Engine Mount of type JWT/OIDC
- Policy Configures Vault Policies
- PasswordPolicy Configures Vault Password Policies
- SecretEngineMount Configures a Mount point for a SecretEngine
- DatabaseSecretEngineConfig Configures a Database Secret Engine Connection
- DatabaseSecretEngineRole Configures a Database Secret Engine Role
- GitHubSecretEngineConfig Configures a Github Application to produce tokens, see the also the vault-plugin-secrets-github
- GitHubSecretEngineRole Configures a Github Application to produce scoped tokens, see the also the vault-plugin-secrets-github
- PKISecretEngineConfig Configures a PKI Secret Engine
- PKISecretEngineRole Configures a PKI Secret Engine Role
- QuaySecretEngineConfig Configures a Quay server to produce Robot accounts, see the also the vault-plugin-secrets-quay
- QuaySecretEngineRole Configures a Quay server to produce credentials for a Robot account, see the also the vault-plugin-secrets-quay
- QuaySecretEngineStaticRole Configures a Quay server to produce credentials for a Robot account using a fixed username and generated credentials, see the also the vault-plugin-secrets-quay
- RabbitMQSecretEngineConfig Configures a RabbitMQ Secret Engine
- RabbitMQSecretEngineRole Configures a RabbitMQ Secret Engine Role
- RandomSecret Creates a random secret in a vault kv Secret Engine with one password field generated using a PasswordPolicy
- VaultSecret Creates a K8s Secret from one or more Vault Secrets
All APIs share a common authentication section, details can be found here Every time a resource is reconciled a new authentication is performed and a new token is requeued from Vault. For the token will be used for reconcile cycle possible for a few calls and then forgotten. The whole process should normally take a fraction of a second, maybe a few seconds on a very busy pod. It is recommended to dedicate an authentication engine just for the vault-configuration-operator controller and keep the token TTL so 10-20s. This will prevent a proliferation of tokens on the Vault side.
See this section for an example scenario in which this operator could be used.
See this section of the documentation for a details on how to contribute a new type.
At the moment the connection to Vault can be initialized with Vault's standard environment variables. See the OLM documentation on how to pass environment variables via a Subscription. The variable that are read at client initialization are listed here.
For certificates, the recommended approach is to mount the secret or configmap containing the certificate as described here, and the configure the corresponding variables to point at the files location in the mounted path.
Here is an example:
This is a configmap that will be injected with the OpenShift service-ca, you will have different ways of injecting your CA:
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
service.beta.openshift.io/inject-cabundle: "true"
name: ocp-service-ca
data: []
Here is the subscription using that configmap:
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: vault-config-operator
spec:
channel: alpha
installPlanApproval: Automatic
name: vault-config-operator
source: community-operators
sourceNamespace: openshift-marketplace
config:
env:
- name: VAULT_CACERT
value: /vault-ca/ca.crt
volumes:
- name: vault-ca
configMap:
name: ocp-service-ca
volumeMounts:
- mountPath: /vault-ca
name: vault-ca
All APIs share a common connection section. This section can be used to override how connections to Vault are established, essentially changing the above explained behavior. With feature it is potentially possible to connect to multiple vault instance from a single cluster.
Here is an example:
connection:
address: 'https://vault.vault2.svc:8200'
tLSConfig:
cacert: |
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIW+spyyCjRaIwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY3NDc3NDQ3MzAe
Fw0yMzAxMjYyMzA3NTNaFw0yNTAzMjYyMzA3NTRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE2NzQ3NzQ0NzMwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8+qF40DBauBGBgk14iCwzDZ5s9J/0rzRi
x/aLm9DYabtrar1mz6T41UstX4ulgxnU+QrQYN8X1NROzlksC4fFPI3lmmjI95PU
xshv5z3wjKlEq5sI4RqyzAqgww5EMnBz5aaaElFBRZhgFymylc347kR7ZMpx8H9b
LuyAf4zOuherbUO6xYREPKyZwFC1HHZpjnXtrTYNsFvB2GQjnTNWHZdCfdZVHtZx
/vmTlPQGXMnoYBvKH74jxbQVYcu/UGtifj/Q4LpPn05YYvq6xQcYq+H6dLC4gD/W
Z1bFgLZUH5+Xx/mPldrvZBhqINvVo3qR+7U8TiGnoyGUYCdv8k2RAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSaW5QB
6lFBObTYaebBNKMlOXp8VDAfBgNVHSMEGDAWgBSaW5QB6lFBObTYaebBNKMlOXp8
VDANBgkqhkiG9w0BAQsFAAOCAQEAmVnCbvFRs0Z13tgWE3J51dcrzhyuBEbdAWfU
5MHrDKw7Sm3N67UC/Av01G6XDIZh8lW6s+foOES3IXWv/ayH4Sc91U92/K17Cls6
s6ZuC6KDd/X0ez6G8URbvB9gwlrst3WVnD0Qefb36K4YqfRF9j4KTAbDXLZuGpZK
lMk0STHHRm7IeVBnSEqrs8AxLdn+wgnPEdag7Z89BvRhhscQivZwsSBwSegalmdb
UIfKX64wKllQs2sd6eSvDZxDheB1YGM2nYQDYZeWb0gskPsakYUcTfouvXBYrXMf
mHyYxYdDK+drEiV6RB1s8swFPdVaLMe2U0KedY486JFk1aKHqA==
-----END CERTIFICATE-----
This section features the same options that are available via environment variables when using the vault
client. Keep in mind that this configuration override the default explained above, so you need to specify only the field that need to be different.
This is a cluster-level operator that you can deploy in any namespace, vault-config-operator
is recommended.
It is recommended to deploy this operator via OperatorHub
, but you can also deploy it using Helm
.
Arch | Support |
---|---|
amd64 | âś… |
arm64 | âś… |
ppc64le | âś… |
s390x | âś… |
Note: This operator supports being installed disconnected environments
If you want to utilize the Operator Lifecycle Manager (OLM) to install this operator, you can do so in two ways: from the UI or the CLI.
- If you would like to launch this operator from the UI, you'll need to navigate to the OperatorHub tab in the console. Before starting, make sure you've created the namespace that you want to install this operator to with the following:
oc new-project vault-config-operator
- Once there, you can search for this operator by name:
vault config operator
. This will then return an item for our operator and you can select it to get started. Once you've arrived here, you'll be presented with an option to install, which will begin the process. - After clicking the install button, you can then select the namespace that you would like to install this to as well as the installation strategy you would like to proceed with (
Automatic
orManual
). - Once you've made your selection, you can select
Subscribe
and the installation will begin. After a few moments you can go ahead and check your namespace and you should see the operator running.
If you'd like to launch this operator from the command line, you can use the manifests contained in this repository by running the following:
oc new-project vault-config-operator
oc apply -f config/operatorhub -n vault-config-operator
This will create the appropriate OperatorGroup and Subscription and will trigger OLM to launch the operator in the specified namespace.
Here are the instructions to install the latest release with Helm.
oc new-project vault-config-operator
helm repo add vault-config-operator https://redhat-cop.github.io/vault-config-operator
helm repo update
helm install vault-config-operator vault-config-operator/vault-config-operator
This can later be updated with the following commands:
helm repo update
helm upgrade vault-config-operator vault-config-operator/vault-config-operator
If you want the installation to create self-signed certificates for the webhook and the metrics server, set enableCertManager
to true
. The installation will create a self-signed issuer and the required certificates.
If you want to provide your own cert-manager
certificates instead, please create the following two certificates:
- A certificate named
serving-cert
in the operator's target namespace, with secret namewebhook-server-cert
, for the following domain names:vault-config-operator-webhook-service.<target namespace>.svc
vault-config-operator-webhook-service.<target namespace>.svc.cluster.local
- A certificate with a name of your choice in the operator's target namespace, with secret name
vault-config-operator-certs
, for the following domain names:vault-config-operator-controller-manager-metrics-service.<target namespace>.svc
vault-config-operator-controller-manager-metrics-service.<target namespace>.svc.cluster.local
Prometheus compatible metrics are exposed by the Operator and can be integrated into OpenShift's default cluster monitoring. To enable OpenShift cluster monitoring, label the namespace the operator is deployed in with the label openshift.io/cluster-monitoring="true"
.
oc label namespace <namespace> openshift.io/cluster-monitoring="true"
Openshift monitoring...
export operatorNamespace=vault-config-operator
oc label namespace ${operatorNamespace} openshift.io/cluster-monitoring="true"
oc rsh -n openshift-monitoring -c prometheus prometheus-k8s-0 /bin/bash
export operatorNamespace=vault-config-operator
curl -v --data-urlencode "query=controller_runtime_active_workers{namespace=\"${operatorNamespace}\"}" localhost:9090/api/v1/query
exit
in Kubernetes...
See the Test helm chart locally section to run the helmchart test which will also test that metrics work against a k8s kind cluster.
If using vscode, you may need a ./.vscode/settings.json
file in this directory to make gopls happy since build tags are used.
See https://github.com/golang/vscode-go/blob/master/docs/settings.md#buildbuildflags
{
"gopls": {
"ui.completion.usePlaceholders": true,
"build.buildFlags": [
"--tags=integration"
]
},
}
If you don't have a Vault instance available for testing, deploy one with these steps:
helm repo add hashicorp https://helm.releases.hashicorp.com
oc new-project vault
oc adm policy add-role-to-user admin -z vault -n vault
export cluster_base_domain=$(oc get dns cluster -o jsonpath='{.spec.baseDomain}')
helm upgrade vault hashicorp/vault -i --create-namespace -n vault --atomic -f ./config/local-development/vault-values.yaml --set server.route.host=vault-vault.apps.${cluster_base_domain}
Note: you may need to manually remove the
service.beta.openshift.io/serving-cert-secret-name: vault-server-tls
annotation from the Servicevault-internal
and force the recreation of the service serving certificate since the annotation gets applied to bothvault
andvault-internal
Services by the helm chart. The implementation expects the vault.vault.svc certificate. See server-service.yaml and server-headless-service.yaml.
All the configuration made by the operator need to authenticate via a Kubernetes Authentication. So you need a root Kubernetes Authentication mount point and role. The you can create more roles via the operator. If you don't have a root mount point and role, you can create them as follows:
oc new-project vault-admin
export cluster_base_domain=$(oc get dns cluster -o jsonpath='{.spec.baseDomain}')
export VAULT_ADDR=https://vault-vault.apps.${cluster_base_domain}
export VAULT_TOKEN=$(oc get secret vault-init -n vault -o jsonpath='{.data.root_token}' | base64 -d )
# this policy is intentionally broad to allow to test anything in Vault. In a real life scenario this policy would be scoped down.
vault policy write -tls-skip-verify vault-admin ./config/local-development/vault-admin-policy.hcl
vault auth enable -tls-skip-verify kubernetes
vault write -tls-skip-verify auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc:443
vault write -tls-skip-verify auth/kubernetes/role/policy-admin bound_service_account_names=default bound_service_account_namespaces=vault-admin policies=vault-admin ttl=10s
# noticed how we created a 10s TTL for the tokens created by this authentication engine
export accessor=$(vault read -tls-skip-verify -format json sys/auth | jq -r '.data["kubernetes/"].accessor')
verify that kube authentication works:
export token=$(oc create token default -n vault-admin) #requires oc client 4.11.x
vault write -tls-skip-verify auth/kubernetes/login role=policy-admin jwt=${token}
Note: this operator build process is tested with podman, but some of the build files (Makefile specifically) use docker because they are generated automatically by operator-sdk. It is recommended remap the docker command to the podman command.
export repo=raffaelespazzoli #change the organization name to your own and be sure to make the repo public
docker login quay.io/$repo
oc new-project vault-config-operator
oc project vault-config-operator
tilt up
Policy
envsubst < ./test/database-engine-admin-policy.yaml | oc apply -f - -n vault-admin
Vault Role
oc new-project test-vault-config-operator
oc label namespace test-vault-config-operator database-engine-admin=true
oc apply -f ./test/database-engine-admin-role.yaml -n vault-admin
Secret Engine Mount
oc apply -f ./test/database-secret-engine.yaml -n test-vault-config-operator
Database secret engine connection. This will deploy a postgresql database to connect to
oc create secret generic postgresql-admin-password --from-literal=postgres-password=changeit -n test-vault-config-operator
helm repo add bitnami https://charts.bitnami.com/bitnami
helm upgrade helm upgrade my-postgresql-database bitnami/postgresql -i --create-namespace -n test-vault-config-operator -f ./docs/examples/postgresql/postgresql-values.yaml --set primary.podSecurityContext.fsGroup=null,primary.podSecurityContext.seccompProfile.type=RuntimeDefault,primary.containerSecurityContext.runAsUser=null,primary.containerSecurityContext.allowPrivilegeEscalation=false,primary.containerSecurityContext.runAsNonRoot=true,primary.containerSecurityContext.seccompProfile.type=RuntimeDefault,volumePermissions.enabled=false,shmVolume.enabled=false #For OpenShift 4.11 and higher, let set OpenShift the runAsUser and fsGroup automatically. Configure the pod and container security context to restrictive defaults and disable the volume permissions setup. See https://github.com/bitnami/charts/tree/main/bitnami/postgresql
oc adm policy add-scc-to-user anyuid -z default -n test-vault-config-operator
oc apply -f ./test/database-engine-config.yaml -n test-vault-config-operator
Database Secret engine role
oc apply -f ./test/database-engine-read-only-role.yaml -n test-vault-config-operator
Database Secret engine static role
export POSTGRES_PASSWORD=$(kubectl get secret --namespace test-vault-config-operator postgresql-admin-password -o jsonpath="{.data.postgres-password}" | base64 -d)
oc rsh -c postgresql -n test-vault-config-operator my-postgresql-database-0
/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash
psql --host 127.0.0.1 -U postgres -d postgres -p 5432
type $POSTGRES_PASSWORD here
CREATE ROLE helloworld;
exit
exit
exit
oc apply -f ./test/database-engine-read-only-static-role.yaml -n test-vault-config-operator
Database root password rotation. Note we are going to overwrite the previous definition
oc apply -f ./test/database-engine-config-password-rotation.yaml -n test-vault-config-operator
export POSTGRES_PASSWORD=$(kubectl get secret --namespace test-vault-config-operator postgresql-admin-password -o jsonpath="{.data.postgres-password}" | base64 -d)
oc rsh -c postgresql -n test-vault-config-operator my-postgresql-database-0
/opt/bitnami/scripts/postgresql/entrypoint.sh /bin/bash
psql --host 127.0.0.1 -U postgres -d postgres -p 5432
type $POSTGRES_PASSWORD here
this should fail now
exit
exit
exit
RandomSecret
oc apply -f ./test/password-policy.yaml -n vault-admin
envsubst < ./test/kv-engine-admin-policy.yaml | oc apply -f - -n vault-admin
envsubst < ./test/secret-writer-policy.yaml | oc apply -f - -n vault-admin
oc apply -f ./test/kv-engine-admin-role.yaml -n vault-admin
oc apply -f ./test/secret-writer-role.yaml -n vault-admin
oc apply -f ./test/kv-secret-engine.yaml -n test-vault-config-operator
oc apply -f ./test/random-secret.yaml -n test-vault-config-operator
VaultSecret
Note: you must run the previous tests
oc apply -f ./test/vaultsecret/kubernetesauthenginerole-secret-reader.yaml -n vault-admin
envsubst < ./test/vaultsecret/policy-secret-reader.yaml | oc apply -f - -n vault-admin
oc apply -f ./test/vaultsecret/randomsecret-another-password.yaml -n test-vault-config-operator
oc apply -f ./test/vaultsecret/vaultsecret-randomsecret.yaml -n test-vault-config-operator
oc apply -f ./test/vaultsecret/vaultsecret-dynamicsecret.yaml -n test-vault-config-operator
# test a pullsecret... Create the secret in test-vault-config-operator/kv/pullsecret using the Vault UI first
oc apply -f ./test/vaultsecret/vaultsecret-pullsecret.yaml -n test-vault-config-operator
VaultSecret & RandomSecret using KV Secret Engine V2
#Random Secret using v2 steps
oc apply -f ./test/randomsecret/v2/00-passwordpolicy-simple-password-policy-v2.yaml -n vault-admin
envsubst < ./test/randomsecret/v2/01-policy-kv-engine-admin-v2.yaml | oc apply -f - -n vault-admin
envsubst < ./test/randomsecret/v2/04-policy-secret-writer-v2.yaml | oc apply -f - -n vault-admin
envsubst < ./test/vaultsecret/v2/00-policy-secret-reader-v2.yaml | oc apply -f - -n vault-admin
oc apply -f ./test/randomsecret/v2/02-kubernetesauthenginerole-kv-engine-admin-v2.yaml -n vault-admin
oc apply -f ./test/randomsecret/v2/05-kubernetesauthenginerole-secret-writer-v2.yaml -n vault-admin
oc apply -f ./test/vaultsecret/v2/00-kubernetesauthenginerole-secret-reader-v2.yaml -n vault-admin
oc apply -f test/randomsecret/v2/03-secretenginemount-kv-v2.yaml -n test-vault-config-operator
oc apply -f ./test/randomsecret/v2/06-randomsecret-randomsecret-password-v2.yaml -n test-vault-config-operator
oc apply -f ./test/randomsecret/v2/07-randomsecret-randomsecret-another-password-v2.yaml -n test-vault-config-operator
#VaultSecret reading RandomSecret v2 steps
oc apply -f ./test/vaultsecret/v2/07-vaultsecret-randomsecret-v2.yaml -n test-vault-config-operator
Kube auth engine mount and config and role
oc apply -f ./test/kube-auth-engine-mount.yaml -n vault-admin
oc apply -f ./test/kube-auth-engine-config.yaml -n vault-admin
oc apply -f ./test/kube-auth-engine-role.yaml -n vault-admin
Github secret engine
create a github application following the instructions here. save the ssh key in a file called: ./test/vault-secret-engine.private-key.pem and the application id
export application_id=163698 #replace with your own
export org_name=raf-backstage-demo
export ssh_key=$(cat ./test/vault-secret-engine.private-key.pem | base64 -w 0)
envsubst < ./test/github-secret-engine-config-secret-template.yaml | oc apply -f - -n vault-admin
oc apply -f ./test/github-secret-engine-mount.yaml -n vault-admin
envsubst < ./test/github-secret-engine-config.yaml | oc apply -f - -n vault-admin
#vault read -tls-skip-verify github/raf-backstage-demo/token
vault write -tls-skip-verify github/raf-backstage-demo/token org_name=${org_name}
envsubst < ./test/github-secret-engine-role.yaml | oc apply -f - -n vault-admin
vault read -tls-skip-verify github/raf-backstage-demo/token/one-repo-only
make helmchart-test
Since this will run a kind instance, you may then test creating CRs manually...
export VAULT_TOKEN=$(kubectl get secret vault-init -n vault -o jsonpath='{.data.root_token}' | base64 -d)
export VAULT_ADDR="http://localhost"
export accessor=$(vault read -tls-skip-verify -format json sys/auth | jq -r '.data["kubernetes/"].accessor')
kubectl create namespace vault-admin
kubectl create namespace test-vault-config-operator
Next, run through the Test Manually steps.
Define an image and tag. For example...
export imageRepository="quay.io/redhat-cop/vault-config-operator"
export imageTag="$(git -c 'versionsort.suffix=-' ls-remote --exit-code --refs --sort='version:refname' --tags https://github.com/redhat-cop/vault-config-operator.git '*.*.*' | tail --lines=1 | cut --delimiter='/' --fields=3)"
Deploy chart...
make helmchart IMG=${imageRepository} VERSION=${imageTag}
helm upgrade -i vault-config-operator-local charts/vault-config-operator -n vault-config-operator-local --create-namespace
Delete...
helm delete vault-config-operator-local -n vault-config-operator-local
kubectl delete -f charts/vault-config-operator/crds/crds.yaml
export repo=raffaelespazzoli #replace with yours
docker login quay.io/$repo
make docker-build IMG=quay.io/$repo/vault-config-operator:latest
make docker-push IMG=quay.io/$repo/vault-config-operator:latest
make manifests
make bundle IMG=quay.io/$repo/vault-config-operator:latest
operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.25
make bundle-build BUNDLE_IMG=quay.io/$repo/vault-config-operator-bundle:latest
docker push quay.io/$repo/vault-config-operator-bundle:latest
operator-sdk bundle validate quay.io/$repo/vault-config-operator-bundle:latest suite=operatorframework --optional-values=k8s-version=1.25
oc new-project vault-config-operator
oc label namespace vault-config-operator openshift.io/cluster-monitoring="true"
operator-sdk cleanup vault-config-operator -n vault-config-operator
operator-sdk run bundle --install-mode AllNamespaces -n vault-config-operator quay.io/$repo/vault-config-operator-bundle:latest
make integration
git tag -a "<tagname>" -m "<commit message>"
git push upstream <tagname>
If you need to remove a release:
git tag -d <tagname>
git push upstream --delete <tagname>
If you need to "move" a release to the current main
git tag -f <tagname>
git push upstream -f <tagname>
operator-sdk cleanup vault-config-operator -n vault-config-operator
oc delete operatorgroup operator-sdk-og
oc delete catalogsource vault-config-operator-catalog