Update fails when SSL cert in secret is updated

[nikhiljindal]

Ref #124

Steps to repro:

• Create an ingress using SSL certs stored in a secret.
• Update the SSL cert stored in secret and then run kubemci create again to update the MCI to use the updated SSL cert.

Expected:

• MCI is updated to use the new SSL cert

Actual:

• User gets "resource already in use" error.

Whats happening:
Since we cannot update a GCP SSL cert (GCP does not support it), we try to delete it and recreate it with the new value. But since the cert is being refered to by target proxy, deletion fails.

Ways to fix it:
Ideally we should do what ingress-gce does: #124. Create a new GCP SSL cert, use that in the target proxy and then delete the old cert. This will require refactoring existing ingress-gce code to enable reshare.
Or come up with our own creative way to fix this.

[nikhiljindal]

While there are different ways in which this can be fixed, I wanted to get @mdelio's opinion on how important this is, before spending too much time.

Users can use GCP SSL certs to manage their certs instead of using kubernetes secrets. To update the SSL cert in that case, users can:

• Create a new SSL cert
• Update the annotation on their ingress definition to point to this new SSL cert.
• Run kubemci create again to update the MCI.

This will work fine.

Other problems with using k8s secrets to manage SSL certs:
Users will have to create their secret in all clusters. Since we do not have a master cluster, kubemci picks any cluster at random and reads the secret from there. Thus requiring the user to keep the secret in all clusters in sync. When they remove an MCI from an ingress, they have to remember to delete the secret from that cluster as well.

@mdelio Should we support managing SSL certs using secrets?
cc @csbell @G-Harmon

[nikhiljindal]

cc @prameshj @nicksardo as FYI

[riaan53]

Hi, Any update on this issue? Im using https://github.com/jetstack/cert-manager to maintain an up to date letsencrypt cert in all my clusters.

[nikhiljindal]

@riaan53 No We havent been able to get to this yet.

Am curious if cert-manager supports multi-cluster natively or you've had to do some special setup to achieve it?
Also, does exporting those secrets as GCP SSL Certs work for you? (A controller that watches secrets and creates corresponding GCP SSL certs). That will ensure you do not run into this issue.

[riaan53]

Hi @nikhiljindal ,

Thanks for the reply. I just use dns01 validation so it does not matter for cert-manager, just as long as you stay below letsencrypt limits (Duplicate Certificate limit of 5 certificates per week) which is fine for me but you can add some subdomains to increase the limits to 20 if you need more.

Yes a controller like that should work nice - then cert-manager and that new controller will only be needed on one cluster. Do you have an open source project in mind that does something like that?

How will the syncing with kubemci work after the new controller makes changes? Probably have a flag to tell kubemci that certs are managed externally so it does not revert the changes if you do updates?

[riaan53]

Hi @nikhiljindal

I created a simple kubernetes cronjob that runs https://github.com/xenolf/lego and rolls the GCE loadbalancer cert if renewed. When creating the multi ingress for the first time use a temp self signed cert in the pre-shared-cert annotation. Used this repo as a start: https://github.com/bloomapi/letsencrypt-gcloud-balancer . The solution seems to work nice.

[nikhiljindal]

This is awesome @riaan53!
Would you be up to documenting the exact steps so that we can point other users to it?
Can be a .md file in this repo or a one-page doc or blog post. Whatever you prefer.

How will the syncing with kubemci work after the new controller makes changes? Probably have a flag to tell kubemci that certs are managed externally so it does not revert the changes if you do updates?

What I had in mind was that the controller will generate the new cert, create a new GCP SSL cert object and then run kubemci create with pre-shared-cert-annotation updated to new cert name. Running kubemci create again will update GCLB to use the new cert.

I see that https://github.com/bloomapi/letsencrypt-gcloud-balancer generates a new cert and then also updates target proxy directly. This works as well but then you need to ensure that you update the annotation in your ingress spec as well so that running kubemci create next time does not revert the changes.

[riaan53]

Hi @nikhiljindal

Yes ill do a small readme and a repo. I have modified it like you suggested to let kubemci update the new certs and all ingress changes but it looks like theres a small issue. kubemci checks that the cert exist (and fails if it doesnt) but it does not update the loadblaancer with the new one, can you please check if you see the same behaviour?

[pdecat]

The recently released beta of Google-managed SSL certificates provides a working solution for Let's Encrypt issued certificates without the renewal issue as the certificate resource stays the same at all times, no need to update the forwarding rule.

Just create a certificate then reference it by its static name in the ingress.gcp.kubernetes.io/pre-shared-cert annotation of the multi-cluster ingress then update the MCI with kubemci create.
Also, the tls section should/must be absent as the certificate is no longer stored in a kubernetes secret by kube-lego/cert-manager.

[Arachnid]

Are there any plans to improve this workflow so changes to Kubernetes secrets are picked up? Google managed SSL certificates don't support wildcards, so this is still very useful for some of us.

[guanzo]

How do I keep the google cert in sync with a wildcard cert stored in a k8s secret? My service went down recently b/c my secret was updated, but the google cert wasn't.

EDIT: Besides completely deleting kubemci, then recreating it from scratch, idk what to do. My cert is due to expire on Jan 5th.. so I'd really like some advice.

[dzikidzik]

In my case deleting kubemci was not an option, what is more I am using helm secrets to manage certificates, and simple update of certificate file in secrets.yaml was returning error.

I manually added certificate with gcloud compute ssl-certificates create ... and then updated target-https-proxies for kubemci with: gcloud compute target-https-proxies update <mciLoadBalancer> --ssl-certificates <newCert> after doing this I am able to use helm and update secrets.