fix(spin/certs): inject CA bunde in the filesystem

[radu-matei]

In spinkube/containerd-shim-spin#44 and deislabs/containerd-wasm-shims#176, a Spin application cannot make a TLS connection because the file system the host component is running in does not have a CA bundle.

To address it, this commit injects a CA bundle in the file system for every application pod created by injecting a default secret into /etc/ssl/certs/ca-certificates.crt.

There are a few alternatives to this approach:

• using the node's CA bundle
• compiling the default CA bundle into the shim executing the Spin application
• requiring each application to provide its own bundle

All of those alternatives have the main limitation of tying the bundle to either the app, shim, or node artifact (and all the lifecycle issue arising from that).
In contrast, this allows a cluster admin to configure this (while also allowing us to have a default bundle that works for a majority of use cases out of the box).

I tested this PR with an app that makes TLS connections, and used Alpine's bundle, which is ~215 KB in size.

The main limitation of this approach is having to respect the 1MB secret size of Kubernetes.
(I can see an experience for working around the 1MB size where a cluster admin could create a volume with the CA bundle, and the Spin operator could inject that instead of a (the?) default secret. That is not is scope for this PR.)

To test this draft, you need to fetch a bundle and create a secret from it:

# Alpine's bundle pushed to filebin by me.
$ wget https://filebin.net/yc0ako929ozv2slu/ca-certificates.crt
$ kubectl create secret generic spin-ca --from-file=ca-certificates.crt

Marking as a draft, as it needs testing and a way to create the secret at deployment time.

[jsturtevant]

this also serves as an example way to address deislabs/containerd-wasm-shims#89

[adamreese]

Updated test setup for creating the ca-certificates secret:

curl -sfL https://curl.se/ca/cacert.pem -o ca-certificates.crt
kubectl create secret generic spin-ca --from-file=ca-certificates.crt

[adamreese]

One issue with mounting a secret to every deployment is namespacing. A CA secret will need to be created in each namespace a spin app will be running

[adamreese]

If we go ahead with creating the secret automatically we could bundle the certs with the operator. This would allow the operator to create the required secret in any namespace that it puts a SpinApp deployment.

Rather than adding default certs to every application we could add a include_default_ca_certs to the SpinApp CRD. When set the operator would add the volume to the deployment.

Reasons against certs by default...

• A ca cert secret would need to be created for every namespace that a SpinApp will run in.
• If an application should not be able to connect to the outside world the certs would need to be disabled for all applications running in the namespace.
• It breaks the convention of container images, which don't have default certs.

Another path forward would be to keep it manual and provide documentation on how to configure certs using the workaround below.

---

As a workaround you can create the volume mount directly in the SpinApp manifest.

Fetch the certs and create the secret

curl -sfL https://curl.se/ca/cacert.pem -o ca-certificates.crt
kubectl create secret generic spin-ca --from-file=ca-certificates.crt

Add the secret as a volume to the SpinApp manifest

spec:
  volumes:
    - name: spin-ca
      secret:
        secretName: spin-ca
        items:
          - key: ca-certificates.crt
            path: ca-certificates.crt
  volumeMounts:
    - name: spin-ca
      mountPath: "/etc/ssl/certs/ca-certificates.crt"
      readOnly: true
      subPath: ca-certificates.crt

[radu-matei]

The primary reason I feel strongly about injecting a bundle by default is the difference in experience when testing the application locally and when deploying with SpinKube.

If the app I'm working on is able to make requests to HTTPS endpoints when running locally (and potentially in CI when doing things like end to end testing), but no when deploying, that is not a good experience (and the main difference compared to your container image analogy).

If an application should not be able to connect to the outside world the certs would need to be disabled for all applications

This is problematic because guest code (running in Wasm) connecting to the outside world is controlled through allowed_outbound_hosts — but certs are used by host components.

[radu-matei]

Superseded by #207.
Thanks, @adamreese!