Add example of AKS attestation and secret provisioning

examples/aks-attestation/.gitignore

@@ -0,0 +1 @@
+/gramine

examples/aks-attestation/README.md

@@ -0,0 +1,140 @@
+# Gramine Attestation Inside AKS cluster
+
+This guide demonstrates how Gramine DCAP attestation quote can be generated and verified from
+within an AKS cluster. Here, we provide an end-to-end example to help Cloud Solution Providers
+integrate Gramine’s RA-TLS attestation and secret provisioning feature with a confidential compute
+cluster managed by Azure Kubernetes Service. This guide contains necessary reference wrappers that
+enable Gramine to use AKS components such as AESMD and DCAP quote provider libraries. This guide
+also describes a microservice deployment for the RA-TLS verifier (server) that can be readily
+deployed to the AKS cluster.
+
+## Preparing client and server images
+
+This demonstration is based on the `ra-tls-secret-prov` example from
+https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov. Familiarity
+with this example is highly recommended before proceeding further. The sample contains client and
+server applications, where by default server is running on localhost:4433. In the example, the
+client sends its SGX quote to the server for verification. After successful quote verification, the
+server sends a secret to the client. To run these client and server applications inside the AKS
+cluster, user needs to prepare two docker images, one for the client and one for the server. In our
+AKS attestation example, the server will no longer run on localhost, instead it will run in a Docker
+container inside the AKS cluster. The server container should be assigned a DNS name
+(e.g., `<AKS-DNS-NAME>`) to be accessible from the outside of the container. The client will send
+requests to this DNS name. Therefore, for demonstration we updated the example certificates from
+https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov/certs by
+replacing the "Common Name" field in the server certificate (i.e., `server2-sha256.crt`) from
+`localhost` to `<AKS-DNS-NAME.*.cloudapp.azure.com>`.
+
+In order to create base client and server images for the AKS environment, user can execute the
+`base-image-generation-script.sh` script. Since both client and server applications will
+run inside containers in the AKS cluster, and the client application will send its SGX quote to the
+server for verification, therefore the user needs to graminize the client application. Hence, the
+following two steps create a native Docker server image and a graminized GSC client image for the
+AKS cluster.
+
+Note: This example is Ubuntu-specific (tested version is Ubuntu 18.04).
+
+### Creating server image
+
+1. The `base-image-generation-script.sh` script will create the native Docker server image with the
+   name `aks-secret-prov-server-img:latest`.
+
+2. Push the server image to Docker Hub or your preferred registry:
+
+    ```sh
+    $ docker tag aks-secret-prov-server-img:latest \
+        <dockerhubusername>/aks-secret-prov-server-img:latest
+    $ docker push <dockerhubusername>/aks-secret-prov-server-img:latest
+    ```
+
+3. Deploy `aks-secret-prov-server-img:latest` in the AKS confidential compute cluster:
+    - Reference deployment file:
+        `aks-secret-prov-server-deployment.yaml`
+
+### Creating client image
+
+1. The `base-image-generation-script.sh` script will create the native Docker client image with the
+   name `aks-secret-prov-client-img:latest`.
+
+2. Create the GSC client image:
+
+    Note: We tested this example with DCAP driver 1.11 specified in the GSC configuration file.
+
+    ```sh
+    $ cd gsc
+    $ cp config.yaml.template config.yaml
+    $ openssl genrsa -3 -out enclave-key.pem 3072
+    $ ./gsc build aks-secret-prov-client-img:latest \
+        examples/aks-attestation/aks-secret-prov-client.manifest
+    $ ./gsc sign-image aks-secret-prov-client-img:latest enclave-key.pem
+    ```
+
+5. Push resulting image to Docker Hub or your preferred registry:
+
+    ```sh
+    $ docker tag gsc-aks-secret-prov-client-img:latest \
+        <dockerhubusername>/gsc-aks-secret-prov-client-img:latest
+    $ docker push <dockerhubusername>/gsc-aks-secret-prov-client-img:latest
+    ```
+
+6. Deploy `gsc-aks-secret-prov-client-img:latest` in AKS confidential compute cluster:
+    - Reference deployment file:
+        `aks-secret-prov-client-deployment.yaml`
+
+## Deploying client and server images in AKS Confidential Compute cluster
+
+AKS confidential compute cluster can be created using the following
+[link](https://docs.microsoft.com/en-us/azure/confidential-computing/confidential-enclave-nodes-aks-get-started).
+
+Gramine performs out-of-proc mode DCAP quote generation. Out-of-proc mode quote generation requires
+aesmd service. To fulfill this requirement, AKS provides the
+[sgxquotehelper daemonset](https://docs.microsoft.com/en-us/azure/confidential-computing/confidential-nodes-aks-addon#out-of-proc-attestation-for-confidential-workloads)
+(can be enabled by `--enable-sgxquotehelper` during cluster creation). This feature exposes aesmd
+service for the container node. The service will internally connect with az-dcap-client to fetch the
+platform collateral required for quote generation. In this demo, the
+`aks-secret-prov-client-deployment.yaml` uses aesmd service exposed by AKS with the help of the
+sgxquotehelper plugin.
+
+In our example, the client will generate the SGX quote that will be embedded inside the RA-TLS
+certificate. On receiving the quote, the server will internally verify it using the
+libsgx-dcap-quote-verify library via the az-dcap-client library.
+
+### Deployment
+
+```sh
+$ kubectl apply -f aks-secret-prov-server-deployment.yaml
+```
+
+Once the server container is in running state, start the client container as shown below:
+
+```sh
+$ kubectl apply -f aks-secret-prov-client-deployment.yaml
+```
+
+At this stage, a successful RA-TLS verification will be completed, and the secrets will be
+provisioned from the server to the client.
+
+## Checking SGX quote generation and verification
+
+Verify the client job is completed:
+
+```sh
+$ kubectl get pods
+```
+
+Receive logs to verify the secret has been provisioned to the client:
+
+```sh
+$ kubectl logs -l app=gsc-ra-tls-secret-prov-client --tail=50
+```
+
+Expected output:
+
+`--- Received secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'`
+
+Delete both client and server containers:
+
+```sh
+$ kubectl delete -f aks-secret-prov-server-deployment.yaml
+$ kubectl delete -f aks-secret-prov-client-deployment.yaml
+```

examples/aks-attestation/aks-secret-prov-client-deployment.yaml

@@ -0,0 +1,29 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+   name: secret-prov-client
+   labels:
+      app: secret-prov-client
+spec:
+   template:
+    metadata:
+       labels:
+           app: gsc-ra-tls-secret-prov-client
+    spec:
+      volumes:
+      - name: var-run-aesmd
+        hostPath:
+          path: /var/run/aesmd
+      containers:
+      - name: gsc-ra-tls-secret-prov-client-container
+        image: <dockerhubusername>/gsc-aks-secret-prov-client-img:latest
+        env:
+        - name: SECRET_PROVISION_SERVERS
+          value: "ra-tls-server-aks-dns.eastus.cloudapp.azure.com:4433"
+        resources:
+          limits:
+            kubernetes.azure.com/sgx_epc_mem_in_MiB: 10
+        volumeMounts:
+        - name: var-run-aesmd
+          mountPath: /var/run/aesmd
+      restartPolicy: Never

examples/aks-attestation/aks-secret-prov-client.dockerfile

@@ -0,0 +1,27 @@
+FROM ubuntu:18.04
+
+RUN apt-get update \
+    && env DEBIAN_FRONTEND=noninteractive apt-get install -y wget \
+    build-essential \
+    gnupg2 \
+    libcurl3-gnutls \
+    python3
+
+# Installing DCAP libraries
+RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main' \
+    > /etc/apt/sources.list.d/intel-sgx.list \
+    && wget https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key \
+    && apt-key add intel-sgx-deb.key
+
+RUN apt-get update \
+    && apt-get install -y libsgx-urts \
+    libsgx-dcap-ql \
+    libsgx-quote-ex
+
+WORKDIR /ra-tls-secret-prov
+
+COPY gramine/CI-Examples/ra-tls-secret-prov/certs ./certs
+
+COPY gramine/CI-Examples/ra-tls-secret-prov/secret_prov_min_client /usr/local/bin
+
+ENTRYPOINT ["secret_prov_min_client"]

examples/aks-attestation/aks-secret-prov-client.manifest

@@ -0,0 +1,19 @@
+# Secret Provisioning library (client-side) is preloaded
+loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
+
+loader.env.SECRET_PROVISION_SERVERS = { passthrough = true }
+loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
+loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "certs/test-ca-sha256.crt"
+
+sgx.remote_attestation = true
+
+sgx.allowed_files = [
+  "file:/etc/ethers",
+  "file:/etc/gai.conf",
+  "file:/etc/group",
+  "file:/etc/host.conf",
+  "file:/etc/hosts",
+  "file:/etc/nsswitch.conf",
+  "file:/etc/passwd",
+  "file:/etc/resolv.conf",
+]

examples/aks-attestation/aks-secret-prov-server-deployment.yaml

@@ -0,0 +1,39 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ra-tls-secret-prov-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: ra-tls-secret-prov-server
+  template:
+    metadata:
+      labels:
+        app: ra-tls-secret-prov-server
+    spec:
+      containers:
+      - name: ra-tls-secret-prov-server-container
+        image: <dockerhubusername>/aks-secret-prov-server-img:latest
+        ports:
+        - containerPort: 4433
+        resources:
+          limits:
+            kubernetes.azure.com/sgx_epc_mem_in_MiB: 10
+
+# TODO: Requesting an SGX machine is not needed, but Intel DCAP libraries have a bug of trying to
+# open the SGX driver (see  https://github.com/intel/linux-sgx/issues/756)
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    service.beta.kubernetes.io/azure-dns-label-name: ra-tls-server-aks-dns
+  name: ra-tls-secret-prov-server
+spec:
+  type: LoadBalancer
+  ports:
+  - port: 4433
+  selector:
+    app: ra-tls-secret-prov-server

examples/aks-attestation/aks-secret-prov-server.dockerfile

@@ -0,0 +1,44 @@
+FROM ubuntu:18.04
+
+RUN apt-get update \
+    && env DEBIAN_FRONTEND=noninteractive apt-get install -y \
+    build-essential \
+    gnupg2 \
+    libcurl3-gnutls \
+    libcurl4-openssl-dev \
+    python3 \
+    wget
+
+# Installing Azure DCAP Quote Provider Library (az-dcap-client).
+# Here, the version of az-dcap-client should be in sync with the az-dcap-client
+# version used for quote generation. User can replace the below package with the
+# latest package.
+RUN wget https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/a/az-dcap-client/az-dcap-client_1.10_amd64.deb \
+ && dpkg -i az-dcap-client_1.10_amd64.deb
+
+# Installing DCAP Quote Verification Library
+RUN echo 'deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main' \
+    > /etc/apt/sources.list.d/intel-sgx.list \
+    && wget https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key \
+    && apt-key add intel-sgx-deb.key
+
+RUN apt-get update && apt-get install -y libsgx-dcap-quote-verify
+
+WORKDIR /ra-tls-secret-prov
+
+COPY gramine/CI-Examples/ra-tls-secret-prov/certs ./certs
+COPY gramine/CI-Examples/ra-tls-secret-prov/files ./files
+
+COPY gramine/CI-Examples/ra-tls-secret-prov/secret_prov_server_dcap /usr/local/bin
+
+RUN mkdir libs
+
+COPY gramine/meson_build_output/lib/x86_64-linux-gnu/libsecret_prov_verify_dcap.so libs
+COPY gramine/meson_build_output/lib/x86_64-linux-gnu/libsgx_util.so libs
+COPY gramine/meson_build_output/lib/x86_64-linux-gnu/libmbedcrypto_gramine.so.* libs
+COPY gramine/meson_build_output/lib/x86_64-linux-gnu/libmbedtls_gramine.so.* libs
+COPY gramine/meson_build_output/lib/x86_64-linux-gnu/libmbedx509_gramine.so.* libs
+
+ENV LD_LIBRARY_PATH = "${LD_LIBRARY_PATH}:/ra-tls-secret-prov/libs"
+
+ENTRYPOINT ["secret_prov_server_dcap"]

examples/aks-attestation/base-image-generation-script.sh

@@ -0,0 +1,24 @@
+# Download and build Gramine. We need it to build and copy ra-tls-secret-prov files and
+# relevant libraries into the server and client Dockerfiles.
+if [ -d "gramine" ]; then
+    echo "***** gramine directory exists, proceeding to image generation *****"
+else
+    bash ./gramine_build.sh
+fi
+
+# Include Meson build output packages dir in $PKG_CONFIG_PATH, contains mbedTLS and util libs
+export PKG_CONFIG_PATH="$(find $PWD/gramine/meson_build_output/lib -type d -path '*/pkgconfig'):${PKG_CONFIG_PATH}"
+
+# Create Server image
+cd gramine/CI-Examples/ra-tls-secret-prov
+make clean && make dcap
+cd ../../../
+docker build -f aks-secret-prov-server.dockerfile -t aks-secret-prov-server-img .
+
+# Create Client image
+cd gramine/CI-Examples/ra-tls-secret-prov
+make clean && make secret_prov_min_client
+cd ../../../
+docker build -f aks-secret-prov-client.dockerfile -t aks-secret-prov-client-img .
+
+rm -rf gramine/

examples/aks-attestation/certs/README

@@ -0,0 +1,13 @@
+This directory contains pre-generated example certificates. In particular:
+
+- `test-ca-sha256.crt` -- RSA SHA256 root CA certificate in PEM format. Loaded
+  in client (attester) so that client can verify the server's certificate.
+- `server2-sha256.crt` -- RSA SHA256 leaf server certificate in PEM format.
+  Loaded in server (verifier), so it will send it to the client during TLS
+  handshake. The "Common Name" field is set to
+  `ra-tls-server-aks-dns.eastus.cloudapp.azure.com`.
+- `server2.key` -- RSA private key in PEM format. Loaded in server (verifier).
+
+Note: These certificates should not be used in production. Here, user should
+      update the server2-sha256.crt with ``Common Name`` pointing to the DNS
+      name of the ra-tls-secret-prov server deployed in the AKS cluster.