This repository can be used to deploy OpenWhisk to a Kubernetes cluster. To accomplish this, we have created a Kubernetes job responsible for deploying OpenWhisk from inside of Kubernetes. This job runs through the OpenWhisk Ansible playbooks with some modifications to "Kube-ify" specific actions. The reason for this approach is to try and streamline a one size fits all way of deploying OpenWhisk.
Currently, the OpenWhisk deployment is going to be a static set of Kube yaml files. It should be easy to use the tools from this repo to build your own OpenWhisk deployment job, allowing you to set up your own configurations if need be.
The scripts and Docker images should be able to:
- Build the Docker image used for deploying OpenWhisk.
- Uses a Kubernetes job to deploy OpenWhisk.
- Requirements
- Setting up Kuberentes
- Configuring OpenWhisk
- Cleanup
- Troubleshooting
- Manually Building Custom Docker Files
- Limitations and Enhancements
- Issues
A number of requirements must be met for OpenWhisk to deploy on Kubernetes.
Kubernetes
- Kubernetes version 1.5.6 and 1.6.2
- Kubernetes has KubeDNS deployed
- (Optional) Kubernetes Pods can receive public addresses. This will be required if you wish to reach Nginx from outside of the Kubernetes cluster network.
OpenWhisk
- Docker version 1.12+
NOTE: If you do not have an environment that meets these requirements then you can set one up following the Setting up Kubernetes section.
If you are deploying Kubernetes from scratch to try out OpenWhisk, there are a number of ways to setup a Dev/Test environmnet depending on your host OS. To mitigate that, we are using VirtualBox with a Ubuntu VM. For the full instructions on setting an environment up, take a look at these instructions.
We also have experimental support for Minikube, see the Minikube-specific install instructions for more details.
To configure OpenWhisk on Kubernetes, you will need to target a Kubernetes
environment. If you do not have one up and running, then you can look
at the Setting up Kubernetes section. Once you
are successfully up, running, and targetd, you will then need to create a
namespace called openwhisk. To do this, you can just run the following command.
kubectl apply -f configure/openwhisk_kube_namespace.yml
From here, you should just need to run the Kubernetes job to
setup the OpenWhisk environment. The only caveat is that
the default image is used to deploy to kube v1.5.6.
Take a look
here
if you wish to change to kube v1.6.2 by replacing v1.5.2 to v1.6.2.
NOTE Unfortunately Kube does not have backward compatibility requirements between the cli and Kube api server. However, the v1.5.6 image will probably work with any Kube v1.5+ and the v1.6.2 image will probably work with any Kube v1.6+. If the configuration image does return compatibility issues then try building a custom image.
kubectl apply -f configure/configure_whisk.yml
To see what is happening during the deployment process, you can view the logs from the configuration Pod creted by the previous command.
kubectl -n openwhisk logs configure-openwhisk-XXXXX
Once the configuration job successfully finishes, you will need manually deploy the rest of the OpenWhisk components.
From here, you will now need to get the publicly available address of Nginx.
- Obtain the IP address of the Kubernetes nodes.
kubectl get nodes
- Obtain the public port for the Kubernetes Nginx Service
kubectl -n openwhisk describe service nginx
From here you should note the port used for the api endpoint. E.g:
export WSK_PORT=$(kubectl -n openwhisk describe service nginx | grep https-api | grep NodePort| awk '{print $3}' | cut -d'/' -f1)
Now you should be able to setup the wsk cli like normal and interact with Openwhisk.
wsk property set --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP --apihost https://[nginx_ip]:$WSK_PORT
Lastly, you will need to install the initial catalog. To do this, you will need
to set the OPENWHISK_HOME environment variable:
export OPENWHISK_HOME [location of the openwhisk repo]
Then you should be able to run the following commands. Just make sure to
replace the [nginx_ip] bellow.
pushd /tmp
git clone https://github.com/apache/incubator-openwhisk-catalog
cd incubator-openwhisk-catalog/packages
./installCatalog.sh 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP https://[nginx_ip]:$WSK_PORT
popd
At some point there might be a need to cleanup the Kubernetes environment. For this, we want to delete all the OpenWhisk deployments, services, jobs and whatever else might be there. For this, you can run the following script:
./kube_environment/cleanup.sh
When inspecting kafka logs of various components and they are not able to send/receive message then Kafka is the usual problem. If everything is deployed on a single machine, then you might need to allow Kube Pods to communicate with themselves over a Kube Service. Setting a network to promiscous mode can be the solution will enable network traffic to route in a loop back to itself. E.g:
ip link set docker0 promisc on
When deploying the configuration pod, if it fails with a
error validating data: the server does not allow access to the requested resource;
error then you probably do not have permissions to create Pods from a Pod running
in the Kube cluster. You will need to create a ClusterRoleBinding with proper
security settings. For information about the role bindings,
take a look at the info here.
There are two images that are required when deploying OpenWhisk on Kube, Nginx and the OpenWhisk configuration image. Right now the the configuration images built will work with a Kube version 1.5.6 and 1.6.2. To build the configuration image with a custom Kube version you can edit the build script here
To build these images, there is a helper script that installs all required dependencies and the Docker images themselves. For example, one the required dependencies is the wsk cli and to build it you will need to download the OpenWhisk repo and setup your invironment to build the docker images via gradle. That setup can be found here.
Important
To build custom docker images, you will need to be on a Linux machine.
During the wsk cli build process it mounts a number of files from the
host machine. Because of this, Golang determines that the wsk build
architecture should be for macOS, but of course this is the wrong version
when running later. It needs to be built for the Linux architecture.
To use the script, it takes in 2 arguments:
-
(Required) The first argument is the Docker account to push the built images to. For Nginx, it will tag the image as
account_name/whisk_nginx:latestand the OpenWhisk configuration image will be taggedaccount_name/whisk_config:dev-v1.5.6andaccount_name/whisk_config:dev-v1.6.2.NOTE: log into Docker before running the script or it will fail to properly upload the docker images.
-
(Optional) The second argument is the location of where the repo is installed locally. By default it assumes that this repo exists at
$HOME/workspace/openwhisk.
If you plan on building your own images and would like to change from danlavine's,
then make sure to update the
configure_whisk.yml and
nginx with your images.
To run the script, use the command:
docker/build <Docker username> <(optional) openwhisk dir>
The current Kube Deployment and Services files that define the OpenWhisk cluster can be found here. Only one instance of each OpenWhisk process is created, but if you would like to increase that number, then this would be the place to do it. Simply edit the appropriate file and Manually Build Custom Docker Files
When in the process of creating a new deployment, it is nice to
run things by hand to see what is going on inside the container and
not have it be removed as soon as it finishes or fails. For this,
you can change the command of configure_whisk.yml
to command: [ "tail", "-f", "/dev/null" ]. Then just run the
original command from inside the Pod's container.
During the deployment process, OpenWhisk needs to generate a CA-cert
for Nginx and currently it has a static dns entry. Because of this, you
will need to connect to OpenWhisk using the insecure mode (e.g. wsk -i).
There is future work to make this CA-cert configurable.
A couple of components for OpenWhisk on Kube deployment strategy requires custom built Docker images. One such component is Nginx and currently resides at danlavine/whisk_nginx. There is currently and open issue to make a public image and once it is resolved, then we can switch to the public image.
The second Docker image this deployment strategy relies on is the OpenWhisk configuration image. For now, it is hosted at danlavine/whisk_config, but ideally an official images can be built an maintained at some point. If you would like to build your own deployment image, see Manually Build Custom Docker Files
Lastly, since OpenWhisk is configured/deployed via a Kubernetes Pod it requires
the correct kubectl version to be built into danlavine/whisk_config. For now,
there is only a version for Kube 1.5, and one can be built for 1.6, but there
is no CI to test it against at the moment.
Minikube (experimental) We also have experimental support for
- Minikube, see the
- Minikube-specific install instructions for more details.
Bad Kube versions
- Kube 1.6.3 has an issue with volume mount subpaths. See here for more information.
- Enable the configuration job to run any number of times. This way it updates an already running OpenWhisk deployment on all subsequent runs
- Use a public Edge Docker image once this issue is resolved
Report bugs, ask questions and request features here on GitHub.
You can also join our slack channel and chat with developers. To get access to our slack channel, request an invite here.