Table of Contents generated with DocToc
- Using Kubeflow
This guide will walk you through the basics of deploying and interacting with Kubeflow. Some understanding of Kubernetes, Tensorflow, and Ksonnet are useful in completing the contents of this guide.
For an end to end example illustrating in details how to deploy kubeflow and run a training job from scratch, check out this tutorial.
- Kubernetes >= 1.8 see here
- ksonnet version 0.9.2. (See below for an explanation of why we use ksonnet)
- An existing kubernetes cluster:
- A minimum of 0.6 CPU in cluster (Reserved for 3 replicated ambassador pods and according to your need add additional CPUs)
- Node with storage >= 10 GB (Due to the ML libraries and third party packages being bundled in Kubeflow Docker images)
We will be using Ksonnet to deploy kubeflow into your cluster.
Initialize a directory to contain your ksonnet application.
ks init my-kubeflow
Install the Kubeflow packages into your application.
# For a list of releases see:
# https://github.com/kubeflow/kubeflow/releases
VERSION=v0.1.2
cd my-kubeflow
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/${VERSION}/kubeflow
ks pkg install kubeflow/core@${VERSION}
ks pkg install kubeflow/tf-serving@${VERSION}
ks pkg install kubeflow/tf-job@${VERSION}
Create the Kubeflow core component. The core component includes:
- JupyterHub
- TensorFlow job controller
ks generate core kubeflow-core --name=kubeflow-core
# Enable collection of anonymous usage metrics
# Skip this step if you don't want to enable collection.
# Or set reportUsage to false (the default).
ks param set kubeflow-core reportUsage true
ks param set kubeflow-core usageId $(uuidgen)
Ksonnet allows us to parameterize the Kubeflow deployment according to our needs. We will define two environments: nocloud, and cloud.
ks env add nocloud
ks env add cloud
The nocloud
environment can be used for minikube or other basic k8s clusters, the cloud
environment will be used for GKE or Azure in this guide.
If using GKE, we can configure our cloud environment to use GCP features with a single parameter:
ks param set kubeflow-core cloud gke --env=cloud
If the cluster was created on Azure with AKS/ACS:
ks param set kubeflow-core cloud aks --env=cloud
If it was created with acs-engine instead:
ks param set kubeflow-core cloud acsengine --env=cloud
Now let's set ${KF_ENV}
to cloud
or nocloud
to reflect our environment for the rest of the guide:
$ KF_ENV=cloud|nocloud
- By default Kubeflow does not persist any work that is done within the Jupyter notebook.
- If the container is destroyed or recreated, all of its contents, including users working notebooks and other files are going to be deleted.
- To enable the persistence of such files, the user will need to have a default StorageClass defined for persistent volumes.
- You can run the following command to check if you have a storage class
kubectl get storageclass
-
Users with a default storage class defined can use the jupyterNotebookPVCMount parameter to create a volume that will be mounted within the notebook
ks param set kubeflow-core jupyterNotebookPVCMount /home/jovyan
- Here we mount the volume at
/home/jovyan
because the notebook always executes as user jovyan - The selected directory will be stored on whatever storage is the default for the cluster (typically some form of persistent disk)
- Here we mount the volume at
Create a namespace for your deployment and set it as part of the environment. Feel free to change the namespace to a value that better suits your kubernetes cluster.
NAMESPACE=kubeflow
kubectl create namespace ${NAMESPACE}
ks env set ${KF_ENV} --namespace ${NAMESPACE}
And apply the components to our Kubernetes cluster
ks apply ${KF_ENV} -c kubeflow-core
At any time you can inspect the kubernetes objects definitions for a particular ksonnet component using ks show
e.g
ks show ${KF_ENV} -c kubeflow-core
When enabled, Kubeflow will report anonymous usage data using spartakus, Kubernetes' reporting tool. Spartakus does not report any personal information. See here for more detail. This is entirely voluntary and you can opt out by doing the following
ks param set kubeflow-core reportUsage false
# Delete any existing deployments of spartakus
kubectl delete -n ${NAMESPACE} deploy spartakus-volunteer
To explictly enable usage reporting repeat the above steps setting reportUsage to true
ks param set kubeflow-core reportUsage true
# Delete any existing deployments of spartakus
kubectl delete -n ${NAMESPACE} deploy spartakus-volunteer
Reporting usage data is one of the most signifcant contributions you can make to Kubeflow; so please consider turning it on. This data allows us to improve the project and helps the many companies working on Kubeflow justify continued investement.
You can improve the quality of the data by giving each Kubeflow deployment a unique id
ks param set kubeflow-core usageId $(uuidgen)
The kubeflow-core component deployed JupyterHub and a corresponding load balancer service. You can check its status using the kubectl command line.
kubectl get svc -n=${NAMESPACE}
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
tf-hub-0 ClusterIP None <none> 8000/TCP 1m
tf-hub-lb ClusterIP 10.11.245.94 <none> 80/TCP 1m
...
By default we are using ClusterIPs for the JupyterHub UI. This can be changed to one of the following:
-
NodePort (for non-cloud) by issuing
ks param set kubeflow-core jupyterHubServiceType NodePort ks apply ${KF_ENV}
-
LoadBalancer (for cloud) by issuing
ks param set kubeflow-core jupyterHubServiceType LoadBalancer ks apply ${KF_ENV}
however this will leave your Jupyter notebook open to the Internet.
To connect to your Jupyter Notebook locally:
PODNAME=`kubectl get pods --namespace=${NAMESPACE} --selector="app=tf-hub" --output=template --template="{{with index .items 0}}{{.metadata.name}}{{end}}"`
kubectl port-forward --namespace=${NAMESPACE} $PODNAME 8000:8000
Then, open http://127.0.0.1:8000 in your browser.
You should see a sign in prompt.
-
Sign in using any username/password
-
Click the "Start My Server" button, and you will be greeted by a dialog screen.
-
Select a CPU or GPU image from the Image dropdown menu depending on whether you are doing CPU or GPU training, or whether or not you have GPUs in your cluster. We currently offer a cpu and gpu image for each tensorflow minor version(eg: 1.4.1,1.5.1,1.6.0). Or you can type in the name of any TF image you want to run.
-
Allocate memory, CPU, GPU, or other resources according to your need (1 CPU and 2Gi of Memory are good starting points)
- To allocate GPUs, make sure that you have GPUs available in your cluster
- Run the following command to check if there are any nvidia gpus available:
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
- If you have GPUs available, you can schedule your server on a GPU node by specifying the following json in
Extra Resource Limits
section:{"nvidia.com/gpu": "1"}
-
Click Spawn
-
The images are 10's of GBs in size and can take a long time to download depending on your network connection
-
You can check the status of your pod by doing
kubectl -n ${NAMESPACE} describe pods jupyter-${USERNAME}
-
Where ${USERNAME} is the name you used to login
-
GKE users if you have IAP turned on the pod will be named differently
- If you signed on as [email protected] the pod will be named
jupyter-accounts-2egoogle-2ecom-3USER-40DOMAIN-2eEXT
-
-
-
You should now be greeted with a Jupyter Notebook interface.
The image supplied above can be used for training Tensorflow models with Jupyter. The images include all the requisite plugins, including Tensorboard that you can use for rich visualizations and insights into your models.
To test the install, we can run a basic hello world (adapted from mnist_softmax.py )
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
import tensorflow as tf
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
for _ in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))
Paste the example into a new Python 3 Jupyter notebook and execute the code. This should result in a 0.9014 accuracy result against the test data.
Please note that when running on most cloud providers, the public IP address will be exposed to the internet and is an unsecured endpoint by default. For a production deployment with SSL and authentication, refer to the documentation.
We treat each deployed model as a component in your APP.
Create a component for your model located on cloud
MODEL_COMPONENT=serveInception
MODEL_NAME=inception
MODEL_PATH=gs://kubeflow-models/inception
ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}
ks param set ${MODEL_COMPONENT} modelPath ${MODEL_PATH}
(Or) create a component for your model located on nfs, learn more from components/k8s-model-server
MODEL_COMPONENT=serveInceptionNFS
MODEL_NAME=inception-nfs
MODEL_PATH=/mnt/var/nfs/general/inception
MODEL_STORAGE_TYPE=nfs
NFS_PVC_NAME=nfs
ks generate tf-serving ${MODEL_COMPONENT} --name=${MODEL_NAME}
ks param set ${MODEL_COMPONENT} modelPath ${MODEL_PATH}
ks param set ${MODEL_COMPONENT} modelStorageType ${MODEL_STORAGE_TYPE}
ks param set ${MODEL_COMPONENT} nfsPVC ${NFS_PVC_NAME}
Deploy the model component. Ksonnet will pick up existing parameters for your environment (e.g. cloud, nocloud) and customize the resulting deployment appropriately
ks apply ${KF_ENV} -c ${MODEL_COMPONENT}
As before, a few pods and services have been created in your cluster. You can get the inception serving endpoint by querying kubernetes:
kubectl get svc inception -n=${NAMESPACE}
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
inception LoadBalancer 10.35.255.136 ww.xx.yy.zz 9000:30936/TCP 28m
...
In this example, you should be able to use the inception_client to hit ww.xx.yy.zz:9000
The model at gs://kubeflow-models/inception is publicly accessible. However, if your environment doesn't
have google cloud credential setup, TF serving will not be able to read the model.
See this issue for example.
To setup the google cloud credential, you should either have the environment variable
GOOGLE_APPLICATION_CREDENTIALS
pointing to the credential file, or run gcloud auth login
.
See doc for more detail.
Seldon-core provides deployment for any machine learning runtime that can be packaged in a Docker container.
Install the seldon package
ks pkg install kubeflow/seldon
Generate the core components
ks generate seldon seldon
Seldon allows complex runtime graphs for model inference to be deployed. For an example end-to-end integration see the kubeflow-seldon example. For more details see the seldon-core documentation.
Note: Before submitting a training job, you should have deployed kubeflow to your cluster. Doing so ensures that
the TFJob
custom resource is available when you submit the training job.
We treat each TensorFlow job as a component in your APP.
Create a component for your job.
JOB_NAME=myjob
ks generate tf-job ${JOB_NAME} --name=${JOB_NAME}
To configure your job you need to set a bunch of parameters. To see a list of parameters run
ks prototype describe tf-job
Parameters can be set using ks param
e.g. to set the Docker image used
IMAGE=gcr.io/tf-on-k8s-dogfood/tf_sample:d4ef871-dirty-991dde4
ks param set ${JOB_NAME} image ${IMAGE}
You can also edit the params.libsonnet
files directly to set parameters.
Warning Currently setting args via the command line doesn't work because of escaping issues (see ksonnet/ksonnet/issues/235). So to set the parameters you will need
to directly edit the params.libsonnet
file directly.
To run your job
ks apply ${KF_ENV} -c ${JOB_NAME}
For information on monitoring your job please refer to the TfJob docs.
To delete your job
ks delete ${KF_ENV} -c ${JOB_NAME}
Kubeflow ships with a ksonnet prototype suitable for running the TensorFlow CNN Benchmarks.
Create the component
CNN_JOB_NAME=mycnnjob
ks generate tf-cnn ${CNN_JOB_NAME} --name=${CNN_JOB_NAME}
Submit it
ks apply ${KF_ENV} -c ${CNN_JOB_NAME}
Monitor it (Noted that tf-cnn job is also a tfjobs. Please refer to the TfJob docs)
kubectl get -o yaml tfjobs ${CNN_JOB_NAME}
Delete it
ks delete ${KF_ENV} -c ${CNN_JOB_NAME}
The prototype provides a bunch of parameters to control how the job runs (e.g. use GPUs run distributed etc...). To see a list of paramets
ks prototype describe tf-cnn
Note: Before submitting a training job, you should have deployed kubeflow to your cluster. Doing so ensures that
the PyTorchJob
custom resource is available when you submit the training job.
We treat each PyTorch job as a component in your APP.
Create a component for your job.
JOB_NAME=myjob
ks generate pytorch-job ${JOB_NAME} --name=${JOB_NAME}
To configure your job you need to set a bunch of parameters. To see a list of parameters run
ks prototype describe pytorch-job
Parameters can be set using ks param
e.g. to set the Docker image used
IMAGE=<your pytorch image>
ks param set ${JOB_NAME} image ${IMAGE}
You can also edit the params.libsonnet
files directly to set parameters.
Warning Currently setting args via the command line doesn't work because of escaping issues (see ksonnet/ksonnet/issues/235). So to set the parameters you will need
to directly edit the params.libsonnet
file directly.
To run your job
ks apply ${KF_ENV} -c ${JOB_NAME}
To delete your job
ks delete ${KF_ENV} -c ${JOB_NAME}
- Often times data scientists require a POSIX compliant filesystem
- For example, most HDF5 libraries require POSIX and don't work with an object store like GCS or S3
- When working with teams you might want a shared POSIX filesystem to be mounted into your notebook environments so that data scientists can work collaboratively on the same datasets.
- Here we show how to customize your Kubeflow deployment to achieve this.
Set the disks parameter to a comma separated list of the Google persistent disks you want to mount
- These disks should be in the same zone as your cluster
- These disks need to be created manually via gcloud or the Cloud console e.g.
- These disks can't be attached to any existing VM or POD.
Create the disks
gcloud --project=${PROJECT} compute disks create --zone=${ZONE} ${PD_DISK1} --description="PD to back NFS storage on GKE." --size=1TB
gcloud --project=${PROJECT} compute disks create --zone=${ZONE} ${PD_DISK2} --description="PD to back NFS storage on GKE." --size=1TB
Configure the environment to use the disks.
ks param set --env=cloud kubeflow-core disks ${PD_DISK1},${PD_DISK2}
Deploy the environment
ks apply cloud
Start Juptyer
You should see your NFS volumes mounted as /mnt/${DISK_NAME}
In a Juptyer cell you can run
!df
You should see output like the following
https://github.com/jlewi/deepvariant_on_k8s
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 98884832 8336440 90532008 9% /
tmpfs 15444244 0 15444244 0% /dev
tmpfs 15444244 0 15444244 0% /sys/fs/cgroup
10.11.254.34:/export/pvc-d414c86a-e0db-11e7-a056-42010af00205 1055841280 77824 1002059776 1% /mnt/jlewi-kubeflow-test1
10.11.242.82:/export/pvc-33f0a5b3-e0dc-11e7-a056-42010af00205 1055841280 77824 1002059776 1% /mnt/jlewi-kubeflow-test2
/dev/sda1 98884832 8336440 90532008 9% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 15444244 0 15444244 0% /sys/firmware
- Here
jlewi-kubeflow-test1
andjlewi-kubeflow-test2
are the names of the PDs.
On Minikube the Virtualbox/VMware drivers for Minikube are recommended as there is a known issue between the KVM/KVM2 driver and TensorFlow Serving. The issue is tracked in kubernetes/minikube#2377.
We recommend increasing the amount of resources Minikube allocates
minikube start --cpus 4 --memory 8096 --disk-size=40g
- Minikube by default allocates 2048Mb of RAM for its VM which is not enough for JupyterHub.
- The larger disk is needed to accomodate Kubeflow's Jupyter images which are 10s of GBs due to all the extra Python libraries we include.
If you encounter a jupyter-xxxx pod in Pending status, described with:
Warning FailedScheduling 8s (x22 over 5m) default-scheduler 0/1 nodes are available: 1 Insufficient memory.
- Then try recreating your Minikube cluster (and re-apply Kubeflow using Ksonnet) with more resources (as your environment allows):
If you are running on a K8s cluster with RBAC enabled, you may get an error like the following when deploying Kubeflow:
ERROR Error updating roles kubeflow-test-infra.jupyter-role: roles.rbac.authorization.k8s.io "jupyter-role" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["*"], APIGroups:["*"], Verbs:["*"]}] user=&{[email protected] [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
This error indicates you do not have sufficient permissions. In many cases you can resolve this just by creating an appropriate clusterrole binding like so and then redeploying kubeflow
kubectl create clusterrolebinding default-admin --clusterrole=cluster-admin [email protected]
- Replace
[email protected]
with the user listed in the error message.
If you're using GKE, you may want to refer to GKE's RBAC docs to understand how RBAC interacts with IAM on GCP.
If you're having trouble spawning jupyter notebooks, check that the pod is getting scheduled
kubectl -n ${NAMESPACE} get pods
- Look for pods whose name starts with juypter
- If you are using username/password auth with Jupyter the pod will be named
jupyter-${USERNAME}
-
If you are using IAP on GKE the pod will be named
jupyter-accounts-2egoogle-2ecom-3USER-40DOMAIN-2eEXT
- Where [email protected] is the Google account used with IAP
Once you know the name of the pod do
kubectl -n ${NAMESPACE} describe pods ${PODNAME}
- Look at the events to see if there are any errors trying to schedule the pod
- One common error is not being able to schedule the pod because there aren't enough resources in the cluster.
If you are deploying Kubeflow in an OpenShift environment which encapsulates Kubernetes, you will need to adjust the security contexts for the ambassador and jupyter-hub deployments in order to get the pods to run.
oc adm policy add-scc-to-user anyuid -z ambassador
oc adm policy add-scc-to-user anyuid -z jupyter-hub
Once the anyuid policy has been set, you must delete the failing pods and allow them to be recreated in the project deployment.
You will also need to adjust the privileges of the tf-job-operator service account for TFJobs to run. Do this in the project where you are running TFJobs:
oc adm policy add-role-to-user cluster-admin -z tf-job-operator
The Docker for Mac Community Edition now ships with Kubernetes support (1.9.2) which can be enabled from their edge channel. If you decide to use this as your Kubernetes environment on Mac, you may encounter the following error when deploying Kubeflow:
ks apply default -c kubeflow-core
ERROR Attempting to deploy to environment 'default' at 'https://127.0.0.1:8443', but cannot locate a server at that address
This error is due to the fact that the default cluster installed by Docker for Mac is actually set to https://localhost:6443
. One option is to directly edit the generated environments/default/spec.json
file to set the "server" variable to the correct location, then retry the deployment. However, it is preferable to initialize your Ksonnet app using the desired kube config:
kubectl config use-context docker-for-desktop
ks init my-kubeflow
Because ksonnet uses Github to pull kubeflow, unless user specifies Github API token, it will quickly consume maximum API call quota for anonymous. To fix this issue first create Github API token using this guide, and assign this token to GITHUB_TOKEN environment variable.
export GITHUB_TOKEN=<< token >>
Kubeflow requires ksonnet version 0.9.2 or later see here. If you run ks apply
with an older version of ksonnet you will likely get the error Unknown variable: env
as illustrated below:
ks apply ${KF_ENV} -c kubeflow-core
ERROR Error reading /Users/xxx/projects/devel/go/src/github.com/kubeflow/kubeflow/my-kubeflow/environments/nocloud/main.jsonnet: /Users/xxx/projects/devel/go/src/github.com/kubeflow/kubeflow/my-kubeflow/components/kubeflow-core.jsonnet:8:49-52 Unknown variable: env
namespace: if params.namespace == "null" then env.namespace else params.namespace
You can check the ksonnet version as follows:
ks version
If your ksonnet version is lower than v0.9.2, please upgrade it and follow the user_guide to recreate the app.
Ksonnet is a command line tool that makes it easier to manage complex deployments consisting of multiple components. It is designed to work side by side with kubectl.
Ksonnet allows us to generate Kubernetes manifests from parameterized templates. This makes it easy to customize Kubernetes manifests for your particular use case. In the examples above we used this functionality to generate manifests for TfServing with a user supplied URI for the model.
One of the reasons we really like ksonnet is because it treats environment as in (dev, test, staging, prod) as a first class concept. For each environment we can easily deploy the same components but with slightly different parameters to customize it for a particular environments. We think this maps really well to common workflows. For example, this feature makes it really easy to run a job locally without GPUs for a small number of steps to make sure the code doesn't crash, and then easily move that to the Cloud to run at scale with GPUs.