recipes/Keras/Keras-GPU/Keras-GPU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras GPU\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This recipe shows how to run Keras using Batch AI. Keras supports tensorflow, cntk and theano backends. Currently only tensorflow and cntk backends supports running on GPU. Batch AI will automatic setup backend when toolkit is specified.\n",
    "\n",
    "## Details\n",
    "- Keras can run with CNTK or Tensorflow backend.\n",
    "- Standard keras sample script [mnist_cnn.py](https://raw.githubusercontent.com/fchollet/keras/master/examples/mnist_cnn.py) is used;\n",
    "- The script downloads the standard MNIST Database on its own;\n",
    "- Standard output of the job will be stored on Azure File Share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Install Dependencies and Create Configuration file.\n",
    "Follow [instructions](/recipes) to install all dependencies and create configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from datetime import datetime\n",
    "import sys\n",
    "\n",
    "from azure.storage.file import FileService\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# The BatchAI/utilities folder contains helper functions used by different notebooks\n",
    "sys.path.append('../../../')\n",
    "import utilities as utils\n",
    "\n",
    "cfg = utils.config.Configuration('../../configuration.json')\n",
    "client = utils.config.create_batchai_client(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create Resoruce Group and Batch AI workspace if not exists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "utils.config.create_resource_group(cfg)\n",
    "_ = client.workspaces.create(cfg.resource_group, cfg.workspace, cfg.location).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prepare Training Dataset and Script in Azure Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaidsvmsample` under your storage account.\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "azure_file_share_name = 'batchaisample'\n",
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.create_share(azure_file_share_name, fail_on_exist=False)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Sample Script and Configure the Input Directories\n",
    "\n",
    "- Download original sample script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sample_script_url = 'https://raw.githubusercontent.com/fchollet/keras/master/examples/mnist_cnn.py'\n",
    "utils.dataset.download_file(sample_script_url, 'mnist_cnn.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For each job we will create a folder containing a copy of [train_mnist.py](https://github.com/chainer/chainer/blob/master/examples/mnist/train_mnist.py). This allows each job to have it's own copy of the sample script (in case you would like to change it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "keras_sample_dir = \"KerasSamples\"\n",
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, keras_sample_dir, fail_on_exist=False)\n",
    "service.create_file_from_path(\n",
    "    azure_file_share_name, keras_sample_dir, 'mnist_cnn.py', 'mnist_cnn.py')\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a GPU cluster of `STANDARD_NC6` nodes. Number of nodes in the cluster is configured with `nodes_count` variable;\n",
    "- We will mount file share at folder with name `afs`. Full path of this folder on a computer node will be `$AZ_BATCHAI_MOUNT_ROOT/afs`;\n",
    "- We will call the cluster `nc6`;\n",
    "\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nodes_count = 1\n",
    "cluster_name = 'nc6'\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=cfg.location,\n",
    "    vm_size='STANDARD_NC6',\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=cfg.admin,\n",
    "        admin_user_password=cfg.admin_password or None,\n",
    "        admin_user_ssh_public_key=cfg.admin_ssh_key or None,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.create(cfg.resource_group, cfg.workspace, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "The `utilities` module contains a helper function allowing to wait for the cluster to become available - all nodes are allocated and finished preparation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cluster = client.clusters.get(cfg.resource_group, cfg.workspace, cluster_name)\n",
    "utils.cluster.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Job\n",
    "\n",
    "- Will use configured previously input and output directories;\n",
    "- Will run standard `mnist_cnn.py` from SCRIPT input directory using custom framework;\n",
    "- Will output standard output and error streams to file share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please choose what backend of Keras will be used: `'cntk'` or `'tensorflow'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "backend = 'cntk'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `'cntk'` backend is used:\n",
    "- The job will use `microsoft/2.5.1-gpu-python2.7-cuda9.0-cudnn7.0` container.\n",
    "- Keras framework has been preinstalled in the container\n",
    "- The job needs to have `cntk_settings` to be configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if backend == 'cntk':\n",
    "    parameters = models.JobCreateParameters(\n",
    "         location=cfg.location,\n",
    "         cluster=models.ResourceId(id=cluster.id),\n",
    "         node_count=1,\n",
    "         container_settings=models.ContainerSettings(\n",
    "             image_source_registry=models.ImageSourceRegistry(image='microsoft/cntk:2.5.1-gpu-python2.7-cuda9.0-cudnn7.0')),\n",
    "         mount_volumes=models.MountVolumes(\n",
    "            azure_file_shares=[\n",
    "                models.AzureFileShareReference(\n",
    "                    account_name=cfg.storage_account_name,\n",
    "                    credentials=models.AzureStorageCredentialsInfo(\n",
    "                        account_key=cfg.storage_account_key),\n",
    "                    azure_file_url='https://{0}.file.core.windows.net/{1}'.format(\n",
    "                        cfg.storage_account_name, azure_file_share_name),\n",
    "                    relative_mount_path='afs')\n",
    "            ]\n",
    "         ),\n",
    "         std_out_err_path_prefix='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}'.format('afs'),\n",
    "         cntk_settings = models.CNTKsettings(\n",
    "             python_script_file_path='$AZ_BATCHAI_JOB_MOUNT_ROOT/afs/{0}/mnist_cnn.py'.format(keras_sample_dir)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `'tensorflow'` backend is used:\n",
    "- The job will use `tensorflow/tensorflow:1.8.0-gpu` container.\n",
    "- Keras framework will be installed by job preparation command line.\n",
    "- The job needs to have `tensor_flow_settings` to be configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if backend == 'tensorflow':\n",
    "    parameters = models.JobCreateParameters(\n",
    "         location=cfg.location,\n",
    "         cluster=models.ResourceId(id=cluster.id),\n",
    "         node_count=1,\n",
    "         job_preparation=models.JobPreparation(command_line='pip install keras'),\n",
    "         container_settings=models.ContainerSettings(\n",
    "             image_source_registry=models.ImageSourceRegistry(image='tensorflow/tensorflow:1.8.0-gpu')),\n",
    "         mount_volumes=models.MountVolumes(\n",
    "            azure_file_shares=[\n",
    "                models.AzureFileShareReference(\n",
    "                    account_name=cfg.storage_account_name,\n",
    "                    credentials=models.AzureStorageCredentialsInfo(\n",
    "                        account_key=cfg.storage_account_key),\n",
    "                    azure_file_url='https://{0}.file.core.windows.net/{1}'.format(\n",
    "                        cfg.storage_account_name, azure_file_share_name),\n",
    "                    relative_mount_path='afs')\n",
    "            ]\n",
    "         ),\n",
    "         std_out_err_path_prefix='$AZ_BATCHAI_JOB_MOUNT_ROOT/{0}'.format('afs'),\n",
    "         tensor_flow_settings=models.TensorFlowSettings(\n",
    "             python_script_file_path='$AZ_BATCHAI_JOB_MOUNT_ROOT/afs/{0}/mnist_cnn.py'.format(keras_sample_dir)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a training Job and wait for Job completion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "experiment_name = 'keras_experiment'\n",
    "experiment = client.experiments.create(cfg.resource_group, cfg.workspace, experiment_name).result()\n",
    "job_name = datetime.utcnow().strftime('keras_{}_%m_%d_%Y_%H%M%S'.format(backend))\n",
    "job = client.jobs.create(cfg.resource_group, cfg.workspace, experiment_name, job_name, parameters).result()\n",
    "print('Created Job {0} in Experiment {1}'.format(job.name, experiment.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Job to Finish\n",
    "The job will start running when the cluster will have enough idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdout file.\n",
    "\n",
    "**Note** Execution may take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if backend == 'tensorflow':\n",
    "    read_file = 'stdout-wk-0.txt'\n",
    "elif backend == 'cntk':\n",
    "    read_file = 'stdout.txt'\n",
    "\n",
    "utils.job.wait_for_job_completion(client, cfg.resource_group, cfg.workspace, \n",
    "                                  experiment_name, job_name, cluster_name, 'stdouterr', read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### List log files for the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "files = client.jobs.list_output_files(cfg.resource_group, cfg.workspace, experiment_name, job_name,\n",
    "                                      models.JobsListOutputFilesOptions(outputdirectoryid='stdouterr')) \n",
    "for f in list(files):\n",
    "    print(f.name, f.download_url or 'directory')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean Up (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.jobs.delete(cfg.resource_group, cfg.workspace, experiment_name, job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(cfg.resource_group, cfg.workspace, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}