recipes/Caffe2/Caffe2-GPU-Distributed/Caffe2-GPU-Distributed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caffe2 GPU Distributed\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This example demonstrates how to run standard Caffe2 [resnet50_trainer.py](https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/resnet50_trainer.py) example using Batch AI. You can run it on a single or multiple compute nodes.\n",
    "\n",
    "## Details\n",
    "\n",
    "- Standard Caffe2 sample script [resnet50_trainer.py](https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/resnet50_trainer.py) is used;\n",
    "- MNIST Dataset has been translated into a lmdb database, and can be obtained at http://download.caffe2.ai/databases/mnist-lmdb.zip;\n",
    "- NFS will be used for rendezvous temp files to coordinate between each shard/node \n",
    "- Standard output of the job will be stored on Azure File Share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Install Dependencies and Create Configuration file.\n",
    "Follow [instructions](/recipes) to install all dependencies and create configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "from datetime import datetime\n",
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "\n",
    "from azure.storage.file import FileService, FilePermissions\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# utilities.py contains helper functions used by different notebooks\n",
    "sys.path.append('../../')\n",
    "import utilities\n",
    "\n",
    "cfg = utilities.Configuration('../../configuration.json')\n",
    "client = utilities.create_batchai_client(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaicntkgpupythonsample` under your storage account.\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "azure_file_share_name = 'batchaichaisample'\n",
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.create_share(azure_file_share_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Single Node NFS\n",
    "\n",
    "- To run distributed Caffe2 training, we need to mount a single node NFS file server to all the GPU nodes, so that they can use a shared for rendezvous temp files to coordinate between each shard/node\n",
    "\n",
    "The file server will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fs_Name = 'caffe2nfs'\n",
    "parameters = models.FileServerCreateParameters(\n",
    "    location = cfg.location,\n",
    "    vm_size='Standard_D14_V2',\n",
    "    ssh_configuration=models.SshConfiguration(\n",
    "        user_account_settings=models.UserAccountSettings(\n",
    "            admin_user_name=cfg.admin,\n",
    "            admin_user_password=cfg.admin_password)),\n",
    "    data_disks=models.DataDisks(\n",
    "        disk_size_in_gb=10,\n",
    "        disk_count=2,\n",
    "        storage_account_type='Standard_LRS')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the single node NFS file server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "utilities.create_resource_group(cfg)\n",
    "_ = client.file_servers.create(cfg.resource_group, fs_Name, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us wait until the provisioning state FileServer enters \"Succeed\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FileServer /subscriptions/8001b49e-e169-4ee6-a5ff-9c3ea5385053/resourceGroups/batchaitests/providers/Microsoft.BatchAI/fileservers/caffe2nfs provisioning state = succeeded\n"
     ]
    }
   ],
   "source": [
    "fs = client.file_servers.get(cfg.resource_group, fs_Name)\n",
    "nfs_id = fs.id\n",
    "print (\"FileServer {0} provisioning state = {1}\".format(nfs_id, fs.provisioning_state.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a gpu cluster of `STANDARD_NC6` nodes. Number of nodes in the cluster is configured with `nodes_count` variable;\n",
    "- We will mount file share at folder with name `external`. Full path of this folder on a computer node will be `$AZ_BATCHAI_MOUNT_ROOT/external`;\n",
    "- We will mount NFS at folder with name `fileserver`. Full path of this folder on a computer node will be `$AZ_BATCHAI_MOUNT_ROOT/fileserver`;\n",
    "- We will call the cluster `nc6`;\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "azure_file_share = 'external'\n",
    "nfs_file_share = 'fileserver'\n",
    "nodes_count = 2\n",
    "cluster_name = 'dsvm'\n",
    "\n",
    "volumes = models.MountVolumes(\n",
    "    azure_file_shares=[\n",
    "        models.AzureFileShareReference(\n",
    "            account_name=cfg.storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=cfg.storage_account_key),\n",
    "            azure_file_url = 'https://{0}.file.core.windows.net/{1}'.format(\n",
    "                cfg.storage_account_name, azure_file_share_name),\n",
    "            relative_mount_path=azure_file_share)],\n",
    "    file_servers = [\n",
    "         models.FileServerReference(\n",
    "             file_server = models.ResourceId(nfs_id),\n",
    "             relative_mount_path = nfs_file_share,\n",
    "             mount_options = \"rw\")\n",
    "    ]\n",
    ")\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=cfg.location,\n",
    "    vm_size=\"STANDARD_NC6\",\n",
    "    virtual_machine_configuration=models.VirtualMachineConfiguration(\n",
    "        image_reference=models.ImageReference(\n",
    "            publisher=\"microsoft-ads\",\n",
    "            offer=\"linux-data-science-vm-ubuntu\",\n",
    "            sku=\"linuxdsvmubuntu\",\n",
    "            version=\"latest\")),\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    node_setup=models.NodeSetup(\n",
    "        mount_volumes=volumes\n",
    "    ),\n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=cfg.admin,\n",
    "        admin_user_password=cfg.admin_password,\n",
    "        admin_user_ssh_public_key=cfg.admin_ssh_key\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.create(cfg.resource_group, cluster_name, parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Get the just created cluster. utilities.py contains a helper function to print out all kind of nodes count in the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 0; Unusable: 0; Running: 2; Preparing: 0\n"
     ]
    }
   ],
   "source": [
    "cluster = client.clusters.get(cfg.resource_group, cluster_name)\n",
    "utilities.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy MNIST Dataset\n",
    "\n",
    "For demonstration purposes, we will download preprocessed MNIST dataset to the current directory and upload it to file share directory named `mnist`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://download.caffe2.ai/databases/mnist-lmdb.zip ...Done\n",
      "Extracting Caffe2 MNIST dataset...Done\n"
     ]
    }
   ],
   "source": [
    "mnist_dataset_url = 'http://download.caffe2.ai/databases/mnist-lmdb.zip'\n",
    "if not os.path.exists('mnist_train_lmdb') or not os.path.exists('mnist_test_lmdb'):\n",
    "    utilities.download_file(mnist_dataset_url, 'caffe2_mnist_dataset.zip')\n",
    "    print('Extracting Caffe2 MNIST dataset...', end='')\n",
    "    with zipfile.ZipFile('caffe2_mnist_dataset.zip', 'r') as z:\n",
    "        z.extractall('.')\n",
    "    print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create File Share and Upload MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnist_dataset_directory = 'mnist'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to create folders and upload files into Azure File Share - you can use [Azure Portal](https://ms.portal.azure.com), [Storage Explorer](http://storageexplorer.com/), [Azure CLI2](/azure-cli-extension) or Azure SDK for your preferable programming language.\n",
    "In this example we will use Azure SDK for python to copy files into file share."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, mnist_dataset_directory,\n",
    "    fail_on_exist=False)\n",
    "# Since uploading can take significant time, let's check first if the\n",
    "# file has been uploaded already.\n",
    "for d in ['mnist-train-nchw-lmdb', 'mnist-test-nchw-lmdb']:\n",
    "    service.create_directory(\n",
    "        azure_file_share_name, os.path.join(mnist_dataset_directory, d),\n",
    "        fail_on_exist=False)\n",
    "    for f in ['data.mdb', 'lock.mdb']:\n",
    "        if service.exists(azure_file_share_name, os.path.join(mnist_dataset_directory, d, f)):\n",
    "            continue\n",
    "        service.create_file_from_path(\n",
    "            azure_file_share_name, os.path.join(mnist_dataset_directory, d), f, os.path.join(d,f))\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Sample Script and Configure the Input Directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For each job we will create a folder containing a copy of [resnet50_trainer.py](https://github.com/chainer/chainer/blob/master/examples/mnist/train_mnist.py). This allows each job to have it's own copy of the sample script (in case you would like to change it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "caffe2SampleDir = \"Caffe2Samples\"\n",
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, caffe2SampleDir, fail_on_exist=False)\n",
    "service.create_file_from_path(\n",
    "    azure_file_share_name, caffe2SampleDir, 'resnet50_trainer.py', 'resnet50_trainer.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The job needs to know where to find resnet50_trainer.py script. So, we will configure an input directory for the script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_directories = [\n",
    "    models.InputDirectory(\n",
    "        id='SCRIPT',\n",
    "        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, caffe2SampleDir)),\n",
    "    models.InputDirectory(\n",
    "        id='DATASET',\n",
    "        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, mnist_dataset_directory))\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job will be able to reference those directories using ```$AZ_BATCHAI_INPUT_SCRIPT``` and ```$AZ_BATCHAI_INPUT_DATASET``` environment variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Output Directories\n",
    "We will store standard and error output of the job in Azure File Share:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "std_output_path_prefix = \"$AZ_BATCHAI_MOUNT_ROOT/{0}\".format(azure_file_share)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The temp output will be stored in NFS File Share:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output_directories = [\n",
    "    models.OutputDirectory(\n",
    "        id='TEMP',\n",
    "        path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(nfs_file_share),\n",
    "        path_suffix=\"temp\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Job\n",
    "- The job will use `caffe2ai/caffe2` container.\n",
    "- Will use configured previously input and output directories;\n",
    "- Will run modified `resnet50_trainer.py` from SCRIPT input directory;\n",
    "- Will output standard output and error streams to file share;\n",
    "- Will use TEMP output directory shared directory for rendezvous temp files to coordinate between each node. \n",
    "- For demostration purpose, we will only run 5 epochs with epoch size as 2000\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "job_name = datetime.utcnow().strftime(\"caffe2_%m_%d_%Y_%H%M%S\")\n",
    "parameters = models.job_create_parameters.JobCreateParameters(\n",
    "     location=cfg.location,\n",
    "     cluster=models.ResourceId(cluster.id),\n",
    "     node_count=2,\n",
    "     input_directories=input_directories,\n",
    "     std_out_err_path_prefix=std_output_path_prefix,\n",
    "     output_directories=output_directories,\n",
    "     container_settings=models.ContainerSettings(\n",
    "         models.ImageSourceRegistry(image='caffe2ai/caffe2')),\n",
    "     caffe2_settings = models.Caffe2Settings(\n",
    "         python_script_file_path='$AZ_BATCHAI_INPUT_SCRIPT/resnet50_trainer.py',\n",
    "         command_line_args='--num_shards 2 --shard_id $AZ_BATCHAI_TASK_INDEX --run_id 0 --epoch_size 2000 --num_epochs 5 --train_data $AZ_BATCHAI_INPUT_DATASET/mnist-train-nchw-lmdb --file_store_path $AZ_BATCHAI_OUTPUT_TEMP'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a training Job and wait for Job completion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Job: caffe2_09_29_2017_011958\n"
     ]
    }
   ],
   "source": [
    "_ = client.jobs.create(cfg.resource_group, job_name, parameters)\n",
    "print('Created Job: {}'.format(job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Job to Finish\n",
    "The job will start running when the cluster will have enought idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdout.txt.\n",
    "\n",
    "**Note** Execution may take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 2; Allocated: 2; Idle: 0; Unusable: 0; Running: 2; Preparing: 0\n",
      "Job state: running ExitCode: None\n",
      "Waiting for job output to become available...\n",
      "2017/09/29 01:20:16 cmd is  export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64;export PYTHONPATH=/usr/local; python /mnt/batch/tasks/shared/LS_root/mounts/external/Caffe2Samples/resnet50_trainer.py --num_shards 2 --shard_id 0 --run_id 0 --epoch_size 2000 --num_epochs 5 --train_data /mnt/batch/tasks/shared/LS_root/mounts/external/mnist/mnist-train-nchw-lmdb --file_store_path /mnt/batch/tasks/shared/LS_root/mounts/fileserver/8001b49e-e169-4ee6-a5ff-9c3ea5385053/batchaitests/jobs/caffe2_09_29_2017_011958/outputs/temp \n",
      "INFO:resnet50_trainer:Running on GPUs: [0]\n",
      "INFO:resnet50_trainer:Using epoch size: 1984\n",
      "INFO:data_parallel_model:Parallelizing model for devices: [0]\n",
      "INFO:data_parallel_model:Create input and model training operators\n",
      "INFO:data_parallel_model:Model for GPU : 0\n",
      "INFO:data_parallel_model:Adding gradient operators\n",
      "INFO:data_parallel_model:Add gradient all-reduces for SyncSGD\n",
      "WARNING:data_parallel_model:Distributed computed params all-reduce not implemented yet\n",
      "INFO:data_parallel_model:Post-iteration operators for updating params\n",
      "INFO:data_parallel_model:Calling optimizer builder function\n",
      "INFO:data_parallel_model:Add initial parameter sync\n",
      "WARNING:data_parallel_model:------- DEPRECATED API, please use data_parallel_model.OptimizeGradientMemory() ----- \n",
      "WARNING:memonger:NOTE: Executing memonger to optimize gradient memory\n",
      "INFO:memonger:Remapping 112 blobs, using 5 shared\n",
      "INFO:memonger:Memonger memory optimization took 0.0299940109253 secs\n",
      "INFO:resnet50_trainer:Starting epoch 0/5\n",
      "INFO:resnet50_trainer:Finished iteration 1/31 of epoch 0 (4.34 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 8.0980386734, accuracy: 0.0\n",
      "INFO:resnet50_trainer:Finished iteration 2/31 of epoch 0 (41.92 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 3/31 of epoch 0 (44.58 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 4/31 of epoch 0 (45.42 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 5/31 of epoch 0 (41.06 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 6/31 of epoch 0 (45.87 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 7/31 of epoch 0 (45.92 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 8/31 of epoch 0 (45.52 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 9/31 of epoch 0 (43.06 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 10/31 of epoch 0 (43.06 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 11/31 of epoch 0 (42.99 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 12/31 of epoch 0 (44.26 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 13/31 of epoch 0 (45.53 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 14/31 of epoch 0 (45.62 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 15/31 of epoch 0 (46.01 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 16/31 of epoch 0 (45.30 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 17/31 of epoch 0 (44.75 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 18/31 of epoch 0 (43.72 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 19/31 of epoch 0 (42.84 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 20/31 of epoch 0 (45.70 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 21/31 of epoch 0 (44.96 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 22/31 of epoch 0 (45.92 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 23/31 of epoch 0 (45.38 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 24/31 of epoch 0 (45.26 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 25/31 of epoch 0 (45.41 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 26/31 of epoch 0 (44.11 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 27/31 of epoch 0 (43.00 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 28/31 of epoch 0 (41.48 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 29/31 of epoch 0 (43.69 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 30/31 of epoch 0 (44.82 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 31/31 of epoch 0 (44.92 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Starting epoch 1/5\n",
      "INFO:resnet50_trainer:Finished iteration 1/31 of epoch 1 (38.74 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 2/31 of epoch 1 (41.53 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 3/31 of epoch 1 (44.06 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 4/31 of epoch 1 (44.82 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 5/31 of epoch 1 (45.28 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 6/31 of epoch 1 (44.69 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 7/31 of epoch 1 (45.31 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 8/31 of epoch 1 (44.83 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 9/31 of epoch 1 (45.37 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 10/31 of epoch 1 (43.46 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 11/31 of epoch 1 (42.42 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 12/31 of epoch 1 (44.69 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 13/31 of epoch 1 (44.09 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 14/31 of epoch 1 (46.21 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 15/31 of epoch 1 (44.50 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 16/31 of epoch 1 (43.81 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 17/31 of epoch 1 (45.40 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 18/31 of epoch 1 (46.03 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 19/31 of epoch 1 (44.05 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 20/31 of epoch 1 (43.51 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 21/31 of epoch 1 (44.18 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 22/31 of epoch 1 (45.64 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 23/31 of epoch 1 (45.03 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 24/31 of epoch 1 (45.12 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 25/31 of epoch 1 (44.74 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 26/31 of epoch 1 (45.83 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 27/31 of epoch 1 (43.20 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 28/31 of epoch 1 (45.53 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 29/31 of epoch 1 (41.33 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 30/31 of epoch 1 (43.33 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 31/31 of epoch 1 (44.93 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Starting epoch 2/5\n",
      "INFO:resnet50_trainer:Finished iteration 1/31 of epoch 2 (44.89 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 2/31 of epoch 2 (43.73 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 3/31 of epoch 2 (43.29 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 4/31 of epoch 2 (42.71 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 5/31 of epoch 2 (46.01 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 6/31 of epoch 2 (44.99 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 7/31 of epoch 2 (46.00 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 8/31 of epoch 2 (44.89 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 9/31 of epoch 2 (44.75 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 10/31 of epoch 2 (45.03 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 11/31 of epoch 2 (45.12 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 12/31 of epoch 2 (42.32 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 13/31 of epoch 2 (44.39 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 14/31 of epoch 2 (44.86 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 15/31 of epoch 2 (45.80 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 16/31 of epoch 2 (45.05 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 17/31 of epoch 2 (42.65 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 18/31 of epoch 2 (44.97 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 19/31 of epoch 2 (44.37 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 20/31 of epoch 2 (43.62 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 21/31 of epoch 2 (44.30 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 22/31 of epoch 2 (43.63 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 23/31 of epoch 2 (45.23 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 24/31 of epoch 2 (45.55 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 25/31 of epoch 2 (44.00 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 26/31 of epoch 2 (45.22 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 27/31 of epoch 2 (45.55 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 28/31 of epoch 2 (45.84 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 29/31 of epoch 2 (44.49 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 30/31 of epoch 2 (44.02 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 31/31 of epoch 2 (44.97 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Starting epoch 3/5\n",
      "INFO:resnet50_trainer:Finished iteration 1/31 of epoch 3 (38.87 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 2/31 of epoch 3 (45.41 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 3/31 of epoch 3 (42.68 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 4/31 of epoch 3 (41.24 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 5/31 of epoch 3 (42.87 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 6/31 of epoch 3 (45.00 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 7/31 of epoch 3 (45.60 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 8/31 of epoch 3 (46.19 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 9/31 of epoch 3 (44.96 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 10/31 of epoch 3 (44.48 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 11/31 of epoch 3 (44.84 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 12/31 of epoch 3 (44.89 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 13/31 of epoch 3 (43.29 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 14/31 of epoch 3 (42.90 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 15/31 of epoch 3 (41.40 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 16/31 of epoch 3 (45.09 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 17/31 of epoch 3 (44.79 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 18/31 of epoch 3 (45.18 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 19/31 of epoch 3 (45.37 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 20/31 of epoch 3 (45.25 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 21/31 of epoch 3 (43.37 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 22/31 of epoch 3 (43.51 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 23/31 of epoch 3 (44.32 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 24/31 of epoch 3 (44.29 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 25/31 of epoch 3 (45.37 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 26/31 of epoch 3 (45.27 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 27/31 of epoch 3 (45.09 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 28/31 of epoch 3 (45.02 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 29/31 of epoch 3 (45.60 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 30/31 of epoch 3 (44.92 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 31/31 of epoch 3 (41.39 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Starting epoch 4/5\n",
      "INFO:resnet50_trainer:Finished iteration 1/31 of epoch 4 (45.05 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 2/31 of epoch 4 (44.77 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 3/31 of epoch 4 (44.03 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 4/31 of epoch 4 (45.21 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 5/31 of epoch 4 (42.67 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 6/31 of epoch 4 (42.10 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 7/31 of epoch 4 (44.59 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 8/31 of epoch 4 (45.03 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 9/31 of epoch 4 (44.06 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 10/31 of epoch 4 (44.46 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 11/31 of epoch 4 (44.40 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 12/31 of epoch 4 (45.10 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 13/31 of epoch 4 (44.88 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 14/31 of epoch 4 (44.79 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 15/31 of epoch 4 (42.57 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 16/31 of epoch 4 (44.07 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 17/31 of epoch 4 (45.47 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 18/31 of epoch 4 (46.04 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 19/31 of epoch 4 (44.96 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 20/31 of epoch 4 (43.22 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 21/31 of epoch 4 (44.42 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 22/31 of epoch 4 (44.85 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 23/31 of epoch 4 (43.21 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 24/31 of epoch 4 (43.09 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 25/31 of epoch 4 (41.87 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 26/31 of epoch 4 (44.83 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 27/31 of epoch 4 (45.04 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 28/31 of epoch 4 (44.67 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 29/31 of epoch 4 (45.22 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 30/31 of epoch 4 (45.57 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "INFO:resnet50_trainer:Finished iteration 31/31 of epoch 4 (44.18 images/sec)\n",
      "INFO:resnet50_trainer:Training loss: 0.0, accuracy: 1.0\n",
      "Job state: succeeded ExitCode: 0\n"
     ]
    }
   ],
   "source": [
    "utilities.wait_for_job_completion(client, cfg.resource_group, job_name, cluster_name, 'stdouterr', 'stderr-0.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Download stdout.txt and stderr.txt files for the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "files = client.jobs.list_output_files(cfg.resource_group, job_name, models.JobsListOutputFilesOptions(\"stdOuterr\")) \n",
    "for file in list(files):\n",
    "    utilities.download_file(file.download_url, file.name)\n",
    "print(\"All files Downloaded\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for n in range(nodes_count):\n",
    "    print('stdout-{0}.txt content:'.format(n))\n",
    "    with open('stderr-{0}.txt'.format(n)) as f:\n",
    "        print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "client.jobs.delete(cfg.resource_group, job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "client.clusters.delete(cfg.resource_group, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "service = FileService(cfg.storage_account_name, cfg.storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}