etc/notebooks/art/ART_model_robustness_check.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Adversarial Samples for Deep Learning Models with the Adversarial Robustness Toolbox (ART)\n",
    "\n",
    "This notebook shows how to use adversarial attack techniques from the [Adversarial Robustness Toolbox (ART)](https://developer.ibm.com/code/open/projects/adversarial-robustness-toolbox/) on Deep Learning models trained with *FfDL*. The *ART* library supports crafting and analyzing various attack and defense methods for deep learning models. \n",
    "\n",
    "In this notebook, you will learn how to incorporate one of the attack methods supported by *ART*, the *Fast Gradient Method* (*FGM*), into your training pipeline to generate adversarial samples for the purposes of evaluating the robustness of the trained model. The model is a Convolutional Neural Network (CNN) trained on the *[MNIST handwritten digit data](http://yann.lecun.com/exdb/mnist/)* using [Keras](https://keras.io/) with a [TensorFlow](https://www.tensorflow.org/) backend.\n",
    "\n",
    "The *ART* Github repository can be found here - https://github.com/IBM/adversarial-robustness-toolbox\n",
    "\n",
    "This notebook uses Python 3.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "1.\t[Set up the environment](#setup)\n",
    "2.\t[Create a Keras model](#model)\n",
    "3.  [Train the model](#train)\n",
    "4.\t[Generate adversarial samples for a robustness check](#art)\n",
    "5.\t[Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Setup\n",
    "\n",
    "It is recommended that you run this notebook inside a Python 3 virtual environment. Make sure you have all required libraries installed.\n",
    "\n",
    "To store model and training data, this notebook requires access to a Cloud Object Storage (COS) instance. [BlueMix Cloud Object Storage](https://console.bluemix.net/catalog/services/cloud-object-storage) offers a free *lite plan*. Follow [these instructions](https://dataplatform.ibm.com/docs/content/analyze-data/ml_dlaas_object_store.html) to create your COS instance and generate [service credentials](https://console.bluemix.net/docs/services/cloud-object-storage/iam/service-credentials.html#service-credentials) with [HMAC keys](https://console.bluemix.net/docs/services/cloud-object-storage/hmac/credentials.html#using-hmac-credentials).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Enter your cluster and object storage information:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "env = dict(os.environ)\n",
    "\n",
    "user_data = {\n",
    "    \"public_ip\"                  : env.get(\"PUBLIC_IP\"),             # Public IP of your Kubernetes cluster with FfDL deployed on it \n",
    "    \"kubeconfig_file\"            : env.get(\"KUBECONFIG\"),            # Path pointing to your Kubernetes cluster configuration file \n",
    "    \"cos_hmac_access_key_id\"     : env.get(\"AWS_ACCESS_KEY_ID\"),     # Cloud Object Storage (AWS) Access Key ID \n",
    "    \"cos_hmac_secret_access_key\" : env.get(\"AWS_SECRET_ACCESS_KEY\"), # Cloud Object Storage (AWS) Secret Access Key \n",
    "    \"cos_service_endpoint\"       : env.get(\"AWS_ENDPOINT_URL\"),      # Cloud Object Storage endpoint, i.e. 'https://s3-api.us-geo.objectstorage.softlayer.net' \n",
    "    \"cos_region_name\"            : env.get(\"AWS_DEFAULT_REGION\")     # Cloud Object Storage default region, i.e. 'us-east-1' \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "unset_vars = [key for (key, value) in user_data.items() if not value]\n",
    "\n",
    "for var in unset_vars:\n",
    "    print(\"Dictionary 'user_data' is missing '%s'\" % var)\n",
    "    \n",
    "assert not unset_vars, \"Enter 'user_data' to run this notebook!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Verify or Install Required Python Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All required libraries are installed.\n",
      "keras>=2.1.6\r\n",
      "tensorflow>=1.8\r\n",
      "ipython>=5.0.0\r\n",
      "jupyter>=1.0.0\r\n",
      "requests>=2.12.0,<=2.18.4\r\n",
      "wget\r\n",
      "boto3\r\n",
      "git+git://github.com/IBM/adversarial-robustness-toolbox@master\r\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "def is_venv():\n",
    "    return (hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix))\n",
    "\n",
    "try:\n",
    "    import keras, tensorflow, requests, wget, boto3, art\n",
    "    print(\"All required libraries are installed.\")\n",
    "    !cat requirements.txt\n",
    "except ModuleNotFoundError:\n",
    "    if is_venv:\n",
    "        print(\"Installing required libraries into virtual environment.\")\n",
    "        !python -m pip install -r requirements.txt\n",
    "    else:\n",
    "        print(\"Please install the required libraries.\")\n",
    "        !cat requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Connect to Cloud Object Storage  (COS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `boto3.resource` to interact with the COS instance. The `boto3` library allows Python developers to manage Cloud Object Storage (COS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos = boto3.resource(\"s3\", \n",
    "                     aws_access_key_id     = user_data[\"cos_hmac_access_key_id\"],\n",
    "                     aws_secret_access_key = user_data[\"cos_hmac_secret_access_key\"],\n",
    "                     endpoint_url          = user_data[\"cos_service_endpoint\"],\n",
    "                     region_name           = user_data[\"cos_region_name\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for bucket in cos.buckets.all():\n",
    "#     print(bucket.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create two buckets, which you will use to store training data and training results.\n",
    "\n",
    "**Note:** The bucket names must be unique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating bucket \"training-data-d7e9e034-b6ee-4031-a4f3-93521829c3ee\" ...\n",
      "Creating bucket \"training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee\" ...\n"
     ]
    }
   ],
   "source": [
    "from uuid import uuid4\n",
    "\n",
    "bucket_uid             = str(uuid4())\n",
    "training_data_bucket   = 'training-data-' + bucket_uid\n",
    "training_result_bucket = 'training-results-' + bucket_uid\n",
    "\n",
    "def create_buckets(bucket_names):\n",
    "    for bucket in bucket_names:\n",
    "        print('Creating bucket \"{}\" ...'.format(bucket))\n",
    "        try:\n",
    "            cos.create_bucket(Bucket=bucket)\n",
    "        except boto3.exceptions.botocore.client.ClientError as e:\n",
    "            print('Error: {}.'.format(e.response['Error']['Message']))\n",
    "\n",
    "buckets = [training_data_bucket, training_result_bucket]\n",
    "\n",
    "create_buckets(buckets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should have 2 buckets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Download MNIST Training Data and Upload it to the COS Buckets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the training data and upload it to the `training-data` bucket.\n",
    "First, create a list of links for the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training data links\n",
    "data_links = ['https://s3.amazonaws.com/img-datasets/mnist.npz']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in the next cell uploads files from links to your COS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading data mnist.npz ...\n",
      "mnist.npz is uploaded.\n"
     ]
    }
   ],
   "source": [
    "# Upload files to COS\n",
    "from urllib.request import urlopen\n",
    "\n",
    "bucket_obj = cos.Bucket(training_data_bucket)\n",
    "\n",
    "for data_link in data_links:\n",
    "    filename = data_link.split('/')[-1]\n",
    "    print('Uploading data {} ...'.format(filename))\n",
    "    with urlopen(data_link) as data:\n",
    "        bucket_obj.upload_fileobj(data, filename)\n",
    "        print('{} is uploaded.'.format(filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at the list of the created buckets and their contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-data-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n",
      "  File: mnist.npz, 11221.13kB\n",
      "training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n"
     ]
    }
   ],
   "source": [
    "def print_bucket_contents(buckets):\n",
    "    for bucket_name in buckets:\n",
    "        print(bucket_name)\n",
    "        bucket_obj = cos.Bucket(bucket_name)\n",
    "        for obj in bucket_obj.objects.all():\n",
    "            print(\"  File: {}, {:4.2f}kB\".format(obj.key, obj.size/1024))\n",
    "\n",
    "print_bucket_contents(buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are done with COS, and you are ready to train your model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"model\"></a>\n",
    "## 2. Create the Keras model\n",
    "\n",
    "In this section we:\n",
    "\n",
    "- [2.1 Package the model definition](#zip)\n",
    "- [2.2 Prepare the training definition metadata](#manifest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Create the Model Zip File <a id=\"zip\"></a>\n",
    "\n",
    "Let's create the model [`convolutional_keras.py`](../edit/convolutional_keras.py) and add it to a zip file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_filename  = \"convolutional_keras.py\"\n",
    "archive_filename = 'model.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing convolutional_keras.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $script_filename\n",
    "\n",
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "import keras\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 1\n",
    "\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "\n",
    "def main(argv):\n",
    "    if len(argv) < 2:\n",
    "        sys.exit(\"Not enough arguments provided.\")\n",
    "    global image_path\n",
    "    i = 1\n",
    "    while i <= 2:\n",
    "        arg = str(argv[i])\n",
    "        if arg == \"--data\":\n",
    "            image_path = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        i += 2\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main(sys.argv)\n",
    "\n",
    "\n",
    "# load mnist npz file\n",
    "f = np.load(image_path)\n",
    "x_train = f['x_train']\n",
    "y_train = f['y_train']\n",
    "x_test = f['x_test']\n",
    "y_test = f['y_test']\n",
    "f.close()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "# model\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_split=0.1)\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "\n",
    "model_wt_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.hdf5\"\n",
    "model.save(model_wt_path)\n",
    "print(\"Model saved to file: %s\" % model_wt_path)\n",
    "\n",
    "model_def_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.json\"\n",
    "model_json = model.to_json()\n",
    "with open(model_def_path, \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "print(\"Model definition saved to file: %s\" % model_def_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  adding: convolutional_keras.py (deflated 60%)\r\n"
     ]
    }
   ],
   "source": [
    "import zipfile\n",
    "\n",
    "zipfile.ZipFile(archive_filename, mode='w').write(script_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Prepare the Training Definition Metadata <a id=\"manifest\"></a>\n",
    "- *FfDL* does not have a *Keras* community image so we need to `pip`-install *Keras* prior to running the `training_command` \n",
    "- Your COS credentials are referenced in the `data_stores` > `connection` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "training_command = \"pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/mnist.npz\"\n",
    "\n",
    "manifest = {\n",
    "  \"name\":        \"keras_digit_recognition\",\n",
    "  \"description\": \"Hand-written Digit Recognition Training\",\n",
    "  \"version\":     \"1.0\",\n",
    "  \"memory\":      \"2Gb\",\n",
    "  \"gpus\": 0,\n",
    "  \"cpus\": 2,\n",
    "  \"data_stores\": [\n",
    "    {\n",
    "      \"id\":   \"s3-art-notebook-files\",\n",
    "      \"type\": \"mount_cos\",\n",
    "      \"training_data\": {\n",
    "        \"container\": training_data_bucket\n",
    "      },\n",
    "      \"training_results\": {\n",
    "        \"container\": training_result_bucket\n",
    "      },\n",
    "      \"connection\": {\n",
    "        \"auth_url\":  user_data[\"cos_service_endpoint\"],\n",
    "        \"user_name\": user_data[\"cos_hmac_access_key_id\"],\n",
    "        \"password\":  user_data[\"cos_hmac_secret_access_key\"]\n",
    "      }\n",
    "    }\n",
    "  ],\n",
    "  \"framework\": {\n",
    "    \"name\":    \"tensorflow\",\n",
    "    \"version\": \"1.5.0-py3\",\n",
    "    \"command\": training_command\n",
    "  },\n",
    "  \"evaluation_metrics\": {\n",
    "    \"type\": \"tensorboard\",\n",
    "    \"in\":   \"$JOB_STATE_DIR/logs/tb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "yaml.dump(manifest, open(\"manifest.yml\", \"w\"), default_flow_style=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Train the Model<a id=\"train\"></a>\n",
    "\n",
    "In this section, learn how to:\n",
    "- [3.1 Setup the command line environment](#cmd_setup)\n",
    "- [3.2 Train the model in the background](#backg)\n",
    "- [3.3 Monitor the training log](#log)\n",
    "- [3.4 Cancel the training](#cancel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Setup the Command Line Environment <a id=\"cmd_setup\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PUBLIC_IP=169.60.36.238\n",
      "env: KUBECONFIG=~/.bluemix/plugins/container-service/clusters/ffdl-with-art-cluster/kube-config-dal12-ffdl-with-art-cluster.yml\n"
     ]
    }
   ],
   "source": [
    "%env PUBLIC_IP  {user_data[\"public_ip\"]}\n",
    "%env KUBECONFIG {user_data[\"kubeconfig_file\"]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the DLaaS URL, username and password"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: DLAAS_URL=http://169.60.36.238:30154\n",
      "env: DLAAS_USERNAME=test-user\n",
      "env: DLAAS_PASSWORD=test\n"
     ]
    }
   ],
   "source": [
    "restapi_port = !kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}'\n",
    "dlaas_url    = \"http://%s:%s\" % (user_data[\"public_ip\"], restapi_port[0])\n",
    "\n",
    "%env DLAAS_URL        $dlaas_url\n",
    "%env DLAAS_USERNAME = test-user\n",
    "%env DLAAS_PASSWORD = test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain the correct FfDL CLI for your machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "\n",
    "ffdl_dir = os.getcwd().replace(\"/etc/notebooks/art\", \"\")\n",
    "ffdl     = \"%s/cli/bin/ffdl-%s\" % (ffdl_dir, \"osx\" if platform.system() == \"Darwin\" else \"linux\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Start the Training Job<a id=\"backg\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Deploying model with manifest 'manifest.yml' and model file 'model.zip'...\",\n",
       " 'Model ID: training-mSVi6tdiR',\n",
       " 'OK']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = !{ffdl} train \"manifest.yml\" \"model.zip\"\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.  Monitor the Training Logs<a id=\"log\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting model training logs for '\u001b[1;36mtraining-mSVi6tdiR\u001b[0m'...\n",
      "Training with training/test data at:\n",
      "  DATA_DIR: /mnt/data/training-data-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n",
      "  MODEL_DIR: /job/model-code\n",
      "  TRAINING_JOB: \n",
      "  TRAINING_COMMAND: pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/mnist.npz\n",
      "Storing trained model at:\n",
      "  RESULT_DIR: /mnt/results/training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee/training-mSVi6tdiR\n",
      "Contents of $MODEL_DIR\n",
      "total 12\n",
      "drwxrwxrwx 2 6342627 4294967294 4096 Jul 14 01:35 .\n",
      "drwxrwxrwx 4 nobody  4294967294 4096 Jul 14 01:35 ..\n",
      "-rwxrwxrwx 1 6342627 4294967294 2649 Jul 13 18:34 convolutional_keras.py\n",
      "Contents of $DATA_DIR\n",
      "total 11226\n",
      "drwxrwxr-x 1 root root        0 Jan  1  1970 .\n",
      "drwxr-xr-x 3 root root     4096 Jul 14 01:34 ..\n",
      "---------- 1 root root 11490434 Jul 14 01:34 mnist.npz\n",
      "CHECKPOINT_DIR=/mnt/results/training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee/_wml_checkpoints\n",
      "DATA_DIR=/mnt/data/training-data-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n",
      "DOWNWARD_API_POD_NAME=learner-c74f93c5-8d4e-4e5e-666b-735a1c4a401f-0\n",
      "DOWNWARD_API_POD_NAMESPACE=default\n",
      "ELASTICSEARCH_PORT=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_ADDR=172.21.204.56\n",
      "ELASTICSEARCH_PORT_9200_TCP_PORT=9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp\n",
      "ELASTICSEARCH_SERVICE_HOST=172.21.204.56\n",
      "ELASTICSEARCH_SERVICE_PORT=9200\n",
      "ELASTICSEARCH_SERVICE_PORT_HTTP=9200\n",
      "FFDL_LCM_PORT=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP_ADDR=172.21.37.214\n",
      "FFDL_LCM_PORT_80_TCP_PORT=80\n",
      "FFDL_LCM_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_LCM_SERVICE_HOST=172.21.37.214\n",
      "FFDL_LCM_SERVICE_PORT=80\n",
      "FFDL_LCM_SERVICE_PORT_GRPC=80\n",
      "FFDL_RESTAPI_PORT=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP_ADDR=172.21.137.113\n",
      "FFDL_RESTAPI_PORT_80_TCP_PORT=80\n",
      "FFDL_RESTAPI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_RESTAPI_SERVICE_HOST=172.21.137.113\n",
      "FFDL_RESTAPI_SERVICE_PORT=80\n",
      "FFDL_RESTAPI_SERVICE_PORT_FFDL=80\n",
      "FFDL_TRAINER_PORT=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP_ADDR=172.21.146.32\n",
      "FFDL_TRAINER_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAINER_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAINER_SERVICE_HOST=172.21.146.32\n",
      "FFDL_TRAINER_SERVICE_PORT=80\n",
      "FFDL_TRAINER_SERVICE_PORT_GRPC=80\n",
      "FFDL_TRAININGDATA_PORT=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_ADDR=172.21.180.242\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAININGDATA_SERVICE_HOST=172.21.180.242\n",
      "FFDL_TRAININGDATA_SERVICE_PORT=80\n",
      "FFDL_TRAININGDATA_SERVICE_PORT_GRPC=80\n",
      "FFDL_UI_PORT=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP_ADDR=172.21.53.228\n",
      "FFDL_UI_PORT_80_TCP_PORT=80\n",
      "FFDL_UI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_UI_SERVICE_HOST=172.21.53.228\n",
      "FFDL_UI_SERVICE_PORT=80\n",
      "FFDL_UI_SERVICE_PORT_HTTP=80\n",
      "GPU_COUNT=0.000000\n",
      "HOME=/root\n",
      "JOB_STATE_DIR=/job\n",
      "LEARNER_ID=1\n",
      "LEARNER_NAME_PREFIX=learner-c74f93c5-8d4e-4e5e-666b-735a1c4a401f\n",
      "LOG_DIR=/job/logs\n",
      "MODEL_DIR=/job/model-code\n",
      "NUM_LEARNERS=1\n",
      "OLDPWD=/notebooks\n",
      "PATH=/usr/local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n",
      "PROMETHEUS_PORT=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP_ADDR=172.21.74.212\n",
      "PROMETHEUS_PORT_9090_TCP_PORT=9090\n",
      "PROMETHEUS_PORT_9090_TCP_PROTO=tcp\n",
      "PROMETHEUS_SERVICE_HOST=172.21.74.212\n",
      "PROMETHEUS_SERVICE_PORT=9090\n",
      "PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090\n",
      "PWD=/job/model-code\n",
      "PYTHONPATH=:/job/model-code\n",
      "RESULT_DIR=/mnt/results/training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee/training-mSVi6tdiR\n",
      "S3_PORT=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP_ADDR=172.21.46.129\n",
      "S3_PORT_80_TCP_PORT=80\n",
      "S3_PORT_80_TCP_PROTO=tcp\n",
      "S3_SERVICE_HOST=172.21.46.129\n",
      "S3_SERVICE_PORT=80\n",
      "SHLVL=3\n",
      "TRAINING_COMMAND=pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/mnist.npz\n",
      "TRAINING_ID=training-mSVi6tdiR\n",
      "_=/usr/bin/env\n",
      "Sat Jul 14 01:35:49 UTC 2018: Running training job\n",
      "Collecting keras\n",
      "  Downloading https://files.pythonhosted.org/packages/68/12/4cabc5c01451eb3b413d19ea151f36e33026fc0efb932bf51bcaf54acbf5/Keras-2.2.0-py2.py3-none-any.whl (300kB)\n",
      "Collecting pyyaml (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting keras-preprocessing==1.0.1 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/f8/33/275506afe1d96b221f66f95adba94d1b73f6b6087cfb6132a5655b6fe338/Keras_Preprocessing-1.0.1-py2.py3-none-any.whl\n",
      "Collecting keras-applications==1.0.2 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/e2/60/c557075e586e968d7a9c314aa38c236b37cb3ee6b37e8d57152b1a5e0b47/Keras_Applications-1.0.2-py2.py3-none-any.whl (43kB)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Building wheels for collected packages: pyyaml\n",
      "  Running setup.py bdist_wheel for pyyaml: started\n",
      "  Running setup.py bdist_wheel for pyyaml: finished with status 'done'\n",
      "  Stored in directory: /root/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f\n",
      "Successfully built pyyaml\n",
      "Installing collected packages: pyyaml, keras-preprocessing, keras-applications, keras\n",
      "Successfully installed keras-2.2.0 keras-applications-1.0.2 keras-preprocessing-1.0.1 pyyaml-3.13\n",
      "You are using pip version 9.0.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "2018-07-14 01:35:56.196549: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "Train on 54000 samples, validate on 6000 samples\n",
      "Epoch 1/1\n",
      "\n",
      "  128/54000 [..............................] - ETA: 5:19 - loss: 2.2974 - acc: 0.1016\n",
      "  256/54000 [..............................] - ETA: 4:34 - loss: 2.2476 - acc: 0.1484\n",
      "  384/54000 [..............................] - ETA: 4:20 - loss: 2.2112 - acc: 0.1849\n",
      "  512/54000 [..............................] - ETA: 4:11 - loss: 2.1647 - acc: 0.2285\n",
      "  640/54000 [..............................] - ETA: 4:07 - loss: 2.1029 - acc: 0.2609\n",
      "  768/54000 [..............................] - ETA: 4:06 - loss: 2.0633 - acc: 0.2812\n",
      "  896/54000 [..............................] - ETA: 4:03 - loss: 2.0218 - acc: 0.3080\n",
      " 1024/54000 [..............................] - ETA: 4:03 - loss: 1.9693 - acc: 0.3281\n",
      " 1152/54000 [..............................] - ETA: 4:02 - loss: 1.9130 - acc: 0.3490\n",
      " 1280/54000 [..............................] - ETA: 4:01 - loss: 1.9061 - acc: 0.3633\n",
      " 1408/54000 [..............................] - ETA: 3:59 - loss: 1.8777 - acc: 0.3764\n",
      " 1536/54000 [..............................] - ETA: 4:01 - loss: 1.8276 - acc: 0.3926\n",
      " 1664/54000 [..............................] - ETA: 3:59 - loss: 1.7747 - acc: 0.4075\n",
      " 1792/54000 [..............................] - ETA: 3:58 - loss: 1.7580 - acc: 0.4163\n",
      " 1920/54000 [>.............................] - ETA: 3:58 - loss: 1.7196 - acc: 0.4302\n",
      " 2048/54000 [>.............................] - ETA: 3:57 - loss: 1.6741 - acc: 0.4443\n",
      " 2176/54000 [>.............................] - ETA: 3:57 - loss: 1.6380 - acc: 0.4568\n",
      " 2304/54000 [>.............................] - ETA: 3:56 - loss: 1.6123 - acc: 0.4666\n",
      " 2432/54000 [>.............................] - ETA: 3:54 - loss: 1.5801 - acc: 0.4770\n",
      " 2560/54000 [>.............................] - ETA: 3:53 - loss: 1.5462 - acc: 0.4863\n",
      " 2688/54000 [>.............................] - ETA: 3:52 - loss: 1.5156 - acc: 0.4981\n",
      " 2816/54000 [>.............................] - ETA: 3:52 - loss: 1.4857 - acc: 0.5089\n",
      " 2944/54000 [>.............................] - ETA: 3:51 - loss: 1.4548 - acc: 0.5183\n",
      " 3072/54000 [>.............................] - ETA: 3:50 - loss: 1.4268 - acc: 0.5270\n",
      " 3200/54000 [>... \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             ... ========================>.] - ETA: 2s - loss: 0.2852 - acc: 0.9114\n",
      "53632/54000 [============================>.] - ETA: 1s - loss: 0.2848 - acc: 0.9115\n",
      "53760/54000 [============================>.] - ETA: 1s - loss: 0.2843 - acc: 0.9116\n",
      "53888/54000 [============================>.] - ETA: 0s - loss: 0.2839 - acc: 0.9118\n",
      "54000/54000 [==============================] - 253s 5ms/step - loss: 0.2835 - acc: 0.9119 - val_loss: 0.0570 - val_acc: 0.9842\n",
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "Test loss: 0.060927866772469134\n",
      "Test accuracy: 0.9821\n",
      "Model saved to file: /mnt/results/training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee/training-mSVi6tdiR/keras_original_model.hdf5\n",
      "Model definition saved to file: /mnt/results/training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee/training-mSVi6tdiR/keras_original_model.json\n",
      "Training process finished. Exit code: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if \"Model ID\" in out[1]:\n",
    "    model_id = out.fields()[1][-1]\n",
    "    !{ffdl} logs --follow {model_id}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting all models ...\n",
      "\u001b[1;38mID\u001b[0m                   \u001b[1;38mName\u001b[0m                      \u001b[1;38mFramework\u001b[0m              \u001b[1;38mTraining status\u001b[0m   \u001b[1;38mSubmitted\u001b[0m   \u001b[1;38mCompleted\u001b[0m   \n",
      "\u001b[1;36mtraining-mSVi6tdiR\u001b[0m   keras_digit_recognition   tensorflow:1.5.0-py3   COMPLETED         N/A         N/A   \n",
      "\n",
      "1 records found.\n"
     ]
    }
   ],
   "source": [
    "!{ffdl} list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Generate Adversarial Samples <a id=\"art\"></a>\n",
    "\n",
    "In this section, we learn how to:\n",
    "- [4.1 Generate adversarial samples with ART (synchronously in notebook)](#artLocal)\n",
    "- [4.2 Generate adversarial samples with ART (asynchronously using FfDL)](#artWithFfDL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Generate Adversarial Samples Locally <a id=\"artLocal\"></a>\n",
    "\n",
    "This section shows how to use the ART Fast Gradient Method (FGM) to generate adversarial samples for the model previously trained synchronously in this notebook. \n",
    "\n",
    "A trained model should have been created in the `training_result_bucket`. Now ART can be used to check the robustness of the trained model. \n",
    "\n",
    "The original dataset used to train the model as well as the trained model serve as inputs to the `robustness_check.py` script. We can download both from the `training_data_bucket` and the `training_result_bucket` respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download the original data set and the trained model from Cloud Object Store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_filename = \"mnist.npz\"\n",
    "weights_filename = \"keras_original_model.hdf5\"\n",
    "network_definition_filename = \"keras_original_model.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print contents of COS buckets used in the previous training run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-data-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n",
      "  File: mnist.npz, 11221.13kB\n",
      "training-results-d7e9e034-b6ee-4031-a4f3-93521829c3ee\n",
      "  File: _wml_checkpoints/, 0.00kB\n",
      "  File: training-mSVi6tdiR/_submitted_code/model.zip, 2.73kB\n",
      "  File: training-mSVi6tdiR/keras_original_model.hdf5, 14092.55kB\n",
      "  File: training-mSVi6tdiR/keras_original_model.json, 2.75kB\n",
      "  File: training-mSVi6tdiR/learner-1/, 0.00kB\n",
      "  File: training-mSVi6tdiR/learner-1/.log-copy-complete, 0.00kB\n",
      "  File: training-mSVi6tdiR/learner-1/training-log.txt, 42.00kB\n"
     ]
    }
   ],
   "source": [
    "print_bucket_contents([training_data_bucket, training_result_bucket])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded keras_original_model.hdf5\n",
      "Downloaded keras_original_model.json\n"
     ]
    }
   ],
   "source": [
    "# download network definition and weights to current working directory\n",
    "\n",
    "weights_file_in_cos_bucket = os.path.join(model_id, weights_filename)\n",
    "network_definition_file_in_cos_bucket = os.path.join(model_id, network_definition_filename)\n",
    "\n",
    "bucket_obj = cos.Bucket(training_result_bucket)\n",
    "\n",
    "bucket_obj.download_file(weights_file_in_cos_bucket, weights_filename)\n",
    "print('Downloaded', weights_filename)\n",
    "\n",
    "bucket_obj.download_file(network_definition_file_in_cos_bucket, network_definition_filename)\n",
    "print('Downloaded', network_definition_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the original data set (mnist.npz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded mnist.npz\n"
     ]
    }
   ],
   "source": [
    "# download dataset\n",
    "bucket_obj = cos.Bucket(training_data_bucket)\n",
    "bucket_obj.download_file(dataset_filename, dataset_filename)\n",
    "print('Downloaded', dataset_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load & compile the model that we created using `convolutional_keras.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Definition: keras_original_model.json\n",
      "Weights:            keras_original_model.hdf5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from keras import backend as K\n",
    "from keras.models import model_from_json\n",
    "\n",
    "print('Network Definition:', network_definition_filename)\n",
    "print('Weights:           ', weights_filename)\n",
    "\n",
    "# load model\n",
    "json_file = open(network_definition_filename, 'r')\n",
    "model_json = json_file.read()\n",
    "json_file.close()\n",
    "\n",
    "model = model_from_json(model_json)\n",
    "model.load_weights(weights_filename)\n",
    "comp_params = {'loss': 'categorical_crossentropy',\n",
    "                       'optimizer': 'adam',\n",
    "                       'metrics': ['accuracy']}\n",
    "model.compile(**comp_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After loading & compiling the model, the next step is to create a KerasClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create ART classifier object\n",
    "from art.classifiers.keras import KerasClassifier\n",
    "\n",
    "classifier = KerasClassifier((0, 1), model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the test data and labels from `mnist.npz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "\n",
    "f = np.load(dataset_filename)\n",
    "x_original = f['x_test']\n",
    "y = f['y_test']\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the original (non-adversarial) sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADXZJREFUeJzt3X+IHPUZx/HPU5uAaFGT0uMwttGohSj+CKcUCaVFjVZiYkA0wT9SWnr9o0LF+ItUUChiKf1B/wpEDCba2jRcjFFL0zZUTSEJOSVGo1ETuWjCJdcQ0QSRmuTpHzvXXvXmu5uZ2Z29PO8XHLc7z+7Mw3Kfm5md3e/X3F0A4vlS3Q0AqAfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1Jc7uTEz4+OEQJu5u7XyuFJ7fjO70czeNrPdZvZAmXUB6Cwr+tl+MztN0juSrpe0T9I2SYvc/c3Ec9jzA23WiT3/1ZJ2u/t77v5vSX+UNL/E+gB0UJnwnyvpgzH392XL/o+Z9ZvZoJkNltgWgIq1/Q0/d18uabnEYT/QTcrs+fdLOm/M/WnZMgATQJnwb5N0kZmdb2aTJS2UtL6atgC0W+HDfnc/ZmZ3Stog6TRJK9x9Z2WdAWirwpf6Cm2Mc36g7TryIR8AExfhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBdXTobhRzzz33JOunn356bu2yyy5LPvfWW28t1NOoZcuWJeubN2/OrT355JOlto1y2PMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCM3tsFVq9enayXvRZfpz179uTWrrvuuuRz33///arbCYHRewEkEX4gKMIPBEX4gaAIPxAU4QeCIvxAUKW+z29mQ5KOSDou6Zi791XR1Kmmzuv4u3btStY3bNiQrF9wwQXJ+s0335ysz5gxI7d2xx13JJ/76KOPJusop4rBPL7r7ocqWA+ADuKwHwiqbPhd0l/N7BUz66+iIQCdUfawf7a77zezr0n6m5ntcveXxz4g+6fAPwagy5Ta87v7/uz3iKRnJF09zmOWu3sfbwYC3aVw+M3sDDP7yuhtSXMkvVFVYwDaq8xhf4+kZ8xsdD1/cPe/VNIVgLYrHH53f0/S5RX2MmH19aXPaBYsWFBq/Tt37kzW582bl1s7dCh9Ffbo0aPJ+uTJk5P1LVu2JOuXX57/JzJ16tTkc9FeXOoDgiL8QFCEHwiK8ANBEX4gKMIPBMUU3RXo7e1N1rPPQuRqdinvhhtuSNaHh4eT9TKWLFmSrM+cObPwul944YXCz0V57PmBoAg/EBThB4Ii/EBQhB8IivADQRF+ICiu81fgueeeS9YvvPDCZP3IkSPJ+uHDh0+6p6osXLgwWZ80aVKHOkHV2PMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFBc5++AvXv31t1CrnvvvTdZv/jii0utf+vWrYVqaD/2/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QlLl7+gFmKyTNlTTi7pdmy6ZIWi1puqQhSbe5+4dNN2aW3hgqN3fu3GR9zZo1yXqzKbpHRkaS9dR4AC+99FLyuSjG3dMTRWRa2fM/IenGzy17QNJGd79I0sbsPoAJpGn43f1lSZ8fSma+pJXZ7ZWSbqm4LwBtVvScv8fdR+eIOiCpp6J+AHRI6c/2u7unzuXNrF9Sf9ntAKhW0T3/QTPrlaTsd+67Pu6+3N373L2v4LYAtEHR8K+XtDi7vVjSs9W0A6BTmobfzJ6WtFnSN81sn5n9UNIvJF1vZu9Kui67D2ACaXrO7+6LckrXVtwL2qCvL3221ew6fjOrV69O1rmW3734hB8QFOEHgiL8QFCEHwiK8ANBEX4gKIbuPgWsW7cutzZnzpxS6161alWy/uCDD5ZaP+rDnh8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmo6dHelG2Po7kJ6e3uT9ddeey23NnXq1ORzDx06lKxfc801yfqePXuSdXRelUN3AzgFEX4gKMIPBEX4gaAIPxAU4QeCIvxAUHyffwIYGBhI1ptdy0956qmnknWu45+62PMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFBNr/Ob2QpJcyWNuPul2bKHJf1I0r+yhy119z+3q8lT3bx585L1WbNmFV73iy++mKw/9NBDhdeNia2VPf8Tkm4cZ/lv3f2K7IfgAxNM0/C7+8uSDnegFwAdVOac/04z22FmK8zsnMo6AtARRcO/TNIMSVdIGpb067wHmlm/mQ2a2WDBbQFog0Lhd/eD7n7c3U9IekzS1YnHLnf3PnfvK9okgOoVCr+ZjR1OdoGkN6ppB0CntHKp72lJ35H0VTPbJ+khSd8xsyskuaQhST9uY48A2qBp+N190TiLH29DL6esZt+3X7p0abI+adKkwtvevn17sn706NHC68bExif8gKAIPxAU4QeCIvxAUIQfCIrwA0ExdHcHLFmyJFm/6qqrSq1/3bp1uTW+sos87PmBoAg/EBThB4Ii/EBQhB8IivADQRF+IChz985tzKxzG+sin376abJe5iu7kjRt2rTc2vDwcKl1Y+Jxd2vlcez5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAovs9/CpgyZUpu7bPPPutgJ1/00Ucf5daa9dbs8w9nnXVWoZ4k6eyzz07W77777sLrbsXx48dza/fff3/yuZ988kklPbDnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgml7nN7PzJK2S1CPJJS1399+Z2RRJqyVNlzQk6TZ3/7B9rSLPjh076m4h15o1a3JrzcYa6OnpSdZvv/32Qj11uwMHDiTrjzzySCXbaWXPf0zSEnefKelbkn5iZjMlPSBpo7tfJGljdh/ABNE0/O4+7O6vZrePSHpL0rmS5ktamT1spaRb2tUkgOqd1Dm/mU2XdKWkrZJ63H30uO2AGqcFACaIlj/bb2ZnShqQdJe7f2z2v2HC3N3zxuczs35J/WUbBVCtlvb8ZjZJjeD/3t3XZosPmllvVu+VNDLec919ubv3uXtfFQ0DqEbT8FtjF/+4pLfc/TdjSuslLc5uL5b0bPXtAWiXpkN3m9lsSZskvS7pRLZ4qRrn/X+S9HVJe9W41He4ybpCDt29du3aZH3+/Pkd6iSWY8eO5dZOnDiRW2vF+vXrk/XBwcHC6960aVOyvmXLlmS91aG7m57zu/s/JeWt7NpWNgKg+/AJPyAowg8ERfiBoAg/EBThB4Ii/EBQTNHdBe67775kvewU3imXXHJJst7Or82uWLEiWR8aGiq1/oGBgdzarl27Sq27mzFFN4Akwg8ERfiBoAg/EBThB4Ii/EBQhB8Iiuv8wCmG6/wAkgg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqKbhN7PzzOwfZvamme00s59myx82s/1mtj37uan97QKoStPBPMysV1Kvu79qZl+R9IqkWyTdJumou/+q5Y0xmAfQdq0O5vHlFlY0LGk4u33EzN6SdG659gDU7aTO+c1suqQrJW3NFt1pZjvMbIWZnZPznH4zGzSzwVKdAqhUy2P4mdmZkl6S9Ii7rzWzHkmHJLmkn6txavCDJuvgsB9os1YP+1sKv5lNkvS8pA3u/ptx6tMlPe/ulzZZD+EH2qyyATzNzCQ9LumtscHP3ggctUDSGyfbJID6tPJu/2xJmyS9LulEtnippEWSrlDjsH9I0o+zNwdT62LPD7RZpYf9VSH8QPsxbj+AJMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQTQfwrNghSXvH3P9qtqwbdWtv3dqXRG9FVdnbN1p9YEe/z/+FjZsNuntfbQ0kdGtv3dqXRG9F1dUbh/1AUIQfCKru8C+vefsp3dpbt/Yl0VtRtfRW6zk/gPrUvecHUJNawm9mN5rZ22a228weqKOHPGY2ZGavZzMP1zrFWDYN2oiZvTFm2RQz+5uZvZv9HneatJp664qZmxMzS9f62nXbjNcdP+w3s9MkvSPpekn7JG2TtMjd3+xoIznMbEhSn7vXfk3YzL4t6aikVaOzIZnZLyUddvdfZP84z3H3+7ukt4d1kjM3t6m3vJmlv68aX7sqZ7yuQh17/qsl7Xb399z935L+KGl+DX10PXd/WdLhzy2eL2lldnulGn88HZfTW1dw92F3fzW7fUTS6MzStb52ib5qUUf4z5X0wZj7+9RdU367pL+a2Stm1l93M+PoGTMz0gFJPXU2M46mMzd30udmlu6a167IjNdV4w2/L5rt7rMkfU/ST7LD267kjXO2brpcs0zSDDWmcRuW9Os6m8lmlh6QdJe7fzy2VudrN05ftbxudYR/v6Tzxtyfli3rCu6+P/s9IukZNU5TusnB0UlSs98jNffzX+5+0N2Pu/sJSY+pxtcum1l6QNLv3X1ttrj21268vup63eoI/zZJF5nZ+WY2WdJCSetr6OMLzOyM7I0YmdkZkuao+2YfXi9pcXZ7saRna+zl/3TLzM15M0ur5teu62a8dveO/0i6SY13/PdI+lkdPeT0dYGk17KfnXX3JulpNQ4DP1PjvZEfSpoqaaOkdyX9XdKULurtSTVmc96hRtB6a+ptthqH9Dskbc9+bqr7tUv0Vcvrxif8gKB4ww8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFD/Abw9Wv8QfFP9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.imshow(x_original[1], cmap='gray')\n",
    "print(y[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardize the Numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocess\n",
    "x_original = np.expand_dims(x_original, axis=3)\n",
    "x_original = x_original.astype('float32') / 255\n",
    "y = np_utils.to_categorical(y, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the model and calculated test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model test loss:     6.0927865943312645\n",
      "model test accuracy: 98.21\n"
     ]
    }
   ],
   "source": [
    "# evaluate\n",
    "scores = model.evaluate(x_original, y, verbose=0)\n",
    "print('model test loss:    ', scores[0]*100)\n",
    "print('model test accuracy:', scores[1]*100)\n",
    "model_accuracy = scores[1]*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ART exposes many attacks like FGM, NewtonFool, DeepFool, Carlini etc. The code below shows how to use one of ART's attack methods (Fast Gradient Method or FGM) to craft adversarial samples based on x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of adversarial samples crafted: 10000\n",
      "adversarial samples saved to: etc/notebooks/art/adv_samples\n"
     ]
    }
   ],
   "source": [
    "from art.attacks.fast_gradient import FastGradientMethod\n",
    "\n",
    "# configuration\n",
    "epsilon = 0.2\n",
    "\n",
    "# create crafter object\n",
    "crafter = FastGradientMethod(classifier, eps=epsilon)\n",
    "\n",
    "# craft samples on x_test (stored in variable x_original)\n",
    "x_adv_samples = crafter.generate(x_original)\n",
    "\n",
    "outfile = os.path.join(os.getcwd(), 'adv_samples')\n",
    "np.savez(outfile, x_original=x_original, x_adversarial=x_adv_samples, y=y)\n",
    "\n",
    "print(\"Number of adversarial samples crafted:\", len(x_adv_samples))\n",
    "print(\"adversarial samples saved to:\", outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions can be used for gathering metrics like model robustness, confidence metric, perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy.linalg as la\n",
    "import json\n",
    "\n",
    "\n",
    "def get_metrics(model, x_original, x_adv_samples, y):\n",
    "    scores = model.evaluate(x_original, y, verbose=0)\n",
    "    model_accuracy_on_non_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    y_pred = model.predict(x_original, verbose=0)\n",
    "    y_pred_adv = model.predict(x_adv_samples, verbose=0)\n",
    "\n",
    "    scores = model.evaluate(x_adv_samples, y, verbose=0)\n",
    "    model_accuracy_on_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    pert_metric = get_perturbation_metric(x_original, x_adv_samples, y_pred, y_pred_adv, ord=2)\n",
    "    conf_metric = get_confidence_metric(y_pred, y_pred_adv)\n",
    "\n",
    "    data = {\n",
    "        \"model accuracy on test data:\": model_accuracy_on_non_adversarial_samples,\n",
    "        \"model accuracy on adversarial samples\": model_accuracy_on_adversarial_samples,\n",
    "        \"reduction in confidence\": conf_metric * 100,\n",
    "        \"average perturbation\": pert_metric * 100\n",
    "    }\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_perturbation_metric(x_original, x_adv, y_pred, y_pred_adv, ord=2):\n",
    "\n",
    "    idxs = (np.argmax(y_pred_adv, axis=1) != np.argmax(y_pred, axis=1))\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    perts_norm = la.norm((x_adv - x_original).reshape(x_original.shape[0], -1), ord, axis=1)\n",
    "    perts_norm = perts_norm[idxs]\n",
    "\n",
    "    return np.mean(perts_norm / la.norm(x_original[idxs].reshape(np.sum(idxs), -1), ord, axis=1))\n",
    "\n",
    "\n",
    "# This computes the change in confidence for all images in the test set\n",
    "def get_confidence_metric(y_pred, y_pred_adv):\n",
    "\n",
    "    y_classidx = np.argmax(y_pred, axis=1)\n",
    "    y_classconf = y_pred[np.arange(y_pred.shape[0]), y_classidx]\n",
    "\n",
    "    y_adv_classidx = np.argmax(y_pred_adv, axis=1)\n",
    "    y_adv_classconf = y_pred_adv[np.arange(y_pred_adv.shape[0]), y_adv_classidx]\n",
    "\n",
    "    idxs = (y_classidx == y_adv_classidx)\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    idxnonzero = y_classconf != 0\n",
    "    idxs = idxs & idxnonzero\n",
    "\n",
    "    return np.mean((y_classconf[idxs] - y_adv_classconf[idxs]) / y_classconf[idxs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below cell will display the following\n",
    "\n",
    "1. Model accuracy on test data\n",
    "2. Model robustness on adversarial samples\n",
    "3. Reduction in confidence\n",
    "4. Perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"model accuracy on test data:\": 98.21,\n",
      "    \"model accuracy on adversarial samples\": 55.800000000000004,\n",
      "    \"reduction in confidence\": 24.336346983909607,\n",
      "    \"average perturbation\": 46.15243971347809\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = get_metrics(model, x_original, x_adv_samples, y)\n",
    "\n",
    "print(json.dumps(result, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the numpy array, so that the adversarial images can be visualized\n",
    "\n",
    "x_adv = (x_adv_samples * 255).astype('int')\n",
    "x_adv = x_adv[:, :, :, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And visualize an adversarial sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAD3FJREFUeJzt3X+MVfWZx/HPsw5DZNoYkN0JCq61wQ3+pOtATNZsMNDGEnREjRSTlU2a4h+t2SaNLNGY9Z81ZrNt18QNCV1JwXShmIKC0VqXrHHRDWEQV23HXdFQfmQEDU0QMKmDz/4xZ5qpzv2eyz333HMuz/uVTObO/d5zzsOZ++HeO88552vuLgDx/EnVBQCoBuEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUT0c31tPjU6dO7eQmm3bmzJmWl502bVqhdectj/op+jst8nzL4+7WzOMKhd/MbpH0uKQLJP2buz+WevzUqVM1b968Ipsszb59+1peNu/flLfuuu4TNFb0d1rk+dYuLb/tN7MLJP2rpG9KukrSSjO7ql2FAShXkc/8CyUdcPf33f33krZIGmxPWQDKViT8l0o6POHnI9l9f8TMVpvZkJkNjY6OFtgcgHYq/a/97r7e3QfcfaCnp6N/XwSQUCT8RyXNmfDz7Ow+AF2gSPj3SpprZl8xs15J35K0oz1lASibFbmSj5ktlfQvGmv1bXD3f0w9vq+vz2lroVl57bAbbrihQ5V0VtE2YEf6/O7+vKTni6wDQDU4vBcIivADQRF+ICjCDwRF+IGgCD8QVKE+/zlvzKzQxs7Xvi46r5uPIcirvdk+P6/8QFCEHwiK8ANBEX4gKMIPBEX4gaC66tI6ZV7xtMrWTt6/a9GiRcnxCy+8sOHY9ddfn1z2rrvuSo7nWbduXXL82muvbTj21FNPJZetsh1X9LlW51bhOF75gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCorjqlt0xV9mUXL16cHC/ai6/Se++913Bs69atyWUPHTqUHK/DTLetKvJ845ReAIUQfiAowg8ERfiBoAg/EBThB4Ii/EBQRafoPijpY0lnJY26+0Dq8UWn6K7r+fx5da1ZsyY5XmYf/5133kmOv/jii8nxK664Ijl+6623nnNN4w4fPpwcf/TRR1ted55uPkYgT0em6M7c7O4ftWE9ADqIt/1AUEXD75J+ZWb7zGx1OwoC0BlF3/bf5O5HzezPJL1kZu+4+ysTH5D9p7Baknp7ewtuDkC7FHrld/ej2ffjkrZLWjjJY9a7+4C7D/T0dNX1QoHzWsvhN7M+M/vy+G1J35D0drsKA1CuIi/F/ZK2m9n4ev7d3X/ZlqoAlK7l8Lv7+5LSF4VvszJ78WX2fZcvX15o+QULFiTHU8cJfPRRugt76tSp5HjqfHxJuuyyy5LjqXkDZsyYkVy2THnPpfP5OIBxtPqAoAg/EBThB4Ii/EBQhB8IivADQXX0kLszZ84kWyjdMK3xZC655JLkeHYsREN5rby802ZHRkaS40U88MADyfE77rgjOb53796GY4ODg8llT548mRwv8/kSoRXIKz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBFWrKbqr7PPn9W2L1JZ32mveabUnTpxoedtFDQ0NFVo+1edfsmRJctm5c+cW2naVqjwOgCm6ASQRfiAowg8ERfiBoAg/EBThB4Ii/EBQHT2ff9q0aSoyRXeZyjzG4NChQ6WtWyrWU96yZUsbK/miPXv2NBzLO1+/TGUe19HM8nW4HgCv/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVG6f38w2SFom6bi7X5PdN0PSzyVdLumgpLvd/XfllRlbkZ70smXLksuuWLGipZrGPffcc8nxtWvXFlp/Waruw6e236ljAJp55f+ppFs+d99aSbvcfa6kXdnPALpIbvjd/RVJn7+UzKCkjdntjZJub3NdAErW6mf+fncfnyPqA0n9baoHQIcU/oOfj10EsOG1+cxstZkNmdnQ6Oho0c0BaJNWw3/MzGZJUvb9eKMHuvt6dx9w94Geno6eRwQgodXw75C0Kru9StKz7SkHQKfkht/MNkv6b0l/YWZHzOzbkh6T9HUze1fSkuxnAF0k9324u69sMLS4zbWUquzzt4sos687MDBQ2rolaevWrcnx06dPNxyr8z4v+zgAzucHUBnCDwRF+IGgCD8QFOEHgiL8QFAdnaK7r6/P63rp7m72zDPPNBybPXt2oXXff//9yfEnnnii5XVX2eorG1N0A6gtwg8ERfiBoAg/EBThB4Ii/EBQhB8Iqqv6/Kne6fncM541a1ZyfOfOnS2v+4UXXkiOL126tOV1l62bf+dlHgdAnx9AEuEHgiL8QFCEHwiK8ANBEX4gKMIPBNXRPr+ZFdpYXfu6ZV8W/LXXXkuO9/b2trzue+65Jzm+efPmltctFZuKuq6/76rl7Tf6/ACSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqNw+v5ltkLRM0nF3vya77xFJ35H0YfawB939+dyNFezzp3RzT/i2225Lji9ZsiQ5PmXKlIZjL7/8cnLZNWvWJMfzlDmVdTf/TqsyPDys06dPt63P/1NJt0xy/4/dfX72lRt8APWSG353f0XSiQ7UAqCDinzm/56ZvWlmG8xsetsqAtARrYZ/naSvSpovaUTSDxs90MxWm9mQmQ21uC0AJWgp/O5+zN3Puvtnkn4iaWHisevdfcDdB1otEkD7tRR+M5t4Odnlkt5uTzkAOqUn7wFmtlnSIkkzzeyIpH+QtMjM5ktySQcl3VdijQBKkBt+d185yd1PllDLeeviiy9Ojs+cOTM5nurj59m/f3/LyzaDPn734gg/ICjCDwRF+IGgCD8QFOEHgiL8QFC5rb52mjZtmlJTdBdpGxWd8rhI2ylv23feeWdyfMGCBS1vW5K2b9/ecCzv0tu027pPu6b35pUfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Lq6BTdfX19nurzl6nM6aDz1v3qq68mx4ucsitJCxc2vJBSYRwHUD9M0Q2gEMIPBEX4gaAIPxAU4QeCIvxAUIQfCKqj5/OXqcw+ft3de++9Dcc2bdpUaN1Fzx2/8cYbG459+umnyWXzjn+46KKLWqpJkqZPT08v2dfX1/K6m3H27NmGYwcOHEgum3ouDw8PN10Dr/xAUIQfCIrwA0ERfiAowg8ERfiBoAg/EFRun9/M5kjaJKlfkkta7+6Pm9kMST+XdLmkg5LudvfflVdqMd18HEDedf03btzY0lgz9u7dW2j5p59+uuHYyMhIctn+/v7k+IoVK1qqqe4efvjh5Pgnn3zSlu0088o/KukH7n6VpBslfdfMrpK0VtIud58raVf2M4AukRt+dx9x99ez2x9LGpZ0qaRBSeMvKxsl3V5WkQDa75w+85vZ5ZK+JmmPpH53H3/f9oHGPhYA6BJNh9/MviTpF5K+7+4nJ4752IUAJ70YoJmtNrMhMxsaHR0tVCyA9mkq/GY2RWPB/5m7b8vuPmZms7LxWZKOT7asu6939wF3H+jpOW/OIwK6Xm74zcwkPSlp2N1/NGFoh6RV2e1Vkp5tf3kAypJ76W4zu0nSf0l6S9Jn2d0Pauxz/1ZJl0n6rcZafSdy1tW564Sfo7xWX5FTWx966KHk+ODgYHK86BTeZSraCixT6mNm6pTaZuzcuTM5XmS/7N69Ozl+3XXXNRwbHh7W6dOnm7p0d+77cHffLanRyhY3sxEA9cMRfkBQhB8IivADQRF+ICjCDwRF+IGgOjpFd9Q+f56bb745Od7b29vyuu+7777k+NVXX50cv/LKK1vedp79+/cnxw8ePFho/du2bWs4di6XuK4bpugGUAjhB4Ii/EBQhB8IivADQRF+ICjCDwTV0T5/X1+fz5s3r+Xly+y11/nS3XmK7Jdu/ndHRZ8fQCGEHwiK8ANBEX4gKMIPBEX4gaAIPxAUU+hkunkK7yLKPHZCOn/3W5VS+/RcrlPAKz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBJV7Pr+ZzZG0SVK/JJe03t0fN7NHJH1H0ofZQx909+dT6yp6Pn+Zyu5349zV+RiBos+XIv+2dp3P38xBPqOSfuDur5vZlyXtM7OXsrEfu/s/N7MhAPWSG353H5E0kt3+2MyGJV1admEAynVOn/nN7HJJX5O0J7vre2b2ppltMLPpDZZZbWZDZjY0OjpaqFgA7dN0+M3sS5J+Ien77n5S0jpJX5U0X2PvDH442XLuvt7dB9x9oKeHUwmAumgq/GY2RWPB/5m7b5Mkdz/m7mfd/TNJP5G0sLwyAbRbbvjNzCQ9KWnY3X804f5ZEx62XNLb7S8PQFmaeR/+V5L+RtJbZvZGdt+Dklaa2XyNtf8OSkrPBV1zVU7RjXjq8Hxq5q/9uyVN1jdM9vQB1BtH+AFBEX4gKMIPBEX4gaAIPxAU4QeC6ugU3WbWuY2hI+p82m0RdejDt4opugEkEX4gKMIPBEX4gaAIPxAU4QeCIvxAUJ3u838o6bcT7pop6aOOFXBu6lpbXeuSqK1V7aztz939T5t5YEfD/4WNmw25+0BlBSTUtba61iVRW6uqqo23/UBQhB8Iqurwr694+yl1ra2udUnU1qpKaqv0Mz+A6lT9yg+gIpWE38xuMbP/NbMDZra2ihoaMbODZvaWmb1hZkMV17LBzI6b2dsT7pthZi+Z2bvZ90mnSauotkfM7Gi2794ws6UV1TbHzP7TzH5jZr82s7/L7q903yXqqmS/dfxtv5ldIOn/JH1d0hFJeyWtdPffdLSQBszsoKQBd6+8J2xmfy3plKRN7n5Ndt8/STrh7o9l/3FOd/e/r0ltj0g6VfXMzdmEMrMmziwt6XZJf6sK912irrtVwX6r4pV/oaQD7v6+u/9e0hZJgxXUUXvu/oqkE5+7e1DSxuz2Ro09eTquQW214O4j7v56dvtjSeMzS1e67xJ1VaKK8F8q6fCEn4+oXlN+u6Rfmdk+M1tddTGT6M+mTZekDyT1V1nMJHJnbu6kz80sXZt918qM1+3GH/y+6CZ3/0tJ35T03eztbS352Ge2OrVrmpq5uVMmmVn6D6rcd63OeN1uVYT/qKQ5E36end1XC+5+NPt+XNJ21W/24WPjk6Rm349XXM8f1Gnm5slmllYN9l2dZryuIvx7Jc01s6+YWa+kb0naUUEdX2BmfdkfYmRmfZK+ofrNPrxD0qrs9ipJz1ZYyx+py8zNjWaWVsX7rnYzXrt7x78kLdXYX/zfk/RQFTU0qOsKSf+Tff266tokbdbY28BPNfa3kW9LuljSLknvSvoPSTNqVNtTkt6S9KbGgjarotpu0thb+jclvZF9La163yXqqmS/cYQfEBR/8AOCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/ENT/Ax/ZYm8/nrahAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_adv[1], cmap='gray');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Generate Adversarial Samples Asynchronously using FfDL <a id=\"artWithFfDL\"></a>\n",
    "\n",
    "As its name suggests, FGM is a relatively short running method to generate adversarial samples. Other attack techniques can take much longer and therefore you may want to use additional computing resources and generate adversarial samples through an asynchronous Watsom ML training request. This section shows how to gather the relevant information to execute FGM as an asynchronous FfDL training job. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets first create two COS buckets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating bucket \"robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7\" ...\n",
      "Creating bucket \"robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7\" ...\n"
     ]
    }
   ],
   "source": [
    "from uuid import uuid4\n",
    "\n",
    "bucket_uid = str(uuid4())\n",
    "\n",
    "robustnesscheck_data_bucket   = 'robustnesscheck-data-'    + bucket_uid\n",
    "robustnesscheck_result_bucket = 'robustnesscheck-results-' + bucket_uid\n",
    "\n",
    "buckets = [robustnesscheck_data_bucket,\n",
    "           robustnesscheck_result_bucket]\n",
    "\n",
    "create_buckets(buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload all the artifacts (`mnist.npz`, `keras_original_model.json`, `keras_original_model.hdf5`) to the `robustnesscheck_data_bucket`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading files to robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7:\n",
      "- mnist.npz was uploaded\n",
      "- keras_original_model.hdf5 was uploaded\n",
      "- keras_original_model.json was uploaded\n"
     ]
    }
   ],
   "source": [
    "# upload\n",
    "\n",
    "bucket_obj = cos.Bucket(robustnesscheck_data_bucket)\n",
    "print(\"Uploading files to {}:\".format(robustnesscheck_data_bucket))\n",
    "\n",
    "bucket_obj.upload_file(dataset_filename, dataset_filename)\n",
    "print('- {} was uploaded'.format(dataset_filename)) \n",
    "\n",
    "bucket_obj.upload_file(weights_filename, weights_filename)\n",
    "print('- {} was uploaded'.format(weights_filename))\n",
    "\n",
    "bucket_obj.upload_file(network_definition_filename, network_definition_filename)\n",
    "print('- {} was uploaded'.format(network_definition_filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Create a Python script that generates adversarial samples to check robustness using FGM (Fast Gradient Method) from the ART library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_filename  = \"robustness_check.py\"\n",
    "archive_filename = 'model.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing robustness_check.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $script_filename\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import numpy.linalg as la\n",
    "import json\n",
    "\n",
    "from keras.models import model_from_json\n",
    "from art.classifiers.keras import KerasClassifier\n",
    "from keras.utils import np_utils\n",
    "from art.attacks.fast_gradient import FastGradientMethod\n",
    "\n",
    "\n",
    "def get_metrics(model, x_original, x_adv_samples, y):\n",
    "    scores = model.evaluate(x_original, y, verbose=0)\n",
    "    model_accuracy_on_non_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    y_pred = model.predict(x_original, verbose=0)\n",
    "    y_pred_adv = model.predict(x_adv_samples, verbose=0)\n",
    "\n",
    "    scores = model.evaluate(x_adv_samples, y, verbose=0)\n",
    "    model_accuracy_on_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    pert_metric = get_perturbation_metric(x_original, x_adv_samples, y_pred, y_pred_adv, ord=2)\n",
    "    conf_metric = get_confidence_metric(y_pred, y_pred_adv)\n",
    "\n",
    "    data = {\n",
    "        \"model accuracy on test data:\": model_accuracy_on_non_adversarial_samples,\n",
    "        \"model accuracy on adversarial samples\": model_accuracy_on_adversarial_samples,\n",
    "        \"reduction in confidence\": conf_metric * 100,\n",
    "        \"average perturbation\": pert_metric * 100\n",
    "    }\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_perturbation_metric(x_original, x_adv, y_pred, y_pred_adv, ord=2):\n",
    "\n",
    "    idxs = (np.argmax(y_pred_adv, axis=1) != np.argmax(y_pred, axis=1))\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    perts_norm = la.norm((x_adv - x_original).reshape(x_original.shape[0], -1), ord, axis=1)\n",
    "    perts_norm = perts_norm[idxs]\n",
    "\n",
    "    return np.mean(perts_norm / la.norm(x_original[idxs].reshape(np.sum(idxs), -1), ord, axis=1))\n",
    "\n",
    "\n",
    "# This computes the change in confidence for all images in the test set\n",
    "def get_confidence_metric(y_pred, y_pred_adv):\n",
    "\n",
    "    y_classidx = np.argmax(y_pred, axis=1)\n",
    "    y_classconf = y_pred[np.arange(y_pred.shape[0]), y_classidx]\n",
    "\n",
    "    y_adv_classidx = np.argmax(y_pred_adv, axis=1)\n",
    "    y_adv_classconf = y_pred_adv[np.arange(y_pred_adv.shape[0]), y_adv_classidx]\n",
    "\n",
    "    idxs = (y_classidx == y_adv_classidx)\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    idxnonzero = y_classconf != 0\n",
    "    idxs = idxs & idxnonzero\n",
    "\n",
    "    return np.mean((y_classconf[idxs] - y_adv_classconf[idxs]) / y_classconf[idxs])\n",
    "\n",
    "\n",
    "def main(argv):\n",
    "    if len(argv) < 2:\n",
    "        sys.exit(\"Not enough arguments provided.\")\n",
    "\n",
    "    global network_definition_filename, weights_filename, dataset_filename\n",
    "\n",
    "    i = 1\n",
    "    while i <= 8:\n",
    "        arg = str(argv[i])\n",
    "        print(arg)\n",
    "        if arg == \"--data\":\n",
    "            dataset_filename = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        if arg == \"--networkdefinition\":\n",
    "            network_definition_filename = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        if arg == \"--weights\":\n",
    "            weights_filename = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        if arg == \"--epsilon\":\n",
    "            epsilon = float(argv[i+1])\n",
    "\n",
    "        i += 2\n",
    "\n",
    "    print(\"dataset: \", dataset_filename)\n",
    "    print(\"network definition: \", network_definition_filename)\n",
    "    print(\"weights: \", weights_filename)\n",
    "\n",
    "    # load & compile model\n",
    "    json_file = open(network_definition_filename, 'r')\n",
    "    model_json = json_file.read()\n",
    "    json_file.close()\n",
    "    model = model_from_json(model_json)\n",
    "    model.load_weights(weights_filename)\n",
    "    comp_params = {'loss': 'categorical_crossentropy',\n",
    "                   'optimizer': 'adam',\n",
    "                   'metrics': ['accuracy']}\n",
    "    model.compile(**comp_params)\n",
    "\n",
    "    # create keras classifier\n",
    "    classifier = KerasClassifier((0, 1), model)\n",
    "\n",
    "    # load data set\n",
    "    pf = np.load(dataset_filename)\n",
    "\n",
    "    x = pf['x_test']\n",
    "    y = pf['y_test']\n",
    "\n",
    "    # pre-process numpy array\n",
    "\n",
    "    x = np.expand_dims(x, axis=3)\n",
    "    x = x.astype('float32') / 255\n",
    "\n",
    "    y = np_utils.to_categorical(y, 10)\n",
    "\n",
    "    # craft adversarial samples using FGSM\n",
    "    crafter = FastGradientMethod(classifier, eps=epsilon)\n",
    "    x_samples = crafter.generate(x)\n",
    "\n",
    "    # obtain all metrics (robustness score, perturbation metric, reduction in confidence)\n",
    "    metrics = get_metrics(model, x, x_samples, y)\n",
    "\n",
    "    print(\"metrics : \", metrics)\n",
    "    \n",
    "    report_file = os.path.join(os.environ[\"RESULT_DIR\"], \"report.txt\")\n",
    "\n",
    "    with open(report_file, \"w\") as report:\n",
    "        report.write(json.dumps(metrics))\n",
    "    \n",
    "    adv_samples_file = os.path.join(os.environ[\"RESULT_DIR\"], 'adv_samples')\n",
    "    print(\"adversarial samples saved to : \", adv_samples_file)\n",
    "    np.savez(adv_samples_file, x_original=x, x_adversarial=x_samples, y=y)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main(sys.argv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a zip archive to package the training script `robustness_check.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  adding: robustness_check.py (deflated 67%)\r\n"
     ]
    }
   ],
   "source": [
    "zipfile.ZipFile(archive_filename, mode='w').write(script_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Model Definition Metadata. \n",
    "\n",
    "The training command now points to `robustness_check.py` and four arguments are passed to the script:\n",
    "\n",
    "1. data (`mnist.npz`)\n",
    "2. networkdefinition (`keras_original_model.json`)\n",
    "3. weights (`keras_original_model.hdf5`)\n",
    "4. epsilon (`0.2` or `0.1`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_command = \"\\\n",
    "pip3 install keras; \\\n",
    "pip3 install https://github.com/IBM/adversarial-robustness-toolbox/zipball/master; \\\n",
    "python3 robustness_check.py \\\n",
    "  --epsilon 0.2 \\\n",
    "  --data ${DATA_DIR}/mnist.npz \\\n",
    "  --networkdefinition ${DATA_DIR}/keras_original_model.json \\\n",
    "  --weights ${DATA_DIR}/keras_original_model.hdf5\"\n",
    "\n",
    "manifest = {\n",
    "  \"name\":        \"art_robustness_check\",\n",
    "  \"description\": \"Generates adversarial samples to check robustness using FGM\",\n",
    "  \"version\":     \"1.0\",\n",
    "  \"memory\": \"2Gb\",\n",
    "  \"gpus\":    0,\n",
    "  \"cpus\":    2,\n",
    "  \"data_stores\": [\n",
    "    {\n",
    "      \"id\":   \"s3-art-robustness-check\",\n",
    "      \"type\": \"mount_cos\",\n",
    "      \"training_data\": {\n",
    "        \"container\": robustnesscheck_data_bucket\n",
    "      },\n",
    "      \"training_results\": {\n",
    "        \"container\": robustnesscheck_result_bucket\n",
    "      },\n",
    "      \"connection\": {\n",
    "        \"auth_url\":  user_data[\"cos_service_endpoint\"],\n",
    "        \"user_name\": user_data[\"cos_hmac_access_key_id\"],\n",
    "        \"password\":  user_data[\"cos_hmac_secret_access_key\"]\n",
    "      }\n",
    "    }\n",
    "  ],\n",
    "  \"framework\": {\n",
    "    \"name\":    \"tensorflow\",\n",
    "    \"version\": \"1.5.0-py3\",\n",
    "    \"command\": training_command\n",
    "  },\n",
    "  \"evaluation_metrics\": {\n",
    "    \"type\": \"tensorboard\",\n",
    "    \"in\":   \"$JOB_STATE_DIR/logs/tb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "yaml.dump(manifest, open(\"manifest.yml\", \"w\"), default_flow_style=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now start the training job with FfDL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Deploying model with manifest 'manifest.yml' and model file 'model.zip'...\",\n",
       " 'Model ID: training-Vqpx6tOiR',\n",
       " 'OK']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = !{ffdl} train \"manifest.yml\" \"model.zip\"\n",
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting model training logs for '\u001b[1;36mtraining-Vqpx6tOiR\u001b[0m'...\n",
      "Training with training/test data at:\n",
      "  DATA_DIR: /mnt/data/robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7\n",
      "  MODEL_DIR: /job/model-code\n",
      "  TRAINING_JOB: \n",
      "  TRAINING_COMMAND: pip3 install keras; pip3 install https://github.com/IBM/adversarial-robustness-toolbox/zipball/master; python3 robustness_check.py   --epsilon 0.2   --data ${DATA_DIR}/mnist.npz   --networkdefinition ${DATA_DIR}/keras_original_model.json   --weights ${DATA_DIR}/keras_original_model.hdf5\n",
      "Storing trained model at:\n",
      "  RESULT_DIR: /mnt/results/robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7/training-Vqpx6tOiR\n",
      "Contents of $MODEL_DIR\n",
      "total 16\n",
      "drwxrwxrwx 2 6342627 4294967294 4096 Jul 14 01:44 .\n",
      "drwxrwxrwx 4 nobody  4294967294 4096 Jul 14 01:44 ..\n",
      "-rwxrwxrwx 1 6342627 4294967294 4442 Jul 13 18:43 robustness_check.py\n",
      "Contents of $DATA_DIR\n",
      "total 25322\n",
      "drwxrwxr-x 1 root root        0 Jan  1  1970 .\n",
      "drwxr-xr-x 3 root root     4096 Jul 14 01:43 ..\n",
      "---------- 1 root root 14430768 Jul 14 01:43 keras_original_model.hdf5\n",
      "---------- 1 root root     2813 Jul 14 01:43 keras_original_model.json\n",
      "---------- 1 root root 11490434 Jul 14 01:43 mnist.npz\n",
      "CHECKPOINT_DIR=/mnt/results/robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7/_wml_checkpoints\n",
      "DATA_DIR=/mnt/data/robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7\n",
      "DOWNWARD_API_POD_NAME=learner-a4e68235-4d8b-4d3d-5b1e-59030e896647-0\n",
      "DOWNWARD_API_POD_NAMESPACE=default\n",
      "ELASTICSEARCH_PORT=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_ADDR=172.21.204.56\n",
      "ELASTICSEARCH_PORT_9200_TCP_PORT=9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp\n",
      "ELASTICSEARCH_SERVICE_HOST=172.21.204.56\n",
      "ELASTICSEARCH_SERVICE_PORT=9200\n",
      "ELASTICSEARCH_SERVICE_PORT_HTTP=9200\n",
      "FFDL_LCM_PORT=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP_ADDR=172.21.37.214\n",
      "FFDL_LCM_PORT_80_TCP_PORT=80\n",
      "FFDL_LCM_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_LCM_SERVICE_HOST=172.21.37.214\n",
      "FFDL_LCM_SERVICE_PORT=80\n",
      "FFDL_LCM_SERVICE_PORT_GRPC=80\n",
      "FFDL_RESTAPI_PORT=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP_ADDR=172.21.137.113\n",
      "FFDL_RESTAPI_PORT_80_TCP_PORT=80\n",
      "FFDL_RESTAPI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_RESTAPI_SERVICE_HOST=172.21.137.113\n",
      "FFDL_RESTAPI_SERVICE_PORT=80\n",
      "FFDL_RESTAPI_SERVICE_PORT_FFDL=80\n",
      "FFDL_TRAINER_PORT=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP_ADDR=172.21.146.32\n",
      "FFDL_TRAINER_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAINER_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAINER_SERVICE_HOST=172.21.146.32\n",
      "FFDL_TRAINER_SERVICE_PORT=80\n",
      "FFDL_TRAINER_SERVICE_PORT_GRPC=80\n",
      "FFDL_TRAININGDATA_PORT=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_ADDR=172.21.180.242\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAININGDATA_SERVICE_HOST=172.21.180.242\n",
      "FFDL_TRAININGDATA_SERVICE_PORT=80\n",
      "FFDL_TRAININGDATA_SERVICE_PORT_GRPC=80\n",
      "FFDL_UI_PORT=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP_ADDR=172.21.53.228\n",
      "FFDL_UI_PORT_80_TCP_PORT=80\n",
      "FFDL_UI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_UI_SERVICE_HOST=172.21.53.228\n",
      "FFDL_UI_SERVICE_PORT=80\n",
      "FFDL_UI_SERVICE_PORT_HTTP=80\n",
      "GPU_COUNT=0.000000\n",
      "HOME=/root\n",
      "JOB_STATE_DIR=/job\n",
      "LEARNER_ID=1\n",
      "LEARNER_NAME_PREFIX=learner-a4e68235-4d8b-4d3d-5b1e-59030e896647\n",
      "LOG_DIR=/job/logs\n",
      "MODEL_DIR=/job/model-code\n",
      "NUM_LEARNERS=1\n",
      "OLDPWD=/notebooks\n",
      "PATH=/usr/local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n",
      "PROMETHEUS_PORT=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP_ADDR=172.21.74.212\n",
      "PROMETHEUS_PORT_9090_TCP_PORT=9090\n",
      "PROMETHEUS_PORT_9090_TCP_PROTO=tcp\n",
      "PROMETHEUS_SERVICE_HOST=172.21.74.212\n",
      "PROMETHEUS_SERVICE_PORT=9090\n",
      "PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090\n",
      "PWD=/job/model-code\n",
      "PYTHONPATH=:/job/model-code\n",
      "RESULT_DIR=/mnt/results/robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7/training-Vqpx6tOiR\n",
      "S3_PORT=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP_ADDR=172.21.46.129\n",
      "S3_PORT_80_TCP_PORT=80\n",
      "S3_PORT_80_TCP_PROTO=tcp\n",
      "S3_SERVICE_HOST=172.21.46.129\n",
      "S3_SERVICE_PORT=80\n",
      "SHLVL=3\n",
      "TRAINING_COMMAND=pip3 install keras; pip3 install https://github.com/IBM/adversarial-robustness-toolbox/zipball/master; python3 robustness_check.py   --epsilon 0.2   --data ${DATA_DIR}/mnist.npz   --networkdefinition ${DATA_DIR}/keras_original_model.json   --weights ${DATA_DIR}/keras_original_model.hdf5\n",
      "TRAINING_ID=training-Vqpx6tOiR\n",
      "_=/usr/bin/env\n",
      "Sat Jul 14 01:44:18 UTC 2018: Running training job\n",
      "Collecting keras\n",
      "  Downloading https://files.pythonhosted.org/packages/68/12/4cabc5c01451eb3b413d19ea151f36e33026fc0efb932bf51bcaf54acbf5/Keras-2.2.0-py2.py3-none-any.whl (300kB)\n",
      "Collecting keras-preprocessing==1.0.1 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/f8/33/275506afe1d96b221f66f95adba94d1b73f6b6087cfb6132a5655b6fe338/Keras_Preprocessing-1.0.1-py2.py3-none-any.whl\n",
      "Collecting keras-applications==1.0.2 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/e2/60/c557075e586e968d7a9c314aa38c236b37cb3ee6b37e8d57152b1a5e0b47/Keras_Applications-1.0.2-py2.py3-none-any.whl (43kB)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting pyyaml (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Building wheels for collected packages: pyyaml\n",
      "  Running setup.py bdist_wheel for pyyaml: started\n",
      "  Running setup.py bdist_wheel for pyyaml: finished with status 'done'\n",
      "  Stored in directory: /root/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f\n",
      "Successfully built pyyaml\n",
      "Installing collected packages: keras-preprocessing, keras-applications, pyyaml, keras\n",
      "Successfully installed keras-2.2.0 keras-applications-1.0.2 keras-preprocessing-1.0.1 pyyaml-3.13\n",
      "You are using pip version 9.0.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "Collecting https://github.com/IBM/adversarial-robustness-toolbox/zipball/master\n",
      "  Downloading https://github.com/IBM/adversarial-robustness-toolbox/zipball/master\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.5/dist-packages (from adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: Keras in /usr/local/lib/python3.5/dist-packages (from adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: scipy in /usr/local/lib/python3.5/dist-packages (from adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: matplotlib in /usr/local/lib/python3.5/dist-packages (from adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.5/dist-packages (from adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.5/dist-packages (from h5py->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python3.5/dist-packages (from h5py->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: keras-applications==1.0.2 in /usr/local/lib/python3.5/dist-packages (from Keras->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.5/dist-packages (from Keras->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: keras-preprocessing==1.0.1 in /usr/local/lib/python3.5/dist-packages (from Keras->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.5/dist-packages (from matplotlib->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: pytz in /usr/local/lib/python3.5/dist-packages (from matplotlib->adversarial-robustness-toolbox==0.2.2)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.5/dist-packages (from matplotlib->adversarial-robustness-toolbox==0.2.2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.5/dist-packages (from matplotlib->adversarial-robustness-toolbox==0.2.2)\n",
      "Installing collected packages: adversarial-robustness-toolbox\n",
      "  Running setup.py install for adversarial-robustness-toolbox: started\n",
      "    Running setup.py install for adversarial-robustness-toolbox: finished with status 'done'\n",
      "Successfully installed adversarial-robustness-toolbox-0.2.2\n",
      "You are using pip version 9.0.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "2018-07-14 01:44:26.863805: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "--epsilon\n",
      "--data\n",
      "--networkdefinition\n",
      "--weights\n",
      "dataset:  /mnt/data/robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7/mnist.npz\n",
      "network definition:  /mnt/data/robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7/keras_original_model.json\n",
      "weights:  /mnt/data/robustnesscheck-data-3e656301-8ea7-4065-8a6c-9e4601e27dd7/keras_original_model.hdf5\n",
      "metrics :  {'model accuracy on adversarial samples': 55.800000000000004, 'average perturbation': 46.15243971347809, 'model accuracy on test data:': 98.21, 'reduction in confidence': 24.336346983909607}\n",
      "adversarial samples saved to :  /mnt/results/robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7/training-Vqpx6tOiR/adv_samples\n",
      "Training process finished. Exit code: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Monitor the training logs\n",
    "\n",
    "if \"Model ID\" in out[1]:\n",
    "    model_id = out.fields()[1][-1]\n",
    "    !{ffdl} logs --follow {model_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above training job will create a report (`report.txt`). The file is available in COS (bucket = `robustnesscheck_result_bucket`) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"model accuracy on adversarial samples\": 55.800000000000004,\n",
      "    \"average perturbation\": 46.15243971347809,\n",
      "    \"model accuracy on test data:\": 98.21,\n",
      "    \"reduction in confidence\": 24.336346983909607\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "report_file = os.path.join(model_id, \"report.txt\")\n",
    "bucket_obj = cos.Bucket(robustnesscheck_result_bucket)\n",
    "bucket_obj.download_file(report_file, \"report.txt\")\n",
    "\n",
    "with open('report.txt', \"r\") as report:\n",
    "    result = json.load(report)\n",
    "    \n",
    "print(json.dumps(result, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the adversarial samples that were stored to COS using the below command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading robustnesscheck-results-3e656301-8ea7-4065-8a6c-9e4601e27dd7/training-Vqpx6tOiR/adv_samples.npz ...\n",
      "Downloaded: adv_samples_from_cos.npz\n"
     ]
    }
   ],
   "source": [
    "samples_file = os.path.join(model_id, \"adv_samples.npz\")\n",
    "print('Downloading {}/{} ...'.format(robustnesscheck_result_bucket, samples_file))\n",
    "bucket_obj = cos.Bucket(robustnesscheck_result_bucket)\n",
    "bucket_obj.download_file(samples_file, \"adv_samples_from_cos.npz\")\n",
    "print('Downloaded:', \"adv_samples_from_cos.npz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare original with adversarial images and test model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1oAAAC/CAYAAADnw60yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XfAFMX9x/H3/kBQsCuKFRRUbNiIGiOxYVdExYiisddgF0tMfDyjMVGjRqMQTWyY2DX2ir1HLIiCBmMHC/besr8/npu5OW6fu73dub295/m8/mGYu9udm2fL7c53vxOEYYiIiIiIiIj483/NboCIiIiIiEhnowstERERERERz3ShJSIiIiIi4pkutERERERERDzThZaIiIiIiIhnutASERERERHxTBdaIiIiIiIinulCS0RERERExDNdaImIiIiIiHjWvZ43B0EQNqohncCsMAz71Psh9WlV6lP/1Kf+qU/9U5/6pz71L1Gfgvq1mjAMgySfU59Wpf3fv1h9qhEtf95odgM6IfWpf+pT/9Sn/qlP/VOf+qc+lVahbdW/WH2qCy0RERERERHPdKElIiIiIiLimS60REREREREPNOFloiIiIiIiGe60BIREREREfFMF1oiIiIiIiKe1TWPlrSmo48+2pbnmmsuAAYPHmzrRo4cWfGZcePG2fLjjz8OwIQJExrVRBERERGRTkUjWiIiIiIiIp5pRKsTu/rqq4HoESvX//73v4q6Aw44wJaHDRsGwIMPPmjr3nzzTR9N7LKWX355AKZNm2brDjvsMADOO++8prQpj3r37m3LZ5xxBlC+bU6aNMmWd9ppJwDeeEPzMoqISH4tsMACtrz00kt3+D73fHbEEUcAMGXKFFv3yiuvAPD888/7bqJ4ohEtERERERERz3ShJSIiIiIi4plCBzsZEy4I1UMG3ZC1u+66C4Bll13W1m277ba2PGDAAABGjx5t60477bT0je3C1lhjDaA8bPPtt99uVnNya7HFFrPl/fbbDyjvs7XWWsuWt9lmGwDOP//8jFqXf2uuuaYt33DDDQD079/fy7I322wzW546dSoAb731lpdld2bm2HrzzTfbujFjxgAwfvx4W/fjjz9m27AmWGSRRQC45pprbN1jjz0GwIUXXmjrXn/9da/rnW+++Wz55z//OQB33nmnrfv++++9rk+6tq233tqWhw8fDsCGG25o6wYOHNjhZ01oIEC/fv0A6NmzZ8X7unXrlraZ0iAa0RIREREREfFMI1qdxJAhQwDYfvvtK1578cUXbdncTZk1a5at++KLLwDo0aOHrXviiSdsebXVVgNgoYUW8tjirm311VcH4Msvv7R1N954Y7Oakzt9+vQB4LLLLmtyS1rb5ptvbstRd0HTcEe99957bwBGjRrldR2dhXvsvOCCCype/8tf/gLAxRdfbOu+/vrrxjesCdwkAObc5I4wvffee4D/USx3PW4SHXOscUfHp0+f7n3dzTbvvPMC5dEoq6yyClBKeAUazUvCRP0A/OpXvwJKERhQmlYHIAiCupZtEmdJ69KIloiIiIiIiGe60BIREREREfEss9BBNzGDGVKdMWOGrfvmm28A+Mc//mHr3n33XaBzDuP7ZpIGuMPSJizDDR+aOXNmh8s46qijbHmllVaqeP22225L3c6uzIRpQOnh9wkTJjSrOblz6KGH2vKIESMAWHvttWN/3jzU/n//V7p/ZOYWeeihh3w0sWV0795+aN9qq60atg43/OrII48Eyuc9c8NiuzqzbQIsueSSFa9feeWVQOk82BktvPDCQHnCpgUXXBAoD6c85JBDGtaG3/zmNwAss8wyts7My9cZf2e4CaxOPfVUAJZaaqmK95mwQoAPP/yw8Q3rZNx92syHmZZJWOY++tFVmWQh5hgC5Y/JmMQibqIsk1jo0UcftXXN2sc1oiUiIiIiIuJZZiNap59+ui1XSy9s7i4BfP7550BjruhNKm23XU8//bT39WTllltuAcrThJr+++ijj2Itw32QfY455vDYOgEYNGiQLZs7/+7d3a7u7LPPtmX3zlRcO+ywQ9m/AG+88QYAO++8s61zR2I6q4022giAn/70p7bOPdb54CY1MCPgvXr1snVdfUTLTT5ywgknVH2vGdkOw7ChbWomM9WAm9baOPnkkxu23pVXXtmWTdSGm3ioMx6DzQjLOeecY+tMQpaobey8886zZRNtAfF/O3RW7giKGalyR0jMlADffvutrfv000+B8uOfO9J/9913AzBlyhRb9+STTwLw7LPP2jqTDKerHUdN5I+7HZpzuvv3qGWdddYB4IcffrB1L7/8MgCPPPKIrTN/1++++y5hi2vTiJaIiIiIiIhnutASERERERHxLLPQQXdOgcGDBwMwdepUW7fiiisCpfACKIUYrLvuurburbfeAqIf6HS5w4UffPABUEoY4XrzzTdtuZVDBw0TKlWPsWPHAh3P12CGtc2/kswxxxxjy+bv1Bm2ubRuv/12oDyJRVzug9tmPrh+/frZOvPQ+1NPPWXrunXrlqideecmWzHJFV599VVb9/vf/97r+rbbbjuvy+tsVl11VVt252gy3HPUHXfckUmbsrbIIovY8o477ljx+j777AOUztE+mZDBe++9t+I1N3TQhNh3JkcffTRQSjZSixtavcUWW9iySaDhhhY2MsQqL0yonwnzg9J8olFzlbrzjprfsO48cEsvvbQtm8dWkoTHdzbmWsDMPQalbdFN0GK88847tvzwww/b8muvvQaU/8Yyjwi4CbXM/uAmiTIJs0zyjEbQiJaIiIiIiIhnmY1oTZw4MbJsmIcKXeZh69VXX93WmavUn/zkJ1XX56bJfeWVV4DyETRzZeve8e1KttlmG1s2DyL36NHD1r3//vu2fPzxxwPw1VdfZdS6zsNN/DJkyBBbNttkV3vQ1dhggw1seYUVVgDK7/BVu9vn3nly7ziah5A33nhjWxeVhOCggw4CYNy4cfU2O9dM6moo3ZF1706bEb+0zLHT/Rvq7mylqBEcl7vtdlZ/+tOfbHm33XYDypPRXHvttQ1b99ChQwFYdNFFbd2ll14KwBVXXNGw9TaLO5K/1157Vbw+efJkAN577z1bN2zYsIr3zTfffLZsRsaipt3pbNzfP//85z+B0igWlCICokZIXe5IluFGTnV1f/3rX23ZjA5GJblwrxNeeOEFAH7961/buqipMNZbbz1bNuf5iy++2NaZawl3Hzj//PMBuP76622d7xF2jWiJiIiIiIh4pgstERERERERzzILHUzi448/BuD++++veC0q/LAjJoTDnffFDEV2xjk04nDD2Nwhc8PtlwcffDCTNnVGbniVqxEPf7cCE0p51VVX2bpqc2O4yV3M0H6hULB1UeGs7mf2339/APr06WPrzHxSc845p637y1/+AsD3339f+0vkzMiRI4HyB3ynT58ONCbZignHdMMFH3jgAQA++eQT7+trVT//+c8j600ygVpza3UG7pxNZnuZMWOGrfOVWGGuueYCykOLDj744Io27L333l7Wl0fuIxbzzDMPUJ4wwJyL3OPeLrvsApT324ABA2y5b9++ANx00022bssttwQ6zxxbc889N1B6RAJKj1bMmjXL1p155pmAHqGoh7utmUQV++67r60LggAo/z1kQvrPOOMMWxf3EQszVxyUkl6ddNJJts48ouSG2WZBI1oiIiIiIiKe5XpEKw03rewFF1wAlKePNgkgOstdmbj+9a9/AbDZZptVvHb55ZfbsvtgvSTnpnh2mVGVrqZ79/ZDTq0Z3s0o6qhRo2yde3exGndE67TTTgPgrLPOsnW9evUCyv8GN998M9CayXF22mknoPS9oHTM88VN6jJ69GgAfvzxR1t3yimnAK05IuibeSDbfTDbZe7OPvfcc5m1KU+23nprWzYJQdyR0LhJatxogaipYIzrrrsuSTNbTs+ePW3ZjOKdffbZFe9zkwhccsklQOkYArDssstWfMYdxels6d1HjBgBwHHHHWfrTPIKk1AFSsmWJD6zX0JpGiEzigWldO1u4iB3KpZq3GlazHRP7m9YM22MG8lmuG2YMGEC0NhoDI1oiYiIiIiIeKYLLREREREREc86beigO9O0eRDeJNcAePnllzNvU7MstthitmzCWdwwAxOSZcJ/wN+cO12VCWFx5zN59tlnbfmee+7JvE155yZuMA+txw0X7IgJCTThblB7Dr5W4M51ExUu5XuOMJNUBEphn+68hFEJi7qqWttXZ5u/rZo///nPtrzRRhsBsPjii9s6kzDEDeUZPnx4rGW7n3ETXhj//e9/gfJED52ZSWzhcsM0zWMDUdzkWFGeeOIJW+5svw2iQnzNufrtt9/Oujmdihve54aaGz/88AMA66yzjq0zyZ0GDRpU8f6vv/7alldcccWKsvt7wZ0/b3buPFpZhL1rREtERERERMSzTjei9bOf/Qwof7DRMA89AkyZMiWzNjWbO+O1m/7SuOKKK4DWTASQV8OGDQNgwQUXtHUmtShEz2relbiJaQz3rpYv5q63u76odZsUsLvvvrv3NjSCOyK9xBJLAHDllVc2bH1uymejKx1D6xE1OpAk2UNnMGnSJFsePHgwUJ6GfIsttgBKD8pDKdXzZZddVnXZ5iF2gOeff77i9cceewzoOuc1d/83o4Lu6KoZIXATNG2//fZAecIAd1s19fvtt5+tM/3+0ksveWt7M5kRFJfZLtva2mydSXHfVZPYJHHffffZsol6ML+NAJZeemkAzj33XFsXNTptRsPcEbIoUaNY7jQkN954IwCHHnqorZs5c2bVZfqgES0RERERERHPdKElIiIiIiLiWacLHdxqq60AmGOOOWzdxIkTAXj88ceb0qZmMeEDa665ZsVrDzzwgC27w+Pix2qrrQaUD4N3lflcqjnwwAOB8uH8Rtp2220BWGONNWydWbfbBnf2+Fbw+eef27IJZTGhWVAKWU07T6CZjzAqvOaRRx5JtezOZP3117flXXfdteJ1dw6ervqAvUlG5SZOMeVjjz227uW58z2ZEGE3rOvoo49O1M5Wde+999qy2d7cMEET6hcVmuV+1k0kduuttwKw3HLL2ToTdmWO5a3OJEtzzwcmNPvEE0+0dWZu0fHjx9s6kyTEhMABTJ8+HYAXX3wxcn0rr7wyUP57tLMeE9zkFSZMdf7557d15hEf88gPwIcffgiU5jKD0t/D/K4CWHvttWO14cILL7RlkxinkXNmRdGIloiIiIiIiGe60BIREREREfGsU4QOzjXXXLZsssV89913ts6ExjUyT35euFkFzTCpG0ZpuCEWnW1ejGbp27evLQ8dOhQon6/NZLzpykwoXyOYEJCVVlrJ1lWbQ8dkN4PWOza4IRkmq9qOO+5o62677TYAzjrrrFjLW2WVVWzZDcnq378/EB1ulFX4Zytwj7tRWS01b55/bliX2T7dEER3/+4K3DDhX/ziF0B5uLo7955x3nnnAeX95mbEveGGG4DyLM6bb745UJ6JtJUzO5555pkAHHnkkVXfZ/brgw8+2Na55Xq526d5lGPUqFGJl9cq3LC9qOzg1Vx++eW2HBU66IbUm7/npZdeauui5vLKgka0REREREREPOsUI1ruHBzmoXd3ziIzn0ZXcNRRR9myO4eGYWaHVwIM//bcc09bNkkE7rjjjia1pus54YQTgPKHuaO8/vrrAOyxxx62zn3wttWYfdkkBADYeuutgfhza82aNcuW3dGrhRdeuMPPuHcKu7qoZCHundu//vWvWTan09ppp51s+Ze//KUtmzvZ5kH6rs4kt3C3S5Okxd0uzahgR/M6/u53vwNgxRVXtHUmyZY7ougeS1uNGVW5+uqrbd0///lPALp3L/1EXmqppYDoEeskTAQGlP5OJuEGwCmnnOJlPZ3BMcccA9Qe8XMTtDRyXsl6aURLRERERETEM11oiYiIiIiIeNayoYMmNAbgt7/9rS1/9tlnAJx88smZtykPaj3QOWbMGEAJMBqhX79+FXVm7hhpjNtvv92WV1hhhVifMfPJdJZ5oKZNmwaUHoAHWH311QEYOHBgrGV0NMfbZZddBsDo0aMrXnMTcnRVSy65JBA9d5Y7N87TTz+dWZs6sy233DKy3sz39Mwzz2TZnNxz58dyy3GZfdwNqzOhgxtttJGt8zVvXzOYBAnuPrr88stXvG+TTTYBypOLmfkXox7TqIcJ+15rrbVSLacz2XfffW3ZhFS6oZwuM2eZSd6SNxrREhERERER8azlRrRMGt1zzz3X1nXr1s2WzR1uM2O3lDN3nupJZ21mmXc/Y+7qRKWMdWf+rjbC5qbaNOllv/rqq9jtypttttmmou6WW25pQkvyy9y5i3qgOOputTur++KLL17xurucuOnGG5liPi/M9A3uNA5J/Pe//+3wNTcl/JQpU1Ktp1Wtt956QPT2bBIPiT/uMeLLL7+05T/96U/NaE6Xcc0119iyGdHaeeedbZ2JlOnMkUQTJ06sqDORA+6I1g8//ADAJZdcYusuuugiWz788MOB6FFwKaVtd/fpueeeu+J9blSWSYLx7bffNrh1yWhES0RERERExDNdaImIiIiIiHjWEqGDbmigmR9rmWWWsXXurORuYgypNHny5Lo/c+211wIwc+ZMW7fooosC5eEDabz77rsAnHrqqV6Wl6X1118fgL59+za5Jfk3btw4AE4//fSK18wD7RAdBlgrNLDa6+PHj4/bRHGYUE93ji6jq4YLukwou8vMSfbnP/856+Z0WiY0yJx3AN5//31bVhKMxnKPrebYvd1229k6M5ffVVddZeteeeWVjFrXPHfffTdQ/rvFJGzYb7/9bJ2blGjDDTfscHluAp2uyoT2zzPPPBWvueHCJoQV4NFHH218w1LQiJaIiIiIiIhnLTGiNWDAAFuOSn/pJlxwR7e6IjfdtXvHKY2ddtop1vvMQ6BRIws333yzLUelOn744YcTtq75tt9+e6B85PXZZ58F4KGHHmpKm/LKpF8dO3asrevTp4+XZX/wwQcATJ061dbtv//+QPlorMQXhmHZv1Ju8803r6h78803gVISIUnPjGi52+Ftt91W8T73LvgCCywAlP4e4odJsHPiiSfaujPOOAOA3//+97Zu9913Bzr3NBDmXOMmC3Gn2TDcVPiGmwzMbMvHHXec7ya2BHe/PeaYYzp83z/+8Q9bfuCBBxrZJK80oiUiIiIiIuKZLrREREREREQ8y3XoYL9+/YDSA4cuN/TIfYi+q9thhx1s2QzBujOZR1l55ZWB2oktLr74Ylt+/fXXK16//vrrAZg2bVqstrayXr162fJWW21V8fp1110HlIcHCLzxxhsAjBo1ytaNGDECgMMOOyzVss0Dyeeff36q5UjJnHPOWVHXmUOB4nCPp25Yu/HNN98A9c1VKPVzj62jR48G4IgjjrB1L774IgB77LFHtg3rIi6//HJbPuCAA4Dy3x9mTq0kCbhahTkWmrmxoDTn05AhQ2zdIossYsvmt9OECRNs3UknndTAVuaX6auXXnrJ1kX9XjXbkNvPrUQjWiIiIiIiIp7lekTLPMi+9NJLV7z24IMP2rIe1I4WlUK7Gs1UHp97t/rjjz8GyhN+KLVzdW6SEFN2R67Nvm9SvUKpfy+88EJb56Ydd++KiR977bUXAJ988omt+93vftes5uSCm+zHJPZZZZVVbN306dMzb1NXtO+++9ryPvvsA8Df//53W9fVt9NGM8mHAIYNGwaUR7oce+yxQGm0sTN77733bNmcs0wyEIB1113XlguFAlA+PUFXtfHGGwOw5JJL2rqo3/NmpNpEC7QajWiJiIiIiIh4pgstERERERERz3IXOrj++uvb8iGHHNLEloh0zA0dXG+99ZrYks7jzjvvjCxL8/z73/8G4KyzzrJ1999/f7OakwtuEoYTTjgBKA93mTRpUuZt6uzGjBkDlBIsQHn48bhx44BSGDfAd999l1HrxMxVdu+999q64cOHA7DSSivZuq4U3u0mu3DLUmLCe6PCBc3cbND65xyNaImIiIiIiHiWuxGtoUOH2rJJ/eh69dVXAfjiiy8ya5OISFfkJiORSjNmzABg7733bnJLOrdHHnkEKD08L/k0cuRIW37++ecBGDhwoK3rSiNaUtuCCy4IlCe1MklCzjnnnKa0qRE0oiUiIiIiIuKZLrREREREREQ8y13oYBQzBA2wySabAPDRRx81qzkiIiIi4vjss89seZlllmliS6QVmCRLbrIlkyBj5syZTWlTI2hES0RERERExLMgKq1ih28Ogvhv7nomhWE4pN4PqU+rUp/6pz71T33qn/rUP/Wpf4n6FNSv1YRhGNR+VyX1aVXa//2L1aca0RIREREREfFMF1oiIiIiIiKe1ZsMYxbwRiMa0gn0S/g59WnH1Kf+qU/9U5/6pz71T33qX9I+BfVrR9SnjaH9379YfVrXM1oiIiIiIiJSm0IHRUREREREPNOFloiIiIiIiGe60BIREREREfGs3mQYdQkKwY/AC8X1TAX2CNvCrxIua0Pg6LAt3CYoBMOBlcK28A8dvHd+YNewLbwgxnIfBuYp/ncR4KmwLRyRpI1ZaJE+/QcwBPgeeAo4IGwLv0/Sxqy0SL+OAQ4HBgB9wrZwVpL2ZaVF+nQZ4CpgIWASsHvYFn6XpI1ZaIU+dT5zLrB32BbOnaR9WWmFPtW+r+20Ffo0KAQbA2cCPWg/nu4TtoU/JGljFlqkTy8FNgA+LVbtGbaFzyVpYxZapE8z3U4bPaL1ddgWrh62hasA3wEHui8GhSAICkHdbQjbwps76uyi+YGDYy5raLGNqwOPAzfU256M5b5PgX8Ag4BVgbmAfettTxO0Qr8+CgyjdTIAtUKf/hE4O2wLBwIfA/vU256MtUKfEhSCIcAC9bajSVqhT7Xvo+00z31aXPdlwKhiG98A9qi3PRnLdZ86xprfqXm+yCrKdZ82Yztt6IjWbB4GBgeFoD9wF/AksBawVVAIVgAKQE/gVWCvsC38IigEWwDnAF8Bj5gFBYVgT2BI2BaOCQrBosB4YNniywcBhwIDgkLwHHBP2BaOrdW4oBDMC2wM7JX+q2Yml30atoW3O8t9CljSy7fNTl779dniMv190+zkrk+DQhDQvs/vWqy6DDgJGOfnKzdc7vq0uKxuwBm09+v23r5tNnLZp9r322k7tfLYpwsB34Vt4SvF/98DHA/83cs3brw89mmry2OfZr6dZvKMVlAIugNb0j6cCLAccEHYFq4MfAn8BhgWtoVrAk8DRwaFYE7gImBb2v8wfTtY/LnAg2FbuBqwJvAicBzwavGqemyxDbXuAowAJoZt4WcJv2amWqFPg0IwB7A7cGfiL5qxVujXVpPjPl0I+MQJGXgbWCLVl81IjvsUYAxwc9gWzkz5NTOV8z5tSTnvU22nldL06Syge3GUEGAksFSqL5uRHPepcWpQCCYHheDsoBD0TPNds5LjPs18O230hdZcxS/6NPAmpSvGN8K28IlieV1gJeDR4nv3oH0SsEHAa2Fb+J+wLQyBKzpYx8YU70CHbeGPYVv4adSbwvbQwGp2Aa6M97WaqpX69ALgobAtfDjeV2uqVurXVqE+9S/XfRoUgsWBnYDzkny5Jsl1n7aoXPeptlP/fVpc7ijg7GIky+fAj/V/zUzluk+Lji+u6yfAgsCxdXy/Zsh1nzZjO2106ODXs3/RYujDl24V7cN8u8z2vsxOOEEhWBhYm9YIH2iVPm0D+gAHZLXOlFqiX1tM3vv0Q2D+oBB0L45qLQm8k8F608h7n64BDASmF9vVKygE08P2Z+DyKu992ory3qfaThsgbAsfB4YW17kZsHwW602hFfrUjLh+GxSCS4Cjs1hvCq3Qp5lup3lI7/4E8LOgEAwECApB76AQLA9MA/oHhWBA8X27dPD5ibTHZxIUgm5BIZiP9ivUeTp4f5SRwK1hW/hNki+QQ03t06AQ7AtsDuwStoX/S/41cicP22pn07Q+Ld7Zup/2/R/a76rdlPSL5Egz+/S2sC3sG7aF/cO2sD/wVc5/vMalfd8/baf+Nfvcv0jx3560j7yMT/pFcqTZfbpY8d+A9kdcpiT9IjnS7D7NdDtt+oVW2BZ+AOwJXBkUgsm0Z/4bVLzo2R+4LSgEzwDvd7CIw4CNgkLwAu1pGlcK28IPaR+SnBIUgjOgZvzrKFojbDCWHPTpeGBR4PGgEDwXFIITfX23Zmp2vwaF4NCgELxN+8jL5KAQ/M3j12uKZvcp7QfZI4NCMJ32Z7Za5cHtDuWgTzudZvep9v1I2k5nk4M+HRsUgqnAZOCWsC28z9d3a5Yc9Ok/ip99AVgYOMXTV2uaHPRppttpEIZhI5cvIiIiIiLS5TR9REtERERERKSz0YWWiIiIiIiIZ7rQEhERERER8UwXWiIiIiIiIp7VNY9WEARVM2f06tULgK+++irW8sz762GW7X427vritiHh8maFYdin3g9179497Nmzvom+o9oX1fdJ+rdRotoVo58z69N6xe3njr6jj79Nte2gyvsS9WmtfT+uau2L2ydp9vckGtWnvvZ933wdE5PsF47c7fsJ97fE6u2/LPf9Wvtq3PbF/Wya9TZYoj6FbM5TcdVznqr375NkWw3DMOhwJVVE9Wkjtwlfv7Ey2m4beu5P8ps8zTEh7rEgzTHD1zHV64TFK664IgCTJk2q6/31MMt2Pxt3fXHbkHB5byT5UM+ePevuh6j2RfV9kv5tlKh2xejnzPq0XnH7uaPv6KN91baDKu9L1Ke+VGtf3D5Js78n0ag+9bXv++brmJhkv3Dkbt9PuL8lVm//Zbnv1+rjuO2L+9k0622wxH2axXkqrnrOU/X+fbLcb6L6tJHbhK+/X0bbbUPP/Ul+k6c5JsQ9FqQ5Zvg6pip0UERERERExLO65tHyFT5krLXWWl6W4+tugGlP1PLctnawvklhGA6pd529e/cO83JXqx6mDxr8N8xdn1bb1pL0hbu8ej9fa7vvYHtO1Ke+9/00fPVzkuOGzz5Ns52mOeZF9V+a7TCtPO/7cfexWp/13afN3vfjfp9G3qkzbb5WAAAgAElEQVSv1YZq56i4f5sY7U/Up5CuX8sa4OFc3ISRQCtqW00aOhi1/zfzu2Upxv7QkHN/yv3HizTn9JTriNWnGtESERERERHxrK4RLfduQaOuHNPcpapH3PbXcaWc+d3CvPI4StOyd2CTrC+jO5INGSnI8q5hVvtHs/u0Xo34G2R5LOos+36URvZjs7fTes/JWW2ntSJTkvKxnYK/KIF67+bX0wdx+9D33zTpiFaS/T/uyKdvabbZJNt20hGtuKOEUVFfMSLBGi5uGxK2VSNaIiIiIiIizaALLREREREREc8Shw5GSRPWl3XIVtR6U4Y+ZJZkoN52brjhhrZurrnmAmC11VazdSNHjqz47Lhx42x51VVXBWDChAkVy04y3F7Hw5MtGz4UN1whyWeTLte8pZlhbr7DB7IIbcsybLiWPITBxdUZwoZ9Hd/SJF6JK4vtNG6YWh6200YdG3yFDjbyUYy4ib1qqffzvr5HlqGDrShhMpfM9v84bau1nKzCrbNIhqMRLREREREREc+8jmj5kId0sHE/06y02XGv9DfZZBMgesSqHq+++ioA11xzja178803gcammabBfdrI1PRxt7V///vftu6www4D4Ntvv0217GqfpclJW+rdXn7+85/b8nLLLQfAAQccENkGs52bbbOetuSlT5PI+iHuetV6SHr218x/W3Xfj1pHs0ZyW3FqhzR3mH2NpDX6vA/JIoTq3Y7yNpoTt/0+RrTSjOr5nr4mygILLGDLSy21VIfvc89nhx9+OAAvvviirXv55ZcBmDx5csVnm73/521EK8360pz7NaIlIiIiIiLimS60REREREREPOve7AbMrplD3VFDzY0OwenVqxc+ZjI3nznmmGNsXbWQwWnTptnyXXfdBcCyyy5r67bddltbHjBgAAC77babrfv9738PpAtbanZ4U5IHOuv9bh1tzwMHDgTgf//7n6175513AFh44YUr1l1rm4x6XxazpVfjY72LL764Lf/tb38D4Mknn7R1//d/pXtFZpvde++9K5aTh/k8kqq1j8X9Ptdffz1Q2nc7EjekcrPNNrPl0047Ldby8qKRCS2iPuvWzZw5E4CbbrrJ1h1yyCEAPPHEE7HW4SuELo6oc1Qt9f7NP//8c1t+7LHHALjwwgtt3euvvw50/L3SbGPmGDNjxoyqyzPrboVjSbOSBNVabzPOSWn+Xr6S2Gy99da2PHz48Ir3uY8QzM6EBgLMP//8AGy88ca2zpQPOuigqm1p9raaZv2N/K1W7f2+9iONaImIiIiIiHhW14jWV199VXHVl2TW6kZeOfrW7DswRtw72Ntvv31F3U9+8hNbNqNcs2bNsnVffPEFUEp6AbD00kvbskkFv+CCC9ZsO+TrLooPte7wJxnBM3+HL7/80ta98cYbQPmIVtR6o9YXt86nNPtx3AdP11tvPVvnjmRFce+Ad7S8Wq83645soxJabLrpprbco0ePxG2JYu7MAvzwww8AnHHGGbE+26hts1aEQNy/X9xR47gWWmghW3766acrXn/88ccBGDp0qK37+uuvY7Uris/+jXveT+LOO+8EYO6557Z1JoLCjGJ1JMl3nHfeeQG4+uqrbV2fPn2A8lGCauvz9d1r9avvVPrmu7ujzyuvvDJQfpyIGlXN6++zanzvw3ENGjQIgP3339/WuUmdzOhVtVEs1worrBDrfUmSO8VRa0S7Wl9mMa1F2vVlsTyNaImIiIiIiHimCy0RERERERHPUifDiDu09uGHH9qyeRjbffj0m2++AeCKK66wddtttx0A06dPT9vMumQ5TO6GD6RZr3mgNwhK00+YkEE3sYV5EDvK2LFjbXmHHXawZTPEbf4eAJ999lmsNjcj9DLJw9tGmhCzev5+5uH3yy+/PNbn85TkolEOPfRQWx4xYgQAU6dOrfoZN/zChGe4CTKee+45gLLtod5Qh2Y9WB7nNVdUeKn7ELZvbgjcSiutFOszeQnFThKqmWYOFjckMCp0cJ555gFK58Fa62n2vp9m/aeffrotm3D0vfbay9ZdeumlQPxzSy1uW0eNGgXAMsssY+tMiFceHm+IG/4Vty2jR4+25X79+gHlx0cTstlREhZfoeBZcc/91cLpfT/e0Lt3b1s282G64oYJukzCMnfOrLiymAdsdkn6tNpxOOq17t1Llyzub9QNN9wQgB9//NHW/fWvfwVgypQpsdYb1QZfNKIlIiIiIiLiWRCG8Sd9TjND9LXXXmvL/fv3j/UZk/I1yRV9LW+//TYAf/zjHyteS/hgeupZt+u9Qxf1fjeJhUly8dFHH8VabtSdVijdjRk2bJitW2655WItM0oddwsS9Wnv3r3DOCnzk9wRjRJ3Oe6ortkfzJ0YKD1EG3fdWW6nbp82ajb3jrY/IyoZhnt31k2Vb5gEIxMnTrR1DUgcknrf98Ht+9deew2AO+64w9Z98sknAPz617+ue9lR/bPBBhvYshmlWGeddepedkerzMN2mmbf79mzpy0/8sgjVZdpRh6XWmqpWO2KksU5Ko3jjz8eKE/YZKIu3EiMuJL8Dc257IYbbrB1JhGJm2I+7vJI2KcQ3a++RiCWXHJJAP71r3/ZOnN8dX/zmf4/+uijbd2f/vSnWO3xPcWAu7wwDOvfIIi//ydhtlt39P7UU08FYI455rB1t99+O1Ce6Mod8br77rsBeOGFF2zdU089BZQnLjPJcNzlRGnG76la60+zHa+66qoA/OpXv7J1Q4Yk2sU65I4wmkia77//vuJ9vo6pGtESERERERHxTBdaIiIiIiIinqVOhhE3fMkdqvvuu+8AeOmll2ydCa1YY401bJ0Jq1p33XVt3VtvvQXUDrEw87oAfPDBBwAstthiFe978803bfnKK6+seL3RczEkSdxQbVjW/T5Rotp81VVXVf2MCdkyCTBqqfVAb54f6K714Hya2cbHjRtnyyakrVZYQL19lcUcZtXalGQuDxNqUYsbJmi44ZgmVNY89A2lh9733XdfW1dtvi1XmrCZOGrN+VQvE3IBpYRD7tx4u+66K+AvPMmdR8vYYostbNkcd5u1n8fdTmd/f0d19T5gPnjw4Krrc89RUeezRoXo1ivNPDquHXfcESgP8XW3Wd/MHFEnnXRSxWs33nijLbthXc2WJhzP/awJb3N/d0WFZ5rXd955Z1vn7sOnnHIKAN9++62tMyFWcX/7NXv/d9XblqhHa04++eSq6xg/fjxQPg+c+0iH+Q3rLtucp9z5TePyFQZdD1/LMsdIN0ww7vd55ZVXbNmEym+++ea2zhxf3QQat9xyC1CeIG6fffYBSn+3RtCIloiIiIiIiGepR7SiRF2Rug+ju+XZPfzww7Zs0jK6o1zmgU73ocEobprcc889F4C77rrL1pm0sm6CB/cKuZpmj8TUK2qUYZtttrF17t0s49Zbb7Xl4447rq71xR1VyWL0JWpdzVqe+0Bn1LZWb+r4Zo8cpvn7mVTsAIsssghQfvfV7N9uAgyTaME8vD77659++ilQnjL/hBNOqFj32muvDZQeQIbGj17FlWY7db/ru+++C5TfnTZ8PcTuJsPwkQY6i+00zXrr/du4SR+i1EoKVW2fj9r3fdzBT6Jav0yYMMGWv/rqKwCeffZZWxeVejnOcjvifl8zgtO3b19bZ1LHJxnFytN5v1Zb3HT5xuTJkwF47733bJ37+8eYb775bNkkyXBHAKJU2waz3C5rTZcT95xoRp1cd955JwD33HNP1c+6I1lGrUijRmnWMTXuZ80x0h3dM7/x3euEn/70p0B5IqeoqTDWW289Wz7ooIMAuPjii23dAgssAMD1119v60y0i1tnojF80YiWiIiIiIiIZ7rQEhERERER8Sx16GCSh9/j+vjjjwG47777Kl6rFn44OxOiYYYNoTSHQa1EEHlM3ODj4cda8xJcc801tmwSNiRZb70PpPtQK3wgrmrtq/X3MK+vssoqkZ83Q9Nxwypqhek1YyZ4V9z1mn3xrLPOivV+d14ms8+7D7ebuUZc7oPH5vNuiKGZ88kNDzn//POB8rk0mrHPJ9nPR44cCZTmBoRSspVa8yQm2V522WUXoDypgQnbNHN11VpHo84bSfZ93/uMWf+0adNsnRvSYpJB/eY3v6l72VHH02bv+9W4259pn3kgHUqJrqLmsKnHXHPNBcCIESNs3cEHH1zRhieeeCLxOrL+LVBvyJ2ZOwtgo402AkoJgqAUSjXnnHPaOhMmaPoKyuczNAnELrjgAlvnvjdOu5q1Xda7X7ihvm5fGltuuWWHn83DvpfF7ykj7vd1t7V5550XKE9MZcJ73d+j5rGVeh5ZMe2ZZ555bF23bt2A8t8LO+ywA1CeMCvOciF6e4rb5xrREhERERER8awhyTDywL3SNOmj3fTQ5mHxjz76KNuGJZTkjknUZ8xM8VF3bA455BBbdh9i9rHeLBNfuNLMWu7jLlVH6YvNqErUlANR4o5yNXvktZY55pijos6MNrn7p+mfY4891tbF/Xu4Dx5fdtllAOyxxx62ziTdWG655Wxd3DvdjX6wO8k2t9NOOwHlqe7dqQR822233YDyhETHHHMMED0yUat/GrXNNmukp0ePHkD5KJbLRAg899xzVZeTZioJn3xFCBjuHeajjjoKKB8JNcmv9t9//6rLcZOxmKlgTMIHl0l81ZFmjcbUmt6h3r9nz549K+rOOeecijo3iYBJFGBGxaF8dMGMbplEJklkeU5y+zTuOXHQoEEA3HDDDRWvuSOkWWhkhJgP9bbJ7JdQmirAHTE125o7SmqSVNVzrjWRMm6Emjn3u8y63akOzDRTUdEYrjR/D41oiYiIiIiIeKYLLREREREREc/qCh30PdTtqjckp9Yw3vDhw225T58+QCm5BpTn44+z7CzngvA9fOyGp0WFDN5xxx0A/OUvf4ndLiNNSGOzwtyyCCkyIXLufCbu/DFmJviouSCShFlm2adptgMztH/44YdXvOaGFLghg9XWG8Vty0033QTA6NGjY322o+U0QpqHjM2DxVAKl1piiSVsnZv8w4cdd9zRlhdeeOGK103CoiRzdDV6m/V13Iq7jvXXX7/qe3/88cdYy4krat9v1PGgWl9GvbbyyivbstkX3fORCT91H2I356MotebPdP33v/8FSvPmuW3NqzTH/htvvLHita233tqWzWMDLrOOZ555xtYttNBCtmxCrHwkEYlar/t6s34P/OxnP6uoM+HnZputpZ6215tULM2jDj76NOp3f1wmIUVHzNxVzz//vK0bOHAgUArpdG2yySa27LbJlN35r9zfE7Nz+2rXXXcF0j0+UItGtERERERERDxLnQyjVvrD2V/r6PUk64ti7k6MGjWq4rXtttsu1jripu72Ic1dbVe1vndnvI5SK/FFtTumaTTj7nZH66r2fZJ8b3PnxX0Q271TGzWSFaXe9PhZJB3xcQfSTXxhRKUNrkfU38as529/+5utO/DAA4FSUgyAI444AoCzzz47VRvSSnLs+cMf/gDUnqoijQEDBlTU/ec///Gy7Gbc0U7Sz3H3/aipM9wHrcePH9/hZ5McE9OkHc6SOf+6o7FbbLEFAGPHjrV15q501MPsLpPSHeBXv/pVxeuPPfZYRV0e+6Uj1baFqNeuvPJKWzbJaUxiESiNELgJmkyyAnfqm6hRw/3228+Wr7vuuljtj5LHbdVNBGKYKULcEUEzEutGphiN/M2YxYh8NVG/UWu1JSoRiZkOZNNNN7V1pp/PPfdcWxc1JYmJAqg1Qta3b19bjjoOv/baawBsu+22tm7mzJlVlzm7JH9LjWiJiIiIiIh4pgstERERERERz+oKHaw1hGgkCXWIUu9Dg1AainTn65k4cSLQOnNBJAlVi3qfSQjiPhRo+uWBBx6wdW7IQdTysvzuPtZbK2lLXGnmsTGzmrvD4LXmc6mm1nbQjHm0koQv7LLLLkApjCDJsuvZP0xYgBtCY7htqDdksNnhLi4zH1NHc7alscgiiwDR4TW//OUvbTlufzQ6EY2vfd+I284555zTls027vr0009t+e233461zLih+NXqfIj7MHzcY/dnn31my9dcc03Zv3GWHbWOMWPGAOVzk7nhSEk1cl6juCFZtf7W5n333nuvrTOhg2549BVXXFGxHPP6sssua+vcxy5MOPJmm21m60wY4UUXXdRh22tp9vHT/Vua8FWX6Ze2tjZbZ84Xbujv4MGDATj++ONtnel7k4wFYPfdd7dlkyDm8ccft3XmmJDk908zjqlR66/FzKNl/oXSHJBuQhIzH6Q7H6aZI2611VazdVGPGri/dU3ZPc+bJHi15szyTSNaIiIiIiIinulCS0RERERExLPUWQddaebz8JVZZcsttwTgu+++s3UnnnhirM/mfY6NONw5MMx8N24YpRGVOcfV7Ew3vsXNjtnRZ+JwMxQNHToUgGnTptm6qHlMfIkbatcM7vqPOuqoDt+XJkwWSvPluWEOG2+8cYfLWWeddWKtLwtpMu+9+uqrQPlcV+b4d9ZZZ9m6Cy+8sOKzZm7BVVZZxda5GQb79esHRGeCiqqLK+/hyPWGrbvzE0Zl1DTz5sVdXi1JwuqzlMW54pBDDrFlsy1Gzb/nStNvvvvS97ykH330kS2bMHU3Q6BZnrvfmrkzL7/8clvn9tHdd98NlIcOmuOIGw5rjkFZhrQmEfX3v//++wHYaKONqn7W7Ne1suOacMt65nybPn06UB62medzehru97r22muB8jDLag477LCqr3/++ee2fOSRRwLQo0cPWxc1h2HcjIqpsiwn/qSIiIiIiIhEqmtEK8lDsVF11R7cT3LHyYxiQemh9zvvvNPWuaNb9a4n6v0+7ya4fZpkrqTZ3+fe1Y66o2Jmj3cTYORppMpH39aam8z3vDnGnnvuacsmiYA7d1ZcvubH8vl3TTLfm4+/ZT0Po59wwgkArL/++rbuySef7PD9Zu6spO1JK+p4muRvdvrppwNwwAEH2Doziu3e4Tflp556qmIZs2bNsmX34WEzShglaqTc1Yw7snG3U1/7hlmOGcF2uQ9c15rLMKpd1c4HWSYr8jXXow/m4XkozVUIpTvZ7qiOkaStzZjbzV1vPeuO2iZMcgszIgXwxRdfAPCLX/zC1rkjWVHr/d3vfgeURwkstdRSAPz2t7+1de65r9WYxFVrrrmmrRs3blzF+8y5JGrEOoqbiKTW6NbAgQOB8uOtmU8uriyPqVFqbbtpjsMmwchKK60U+bpJfHHQQQfZupdffjnWsuNKMz+aRrREREREREQ804WWiIiIiIiIZ4mTYaQJtaqWNCPuevv27Wvr3LkOzBwd7733Xt3Lnn0drqjQM99DtT5CbcwDgB057bTTOlx2nkIIfavVf/Vuf1HvN4kDXCbZQD2ShDbm/SHkRrn99ttt+cEHHwSqhwsCvPvuuwA8/PDDti7NscunWsfEau10Q4JMCLWb2KKaqIfmAQ499FAARo8eXfGZr7/+umq7qu0racIw6pUmJLsWkwQjau4sd74sN4wobhvinicbvU0mmZusUecS9zEB95GAW265pcPPpAnFbmSIphuS1aj+cufWMp544glbrrXtPPLIIwBcffXVtu7oo48GyhPoRM0RWe8xIYvzVdT6TYIEdx8dMmRIxWdNqKobMn3SSScB0L176ad01L4eVRfFXa8739Tssuy/uPNo+U6gtu+++9qye26L8uKLLwL+wgV9H1M1oiUiIiIiIuKZ1/Tuce/O+LhKdGd979atmy2bO4zurNutzPcdr1/+8pdA7YdgXeuuuy4A33//va0zd3Xmm2++ivcvsMACtty7d+8O2+Km2jTpTX3cQYybtCXJ8qv9HWbOnGnLiy++OAC33nqrrTMPvNZqT5K081k+vJ1mWwyCAIh+oNi9W224M71Hcb9v//79O3yf+zCyaYMrL+myfe3nZvqGWtM41GrDsGHDOnzfqquuassvvPAC0NjRozji7vu+rLfeekD09nzTTTdFfqbaNlRr9KpVog/q3U/ifp8333zTluedd15bNoke4t7p95VwyJe8Rh+YPnFHtIYPHw7A8ssvb+vGjBkDRCcjyVLcxA1JoqkmTpxYUbf66qsDMHLkSFtnzjX777+/rbvooots2SRhckfBzWfcSC2TSCtJ8qtmTJ/haz8zx9IDDzyw6nofeOABWzaJoNL0QdwkfkloREtERERERMQzXWiJiIiIiIh4ljh0MG7iBh9JBtzQwPHjxwOwzDLL2Dr3AcKdd9451vqqiTvnV5aS9Knhhk1ddtllZf+6Onpg08ze7YbGLbroooCf/obSnBzuA/ZJZT2XjplDx/QJlLZJN3zgvvvuq7qcNOF/Wc6vkyZ8ycxPYuZ+cpmQFIiedyQqyYUbsuXO/2Sss846QHkocVQ/p5n3o9nHBsN3KCyUwiyjwi1NuKDP9aXla86nuNv4QgstVFFn5iRzj2W+9+mo80GW22atdcU9lsX925gwIje06v3337dlNzRr9mXHbWutdrXS3Fpx1Vpe1Da4wgorABCGoa0zx+6rrrrK1r3yyisVy8sy5DXub9Q0j7yYecrc0EHjwgsvtGX3sYENN9wQiD7HvfPOOxV1zZ4HLsnvqTTh0WbutigPPfSQLbu/F7799tua662lkUmHNKIlIiIiIiLiWeDelaild+/eYb0PGsdNnRp3GVGjLrfddpstV0vzWmvZ1doV4yp2UhiGlTlBawiCIP4foKjalfUJJ5xgy9tttx1Qe1byKHHTkbp++OEHoDzJheH+XaKWbdLIDh482NZNmjQpsz6NEvcOhnko0038YUa0zIgKwFNPPVXXet11e7xb3ZA+rXdqAnd76NOnD1B7dCqK+xmznX/66ae2zowozpgxw9ZFjZrWm7Bhtvdn3qdV2uLVzTffDJSP1hoHH3xw1c/We+zPok+rtSVJP5rkDEsttVTFaz/96U9t2U2pXa8k+34H38lbn6YZmUgSnWGS4rjnjksuucSWzUi5a/vttwfKE2hEtaGaRm2nUN6vPqZ2icvXCNkGG2xgy2eeeSYACy+8sK3bfffdgfhRKm5bwjCsHEKPoZG/Uau9zx29W2655epaP8Bbb70FwG677Wbrvvzyy4r31RtN0ornqWrXJG5yrKhkGVlMuZSkTzWiJSIiIiIi4pkutERERERERDzzOo9WlDQhBksvvTQAv/71ryteM7OTQ/UhVl/tyvND8K5TTz3Vlh977DEAevToUfE+M+8AwMorrwyUz4sRFW7ozsnz+uuvV7x+ww03ADB16tQ6W10KGcxjn9Zihv0HDRpk666//nogOoyyFo9hLbnlJlExIT6HHXZYqmWaUKLzzz+/7s/GTZDhkzvnU71zkMSdbymtOeecs6Lum2++6bANcdebJrmPb3H7OYpJcLPHHnvYOtM/acIFa6kV0tTobTfu3953EiL3vOQmyTLhWmaOIoApU6YAsOeee8ZqQzO3yWp95zuU3Ne28eCDD9qymQfTZf4mkydPjrU8852S/H5olLh95f52KhQKAJx44omR7zXJxSZMmGDr0jzy0uhtNWpuwjSJpKLe7845aMID3SR3xuGHHx6z1dWlmW8vCY1oiYiIiIiIeJZ4RKveJBJJrrrNqEtUCl139MB9QM538o00y4mj1t2CNO347LPPKurMsu+6667Yy+kMkqRujeuTTz4BSokDoHqK0o7alSYlaqNFbadx1WrvtGnTgPK7Wsa2225ry+Z1N5WzOzL70ksvVXw+zd81y3TOPpILuMtI83C9+9lNN9204nUznUE9y2ymRo74meQM7ohW1N39NBqZdjiOqJHXJOdS378P1lxzzYryM888Y+vMMTjP22Y19SboqXUuaWRynV122QUoT1Yy//zz17WMRk9H0hHf+09bWxsAV199ta1zk2K5SVx8SLMPJZVkRD1qhPbdd98FYIkllrB1UckwjjnmGKAULdDROuK2NevpMTSiJSIiIiIi4pkutERERERERDxreDKMerlhKT179oz1mTRhVUlCypodiuB7qDjLUMlafM3xMbu44Te12hS1vEMOOaTu9tTbhrhJGho1/B13dvg07rzzzlh1/fr1q/p6V5UkGUGtbWPmzJkAnH322bbOzIsTN1Sp1jqakbghbh/U2sbNHE3PP/+8rXPD1+rlKySn2vsbpdb60yRy6NWrFwAnn3yyrXvooYds2czh+PHHH9u6NMlIsui3JI8NxD33x03u4/sYfu+999ryiBEjAJg1a5at8x3e7VPcpCOuam13v2vU907SrmrrbdTjBe65P+7vlmrtc99vwnvdcEGTBGPixIm2ziQdagQf360WjWiJiIiIiIh4VteIVtRd7bjiXm2/+OKLtjx69OiK183V7jnnnJNo3bNLcmeo0eLeRcmqTT7uhNWbPCWNRt4pNBo5mtjIOys+NTtBR0eyuHPbWdTqFzcZSdLlZLntJknakmbbmDFjBgD77LNP3ctNE0kQd6SoUbJOtmBGUceOHVv3Z+PKwzE2SXKkqM/GXZ7v73f88cfb8nXXXQeUj0b07du34jPN+I0VJcn6fW0z1fogTxFHRpL9P+r1BRdcEIAgCCpeGzZsWMLWdby+uP3su081oiUiIiIiIuKZLrREREREREQ8qyt0MGo+DVeaoV/z2ah5W9wHjd2Z4ePyMVzYqCQNSeRlqD2tZodjtnr/xZ0fIgt5CNfzEWoTJU/7vpEmxKij5cRVbx+0Sihs1nxvV1ns+/X+3fK479SjkdtpmkcxoiRJ+JRkLqK4nx05cmRFXVTo4OzrmDp1aqw2xZUmfDLJPhX1+6yR55os96s055oon3/+OVD+vS+99NK6l1OtPVmc42rRiJaIiIiIiIhnQdQszB2+OQjiv7nBkjxo3OC7a5PCMBxS74fy1KeurFLH1/p4Xvu0EXeMM7ozlds+jdLIUauXHngAAAeqSURBVDGP/d1SferK4mHuhHLXp9W+Yx7umsZZZd761Le0CUgSSNSn0Fr9WouP47T7twnDsDI7Qgyteu6PqzP8nmpE0pwmRkjE6lONaImIiIiIiHimCy0RERERERHP6g0d/AB4o3HNaWn9wjDsU++H1KdVqU/9U5/6pz71T33qn/rUv0R9CurXKtSnjaH9379YfVrXhZaIiIiIiIjUptBBERERERERz3ShJSIiIiIi4pkutERERERERDxr6IVWUAh+DArBc0EhmBIUgmuDQtArxbI2DArBrcXy8KAQHFflvfMHheDgmMt9uNjG54JCMCMoBP9K2sbMBMGPBMFzBMEUguBaguT9ShBsSNDerwTBcIKO+5UgmJ8gXr8SBMsQBE8SBNMJgqsJgh6J25iBFtlWxwSFYHpQCMKgECyctH1ZaZE+XSYoBE8W+/XqoJDv7bRF9v2NCYJnim28jCDonriNGWiR7bSl9v0W2U43KW6nzxEEjxAEAxO3MQut0Kelz5xLEHyRuH1ZaYU+DYKAIDiVIHiFIJhKEByauI1ZaI0+fbjYxucIghkEjf3d3+gRra/DtnD1sC1cBfgOONB9MSgEQVAI6m5D2BbeHLaFf6jylvmBWB0etoVDi21cHXgcuKHe9jTB14Th6oTR/VrcMev/24bhzYR++hX4I3A2YTgQ+BjYp+72ZCv32yrwKDCM1skA1Ap9+kfg7LCtdbbTXO/77eu+DBhVbOMbwB51tydbrbCdtty+n+vttN04YDRhuDrwT+A3dbcnW63QpxAEQ4AF6m5Hc7RCn+4JLAUMIgxXBK6quz3Zyn+fhuHQYhsz+d2f5Z3Gh4HBQSHoD9wFPAmsBWwVFIIVgALQE3gV2CtsC78ICsEWwDnAV8AjZkFBIdgTGBK2hWOCQrAoMB5YtvjyQcChwICgEDwH3BO2hWNrNS4oBPMCGwN7pf+qmXoYGEwQ9Ge2fiWo7FfC8AuC6H4laO9XwnAMQcf9StDer4Qd9GsQBLT35a7FmsuAk2g/sbWCXG6rYVv4bHGZ/r5pdnLXp0Ghc2ynudr3YSHgO8LwleL/7wGOB/7u4wtnIHfbKXSOfT9n2ylACMxbLM8HzEj9TbOTzz4Ngm7AGbQfU7f39F2zks8+bX//roTh/wAIw/d9fNmM5LVPzTIz+d2fyTNaQSHoDmwJvFCsWg64IGwLVwa+pP1O0rCwLVwTeBo4MigEcwIXAdvS/ofp28HizwUeDNvC1YA1gReB44BXi3cpxxbb8FyNZo4AJoZt4WcJv2b2guh+JSzvV8JSvxLU16+Elf1avBMwttiGqH5dCPiEMPyh+P+3gSXSfNWstMi22lJy3KcLAZ+Eba23neZ4358FdC/e1QYYSfvd2NzL8XbauvK7nQLsC9xOELwN7A5Uu1ueH/nu0zHAzYThzHRfMmP57tMBwM4EwdMEwR0EwXLpvmxG8t2nxghgImFjf/c3+kJrruKJ42ngTUp3Nd8I28IniuV1gZWAR4vv3QPoBwwCXgvbwv+EbWEIXNHBOjameAc6bAt/DNvCT6PeVAwNrGYX4Mp4X6vp5ipuQBX9SljZr8X3lvUrYfgfwnj9Shj+SBjdr8Wh186glbbVVqE+9S/f+377ckcBZxMETwGfAz/W+R2zpu3Uv3xvp+2OALYiDJcELgHOiv/1miLffRoEiwM7AefV/c2aJ9992q4n8A1hOIT2i5CL43+9pmiFPjUy+d3f6NDBr2c/cRRDH750q2gPm9hltvdldsIpPly8Nq0z1P11xQYURPcrYXm/EmTSrx8C8xME3YujWksC72Sw3jRaYlttMXnv0w+B+YNC0L04qtUS22nO930Iw8eBocV1bgYsn8l6k8v7dtqK8r2dBkEfYDXC8MlizdXAnQ1fbzr57lNYAxgITC+2qxdBMJ3257TzKu99Cu2RFuYZohtpvymQZ63QpxBk97s/D+ndnwB+FhTaM/4EhaB3UAiWB6YB/YNCMKD4vl06+PxE2uMzCQpBt6AQzEf7XdR56mjDSODWsC38JskXyKkngJ9hMikFQW+CUr8SxO9XgqAbQR392n4n4n7a+xXa71bclOhb5EsettXOpml9WhyB6LTbaVP2/fbPLFL8tydwLO2x9K1O+75/zdxOPwbmK64PYFNgapIvkTPNPO/fRhj2JQz7E4b9ga9yfpEVV3OPp/AvYKNieQPglSrvbRXN7lMo/u4nbPzv/qZfaIVt4Qe0Z1W5MigEk2nPADKoeNGzP3BbUAieATp6APAwYKOgELwATAJWCtvCD2kP8ZgSFIIzoGbs+yhaJ2wwnrDUrwSlfi1uVPsDtxHU7leCUr8Stvcr7Wk7zwCqxcAeS3vM7XTan4VplYfhO9TsbTUoBIcGheBt2kdeJgeF4G8ev15TNLtPKW6nQaHzbKc52PfHEgRTgcnALYThfZ6+WdM0ezvtjPt+U7fT9kiL/YDrCYLnaX9Gq/bD83nX/H2/82l+n/4B2LH4+dNof7awtTW/TyHD3/1B++CDiIiIiIiI+NL0ES0REREREZHORhdaIiIiIiIinulCS0RERERExDNdaImIiIiIiHimCy0RERERERHPdKElIiIiIiLimS60REREREREPPt/9WMIMSdPbJkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x216 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_adv_samples = np.load(\"adv_samples_from_cos.npz\")\n",
    "x_original    = x_adv_samples[\"x_original\"]\n",
    "x_adversarial = x_adv_samples[\"x_adversarial\"]\n",
    "y             = x_adv_samples[\"y\"]\n",
    "\n",
    "x_orig = (x_original    * 255).astype('int')[:, :, :, 0]\n",
    "x_adv  = (x_adversarial * 255).astype('int')[:, :, :, 0]\n",
    "\n",
    "y_pred_orig = model.predict(x_original,    verbose=0)\n",
    "y_pred_adv  = model.predict(x_adversarial, verbose=0)\n",
    "\n",
    "fig    = plt.figure(figsize=(15, 3))\n",
    "cols   = 10\n",
    "rows   = 2\n",
    "images = list(x_orig[:cols])      + list(x_adv[:cols])\n",
    "preds  = list(y_pred_orig[:cols]) + list(y_pred_adv[:cols])\n",
    "labels = list(y[:cols])           + list(y[:cols])\n",
    "\n",
    "for i in range(0, len(images)):\n",
    "    ax = fig.add_subplot(rows, cols, i+1)\n",
    "    y_pred = np.argmax(preds[i])\n",
    "    y_orig = np.argmax(labels[i])\n",
    "    ax.set_xlabel(\"Predict: %s\" % y_pred,\n",
    "                  color = \"green\" if y_pred == y_orig else \"red\")\n",
    "    ax.tick_params(axis='both', which='both',\n",
    "                   bottom=False, top=False,\n",
    "                   right=False, left=False,\n",
    "                   labelbottom=False, labelleft=False)\n",
    "    plt.imshow(images[i], cmap='gray')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Summary and Next Steps <a id=\"summary\"></a>\n",
    "\n",
    "This notebook only looked at one adversarial robustness technique (FGM). The *ART* library contains many more attacks, metrics and defenses to help you understand and improve your model's robustness. You can use this notebook as a template to experiment with all aspects of *ART*. Find more state-of-the-art methods for attacking and defending classifiers here:\n",
    "\n",
    "https://github.com/IBM/adversarial-robustness-toolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements\n",
    "\n",
    "Special thanks to [Anupama-Murthi](https://github.ibm.com/Anupama-Murthi) and [Vijay Arya](https://github.ibm.com/vijay-arya) who created the original notebook which we modified here to showcase how to use *ART* with *FfDL*. If you would like to try *[Watson Machine Learning (WML) Service](https://console.bluemix.net/catalog/services/machine-learning)* with *ART* check out Anupama and Vijay's notebook here:\n",
    "\n",
    "[https://github.ibm.com/robust-dlaas/ART-in-WML/Use ART to check robustness of deep learning models.ipynb](https://github.ibm.com/robust-dlaas/ART-in-WML/blob/master/Use%20ART%20to%20check%20robustness%20of%20deep%20learning%20models.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2018 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}