t81_558_class_14_01_automl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* **Part 14.1: What is AutoML** [[Video]](https://www.youtube.com/watch?v=TFUysIR5AB0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_02_auto_encode.ipynb)\n",
    "* Part 14.3: Training an Intrusion Detection System with KDD99 [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* Part 14.5: The Deep Learning Technologies I am Excited About [[Video]]() [[Notebook]](t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.1: What is AutoML\n",
    "\n",
    "Automatic Machine Learning (AutoML) attempts to use machine learning to automate itself.  Data is passed to the AutoML application in raw form and models are automatically generated.\n",
    "\n",
    "### AutoML from your Local Computer\n",
    "\n",
    "The following AutoML applications are commercial.\n",
    "\n",
    "* [Rapid Miner](https://rapidminer.com/educational-program/) - Free student version available.\n",
    "* [Dataiku](https://www.dataiku.com/dss/editions/) - Free community version available.\n",
    "* [DataRobot](https://www.datarobot.com/) - Commercial\n",
    "* [H2O Driverless](https://www.h2o.ai/products/h2o-driverless-ai/) - Commercial\n",
    "\n",
    "### AutoML from Google Cloud\n",
    "\n",
    "* [Google Cloud AutoML Tutorial](https://cloud.google.com/vision/automl/docs/tutorial)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Simple AutoML System\n",
    "\n",
    "The following program is a very simple implementation of AutoML.  It is able to take RAW tabular data and construct a neural network.  \n",
    "\n",
    "We begin by defining a class that abstracts the differences between reading CSV over local file system or HTTP/HTTPS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import csv\n",
    "\n",
    "class CSVSource():\n",
    "    def __init__(self, filename):\n",
    "        self.filename = filename\n",
    "    def __enter__(self):\n",
    "        if self.filename.lower().startswith(\"https:\") \n",
    "            or self.filename.lower().startswith(\"https:\"):\n",
    "            r = requests.get(self.filename, stream=True)\n",
    "            self.infile = (line.decode('utf-8') for line in r.iter_lines())\n",
    "            return csv.reader(self.infile)\n",
    "        else:\n",
    "            self.infile = codecs.open(self.filename, \"r\", \"utf-8\")\n",
    "            return csv.reader(self.infile)\n",
    "    def __exit__(self, type, value, traceback):\n",
    "        self.infile.close()\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code analyzes the tabular data and determines a way of encoding the feature vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import codecs\n",
    "import math\n",
    "import os\n",
    "import re\n",
    "from numpy import genfromtxt\n",
    "\n",
    "MAX_UNIQUES = 200\n",
    "\n",
    "INPUT_ENCODING = 'latin-1'\n",
    "\n",
    "CMD_CAT_DUMMY = 'dummy-cat'\n",
    "CMD_CAT_NUMERIC = 'numeric-cat'\n",
    "CMD_IGNORE = 'ignore'\n",
    "CMD_MAP = 'map'\n",
    "CMD_PASS = 'pass'\n",
    "CMD_BITS = 'bits'\n",
    "\n",
    "CONTROL_INDEX = 'index'\n",
    "CONTROL_NAME = 'name'\n",
    "CONTROL_COMMAND = 'command'\n",
    "CONTROL_TYPE = 'type'\n",
    "CONTROL_LENGTH = 'length'\n",
    "CONTROL_UNIQUE_COUNT = 'unique_count'\n",
    "CONTROL_UNIQUE_LIST = 'unique_list'\n",
    "CONTROL_MISSING = 'missing'\n",
    "CONTROL_MEAN = 'mean'\n",
    "CONTROL_SDEV = 'sdev'\n",
    "\n",
    "\n",
    "MAP_SKIP = True\n",
    "MISSING_SKIP = False\n",
    "\n",
    "current_row = 0\n",
    "\n",
    "def is_number(s):\n",
    "    try:\n",
    "        float(s)\n",
    "        return True\n",
    "    except ValueError:\n",
    "        return False\n",
    "\n",
    "\n",
    "def isna(s):\n",
    "    return s.upper() == 'NA' or s.upper() == 'N/A' \n",
    "        or s.upper() == 'NULL' or len(s) < 1 or s.upper() == '?'\n",
    "\n",
    "def analyze(filename):\n",
    "    fields = []\n",
    "    first_header = None\n",
    "\n",
    "    # Pass 1 (very short. First, look at the first row of each of the \n",
    "    # provided files.\n",
    "    # Build field blocks from the first file, and ensure that other files\n",
    "    # match the first one.\n",
    "    \n",
    "    with CSVSource(filename) as reader:\n",
    "        header = next(reader)\n",
    "\n",
    "        if first_header is None:\n",
    "            first_header = header\n",
    "\n",
    "            for idx, field_name in enumerate(header):\n",
    "                fields.append({\n",
    "                    'name': field_name,\n",
    "                    'command': '?',\n",
    "                    'index': idx,\n",
    "                    'type': None,\n",
    "                    'missing': False,\n",
    "                    'unique': {},\n",
    "                    'count': 0,\n",
    "                    'mean': '',\n",
    "                    'sum': 0,\n",
    "                    'sdev': '',\n",
    "                    'length': 0})\n",
    "        else:\n",
    "            for x, y in zip(header, first_header):\n",
    "                if x != y:\n",
    "                    raise ValueError(\\\n",
    "                        'The headers do not match on the input files')\n",
    "\n",
    "\n",
    "    # Pass 2 over the files\n",
    "    with CSVSource(filename) as reader:\n",
    "        next(reader)\n",
    "\n",
    "        # Determine types and calculate sum\n",
    "        for row in reader:\n",
    "            if len(row) != len(fields):\n",
    "                continue\n",
    "            for data, field_info in zip(row, fields):\n",
    "                data = data.strip()\n",
    "                field_info['length'] = max(len(data),field_info['length'])\n",
    "                if len(data) < 1 or data.upper() == 'NULL' or isna(data):\n",
    "                    field_info[CONTROL_MISSING] = True\n",
    "                else:\n",
    "                    if not is_number(data):\n",
    "                        field_info['type'] = 'text'\n",
    "\n",
    "                    # Track the unique values and counts per unique item\n",
    "                    cat_map = field_info['unique']\n",
    "                    if data in cat_map:\n",
    "                        cat_map[data]['count']+=1\n",
    "                    else:\n",
    "                        cat_map[data] = {'name':data,'count':1}\n",
    "\n",
    "                    if field_info['type'] != 'text':\n",
    "                        field_info['count'] += 1\n",
    "                        field_info['sum'] += float(data)\n",
    "\n",
    "    # Finalize types\n",
    "    for field in fields:\n",
    "        if field['type'] is None:\n",
    "            field['type'] = 'numeric'\n",
    "        field[CONTROL_UNIQUE_COUNT] = len(field['unique'])\n",
    "\n",
    "    # Calculate mean\n",
    "    for field in fields:\n",
    "        if field['type'] == 'numeric' and field['count'] > 0:\n",
    "            field['mean'] = field['sum'] / field['count']\n",
    "\n",
    "\n",
    "    # Pass 3 over the files, calculate standard deviation and \n",
    "    # finailize fields.\n",
    "    sums = [0] * len(fields)\n",
    "    \n",
    "    with CSVSource(filename) as reader:\n",
    "        next(reader)\n",
    "\n",
    "        for row in reader:\n",
    "            if len(row) != len(fields):\n",
    "                continue\n",
    "            for data, field_info in zip(row, fields):\n",
    "                data = data.strip()\n",
    "                if field_info['type'] == 'numeric' \n",
    "                        and len(data) > 0 and not isna(data):\n",
    "                    sums[field_info['index']] += (float(data) - \\\n",
    "                                    field_info['mean']) ** 2\n",
    "\n",
    "    # Examine fields\n",
    "    for idx, field in enumerate(fields):\n",
    "        if field['type'] == 'numeric' and field['count'] > 0:\n",
    "            field['sdev'] = math.sqrt(sums[field['index']] / field['count'])\n",
    "\n",
    "        # Assign a default command\n",
    "        if field['name'] == 'ID' or field['name'] == 'FOLD':\n",
    "            field['command'] = 'pass'\n",
    "        elif \"DATE\" in field['name'].upper():\n",
    "            field['command'] = 'date'\n",
    "        elif field['unique_count'] == 2 and field['type'] == 'numeric':\n",
    "            field['command'] = CMD_PASS\n",
    "        elif field['type'] == 'numeric' and field['unique_count'] < 25:\n",
    "            field['command'] = CMD_CAT_DUMMY\n",
    "        elif field['type'] == 'numeric':\n",
    "            field['command'] = 'zscore'\n",
    "        elif field['type'] == 'text' and field['unique_count'] \\\n",
    "                    <= MAX_UNIQUES:\n",
    "            field['command'] = CMD_CAT_DUMMY\n",
    "        else:\n",
    "            field['command'] = CMD_IGNORE\n",
    "\n",
    "    return fields\n",
    "\n",
    "def write_control_file(filename, fields):\n",
    "    with codecs.open(filename, \"w\", \"utf-8\") as outfile:\n",
    "        writer = csv.writer(outfile,quoting=csv.QUOTE_NONNUMERIC)\n",
    "\n",
    "        writer.writerow([CONTROL_INDEX, CONTROL_NAME, CONTROL_COMMAND, \n",
    "                         CONTROL_TYPE, CONTROL_LENGTH, CONTROL_UNIQUE_COUNT, \n",
    "                         CONTROL_MISSING, CONTROL_MEAN, CONTROL_SDEV])\n",
    "        for field in fields:\n",
    "\n",
    "            # Write the main row for the field (left-justified)\n",
    "            writer.writerow([field[CONTROL_INDEX], field[CONTROL_NAME], \n",
    "                            field[CONTROL_COMMAND], field[CONTROL_TYPE], \n",
    "                             field[CONTROL_LENGTH], \n",
    "                                 field[CONTROL_UNIQUE_COUNT],\n",
    "                             field[CONTROL_MISSING], field[CONTROL_MEAN], \n",
    "                             field[CONTROL_SDEV]])\n",
    "\n",
    "            # Write out any needed category information\n",
    "            if field[CONTROL_UNIQUE_COUNT] <= MAX_UNIQUES:\n",
    "                sorted_cat = field['unique'].values()\n",
    "                sorted_cat = sorted(sorted_cat, key=lambda k: \n",
    "                                    k[CONTROL_NAME])\n",
    "                for category in sorted_cat:\n",
    "                    writer.writerow([\"\",\"\",\n",
    "                        category[CONTROL_NAME],\n",
    "                            category['count']])\n",
    "            else:\n",
    "                catagories = \"\"\n",
    "\n",
    "\n",
    "\n",
    "def read_control_file(filename):\n",
    "    with codecs.open(filename, \"r\", \"utf-8\") as infile:\n",
    "        reader = csv.reader(infile)\n",
    "        header = next(reader)\n",
    "\n",
    "        lookup = {}\n",
    "        for i, name in enumerate(header):\n",
    "            lookup[name] = i\n",
    "\n",
    "        fields = []\n",
    "        categories = {}\n",
    "\n",
    "        for row in reader:\n",
    "            if row[0] == '':\n",
    "                name = row[2]\n",
    "                mp = '' if len(row)<=4 else row[4]\n",
    "                categories[name] = {'name':name,'count':int(row[3]),\n",
    "                                    'map':mp}\n",
    "                if len(categories)>0:\n",
    "                    field[CONTROL_UNIQUE_LIST] = \\\n",
    "                        sorted(categories.keys())\n",
    "            else:\n",
    "                # New field\n",
    "                field = {}\n",
    "                categories = {}\n",
    "                field['unique'] = categories\n",
    "                for key in lookup.keys():\n",
    "                    value = row[lookup[key]]\n",
    "                    if key in ['unique_count', 'count', 'index', 'length']:\n",
    "                        value = int(value)\n",
    "                    elif key in ['sdev', 'mean', 'sum']:\n",
    "                        if len(value) > 0:\n",
    "                            value = float(value)\n",
    "                    field[key] = value\n",
    "\n",
    "                field['len'] = -1\n",
    "                fields.append(field)\n",
    "        return fields\n",
    "\n",
    "def header_cat_dummy(field, header):\n",
    "    name = str(field['name'])\n",
    "\n",
    "    for c in field['unique']:\n",
    "        dname = \"{}-D:{}\".format(name, c)\n",
    "        header.append(dname)\n",
    "\n",
    "def header_bits(field, header):\n",
    "    for i in range(field['length']):\n",
    "        header.append(\"{}-B:{}\".format(field['name'], i))\n",
    "\n",
    "\n",
    "def header_other(field, header):\n",
    "    header.append(field['name'])\n",
    "\n",
    "\n",
    "def column_zscore(field,write_row,value,has_na):\n",
    "    if isna(value) or field['sdev'] == 0:\n",
    "        #write_row.append('NA')\n",
    "        #has_na = True\n",
    "        write_row.append(0)\n",
    "    elif not is_number(value):\n",
    "        raise ValueError(\"Row {}: Non-numeric for zscore: {}\"\\\n",
    "            \" on field {}\".format(current_row,value,field['name']))\n",
    "    else:\n",
    "        value = (float(value) - field['mean']) / field['sdev']\n",
    "        write_row.append(value)\n",
    "    return has_na\n",
    "\n",
    "def column_cat_numeric(field,write_row,value,has_na):\n",
    "    if CONTROL_UNIQUE_LIST not in field:\n",
    "        raise ValueError(\"No value list, can't encode {}\"\\\n",
    "            \" to numeric categorical.\".format(field[CONTROL_NAME]))\n",
    "\n",
    "    if value not in field[CONTROL_UNIQUE_LIST]:\n",
    "        write_row.append(\"NA\")\n",
    "        has_na = True\n",
    "    else:\n",
    "        idx = field[CONTROL_UNIQUE_LIST].index(value)\n",
    "        write_row.append('class-' + str(idx))\n",
    "    return has_na\n",
    "\n",
    "def column_map(field,write_row,value,has_na):\n",
    "    if value in field['unique']:\n",
    "        mapping = field['unique'][value]['map']\n",
    "        write_row.append(mapping)\n",
    "    else:\n",
    "        write_row.append(\"NA\")\n",
    "        return True\n",
    "    return has_na\n",
    "\n",
    "\n",
    "def column_cat_dummy(field,write_row,value,has_na):\n",
    "    for c in field['unique']:\n",
    "        write_row.append(0 if value != c else 1)\n",
    "    return has_na\n",
    "\n",
    "def column_bits(field,write_row,value,has_na):\n",
    "    if len(value)!=field['length']:\n",
    "        raise ValueError(\"Invalid bits length: {}, expected: {}\".format(\n",
    "            len(value),field['length']))\n",
    "\n",
    "    for c in value:\n",
    "        if c == 'Y':\n",
    "            write_row.append(1)\n",
    "        elif c == 'N':\n",
    "            write_row.append(-1)\n",
    "        else:\n",
    "            write_row.append(0)\n",
    "    return has_na\n",
    "\n",
    "def transform_file(input_file, output_file, fields):\n",
    "    print(\"**Transforming to file: {}\".format(output_file))\n",
    "    with CSVSource(input_file) as reader, \\\n",
    "            codecs.open(output_file, \"w\", \"utf-8\") as outfile:\n",
    "        writer = csv.writer(outfile)\n",
    "\n",
    "        next(reader)\n",
    "        header = []\n",
    "\n",
    "        # Write the header\n",
    "        for field in fields:\n",
    "            if field['command'] == CMD_IGNORE:\n",
    "                pass\n",
    "            elif field['command'] == CMD_CAT_DUMMY:\n",
    "                header_cat_dummy(field,header)\n",
    "            elif field['command'] == CMD_BITS:\n",
    "                header_bits(field,header)\n",
    "            else:\n",
    "                header_other(field,header)\n",
    "\n",
    "        print(\"Columns generated: {}\".format(len(header)))\n",
    "\n",
    "        writer.writerow(header)\n",
    "        line_count = 0\n",
    "        lines_skipped = 0\n",
    "\n",
    "        # Process the actual file\n",
    "        current_row = -1\n",
    "        header_len = len(header)\n",
    "        for row in reader:\n",
    "            if len(row) != len(fields):\n",
    "                continue\n",
    "                \n",
    "            current_row+=1\n",
    "            has_na = False\n",
    "            write_row = []\n",
    "            for field in fields:\n",
    "                value = row[field['index']].strip()\n",
    "\n",
    "                cmd = field['command']\n",
    "                if cmd == 'zscore':\n",
    "                    has_na = column_zscore(field,write_row,value, has_na)\n",
    "                elif cmd == CMD_CAT_NUMERIC:\n",
    "                    has_na = column_cat_numeric(field,write_row,value, \\\n",
    "                                                has_na)\n",
    "                elif cmd == CMD_IGNORE:\n",
    "                    pass\n",
    "                elif cmd == CMD_MAP:\n",
    "                    has_na = column_map(field,write_row,value, has_na)\n",
    "                elif cmd == CMD_PASS:\n",
    "                    write_row.append(value)\n",
    "                elif cmd == 'date':\n",
    "                    write_row.append(str(value[-4:]))\n",
    "                elif cmd == CMD_CAT_DUMMY:\n",
    "                    has_na = column_cat_dummy(field,write_row,value, \n",
    "                                              has_na)\n",
    "                elif cmd == CMD_BITS:\n",
    "                    has_na = column_bits(field,write_row,value,has_na)\n",
    "                else:\n",
    "                    raise ValueError(\\\n",
    "                        \"Unknown command: {}, stopping.\".format(cmd))\n",
    "\n",
    "\n",
    "            if MISSING_SKIP and has_na:\n",
    "                lines_skipped += 1\n",
    "                pass\n",
    "            else:\n",
    "                line_count += 1\n",
    "                writer.writerow(write_row)\n",
    "\n",
    "                # Double check!\n",
    "                if len(write_row) != header_len:\n",
    "                    raise ValueError(\"Inconsistant column \"\\\n",
    "                        \"count near line: {}, only had: {}\" \\\n",
    "                        .format(line_count,len(write_row)))\n",
    "\n",
    "    print(\"Data rows written: {}, skipped: {}\"\\\n",
    "          .format(line_count,lines_skipped))\n",
    "    print()\n",
    "\n",
    "def find_field(control, name):\n",
    "    for field in control:\n",
    "        if field['name'] == name:\n",
    "            return field\n",
    "    return None\n",
    "\n",
    "def find_transformed_fields(header, name):\n",
    "    y = []\n",
    "    x = []\n",
    "    for idx, field in enumerate(header):\n",
    "        if field.startswith(name + '-') or field==name:\n",
    "            y.append(idx)\n",
    "        else:\n",
    "            x.append(idx)\n",
    "            \n",
    "    return x,y\n",
    "\n",
    "def process_for_fit(control, transformed_file, target):\n",
    "    \n",
    "    with CSVSource(transformed_file) as reader:\n",
    "        header = next(reader)\n",
    "    \n",
    "    field = find_field(control, target)\n",
    "    if field is None:\n",
    "        raise ValueError(f\"Unknown target column specified:{target}\")\n",
    "\n",
    "    if field['command'] == 'dummy-cat':\n",
    "        print(f\"Performing classification on: {target}\")\n",
    "    else:\n",
    "        print(f\"Performing regression on: {target}\")\n",
    "        \n",
    "    x_ids, y_ids = find_transformed_fields(header, target)\n",
    "    \n",
    "    x = genfromtxt(\"transformed.csv\", delimiter=',', skip_header=1)\n",
    "    y = x[:,y_ids]\n",
    "    x = x[:,x_ids]\n",
    "    return x,y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code takes the data processed from above and trains a neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "def generate_network(x,y,task):\n",
    "    model = Sequential()\n",
    "    model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "    model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "    \n",
    "    if task == 'classify':\n",
    "        model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "        model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "    else:\n",
    "        model.add(Dense(1))\n",
    "        model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "        \n",
    "    return model\n",
    "\n",
    "def cross_validate(x,y,folds,task):\n",
    "    \n",
    "    if task == 'classify':\n",
    "        cats = y.argmax(axis=1)\n",
    "        kf = StratifiedKFold(folds, shuffle=True, random_state=42).split(\\\n",
    "                x,cats)\n",
    "    else:\n",
    "        kf = KFold(folds, shuffle=True, random_state=42).split(x) \n",
    "    \n",
    "    oos_y = []\n",
    "    oos_pred = []\n",
    "    fold = 0\n",
    "    \n",
    "    for train, test in kf:\n",
    "        fold+=1\n",
    "        print(f\"Fold #{fold}\")\n",
    "\n",
    "        x_train = x[train]\n",
    "        y_train = y[train]\n",
    "        x_test = x[test]\n",
    "        y_test = y[test]\n",
    "\n",
    "        model = generate_network(x,y,task)\n",
    "        model.fit(x_train,y_train,validation_data=(x_test,y_test),verbose=0,\n",
    "                  epochs=500)\n",
    "\n",
    "        pred = model.predict(x_test)\n",
    "\n",
    "        oos_y.append(y_test)\n",
    "        \n",
    "        if task == 'classify':\n",
    "            # raw probabilities to chosen class (highest probability)\n",
    "            pred = np.argmax(pred,axis=1) \n",
    "        oos_pred.append(pred)  \n",
    "\n",
    "        if task == 'classify':\n",
    "            # Measure this fold's accuracy\n",
    "            y_compare = np.argmax(y_test,axis=1) # For accuracy calculation\n",
    "            score = metrics.accuracy_score(y_compare, pred)\n",
    "            print(f\"Fold score (accuracy): {score}\")\n",
    "        else:\n",
    "            score = np.sqrt(metrics.mean_squared_error(pred,y_test))\n",
    "            print(f\"Fold score (RMSE): {score}\")\n",
    "            \n",
    "        \n",
    "    # Build the oos prediction list and calculate the error.\n",
    "    oos_y = np.concatenate(oos_y)\n",
    "    oos_pred = np.concatenate(oos_pred)\n",
    "    \n",
    "    if task == 'classify':\n",
    "        oos_y_compare = np.argmax(oos_y,axis=1) # For accuracy calculation\n",
    "        score = metrics.accuracy_score(oos_y_compare, oos_pred)\n",
    "        print(f\"Final score (accuracy): {score}\") \n",
    "    else:\n",
    "        score = np.sqrt(metrics.mean_squared_error(oos_y, oos_pred))\n",
    "        print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running My Sample AutoML Program\n",
    "\n",
    "These three variables are all you really need to define."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOURCE_DATA = \\\n",
    "'https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv'\n",
    "TARGET_FIELD = 'product'\n",
    "TASK = 'classify'\n",
    "\n",
    "#SOURCE_DATA = 'https://data.heatonresearch.com/data/t81-558/iris.csv'\n",
    "#TARGET_FIELD = 'species'\n",
    "#TASK = 'classify'\n",
    "\n",
    "#SOURCE_DATA = 'https://data.heatonresearch.com/data/t81-558/auto-mpg.csv'\n",
    "#TARGET_FIELD = 'mpg'\n",
    "#TASK = 'reg'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following lines of code analyze your source data file and figure out how to encode each column.  The result is a control file that you can modify to control how each column is handled.  The below code should only be run ONCE to generate a control file as a starting point for you to modify."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import requests\n",
    "import codecs\n",
    "\n",
    "control = analyze(SOURCE_DATA)\n",
    "write_control_file(\"control.csv\",control)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your control file is already create, you can start here (after defining the above constants).  Do not rerun the previous section, as it will overwrite your control file.  Now transform the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Transforming to file: transformed.csv\n",
      "Columns generated: 59\n",
      "Data rows written: 2000, skipped: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "control = read_control_file(\"control.csv\")\n",
    "transform_file(SOURCE_DATA,\"transformed.csv\",control)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the transformed data into properly preprocessed $x$ and $y$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing classification on: product\n",
      "(2000, 52)\n",
      "(2000, 7)\n"
     ]
    }
   ],
   "source": [
    "x,y = process_for_fit(control, \"transformed.csv\", TARGET_FIELD)\n",
    "print(x.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double check to be sure there are no missing values remaining."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.isnan(x).any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to cross validate and train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0810 13:28:42.553956 140735657337728 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow_core/python/ops/math_grad.py:1366: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0810 13:28:42.720788 140735657337728 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py:468: BaseResourceVariable.constraint (from tensorflow.python.ops.resource_variable_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Apply a constraint manually following the optimizer update step.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold score (accuracy): 0.6915422885572139\n",
      "Fold #2\n",
      "Fold score (accuracy): 0.7064676616915423\n",
      "Fold #3\n",
      "Fold score (accuracy): 0.6807980049875312\n",
      "Fold #4\n",
      "Fold score (accuracy): 0.6658291457286433\n",
      "Fold #5\n",
      "Fold score (accuracy): 0.6675062972292192\n",
      "Final score (accuracy): 0.6825\n"
     ]
    }
   ],
   "source": [
    "cross_validate(x,y,5,TASK)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}