卷积网络/GoogLeNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GoogLeNet\n",
    "\n",
    "GoogLeNet是google研究院在ImageNet大规模视觉识别挑战赛2014（ILSVRC14）上提出了一种代号为Inception的深度卷积神经网络结构的一个特例，最终在比赛中Top-5 error为6.67%,从而获得当年比赛冠军。GoogLeNet设计基于以下动机：提升网络性能最直接的办法是增加网络深度和宽度，增加深度和宽度会产生巨大的参数，一般的解决方法是将全连接或者一般卷积转化为稀疏连接，利用密集矩阵的计算性能来计算稀疏矩阵，从而达到计算性能的提升。\n",
    "\n",
    "## Inception架构\n",
    "\n",
    "Inception架构的主要架构如图所示：\n",
    "![](img/inception.PNG)\n",
    "> 图片来源：[Going deeper with convolutions](https://arxiv.org/pdf/1409.4842v1.pdf)\n",
    "\n",
    "上图（a）是一个简单版本，说明如下：\n",
    "- 1、采用不同大小的卷积核表示不同的感受野，filter concatenation表示不同尺度特征的融合;\n",
    "- 2、卷积核采大小用1/3/5是为了对齐，设定步长stride=1，pad=0/1/2,卷积之后就可以得到相同维度的特征;\n",
    "- 3、除了卷积核之外加入的max pooling是被多种实现证实有效加入的;\n",
    "- 4、网络越到后面特征抽象程度越高，特征所涉及的感受野变大，卷积大小3/5的卷积比例需要增加。\n",
    "\n",
    "考虑到5×5卷积核带来的巨大计算量，inception架构采用1×1卷积核来进行降维，得到的新架构如上图（b）所示。\n",
    "\n",
    "\n",
    "## GoogLeNet架构\n",
    "\n",
    "Googlenet整体架构如下图所示：\n",
    "![](img/googlenet.jpg)\n",
    "> 图片来源：[Going deeper with convolutions](https://arxiv.org/pdf/1409.4842v1.pdf)\n",
    "\n",
    "说明如下：\n",
    "- 1、GoogLeNet采用模块化设计，方便修改;\n",
    "- 2、为了应对梯度消失问题，网络额外增加了两个辅助的softmax分类器用来向前传递梯度，这两个辅助分类器只用于训练，测试时需要去掉;\n",
    "- 3、实验表明网络末尾采用average pooling替代全连接层可以提升top-1精度，网络内部依然采用了dropout。\n",
    "- 4、所有的卷积都采用了线性修正激活。\n",
    "\n",
    "\n",
    "## 代码实现\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from __future__ import division, print_function, absolute_import\n",
    "\n",
    "import os, tflearn, random, pickle, gzip\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from datetime import datetime\n",
    "from collections import namedtuple\n",
    "\n",
    "from tflearn.layers.core import input_data, dropout, fully_connected\n",
    "from tflearn.layers.conv import conv_2d, max_pool_2d, avg_pool_2d\n",
    "from tflearn.layers.normalization import local_response_normalization\n",
    "from tflearn.layers.merge_ops import merge\n",
    "from tflearn.layers.estimator import regression\n",
    "\n",
    "_EPSILON = 1e-8\n",
    "\n",
    "# import tflearn.datasets.oxflower17 as oxflower17\n",
    "# X, Y = oxflower17.load_data(one_hot=True, resize_pics=(227, 227))\n",
    "\n",
    "HParams = namedtuple('HParams',\n",
    "                     'img_size, label_size, gpu_memory_fraction, model_name, gpu_usage')\n",
    "\n",
    "\n",
    "class GoogLeNet(object):\n",
    "\n",
    "    def __init__(self, hps):\n",
    "        # ctrl\n",
    "        self.img_size = hps.img_size\n",
    "        self.label_size = hps.label_size\n",
    "        self.model_name = hps.model_name\n",
    "        self.gpu_usage = hps.gpu_usage\n",
    "\n",
    "        # net\n",
    "        tflearn.config.init_graph(gpu_memory_fraction=self.gpu_usage)\n",
    "        network = input_data(shape=[None, self.img_size, self.img_size, 3])\n",
    "        conv1_7_7 = conv_2d(network, 64, 7, strides=2, activation='relu', name = 'conv1_7_7_s2')\n",
    "        pool1_3_3 = max_pool_2d(conv1_7_7, 3,strides=2)\n",
    "        pool1_3_3 = local_response_normalization(pool1_3_3)\n",
    "        conv2_3_3_reduce = conv_2d(pool1_3_3, 64,1, activation='relu',name = 'conv2_3_3_reduce')\n",
    "        conv2_3_3 = conv_2d(conv2_3_3_reduce, 192,3, activation='relu', name='conv2_3_3')\n",
    "        conv2_3_3 = local_response_normalization(conv2_3_3)\n",
    "        pool2_3_3 = max_pool_2d(conv2_3_3, kernel_size=3, strides=2, name='pool2_3_3_s2')\n",
    "        inception_3a_1_1 = conv_2d(pool2_3_3, 64, 1, activation='relu', name='inception_3a_1_1')\n",
    "        inception_3a_3_3_reduce = conv_2d(pool2_3_3, 96,1, activation='relu', name='inception_3a_3_3_reduce')\n",
    "        inception_3a_3_3 = conv_2d(inception_3a_3_3_reduce, 128,filter_size=3,  activation='relu', name = 'inception_3a_3_3')\n",
    "        inception_3a_5_5_reduce = conv_2d(pool2_3_3,16, filter_size=1,activation='relu', name ='inception_3a_5_5_reduce' )\n",
    "        inception_3a_5_5 = conv_2d(inception_3a_5_5_reduce, 32, filter_size=5, activation='relu', name= 'inception_3a_5_5')\n",
    "        inception_3a_pool = max_pool_2d(pool2_3_3, kernel_size=3, strides=1, )\n",
    "        inception_3a_pool_1_1 = conv_2d(inception_3a_pool, 32, filter_size=1, activation='relu', name='inception_3a_pool_1_1')\n",
    "\n",
    "        # merge the inception_3a__\n",
    "        inception_3a_output = merge([inception_3a_1_1, inception_3a_3_3, inception_3a_5_5, inception_3a_pool_1_1], mode='concat', axis=3)\n",
    "\n",
    "        inception_3b_1_1 = conv_2d(inception_3a_output, 128,filter_size=1,activation='relu', name= 'inception_3b_1_1' )\n",
    "        inception_3b_3_3_reduce = conv_2d(inception_3a_output, 128, filter_size=1, activation='relu', name='inception_3b_3_3_reduce')\n",
    "        inception_3b_3_3 = conv_2d(inception_3b_3_3_reduce, 192, filter_size=3,  activation='relu',name='inception_3b_3_3')\n",
    "        inception_3b_5_5_reduce = conv_2d(inception_3a_output, 32, filter_size=1, activation='relu', name = 'inception_3b_5_5_reduce')\n",
    "        inception_3b_5_5 = conv_2d(inception_3b_5_5_reduce, 96, filter_size=5,  name = 'inception_3b_5_5')\n",
    "        inception_3b_pool = max_pool_2d(inception_3a_output, kernel_size=3, strides=1,  name='inception_3b_pool')\n",
    "        inception_3b_pool_1_1 = conv_2d(inception_3b_pool, 64, filter_size=1,activation='relu', name='inception_3b_pool_1_1')\n",
    "\n",
    "        #merge the inception_3b_*\n",
    "        inception_3b_output = merge([inception_3b_1_1, inception_3b_3_3, inception_3b_5_5, inception_3b_pool_1_1], mode='concat',axis=3,name='inception_3b_output')\n",
    "\n",
    "        pool3_3_3 = max_pool_2d(inception_3b_output, kernel_size=3, strides=2, name='pool3_3_3')\n",
    "        inception_4a_1_1 = conv_2d(pool3_3_3, 192, filter_size=1, activation='relu', name='inception_4a_1_1')\n",
    "        inception_4a_3_3_reduce = conv_2d(pool3_3_3, 96, filter_size=1, activation='relu', name='inception_4a_3_3_reduce')\n",
    "        inception_4a_3_3 = conv_2d(inception_4a_3_3_reduce, 208, filter_size=3,  activation='relu', name='inception_4a_3_3')\n",
    "        inception_4a_5_5_reduce = conv_2d(pool3_3_3, 16, filter_size=1, activation='relu', name='inception_4a_5_5_reduce')\n",
    "        inception_4a_5_5 = conv_2d(inception_4a_5_5_reduce, 48, filter_size=5,  activation='relu', name='inception_4a_5_5')\n",
    "        inception_4a_pool = max_pool_2d(pool3_3_3, kernel_size=3, strides=1,  name='inception_4a_pool')\n",
    "        inception_4a_pool_1_1 = conv_2d(inception_4a_pool, 64, filter_size=1, activation='relu', name='inception_4a_pool_1_1')\n",
    "\n",
    "        inception_4a_output = merge([inception_4a_1_1, inception_4a_3_3, inception_4a_5_5, inception_4a_pool_1_1], mode='concat', axis=3, name='inception_4a_output')\n",
    "\n",
    "\n",
    "        inception_4b_1_1 = conv_2d(inception_4a_output, 160, filter_size=1, activation='relu', name='inception_4a_1_1')\n",
    "        inception_4b_3_3_reduce = conv_2d(inception_4a_output, 112, filter_size=1, activation='relu', name='inception_4b_3_3_reduce')\n",
    "        inception_4b_3_3 = conv_2d(inception_4b_3_3_reduce, 224, filter_size=3, activation='relu', name='inception_4b_3_3')\n",
    "        inception_4b_5_5_reduce = conv_2d(inception_4a_output, 24, filter_size=1, activation='relu', name='inception_4b_5_5_reduce')\n",
    "        inception_4b_5_5 = conv_2d(inception_4b_5_5_reduce, 64, filter_size=5,  activation='relu', name='inception_4b_5_5')\n",
    "\n",
    "        inception_4b_pool = max_pool_2d(inception_4a_output, kernel_size=3, strides=1,  name='inception_4b_pool')\n",
    "        inception_4b_pool_1_1 = conv_2d(inception_4b_pool, 64, filter_size=1, activation='relu', name='inception_4b_pool_1_1')\n",
    "\n",
    "        inception_4b_output = merge([inception_4b_1_1, inception_4b_3_3, inception_4b_5_5, inception_4b_pool_1_1], mode='concat', axis=3, name='inception_4b_output')\n",
    "\n",
    "\n",
    "        inception_4c_1_1 = conv_2d(inception_4b_output, 128, filter_size=1, activation='relu',name='inception_4c_1_1')\n",
    "        inception_4c_3_3_reduce = conv_2d(inception_4b_output, 128, filter_size=1, activation='relu', name='inception_4c_3_3_reduce')\n",
    "        inception_4c_3_3 = conv_2d(inception_4c_3_3_reduce, 256,  filter_size=3, activation='relu', name='inception_4c_3_3')\n",
    "        inception_4c_5_5_reduce = conv_2d(inception_4b_output, 24, filter_size=1, activation='relu', name='inception_4c_5_5_reduce')\n",
    "        inception_4c_5_5 = conv_2d(inception_4c_5_5_reduce, 64,  filter_size=5, activation='relu', name='inception_4c_5_5')\n",
    "\n",
    "        inception_4c_pool = max_pool_2d(inception_4b_output, kernel_size=3, strides=1)\n",
    "        inception_4c_pool_1_1 = conv_2d(inception_4c_pool, 64, filter_size=1, activation='relu', name='inception_4c_pool_1_1')\n",
    "\n",
    "        inception_4c_output = merge([inception_4c_1_1, inception_4c_3_3, inception_4c_5_5, inception_4c_pool_1_1], mode='concat', axis=3,name='inception_4c_output')\n",
    "\n",
    "        inception_4d_1_1 = conv_2d(inception_4c_output, 112, filter_size=1, activation='relu', name='inception_4d_1_1')\n",
    "        inception_4d_3_3_reduce = conv_2d(inception_4c_output, 144, filter_size=1, activation='relu', name='inception_4d_3_3_reduce')\n",
    "        inception_4d_3_3 = conv_2d(inception_4d_3_3_reduce, 288, filter_size=3, activation='relu', name='inception_4d_3_3')\n",
    "        inception_4d_5_5_reduce = conv_2d(inception_4c_output, 32, filter_size=1, activation='relu', name='inception_4d_5_5_reduce')\n",
    "        inception_4d_5_5 = conv_2d(inception_4d_5_5_reduce, 64, filter_size=5,  activation='relu', name='inception_4d_5_5')\n",
    "        inception_4d_pool = max_pool_2d(inception_4c_output, kernel_size=3, strides=1,  name='inception_4d_pool')\n",
    "        inception_4d_pool_1_1 = conv_2d(inception_4d_pool, 64, filter_size=1, activation='relu', name='inception_4d_pool_1_1')\n",
    "\n",
    "        inception_4d_output = merge([inception_4d_1_1, inception_4d_3_3, inception_4d_5_5, inception_4d_pool_1_1], mode='concat', axis=3, name='inception_4d_output')\n",
    "\n",
    "        inception_4e_1_1 = conv_2d(inception_4d_output, 256, filter_size=1, activation='relu', name='inception_4e_1_1')\n",
    "        inception_4e_3_3_reduce = conv_2d(inception_4d_output, 160, filter_size=1, activation='relu', name='inception_4e_3_3_reduce')\n",
    "        inception_4e_3_3 = conv_2d(inception_4e_3_3_reduce, 320, filter_size=3, activation='relu', name='inception_4e_3_3')\n",
    "        inception_4e_5_5_reduce = conv_2d(inception_4d_output, 32, filter_size=1, activation='relu', name='inception_4e_5_5_reduce')\n",
    "        inception_4e_5_5 = conv_2d(inception_4e_5_5_reduce, 128,  filter_size=5, activation='relu', name='inception_4e_5_5')\n",
    "        inception_4e_pool = max_pool_2d(inception_4d_output, kernel_size=3, strides=1,  name='inception_4e_pool')\n",
    "        inception_4e_pool_1_1 = conv_2d(inception_4e_pool, 128, filter_size=1, activation='relu', name='inception_4e_pool_1_1')\n",
    "\n",
    "\n",
    "        inception_4e_output = merge([inception_4e_1_1, inception_4e_3_3, inception_4e_5_5,inception_4e_pool_1_1],axis=3, mode='concat')\n",
    "\n",
    "        pool4_3_3 = max_pool_2d(inception_4e_output, kernel_size=3, strides=2, name='pool_3_3')\n",
    "\n",
    "\n",
    "        inception_5a_1_1 = conv_2d(pool4_3_3, 256, filter_size=1, activation='relu', name='inception_5a_1_1')\n",
    "        inception_5a_3_3_reduce = conv_2d(pool4_3_3, 160, filter_size=1, activation='relu', name='inception_5a_3_3_reduce')\n",
    "        inception_5a_3_3 = conv_2d(inception_5a_3_3_reduce, 320, filter_size=3, activation='relu', name='inception_5a_3_3')\n",
    "        inception_5a_5_5_reduce = conv_2d(pool4_3_3, 32, filter_size=1, activation='relu', name='inception_5a_5_5_reduce')\n",
    "        inception_5a_5_5 = conv_2d(inception_5a_5_5_reduce, 128, filter_size=5,  activation='relu', name='inception_5a_5_5')\n",
    "        inception_5a_pool = max_pool_2d(pool4_3_3, kernel_size=3, strides=1,  name='inception_5a_pool')\n",
    "        inception_5a_pool_1_1 = conv_2d(inception_5a_pool, 128, filter_size=1,activation='relu', name='inception_5a_pool_1_1')\n",
    "\n",
    "        inception_5a_output = merge([inception_5a_1_1, inception_5a_3_3, inception_5a_5_5, inception_5a_pool_1_1], axis=3,mode='concat')\n",
    "\n",
    "\n",
    "        inception_5b_1_1 = conv_2d(inception_5a_output, 384, filter_size=1,activation='relu', name='inception_5b_1_1')\n",
    "        inception_5b_3_3_reduce = conv_2d(inception_5a_output, 192, filter_size=1, activation='relu', name='inception_5b_3_3_reduce')\n",
    "        inception_5b_3_3 = conv_2d(inception_5b_3_3_reduce, 384,  filter_size=3,activation='relu', name='inception_5b_3_3')\n",
    "        inception_5b_5_5_reduce = conv_2d(inception_5a_output, 48, filter_size=1, activation='relu', name='inception_5b_5_5_reduce')\n",
    "        inception_5b_5_5 = conv_2d(inception_5b_5_5_reduce,128, filter_size=5,  activation='relu', name='inception_5b_5_5' )\n",
    "        inception_5b_pool = max_pool_2d(inception_5a_output, kernel_size=3, strides=1,  name='inception_5b_pool')\n",
    "        inception_5b_pool_1_1 = conv_2d(inception_5b_pool, 128, filter_size=1, activation='relu', name='inception_5b_pool_1_1')\n",
    "        inception_5b_output = merge([inception_5b_1_1, inception_5b_3_3, inception_5b_5_5, inception_5b_pool_1_1], axis=3, mode='concat')\n",
    "\n",
    "        pool5_7_7 = avg_pool_2d(inception_5b_output, kernel_size=7, strides=1)\n",
    "        pool5_7_7 = dropout(pool5_7_7, 0.4)\n",
    "\n",
    "        # output stage\n",
    "        loss = fully_connected(pool5_7_7, self.label_size ,activation='softmax')\n",
    "        network = regression(loss, optimizer='momentum',\n",
    "                             loss='categorical_crossentropy',\n",
    "                             learning_rate=0.001)\n",
    "\n",
    "        model_path = 'models/%s' % self.model_name\n",
    "        if not os.path.exists(model_path): os.makedirs(model_path)\n",
    "        self.model = tflearn.DNN(network, checkpoint_path=model_path+'/model',\n",
    "                        max_checkpoints=3, tensorboard_verbose=2)\n",
    "        # load existing model checkpoint\n",
    "        ckpt = self.get_checkpoint(model_path)\n",
    "        if ckpt:    \n",
    "            self.model.load(ckpt)\n",
    "            print(\"load existing checkpoint from %s\" % ckpt)\n",
    "\n",
    "\n",
    "\n",
    "    def get_checkpoint(self, model_path):\n",
    "        ckpt_path = '%s/checkpoint' % model_path\n",
    "        if not os.path.exists(ckpt_path): return\n",
    "        with open(ckpt_path, 'r') as f:\n",
    "            lines = [line.split(':')[1].replace('\\\"', '').strip() for line in f.readlines()]\n",
    "            lines = lines[1:][::-1]\n",
    "        for line in lines:\n",
    "            path = \"%s/%s\" % (model_path, line)\n",
    "            if os.path.exists(path):\n",
    "                return path\n",
    "\n",
    "\n",
    "\n",
    "    def fit(self, X, Y, n_epoch=1000):\n",
    "        print(\"fit data dim: X=%s, Y=%s\" % (np.shape(X), np.shape(Y)))\n",
    "        self.model.fit(X, Y, n_epoch=n_epoch, validation_set=0.1, shuffle=True,\n",
    "                  show_metric=True, batch_size=32, snapshot_step=200,\n",
    "                  snapshot_epoch=True, run_id=self.model_name)\n",
    "        \n",
    "\n",
    "\n",
    "    def predict(self, X):\n",
    "        return self.model.predict(X)\n",
    "\n",
    "\n",
    "\n",
    "    def get_data(self, dirname='17flowers', resize_pics=(227, 227), down_sampling=None):\n",
    "        pkl_fnames = [\"images/%s/%s\" % (dirname, f) for f in os.listdir(\"images/%s/\" % dirname) if \"samples_\" in f]\n",
    "        if not pkl_fnames:\n",
    "            pkl_fnames = image_dirs_to_samples(dirname, self.label_size,\n",
    "                resize=resize_pics, convert_gray=False, filetypes=['.jpg', '.jpeg'], \n",
    "                down_sampling=down_sampling, categorical_Y=True, shuffle_data=True)\n",
    "\n",
    "        # print(\"[get_data]:\", pkl_fnames)\n",
    "        return pkl_fnames\n",
    "\n",
    "\n",
    "\n",
    "    def to_categorical(y, nb_classes):\n",
    "        \"\"\" to_categorical.\n",
    "\n",
    "        Convert class vector (integers from 0 to nb_classes)\n",
    "        to binary class matrix, for use with categorical_crossentropy.\n",
    "\n",
    "        Arguments:\n",
    "            y: `array`. Class vector to convert.\n",
    "            nb_classes: `int`. Total number of classes.\n",
    "\n",
    "        \"\"\"\n",
    "        y = np.asarray(y, dtype='int32')\n",
    "        if not nb_classes:\n",
    "            nb_classes = np.max(y)+1\n",
    "        Y = np.zeros((len(y), nb_classes))\n",
    "        for i in range(len(y)):\n",
    "            Y[i, y[i]] = 1.\n",
    "        return Y\n",
    "\n",
    "\n",
    "    # =====================\n",
    "    #    SEQUENCES UTILS\n",
    "    # =====================\n",
    "\n",
    "\n",
    "    def pad_sequences(sequences, maxlen=None, dtype='int32', padding='post',\n",
    "                      truncating='post', value=0.):\n",
    "        \"\"\" pad_sequences.\n",
    "\n",
    "        Pad each sequence to the same length: the length of the longest sequence.\n",
    "        If maxlen is provided, any sequence longer than maxlen is truncated to\n",
    "        maxlen. Truncation happens off either the beginning or the end (default)\n",
    "        of the sequence. Supports pre-padding and post-padding (default).\n",
    "\n",
    "        Arguments:\n",
    "            sequences: list of lists where each element is a sequence.\n",
    "            maxlen: int, maximum length.\n",
    "            dtype: type to cast the resulting sequence.\n",
    "            padding: 'pre' or 'post', pad either before or after each sequence.\n",
    "            truncating: 'pre' or 'post', remove values from sequences larger than\n",
    "                maxlen either in the beginning or in the end of the sequence\n",
    "            value: float, value to pad the sequences to the desired value.\n",
    "\n",
    "        Returns:\n",
    "            x: `numpy array` with dimensions (number_of_sequences, maxlen)\n",
    "\n",
    "        Credits: From Keras `pad_sequences` function.\n",
    "        \"\"\"\n",
    "        lengths = [len(s) for s in sequences]\n",
    "\n",
    "        nb_samples = len(sequences)\n",
    "        if maxlen is None:\n",
    "            maxlen = np.max(lengths)\n",
    "\n",
    "        x = (np.ones((nb_samples, maxlen)) * value).astype(dtype)\n",
    "        for idx, s in enumerate(sequences):\n",
    "            if len(s) == 0:\n",
    "                continue  # empty list was found\n",
    "            if truncating == 'pre':\n",
    "                trunc = s[-maxlen:]\n",
    "            elif truncating == 'post':\n",
    "                trunc = s[:maxlen]\n",
    "            else:\n",
    "                raise ValueError(\"Truncating type '%s' not understood\" % padding)\n",
    "\n",
    "            if padding == 'post':\n",
    "                x[idx, :len(trunc)] = trunc\n",
    "            elif padding == 'pre':\n",
    "                x[idx, -len(trunc):] = trunc\n",
    "            else:\n",
    "                raise ValueError(\"Padding type '%s' not understood\" % padding)\n",
    "        return x\n",
    "\n",
    "\n",
    "    def string_to_semi_redundant_sequences(string, seq_maxlen=25, redun_step=3, char_idx=None):\n",
    "        \"\"\" string_to_semi_redundant_sequences.\n",
    "\n",
    "        Vectorize a string and returns parsed sequences and targets, along with\n",
    "        the associated dictionary.\n",
    "\n",
    "        Arguments:\n",
    "            string: `str`. Lower-case text from input text file.\n",
    "            seq_maxlen: `int`. Maximum length of a sequence. Default: 25.\n",
    "            redun_step: `int`. Redundancy step. Default: 3.\n",
    "            char_idx: 'dict'. A dictionary to convert chars to positions. Will be automatically generated if None\n",
    "\n",
    "        Returns:\n",
    "            A tuple: (inputs, targets, dictionary)\n",
    "        \"\"\"\n",
    "\n",
    "        print(\"Vectorizing text...\")\n",
    "\n",
    "        if char_idx is None:\n",
    "          char_idx = chars_to_dictionary(string)\n",
    "\n",
    "        len_chars = len(char_idx)\n",
    "\n",
    "        sequences = []\n",
    "        next_chars = []\n",
    "        for i in range(0, len(string) - seq_maxlen, redun_step):\n",
    "            sequences.append(string[i: i + seq_maxlen])\n",
    "            next_chars.append(string[i + seq_maxlen])\n",
    "\n",
    "        X = np.zeros((len(sequences), seq_maxlen, len_chars), dtype=np.bool)\n",
    "        Y = np.zeros((len(sequences), len_chars), dtype=np.bool)\n",
    "        for i, seq in enumerate(sequences):\n",
    "            for t, char in enumerate(seq):\n",
    "                X[i, t, char_idx[char]] = 1\n",
    "            Y[i, char_idx[next_chars[i]]] = 1\n",
    "\n",
    "        print(\"Text total length: {:,}\".format(len(string)))\n",
    "        print(\"Distinct chars   : {:,}\".format(len_chars))\n",
    "        print(\"Total sequences  : {:,}\".format(len(sequences)))\n",
    "\n",
    "        return X, Y, char_idx\n",
    "\n",
    "\n",
    "    def textfile_to_semi_redundant_sequences(path, seq_maxlen=25, redun_step=3,\n",
    "                                             to_lower_case=False, pre_defined_char_idx=None):\n",
    "        \"\"\" Vectorize Text file \"\"\"\n",
    "        text = open(path).read()\n",
    "        if to_lower_case:\n",
    "            text = text.lower()\n",
    "        return string_to_semi_redundant_sequences(text, seq_maxlen, redun_step, pre_defined_char_idx)\n",
    "\n",
    "\n",
    "    def chars_to_dictionary(string):\n",
    "        \"\"\" Creates a dictionary char:integer for each unique character \"\"\"\n",
    "        chars = set(string)\n",
    "        # sorted tries to keep a consistent dictionary, if you run a second time for the same char set\n",
    "        char_idx = {c: i for i, c in enumerate(sorted(chars))}\n",
    "        return char_idx\n",
    "\n",
    "\n",
    "    def random_sequence_from_string(string, seq_maxlen):\n",
    "        rand_index = random.randint(0, len(string) - seq_maxlen - 1)\n",
    "        return string[rand_index: rand_index + seq_maxlen]\n",
    "\n",
    "\n",
    "    def random_sequence_from_textfile(path, seq_maxlen):\n",
    "        text = open(path).read()\n",
    "        return random_sequence_from_string(text, seq_maxlen)\n",
    "\n",
    "    try:\n",
    "        from tensorflow.contrib.learn.python.learn.preprocessing.text import \\\n",
    "            VocabularyProcessor as _VocabularyProcessor\n",
    "    except Exception:\n",
    "        _VocabularyProcessor = object\n",
    "\n",
    "\n",
    "    # Mirroring TensorFLow `VocabularyProcessor`\n",
    "    class VocabularyProcessor(_VocabularyProcessor):\n",
    "        \"\"\" Vocabulary Processor.\n",
    "\n",
    "        Maps documents to sequences of word ids.\n",
    "\n",
    "        Arguments:\n",
    "            max_document_length: Maximum length of documents.\n",
    "                if documents are longer, they will be trimmed, if shorter - padded.\n",
    "            min_frequency: Minimum frequency of words in the vocabulary.\n",
    "            vocabulary: CategoricalVocabulary object.\n",
    "\n",
    "        Attributes:\n",
    "            vocabulary_: CategoricalVocabulary object.\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        def __init__(self,\n",
    "                     max_document_length,\n",
    "                     min_frequency=0,\n",
    "                     vocabulary=None,\n",
    "                     tokenizer_fn=None):\n",
    "            super(VocabularyProcessor, self).__init__(max_document_length,\n",
    "                                                      min_frequency,\n",
    "                                                      vocabulary,\n",
    "                                                      tokenizer_fn)\n",
    "\n",
    "        def fit(self, raw_documents, unused_y=None):\n",
    "            \"\"\" fit.\n",
    "\n",
    "            Learn a vocabulary dictionary of all tokens in the raw documents.\n",
    "\n",
    "            Arguments:\n",
    "                raw_documents: An iterable which yield either str or unicode.\n",
    "                unused_y: to match fit format signature of estimators.\n",
    "\n",
    "            Returns:\n",
    "                self\n",
    "            \"\"\"\n",
    "            return super(VocabularyProcessor, self).fit(raw_documents, unused_y)\n",
    "\n",
    "        def fit_transform(self, raw_documents, unused_y=None):\n",
    "            \"\"\" fit_transform.\n",
    "\n",
    "            Learn the vocabulary dictionary and return indexies of words.\n",
    "\n",
    "            Arguments:\n",
    "                raw_documents: An iterable which yield either str or unicode.\n",
    "                unused_y: to match fit_transform signature of estimators.\n",
    "\n",
    "            Returns:\n",
    "                X: iterable, [n_samples, max_document_length] Word-id matrix.\n",
    "            \"\"\"\n",
    "            return super(VocabularyProcessor, self).fit_transform(raw_documents,\n",
    "                                                                  unused_y)\n",
    "\n",
    "        def transform(self, raw_documents):\n",
    "            \"\"\" transform.\n",
    "\n",
    "            Transform documents to word-id matrix.\n",
    "\n",
    "            Convert words to ids with vocabulary fitted with fit or the one\n",
    "            provided in the constructor.\n",
    "\n",
    "            Arguments:\n",
    "                raw_documents: An iterable which yield either str or unicode.\n",
    "\n",
    "            Yields:\n",
    "                X: iterable, [n_samples, max_document_length] Word-id matrix.\n",
    "            \"\"\"\n",
    "            return super(VocabularyProcessor, self).transform(raw_documents)\n",
    "\n",
    "        def reverse(self, documents):\n",
    "            \"\"\" reverse.\n",
    "\n",
    "            Reverses output of vocabulary mapping to words.\n",
    "\n",
    "            Arguments:\n",
    "                documents: iterable, list of class ids.\n",
    "\n",
    "            Returns:\n",
    "                Iterator over mapped in words documents.\n",
    "            \"\"\"\n",
    "            return super(VocabularyProcessor, self).reverse(documents)\n",
    "\n",
    "        def save(self, filename):\n",
    "            \"\"\" save.\n",
    "\n",
    "            Saves vocabulary processor into given file.\n",
    "\n",
    "            Arguments:\n",
    "                filename: Path to output file.\n",
    "            \"\"\"\n",
    "            super(VocabularyProcessor, self).save(filename)\n",
    "\n",
    "        @classmethod\n",
    "        def restore(cls, filename):\n",
    "            \"\"\" restore.\n",
    "\n",
    "            Restores vocabulary processor from given file.\n",
    "\n",
    "            Arguments:\n",
    "                filename: Path to file to load from.\n",
    "\n",
    "            Returns:\n",
    "                VocabularyProcessor object.\n",
    "            \"\"\"\n",
    "            return super(VocabularyProcessor, cls).restore(filename)\n",
    "\n",
    "\n",
    "    # ===================\n",
    "    #    IMAGES UTILS\n",
    "    # ===================\n",
    "\n",
    "    def build_hdf5_image_dataset(target_path, image_shape, output_path='dataset.h5',\n",
    "                                 mode='file', categorical_labels=True,\n",
    "                                 normalize=True, grayscale=False,\n",
    "                                 files_extension=None, chunks=True):\n",
    "        \"\"\" Build HDF5 Image Dataset.\n",
    "\n",
    "        Build an HDF5 dataset by providing either a root folder or a plain text\n",
    "        file with images path and class id.\n",
    "\n",
    "        'folder' mode: Root folder should be arranged as follow:\n",
    "        ```\n",
    "        ROOT_FOLDER -> SUBFOLDER_0 (CLASS 0) -> CLASS0_IMG1.jpg\n",
    "                                             -> CLASS0_IMG2.jpg\n",
    "                                             -> ...\n",
    "                    -> SUBFOLDER_1 (CLASS 1) -> CLASS1_IMG1.jpg\n",
    "                                             -> ...\n",
    "                    -> ...\n",
    "        ```\n",
    "        Note that if sub-folders are not integers from 0 to n_classes, an id will\n",
    "        be assigned to each sub-folder following alphabetical order.\n",
    "\n",
    "        'file' mode: Plain text file should be formatted as follow:\n",
    "        ```\n",
    "        /path/to/img1 class_id\n",
    "        /path/to/img2 class_id\n",
    "        /path/to/img3 class_id\n",
    "        ```\n",
    "\n",
    "        Examples:\n",
    "            ```\n",
    "            # Load path/class_id image file:\n",
    "            dataset_file = 'my_dataset.txt'\n",
    "\n",
    "            # Build a HDF5 dataset (only required once)\n",
    "            from tflearn.data_utils import build_hdf5_image_dataset\n",
    "            build_hdf5_image_dataset(dataset_file, image_shape=(128, 128),\n",
    "                                     mode='file', output_path='dataset.h5',\n",
    "                                     categorical_labels=True, normalize=True)\n",
    "\n",
    "            # Load HDF5 dataset\n",
    "            import h5py\n",
    "            h5f = h5py.File('dataset.h5', 'w')\n",
    "            X = h5f['X']\n",
    "            Y = h5f['Y']\n",
    "\n",
    "            # Build neural network and train\n",
    "            network = ...\n",
    "            model = DNN(network, ...)\n",
    "            model.fit(X, Y)\n",
    "            ```\n",
    "\n",
    "        Arguments:\n",
    "            target_path: `str`. Path of root folder or images plain text file.\n",
    "            image_shape: `tuple (height, width)`. The images shape. Images that\n",
    "                doesn't match that shape will be resized.\n",
    "            output_path: `str`. The output path for the hdf5 dataset. Default:\n",
    "                'dataset.h5'\n",
    "            mode: `str` in ['file', 'folder']. The data source mode. 'folder'\n",
    "                accepts a root folder with each of his sub-folder representing a\n",
    "                class containing the images to classify.\n",
    "                'file' accepts a single plain text file that contains every\n",
    "                image path with their class id.\n",
    "                Default: 'folder'.\n",
    "            categorical_labels: `bool`. If True, labels are converted to binary\n",
    "                vectors.\n",
    "            normalize: `bool`. If True, normalize all pictures by dividing\n",
    "                every image array by 255.\n",
    "            grayscale: `bool`. If true, images are converted to grayscale.\n",
    "            files_extension: `list of str`. A list of allowed image file\n",
    "                extension, for example ['.jpg', '.jpeg', '.png']. If None,\n",
    "                all files are allowed.\n",
    "            chunks: `bool` or `list of int`. Whether to chunks the dataset or not.\n",
    "                Additionaly, a specific shape for each chunk can be provided.\n",
    "\n",
    "        \"\"\"\n",
    "        import h5py\n",
    "\n",
    "        assert image_shape, \"Image shape must be defined.\"\n",
    "        assert image_shape[0] and image_shape[1], \\\n",
    "            \"Image shape error. It must be a tuple of int: ('width', 'height').\"\n",
    "        assert mode in ['folder', 'file'], \"`mode` arg must be 'folder' or 'file'\"\n",
    "\n",
    "        if mode == 'folder':\n",
    "            images, labels = directory_to_samples(target_path,\n",
    "                                                  flags=files_extension)\n",
    "        else:\n",
    "            with open(target_path, 'r') as f:\n",
    "                images, labels = [], []\n",
    "                for l in f.readlines():\n",
    "                    l = l.strip('\\n').split()\n",
    "                    images.append(l[0])\n",
    "                    labels.append(int(l[1]))\n",
    "\n",
    "        n_classes = np.max(labels) + 1\n",
    "\n",
    "        d_imgshape = (len(images), image_shape[0], image_shape[1], 3) \\\n",
    "            if not grayscale else (len(images), image_shape[0], image_shape[1])\n",
    "        d_labelshape = (len(images), n_classes) \\\n",
    "            if categorical_labels else (len(images), )\n",
    "\n",
    "        dataset = h5py.File(output_path, 'w')\n",
    "        dataset.create_dataset('X', d_imgshape, chunks=chunks)\n",
    "        dataset.create_dataset('Y', d_labelshape, chunks=chunks)\n",
    "\n",
    "        for i in range(len(images)):\n",
    "            img = load_image(images[i])\n",
    "            width, height = img.size\n",
    "            if width != image_shape[0] or height != image_shape[1]:\n",
    "                img = resize_image(img, image_shape[0], image_shape[1])\n",
    "            if grayscale:\n",
    "                img = convert_color(img, 'L')\n",
    "            elif img.mode == 'L':\n",
    "                img = convert_color(img, 'RGB')\n",
    "\n",
    "            img = pil_to_nparray(img)\n",
    "            if normalize:\n",
    "                img /= 255.\n",
    "            dataset['X'][i] = img\n",
    "            if categorical_labels:\n",
    "                dataset['Y'][i] = to_categorical([labels[i]], n_classes)[0]\n",
    "            else:\n",
    "                dataset['Y'][i] = labels[i]\n",
    "\n",
    "    def get_img_channel(image_path):\n",
    "        \"\"\"\n",
    "        Load a image and return the channel of the image\n",
    "        :param image_path:\n",
    "        :return: the channel of the image\n",
    "        \"\"\"\n",
    "        try:\n",
    "            img = load_image(image_path)\n",
    "            img = pil_to_nparray(img)\n",
    "            channel = img.shape[2]\n",
    "        except:\n",
    "            channel = 1\n",
    "        return channel\n",
    "\n",
    "    def image_preloader(target_path, image_shape, mode='file', normalize=True,\n",
    "                        grayscale=False, categorical_labels=True,\n",
    "                        files_extension=None, filter_channel=False):\n",
    "        \"\"\" Image PreLoader.\n",
    "\n",
    "        Create a python array (`Preloader`) that loads images on the fly (from\n",
    "        disk or url). There is two ways to provide image samples 'folder' or\n",
    "        'file', see the specifications below.\n",
    "\n",
    "        'folder' mode: Load images from disk, given a root folder. This folder\n",
    "        should be arranged as follow:\n",
    "        ```\n",
    "        ROOT_FOLDER -> SUBFOLDER_0 (CLASS 0) -> CLASS0_IMG1.jpg\n",
    "                                             -> CLASS0_IMG2.jpg\n",
    "                                             -> ...\n",
    "                    -> SUBFOLDER_1 (CLASS 1) -> CLASS1_IMG1.jpg\n",
    "                                             -> ...\n",
    "                    -> ...\n",
    "        ```\n",
    "        Note that if sub-folders are not integers from 0 to n_classes, an id will\n",
    "        be assigned to each sub-folder following alphabetical order.\n",
    "\n",
    "        'file' mode: A plain text file listing every image path and class id.\n",
    "        This file should be formatted as follow:\n",
    "        ```\n",
    "        /path/to/img1 class_id\n",
    "        /path/to/img2 class_id\n",
    "        /path/to/img3 class_id\n",
    "        ```\n",
    "\n",
    "        Note that load images on the fly and convert is time inefficient,\n",
    "        so you can instead use `build_hdf5_image_dataset` to build a HDF5 dataset\n",
    "        that enable fast retrieval (this function takes similar arguments).\n",
    "\n",
    "        Examples:\n",
    "            ```\n",
    "            # Load path/class_id image file:\n",
    "            dataset_file = 'my_dataset.txt'\n",
    "\n",
    "            # Build the preloader array, resize images to 128x128\n",
    "            from tflearn.data_utils import image_preloader\n",
    "            X, Y = image_preloader(dataset_file, image_shape=(128, 128),\n",
    "                                   mode='file', categorical_labels=True,\n",
    "                                   normalize=True)\n",
    "\n",
    "            # Build neural network and train\n",
    "            network = ...\n",
    "            model = DNN(network, ...)\n",
    "            model.fit(X, Y)\n",
    "            ```\n",
    "\n",
    "        Arguments:\n",
    "            target_path: `str`. Path of root folder or images plain text file.\n",
    "            image_shape: `tuple (height, width)`. The images shape. Images that\n",
    "                doesn't match that shape will be resized.\n",
    "            mode: `str` in ['file', 'folder']. The data source mode. 'folder'\n",
    "                accepts a root folder with each of his sub-folder representing a\n",
    "                class containing the images to classify.\n",
    "                'file' accepts a single plain text file that contains every\n",
    "                image path with their class id.\n",
    "                Default: 'folder'.\n",
    "            categorical_labels: `bool`. If True, labels are converted to binary\n",
    "                vectors.\n",
    "            normalize: `bool`. If True, normalize all pictures by dividing\n",
    "                every image array by 255.\n",
    "            grayscale: `bool`. If true, images are converted to grayscale.\n",
    "            files_extension: `list of str`. A list of allowed image file\n",
    "                extension, for example ['.jpg', '.jpeg', '.png']. If None,\n",
    "                all files are allowed.\n",
    "            filter_channel: `bool`. If true, images which the channel is not 3 should\n",
    "                be filter.\n",
    "\n",
    "        Returns:\n",
    "            (X, Y): with X the images array and Y the labels array.\n",
    "\n",
    "        \"\"\"\n",
    "        assert mode in ['folder', 'file']\n",
    "        if mode == 'folder':\n",
    "            images, labels = directory_to_samples(target_path,\n",
    "                                                  flags=files_extension, filter_channel=filter_channel)\n",
    "        else:\n",
    "            with open(target_path, 'r') as f:\n",
    "                images, labels = [], []\n",
    "                for l in f.readlines():\n",
    "                    l = l.strip('\\n').split()\n",
    "                    if not files_extension or any(flag in l(0) for flag in files_extension):\n",
    "                        if filter_channel:\n",
    "                            channel_cnt = get_img_channel(l[0])\n",
    "                            if channel_cnt != 3:\n",
    "                                continue\n",
    "                        images.append(l[0])\n",
    "                        labels.append(int(l[1]))\n",
    "\n",
    "        n_classes = np.max(labels) + 1\n",
    "        X = ImagePreloader(images, image_shape, normalize, grayscale)\n",
    "        Y = LabelPreloader(labels, n_classes, categorical_labels)\n",
    "\n",
    "        return X, Y\n",
    "\n",
    "\n",
    "    def load_image(in_image):\n",
    "        \"\"\" Load an image, returns PIL.Image. \"\"\"\n",
    "        img = Image.open(in_image)\n",
    "        return img\n",
    "\n",
    "\n",
    "    def resize_image(in_image, new_width, new_height, out_image=None,\n",
    "                     resize_mode=Image.ANTIALIAS):\n",
    "        \"\"\" Resize an image.\n",
    "\n",
    "        Arguments:\n",
    "            in_image: `PIL.Image`. The image to resize.\n",
    "            new_width: `int`. The image new width.\n",
    "            new_height: `int`. The image new height.\n",
    "            out_image: `str`. If specified, save the image to the given path.\n",
    "            resize_mode: `PIL.Image.mode`. The resizing mode.\n",
    "\n",
    "        Returns:\n",
    "            `PIL.Image`. The resize image.\n",
    "\n",
    "        \"\"\"\n",
    "        img = in_image.resize((new_width, new_height), resize_mode)\n",
    "        if out_image:\n",
    "            img.save(out_image)\n",
    "        return img\n",
    "\n",
    "\n",
    "    def convert_color(in_image, mode):\n",
    "        \"\"\" Convert image color with provided `mode`. \"\"\"\n",
    "        return in_image.convert(mode)\n",
    "\n",
    "\n",
    "    def pil_to_nparray(pil_image):\n",
    "        \"\"\" Convert a PIL.Image to numpy array. \"\"\"\n",
    "        pil_image.load()\n",
    "        return np.asarray(pil_image, dtype=\"float32\")\n",
    "\n",
    "\n",
    "    def image_dirs_to_samples(dirname, label_size, resize=None, convert_gray=None, filetypes=None, \n",
    "                            down_sampling=None, categorical_Y=True, shuffle_data=True):\n",
    "        #------------------------------------------------------\n",
    "        def dump_samples(X, Y, fptr):\n",
    "            # X, Y = np.array(X), np.array(Y)\n",
    "            if categorical_Y:\n",
    "                Y = to_categorical(Y, label_size) # First class is '0'\n",
    "            fname = \"images/%s/samples_%i.pkl.gz\" % (dirname, fptr)\n",
    "            pickle.dump((X, Y), gzip.open(fname, 'wb'))\n",
    "            print(\"dumped samples shape %s, %s in %s @ %s\" % (np.shape(X), np.shape(Y), fname, datetime.now()))\n",
    "            return fname\n",
    "            \n",
    "        #------------------------------------------------------\n",
    "        print(\"Starting to parse images...\")\n",
    "        if filetypes:\n",
    "            if filetypes not in [list, tuple]: filetypes = list(filetypes)\n",
    "        directory = \"images/%s/jpg/\" % dirname\n",
    "        raw_samples, raw_targets = directory_to_samples(directory, flags=filetypes, down_sampling=down_sampling)\n",
    "        if shuffle_data:\n",
    "            raw_samples, raw_targets = shuffle(raw_samples, raw_targets)\n",
    "            \n",
    "        raw_samples, raw_targets = list(raw_samples), list(raw_targets)\n",
    "        # generate sample caches\n",
    "        pkl_files = []\n",
    "        print(\"%i samples to go @ %s\" % (len(raw_samples), datetime.now()))\n",
    "        samples, targets, i, fptr = [], [], 0, 0\n",
    "        while raw_samples:\n",
    "            s, t = raw_samples.pop(0), raw_targets.pop(0)\n",
    "            try:\n",
    "                img = load_image(s)\n",
    "            except Exception as e:\n",
    "                print(e)\n",
    "                continue\n",
    "            if resize:\n",
    "                img = resize_image(img, resize[0], resize[1])\n",
    "            if convert_gray:\n",
    "                img = convert_color(img, 'L')\n",
    "            img = pil_to_nparray(img)\n",
    "            # ensure image_channel = 3\n",
    "            if (len(img.shape) != 3) or (img.shape[2] != 3): continue\n",
    "            # normalize\n",
    "            img /= 255.\n",
    "            samples.append(img)\n",
    "            targets.append(t)\n",
    "            i += 1\n",
    "            if (i % 500 == 0): \n",
    "                print(\"%i img loaded @ %s\" % (i, datetime.now()))\n",
    "            # if (i % 5000 == 0):\n",
    "                fname = dump_samples(samples, targets, fptr)\n",
    "                pkl_files.append(fname)\n",
    "                samples, targets, fptr = [], [], (fptr+1)\n",
    "        if samples:\n",
    "            fname = dump_samples(samples, targets, fptr)\n",
    "            pkl_files.append(fname)\n",
    "            samples, targets, fptr = [], [], (fptr+1)\n",
    "        print(\"Parsing Done, %i samples, %i pkl files @ %s\" % (i, fptr, datetime.now()))\n",
    "        return pkl_files\n",
    "\n",
    "\n",
    "\n",
    "    # def build_image_dataset_from_dir(dirname,\n",
    "    #                                  # directory,\n",
    "    #                                  # dataset_fname,\n",
    "    #                                  resize=None, convert_gray=None,\n",
    "    #                                  filetypes=None, shuffle_data=False,\n",
    "    #                                  categorical_Y=False,\n",
    "    #                                  down_sampling=None,\n",
    "    #                                  ):\n",
    "        \n",
    "    #     dataset_cache = [f for f in os.listdir(\"images/%s/\" % dirname) if \"samples_\" in f]\n",
    "    #     if not dataset_cache:\n",
    "    #         X, Y = image_dirs_to_samples(\"images/%s/jpg/\" % dirname, resize, convert_gray, filetypes, down_sampling=down_sampling, \n",
    "    #             dataset_fname=dataset_fname, categorical_Y=categorical_Y, shuffle_data=shuffle_data)\n",
    "    #     return dataset_cache\n",
    "\n",
    "\n",
    "    def random_flip_leftright(x):\n",
    "        if bool(random.getrandbits(1)):\n",
    "            return np.fliplr(x)\n",
    "        else:\n",
    "            return x\n",
    "\n",
    "\n",
    "    def random_flip_updown(x):\n",
    "        if bool(random.getrandbits(1)):\n",
    "            return np.flipud(x)\n",
    "        else:\n",
    "            return x\n",
    "\n",
    "\n",
    "    # ==================\n",
    "    #     DATA UTILS\n",
    "    # ==================\n",
    "\n",
    "\n",
    "    def shuffle(*arrs):\n",
    "        \"\"\" shuffle.\n",
    "\n",
    "        Shuffle given arrays at unison, along first axis.\n",
    "\n",
    "        Arguments:\n",
    "            *arrs: Each array to shuffle at unison.\n",
    "\n",
    "        Returns:\n",
    "            Tuple of shuffled arrays.\n",
    "\n",
    "        \"\"\"\n",
    "        arrs = list(arrs)\n",
    "        for i, arr in enumerate(arrs):\n",
    "            assert len(arrs[0]) == len(arrs[i])\n",
    "            arrs[i] = np.array(arr)\n",
    "        p = np.random.permutation(len(arrs[0]))\n",
    "        return tuple(arr[p] for arr in arrs)\n",
    "\n",
    "\n",
    "    def samplewise_zero_center(X):\n",
    "        \"\"\" samplewise_zero_center.\n",
    "\n",
    "        Zero center each sample by subtracting it by its mean.\n",
    "\n",
    "        Arguments:\n",
    "            X: `array`. The batch of samples to center.\n",
    "\n",
    "        Returns:\n",
    "            A numpy array with same shape as input.\n",
    "\n",
    "        \"\"\"\n",
    "        for i in range(len(X)):\n",
    "            X[i] -= np.mean(X[i], axis=1, keepdims=True)\n",
    "        return X\n",
    "\n",
    "\n",
    "    def samplewise_std_normalization(X):\n",
    "        \"\"\" samplewise_std_normalization.\n",
    "\n",
    "        Scale each sample with its standard deviation.\n",
    "\n",
    "        Arguments:\n",
    "            X: `array`. The batch of samples to scale.\n",
    "\n",
    "        Returns:\n",
    "            A numpy array with same shape as input.\n",
    "\n",
    "        \"\"\"\n",
    "        for i in range(len(X)):\n",
    "            X[i] /= (np.std(X[i], axis=1, keepdims=True) + _EPSILON)\n",
    "        return X\n",
    "\n",
    "\n",
    "    def featurewise_zero_center(X, mean=None):\n",
    "        \"\"\" featurewise_zero_center.\n",
    "\n",
    "        Zero center every sample with specified mean. If not specified, the mean\n",
    "        is evaluated over all samples.\n",
    "\n",
    "        Arguments:\n",
    "            X: `array`. The batch of samples to center.\n",
    "            mean: `float`. The mean to use for zero centering. If not specified, it\n",
    "                will be evaluated on provided data.\n",
    "\n",
    "        Returns:\n",
    "            A numpy array with same shape as input. Or a tuple (array, mean) if no\n",
    "            mean value was specified.\n",
    "\n",
    "        \"\"\"\n",
    "        if mean is None:\n",
    "            mean = np.mean(X, axis=0)\n",
    "            return X - mean, mean\n",
    "        else:\n",
    "            return X - mean\n",
    "\n",
    "\n",
    "    def featurewise_std_normalization(X, std=None):\n",
    "        \"\"\" featurewise_std_normalization.\n",
    "\n",
    "        Scale each sample by the specified standard deviation. If no std\n",
    "        specified, std is evaluated over all samples data.\n",
    "\n",
    "        Arguments:\n",
    "            X: `array`. The batch of samples to scale.\n",
    "            std: `float`. The std to use for scaling data. If not specified, it\n",
    "                will be evaluated over the provided data.\n",
    "\n",
    "        Returns:\n",
    "            A numpy array with same shape as input. Or a tuple (array, std) if no\n",
    "            std value was specified.\n",
    "\n",
    "        \"\"\"\n",
    "        if std is None:\n",
    "            std = np.std(X, axis=0)\n",
    "            return X / std, std\n",
    "        else:\n",
    "            return X / std\n",
    "\n",
    "\n",
    "    def directory_to_samples(directory, flags=None, filter_channel=False, down_sampling=None):\n",
    "        \"\"\" Read a directory, and list all subdirectories files as class sample \"\"\"\n",
    "        samples = []\n",
    "        targets = []\n",
    "        label = 0\n",
    "        # try: # Python 2\n",
    "        #     classes = sorted(os.walk(directory).next()[1])\n",
    "        #     # classes = sorted(next(os.walk(directory))[1])\n",
    "        # except Exception: # Python 3\n",
    "        print(directory)\n",
    "        classes = os.walk(directory).__next__()[1]\n",
    "        # keep category numerical order, if all categories are digits\n",
    "        if all([n.isdigit() for n in classes]):\n",
    "            classes = sorted(classes, key=lambda v: int(v))\n",
    "        else:\n",
    "            classes = sorted(classes)\n",
    "        for c in classes:\n",
    "            c_dir = os.path.join(directory, c)\n",
    "            # try: # Python 2\n",
    "            #     walk = os.walk(c_dir).next()\n",
    "            #     # walk = next(os.walk(c_dir))\n",
    "            # except Exception: # Python 3\n",
    "            walk = os.walk(c_dir).__next__()\n",
    "            cands = walk[2]\n",
    "            # if down sampling, discard this sample with probability = 1-down_sampling \n",
    "            if down_sampling and (c in down_sampling):\n",
    "                if down_sampling[c] > 1:\n",
    "                    l = down_sampling[c]\n",
    "                else:\n",
    "                    l = max(50, int(len(cands) * down_sampling[c]))\n",
    "                cands = cands[:l]\n",
    "            for sample in cands:\n",
    "                if not flags or any(flag in sample for flag in flags):\n",
    "                    if filter_channel:\n",
    "                        channel_cnt = get_img_channel(os.path.join(c_dir, sample))\n",
    "                        if channel_cnt != 3:\n",
    "                            continue\n",
    "                    samples.append(os.path.join(c_dir, sample))\n",
    "                    targets.append(label)\n",
    "            print(\"samples dim %s, target dim %s\" % (np.shape(samples), np.shape(targets)))\n",
    "            label += 1\n",
    "        return samples, targets\n",
    "\n",
    "\n",
    "\n",
    "    def load_csv(filepath, target_column=-1, columns_to_ignore=None,\n",
    "                 has_header=True, categorical_labels=False, n_classes=None):\n",
    "        \"\"\" load_csv.\n",
    "\n",
    "        Load data from a CSV file. By default the labels are considered to be the\n",
    "        last column, but it can be changed by filling 'target_column' parameter.\n",
    "\n",
    "        Arguments:\n",
    "            filepath: `str`. The csv file path.\n",
    "            target_column: The id of the column representing the labels.\n",
    "                Default: -1 (The last column).\n",
    "            columns_to_ignore: `list of int`. A list of columns index to ignore.\n",
    "            has_header: `bool`. Whether the csv file has a header or not.\n",
    "            categorical_labels: `bool`. If True, labels are returned as binary\n",
    "                vectors (to be used with 'categorical_crossentropy').\n",
    "            n_classes: `int`. Total number of class (needed if\n",
    "                categorical_labels is True).\n",
    "\n",
    "        Returns:\n",
    "            A tuple (data, target).\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        from tensorflow.python.platform import gfile\n",
    "        with gfile.Open(filepath) as csv_file:\n",
    "            data_file = csv.reader(csv_file)\n",
    "            if not columns_to_ignore:\n",
    "                columns_to_ignore = []\n",
    "            if has_header:\n",
    "                header = next(data_file)\n",
    "            data, target = [], []\n",
    "            # Fix column to ignore ids after removing target_column\n",
    "            for i, c in enumerate(columns_to_ignore):\n",
    "                if c > target_column:\n",
    "                    columns_to_ignore[i] -= 1\n",
    "            for i, d in enumerate(data_file):\n",
    "                target.append(d.pop(target_column))\n",
    "                data.append([_d for j, _d in enumerate(d) if j not in columns_to_ignore])\n",
    "            if categorical_labels:\n",
    "                assert isinstance(n_classes, int), \"n_classes not specified!\"\n",
    "                target = to_categorical(target, n_classes)\n",
    "            return data, target\n",
    "\n",
    "\n",
    "    class Preloader(object):\n",
    "        def __init__(self, array, function):\n",
    "            self.array = array\n",
    "            self.function = function\n",
    "\n",
    "        def __getitem__(self, id):\n",
    "            if type(id) in [list, np.ndarray]:\n",
    "                return [self.function(self.array[i]) for i in id]\n",
    "            elif isinstance(id, slice):\n",
    "                return [self.function(arr) for arr in self.array[id]]\n",
    "            else:\n",
    "                return self.function(self.array[id])\n",
    "\n",
    "        def __len__(self):\n",
    "            return len(self.array)\n",
    "\n",
    "\n",
    "    class ImagePreloader(Preloader):\n",
    "        def __init__(self, array, image_shape, normalize=True, grayscale=False):\n",
    "            fn = lambda x: self.preload(x, image_shape, normalize, grayscale)\n",
    "            super(ImagePreloader, self).__init__(array, fn)\n",
    "\n",
    "        def preload(self, path, image_shape, normalize=True, grayscale=False):\n",
    "            img = load_image(path)\n",
    "            width, height = img.size\n",
    "            if width != image_shape[0] or height != image_shape[1]:\n",
    "                img = resize_image(img, image_shape[0], image_shape[1])\n",
    "            if grayscale:\n",
    "                img = convert_color(img, 'L')\n",
    "            img = pil_to_nparray(img)\n",
    "            if normalize:\n",
    "                img /= 255.\n",
    "            return img\n",
    "\n",
    "\n",
    "    class LabelPreloader(Preloader):\n",
    "        def __init__(self, array, n_class=None, categorical_label=True):\n",
    "            fn = lambda x: self.preload(x, n_class, categorical_label)\n",
    "            super(LabelPreloader, self).__init__(array, fn)\n",
    "\n",
    "        def preload(self, label, n_class, categorical_label):\n",
    "            if categorical_label:\n",
    "                #TODO: inspect assert bug\n",
    "                #assert isinstance(n_class, int)\n",
    "                return to_categorical([label], n_class)[0]\n",
    "            else:\n",
    "                return label\n",
    "\n",
    "\n",
    "    def get_max(X):\n",
    "        return np.max(X)\n",
    "\n",
    "\n",
    "    def get_mean(X):\n",
    "        return np.mean(X)\n",
    "\n",
    "\n",
    "    def get_std(X):\n",
    "        return np.std(X)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在flowers数据上进行训练\n",
    "\n",
    "先将数据download到本地，数据地址如下：\n",
    "http://www.robots.ox.ac.uk/~vgg/data/flowers/17/17flowers.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division, print_function, absolute_import\n",
    "\n",
    "from datetime import datetime\n",
    "\n",
    "import pickle, gzip\n",
    "import numpy as np\n",
    "import tflearn.datasets.oxflower17 as oxflower17\n",
    "\n",
    "hps = HParams(img_size=227,\n",
    "             label_size=17,\n",
    "             gpu_memory_fraction=0.4,\n",
    "             model_name=\"17flowers\",\n",
    "             gpu_usage=0.5)\n",
    "gnet = GoogLeNet(hps=hps) \n",
    "pkl_files = gnet.get_data(dirname=\"17flowers\", down_sampling={str(n): 10000 for n in range(13)})\n",
    "\n",
    "epoch = 0\n",
    "while True:\n",
    "    for f in pkl_files:\n",
    "        X, Y = pickle.load(gzip.open(f, 'rb'))\n",
    "        gnet.fit(X, Y, n_epoch=10)\n",
    "        print('[pkl_files] done with %s @ %s' % (f, datetime.now()))\n",
    "    epoch += 1\n",
    "    print(\"[Finish] all pkl_files been trained %i times.\" % epoch)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练结果\n",
    "\n",
    "训练过程如下图所示：\n",
    "![](img/training_googlenet.PNG)\n",
    "\n",
    "训练过程中precision平稳升高，loss平稳降低，在step为16k时收敛，precision达到0.97，loss降至0.01。\n",
    "\n",
    "## 参考文献\n",
    "<b>相关论文</b>\n",
    "\n",
    "Going Deeper with Convolutions\n",
    "https://arxiv.org/abs/1409.4842\n",
    "\n",
    "<b>参考代码</b>\n",
    "https://github.com/Marsan-Ma/imgrec\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}