01-1. 標準神經網路做手寫辨識（MNIST）.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 主題 01-1. 標準神經網路做手寫辨識\n",
    "\n",
    "我們終於要開始做生命中第一個神經網路..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 初始準備\n",
    "\n",
    "Keras 可以用各種不同的深度學習套件當底層, 我們在此指定用 Tensorflow 以確保執行的一致性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: KERAS_BACKEND=tensorflow\n"
     ]
    }
   ],
   "source": [
    "%env KERAS_BACKEND=tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再來是我們標準數據分析動作!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 讀入 MNIST 數據庫\n",
    "\n",
    "MNIST 是有一堆 0-9 的手寫數字圖庫。有 6 萬筆訓練資料, 1 萬筆測試資料。它是 \"Modified\" 版的 NIST 數據庫, 原來的版本有更多資料。這個 Modified 的版本是由 LeCun, Cortes, 及 Burges 等人做的。可以參考這個數據庫的[原始網頁](http://yann.lecun.com/exdb/mnist/)。\n",
    "\n",
    "MNIST 可以說是 Deep Learning 最有名的範例, 它被 Deep Learning 大師 Hinton 稱為「機器學習的果蠅」。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 由 Keras 讀入 MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras 很貼心的幫我們準備好 MNIST 數據庫, 我們可以這樣讀進來 (第一次要花點時間)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們來看看訓練資料是不是 6 萬筆、測試資料是不是有 1 筆。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特別要注意的是, 萬一在讀的過程中失敗, 你需要找到下載的部份數據集刪去, 然後在一個網路通𣈱的地方再下載一次。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 數據庫的內容\n",
    "\n",
    "每筆輸入 (x) 就是一個手寫的 0-9 中一個數字的圖檔, 大小為 28x28。而輸出 (y) 當然就是「正確答案」。我們來看看編號 9487 的訓練資料。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28, 28)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[9487].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因為是圖檔, 當然可以顯示出來!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<matplotlib.image.AxesImage at 0x125010ba8>"
=======
       "<matplotlib.image.AxesImage at 0x7f2a391f4e80>"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADI1JREFUeJzt3W+oXPWdx/HPR5OCSasYctVgdW+36GJQNl2GsKAsLuUWswRjH1QasGS17O2DKFsssiJCfaAgy9puhU0lXUMTbdMWWtc8kN0GWXCDS3AMkthkd6tyt80mJDeoSQqBovnug3tSbuKdM5OZ82duvu8XhJk5vzP3fJjkkzMzv7nzc0QIQD6XtR0AQDsoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpJY0ebCVK1fG5ORkk4cEUpmZmdGJEyc8yL4jld/2XZK+J+lySf8cEU+X7T85OalutzvKIQGU6HQ6A+879NN+25dL+idJ6yStlrTR9uphfx6AZo3ymn+tpHci4r2I+L2kn0jaUE0sAHUbpfzXS/rtvNuHi23nsT1tu2u7Ozs7O8LhAFRplPIv9KbCJ34/OCK2RkQnIjoTExMjHA5AlUYp/2FJN8y7/VlJR0aLA6Apo5T/DUk32f6c7U9J+qqkXdXEAlC3oaf6IuIj2w9K+jfNTfVti4hfVZYMQK1GmuePiFckvVJRFgAN4uO9QFKUH0iK8gNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSVF+ICnKDyTV6BLdaN6KFStKx0+ePFk6vnnz5tLxZ5999qIzYTxw5geSovxAUpQfSIryA0lRfiApyg8kRfmBpEaa57c9I+m0pI8lfRQRnSpC4eJ8+OGHPcciovS+tkvHt2zZUjrOPP/iVcWHfP4yIk5U8HMANIin/UBSo5Y/JP3S9pu2p6sIBKAZoz7tvz0ijti+RtJu2/8VEa/N36H4T2Fakm688cYRDwegKiOd+SPiSHF5XNJLktYusM/WiOhERGdiYmKUwwGo0NDlt73c9mfOXZf0JUlvVxUMQL1Gedp/raSXiqmiJZJ+HBH/WkkqALUbuvwR8Z6kP60wC3roN1f/3HPP9Rw7ffp01XHOs3v37tLxqampWo+P4THVByRF+YGkKD+QFOUHkqL8QFKUH0iKr+5eBE6dOlU6/vjjjzeU5JP27NlTOs5U3/jizA8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSTHPvwgsXbq0dHz16tU9xw4ePFh1HFwiOPMDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFLM8y8Cy5YtKx2///77e4498sgjVcc5z86dO0vHH3744Z5jV111VdVxcBE48wNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUn3n+W1vk7Re0vGIuLXYtkLSTyVNSpqRdG9EfFBfTIyrd999t3T8zJkzPceY52/XIGf+H0q664Jtj0p6NSJukvRqcRvAItK3/BHxmqT3L9i8QdL24vp2SfdUnAtAzYZ9zX9tRByVpOLymuoiAWhC7W/42Z623bXdnZ2drftwAAY0bPmP2V4lScXl8V47RsTWiOhERGdiYmLIwwGo2rDl3yVpU3F9k6SXq4kDoCl9y297p6T/lPQntg/b/rqkpyVN2f61pKniNoBFpO88f0Rs7DH0xYqzYEgPPfRQz7Enn3yy9L4nT56sOg4WCT7hByRF+YGkKD+QFOUHkqL8QFKUH0iKr+6+BCxZ0vuv0XaDSbCYcOYHkqL8QFKUH0iK8gNJUX4gKcoPJEX5gaQoP5AU5QeSovxAUpQfSIryA0lRfiApyg8kRfmBpPh9/kvc+vXrS8dfeOGFkX7+2bNnS8f37t3bc2zDhg0jHRuj4cwPJEX5gaQoP5AU5QeSovxAUpQfSIryA0n1nee3vU3SeknHI+LWYtsTkv5G0myx22MR8UpdITG8Bx54oHT8xRdfHOnnX3ZZ+fnjmWee6Tk2NTVVet9ly5YNlQmDGeTM/0NJdy2w/bsRsab4Q/GBRaZv+SPiNUnvN5AFQINGec3/oO39trfZvrqyRAAaMWz5vy/p85LWSDoqqecLO9vTtru2u7Ozs712A9CwocofEcci4uOIOCvpB5LWluy7NSI6EdGZmJgYNieAig1Vftur5t38sqS3q4kDoCmDTPXtlHSnpJW2D0v6tqQ7ba+RFJJmJH2jxowAatC3/BGxcYHNz9eQBZeg119/vefYvn37Su97xx13VB0H8/AJPyApyg8kRfmBpCg/kBTlB5Ki/EBSlB9IivIDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFIs0X2Ju+6660rH+327Up1fvTY9PV06fvDgwdqODc78QFqUH0iK8gNJUX4gKcoPJEX5gaQoP5AU8/yXuJtvvrl0fN26daXjO3bsqDLOeVi+rV2c+YGkKD+QFOUHkqL8QFKUH0iK8gNJUX4gqb7z/LZvkLRD0nWSzkraGhHfs71C0k8lTUqakXRvRHxQX1TU4amnniodr3Oe/8yZM6XjBw4cKB2/7bbbqoyTziBn/o8kfSsibpH055I2214t6VFJr0bETZJeLW4DWCT6lj8ijkbEvuL6aUmHJF0vaYOk7cVu2yXdU1dIANW7qNf8ticlfUHSXknXRsRRae4/CEnXVB0OQH0GLr/tT0v6uaRvRsSpi7jftO2u7S6f5QbGx0Dlt71Uc8X/UUT8oth8zPaqYnyVpOML3TcitkZEJyI6/b4sEkBz+pbftiU9L+lQRHxn3tAuSZuK65skvVx9PAB1GeRXem+X9DVJB2y/VWx7TNLTkn5m++uSfiPpK/VERJ2WL19eOt5vOm3//v1DH/uKK64oHb/llluG/tnor2/5I2KPJPcY/mK1cQA0hU/4AUlRfiApyg8kRfmBpCg/kBTlB5Liq7uTu/LKK0vH77777tLxUeb5+1myhH+edeLMDyRF+YGkKD+QFOUHkqL8QFKUH0iK8gNJMZGKUvfdd1/p+AcflH9b+5YtW6qMgwpx5geSovxAUpQfSIryA0lRfiApyg8kRfmBpBwRjR2s0+lEt9tt7HhANp1OR91ut9dX7Z+HMz+QFOUHkqL8QFKUH0iK8gNJUX4gKcoPJNW3/LZvsP3vtg/Z/pXtvy22P2H7/2y/Vfz5q/rjAqjKIF/m8ZGkb0XEPtufkfSm7d3F2Hcj4h/qiwegLn3LHxFHJR0trp+2fUjS9XUHA1Cvi3rNb3tS0hck7S02PWh7v+1ttq/ucZ9p213b3dnZ2ZHCAqjOwOW3/WlJP5f0zYg4Jen7kj4vaY3mnhk8s9D9ImJrRHQiojMxMVFBZABVGKj8tpdqrvg/iohfSFJEHIuIjyPirKQfSFpbX0wAVRvk3X5Lel7SoYj4zrztq+bt9mVJb1cfD0BdBnm3/3ZJX5N0wPZbxbbHJG20vUZSSJqR9I1aEgKoxSDv9u+RtNDvB79SfRwATeETfkBSlB9IivIDSVF+ICnKDyRF+YGkKD+QFOUHkqL8QFKUH0iK8gNJUX4gKcoPJEX5gaQaXaLb9qyk/523aaWkE40FuDjjmm1cc0lkG1aV2f4oIgb6vrxGy/+Jg9vdiOi0FqDEuGYb11wS2YbVVjae9gNJUX4gqbbLv7Xl45cZ12zjmksi27Baydbqa34A7Wn7zA+gJa2U3/Zdtv/b9ju2H20jQy+2Z2wfKFYe7racZZvt47bfnrdthe3dtn9dXC64TFpL2cZi5eaSlaVbfezGbcXrxp/2275c0v9ImpJ0WNIbkjZGxMFGg/Rge0ZSJyJanxO2/ReSfidpR0TcWmz7e0nvR8TTxX+cV0fE341Jtick/a7tlZuLBWVWzV9ZWtI9kv5aLT52JbnuVQuPWxtn/rWS3omI9yLi95J+ImlDCznGXkS8Jun9CzZvkLS9uL5dc/94Gtcj21iIiKMRsa+4flrSuZWlW33sSnK1oo3yXy/pt/NuH9Z4Lfkdkn5p+03b022HWcC1xbLp55ZPv6blPBfqu3Jzky5YWXpsHrthVryuWhvlX2j1n3Gacrg9Iv5M0jpJm4untxjMQCs3N2WBlaXHwrArXletjfIflnTDvNuflXSkhRwLiogjxeVxSS9p/FYfPnZukdTi8njLef5gnFZuXmhlaY3BYzdOK163Uf43JN1k+3O2PyXpq5J2tZDjE2wvL96Ike3lkr6k8Vt9eJekTcX1TZJebjHLecZl5eZeK0ur5cdu3Fa8buVDPsVUxj9KulzStoh4qvEQC7D9x5o720tzi5j+uM1stndKulNzv/V1TNK3Jf2LpJ9JulHSbyR9JSIaf+OtR7Y7NffU9Q8rN597jd1wtjsk/YekA5LOFpsf09zr69Yeu5JcG9XC48Yn/ICk+IQfkBTlB5Ki/EBSlB9IivIDSVF+ICnKDyRF+YGk/h85G4rir6/+MAAAAABJRU5ErkJggg==\n",
      "text/plain": [
<<<<<<< HEAD
       "<matplotlib.figure.Figure at 0x121ae5278>"
=======
       "<Figure size 432x288 with 1 Axes>"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_train[9487], cmap='Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們人眼辨識就知道這是 1, 我們看答案是不是和我們想的一樣。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[9487]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 輸入格式整理\n",
    "\n",
    "我們現在要用標準神經網路學學手寫辨識。原來的每筆數據是個 28x28 的矩陣 (array), 但標準神經網路只吃「平平的」, 也就是每次要 28x28=784 長的向量。因此我們要用 `reshape` 調校一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(60000, 784)\n",
    "x_test = x_test.reshape(10000, 784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 輸出格式整理\n",
    "\n",
    "我們可能會想, 我們想學的函數是這樣的型式:\n",
    "\n",
    "$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}$$\n",
    "\n",
    "其實這樣不太好! 為什麼呢? 比如說我們的輸入 x 是一張 0 的圖, 因為我們訓練的神經網路總會有點誤差, 所以可能會得到:\n",
    "\n",
    "$$\\hat{f}(x) = 0.5$$\n",
    "\n",
    "那這意思是有可能是 0, 也有可能是 1 嗎!!?? 可是 0 和 1 根本不像啊。換句話說分類的問題這樣做其實不合理!\n",
    "\n",
    "於是我們會做 \"1-hot enconding\", 也就是\n",
    "\n",
    "* 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0]\n",
    "* 5 -> [0, 0, 0, 0, 0, 1, 0, 0, 0]\n",
    "\n",
    "等等。因為分類問題基本上都要做這件事, Keras 其實已幫我們準備好套件!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np_utils.to_categorical(y_train,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test = np_utils.to_categorical(y_test,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們來看看剛剛是 1 的 9487 號數據的答案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[9487]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "和我們想的一樣! 至此我們可以打造我們的神經網路了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 打造第一個神經網路\n",
    "\n",
    "我們決定了我們的函數是\n",
    "\n",
    "$$\\hat{f} \\colon \\mathbb{R}^{784} \\to \\mathbb{R}^{10}$$\n",
    "\n",
    "這個樣子。而我們又說第一次要用標準神網路試試, 所以我們只需要再決定要幾個隱藏層、每層要幾個神經元, 用哪個激發函數就可以了。\n",
    "\n",
    "### 3.1 決定神經網路架構、讀入相關套件\n",
    "\n",
    "假如我們要這麼做:\n",
    "\n",
    "* 使用 <span style=\"color:red;\">2</span> 個 hidden layers\n",
    "* 每個 hidden layer 用 <span style=\"color:red;\">500</span> 個神經元\n",
    "* Activation Function 唯一指名 <span style=\"color:red;\">sigmoid</span>\n",
    "\n",
    "於是從 Keras 把相關套件讀進來。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 建構我們的神經網路\n",
    "\n",
    "和以前做迴歸或機器學習一樣, 我們就打開個「函數學習機」。標準一層一層傳遞的神經網路叫 `Sequential`, 於是我們打開一個空的神經網路。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們每次用 `add` 去加一層, 從第一個隱藏層開始。而第一個隱藏層因為 Keras 當然猜不到輸入有幾個 features, 所以我們要告訴它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(500, input_dim=784))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二層 hidden layer 因為前面輸出是 500, 現在輸入是 500, 就不用再說了! 這裡的 500 只告訴 Keras, 我們第二層還是用 500!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(500))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "輸出有 10 個數字, 所以輸出層的神經元是 10 個! 而如果我們的網路輸出是 \n",
    "\n",
    "$$(y_1, y_2, \\ldots, y_{10})$$\n",
    "\n",
    "我們還希望\n",
    "\n",
    "$$\\sum_{i=1}^{10} y_i = 1$$\n",
    "\n",
    "這可能嗎, 結果是很容易, 就用 `softmax` 當激發函數就可以!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我們的第一個神經網路就建好了!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 組裝\n",
    "\n",
    "和之前比較不一樣的是我們還要做 `compile` 才正式把我們的神經網路建好。你可以發現我們還需要做幾件事:\n",
    "\n",
    "* 決定使用的 loss function, 一般是 `mse`\n",
    "* 決定 optimizer, 我們用標準的 SGD\n",
    "* 設 learning rate\n",
    "\n",
    "為了一邊訓練一邊看到結果, 我們加設\n",
    "\n",
    "    metrics=['accuracy']\n",
    "    \n",
    "本行基本上和我們的神經網路功能沒有什麼關係。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='mse', optimizer=SGD(lr=0.1), metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 檢視我們的神經網路"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以檢視我們神經網路的架構, 可以確認一下是不是和我們想像的一樣。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 看 model 的 summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                5010      \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 648,010\n",
      "Trainable params: 648,010\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 畫出結構圖\n",
    "\n",
    "要使用這個功能要安裝 `pydot` 及 `graphviz` 兩個套件, 請在終端機 (Anaconda Prompt) 安裝:\n",
    "\n",
    "    conda install pydot\n",
    "    conda install graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import plot_model\n",
    "plot_model(model, show_shapes=True, to_file='model01.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![我的神經網路](model01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 訓練你的第一個神經網路\n",
    "\n",
    "恭喜! 我們完成了第一個神經網路。現在要訓練的時候, 你會發現不是像以前沒頭沒腦把訓練資料送進去就好。這裡我們還有兩件事要決定:\n",
    "\n",
    "* 一次要訓練幾筆資料 (`batch_size`), 我們就 100 筆調一次參數好了\n",
    "* 這 6 萬筆資料一共要訓練幾次 (`epochs`), 我們訓練個 20 次試試\n",
    "\n",
    "於是最精彩的就來了。你要有等待的心理準備..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
<<<<<<< HEAD
      "Epoch 1/2\n",
      "60000/60000 [==============================] - 28s 466us/step - loss: 0.0831 - acc: 0.3281\n",
      "Epoch 2/2\n",
      "60000/60000 [==============================] - 28s 462us/step - loss: 0.0637 - acc: 0.6190\n"
=======
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 1s 23us/step - loss: 0.0832 - acc: 0.3194\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0634 - acc: 0.5955\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0460 - acc: 0.7660\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0334 - acc: 0.8482\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0257 - acc: 0.8769\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0213 - acc: 0.8908\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0186 - acc: 0.9003\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0168 - acc: 0.9071\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0155 - acc: 0.9129\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0144 - acc: 0.9177\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0136 - acc: 0.9214\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0129 - acc: 0.9250\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0123 - acc: 0.9281\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0118 - acc: 0.9314\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0113 - acc: 0.9337\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0109 - acc: 0.9364\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0105 - acc: 0.9390\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 1s 17us/step - loss: 0.0101 - acc: 0.9414\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0098 - acc: 0.9428\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 1s 16us/step - loss: 0.0095 - acc: 0.9447\n"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
     ]
    },
    {
     "data": {
      "text/plain": [
<<<<<<< HEAD
       "<keras.callbacks.History at 0x12d8a8cc0>"
=======
       "<keras.callbacks.History at 0x7f2a38621710>"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, batch_size=100, epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 試用我們的結果\n",
    "\n",
    "我們來用比較炫的方式來看看可愛的神經網路學習成果。對指令有問題可以參考我們之前的 MOOC 影片教學。"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
=======
   "execution_count": 24,
   "metadata": {},
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "outputs": [],
   "source": [
    "from ipywidgets import interact_manual"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們 \"predict\" 放的是我們神經網路的學習結果。這裡用 `predict_classes` 會讓我們 Keras 選 10 個輸出機率最大的那類。"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 3s 293us/step\n"
     ]
    }
   ],
=======
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "source": [
    "predict = model.predict_classes(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不要忘了我們的 `x_test` 每筆資料已經換成 784 維的向量, 我們要整型回 28x28 的矩陣才能當成圖形顯示出來!"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
=======
   "execution_count": 26,
   "metadata": {},
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "outputs": [],
   "source": [
    "def test(測試編號):\n",
    "    plt.imshow(x_test[測試編號].reshape(28,28), cmap=\"Greys\")\n",
    "    print(\"神經網路判斷為:\", predict[測試編號])"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 26,
=======
   "execution_count": 27,
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
<<<<<<< HEAD
       "model_id": "daf7379f90a9433a8c5b4a9e3c7dc0f6",
=======
       "model_id": "a2f6ef52950849928ba77c3e053ea848",
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
<<<<<<< HEAD
       "A Jupyter Widget"
=======
       "interactive(children=(IntSlider(value=4999, description='測試編號', max=9999), Button(description='Run Interact', …"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "interact_manual(test, 測試編號 = (0, 9999));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到底測試資料總的狀況如何呢? 我們可以給我們神經網路「考一下試」。"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 27,
=======
   "execution_count": 28,
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
<<<<<<< HEAD
      "10000/10000 [==============================] - 3s 341us/step\n"
=======
      "10000/10000 [==============================] - 0s 20us/step\n"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
<<<<<<< HEAD
   "execution_count": 28,
=======
   "execution_count": 29,
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
<<<<<<< HEAD
      "測試資料的 loss: 0.0537684477806\n",
      "測試資料正確率: 0.7196\n"
=======
      "測試資料的 loss: 0.010567067707702518\n",
      "測試資料正確率: 0.9338\n"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
     ]
    }
   ],
   "source": [
    "print('測試資料的 loss:', score[0])\n",
    "print('測試資料正確率:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 訓練好的神經網路存起來!\n",
    "\n",
    "如果對訓練成果滿意, 我們當然不想每次都再訓練一次! 我們可以把神經網路的架構和訓練好的參數都存起來, 以供日後使用!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之前還沒裝 pyh5 要在終端機 (Anaconda Prompt) 下安裝:\n",
    "    \n",
    "    conda install h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_json = model.to_json()\n",
    "open('handwriting_model_architecture.json', 'w').write(model_json)\n",
    "model.save_weights('handwriting_model_weights.h5')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}