3.4-classifying-movie-reviews.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 영화 리뷰 분류: 이진 분류 예제\n",
    "\n",
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝](https://tensorflow.blog/%EC%BC%80%EB%9D%BC%EC%8A%A4-%EB%94%A5%EB%9F%AC%EB%8B%9D/) 책의 3장 4절의 코드 예제입니다. 책에는 더 많은 내용과 그림이 있습니다. 이 노트북에는 소스 코드에 관련된 설명만 포함합니다.\n",
    "\n",
    "----\n",
    "\n",
    "2종 분류 또는 이진 분류는 아마도 가장 널리 적용된 머신 러닝 문제일 것입니다. 이 예제에서 리뷰 텍스트를 기반으로 영화 리뷰를 긍정과 부정로 분류하는 법을 배우겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDB 데이터셋\n",
    "\n",
    "인터넷 영화 데이터베이스로부터 가져온 양극단의 리뷰 50,000개로 이루어진 IMDB 데이터셋을 사용하겠습니다. 이 데이터셋은 훈련 데이터 25,000개와 테스트 데이터 25,000개로 나뉘어 있고 각각 50%는 부정, 50%는 긍정 리뷰로 구성되어 있습니다.\n",
    "\n",
    "왜 훈련 데이터와 테스트 데이터를 나눌까요? 같은 데이터에서 머신 러닝 모델을 훈련하고 테스트해서는 절대 안 되기 때문입니다! 모델이 훈련 데이터에서 잘 작동한다는 것이 처음 만난 데이터에서도 잘 동작한다는 것을 보장하지 않습니다. 중요한 것은 새로운 데이터에 대한 모델의 성능입니다(사실 훈련 데이터의 레이블은 이미 알고 있기 때문에 이를 예측하는 모델은 필요하지 않습니다). 예를 들어 모델이 훈련 샘플과 타깃 사이의 매핑을 모두 외워버릴 수 있습니다. 이런 모델은 처음 만나는 데이터에서 타깃을 예측하는 작업에는 쓸모가 없습니다. 다음 장에서 이에 대해 더 자세히 살펴보겠습니다.\n",
    "\n",
    "MNIST 데이터셋처럼 IMDB 데이터셋도 케라스에 포함되어 있습니다. 이 데이터는 전처리되어 있어 각 리뷰(단어 시퀀스)가 숫자 시퀀스로 변환되어 있습니다. 여기서 각 숫자는 사전에 있는 고유한 단어를 나타냅니다.\n",
    "\n",
    "다음 코드는 데이터셋을 로드합니다(처음 실행하면 17MB 정도의 데이터가 컴퓨터에 다운로드됩니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "매개변수 `num_words=10000`은 훈련 데이터에서 가장 자주 나타나는 단어 10,000개만 사용하겠다는 의미입니다. 드물게 나타나는 단어는 무시하겠습니다. 이렇게 하면 적절한 크기의 벡터 데이터를 얻을 수 있습니다.\n",
    "\n",
    "변수 `train_data`와 `test_data`는 리뷰의 목록입니다. 각 리뷰는 단어 인덱스의 리스트입니다(단어 시퀀스가 인코딩된 것입니다). `train_labels`와 `test_labels`는 부정을 나타내는 0과 긍정을 나타내는 1의 리스트입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 14,\n",
       " 22,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 973,\n",
       " 1622,\n",
       " 1385,\n",
       " 65,\n",
       " 458,\n",
       " 4468,\n",
       " 66,\n",
       " 3941,\n",
       " 4,\n",
       " 173,\n",
       " 36,\n",
       " 256,\n",
       " 5,\n",
       " 25,\n",
       " 100,\n",
       " 43,\n",
       " 838,\n",
       " 112,\n",
       " 50,\n",
       " 670,\n",
       " 2,\n",
       " 9,\n",
       " 35,\n",
       " 480,\n",
       " 284,\n",
       " 5,\n",
       " 150,\n",
       " 4,\n",
       " 172,\n",
       " 112,\n",
       " 167,\n",
       " 2,\n",
       " 336,\n",
       " 385,\n",
       " 39,\n",
       " 4,\n",
       " 172,\n",
       " 4536,\n",
       " 1111,\n",
       " 17,\n",
       " 546,\n",
       " 38,\n",
       " 13,\n",
       " 447,\n",
       " 4,\n",
       " 192,\n",
       " 50,\n",
       " 16,\n",
       " 6,\n",
       " 147,\n",
       " 2025,\n",
       " 19,\n",
       " 14,\n",
       " 22,\n",
       " 4,\n",
       " 1920,\n",
       " 4613,\n",
       " 469,\n",
       " 4,\n",
       " 22,\n",
       " 71,\n",
       " 87,\n",
       " 12,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 38,\n",
       " 76,\n",
       " 15,\n",
       " 13,\n",
       " 1247,\n",
       " 4,\n",
       " 22,\n",
       " 17,\n",
       " 515,\n",
       " 17,\n",
       " 12,\n",
       " 16,\n",
       " 626,\n",
       " 18,\n",
       " 2,\n",
       " 5,\n",
       " 62,\n",
       " 386,\n",
       " 12,\n",
       " 8,\n",
       " 316,\n",
       " 8,\n",
       " 106,\n",
       " 5,\n",
       " 4,\n",
       " 2223,\n",
       " 5244,\n",
       " 16,\n",
       " 480,\n",
       " 66,\n",
       " 3785,\n",
       " 33,\n",
       " 4,\n",
       " 130,\n",
       " 12,\n",
       " 16,\n",
       " 38,\n",
       " 619,\n",
       " 5,\n",
       " 25,\n",
       " 124,\n",
       " 51,\n",
       " 36,\n",
       " 135,\n",
       " 48,\n",
       " 25,\n",
       " 1415,\n",
       " 33,\n",
       " 6,\n",
       " 22,\n",
       " 12,\n",
       " 215,\n",
       " 28,\n",
       " 77,\n",
       " 52,\n",
       " 5,\n",
       " 14,\n",
       " 407,\n",
       " 16,\n",
       " 82,\n",
       " 2,\n",
       " 8,\n",
       " 4,\n",
       " 107,\n",
       " 117,\n",
       " 5952,\n",
       " 15,\n",
       " 256,\n",
       " 4,\n",
       " 2,\n",
       " 7,\n",
       " 3766,\n",
       " 5,\n",
       " 723,\n",
       " 36,\n",
       " 71,\n",
       " 43,\n",
       " 530,\n",
       " 476,\n",
       " 26,\n",
       " 400,\n",
       " 317,\n",
       " 46,\n",
       " 7,\n",
       " 4,\n",
       " 2,\n",
       " 1029,\n",
       " 13,\n",
       " 104,\n",
       " 88,\n",
       " 4,\n",
       " 381,\n",
       " 15,\n",
       " 297,\n",
       " 98,\n",
       " 32,\n",
       " 2071,\n",
       " 56,\n",
       " 26,\n",
       " 141,\n",
       " 6,\n",
       " 194,\n",
       " 7486,\n",
       " 18,\n",
       " 4,\n",
       " 226,\n",
       " 22,\n",
       " 21,\n",
       " 134,\n",
       " 476,\n",
       " 26,\n",
       " 480,\n",
       " 5,\n",
       " 144,\n",
       " 30,\n",
       " 5535,\n",
       " 18,\n",
       " 51,\n",
       " 36,\n",
       " 28,\n",
       " 224,\n",
       " 92,\n",
       " 25,\n",
       " 104,\n",
       " 4,\n",
       " 226,\n",
       " 65,\n",
       " 16,\n",
       " 38,\n",
       " 1334,\n",
       " 88,\n",
       " 12,\n",
       " 16,\n",
       " 283,\n",
       " 5,\n",
       " 16,\n",
       " 4472,\n",
       " 113,\n",
       " 103,\n",
       " 32,\n",
       " 15,\n",
       " 16,\n",
       " 5345,\n",
       " 19,\n",
       " 178,\n",
       " 32]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 자주 등장하는 단어 10,000개로 제한했기 때문에 단어 인덱스는 10,000을 넘지 않습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9999"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max([max(sequence) for sequence in train_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "재미 삼아 이 리뷰 데이터 하나를 원래 영어 단어로 어떻게 바꾸는지 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word_index는 단어와 정수 인덱스를 매핑한 딕셔너리입니다\n",
    "word_index = imdb.get_word_index()\n",
    "# 정수 인덱스와 단어를 매핑하도록 뒤집습니다\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# 리뷰를 디코딩합니다. \n",
    "# 0, 1, 2는 '패딩', '문서 시작', '사전에 없음'을 위한 인덱스이므로 3을 뺍니다\n",
    "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 데이터 준비\n",
    "\n",
    "신경망에 숫자 리스트를 주입할 수는 없습니다. 리스트를 텐서로 바꾸는 두 가지 방법이 있습니다:\n",
    "\n",
    "* 같은 길이가 되도록 리스트에 패딩을 추가하고 `(samples, sequence_length)` 크기의 정수 텐서로 변환합니다. 그다음 이 정수 텐서를 다룰 수 있는 층을 신경망의 첫 번째 층으로 사용합니다(`Embedding` 층을 말하며 나중에 자세히 다루겠습니다).\n",
    "* 리스트를 원-핫 인코딩하여 0과 1의 벡터로 변환합니다. 예를 들면 시퀀스 `[3, 5]`를 인덱스 3과 5의 위치는 1이고 그 외는 모두 0인 10,000차원의 벡터로 각각 변환합니다. 그다음 부동 소수 벡터 데이터를 다룰 수 있는 `Dense` 층을 신경망의 첫 번째 층으로 사용합니다.\n",
    "\n",
    "여기서는 두 번째 방식을 사용하고 이해를 돕기 위해 직접 데이터를 원-핫 벡터로 만들겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # 크기가 (len(sequences), dimension))이고 모든 원소가 0인 행렬을 만듭니다\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # results[i]에서 특정 인덱스의 위치를 1로 만듭니다\n",
    "    return results\n",
    "\n",
    "# 훈련 데이터를 벡터로 변환합니다\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 테스트 데이터를 벡터로 변환합니다\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 샘플은 다음과 같이 나타납니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "레이블은 쉽게 벡터로 바꿀 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 레이블을 벡터로 바꿉니다\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 신경망에 주입할 데이터가 준비되었습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 신경망 모델 만들기\n",
    "\n",
    "입력 데이터가 벡터이고 레이블은 스칼라(1 또는 0)입니다. 아마 앞으로 볼 수 있는 문제 중에서 가장 간단할 것입니다. 이런 문제에 잘 작동하는 네트워크 종류는 `relu` 활성화 함수를 사용한 완전 연결 층(즉, `Dense(16, activation='relu')`)을 그냥 쌓은 것입니다.\n",
    "\n",
    "`Dense` 층에 전달한 매개변수(16)는 은닉 유닛의 개수입니다. 하나의 은닉 유닛은 층이 나타내는 표현 공간에서 하나의 차원이 됩니다. 2장에서 `relu` 활성화 함수를 사용한 `Dense` 층을 다음과 같은 텐서 연산을 연결하여 구현하였습니다:\n",
    "\n",
    "`output = relu(dot(W, input) + b)`\n",
    "\n",
    "16개의 은닉 유닛이 있다는 것은 가중치 행렬 `W`의 크기가 `(input_dimension, 16)`이라는 뜻입니다. 입력 데이터와 `W`를 점곱하면 입력 데이터가 16 차원으로 표현된 공간으로 투영됩니다(그리고 편향 벡터 `b`를 더하고 `relu` 연산을 적용합니다). 표현 공간의 차원을 '신경망이 내재된 표현을 학습할 때 가질 수 있는 자유도'로 이해할 수 있습니다. 은닉 유닛을 늘리면 (표현 공간을 더 고차원으로 만들면) 신경망이 더욱 복잡한 표현을 학습할 수 있지만 계산 비용이 커지고 원치 않은 패턴을 학습할 수도 있습니다(훈련 데이터에서는 성능이 향상되지만 테스트 데이터에서는 그렇지 않은 패턴입니다).\n",
    "\n",
    "`Dense` 층을 쌓을 때 두 가진 중요한 구조상의 결정이 필요합니다:\n",
    "\n",
    "* 얼마나 많은 층을 사용할 것인가\n",
    "* 각 층에 얼마나 많은 은닉 유닛을 둘 것인가\n",
    "\n",
    "4장에서 이런 결정을 하는 데 도움이 되는 일반적인 원리를 배우겠습니다. 당분간은 저를 믿고 선택한 다음 구조를 따라 주세요.\n",
    "\n",
    "* 16개의 은닉 유닛을 가진 두 개의 은닉층\n",
    "* 현재 리뷰의 감정을 스칼라 값의 예측으로 출력하는 세 번째 층\n",
    "\n",
    "중간에 있는 은닉층은 활성화 함수로 `relu`를 사용하고 마지막 층은 확률(0과 1 사이의 점수로, 어떤 샘플이 타깃 '1'일 가능성이 높다는 것은 그 리뷰가 긍정일 가능성이 높다는 것을 의미합니다)을 출력하기 위해 시그모이드 활성화 함수를 사용합니다. `relu`는 음수를 0으로 만드는 함수입니다. 시그모이드는 임의의 값을 [0, 1] 사이로 압축하므로 출력 값을 확률처럼 해석할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음이 이 신경망의 모습입니다:\n",
    "\n",
    "![3-layer network](https://s3.amazonaws.com/book.keras.io/img/ch3/3_layer_network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음은 이 신경망의 케라스 구현입니다. 이전에 보았던 MNIST 예제와 비슷합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 손실 함수와 옵티마이저를 선택해야 합니다. 이진 분류 문제이고 신경망의 출력이 확률이기 때문에(네트워크의 끝에 시그모이드 활성화 함수를 사용한 하나의 유닛으로 된 층을 놓았습니다), `binary_crossentropy` 손실이 적합합니다. 이 함수가 유일한 선택은 아니고 예를 들어 `mean_squared_error`를 사용할 수도 있습니다. 확률을 출력하는 모델을 사용할 때는 크로스엔트로피가 최선의 선택입니다. 크로스엔트로피는 정보 이론 분야에서 온 개념으로 확률 분포 간의 차이를 측정합니다. 여기에서는 원본 분포와 예측 분포 사이를 측정합니다.\n",
    "\n",
    "다음은 `rmsprop` 옵티마이저와 `binary_crossentropy` 손실 함수로 모델을 설정하는 단계입니다. 훈련하는 동안 정확도를 사용해 모니터링하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "케라스에 `rmsprop`, `binary_crossentropy`, `accuracy`가 포함되어 있기 때문에 옵티마이저, 손실 함수, 측정 지표를 문자열로 지정하는 것이 가능합니다. 이따금 옵티마이저의 매개변수를 바꾸거나 자신만의 손실 함수, 측정 함수를 전달해야 할 경우가 있습니다. 전자의 경우에는 옵티마이저 파이썬 클래스를 사용해 객체를 직접 만들어 `optimizer` 매개변수에 전달하면 됩니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "후자의 경우는 `loss`와 `metrics` 매개변수에 함수 객체를 전달하면 됩니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import losses\n",
    "from keras import metrics\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss=losses.binary_crossentropy,\n",
    "              metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련 검증\n",
    "\n",
    "훈련하는 동안 처음 본 데이터에 대한 모델의 정확도를 측정하기 위해서는 원본 훈련 데이터에서 10,000의 샘플을 떼어서 검증 세트를 만들어야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "이제 모델을 512개 샘플씩 미니 배치를 만들어 20번의 에포크 동안 훈련시킵니다(`x_train`과 `y_train` 텐서에 있는 모든 샘플에 대해 20번 반복합니다). 동시에 따로 떼어 놓은 10,000개의 샘플에서 손실과 정확도를 측정할 것입니다. 이렇게 하려면 `validation_data` 매개변수에 검증 데이터를 전달해야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "15000/15000 [==============================] - 2s 119us/step - loss: 0.5084 - acc: 0.7813 - val_loss: 0.3797 - val_acc: 0.8684\n",
      "Epoch 2/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.3004 - acc: 0.9047 - val_loss: 0.3004 - val_acc: 0.8897\n",
      "Epoch 3/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.2179 - acc: 0.9285 - val_loss: 0.3085 - val_acc: 0.8711\n",
      "Epoch 4/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.1750 - acc: 0.9437 - val_loss: 0.2840 - val_acc: 0.8832\n",
      "Epoch 5/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.1427 - acc: 0.9543 - val_loss: 0.2841 - val_acc: 0.8872\n",
      "Epoch 6/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.1150 - acc: 0.9650 - val_loss: 0.3166 - val_acc: 0.8772\n",
      "Epoch 7/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0980 - acc: 0.9705 - val_loss: 0.3127 - val_acc: 0.8846\n",
      "Epoch 8/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0807 - acc: 0.9763 - val_loss: 0.3859 - val_acc: 0.8649\n",
      "Epoch 9/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0661 - acc: 0.9821 - val_loss: 0.3635 - val_acc: 0.8782\n",
      "Epoch 10/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0561 - acc: 0.9853 - val_loss: 0.3843 - val_acc: 0.8792\n",
      "Epoch 11/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0439 - acc: 0.9893 - val_loss: 0.4153 - val_acc: 0.8779\n",
      "Epoch 12/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0381 - acc: 0.9921 - val_loss: 0.4525 - val_acc: 0.8690\n",
      "Epoch 13/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0300 - acc: 0.9928 - val_loss: 0.4698 - val_acc: 0.8729\n",
      "Epoch 14/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0247 - acc: 0.9945 - val_loss: 0.5023 - val_acc: 0.8726\n",
      "Epoch 15/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0175 - acc: 0.9979 - val_loss: 0.5342 - val_acc: 0.8693\n",
      "Epoch 16/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0149 - acc: 0.9983 - val_loss: 0.5710 - val_acc: 0.8697\n",
      "Epoch 17/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0151 - acc: 0.9971 - val_loss: 0.6024 - val_acc: 0.8697\n",
      "Epoch 18/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0075 - acc: 0.9996 - val_loss: 0.6789 - val_acc: 0.8632\n",
      "Epoch 19/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0118 - acc: 0.9975 - val_loss: 0.6698 - val_acc: 0.8680\n",
      "Epoch 20/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0041 - acc: 0.9999 - val_loss: 0.6931 - val_acc: 0.8654\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CPU를 사용해도 에포크마다 2초가 걸리지 않습니다. 전체 훈련은 20초 이상 걸립니다. 에포크가 끝날 때마다 10,000개의 검증 샘플 데이터에서 손실과 정확도를 계산하기 때문에 약간씩 지연됩니다.\n",
    "\n",
    "`model.fit()` 메서드는 `History` 객체를 반환합니다. 이 객체는 훈련하는 동안 발생한 모든 정보를 담고 있는 딕셔너리인 `history` 속성을 가지고 있습니다. 한 번 확인해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 딕셔너리는 훈련과 검증하는 동안 모니터링할 측정 지표당 하나씩 모두 네 개의 항목을 담고 있습니다. 맷플롯립을 사용해 훈련과 검증 데이터에 대한 손실과 정확도를 그려 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYFNXZ9/Hv7bCJICBgVJBFw6MCIowjmogCanzBBdQQAcFdCRoSjTER1xgiEY1RgiEmmEdiZARcoqJRyBM3XJFBEQREEEFGEIEIiKA4cL9/nBpshp5hlq7p7pnf57r6mq6q09V3V/f03efUqXPM3REREck0e6U7ABERkWSUoEREJCMpQYmISEZSghIRkYykBCUiIhlJCUpERDKSEpRkJDPLMbPNZtYmlWXTycy+a2Ypv67DzE4xs+UJy4vN7ITylK3Ec/3NzG6o7OPL2O9tZvb3VO9XsluddAcgNYOZbU5YbAh8DWyPln/s7vkV2Z+7bwcapbpsbeDuh6ViP2Z2GTDU3Xsl7PuyVOxbpDyUoCQl3H1ngoh+oV/m7v8prbyZ1XH3ouqITUSyk5r4pFpETThTzWyymX0BDDWz75nZm2a2wcxWm9k4M6sbla9jZm5m7aLlSdH258zsCzN7w8zaV7RstL2vmX1gZhvN7F4ze83MLiol7vLE+GMzW2pmn5vZuITH5pjZPWa23sw+BPqUcXxuMrMpJdaNN7O7o/uXmdmi6PV8GNVuSttXoZn1iu43NLOHotgWAEcned5l0X4XmFm/aP2RwJ+AE6Lm03UJx/bWhMcPj177ejN70swOLM+x2RMzOyuKZ4OZvWBmhyVsu8HMVpnZJjN7P+G1Hmdmb0fr15jZ78v7fJKh3F033VJ6A5YDp5RYdxuwDTiT8MNob+AY4FhCTf4Q4ANgRFS+DuBAu2h5ErAOyAPqAlOBSZUouz/wBdA/2nYN8A1wUSmvpTwxPgU0AdoB/y1+7cAIYAHQGmgOzAz/ckmf5xBgM7BPwr4/A/Ki5TOjMgacBGwFukTbTgGWJ+yrEOgV3b8LeAloBrQFFpYoey5wYPSenBfF8J1o22XASyXinATcGt0/NYqxK9AA+DPwQnmOTZLXfxvw9+j+EVEcJ0Xv0Q3Rca8LdAJWAAdEZdsDh0T3ZwODo/uNgWPT/b+gW9VuqkFJdXrV3Z929x3uvtXdZ7v7LHcvcvdlwASgZxmPf8zdC9z9GyCf8MVY0bJnAHPd/alo2z2EZJZUOWO83d03uvtyQjIofq5zgXvcvdDd1wNjynieZcB7hMQJ8ANgg7sXRNufdvdlHrwAPA8k7QhRwrnAbe7+ubuvINSKEp/3EXdfHb0nDxN+XOSVY78AQ4C/uftcd/8KGAn0NLPWCWVKOzZlGQRMc/cXovdoDLAv4YdCESEZdoqaiT+Kjh2EHxodzKy5u3/h7rPK+TokQylBSXVambhgZoeb2b/M7FMz2wSMAlqU8fhPE+5voeyOEaWVPSgxDnd3Qo0jqXLGWK7nIvzyL8vDwODo/nmExFocxxlmNsvM/mtmGwi1l7KOVbEDy4rBzC4ys3ejprQNwOHl3C+E17dzf+6+CfgcaJVQpiLvWWn73UF4j1q5+2LgF4T34bOoyfiAqOjFQEdgsZm9ZWanlfN1SIZSgpLqVLKL9V8JtYbvuvu+wC2EJqw4rSY0uQFgZsauX6glVSXG1cDBCct76gY/FTglqoH0JyQszGxv4DHgdkLzW1Pg3+WM49PSYjCzQ4D7gCuA5tF+30/Y7566xK8iNBsW768xoSnxk3LEVZH97kV4zz4BcPdJ7n48oXkvh3BccPfF7j6I0Iz7B+BxM2tQxVgkjZSgJJ0aAxuBL83sCODH1fCczwC5ZnammdUBrgJaxhTjI8DVZtbKzJoD15VV2N3XAK8CE4HF7r4k2lQfqAesBbab2RnAyRWI4QYza2rhOrERCdsaEZLQWkKuvoxQgyq2Bmhd3CkkicnApWbWxczqExLFK+5eao20AjH3M7Ne0XP/knDecJaZHWFmvaPn2xrdthNewPlm1iKqcW2MXtuOKsYiaaQEJen0C+BCwpfPXwk1iFhFSWAgcDewHjgUeIdw3VaqY7yPcK5oPuEE/mPleMzDhE4PDyfEvAH4OfAEoaPBAEKiLY9fE2pyy4HngH8k7HceMA54KypzOJB43ub/gCXAGjNLbKorfvx0QlPbE9Hj2xDOS1WJuy8gHPP7CMmzD9AvOh9VH7iTcN7wU0KN7abooacBiyz0Er0LGOju26oaj6SPhSZ4kdrJzHIITUoD3P2VdMcjIt9SDUpqHTPrY2ZNomaimwk9w95Kc1giUoISlNRGPYBlhGaiPsBZ7l5aE5+IpIma+EREJCOpBiUiIhkp6waLbdGihbdr1y7dYYiISCXNmTNnnbuXdXkHkIUJql27dhQUFKQ7DBERqSQz29OoKoCa+EREJEMpQYmISEZSghIRkYwU6zkoM+sD/JEwoOPf3H1Mie33AL2jxYbA/tGAlRXyzTffUFhYyFdffVXVkKUaNGjQgNatW1O3bmlDvImIxJigoiFkxhPmtSkEZpvZNHdfWFzG3X+eUP6nQLfKPFdhYSGNGzemXbt2hMGpJVO5O+vXr6ewsJD27dvv+QEiUmvF2cTXHVgaTbK2DZjCt5OxJTOYMDpyhX311Vc0b95cySkLmBnNmzdXbVdE9ijOBNWKXSdKK6SUeXfMrC1hbpcXStk+zMwKzKxg7dq1SZ9MySl76L0SkfKIM0El+xYqbVylQYQpurcn2+juE9w9z93zWrbc47VdIiISg6VLYcKE6nu+OBNUIbvO5NmaMK1BMoOoZPNeJli/fj1du3ala9euHHDAAbRq1Wrn8rZt5ZuO5uKLL2bx4sVllhk/fjz5+flllimvHj16MHfu3JTsS0Rqrs2b4e9/h549oUMHuPJK+HS32cHiEWcvvtlABzNrT5iqeRBwXslCZnYYYdKxN2KMZRf5+XDjjfDxx9CmDYweDUOqMM1a8+bNd37Z33rrrTRq1Ihrr712lzLujruz117JfxNMnDhxj8/zk5/8pPJBioiUkzu8/jo88AA88khIUh06wO23w/nnwwEHVE8csdWg3L2IML30DGAR8Ii7LzCzUWbWL6HoYGCKV9Ow6vn5MGwYrFgR3oQVK8Jyiiomu1i6dCmdO3dm+PDh5Obmsnr1aoYNG0ZeXh6dOnVi1KhRO8sW12iKiopo2rQpI0eO5KijjuJ73/sen332GQA33XQTY8eO3Vl+5MiRdO/encMOO4zXX38dgC+//JIf/vCHHHXUUQwePJi8vLw91pQmTZrEkUceSefOnbnhhhsAKCoq4vzzz9+5fty4cQDcc889dOzYkaOOOoqhQ4em/JiJSPqsWgVjxsDhh0OPHjB1Kpx7LrzyCixeDCNHQqukPQniEet1UO7+LPBsiXW3lFi+Nc4YSrrxRtiyZdd1W7aE9VWpRZVm4cKFTJw4kb/85S8AjBkzhv3224+ioiJ69+7NgAED6Nix4y6P2bhxIz179mTMmDFcc801PPDAA4wcOXK3fbs7b731FtOmTWPUqFFMnz6de++9lwMOOIDHH3+cd999l9zc3DLjKyws5KabbqKgoIAmTZpwyimn8Mwzz9CyZUvWrVvH/PnzAdiwYQMAd955JytWrKBevXo714lIfNasCU1qHTpAw4ap3/+2bfD006G2NH067NgBJ5wA118PAwZAo0apf87yqnUjSXz8ccXWV9Whhx7KMcccs3N58uTJ5Obmkpuby6JFi1i4cOFuj9l7773p27cvAEcffTTLly9Puu9zzjlntzKvvvoqgwYNAuCoo46iU6dOZcY3a9YsTjrpJFq0aEHdunU577zzmDlzJt/97ndZvHgxV111FTNmzKBJkyYAdOrUiaFDh5Kfn68LbUVitmhRSExdu8I++4RTEqecEs4DjR0Lzz4bOi4UFVV83+++C1dfDQcdFBLRu++GGtIHH8DMmXDRRelNTpCFo5lXVZs2oVkv2fo47LPPPjvvL1myhD/+8Y+89dZbNG3alKFDhya9HqhevXo77+fk5FBUyqevfv36u5WpaEtpaeWbN2/OvHnzeO655xg3bhyPP/44EyZMYMaMGbz88ss89dRT3Hbbbbz33nvk5ORU6DlFZM82boSzzoK994bx48P31gcfhNvkyZDYgFGnDhx6KPzP/+x6O+ywcL6o+MqO//4XHn4YJk6Et9+GevWgf3+45BL4wQ8g0/6Va12CGj06nHNKbOZr2DCsj9umTZto3Lgx++67L6tXr2bGjBn06dMnpc/Ro0cPHnnkEU444QTmz5+ftIaW6LjjjuOXv/wl69evp0mTJkyZMoVrr72WtWvX0qBBA370ox/Rvn17hg8fzvbt2yksLOSkk06iR48e5Ofns2XLFho3bpzS1yBS2+3YAUOHwrJl8MILocktkTusXx+S1eLF3yauDz6Af/8bvv7627KNGoVk1aIFvPRSaNLr1g3GjYPzzoPmzav1pVVIrUtQxeeZUtmLr7xyc3Pp2LEjnTt35pBDDuH4449P+XP89Kc/5YILLqBLly7k5ubSuXPnnc1zybRu3ZpRo0bRq1cv3J0zzzyT008/nbfffptLL70Ud8fMuOOOOygqKuK8887jiy++YMeOHVx33XVKTiIx+M1v4Jln4E9/2j05QagRtWgRbt///q7bduyAlSt3T16FhTB8OFx8cWgyzAZWTZ3nUiYvL89LTli4aNEijjjiiDRFlFmKioooKiqiQYMGLFmyhFNPPZUlS5ZQp05m/RbReyaS3JNPwtlnh0Tyv//7bfNcTWJmc9w9b0/lMutbS6ps8+bNnHzyyRQVFeHu/PWvf8245CQiyS1aFK4zOuYY+POfa2Zyqgh9c9UwTZs2Zc6cOekOQ0QqqLhTRMOG8M9/QoMG6Y4o/ZSgRETSrGSniNat0x1RZlCCEhFJsz11iqitat2FuiIimeTJJ2HUqHBh7JVXpjuazKIEJSKSJomdIu67T50iSlKCSoFevXoxY8aMXdaNHTuWK/fwc6hRNI7IqlWrGDBgQKn7LtmtvqSxY8eyJeHK49NOOy0l4+Tdeuut3HXXXVXej4jsTp0i9kwJKgUGDx7MlClTdlk3ZcoUBg8eXK7HH3TQQTz22GOVfv6SCerZZ5+ladOmld6fiMQrsVPEo4+qU0RplKBSYMCAATzzzDN8HY0vsnz5clatWkWPHj12XpeUm5vLkUceyVNPPbXb45cvX07nzp0B2Lp1K4MGDaJLly4MHDiQrVu37ix3xRVX7Jyq49e//jUA48aNY9WqVfTu3ZvevXsD0K5dO9atWwfA3XffTefOnencufPOqTqWL1/OEUccweWXX06nTp049dRTd3meZObOnctxxx1Hly5dOPvss/n88893Pn/Hjh3p0qXLzkFqX3755Z0TNnbr1o0vvvii0sdWpCYq7hRxzz1w4onpjiZz1bhefFdfDameKLZr1zBycGmaN29O9+7dmT59Ov3792fKlCkMHDgQM6NBgwY88cQT7Lvvvqxbt47jjjuOfv36YaU0Nt933300bNiQefPmMW/evF2myxg9ejT77bcf27dv5+STT2bevHn87Gc/4+677+bFF1+kRYsWu+xrzpw5TJw4kVmzZuHuHHvssfTs2ZNmzZqxZMkSJk+ezP3338+5557L448/Xub8ThdccAH33nsvPXv25JZbbuE3v/kNY8eOZcyYMXz00UfUr19/Z7PiXXfdxfjx4zn++OPZvHkzDdR2IbJTYqcIzUFaNtWgUiSxmS+xec/dueGGG+jSpQunnHIKn3zyCWvWrCl1PzNnztyZKLp06UKXLl12bnvkkUfIzc2lW7duLFiwYI8Dwb766qucffbZ7LPPPjRq1IhzzjmHV155BYD27dvTNRqQq6wpPSDMT7VhwwZ69uwJwIUXXsjMmTN3xjhkyBAmTZq0c8SK448/nmuuuYZx48axYcMGjWQhElm0CC64QJ0iyqvGfXOUVdOJ01lnncU111zD22+/zdatW3fWfPLz81m7di1z5syhbt26tGvXLukUG4mS1a4++ugj7rrrLmbPnk2zZs246KKL9rifssZZLJ6qA8J0HXtq4ivNv/71L2bOnMm0adP47W9/y4IFCxg5ciSnn346zz77LMcddxz/+c9/OPzwwyu1f5GaInH6jMcfV6eI8lANKkUaNWpEr169uOSSS3bpHLFx40b2339/6taty4svvsiKZJNRJTjxxBPJj+aff++995g3bx4QpurYZ599aNKkCWvWrOG5557b+ZjGjRsnPc9z4okn8uSTT7Jlyxa+/PJLnnjiCU6oxFWATZo0oVmzZjtrXw899BA9e/Zkx44drFy5kt69e3PnnXeyYcMGNm/ezIcffsiRRx7JddddR15eHu+//36Fn1OkJinZKeLgg9MdUXaocTWodBo8eDDnnHPOLj36hgwZwplnnkleXh5du3bdY03iiiuu4OKLL6ZLly507dqV7t27A2F23G7dutGpU6fdpuoYNmwYffv25cADD+TFF1/cuT43N5eLLrpo5z4uu+wyunXrVmZzXmkefPBBhg8fzpYtWzjkkEOYOHEi27dvZ+jQoWzcuBF35+c//zlNmzbl5ptv5sUXXyQnJ4eOHTvunB1YpLYq7hRx773qFFERmm5D0kLvmdQWTz0VmvYuuggeeEDnnaD8023E2sRnZn3MbLGZLTWzkaWUOdfMFprZAjN7OM54RESq0/vvh5Ei8vLUKaIyYmviM7McYDzwA6AQmG1m09x9YUKZDsD1wPHu/rmZ7R9XPCIi1WnjRujfP3SK0EgRlRNnDao7sNTdl7n7NmAK0L9EmcuB8e7+OYC7f1bZJ8u2psraTO+V1GTbt8M//gG5ueoUUVVxJqhWwMqE5cJoXaL/Af7HzF4zszfNrE+yHZnZMDMrMLOCtWvX7ra9QYMGrF+/Xl98WcDdWb9+vS7elRpnxw6YOhU6dYILL4QmTWD6dHWKqIo4e/Ela20tmUHqAB2AXkBr4BUz6+zuu4x06u4TgAkQOkmU3Gnr1q0pLCwkWfKSzNOgQQNaa/AxqSHcYdo0uPlmmD8/JKh//jN0jNA5p6qJM0EVAokV29bAqiRl3nT3b4CPzGwxIWHNrsgT1a1bl/bt21clVhGRCnGHGTNCYioogA4d4OGH4dxzIScn3dHVDHE28c0GOphZezOrBwwCppUo8yTQG8DMWhCa/JbFGJOISJW99FKY+bZvX1i3DiZOhIULYfBgJadUii1BuXsRMAKYASwCHnH3BWY2ysz6RcVmAOvNbCHwIvBLd18fV0wiIlXx+utw8snQuzcsXx66ji9eHK5x0pCTqVcjLtQVEYnTnDmhKe+552D//eGGG+DHP1bX8crKiAt1RUSy2fz5cM454ULbN9+EMWNC1/GrrlJyqg6qlIqIlLB4Mdx6a+g23rhxuH/11aHruFQfJSgRkciGDfDrX8Of/hRqSCNHwrXXwn77pTuy2kkJSkRqPXd46CH45S9h7VoYPjzUmvbX4GtppQQlIrXavHlh6vVXX4Vjjw0dIaL5RiXN1ElCRGqljRvDeaXc3DAV+9/+FrqRKzllDtWgRKRWcYf8/HBu6bPPQnfx0aN1nikTKUGJSK0xf35oznvlFTjmmDDLbd4er8aRdFETn4jUeJs2wTXXQLduYUiiCRPCdU1KTplNNSgRqbHcYfJk+MUvYM0auPxy+N3voHnzdEcm5aEEJSI10oIFoTnv5ZdDTempp6B793RHJRWhJj4RqVG++CJ0gOjaNXQh/8tfQnOeklP2UQ1KRGqE4hltr70WVq2Cyy6D22+HFi3SHZlUlmpQIpLV3OFf/4Kjj4bzzoMDDww1pvvvV3LKdkpQIpK1XnoJevSAM84IPfUeeghmzQojQkj2U4ISkawzezacemqYOHDFinCe6f33YehQzWhbkyhBiUjWeO89OPvs0OHhnXfgD3+AJUvCaBB166Y7Okk1dZIQkYy3dGmYBmPy5DA/029/GyYNbNw43ZFJnGpVDSo/H9q1g732Cn/z89MdkYiUZeVKGDYMDj8cnngCrrsOPvoIbrpJyak2qDU1qPz88EHfsiUsr1gRlgGGDElfXCKyu88+C13E77svdB+/8kq44QY44IB0RybVKdYalJn1MbPFZrbUzEYm2X6Rma01s7nR7bK4Yrnxxm+TU7EtW8J6EckMGzaE2tEhh8C4ceHH45Il4b6SU+0TWw3KzHKA8cAPgEJgtplNc/eFJYpOdfcRccVR7OOPK7ZeRKrPl1+GJHTnnSFJDRwIv/kNHHZYuiOTdIqzBtUdWOruy9x9GzAF6B/j85WpTZuKrReR+G3bBuPHw6GHhia8Hj1C77wpU5ScJN4E1QpYmbBcGK0r6YdmNs/MHjOzg5PtyMyGmVmBmRWsXbu2UsGMHg0NG+66rmHDsF5EqteOHeG88OGHw4gRIRm99ho8/XQYQ08E4k1QlmSdl1h+Gmjn7l2A/wAPJtuRu09w9zx3z2vZsmWlghkyJMwB07YtmIW/Eyaog4RIdSoelqhbt3BRbZMm8NxzYUSI738/3dFJpomzF18hkFgjag2sSizg7usTFu8H7ogxHoYMUUISSZdXX4Xrrw9/Dz00XNN07rnhsg+RZOL8aMwGOphZezOrBwwCpiUWMLMDExb7AYtijEdE0mDevDBW3gknhAtu77sPFi2CQYOUnKRssdWg3L3IzEYAM4Ac4AF3X2Bmo4ACd58G/MzM+gFFwH+Bi+KKR0Sq17JlcMst8PDDoSlvzBj46U93PxcsUhpzL3laKLPl5eV5QUFBusMQkVJ8+incdls4x1unThiS6Fe/gmbN0h2ZZAozm+PueXsqV2tGkhCReG3cCL//PdxzD3z9NVx+Odx8Mxx0ULojk2ylBCUiVbJ1a7iW6fbb4b//DeeWRo2CDh3SHZlkOyUoEamwbdvgjTdCF/FJk+CTT6BPH/jd70IXcpFUUIISkXJZsQKmTw+355+HL74I55h69QoX3fbsme4IpaZRghKRpL76Cl55JdSSpk8PXcMhDA923nmhxnTSSbDvvumNU2ouJSgR2Wnp0m8T0osvhvNL9euH2tHll0PfvmFYIks2ToxIiilBidRiX34ZElFx092HH4b1HTrAZZeFhNSzp65dkvRQghKphebODaOHP/986PDQsGForrvmGvh//y8MRSSSbkpQIrXIpk1hdId774XmzcPIDn37hmku6tdPd3Qiu1KCEqkF3GHq1FBD+vRTuOKKMNqDRneQTKYEJVLDvf8+/OQn8MILkJcH06aFvyKZTmMJi9RQW7bAjTdCly4wZw78+c/w5ptKTpI9VIMSqYGefjqcX1qxAi64AO68E77znXRHJVIxqkGJ1CDLl0P//tCvHzRqBC+/DA8+qOQk2UkJSqQG2LYtDNbasSP85z+hxvTOO3DiiemOTKTy1MQnkuVeeCF0gnj/fTjnHBg7Fg4+ON1RiVSdalAiWWr1ahgyBE4+OdSg/vUvePxxJSepOZSgRLJMUVG40Pbww+Gxx8KFt++9B6edlu7IRFJLTXwiWcIdnnwSfv1rmD8fTj0V/vQnTQwoNZdqUCIZzj1cXHv00eEc09dfwyOPhMFdlZykJos1QZlZHzNbbGZLzWxkGeUGmJmbmS4hFIm4w7PPQvfuoev4pk3wj3/AggXwox9pygup+WJLUGaWA4wH+gIdgcFm1jFJucbAz4BZccUikk3c4d//hu99D04/HdatgwceCBMGnn9+mMVWpDaIswbVHVjq7svcfRswBeifpNxvgTuBr2KMRSTjuYcu4yecEKa8WL0aJkyAxYvh4ouhbt10RyhSveJMUK2AlQnLhdG6ncysG3Cwuz9T1o7MbJiZFZhZwdq1a1MfqUiavfwy9O4duowvXx7GzfvggzCLbb166Y5OJD3iTFDJWsh950azvYB7gF/saUfuPsHd89w9r2XLlikMUSS9XnsNTjkFevUKCenee8O061dcofmZROJMUIVA4iWDrYFVCcuNgc7AS2a2HDgOmKaOElIbvPlmaMbr0SN0Gb/nnjDd+ogR0KBBuqMTyQxxJqjZQAcza29m9YBBwLTije6+0d1buHs7d28HvAn0c/eCGGOSWua558J5nR070h1JUFAQOj5873vw9tvw+9/DsmVw9dWw997pjk4ks8TWH8jdi8xsBDADyAEecPcFZjYKKHD3aWXvQaRqHnssdMeGMPzP0KFh6onDD6/eOFavDrFMnRqa9PbbD8aMCePnNWpUvbGIZBNz9z2XyiB5eXleUFC5SlZREVx/PRxxBFxySYoDk4wyZ07oDdetW2g2e+ih0HV7+3Y45piQqAYNghYt4nn+zz4L4+JNnQozZ4Yeel26hLHzhg+HffeN53lFsoGZzXH3PZ7OqVUJavv20O7/2mvwxhvQtWuKg5OMsGpVSEJ16sDs2bD//mH9p5/C5MnhYte5c8P2004LyeqMM6reKWH9enjiiZCUipsVjzgCBg6Ec88N90UkxQnKzA4FCt39azPrBXQB/uHuG6ocaQVVJUFB+GWbmxu+jObMgaZNUxicpN2WLdCzZ5h64rXXQq0lmXnzQq0qPz80wTVrFhLJBRfAcceVf5SGDRvC+HhTp4Z5mIqK4LvfDfsaOBA6d9aIDyIlpTpBzQXygHaEc0rTgMPcvdrHT65qggJ4/fXwJXbaaeEX714akbBGcA/Ndo8+Ck89BWeeuefHbN8Ozz8falX//Cds3RoSzPnnh1v79rs/5osvwth4U6fCjBlhqot27UItaeDA0KyopCRSuvImKNx9jzfg7ejvL4GfRvffKc9jU307+uijPRXGjnUH9zFjUrI7yQC33hre0zvvrNzjN21ynzjRvXfvsB9wP+EE9/vvd//kE/epU93POce9QYOwrXVr92uucZ81y33HjpS+FJEajdBRbo/f9+WtQc0CxgI3Ame6+0dm9p67d650Cq2kVNSg4Ntf2489Fn5B9+pV9dgkfaZODe/nRReFceuqWoNZsSI0//3jH2GooWIHHBB6Bg4cGLqKq/YtUnGpbuLrCAwH3nD3yWbWHhjo7mOqHmrFpCpBQWiqOeY8rhwIAAAUY0lEQVQY+PxzeOcdOOiglOxWqtns2XDiiZCXF84DpXIEBvdw7dJ//hMS0gknQE5O6vYvUhvF1ovPzJoRxs+bV9ngqiKVCQrC1AXdu4eOEy+8oAE5s01hYXj/6teHt94CjYQlkvnKm6DK1UBhZi+Z2b5mth/wLjDRzO6uapCZoFMnuP9+ePVVGFnqjFWSib78MsyTtHkzPP20kpNITVPeFvQm7r4JOAeY6O5HA6fEF1b1Ou+8cFX/3XeHiysl8+3YEc43vfNOuLapc7WfDRWRuJU3QdUxswOBc4Eyp8bIVn/4Q2gquvjiMKq0ZLZbbw0dXO66K4xtJyI1T3kT1CjC9U8fuvtsMzsEWBJfWNWvfv1w/Uy9evDDH4bmI8lMDz8Mv/0tXHop/Pzn6Y5GROJSrgTl7o+6exd3vyJaXubuP4w3tOrXpk348luwIIyXlmWjQNUKs2aFcRRPPDFM6qcLYkVqrvJ2kmhtZk+Y2WdmtsbMHjez1nEHlw6nnhqajyZNgr/+Nd3RSKKVK0OniFatwrlCzTQrUrOVt4lvImF4o4MI07Y/Ha2rkW66Cfr0gauuCtfASPpt3hyGLtq6NfTYi2sUchHJHOVNUC3dfaK7F0W3vwM1tlPvXnuFGtQBB8CAAWGUakmfHTvCuHjz54cRIzp2THdEIlIdypug1pnZUDPLiW5DgRr9td28eegltnp1+HLMlBlZa6Obbw4jht99d6jZikjtUN4EdQmhi/mnwGpgAHBxXEFlimOOgbFjw7Tho0enO5raadIk+N3vYNgw+NnP0h2NiFSnSk9YaGZXu/vYFMezR6ke6mhP3MMcQfn5MH166EQhpfv885DQGzQIcyw1bRpuzZqFWWQrMrjqG2+EQXy///0wG66GoRKpGWKfUdfMPnb3NpV6cBVUd4KCcE3UsceGGVnfeQcOPrhanz4rbNkC48bBHXeESfySMQtJqmTiSna/YUO48spQ/s03Q5OriNQM5U1QdaryHFV4bFbZZ5/QrfmYY8JUCzNnpqaLs3uYgTWbawbffAN/+xuMGhUS+Jlnwg03wN57h0T1+efhb+L9xHVLlny7ruTF0U2ahB57Sk4itVNVEtQeq15m1gf4I5AD/K3k9BxmNhz4CbAd2AwMc/eFVYgpNocdFuYZ+tGP4IwzwnBIH38cLu4dPRqGDEn+uK++guXLYdmy5LeiojAT67BhcPzx2XPh6Y4doUfdzTfDhx9Cjx6hU8nxx1d+n998Axs3fpvA2rSB73wndTGLSHYps4nPzL4geSIyYG93LzXBmVkO8AHwA6AQmA0MTkxAZrZvNAgtZtYPuNLdy+ynlY4mvkR9+4ZzUYn23ht+9Svo0GH3BPTJJ7uOSNGwIRxyyLe3rVvDYKebNoWR1YcNC70GmzWr3tdVXu7hHNMNN8C770KXLnD77eG4ZEtyFZH0SkkTn7s3rkIM3YGl7r4sCmgK0B/YmaCKk1NkH8pRK0u3hUnqd1u3wm9+8+1yq1Yh+Zx88reJ6NBDw9/999/9i/wPf4ApU8LIFVddBdddF2ZsHTYsTJKXKV/8r70G118Pr7wSXsvDD4c4NausiMShKk18e9IKWJmwXAgcW7KQmf0EuAaoB5yUbEdmNgwYBtCmTbX3y9jFypWlb1u4ENq1CzWqithnnzDw6aWXhk4YEyaEXoMPPghHHhkS1dChofNAOsybBzfeCM88Ey5e/vOfQ6waakhE4hTnb99kv/t3qyG5+3h3PxS4Drgp2Y7cfYK757l7Xss0z0pXWn5s2xaOOKLiyamkbt3gvvtg1aqQqOrXh5/+NExHf8kloUdbdQ1iu2xZaG7s2jXUmn73O1i6FK64QslJROIXZ4IqBBI7ZLcGVpVRfgpwVozxpMTo0eE8UqKGDVN/IW+jRnD55TB7NsyZExLFo4+GJr+uXUMtZuPG1D5nsU8/hREj4PDDQ8eHX/0qJKvrrw+1PRGR6hBngpoNdDCz9mZWDxhEGHB2JzPrkLB4Olkwx9SQIaFm07ZtODfUtm1YLq0XXyrk5obzU6tWwV/+Ajk5YQbggw4KTW1vvZWaWtWGDaEp79BDw/NcemnooTdmDOy3X9X3LyJSEZW+ULdcOzc7DRhL6Gb+gLuPNrNRQIG7TzOzPxKmjv8G+BwY4e4LytpnunvxZYqCgpC0Jk8O1w8V12yK3073b2/lXS42aFC4rqlD4s8HEZEUiX0kiXRRgtrVpk2hB+D774caXfENKr6ckwP9+oXzYCIicamOkSQkA+y7b+jlJyJS0+gKFhERyUhKUCIikpGUoEREJCMpQYmISEZSghIRkYykBCUiIhlJCUpERDKSEpSIiGQkJSgREclISlAiIpKRlKBERCQjKUGJiEhGUoISEZGMpAQlIiIZSQkqTfLzoV072Guv8Dc/P90RiYhkFs0HlQb5+WEOpy1bwvKKFd/O6RTn1PEiItlENag0uPHGb5NTsS1bwnoREQmUoNLg448rtl5EpDZSgkqDNm0qtl5EpDaKNUGZWR8zW2xmS81sZJLt15jZQjObZ2bPm1nbOOPJFKNHQ8OGu65r2DCsFxGRILYEZWY5wHigL9ARGGxmHUsUewfIc/cuwGPAnXHFk0mGDIEJE6BtWzALfydMUAcJEZFEcfbi6w4sdfdlAGY2BegPLCwu4O4vJpR/ExgaYzwZZcgQJSQRkbLE2cTXCliZsFwYrSvNpcBzyTaY2TAzKzCzgrVr16YwRBERyVRxJihLss6TFjQbCuQBv0+23d0nuHueu+e1bNkyhSGKiEimirOJrxA4OGG5NbCqZCEzOwW4Eejp7l/HGI+IiGSROGtQs4EOZtbezOoBg4BpiQXMrBvwV6Cfu38WYywiIpJlYktQ7l4EjABmAIuAR9x9gZmNMrN+UbHfA42AR81srplNK2V3UgqN6SciNVWsY/G5+7PAsyXW3ZJw/5Q4n7+m05h+IlKTaSSJLKYx/USkJlOCymIa009EajIlqCymMf1EpCZTgspiGtNPRGoyJagspjH9RKQm04y6WU5j+olITaUalIiIZCQlKBERyUhKUAJoRAoRyTw6ByUakUJEMpJqUKIRKUQkIylBiUakEJGMpAQlGpFCRDKSEpRoRAoRyUhKUKIRKUQkI6kXnwAakUJEMo9qUJIyupZKRFJJNShJCV1LJSKpphqUpISupRKRVFOCkpTQtVQikmqxJigz62Nmi81sqZmNTLL9RDN728yKzGxAnLFIvHQtlYikWmwJysxygPFAX6AjMNjMOpYo9jFwEfBwXHFI9UjltVTqbCEiEG8Nqjuw1N2Xufs2YArQP7GAuy9393nAjhjjkGqQqmupijtbrFgB7t92tlCSEql94kxQrYCVCcuF0boKM7NhZlZgZgVr165NSXCSekOGwPLlsGNH+FuZ3nvqbCEixeJMUJZknVdmR+4+wd3z3D2vZcuWVQxLMpk6W4hIsTgTVCFwcMJya2BVjM8nNYA6W4hIsTgT1Gygg5m1N7N6wCBgWozPJzWAOluISLHYEpS7FwEjgBnAIuARd19gZqPMrB+AmR1jZoXAj4C/mtmCuOKR7KDOFiJSzNwrdVoobfLy8rygoCDdYUiGa9cuJKWS2rYNHThEJH3MbI675+2pnEaSkBpJnS1Esp8SlNRIqexsoXNZIumhBCU1Uqo6W+hclkj6KEFJjZSqzha6cFgkfZSgpMZKxcgWqTqXpWZCkYpTghIpQyrOZamZUKRylKBEypCKc1lqJhSpHCUokTKk4lxWKru8q6lQapM66Q5AJNMNGVK581fF2rRJftFwRbu8FzcVFtfGipsKi2MUqWlUgxKJWaq6vKeyqVA1MckGSlAiMUtVl/dU9ihUpw3JBkpQItUgFV3eUzU6RqpqYqqFSdyUoESyRKqaClNRE1MtTKqDEpRIlkhVU2EqamI6HybVQQlKJIukoqkwFTUxnQ+T6qAEJVLLpKImlmnnw0A1sZpICUqkFqpqTSyTzodB6mpiqUpySpYp4u5ZdTv66KNdRNJv0iT3tm3dzcLfSZMqvo+2bd1DStn11rZt9e9n0iT3hg13fXzDhhV/XanaTyql4r1KJaDAy/F9n/aEU9GbEpRIzZGqL3Oz5AnKrPz7yKRkWSwViSUTE2Z5E5Sa+EQkbTKpZ2Kqmhszrdkym697izVBmVkfM1tsZkvNbGSS7fXNbGq0fZaZtYszHhHJPJnSMzFVHT8yrQNJNl/3FluCMrMcYDzQF+gIDDazjiWKXQp87u7fBe4B7ogrHhGpuVJRE0tVx49M60CSade9VUScNajuwFJ3X+bu24ApQP8SZfoDD0b3HwNONjOLMSYRqaGqWhNLVXNjJjVbQmZd91ZRcSaoVsDKhOXCaF3SMu5eBGwEmpfckZkNM7MCMytYu3ZtTOGKSG2XiubGVO0nVTWxTLruraLiTFDJakJeiTK4+wR3z3P3vJYtW6YkOBGRTJaqmljxvjLhureKinPCwkLg4ITl1sCqUsoUmlkdoAnw3xhjEhHJGlWdLDOVcUA45/Txx6HmNHp0/LHFmaBmAx3MrD3wCTAIOK9EmWnAhcAbwADghaiPvIiIZJB0JMvYEpS7F5nZCGAGkAM84O4LzGwU4SKtacD/Ag+Z2VJCzWlQXPGIiEh2ibMGhbs/CzxbYt0tCfe/An4UZwwiIpKdNJKEiIhkJCUoERHJSEpQIiKSkSzbOs2Z2VpgRbrjKKcWwLp0B1FB2Raz4o1XtsUL2RdzbYy3rbvv8aLWrEtQ2cTMCtw9L91xVES2xax445Vt8UL2xax4S6cmPhERyUhKUCIikpGUoOI1Id0BVEK2xax445Vt8UL2xax4S6FzUCIikpFUgxIRkYykBCUiIhlJCaqKzOxgM3vRzBaZ2QIzuypJmV5mttHM5ka3W5LtqzqZ2XIzmx/FU5Bku5nZODNbambzzCw3HXFGsRyWcOzmmtkmM7u6RJm0HmMze8DMPjOz9xLW7Wdm/2dmS6K/zUp57IVRmSVmdmEa4/29mb0fvd9PmFnTUh5b5menmmO+1cw+SXjfTyvlsX3MbHH0eR6ZxninJsS63MzmlvLYaj/GpX2XpfVz7O66VeEGHAjkRvcbAx8AHUuU6QU8k+5YS8S0HGhRxvbTgOcIk0oeB8xKd8xRXDnAp4QL/TLmGAMnArnAewnr7gRGRvdHAnckedx+wLLob7PofrM0xXsqUCe6f0eyeMvz2anmmG8Fri3HZ+ZD4BCgHvBuyf/R6oq3xPY/ALdkyjEu7bssnZ9j1aCqyN1Xu/vb0f0vgEXsPrV9NuoP/MODN4GmZnZguoMCTgY+dPeMGk3E3Wey+2Sb/YEHo/sPAmcleej/A/7P3f/r7p8D/wf0iS3QSLJ43f3f7l4ULb5JmGQ0Y5RyjMujO7DU3Ze5+zZgCuG9iVVZ8ZqZAecCk+OOo7zK+C5L2+dYCSqFzKwd0A2YlWTz98zsXTN7zsw6VWtgyTnwbzObY2bDkmxvBaxMWC4kMxLvIEr/p860Y/wdd18N4Z8f2D9JmUw9zpcQatDJ7OmzU91GRM2SD5TS/JSJx/gEYI27Lylle1qPcYnvsrR9jpWgUsTMGgGPA1e7+6YSm98mNEkdBdwLPFnd8SVxvLvnAn2Bn5jZiSW2W5LHpPWaBDOrB/QDHk2yOROPcXlk4nG+ESgC8kspsqfPTnW6DzgU6AqsJjSblZRxxxgYTNm1p7Qd4z18l5X6sCTrqnyMlaBSwMzqEt7QfHf/Z8nt7r7J3TdH958F6ppZi2oOs2RMq6K/nwFPEJpBEhUCBycstwZWVU90peoLvO3ua0puyMRjDKwpbhaN/n6WpExGHefo5PYZwBCPTi6UVI7PTrVx9zXuvt3ddwD3lxJLph3jOsA5wNTSyqTrGJfyXZa2z7ESVBVFbcn/Cyxy97tLKXNAVA4z60447uurL8rd4tnHzBoX3yecHH+vRLFpwAVRb77jgI3F1fw0KvVXZ6Yd48g0oLg304XAU0nKzABONbNmUfPUqdG6amdmfYDrgH7uvqWUMuX57FSbEudFzy4lltlABzNrH9XCBxHem3Q5BXjf3QuTbUzXMS7juyx9n+Pq7CVSE29AD0JVdh4wN7qdBgwHhkdlRgALCL2H3gS+n+aYD4lieTeK68ZofWLMBown9H6aD+SlOeaGhITTJGFdxhxjQuJcDXxD+DV5KdAceB5YEv3dLyqbB/wt4bGXAEuj28VpjHcp4TxC8ef4L1HZg4Bny/rspDHmh6LP5zzCF+mBJWOOlk8j9Er7sLpiThZvtP7vxZ/bhLJpP8ZlfJel7XOsoY5ERCQjqYlPREQykhKUiIhkJCUoERHJSEpQIiKSkZSgREQkIylBiaSQmW23XUdeT9nI2WbWLnFkbJGark66AxCpYba6e9d0ByFSE6gGJVINovl97jCzt6Lbd6P1bc3s+Wiw0+fNrE20/jsW5mR6N7p9P9pVjpndH83X828z2zsq/zMzWxjtZ0qaXqZISilBiaTW3iWa+AYmbNvk7t2BPwFjo3V/Ikxr0oUwOOu4aP044GUPg9/mEkYUAOgAjHf3TsAG4IfR+pFAt2g/w+N6cSLVSSNJiKSQmW1290ZJ1i8HTnL3ZdGAnJ+6e3MzW0cYnuebaP1qd29hZmuB1u7+dcI+2hHm3OkQLV8H1HX328xsOrCZMIr7kx4NnCuSzVSDEqk+Xsr90sok83XC/e18ex75dMLYiUcDc6IRs0WymhKUSPUZmPD3jej+64TRtQGGAK9G958HrgAwsxwz27e0nZrZXsDB7v4i8CugKbBbLU4k2+hXlkhq7W1mcxOWp7t7cVfz+mY2i/DDcHC07mfAA2b2S2AtcHG0/ipggpldSqgpXUEYGTuZHGCSmTUhjEJ/j7tvSNkrEkkTnYMSqQbROag8d1+X7lhEsoWa+EREJCOpBiUiIhlJNSgREclISlAiIpKRlKBERCQjKUGJiEhGUoISEZGM9P8B/oUMTOUg1lQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# ‘bo’는 파란색 점을 의미합니다\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# ‘b’는 파란색 실선을 의미합니다\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcFNW9///Xh0VZZXcDYZAYAwwMjOMQI+7GLYprBII34kY0ala/iQneaEyMibtGf0aMJlGJyI1XRa9LlOCWBGFQZNEoCIgjiMO+qiyf3x+nZmiG7pmemV5qZt7Px6Me3VV1qvrTNT396XPq1Clzd0REROKmRb4DEBERSUYJSkREYkkJSkREYkkJSkREYkkJSkREYkkJSkREYkkJSrLOzFqa2UYz653JsvlkZl8ys4xfo2Fmx5vZkoT598zsiHTK1uO1/mhmP6/v9iLZ1irfAUj8mNnGhNl2wOfA9mj+O+4+sS77c/ftQIdMl20O3P3gTOzHzC4GznP3oxP2fXEm9i2SLUpQsht3r0oQ0S/0i939pVTlzayVu2/LRWwitdHnselQE5/UmZn92sweM7NHzWwDcJ6ZHWZm081srZktN7O7zKx1VL6VmbmZFUTzj0TrnzOzDWb2bzPrW9ey0fqTzex9M1tnZr83s3+a2dgUcacT43fMbKGZrTGzuxK2bWlmt5vZKjP7ADiphuNzjZlNqrbsHjO7LXp+sZm9G72fD6LaTap9lZvZ0dHzdmb2cBTbfOCQJK+7KNrvfDMbES0fBNwNHBE1n65MOLbXJWx/afTeV5nZk2a2XzrHpi7HuTIeM3vJzFab2Sdm9pOE1/nv6JisN7MyM9s/WXOqmb1e+XeOjuer0eusBq4xs4PMbFr0XlZGx61TwvZ9ovdYEa2/08zaRDH3Tyi3n5ltNrNuqd6vZJG7a9KUcgKWAMdXW/Zr4AvgNMKPnLbAocAwQq38QOB94IqofCvAgYJo/hFgJVACtAYeAx6pR9m9gQ3A6dG6HwFbgbEp3ks6MT4FdAIKgNWV7x24ApgP9AK6Aa+Gf5+kr3MgsBFon7DvT4GSaP60qIwBxwJbgMHRuuOBJQn7KgeOjp7fArwMdAH6AO9UK3susF/0N/lWFMM+0bqLgZerxfkIcF30/IQoxiFAG+D/A/6RzrGp43HuBKwAvg/sCewFlEbrfga8DRwUvYchQFfgS9WPNfB65d85em/bgMuAloTP45eB44A9os/JP4FbEt7PvOh4to/KHx6tmwDckPA6PwaeyPf/YXOd8h6ApnhPpE5Q/6hlu6uA/4meJ0s6f0goOwKYV4+yFwKvJawzYDkpElSaMX41Yf3/AldFz18lNHVWrjul+pdmtX1PB74VPT8ZeL+Gss8Al0fPa0pQSxP/FsB3E8sm2e884BvR89oS1F+A3ySs24tw3rFXbcemjsf5v4CyFOU+qIy32vJ0EtSiWmI4B5gZPT8C+ARomaTc4cBiwKL52cBZmf6/0pTepCY+qa+PEmfM7Ctm9n9Rk8164Hqgew3bf5LwfDM1d4xIVXb/xDg8fKOUp9pJmjGm9VrAhzXEC/BXYHT0/FtAVccSMzvVzN6ImrjWEmovNR2rSvvVFIOZjTWzt6NmqrXAV9LcL4T3V7U/d18PrAF6JpRJ629Wy3E+AFiYIoYDCEmqPqp/Hvc1s8lm9nEUw5+rxbDEQ4ecXbj7Pwm1seFmVgj0Bv6vnjFJAylBSX1V72J9H+EX+5fcfS/gF4QaTTYtJ/zCB8DMjF2/UKtrSIzLCV9slWrrBv8YcLyZ9SI0Qf41irEt8DfgRkLzW2fg72nG8UmqGMzsQOBeQjNXt2i//0nYb21d4pcRmg0r99eR0JT4cRpxVVfTcf4I6Jdiu1TrNkUxtUtYtm+1MtXf3+8IvU8HRTGMrRZDHzNrmSKOh4DzCLW9ye7+eYpykmVKUJIpHYF1wKboJPN3cvCazwDFZnaambUinNfokaUYJwM/MLOe0Qnzn9ZU2N1XEJqh/gS85+4LolV7Es6LVADbzexUwrmSdGP4uZl1tnCd2BUJ6zoQvqQrCLn6YkINqtIKoFdiZ4VqHgUuMrPBZrYnIYG+5u4pa6Q1qOk4TwF6m9kVZraHme1lZqXRuj8CvzazfhYMMbOuhMT8CaEzTkszG0dCMq0hhk3AOjM7gNDMWOnfwCrgNxY6nrQ1s8MT1j9MaBL8FiFZSZ4oQUmm/Bg4n9Bp4T5CDSKroiQwEriN8IXTD3iL8Ms50zHeC0wF5gIzCbWg2vyVcE7prwkxrwV+CDxB6GhwDiHRpuNaQk1uCfAcCV+e7j4HuAuYEZX5CvBGwrYvAguAFWaW2FRXuf3zhKa4J6LtewNj0oyrupTH2d3XAV8HziZ0yngfOCpafTPwJOE4ryd0WGgTNd1eAvyc0GHmS9XeWzLXAqWERDkFeDwhhm3AqUB/Qm1qKeHvULl+CeHv/IW7/6uO710yqPJEoEijFzXZLAPOcffX8h2PNF5m9hCh48V1+Y6lOdOFutKomdlJhCabzwjdlLcRahEi9RKdzzsdGJTvWJo7NfFJYzccWERo+jkJOEMntaW+zOxGwrVYv3H3pfmOp7lTE5+IiMSSalAiIhJLTeYcVPfu3b2goCDfYYiISC1mzZq10t1ruiQEaEIJqqCggLKysnyHISIitTCz2kZiAdTEJyIiMaUEJSIisaQEJSIisdRkzkEls3XrVsrLy/nss8/yHYrUoE2bNvTq1YvWrVMNEycizVGTTlDl5eV07NiRgoICwkDXEjfuzqpVqygvL6dv3761byAizUbWmvjM7EEz+9TM5qVYb9Etmhea2RwzK05Yd76ZLYim8+sbw2effUa3bt2UnGLMzOjWrZtquSKNwMSJUFAALVqEx4kTa9uiYbJ5DurPhKFnUjmZcGvng4BxhNGiiYbXv5Zwy+hS4Foz61LfIJSc4k9/I5HUcp0Uaopj3Dj48ENwD4/jxmU3nqwlKHd/lXA7gVROBx7yYDrQ2cz2A04EXnT31e6+hnCbgJoSnYhIlUx8oTfnpJDK+PGwefOuyzZvDsuzJZ+9+Hqy622ay6NlqZbvxszGmVmZmZVVVFRkLdD6WrVqFUOGDGHIkCHsu+++9OzZs2r+iy++SGsfF1xwAe+9916NZe655x4m5us/SCRGMvGFnqmkkIkkl6mkkIlYlqYYOjfV8oxw96xNQAEwL8W6/wOGJ8xPBQ4B/h9wTcLy/wZ+XNtrHXLIIV7dO++8s9uymjzyiHufPu5m4fGRR+q0eY2uvfZav/nmm3dbvmPHDt++fXvmXqiRquvfSiSZPn3cQ1rZderTJ7f7eOQR93btdt2+Xbu6f6eYJY/FLPexZOK4VALKPI0cks8aVDlwQMJ8L8LN5lItz6pcVqUXLlxIYWEhl156KcXFxSxfvpxx48ZRUlLCwIEDuf7666vKDh8+nNmzZ7Nt2zY6d+7M1VdfTVFREYcddhiffvopANdccw133HFHVfmrr76a0tJSDj74YP71r3BD0E2bNnH22WdTVFTE6NGjKSkpYfbs2bvFdu2113LooYdWxefRaPfvv/8+xx57LEVFRRQXF7NkyRIAfvOb3zBo0CCKiooYn826vjR5cfmVn4l9ZKrm07t33ZZnM5YbboB27XZd1q5dWJ416WSx+k7UXIP6BuG21QZ8FZgRLe8KLAa6RNNioGttr9XQGlQmfx0kk1iDWrBggZuZz5gxo2r9qlWr3N1969atPnz4cJ8/f767ux9++OH+1ltv+datWx3wZ5991t3df/jDH/qNN97o7u7jx4/322+/var8T37yE3d3f+qpp/zEE090d/cbb7zRv/vd77q7++zZs71Fixb+1ltv7RZnZRw7duzwUaNGVb1ecXGxT5kyxd3dt2zZ4ps2bfIpU6b48OHDffPmzbtsWx+qQTVvcfqVn4l9ZKLm456Z45KpWCrjyUQrE/muQZnZo4Q7nR5sZuVmdpGZXWpml0ZFniXcaG4hcD/w3ShhrgZ+BcyMpuujZVmV6/bVfv36ceihh1bNP/rooxQXF1NcXMy7777LO++8s9s2bdu25eSTTwbgkEMOqarFVHfWWWftVub1119n1KhRABQVFTFw4MCk206dOpXS0lKKiop45ZVXmD9/PmvWrGHlypWcdtppQLiwtl27drz00ktceOGFtG3bFoCuXbvW/UBIXmWqM0BD9xOnX/mZ2Ecmaj4AY8bAhAnQpw+YhccJE8LyXMdSGc+SJbBjR3isSxz1kbULdd19dC3rHbg8xboHgQezEVcqvXuHZr1ky7Ohffv2Vc8XLFjAnXfeyYwZM+jcuTPnnXde0uuC9thjj6rnLVu2ZNu2bUn3veeee+5Wxr32G1Nu3ryZK664gjfffJOePXtyzTXXVMWRrCu4u6uLeCNW2axdmRgqm7Whbl88mdhPpn4gVr7e+PFh2969Q2Kpy/vJxD5uuGHXYwL1bw4bM6ZhiSCTseSaxuKL5KV9NbJ+/Xo6duzIXnvtxfLly3nhhRcy/hrDhw9n8uTJAMydOzdpDW3Lli20aNGC7t27s2HDBh5//HEAunTpQvfu3Xn66aeBcAH05s2bOeGEE3jggQfYsmULAKtXZ72iKxmUqVpLJvYTt1/5Dd1HJmo+mRKnWOpKCSqSzz9icXExAwYMoLCwkEsuuYTDDz88469x5ZVX8vHHHzN48GBuvfVWCgsL6dSp0y5lunXrxvnnn09hYSFnnnkmw4YNq1o3ceJEbr31VgYPHszw4cOpqKjg1FNP5aSTTqKkpIQhQ4Zw++23ZzxuSa2hzWqZqrVkYj/5/IGYLbluDmsssdRJOieqGsOUiW7mTdnWrVt9y5Yt7u7+/vvve0FBgW/dujXPUe2kv1XdZOLkeaY6BmVqP9m8zEPihXx3kpB42bhxI4cffjhFRUWcffbZ3HfffbRq1aTHCm7SMtGslqlaS6b202h/5UvW6BuqmejcuTOzZs3KdxiSIZloVstEZ4BM7kekOtWgRPKgoeePMtmNORO1FtV+JBuUoERyLBOjljTFTgUi1SlBieRYJs4fNeauwyLpUoISqYO4jBcHalaTpk8JKouOPvro3S66veOOO/jud79b43YdOnQAYNmyZZxzzjkp911WVlbjfu644w42J/xUP+WUU1i7dm06oUsSmRpQOJMXpYo0ZUpQWTR69GgmTZq0y7JJkyYxenSNo0BV2X///fnb3/5W79evnqCeffZZOnfuXO/9NXdxGi9OpDlQgsqic845h2eeeYbPP/8cgCVLlrBs2TKGDx/Oxo0bOe644yguLmbQoEE89dRTu22/ZMkSCgsLgTAM0ahRoxg8eDAjR46sGl4I4LLLLqu6Vce1114LwF133cWyZcs45phjOOaYYwAoKChg5cqVANx2220UFhZSWFhYdauOJUuW0L9/fy655BIGDhzICSecsMvrVHr66acZNmwYQ4cO5fjjj2fFihVAuNbqggsuYNCgQQwePLhqqKTnn3+e4uJiioqKOO644zJybPMhk01zOn8kUrtmcx3UD34ASW5/1CBDhkD03Z5Ut27dKC0t5fnnn+f0009n0qRJjBw5EjOjTZs2PPHEE+y1116sXLmSr371q4wYMSLl4Kv33nsv7dq1Y86cOcyZM4fi4uKqdTfccANdu3Zl+/btHHfcccyZM4fvfe973HbbbUybNo3u3bvvsq9Zs2bxpz/9iTfeeAN3Z9iwYRx11FF06dKFBQsW8Oijj3L//fdz7rnn8vjjj3Peeeftsv3w4cOZPn06ZsYf//hHbrrpJm699VZ+9atf0alTJ+bOnQvAmjVrqKio4JJLLuHVV1+lb9++jXq8vkwOKNzQAUBFmgPVoLIssZkvsXnP3fn5z3/O4MGDOf744/n444+raiLJvPrqq1WJYvDgwQwePLhq3eTJkykuLmbo0KHMnz8/6UCwiV5//XXOPPNM2rdvT4cOHTjrrLN47bXXAOjbty9DhgwBUt/So7y8nBNPPJFBgwZx8803M3/+fABeeuklLr985wD1Xbp0Yfr06Rx55JH07dsXyO8tORrawUFNcyK51WxqUDXVdLLpjDPO4Ec/+hFvvvkmW7Zsqar5TJw4kYqKCmbNmkXr1q0pKChIeouNRMlqV4sXL+aWW25h5syZdOnShbFjx9a6H6/h1huVt+qAcLuOZE18V155JT/60Y8YMWIEL7/8Mtddd13VfqvHmGxZPmTilhAaMUEkt1SDyrIOHTpw9NFHc+GFF+7SOWLdunXsvffetG7dmmnTpvFhsrajBEceeSQTo5/88+bNY86cOUC4VUf79u3p1KkTK1as4LnnnqvapmPHjmzYsCHpvp588kk2b97Mpk2beOKJJzjiiCPSfk/r1q2jZ8+eAPzlL3+pWn7CCSdw9913V82vWbOGww47jFdeeYXFixcD+bslR6Y6OKhrt0juKEHlwOjRo3n77ber7mgLMGbMGMrKyigpKWHixIl85StfqXEfl112GRs3bmTw4MHcdNNNlJaWAuHuuEOHDmXgwIFceOGFu9yqY9y4cZx88slVnSQqFRcXM3bsWEpLSxk2bBgXX3wxQ4cOTfv9XHfddXzzm9/kiCOO2OX81jXXXMOaNWsoLCykqKiIadOm0aNHDyZMmMBZZ51FUVERI0eOTPt1MinXd0wWkYazmpp7GpOSkhKvfl3Qu+++S//+/fMUkdRFtv9WBQXJOzj06RNqQiKSO2Y2y91LaiunGpTEXiZGb1AHB5HGRwlKYi1Tozfo2iORxqfJJ6im0oTZlNX0N8pU5wZQBweRxqZJJ6g2bdqwatUqJakYc3dWrVpFmzZtkq5X5waR5qtJXwfVq1cvysvLqaioyHcoUoM2bdrQq1evpOsyOXqDiDQuTTpBtW7dumoEA8mPiRMbdmHrDTfseoEtqHODSHPRpJv4JL8y0cFBnRtEmq8mfR2U5JeuPRKRZHQdlOSdOjiISEMoQUnW6M6xItIQSlCSNRq9QUQaQglKskYdHESkIZp0N3PJP905VkTqSzUoERGJJSUoERGJJSUoSSkTt7kQEakvnYOSpCpHgagcYqhyFAjQOSURyQ3VoCSpTN7mQkSkPpSgJCmNAiEi+ZbVBGVmJ5nZe2a20MyuTrK+j5lNNbM5ZvaymfVKWLfdzGZH05Rsxim70ygQIpJvWUtQZtYSuAc4GRgAjDazAdWK3QI85O6DgeuBGxPWbXH3IdE0IltxSnIaBUJE8i2bNahSYKG7L3L3L4BJwOnVygwApkbPpyVZL3miUSBEJN+ymaB6Ah8lzJdHyxK9DZwdPT8T6Ghm3aL5NmZWZmbTzeyMZC9gZuOiMmW6a27mjRkTbouxY0d4VHISkVzKZoKyJMuq33zqKuAoM3sLOAr4GNgWresd3S/kW8AdZtZvt525T3D3Encv6dGjRwZDFxGRfMvmdVDlwAEJ872AZYkF3H0ZcBaAmXUAznb3dQnrcPdFZvYyMBT4IIvxiohIjGSzBjUTOMjM+prZHsAoYJfeeGbW3cwqY/gZ8GC0vIuZ7VlZBjgceCeLsTYpGgFCRJqCrNWg3H2bmV0BvAC0BB509/lmdj1Q5u5TgKOBG83MgVeBy6PN+wP3mdkOQhL9rbsrQaVBI0CISFNh7tVPCzVOJSUlXlZWlu8w8q6gICSl6vr0CR0dRETyzcxmRX0MaqSRJJoYjQAhIk2FElQToxEgRKSpUIJqYjQChIg0FUpQTYxGgBCRpkL3g2qCxoxRQhKRxk81KBERiSUlKBERiSUlKBERiSUlKBERiSUlKBERiSUlKBERiSUlqJjRSOQiIoGug4oRjUQuIrKTalAxMn78zuRUafPmsFxEpLlRgooRjUQuIrKTElSMaCRyEZGdlKBiRCORi4jspAQVIxqJXERkJ/XiixmNRC4iEqgGJSIisaQEJSIisaQEJSIisaQEJSIisaQEJSIisaQEJSIisaQEJSIisaQEJSIisaQEJSIisVRrgjKzK8ysSy6CERERqZRODWpfYKaZTTazk8zMsh2UiIhIrQnK3a8BDgIeAMYCC8zsN2bWL8uxiYhIM5bWOSh3d+CTaNoGdAH+ZmY3ZTE2ERFpxtI5B/U9M5sF3AT8Exjk7pcBhwBnZzm+RmPiRCgogBYtwuPEifmOSESkcUvndhvdgbPc/cPEhe6+w8xOzU5YjcvEiTBuHGzeHOY//DDMg26dISJSX+k08T0LrK6cMbOOZjYMwN3fzVZgjcn48TuTU6XNm8NyERGpn3QS1L3AxoT5TdEyiSxdWrflIiJSu3QSlEWdJIDQtEead+KNuqW/Z2YLzezqJOv7mNlUM5tjZi+bWa+Edeeb2YJoOj+d18uX3r3rtlxERGqXToJaFHWUaB1N3wcW1baRmbUE7gFOBgYAo81sQLVitwAPuftg4HrgxmjbrsC1wDCgFLg2zhcL33ADtGu367J27cJyERGpn3QS1KXA14CPgXJC0hiXxnalwEJ3X+TuXwCTgNOrlRkATI2eT0tYfyLworuvdvc1wIvASWm8Zl6MGQMTJkCfPmAWHidMUAcJEZGGqLWpzt0/BUbVY989gY8S5iuTW6K3CV3V7wTOBDqaWbcU2/asRww5M2aMEpKISCbVmqDMrA1wETAQaFO53N0vrG3TJMu82vxVwN1mNhZ4lVBL25bmtpjZOKLaXG+d8BERaVLSaeJ7mDAe34nAK0AvYEMa25UDByTM9wKWJRZw92Xufpa7DwXGR8vWpbNtVHaCu5e4e0mPHj3SCElERBqLdBLUl9z9v4FN7v4X4BvAoDS2mwkcZGZ9zWwPQjPhlMQCZtbdzCpj+BnwYPT8BeAEM+sSdY44IVomIiLNRDoJamv0uNbMCoFOQEFtG7n7NuAKQmJ5F5js7vPN7HozGxEVOxp4z8zeB/YBboi2XQ38ipDkZgLXR8tERKSZsIRLnJIXMLsYeJxQa/oz0AH4b3e/L+vR1UFJSYmXlZXlOwwREamFmc1y95LaytXYSSJqflsfdfV+FTgwQ/GJiIjUqMYmvmjUiCtyFIuIiEiVdM5BvWhmV5nZAWbWtXLKemQiItKspZOgLgQuJzTxzYomnexJ4qWX4KSTYOHCfEciItL4pTOSRN9cBNLYzZoFZ5wBmzbBUUfB1Knwla/kOyoRkcYrnZEkvp1subs/lPlwGqcPPoBTToHu3eGJJ+C88+Doo0OSGjgw39HVnTssWQIzZsB778FZZ0FhYb6jEpHmJp3bZhya8LwNcBzwJqAEBXz6aWjW274dXngBDj4YXnkFjj02JKmXXoKionxHWbOKipCMZs4MjzNmwKpVO9dfd10YZ/CXv4QD1Y9TRHKk1uugdtvArBPwsLuPqLVwDuXjOqiNG0MimjcP/vEP+OpXd65bsCCs27QJXnwRDjkkp6GltHEjvPnmrsloyZKwrkULGDAASkvDdOih0LMn3HYb/P73sHUrXHIJXHMN7L9/buPevj0k0n33ze3rikjmpXsdVH0SVGtgjrv3r29w2ZDrBLV1K5x+eqg1PfkknHba7mUWL4ZjjoG1a0O5YdXHcs9BjPPm7UxEM2fC/PmwY0dYX1CwMxGVlkJxMXTokHxfy5aF+1tNmACtWsGVV8JPfwrdumX3PXz4IfzpT/Dgg1BeDg89FJpQRaTxSjdB4e41TsDThDH0pgDPEG5W+Nvatsv1dMghh3iu7NjhPnasO7jff3/NZZcscT/wQPeOHd1ffz038a1e7T5unHubNiFGcO/Wzf3kk92vvdb9mWfcV6yo374/+MD9v/7L3cx9r73cr7/eff36jIbvn33mPmmS+9e/Hl7HzP2EE9wPP9y9RQv3yZMz+3oikltAmafxvZ5OgjoqYToc6JXOjnM95TJB/fzn4chdd1165T/6yP3LX3Zv39795ZezF9eOHe6PPea+zz7uLVuGJDVpkvuiRWFdJs2b537mmeE4dO/ufttt7lu2NGyfc+a4f//7IZmCe+/e4RgvWRLWb9gQklSrVu5PPdXw9yAi+ZHJBNUXaJMw3xYoSGfnuZxylaDuuScctUsuqduX/rJl7v37u7dt6/7ii5mP68MP3b/xjRBbSYn7W29l/jWSeeMN9+OPD6/bq1eoUW7dmv7269a533efe2lp2Efr1u7f/Kb7Cy+4b9uWvHxpqfsee7g//3zm3oeI5E4mE1QZsEfC/B7AzHR2nsspFwnq8cdDc9OIEXX7Eq60YoX7oEHue+7p/txzmYlp2zb3O+4ItbP27d1vvz35F3u2TZ3qPmxY+EQddJD7o4+6b9+evOyOHe6vvRaaSdu1C9sMHBhir6io/bVWr3YfMiQ0Yf7jH5l9H9myY4f7pk3un3zivmCB+6xZ4Ri89Vb4cbF+feZruSJxlW6CSmc089nuPqTasrfdPVadp7PdSeK11+DrXw8dCV56Cdq1q99+Vq0K+5k/H/72t+SdK9L19tuhV93MmXDyyXDvvdCnT/3311Du8PTTMH586JxRVBQ6VpxyCpjBihWhk8MDD4Trqzp0gNGj4aKLQicNS3Yf5RQqKkIHlMWLQweU4cOz976q++ST0Gtz/fowbdgQpsrnqR63b695v61aQZcu0LVreEx8XtOyvfeGli1z895FMiFjvfjM7EXg9+4+JZo/Hfieux+XkUgzJJsJav788AW4zz7wz382vOfamjVw4onw1lvw2GPhQti62LIFrr8ebr45xHLnnTByZN2+4LNp+/bwvn7xi3AR89e+Fo7d00/Dtm1h/uKL4ZvfTN1rMB2ffBJG7Vi+PPxoKC3N3HtIZfJkuPTS8DesZAYdO4Zpr712faxpWbt2odv/mjWwenXyx8rn69aFHwDJ9O4NV10VEn19fziJ5FIme/H1A6YDS6PpX4S77Oa9WS9xylYT30cfhXMr++7rvnhx5va7dq37YYeFzgyTJqW/3YsvuvfrF5rFLrzQfdWqzMWUaV984f6HP7jvv797jx7uV13l/s47mX2N8vLQS7Jz5+yed1u71v2888JxP/RQ9+nTw3nFDRtSN2Vm0rZt4W+9cKH7jBnhHN2kSe6//7378OFe1VPzl790X7ky+/GINASZOgdVVTDcqLBjuuVzPWUjQa1Z415YGLqIZ+PLb/169yMrrxRAAAASFUlEQVSOCF2nH3qo5rIrV7qff/7OczyN5dyLezi3ks0v8SVL3A84IHxBz52b+f2//HLoUdiyZeim/8UXmX+Nhnr9dffTTgufj3bt3H/wA/elS/MdlUhyGUtQwG+AzgnzXYBfp7PzXE6ZTlBbtrgfeWToVTZ1akZ3vYuNG92PPTZ0vnjggd3X79jh/sgjoSt3q1ahi/vmzdmLp7FasMB9v/1CF/v//Ccz+/zss1DrM3P/0pdCrSnu5s51//a3w2elVavwo2b+/HxHJbKrdBNUOueg3nL3odWWvenuxXVocsy6TJ6D2r4dRo0KnRgefTQ8z6YtW8JI6H//O/zhD/Cd74TlixfDZZftHIXi/vth0KDsxtKY/ec/4ZxUq1bw6qvQr1/99zV3bhh/cO7c8Pe49VZo3z5zsWbb0qVhiKr774fNm2HEiDDyx9e+lvnXWr8+jOb/5pvhtVq12n1q3Tr58mRT587w5S/DHntkPlaJh0yeg5oD7Jkw3xaYn072y+WUqRrUjh3uV1wR6pa33ZaRXaZly5ad1zHdfrv7zTeHa6Y6dAjnGfLRdbwxmjPHvWvX0CT34Yd13377dvdbbgnXWe29dxh1ozFbuTJc7Ny1a/hsHXFEeE/17dK+aVNoTrzjDvcxY9wPPtirRivJ5NSqlfuAAe7nnuv+q1+5P/FEqCXn4nyfZB8ZrEH9BBgB/CladAEwxd1vqnf6zIJM1aB++1v42c/gxz+GW27JQGB18MUXoTfek0+G+REj4O674YADchtHY/fmm2Gg3u7dQ00q3YFtly6F88+Hl18O4yzefz/06JHVUHNm06bQvf+WW+Cjj0JN/Kc/DZ+3VinuafD55zBnDpSVhUsZysp2Hctx//2hpCSM5VhSEgZE7tIl9NRsyFRRES5TqJwWLdoZU9u24RY2hYVhGjQoPO63X3x6sUrtMjpYrJmdBBwPGLAG2M/dL29wlBmUiQT10EPhC+pb34KHHw6je+fa1q3w61+Ha4jOPFP/dPU1fXq43qxXr5Bw9tkndVl3+Otf4fLLQ/PunXfCBRc0zWO/dStMmgS/+11INn36hC7q3/52aFJOTEZz5oTyEJJ9ZSKqnHI1ov3GjfDOOyFZzZ27M3F98snOMl277kxahYXQv39oVvzii/AeKh9TPU+1vjIZN0T79nDqqaF5NR/fKXGU6QQ1BPgWcC6wGHjc3e9ucJQZ1NAE9fzz4aLZo46CZ59V+3dT8Npr4V5dBx4I06aFL9nqVq8O5/kmTw5fIA8/3DzuebVjR/ic//a34dq+RJ067ZqIDj00XGsVt4RdURGSbPXEtX59w/bbuvXOKRMXQG/YEJJdr17h2r9Ro8IxzfXxrPyqj8PfscEJysy+DIwCRgOrgMeAq9w9j2MVpNaQBLVjx877Nb3ySriIUpqGqVPhG98I97n6xz/CCfhKL74IY8eGm07+8pehyas5jsjw+uuhI07//uGLs1+/xvtL3z00Yb7/fnjeunX4sVmZcJI9T1zWqlXmv8A3bIApU8LF688/H5JV375w7rkhWRUVZS9pLFkSWhCmTQuPy5eH2+z067dzOvDAnY+5utA7EwlqB/AacJG7L4yWLXL3WP6+bGgN6tNPQ/POfvtlMCiJheeeC+eUiotDUmrVCq6+Gu66K3wpP/JIWCeSbWvXhnPMjz0WPovbt4ceiyNHhmQ1YEDD9r906a4JqfJmpN27hzt8H3hgaMr94IMwrVu36/b77bdr8kqcunXLXCLNRII6k1CD+hrwPDAJ+KO7981MiJmVjzvqSuPx5JNwzjlhOKS1a+Hdd8NNF3/3u3DiXSTXVq6E//3fkKxefjm05BQWhmQ1ciQcdFDt+ygv35mMXn55Z4eSbt3C6YpjjgmJaeDA3ZOLe2jirkxW1adly3Ytv9deO2tbV1216x3E6yqTY/G1B84gNPUdC/wFeMLd/17/8DJPCUpq89hjoQPMPvuEu/SeeGK+IxIJPvkkXHf52GOhyRVCrX7kyNAUWFAQli1btjMhTZsWEgmE3pOJCamwsOHNtFu27FrbSpzuvTf0lK2vrNzy3cy6At8ERrp7A8LLPCUoSce8eeFkdeK5KJE4KS+H//mf0Ntyxoyw7NBDQ81/wYIw37lzSEhHHx2mwYMb13nDrCSoOFOCEpGmZvHi0MP0qad2nkc65piQkBpzhx4lKBERiaV0E1QjqhSKiEhzogQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxpAQlIiKxlNUEZWYnmdl7ZrbQzK5Osr63mU0zs7fMbI6ZnRItLzCzLWY2O5r+kM04RUQkflLcS7PhzKwlcA/wdaAcmGlmU9z9nYRi1wCT3f1eMxsAPAsUROs+cPch2YpPRETiLZs1qFJgobsvcvcvCKOhn16tjAOVd1/qBFQbP1dERJqrbCaonsBHCfPl0bJE1wHnmVk5ofZ0ZcK6vlHT3ytmdkSyFzCzcWZWZmZlFRUVGQxdRETyLZsJKtmtraoP/Dca+LO79wJOAR42sxbAcqC3uw8FfgT81cx2u8+tu09w9xJ3L+nRo0eGwxcRkXzKZoIqBw5ImO/F7k14FwGTAdz930AboLu7f+7uq6Lls4APgC9nMVYREYmZbCaomcBBZtbXzPYg3J13SrUyS4HjAMysPyFBVZhZj6iTBWZ2IHAQsCiLsYqISMxkrRefu28zsyuAF4CWwIPuPt/MrgfK3H0K8GPgfjP7IaH5b6y7u5kdCVxvZtuA7cCl7r46W7GKiEj86H5QIiKSU7oflIiINGpKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktKUCIiEktZTVBmdpKZvWdmC83s6iTre5vZNDN7y8zmmNkpCet+Fm33npmdmM04RUQkflpla8dm1hK4B/g6UA7MNLMp7v5OQrFrgMnufq+ZDQCeBQqi56OAgcD+wEtm9mV3356teEVEJF6yWYMqBRa6+yJ3/wKYBJxerYwDe0XPOwHLouenA5Pc/XN3XwwsjPYnIiLNRDYTVE/go4T58mhZouuA88ysnFB7urIO22Jm48yszMzKKioqMhW3iIjEQDYTlCVZ5tXmRwN/dvdewCnAw2bWIs1tcfcJ7l7i7iU9evRocMAiIhIfWTsHRaj1HJAw34udTXiVLgJOAnD3f5tZG6B7mtuKiEgTls0a1EzgIDPra2Z7EDo9TKlWZilwHICZ9QfaABVRuVFmtqeZ9QUOAmZkMVYREYmZrNWg3H2bmV0BvAC0BB509/lmdj1Q5u5TgB8D95vZDwlNeGPd3YH5ZjYZeAfYBlyuHnwiIs2LhXzQ+JWUlHhZWVm+wxARkVqY2Sx3L6mtnEaSEBGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCEhGRWFKCAiZOhIICaNEiPE6cmO+IREQkm4PFNgoTJ8K4cbB5c5j/8MMwDzBmTP7iEhFp7pp9DWr8+J3JqdLmzWG5iIjkT7NPUEuX1m25iIjkRrNPUL171225iIjkRrNPUDfcAO3a7bqsXbuwXERE8qfZJ6gxY2DCBOjTB8zC44QJ6iAhIpJvzb4XH4RkpIQkIhIvzb4GJSIi8aQEJSIisaQEJSIisaQEJSIisaQEJSIisWTunu8YMsLMKoAP8x1HmroDK/MdRB00pngbU6zQuOJtTLFC44q3McUKDY+3j7v3qK1Qk0lQjYmZlbl7Sb7jSFdjircxxQqNK97GFCs0rngbU6yQu3jVxCciIrGkBCUiIrGkBJUfE/IdQB01pngbU6zQuOJtTLFC44q3McUKOYpX56BERCSWVIMSEZFYUoISEZFYUoLKEjM7wMymmdm7ZjbfzL6fpMzRZrbOzGZH0y/yEWsUyxIzmxvFUZZkvZnZXWa20MzmmFlxPuKMYjk44ZjNNrP1ZvaDamXyemzN7EEz+9TM5iUs62pmL5rZguixS4ptz4/KLDCz8/MU681m9p/ob/2EmXVOsW2Nn5scxnudmX2c8Pc+JcW2J5nZe9Hn+Oo8xfpYQpxLzGx2im1zemxTfWfl9XPr7pqyMAH7AcXR847A+8CAamWOBp7Jd6xRLEuA7jWsPwV4DjDgq8Ab+Y45iqsl8Anhwr/YHFvgSKAYmJew7Cbg6uj51cDvkmzXFVgUPXaJnnfJQ6wnAK2i579LFms6n5scxnsdcFUan5UPgAOBPYC3q/9P5iLWautvBX4Rh2Ob6jsrn59b1aCyxN2Xu/ub0fMNwLtAz/xG1SCnAw95MB3obGb75Tso4DjgA3eP1Sgi7v4qsLra4tOBv0TP/wKckWTTE4EX3X21u68BXgROylqgJI/V3f/u7tui2elAr2zGUBcpjm06SoGF7r7I3b8AJhH+JllTU6xmZsC5wKPZjCFdNXxn5e1zqwSVA2ZWAAwF3kiy+jAze9vMnjOzgTkNbFcO/N3MZpnZuCTrewIfJcyXE4+EO4rU/+BxObaV9nH35RC+DIC9k5SJ43G+kFB7Tqa2z00uXRE1ST6Yohkqbsf2CGCFuy9IsT5vx7bad1bePrdKUFlmZh2Ax4EfuPv6aqvfJDRNFQG/B57MdXwJDnf3YuBk4HIzO7LaekuyTV6vUTCzPYARwP8kWR2nY1sXsTrOZjYe2AZMTFGkts9NrtwL9AOGAMsJTWfVxerYAqOpufaUl2Nby3dWys2SLGvwsVWCyiIza034Q0909/+tvt7d17v7xuj5s0BrM+ue4zArY1kWPX4KPEFoDklUDhyQMN8LWJab6FI6GXjT3VdUXxGnY5tgRWWzaPT4aZIysTnO0YnuU4ExHp1oqC6Nz01OuPsKd9/u7juA+1PEEadj2wo4C3gsVZl8HNsU31l5+9wqQWVJ1L78APCuu9+Wosy+UTnMrJTw91iVuyir4mhvZh0rnxNOkM+rVmwK8O2oN99XgXWV1f48SvkLNC7HtpopQGXvpvOBp5KUeQE4wcy6RM1UJ0TLcsrMTgJ+Coxw980pyqTzucmJaudDz0wRx0zgIDPrG9W+RxH+JvlwPPAfdy9PtjIfx7aG76z8fW5z1UOkuU3AcEIVdw4wO5pOAS4FLo3KXAHMJ/Qmmg58LU+xHhjF8HYUz/hoeWKsBtxD6AU1FyjJ8/FtR0g4nRKWxebYEhLncmAr4dflRUA3YCqwIHrsGpUtAf6YsO2FwMJouiBPsS4knFOo/Oz+ISq7P/BsTZ+bPMX7cPS5nEP4Qt2verzR/CmE3mkf5CLeZLFGy/9c+VlNKJvXY1vDd1bePrca6khERGJJTXwiIhJLSlAiIhJLSlAiIhJLSlAiIhJLSlAiIhJLSlAiWWJm223XUdczNnq2mRUkjpAt0hS1yncAIk3YFncfku8gRBor1aBEciy6z8/vzGxGNH0pWt7HzKZGA55ONbPe0fJ9LNyT6e1o+lq0q5Zmdn90756/m1nbqPz3zOydaD+T8vQ2RRpMCUoke9pWa+IbmbBuvbuXAncDd0TL7ibc0mQwYXDWu6LldwGveBj4tpgwsgDAQcA97j4QWAucHS2/Ghga7efSbL05kWzTSBIiWWJmG929Q5LlS4Bj3X1RNDjnJ+7ezcxWEobo2RotX+7u3c2sAujl7p8n7KOAcP+dg6L5nwKt3f3XZvY8sJEwgvuTHg2aK9LYqAYlkh+e4nmqMsl8nvB8OzvPKX+DMG7iIcCsaORskUZHCUokP0YmPP47ev4vwgjbAGOA16PnU4HLAMyspZntlWqnZtYCOMDdpwE/AToDu9XiRBoD/bISyZ62ZjY7Yf55d6/sar6nmb1B+JE4Olr2PeBBM/t/QAVwQbT8+8AEM7uIUFO6jDBCdjItgUfMrBNhBPrb3X1txt6RSA7pHJRIjkXnoErcfWW+YxGJMzXxiYhILKkGJSIisaQalIiIxJISlIiIxJISlIiIxJISlIiIxJISlIiIxNL/D9jkMfx9x62NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # 그래프를 초기화합니다\n",
    "acc = history_dict['acc']\n",
    "val_acc = history_dict['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "점선은 훈련 손실과 정확도이고 실선은 검증 손실과 정확도입니다. 신경망의 무작위한 초기화 때문에 사람마다 결과거 조금 다를 수 있습니다.\n",
    "\n",
    "여기에서 볼 수 있듯이 훈련 손실이 에포크마다 감소하고 훈련 정확도는 에포크마다 증가합니다. 경사 하강법 최적화를 사용했을 때 반복마다 최소화되는 것이 손실이므로 기대했던 대로입니다. 검증 손실과 정확도는 이와 같지 않습니다. 4번째 에포크에서 그래프가 역전되는 것 같습니다. 이것이 훈련 세트에서 잘 작동하는 모델이 처음 보는 데이터에 잘 작동하지 않을 수 있다고 앞서 언급한 경고의 한 사례입니다. 정확한 용어로 말하면 과대적합되었다고 합니다. 2번째 에포크 이후부터 훈련 데이터에 과도하게 최적화되어 훈련 데이터에 특화된 표현을 학습하므로 훈련 세트 이외의 데이터에는 일반화되지 못합니다.\n",
    "\n",
    "이런 경우에 과대적합을 방지하기 위해서 3번째 에포크 이후에 훈련을 중지할 수 있습니다. 일반적으로 4장에서 보게 될 과대적합을 완화하는 다양한 종류의 기술을 사용할 수 있습니다.\n",
    "\n",
    "처음부터 다시 새로운 신경망을 4번의 에포크 동안만 훈련하고 테스트 데이터에서 평가해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "25000/25000 [==============================] - 1s 54us/step - loss: 0.4749 - acc: 0.8217\n",
      "Epoch 2/4\n",
      "25000/25000 [==============================] - 1s 48us/step - loss: 0.2658 - acc: 0.9097\n",
      "Epoch 3/4\n",
      "25000/25000 [==============================] - 1s 48us/step - loss: 0.1982 - acc: 0.9299\n",
      "Epoch 4/4\n",
      "25000/25000 [==============================] - 1s 48us/step - loss: 0.1679 - acc: 0.9404\n",
      "25000/25000 [==============================] - 1s 50us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.3231545869159698, 0.87348]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 단순한 방식으로도 87%의 정확도를 달성했습니다. 최고 수준의 기법을 사용하면 95%에 가까운 성능을 얻을 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련된 모델로 새로운 데이터에 대해 예측하기\n",
    "\n",
    "모델을 훈련시킨 후에 이를 실전 환경에서 사용하고 싶을 것입니다. `predict` 메서드를 사용해서 어떤 리뷰가 긍정일 확률을 예측할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1402615 ],\n",
       "       [0.9997029 ],\n",
       "       [0.29552558],\n",
       "       ...,\n",
       "       [0.07234979],\n",
       "       [0.04342841],\n",
       "       [0.48153415]], dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기에서처럼 이 모델은 어떤 샘플에 대해 확신을 가지고 있지만(0.99 또는 그 이상, 0.01 또는 그 이하) 어떤 샘플에 대해서는 확신이 부족합니다(0.6, 0.4). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 추가 실험\n",
    "\n",
    "* 여기에서는 두 개의 은닉층을 사용했습니다. 한 개 또는 세 개의 은닉층을 사용하고 검증과 테스트 정확도에 어떤 영향을 미치는지 확인해 보세요.\n",
    "* 층의 은닉 유닛을 추가하거나 줄여 보세요: 32개 유닛, 64개 유닛 등\n",
    "* `binary_crossentropy` 대신에 `mse` 손실 함수를 사용해 보세요.\n",
    "* `relu` 대신에 `tanh` 활성화 함수(초창기 신경망에서 인기 있었던 함수입니다)를 사용해 보세요.\n",
    "\n",
    "다음 실험을 진행하면 여기에서 선택한 구조가 향상의 여지는 있지만 어느 정도 납득할 만한 수준이라는 것을 알게 것입니다!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 정리\n",
    "\n",
    "다음은 이 예제에서 배운 것들입니다:\n",
    "\n",
    "* 원본 데이터를 신경망에 텐서로 주입하기 위해서는 꽤 많은 전처리가 필요합니다. 단어 시퀀스는 이진 벡터로 인코딩될 수 있고 다른 인코딩 방식도 있습니다.\n",
    "* `relu` 활성화 함수와 함께 `Dense` 층을 쌓은 네트워크는 (감성 분류를 포함하여) 여러 종류의 문제에 적용할 수 있어서 앞으로 자주 사용하게 될 것입니다.\n",
    "* (출력 클래스가 두 개인) 이진 분류 문제에서 네트워크는 하나의 유닛과 `sigmoid` 활성화 함수를 가진 `Dense` 층으로 끝나야 합니다. 이 신경망의 출력은 확률을 나타내는 0과 1 사이의 스칼라 값입니다.\n",
    "* 이진 분류 문제에서 이런 스칼라 시그모이드 출력에 대해 사용할 손실 함수는 `binary_crossentropy`입니다.\n",
    "* `rmsprop` 옵티마이저는 문제에 상관없이 일반적으로 충분히 좋은 선택입니다. 걱정할 거리가 하나 줄은 셈입니다.\n",
    "* 훈련 데이터에 대해 성능이 향상됨에 따라 신경망은 과대적합되기 시작하고 이전에 본적 없는 데이터에서는 결과가 점점 나빠지게 됩니다. 항상 훈련 세트 이외의 데이터에서 성능을 모니터링해야 합니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}