{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chap03-4. 영화 리뷰 분류: 이진 분류 문제\n",
    "이 notebook에서는 리뷰 텍스트 기반으로 영화 리뷰를 긍정 혹은 부정으로 나누는 이진 분류를 IMDB 데이터셋을 이용해 보여주겠습니다.  \n",
    "<br />\n",
    "\n",
    "### IMDB 데이터셋\n",
    "인터넷 영화 데이터베이스로부터 가져온 양극단의 리뷰 50,000개로 이루어진 데이터셋입니다.  \n",
    "이 데이터셋은 훈련 데이터 25,000개와 테스트 데이터 25,000개로 나뉘어 있고 각각 50%는 부정, 50%는 긍정 리뷰로 구성되어 있습니다.  \n",
    "<br />\n",
    "한편 IMDB 데이터셋은 Keras에 포함되어 있습니다.  \n",
    "이 데이터는 전처리되어 있어 각 리뷰(단어 시퀀스)가 숫자 시퀀스로 변환되어 있습니다. 여기서 각 숫자는 사전에 있는 고유한 단어를 나타냅니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imdb 데이터셋 로드하기\n",
    "from tensorflow.keras.datasets import imdb\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`imdb.load_data()`를 통해서 데이터를 로드할 수 있으며, 파라미터인 `num_words`는 가장 많은 빈도로 존재하는 단어 10000개만 사용하겠다는 것을 나타냅니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이렇게 불러온 데이터셋은 전처리가 진행된 데이터이고, 출력해보면 다음과 같은 결과가 나옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st train data:  [1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n",
      "1st train label:  1\n"
     ]
    }
   ],
   "source": [
    "print(\"1st train data: \", train_data[0])\n",
    "print(\"1st train label: \", train_labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위와 같이 각 리뷰 데이터는 단어 인덱스의 리스트입니다.(단어가 인코딩 된 것) 그리고 라벨에서 0은 **부정**을, 1은 **긍정**을 나타냅니다.  \n",
    "재미삼아 각 리뷰를 우리가 알아볼 수 있도록 원래의 텍스트로 돌려보도록 하겠습니다.  \n",
    "자주 사용된 10000개의 단어만 사용했으므로 인덱스에 포함되지 않은 단어는 ?로 출력이 되도록 하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index = imdb.get_word_index()\n",
    "\n",
    "def decode_review(review):\n",
    "    reverse_word_index = dict(\n",
    "        [(value, key) for (key, value) in word_index.items()])\n",
    "    decode = ' '.join(\n",
    "        [reverse_word_index.get(i - 3, '?') for i in review])\n",
    "    return decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decode_review(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위와 같이 영화에 대한 긍정적인 리뷰를 볼 수 있습니다.  \n",
    "<br />\n",
    "한편 keras에서 제공한 데이터셋과 같은 숫자 리스트는 바로 신경망에 넣어줄 수 없습니다.  \n",
    "이 리스트를 텐서(벡터)로 변환하는 과정을 거쳐야 합니다.  \n",
    "여기에서는 **one-hot encoding**을 사용하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train  (25000, 10000)\n",
      "x_test  (25000, 10000)\n"
     ]
    }
   ],
   "source": [
    "x_train = vectorize_sequences(train_data)\n",
    "x_test = vectorize_sequences(test_data)\n",
    "\n",
    "print(\"x_train \", x_train.shape)\n",
    "print(\"x_test \", x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "라벨 또한 벡터로 만들겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train  (25000,)\n",
      "y_test  (25000,)\n"
     ]
    }
   ],
   "source": [
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')\n",
    "\n",
    "print(\"y_train \", y_train.shape)\n",
    "print(\"y_test \", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "그리고 훈련하는 동안 처음 본 데이터에 대한 모델의 정확도를 측정하기 위해서 훈련 데이터에서 10000개의 샘플을 추출해 검증 세트를 만들겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 신경망에 주입할 데이터는 모두 준비가 되었으니, 신경망 모델을 만들겠습니다.  \n",
    "여기에서는 Sequential 클래스를 사용해 모델을 정의를 할 것인데, `relu` 활성화 함수를 사용한 Dense 층 두 개를 쌓고 마지막 층은 확률을 출력하기 위해 `sigmoid` 활성화 함수를 사용하겠습니다.  \n",
    "![neural networks structure](./images/nn-2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 손실함수와 옵티마이저를 선택하면 학습 할 준비가 끝납니다.  \n",
    "이 문제는 이진 분류 문제이고 신경망의 출력이 확률이기 때문에, `binary_crossentropy`손실이 적합합니다.(`mean_squared_error`도 사용가능합니다.)  \n",
    "또한 옵티마이저는 SGD 종류 중 하나인 `RMSProp`를 사용하겠습니다.  \n",
    "`metrics`에는 평가 할 지표를 넣으면 되는데 이진 분류에서의 정확도를 나타내는 `binary_accuracy`를 사용하겠습니다. ([[링크](https://keras.io/metrics/#available-metrics)] 참조)  \n",
    "케라스에는 `rmsprop`, `binary_crossentropy`, `accuracy`가 이미 포함되어 있지만 여기에서는 모듈을 불러와서 사용하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "from tensorflow.keras import losses\n",
    "from tensorflow.keras import metrics\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "             loss=losses.binary_crossentropy,\n",
    "             metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 컴파일이 끝나면 모델을 학습시키면 됩니다.  \n",
    "512개의 샘플씩 미니 배치를 만들어 20번의 에포크 동안 훈련시킵니다.  \n",
    "위에서 만든 검증 세트를 사용해서 훈련하는 동안의 정확도를 측정할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "15000/15000 [==============================] - 2s 143us/step - loss: 0.4974 - binary_accuracy: 0.7951 - val_loss: 0.3795 - val_binary_accuracy: 0.8655\n",
      "Epoch 2/20\n",
      "15000/15000 [==============================] - 1s 98us/step - loss: 0.2939 - binary_accuracy: 0.9038 - val_loss: 0.3006 - val_binary_accuracy: 0.8870\n",
      "Epoch 3/20\n",
      "15000/15000 [==============================] - 1s 89us/step - loss: 0.2146 - binary_accuracy: 0.9297 - val_loss: 0.2771 - val_binary_accuracy: 0.8904\n",
      "Epoch 4/20\n",
      "15000/15000 [==============================] - 1s 93us/step - loss: 0.1708 - binary_accuracy: 0.9437 - val_loss: 0.2768 - val_binary_accuracy: 0.8866\n",
      "Epoch 5/20\n",
      "15000/15000 [==============================] - 1s 96us/step - loss: 0.1357 - binary_accuracy: 0.9563 - val_loss: 0.2823 - val_binary_accuracy: 0.8878\n",
      "Epoch 6/20\n",
      "15000/15000 [==============================] - 1s 89us/step - loss: 0.1144 - binary_accuracy: 0.9631 - val_loss: 0.3021 - val_binary_accuracy: 0.8856\n",
      "Epoch 7/20\n",
      "15000/15000 [==============================] - 1s 82us/step - loss: 0.0944 - binary_accuracy: 0.9727 - val_loss: 0.3191 - val_binary_accuracy: 0.8815\n",
      "Epoch 8/20\n",
      "15000/15000 [==============================] - 1s 82us/step - loss: 0.0783 - binary_accuracy: 0.9780 - val_loss: 0.3450 - val_binary_accuracy: 0.8787\n",
      "Epoch 9/20\n",
      "15000/15000 [==============================] - 1s 87us/step - loss: 0.0653 - binary_accuracy: 0.9818 - val_loss: 0.3961 - val_binary_accuracy: 0.8691\n",
      "Epoch 10/20\n",
      "15000/15000 [==============================] - 1s 88us/step - loss: 0.0537 - binary_accuracy: 0.9859 - val_loss: 0.4247 - val_binary_accuracy: 0.8696\n",
      "Epoch 11/20\n",
      "15000/15000 [==============================] - 1s 94us/step - loss: 0.0428 - binary_accuracy: 0.9904 - val_loss: 0.4273 - val_binary_accuracy: 0.8740\n",
      "Epoch 12/20\n",
      "15000/15000 [==============================] - 1s 93us/step - loss: 0.0369 - binary_accuracy: 0.9911 - val_loss: 0.4510 - val_binary_accuracy: 0.8739\n",
      "Epoch 13/20\n",
      "15000/15000 [==============================] - 1s 90us/step - loss: 0.0307 - binary_accuracy: 0.9929 - val_loss: 0.4768 - val_binary_accuracy: 0.8733\n",
      "Epoch 14/20\n",
      "15000/15000 [==============================] - 1s 82us/step - loss: 0.0235 - binary_accuracy: 0.9945 - val_loss: 0.5131 - val_binary_accuracy: 0.8697\n",
      "Epoch 15/20\n",
      "15000/15000 [==============================] - 1s 81us/step - loss: 0.0191 - binary_accuracy: 0.9968 - val_loss: 0.5399 - val_binary_accuracy: 0.8707\n",
      "Epoch 16/20\n",
      "15000/15000 [==============================] - 1s 81us/step - loss: 0.0166 - binary_accuracy: 0.9969 - val_loss: 0.5722 - val_binary_accuracy: 0.8685\n",
      "Epoch 17/20\n",
      "15000/15000 [==============================] - 1s 81us/step - loss: 0.0089 - binary_accuracy: 0.9995 - val_loss: 0.6328 - val_binary_accuracy: 0.8642\n",
      "Epoch 18/20\n",
      "15000/15000 [==============================] - 1s 82us/step - loss: 0.0100 - binary_accuracy: 0.9991 - val_loss: 0.6419 - val_binary_accuracy: 0.8669\n",
      "Epoch 19/20\n",
      "15000/15000 [==============================] - 1s 81us/step - loss: 0.0096 - binary_accuracy: 0.9985 - val_loss: 0.6722 - val_binary_accuracy: 0.8666\n",
      "Epoch 20/20\n",
      "15000/15000 [==============================] - 1s 83us/step - loss: 0.0041 - binary_accuracy: 0.9999 - val_loss: 0.7095 - val_binary_accuracy: 0.8639\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                  partial_y_train,\n",
    "                  epochs=20,\n",
    "                  batch_size=512,\n",
    "                  validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`model.fit()` 메서드는 `History` 객체를 반환합니다. 이 객체는 훈련하는 동안 발생한 모든 정보를 담고 있는 딕셔너리인 history 속성을 가지고 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_loss', 'val_binary_accuracy', 'loss', 'binary_accuracy'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib을 통해서 훈련과 검증 손실, 훈련과 검증 정확도를 그려보도록 하겠습니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VOXZ//HPBYKIICDQouzuBAHF/ECLyCJV1AouqCAuuKG0alttLY9Ya6k8xaUuUGrBtUqUWn2sqCC2SuvSigSrKCKCCBoEgVRAxIXA9fvjPhmHmIQJmTMzSb7v12teOXPmnjPXTJJzzb2c+zZ3R0REBKBetgMQEZHcoaQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKklZmVt/MNptZh3SWzSYzO8DM0j5228wGmdmKpPtLzKxvKmV34bXuMbNrd/X5lRz3RjN7IN3HlezZLdsBSHaZ2eaku42Br4Bt0f1L3b2gKsdz921Ak3SXrQvc/eB0HMfMLgbOcff+Sce+OB3HltpPSaGOc/fESTn6Jnqxu/+9ovJmtpu7l2QiNhHJPDUfSaWi5oE/m9kjZvYZcI6ZHWVmr5rZBjNbbWaTzKxBVH43M3Mz6xTdnx49PtvMPjOzf5tZ56qWjR4/wczeM7ONZjbZzF4xs1EVxJ1KjJea2TIz+9TMJiU9t76Z3W5mxWa2HBhcyeczzsxmlNk3xcxui7YvNrPF0ft5P/oWX9Gxisysf7Td2MweimJbBBxRpux1ZrY8Ou4iMxsS7e8G/B7oGzXNrU/6bG9Iev5l0XsvNrO/mtk+qXw2O2Nmp0bxbDCzF8zs4KTHrjWzj81sk5m9m/RejzSz16P9n5jZLam+nsTA3XXTDXcHWAEMKrPvRuBr4GTCl4g9gP8H9CbUNPcD3gMuj8rvBjjQKbo/HVgP5AMNgD8D03eh7HeAz4Ch0WNXAVuBURW8l1RifBJoBnQC/lv63oHLgUVAO6Al8GL4Vyn3dfYDNgN7Jh17LZAf3T85KmPAQOALoHv02CBgRdKxioD+0fatwD+AFkBH4J0yZc8E9ol+J2dHMXw3euxi4B9l4pwO3BBtHxfFeBjQCPgD8EIqn0057/9G4IFou0sUx8Dod3QtsCTa7gqsBNpEZTsD+0Xb84ER0XZToHe2/xfq8k01BUnFy+7+lLtvd/cv3H2+u89z9xJ3Xw5MA/pV8vzH3L3Q3bcCBYSTUVXL/gB4w92fjB67nZBAypVijL91943uvoJwAi59rTOB2929yN2LgYmVvM5y4G1CsgL4PvCpuxdGjz/l7ss9eAF4Hii3M7mMM4Eb3f1Td19J+Paf/LqPuvvq6HfyMCGh56dwXICRwD3u/oa7fwmMBfqZWbukMhV9NpUZDsx09xei39FEQmLpDZQQElDXqAnyg+izg5DcDzSzlu7+mbvPS/F9SAyUFCQVHyXfMbNDzOwZM1tjZpuA8UCrSp6/Jml7C5V3LldUdt/kONzdCd+sy5VijCm9FuEbbmUeBkZE22dH90vj+IGZzTOz/5rZBsK39Mo+q1L7VBaDmY0yszejZpoNwCEpHhfC+0scz903AZ8CbZPKVOV3VtFxtxN+R23dfQlwNeH3sDZqjmwTFb0AyAOWmNlrZnZiiu9DYqCkIKkoOxxzKuHb8QHuvhdwPaF5JE6rCc05AJiZseNJrKzqxLgaaJ90f2dDZh8FBplZW0KN4eEoxj2Ax4DfEpp2mgPPpRjHmopiMLP9gLuAMUDL6LjvJh13Z8NnPyY0SZUerymhmWpVCnFV5bj1CL+zVQDuPt3d+xCajuoTPhfcfYm7Dyc0Ef4OeNzMGlUzFtlFSgqyK5oCG4HPzawLcGkGXvNpoKeZnWxmuwE/BlrHFOOjwE/MrK2ZtQR+UVlhd18DvAw8ACxx96XRQ7sDDYF1wDYz+wFwbBViuNbMmlu4juPypMeaEE786wj58RJCTaHUJ0C70o71cjwCXGRm3c1sd8LJ+SV3r7DmVYWYh5hZ/+i1f07oB5pnZl3MbED0el9Et+2EN3CumbWKahYbo/e2vZqxyC5SUpBdcTVwPuEffiqhQzhW7v4JcBZwG1AM7A/8h3BdRbpjvIvQ9v8WoRP0sRSe8zCh4zjRdOTuG4CfAk8QOmuHEZJbKn5FqLGsAGYDDyYddyEwGXgtKnMwkNwO/zdgKfCJmSU3A5U+/1lCM84T0fM7EPoZqsXdFxE+87sICWswMCTqX9gduJnQD7SGUDMZFz31RGCxhdFttwJnufvX1Y1Hdo2FplmRmsXM6hOaK4a5+0vZjkektlBNQWoMMxscNafsDvySMGrltSyHJVKrKClITXI0sJzQNHE8cKq7V9R8JCK7QM1HIiKSoJqCiIgk1LgJ8Vq1auWdOnXKdhgiIjXKggUL1rt7ZcO4gRqYFDp16kRhYWG2wxARqVHMbGdX5gNqPhIRkSRKCiIikqCkICIiCTWuT6E8W7dupaioiC+//DLboUgKGjVqRLt27WjQoKKpeUQkW2pFUigqKqJp06Z06tSJMHmm5Cp3p7i4mKKiIjp37rzzJ4hIRtWK5qMvv/ySli1bKiHUAGZGy5YtVasTyVG1IikASgg1iH5XIrmr1iQFEZHaassW+MUvYGVKVxpUT6xJIZrVcomZLTOzseU8fruZvRHd3ouWFaxxiouLOeywwzjssMNo06YNbdu2Tdz/+uvUpoW/4IILWLJkSaVlpkyZQkFBQTpC5uijj+aNN95Iy7FEJD7/+Ad07w433wyzZsX/erF1NEfz3U8hLGReBMw3s5nu/k5pGXf/aVL5K4DD44onWUEBjBsHH34IHTrAhAkwshpLjLRs2TJxgr3hhhto0qQJP/vZz3Yo4+64O/XqlZ+H77///p2+zo9+9KNdD1JEapRNm+Caa2DqVDjggJAc+vWL/3XjrCn0Apa5+/JoFaUZhPVrKzKCsExgrAoKYPToUA1zDz9Hjw77023ZsmXk5eUxcuRIunbtyurVqxk9ejT5+fl07dqV8ePHJ8qWfnMvKSmhefPmjB07lh49enDUUUexdu1aAK677jruuOOORPmxY8fSq1cvDj74YP71r38B8Pnnn3P66aeTl5fHsGHDyM/P32mNYPr06XTr1o1DDz2Ua6+9FoCSkhLOPffcxP5JkyYBcPvtt5OXl0f37t0555xz0v6ZiUioEXTtCnffDVdfDW++mZmEAPEOSW0LfJR0vwjoXV5BM+tIWMz7hQoeHw2MBujQYWdrqFdu3LjQPpdsy5awvzq1hYq8++67PPjgg+Tn5wMwceJE9t57b0pKShgwYADDhg0jLy9vh+ds3LiRfv36MXHiRK666iruu+8+xo79Vusb7s5rr73GzJkzGT9+PM8++yyTJ0+mTZs2PP7447z55pv07Nmz0viKioq47rrrKCwspFmzZgwaNIinn36a1q1bs379et566y0ANmwILXs333wzK1eupGHDhol9IpIexcXw05/CQw+FpPD449CrV2ZjyJWO5uHAY+6+rbwH3X2au+e7e37r1jud5K9SH35Ytf3Vtf/++ycSAsAjjzxCz5496dmzJ4sXL+add9751nP22GMPTjjhBACOOOIIVqxYUe6xTzvttG+Vefnllxk+fDgAPXr0oGvXrpXGN2/ePAYOHEirVq1o0KABZ599Ni+++CIHHHAAS5Ys4corr2TOnDk0a9YMgK5du3LOOedQUFCgi89E0uixxyAvDx55BK6/HhYsyHxCgHiTwiqgfdL9dtG+8gwnA01HEPoQqrK/uvbcc8/E9tKlS7nzzjt54YUXWLhwIYMHDy53vH7Dhg0T2/Xr16ekpKTcY+++++47LbOrWrZsycKFC+nbty9Tpkzh0ksvBWDOnDlcdtllzJ8/n169erFtW7l5XERStGYNnH46nHEGtG8PhYXw619D9O+dcXEmhfnAgWbW2cwaEk78M8sWMrNDgBbAv2OMJWHCBGjceMd9jRuH/XHbtGkTTZs2Za+99mL16tXMmTMn7a/Rp08fHn30UQDeeuutcmsiyXr37s3cuXMpLi6mpKSEGTNm0K9fP9atW4e7c8YZZzB+/Hhef/11tm3bRlFREQMHDuTmm29m/fr1bCnbFiciKXGHBx8MtYNnnoGJE+HVV6FHj+zGFVufgruXmNnlwBygPnCfuy8ys/FAobuXJojhwAzP0Lqgpf0G6Rx9lKqePXuSl5fHIYccQseOHenTp0/aX+OKK67gvPPOIy8vL3ErbfopT7t27fjNb35D//79cXdOPvlkTjrpJF5//XUuuugi3B0z46abbqKkpISzzz6bzz77jO3bt/Ozn/2Mpk2bpv09iNR2H34Il14Kzz4LffrAvffCwQdnO6qgxq3RnJ+f72UX2Vm8eDFdunTJUkS5paSkhJKSEho1asTSpUs57rjjWLp0KbvtllvTXOl3JnXR9u1hiOk114SawsSJ8MMfQgUj1dPKzBa4e/7OyuXWmUKqbfPmzRx77LGUlJTg7kydOjXnEoJIXbR0KVx8Mbz4IgwaFIab5uLKwjpb1DLNmzdnwYIF2Q5DRCLFxaFGMHkyNGoUmoouuABydQowJQURkRhs3gx33AG33BK2zz0X/vd/Yd99sx1Z5ZQURETS6KuvYNo0uPFGWLsWTjklbO/kkqGcoaQgIpIG27aF6XKuvz5MnzNgADz5JBx5ZLYjq5pcuaJZRKRGcg8n/x494PzzoVUreO45eP75mpcQQEkhLQYMGPCtC9HuuOMOxowZU+nzmjRpAsDHH3/MsGHDyi3Tv39/yg7BLeuOO+7Y4SKyE088MS3zEt1www3ceuut1T6OSG31j3/A974Xmoi2boW//AXmz4fvfz93O5J3RkkhDUaMGMGMGTN22DdjxgxGjBiR0vP33XdfHnvssV1+/bJJYdasWTRv3nyXjycilVuwAI4/PjQRFRXBPffAokUwbFjNTQallBTSYNiwYTzzzDOJBXVWrFjBxx9/TN++fRPXDfTs2ZNu3brx5JNPfuv5K1as4NBDDwXgiy++YPjw4XTp0oVTTz2VL774IlFuzJgxiWm3f/WrXwEwadIkPv74YwYMGMCAAQMA6NSpE+vXrwfgtttu49BDD+XQQw9NTLu9YsUKunTpwiWXXELXrl057rjjdnid8rzxxhsceeSRdO/enVNPPZVPP/008fqlU2mXTsT3z3/+M7HI0OGHH85nn322y5+tSC5ZsgTOPBPy80NiuPVWeO89uOgiqC2XA9WSt/GNn/wE0r2g2GGHhaFlFdl7773p1asXs2fPZujQocyYMYMzzzwTM6NRo0Y88cQT7LXXXqxfv54jjzySIUOGVLhO8V133UXjxo1ZvHgxCxcu3GHq6wkTJrD33nuzbds2jj32WBYuXMiVV17Jbbfdxty5c2nVqtUOx1qwYAH3338/8+bNw93p3bs3/fr1o0WLFixdupRHHnmEu+++mzPPPJPHH3+80vURzjvvPCZPnky/fv24/vrr+fWvf80dd9zBxIkT+eCDD9h9990TTVa33norU6ZMoU+fPmzevJlGjRpV4dMWyR3r18PixeH2yiuhI7lRo9CZfPXVsNde2Y4w/VRTSJPkJqTkpiN359prr6V79+4MGjSIVatW8cknn1R4nBdffDFxcu7evTvdu3dPPPboo4/Ss2dPDj/8cBYtWrTTye5efvllTj31VPbcc0+aNGnCaaedxksvvQRA586dOeyww4DKp+eGsL7Dhg0b6Bet8nH++efz4osvJmIcOXIk06dPT1w53adPH6666iomTZrEhg0bdEW15LTt22HFijAP0e23h0W3+vYNHcatW8Mxx4R5ih5/HC6/HJYvD7OY1saEALWwplDZN/o4DR06lJ/+9Ke8/vrrbNmyhSOOOAKAgoIC1q1bx4IFC2jQoAGdOnUqd7rsnfnggw+49dZbmT9/Pi1atGDUqFG7dJxSuyfNy1u/fv2dNh9V5JlnnuHFF1/kqaeeYsKECbz11luMHTuWk046iVmzZtGnTx/mzJnDIYccssuxiqTD11+HqSbeffebb/+LF4cmoeTJflu2hC5d4LTTws/SW/v2mZmjKNtqXVLIliZNmjBgwAAuvPDCHTqYN27cyHe+8x0aNGjA3LlzWblyZaXHOeaYY3j44YcZOHAgb7/9NgsXLgTCtNt77rknzZo145NPPmH27Nn0798fgKZNm/LZZ599q/mob9++jBo1irFjx+LuPPHEEzz00ENVfm/NmjWjRYsWvPTSS/Tt25eHHnqIfv36sX37dj766CMGDBjA0UcfzYwZM9i8eTPFxcV069aNbt26MX/+fN59910lBcmqf/4ThgwJ6x6X6tAhnOz79dvx5F/m36jOUVJIoxEjRnDqqafuMBJp5MiRnHzyyXTr1o38/PydnhzHjBnDBRdcQJcuXejSpUuixtGjRw8OP/xwDjnkENq3b7/DtNujR49m8ODB7LvvvsydOzexv2fPnowaNYpe0fJNF198MYcffnilTUUV+dOf/sRll13Gli1b2G+//bj//vvZtm0b55xzDhs3bsTdufLKK2nevDm//OUvmTt3LvXq1aNr166JVeREsmH+fDj5ZGjXDv7wh3DiP+ggiEaESxmaOluyQr8zyYS33w41gWbN4KWXoG3bbEeUPalOnV0HWshEpC5atixcRNaoEfz973U7IVSFmo9EpNYpKgprFmzdGtYv2G+/bEdUc9SapFC6bKTkvprWZCk1y9q1oYbw6afwwgthDWRJXa1oPmrUqBHFxcU62dQA7k5xcbEuaJNYbNgQpp9YuRKeeQaicRpSBbHWFMxsMHAnUB+4x90nllPmTOAGwIE33f3sqr5Ou3btKCoqYt26ddWMWDKhUaNGtGvXLtthSC3z+edw0klhDqKnnoKjj852RDVTbEnBzOoDU4DvA0XAfDOb6e7vJJU5EPgfoI+7f2pm39mV12rQoAGdO3dOR9giUgN99RWceiq8+io8+mioLciuibP5qBewzN2Xu/vXwAxgaJkylwBT3P1TAHdfG2M8IlILlZTA8OHwt7/BfffB6adnO6KaLc6k0Bb4KOl+UbQv2UHAQWb2ipm9GjU3fYuZjTazQjMrVBORiJTavh0uvBD++leYPDksciPVk+2O5t2AA4H+wAjgbjP71kIA7j7N3fPdPb9169YZDlFEcpE7XHEFPPQQTJgQJquT6oszKawC2ifdbxftS1YEzHT3re7+AfAeIUmIiFTq2mvDtBXXXAP/8z/Zjqb2iDMpzAcONLPOZtYQGA7MLFPmr4RaAmbWitCctDzGmESkFvjtb2HiRLjssvBTlyilT2xJwd1LgMuBOcBi4FF3X2Rm481sSFRsDlBsZu8Ac4Gfu3txXDGJSM03ZUqoJYwcGbaVENKrVkyIJyJ1w4MPhs7koUPhL3+BBg2yHVHNoQnxRKRWeeIJuOACOPZYmDFDCSEuSgoiktM2bYKxY+Gss6B37zD8VLOkxEdJQURyUkkJTJ0KBxwAN90EI0bArFlaHCdutWaWVBGpPZ57Dq6+OiyS07dvSAb5O20Nl3RQTUFEcsbixWFSu+OPhy1b4LHHwvrKSgiZo6QgIlm3fn24OrlbN3j5ZbjlFnjnnTCPkYacZpaaj0Qka77+Gn7/e/jNb0KH8qWXwq9/DZrNJnuUFEQk49zDKKJrrglrKR9/PPzud9C1a7YjEzUfiUhG/ec/MHAgnHYaNGwIs2fDs88qIeQKJQURyYiPPw4Xnx1xRBhVNGUKvPkmDC53wnzJFjUfiUisPvoIbr89XHOwdWsYajpuHDT/1iT5kguUFEQkFosXw803w/TpoQ9hxAi44QbYf/9sRyaVUVIQkbSaNy9cgVw6HcWYMXDVVdCpU7Yjk1QoKYhItbmHNZInToS5c0PT0HXXhWsPNLy0ZlFSEJFdtm0bPP54SAb/+Q/su28YWnrJJdC0abajk12hpCAiVfbll2Ftg5tvhvffh4MOgnvvDQvf7L57tqOT6lBSEJGUbdoEf/xjGE20Zk2Yk+ixx+CUU6B+/WxHJ+mgpCAiO7VpU+g8njIFNm6EQYPCqKKBAzU3UW2jpCAilZo5E374w3Dx2emnwy9+oVlLa7NYr2g2s8FmtsTMlpnZ2HIeH2Vm68zsjeh2cZzxiEjqVq+GYcPCesh77w3//ndYF1kJoXaLLSmYWX1gCnACkAeMMLO8cor+2d0Pi273xBFLQUEYI12vXvhZUBDHq4jUDtu3w7Rp0KULPP00/O//woIFYSlMqf3ibD7qBSxz9+UAZjYDGAq8E+NrfktBAYweHRbsAFi5MtyHMFJCRL7x7rvh/+Oll2DAgDA1xYEHZjsqyaQ4m4/aAh8l3S+K9pV1upktNLPHzKx9eQcys9FmVmhmhevWratSEOPGfZMQSm3ZEvaLSPD11zB+PPToESaru+8+eP55JYS6KNuzpD4FdHL37sDfgD+VV8jdp7l7vrvnt67i5ZEffli1/SJ1zSuvwOGHw69+FaazXrw4zGaqUUV1U5xJYRWQ/M2/XbQvwd2L3f2r6O49wBHpDqJDh6rtF6krNm4Mo4qOPho2b4ZnnoFHHoHvfjfbkUk2xZkU5gMHmllnM2sIDAdmJhcws32S7g4BFqc7iAkToHHjHfc1bhz2i9RVTzwBeXmhz+AnP4FFi+DEE7MdleSC2JKCu5cAlwNzCCf7R919kZmNN7MhUbErzWyRmb0JXAmMSnccI0eGkRQdO4bqcMeO4b46maUuWrUqNBGddlqYqO7VV8PVyU2aZDsyyRXm7tmOoUry8/O9sLAw22GI1Chbt8I998DYsaFT+YYbwnTWDRpkOzLJFDNb4O47vcpEVzSL1GKbNoVkcOedYXDFsceGJiMtdCMVUVIQqYU++ggmTQpNpZs2Qb9+8Pvfww9+oFFFUjklBZFa5D//CesZ/PnPYeGbM84IayJragpJlZKCSA23fTs8+2xIBi+8EDqNr7gCfvzjMLBCpCqUFERqqC+/DNO43HYbvPMOtG0bFr255JKwHKbIrlBSEKlhiovhrrtg8mRYuzZMTfHQQ3DmmdCwYbajk5pOSUGkhli2LFxTcP/98MUXcMIJob9AC91IOikpiOSwbdtCf8Ef/gCzZ4frCs45J1xj0LVrtqOT2khJQSQHrV8fZir94x/hgw+gTRu47rowV1GbNtmOTmozJQWRHOEO8+aFWsGjj8JXX4XrC266CU45RVcfS2YoKYhk2eefh9lJ//CHcJ1B06Zw8cUwZoyaiCTzlBREsmTJktA8dP/9YRrrQw8No4pGjgyJQSQblBREMqikBJ56KtQK/v730CQ0bFjoK+jTR6OIJPuUFETSoKQkNAN9/nlYsGbz5m+2S38uXx46j1etgvbt4cYb4aKL1HEsuUVJQWQn3MOqZA88AJ9+uuOJvnT7yy9TO9Zxx8GUKXDSSbCb/vskB+nPUqQCpcnghhtgwQLYd1/o3BlatIB27cIcQ02awJ57prbdvHl4rkguU1IQKcMdZs0KyaCwMCSCe++Fc8/VsFCp/eJco1mkRimtGfTqFdYdWL8+LFCzZAlceKESgtQNSgpS55XWDHr33jEZvPde6AhWMpC6JNakYGaDzWyJmS0zs7GVlDvdzNzMtBSIZIx7mE/oyCNDx+/atXD33aFmoGQgdVVsScHM6gNTgBOAPGCEmeWVU64p8GNgXlyxiCRLTgYnngiffBKSwXvvhSuJNf201GVx1hR6Acvcfbm7fw3MAIaWU+43wE1AioP6RHaNe5hx9KijvkkG06YpGYgkizMptAU+SrpfFO1LMLOeQHt3f6ayA5nZaDMrNLPCdevWpT9SqfXefRe+972wBsHq1d8kg0suUTIQSZa1IalmVg+4DRi1s7LuPg2YBpCfn+/xRia1zb/+BSefDPXrw9SpMGqUEoFIReJMCquA9kn320X7SjUFDgX+YWHClzbATDMb4u6FMcYldciTT8Lw4eFis2efhf33z3ZEIrktzuaj+cCBZtbZzBoCw4GZpQ+6+0Z3b+Xundy9E/AqoIQgaTN1Kpx2GnTrBq+8ooQgkoqUkoKZ7W9mu0fb/c3sSjNrXtlz3L0EuByYAywGHnX3RWY23syGVDfwXVFUlI1XlUxzh1/+Ei67DAYPhrlz4TvfyXZUIjVDqjWFx4FtZnYAoW2/PfDwzp7k7rPc/SB339/dJ0T7rnf3meWU7R9nLeG3vw3z1a9cGdcrSC7YujWMJLrxxnAV8pNPhnmHRCQ1qSaF7dE3/1OBye7+c2Cf+MJKv7POgu3bwwImJSXZjkbi8PnnYdnK++6D668PVyVrJlKRqkk1KWw1sxHA+cDT0b4adb3nfvuFVa5eeQV+85tsRyPptnYtDBgQOpOnToVf/1oL1ojsilSTwgXAUcAEd//AzDoDD8UXVjzOPhvOOy80Lbz0UrajkXR5//2watnbb8MTT8Do0dmOSKTmMveqDfs3sxaEC84WxhNS5fLz872wcNe7Hj77DHr2hK++gjfegL33TmNwknGFheHq5O3bwzKXRx2V7YhEcpOZLXD3nc4vl+roo3+Y2V5mtjfwOnC3md1W3SCzoWlTeOQRWLMmXM1axZwoOWT2bOjfP3Qkv/KKEoJIOqTafNTM3TcBpwEPuntvYFB8YcUrPx8mTID/+78wEZrUPA88EK5SPvDAcMXywQdnOyKR2iHVpLCbme0DnMk3Hc012tVXw/e/Dz/5CbzzTrajkVS5h4R+wQWhY/mf/4R9atQ4OJHclmpSGE+4CO19d59vZvsBS+MLK3716sGDD4b1c4cPT33hdcmebdvgRz+C664LQ4ufeQb22ivbUYnULiklBXf/i7t3d/cx0f3l7n56vKHFr02b0Azx1ltwzTXZjkYq89//whlnwF13hd/Vgw9qUjuROKTa0dzOzJ4ws7XR7XEzaxd3cJlw4omhCWny5DB6RXLLBx/AlVdC+/bw17/CnXfCTTeFmp6IpF+q/1r3Eyaz2ze6PRXtqxUmToTDDgvt1B9/nO1oBGD+/HAV+gEHhIsOzzgD3nwzJAgRiU+qSaG1u9/v7iXR7QGgdYxxZdTuu4dhql98AeeeG9quJfNKrzXo1w969QpXJ//sZ6G28MADYbZTEYlXqkmh2MwiMenvAAASTklEQVTOMbP60e0coDjOwDLtkENg0iR44QW45ZZsR1O3fPllmKeoa1cYMiQkgdtug48+Ck1Fbdvu/Bgikh6pJoULCcNR1wCrgWGksGJaTXPhhaGZ4pe/hNdey3Y0tV9xcZhypGPHcCFho0ZQUBCmrfjpTzWySCQbUh19tNLdh7h7a3f/jrufAtT40UdlmYW1e9u2hREjYNOmbEdUOy1fDldcAR06hATcsyc8/zy8/nqYn6pBjZpqUaR2qc4YjqvSFkUOad48fFtdsQJ++MNsR1N7uMO8eXDmmeEq5KlTw/Zbb4XpKgYO1KymIrmgOrPN19p/4T594Fe/Crfjjw+dz1J1X38NL74ITz8dbu+/D82ahesMrrgC9t032xGKSFnVSQq1eiq5ceNCk8YPfxgmWjvggGxHVDOsXQuzZoUk8NxzYVbaRo1CTeDnPw/NQ02bZjtKEalIpUnBzD6j/JO/AXvEElGOqF8fpk+HHj1C/8Irr+gK2vK4h+sHSmsDr70W9rVtGxLAD34QEkLjxtmOVERSUWlScPdqfaczs8HAnUB94B53n1jm8cuAHwHbgM3AaHfPmenp2rcPQyVPPz10iN50U7Yjyg1btoShu6WJYNWq0B/QqxeMHx8SQY8e6iMQqYliW8HWzOoDU4DvA0XAfDObWeak/7C7/zEqPwS4DRgcV0y74rTT4NJL4eabYdCgMLNqXbRmTZhm4umnQ7Pal1+GyQSPPz4kgRNOgO9+N9tRikh1xTmDTC9gWTR53tfADGBocoFojYZSe5Kj/RS9eoVhkscdBy1awB/+kO2IMuPTT+Hee0MybNsWxoyBxYtDkvzb32D9enjsMRg1SglBpLaIraYAtAU+SrpfBPQuW8jMfkQY3toQGFjegcxsNDAaoEOHDmkPtDIFBWGkzNat4f6GDWH65oceCnMmHXNM7Wom+fxzmDkzTPvx7LPhfe+/P1x7bZhiPC+vdr1fEdlRlddoTvnAZsOAwe5+cXT/XKC3u19eQfmzgePd/fzKjlvdNZqrqlMnWLny2/vr1Qtz9eTlwWWXwXnnheGWNdFXX4UEMGNGSAhbtoSawVlnhU72I45QIhCp6dK6RvMuWgW0T7rfLtpXkRnAKTHGs0s+/LD8/du3w333hfWBr7wyjLm/5JJwVW5NUFISmoAuuiisK3HKKeH+eeeF1cw+/BB+97uwdKkSgkjdEWdSmA8caGadzawhMJww/XaCmR2YdPckcnA1t4paqzp2DFNtv/YaFBaGb9QFBeFbde/eYVbPL77IaKg7tX17GFp7xRWhJnDccfCXv4RJ6GbPhtWrwyI2xxyj9QpE6qrYmo8AzOxE4A7CkNT73H2CmY0HCt19ppndCQwCtgKfApe7+6LKjpnp5qOCAhg9OjSplGrcOMyRNHLkjmU3bAgrgt11F7z7buiUHjUqNC8ddFD8sW7ZEkYJrV4dbmW33347zDzaqFEYMTRiRBg1tEetvuJERCD15qNYk0IcMp0UICSGceNCk0qHDmHh+LIJIZl7aIK56y74v/8LTTXHHhues9deoTmm9FavXurbJSXfnOjLO/mXN4Ff/fphZNA++4T+kaFDw00zkIrULUoKOWLNmnAB3LRp4Vt6uuy5ZzjRt2kTfla03aqVmoJEREkh55SUwHvvhVXdtm8PtQn3irfLe2y33cK3/jZtNH+QiFRNqkkhzusUJMluu4XhqyIiuUwNCyIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKChlQUBDWMqhXL/wsKMh2RCIi5dMsqTEru3LbypXhPlS+UI+ISDaophCzceN2XMoTwv1x47ITj4hIZZQUYvbhh1XbLyKSTbEmBTMbbGZLzGyZmY0t5/GrzOwdM1toZs+bWcc448mGDh2qtl9EJJtiSwpmVh+YApwA5AEjzKzs2mP/AfLdvTvwGHBzXPFky4QJ0LjxjvsaNw77RURyTZw1hV7AMndf7u5fAzOAockF3H2uu5e2uL8KtIsxnqwYORKmTYOOHcEs/Jw2TZ3MIpKb4hx91Bb4KOl+EdC7kvIXAbPLe8DMRgOjATrUwHaXkSOVBESkZsiJjmYzOwfIB24p73F3n+bu+e6e37p168wGJyJSh8RZU1gFtE+63y7atwMzGwSMA/q5+1cxxiMiIjsRZ01hPnCgmXU2s4bAcGBmcgEzOxyYCgxx97UxxiIiIimILSm4ewlwOTAHWAw86u6LzGy8mQ2Jit0CNAH+YmZvmNnMCg4nIiIZEGufgrvPcveD3H1/d58Q7bve3WdG24Pc/bvuflh0G1L5EesmzZ0kIpmiuY9ynOZOEpFMyonRR1IxzZ0kIpmkpJDjNHeSiGSSkkKO09xJIpJJSgo5TnMniUgmKSnkOM2dJCKZpNFHNYDmThKRTFFNQUREEpQUREQkQUmhDtAV0SKSKvUp1HK6IlpEqkI1hVpOV0SLSFUoKdRyuiJaRKpCSaGW0xXRIlIVSgq1nK6IFpGqUFKo5dJxRbRGL4nUHRp9VAdU54pojV4SqVtUU5BKafSSSN2ipCCV0uglkbol1qRgZoPNbImZLTOzseU8foyZvW5mJWY2LM5YZNdo9JJI3RJbUjCz+sAU4AQgDxhhZnllin0IjAIejisOqR6NXhKpW+KsKfQClrn7cnf/GpgBDE0u4O4r3H0hsD3GOKQaNHpJpG6Jc/RRW+CjpPtFQO9dOZCZjQZGA3RQu0XGafSSSN1RIzqa3X2au+e7e37r1q2zHY5UgUYvidQscSaFVUD7pPvton1Sh2j0kkjNEmdSmA8caGadzawhMByYGePrSQ5Kx+gl9UmIZE5sScHdS4DLgTnAYuBRd19kZuPNbAiAmf0/MysCzgCmmtmiuOKR7Kju6KXSPomVK8H9mz4JJQaReJi7ZzuGKsnPz/fCwsJshyFVUFAQ+hA+/DDUECZMSL2TuVOnkAjK6tgRVqxIZ5QitZuZLXD3/J2WU1KQXFavXqghlGUG2zWQWSRlqSaFGjH6SOou9UmIZJaSguQ09UmIZJaSguS06l5RreskRKpGSUFy3siRoVN5+/bwsypXQqfjOgk1P0ldoqQgtVp1+yTU/CR1jZKC1GrV7ZNQ85PUNUoKUqtVt09CzU9S12iNZqn1qjPLa4cO5V88V9XmJ80SKzWFagoilciV5ifVNiRTlBREKpErzU/V7exWUpFUaZoLkRilY+6m6h6jbBMWhNpOVVfQk5pN01yI5IB0rHFd3dpGOpqwVNOoO5QURGKUjjWuq3utRXWTiq7VqFuUFERiVp0rsqH6tY3qJpVcqGmoppI5SgoiOa66tY3qJpVs1zRUU8kwd69RtyOOOMJFpGqmT3fv2NHdLPycPj3153bs6B5OxzveOnasGc9Ph+p8ful4fjoAhZ7COTbrJ/mq3pQURDJr+nT3xo13PCE3bpz6ic2s/JO6WWaeX/oedvWkXN33X93np4uSgoikTU2uaVT3pJzt+EvfQ3VrGjmRFIDBwBJgGTC2nMd3B/4cPT4P6LSzYyopiNQs2f6mXd2TcrZrOumqaaSaFGLraDaz+sAU4AQgDxhhZnllil0EfOruBwC3AzfFFY+IZEd1O8qzfVV5dUdv5cLor6qIc/RRL2CZuy9396+BGcDQMmWGAn+Kth8DjjUzizEmEcmC6g7Lrc7zq3tSru7orWyP/qqqOJNCW+CjpPtF0b5yy7h7CbARaFn2QGY22swKzaxw3bp1MYUrIrVRdU/K2a7pVDepVVWNuE7B3ae5e76757du3Trb4YhIDZKOq8qzWdNJx1QpVRHnegqrgPZJ99tF+8orU2RmuwHNgOIYYxKROqg6a2pkW2nc48aFJqMOHUJCiOv9xJkU5gMHmllnwsl/OHB2mTIzgfOBfwPDgBeiXnIREYlkMqnFlhTcvcTMLgfmAPWB+9x9kZmNJwyNmgncCzxkZsuA/xISh4iIZEmsy3G6+yxgVpl91ydtfwmcEWcMIiKSuhrR0SwiIpmhpCAiIglKCiIiklDj1mg2s3VAOSvW5oRWwPpsB1EJxVc9uR4f5H6Miq96qhNfR3ff6YVeNS4p5DIzK/QUFsbOFsVXPbkeH+R+jIqvejIRn5qPREQkQUlBREQSlBTSa1q2A9gJxVc9uR4f5H6Miq96Yo9PfQoiIpKgmoKIiCQoKYiISIKSQhWZWXszm2tm75jZIjP7cTll+pvZRjN7I7pdX96xYoxxhZm9Fb12YTmPm5lNMrNlZrbQzHpmMLaDkz6XN8xsk5n9pEyZjH9+Znafma01s7eT9u1tZn8zs6XRzxYVPPf8qMxSMzs/Q7HdYmbvRr+/J8yseQXPrfRvIeYYbzCzVUm/xxMreO5gM1sS/T2OzWB8f06KbYWZvVHBc2P9DCs6p2Tt7y+VhZx1++YG7AP0jLabAu8BeWXK9AeezmKMK4BWlTx+IjAbMOBIYF6W4qwPrCFcVJPVzw84BugJvJ2072ZgbLQ9FripnOftDSyPfraItltkILbjgN2i7ZvKiy2Vv4WYY7wB+FkKfwPvA/sBDYE3y/4/xRVfmcd/B1yfjc+wonNKtv7+VFOoIndf7e6vR9ufAYv59jKjuW4o8KAHrwLNzWyfLMRxLPC+u2f9CnV3f5EwfXuy5DXE/wScUs5Tjwf+5u7/dfdPgb8Bg+OOzd2f87CELcCrhEWssqaCzy8VqazlXm2VxRetC38m8Ei6XzcVlZxTsvL3p6RQDWbWCTgcmFfOw0eZ2ZtmNtvMumY0MHDgOTNbYGajy3k8lfWzM2E4Ff8jZvPzK/Vdd18dba8BvltOmVz4LC8k1PzKs7O/hbhdHjVx3VdB80cufH59gU/cfWkFj2fsMyxzTsnK35+Swi4ysybA48BP3H1TmYdfJzSJ9AAmA3/NcHhHu3tP4ATgR2Z2TIZff6fMrCEwBPhLOQ9n+/P7Fg919Zwbv21m44ASoKCCItn8W7gL2B84DFhNaKLJRSOovJaQkc+wsnNKJv/+lBR2gZk1IPzyCtz9/8o+7u6b3H1ztD0LaGBmrTIVn7uvin6uBZ4gVNGTpbJ+dtxOAF5390/KPpDtzy/JJ6XNatHPteWUydpnaWajgB8AI6OTxrek8LcQG3f/xN23uft24O4KXjurf4sW1oY/DfhzRWUy8RlWcE7Jyt+fkkIVRe2P9wKL3f22Csq0icphZr0In3NxhuLb08yalm4TOiTfLlNsJnBeNArpSGBjUjU1Uyr8dpbNz6+M0jXEiX4+WU6ZOcBxZtYiah45LtoXKzMbDFwDDHH3LRWUSeVvIc4Yk/upTq3gtRNruUe1x+GEzz1TBgHvuntReQ9m4jOs5JySnb+/uHrUa+sNOJpQjVsIvBHdTgQuAy6LylwOLCKMpHgV+F4G49svet03oxjGRfuT4zNgCmHUx1tAfoY/wz0JJ/lmSfuy+vkREtRqYCuhXfYioCXwPLAU+Duwd1Q2H7gn6bkXAsui2wUZim0ZoS259G/wj1HZfYFZlf0tZPDzeyj6+1pIOMHtUzbG6P6JhBE378cVY3nxRfsfKP27Syqb0c+wknNKVv7+NM2FiIgkqPlIREQSlBRERCRBSUFERBKUFEREJEFJQUREEpQURCJmts12nME1bTN2mlmn5Bk6RXLVbtkOQCSHfOHuh2U7CJFsUk1BZCei+fRvjubUf83MDoj2dzKzF6IJ3543sw7R/u9aWOPgzej2vehQ9c3s7mjO/OfMbI+o/JXRXPoLzWxGlt6mCKCkIJJsjzLNR2clPbbR3bsBvwfuiPZNBv7k7t0JE9JNivZPAv7pYUK/noQrYQEOBKa4e1dgA3B6tH8scHh0nMvienMiqdAVzSIRM9vs7k3K2b8CGOjuy6OJy9a4e0szW0+YumFrtH+1u7cys3VAO3f/KukYnQjz3h8Y3f8F0MDdbzSzZ4HNhNlg/+rRZIAi2aCagkhqvILtqvgqaXsb3/TpnUSYi6onMD+auVMkK5QURFJzVtLPf0fb/yLM6gkwEngp2n4eGANgZvXNrFlFBzWzekB7d58L/AJoBnyrtiKSKfpGIvKNPWzHxdufdffSYaktzGwh4dv+iGjfFcD9ZvZzYB1wQbT/x8A0M7uIUCMYQ5ihszz1gelR4jBgkrtvSNs7Eqki9SmI7ETUp5Dv7uuzHYtI3NR8JCIiCaopiIhIgmoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikvD/ARgqtbJRaNhKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "loss = history_dict['loss']\n",
    "val_loss = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label=\"Training loss\")\n",
    "plt.plot(epochs, val_loss, 'b', label=\"Validation loss\")\n",
    "plt.title(\"Training and validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXh32V3Q1k0VpZVLYI7qioRatyq1ZF+nOhFvUWt1tvS4WqdelirbVar1dqtVpR6q114V53pKKlKkFJVFBBDBpAjBARDAKBz++P7xkyCUlmyGyZ5P18PM5j5uyfOZmcz3zP93u+x9wdERGR+rTIdQAiItL4KVmIiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFpJVZtbSzDaaWd90LptLZvYNM0t7G3QzO97MSuLG3zezo5JZtgH7utfMrmno+tL0tcp1ANK4mdnGuNEOwGZgWzR+sbvP3JXtufs2oFO6l20O3P2AdGzHzC4Cvufux8Rt+6J0bFuaLiULqZe77zhZR79cL3L3F+ta3sxauXtlNmITkezRZShJiZndZGZ/NbNHzGwD8D0zO8zMXjOzL8xstZndYWato+VbmZmbWf9o/KFo/jNmtsHM/mVmA3Z12Wj+SWb2gZmtN7M7zeyfZnZBHXEnE+PFZrbMzMrN7I64dVua2e/MbK2ZLQfG1XN8ppnZrBrT7jKz26L3F5nZkujzfBj96q9rW6Vmdkz0voOZ/SWK7V1gZI1lp5vZ8mi775rZadH0g4A/AEdFl/g+jzu218etf0n02dea2RNmtlcyx2ZXjnMsHjN70czWmdmnZvbjuP38LDomX5pZoZntXdd+JAvcXYOGpAagBDi+xrSbgC3AqYQfH+2BQ4DRhJLrvsAHwJRo+VaAA/2j8YeAz4ECoDXwV+ChBiy7O7ABGB/N+w9gK3BBHZ8lmRifBLoA/YF1sc8OTAHeBfoAPYB54V+p1v3sC2wEOsZt+zOgIBo/NVrGgOOATcDB0bzjgZK4bZUCx0TvbwX+AXQD+gGLayx7FrBX9Dc5N4phj2jeRcA/asT5EHB99P7EKMZhQDvgv4CXkjk2u3icuwBrgCuAtsBuwKho3k+BImD/6DMMA7rn+n+gOQ8qWUg6vOrus919u7tvcvcF7v66u1e6+3JgBjCmnvX/5u6F7r4VmEk4MezqsqcAi9z9yWje7wiJpVZJxvhLd1/v7iWEE3NsX2cBv3P3UndfC/yqnv0sB94hJDGAE4Bydy+M5s929+UevATMAWqtxK7hLOAmdy939xWE0kL8fh9199XR3+RhQqIvSGK7ABOBe919kbt/DUwFxphZn7hl6jo21SQ4zqcBH7v77919s7t/6e5vRPMuAq5x96XRZ1jk7uuSjF8yQMlC0uGT+BEzG2hm/xddVvgSuAHoWc/6n8a9r6D+Su26lt07Pg53d8Iv8VolGWNS+wJW1BMvwMPAhOj9udF4LI5TzOz16DLMF4Rf9fUdq5i96ovBzC4ws6Lo8s8XwMAktwvh8+3Ynrt/CZQDveOWSepvluA47wN8WEcM9c2THFCykHSo2Wz0HsKv6W+4+27AtYTLLJm0mnBZCAAzM6qf3GpKJcbVhJNZTKKmvY8Cx5tZb0IJ4+EoxvbA34BfEi4RdQWeTzKOT+uKwcz2Be4GLgV6RNt9L267iZr5riJc2optrzPhctfKJOKqqb7j/AmwXx3r1TdPckDJQjKhM7Ae+MrMBgEXZ2Gf/wuMMLNTzawV4Tp4rwzF+ChwpZn1NrMewE/qW9jdPwVeBf4MvO/uS6NZbYE2QBmwzcxOAcbuQgzXmFlXC/ehTImb14mQEMoIefMHhJJFzBqgT3xFcw2PAN83s4PNrC0hmb3i7nWW1OpR33F+CuhrZlPMrK2Z7WZmo6J59wI3mdl+Fgwzs+4N2L+kiZKFZMKPgPMJFc73ECqiM8rd1wBnA7cBawm/St8i3BeS7hjvJtQtvA0sIJQOEnmYUGG94xKUu38BXAU8TqgkPpOQ9JJxHaGEUwI8AzwYt91i4E7gjWiZA4DX49Z9AVgKrDGz+MtJsfWfJVwuejxavy+hHqMh6jzO7r6eUIdzBiGBfUBVfcZvgCcIx/lLQl1HuwbGIGlg4dKuSNNiZi0Jl1POdPdXch2PSL5TyUKaDDMbF12WaQv8jNB09o0Eq4lIEpQspCk5ElhOuFb/LeA77l7XZSgR2QW6DCUiIgmpZCEiIgk1mY4Ee/bs6f379891GCIieWXhwoWfu3t9zcyBJpQs+vfvT2FhYa7DEBHJK2aWqAcCQJehREQkCUoWIiKSkJKFiIgkpGQhIiIJKVmIiEhCGUsWZnafmX1mZu/UMd+iRywuM7NiMxsRN+98M1saDednKkYRkXw2cyb07w8tWoTXmTMzt69Mliz+TD3PJgZOIjwycX9gMqEnT6JuiK8jPIpxFHCdmXXLYJwi0kylerLN5sm6tn1PngwrVoB7eJ08OXMxZCxZuPs8QrfLdRkPPBg9TvI1oGv0UPhvAS+4+zp3Lyd0p1xf0hER2WWpnmzTcbJOJdlMmwYVFdWnVVSE6ZmQyzqL3lR/LGRpNK2u6Tsxs8lmVmhmhWVlZRkLVEQyI5e/7FM92aa6fqrJ5uOPd216qvK6gtvdZ7h7gbsX9OqV8G51EWlEcv3LPtWTbarrp5ps+tbxMN+6pqcql8liJdWfIdwnmlbXdBFpZPL5l32qJ9tU10812dx8M3ToUH1ahw5heibkMlk8BZwXtYo6FFjv7quB54ATzaxbVLF9YjRNRBqRfP9ln+rJNtX1U002EyfCjBnQrx+YhdcZM8L0jHD3jAyEh76vJjytrBT4PnAJcEk034C7gA8JzzIuiFt3ErAsGi5MZn8jR450Edk1Dz3k3q+fu1l4feih5Nft1889pInqQ79++bG+e2qfP9X1H3rIvUOH6rF36LDrMaQKKPRkzunJLJQPg5KFyK5J9WRlVvvJ2iw7+28sJ9tUpJqs0kHJQiQP5PKXbXP/ZS9BssmiyTxWtaCgwPU8C8knsWv+8ZW0HTokf9051fVbtAin95rMYPv2zO9fGgczW+juBYmWy+umsyL5LN9bA2W9glVySiULkRxJ9Ze9SgaSDipZiDRyuW7nr5KB7AolC5EUpHJTWq7b+UNIDCUloSRSUqJEIXVTshBpoFRvSkv1l71KBpJNqrMQaaD+/UOCqKlfv/ArXSQfqM5CJMOy3eunSC4pWUizlkqdQ7Z7/RTJJSULabZSrXPIdq+fIrmkZCHNVqo3tamCWZoTJQvJa6lcRkpHnYOankpzoWQheSvVy0iqcxBJnpKF5K1ULyOpzkEkeUoWkrdSvYykOgeR5LXKdQAiDdW3b+03xe3KZaSJE5UcRJKhkoXkVC77VhKR5ClZSM7kum8lEUme+oaSnFHfSiK5p76hpNFT30oi+UPJQnJG9zmI5A8lC8kZVVCL5A8lC8kZVVCL5A/dZyE5pfscRPKDShYiIpKQkoWIiCSkZCEpSeUObBHJH6qzkAaL3YEd6/k1dgc2qB5CpKlRyUIaLNUuwkUkfyhZSIPpDmyR5kPJQhpMd2CLNB9KFtJgugNbpPlQsmjmUmnNpDuwRZoPtYZqxtLRmkl3YIs0DypZNGNqzSQiyVKyaMbUmklEkqVk0YypNZOIJEvJohlTayYRSVZGk4WZjTOz981smZlNrWV+PzObY2bFZvYPM+sTN2+bmS2KhqcyGWdzpdZMIpIsc/fMbNisJfABcAJQCiwAJrj74rhl/gf4X3d/wMyOAy509/8Xzdvo7p2S3V9BQYEXFham9TOIiDR1ZrbQ3QsSLZfJksUoYJm7L3f3LcAsYHyNZQYDL0Xv59YyX0REGoFMJovewCdx46XRtHhFwOnR++8Anc2sRzTezswKzew1M/u32nZgZpOjZQrLysrSGbuIiMTJdQX31cAYM3sLGAOsBLZF8/pFRaNzgdvNbL+aK7v7DHcvcPeCXr16ZS1oEZHmJpN3cK8E9okb7xNN28HdVxGVLMysE3CGu38RzVsZvS43s38Aw4EPMxiviIjUIZMliwXA/mY2wMzaAOcA1Vo1mVlPM4vF8FPgvmh6NzNrG1sGOAJYjOxET6oTkWzIWMnC3SvNbArwHNASuM/d3zWzG4BCd38KOAb4pZk5MA/4YbT6IOAeM9tOSGi/im9FJYGeVCci2ZKxprPZ1hybzvbvHxJETf36QUlJtqMRkXzUGJrOSoapbycRyRYlizymvp1EJFuULPKY+nYSkWxRsshj6ttJRLJFT8rLc3pSnYhkg0oWIiKSkJKFiIgkpGQhIiIJKVmIiEhCShYiIpKQkoWIiCSkZJFj6jVWRPKB7rPIIfUaKyL5QiWLHJo2rSpRxFRUhOkiIo2JkkUOqddYEckXShY5pF5jRSRfKFnkkHqNFZF8oWSRQ+o1VkTyhVpD5Zh6jRWRfKCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJJUwWZnaZmXXLRjAiItI4JVOy2ANYYGaPmtk4M7NMByUiIo1LwmTh7tOB/YE/ARcAS83sF2a2X4ZjExGRRiKpOgt3d+DTaKgEugF/M7NbMhibiIg0Egm7+zCzK4DzgM+Be4H/dPetZtYCWAr8OLMhiohIriXTN1R34HR3XxE/0d23m9kpmQlLREQak2QuQz0DrIuNmNluZjYawN2XZCowERFpPJJJFncDG+PGN0bTRESkmUgmWVhUwQ2Ey0+oa3MRkWYlmWSx3MwuN7PW0XAFsDzTgYmISOORTLK4BDgcWAmUAqOByZkMSkREGpdkbsr7zN3Pcffd3X0Pdz/X3T/LRnD5YOZM6N8fWrQIrzNn5joiEZH0S+Y+i3bA94EhQLvYdHeflMG48sLMmTB5MlRUhPEVK8I46FGpItK0JHMZ6i/AnsC3gJeBPsCGTAaVL6ZNq0oUMRUVYbqISFOSTLL4hrv/DPjK3R8Avk2ot0go6njwfTNbZmZTa5nfz8zmmFmxmf3DzPrEzTvfzJZGw/nJfqBs+vjjXZsuIpKvkkkWW6PXL8zsQKALsHuilcysJXAXcBIwGJhgZoNrLHYr8KC7HwzcAPwyWrc7cB0hKY0CrmuM3aT37btr00VE8lUyyWJGdKKeDjwFLAZ+ncR6o4Bl7r7c3bcAs4DxNZYZDLwUvZ8bN/9bwAvuvs7dy4EXgHFJ7DOrbr4ZOnSoPq1DhzBdRKQpqTdZRJ0Ffunu5e4+z933jVpF3ZPEtnsDn8SNl0bT4hUBp0fvvwN0NrMeSa6bcxMnwowZ0K8fmIXXGTNUuS0iTU+9ySK6WzuTvcpeDYwxs7eAMYR7ObYlu7KZTTazQjMrLCsry1SM9Zo4EUpKYPv28KpEISJNUTKXoV40s6vNbB8z6x4bklhvJbBP3HifaNoO7r7K3U939+HAtGjaF8msGy07w90L3L2gV69eSYQkIiINkUwfT2dHrz+Mm+bAvgnWWwDsb2YDCCf6c4Bz4xcws57AuqgE81PgvmjWc8Av4iq1T4zmi4hIDiRMFu4+oCEbdvdKM5tCOPG3BO5z93fN7Aag0N2fAo4BfmlmDswjSkjuvs7MbiQkHIAb3H3dTjsREZGssLgOZWtfwOy82qa7+4MZiaiBCgoKvLCwMNdhiIjkFTNb6O4FiZZL5jLUIXHv2wFjgTeBRpUsREQkc5K5DHVZ/LiZdSXcMyEiIs1EMq2havoKaFA9hoiI5Kdkep2dTWj9BCG5DAYezWRQTd3XX0NxMSxYAIWFUFQE++wDRxwBhx8OBQXQrl3i7YiIZEsydRa3xr2vBFa4e2mG4mlytm6Fd98NSSGWHIqLobIyzO/VC4YOhSVL4KmnwrQ2bWDkyJA4Yglkjz1y9xlERJJpDTUAWO3uX0fj7YE93L0k8+ElrzG0htq2Dd5/PySEWHJYtCiUJAC6dg2lhoICOOSQ8LrPPqGrEICyMpg/H/75z/C6YAFs2RLm7bdfVeI44ggYPDg8cKkhNm2CtWth3brwWl4OPXrAvvvC3ntDy5apHwsRyQ/JtoZKJlkUAodHnQFiZm2Af7r7IfWumGW5ShbbtsEvfwkvvABvvgkbN4bpHTuG0kF8Ythvv6rEkIzNm2HhwqoE8s9/hoQC0KULHHZYSByHHQatW1ed/BO9xpJXbdq0CX1c7btv7cNuuzX8WIlI45POZLHI3YfVmFbk7kNTjDGtcpUsrr0WbrwxJITRo6sSwwEHpP8Xujt8+GFV4pg/P1ziqk3r1qG00KMHdO9e/2vXrvD557B8+c5DeXn17cZKIAMGVE8iw4eH7YlIfknnfRZlZnZadMc1ZjYe+DzVAJuCp58OiWLSJPjTnzK/PzP4xjfCcH70OKjy8nDJq0WL6kmgY8ddK8XUpbwcPvqoegL56KNQivr736vqXszg4INhzJgwHH009OyZ+v5FpHFIpmSxHzAT2DuaVAqc5+7LMhzbLsl2yaKkBEaMCJds5s+H9u2ztutGo7ISVq6EpUvhX/+Cl18Ox2LTpjD/wAOrkseYMbB7wkdmiUi2pe0yVNwGOwG4+8YUY8uIbCaLr7+GI4+EZctCncJ++2Vlt3lhy5ZQMf/yy2F49dWq55QPGlQ9eey1V25jFZH01ln8Argl6jqcqCfYH7n79LREmibZTBYXXxwecvTkk3DaaVnZZd7aujUk1FjyeOWVqkYA3/xmVeKINQBolcyFURFJm3Qmi7ei503ET3vT3UekGGNaZStZPPAAXHABTJ0aWkHJrqmshLfeqkoe8+bBl1+GeW3bwsCBMGRI9WHAADXnFcmUdCaLYuAQd98cjbcndDE+JC2Rpkk2kkVxcWjxdNhh8Pzz+hWcDtu2heNaVBRadsWGjz+uWqZ9+3AJq2YS6dev4feaiEiQztZQM4E5ZnY/YMAFwAOphZd/1q+HM86Abt3gkUeUKNKlZcvQ7Hb48OrTN2yAxYvhnXeqEshLL8Ff/lK1TMeO4ebEIUPg3HPhhBOyG7tIc5JMr7O/NrMi4HhCH1HPAf0yHVhj4h4uPZWUwD/+oa43sqFz51CKGz26+vQvvtg5icyeDX/+c6g/+u1vQ9NiEUmvZH8fryEkiu8CHwGPZSyiRujWW+GJJ+B3vwt3TEvudO0aujw5/PCqaZs3w+23w003hZLGVVfBtGm621wkneq84mtm3zSz68zsPeBO4GNCHcex7v6HrEWYYy+/DD/9KXz3u3DFFbmORmrTti385CfwwQcwcSLccktoaXX//bB9e66jE2ka6qsefA84DjjF3Y909zuBbdkJq3FYvRrOPjtc1rj33vTcES2Zs9deIUG88UZoQTVpUriMNX9+riOr3/btoYnxtm3hfZK3PolkVX2XoU4HzgHmmtmzhKfjNZvT5datIVFs2ABz5uiSRj455JCQIB5+GH7843Dp8Nxz4de/hj59ch1dSAbvvRe+Vy++CHPnVjUfjjELQ4sWdb/Gv2/TJrQaiw0dOtQ/XnPagAGhR4IOHXJzTKTxS6bpbEdgPDCBUNJ4EHjc3Z/PfHjJS3fT2f/8z1BX8dBD4dKG5KeNG0OS+M1vQsurqVPh6quz3z3LqlVVyeHFF8M4hE4Yx46Fvn1DEomVLOJfa5sWP2/79nDn/KZNVUNFRf3jtWnZMjxbZfRoOPTQMOy/v0rUTV3au/uINtqNUMl9truPTSG+tEtnsvj730Mz2X//d7jrrrRsUnLso4/CD4DHHgv3Z9x6a/gbZ+pE+OWXob4rlhwWLw7Te/QIyeH448PrvvtmZv/1cQ+NAmIJpKIilHReew1efz1cxtuwISzbrVtVq7RDD4VRo9S7cFOTkWTRmKUrWXzwQeh6YtCgcHdx27ZpCE4ajblz4corw42AY8bA738ffk2nasuWcLKNJYc33gh1EO3bhx54Y8lh6NDGfyPhtm3hyY2vv16VQN55p6ou5ZvfrEoeo0eH3oZbtw7zvv463JO0fn1o5hz/Wt+0jh1rf37KXns1/uOV75QsGqCiIvwDrFoVuuDu2zdNwUmjUlkZGixMnx66YP/BD2DcuHCiiw2bNyf/vqIi3IFeURFObIccEpLD8ceHu/2bwg+ODRtCV/ix5PHaa7BmTZjXrl2o01u/PhyT+rRoEZbt2jU8wCs2bNgQur//5JPqFfxt2+787JTYMGAAdOqUuc/cXChZ7CL38IyIhx6CZ5+FE09MY3DSKJWXw89/Dn/4Q/g1XZfWrcNJq127qiF+PNan1QknwDHHhBNhU+ceumSJJY+KiqoTfywR1Hzt0iWc3OsrKWzZAitW7PwMleXLw4O/ajYE2H33kDgOPhiOOw6OPVZd4e8qJYtddM89cMkl4eRx7bVpDEwavdLS8Lja2pJB27bqxLCxcA8JvrYkUlhYlUgOOigkjrFjwyXALl0yG9Onn4a4WrcOQ6tWO79v1SoMjbGxgJLFLigsDM0rjzsO/u//dI1UJN9UVoau8F96KQyvvhouEbZoEeogx44N/9+HH96w5sHbt4eS1JIlobFC7HXx4nD5LVmxpFEzmfTqFepJBw8Ow6BB4f6uWF1QJilZJGntWhg5MvxCePPN0FpFRPLb11+HS2Sx5PH66yGhtGkTEsZxx4Vh1KjqJ+TKylBaiU8IS5aEIfYQLwiXuuJP7j17hnW3bg1D7H2y01avDvsoKanaR6tWoTFBzSRywAGh1JsuShZJKisLdRU//3momBSRpmfDhlDaeOmlcL/LokXhB2LHjuFSVefOITF88EGoN4np06fqJB1/ws7Uj8qvvgrNmONLLkuWhKdyxrquadEi1NPEYho0KNTZ1Oy5OVlKFiIidVi7NtwHEyt5bN5cPRkMHhwaLTSWnhs2bw6JrGYSef/9UDI55JDQXLshlCxERJq4yspQwf/VV6G7loZI58OPRESkEWrVKtRhZIPa/YiISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJZTRZmNk4M3vfzJaZ2dRa5vc1s7lm9paZFZvZydH0/ma2ycwWRcN/ZzJOERGpX8bu4DazlsBdwAlAKbDAzJ5y98Vxi00HHnX3u81sMPA00D+a96G7D8tUfCIikrxMlixGAcvcfbm7bwFmAeNrLONArKuuLsCqDMYjIiINlMlk0Rv4JG68NJoW73rge2ZWSihVXBY3b0B0eeplMzuqth2Y2WQzKzSzwrKysjSGLiIi8XJdwT0B+LO79wFOBv5iZi2A1UBfdx8O/AfwsJnt1Fmwu89w9wJ3L+jVq1dWAxcRaU4ymSxWAvvEjfeJpsX7PvAogLv/C2gH9HT3ze6+Npq+EPgQ+GYGYxURkXpkMlksAPY3swFm1gY4B3iqxjIfA2MBzGwQIVmUmVmvqIIcM9sX2B9YnsFYRUSkHhlrDeXulWY2BXgOaAnc5+7vmtkNQKG7PwX8CPijmV1FqOy+wN3dzI4GbjCzrcB24BJ3X5epWEVEpH56Up6ISDOW7JPycl3BLSIieUDJQkREElKyEBGRhJQsREQkISULERFJSMlCREQSUrIQEZGElCxERCQhJQsREUlIyUJERBJSshARkYSULEREJCElCxERSUjJQkREElKyEBGRhJQsREQkISULERFJSMlCREQSUrIQEZGEWuU6ABHJf1u3bqW0tJSvv/4616FIHdq1a0efPn1o3bp1g9ZXshCRlJWWltK5c2f69++PmeU6HKnB3Vm7di2lpaUMGDCgQdvQZSgRSdnXX39Njx49lCgaKTOjR48eKZX8lCxEJC2UKBq3VP8+ShYiIpKQkoWIZN3MmdC/P7RoEV5nzkxte2vXrmXYsGEMGzaMPffck969e+8Y37JlS1LbuPDCC3n//ffrXeauu+5iZqrB5ilVcItIVs2cCZMnQ0VFGF+xIowDTJzYsG326NGDRYsWAXD99dfTqVMnrr766mrLuDvuTosWtf9Gvv/++xPu54c//GHDAmwCVLIQkayaNq0qUcRUVITp6bZs2TIGDx7MxIkTGTJkCKtXr2by5MkUFBQwZMgQbrjhhh3LHnnkkSxatIjKykq6du3K1KlTGTp0KIcddhifffYZANOnT+f222/fsfzUqVMZNWoUBxxwAPPnzwfgq6++4owzzmDw4MGceeaZFBQU7Ehk8a677joOOeQQDjzwQC655BLcHYAPPviA4447jqFDhzJixAhKSkoA+MUvfsFBBx3E0KFDmZaJg5WAkoWIZNXHH+/a9FS99957XHXVVSxevJjevXvzq1/9isLCQoqKinjhhRdYvHjxTuusX7+eMWPGUFRUxGGHHcZ9991X67bdnTfeeIPf/OY3OxLPnXfeyZ577snixYv52c9+xltvvVXruldccQULFizg7bffZv369Tz77LMATJgwgauuuoqioiLmz5/P7rvvzuzZs3nmmWd44403KCoq4kc/+lGajk7ylCxEJKv69t216anab7/9KCgo2DH+yCOPMGLECEaMGMGSJUtqTRbt27fnpJNOAmDkyJE7ft3XdPrpp++0zKuvvso555wDwNChQxkyZEit686ZM4dRo0YxdOhQXn75Zd59913Ky8v5/PPPOfXUU4FwI12HDh148cUXmTRpEu3btwege/fuu34gUqRkISJZdfPN0KFD9WkdOoTpmdCxY8cd75cuXcrvf/97XnrpJYqLixk3blyt9x60adNmx/uWLVtSWVlZ67bbtm2bcJnaVFRUMGXKFB5//HGKi4uZNGlSo7/7XclCRLJq4kSYMQP69QOz8DpjRsMrt3fFl19+SefOndltt91YvXo1zz33XNr3ccQRR/Doo48C8Pbbb9dactm0aRMtWrSgZ8+ebNiwgcceewyAbt260atXL2bPng2Emx0rKio44YQTuO+++9i0aRMA69atS3vciag1lIhk3cSJ2UkONY0YMYLBgwczcOBA+vXrxxFHHJH2fVx22WWcd955DB48eMfQpUuXasv06NGD888/n8GDB7PXXnsxevToHfNmzpzJxRdfzLRp02jTpg2PPfYYp5xyCkVFRRQUFNC6dWtOPfVUbrzxxrTHXh+L1cDnu4KCAi8sLMx1GCLN0pIlSxg0aFCuw2gUKisrqayspF27dixdupTmNhsCAAANUUlEQVQTTzyRpUuX0qpV7n+b1/Z3MrOF7l5Qxyo75D56EZEmZOPGjYwdO5bKykrcnXvuuadRJIpU5f8nEBFpRLp27crChQtzHUbaqYJbREQSUrIQEZGElCxERCQhJQsREUkoo8nCzMaZ2ftmtszMptYyv6+ZzTWzt8ys2MxOjpv302i9983sW5mMU0Ty27HHHrvTDXa33347l156ab3rderUCYBVq1Zx5pln1rrMMcccQ6Jm+bfffjsVcb0jnnzyyXzxxRfJhJ43MpYszKwlcBdwEjAYmGBmg2ssNh141N2HA+cA/xWtOzgaHwKMA/4r2p6IyE4mTJjArFmzqk2bNWsWEyZMSGr9vffem7/97W8N3n/NZPH000/TtWvXBm+vMcpk09lRwDJ3Xw5gZrOA8UD8ve8O7Ba97wKsit6PB2a5+2bgIzNbFm3vXxmMV0TS4MoroZYeuVMybBhEPYPX6swzz2T69Ols2bKFNm3aUFJSwqpVqzjqqKPYuHEj48ePp7y8nK1bt3LTTTcxfvz4auuXlJRwyimn8M4777Bp0yYuvPBCioqKGDhw4I4uNgAuvfRSFixYwKZNmzjzzDP5+c9/zh133MGqVas49thj6dmzJ3PnzqV///4UFhbSs2dPbrvtth291l500UVceeWVlJSUcNJJJ3HkkUcyf/58evfuzZNPPrmjo8CY2bNnc9NNN7FlyxZ69OjBzJkz2WOPPdi4cSOXXXYZhYWFmBnXXXcdZ5xxBs8++yzXXHMN27Zto2fPnsyZMydtf4NMJovewCdx46XA6BrLXA88b2aXAR2B4+PWfa3Gur1r7sDMJgOTAfpmqstKEWn0unfvzqhRo3jmmWcYP348s2bN4qyzzsLMaNeuHY8//ji77bYbn3/+OYceeiinnXZanc+kvvvuu+nQoQNLliyhuLiYESNG7Jh388030717d7Zt28bYsWMpLi7m8ssv57bbbmPu3Ln07Nmz2rYWLlzI/fffz+uvv467M3r0aMaMGUO3bt1YunQpjzzyCH/84x8566yzeOyxx/je975Xbf0jjzyS1157DTPj3nvv5ZZbbuG3v/0tN954I126dOHtt98GoLy8nLKyMn7wgx8wb948BgwYkPb+o3J9U94E4M/u/lszOwz4i5kdmOzK7j4DmAGhu48MxSgiu6C+EkAmxS5FxZLFn/70JyA8c+Kaa65h3rx5tGjRgpUrV7JmzRr23HPPWrczb948Lr/8cgAOPvhgDj744B3zHn30UWbMmEFlZSWrV69m8eLF1ebX9Oqrr/Kd73xnR8+3p59+Oq+88gqnnXYaAwYMYNiwYUDd3aCXlpZy9tlns3r1arZs2cKAAQMAePHFF6tdduvWrRuzZ8/m6KOP3rFMursxz2QF90pgn7jxPtG0eN8HHgVw938B7YCeSa6bFul+FrCI5Mb48eOZM2cOb775JhUVFYwcORIIHfOVlZWxcOFCFi1axB577NGg7sA/+ugjbr31VubMmUNxcTHf/va3U+pWPNa9OdTdxflll13GlClTePvtt7nnnnty2o15JpPFAmB/MxtgZm0IFdZP1VjmY2AsgJkNIiSLsmi5c8ysrZkNAPYH3kh3gLFnAa9YAe5VzwJWwhDJP506deLYY49l0qRJ1Sq2169fz+67707r1q2ZO3cuK1asqHc7Rx99NA8//DAA77zzDsXFxUDo3rxjx4506dKFNWvW8Mwzz+xYp3PnzmzYsGGnbR111FE88cQTVFRU8NVXX/H4449z1FFHJf2Z1q9fT+/e4Qr8Aw88sGP6CSecwF133bVjvLy8nEMPPZR58+bx0UcfAenvxjxjycLdK4EpwHPAEkKrp3fN7AYzOy1a7EfAD8ysCHgEuMCDdwkljsXAs8AP3X1bumPM5rOARSTzJkyYQFFRUbVkMXHiRAoLCznooIN48MEHGThwYL3buPTSS9m4cSODBg3i2muv3VFCGTp0KMOHD2fgwIGce+651bo3nzx5MuPGjePYY4+ttq0RI0ZwwQUXMGrUKEaPHs1FF13E8OHDk/48119/Pd/97ncZOXJktfqQ6dOnU15ezoEHHsjQoUOZO3cuvXr1YsaMGZx++ukMHTqUs88+O+n9JKNZd1HeokUoUdRkBtu3pykwkWZAXZTnh1S6KG/Wd3Bn+1nAIiL5qlkni2w/C1hEJF8162SRy2cBizQ1TeWSdlOV6t8n1/dZ5FyungUs0pS0a9eOtWvX0qNHjzpvdpPccXfWrl1Lu3btGryNZp8sRCR1ffr0obS0lLKyslyHInVo164dffr0afD6ShYikrLWrVvvuHNYmqZmXWchIiLJUbIQEZGElCxERCShJnMHt5mVAfV3+pJbPYHPcx1EPRRfahRfahRfalKJr5+790q0UJNJFo2dmRUmc0t9rii+1Ci+1Ci+1GQjPl2GEhGRhJQsREQkISWL7JmR6wASUHypUXypUXypyXh8qrMQEZGEVLIQEZGElCxERCQhJYs0MbN9zGyumS02s3fN7IpaljnGzNab2aJouDYHcZaY2dvR/nd6tKAFd5jZMjMrNrMRWYztgLhjs8jMvjSzK2ssk9VjaGb3mdlnZvZO3LTuZvaCmS2NXrvVse750TJLzez8LMb3GzN7L/r7PW5mXetYt97vQgbju97MVsb9DU+uY91xZvZ+9F2cmsX4/hoXW4mZLapj3Wwcv1rPKzn5Drq7hjQMwF7AiOh9Z+ADYHCNZY4B/jfHcZYAPeuZfzLwDGDAocDrOYqzJfAp4YahnB1D4GhgBPBO3LRbgKnR+6nAr2tZrzuwPHrtFr3vlqX4TgRaRe9/XVt8yXwXMhjf9cDVSfz9PwT2BdoARTX/nzIVX435vwWuzeHxq/W8kovvoEoWaeLuq939zej9BmAJ0Du3UTXIeOBBD14DuprZXjmIYyzwobvn9K58d58HrKsxeTzwQPT+AeDfaln1W8AL7r7O3cuBF4Bx2YjP3Z9398po9DWg4f1Sp6iO45eMUcAyd1/u7luAWYTjnlb1xWfhwRxnAY+ke7/Jque8kvXvoJJFBphZf2A48Hotsw8zsyIze8bMhmQ1sMCB581soZlNrmV+b+CTuPFScpP0zqHuf9JcH8M93H119P5TYI9almksx3ESoaRYm0TfhUyaEl0mu6+OSyiN4fgdBaxx96V1zM/q8atxXsn6d1DJIs3MrBPwGHClu39ZY/abhMsqQ4E7gSeyHR9wpLuPAE4CfmhmR+cghnqZWRvgNOB/apndGI7hDh7K+42y/bmZTQMqgZl1LJKr78LdwH7AMGA14VJPYzSB+ksVWTt+9Z1XsvUdVLJIIzNrTfiDznT3v9ec7+5fuvvG6P3TQGsz65nNGN19ZfT6GfA4obgfbyWwT9x4n2haNp0EvOnua2rOaAzHEFgTuzQXvX5WyzI5PY5mdgFwCjAxOpnsJInvQka4+xp33+bu24E/1rHfXB+/VsDpwF/rWiZbx6+O80rWv4NKFmkSXd/8E7DE3W+rY5k9o+Uws1GE4782izF2NLPOsfeEitB3aiz2FHBe1CrqUGB9XHE3W+r8RZfrYxh5Coi1LDkfeLKWZZ4DTjSzbtFllhOjaRlnZuOAHwOnuXtFHcsk813IVHzxdWDfqWO/C4D9zWxAVNI8h3Dcs+V44D13L61tZraOXz3nlex/BzNZk9+cBuBIQlGwGFgUDScDlwCXRMtMAd4ltOx4DTg8yzHuG+27KIpjWjQ9PkYD7iK0RHkbKMhyjB0JJ/8ucdNydgwJSWs1sJVwzff7QA9gDrAUeBHoHi1bANwbt+4kYFk0XJjF+JYRrlXHvof/HS27N/B0fd+FLMX3l+i7VUw46e1VM75o/GRC658PsxlfNP3Pse9c3LK5OH51nVey/h1Udx8iIpKQLkOJiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIJmNk2q94bbtp6QDWz/vE9noo0Vq1yHYBIHtjk7sNyHYRILqlkIdJA0fMMbomeafCGmX0jmt7fzF6KOsqbY2Z9o+l7WHi+RFE0HB5tqqWZ/TF6XsHzZtY+Wv7y6DkGxWY2K0cfUwRQshBJRvsal6HOjpu33t0PAv4A3B5NuxN4wN0PJnTid0c0/Q7gZQ+dII4g3PkLsD9wl7sPAb4AzoimTwWGR9u5JFMfTiQZuoNbJAEz2+junWqZXgIc5+7Lo87ePnX3Hmb2OaELi63R9NXu3tPMyoA+7r45bhv9Cc8c2D8a/wnQ2t1vMrNngY2EnnWf8KgDRZFcUMlCJDVex/tdsTnu/Taq6hK/TeinawSwIOoJVSQnlCxEUnN23Ou/ovfzCb2kAkwEXonezwEuBTCzlmbWpa6NmlkLYB93nwv8BOgC7FS6EckW/VIRSay9mS2KG3/W3WPNZ7uZWTGhdDAhmnYZcL+Z/SdQBlwYTb8CmGFm3yeUIC4l9Hham5bAQ1FCMeAOd/8ibZ9IZBepzkKkgaI6iwJ3/zzXsYhkmi5DiYhIQipZiIhIQipZiIhIQkoWIiKSkJKFiIgkpGQhIiIJKVmIiEhC/x+1ZiBob8SYJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()\n",
    "acc = history_dict['binary_accuracy']\n",
    "val_acc = history_dict['val_binary_accuracy']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label=\"Training acc\")\n",
    "plt.plot(epochs, val_acc, 'b', label=\"Validation acc\")\n",
    "plt.title(\"Training and validation acc\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기에서 보면 기대한 대로 학습 손실은 감소하고 학습 정확도는 증가함을 볼 수 있는데, 검증 세트에 대한 손실과 정확도는 그렇지 않다는 것을 볼 수 있습니다.  \n",
    "이는 우리의 모델이 훈련 데이터에 **과적합** 되었다는 의미이고, 이런 모델은 새로운 데이터에 대해서 좋은 예측을 할 수 없습니다.  \n",
    "그렇기 때문에 모델이 과적합 되기 전인 네 번째 에포크 동안만 훈련을 처음부터 다시 시켜 이번에는 테스트 데이터로 예측 및 평가를 하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "25000/25000 [==============================] - 2s 62us/step - loss: 0.4577 - binary_accuracy: 0.8222\n",
      "Epoch 2/4\n",
      "25000/25000 [==============================] - 1s 52us/step - loss: 0.2631 - binary_accuracy: 0.9073\n",
      "Epoch 3/4\n",
      "25000/25000 [==============================] - 1s 59us/step - loss: 0.2026 - binary_accuracy: 0.9271\n",
      "Epoch 4/4\n",
      "25000/25000 [==============================] - 1s 57us/step - loss: 0.1691 - binary_accuracy: 0.9406\n",
      "25000/25000 [==============================] - 3s 107us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "             loss=losses.binary_crossentropy,\n",
    "             metrics=[metrics.binary_accuracy])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.2971209902191162, 0.88256]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이와 같이 간단한 모델을 사용해 87%의 정확도를 달성했습니다.  \n",
    "<br />\n",
    "또한 `predict`메서드를 이용하면 훈련시킨 모델을 실전 환경에서도 사용할 수 있어 어떤 리뷰가 긍정일 확률을 예측할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.19154617],\n",
       "       [0.9998211 ],\n",
       "       [0.9308494 ],\n",
       "       ...,\n",
       "       [0.18322498],\n",
       "       [0.1177181 ],\n",
       "       [0.719048  ]], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 요약\n",
    "* 원본 데이터를 신경망에 텐서로 주입하기 위해서는 꽤 많은 전처리가 필요합니다.\n",
    "* `relu` 활성화 함수와 함께 Dense 층을 쌓은 네트워크는 여러 종류의 문제에 적용할 수 있습니다.\n",
    "* 이진 분류 분제에서 네트워크는 하나의 유닛과 `sigmoid` 활성화 함수를 가진 Dense 층으로 끝나야 합니다. 이 신경망의 출력은 0과 1사이의 스칼라 값입니다.\n",
    "* 이진 분류 문제에서 이런 스칼라 시그모이드 출력에 대해 사용할 손실 함수는 `binary_crossentropy`입니다.\n",
    "* `rmsprop` 옵티마이저는 문제에 상관없이 일반적으로 충분히 좋은 선택입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## +) TensorFlow를 이용해 구현해보기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/7\n",
      "loss: 2.5925 | accuracy: 0.5234\n",
      "Epoch 2/7\n",
      "loss: 0.7361 | accuracy: 0.5566\n",
      "Epoch 3/7\n",
      "loss: 0.6008 | accuracy: 0.6699\n",
      "Epoch 4/7\n",
      "loss: 0.3553 | accuracy: 0.8496\n",
      "Epoch 5/7\n",
      "loss: 0.2147 | accuracy: 0.9160\n",
      "Epoch 6/7\n",
      "loss: 0.1533 | accuracy: 0.9570\n",
      "Epoch 7/7\n",
      "loss: 0.1240 | accuracy: 0.9570\n",
      "\n",
      "test loss: 0.4149 | test accuracy: 0.8549\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "# label의 데이터 사이즈 변경(축 추가) 및 데이터 셔플\n",
    "y_train_ = y_train[:, np.newaxis]\n",
    "y_test_ = y_test[:, np.newaxis]\n",
    "x_train_, y_train_ = shuffle(x_train, y_train_)\n",
    "x_test_, y_test_ = shuffle(x_test, y_test_)\n",
    "\n",
    "# Placeholder 선언(feed할 데이터와 라벨)\n",
    "inputs = tf.placeholder(tf.float32, [None, 10000])\n",
    "labels = tf.placeholder(tf.float32, [None, 1])\n",
    "\n",
    "# 모델 정의\n",
    "# layer 1 (10000, 16) - activation: relu\n",
    "W1 = tf.Variable(tf.random_normal([10000, 16]))\n",
    "b1 = tf.Variable(tf.random_normal([16]))\n",
    "y1 = tf.nn.relu(tf.matmul(inputs, W1) + b1)\n",
    "\n",
    "# layer 2 (16, 16) - activation: relu\n",
    "W2 = tf.Variable(tf.random_normal([16, 16]))\n",
    "b2 = tf.Variable(tf.random_normal([16]))\n",
    "y2 = tf.nn.relu(tf.matmul(y1, W2) + b2)\n",
    "\n",
    "# layer 3 (16, 1) - activation: sigmoid\n",
    "W3 = tf.Variable(tf.random_normal([16, 1]))\n",
    "b3 = tf.Variable(tf.random_normal([1]))\n",
    "\n",
    "logits = tf.matmul(y2, W3) + b3\n",
    "\n",
    "# 손실 함수는 cross_entropy, 옵티마이저는 rmsprop 사용 (learning rate는 0.04)\n",
    "cross_entropy = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits))\n",
    "train_step = tf.train.RMSPropOptimizer(0.04).minimize(cross_entropy)\n",
    "\n",
    "predicted = tf.cast(tf.nn.sigmoid(logits) > 0.5, dtype=tf.float32)\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, labels), dtype=tf.float32))\n",
    "\n",
    "# 학습 및 평가\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    # 배치 사이즈와 에포크 지정\n",
    "    batch_size = 512\n",
    "    training_epochs = 7\n",
    "\n",
    "    for epoch in range(training_epochs):\n",
    "        batch_count = int(x_train_.shape[0] / batch_size)\n",
    "        for i in range(batch_count):\n",
    "            batch_xs = x_train_[i * batch_size : min((i + 1) * batch_size, x_train_.shape[0])]\n",
    "            batch_ys = y_train_[i * batch_size : min((i + 1) * batch_size, x_train_.shape[0])]\n",
    "\n",
    "            sess.run(train_step, feed_dict={inputs: batch_xs, labels: batch_ys})\n",
    "\n",
    "        loss, acc = sess.run([cross_entropy, accuracy], feed_dict={inputs: batch_xs, labels: batch_ys})\n",
    "        print('Epoch ' + str(epoch + 1) + '/' + str(training_epochs))\n",
    "        print('loss: {:.4f} | accuracy: {:.4f}'.format(loss, acc))\n",
    "\n",
    "    test_loss, test_acc = sess.run([cross_entropy, accuracy], feed_dict={inputs: x_test_, labels: y_test_})\n",
    "    print('\\ntest loss: {:.4f} | test accuracy: {:.4f}'.format(test_loss, test_acc))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (myenv)",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}