{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "TensorFlow 2.3 on Python 3.6 (CUDA 10.1)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "9-2.deep_net_in_keras.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eo2G1vSIzfct"
      },
      "source": [
        "# 케라스로 심층 신경망 만들기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3YlzZAYzfc1"
      },
      "source": [
        "이 노트북에서는 지금까지 다룬 이론을 적용하여 [은닉층을 추가한 신경망](https://github.com/rickiepark/dl-illustrated/blob/master/notebooks/8-4.intermediate_net_in_keras.ipynb)을 향상시켜 보겠습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ipy3vZNuzfc2"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/rickiepark/dl-illustrated/blob/master/notebooks/9-2.deep_net_in_keras.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kfptzy2Azfc2"
      },
      "source": [
        "#### 라이브러리를 적재합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3K6MtDtszfc2"
      },
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras.datasets import mnist\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Dropout # new!\n",
        "from tensorflow.keras.layers import BatchNormalization # new!\n",
        "from tensorflow.keras.optimizers import SGD"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmF-LF_wzfc3"
      },
      "source": [
        "#### 데이터를 적재합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5bYUXOCuzfc3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "04bfbd02-9436-4ae1-9152-35eb51e21364"
      },
      "source": [
        "(X_train, y_train), (X_valid, y_valid) = mnist.load_data()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 2s 0us/step\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kY6JdpLCzfc3"
      },
      "source": [
        "#### 데이터를 전처리합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78EKxfkOzfc4"
      },
      "source": [
        "X_train = X_train.reshape(60000, 784).astype('float32')\n",
        "X_valid = X_valid.reshape(10000, 784).astype('float32')"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7edPSglQzfc4"
      },
      "source": [
        "X_train /= 255\n",
        "X_valid /= 255"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FGC1vEeTzfc4"
      },
      "source": [
        "n_classes = 10\n",
        "y_train = keras.utils.to_categorical(y_train, n_classes)\n",
        "y_valid = keras.utils.to_categorical(y_valid, n_classes)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPSgbdqjzfc4"
      },
      "source": [
        "#### 신경망을 만듭니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YnsPCfMdzfc5"
      },
      "source": [
        "model = Sequential()\n",
        "\n",
        "model.add(Dense(64, activation='relu', input_shape=(784,)))\n",
        "model.add(BatchNormalization())\n",
        "\n",
        "model.add(Dense(64, activation='relu'))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Dropout(0.2))\n",
        "\n",
        "model.add(Dense(64, activation='relu'))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Dropout(0.4))\n",
        "\n",
        "model.add(Dense(10, activation='softmax'))"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D93Id0Omzfc5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8b905d4-4c4e-428a-a1a9-6895be6c3c82"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " dense (Dense)               (None, 64)                50240     \n",
            "                                                                 \n",
            " batch_normalization (BatchN  (None, 64)               256       \n",
            " ormalization)                                                   \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 64)                4160      \n",
            "                                                                 \n",
            " batch_normalization_1 (Batc  (None, 64)               256       \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 64)                0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 64)                4160      \n",
            "                                                                 \n",
            " batch_normalization_2 (Batc  (None, 64)               256       \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 64)                0         \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 10)                650       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 59,978\n",
            "Trainable params: 59,594\n",
            "Non-trainable params: 384\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JHzdMWXizfc6"
      },
      "source": [
        "#### 모델 설정을 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fuPa-i31zfc6"
      },
      "source": [
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iGGleXnizfc6"
      },
      "source": [
        "#### 훈련!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SlgVOfFuzfc7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f80c6c35-a6f3-4e3b-99b6-1fd1675435e4"
      },
      "source": [
        "model.fit(X_train, y_train, batch_size=128, epochs=20, verbose=1, validation_data=(X_valid, y_valid))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "469/469 [==============================] - 6s 6ms/step - loss: 0.5565 - accuracy: 0.8329 - val_loss: 0.1835 - val_accuracy: 0.9446\n",
            "Epoch 2/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.2256 - accuracy: 0.9350 - val_loss: 0.1502 - val_accuracy: 0.9527\n",
            "Epoch 3/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.1700 - accuracy: 0.9508 - val_loss: 0.1369 - val_accuracy: 0.9568\n",
            "Epoch 4/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.1391 - accuracy: 0.9594 - val_loss: 0.1028 - val_accuracy: 0.9684\n",
            "Epoch 5/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.1176 - accuracy: 0.9655 - val_loss: 0.1006 - val_accuracy: 0.9693\n",
            "Epoch 6/20\n",
            "469/469 [==============================] - 3s 6ms/step - loss: 0.1063 - accuracy: 0.9685 - val_loss: 0.0961 - val_accuracy: 0.9714\n",
            "Epoch 7/20\n",
            "469/469 [==============================] - 3s 6ms/step - loss: 0.0939 - accuracy: 0.9725 - val_loss: 0.0944 - val_accuracy: 0.9725\n",
            "Epoch 8/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0837 - accuracy: 0.9749 - val_loss: 0.0924 - val_accuracy: 0.9729\n",
            "Epoch 9/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0804 - accuracy: 0.9766 - val_loss: 0.0885 - val_accuracy: 0.9753\n",
            "Epoch 10/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0724 - accuracy: 0.9782 - val_loss: 0.0894 - val_accuracy: 0.9743\n",
            "Epoch 11/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0703 - accuracy: 0.9791 - val_loss: 0.0892 - val_accuracy: 0.9726\n",
            "Epoch 12/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0650 - accuracy: 0.9804 - val_loss: 0.0831 - val_accuracy: 0.9765\n",
            "Epoch 13/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0592 - accuracy: 0.9821 - val_loss: 0.1061 - val_accuracy: 0.9680\n",
            "Epoch 14/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0560 - accuracy: 0.9832 - val_loss: 0.0998 - val_accuracy: 0.9741\n",
            "Epoch 15/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0539 - accuracy: 0.9832 - val_loss: 0.0921 - val_accuracy: 0.9738\n",
            "Epoch 16/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0490 - accuracy: 0.9848 - val_loss: 0.0831 - val_accuracy: 0.9764\n",
            "Epoch 17/20\n",
            "469/469 [==============================] - 2s 5ms/step - loss: 0.0484 - accuracy: 0.9847 - val_loss: 0.0898 - val_accuracy: 0.9752\n",
            "Epoch 18/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0443 - accuracy: 0.9863 - val_loss: 0.0881 - val_accuracy: 0.9774\n",
            "Epoch 19/20\n",
            "469/469 [==============================] - 3s 5ms/step - loss: 0.0406 - accuracy: 0.9869 - val_loss: 0.0831 - val_accuracy: 0.9765\n",
            "Epoch 20/20\n",
            "469/469 [==============================] - 3s 6ms/step - loss: 0.0409 - accuracy: 0.9874 - val_loss: 0.0936 - val_accuracy: 0.9755\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f7d301d6d30>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    }
  ]
}