{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qofg_NrDnFSH"
   },
   "source": [
    "**12장 – 텐서플로를 사용한 사용자 정의 모델과 훈련**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oAgZdgLUnFSJ"
   },
   "source": [
    "_이 노트북은 12장에 있는 모든 샘플 코드와 연습문제 해답을 가지고 있습니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i8anu_aEnFSJ"
   },
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/handson-ml2/blob/master/12_custom_models_and_training_with_tensorflow.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />구글 코랩에서 실행하기</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_tnhYgkcnFSK"
   },
   "source": [
    "# 설정"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z_MO6EFinFSL"
   },
   "source": [
    "먼저 몇 개의 모듈을 임포트합니다. 맷플롯립 그래프를 인라인으로 출력하도록 만들고 그림을 저장하는 함수를 준비합니다. 또한 파이썬 버전이 3.5 이상인지 확인합니다(파이썬 2.x에서도 동작하지만 곧 지원이 중단되므로 파이썬 3을 사용하는 것이 좋습니다). 사이킷런 버전이 0.20 이상인지와 텐서플로 버전이 2.0 이상인지 확인합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:30.231979Z",
     "iopub.status.busy": "2021-07-04T08:14:30.230153Z",
     "iopub.status.idle": "2021-07-04T08:14:34.245013Z",
     "shell.execute_reply": "2021-07-04T08:14:34.246445Z"
    },
    "id": "VBj7sKkwnFSM"
   },
   "outputs": [],
   "source": [
    "# 파이썬 ≥3.5 필수\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# 사이킷런 ≥0.20 필수\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version은 코랩 명령입니다.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# 이 노트북은 텐서플로 ≥2.4이 필요합니다\n",
    "# 2.x 버전은 대부분 동일한 결과를 만들지만 몇 가지 버그가 있습니다.\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.4\"\n",
    "\n",
    "# 공통 모듈 임포트\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# 노트북 실행 결과를 동일하게 유지하기 위해\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# 깔끔한 그래프 출력을 위해\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# 그림을 저장할 위치\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"그림 저장:\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iHim5E7UnFSN"
   },
   "source": [
    "## 텐서와 연산"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RkZT7K1pnFSN"
   },
   "source": [
    "### 텐서"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.460590Z",
     "iopub.status.busy": "2021-07-04T08:14:35.458827Z",
     "iopub.status.idle": "2021-07-04T08:14:35.467218Z",
     "shell.execute_reply": "2021-07-04T08:14:35.468643Z"
    },
    "id": "Jqh1vBv7nFSO",
    "outputId": "89bdeae2-2212-4bed-e339-2b79322b966d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant([[1., 2., 3.], [4., 5., 6.]]) # 행렬"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.477378Z",
     "iopub.status.busy": "2021-07-04T08:14:35.476485Z",
     "iopub.status.idle": "2021-07-04T08:14:35.481997Z",
     "shell.execute_reply": "2021-07-04T08:14:35.482893Z"
    },
    "id": "zz58NEmvnFSP",
    "outputId": "1eb3be74-ddf3-4e97-d19e-047220ddc7ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=42>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(42) # 스칼라"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.492311Z",
     "iopub.status.busy": "2021-07-04T08:14:35.491089Z",
     "iopub.status.idle": "2021-07-04T08:14:35.495539Z",
     "shell.execute_reply": "2021-07-04T08:14:35.496432Z"
    },
    "id": "cISdcS3mnFSP",
    "outputId": "bdee6051-bb43-4b7c-dbee-7aa379848add"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = tf.constant([[1., 2., 3.], [4., 5., 6.]])\n",
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.503752Z",
     "iopub.status.busy": "2021-07-04T08:14:35.502688Z",
     "iopub.status.idle": "2021-07-04T08:14:35.506860Z",
     "shell.execute_reply": "2021-07-04T08:14:35.507697Z"
    },
    "id": "sCixixr5nFSP",
    "outputId": "a5ea39a6-034c-4328-c9ac-e7da005f800a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 3])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.514996Z",
     "iopub.status.busy": "2021-07-04T08:14:35.513990Z",
     "iopub.status.idle": "2021-07-04T08:14:35.518551Z",
     "shell.execute_reply": "2021-07-04T08:14:35.519382Z"
    },
    "id": "uuds8EZnnFSQ",
    "outputId": "391b6994-18ac-41c3-8728-71af3352e8e0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tf.float32"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5LcmbcOznFSQ"
   },
   "source": [
    "### 인덱싱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:35.527030Z",
     "iopub.status.busy": "2021-07-04T08:14:35.525947Z",
     "iopub.status.idle": "2021-07-04T08:14:36.017537Z",
     "shell.execute_reply": "2021-07-04T08:14:36.018983Z"
    },
    "id": "rQt1EHV-nFSQ",
    "outputId": "f06c31e5-c53f-4318-e86a-fff585ef9335"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[2., 3.],\n",
       "       [5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[:, 1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:36.027301Z",
     "iopub.status.busy": "2021-07-04T08:14:36.026287Z",
     "iopub.status.idle": "2021-07-04T08:14:36.032628Z",
     "shell.execute_reply": "2021-07-04T08:14:36.033409Z"
    },
    "id": "-8jI6UyZnFSR",
    "outputId": "57edf7a3-84dc-46e2-9ca4-63e2feb26c0d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[..., 1, tf.newaxis]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7Ny0ZhPFnFSR"
   },
   "source": [
    "### 연산"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:36.040360Z",
     "iopub.status.busy": "2021-07-04T08:14:36.039355Z",
     "iopub.status.idle": "2021-07-04T08:14:36.046051Z",
     "shell.execute_reply": "2021-07-04T08:14:36.046883Z"
    },
    "id": "k_S2puMnnFSR",
    "outputId": "fd744bb0-e740-4762-80ec-9e22b4559357"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[11., 12., 13.],\n",
       "       [14., 15., 16.]], dtype=float32)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t + 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:36.053653Z",
     "iopub.status.busy": "2021-07-04T08:14:36.052594Z",
     "iopub.status.idle": "2021-07-04T08:14:36.059689Z",
     "shell.execute_reply": "2021-07-04T08:14:36.060521Z"
    },
    "id": "FQkXtyMFnFSR",
    "outputId": "e331d0ac-2858-4e70-8e85-97ca9683c772"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[ 1.,  4.,  9.],\n",
       "       [16., 25., 36.]], dtype=float32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.square(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:36.068783Z",
     "iopub.status.busy": "2021-07-04T08:14:36.067640Z",
     "iopub.status.idle": "2021-07-04T08:14:37.295123Z",
     "shell.execute_reply": "2021-07-04T08:14:37.296584Z"
    },
    "id": "9RgHRMwpnFSR",
    "outputId": "646a5b4c-c04c-4f68-b513-e7be218c5b52"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[14., 32.],\n",
       "       [32., 77.]], dtype=float32)>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t @ tf.transpose(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FepPlGrBnFSS"
   },
   "source": [
    "### `keras.backend` 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.307817Z",
     "iopub.status.busy": "2021-07-04T08:14:37.306372Z",
     "iopub.status.idle": "2021-07-04T08:14:37.313260Z",
     "shell.execute_reply": "2021-07-04T08:14:37.314389Z"
    },
    "id": "ll9jO3G6nFSS",
    "outputId": "4e09e7a9-a30b-4672-da11-685109b18d81"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[11., 26.],\n",
       "       [14., 35.],\n",
       "       [19., 46.]], dtype=float32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "K = keras.backend\n",
    "K.square(K.transpose(t)) + 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QrqIuM0NnFSS"
   },
   "source": [
    "### 넘파이 변환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.323539Z",
     "iopub.status.busy": "2021-07-04T08:14:37.322635Z",
     "iopub.status.idle": "2021-07-04T08:14:37.329008Z",
     "shell.execute_reply": "2021-07-04T08:14:37.328192Z"
    },
    "id": "_J3NWmTVnFSS",
    "outputId": "4d10c3d1-a661-46d6-e4e2-9de4083741ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 4., 5.])>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([2., 4., 5.])\n",
    "tf.constant(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.337928Z",
     "iopub.status.busy": "2021-07-04T08:14:37.336692Z",
     "iopub.status.idle": "2021-07-04T08:14:37.341174Z",
     "shell.execute_reply": "2021-07-04T08:14:37.342140Z"
    },
    "id": "LSnlGqM4nFSS",
    "outputId": "7d120ee6-d494-4e76-925f-13f64b1d0dd2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.350290Z",
     "iopub.status.busy": "2021-07-04T08:14:37.349129Z",
     "iopub.status.idle": "2021-07-04T08:14:37.354304Z",
     "shell.execute_reply": "2021-07-04T08:14:37.355414Z"
    },
    "id": "IpmcaxRRnFSS",
    "outputId": "ab99b995-ddd7-454b-ad25-25ed3a2aa8c1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.363283Z",
     "iopub.status.busy": "2021-07-04T08:14:37.361860Z",
     "iopub.status.idle": "2021-07-04T08:14:37.367717Z",
     "shell.execute_reply": "2021-07-04T08:14:37.368496Z"
    },
    "id": "xME7iW_VnFST",
    "outputId": "076e8535-e490-4e25-b713-019f5d0e0134"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 4., 16., 25.])>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.square(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.376267Z",
     "iopub.status.busy": "2021-07-04T08:14:37.375251Z",
     "iopub.status.idle": "2021-07-04T08:14:37.379815Z",
     "shell.execute_reply": "2021-07-04T08:14:37.380710Z"
    },
    "id": "GaHgPvyNnFST",
    "outputId": "4ea536f5-09a1-400a-e90b-a3e84bd1cbbc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  4.,  9.],\n",
       "       [16., 25., 36.]], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.square(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qEscuNeMnFST"
   },
   "source": [
    "### 타입 변환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.389192Z",
     "iopub.status.busy": "2021-07-04T08:14:37.387744Z",
     "iopub.status.idle": "2021-07-04T08:14:37.393285Z",
     "shell.execute_reply": "2021-07-04T08:14:37.394243Z"
    },
    "id": "zmBGBSFxnFSU",
    "outputId": "c560f0a8-e5c3-40ad-aedc-e7ca447c85ec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.constant(2.0) + tf.constant(40)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.403920Z",
     "iopub.status.busy": "2021-07-04T08:14:37.402792Z",
     "iopub.status.idle": "2021-07-04T08:14:37.407612Z",
     "shell.execute_reply": "2021-07-04T08:14:37.408498Z"
    },
    "id": "1dztSrtpnFSU",
    "outputId": "24d5960f-d0b8-4be0-bac7-3615e1a73f87"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a double tensor [Op:AddV2]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.constant(2.0) + tf.constant(40., dtype=tf.float64)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.417591Z",
     "iopub.status.busy": "2021-07-04T08:14:37.416454Z",
     "iopub.status.idle": "2021-07-04T08:14:37.423410Z",
     "shell.execute_reply": "2021-07-04T08:14:37.422467Z"
    },
    "id": "bml4toHdnFSU",
    "outputId": "4fbd9268-815d-4c81-913a-591a112a16a1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=42.0>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = tf.constant(40., dtype=tf.float64)\n",
    "tf.constant(2.0) + tf.cast(t2, tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rUX0FqA2nFSU"
   },
   "source": [
    "### 문자열"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.431405Z",
     "iopub.status.busy": "2021-07-04T08:14:37.430323Z",
     "iopub.status.idle": "2021-07-04T08:14:37.435955Z",
     "shell.execute_reply": "2021-07-04T08:14:37.435068Z"
    },
    "id": "kmfyUUWDnFSU",
    "outputId": "ab1a8bc7-fc20-4b11-8889-e6c8e962f9cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'hello world'>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(b\"hello world\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.443720Z",
     "iopub.status.busy": "2021-07-04T08:14:37.442656Z",
     "iopub.status.idle": "2021-07-04T08:14:37.447589Z",
     "shell.execute_reply": "2021-07-04T08:14:37.448418Z"
    },
    "id": "S4eTX-ZvnFSU",
    "outputId": "fe1b8126-82ec-4085-c591-0d60c9f374f1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'caf\\xc3\\xa9'>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(\"café\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.457168Z",
     "iopub.status.busy": "2021-07-04T08:14:37.456101Z",
     "iopub.status.idle": "2021-07-04T08:14:37.461581Z",
     "shell.execute_reply": "2021-07-04T08:14:37.462492Z"
    },
    "id": "M3FuAE4gnFSV",
    "outputId": "84215369-038c-41b3-a35b-fb29fadc32fc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233], dtype=int32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u = tf.constant([ord(c) for c in \"café\"])\n",
    "u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.471695Z",
     "iopub.status.busy": "2021-07-04T08:14:37.470569Z",
     "iopub.status.idle": "2021-07-04T08:14:37.480015Z",
     "shell.execute_reply": "2021-07-04T08:14:37.479070Z"
    },
    "id": "tg50H0XNnFSV",
    "outputId": "4ed9cbb9-a09c-4168-ee44-901c1c2a1832"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=4>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = tf.strings.unicode_encode(u, \"UTF-8\")\n",
    "tf.strings.length(b, unit=\"UTF8_CHAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.487508Z",
     "iopub.status.busy": "2021-07-04T08:14:37.486374Z",
     "iopub.status.idle": "2021-07-04T08:14:37.492583Z",
     "shell.execute_reply": "2021-07-04T08:14:37.493464Z"
    },
    "id": "0gjS_MnenFSV",
    "outputId": "24b37c8f-6b91-4796-b68e-0723c6517e58"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233], dtype=int32)>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.unicode_decode(b, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xIunxUy1nFSV"
   },
   "source": [
    "### 문자열 배열"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.501430Z",
     "iopub.status.busy": "2021-07-04T08:14:37.500296Z",
     "iopub.status.idle": "2021-07-04T08:14:37.503764Z",
     "shell.execute_reply": "2021-07-04T08:14:37.504594Z"
    },
    "id": "huIxMcD8nFSV"
   },
   "outputs": [],
   "source": [
    "p = tf.constant([\"Café\", \"Coffee\", \"caffè\", \"咖啡\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.512697Z",
     "iopub.status.busy": "2021-07-04T08:14:37.511629Z",
     "iopub.status.idle": "2021-07-04T08:14:37.516662Z",
     "shell.execute_reply": "2021-07-04T08:14:37.517486Z"
    },
    "id": "79sKVmlsnFSV",
    "outputId": "f630892f-5661-4cf9-e20d-e3df377296bb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([4, 6, 5, 2], dtype=int32)>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.length(p, unit=\"UTF8_CHAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.524818Z",
     "iopub.status.busy": "2021-07-04T08:14:37.523764Z",
     "iopub.status.idle": "2021-07-04T08:14:37.529752Z",
     "shell.execute_reply": "2021-07-04T08:14:37.530586Z"
    },
    "id": "iZn8rFS3nFSV",
    "outputId": "d9c297eb-6a3f-4b2b-c7a3-ccc4dee2aea1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857]]>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = tf.strings.unicode_decode(p, \"UTF8\")\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.537541Z",
     "iopub.status.busy": "2021-07-04T08:14:37.536487Z",
     "iopub.status.idle": "2021-07-04T08:14:37.541569Z",
     "shell.execute_reply": "2021-07-04T08:14:37.540647Z"
    },
    "id": "w_zPFyFCnFSV",
    "outputId": "475fcba7-c220-4c49-e49e-dedf29b7418e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857]]>\n"
     ]
    }
   ],
   "source": [
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vk6tGFd4nFSW"
   },
   "source": [
    "### 래그드 텐서"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.548915Z",
     "iopub.status.busy": "2021-07-04T08:14:37.547863Z",
     "iopub.status.idle": "2021-07-04T08:14:37.555844Z",
     "shell.execute_reply": "2021-07-04T08:14:37.556673Z"
    },
    "id": "kc0CcQ9-nFSW",
    "outputId": "96652dc2-c317-46f8-99dd-e24087c6bf51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 67 111 102 102 101 101], shape=(6,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "print(r[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.564019Z",
     "iopub.status.busy": "2021-07-04T08:14:37.562968Z",
     "iopub.status.idle": "2021-07-04T08:14:37.572286Z",
     "shell.execute_reply": "2021-07-04T08:14:37.573167Z"
    },
    "id": "RC13J_djnFSW",
    "outputId": "fb0c7250-9608-416d-b969-6ca40cf2244f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232]]>\n"
     ]
    }
   ],
   "source": [
    "print(r[1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.582723Z",
     "iopub.status.busy": "2021-07-04T08:14:37.581478Z",
     "iopub.status.idle": "2021-07-04T08:14:37.589220Z",
     "shell.execute_reply": "2021-07-04T08:14:37.590190Z"
    },
    "id": "hGTY7YJunFSW",
    "outputId": "87121768-2448-4592-f557-04eedfe1e523"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857], [65, 66], [], [67]]>\n"
     ]
    }
   ],
   "source": [
    "r2 = tf.ragged.constant([[65, 66], [], [67]])\n",
    "print(tf.concat([r, r2], axis=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.601046Z",
     "iopub.status.busy": "2021-07-04T08:14:37.599831Z",
     "iopub.status.idle": "2021-07-04T08:14:37.610876Z",
     "shell.execute_reply": "2021-07-04T08:14:37.611756Z"
    },
    "id": "jK9hMKkmnFSW",
    "outputId": "2b0e09d2-25cd-4481-8cf3-fe03e59c3818"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233, 68, 69, 70], [67, 111, 102, 102, 101, 101, 71], [99, 97, 102, 102, 232], [21654, 21857, 72, 73]]>\n"
     ]
    }
   ],
   "source": [
    "r3 = tf.ragged.constant([[68, 69, 70], [71], [], [72, 73]])\n",
    "print(tf.concat([r, r3], axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.618402Z",
     "iopub.status.busy": "2021-07-04T08:14:37.617243Z",
     "iopub.status.idle": "2021-07-04T08:14:37.623622Z",
     "shell.execute_reply": "2021-07-04T08:14:37.624500Z"
    },
    "id": "twTle-bpnFSW",
    "outputId": "a845e545-1959-4241-c533-96c0c6ae4c11"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=string, numpy=array([b'DEF', b'G', b'', b'HI'], dtype=object)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.unicode_encode(r3, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.631450Z",
     "iopub.status.busy": "2021-07-04T08:14:37.630382Z",
     "iopub.status.idle": "2021-07-04T08:14:37.638374Z",
     "shell.execute_reply": "2021-07-04T08:14:37.639202Z"
    },
    "id": "qGBmIeoLnFSW",
    "outputId": "ec66f750-2fc3-4fcc-b320-3ffb2b35270c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 6), dtype=int32, numpy=\n",
       "array([[   67,    97,   102,   233,     0,     0],\n",
       "       [   67,   111,   102,   102,   101,   101],\n",
       "       [   99,    97,   102,   102,   232,     0],\n",
       "       [21654, 21857,     0,     0,     0,     0]], dtype=int32)>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.to_tensor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tMc8-5sjnFSW"
   },
   "source": [
    "### 희소 텐서"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.650301Z",
     "iopub.status.busy": "2021-07-04T08:14:37.649286Z",
     "iopub.status.idle": "2021-07-04T08:14:37.653371Z",
     "shell.execute_reply": "2021-07-04T08:14:37.654415Z"
    },
    "id": "71zD-IL-nFSW"
   },
   "outputs": [],
   "source": [
    "s = tf.SparseTensor(indices=[[0, 1], [1, 0], [2, 3]],\n",
    "                    values=[1., 2., 3.],\n",
    "                    dense_shape=[3, 4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.662711Z",
     "iopub.status.busy": "2021-07-04T08:14:37.661456Z",
     "iopub.status.idle": "2021-07-04T08:14:37.666556Z",
     "shell.execute_reply": "2021-07-04T08:14:37.665402Z"
    },
    "id": "gDXfjz0onFSW",
    "outputId": "286ee4b1-a165-459c-8c08-328abb208230"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 1]\n",
      " [1 0]\n",
      " [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.673495Z",
     "iopub.status.busy": "2021-07-04T08:14:37.672384Z",
     "iopub.status.idle": "2021-07-04T08:14:37.679187Z",
     "shell.execute_reply": "2021-07-04T08:14:37.680028Z"
    },
    "id": "6EECeDVVnFSX",
    "outputId": "bedaca50-177f-40f4-9c41-b21f2f13f222"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 1., 0., 0.],\n",
       "       [2., 0., 0., 0.],\n",
       "       [0., 0., 0., 3.]], dtype=float32)>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sparse.to_dense(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.686416Z",
     "iopub.status.busy": "2021-07-04T08:14:37.685349Z",
     "iopub.status.idle": "2021-07-04T08:14:37.689468Z",
     "shell.execute_reply": "2021-07-04T08:14:37.690529Z"
    },
    "id": "R2glxW27nFSX"
   },
   "outputs": [],
   "source": [
    "s2 = s * 2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.698394Z",
     "iopub.status.busy": "2021-07-04T08:14:37.697217Z",
     "iopub.status.idle": "2021-07-04T08:14:37.702468Z",
     "shell.execute_reply": "2021-07-04T08:14:37.701461Z"
    },
    "id": "NAYv7KbanFSX",
    "outputId": "f15fdcb7-5d3d-418c-e188-058565b8eced"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unsupported operand type(s) for +: 'SparseTensor' and 'float'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    s3 = s + 1.\n",
    "except TypeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.712098Z",
     "iopub.status.busy": "2021-07-04T08:14:37.710568Z",
     "iopub.status.idle": "2021-07-04T08:14:37.716915Z",
     "shell.execute_reply": "2021-07-04T08:14:37.717891Z"
    },
    "id": "KBvWo_EnnFSX",
    "outputId": "62ea5d33-8add-4ce1-ab4b-68dc66af270b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[ 30.,  40.],\n",
       "       [ 20.,  40.],\n",
       "       [210., 240.]], dtype=float32)>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s4 = tf.constant([[10., 20.], [30., 40.], [50., 60.], [70., 80.]])\n",
    "tf.sparse.sparse_dense_matmul(s, s4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.728268Z",
     "iopub.status.busy": "2021-07-04T08:14:37.727064Z",
     "iopub.status.idle": "2021-07-04T08:14:37.731227Z",
     "shell.execute_reply": "2021-07-04T08:14:37.732106Z"
    },
    "id": "iVef5eaJnFSX",
    "outputId": "ecfb8acb-6767-49c2-d254-afe7d8b5886a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 2]\n",
      " [0 1]], shape=(2, 2), dtype=int64), values=tf.Tensor([1. 2.], shape=(2,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "s5 = tf.SparseTensor(indices=[[0, 2], [0, 1]],\n",
    "                     values=[1., 2.],\n",
    "                     dense_shape=[3, 4])\n",
    "print(s5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.739701Z",
     "iopub.status.busy": "2021-07-04T08:14:37.738641Z",
     "iopub.status.idle": "2021-07-04T08:14:37.743634Z",
     "shell.execute_reply": "2021-07-04T08:14:37.744482Z"
    },
    "id": "RvZ3zednnFSX",
    "outputId": "605e4642-1285-4eef-cca8-1f6a5120008f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "indices[1] is out of order. Many sparse ops require sorted indices.\n",
      "  Use `tf.sparse.reorder` to create a correctly ordered copy.\n",
      "\n",
      " [Op:SparseToDense]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.sparse.to_dense(s5)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.752642Z",
     "iopub.status.busy": "2021-07-04T08:14:37.751599Z",
     "iopub.status.idle": "2021-07-04T08:14:37.759282Z",
     "shell.execute_reply": "2021-07-04T08:14:37.760238Z"
    },
    "id": "049PReJ_nFSX",
    "outputId": "5ae7117f-5231-4421-a8f6-d42b6486e290"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 2., 1., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s6 = tf.sparse.reorder(s5)\n",
    "tf.sparse.to_dense(s6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kwMI9FoNnFSX"
   },
   "source": [
    "### 집합"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.771543Z",
     "iopub.status.busy": "2021-07-04T08:14:37.770349Z",
     "iopub.status.idle": "2021-07-04T08:14:37.775886Z",
     "shell.execute_reply": "2021-07-04T08:14:37.776817Z"
    },
    "id": "gaYypzW3nFSX",
    "outputId": "a8a50dff-d462-409f-8d25-4f965d5a7725"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 6), dtype=int32, numpy=\n",
       "array([[ 2,  3,  4,  5,  6,  7],\n",
       "       [ 0,  7,  9, 10,  0,  0]], dtype=int32)>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set1 = tf.constant([[2, 3, 5, 7], [7, 9, 0, 0]])\n",
    "set2 = tf.constant([[4, 5, 6], [9, 10, 0]])\n",
    "tf.sparse.to_dense(tf.sets.union(set1, set2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.784203Z",
     "iopub.status.busy": "2021-07-04T08:14:37.783011Z",
     "iopub.status.idle": "2021-07-04T08:14:37.789094Z",
     "shell.execute_reply": "2021-07-04T08:14:37.789950Z"
    },
    "id": "ywZU2OS-nFSY",
    "outputId": "5ff120ad-06c6-40cc-deaf-96f073507f5e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=int32, numpy=\n",
       "array([[2, 3, 7],\n",
       "       [7, 0, 0]], dtype=int32)>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sparse.to_dense(tf.sets.difference(set1, set2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.797847Z",
     "iopub.status.busy": "2021-07-04T08:14:37.796714Z",
     "iopub.status.idle": "2021-07-04T08:14:37.802344Z",
     "shell.execute_reply": "2021-07-04T08:14:37.803228Z"
    },
    "id": "1C8SMyv-nFSY",
    "outputId": "1d1e6ed1-3275-446b-83f9-ef168d10aa16"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[5, 0],\n",
       "       [0, 9]], dtype=int32)>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sparse.to_dense(tf.sets.intersection(set1, set2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aHM1uuinnFSY"
   },
   "source": [
    "### 변수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.812882Z",
     "iopub.status.busy": "2021-07-04T08:14:37.811337Z",
     "iopub.status.idle": "2021-07-04T08:14:37.815257Z",
     "shell.execute_reply": "2021-07-04T08:14:37.816519Z"
    },
    "id": "qSn5cvBrnFSY"
   },
   "outputs": [],
   "source": [
    "v = tf.Variable([[1., 2., 3.], [4., 5., 6.]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.824588Z",
     "iopub.status.busy": "2021-07-04T08:14:37.823168Z",
     "iopub.status.idle": "2021-07-04T08:14:37.829715Z",
     "shell.execute_reply": "2021-07-04T08:14:37.830725Z"
    },
    "id": "OU0pq8XQnFSY",
    "outputId": "42d986af-88e0-49b3-993e-e023b65d4f24"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2.,  4.,  6.],\n",
       "       [ 8., 10., 12.]], dtype=float32)>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.assign(2 * v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.838313Z",
     "iopub.status.busy": "2021-07-04T08:14:37.836989Z",
     "iopub.status.idle": "2021-07-04T08:14:37.844777Z",
     "shell.execute_reply": "2021-07-04T08:14:37.846270Z"
    },
    "id": "sdy-xfQLnFSY",
    "outputId": "e102663e-464b-4b80-f751-f1735c851d30"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2., 42.,  6.],\n",
       "       [ 8., 10., 12.]], dtype=float32)>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[0, 1].assign(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.855848Z",
     "iopub.status.busy": "2021-07-04T08:14:37.854468Z",
     "iopub.status.idle": "2021-07-04T08:14:37.862352Z",
     "shell.execute_reply": "2021-07-04T08:14:37.863231Z"
    },
    "id": "p-o6a231nFSY",
    "outputId": "b57aa092-a87d-45bd-89a9-0d5711b2b206"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2., 42.,  0.],\n",
       "       [ 8., 10.,  1.]], dtype=float32)>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[:, 2].assign([0., 1.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.872273Z",
     "iopub.status.busy": "2021-07-04T08:14:37.871171Z",
     "iopub.status.idle": "2021-07-04T08:14:37.876014Z",
     "shell.execute_reply": "2021-07-04T08:14:37.876881Z"
    },
    "id": "qdNmt8zPnFSY",
    "outputId": "858b660a-ebe7-4879-d896-5ff1527c0a22"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'ResourceVariable' object does not support item assignment\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    v[1] = [7., 8., 9.]\n",
    "except TypeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.884868Z",
     "iopub.status.busy": "2021-07-04T08:14:37.883764Z",
     "iopub.status.idle": "2021-07-04T08:14:37.890751Z",
     "shell.execute_reply": "2021-07-04T08:14:37.891615Z"
    },
    "id": "-Vne6RM8nFSZ",
    "outputId": "4c4f03f1-f7c9-45e3-8bb5-351f5eabd281"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[100.,  42.,   0.],\n",
       "       [  8.,  10., 200.]], dtype=float32)>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.scatter_nd_update(indices=[[0, 0], [1, 2]],\n",
    "                    updates=[100., 200.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.900631Z",
     "iopub.status.busy": "2021-07-04T08:14:37.899531Z",
     "iopub.status.idle": "2021-07-04T08:14:37.906859Z",
     "shell.execute_reply": "2021-07-04T08:14:37.907721Z"
    },
    "id": "IO7z_1ZxnFSZ",
    "outputId": "3db41b44-6b3c-4fde-bc22-b70ed1a744f0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[4., 5., 6.],\n",
       "       [1., 2., 3.]], dtype=float32)>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sparse_delta = tf.IndexedSlices(values=[[1., 2., 3.], [4., 5., 6.]],\n",
    "                                indices=[1, 0])\n",
    "v.scatter_update(sparse_delta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IG6tPHOxnFSZ"
   },
   "source": [
    "### 텐서 배열"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.918915Z",
     "iopub.status.busy": "2021-07-04T08:14:37.917726Z",
     "iopub.status.idle": "2021-07-04T08:14:37.921144Z",
     "shell.execute_reply": "2021-07-04T08:14:37.922086Z"
    },
    "id": "4ePYHLefnFSZ"
   },
   "outputs": [],
   "source": [
    "array = tf.TensorArray(dtype=tf.float32, size=3)\n",
    "array = array.write(0, tf.constant([1., 2.]))\n",
    "array = array.write(1, tf.constant([3., 10.]))\n",
    "array = array.write(2, tf.constant([5., 7.]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.930696Z",
     "iopub.status.busy": "2021-07-04T08:14:37.928997Z",
     "iopub.status.idle": "2021-07-04T08:14:37.933286Z",
     "shell.execute_reply": "2021-07-04T08:14:37.934179Z"
    },
    "id": "ZWrJYjaNnFSZ",
    "outputId": "5718f4cb-5884-4e08-989e-2ef6d72622d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([ 3., 10.], dtype=float32)>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array.read(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.940457Z",
     "iopub.status.busy": "2021-07-04T08:14:37.939418Z",
     "iopub.status.idle": "2021-07-04T08:14:37.945931Z",
     "shell.execute_reply": "2021-07-04T08:14:37.946707Z"
    },
    "id": "qqJJof_tnFSa",
    "outputId": "f9fb09d2-c693-4fc2-9bb6-e4e01b0f9c45"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[1., 2.],\n",
       "       [0., 0.],\n",
       "       [5., 7.]], dtype=float32)>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.954033Z",
     "iopub.status.busy": "2021-07-04T08:14:37.953006Z",
     "iopub.status.idle": "2021-07-04T08:14:37.959558Z",
     "shell.execute_reply": "2021-07-04T08:14:37.960437Z"
    },
    "id": "Af5sF8MhnFSa",
    "outputId": "ae026565-749d-4100-bfe0-b32f97b9bba9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([2., 3.], dtype=float32)>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean, variance = tf.nn.moments(array.stack(), axes=0)\n",
    "mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.966999Z",
     "iopub.status.busy": "2021-07-04T08:14:37.965953Z",
     "iopub.status.idle": "2021-07-04T08:14:37.971590Z",
     "shell.execute_reply": "2021-07-04T08:14:37.972416Z"
    },
    "id": "32DE6kdEnFSa",
    "outputId": "bc3ba2d5-c6ff-480f-9554-0c2939f5a223"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([4.6666665, 8.666667 ], dtype=float32)>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XPbiThrvnFSa"
   },
   "source": [
    "## 사용자 정의 손실 함수"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g6PwQCm3nFSa"
   },
   "source": [
    "캘리포니아 주택 데이터셋을 로드하여 준비해 보겠습니다. 먼저 이 데이터셋을 로드한 다음 훈련 세트, 검증 세트, 테스트 세트로 나눕니다. 마지막으로 스케일을 변경합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:37.984999Z",
     "iopub.status.busy": "2021-07-04T08:14:37.983928Z",
     "iopub.status.idle": "2021-07-04T08:14:38.072608Z",
     "shell.execute_reply": "2021-07-04T08:14:38.073492Z"
    },
    "id": "4Ze9UWvrnFSa",
    "outputId": "853c189b-f078-47dc-d0bf-92fde2030287"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading Cal. housing from https://ndownloader.figshare.com/files/5976036 to /root/scikit_learn_data\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target.reshape(-1, 1), random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_valid_scaled = scaler.transform(X_valid)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:38.082413Z",
     "iopub.status.busy": "2021-07-04T08:14:38.081277Z",
     "iopub.status.idle": "2021-07-04T08:14:38.084633Z",
     "shell.execute_reply": "2021-07-04T08:14:38.085502Z"
    },
    "id": "SGVUuLCGnFSb"
   },
   "outputs": [],
   "source": [
    "def huber_fn(y_true, y_pred):\n",
    "    error = y_true - y_pred\n",
    "    is_small_error = tf.abs(error) < 1\n",
    "    squared_loss = tf.square(error) / 2\n",
    "    linear_loss  = tf.abs(error) - 0.5\n",
    "    return tf.where(is_small_error, squared_loss, linear_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 276
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:38.104690Z",
     "iopub.status.busy": "2021-07-04T08:14:38.103561Z",
     "iopub.status.idle": "2021-07-04T08:14:38.706192Z",
     "shell.execute_reply": "2021-07-04T08:14:38.707483Z"
    },
    "id": "pr1qnz3_nFSb",
    "outputId": "b1e8262a-bf23-47df-8770-3641de052d55"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAEDCAYAAAB0/A4MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZyN5fvA8c89Y2KGsWXfIhkiWypJMomoJNVXe6hEql+WVJTWLxVSIkQbUpZKElkiI2QpxVcpsu+7MTNmMcv9++OaYSxjzsycc56zXO/X67xm5swz57meOTPnOs9zX/d1G2stSimllPKeEKcDUEoppYKNJl+llFLKyzT5KqWUUl6myVcppZTyMk2+SimllJdp8lVKKaW8TJOvUj7EGBNtjLHGmDJe2l9XY0yCN/allDpNk69SBWSMmWCMmX2e+6/KTKTVvR+VUsqXafJVKggYYy5yOgal1GmafJXykvNdUjbGVM+876qzNr/WGLPWGJNsjFljjGly1mNdZ4xZYoxJNMbsMcaMNcYUz/b9mMz73jHGHAKW5yHOHsaYzcaYk5kfHz/P9zdlxnbYGDPfGFMo83v1jTGLjDFxxpgEY8w6Y8yNefk9KRUMNPkq5ZveAV4ArgK2ArONMREgCQ5YAMwCGgJ3AY2AT896jIcAA7QAOruyU2PMncAHwAjgCuB9YIwx5vbM718FjAZeB2oDNwHzsj3El8A+4JrMmF4Dkl0+aqWCRCGnA1AqQLQ7T+FSQd7c/tdaOx/AGPMIsBt4APgYeA6YZq0dnrWxMaYn8Icxppy19mDm3dustc/mcb/9gM+ttR9kfr0p86z7BeB7oBpwAphlrY0HdgDrsv38JcA71tp/Mr/enMf9KxUU9MxXKff4GTnTy357oACPtyLrE2ttArAeqJt5VxPgoczLugmZST/rsnLNbI+xJh/7vZxzL1Evy7bvH5GEu80Y84UxposxJjLbtu8CHxtjfjLGvGSMqZOPGJQKeJp8lXKPRGvt5uw35Gw1u4zMjybbfWH52FcIcgacPdE3BGoBa7NtdyIfj50TC5B5tnslcA+wExgA/GOMqZT5/deQRD0TuA74nzHmUTfGoVRA0OSrlPccyvxYMdt9jXLY9tqsT4wxRZHx178z7/odqHd2ss+8JRUwxr+B5mfddz2wIesLa22atfYna+0AoAFQFGif7fv/WmtHWmtvAz4BuhUwJqUCjo75KuU9m4FdwGvGmP5AdWBgDtsOzKxS3gu8ApxEipkAhgArjTEfAuOAeKAOcLu1tkcBYxwGfGWMWYMUdbUDHkSKujDGtEcubf8MHAVuBCKBv40x4Uih2FfAdqA8krhXFTAmpQKOJl+lvMRam2qMuQ8YgxQprQVeBM5p0AH0B4YjFcV/Ae2ttScyH+d/xpgbgEHAEiAUqYj+1g0xzjTG/B9SeDUCGd990lr7feYmsUBH5A1BBLAF6GatXZo5l7gUMAE5uz+SeWz9ChqXUoHGWGudjkEppZQKKjrmq5RSSnlZnpKvMaZWZlebyZ4KSCmllAp0eT3zHQ386olAlFJKqWDhcvLNLBSJBRZ5LhyllFIq8LmUfDMbtr8B9PVsOEoppVTgc3Wq0X+BT6y1u40xOW5kjOkOdAcoUqRIk2rVqhU8Qh+VkZFBSEjO712shSNHClOmTIoXo3KP3I7N3wXy8e3atQtrLcH8v+fv/PH4kpNDKVw4A2MuPHvGH48tLzZt2nTYWlvWlW1zTb7GmEZAa6Bxbttaa8cD4wFq165tN27c6EoMfikmJobo6Ohct0tMhIgIz8fjTq4em78K5OOLjo4mNjaWtWvX5r6xnwrk5w/87/hWroR69SAyMvdt/e3Y8soYs8PVbV15CxKNdOLZaYzZj0yYv9sY83u+ogsix45BgwaQluZ0JEop5RkTJ8LevU5H4X9cuew8Hpia7et+SDLu6YmAAkmpUvDnn1BI+4gppQLU2LFOR+Cfcj3ztdYmWmv3Z92ABCDZWnsot59VkJoKL74oY8BKKRVI7rgD/vrL6Sj8U57PyTKXDFMuKlYMqlaVS89h+Vk8TimlfNR770EA1/Z5VOCWnfkIY6BnT9i/3+lIlFLKfb75BkqW1GG1/NLk6wVJSXDbbfJRKaUCwapVOpxWEPqexQvCw2HdOjkLVkopf5eaCkOHOh2Ff9MzXy9JT4d775V5v0op5a+shcaNYc8epyPxb3rm6yWFCkG3bjo+opTyb8bAL79A8eJOR+Lf9MzXi1q3htWrdZxEKeW/3npLX8PcQZOvl733HhzSGdJKKT+Ung4XXQRFizodif/Ti6BeZIyU5yullD86fBiefdbpKAKDnvl6mbXQpg3s3Ol0JEop5bqUFGjZEhISnI4kMOiZr5cZA6NHQ5UqTkeilFKuK1wYNmyAAF4R0Kv01+iAqCj46iuddqSU8g8nT0KXLjK/V7mHJl+HbNgABw86HYVSSrnmrrvk7Fe5h152dsjrr8tiC9Zq5yullG9bvx46dHA6isCiZ74Ouu02WLPG6SiUUipnR4/KsqgZGU5HElj0zNdB06bJqiBKKeWrSpeG+fOdjiLw6Jmvg0qWhI8/ho0bnY5EKaXOtWsX3HKLdrTyBI8l39hYXTneFaVKaem+Uso3VaoE77yjdSmu+OyzvG3vsZf9gweL0KuXtCNTObv7bqhQAeLinI5EKaVOO34cZs2CevWcjsS3ZWTAgAHw6KN5+zmPnnONHAl33AHx8Z7ci/8bOBDmzXM6CqWUOu3AAZkSqXKWmChLxb79NoSG5u1nPZZ8q1ZNpHRpmDMHWrSQsQN1fiNGwD33OB2FUkqJ9HSoWRNeesnpSHzX/v1w443w9deyvOIPP+Tt5z2WfMPD01m1Sro5rVsHTZvqtJqcGAOTJsEXXzgdiVJKwYIF0Lmz01H4rj//lJy2ejVUry7rG998c94ew6OXnS+7DFasgOho2LdPzoBnzvTkHv3X1VfDddc5HYVSSkmF89ixTkfhm+bNk9fqnTvh2mth5cr8jYt7vM42a45Y166QlCQtyt55R0vXz3b55acblyullFNWrZIeBMWLOx2J7xkzRpojxcfLUOFPP0H58vl7LK9McrnoIvj0U3jzTUm6zz0HPXpok+6zLV8OMTFOR6GUCmYRETIFUp2Wng59+sBTT0l188CBMGUKhIfn/zG91uHKGCnHvuwyGUv46CPYulUGq7XLk+jUST5qv2ellBMOH5ZCq/r1nY7EdyQkwP33w+zZEBYmuatLl4I/rtfbO3TqJGd35crBokXQrJkkYSV++AF69nQ6CqVUMPrqK3jvPaej8B27d0ut0uzZcjXgxx/dk3jBod7OTZvKuEL79vDXX/L1d99pwRHIE33ttU5HoZQKRj17aj1Olt9/h9tvh717oVYtScBRUe57fMcaG1avLmOcbdvKpY5WreQaerCLjJTLHN9843QkSqlgMnKknATpkJf8Hlq0kMR7ww0ya8ediRccXlihRAl5N9GzJ6SkwAMPwBtv6Duv9HS53KGUUt7Srh00auR0FM6yFt59F+68U7pXde4sc54vvtj9+3K8pX+hQjB6tIwzGAOvvioHnJLidGTOqVEDevWS3qpKKeVpq1fLmOYllzgdiXNSU+VE8NlnJQkPGgQTJsgUUE9wPPmCJN3eveVUv2hRmDwZWreWy9HBats2uRQf7FcBlFKeN38+/Puv01E45/hxqUEaN06S7dSp0lrTk5fgfSL5Zrn9dli6FCpXhmXLpPAoWNe6rVFDxhl0/EUp5UkZGfDyy8Fb8Lp9uxz7ggVQtiwsXiyLJXiaTyVfgMaNpRK6cWPYskUS8OLFTkfljJMn4fHHdVlGpZTn3HSTzDoJRitXymybDRugbl3JPc2aeWffPpd8Qc58ly6FDh0gNlYaVud1oeJAULSoFEFkZDgdiVIqUE2bJokn2EyfLqsSHTwIbdrI7JsaNby3f59MviCJZ8YMGfxOS5OFigcMCK5EZIz0wl6zRsd+lVLuN3SojHEG0/CWtdLq+N57ITkZuneXpW+93WnRZ5MvyOLE77wDH34on7/9tvzCEhOdjsy7hg6VtSOVUspd0tIkERUt6nQk3nPyJDzyyOliquHDJb+EhXk/Fp9Ovll69IC5c2WVja+/lksFwZKMjJErABUqOB2JUiqQHDwIL7wg0z2DwdGjMoQ5caIsHjFjBvTt69xZv18kX5Br8itWSGes1atlkHz9eqej8p577oHffnM6CqVUIDh6VNbsTUtzOhLv+PdfKd5dsgQqVoSff4aOHZ2NyaXka4yZbIzZZ4yJM8ZsMsZ083Rg51O3rlSnXXutLGTcvLksbBwMhg2DK690OgqlVCAoXRrWrg2Os96ff5ac8e+/0LChnLw1aeJ0VK6f+b4FVLfWFgc6AIOMMY6EX768LGB8772yoPFtt8kCx4GuenVZ8ShY5z0rpdxj61bo1i04iqwmTZKGTUePShONZcugShWnoxIuJV9r7V/W2qyGjzbzVtNjUeUiPBy+/FIWNM7IkAWOe/cO/Pmwx4/LGw6llMqvChXgscecjsKzshqHdOkibSN79YKZM6FYMacjO81YF+ewGGPGAF2BcOAP4AZrbcJZ23QHugOULVu2yfTp090a7PnMn1+ed96pTVpaCM2aHWbgwL+JiPB8Fk5ISKCYA89kerohJSXEo8fo1LF5SyAfX+/evUlPT2fUqFFOh+Ixgfz8gWeP79ixMPbtC6du3TiPPH5uvPHcnTwZwttv12Hx4nKEhFiefvpf7rxzr0f3meXGG29cY629yqWNrbUu34BQ4HpgIBB2oW2joqKstyxZYm3p0taCtQ0bWrtrl+f3uXjxYs/v5DwGDbJ2+HDP7sOpY/OWQD6+li1b2oYNGzodhkcF8vNnrWePb8UKeQ1xiqefuwMHrG3WTHJBZKS1c+d6dHfnAH6zLubTPFU7W2vTrbXLgCpAzzy9JfCgG26QQqxatWDdOrjmGmlMEYief17K45VSKi/S0qTw6KWXnI7EMzZskFkwK1ZAtWrSsapdO6ejyll+pxoVwsEx3/OpVUsScMuWsG+fJOSZM52Oyv3CwuSPasgQpyNRSvmT99+XRkWB6McfpSfz9u1w9dXSo7l+faejurBck68xppwx5j5jTDFjTKgxpi1wP7DI8+HlTenSsjJFly7SBeuuu6SDSaC1Zrz0UoiOdjoKpZQ/6dVL1qsNNB99JHOW4+Lg7rshJsY/mhK5cuZrkUvMu4FjwDtAb2vtLE8Gll8XXSSLMAweLEm3Xz944gmpeAsUFSvKu7qlS52ORCnlDyZOhP/9D0qUcDoS90lPh+eek97M6enQv78slhAR4XRkrsl1irW19hDQ0guxuI0x8OKLcNllchY8frzMbfvqK+83z/aUI0fg00+hRQunI1FK+brSpQMr8Z44AQ89JEOLhQpJf2Z/mz7lN+0l8+Oee2Qt4HLlYOFCWTB52zano3KPqlXlDD/Q5zYrpQpm40ZpRlTTp6p08m/vXqntmTlTTqbmz/e/xAsBnnxBqvtWrYJ69eDvv09XwwWCpCS44orgW+VJKeW6Z5+FLVucjsI91q2T1/A1a6T2ZcUKaNXK6ajyJ+CTL0hrxuXLZUWLQ4dkVaSpU52OquDCw2Xc11/GOJRS3jd7tswG8Xdz5kg//9275eOqVVCnjtNR5V9QJF+Q8Y45c6T4KiUF7r8fBg3y/0roMmXkOI4edToSpZQvSUyE668PjCtjI0dChw4y1vvAAzKMWKaM01EVTNAkX5CB+TFj4L33pCgrq/dnSkruP+vLqlULnqXBlFKuiYiQYlN/vjKWlgZPPy3TpDIy4LXXYPJkKFLE6cgKLqiSL0jS7d1bBuuLFoXPP5e1go8ccTqy/OvcWf4wjx93OhKllC84cQI++USWYfVXcXFytjt6tEwhnTwZXn01cFZjCrrkm6VDBxkvrVRJPl57rX8v1/fmm/DLL05HoZTyBUePwuHDTkeRfzt3yiXzuXPl8vKiRfDgg05H5V5Bm3wBGjeWhZUbN4bNm6U9WUyM01Hlz8iR0uVFKRXckpPh4ovhhRecjiR/fv1V+vOvXw+1a0vb4Ouvdzoq9wvq5AtQuTL8/LOcCR87JhXRn33mdFT5M2kSDB3qdBRKKSctWCDjpP7om29kDu+BAzIrZcWKwJmffLagT74gCyzPmCGrBaWmwqOPSoesjAynI8ub1q0ldqVU8OrQAcaNczqKvLFWThz+8x/pX/DYYzBvHpQq5XRknqPJN1NoqCzCMHasfP7WW3DvvfKH4C8qVZLK7a++cjoSpZQTRo6EWbNk9TN/cfIkPP746cvkQ4bIYgkXXeRsXJ6Wa2/nYPPEE9I5pVMn+PprGfifNQvKl3c6MtdkZMiyWkqp4NO2rX9Nwzl2TM52f/pJ4p48WVYmCgZ65nseN98slcOXXCIFWU2bwp9/Oh2Va6pWlZU+DhxwOhKllDfNny8nCZdc4nQkrtmyRYpcf/pJ4l6yJHgSL2jyzVG9etK+rGlT2LFDFmWYP9/pqFyTkAA33SRVj0qp4LBggcyN9QfLl5+e3lm/vpzkXHON01F5lybfCyhfXlZFuuceiI+XlUHGjnU6qtwVKyZrd/rT5SelVP6dOCE1K9WqOR1J7r78UhZDOHwY2rWDZcv8I2530+Sbi/BwmDIFXnpJlu978kno08f3l/ILCYGuXf27cYhSKnfx8dCoke9f6bIWJk68hAcflCKrp56C77+H4sWdjswZmnxdEBIiixdMmCBVhCNGwCuvXEFCgtORXdjTT8uKTkqpwBUZKUvt+fKVrpQUePhhmDChBiEh8P778MEH0m8/WGnyzYMuXeDHH2Xu2S+/lKFFC1neyldddZWse/n3305HopTyhE2boH9/31484fBh6UHwxRdQpEg6330HzzzjdFTO0+SbRy1bSruzKlUSWbtWCrJ+/93pqHK2dSvs3et0FEopTyhTRmZn+Kp//pHXyGXLoEoVGDXqD9q3dzoq36DJNx+iouCDD37nhhsksbVoAd9953RU5/fQQ1LcoCseKRVYNm+W1dhatXI6kvP76SeZSrR1K1x5pcweuewyHx+r8yJNvvlUokQaCxbIpejERLjzTnj3XSkq8DUzZ8KzzzodhVLKndau9d2FYD79VBp+xMbCHXdI//xKlZyOyrcE8XB3wRUuLIsw1KoFAwdKgtu0CUaN8q32bh06yE0pFRhOnpTOUL4mI0P64g8ZIl/36wdvvy0te9WZ9My3gIyRaUjTpkkyHjdO5gP70mXe0FApenjgAf9bLEIpda777/e9s97EROmJMGSIvOaMGwfDhmnizYkmXze55x75ZyhbViqir7sOtm1zOqrTypWDbt3kzYJSyr9NnCi1Jr5i/36IjpYlAYsXh7lzoXt3p6PybZp83ejaa6WooG5d2LBBqvxWrHA6KmGMFGZMn+5fKzUppU5LS4MePeRzXzmjXL9eXut+/VX6CqxYAW3aOB2V79Pk62Y1asiiDG3awKFDsiD0tGlOR3Xan3/qogtK+bM2baBoUaejEPPmQfPmsvpb9pMPlTtNvh5QogTMmSPvUFNS4L77pEOWL1RC//e/UnUYH+90JEqpvDhxQt7Y/+c/vjF8NGaM1LfEx8va5z/9JMNbyjWafD0kLEwWYRg+XP5RXn5Zei2npDgdmSTgyZOdjkIplRdbt0ovZKelp0Pv3tKbOSNDZnp8+aX0wVeu06lGHmQM9O0LNWtKpfGkSbLQ/YwZcPHFzsX16qvB3VNVKX+TnAxXXCHVw05KSJBK69mz5QTj44+hc2dnY/JXeubrBXfcAUuXyuXen3+WsZFNm5yLp1AhaYn51FPOxaCUct0778iCLk7avVsqrGfPhtKlYeFCTbwFocnXS7LaqzVqJG3hmjWDJUuci6dOHZl6pJTyfQMGnK5ydsKaNbLY/dq10lRo5Uq44Qbn4gkEmny9qEoVOQO+/XY4elSqFidOdCaWiAioV08uG/lCIZhS6vyeeQZ27HBu5aLvvpNEu2+ffFyxQhKwKhhNvl5WrBh8+y306QOpqVKENXCgM52nQkPl8ndiovf3rZRyza23yht3b7NWCkbvvFNeI7KWVHWyXiWQaPJ1QGioLMIwZox8PniwFDF4u/lFaCgMHSr7PXnSu/tWSl1YWpo0xWnbFi66yLv7Tk2Fnj2lN7O1MlXys8+8H0cg0+TroJ49ZT5wZKT8k914ozMNMHr3huXLvb9fpVTODh2SsVVvO35c5u+OGyf96qdNk/71vjC3OJBo8nVY27Yycf6SS6Qgq2lT+Osv78YwaZIkfqWUb0hIgFKl5AqZN5Petm3Sl/7HH6VPfUyM9K1X7qfJ1wdcccXpxLtjh/zxz5/vvf2HhMgZeN++3tunUipn06fLfHxvWrFCXoM2bJAWkatWybRI5Rm5Jl9jTGFjzCfGmB3GmHhjzFpjzC3eCC6YlC8PixdDp04QFyeXfT780Hv7v/56Gd9RSjnv0UdlnNVbpk2Tq1+HDsksjF9+kT71ynNcOfMtBOwCWgIlgIHAdGNMdc+FFZzCw2HqVFmMOj1dxoT79pXPPa1ECRl7fvNNnXqklJOGDq3N2rXSQcrTrJWCz/vuk9a3PXrIVbASJTy/72CXa/K11p6w1r5mrd1urc2w1s4GtgFNPB9e8AkJkX+Gzz6Tf7733oO77pIxIE+LiJACi7Q0raxQyil3373bKysDpaScnupojEwrGjvWO0lf5WPM1xhTHogCvFwWFFy6doUFC6ToYtYsaeu2e7dn9xkaCs8+C/v2hevcX6W8LCVFEmD16ic8PqXnyBG4+WYptoyIkN4DfftqRbM35am9vjEmDPgCmGit/ec83+8OdAcoW7YsMTEx7ojRJyUkJHjl+N5/P5wBA+qzdm0EjRun8Oab66lVy7OnwVOm1CAhYQ116wbmuoPeeu6cEBsbS3p6esAeHwTu8xcXV4gtWypQu7Znj2/3bnlN2b07gjJlUhg8eD0lSiTgjV9poD53+WKtdemGnCVPBX4AwnLbPioqygayxYsXe21fhw5Z26KFtWBtRIS1333n2f1lHVtKimf34xRvPnfe1rJlS9uwYUOnw/CoQHz+9u+39sAB+dyTxxcTY23p0vJa0rChtbt2eWxX5xWIz112wG/WxZzq0mVnY4wBPgHKA3dba1M99F5AnUeZMjLv7uGHpc1bx44yFuzJwqiZM+HJJz33+Eqp0xYtgo8+8uw+Jk2SSuajR6F9e1i2zJm2lUq4etl5LHA50Npa6+UmiAqkEGriRIiKgpdflvGZTZtg1CjPrM17660yJqSU8qy0NFnv21MyMmTOcNbUpd69ZYnC0FDP7VPlzpV5vpcAPYBGwH5jTELm7UGPR6fOYIxUJk6ZIsn4ww9lPvDx4+7fV1bBx4MPer/ntFLBpG1bWL/eM4+dlCSJfdAgmUkxerRcNdPE67xcz5mstTsArYHzIffdJ+0o77hDKqKbN5cFrqtXd+9+IiLkUrc2U1fKc6ZOlaEldzt4UF4jVq483T++XTv370flj7aX9FPNmkn7t8svl17QTZt6pgl7u3bS33XrVvc/tlLBbNcueOopSbzunuKzYcPp14Rq1WThFE28vkWTrx+rUUPawLVpI+9yb7xR3t262/btzqy2pFQgK11a1sp1d+L98Ud5c759O1x9tbxJr1/fvftQBafJ18+VLCnt4Lp3h+RkuPde97eIfOwxeRe9bZv7HlOpYLZsGezcCa1bu/dxx4+HW26R/vD/+Y9ctapQwb37UO6hyTcAhIVJ8dXw4fIu+qWX4JFH4ORJ9+1j7VpdeEEpd9m5E/btc9/jpafL/2ePHvJ5//6yWEJEhPv2odzLA5NUlBOMkelHl14qFcoTJ8plp2++gYsvLvjjX3klfP21/GNrpaRS+bdtm3unFp04If/z330n0w7HjZNVkZRv0zPfANOxIyxdCpUqwZIlMvbz77/ue/xWrWR+sVIq706elMXpY2Pd83h798INN0jiLVlSZj9o4vUPmnwD0JVXSpFFw4aSeK+9VhJxQRkjBV1RUQV/LKWCTUaGnJmuXi2JsqDWroVrroHff4eaNWHFCim6VP5Bk2+AqlJFijrat5d2cm3aSHu5gipfHubNk8n6SinXTZ8uq4a5o7p59my4/nrYs0fm+a9cCXXqFPxxlfdo8g1gxYpJj+bevSE1Fbp0kQ5ZGRkFe9w6deQfXynluk6dpBiyIKyF99+X5hlZY72LFnmmSYfyLE2+AS40VNrJjR4tnw8eLMUeBWkZWb26NPf4+GPPLu6gVCCwVpppbNtWsCSZlgb/93/yZjojA15/HT7/XFrNKv+jyTdIPPmkXKqKjJQpCK1aSWOO/AoJgc2bZZUlpVTOjJFmGtWq5f8x4uLg9tvlTfRFF8EXX8Arr7i/QYfyHk2+QaRdO2kzV62ajBE1bSqtKfOjUCF4+21ISID9+90bp1KB4sgRmfbXunX+e6Tv2CHjuvPmyZnzTz95dhUk5R2afINM/fpSCX3NNTIP+LrrpB1dfk2cWLCfVyqQxcbKWWt+/fqrvEn+80+ptVi5UhKx8n+afINQhQqweLG0n4uLk3Z048bl77Gef15WPjpxwr0xKuXv1q+X+fb/93/5+/lvvoGWLaWveqtW0se9Zk33xqico8k3SEVEyNjvgAHSteqJJ2QaRHp63h/r4EGZS5yW5v44lfJXH30Ef/yR95+zFoYMkTfHSUnSW33ePChVyv0xKudoe8kgFhIiizDUqiULM7z7LmzZAj165O09WblycjmskP41KQXI5eaRI/P+cydPwjvv1OaHH+TrIUPguee0sCoQ6Zmv4pFHpC1dqVLSpq5Xr8bs2ZO3xyhaVKovZ8zwTIxK+YuNG6W5TV6n4R07JkWRP/xQkfBwuez8/POaeAOVJl8FSFu6FStkTOnffyNp2jTvl8y6dIG2bT0Tn1L+wFqoXVsaX+QlaW7ZIn3YFy+G0qVTWLIE7rrLc3Eq52nyVafUri2Xjxs0iGXPHmjRAr7/3vWfr1lTph49/7w231DB6cknYf78vDW+WLZMKpo3bpTZCGPG/M7VV3suRuUbNPmqM5QpA8OGreOhh6SC+Y47YMQI15Np6X1LvrcAAB2FSURBVNLQqJFnY1TKV738sqwy5KovvoCbbpL5wLfcIom4fPkUzwWofIYmX3WOiy6yTJoEb7whSbdPH2mP50o1c1iYNABYuFA6YCkVDHbvhp49oWJFCA/PfXtr4bXX4KGHpMjq6adh1iwoXtzjoSofoclXnZcx8i7+yy/lEtrYsVJEcvy4az+/ezccOuTZGJXyFRdfLC0kXRnnTU6WpPv66zLjYORIGDVKZwsEG02+6oLuv1/a2ZUtK2NZzZtLZ6zcPPKIjGP9+qvHQ1TKUV9+Cbt2wc03577toUPSavLLL2XVsVmz8t+EQ/k3x95rxcXFcfDgQVJTU50KoUBKlCjB33//7XQYHnH2sZUrF8bSpeW4887i/PWXJNVZs+TjhRw8CIMGyfSj0FAPB62UQ5KTZbglN//8A7fdBlu3ynrbs2dDw4aej0/5JkeSb1xcHAcOHKBy5cqEh4dj/HAiW3x8PJGRkU6H4RHZj81aS1JSEnv27GHhQujSpTgLF0J0NEyaJGuU5qRCBZk3nJAgY1wB+utSQSo5WRYqefTR3Lf96Se4+25pvtGkicwiqFjR8zEq3+XIZeeDBw9SuXJlIiIi/DLxBhNjDBEREVSuXJnExIP88AM8/ri88NxzD7z1Vu6V0EOGwPTp3olXKW/ZsUPOXnPzyScy/z02Fjp2hCVLNPEqh5Jvamoq4a6UBCqfER4eTmpqKmFhsgjDO+9IccmLL8o7/5Mnc/7Z116T/rQX2kYpf7Jpk7Rlfe+9nLfJyID+/aFbN5kp8Nxz0rWqaFHvxal8l2MFV3rG61+yP1/GyCIMM2bIAg0TJkixydGj5//Z0FC59Ny4sa5+pALDSy/JMn85SUyUK0NDhsjf//jxMHSoVDcrBVrtrAqgY0f4+We5hLZkibTH+/ff829brJg0EChaVLtfKf+VlibLcE6fDg0anH+b/fulJuKbb6BECVmR6PHHvRqm8gOafFWBNGkCq1dL1eamTbK04NKl59+2VCmZYvHGG96NUSl3+eEH6Ns35/m869efnmJXo4aswdu6tXdjVP5Bk68qsCpVJOHedptcer7pJvj88/Nv266drB2slL9JS4MOHWDMmPN/f+5cmQe/c6dcBVq5EurW9W6Myn9o8s2j6Ohonn76accfIzfHjh2jfPnybNmyJddtO3XqxPDhwwu0v8jIrOUIITUVOneWJQbPvsRcurSs//vII1ItqpQ/SEuDq6+Gw4fhoovO/f7o0dIBLj4e7rtPphaVK+f9OJX/0OQboN58801uvfVWatasmeu2r7zyCoMHD+a4q70jcxAaKoswfPCBFJb897/S5zk5+cztjJHkW6lSgXanlFdYK60f58+XhUeyS0+H3r2lN3NGhrRk/eILKFLEmViV/9DkG0BOZs7lSUxM5OOPP+axxx5z6efq16/PpZdeyuTJk90Sx1NPyfzHyEiYOhVatZJuV9ndcIOc+b75plt2qZTHDB4sFf1nn8nGx8uqX++/Lx2ushYj0Ypm5Qr9M8mHjIwMXn/9dcqUKUO5cuXo168fGRkZwPkvKXft2pX27dufcV9aWhq9evWiVKlSlCpViueee+7UY4B0lho6dCg1a9YkPDyc+vXrn5Mco6Oj6dmzJ/369aNs2bI0b94cgB9++AFjzKmvAYYOHYox5pzbK6+8AkCHDh2YMmWK235Ht9wi3X+qVYMVK6QIZcOGM7cpW1bmSirly554QsZ6s9u9W9a7njNHhlIWLoSHH3YmPuWffCL5GuPMLb+++OILQkND+eWXX/jggw8YMWIE06ZNy/NjZGRksGLFCsaNG8f48eMZMWLEqe8PHDiQTz75hNGjR7NhwwYGDBhAjx49mDNnzhmPM3nyZKy1LF26lEmTJgGwdOlSmjRpcsbc3J49e7Jv375Tt2effZYKFSrQuXNnAK655hpWr15NUlJSfn8t56hfH1atkrGy7dulCOXHH09/v0QJaU85YwasXeu23SrlFlu3woMPyopFpUufvn/NGrjmGli3DqKipLAqL2v4KgUOLqzgz+rWrcvAgQOJjIwkKiqKjz76iEWLFnH//fe7/BgVK1Zk5MiRGGOoU6cOmzZt4t1336Vv376cOHGCd999lwULFtCiRQsAatSowerVqxk9ejS33XbbqcepUaPGOcVSO3bsoNJZA6qRkZGn+jUPGTKEKVOmEBMTw2WXXQZApUqVSE1NZe/evZRzY6VIhQoQEyMFWN98I2fEY8ZA9+6ntwkN1bm/yvdUqyYFhNnfqM+cKQk5MfH0XN7siVkpV/nEma+1ztzyq8FZs+srVarEwbMHNXNx7bXXnnFm2qxZM/bs2UNcXBwbNmwgOTmZdu3aUaxYsVO3sWPHnlO93KRJk3MeOykpiSI5VHy89dZbjBo1isWLF1O7du1T92e1+3TnmW+WiAhpStC/vxSo9OgB/frJ5yDjZg0awEcfSVWpUk6yVuby7tghZ7hZ9w0fDnfdJYm3SxcpwNLEq/LLpTNfY8zTQFegPjDFWtvVgzH5vLCz1g8zxpwarw0JCcGeldnzumxi1mN9//33VKtW7YL7LnqeRrFlypTh2LFj59w/aNAgPvzwwzPOeLMczewNWbZs2TzF6qqQEFmEoVYtSb7Dh8OWLTB58umuV9u3S/vJEiU8EoJSLjEG2rSBypXl69RUqWYeP16+fvNNeSOpHXJVQbh65rsXGAR86sFYAkLZsmXZt2/fGfetW7funO1WrVp1RpJeuXIllSpVonjx4tStW5fChQuzY8cOLrvssjNul1xySa4xNG7cmA1nVTe98cYbjB8/niVLlpyTeAH+/PNPKleuTPny5V091Hx59FFYsABKlpRLeDfcAHv3ylSOwYPlzHfRIo+GoFSOZs6UGoRbbpHpQrGxcOutkniLFJErOAMGaOJVBedS8rXWzrDWzgSOeDgev9eqVSvmzp3LrFmz2LhxI3379mXXrl3nbLd371569+7Nxo0b+frrrxk2bBh9+vQBZHy2X79+9OvXj08//ZTNmzezdu1aPvzwQ8Znvf2+gLZt2/L3339z5Ig8XYMGDWLkyJFMnTqVokWLsn//fvbv309ytgm4S5cupW3btm76LVzYjTdKkUrNmvD773JpL6vgas8e6QGtlBMuvRSqV5fPt22TjlULF8o0o5iYC69frVReuLXgyhjTHegOcgYYExNz3u1KlChBfHy8O3ftNenp6Zw8eZL09PRTx5CamkpaWhrx8fF06tSJ3377jUceeQSAxx9/nPbt23PkyJFT26enp3PPPfeQlJRE06ZNMcbw8MMP061bt1PbPP/885QoUYKhQ4fSs2dPIiMjadCgAb169TrjcU6ePHnO77J69eo0adKECRMm8PjjjzNs2DDi4uLOmHoEMGvWLKKjo0lOTubbb79lxowZxMfHn3Fs2SUnJ+f4nObH8OFhvPxyPdavL0mzZum88soGmjU7QsuW8PHHxShePJVy5VLctr8sCQkJbj0OXxIbG0t6enrAHh945vmLjQ3ju+8q0bnzDoyB0aOLM3DgFcTGXkT16id46631JCUl441fayD/fQbyseWZtdblG3LpeYIr20ZFRdmcbNiwIcfv+Yu4uDinQ7iguXPn2qioKJuWlpbrth988IFt06bNqa9zOjZPPG/JydY+9JCUwIWEWDtihLUZGdaOHGnt3Llu35211trFixd75oF9QMuWLW3Dhg2dDsOjPPH8HT1q7cSJ8vnUqdYWLix/kzffbG1srNt3d0GB/PcZyMdmrbXAb9bFfOoT1c7K/dq1a8dTTz3F7t27c902LCyMUaNGeSGqcxUuLJ2BXn9d2vNlterr2VMWYVi06HRVtFLuZq0UAlorTTIGDZLezCkp0lxjzhwtAFSeofN8A9gzzzzj0nbds0+6dYAxsghDrVrQtavMA966FaZMkWroOnVOV54q5U7WShtUY+Rvb9Ik+fzdd8+d46uUO7k61ahQ5rahQKgxpgiQZq3VWZnKbe6/XxobdOwoC5C3aCE9osuXl2KX6GinI1SB5Isv4PLL5e+uY0f4+WeZkz5lyrntJJVyN1cvOw8EkoD+wEOZnw/0VFAqeDVvLi0p69SBP/+UntBz50pje+2CpdypWDFZ8OPaayXxVqok61Jr4lXe4OpUo9esteas22sejk0FqUsvlcUYbroJDhyAe+6RtVIPHYJff3U6OuXv/vc/ubxcsqS0ity8GRo3htWr4cornY5OBQstuFI+qWRJOePt1k3WA+7UCV56CRYvdjoy5e+KFJHFEdq0gaNH4fbb5cxX6wqUN2nyVT4rLEw6Cw0dKoUvH38M//wjTThcKOJW6gz798OLL8LEiTBypLSN7NMHvv1WLkEr5U2afJVPMwaee05WjwkPh88+g0cegd9+czoy5W8KFZIrJ2++KStpjRkjVc2hoU5HpoKRJl/lF+68Uy4NVqggY3QvvCAvnMePOx2Z8nUpKbKkZbt20tY0MlLm7/bs6XRkKphp8lV+46qrpCimQQPYtAkGDoQff3Q6KuXr/v1Xpq6tWQOXXAK//AJeamOuVI40+Sq/UrWqjPneeiskJcEDD0g1tAeWIVYB4JZbZCrRoUMybW3VKrjiCqejUkqTb57deeedlCpViocfftjpUIJWZCR89x0884wUzXz1lbQF1HnAKou1MG6cLF954oRUyy9eLA1blPIFmnzzqFevXkyaNCnPP7dr1y6io6OpW7cuDRo04KuvvvJAdMGjUCF4/30YNQpCQqSIpk4deaFVwS09XebrPvGE9At/8UWYOlUK9pTyFZp88yg6OprIyMg8/1yhQoUYMWIEGzZsYMGCBfTu3ZsTmikK7Omn4fvvoWhRGQe++Wa5xKiC0/HjcNddsj50oUJSHT94sLxBU8qX6J+kl1SsWJFGjRoBUKFCBcqUKcPRo0cdjiow3HqrFNFUrSofa9SAv/92OirlbXv2SK/mWbOgVCkpxuva1emolDo/Tb4OWLNmDenp6VStWtXpUAJGgwZSTHPVVXLpuVkzmD/f6aiUt6xYIc/9vn1Qs6ZMKdKFOJQv0+TrZUePHqVz586MHz/e6VACTsWKsGQJ3H23XH685Rb46COno1Ke9v33kmj375eVsFauhKgop6NS6sI0+brR0KFDMcacc3vllVcASElJoWPHjvTv35/rrrvO4WgDU0QETJ8Ozz8vFa/du8OTT0rhjQos1kqVe4cOcPIkPPSQXGouU8bpyJTKnSbfPGrdujWdOnViwYIFVKlShRUrVpz6Xs+ePdm3b9+p27PPPkuFChXo3Lkz1lq6du1Kq1atdJqSh4WEwJAh0gs6JATGjpWzYa1vCxxpaVJs9/LL8vV//ysrFRUu7GxcSrmqkNMB+JuFCxcCEB8ff07Vc2Rk5Kn7hgwZwpQpU4iJieGyyy5j2bJlTJs2jQYNGjBz5kwAPv/8c+rXr+/dAwgijz0mxVd33QUzZ8qY4KJFTkelCurEiVCio2H5ckm2n30G99/vdFRK5Y0mXw946623GD16NIsXLyYqc/Dp+uuvJ0OvfXpdq1YyBtiqlayI1LQpvPpqUS3G8VM7dsDTTzdm+3YoXlyWndQRHOWPfOay82uvyQ2kWGLTJunF2qSJ3PfsszB8uHxeqRLs3QsxMacrGrt3l+XnQDogxcdLIcbtt8t9DzwAX34pnxuTvxizj+MWL178nLFdgEGDBjF69GhiYmJOJV7lrDp1ZAH15s1lKcInn7ySOXOcjkrl1erV0KgRbN9ejMsvhz/+0MSr/Ji11iO3qKgom5MNGzbk+D1ftnPnTtuyZUt7+eWX23r16tnp06ef8f3XX3/dVq1a1W7evNmhCN0jLi7uvPf76/OWJSnJ2sdu3mmrsNMaY+277zodkfu1bNnSNmzY0Okw3G7CBGsLF7a2CjvtTVF/2mPHnI7IcxYvXux0CB4TyMdmrbXAb9bFHOkzZ77+IHuXqpkzZ57RpWrQoEGMHDmSqVOnUrRoUfbv38/+/ftJTk52OGqVpUgR+GheVVp3TcNa6NtXinbS0pyOTOUkLU2ep65dZWnAW7tXpf/ow5Qs6XRkShWMJt88yN6lqnz58qe6VFlrGTZsGEeOHKF58+ZUrFjx1G358uUOR62yM9On0a/qZ0yaBGFhMHq0rHqzb5/TkamzHT4M118P770nrSJHj4ZxraZR8WetmlP+Twuu8umPP/44o0vVcV3V3T+MHUvl2FjqrX2DmjVljuiaNdC4MUybBi1bOh2gAvj9d5kellVYNWeOJGKi5fnjjTecDlGpAtEz33w4evQoPXr00C5Vfu666+Cvv+DGG+HAAfk4dKguTegka2W1qquvlsR79dXyHF1/vdORKeVemnzzKKtLVZ8+fbRLVQAoX17WfO3TR174X3hB2lIePOh0ZMHnyBFo3Rp695aOZI89Bj//DFWqOB2ZUu6nyTcPbLYuVffrrP6AUagQvPsufPedXOKcPx+uuAKdjuRFS5fKNKKffpLlIb/5RjqUFSnidGRKeYYm3zxYvnw506ZNY+bMmTRv3pxGjRqxfv16p8NSbtKhA6xfL835Dx2C9u3l7Csx0enIAldSkszhv+EGmYPdrBn8+ad0JVMqkGnBVR5k71J1vvaSyg98/TV/LV9O8xy+Xa2aNG8ZNgxefBE+/VQufX72mY47utsvv8CDD8rYrjGShN98U6rQc5TL86eUv9AzXxVcypQhtUSJC24SEiJjv7//DvXqwebNcjb8xBMQG+ulOANYYiL06ycdx7Zvh9q1ZS3mYcNySbzg0vOnlD/Q5KuCy4QJVJg3z6VNGzaE336DAQNkXHjcOLj0UhmP1IrovLNWxtVr1ZJWscbIm5y1a6Wq2SV5eP6U8mWafFVwyeOLd5Eicil07Vpo0ACOHYP//Ed6iv/9t+fCDDSbN8sc6o4dpS/7ZZfJghdvv53HoipNvipAaPJVygX16kkj/7FjpRr355+hfn1pT3n4sNPR+a7jx+H556FuXaloDg+HkSPljcs11zgdnVLO0eSrlItCQmTcd+tW6NFD5qKOHg01a8pl1KQkpyP0HUlJ8ju59FIZy01NhS5dYNs2+L//k8v4SgUzx5Kv1UEzv6LP12nlysGHH8K6dVKIFRcnBURVqsCIEcGdhFNT4ZNP5A1Jv35w9KgUVv3yC0yYIE1NlFIOJd+wsDCSgvkVyg8lJSURlmspanCpXx+WLJFmHLVqSaLp0wcqVJCmHZkLXgWFhAR541G5MnTrJgtVREXJYvdLl8r8XaXUaY4k33LlyrFnzx4SExP1jMrHWWtJTExkz549lCtXzulwCu6HH/jf22+77eGMgVtvhY0bYdYsGduMi5M5qxUqQK9eMp0mUB04AAMHyhltnz7SnKRqVZgyRcZ127WT35HbuPn5U8opjoy8FC9eHIC9e/eSmprqRAgFlpycTJEA7X139rGFhYVRvnz5U8+bX4uIIMMDz5sxcPvt0hVrzhwYPFiqeUeOhFGjoFUr6N9fPob4eaVFRgYsXCjHNm8epKfL/VddBS+/LL8Djx2jh54/pbzNsbKH4sWL+/WLeUxMDI0bN3Y6DI8I5GNjzBgqbdokc4U8wBhJPu3bw+rVkninTIFFi+RWtix07iyLw19xhUdC8JjNm2HqVEm6hw7JfcbAHXfAc8/J2K7Hefj5U8pbtOZQBZfp0ynnpTZV11wDn38u1b7jx0sh0s6dUgU8fLjMde3SRS5bN27s5suzbrJhA3z9tbyB+Oef0/dXqgQ9e8Ijj8g4r9d48flTypNcujhkjCltjPnWGHPCGLPDGPOApwNTKlBUqACvvCLTbJYulWlKxYrJmeTLL0OTJlCxoizi8NVX0oTCKfv3S6J97DF5c1CvHrz6qiTeIkWkF/P8+bBrl4z1ejXxKhVAXD3zHQ2cBMoDjYA5xph11tq/PBaZUgEmJEQWZ7j+elkwfv58mD1bziwPHJBFHD79VLYtV04u47ZsKZ216tSRJO7Os+MjR2S61B9/SAevxYthz54zt4mMlLHsBx+Em26CwoXdt3+lglmuydcYUxS4G7jCWpsALDPGzAIeBvp7OD6lAlLhwrKEYYcO0jP6f/+TaumFCyURHjwI334rtyxFi0KNGjJWXLKkzCsuX17GkcPDpXHF8eNw4kQhli2D+PjTt+PHpTnI7t2wYwds2SL3ny08XN4cREdDmzZyOVwbYijlfia3qT7GmMbAcmttRLb7+gEtrbW35/RzERER9poA7h8XGxtLyZIlnQ7DIwL52Fi7lrS0NApddZXTkeTIWln5Jy5OkmZCAqSkQFqaKz+9NvNjo1y3DAmBiAhJ6sWLy6XwyEjfHHs+xQ+ev4IK5P+/QD42gCVLlqyx1rr0x+lK8m0BfGWtrZDtvseBB6210Wdt2x3onvnlFcCfeYjb35QBArWrbyAfG+jx+Ts9Pv8VyMcGUNta69JC765cUEoAzp4TVBw456KVtXY8MB7AGPObq+8A/FEgH18gHxvo8fk7PT7/FcjHBnJ8rm7rSrXzJqCQMaZWtvsaAlpspZRSSuVDrsnXWnsCmAG8YYwpaoxpDtwBfO7p4JRSSqlA5GoTuCeBcOAgMAXo6cI0o/EFCcwPBPLxBfKxgR6fv9Pj81+BfGyQh+PLteBKKaWUUu7l5y3elVJKKf+jyVcppZTyMq8kX2NMLWNMsjFmsjf25y3GmMnGmH3GmDhjzCZjTDenY3IXY0xhY8wnmb28440xa40xtzgdlzsZY542xvxmjEkxxkxwOp6CCuQe7IH2XJ0t0P/fAvm1Mru85DpvnfmOBn710r686S2gurW2ONABGGSMaeJwTO5SCNgFtARKAAOB6caY6g7G5G57gUHAp04H4ibZe7A/CIw1xtRzNiS3CbTn6myB/v8WyK+V2bmc6zyefI0x9wGxwCJP78vbrLV/WWtTsr7MvNV0MCS3sdaesNa+Zq3dbq3NsNbOBrYBAfMPY62dYa2dCRxxOpaCytaD/WVrbYK1dhmQ1YPd7wXSc3U+gf7/FsivlVnymus8mnyNMcWBN4C+ntyPk4wxY4wxicA/wD7gB4dD8ghjTHkgCm2u4quigDRr7aZs960DAuXMN6gE4v9bIL9W5ifXefrM97/AJ9ba3R7ej2OstU8CkUALpBlJyoV/wv8YY8KAL4CJ1tp/ctteOaIYEHfWfceRv03lRwL1/y3AXyvznOvynXyNMTHGGJvDbZkxphHQGngvv/twUm7Hl31ba2165mW+KkBPZyLOG1ePzxgTgnQzOwk87VjAeZSX5y9AuNyDXfkuf/1/c5U/vlbmJr+5Lt8rdZ69otF5AuoNVAd2GlmjrBgQaoypa629Mr/79Zbcji8HhfCTcQxXjs/IE/cJUsBzq7U21dNxuUs+nz9/dqoHu7X238z7tAe7H/Hn/7d88JvXShdEk49c58nLzuORX26jzNuHwBygrQf36TXGmHLGmPuMMcWMMaHGmLbA/QRWYdlY4HLgdmttktPBuJsxppAxpggQivyzFDHG+OXS8YHegz2QnqsLCMj/tyB4rcxXrvNY8rXWJlpr92fdkMtiydbaQ57ap5dZ5LLJbuAY8A7Q21o7y9Go3MQYcwnQA/lj2m+MSci8PehwaO40EEgC+gMPZX4+0NGICiY/Pdj9RaA9V2cI8P+3gH6tzG+u097OSimllJdpe0mllFLKyzT5KqWUUl6myVcppZTyMk2+SimllJdp8lVKKaW8TJOvUkop5WWafJVSSikv0+SrlFJKeZkmX6WUUsrLNPkqFQCMMc/nsILTG07HppQ6l7aXVCoAGGMigaLZ7uoHPAi0sNZudiYqpVRONPkqFWCMMS8AzwCtrLUbnY5HKXWuQFuSS6mgZowZADwF3Git3eR0PEqp89Pkq1SAMMYMBJ4AovVSs1K+TZOvUgHAGPMK0A1oaa3d4nQ8SqkL0+SrlJ/LPON9BugAnDDGVMj8Vqy1Ntm5yJRSOdGCK6X8mDHGALFA8fN8u7W1dpGXQ1JKuUCTr1JKKeVl2mRDKaWU8jJNvkoppZSXafJVSimlvEyTr1JKKeVlmnyVUkopL9Pkq5RSSnmZJl+llFLKyzT5KqWUUl6myVcppZTysv8HMk22K5kLKGQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x252 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 3.5))\n",
    "z = np.linspace(-4, 4, 200)\n",
    "plt.plot(z, huber_fn(0, z), \"b-\", linewidth=2, label=\"huber($z$)\")\n",
    "plt.plot(z, z**2 / 2, \"b:\", linewidth=1, label=r\"$\\frac{1}{2}z^2$\")\n",
    "plt.plot([-1, -1], [0, huber_fn(0., -1.)], \"r--\")\n",
    "plt.plot([1, 1], [0, huber_fn(0., 1.)], \"r--\")\n",
    "plt.gca().axhline(y=0, color='k')\n",
    "plt.gca().axvline(x=0, color='k')\n",
    "plt.axis([-4, 4, 0, 4])\n",
    "plt.grid(True)\n",
    "plt.xlabel(\"$z$\")\n",
    "plt.legend(fontsize=14)\n",
    "plt.title(\"Huber loss\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:38.725999Z",
     "iopub.status.busy": "2021-07-04T08:14:38.713775Z",
     "iopub.status.idle": "2021-07-04T08:14:38.743016Z",
     "shell.execute_reply": "2021-07-04T08:14:38.741652Z"
    },
    "id": "jaC8PoiYnFSb"
   },
   "outputs": [],
   "source": [
    "input_shape = X_train.shape[1:]\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:38.762163Z",
     "iopub.status.busy": "2021-07-04T08:14:38.760591Z",
     "iopub.status.idle": "2021-07-04T08:14:38.764640Z",
     "shell.execute_reply": "2021-07-04T08:14:38.765729Z"
    },
    "id": "SiLyH6ncnFSc"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=huber_fn, optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:38.772068Z",
     "iopub.status.busy": "2021-07-04T08:14:38.770797Z",
     "iopub.status.idle": "2021-07-04T08:14:43.521361Z",
     "shell.execute_reply": "2021-07-04T08:14:43.522572Z"
    },
    "id": "tMH11jIEnFSc",
    "outputId": "b1b9aa46-2efe-4bd4-eabf-a23b88cd869d",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 3s 4ms/step - loss: 0.6235 - mae: 0.9953 - val_loss: 0.2862 - val_mae: 0.5866\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.2197 - mae: 0.5177 - val_loss: 0.2382 - val_mae: 0.5281\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47b0dab50>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GiDKxlo0nFSc"
   },
   "source": [
    "## 사용자 정의 요소를 가진 모델을 저장하고 로드하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:43.554664Z",
     "iopub.status.busy": "2021-07-04T08:14:43.553745Z",
     "iopub.status.idle": "2021-07-04T08:14:43.583902Z",
     "shell.execute_reply": "2021-07-04T08:14:43.585209Z"
    },
    "id": "4vsC0bFSnFSc"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:43.593971Z",
     "iopub.status.busy": "2021-07-04T08:14:43.592418Z",
     "iopub.status.idle": "2021-07-04T08:14:43.677265Z",
     "shell.execute_reply": "2021-07-04T08:14:43.678558Z"
    },
    "id": "DU0qVKUQnFSc"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss.h5\",\n",
    "                                custom_objects={\"huber_fn\": huber_fn})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:43.686580Z",
     "iopub.status.busy": "2021-07-04T08:14:43.685128Z",
     "iopub.status.idle": "2021-07-04T08:14:47.493380Z",
     "shell.execute_reply": "2021-07-04T08:14:47.494505Z"
    },
    "id": "3D4RbreznFSc",
    "outputId": "790f5d75-8dde-43b5-b9dc-9f6571c53a4c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.2054 - mae: 0.4982 - val_loss: 0.2209 - val_mae: 0.5050\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.1999 - mae: 0.4900 - val_loss: 0.2127 - val_mae: 0.4986\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd476f36ed0>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:47.505642Z",
     "iopub.status.busy": "2021-07-04T08:14:47.504658Z",
     "iopub.status.idle": "2021-07-04T08:14:47.508213Z",
     "shell.execute_reply": "2021-07-04T08:14:47.508984Z"
    },
    "id": "XU-KvpzJnFSc"
   },
   "outputs": [],
   "source": [
    "def create_huber(threshold=1.0):\n",
    "    def huber_fn(y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < threshold\n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss  = threshold * tf.abs(error) - threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    return huber_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:47.520740Z",
     "iopub.status.busy": "2021-07-04T08:14:47.516256Z",
     "iopub.status.idle": "2021-07-04T08:14:47.531481Z",
     "shell.execute_reply": "2021-07-04T08:14:47.532301Z"
    },
    "id": "pagOIVIvnFSd"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:47.539616Z",
     "iopub.status.busy": "2021-07-04T08:14:47.538294Z",
     "iopub.status.idle": "2021-07-04T08:14:51.879755Z",
     "shell.execute_reply": "2021-07-04T08:14:51.880935Z"
    },
    "id": "VpXt3HOpnFSd",
    "outputId": "2c90da98-3ac4-4afc-aab9-3e0b375ff03e",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.2226 - mae: 0.4892 - val_loss: 0.2540 - val_mae: 0.4907\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.2184 - mae: 0.4844 - val_loss: 0.2372 - val_mae: 0.4879\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47afc66d0>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:51.914825Z",
     "iopub.status.busy": "2021-07-04T08:14:51.913722Z",
     "iopub.status.idle": "2021-07-04T08:14:51.946973Z",
     "shell.execute_reply": "2021-07-04T08:14:51.947805Z"
    },
    "id": "or-QRU9DnFSd"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss_threshold_2.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:51.956901Z",
     "iopub.status.busy": "2021-07-04T08:14:51.955368Z",
     "iopub.status.idle": "2021-07-04T08:14:52.037871Z",
     "shell.execute_reply": "2021-07-04T08:14:52.039072Z"
    },
    "id": "VRAUNTJxnFSd"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss_threshold_2.h5\",\n",
    "                                custom_objects={\"huber_fn\": create_huber(2.0)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:52.047973Z",
     "iopub.status.busy": "2021-07-04T08:14:52.046462Z",
     "iopub.status.idle": "2021-07-04T08:14:55.927195Z",
     "shell.execute_reply": "2021-07-04T08:14:55.928267Z"
    },
    "id": "ZQIXRq1snFSd",
    "outputId": "768a1bf8-afc2-4a26-d918-9cb0b6baef1e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.2147 - mae: 0.4800 - val_loss: 0.2133 - val_mae: 0.4654\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2119 - mae: 0.4762 - val_loss: 0.1992 - val_mae: 0.4643\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47adb2b10>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:55.944343Z",
     "iopub.status.busy": "2021-07-04T08:14:55.943056Z",
     "iopub.status.idle": "2021-07-04T08:14:55.946321Z",
     "shell.execute_reply": "2021-07-04T08:14:55.947381Z"
    },
    "id": "3u4WR3alnFSd"
   },
   "outputs": [],
   "source": [
    "class HuberLoss(keras.losses.Loss):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        self.threshold = threshold\n",
    "        super().__init__(**kwargs)\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < self.threshold\n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss  = self.threshold * tf.abs(error) - self.threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:55.961261Z",
     "iopub.status.busy": "2021-07-04T08:14:55.960228Z",
     "iopub.status.idle": "2021-07-04T08:14:55.981249Z",
     "shell.execute_reply": "2021-07-04T08:14:55.982560Z"
    },
    "id": "y1JdYpsKnFSd"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:56.000735Z",
     "iopub.status.busy": "2021-07-04T08:14:55.999638Z",
     "iopub.status.idle": "2021-07-04T08:14:56.007356Z",
     "shell.execute_reply": "2021-07-04T08:14:56.008433Z"
    },
    "id": "MSvlez6KnFSd"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=HuberLoss(2.), optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:14:56.016116Z",
     "iopub.status.busy": "2021-07-04T08:14:56.014749Z",
     "iopub.status.idle": "2021-07-04T08:15:00.300730Z",
     "shell.execute_reply": "2021-07-04T08:15:00.301680Z"
    },
    "id": "AgKhF3hKnFSd",
    "outputId": "84fb0610-d355-4fc0-a289-2de847fc33be",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.7095 - mae: 0.8863 - val_loss: 0.3378 - val_mae: 0.5485\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2416 - mae: 0.5083 - val_loss: 0.2660 - val_mae: 0.5089\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47ac65a10>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:00.335267Z",
     "iopub.status.busy": "2021-07-04T08:15:00.333773Z",
     "iopub.status.idle": "2021-07-04T08:15:00.369221Z",
     "shell.execute_reply": "2021-07-04T08:15:00.370473Z"
    },
    "id": "uyTT256tnFSe"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss_class.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:00.379781Z",
     "iopub.status.busy": "2021-07-04T08:15:00.378195Z",
     "iopub.status.idle": "2021-07-04T08:15:00.460926Z",
     "shell.execute_reply": "2021-07-04T08:15:00.462302Z"
    },
    "id": "Ot6cK1oYnFSe"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss_class.h5\",\n",
    "                                custom_objects={\"HuberLoss\": HuberLoss})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:00.471143Z",
     "iopub.status.busy": "2021-07-04T08:15:00.469597Z",
     "iopub.status.idle": "2021-07-04T08:15:04.566644Z",
     "shell.execute_reply": "2021-07-04T08:15:04.567732Z"
    },
    "id": "M9OIPCgNnFSe",
    "outputId": "922f5b7f-623c-4dca-9680-fd293812be3b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2286 - mae: 0.4970 - val_loss: 0.2120 - val_mae: 0.4723\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.2216 - mae: 0.4904 - val_loss: 0.2045 - val_mae: 0.4725\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47aa55150>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.576492Z",
     "iopub.status.busy": "2021-07-04T08:15:04.575437Z",
     "iopub.status.idle": "2021-07-04T08:15:04.580464Z",
     "shell.execute_reply": "2021-07-04T08:15:04.581297Z"
    },
    "id": "jpZgLrW-nFSe",
    "outputId": "1b906c14-5f76-4c05-edbc-3dc56130f2e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.loss.threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "truxJ-5enFSe"
   },
   "source": [
    "## 그외 사용자 정의 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.589988Z",
     "iopub.status.busy": "2021-07-04T08:15:04.588495Z",
     "iopub.status.idle": "2021-07-04T08:15:04.629460Z",
     "shell.execute_reply": "2021-07-04T08:15:04.627925Z"
    },
    "id": "o1ZGp6XNnFSe"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.651373Z",
     "iopub.status.busy": "2021-07-04T08:15:04.649654Z",
     "iopub.status.idle": "2021-07-04T08:15:04.653191Z",
     "shell.execute_reply": "2021-07-04T08:15:04.654590Z"
    },
    "id": "yB3e_4JbnFSe"
   },
   "outputs": [],
   "source": [
    "def my_softplus(z): # tf.nn.softplus(z) 값을 반환합니다\n",
    "    return tf.math.log(tf.exp(z) + 1.0)\n",
    "\n",
    "def my_glorot_initializer(shape, dtype=tf.float32):\n",
    "    stddev = tf.sqrt(2. / (shape[0] + shape[1]))\n",
    "    return tf.random.normal(shape, stddev=stddev, dtype=dtype)\n",
    "\n",
    "def my_l1_regularizer(weights):\n",
    "    return tf.reduce_sum(tf.abs(0.01 * weights))\n",
    "\n",
    "def my_positive_weights(weights): # tf.nn.relu(weights) 값을 반환합니다\n",
    "    return tf.where(weights < 0., tf.zeros_like(weights), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.665873Z",
     "iopub.status.busy": "2021-07-04T08:15:04.664915Z",
     "iopub.status.idle": "2021-07-04T08:15:04.668497Z",
     "shell.execute_reply": "2021-07-04T08:15:04.669224Z"
    },
    "id": "KoYamcfmnFSf"
   },
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(1, activation=my_softplus,\n",
    "                           kernel_initializer=my_glorot_initializer,\n",
    "                           kernel_regularizer=my_l1_regularizer,\n",
    "                           kernel_constraint=my_positive_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.676960Z",
     "iopub.status.busy": "2021-07-04T08:15:04.675394Z",
     "iopub.status.idle": "2021-07-04T08:15:04.681627Z",
     "shell.execute_reply": "2021-07-04T08:15:04.682609Z"
    },
    "id": "OjTWCzNknFSf"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.698955Z",
     "iopub.status.busy": "2021-07-04T08:15:04.697532Z",
     "iopub.status.idle": "2021-07-04T08:15:04.725625Z",
     "shell.execute_reply": "2021-07-04T08:15:04.726757Z"
    },
    "id": "eOl39GkynFSf"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=my_l1_regularizer,\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.744912Z",
     "iopub.status.busy": "2021-07-04T08:15:04.743523Z",
     "iopub.status.idle": "2021-07-04T08:15:04.751176Z",
     "shell.execute_reply": "2021-07-04T08:15:04.752176Z"
    },
    "id": "Pj0gQWgnnFSf"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:04.761800Z",
     "iopub.status.busy": "2021-07-04T08:15:04.760020Z",
     "iopub.status.idle": "2021-07-04T08:15:09.144757Z",
     "shell.execute_reply": "2021-07-04T08:15:09.145795Z"
    },
    "id": "c1nX7r3RnFSf",
    "outputId": "06dffd5a-07a7-4d09-ea34-acdd8a34a1e8",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 1.5542 - mae: 0.8962 - val_loss: 1.4154 - val_mae: 0.5607\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.5943 - mae: 0.5256 - val_loss: 1.4399 - val_mae: 0.5137\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47a9865d0>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.167536Z",
     "iopub.status.busy": "2021-07-04T08:15:09.166470Z",
     "iopub.status.idle": "2021-07-04T08:15:09.199791Z",
     "shell.execute_reply": "2021-07-04T08:15:09.201240Z"
    },
    "id": "--fx4XjEnFSf"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.211908Z",
     "iopub.status.busy": "2021-07-04T08:15:09.210859Z",
     "iopub.status.idle": "2021-07-04T08:15:09.299636Z",
     "shell.execute_reply": "2021-07-04T08:15:09.298692Z"
    },
    "id": "mzao-J6hnFSf"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\n",
    "    \"my_model_with_many_custom_parts.h5\",\n",
    "    custom_objects={\n",
    "       \"my_l1_regularizer\": my_l1_regularizer,\n",
    "       \"my_positive_weights\": my_positive_weights,\n",
    "       \"my_glorot_initializer\": my_glorot_initializer,\n",
    "       \"my_softplus\": my_softplus,\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.311370Z",
     "iopub.status.busy": "2021-07-04T08:15:09.309901Z",
     "iopub.status.idle": "2021-07-04T08:15:09.313019Z",
     "shell.execute_reply": "2021-07-04T08:15:09.314231Z"
    },
    "id": "jCVaCcxwnFSf"
   },
   "outputs": [],
   "source": [
    "class MyL1Regularizer(keras.regularizers.Regularizer):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "    def __call__(self, weights):\n",
    "        return tf.reduce_sum(tf.abs(self.factor * weights))\n",
    "    def get_config(self):\n",
    "        return {\"factor\": self.factor}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.323296Z",
     "iopub.status.busy": "2021-07-04T08:15:09.322226Z",
     "iopub.status.idle": "2021-07-04T08:15:09.331068Z",
     "shell.execute_reply": "2021-07-04T08:15:09.331982Z"
    },
    "id": "8B62Te2ZnFSf"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.336986Z",
     "iopub.status.busy": "2021-07-04T08:15:09.335822Z",
     "iopub.status.idle": "2021-07-04T08:15:09.371820Z",
     "shell.execute_reply": "2021-07-04T08:15:09.372684Z"
    },
    "id": "D5zjm10KnFSg"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=MyL1Regularizer(0.01),\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.376664Z",
     "iopub.status.busy": "2021-07-04T08:15:09.375792Z",
     "iopub.status.idle": "2021-07-04T08:15:09.394320Z",
     "shell.execute_reply": "2021-07-04T08:15:09.393397Z"
    },
    "id": "YffhdIG-nFSg"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:09.400219Z",
     "iopub.status.busy": "2021-07-04T08:15:09.397613Z",
     "iopub.status.idle": "2021-07-04T08:15:13.847502Z",
     "shell.execute_reply": "2021-07-04T08:15:13.848611Z"
    },
    "id": "KgQEiQR6nFSg",
    "outputId": "4b4043b0-0f69-4636-8544-5cc37ae97ef5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 1.5542 - mae: 0.8962 - val_loss: 1.4154 - val_mae: 0.5607\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.5943 - mae: 0.5256 - val_loss: 1.4399 - val_mae: 0.5137\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd479f76250>"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:13.868331Z",
     "iopub.status.busy": "2021-07-04T08:15:13.866654Z",
     "iopub.status.idle": "2021-07-04T08:15:13.896032Z",
     "shell.execute_reply": "2021-07-04T08:15:13.897428Z"
    },
    "id": "RfI--9P8nFSg"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:13.907144Z",
     "iopub.status.busy": "2021-07-04T08:15:13.905960Z",
     "iopub.status.idle": "2021-07-04T08:15:13.987931Z",
     "shell.execute_reply": "2021-07-04T08:15:13.988742Z"
    },
    "id": "bKcPzkuGnFSh"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\n",
    "    \"my_model_with_many_custom_parts.h5\",\n",
    "    custom_objects={\n",
    "       \"MyL1Regularizer\": MyL1Regularizer,\n",
    "       \"my_positive_weights\": my_positive_weights,\n",
    "       \"my_glorot_initializer\": my_glorot_initializer,\n",
    "       \"my_softplus\": my_softplus,\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8i816JgTnFSh"
   },
   "source": [
    "## 사용자 정의 지표"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:14.004630Z",
     "iopub.status.busy": "2021-07-04T08:15:13.996233Z",
     "iopub.status.idle": "2021-07-04T08:15:14.010998Z",
     "shell.execute_reply": "2021-07-04T08:15:14.009850Z"
    },
    "id": "EZWWawkinFSh"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:14.025252Z",
     "iopub.status.busy": "2021-07-04T08:15:14.023888Z",
     "iopub.status.idle": "2021-07-04T08:15:14.047903Z",
     "shell.execute_reply": "2021-07-04T08:15:14.048881Z"
    },
    "id": "K1fe2oELnFSh"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:14.065249Z",
     "iopub.status.busy": "2021-07-04T08:15:14.064079Z",
     "iopub.status.idle": "2021-07-04T08:15:14.070303Z",
     "shell.execute_reply": "2021-07-04T08:15:14.071380Z"
    },
    "id": "SIHTKaKNnFSh"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[create_huber(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:14.078905Z",
     "iopub.status.busy": "2021-07-04T08:15:14.077659Z",
     "iopub.status.idle": "2021-07-04T08:15:17.591938Z",
     "shell.execute_reply": "2021-07-04T08:15:17.592933Z"
    },
    "id": "g4l-_an2nFSh",
    "outputId": "0b42b348-4068-4e70-f444-1b5883d8607d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 2.0982 - huber_fn: 0.9192\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.6052 - huber_fn: 0.2733\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd476f3bbd0>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZqeCDPtvnFSh"
   },
   "source": [
    "**노트**: 손실과 지표에 같은 함수를 사용하면 다른 결과가 나올 수 있습니다. 이는 일반적으로 부동 소수점 정밀도 오차 때문입니다. 수학 식이 동일하더라도 연산은 동일한 순서대로 실행되지 않습니다. 이로 인해 작은 차이가 발생합니다. 또한 샘플 가중치를 사용하면 정밀도보다 더 큰 오차가 생깁니다:\n",
    "\n",
    "* 에포크에서 손실은 지금까지 본 모든 배치 손실의 평균입니다. 각 배치 손실은 가중치가 적용된 샘플 손실의 합을 _배치 크기_ 로 나눈 것입니다(샘플 가중치의 합으로 나눈 것이 아닙니다. 따라서 배치 손실은 손실의 가중 평균이 아닙니다).\n",
    "* 에포크에서 지표는 가중치가 적용된 샘플 손실의 합을 지금까지 본 모든 샘플 가중치의 합으로 나눈 것입니다. 다른 말로하면 모든 샘플 손실의 가중 평균입니다. 따라서 위와 같지 않습니다.\n",
    "\n",
    "수학적으로 말하면 손실 = 지표 * 샘플 가중치의 평균(더하기 약간의 부동 소수점 정밀도 오차)입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:17.602557Z",
     "iopub.status.busy": "2021-07-04T08:15:17.601483Z",
     "iopub.status.idle": "2021-07-04T08:15:17.618592Z",
     "shell.execute_reply": "2021-07-04T08:15:17.619398Z"
    },
    "id": "9DGQB6V3nFSh"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[create_huber(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:17.626600Z",
     "iopub.status.busy": "2021-07-04T08:15:17.625547Z",
     "iopub.status.idle": "2021-07-04T08:15:22.357734Z",
     "shell.execute_reply": "2021-07-04T08:15:22.356536Z"
    },
    "id": "EjeISBIOnFSh",
    "outputId": "3484be7d-b172-4be9-c8f1-8758a24467bc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 3ms/step - loss: 0.1175 - huber_fn: 0.2399\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.1131 - huber_fn: 0.2297\n"
     ]
    }
   ],
   "source": [
    "sample_weight = np.random.rand(len(y_train))\n",
    "history = model.fit(X_train_scaled, y_train, epochs=2, sample_weight=sample_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.369800Z",
     "iopub.status.busy": "2021-07-04T08:15:22.368364Z",
     "iopub.status.idle": "2021-07-04T08:15:22.373044Z",
     "shell.execute_reply": "2021-07-04T08:15:22.374077Z"
    },
    "id": "eKbXc3FInFSi",
    "outputId": "e199de4c-bb02-41d8-8c04-7d88d1bb1988"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.11749907582998276, 0.11906625573138947)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history[\"loss\"][0], history.history[\"huber_fn\"][0] * sample_weight.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8_Df63lFnFSi"
   },
   "source": [
    "### 스트리밍 지표"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.626671Z",
     "iopub.status.busy": "2021-07-04T08:15:22.624853Z",
     "iopub.status.idle": "2021-07-04T08:15:22.749065Z",
     "shell.execute_reply": "2021-07-04T08:15:22.750374Z"
    },
    "id": "7wW4uLd9nFSi",
    "outputId": "9a8d3c26-bc07-4bf1-d104-83474e997416"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.8>"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision = keras.metrics.Precision()\n",
    "precision([0, 1, 1, 1, 0, 1, 0, 1], [1, 1, 0, 1, 0, 1, 0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.759945Z",
     "iopub.status.busy": "2021-07-04T08:15:22.759008Z",
     "iopub.status.idle": "2021-07-04T08:15:22.769857Z",
     "shell.execute_reply": "2021-07-04T08:15:22.770856Z"
    },
    "id": "zed4nGD3nFSi",
    "outputId": "4461d5d8-ce7f-4589-b3b5-f134b9102915"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.5>"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision([0, 1, 0, 0, 1, 0, 1, 1], [1, 0, 1, 1, 0, 0, 0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.777898Z",
     "iopub.status.busy": "2021-07-04T08:15:22.776585Z",
     "iopub.status.idle": "2021-07-04T08:15:22.783204Z",
     "shell.execute_reply": "2021-07-04T08:15:22.784016Z"
    },
    "id": "RqFgNHLPnFSi",
    "outputId": "52e2add8-3f95-4498-e6c8-c971d3e637ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.5>"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.791061Z",
     "iopub.status.busy": "2021-07-04T08:15:22.789955Z",
     "iopub.status.idle": "2021-07-04T08:15:22.796509Z",
     "shell.execute_reply": "2021-07-04T08:15:22.797376Z"
    },
    "id": "DoCvoOt5nFSi",
    "outputId": "c847c2d0-e801-4dfe-fb5d-615ee2c0fe21"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'true_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>,\n",
       " <tf.Variable 'false_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.803983Z",
     "iopub.status.busy": "2021-07-04T08:15:22.802886Z",
     "iopub.status.idle": "2021-07-04T08:15:22.807339Z",
     "shell.execute_reply": "2021-07-04T08:15:22.808102Z"
    },
    "id": "Bb3EbnzGnFSi"
   },
   "outputs": [],
   "source": [
    "precision.reset_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WuhZp4TEnFSi"
   },
   "source": [
    "스트리밍 지표 만들기:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.825852Z",
     "iopub.status.busy": "2021-07-04T08:15:22.824787Z",
     "iopub.status.idle": "2021-07-04T08:15:22.828894Z",
     "shell.execute_reply": "2021-07-04T08:15:22.829904Z"
    },
    "id": "PseG5pPUnFSi"
   },
   "outputs": [],
   "source": [
    "class HuberMetric(keras.metrics.Metric):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        super().__init__(**kwargs) # 기본 매개변수 처리 (예를 들면, dtype)\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        self.total = self.add_weight(\"total\", initializer=\"zeros\")\n",
    "        self.count = self.add_weight(\"count\", initializer=\"zeros\")\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        metric = self.huber_fn(y_true, y_pred)\n",
    "        self.total.assign_add(tf.reduce_sum(metric))\n",
    "        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))\n",
    "    def result(self):\n",
    "        return self.total / self.count\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.849379Z",
     "iopub.status.busy": "2021-07-04T08:15:22.842123Z",
     "iopub.status.idle": "2021-07-04T08:15:22.979409Z",
     "shell.execute_reply": "2021-07-04T08:15:22.980690Z"
    },
    "id": "N3oSW-yJnFSi",
    "outputId": "c6168f06-a1a0-4a27-c304-b863fc4a0f13"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=14.0>"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = HuberMetric(2.)\n",
    "\n",
    "# total = 2 * |10 - 2| - 2²/2 = 14\n",
    "# count = 1\n",
    "# result = 14 / 1 = 14\n",
    "m(tf.constant([[2.]]), tf.constant([[10.]])) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:22.992007Z",
     "iopub.status.busy": "2021-07-04T08:15:22.990814Z",
     "iopub.status.idle": "2021-07-04T08:15:23.000738Z",
     "shell.execute_reply": "2021-07-04T08:15:23.001656Z"
    },
    "id": "fZxdkhK_nFSi",
    "outputId": "096a373e-d98a-4fff-8320-7e78788b6d32"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=7.0>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total = total + (|1 - 0|² / 2) + (2 * |9.25 - 5| - 2² / 2) = 14 + 7 = 21\n",
    "# count = count + 2 = 3\n",
    "# result = total / count = 21 / 3 = 7\n",
    "m(tf.constant([[0.], [5.]]), tf.constant([[1.], [9.25]]))\n",
    "\n",
    "m.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.007901Z",
     "iopub.status.busy": "2021-07-04T08:15:23.006876Z",
     "iopub.status.idle": "2021-07-04T08:15:23.012642Z",
     "shell.execute_reply": "2021-07-04T08:15:23.013453Z"
    },
    "id": "V4Yj14hCnFSj",
    "outputId": "1e5d3580-85db-4d3e-87d6-6147b9ab4139"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'total:0' shape=() dtype=float32, numpy=21.0>,\n",
       " <tf.Variable 'count:0' shape=() dtype=float32, numpy=3.0>]"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.020099Z",
     "iopub.status.busy": "2021-07-04T08:15:23.019059Z",
     "iopub.status.idle": "2021-07-04T08:15:23.025577Z",
     "shell.execute_reply": "2021-07-04T08:15:23.026477Z"
    },
    "id": "AR4lyDg4nFSj",
    "outputId": "e4cc7879-23d8-4a53-dd57-95400cb52a3c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'total:0' shape=() dtype=float32, numpy=0.0>,\n",
       " <tf.Variable 'count:0' shape=() dtype=float32, numpy=0.0>]"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.reset_states()\n",
    "m.variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NWT6cVPunFSj"
   },
   "source": [
    "`HuberMetric` 클래스가 잘 동작하는지 확인해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.035392Z",
     "iopub.status.busy": "2021-07-04T08:15:23.034294Z",
     "iopub.status.idle": "2021-07-04T08:15:23.044938Z",
     "shell.execute_reply": "2021-07-04T08:15:23.045831Z"
    },
    "id": "o_JZayvonFSj"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.059263Z",
     "iopub.status.busy": "2021-07-04T08:15:23.057895Z",
     "iopub.status.idle": "2021-07-04T08:15:23.080858Z",
     "shell.execute_reply": "2021-07-04T08:15:23.081963Z"
    },
    "id": "JI1L36IJnFSj"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.096408Z",
     "iopub.status.busy": "2021-07-04T08:15:23.092739Z",
     "iopub.status.idle": "2021-07-04T08:15:23.109658Z",
     "shell.execute_reply": "2021-07-04T08:15:23.110862Z"
    },
    "id": "6sJSLY0HnFSj"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[HuberMetric(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:23.120177Z",
     "iopub.status.busy": "2021-07-04T08:15:23.118905Z",
     "iopub.status.idle": "2021-07-04T08:15:26.468091Z",
     "shell.execute_reply": "2021-07-04T08:15:26.469262Z"
    },
    "id": "68uiyAcUnFSj",
    "outputId": "fb70d71b-7ded-43dd-c69c-7b0d93e4c53b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 3ms/step - loss: 0.8707 - huber_metric: 0.8707\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.2595 - huber_metric: 0.2595\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47a4e9fd0>"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:26.501237Z",
     "iopub.status.busy": "2021-07-04T08:15:26.499598Z",
     "iopub.status.idle": "2021-07-04T08:15:26.526806Z",
     "shell.execute_reply": "2021-07-04T08:15:26.527883Z"
    },
    "id": "bxInm5qnnFSj"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:26.536250Z",
     "iopub.status.busy": "2021-07-04T08:15:26.534870Z",
     "iopub.status.idle": "2021-07-04T08:15:26.603542Z",
     "shell.execute_reply": "2021-07-04T08:15:26.604356Z"
    },
    "id": "7JTorko_nFSj"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric.h5\",\n",
    "                                custom_objects={\"huber_fn\": create_huber(2.0),\n",
    "                                                \"HuberMetric\": HuberMetric})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:26.612094Z",
     "iopub.status.busy": "2021-07-04T08:15:26.610601Z",
     "iopub.status.idle": "2021-07-04T08:15:29.811418Z",
     "shell.execute_reply": "2021-07-04T08:15:29.810056Z"
    },
    "id": "EO4pBgVxnFSj",
    "outputId": "89d66c65-882d-4ac5-f9dd-12527a04f0f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 3ms/step - loss: 0.2350 - huber_metric: 0.2350\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.2278 - huber_metric: 0.2278\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd47a326690>"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mH6-Xw2HnFSj"
   },
   "source": [
    "**경고**: 텐서플로 2.2에서 tf.keras가 `model.metrics`의 0번째 위치에 지표를 추가합니다([텐서플로 이슈 #38150](https://github.com/tensorflow/tensorflow/issues/38150) 참조). 따라서 `HuberMetric`에 접근하려면 `model.metrics[0]` 대신 `model.metrics[-1]`를 사용해야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.820526Z",
     "iopub.status.busy": "2021-07-04T08:15:29.819586Z",
     "iopub.status.idle": "2021-07-04T08:15:29.825170Z",
     "shell.execute_reply": "2021-07-04T08:15:29.824379Z"
    },
    "id": "4KnXWivunFSk",
    "outputId": "0404c5ad-3def-49a5-bd71-c5c1469612f4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics[-1].threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ViTVxzoBnFSk"
   },
   "source": [
    "잘 동작하는군요! 다음처럼 더 간단하게 클래스를 만들 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.840294Z",
     "iopub.status.busy": "2021-07-04T08:15:29.838769Z",
     "iopub.status.idle": "2021-07-04T08:15:29.842340Z",
     "shell.execute_reply": "2021-07-04T08:15:29.843598Z"
    },
    "id": "f43iC6UWnFSk"
   },
   "outputs": [],
   "source": [
    "class HuberMetric(keras.metrics.Mean):\n",
    "    def __init__(self, threshold=1.0, name='HuberMetric', dtype=None):\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        super().__init__(name=name, dtype=dtype)\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        metric = self.huber_fn(y_true, y_pred)\n",
    "        super(HuberMetric, self).update_state(metric, sample_weight)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FtBpUQKWnFSk"
   },
   "source": [
    "이 클래스는 크기를 잘 처리하고 샘플 가중치도 지원합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.852413Z",
     "iopub.status.busy": "2021-07-04T08:15:29.850905Z",
     "iopub.status.idle": "2021-07-04T08:15:29.863225Z",
     "shell.execute_reply": "2021-07-04T08:15:29.863996Z"
    },
    "id": "CFi9R0PsnFSk"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.880400Z",
     "iopub.status.busy": "2021-07-04T08:15:29.878725Z",
     "iopub.status.idle": "2021-07-04T08:15:29.903763Z",
     "shell.execute_reply": "2021-07-04T08:15:29.905048Z"
    },
    "id": "YNuyOkfjnFSk"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.923238Z",
     "iopub.status.busy": "2021-07-04T08:15:29.922379Z",
     "iopub.status.idle": "2021-07-04T08:15:29.948652Z",
     "shell.execute_reply": "2021-07-04T08:15:29.949313Z"
    },
    "id": "uLjhx8PCnFSk"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=keras.losses.Huber(2.0), optimizer=\"nadam\", weighted_metrics=[HuberMetric(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:29.956912Z",
     "iopub.status.busy": "2021-07-04T08:15:29.956048Z",
     "iopub.status.idle": "2021-07-04T08:15:35.132118Z",
     "shell.execute_reply": "2021-07-04T08:15:35.131115Z"
    },
    "id": "09rxtKpanFSk",
    "outputId": "98c06747-17aa-4c68-f947-37b5da6ffe8d",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.4455 - HuberMetric: 0.8978\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.1305 - HuberMetric: 0.2631\n"
     ]
    }
   ],
   "source": [
    "sample_weight = np.random.rand(len(y_train))\n",
    "history = model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32),\n",
    "                    epochs=2, sample_weight=sample_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:35.143922Z",
     "iopub.status.busy": "2021-07-04T08:15:35.142733Z",
     "iopub.status.idle": "2021-07-04T08:15:35.147546Z",
     "shell.execute_reply": "2021-07-04T08:15:35.148541Z"
    },
    "id": "mJ4-Y0E2nFSk",
    "outputId": "a0b07151-f308-4339-d169-ca57abf441cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.44554364681243896, 0.44554368685750223)"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history[\"loss\"][0], history.history[\"HuberMetric\"][0] * sample_weight.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:35.180715Z",
     "iopub.status.busy": "2021-07-04T08:15:35.179527Z",
     "iopub.status.idle": "2021-07-04T08:15:35.212988Z",
     "shell.execute_reply": "2021-07-04T08:15:35.213970Z"
    },
    "id": "adIdqEocnFSk"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric_v2.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:35.223140Z",
     "iopub.status.busy": "2021-07-04T08:15:35.221582Z",
     "iopub.status.idle": "2021-07-04T08:15:35.311024Z",
     "shell.execute_reply": "2021-07-04T08:15:35.312329Z"
    },
    "id": "F_niUjqDnFSk"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric_v2.h5\",\n",
    "                                custom_objects={\"HuberMetric\": HuberMetric})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:35.321452Z",
     "iopub.status.busy": "2021-07-04T08:15:35.319795Z",
     "iopub.status.idle": "2021-07-04T08:15:38.912190Z",
     "shell.execute_reply": "2021-07-04T08:15:38.913187Z"
    },
    "id": "7V2PkjHknFSl",
    "outputId": "ceb9ae9b-5821-47b5-f803-50b6820bdd7b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2377 - HuberMetric: 0.2377\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.2279 - HuberMetric: 0.2279\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd479ff9c90>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:38.921614Z",
     "iopub.status.busy": "2021-07-04T08:15:38.920593Z",
     "iopub.status.idle": "2021-07-04T08:15:38.925260Z",
     "shell.execute_reply": "2021-07-04T08:15:38.926140Z"
    },
    "id": "LiY6tzrrnFSl",
    "outputId": "bba96dba-f439-4c16-fee6-88e7166fd4b4",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics[-1].threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wHvCg6OCnFSl"
   },
   "source": [
    "## 사용자 정의 층"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:38.934702Z",
     "iopub.status.busy": "2021-07-04T08:15:38.933594Z",
     "iopub.status.idle": "2021-07-04T08:15:38.937240Z",
     "shell.execute_reply": "2021-07-04T08:15:38.938128Z"
    },
    "id": "yaYQR8DxnFSm"
   },
   "outputs": [],
   "source": [
    "exponential_layer = keras.layers.Lambda(lambda x: tf.exp(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:38.945934Z",
     "iopub.status.busy": "2021-07-04T08:15:38.944822Z",
     "iopub.status.idle": "2021-07-04T08:15:38.951723Z",
     "shell.execute_reply": "2021-07-04T08:15:38.952589Z"
    },
    "id": "JbXxzU34nFSm",
    "outputId": "a703b63f-c919-4574-8917-31a565c05890"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([0.36787948, 1.        , 2.7182817 ], dtype=float32)>"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exponential_layer([-1., 0., 1.])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mM-jjL37nFSm"
   },
   "source": [
    "회귀 모델이 예측할 값이 양수이고 스케일이 매우 다른 경우 (예를 들어, 0.001, 10., 10000) 출력층에 지수 함수를 추가하면 유용할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:38.960487Z",
     "iopub.status.busy": "2021-07-04T08:15:38.959393Z",
     "iopub.status.idle": "2021-07-04T08:15:38.976960Z",
     "shell.execute_reply": "2021-07-04T08:15:38.976101Z"
    },
    "id": "H_fZsjonnFSm"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:38.997361Z",
     "iopub.status.busy": "2021-07-04T08:15:38.995721Z",
     "iopub.status.idle": "2021-07-04T08:15:45.243279Z",
     "shell.execute_reply": "2021-07-04T08:15:45.244491Z"
    },
    "id": "GVDT8nyxnFSm",
    "outputId": "576b6915-f500-4db9-b9c5-58107c0b26fa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 2s 3ms/step - loss: 1.0631 - val_loss: 0.4457\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4562 - val_loss: 0.3798\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4029 - val_loss: 0.3548\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3851 - val_loss: 0.3464\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3708 - val_loss: 0.3449\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.3586\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.3586340546607971"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "    exponential_layer\n",
    "])\n",
    "model.compile(loss=\"mse\", optimizer=\"sgd\")\n",
    "model.fit(X_train_scaled, y_train, epochs=5,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:45.266884Z",
     "iopub.status.busy": "2021-07-04T08:15:45.265426Z",
     "iopub.status.idle": "2021-07-04T08:15:45.269058Z",
     "shell.execute_reply": "2021-07-04T08:15:45.270234Z"
    },
    "id": "BD9ygbq5nFSm"
   },
   "outputs": [],
   "source": [
    "class MyDense(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=None, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.activation = keras.activations.get(activation)\n",
    "\n",
    "    def build(self, batch_input_shape):\n",
    "        self.kernel = self.add_weight(\n",
    "            name=\"kernel\", shape=[batch_input_shape[-1], self.units],\n",
    "            initializer=\"glorot_normal\")\n",
    "        self.bias = self.add_weight(\n",
    "            name=\"bias\", shape=[self.units], initializer=\"zeros\")\n",
    "        super().build(batch_input_shape) # must be at the end\n",
    "\n",
    "    def call(self, X):\n",
    "        return self.activation(X @ self.kernel + self.bias)\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return tf.TensorShape(batch_input_shape.as_list()[:-1] + [self.units])\n",
    "\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"units\": self.units,\n",
    "                \"activation\": keras.activations.serialize(self.activation)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:45.281847Z",
     "iopub.status.busy": "2021-07-04T08:15:45.278785Z",
     "iopub.status.idle": "2021-07-04T08:15:45.285433Z",
     "shell.execute_reply": "2021-07-04T08:15:45.286330Z"
    },
    "id": "PhAOl_-xnFSm"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:45.298411Z",
     "iopub.status.busy": "2021-07-04T08:15:45.297225Z",
     "iopub.status.idle": "2021-07-04T08:15:45.381189Z",
     "shell.execute_reply": "2021-07-04T08:15:45.382152Z"
    },
    "id": "Ux2CIm0QnFSm"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    MyDense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    MyDense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:45.400162Z",
     "iopub.status.busy": "2021-07-04T08:15:45.398953Z",
     "iopub.status.idle": "2021-07-04T08:15:49.494875Z",
     "shell.execute_reply": "2021-07-04T08:15:49.495803Z"
    },
    "id": "ZzYKMKhfnFSm",
    "outputId": "1e267b1f-84f6-466e-d1ac-3a1c17ba70e0",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 2.2563 - val_loss: 0.9472\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.6485 - val_loss: 0.6219\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.5474\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5473726987838745"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.526507Z",
     "iopub.status.busy": "2021-07-04T08:15:49.525256Z",
     "iopub.status.idle": "2021-07-04T08:15:49.556131Z",
     "shell.execute_reply": "2021-07-04T08:15:49.557059Z"
    },
    "id": "D3dQrUgwnFSm"
   },
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_layer.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.565553Z",
     "iopub.status.busy": "2021-07-04T08:15:49.564073Z",
     "iopub.status.idle": "2021-07-04T08:15:49.637729Z",
     "shell.execute_reply": "2021-07-04T08:15:49.638951Z"
    },
    "id": "vqqfaYhpnFSn"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_layer.h5\",\n",
    "                                custom_objects={\"MyDense\": MyDense})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.650052Z",
     "iopub.status.busy": "2021-07-04T08:15:49.648530Z",
     "iopub.status.idle": "2021-07-04T08:15:49.651608Z",
     "shell.execute_reply": "2021-07-04T08:15:49.652527Z"
    },
    "id": "XE8kj1ZCnFSn"
   },
   "outputs": [],
   "source": [
    "class MyMultiLayer(keras.layers.Layer):\n",
    "    def call(self, X):\n",
    "        X1, X2 = X\n",
    "        print(\"X1.shape: \", X1.shape ,\" X2.shape: \", X2.shape) # 사용자 정의 층 디버깅\n",
    "        return X1 + X2, X1 * X2\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        batch_input_shape1, batch_input_shape2 = batch_input_shape\n",
    "        return [batch_input_shape1, batch_input_shape2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZRR1NXSpnFSn"
   },
   "source": [
    "사용자 정의 층은 다음처럼 함수형 API를 사용해 호출할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.671914Z",
     "iopub.status.busy": "2021-07-04T08:15:49.670532Z",
     "iopub.status.idle": "2021-07-04T08:15:49.736314Z",
     "shell.execute_reply": "2021-07-04T08:15:49.737569Z"
    },
    "id": "qlHFFWy8nFSn",
    "outputId": "e2b1de3b-ec1a-4fcc-a80b-1a24cc9780ee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (None, 2)  X2.shape:  (None, 2)\n"
     ]
    }
   ],
   "source": [
    "inputs1 = keras.layers.Input(shape=[2])\n",
    "inputs2 = keras.layers.Input(shape=[2])\n",
    "outputs1, outputs2 = MyMultiLayer()((inputs1, inputs2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rnu1SI3pnFSn"
   },
   "source": [
    "`call()` 메서드는 심볼릭 입력을 받습니다. 이 입력의 크기는 부분적으로만 지정되어 있습니다(이 시점에서는 배치 크기를 모릅니다. 그래서 첫 번째 차원이 None입니다):\n",
    "\n",
    "사용자 층에 실제 데이터를 전달할 수도 있습니다. 이를 테스트하기 위해 각 데이터셋의 입력을 각각 네 개의 특성을 가진 두 부분으로 나누겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.750493Z",
     "iopub.status.busy": "2021-07-04T08:15:49.749103Z",
     "iopub.status.idle": "2021-07-04T08:15:49.753632Z",
     "shell.execute_reply": "2021-07-04T08:15:49.754572Z"
    },
    "id": "5eIOchaEnFSn",
    "outputId": "5e6292de-75f8-41b1-fb1d-a09e0c08694c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((11610, 4), (11610, 4))"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def split_data(data):\n",
    "    columns_count = data.shape[-1]\n",
    "    half = columns_count // 2\n",
    "    return data[:, :half], data[:, half:]\n",
    "\n",
    "X_train_scaled_A, X_train_scaled_B = split_data(X_train_scaled)\n",
    "X_valid_scaled_A, X_valid_scaled_B = split_data(X_valid_scaled)\n",
    "X_test_scaled_A, X_test_scaled_B = split_data(X_test_scaled)\n",
    "\n",
    "# 분할된 데이터 크기 출력\n",
    "X_train_scaled_A.shape, X_train_scaled_B.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2Kjpmi_lnFSn"
   },
   "source": [
    "크기가 완전하게 지정된 것을 볼 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.762391Z",
     "iopub.status.busy": "2021-07-04T08:15:49.761275Z",
     "iopub.status.idle": "2021-07-04T08:15:49.769201Z",
     "shell.execute_reply": "2021-07-04T08:15:49.770026Z"
    },
    "id": "QUFlvjdNnFSn",
    "outputId": "1482c85a-5fe0-40f1-ccc3-0e5c3e2a0819"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (11610, 4)  X2.shape:  (11610, 4)\n"
     ]
    }
   ],
   "source": [
    "outputs1, outputs2 = MyMultiLayer()((X_train_scaled_A, X_train_scaled_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Gj0EYDB9nFSn"
   },
   "source": [
    "함수형 API를 사용해 완전한 모델을 만들어 보겠습니다(이 모델은 간단한 예제이므로 놀라운 성능을 기대하지 마세요):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.784955Z",
     "iopub.status.busy": "2021-07-04T08:15:49.783919Z",
     "iopub.status.idle": "2021-07-04T08:15:49.838940Z",
     "shell.execute_reply": "2021-07-04T08:15:49.840327Z"
    },
    "id": "ITI2PrCbnFSn",
    "outputId": "7953ca01-ec92-4a0c-c10b-6fd3733cbdd8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "input_A = keras.layers.Input(shape=X_train_scaled_A.shape[-1])\n",
    "input_B = keras.layers.Input(shape=X_train_scaled_B.shape[-1])\n",
    "hidden_A, hidden_B = MyMultiLayer()((input_A, input_B))\n",
    "hidden_A = keras.layers.Dense(30, activation='selu')(hidden_A)\n",
    "hidden_B = keras.layers.Dense(30, activation='selu')(hidden_B)\n",
    "concat = keras.layers.Concatenate()((hidden_A, hidden_B))\n",
    "output = keras.layers.Dense(1)(concat)\n",
    "model = keras.models.Model(inputs=[input_A, input_B], outputs=[output])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.858386Z",
     "iopub.status.busy": "2021-07-04T08:15:49.856892Z",
     "iopub.status.idle": "2021-07-04T08:15:49.862315Z",
     "shell.execute_reply": "2021-07-04T08:15:49.863304Z"
    },
    "id": "Hf5QYYp6nFSn"
   },
   "outputs": [],
   "source": [
    "model.compile(loss='mse', optimizer='nadam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:49.870419Z",
     "iopub.status.busy": "2021-07-04T08:15:49.869216Z",
     "iopub.status.idle": "2021-07-04T08:15:54.377789Z",
     "shell.execute_reply": "2021-07-04T08:15:54.378814Z"
    },
    "id": "TlnQRPmenFSo",
    "outputId": "448960f2-5023-4334-f24d-92870bc1b192"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "353/363 [============================>.] - ETA: 0s - loss: 2.1461X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 2.1142 - val_loss: 1.3630\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.9684 - val_loss: 0.9773\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd479af8410>"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit((X_train_scaled_A, X_train_scaled_B), y_train, epochs=2,\n",
    "          validation_data=((X_valid_scaled_A, X_valid_scaled_B), y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7wMgYJv8nFSo"
   },
   "source": [
    "훈련과 테스트에서 다르게 동작하는 층을 만들어 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:54.393887Z",
     "iopub.status.busy": "2021-07-04T08:15:54.392577Z",
     "iopub.status.idle": "2021-07-04T08:15:54.396213Z",
     "shell.execute_reply": "2021-07-04T08:15:54.397212Z"
    },
    "id": "NeKKkO32nFSo"
   },
   "outputs": [],
   "source": [
    "class AddGaussianNoise(keras.layers.Layer):\n",
    "    def __init__(self, stddev, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.stddev = stddev\n",
    "\n",
    "    def call(self, X, training=None):\n",
    "        if training:\n",
    "            noise = tf.random.normal(tf.shape(X), stddev=self.stddev)\n",
    "            return X + noise\n",
    "        else:\n",
    "            return X\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return batch_input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jh5WfjzunFSo"
   },
   "source": [
    "다음은 사용자 정의 층을 사용하는 간단한 모델입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:54.420349Z",
     "iopub.status.busy": "2021-07-04T08:15:54.407706Z",
     "iopub.status.idle": "2021-07-04T08:15:54.426847Z",
     "shell.execute_reply": "2021-07-04T08:15:54.428025Z"
    },
    "id": "Ln23QcsdnFSo"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    AddGaussianNoise(stddev=1.0),\n",
    "    keras.layers.Dense(30, activation=\"selu\"),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:54.446870Z",
     "iopub.status.busy": "2021-07-04T08:15:54.445458Z",
     "iopub.status.idle": "2021-07-04T08:15:58.590668Z",
     "shell.execute_reply": "2021-07-04T08:15:58.591605Z"
    },
    "id": "Gj0MuOcKnFSo",
    "outputId": "61067f1c-8caa-47e8-8853-334f42336ea4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 3s 4ms/step - loss: 2.3857 - val_loss: 7.6082\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 1.0571 - val_loss: 4.4597\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.7560\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7559615969657898"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bo5Aksa0nFSo"
   },
   "source": [
    "## 사용자 정의 모델"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:58.599593Z",
     "iopub.status.busy": "2021-07-04T08:15:58.598425Z",
     "iopub.status.idle": "2021-07-04T08:15:58.601757Z",
     "shell.execute_reply": "2021-07-04T08:15:58.602628Z"
    },
    "id": "qZ9CBGORnFSo"
   },
   "outputs": [],
   "source": [
    "X_new_scaled = X_test_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:58.616167Z",
     "iopub.status.busy": "2021-07-04T08:15:58.614894Z",
     "iopub.status.idle": "2021-07-04T08:15:58.618458Z",
     "shell.execute_reply": "2021-07-04T08:15:58.619457Z"
    },
    "id": "ScNXzm1GnFSo"
   },
   "outputs": [],
   "source": [
    "class ResidualBlock(keras.layers.Layer):\n",
    "    def __init__(self, n_layers, n_neurons, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(n_neurons, activation=\"elu\",\n",
    "                                          kernel_initializer=\"he_normal\")\n",
    "                       for _ in range(n_layers)]\n",
    "\n",
    "    def call(self, inputs):\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        return inputs + Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:58.635663Z",
     "iopub.status.busy": "2021-07-04T08:15:58.634461Z",
     "iopub.status.idle": "2021-07-04T08:15:58.637358Z",
     "shell.execute_reply": "2021-07-04T08:15:58.638363Z"
    },
    "id": "LjmHGRPOnFSo"
   },
   "outputs": [],
   "source": [
    "class ResidualRegressor(keras.models.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden1 = keras.layers.Dense(30, activation=\"elu\",\n",
    "                                          kernel_initializer=\"he_normal\")\n",
    "        self.block1 = ResidualBlock(2, 30)\n",
    "        self.block2 = ResidualBlock(2, 30)\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        Z = self.hidden1(inputs)\n",
    "        for _ in range(1 + 3):\n",
    "            Z = self.block1(Z)\n",
    "        Z = self.block2(Z)\n",
    "        return self.out(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:58.645468Z",
     "iopub.status.busy": "2021-07-04T08:15:58.644301Z",
     "iopub.status.idle": "2021-07-04T08:15:58.653372Z",
     "shell.execute_reply": "2021-07-04T08:15:58.654158Z"
    },
    "id": "oP_n1Vf1nFSo"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:15:58.666308Z",
     "iopub.status.busy": "2021-07-04T08:15:58.665238Z",
     "iopub.status.idle": "2021-07-04T08:16:14.531553Z",
     "shell.execute_reply": "2021-07-04T08:16:14.532718Z"
    },
    "id": "FTnG6qh4nFSp",
    "outputId": "22a7e069-f609-4f0b-bdaf-ace1c3411bfa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 9.1324\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 1.0580\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.8870\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.5834\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.6460\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.6507\n"
     ]
    }
   ],
   "source": [
    "model = ResidualRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:14.544665Z",
     "iopub.status.busy": "2021-07-04T08:16:14.543467Z",
     "iopub.status.idle": "2021-07-04T08:16:15.547522Z",
     "shell.execute_reply": "2021-07-04T08:16:15.548926Z"
    },
    "id": "aqkYIQvOnFSp",
    "outputId": "3ecc3fbc-7b8a-4b76-d73d-75234cc81740"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:Found untraced functions such as dense_1_layer_call_and_return_conditional_losses, dense_1_layer_call_fn, dense_2_layer_call_and_return_conditional_losses, dense_2_layer_call_fn, dense_3_layer_call_and_return_conditional_losses while saving (showing 5 of 20). These functions will not be directly callable after loading.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_custom_model.ckpt/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_custom_model.ckpt/assets\n"
     ]
    }
   ],
   "source": [
    "model.save(\"my_custom_model.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:15.556474Z",
     "iopub.status.busy": "2021-07-04T08:16:15.555185Z",
     "iopub.status.idle": "2021-07-04T08:16:16.173104Z",
     "shell.execute_reply": "2021-07-04T08:16:16.171956Z"
    },
    "id": "j2fgl5UknFSp"
   },
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_custom_model.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:16.182435Z",
     "iopub.status.busy": "2021-07-04T08:16:16.180781Z",
     "iopub.status.idle": "2021-07-04T08:16:30.488212Z",
     "shell.execute_reply": "2021-07-04T08:16:30.487051Z"
    },
    "id": "yaRZkdOCnFSp",
    "outputId": "5f0fbc3b-92c5-4a16-d7bf-d7b6ea2327ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 0.8036\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.5590\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.4959\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.4256\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.4237\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VoxJM__-nFSp"
   },
   "source": [
    "대신 시퀀셜 API를 사용하는 모델을 정의할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:30.499599Z",
     "iopub.status.busy": "2021-07-04T08:16:30.498333Z",
     "iopub.status.idle": "2021-07-04T08:16:30.521900Z",
     "shell.execute_reply": "2021-07-04T08:16:30.522906Z"
    },
    "id": "SNXYstMCnFSp"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:30.547691Z",
     "iopub.status.busy": "2021-07-04T08:16:30.546181Z",
     "iopub.status.idle": "2021-07-04T08:16:30.554408Z",
     "shell.execute_reply": "2021-07-04T08:16:30.553132Z"
    },
    "id": "j0lwOBjenFSp"
   },
   "outputs": [],
   "source": [
    "block1 = ResidualBlock(2, 30)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    block1, block1, block1, block1,\n",
    "    ResidualBlock(2, 30),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:30.573271Z",
     "iopub.status.busy": "2021-07-04T08:16:30.572301Z",
     "iopub.status.idle": "2021-07-04T08:16:44.015874Z",
     "shell.execute_reply": "2021-07-04T08:16:44.017074Z"
    },
    "id": "6Ol9WZK-nFSp",
    "outputId": "fee72b7b-9161-408a-e1e7-053bd689e584"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 0.8695\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.4720\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.5537\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.3809\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 2s 5ms/step - loss: 0.4012\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.4852\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0ivfG84mnFSp"
   },
   "source": [
    "## 모델 구성 요소에 기반한 손실과 지표"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jbr8J6n1nFSp"
   },
   "source": [
    "**노트**: 다음 코드는 책의 코드와 두 가지 다른 점이 있습니다:\n",
    "1. 생성자에서 `keras.metrics.Mean()` 측정 지표를 만들고 `call()` 메서드에서 사용하여 평균 재구성 손실을 추적합니다. 훈련에서만 사용해야 하기 때문에 `call()` 메서드에 `training` 매개변수를 추가합니다. `training`이 `True`이면 `reconstruction_mean`를 업데이트하고 `self.add_metric()`를 호출합니다.\n",
    "2. TF 2.2에 있는 이슈([#46858](https://github.com/tensorflow/tensorflow/issues/46858)) 때문에 `build()` 메서드 안에서 `super().build()`를 호출하면 안됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:44.044268Z",
     "iopub.status.busy": "2021-07-04T08:16:44.042711Z",
     "iopub.status.idle": "2021-07-04T08:16:44.045933Z",
     "shell.execute_reply": "2021-07-04T08:16:44.047194Z"
    },
    "id": "YHTjLpT3nFSp"
   },
   "outputs": [],
   "source": [
    "class ReconstructingRegressor(keras.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(30, activation=\"selu\",\n",
    "                                          kernel_initializer=\"lecun_normal\")\n",
    "                       for _ in range(5)]\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "        self.reconstruction_mean = keras.metrics.Mean(name=\"reconstruction_error\")\n",
    "\n",
    "    def build(self, batch_input_shape):\n",
    "        n_inputs = batch_input_shape[-1]\n",
    "        self.reconstruct = keras.layers.Dense(n_inputs)\n",
    "        #super().build(batch_input_shape)\n",
    "\n",
    "    def call(self, inputs, training=None):\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        reconstruction = self.reconstruct(Z)\n",
    "        self.recon_loss = 0.05 * tf.reduce_mean(tf.square(reconstruction - inputs))\n",
    "        \n",
    "        if training:\n",
    "           result = self.reconstruction_mean(recon_loss)\n",
    "           self.add_metric(result)\n",
    "        return self.out(Z)\n",
    "    \n",
    "    def train_step(self, data):\n",
    "        x, y = data\n",
    "\n",
    "        with tf.GradientTape() as tape:\n",
    "            y_pred = self(x)\n",
    "            loss = self.compiled_loss(y, y_pred, regularization_losses=[self.recon_loss])\n",
    "\n",
    "        gradients = tape.gradient(loss, self.trainable_variables)\n",
    "        self.optimizer.apply_gradients(zip(gradients, self.trainable_variables))\n",
    "\n",
    "        return {m.name: m.result() for m in self.metrics}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:44.062713Z",
     "iopub.status.busy": "2021-07-04T08:16:44.055839Z",
     "iopub.status.idle": "2021-07-04T08:16:44.066069Z",
     "shell.execute_reply": "2021-07-04T08:16:44.066880Z"
    },
    "id": "XQah3_DFnFSq"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:44.079276Z",
     "iopub.status.busy": "2021-07-04T08:16:44.077540Z",
     "iopub.status.idle": "2021-07-04T08:16:51.038448Z",
     "shell.execute_reply": "2021-07-04T08:16:51.039594Z"
    },
    "id": "fIBmQGYanFSq",
    "outputId": "057adc0b-9382-4ae4-ed3a-288d19ae9491"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 4s 6ms/step - loss: 0.7886 - reconstruction_error: 0.0000e+00\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 6ms/step - loss: 0.4134 - reconstruction_error: 0.0000e+00\n"
     ]
    }
   ],
   "source": [
    "model = ReconstructingRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=2)\n",
    "y_pred = model.predict(X_test_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rGDLWUXhnFSq"
   },
   "source": [
    "## 자동 미분을 사용하여 그레이디언트 계산하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.049280Z",
     "iopub.status.busy": "2021-07-04T08:16:51.047892Z",
     "iopub.status.idle": "2021-07-04T08:16:51.050911Z",
     "shell.execute_reply": "2021-07-04T08:16:51.051795Z"
    },
    "id": "NUl_neZpnFSr"
   },
   "outputs": [],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + 2 * w1 * w2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.060463Z",
     "iopub.status.busy": "2021-07-04T08:16:51.059412Z",
     "iopub.status.idle": "2021-07-04T08:16:51.064429Z",
     "shell.execute_reply": "2021-07-04T08:16:51.065316Z"
    },
    "id": "X1cstf8lnFSr",
    "outputId": "895a06e6-b96e-47f1-befd-d976f361c2a4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36.000003007075065"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w1, w2 = 5, 3\n",
    "eps = 1e-6\n",
    "(f(w1 + eps, w2) - f(w1, w2)) / eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.073140Z",
     "iopub.status.busy": "2021-07-04T08:16:51.072105Z",
     "iopub.status.idle": "2021-07-04T08:16:51.076959Z",
     "shell.execute_reply": "2021-07-04T08:16:51.077915Z"
    },
    "id": "bLBVUzu9nFSs",
    "outputId": "2f0d4b05-69df-4623-f8c4-530484a94004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.000000003174137"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(f(w1, w2 + eps) - f(w1, w2)) / eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.088035Z",
     "iopub.status.busy": "2021-07-04T08:16:51.086885Z",
     "iopub.status.idle": "2021-07-04T08:16:51.094155Z",
     "shell.execute_reply": "2021-07-04T08:16:51.095031Z"
    },
    "id": "wGGD9esynFSs"
   },
   "outputs": [],
   "source": [
    "w1, w2 = tf.Variable(5.), tf.Variable(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "gradients = tape.gradient(z, [w1, w2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.101660Z",
     "iopub.status.busy": "2021-07-04T08:16:51.100543Z",
     "iopub.status.idle": "2021-07-04T08:16:51.105890Z",
     "shell.execute_reply": "2021-07-04T08:16:51.106765Z"
    },
    "id": "WHexGsK7nFSs",
    "outputId": "bd147c62-d029-4ca2-a886-52e0680bc8e7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.116638Z",
     "iopub.status.busy": "2021-07-04T08:16:51.115509Z",
     "iopub.status.idle": "2021-07-04T08:16:51.122370Z",
     "shell.execute_reply": "2021-07-04T08:16:51.123249Z"
    },
    "id": "jDJYd-manFSs",
    "outputId": "606d4971-cefd-4c1d-e5b8-76c618438f49",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A non-persistent GradientTape can only be used to compute one set of gradients (or jacobians)\n"
     ]
    }
   ],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "dz_dw1 = tape.gradient(z, w1)\n",
    "try:\n",
    "    dz_dw2 = tape.gradient(z, w2)\n",
    "except RuntimeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.132420Z",
     "iopub.status.busy": "2021-07-04T08:16:51.131290Z",
     "iopub.status.idle": "2021-07-04T08:16:51.138370Z",
     "shell.execute_reply": "2021-07-04T08:16:51.139248Z"
    },
    "id": "DXw2nnLcnFSs"
   },
   "outputs": [],
   "source": [
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "dz_dw1 = tape.gradient(z, w1)\n",
    "dz_dw2 = tape.gradient(z, w2) # works now!\n",
    "del tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.146434Z",
     "iopub.status.busy": "2021-07-04T08:16:51.145227Z",
     "iopub.status.idle": "2021-07-04T08:16:51.151796Z",
     "shell.execute_reply": "2021-07-04T08:16:51.150789Z"
    },
    "id": "gJidVNB7nFSs",
    "outputId": "d55f4da2-5005-40c3-d52e-488a7f8d561b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>)"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dz_dw1, dz_dw2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.161659Z",
     "iopub.status.busy": "2021-07-04T08:16:51.160458Z",
     "iopub.status.idle": "2021-07-04T08:16:51.164923Z",
     "shell.execute_reply": "2021-07-04T08:16:51.165762Z"
    },
    "id": "tyXHSb1SnFSs"
   },
   "outputs": [],
   "source": [
    "c1, c2 = tf.constant(5.), tf.constant(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(c1, c2)\n",
    "\n",
    "gradients = tape.gradient(z, [c1, c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.172964Z",
     "iopub.status.busy": "2021-07-04T08:16:51.171909Z",
     "iopub.status.idle": "2021-07-04T08:16:51.176687Z",
     "shell.execute_reply": "2021-07-04T08:16:51.177570Z"
    },
    "id": "YrBr2oY_nFSs",
    "outputId": "fc41953f-590d-41ba-a3ae-5e38ce635dc7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None]"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.186564Z",
     "iopub.status.busy": "2021-07-04T08:16:51.185398Z",
     "iopub.status.idle": "2021-07-04T08:16:51.191128Z",
     "shell.execute_reply": "2021-07-04T08:16:51.192003Z"
    },
    "id": "PxGzWYUbnFSt"
   },
   "outputs": [],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    tape.watch(c1)\n",
    "    tape.watch(c2)\n",
    "    z = f(c1, c2)\n",
    "\n",
    "gradients = tape.gradient(z, [c1, c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.199435Z",
     "iopub.status.busy": "2021-07-04T08:16:51.198312Z",
     "iopub.status.idle": "2021-07-04T08:16:51.203980Z",
     "shell.execute_reply": "2021-07-04T08:16:51.204859Z"
    },
    "id": "sMfZRrDZnFSt",
    "outputId": "c80cc715-c429-4011-d89d-9cb91a5b6306"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.214904Z",
     "iopub.status.busy": "2021-07-04T08:16:51.213767Z",
     "iopub.status.idle": "2021-07-04T08:16:51.224864Z",
     "shell.execute_reply": "2021-07-04T08:16:51.225789Z"
    },
    "id": "ICrkGqt9nFSt",
    "outputId": "8ac4df3e-6f58-4389-b750-a28184ca9dee"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=136.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=30.0>]"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    z1 = f(w1, w2 + 2.)\n",
    "    z2 = f(w1, w2 + 5.)\n",
    "    z3 = f(w1, w2 + 7.)\n",
    "\n",
    "tape.gradient([z1, z2, z3], [w1, w2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.235927Z",
     "iopub.status.busy": "2021-07-04T08:16:51.234807Z",
     "iopub.status.idle": "2021-07-04T08:16:51.244752Z",
     "shell.execute_reply": "2021-07-04T08:16:51.245705Z"
    },
    "id": "ZrZ77owOnFSt"
   },
   "outputs": [],
   "source": [
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    z1 = f(w1, w2 + 2.)\n",
    "    z2 = f(w1, w2 + 5.)\n",
    "    z3 = f(w1, w2 + 7.)\n",
    "\n",
    "tf.reduce_sum(tf.stack([tape.gradient(z, [w1, w2]) for z in (z1, z2, z3)]), axis=0)\n",
    "del tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.256243Z",
     "iopub.status.busy": "2021-07-04T08:16:51.255046Z",
     "iopub.status.idle": "2021-07-04T08:16:51.262017Z",
     "shell.execute_reply": "2021-07-04T08:16:51.263038Z"
    },
    "id": "5VDDlIkVnFSt"
   },
   "outputs": [],
   "source": [
    "with tf.GradientTape(persistent=True) as hessian_tape:\n",
    "    with tf.GradientTape() as jacobian_tape:\n",
    "        z = f(w1, w2)\n",
    "    jacobians = jacobian_tape.gradient(z, [w1, w2])\n",
    "hessians = [hessian_tape.gradient(jacobian, [w1, w2])\n",
    "            for jacobian in jacobians]\n",
    "del hessian_tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.270264Z",
     "iopub.status.busy": "2021-07-04T08:16:51.268958Z",
     "iopub.status.idle": "2021-07-04T08:16:51.273742Z",
     "shell.execute_reply": "2021-07-04T08:16:51.274587Z"
    },
    "id": "HFk0AReSnFSt",
    "outputId": "2f375eca-0ff7-4f21-d86a-9d30d1cf12a9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jacobians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.281023Z",
     "iopub.status.busy": "2021-07-04T08:16:51.279999Z",
     "iopub.status.idle": "2021-07-04T08:16:51.286261Z",
     "shell.execute_reply": "2021-07-04T08:16:51.287093Z"
    },
    "id": "yZVuaFulnFSt",
    "outputId": "7974c698-77d9-4186-9860-ac1cc9111e37"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[<tf.Tensor: shape=(), dtype=float32, numpy=6.0>,\n",
       "  <tf.Tensor: shape=(), dtype=float32, numpy=2.0>],\n",
       " [<tf.Tensor: shape=(), dtype=float32, numpy=2.0>, None]]"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hessians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.296820Z",
     "iopub.status.busy": "2021-07-04T08:16:51.295762Z",
     "iopub.status.idle": "2021-07-04T08:16:51.303440Z",
     "shell.execute_reply": "2021-07-04T08:16:51.304381Z"
    },
    "id": "Z29IDrk0nFSu",
    "outputId": "9ce03f6b-7847-4c46-beb4-eb2a023f1582"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=30.0>, None]"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + tf.stop_gradient(2 * w1 * w2)\n",
    "\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "tape.gradient(z, [w1, w2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.313255Z",
     "iopub.status.busy": "2021-07-04T08:16:51.312065Z",
     "iopub.status.idle": "2021-07-04T08:16:51.319779Z",
     "shell.execute_reply": "2021-07-04T08:16:51.320650Z"
    },
    "id": "O5-EE34anFSu",
    "outputId": "b9f03eba-7437-4593-bf6c-3670c8003a00"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=nan>]"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable(100.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_softplus(x)\n",
    "\n",
    "tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.328636Z",
     "iopub.status.busy": "2021-07-04T08:16:51.327510Z",
     "iopub.status.idle": "2021-07-04T08:16:51.333810Z",
     "shell.execute_reply": "2021-07-04T08:16:51.334689Z"
    },
    "id": "Da4WXjSvnFSu",
    "outputId": "d5c30d95-563a-42a2-b39d-8a7c506f6f87"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=30.0>"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.math.log(tf.exp(tf.constant(30., dtype=tf.float32)) + 1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.343670Z",
     "iopub.status.busy": "2021-07-04T08:16:51.342529Z",
     "iopub.status.idle": "2021-07-04T08:16:51.351679Z",
     "shell.execute_reply": "2021-07-04T08:16:51.352632Z"
    },
    "id": "isJYdf_DnFSu",
    "outputId": "e93f7627-b12c-4d5e-fa0e-3282b37f6160"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>]"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable([100.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_softplus(x)\n",
    "\n",
    "tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.364281Z",
     "iopub.status.busy": "2021-07-04T08:16:51.363154Z",
     "iopub.status.idle": "2021-07-04T08:16:51.367261Z",
     "shell.execute_reply": "2021-07-04T08:16:51.368341Z"
    },
    "id": "v4CtFIKOnFSu"
   },
   "outputs": [],
   "source": [
    "@tf.custom_gradient\n",
    "def my_better_softplus(z):\n",
    "    exp = tf.exp(z)\n",
    "    def my_softplus_gradients(grad):\n",
    "        return grad / (1 + 1 / exp)\n",
    "    return tf.math.log(exp + 1), my_softplus_gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.378090Z",
     "iopub.status.busy": "2021-07-04T08:16:51.376652Z",
     "iopub.status.idle": "2021-07-04T08:16:51.379875Z",
     "shell.execute_reply": "2021-07-04T08:16:51.380804Z"
    },
    "id": "tBHtwlqMnFSu"
   },
   "outputs": [],
   "source": [
    "def my_better_softplus(z):\n",
    "    return tf.where(z > 30., z, tf.math.log(tf.exp(z) + 1.))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.389760Z",
     "iopub.status.busy": "2021-07-04T08:16:51.388584Z",
     "iopub.status.idle": "2021-07-04T08:16:51.403887Z",
     "shell.execute_reply": "2021-07-04T08:16:51.402464Z"
    },
    "id": "mkpIj-COnFSu",
    "outputId": "e7e0cfa5-cbe6-43a0-f9e9-5687ed90477d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(1,), dtype=float32, numpy=array([1000.], dtype=float32)>,\n",
       " [<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>])"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable([1000.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_better_softplus(x)\n",
    "\n",
    "z, tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XI8B76oxnFSu"
   },
   "source": [
    "# 사용자 정의 훈련 반복"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.412781Z",
     "iopub.status.busy": "2021-07-04T08:16:51.411671Z",
     "iopub.status.idle": "2021-07-04T08:16:51.424600Z",
     "shell.execute_reply": "2021-07-04T08:16:51.425708Z"
    },
    "id": "ANdypoJYnFSu"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.446035Z",
     "iopub.status.busy": "2021-07-04T08:16:51.444592Z",
     "iopub.status.idle": "2021-07-04T08:16:51.450062Z",
     "shell.execute_reply": "2021-07-04T08:16:51.451073Z"
    },
    "id": "CnTqizVwnFSu"
   },
   "outputs": [],
   "source": [
    "l2_reg = keras.regularizers.l2(0.05)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=l2_reg),\n",
    "    keras.layers.Dense(1, kernel_regularizer=l2_reg)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.459636Z",
     "iopub.status.busy": "2021-07-04T08:16:51.458431Z",
     "iopub.status.idle": "2021-07-04T08:16:51.462294Z",
     "shell.execute_reply": "2021-07-04T08:16:51.463223Z"
    },
    "id": "wFdg6eRenFSv"
   },
   "outputs": [],
   "source": [
    "def random_batch(X, y, batch_size=32):\n",
    "    idx = np.random.randint(len(X), size=batch_size)\n",
    "    return X[idx], y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.475177Z",
     "iopub.status.busy": "2021-07-04T08:16:51.474083Z",
     "iopub.status.idle": "2021-07-04T08:16:51.477820Z",
     "shell.execute_reply": "2021-07-04T08:16:51.478652Z"
    },
    "id": "LSqc9FVNnFSv"
   },
   "outputs": [],
   "source": [
    "def print_status_bar(iteration, total, loss, metrics=None):\n",
    "    metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result())\n",
    "                         for m in [loss] + (metrics or [])])\n",
    "    end = \"\" if iteration < total else \"\\n\"\n",
    "    print(\"\\r{}/{} - \".format(iteration, total) + metrics,\n",
    "          end=end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:51.499573Z",
     "iopub.status.busy": "2021-07-04T08:16:51.498167Z",
     "iopub.status.idle": "2021-07-04T08:16:54.379440Z",
     "shell.execute_reply": "2021-07-04T08:16:54.380763Z"
    },
    "id": "I-sdMEKxnFSv",
    "outputId": "9d98b783-ffa7-4d12-df12-d961f7a501a0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 - loss: 0.0900 - mean_square: 858.5000\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "mean_loss = keras.metrics.Mean(name=\"loss\")\n",
    "mean_square = keras.metrics.Mean(name=\"mean_square\")\n",
    "for i in range(1, 50 + 1):\n",
    "    loss = 1 / i\n",
    "    mean_loss(loss)\n",
    "    mean_square(i ** 2)\n",
    "    print_status_bar(i, 50, mean_loss, [mean_square])\n",
    "    time.sleep(0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T74QlMxbnFSv"
   },
   "source": [
    "A fancier version with a progress bar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:54.394881Z",
     "iopub.status.busy": "2021-07-04T08:16:54.393436Z",
     "iopub.status.idle": "2021-07-04T08:16:54.396503Z",
     "shell.execute_reply": "2021-07-04T08:16:54.397488Z"
    },
    "id": "JCoqZPL1nFSv"
   },
   "outputs": [],
   "source": [
    "def progress_bar(iteration, total, size=30):\n",
    "    running = iteration < total\n",
    "    c = \">\" if running else \"=\"\n",
    "    p = (size - 1) * iteration // total\n",
    "    fmt = \"{{:-{}d}}/{{}} [{{}}]\".format(len(str(total)))\n",
    "    params = [iteration, total, \"=\" * p + c + \".\" * (size - p - 1)]\n",
    "    return fmt.format(*params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:54.405527Z",
     "iopub.status.busy": "2021-07-04T08:16:54.404360Z",
     "iopub.status.idle": "2021-07-04T08:16:54.408663Z",
     "shell.execute_reply": "2021-07-04T08:16:54.409533Z"
    },
    "id": "palDPRM_nFSv",
    "outputId": "2038faef-dd24-49ab-d42e-72169e9f492c"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "' 3500/10000 [=>....]'"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "progress_bar(3500, 10000, size=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:54.420702Z",
     "iopub.status.busy": "2021-07-04T08:16:54.419667Z",
     "iopub.status.idle": "2021-07-04T08:16:54.423451Z",
     "shell.execute_reply": "2021-07-04T08:16:54.424317Z"
    },
    "id": "_INP4LY-nFSv"
   },
   "outputs": [],
   "source": [
    "def print_status_bar(iteration, total, loss, metrics=None, size=30):\n",
    "    metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result())\n",
    "                         for m in [loss] + (metrics or [])])\n",
    "    end = \"\" if iteration < total else \"\\n\"\n",
    "    print(\"\\r{} - {}\".format(progress_bar(iteration, total), metrics), end=end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:54.451621Z",
     "iopub.status.busy": "2021-07-04T08:16:54.442475Z",
     "iopub.status.idle": "2021-07-04T08:16:57.309328Z",
     "shell.execute_reply": "2021-07-04T08:16:57.310825Z"
    },
    "id": "_wv5QF3bnFSv",
    "outputId": "91d9da4b-8e24-41c5-e4ac-903f69a0c524"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - loss: 0.0900 - mean_square: 858.5000\n"
     ]
    }
   ],
   "source": [
    "mean_loss = keras.metrics.Mean(name=\"loss\")\n",
    "mean_square = keras.metrics.Mean(name=\"mean_square\")\n",
    "for i in range(1, 50 + 1):\n",
    "    loss = 1 / i\n",
    "    mean_loss(loss)\n",
    "    mean_square(i ** 2)\n",
    "    print_status_bar(i, 50, mean_loss, [mean_square])\n",
    "    time.sleep(0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:57.320138Z",
     "iopub.status.busy": "2021-07-04T08:16:57.318748Z",
     "iopub.status.idle": "2021-07-04T08:16:57.326326Z",
     "shell.execute_reply": "2021-07-04T08:16:57.327156Z"
    },
    "id": "GY73UpFLnFSv"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:57.344821Z",
     "iopub.status.busy": "2021-07-04T08:16:57.343764Z",
     "iopub.status.idle": "2021-07-04T08:16:57.360548Z",
     "shell.execute_reply": "2021-07-04T08:16:57.362034Z"
    },
    "id": "PG-ZQpginFSv",
    "outputId": "6d9d9693-adcc-4dac-aec0-199d095b4e8d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=0.01)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.MeanAbsoluteError()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:16:57.381663Z",
     "iopub.status.busy": "2021-07-04T08:16:57.380524Z",
     "iopub.status.idle": "2021-07-04T08:17:33.871593Z",
     "shell.execute_reply": "2021-07-04T08:17:33.872920Z"
    },
    "id": "ITOnR9uLnFSv",
    "outputId": "b0a13b06-c286-46c6-f832-791df6a47ef5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "11610/11610 [==============================] - mean: 1.3955 - mean_absolute_error: 0.5722\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - mean: 0.6774 - mean_absolute_error: 0.5280\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - mean: 0.6351 - mean_absolute_error: 0.5177\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - mean: 0.6384 - mean_absolute_error: 0.5181\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - mean: 0.6440 - mean_absolute_error: 0.5222\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(1, n_epochs + 1):\n",
    "    print(\"Epoch {}/{}\".format(epoch, n_epochs))\n",
    "    for step in range(1, n_steps + 1):\n",
    "        X_batch, y_batch = random_batch(X_train_scaled, y_train)\n",
    "        with tf.GradientTape() as tape:\n",
    "            y_pred = model(X_batch)\n",
    "            main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "            loss = tf.add_n([main_loss] + model.losses)\n",
    "        gradients = tape.gradient(loss, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "        for variable in model.variables:\n",
    "            if variable.constraint is not None:\n",
    "                variable.assign(variable.constraint(variable))\n",
    "        mean_loss(loss)\n",
    "        for metric in metrics:\n",
    "            metric(y_batch, y_pred)\n",
    "        print_status_bar(step * batch_size, len(y_train), mean_loss, metrics)\n",
    "    print_status_bar(len(y_train), len(y_train), mean_loss, metrics)\n",
    "    for metric in [mean_loss] + metrics:\n",
    "        metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209,
     "referenced_widgets": [
      "bcf308ed877d40cabcca590040b19db9",
      "0550b859d58c4b21b730ef45b52a97f2",
      "6dfcfab30ebf42bfa4e8f99cb6654f37",
      "2a52efd06cbd4bfb8e5dbbe17c6236ab",
      "503dcfb1aa1a47cca810d3ae18347faf",
      "b442291bc2fd4afeb00345c601401a48",
      "d00ac2178fda4662bac2b5894225e695",
      "d40d0e4d6dbc4fe7837af600a714baf5",
      "15750aba26bc47b1b03a133a4cadad3a",
      "844cea147d9d46c081ade1ef0e117c72",
      "d0295bc6f1bb4ee38d7ac9244ee54d8e",
      "b531e026313f4933943884f6e1e795f5",
      "fb52f438720d42e7bcf99a27ad7d2eb7",
      "e09a357ca8bf49d1b9da060f8eb29224",
      "5f29d13ac54e4d968770d64d1fad99aa",
      "306c5a5eb7dd4afdaab6886e31e99a02",
      "b6a618847516464680c2a16aef583371",
      "75d93ec1d61648679862d7ddee5547b9",
      "4abc8ad70e6d4112a18aa58394942015",
      "4bf89d5e43274887bd1724a2d305e0a4",
      "6cdc302c227b4da9a76120662368e6b8",
      "8c863503d34b45ebbcb9190940434880",
      "c715ff27d24e45f9b4aacede06814708",
      "cdd33bdc83ef4ba08e7aaf223624f89a",
      "d0e360ece5814c02ac20a2fe42e5c66f",
      "51a6f9642c7e45408f3d984b6a25344a",
      "be5e3d53ea4144539386c29ffef07a3d",
      "f1e80d88677d40f1bdf9ffcf4fe649d8",
      "1066152b45a7421393c6f2110a338fcb",
      "48a18bc2cb684c1b923a8c1da67238fc",
      "0bd851e4cb03463985643ae9f4c37aa7",
      "b5dadf6cb1e8404bb156b17058f73f4a",
      "c80da36d70744be0b6343dc7b5f24f53",
      "ab5fb1fc8fe7467495e0ee03f6839479",
      "d4e99b5a737f4a93a8ad75d870dc485a",
      "d6384c2670af484a8d0ce6019929b947",
      "a4d0b9ed8c1b44819157609d8faa8b5d",
      "4dc3605db14b461b8fd91761de341563",
      "df7ed54f99b54d2abff6b5bdde6db72e",
      "645ace54d7b9427aabe2b2dfa6dc6dea",
      "cf960f823e2741b4b0117ff9eb298414",
      "f8cbaeb84c544413b44be1717d723be2",
      "d5eff707caaa4775a0a2751fec7f1eba",
      "0797d34c1de449309ddbaebb42e85c07",
      "74bea35b82934882b9098623f450fab7",
      "e216a70f79254c9a864c952d1cb6f7f4",
      "252a5e08473f4751a6b63bda7ce5055d",
      "0b1a5bde488248e483f211a75e50b9a7",
      "df2eb47bbf0c47dfbcc6c304733ebcf6",
      "eb43a6e971134e57b35e636ac843b377",
      "40e7a51a0db544ea80d4f5e3805a2a0d",
      "b8ab2b969e9641b98c8934443cb199c7",
      "004faff048744caf8c45ec588d15c83c",
      "434a0e19af3d45cab0e00916a6f1d504",
      "0af626a080824117a0e04705562e6418",
      "3697fc6dc6c34b8fbca2c522d3fe0543",
      "d7b0f08b1a8241ae8f5857893c7e6196",
      "f087a4deddfb4b13b39d053bc39ed254",
      "4d85525433584eb4ae759bc56f0332eb",
      "2d1ae6fa9fe146a9a288fd5ea41667a2",
      "4fb6a7ada2cf4ab9812a62110edd7169",
      "19386467e8344d93b420b0f8c2483400",
      "8f9ea35424a049e2826fcd12e782091b",
      "b20bc4ddb0004894b3fb69a7dc5a6b37",
      "6157ade5095b4bda96075e2b5f51df95",
      "e3c0df173966467db5fb15ded865d25b"
     ]
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:17:33.898309Z",
     "iopub.status.busy": "2021-07-04T08:17:33.896576Z",
     "iopub.status.idle": "2021-07-04T08:18:11.822625Z",
     "shell.execute_reply": "2021-07-04T08:18:11.820044Z"
    },
    "id": "_VOoVK34nFSw",
    "outputId": "7bcb9f91-cced-4c12-e04c-59faa1778280"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bcf308ed877d40cabcca590040b19db9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "All epochs:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b531e026313f4933943884f6e1e795f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 1/5:   0%|          | 0/362 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c715ff27d24e45f9b4aacede06814708",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 2/5:   0%|          | 0/362 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab5fb1fc8fe7467495e0ee03f6839479",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 3/5:   0%|          | 0/362 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74bea35b82934882b9098623f450fab7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 4/5:   0%|          | 0/362 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3697fc6dc6c34b8fbca2c522d3fe0543",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 5/5:   0%|          | 0/362 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    from tqdm.notebook import trange\n",
    "    from collections import OrderedDict\n",
    "    with trange(1, n_epochs + 1, desc=\"All epochs\") as epochs:\n",
    "        for epoch in epochs:\n",
    "            with trange(1, n_steps + 1, desc=\"Epoch {}/{}\".format(epoch, n_epochs)) as steps:\n",
    "                for step in steps:\n",
    "                    X_batch, y_batch = random_batch(X_train_scaled, y_train)\n",
    "                    with tf.GradientTape() as tape:\n",
    "                        y_pred = model(X_batch)\n",
    "                        main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                        loss = tf.add_n([main_loss] + model.losses)\n",
    "                    gradients = tape.gradient(loss, model.trainable_variables)\n",
    "                    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "                    for variable in model.variables:\n",
    "                        if variable.constraint is not None:\n",
    "                            variable.assign(variable.constraint(variable))                    \n",
    "                    status = OrderedDict()\n",
    "                    mean_loss(loss)\n",
    "                    status[\"loss\"] = mean_loss.result().numpy()\n",
    "                    for metric in metrics:\n",
    "                        metric(y_batch, y_pred)\n",
    "                        status[metric.name] = metric.result().numpy()\n",
    "                    steps.set_postfix(status)\n",
    "            for metric in [mean_loss] + metrics:\n",
    "                metric.reset_states()\n",
    "except ImportError as ex:\n",
    "    print(\"To run this cell, please install tqdm, ipywidgets and restart Jupyter\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CnRi0O4YnFSw"
   },
   "source": [
    "## 텐서플로 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.830499Z",
     "iopub.status.busy": "2021-07-04T08:18:11.829618Z",
     "iopub.status.idle": "2021-07-04T08:18:11.833538Z",
     "shell.execute_reply": "2021-07-04T08:18:11.834319Z"
    },
    "id": "Z4ZR1OPsnFSx"
   },
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    return x ** 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.842745Z",
     "iopub.status.busy": "2021-07-04T08:18:11.841606Z",
     "iopub.status.idle": "2021-07-04T08:18:11.846105Z",
     "shell.execute_reply": "2021-07-04T08:18:11.846996Z"
    },
    "id": "8l01IQPXnFSx",
    "outputId": "1ed93f76-dc77-44d6-e02b-0d2f4dd0eeca"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cube(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.854139Z",
     "iopub.status.busy": "2021-07-04T08:18:11.853046Z",
     "iopub.status.idle": "2021-07-04T08:18:11.861946Z",
     "shell.execute_reply": "2021-07-04T08:18:11.861036Z"
    },
    "id": "i7FxLva0nFSx",
    "outputId": "c9f65ec5-995d-464a-8835-e1cf7a6dbd21"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cube(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.871190Z",
     "iopub.status.busy": "2021-07-04T08:18:11.869990Z",
     "iopub.status.idle": "2021-07-04T08:18:11.875579Z",
     "shell.execute_reply": "2021-07-04T08:18:11.874509Z"
    },
    "id": "T03G4Mt8nFSx",
    "outputId": "eb48f1b7-91a0-4558-b7fd-87a72160a584"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.eager.def_function.Function at 0x7fd479c64b10>"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube = tf.function(cube)\n",
    "tf_cube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.906924Z",
     "iopub.status.busy": "2021-07-04T08:18:11.882671Z",
     "iopub.status.idle": "2021-07-04T08:18:11.967643Z",
     "shell.execute_reply": "2021-07-04T08:18:11.968947Z"
    },
    "id": "1FjYnl2snFSx",
    "outputId": "5ce6e3b9-c3dc-4687-f810-b2003cbd1555"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=8>"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:11.978396Z",
     "iopub.status.busy": "2021-07-04T08:18:11.976707Z",
     "iopub.status.idle": "2021-07-04T08:18:11.994754Z",
     "shell.execute_reply": "2021-07-04T08:18:11.995845Z"
    },
    "id": "CJ98sHrKnFSx",
    "outputId": "f6cf2a50-4345-4c52-caef-169b2f08d19c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "69nqOhgSnFSx"
   },
   "source": [
    "### TF 함수와 콘크리트 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.005258Z",
     "iopub.status.busy": "2021-07-04T08:18:12.003881Z",
     "iopub.status.idle": "2021-07-04T08:18:12.008314Z",
     "shell.execute_reply": "2021-07-04T08:18:12.009204Z"
    },
    "id": "TKtHslXenFSx",
    "outputId": "ba36d1c1-5305-486a-a8ab-69699c0b3fe5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.func_graph.FuncGraph at 0x7fd479280850>"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function = tf_cube.get_concrete_function(tf.constant(2.0))\n",
    "concrete_function.graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.016194Z",
     "iopub.status.busy": "2021-07-04T08:18:12.015274Z",
     "iopub.status.idle": "2021-07-04T08:18:12.021028Z",
     "shell.execute_reply": "2021-07-04T08:18:12.021740Z"
    },
    "id": "_rPVLwFKnFSx",
    "outputId": "02127582-2606-4521-f4ed-36384f0f33fd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.029029Z",
     "iopub.status.busy": "2021-07-04T08:18:12.028140Z",
     "iopub.status.idle": "2021-07-04T08:18:12.032917Z",
     "shell.execute_reply": "2021-07-04T08:18:12.033810Z"
    },
    "id": "Vvpikc9TnFSx",
    "outputId": "f26069b6-22c4-4351-d186-c758751df25d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function is tf_cube.get_concrete_function(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3D-erxOBnFSx"
   },
   "source": [
    "### 함수 정의와 그래프"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.041193Z",
     "iopub.status.busy": "2021-07-04T08:18:12.040072Z",
     "iopub.status.idle": "2021-07-04T08:18:12.044509Z",
     "shell.execute_reply": "2021-07-04T08:18:12.045398Z"
    },
    "id": "_IvdEuL2nFSy",
    "outputId": "e7f2706a-82e1-44c1-f951-596f220b86ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.func_graph.FuncGraph at 0x7fd479280850>"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.053175Z",
     "iopub.status.busy": "2021-07-04T08:18:12.052119Z",
     "iopub.status.idle": "2021-07-04T08:18:12.056725Z",
     "shell.execute_reply": "2021-07-04T08:18:12.057552Z"
    },
    "id": "G6izh2CRnFSy",
    "outputId": "749caa01-b752-4265-a212-372fc096e25d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'pow/y' type=Const>,\n",
       " <tf.Operation 'pow' type=Pow>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ops = concrete_function.graph.get_operations()\n",
    "ops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.065323Z",
     "iopub.status.busy": "2021-07-04T08:18:12.064271Z",
     "iopub.status.idle": "2021-07-04T08:18:12.068512Z",
     "shell.execute_reply": "2021-07-04T08:18:12.069401Z"
    },
    "id": "AoSFwR_ynFSy",
    "outputId": "fdd94e28-e51a-4c02-c32a-9b6c659135e1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'x:0' shape=() dtype=float32>,\n",
       " <tf.Tensor 'pow/y:0' shape=() dtype=float32>]"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pow_op = ops[2]\n",
    "list(pow_op.inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.076592Z",
     "iopub.status.busy": "2021-07-04T08:18:12.075541Z",
     "iopub.status.idle": "2021-07-04T08:18:12.080084Z",
     "shell.execute_reply": "2021-07-04T08:18:12.080859Z"
    },
    "id": "9Fzk_02ynFSy",
    "outputId": "7d7f5203-6fe3-42d2-9467-632c972e8b15"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'pow:0' shape=() dtype=float32>]"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pow_op.outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.088380Z",
     "iopub.status.busy": "2021-07-04T08:18:12.087383Z",
     "iopub.status.idle": "2021-07-04T08:18:12.091856Z",
     "shell.execute_reply": "2021-07-04T08:18:12.092684Z"
    },
    "id": "aGRFv-VxnFSy",
    "outputId": "380b405d-5e63-4615-a49a-279199d6522e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Operation 'x' type=Placeholder>"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph.get_operation_by_name('x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.101366Z",
     "iopub.status.busy": "2021-07-04T08:18:12.099700Z",
     "iopub.status.idle": "2021-07-04T08:18:12.105564Z",
     "shell.execute_reply": "2021-07-04T08:18:12.106993Z"
    },
    "id": "COu4KwZSnFSy",
    "outputId": "2e1730e8-18ab-4ce0-fc8c-2d302ea248aa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Identity:0' shape=() dtype=float32>"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph.get_tensor_by_name('Identity:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.116483Z",
     "iopub.status.busy": "2021-07-04T08:18:12.114823Z",
     "iopub.status.idle": "2021-07-04T08:18:12.119524Z",
     "shell.execute_reply": "2021-07-04T08:18:12.120359Z"
    },
    "id": "eJy4X6jbnFSy",
    "outputId": "428aa840-dcdc-4e48-fc74-97e458373e99"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"__inference_cube_1065956\"\n",
       "input_arg {\n",
       "  name: \"x\"\n",
       "  type: DT_FLOAT\n",
       "}\n",
       "output_arg {\n",
       "  name: \"identity\"\n",
       "  type: DT_FLOAT\n",
       "}"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.function_def.signature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9Nh2laLOnFSy"
   },
   "source": [
    "### TF 함수가 계산 그래프를 추출하기 위해 파이썬 함수를 트레이싱하는 방법"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.129008Z",
     "iopub.status.busy": "2021-07-04T08:18:12.128011Z",
     "iopub.status.idle": "2021-07-04T08:18:12.131970Z",
     "shell.execute_reply": "2021-07-04T08:18:12.132795Z"
    },
    "id": "SZK5xGMGnFSy"
   },
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def tf_cube(x):\n",
    "    print(\"print:\", x)\n",
    "    return x ** 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.146742Z",
     "iopub.status.busy": "2021-07-04T08:18:12.139776Z",
     "iopub.status.idle": "2021-07-04T08:18:12.210946Z",
     "shell.execute_reply": "2021-07-04T08:18:12.211878Z"
    },
    "id": "kZa67mMWnFSy",
    "outputId": "da0c33bf-94b3-4889-82a8-ad182f5f7326"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print: Tensor(\"x:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "result = tf_cube(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.218833Z",
     "iopub.status.busy": "2021-07-04T08:18:12.217592Z",
     "iopub.status.idle": "2021-07-04T08:18:12.222986Z",
     "shell.execute_reply": "2021-07-04T08:18:12.223763Z"
    },
    "id": "0MW_WncUnFSy",
    "outputId": "eee3f3d4-a620-4359-f7b3-da5c47d8752b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.235570Z",
     "iopub.status.busy": "2021-07-04T08:18:12.233860Z",
     "iopub.status.idle": "2021-07-04T08:18:12.290992Z",
     "shell.execute_reply": "2021-07-04T08:18:12.291999Z"
    },
    "id": "N4ejN4cKnFSz",
    "outputId": "432b2eab-eeef-4894-ba6e-a7c32e5ff29c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print: 2\n",
      "print: 3\n",
      "print: Tensor(\"x:0\", shape=(1, 2), dtype=float32)\n",
      "print: Tensor(\"x:0\", shape=(2, 2), dtype=float32)\n",
      "WARNING:tensorflow:5 out of the last 5 calls to <function tf_cube at 0x7fd47a20fd40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:5 out of the last 5 calls to <function tf_cube at 0x7fd47a20fd40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print: Tensor(\"x:0\", shape=(3, 2), dtype=float32)\n",
      "WARNING:tensorflow:6 out of the last 6 calls to <function tf_cube at 0x7fd47a20fd40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:6 out of the last 6 calls to <function tf_cube at 0x7fd47a20fd40> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.\n"
     ]
    }
   ],
   "source": [
    "result = tf_cube(2)\n",
    "result = tf_cube(3)\n",
    "result = tf_cube(tf.constant([[1., 2.]])) # New shape: trace!\n",
    "result = tf_cube(tf.constant([[3., 4.], [5., 6.]])) # New shape: trace!\n",
    "result = tf_cube(tf.constant([[7., 8.], [9., 10.], [11., 12.]])) # New shape: trace!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OtxmwBibnFSz"
   },
   "source": [
    "특정 입력 시그니처를 지정하는 것도 가능합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.302653Z",
     "iopub.status.busy": "2021-07-04T08:18:12.301378Z",
     "iopub.status.idle": "2021-07-04T08:18:12.304438Z",
     "shell.execute_reply": "2021-07-04T08:18:12.305373Z"
    },
    "id": "ilXpmEtbnFSz"
   },
   "outputs": [],
   "source": [
    "@tf.function(input_signature=[tf.TensorSpec([None, 28, 28], tf.float32)])\n",
    "def shrink(images):\n",
    "    print(\"트레이싱\", images)\n",
    "    return images[:, ::2, ::2] # 행과 열의 절반을 버립니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.313289Z",
     "iopub.status.busy": "2021-07-04T08:18:12.312167Z",
     "iopub.status.idle": "2021-07-04T08:18:12.321014Z",
     "shell.execute_reply": "2021-07-04T08:18:12.320129Z"
    },
    "id": "_SaQ3wV8nFSz"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.331311Z",
     "iopub.status.busy": "2021-07-04T08:18:12.330246Z",
     "iopub.status.idle": "2021-07-04T08:18:12.411510Z",
     "shell.execute_reply": "2021-07-04T08:18:12.412508Z"
    },
    "id": "rnZJvhf3nFSz",
    "outputId": "26fca1a2-c0cd-45ea-a39e-6f8b57a519b6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "트레이싱 Tensor(\"images:0\", shape=(None, 28, 28), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "img_batch_1 = tf.random.uniform(shape=[100, 28, 28])\n",
    "img_batch_2 = tf.random.uniform(shape=[50, 28, 28])\n",
    "preprocessed_images = shrink(img_batch_1) # 함수 트레이싱\n",
    "preprocessed_images = shrink(img_batch_2) # 동일한 콘크리트 함수 재사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.421961Z",
     "iopub.status.busy": "2021-07-04T08:18:12.420578Z",
     "iopub.status.idle": "2021-07-04T08:18:12.427046Z",
     "shell.execute_reply": "2021-07-04T08:18:12.427918Z"
    },
    "id": "g0a7ieyFnFSz",
    "outputId": "18258769-b77b-4d48-ce1e-7457a7a90821"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python inputs incompatible with input_signature:\n",
      "  inputs: (\n",
      "    tf.Tensor(\n",
      "[[[0.7413678  0.62854624]\n",
      "  [0.01738465 0.3431449 ]]\n",
      "\n",
      " [[0.51063764 0.3777541 ]\n",
      "  [0.07321596 0.02137029]]], shape=(2, 2, 2), dtype=float32))\n",
      "  input_signature: (\n",
      "    TensorSpec(shape=(None, 28, 28), dtype=tf.float32, name=None))\n"
     ]
    }
   ],
   "source": [
    "img_batch_3 = tf.random.uniform(shape=[2, 2, 2])\n",
    "try:\n",
    "    preprocessed_images = shrink(img_batch_3)  # 다른 타입이나 크기 거부\n",
    "except ValueError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iGEDLTpVnFSz"
   },
   "source": [
    "### 오토그래프를 사용해 제어 흐름 나타내기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tEAizr5BnFSz"
   },
   "source": [
    "`range()`를  사용한 정적인 `for` 반복:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.437138Z",
     "iopub.status.busy": "2021-07-04T08:18:12.436013Z",
     "iopub.status.idle": "2021-07-04T08:18:12.439787Z",
     "shell.execute_reply": "2021-07-04T08:18:12.440708Z"
    },
    "id": "XF0hg7XcnFSz"
   },
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in range(10):\n",
    "        x += 1\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.453379Z",
     "iopub.status.busy": "2021-07-04T08:18:12.448072Z",
     "iopub.status.idle": "2021-07-04T08:18:12.573527Z",
     "shell.execute_reply": "2021-07-04T08:18:12.574425Z"
    },
    "id": "_C_KpXXAnFSz",
    "outputId": "ffe73dbb-0a5e-4eb6-f2c0-731a140b6061"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=15>"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10(tf.constant(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.585088Z",
     "iopub.status.busy": "2021-07-04T08:18:12.583963Z",
     "iopub.status.idle": "2021-07-04T08:18:12.589061Z",
     "shell.execute_reply": "2021-07-04T08:18:12.590033Z"
    },
    "id": "ytBDurUinFSz",
    "outputId": "2a1da7fe-cfca-4344-b09c-1b9ac44df20e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'add/y' type=Const>,\n",
       " <tf.Operation 'add' type=AddV2>,\n",
       " <tf.Operation 'add_1/y' type=Const>,\n",
       " <tf.Operation 'add_1' type=AddV2>,\n",
       " <tf.Operation 'add_2/y' type=Const>,\n",
       " <tf.Operation 'add_2' type=AddV2>,\n",
       " <tf.Operation 'add_3/y' type=Const>,\n",
       " <tf.Operation 'add_3' type=AddV2>,\n",
       " <tf.Operation 'add_4/y' type=Const>,\n",
       " <tf.Operation 'add_4' type=AddV2>,\n",
       " <tf.Operation 'add_5/y' type=Const>,\n",
       " <tf.Operation 'add_5' type=AddV2>,\n",
       " <tf.Operation 'add_6/y' type=Const>,\n",
       " <tf.Operation 'add_6' type=AddV2>,\n",
       " <tf.Operation 'add_7/y' type=Const>,\n",
       " <tf.Operation 'add_7' type=AddV2>,\n",
       " <tf.Operation 'add_8/y' type=Const>,\n",
       " <tf.Operation 'add_8' type=AddV2>,\n",
       " <tf.Operation 'add_9/y' type=Const>,\n",
       " <tf.Operation 'add_9' type=AddV2>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10.get_concrete_function(tf.constant(5)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cNEX0KW4nFSz"
   },
   "source": [
    "`tf.while_loop()`를 사용한 동적인 반복:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.602151Z",
     "iopub.status.busy": "2021-07-04T08:18:12.600905Z",
     "iopub.status.idle": "2021-07-04T08:18:12.604577Z",
     "shell.execute_reply": "2021-07-04T08:18:12.605453Z"
    },
    "id": "WmjQ3HJWnFSz"
   },
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def add_10(x):\n",
    "    condition = lambda i, x: tf.less(i, 10)\n",
    "    body = lambda i, x: (tf.add(i, 1), tf.add(x, 1))\n",
    "    final_i, final_x = tf.while_loop(condition, body, [tf.constant(0), x])\n",
    "    return final_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.613134Z",
     "iopub.status.busy": "2021-07-04T08:18:12.612020Z",
     "iopub.status.idle": "2021-07-04T08:18:12.782863Z",
     "shell.execute_reply": "2021-07-04T08:18:12.784125Z"
    },
    "id": "MTYxWdZanFS0",
    "outputId": "3884b63f-6298-4d96-b5c9-8cf13ee507cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=15>"
      ]
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10(tf.constant(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.794353Z",
     "iopub.status.busy": "2021-07-04T08:18:12.793301Z",
     "iopub.status.idle": "2021-07-04T08:18:12.797961Z",
     "shell.execute_reply": "2021-07-04T08:18:12.798894Z"
    },
    "id": "7qYJYD_cnFS0",
    "outputId": "3dc3c05c-4f46-48fb-fdee-df40a8299933"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'Const' type=Const>,\n",
       " <tf.Operation 'while/maximum_iterations' type=Const>,\n",
       " <tf.Operation 'while/loop_counter' type=Const>,\n",
       " <tf.Operation 'while' type=StatelessWhile>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 234,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10.get_concrete_function(tf.constant(5)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oEcvIYTrnFS0"
   },
   "source": [
    "(오토그래프에 의한) `tf.range()`를 사용한 동적인 `for` 반복:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.808335Z",
     "iopub.status.busy": "2021-07-04T08:18:12.807158Z",
     "iopub.status.idle": "2021-07-04T08:18:12.810195Z",
     "shell.execute_reply": "2021-07-04T08:18:12.811112Z"
    },
    "id": "s5E-YpJonFS0"
   },
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in tf.range(10):\n",
    "        x = x + 1\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.819555Z",
     "iopub.status.busy": "2021-07-04T08:18:12.818460Z",
     "iopub.status.idle": "2021-07-04T08:18:12.935210Z",
     "shell.execute_reply": "2021-07-04T08:18:12.936071Z"
    },
    "id": "kxac-wYhnFS0",
    "outputId": "2733f1fa-b174-430c-9e49-75b8e85d555a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'range/start' type=Const>,\n",
       " <tf.Operation 'range/limit' type=Const>,\n",
       " <tf.Operation 'range/delta' type=Const>,\n",
       " <tf.Operation 'range' type=Range>,\n",
       " <tf.Operation 'sub' type=Sub>,\n",
       " <tf.Operation 'floordiv' type=FloorDiv>,\n",
       " <tf.Operation 'mod' type=FloorMod>,\n",
       " <tf.Operation 'zeros_like' type=Const>,\n",
       " <tf.Operation 'NotEqual' type=NotEqual>,\n",
       " <tf.Operation 'Cast' type=Cast>,\n",
       " <tf.Operation 'add' type=AddV2>,\n",
       " <tf.Operation 'zeros_like_1' type=Const>,\n",
       " <tf.Operation 'Maximum' type=Maximum>,\n",
       " <tf.Operation 'while/maximum_iterations' type=Const>,\n",
       " <tf.Operation 'while/loop_counter' type=Const>,\n",
       " <tf.Operation 'while' type=StatelessWhile>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10.get_concrete_function(tf.constant(0)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jKv6EsoMnFS0"
   },
   "source": [
    "### TF 함수에서 변수와 다른 자원 다루기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.943690Z",
     "iopub.status.busy": "2021-07-04T08:18:12.942848Z",
     "iopub.status.idle": "2021-07-04T08:18:12.948164Z",
     "shell.execute_reply": "2021-07-04T08:18:12.949454Z"
    },
    "id": "8lALRLVnnFS0"
   },
   "outputs": [],
   "source": [
    "counter = tf.Variable(0)\n",
    "\n",
    "@tf.function\n",
    "def increment(counter, c=1):\n",
    "    return counter.assign_add(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:12.962500Z",
     "iopub.status.busy": "2021-07-04T08:18:12.957913Z",
     "iopub.status.idle": "2021-07-04T08:18:13.026064Z",
     "shell.execute_reply": "2021-07-04T08:18:13.026933Z"
    },
    "id": "Rb1pt6VInFS0",
    "outputId": "26bddd04-3f11-4f64-e48e-d2e3e669ac74"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=2>"
      ]
     },
     "execution_count": 238,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "increment(counter)\n",
    "increment(counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.034814Z",
     "iopub.status.busy": "2021-07-04T08:18:13.033671Z",
     "iopub.status.idle": "2021-07-04T08:18:13.038954Z",
     "shell.execute_reply": "2021-07-04T08:18:13.037986Z"
    },
    "id": "L2hj3cM9nFS1",
    "outputId": "2f6d7af9-b74c-4430-9aab-0923829a23e7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"counter\"\n",
       "type: DT_RESOURCE"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function_def = increment.get_concrete_function(counter).function_def\n",
    "function_def.signature.input_arg[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.047553Z",
     "iopub.status.busy": "2021-07-04T08:18:13.046451Z",
     "iopub.status.idle": "2021-07-04T08:18:13.050503Z",
     "shell.execute_reply": "2021-07-04T08:18:13.051315Z"
    },
    "id": "6BQEr96bnFS1"
   },
   "outputs": [],
   "source": [
    "counter = tf.Variable(0)\n",
    "\n",
    "@tf.function\n",
    "def increment(c=1):\n",
    "    return counter.assign_add(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.062532Z",
     "iopub.status.busy": "2021-07-04T08:18:13.061357Z",
     "iopub.status.idle": "2021-07-04T08:18:13.124305Z",
     "shell.execute_reply": "2021-07-04T08:18:13.125231Z"
    },
    "id": "Iq0eZORHnFS1",
    "outputId": "ccdc723f-af45-4319-97fc-628a6ef2f273"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=2>"
      ]
     },
     "execution_count": 241,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "increment()\n",
    "increment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.132651Z",
     "iopub.status.busy": "2021-07-04T08:18:13.131477Z",
     "iopub.status.idle": "2021-07-04T08:18:13.135757Z",
     "shell.execute_reply": "2021-07-04T08:18:13.136623Z"
    },
    "id": "m3AP7PgGnFS1",
    "outputId": "5d54dc3a-6b79-4528-8084-d96a8e59bfa9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"assignaddvariableop_resource\"\n",
       "type: DT_RESOURCE"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function_def = increment.get_concrete_function().function_def\n",
    "function_def.signature.input_arg[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.146041Z",
     "iopub.status.busy": "2021-07-04T08:18:13.144979Z",
     "iopub.status.idle": "2021-07-04T08:18:13.148372Z",
     "shell.execute_reply": "2021-07-04T08:18:13.149229Z"
    },
    "id": "ixmvKjGtnFS1"
   },
   "outputs": [],
   "source": [
    "class Counter:\n",
    "    def __init__(self):\n",
    "        self.counter = tf.Variable(0)\n",
    "\n",
    "    @tf.function\n",
    "    def increment(self, c=1):\n",
    "        return self.counter.assign_add(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.156955Z",
     "iopub.status.busy": "2021-07-04T08:18:13.155860Z",
     "iopub.status.idle": "2021-07-04T08:18:13.225991Z",
     "shell.execute_reply": "2021-07-04T08:18:13.227379Z"
    },
    "id": "iE07HYZPnFS1",
    "outputId": "f6a42171-4846-4d73-b1b7-34206ff69773"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=2>"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = Counter()\n",
    "c.increment()\n",
    "c.increment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.239411Z",
     "iopub.status.busy": "2021-07-04T08:18:13.238495Z",
     "iopub.status.idle": "2021-07-04T08:18:13.324707Z",
     "shell.execute_reply": "2021-07-04T08:18:13.323426Z"
    },
    "id": "rDYlFn4xnFS1",
    "outputId": "90c634c6-64ae-4ce6-de19-9e4f7bb8e262",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def tf__add(x):\n",
      "    with ag__.FunctionScope('add_10', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:\n",
      "        do_return = False\n",
      "        retval_ = ag__.UndefinedReturnValue()\n",
      "\n",
      "        def get_state():\n",
      "            return (x,)\n",
      "\n",
      "        def set_state(vars_):\n",
      "            nonlocal x\n",
      "            (x,) = vars_\n",
      "\n",
      "        def loop_body(itr):\n",
      "            nonlocal x\n",
      "            i = itr\n",
      "            x = ag__.ld(x)\n",
      "            x += 1\n",
      "        i = ag__.Undefined('i')\n",
      "        ag__.for_stmt(ag__.converted_call(ag__.ld(tf).range, (10,), None, fscope), None, loop_body, get_state, set_state, ('x',), {'iterate_names': 'i'})\n",
      "        try:\n",
      "            do_return = True\n",
      "            retval_ = ag__.ld(x)\n",
      "        except:\n",
      "            do_return = False\n",
      "            raise\n",
      "        return fscope.ret(retval_, do_return)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in tf.range(10):\n",
    "        x += 1\n",
    "    return x\n",
    "\n",
    "print(tf.autograph.to_code(add_10.python_function))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.335064Z",
     "iopub.status.busy": "2021-07-04T08:18:13.333311Z",
     "iopub.status.idle": "2021-07-04T08:18:13.336906Z",
     "shell.execute_reply": "2021-07-04T08:18:13.338368Z"
    },
    "id": "q5-0PLgtnFS2"
   },
   "outputs": [],
   "source": [
    "def display_tf_code(func):\n",
    "    from IPython.display import display, Markdown\n",
    "    if hasattr(func, \"python_function\"):\n",
    "        func = func.python_function\n",
    "    code = tf.autograph.to_code(func)\n",
    "    display(Markdown('```python\\n{}\\n```'.format(code)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 486
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.351151Z",
     "iopub.status.busy": "2021-07-04T08:18:13.349941Z",
     "iopub.status.idle": "2021-07-04T08:18:13.355816Z",
     "shell.execute_reply": "2021-07-04T08:18:13.356679Z"
    },
    "id": "QLMou_bhnFS2",
    "outputId": "14751d22-48d1-4fcc-b61d-54558a752b81"
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "def tf__add(x):\n",
       "    with ag__.FunctionScope('add_10', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:\n",
       "        do_return = False\n",
       "        retval_ = ag__.UndefinedReturnValue()\n",
       "\n",
       "        def get_state():\n",
       "            return (x,)\n",
       "\n",
       "        def set_state(vars_):\n",
       "            nonlocal x\n",
       "            (x,) = vars_\n",
       "\n",
       "        def loop_body(itr):\n",
       "            nonlocal x\n",
       "            i = itr\n",
       "            x = ag__.ld(x)\n",
       "            x += 1\n",
       "        i = ag__.Undefined('i')\n",
       "        ag__.for_stmt(ag__.converted_call(ag__.ld(tf).range, (10,), None, fscope), None, loop_body, get_state, set_state, ('x',), {'iterate_names': 'i'})\n",
       "        try:\n",
       "            do_return = True\n",
       "            retval_ = ag__.ld(x)\n",
       "        except:\n",
       "            do_return = False\n",
       "            raise\n",
       "        return fscope.ret(retval_, do_return)\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_tf_code(add_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1gFgVxYPnFS2"
   },
   "source": [
    "## tf.keras와 TF 함수를 함께 사용하거나 사용하지 않기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W1lPoDfknFS2"
   },
   "source": [
    "기본적으로 tf.keras는 자동으로 사용자 정의 코드를 TF 함수로 변환하기 때문에 `tf.function()`을 사용할 필요가 없습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.364674Z",
     "iopub.status.busy": "2021-07-04T08:18:13.363574Z",
     "iopub.status.idle": "2021-07-04T08:18:13.366662Z",
     "shell.execute_reply": "2021-07-04T08:18:13.367470Z"
    },
    "id": "W2OWiyhpnFS2"
   },
   "outputs": [],
   "source": [
    "# 사용자 손실 함수\n",
    "def my_mse(y_true, y_pred):\n",
    "    print(\"my_mse() 손실 트레이싱\")\n",
    "    return tf.reduce_mean(tf.square(y_pred - y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.375494Z",
     "iopub.status.busy": "2021-07-04T08:18:13.374453Z",
     "iopub.status.idle": "2021-07-04T08:18:13.377935Z",
     "shell.execute_reply": "2021-07-04T08:18:13.378800Z"
    },
    "id": "-Vw6sZmSnFS2"
   },
   "outputs": [],
   "source": [
    "# 사용자 지표 함수\n",
    "def my_mae(y_true, y_pred):\n",
    "    print(\"my_mae() 지표 트레이싱\")\n",
    "    return tf.reduce_mean(tf.abs(y_pred - y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.395747Z",
     "iopub.status.busy": "2021-07-04T08:18:13.394310Z",
     "iopub.status.idle": "2021-07-04T08:18:13.397704Z",
     "shell.execute_reply": "2021-07-04T08:18:13.398856Z"
    },
    "id": "i31eB7X0nFS2"
   },
   "outputs": [],
   "source": [
    "# 사용자 정의 층\n",
    "class MyDense(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=None, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.activation = keras.activations.get(activation)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.kernel = self.add_weight(name='kernel', \n",
    "                                      shape=(input_shape[1], self.units),\n",
    "                                      initializer='uniform',\n",
    "                                      trainable=True)\n",
    "        self.biases = self.add_weight(name='bias', \n",
    "                                      shape=(self.units,),\n",
    "                                      initializer='zeros',\n",
    "                                      trainable=True)\n",
    "        super().build(input_shape)\n",
    "\n",
    "    def call(self, X):\n",
    "        print(\"MyDense.call() 트레이싱\")\n",
    "        return self.activation(X @ self.kernel + self.biases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.406507Z",
     "iopub.status.busy": "2021-07-04T08:18:13.405319Z",
     "iopub.status.idle": "2021-07-04T08:18:13.410643Z",
     "shell.execute_reply": "2021-07-04T08:18:13.411398Z"
    },
    "id": "guEjwBD3nFS2"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.428336Z",
     "iopub.status.busy": "2021-07-04T08:18:13.427336Z",
     "iopub.status.idle": "2021-07-04T08:18:13.436439Z",
     "shell.execute_reply": "2021-07-04T08:18:13.437426Z"
    },
    "id": "GsUahIc4nFS2"
   },
   "outputs": [],
   "source": [
    "# 사용자 정의 모델\n",
    "class MyModel(keras.models.Model):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden1 = MyDense(30, activation=\"relu\")\n",
    "        self.hidden2 = MyDense(30, activation=\"relu\")\n",
    "        self.output_ = MyDense(1)\n",
    "\n",
    "    def call(self, input):\n",
    "        print(\"MyModel.call() 트레이싱\")\n",
    "        hidden1 = self.hidden1(input)\n",
    "        hidden2 = self.hidden2(hidden1)\n",
    "        concat = keras.layers.concatenate([input, hidden2])\n",
    "        output = self.output_(concat)\n",
    "        return output\n",
    "\n",
    "model = MyModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.455504Z",
     "iopub.status.busy": "2021-07-04T08:18:13.454006Z",
     "iopub.status.idle": "2021-07-04T08:18:13.461364Z",
     "shell.execute_reply": "2021-07-04T08:18:13.460118Z"
    },
    "id": "cXNdmo_ZnFS2"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:13.469768Z",
     "iopub.status.busy": "2021-07-04T08:18:13.468230Z",
     "iopub.status.idle": "2021-07-04T08:18:18.848304Z",
     "shell.execute_reply": "2021-07-04T08:18:18.849388Z"
    },
    "id": "ZiUY_qRjnFS2",
    "outputId": "adb3c763-db1c-4a28-cab4-dc3e6984ae4e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "353/363 [============================>.] - ETA: 0s - loss: 1.3459 - my_mae: 0.7978MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 1.3255 - my_mae: 0.7900 - val_loss: 0.5569 - val_my_mae: 0.4819\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.4419 - my_mae: 0.4767 - val_loss: 0.4664 - val_my_mae: 0.4576\n",
      "162/162 [==============================] - 0s 2ms/step - loss: 0.4164 - my_mae: 0.4639\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4163525104522705, 0.4639028012752533]"
      ]
     },
     "execution_count": 254,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DqKwtqeZnFS2"
   },
   "source": [
    "`dynamic=True`로 모델을 만들어 이 기능을 끌 수 있습니다(또는 모델의 생성자에서 `super().__init__(dynamic=True, **kwargs)`를 호출합니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:18.858643Z",
     "iopub.status.busy": "2021-07-04T08:18:18.857470Z",
     "iopub.status.idle": "2021-07-04T08:18:18.866575Z",
     "shell.execute_reply": "2021-07-04T08:18:18.867385Z"
    },
    "id": "RIT3wTOenFS2"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:18.877147Z",
     "iopub.status.busy": "2021-07-04T08:18:18.875979Z",
     "iopub.status.idle": "2021-07-04T08:18:18.883340Z",
     "shell.execute_reply": "2021-07-04T08:18:18.884263Z"
    },
    "id": "NXF_pU75nFS3"
   },
   "outputs": [],
   "source": [
    "model = MyModel(dynamic=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:18.902153Z",
     "iopub.status.busy": "2021-07-04T08:18:18.900942Z",
     "iopub.status.idle": "2021-07-04T08:18:18.907182Z",
     "shell.execute_reply": "2021-07-04T08:18:18.908345Z"
    },
    "id": "W1Ijplz0nFS3"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PpNoK8_WnFS3"
   },
   "source": [
    "사용자 정의 코드는 반복마다 호출됩니다. 너무 많이 출력되는 것을 피하기 위해 작은 데이터셋으로 훈련, 검증, 평가해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:18.918530Z",
     "iopub.status.busy": "2021-07-04T08:18:18.917237Z",
     "iopub.status.idle": "2021-07-04T08:18:19.275612Z",
     "shell.execute_reply": "2021-07-04T08:18:19.276677Z"
    },
    "id": "clBvdSLJnFS3",
    "outputId": "7e56c770-cca3-4017-855b-06a8113c64d9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[5.507259368896484, 2.0566811561584473]"
      ]
     },
     "execution_count": 258,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled[:64], y_train[:64], epochs=1,\n",
    "          validation_data=(X_valid_scaled[:64], y_valid[:64]), verbose=0)\n",
    "model.evaluate(X_test_scaled[:64], y_test[:64], verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NgR_CLQHnFS3"
   },
   "source": [
    "또는 모델을 컴파일할 때 `run_eagerly=True`를 지정합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.285647Z",
     "iopub.status.busy": "2021-07-04T08:18:19.284388Z",
     "iopub.status.idle": "2021-07-04T08:18:19.289539Z",
     "shell.execute_reply": "2021-07-04T08:18:19.290366Z"
    },
    "id": "dXpvcUGfnFS3"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.300177Z",
     "iopub.status.busy": "2021-07-04T08:18:19.298916Z",
     "iopub.status.idle": "2021-07-04T08:18:19.306169Z",
     "shell.execute_reply": "2021-07-04T08:18:19.307159Z"
    },
    "id": "I6k23xDJnFS3"
   },
   "outputs": [],
   "source": [
    "model = MyModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.323376Z",
     "iopub.status.busy": "2021-07-04T08:18:19.322345Z",
     "iopub.status.idle": "2021-07-04T08:18:19.330227Z",
     "shell.execute_reply": "2021-07-04T08:18:19.328987Z"
    },
    "id": "3L_mdyoknFS3"
   },
   "outputs": [],
   "source": [
    "model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae], run_eagerly=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.339719Z",
     "iopub.status.busy": "2021-07-04T08:18:19.338131Z",
     "iopub.status.idle": "2021-07-04T08:18:19.687021Z",
     "shell.execute_reply": "2021-07-04T08:18:19.688268Z"
    },
    "id": "Zne2esoMnFS3",
    "outputId": "6f5cd521-64d4-4d2e-dae0-c614fd0c2f46"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n",
      "MyModel.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "MyDense.call() 트레이싱\n",
      "my_mse() 손실 트레이싱\n",
      "my_mae() 지표 트레이싱\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[5.507259368896484, 2.0566811561584473]"
      ]
     },
     "execution_count": 262,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled[:64], y_train[:64], epochs=1,\n",
    "          validation_data=(X_valid_scaled[:64], y_valid[:64]), verbose=0)\n",
    "model.evaluate(X_test_scaled[:64], y_test[:64], verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w8yHh6uVnFS3"
   },
   "source": [
    "## 사용자 정의 옵티마이저"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kb-o4KXMnFS3"
   },
   "source": [
    "사용자 정의 옵티마이저를 정의하는 것은 일반적이지 않습니다. 하지만 어쩔 수 없이 만들어야 하는 상황이라면 다음 예를 참고하세요:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.712137Z",
     "iopub.status.busy": "2021-07-04T08:18:19.710793Z",
     "iopub.status.idle": "2021-07-04T08:18:19.713978Z",
     "shell.execute_reply": "2021-07-04T08:18:19.715041Z"
    },
    "id": "cu5eHM8NnFS3"
   },
   "outputs": [],
   "source": [
    "class MyMomentumOptimizer(keras.optimizers.Optimizer):\n",
    "    def __init__(self, learning_rate=0.001, momentum=0.9, name=\"MyMomentumOptimizer\", **kwargs):\n",
    "        \"\"\"super().__init__()를 호출하고 _set_hyper()를 사용해 하이퍼파라미터를 저장합니다\"\"\"\n",
    "        super().__init__(name, **kwargs)\n",
    "        self._set_hyper(\"learning_rate\", kwargs.get(\"lr\", learning_rate)) # lr=learning_rate을 처리\n",
    "        self._set_hyper(\"decay\", self._initial_decay) # \n",
    "        self._set_hyper(\"momentum\", momentum)\n",
    "    \n",
    "    def _create_slots(self, var_list):\n",
    "        \"\"\"모델 파라미터마다 연관된 옵티마이저 변수를 만듭니다.\n",
    "        텐서플로는 이런 옵티마이저 변수를 '슬롯'이라고 부릅니다.\n",
    "        모멘텀 옵티마이저에서는 모델 파라미터마다 하나의 모멘텀 슬롯이 필요합니다.\n",
    "        \"\"\"\n",
    "        for var in var_list:\n",
    "            self.add_slot(var, \"momentum\")\n",
    "\n",
    "    @tf.function\n",
    "    def _resource_apply_dense(self, grad, var):\n",
    "        \"\"\"슬롯을 업데이트하고 모델 파라미터에 대한 옵티마이저 스텝을 수행합니다.\n",
    "        \"\"\"\n",
    "        var_dtype = var.dtype.base_dtype\n",
    "        lr_t = self._decayed_lr(var_dtype) # 학습률 감쇠 처리\n",
    "        momentum_var = self.get_slot(var, \"momentum\")\n",
    "        momentum_hyper = self._get_hyper(\"momentum\", var_dtype)\n",
    "        momentum_var.assign(momentum_var * momentum_hyper - (1. - momentum_hyper)* grad)\n",
    "        var.assign_add(momentum_var * lr_t)\n",
    "\n",
    "    def _resource_apply_sparse(self, grad, var):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {\n",
    "            **base_config,\n",
    "            \"learning_rate\": self._serialize_hyperparameter(\"learning_rate\"),\n",
    "            \"decay\": self._serialize_hyperparameter(\"decay\"),\n",
    "            \"momentum\": self._serialize_hyperparameter(\"momentum\"),\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.723704Z",
     "iopub.status.busy": "2021-07-04T08:18:19.722551Z",
     "iopub.status.idle": "2021-07-04T08:18:19.727526Z",
     "shell.execute_reply": "2021-07-04T08:18:19.728330Z"
    },
    "id": "9a-d3UwinFS3"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:19.741747Z",
     "iopub.status.busy": "2021-07-04T08:18:19.740680Z",
     "iopub.status.idle": "2021-07-04T08:18:24.082650Z",
     "shell.execute_reply": "2021-07-04T08:18:24.083732Z"
    },
    "id": "JPvzY12KnFS3",
    "outputId": "4ba0634f-8c25-4bce-9771-0ee48530b439",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 3.8128\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.4877\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.9162\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.7587\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.7050\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd478dd75d0>"
      ]
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.Sequential([keras.layers.Dense(1, input_shape=[8])])\n",
    "model.compile(loss=\"mse\", optimizer=MyMomentumOptimizer())\n",
    "model.fit(X_train_scaled, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "goUFeTllnFS4"
   },
   "source": [
    "# 연습문제"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FN1NBFYMnFS4"
   },
   "source": [
    "## 1. to 11.\n",
    "부록 A 참조."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "inl1mvaknFS4"
   },
   "source": [
    "## 12. _층 정규화_ 를 수행하는 사용자 정의 층을 구현하세요.\n",
    "\n",
    "_15장에서 순환 신경망을 사용할 때 이런 종류의 층을 사용합니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RVTVssxNnFS4"
   },
   "source": [
    "### a.\n",
    "_문제: `build()` 메서드에서 두 개의 훈련 가능한 가중치 *α*와 *β*를 정의합니다. 두 가중치 모두 크기가 `input_shape[-1:]`이고 데이터 타입은 `tf.float32`입니다. *α*는 1로 초기화되고 *β*는 0으로 초기화되어야 합니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ArXtJ4jznFS4"
   },
   "source": [
    "솔루션: 아래 참조."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OtZf7k4bnFS4"
   },
   "source": [
    "### b.\n",
    "_문제: `call()` 메서드는 샘플의 특성마다 평균 μ와 표준편차 σ를 계산해야 합니다. 이를 위해 전체 샘플의 평균 μ와 분산 σ<sup>2</sup>을 반환하는 `tf.nn.moments(inputs, axes=-1, keepdims=True)`을 사용할 수 있습니다(분산의 제곱근으로 표준편차를 계산합니다). 그다음 *α*⊗(*X* - μ)/(σ + ε) + *β*를 계산하여 반환합니다. 여기에서 ⊗는 원소별\n",
    "곱셈(`*`)을 나타냅니다. ε은 안전을 위한 항입니다(0으로 나누어지는 것을 막기 위한 작은 상수. 예를 들면 0.001)._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:24.104782Z",
     "iopub.status.busy": "2021-07-04T08:18:24.103721Z",
     "iopub.status.idle": "2021-07-04T08:18:24.107336Z",
     "shell.execute_reply": "2021-07-04T08:18:24.108585Z"
    },
    "id": "zaJez0D7nFS4"
   },
   "outputs": [],
   "source": [
    "class LayerNormalization(keras.layers.Layer):\n",
    "    def __init__(self, eps=0.001, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.eps = eps\n",
    "\n",
    "    def build(self, batch_input_shape):\n",
    "        self.alpha = self.add_weight(\n",
    "            name=\"alpha\", shape=batch_input_shape[-1:],\n",
    "            initializer=\"ones\")\n",
    "        self.beta = self.add_weight(\n",
    "            name=\"beta\", shape=batch_input_shape[-1:],\n",
    "            initializer=\"zeros\")\n",
    "        super().build(batch_input_shape) # 반드시 끝에 와야 합니다\n",
    "\n",
    "    def call(self, X):\n",
    "        mean, variance = tf.nn.moments(X, axes=-1, keepdims=True)\n",
    "        return self.alpha * (X - mean) / (tf.sqrt(variance + self.eps)) + self.beta\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return batch_input_shape\n",
    "\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"eps\": self.eps}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZmgaOCRenFS4"
   },
   "source": [
    "_ε_ 하이퍼파라미터(`eps`)는 필수가 아닙니다. 또한 `tf.sqrt(variance) + self.eps` 보다 `tf.sqrt(variance + self.eps)`를 계산하는 것이 좋습니다. sqrt(z)의 도함수는 z=0에서 정의되지 않기 때문에 분산 벡터의 한 원소가 0에 가까우면 훈련이 이리저리 널뜁니다. 제곱근 안에 _ε_를 넣으면 이런 현상을 방지할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-txGpum2nFS4"
   },
   "source": [
    "### c.\n",
    "_문제: 사용자 정의 층이 `keras.layers.LayerNormalization` 층과 동일한(또는 거의 동일한) 출력을 만드는지 확인하세요._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XayuPDh8nFS4"
   },
   "source": [
    "각 클래스의 객체를 만들고 데이터(예를 들면, 훈련 세트)를 적용해 보죠. 차이는 무시할 수 있는 수준입니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:24.118152Z",
     "iopub.status.busy": "2021-07-04T08:18:24.116642Z",
     "iopub.status.idle": "2021-07-04T08:18:26.833501Z",
     "shell.execute_reply": "2021-07-04T08:18:26.834923Z"
    },
    "id": "G-JdGdrpnFS4",
    "outputId": "2b92e9eb-5b06-4f0d-d830-c64ee33e18df"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=3.80914e-08>"
      ]
     },
     "execution_count": 267,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X_train.astype(np.float32)\n",
    "\n",
    "custom_layer_norm = LayerNormalization()\n",
    "keras_layer_norm = keras.layers.LayerNormalization()\n",
    "\n",
    "tf.reduce_mean(keras.losses.mean_absolute_error(\n",
    "    keras_layer_norm(X), custom_layer_norm(X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZhYijbpJnFS5"
   },
   "source": [
    "네 충분히 가깝네요. 조금 더 확실하게 알파와 베타를 완전히 랜덤하게 지정하고 다시 비교해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:26.847454Z",
     "iopub.status.busy": "2021-07-04T08:18:26.846347Z",
     "iopub.status.idle": "2021-07-04T08:18:26.856648Z",
     "shell.execute_reply": "2021-07-04T08:18:26.855676Z"
    },
    "id": "LmRALOLQnFS5",
    "outputId": "256086be-d91d-494a-99f5-bcbae13bdfa3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=1.695759e-08>"
      ]
     },
     "execution_count": 268,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_alpha = np.random.rand(X.shape[-1])\n",
    "random_beta = np.random.rand(X.shape[-1])\n",
    "\n",
    "custom_layer_norm.set_weights([random_alpha, random_beta])\n",
    "keras_layer_norm.set_weights([random_alpha, random_beta])\n",
    "\n",
    "tf.reduce_mean(keras.losses.mean_absolute_error(\n",
    "    keras_layer_norm(X), custom_layer_norm(X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "y7nGTB_bnFS6"
   },
   "source": [
    "여전히 무시할 수 있는 수준입니다! 사용자 정의 층이 잘 동작합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8OldjLnsnFS6"
   },
   "source": [
    "## 13. 사용자 정의 훈련 반복을 사용해 패션 MNIST 데이터셋으로 모델을 훈련해보세요.\n",
    "\n",
    "_패션 MNIST 데이터셋은 10장에서 소개했습니다._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3Sc-mwzpnFS6"
   },
   "source": [
    "### a.\n",
    "_문제: 에포크, 반복, 평균 훈련 손실, (반복마다 업데이트되는) 에포크의 평균 정확도는 물론 에포크 끝에서 검증 손실과 정확도를 출력하세요._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:26.864056Z",
     "iopub.status.busy": "2021-07-04T08:18:26.863020Z",
     "iopub.status.idle": "2021-07-04T08:18:27.305663Z",
     "shell.execute_reply": "2021-07-04T08:18:27.304709Z"
    },
    "id": "4csPXhr8nFS6",
    "outputId": "4de09f00-b38c-42cd-8a53-b1ed1de8d6bf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz\n",
      "32768/29515 [=================================] - 0s 0us/step\n",
      "40960/29515 [=========================================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz\n",
      "26427392/26421880 [==============================] - 0s 0us/step\n",
      "26435584/26421880 [==============================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz\n",
      "16384/5148 [===============================================================================================] - 0s 0us/step\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz\n",
      "4423680/4422102 [==============================] - 0s 0us/step\n",
      "4431872/4422102 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full.astype(np.float32) / 255.\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000:]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "X_test = X_test.astype(np.float32) / 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:27.315092Z",
     "iopub.status.busy": "2021-07-04T08:18:27.313431Z",
     "iopub.status.idle": "2021-07-04T08:18:27.320157Z",
     "shell.execute_reply": "2021-07-04T08:18:27.321148Z"
    },
    "id": "pU9Tyc3LnFS6"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:27.336428Z",
     "iopub.status.busy": "2021-07-04T08:18:27.335214Z",
     "iopub.status.idle": "2021-07-04T08:18:27.357760Z",
     "shell.execute_reply": "2021-07-04T08:18:27.356712Z"
    },
    "id": "1-7YuvHCnFS6"
   },
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\"),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:27.365537Z",
     "iopub.status.busy": "2021-07-04T08:18:27.364057Z",
     "iopub.status.idle": "2021-07-04T08:18:27.385914Z",
     "shell.execute_reply": "2021-07-04T08:18:27.384561Z"
    },
    "id": "FDtSEXLanFS6",
    "outputId": "1474eae4-4734-42c6-c961-0bea5d8777f2"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=0.01)\n",
    "loss_fn = keras.losses.sparse_categorical_crossentropy\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.SparseCategoricalAccuracy()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209,
     "referenced_widgets": [
      "954df2e8c6f94ad784aaabfbd4d4ab85",
      "dba56a9e52cb428f968f0821f3e7c417",
      "cd13ef0f97764193aedd194cbe18d00c",
      "fbfb5813c7c54c47a7f4715c215de32b",
      "01101b8e43d845e98557ae7d17a4aedd",
      "077ffc2e3d8d420bbfaf84eb275ce5fb",
      "88cbf84652964ec7815386f65d541829",
      "c77cec643eab4e4c9c963d7704008d2b",
      "f7b7b3259e1f4ea299a3d3c5c0abaf9d",
      "fa6c3a02385541c584bcf9c90cebbca0",
      "7a9231435c1c429e89e0eab0a539353e",
      "bdfaca20ecde4cf2aba21ec1dfda072e",
      "dbf564f628284554bd3aeaa0aa5fd0f8",
      "1b2df2881c86485fb4087c0fa8bf7575",
      "f1bc96fa597c483787ec73c6074360c9",
      "97fcfb43deca469487e6a6245efe6795",
      "e51664bf98454402b1174f541dd8ac77",
      "96d5de7a38a14125a9fce81b87c55b63",
      "fffcbe1917fe4729a2e4a5fcabbe2199",
      "d86e2684ae994c91a2bab1a48d52f303",
      "78fc4e9aa67b451ca54d192050b8e6a1",
      "3aeb59b8c260494c87688ab566c0550b",
      "bc621bb1bc1f425c9ec08b5e04e268c6",
      "8807073925614c1faedc10485d813611",
      "c596cd26628e4c23a49b12fc8610ceaf",
      "e447fde6569049eca870b7414ad01b1c",
      "2a0f4067caf94a99a21ca5091812b47f",
      "4e2faca0583c4a66b2ded1cad440319f",
      "ff90e29f3e024f4d9348372aca149fef",
      "0f85d383bfb1482f8c1e1676f9b76b20",
      "ed486814f2a8437983e66318b028dc1e",
      "f04e6eb59086470090065fda813b15c2",
      "1b6c735c79014db8b17d0ad7d764da9f",
      "30ccc72c804d435f85a75c0d585b9f16",
      "d7e0647152754ed499fb50725f1366eb",
      "21fb1c98fb42438897a7e47e805319e2",
      "fe691b3fa59344a9bda14da651e75f3b",
      "8a534692d07f4b7cbe53f4b78beb0827",
      "5e06565a48bf460bb9bee34f0adf503a",
      "9011c19bf8b04e239a12c3a62b221e1e",
      "1666384c6cc9435a8ad3e4769c7b8913",
      "c66b44bd19654a5f96c822c0e70d13bb",
      "52111926cbf346e38ebb3eaebc6ea30e",
      "81a80b17c1e947b98cabb1f726c209ff",
      "422e521b86cf4c0fa959fcced52804e7",
      "6428322789614bc581c5fcc6fc7420a1",
      "e9a2f5cc32c041a49016ce0dcda1aaa9",
      "3f97741a9ac243c3b24aeae50c6f1e2e",
      "99d1e2fffef44dcb8d3b278a7fdc78bb",
      "29eb106ec39e48f68cd9d351cf6ceac8",
      "10ea01782e764313a738b1f2036b593f",
      "d646b7a00e8e4e95b360a496d8c87935",
      "ca2720f9581241b9a4115740bad32d4a",
      "c259426907fc4c0b963379279a939065",
      "2989466ffc2e4c159b542c50484d248e",
      "261acd00b45c48798c67b9dbe3fccca1",
      "b39754b6bd1b425cad2defffd36347ac",
      "39c1b83bb9ff46aca393ac85aa40c87c",
      "4e38d1bfa23b42158f4b3e6fe455e598",
      "618e97706ef24d1f86cac54878f553f2",
      "51b5d2a15fa74330b16ba332a02ab0de",
      "763b5221f3f948e79a1645770a386cc5",
      "1c7a6f444e3b46f49242009f3648e98c",
      "bd38c3d7fc5d47cfb6eb91958a5b8be1",
      "4494340079f74e7c9e337ea2c5d89233",
      "ef4cd6ea805c430d8086f989bbaf05cb"
     ]
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:18:27.426092Z",
     "iopub.status.busy": "2021-07-04T08:18:27.424333Z",
     "iopub.status.idle": "2021-07-04T08:20:57.724860Z",
     "shell.execute_reply": "2021-07-04T08:20:57.723816Z"
    },
    "id": "rrbSZk-rnFS6",
    "outputId": "62c98fbb-39ee-48cd-e2af-3fd14fbbb5ef"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "954df2e8c6f94ad784aaabfbd4d4ab85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "All epochs:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdfaca20ecde4cf2aba21ec1dfda072e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 1/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc621bb1bc1f425c9ec08b5e04e268c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 2/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "30ccc72c804d435f85a75c0d585b9f16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 3/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "422e521b86cf4c0fa959fcced52804e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 4/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "261acd00b45c48798c67b9dbe3fccca1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 5/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with trange(1, n_epochs + 1, desc=\"All epochs\") as epochs:\n",
    "    for epoch in epochs:\n",
    "        with trange(1, n_steps + 1, desc=\"Epoch {}/{}\".format(epoch, n_epochs)) as steps:\n",
    "            for step in steps:\n",
    "                X_batch, y_batch = random_batch(X_train, y_train)\n",
    "                with tf.GradientTape() as tape:\n",
    "                    y_pred = model(X_batch)\n",
    "                    main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                    loss = tf.add_n([main_loss] + model.losses)\n",
    "                gradients = tape.gradient(loss, model.trainable_variables)\n",
    "                optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "                for variable in model.variables:\n",
    "                    if variable.constraint is not None:\n",
    "                        variable.assign(variable.constraint(variable))                    \n",
    "                status = OrderedDict()\n",
    "                mean_loss(loss)\n",
    "                status[\"loss\"] = mean_loss.result().numpy()\n",
    "                for metric in metrics:\n",
    "                    metric(y_batch, y_pred)\n",
    "                    status[metric.name] = metric.result().numpy()\n",
    "                steps.set_postfix(status)\n",
    "            y_pred = model(X_valid)\n",
    "            status[\"val_loss\"] = np.mean(loss_fn(y_valid, y_pred))\n",
    "            status[\"val_accuracy\"] = np.mean(keras.metrics.sparse_categorical_accuracy(\n",
    "                tf.constant(y_valid, dtype=np.float32), y_pred))\n",
    "            steps.set_postfix(status)\n",
    "        for metric in [mean_loss] + metrics:\n",
    "            metric.reset_states()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VXaxk5WwnFS6"
   },
   "source": [
    "### b.\n",
    "_문제: 상위 층과 하위 층에 학습률이 다른 옵티마이저를 따로 사용해보세요._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:20:57.735116Z",
     "iopub.status.busy": "2021-07-04T08:20:57.733165Z",
     "iopub.status.idle": "2021-07-04T08:20:57.741263Z",
     "shell.execute_reply": "2021-07-04T08:20:57.742096Z"
    },
    "id": "W_NUUQJLnFS6"
   },
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:20:57.757785Z",
     "iopub.status.busy": "2021-07-04T08:20:57.756644Z",
     "iopub.status.idle": "2021-07-04T08:20:57.808014Z",
     "shell.execute_reply": "2021-07-04T08:20:57.809090Z"
    },
    "id": "jMk78kcFnFS6"
   },
   "outputs": [],
   "source": [
    "lower_layers = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "])\n",
    "upper_layers = keras.models.Sequential([\n",
    "    keras.layers.Dense(10, activation=\"softmax\"),\n",
    "])\n",
    "model = keras.models.Sequential([\n",
    "    lower_layers, upper_layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:20:57.817373Z",
     "iopub.status.busy": "2021-07-04T08:20:57.816005Z",
     "iopub.status.idle": "2021-07-04T08:20:57.819046Z",
     "shell.execute_reply": "2021-07-04T08:20:57.820186Z"
    },
    "id": "_vvluGEWnFS7",
    "outputId": "3282292b-a4ae-4feb-9d8b-cdcee47d3a81"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n"
     ]
    }
   ],
   "source": [
    "lower_optimizer = keras.optimizers.SGD(learning_rate=1e-4)\n",
    "upper_optimizer = keras.optimizers.Nadam(learning_rate=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-07-04T08:20:57.838414Z",
     "iopub.status.busy": "2021-07-04T08:20:57.837124Z",
     "iopub.status.idle": "2021-07-04T08:20:57.861214Z",
     "shell.execute_reply": "2021-07-04T08:20:57.859890Z"
    },
    "id": "Y1Ah5AEMnFS7"
   },
   "outputs": [],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "loss_fn = keras.losses.sparse_categorical_crossentropy\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.SparseCategoricalAccuracy()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 209,
     "referenced_widgets": [
      "b64df0792ca744b0956faca854697247",
      "240dd8d4d60c4592a6e6d97f459e4ed3",
      "4a762cb8d0884184ba530d0cf66808c7",
      "7c9209ea773442ffb69c94993d280146",
      "702a63248dbd41c0ab76eade65b4e982",
      "5781adabc8434b55ab33b338d0675fee",
      "2fcf03fd1fc9423a8ac00fac090838b2",
      "9ab3309cdccc4d1d87b478b25d2126ea",
      "07b0a60ac77e450cbf9e583bbd0d043e",
      "ce807f932d1f437096dfac5d75a2a19f",
      "7736bf847415410086f651292abfcf5a",
      "16b039d22e054e1c85b0896ec20089dd",
      "69bee84df81845378fd777fe2cb80e92",
      "52431da65bab47f89e903d64340d4c7b",
      "cf9c56ae1e7f4b4eb3c8081d0edea083",
      "01460dbc26b34e3295ecf13046b384a2",
      "26ab338c09f944e1bb4132c9d699b068",
      "32343adfca8c4590a2389d2a34dc2985",
      "2cfd5ac794f74386a5d491877ecef946",
      "6db5e6485eda4dd8b6a8817bd4041f9c",
      "dee1df40cd38444ebad124cf5c58bb80",
      "982450d3a33144baae5a36bad4e84473",
      "cec12d5e2bce479fa6ef9441420ee512",
      "bba4a2ffffa643fdb726b64292b7d75e",
      "5ff5b68102864f11bf7d7e12248cd305",
      "41fca9356d384fa58c4cd2dcf5ba860f",
      "c6648f3fe191404f97adfa8e849f9538",
      "ea23077adb7c4a28bf6e2fe66f4aafff",
      "0dfd299aaff848fba46d52e485d98177",
      "cd3e3b06a1e846ab88990f099769e381",
      "c7a17800e32843ccb7c940a3c6ff40d0",
      "c06811c055164e80ada214e67f974892",
      "21d9cd9bb6cf4504a0c9937e710c04fd",
      "8d632b8938824212b72b0273667fa529",
      "2421b8871d6540fe8a1737bdca64ce00",
      "f8b2396a2aba44f7897137d1f14ba1da",
      "0c9c988d7264499a93aaf185234388d2",
      "990cf772d4b04745a97c08bb6df3f878",
      "444fa7320f2749e1849109a7f195d04a",
      "7c671360dc164e55a58cbcff4eb3199d",
      "dc3cc72da41747bfb203b01cc47377e7",
      "6505729fb17e453fb56fb1341cdd11ad",
      "9524b15918c8407c89bb33e671cf3c60",
      "33cbe81c8aa1409595f301c995fba1d2",
      "8861a8a339b24927a70c7a4d34f661aa",
      "e2701922f52e4bf9a6170af1d3c2e4b4",
      "d2d60853305f45878e742cfab60b3eaa",
      "fb13d30cc3fa4671935157b7a6b9c412",
      "542b77db259740e9be24d4554e9e5cbf",
      "a7aaa6c4fbb24547a115776d0919aa3e",
      "997db3cd02f24749b8130ed33ba68ef0",
      "84bb8f90247f4ff2a6b556c6b733030b",
      "80cfa25092094cd5a00cf0ff3ac0e6e9",
      "d084086314d24693aa89bff3eae1e363",
      "397a32ea60e04325a591bb8a5084db14",
      "55f6f8f85ad44cd092d737e22bcdbb12",
      "745eef7ab298438ca73c7eebf0c8d3f1",
      "4cba705cfe734505ba5e50e07b2d13ab",
      "52a376e476db4bb4b8bb5e938254dbb5",
      "942d2aefa11e4075951993c98e872d5e",
      "d32d6345488f479f91296371b3f5c248",
      "be7445ae4f9f4514a502e340b6f1fd94",
      "6aaa7c3b09904b189efe4a4a634d3e95",
      "529fd9384a9e42b68c4b6ffcbec8dfa3",
      "aa5cd820dabb4146afa9e660b0791cdc",
      "9ff2ccdefc944a83b69345b3ae6df4c7"
     ]
    },
    "execution": {
     "iopub.execute_input": "2021-07-04T08:20:57.916303Z",
     "iopub.status.busy": "2021-07-04T08:20:57.871210Z",
     "iopub.status.idle": "2021-07-04T08:23:28.076184Z",
     "shell.execute_reply": "2021-07-04T08:23:28.074944Z"
    },
    "id": "E2BbBpTMnFS7",
    "outputId": "09a19422-66c9-4d2c-e676-87e650d43e4a"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b64df0792ca744b0956faca854697247",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "All epochs:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16b039d22e054e1c85b0896ec20089dd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 1/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cec12d5e2bce479fa6ef9441420ee512",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 2/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d632b8938824212b72b0273667fa529",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 3/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8861a8a339b24927a70c7a4d34f661aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 4/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "55f6f8f85ad44cd092d737e22bcdbb12",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch 5/5:   0%|          | 0/1718 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with trange(1, n_epochs + 1, desc=\"All epochs\") as epochs:\n",
    "    for epoch in epochs:\n",
    "        with trange(1, n_steps + 1, desc=\"Epoch {}/{}\".format(epoch, n_epochs)) as steps:\n",
    "            for step in steps:\n",
    "                X_batch, y_batch = random_batch(X_train, y_train)\n",
    "                with tf.GradientTape(persistent=True) as tape:\n",
    "                    y_pred = model(X_batch)\n",
    "                    main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                    loss = tf.add_n([main_loss] + model.losses)\n",
    "                for layers, optimizer in ((lower_layers, lower_optimizer),\n",
    "                                          (upper_layers, upper_optimizer)):\n",
    "                    gradients = tape.gradient(loss, layers.trainable_variables)\n",
    "                    optimizer.apply_gradients(zip(gradients, layers.trainable_variables))\n",
    "                del tape\n",
    "                for variable in model.variables:\n",
    "                    if variable.constraint is not None:\n",
    "                        variable.assign(variable.constraint(variable))                    \n",
    "                status = OrderedDict()\n",
    "                mean_loss(loss)\n",
    "                status[\"loss\"] = mean_loss.result().numpy()\n",
    "                for metric in metrics:\n",
    "                    metric(y_batch, y_pred)\n",
    "                    status[metric.name] = metric.result().numpy()\n",
    "                steps.set_postfix(status)\n",
    "            y_pred = model(X_valid)\n",
    "            status[\"val_loss\"] = np.mean(loss_fn(y_valid, y_pred))\n",
    "            status[\"val_accuracy\"] = np.mean(keras.metrics.sparse_categorical_accuracy(\n",
    "                tf.constant(y_valid, dtype=np.float32), y_pred))\n",
    "            steps.set_postfix(status)\n",
    "        for metric in [mean_loss] + metrics:\n",
    "            metric.reset_states()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "12_custom_models_and_training_with_tensorflow.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "004faff048744caf8c45ec588d15c83c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "01101b8e43d845e98557ae7d17a4aedd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7a9231435c1c429e89e0eab0a539353e",
      "placeholder": "​",
      "style": "IPY_MODEL_fa6c3a02385541c584bcf9c90cebbca0",
      "value": " 5/5 [03:07&lt;00:00, 37.36s/it]"
     }
    },
    "01460dbc26b34e3295ecf13046b384a2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_982450d3a33144baae5a36bad4e84473",
      "placeholder": "​",
      "style": "IPY_MODEL_dee1df40cd38444ebad124cf5c58bb80",
      "value": " 1718/1718 [00:36&lt;00:00, 49.65it/s, loss=1.06, sparse_categorical_accuracy=0.679]"
     }
    },
    "0550b859d58c4b21b730ef45b52a97f2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "077ffc2e3d8d420bbfaf84eb275ce5fb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0797d34c1de449309ddbaebb42e85c07": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "07b0a60ac77e450cbf9e583bbd0d043e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0af626a080824117a0e04705562e6418": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0b1a5bde488248e483f211a75e50b9a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_004faff048744caf8c45ec588d15c83c",
      "max": 362,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b8ab2b969e9641b98c8934443cb199c7",
      "value": 362
     }
    },
    "0bd851e4cb03463985643ae9f4c37aa7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0c9c988d7264499a93aaf185234388d2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6505729fb17e453fb56fb1341cdd11ad",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dc3cc72da41747bfb203b01cc47377e7",
      "value": 1718
     }
    },
    "0dfd299aaff848fba46d52e485d98177": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0f85d383bfb1482f8c1e1676f9b76b20": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1066152b45a7421393c6f2110a338fcb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "10ea01782e764313a738b1f2036b593f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "15750aba26bc47b1b03a133a4cadad3a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1666384c6cc9435a8ad3e4769c7b8913": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "16b039d22e054e1c85b0896ec20089dd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_52431da65bab47f89e903d64340d4c7b",
       "IPY_MODEL_cf9c56ae1e7f4b4eb3c8081d0edea083",
       "IPY_MODEL_01460dbc26b34e3295ecf13046b384a2"
      ],
      "layout": "IPY_MODEL_69bee84df81845378fd777fe2cb80e92"
     }
    },
    "19386467e8344d93b420b0f8c2483400": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1b2df2881c86485fb4087c0fa8bf7575": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_96d5de7a38a14125a9fce81b87c55b63",
      "placeholder": "​",
      "style": "IPY_MODEL_e51664bf98454402b1174f541dd8ac77",
      "value": "Epoch 1/5: 100%"
     }
    },
    "1b6c735c79014db8b17d0ad7d764da9f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1c7a6f444e3b46f49242009f3648e98c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "21d9cd9bb6cf4504a0c9937e710c04fd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "21fb1c98fb42438897a7e47e805319e2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9011c19bf8b04e239a12c3a62b221e1e",
      "placeholder": "​",
      "style": "IPY_MODEL_5e06565a48bf460bb9bee34f0adf503a",
      "value": "Epoch 3/5: 100%"
     }
    },
    "240dd8d4d60c4592a6e6d97f459e4ed3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2421b8871d6540fe8a1737bdca64ce00": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "252a5e08473f4751a6b63bda7ce5055d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_40e7a51a0db544ea80d4f5e3805a2a0d",
      "placeholder": "​",
      "style": "IPY_MODEL_eb43a6e971134e57b35e636ac843b377",
      "value": "Epoch 4/5: 100%"
     }
    },
    "261acd00b45c48798c67b9dbe3fccca1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_39c1b83bb9ff46aca393ac85aa40c87c",
       "IPY_MODEL_4e38d1bfa23b42158f4b3e6fe455e598",
       "IPY_MODEL_618e97706ef24d1f86cac54878f553f2"
      ],
      "layout": "IPY_MODEL_b39754b6bd1b425cad2defffd36347ac"
     }
    },
    "26ab338c09f944e1bb4132c9d699b068": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2989466ffc2e4c159b542c50484d248e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "29eb106ec39e48f68cd9d351cf6ceac8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2a0f4067caf94a99a21ca5091812b47f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1b6c735c79014db8b17d0ad7d764da9f",
      "placeholder": "​",
      "style": "IPY_MODEL_f04e6eb59086470090065fda813b15c2",
      "value": " 1718/1718 [00:37&lt;00:00, 46.25it/s, loss=0.408, sparse_categorical_accuracy=0.854]"
     }
    },
    "2a52efd06cbd4bfb8e5dbbe17c6236ab": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_15750aba26bc47b1b03a133a4cadad3a",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d40d0e4d6dbc4fe7837af600a714baf5",
      "value": 5
     }
    },
    "2cfd5ac794f74386a5d491877ecef946": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2d1ae6fa9fe146a9a288fd5ea41667a2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e3c0df173966467db5fb15ded865d25b",
      "placeholder": "​",
      "style": "IPY_MODEL_6157ade5095b4bda96075e2b5f51df95",
      "value": " 362/362 [00:09&lt;00:00, 37.30it/s, loss=0.637, mean_absolute_error=0.513]"
     }
    },
    "2fcf03fd1fc9423a8ac00fac090838b2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "306c5a5eb7dd4afdaab6886e31e99a02": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8c863503d34b45ebbcb9190940434880",
      "placeholder": "​",
      "style": "IPY_MODEL_6cdc302c227b4da9a76120662368e6b8",
      "value": " 362/362 [00:10&lt;00:00, 37.10it/s, loss=0.65, mean_absolute_error=0.522]"
     }
    },
    "30ccc72c804d435f85a75c0d585b9f16": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_21fb1c98fb42438897a7e47e805319e2",
       "IPY_MODEL_fe691b3fa59344a9bda14da651e75f3b",
       "IPY_MODEL_8a534692d07f4b7cbe53f4b78beb0827"
      ],
      "layout": "IPY_MODEL_d7e0647152754ed499fb50725f1366eb"
     }
    },
    "32343adfca8c4590a2389d2a34dc2985": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "33cbe81c8aa1409595f301c995fba1d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3697fc6dc6c34b8fbca2c522d3fe0543": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f087a4deddfb4b13b39d053bc39ed254",
       "IPY_MODEL_4d85525433584eb4ae759bc56f0332eb",
       "IPY_MODEL_2d1ae6fa9fe146a9a288fd5ea41667a2"
      ],
      "layout": "IPY_MODEL_d7b0f08b1a8241ae8f5857893c7e6196"
     }
    },
    "397a32ea60e04325a591bb8a5084db14": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39c1b83bb9ff46aca393ac85aa40c87c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_763b5221f3f948e79a1645770a386cc5",
      "placeholder": "​",
      "style": "IPY_MODEL_51b5d2a15fa74330b16ba332a02ab0de",
      "value": "Epoch 5/5: 100%"
     }
    },
    "3aeb59b8c260494c87688ab566c0550b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3f97741a9ac243c3b24aeae50c6f1e2e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ca2720f9581241b9a4115740bad32d4a",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d646b7a00e8e4e95b360a496d8c87935",
      "value": 1718
     }
    },
    "40e7a51a0db544ea80d4f5e3805a2a0d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "41fca9356d384fa58c4cd2dcf5ba860f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c7a17800e32843ccb7c940a3c6ff40d0",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cd3e3b06a1e846ab88990f099769e381",
      "value": 1718
     }
    },
    "422e521b86cf4c0fa959fcced52804e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e9a2f5cc32c041a49016ce0dcda1aaa9",
       "IPY_MODEL_3f97741a9ac243c3b24aeae50c6f1e2e",
       "IPY_MODEL_99d1e2fffef44dcb8d3b278a7fdc78bb"
      ],
      "layout": "IPY_MODEL_6428322789614bc581c5fcc6fc7420a1"
     }
    },
    "434a0e19af3d45cab0e00916a6f1d504": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "444fa7320f2749e1849109a7f195d04a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4494340079f74e7c9e337ea2c5d89233": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "48a18bc2cb684c1b923a8c1da67238fc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4a762cb8d0884184ba530d0cf66808c7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2fcf03fd1fc9423a8ac00fac090838b2",
      "placeholder": "​",
      "style": "IPY_MODEL_5781adabc8434b55ab33b338d0675fee",
      "value": "All epochs: 100%"
     }
    },
    "4abc8ad70e6d4112a18aa58394942015": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "4bf89d5e43274887bd1724a2d305e0a4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4cba705cfe734505ba5e50e07b2d13ab": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_be7445ae4f9f4514a502e340b6f1fd94",
      "placeholder": "​",
      "style": "IPY_MODEL_d32d6345488f479f91296371b3f5c248",
      "value": "Epoch 5/5: 100%"
     }
    },
    "4d85525433584eb4ae759bc56f0332eb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b20bc4ddb0004894b3fb69a7dc5a6b37",
      "max": 362,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8f9ea35424a049e2826fcd12e782091b",
      "value": 362
     }
    },
    "4dc3605db14b461b8fd91761de341563": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0797d34c1de449309ddbaebb42e85c07",
      "placeholder": "​",
      "style": "IPY_MODEL_d5eff707caaa4775a0a2751fec7f1eba",
      "value": " 362/362 [00:09&lt;00:00, 36.28it/s, loss=0.626, mean_absolute_error=0.51]"
     }
    },
    "4e2faca0583c4a66b2ded1cad440319f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4e38d1bfa23b42158f4b3e6fe455e598": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bd38c3d7fc5d47cfb6eb91958a5b8be1",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1c7a6f444e3b46f49242009f3648e98c",
      "value": 1718
     }
    },
    "4fb6a7ada2cf4ab9812a62110edd7169": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "503dcfb1aa1a47cca810d3ae18347faf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d0295bc6f1bb4ee38d7ac9244ee54d8e",
      "placeholder": "​",
      "style": "IPY_MODEL_844cea147d9d46c081ade1ef0e117c72",
      "value": " 5/5 [00:49&lt;00:00,  9.84s/it]"
     }
    },
    "51a6f9642c7e45408f3d984b6a25344a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0bd851e4cb03463985643ae9f4c37aa7",
      "max": 362,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_48a18bc2cb684c1b923a8c1da67238fc",
      "value": 362
     }
    },
    "51b5d2a15fa74330b16ba332a02ab0de": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "52111926cbf346e38ebb3eaebc6ea30e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "52431da65bab47f89e903d64340d4c7b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_32343adfca8c4590a2389d2a34dc2985",
      "placeholder": "​",
      "style": "IPY_MODEL_26ab338c09f944e1bb4132c9d699b068",
      "value": "Epoch 1/5: 100%"
     }
    },
    "529fd9384a9e42b68c4b6ffcbec8dfa3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "52a376e476db4bb4b8bb5e938254dbb5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_529fd9384a9e42b68c4b6ffcbec8dfa3",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6aaa7c3b09904b189efe4a4a634d3e95",
      "value": 1718
     }
    },
    "542b77db259740e9be24d4554e9e5cbf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_397a32ea60e04325a591bb8a5084db14",
      "placeholder": "​",
      "style": "IPY_MODEL_d084086314d24693aa89bff3eae1e363",
      "value": " 1718/1718 [00:36&lt;00:00, 48.74it/s, loss=0.537, sparse_categorical_accuracy=0.811]"
     }
    },
    "55f6f8f85ad44cd092d737e22bcdbb12": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4cba705cfe734505ba5e50e07b2d13ab",
       "IPY_MODEL_52a376e476db4bb4b8bb5e938254dbb5",
       "IPY_MODEL_942d2aefa11e4075951993c98e872d5e"
      ],
      "layout": "IPY_MODEL_745eef7ab298438ca73c7eebf0c8d3f1"
     }
    },
    "5781adabc8434b55ab33b338d0675fee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5e06565a48bf460bb9bee34f0adf503a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5f29d13ac54e4d968770d64d1fad99aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4bf89d5e43274887bd1724a2d305e0a4",
      "max": 362,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4abc8ad70e6d4112a18aa58394942015",
      "value": 362
     }
    },
    "5ff5b68102864f11bf7d7e12248cd305": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0dfd299aaff848fba46d52e485d98177",
      "placeholder": "​",
      "style": "IPY_MODEL_ea23077adb7c4a28bf6e2fe66f4aafff",
      "value": "Epoch 2/5: 100%"
     }
    },
    "6157ade5095b4bda96075e2b5f51df95": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "618e97706ef24d1f86cac54878f553f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ef4cd6ea805c430d8086f989bbaf05cb",
      "placeholder": "​",
      "style": "IPY_MODEL_4494340079f74e7c9e337ea2c5d89233",
      "value": " 1718/1718 [00:37&lt;00:00, 47.02it/s, loss=0.362, sparse_categorical_accuracy=0.869]"
     }
    },
    "6428322789614bc581c5fcc6fc7420a1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "645ace54d7b9427aabe2b2dfa6dc6dea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6505729fb17e453fb56fb1341cdd11ad": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "69bee84df81845378fd777fe2cb80e92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6aaa7c3b09904b189efe4a4a634d3e95": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6cdc302c227b4da9a76120662368e6b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6db5e6485eda4dd8b6a8817bd4041f9c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6dfcfab30ebf42bfa4e8f99cb6654f37": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d00ac2178fda4662bac2b5894225e695",
      "placeholder": "​",
      "style": "IPY_MODEL_b442291bc2fd4afeb00345c601401a48",
      "value": "All epochs: 100%"
     }
    },
    "702a63248dbd41c0ab76eade65b4e982": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7736bf847415410086f651292abfcf5a",
      "placeholder": "​",
      "style": "IPY_MODEL_ce807f932d1f437096dfac5d75a2a19f",
      "value": " 5/5 [03:00&lt;00:00, 36.04s/it]"
     }
    },
    "745eef7ab298438ca73c7eebf0c8d3f1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "74bea35b82934882b9098623f450fab7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_252a5e08473f4751a6b63bda7ce5055d",
       "IPY_MODEL_0b1a5bde488248e483f211a75e50b9a7",
       "IPY_MODEL_df2eb47bbf0c47dfbcc6c304733ebcf6"
      ],
      "layout": "IPY_MODEL_e216a70f79254c9a864c952d1cb6f7f4"
     }
    },
    "75d93ec1d61648679862d7ddee5547b9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "763b5221f3f948e79a1645770a386cc5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7736bf847415410086f651292abfcf5a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "78fc4e9aa67b451ca54d192050b8e6a1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7a9231435c1c429e89e0eab0a539353e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7c671360dc164e55a58cbcff4eb3199d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7c9209ea773442ffb69c94993d280146": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_07b0a60ac77e450cbf9e583bbd0d043e",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9ab3309cdccc4d1d87b478b25d2126ea",
      "value": 5
     }
    },
    "80cfa25092094cd5a00cf0ff3ac0e6e9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "81a80b17c1e947b98cabb1f726c209ff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "844cea147d9d46c081ade1ef0e117c72": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "84bb8f90247f4ff2a6b556c6b733030b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8807073925614c1faedc10485d813611": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8861a8a339b24927a70c7a4d34f661aa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d2d60853305f45878e742cfab60b3eaa",
       "IPY_MODEL_fb13d30cc3fa4671935157b7a6b9c412",
       "IPY_MODEL_542b77db259740e9be24d4554e9e5cbf"
      ],
      "layout": "IPY_MODEL_e2701922f52e4bf9a6170af1d3c2e4b4"
     }
    },
    "88cbf84652964ec7815386f65d541829": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8a534692d07f4b7cbe53f4b78beb0827": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_81a80b17c1e947b98cabb1f726c209ff",
      "placeholder": "​",
      "style": "IPY_MODEL_52111926cbf346e38ebb3eaebc6ea30e",
      "value": " 1718/1718 [00:37&lt;00:00, 48.13it/s, loss=0.374, sparse_categorical_accuracy=0.865]"
     }
    },
    "8c863503d34b45ebbcb9190940434880": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8d632b8938824212b72b0273667fa529": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f8b2396a2aba44f7897137d1f14ba1da",
       "IPY_MODEL_0c9c988d7264499a93aaf185234388d2",
       "IPY_MODEL_990cf772d4b04745a97c08bb6df3f878"
      ],
      "layout": "IPY_MODEL_2421b8871d6540fe8a1737bdca64ce00"
     }
    },
    "8f9ea35424a049e2826fcd12e782091b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9011c19bf8b04e239a12c3a62b221e1e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "942d2aefa11e4075951993c98e872d5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9ff2ccdefc944a83b69345b3ae6df4c7",
      "placeholder": "​",
      "style": "IPY_MODEL_aa5cd820dabb4146afa9e660b0791cdc",
      "value": " 1718/1718 [00:36&lt;00:00, 47.57it/s, loss=0.523, sparse_categorical_accuracy=0.814]"
     }
    },
    "9524b15918c8407c89bb33e671cf3c60": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "954df2e8c6f94ad784aaabfbd4d4ab85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_cd13ef0f97764193aedd194cbe18d00c",
       "IPY_MODEL_fbfb5813c7c54c47a7f4715c215de32b",
       "IPY_MODEL_01101b8e43d845e98557ae7d17a4aedd"
      ],
      "layout": "IPY_MODEL_dba56a9e52cb428f968f0821f3e7c417"
     }
    },
    "96d5de7a38a14125a9fce81b87c55b63": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "97fcfb43deca469487e6a6245efe6795": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3aeb59b8c260494c87688ab566c0550b",
      "placeholder": "​",
      "style": "IPY_MODEL_78fc4e9aa67b451ca54d192050b8e6a1",
      "value": " 1718/1718 [00:38&lt;00:00, 47.59it/s, loss=0.511, sparse_categorical_accuracy=0.816]"
     }
    },
    "982450d3a33144baae5a36bad4e84473": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "990cf772d4b04745a97c08bb6df3f878": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_33cbe81c8aa1409595f301c995fba1d2",
      "placeholder": "​",
      "style": "IPY_MODEL_9524b15918c8407c89bb33e671cf3c60",
      "value": " 1718/1718 [00:35&lt;00:00, 49.57it/s, loss=0.572, sparse_categorical_accuracy=0.802]"
     }
    },
    "997db3cd02f24749b8130ed33ba68ef0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "99d1e2fffef44dcb8d3b278a7fdc78bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2989466ffc2e4c159b542c50484d248e",
      "placeholder": "​",
      "style": "IPY_MODEL_c259426907fc4c0b963379279a939065",
      "value": " 1718/1718 [00:37&lt;00:00, 46.01it/s, loss=0.368, sparse_categorical_accuracy=0.869]"
     }
    },
    "9ab3309cdccc4d1d87b478b25d2126ea": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "9ff2ccdefc944a83b69345b3ae6df4c7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a4d0b9ed8c1b44819157609d8faa8b5d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f8cbaeb84c544413b44be1717d723be2",
      "max": 362,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cf960f823e2741b4b0117ff9eb298414",
      "value": 362
     }
    },
    "a7aaa6c4fbb24547a115776d0919aa3e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "aa5cd820dabb4146afa9e660b0791cdc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ab5fb1fc8fe7467495e0ee03f6839479": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d6384c2670af484a8d0ce6019929b947",
       "IPY_MODEL_a4d0b9ed8c1b44819157609d8faa8b5d",
       "IPY_MODEL_4dc3605db14b461b8fd91761de341563"
      ],
      "layout": "IPY_MODEL_d4e99b5a737f4a93a8ad75d870dc485a"
     }
    },
    "b20bc4ddb0004894b3fb69a7dc5a6b37": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b39754b6bd1b425cad2defffd36347ac": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b442291bc2fd4afeb00345c601401a48": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b531e026313f4933943884f6e1e795f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e09a357ca8bf49d1b9da060f8eb29224",
       "IPY_MODEL_5f29d13ac54e4d968770d64d1fad99aa",
       "IPY_MODEL_306c5a5eb7dd4afdaab6886e31e99a02"
      ],
      "layout": "IPY_MODEL_fb52f438720d42e7bcf99a27ad7d2eb7"
     }
    },
    "b5dadf6cb1e8404bb156b17058f73f4a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b64df0792ca744b0956faca854697247": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4a762cb8d0884184ba530d0cf66808c7",
       "IPY_MODEL_7c9209ea773442ffb69c94993d280146",
       "IPY_MODEL_702a63248dbd41c0ab76eade65b4e982"
      ],
      "layout": "IPY_MODEL_240dd8d4d60c4592a6e6d97f459e4ed3"
     }
    },
    "b6a618847516464680c2a16aef583371": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b8ab2b969e9641b98c8934443cb199c7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "bba4a2ffffa643fdb726b64292b7d75e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bc621bb1bc1f425c9ec08b5e04e268c6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c596cd26628e4c23a49b12fc8610ceaf",
       "IPY_MODEL_e447fde6569049eca870b7414ad01b1c",
       "IPY_MODEL_2a0f4067caf94a99a21ca5091812b47f"
      ],
      "layout": "IPY_MODEL_8807073925614c1faedc10485d813611"
     }
    },
    "bcf308ed877d40cabcca590040b19db9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6dfcfab30ebf42bfa4e8f99cb6654f37",
       "IPY_MODEL_2a52efd06cbd4bfb8e5dbbe17c6236ab",
       "IPY_MODEL_503dcfb1aa1a47cca810d3ae18347faf"
      ],
      "layout": "IPY_MODEL_0550b859d58c4b21b730ef45b52a97f2"
     }
    },
    "bd38c3d7fc5d47cfb6eb91958a5b8be1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bdfaca20ecde4cf2aba21ec1dfda072e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1b2df2881c86485fb4087c0fa8bf7575",
       "IPY_MODEL_f1bc96fa597c483787ec73c6074360c9",
       "IPY_MODEL_97fcfb43deca469487e6a6245efe6795"
      ],
      "layout": "IPY_MODEL_dbf564f628284554bd3aeaa0aa5fd0f8"
     }
    },
    "be5e3d53ea4144539386c29ffef07a3d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c80da36d70744be0b6343dc7b5f24f53",
      "placeholder": "​",
      "style": "IPY_MODEL_b5dadf6cb1e8404bb156b17058f73f4a",
      "value": " 362/362 [00:09&lt;00:00, 39.33it/s, loss=0.63, mean_absolute_error=0.512]"
     }
    },
    "be7445ae4f9f4514a502e340b6f1fd94": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c06811c055164e80ada214e67f974892": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c259426907fc4c0b963379279a939065": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c596cd26628e4c23a49b12fc8610ceaf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ff90e29f3e024f4d9348372aca149fef",
      "placeholder": "​",
      "style": "IPY_MODEL_4e2faca0583c4a66b2ded1cad440319f",
      "value": "Epoch 2/5: 100%"
     }
    },
    "c6648f3fe191404f97adfa8e849f9538": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_21d9cd9bb6cf4504a0c9937e710c04fd",
      "placeholder": "​",
      "style": "IPY_MODEL_c06811c055164e80ada214e67f974892",
      "value": " 1718/1718 [00:35&lt;00:00, 48.54it/s, loss=0.648, sparse_categorical_accuracy=0.783]"
     }
    },
    "c66b44bd19654a5f96c822c0e70d13bb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c715ff27d24e45f9b4aacede06814708": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d0e360ece5814c02ac20a2fe42e5c66f",
       "IPY_MODEL_51a6f9642c7e45408f3d984b6a25344a",
       "IPY_MODEL_be5e3d53ea4144539386c29ffef07a3d"
      ],
      "layout": "IPY_MODEL_cdd33bdc83ef4ba08e7aaf223624f89a"
     }
    },
    "c77cec643eab4e4c9c963d7704008d2b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c7a17800e32843ccb7c940a3c6ff40d0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c80da36d70744be0b6343dc7b5f24f53": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ca2720f9581241b9a4115740bad32d4a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cd13ef0f97764193aedd194cbe18d00c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_88cbf84652964ec7815386f65d541829",
      "placeholder": "​",
      "style": "IPY_MODEL_077ffc2e3d8d420bbfaf84eb275ce5fb",
      "value": "All epochs: 100%"
     }
    },
    "cd3e3b06a1e846ab88990f099769e381": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cdd33bdc83ef4ba08e7aaf223624f89a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ce807f932d1f437096dfac5d75a2a19f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cec12d5e2bce479fa6ef9441420ee512": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5ff5b68102864f11bf7d7e12248cd305",
       "IPY_MODEL_41fca9356d384fa58c4cd2dcf5ba860f",
       "IPY_MODEL_c6648f3fe191404f97adfa8e849f9538"
      ],
      "layout": "IPY_MODEL_bba4a2ffffa643fdb726b64292b7d75e"
     }
    },
    "cf960f823e2741b4b0117ff9eb298414": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cf9c56ae1e7f4b4eb3c8081d0edea083": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6db5e6485eda4dd8b6a8817bd4041f9c",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2cfd5ac794f74386a5d491877ecef946",
      "value": 1718
     }
    },
    "d00ac2178fda4662bac2b5894225e695": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d0295bc6f1bb4ee38d7ac9244ee54d8e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d084086314d24693aa89bff3eae1e363": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d0e360ece5814c02ac20a2fe42e5c66f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1066152b45a7421393c6f2110a338fcb",
      "placeholder": "​",
      "style": "IPY_MODEL_f1e80d88677d40f1bdf9ffcf4fe649d8",
      "value": "Epoch 2/5: 100%"
     }
    },
    "d2d60853305f45878e742cfab60b3eaa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_997db3cd02f24749b8130ed33ba68ef0",
      "placeholder": "​",
      "style": "IPY_MODEL_a7aaa6c4fbb24547a115776d0919aa3e",
      "value": "Epoch 4/5: 100%"
     }
    },
    "d32d6345488f479f91296371b3f5c248": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d40d0e4d6dbc4fe7837af600a714baf5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "d4e99b5a737f4a93a8ad75d870dc485a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d5eff707caaa4775a0a2751fec7f1eba": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d6384c2670af484a8d0ce6019929b947": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_645ace54d7b9427aabe2b2dfa6dc6dea",
      "placeholder": "​",
      "style": "IPY_MODEL_df7ed54f99b54d2abff6b5bdde6db72e",
      "value": "Epoch 3/5: 100%"
     }
    },
    "d646b7a00e8e4e95b360a496d8c87935": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "d7b0f08b1a8241ae8f5857893c7e6196": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d7e0647152754ed499fb50725f1366eb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d86e2684ae994c91a2bab1a48d52f303": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dba56a9e52cb428f968f0821f3e7c417": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dbf564f628284554bd3aeaa0aa5fd0f8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dc3cc72da41747bfb203b01cc47377e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "dee1df40cd38444ebad124cf5c58bb80": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "df2eb47bbf0c47dfbcc6c304733ebcf6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0af626a080824117a0e04705562e6418",
      "placeholder": "​",
      "style": "IPY_MODEL_434a0e19af3d45cab0e00916a6f1d504",
      "value": " 362/362 [00:09&lt;00:00, 38.69it/s, loss=0.612, mean_absolute_error=0.505]"
     }
    },
    "df7ed54f99b54d2abff6b5bdde6db72e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e09a357ca8bf49d1b9da060f8eb29224": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_75d93ec1d61648679862d7ddee5547b9",
      "placeholder": "​",
      "style": "IPY_MODEL_b6a618847516464680c2a16aef583371",
      "value": "Epoch 1/5: 100%"
     }
    },
    "e216a70f79254c9a864c952d1cb6f7f4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e2701922f52e4bf9a6170af1d3c2e4b4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e3c0df173966467db5fb15ded865d25b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e447fde6569049eca870b7414ad01b1c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ed486814f2a8437983e66318b028dc1e",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0f85d383bfb1482f8c1e1676f9b76b20",
      "value": 1718
     }
    },
    "e51664bf98454402b1174f541dd8ac77": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e9a2f5cc32c041a49016ce0dcda1aaa9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_10ea01782e764313a738b1f2036b593f",
      "placeholder": "​",
      "style": "IPY_MODEL_29eb106ec39e48f68cd9d351cf6ceac8",
      "value": "Epoch 4/5: 100%"
     }
    },
    "ea23077adb7c4a28bf6e2fe66f4aafff": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "eb43a6e971134e57b35e636ac843b377": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ed486814f2a8437983e66318b028dc1e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef4cd6ea805c430d8086f989bbaf05cb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f04e6eb59086470090065fda813b15c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f087a4deddfb4b13b39d053bc39ed254": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_19386467e8344d93b420b0f8c2483400",
      "placeholder": "​",
      "style": "IPY_MODEL_4fb6a7ada2cf4ab9812a62110edd7169",
      "value": "Epoch 5/5: 100%"
     }
    },
    "f1bc96fa597c483787ec73c6074360c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d86e2684ae994c91a2bab1a48d52f303",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_fffcbe1917fe4729a2e4a5fcabbe2199",
      "value": 1718
     }
    },
    "f1e80d88677d40f1bdf9ffcf4fe649d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f7b7b3259e1f4ea299a3d3c5c0abaf9d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f8b2396a2aba44f7897137d1f14ba1da": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c671360dc164e55a58cbcff4eb3199d",
      "placeholder": "​",
      "style": "IPY_MODEL_444fa7320f2749e1849109a7f195d04a",
      "value": "Epoch 3/5: 100%"
     }
    },
    "f8cbaeb84c544413b44be1717d723be2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fa6c3a02385541c584bcf9c90cebbca0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fb13d30cc3fa4671935157b7a6b9c412": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_80cfa25092094cd5a00cf0ff3ac0e6e9",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_84bb8f90247f4ff2a6b556c6b733030b",
      "value": 1718
     }
    },
    "fb52f438720d42e7bcf99a27ad7d2eb7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fbfb5813c7c54c47a7f4715c215de32b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f7b7b3259e1f4ea299a3d3c5c0abaf9d",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c77cec643eab4e4c9c963d7704008d2b",
      "value": 5
     }
    },
    "fe691b3fa59344a9bda14da651e75f3b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c66b44bd19654a5f96c822c0e70d13bb",
      "max": 1718,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1666384c6cc9435a8ad3e4769c7b8913",
      "value": 1718
     }
    },
    "ff90e29f3e024f4d9348372aca149fef": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fffcbe1917fe4729a2e4a5fcabbe2199": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}