{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "day34_04_lecture_chap2_exercise_public.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rkti498/e_shikaku/blob/main/day34_04_lecture_chap2_exercise_public.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c183rGQUp2aJ"
      },
      "source": [
        "# 演習 Transformerモデル"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-dQ9b9Yjp2aL"
      },
      "source": [
        "TransformerはRNNやCNNを使用せず、Attentionのみを用いるSeq2Seqモデルです。\n",
        "\n",
        "並列計算が可能なためRNNに比べて計算が高速な上、Self-Attentionと呼ばれる機構を用いることにより、局所的な位置しか参照できないCNNと異なり、系列内の任意の位置の情報を参照することを可能にしています。\n",
        "\n",
        "その他にもいくつかの工夫が加えられており、翻訳に限らない自然言語処理のあらゆるタスクで圧倒的な性能を示すことが知られています。\n",
        "\n",
        "原論文：[Attention is All You Need](https://arxiv.org/abs/1706.03762)\n",
        "\n",
        "参考実装：https://github.com/jadore801120/attention-is-all-you-need-pytorch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lA-aG0SHg5Iw"
      },
      "source": [
        ""
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WwR-X5fqqPHC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ef9e8c21-ccd6-4b59-9935-cbefe3ed1122"
      },
      "source": [
        "! wget https://www.dropbox.com/s/9narw5x4uizmehh/utils.py\n",
        "! mkdir images data\n",
        "\n",
        "# data取得\n",
        "! wget https://www.dropbox.com/s/o4kyc52a8we25wy/dev.en -P data/\n",
        "! wget https://www.dropbox.com/s/kdgskm5hzg6znuc/dev.ja -P data/\n",
        "! wget https://www.dropbox.com/s/gyyx4gohv9v65uh/test.en -P data/\n",
        "! wget https://www.dropbox.com/s/hotxwbgoe2n013k/test.ja -P data/\n",
        "! wget https://www.dropbox.com/s/5lsftkmb20ay9e1/train.en -P data/\n",
        "! wget https://www.dropbox.com/s/ak53qirssci6f1j/train.ja -P data/"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-07-18 12:08:07--  https://www.dropbox.com/s/9narw5x4uizmehh/utils.py\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/9narw5x4uizmehh/utils.py [following]\n",
            "--2021-07-18 12:08:07--  https://www.dropbox.com/s/raw/9narw5x4uizmehh/utils.py\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com/cd/0/inline/BSjM4QxI-YkvO4k_X5FoMHZokPNnkyJRhKeIDkIq-EFFjxqkJEjo-ygKrclBmTTVsG3GrFPvhfX-fk4ZKjm-4n0mn2Mo-8KsGEx5NDBX0aSB5T4M0j5xVdtkuwsWnUMWMiAOM8CQgi8rjUojfBkXnWxk/file# [following]\n",
            "--2021-07-18 12:08:07--  https://uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com/cd/0/inline/BSjM4QxI-YkvO4k_X5FoMHZokPNnkyJRhKeIDkIq-EFFjxqkJEjo-ygKrclBmTTVsG3GrFPvhfX-fk4ZKjm-4n0mn2Mo-8KsGEx5NDBX0aSB5T4M0j5xVdtkuwsWnUMWMiAOM8CQgi8rjUojfBkXnWxk/file\n",
            "Resolving uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com (uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6017:15::a27d:20f\n",
            "Connecting to uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com (uc0e4ec689d9019c5836c5a442e0.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 949 [text/plain]\n",
            "Saving to: ‘utils.py.1’\n",
            "\n",
            "utils.py.1          100%[===================>]     949  --.-KB/s    in 0s      \n",
            "\n",
            "2021-07-18 12:08:07 (176 MB/s) - ‘utils.py.1’ saved [949/949]\n",
            "\n",
            "mkdir: cannot create directory ‘images’: File exists\n",
            "mkdir: cannot create directory ‘data’: File exists\n",
            "--2021-07-18 12:08:08--  https://www.dropbox.com/s/o4kyc52a8we25wy/dev.en\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/o4kyc52a8we25wy/dev.en [following]\n",
            "--2021-07-18 12:08:08--  https://www.dropbox.com/s/raw/o4kyc52a8we25wy/dev.en\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com/cd/0/inline/BSjWZRTKeJDRVIcWMLjUqUmg0EF0yMaGnTIXrDOKCZCUwfYRwtTa2Yuwy7KsdR6H8rCrhO54D90_TiOVC1KlZOPdUDdlQYbG3lehuh13Vag_ZFXFXk7NeJDUbCzPu9Xzic5pfKHkTwelk4snaZTZOWiE/file# [following]\n",
            "--2021-07-18 12:08:08--  https://uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com/cd/0/inline/BSjWZRTKeJDRVIcWMLjUqUmg0EF0yMaGnTIXrDOKCZCUwfYRwtTa2Yuwy7KsdR6H8rCrhO54D90_TiOVC1KlZOPdUDdlQYbG3lehuh13Vag_ZFXFXk7NeJDUbCzPu9Xzic5pfKHkTwelk4snaZTZOWiE/file\n",
            "Resolving uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com (uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6022:15::a27d:420f\n",
            "Connecting to uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com (uc69432a490bf7bca59d9efd0a8d.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 17054 (17K) [text/plain]\n",
            "Saving to: ‘data/dev.en.1’\n",
            "\n",
            "dev.en.1            100%[===================>]  16.65K  --.-KB/s    in 0.001s  \n",
            "\n",
            "2021-07-18 12:08:08 (15.6 MB/s) - ‘data/dev.en.1’ saved [17054/17054]\n",
            "\n",
            "--2021-07-18 12:08:08--  https://www.dropbox.com/s/kdgskm5hzg6znuc/dev.ja\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/kdgskm5hzg6znuc/dev.ja [following]\n",
            "--2021-07-18 12:08:08--  https://www.dropbox.com/s/raw/kdgskm5hzg6znuc/dev.ja\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com/cd/0/inline/BShJS8UndOASy5GQQSPsXjT-MMUHly5DIb6KV0kZrmfT4A_b9fPxvLeKehfibExx7MG1ys6oVBrg8-jcYufSRON374Tzx_6v5p94c-gNLlB_x6lWBvI78qv4IlWecRnMHSbgOaqPp5xnAlcwakRE-quL/file# [following]\n",
            "--2021-07-18 12:08:08--  https://uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com/cd/0/inline/BShJS8UndOASy5GQQSPsXjT-MMUHly5DIb6KV0kZrmfT4A_b9fPxvLeKehfibExx7MG1ys6oVBrg8-jcYufSRON374Tzx_6v5p94c-gNLlB_x6lWBvI78qv4IlWecRnMHSbgOaqPp5xnAlcwakRE-quL/file\n",
            "Resolving uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com (uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6022:15::a27d:420f\n",
            "Connecting to uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com (uc48ad60be0b228ded9d109fc845.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 27781 (27K) [text/plain]\n",
            "Saving to: ‘data/dev.ja.1’\n",
            "\n",
            "dev.ja.1            100%[===================>]  27.13K  --.-KB/s    in 0.07s   \n",
            "\n",
            "2021-07-18 12:08:09 (370 KB/s) - ‘data/dev.ja.1’ saved [27781/27781]\n",
            "\n",
            "--2021-07-18 12:08:09--  https://www.dropbox.com/s/gyyx4gohv9v65uh/test.en\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/gyyx4gohv9v65uh/test.en [following]\n",
            "--2021-07-18 12:08:09--  https://www.dropbox.com/s/raw/gyyx4gohv9v65uh/test.en\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com/cd/0/inline/BSjWxDLfydpQZeBMD313_Q8p6fbw5BknPyCeefaJvJ4gyEkcM8ttsW2aVa-zfLtXGwGgp4o5LiFEgjgl5EcynVdCuZvG3F0tRjSmbe_BykLTiZ73wXWAvIuvtRBNajE4ZXGd3ZFZjTN01MAkqZl6E9za/file# [following]\n",
            "--2021-07-18 12:08:09--  https://ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com/cd/0/inline/BSjWxDLfydpQZeBMD313_Q8p6fbw5BknPyCeefaJvJ4gyEkcM8ttsW2aVa-zfLtXGwGgp4o5LiFEgjgl5EcynVdCuZvG3F0tRjSmbe_BykLTiZ73wXWAvIuvtRBNajE4ZXGd3ZFZjTN01MAkqZl6E9za/file\n",
            "Resolving ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com (ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6017:15::a27d:20f\n",
            "Connecting to ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com (ucb8d14c3cbc1cbce1012d5c483d.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 17301 (17K) [text/plain]\n",
            "Saving to: ‘data/test.en.1’\n",
            "\n",
            "test.en.1           100%[===================>]  16.90K  --.-KB/s    in 0.001s  \n",
            "\n",
            "2021-07-18 12:08:10 (11.8 MB/s) - ‘data/test.en.1’ saved [17301/17301]\n",
            "\n",
            "--2021-07-18 12:08:10--  https://www.dropbox.com/s/hotxwbgoe2n013k/test.ja\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/hotxwbgoe2n013k/test.ja [following]\n",
            "--2021-07-18 12:08:10--  https://www.dropbox.com/s/raw/hotxwbgoe2n013k/test.ja\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com/cd/0/inline/BSgTviouSpD9VGsNL8yCbZdFlqLeah68bzs-zu1qWajVW8dTfh2dm_KyrbZU7b_sNf7GCn46U1tZ55HuTmhd848pZT61O36vPpFRR-DIklYbM91pFbHxTPJh-7C00U_99FswPq5ShDEB-lfg2m08wNzY/file# [following]\n",
            "--2021-07-18 12:08:10--  https://uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com/cd/0/inline/BSgTviouSpD9VGsNL8yCbZdFlqLeah68bzs-zu1qWajVW8dTfh2dm_KyrbZU7b_sNf7GCn46U1tZ55HuTmhd848pZT61O36vPpFRR-DIklYbM91pFbHxTPJh-7C00U_99FswPq5ShDEB-lfg2m08wNzY/file\n",
            "Resolving uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com (uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com)... 162.125.4.15, 2620:100:6017:15::a27d:20f\n",
            "Connecting to uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com (uc8a502e4ecc5cc71aa3b2a82d48.dl.dropboxusercontent.com)|162.125.4.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 27793 (27K) [text/plain]\n",
            "Saving to: ‘data/test.ja.1’\n",
            "\n",
            "test.ja.1           100%[===================>]  27.14K  --.-KB/s    in 0.07s   \n",
            "\n",
            "2021-07-18 12:08:11 (370 KB/s) - ‘data/test.ja.1’ saved [27793/27793]\n",
            "\n",
            "--2021-07-18 12:08:11--  https://www.dropbox.com/s/5lsftkmb20ay9e1/train.en\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/5lsftkmb20ay9e1/train.en [following]\n",
            "--2021-07-18 12:08:11--  https://www.dropbox.com/s/raw/5lsftkmb20ay9e1/train.en\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com/cd/0/inline/BSiY7o32yWwPhAOtzpjqM73yDUgwxtaIxCJ4JNcCh8LPNYG7NcBoD8VzAPYo88OU6x47wori6x_CVKoa_FsSQuE4MTocf_GhYLOtba2tr5NdKUvVFDQ3Bq7qcX_lqowbKoSOxAgiV_5zwU-jHZRB_hh6/file# [following]\n",
            "--2021-07-18 12:08:11--  https://ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com/cd/0/inline/BSiY7o32yWwPhAOtzpjqM73yDUgwxtaIxCJ4JNcCh8LPNYG7NcBoD8VzAPYo88OU6x47wori6x_CVKoa_FsSQuE4MTocf_GhYLOtba2tr5NdKUvVFDQ3Bq7qcX_lqowbKoSOxAgiV_5zwU-jHZRB_hh6/file\n",
            "Resolving ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com (ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6017:15::a27d:20f\n",
            "Connecting to ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com (ucaf21ac6da73708f845506de743.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1701356 (1.6M) [text/plain]\n",
            "Saving to: ‘data/train.en.1’\n",
            "\n",
            "train.en.1          100%[===================>]   1.62M  10.8MB/s    in 0.2s    \n",
            "\n",
            "2021-07-18 12:08:11 (10.8 MB/s) - ‘data/train.en.1’ saved [1701356/1701356]\n",
            "\n",
            "--2021-07-18 12:08:11--  https://www.dropbox.com/s/ak53qirssci6f1j/train.ja\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6017:18::a27d:212\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: /s/raw/ak53qirssci6f1j/train.ja [following]\n",
            "--2021-07-18 12:08:12--  https://www.dropbox.com/s/raw/ak53qirssci6f1j/train.ja\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com/cd/0/inline/BSjOUBH4vUgdjmnAtTH8v1AlA62Kd3MY-H414kKwrEAjHhKiVisM6k4KIPoLM3sFQIhdY7tO3t8ZEMAFeI44Un_7sGFgBljiVq3uWN7WSeaxNbFntjlRv6eueoKJuCSvUbM5HT2HZ_xkh3uUlKCC8bEE/file# [following]\n",
            "--2021-07-18 12:08:12--  https://uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com/cd/0/inline/BSjOUBH4vUgdjmnAtTH8v1AlA62Kd3MY-H414kKwrEAjHhKiVisM6k4KIPoLM3sFQIhdY7tO3t8ZEMAFeI44Un_7sGFgBljiVq3uWN7WSeaxNbFntjlRv6eueoKJuCSvUbM5HT2HZ_xkh3uUlKCC8bEE/file\n",
            "Resolving uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com (uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6017:15::a27d:20f\n",
            "Connecting to uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com (uce89486d5d0b1d7bfce195839fb.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2784447 (2.7M) [text/plain]\n",
            "Saving to: ‘data/train.ja.1’\n",
            "\n",
            "train.ja.1          100%[===================>]   2.66M  --.-KB/s    in 0.06s   \n",
            "\n",
            "2021-07-18 12:08:12 (43.8 MB/s) - ‘data/train.ja.1’ saved [2784447/2784447]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zmG3Ihbpp2aN"
      },
      "source": [
        "import time\n",
        "import numpy as np\n",
        "from sklearn.utils import shuffle\n",
        "from sklearn.model_selection import train_test_split\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "\n",
        "from nltk import bleu_score\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "from utils import Vocab\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "torch.manual_seed(1)\n",
        "random_state = 42"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e50saJX2p2aQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67a161fd-8dfd-43e1-d2b5-0cd1787ae228"
      },
      "source": [
        "print(torch.__version__)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.9.0+cu102\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YxRYfAn4p2aW"
      },
      "source": [
        "PAD = 0\n",
        "UNK = 1\n",
        "BOS = 2\n",
        "EOS = 3\n",
        "\n",
        "PAD_TOKEN = '<PAD>'\n",
        "UNK_TOKEN = '<UNK>'\n",
        "BOS_TOKEN = '<S>'\n",
        "EOS_TOKEN = '</S>'"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLfYkF9Cp2aZ"
      },
      "source": [
        "## 1.データセット"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8yesDStlp2aa"
      },
      "source": [
        "def load_data(file_path):\n",
        "    \"\"\"\n",
        "    テキストファイルからデータを読み込む\n",
        "    :param file_path: str, テキストファイルのパス\n",
        "    :return data: list, 文章（単語のリスト）のリスト\n",
        "    \"\"\"\n",
        "    data = []\n",
        "    for line in open(file_path, encoding='utf-8'):\n",
        "        words = line.strip().split()  # スペースで単語を分割\n",
        "        data.append(words)\n",
        "    return data"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AN7_LZUdp2ad"
      },
      "source": [
        "train_X = load_data('./data/train.en')\n",
        "train_Y = load_data('./data/train.ja')"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TkT-ADq-p2al"
      },
      "source": [
        "# 訓練データと検証データに分割\n",
        "train_X, valid_X, train_Y, valid_Y = train_test_split(train_X, train_Y, test_size=0.2, random_state=random_state)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rlncckt2p2as",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30f5e8c5-1233-412d-8281-6261b141238d"
      },
      "source": [
        "# データセットの中身を確認\n",
        "print('train_X:', train_X[:5])\n",
        "print('train_Y:', train_Y[:5])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_X: [['where', 'shall', 'we', 'eat', 'tonight', '?'], ['i', 'made', 'a', 'big', 'mistake', 'in', 'choosing', 'my', 'wife', '.'], ['i', \"'ll\", 'have', 'to', 'think', 'about', 'it', '.'], ['it', 'is', 'called', 'a', 'lily', '.'], ['could', 'you', 'lend', 'me', 'some', 'money', 'until', 'this', 'weekend', '?']]\n",
            "train_Y: [['今夜', 'は', 'どこ', 'で', '食事', 'を', 'し', 'よ', 'う', 'か', '。'], ['僕', 'は', '妻', 'を', '選', 'ぶ', 'の', 'に', '大変', 'な', '間違い', 'を', 'し', 'た', '。'], ['考え', 'と', 'く', 'よ', '。'], ['ｌｉｌｙ', 'と', '呼', 'ば', 'れ', 'て', 'い', 'ま', 'す', '。'], ['今週末', 'まで', 'いくら', 'か', '金', 'を', '貸', 'し', 'て', 'くれ', 'ま', 'せ', 'ん', 'か', '。']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0r_RMRdPp2a1"
      },
      "source": [
        "MIN_COUNT = 2  # 語彙に含める単語の最低出現回数\n",
        "\n",
        "word2id = {\n",
        "    PAD_TOKEN: PAD,\n",
        "    BOS_TOKEN: BOS,\n",
        "    EOS_TOKEN: EOS,\n",
        "    UNK_TOKEN: UNK,\n",
        "    }\n",
        "\n",
        "vocab_X = Vocab(word2id=word2id)\n",
        "vocab_Y = Vocab(word2id=word2id)\n",
        "vocab_X.build_vocab(train_X, min_count=MIN_COUNT)\n",
        "vocab_Y.build_vocab(train_Y, min_count=MIN_COUNT)\n",
        "\n",
        "vocab_size_X = len(vocab_X.id2word)\n",
        "vocab_size_Y = len(vocab_Y.id2word)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wMP4omn-p2a5"
      },
      "source": [
        "def sentence_to_ids(vocab, sentence):\n",
        "    \"\"\"\n",
        "    単語のリストをインデックスのリストに変換する\n",
        "    :param vocab: Vocabのインスタンス\n",
        "    :param sentence: list of str\n",
        "    :return indices: list of int\n",
        "    \"\"\"\n",
        "    ids = [vocab.word2id.get(word, UNK) for word in sentence]\n",
        "    ids = [BOS] + ids + [EOS]  # EOSを末尾に加える\n",
        "    return ids"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6qCMnmqFp2a7"
      },
      "source": [
        "train_X = [sentence_to_ids(vocab_X, sentence) for sentence in train_X]\n",
        "train_Y = [sentence_to_ids(vocab_Y, sentence) for sentence in train_Y]\n",
        "valid_X = [sentence_to_ids(vocab_X, sentence) for sentence in valid_X]\n",
        "valid_Y = [sentence_to_ids(vocab_Y, sentence) for sentence in valid_Y]"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "umBYKCRFp2a-"
      },
      "source": [
        "class DataLoader(object):\n",
        "    def __init__(self, src_insts, tgt_insts, batch_size, shuffle=True):\n",
        "        \"\"\"\n",
        "        :param src_insts: list, 入力言語の文章（単語IDのリスト）のリスト\n",
        "        :param tgt_insts: list, 出力言語の文章（単語IDのリスト）のリスト\n",
        "        :param batch_size: int, バッチサイズ\n",
        "        :param shuffle: bool, サンプルの順番をシャッフルするか否か\n",
        "        \"\"\"\n",
        "        self.data = list(zip(src_insts, tgt_insts))\n",
        "\n",
        "        self.batch_size = batch_size\n",
        "        self.shuffle = shuffle\n",
        "        self.start_index = 0\n",
        "        \n",
        "        self.reset()\n",
        "    \n",
        "    def reset(self):\n",
        "        if self.shuffle:\n",
        "            self.data = shuffle(self.data, random_state=random_state)\n",
        "        self.start_index = 0\n",
        "    \n",
        "    def __iter__(self):\n",
        "        return self\n",
        "    \n",
        "    def __next__(self):\n",
        "\n",
        "        def preprocess_seqs(seqs):\n",
        "            # パディング\n",
        "            max_length = max([len(s) for s in seqs])\n",
        "            data = [s + [PAD] * (max_length - len(s)) for s in seqs]\n",
        "            # 単語の位置を表現するベクトルを作成\n",
        "            positions = [[pos+1 if w != PAD else 0 for pos, w in enumerate(seq)] for seq in data]\n",
        "            # テンソルに変換\n",
        "            data_tensor = torch.tensor(data, dtype=torch.long, device=device)\n",
        "            position_tensor = torch.tensor(positions, dtype=torch.long, device=device)\n",
        "            return data_tensor, position_tensor            \n",
        "\n",
        "        # ポインタが最後まで到達したら初期化する\n",
        "        if self.start_index >= len(self.data):\n",
        "            self.reset()\n",
        "            raise StopIteration()\n",
        "\n",
        "        # バッチを取得して前処理\n",
        "        src_seqs, tgt_seqs = zip(*self.data[self.start_index:self.start_index+self.batch_size])\n",
        "        src_data, src_pos = preprocess_seqs(src_seqs)\n",
        "        tgt_data, tgt_pos = preprocess_seqs(tgt_seqs)\n",
        "\n",
        "        # ポインタを更新する\n",
        "        self.start_index += self.batch_size\n",
        "\n",
        "        return (src_data, src_pos), (tgt_data, tgt_pos)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-rcK5zFp2bB"
      },
      "source": [
        "## 2.各モジュールの定義\n",
        "TransformerのモデルもEncoder-Decoderモデルの構造になっています。\n",
        "EncoderとDecoderは\n",
        "- Positional Encoding: 入出力の単語のEmbedding時に単語の位置情報を埋め込む\n",
        "- Scaled Dot-Product Attention: 内積でAttentionを計算し、スケーリングを行う\n",
        "- Multi-head Attention: Scaled Dot-Product Attentionを複数のヘッドで並列化する\n",
        "- Position-Wise Feed Forward Network: 単語列の位置ごとに独立して処理を行う\n",
        "など、いくつかのモジュールから構成されているため、それぞれのモジュールを個別に定義していきます。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IDcDkr4ip2bF"
      },
      "source": [
        "### ① Position Encoding\n",
        "Transformerは系列の処理にRNNを使用しないので、そのままでは単語列の語順を考慮することができません。\n",
        "\n",
        "そのため、入力系列の埋め込み行列に単語の位置情報を埋め込むPosition Encodingを加算します。\n",
        "\n",
        "Positional Encodingの行列$PE$の各成分は次式で表されます。\n",
        "$PE_{(pos, 2i)} = \\sin(pos/10000^{2i/d_{model}})$\n",
        "\n",
        "$PE_{(pos, 2i+1)} = \\cos(pos/10000^{2i/d_{model}})$\n",
        "ここで$pos$は単語の位置を、$i$は成分の次元を表しています。\n",
        "\n",
        "Positional Encodingの各成分は、波長が$2\\pi$から$10000*2\\pi$に幾何学的に伸びる正弦波に対応します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9T8VxiHfp2bG"
      },
      "source": [
        "def position_encoding_init(n_position, d_pos_vec):\n",
        "    \"\"\"\n",
        "    Positional Encodingのための行列の初期化を行う\n",
        "    :param n_position: int, 系列長\n",
        "    :param d_pos_vec: int, 隠れ層の次元数\n",
        "    :return torch.tensor, size=(n_position, d_pos_vec)\n",
        "    \"\"\"\n",
        "    # PADがある単語の位置はpos=0にしておき、position_encも0にする\n",
        "    position_enc = np.array([\n",
        "        [pos / np.power(10000, 2 * (j // 2) / d_pos_vec) for j in range(d_pos_vec)]\n",
        "        if pos != 0 else np.zeros(d_pos_vec) for pos in range(n_position)])\n",
        "    position_enc[1:, 0::2] = np.sin(position_enc[1:, 0::2])  # dim 2i\n",
        "    position_enc[1:, 1::2] = np.cos(position_enc[1:, 1::2])  # dim 2i+1\n",
        "    return torch.tensor(position_enc, dtype=torch.float)"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TaqSqkAup2bJ"
      },
      "source": [
        "ちなみに、Position Encodingを可視化すると以下のようになります。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cUu6Kxlop2bK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "outputId": "1b412ae5-c69d-4b1b-d8bc-56b554af6d14"
      },
      "source": [
        "pe = position_encoding_init(50, 256).numpy()\n",
        "plt.figure(figsize=(16,8))\n",
        "sns.heatmap(pe, cmap='Blues')\n",
        "plt.show()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAHfCAYAAACxoFIrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZgcVdk28PuZfd+zTvYdkkAIkbCIkgBhlSDKFtaAhB30cwEV8AUEQUVUJEDUEFlkFUOACEQggEIgA7IkBMhO9j0zmX165nx/VPU5p6erepnpTnom9++66nqfefpU1emaid93qO66RSkFIiIiIiIiik3avp4AERERERFRV8JFFBERERERURy4iCIiIiIiIooDF1FERERERERx4CKKiIiIiIgoDlxEERERERERxYGLKCIiIiIiSmkiMltEtorIEp/XRUT+KCIrROQTERlvvXaRiCx3t4sSMR8uooiIiIiIKNXNAXBihNdPAjDc3WYAeAAARKQMwC8ATARwGIBfiEhpZyfDRRQREREREaU0pdRbAHZGGDIVwCPKsQhAiYj0AXACgAVKqZ1KqV0AFiDyYiwmXEQREREREVFXVwlgnfXzerfn1++UjM4eIJpr/7lMJfscRERERERdzX3fPkD29RzilXvINUn5/9s3fnT/5XA+hhc0Syk1KxnnSoSkL6KIiIiIiIgicRdMnVk0bQDQ3/q5n9vbAOCYdv2FnTgPgBgWUSIyCs5nDIO3vTYAmKeUWtbZkxMRERERURciKfttoHkArhGRJ+E8RKJaKbVJRF4BcKf1MIkpAH7a2ZNFXESJyA0AzgXwJID33XY/AE+IyJNKqbs6OwEiIiIiIqJIROQJOHeUKkRkPZwn7mUCgFLqQQDzAZwMYAWAegDT3dd2isjtABa7h7pNKRXpARWxzUcp/481isiXAEYrpVra9bMALFVKDffZbwbczzQec8X/HTpmylmdnScRERERUbfSJb8Tdej1SflOVMMHf+hS1yLax/naAPQFsLZdv4/7mif7M418sAQRERERUTeRuh/n26uiLaK+D+A1EVkO82jAAQCGAbgmmRMjIiIiIiJKRREXUUqpl0VkBJx0X/vBEouVUq3JnhwREREREaUQ6VKfukuaqE/nU0q1AVjU0RP85YEXAQAVI0bo3vbV5tOBxX376rp6m/mOV15xIQCgsb7RTDbDmq71+ws0B3Sdk58DAKjfXaN7RT3KdF2zeYuuy/qbnK2dq1cDAHoMH6Z72z43DyDsM3asrjd9/JGu+x86HgCw7r33dW/wUYfrevXb/9H18EnfAAAsf/1N3TtgyiRdL3v537oec8oUXS958WUAwMFTT9K9j+fO1/X475yq6w+fnafrCWdNBQBUPTVX9yZOO0PX7z3+rK6PuOBMXb/76DMAgK9fZL7L9p85T+n6G5eco+u3Zj8JAJj0vXN1740/P67rY2ecr+vXZj2m6ylXXAAAePXBR3XvhCsv0PUrMx/R9cnXXKTr+X/6GwDglGsv1r2X7puj61OvM/0X/2j6U6+fDgB4/vd/1b3Tf/A9Xc+99y+6PuP/Of3nfmd63/nhZbr+xz1/1vV3f2T6z/72z749ADjzxyb64JnfmCd4nv0Tp//Ur03vnBsu1/WTdz8UsX+u1XvCGuvVP+/GK3Tv8bse1LVf//yfOv3HfhXei6V/wc+u1L1H73xA19H6du/Cn5uxj9wR3vfqxdK/+KardG/OL2fqOlo/nrF2f/rNpvfw7WZstH6yxtr9RMztkluu1vXs2+6P2E/WWLsfz9i9fb5UnltHz5fKc+vo+VJ5bnv7fKk8t46eL1Xmdt+3/wTqmpgTRUREREREseF3ogBwEUVERERERLHix/kAAFxKEhERERERxYF3ooiIiIiIKDb8OB8A3okiIiIiIiKKC+9EERERERFRbPidKAC8E0VERERERBQX3okiIiIiIqLY8DtRAPbGIqreCb3969VH6tbUi97W9T23mfDX7132a13/7N7vAwBu+v7vdG/G7dfq+k8336frYFgpADz1GycgdcqVF+req7P+ruuJ531X1+89ZYJpx5x6IgBgycuv696Qb3xd16sWVem690EH63rdJ58BAMpHH6R7qz9doevCEWN0vXyJEzKcM2iU6S1dp+uM/iN1vWLZBl2jz3AAwMovNptez0Fmbsu3mn55PzOPldudorS37q1ZbQKNUdxTl2vX7jb9wnIAwFfrqk0vv1SXGzaaIGPkOqHIGzbtMb2cAl1u2lpr+tn5pr/N7Wfl6t6WHfVmbGa26e8M72/d1WB6GVm63F5twpmRbv68t9W4/fRM3duxxx5r+jtrm5wiLV33dtU1mbFWv7quOaxfU99ietb/0NQ2ePf3BPv22EbvsfVNgbB+SM/S0Nzq0fMe29gSPtav3xRo8xzr1W8OeB+3pdX7GF79Fp/zefVbW5XnWK9+wGcOfv3WtvBjePUSMTbeYyiPtlcvEWP9j+E3t/B+ssYm83xEROTix/kA8ON8REREREREcYm6iBKRUSJyrIgUtOufmLxpERERERFRypG05GxdTMQZi8h1AJ4HcC2AJSIy1Xr5zmROjIiIiIiIKBVF+07UZQAOVUrVisggAM+KyCCl1B8A8AORRERERET7E34nCkD0RVSaUqoWAJRSa0TkGDgLqYGIsIgSkRkAZgBAxtCTkdF7fIKmS0RERERE+0wX/OhdMkS7CltEZFzwB3dBdSqACgBj/XZSSs1SSk1QSk3gAoqIiIiIiLqTaHeiLgQQ8kxkpVQAwIUi8lDSZkVERERERKmHd6IARFlEKaXWR3jtv7Gc4JRLvw0AmLVoHR4937krddC3v6VfP210X133OnqKri8Y3x8AcMcYky911eEDdf2n/qN1/aOjh+j6qb84mUg3Thqme68+nKPrHx47VNdnPWayf6443jnGNc8+q3vnTDLHvXO+yZQ6+Rvf1vXshf8CABw9bbLuzX3vDV1POGWCrt+Y/TQAYPRZ5vkcHzz3L12POuE4XX/++n90PeDwiQCArz74n+71PNC8/63LTS5V6QBzjXZs2AIAyO9jsqO2bdim66yelbrevnG7rtPKnd/J9k1WplSZ+T1t3bTL9Euc671ti5UpVVRhxm6xMqUKTNbUtq11TpFXZOaw3cqD8uu7WVO7dlk9K2tq9+5Gz351MD/Kyp+qrrGyn6ysqeo9Tv+Qs7+D//3jBQBATa2VB2VlSnnlRO2x86CsrKqQvpU1VdsYCOvVNdp5UBKxH5ITZR0jJBPK/R+85+6djZOuviikB7TLg/LqW70mn7HNHplSfhlPXmP9xvtlSgXczKQzfzwDz/xmltvzGxveD8SZ2+SVH9XmM9arH89Yv75fhFE8mVJtHgfx6sXbT1YuVbwZVsmSyFyq6TdfhYdvnxlxbFcz+7b7ccktV+/raSTUJbdcjdm33b+vp0FEKSr5Ybuu4AKKqKsILqC6E72A6kaCCyiiriK4gOpOutsCCgAXUER+0vhgCWAvLqKIiIiIiKiL48f5AMQQtktEREREREQG70QREREREVFsmBMFgHeiiIiIiIiI4sI7UUREREREFBt+JwoA70QRERERERHFhXeiiIiIiIgoNvxOFIC9sIj67bcOBACs3lane/efc4iu65pN6Oavp5ssqdwsJzT06vO+pnt9Skxo7infnqjrob0KdH3QcUcBAMb0M2Gtvb5mAnsnDioz5zjAHHvykJ5OYYX4fntUb13faYXNnj+2j65n5zjnnnaI6c21bnN+Z7w5xhsPOYGvJ4wzYz943ITRHnmQ6X8+d6uux4/uBQD46t+bdG/0gSbcd+uiN3U95FgT7vvBp1XOWxrvHeLb+3BzDb2CfP1CfHdtNoG9+eXO9azebsJ2s0pN2G7NDvP+0oqt/k63X9RT96p32sG85WbsbvO3g/xiAMDuXd7BvLt3NZh+Tr45dnWTR887mHePG7aLTPP3VmuH7VqBvXV22K4b2BvSs4J56+rtvvmnVx8M4bXDdn2Ceb2CdRuavcNvQ/ru2NBQXfM/go3N3v2mYN/u+QTlNttBue48mjx6ANDc6t3Xwbp2zyewt9Uj/NZvbKA1PNDUKzwX8A/h9QqY9Qv3bfUY6xd+69f3ymD1C+aNJ/A1nhDfePrxBPPGG+4bj2SF++5tHQnxjbVPRNQp/DgfAH6cj4iIiIiIKC78OB8REREREcWGH+cD0IE7USLySDImQkRERERE1BVEvBMlIvPatwBMEpESAFBKnZasiRERERERUYrhd6IARL8T1Q9ADYDfAbjH3fZYtScRmSEiVSJS9fdH/pqouRIRERER0b4kkpyti4n2nagJAK4H8HMAP1ZKfSQiDUqpNyPtpJSaBWAWAKzZ3sjHAxERERERUbcRcRGllGoDcK+IPOP+3y3R9iEiIiIiom6KH+cDEOOCSCm1HsCZInIKnI/3ERERERER7ZfiuquklHoJwEvx7FNR6ASTnnz/O7r36nVH6fr/Xv1S1z+dNEzXC790Al0vO2yA7q3YXKvrn002Y3dbIaY3nDwibA6XnDpK10W5Jvx0ynEH6DoY5DvuKBO2O7BHnq57H3SwrodZ4b55w53+2L7F5oSV5nwTK024L8r7AQCOH2xCZ+/MNUGxJw03Y2e7wa0AcMIBTn+uFZz49eEmjPaNFhMaO35YD11/0OCsd0cPNWM/n7fDvI8hpbr+6o3tuh4w0OlvrTKhur0rzfvftewTXVccMAgAsLZqje6Vjh6j6y1fLjf9gVZg78YtAICCcvOea3eb3292UZFnP73QmVtdtR3Aa95HXY3Vzyux+m44b4753dXVWmG7Vghv7Z7msJ4O4AWALL8QXqdfXx8ewOv0Wzz7DcFgXSuAt7HRCtW1AnsbG8NDeBs8egDQ6BXM69ED2gXoWv+FSYfl2kG5IQG6ErHf7BPu6xeK69UPCb/1Cuy1en5jvYJ1vQJ4Ae8QX7/x8QToxhsw6xXY6zu2kwG6iQhxTVagbbzXLRGBvV66YmCvl/h+pyn8Roho3+mC319KBn40j4iIiIiIYsOP8wHoQE4UERERERHR/ox3ooiIiIiIKDa8EwWAd6KIiIiIiIjiwjtRREREREQUGz5YAgDvRBEREREREcWFd6KIiIiIiCg2/E4UgL1wJ2rukg2Yu2QDZnxzIBY/+U8sfvKfWLmlTm9/mLVQb4G2Nr398NH/4YeP/g/H3fUGehZlo2dRNn752nK9jepbqLfnP9uot6OH9MDRQ3qgsaUNS9fXYOn6Gpw9tq/eNuxq0NtVRwzUW11TAHVNAbx4zZG45JiBuOSYgUgX0dvJ3xiit8LcDL0ddsQQHHbEEPQszNbb8IOH6q2yNFdvpcNGoHTYCJzxm9cxoDwPA8rzkNF/pN5G9ijSGyr66+3gnqU4uGcp3vnnHUBBGVBQhiP6legNWbl6O3pQkd6Qlg6kpeMf987GxMFFmDi4CGhr1du4ASV6Q0uj3kb1L8Go/iU4//vTgMZaoLEWQ/qX6A311XqrrCxCZWURjpz6TaB2F1C7C737FOoNe3bqraJnkd6wZzuwZztq13yJ4vIiFJcXQe3ZobfC0kK9tezZrbf8onzkF+UjMzsTDbX1aKitR15hvt7qa+v1lplfoLdgL7usBySvGJJXjIbaBr0ht0hvDXUNaKhrQPHgYUB2HpCdh4a6Rr0hO19vDfXNekN2LpCdixWLPkRdXQvq6lqcTCl3q69v0RsysvXW0NCChoYWDJk82cmPysjSvYaGFt1DRhYaGwN6Q3omkJ6JD577F5qaWtHU1OpkTblbU1NAb8G/hUVPPI/m5lY0N7fqHtLS0dTcqjev/sTzz3Ru34ugsblVb/bY5kCr3iBpgKRhway/o6W1DS2tbboHSdM9py96aw60oTnQhmMvv0D3WgJterO1tiq0tiqcdPVFuhdobdObLdCm9Bacw9zfz0abUk6ukDW31jalt5Dzub3Tf/A93WtrU3qzefWf+vUsz+O2KbN5nS90rDJztih3O+eGy83+Sukt2jEe+9WDnudTymxe/fNuvMLqKb2Fjg3vP3rnA/Didb5H7ngg4hz8oowu/PmVEd+zX3/OL2d6jo3m4puuijomOF97bLT3YXv49plxzSlo+s3R5xbP+bx+p7Nvuz/m37/tkluu7tDcZt92f4f2i+d8qTy3RJwvHt3xWqTy3FKe9f9uJ3SLelo5UUS+EJEVInKjx+v3ishH7valiOy2Xmu1XpuXiMuw1+5EXXbl7zu03yd3ntih/VZuqY0+yMNzSzZ0aL94VN11aof2O/KCezq036//+P0O7ffYXbNiHvvO43M7dI7KQyd0aL+aHdUd2q+5sTn6IFf1xo0dOkefsWM7tN+qt/7Tof3GnDIl5rHjv9Oxv733Hn+2Q/sdc+k5HdrvtVmPxTz2X/f/rUPnOO366R3ab+69f+nQft/90WUd2i8eT979UIf2O9dafMXj8bse7NB+F/zsyuiDOjDW9sgd3gu1aOzFVzzm/DL2BU48Y22JXgwl8nwdnVsq/39kU3luiThfPLrjtUjluVE4EUkHcD+A4wGsB7BYROYppT4LjlFK/cAafy2AQ6xDNCilxiVyTrwfR0REREREsbE+vZHQLbLDAKxQSq1SSjUDeBLA1AjjzwXwRILesaeIMxaRiSJS5Na5InKriLwgIneLSHEyJ0ZERERERPsHEZkhIlXWNsN6uRLAOuvn9W7P6zgDAQwG8LrVznGPuUhETk/EfKN9nG82gIPd+g8A6gHcDeBYAA8DOCMRkyAiIiIioi4gSY84V0rNAhD7d0n8nQPgWaVUq9UbqJTaICJDALwuIp8qpVZ25iTR7p2lKaUCbj1BKfV9pdR/lFK3Ahjit5O9knz9ucc7Mz8iIiIiIkoRIpKULYoNAPpbP/dze17OQbuP8imlNrj/dxWAhQj9vlSHRFtELRGR4LevPxaRCQAgIiMAtPjtpJSapZSaoJSaMPmM8zo7RyIiIiIi2n8tBjBcRAaLSBachVLYU/ZEZBSAUgDvWr1SEcl26woARwH4rP2+8Yr2cb7vAfiDiNwEYDuAd0VkHZzPJH4v4p5ERERERNStxHDXKOGUUgERuQbAKwDSAcxWSi0VkdsAVCmlgguqcwA8qUJzFQ4A8JCItMG5gXSX/VS/joq4iFJKVQO42H24xGB3/Hql1JbOnpiIiIiIiCgWSqn5AOa3693S7uf/89jvHQAdy5+JIKacKKVUDYCPO3KC637/FgAgd+R4NNTWAwB+MPdTM2D9Ml3+41Pz0cZ1r78CAPhi41G69/zTb+u69owxur77qaW6vmjCIADAXQtX6N4TF5ksogfeXWXGHjpA10s31AAAJg3uaaa2q0HX54/to+s9DQFdn/O1vgCcoMug4yb003VuVrquxx/ifJSzJD9T94YdaD7e2bMoW9cVQ8xXzvqU5AAAsiqH6t6A0jxzwgpzjBFlhaZfVAEAGNejxPSy83V5aF9rbJqZ56H9CwAAjykTWHqgNfalVvNJzmF9igAA7zSba9W/txn7caPJ6+rdq0DXy+ud692zp5nPhjqdiYaycjN2+1KTCVVUOgIAUPPVGt3L72d+Z/U7tum6oG9fXVdv2wkAyCs2c2uoM3POyssL66flmrGN9Y26Rk6Bd9+9tk0NTaaXlWvGNlgZVVnmd11f717PrBwzB+tvDOnm76Wx0e47/3ybmrzHNjVZ36d0f7/NzeE9px/w7Le0tPr3gJBHkja3WAG37n+lavLohY81x2gJhuR69dofww3fPfbyC3SuVKBVeY5tH77r1wOcYF6vuQXawufWPpw2Ur99IG+0vlc4aTzn8ws39er65bz6Bc569f3CYr36vnPzPG58obfe54vrEJ7iuRaJkKz3kSoS8btOtktuubp7BaQSJcrevxGVkvZa2G5wAUVElEjxBPMSEcWKCygib/vi43ypiGG7REREREREcdhrd6KIiIiIiKhr450oB+9EERERERERxYF3ooiIiIiIKCa8E+XgnSgiIiIiIqI48E4UERERERHFhHeiHFxEERERERFRbLiGArAXFlENSxcBAG6953rd+8VNs3U98tRTdX3LXxebHXsNdvZb8KXpbTIBuq9+uUXXm995Q9erth4DAHjlxQ91r/7ccbp+8EVzvKuONIG2M99dCwCYPc2MfaRqra6/M8YE6K7cYgJkD+9X7syh2oSunj7KhL/WWUGoU8e5fStP8MiDTIhvdqb5dOWYMb11XZTr/JoGDjdjywqydF3a38ythxXYm9HTCRPuXWxCXFFqjju4xITGIr9UlyODgb1WUOzYXiYU1w5eHdPHDam1gnlHWKG6sIJ5B/Yw/bdbnEDavlbvf00mS6yiwoTfftlgrnd5uTOn9W5YLwCUWMHD2+pNMG9B8XBdV69dDQDI69dL9+q3m2DenD7m2tbscI6RV2jesx2gm5mT49mX7Lywnh1u3Nxoh+3mhvczzXGbm1qsseZ3GhK2m5kd3rPCdkMCdKME8/qH8EYJ243SDwR8QnUD3gG6uu8XlBsSdBsefqsDcdvxCuEN6VniCcX1C+YNCWB1+/EE5fqdzz/Q1iPc12dsZ4N5/frJCub1k4hg1u4czBup39V0hWBeItr/8E4UERERERHFhB/nc0RcRIlIFoBzAGxUSv1bRKYBOBLAMgCzlFItkfYnIiIiIiLqbqLdiXrYHZMnIhcBKADwHIBjARwG4KLkTo+IiIiIiFIF70Q5oi2ixiqlDhKRDAAbAPRVSrWKyGMAPk7+9IiIiIiIKFVwEeWIlhOV5n6krxBAHoBit58NINNvJxGZISJVIlIV2L4kMTMlIiIiIiJKAdHuRP0VwOcA0gH8HMAzIrIKwOEAnvTbSSk1C8AsAMgdfx0foUNERERE1A3wTpQj4iJKKXWviDzl1htF5BEAxwH4s1Lq/b0xQSIiIiIiolQS9RHnSqmNVr0bwLNJnREREREREaUm3ogCsBdyooafehoA4LKJg3TvF9kmHPWB8w/V9eQzb9L1uTdcDgB4Ys4C3SubOEnXv3xmqTlJjglsffD9r5xi7Se69+FXu3W97t13dL1p9zG6fmXBZwCA1rMP1r2HF5qw3Yu/Zub/+KebdH3nySMBAC8v26x7Rw6qMOfb0aDrw/qWAQB21pnQ1ZOGl+m6qcUEhU4aZY4RvG06zgrxzc0yIadDh5t+Ua75qlrvAU6wbGm+6RX07qvr8kIT2ItyE9jbu9ANfS0ycxhUZIXtWtd7eDCwN8Mc64Ce5vdrB5COsPtuOO+AcqtnBfNWllvnazFBxj3L3PHN5rqWlprgWjTW6bKkxITXbmjYAwAoLDJjtzeaEN+8QjOPmvXrAAA5vct1r37XLjO2h7kue3bv0XVOnnM+O1Q3PdsE5YaE7VohvC3BYF2vAF4gJIS3xQ7hdcN2m+0A3Uzze2hsbA0b29QUJVQX0MG8gBWgGxKqawXa+oXtur/30J4VoBsIHwsAra3hAbp+wbzNgfBg3RafcN9WjxDekGBe33Bf0/cKqfUL5vUa2+oRwAt4B/P6HiOOYF6/ucUTzOsbwusVfup9iLiCeb3PFW8/OcGsycp2TVYAr5/9MZiXiBKLH+dzRHuwBBEREREREVmSfieKiIiIiIi6B96JcvBOFBERERERURx4J4qIiIiIiGLCO1EO3okiIiIiIiKKA+9EERERERFRbHgjCgAXUUREREREFCN+nM+R9EXUX6d/DQCwYkst+ruZQFMvOlm/fsigEl1njZqo659PHgYAeOI3f9G9n918hq5/9MMHdD32tJN0/cTzHzvFgLG6d++bq8yE6qt1+fqqrbpu/LwKALB621Td++S/n+p6z7VH6fqlt1fr+tenHuCc9wOTHXXK6D66/seSDbqeNq4/AGDZRpMtNLJHka631jTp+qh+Jj+q1s0BmjzcXCs7L+bQYSa3KCPd/GGPGOYcIy/bZPlUDjRji3LNr7+8j+mX5jtZQ5nl5n2UFliZUiW9dNknmCmVZ+Y2wMpcsrOPBtt5Tm6+0LAKk4FkG1hmjW0zmUJ9gjlRAZOj1NM+bou5hqUlVt/NlSouts5nZUoVFZn+5iann1dg9t/ZZMZm51Xqes8Wkw+WXV4MAKjetlP38ooLzenqTd5VVra5nk2NzpzTsswcmpu8c6K8+i3NVnaUldcV0nezn0Jym6xMqZCcqLSM8L5fHpTVDwTC86O8ejH1QzKlfLKfWp1/AxPPPxPvPe5kgAda7cwlO/spPIvJqwcAAZ/QJD3eLzvKL2vKFU+mFGDlSllz88vA8TqEf8ZT7GN9c6niOIbnHHz6XplJ8eYoxTM8EZlS3tczrkN4SsS16KzuFLnUkd/1Jbdcjdm33Z+sKRFRF7fX7kT1t0NViYgSJLiAIiJKJC6giLzxTpSDD5YgIiIiIiKKA78TRUREREREMeGdKAcXUUREREREFBMuohwRP84nIsUicpeIfC4iO0Vkh4gsc3slEfabISJVIlL1zyfmJHzSRERERERE+0q0O1FPA3gdwDFKqc0AICK9AVzkvjbFayel1CwAswBg8erqbvR8HyIiIiKi/RhvRAGI/mCJQUqpu4MLKABQSm1WSt0NYGByp0ZERERERJR6oi2i1orIT0REBwOJSC8RuQHAuuROjYiIiIiIUomIJGXraqJ9nO9sADcCeFNEerq9LQDmATgzlhOM7e8EkP7whWW6d8dJo3T91fZ6Xf9g+uG6rnTDVnsceazufXesCTn9kRXQ+YvTDjRjLvonAGDKFRfo3qsvfajrvNHmHA8usEJ4M7MBAM8tM6G52PC5LldurtX1po8/0vWO2uMAAO8sMgG86oLxun6uyhxvxuGDAQAvr9ymez85Zpiu/7Nyu67H9CnW9dZqJ4x1bA/zNbSaBhOk+vVBZmyLFUz6tcGlAIA06w9z5GAT4pudYa5hvwGlus53w3nLe5frXnFupq7zynuYfp7btwJ4y/KtYN58M7d++Vb4bXY+AGBAkdVLN+cYXJ4NLwNK3b4y77N3sXWMVnNdetrBum44b6nds4J5i4qs8zU5wbyFhdb7sIJ58/KtYzSZv9+cPKdfbQfz5pgQ4/pdu3WdVWB+D3U1zvisHHO+QHNA15VKTqkAACAASURBVGlZ3n3PsF0rmDdkrBvCGxrAa653ix22m+ERtpuRGd4DoobwxhTM6xWsGxKq6z1W9/3GWgIex4gp/NY6tlfQqW8YrUcIb0xjLV45oK0qytzsaxFHuG88c3D6XkGwsY/1DQL2PoSneIJnExEaG28Ib7Lm4SUVQni7ezAvERldccGTDBEXUUqpXQBucLcQIjIdwMNJmhcREREREVFK6kzY7q0JmwUREREREaU8fpzPEfFOlIh84vcSgF4+rxEREREREXVb0b4T1QvACQB2tesLgHeSMiMiIiIiIkpNXe+mUVJEW0S9CKBAKfVR+xdEZGFSZkRERERERJTCoj1Y4tIIr01L/HSIiIiIiChVdcXvLyVDtDtRREREREREALiICurM0/mIiIiIiIj2O0m/E/XpV9UAgNkz5+neb081sVPn/q1K1w+ceZCuP1tfAwD4vwvH6V5hjpnu8OMm6/rIISYUFsVOJvCNk0yI7asz5+h66g2X6/qJOQt0XTZuIgDgydetAN5CE5T6+KdWCO/OjbpcttmZZ/Xn5kGGW2tO1fUnH6zRdUvACfr990fmWDcfP0LXLy/foetvDDfnfn+d0z9ykOltqTZBscPLCnRdbYXwTuhbBABoajFBo4cOKIKXEVbYbmaGs7bu198E5eZmmaDUsl5mbIH7OykoM0HARVYwL4p66lIH8wI6hLdXnhVcm52ny755VoBuhgmbHVjijrdCRfuXWKG4ll52sG6bE/paUWT1rGDe0kIrbNcN5i20ey2NugwJ4bXCdvOCIcPNDbqXnWcdw+rbwbq1253fb1ax+T3u2b1H1znWNbLDctMzM9zp2gG85nyBlvBgXq8A3rCxIf3W4MnMHKy/p9AQXqvvBuuGhOr6he169X2DedPC+14BvO3GtnkE6HoF8Ib1LQGvcN8277GtXmG0HgG8gH9QqlcArl8GqNcxQjrWnDsblOs3D7+xfseO9Xx+u/v1OxvCG2/OameDWZOZ67q3Q3i9pMAUEoIBvEQG70Q5eCeKiIiIiIhSmoicKCJfiMgKEbnR4/WLRWSbiHzkbt+zXrtIRJa720WJmA+/E0VERERERDHZF3eiRCQdwP0AjgewHsBiEZmnlPqs3dCnlFLXtNu3DMAvAEyA80GGD9x920c4xYV3ooiIiIiIKDaSpC2ywwCsUEqtUko1A3gSwNQYZ3wCgAVKqZ3uwmkBgBNj3NcXF1FERERERJTKKgGss35e7/ba+46IfCIiz4pI/zj3jUvERZSIFInIr0TkURGZ1u61mRH2myEiVSJS9c8n5nR2jkRERERElAJEJFmbXj+424w4p/YCgEFKqYPg3G36W+LfvRHtO1EPA1gO4B8ALhGR7wCYppRqAnC4305KqVkAZgHA4lXVfKQNERERERH5stcPHjYA6G/93M/t2fvvsH78C4BfW/se027fhZ2YKoDoH+cbqpS6USk1Vyl1GoAPAbwuIuVR9iMiIiIiom4mWXeiolgMYLiIDBaRLADnAJhnDxCRPtaPpwFY5tavAJgiIqUiUgpgitvrlGh3orJFJE0p1QYASqk7RGQDgLcAFETe1XHJw+8DAIYdPh4r3ngLALBolVkovvLIi7ouuvBQXU+b4+RHPX3pYbr3sZs5BQA3nXGgeRPp5sKPPe4oAMCYflYeUs9Burxq4kBdP3G3yYQ64/KTAAB/uf953et1yARdv/T2anO8UvM7euqTzU5Ru1P3lm4282xYuUTXW2qcbKdlH6/VPTvD6b9WFlXmtw7Q9YLlzsNDTjrAnPfNVVt1PWmoyWLatMvkGQ0qyQcQmh11cC9zXRqDGUAAxvUzv85gHMbwSpMTZV/jykpzjGB+VFlPkxNVkG3+rArLzNjCXNOXoh5uz86OMseosPOVsvN12TPYt7KMKoussVYeTmVxeH5UL3tsm3n/5XZ+lJsTVVIQnh0FAAU+/XydE2V+B3l2NpaVNZWdG55BZWdHocXkgGVmF+q6sd4cI5gfZWdHZWSaa2z3g9dr9/bdKAjmUfnlRFmZULqfbo7bGjDXzTfPyc2PCsmUSsvwHut1jJDsJ+U5NpgJNXbqKfj0hZfDx/rmR4nHWL9MKesYHoFHrR75U2HHCI6NJQ8qyvm8juvX98sI8sqPijfjyTPPKY6xfsf16sebz+M12jeLq5OZUv5jO/8BjGTFEiUrO2rOL2fi4puuinl8V4hdmn7zVXj4dt9vLgDw+7fQBd4cUSfsi5gopVRARK6Bs/hJBzBbKbVURG4DUKWUmgfgOhE5DUAAwE4AF7v77hSR2+EsxADgNqXUzrCTxCnaIuoFAJMB/Nt6E3NEZDOA++I5UXABRUT7TkFxTP/to0sJLqCIaN+JZwHVVURbQBHR3qWUmg9gfrveLVb9UwA/9dl3NoDZiZxPxEWUUuonPv2XReTORE6EiIiIiIhS277IiUpFnXnE+a0JmwUREREREVEXEfFOlIh84vcSgF6Jnw4REREREaUq3ohyRPtOVC84Kb+72vUFwDtJmREREREREVEKi7aIehFAgVLqo/YviMjCpMyIiIiIiIhSEr8T5Yj2YIlLI7w2LfHTISIiIiKiVMU1lKMzD5YgIiIiIiLa70T7OF+nrXjpBQDAqdddrHtXPlzlOfadlSaEd9GzTvZL7hVH6N4N85bq+rkZE3Vth/DeeMrIsOOOm2wCe0f2sXJyeg3R5fRDKgEAf9mxXvdOn3y6rh/643O67n3wOF2/9q4bnFvW18xtiQnCRUONLpdtceqWrz7Xva01JlR1+dJ1uraDcBd/tgUAkPHt0br3xorduj5tdKWuX99hzn3sMOfZHxt2NeheZXGurmsaTMDq6Aor0NU990GV5lrZAY3D+pgA3fQ05z9H9O1r9s/ONGvzsh4msDcvywrhLXXGF+SY8FQpLNd1QY71p5lvjlGW44bUZpn30cMO5rUCZPsUWuG1bqhonyIr/NZSUWCNdbKlUVZoHbfVXKsSe2xICG9WWE8H8AL+IbzNzu8nJGy3uV6XWdmRQ3jtAN7cfHNdvEJ4Q0J1/cJ2rb4O1vUK4AWihvCGhOqmm9+1XwivDsX1C/H1CuG1Qm5DQnVDgnm9gmujHLfdsXWgbZRA3LC+ewy/oFy/EF4d2BnD+bwO4RvM6xUOav9gved4Qnj9xnq14wkjjSfw10880afxhtHuzRDeZGa4JiuENx4pMIWkYggvdRdpabwVBfBOFBERERERUVySfieKiIiIiIi6B34nysFFFBERERERxYRP53PE/XE+EemZjIkQERERERF1BRHvRIlIWfsWgPdF5BAAopTa6bPfDAAzACCj/yRkVIxJxFyJiIiIiGgf4o0oR7SP820HsLZdrxLAh3AeeDQkbA8ASqlZAGYBQO746/g4GiIiIiIi6jaiLaJ+DOB4AD9WSn0KACKyWik1OOkzIyIiIiKilMLvRDkiLqKUUveIyFMA7hWRdQB+gfgiN4iIiIiIqJvgIsoR9el8Sqn1AM4UkdMALACQF88J8sY4Ybm/+daBunfAfTfoesoVF+j6h098ZJ8YAFC1ZpduLX7+NV3nXnOkrm/5lwmvfWr61wAAS9ebkNvvTxlqjmv93sccPV7Xw3q7wbI9B+ne+QeZAN2Hdm7U9cnf+LauZ9//PACg15iDdO/NxSawF6V9dDlv2TansAJ4v9y2R9eBdV/oesceE9i66vMNAIAmK6D0g89NqG7GGeY7Z2+tNsHDU8c4Ibxf7DTnmzTUPBdk0y4T0lpZZEJa9zQ6YaqjykzYbrMVeHpgH/MnEMwOHNzLhO2mWyFsvXt7h/CWlDuBvblZJhC1oMScLySEt6DM6rt/snkm8FcH8AIhIbwVueEhvL3ywwN4AaB3YXgIr1cALwCU2MewQniL8sLDdvPyvIN5c3Ot87kBurm5GZ5jM7PtseZ3pkN4rQDejCxzvRvqTMhyTl6Oc1grKDcYwNu+7xnC6xXAC0QN4fUK4AViCOH1CuD1O0YMQblex/AK4AViCOH1CuBtP9YjIdYrgDfsGPZ4r1BcFfv5/HI9vc7nF7QaTwhvPKG4fnPrLiG8ezOAN97zxSMVAngBhvASUWqK+el8Sql5ACYBOA4ARGR6siZFRERERESpRyQ5W1cT1yPOlVINSqkl7o+3JmE+REREREREKS3aI84/8XsJQK/ET4eIiIiIiFIVvxPliPadqF4ATgCwq11fALyTlBkREREREVFK4hrKEW0R9SKAAqXUR+1fEJGFSZkRERERERFRCov2iPNLI7w2LfHTISIiIiKiVMWP8znierAEERERERHR/i5qTlRn/fH6owEA/12zXecWpQ0ep1//tZUfNe60W3R92LlOFtMv5i9Dc7ObNdNoMpU+s3Kg3n3xP7ouuNLJpbpr4Qrde+BMk+G0fFOtrq84foiuM9xso2GHHax7Q3vlmzdi5T2dP9bUs3c4mVCTjzhF956Ys0DXZSPN+3v7QyfvaeAJ38Lad94FALy8fIc5h5UftWpHna6b1y8HAOysM9lBa5Zv1nWLlbnz0fLt5j2lO+/p3bXmun1rtMm+WrnbXIsjB1XoeluNkzvUpyhH92obTQbQiDJzXYL5UfdOHY2HFq0GEJrpMaCnyX6y86N69nSOcfqDi/Cva44CABSXmeynnEyTB5RfZM6Xl+3280vN69lmLHLN+UqzrYymTCczqtzOlEo3f/498q0sJjcP50+33I9b77ke7ZXle+dHFbv9M39wMZ65568AgKI867hWplSe3XczoUKyo5pNHlRoppTp6/wou5cVnj/l9Av163t2O38PwewoZ2omzyk9Iz2sn1tcrHOnQrKfouRHhfYyvMd6ZT95ZUcBntlPQycdg5Vvvh3Sc8ZGzo/66PmXMfbUKR7HjSE/SvdiyH5y5+GV5RTWj5If5ZUdBZicmeOvuAALHnrM2d93bGy9iMfwmK9f1o3X2/Yb69V+5jez8N0fXeY9wQjO/skMPPXrWRHP5zmHuM8Uu+A0zrvxCjx+14NRxobP5NE7H8AFP7sy7vMlmtffxZxfzsTFN10V97EuvukqzPnlzLj32xvRStNvvgoP3x7/3B6+fSam3xz/tUhll9xyNWbfdv++nkZCzb7tflxyy9Vx79cdr0W8eCPKsdfuRAUXUPHSC6huJLiA6k6CC6h4BRdQqchrARWL4AIqFQUXUPGyg3tTTXABFS+9gOpGgguo7qQjCygAegGViqItoPzEs4Da2zqygALQoQXU3tKRBRSAbreAAtAtFw0dWUAB3fNaUMck/U4UERERERF1D/xOlIOLKCIiIiIiignXUI64P84nIuXJmAgREREREVFXEHERJSJ3iUiFW08QkVUA3hORtSLyzQj7zRCRKhGpev25xxM8ZSIiIiIi2hdEJClbVxPtTtQpSqng495+A+BspdQwAMcDuMdvJ6XULKXUBKXUhMlnnJegqRIREREREe170b4TlSEiGUqpAIBcpdRiAFBKfSki2VH2JSIiIiKibqQL3jRKimiLqJkA5ovIXQBeFpE/AHgOwGQAHyV7ckRERERElDq64kfvkiHiIkopdZ+IfArgSgAj3PHDAcwFcHssJzh9rJMP9c9PN+jejVccreuBFXlmcGlvXd5z+lgAwNEX/lb3hh5vcl1u//eXZj838BYAVm11QmpfefFDc9iLJ+j6llfMfj+fPEzXa7fXAwDOm2QCeO3A194Hm4DgYb1MoGsw9PXsg8zcn9i6RteHn2W+OjZ/7vsAgIKhB+jefz/dZI6Va8JmF67Zafq1Tr1xl8nqadhgcpl217foevWKLboOuKGhH682x8pMN5/g/GCjCdudMtLM/8PqXQCAgytLzDnqzDn6Fubquq7JCZAdVmZ+jwEr8HRYD+v3a6mscAJ0g4HAAFBh/S1kZ5p5FpSY653r/k7yrADeXDtsN8/MOS/b+vPOdcJmS3KsMNpMEzZblmMF6LqhsD0LwgN4AaDCDua1lOS5x2gz2WZFedZxW801LPAI4Q0J1bWCeXNyrPfRYgKXs4PvL2B6IWG7ARO2m5EVPjYj01zXpgYzNsu6FoEWZx52AG+w57xgzhc1bNcK1W1t9e63Bf92QsZaIbfpHsG8HgG8kfvO35xXIG543yP8NiTENzyA1+8YfufzD7oNn0PUsb5zsN6fx0G8gn0jny/8hZBOlBBen9N5jvWbQzzhvn7iCeH14xdInAyJmO/eDODdF1JkGp2WiN81ESVX1EecK6UWAljYvi8i0wE8nPgpERERERFRKuKNKEfcjzi33JqwWRAREREREXUREe9Eicgnfi8B6JX46RARERERUarid6Ic0T7O1wvACQB2tesLgHeSMiMiIiIiIkpJXEM5oi2iXgRQoJQKexKfiCxMyoyIiIiIiIhSWLSn810a4bVpiZ8OERERERGlKn6cz9GZB0sQERERERHtd6I+4pyIiIiIiAjgnaigpC+ittU4IZ7X/f4t3fti5lm6/mjNbl1PnXasrsf0d4NnrXDQO84eq+tzbnxa18XjTXjvg+9/5RRrzYMFt1SbINF/zl+i6/vOGKPre95cAQA4fVQf3dtaY/Y7+RsmhLcw11y2wpHOnEb3MUG5yDBhpd89xITYzn9gufPejv2u7lW9s9zs1n+krt9ets0czw2FfW+T9XyP3SZU157nnvVf6bq20QlFXb1yu+7ZgYgfrzXHy8owNyU/3eqE8H59aIXufbFlj66HVpiQ1poG5xz9Ck1Qbn2zCVIdVm4CbYPhvwAwqIcJyw3qU2F6GWlmPuXl4SG8+XbYrhWKnF1g9bOsEF43yDgvy/qTzzFjS7OtUFz391di96zw1x4F3v9sKvLD+0V2qK4VwluYE97PCwnbNcG8uT59HcJrBfBm2QHDXiG8IWG7ZqwKtFh9c70bmpyA5wzrutkBumlWCG9o2K4zPhCIJZjXOnYwhNcvbDek7/49pXv02o31Cr8NGRslmDdkvFcAb7uxXuG1foG2rXEco9UvCDiuQFvrh2DwsE+wp1/gp1fb9xj2D5K4Dz/EE8KbiADeeKJP4wmeTUSmKkN49x8M4SVKHbwTRUREREREMeGNKAcXUUREREREFBN+nM/BB0sQEREREVFKE5ETReQLEVkhIjd6vP7/ROQzEflERF4TkYHWa60i8pG7zUvEfHgnioiIiIiIYrIvbkSJSDqA+wEcD2A9gMUiMk8p9Zk17H8AJiil6kXkSgC/BnC2+1qDUmpcIucU8U6UiEwQkTdE5DER6S8iC0SkWkQWi8ghEfabISJVIlL190f+msj5EhERERHR/uUwACuUUquUUs0AngQw1R6glHpDKVXv/rgIQL9kTijanaiZAH4BoATAOwB+oJQ6XkSOdV87wmsnpdQsALMAYM32Rj5KhoiIiIioG9hH34mqBLDO+nk9gIkRxl8K4F/WzzkiUgUgAOAupdTczk4o2neiMpVS/1JKPQFAKaWehVO8BiAn8q5ERERERNSdiCRrM59kc7cZHZufnA9gAoDfWO2BSqkJAKYB+L2IDO3sdYh2J6pRRKYAKAagROR0pdRcEfkmgNYo+xIREREREUVlf5LNwwYA/a2f+7m9ECJyHICfA/imUkoHqSqlNrj/d5WILARwCICVnZlvtEXUFXC+lNUG4AQAV4rIHHfSl8Vygh/OWwoAaFi6SPcyM87W9RWPfaDrJy83nw7cuKsRADDqxCm6d8ywnubAG7/U5dU/OFnXf3isyikGmGDeV5Zv1nXdkvdM3WRCb//2ihO2e93XTaju/M826fr8sSaEd0+jCRA98ghnfHlBtu6lDRit6/F9Ss2cW539Th1nAngXPf2SrkdNMWHDX3y20ezXcxAA4M0vdsLL0u3V5oedZr/ttc7fzvb15v3XN5m174pVVniv5eP1NQBMsC0ALN9Vq+tx/UvMMTY7/fICE0xba12f/lb4bWOLOffwCudGph0oWllmQl7tO8U9rLDdjHTnhZJS07ODgvOs0N/cLNPPyMt3e1YAb44JDQ4J4XXDjUPCdq1A2JJsK/zWCnQtzwv/51Sa6/1PrDA3PGw3pNdqrqEO1fXrewXwAu3CdjPCenaALgImsDndCtBtc0N4M/LMjeemRmtsSNCt+f2Ke73a7OBav7DdNI/AXjso1zds1+1bAa6BgB2UGzlA1y/E1yuYN6TvF4gbJYQ3NOTWZ6yl1eN8/uG34UHAyicettXjfP7BtX792I8R6/6AdyhuYoJkYz9f3MeOY+zeDuHtrGTOYW+H8KbC9UwWBvDS3pa2bz7OtxjAcBEZDGcdcg6cu0qa+7yGhwCcqJTaavVLAdQrpZpEpALAUXDWN50ScRGllPoYzuIp6Hp3g4hMh/M9KSIiIiIioqRQSgVE5BoArwBIBzBbKbVURG4DUKWUmgfn43sFAJ5xv7f1lVLqNAAHAHhIRNrgfJXprnZP9euQzjzi/FYAD3d2AkRERERE1DXsq6xdpdR8APPb9W6x6uN89nsHwFiv1zoj4iJKRD7xewlAr0RPhoiIiIiIUtc+ejpfyol2J6oXnI/ztf/yjIAf5SMiIiIiov1QtEXUiwAKlFIftX/BfbIFERERERHtJ9J4IwpA9AdLXBrhtWl+rxEREREREXVXnXmwBBERERER7Uf4nShH0hdR82fPdYq8Ygyf/E0AwD8+NdlYX7z4oq4H3WRykq59bgkA4I7vjtE9O/ckb+xRur7gkH66vvOG+wAAU664QPf+NH+5mZCVDbR0Q42u1733PgAgTcwT3Wf95ytdP3nx13S9crPJTDr3UCc/KtBmMmcOPGSwrvuUmHwdVDgZYd8cUGF69Sbj6ciDTBbV7Lff13Wv0c41WPL5NrNfiXmux9urrJwoKwdo7a56p9i2Vveq602m0Nb15njNVr7Oiq92AwjNAfh4Y52uLzzUZOqsrnGuxaAeJsNrk5vxBQCleSYbqL7ZZANV5uc657WyegaXm6wtO/aij5UJFZxTWZl3TlRhcb5nP7fAOZ+dHSW5heb1kPwo5xj5di/T/B6L7ZwoK/uoJJjRZGX1lHlkRwFAiXVdoJxrUGBnPLWZa1WQY+dH2ZlQmWG97Gw7+8nup4eNzcj0zp/KzArPq0rPtPKZas3cMq33EWgxxwhmTYXkQVlZW375UbrvlR3Vrh8c22vswdiy1Pnfi9ZW6w/HyrAKBKx+mpth5ZHlFHYMr0yoKHlQYX3dC8+qCj+GlfPkkf3ilfHk1/fLn/LKjwo5l30t/I4RR56TVx5QSCfKe/bLcvLPzPIeH6t4z+c5Nq7zJS87ivlBkUW7PNNvvgoP3z5z70wmCfj7J0quvXYnKriAIiJKpOACiogokbryAooomXgjysGP8xERERERUUwEXEUBTmovERERERERxSjiIkpEikXkLhH5XER2isgOEVnm9kr21iSJiIiIiGjfS5PkbF1NtDtRT8MJ2j1GKVWmlCoHMMntPe23k4jMEJEqEakKbP4wcbMlIiIiIiLax6ItogYppe5WSm0ONpRSm5VSdwMY6LeTUmqWUmqCUmpCRu/xiZorERERERHtQyKSlK2ribaIWisiPxER/TxtEeklIjcAWJfcqRERERERUSoRSc7W1URbRJ0NoBzAmyKyS0R2AlgIoAzAWUmeGxERERERUcqJ+IhzpdQuEXkYwAIAi5RSOmVWRE4E8HLUM+QVAQBmTz9Mt068db55vXKULr/cZEJsH3v0LQDAPc9coXuvfblV15ecZT4m2LvYCrQtdkJfb5w0TLcmX3afOd3hR+p65rsmhBY1zrHX72zQrUVvfq7rwiuP0PXjn27S9fVHDQIAbNptAmZPmlCpazvwtcfIkQCAfuW55ry5RWa/4WW6nr3LnOOg0U4I8YJ/fWx2qzSBvh+v2G6OZ4UJf7DJDRNuMKHC22qadN281dxM3NNoglI3fLUDQGjQ5pcbTKBvRrr5zwXLtjrX6+QDTLDpplpzLUb3Me9vT4M5R498J1i3sdkEkPYvMr/HgBXG2q/U+v26epWaa2iHApdaY7PSzbXPL3QCdLMzzDxz8szYnEzrvye41zAkgDfbnK8o0wqjzcgy/WBIrRUIW+oXtpubHtYrzAkP4AX8Q3jzgn0rKFeH6ob13bEtJow5K8sO5jX9kBDegPP3kpHhPTY9w4QbNzeafqYbSGyH6qZZvw+/sN3W1tawXshYO4Q3ONYjgBdAuwBd+xhOPxCwx9oBun79YNhu5FDdsL57DN/wW7+QWo/xIUOteXgdImT3kEDbKGP95ma/b68AXd9jeJ3PLwjY4s7ZP8TX73zxzK3zYaRdLdA0Me85ARPxEE/wcDKlyDSIUlZaV7xtlATRns53HYDnAVwDYImITLVevjOZEyMiIiIiIkpF0cJ2LwNwqFKqVkQGAXhWRAYppf4AMGmLiIiIiGh/whtRjmiLqLTgR/iUUmtE5Bg4C6mB4CKKiIiIiGi/0hWfpJcM0R4ssUVExgV/cBdUpwKoADA2mRMjIiIiIiJKRdHuRF0IIGA3lFIBABeKyENJmxUREREREaUc3ohyRHs63/oIr/038dMhIiIiIiJKbdHuRBEREREREQHgI86Don0nioiIiIiIiCxJvxN12VXfAgCM6W9CV+s+/o+ur/3ltbr+6fzPzI4bvwAQGvh601Of6HrudUfret0OE5A79rijAACj+5nzoXqbLi8/daSu7/zLu2bM4EMAAK+u3KJbbas/0rUdRvvS26t1/auTnbDguUs26N4pw3rquqahRdeHje8HACjONUGi6f1G6HpEj0J4mXxABQBgwZxVZrrjj9f1mhUmhBgVA3T5/qpdTmGFkX6+ywTvosaE9O6uM0Gp1Vuc69XQbIJd160zYbt2EOGyTXsAhIYKr6qu0/Vhg0yA8AYryLhHkRO2W9tkrmuffBNo22QFoQ4qy9Z1MIC0V7EZa/8HkbIS00+3QoGL3EDmzAzTyy0wY7OtsN3MPCdA3HkXwwAAIABJREFUNifTDts1obI5dghvpgnsLQyG7XoF8AIhv4eSnPB/eiW53v8c80NCeM3FzwsG6FoBvLn2WKuf7TE2NJjX/J1mZIYH9qZn+o01c24LmH66G2Tc2GyCl+3A3kDA/N7T0q0A3UB4gK4O1W3X18G6sYy1k1ndvlcgbnjfI7DXd2zkY8R7PhPu6zPW0uoRBOwXqhoaoOsGAccdaOsx31jO137/FBRvuG9cx45jbDzBs4m4np0N4U3m7zRVQni7i64WEE2ph/ehHPw4HxERERERxYSPOHfw43xERERERERx4J0oIiIiIiKKSRpvRAGIcidKRIpE5Fci8qiITGv32swI+80QkSoRqVry6tOJmisREREREdE+F+3jfA/D+f7YPwCcIyL/EJHgt/wP99tJKTVLKTVBKTVhzJSzEjRVIiIiIiLal0QkKVtXE+3jfEOVUt9x67ki8nMAr4vIaUmeFxERERERpZguuN5JimiLqGwRSVNKtQGAUuoOEdkA4C0ABUmfHRERERERUYqJtoh6AcBkAP8ONpRSc0RkM4D7YjnBzycPBQBU17fgq+31AIDyI47Tr/+/rw/R9eDT79J15aQTAQCvfrFZ91b9e4Gu+986Rdc/m/+5rn9yipMDZS+SM0dM0PW3D+yj61s+r9L11y8+GwDwyMK1ZscskyO0cnOtrjd9bPKj0tJOAgD8ffFG3Zs97RBdr3XfMwCcNrYHgNDMi+GjTa5TzyKTh4RSM88j+rpZSw0m4+nQUSaL6rO3zfuoGG5yp75cucMpCsp17/115n0gYLKhNtSYDCfsdN5LTYPJ8tm+aYeuW1pNhtPajc6c7PTqZVvMsXKs/KX1teZaDO7h5C5tqW7SvSIr48jOqOqVa7KYmt1z9y8xWUx25EXPYjPWnlOx289IM/PJLzS/XzvnKsfNOMq2esgx/83AzpRCtjlGfjA/ysqJKrRylJBu3l+xnRPlZvUU5VhZTJZCe6wy174g2Ld6uVneeU5Zbv+w75yI959+wZl6tnVcK+PJKz/KzoMKZkcBQHpGeKaU3W+z/lbSss11C7TYOVFWDlJwfHpGeA8IuYbBfsmAgdi9fj0AQHnkQYUdw73era3WWCurKhCwj2HNwyOLKWrGk3U+v4wnv75Xlotf1pTX2NZomVJRzgUAyifZKJ6cGa+h/llMHnOzf4jynp1jxH6+ZMXlxHV9kjMFdx5JPHg7j975AC742ZV774R7wcU3XYU5v3S+/t3do5WYHUXx6IofvUuGiN+JUkr9BMB6ETlWRAqs/ssArovnRF9Ziwki2jeCC6juJLiAIqJ9p7stoADoBRQRkZdoT+e7FsDzAK4FsEREplov35HMiRERERERUWpJk+RsXU20j/PNAHCoUqpWRAYBeFZEBiml/oDQT8wREREREVE3x4/zOaItotKUUrUAoJRaIyLHwFlIDQQXUUREREREtB+KlhO1RUTGBX9wF1SnAqgAMDaZEyMiIiIiotQiSdq6mmiLqAsBbLYbSqmAUupCAN9I2qyIiIiIiIhSVMSP8ymlfB97pZT6b+KnQ0REREREqSqN34kCEP1OFBEREREREVmiPVii04IhptNnv697v798oq7z7WDPxjpd/vZCJ7D2x4+aYFvkFely8+5GXc955kNd/3TmWQCApetNMO3JJ5mvb/UrM+GoyDThtld/YxAA4NyfPq17xWNMSO/jn24y++00wbrbapzA2vcXrdK9ku8dpuu/Lt6m69NHOQG62/eYkNsjDzKhujmZ5loUDjQhxJVlboBspgmSnTSsRNeP7jJzGzbiKF1/8O4KAEB6LxPou2TNLvM+rOMt3b7H9OurAQC76sw8m7abT3XWN5kg3M0bnbF2SObqLeZYdrjtyp3mdzZlpPNedzaYsN3B5SbQts46R48883tqanFCUyuLTKCtHR7a2wrbtVW4/XTrGZpFVrhxphX4mpPvjLUDeLNzzdhsO2A2Oz+8n2XmUBAStpvh3XdDYUtyvcN2i31CePOCYbltreE9ICQdMsdjbJYdzGv1M62/QwQC4T0rxDc0hNf0dQivTzBvc6P528rICg/W9QzgdV4wp2tt9e+167e1hR8jpOcXlJvm0bcCX/3Cb70CdOMJ5g09nxnrGzDrGaBr/RByDI/9Q8amRRwbNj7K+bwCdP14z80vCNg+X+L+e6DfdOMJ9/U/ducCTeO5lomQiADWZE15b18LIjJ4I8qR9EUUERERERF1D3zEuYMf5yMiIiIiIopD3HeiRKSnUmprMiZDRERERESpizeiHBEXUSJS1r4F4H0ROQSAKKV2Jm1mREREREREKSjax/m2A/jA2qoAVAL40K09icgMEakSkao5s/+cqLkSEREREdE+lCaSlK2ribaI+jGALwCcppQarJQaDGC9Ww/x20kpNUspNUEpNeHiSy5L5HyJiIiIiGgfEUnOFv28cqKIfCEiK0TkRo/Xs0XkKff190RkkPXaT93+FyJyQiKuQ8RFlFLqHgDfA3CLiPxORArR7smyREREREREySIi6QDuB3ASgAMBnCsiB7YbdimAXUqpYQDuBXC3u++BAM4BMBrAiQBmusfrlKhP51NKrVdKnQlgIYAFAPI6e1IiIiIiIup6RCQpWxSHAVihlFqllGoG8CSAqe3GTAXwN7d+FsCx4hx4KoAnlVJNSqnVAFa4x+uUqE/nE5FRcL4H9TqcRdRQt3+iUurlaPs/8O4aAMDK+S/o3ok3H6freUs36PrgqSfp+tgRPQEA6xe+qnsTz/uurh/533pdNyx5R9cFOdMAAHctXKF7Nx4zTNd7Gkz4Z9k4E/o7caD7DI2NX+relPOO0fVLb682b6rUBOR+umk3AKDuy090ryVwlq5frDLBvFcdMRgA8L+vduveScPNszuCQbIAMPLAvrouyXODZXuY0NwDy03wMAImuPRrw8p1vWjuawCAykNNaPCa1TvMfsU9dPm/dVbYrnLm8VVNvelVm9DgmgYTqrpz686wuW/YZI5l/5tYvs0cLxhku762QfdG9y3W9Y5aE8JbkGP+TBtanDDVHrkm0LbFCmOtLM7UtR3GWF6YHTafoiJzDDuEt6DQ6YcE8OaZsXYIb1q2CW/OznT7VoizHaCMDBMQnO8Rtltghc7a4aFFPmG7Bdnh/ZCwXStANzcYrOsXtmuF4oaG8AYijrUDdL3Ddk0vNEDXzCM93VyvlmZnfFqaHTprfr9iBRarYMqpHarrE8wb0nf/CEJ70QN0W1vDw3ZbPY4bfgyJeNyQQFOvUFzfuUUe6x8O6xXM6xNo6xt0G88xYutFOkY8vI7hF4ibiPN5iSeA1088h+hsoDF1XHe/nsn6N0LkRURmAJhhtWYppWa5dSWAddZr6wFMRCg9RikVEJFqAOVuf1G7fSs7O99oT+e7DsDVAJYB+CuA65VSz7sv3wkg6iKKiIiIiIi6h2SFzLoLpllRB6aIaHeiLgNwqFKq1v1y1rMiMkgp9Qc4jzsnIiIiIqL9RAwfvUuGDQD6Wz/3c3teY9aLSAaAYgA7Ytw3btEWk2lKqVoAUEqtAXAMgJNE5HfgIoqIiIiIiJJvMYDhIjJYRLLgPChiXrsx8wBc5NbfBfC6cj6TOg/AOe7T+wYDGA7g/c5OKNqdqC0iMk4p9REAuHekTgUwG8DYzp6ciIiIiIi6jrR9cBvF/Y7TNQBeAZAOYLZSaqmI3AagSik1D85Xjx4VkRUAdsJZaMEd9zSAzwAEAFytlGr1PFEcoi2iLnRPFvImAFwoIg919uRERERERETRKKXmA5jfrneLVTcCONNn3zsA3JHI+URcRCml1kd47b+JnAgREREREaW2fXEnKhUl6wEbRERERERE3VLUnKjOuv0+J6sIA8Ygr7gQAFBt5Qz98MH3dP3Pnxz7/9m78zg5qnJ94M87vc0+EzLZIJAQQlgDgYSAICCbwgXZREVBwKtE8eJ6rxcQl6uCF1HkJ3hBAwiyehUFWQQuWxDZwxoIkBCSkD2ZZDJ7Zn1/f1T1Oaenq6a6h+mke/J8+dQnJ2+fqjpV3QOcqep6THuLnzv0wRNXYpdTrwQAXHGSDSY+5YpH7U52nGaaSze0AwAeeeAVU/vDWQea9j+XNJr26R/f07RHVfkZPrVjTe3Lsyaa9p9vesC0x+8/w7T/+Npar9HZYmprm23G0Zuv2HypiuQRAIBDp47GpQ+9CwD4ymz7sJANrXa9w/ey44jHvCn/6Ek2J2pcnc0tQnm1aR62i81aurbFO9Y9drfZUU88ssCuNs4e3+IVNrsKqSoAwMINbbbW1W6aTe32/etvWgcA2PHkX+C9u7/jHcfaZvN6nxOYsny93V46l2nJxi12PAk7p9/YYbOvxtTaHKGOLu8W1lHlNnPJzagaV2X7uvse72wjbbRTK3OeNFPjZ0qdcM0/8di3DwcAVFTZPKhk3PZNVdhtpNL5Uf75y6gBQNK+ZzVuTpSfH1Udz86OAoCagDwoAKgLyI+qcnOitD+rfvo3zsFf/9/N3jGFZEoFZUJl1mzfRCI4PyqePj4nJyoeD95fZn6UN+a4k5nl5jmJ8+uvdL2yvh4drd7n082UQlnwNtLndv3i99Cw25SMWlZf53NhMqHK3Fpw9lPGOPx6RtxKWKaUI6gelec067OnYv7/3jvodoPGERYFk7GJjAyqiL5hY05nZuWRKfXANbfgxK+fF7C/sAyr9L5yycwKGkPuYxus/2A++59z8L9Xzh3y+tta0Jhv+9n1+ML3LshjG8M5IiufzKwo533/a7jlsuvyXu/mn16H877/tWEbRzH44g++hpt/mv+5KGa//8n/4F9/+G/behglaRs9na/oFHwSlZaeQOUrPYEaSdITqJEkPYEaSdITqJEkPYEaSdITqHyZCdQIkp5AjSRBE6hSl55AjST5TKBKxVAmUABG3AQKwIibQAHgBOpD4O18Ht7OR0RERERElIe8r0SJyGhV3ViIwRARERERUfHi3XyeQa9EicgVItLgt2eJyPsAXhCR5SJy5FYZIRERERERURGJup3vRFVNP4nhFwA+q6pTARwH4KqwlURkjojMF5H5vcv+MUxDJSIiIiKibalMpCBLqYmaRMVFJH3LX4WqvgQAqroIQPbjznyqOldVZ6nqrPjkI4ZpqEREREREtC2VFWgpNVFjvg7A30XkaAAPi8ivReRIEfkxgNcKPzwiIiIiIqLiMuiDJVT1WhFZAOACANP8/rsDuBfAZYUfHhERERERFYsSvPOuIHJ5Ot9aAHMBvKCqJi1VRI4H8HDk2qsXAQCu/eG3TOmKJ5eY9uaXnjTt/Sadbto3vrAMAHD46Ufb150g2ZZXnjbtj19wjmn/9sUVXmP5G6aWDqsFgKuesPv+5Sn7mHZTuxfuOv7Amaa214RaexybVpvmCYefatr3P77Ya4ydbGqvrWky7d5lb5p2OhT2yVdXmdoPjp1q2s8ssQ89/Niuo0y7s9sLJt1jr3GmVlvhvHVjJpnmbqNs8G460HTmZLutJxpXmPaEjx5q2is/sGNGvbefBStbEWS5m8vT6o25tdMGrTZvtGG73b02dHTduuztLW3sMO2kE7q6qr3TtA9I1Jv2ej/IuMY5/s4eG9zaUG7vMu3ptaGLE2oTADKDGHeotn3dfyHU+mG7MScIoarahvsmnHG6YbsJP1i3LGVDdZNu2G4ipB7zjqXKDeCN2Xa1G2gbEcJbFRLMW5EOr3WOv9zdrhPMm0xmh+Jmhur2OH3Lsvp6w4+F1wZsw633+0G+sZg9r91dNng5FrN9+/y+UpYd1gsgM0C3P7uubtJqRF+v7vfPCNUNC9sNCJjNJVQ3aBthwbwRfTNDdUPGlq4FBOJ62xg83Dcj0Ba5h5zmk4eab/htMQsLJM5VvqvnEzxbiueTto1SDIgmKoSop/N9A8DfAHwdwJsicorz8s8KOTAiIiIiIioufLCEJ+pK1PkAZqpqm4hMBnC3iExW1V8DKL2jJSIiIiKiISvB+U5BRE2iytK38KnqMhH5GLyJ1CRwEkVERERERNuhqKfzrRORGem/+BOqkwA0AJheyIEREREREVFxKZPCLKUmahJ1DrwHSxiq2quq5wBgABQREREREW13oh5xvnKQ154Z/uEQEREREVGxKsWHQBRCKQYEExERERERbTO55EQRERERERHx6Xy+gk+iDv68F6B7yr47mdqXfnCtadfOPNK0G1ttqOZPb3oRAPD3Hx5vai1bbKArGnY2ze8fu7tpn/Djh7zGLva5F++vt+Gwzz36qmlPnXOwaT++aD0A4PSjbfitG+iKURNM8wv77WjaN1/7FwDATjNnmdpfXl9n1+u2obHrmrcAAN5dsNzU3MDTR97bZNrfONQG6G5o9QJmD5vWYGplzjfwGnax42motSGlqPDCgmfv5IQGt9tQ3alOCO+jD31gVxszHgDw/iobmotyG+L7zgYbkJs+vs0d9r3TzfZrdO1dNmx14/oW0+7zUydXNtr3xj2mZZu6TDuVsBdMN3R49bF19jjbt9h91JUn7NCcoN8xlV5Ybl+fDQkcV2MDdF07+GG74vxbotoJ24074yyvdAJ0/RDelBP4mxGqm6ywzYRb97ZR6YbRxuxxVMadz6ETblodEKxbkwq+uFyR9Ps6obqmBmSE4pYH1BOJ4FDdjBDeXhugG08fd5/9mQ0N241l18ucQGM3FLcskV0X5/3ILUA37h+GU8sItHXrAeMIC791xuF+ztLbCN1uRiju4EG3YRmXwX0jgnJz2m5wPSg0Nr++weG+QeGw+eZ6Bm7D/UvGuQ8KHg7ebj6hv8MRRlpqgabDc8zDMJAQ+QQPF0oRDIFoWJTiQyAKgbfzERERERER5YG38xERERERUU6EUbEAeCWKiIiIiIgoL4NOokRklog8KSK3i8jOIvKoiDSLyEsicsDWGiQREREREW17DNv1RF2Jug7AlQAeBPAsgN+pah2Ai/3XAonIHBGZLyLzVz/7t2EbLBERERERbTucRHmiJlEJVX1IVe8CoKp6N7zG4wDKw1ZS1bmqOktVZ+146CnDOFwiIiIiIqJtK+rBEltE5OMA6gCoiJyqqveKyJEA+iLWJSIiIiKiEUQYFAUgehL1VXi38/UD+ASAC0TkFgCrAJxf2KEREREREREVn0EnUar6uoh8C8COAFaq6jcBfBMAROT4wdZNu/4z+wMA1rfY8FRssMGuV/3kdNP+1dNLTbvttacBAPtM/Iyp3fTiMtP+6MmHm/Y+Tphs+4LnAADHfeVsu97LK+2+Vy40TTc09TdPefv+xSf3MbWmdhsgO37/GaY9dZwNnsVmL1j340447v1PLLavj7H1N9Z54bX9K94xta4eG8D5zzdWm/ZPPmEDhJ99fyMA4KO71JtaZ7e9ELj7tLGmXVPuvKV+IPHk+ipbc4JSZ0yy23t0oz1HY/f4CABg9crNdr1aG/T79po2DLSi1QngbbOBvm1OQHLLJhu2mw7CXb8+e1sAsGyj3V7SCV5d1+kFFs9K2KDg9c32s+Uef2ePPdbRfgBujxOCOr7GBtq6QYz1VV6wrvuLlppqG6Abcz43VW4Ib8yrpyps34QTtluWyg7m9Tp59Yxg3pg9jqpEPLieDq91QmVrAgJ4AaAqoF6RdLbrHL8bAJ0O5026fcPCdp16MlmWVcsI242ou7X+Ptu3rMye2+4e7+fTDevtc/pKmT2fGcG6/vnKDOC124gM7M2oZYfqhtUzayGBvQ5TD91uwDZy6JsZ7hswNncMIaG4keG+bqAtck8YzSeMNJ/w22IWFu6bj3w2kU/obKmdS9q2Si0gmoauFL+/VAhRT+f7BoB7AHwdwJsi4n7B6WeFHBgRERERERUXkcIspSbqdr7zAcxS1TYRmQzgbhGZrKq/Bpi0RURERERE25+oSVSZqrYBgKouE5GPwZtITQInUURERERE25WyUrxsVABRjzhfJyLmy0D+hOokAA0AphdyYERERERERMUo6krUOQB63YKq9gI4R0R+V7BRERERERFR0eGDJTxRT+dbOchrzwz/cIiIiIiIiIpb1JUoIiIiIiIiAKX5JL1CKPgkavIYm1F09m0vAwCmn3qSqZ28z46mff7lN5t2xb6HAgA2t/eY2n/f9qpp//nfjzLtdiczCXVeZtJFR001pTN+8YR9fYLNX1reaLOInn3iLQDAlC/PtjU/nwkATjh8V9OuqXBOW42Xn/SZfcab0s3X3Wfa4/bZ17Tve2uD1+hqN7UNrTbj6L2F9sKfm9Xz5FIvd+n8g3Y2tY1tNsNq9tTRpu1mGI2eOMH7s8ZmGaHcZlwdOKHG1tttttOUyV4G07xHV5laqsEe39LVzXa9lPf+LnJyndDdaZrNHfb90+b1pt3hv2dNjTYnqs8JTFm9yW6vzDmmZZu885VK2K/zbeq052Jsrc0R6nA+F7UpLxMqnU8FAA2V9ry4+x7rZD+l1VXZmpvUXeXU07ljGTlRMds3VR6cH4VkhfdHRs1mSlW4OUoxm21VGfc/h04mT1UyJCcqmf31x1QyOw8KAFIB2U/lyeCMp4TzPrj1eHrMfW7N6dtn7xIuczOz+rzPi5v95G7X7ZvOc+rr70Pcz7FyM57E+dwEZT9l5kTZn+mMekYWU396w8HbDcti8sfR1xeWKdUfWDfbiMhnytpfZN+IjKeI7YbVw/KO8uubfT7Dco3yiaQJ3Yb7F//c55t1E3w+c+/7YfdV7IZjzIU67KjMrPO+/zXcctl1hdm5owTfVtrOlfHZcgCiHywxbNITKCKi4RRP8oI6EQ2/rTGBIqLSxf/7ICIiIiKinPB2Ps9WuxJFREREREQ0Egx6JUpE4gC+BOA0AOkvL60C8DcAN6lqT9i6REREREQ0svAR556oK1G3AZgB4L8A/Iu//BjA/gBuD1tJROaIyHwRmX/TjXOHaahERERERLQtlYkUZCk1Ud+Jmqmq0wbUVgJ4XkQWha2kqnMBzAWAzh7wuTNERERERDRiRF2J2iQinxaxz90VkTIR+SyApkHWIyIiIiKiEUakMEupiZpEnQngDABrRWSRf/VpLYDT/deIiIiIiIi2GRHZQUQeFZHF/p+jAvrMEJHnROQtEXnDvyiUfu0WEVkqIq/5y4yofQ56O5+qLhORXwG4CsASAHsC+AiAhaq6NJeDemqRFzD795v+amr/uP0/TdsNRMWy103z4qu+CQC44zUbQNv04jzT3m+XU0377wvXmPb0Yw/z/ty5ztQ2v/acaR905mmmfbezHpa9BgBIOcGmN72wwrS/c/gU027dYoNCR+29HwBg2jgbYouNdsyHzz7BtJ952Q+vHTXB1BZtaDXtnhXv2nafDeB8ZuE6AMAlToDways2m/ZhzrF29dj1Jk/xQnhrym1Aq7vvyfU2CNkNNN1nore9JzfZsN2GmbNMe83qFrtezQ4AgHfXOWG7jtXtNngX7XbM7V3eOWxpsttyj7mxMXh7K5u2AAASTujq+s4tpj0jWW/abpBxpZ8ltKXXHueolA3K7XWCUMfWeH3dIMZRGWG7djxu2G46FLgyI4DXjjNZbuvu+Mv8YN2MAN647ZsRwhuzP7Ll6c9qmf3MVrpBuRkhvNm/LwmqAUBFRgivZo/BCeZNBATzenW/f39vdm1A31g8extBAbwD6/1+kG8sZkOMe7qdvmXBgbbi18OCcvv73BBeOzbTPyiA19uwU88O1s0I1AztK8H1QbbrbTt7bPnsL6xvWFBqUEhpXuG+edzlnW8QaaECacMCggtlOPaXzyaigmcztsub9ClHpRgQTdGK9PtLFwN4XFWvEJGL/b9fNKBPB4BzVHWxiOwI4GUReURV0/+D+l1VvTvXHUY9ne9HAE7w+z0KYDaAeQAuFpEDVPXyXHdERERERESlrTjnUDgFwMf89h/gzVcyJlGqushprxaR9QDGANiMIYh6sMQZ8J7Ol4J3G99EVW0RkV8CeAEAJ1FERERERLQtjVPV9C1mawGMG6yziMwGkIR3p13a5SLyQwCPA7hYVbsCV/ZFTaJ6VbUPQIeILFHVFgBQ1U4R6Y9Yl4iIiIiIRpCoByoMlYjMATDHKc31n/idfv0xAOMDVr3U/YuqqoiE3ksqIhPgxTidq2q+o3AJvMlXEt4Txi8C8JPBxhs1ieoWkUpV7QAw09l5HQBOooiIiIiI6ENzI5JCXj827DURWSciE1R1jT9JWh/SrxbAgwAuVdXnnW2nr2J1icjNAP4jarxRk8kj/AkUnJkaACQAnBu1cSIiIiIiGjlEpCDLh3Qf7NzkXAB/Cxh3EsA9AG4d+AAJf+IF8QZyKoA3o3Y46CQq7F5AVW1U1QVRGyciIiIiopFDCrR8SFcAOE5EFgM41v87RGSWiNzo9/kMgCMAnBfwKPM7RGQBgAUAGgBcFrXDqNv5iIiIiIiIipaqbgRwTEB9PoAv++3bAdwesv7R+e6TkygiIiIiIspJkeZEbXUFn0Sdf92zXqPaBge7Qbg/+T/zyHaMOcx+X+zcmbsAAPb7zr12YztOM003HPWyvy407R+dsS+AAZcFnYDS7xy7m2l/97bXbJ/asQCAdc02uPXJJ2347W8/vb9pL1nXZtpHfGRXAMAO1TYcFeU2ePdT0+0TFu++8ykAQN1ue5jaw4s32fU6mk1zU5sNDV20cDUAoDJlgz+fW9Vk2qfsaR9UsrnDrrefH7Ybj9mzUTPBhu2OdsecKDfNA3b0Q3jb7D52dt6zl597z7Rjo7zjW7bWhga721q62QnN7bLt1k4vhLWvudHUtnTbr91tarTn2A2BXNPkbcP9AV7Z3G3aKScUtqXLnovR1amsfdSlbAhxd6+tj67w6m6Oasa5ctS7Ybv+mCor7Xbdc58ZtmvrCX8cbgAvkpVOX6eesMGylemQWufzXRFzgmudoNga57OTVp3MrgFAeUbYrncSyhPZAbwAkExk9wWcEF5IVDf+AAAgAElEQVQnVDfuBvb2hdT9/kEBvAAQiw0ezOsG5caT8cC6+KHIYaG64QG6/Vl9M4IkQ7fh789NTy2LDsW1AbplwX0dkcG8jsDw27BQ3dBt+I2wcF93GwHHl1PfIG4Y8TCEwwaGBmfsb3ifQRU0juEIIy3FQNMPO+ZCHnI+ny0i2n7xShQREREREeWE16E8hXrUOxERERER0Yg06CRKRGIi8hUR+amIHDbgte8XdmhERERERFRMRAqzlJqoK1G/A3AkgI0ArhGRXzmvnR62kojMEZH5IjK/7a1HhmGYRERERES0rRVpTtRWFzWJmq2qn1fV/wfgYADVIvJXEUlhkFsiVXWuqs5S1VnV+3xiOMdLRERERES0TUVNoszjxFS1V1XnAHgdwBMAqkPXIiIiIiKiEaesQEupiRrzfBE53i2o6o8B3AxgcqEGRUREREREVKwGfcS5qp4tIrNF5CBVfUlE9gZwPIB3VDUx2Lppjc8+atrn/+hCAMD769tN7eq580z79z860bSrUt7QWl5+ytRO/faXTfuZ922+0HuPP2naR/7nUQCAt1a1mNqYWYea9kcmjzbtlS++aNq7HHKwt93ldrud775i2jUVnzftOxesNe0zD/Byl9ycoeSu+5j29Ak2XwnrlgIA9j/xYHscC9bY1ytqTXN1U6dpt3+wBEBmZss/F2007S8fNMm0lzfaLKbDJnnbczO1Jk5qMO3aCuctrBtjmlPra7xGr81fmraTPY7nN60z7YZpXnbXmjVOTlSV7bt4g83dcnOENrR3eQ0ni6q9u9e0W5rs++eOf8NG7/jcW2dXbO4ybTeXqbHT1vdKeOei2cnRqnDykNz3rz7lXYDtdbJ+xlTZHxU3QqTWyYRKj6kqIDsKAMorbD0jP8rfn5sd5eZBJeJOPW63kUznK8XsGCriwTlR6foNN16M88+/0qslgn+HUpHIzo8Kyo4CBuRHOXlOQTlRiYy+vU49OyfKzX5ytxFUb9+4CTVjvM+1m89UVmb79vb0ZtUzspzc7KeMPKeAupvb5PZ1M5My6un9ZdcAoK8vuG76Z2QxhezP5DYF5EwNkE9WUWSGlSMsWyefXCpF7n3DbO0spqBsq0LlNkXmaOWgUAlIt//3b3HWxV8t0Na3jXMuvQC3Xn79Ntn3SI+qKsVsM7JK8ftLhTDoJEpEfgTgBABxEXkU3veingRwsYgcoKqX57qj9ASKiLad9ARqJElPoIho2xlpEygA22wCRVTsOIXyRIXtngFgBoAUgLUAJqpqi4j8EsALAHKeRBEREREREY0EUZOoXlXtA9AhIktUtQUAVLVTRPoj1iUiIiIiohGEt/N5oh4s0S0ilX57ZrooInUAOIkiIiIiIqLtTtSVqCNUtQsAVNWdNCUAnFuwURERERERUdEpxceRF0LU0/m6QuqNABqDXiMiIiIiopGJt/N5OJkkIiIiIiLKQ9TtfERERERERAD4iPO0gk+ixh7+CQDApUfvZmrn3G5DbLHybdM8Yc8vmfY/3tvgNXaZbmqXHjPVtL94y3y7DefrWtXl3iFdOW+JqZ130l6mXeeEo6LV3pF47se9bd/w9AeB223dYsM6H3x6qWl/56NeuO+qTTYcd7+Zu5r2uNryrO2dvP84U7rk2n+YdmziNNP+58pNdr3N67LGsPDt9aZdU27fxgWNzaa9/5h6AEBbl11vj8k7mHbKCTlNNuxo2qOr/UBXJ2h0/x2r7Hha7HmbsONBAIAl7zqhwXX2+Jatd0J4ne2tavfPV5cNB27fYkNVu1vscXT12Pdhk3+e3Zy+dZvtuY+V2R/tNW02LDgdTOuei/F19r3Z4uyjLul9RnqcENTRlfYcu6Gi9ZU2/Dat2vmMOXmvqHTq7jiT5emwXds5kbR93bobwpuM5R62W56uO5fgq5LBF6IrA0J4U25QrnP8SbdvUN0Jyo3HgwN04/HswN6gGjAwbLc3oBbct7/Lvr/xuPdehgXzZoTwSkDdOa+RfQFzzsODckNCXAOSVcMCe4MDbZ2/hAb2luXeN2zbkfvL7ptxaBnHEdE3h/2Fhf5GbmMI6xdSMYSRbu1zMTyhyMMwkADF8rkgouLBK1FERERERJQTfiXKw+9EERERERER5YFXooiIiIiIKCdl/FYUgCFMokRkkapOi+5JREREREQjCW/n8wx6O5+ItIpIi7+0ikgrgN3S9UHWmyMi80VkftubDw/7oImIiIiIiLaVqO9E3QzgXgC7q2qNqtYA+MBv14atpKpzVXWWqs6q3vf44RwvERERERFtI1Kgf0rNoJMoVf0GgF8DuEtEviEiZQD4nE8iIiIiItpuRT6dT1VfBnCs/9enAJQP0p2IiIiIiEYokcIspSbywRIiMhuAquo1IvIqgKNE5F9U9e+57ODGCz4CwIadAsA/7rjPtKedeJJpu/mU37ntVQDAmZ871NSmjqs27TcfmWfaux51tGkv3dAOAHjo/pdN7WfXn23aq5u22J1M3t80T9t7PADgsuueMrXUtANNe8m6NruN11417bG13u2Kd7xqQ3pPnmmDa91QVYydDAA4dOJoU+pZ8a5p73GcPY4n3t5g1/M/Wetbukxp/dIVpl3m7OPpJZtN+9jdvNDbja02dHbWJHsXppsd2LBjg2nXpkNhK+vs2Haw5x7dNtx21528PgueW2hqNWPsttY45w3ldhuLG/33oa/H1Jo7bRvtTabZ2WMDVFubvPDePieBc32TDewtc34KVzVnh+1u6rLncHKi0rQ3d9h9Vya9H4ueXieAN2VDdd19j67K/hGqrbB93fFEhe3GYraWSNm+caeORLlT93+mEnZ/qVhw2G5Fuu7UMkJ1ncDToBDelBt+64RQh9UT6Z9350OWCAnszQjh7fPea/dcZAbzxrPqoWG7ToCuOu+Z+Oe+v68/qzawr3u+TF2CtxvY1+kfVAMGBOg64+gP2l9UMK8ErD9AcFBuWN+Ieg5j+7A05OaHfPY3HEPL2ETEcYcHBEeEIg+zYgjspeHFt5SKAZ/O5xl0EiUiPwJwAoC4iDwKYDaAeQAuFpEDVPXywg+RiIiIiIioeERdiToDwAwAKQBrAUxU1RYR+SWAFwBwEkVEREREtJ0oxVvvCiHqO1G9qtqnqh0AlqhqCwCoaieA/sFXJSIiIiIiGnmirkR1i0ilP4mamS6KSB04iSIiIiIi2q7wSpQnahJ1hKp2AYCqupOmBIBzCzYqIiIiIiIqOqWY6VQIg06i0hOogHojgMaCjIiIiIiIiKiIRT7inIiIiIiICMhI49iuFXwS9dGpNjPo6n8s8Ropm89z3dk2i+met1aZ9vLHHvZq37zM1DIynrbY/KFLP7W3af/2RT8/afkbprZLQ4Vp3/jCMtM+/Jh9THvSaG9MuvQ1UzvygnNM+84Fa+2+N602zXRkwx9ftLVffNJud3OHzSoav+c0AMDEHex40NFsmofuN8G07398se0zeiIA4J3GFlvbsNw0u3rsnZZvLLYXCOtO9cbx1lq7j+lja0x7i5O/NHlyvWlXJv28m1F2POOqnYxl587OPcZ72U/fvfA4/OLyOwAADdOnmtcb17fa9WpsPtYyP8/LtabD5k+hwx5rR5cdZ3uLt16Pk/HT1OSs51jbbD8vcf8nfuOW7OwoAOjstvuoqfB+LJo7e1DuZxvVJW1uU2+fkxNVaX+E+v0Aj1onD8pVVWnznGIB+VFxNzvKyaVKODlIZYmEU/f7x5PZNQCI2bFV+HlOD952KU4894qMGoCMG5wrA3KigmoAzPkBkBFgYvKjnM9KRh5URvZTdj2RCMl+CsiEalr6Phqm7j5oX3VC6NL5Ub09vVk1AOjvd/OjAupOHpTb1z2HGfk8fv+wvv1R+VFhmVIZ+0OWjDFEbCNsDBnbjdxfdg2wPxfuNnLKpfLH8cQNd+Ho8z8XvPHAbeTcdavnNqWd/p0v46+/unHYtheWS5WPfDbRH3CS7rjitzjr4q9mb7eEc43OufQC3Hr59dt6GNuVrZlt9vuf/A/+9Yf/ttX2RyPPVrsSZSZQNCKlJ1AjScYEYYRIT6BGEjOBohEpnwlUqRjOCVSxCJpAlTpOoEY2TqCGjt+J8kQ94pyIiIiIiIgc/E4UERERERHlhI849wx6JUpELhSRBr89VUT+ISKbReQFEZm+dYZIRERERETFQAr0T6mJup3vAv9x5gDwawBXq2o9gIsA/DZsJRGZIyLzRWT+TTfOHaahEhERERERbXtRt/O5r49V1XsAQFXniUhNyDpQ1bkA5gJAZ09eD/0hIiIiIqIixUece6KuRN0tIreIyBQA94jIt0Rkkoh8EcAHW2F8RERERERERWXQK1GqeqmInAfgLgC7AUgBmAPgXgBnFXx0RERERERUNErx+0uFkMvT+RYCuFBVXxKRfQAcD+BtVW2OWA8A8P56Lxz1p9c+bmonn/svpn3g5FGm/elfPmlXHDsZALDrmCpT+t7f3zHt8YcdZdofnzbetL957d1eYxf73Ist3Tbk8roHFpn2z8/a37R702mF5dWmdv5HdrbbvfkVOzYnhHZdcxcA4OXnbQ7Wzl88yLQXrbFhs4ceuBMAG+YKAKioNc0TdrdhtDff+Kjd3a67AQDmLXVOeZcNq21yAn2XL1ln2umco/mrbXDtGfvYsTd39pj23hNt2G7Mv05bO9YGJddXOQGySRsWvM9YPzi50+5j/Hh7p+eKJWtMO15nj29Voz/+hA3xXeGE46KnyzTbtthQVG3bBCAzYLi5qcO03dDQ9c72yvxjWtMSHLbb3GXPRUNNCkBmGHF10r5nbtDvqAp7XtJZqqNCwnZrnL7iPNqmwq/H3LDdchug64bwJpzQ33g6TNY5h3E3jDaRMs1Uuu4ExaZibtiuXa8iERC2G1ADgKRbd4J1zbl1A3hD+maE7fr1uBsE3Gffh5gbJuwH6wYF8AKZAbpBIbzqfFYk7gTJuvWygHpGUK4boBsc2GsDbYP7akAYrVdPDzgk5DZof2FBuY58AmZzCsWN7JtbDQgOjc2nbwb3fRqGAM+wbZiqDG9iSGSA8pC3W3p32A/PcQ/DQAIMx2eLqNTw6XyeQSdRIvIjACcAiIvIowBmA5gH4GIROUBVLy/8EImIiIiIiIKJyA4A/hfAZADLAHxGVZsC+vUBWOD/9QNVPdmv7wrgjwBGA3gZwBdUtXvg+q6oX52dAeAwAEcA+DcAp6nqTwF8AsBnczoqIiIiIiIaEaRAy4d0MYDHVXV3AI/7fw/Sqaoz/OVkp/5zeE8hnwqgCcCXonYYNYnqVdU+Ve0AsERVWwBAVTsB9A++KhERERERUcGdAuAPfvsPAE7NdUXxvmNxNIC781k/6jtR3SJS6U+iZjo7qwMnUURERERE25Wy4vxS1DhVTX8Rfy2AcSH9ykVkPoBeAFeo6r3wbuHbrKrpL+GvBLBT1A6jJlFHqGoXAKiqO2lKADg3auNERERERDRyFGoKJSJz4D0FPG2unz2bfv0xAOOzVgQudf+iqioiYU99maSqq/z4pidEZAGAnB6WN1DUI867QuqNABqHskMiIiIiIiKXP2GaO8jrx4a9JiLrRGSCqq4RkQkA1odsY5X/5/siMg/AAQD+AqBeROL+1aiJAFZFjXd4n8lKREREREQjV3E+WeI+2LvkzgXwt6xhi4wSkZTfboD38LyF6uUoPAnvgXqh6w/ESRQREREREZWyKwAcJyKLARzr/x0iMktEbvT77AVgvoi8Dm/SdIWqLvRfuwjAd0TkPXjfkbopaoe5hO1+KF/946teY7UNuf3ZCV8w7RUbbVBq04tPmPanv+vdErlyU6ep3Xjni6b9y28dadpVKRvM2b7gOQDAcV8529ReW7HZtJc/+5xpz77EBva+t7YNADBq+ixTO3CiDQJe/dqrpj1+vxl226u9R9B3LVlgajXldt/3L95g2ifvOwZAZlhrbOI0095jrA2pxYblprnn8bMBAPPfca5MpmwI8brN9q7LjlV2vXQI4Pyl9jH5c2ZPMu1VTfbczpxot9fnp1iO32kHU6sudz4q1fa8TKrx1+u1j9KfMs4ex3Ob7fGP3m2qaa/3Q5jdsOH3Nzl3jzpfwdu0xam3e7etdjpBuO0tNni4z0ngbNpsjy9tdYsN1Y07wa1NXXb86VDYFieMOB1cDGS+f7UJG36b3vfoSnuu3BzG6oywXVtPh+26X9RMpmzfjBDelA3hTaTH74TqJtww2pgdRzIdSBuz2y3PCNt1wn/j2SG8QQG8Xj0WWE+m6877mAgJxc0I2/XrmaG6Nmw5kcjuGxq2GxHC64bfusG8vT29gXUT+OkEFrvBvKH1dAhrRlBucLhvZoCuZtX6+iICbTPCeiNCfJ1t5xOUO1j/gdsN6xsaXJtPiC9y30aYrR1oGxwmXLxhrYUcWREfdsnhuaStTQr2raihU9WNAI4JqM8H8GW//SyA6SHrvw8vDzdnvBJFRERERESUh4JfiSIiIiIiopGhOJ9wvvVxEkVERERERDnhHMoz6O18IjJFRH4vIpeJSLWI3CAib4rIn0Vk8tYZIhERERERUfGI+k7ULQBeAtAG4HkA7wA4AcDDAH4ftpKIzBGR+SIyf+3z9w3TUImIiIiIaJsqzkecb3VRk6gaVb1eVa8AUKuqV6nqClW9CcCosJVUda6qzlLVWeMPOXlYB0xERERERLQtRX0nql9EpgGoB1ApIrNUdb6ITAUQ/GxjIiIiIiIakYrxEefbQtQk6j8B3A+gH8CpAC4Rkf0A1AE4v8BjIyIiIiKiIsKn83kGnUSp6uMicg6AflV9SUSa4H0naqGq/j2XHcz/418BAAefdYap7TiqwrTPuvVlO5hpB5n2D47ZHQDwq38uNbWed23Y7if3OtO031ndandYNxYAcNFRNtj1v594z77eboNnR1XZ4NLLnlgCADj+qN1NbYdq+zo2rTbNE4441bTvfGWN1+i2wa5dPTbE85GXVpr2eXMOAQBsaLHBrlP33tm0x9ba0FQ3vPZje3khvdfcMd++Pm6Kab7RaMOEsXmdaXZ2ewGj77y30dSqyu0FxPea7Hnbvd4G5Hb4602eWGdqKSeAtWzUBNMelT5HTrjmHuMq7Xja7PkeM84G665e4Y+pxgb6rtpoQ3Pd7a3vcMJ2u7w+7V02EHVLa5tpdztBuE1NW0w7HUbY2GJrbohtY4c93+lg2vT5A4DRzmfBfX9rkvZHqNcPb62vcEJXnRTEOids11VV4W3DGQ4qKux23XHGk9n1eMLW4m7AbDyZXXcCYVNuyK1bd7fh/5uyPBH8b8zyeHDdbNs5/qQbzOvUM8J2/XpQDQBiAQG6QTVgYNiu/bxI+nwGBPB6u9PAejqcV0IDbUMCZv3+brhvVN+sej77M+PNvW9mAG8uxxfUN2uzofWw/UWS6GjD4EDb3HcRppgDTfMJ9y2UsADlYlaoIZfiuShmxRxITdu3QSdRIvIjeJOmuIg8Ci/Jdx6Ai0XkAFW9vPBDJCIiIiKiYsALUZ6o2/nOADADQArAWgATVbVFRH4J4AUAnEQREREREdF2JWoS1auqfQA6RGSJqrYAgKp2ikh/xLpERERERDSS8FIUgOhJVLeIVKpqB4CZ6aKI1MF72AQREREREW0n+HQ+T9Qk6ghV7QIAVXUnTQkA5xZsVEREREREREUq6ul8XSH1RgCNBRkREREREREVJT7i3BP9vFgiIiIiIiIyom7n+/B23gcA8MI/3sYr/+NlO73w/ibz8kO3Pmja/3HJ5+xqo70sqT/c+YKpJfc62LTrK20GzoV/WWDa+x5zGABg+s424+jxR94w7fK9Zpv2pjabDXTPI28DAP7070eZWtsWmy0DJxvprOm2fcYvnvAa43cztdVNNjPqnddtztX4+iMBAM87x3/ofnZb5W6OTk2DaR6+yygAwH+vXGZq4/bZ17SfXtJs1+vrMc2mdq+99oP1ppZwsnPmr7L5SgfMGmXaLR3eenvtZHOd3N861I+pN+3adJ5RymZD7b6DkxO1xe5jwrhq0373De9YUnV2W2sbnZyopM0SW77ZZjul87M6umzGj5v95WY4tbd0mHY6t8PNiSpzDmptqz1vCT/7qLnL1ibUl5t2h5MfVelkNPX0efuoS9k8qD4n36K+IvjHrarc6+/mD1U4mVJxJycqmXKzn7x6IpnIqnkv2DEn0vWEXb885nzenJyojLqfy1PpfjYd5Yng38OYTCjnLmD3s+fWU+62/eymjJwoJ88pMxPKO7erXnoJuxxycFZfN+Mpcxvpsdn3RpxzrE64jsSz62F93XOYkQlVFvd3F5wHFZofld52WBaTM46+vnTfwfOgsrbhbzunvhn1iAyrHOq57i8s6yif6JjQcx+VmZWDwG24f8nhPQkSmec1zEoti2drjPecSy/ArZdfX/D9FEqJvaVUQnghyrPVrkSlJ1BERMPJTKCIiIZRKU+giKjwCn8lioiIiIiIRgZeigLASRQREREREeWIjzj3DDqJEpE4gC8BOA3Ajn55FYC/AbhJVXvC1iUiIiIiIhqJoq5E3QZgM4D/ArDSr02ElxF1O4DPBq0kInMAzAGA+IwvIL7rkcMxViIiIiIi2ob4iHNP1CRqpqpOG1BbCeB5EVkUtpKqzgUwFwAqTr+Jz4chIiIiIqIRI+rpfJtE5NMi9hmtIlImIp8F0DTIekRERERENMJIgZZSEzWJOhPAGQDWicgiEVkMYC2A0/3XiIiIiIhoe8FZFICI2/lUdRn87z2JyGi//GtVPTvXHVz2TS+8dnJDlal96tpnbId+G2j71UMmmfbbq1q9l5e8bGpfuezrpv3eOhvi+vd7bCDvbZedBmDAe7H0VdP85HfnmPYLH9jQ26bXXwQA7DHhZFN7d02raY/fb4ZpTxtfY9qb3/GCfnf96KGm9tJqu12seMs002G69727wdRO2H20abtBseW77G7ak3bwz13TGlPbZ++jTXvB4ka7v0obMry22QuW7Vm33NT6neTK15fbi4n1R04x7aXrvZDa/cbb0NzePrve+B1tQG5Vyv8IVduw3p2qbVCuG3I6aYwN29UW7xzU7riXqTU22nBcVNi+HzR1YaDGTqfWad+nzh67v/ZWG97b5x93c7MT3OvY0GafkRL3Q1qbum0Yc9IJf93UbuspJ2y2p9d7/6qdAN4+53zXV9gwVjcEsaYiHbZra+XldhtuCG8yZesxP2w1nrQ1N5hXEinbNx22GwsJ5nXqSTfQ1g+QLXdrTnhoZUjYbnk8u54K6RsUwhuPx7JqXt0N0O31axLYtyxjuwHBuiHBvG74rVvv7ekdtG9GKG5/drhrZs0dG7L6ekPODtDtD9mGObyg2oDt9gek12b2zTOwN6hvWECwv+2w7QaH3+YX4qvI/Q7yIQf2DqOwMOFCGY79FWrIDIelfJRaQDSNPFFP57svoHx0uq6qJwe8TkREREREIxAfce6JerDERAALAdwI75dPAuAgAFcVeFxERERERERFKeo7UbMAvAzgUgDNqjoPQKeqPqWqTxV6cEREREREVDxECrOUmqjvRPUDuFpE/uz/uS5qHSIiIiIiGplKcL5TEDlNiFR1JYBPi8iJAFoKOyQiIiIiIqLilddVJVV9EMCDBRoLEREREREVM16KAhD9nSgiIiIiIiJy8PtNRERERESUEz7i3FPwSdSXZnsBuk8tsgGzSx+xdwQe9xWb2zuq0gZ+nnf7KwCAyumHmdrXDrZhvD/8v0V2JysXmuaRU7ww3bdWOV/dGmPX+/pHbPviB+x66PHCW2uckNObXl5p2iccsatp11Q4p63VC7o9wdnu3a+sta87gZ6d3V776VdXmdq3Dpts2htabYDsbnvuaNoNNUmv4QTLHe6E9P7imSV2f2PtON/Y0Ow1WmwYb3u3Hc+S923YbqUT2PreZi+8drd6G3jb6aw3eada004HpcZGjTO1uir7PrrBnbuPKbf1dm9sDU4A78plTmhwtT2+NZucEF4//HWDG7bb3WnH2WXH2d1mA5m7/SDczZtt2K6b09fYauvpLNVNHTaAN+GEvLrnorrGnrf0PmoS9vjdkOJ653PjhgTWlDvny1dVYWtOtisqnHo6bDeRDA7QjTuhvybQNp7MrgEZYbupmBN065/vjABe5xE6QaG6Xj37X7DJjABde/zuuU3Xg2qAExrs1GPu2Jyft7C6CeF1gr7LwvoGBOvG48Hvo9s3M3i2LGN9r+aG3/Zn9c2oBwXwDhBUz6lvRPhtWGBvUBBwfsG1OdTzeVSTDH5TRU4Bs/7+ggJ/B1PMeZ/FEEaa7/n8sIbjmAs15K19LoiosHglioiIiIiIclKKjyMvBE6iiIiIiIgoJ5xDeQa9B0JEYiLyFRH5qYgcNuC17xd2aERERERERMUn6ul8vwNwJICNAK4RkV85r50etpKIzBGR+SIy/5abbhiGYRIRERER0TYnBVpKTNTtfLNVdT8AEJHfALhORP4K4HMY5HBVdS6AuQCwubOP36QkIiIiIqIRI+pKlHmUl6r2quocAK8DeAJAdehaREREREQ04kiB/ik1UZOo+SJyvFtQ1R8DuBnA5EINioiIiIiIio9IYZZSM+jtfKp69sCaiNyqqucAuDGXHWzp9rJOZkysx6xLvHwomby/ef3KT+5j2i8s3WTa//zTwwCA7156lqmNr7c5Q3f/Zb5pp/Y+xLSr/ZynK+fZ7KTpHzvItPecUGPazz7xlmlX7OX12dTWbWoPP7nYtP/8H0eZdusWmy+DURMAAGfsZXOSTrtvgX19/G6mudbPKHrvzeWmNrb2aDue9zea9sH7jDftZDozp6bB1A7ZaZRpd6y22xu/73TTfm6pnxPl5N40O9lHG1bZ7C43X+iV1e0AgIMm7mDX67Tr7THensP0h76+od7Uat3co1SlaU6pt210efsYP9Ze0Hzn9WV2tVqbRbsS9CwAACAASURBVLXezYlKeJ+Blc1OTlSvfc/anZwodDTb3fkZTh2tNlPKzezY2GK3V+Yf1Lo2JyfKOT+t3fb93yleYXfn50dVJGwekpsTVeNkcfU5+64rd/KTfFVOXpn725lypx6T7JyodHYUkJkTZepOTlTc6Qsn+yjlZib5OVHlbnaUk8lTngjJiQqoZ2Y/2WykVEA9o6/z+Y0H1Bc/9gT2/MSxfs3NlIrIfgrJg3JDYsQ9R+myW3IDiMLq/jnMyK8pC86aisxi6g/rm96urfX1ZedBZe1v4PphYxisf2Tf3DOsct0XEJ79lF9e1YcfR5CwPCBTjXg/wseQ+/sxHIohZ2pb+sL3LsBtP7t+Ww9j2G3nbyvRsBl0EiUi9w0sAThKROoBQFVPznVH6QkUEdFwMhMoIqJhNBInUETDoQQvGhVE1IMldgbwFryrTgrvvM0CcFWBx0VERERERFSUor4TNRPAywAuBdCsqvMAdKrqU6r6VKEHR0RERERERYSPOAcQMYlS1X5VvRrAFwFc6j/mPOrqFRERERERjUDF+HQ+EdlBRB4VkcX+n6MC+hwlIq85yxYROdV/7RYRWeq8NiNqn1FXogAAqrpSVT8N4CEAt+d7YERERERERAVyMYDHVXV3AI/7f8+gqk+q6gxVnQHgaAAdAP7P6fLd9Ouq+lrUDvO6qqSqDwLgEyKIiIiIiLZDRfo48lMAfMxv/wHAPAAXDdL/DAAPqWrHIH0GldOVKCIiIiIiokIRkTkiMt9Z5uSx+jhVXeO31wIYN1hnAGcCuGtA7XIReUNErhaRVNQO+f0mIiIiIiLKSaEuRKnqXABzQ/cr8hiA8QEvXTpgOyoioYloIjIBwHQAjzjlS+BNvpL+GC4C8JPBxlvwSdSPH/MCazc+95ipXXLlt0x78hgbwHrm756zK/Z5QafnH7SLKS1e22ba+v4rpn3Ojy407aUbvBDXhx541dRu+sHxpi3uNchl9nbHT3z7ywCAV1Y2mVrTAhvou/v4T5r2knV2HGP28cJtp4ytMrXmRQtNe+eDZ5v2q2v9ba96x9TKnWDWhxfbsOHjdrPfh+v2g2KTO+1qt7uDDXnF5nWmOW2aDQVe+L6/vQobXLvBCZXtWb/CtN1wyIUrNwMA6o6cYmofNNqrnXuPs+9ZOkx27Hi7j4qkE8xabY9jx2pnzH7Q6c4N9rxpqw0brpkwzbQ3brQBuajwwnlXbrYBu66mLU69s9U0t/R4+2tvbTe1Piets6VlS9a2Njhhu26IbXO33UfSCX9tavf6p5yg2Z4+Gypb7YTfuvuur0iHsdp9V6WcsF3nI+uG7aY/y8lUQKguMsN24wFhuzEnQBgxG9jrBi+ng2KTAQG8wIBgXjeEN559kTuoBgwI1k3XYsHBvPF4LKueGcDb6/SVrL7e8P3+YaG6ISG8/f39WbXent5B+3obF38IIUG5/cGhuDbcNyT8Vtz9BQTzZgTohtUle7uO8G1EBPZm1KPCfQffbmhwbT7hvihcumihAmnDwoQL5cPur5DDZTgs5Wp7D4ge6VQ1NBhSRNaJyARVXeNPktYPsqnPALhHVc3/6DlXsbpE5GYA/xE1Ht7OR0REREREuSnOR5zfB+Bcv30ugL8N0vdzGHArnz/xgni/oT4VwJtRO+QkioiIiIiIclKMjzgHcAWA40RkMYBj/b9DRGaJyI1m7CKTAewMYGDe7R0isgDAAgANAC6L2iG/E0VERERERCVLVTcCOCagPh/Al52/LwOwU0C/o/PdZ96TKBFZpKrTonsSEREREdFIUqSPON/qBr2dT0RaRaTFX1pFpBXAbun6IOuZRxQufPRPwz5oIiIiIiKibSXqO1E3A7gXwO6qWqOqNQA+8Nu1YSup6lxVnaWqs/Y+7jPDOV4iIiIiItpGivO5ElvfoLfzqeo3RGQmgLtE5F4Av0Fhn2RKRERERERFirfzeSKfzqeqL8N7ygXgPcmivKAjIiIiIiIiKmI5PVhCVfsBXCMif0YOz0133fLb+wEAYw//hKldcMhk035t+WbTfvfvD5n2zM+cAgBoqEmZ2nfue8u0y6bONO0LD7GBvL95/gOv4QTpHjnlHNN2A3tRP840v3bIJADAVf9YYl/fYvvWVtgw0jsXrDXtow/1jqWu0r6OFpvvddTsnU37bws2eI1eG9ba1WNDOZ9ZsMa0L3DW29jm9Z+8+46mNrrahqamg4kB4ODddjDt619a5jUa7LYWbnK+ytbSaJpbuu043l/mvSeVKRtsurTFhtROrrUBuZ1+iO3E8TWmlnBCTqV2jGm75zAdsLnbaGdO3m4/C6Mbqk17zUobQozKOq/WZMN/3bDOxi02TBg9tp0+vu4OG9zrBuG2OCHEaU1ttuaG2G7stOfbDYXt8s9FZcq+N905hO3WlqfDdm2tqtw5V45KJ2w3PaTyjJoTtpvMDuHNDOB1focSC6kHhe06+0iF1MsT2b+mKo8H/+oqM1jXOwcZAbzOeckICPbrQTWv7obw9mXX3VDdWHBgb0YIr79tN1TXfc/C6hIUaBsWMOucQxPYm0Pf4EDb3MNo+0MCfyNDeHMIAg4K9w0TtLt81s8g0QkeQQGzw5HVybzPaGEhysWqkMMttXNBVJo33w2/QSdRInJfQDmVrqvqyQUZFRERERERUZGKuhI1EcBCADfC+y6UADgIwFUFHhcRERERERUZfifKE3W/wywALwO4FECzqs4D0KmqT6nqwKRfIiIiIiIawfh0Pk/U0/n6AVztfxfqahFZF7UOERERERHRSJbrgyVWAvi0iJwIIDRkl4iIiIiIRi7ezufJ66qSqj4I4MECjYWIiIiIiKjo8dY8IiIiIiLKiZTkN5iGX+EnUVu8fKH1r76EB274BgAg6WTAXHD7K7Zvnc0Uuvr0/QAA765pNbUH/vS0aX/unKNNe5eGStO+62+ve42d9zE1N1PpJ48ttuvNnm3ae+/k5Rw9Ne9dU5NdZ5h26xabHfPg00tN+9p/9fKqtvTYzBlUjTLN0/cea9pf+d0LXmMHm/e0odVmES1+a4Vpj6073LTfWuXdQXnAnvb8lCdshlM6OwkADplo279Ys8rb3a67mtqLH9jz6eZLNTvZR+tXeflRbn7Pm+tsTtT06XYfbf55mTLO5kS5WUV1o23fmgrn45bw8r92HVVha102+2lMg82iem+hPS+JmnoAQONmm/eU3hYArHbznpw8ro5u//3raDa17l6b4dTeareXzsz5/edm4NQbXsg6po0d9rMQdzKKWnq8czgubrOv3M9FRdy+Z719Nhek2s9z6nOyQuqcjC5XZcqew3T+UCqVnQcFAIlkIqv+3g2fx55f+1PW2BG35zCz7m0jGZAdBQzMj7Lt8nj2M2tSATUASLqfZfXek4zsKLdvRn6U1/f1ex7AgWecnFEDBuZEuXlOktXXzXhCSPZTOlfKzY5SZ7sSD6n7/bU/Ig/K6+wMQz9U34zombKQur+NyDyorG0H5VIFbuJDZ1iFCes6rNlP7rkfhiyf9DZO+sZ5eOCaWwB4j721+4vOtspHcO7Whz+OoG388ee/w5kXfeVDb7tQhnLcX/jeBbjtZ9cXYDTFh1FVlBfOoQBEP51v2KQnUESlIj2BGknSE6iRxEygiEpEegI1khTzBGqotpcJFBENDW/nIyIiIiKinPBClGerXYkiIiIiIiIaCQadRInIhSLS4Lenisg/RGSziLwgItO3zhCJiIiIiKgYiBRmKTVRV6IuUNVGv/1rAFeraj2AiwD8NmwlEZkjIvNFZH7v2lfCuhEREREREZWcqEmU+52psap6DwCo6jwANYFreK/PVdVZqjorPv7ADz9KIiIiIiLa5qRA/5SaqEnU3SJyi4hMAXCPiHxLRCaJyBcBfLAVxkdERERERMVCCrSUmEGfzqeql/oTprsA7AYgBWAOgHsBnFX44RERERERERWXyEecq+rNAG5O/11EblPV7+W6g1O+fBoA4LCpDaZ256v2ItY7999v2idd+AXT3ndnL6T1zFvm242tXWKa3zn8S6a9rtkGrLa98SwA4Jg5Z5vaeieA9a8Pvmna3z57lmlX+oGnW9592dQOPusM016yrs20V7/+uh3nBC/0d8VGG9ZaNdUG/e45rtaO891FAIAddt/D1BZtsOG3PSts0G+VE6D61AebAABH7W5DfPvcRMlxU0xz11E2pBabVnu1Yw8ypbeWbbKvJ2wo7KY2G0y7ZcMaAJnhe+nAXwCoPXiSaa/3z/3e42xobr8zttFj7fFXJt2AYK8+ocoJ23XCf3cabQOU+1qb7L4neftuanLCdpN2G6tb7DbcMNWWLr/eac93V499fUv7Frs/f/wtThCya0ObE7brhJi2+IG+blBsS4cdTzJh6z1u2G7ce6/dHNXacnuu3PchM2zX+7O83NbcUOBEMjuEN54IDuZF3AZSu8eEWDzrmNyw3ZTTdr8Vmkr3zwjgDf41UyqWXU8EhOpmjcM/MfF4diAuAMTc7TrbMP2dvkGhugBQFsuuu301JJi3r9duIx2KnBH26Z63gPBbb8iaVUNY+G1A37Bw0f6ANNp8QnUz+uewvyEHvkYEAYeF3+YV7ovc+4YpVEhp0DiCgoTD+haLQo2siA+ZaMQrwYtGBTHoJEpE7gsoH52uqypTLomIiIiIaLsSdSVqIoCFAG6E9wslAXAQgKsKPC4iIiIiIioypfg48kKIerDELAAvA7gUQLP/VL5OVX1KVZ8q9OCIiIiIiKh48Ol8nqgHS/QDuFpE/uz/uS5qHSIiIiIiopEspwmRqq4E8GkRORFAS1R/IiIiIiIaeXg7nyevq0qq+iCABws0FiIiIiIioqIX9Z0oIiIiIiIicvD7TURERERElBPezucp+CTq5yftDQB4zwmrvei652yHHaeZ5k8+bkNo1zZ74af/d/c8Uxt9yNGmvdvYatP+9dM2hBd1YwEAlxw91ZQeW7LOtNsWPG/ap+19Utb+3PDI8w/fxbTvXLDW7mPjStNsqPFCSu9duNrUDpg12bTH1NgQU2xYDgCY9akjTOnhxU74bUezabqBmPPe3gAA+MUnbYhvS6cNcR0/abwznpTdXo93TPtNGW1K9z+x2L5eP840lza323qzt7/uXhtQunSlHVtVygaFrmnzQm8n19lw3C5nvXHj7PuUdENRa7zw5fqKBILssoMTwttu911b74UJNzd12Ner6k1z7WYnhNexaYsfJtxtX9/ihO12ttt6r3/uW0PCdpvabT2WEbbrvScJJ+TVPRc1FfbHrafP1qv8AFw3QDkjbNeJq3TDdtPKU27YrlMvzz1styweC6wj5r0/GQG8TlBsMiyE14Tt2vXKE8EXvjM+F0HbdVI147HsEN5YSN9YQFCuV5esvmUB2wUGhPD6dSlzQ25tX3GOtb8/exv9OfR1z6EJUA0Lvw0Kus3YbnDfoHo+Qblh/XMK7DX7C+ub+xgy5PNfdBnaDRj5hLvmez6LVVi4b+H2V2IniEpGMQdSU+nilSgiIiIiIspJKT6OvBD4nSgiIiIiIqI88EoUERERERHlhN+J8gx6JUpEpojI70XkMhGpFpEbRORNEfmziEzeOkMkIiIiIiIqHlG3890C4CUAbQCeB/AOgBMAPAzg92EricgcEZkvIvPv/MONwzRUIiIiIiLalqRAS6mJup2vRlWvBwAR+ZqqXuXXbxKRC8NWUtW5AOYCwPKNXXwkChERERHRSFCKM54CiLoS1S8i00TkIACVIjILAERkKoDY4KsSERERERGNPFGTqP8EcD+AWwGcCuASEVkM4FkAP8xlB6OrkxhdncTkhip89a5X8dW7XkXba0+b5cI5R5tl17FVZrnyqfdx5VPv47w5/wI0rgAaV+D7X5hhlqb2brNc87+vm2XfYw7Dvscchm/96XVM37kO03euwzUPLjYLUpVmmTTaLo8tWYfHlqzDNdd+HRV7HIiKPQ7EYZMazPLg00vNglETzNLfD/T3A39+YZVZPjVzvFkyr1WWAVKG//vdHTh5/7E4ef+xeGbBGrOgotYsmzt6zPL2W2vw9ltrcNIVj2FcfTnG1ZdjXXOXWfac1mCWqvKYWZAoBxLluPmK3+PQSbU4dFItGldtMEvFuJ3M8vqaNrOgqx3oakdbV69Z1q7abJZkvMws7zW1472mdryzsQ3jqsoxrqoc7V19Zpk0ptossTIxS2V9PSrr63Hk9+5HdXkc1eVxIJ40y5QdUmZJjwdd7WhoqERDQyUmTxmNtuY2tDW3QarqzbKhZYtZ3O2tb+/G+vZuoNcund19ZtHOVrN09/aju7cfT1x0FNpau9DW2oV+VbM0t3ebRUTMsrG9Fxvbe3HrKyuQiJUhEStDe0+vWZKxMrP09PabpTIRQ2UihkVrW9HXr+jrV9Sm4mZRhVlqy+NmSatMxc3ift6SyZhZykRQJoJ3fn0a4sk44sl4xvsRT8TNEouJWdLn77ALb3VqCbMkysrMkv58Q8qQjMWQjMVw582XODUxi6s8LmZJS8TLzOLKqPsn5blb/4R4TBCPiZfl5C8Zx5FRL0MsVoY9jjjE1MrKxCzo7zOLlIlZ0rWW1atRVlbm5T8523X7qqpZ0p8P7c+uiWT2hYhZbM2e1/7+frO40n3H7LlXVi17u/bzhDIBygTvP/uCrTn7y9yGW/f67vmJY3P5z0CWjO1G1F+4868h23COI6D+0fM+a2r9apeobTz6u9sD+2Zwzmf63wv5OPHr5zlj05y3ce/VQ7s9/lP/fn5kn7D3JFd//PnvhrTe5y76ypDWu+OK3w5pPVfUMX/hexfksI3sz9Ctl18/pPGcc6ndXz6fi1suu25I+8vHF3/wtSGtd/NPhza2oe4vH7//yf8Mab1//eG/bdX9FSMp0D+lZtDb+VT1cQB7OKV/isgDAE5W1f6Q1QIdf+0/hzA84JafDe1fzI//+xHRnQJc9JtnhrRePn5z3beHtN5LV5wU3SnADdd/a0jr5ePIXRqGtN7LvzptSOu9+eryIa2Xj9nff3hI633poElDWs8NkC6Ufb97/5DWe/76c4e03uf/9edDWi8fh3/xs9GdArz71LNDWq9+4sQhrbc1/vuw4Z23h7Terh85eEjrvfPIY0NaLx8Hf/70Ia33z1v+d0jrHTPn7CGtl48Hr71lSOud+u0vD2m9v1x1w5DWy8eZQ5wM3TXEyddZF391SOvl47afffjJUD6GOvk67/uFn3Bs7cnQUPeXj609GRrq/qh4DTqJEpH7AsofA3Cv/xvUkwsyKiIiIiIiKjp8xLkn6sESOwN4C8CNABTe71IPAnDVYCsREREREdHIwzmUJ+o7UTMBvAzgUgDNqjoPQKeqPqWqTxV6cERERERERMUm6jtR/QCuFpE/+3+ui1qHiIiIiIhGKF6KApDjhEhVVwL4tIicCKClsEMiIiIiIiIqXnldVVLVBwE8WKCxEBERERFRESvFx5EXAm/NIyIiIiKinPDpfJ6oB0sQERERERGRy03sLuQCYA77Fs84iqFvsYyj1PoWyziKoW+xjKMY+hbLOIqhb7GMo9T6Fss4iqFvsYyjGPoWyziKoW+ht82ltJattyNgPvsWzziKoW+xjKPU+hbLOIqhb7GMoxj6Fss4iqFvsYyj1PoWyziKoW+xjKMY+hbLOIqhb6G3zaW0Ft7OR0RERERElAdOooiIiIiIiPKwNSdRc9m3qMZRDH2LZRyl1rdYxlEMfYtlHMXQt1jGUQx9i2Ucpda3WMZRDH2LZRzF0LdYxlEMfQu9bSoh4t+zSURERERERDng7XxERERERER54CSKiIiIiIgoD/FCbFRE9gRwCoCd/NIqAPep6ts5rHurqp4T8loSwJkAVqvqYyLyeQCHAngbwFxV7RmWAyAiIiIiIgox7N+JEpGLAHwOwB8BrPTLE+FNfv6oqlc4fe8buDqAowA8AQCqevKAbd8Bb+JXCWAzgGoAfwVwjH8s5w7rwRSAiIxV1fXbehwjnYiMVtWNBdjuNn//CnVsxWKkHx8RFScRGQfnl7+qui6PdU9W1YH/T5N+La6qvX67GsCeAN5X1U0h/RMDfyksIg2q2uj8PQmgR/3/iRORowAcCGChqj4UsE0BMBuZv9x+UXP4n0AR+ZqqXhfy2i4AWlR1s4hMBjALwDuq+mbUdvMhIlMB7A/gbVVdGNJnSO/f1n7vaAQZ7uApAIsAJALqSQCLB9ReAXA7gI8BONL/c83/b+9Mg60orgD8DbyoiLJqHgoCIhKDGyqC5RLXKBjLBUgZMRqtuMQFCGoEokYTd5OYGNcIUaMpDGqiuOEWISYqiALCewFEUSFxwxW0UimXzo9zbhj79dw7fe+bd2de+lSdenN7vunXPWem+0xPz2nd3teRx2L92wC8DXQstR+lfa1Yj54J6V2BK4BlwPvAe8ibsCuAbhbbw9KewGtAd6CHxQ4FZuv52Ap4HPgImA/s4ihHR+BU4GJgL2vf+dbvM4HNdHsg8BTyEDoP2NFiGzTfR4DFqrOAH7js6rJ/QvoA4BbgEuThdyrQBNwN9LfYLsDlwB3AWGvfDY68r4jVbyiwEngZeN2+jvJgP5/6edatJtvVw34+9bOOa0Qchl2BRo/7+vAy+xpi25toeXqU4V3t3GbW7w3QwSr9vT9wNjCyTL4RMBwYpTo8nkeF+p1eZl/f0jUO9AfGADukPXcp//9AYDQwuC1tlxf75cl2GV6fhbj3fOoHDAHmIn3BE6rLNG1XR76jLB0NvFX6bbEnIP3MS8BIpI37C7AaOMZi90cGn98FHiPWtgILLPZFoLtu/wh4Bjgf6Xsut9iDkXZ1FjBN9RFNO9hiz7L0bC3PWcBZFjsZeFXP1Un693dAs80qv6Oe09VI1LrusX3PWexs1vcNx+n5mwYsAcZVa7882C5o+9HWz1Au3H6O9H7AciutAzBRb/ohmrayTN5NSKPYHViHNrLARsjoRJzNylF9FJgE9Iql9dK0xyz2C21g4vqp/l1psc/pTXqM3qBjNP1A4FnHuZgGTAd+CLwAXB3bZze2zbHth4CjdHs/4GmLvRO4EdgDeYPYR7dvBGZY7Dpgreo61c9L6Rb7FHAa0ug2IQ3zVsD3gSct9k9qkyOB+/X3hq66adqS2PZsYHfdHoS1Wnge7OdTP8+6pbZdXuznUz9NL1RniYejo0zdnR0K5ujkxX55sF3G9SvUvedTP2ARMNxx/B7Ai470T4EHkYGlW1XX6d9b7DYO2AzYGmlrt9H0RqzBX2TQbXvdHgOsAPbQ3wsttim2/TzQSbcbHPkuxRrs0vStaek7rQNmAD8BLlT9oLRtsc1AJ2SAcR2wuaZ3jpcvxv8dGAF0A87R47dJUb/56MA2MhPJrl9q++XBdkHbj7Z+hnKDlDqSm1VLHcmIhGP6ICPa1wGryuQ9EWlgXwfGIw3tVL3QL7TYrBzV5WXKZz8knq113zGW9mrCsQtj26uS9sXSFse2G/Q8/xnY0NEYLY9tz0/KR38730S49gG/AW4nNhrZGvUDFlm/zwOeRhpqV0e5FB3JBOYm2TYv9vOpn2fdUtsuL/bzqV8pbwrUWeLh6MTOR39Heps5OxTM0cmL/fJgu4zrV6h7z6d+WLNkrDxedqTtjvgfp8XSXk04flFs+w1rn10/+zxuDyxH/Bi77XwGfRuJ9FOlh8WN7OtCz1ODo2wb2PVD3nreDVwJbKxpzsFt1s8O6gi8A3Rwnfsy9du/ZENH/RYCvXV7NrBR7H812/VLa7882C5o+9FsMpU3THsgo0+jdbtjiuO+BVxWgdkS2FK3u2lDOszBZeWoPgacy5cdz0bkTcYTjnKUHhCvBjYt0xg9i4xkfht5SDxS0/fFPSK/zJF2odbRnjZ5KXAbMiXrx8jbq37AicCDFjtXyxBvDDsARwPzHP9zN+QbtvHKJdXvBeShdHdkFHGopg90NEZL4/9f005AnInXHXmPU7scAFwEXKPn7afAHXmzn0/9POvmZbsq7TfMst+2tdjPp37KF6qzxMPRKdWPOjs7jvrl2tHJi/3yYLus65eF/bKynU/9kAGlh5C2ck/VozXtuoRydwAm6HU/rIz97kdmxVyHtLW/BPZC+utHLfZ5YrMkNK0P8gC7zkrfCXnTdrvqK8hD6vO0nH0zBblXJwFjVSdp2pSEch+B+BNjytTtNmRGzExkJsQdwLHIm9K7XNcm0NVRjxXAe1b6fkif8TM9d8/oOXscOMdivexXb9sFbT9a9wJkVrHsHNXuSMe3DBk5fF//15WUn8N9OOLkvpWwf2dkqtks5MPFa5DvlpqBPR38H3C82UOme3zqSD8B+QbqXWRE8B/AZY4GrT8yQroGmV6xAum4ZwBbJ5S9A+KE/w2rE4wxByKd3VJgb+TNYCnvIy32KuAgRx4jSOjItcGdgXQKS4CHgVOw5s63sf0+UPvZ36x51Q9xYtPUrWS7d9R2L1WyXSvZ74ga61ey3YJY/U6166dsoTpLPBwd5evu7FAwRycv9kuw3eS2tF3G9SvUvVdF/UYCNwEPqN4EHJp0zcWO6w3cVaZ+XfTamIx87zUGeUt3A7CFxR4E7OzIoytwniO9o5Z7AjJz4misb3tj7GAtw7Wqkynz7aIe0xn4OfBUwv4GZAr7d3R7T7XjuUBnBz8WfXNopfcFpibU+zTgV1rmScB2CWXxth8yKO9ju9Fqu+s9bNfNZbug7UNbPTpfXiSKoquQb1yesNJHANcaY7a10vdDbtZBSGOwGrgPmXrwmcVuhzTcc40xH8fzNsY84mB7Iw8wnyNTFZoS2K8jN/W8Svlq+jDAGGPmR1E0GHFSlxljHq7Abq/sUhcbO6anbl5jjPluEhfjt0BG93pWYpV/EPnY+IsK3N5IJ9xkjHksRb77KL/E5qMoGo6co4+iKNoYaRx3RRzBy4wxH1nsUmPMWmUvUvaFBLaUbyekMhVXjwAACFNJREFU4U3KdzxwrzFmdYq6+LAbIB3aG8gDyQjEGWnGsQSAY8mA45CBg18gHdqnjrz/peyxFfIegHz/sBVy3b8ETDfGrHWUewDSOZXY5Ums8iNxL6FQ7lrujXTEQ40xAxz7uwBnAAZxAkYgAw+rgIuNMW/G2IOANcaYF608ugJnGmMutdI7Im8pS23LPxHn8MOEsg5GHtrt+jkjUukxnZHrc7gx5huO/Q3IW0oD3IPcH2O1ftcbYz6JsWMRp2KulUdf4AJjzMmOeo+16jfTGLPMUY5qbLcl8GsSbKeMbb9DkDftrwOXpLRfN+CMWuynbbirfq1pu+HIvdjCdhXqV/P1GUXRobivzXrfe07b+dYvSPsRn0i6URQ1mvSR/OoeoTdIfqTdPkSVkyiKTjTG3FoNq07tGciI/BBggjFmpu5bYIzZtQb2dOQNSVlW0y5ERl4akFHf4ciI3zeRDuLSMuwwYE4C6wrzeQCOsPMZss8ZY4bp9snIObwX6QgfMLEw+Q7+JOXvc/FRFDUjo0WfRVF0M/AJ8lblQE0f1QbsR7r/FWRk+W5jzBrH+bHZ6co6Q6XGlgDohEQG7KznzbkEgGPJgETeJ2+9lg9DglEciozEfwgchUQgmxNjJyDTeCuyQdqfZOXo+OadlfiE68+KDVJZ9AFzCvIA3Ig80L2DvP27wn7givFHAl8tx/uwFco4yxgzshpWH1SnIIO/Dxtj7oztu8EYc3oCO8sYM701WE3rhbxh/AL5HnAcMoC2FPF53qzAjkJ8JJvt4TgNC4BdkP7p/TZg/zfYrTa/GpnW2gRM9Gm7ghRI7FdT/w9KmeAVlVhkutEmut0fmRYwQX/bH+1mwsb4jogDvBbooumdaDmP3IdNHXbek13ow8a25/PlD6ldwQZS88Q+8Kbl9wH2d3RZsQuRqS4HI1Ny1iDz9b8HbFoD67UEgA/vyS6J7d8YmKPbfXFf96lYTS+FqF9K5RD1PuHsU7MV2otZtbB8OaqoHZnMjirqE4HUh+2FRHS8HvmG9CK10120nMLiYhcnsPaSAT1IXjIgNVtF3iNi212R+2oxMkjRWAMbj/K6GxIEaQXuKK+uiLA1s8osQCLQbZPiGvRhd6flMg4f4l7GwbXkQ2uwmyBTR5uRwZw1yDTrExLKnIonOWrrZKyorRV4V5TX1HmzPnS8rbsBb9bA+gTayoTVtEeQh6HJyH00SW0+Dnl7XS3rE0k3K3ZBbHsashxIPyQg2n2V7q+gxdS6FyCziq1fJ8fWJcB/amDtj6U30Zv9alo6y5mwun+ha1t/t3DaPdjUYeczZF9EnJ+etAzK4HKsU/PIx9wn6vatrA+QMIiWkQuzYu2O6CvIFJk7kakq1bKplwDw5T3ZJazvSLvHbULLj9VTs5rm45AUytFRvu7ODgVzdKrIO7Wz48n6RHnNhNX0V5EpuauQpRcmosGYamR9lnHIip2JTPXrg4SDvwAJbvN7HEGp0vJ4RG315T3Zz5HZGbMd+u8aWJ9AW5mwur9clNdyfksl1ieSblbsgjLla+HDBW0fWvcCZFYxGS0fgnR4ce1PywhAPuyT6ENALK0B+Wj187Zgdd881kd1ikdr6upo6FKzsX2pws5nwSIjyCtRBwgd0UYeLF0PlKl5rfNtyPS4eYijtRL4K9ZHoRmyiWtGlOxUJZt6CQBf3pOdgDjUU5E3O6WHy82xPlD2YTU9K+el7o6O8nV3diiYo1NF3qmdHU/WJ8prJqyjzPsgAQze0mvulBpYn2UOsmLt6Hzz9W8H3BFrU/H4R21NzXuyTcC2Cdft6hpYn0BbmbC2PZBvFctd96lZTUsVSTcrFvnWrrTm20q+vMhzi5kgQduH1r0AmVVMpl3snbBveg1sH6zoQLF9dhS2TFhN2zCB3YyYI+HLOpiKYeezZmPHbEyZKHM+PDK9aWfkjUBjhXxalQUGedQhNat8qiUAquE92e11vzOSUg1sVs5L3R0dTau7s0MBHR0fHg9nx5P1ifKaCau86w1ARyRgw601sD7LOGTFPoP21cgb+Udj+1xvjFLxeEZt9eE92THA1xKuWzuKrQ+bOmJqVqym/wz9bMFKHwjcUy1r7S8bSTcrlvXrvZW09FlBL+D2SvkHLabWvQBBgwYNmlYth+R9yyHp3kZsJo6OptXd2Smyo5OG93F2fB0jkpdacK0flRX7R4/7yYdNvQxHhuxOyPS/D5BFoQdp+ubAeEeZU/P6vw+yr2ccS4n48lWwB7YhO7Kt2LaqH/K99w71ZNNcQ0GLr3UvQNCgQYO2hqJTAdsjm5dytCZrOSStxlbD1/tchGuovucCmaK8HInq+hqxte9wv61LzeeEHVdvNuNyFOocB20/WvcCBA0aNGhrKDVE3cw7m5dyFI3NSznywOalHHlgbZ7qouPWNUpv0di8lCMPbND2ow0ECRIkSEEkiqLFSbuQb5gKy+alHEVj81KOPLB5KUceWE++g9EF7o0xr0VRtB9wTxRF/ZS1xYcPbL7KkQc2SDuR8BAVJEiQIkkjcAjyjUNcIuQj8iKzeSlH0di8lCMPbF7KkQfWh387iqIhxphFAMaYj6MoOgy4BdjRka8PH9h8lSMPbJB2IuEhKkiQIEWSB5EpE4vsHVEUzSk4m5dyFI3NSznywOalHHlgffjjgc/i+40xnwHHR1H0W0e+Pnxg81WOPLBB2olERuZrBgkSJEiQIEGCBAkSJEiQFNKh3gUIEiRIkCBBggQJEiRIkCJJeIgKEiRIkCBBggQJEiRIEA8JD1FBggQJEiRIkCBBggQJ4iHhISpIkCBBggQJEiRIkCBBPCQ8RAUJEiRIkCBBggQJEiSIh/wX2fcjqStw1fMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 1152x576 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6iudy2Vp2bV"
      },
      "source": [
        "縦軸が単語の位置を、横軸が成分の次元を表しており、濃淡が加算される値です。\n",
        "\n",
        "ここでは最大系列長を50、隠れ層の次元数を256としました。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0sYQJIaHp2bY"
      },
      "source": [
        "### ② Multihead Attention\n",
        "ソース・ターゲット注意機構と自己注意機構\n",
        "Attentionは一般に、queryベクトルとkeyベクトルの類似度を求めて、その正規化した重みをvalueベクトルに適用して値を取り出す処理を行います。\n",
        "\n",
        "一般的な翻訳モデルで用いられるAttentionはソース・ターゲット注意機構と呼ばれ、この場合queryはDecoderの隠れ状態(Target)、keyはEncoderの隠れ状態(Source)、valueもEncoderの隠れ状態(Source)で表現されるのが一般的です。モデル全体の図では、右側のDecoderブロックの中央にあるAttentionがこれに相当します。\n",
        "\n",
        "Transformerでは、このソース・ターゲット注意機構に加えて、query,key,valueを同じ系列内で定義する自己注意機構を用います。これにより、ある単語位置の出力を求める際にあらゆる位置を参照できるため、局所的な位置しか参照できない畳み込み層よりも良い性能を発揮できると言われています。モデル全体の図では、左側のEncoderブロックと右側のDecoderブロックの下部にあるAttentionがこれに当たります。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LaR4TfW2p2bZ"
      },
      "source": [
        "Transformerでは、Scaled Dot-Product Attentionと呼ばれるAttentionを、複数のヘッドで並列に扱うMulti-Head Attentionによって、Source-Target-AttentionとSelf-Attentionを実現します。\n",
        "\n",
        "#### Scaled Dot-Product Attention\n",
        "Attentionには、注意の重みを隠れ層 1 つのフィードフォワードネットワークで求めるAdditive Attentionと、注意の重みを内積で求めるDot-Product Attentionが存在します。  一般に、Dot-Product Attentionのほうがパラメータが少なく高速であり、Transformerでもこちらを使います。\n",
        "\n",
        "\n",
        "Tranformerではさらなる工夫として、query($Q$)とkey($K$)の内積をスケーリング因子 $\\sqrt{d_k}$ で除算します。\n",
        "\n",
        "$Attention(Q, K, V) = softmax(\\frac{QK^T}{\\sqrt{d_k}})V$ \n",
        "\n",
        "\n",
        "これは、$d_k$（keyベクトルの次元数）が大きい場合に内積が大きくなりすぎて逆伝播のsoftmaxの勾配が極端に小さくなることを防ぐ役割を果たします。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N7FawizWp2ba"
      },
      "source": [
        "class ScaledDotProductAttention(nn.Module):\n",
        "    \n",
        "    def __init__(self, d_model, attn_dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param attn_dropout: float, ドロップアウト率\n",
        "        \"\"\"\n",
        "        super(ScaledDotProductAttention, self).__init__()\n",
        "        self.temper = np.power(d_model, 0.5)  # スケーリング因子\n",
        "        self.dropout = nn.Dropout(attn_dropout)\n",
        "        self.softmax = nn.Softmax(dim=-1)\n",
        "\n",
        "    def forward(self, q, k, v, attn_mask):\n",
        "        \"\"\"\n",
        "        :param q: torch.tensor, queryベクトル, \n",
        "            size=(n_head*batch_size, len_q, d_model/n_head)\n",
        "        :param k: torch.tensor, key, \n",
        "            size=(n_head*batch_size, len_k, d_model/n_head)\n",
        "        :param v: torch.tensor, valueベクトル, \n",
        "            size=(n_head*batch_size, len_v, d_model/n_head)\n",
        "        :param attn_mask: torch.tensor, Attentionに適用するマスク, \n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "        :return output: 出力ベクトル, \n",
        "            size=(n_head*batch_size, len_q, d_model/n_head)\n",
        "        :return attn: Attention\n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "        \"\"\"\n",
        "        # QとKの内積でAttentionの重みを求め、スケーリングする\n",
        "        attn = torch.bmm(q, k.transpose(1, 2)) / self.temper  # (n_head*batch_size, len_q, len_k)\n",
        "        # Attentionをかけたくない部分がある場合は、その部分を負の無限大に飛ばしてSoftmaxの値が0になるようにする\n",
        "        attn.data.masked_fill_(attn_mask, -float('inf'))\n",
        "        \n",
        "        attn = self.softmax(attn)\n",
        "        attn = self.dropout(attn)\n",
        "        output = torch.bmm(attn, v)\n",
        "\n",
        "        return output, attn"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iw1FIuWip2bd"
      },
      "source": [
        "#### Multi-Head Attention\n",
        "TransformerではAttentionを複数のヘッドで並列に行うMulti-Head Attentionを採用しています。\n",
        "\n",
        "複数のヘッドでAttentionを行うことにより、各ヘッドが異なる部分空間を処理でき、精度が向上するとされています。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tYZJzsfxp2bf"
      },
      "source": [
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self, n_head, d_model, d_k, d_v, dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param n_head: int, ヘッド数\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param dropout: float, ドロップアウト率\n",
        "        \"\"\"\n",
        "        super(MultiHeadAttention, self).__init__()\n",
        "\n",
        "        self.n_head = n_head\n",
        "        self.d_k = d_k\n",
        "        self.d_v = d_v\n",
        "\n",
        "        # 各ヘッドごとに異なる重みで線形変換を行うための重み\n",
        "        # nn.Parameterを使うことで、Moduleのパラメータとして登録できる. TFでは更新が必要な変数はtf.Variableでラップするのでわかりやすい\n",
        "        self.w_qs = nn.Parameter(torch.empty([n_head, d_model, d_k], dtype=torch.float))\n",
        "        self.w_ks = nn.Parameter(torch.empty([n_head, d_model, d_k], dtype=torch.float))\n",
        "        self.w_vs = nn.Parameter(torch.empty([n_head, d_model, d_v], dtype=torch.float))\n",
        "        # nn.init.xavier_normal_で重みの値を初期化\n",
        "        nn.init.xavier_normal_(self.w_qs)\n",
        "        nn.init.xavier_normal_(self.w_ks)\n",
        "        nn.init.xavier_normal_(self.w_vs)\n",
        "\n",
        "        self.attention = ScaledDotProductAttention(d_model)\n",
        "        self.layer_norm = nn.LayerNorm(d_model) # 各層においてバイアスを除く活性化関数への入力を平均０、分散１に正則化\n",
        "        self.proj = nn.Linear(n_head*d_v, d_model)  # 複数ヘッド分のAttentionの結果を元のサイズに写像するための線形層\n",
        "        # nn.init.xavier_normal_で重みの値を初期化\n",
        "        nn.init.xavier_normal_(self.proj.weight)\n",
        "        \n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "\n",
        "    def forward(self, q, k, v, attn_mask=None):\n",
        "        \"\"\"\n",
        "        :param q: torch.tensor, queryベクトル, \n",
        "            size=(batch_size, len_q, d_model)\n",
        "        :param k: torch.tensor, key, \n",
        "            size=(batch_size, len_k, d_model)\n",
        "        :param v: torch.tensor, valueベクトル, \n",
        "            size=(batch_size, len_v, d_model)\n",
        "        :param attn_mask: torch.tensor, Attentionに適用するマスク, \n",
        "            size=(batch_size, len_q, len_k)\n",
        "        :return outputs: 出力ベクトル, \n",
        "            size=(batch_size, len_q, d_model)\n",
        "        :return attns: Attention\n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "            \n",
        "        \"\"\"\n",
        "        d_k, d_v = self.d_k, self.d_v\n",
        "        n_head = self.n_head\n",
        "\n",
        "        # residual connectionのための入力 出力に入力をそのまま加算する\n",
        "        residual = q\n",
        "\n",
        "        batch_size, len_q, d_model = q.size()\n",
        "        batch_size, len_k, d_model = k.size()\n",
        "        batch_size, len_v, d_model = v.size()\n",
        "\n",
        "        # 複数ヘッド化\n",
        "        # torch.repeat または .repeatで指定したdimに沿って同じテンソルを作成\n",
        "        q_s = q.repeat(n_head, 1, 1) # (n_head*batch_size, len_q, d_model)\n",
        "        k_s = k.repeat(n_head, 1, 1) # (n_head*batch_size, len_k, d_model)\n",
        "        v_s = v.repeat(n_head, 1, 1) # (n_head*batch_size, len_v, d_model)\n",
        "        # ヘッドごとに並列計算させるために、n_headをdim=0に、batch_sizeをdim=1に寄せる\n",
        "        q_s = q_s.view(n_head, -1, d_model) # (n_head, batch_size*len_q, d_model)\n",
        "        k_s = k_s.view(n_head, -1, d_model) # (n_head, batch_size*len_k, d_model)\n",
        "        v_s = v_s.view(n_head, -1, d_model) # (n_head, batch_size*len_v, d_model)\n",
        "\n",
        "        # 各ヘッドで線形変換を並列計算(p16左側`Linear`)\n",
        "        q_s = torch.bmm(q_s, self.w_qs)  # (n_head, batch_size*len_q, d_k)\n",
        "        k_s = torch.bmm(k_s, self.w_ks)  # (n_head, batch_size*len_k, d_k)\n",
        "        v_s = torch.bmm(v_s, self.w_vs)  # (n_head, batch_size*len_v, d_v)\n",
        "        # Attentionは各バッチ各ヘッドごとに計算させるためにbatch_sizeをdim=0に寄せる\n",
        "        q_s = q_s.view(-1, len_q, d_k)   # (n_head*batch_size, len_q, d_k)\n",
        "        k_s = k_s.view(-1, len_k, d_k)   # (n_head*batch_size, len_k, d_k)\n",
        "        v_s = v_s.view(-1, len_v, d_v)   # (n_head*batch_size, len_v, d_v)\n",
        "\n",
        "        # Attentionを計算(p16.左側`Scaled Dot-Product Attention * h`)\n",
        "        outputs, attns = self.attention(q_s, k_s, v_s, attn_mask=attn_mask.repeat(n_head, 1, 1))\n",
        "\n",
        "        # 各ヘッドの結果を連結(p16左側`Concat`)\n",
        "        # torch.splitでbatch_sizeごとのn_head個のテンソルに分割\n",
        "        outputs = torch.split(outputs, batch_size, dim=0)  # (batch_size, len_q, d_model) * n_head\n",
        "        # dim=-1で連結\n",
        "        outputs = torch.cat(outputs, dim=-1)  # (batch_size, len_q, d_model*n_head)\n",
        "\n",
        "        # residual connectionのために元の大きさに写像(p16左側`Linear`)\n",
        "        outputs = self.proj(outputs)  # (batch_size, len_q, d_model)\n",
        "        outputs = self.dropout(outputs)\n",
        "        outputs = self.layer_norm(outputs + residual)\n",
        "\n",
        "        return outputs, attns"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SR37dOECp2bn"
      },
      "source": [
        "### ③ Position-Wise Feed Forward Network\n",
        "単語列の位置ごとに独立して処理する2層のネットワークであるPosition-Wise Feed Forward Networkを定義します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Eor7Q0Zp2bo"
      },
      "source": [
        "class PositionwiseFeedForward(nn.Module):\n",
        "    \"\"\"\n",
        "    :param d_hid: int, 隠れ層1層目の次元数\n",
        "    :param d_inner_hid: int, 隠れ層2層目の次元数\n",
        "    :param dropout: float, ドロップアウト率\n",
        "    \"\"\"\n",
        "    def __init__(self, d_hid, d_inner_hid, dropout=0.1):\n",
        "        super(PositionwiseFeedForward, self).__init__()\n",
        "        # window size 1のconv層を定義することでPosition wiseな全結合層を実現する.\n",
        "        self.w_1 = nn.Conv1d(d_hid, d_inner_hid, 1)\n",
        "        self.w_2 = nn.Conv1d(d_inner_hid, d_hid, 1)\n",
        "        self.layer_norm = nn.LayerNorm(d_hid)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.relu = nn.ReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        :param x: torch.tensor,\n",
        "            size=(batch_size, max_length, d_hid)\n",
        "        :return: torch.tensor,\n",
        "            size=(batch_size, max_length, d_hid) \n",
        "        \"\"\"\n",
        "        residual = x\n",
        "        output = self.relu(self.w_1(x.transpose(1, 2)))\n",
        "        output = self.w_2(output).transpose(2, 1)\n",
        "        output = self.dropout(output)\n",
        "        return self.layer_norm(output + residual)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzXB_HIVp2bu"
      },
      "source": [
        "### ④ Masking\n",
        "TransformerではAttentionに対して２つのマスクを定義します。\n",
        "\n",
        "一つはkey側の系列のPADトークンに対してAttentionを行わないようにするマスクです。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "amN70mbwp2bv"
      },
      "source": [
        "def get_attn_padding_mask(seq_q, seq_k):\n",
        "    \"\"\"\n",
        "    keyのPADに対するattentionを0にするためのマスクを作成する\n",
        "    :param seq_q: tensor, queryの系列, size=(batch_size, len_q)\n",
        "    :param seq_k: tensor, keyの系列, size=(batch_size, len_k)\n",
        "    :return pad_attn_mask: tensor, size=(batch_size, len_q, len_k)\n",
        "    \"\"\"\n",
        "    batch_size, len_q = seq_q.size()\n",
        "    batch_size, len_k = seq_k.size()\n",
        "    pad_attn_mask = seq_k.data.eq(PAD).unsqueeze(1)   # (N, 1, len_k) PAD以外のidを全て0にする\n",
        "    pad_attn_mask = pad_attn_mask.expand(batch_size, len_q, len_k) # (N, len_q, len_k)\n",
        "    return pad_attn_mask"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6HtLq5nop2bz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "849817fd-e78b-4cf7-d62a-9fd452e69db0"
      },
      "source": [
        "_seq_q = torch.tensor([[1, 2, 3]])\n",
        "_seq_k = torch.tensor([[4, 5, 6, 7, PAD]])\n",
        "_mask = get_attn_padding_mask(_seq_q, _seq_k)  # 行がquery、列がkeyに対応し、key側がPAD(=0)の時刻だけ1で他が0の行列ができる\n",
        "print('query:\\n', _seq_q)\n",
        "print('key:\\n', _seq_k)\n",
        "print('mask:\\n', _mask)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "query:\n",
            " tensor([[1, 2, 3]])\n",
            "key:\n",
            " tensor([[4, 5, 6, 7, 0]])\n",
            "mask:\n",
            " tensor([[[False, False, False, False,  True],\n",
            "         [False, False, False, False,  True],\n",
            "         [False, False, False, False,  True]]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5mPZVJ7p2b2"
      },
      "source": [
        "もう一つはDecoder側でSelf Attentionを行う際に、各時刻で未来の情報に対するAttentionを行わないようにするマスクです。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IW8zvfLcp2b3"
      },
      "source": [
        "def get_attn_subsequent_mask(seq):\n",
        "    \"\"\"\n",
        "    未来の情報に対するattentionを0にするためのマスクを作成する\n",
        "    :param seq: tensor, size=(batch_size, length)\n",
        "    :return subsequent_mask: tensor, size=(batch_size, length, length)\n",
        "    \"\"\"\n",
        "    attn_shape = (seq.size(1), seq.size(1))\n",
        "    # 上三角行列(diagonal=1: 対角線より上が1で下が0)\n",
        "    subsequent_mask = torch.triu(torch.ones(attn_shape, dtype=torch.uint8, device=device), diagonal=1)\n",
        "    subsequent_mask = subsequent_mask.repeat(seq.size(0), 1, 1)\n",
        "    return subsequent_mask"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "579z_7Krp2b6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b973f0bd-12d3-4a75-8d04-6e235ef6e802"
      },
      "source": [
        "_seq = torch.tensor([[1,2,3,4]])\n",
        "_mask = get_attn_subsequent_mask(_seq)  # 行がquery、列がkeyに対応し、queryより未来のkeyの値が1で他は0の行列ができいる\n",
        "print('seq:\\n', _seq)\n",
        "print('mask:\\n', _mask)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "seq:\n",
            " tensor([[1, 2, 3, 4]])\n",
            "mask:\n",
            " tensor([[[0, 1, 1, 1],\n",
            "         [0, 0, 1, 1],\n",
            "         [0, 0, 0, 1],\n",
            "         [0, 0, 0, 0]]], device='cuda:0', dtype=torch.uint8)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJmM096Bp2b_"
      },
      "source": [
        "## 3. モデルの定義\n",
        "\n",
        "### Encoder\n",
        "これまで定義してきたサブレイヤーを統合して、Encoderを定義します。\n",
        "\n",
        "EncoderではSelf AttentionとPosition-Wise Feed Forward Networkからなるブロックを複数層繰り返すので、ブロックのクラスEncoderLayerを定義した後にEncoderを定義します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LtpNJZXxp2cA"
      },
      "source": [
        "class EncoderLayer(nn.Module):\n",
        "    \"\"\"Encoderのブロックのクラス\"\"\"\n",
        "    def __init__(self, d_model, d_inner_hid, n_head, d_k, d_v, dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_inner_hid: int, Position Wise Feed Forward Networkの隠れ層2層目の次元数\n",
        "        :param n_head: int,　ヘッド数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param dropout: float, ドロップアウト率\n",
        "        \"\"\"\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        # Encoder内のSelf-Attention\n",
        "        self.slf_attn = MultiHeadAttention(\n",
        "            n_head, d_model, d_k, d_v, dropout=dropout)\n",
        "        # Postionwise FFN\n",
        "        self.pos_ffn = PositionwiseFeedForward(d_model, d_inner_hid, dropout=dropout)\n",
        "\n",
        "    def forward(self, enc_input, slf_attn_mask=None):\n",
        "        \"\"\"\n",
        "        :param enc_input: tensor, Encoderの入力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :param slf_attn_mask: tensor, Self Attentionの行列にかけるマスク, \n",
        "            size=(batch_size, len_q, len_k)\n",
        "        :return enc_output: tensor, Encoderの出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :return enc_slf_attn: tensor, EncoderのSelf Attentionの行列, \n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "        \"\"\"\n",
        "        # Self-Attentionのquery, key, valueにはすべてEncoderの入力（enc_input）が入る\n",
        "        enc_output, enc_slf_attn = self.slf_attn(\n",
        "            enc_input, enc_input, enc_input, attn_mask=slf_attn_mask)\n",
        "        enc_output = self.pos_ffn(enc_output)\n",
        "        return enc_output, enc_slf_attn"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MHrEiFeEp2cF"
      },
      "source": [
        "class Encoder(nn.Module):\n",
        "    \"\"\"EncoderLayerブロックからなるEncoderのクラス\"\"\"\n",
        "    def __init__(\n",
        "            self, n_src_vocab, max_length, n_layers=6, n_head=8, d_k=64, d_v=64,\n",
        "            d_word_vec=512, d_model=512, d_inner_hid=1024, dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param n_src_vocab: int, 入力言語の語彙数\n",
        "        :param max_length: int, 最大系列長\n",
        "        :param n_layers: int, レイヤー数\n",
        "        :param n_head: int,　ヘッド数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param d_word_vec: int, 単語の埋め込みの次元数\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_inner_hid: int, Position Wise Feed Forward Networkの隠れ層2層目の次元数\n",
        "        :param dropout: float, ドロップアウト率        \n",
        "        \"\"\"\n",
        "        super(Encoder, self).__init__()\n",
        "\n",
        "        n_position = max_length + 1\n",
        "        self.max_length = max_length\n",
        "        self.d_model = d_model\n",
        "\n",
        "        # Positional Encodingを用いたEmbedding\n",
        "        self.position_enc = nn.Embedding(n_position, d_word_vec, padding_idx=PAD)\n",
        "        self.position_enc.weight.data = position_encoding_init(n_position, d_word_vec)\n",
        "\n",
        "        # 一般的なEmbedding\n",
        "        self.src_word_emb = nn.Embedding(n_src_vocab, d_word_vec, padding_idx=PAD)\n",
        "\n",
        "        # EncoderLayerをn_layers個積み重ねる\n",
        "        self.layer_stack = nn.ModuleList([\n",
        "            EncoderLayer(d_model, d_inner_hid, n_head, d_k, d_v, dropout=dropout)\n",
        "            for _ in range(n_layers)])\n",
        "\n",
        "    def forward(self, src_seq, src_pos):\n",
        "        \"\"\"\n",
        "        :param src_seq: tensor, 入力系列, \n",
        "            size=(batch_size, max_length)\n",
        "        :param src_pos: tensor, 入力系列の各単語の位置情報,\n",
        "            size=(batch_size, max_length)\n",
        "        :return enc_output: tensor, Encoderの最終出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :return enc_slf_attns: list, EncoderのSelf Attentionの行列のリスト\n",
        "        \"\"\"\n",
        "        # 一般的な単語のEmbeddingを行う\n",
        "        enc_input = self.src_word_emb(src_seq)\n",
        "        # Positional EncodingのEmbeddingを加算する\n",
        "        enc_input += self.position_enc(src_pos)\n",
        "\n",
        "        enc_slf_attns = []\n",
        "        enc_output = enc_input\n",
        "        # key(=enc_input)のPADに対応する部分のみ1のマスクを作成\n",
        "        enc_slf_attn_mask = get_attn_padding_mask(src_seq, src_seq)\n",
        "\n",
        "        # n_layers個のEncoderLayerに入力を通す\n",
        "        for enc_layer in self.layer_stack:\n",
        "            enc_output, enc_slf_attn = enc_layer(\n",
        "                enc_output, slf_attn_mask=enc_slf_attn_mask)\n",
        "            enc_slf_attns += [enc_slf_attn]\n",
        "\n",
        "        return enc_output, enc_slf_attns"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K4uxRtsWp2cI"
      },
      "source": [
        "### Decoder\n",
        "\n",
        "Deocoderも同様にSelf Attention, Source-Target Attention, Position-Wise Feed Forward Networkからなるブロックを複数層繰り返ので、ブロックのクラスDecoderLayerを定義した後にDecoderを定義します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2WFWRsXYp2cJ"
      },
      "source": [
        "class DecoderLayer(nn.Module):\n",
        "    \"\"\"Decoderのブロックのクラス\"\"\"\n",
        "    def __init__(self, d_model, d_inner_hid, n_head, d_k, d_v, dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_inner_hid: int, Position Wise Feed Forward Networkの隠れ層2層目の次元数\n",
        "        :param n_head: int,　ヘッド数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param dropout: float, ドロップアウト率\n",
        "        \"\"\"\n",
        "        super(DecoderLayer, self).__init__()\n",
        "        # Decoder内のSelf-Attention\n",
        "        self.slf_attn = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)\n",
        "        # Encoder-Decoder間のSource-Target Attention\n",
        "        self.enc_attn = MultiHeadAttention(n_head, d_model, d_k, d_v, dropout=dropout)\n",
        "        # Positionwise FFN\n",
        "        self.pos_ffn = PositionwiseFeedForward(d_model, d_inner_hid, dropout=dropout)\n",
        "\n",
        "    def forward(self, dec_input, enc_output, slf_attn_mask=None, dec_enc_attn_mask=None):\n",
        "        \"\"\"\n",
        "        :param dec_input: tensor, Decoderの入力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :param enc_output: tensor, Encoderの出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :param slf_attn_mask: tensor, Self Attentionの行列にかけるマスク, \n",
        "            size=(batch_size, len_q, len_k)\n",
        "        :param dec_enc_attn_mask: tensor, Soutce-Target Attentionの行列にかけるマスク, \n",
        "            size=(batch_size, len_q, len_k)\n",
        "        :return dec_output: tensor, Decoderの出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :return dec_slf_attn: tensor, DecoderのSelf Attentionの行列, \n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "        :return dec_enc_attn: tensor, DecoderのSoutce-Target Attentionの行列, \n",
        "            size=(n_head*batch_size, len_q, len_k)\n",
        "        \"\"\"\n",
        "        # Self-Attentionのquery, key, valueにはすべてDecoderの入力（dec_input）が入る\n",
        "        dec_output, dec_slf_attn = self.slf_attn(\n",
        "            dec_input, dec_input, dec_input, attn_mask=slf_attn_mask)\n",
        "        # Source-Target-AttentionのqueryにはDecoderの出力(dec_output), key, valueにはEncoderの出力（enc_output）が入る\n",
        "        dec_output, dec_enc_attn = self.enc_attn(\n",
        "            dec_output, enc_output, enc_output, attn_mask=dec_enc_attn_mask)\n",
        "        dec_output = self.pos_ffn(dec_output)\n",
        "\n",
        "        return dec_output, dec_slf_attn, dec_enc_attn"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnBteVrjp2cL"
      },
      "source": [
        "class Decoder(nn.Module):\n",
        "    \"\"\"DecoderLayerブロックからなるDecoderのクラス\"\"\"\n",
        "    def __init__(\n",
        "            self, n_tgt_vocab, max_length, n_layers=6, n_head=8, d_k=64, d_v=64,\n",
        "            d_word_vec=512, d_model=512, d_inner_hid=1024, dropout=0.1):\n",
        "        \"\"\"\n",
        "        :param n_tgt_vocab: int, 出力言語の語彙数\n",
        "        :param max_length: int, 最大系列長\n",
        "        :param n_layers: int, レイヤー数\n",
        "        :param n_head: int,　ヘッド数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param d_word_vec: int, 単語の埋め込みの次元数\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_inner_hid: int, Position Wise Feed Forward Networkの隠れ層2層目の次元数\n",
        "        :param dropout: float, ドロップアウト率        \n",
        "        \"\"\"\n",
        "        super(Decoder, self).__init__()\n",
        "        n_position = max_length + 1\n",
        "        self.max_length = max_length\n",
        "        self.d_model = d_model\n",
        "\n",
        "        # Positional Encodingを用いたEmbedding\n",
        "        self.position_enc = nn.Embedding(\n",
        "            n_position, d_word_vec, padding_idx=PAD)\n",
        "        self.position_enc.weight.data = position_encoding_init(n_position, d_word_vec)\n",
        "\n",
        "        # 一般的なEmbedding\n",
        "        self.tgt_word_emb = nn.Embedding(\n",
        "            n_tgt_vocab, d_word_vec, padding_idx=PAD)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        # DecoderLayerをn_layers個積み重ねる\n",
        "        self.layer_stack = nn.ModuleList([\n",
        "            DecoderLayer(d_model, d_inner_hid, n_head, d_k, d_v, dropout=dropout)\n",
        "            for _ in range(n_layers)])\n",
        "\n",
        "    def forward(self, tgt_seq, tgt_pos, src_seq, enc_output):\n",
        "        \"\"\"\n",
        "        :param tgt_seq: tensor, 出力系列, \n",
        "            size=(batch_size, max_length)\n",
        "        :param tgt_pos: tensor, 出力系列の各単語の位置情報,\n",
        "            size=(batch_size, max_length)\n",
        "        :param src_seq: tensor, 入力系列, \n",
        "            size=(batch_size, n_src_vocab)\n",
        "        :param enc_output: tensor, Encoderの出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :return dec_output: tensor, Decoderの最終出力, \n",
        "            size=(batch_size, max_length, d_model)\n",
        "        :return dec_slf_attns: list, DecoderのSelf Attentionの行列のリスト \n",
        "        :return dec_slf_attns: list, DecoderのSelf Attentionの行列のリスト\n",
        "        \"\"\"\n",
        "        # 一般的な単語のEmbeddingを行う\n",
        "        dec_input = self.tgt_word_emb(tgt_seq)\n",
        "        # Positional EncodingのEmbeddingを加算する\n",
        "        dec_input += self.position_enc(tgt_pos)\n",
        "\n",
        "        # Self-Attention用のマスクを作成\n",
        "        # key(=dec_input)のPADに対応する部分が1のマスクと、queryから見たkeyの未来の情報に対応する部分が1のマスクのORをとる\n",
        "        dec_slf_attn_pad_mask = get_attn_padding_mask(tgt_seq, tgt_seq)  # (N, max_length, max_length)\n",
        "        dec_slf_attn_sub_mask = get_attn_subsequent_mask(tgt_seq)  # (N, max_length, max_length)\n",
        "        dec_slf_attn_mask = torch.gt(dec_slf_attn_pad_mask + dec_slf_attn_sub_mask, 0)  # ORをとる\n",
        "\n",
        "        # key(=dec_input)のPADに対応する部分のみ1のマスクを作成\n",
        "        dec_enc_attn_pad_mask = get_attn_padding_mask(tgt_seq, src_seq)  # (N, max_length, max_length)\n",
        "\n",
        "        dec_slf_attns, dec_enc_attns = [], []\n",
        "\n",
        "        dec_output = dec_input\n",
        "        # n_layers個のDecoderLayerに入力を通す\n",
        "        for dec_layer in self.layer_stack:\n",
        "            dec_output, dec_slf_attn, dec_enc_attn = dec_layer(\n",
        "                dec_output, enc_output,\n",
        "                slf_attn_mask=dec_slf_attn_mask,\n",
        "                dec_enc_attn_mask=dec_enc_attn_pad_mask)\n",
        "\n",
        "            dec_slf_attns += [dec_slf_attn]\n",
        "            dec_enc_attns += [dec_enc_attn]\n",
        "\n",
        "        return dec_output, dec_slf_attns, dec_enc_attns"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uGZTXI2xp2cN"
      },
      "source": [
        "class Transformer(nn.Module):\n",
        "    \"\"\"Transformerのモデル全体のクラス\"\"\"\n",
        "    def __init__(\n",
        "            self, n_src_vocab, n_tgt_vocab, max_length, n_layers=6, n_head=8,\n",
        "            d_word_vec=512, d_model=512, d_inner_hid=1024, d_k=64, d_v=64,\n",
        "            dropout=0.1, proj_share_weight=True):\n",
        "        \"\"\"\n",
        "        :param n_src_vocab: int, 入力言語の語彙数\n",
        "        :param n_tgt_vocab: int, 出力言語の語彙数\n",
        "        :param max_length: int, 最大系列長\n",
        "        :param n_layers: int, レイヤー数\n",
        "        :param n_head: int,　ヘッド数\n",
        "        :param d_k: int, keyベクトルの次元数\n",
        "        :param d_v: int, valueベクトルの次元数\n",
        "        :param d_word_vec: int, 単語の埋め込みの次元数\n",
        "        :param d_model: int, 隠れ層の次元数\n",
        "        :param d_inner_hid: int, Position Wise Feed Forward Networkの隠れ層2層目の次元数\n",
        "        :param dropout: float, ドロップアウト率        \n",
        "        :param proj_share_weight: bool, 出力言語の単語のEmbeddingと出力の写像で重みを共有する        \n",
        "        \"\"\"\n",
        "        super(Transformer, self).__init__()\n",
        "        self.encoder = Encoder(\n",
        "            n_src_vocab, max_length, n_layers=n_layers, n_head=n_head,\n",
        "            d_word_vec=d_word_vec, d_model=d_model,\n",
        "            d_inner_hid=d_inner_hid, dropout=dropout)\n",
        "        self.decoder = Decoder(\n",
        "            n_tgt_vocab, max_length, n_layers=n_layers, n_head=n_head,\n",
        "            d_word_vec=d_word_vec, d_model=d_model,\n",
        "            d_inner_hid=d_inner_hid, dropout=dropout)\n",
        "        self.tgt_word_proj = nn.Linear(d_model, n_tgt_vocab, bias=False)\n",
        "        nn.init.xavier_normal_(self.tgt_word_proj.weight)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        assert d_model == d_word_vec  # 各モジュールの出力のサイズは揃える\n",
        "\n",
        "        if proj_share_weight:\n",
        "            # 出力言語の単語のEmbeddingと出力の写像で重みを共有する\n",
        "            assert d_model == d_word_vec\n",
        "            self.tgt_word_proj.weight = self.decoder.tgt_word_emb.weight\n",
        "\n",
        "    def get_trainable_parameters(self):\n",
        "        # Positional Encoding以外のパラメータを更新する\n",
        "        enc_freezed_param_ids = set(map(id, self.encoder.position_enc.parameters()))\n",
        "        dec_freezed_param_ids = set(map(id, self.decoder.position_enc.parameters()))\n",
        "        freezed_param_ids = enc_freezed_param_ids | dec_freezed_param_ids\n",
        "        return (p for p in self.parameters() if id(p) not in freezed_param_ids)\n",
        "\n",
        "    def forward(self, src, tgt):\n",
        "        src_seq, src_pos = src\n",
        "        tgt_seq, tgt_pos = tgt\n",
        "\n",
        "        src_seq = src_seq[:, 1:]\n",
        "        src_pos = src_pos[:, 1:]\n",
        "        tgt_seq = tgt_seq[:, :-1]\n",
        "        tgt_pos = tgt_pos[:, :-1]\n",
        "\n",
        "        enc_output, *_ = self.encoder(src_seq, src_pos)\n",
        "        dec_output, *_ = self.decoder(tgt_seq, tgt_pos, src_seq, enc_output)\n",
        "        seq_logit = self.tgt_word_proj(dec_output)\n",
        "\n",
        "        return seq_logit"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5iO0SDdJp2cQ"
      },
      "source": [
        "## 4. 学習"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qRsUUhMup2cR"
      },
      "source": [
        "def compute_loss(batch_X, batch_Y, model, criterion, optimizer=None, is_train=True):\n",
        "    # バッチの損失を計算\n",
        "    model.train(is_train)\n",
        "    \n",
        "    pred_Y = model(batch_X, batch_Y)\n",
        "    gold = batch_Y[0][:, 1:].contiguous()\n",
        "#     gold = batch_Y[0].contiguous()\n",
        "    loss = criterion(pred_Y.view(-1, pred_Y.size(2)), gold.view(-1))\n",
        "\n",
        "    if is_train:  # 訓練時はパラメータを更新\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "    gold = gold.data.cpu().numpy().tolist()\n",
        "    pred = pred_Y.max(dim=-1)[1].data.cpu().numpy().tolist()\n",
        "\n",
        "    return loss.item(), gold, pred"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gVeXwSp3p2cW"
      },
      "source": [
        "MAX_LENGTH = 20\n",
        "batch_size = 64\n",
        "num_epochs = 15\n",
        "lr = 0.001\n",
        "ckpt_path = 'transformer.pth'\n",
        "max_length = MAX_LENGTH + 2"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B5UNWBmSp2cY"
      },
      "source": [
        "model_args = {\n",
        "    'n_src_vocab': vocab_size_X,\n",
        "    'n_tgt_vocab': vocab_size_Y,\n",
        "    'max_length': max_length,\n",
        "    'proj_share_weight': True,\n",
        "    'd_k': 32,\n",
        "    'd_v': 32,\n",
        "    'd_model': 128,\n",
        "    'd_word_vec': 128,\n",
        "    'd_inner_hid': 256,\n",
        "    'n_layers': 3,\n",
        "    'n_head': 6,\n",
        "    'dropout': 0.1,\n",
        "}"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4grdni3dp2ca",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48ca2b3c-8eee-40d5-f9cc-faf3185b69c7"
      },
      "source": [
        "# DataLoaderやモデルを定義\n",
        "train_dataloader = DataLoader(\n",
        "    train_X, train_Y, batch_size\n",
        "    )\n",
        "valid_dataloader = DataLoader(\n",
        "    valid_X, valid_Y, batch_size, \n",
        "    shuffle=False\n",
        "    )\n",
        "\n",
        "model = Transformer(**model_args).to(device)\n",
        "\n",
        "optimizer = optim.Adam(model.get_trainable_parameters(), lr=lr)\n",
        "\n",
        "criterion = nn.CrossEntropyLoss(ignore_index=PAD, size_average=False).to(device)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
            "  warnings.warn(warning.format(ret))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SnHO46bip2cd"
      },
      "source": [
        "def calc_bleu(refs, hyps):\n",
        "    \"\"\"\n",
        "    BLEUスコアを計算する関数\n",
        "    :param refs: list, 参照訳。単語のリストのリスト (例： [['I', 'have', 'a', 'pen'], ...])\n",
        "    :param hyps: list, モデルの生成した訳。単語のリストのリスト (例： [['I', 'have', 'a', 'pen'], ...])\n",
        "    :return: float, BLEUスコア(0~100)\n",
        "    \"\"\"\n",
        "    refs = [[ref[:ref.index(EOS)]] for ref in refs]\n",
        "    hyps = [hyp[:hyp.index(EOS)] if EOS in hyp else hyp for hyp in hyps]\n",
        "    return 100 * bleu_score.corpus_bleu(refs, hyps)"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5jKt9aop2cf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "21acf18b-3e26-4395-9234-d6d066b2833f"
      },
      "source": [
        "# 訓練\n",
        "best_valid_bleu = 0.\n",
        "\n",
        "for epoch in range(1, num_epochs+1):\n",
        "    start = time.time()\n",
        "    train_loss = 0.\n",
        "    train_refs = []\n",
        "    train_hyps = []\n",
        "    valid_loss = 0.\n",
        "    valid_refs = []\n",
        "    valid_hyps = []\n",
        "    # train\n",
        "    for batch in train_dataloader:\n",
        "        batch_X, batch_Y = batch\n",
        "        loss, gold, pred = compute_loss(\n",
        "            batch_X, batch_Y, model, criterion, optimizer, is_train=True\n",
        "            )\n",
        "        train_loss += loss\n",
        "        train_refs += gold\n",
        "        train_hyps += pred\n",
        "    # valid\n",
        "    for batch in valid_dataloader:\n",
        "        batch_X, batch_Y = batch\n",
        "        loss, gold, pred = compute_loss(\n",
        "            batch_X, batch_Y, model, criterion, is_train=False\n",
        "            )\n",
        "        valid_loss += loss\n",
        "        valid_refs += gold\n",
        "        valid_hyps += pred\n",
        "    # 損失をサンプル数で割って正規化\n",
        "    train_loss /= len(train_dataloader.data) \n",
        "    valid_loss /= len(valid_dataloader.data) \n",
        "    # BLEUを計算\n",
        "    train_bleu = calc_bleu(train_refs, train_hyps)\n",
        "    valid_bleu = calc_bleu(valid_refs, valid_hyps)\n",
        "\n",
        "    # validationデータでBLEUが改善した場合にはモデルを保存\n",
        "    if valid_bleu > best_valid_bleu:\n",
        "        ckpt = model.state_dict()\n",
        "        torch.save(ckpt, ckpt_path)\n",
        "        best_valid_bleu = valid_bleu\n",
        "\n",
        "    elapsed_time = (time.time()-start) / 60\n",
        "    print('Epoch {} [{:.1f}min]: train_loss: {:5.2f}  train_bleu: {:2.2f}  valid_loss: {:5.2f}  valid_bleu: {:2.2f}'.format(\n",
        "            epoch, elapsed_time, train_loss, train_bleu, valid_loss, valid_bleu))\n",
        "    print('-'*80)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1 [0.4min]: train_loss: 77.35  train_bleu: 4.68  valid_loss: 41.18  valid_bleu: 10.77\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 2 [0.3min]: train_loss: 39.40  train_bleu: 12.26  valid_loss: 32.24  valid_bleu: 17.36\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 3 [0.3min]: train_loss: 32.03  train_bleu: 18.01  valid_loss: 28.11  valid_bleu: 22.09\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 4 [0.4min]: train_loss: 28.30  train_bleu: 21.60  valid_loss: 25.78  valid_bleu: 24.94\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 5 [0.4min]: train_loss: 25.82  train_bleu: 24.42  valid_loss: 24.29  valid_bleu: 27.22\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 6 [0.3min]: train_loss: 23.99  train_bleu: 26.68  valid_loss: 23.03  valid_bleu: 29.07\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 7 [0.3min]: train_loss: 22.54  train_bleu: 28.64  valid_loss: 22.22  valid_bleu: 30.07\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 8 [0.4min]: train_loss: 21.37  train_bleu: 30.12  valid_loss: 21.53  valid_bleu: 31.17\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 9 [0.3min]: train_loss: 20.35  train_bleu: 31.42  valid_loss: 20.95  valid_bleu: 31.49\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 10 [0.3min]: train_loss: 19.42  train_bleu: 32.88  valid_loss: 20.39  valid_bleu: 33.22\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 11 [0.4min]: train_loss: 18.67  train_bleu: 33.86  valid_loss: 20.12  valid_bleu: 33.81\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 12 [0.4min]: train_loss: 17.93  train_bleu: 35.10  valid_loss: 19.78  valid_bleu: 33.89\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 13 [0.3min]: train_loss: 17.29  train_bleu: 36.01  valid_loss: 19.39  valid_bleu: 34.61\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 14 [0.3min]: train_loss: 16.71  train_bleu: 36.88  valid_loss: 19.14  valid_bleu: 35.31\n",
            "--------------------------------------------------------------------------------\n",
            "Epoch 15 [0.4min]: train_loss: 16.15  train_bleu: 37.92  valid_loss: 18.95  valid_bleu: 35.35\n",
            "--------------------------------------------------------------------------------\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yVbfslwXp2ch"
      },
      "source": [
        "## 5. 評価"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kVsIBMSpp2ck"
      },
      "source": [
        "def test(model, src, max_length=20):\n",
        "    # 学習済みモデルで系列を生成する\n",
        "    model.eval()\n",
        "    \n",
        "    src_seq, src_pos = src\n",
        "    batch_size = src_seq.size(0)\n",
        "    enc_output, enc_slf_attns = model.encoder(src_seq, src_pos)\n",
        "        \n",
        "    tgt_seq = torch.full([batch_size, 1], BOS, dtype=torch.long, device=device)\n",
        "    tgt_pos = torch.arange(1, dtype=torch.long, device=device)\n",
        "    tgt_pos = tgt_pos.unsqueeze(0).repeat(batch_size, 1)\n",
        "\n",
        "    # 時刻ごとに処理\n",
        "    for t in range(1, max_length+1):\n",
        "        dec_output, dec_slf_attns, dec_enc_attns = model.decoder(\n",
        "            tgt_seq, tgt_pos, src_seq, enc_output)\n",
        "        dec_output = model.tgt_word_proj(dec_output)\n",
        "        out = dec_output[:, -1, :].max(dim=-1)[1].unsqueeze(1)\n",
        "        # 自身の出力を次の時刻の入力にする\n",
        "        tgt_seq = torch.cat([tgt_seq, out], dim=-1)\n",
        "        tgt_pos = torch.arange(t+1, dtype=torch.long, device=device)\n",
        "        tgt_pos = tgt_pos.unsqueeze(0).repeat(batch_size, 1)\n",
        "\n",
        "    return tgt_seq[:, 1:], enc_slf_attns, dec_slf_attns, dec_enc_attns"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mvsDBT7Up2ct"
      },
      "source": [
        "def ids_to_sentence(vocab, ids):\n",
        "    # IDのリストを単語のリストに変換する\n",
        "    return [vocab.id2word[_id] for _id in ids]\n",
        "\n",
        "def trim_eos(ids):\n",
        "    # IDのリストからEOS以降の単語を除外する\n",
        "    if EOS in ids:\n",
        "        return ids[:ids.index(EOS)]\n",
        "    else:\n",
        "        return ids"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lf7c2Hglp2c1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2fd5499a-f34b-40ef-92b4-0d2c3747ffc8"
      },
      "source": [
        "# 学習済みモデルの読み込み\n",
        "model = Transformer(**model_args).to(device)\n",
        "ckpt = torch.load(ckpt_path)\n",
        "model.load_state_dict(ckpt)"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xd9DvONGp2c3"
      },
      "source": [
        "# テストデータの読み込み\n",
        "test_X = load_data('./data/dev.en')\n",
        "test_Y = load_data('./data/dev.ja')\n",
        "test_X = [sentence_to_ids(vocab_X, sentence) for sentence in test_X]\n",
        "test_Y = [sentence_to_ids(vocab_Y, sentence) for sentence in test_Y]"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvFexG80p2c6"
      },
      "source": [
        "### 生成"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4CTH8DSyp2c8"
      },
      "source": [
        "test_dataloader = DataLoader(\n",
        "    test_X, test_Y, 1,\n",
        "    shuffle=False\n",
        "    )"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-pelUkHkp2c_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84756d99-0799-4872-8594-9b1cdb29dc78"
      },
      "source": [
        "src, tgt = next(test_dataloader)\n",
        "\n",
        "src_ids = src[0][0].cpu().numpy()\n",
        "tgt_ids = tgt[0][0].cpu().numpy()\n",
        "\n",
        "print('src: {}'.format(' '.join(ids_to_sentence(vocab_X, src_ids[1:-1]))))\n",
        "print('tgt: {}'.format(' '.join(ids_to_sentence(vocab_Y, tgt_ids[1:-1]))))\n",
        "\n",
        "preds, enc_slf_attns, dec_slf_attns, dec_enc_attns = test(model, src)\n",
        "pred_ids = preds[0].data.cpu().numpy().tolist()\n",
        "print('out: {}'.format(' '.join(ids_to_sentence(vocab_Y, trim_eos(pred_ids)))))"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "src: show your own business . </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
            "tgt: 自分 の 事 を しろ 。 </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
            "out: 自分 の こと を <UNK> し て い た 。\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F175okKMp2dC"
      },
      "source": [
        "## BLEUの評価"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q1MNLkgfp2dD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "21bd6bd4-55bb-4d9d-998f-64170103b113"
      },
      "source": [
        "# BLEUの評価\n",
        "test_dataloader = DataLoader(\n",
        "    test_X, test_Y, 128,\n",
        "    shuffle=False\n",
        "    )\n",
        "refs_list = []\n",
        "hyp_list = []\n",
        "\n",
        "for batch in test_dataloader:\n",
        "    batch_X, batch_Y = batch\n",
        "    preds, *_ = test(model, batch_X)\n",
        "    preds = preds.data.cpu().numpy().tolist()\n",
        "    refs = batch_Y[0].data.cpu().numpy()[:, 1:].tolist()\n",
        "    refs_list += refs\n",
        "    hyp_list += preds\n",
        "bleu = calc_bleu(refs_list, hyp_list)\n",
        "print(bleu)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24.052339616406652\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bVFIh5e1ma7m"
      },
      "source": [
        "\n",
        "# 何度か実施すると"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dr0uTerQmvv6"
      },
      "source": [
        "### 考察\n",
        "\n",
        "TransformerOnlyのモデルということで、前回のRNNありのモデルと比べて性能が向上することを期待していたが、\n",
        "劇的に変わるということはなかった。\n",
        "Attentionはどこまでも大きくスケールしていく、というような論文が2020年頃にGPTで有名なOpenAIから出されたということを思い出した。\n",
        "つまり、このサンプルくらいのサイズで、小規模の田中コーパスだと、AttentionだけとかRNNありとかの違いはそれほど違いが出ないのではないかと予想される。\n",
        "大規模な学習データで巨大なモデルで学習してこそ際立った性能が出るのではないか、という感想を持った。\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rkG_UPpCp2dL"
      },
      "source": [
        "'''\n",
        "src: show your own business .\n",
        "tgt: 自分 の 事 を しろ 。\n",
        "out: 自分 の こと を <UNK> し て い た 。\n",
        "\n",
        "src: the water was cut off yesterday . </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
        "tgt: 昨日 水道 を 止め られ た 。 </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
        "out: 昨日 水 は <UNK> を 切 っ た \n",
        "\n",
        "src: i should like to see you this afternoon . </S> <PAD> <PAD> <PAD> <PAD>\n",
        "tgt: 今日 の 午後 お 会 い し た い の で す が 。 </S> <PAD>\n",
        "out: 午後 あなた に 会 う の を 見 る べ き だ 。\n",
        "\n",
        "src: i gave him a call . </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
        "tgt: 私 は 彼 に 電話 を し た 。 </S> <PAD> <PAD> <PAD> <PAD> <PAD> <PAD>\n",
        "out: 彼 に 電話 を くれ た 。\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "'''\n"
      ],
      "execution_count": 40,
      "outputs": []
    }
  ]
}