{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eJm75_6N2A66"
      },
      "source": [
        "## FrozenLake-v1\n",
        "\n",
        "Here is a [better rendering](https://nbviewer.org/github/facebookresearch/Pearl/blob/main/tutorials/frozen_lake/frozen_lake.ipynb) of this notebook on [nbviewer](https://nbviewer.org/).\n",
        "\n",
        "This example shows how to use DQN to solve the `FrozenLake-v1` environment from gymnasium. This environment has observations as indices (tabular observation)which is not suitable for learning with a neural network. In what follows, we show how to use Pearl's `OneHotObservationsFromDiscrete` wrapper to convert observations to their one-hot representations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8NNfwWXGvn_o",
        "output": {
          "id": 361938766731901,
          "loadingStatus": "loaded"
        }
      },
      "outputs": [],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnPVwbqD2A7T"
      },
      "source": [
        "## Installation\n",
        "If you haven't installed Pearl, please make sure you install Pearl with the following cell. Otherwise, you can skip the cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1uLHbYlegKX-",
        "output": {
          "id": 1081297929584504,
          "loadingStatus": "loaded"
        },
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "841246f2-6a68-452a-f952-a6fa75428942"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Skipping Pearl as it is not installed.\u001b[0m\u001b[33m\n",
            "\u001b[0mCloning into 'Pearl'...\n",
            "remote: Enumerating objects: 6409, done.\u001b[K\n",
            "remote: Counting objects: 100% (1554/1554), done.\u001b[K\n",
            "remote: Compressing objects: 100% (378/378), done.\u001b[K\n",
            "remote: Total 6409 (delta 1322), reused 1262 (delta 1175), pack-reused 4855 (from 1)\u001b[K\n",
            "Receiving objects: 100% (6409/6409), 55.14 MiB | 9.88 MiB/s, done.\n",
            "Resolving deltas: 100% (4377/4377), done.\n",
            "/content/Pearl\n",
            "Processing /content/Pearl\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: gym in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (0.25.2)\n",
            "Collecting gymnasium[accept-rom-license,atari,mujoco] (from Pearl==0.1.0)\n",
            "  Downloading gymnasium-1.0.0-py3-none-any.whl.metadata (9.5 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (1.26.4)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (3.7.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (2.2.2)\n",
            "Collecting parameterized (from Pearl==0.1.0)\n",
            "  Downloading parameterized-0.9.0-py2.py3-none-any.whl.metadata (18 kB)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (2.32.3)\n",
            "Collecting mujoco (from Pearl==0.1.0)\n",
            "  Downloading mujoco-3.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.4/44.4 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (2.5.0+cu121)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (0.20.0+cu121)\n",
            "Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (from Pearl==0.1.0) (2.5.0+cu121)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from gym->Pearl==0.1.0) (3.1.0)\n",
            "Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from gym->Pearl==0.1.0) (0.0.8)\n",
            "\u001b[33mWARNING: gymnasium 1.0.0 does not provide the extra 'accept-rom-license'\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari,mujoco]->Pearl==0.1.0) (4.12.2)\n",
            "Collecting farama-notifications>=0.0.1 (from gymnasium[accept-rom-license,atari,mujoco]->Pearl==0.1.0)\n",
            "  Downloading Farama_Notifications-0.0.4-py3-none-any.whl.metadata (558 bytes)\n",
            "Collecting ale-py>=0.9 (from gymnasium[accept-rom-license,atari,mujoco]->Pearl==0.1.0)\n",
            "  Downloading ale_py-0.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.6 kB)\n",
            "Requirement already satisfied: imageio>=2.14.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari,mujoco]->Pearl==0.1.0) (2.35.1)\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from mujoco->Pearl==0.1.0) (1.4.0)\n",
            "Requirement already satisfied: etils[epath] in /usr/local/lib/python3.10/dist-packages (from mujoco->Pearl==0.1.0) (1.10.0)\n",
            "Collecting glfw (from mujoco->Pearl==0.1.0)\n",
            "  Downloading glfw-2.7.0-py2.py27.py3.py30.py31.py32.py33.py34.py35.py36.py37.py38-none-manylinux2014_x86_64.whl.metadata (5.4 kB)\n",
            "Requirement already satisfied: pyopengl in /usr/local/lib/python3.10/dist-packages (from mujoco->Pearl==0.1.0) (3.1.7)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (1.3.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (4.54.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (1.4.7)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (24.1)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (10.4.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (3.2.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->Pearl==0.1.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->Pearl==0.1.0) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->Pearl==0.1.0) (2024.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->Pearl==0.1.0) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->Pearl==0.1.0) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->Pearl==0.1.0) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->Pearl==0.1.0) (2024.8.30)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->Pearl==0.1.0) (3.16.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->Pearl==0.1.0) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->Pearl==0.1.0) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->Pearl==0.1.0) (2024.6.1)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch->Pearl==0.1.0) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch->Pearl==0.1.0) (1.3.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib->Pearl==0.1.0) (1.16.0)\n",
            "Requirement already satisfied: importlib_resources in /usr/local/lib/python3.10/dist-packages (from etils[epath]->mujoco->Pearl==0.1.0) (6.4.5)\n",
            "Requirement already satisfied: zipp in /usr/local/lib/python3.10/dist-packages (from etils[epath]->mujoco->Pearl==0.1.0) (3.20.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->Pearl==0.1.0) (3.0.2)\n",
            "Downloading mujoco-3.2.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m61.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading parameterized-0.9.0-py2.py3-none-any.whl (20 kB)\n",
            "Downloading ale_py-0.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m54.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading Farama_Notifications-0.0.4-py3-none-any.whl (2.5 kB)\n",
            "Downloading glfw-2.7.0-py2.py27.py3.py30.py31.py32.py33.py34.py35.py36.py37.py38-none-manylinux2014_x86_64.whl (211 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.8/211.8 kB\u001b[0m \u001b[31m14.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading gymnasium-1.0.0-py3-none-any.whl (958 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m958.1/958.1 kB\u001b[0m \u001b[31m46.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hBuilding wheels for collected packages: Pearl\n",
            "  Building wheel for Pearl (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for Pearl: filename=Pearl-0.1.0-py3-none-any.whl size=217554 sha256=20f65dc7bd6f7d90277bd9f5042c28bf50ee6c37710dab95e7c08ab39878e03b\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-2g2jb8uo/wheels/83/80/1d/d9211ba70ee392341daf21a07252739e0cb2af9f95439a28cd\n",
            "Successfully built Pearl\n",
            "Installing collected packages: glfw, farama-notifications, parameterized, gymnasium, ale-py, mujoco, Pearl\n",
            "Successfully installed Pearl-0.1.0 ale-py-0.10.1 farama-notifications-0.0.4 glfw-2.7.0 gymnasium-1.0.0 mujoco-3.2.4 parameterized-0.9.0\n",
            "/content\n"
          ]
        }
      ],
      "source": [
        "%pip uninstall Pearl -y\n",
        "%rm -rf Pearl\n",
        "!git clone https://github.com/facebookresearch/Pearl.git\n",
        "%cd Pearl\n",
        "%pip install .\n",
        "%cd .."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ej7hBEZT2A7Y"
      },
      "source": [
        "## Import Modules"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcb70ZC_h3OA",
        "output": {
          "id": 716826897185117,
          "loadingStatus": "loaded"
        }
      },
      "outputs": [],
      "source": [
        "from pearl.utils.functional_utils.experimentation.set_seed import set_seed\n",
        "from pearl.policy_learners.sequential_decision_making.deep_q_learning import DeepQLearning\n",
        "from pearl.replay_buffers import BasicReplayBuffer\n",
        "from pearl.utils.functional_utils.train_and_eval.online_learning import online_learning\n",
        "from pearl.pearl_agent import PearlAgent\n",
        "from pearl.utils.instantiations.environments.gym_environment import GymEnvironment\n",
        "from pearl.utils.instantiations.environments.environments import (\n",
        "    OneHotObservationsFromDiscrete,\n",
        ")\n",
        "from pearl.utils.instantiations.spaces.discrete import DiscreteSpace\n",
        "import torch\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from pearl.action_representation_modules.one_hot_action_representation_module import (\n",
        "    OneHotActionTensorRepresentationModule,\n",
        ")\n",
        "\n",
        "set_seed(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5WBfzaY2A7a"
      },
      "source": [
        "## Vanilla DQN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kulkpFAvnOQx",
        "output": {
          "id": 359198433580269,
          "loadingStatus": "loaded"
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "98f48799-1d93-45eb-88e6-973b071c1503"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "episode 2, step 147, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 3, step 247, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 4, step 347, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 6, step 427, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 9, step 540, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 10, step 640, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 11, step 716, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 12, step 816, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 13, step 916, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 14, step 1016, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 16, step 1150, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 19, step 1298, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 20, step 1348, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 21, step 1448, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 22, step 1548, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 23, step 1629, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 24, step 1729, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 25, step 1817, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 27, step 1960, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 29, step 2030, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 32, step 2186, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 33, step 2252, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 34, step 2352, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 37, step 2493, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 38, step 2589, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 39, step 2689, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 40, step 2702, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 43, step 2822, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 44, step 2922, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 46, step 3032, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 48, step 3138, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 55, step 3229, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 62, step 3317, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 67, step 3487, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 68, step 3587, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 70, step 3610, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 72, step 3764, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 73, step 3830, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 74, step 3930, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 75, step 4030, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 77, step 4147, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 79, step 4297, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 80, step 4364, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 82, step 4401, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 84, step 4569, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 85, step 4669, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 87, step 4782, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 88, step 4882, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 89, step 4982, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 90, step 5008, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 103, step 5102, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 119, step 5202, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 136, step 5305, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 153, step 5404, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 169, step 5505, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 184, step 5603, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 201, step 5705, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 216, step 5803, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 233, step 5903, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 249, step 6000, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 265, step 6101, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 282, step 6203, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 298, step 6301, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 315, step 6402, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 332, step 6503, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 349, step 6606, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 365, step 6705, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 381, step 6803, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 398, step 6904, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 414, step 7004, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 429, step 7104, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 445, step 7201, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 461, step 7300, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 478, step 7402, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 493, step 7500, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 510, step 7602, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 526, step 7702, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 542, step 7801, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 559, step 7900, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 575, step 8000, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 592, step 8102, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 608, step 8204, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 624, step 8302, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 642, step 8405, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 659, step 8505, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 675, step 8600, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 692, step 8702, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 710, step 8805, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 726, step 8905, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 742, step 9007, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 757, step 9100, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 775, step 9203, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 791, step 9302, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 807, step 9400, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 823, step 9503, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 840, step 9603, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 856, step 9703, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 872, step 9800, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 888, step 9902, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 905, step 10005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 921, step 10105, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 936, step 10202, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 954, step 10302, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 971, step 10404, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 987, step 10503, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1004, step 10605, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1020, step 10705, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1036, step 10805, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1051, step 10901, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1067, step 11000, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1083, step 11103, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1099, step 11202, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1116, step 11304, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1133, step 11405, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1149, step 11504, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1165, step 11602, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1180, step 11700, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1197, step 11800, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1213, step 11903, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1230, step 12009, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1244, step 12107, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1260, step 12200, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1277, step 12303, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 1293, step 12400, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1310, step 12505, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1327, step 12605, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1342, step 12701, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1359, step 12803, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1376, step 12904, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1391, step 13005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1407, step 13102, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1423, step 13201, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1440, step 13301, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1453, step 13403, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 1469, step 13501, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1485, step 13619, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1500, step 13704, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1519, step 13804, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1535, step 13906, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1551, step 14004, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1567, step 14100, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1584, step 14202, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1600, step 14300, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1617, step 14404, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1633, step 14500, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1649, step 14601, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1665, step 14702, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1682, step 14804, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1698, step 14905, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1714, step 15005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1730, step 15105, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1746, step 15202, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1763, step 15302, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1780, step 15404, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1796, step 15505, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1812, step 15600, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1828, step 15701, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1844, step 15802, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1860, step 15902, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1876, step 16005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1892, step 16102, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1909, step 16204, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1925, step 16303, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1941, step 16402, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1957, step 16500, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1973, step 16601, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 1990, step 16700, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 2006, step 16800, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2023, step 16902, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2040, step 17004, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2056, step 17105, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2071, step 17200, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2088, step 17300, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2106, step 17403, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 2122, step 17504, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2139, step 17606, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2155, step 17702, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2172, step 17805, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2187, step 17901, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2205, step 18001, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2221, step 18101, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2238, step 18205, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2254, step 18303, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2270, step 18404, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2286, step 18503, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 0.0\n",
            "episode 2302, step 18602, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2318, step 18705, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2333, step 18802, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2349, step 18904, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2366, step 19005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2382, step 19103, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2398, step 19201, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2415, step 19303, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2431, step 19403, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2448, step 19509, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2463, step 19603, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2479, step 19701, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2495, step 19801, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2511, step 19902, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n",
            "episode 2528, step 20005, agent=PearlAgent with DeepQLearning, BasicReplayBuffer, env=One-hot observations on FrozenLake-v1 from FrozenLake-v1\n",
            "return: 1.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAGwCAYAAABGogSnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeuElEQVR4nO3deXhTVfoH8G+SJk0LdKcLpVD2tWxFsCAgUAVEFtGfiIwsIo4KI1pFwFEQdcAVccdRAR1HRR3ABQShUEGoIPu+F8rSBVq6t0manN8f6U0buqVptnK/n+fpo725Sc4l7T1v3/OecxRCCAEiIiIimVC6uwFERERErsTgh4iIiGSFwQ8RERHJCoMfIiIikhUGP0RERCQrDH6IiIhIVhj8EBERkax4ubsBrmYymXDlyhU0adIECoXC3c0hIiIiGwghkJ+fj2bNmkGprF/uRnbBz5UrVxAVFeXuZhAREZEdLl68iObNm9frNWQX/DRp0gSA+R/Pz8/Pza0hIiIiW+Tl5SEqKsrSj9eH7IIfaajLz8+PwQ8REVED44iSFRY8ExERkaww+CEiIiJZYfBDREREsiK7mh9bGY1GGAwGdzejwVGr1VCpVO5uBhERUbUY/NxACIH09HTk5OS4uykNVkBAAMLDw7mOEhEReSQGPzeQAp/Q0FD4+vqyA68DIQSKioqQmZkJAIiIiHBzi4iIiCpj8FOB0Wi0BD7BwcHubk6D5OPjAwDIzMxEaGgoh8CIiMjjsOC5AqnGx9fX180tadikfz/WTBERkSdi8FMFDnXVD//9iIjIkzH4ISIiIllxa/Czbds2jBo1Cs2aNYNCocDatWtrfU5SUhJ69eoFb29vtG3bFitXrnR6O4mIiOjm4dbgp7CwEN27d8eHH35o0/kpKSkYOXIkBg8ejAMHDuCpp57CI488go0bNzq5pURERHSzcGvwM2LECLz66qu45557bDp/2bJlaNWqFd5++2106tQJM2fOxH333Yd33nnHyS31fFOmTIFCoYBCoYBarUZYWBjuuOMOLF++HCaTyercnTt34q677kJgYCC0Wi1iYmKwZMkSGI1Gq/MUCgW0Wi0uXLhgdXzs2LGYMmWKsy+JiIgaoN9PXUWp0VT7iW7UoGp+kpOTER8fb3Vs2LBhSE5OrvY5Op0OeXl5Vl83q+HDhyMtLQ3nz5/Hr7/+isGDB2PWrFm4++67UVpaCgBYs2YNBg0ahObNm2Pr1q04ceIEZs2ahVdffRUPPPAAhBBWr6lQKDB//nx3XA4RETUwu1OyMXn5boz6YAdKDMban+AmDWqdn/T0dISFhVkdCwsLQ15eHoqLiy1rzFS0ePFiLFy40O73FEKg2E0foI9aVaeZU97e3ggPDwcAREZGolevXrj11lsxdOhQrFy5EhMmTMD06dMxevRo/Pvf/7Y875FHHkFYWBhGjx6N7777DuPHj7c8NnPmTCxZsgSzZ89G165dHXdxRER0Uyk1mjD/xyMAgB5R/tCqPXedtwYV/Nhj3rx5SEhIsHyfl5eHqKgom59fbDCi83z31BQde3kYfDX1+4iGDBmC7t27Y/Xq1QgODkZWVhaeffbZSueNGjUK7du3xzfffGMV/PTv3x+nTp3C3Llz8csvv9SrLUREdPP6encqTqTnw99HjdnDOrq7OTVqUMNe4eHhyMjIsDqWkZEBPz+/KrM+gDkb4ufnZ/UlNx07dsT58+dx6tQpAECnTp2qPU86p6LFixdjw4YN2L59u1PbSUREDVNWgQ5vbTwJAHj2zvYIaqRxc4tq1qAyP3FxcVi/fr3VsU2bNiEuLs5p7+mjVuHYy8Oc9vq1vbcjCCGshs9urOupSKOp/APbuXNnTJo0CXPnzsWOHTsc0iYiIrp5vLnxJPJKStE5wg8P9m3p7ubUyq3BT0FBAc6cOWP5PiUlBQcOHEBQUBBatGiBefPm4fLly/jyyy8BAI899hg++OADPPfcc3j44YexZcsWfPfdd1i3bp3T2qhQKOo99ORux48fR6tWrdCuXTvL9/369avyvB49elT5GgsXLkT79u1tWotJrk5l5OPv/9mLyXEtMaV/K3c3h6jByirQ4f0tZ3A6Mx9L7u+BMD+tu5vkkUoMRkz8bBdOpedDpVLAS6mEl1IBL5UCapUSKqXC8r0CCpSaBEqNJhhNAgaTCUajgKHsWKlJoGljb7xxXzf0jg6qUzsOXMzBqj0XAQAvj+kCldLzV/l3a6++Z88eDB482PK9VJszefJkrFy5EmlpaUhNTbU83qpVK6xbtw5PP/003n33XTRv3hyfffYZhg1zT2amIdiyZQsOHz6Mp59+GsOGDUNQUBDefvvtSsHPTz/9hNOnT2Pp0qVVvk5UVBRmzpyJ559/Hm3atHFByxuepZtPIeVaIRatP4GB7ZuiddPG7m4SUYNSYjDi8z9SsCzpLPJ15hmqy3ekYN6Iqofq5e6HvZew98J1h71efkkppn2xB/97PA5tQ5vY9ByTSWDBj0cgBDCuZ2SdAyd3cWvwc/vtt9c4BFPV6s2333479u/f78RWNVw6nQ7p6ekwGo3IyMjAhg0bsHjxYtx9992YNGkSVCoVPvnkEzzwwAN49NFHMXPmTPj5+SExMRGzZ8/G9OnTcdddd1X7+vPmzcOnn36KlJQUq6JoAi5mF2HDkXQAgN5owoKfjuLLh/twnzMiG5hMAmv2X8bbv53EldwSAECEvxZpuSX4cf8VPDesY4PIJriS0STw+R8pAIBn7miPETHhMBiFOasjZXek700mQABeKgVUyvKskFpZ9l+VeY242T8cxP7UHExe/hdWP9HPpozb93sv4uClXDT29sLcEZ5d5FxRwx7PISsbNmxAREQEvLy8EBgYiO7du+O9997D5MmToVSaa9vvu+8+bN26Ff/6178wYMAAy7pHr7/+Op577rkaXz8oKAhz5szB888/7/RraWhW7DgPkwC6RvrhVEYBtp++hvWH0zGyW4S7m0bk0XacuYZF64/j6BXzvSgywAfPDmuPEV0j0Odfm5GeV4I/z2Whf9sQN7fUs2w+noGUa4Xw91Hj4dtaoZF3/bvzzyffgns/3omUa4WYsuIvfPf3W9FEq672/NwiA17fYC5yfiq+HUIb0PCkQtSUerkJ5eXlwd/fH7m5uZVmfpWUlCAlJQWtWrWCVttwPsT6KCkpwZgxY3Dx4kX8/vvvaNq0qUNeU07/jnklBsQtSkSh3ogvHu6DvReu473E0wj302LzM4PQ2AE3JaKbzamMfCxefxxbT14FADTx9sITg9tiav9oy/ow81Yfxje7U3Fvr+Z4+/7u7myux7n3453Ye+E6Zgxu49Bp5alZRRj38U5cK9Chf9tgrJjSBxqvqieGL/jxCL5IvoC2oY3x66wBUKucO4G8pv67rhrUVHdyPK1Wix9//BGTJk3Ctm3b3N2cBmnV7oso1BvRLrQxBrYLwRO3t0FUkA/S80rwXuJpdzfvppWaVYQnv9mPrScz3d0Uj2UyCfxy6AoW/3ocabnF7m6OxTubTmH40m3YevIqvJQKTOkXjd+fG4zHb29jtTDeuF6RAIANR9JQrPfc1YIr+vHAZUz/cg8y80uc9h57L2Rj74Xr0KiUmBwX7dDXbhHsi5VTb0EjjQo7zmThuR8OwmSqnCM5diUP//nTvPXRwtFdnB74OFrDai05hVarxdy5c3Hvvfe6uykNTqnRhBU7zOPujwxoZd4PTa3Cy6PNq2Ev/yMFpzLy3dnEm1JWgQ6Tlu/CTwev4B9f70dmnvM6moZICIHfjqbjrve2Y+bX+/HJ7+cQ//bvWLEjBcYqOjJXOpOZj3cTT8MkgOFdwrEpYRBeGt2lynVhercMRFSQDwr1Rvx2LN0Nra2b346m4+lVB7DpWAbeTzxT+xPs9Ok28z1nbM9mThlq6hrpj4//FgsvpQJrD1zB6xtPWD0uhMCCn47AJICRMRENckiSwQ9RPfx6JB1XcksQ3EiDMT0iLccHdwzFnZ3DUGoSeGHtkRoL+6luivSlePiLPTifVQQAKNCV4tV1x93cKs8ghMD201cx9qOdePQ/e3EiPR9NvL3QOcIPhXojFv58DPd8tANHLue6rY1Sx31n5zAseygWrUIaVXuuQqHAPT2bAwBW77vskvbZa++F6/jHN/shxZar9lx0Svbn/LVCbCwLBKcPaO3w15cMbN8Ur9/bDQDwye/nLH/kAcCPB67gr/PX4aNW4fmRDXMmHoOfKrCjqh+5/PsJIfDZ9nMAgIfiWlbax2b+qM7QqpXYnZKNtQc8+8bdUJQaTZj59X4cvJiDAF813n2gB5QK4KeDV7DjzDV3N8+t9pzPxgP//hMPfb4bBy/mwEetwhO3t8H2OYPxyz9uw6tju6KJ1guHLuVi9Ad/4NVfjqGwbDq5q2Tml2DNfvPvwt8H2dZx39PT/EfF9tNXPTbDd/ZqAaZ98Rd0pSYM6RiKni0CoC81WWZjOdLnf6RACGBIx1C0C7NtOrq97o1tjtnDOgAAXv7lGH49nIYCXSkWrTf/sTFzSFtEBlS9u4KnY/BTgVptrmovKipyc0saNunfT/r3vFntvXAdBy/lQuOlxN9urbyiafNAX/xjiHlhyX+tO4HcYoOrm3hTEcKcRdtyIhPeXkp8Prk3xvSIxKSymocXfzwCXWnDqAtxpMOXcjF5+W7ctywZu1KyoVEpMbV/NLY9NxjPDe+IAF8NlEoF/nZrSyQmDMLd3SJgEsBnf6TgjiW/Y/OxjNrfxEG+2HkeeqMJvVoEILalbevBtApphJ4tAmAS5iDX02Tml2Dy8t3IKTKge1QAPniwJ2YObgsA+Cr5AnKLHPd7n12ox/d7zYsJOjPrU9ETt7fBQ7e2hBDArFUHMOub/cjM1yE62BePDGi4i7lyGkoFKpUKAQEByMw0F1D6+vpynZY6EEKgqKgImZmZCAgIgErluTv6OsJn281/1Y3rGYmQxt5VnjN9QGv8b98lnLtaiHc2ncJLo7vU+rpnrxbgnU2nUGIwYu6IjjYvNlaVfanXsXrfJehLTTWep1Iq8MAtLdA9KsDu93K29xLP4Nu/LkKpAN6b0NPSeSbc2R6/HErDuauF+Gx7CmaUdTyuZjIJfL07FV/vSoXRJKD2UkCjUkKtUkLjpYSm7L9qlRI+ahVGxITj9g6hdr/ftQIdFv58DD+XBQReSgX+r3cU/jGkLZpV89d4qJ8WHzzYC/fGZuLFtUdw6XoxHvlyD4Z3CcdLo7sg3N95szMLdaX46k/zorWPDqzbQqnjekZif2oOVu+7jEec0OkLIfDVnxfgpVLi3l7Nq53ddKMCXSmmrvgLl64XIzrYF8sn94avxgtDOoaiY3gTnEjPxxfJ5/Hk0HYOaed/ki+gxGBCTKQ/bm3tmsUEFQoFXhrdBRl5JfjtWAYST5j7xwWjusDbq+He4znV/QZCCKSnpyMnJ8f1jbtJBAQEIDw8/KYOHC9kFeL2t5IgBPDb0wPRvob0844z1zDxs13m4ZmZt6FrpH+V510r0OHdzafx9e5US1GqWqXAY4PaYMbgtpWG1WpyrUCH1389ge/3XrL5OeF+WmxKGFjjuh7usuqvVMz532EAwKtju1bKtK3dfxlPrToArVqJTU8PQlSQr0vbl5pVhDn/O4Tkc1l1et7d3SIwf1RnhDaxPegQQuCng1fw0k9Hcb3IAIUCGNsjEk/Ft0PL4OrrZ25UrDdiaeIpfLbdXATd2NsLK6begluctELvih0pWPjzMbQKaYTNCYPqtGjh9UI9+izaDINRYONTA9Eh3LHDPb8cuoKZX5sXz40O9sW8uzrhzs5hNd7D9KUmTPviL2w/fQ3BjTRY/UQ/q3//nw5ewZPf7EeArxo75gyp9zo8JQYj+r+2BVmFerw3oSdGd29Wr9ez5/0nfrYLey9cx9COofh8yi0ufX/AsVPdGfxUw2g0wmDgMEVdqdXqmz7jAwAv/XQUK3eex6D2TfHFw31qPX/m1/vwy6E09GwRgP891g/KCjf+Yr0Ry3ek4OOksygoq8GI7xQGIYTlr6yWwb54dWxXDGhX8zpMpUYT/rsrFW//Zt5kEADG9mhWa23At3+l4mJ2Mab0i7YpO+VKW05kYPqXe2E0Ccwc3BbPltUgVCSEwIRP/8Sf57IR3ykMn03u7ZK2mUwCK3eex5sbT6LYYISPWoWEO9qjczM/6EtN0BtN0JeaYCj7r/T9+axCfL0rFSYB+Gm98PxdnXB/7yirn4uqZOaV4Pk1R7D5uHmoqlOEH968r1u1AbUtjqflYe7/DuHgpVzc1jYEXz3S1+7Xqk6p0YTb30rCpevFVQavtpj+5R5sOpaBvw9q7dDtLkoMRgx9+3dczimGRqWE3mjOkvZtFYQX7+5c5b+tEALPfHcQq/dfhq9GhW8fvRXdmgdYnWM0CQx9Ownns4rwwshO9c5Yfb0rFc+vOYzIAB/8Pvt2eLlhanl+iQE/H0zDyJgI+Pu6/o8kBj/14Mh/PJKn3GID4hYnokhvxH+m9ak1IAGA9NwSDH07CYV6I16/Nwbjb2lhWdL/rd9OIq1sSf9uzf3x/F2dcGvrYAghsPFoOl766RjSywo9R3dvhhfu7lRlpmDP+Wy8+ONRHE8zr5TbpZkfXh7TFbEtA2tt3x+nr+Fvn++CQgGsfrwferao/TmucPBiDh74958oNhhxb6/meOv/ulX71/iZzHwMX7odpSaBTyf1xh2dw5zatrNXCzDnh0PYU7a30q2tg/D6vd1szr4cuZyLuasP4chl8+fVp1UQFt0Tg7ahlfeEE0Lgh72X8Movx5BXUgq1SoF/DGmHx29v45D1VS5mF2HAG1uhUAB/zBni8CLWnw9ewT++2Y+gRhrsnDukTllMya+H0/D4f/ch3E+LHXOHOGy7i/cTT+PtTafQzF+LH2fehi92nsen289BV2qCQgGM62ku+q04JPjGhhP4KOksVEoFPpvcG4OrGb78dncq5q4+jDA/b2x7brDdw0Qmk0D8kt9x7lohXry7M6bd1nBrbeqDixwSudG3u1NRpDeiQ1gT3Gbj+hbh/lo8fUd7AMBrv57AhiNpuPv9P/DM9weRlluCyAAfvPtAD6x9oj9ubR0MwDzWPrxrBDY/MwhT+0dbZjXFv/07/rvrgmXhscz8EiR8dwD3LUvG8bQ8+Puo8crYrvhp5m02BT4AcFu7EIzrGQkhzKvqGow11wi5wvlrhXh45V8oNhgxsH1TvHZvTI3DEG1Dm2D6QPNf1y/9dNRpi+IZTQKf/H4Wd727HXsuXEcjjQqvju2Krx+5tU7DTl0j/bH2if54YWQn+KhV2J2Sjbve3Y53N5+2Kty+klOMKSv+wuwfDiGvpBTdmvvj53/chieHtnPYwnJRQb64tXUQhADW7LN9qNQWQgj8e5t5VuSkKmZF2mpIp1D4ab0s2104QnpuCT5KOgsAmDOiI5o28cazwzpgy7O3Y2yPZhAC+N++S7j9ra1YsukUCnWl+E/yectzFo+LqTbwAYB7ekUi3E+LjDwd/rfX/hmfiScyce5aIfy0Xhh/S5Tdr0PlmPkhqgOD0YSBb2xFWm4J3rivG+7vbfuNyGA04e73/sDJCoseNtF6YebgtpjcL7rWTuHwpVw8v+YwDpet0dKrRQCGdAzFJ7+fQ76uFAoFML53FJ4b3rHKBeNqk1WgQ/yS33G9yIA5wzvi8dvrVpRqq4vZRdh0LAOmWm49//nzAi5kFaFrpB++fTTOpm1CivSluGPJNlzOKbZ52X8hBP48l43Tmfnw91EjwFeDQF81Anw08PdVw0/rZQm6TmXkY/YPh3DwYg4AYEC7ECweF4PmgfWrMbqYXYQXfzyCpLKtHtqGNsaie2JwJrMAi9YfR4GuFBovJZ6Ob4/pA1o5Zcjjh72X8Oz3BxEd7Iutz97usJq95LNZmPDpn9Cqldg5d6hdP5uS59ccxte7UjGuVySW3N+j3m17etUBrNl/GbEtA/HDY3GVrvnAxRy8+ssxS3YvpLE3sgp1EAJIuKO9TYXMn/+Rgld+OYYWQb7Y8swguz67+5clY/f5bDx+exvMGd5wNg91NA571QODH7pRalYR9l+8jvhOYbUWJf544DJmfXsAIY29sWNu3dPYu1Oy8cC/k6FUmKcePzm0XZ06A6NJ4Mvk83hr40kUVshsdGvuj5fHdEWPes7WkjpAby8lfnt6YJ0yGbYwmQSGLd2G05kFNp0fFeSD/z3er04FwRuPpuPv/9kLtUqBX2cNrHIYSbI7JRtvbTyJ3eezqz1HpVSYgyIfNS5dL4beaEITrRdeHNkZ/9e7ucOCBCEEfj6Uhpd/PoprBXqrx3q2CMCb93Wv8Vrqq1BXilv+tRlFeiO+fyzOYYXPU1fsxtaTV/HQrS3xytiu9XqtPeezcd+yZPhqVNjzQjx8NfYXEe9LvY5xH+0EAPw0s3+lmh2JEAIbjqRj8a8nkJptXsZjQp8WWHRPV5s++yJ9Kfq/tgXXiwx494EeVouh2mJ/6nXc89FOqFUK/DFniE07rd+sHNl/c6o7yd7c1Yew82wW/LRe+NutLTGlX3SVS8YLISyLlk2Ka2nX+H2fVkHY8NRANPL2squuQqVUYGr/VhjeNRyvrjuOA6k5mDmkLe7vHeWQGoh7e0Vi9b5L2Hk2Cy+sPYIvH+7j0Fl7vx5Jx+nMAjTx9sLQTjVP8/bzUWP6gNZ1CnwA88rBQzqGYsuJTCz46Qi+mta30jUcupSDt347hW2nzJkWjZcSt7UNQbHeiOtFeuQWG5BTZECxwQijSSC7UI/sQnNAMrRjKP51T4zDp4UrFAqM7t4MA9uFYPH6E1i15yK0aiWevbMDpvZv5bAal+o08vbCyJgIfL/3En7Yc8khwc+pjHxsPXkVCgUcUqcS2zIQLYJ8kZpdhN+OZmBsz7oFEhKTSeDln48BAO6LbV5t4AOYP5cRMREY0ikU3+6+iAJdKf4+sLXNvxe+Gi883L8V3t50Ch9tPYtR3ZrVWthe0adlC6mO6REp68DH0Rj8kOxllf2VnVdSio+SzuLT7ecwtkckHhnQ2mpK7V/nr+PQpVx4eykxsW8Lu9+vpmnxtorw98GHD/aq9+vcSKFQYNE9MRi2dBu2n76GtQcuW7YXqC+TSeD9LeaNXqfe1goJZTVQjqZQKPDSqC7YceYadpzJws+H0izTgk+m52PJppPYeNQ8W8pLqcD9t5jXxonwrxyMlhiMyCs24HqRATlFemjVKnRr7u/UZRwCfDV4/b5umDagFfy0aqeuvXOj+2Kb4/u9l7DucBoWjO5cr8wKAHxaVuszvEs4omvYxsJWCoUCY3tG4r3E01i9/7Ldwc/aA5dx4GIOGmlUeK6K2YNV8fZSYXK/aLveb1JcND7Zdg4nM/KReCLT5mL8C1mF2HDE+VtZyBELnkn2pKmt/xjSFrdEB8JgFPh+7yUMW7oNk5fvxo4z18qyPuYb+bhezRFczaKGN4PokEaWWoZXfjmO64X6Wp5hm83HM3AiPR+NNCo83D/aIa9ZnRbBvpZVdl/55RgOX8rFrG/3Y/i727DxaEbZLJ5IJD4zCIvuiaky8AEArVqFUD8tOoQ3Qd/WwegeFeCy9avahzVxaeADALdEB6FFkC8KdKXYeLR+G4lm5JVYtnWRCtEdQdru4g87t7so1JXi9Q3mjTpnDGnrlI1Bb+Tvq7ZM7/9g6xmbtwBa/kcKTAIY1L6pw9c2kjsGPyR7OoO5dia+Uxi+f6wfVj/RD3fFhEOpAH4/dRUTP9uFu977A7+VbQMw7bZoN7bWNaYPaI0OYU2QXajHv9bXf9NQIQTe32Le5XpSv2gE+Npf9GqrRwe1RquQRriar8OoD/7AjweuQAjgrphw/PbUQCwZ38PhNU0NnVKpwL29zJm+7/fUb9bXyp3nYTAK3BIdiF4OXDqhvttdLPv9LDLydIgK8sHD/V03ZXzaba3g7aXEwYs5SD5b+2y1fanX8V3ZZ/CoA4NHMmPwQ7InZX681eZfh14tAvHRxFhsffZ2TI5rCR+1CsfT8iAEMLhD03ptN9FQaLyUWDQuBgqFuQh6Zz03DU06dRWHL+fCR63CIy5ao8TbS4WXx5Qv2Di4Q1P88o/b8NHEWKdvCNmQ3RtrzqzsPJuFS9ft2+ewQFeKr/68AKDuW1nYYlxZ9qeuO71fzC7CJ2VDcf+8q5Pd0+7t0bSJt2Wa+odJZ6o8x2QS2HQsA/+3bCfGfbQTxQYjYiL90a9NsMvaKRcMfkj2dAZz8KO5YQpqy+BGWDimK3bOHYJn72yPge2b4vm7HLeyrKeLbRmIv/U1p+qfX3MYJQb71s0RQuD9RHOtz8S+LVw6ZDigXVOsevRW/DSzP1ZM7VOvlZDlonmgr6WzrWtwIVn110Xkl5SiddNGGNrR/v3LqnN3t2ZQqxQ4lpaHE+l5Nj/vtV9PQF9qQlzrYAzrEu7wdtXm0YGt4aVUYMeZLOxPvW45XmIw4pvdqYh/53dM/3IP/jp/HWqVAvfFNse/J8Xe1FsFuQuDH5I9nSXzU/VfgYGNNJg5pB2+fLiP7DIGs4d3QJifN85nFeGDLVX/tVqbnWezsC81BxovpVvS931bB9c4m4cquy/WPPT1w95LNtenSAxGE5aXzYqcPqB1nWY22SqwkcayuOAaGwO0XeeysO5wGpQKYP6ozm4JKJoH+lqKtD9KOovrhXq8n3gat72+BfNWH8a5q4VoovXCY4Pa4I85Q/DW/3Wvth6N6ofBD8maEMKy4/mNmR8C/LRqLCzb62vZ72dxMj2/lmdU9l5Z1mfCLVEuKS6l+hveNRyNvb2Qml2E3SnVr4FUlfWH03A5pxghjTWW4mRnGNfL/NprD1y2bARcHaNJ4OVfzFPbH+jTAp0i3LfG22OD2kChADYdy0C/17bg7U2ncK1Aj2b+WrwwshOS5w3F3BEdOa3dyXi3J1nTV9jGQePFX4eqDOsSjjs6h6HUJPDM9wcsm6/aYndKNnalZEOtUuDvg5yzYjQ5nq/GvOYPYM7+2KriVhaT42pftbw+BncMhb+PGhl5Ory67hi+33MR209fxamMfOQWG6wyVt/vuYijV/LQROuFZ5y0xIKt2oY2xoiu5iG3YoMRnSP88O4DPfD7c4PxyIDWNq1kTvXHf2WSNV1pefDjzeCnSgqFAi+P6YK/zmfjyOU8PLziL6x8+Bab1oCR1vW5LzYKzRy8WSY51329m2PVnotYdzgNL43uUuvq5wDw2oYTOHolDz5qlV07t9eFt5cKo7pH4Ks/U7Fix/lKj2vVSoT7aRHqp7VkLGcNbecRy1QsHN0V0cGN0K9NCPq3DWZNjxvwbk+ypq8Q/HDYq3oR/j748uE+aOLthd3ns/HIF3tqLYDen3od209fg0qpwBNO2ieMnKd3y0BEB/uiSG+0LLRXk8+2n8Mnv5uzPi+P6YLAeuzhZatn7uiAp+Pb4/9im2Ng+6boENYE/j5qAECJwYTzWeZhu9xiA1qHNMKkuGint8kWTZt447nhHXFbuxAGPm7CzA/Jmq5CvY8zCjNvJt2aB+CLaX3w0Ge7sPNsFqZ/uQefTupd7dCGtK7P2B6RiAqq38af5HoKhXm20Vu/ncL3ey/i3tjqV/r+8cBlvLrOvB7Uc8M74P/qsOFvfQQ20mBWfOXNRUsMRmTklSAjT4f0vBJkF+gwpGMYh7bJgj8JJGuWYmfeFG3Sq0UgVj7cBz5qFbafvoYn/rvPKnsmOXI5F1tOZEKpAGYMZtanobqnV3MoFMCf57JxMbvqNX+2nbqKZ78/CACY0i8aj3tAbZdWrULL4Ebo0yoIo7s3w5T+rdAimAE4leMdn2RNV2oeumG9j+1uiQ7C51N6w9tLiS0nMvGPb/bBYLQOgKRan7u7NUPrps7biZycKzLAB/3bhAAA/revcuHzoUs5eOyrvTAYBe7uFoH5d7tnCjlRXfGOT7LGzI99+rUJwaeTekPjpcTGoxl4atUBlJYFQCfS8ywbh84c0tadzSQHkNb8+d++SzBVmFKecq0QU1f8hSK9Ef3bBuPt+7tz6JgaDN7xSdakmh9mfupuYPum+ORvsVCrFFh3KA2zfzgEo0lYFkMc0TXcITvYk3sN6xKOJt5euJhdjN3nzWv+ZOaXYNLyXcgq1KNLMz8s+1ssvL1ct1UEUX3xjk+yxsxP/QzuGIoPH+wFL6UCa/Zfxt//sxfrDqcBYNbnZuGjUeHu7uY1f77fcwn5JQZMWf4XLmYXo2WwL1ZO7YMmWrWbW0lUN7zjk6yV1/zwr1Z73dklHO8+0BNKBbD5eAaEAOI7haJLM+6jdbOQhr7WH07DI1/swbG0PIQ01uDLh/ugaRP3r5tDVFcMfkjWmPlxjJHdIvDO+B6Qal1nDqk8/Zgarl4tAtE6pBGKDUbsSslGI40KK6f2QcvgRu5uGpFduM4PyRprfhxnTI9ING3ijSKdET2iAtzdHHIghUKBe2Ob482NJ6FWKfDJQ73RNZKZPWq4GPyQrOmY+XGofmXTounmM7lfNC7nFOPOzmG4rR0/Z2rYGPyQrDHzQ2Sbxt5eWHRPjLubQeQQvOOTrJXX/LDgmYhILhj8kKxxhWciIvnhHZ9kjbO9iIjkh3d8kjXW/BARyQ/v+CRrzPwQEckP7/gka1zhmYhIfhj8kKzpOexFRCQ7vOOTrLHmh4hIfnjHJ1ljzQ8Rkfzwjk+yxmEvIiL54R2fZI17exERyQ/v+CRr5ZkfzvYiIpILBj8ka9JUd42KvwpERHLBOz7JmmW2l5q/CkREcsE7PsmaZbYXMz9ERLLBOz7JWnnmhzU/RERyweCHZE3HzA8Rkezwjk+yppf29mLNDxGRbPCOT7LGzA8Rkfzwjk+yJYSA3sjZXkREcsM7PsmWwSgghPn/vVUseCYikgu3Bz8ffvghoqOjodVq0bdvX+zevbvG85cuXYoOHTrAx8cHUVFRePrpp1FSUuKi1tLNRMr6AMz8EBHJiVvv+KtWrUJCQgIWLFiAffv2oXv37hg2bBgyMzOrPP/rr7/G3LlzsWDBAhw/fhyff/45Vq1aheeff97FLaebgc5gtPw/a36IiOTDrXf8JUuWYPr06Zg6dSo6d+6MZcuWwdfXF8uXL6/y/J07d6J///548MEHER0djTvvvBMTJkyoMVuk0+mQl5dn9UUElGd+1CoFlEqFm1tDRESu4rbgR6/XY+/evYiPjy9vjFKJ+Ph4JCcnV/mcfv36Ye/evZZg59y5c1i/fj3uuuuuat9n8eLF8Pf3t3xFRUU59kKowdIZONOLiEiOvNz1xteuXYPRaERYWJjV8bCwMJw4caLK5zz44IO4du0abrvtNgghUFpaiscee6zGYa958+YhISHB8n1eXh4DIAKACjO9WOxMRCQnDepP3qSkJCxatAgfffQR9u3bh9WrV2PdunV45ZVXqn2Ot7c3/Pz8rL6IAGZ+iIjkym2Zn5CQEKhUKmRkZFgdz8jIQHh4eJXPefHFF/HQQw/hkUceAQDExMSgsLAQjz76KP75z39CqWQnRrbTG7m6MxGRHLntrq/RaBAbG4vExETLMZPJhMTERMTFxVX5nKKiokoBjqpsfRYhLdhCZCNmfoiI5MltmR8ASEhIwOTJk9G7d2/06dMHS5cuRWFhIaZOnQoAmDRpEiIjI7F48WIAwKhRo7BkyRL07NkTffv2xZkzZ/Diiy9i1KhRliCIyFY6ru5MRCRLbg1+xo8fj6tXr2L+/PlIT09Hjx49sGHDBksRdGpqqlWm54UXXoBCocALL7yAy5cvo2nTphg1ahT+9a9/uesSqAFj5oeISJ4UQmbjRXl5efD390dubi6Ln2Xup4NX8OQ3+xHXOhjfPHqru5tDREQ1cGT/zT95SbakFZ41Xvw1ICKSE971SbYs6/ww+CEikhXe9Um2LDU/DH6IiGSFd32SrfLMD2cKEhHJCYMfki1mfoiI5Il3fZItywrPDH6IiGSFd32SLSnzw+CHiEheeNcn2eJsLyIieeJdn2SLNT9ERPLEuz7JFmd7ERHJE4Mfki1dKVd4JiKSI971Sbb0paz5ISKSI971SbZ0paz5ISKSI971SbZ0paz5ISKSIwY/JFvM/BARyRPv+iRbrPkhIpIn3vVJtjjbi4hInnjXJ9li5oeISJ541yfZYs0PEZE88a5PsqXnbC8iIlli8EOyJdX8cNiLiEheeNcnWRJCsOaHiEimeNcnWSo1CZiE+f9Z80NEJC+865MsSVkfgDU/RERyw+CHZElXIfhh5oeISF541ydZkjI/XkoFVEqFm1tDRESuxOCHZImrOxMRyRfv/CRLnOlFRCRfvPOTLHF1ZyIi+eKdn2RJx9WdiYhki8EPyRJrfoiI5It3fpIl1vwQEckX7/wkS6z5ISKSL975SZaY+SEiki/e+UmWyjM/LHgmIpIbBj/kMEIIFOuN7m6GTZj5ISKSL975yWGeX3MYPV/5DRezi9zdlFpxthcRkXzxzk8Os/fCdZQYTDh0KdfdTakVMz9ERPLFOz85TFHZkFd2oc7NLamdjsEPEZFs8c5PDlNiMAc/WYV6N7ekdnqu8ExEJFsMfshhyjM/nh/8sOaHiEi+eOcnhxBCoLhBZn74K0BEJDe885ND6EpNEML8/9kFnh/8WNb5UfFXgIhIbnjnJ4eouL5PQxj2smR+1PwVICKSG975ySGKDOXBT0MY9tIZmfkhIpIr3vnJISpmfq4X6WEyCTe2pnY6g5T54WwvIiK5YfBDDlEx+DGaBPJKDG5sTe30zPwQEckW7/zkEMUG6z29PH3oS2fgVHciIrninZ8cokhfavW9pxc9S5kfTnUnIpIf3vnJIW7czT3Lw6e7SzU/zPwQEckP7/zkEDcOezWczA8LnomI5IbBDzlEkf7G4MezNzfl9hZERPLFOz85REkDK3jm9hZERPLFOz85xI2ZH4+v+WHwQ0QkW7zzk0NINT8BvmoADaDmp5Q1P0REcsXghxxCmu3VPNAHgOcPe1k2NmXmh4hIdtx+5//www8RHR0NrVaLvn37Yvfu3TWen5OTgxkzZiAiIgLe3t5o37491q9f76LWUnWk4CcywBz8eHLBc6nRBGPZ9hsc9iIikh8vd775qlWrkJCQgGXLlqFv375YunQphg0bhpMnTyI0NLTS+Xq9HnfccQdCQ0Pxww8/IDIyEhcuXEBAQIDrG09WpI1Nmwf6AjAPewkhoFAo3NmsKknT3AFmfoiI5Mitwc+SJUswffp0TJ06FQCwbNkyrFu3DsuXL8fcuXMrnb98+XJkZ2dj586dUKvNtSXR0dE1vodOp4NOV56FyMvLc9wFkMWNw14Go0C+rhR+WrU7m1Ulqd4HYOaHiEiO3Hbn1+v12Lt3L+Lj48sbo1QiPj4eycnJVT7np59+QlxcHGbMmIGwsDB07doVixYtgtForPJ8AFi8eDH8/f0tX1FRUQ6/FgKKDebtLYIaaeCrMRcRZ3vojC+p3kepALy4sSkRkey47c5/7do1GI1GhIWFWR0PCwtDenp6lc85d+4cfvjhBxiNRqxfvx4vvvgi3n77bbz66qvVvs+8efOQm5tr+bp48aJDr4PMpMyPVq1CUCMNAM8teuZMLyIieXPrsFddmUwmhIaG4t///jdUKhViY2Nx+fJlvPnmm1iwYEGVz/H29oa3t7eLWyo/0jo/vhoVghtpcOl6scdOd+fqzkRE8ua24CckJAQqlQoZGRlWxzMyMhAeHl7lcyIiIqBWq6FSlf/F3qlTJ6Snp0Ov10Oj0Ti1zVQ9aYVnnwqZH0+d8cUFDomI5M1td3+NRoPY2FgkJiZajplMJiQmJiIuLq7K5/Tv3x9nzpyByVResHrq1ClEREQw8HEzKfPjo1EhqJE50+apw15c44eISN7cevdPSEjAp59+ii+++ALHjx/H448/jsLCQsvsr0mTJmHevHmW8x9//HFkZ2dj1qxZOHXqFNatW4dFixZhxowZ7roEKlNcIfMT3Lgs8+OhBc/c14uISN7cWvMzfvx4XL16FfPnz0d6ejp69OiBDRs2WIqgU1NToVSWd1BRUVHYuHEjnn76aXTr1g2RkZGYNWsW5syZ465LoDLFlpofrwrDXp4Z/JRnfljwTEQkR24veJ45cyZmzpxZ5WNJSUmVjsXFxeHPP/90cquoLgxGE0rLVkz2aVCzvZj5ISKSI979qd4q7ujuUzbbC/DkzA9nexERyRnv/lRv0kwvlVIBtUrh8cNezPwQEckb7/5Ub5Y1ftQqKBQKBFtme3GqOxEReR7e/aneLKs7l21rEVQ226vEYEKRvtRt7aoOV3gmIpI3Bj9Ub9K+XtKeXo00Kks9TZYHTndnzQ8Rkbzx7k/1ZlngUG0OfsxDX55b98OaHyIieePdn+qtuMLqzhJPLnrmCs9ERPLGuz/VW8XVnSWevNYPMz9ERPLGuz/VW3GFHd0lwR68uSkzP0RE8mb3Cs85OTnYvXs3MjMzrTYaBcx7cpF8SDU/WqvMj+dubqrjbC8iIlmzK/j5+eefMXHiRBQUFMDPzw8KhcLymEKhYPAjM9Kwl1Xmx4M3N+VsLyIiebPr7v/MM8/g4YcfRkFBAXJycnD9+nXLV3Z2tqPbSB6uWF99zY8nFjyz5oeISN7suvtfvnwZTz75JHx9fR3dHmqALAXPmvJEoicXPLPmh4hI3uy6+w8bNgx79uxxdFuogbpxnR8ADWSdH9b8EBHJkV01PyNHjsTs2bNx7NgxxMTEQK1WWz0+evRohzSOGoaSKmp+LJmfAk+c7cWaHyIiObMr+Jk+fToA4OWXX670mEKhgNForF+rqEGR9u/SWk11N8/2KtQbUWIwWs0EczfW/BARyZtdd3+TyVTtFwMf+Sk2mIMJ3woBjp+PF7yU5lmAnjb0xZofIiJ5q/Pd32AwwMvLC0eOHHFGe6gBKi7L/FTc3kKhUCDQQ+t+mPkhIpK3Ot/91Wo1WrRowQwPWZTP9rIe2gr20BlfOgY/RESyZtfd/5///Ceef/55rulDAKqe7QVUXOvHs4qeOduLiEje7Cp4/uCDD3DmzBk0a9YMLVu2RKNGjawe37dvn0MaRw1DSRV7ewEVZ3x5WuaHs72IiOTMruBn7NixDm4GNWRFVUx1Bzx3rR/W/BARyZtdwc+CBQsc3Q5qwIqr2NgUKN/c1NOCH872IiKSN979qV6MJmEJJnw11rG0tLmpJxU8G00CpSYBgDU/RERyZVfmR6lUWu3kfiPOBJMPaXVnoHLBsycOe0lDXgAzP0REcmVX8LNmzRqr7w0GA/bv348vvvgCCxcudEjDqGGQZnoBgFZtHUx44s7uFYMf1vwQEcmTXcHPmDFjKh2777770KVLF6xatQrTpk2rd8OoYZAyPz5qVaVsoGXYy4P295JmeikUsKxATURE8uLQP31vvfVWJCYmOvIlycMVVTPNHSgveM4rKYXBaKr0uDtUXOCwpqFbIiK6eTks+CkuLsZ7772HyMhIR70kNQDS6s5VbVwa4KOGlFy57iFDX5aZXioOeRERyZVdw16BgYFWfzULIZCfnw9fX1989dVXDmsceT5pR/eqMj9KpQKBvhpkFeqRVahHqJ/W1c2rxLLGjwftMk9ERK5lV/DzzjvvWAU/SqUSTZs2Rd++fREYGOiwxpHnk9b4uXFfL0lQI3Pw4ylFz5bVnZn5ISKSLbuCnyFDhiAqKqrKmonU1FS0aNGi3g2jhqHYUPW+XpIgD9vctDzzw+CHiEiu7OoBWrVqhatXr1Y6npWVhVatWtW7UdRwFNWS+ZFmfGV7yIwv1vwQEZFdPYAQosrjBQUF0GrdX9dBrlNSzb5eEk9b64c1P0REVKdhr4SEBACAQqHA/Pnz4evra3nMaDRi165d6NGjh0MbSJ6tqJp9vSTSdHdPGfayTHVn5oeISLbqFPzs378fgDnzc/jwYWg0GstjGo0G3bt3x7PPPuvYFpJHK65hnR/A87a40JdtvcKaHyIi+apT8LN161YAwNSpU/Huu+/Cz8/PKY2ihqOhFTzrDKz5ISKSO7t6gBUrVsDPzw9nzpzBxo0bUVxcDKD6WiC6eZVPda86jva8zA9nexERyZ1dPUB2djaGDh2K9u3b46677kJaWhoAYNq0aXjmmWcc2kDybJbZXtVlfhp7VvDDzA8REdnVAzz11FNQq9VITU21KnoeP348NmzY4LDGkeezdbbX9SI9jCb3ZwYtmR8vzvYiIpIruxY5/O2337Bx40Y0b97c6ni7du1w4cIFhzSMGgZpe4vqMj+BvubgRwggp0iP4MbeLmtbVXRlwZrGi5kfIiK5sqsHKCwstMr4SLKzs+Ht7d7OjVzLUvBcTeZHrVLC30cNwDOGvnTG8l3diYhInuzqAQYMGIAvv/zS8r1CoYDJZMIbb7yBwYMHO6xx5PmKa6n5AcqLnq8VeEDwI9X8MPghIpItu4a93nzzTQwZMgR79uyBXq/Hc889h6NHjyI7Oxs7duxwdBvJgxXXUvMDmOt+zl0r9IjMD2t+iIiozsGPwWDAk08+iZ9//hmbNm1CkyZNUFBQgHHjxmHGjBmIiIhwRjvJQ1lWeK4l+AGA7EL37+8lbW/BzA8RkXzVOfhRq9U4dOgQAgMD8c9//tMZbaIGpLbZXkD55qaesNChZXsLBj9ERLJlVw/wt7/9DZ9//rmj20INUG3r/ACetbmpvpSzvYiI5M6ump/S0lIsX74cmzdvRmxsLBo1amT1+JIlSxzSOPJsQohaZ3sBnrW5KTM/RERkV/Bz5MgR9OrVCwBw6tQpq8cUCkX9W0UNgq7UBGlHE1tme2V7wGwv1vwQEZFdwY+0wSnJmzTNHWg4w17lmR/O9iIikiv++Ut2K5JWS1Yp4VXDXlmetLO7nsNeRESyxx6A7Fa+o3vNWRRpttf1Ij1Mbt7fS1dW8Mzgh4hIvtgDkN2k4Kemae5AeebHaBLIKzE4vV01Yc0PERGxByC7WWZ61VDvA5jraxp7m8vL3D30xZofIiJi8EN2s+zoXkvmB/CcomdmfoiIiD0A2c2WTU0lllWe3Tzdnev8EBGRR/QAH374IaKjo6HVatG3b1/s3r3bpud9++23UCgUGDt2rHMbSFWyZYFDSTAzP0RE5CHc3gOsWrUKCQkJWLBgAfbt24fu3btj2LBhyMzMrPF558+fx7PPPosBAwa4qKV0I1u2tpB4wuamJpOosKu723/0iYjITdzeAyxZsgTTp0/H1KlT0blzZyxbtgy+vr5Yvnx5tc8xGo2YOHEiFi5ciNatW7uwtVSRLZuaSjxhiwsp8AGY+SEikjO39gB6vR579+5FfHy85ZhSqUR8fDySk5Orfd7LL7+M0NBQTJs2rdb30Ol0yMvLs/oixyiycZ0fwDOGvaR6H4CzvYiI5Mytwc+1a9dgNBoRFhZmdTwsLAzp6elVPuePP/7A559/jk8//dSm91i8eDH8/f0tX1FRUfVuN5mVT3WvfZcUT5jtpa8Q/KhV3IOOiEiuGlTuPz8/Hw899BA+/fRThISE2PScefPmITc31/J18eJFJ7dSPspXeK79xyjIA2Z7VVzdmRvwEhHJl10bmzpKSEgIVCoVMjIyrI5nZGQgPDy80vlnz57F+fPnMWrUKMsxk8n817yXlxdOnjyJNm3aWD3H29sb3t7eTmg9la/wXPuPkScMe3GmFxERAW7O/Gg0GsTGxiIxMdFyzGQyITExEXFxcZXO79ixIw4fPowDBw5YvkaPHo3BgwfjwIEDHNJyMWljU22dZnvpIYR79vfi6s5ERAS4OfMDAAkJCZg8eTJ69+6NPn36YOnSpSgsLMTUqVMBAJMmTUJkZCQWL14MrVaLrl27Wj0/ICAAACodJ+ezdW8vAAgum+2lN5pQoCtFE63aqW2rCnd0JyIiwAOCn/Hjx+Pq1auYP38+0tPT0aNHD2zYsMFSBJ2amgqlkp2VJyo2lG1vYUPmx0ejgo9ahWKDEdmFercEP1zdmYiIAA8IfgBg5syZmDlzZpWPJSUl1fjclStXOr5BZJPiOkx1B8xDX5dzipFVqEfL4EbObFqVWPNDRERAA5vtRZ6lLis8A+X7e2W7acZXxdleREQkX+wFyG51WeEZcP9aP8z8EBERwOCH6kHK/Ngy2wsoD37ctcUFZ3sRERHA4IfqobiOmR9prZ+sAvdsbsrMDxERAQx+qB7qXvBsnu7urmEv1vwQERHA4IfsZDCaUGoyL1boa8PeXkCFzI+bh72Y+SEikjf2AmQXqd4HALQ27O0FuL/g2RL8qPhjT0QkZ+wFyC7STC+VUmFzMCFtburu2V7eav7YExHJGXsBskvFNX5s3SG9fNjLPQXP5ZkfzvYiIpIzBj9kl7oWOwPlw14lBhOK9KVOaVdNmPkhIiKAwQ/ZqS77ekkae3tZhsiy3LDKszTbizU/RETyxl6A7FKsN2dRbF3jBwAUCoVbi56Z+SEiIoDBD9lJGraydXVniTuDH872IiIigMEP2amuqztLpM1N3bHWT3nmhwXPRERyxuCH7FJcxx3dJeWZH9fP+LKs8MzMDxGRrLEXILsU2THbC3Dv5qZ6I2t+iIiIwQ/Zye5hLynz447ZXgbW/BAREYMfspP9w17u29yUmR8iIgIY/JCdpMyPj8a2TU0l7hz2Ks/8sOCZiEjOGPyQXYrszPyEuHF/L2Z+iIgIYPBDdiqxs+bHrev8GLjCMxERMfghO1kWOaxzwbO55qdAV2qZeu4qzPwQERHA4IfsVFxWP+Nbx2EvPx8veCnNu8C7OvvD2V5ERAQw+CE7FZdlfuq6zo9CoUCgVPTs4unuOiNXeCYiIgY/ZKfy2V51DySC3VD3I4SwbG/BzA8RkbyxFyC72DvbC3BP0bNU7wOw5oeISO7YC5BdSvT2zfYC3LPWj7SjO8DMDxGR3LEXILsUGezP/AS7YXNTfYXgx9uLP/ZERHLGXoDsUmznxqaAe7a40FWo91EoFC57XyIi8jwMfqjOjCZhCSbsqvlp7PrZXlLmh1kfIiJiT0B1Jq3uDAC+ddzbC3DPbC9pQUUNgx8iItljT0B1Js30AuzLpLij4JmZHyIikrAnoDorqVDsrFTWvX4m2LLIoesKni01Pwx+iIhkjz0B1VlRPYqdgfLMT15JKQwV1t9xpvLMD1d3JiKSOwY/VGfF9ZjmDgABvhpIE66uu2joizU/REQkYU9AdVZk575eEpVSgUBf19b9sOaHiIgk7AmozqSaH3tWd5a4eosL1vwQEZGEPQHVmVTzo63H7uiunvGlY+aHiIjKsCegOiuqx75eEstaPy6a8cXMDxERSdgTUJ2V1LPgGXD9sBdnexERkYTBD9VZfae6AxXW+uFsLyIicjH2BFRnlk1NG2Tmhz/yRERyx56A6qzYEbO9Gpt3dnd1wTMzP0RExJ6A6swRmR9Xb27Kmh8iIpIw+KE6K6/5qfuO7hJ3DXsx80NEROwJqM4cscihlPm5XqSH0SQc0q6aSAXPrPkhIiL2BFRnlu0t6jHsFVgW/AgB5BQ5P/vDgmciIpKwJ6A6s2xsWo/Mj1qlhL+PGoBrhr64wjMREUnYE1CdOaLgGXDtWj+s+SEiIgl7AqozR0x1B1xb9KzjbC8iIirD4IfqzLKxqYOCH2Z+iIjIldgTUJ05YrYXAAQ3ljY3dUXmh7O9iIjIjD0B1VmRg2p+yoe9nL+zO1d4JiIiCXsCqhMhhENmewFAUCPXbXHBFZ6JiEjC4IfqRFdqgihbk9BRs71cWfDMzA8REbEnoDqRprkDjhz24jo/RETkOh7RE3z44YeIjo6GVqtF3759sXv37mrP/fTTTzFgwAAEBgYiMDAQ8fHxNZ5PjlVUNuSlUSnhparfj49rZ3uVtZvBDxGR7Lm9J1i1ahUSEhKwYMEC7Nu3D927d8ewYcOQmZlZ5flJSUmYMGECtm7diuTkZERFReHOO+/E5cuXXdxyebIscFjPeh+gfLbX9UI9hHDu/l7M/BARkcTtPcGSJUswffp0TJ06FZ07d8ayZcvg6+uL5cuXV3n+f//7XzzxxBPo0aMHOnbsiM8++wwmkwmJiYkubrk8OWp1Z6A881NqEsgrLq3361VHCAG9kTU/RERk5taeQK/XY+/evYiPj7ccUyqViI+PR3Jysk2vUVRUBIPBgKCgoCof1+l0yMvLs/oi+zlqdWfAPPOqsbcXACDLidPdDUZhKdLmbC8iInJr8HPt2jUYjUaEhYVZHQ8LC0N6erpNrzFnzhw0a9bMKoCqaPHixfD397d8RUVF1bvdcibt6K51QOYHcE3dj5T1ATjsRUREHjDsVR+vvfYavv32W6xZswZarbbKc+bNm4fc3FzL18WLF13cypuLo1Z3lliCHyeu8qwzlM9Q09SzSJuIiBo+L3e+eUhICFQqFTIyMqyOZ2RkIDw8vMbnvvXWW3jttdewefNmdOvWrdrzvL294e3t7ZD2UoXVnR0U/LhirR8p86NWKaBUKpz2PkRE1DC49c9gjUaD2NhYq2JlqXg5Li6u2ue98cYbeOWVV7Bhwwb07t3bFU2lMo7a2kLiii0udAau7kxEROXcmvkBgISEBEyePBm9e/dGnz59sHTpUhQWFmLq1KkAgEmTJiEyMhKLFy8GALz++uuYP38+vv76a0RHR1tqgxo3bozGjRu77TrkosRBW1tIghq7ruaHM72IiAjwgOBn/PjxuHr1KubPn4/09HT06NEDGzZssBRBp6amQqks77Q+/vhj6PV63HfffVavs2DBArz00kuubLosSZkfR9X8uGLYqzzzw+CHiIg8IPgBgJkzZ2LmzJlVPpaUlGT1/fnz553fIKqWNNXdcbO9zPVYzq354erORERUjr0B1UmxkzI/zp3txcwPERGVY29AdeLIFZ4B12xuqmPNDxERVcDegOqkyFLw7JgR04rBj7P29+JsLyIiqojBD9WJozM/0uameqMJBTrn7O9lme3FBQ6JiAgMfqiOig3mAMVRNT++Gi9o1eYfQ2cNfUkrPHur+eNOREQMfqiOpMyPo2Z7AUBw2YwvZ631w8wPERFVxN6A6sTR6/wAFep+nDTjy1Lz48CAjYiIGi4GP1Qnjl7hGXD+jC9mfoiIqCL2BlQnjt7bCygvenbWsFd55oc/7kRExOCH6kha4dmRw17BTt7c1LLCMzM/REQEBj9UR5ap7g4d9rKv4NlkEjatDcTMDxERVcTegGxmMJpQajIHG75qx20LZ8/mptmFevRZtBmPf7Wv1nOlmh9vZn6IiAgMfqgOpHofANBqHPejY0/B85/nsnCtQI9NxzMsRdjV4WwvIiKqiMEP2UwKMlRKhUPrZ4Ia131z06NXcgEARpPAyfT8Gs/lbC8iIqqIvQHZrOJML4VC4bDXtWfY6+iVvCr/vyq6Uq7wTERE5dgbkM2cUewMlA97FRuMlveojXXwk1vjufpSZn6IiKgcewOymbSvlyPX+AGAxt5elsAky4bp7pn5JbiaX35e7ZkfzvYiIqJy7A3IZsV6cxDhyDV+AEChUNSp6FkKdhp7m2ecnUjPg9FU/ZR3nSXzw4JnIiJi8EN1UKQ3Z34cuampRAp+bFnr51hZ8DO4Yyh8NSqUGEw4d7Wg2vMtmR8v/rgTERGDH6oDZ6zuLJG2uLBlc1Opxicm0g+dI/wAAEdqqPux1Pww+CEiIjD4oToodsK+XpK6DHsduWzO/HRt5o8uzczBz9HL1df9WGZ7MfghIiIAjluml256RU6a7QXYPuyVV2JAanYRAKBzMz9cul4MoOaiZ2Z+iIioIvYGZDNp2MsZmR9prZ+sgppne0n1PpEBPgjw1aCzlPm5klvtPl/lNT8seCYiIgY/VAfSsJczan6kzU1rG/aSMjzScFf7sCZQqxTIKym1ZIFuxMwPERFVxN6AbCZlfrRuHPaSip27NPMHYA5o2oU2KXus6qEv1vwQEVFF7A3IZlLNjyN3dJdYZnvVEvwcuyHzU/H/j1Ux40sIYcn8MPghIiKAwQ/VgbSxqY8Dd3SX2DLbq8RgxOlM83o+XSIrBz9VZX5KTQLS+oes+SEiIoDBD9WBtMihj8YJmZ+y4KdAV2oZprrRqYx8GE0CQY00CPfTWo53iTQPgVUV/EhZH4A1P0REZMbegGxWbDAHEs6Y7eWnVUOlNO8UX132p2Kxc8Vd5TtF+EGhANLzSnDthtliOgY/RER0A/YGZLPissyPM2Z7KZUKBPpK092rC37MNT2dK9T7AOY9vqKDG5WdY539kTI/XkqFJbgiIiJ5Y/BDNnPmOj9A+dBX7Zkf/0qPVVzvpyLO9CIiohuxRyCbOXOFZ6DmomejSeBEWj4A65lekuqKnrnGDxER3Yg9AtmsxIl7ewFAUOPq1/pJuVaAYoMRvhoVWpUNcVXUtSwbdOyG4EfH4IeIiG7AHoFsVuTEXd2BisNelbe4kDYz7RThB2UVtTtS5iflWiEKdKWW49zagoiIbsTgh2wmbW+hdVbmp4Zhr/KVnSsPeQFAcGNvy/T342nl2R+p5oeZHyIikrBHIJsYTcKSRXF25qeq2V437ulVFUvdz+Xyomeu7kxERDdij0A2kVZ3BpxX8BzcuOrNTYUQNc70klRV9MyaHyIiuhF7BLKJNNMLALROqp+pbtjrck4xcosN8FIq0C6scbXP79ys8krPzPwQEdGN2COQTUoqrPFTVcGxIwRXs7O7FMy0D2tSY+GylPk5nZlvqfUpz/yw4JmIiMwY/JBNnL3GD1Ce+cktNsBgLN+WwpZ6HwBoHugDfx81DEaB0xnmDVCZ+SEiohuxRyCbOHt1ZwAI8NVA2rLrelF59udYLTO9JAqFAp0jrFd65mwvIiK6EXsEskn5ju7OC35UFfb3qlj3Y8n8RFZf7Cy5seiZmR8iIroRewSySYmTFziUWIqey6a7ZxfqkZZbAoXCvMBhbbpEWgc/OgY/RER0A/YIZJMiJy9wKAm6oehZGr6KDm6Ext5etT5fmgp/PC0PRpOokPlhwTMREZkx+CGbSKs7Ozvzc+PO7lIGp3Mt9T6S1iGNoFUrUaQ34nxWIWt+iIioEvYIZBNXFDwDVWV+bJvpJfFSKdExvHzoizU/RER0I/YIZBNXTHUHKm9uKm1VUdPKzjcqL3rOhb5syrxGxR91IiIyY49ANpGGvVyV+cku1KNQV4qUrEIAtmd+zOeaA6VjV/KgM5RlftT8USciIjP2CGSTYlfN9irb3yurQI/jaXkQAgjz80ZI2XFbVJzublnhmZkfIiIqwx6BbOKqzE/FgmdbNjOtSofwJlApFcgu1ONCtjlz5O3kdhMRUcPB4IdsUl7zU/t08/qoWPB81MaVnW+kVavQtql5A9TjafkAmPkhIqJy7BHIJuUbmzr3R0bK/Fwv0uPw5brN9KpIeo7RJACw5oeIiMqxRyCbSNtb+Do58xNYFvwIYV6oEKj7sBdQeV0gZn6IiEjCHoFsIhU8a51c8KxWKeGnLQ+w/LReaB7oU+fXuTFgYs0PERFJGPyQTSwrPLsgiAiuMLOrczM/KKSt3uuAmR8iIqoOewSyiWWFZydnfoDyomcA6GrHkBcA+PuoERVUnjFizQ8REUk8okf48MMPER0dDa1Wi759+2L37t01nv/999+jY8eO0Gq1iImJwfr1613UUvly1QrPgHXwI+3Sbo+KgRMzP0REJHF7j7Bq1SokJCRgwYIF2LdvH7p3745hw4YhMzOzyvN37tyJCRMmYNq0adi/fz/Gjh2LsWPH4siRIy5uubyUuGhvL6B8xhdgX7Fz+XPLAyctMz9ERFTGuVN3bLBkyRJMnz4dU6dOBQAsW7YM69atw/LlyzF37txK57/77rsYPnw4Zs+eDQB45ZVXsGnTJnzwwQdYtmyZS9teka7UiKv5Ore9v7MV6lyzwjNQnvnx9lKidUgju1+ni1XmhwXPRERk5tbgR6/XY+/evZg3b57lmFKpRHx8PJKTk6t8TnJyMhISEqyODRs2DGvXrq3yfJ1OB52uPCjJy8urf8OrcPRKHsZ9tNMpr+1JXJH5kYKfjhF+8KrHcFXFzI+Gu7oTEVEZtwY/165dg9FoRFhYmNXxsLAwnDhxosrnpKenV3l+enp6lecvXrwYCxcudEyDa6CAOVNxM+vTKghNm9i+x5a9BrVviv/8eQETbomq1+uE+mkxslsEsgv0CHVBu4mIqGFw+7CXs82bN88qU5SXl4eoqPp1qlXp2SIQJ18d4fDXlaN2YU3w++zBDnmtDx/s5ZDXISKim4dbg5+QkBCoVCpkZGRYHc/IyEB4eHiVzwkPD6/T+d7e3vD25l/9REREZObWcRqNRoPY2FgkJiZajplMJiQmJiIuLq7K58TFxVmdDwCbNm2q9nwiIiKiitw+7JWQkIDJkyejd+/e6NOnD5YuXYrCwkLL7K9JkyYhMjISixcvBgDMmjULgwYNwttvv42RI0fi22+/xZ49e/Dvf//bnZdBREREDYTbg5/x48fj6tWrmD9/PtLT09GjRw9s2LDBUtScmpoKpbI8QdWvXz98/fXXeOGFF/D888+jXbt2WLt2Lbp27equSyAiIqIGRCGEEO5uhCvl5eXB398fubm58POzf/VgIiIich1H9t8399xsIiIiohsw+CEiIiJZYfBDREREssLgh4iIiGSFwQ8RERHJCoMfIiIikhUGP0RERCQrDH6IiIhIVhj8EBERkay4fXsLV5MWtM7Ly3NzS4iIiMhWUr/tiI0pZBf85OfnAwCioqLc3BIiIiKqq/z8fPj7+9frNWS3t5fJZMKVK1fQpEkTKBQKh752Xl4eoqKicPHixZt23zA5XCMgj+uUwzUC8rhOOVwjII/rlMM1AvZdpxAC+fn5aNasmdWG5/aQXeZHqVSiefPmTn0PPz+/m/qHFpDHNQLyuE45XCMgj+uUwzUC8rhOOVwjUPfrrG/GR8KCZyIiIpIVBj9EREQkKwx+HMjb2xsLFiyAt7e3u5viNHK4RkAe1ymHawTkcZ1yuEZAHtcph2sE3H+dsit4JiIiInlj5oeIiIhkhcEPERERyQqDHyIiIpIVBj9EREQkKwx+HOTDDz9EdHQ0tFot+vbti927d7u7SdVavHgxbrnlFjRp0gShoaEYO3YsTp48aXXO7bffDoVCYfX12GOPWZ2TmpqKkSNHwtfXF6GhoZg9ezZKS0utzklKSkKvXr3g7e2Ntm3bYuXKlc6+PADASy+9VKn9HTt2tDxeUlKCGTNmIDg4GI0bN8a9996LjIwMq9fw5OuTREdHV7pOhUKBGTNmAGi4n+O2bdswatQoNGvWDAqFAmvXrrV6XAiB+fPnIyIiAj4+PoiPj8fp06etzsnOzsbEiRPh5+eHgIAATJs2DQUFBVbnHDp0CAMGDIBWq0VUVBTeeOONSm35/vvv0bFjR2i1WsTExGD9+vVOv0aDwYA5c+YgJiYGjRo1QrNmzTBp0iRcuXLF6jWq+vxfe+01j7nG2q4TAKZMmVLpGoYPH251TkP+LAFU+TuqUCjw5ptvWs7x9M/Sln7DlffVeve5gurt22+/FRqNRixfvlwcPXpUTJ8+XQQEBIiMjAx3N61Kw4YNEytWrBBHjhwRBw4cEHfddZdo0aKFKCgosJwzaNAgMX36dJGWlmb5ys3NtTxeWloqunbtKuLj48X+/fvF+vXrRUhIiJg3b57lnHPnzglfX1+RkJAgjh07Jt5//32hUqnEhg0bnH6NCxYsEF26dLFq/9WrVy2PP/bYYyIqKkokJiaKPXv2iFtvvVX069evwVyfJDMz0+oaN23aJACIrVu3CiEa7ue4fv168c9//lOsXr1aABBr1qyxevy1114T/v7+Yu3ateLgwYNi9OjRolWrVqK4uNhyzvDhw0X37t3Fn3/+KbZv3y7atm0rJkyYYHk8NzdXhIWFiYkTJ4ojR46Ib775Rvj4+IhPPvnEcs6OHTuESqUSb7zxhjh27Jh44YUXhFqtFocPH3bqNebk5Ij4+HixatUqceLECZGcnCz69OkjYmNjrV6jZcuW4uWXX7b6fCv+Hrv7Gmu7TiGEmDx5shg+fLjVNWRnZ1ud05A/SyGE1bWlpaWJ5cuXC4VCIc6ePWs5x9M/S1v6DVfdVx3R5zL4cYA+ffqIGTNmWL43Go2iWbNmYvHixW5sle0yMzMFAPH7779bjg0aNEjMmjWr2uesX79eKJVKkZ6ebjn28ccfCz8/P6HT6YQQQjz33HOiS5cuVs8bP368GDZsmGMvoAoLFiwQ3bt3r/KxnJwcoVarxffff285dvz4cQFAJCcnCyE8//qqM2vWLNGmTRthMpmEEA3/cxRCVOpMTCaTCA8PF2+++ablWE5OjvD29hbffPONEEKIY8eOCQDir7/+spzz66+/CoVCIS5fviyEEOKjjz4SgYGBlusUQog5c+aIDh06WL6///77xciRI63a07dvX/H3v//dqddYld27dwsA4sKFC5ZjLVu2FO+88061z/GkaxSi6uucPHmyGDNmTLXPuRk/yzFjxoghQ4ZYHWton+WN/YYr76uO6HM57FVPer0ee/fuRXx8vOWYUqlEfHw8kpOT3dgy2+Xm5gIAgoKCrI7/97//RUhICLp27Yp58+ahqKjI8lhycjJiYmIQFhZmOTZs2DDk5eXh6NGjlnMq/rtI57jq3+X06dNo1qwZWrdujYkTJyI1NRUAsHfvXhgMBqu2dezYES1atLC0rSFc3430ej2++uorPPzww1ab9jb0z/FGKSkpSE9Pt2qTv78/+vbta/X5BQQEoHfv3pZz4uPjoVQqsWvXLss5AwcOhEajsZwzbNgwnDx5EtevX7ec4ynXnpubC4VCgYCAAKvjr732GoKDg9GzZ0+8+eabVkMIDeUak5KSEBoaig4dOuDxxx9HVlaW5bGb7bPMyMjAunXrMG3atEqPNaTP8sZ+w1X3VUf1ubLb2NTRrl27BqPRaPVhAkBYWBhOnDjhplbZzmQy4amnnkL//v3RtWtXy/EHH3wQLVu2RLNmzXDo0CHMmTMHJ0+exOrVqwEA6enpVV6z9FhN5+Tl5aG4uBg+Pj5Ou66+ffti5cqV6NChA9LS0rBw4UIMGDAAR44cQXp6OjQaTaVOJCwsrNa2S4/VdI4rrq8qa9euRU5ODqZMmWI51tA/x6pI7aqqTRXbHBoaavW4l5cXgoKCrM5p1apVpdeQHgsMDKz22qXXcJWSkhLMmTMHEyZMsNoE8sknn0SvXr0QFBSEnTt3Yt68eUhLS8OSJUss1+Hp1zh8+HCMGzcOrVq1wtmzZ/H8889jxIgRSE5Ohkqluuk+yy+++AJNmjTBuHHjrI43pM+yqn7DVffV69evO6TPZfAjczNmzMCRI0fwxx9/WB1/9NFHLf8fExODiIgIDB06FGfPnkWbNm1c3cw6GzFihOX/u3Xrhr59+6Jly5b47rvvXN5Zu8rnn3+OESNGoFmzZpZjDf1zJHPx8/333w8hBD7++GOrxxISEiz/361bN2g0Gvz973/H4sWLG8z2CA888IDl/2NiYtCtWze0adMGSUlJGDp0qBtb5hzLly/HxIkTodVqrY43pM+yun6jIeGwVz2FhIRApVJVqmjPyMhAeHi4m1plm5kzZ+KXX37B1q1b0bx58xrP7du3LwDgzJkzAIDw8PAqr1l6rKZz/Pz8XB6ABAQEoH379jhz5gzCw8Oh1+uRk5NTqW21tV16rKZz3HF9Fy5cwObNm/HII4/UeF5D/xwrtqum37nw8HBkZmZaPV5aWors7GyHfMau+t2WAp8LFy5g06ZNVlmfqvTt2xelpaU4f/48gIZxjTdq3bo1QkJCrH5Gb4bPEgC2b9+OkydP1vp7CnjuZ1ldv+Gq+6qj+lwGP/Wk0WgQGxuLxMREyzGTyYTExETExcW5sWXVE0Jg5syZWLNmDbZs2VIplVqVAwcOAAAiIiIAAHFxcTh8+LDVTUm6OXfu3NlyTsV/F+kcd/y7FBQU4OzZs4iIiEBsbCzUarVV206ePInU1FRL2xra9a1YsQKhoaEYOXJkjec19M8RAFq1aoXw8HCrNuXl5WHXrl1Wn19OTg727t1rOWfLli0wmUyWADAuLg7btm2DwWCwnLNp0yZ06NABgYGBlnPcde1S4HP69Gls3rwZwcHBtT7nwIEDUCqVlmEiT7/Gqly6dAlZWVlWP6MN/bOUfP7554iNjUX37t1rPdfTPsva+g1X3Vcd1ufaXBpN1fr222+Ft7e3WLlypTh27Jh49NFHRUBAgFVFuyd5/PHHhb+/v0hKSrKaVllUVCSEEOLMmTPi5ZdfFnv27BEpKSnixx9/FK1btxYDBw60vIY0ZfHOO+8UBw4cEBs2bBBNmzatcsri7NmzxfHjx8WHH37osqngzzzzjEhKShIpKSlix44dIj4+XoSEhIjMzEwhhHlKZosWLcSWLVvEnj17RFxcnIiLi2sw11eR0WgULVq0EHPmzLE63pA/x/z8fLF//36xf/9+AUAsWbJE7N+/3zLT6bXXXhMBAQHixx9/FIcOHRJjxoypcqp7z549xa5du8Qff/wh2rVrZzU9OicnR4SFhYmHHnpIHDlyRHz77bfC19e30tRhLy8v8dZbb4njx4+LBQsWOGzqcE3XqNfrxejRo0Xz5s3FgQMHrH5PpVkxO3fuFO+88444cOCAOHv2rPjqq69E06ZNxaRJkzzmGmu7zvz8fPHss8+K5ORkkZKSIjZv3ix69eol2rVrJ0pKSiyv0ZA/S0lubq7w9fUVH3/8caXnN4TPsrZ+QwjX3Vcd0ecy+HGQ999/X7Ro0UJoNBrRp08f8eeff7q7SdUCUOXXihUrhBBCpKamioEDB4qgoCDh7e0t2rZtK2bPnm21PowQQpw/f16MGDFC+Pj4iJCQEPHMM88Ig8Fgdc7WrVtFjx49hEajEa1bt7a8h7ONHz9eRERECI1GIyIjI8X48ePFmTNnLI8XFxeLJ554QgQGBgpfX19xzz33iLS0NKvX8OTrq2jjxo0CgDh58qTV8Yb8OW7durXKn9HJkycLIczT3V988UURFhYmvL29xdChQytdf1ZWlpgwYYJo3Lix8PPzE1OnThX5+flW5xw8eFDcdtttwtvbW0RGRorXXnutUlu+++470b59e6HRaESXLl3EunXrnH6NKSkp1f6eSms47d27V/Tt21f4+/sLrVYrOnXqJBYtWmQVNLj7Gmu7zqKiInHnnXeKpk2bCrVaLVq2bCmmT59eqRNryJ+l5JNPPhE+Pj4iJyen0vMbwmdZW78hhGvvq/XtcxVlF0VEREQkC6z5ISIiIllh8ENERESywuCHiIiIZIXBDxEREckKgx8iIiKSFQY/REREJCsMfoiIiEhWGPwQERGRrDD4ISIiIllh8ENEHm/KlCkYO3asu5tBRDcJBj9EREQkKwx+iMhj/PDDD4iJiYGPjw+Cg4MRHx+P2bNn44svvsCPP/4IhUIBhUKBpKQkAMDFixdx//33IyAgAEFBQRgzZgzOnz9veT0pY7Rw4UI0bdoUfn5+eOyxx6DX62t8z8LCQhdfORG5kpe7G0BEBABpaWmYMGEC3njjDdxzzz3Iz8/H9u3bMWnSJKSmpiIvLw8rVqwAAAQFBcFgMGDYsGGIi4vD9u3b4eXlhVdffRXDhw/HoUOHoNFoAACJiYnQarVISkrC+fPnMXXqVAQHB+Nf//pXte/J/Z6Jbm4MfojII6SlpaG0tBTjxo1Dy5YtAQAxMTEAAB8fH+h0OoSHh1vO/+qrr2AymfDZZ59BoVAAAFasWIGAgAAkJSXhzjvvBABoNBosX74cvr6+6NKlC15++WXMnj0br7zySo3vSUQ3Lw57EZFH6N69O4YOHYqYmBj83//9Hz799FNcv3692vMPHjyIM2fOoEmTJmjcuDEaN26MoKAglJSU4OzZs1av6+vra/k+Li4OBQUFuHjxYp3fk4huDgx+iMgjqFQqbNq0Cb/++is6d+6M999/Hx06dEBKSkqV5xcUFCA2NhYHDhyw+jp16hQefPBBp7wnEd0cGPwQkcdQKBTo378/Fi5ciP3790Oj0WDNmjXQaDQwGo1W5/bq1QunT59GaGgo2rZta/Xl7+9vOe/gwYMoLi62fP/nn3+icePGiIqKqvE9iejmxeCHiDzCrl27sGjRIuzZswepqalYvXo1rl69ik6dOiE6OhqHDh3CyZMnce3aNRgMBkycOBEhISEYM2YMtm/fjpSUFCQlJeHJJ5/EpUuXLK+r1+sxbdo0HDt2DOvXr8eCBQswc+ZMKJXKGt+TiG5eLHgmIo/g5+eHbdu2YenSpcjLy0PLli3x9ttvY8SIEejduzeSkpLQu3dvFBQUYOvWrbj99tuxbds2zJkzB+PGjUN+fj4iIyMxdOhQ+Pn5WV536NChaNeuHQYOHAidTocJEybgpZdeqvU9iejmpRCc00lEN6kpU6YgJycHa9eudXdTiMiDcNiLiIiIZIXBDxEREckKh72IiIhIVpj5ISIiIllh8ENERESywuCHiIiIZIXBDxEREckKgx8iIiKSFQY/REREJCsMfoiIiEhWGPwQERGRrPw/uSXOjtxi+fIAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "number_of_steps = 20000\n",
        "record_period = 400\n",
        "\n",
        "\"\"\"\n",
        "This test is checking if DQN will eventually solve FrozenLake-v1\n",
        "whose observations need to be wrapped in a one-hot representation.\n",
        "\"\"\"\n",
        "env = OneHotObservationsFromDiscrete(\n",
        "    GymEnvironment(\n",
        "        \"FrozenLake-v1\", is_slippery=False, map_name=\"4x4\",\n",
        "    )\n",
        ")\n",
        "\n",
        "action_representation_module = OneHotActionTensorRepresentationModule(\n",
        "    max_number_actions= env.action_space.n,\n",
        ")\n",
        "\n",
        "assert isinstance(env.action_space, DiscreteSpace)\n",
        "state_dim = env.observation_space.n\n",
        "agent = PearlAgent(\n",
        "    policy_learner=DeepQLearning(\n",
        "        state_dim=state_dim,\n",
        "        action_space=env.action_space,\n",
        "        hidden_dims=[64, 64],\n",
        "        training_rounds=1,\n",
        "        action_representation_module=action_representation_module\n",
        "    ),\n",
        "    replay_buffer=BasicReplayBuffer(1000),\n",
        ")\n",
        "\n",
        "info = online_learning(\n",
        "    agent=agent,\n",
        "    env=env,\n",
        "    number_of_steps=number_of_steps,\n",
        "    print_every_x_steps=100,\n",
        "    record_period=record_period,\n",
        "    learn_after_episode=False,\n",
        ")\n",
        "torch.save(info[\"return\"], \"DQN-return.pt\")\n",
        "plt.plot(record_period * np.arange(len(info[\"return\"])), info[\"return\"], label=\"DQN\")\n",
        "plt.xlabel(\"steps\")\n",
        "plt.ylabel(\"return\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UO6AKi7b2A7d"
      },
      "source": []
    }
  ],
  "metadata": {
    "custom": {
      "cells": [],
      "metadata": {
        "custom": {
          "cells": [],
          "metadata": {
            "custom": {
              "cells": [],
              "metadata": {
                "custom": {
                  "cells": [],
                  "metadata": {
                    "custom": {
                      "cells": [],
                      "metadata": {
                        "accelerator": "GPU",
                        "colab": {
                          "gpuType": "T4",
                          "include_colab_link": true,
                          "provenance": []
                        },
                        "fileHeader": "",
                        "fileUid": "4316417e-7688-45f2-a94f-24148bfc425e",
                        "isAdHoc": false,
                        "kernelspec": {
                          "display_name": "pearl (local)",
                          "language": "python",
                          "name": "pearl_local"
                        },
                        "language_info": {
                          "name": "python"
                        }
                      },
                      "nbformat": 4,
                      "nbformat_minor": 2
                    },
                    "fileHeader": "",
                    "fileUid": "1158a851-91bb-437e-a391-aba92448f600",
                    "indentAmount": 2,
                    "isAdHoc": false,
                    "language_info": {
                      "name": "plaintext"
                    }
                  },
                  "nbformat": 4,
                  "nbformat_minor": 2
                },
                "fileHeader": "",
                "fileUid": "ddf9fa29-09d7-404d-bc1b-62a580952524",
                "indentAmount": 2,
                "isAdHoc": false,
                "language_info": {
                  "name": "plaintext"
                }
              },
              "nbformat": 4,
              "nbformat_minor": 2
            },
            "fileHeader": "",
            "fileUid": "e751f6fa-be9e-4f88-9fef-36812551b013",
            "indentAmount": 2,
            "isAdHoc": false,
            "kernelspec": {
              "display_name": "pearl2",
              "language": "python",
              "name": "python3"
            },
            "language_info": {
              "codemirror_mode": {
                "name": "ipython",
                "version": 3
              },
              "file_extension": ".py",
              "mimetype": "text/x-python",
              "name": "python",
              "nbconvert_exporter": "python",
              "pygments_lexer": "ipython3",
              "version": "3.8.18"
            }
          },
          "nbformat": 4,
          "nbformat_minor": 2
        },
        "fileHeader": "",
        "fileUid": "a6d73a55-c990-483c-9ad3-c926ad0c935a",
        "indentAmount": 2,
        "isAdHoc": false,
        "language_info": {
          "name": "plaintext"
        }
      },
      "nbformat": 4,
      "nbformat_minor": 2
    },
    "indentAmount": 2,
    "colab": {
      "provenance": []
    },
    "language_info": {
      "name": "python"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}