{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S0lz4fO49hX-"
   },
   "source": [
    "# HW1 Solutions: Introduction to RL \n",
    "\n",
    "\n",
    "This notebook is designed for is designed to provide hands-on experience with RL modeling, algorithm implementation, and performance evaluation. Students will explore RL concepts through predefined environments and custom-designed settings.\n",
    "\n",
    "Follow the instructions in each section to complete the homework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_MVMgXuu9hX_"
   },
   "source": [
    "## Setup Instructions\n",
    "Seting up RL dependecies for first time may be challenging. Some  torch or gymnasium (Sklearn lib in SL world!) environments need additional set up on your system. If you encountered error and failure after hours of search and try feel free to be in contact with TA's.  Run the following commands to install dependencies before starting the notebook:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "PwJlcM67zK6e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)\n",
      "E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?\n"
     ]
    }
   ],
   "source": [
    "!apt-get install x11-utils > /dev/null 2>&1\n",
    "!pip install pyglet > /dev/null 2>&1\n",
    "!apt-get install -y xvfb python-opengl > /dev/null 2>&1\n",
    "!apt-get install xvfb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Vuzy-9Ky-adf"
   },
   "outputs": [],
   "source": [
    "!pip install pyvirtualdisplay > /dev/null 2>&1\n",
    "!pip install swig\n",
    "!pip install stable-baselines3 \"gymnasium[all]\" pygame matplotlib numpy pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L-wa1iuOt4am"
   },
   "source": [
    "But for saving game as **video** he defined a function (it's okay if you don't understand just try to run the code and see the output, then try to modify envs!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "bhc87xchyeHl"
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "from gymnasium.wrappers import RecordEpisodeStatistics, RecordVideo\n",
    "\n",
    "training_period = 250  # record the agent's episode every 250\n",
    "num_training_episodes = 1000  # total number of training episodes\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\", render_mode=\"rgb_array\")\n",
    "env = RecordVideo(env, video_folder=\"MountainCar-v0-agent\", name_prefix=\"training\",\n",
    "                  episode_trigger=lambda x: x % training_period == 0)\n",
    "env = RecordEpisodeStatistics(env)\n",
    "\n",
    "for episode_num in range(num_training_episodes):\n",
    "    obs, info = env.reset()\n",
    "\n",
    "    episode_over = False\n",
    "    while not episode_over:\n",
    "        action = env.action_space.sample()  # replace with actual agent\n",
    "        obs, reward, terminated, truncated, info = env.step(action)\n",
    "\n",
    "        episode_over = terminated or truncated\n",
    "\n",
    "    logging.info(f\"episode-{episode_num}\", info[\"episode\"])\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ZKGl8Ql35k_"
   },
   "source": [
    "The videos are in MountainCar-v0-agent folder of your colab folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n5URzrgeBjA2"
   },
   "source": [
    "**Loading saved model**\n",
    "\n",
    "After training model using PPO and saving it, Hamid started to load the model with the name he saved in cell above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qHRHFr9-6cgt"
   },
   "outputs": [],
   "source": [
    "model = PPO.load(\"ppo_MountainCar\")\n",
    "\n",
    "obs = vec_env.reset()\n",
    "while True:\n",
    "    action, _states = model.predict(obs)\n",
    "    obs, rewards, dones, info = vec_env.step(action)\n",
    "    print(obs, rewards, dones, info)\n",
    "    if dones[0]:\n",
    "      break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n-Jjj_0Q9hX_"
   },
   "source": [
    "# **Task 1: Solving Predefined Environments (45 points)**\n",
    "1.1. Choose two environments from the list which are implemented by  other developers and communities and train RL agents using stable-baselines3. Don't forget to check workshop notebook.\n",
    "\n",
    "**Environments:**\n",
    "- [CartPole](https://gymnasium.farama.org/environments/classic_control/cart_pole/)\n",
    "- [FrozenLake](https://gymnasium.farama.org/environments/toy_text/frozen_lake/)\n",
    "- [Taxi](https://gymnasium.farama.org/environments/toy_text/taxi/)\n",
    "- Flappy Bird (Custom env which you can google it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ryQkECp8hr9v"
   },
   "source": [
    "📊 1.2. Algorithm Comparison:\n",
    "\n",
    "\n",
    " Compare RL algorithms and results (at least two algorithms e.g., PPO, DQN) based on:\n",
    "- Total reward over time\n",
    "- Hyperparameters (check the docs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kis3E_faEs6H"
   },
   "source": [
    "## 1.1 CartPole Solution: \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "ZHfbghaAEcp5",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cpu device\n",
      "Wrapping the env with a `Monitor` wrapper\n",
      "Wrapping the env in a DummyVecEnv.\n",
      "Logging to ./a2c_CartPole_tensorboard/A2C_3\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 41.4     |\n",
      "|    ep_rew_mean        | 41.4     |\n",
      "| time/                 |          |\n",
      "|    fps                | 657      |\n",
      "|    iterations         | 100      |\n",
      "|    time_elapsed       | 0        |\n",
      "|    total_timesteps    | 500      |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.583   |\n",
      "|    explained_variance | 0.248    |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 99       |\n",
      "|    policy_loss        | 1.32     |\n",
      "|    value_loss         | 4.74     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 37.3     |\n",
      "|    ep_rew_mean        | 37.3     |\n",
      "| time/                 |          |\n",
      "|    fps                | 708      |\n",
      "|    iterations         | 200      |\n",
      "|    time_elapsed       | 1        |\n",
      "|    total_timesteps    | 1000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.614   |\n",
      "|    explained_variance | -0.921   |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 199      |\n",
      "|    policy_loss        | 0.991    |\n",
      "|    value_loss         | 10.1     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 36.3     |\n",
      "|    ep_rew_mean        | 36.3     |\n",
      "| time/                 |          |\n",
      "|    fps                | 718      |\n",
      "|    iterations         | 300      |\n",
      "|    time_elapsed       | 2        |\n",
      "|    total_timesteps    | 1500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.595   |\n",
      "|    explained_variance | 0.218    |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 299      |\n",
      "|    policy_loss        | 1.39     |\n",
      "|    value_loss         | 6.2      |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 37.9     |\n",
      "|    ep_rew_mean        | 37.9     |\n",
      "| time/                 |          |\n",
      "|    fps                | 720      |\n",
      "|    iterations         | 400      |\n",
      "|    time_elapsed       | 2        |\n",
      "|    total_timesteps    | 2000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.617   |\n",
      "|    explained_variance | -0.0144  |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 399      |\n",
      "|    policy_loss        | 1.11     |\n",
      "|    value_loss         | 6.31     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 39       |\n",
      "|    ep_rew_mean        | 39       |\n",
      "| time/                 |          |\n",
      "|    fps                | 816      |\n",
      "|    iterations         | 500      |\n",
      "|    time_elapsed       | 3        |\n",
      "|    total_timesteps    | 2500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.467   |\n",
      "|    explained_variance | 0.0353   |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 499      |\n",
      "|    policy_loss        | 0.594    |\n",
      "|    value_loss         | 5.39     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 41.6     |\n",
      "|    ep_rew_mean        | 41.6     |\n",
      "| time/                 |          |\n",
      "|    fps                | 891      |\n",
      "|    iterations         | 600      |\n",
      "|    time_elapsed       | 3        |\n",
      "|    total_timesteps    | 3000     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.557   |\n",
      "|    explained_variance | 0.0422   |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 599      |\n",
      "|    policy_loss        | 1.25     |\n",
      "|    value_loss         | 4.68     |\n",
      "------------------------------------\n",
      "------------------------------------\n",
      "| rollout/              |          |\n",
      "|    ep_len_mean        | 46.1     |\n",
      "|    ep_rew_mean        | 46.1     |\n",
      "| time/                 |          |\n",
      "|    fps                | 957      |\n",
      "|    iterations         | 700      |\n",
      "|    time_elapsed       | 3        |\n",
      "|    total_timesteps    | 3500     |\n",
      "| train/                |          |\n",
      "|    entropy_loss       | -0.538   |\n",
      "|    explained_variance | 0.00111  |\n",
      "|    learning_rate      | 0.0007   |\n",
      "|    n_updates          | 699      |\n",
      "|    policy_loss        | 1.32     |\n",
      "|    value_loss         | 4.25     |\n",
      "------------------------------------\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 11\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[38;5;66;03m# Train env 1\u001b[39;00m\n\u001b[1;32m     10\u001b[0m model \u001b[38;5;241m=\u001b[39m A2C(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMlpPolicy\u001b[39m\u001b[38;5;124m\"\u001b[39m, env, verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m, tensorboard_log\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m./a2c_CartPole_tensorboard/\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m---> 11\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlearn\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtotal_timesteps\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m1000_000\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m     12\u001b[0m model\u001b[38;5;241m.\u001b[39msave(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124ma2c_cartpole\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/a2c/a2c.py:201\u001b[0m, in \u001b[0;36mA2C.learn\u001b[0;34m(self, total_timesteps, callback, log_interval, tb_log_name, reset_num_timesteps, progress_bar)\u001b[0m\n\u001b[1;32m    192\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mlearn\u001b[39m(\n\u001b[1;32m    193\u001b[0m     \u001b[38;5;28mself\u001b[39m: SelfA2C,\n\u001b[1;32m    194\u001b[0m     total_timesteps: \u001b[38;5;28mint\u001b[39m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    199\u001b[0m     progress_bar: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m    200\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m SelfA2C:\n\u001b[0;32m--> 201\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlearn\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    202\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtotal_timesteps\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtotal_timesteps\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    203\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcallback\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcallback\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    204\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlog_interval\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlog_interval\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    205\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtb_log_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtb_log_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    206\u001b[0m \u001b[43m        \u001b[49m\u001b[43mreset_num_timesteps\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreset_num_timesteps\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    207\u001b[0m \u001b[43m        \u001b[49m\u001b[43mprogress_bar\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mprogress_bar\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    208\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/common/on_policy_algorithm.py:336\u001b[0m, in \u001b[0;36mOnPolicyAlgorithm.learn\u001b[0;34m(self, total_timesteps, callback, log_interval, tb_log_name, reset_num_timesteps, progress_bar)\u001b[0m\n\u001b[1;32m    333\u001b[0m         \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mep_info_buffer \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    334\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_dump_logs(iteration)\n\u001b[0;32m--> 336\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    338\u001b[0m callback\u001b[38;5;241m.\u001b[39mon_training_end()\n\u001b[1;32m    340\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/a2c/a2c.py:150\u001b[0m, in \u001b[0;36mA2C.train\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    146\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maction_space, spaces\u001b[38;5;241m.\u001b[39mDiscrete):\n\u001b[1;32m    147\u001b[0m     \u001b[38;5;66;03m# Convert discrete action from float to long\u001b[39;00m\n\u001b[1;32m    148\u001b[0m     actions \u001b[38;5;241m=\u001b[39m actions\u001b[38;5;241m.\u001b[39mlong()\u001b[38;5;241m.\u001b[39mflatten()\n\u001b[0;32m--> 150\u001b[0m values, log_prob, entropy \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpolicy\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mevaluate_actions\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrollout_data\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mobservations\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mactions\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    151\u001b[0m values \u001b[38;5;241m=\u001b[39m values\u001b[38;5;241m.\u001b[39mflatten()\n\u001b[1;32m    153\u001b[0m \u001b[38;5;66;03m# Normalize advantage (not present in the original implementation)\u001b[39;00m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/common/policies.py:732\u001b[0m, in \u001b[0;36mActorCriticPolicy.evaluate_actions\u001b[0;34m(self, obs, actions)\u001b[0m\n\u001b[1;32m    730\u001b[0m features \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mextract_features(obs)\n\u001b[1;32m    731\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mshare_features_extractor:\n\u001b[0;32m--> 732\u001b[0m     latent_pi, latent_vf \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmlp_extractor\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfeatures\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    733\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    734\u001b[0m     pi_features, vf_features \u001b[38;5;241m=\u001b[39m features\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1739\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1737\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1738\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1739\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1750\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1745\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1746\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1747\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1748\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1749\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1750\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1752\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m   1753\u001b[0m called_always_called_hooks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mset\u001b[39m()\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/common/torch_layers.py:257\u001b[0m, in \u001b[0;36mMlpExtractor.forward\u001b[0;34m(self, features)\u001b[0m\n\u001b[1;32m    252\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, features: th\u001b[38;5;241m.\u001b[39mTensor) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mtuple\u001b[39m[th\u001b[38;5;241m.\u001b[39mTensor, th\u001b[38;5;241m.\u001b[39mTensor]:\n\u001b[1;32m    253\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    254\u001b[0m \u001b[38;5;124;03m    :return: latent_policy, latent_value of the specified network.\u001b[39;00m\n\u001b[1;32m    255\u001b[0m \u001b[38;5;124;03m        If all layers are shared, then ``latent_policy == latent_value``\u001b[39;00m\n\u001b[1;32m    256\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 257\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mforward_actor\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfeatures\u001b[49m\u001b[43m)\u001b[49m, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mforward_critic(features)\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/stable_baselines3/common/torch_layers.py:260\u001b[0m, in \u001b[0;36mMlpExtractor.forward_actor\u001b[0;34m(self, features)\u001b[0m\n\u001b[1;32m    259\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mforward_actor\u001b[39m(\u001b[38;5;28mself\u001b[39m, features: th\u001b[38;5;241m.\u001b[39mTensor) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m th\u001b[38;5;241m.\u001b[39mTensor:\n\u001b[0;32m--> 260\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpolicy_net\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfeatures\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1739\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1737\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1738\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1739\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1750\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1745\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1746\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1747\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1748\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1749\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1750\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1752\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m   1753\u001b[0m called_always_called_hooks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mset\u001b[39m()\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/container.py:250\u001b[0m, in \u001b[0;36mSequential.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    248\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;28minput\u001b[39m):\n\u001b[1;32m    249\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m module \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m:\n\u001b[0;32m--> 250\u001b[0m         \u001b[38;5;28minput\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[43mmodule\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m    251\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28minput\u001b[39m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1739\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1737\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1738\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1739\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1750\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1745\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1746\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1747\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1748\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1749\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1750\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1752\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m   1753\u001b[0m called_always_called_hooks \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mset\u001b[39m()\n",
      "File \u001b[0;32m~/Self_learn/RL_Course/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py:125\u001b[0m, in \u001b[0;36mLinear.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    124\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;28minput\u001b[39m: Tensor) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tensor:\n\u001b[0;32m--> 125\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mF\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlinear\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbias\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Import CartPole env \n",
    "from stable_baselines3 import A2C , PPO , DQN\n",
    "import gymnasium as gym\n",
    "from gymnasium import spaces\n",
    "import numpy as np\n",
    "\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "# Train env With A2C\n",
    "model = A2C(\"MlpPolicy\", env, verbose=1, tensorboard_log=\"./a2c_CartPole_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"a2c_cartpole\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train env With PPO\n",
    "model = PPO(\"MlpPolicy\", env, verbose=1, tensorboard_log=\"./PPO_CartPole_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"ppo_cartpole\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow installation not found - running with reduced feature set.\n",
      "\n",
      "NOTE: Using experimental fast data loading logic. To disable, pass\n",
      "    \"--load_fast=false\" and report issues on GitHub. More details:\n",
      "    https://github.com/tensorflow/tensorboard/issues/4784\n",
      "\n",
      "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
      "TensorBoard 2.18.0 at http://localhost:6007/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "# plot env1 total rewards on sepearate lines for each algorithm\n",
    "!tensorboard --logdir ./a2c_CartPole_tensorboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modify hyperparameters for cartpole"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modify hyperparameters and plot reward curves\n",
    "learning_rates = [0.0001, 0.01]  \n",
    "gammas = [0.95, 0.99]\n",
    "\n",
    "for lr in learning_rates:\n",
    "    for gamma in gammas:\n",
    "        model = PPO(\"MlpPolicy\",\n",
    "                    env, verbose=0,\n",
    "                    learning_rate=lr, \n",
    "                    gamma=gamma ,\n",
    "                    tensorboard_log=f'./PPO_CartPole_tensorboard/ppo_lr{lr}_gamma{gamma}')\n",
    "        model.learn(total_timesteps=200_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## wrapper for reward function of Cartpole"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gymnasium import  RewardWrapper\n",
    "class DoubledReward(RewardWrapper):\n",
    "    def reward(self, reward):\n",
    "        return reward * 2\n",
    "\n",
    "wrapped_env = DoubledReward(env)\n",
    "\n",
    "model = PPO(\"MlpPolicy\", wrapped_env, verbose=0, tensorboard_log=\"./PPO_CartPole_tensorboard/reward_wrapped_env_ppo\")\n",
    "model.learn(total_timesteps=500_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KQ5mY77EE9o3"
   },
   "source": [
    "## 1.2 FrozenLake implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines3 import A2C , PPO , DQN\n",
    "import gymnasium as gym\n",
    "from gymnasium import spaces\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "env = gym.make(\"FrozenLake-v1\" , is_slippery=True)\n",
    "# Train env With DQN\n",
    "model = DQN(\"MlpPolicy\" ,env , verbose=0, tensorboard_log=\"./DQN_FrozenLake_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"DQN_FrozenLake\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train env With A2C\n",
    "model = A2C(\"MlpPolicy\" ,env , verbose=0, tensorboard_log=\"./A2C_FrozenLake_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"A2C_FrozenLake\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modify DQN hyperparameters for FrozenLake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modify hyperparameters and plot reward curves\n",
    "learning_rates = [0.0001, 0.01]  \n",
    "gammas = [0.95, 0.99]\n",
    "\n",
    "for lr in learning_rates:\n",
    "    for gamma in gammas:\n",
    "        model = DQN(\"MlpPolicy\",\n",
    "                    env, verbose=0,\n",
    "                    learning_rate=lr, \n",
    "                    gamma=gamma ,\n",
    "                    tensorboard_log=f'./DQN_FrozenLake_tensorboard/DQN_lr{lr}_gamma{gamma}')\n",
    "        model.learn(total_timesteps=300_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## wrapper for reward function of FrozenLake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<stable_baselines3.dqn.dqn.DQN at 0x7b9bf3fe9240>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gymnasium import  RewardWrapper\n",
    "class ModifiedFrozenLakeReward(RewardWrapper):\n",
    "    def reward(self, reward):\n",
    "        if reward == 0:            \n",
    "            return -0.01\n",
    "        elif reward == 1:           \n",
    "            return 2\n",
    "        else:\n",
    "            return reward\n",
    "\n",
    "wrapped_env = ModifiedFrozenLakeReward(env)\n",
    "\n",
    "model = DQN(\"MlpPolicy\", wrapped_env, verbose=0, tensorboard_log=\"./DQN_FrozenLake_tensorboard/reward_wrapped_env_DQN\")\n",
    "model.learn(total_timesteps=500_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Taxi Env implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines3 import A2C , PPO , DQN\n",
    "import gymnasium as gym\n",
    "from gymnasium import spaces\n",
    "import numpy as np\n",
    "\n",
    "env = gym.make(\"Taxi-v3\")\n",
    "# Train env With DQN\n",
    "model = DQN(\"MlpPolicy\" ,env , verbose=0, tensorboard_log=\"./DQN_Taxi_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"DQN_Taxi\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train env With PPO\n",
    "model = PPO(\"MlpPolicy\", env, verbose=0, tensorboard_log=\"./PPO_Taxi_tensorboard/\")\n",
    "model.learn(total_timesteps=500_000)\n",
    "model.save(\"ppo_Taxi\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "seems we should solve taxi using better algorithms. we use q learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Learning for Taxi "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import gymnasium as gym\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "\n",
    "\n",
    "\n",
    "def qlearning_train(taxi_env , learning_rate , discount_factor ):\n",
    "    \n",
    "    # we use the tensorboard writer\n",
    "    writer = SummaryWriter(f\"Qlearning_Taxi/taxi_experiment_lr_{learning_rate}_{discount_factor}\")\n",
    "    \n",
    "    # q table dimensions based on state and action space\n",
    "    num_states = taxi_env.observation_space.n\n",
    "    num_actions = taxi_env.action_space.n\n",
    "    q_values = np.zeros((num_states, num_actions))\n",
    "    \n",
    "    # hyperparameters for the problem\n",
    "    alpha = learning_rate          # learning rate\n",
    "    gamma = discount_factor         # discount factor\n",
    "    eps = 1.0            # initial exploration probability\n",
    "    decay = 0.005        # epsilon decay rate per episode\n",
    "    episodes = 5000\n",
    "    max_steps_per_ep = 99\n",
    "    \n",
    "    \n",
    "    episode_rewards = []\n",
    "    print(\"training...\")\n",
    "    \n",
    "    for ep in range(episodes):\n",
    "        # reseting the environment\n",
    "        state, _ = taxi_env.reset()\n",
    "        total_reward = 0\n",
    "    \n",
    "        for step in range(max_steps_per_ep):\n",
    "            \n",
    "            if random.random() < eps:\n",
    "                chosen_action = taxi_env.action_space.sample()\n",
    "            else:\n",
    "                chosen_action = np.argmax(q_values[state, :])\n",
    "            \n",
    "            next_state, reward, done, _, _ = taxi_env.step(chosen_action)\n",
    "            \n",
    "            q_values[state, chosen_action] += alpha * (\n",
    "                reward + gamma * np.max(q_values[next_state, :]) - q_values[state, chosen_action]\n",
    "            )\n",
    "    \n",
    "            state = next_state\n",
    "            total_reward += reward\n",
    "    \n",
    "            if done:\n",
    "                break\n",
    "    \n",
    "        # decay epsilon over episodes\n",
    "        eps = 1  / (1  + decay * ep)\n",
    "        episode_rewards.append(total_reward)\n",
    "        \n",
    "        avg_reward = np.mean(episode_rewards[-100:]) if ep >= 100 else np.mean(episode_rewards)\n",
    "        writer.add_scalar(\"Mean_Episode_Reward\", avg_reward, ep)\n",
    "    \n",
    "    print(f\"Training completed over {episodes} episodes.\")\n",
    "    writer.close()\n",
    "    taxi_env.close()\n",
    "    return q_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we test with different hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n"
     ]
    }
   ],
   "source": [
    "# create Taxi environment\n",
    "taxi_env = gym.make(\"Taxi-v3\", render_mode=\"ansi\")\n",
    "\n",
    "learning_rates = [0.1, 0.9]  \n",
    "gammas = [0.95, 0.99]\n",
    "\n",
    "for lr in learning_rates:\n",
    "    for gamma in gammas:\n",
    "        q_values = qlearning_train(taxi_env = taxi_env , learning_rate = lr , discount_factor = gamma )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can render the result here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run 1\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "|\u001b[43m \u001b[0m: | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "|\u001b[43m \u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "|\u001b[43m \u001b[0m| : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1m\u001b[43mY\u001b[0m\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[42mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (Pickup)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "|\u001b[42m_\u001b[0m| : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "|\u001b[42m_\u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| :\u001b[42m_\u001b[0m: : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (East)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : :\u001b[42m_\u001b[0m: : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (East)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : |\u001b[42m_\u001b[0m: : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|R: |\u001b[42m_\u001b[0m: :\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|R: | :\u001b[42m_\u001b[0m:\u001b[35mG\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (East)\n",
      "\n",
      "+---------+\n",
      "|R: | : :\u001b[35m\u001b[42mG\u001b[0m\u001b[0m|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (East)\n",
      "\n",
      "Run completed in 13 steps.\n",
      "\n",
      "Run 2\n",
      "\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: | : :G|\n",
      "| : | : : |\n",
      "| :\u001b[43m \u001b[0m: : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "\n",
      "\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: | : :G|\n",
      "| : | : : |\n",
      "|\u001b[43m \u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: | : :G|\n",
      "|\u001b[43m \u001b[0m: | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|\u001b[34;1m\u001b[43mR\u001b[0m\u001b[0m: | : :G|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "\n",
      "+---------+\n",
      "|\u001b[42mR\u001b[0m: | : :G|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (Pickup)\n",
      "\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "|\u001b[42m_\u001b[0m: | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : | : : |\n",
      "|\u001b[42m_\u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "|\u001b[42m_\u001b[0m| : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : | : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35m\u001b[42mY\u001b[0m\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "\n",
      "Run completed in 9 steps.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from time import sleep\n",
    "\n",
    "def simulate_agent(environment, q_matrix, num_runs=5, max_steps_per_run= 100):\n",
    "    for run in range(num_runs):\n",
    "        current_state, _ = environment.reset()\n",
    "        finished = False\n",
    "        print(f\"Run {run + 1}\\n\")\n",
    "        sleep(1)\n",
    "\n",
    "        for step in range(max_steps_per_run):            \n",
    "            print(environment.render())\n",
    "            sleep(0.5)  \n",
    "            \n",
    "            chosen_action = np.argmax(q_matrix[current_state, :])\n",
    "            next_state, reward, finished, _, _ = environment.step(chosen_action)\n",
    "            current_state = next_state\n",
    "\n",
    "            if finished:\n",
    "                print(f\"Run completed in {step + 1} steps.\\n\")\n",
    "                sleep(2)\n",
    "                break\n",
    "\n",
    "taxi_env = gym.make(\"Taxi-v3\", render_mode=\"ansi\")\n",
    "simulate_agent(taxi_env, q_values, num_runs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping Taxi reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training...\n",
      "Training completed over 5000 episodes.\n"
     ]
    }
   ],
   "source": [
    "from gymnasium import  RewardWrapper\n",
    "class CustomTaxiReward(RewardWrapper):\n",
    "    def reward(self, reward):\n",
    "        if reward == -10:            \n",
    "            return -40     \n",
    "        else:\n",
    "            return reward\n",
    "            \n",
    "taxi_env.reset()\n",
    "wrapped_taxi_env = CustomTaxiReward(taxi_env)\n",
    "\n",
    "qlearning_train(taxi_env = wrapped_taxi_env , learning_rate = 0.9 , discount_factor = .99 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Implementing Flappy Bird"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we used the third party repo for flappy bird game from this [link](https://github.com/markub3327/flappy-bird-gymnasium).\n",
    "\n",
    "First we install it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install flappy-bird-gymnasium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we check render in Human mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import flappy_bird_gymnasium\n",
    "import gymnasium\n",
    "env = gymnasium.make(\"FlappyBird-v0\", render_mode=\"human\", use_lidar=True)\n",
    "\n",
    "obs, _ = env.reset()\n",
    "while True:    \n",
    "    action = env.action_space.sample()\n",
    "\n",
    "    # Processing:\n",
    "    obs, reward, terminated, _, info = env.step(action)\n",
    "    \n",
    "    # Checking if the player is still alive\n",
    "    if terminated:\n",
    "        break\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we check the action and observation space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action Space Type: <class 'gymnasium.spaces.discrete.Discrete'>\n",
      "Action Space: Discrete(2)\n",
      "\n",
      "Observation Space Type: <class 'gymnasium.spaces.box.Box'>\n",
      "Observation Space: Box(0.0, 1.0, (180,), float64)\n",
      "\n",
      "Example Step Results:\n",
      "Action Taken: 1\n",
      "New State: [1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 0.99428291 0.9916614  0.98996685 0.98850642 0.98692514 0.98601476\n",
      " 0.9851696  0.98481283 0.98469388 0.98474675 0.98502427 0.98576393\n",
      " 0.98659539 0.98807182 0.98945397 0.99107061 0.99358888 0.99651883\n",
      " 0.99985682 1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         1.         1.         1.         1.\n",
      " 1.         1.         0.98991426 0.97514403 0.96365112 0.95245687\n",
      " 0.94157192 0.93100712 0.92584822 0.91578435 0.90606844 0.8967118\n",
      " 0.88772578 0.88123633 0.87496653 0.86892114 0.86310487 0.85752239]\n",
      "Reward Received: 0.1\n"
     ]
    }
   ],
   "source": [
    "env = gymnasium.make(\"FlappyBird-v0\", render_mode=\"rgb_array\", use_lidar=True)\n",
    "# Display action space\n",
    "print(f\"Action Space Type: {type(env.action_space)}\")\n",
    "print(f\"Action Space: {env.action_space}\")\n",
    "\n",
    "# Display observation space\n",
    "print(f\"\\nObservation Space Type: {type(env.observation_space)}\")\n",
    "print(f\"Observation Space: {env.observation_space}\")\n",
    "\n",
    "# Take a random action\n",
    "action = env.action_space.sample()\n",
    "env.reset()\n",
    "next_state, reward, done, truncated, info = env.step(action)\n",
    "\n",
    "# Display step results\n",
    "print(\"\\nExample Step Results:\")\n",
    "print(f\"Action Taken: {action}\")\n",
    "print(f\"New State: {next_state}\")\n",
    "print(f\"Reward Received: {reward}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train env With A2C\n",
    "model = A2C(\"MlpPolicy\" ,env , verbose=0, tensorboard_log=\"./A2C_FlappyBird_tensorboard/\")\n",
    "model.learn(total_timesteps=5_000_000)\n",
    "model.save(\"A2C_FlappyBird\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train env With PPO\n",
    "model = PPO(\"MlpPolicy\" ,env , verbose=0, tensorboard_log=\"./PPO_mlp_FlappyBird_tensorboard/\")\n",
    "model.learn(total_timesteps=5_000_000)\n",
    "model.save(\"PPO_FlappyBird\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping FlappyBird reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gymnasium import  RewardWrapper\n",
    "class ModifiedFlappyBirdReward(RewardWrapper):\n",
    "    def reward(self, reward):\n",
    "        if reward == - 0.5:  #- touch the top of the screen          \n",
    "            return -2 # we penalty touching the screen more\n",
    "        elif reward == 1:           \n",
    "            return 2\n",
    "        else:\n",
    "            return reward\n",
    "\n",
    "wrapped_env = ModifiedFlappyBirdReward(env)\n",
    "\n",
    "model = DQN(\"MlpPolicy\", wrapped_env, verbose=0, tensorboard_log=\"./DQN_FrozenLake_tensorboard/reward_wrapped_env_DQN\")\n",
    "model.learn(total_timesteps=500_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Py94B6gw9hYA"
   },
   "source": [
    "# Task 2 Solution: Creating Custom Environment (45 points)\n",
    "In this question, you were required to model **a custom 4*4 gridworld problem** as Markov Decision Processes (MDPs). \n",
    "\n",
    "![gridworld_4x4.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABMYAAAUMCAYAAADbCZ6qAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAC4jAAAuIwF4pT92AAB0YUlEQVR4nOzdZ5wV5d34/+/SO0hHFBVRQZogsaLRWGJNTMxtTdFEo8ZujHeUqBiVmFhCjLEmlhg1pNlboti9LYCiKCpFJCId6X3Z/wP/+EN3ZtlytnG936+XD5w5O3Ptcnb2nM+ZuaaopKSkJAAAAAAgMQ1qewAAAAAAUBuEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJKlRbQ8AgJoxZsyYOPXUU0stHzx4cNx66621MKJ8P/7xj2PcuHGllt98880xZMiQWhgR9dnDDz8cl112Wanlhx12WAwfPrzmB1SGww8/PGbOnFlq+UMPPRSbb755LYyIVHkuApAKYQxI1sqVK2Pp0qWxcuXKWLNmTTRu3DiaNGkSbdq0iWbNmtX28AAAAKhmwhj1yrp16+JHP/pRvP3227mPqYtnv9Q1K1eujKOPPjpmzJiR+5i6eCZFVUyZMiXefPPNmDBhQkyZMiVmzpwZn376ae7jW7ZsGd26dYutt946evXqFf3794/+/ftHixYtanDUVNSbb74Zr776aua6o48+Otq1a1ewfb388su5x6KmTZvGCSecULB9RUQ88MADMXv27FLLW7ZsGd/97ncLui/IsmLFijL/blSnjh07FvT3FwBgPWGMeuXvf/97mVGM8rn55ptr7c1NTfrkk0/iX//6Vzz99NPx3//+t0Jfu2zZspg8eXJMnjw5nnrqqYiIaNSoUQwaNCj23nvv+PrXvx7t27evjmFTBcuWLYvbbrstc13Pnj3jgAMOKNi+7rrrrhg7dmzu+oMOOii6du1akH0VFxfHddddF8uXLy+1bujQocIYNeKdd97JvBy7Jpx33nlx3HHH1cq+AYBNmzBGvTFnzpy48cYba3sY9d57770X9913X20Po1p99NFHcfPNN8fo0aOjuLi4YNtdu3ZtvP766/H666/HyJEjPw8SgwYNKtg+qJqBAwdGw4YNM//dx40bV7AwtmbNmpgwYUKZjxk3blwccsghBdnfe++9lxnFIj47SxYAoC669957Y8mSJaWWH3fccdG6detaGBGUJoxRb/zmN7+JZcuW1fYw6rXi4uK48sorCxqL6pJVq1bFLbfcEvfdd1+sWbOmWvdVXFwczz33XDz33HMxcODAOOuss2LgwIHVuk82rlWrVrHDDjvEu+++W2pd1mT+lfXOO+/EqlWrynxMIcPYG2+8kbtOGAMA6qr77rsv80Yehx9+uDBGndGgtgcA5TF69Oh49tlna3sY9d59990XEydOrO1hVIuPPvooTjzxxPjzn/9c7VHsy8aPHx8/+tGP4vnnn6/R/ZIt7wy+qVOnxsKFCwuyj/JEtkKGuLxLNlu0aBG9e/cu2H4AACA1zhijzlu6dGlcffXVtT2Mem/mzJlxyy231PYwqsX48ePjnHPOyTxN+8u22GKL+MpXvhJ9+/aNHj16RLdu3aJ169bRrFmzKCkpieXLl8fixYtjxowZMW3atBg/fny8+eabMXfu3I1ue+nSpYX4dqiiwYMHxz333FNqeUlJSbz55puxzz77VHkfZZ3Btd706dNj3rx50bFjxyrtq6SkJMaPH5+5buDAgdGokT/lAABQWV5NU+fdcMMNpaJE9+7do127dvHOO+/U0qjqn1/96lexYsWKLywbMGBAzJkzJ2bNmlVLo6q6MWPGxDnnnBMrV67MfUzTpk3jsMMOiyOPPDK23377MrfXtm3baNu2bWy55Zax2267xTHHHBMlJSXx9ttvx3/+8594+OGH620AGzJkSIwZM6a2h1HtBg0aFA0aNIh169aVWjdu3Lgqh7Hi4uJ46623yvXYN954o8rzmk2ePDkWL16cuW7nnXeu0rahUC699NI4/PDDa3sYFNDDDz9c20MAgBrhUkrqtPHjx8c///nPUssvuOCCaNq0aS2MqH564okn4uWXX/7CsoYNG8ZFF10URUVFtTSqqvvwww/j/PPPLzOKHXjggfHAAw/EhRdeuNEolqeoqCgGDBgQP/3pT+Oxxx6Ln/70p+5IWYe1adMmevXqlbmuEJc3vvfee5nzHQ4YMKBa9lfWnS/NLwYAAFUjjFFnrV27Nq688sooKSn5wvL99tsv9txzz1oaVf2zePHiuO6660otP+6443LjQX2wbNmyOPfcc3PP3mratGlcdtllMWLEiOjUqVPB9tuiRYs49thj4/77748TTzwxGjZsWLBtUzh5wWjSpElVPuMv6zLKrl27xsEHH1xqeSHCWN5lm82aNYsdd9yxytsHAICUCWPUWXfeeWdMnTr1C8tatmwZ559/fi2NqH4aOXJkLFiw4AvLunbtGj/+8Y9raUSFMXLkyPj4448z1zVr1ix+97vfxaGHHlpt+2/ZsmWcfvrpcdddd8W2225bbfuhcvLCWHFxcbz55ptV2nZW7Bo0aFDmpP+FmPA/b7wDBgwwvxgAAFSRMEad9NFHH8Xtt99eavmpp55a0LN/NnVjx47NnCPk/PPPj+bNm9fCiArjjTfeiPvvvz9zXVFRUVxxxRUxZMiQGhlL796946677oqDDjqoRvZH+QwePDj3MuHyTJyfZ/0E/l82aNCg2HbbbaNNmzblenx5TZs2LebPn5+5zmWUAABQdcIYddKIESNi9erVX1jWu3fvOOqoo2ppRPXP6tWrMy9F3XvvvQtyV77adMMNN+SuO/bYY2v8+2vWrFlcccUVMXTo0BrdL/natWsX22yzTea6qlzemDcR/voQN3DgwFLrypojbGPKGquJ9wEAoOpcg0Gd88ADD5R6I9mgQYO48MILzedUAX/6059i+vTpX1jWvHnzuOCCC2ppRIXxyiuvxPjx4zPXdenSJU477bQaHtH/8+WzharDokWL4v/+7/9izJgxMXXq1Pjkk09i2bJlsXr16mjdunVsscUWcdJJJ9VqpFuwYEE888wz8dprr8WUKVNi7ty5sXLlymjWrFm0bt06ttpqq9hxxx1jr732iv79+1fbDSAGDx5c6nLsiIiJEyd+Pp6KygpV7du3j6233joiInbaaad44YUXvrC+Kmeo5X1t06ZNo2/fvpXe7tKlS+Pll1+ON954IyZPnhyffPJJLFmyJFatWhWNGzeOli1bxuabbx5bb7117LTTTrHnnntGx44dK72/Qpg3b1689NJLMXbs2Jg2bVrMmjUrli1bFsXFxdGmTZvo0aNHnHfeebU679rMmTNj9OjRMWbMmPjwww9jwYIFsWrVqmjWrNnnsbZfv36x9957V/pmIBRecXFxjBs3Ll544YV47733Yvr06bFkyZKI+Cyyt2/fPrbZZpvYc889Y/fdd6+RY33KVq9eHWPHjo1XXnklJk2aFP/9739j8eLFnx+318/peMIJJ1Rou/XxuLdo0aJ4/vnn47XXXotJkybF3LlzY+nSpdG8efNo3759dOjQIQYOHBhDhw6N/v3719rr5AULFsTzzz8fr7/+ekydOjXmzJkTy5Yti8aNG39+Q5xBgwbFwQcfHF26dKnw9qdNmxZPP/10vPPOOzFlypRYtGhRrFixIpo2bRqdO3eO7bbbLvbcc8/Yb7/9avSKiI8//jheeeWVeOutt+Kjjz6KWbNmxfLly2PVqlXRtGnTaNOmTXTv3j122GGHGDJkSOy22261evOwuXPnxrPPPhtjx46NyZMnx/z582P58uXRqlWr2GyzzaJjx44xZMiQ2HPPPaN3796Ven22YsWKmDFjxheWrV27NvOx06dPj+XLl5d72+3bt3fzK6qNMEadMn/+/Lj++utLLT/yyCOr9CawPObNmxfHHXdcqfm4Ij4LHvfee2907dq1yvt55JFHYvjw4Znr9t5778yJ8itqypQp8ec//7nU8pNPPrkg30NtyrpL6XonnnhivbtE9PDDD4+ZM2eWWv7QQw/F5ptv/vn/z5w5M2677bZ48sknY9WqVZnbWrhwYSxcuLBUEF1vzJgxceqpp5ZaPnjw4Lj11lsr+R38P/PmzYubbropHn/88VJnfEZ8dsOEZcuWxaxZs+LVV1+NO+64I7bddts4+eSTY//996/y/r9s8ODB8Y9//KPU8rVr18b48eNj1113rfA2s8LYhmeJZc0z9sEHH8TSpUujVatWBdlfRES/fv2iSZMmFd7elClT4s4774zRo0fnPo+Ki4tj5cqVMX/+/Hj77bfj4YcfjoYNG8auu+4aJ554Yub3WBl5lzuPGTPmC/8/efLkuO222+LZZ5+N4uLizK9ZsGBBLFiwIObMmVMrYWz69Olxww03xHPPPZc5xvXP/RkzZsSLL74YN998cwwYMCBOPfXU2GWXXWp8vJu64cOHxyOPPFJq+aWXXhqHH3745/9fUlISDz30UNxxxx25c1bOnj07Zs+eHRMnTozHHnssmjRpEscee2yceOKJFf6dfuqpp+LnP/95qeXNmjWLf//739GiRYsKba88fvWrX2X+3SzPcb+8f582prz/HitWrIg///nP8c9//jPztVjEZ79LU6ZMibfffrvc+6+Px71PP/00br311njooYcyx7x06dJYunRpTJ8+Pd5444248847o3v37nH22WfH1772tSqPs7z/9nPmzImbb745Hn/88VizZk2px69duzZWrFgRs2fPjpdeeiluuummOOSQQ+Lss8+Odu3abXQcb775Ztx44425fwuXL18e06ZNi2nTpsV//vOfuO666+JHP/pRHHvssdGgQfVcGFVcXBxPPvlk/PWvf413330393ErVqz4/HsfN25c3HfffdGqVav4xje+Ed///verHF0/+eST+MY3vlFqebdu3UpNozJjxoy4+eab49///nfm36hFixbFokWLYtq0aTFmzJi4+eabY/vtt4/zzjuvwlOTvPPOO5mvNbOcccYZFdr2ySefHKecckqFvgbKy6WU1CnXXHNNqcuUOnToEKeffnq177tjx44xfPjwzE9HFi9eHMOGDct9U1Ze06dPj9/85jeZ67p06RKXXnpplbYf8dkL/SuvvLLUC5RevXrFcccdV+Xt16b1n0hmadeuXRx22GE1PKKaMWrUqPif//mf3BfIdcGTTz4ZRx55ZDz44IOZUSzPlClT4uc//3mcf/75Vb5b5JeVNQdXZc/iypovbMP99OnTp9SnwevWravUPGMzZsyI2bNnZ66r6PxiK1asiKuvvjqOPfbYePzxxyv8PCouLo6XX345Tj755PjZz34W8+bNq9DXV8a6devixhtvjOOPPz6efvrpKh9/q8u9994bRx99dIwePbpCY3zrrbfiJz/5SYwYMSL303Sqz5w5c+KUU06Jyy+/PDeKZVm9enXcddddceSRR8bEiRMrtM+9994782yzlStXxtNPP12hbZXHmjVr4j//+U/muuq8OU1ljB07Nv7nf/4nbrvtttwoVlH18bgXEfHMM8/Ed77znfj73/9eoTHPmDEjLrjggjj33HNj5cqV1TjCzzz55JNx1FFHxUMPPZQZxbIUFxfHww8/HEcffXSZvz+rV6+OX//613HSSSdVaPqDxYsXx29/+9s444wzYtmyZeX+uvIaM2ZMHHXUUXHJJZeUGcXyLF26NO6999448sgjMz+4qw7/+Mc/4uijj47HH3+8Qn+jPvjggzj11FPjyiuvjHXr1lXjCKFuEMaoM1566aXMF3Dnnntupc60qIw99tgjvvvd72auGz9+fNxyyy2V3vaaNWvioosuyjxluGHDhnH55ZdH27ZtK7399f7xj3/EW2+99YVlRUVFceGFF9b7O9i98MILuX/U999//0pdGlfX/frXv46rr766Rl7kVtZtt90Ww4YNq9KL0GeffTZ+/OMfx6efflqwcXXs2DF69OiRua4y84zlTYS/4ZkEjRs3jn79+hVkf2XFu4rMLzZjxoz4wQ9+EKNGjSrIi9tnnnkmjj/++JgwYUKVt5Vn7dq18bOf/Sxuv/32OhvE1q1bF1dccUVcd9115X5TmOVf//pXnHXWWXX6d3xTM2nSpDjhhBOqNN/g/Pnz47TTTsu9tD9LkyZNcs+OffTRRys9ljzPP/985pyIzZo1q5azdCvr8ccfj9NPPz1mzZpVsG3Wx+NeRMTdd98dF1xwQSxatKjS23jhhRfinHPOqdAlahX15z//OX7xi19U+gOt+fPnx49//ON47733Sq1bunRpnHbaafH3v/+90uN77bXX4uyzz67QB3VlWbduXfzhD3+IU089NT766KMqb2/ZsmVx1VVXxcUXX1ylvx9lKSkpiauvvjquuuqqKv19uf/+++Piiy/2AQ6bPGGMOmHFihVx1VVXlVq+66671vjd/k4//fTcyzbvvPPOUqe7l9f111+f+QIgIuKHP/xhQe4wN3fu3MyJ6b/5zW9mTgpe37z00ku56/bbb78aHEnNuPHGG6v0wrAm3HnnnVUKxhv64IMP4vzzzy/oi8S8gDRhwoQKv2DOehPdsmXLUnNFZV1yU5k34HmT9jdp0iT69+9frm18/PHHcdJJJ2XOtVYV8+fPj1NPPbVKd9wsy2WXXRbPPfdctWy7UK666qp44IEHCrKt1157LYYPH17qZikU3scffxxnnHFGzJkzp8rbWrp0aZx//vmxcOHCcn9N3pnNY8eOLWgYisiPbfvss0+0bNmyoPuqrJdffjmGDx9e0Dfd9fW4N2rUqPjd735XkOPAmDFj4uqrry7AqEp78MEH4/rrr6/yOFesWBE//elPvxBvV61aFWeddVaFgnOeN998M2666aYqb2ft2rVx0UUXxR133FHlbX3Z448/Hj//+c+rJTpdf/31MWrUqIJs68knn6yW7x/qkvp9+gibjJtuuqnUXAZNmjTJnIujujVq1ChGjBgRxx13XKkzYNatWxcXX3xx3HvvvbHZZpuVe5svvPBC3HfffZnrBg8eHCeddFKVxrzeb37zm1Jj3myzzeLMM88syPZrW14oaNq06SYR/jb0yiuvxO23315qeYcOHWLo0KExaNCg6NixY7Rr1y7Wrl0bs2bNinfeeadaLsfJ8/zzz8cf/vCHMh+z1VZbxde//vXo2bNndO7cOUpKSmLu3LkxefLkePLJJ+O///3vFx4/fvz4uPHGGws2xsGDB8f9999favnq1atjwoQJFQrSWWdwDRw4sNQ8JjvttFOpx02cODFWrFhRoTnw8s4Y69u3b7km712yZEmcccYZMXfu3NzHNGrUKHbdddfYa6+9omvXrtGhQ4dYtGhRzJ07N1577bV49tlnY8WKFZlfu3Llyjj77LPjz3/+c2y11Vbl+6bK4f7774/HH3+81PJu3brFXnvtFQMGDIj27dtH27ZtY9WqVfHJJ5/E+PHjY/To0QUbw8b87W9/i3/9619lPqZ3796x//77x1ZbbRWdOnWKNWvWxJw5c2LixInx73//u1SYeeqpp9xptJqtWrUqzjvvvFJnfrZq1Sp23XXX2HXXXaNz587Rvn37WL16dcybNy/Gjh0bzzzzTO5ldJ9++mmMHDkyd+7QLxswYED06NGj1FyQJSUl8dhjj8UPf/jDSn1vX7Zw4cJ4+eWXM9fVlWkH5s+fHyNHjix1Vmjz5s1jl112id133z26dOkS7du3j6KioliwYEFMmjQpnn322dxt1tfj3pgxY+Laa68ttbxHjx6x5557Rv/+/aN9+/bRqlWrWLRoUcyYMSNeeumlePHFF3PPqn344Yfj0EMPrfAcUWX54IMPMj/I7tWrV3z1q1+N7bffPjp06BBNmjSJBQsWfP66JC9Szp49O2666ab43//934j47HXsl696aNq0aey8886x5557Rvfu3T///ZwzZ0688sorMXr06Nwz1+677744/PDDo2fPnpX+nocPHx5PPfVU7vqioqLo27dv7LzzztGnT59o165dtGrVKpYuXRrz5s2LcePGxYsvvpgb45977rkYOXJknH/++ZUe45c98sgjcffdd5davv3228eee+4ZO+ywQ7Rv3z5atGgRn376aXz00Uef3zwhL3jecccdccABB3x+syHY1Ahj1LqJEydmfqJxwgknxJZbblkLI4ro3r17DBs2LC666KJS6+bOnRvDhw+PkSNHlutuLXPmzMl9wdy2bdu4/PLLC3IXoWeffTaeeeaZUsvPPvvsglyiWdtmzZqVe2lBnz59KjUReV32u9/97gv/37Zt2/jJT34S3/rWtzInlO3Xr1/sv//+cfrppxf0csQ8CxYsiMsvvzz3BVSXLl1i2LBhsccee2Su33///ePUU0+N559/PkaMGPGFN50VDc9lKSt8jRs3rkJhLOusr6yzwwYMGBANGzb8wpuV4uLiGD9+fOy2227l2tfcuXNz5z0q75h/+ctfljl30v777x8/+9nPokOHDpnrDz/88FixYkXcdNNNcd9992X+Wy9btiz+93//N+6+++5o3Lhxuca1MSNHjvzC/3ft2jXOOeec3Eu/BgwYEAcddFCcffbZuW9mC+nDDz8s9fu5oZ49e8Yll1ySeUltRMTXv/71OOuss+KRRx6Ja6+99gsfZtxwww3VdpdWPrtb84bBpFmzZvG9730vvv/97+dG6/333z/OPvvsuO222+Kuu+7KfMwjjzwSxx9/fGy33XblGschhxwSN998c6nlhQxjTz75ZOZZKJ06daozN3y44447vvD8b9SoURx99NFx8skn506hMXTo0DjxxBNzz66rr8e9iy+++AuXfG6zzTZx1llnxV577ZX7Nd/+9rdj6tSp8ctf/jL3Es+RI0fGX/7yl4KMMeKzmzlseFZ3z54947zzzsv92zZ06NA4+eST429/+1v89re/zYx4//rXv+IHP/hBTJgwIR588MHPlxcVFcWhhx4aZ5xxRu5E9QceeGD85Cc/iSuvvLLUHaEjPjvb6w9/+ENmdCyPu+66K5544onc9QcffHCceOKJZYa3gw46KNauXRv/+Mc/4rbbbst8LfvXv/41dtlll9h7770rNc4NLV68uNR8xv37949zzjkn90Pk3XffPY455ph466234rLLLsu8XHT16tXxhz/8YaNnIg4ZMqTU1TWFupEHVCeXUlKriouL44orrij1h7JHjx4Vvg13oR144IFxxBFHZK576aWX4p577tnoNtafYZYXdC655JJK3bb6y5YtW5b5h2rw4MF15pPhqnr//fdz1/Xq1asGR1IzNnyz0LNnzxg1alQceeSRG73LUqNGjaJTp07VPby48cYbcwPcTjvtFKNGjcqNYhvae++9Y9SoUV+4fLm4uLhgkxx36dIlunfvnrmuIpc35k2EnxXGWrRoETvssEOp5RWZ8D/v7MiI8oWxvFAe8dmbjWHDhsVVV12V++ZwvebNm8d5550Xv//973PPUps8eXLmXXAra8Pn/uDBg+Nvf/tbueZDatasWcGCalmuueaa3Amx99tvv/jLX/6SG8XWa9CgQXzjG9+IUaNGfeH5uWzZsoLfhIL/Z8Mo1rFjx7j99tvjlFNO2eiZnM2aNYszzzwzzj333NzHPPTQQ+Uex6GHHpoZQKdNm1awOazyLqM8+OCDq+1ufRW14e96y5Yt4+abby73vLJZd9iuz8e9DZ+b++67b9xzzz1lRrH1evbsGTfeeGMMGDAgc/17770XkyZNKtg4NzzbcujQoXH33Xdv9AOfBg0axDHHHJP7IXFxcXHcfffdX4hXDRs2jMsuuyyGDx++0bs3duzYMa6++urYfffdM9e/+OKLFbrceb1JkyblXorZqlWruOaaa+Lyyy8v19lojRo1imOOOSbuuOOO3Nckv/71rwtyg6Vly5Z9YX65o446Kv70pz+V68qKAQMGxC233JI7P+sLL7xQqZ8l1Ad14y8jybr33nszg8fPf/7zOnEG0Pnnn5/7B+8Pf/jDRu9I9ac//Sn3De7RRx8dX/3qV6s8xojPzjL48pv2xo0bx4UXXliQ7dcFM2bMyF23KZ/WveWWW8Ytt9xS5dt6F9L06dNL3Qp8vZ49e8Z1111XoRtmtG3bNkaOHJn7Qqyq8kLS22+/Xe55PbKiVtOmTXPnI8y6nLIiIS4vojVq1GijL27XrVsX119/fe76s88+O771rW+VeywREbvttltcccUVuW+o77zzzoK/WB4wYEBcf/310aJFi4Jutypef/31ePXVVzPX7bzzznH55ZdX6G9X165d4/e//32NBD3+n9atW8cf//jHUvMDbszxxx8fu+66a+a6J554otzHk27duuUelwoxCf+0adNy75hX1+5GGfHZ65Ubbrgh87hZXpvKcW/fffeNX//61xU6jrRo0SJ++ctf5ka8ikTb8tpll13immuuKddl/esdfPDBubFv1KhRX4iDl112WRxyyCHl3najRo1i2LBhmeMpLi6OJ598stzbWu+qq67K/J1u1qxZ3HjjjbHPPvtUeJs9evSIW265JfNKjtmzZ2dO/VAVxxxzTFxwwQUViuEdO3aMSy65JDPer127NnOqA9gUCGPUmhkzZmRO2n3QQQfVmdP8mzVrFiNGjMj8Q7v+LpN5d+J744034o9//GPmuu233z7OPvvsgozx7bffjn/+85+lln/3u9+NbbbZpiD7qAvKmii5Js6Qqi3Dhg2rc2+a77vvvszLIRo0aBBXXHFFtGnTpsLb3GyzzeKXv/xltVxGlvcGdMWKFRuN2+tlRa2+ffvmXkaTtc933nmn3J8G54WxHXfccaN3X33mmWdKzV+03l577ZV7592N2XfffeO4447LXLdixYqCTfIb8dkck5deemmdu9Ns3iVJLVq0iCuvvLJSH+j06NHj8/l16rPLLrsshgwZUi3/3XvvvQUd60UXXRRbbLFFpb42b87OTz/9NCZPnlzu7eSdzf3vf/+7yhNx58W1Pn36xLbbblulbVeHE088sdw3FMmzKRz3unTpEr/4xS8qdUbfFltskRv+KnvTqDytW7eOSy+9tFJ3Ov/Rj3600cccdNBBlbrxVteuXXN/r1555ZUKbevll1/OvQHA8OHDY8cdd6zw+Nbr2rVrXHzxxZnr8i7frYyqvNfYaaedciPm66+/XpVhQZ0ljFFrsm4f3Lp16zIvVagNvXr1ivPOOy9z3X//+9/MSUgXLVoUv/jFLzLjQfPmzWPEiBEFOSNu7dq1ccUVV5S6FXn37t3L9eKjPinr0rqNXRZRXx144IEFnTS3ENasWZP7aeE3v/nNCp+BsaF+/fpV6BPi8irr0sOyLlncUFYYK2u7WWc+rJ/wf2MWLlyYO1FxeS6j3HCOlg01adIk91hWXieddFLu79vDDz9csBf0xx13XEEnti6EefPmxf/93/9lrjvhhBOqdFbn/vvvX5A7E7NxO++8cxxwwAGV/vrevXvnfuhU1iX/X7bffvtlht9FixbFiy++WOnxlZSU5B6j6+LZYt26dYsTTzyxytvZFI57p512WpXmhM2LSVOnTq3wXZjL8oMf/KDS04D069cv8zLY9Ro1ahTnnHNOJUcW8bWvfS1z+QcffFCh7eTdMGu33XYr16X9G7PPPvtkXnI/Y8aMgt319Kc//WmV5sA7+OCDM5dX5DgH9YkwRq14/PHHM99gnH766XUychx55JGx3377Za57/PHH45FHHvnCsssvvzxzPqKIiAsuuKBgl/7dddddMWXKlMx91LUzLaqqrEm1K3LZXn3y7W9/u7aHUMorr7ySOwdSIeYFLNTE0xvq3r177ov48sz7lTcRftb8Yuu1a9cu881zefZX1iWXG4snS5Ysyb3U7+tf/3qVb2jSqlWrOP744zPXzZo1K/cT9oooKiqq8CVPNeHpp58u9SFExGdnFh977LFV3n51PPcprRD/VnlzKlXkDWOLFi1i3333zVxXlcspx44dmzkxfaNGjSp1Fk51O+yww6o8gf2mcNxr3759HHjggVXaRr9+/TLP2C4uLs58rVgZTZs2zZ1/t7zK+tu57777VulDhkGDBmXe0Gr27Nm58/1mPTbv+XTaaadVemxflncsqkoYX2/77bev8l2O8+Zsmz17tnnG2CQJY9S4RYsWxXXXXVdqed++fetkCFjv4osvzr1zym9+85vP7+AyatSo3FuJH3TQQXH44YcXZDzTp0+P22+/vdTy/fbbL/bcc8+C7KMuKevTzkLdFeryyy+v8iU/eZPLVlS3bt2q/KKmOuS9WOzXr1/uhLIVsdVWW0WfPn2qvJ0vywtKb775Zmbs2FBWqGrYsGHuZMfrVXaesbx41rBhw43OwfP6669nnqkaEQU7G++ggw7KvdTntddeq/L2d9ppp4I8lwot77m/1157bXTy9vL4yle+Uic/GNqUNG/ePIYOHVrl7eTd8CXrrmtlybvs68UXXyz3m/gvy4tqe+yxR7Rr165S26xOhTiLbVM47u2zzz4FuZIg71LZTz75pMrbjvjsjoNVfR6V9eFwVc/GatKkSe5r9fL+DF544YXM1wVbbrll7ryilbHrrrtmTh9RiEtfqxpZIz4LwnkfKlb0WAf1gTBGjRs5cmSpu9k1bNgwLrroojpzp6QsrVq1iiuvvDLzk6jly5fHhRdeGBMmTIjf/e53mV+/xRZbFHQy/BEjRpSar6hly5Zx/vnnF2wfdcmGtwf/ssrMc1HXDRw4sFrm26qqvFP8886orIxCXKbwZXmRcdmyZRu9xCIrZvXu3XujMSTrU/G33npro/MH5V3e2bt3741ORJ/379O+ffuChdbOnTvnBrpCXAJSlQm4q1PeWSGFeu43bNgw9wwiCqNfv34F+XuRdwZS3pyjeb7yla9kvvFcs2ZN/Oc//6nwuFauXBmjR4/OXFcX71DdqVOnSs/1tqFN4bhXnjsGlkfez7Oiz808VZ0LLiJyw1Whtt+tW7fM5eW942/ehyCVmWy/LO3atcucMmDy5Mm5obe8CvV8KtSxDuqDTe/dJHXamDFjMu9md9RRR8UOO+xQCyOqmP79+8dpp50WN9xwQ6l1H3zwQZx88smZAadRo0YxYsSIaNmyZUHG8dBDD2V+onTqqadushPRl/VJalUnKq6LevfuXdtDKGXdunXx4YcfZq4r5FleVZnUNs/G5hkr6+eddQZXWZeClPWYlStXxrvvvpt7ttnSpUtzJ/AuzxxUeV/bp0+fgn7w0L9//8xgWIjLderic3/u3Lm5Z/AU8vlaHWdL1pTTTjutYHda/rJC3ZW3UK8z8v6Wl/eN93oNGjSIgw8+OO68885S6x599NH4zne+U6HtPfPMM5lvWNu2bZs7kXZtKtTv+qZw3CvUczNvaolChYxC/JvlfajUrl276Ny5c5W3n/cBUnl/Bnl3dN1uu+0qPaY8nTt3jmnTpn1h2apVq2LmzJlVisZ17VgH9YEwRo1ZtWpVjBgxotTyzp07x6mnnloLI6qcH/zgBzFmzJjMO9zkndV0xhlnFOzN04IFCzLPSuvdu3ccddRRBdlHXVRWGCvrbLL6qi7eUXTmzJm5d1WsyqT7X1YdLz579OgRHTt2zLyJwxtvvJE7f8zChQszY2B5wli3bt2iS5cupeYbHDduXG4YK+vSzvKc+ZB3V7ZCf/CQ9+89f/78WLp0aZXm/evZs2elv7a6rL9U/statWpV5tkPFVXI36Oa1rlz59xLDOuKQl1KmPdmsTLx4dBDD80MY2+//XZMnz49evToUe5tPfbYY5nLDzzwwIJNOVBIhfo7tykc9wr13KxqFNqYQowzb4yF+hnkhbfy/AyWLFmSO0dwoeYH3lDezRbmzp1b6TDWtGnTjZ5dXl7V/XyCukQYo8b88Y9/zHzx8tOf/rRgZ1LVhKKiorjsssviuOOOi/nz52/08XvssUfum+7KuPbaa0ududCgQYO48MILMy/z3FSU9Ud+8eLFNTiSmtG6devaHkIpc+fOzVy+2WabZU74W1nt2rWLdu3aFXxy10GDBmVenvTGG29ESUlJ5qWr69dtqKioqNyX+w0aNCieeOKJLywbN25c7o0K8uYga9CgwUb3WVJSknv31kKH1rx5bCI+e55U5Q1ifXruVyRalEd1vPHi/ynUcyvvLKSNzVeYZZtttokdd9wx8yyVRx99tNyTfc+bNy93rqu6eDfKiML8e2wqx71C/Q3Ne25W9dK89Qpxs6O8MRbq9zPvtXB5fj+zblyx3ve///1Kj6miKjvHYERhb0hVlZ8l1DfCGDVi8uTJcffdd5davueeexZ0bqKa0qFDh7jsssvizDPPLPM23R07dozLLrusYHNFvfzyy/Hkk0+WWn7kkUcWdELQuqisS0TLEyjL49RTTy33HcvOPPPM3DfLhVAXY3FeqKqOkNGqVauCh7Gdd945M4wtWrQopkyZknm2S1ao6tmzZ+6nvF+20047lQpj48ePj+Li4swXnHlhbPvtt9/oi92lS5fmXlZc6H+jssby6aefVukNacrP/ebNm0fDhg0L9iaWLyrE5ObV4bDDDssMY48//niceuqp5XoN8fjjj2c+b7baaqvo169fQcZZaIX4Xd9Ujnt19bn5ZdU5zrpwVmN1vq6riJUrV1b6a+vLcwnqGmGMardu3bq48sorS71wadq0aVxwwQW1NKqq22233eKwww7LnDMt4rNPxH75y1/GZpttVpD9rVy5Mq666qpSyzt06BCnn356QfZRl5U170Teae8V1alTp3LP0VbdE/4X4i53hZb3Qq1Qp+xvqDriSFlzdI0bN67cYaw8c32tl3XJ5foJ/788n9TKlSvjvffey9xOefaZd5lrROF/nmW9QSxrHOXhud9ykzwLlnwHHnhg/Pa3vy01LcAnn3wS48aNK9dl1HmXUdbVs8UiCvO7vqkc96gbli9fXttDiIhNc+5cqOvq7i0A2WT84x//iLfffrvU8pNOOim6d+9eCyMqjFmzZsVzzz1X5mMKeVfBW265JfNW0+eee25BT5uuq/LujBMRMXXq1BocSbryXqg1a9as4PuqjjjSs2fP3FCdNcH+0qVLY9KkSaWWl2d+sQ33mXV2WVZwGz9+fO7PuDxhrKwX0oX+Nypre5vinH/1/blP3dauXbsYOnRo5rpHH310o1///vvvZx6rGjRoEIccckiVx1eXOe5RSKtXr67tIURElHk1ClA9nDFGtcu67XGHDh1i9913z72TUHnkfYK/cuXK3O02b968IDGuuLg4hg0bVuan+uvWrYuLL7447rvvvoKcNZb1c+zVq1f06tWrSj/HvBeVS5Ysyd1umzZtCnLnoIooaxLdqnz/lF/eWXJVOeU/z4oVKwq+zYjPotbo0aNLLc8KVXkT4VckjBUVFcXAgQPj+eefL7W/L889mHcZZVFRUbn2WdZZjIX+Nypre3XhcphC2xSe+9Rthx56aDzzzDOllj/99NNxwQUXlBll8s4W23nnnaNr164FG2Nd5LhHIRXyLqZA/SKMUSvmz58f3/3ud6tl2++++24cc8wxmesGDx4ct956a5X3ceutt8b48eM3+rh58+bF8OHDY+TIkQU9e2y9yZMn536vVfXcc8/lnhF32GGHxfDhw6tlv3k6dOgQnTt3jjlz5pRa9/7778eyZcvq5NxEm5K8N2bVcelBdd3xaPDgwZlhbP78+fHRRx/FVltt9fmyrFC1xRZblPty2/UGDRpUKoy9+eabpSb8zzprLeKzu3SWZ06zpk2b5q4r9M+zrFu1lzWO+irve6pPz33qtqFDh2bedGTZsmXx7LPPxkEHHZT5dcXFxaXmMVyvLl9GWSiOexRSWQH6t7/9bXTr1q1GxtGlS5ca2Q/w/whjUEFjxoyJO+64o9yPf+mll+Lee+8t6J0pU/WVr3wl87KS4uLiGDNmTHz1q1+thVGlI+9W6kuWLCn4vsp6A1IVG5tnbMMwlhWqKnK2WFlf8+UJ/1evXh3vvPNO5teXd06zVq1aRaNGjTLPAi30v1FZ/z6FmlexLsn7ngr9c12xYoWJ9xPVqFGj+PrXvx6jRo0qte6xxx7LDWOvvPJK5g1omjdvXi9vblRRjnsUUlkfQrVr1y5zLlJg0+B8UaiAhQsXxsUXX5x5eVXr1q1jiy22yPy6G264IXdSbcpvzz33zF331FNP1eBI0pR3ptSnn35a0MnCFy5cWPA7Uq7Xq1evaNOmTea6Dc8QW7lyZUycOLHUYyoTxnr37p35KfSG+5swYULu5M3lDWNFRUXRsWPHzHUffvhhubZRXmXN61fRM+rqg7zvafr06QXdz7Rp0wq6PeqXvDO8Xn311Zg3b17murw5yL72ta8lMV+d4x6FVNalx4sWLarBkQA1TRiDChg+fHjurZx/8YtfxFVXXZU5z8SaNWvioosuqjN3u6mv9thjj9zT3J999lkvWqpZ165dcy8XyZr4ubIKua0va9CgQey0006Z6zY8QyxvIvyK3JFyvUaNGsWAAQNKLd8wjJU1v1hF9tmjR4/M5R988EG5t1Eeedvr0KHDJnkzkA3PJNzQ0qVLY+bMmQXbT6H/nahfdtxxx+jZs2ep5XmXSy5dujR3yoMULqNcz3GPQunSpUvu3Yb/+9//1vBogJokjEE53XvvvfHiiy9mrvv2t78d++23X/Tu3TvOPPPMzMdMnz49fv3rX1fnEDd5rVq1igMOOCBz3YoVK+Lvf/97DY8oLQ0bNoxtttkmc927775bsP0UcltZ8kLTrFmzPr/za9ZllJ06dco9K3RjsmLchvvIm19sm222yb2ENct2222XuXzixImZZ7pW1oQJEzKXb7vttgXbR13SqVOn3Ets8i6BrYyssxRJS95dJLPODHv66aczzzTt0qVLDBkypOBjq6sc9yiUBg0aRO/evTPX5X2ABWwazDFGtbv22murZbs//vGPM/9IFWqC/Q2999578fvf/z5zXc+ePeO88877/P+PPfbYeO211zIj2qOPPhq77bZbHHzwwRUew7333lvhrymPww8/PPOMh9qYYL88jjrqqHj44Ycz1919991xxBFH5F5WQdXttNNOmZcFP/300/G9732vIPuo7stid95559x148aNi8033zzz2JJ3pll5ZF2COX/+/Jg2bVpsscUW8dZbb1V4rFl22mmnuOeeezL3NW7cuIK8WZ43b17uG4Sq/Izquqy7i0Z89tzff//9q7z94uLizLsSkpZDDjkkbrzxxlJBZ9KkSTFp0qQvRKC8yygPOeSQpO6u57hHIe2yyy6Z/9bjxo2LtWvXlnknVLLlHY/WrFlTwyOBfOn81YRKWr58eVx00UWZB++mTZvGiBEjvnB5X1FRUQwfPjw3zlx11VXx8ccfV9t4N3V9+vTJnWR/2bJlcfXVV9fwiNKyyy67ZC6fMGFCzJgxo8rbnz59erWfNbP99tvn3sF03LhxuRPhV+YyyvX69++f+WL6jTfeiPfeey9WrFiR+XUV3eeQIUOiYcOGmevy7lxXUU888UTuWRi77rprQfZRF+U991944YXcf7+KeP311zMnUSctnTt3zn2ubRjCZs6cmXumaUqXUUY47lFY++yzT+byRYsWxWOPPVazg9lE5E3DkTe3KtQGYQw24qqrrsqdYPncc8/NvENNu3bt4vLLL8/8hGTZsmVx0UUXZc5fRPmcfvrpuZ/YPf3005mfHFMYu+22W25UuvPOO6u8/T/96U9V3sbGNGzYMPcT/nHjxsU777yT+WKtMhPvr9esWbPMyzPGjRtX5uUZFd1n69atY7fddstc9/jjj39+qWhlLV++PPf3q1u3bplzqW0q9ttvv8xj+sqVK+O+++6r8vZvv/32Km+DTUNe2HriiSc+v2vpo48+GiUlJaUe07dv39h6662rc3h1juMehdSrV6/o27dv5ro777zT6/dKyLvp0Zw5c2p4JJBPGIMyPProo7mfDu27777xne98J/drv/KVr8QPfvCDzHXvvvtu3HDDDQUZY4p69uwZP/zhD3PXjxw5Mh544IGaG1BCmjRpEgcddFDmugcffLBKE+e/8847NfZpbN6ZWB9//HE8+eSTpZa3bdu2yvPIZEWussLYVlttVanLgr/xjW9kLl+1alX89re/rfD2NnT77bfn3oDksMMOi6Kioiptvy7r1KlT7pvvO++8M/eugeXx1FNPmb+Gz+27776ZH0DMmzcvXn311YiI3GPlYYcdVq1jq6sc9yik448/PnP59OnTq22KmE1Z586dM5e7EzN1iTAGOcqaLL9r165x8cUXb3Qbp5xySu4niffcc0+8/PLLVRpjyn74wx9G//79M9eVlJTElVdeGTfccMPnn65TOMcee2zmmTPr1q2LYcOGxeLFiyu8zU8//TQuvvjizDMgqkNZlyg+9NBDpZYNGDCgym9+ss5Smz17drz22muZj6/o/GLr7bvvvrl3UXzmmWdi1KhRldruCy+8EHfffXfmuubNm8fRRx9dqe3WJ3lvlpYvXx7Dhg2L1atXV3ibbszClzVr1iy+9rWvZa579NFHY8KECZlnsjdu3DgOPPDA6h5eneS4RyEdcMABseOOO2au+/vf/x5//etfC77P4uLiap9jtbbk3bgpa95OqC3CGGRYs2ZNXHTRRbF8+fJS6xo2bBhXXHFF7mnBG2rUqFFceeWV0bp161LrSkpKYvjw4VU6yyBljRo1imuuuSa6deuWub6kpCTuvPPO+P73vx9vvvlmwfa7ZMmSuPnmm5P+d9t6661zz0qYOnVqnHfeebF06dJyb2/RokVxzjnn5F6yXB369OkTzZs3z1yXFTeqMr/YejvttFNmXMuLKZXdZ4MGDXLvjhvx2Q1R8ibtzjNmzJi48MILc0PzCSecUKG7Z9ZXu+66a3zlK1/JXDd27Ni4+OKLKxTHZs2aFWeeeWZ8+umnhRoim4i8Y+yzzz4bf/vb3zLXDR06NPfuqZs6xz0KqaioKC666KJo3Lhx5vprrrkmLr/88oLMkbVy5cr4xz/+Ed/+9rdj2LBhVd5eXZR3ksC4ceNyb6gFNc1tNSDD9ddfn3nnvYiIk046qUJ3IOrWrVsMGzYsfv7zn5dat2DBgrj00kvjhhtucCp+JXTo0CGuv/76+MlPfpJ7mcP7778fJ510UgwaNCiOPPLIGDp0aLRq1apC+ykpKYm33norRo8eHQ899FAsWbKkEMOv104//fR4/vnnY+HChaXWvfnmm3H00UfHsGHDYo899ihzO88//3yMGDHiC6GxYcOGsdlmm1VrfGzUqFEMGDDg88uSNqYq84ut17Zt29hmm21i6tSp5Xp8Zc8Yi/hs8uB999038y6H69ati0svvTReeeWVOO+882KzzTbL3c7KlSvjlltuiXvuuSd34ulevXrF97///UqPtb45//zz43vf+15mAHv66afjww8/jEsuuST69euXu41169bFo48+Gtdee+0XInLLli2jqKioQmG5rpgzZ05Mnjy5WvfRs2fPZO62OHjw4Nh8881LzY+1atWq3MsoU5t0/8sc9yik3r17x5lnnhnXXXdd5voHH3wwXn311Tj++OPjiCOOyP2wLcuKFSvipZdeiqeffjpeeumlzz+Iz7uJRH03ePDgaN26debr58suuyweeeSR2HvvvWPbbbeN1q1b507W3759+2jfvn11D5dECWPwJS+++GLuKdKDBw+OH/3oRxXe5v777x/f+ta34v777y+17tVXX4277rorTjjhhApvl89Oz77tttvi9NNPL/OuiG+88Ua88cYb0bhx4+jTp0/07ds3ttpqq+jatWu0adMmmjZtGiUlJbF8+fJYvnx5LFy4MKZNmxZTpkyJd999t9x3i6vIC6P6rEOHDvGLX/wifvazn2Ve/jh79uw466yzYuutt44DDzwwtt122+jcuXOUlJTEnDlzYsqUKfHkk09mniV27LHHxrvvvlvtZ+UNHjy4XGGsefPmmRPnV8agQYPKFca23HLL6NSpU5X2dckll8SkSZNy74L7+OOPx1NPPRW77757DB06NLp06RIdOnSIxYsXx5w5c+K1116LZ599NvPM2fVatmwZv/71r3M/Vd8UbbvttnHWWWfFNddck7l+6tSpccIJJ0SfPn1i//33/3yuuLVr18acOXNi4sSJ8eSTT2ZOOvyTn/wk/vKXv9TLMHbTTTfFTTfdVK37eOaZZzLPwN4UFRUVxSGHHBJ//OMfy/X4du3axdChQ6t5VHWf4x6FdNxxx8WMGTNyL8WdNWtWXHvttXHjjTdG//79Y/DgwbH11ltHmzZtok2bNrF27dpYunRpLFmyJObOnRsffPBBvP/++zF16tSkpvpo3LhxHHHEEbmXJY8dOzbGjh270e2cfPLJccoppxR6eBARwhh8wdy5c2P48OGZb/Tbtm0bV1xxRaU/rf7pT38a48ePz3xTfPPNN8eQIUPKPMOAfFtssUXcfffdcdlll8Vzzz1X5mPXrFkTb731Vrz11lsFHUPTpk3jpJNOiu9973sF3W5dts8++8Qpp5wSN998c+5jpk2bFrfeemu5tzlgwID4yU9+EmeccUYhhlim8p6R1b9//9y7oFbUoEGD4p///OdGH1eISzdbt24dN9xwQ5x88sm5Z1SuWbMmnn/++UrN89GsWbMYOXJk7rw+m7JjjjkmJk2aFA8++GDuYyZOnBgTJ04s9za/9rWvxVFHHRV/+ctfCjFENgGHHnpoucPY17/+9YIdp+ozxz0K7fzzz48mTZrkRp2Iz84Ae+2113LnDCXixBNPjCeeeCL39xJqWxrno0M5rFu3Li6++OLMS8MiPvsUMu+uKuXRrFmzGDFiRObpwWvXro1hw4bVy7ME6oo2bdrEtddeG5dcckl06NChxvbbqFGj+Pa3vx33339/nHjiicm9MTnppJPipJNOKsi2evXqFddcc000adKkINvbmL59++aerr+hQkSq9cp7GXah9rnFFlvEH//4x+jZs2dBtrdehw4d4qabbirIJab11UUXXRSHH354QbY1ZMiQ+OUvf+mSer5gyy23zJ2b58tSvRtlFsc9CqmoqCjOPvvsGD58eLRo0aK2h1NvtWnTJkaOHFmjr9GhIoQx+P/dcccdMWbMmMx1Rx11VHz1q1+t8j569eoV5557bua6GTNmxIgRI6q8j9R94xvfiH/+859x8sknlzmHSFW1b98+TjzxxHjwwQfjoosuqlI0re9OPfXUuPzyy6v0gnGvvfaK2267rUbnjmjcuHHunU03VJE5BTema9euuTeM2FBV5hf7su7du8edd94ZRx11VEHCy7777hv33HNPuX52m7KGDRvGJZdcEmeffXaVgvg3vvGNuP7666NZs2YFHB2bivIEr549e0afPn1qYDT1h+MehXbYYYfFfffdF/vuu2/Bt92wYcPYY4894oorrij4tuuSHXbYIe6777741re+VWMfgkJ5pXVqA5uUww8/PPPN4+abb17hbY0fPz73cq/tttsuzjnnnApvM893vvOdePXVVzMnh/33v/8du+66a3zzm98s2P425thjj82cDHOHHXaosTEUWqtWreKUU06JE044IZ5++ul46qmn4tVXX63y3YO6d+8eu+yyS+y3334xZMiQ5M4OK8vBBx8cQ4YMiRtvvDGeeOKJWLNmTbm+bptttokf/ehHcdBBB1XzCLMNHjw4N4hHlD+eVcROO+0UM2fOzF2/+eabR9euXQu6zxYtWsQFF1wQRx55ZNx5550xevToCv0+NGzYMHbdddc44YQTCnoGXX1XVFQU3/ve92KvvfaKP/zhD/Hcc8/lTtb9ZX379o3TTjstdtttt2oeJfXZAQccENdee22Zv6+HHHJIDY6o/nDco9C6d+8eV199dXzwwQcxatSoePrppyt9tUfz5s1j0KBBsdtuu8UBBxxQ5XlF64v27dvHsGHD4uyzz46XXnop3nzzzZg6dWrMmjUrFi5cGCtXrkxq/jXqjqKSrMmUADYxK1eujIkTJ8aECRNi8uTJMXPmzJg9e3YsXbo0Vq5cGatXr47GjRtHkyZNok2bNtGxY8fo3LlzbLPNNrHttttGv379Ch4rNlXz58+P0aNHx2uvvRZTpkyJ+fPnx8qVKz//2W611VbRp0+f2GuvvWLgwIEuH6sFS5cu/fwF6aRJk+KTTz6JxYsXf/570LJly+jWrVv07NkzBg4cGEOHDo2OHTvW9rDrvE8++SRGjx4dY8eOjalTp8aCBQti9erV0axZs2jbtm1svfXW0a9fv9h7770LdkMHoHwc9yi0tWvXxtixY+Ott96K999/Pz755JOYM2dOrFixItasWRPNmjWLFi1aRMuWLaNz586x9dZbx9Zbbx3bb7999OvXzw0coA4RxgAAAABIkjnGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJjWprx0VFRbW1awAAAADqmJKSkhrfpzPGAAAAAEiSMAYAAABAkoQxAAAAAJJUa3OMfdkDDzwQvXr1qu1hAJuwyZMnxxFHHFHmYxyLgOrkOATUBY5FQF1QnmNRTagzYaxXr17Rt2/f2h4GkDjHIqC2OQ4BdYFjEZAKl1ICAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJDWq7QFQ961bty7mz59f28OoNzp06BANGmjOAAAAUNcJY2zU/Pnzo3PnzrU9jHpjzpw50alTp9oeBgAAALARTmsBAAAAIEnCGAAAAABJEsYAAAAASJI5xqiU3z36XLTerH1tD6PWLfl0QZx96FdrexgAAABAJQhjVErrzdpH2/YdansYAAAAAJXmUkoAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASWpU2wOAili5bGnM/ujDWLZ4YaxavixWLV8eURTRuGmzaNK0WbTarH2069Ql2nToFA0beXoDAAAA+ZQD6rwZk9+P8c/+J6a/NyEWzPokoqRko19TVNQg2nfbPLr13C66bdMrtuzdN7ps1TOKiopqYMQAAABAfSCMUWdNfWtcjL7vzpgz/cMKf21JybqY/8nHMf+Tj2PCi89ERETLtu1im/6DYq9vHxubdelW6OECAAAA9YwwRp2zeuWK+M/dt8X4Z/9T0O0uW7QwJrz4TOzwld2FMQAAAEAYo25ZvXJl/PU3w+Pj99+t7aEAAAAAmzhhjDqjZN26+Md1V+RGsaIGDaJHn36x3eBdo0uPbaJ9182jSfPm0bhJ01i5fFmsWLokliyYFzOnTo5PpnwQ/33/3Vi+eGHNfhMAAABAvSGMUWeMG/1ETHtnfOa67YfsFvsd/6PYrHPXzPUtWreJFq3bRIdu3WPrvgMjImLduuKYPvGdeP/1l2Piqy/G8sWLqm3sAAAAQP0jjFEnrFy2NJ7/+18y1+39ne/G0G8dXeFtNmjQMLbuOyC27jsg9jvuh/HOy8/F608+VNWhAgAAAJsIYYw64YMxr8SKpUtKLd9x970rFcW+rFGTJjFwnwNi4D4HxNrVq6u8PQAAAKD+a1DbA4CIiA/GvpK5fJ+jvlfwfTVq0qTg2wQAAADqH2GMOuHjSe+XWta5xzbRLmdOMQAAAICqEsaodevWFceKJYtLLW/XuUstjAYAAABIhTBGrVu+ZHGUlKwrtbxho8a1MBoAAAAgFcIYta5Bg4aZy5d8Or+GRwIAAACkRBij1jVv2SqKiko/FWdO+SBWLltaCyMCAAAAUiCMUeuKGjTInE+seO3aePH+v9bCiAAAAIAUCGPUCVv1HZC5/LXHH4yXHhgV69YV1/CIAAAAgE2dMEad0GfXvXLXPff3v8SfLjw73nzmyVi5fFkNjgoAAADYlDWq7QFARMQ2/QbGFtv3iY8/mJi5fu7HH8Vjf7whnrjj5ujRu29sscOOscV2vaPbtttH85atani0AAAAwKZAGKPOOPhHp8ddl/4sVq9ckfuYdcVrY9o742PaO+M/X7ZZl26x+bbbR/ftekeP3n2j05ZbR1FRUU0MGQAAAKjHhDHqjE5bbBXfOut/41+/+1WsWbWq3F/36eyZ8ensmfHOy89FRETzVq1j6347Rf+9vhY9BwyKBg0aVteQAQAAgHrMHGPUKdsO3Dl+cNm10WHzLSq9jRVLl8TEV16Iv119Wdxw5g/j9ScfjnXFJu8HAAAAvkgYo87pvOVWcdKvfh8HnfiTaL1Zhypta+nCBfGfP98at/7v6TFj8vsFGiEAAACwKRDGqJMaNmoUg/c/OE7/3Z/if356cfTZba9o0qx5pbe3YOaMuOeKi+Kdl54t3CABAACAes0cY9RpDRo2jO0G7xLbDd4l1hUXx8wPJ8f0iRNixuT3YtaHk2Px/Hnl3tbaNavjoZt+G81atoptdxpSjaMGAAAA6gNhjHqjQcOG0b3XDtG91w6fL1u+ZFHM+nBK/Pf9d+Ojd9+KGZPej5KSdbnbKClZFw/eeG2c9KvfR5sOHWti2AAAAEAdJYxRr7Vo3TZ6DhgcPQcMjoiIJQvmx1svPB2vPfZArFi6JPNrVi5bGi89+Lc4+Ic/qcmhAgAAAHWMOcbYpLRu3yH2/OZRcdpvb4sdd98793FvPf9ULFu8qAZHBgAAANQ1whibpGYtWsYRZ/wsBnx1/8z1xWvWxLQJb9bsoAAAAIA6RRhjk/b1E06NVpu1z1z30btv1/BoAAAAgLpEGGOT1rhJ0xi4zwGZ6xbOmVXDowEAAADqEmGMTV6P3v0zly9fsriGRwIAAADUJcIYm7zWOZdSrlm1soZHAgAAANQlwhibvHXFazOXN27atIZHAgAAANQlwhibvMUL5mcub92+Yw2PBAAAAKhLhDE2eVPHj81c3ql7jxoeCQAAAFCXCGPUCVPfGlct2122eFG88/Jzmeu26jugWvYJAAAA1A+NansAEBHxz5G/ik5b9Ii9jjwuth24c0G2ua64OB677fexYumSUutatGkX2/TbqSD7AQAAAOonYYw645MpH8So3wyPbj23i532PTD67LZXNGvRslLbWrrw03j01t/FlJzLKHc95Iho0LBhVYYLAAAA1HPCGHXOzKmTYubUSfGfP98W2w/ZLbbbedfYescB0bJtu41+7dyPp8c7Lz8bY558JFavXJH5mE5bbh27HPyNAo8aAAAAqG+EMeqstWtWx7v/93y8+3/PR0REmw4do9MWW0WbDp2ieavW0bBx41i7elWsWrEiFs2dHXOmT4sln2bfgXK9Fm3axpHnXBgNGzWuiW8BAAAAqMOEMeqNxfPnxeL58yr99e27bh7/89OLo33XzQs4KgAAAKC+EsaoE4YceGhMfPWlWDhnVsG33bBx49jtsG/Hnt84Kho1aVLw7QMAAAD1kzBGnbDvMSfEvsecELM/+jA+GPtKfPTu2zFz6gexZtWqSm+zXeeu0XfPr8bAvfePdp27FnC0AAAAwKZAGKNO6bLVNtFlq21ir28fG8Vr18asDyfHrGlTYsGsT2LBrE9i4ZzZsWrFsli9YkWsWb0qGjdpGk2aNY8mzZtHyzbtotOWW0WXrXpGt222ja7b9KrtbwcAAACow4Qx6qyGjRpF9+16R/ftetf2UAAAAIBNUIPaHgAAAAAA1AZhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACS1Ki2B0D9tOTTBbU9hDrBzwEAAADqL2GMSjn70K/W9hAAAAAAqsSllAAAAAAkSRgDAAAAIEnCGAAAAABJMscYG9WhQ4eYM2dObQ+j3ujQoUNtDwEAAAAoB2GMjWrQoEF06tSptocBAAAAUFAupQQAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJalTbA1hv8uTJtT0EYBNXnuOMYxFQnRyHgLrAsQioC+rKcaaopKSkpFZ2XFRUG7sFAAAAoA6qjUTlUkoAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIUqPaHsB6DzzwQPTq1au2hwFswiZPnhxHHHFEmY855phjon379jUzICA5CxYsiL/+9a9lPsZrIqC6lec1kWMRUN3KcyyqCXUmjPXq1Sv69u1b28MAEte+ffvo3LlzbQ8DSJjXREBd4FgEpMKllAAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACS1Ki2B7De5MmTa3sIwCauPMeZBQsW1MBIgFSV5xjjNRFQ3cpznHEsAqpbXTnOFJWUlJTUyo6LimpjtwAAAADUQbWRqFxKCQAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAElqVNsDWO+BBx6IXr161fYwgE3Y5MmT44gjjijzMY5FQHUqz3HomGOOifbt29fMgIAkLViwIP7617+W+RiviYDqVp7XRTWhzoSxXr16Rd++fWt7GEDiHIuA2ta+ffvo3LlzbQ8DSJzXREAqXEoJAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkqVFtD2C9yZMn1/YQgE1ceY4zjkVAdSrPMWbBggU1MBIgZeU5znhNBFS3unKcKSopKSmplR0XFdXGbgEAAACog2ojUbmUEgAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJLUqLYHsN4DDzwQvXr1qu1hAJuwyZMnxxFHHFHmYxyLgOrkOATUBY5FQF1QnmNRTagzYaxXr17Rt2/f2h4GkDjHIqC2OQ4BdYFjEZAKl1ICAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAAAAJEkYAwAAACBJwhgAAAAASRLGAAAAAEiSMAYAAABAkoQxAAAAAJIkjAEAAACQJGEMAAAAgCQJYwAAAAAkSRgDAAAAIEnCGAAAAABJEsYAAAAASJIwBgAAAECShDEAAAAAkiSMAQAAAJAkYQwAAACAJAljAAAAACRJGAMAAAAgScIYAAAAAEkSxgAAAABIkjAGAAAAQJKEMQAAAACSJIwBAAAAkCRhDAAAAIAkCWMAAP9fe3cTG9d5HWD4cEhqGHFMihyJsmCRYqzICRT9BF5EjhrJRlG0cbJ2kbp1umo3XXjTRYKmQBdBoVUX/dm1KJC0gIMsXNRt4i6UBqqcOIgbR4wk2JbkWKbFkKNQlESO+T/sooibgiOFFsm5lM7zLL87vN8ZYjCLF/feAQAgJWEMAAAAgJSEMQAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAICVhDAAAAICUhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASKmj6AEAAACA5hqNRkxOThY9xgOjWq1GqeQaIf6PMAYAAABb1OTkZAwMDBQ9xgOjVqvFrl27ih6DLUQmBQAAACAlYQwAAACAlIQxAAAAAFLyjDEAAAC4j3z5B1+OSrVS9Bhb3szkTJz6zKmix2CLE8YAAADgPlKpVqKyUxiDjeBWSgAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAICVhDAAAAICUhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASKmj6AEAAACA+89CfSGmr01HvVaPuZtzsTS/FI3FRnR0dcS27m3RWemM8kPl6N3XG+WHykWPC00JYwAAAMCvtbywHOOvj8e73383rp+/HrdGb0WsrO1vux/ujv79/bHzEztj6LND0bO3Z3OHhTUSxgAAAIA7mr0xG2+8+EZc+valmL89f0/nqI/Xoz5ej9FXRuP1f3g9+h7ti30n98X+39kf23du3+CJYe2EMQAAAGCV5cXluPiti3H+hfOxNLu0oeeeensqpt6eipF/HokDTx+IQ793SCCjEB6+DwAAAPw/02PT8fLzL8dP/vEnGx7FflVjsRFv/uub8eKXXoyx18Y2bR+4E1eMAQAAAB+4fvF6fPer342F6YU7vqbUWYo9j++JgUMD0X+gPyq7K9HV1xUd5f/NDIuzizF3Yy6mfz4dk5cmozZSi9r5WjSWGk3P11hsxOzk7Ka8H7gbYQwAAACIiIja+Vqc/srpWJprfpVY90B3HHzmYOz/7f3Rub3zjucpP1T+4Bcp9z6xNyIi5m/Px9X/uhpv/MsbceudW5syP3xYwhgAAAAQ02PT8b2/+F7zKNYWcfCZg3H0uaPR0XVvKaHcU47HvvBYPPaFx2L0+6Px47//cdwevb3OqWF9hDEAAABIrrHciDNfOxPzt1b/6mSpsxQn/uxEDP3G0IbtN3h8MB459khc+OaFGPnGyB1vsYTN5uH7AAAAkNz5F87HjUs3Vq23ldri5FdPbmgU+6VSeykOP3s4nv6bp6PycGXDzw9r4YoxAAAASGx2ajYuvHCh6bHDzx6OweODm7p//8f64/N/9/mo1+qbug80I4wBAABAYhe+eaHpc8V6Bnvi8O8fbskM5Z5ylHvKLdkLfpVbKQEAACCppfmluPIfV5oe+9QffipKHbIBDzafcAAAAEhq9JXRWJhZWLXevbs7hk5s/HPFYKsRxgAAACCp937wXtP14aeGo63U1uJpoPWEMQAAAEhoZWUlxv57rOmxvU/sbfE0UAxhDAAAABKavjYdC9Orb6NsL7dH9ePVAiaC1hPGAAAAIKEbV240Xe8d6o32zvYWTwPFEMYAAAAgoelr003Xe4d6WzwJFEcYAwAAgITe/8X7Tde3V7e3eBIojjAGAAAACc3dnGu63tXX1eJJoDjCGAAAACS0PL/cdL2j3NHiSaA4Pu0AAACQ0PJC8zBW6lzfNTRnT52Nn53+2T397ZHnjsTRLx1d1/7wYbhiDAAAABJqK7U1XV9prLR4EiiOMAYAAAAJtZfbm643FhstngSKI4wBAABAQp3bO5uuz8/Mt3gSKI5njAEAAEBC26vbm67P3Wj+a5Vrte/kvugZ7Lnra2bGZ+LKy1fWtQ9sBGEMAAAAEuoe6G66Pv3z6XWdd/D4YAweH7zra8bPjQtjbAlupQQAAICEeod7m67funqrxZNAcYQxAAAASKj/Y/0RTX6Ysl6rx/u/eL/1A0EBhDEAAABIaFv3tuj7aF/TY+Ovj7d4GiiGMAYAAABJPfLEI03Xr5652uJJoBjCGAAAACQ1/ORw0/Wx18aifr3e2mGgAMIYAAAAJNX3aF/s/MTOVeuNpUZc/NbFAiaC1hLGAAAAILFP/u4nm66/9dJbcfPqzdYOAy0mjAEAAEBiQyeGovrx6qr1xlIjzv7l2ViaXypgKmgNYQwAAACSO/b8sSh1rE4EU29PxZmvnYnlxeUCpoLNJ4wBAABActUD1Tjy3JGmx669ei1Of+V0zE7Ntngq2HzCGAAAABCHnz0cw7853PTYxLmJeOmPX4pL37kUjeXGuvaZmZiJS/9+aV3ngI3SUfQAAAAAwNZw/E+Px/L8coy+Mrrq2PzN+Xj1r16Nn/7TT2P/5/bH0GeHYsfwjmhra/u1512oL8TYa2MxenY03j37bjSW1hfXYKMIYwAAAEBERLR3tsfJPz8ZP/rbH8Vb//ZW09fUa/UY+fpIjHx9JLZVtkXf/r6o7K5E146uaC+3R6xELM4uxtLcUtRr9bj93u2oT9RjpbFy1707ujqi/0D/ZrwtuCNhDAAAAPhAqb0Ux54/FruP7o4f/vUPY2F64Y6vXZhZiIlzEzERE/e+YVvE8JPD8fgfPR7dA933fh64B8IYAAAAsMrwU8Ox5/E9ce4b5+Lyty/H8sLG/jJlqbMUw08Nx8FnDkbfR/s29NywVsIYAAAA0FS5pxyf/pNPx5E/OBKXv3M53vnPd2Lq7al7Pl+psxQDhwZi6MRQDD85HOWe8gZOCx+eMAYAAADcVVdvVxz64qE49MVDMTM+E7XztZh8czJuv3c7ZiZmYv7WfCzNL0VjsRFt7W3Rvq09unq74iP9H4nKnkrsGN4R/Qf6Y9fBXdFRliLYOnwaAQAAgDWrPFyJysOVePS3Hi16FFi3UtEDAAAAAEARhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAICVhDAAAAICUhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAIKWOogcAAAAA1m5mcqboEe4L/k+shTAGAAAA95FTnzlV9AjwwHArJQAAAAApCWMAAAAApCSMAQAAAJCSZ4wBAADAFlWtVqNWqxU9xgOjWq0WPQJbjDAGAAAAW1SpVIpdu3YVPQY8sNxKCQAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApCWMAAAAApCSMAQAAAJCSMAYAAABASsIYAAAAACkJYwAAAACkJIwBAAAAkJIwBgAAAEBKwhgAAAAAKQljAAAAAKQkjAEAAACQkjAGAAAAQErCGAAAAAApdRQ9wC9dvny56BGAB9xavmd8FwGbyfcQsBX4LgK2gq3yPdO2srKyUsjGbW1FbAsAAADAFlREonIrJQAAAAApCWMAAAAApCSMAQAAAJBSYQ/fL+jRZgAAAAAQEa4YAwAAACApYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAICVhDAAAAICUhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAICVhDAAAAICUhDEAAAAAUhLGAAAAAEhJGAMAAAAgJWEMAAAAgJSEMQAAAABSEsYAAAAASEkYAwAAACAlYQwAAACAlIQxAAAAAFISxgAAAABISRgDAAAAIKX/AXGUERyCrg/cAAAAAElFTkSuQmCC)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **State Space ($ S $)**: The state space is all 4*4 = 16 cells in the grid. So the states are tuples like (row, column):\n",
    "\n",
    "  -  Starts at (0,0) for the top-left “S.”\n",
    "  - Ends at (3,3) for the bottom-right “G.”\n",
    "  -  Holes are at (1,1) and (2,2).\n",
    "  -  the other 10 cells are just regular places\n",
    "\n",
    "   So, S = {(0,0), (0,1), ..., (3,2), (3,3)}. That’s 16 possible states where the agent can be at any time.\n",
    "\n",
    "- **Action Space ($ A $)**: \n",
    "The agent can move in four directions: up, down, left, or right. We’ll keep it simple and define:\n",
    "\n",
    "    A = {0: up, 1: down, 2: left, 3: right}. Four actions. In code, we’ll use numbers 0 to 3 to keep it easy for gymnasium.\n",
    "\n",
    "\n",
    "- **Reward Function ($ R $)**:\n",
    "\n",
    "  - Reaching the goal (G) at (3,3): +10 reward.\n",
    "  - Falling into a hole at (1,1) or (2,2): -1 reward. \n",
    "  - Moving anywhere else (white cells or hitting a wall): 0 reward.\n",
    "  \n",
    "\n",
    "\n",
    "- **Transition Probability ($ P $)**: We assume this is deterministic environment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "id": "hgkvovqQiB8c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A    .    .    .\n",
      ".    H    .    .\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "Starting state: (0, {})\n",
      "After moving right: 1 Reward: 0 Done: False\n"
     ]
    }
   ],
   "source": [
    "import gymnasium as gym\n",
    "from gymnasium import spaces\n",
    "import numpy as np\n",
    "\n",
    "class GridWorldEnv(gym.Env):\n",
    "    def __init__(self):       \n",
    "        self.grid_size = 4\n",
    "        self.start = (0, 0)  \n",
    "        self.goal = (3, 3)  \n",
    "        self.holes = [(1, 1), (2, 2)]  \n",
    "        self.state = self.start\n",
    "        \n",
    "        \n",
    "        self.action_space = spaces.Discrete(4)  # 0: up, 1: down, 2: left, 3: right\n",
    "        self.observation_space = spaces.Discrete(16)  # from 0 to 15 for the problem states\n",
    "\n",
    "    def get_observation(self):\n",
    "        \"\"\"we convert internal state (i, j) to integer observation using state = 4*i + j.\"\"\"\n",
    "        i, j = self.state\n",
    "        return 4 * i + j\n",
    "\n",
    "    def step(self, action):        \n",
    "        row, col = self.state\n",
    "        \n",
    "         \n",
    "        if action == 0:  # up\n",
    "            next_row = max(row - 1, 0)\n",
    "            next_col = col\n",
    "        elif action == 1:  # down\n",
    "            next_row = min(row + 1, 3)\n",
    "            next_col = col\n",
    "        elif action == 2:  # left\n",
    "            next_row = row\n",
    "            next_col = max(col - 1, 0)\n",
    "        elif action == 3:  # right\n",
    "            next_row = row\n",
    "            next_col = min(col + 1, 3)\n",
    "\n",
    "        next_state = (next_row, next_col)\n",
    "        \n",
    "        # rewards\n",
    "        if next_state in self.holes:\n",
    "            reward = -1\n",
    "            done = True\n",
    "        elif next_state == self.goal:\n",
    "            reward = 10\n",
    "            done = True\n",
    "        else:\n",
    "            reward = 0\n",
    "            done = False\n",
    "        \n",
    "        # Update the internal state\n",
    "        self.state = next_state\n",
    "        # Return observation as integer, along with reward and status\n",
    "        return self.get_observation(), reward, done, False, {}\n",
    "\n",
    "    def render(self):\n",
    "        \"\"\"Render the grid world with the agent's position.\"\"\"\n",
    "        # Initialize a 4x4 grid with empty cells\n",
    "        grid = [['.' for _ in range(4)] for _ in range(4)]\n",
    "        \n",
    "        grid[0][0] = 'S'               # Start\n",
    "        grid[3][3] = 'G'               # Goal\n",
    "        for hole in self.holes:        # Holes\n",
    "            grid[hole[0]][hole[1]] = 'H'\n",
    "        \n",
    "        agent_row, agent_col = self.state\n",
    "        if grid[agent_row][agent_col] == 'S':\n",
    "            grid[agent_row][agent_col] = 'A'  # Agent on start\n",
    "        elif grid[agent_row][agent_col] == 'G':\n",
    "            grid[agent_row][agent_col] = 'A'  # Agent on goal\n",
    "        elif grid[agent_row][agent_col] == 'H':\n",
    "            grid[agent_row][agent_col] = 'A(H)'  # Agent on hole\n",
    "        else:\n",
    "            grid[agent_row][agent_col] = 'A'     # Agent on empty cell\n",
    "        \n",
    "        # Print the grid\n",
    "        for row in grid:\n",
    "            print('    '.join(row))\n",
    "        print()  # empty line \n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"Reset the environment to the start state and return initial observation.\"\"\"\n",
    "        self.state = self.start\n",
    "        return self.get_observation(), {}\n",
    "\n",
    "# quick test\n",
    "env = GridWorldEnv()\n",
    "env.render()\n",
    "print(\"Starting state:\", env.reset())\n",
    "action = 3  # move right action for test\n",
    "next_state, reward, done, truncated , _ = env.step(action)\n",
    "print(\"After moving right:\", next_state, \"Reward:\", reward, \"Done:\", done)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [],
   "source": [
    "def qlearning_train_gridworld(env , learning_rate , discount_factor ):\n",
    "    \n",
    "    # we use the tensorboard writer\n",
    "    writer = SummaryWriter(f\"Qlearning_GridWorld/experiment_lr_{learning_rate}_{discount_factor}\")\n",
    "    \n",
    "    # q table dimensions based on state and action space\n",
    "    num_states = env.observation_space.n\n",
    "    num_actions = env.action_space.n\n",
    "    q_values = np.zeros((num_states, num_actions))\n",
    "    \n",
    "    # hyperparameters for the problem\n",
    "    alpha = learning_rate          # learning rate\n",
    "    gamma = discount_factor         # discount factor\n",
    "    eps = 1.0            # initial exploration probability\n",
    "    decay = 0.005        # epsilon decay rate per episode\n",
    "    episodes = 5000\n",
    "    max_steps_per_ep = 99\n",
    "    \n",
    "    \n",
    "    episode_rewards = []\n",
    "    print(\"training...\")\n",
    "    \n",
    "    for ep in range(episodes):\n",
    "        # reseting the environment\n",
    "        state, _ = env.reset()\n",
    "        total_reward = 0\n",
    "    \n",
    "        for step in range(max_steps_per_ep):\n",
    "            \n",
    "            if random.random() < eps:\n",
    "                chosen_action = env.action_space.sample()\n",
    "            else:\n",
    "                chosen_action = np.argmax(q_values[state, :])\n",
    "            \n",
    "            next_state, reward, done, _, _ = env.step(chosen_action)\n",
    "            \n",
    "            q_values[state, chosen_action] += alpha * (\n",
    "                reward + gamma * np.max(q_values[next_state, :]) - q_values[state, chosen_action]\n",
    "            )\n",
    "    \n",
    "            state = next_state\n",
    "            total_reward += reward\n",
    "    \n",
    "            if done:\n",
    "                break\n",
    "    \n",
    "        # decay epsilon over episodes\n",
    "        eps = 1  / (1  + decay * ep)\n",
    "        episode_rewards.append(total_reward)\n",
    "        \n",
    "        avg_reward = np.mean(episode_rewards[-100:]) if ep >= 100 else np.mean(episode_rewards)\n",
    "        writer.add_scalar(\"Mean_Episode_Reward\", avg_reward, ep)\n",
    "    \n",
    "    print(f\"Training completed over {episodes} episodes.\")\n",
    "    writer.close()\n",
    "    env.close()\n",
    "\n",
    "    return q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simulate_gridworld_agent(environment, q_matrix,  max_steps_per_run= 100):\n",
    "    print(\"environment\" , environment)\n",
    "    \n",
    "    current_state, _ = environment.reset()\n",
    "    print(current_state)\n",
    "    finished = False    \n",
    "    sleep(1)\n",
    "\n",
    "    for step in range(max_steps_per_run):            \n",
    "        environment.render()\n",
    "        sleep(0.5)  \n",
    "        \n",
    "        chosen_action = np.argmax(q_matrix[current_state, :])\n",
    "        next_state, reward, finished, _, _ = environment.step(chosen_action)\n",
    "        current_state = next_state\n",
    "\n",
    "        if finished:\n",
    "            environment.render()\n",
    "            print(f\"Run completed in {step + 1} steps.\\n\")\n",
    "            sleep(2)\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training...\n",
      "Training completed over 5000 episodes.\n"
     ]
    }
   ],
   "source": [
    "env = GridWorldEnv()\n",
    "q_values = qlearning_train_gridworld(env = env , learning_rate = 0.9 , discount_factor = .99 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 9.41480149,  9.32065348,  9.41480149,  9.5099005 ],\n",
       "       [ 9.5099005 , -1.        ,  9.41480149,  9.6059601 ],\n",
       "       [ 9.6059601 ,  9.70299   ,  9.5099005 ,  9.70299   ],\n",
       "       [ 9.70299   ,  9.801     ,  9.6059601 ,  9.70299   ],\n",
       "       [ 9.41480149,  9.22737868,  9.32065347, -1.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 9.6059601 , -1.        , -1.        ,  9.801     ],\n",
       "       [ 9.70299   ,  9.9       ,  9.70299   ,  9.801     ],\n",
       "       [ 9.32065234,  0.        ,  7.45897358,  0.        ],\n",
       "       [-0.99      ,  0.        ,  0.        , -0.999     ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 9.801     , 10.        , -1.        ,  9.9       ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.999     ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.        ,  0.        ,  0.        ,  0.        ]])"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "environment <GridWorldEnv instance>\n",
      "0\n",
      "A    .    .    .\n",
      ".    H    .    .\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "S    A    .    .\n",
      ".    H    .    .\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "S    .    A    .\n",
      ".    H    .    .\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "S    .    .    .\n",
      ".    H    A    .\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "S    .    .    .\n",
      ".    H    .    A\n",
      ".    .    H    .\n",
      ".    .    .    G\n",
      "\n",
      "S    .    .    .\n",
      ".    H    .    .\n",
      ".    .    H    A\n",
      ".    .    .    G\n",
      "\n",
      "S    .    .    .\n",
      ".    H    .    .\n",
      ".    .    H    .\n",
      ".    .    .    A\n",
      "\n",
      "Run completed in 6 steps.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "simulate_gridworld_agent(environment = env, q_matrix = q_values , max_steps_per_run= 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fAzGwDFc56Bv"
   },
   "source": [
    "## Modify Q-Learning hyperparameters for gridworld"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n",
      "training...\n",
      "Training completed over 5000 episodes.\n"
     ]
    }
   ],
   "source": [
    "learning_rates = [0.1, 0.9]  \n",
    "gammas = [0.95, 0.99]\n",
    "\n",
    "for lr in learning_rates:\n",
    "    for gamma in gammas:\n",
    "        qlearning_train_gridworld(env = env , learning_rate = lr , discount_factor = gamma )"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}