{ "cells": [ { "cell_type": "markdown", "id": "c3e89aca-6c52-4b20-b1a3-9279ef0fd99b", "metadata": {}, "source": [ "# Deep Q-Network (DQN) on LunarLander-v2\n", "\n", "> In this post, We will take a hands-on-lab of Simple Deep Q-Network (DQN) on openAI LunarLander-v2 environment. This is the coding exercise from udacity Deep Reinforcement Learning Nanodegree.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Reinforcement_Learning, PyTorch, Udacity]\n", "- image: images/LunarLander-v2.gif" ] }, { "cell_type": "markdown", "id": "fd834f1b-51c4-4910-a4d6-3b212e1a2a5a", "metadata": {}, "source": [ "## Deep Q-Network (DQN)\n", "---\n", "In this notebook, you will implement a DQN agent with OpenAI Gym's LunarLander-v2 environment.\n", "\n", "### Import the Necessary Packages" ] }, { "cell_type": "code", "execution_count": 1, "id": "22518486-3c2f-47fe-92b3-1502875eacfe", "metadata": {}, "outputs": [], "source": [ "import gym\n", "import random\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import matplotlib.pyplot as plt\n", "import base64, io\n", "\n", "import numpy as np\n", "from collections import deque, namedtuple\n", "\n", "# For visualization\n", "from gym.wrappers.monitoring import video_recorder\n", "from IPython.display import HTML\n", "from IPython import display \n", "import glob" ] }, { "cell_type": "markdown", "id": "f75f934c-6921-43aa-8389-6df4b993eca4", "metadata": {}, "source": [ "### Instantiate the Environment and Agent\n", "\n", "Initialize the environment." ] }, { "cell_type": "code", "execution_count": 2, "id": "594828b6-da33-481d-ab42-041e8c17ffea", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "State shape: (8,)\n", "Number of actions: 4\n" ] } ], "source": [ "env = gym.make('LunarLander-v2')\n", "env.seed(0)\n", "print('State shape: ', env.observation_space.shape)\n", "print('Number of actions: ', env.action_space.n)" ] }, { "cell_type": "markdown", "id": "03735bdc-c07e-4c87-b208-cce894bb8e43", "metadata": {}, "source": [ "### Define Neural Network Architecture.\n", "\n", "Since `LunarLander-v2` environment is sort of simple envs, we don't need complicated architecture. We just need non-linear function approximator that maps from state to action." ] }, { "cell_type": "code", "execution_count": 3, "id": "ae834607-433e-4ed5-8b23-8de7b53230a8", "metadata": {}, "outputs": [], "source": [ "class QNetwork(nn.Module):\n", " \"\"\"Actor (Policy) Model.\"\"\"\n", "\n", " def __init__(self, state_size, action_size, seed):\n", " \"\"\"Initialize parameters and build model.\n", " Params\n", " ======\n", " state_size (int): Dimension of each state\n", " action_size (int): Dimension of each action\n", " seed (int): Random seed\n", " \"\"\"\n", " super(QNetwork, self).__init__()\n", " self.seed = torch.manual_seed(seed)\n", " self.fc1 = nn.Linear(state_size, 64)\n", " self.fc2 = nn.Linear(64, 64)\n", " self.fc3 = nn.Linear(64, action_size)\n", " \n", " def forward(self, state):\n", " \"\"\"Build a network that maps state -> action values.\"\"\"\n", " x = self.fc1(state)\n", " x = F.relu(x)\n", " x = self.fc2(x)\n", " x = F.relu(x)\n", " return self.fc3(x)" ] }, { "cell_type": "markdown", "id": "b0873298-cab0-4dcb-9c84-4782dc914dd7", "metadata": {}, "source": [ "### Define some hyperparameter" ] }, { "cell_type": "code", "execution_count": 4, "id": "7010c525-29d8-445c-8769-6cfb7d00948b", "metadata": {}, "outputs": [], "source": [ "BUFFER_SIZE = int(1e5) # replay buffer size\n", "BATCH_SIZE = 64 # minibatch size\n", "GAMMA = 0.99 # discount factor\n", "TAU = 1e-3 # for soft update of target parameters\n", "LR = 5e-4 # learning rate \n", "UPDATE_EVERY = 4 # how often to update the network" ] }, { "cell_type": "code", "execution_count": 5, "id": "334bb7d8-7d62-4cfb-96f7-f8809ba8e089", "metadata": {}, "outputs": [], "source": [ "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")" ] }, { "cell_type": "markdown", "id": "1d861efe-200c-4690-9698-722abbf0b77c", "metadata": {}, "source": [ "### Define Agent " ] }, { "cell_type": "code", "execution_count": 6, "id": "0530f456-2bfd-4061-ad62-f14846a9a284", "metadata": {}, "outputs": [], "source": [ "class Agent():\n", " \"\"\"Interacts with and learns from the environment.\"\"\"\n", "\n", " def __init__(self, state_size, action_size, seed):\n", " \"\"\"Initialize an Agent object.\n", " \n", " Params\n", " ======\n", " state_size (int): dimension of each state\n", " action_size (int): dimension of each action\n", " seed (int): random seed\n", " \"\"\"\n", " self.state_size = state_size\n", " self.action_size = action_size\n", " self.seed = random.seed(seed)\n", "\n", " # Q-Network\n", " self.qnetwork_local = QNetwork(state_size, action_size, seed).to(device)\n", " self.qnetwork_target = QNetwork(state_size, action_size, seed).to(device)\n", " self.optimizer = optim.Adam(self.qnetwork_local.parameters(), lr=LR)\n", "\n", " # Replay memory\n", " self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, seed)\n", " # Initialize time step (for updating every UPDATE_EVERY steps)\n", " self.t_step = 0\n", " \n", " def step(self, state, action, reward, next_state, done):\n", " # Save experience in replay memory\n", " self.memory.add(state, action, reward, next_state, done)\n", " \n", " # Learn every UPDATE_EVERY time steps.\n", " self.t_step = (self.t_step + 1) % UPDATE_EVERY\n", " if self.t_step == 0:\n", " # If enough samples are available in memory, get random subset and learn\n", " if len(self.memory) > BATCH_SIZE:\n", " experiences = self.memory.sample()\n", " self.learn(experiences, GAMMA)\n", "\n", " def act(self, state, eps=0.):\n", " \"\"\"Returns actions for given state as per current policy.\n", " \n", " Params\n", " ======\n", " state (array_like): current state\n", " eps (float): epsilon, for epsilon-greedy action selection\n", " \"\"\"\n", " state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n", " self.qnetwork_local.eval()\n", " with torch.no_grad():\n", " action_values = self.qnetwork_local(state)\n", " self.qnetwork_local.train()\n", "\n", " # Epsilon-greedy action selection\n", " if random.random() > eps:\n", " return np.argmax(action_values.cpu().data.numpy())\n", " else:\n", " return random.choice(np.arange(self.action_size))\n", "\n", " def learn(self, experiences, gamma):\n", " \"\"\"Update value parameters using given batch of experience tuples.\n", "\n", " Params\n", " ======\n", " experiences (Tuple[torch.Variable]): tuple of (s, a, r, s', done) tuples \n", " gamma (float): discount factor\n", " \"\"\"\n", " # Obtain random minibatch of tuples from D\n", " states, actions, rewards, next_states, dones = experiences\n", "\n", " ## Compute and minimize the loss\n", " ### Extract next maximum estimated value from target network\n", " q_targets_next = self.qnetwork_target(next_states).detach().max(1)[0].unsqueeze(1)\n", " ### Calculate target value from bellman equation\n", " q_targets = rewards + gamma * q_targets_next * (1 - dones)\n", " ### Calculate expected value from local network\n", " q_expected = self.qnetwork_local(states).gather(1, actions)\n", " \n", " ### Loss calculation (we used Mean squared error)\n", " loss = F.mse_loss(q_expected, q_targets)\n", " self.optimizer.zero_grad()\n", " loss.backward()\n", " self.optimizer.step()\n", "\n", " # ------------------- update target network ------------------- #\n", " self.soft_update(self.qnetwork_local, self.qnetwork_target, TAU) \n", "\n", " def soft_update(self, local_model, target_model, tau):\n", " \"\"\"Soft update model parameters.\n", " θ_target = τ*θ_local + (1 - τ)*θ_target\n", "\n", " Params\n", " ======\n", " local_model (PyTorch model): weights will be copied from\n", " target_model (PyTorch model): weights will be copied to\n", " tau (float): interpolation parameter \n", " \"\"\"\n", " for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):\n", " target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)" ] }, { "cell_type": "markdown", "id": "4eb5db66-ecb3-4bf4-aee1-3e18003c17b0", "metadata": {}, "source": [ "### Define Replay Buffer" ] }, { "cell_type": "code", "execution_count": 7, "id": "f74bb08e-0b95-42db-9fc8-d609514d55af", "metadata": {}, "outputs": [], "source": [ "class ReplayBuffer:\n", " \"\"\"Fixed-size buffer to store experience tuples.\"\"\"\n", "\n", " def __init__(self, action_size, buffer_size, batch_size, seed):\n", " \"\"\"Initialize a ReplayBuffer object.\n", "\n", " Params\n", " ======\n", " action_size (int): dimension of each action\n", " buffer_size (int): maximum size of buffer\n", " batch_size (int): size of each training batch\n", " seed (int): random seed\n", " \"\"\"\n", " self.action_size = action_size\n", " self.memory = deque(maxlen=buffer_size) \n", " self.batch_size = batch_size\n", " self.experience = namedtuple(\"Experience\", field_names=[\"state\", \"action\", \"reward\", \"next_state\", \"done\"])\n", " self.seed = random.seed(seed)\n", " \n", " def add(self, state, action, reward, next_state, done):\n", " \"\"\"Add a new experience to memory.\"\"\"\n", " e = self.experience(state, action, reward, next_state, done)\n", " self.memory.append(e)\n", " \n", " def sample(self):\n", " \"\"\"Randomly sample a batch of experiences from memory.\"\"\"\n", " experiences = random.sample(self.memory, k=self.batch_size)\n", "\n", " states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)\n", " actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).long().to(device)\n", " rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)\n", " next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)\n", " dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)\n", " \n", " return (states, actions, rewards, next_states, dones)\n", "\n", " def __len__(self):\n", " \"\"\"Return the current size of internal memory.\"\"\"\n", " return len(self.memory)" ] }, { "cell_type": "markdown", "id": "6bd61b8d-b63e-444f-9cb7-6295df46995d", "metadata": {}, "source": [ "### Training Process" ] }, { "cell_type": "code", "execution_count": 8, "id": "c907ab57-ed48-4824-b27c-8c7d707d6919", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Episode 100\tAverage Score: -203.18\n", "Episode 200\tAverage Score: -121.23\n", "Episode 300\tAverage Score: -55.393\n", "Episode 400\tAverage Score: -12.05\n", "Episode 500\tAverage Score: 41.702\n", "Episode 600\tAverage Score: 84.14\n", "Episode 700\tAverage Score: 142.01\n", "Episode 800\tAverage Score: 173.82\n", "Episode 900\tAverage Score: 164.89\n", "Episode 1000\tAverage Score: 153.46\n", "Episode 1100\tAverage Score: 184.06\n", "Episode 1200\tAverage Score: 172.37\n", "Episode 1278\tAverage Score: 200.02\n", "Environment solved in 1178 episodes!\tAverage Score: 200.02\n" ] } ], "source": [ "def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n", " \"\"\"Deep Q-Learning.\n", " \n", " Params\n", " ======\n", " n_episodes (int): maximum number of training episodes\n", " max_t (int): maximum number of timesteps per episode\n", " eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n", " eps_end (float): minimum value of epsilon\n", " eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n", " \"\"\"\n", " scores = [] # list containing scores from each episode\n", " scores_window = deque(maxlen=100) # last 100 scores\n", " eps = eps_start # initialize epsilon\n", " for i_episode in range(1, n_episodes+1):\n", " state = env.reset()\n", " score = 0\n", " for t in range(max_t):\n", " action = agent.act(state, eps)\n", " next_state, reward, done, _ = env.step(action)\n", " agent.step(state, action, reward, next_state, done)\n", " state = next_state\n", " score += reward\n", " if done:\n", " break \n", " scores_window.append(score) # save most recent score\n", " scores.append(score) # save most recent score\n", " eps = max(eps_end, eps_decay*eps) # decrease epsilon\n", " print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n", " if i_episode % 100 == 0:\n", " print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n", " if np.mean(scores_window)>=200.0:\n", " print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n", " torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')\n", " break\n", " return scores\n", "\n", "agent = Agent(state_size=8, action_size=4, seed=0)\n", "scores = dqn()" ] }, { "cell_type": "markdown", "id": "1ba6726a-977a-4345-8897-021302bfc262", "metadata": {}, "source": [ "### Plot the learning progress" ] }, { "cell_type": "code", "execution_count": 9, "id": "d2d491c9-a5dc-4c32-a95d-796f85c60c83", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEGCAYAAACZ0MnKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABHF0lEQVR4nO2dd5wV5dXHf2cLW1iWpfdeRBCkgx2FCIotmrxiN5qgxFgSo8GSWDHGGKPGFo1dY68RC0UQRbrSe5Xee1l29573j5m5d2bu1Htnbtk9Xz589t4pz5w7d+5znnPOc85DzAxBEARB8EJOugUQBEEQsgdRGoIgCIJnRGkIgiAInhGlIQiCIHhGlIYgCILgmbx0CxA2DRs25LZt26ZbDEEQhKxizpw5O5i5kXl7tVcabdu2xezZs9MthiAIQlZBROustot7ShAEQfCMKA1BEATBM6I0BEEQBM+I0hAEQRA8I0pDEARB8IwoDUEQBMEzojQEQRAEz4jSEARByBImLduGjXsOp1UGURqCIAhZwq9enoWzHp+SVhlEaQiCICRBqhay27bvCABg35FKVFZFUnJNK0RpCIIgJMGlL8xAxzs/D/06/R+aGH1dUZW+FVerfe0pQRCEMJm2emfKr1kZiQDITfl1AbE0BCF0Pp23CWt3HMT5T0/Fjz/tjttfWRXBxCVb8fp0pT5chQfXw66DR/HAZ4s9HZstMDO+XLgFRyvT85k27D6Et2b+lJZrA8C0VTvx1y+WeDq2UrU0lmzeh6pIaq0OURqCECIVVRHc9NaPGPToZMxbvwf3froo7pjfvDYb1746G3/+eCHen7MBne76Aqu3H4g7bs+ho1H/+YOfLcaL363BuEVbA5W3KsJ45Mul2HXwaKDtemHSsm24/o05eHrSypRfGwB++dw03PHhAhypqMLXS7fiQHllSq5bFWHsPngUl7wwHf/+ZrXlMea4SUUkgqVb9uGsJ77FExOWY/+RCizbsh/3/28xBv19UqjyitIQhABYu+Mgnpy4Iu7HbY6RWg0KJy3bHn2tdZgLNykjyL2HKgAA63YeRM/7x+OV79cCAA5XVAEAiAL6ACqTl23DM5NX4R4L5RY2a3ccAgDsPVyR8msDwOa9SqB52uqduOaV2Rj9wfyUXPdvXy5FrwfGOx5TaXpwKqsY2/aVAwC+XrYN5z81FUMfn4KXpq7B2p2HQpMVEKUhCIFw1csz8dj45di2X/khX/HiDDwxYUXccQxnV8KaHQcBAFv2HsbdHy/A8fePQ0VVBKtUy2PCEsWy0AKhuTmET+ZuRN8HJwQyo0Zr94iqlFLJHlVZlBblJ9zGzgPluPKlmdh5oDzhNn718iwAwIbd/vIhEs2fGDt/s+sxZjdkZRUjN0cZMSzcuA+r1ecmFYjSEIQA0DrZiGpafLtiB/45YXmckvA6O/Ohz5firZnrASjui2teURYS03zZVRGlE8klwp8/XogdB8qx/0hq3ClhsU9TGoWJz895ddo6TFm+HX0enIDNe+M78VXbD+D7VTs8tcXMuOmtH7HJQRls238k+vqkh792bO+5b1bhhv/+ELdd6/ydMMd5KiORwK1Mr4jSEAQb3pi+zjIGYUWO+gs2KwW3914o13UYEWZ8v3IHpq5UZuz8+rXZ2KcqC7emmRkf/rDBMnh+tDKCPYeORjuiFKUeGNAUbp6HTtQLizbuM7z/ftUODP7HN7j0hRlxx1oF3xdu2odP523CH96dixmrd2LBhr1xx5z+98me5Xn4i6WWVoX58776/Vrc/7/FjvJVRhgvfbfW87WDRJSGINhw98cLozEENzSlceLDX0ddSVYk0hcff9+46OtZa3fj0v/MwFGLjt/NPfXJ3E34w7vz8PyU+GDr9W/MQc/7x8Nvd712x0FfyW0rtx3A2PmbMWfd7jjlpTVDAQ2hiwuMU1L1yuLql2ca9nW++4u48xuW1AIAbNl7BBc/Px3nPvVd3DEHjxrdeIePurv1Vm47AGbGiq378ds358TFK+75dBFemrrGsM38fe86eDTqqrQizBlVojSEGg0zY8veI+4H+uDLhVuiryOmDnXvoaOIRBg7DpRjw+5DmLYquDn+VopEjzYjavv+eH//10u3+b7ezDW7MOjRyXh39nrP5wx57Bvc8N8fcNGz3+PhL5Ya9mmuvER1xrb9R7BhVywI7DR1d7Ju8kGQ/Ovr+DhWJMKYt35P9P2Qx77Bp/M24fIXZ+DzBVvw0y73wLX5s4x4frrj8de8MsubwAkgyX1CxvDkxBXo0rQOzuzWNGXX/O/Mn3DXRwvx2Y0n47gWdRNuJ0c3/NL7qM2D8E17j2D4v77Dks37kJ9LgWb2Vrq0leiVmBlvzvgJP+/VArULYl3Gsq37ASjWz8HyKpQU5OHMbk1QVlzLU7uLNhndPdq9+ssni3DZgDaG+zhn3S70bl0vaoVEIoz95ZWoW5QPZsaTE1finxOWG9pzy/dgZtz+/nyc3Kmh5f6t+/wH0w9bTCB4e9Z63PnRAsO2xZv3GdyObvi1HL5ZHo5SBMTSEDKIx8Yvx8jX54TS9n3/W4S2o8fGbf9ejQ2s8Tj7pLIqgg27D6GiKoK2o8fisfFKR0U6x06ubqhs9VNfslnxtQddCsJPot/4xVsxZ93uuPPY4tXUlTtx98cLo3729bsOYdv+I4ioHdn7czbg/s8W4/YP5uP378y1vebt788zvDf3g/q3B4/GgvpfL92Ki56dhtemrYtuG/P5Ehx/3zgcOlqJRZv2xSkMAFixzd5NCChTbN+bswE3v20vsx2VVRFLd6CVp27Njng5ivJzPbsCV27bjy901mu6EaUh1AhenrrWcrs2gvMygwVQ5tSf/LdJWKfOhX/xWyU+oD89x2BppC6ibHZPHamowtOTVkaVgv4T/ua12bjo2e+V8/SBdvV+MMdea0luO1X31imPTEL/MRPjfPGA8+j83dkbDO837j6MN2ess8zL0N+2LXuVNu/5dBF+0u77d4rPv7zCfhbR379aZisLAMuZTF7pO2YC+o6Z4HrcN8u344Vv18RtL8r3XgJkyGNTooOTTECUhlCjqVJ7pxyPjvRvVyjTNc0Z0/rg7QOfxWa+pHISktlyeWHKavz9q2V48bs1+L/npmHRJsXCcVJkmiKYuHQb2t/5uapUrWMNEQul4ScesXHPYdz10UIcf984jHpjjkFR/Ofb1Vi3U7H+CvJi3dSpf59kuPe9HhiPG9/60f7zVEXwr4krsNLC6vjxpz2e5LSayrznUAX2HLJSdox9Ryrw2rS1YGbMXGMdsyqulRtYwD/ViNIQsp6lW/YlnIwW8WlpaOh9zLPW7sJ6m2BmKqeumt1T2sye6at3YubaXfjgB2Wkv99UHkMvYmXE2Mb+I/bZ2VaWhhVb9x1xLclhdr/86+uVuPjfSrC3IN/YTZlnPq3ebu9afH/OBvxj/HIMeewbT7JasVOnpNqOHhstUW4FA/jLxwvxl08WYeaaXQa3pZ6C/FzXUi1rdxxEeWXiSZZhWbmiNISsYt76PXhbV1Ruz6GjGPb4t/jje/Mczophjmtolkauh1/C4aNVUReQXmn88rlp9h1oCpXGlS/ORMc7P8f/5m0CoCSTAYjrtswWgr5z+WKBsfPWu47i2vHYKQ14aCKOu+crD0ca29txoBzb95fHzW6bb5EvYUciwWw3bncoL8IM7DigKIPyygjsxiK7PdT2GvToZFyfRIxPc+EFjSgNIas4/+mpGP1hbCbKIXU0rQV1F27ci7ajx2KuboqjE1rn7+Se0jrZY//yZXRUax6R2+FWNiRIDldUoTLCuP8zY2LYJNP0Ur1EkQjjjekxJTxusXHu/97DFbr8CeP1gs4FMOugygjjhL9OxINjvVV+tSKMOlZO03VZ/Qeo98vmuaryqHDN350fvP4G/CJKQ8hKtNGx+ac3eZmSbzB+sf1sE/2sF220PHnZdlsXl5UVoXWYbn7pdGRW7z541NE1obcmxi7YjL99udT22AlLthlcS/p2vQRn/bhIrA716gKzY8+h1FbrXbXtYDTv4ooXZ+LJifF5G0BwWe9OfDZ/cyg1xNKmNIioFRFNIqIlRLSIiG5Wt9cnovFEtEL9W093zh1EtJKIlhHR0HTJLqQf83x47Sf46DilI3Oafdrxri/w/pwN6nFKp/TK92sxxmZEa2VVeO3M0rG+WmWEHa0A/Syrf09Z5djWkxNX4Lb3FXcMgfCETSeoYdahSzbvd5E2RhhW2Yc/bgy8TSemrd6J9bvcCxdOWhpeHoWefC9+V5+k09KoBHArMx8LYCCAG4ioK4DRACYycycAE9X3UPeNANANwDAAzxBRepauEtLOAZfifG7+ds3vr9cHizfvw8X/nhZX6M7J0nAjlVNu9Xh1fyw01WdygkhZUMqNw0ercFC1Ts5+8lvP7afpVqWFVK3253eChxfSpjSYeTMz/6C+3g9gCYAWAM4H8Kp62KsALlBfnw/gbWYuZ+Y1AFYC6J9SoYWMwW02jtV0UD2ae0Dfuc5Ztxsz1uzCC1OMAcSvLBKrPCsNT0cFj8eQiy+I3O8roMR+ut3zFdamsFy3kDoyIqZBRG0B9AIwA0ATZt4MKIoFQGP1sBYA9EVuNqjbrNobSUSziWj29u2pMQOF4Fi8aR/ajh6LMWMX2x6jWRJ2I3m3kbY2ArOySMzbbnt/PlZuM7pZojENx6ukb/Qc1noYblns+immb8/yXpMKAN6bs8H9ICHtpF1pEFEJgA8A3MLMTray1e/T8glm5ueZuS8z923UqFEQYgopZOwCxQVilUmrYVXTSb8IzstT10ZXvbMiLzdWw8iMlRUx5LEphvfeYxqp0Rqt6hcZ3g96dHLg1/h8wZaEFxoSUsfFfVuF2n5alQYR5UNRGG8y84fq5q1E1Ezd3wyAVn5zAwD93WgJwN3BKlRLtK5YP3vJvN7BtNX2i+1oU2ytLBIvwUMvbhoAKfNPFecba4+ma8lUPVqeSLbRrXkp+rap535ghtK4tCDU9tM5e4oAvAhgCTM/ptv1KYCr1NdXAfhEt30EERUQUTsAnQAYU0OFaoEXl471McaNdYvsq63OXqvkdVjNsnpp6hqMX2y/VgGQ/tlTvVqXGd4X1ZI5IWYSrdKRpdU90KFRbQDKgOj70Wfgm9sGhXKddFoaJwG4AsAZRDRX/X82gIcB/IyIVgD4mfoezLwIwLsAFgP4EsANzJz6hYyFjGDFtv24/3+LDTGNg+XGx+GSF6Zj1fYD6PLn+AV2tqilIOwsBnNFVjNVXpP7QtIa5naLRWkYyM+lpO59NiqOdg0VpZGbQ2heVoQ2DWqHcp20rafBzN/BPo442OacMQDGhCaUkDX87r9Kkbqzu8fW3rjVopTIsi37caTCvoO3iznk5jiPpzItppEpSmOrQ12mVPLp707GWU94n+6rx65eVLYQxjRbPWkPhAvVF2bGvyau8B081XezbnkOYSQvAe4Zu97zNIKQxqJd0/sCH6W2w2SbxaqAqaZNg2Ic26zU07F/PLNz3LZstDL0iNIQspZV2w/iH+OXY9QbiRddc+t0w/qBuLWrKQ03neBlOdfRZ3XxKlYUs3R1i/J9t1FdKSmo2QuS5oas9URpVBO27juCcYsyZ3UvIJbvcPho4qEnt8xur5VWzRTmOz/6bkpDMzTcLA4rl5meTo1L0L9dfcdjrOjcpARXn9gW/dsq5+bnEFqUFbmclXlowdtUkulK5aSODfD5TadY7lv6wDDX83PE0hC88IvnvsfI1+fguxU7sHCj99LR6eSTuRvR/Z6vHJcpdfMCue3/doV1cueRigjmrNtta8m4Kw1VISaZRBdhdqyw27FxieX2HCLce143XfAzJ21ulcl/HJTwuWG4F7X78NSlvaz3615bff/6/Rf2sswfDpXSwvy4vBsNL/cr7FqIojSqCVqRtMtfnIFz/vVdmqXxxgOfLcb+8sq4FdD0P2Q3S8JtpP/WTPus5DdnrLPd56Y0gqopxXD+kdvFVrSOUYvXE6VvBK2f7tuzVZmvc2vlhdcF1bLpYP18c12a1QlGGB+UFubbPn9Oz0q0hH0IMhlkCLl9QbDFS7/rpjSS6bw//GEjVlgsAwq4+4WDCnBHIs6Whr3yIsPfHAJ+d0bHYISy4aSODawl0YlYu8BfQD4US8Ol29Q/M5Zfo+4DuS0DfNmA1n5E80RpUZ7tdZ1K8ccGEuKeEmowbu6noBcC0li21bmkd1CXrXJxT9kpDe2U2F/COT2ah5bQBSgjYDcK8+KVRv3axiTLIt1MLy8r2AWNn6/Obb2UlvWKfV27S1N3y6UoP9fR0rUTSZNVLA0ha9EGdBF16u0+h/WmDefpftZ6S8NqQZ2QdIYr/5zgvgCRF6qqGE4pIbYjzuh+4/tWPjsxP9j2n7rvwEtmuj6nJIxyJ26xHb2V6BbTcBu0a/tP6dQQF/Rs7njs0geG4d3rT3BuEACIHC3dL28+Ffed1w392hpLnaQqpCVKo5rCzJi8bJv3Gkkhok29HfOZ9SJHTj9yVmPkP/y0Gz3vHx+3P9HZU5kCESVnacCoNXJyCMc0sR7NJltPyUvSm9mq0IkWJeySJ9r17K7jJ+HSzT2lP67YJaZUmJ9rG2eJa89BWx3TtA6uOrGtpQwAQk80EaVRTflq0RZc/fIsvDQ1nMXlvWB+dg8cNa6B4eWnqymFXz43zXJ/WO6pVMHMjqNZu31aBx6nPADsNLl8GpYoBewSyQcxXNNGFk2xdWteaqk0zISVkBlFFfTkjg1x33ndDO4wwD0epf+c7paGcoDXp9BLf+41JFGv2Hivw07q0xClUU3Zslcp57BeXa84I/D6y7KYPWUX8M52SwNw9pvbB0SN+/WHlRUbYw/5udoxhFGDOgQq53WntkeDkgI88oseePnqfiizSDLUTvtFn5ZY+/BwQ6cYxrcXnSJAhKtObBtncRgqDlhIQIgpYbegsv6WeOmyvVhrXsuYPPKLHsbM9xT5p0RpCKGw59BRzF2/x7AtkQ5eK11uZ1Bku85I2D1leq8/7G8X9TDto7hjEsHq9MsGtAEA/F/fVmhcWoh8iym05u/Iq8snUcwuoDhr1EsVZfUgN1mjlgZzoFYEoHz3Q45tbLu/rLgWrjyhTfQ9mf6GRWanRgpZyyUvzMCSzcY1tew6eGZ7S8JNKVQHS8OpI7GPaWiKQJsxEzvO7CLSFpyy6wBP7NAA33sod2J1enmlMbkx3yKqX6i6h7SvSv+ZvEyZziF/Ex4KTNn+5mdEb124lqlxVRre5VKO92BpqIeseuhsLNuyHxOWbHM+IXqeuKeELMasMIB4V4DWYTDY9sebbHJfpsMuU27dOgKr2Ke5I9M6aSKgfUNj2Y6B7evHrc2hp48ueG51q1vVN87W0hSUnjqFxrGp387Nr2VSYLJ2zJNB/Iwz3C7t97P47dfzdffzeJfESe33FLbuEKVRQ/h+5Q4MeGgCDpmC0anE0dLQv9e9di8jkt1KA0gsGSsWADf+BeI7Wf3o/xd9Whr2RRi47jT7OMfNgztFX1dUGsu9PDGiZ9SK0MizCHJrSiPm8ontc7q2ht/7U2DKFTGvzqh/ZqyeHr0icHdP6c7zEq/wYmnoXicy0yzs0u6iNKop5nLkD3+5FFv3lWP5VusM6FTgFJewDXS7aI0sNzQAOLs47HbFBWp1nZG5X9IsDWal02pZT1fXiJWkvaUPDMPLV/eLu47elXTUVCMsz8IVVcvC0uhtmuqr74i7t6gbd3ycDMlaGgk8I7H7G9umzUIzHBfysL5Z3SJc1FtV9Ba/Ef3V3cr5B4UojWrKC98ap9qmKvHHGZu4hcPMeTdDItvdU4DzaNY++1f9C+Nfq/Y0l5HVeugP/vw4AGoOgUUQW9/SUZOlYfWt6RXJyR0bYukDw9BVneGjdeb6vs0uZtOjZUyZ+O0LG9Yxdu6jhxmnGpul/vymU3Byx4bR9/rLmUftH4w6EXcPPza2X29pBPQjM7dzhRrsdnrSuzYrjVpY4p4SgkE3y8ML363YEZ22GxTOgXD9e+uMcCuy3T1FRI4/cntLw0iOJ0vDeK9euLIvOusSAa2upR9Jmy0NK/QxjUd/eTwK83NxdvdmuO609hg9TOls9e4mu9Fxk9LC6Gu/MY0//My4sNI1J7czvDc/Ml2bl6JZ3dj17BUBo0+bejire7M42cJ8DLW1Ujo1ti9BorfYwh4gitKoYXh9ti9/cQaGP5nYcpler629j7C9rVHdlUaygfDYcbHX5vY0945mlJmtlFgj9u12blISZ2lY+c710121IG5+bg7uOOtY1FXzR/TyeUpI89kLmuMsjiQQPM4h69deudoim1uP+Ttv17A23vrNQIxRrUI7UrW0sCiNGkIiow9zZnGyOE2rTXT2VMR98JvxOCoNu+2mHWSzb9mDw6Ltm1158W3EX03bUrco33HdEw19INzuc3lxT5HN61ShdcBWSttg1UH/2p4vb4ktquQ+Iyt+2wkdGvhThiEiSkNIGXGWhoeEPdcqt1luaQCJjVbNnZmdpVGQlxsN5poVsKdktJyYlWK2NKzQu6fslEaD2gWuxxhdROGpDcvZUy6KwM595SRnl6be1izPBkRpZDmrtx/AnR8tSLcYnnCePWV3jrNSCGoxpHTiNKXUrh+Kjjqjk6fsO2s7v7vZsrC6lraJmV0tFcA4vZdsehd9xrp9FV/7GE2QNFaD5oZ7QbrZU26WRgLCuU2JTXTKLJvcj2EhGeFZzm/f/AFLtziv/QDEHqT9Ryqx93BFNLiWSpzqR9nGNFwGtx48JhmNWxkRO6fHKIf8BrMOMrunol+Dg4srJp/yl+EtfqR3N9l9qrq62lhOZeHd2gkCrQyK7bUtLp5jUNBBS5Q8kqchBMpVL83E8feNczwm1aN3RuKWRra7p9yq3Fp1Whf0bB6X9GUsnGc86YELjsPQbk3Qv119w3azsrIaNWvbIuxtEoVBaXgY8trlYKTCPdWzVZl7QUKLbcZAuGrF+QhCJxLT8EKqfgmiNGoA5ZVVvha7STT1obyyCt+v3GG7X9+/T1yyNSoTM9tm6brphGrhnkogEB5/nK6zNv2q2zWsjX9f0TcukOql7WOa1EHT0kL8aegxniwNvyPvVNVL8oPdpILY/uRcZ2F9YjsLMmhEaVRziAh/eGceVm8/6PkcP9NYmRnfrdgBZsaDny3Bpf+ZgcWb4utOmdu99tXZuu32oyQ3WX78aY9nWTMVxzwNjx2AfqTrNa8hbvaUxWm1C/Iw/c7BOLFjQ08z1fyW/PAy5ba0MBwvurfbZGF96XpNTekp2fYByZXm890QpVHNYWaMX7w1bvudHy3AkYoqizOcO+plpvjJ/+ZvxuUvzsCbM36KrqttZ9XYN2tfsNDN/fTRjxsd92cDfpPX9KNzK/+1Wz8cy9MIvnvRu5tKXFayAxwWmdJtf/3aAZbHNKoTX9YjUexnRMUfG3ZMIxHlw451FYJFlEYNwFwqGgD+O+MnvDd7veXxdv30+MVbMfTxKfhkbqyj3rhbqXG1ftehqLlg99DbPdTMMJgadtnh1RVn95T/HiRhS8PleG0w0c5UKVeP39XjvMyeMlfS1Zh+x2Bf1/J17ahijaE9in6LFDrRvlFt3GrKYPfTpuEjRH9/EggXkmDuhr04WG5d2ba8MoKV2w7gx592G7bbWRrLVUtiyeaYtaH9gJR+X02IspHFNi8D9gqlGpSWcsVvINyKRHSruWm3a5k7TbccBi/YurM8tJPs8qaOhRCjn9V5ym30+Wfvnb2+yX9f3gcjT2tvuz8RxD2VIcxauwt7D3kPJqcKt85i3vo9th1veWUEQx77Bj9/5nvTOXstj49Nv+S4bZEI6+aJWz+2TjOkbPfVAK3hNDJMpAPx7O7y2bY2mNCKElp9M/4tDX8yBImXYoOWgfAkZ3YZ3ItEgbkJZfZUBlFeWYVfPjcN17w6K92iBIpdWYhLXphuud26zERsW3TyRgLuKbY5LsLAM5NXWjdYA0hk9Oq1Iza3fUzTUtQpyMPQbk0sj9f0t1Ow228Zc7vPF6YuubhvKwBOqyJav9awKw6Z0EyqDImJ+EGUhge0WSOLNlmPwNNJMg+Il7IQllj0/UquRYLuKXaqS8V45Mtl/mWsYehvX6IxjZKCPCy4byhO7tTI7ipq++r5lm0GY2mE5Zefc/cQDO/RTL22B6VhOdGA4l77ytMwtZWBs44dEaXhg0yMySYjk5XSeHeWdXAciP2YDh6txKvfrwUzx1xWHH+cmQgz/vPtarxrCsA7zfvYbxOPyXbyLRYrssTjTCi3bQk0HYdmaTi5oHzHGRIPaSQEEUXdbJ6mB1taGtavEyGH4i+RqMJM1aQRKSNSgzGvj7B0yz7c/sF82+O1R/mN6T8BAFqUxVaAYzB+cMmZqIwwHhy7JG67ufbUy1PXRl9f9/ocxzazlXevOyEulmRFIt2H93Lq/lrXOiUnSyYo95TG4xf39NWe+/VisRnHDt+jNWWseOtDCN35QVtV4p7KALSHLAMNjaQwWxp+3VUHj1Yakps07H4EFVXWd7AywpiyfLuna5YVp75mVhiY79FnN56MN38dy0fQ1qXQjmtVvwhWJKRU1LP8di7RmIbWoVqNwn32KG4B6KBzD4hi7mbbEiaGjG/nm9SvXX2ce3xzPHxhj4Q76zhLI7FmJBCeSWSyskjmR/W2yRVlNeqbvnonzvznN9h7uMJyhGk5ErO5nl3g/dnJK3Hre/PcxAUA1AkpOzjVmO/RcS3qol5xrej7GXcOxvQ7BkePu7BXS8tzk3k2/bunlKs5uaB8Jyq6bA/a40KgaMKoUSEkFpDPzyX865JeaGvKXbFaOteKnJz4mEbCtae02YtSsNAIEQ0jomVEtJKIRqfimtG8hUzWHgFg9bCOeH46lm89gO9W7HD0n+v9qbtsFm8qr7TOQF+745BnGWvXyhyl0dom6cwLbrGIerVroanNEqRB4bdNDiGmYR+M9tZOr9Zlvq5HObFnVbdelDFhj2CZp+E2k0rjwt4tMOYC+1X29J16DsV/1mS/anFP6SCiXABPAzgLQFcAlxBR17Cvm4kB8DB4aeoa231WCX83vz03GsfQ79XXldKzftfhZMQDoCwdmimY15cAgD5t6nk6N6jRYCKtxKxTf2eHYWnUdik14vbbe8OmxIgdhFg5fb2sxbWsV8XTfxwrWQy2inrwMU3qeLY0LKexJxoIT+gs/2TOL9Ab/QGsZObVzHwUwNsAzg/7ojWhlAUAfPiDfR0nuzvwv3mbwhHGBr8F8cLESml8MOpEXDagteu5fmc92eYzqJsTSYL0eysjHgLhftu061yj7imX892UTly7FHNP6Z+lYhsL1u3jWHXwbjInklPjRiq7qGxTGi0A6B3xG9RtBohoJBHNJqLZ27d7C7A6kYovZMnmfXHlPDKJSIQdR0DJ3CM/A6u8DFIalTYd9eGj1m44N6wUw8Y9inW262C55TlaB55I4rz/2VPu5yVb2kMjrMEBwXoWWO2C3PgD9X9tMNahUnD7LVhZJ4b9Pj66Ibk2RZoj25SG1e2Mu1PM/Dwz92Xmvo0a2SUqeSc2eyq8L+WsJ741TMGcu34Pvlq0BfuPVGDpFqXUeHllVfR1qnErUf769HUpkSOoTikI7O7JYZvqwW5YdRZz1ikDiY/nWlt0MaXh/9n0eyfrFOabzgvOtWLmjrO64NIBrXHu8c0CaU+DKGYh6lNl9JaGWxVhY3vun/fVa/rb7gvycY5VZJBAuJ4NAFrp3rcEELp/JJXOqfW7lKDwz5+Ziuten4NLXpiOYY9/CwC4/3+LMezxb7F5rzL6jETYVxA5GT78YSPGjF0cSttelqvVSKWlUc9leq+VewoArj6xrWvbfn/XdjPPYut/x8tSx8V1YyfDmV2ty4i8d/0JuPfcrijw6K9PhgYlBXjo591RkGcda0gUAummDsduwLk9muuO0R3v43tqUU+ZFt20rrFke9dmpUYZDEF3bzMS/RD2LyTblMYsAJ2IqB0R1QIwAsCnYV80ammkQHuc8sgkdVEj5f3CjfvUa8eS57TZSU9NWhmXoBcW363ckREVZ/NSGAjv27a+4/5Km3s/oH0DXNCzueW+41uV4c1fD7Cp4+VwLZubr+lQ8+53Rg7E+D+c5tCi/Si6SWmh5fZ2DWvj6pPaRScjpLP8RXuH8uxOKHka8TGNusX5eOnqvtH3I/opY9P2jayv06ZB/My5ywe0wX+u7IsLesZ5zG2xHAMleGN/P6QzjmtRitOOSd674kTmzF/0ADNXEtHvAHwFIBfAS8y8KPwLB99kVYRxpKLKMpB3yzs/xm2LcGyKoJacNHPNruAFy3DCsDQ6NS7Bim0H4rY3dlnk56BD7MLukblhUAec1LEhlmyOdzM69RV2Ckrr+MyLVQ1o38C2rUST+zTuO78bGpcWYHCXxok1AGW20qEEYz9/PqcrLu3vPtnAjugsMIcbcGHvlriwt5Ib8/lNp+DsJ7817H/v+hPiVqjMySEMsbHS7AjS0ujYuASf3XhKgmd7J9ssDTDz58zcmZk7MPOYVFxTG8UFqTtufXcuut3zleW+HQfi8xwizNGH/NynvrN1V1R3wohp2H2vt555TOJtBjzQsLPyEolpaJn/id7LhiUFuOfcbo5W36UuM8imjR6c8CJKZUX5KLKZIutGDhF+1rUJWtcvxm9ONa5jYZeH0rg0fvDQuE4hBh2TmNI052kAwLHNSmMrKmZO2M6SrLI00oUWAA9ydoJdYNOOqggbHub9RypTtrxjJhGGpWH3vdavXStuW1lxPvao66qc0qkhvl2xI7rvpjM6xtp0uaZ1x+D/s0XdUz58hx0bl2DLviOOK/B9ecspqLQp++LG2oeHux5TtzgfdZFYSZiksuBJiZdMuf30uH1ajCqVcy00S+OLm0/Bok17cd5TU3FGEhZcKsg6SyPVvDJ1DU7469eW+w6UV2LtjoMpkSPCHF38BtCmDqbk0hmFnW8/Gfzcx2mjY6PjHi3rRl/3b1cff9BZJnaKSOskrJO6vMuhoQ0k/NyWpy/tja9uORWF+faj9S5NS3Fci7q2+8OgUZ0CdGhUG6PP6uLrvCm3xSsAO5xusVWAXE+iT55ZCZ2uUwr6fd2a18Wqh85Gs7rWdcYyBVEaLjwzeVX0tfmhuew/MzDo0ckpkaMqwoZicEF3nkFaUb1bl+Gx/zs+sPb6tY1lWSe8BogDflw7erfIjWd0iq7NYH44ErE0EhngUgLuqbrF+TimaZ0ErhYOowZ1QF4OYdZdQzDx1kG4/rQOjsebn9XWFkFpO5ymo9olLyZreDQoMbq39FUD/GbQ25HK8aMoDRecvtN56/ck3b7XzloJhMeEqYrYL5GaCH/5JLj5BDlE6NW6nuW+ywf6D2Cer5uN0rws+VHYvecaK894vY3T7jjD8L4wPxdXDGzj69rmharcrq2Vt3ju8j740zBlBK4fievXqM5W/jSsC1Y+dLbn48Mq0uilTIpfzj3eehZdVJ4Mj19YIUrDB2H8ML0aDJEIG0YllZFIQglddiSbnPfIRT2ir3OIUKS6Psw1fS5JYNaL3o3SqKRWwtMtNeqbRn5eb6OV28D2N2/T5gkdlFlN1mVE7HuQTk1KMGpQB6x9eDga66bEas+EXc5IdeSYJolbSU6dtHYLzcdoSY36mJVX3AaFSdcgS4PSEaXhgt2Xap5up2fz3sP4/Ttzbau66vFqaVSxMRCeaJDSjmRLjhfryzAQoglg+mmNU0efgW7N3f3ki+8fanivTybr3rIMeV5XvfNIYb6/n8Fr1/THU5f2cjxGU+jDezTDTYM7AQCa1y2MdkAaZPNaQ3s87NwYOdGYRs1QGrPvHoLjW5UlfL6jeypibWnUysvB2oeH4+qT2gVyHT3JGjXn9miO4T2a4bahic/084soDRfsvtTPF2y2PeeeTxbhox83YtLSba7t+7E09DOHjlZFAvVj7j+S3LKq+uqzORRTIpcOaB1bUMhjW2ZFrW/7Z12bBFLptm5RrPMu8llu/dTOjXBOD2e3g8bw7s3w816Ke834ffkLhNvtiiX31Qyl0bDEOnemgTrTrbfPUul6YuVFUjd8TzamUVQrF09f2ts2ITMMRGm4YB4x/PVzZblSczKVxpa9R6JlMex+x0d0tYm8TpuNsPEBu/XdeWlb3+PEDvGJY/o1rwmEgrxcLH1gGP40rEvUUtDE79S4BABQUpCH964/Ia4t8+9IG/n9fkhnAMBFvVuaTzEwsL1zJjcAfDAqdt3aCc7512P+Hg0rGVocH1uZzlu7dp2L1sFFambaTpRJtw3CV7eciv/+ZmDCbWiKN4jaTV49CBLTqAH8e8pqAPbz4n/1yiz8pNaPMh9RXlmFW97+EV3+/GV0m9cBYhWzwS2zYONe70IHzItX9UMX0+wb/ehf+yEU5uciJ4ei5a81C+LxET0BKD/SfqZSHVZmdmlhHtY+PBw3D1HcPL86qS2WPTjMVr63R8YUwtIH4o8jAO0blkTf262l4IUS1a1nDtDHOnvjda1ex7bFb7Xzs2skU7CwOlFaqMwIc5pG7IantcMDJuzigmEgSsOGKcu3o+3osdhpUZL6hSmrsb/c2p2jLw9h/h1/t2JHXFKf1x/7oo1740ab6Uju++uF3VFUKzeuo9UvVTr4WGMpBc3SqFSHw1q+idVnv+F0Y7DxiRE90b+dUbEQkW0hu46NY8qgZb0i205Efys76wKrL1/dz/J4O7o1r4tnLuuNh37e3bA99tHil/NUru/RPaXFNGx6suh6GjVbZwSCdg+DmD2VjcrAK5IRbsPL6ip2Ryri7f4xqovKDS+dutcB4sjX56CFaTQ7a+1ubycHRGlhXnT2k1mBtW5QjNev7Y/2jUrQvK7Rv1qgdtzm8hV2n12L3Vx7cjvDdFs3vh99BkrVWMXSB4Y5+ov1P+qbBneK5uOcnkA27tnd48t36z+a0wwZ73Eeaxqp9bFalKXOp11d0WIa1bnDDwJRGjYEMdowj/6sOskIs+fpktqCPJlAsanQYg4RTulkXV1zRL9W+OsXS6NlOTSlYNep5+XmYPmDZxniJF7Qu4i8uiluG3qM47FNSwuxZd8RX3IA+llPySXyucU0TuvcCM9f0SchZScYsVo7XIhHbo8Nfmc17D9SES1ZrnHTWz9it26blWpgAF8stJ+Jlak8+ose+N3pHXXxCntGntoeK8achTLVhaW5pfqpbqdRg+IzgGvl5YQ24tO+h7UPD49zh5mZeKtzeXG3q9h9BqtAuNWheuVj3Q7hzG5NM2rt9EzEbo0QPbHaU8E9d9XRayiWhg1+LY3u946z3L56x0H0UUfYVjMqOJL48qDppHFpIf449Bi8NHUNjsL5x0FEBquhXcPaeOSiHjire1MASkbws7pyLV6Zf++Z4Ahw/P3W994P74wcaJltnqjF6b7kp1VMI34bO+wTvPP0Zb1dS7G71Z4KkvxcQkXAuVapwvPwhIiKiCh1GSRpxs8axXPWJb6uxafzN2FfkjkSAPDwhd3dD/KIPofBjZM6NgTgr/osEeH/+rWKS3TzS2lhPuq6rK7nlQHtG6BV/fgaRon2H9HO3qYNry6raNkR0RlJkZ+b4/pc29WeSga7lsb9/jQ8eYlzgmim4snSIKJzATwKoBaAdkTUE8D9zHxeiLKlFT8JPjst1r+wwmpc8eePF3q+jhP65J6ercowN4C6WF54ckQvbNh9KKmpjqnGT5eQaJkHfWefjJWgPTOpGP1mMiNPbR/6GjKRFMY02jWs7ViaPpPx6p66F0B/AJMBgJnnElHbcETKDIIqWqb91ues24XrXp8TSJvWF4q/ZsJNOfjPzRTVykWnJGoBBcEHo07A/A3e81a6Ni91P0glaUvD5XxD7kYCMY2awp1nHxv6NTSd5MfLUBPxqlMrmTl92WRpIOiB3QtT1gTW1je3DYrbphe3puV59WlTH7/yWBdo5Ziz0KFRifuBKok+Btp3YGepWAbCnabm1nBLIxVolQq6NvM+qHCjOv4UvVoaC4noUgC5RNQJwE0Avg9PrPTjx0c/ZcV2230HjlRi/oY9AUgUo02D7DRrMwGnJUqtSLSz1uX2WU6A8Jzc52GfEAxDujbBuN+fakj2TJTq/HV5/QXdCKAbgHIA/wWwF8AtIcmUEfhxT70x/Sfbfb998wec99RUHA3IH3tC+/i6T4CxE0raPaX+vXu40SUQ5EJN2ULiloZx3QzA9B35bK+mxzRSRRAKo7rjamkQUS6AT5l5CIC7whcpMwjKHXBALTeiL1IYBmF0KVq2cU0m+fiQdQNW6tfpUuJmzy6q8/DK1dJg5ioAh4gotQsGp5mglxU9HJDSsA9SB9K82hYZ/tZkEnZPRWMa+m0WbirbN0bE0hAyBa8xjSMAFhDReAAHtY3MfFMoUmUAQXtiwk7gy03C9WFGO7+mjm57tipDk9LkrCyGtym3XgPhQnZRnb9Jr0pjrPq/xmAu/Z0sYbun9DGYoPRdTR3dfnzDSUm3oZ895WpheKCmfhdC5uFJaTDzq0RUC0BnddMyZq4IT6z0E/T6BGt3Hgq0PTP5ecFnJJktjWx1V4296WQMf/K7lF4zqjQM+TPO989pd021+rIVLR6Y7Hr2mYinnoaIBgFYAeBpAM8AWE5Ep4YnVvrJtvUJ8nN0iyD5PPeXfaxXwstWJWHGy7rkbnRo5O/HH3VP+TjH6djq8l3UFAa2b4A3fz0AN6vrw1cnvLqn/gHgTGZeBgBE1BnAWwD6hCVYuknHAkfJoF/Vz4/kvVuXoYspmUnrn8QlovD+9Sf4LvkQNVRtbqHl7CmH+y2WRvah1WWrbnhVGvmawgAAZl5ORMFUistQsi0lQV9F1o/sDKt+TVvvIlmp/GFeZCpT6NvWfc1xMzGdEVQ5GtEaQmbgVWnMJqIXAbyuvr8MQIiFlNKP3RrgmUqezj3lJvmVJ7TBa9PWKcdyaqbxurHwvqG+svCzBf091AfErT5p9fv0QnXEa/R0FIBFUMqH3AxgMYDrwxIqE8gulWF0T7mZGveffxy+vOUUAMrKdHadlXl0G2afXlKQl1WVcl2xyNNwQ4wJIRvwamnkAXiCmR8Dolni1TpdOOjZU0FhJ5Z+5TYvkndpWoonRvTE6V0a44M5GyyPMZeHr6euvCe4E4mWRteXd3GZPWWhYl67pj8+mbspWOEEIQm8Ko2JAIYAOKC+LwIwDsCJYQiVCWSZd8qYp+FR9vN7trDcXlwrN65NANE1vgXv2OkJr4/XqZ0b4dTO1muvC0I68Ko0CplZUxhg5gNEFL/MWTWCmUGUeQFxu05IbxX4nfll/owvXd0Pny/YjFb1Yl9x3zb18FCAqwNWd/STp2rXUn5m/du5BNTFPSVkAV6VxkEi6s3MPwAAEfUFcDg8sdIPszLltCrTtIYNOQlYGna0rl+MmwZ3wvpdsYTE167tj+JasqS8VyK6lfvq1a6F8b8/1bCcrGUgXJSGkAV47QVuAfAeEW2CMohqDuDisITKBCLMyCEg3OIfwVG7ViyI7FdpmA/XOi9DNrMMgxNEuW9eVjfMkvGJUMNxnD1FRP2IqCkzzwLQBcA7ACoBfAkguKXoMpAIZ1dHmZebg89uPBlA8jO/tE8tuQGJk4gCKCnIw2mdG+H1a/sHL5AgBITblNt/Aziqvj4BwJ1QSonsBvB8ohclor8T0VIimk9EHxFRmW7fHUS0koiWEdFQ3fY+RLRA3fckhdyjMTjr3AXRJUSTHLJqmeBu61cL9mjfgN00Ze1+6icb5OYQXr2mP07pJIFvIXNxUxq5zLxLfX0xgOeZ+QNm/jOAjklcdzyA45i5B4DlAO4AACLqCmAElFUChwF4Rp3eCwDPAhgJoJP6f1gS13dFi2lkE8FlHxv/CglgMeVWT9PSQtxwege8do1YFUJ24ao0iEiLewwG8LVuX8JRUWYex8yV6tvpALSKeecDeJuZy5l5DYCVAPoTUTMApcw8jZVh9GsALkj0+l6IRDhulHjdae3DvGTSxCyNZNvRyoiI1kgUbR33kgLrhEUiwm1Du6B9o5JUiiUISeOmNN4C8A0RfQJlttS3AEBEHaGsEx4E1wD4Qn3dAsB63b4N6rYW6mvzdkuIaCQRzSai2du3b09IqAjHjxLDjnGMvenkpM5PtI+3c2eJeypx/nphd7x4VV90bCxrTgvVC0drgZnHENFEAM0AjONY75ID4Eanc4loAoCmFrvuYuZP1GPughJYf1M7zUoMh+12cj8PNebSt2/fhMbdVjGNMMtodGxcgoYlySXZa0otsAq9MnsqYWoX5GHwsU3SLYYgBI6ri4mZp1tsW+7hvCFO+4noKgDnABisU0YbALTSHdYSwCZ1e0uL7aFhFdMIc7Q94Q+nYdu+I67HaXfqioFt8Pr0dYZ9mnxhZLOLpSEIAuC9YGGgENEwAH8CcB4z65e0+xTACCIqIKJ2UALeM5l5M4D9RDRQnTV1JYBPwpQxwlaWRsg9p4/mH7jguOhrrfig/vR3Rg7EOT2aJSeP5A0IgmAiXSm+T0EpeDhejRtMZ+brmXkREb0LpYpuJYAbmFnLrxsF4BUoda++QCwOEgqWlkaYF4Q3F5BepGcu641GdQrQpWmpYR8zY0D7Bpi4dFuAsgmCIKRJaTCz7XRdZh4DYIzF9tkAjos/Ixy0jHA9YSe7+W3+7O5mS0KNafjNCPdwvCT6CYIApMk9lQ0otYOCj2k0rlOAa05qZ7kv2eaD7tf1ukRUhiAIgCgNWyqq2LCEKhDMDKKZdw3BX87tarnPbjR/fMu6GPNzdyNLO1tCEYIghIUoDRsqqyKGhY2A8NfMtmu+TmE+2qrJYk40LytCfi7hj2ce49ieV/RuK/FOCYIApC8QnvFURNi4hCrC7ziTXau7MD8XK8acHZg8+rpIEtMQBAEQS8OWyqoI8nOMtyf0QHjQkQOPzdklAzaqU61X9BUEIQFEadhQWZV6S8Ouk5dRviAImYIoDRsU95TJ0gjQEigtzIsrG5Iu3SCL/wiC4BWJadiguKeMvXiQgfAf/3ImAKDDnZ9Ht9k1n+hlpV6UIAhBI5aGDWG7p3JzyBBoBmI1o/JyCCUFwerz24cdE2h7giDUTERp2FARsZpyG/LIXVUazcuK8PbIgdHNyV729mHH4LeD7NfMEu+UIAheEfeUDZVVjLywEzNM1C3Ox73ndsXgY5tg7+GK6PYg8y0EQRCSQZSGDRVVkfhAeAoi1VerJUb2bUp+jatExF31UHB5HoIgVD/EPWVDZcSqjEhyfHHzKZ6PTVdinYTOBUFwQiwNG+4/rxuKC/Lw+YIt0W3J9t0dG3tfDzrVrjENSQkRBMEJURo2nNixYeBt+gmk5+qy0cPux401pqqn1jilU8O0KWJBqE6I0vBBsl2Onz4rz+CeSux6Xk8LbE3xDOb1awekWwRBqBZITCOF+BnFm3M4BEEQMgFRGj5IpevG6EoRBSIIQmYg7ikfhKEznr+iD5qUFsZtzw3APaXBkqghCEJAiNJIM2d2a2q5PS+AQPivTmqHeRv24NIBbRyPE50iCIJXRGn4IJVOInPdq0RoVKcAb/56oPuBgiAIHhGlkaGEEQifdscZOHy0KvB2BUGoOYjS8EOaAuFBXbZZ3SLH/V2blQZzIUEQqi0ye8oH5r5b35nfNLhToNcyBMJBaFZXCZYPaNcg0OvoOe2YRqG1LQhC9UCURhJcrgswd21WihH9WgXWtn56LxHQvlEJptx2Om48w77EuSAIQtiI0vCBm5vo+FZloV6/dYNi5IQQ69Cm5Eo2iCAIbojS8IHb8qnZPnW1mpadEgQhQERp+MBpkE8UXg0n6cwFQcgURGl45JnLeru6hsKyNNwsHEEQhFQhSsMjZ3dv5th1E0Jcazvspcmz3K0mCELqEKXhA/N6GHp3VA5R1ve+VhZNr9ZlqRdEEISMRZL7fJBjo2KLa+Xi9C6N8eaMdaFcN53Oqbd+MxCHJItcEAQVURo+sFt5b/RZXZCbQ9luaFhSmJ+LwvzcdIshCEKGIO4pH9gpDW1rWCXIq+sSrIIgZB+iNHzg1neHZWiEvka4dh3RTYIguCBKwwd2loZGtrqnNLlFZwiC4EZalQYR/ZGImIga6rbdQUQriWgZEQ3Vbe9DRAvUfU9SGnw2bhU8zDqjqcWKfIkgFoAgCJlC2pQGEbUC8DMAP+m2dQUwAkA3AMMAPENEWhT2WQAjAXRS/w9LqcCIjy2YLQtZVlUQhOpOOi2NfwK4HcYB+vkA3mbmcmZeA2AlgP5E1AxAKTNPY6Vnfg3ABakQsm+beigrzgfg7p4KCzE0BEHIFNIy5ZaIzgOwkZnnmUbvLQBM173foG6rUF+bt9u1PxKKVYLWrVsnJev7o06MvnZ1T4VVRiRkZRVWzSxBEKofoSkNIpoAoKnFrrsA3AngTKvTLLaxw3ZLmPl5AM8DQN++fQPrEW0tDXV71ne+EjwRBMGF0JQGMw+x2k5E3QG0A6BZGS0B/EBE/aFYEPqVjFoC2KRub2mxPaXY9qmqiWG2NILqg8Puyod2a4rHJ6zA0G5NQr6SIAjZTspjGsy8gJkbM3NbZm4LRSH0ZuYtAD4FMIKICoioHZSA90xm3gxgPxENVGdNXQngk1TL7jrlNkVyBM2xzUqx9uHh6Na8brpFEQQhw8moMiLMvIiI3gWwGEAlgBuYWSt8NArAKwCKAHyh/k8pru6pbM3uEwRB8EjalYZqbejfjwEwxuK42QCOS5FYlrjnadhrjeZ1E8/ZkPU0BEHIFNKuNLIJt1lMdpbG5D8OQuv6xSFIJAiCkFqkjIgPzJaGV29Ubg65rvrnhExqEgQhUxCl4YO8XKX37tCoNqbcdnrcfnNGuPT1giBUN8Q95YNererhulPb46oT26J5WVGcUogrKxLQdUX5CIKQKYjS8EFODuGOs4+NvjcrBTslkax7SdxTgiBkCuKeCpC45L70iCEIghAaojQCpFGdAsvtydaOkim3giBkCqI0AkDr0kf0a4UHzu8Wt19KpguCUF0QpREgOTmE4T2aB96uxDQEQcgURGkEjFX/nrR7SpSGIAgZgiiNJPDqdRL3lCAI1QVRGgGjtwqCWzxJTA1BEDIDURpJYKUTZKaTIAjVGVEaSZAqr5PENARByBREaQRNCB286AxBEDIFURoBE4ZVIJaGIAiZgiiNANB36lar+8nkKUEQqguiNAKmKD833SIIgiCEhiiNgMlNYrElO2RGliAImYIojSQoK84HANSuFW6FeYlpCIKQKch6Gklw8+BOaFynAOcdH3y9KT2iMwRByBREaSRBYX4ufnVSu3SLIQiCkDLEPSUIgiB4RpRGFhBcDStBEITkEKUhCIIgeEaURhYghoYgCJmCKI0QGXxsYwBAnUKZbyAIQvVAerMQ+cs5XTFqUAeUFddKqh1J7hMEIVMQSyNE8nJz0KxuUbrFEARBCAxRGiHw1wu742ddmyTdztndmwKQmIYgCJmDKI0QuKR/a7xwZd+k2+nVqh4AyQgXBCFzEKWRwTCkprogCJmFKI0sQNxTgiBkCqI0Mpi8HOXryc+Vr0kQhMxAptxmMJcOaI2t+47ghtM7plsUQRAEAKI0MprC/Fzccfax6RZDEAQhStr8HkR0IxEtI6JFRPSIbvsdRLRS3TdUt70PES1Q9z1JUsVPEAQh5aTF0iCi0wGcD6AHM5cTUWN1e1cAIwB0A9AcwAQi6szMVQCeBTASwHQAnwMYBuCLdMgvCIJQU0mXpTEKwMPMXA4AzLxN3X4+gLeZuZyZ1wBYCaA/ETUDUMrM05iZAbwG4II0yC0IglCjSZfS6AzgFCKaQUTfEFE/dXsLAOt1x21Qt7VQX5u3W0JEI4loNhHN3r59e8CiC4Ig1FxCc08R0QQATS123aVetx6AgQD6AXiXiNrDOvmZHbZbwszPA3geAPr27SsZcoIgCAERmtJg5iF2+4hoFIAPVVfTTCKKAGgIxYJopTu0JYBN6vaWFtsFQRCEFJIu99THAM4AACLqDKAWgB0APgUwgogKiKgdgE4AZjLzZgD7iWigOmvqSgCfpEVyQRCEGky68jReAvASES0EcBTAVarVsYiI3gWwGEAlgBvUmVOAEjx/BUARlFlTMnNKEAQhxZDSV1dfiGg7gHUJnt4QigWUrWSz/NksOyDyp5Nslh3IHPnbMHMj88ZqrzSSgYhmM3PyNc7TRDbLn82yAyJ/Oslm2YHMl18q4QmCIAieEaUhCIIgeEaUhjPPp1uAJMlm+bNZdkDkTyfZLDuQ4fJLTEMQBEHwjFgagiAIgmdEaQiCIAieEaVhARENU9fzWElEo9MtjxVE1IqIJhHREnVNkpvV7fWJaDwRrVD/1tOdY7lWSbogolwi+pGIPlPfZ5PsZUT0PhEtVb+DE7JM/t+rz81CInqLiAozWX4ieomItqkJwdo23/KmY10eG9n/rj4784noIyIqy0TZLWFm+a/7DyAXwCoA7aGUN5kHoGu65bKQsxmA3urrOgCWA+gK4BEAo9XtowH8TX3dVf0sBQDaqZ8xN82f4Q8A/gvgM/V9Nsn+KoBfq69rASjLFvmhVIheA6BIff8ugKszWX4ApwLoDWChbptveQHMBHAClCKoXwA4K02ynwkgT339t0yV3eq/WBrx9AewkplXM/NRAG9DWecjo2Dmzcz8g/p6P4AlUDqD86F0aFD/XqC+tlyrJKVC6yCilgCGA/iPbnO2yF4KpSN4EQCY+Sgz70GWyK+SB6CIiPIAFEMpAJqx8jPzFAC7TJt9yZuudXmsZGfmccxcqb6djlhB1oyS3QpRGvHYremRsRBRWwC9AMwA0ISVAo9Q/zZWD8u0z/U4gNsBRHTbskX29gC2A3hZda/9h4hqI0vkZ+aNAB4F8BOAzQD2MvM4ZIn8OvzK62tdnhRyDWK19DJedlEa8fhauyPdEFEJgA8A3MLM+5wOtdiWls9FROcA2MbMc7yeYrEtnd9JHhR3w7PM3AvAQSjuETsySn7V938+FPdHcwC1iehyp1MstmXsbwIBrcuTCojoLijFWd/UNlkcllGyi9KIx25Nj4yDiPKhKIw3mflDdfNW1ZSF+ldbSjeTPtdJAM4jorVQ3H9nENEbyA7ZAUWeDcw8Q33/PhQlki3yDwGwhpm3M3MFgA8BnIjskV/Dr7wZtS4PEV0F4BwAl6kuJyALZBelEc8sAJ2IqB0R1QIwAso6HxmFOnPiRQBLmPkx3a5PAVylvr4KsXVHLNcqSZW8epj5DmZuycxtodzfr5n5cmSB7ADAzFsArCeiY9RNg6GU888K+aG4pQYSUbH6HA2GEhPLFvk1fMnLGbQuDxENA/AnAOcx8yHdroyXPS2zNzL9P4CzocxGWgXgrnTLYyPjyVDM0/kA5qr/zwbQAMBEACvUv/V159ylfqZlSNPMC4vPMQix2VNZIzuAngBmq/f/YyjLF2eT/PcBWApgIYDXoczWyVj5AbwFJf5SAWXUfW0i8gLoq37mVQCegloVIw2yr4QSu9B+u89louxW/6WMiCAIguAZcU8JgiAInhGlIQiCIHhGlIYgCILgGVEagiAIgmdEaQiCIAieEaUhCDYQURURzdX9d6x4TETXE9GVAVx3LRE1TOC8oUR0LxHVI6LPk5VDEKzIS7cAgpDBHGbmnl4PZubnQpTFC6cAmASlmOLUNMsiVFNEaQiCT9TyJ+8AOF3ddCkzrySiewEcYOZHiegmANdDqSu0mJlHEFF9AC9BKXh4CMBIZp5PRA2gJIA1gpJpTbprXQ7gJijl12cA+C0zV5nkuRjAHWq75wNoAmAfEQ1g5vPCuAdCzUXcU4JgT5HJPXWxbt8+Zu4PJTP3cYtzRwPoxcw9oCgPQMnC/lHddieU8tYAcA+A71gpfvgpgNYAQETHArgYwEmqxVMF4DLzhZj5HcTWa+gOJWu4lygMIQzE0hAEe5zcU2/p/v7TYv98AG8S0cdQyowASumXiwCAmb8mogZEVBeKO+lCdftYItqtHj8YQB8As9RF2ooQK8pnphOU8hIAUMzKGiuCEDiiNAQhMdjmtcZwKMrgPAB/JqJucC5vbdUGAXiVme9wEoSIZgNoCCCPiBYDaEZEcwHcyMzfOn4KQfCJuKcEITEu1v2dpt9BRDkAWjHzJCgLTZUBKAEwBap7iYgGAdjByhoo+u1nQSl+CChF+H5BRI3VffWJqI1ZEGbuC2AslHjGI1CKbPYUhSGEgVgagmBPkTpi1/iSmbVptwVENAPKwOsS03m5AN5QXU8E4J/MvEcNlL9MRPOhBMK1st73AXiLiH4A8A2U0uVg5sVEdDeAcaoiqgBwA4B1FrL2hhIw/y2Axyz2C0IgSJVbQfCJOnuqLzPvSLcsgpBqxD0lCIIgeEYsDUEQBMEzYmkIgiAInhGlIQiCIHhGlIYgCILgGVEagiAIgmdEaQiCIAie+X93UElVDqyWtQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# plot the scores\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "plt.plot(np.arange(len(scores)), scores)\n", "plt.ylabel('Score')\n", "plt.xlabel('Episode #')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "ebb450a9-c530-4f10-b629-53a2d69860c2", "metadata": {}, "source": [ "### Animate it with Video" ] }, { "cell_type": "code", "execution_count": 13, "id": "c9806a31-f777-468e-8987-0139708ef532", "metadata": {}, "outputs": [], "source": [ "def show_video(env_name):\n", " mp4list = glob.glob('video/*.mp4')\n", " if len(mp4list) > 0:\n", " mp4 = 'video/{}.mp4'.format(env_name)\n", " video = io.open(mp4, 'r+b').read()\n", " encoded = base64.b64encode(video)\n", " display.display(HTML(data=''''''.format(encoded.decode('ascii'))))\n", " else:\n", " print(\"Could not find video\")\n", " \n", "def show_video_of_model(agent, env_name):\n", " env = gym.make(env_name)\n", " vid = video_recorder.VideoRecorder(env, path=\"video/{}.mp4\".format(env_name))\n", " agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))\n", " state = env.reset()\n", " done = False\n", " while not done:\n", " frame = env.render(mode='rgb_array')\n", " vid.capture_frame()\n", " \n", " action = agent.act(state)\n", "\n", " state, reward, done, _ = env.step(action) \n", " env.close()" ] }, { "cell_type": "code", "execution_count": 17, "id": "4beb1d45-ab9a-4fd7-a769-017bbf1672e1", "metadata": {}, "outputs": [], "source": [ "agent = Agent(state_size=8, action_size=4, seed=0)\n", "show_video_of_model(agent, 'LunarLander-v2')" ] }, { "cell_type": "code", "execution_count": 18, "id": "f54ee0a0-265b-4161-bbb6-25e28543aa2e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_video('LunarLander-v2')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }