{ "cells": [ { "cell_type": "markdown", "id": "3e646804", "metadata": {}, "source": [ "# Cross-Entropy Methods (CEM) on MountainCarContinuous-v0\n", "\n", "> In this post, We will take a hands-on-lab of Cross-Entropy Methods (CEM for short) on openAI gym MountainCarContinuous-v0 environment. This is the coding exercise from udacity Deep Reinforcement Learning Nanodegree.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Reinforcement_Learning, PyTorch, Udacity]\n", "- image: images/MountainCarContinuous-v0.gif" ] }, { "cell_type": "markdown", "id": "b6e1d4f5", "metadata": {}, "source": [ "## Cross-Entropy Methods (CEM)\n", "---\n", "In this notebook, you will implement CEM on OpenAI Gym's MountainCarContinuous-v0 environment. For summary, The **cross-entropy method** is sort of Black box optimization and it iteratively suggests a small number of neighboring policies, and uses a small percentage of the best performing policies to calculate a new estimate.\n", "\n", "### Import the Necessary Packages" ] }, { "cell_type": "code", "execution_count": 1, "id": "063a557c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\numpy\\_distributor_init.py:32: UserWarning: loaded more than 1 DLL from .libs:\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\numpy\\.libs\\libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll\n", "C:\\Users\\kcsgo\\anaconda3\\lib\\site-packages\\numpy\\.libs\\libopenblas.WCDJNK7YVMPZQ2ME2ZZHJJRJ3JIKNDB7.gfortran-win_amd64.dll\n", " stacklevel=1)\n" ] } ], "source": [ "import gym\n", "import math\n", "import numpy as np\n", "from collections import deque\n", "import matplotlib.pyplot as plt\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "from torch.autograd import Variable\n", "\n", "import base64, io\n", "\n", "# For visualization\n", "from gym.wrappers.monitoring import video_recorder\n", "from IPython.display import HTML\n", "from IPython import display \n", "import glob" ] }, { "cell_type": "code", "execution_count": 2, "id": "de727785", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "device(type='cuda', index=0)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", "device" ] }, { "cell_type": "markdown", "id": "67faf368", "metadata": {}, "source": [ "### Instantiate the Environment and Agent\n", "\n", "MountainCar environment has two types: Discrete and Continuous. In this notebook, we used Continuous version of MountainCar. That is, we can move the car to the left (or right) precisely." ] }, { "cell_type": "code", "execution_count": 3, "id": "c6c02db1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "observation space: Box(-1.2000000476837158, 0.6000000238418579, (2,), float32)\n", "action space: Box(-1.0, 1.0, (1,), float32)\n", " - low: [-1.]\n", " - high: [1.]\n" ] } ], "source": [ "env = gym.make('MountainCarContinuous-v0')\n", "env.seed(101)\n", "np.random.seed(101)\n", "\n", "print('observation space:', env.observation_space)\n", "print('action space:', env.action_space)\n", "print(' - low:', env.action_space.low)\n", "print(' - high:', env.action_space.high)" ] }, { "cell_type": "markdown", "id": "d6ebff61", "metadata": {}, "source": [ "### Define Agent " ] }, { "cell_type": "code", "execution_count": 4, "id": "8a769b19", "metadata": {}, "outputs": [], "source": [ "class Agent(nn.Module):\n", " def __init__(self, env, h_size=16):\n", " super(Agent, self).__init__()\n", " self.env = env\n", " # state, hidden layer, action sizes\n", " self.s_size = env.observation_space.shape[0]\n", " self.h_size = h_size\n", " self.a_size = env.action_space.shape[0]\n", " # define layers (we used 2 layers)\n", " self.fc1 = nn.Linear(self.s_size, self.h_size)\n", " self.fc2 = nn.Linear(self.h_size, self.a_size)\n", " \n", " def set_weights(self, weights):\n", " s_size = self.s_size\n", " h_size = self.h_size\n", " a_size = self.a_size\n", " # separate the weights for each layer\n", " fc1_end = (s_size * h_size) + h_size\n", " fc1_W = torch.from_numpy(weights[:s_size*h_size].reshape(s_size, h_size))\n", " fc1_b = torch.from_numpy(weights[s_size*h_size:fc1_end])\n", " fc2_W = torch.from_numpy(weights[fc1_end:fc1_end+(h_size*a_size)].reshape(h_size, a_size))\n", " fc2_b = torch.from_numpy(weights[fc1_end+(h_size*a_size):])\n", " # set the weights for each layer\n", " self.fc1.weight.data.copy_(fc1_W.view_as(self.fc1.weight.data))\n", " self.fc1.bias.data.copy_(fc1_b.view_as(self.fc1.bias.data))\n", " self.fc2.weight.data.copy_(fc2_W.view_as(self.fc2.weight.data))\n", " self.fc2.bias.data.copy_(fc2_b.view_as(self.fc2.bias.data))\n", " \n", " def get_weights_dim(self):\n", " return (self.s_size + 1) * self.h_size + (self.h_size + 1) * self.a_size\n", " \n", " def forward(self, x):\n", " x = F.relu(self.fc1(x))\n", " x = torch.tanh(self.fc2(x))\n", " return x.cpu().data\n", " \n", " def act(self, state):\n", " state = torch.from_numpy(state).float().to(device)\n", " with torch.no_grad():\n", " action = self.forward(state)\n", " return action\n", " \n", " def evaluate(self, weights, gamma=1.0, max_t=5000):\n", " self.set_weights(weights)\n", " episode_return = 0.0\n", " state = self.env.reset()\n", " for t in range(max_t):\n", " state = torch.from_numpy(state).float().to(device)\n", " action = self.forward(state)\n", " state, reward, done, _ = self.env.step(action)\n", " episode_return += reward * math.pow(gamma, t)\n", " if done:\n", " break\n", " return episode_return" ] }, { "cell_type": "markdown", "id": "ea18646f", "metadata": {}, "source": [ "### Cross Entropy Method" ] }, { "cell_type": "code", "execution_count": 5, "id": "e0c930e7", "metadata": {}, "outputs": [], "source": [ "def cem(agent, n_iterations=500, max_t=1000, gamma=1.0, print_every=10, pop_size=50, elite_frac=0.2, sigma=0.5):\n", " \"\"\"PyTorch implementation of the cross-entropy method.\n", " \n", " Params\n", " ======\n", " Agent (object): agent instance\n", " n_iterations (int): maximum number of training iterations\n", " max_t (int): maximum number of timesteps per episode\n", " gamma (float): discount rate\n", " print_every (int): how often to print average score (over last 100 episodes)\n", " pop_size (int): size of population at each iteration\n", " elite_frac (float): percentage of top performers to use in update\n", " sigma (float): standard deviation of additive noise\n", " \"\"\"\n", " n_elite=int(pop_size*elite_frac)\n", "\n", " scores_deque = deque(maxlen=100)\n", " scores = []\n", " # Initialize the weight with random noise\n", " best_weight = sigma * np.random.randn(agent.get_weights_dim())\n", "\n", " for i_iteration in range(1, n_iterations+1):\n", " # Define the cadidates and get the reward of each candidate\n", " weights_pop = [best_weight + (sigma * np.random.randn(agent.get_weights_dim())) for i in range(pop_size)]\n", " rewards = np.array([agent.evaluate(weights, gamma, max_t) for weights in weights_pop])\n", " \n", " # Select best candidates from collected rewards\n", " elite_idxs = rewards.argsort()[-n_elite:]\n", " elite_weights = [weights_pop[i] for i in elite_idxs]\n", " best_weight = np.array(elite_weights).mean(axis=0)\n", "\n", " reward = agent.evaluate(best_weight, gamma=1.0)\n", " scores_deque.append(reward)\n", " scores.append(reward)\n", " \n", " torch.save(agent.state_dict(), 'checkpoint.pth')\n", " \n", " if i_iteration % print_every == 0:\n", " print('Episode {}\\tAverage Score: {:.2f}'.format(i_iteration, np.mean(scores_deque)))\n", "\n", " if np.mean(scores_deque)>=90.0:\n", " print('\\nEnvironment solved in {:d} iterations!\\tAverage Score: {:.2f}'.format(i_iteration-100, np.mean(scores_deque)))\n", " break\n", " return scores" ] }, { "cell_type": "markdown", "id": "3735df07", "metadata": {}, "source": [ "### Run" ] }, { "cell_type": "code", "execution_count": 6, "id": "4c662c01", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Episode 10\tAverage Score: -1.44\n", "Episode 20\tAverage Score: -3.98\n", "Episode 30\tAverage Score: -4.18\n", "Episode 40\tAverage Score: 2.57\n", "Episode 50\tAverage Score: 18.74\n", "Episode 60\tAverage Score: 29.35\n", "Episode 70\tAverage Score: 38.69\n", "Episode 80\tAverage Score: 45.65\n", "Episode 90\tAverage Score: 47.98\n", "Episode 100\tAverage Score: 52.56\n", "Episode 110\tAverage Score: 62.09\n", "Episode 120\tAverage Score: 72.28\n", "Episode 130\tAverage Score: 82.21\n", "Episode 140\tAverage Score: 89.48\n", "\n", "Environment solved in 47 iterations!\tAverage Score: 90.83\n" ] } ], "source": [ "agent = Agent(env).to(device)\n", "scores = cem(agent)" ] }, { "cell_type": "markdown", "id": "51dea6d9", "metadata": {}, "source": [ "### Plot the learning progress" ] }, { "cell_type": "code", "execution_count": 7, "id": "abff3eba", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEGCAYAAACO8lkDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABBwElEQVR4nO2dd5xcdbn/38/MbE2yu9lsyWZTNqQ3SFkCEsDQISBNkSgICl6ugmJHQO8V71WvP1EUFdCIIIqiXIiXSA+9lyRASC+kbdpu6qZtmZnv74+ZMzs1OzN7Jmf25Hm/XnnNzjln5jxzMvP9nKd8n68YY1AURVGUaDxOG6AoiqLkHyoOiqIoSgIqDoqiKEoCKg6KoihKAioOiqIoSgI+pw2wg6qqKtPQ0OC0GYqiKL2KhQsX7jDGVCfb5wpxaGhoYMGCBU6boSiK0qsQkQ2p9mlYSVEURUlAxUFRFEVJQMVBURRFSSDn4iAi94lIs4gsidpWKSLzRWR1+LF/1L5bRGSNiKwUkXNybZ+iKIqSyJHwHP4EnBu37WbgeWPMKOD58HNEZDwwG5gQfs3dIuI9AjYqiqIoUeRcHIwxrwC74jZfBDwQ/vsB4OKo7X83xrQbY9YBa4DpubZRURRFicWpnEOtMWYrQPixJry9HtgUdVxTeFsCInKdiCwQkQUtLS05NVZRFOVoI98S0pJkW9Ke4saYOcaYRmNMY3V10jkciqIovY69hzp58K0NHGj3O2qHU5PgtotInTFmq4jUAc3h7U3AkKjjBgNbjrh1ii0072tjf5ufY6r7ZvXaD5v2srp5P2ePr83qPRSlN/Dcsu08srCJa08ZTn1FCZ+//x1Wbd/PE4u3cv8XjqczEOSRhU1sa23jUEeAhgF9mD68kgmDyhBJdj9tD06JwzzgauCn4cfHorb/TUTuAAYBo4B3HLFQ6RHGGK7780I27TrIG7ecTpEvVFewpnkfv5y/mqVb9vLU106lpDC23qDdH+CuF9Zw90tr8QdDTuNTH27ln9fPwOPJ3Q/hcCxu2kPT7kPMmlTnyPndzuY9h6grK478/37z4feZVF/OF2YMB8AfCCIieNP8/zfGsOdgJxWlBSkHz7bOAEU+T48HV2MMzy7bzsdGDKCsuCDj1z/41gb+87EliAhPL91Gn0IvHhG+9PER/P6VtVw+5y2adh1k54EOCn0ein0eWttCHsVl0wZz+2XH9cj+w5FzcRCRh4CZQJWINAE/ICQKD4vItcBG4DIAY8xSEXkYWAb4gRuMMYFc26jYz8urWnh/0x4Anl6yjYsm1/Powia+88gHhMd8dh/soKSwhLbOAL9+fjUbdh5kyZa9bNh5kEum1HPliUNZsW0f3/vnEua+t5nzJ9XxnUc+oMDr4dZZ46juV5RwXmMM63ceZM/BDop8XsYPKuvR52jafZDP/fEd9rf7mVRfzpDKUh5fvIU7n1vNDy+awEkjqg77+v3tfvoUenN6h9dbWbRxN796bjWvrGrhu+eO5cszR7Bq+z7mLtrMvz7YwskjqxhSWcpn/vAWLfva+e1npzJ5SAWdgSCdgSClhbHD1762Tn7zwhqeXrKNjbsO0jisPzecNpJTRlXh83ZF0P2BILN+/SpVfYv48zXTKS5IvyBy3Y4D/OTJ5dw6axzDq/rwj3c3cfPcDzl3wkDuuXJqzP/zym37+Od7m3lz7Q5G1vTjvy6aQJ+ikM17Dnbw/55ewUPvbOKMsTXcftlx/OPdTby4opkfXjSBcXVlDK8q5ea5H3LC8EruO28cxw2pAGDLnkPc++o67nt9HSccM4BPTRvcg/+F1IgblgltbGw02lspfzDGcMndb9Cyrx2fV6jpV8S9Vx/PaT9/ieFVfThnQi0/eXIFr950GkMqS1m4YRefvOdN6itKGF7Vh2tObuD0sbUABIOGS+55gy17DjF8QB/e3bCLAo+H0iIvv7x8MqeNqYk59x9fW8d/P74s8vySKfXcduEEykvSu6szxvDTp1aw+2AH1558DN99dDFrmvfT4Q/y6eMHc8t545j585do2deOR+Dy44ewZU8bK7a1Mrq2H1OH9ueMcTWMGdiPOS9/xG9eXMNxg8v5n0snMbKmX8L59rV18ujCJh5e0IQ/GOR/Lp3EtGGVPbj6cOND71FeUsB/XTThsKLUvK+NF5Y3M3NMDQPLi3t0zlSsbdnPa6t38NkThlIQNUC/uKKZax54l/6lhZQV++jwB3n1u6dz+zMr+cOrH9G3yMfImr4Mqijh8cVbqO5bxK4DHZw0sor3NuymqMDDkzeeQk1ZyG5jDNf9ZSEvrGjmlFFVTKov59GFTWzZ20ZJgZfJQyr4/gXjmDConCc/3Mr1f10EwJnjavjdldPoCATxiHQrFNf86V1eWNHMqJq+zLmqkQt/+xoeEfYe6uTXn5nChccNAmDn/nZm3v4ShzoDTKwvZ3HTHsYMLONrZ4xkyeZWHnpnI3sOdfLFk4fznXPGxIhXNHsOdlBekugBBYKGz/7hLRY37eXxG09mRJZhVxFZaIxpTLpPxUGxm5dWNvP5+9/lfy6dxL62Tn7y5ArOHFfD8yua+ddXTmbV9n188+EPeOnbM2mo6sNbH+1k9py3+NsXT+CkkYl34os27ubSu9/A5xHuuHwy4+v68fn736WmXxFzr58Rc+wld7/OgXY/N583lvc37eWuF9cwsKyYf331ZCr7FLK/3c+X/rKQmWOq+cKM4Qmhit+/vJb/eWoFPo9Ewlp3fXYqr61p4dFFm/l042AefGsjD157AnMXNTH3vc2MqO7DxPpyVm3fz8ptrQQNFBd4aOsMMnNMNe9t3MPBDj8/uWQSlzUOiTnfv/9lAc8s3c6xg8vZdaCDrXvb+OZZo7nhtJFZXfu9hzqZ8l/PEjTwrbNG89UzRgGhwfP2Z1ayYP1u/vMT4+lX7OPKP77Npl2HEIFTR1Xzm89OiYRGtu1to7asKDIo3fvqR6zavo92f5D2ziDt/gCfOG4Ql05NftcaCBrue20dP392Je3+ILMmDeTO2VMo8HrYuvcQs+58ldqyYh758km8trqFLz24iN9/bhq3zVvKuLoyZk2q49v/+wEAN507hiumD+P7jy1hyea9TG+o5LEPNnPiMQO4//PHIyLMeWUtP3lyBf9xwXiuPTkUjurwB3lu+XbeWbeLf32whQF9C3nixlO44t632bLnEP92yjH8YN5SSgu9HOwIUOjzMG1of04aMYCTRg6gorSQ+cu2s+tAB9fPHMGyLa189t63OX9SHU8t2UpxgRd/0PDkjSfzrf9dzMadB3jmG6dS06+Y7//fhzz0ziae/topjKrtx8urWvjK3xaxr82P1yOcMLyS758/vkee7ba9bZx35ytMGlzBn6/JruL/cOLgiq6sypGjwx/k0nte59tnj2Fm3F27xR9fW0d9RQmfnDqYA+1+fv7sKp5b3sxl0wYzsb6ctS37AQiEb0wC4UE4VUx56tD+/OxTxzK4oiQiHqNq+rJjf0fMcdtb23hv4x6+ffZoTh9by+lja2kc1p+r7nuH55dv57LGIby+Zgevhf89s3QbN583lqlDQxP0n122nZ8+vYLzJ9Vx24UT+POb6ykrLuD8Y+uYMKiMf7y7iQff2sisSQM5eVQVJ4+q4ieXToq529x9oIPnlm/n7XW7OHfCQM4cX8uO/e3c+NB73PrPD2mo6sPxDSHPYH+7nxdXtPD5kxq47cIJ7Gvr5Ja5H3L7Myupryjh4ilJq7gPy9sf7SRoYFJ9Ob+Yvwqf18PFUwbxmxfW8Le3N1Ja6OXiu16nb7EPAeZ8bhpLtrTy6+dX84dXPuJbZ4/hjbU7uOLet/n17Cl84rhBbNh5gB89sZz+pQWUlRRQ5PNwsCPAt//3A2rLipmRRNB//8pafvb0Ss4cV8PE+nJ+9dxqOvyLOG1sNY8ubKLdH+SuK6bSt8jHmeNqqS0r4gePLWVbaxu3zhrHBcfW8cbaHZQVF/Dlj49ARPjNZ6ZE3n9cXT9u+9cy7pi/inZ/kD++to7zJg7kmhkNkWMKfR5mTapj1qQ6Zoys4t/+vIDvPrKYd9bt4nuzxnH1SQ30KfLxwaY9DCwvZs/BDl5fs5NfzF/FL+Z3fRaPwJMfbqWkwEt9RQm/+PRxTB5SwY+fXM53zhnDyJp+/OKyYzn/16/x6d+9yU3njuVvb2/kcycOY1RtyFv8+Ohq5n/j42zYeYCJ9eWR8FJPGFhezL1XNzK0sk+P3ysZKg5KRrS2dbJkcysPvrUhqTh0BoIsWL+by48fQqHPQ6GvkAuPG8RTH27l2+eMAbpEwBKF7sQB4NNxd9xej0ReZzF/2XYAzp4wMLLt5JFVDOhTyJtrd3JZ4xDeWLODkgIvt104nh8/sZxP3vMmx1T34WB7gG2tbYyrK+P2y46ltNDHt84eE3mfhqo+XHDsIJ5aspWbzhkb2R4fhujfp5DLGofEeAhVfYu458ppXPTb1/jyg4v411dnUFdewiurWugIBDlvYsjefsUF/OryyWxvbeN7//yQ44ZUMLwq9oe/v93PbfOW4hGo6VfMF2Y0MKBvV+7ljbU7KS7w8NB1J/Lvf1nA/3t6Bf/v6RUAfHnmCP791GP40RPLeW/jbn535TRG1fbj7AkDWdu8n/teW8fnThzGDx5bijHw2Pub+cRxg3h2aei6zvvKyQypLAXgQLufi+56nRsfeo85V01je2s7tWXFTBvWn2DQ8NA7G/nYMQP4w1WNiAh9Cn38z1PLeW75djwCd3x6ciQU4vN6mH38UO58fjX9inycNb4WEeGOT09O+X246mMNPLe8md+8sAafRzh9bA0/+9SxKcNoZ42v5cxxtcx9bzMlBd7I9+lT0wYnxOx3Hejg7Y92sutgBzPH1NDc2saXH1zE6t37uXP2ZIoLvHzxlOGcMrqKMeHBf2RNP/76xRP40oOLuP6viygr9vH1M0fHvO/A8mLbw3c9DUEeDhUHJSOsAfmV1TvY3+6nb9wd0NItrRzqDETujgF+eOEEvnbGKGrD8WGvxIlD2IPIpBopmTg8u2w7DQNKGVXTFX/1eIQTRwzg9bU7MMbwxtqdHD+8ksuPH8r5xw7iicVbeOz9LVSUFvDx0dWcN6kuIdFp8aNLJnL9aSNoqMr8Tq28pIA5VzVyyV2vc9Mji/nLtScwf9l2+pcWMG1YpLUYPq+HO2dP4bw7X+XGh95j3ldmxAx4/3h3E48sbKK6XxEt+9qpqyjmihOGRfa/sXYHxzdU0rfIx4PXnsDK7ft4cUULtWVFXDKlHhHh50kqXL5x1iieWrKVy37/Jht2HmRifRmvrNpBa1snzyzdxvi6sogwAPQp8vG7K6dx8V2v88l73gSgtNDLy985jbUt+9m06xDfOmtMxPZ/O/UYZk8fwqGOAAVeD/37FMac/zPTh3LXi2uYNakurQSxxyP89rNTeOujnXxsRFVaOaXbLhzPG2t3cOnUespLUx9f2aeQ86Iq0+orSnj8xpN5+6NdESEXEcYOjA0JNTZU8q+vzuD7/1zCJ44blPAZexsqDkpGWANyhz/Iiyua+UQ4AWfx7rpQp5TGhq4Br0+RL8aNTvAcAqFHX6biEJUva23r5M21O/jCjOEJd48zRlTxxOKtvPXRLlY37+eT4TvFvkU+Lj9+KJcfPzStc5YVF1A2MPNyRYvRtf34xlmj+dETy3l++XZeWNHMmeNqE5KRgypK+PePH8PPnl7JnoOdkUEmEDQ88MZ6Gof15/efm8a0Hz0XI5DN+9pYtX1/JA9gDWDxg1gyRtb04+Ip9cxdtJnTx9Zww2kj+eQ9b/CPdzaxcONuvn7G6CSv6cvc609icdNeKvsUcN2fF/Lr51dzqDNA3yIf50R5cBDyjPqlKPccWB7KPzQMKE26PxkVpYWcOzH98uLB/Ut58dsz0y5OiKaqbxHnH9v9uerKS/jj54/P+P3zERUHJSOiB6Onl2xLFIf1uxhaWRrxEpKRIA6W55BBuadHhGCULS+tbKEzYDh7fG3CsTNGDgDgjvkrAThpxIC0z2M3V32sgb+8tYFv/ON9Wtv8nJXEXiDikUUL4Isrmtm46yA3nTsm4RoCvLl2J5D95/vW2WPo8Af57rljqa8oYVB5MXfMX4UxcPaE5HaOru3H6HBo5TPTh/LQOxvxeYWLJ9cnzGHpjsnhUs1ccrjvpRJLvrXPUPIcazAqLyngxZXNtHV2TUMxxrBgw+6YkFIyrPCRNfAF08g5xBPvOSzetIcin4cpQ/snHDu0spT6ihLeXb+bfsU+JgwqT/s8dlPo8/Ddc8fS2uanyOfh1NHJ50lYQhktgPe/sY668mLOmTAw4h1Fi8Mba3ZS1oPPV19Rwm8/O5UhlaV4PMJ5k+o41BlgaGUpYwcmluHGc+MZoyjyhaq0clV7rxw5VByUjLDKO8+ZUMvBjgAvr+pqeri25QC7DnRwfEPiAB2NFT6yBj7rPTMNK/kDXQOjP2go8nmSCoyIRO6mTzxmQEYilAvOmziQU0ZVcf5h8hu+OAH9qGU/r6/ZyZUnDqPA2/U5g1EC+da6nbZ+PmtG+NnhBHF3VPcr4qZzxzJzTHVMHkXpnag4KBlhDUYzRlZR6POwcMPuyL4F60P5huOHH95zsBLSligEs0lIi8QMjIGgOeygeFI4tORkSMlCRPjzNdO54/LJKY+xroUlgE27DwFwQvjadiX1u16z60AHgypKbLNz6tAK/vviiVz38WPSfs3VJzXwpy9M1xnhLkBzDkpGWINVkc/DwLJitu1ti+x7Z/0uKvsUckw31TyeOM8hkKXnEB1S8QcNXk/qe52zxg/kihN2R2awOk13g6cvzjOwPAhLAK2PGi2QwW4EMhsbP3fisO4PVFyJioOSEcHIIBUWh9YucfiwaS9Th1Z0O/B540ImlgeRUUI6ThxCA2Pq4/sW+fjxJZPSfn+nSV3RFfqQ3iQ5iYAxOBwxU1yEhpWUjPBHksdQW17M9rA4GGPYvOdQWrM1rYEvElbKIiHti0tI+4MmMnC6AU/KuSBx+2M8h8xCc4pyONzza1KOCF2zmT0MLCti2942jDHsPdTJwY4A9f27j3nH3/Vmk5D2SJznYAwu0oaEhHT8LPL40ByEroFXY/2KTbjo56QcCSKDlAi1ZcW0+4PsPdQZSZjWV3RfRx4fMskqIe2JnecQCLprYPTEh5WSCGh8OW/A2JtzUI5uVByUjIi+g7X6xGxrbWPLnpA4pFMtk7K3UgaDe3TXVOs93DQwJrQYSZKXCVVshf42xmBMZnkbRTkcKg5KRkSLgzXbdNveLMUhPmTizSwhnUkpa2/DuhaJnkPXT9bjSaz4ctM1UJxFq5WUjIguqawJr8S2vbWNLXvbKPJ5GJBGs7GEZGsWnoNXMitl7W1E8jJxpazRHzE67xJf6qooPcU9vybliBAIhmZdeT1CTVlIHLbtbWfz7kPUV5SkNfmpuxr+dAh5DqFwivVehytl7W1EKroCqT0Hr0hUC5LQNg0rKXbh6M9JRL4hIktFZImIPCQixSJSKSLzRWR1+FHn4ecR1oxcn0co8nmp7FPIttY2Nu85lPbs3ISBL5BdKWvInq7B000J6VShtxjPISop39W88Agaqbgax8RBROqBG4FGY8xEwAvMBm4GnjfGjAKeDz9X8gTLc7DuUGvLimkOJ6QHpVGpBFFlmPGeQyZhpSSDp5tCKukk7b2eroR0UMNKis047Yj7gBIR8QGlwBbgIuCB8P4HgIudMU1JRsRzCCdMB5YVsWn3QZr3tVNfkV4v/q67/tDzYOSuOLN5DqHXWna5SxxS5WViEtIxYaXMZ5kryuFwTByMMZuBnwMbga3AXmPMs0CtMWZr+JitQNKFikXkOhFZICILWlpakh2i5AB/VM4BQou0rG4OrQmdtucQGfiC4fc0GU2Agy6Bsexxmzgk5GWShZVEq5WU3OFkWKk/IS9hODAI6CMiV6b7emPMHGNMozGmsbq6OldmKnHEhzdqy4qxKkrrM8w5RFfaZNr2oWuGMJH3cNPAmJCXMUkS0p7EaiVtn6HYhZNhpTOBdcaYFmNMJzAXOAnYLiJ1AOHHZgdtVOKIv0MdGLWyVjqtM6Jfay3HEMzCc7CmRETnHNwUUvHEl7Im9RwSq5XclJRXnMVJcdgInCgipRKqfzwDWA7MA64OH3M18JhD9ilJiBeH2vIucRhYnl5Yqctz6AorZTqoJUvYZiow+YwvMgmO8GNyz8Hy2rrKgY+cjYq7cWwSnDHmbRF5BFgE+IH3gDlAX+BhEbmWkIBc5pSNSiJd4Y1Yz6G6XxFFvvTWDI5fqCYYzDysZE14iylldZE4WJ5DdE4ltL3rmOiwkpV70EV2FLtwdIa0MeYHwA/iNrcT8iKUPCQQV1lkiUMmK5DFL3GZTULaukN2eylrMO7zRQ/+IklakKg4KDahTqiSEfHdQStKCyj0eRichThYydZgNgnpuLbfbktIx5f7BpK04/ZK1yQ4neeg2I32VlIyIt5zEBG+cFIDU4amP5HdGr9i7vozvOP1xTWmC7otIR2XlwkEE9eriAkrabWSYjMqDkpGJFtX4JZZ4zJ6DxGJWY/Bn0VIqCsmn31oKp9J8BySrHTniWrZHdBqJcVmNKykZEQ26z0nwysSs0xopuKQLCbvprvm+GVAQ6W6scd4o9qWd1WRHTkbFXejXyUlI4JJPIds8HiieytlHitPthiOmzyHSKluoCus5PPGew5JVtNTz0GxCRUHJSP8QXsSnz6PJ2pgD2btOQRcmpCOnygYMIk5FU9Sz8E910BxFhUHJSOCJhTe6Gk9ffRdbzYJ6YR1qF2WkI6EzaxrFEj0jKIXPAqo56DYjIqDkhHZJI+TEdMXKJh5lU2kmqcHcyXyGW9cwj2ZZxTtORitVlJsRsVByYhsksfJ8HokKtkazLorazDac3DRwJg84R53jEhUy/KubYpiByoOSkZk0wcpGV6PRFaAC5jM73jj76xd5zkk7R0Vl5D2HH6lOEXpCfpVUjLCrjYVsesfZz6we+Jj8lnMss5nrI/ijxKH+I/nkcRJcOo5KHah4qBkhF3iEL3+sT8YzD4h3YNZ1vlM/ETBZJ5D8nkO7rkGirOoOCgZEUqM9vxr4/PErkWQ6Vu6vWU3xE4UTOYZeUUS1uF2k/ekOIuKg5IRgYCxZRauxyNR+YJgwl1xd0RPgstmDereQMxEwSTi5/FI4jrcLvKeFGdRcVAyImASwxvZEN1RNKuEdJTnYImM2zyH2ImCiZ5D9BrSVo8lN4XWFGdRcVAyIllJZTbEL1TjzXBMiy71dGtH0sSJgrH7Y8uBtVpJsRf9KikZkSwxmg3R4hCaWJdhWMlaEyLKc3DbXXPsRMEUXVl1PQclR6g4KBmRrKQyG7ye2FLWTPMYnqicg1srdbweT2xX1iTrOSRUK7lMIBXncFQcRKRCRB4RkRUislxEPiYilSIyX0RWhx/TX0VGyTm58ByyaZoXE1ZyrTgQNVEwSSlr9FwRl4bWFOdw2nO4E3jaGDMWOA5YDtwMPG+MGQU8H36u5Al+m9pUxDSNyyKsFL0YjlsT0tGDf7LrLjHtM9RzUOzFMXEQkTLgVOCPAMaYDmPMHuAi4IHwYQ8AFzthn5KcoLFnPoEnLp6eaUI6ehlNt941e70S0zsqoSurJzZhHdrmrmugOIeTnsMxQAtwv4i8JyL3ikgfoNYYsxUg/FiT7MUicp2ILBCRBS0tLUfO6qMcu7qy+uLi5RknpCXRc3DbXXPMJLgkLclj8jbhR5ddAsVBnBQHHzAVuMcYMwU4QAYhJGPMHGNMozGmsbq6Olc2KnHY2ZXVHxNWyuz1lpYEXJ1ziC1Vjb9GHpFIq+7IPAeXXQPFOZwUhyagyRjzdvj5I4TEYruI1AGEH5sdsk9Jgj+LVduS4YmZBJe54FjJ2UAg6NqQSkxvpWQJ6bjQHLjPe1KcwzFxMMZsAzaJyJjwpjOAZcA84OrwtquBxxwwT0lBMGjPAORLKGXNcg1pY9/SpfmGJz6slDBDOrErq9vyLopz+Bw+/1eBv4pIIfAR8AVCgvWwiFwLbAQuc9A+JQ5/MEhRQc+/Nh6P4A9ETYLLUHCsm+hg0Lh2ApjPG9+VNVEcrHCSeg6K3TgqDsaY94HGJLvOOMKmKGkSMPY0d4vuKJrNKm6RsJIxEZFx28AYXcqaPCGdWK2knoNiF07Pc1B6Gdks6ZkMrze2fUbmi/1Y9rjXc4gv903WlTUYV63ktmugOIeKg5IRgaA9d6cxk+CyWMXNexS0z/DF9Z867HoO4clwLrsEioOoOCgZYZvnEJ+QznYluOjGey4bGeMTzslKWRMS0i4LrSnOoeKgZESyqplsCJVphv7OJqwkIqH1DIx7w0rddmX1hBLSxsVzPRTnUHFQMsKu5ThDs3+DPVrFzZpI59qEtKebhLRYzQe7lgl12zVQnEPFQcmIgMk8BJQMa4nLngxq1kQ6N3sOMaWs3sRqJYjtTKvVSopdqDgoGRFaQ9q+3kqRZHKmnffoSti6NSHdXW8lT1TeJZtZ5opyOFQclIywaxDyegR/dOuLbDyHcFjJteIQt+ZFsklwQFhkNaSk2IuKg5IRAZsa71mzewM9CAl5470Pl4qDCX/GZKWs0DXXQ7VBsRMVByUj7BIHa3avtdJZVuIQLuW0Qi9uK+P0hBPSkY6rKcJKwaB93XIVxULFQckIu9ZzsNZHtsNzsBLS8Qnb3o4vnJC2PKOEhHT4aTB8HTWspNiJioOSEYEsJqwlw/IcelKf7w0373Nr0zkrIR1I4RlFJgKGq5W0UkmxExUHJSMCQZNVZVE88SGhbEtZAy7OOXgsz8HyjJKsIQ1EjnHb51ecRcVByQj7PIfQV8+awJbNXa/V0tqt4mCteRFIcY2iPYdA0H05F8VZVByUjEhWUpkN1gSujkAASLwrTus9RAgYUoZdejtWV9ZUnkNMtVIWS60qyuHQr5OSNsGgwRh7ZuFa79Hhz/6uPzR4BrsGTxcmpEOht1ATqoSV4MLPjbFv5rqiWKg4KGmT6g42G6z36Aj3ms7mrt/KW7g1IW11XbUaFMZ/Pm/0mhZBE8lBKIodqDgoaWPnamOWGHT4QyNfVmElqz+TS3MO3jjPIdUM6UC4nNdtn19xFsfFQUS8IvKeiDwefl4pIvNFZHX4sb/TNiohIvX2NrXPAOgMJA+ZpPsegWDQteJgJaQtzyEhrBRTreS+z684i+PiAHwNWB71/GbgeWPMKOD58HMlD7BzJnIkrBT2HLLtrRSdkHbb4OgJr3mRMiEdP8/BXR9fcRhHxUFEBgPnA/dGbb4IeCD89wPAxUfYLCUFQRs9B+suuN0Shyy7skbPA3CbOFhrXgRSJaQjnoN9bU0UxcJpz+FXwE1AMGpbrTFmK0D4sSbZC0XkOhFZICILWlpacm6ogq3LcVqeghVWysZziE9Iu62U1Rte6c2fQpSt/werfYbbPr/iLI6Jg4hcADQbYxZm83pjzBxjTKMxprG6utpm65RkdC2q0/OvjTcurJSNN+KxmvfZ6NHkE5G8jD+5+Fkft2ueg7s+v+IsPgfPPQO4UERmAcVAmYg8CGwXkTpjzFYRqQOaHbRRiSLVHWw2RMShhwnpzoB7E9Jd1ygQ89zCE5VzUM9BsRvHPAdjzC3GmMHGmAZgNvCCMeZKYB5wdfiwq4HHHDJRicPOpSjjq5Wya7zniXgOHsF1df5d3tXhZ0iHlkrVJUIVe3E655CMnwJnichq4KzwcyUPyInn4O+BOEhUu2oXDozW4J/Ku+rKOYTXc3DfJVAcxMmwUgRjzEvAS+G/dwJnOGmPkhw7J8FZA197D0pZrZbdwSTrK7sBTzd5GU9UbyWtVlLsJh89ByVPsTPxGz/w9WSxH3/QnmaA+Ub8XJBUCWmtVlJygYqDkjZ2lozG91bKVhwiOQcXioMnLiGdsBKcp8tz0GolxW5UHJS0yYXn0NkDz8ETNc/BjZ5DZC5IqlLWuGolFQfFTtIWBxEpEZExuTRGyW/snImckGzNMufg5oHRErz2FN6VdQ2NcW/eRXGOtMRBRD4BvA88HX4+WUTm5dAuJQ+x2jjYMRDHx9Oz78rq3oGxu4R0V1iJcM7hyNqnuJt0PYfbgOnAHgBjzPtAQy4MUvKXgLWugJ0J6Z7kHCTUW8mtCenIankpEtLWU2vNBzd6T4pzpCsOfmPM3pxaouQ9fhs9B1vmOXgEv+U5uHBgtNqURDyHFAnpYHg9Bzd6T4pzpDvPYYmIfBbwisgo4EbgjdyZpeQjQRs9BztmSHtcXsoa35wwfvD36jwHJYek6zl8FZgAtAN/A/YCX8+RTUqeYqvnELcSXDZ3vdYaywHjVs8hNvSWMAkuviurC6+B4hzdeg4i4gXmGWPOBL6Xe5OUfCXSldWG8EV3A186eCQUVgoEjOvWj4buQ2+R3krhaiU3XgPFObr1HIwxAeCgiJQfAXuUPMYfsK+U1RNfw5/tDOmge0tZrYR0ewpx6GqfgWuvgeIc6eYc2oAPRWQ+cMDaaIy5MSdWKXlJ0MZ5DlZyNVUNf1rvEVlj2Z0Do5WQTpWXsZbVCIarldRxUOwkXXF4IvxPOYqxsyurJy7nkN1iP6E1lt2ekE6Vl4lZQ9poWEmxl7TEwRjzgIgUAqPDm1YaYzpzZ5aSj9jalTWuWimrGdLhNZaDLk3GeuLmOaRcz8FotZJiP2mJg4jMBB4A1gMCDBGRq40xr+TMMiXvsLO3UvwM6exLWUO5EDfeNfviwkrxAhipVgoa1wqk4hzphpV+AZxtjFkJICKjgYeAabkyTMk/7OzKGt8aIptxzRIYfzDoyrvmyAzpVKWs8fMcXCiQinOkO8+hwBIGAGPMKqAgNyYp+UrEc7BhybHoCV5ej2S1xGd0qacbxcEa/FNVK0UmwRk0rKTYTrqewwIR+SPwl/DzK4CFuTFJyVcCuZjn4A9m/X7Rg2dZifsGRl9c+4zDVisZezw6RbFI13P4MrCUUNuMrwHLgC/15MQiMkREXhSR5SKyVES+Ft5eKSLzRWR1+LF/T86j2IflOdjZPqM9kP1df/SCQW68a45PSMeLaHRvpZDncETNU1xOul8nH3CnMeZSY8wlwK8Bbw/P7Qe+ZYwZB5wI3CAi44GbgeeNMaOA58PPlTzAVnGQnoeEPDZ4H/lM/CzyhIR0JKykjfcU+0lXHJ4HSqKelwDP9eTExpitxphF4b/3AcuBeuAiQpVRhB8v7sl5FPuwVRyi8hbZvp/1Fp0u9RyiK7qSVYhZYqDVSkouSFccio0x+60n4b9L7TJCRBqAKcDbQK0xZmv4PFuBmhSvuU5EFojIgpaWFrtMUQ5DLjyHnrzf0ZKQ7vAHkw78MYv9aLWSYjPpisMBEZlqPRGRRuCQHQaISF/gUeDrxpjWdF9njJljjGk0xjRWV1fbYYrSDX4bxcET9c3LNhzS1V7CnZU60RMFk3sOocdQWMmeyYmKYpFutdLXgf8VkS2AAQYBl/f05CJSQEgY/mqMmRvevF1E6owxW0WkDmju6XkUewhGJsH1PPMZ/R7ZTqqLXinNzeKQKqciIngE/IHkCWtF6QmH/ZWLyPEiMtAY8y4wFvgHoUTy08C6npxYQoXtfwSWG2PuiNo1D7g6/PfVwGM9OY9iH/7IJLiev1f0e2SdkLbCLgF3J6TbA8GYHE38MV0e3REzTTkK6O7r9HugI/z3x4BbgbuA3cCcHp57BvA54HQReT/8bxbwU+AsEVkNnBV+ruQBwXBb6GwmrMVj3fVCbIgpE3w2JLXzmZiKrhTXXES6Zpm78BooztFdWMlrjNkV/vtyYI4x5lHgURF5vycnNsa8RqhPUzLO6Ml7K7nBb3PS0+fx0BEIZh2m8tiQ1M5noj9TqoHfK9Kj5oWKkorufpVeEbEE5Azghah96eYrFJdgd4sGSxOyfctoW9wuDqnyMl5Plzi4MbSmOEd3A/xDwMsisoNQddKrACIyktA60spRhN3iYA1m2c9zOHrEIZVXEEpI29dKXVEsDisOxpgfi8jzQB3wrDHh5johj+OruTZOyS9sFwePJQ7ZhZXSGTx7MzGew2ES0p1WQtp9l0BxkG5DQ8aYt5JsW5Ubc5R8Jnfi0LPXgz1rTOQbMTmVlJ6D0NmDNTEUJRVa/KakjT9X4pBtV1aX5xx8aXw+T1TOQcNKip2oOChpE7S5WqnLc+hZV9aevEc+k07C3SsSacynCWnFTlQclLSx3XPQhPRhERGsj5gqpxJdreTGvIviHCoOStpYk+DswtNDz8Hj8oQ0dHlHqRLSHo9WKym5QcVBSRt/0Nia+O1pWMntCWnoEr3UpaxR1Ur6a1ZsRL9OStoEg/auGWAN7tl3Ze1+BnFvx/qMKSfBRVUrudV7UpxBxUFJG38weevobLFyBll3ZRX3ew4RAU2jWsmNeRfFOVQclLQJBO29O7UzrOTWgTEtz0GrlZQcoOKgpE0gGEyZGM2GnoaVol/n1pBKdxVdIc9BE9KK/ag4KGkTMLnxHLIVnOjX2Sla+UR3AuoR1HNQcoKKg5I2AZtzDt1V4qT7+p68R77TbVgpZob0ETNLOQrQr5OSNgGbq5V8NuYc3JqQjghoqrCSSNc8B5cKpOIMKg5K2gRsnufQ00lw0ba4Nd5uhcsO6zkEtVpJsR8VByVtcraegx2N91x61+ztxnMIVSuZmGMVxQ7yVhxE5FwRWSkia0TkZqftUewXh8hdcZbJ5Jh5Di5PSKfyHERC/y/gXu9JcYa8FAcR8QJ3AecB44HPiMh4Z61SAsberqw9TkhHfXvdGm/vrq350TDXQ3GGfF0HejqwxhjzEYCI/B24CFiWi5MZY3jyw220+wOUlxRQUVpARWkhx1T1QaJ+lGtb9vPzZ1bywaY9AIytK+MXlx1H/z6FGZ1v7qImnlm6jYUb9vDlmSO49uThPbK/eV8bl979Bv998UROG1MT87lWbd9PXUUxZcUFSV/7t7c38sSHW7h11jgmDCpn94EO3l63i+2tbQSN4aqPNUQGHX8gV4v9ZJtz8ET97c6BsbuEtNtXw1OcI1/FoR7YFPW8CTgh+gARuQ64DmDo0KE9OtmijXu44W+LErZfMqWeOz59HAC/nL+Ku15aS0mBl7PG1yLA4x9u5ZP3vMED10xnSGVpWueau6iJbz78AfUVJRR6hQfeWM81MxpiRChTFq7fTdPuQ9z0yGKe/fqplJUU8Oc31/PgWxtY23KAuvJi7r5iKlOG9o95XWtbJz99ajmtbX4u/O3rTG+oZMGGXZEYNsDImr6cMqoasL8ra49nSEeXsrpUHLpLSMeW8x4Rk5SjhHwVh2RfcxPzxJg5wByAxsZGk+T4tHli8VYKfR7mfWUG7Z1B9hzq5KWVzdz/+nqmDevPvjY/v35hDZdOqefW88dR1bcIgM+cMJQvPrCAC3/7Gv9xwXgumVJ/2EH+o5b9fP//ljB9eCV/++IJzH1vMzc9spgPmvYyeUhF1vYv29qKR2D3gQ5unruYgx0BXl29g2nD+vMfFwzjT2+s49O/f5PGYZW07G/n+IZK/vuiCTzw+npa2/z89Ysn8K8PtvDWRzu56mMNzJo0kJp+xZzxi5d5bfWOiDjkbD0HG8JKbk3GZuI5aFhJsZN8FYcmYEjU88HAllycKBg0PPnhVmaOrmbswLLI9lNGVvFRywF++K+ldAYMFx43iJ9fdlzMj/T4hkrmXn8S3/7fD/jmwx8w74Mt/OGqRgqS9E5ev+MAN/ztPQp9Hu6cPRmf18O5Ewfy/f9bwmPvb05LHNbtOMDDCzbxnbPHxNixfGsrI2v6MmtSHb96bjWFPg8/vXQSs6eHPKpPTR3MD+YtYcOug9SVF/PQOxs50O7n5VUtnDmuhhkjq5gxsirhfNOG9eeV1Tu4Jepa5ZPnEB1W8h6lCenozRpWUuwkLxPSwLvAKBEZLiKFwGxgXi5OtHDjbra1tnH+sXUx2z0e4ZeXT2ZgeTEnj6zi9suOTXr3NqK6L49+6SRunTWWl1a28NA7G2P2B4OG2+Yt5cw7Xmb9jgP88tOTqSsvAaCsuIDTx9Tw+OKtkYqTwzF3URP3vLSWzXsOxWxftqWVcXVl3HDaSL551mjmfvmkiDAAlJcW8KvZU/jn9TP4y7Un8M2zRjPvgy3sPdTJ184YnfJ8J4+qYvnWVlr2tQM5XEM668V+ot7LpQNj9+0z1HNQckNeeg7GGL+IfAV4BvAC9xljlubiXE8s3kqRz8MZ42oT9lX2KeS5b36cQq/nsOEij0f4t1OO4cUVLfzqudVcPKU+kgB+v2kPf3pjPZdMqeeWWWOp6Vcc89oLJw/i6aXbeG3NDk48pvKw51rTvB+AzXsORXIcew52sGVvG+PryijwerjxjFHdfuavnj6SAq+H1rZOJg0uT3ncqaOquf2Zlby+ZgcXT6knmG+L/RwNLbu7aWuuYSUlV+Sr54Ax5kljzGhjzAhjzI9zcY5A0PDEh1s5bUwNfYuS62SRz5tWslhEuHXWOHYd6OCel9ZGtm/d0wbAv3/8mARhADh9bOjcV9/3DmO+/zS3zP0w5TkscdgS5Tks29oKwPhBZUlfk8rWL88cwXfPHXvY4yYMKqN/aQGvrG4B7PccelrKejQt9nO4rqyRv13qPSnOkLficCR4d/0uWva1c8Fxdd0fnAaTBpdzyZR67nttHXsOdgCwdW9oIB9YligMAMUFXu6+YirfOWcMYwf2Y+GG3UmP6wwEWbfjABAnDltC4jCuLn1xSBePR5gxsorXVu/AGGN7tZKvm3h6d4hIJObu1rvm7hb78WpYSckRR7U4TB3an/u/cDynj63p/uA0uWjyINr9wchd/vbWNop8HspLks8zADh1dDU3nDaSGSOraNp9CGMS8w8bdh7EH85LbI7zHGr6FUUqqOzm1FHVNO9rZ3Xz/pDnYOckuG4GvnToaWgq38kkIe3WvIviDEe1OBT6PJw2pobSQvtSL4P7h5LN1gC+rbWdgeXFaYWm6itKONQZYPfBzoR9ltgU+TxsDoeqAJZv3ZcTr8Hi2CGhnMSq7fvC7TPs+8pYRV09Gdg9PSyHzXe6C71FC6tLL4HiEEe1OOSCQRUhcWjaHRKH7XvbqE0RUorHEpam3QcT9q1p3gfA9OGVkbBShz/ImuZ9GeUbMqU+6vOExMG+97ZKUXuSTO5p2+98p7vQm4aVlFyh4mAzpYU+KvsURsRhW2sbdeXpiUO95XXsPpSwb03zfgaVFzOqph+bw6Gn1c376AyYnHoO/YpD7USadh+03XPoaUIaet72O9/pNueg1UpKjlBxyAGD+5eweU9oAN/W2pYyGZ3wuopQeWr8PAaANS37GVnbj/r+odDTnoOdLG7aC8DEHHoOEPo8ufAcrPfqSUdVt+ccPN3lHLRaSckRKg45oL6ihM27D7L7YCcd/mDaYaWyEh/9inwRr+PFlc387OkVBIOGNc37GVndl/qK0Htt3nOI9zfuobykgOFVfXL2WSAkWk27D4W6stqacwi9V08GtUgLDpeKQ3dhs5iEtEuvgeIMeTkJrrdTX1HCiyubu8pY0wwriQj14bt0gPteW8erq3cA0NYZZGRN30hOY/OeQ7y/aQ+Th1T0qGlfOgzuH/o8xtib+LUjId1dS+veTneht5icg0uvgeIM6jnkgMH9S2jrDLJ8ayiJnK7nYL22afdBgkHDB5v2IAJ3hyfVjazpG0kQr96+j1XN+3rUsC8Tm9r9oaUo7VxUx467freHlboLvcWElfTXrNiIfp1yQH3/UO5g4YZdAGknpCEcktpziHU7D9Da5ucbZ46OzJEYVdOXyj6FFPk8PL10G8bA5KEVttsfz+D+Xe3I7Yxre2y46/e4PKzUXegtpm25eg6KjWhYKQdYd/cL1u9GBKr7pT9BbXD/Uva1+XllVahlxbkTBzJ5SAUvrGiOLCpUX1HCks2hmdGTB1fYa3wymypLIn/b2cPIjjJU647aveIQetTeSsqRRsUhB1glqaub91PdryhpC+/uXvv44q30LfIxorovo2v7cero6phjPtpxgIYBpRmvQpcNltiBvT2M7ChDdX9COuw5pFxDWj0HJTdoWCkHlJcU0K84pLvplrFaWBPhFm7YzbGDy5MOeoPCLb+PRL4BuuY6gL2egx0Dux2hqXzGGvBTew7Rf7vzGijOoOKQI6y77XQrleJfBzAlRT7Bqlg6UuIAXXbZ6Tl0t1ZBWu9hCYxrF/uxHtPJORwJi5SjBRWHHGF5AJl6DpV9Cikp8AIweUj/pMcMGxBKEE8dlnx/LrA+Ty7Wc+jJe7q9lLW7hHSkeaGQ85Jm5ehCcw45wqrwydRzsOY6rGnen9IzOG/SQMpKGjn2CCSjLazPY+88By1l7Y5uE9Iuz7kozqHikCOsMEwmcxwshlWW0u4PpKxyKvJ5OX1s4sp1ucTyHHKxTGhPQlWu760kh79GHhtCc4qSDBWHHJFtWAngPy4Yz4EOv90m9YiI55CDhLQtXVldOjh6u+lca0fzQkVJhiM5BxG5XURWiMhiEfmniFRE7btFRNaIyEoROccJ++xg5pgavn32aKYPr8z4tQ1VfZgwKPXazk5g9W9KtZxqNthx1+sVQcTNy4SGHlN3ZbUe3fn5FedwKiE9H5hojDkWWAXcAiAi44HZwATgXOBuEfE6ZGOPKCn08pXTR1Hoc0fOf2RNX/55/UnMHFPd/cFpElmroAeVRh6Pe70GSKMra8RzOGImKUcJjoxcxphnjTFW3OQtYHD474uAvxtj2o0x64A1wHQnbFQSmTK0Pz4be3bbUsrqEVffNXcXNnN7Ql5xjny4rb0GeCr8dz2wKWpfU3hbAiJynYgsEJEFLS0tOTZRyQV29EXyejyuHhg93SWktVpJyRE5S0iLyHPAwCS7vmeMeSx8zPcAP/BX62VJjjfJ3t8YMweYA9DY2Jj0GCW/6W4JzHTwirvDSt3NBdFqJSVX5EwcjDFnHm6/iFwNXACcYYyxBvcmYEjUYYOBLbmxUHGakTV9GVpZGpnxnQ1ej7h2djR0iULKhLR6DkqOcKpa6Vzgu8CFxpiDUbvmAbNFpEhEhgOjgHecsFHJPaNq+/HKTadR2YPmgR4RV3sO3SWkI9VMLr4GijM4Nc/ht0ARMD885f8tY8yXjDFLReRhYBmhcNMNxpiAQzYqvQCf190JaW838xi6chJHzCTlKMERcTDGjDzMvh8DPz6C5ii9GI+4XBy6KfeNJKTVc1BsRu83lF6N1yOuDql011jQjhYkipIMbZ+h9GrOmziQUTV9nTYjZ9SVl1Do86Rc1Mnt61kozqHioPRqzp1Y57QJOeXEYyp5/z/PorQw+U9Vq5WUXKFhJUXJY0QkpTCAVispuUPFQVF6MaLVSkqO0K+UovRivFqtpOQIFQdF6cVotZKSK1QcFKUXo9VKSq5QcVCUXkx3y4gqSraoOChKL8bSBPUcFLtRcVCUXoxHF/tRcoSKg6L0YjQhreQKFQdF6cXoGtJKrlBxUJReTHeN+RQlW1QcFKUXY3kMGlZS7EbFQVF6Mbqeg5IrVBwUpRfj1WolJUeoOChKL0arlZRc4ag4iMi3RcSISFXUtltEZI2IrBSRc5y0T1Hyna6wksOGKK7DscV+RGQIcBawMWrbeGA2MAEYBDwnIqONMQFnrFSU/CaSkNacg2IzTnoOvwRuAkzUtouAvxtj2o0x64A1wHQnjFOU3oCGlZRc4Yg4iMiFwGZjzAdxu+qBTVHPm8Lbkr3HdSKyQEQWtLS05MhSRclvtFpJyRU5CyuJyHPAwCS7vgfcCpyd7GVJtpkk2zDGzAHmADQ2NiY9RlHcjnoOSq7ImTgYY85Mtl1EJgHDgQ/CSxwOBhaJyHRCnsKQqMMHA1tyZaOi9Ha6SlkdNkRxHUf8K2WM+dAYU2OMaTDGNBAShKnGmG3APGC2iBSJyHBgFPDOkbZRUXoLVjRJw0qK3ThWrZQMY8xSEXkYWAb4gRu0UklRUqOL/Si5wnFxCHsP0c9/DPzYGWsUpXehjfeUXKGRSkXpxXg0Ia3kCBUHRenFRMJK6jkoNqPioCi9mMg8B/0lKzajXylF6cV4wr9gzTkodqPioCi9GK1WUnKF49VKiqJkj8/r4dZZYzl9bI3TpiguQ8VBUXo51506wmkTFBeiYSVFURQlARUHRVEUJQEVB0VRFCUBFQdFURQlARUHRVEUJQEVB0VRFCUBFQdFURQlARUHRVEUJQExpvcvvywiLcCGDF9WBezIgTl2o3baS2+wszfYCGqnnThl4zBjTHWyHa4Qh2wQkQXGmEan7egOtdNeeoOdvcFGUDvtJB9t1LCSoiiKkoCKg6IoipLA0SwOc5w2IE3UTnvpDXb2BhtB7bSTvLPxqM05KIqiKKk5mj0HRVEUJQUqDoqiKEoCR6U4iMi5IrJSRNaIyM1O22MhIkNE5EURWS4iS0Xka+HtlSIyX0RWhx/754GtXhF5T0Qez2MbK0TkERFZEb6mH8s3O0XkG+H/6yUi8pCIFOeDjSJyn4g0i8iSqG0p7RKRW8K/p5Uico7Ddt4e/j9fLCL/FJGKfLQzat+3RcSISJXTdkZz1ImDiHiBu4DzgPHAZ0RkvLNWRfAD3zLGjANOBG4I23Yz8LwxZhTwfPi503wNWB71PB9tvBN42hgzFjiOkL15Y6eI1AM3Ao3GmImAF5idJzb+CTg3bltSu8Lf0dnAhPBr7g7/zpyycz4w0RhzLLAKuCVP7UREhgBnARujtjlpZ4SjThyA6cAaY8xHxpgO4O/ARQ7bBIAxZqsxZlH4732EBrN6QvY9ED7sAeBiRwwMIyKDgfOBe6M255uNZcCpwB8BjDEdxpg95JmdhJbqLRERH1AKbCEPbDTGvALsitucyq6LgL8bY9qNMeuANYR+Z47YaYx51hjjDz99Cxicj3aG+SVwExBdGeSYndEcjeJQD2yKet4U3pZXiEgDMAV4G6g1xmyFkIAATq8m/ytCX+hg1LZ8s/EYoAW4Pxz+uldE+pBHdhpjNgM/J3TXuBXYa4x5Np9sjCOVXfn8m7oGeCr8d17ZKSIXApuNMR/E7coLO49GcZAk2/KqnldE+gKPAl83xrQ6bU80InIB0GyMWei0Ld3gA6YC9xhjpgAHyI9QV4RwzP4iYDgwCOgjIlc6a1VW5OVvSkS+RyhU+1drU5LDHLFTREqB7wH/mWx3km1H3M6jURyagCFRzwcTcuXzAhEpICQMfzXGzA1v3i4ideH9dUCzU/YBM4ALRWQ9oZDc6SLyIPllI4T+n5uMMW+Hnz9CSCzyyc4zgXXGmBZjTCcwFzgpz2yMJpVdefebEpGrgQuAK0zXZK58snMEoZuCD8K/pcHAIhEZSJ7YeTSKw7vAKBEZLiKFhBI/8xy2CQAREUIx8uXGmDuids0Drg7/fTXw2JG2zcIYc4sxZrAxpoHQtXvBGHMleWQjgDFmG7BJRMaEN50BLCO/7NwInCgipeH/+zMI5ZnyycZoUtk1D5gtIkUiMhwYBbzjgH1AqBoR+C5woTHmYNSuvLHTGPOhMabGGNMQ/i01AVPD39v8sNMYc9T9A2YRqmJYC3zPaXui7DqZkPu4GHg//G8WMIBQdcjq8GOl07aG7Z0JPB7+O+9sBCYDC8LX8/+A/vlmJ/BDYAWwBPgLUJQPNgIPEcqDdBIauK49nF2EQiRrgZXAeQ7buYZQzN76Df0uH+2M278eqHLazuh/2j5DURRFSeBoDCspiqIo3aDioCiKoiSg4qAoiqIkoOKgKIqiJKDioCiKoiSg4qAc1YhIQETej/p32FnUIvIlEbnKhvOuj+7CmcHrzhGR20Skv4g82VM7FCUVPqcNUBSHOWSMmZzuwcaY3+XQlnQ4BXiRUFPB1x22RXExKg6KkoRwS4N/AKeFN33WGLNGRG4D9htjfi4iNwJfItS/Z5kxZraIVAL3EWr8dxC4zhizWEQGEJoIVU1otqtEnetKQq27Cwk1WrzeGBOIs+dyQq2njyHUj6kWaBWRE4wxF+biGihHNxpWUo52SuLCSpdH7Ws1xkwHfkuoE208NwNTTGjdgC+Ft/0QeC+87Vbgz+HtPwBeM6EmgPOAoQAiMg64HJgR9mACwBXxJzLG/INQb6glxphJhGZUT1FhUHKFeg7K0c7hwkoPRT3+Msn+xcBfReT/CLXngFALlE8CGGNeEJEBIlJOKAx0aXj7EyKyO3z8GcA04N1QeyVKSN1obxShlgoApSa05oei5AQVB0VJjUnxt8X5hAb9C4H/EJEJHL7dcrL3EOABY8wthzNERBYAVYBPRJYBdSLyPvBVY8yrh/0UipIFGlZSlNRcHvX4ZvQOEfEAQ4wxLxJa+KgC6Au8QjgsJCIzgR0mtCZH9PbzCDUBhFADu0+JSE14X6WIDIs3xBjTCDxBKN/wM0INIyerMCi5Qj0H5WinJHwHbvG0McYqZy0SkbcJ3UR9Ju51XuDBcMhIgF8aY/aEE9b3i8hiQglpq8X1D4GHRGQR8DLhNYONMctE5PvAs2HB6QRuADYksXUqocT19cAdSfYrim1oV1ZFSUK4WqnRGLPDaVsUxQk0rKQoiqIkoJ6DoiiKkoB6DoqiKEoCKg6KoihKAioOiqIoSgIqDoqiKEoCKg6KoihKAv8fBXQOPQFdcLMAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# plot the scores\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "plt.plot(np.arange(1, len(scores)+1), scores)\n", "plt.ylabel('Score')\n", "plt.xlabel('Episode #')\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "75403f40", "metadata": {}, "source": [ "### Animate it with Video" ] }, { "cell_type": "code", "execution_count": 13, "id": "7fae8a12", "metadata": {}, "outputs": [], "source": [ "def show_video(env_name):\n", " mp4list = glob.glob('video/*.mp4')\n", " if len(mp4list) > 0:\n", " mp4 = 'video/{}.mp4'.format(env_name)\n", " video = io.open(mp4, 'r+b').read()\n", " encoded = base64.b64encode(video)\n", " display.display(HTML(data=''''''.format(encoded.decode('ascii'))))\n", " else:\n", " print(\"Could not find video\")\n", " \n", "def show_video_of_model(agent, env_name):\n", " env = gym.make(env_name)\n", " vid = video_recorder.VideoRecorder(env, path=\"video/{}.mp4\".format(env_name))\n", " agent.load_state_dict(torch.load('checkpoint.pth'))\n", " state = env.reset()\n", " done = False\n", " while not done:\n", " vid.capture_frame()\n", " \n", " action = agent.act(state)\n", " next_state, reward, done, _ = env.step(action)\n", " state = next_state\n", " if done:\n", " break \n", " env.close()" ] }, { "cell_type": "code", "execution_count": 17, "id": "c8a347e5", "metadata": {}, "outputs": [], "source": [ "agent = Agent(env).to(device)\n", "show_video_of_model(agent, 'MountainCarContinuous-v0')" ] }, { "cell_type": "code", "execution_count": 18, "id": "7fd10379", "metadata": {}, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "show_video('MountainCarContinuous-v0')" ] }, { "cell_type": "markdown", "id": "ab1adc92-90a4-47a1-9ab8-65d5e947bec7", "metadata": {}, "source": [ "> Note: While I tried to execute the model with VideoRecorder in Linux, it doesn't show correctly. But in windows, it works! Perhaps, It may be bug in VideoRecoder." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 5 }