{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# PPO\n", "\n", "The goal of this exercise is to use the `tianshou` library to apply PPO on the cartpole environment. `tianshou` is the latest and most up-to-date DRL library. It is based on pytorch for the deep networks and is the only library currently compatible with gymnasium, not gym.\n", "\n", "Github: \\\n", "Documentation: \n", "\n", "Install it in your virtual environment simply with:\n", "\n", "```bash\n", "pip install -U tianshou\n", "```\n", "\n", "It will also install pytorch, which becomes double use with tensorflow, but well, storage is cheap...\n", "\n", "Let's first import the usual stuff:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "try:\n", " import google.colab\n", " IN_COLAB = True\n", "except:\n", " IN_COLAB = False\n", "\n", "if IN_COLAB:\n", " !pip install -U gymnasium pygame moviepy\n", " !pip install -U tianshou" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "gym version: 0.28.1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/vitay/.virtualenvs/tianshou/lib/python3.11/site-packages/pygame/pkgdata.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html\n", " from pkg_resources import resource_stream, resource_exists\n" ] } ], "source": [ "import numpy as np\n", "rng = np.random.default_rng()\n", "import matplotlib.pyplot as plt\n", "import os\n", "from IPython.display import clear_output\n", "from collections import deque\n", "\n", "import gymnasium as gym\n", "print(\"gym version:\", gym.__version__)\n", "\n", "import tianshou as ts\n", "\n", "import torch\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "import pygame\n", "from moviepy.editor import ImageSequenceClip, ipython_display\n", "\n", "\n", "class GymRecorder(object):\n", " \"\"\"\n", " Simple wrapper over moviepy to generate a .gif with the frames of a gym environment.\n", " \n", " The environment must have the render_mode `rgb_array_list`.\n", " \"\"\"\n", " def __init__(self, env):\n", " self.env = env\n", " self._frames = []\n", "\n", " def record(self, frames):\n", " \"To be called at the end of an episode.\"\n", " for frame in frames:\n", " self._frames.append(np.array(frame))\n", "\n", " def make_video(self, filename):\n", " \"Generates the gif video.\"\n", " directory = os.path.dirname(os.path.abspath(filename))\n", " if not os.path.exists(directory):\n", " os.mkdir(directory)\n", " self.clip = ImageSequenceClip(list(self._frames), fps=self.env.metadata[\"render_fps\"])\n", " self.clip.write_gif(filename, fps=self.env.metadata[\"render_fps\"], loop=0)\n", " del self._frames\n", " self._frames = []" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## tianshou\n", "\n", "### Structure\n", "\n", "``tianshou`` provides an implementation of most model-free algorithms seen in the course: DQN and its variants, A3C, DDPG, PPO and more. It also has several offline RL algorithms. You can see the list of algorithms here:\n", "\n", "\n", "\n", "``tianshou`` relies on several concepts, which are explained here:\n", "\n", "\n", "\n", "![](https://tianshou.readthedocs.io/en/latest/_images/concepts_arch2.png)\n", "\n", "* The **policy** is actually the DRL algorithm (DQN, PPO), not the mapping from states into actions used in the course. It relies on one (or more) neural networks called the **model**.\n", "* The interaction of the policy with the environment is done by the **collector**. By default, the collector used **distributed learning**, i.e. it uses parallel workers to interact with copies of the environment, thereby speeding up data collection. This is used even for algorithms which do not need distributed learning (DQN), as it is only beneficial. \n", "* The data collected by the collector is stored in a **buffer**, which can be an ERM for off-policy algorithms or a temporary buffer for on-policy ones.\n", "* The (distributed) data is stored in **batches**. How data circulates between the collector, the policy and the buffer during training is controlled by the **trainer**.\n", "\n", "Let's demonstrate this interaction with a dummy DQN network on Cartpole:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "env = gym.make('CartPole-v0')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Policy\n", "\n", "The first step is to create the neural network for the DQN network. It must have `env.observation_space.shape=4` input neurons and `env.action_space.n=2` discrete output neurons. Let's put a single hidden layer with 32 neurons for now:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "net = ts.utils.net.common.Net(\n", " env.observation_space.shape,\n", " env.action_space.n,\n", " hidden_sizes=[64, 64],\n", " device=device,\n", ").to(device)\n", "\n", "optim = torch.optim.Adam(net.parameters(), lr=0.001)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`optim` is the Adam optimizer in pytorch, modifying all parameters (weights and biases) of the value network. Check the doc of `Net()` if you want a more specific architecture.\n", "\n", "The output layer of the network is discrete, so that tianshou knows how to sample an action from the output (here the output neurons represent the Q-values, but it could be logits of a continuous policy). \n", "\n", "Now that we have the neural network, we can create the DQN policy object:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "policy = ts.policy.DQNPolicy(\n", " model=net, # value network\n", " optim=optim, # optimizer\n", " discount_factor=0.95, # gamma\n", " target_update_freq=1000, # how often to update the target network\n", " action_space=env.action_space, # action space\n", ")\n", "policy.set_eps(0.1) # epsilon-greedy action selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check the doc of `DQNPolicy` for additional parameters (e.g. to implement a double duelling DQN).\n", "\n", "We can now use the policy to interact with the environment as usual and visualize a trial with an untrained network:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Return: 10.0\n", "MoviePy - Building file videos/cartpole-before.gif with imageio.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Evaluation mode\n", "policy.eval() \n", "\n", "# Create a recordable environment\n", "env = gym.make('CartPole-v0', render_mode=\"rgb_array_list\")\n", "recorder = GymRecorder(env)\n", "\n", "# Sample the initial state\n", "state, info = env.reset()\n", "\n", "# One episode:\n", "done = False\n", "return_episode = 0\n", "while not done:\n", "\n", " # Select an action from the learned policy\n", " action = policy.forward(ts.data.Batch(obs=[state], info=None)).act[0]\n", " \n", " # Sample a single transition\n", " next_state, reward, terminal, truncated, info = env.step(action)\n", "\n", " # End of the episode\n", " done = terminal or truncated\n", "\n", " # Update undiscounted return\n", " return_episode += reward\n", " \n", " # Go in the next state\n", " state = next_state\n", "\n", "print(\"Return:\", return_episode)\n", "\n", "recorder.record(env.render())\n", "video = \"videos/cartpole-before.gif\"\n", "recorder.make_video(video)\n", "ipython_display(video)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The action selection is done by calling `policy.forward()` on a batch of data containing only the current state:\n", "\n", "```python\n", "action = policy.forward(ts.data.Batch(obs=[state], info=None)).act[0]\n", "```\n", "\n", "### Collector\n", "\n", "As we have seen in the DQN exercise, using a neural network with a batch size of 1 is extremely inefficient and slow. It is much better to use **distributed learning** and parallel workers to collect data. That way, we can form a minibatch of states that can be processed efficiently by the NN. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " exploration_noise=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`policy` is the exploration policy. The `exploration_noise` flag allows to switch exploration on and off. For a discrete DQN policy, this impacts the $\\epsilon$-greedy action selection scheme, but other algorithms might use another mechanism (softmax, Gaussian policies, Ornstein-Uhlenbeck, noisy parameters, etc).\n", "\n", "`ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)])` means that we create 10 copies of the Cartpole environment which will be acted upon in parallel using the policy.\n", "\n", "Let's collect some data with the collector. We can either collect a fixed number of steps (over the parallel workers) or episodes. Let's start with 10 steps:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'n/ep': 0,\n", " 'n/st': 10,\n", " 'rews': array([], dtype=float64),\n", " 'lens': array([], dtype=int64),\n", " 'idxs': array([], dtype=int64),\n", " 'rew': 0,\n", " 'len': 0,\n", " 'rew_std': 0,\n", " 'len_std': 0}" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.collect(n_step=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's weird, we apparently did not receive any reward. Let's try to collect more steps." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'n/ep': 98,\n", " 'n/st': 1000,\n", " 'rews': array([ 8., 9., 9., 10., 10., 10., 11., 12., 12., 13., 8., 9., 10.,\n", " 10., 9., 11., 10., 9., 10., 17., 10., 10., 10., 11., 10., 10.,\n", " 10., 10., 11., 8., 10., 10., 10., 10., 8., 11., 9., 10., 14.,\n", " 9., 10., 9., 10., 10., 13., 9., 11., 11., 10., 10., 9., 9.,\n", " 13., 9., 9., 10., 9., 10., 8., 10., 10., 8., 9., 10., 10.,\n", " 11., 9., 12., 10., 10., 13., 11., 9., 10., 11., 10., 12., 10.,\n", " 8., 9., 9., 9., 10., 11., 10., 9., 10., 9., 9., 9., 12.,\n", " 10., 9., 9., 10., 10., 12., 9.]),\n", " 'lens': array([ 8, 9, 9, 10, 10, 10, 11, 12, 12, 13, 8, 9, 10, 10, 9, 11, 10,\n", " 9, 10, 17, 10, 10, 10, 11, 10, 10, 10, 10, 11, 8, 10, 10, 10, 10,\n", " 8, 11, 9, 10, 14, 9, 10, 9, 10, 10, 13, 9, 11, 11, 10, 10, 9,\n", " 9, 13, 9, 9, 10, 9, 10, 8, 10, 10, 8, 9, 10, 10, 11, 9, 12,\n", " 10, 10, 13, 11, 9, 10, 11, 10, 12, 10, 8, 9, 9, 9, 10, 11, 10,\n", " 9, 10, 9, 9, 9, 12, 10, 9, 9, 10, 10, 12, 9]),\n", " 'idxs': array([7, 3, 6, 1, 2, 5, 8, 0, 9, 4, 1, 6, 7, 3, 8, 5, 0, 4, 9, 2, 1, 6,\n", " 7, 3, 8, 5, 4, 9, 0, 6, 2, 1, 7, 3, 0, 8, 9, 4, 5, 2, 6, 7, 3, 0,\n", " 1, 4, 8, 9, 5, 2, 7, 3, 6, 1, 4, 0, 9, 8, 7, 5, 2, 3, 0, 1, 4, 6,\n", " 8, 9, 5, 2, 7, 3, 6, 0, 1, 8, 4, 9, 2, 5, 7, 3, 6, 0, 1, 4, 8, 9,\n", " 2, 7, 5, 3, 1, 8, 0, 4, 6, 9]),\n", " 'rew': 10.051020408163266,\n", " 'len': 10.051020408163266,\n", " 'rew_std': 1.395125983482348,\n", " 'len_std': 1.395125983482348}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.collect(n_step=1000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alright, returns are only reported at the end of an episode. With 1000 steps (in parallel over 10 workers, i.e. each of them did 100 steps), we collected around 100 episodes of length 9 or 10, i.e. the cartpole falls right away, as expected with a random policy.\n", "\n", "Can we collect complete episodes? Yes:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'n/ep': 10,\n", " 'n/st': 71,\n", " 'rews': array([10., 9., 11., 10., 8., 9., 10., 9., 10., 10.]),\n", " 'lens': array([10, 9, 11, 10, 8, 9, 10, 9, 10, 10]),\n", " 'idxs': array([2, 5, 7, 3, 9, 0, 1, 4, 8, 6]),\n", " 'rew': 9.6,\n", " 'len': 9.6,\n", " 'rew_std': 0.8,\n", " 'len_std': 0.8}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.collect(n_episode=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "But where is the data, i.e. the collected transitions? Nowhere, because we forgot to create a buffer to store them. Let's fix that mistake." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " buffer=ts.data.VectorReplayBuffer(1000, 10),\n", " exploration_noise=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We preallocate an ERM of max 1000 transitions for each of the 10 workers. One could use a single replay buffer 10 times bigger, but tianshou requires it . Let's collect some episodes and look at the data stored in the first buffer:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ReplayBuffer(\n", " obs: array([[ 3.4645274e-02, -3.8563393e-02, 2.8588787e-02, 4.0672716e-02],\n", " [ 3.3874005e-02, 1.5613718e-01, 2.9402241e-02, -2.4285485e-01],\n", " [ 3.6996752e-02, 3.5082710e-01, 2.4545144e-02, -5.2612048e-01],\n", " [ 4.4013292e-02, 5.4559523e-01, 1.4022734e-02, -8.1096911e-01],\n", " [ 5.4925196e-02, 7.4052227e-01, -2.1966479e-03, -1.0992084e+00],\n", " [ 6.9735639e-02, 9.3567306e-01, -2.4180816e-02, -1.3925797e+00],\n", " [ 8.8449106e-02, 1.1310875e+00, -5.2032411e-02, -1.6927242e+00],\n", " [ 1.1107086e-01, 1.3267703e+00, -8.5886896e-02, -2.0011418e+00],\n", " [ 1.3760626e-01, 1.5226773e+00, -1.2590973e-01, -2.3191388e+00],\n", " [ 1.6805981e-01, 1.7187009e+00, -1.7229250e-01, -2.6477661e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],\n", " dtype=float32),\n", " rew: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n", " truncated: array([False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " act: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n", " done: array([False, False, False, False, False, False, False, False, False,\n", " True, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " policy: Batch(),\n", " info: Batch(\n", " env_id: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n", " ),\n", " terminated: array([False, False, False, False, False, False, False, False, False,\n", " True, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " obs_next: array([[ 3.3874005e-02, 1.5613718e-01, 2.9402241e-02, -2.4285485e-01],\n", " [ 3.6996752e-02, 3.5082710e-01, 2.4545144e-02, -5.2612048e-01],\n", " [ 4.4013292e-02, 5.4559523e-01, 1.4022734e-02, -8.1096911e-01],\n", " [ 5.4925196e-02, 7.4052227e-01, -2.1966479e-03, -1.0992084e+00],\n", " [ 6.9735639e-02, 9.3567306e-01, -2.4180816e-02, -1.3925797e+00],\n", " [ 8.8449106e-02, 1.1310875e+00, -5.2032411e-02, -1.6927242e+00],\n", " [ 1.1107086e-01, 1.3267703e+00, -8.5886896e-02, -2.0011418e+00],\n", " [ 1.3760626e-01, 1.5226773e+00, -1.2590973e-01, -2.3191388e+00],\n", " [ 1.6805981e-01, 1.7187009e+00, -1.7229250e-01, -2.6477661e+00],\n", " [ 2.0243382e-01, 1.9146510e+00, -2.2524783e-01, -2.9877436e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],\n", " dtype=float32),\n", ")" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.collect(n_episode=10)\n", "collector.buffer.buffers[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that each element of the transitions (s, a, r, s', done, terminated) is saved in a preallocated array of 1000 entries. The first replay buffer has only saved one short episode, so most of the data is zero.\n", "\n", "Sampling a minibatch of transitions is easy:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(Batch(\n", " obs: array([[ 0.06973564, 0.93567306, -0.02418082, -1.3925797 ],\n", " [ 0.00501526, 0.438202 , -0.00610071, -0.5498225 ],\n", " [ 0.00501526, 0.438202 , -0.00610071, -0.5498225 ],\n", " [ 0.08791052, 1.4160656 , -0.12085497, -2.0735345 ],\n", " [-0.00449122, 0.5818429 , 0.01416115, -0.8354608 ],\n", " [ 0.1239062 , 1.754971 , -0.17513801, -2.6740465 ],\n", " [-0.04072677, 0.42806664, -0.00650083, -0.5649815 ],\n", " [-0.01800847, 0.19060245, -0.03275078, -0.3052972 ],\n", " [ 0.04785302, 0.36885726, -0.01701378, -0.5969896 ],\n", " [-0.01481533, -0.01626822, -0.01261962, -0.03139872]],\n", " dtype=float32),\n", " act: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1]),\n", " rew: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]),\n", " terminated: array([False, False, False, False, False, True, False, False, False,\n", " False]),\n", " truncated: array([False, False, False, False, False, False, False, False, False,\n", " False]),\n", " done: array([False, False, False, False, False, True, False, False, False,\n", " False]),\n", " obs_next: array([[ 8.84491056e-02, 1.13108754e+00, -5.20324111e-02,\n", " -1.69272423e+00],\n", " [ 1.37792965e-02, 6.33409083e-01, -1.70971639e-02,\n", " -8.44421327e-01],\n", " [ 1.37792965e-02, 6.33409083e-01, -1.70971639e-02,\n", " -8.44421327e-01],\n", " [ 1.16231829e-01, 1.61218965e+00, -1.62325650e-01,\n", " -2.40101957e+00],\n", " [ 7.14564137e-03, 7.76768565e-01, -2.54806434e-03,\n", " -1.12365675e+00],\n", " [ 1.59005612e-01, 1.95090282e+00, -2.28618950e-01,\n", " -3.01467609e+00],\n", " [-3.21654342e-02, 6.23279154e-01, -1.78004559e-02,\n", " -8.59705389e-01],\n", " [-1.41964173e-02, 3.86175454e-01, -3.88567224e-02,\n", " -6.08126342e-01],\n", " [ 5.52301593e-02, 5.64213097e-01, -2.89535765e-02,\n", " -8.94982755e-01],\n", " [-1.51406955e-02, 1.79032415e-01, -1.32475952e-02,\n", " -3.28036398e-01]], dtype=float32),\n", " info: Batch(\n", " env_id: array([0, 1, 1, 1, 2, 2, 4, 6, 8, 9]),\n", " ),\n", " policy: Batch(),\n", " ),\n", " array([ 5, 104, 104, 109, 203, 209, 402, 601, 802, 900]))" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.buffer.sample(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The transitions come randomly from the workers, so we do not need to worry about it.\n", "\n", "We can reset the buffers with the following command:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "collector.reset_buffer(keep_statistics=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we print the buffer, data still seems to be there:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ReplayBuffer(\n", " obs: array([[ 3.4645274e-02, -3.8563393e-02, 2.8588787e-02, 4.0672716e-02],\n", " [ 3.3874005e-02, 1.5613718e-01, 2.9402241e-02, -2.4285485e-01],\n", " [ 3.6996752e-02, 3.5082710e-01, 2.4545144e-02, -5.2612048e-01],\n", " [ 4.4013292e-02, 5.4559523e-01, 1.4022734e-02, -8.1096911e-01],\n", " [ 5.4925196e-02, 7.4052227e-01, -2.1966479e-03, -1.0992084e+00],\n", " [ 6.9735639e-02, 9.3567306e-01, -2.4180816e-02, -1.3925797e+00],\n", " [ 8.8449106e-02, 1.1310875e+00, -5.2032411e-02, -1.6927242e+00],\n", " [ 1.1107086e-01, 1.3267703e+00, -8.5886896e-02, -2.0011418e+00],\n", " [ 1.3760626e-01, 1.5226773e+00, -1.2590973e-01, -2.3191388e+00],\n", " [ 1.6805981e-01, 1.7187009e+00, -1.7229250e-01, -2.6477661e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],\n", " dtype=float32),\n", " rew: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n", " 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n", " truncated: array([False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " act: array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n", " done: array([False, False, False, False, False, False, False, False, False,\n", " True, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " policy: Batch(),\n", " info: Batch(\n", " env_id: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n", " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),\n", " ),\n", " terminated: array([False, False, False, False, False, False, False, False, False,\n", " True, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False, False, False, False, False, False, False, False, False,\n", " False]),\n", " obs_next: array([[ 3.3874005e-02, 1.5613718e-01, 2.9402241e-02, -2.4285485e-01],\n", " [ 3.6996752e-02, 3.5082710e-01, 2.4545144e-02, -5.2612048e-01],\n", " [ 4.4013292e-02, 5.4559523e-01, 1.4022734e-02, -8.1096911e-01],\n", " [ 5.4925196e-02, 7.4052227e-01, -2.1966479e-03, -1.0992084e+00],\n", " [ 6.9735639e-02, 9.3567306e-01, -2.4180816e-02, -1.3925797e+00],\n", " [ 8.8449106e-02, 1.1310875e+00, -5.2032411e-02, -1.6927242e+00],\n", " [ 1.1107086e-01, 1.3267703e+00, -8.5886896e-02, -2.0011418e+00],\n", " [ 1.3760626e-01, 1.5226773e+00, -1.2590973e-01, -2.3191388e+00],\n", " [ 1.6805981e-01, 1.7187009e+00, -1.7229250e-01, -2.6477661e+00],\n", " [ 2.0243382e-01, 1.9146510e+00, -2.2524783e-01, -2.9877436e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00],\n", " [ 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00]],\n", " dtype=float32),\n", ")" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "collector.buffer.buffers[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But sampling returns an error:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "ename": "ValueError", "evalue": "probabilities contain NaN", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[16], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mcollector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbuffer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msample\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m10\u001b[39;49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/.virtualenvs/tianshou/lib/python3.11/site-packages/tianshou/data/buffer/base.py:313\u001b[0m, in \u001b[0;36mReplayBuffer.sample\u001b[0;34m(self, batch_size)\u001b[0m\n\u001b[1;32m 306\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msample\u001b[39m(\u001b[38;5;28mself\u001b[39m, batch_size: \u001b[38;5;28mint\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Tuple[Batch, np\u001b[38;5;241m.\u001b[39mndarray]:\n\u001b[1;32m 307\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Get a random sample from buffer with size = batch_size.\u001b[39;00m\n\u001b[1;32m 308\u001b[0m \n\u001b[1;32m 309\u001b[0m \u001b[38;5;124;03m Return all the data in the buffer if batch_size is 0.\u001b[39;00m\n\u001b[1;32m 310\u001b[0m \n\u001b[1;32m 311\u001b[0m \u001b[38;5;124;03m :return: Sample data and its corresponding index inside the buffer.\u001b[39;00m\n\u001b[1;32m 312\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m--> 313\u001b[0m indices \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msample_indices\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 314\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m[indices], indices\n", "File \u001b[0;32m~/.virtualenvs/tianshou/lib/python3.11/site-packages/tianshou/data/buffer/manager.py:180\u001b[0m, in \u001b[0;36mReplayBufferManager.sample_indices\u001b[0;34m(self, batch_size)\u001b[0m\n\u001b[1;32m 178\u001b[0m sample_num \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mzeros(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuffer_num, \u001b[38;5;28mint\u001b[39m)\n\u001b[1;32m 179\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 180\u001b[0m buffer_idx \u001b[38;5;241m=\u001b[39m \u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrandom\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mchoice\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 181\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbuffer_num\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mp\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_lengths\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m/\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_lengths\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msum\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 182\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 183\u001b[0m sample_num \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mbincount(buffer_idx, minlength\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuffer_num)\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# avoid batch_size > 0 and sample_num == 0 -> get child's all data\u001b[39;00m\n", "File \u001b[0;32mmtrand.pyx:954\u001b[0m, in \u001b[0;36mnumpy.random.mtrand.RandomState.choice\u001b[0;34m()\u001b[0m\n", "\u001b[0;31mValueError\u001b[0m: probabilities contain NaN" ] } ], "source": [ "collector.buffer.sample(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Interaction loop\n", "\n", "We do not even need to actually sample the buffer, because `policy.update()` takes the buffer and a batch size as input. The following code implements DQN on Cartpole, with an okayish choice of hyperparameters. The main interaction loop consists of:\n", "\n", "1. `collector.collect()`: Collect 100 samples using the 10 workers and store them in the ERM.\n", "2. `policy.update()`: Sample `repeat=10` minitaches of 64 transitions from the buffer and learn from them.\n", "3. `test_collector.collect()`: Test the performance by running 10 episodes without exploration.\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGwCAYAAACD0J42AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABwVElEQVR4nO3deXwU9fkH8M/sbnZzJySQhEACARRQblBEKIIgh4paqb+KqHgfBa3wa1Va7/5sqLbVaq3aVqEqVKtVbPFoEQVUAgKKCCoCcpOEI+RONnvM74/NTGZmZ3Y3sMdM8nn7yss9Zne/uySZJ8/3+T5fQRRFEUREREQmYkv0AIiIiIi0GKAQERGR6TBAISIiItNhgEJERESmwwCFiIiITIcBChEREZkOAxQiIiIyHUeiB3Ay/H4/Dh8+jIyMDAiCkOjhEBERUQREUURdXR0KCwths4XOkVgyQDl8+DCKiooSPQwiIiI6CQcOHEDPnj1DHmPJACUjIwNA4A1mZmYmeDREREQUidraWhQVFcnn8VAsGaBI0zqZmZkMUIiIiCwmkvIMFskSERGR6TBAISIiItNhgEJERESmwwCFiIiITIcBChEREZkOAxQiIiIyHQYoREREZDoMUIiIiMh0GKAQERGR6TBAISIiItNpV4BSWlqKs846CxkZGcjLy8Nll12GHTt2qI5pbm7G3LlzkZubi/T0dMycOROVlZWqY/bv34+LLroIqampyMvLw89//nN4vd5TfzdERETUIbQrQFmzZg3mzp2L9evXY+XKlfB4PJgyZQoaGhrkY+bPn49///vfeP3117FmzRocPnwYl19+uXy/z+fDRRddhJaWFqxbtw5/+9vfsGTJEjzwwAPRe1dERERkaYIoiuLJPvjo0aPIy8vDmjVrMH78eNTU1KBbt25YtmwZfvSjHwEAvv32WwwcOBBlZWU455xz8N577+Hiiy/G4cOHkZ+fDwB47rnncM899+Do0aNwOp1hX7e2thZZWVmoqanhZoFERDHW1OJDitMOAPD7RbT4/EhOsquOOdHQgjSXAycaW+AXRaQk2VHvVmfGu6a7VI/z+UU0eXyobmxB96wU2G3hN5CLptpmD2qbPOiW4YLPL6KmyYOCzGR5I7sGtxcnGlsAAMlJdnRNd8mPrW5sQb3bi1SnAw67gNomT9Dz220C7IKAFp8fAJDucgR9JrGi/DzdXh+O1bcg3eVAXXPbONNdDjR7/EhOsqnGZRME7Kiow4T+3SLa1K892nP+PqXdjGtqagAAOTk5AIDNmzfD4/Fg8uTJ8jEDBgxAcXGxHKCUlZVh8ODBcnACAFOnTsXtt9+O7du3Y/jw4UGv43a74Xa7VW+QiIhib/kXh3DXa1uw6PLBuPLsYsz6y3ps3FuFz++/ANmpgT8oV35diVte3oRwf+7mpjmx5u6JSHcFTj0/fr4Mm/adAACMLsnBbRP6YnCPLFUgECvbDtXgh3/6FB5fIJgCgCaPD7NHF+PRHw7GgapGTHtyLRpafKrHXT+2N/p0TcMjK76Gx3fSf9/H3Ll9c7Hs5nOw+2g9Zv15PY7UucM/SGPG0EI8PSv4nBwvJx2g+P1+3HXXXRg7diwGDRoEAKioqIDT6UR2drbq2Pz8fFRUVMjHKIMT6X7pPj2lpaV4+OGHT3aoRER0ku56bQsA4N43v8KVZxdjw54qAMBHO47gh8N7AgA+23NcNzixCUCSPVBJ4Pb6cbyhBfuPN+KMwky0eP1ycAIAG/ZUYcOeKgwoyMD7d42P6XuqafLgt//dIQcYTZ62IGTLgWoAwD82HUBDiw92mwABgNcfOHbxp3sNn9flUFdNtPj8EMXA7W6vX77d6bAhlrkit9ePdbuPY0dFHRZ/ukcVnAit/yYtivEAgWyPwybIYwaAs3p3ieEowzvpAGXu3LnYtm0bPvnkk2iOR9fChQuxYMEC+XptbS2Kiopi/rpERKRPeYI7Vt+ie8zTs0bgoiHdAQBjF32IQ9VN8LROdxyr1/+L/tuKOhw80YieXVJPaXwrth7GvuONmDuxX9B9V/1lPbYf1s/ES+Pb8H0gEFt0+WB0zXDh+sUbVcddfU4xHpxxJn797jfYdaQev71iKPIzk1XHNHt8EEUgxWnHwRONuOefW/E/o4pw6bAep/Tewrnyz2VY/30Vth+uwZcHAzMdj80cgtMLMjC0ZxYEQcDbWw7hqVU7sftooIb0kUvPxOzRvQAAr286gPKaZlzdej1RTipAmTdvHlasWIG1a9eiZ8+e8u0FBQVoaWlBdXW1KotSWVmJgoIC+ZjPPvtM9XzSKh/pGC2XywWXK/YpPyIiikyLYnpDG2z84cph+LaiDtMGtf1Od7ZmF6R6jFBTDp/vrz7lAGXesi8AAOf0ycXIXupMgFFwAgAenwhRFPHdkToAwJmFWchMaTtVZqcm4f6LzsDlI3pAEAQ8OONMw+dS1tv07JKKpTedc1Lvpb2yUpIABP5ddlYG3sfY07qiR3aKfMylw3rgkqGFKFn4LgDg9PwM+b4rRpkjAdCuAEUURdxxxx146623sHr1apSUlKjuHzlyJJKSkrBq1SrMnDkTALBjxw7s378fY8aMAQCMGTMGjz76KI4cOYK8vDwAwMqVK5GZmYkzzjgjGu+JiIhizOtry6AcbQ02ZgwtxI9G9sR5p3fDpZrjna1TPZ7WzEtlbbPhc391sBqXDC0M+fpfH67FXz/5HgsuOB09slNUxZzKtR+7jtSh2ePDuX1zAQB3/P0L1fPkpDlR1dCWAfL4/Dha70Z1owc2AejTLQ3JSXbceX4/OOw23DnptJDjMoM0Z+DU/u5XFfD6RfTKTUVhVnLQcYIgYNnNo3GgqhFn9c6J9zDDaleAMnfuXCxbtgxvv/02MjIy5JqRrKwspKSkICsrCzfeeCMWLFiAnJwcZGZm4o477sCYMWNwzjmByHHKlCk444wzcM011+Cxxx5DRUUF7rvvPsydO5dZEiIii/AoApTjrSf4W8f3waAeWbrHJzlaV5S0Pu5AVaPq/qdmDYffL+Ku17bgna3luHf6wJCremb88RP4/CI++vYIbIKAG39Qgp9M6IfK2mZMe3KtfNyD/9qOZo8fv7xwIM4fmIcVW8tVz3NaXrpcVyO9r0MnmgAEVsJIWZAFU/qH/kBMRFpx9eXBagDAOSW5hqtxzu3bFegbr5G1T7v6oDz77LOoqanBhAkT0L17d/nrtddek4954okncPHFF2PmzJkYP348CgoK8Oabb8r32+12rFixAna7HWPGjMHVV1+Na6+9Fo888kj03hUREcWUcgVLY+sSVWl1jh5tBqVs93H5vv/OH49LhhZi2qACJNkFHK5pRnlNU8jX97UWrZ5o9OB4Qwseez/QNPTZ1btxorFtKW2zJ/B6j777DSb9bk3Q8xTnqKeSPD4RFTWB7E5+pjX/aE5r/XeQEklpIf5dzKzdUzzhJCcn45lnnsEzzzxjeEyvXr3w7rvvtueliYjIRJRFsq2xQsiMh7SaR6pB2XW0HgDw6i3nyPUPyUl2ZCYn4XhDC+qa9fuFPLVqJ9Z8d9TwdZas2xvxewCA7pqpD4/Xj4rW6acCnWkRK0h1qnvUJCdZc1cba46aiIjiSht8KKd4RAQilFA9vaQiWelxUoCjPZlmthZ4LtuwH//zfBmOaGpVfr/yO2xWLE9WMro9lPRk9d/pHr9fzqAUZKboPcT0pBoUibapnlUwQCEiorAcIQIUKYNiCxGhSFM8ja2Nz6THOzW9QzJaA4aX1+/DZ3uq8LM3tuJQdejpHkmkxymlaE7eHp+I2tbsjbQaxmpcmoyJtj+LVVhz1EREFFfSFI1EWYMiTf+HClCkx//yrW1ocHvlDIr2eTM0GY213x3FuN98iANVjXJmw0iNTrv5UP5nVM+g7ILPL8Ld2rhNGzxZhXZ6jBkUIiLqsBz2SDIoxo+XOrECgS6tUi2KUxOgZCYHZy1EEXhuzW6cU7oq5Bj3HWsIeb/Wb2YO0T15S1meJHt89waKlmaPuj0/MyhERNRhOWzaDIoyQJFqUIxP6G5v20lz3e7j8uoaoykeraUb9ocd418/2RP2GMlt5/WFIAhBUzwA0GjxDMpsTQdYZlCIiKjD0iQ65CkeURTl5ayhMihuT1tAs/LrSvly8BRPdOs+7p0+IOTtKc7gk3dTi1d3bFZRkJWMWWcXy9e5ioeIiDoUZWsJQbO9nTRFo+w+EaoGpdnr071dO42Sm+5s7zCDvDBnFJwOG+6afBpuOy+4C9n407vJl/WmPxrc0hSPdU+RykyUy2HNDIo1u7cQEVHMKetGtKSGa35FhBIqQFFmUJS00yjd0k+9OdqkgfnY+uCUoKmNCf27YdqZBZh6ZtseQTadtE+jnEGxZg0KoF6+rV3VYxUMUIiISJdXsVJHG3tINSjKGEYIcR50G2VQNLUteZnRaY6mV3cxsHsmrlRMfQDqoMrlsMHt9SuKZK15YgfUvVCsmkGx7qdPREQx5fHrZz2AtuxKpBmUZp0MisMmBGUwTjWD8pMJwVM6z18zEjOGFmLuxH5B9ymLZKWsQ4cIUBTt7a1ag8IMChER6VJlUDT3SSt21DUoxs/VvyBDbiEv0QsACrNPLoPy5QNTsPd4A84szAy6b6pmWkfp9Px0zDq7CF3TXXht4wEAQEMHmOIZ3adtd+JQeySZmTXDKiIiijmvpteJ3n5s/hCFtEqP/WhI0G16CZfsVCeGF2eHHNfZJTlBt6W57BhalA1HO7MegiCg9PIh+N8p/eWASXpL2h4tVtK3Wzp+e8VQ/O8Fp6NXblqih3NSrPvpExFRTHkUBSZevx96+8WqApQQCYf8zGQ8NOOMiF73if8ZZnjf4uvPwtKbRmPBBaerbm9vYKJHW7CbZNE+KJIfjeyJOyadluhhnDRrf/pERBQzygyK1y9qsiUB/giXGQOAPcIgonfXNHz0swlBt6+793xM7J+HJLsNd046TXc651Rop3SsXIPSEVhzYoqIiGJOud+O1yftWawmqopkQz+fdsPBUEq6puH6sb2xYms5Zp1VhMyUJBRmq3cXPn9AHrYfro34OcMJyqBYuAalI2CAQkREurx+v+qy3hRPpI3aAMDejgAFAB6ccSYeuPgMwxb60W7hrq05sXINSkfAT5+IiHR5NRkUvXqTSGtQgPZlUNqe0/gxV5/TC/3y0nHn+cHLh09GcAaFp8hEYgaFiIh0eTQ1KHqkmwUhdDABBGdQTnUCJSslCR8sOO8Un6WNU9PQTLuDM8UXw0MiItKlDUqUAYtEqkEJN70DBO+InOI019/InOIxF376RESkSxuQKBvLalfxRDJ7o82gpLnM1YJdu3Egp3gSi58+ERHpUtagAIDeOh6pBiXc9A4QXIOSarYMSgfrg2J1/PSJiEiXV7MXj3LGR5Rvk6Z4wj+fXVPTkeY0VwZFOaUjCEBqlFcJUfswQCEiIl0eTQbFr7POWJSneNqfQUkz2R4xygxKutMRtJEhxRcDFCIi0qWd4tHvJBt5kay2BmVg9+h2gj1VygAlMyUpgSMhgAEKEREZ0E7x6LWSVS4zDke7iufOSdHpXxItygAlI9lc2Z3OiAEKERHpCp7iCT7mZDMoj80cYr4iWTsDFDNhgEJERLq82mXGqq6xgWBDbEeRrLIGxYwBgDqDwimeRGOAQkREujx+4xoUKTBpm+JpXwYl3YQBiotTPKbCAIWIiIL4/SJavOoMiniKy4yVrePNmKFQZlDMtsKoM+K/ABERodnjw2sbD2BC/25IdzlwwRNrUdXQojpGnUFpva01hmlvozYzZiiUNShsc5945vsOISKiuPvjh7vwx492hTzmVDMoykLaDBNmKJQZFG3be4o//gsQERE+3X0s7DHKihRR07QtklU8LYqiW7NP8Wjb3lP88V+AiIig0yQ2iF4n2fYsM052tLWOT04y3+lHOa3DjQITr93/AmvXrsWMGTNQWFgIQRCwfPly1f2CIOh+Pf744/IxvXv3Drp/0aJFp/xmiIjo5EQQn6iyJnINSjsatfXumoZfXjgQT/x4aEQ1K/HGDIq5tHsSsKGhAUOHDsUNN9yAyy+/POj+8vJy1fX33nsPN954I2bOnKm6/ZFHHsHNN98sX8/IyGjvUIiIKI6Uq4798jLjyDMoAHDz+D5RH1e0KIMSZlASr90ByvTp0zF9+nTD+wsKClTX3377bUycOBF9+qi/KTMyMoKOJSIi89JbxdOeRm1m52IGxVRi+i9QWVmJd955BzfeeGPQfYsWLUJubi6GDx+Oxx9/HF6v1/B53G43amtrVV9ERBRFERShKLfmEaFu1BZpBsXMnPa2GhkXMygJF9N1Xn/729+QkZERNBV05513YsSIEcjJycG6deuwcOFClJeX4/e//73u85SWluLhhx+O5VCJiDqll8v2IivVGVkNCvT6oAQudID4BC5F4W6SowO8IYuLaYDy4osvYvbs2UhOTlbdvmDBAvnykCFD4HQ6ceutt6K0tBQulyvoeRYuXKh6TG1tLYqKimI3cCKiTmDdrmO4/+3tAIDT8tLDHq/fByXw/46RQVE2arOHOJLiIWY5rI8//hg7duzATTfdFPbY0aNHw+v1Yu/evbr3u1wuZGZmqr6IiOjkiKKITXurcNVfN8i37TxSL1/uluHC3Il9gx7n14lQxHYWyZqZukjW+u/H6mIWoLzwwgsYOXIkhg4dGvbYLVu2wGazIS8vL1bDISKiVv/68jB+9FyZ4f1JNgFen17Pk7bL2hqUDhCfqAMUFskmXLuneOrr67FrV1s75D179mDLli3IyclBcXExgMAUzOuvv47f/e53QY8vKyvDhg0bMHHiRGRkZKCsrAzz58/H1VdfjS5dupzCWyEioki8veVwyPsddhvcmo0CAYO9eDpoBsX678b62h2gbNq0CRMnTpSvS7Uhc+bMwZIlSwAAr776KkRRxKxZs4Ie73K58Oqrr+Khhx6C2+1GSUkJ5s+fr6oxISKi2NG2qddy2AR4fMEBiqpRW+v/5QClAyQcuEGgubQ7QJkwYULYb+5bbrkFt9xyi+59I0aMwPr169v7skREFAVNLT5sOxy6VYPNMEBRXhZVtwkdIOegDFAiWdVEscVwkYioE5nz4mc4WucOeYwoihHUoEi3dZxGbbaO8CY6EAYoRESdyGd7q8IeI4rAbRMCq3h+NLKnfLt+DUrg/2bcW+dUKDc2pMSIaR8UIiKyHhHA6fkZ+PqRqUiy2/DG5oMAtAGKdi+euA8zJn4+tT92VtZhdElOoofS6TFAISIiFSn4SHU64PWp+tsHXZRilo6wigcA5k7sl+ghUCtO8RARdRLhFjhIlLUmyqkbVQ1KB2zURubCAIWIqJNobPFFdJzPrx/IqKZ4OmCjNjIXBihERJ1EdZMnouMaW9p2l1fGHR29URuZCwMUIqJOor7ZG/4gAA1u/UyLqDPF05EatZG58FuKiKgTOHiiEVsOnIjo2BZFYawyMeLX6STb0YpkyTy4ioeIyKKaPT64HLaIepCM+81Hp/x6ep1kpaClo/VBocRjBoWIyIIOVzdhwP3vY96yL8Ie69VpWx8p9Soe4+6yHaUPCpkHAxQiIgv6+2f7AQDvfFUe9lijmpL20ltmzCJZihUGKEREFtTijTwrUtsc2eqdpTeNRk6aE0/PGq57v6izzFjsYJ1kyTxYg0JEZEHudgQo97+9LaLjxvbris33TTasJ1FO8HT0vXgo8ZhBISKyIGWAcqS2OWSX2NU7jkb8vHqBhnSTsgbF30H34iHzYIBCRGRByimes3+9Cg+8vT3mr6mqQdHcJoARCkUXAxQiIgtq0azMeXn9vlN6vq7pLsP7pNBDlaXR7sXDswlFGWtQiIgsyO2JzsocAFj9swnIz0wOe5xeoza/n31QKDYYoBARWZA2g3KyCrOS0btrWshjBEEARBF+xUtKmRMpUOEyY4o2JuWIiCwo0mXGRjsTnwxR5zIbtVGsMEAhIrKgSAOUcMdFEr5IsYfebsYiG7VRjDBAISKyoEineJZuCC6ePX9Annw5xOrkIHqN2tr24on8eYgiwQCFiMiC9KZu/vrx90G3/d873wTd9qvLBsmXxQhyKG19UNpu0zZqYwaFoo0BChGRBemVlugFI3q6hVhSHIqoG6CwURvFBgMUIiILMuoce8Vz6/BS2V4AwI6KOtV9kwfm48XrRsHpaPvVH0kNrdSETW83Y5EZFIoRLjMmIrIgvWABADbuPYGNe0/ggjPyMfXJtar7/jpnVNDxTS2R91NR1aBIre7ZB4VihBkUIiILCpf5mP/aloiep97tDX9Qa+yhfEl/UA1KRC9HFDEGKEREFmSUQZGs/74q+q/pN17FwykeijYGKEREFuSPYgO2cNr6oLTdFtwHJW7DoU6CAQoRkQV54xigSHT34pF2M2YGhaKMAQoRkQV5facWoKS7Il8jIcUeoZcZM0Ch6GKAQkRkQZ4IOsleNLi7fPmPVw1X3XfbeX0AAFPOyI/4NdV1L1INSuAap3go2rjMmIjIgiIJUEb26oLbzuuLgqxkdMtQN2e7fUI/nF2SiyE9s8I+j9QHRRWeaGpQmEChaGt3BmXt2rWYMWMGCgsLIQgCli9frrr/uuuugyAIqq9p06apjqmqqsLs2bORmZmJ7Oxs3Hjjjaivrz+lN0JE1Jl4IpjiSU6yY3DPrKDgBADsNgFnl+QgOcke8Wvq16Bwiodio90BSkNDA4YOHYpnnnnG8Jhp06ahvLxc/vr73/+uun/27NnYvn07Vq5ciRUrVmDt2rW45ZZb2j96IqJOyusPn0FJTorOLL5+DYqouo1FshRt7Z7imT59OqZPnx7yGJfLhYKCAt37vvnmG7z//vvYuHEjRo0KdDV8+umnceGFF+K3v/0tCgsL2zskIqJOobymCdcv3ohj9e6IMyjRpO6D0noba1AoRmJSJLt69Wrk5eWhf//+uP3223H8+HH5vrKyMmRnZ8vBCQBMnjwZNpsNGzZs0H0+t9uN2tpa1RcRUWfzy7e24duKOhyrb4no+KhlUFr/r1eDwikeipWoByjTpk3DSy+9hFWrVuE3v/kN1qxZg+nTp8PnC+z3UFFRgby8PNVjHA4HcnJyUFFRofucpaWlyMrKkr+KioqiPWwiItM7Wudu1/HJjihnUBRzPH55ioeN2ig2or6K58orr5QvDx48GEOGDEHfvn2xevVqTJo06aSec+HChViwYIF8vba2lkEKEXU67Q0Ckp3RCVCk+hJ/8CpjNmqjmIl5H5Q+ffqga9eu2LVrFwCgoKAAR44cUR3j9XpRVVVlWLficrmQmZmp+iIi6mzaGwREO4MichUPxVHMA5SDBw/i+PHj6N490DBozJgxqK6uxubNm+VjPvzwQ/j9fowePTrWwyEisix7O1MomSnRSZK37cWjCFBENmqj2Gr3d299fb2cDQGAPXv2YMuWLcjJyUFOTg4efvhhzJw5EwUFBdi9ezfuvvtu9OvXD1OnTgUADBw4ENOmTcPNN9+M5557Dh6PB/PmzcOVV17JFTxERCG0NwgozEqJ6uurNguU/i9lUBihUJS1O4OyadMmDB8+HMOHB9omL1iwAMOHD8cDDzwAu92OrVu34pJLLsHpp5+OG2+8ESNHjsTHH38Ml6utUdDSpUsxYMAATJo0CRdeeCHGjRuHP//5z9F7V0REHVB7p3iiFjREsBcPZ3go2tqdQZkwYYJqHlLrP//5T9jnyMnJwbJly9r70kREnZo9gijg6VnD8fSHO3HjuJKov766BkU7xcMIhaKLmwUSEVmETfMb+4U5o4KOGdg9E/+dfx5+fFZx1F5XvwZFfRtneCjaGKAQEVmENktxen5G0DEpUVparEe/BiXwf2ZQKNoYoBARWYQ2CNALRpId0f+1LtW+iLp9UETVMUTRwgCFiMgitNMoLp1gJLYZlFA1KDF7WeqkGKAQEVmEsg9Kj+wUZCQn4a7Jp6mOiXZzNkC5m7FxDQrjE4o2BihERBahnEb5vx8OAgDcdl5f+bbLR/SIaT8SZQ1K0F48TKFQlDFAISKyCGUMkNS6pMehuPH6c6O/tBgwWMUj/Z978VCMMEAhIrIIZZGsFJe0t/39qfCHaNTGBApFGwMUIiKL0FvKq8xcdElLivEIgpt0slEbxUp0dpIiIqKYM6rzeP6akaht8qBnl9SYvK4UBPn96ttFUWyrQWF8QlHGAIWIEqbZ48Ntr2zGead3w/VjY1M/0ZEYBQFTzyyIy+v7NduciGJbBoU1KBRtnOIhooR5fdMBrN5xFA//++tED8USfP7gKZZ4aCuSVd8uQlmDwgCFoosBChElTL3bl+ghWEZjixcrtpbL13PTXSGOjg3tRrGiKLJRG8UMAxQiIgt4fs338uV+eenoXxC8D0+sSMmRoCkeKPqgMINCUcYAhYjIAiprm+XLN41LTL2OdoIpUIMi7cUT//FQx8YAhYjIApRFqPHv2tq6iieoBkWUV/Ywg0LRxgCFiMgC7Irf1vYEBQP6q3g4xUOxwQCFiMgClEGJwx7fYEBvs8DA9baOsiySpWhjgEJEZAGqKZ4EZStEvSkeuQaFEQpFFwMUIiILUAYl8dx/B9DfLBDQTvHEdUjUCTBAISIygbpmT9AUipIyAEhULKDfqC1wmRkUijYGKERECbbtUA0GP/Rf/PyNrYbHKFfueOPcUda4BoV78VDsMEAhooQRdXbH7YyeW7MbAPDG5oOGxygTFF7trn1xElyD0tYbhat4KNoYoBARJZjTEf5XsTI48PjinEGR+6AY16AwPqFoY4BCRJRgrggClBZvW9bEG+cARRI0sySCjdooZhigEBElmNMe/lexclon3lM8cg2K5nblMmMGKBRtDFCIiBIskikej7ctPIj3FI+EjdoonhigEBElWJIig+L16WdHPIrbR/bqEvMxKRn2QQHYqI1ihgEKEVGCKTMojR6f7jEtrQHKhYMLMKwoOx7DCqKdWfL5RTZqo5hhgEJElGDK+o1Gt36AImVQzu3bNS5jUpKyI9oMil8U5cLZ+O+wTB0dAxQiogRTTus0tHh1j5HqTiIpqI0VbR8Un1+UAycHAxSKMgYoREQJ5lGs3w2XQUlyJC4Q0DbW8/lFeclzUgIDJ+qY+B1FRJRgkWRQpD4oiQwEtH1Q/GJbBoUBCkVbu7+j1q5dixkzZqCwsBCCIGD58uXyfR6PB/fccw8GDx6MtLQ0FBYW4tprr8Xhw4dVz9G7d28IgqD6WrRo0Sm/GSIiqzlc3YSlG/bL1xsNAhRp/51EBAJSiYy2BkU1xWPnFA9FV7u/0xsaGjB06FA888wzQfc1Njbi888/x/3334/PP/8cb775Jnbs2IFLLrkk6NhHHnkE5eXl8tcdd9xxcu+AiMjCLnzqYzS2tE3rNISZ4klkDYpeBkUOnGzMoFB0Odr7gOnTp2P69Om692VlZWHlypWq2/74xz/i7LPPxv79+1FcXCzfnpGRgYKCgva+PBFRh1Ld6FFdN8qgJHKKx2g3Y6+iBoUZFIq2mH+n19TUQBAEZGdnq25ftGgRcnNzMXz4cDz++OPwevV/KAHA7XajtrZW9UVE1BGFy6AkMhDQneLxJ35c1DG1O4PSHs3Nzbjnnnswa9YsZGZmyrffeeedGDFiBHJycrBu3TosXLgQ5eXl+P3vf6/7PKWlpXj44YdjOVQiIlMwyqB4ErhaRtrNWLvM2OMT5ds4xUPRFrMAxePx4H/+538giiKeffZZ1X0LFiyQLw8ZMgROpxO33norSktL4XK5gp5r4cKFqsfU1taiqKgoVkMnIkoYZT2KkhlrUNyKrrfMoFC0xeQ7XQpO9u3bh5UrV6qyJ3pGjx4Nr9eLvXv36t7vcrmQmZmp+iKijs3j86O22RP+wA7mT6t3Y/O+qqDbE9kHxagGpdnbtjyay4wp2qL+HSUFJzt37sQHH3yA3NzcsI/ZsmULbDYb8vLyoj0cIrKoyb9fgyEP/RfH692JHkrczXy2LOg2c/RBUQcoygwKAxSKtnZP8dTX12PXrl3y9T179mDLli3IyclB9+7d8aMf/Qiff/45VqxYAZ/Ph4qKCgBATk4OnE4nysrKsGHDBkycOBEZGRkoKyvD/PnzcfXVV6NLl/ju0ElE5rXveCMA4NPdx3HJ0MIEjybxpOW8iZjikXI22hoUd2vQJAiAna3uKcraHaBs2rQJEydOlK9LtSFz5szBQw89hH/9618AgGHDhqke99FHH2HChAlwuVx49dVX8dBDD8HtdqOkpATz589X1ZgQEZGaGTq2BmVQpKwOC2QpBtodoEyYMCFoHlIp1H0AMGLECKxfv769L0tE1OF4FC3uQxFFUbGKJxE1KPqreJpbp3hYIEuxwLCXiChBmjz6K3a0pOAEABwmzKBwJ2OKBQYoREQJ0hxxgNKWaUlkDYp2mbE0fhbIUizwu4qIKEGaWyKb4lEGKImY4pEY1qAwQKEY4HcVEVGCNHsjy6C0+BK8Wkbug6K+2e1lDQrFDgMUIjK1cIX3VhbpFI9X0eZeKlhNBO2/hdvDDArFDr+riIgSpMmgrb1WotvcG9WgsEiWYokBChFRgihbxb943SjD49p6oCQ2EDDqJJvIlUXUcfG7iohMLZFTGrEmZVBGFGejIDPF8LgWb+J2MgYUfVA0t0s1NIkOnKhjYoBCRJQgUpFpcpIdTofxr+MWE3SRBYxrUDjFQ7HAAIWIEqYD179GRCqSTUmyh6wvaXR7AQCpTntcxqUVrgYl0YETdUz8riIiU+vIq3ikKZ7kJDuSHMZZiPrWACXN1e7dSaIquA8KG7VR7PC7iogSpgOXl0REKpJNTrKHPMk3tAQClPQEBSjSv5Pfb9DqnjUoFAMMUIgoYTpwciQibRkUW8gApd4dOC5RAYrEcLNA7mZMMcDvKiKiKBBFEc+t2Y11u49F/JhmRZGsK0SRbEOCp3gEGKzi8Zhj+TN1TIkNx4mIOoj3t1Vg0XvfAgD2LrooosdIq2BSwkzx1DdLUzyJKZKVGNWgsA8KxQK/q4iIomDv8cZ2P0Yqfk112UPusZPoIlm5BsVos0AuM6YYYIBCRJQgNU0eAEBWShIAYN7EfgACNSlKiZ7ikUg1slLfE+7FQ7HE7yoioijQrkjadqgGOyrqQj6mplEdoFwzpheAts0BAWDd7mN4ffNBAEBGcoJn5VuHJWV7uJsxxRIDFCKiKKtt9uDipz/B1CfXwqftbqagzaBImQivX5Qfd9VfNsjHpzkTnUFRt9yX3hozKBQL/K4iIkv4+2f78ZOlm9Gi2GDPrI7Xt8iXvX7j8UoBSnaKE0AgUJHKOY43uIOOT1wNSmBQUoCizZiw1T3FAgMUIrKEhW9+hXe/qsAbrdMdZnMyp+jqpkAgI2VQ7DYBuekuAMCR2uAAJeF9UFr/rw1IuIqHYoHfVURkKXXNnkQPoV2MmtG5vT65j4gUoABAXkZrgFLXHPSYtAQtM5bCEem9aPcNYh8UigUGKERkKWZtj9/eprjHWqeBkuyCqvhVDlB0MigJL5Jtpd15mZ1kKRb4XUVEpqPcIFCbgbBCe3xlDKXtHSI5WhcIQLqmu2BTTJnkpAUClKrGlqDHJLoPikQboITa6JDoZDFAISLTsUIQomV0ijZ6L1KAImVMJFIPFL1i4ET3QZEEBSjMoFAM8LuKiEwnVHxi1ikeI0bvRaox6aYJUKSTv1svQEnwMmOJtgaFfVAoFhigEJHpiFZMoRgwmuKRliJ3TVcHKC5HoBBWyqD0yE6R7wvVDj+Wwk3xcBUPxQK/q4jIFFR1JyGOE05qQW/sGWV2jGKtxpZAF9ZUTVakLYPiU11/8bpRURhldGgbs3EvHooFBihEZArKE3moBIrY7vUyiWWUDZICEO2+Oy6HugbF7Qkcp820xJM2KHQxg0JxwO8qIqIYMgq2pB4o0pSOxKWpQWnx6R+XSEFFsqxBoRhggEJEpiCqLou6lwHzTvEYMcr3SJmR8BmUwP+1QUE8BdWgBDVq46mEoo/fVURkOlaskTUKnIyKZKUMSXKSNoMSuL5xb5XqOO20SiIFN2qzVtBI1mCe73gi6tQiXbljhWXGqmyQ4RRPIIOiDTykk/+x+ha8+tl+xRRPAjMomuvBUzw8lVD0tfu7au3atZgxYwYKCwshCAKWL1+uul8URTzwwAPo3r07UlJSMHnyZOzcuVN1TFVVFWbPno3MzExkZ2fjxhtvRH19/Sm9ESKytkhO6uHuMwt1J1z9ATfLRbL6NSgAcO+bX8mXtcclktOuHgv7oFAstDtAaWhowNChQ/HMM8/o3v/YY4/hqaeewnPPPYcNGzYgLS0NU6dORXNz28ZXs2fPxvbt27Fy5UqsWLECa9euxS233HLy74KIOhSrrdQB1Jkdv3JFksHxUm1JUA1KUvCvZUEAUhIZoGjSVtyLh+Kh3W0Jp0+fjunTp+veJ4oinnzySdx333249NJLAQAvvfQS8vPzsXz5clx55ZX45ptv8P7772Pjxo0YNSqwrv/pp5/GhRdeiN/+9rcoLCw8hbdDRFYVaWbEClM8yrDEqAZFyqC4NIGHNjsBAKlJdtV+PYnGVTwUD1ENe/fs2YOKigpMnjxZvi0rKwujR49GWVkZAKCsrAzZ2dlycAIAkydPhs1mw4YNG3Sf1+12o7a2VvVFRB2LauWO9RIoKpH0dGlbZqz+NawXgKUmeA8e7ZDYB4XiIarfVRUVFQCA/Px81e35+fnyfRUVFcjLy1Pd73A4kJOTIx+jVVpaiqysLPmrqKgomsMmIpOJtB7FrESDy0pugxoUqcOsUrpJNgmUaDMmzKBQLFgi7F24cCFqamrkrwMHDiR6SEQUZeqsgwWjEgXltI7fbzDFI9WgaBqwdc9KDjo21ZnYAln2QaFEiOp3VUFBAQCgsrJSdXtlZaV8X0FBAY4cOaK63+v1oqqqSj5Gy+VyITMzU/VFRB2XtcOTyLI+8jJjTVHsoB5ZeHDGGarbzLKLsSSJfVAoDqIaoJSUlKCgoACrVq2Sb6utrcWGDRswZswYAMCYMWNQXV2NzZs3y8d8+OGH8Pv9GD16dDSHQ0QWZfEEimr87W3UBgBXjS5WXa9qbIne4E6CNvzQZn2YQaFYaHdYXl9fj127dsnX9+zZgy1btiAnJwfFxcW466678H//93847bTTUFJSgvvvvx+FhYW47LLLAAADBw7EtGnTcPPNN+O5556Dx+PBvHnzcOWVV3IFD1EnFrr3ibUiFr8YuuBXFEW5lb1eAzbtFMquI+bqE1WgmYZiHxSKhXYHKJs2bcLEiRPl6wsWLAAAzJkzB0uWLMHdd9+NhoYG3HLLLaiursa4cePw/vvvIzm57Rt66dKlmDdvHiZNmgSbzYaZM2fiqaeeisLbISKrUvU+0ZzUDco4LEFv6FL2BNDPoAiaoo8fj0rswgDteAQA14/tjcWf7gXAPigUG+0OUCZMmBDyrxlBEPDII4/gkUceMTwmJycHy5Yta+9LE1EnoV1yrMxIaE+WZhRuikdq0gaEb2HfNd2J+y4eGLWxRYWgntbhKh6KBYa9RGQKoXqHGNVxmFW4KR5pibFNCF9gevGQQmQkJ0V1fO2lHaEAAXbFuFmDQrHA7yoiMoVQvUOsEJ8oMzu+MHvxyEuMk+xhM0LZqYkNTvQImsCKNSgUCwxQiMh0tCd1KwQoSqrNAnXulzIooaZ3fnHhAAwrysYN40qiPbx204uhVBkU1qBQDJhrcT0RdVqhatusNsXjaysxMZjikVbwGDdgu2V8X9wyvm+0hxYVAtQZFDPtE0QdB8NeIjId7XSPqkg27qNpP59i2ZFecCU1adPuZGxWguZTFwQBdmZNKMb4HUZEpmC0/44oipZYZqw8hYthi2TDZ1DMTBC4codijwEKEZmCKiiBuobDao3alEWyusuMvdbKoGjTVgIAmwWWe5O1WeSng4g6Fc0cj8Xik7AZH2kVj5UzKFy5Q7HGAIWIzMGgkawI0RJFssqEgnIH41B9ULQbBZpVcCgioE/X9ASMhDoTruIhIlPQdo9VXvYbBC9mFa5I1t0BMihj++XioRlnoH8Bd5en2GCAQkSmpq1BsUI9ij9MHxRpFY9lMig6szmCIOC6sYnv0UIdlzV+OoiowzMsktVkUKww3RNuvO4QOxlbAatPKB6s+dNBRB2O4TLj1v/07jMT5Uk73F48ylb3VqDXB4Uo1higEFHcnWhoCTlVow1W1BmJ2I0rWnz+0FNSkbS6NzOGJxQPrEEhorhaveMIrlu8ET8eVYTi3FT5dqM6ExGaVTEWKJMNV4NitUZt2oQJEygUD9YM34nIsp5Y+R0A4LVNB1S3G03xQBSDVvWYXfgpHos1atPQTvkQxYI1fzqIqEOIZEWOdi8evwXmeJSbBYYukmUGhcgIAxQiSphIMiOiGLx5oBkpxxUug2L1VTxE8cCfDiKKK6MAw7hRm7qTrFmXGWvHrHdZ0jbFY5EMCqd0KAEYoBBRXKmXEBscE2KzQJPGJ6r3opziCV0ka81fwZzioXiw5k8HEXUIosHcTchW9yaNUJTj8oWb4rF4J1lmVCgerPHTQUQdktGSYW3cop7iie2YokEMMyXV3JpBSbZIkawWMygUDwxQiCiuIglKVLeLIvyqKRNzRijKOETVqE3nWKtlULQYoFA8WPOng4gsy2jljlGRKaAOSsyaQVGOMdxuxi0WW2asxSkeigcGKESUMMZFsorLYmTLkRPNcIwdoFGbdu8dZlAoHqzx00FEHYZhvxOjZcYQNX1FzBmhGPZB0YlQrNaojSgRGKAQUeIYrnZR367eLNCkAYqyBkXV+Tb4WClAsUwGJcx1oliwxk8HEXVIxoWx6mPCdWY1A1WdTJgiWWmKx6oZFE7xUDwwQCGiuIqk7kR1u6YGxbRFsgZj1GZ8vD4/vK0HWKVRW3BAwgiFYs8aPx1E1GEYdYVV3a48HmLYviJmo1pmrBlui6LNrFVa3Wsxg0LxwACFiBLGsCdKiE6yZmUcRKkH3+xpC1CcVsmghLlOFAvW+Okgog5Pu3JHyWqbBYbqfOv2BupPkuwC7DZrnuq1y46JYoEBChEljOGS4xC7GZs0PjHeLFAboHis1+aeAQklQtQDlN69e0MQhKCvuXPnAgAmTJgQdN9tt90W7WEQkQVEtIpHVB9ohQxKqJqZZq+129wDnOKh+HBE+wk3btwIn88nX9+2bRsuuOACXHHFFfJtN998Mx555BH5empqarSHQUQmFVHWJGiKR/+ymRi1utcOV8qgWGmJcVANCiMUioOoByjdunVTXV+0aBH69u2L8847T74tNTUVBQUFET+n2+2G2+2Wr9fW1p76QIkoIUJ1jNU/PnTRqVkYLTPWdr6Vu8haOoPCCIViL6Y/IS0tLXjllVdwww03qOYwly5diq5du2LQoEFYuHAhGhsbQz5PaWkpsrKy5K+ioqJYDpuI4iTyVTyhO7OagdG0jjZjZMUmbdqMCTMoFA9Rz6AoLV++HNXV1bjuuuvk26666ir06tULhYWF2Lp1K+655x7s2LEDb775puHzLFy4EAsWLJCv19bWMkgh6mBC7dETSaYl0dRFssZ78Ug7GVtliTFRosQ0QHnhhRcwffp0FBYWyrfdcsst8uXBgweje/fumDRpEnbv3o2+ffvqPo/L5YLL5YrlUIkoToxma7Q7GCsvh1q2axaGy4w1GR+pi2ySpZYYczdjir+YhfD79u3DBx98gJtuuinkcaNHjwYA7Nq1K1ZDISKTMm57ry4ytcRmgTCY4tEcJ2VXrNoDBeCyY4qPmAUoixcvRl5eHi666KKQx23ZsgUA0L1791gNhYhMxHhpsUHdhqYPiklneNQZFEXWJGgvntY7HXbrnOQZj1AixGSKx+/3Y/HixZgzZw4cjraX2L17N5YtW4YLL7wQubm52Lp1K+bPn4/x48djyJAhsRgKEZmMOhAJH22IUO8ObN4MShtfiICqLYNi3RoUxisUDzEJUD744APs378fN9xwg+p2p9OJDz74AE8++SQaGhpQVFSEmTNn4r777ovFMIjI5IzO49p6FF+IKROzUGdQjItkpRoUCyVQgjCjQvEQkwBlypQpun8ZFRUVYc2aNbF4SSKyOOOdjUXVqhjTFska1KBox2vFDErwZoGMUCj2rPMTQkQdQiQt17QZFCtsFqhux6+4WTNcKYPisHSRbKJHQJ0BAxQiii/Dk7dRh1n15ntmneMxqkHRBlTS9I/dQnM8QY3aEjMM6mQYoBCRKYQKVixRJKvqfWJcM9MRMiiMUCgeGKAQdRBvbD6I21/ZLLdSt4JIusKKEENmJMzCqFGbdo7H17rM2Ep9UFhzQokQ006yRBQ/P3v9SwDAOX0OYM65vRM7mBAMO8Ya3A5NDYrPrHvxKC77VH1Q1Md1hAwKAxaKB2ZQiDoA5fRCXbMngSM5eaGWHCunTDwmjVCMMijaFY0+nwVX8XCzQEoA6/yEEJGhqoYW+XJykrl3yY2oOZtmybHPCgFKhMuMO0YGhSj2GKAQdQDlNc3y5Qa3uWtQ1FM5+g3NtG3vfYoHmTZAMVpmrDnOinvxBGdQrDN2si4GKEQdwNE6t3y5uqklxJHmElE9CtRTPC2+8BmYRFOt4gnai4cZFKJIMEAh6gDq3F75ck2TNWtQlLSLYJSreFq8Zs2gKAt5DTY+RMdYxcMECsUDAxSiDqC+WRGgNJo7QDFagWtUmqJtdW/aKR7FZb/B1BWg2IvHQgGKFlfxUDwwQCHqAOrdbUFJi0lP4HpUdSdGl0VrrOIxasdvtBePpaZ4LDRU6jgYoBB1AMoMis+su+m1iqQ5m/YQ5VvymHaKp+2yL0TnWytuFhiEAQvFgYV/QohIoqxB8Zo9QIlgike70kdVg2LSDIp6ikdx2a8foDistBeP9rp1hk4WxgCFqAOwUgZFyXA3Y02wolrFY4EMirJgVhswdowaFKLYY4BC1AHUWyiDEgltDYoyg+Ix7TJjxSoegxU9yutWqkHR9j1hHxSKBwYoRB2AMkCRlrGa1cms4rFCkay6BqXtMjMoRCeHAQpRB6Cc9jDp+duAflQS1AfFr54y0dZ1mIHRFE9wBiXwD2SpDIr2unWGThbGAIWoA1AWjpo9g2JEvcwYqss+TXrFjIWyyvErgxJtgOK14GaBRInAnxCiDkA57WGlGhTj6R51DYp2+seM0zyqKZ4OV4Oiuc5JHooDBihEHYDHa3xCNBtt8KF7jOaa9j2ZsVDWaF8hryajJQWQNgsFKFqc4qF4YIBC1AEopzy8Jjx5R0I0uKJdxQNYIIMSYorHkhmURA+AOiUGKEQdgLJIVtu51GzU9SXGu/623R7c7MyMvVCU70X5b6ANGL0W3CxQixkUigdHogdARKeuxUI1KJEuLVZe1tb9Nnl8MRjZKTLoHms0PZVkt87fh0F9UJhToTiwzk8IERlSTnkcrXOjvKYpgaM5OUY1HHpTPI0t5gtQjFrdawPGptaxpzjtcRhVbDCDQvHAAIWoA9BuoPfj59cnaCTtEyooUR6jneJpMmOAoly5E2IVT3Nr9ifZYZ1fv0F9UBIyCupsrPMTQkSGtH1B9lc1Jmgk4Wnb2OsfA9Ux2rrfJo8XZqPKoISY4pECFCtnUIjigQEKkcWJomjKZbftp18wK4piUOGvKad4lDUoITYLlOpnkpMsFKBo+6BwjofigAEKkcWZsatqKOrpm8iyKZaY4lFcVi8zVv/7NHsC11OsFKBoMDyheGCAQmRxVsueGPU7MbrZLwY3ajPjKh6jBnTKDIooivLYXUnW+fWrXbXDBArFg3V+Qohi6E+rd+H3K79L9DBOihl7gpyMUAWzVpviMSqSdSv+rSydQWGEQnHAPijU6TV7fHjs/R0AgKtHFyMvMznBI2ofM3ZVjZRxUzl10zPpJO9y2OD2+k06xaMflCgzKM2KzI+ValAYj1AiMINCnZ7yBG+1eg7AehkUdTGp/u3BfVAClzOSA39TmXOKR/+yMliRxu2wCZZq1KbEYIXiJeo/IQ899BAEQVB9DRgwQL6/ubkZc+fORW5uLtLT0zFz5kxUVlZGexhEEVOeJK3YflwKqpyW6auh3xJeVcOhOlqUi2TTXa0BihkzKAareHyqDErg38pK2RNAXRRrvZ8QsqqY/EY788wzUV5eLn998skn8n3z58/Hv//9b7z++utYs2YNDh8+jMsvvzwWwyCKiHKFiM2Cfx5KGSAr1jQYrtxRtY1vO8mntQYopqxBUU7xGAQoUmBltQCFKBFiUoPicDhQUFAQdHtNTQ1eeOEFLFu2DOeffz4AYPHixRg4cCDWr1+Pc845R/f53G433G63fL22tjYWw6ZOyux714QjbUbnskgGxSjTYFyN0tYHRQpQTNmozWCKR/n9VVEb2IIgJy0pXsOKCmXgzgJZipeY/EbbuXMnCgsL0adPH8yePRv79+8HAGzevBkejweTJ0+Wjx0wYACKi4tRVlZm+HylpaXIysqSv4qKimIxbOqkvIo+FSbfCFiX9Be6VWoatEuI5dsN+qP4xbb7zDzFY1SsrOyD8tXBwB9XZxZmxWVM0WJTfGsxPKF4ifpvtNGjR2PJkiV4//338eyzz2LPnj34wQ9+gLq6OlRUVMDpdCI7O1v1mPz8fFRUVBg+58KFC1FTUyN/HThwINrDpk7M6wtOzYsWilSkMVulfkb52Rolr4yWGZt5ikeqL9FSZlC+P1YPABhQkBGXMUWLoMqgJHAg1KlEfYpn+vTp8uUhQ4Zg9OjR6NWrF/7xj38gJSXlpJ7T5XLB5XJFa4hEKsoaAb9fxLZDNbj2xc8wf/JpuGZM78QNLELS+B1WCVCUl1VTPAZFsopW92mt+9eYcRWP26s/JuX3l7TiKtVi+/Aov7W0TduIYiXmOeHs7Gycfvrp2LVrFwoKCtDS0oLq6mrVMZWVlbo1K0TxoO70CTzw9jZUNbTg/re3J3BUkZMyQNbJoLRdVmVQVFkTdbAiHZdm4iket8Fyb1WGrvWN2CzybyVRFY9ba+hkYTEPUOrr67F79250794dI0eORFJSElatWiXfv2PHDuzfvx9jxoyJ9VCIdKkyKBaa2pFI47dOgNK+z9svivJjzDzFYxSg+HXer91i8yRcZkyJEPUpnp/97GeYMWMGevXqhcOHD+PBBx+E3W7HrFmzkJWVhRtvvBELFixATk4OMjMzcccdd2DMmDGGK3iIYk1Z3OgXRQv1EwmQinyVAYqZz3/qIln921XHi211Nlac4vHqdJW1SjAp4codSoSoBygHDx7ErFmzcPz4cXTr1g3jxo3D+vXr0a1bNwDAE088AZvNhpkzZ8LtdmPq1Kn405/+FO1hEEVMnUEBXA5r1Qcoa1Cev2Ykbn15MwqzTq7eKx5Eo6kcg2W6flGEtBDG1FM8BkWy6p2NrRmg2FgkSwkQ9QDl1VdfDXl/cnIynnnmGTzzzDPRfmmik6LdbdYq/UQkypNej+wU1W1mFMkUj6pgVrGKR15m7PHB7xdNVcvRbJDV6RgBSttlFslSvFjrNzFRDGgzKMoun14L7M3TlkGxySc+n4lraVRTPH7l7cY9UaTrytUvzQZTKoliWIPSEQIUi0wfUsfCAIU6PWWjNr8oqhqe1TWbr2OplrKuQQ5QTJ1Babts2KhN0+pe2wcFMNc0jyiKhgGKXtt7qywJl3ARDyUCAxTq9JTLQP2KnhsAUO82f4Ain/TsbQFKVUMLjta5Qz0sYYwyJUrKk7qy1b3dJiA5KfBry0wreULtgq2a4ml9H1bb84mt7ikRGKBQp+fT9EFRrsYwqiswEymDYhME1V/mP331i0QNKaRI9uLxa6bdpMPsNkHeFNFMK3mMsieA/s7GDru1TvI2ZlAoARigUKfn1dQItChONmY6CRqR9npxKKZ4AGDd7uOJGlJIen1BtHzqIhT5OJsApDrNt5InVCCrVyRrtQyKqjDWWkMnC2OAQp2eT1ODovxr2Gh/FTORZhfsNgEOm/l/pP2i/mXV6h6/dtotcFkQBKS0FsqaaYrHaIkxIGWAAm9AWdBsJRYrmaEOwlo/JUQx4NVMJyhPNlaY4pEzKHZ1BiXJpNMI6ik1/Ske1TFQZlCUUzyJrw86WueG1+cPOcUDtAViba3uYz2y6FJtFpjAcVDnYrEfE6LoUxbJBlZjmK8Gpa7ZY7jDctsqHpuqBsWMxYza9+A1WG3k1WRQpIfZBMgZlKaWxGa3vj5ci7Me/QBzFn9m2EVWIgUm8jJjE/7bhMIiWUoEBijU6QVlUJRTPGH+Mo6HPccaMPih/2Le3/WLXpVLV5X9KsyYltcuf1ZdD7H8WJlBSZWneBKbQXl1434AwKe7jkeQQWkNUMQOUCRrraGThTFAoU5PW4PS4jXXFM/S9fsAAO9sLde932gVjxkLMbUJE79mObFEuWpXVCz9FgSYZhWPso4kVA0K0BaISdk6M/7bhKJq1JbAcVDnwgCFOj3tdIJHEbC4TRCgdMtwyZf9OlMiygyK3eQnEu2qHaMpHvVKn7bAxqYokk30Kh5lFiRcV1vp/Uj/t1qRrKpRm8WCK7Iua/2UEMWAtg+Ksv16ov9KB4AuaU758rGG4OZrcl2DXVB1wVU+ziy0AYpf89lLVHVBEOXaFfUUT2L/bZTBYLiOw9L3lNeqRbIwd+BLHZPFfkyIos+j6SSrLOQ0xTJjxYl7//HGoLu9mgzKry4bBABw2s33461NmPhE/QDFp1pyrMygmGeKR1noeixM117p/fj91sygmLGeiTo+a/2UEMWAugZFfXI0Qw2Kso26XvM1afzSX/RDe2YBMMfYtYKKZDXBoXzZYJlxoA+KORq1KT/fo/VhAhSpBkVecRW7ccWCehVPAgdCnYrFfkyIok9bg6I8h5ohg+JRBChfHaoJul+ZQQHadmNOdIZBT6hlxnp71kiP8cuFwDDNFE+j4vM9Uhs6QPFrMih2i2VQ1EEJIxSKD2v9lBDFgOqveL9miidM8WM8KAMUj86mdNL4pZNesiNwAjdDcKWlzaAosyZGwYqo2IvHTI3aGhUbSZbXNIU8NiiDYrE0BDMolAiO8IcQdWzaPijqDIoZAhRRcTk46NBOGyQ7AxeavT6IomiqVRfaGhTle9PbsybwGFHVByXRq3i+P1qPZo8fy7cclm/bp1MbpCQ3apN2ZbZyH5TEDYM6GWZQqNPzavqgKP+qD9ffIh6UfVmUJ3SJTzNtIE3xiKK6fsUMtKt4lIGIYQYFbYGNoJji+WjHUVz74meobfbEbsAaoiji/N+twYVPfay6/VB16AxK0BSPiYLGSKj6oFhr6GRhDFCo0/OqphNE1cnRDHUcYad45N4arTUorVM8ANCc4HbwWsF9UNrGp22Yp7wsZ1BsbVM8ALD2u6P469rvYzXcIOE6xhrRTvFYrARFsxcPIxSKD4v9mBBFn3oliXq5qzmmeNpOil69DIpcgxI4cSTZBTkl394aGlEU8cr6fdi878RJjjY07RSP8v14DbIpUNWgtO3FIznRGL8Myslm1PyKQl/AesuMlSEJMygUL9b6KSGKgeBVPOZaZhx5DUrgzCEoCknbO/7VO47ivuXbMPPZdSc73JC0nXCNpnX8IWpQUp3q0rl49ujQBnyFWcmq6z27pOg+zudXv1fLTfFYbLzUMTBAoU7PF1Qka65GbS1hpnik+5WN2ZKT2r+SRxRFXL9k48kOMyLaKR4lr0HBrLoPClRTPIC6PiLWtAHfyN458uUzumdi+qAC3cdpA18WyRKFx1U81Okp6yACPTfa7jPFMuMwRbLSfkGupOAApT01NJ/vj820jpJ2mbHRfb6gGpTAZZsgIDNF/WvrZOtCToY24OuSmoRlN43G0Xo3Lh3WA6XvfaP7OJ9f7DAZFDOtCqOOjRkU6vS8PuMpHjOs4lHXoESaQWldatyOAOVwdfPJDjFiIeKTENM9bcfYBAG5aS7lw7Bsw37VSqdY0n6emclJOLdfV1w6rIc8Pj1+TfG13WK94xmTUCIwQKFOT3syNHMNSotuBiVwcnYppj6kDMqRMHvEKO2vUvfy0JtOOlWhpnh8qhU9+sGKTQCcjuBfW18dqo7OAMPQZmsyktXZHKPzuM9v7QCFjdooERigUKfn1UwtKP/KN8MyY2UNinI6Snu/MoMiBRt3/v2LiF9H28ujwR39Tq0ha1A0hbF6txtNL5xoiM9KnqAMSkqS6rrRyVubQbFYfKJaFs0AheKFAQp1ekZ/rQOBE5J2/5h4U/VB0ZnKkKY3lDUodc3tDy60jxn2yErURHkJb8Q1KAbBitGJ/aaXNuH7o/WnPsAwtAFKcAZFf4A+v7KhnmC5Og4b+6BQAjBAoU5PVeMRtFeMfmFqPKkCFJ0TvLu1kNdlsEXukx98h3nLPpePM1Kn05H1zS8OtmeoYYWK9YxqUJR1N9KJ8oGLz0CX1CT0yk2V7/vbur1Bz1nv9mJnZd0pjFitWRMgZiarMyhGAZTPL8qBpNNqWxkTJQh/UqjTU/3lrlN3keiVPMqVI7rLjHUyKIN7ZMmXn/xgJ1ZsLUf/+94PmcGobQoOUKLd/yJ0DYpRBiV4PDeMK8Hn91+Agsy2PiRVOtmee97YigueWIve976D/2yvOJWhAwifQTGa//CLovx9lJxkvV+7rEGhRLDeTwpRlBl2MG2V6EJZZWZDFIOnSdzyX+ZtRbJLrj9L97kqao1X6uhNC0W7ViLyZcZttyvrbpQnR0EQUNXQIl+valAXBPv8It75qly+fuvLm3HT3zahMsRn4PeLQTsTN3t8uPXlTXjyg+/kJd2SoBoUg+f1+cW2YmaH3eAo81JP8RDFBwMU6vRC1aAAiV9qXK8JHLRZFL0MSm66Cxmu4DZHB6uMd93VC1CiXSsRapmxYSdZzTJjpV9cOFC+fLy+RXXfnmPBNSkffFOJp1btDLq9ttmDh/+9Hbcv3YwxpR/it//ZId/32sYD+M/2Sjz5wU40tISpQTGa4rF8BqXtstXqZ8i6rPeTQhRlyr/QlRmUk20XH2117tABitugtiFo+gHAwRPGu+5KmRplp9Z4TvGo/x30L2szOhMH5GFxa7ZIyqaIoohbX96Eyb9fq/s6UiDj94v4rrIOfr+I3/1nBxZ/uhf/2V4JAPjjR7tQ3Rg4bu13R+XHvr9NPU0UXINiMMXjF+Xvo+Qk62VQBGZQKAEYoFCnp9qwTnE51dn+bqzRJooi6oMCFPVJXi+DAgDp7QhQPD6/nB3ITm076UZ7ike7F4+SUXM2vRoUpQEFGQACPV8+3nkU9W6vHGgAwPkD8pClmIqRepA8/eEuTHliLZ76cCe+Lq8Net4Pvz0Cr8+PjXur5Nu2HKhWHaMNNiKb4rHer13V9wEjFIqTqP+klJaW4qyzzkJGRgby8vJw2WWXYceOHapjJkyYAEEQVF+33XZbtIdCFBH16pG2M6O0a24i9+NpaPEFrXxRrmrx+0XdPigAVCdlycET+lM8e441AADSXQ50y3DpHhMNvkj34jE4Ti9BkZeRjPTW6axrXvgMK7aWq+7Pz3Spsk7vfFWOphYfnvjgOwCBImK9aYtV3x7BFc+XoVZn6usft47BZ7+YFNH4gECQJU3xuCyYQWENCiVC1AOUNWvWYO7cuVi/fj1WrlwJj8eDKVOmoKGhQXXczTffjPLycvnrsccei/ZQiCLiMyiSTWvdNTeRUzxS/YndJsgdVJWN25SXtSe+cf26yZf75aUDMM6gfNOaQTg9P10V6ES7m2yoZcbhaoEEQb/+wW4TVKuWVmw9rLo/LyM5qBX+i5/uUV3/bE8VtN7ZWo4v9lfrjvWs3l2Ql5kcdLtRfYZfbMugWHOKR3mZIQrFR9Q3C3z//fdV15csWYK8vDxs3rwZ48ePl29PTU1FQYH+zp9abrcbbndbhX5tbXA6luhkGa3iacugJDBAcQfqQjKSHRDFwHROk6JQU9l6XZtBmTQwT84SDOmZhV1H6nHAIIOyoyLQK6R/QSb2Hmv7YyLaG/GFWsVj1ElWEqoe5vqxvVH2/XEAwKe7jqvu65WbGrQ66/H/qLO6SgMKMvBthXHvlH556YYnacMiWX9bkaw1p3gYlFD8xfwnpaamBgCQk5Ojun3p0qXo2rUrBg0ahIULF6Kx0Xh1QWlpKbKysuSvoqKimI6ZOhfltI6UMRAERZFsHHfL1ZJW1qS7HMhJcwIATij6fUjBk90mIMmuPon06ZYmXy7qEmhoVlHTrBskSAHKwO4ZSHIoMyjRbVIXaR8Ur87rhqqHmXJmAe6ZNkD3vgn98/CHK4dFPMbkJDuuPEv/d0zXdBeW3jTa8LHaLqtSvYtfFOWpQstnUBI3DOpkYhqg+P1+3HXXXRg7diwGDRok337VVVfhlVdewUcffYSFCxfi5ZdfxtVXX234PAsXLkRNTY38deDAgVgOmzoZVe1D60nSJggntSNwtCkDlC6txavK3h9Sc7WMZEfQX/WpzrYE6fDibDhsArx+UbcPiDTF0z8/A05FoBPtXYJDd5INvedQuKkFvVVLAJCT5sSlw3rgywemYOF0/SBG6YzCzKBA6p+3n4trzumFD392HvJ1pnbaxqi+LgUoXp8od/JNtngGhckUipeoT/EozZ07F9u2bcMnn3yiuv2WW26RLw8ePBjdu3fHpEmTsHv3bvTt2zfoeVwuF1yu2BXuUeemN8VjE9r+0tU254onaQVPZnISMlMCP64nGhUBSnPb/Xr+efsYbDtUi/NO74bu2ck4UNWEvcca8NWhGpxTkous1CQcqGrE4Zpm2G0CzuyRhSTFVJHbG9iLKFp1B40txp+lMoNSrdMVNtyKokGKOhTJX64dJV/OSk1CuHzQjeNKcOv4Pnh5/T7V7SN7dcHIXl3CPDp4jPmZLhyoasKh6ib5e0u72soKlAGK3Wa98ZM1xSxAmTdvHlasWIG1a9eiZ8+eIY8dPTqQMt21a5dugEIUS+pW98oMSuKXGUtFsunJDmSnBqZ4lBkUqXeJUfZgZK8cjOwVmF7tmZ2KA1VNmP+PLaisdeOs3l3w+m3nyrUbQ3tmId3lUAUof1q9G+t2H8dbPzn3pIKUnZV1+PJgDc47vRu6pjtR1dhieKx0AhdF0SBACf36w4qy8eJ1o9C/IBNNLT4U56TKhcXhFOek4o3bxyAvI5AduekHffD0h7sieqySdoqnf34mDlQ1YUdlHXpmpwAAki3ZSbbt8qgIAjWiaIh6gCKKIu644w689dZbWL16NUpKSsI+ZsuWLQCA7t27R3s4RGEppxM8rZdtgiAv062sdes+Lh5qWwMQZQ2KaoqnNYAxClCUxvTNRdn3x+X3s3HvCfxk6Wa8+1Wg+di5fbsCCOxz868v21bCbDlQjRONHvj8Ik40tuCvH3+PnDQX7g0xXfKPTQfgtNtw12tb5Nt+NuX0iDrJNrb4VKuTJJEUap4/ID/k/VeM7Im/rduLercXAoB7pw/EVaOLg47LSknCX64dhZtf2oQbxob/HSbRDnFAQQY++KYSu4/Uy8XNsVzGHSvK4HTcaV0TOBLqTKIeoMydOxfLli3D22+/jYyMDFRUBH75ZWVlISUlBbt378ayZctw4YUXIjc3F1u3bsX8+fMxfvx4DBkyJNrDIQpLb3mrTQCGFgWmDDbtO5GQcQFtUzzpyQ45YKpRbOrXlkHRn+JROu/0bvj9yu9Ut0nBCQCM6h34y3hYUTauHdMLL5W1TXOU1zTh9lc+x35Fq/wFF5wOp8MGURSx73gjeuWmQhAElNc04e43tga9/m//+x2G9gyehpFItUDKAEwpGrNMuekurLv3fAiCEHbq6oIz8rHhF5OQ146AQvt8PbsEsibfH2uQe81M6J93EiNPLGUGxYpFvmRNUQ9Qnn32WQCBZmxKixcvxnXXXQen04kPPvgATz75JBoaGlBUVISZM2fivvvui/ZQiMLy+UXVCV9axWMTBIwuyYUgAF8eqMY35bUY2D0z7uOrV2RIMluzJMq9eWqbQtegKOk1blNS/mXfPStFdd/rmw6qghMAuOPvn+PKs4rxxf4TeKp1OuStn5yrOz0j+fJgjeF90mcvPd4mhO8iezKkICKSKatQBbG6z6253r11WkcqNu6RnYLT89Pb9ZxmoPysrLhMmqwpJlM8oRQVFWHNmjXRflmik1JZ26xaSis10xIEoDA7Beed3g2rdxzFut3HExOgtGZQMlwOuXV9nVsvgxL+R1nZwl6PMsjR1m4sWbc36Pj/bK9UtZQHgB/+aR3SnCf3F7ZUQFvdFMig5KQ5cUyxAWC02+7HgjbmyUh2IM1pl7cRmDQwz5KNzphBoURgKEyd2gFNVkBqpiUtD+3fus+L9rh4qWlqm8LJcAUCCGUGpU5exRM+QAk3DZSpyLDo7YQMAJcOKwz7Otodf7VGFGdjTJ/coNulYuQGKSiLcCM+M9GO0CYIchYFCOwLZEXMoFAi8DuNOrXDNerW71IzLelkKDU4M9rDJtaO1gUKWrtluNoyKMopnnbUoNjDpCDSFUHJCIOVGucPyMP407vp3qc06+wi/N9lg3Tve/MnY5Gb7gy6vbEl8L7q3b6g8QDWaLFu03zGAoBbx/eBwyZg/OndMK6fNQtMmUGhRIhpHxQis5NqOCRSMy3pZFiUIwUo+nvYxNrR+rYARdpduVYvg5Jy6j/KygCmb7c0DOmZha2KmpG8DBfG9MlF2e62VvI5aU7cOr4PSt/7FgCw7eGpcNjalmiv3nEEH3xzRD6+uPXz1MuGSFM8UgZFG6BYYopHc90mCLhiVBEuGtIdKUl2SwRZepT/XskW7ONC1sQAhTo1qcZD0pZBCVyXurcqC2njRRRFHGldEpyX4ZJPEvW6NSjhMyhK4/p1xZYD1UHvXyIIAv5x6xh4/SIWf7IHxbmpuHRYDwDA/07pj1c3Bro5/3B4D1wyrFAOULRBxdQzC+QAZVSvLnj40jMB6GdzpGW4ypVLSlaY4tEWoUhXlV19rUhZWeiyYB8XsiZr/9QQnSJpiiQ5yYZmj1+1tw3QduJX1n3ES0OLT67L6Jbhkgt4mz1+tHj9cDpscgYokiJZAPjXvLFY//1x3DiuD+w2AaXvfoPn136ve6yUBblj0mmq25WrfaoaWtA9KwX/vP1c3VVCM0f0RNd0F4b0zEJuetvj9GINr19Ei9cvT/Vo62CsmEHpKJS7WjODQvHCAIU6NSnwyEl14nBNc1ANipQRqG/xwu8Xg2oMYkmqP0l3OZDqdCDZIcLpsKHF60dFTTOKc1PbnUEZ0jMbQ3pmy9fvmHQa9hxrwLRBke0sLhnUIxPbDtXi4iGB5opGbeBtNgETdQpD7QbZkCaPDw2tNSjaoCuen/3J0mZ5rJD0iYRyTyZmUCheGKBQpyZNJ+SkBwKUthqUwP3SSVIUgYYWb7unUk7FkdZN/aSMhc0moFdOKnYeqcfe4w2tAUrkq3j0pLsc+LNiv5pILbv5HOysrMeI4uyTel2j6ZqmFp/hFE/qSS5fjift27LEtFQElAFKuGJromhhro46NSmD0qV1nxu3JoOSnGSHs3VvmqdW7Qzb5yea5AJZxdRIr9w0AMDe4w1o9vhQ51aPP14yk5MwsleXky761O43l9S6g3Jji1ee4kl3qYPBFAusHtF+Gh0kPoE7yrtaE0WCAQp1atIJXtrnRtoDRvlXonTbXz7eg+2Ha+M2NqlAtlumMkAJrII5dKJJ7uya4XKEbcJmNtrMQlrrVFpji69tmbEmg2KF5a1BUzwdpCplTN9A35pENCukzotTPNSpaTMoEqO/fL+tqMOgHsb7yUSTXgaloLX1enlNM/YfDwQoxa174FiJ9kSenZKE6kYPaps88oqpbE3RrRWmeLTxSEeZDclKScLXj0xl/QnFFTMo1Kkdaw0Cumep91wxqh3YdaQ+5mOSSEWyeYoMSn7rOCtqm/FtRSCbI2VVrER74pZW+FQ1tqC2NUDRBo0pFghQOuoUDxBYKs36E4onBijUaXl9fjlAkRqySZS/h/95+7lyoWo0ApTj9W54feHn9I/UBWdQpECqsrZZ7i8y1oLdSbUrcqQpthONigxKqrYGxfwJ3+BMFk/oRCeLAQp1WscbWuAXA/UmoTIoI3t1wR9+PAwAsHbnUXmlT3s1uL34ZOcxjHr0Azz+3x1o9vjg9vrQ7PHB5w8uvm3LoLSNrbB1X5d9xxux5UA1AGD8aeFbz5uNNkOV2xqgVNW3yAFKlzRtBsX8v660CQYmHIhOnvn/JCGKkYqa1mW86a6gAkztCbRfXjqAwHLLaU9+jFULzmtXX459xxsw4+lP5Db1z6/5Hs+vaWuQdu2YXnjk0ra9a0RRxDfltfL4JN0zk1W742YmO9CzS9tmdFah/OgEAchunc45VN0oB2u5aU65gR5gjW6s2gSK1WqDiMzE/H+SdDA7K+vk2gFKrMrWPiP5mS4k2dU/CtplsN0yXBhalA0A2HOsATsq69r1Wn/6aLdqDx2tl8r2qbp1vr75oHw5X1GDYrMJOC0/Q75+RmGmJU+CyuDOJgjISQtM50hTaE6HDclJdmQq+s5YYRWPdtWO9f5liMyDAUqcvLH5IP7n+TJc8MRazPzTuoTs7UJqla1TKPmZyUFbyAd3BBWw/Cfn4genBeo93vuqPOLX2fD9cby26UDY4yY8vhrvbC3Hvf/cirvf2AoAOKdPjqpFPAAMVqwimj6oe8TjMBPl52sTgL7dAhmqz/dXA2jbVFDZPt8SfVCCMiiJGQdRR2D+nGkHsPjTPXj431/L1xtafBj68H+R4XLgr3NGYUSvLkF/wVPsHZEzKMmqv9QB/VU8giDgyrOK8fHOY/jT6t24ZFgPeepHT12zBx98U4n5r30p3/bgjDPQu2saBAC7jzagR3YKbntlMwDgUHUT5i77XPUct53XN+h5zx+Qh5fX7wMAXDa8R2Rv1mSUre4FQZCzU5L/uyww3aUMULR9UcxIm83qKJ1kiRKBZ8Uo8OsUOEo276uSg5NBPdRNjurcXvz4z+sx9OH/4rvKOqz57mjYTqVurw8HWht0hXKoukne+K693U+N3k+o9xmK1+fH3mMN7X7c8Xp3TDNNUg1KQVYyMlMcqp14jcpLpp6Zj4xkB7x+EZN/vwbPr9mNmiYPPvy2EqIoQhRF7D/eCL9fxK0vb1YFJxkuB64fW4KJ/fMwoX8ebhxXgmmDCvDs7BG6rzWgIANnl+QE3T6hfzf86tIz8c/bx+hu0GcFys/XJgBd012Y3rof0E8m9MU5fQKNwZTvr2e2+WttGI4QRY/5/ySJs6YWH975qhx//fh71DV7cfmIHrhhbAne2HwQA7tnYs+xenx5sAan56djRHEXfLrrOJ744DsAwGXDClHSNR0HTzRiSFE2Kmqa8MxHuwEEftG+eN1ZcNpteHb1btUOso0tPkx5Yq18/azeXfD0rBE4WufG4k/34M0vDmF0SQ6GFmXjz4rHzZ3YF11SnfD6RXx/tB55GckYd1pXLP50D/6zvTLovTntNuSkOVFR24x+eem4c9JpcrAzaWAeNu09gfuWbwMQOElMPbMAXr+IippmdMtw4d5/boXNJuB/Lzgdy7ccQs8uqeiW4cL674/j68O1qHd7cf/FZ2DvsQYk2W2oa/bA6xexdMN+AECX1CQ8OONM9OiSgpVfV+LzfSfgdNgw9cwC9OmWhpfL9uG/X1ciN82JX18+GD/7x5eoc3sx6+xiDOmZhfKaZlwytDuyUpzw+v1YtmE/DlU3ITfNibpmL34yoR/W7zmOpz/ciTljemPG0EL8Z3sFymuake5yQBCA4UVd8OXBanxbXovlWw4DCGRQBEFAzy4p+LYiUFui3FBPyWG3YVy/rnhvWwUAoPS9b1H63rcRfW89e/VI3dunD+6OFXeMw0P/2o5N+05gYv9uWHjhQJyWl65bXyIIAq4Z0zui1zQrQVDXoADAH68age+P1quyUsoApSjHAgEKp3iIokYQ47m5SJTU1tYiKysLNTU1yMyMXuvlFq8f0/+wFruPtv+v/XDW/HyCvI8KECi0LH33G/z36+BAguLHYRPw6b3nIz8zGde8sAEf7zwGAPjXvLGGQcqR2mbc9NImbD1YE9Fr2ARg/S8mIS8jOeRxbq8PXx6owcheXTp8Q6y/rduLB/+1HQAwrCgby+eO1T3u4X9vx+JP9wIAvv3VNNMXyr6ztVw1Tbfu3vPlpeFE1L7zNzMoCv/YdCDi4EQ6gWSlJKGqocXwuAyXA7dP7KsKTgCgpGsa/nztKFTUNMMnivjqYA2O1rtxf2sGQ2ITAGlmJTPZgeHFXbDmu6O6r6U8NpaSk2zw+wGv3x/x6+VluFDv9qKx5eR6iIRSnJMq70sTKYdNQNd0F+ZfcBryW/uM3DCuBJ/uOoYzC7NUhahaeZnJ+Ne8cfhi/wnMW/YFDlU3yfflpjlxRmGmHOi8dss5EdcYuRx23Smdjuiq0cUAAtN4s1ov67n9vL7YUVGHXrlppg9OAGB0nxx0z0pGeU0zzju9m7w1ARG1HzMoCh99ewSPvvsNrjq7GGP65mJAQQa+rajD+9sqcGZhJnx+ERP658HlsEFE23xzfYsXNY0e1Lu96NElBbuO1GP1jqOY2L8bhvbMble/DL9fxG2vbMauI/V4/IohGFHcBbuPNqB7VmCliaP1RHe4ugkvle3D1ecEfrnnprng8fvx0rq96NstHdMHd4fPL8JuE1BR0wyHXUBuWmA66KtDNTjR0IKy3cfx6e7jeOmGs1HV0IJth2ow5cx8eH0iDtc0Yd/xRpR0TUNFTTPe+PwgfjyqCN0yXBhQkCGn6N1eH1wOO3x+ET6/iF1H6pHqtOP2pZ+jpGsqZo/uhXP75kIQBIiiiHe/qkBlbTPOH5CH3l3TcPBEIwoyk+Gw29Ds8SE5yQ6/X8TX5bUoyknF0bpm7D3WiPzMZBTnpmL1jiPYd7wRDS1edE1z4cZxJbDZBPxj4wHc/c+tKMpJwSs3jsa8ZV8gOzUJt47vi3P75sLt9ePz/SdwvKEF553WDVkGm+sdqGpEZkqSZWs7iIjMrD3nbwYoGj6/CL8oclUNERFRlHGK5xTYbQLsrMUnIiJKKKYJiIiIyHQYoBAREZHpMEAhIiIi02GAQkRERKbDAIWIiIhMhwEKERERmQ4DFCIiIjIdBihERERkOgxQiIiIyHQSGqA888wz6N27N5KTkzF69Gh89tlniRwOERERmUTCApTXXnsNCxYswIMPPojPP/8cQ4cOxdSpU3HkyJFEDYmIiIhMImGbBY4ePRpnnXUW/vjHPwIA/H4/ioqKcMcdd+Dee+9VHet2u+F2u+XrtbW1KCoqislmgURERBQb7dksMCEZlJaWFmzevBmTJ09uG4jNhsmTJ6OsrCzo+NLSUmRlZclfRUVF8RwuERERxVlCdjM+duwYfD4f8vPzVbfn5+fj22+/DTp+4cKFWLBggXy9pqYGxcXFqK2tjflYiYiIKDqk83YkkzcJCVDay+VyweVyydelN8hMChERkfXU1dUhKysr5DEJCVC6du0Ku92OyspK1e2VlZUoKCgI+/jCwkIcOHAAGRkZEAQhqmOT6lsOHDjA+pYY4uccH/yc44Ofc/zws46PWH3Ooiiirq4OhYWFYY9NSIDidDoxcuRIrFq1CpdddhmAQJHsqlWrMG/evLCPt9ls6NmzZ0zHmJmZyW/+OODnHB/8nOODn3P88LOOj1h8zuEyJ5KETfEsWLAAc+bMwahRo3D22WfjySefRENDA66//vpEDYmIiIhMImEByo9//GMcPXoUDzzwACoqKjBs2DC8//77QYWzRERE1PkktEh23rx5EU3pxJPL5cKDDz6oKsql6OPnHB/8nOODn3P88LOODzN8zglr1EZERERkhJsFEhERkekwQCEiIiLTYYBCREREpsMAhYiIiEyHAYrCM888g969eyM5ORmjR4/GZ599lughWUppaSnOOussZGRkIC8vD5dddhl27NihOqa5uRlz585Fbm4u0tPTMXPmzKCOwvv378dFF12E1NRU5OXl4ec//zm8Xm8834qlLFq0CIIg4K677pJv4+ccHYcOHcLVV1+N3NxcpKSkYPDgwdi0aZN8vyiKeOCBB9C9e3ekpKRg8uTJ2Llzp+o5qqqqMHv2bGRmZiI7Oxs33ngj6uvr4/1WTM3n8+H+++9HSUkJUlJS0LdvX/zqV79S7dfCz7r91q5dixkzZqCwsBCCIGD58uWq+6P1mW7duhU/+MEPkJycjKKiIjz22GPReQMiiaIoiq+++qrodDrFF198Udy+fbt48803i9nZ2WJlZWWih2YZU6dOFRcvXixu27ZN3LJli3jhhReKxcXFYn19vXzMbbfdJhYVFYmrVq0SN23aJJ5zzjniueeeK9/v9XrFQYMGiZMnTxa/+OIL8d133xW7du0qLly4MBFvyfQ+++wzsXfv3uKQIUPEn/70p/Lt/JxPXVVVldirVy/xuuuuEzds2CB+//334n/+8x9x165d8jGLFi0Ss7KyxOXLl4tffvmleMkll4glJSViU1OTfMy0adPEoUOHiuvXrxc//vhjsV+/fuKsWbMS8ZZM69FHHxVzc3PFFStWiHv27BFff/11MT09XfzDH/4gH8PPuv3effdd8Ze//KX45ptvigDEt956S3V/ND7TmpoaMT8/X5w9e7a4bds28e9//7uYkpIiPv/886c8fgYorc4++2xx7ty58nWfzycWFhaKpaWlCRyVtR05ckQEIK5Zs0YURVGsrq4Wk5KSxNdff10+5ptvvhEBiGVlZaIoBn6gbDabWFFRIR/z7LPPipmZmaLb7Y7vGzC5uro68bTTThNXrlwpnnfeeXKAws85Ou655x5x3Lhxhvf7/X6xoKBAfPzxx+XbqqurRZfLJf79738XRVEUv/76axGAuHHjRvmY9957TxQEQTx06FDsBm8xF110kXjDDTeobrv88svF2bNni6LIzzoatAFKtD7TP/3pT2KXLl1UvzfuuecesX///qc8Zk7xAGhpacHmzZsxefJk+TabzYbJkyejrKwsgSOztpqaGgBATk4OAGDz5s3weDyqz3nAgAEoLi6WP+eysjIMHjxY1VF46tSpqK2txfbt2+M4evObO3cuLrroItXnCfBzjpZ//etfGDVqFK644grk5eVh+PDh+Mtf/iLfv2fPHlRUVKg+56ysLIwePVr1OWdnZ2PUqFHyMZMnT4bNZsOGDRvi92ZM7txzz8WqVavw3XffAQC+/PJLfPLJJ5g+fToAftaxEK3PtKysDOPHj4fT6ZSPmTp1Knbs2IETJ06c0hgT2knWLI4dOwafzxfUZj8/Px/ffvttgkZlbX6/H3fddRfGjh2LQYMGAQAqKirgdDqRnZ2tOjY/Px8VFRXyMXr/DtJ9FPDqq6/i888/x8aNG4Pu4+ccHd9//z2effZZLFiwAL/4xS+wceNG3HnnnXA6nZgzZ478Oel9jsrPOS8vT3W/w+FATk4OP2eFe++9F7W1tRgwYADsdjt8Ph8effRRzJ49GwD4WcdAtD7TiooKlJSUBD2HdF+XLl1OeowMUCgm5s6di23btuGTTz5J9FA6nAMHDuCnP/0pVq5cieTk5EQPp8Py+/0YNWoUfv3rXwMAhg8fjm3btuG5557DnDlzEjy6juUf//gHli5dimXLluHMM8/Eli1bcNddd6GwsJCfdSfGKR4AXbt2hd1uD1rlUFlZiYKCggSNyrrmzZuHFStW4KOPPkLPnj3l2wsKCtDS0oLq6mrV8crPuaCgQPffQbqPAlM4R44cwYgRI+BwOOBwOLBmzRo89dRTcDgcyM/P5+ccBd27d8cZZ5yhum3gwIHYv38/gLbPKdTvjYKCAhw5ckR1v9frRVVVFT9nhZ///Oe49957ceWVV2Lw4MG45pprMH/+fJSWlgLgZx0L0fpMY/m7hAEKAKfTiZEjR2LVqlXybX6/H6tWrcKYMWMSODJrEUUR8+bNw1tvvYUPP/wwKO03cuRIJCUlqT7nHTt2YP/+/fLnPGbMGHz11VeqH4qVK1ciMzMz6GTRWU2aNAlfffUVtmzZIn+NGjUKs2fPli/zcz51Y8eODVom/91336FXr14AgJKSEhQUFKg+59raWmzYsEH1OVdXV2Pz5s3yMR9++CH8fj9Gjx4dh3dhDY2NjbDZ1Kcju90Ov98PgJ91LETrMx0zZgzWrl0Lj8cjH7Ny5Ur079//lKZ3AHCZseTVV18VXS6XuGTJEvHrr78Wb7nlFjE7O1u1yoFCu/3228WsrCxx9erVYnl5ufzV2NgoH3PbbbeJxcXF4ocffihu2rRJHDNmjDhmzBj5fmn565QpU8QtW7aI77//vtitWzcufw1DuYpHFPk5R8Nnn30mOhwO8dFHHxV37twpLl26VExNTRVfeeUV+ZhFixaJ2dnZ4ttvvy1u3bpVvPTSS3WXaQ4fPlzcsGGD+Mknn4innXZap176qmfOnDlijx495GXGb775pti1a1fx7rvvlo/hZ91+dXV14hdffCF+8cUXIgDx97//vfjFF1+I+/btE0UxOp9pdXW1mJ+fL15zzTXitm3bxFdffVVMTU3lMuNoe/rpp8Xi4mLR6XSKZ599trh+/fpED8lSAOh+LV68WD6mqalJ/MlPfiJ26dJFTE1NFX/4wx+K5eXlqufZu3evOH36dDElJUXs2rWr+L//+7+ix+OJ87uxFm2Aws85Ov7973+LgwYNEl0ulzhgwADxz3/+s+p+v98v3n///WJ+fr7ocrnESZMmiTt27FAdc/z4cXHWrFlienq6mJmZKV5//fViXV1dPN+G6dXW1oo//elPxeLiYjE5OVns06eP+Mtf/lK1dJWfdft99NFHur+T58yZI4pi9D7TL7/8Uhw3bpzocrnEHj16iIsWLYrK+AVRVLTqIyIiIjIB1qAQERGR6TBAISIiItNhgEJERESmwwCFiIiITIcBChEREZkOAxQiIiIyHQYoREREZDoMUIiIiMh0GKAQERGR6TBAIaJ2ue666yAIAgRBQFJSEkpKSnD33Xejubk5osevXr0agiAE7bZMRKTkSPQAiMh6pk2bhsWLF8Pj8WDz5s2YM2cOBEHAb37zm7iOw+PxICkpKa6vSUTxwQwKEbWby+VCQUEBioqKcNlll2Hy5MlYuXIlAMDv96O0tBQlJSVISUnB0KFD8cYbbwAA9u7di4kTJwIAunTpAkEQcN111wEAevfujSeffFL1OsOGDcNDDz0kXxcEAc8++ywuueQSpKWl4dFHH8VDDz2EYcOG4eWXX0bv3r2RlZWFK6+8EnV1dfLj3njjDQwePBgpKSnIzc3F5MmT0dDQELsPiIhOGQMUIjol27Ztw7p16+B0OgEApaWleOmll/Dcc89h+/btmD9/Pq6++mqsWbMGRUVF+Oc//wkA2LFjB8rLy/GHP/yhXa/30EMP4Yc//CG++uor3HDDDQCA3bt3Y/ny5VixYgVWrFiBNWvWYNGiRQCA8vJyzJo1CzfccAO++eYbrF69Gpdffjm4TyqRuXGKh4jabcWKFUhPT4fX64Xb7YbNZsMf//hHuN1u/PrXv8YHH3yAMWPGAAD69OmDTz75BM8//zzOO+885OTkAADy8vKQnZ3d7te+6qqrcP3116tu8/v9WLJkCTIyMgAA11xzDVatWoVHH30U5eXl8Hq9uPzyy9GrVy8AwODBg0/h3RNRPDBAIaJ2mzhxIp599lk0NDTgiSeegMPhwMyZM7F9+3Y0NjbiggsuUB3f0tKC4cOHR+W1R40aFXRb79695eAEALp3744jR44AAIYOHYpJkyZh8ODBmDp1KqZMmYIf/ehH6NKlS1TGQ0SxwQCFiNotLS0N/fr1AwC8+OKLGDp0KF544QUMGjQIAPDOO++gR48eqse4XK6Qz2mz2YKmXTwej+5ra2kLZQVBgN/vBwDY7XasXLkS69atw3//+188/fTT+OUvf4kNGzagpKQkzDslokRhDQoRnRKbzYZf/OIXuO+++3DGGWfA5XJh//796Nevn+qrqKgIAORaFZ/Pp3qebt26oby8XL5eW1uLPXv2RGWMgiBg7NixePjhh/HFF1/A6XTirbfeispzE1FsMINCRKfsiiuuwM9//nM8//zz+NnPfob58+fD7/dj3LhxqKmpwaefforMzEzMmTMHvXr1giAIWLFiBS688EKkpKQgPT0d559/PpYsWYIZM2YgOzsbDzzwAOx2+ymPbcOGDVi1ahWmTJmCvLw8bNiwAUePHsXAgQOj8M6JKFYYoBDRKXM4HJg3bx4ee+wx7NmzB926dUNpaSm+//57ZGdnY8SIEfjFL34BAOjRowcefvhh3Hvvvbj++utx7bXXYsmSJVi4cCH27NmDiy++GFlZWfjVr34VlQxKZmYm1q5diyeffBK1tbXo1asXfve732H69Omn/NxEFDuCyLV2REREZDKsQSEiIiLTYYBCREREpsMAhYiIiEyHAQoRERGZDgMUIiIiMh0GKERERGQ6DFCIiIjIdBigEBERkekwQCEiIiLTYYBCREREpsMAhYiIiEzn/wFUuQzeclcvZwAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Model\n", "net = ts.utils.net.common.Net(\n", " env.observation_space.shape,\n", " env.action_space.n,\n", " hidden_sizes=[64, 64],\n", " device=device,\n", ").to(device)\n", "\n", "optim = torch.optim.Adam(net.parameters(), lr=0.001)\n", "\n", "# Policy\n", "policy = ts.policy.DQNPolicy(\n", " model=net, # value network\n", " optim=optim, # optimizer\n", " discount_factor=0.99, # gamma\n", " estimation_step=1, # n-step returns\n", " is_double=False, # double Q-learning\n", " target_update_freq=120, # how often to update the target network\n", " action_space=env.action_space, # action space\n", ")\n", "policy.set_eps(0.1) # epsilon-greedy action selection\n", "\n", "# Collector for training\n", "collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " buffer=ts.data.VectorReplayBuffer(20000, 10),\n", " exploration_noise=True\n", ")\n", "# Collector for testing (without exploration). No need for a buffer\n", "test_collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " exploration_noise=False\n", ")\n", "# Pre-fill the training buffer with random transitions\n", "collector.collect(n_step=1000, random=True)\n", "\n", "# Interaction\n", "returns = []\n", "for iteration in range(1000):\n", "\n", " # Training mode\n", " policy.train()\n", " \n", " # Collect transitions\n", " result = collector.collect(n_step=100)\n", "\n", " # Train DQN network on minibatch\n", " policy.update(\n", " buffer=collector.buffer,\n", " sample_size=0, # use the whole buffer\n", " batch_size=64,\n", " repeat=10,\n", " )\n", "\n", " # Test 10 episodes\n", " policy.eval()\n", " result = test_collector.collect(n_episode=10)\n", " mean_reward = result['rew']\n", " #print(iteration, \":\", mean_reward)\n", " returns.append(mean_reward)\n", "\n", "plt.figure()\n", "plt.plot(np.array(returns))\n", "plt.xlabel(\"Epochs\")\n", "plt.ylabel(\"Returns\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Return: 200.0\n", "MoviePy - Building file videos/cartpole-dqn.gif with imageio.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Evaluation mode\n", "policy.eval() \n", "\n", "# Create a recordable environment\n", "env = gym.make('CartPole-v0', render_mode=\"rgb_array_list\")\n", "recorder = GymRecorder(env)\n", "\n", "# Sample the initial state\n", "state, info = env.reset()\n", "\n", "# One episode:\n", "done = False\n", "return_episode = 0\n", "while not done:\n", " # Select an action from the learned policy\n", " action = policy.forward(ts.data.Batch(obs=[state], info=None)).act[0]\n", " # Sample a single transition\n", " next_state, reward, terminal, truncated, info = env.step(action)\n", " # End of the episode\n", " done = terminal or truncated\n", " # Update undiscounted return\n", " return_episode += reward\n", " # Go in the next state\n", " state = next_state\n", "print(\"Return:\", return_episode)\n", "\n", "recorder.record(env.render())\n", "video = \"videos/cartpole-dqn.gif\"\n", "recorder.make_video(video)\n", "ipython_display(video)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q:** Understand and run the code. Experiment with the hyperparameters and compare them to the previous exercise. In particular, what is `estimation_step=3` in the constructor of PPO? What is its influence?\n", "\n", "**Q:** Implement scheduling of the exploration parameter with the right hyperparameters." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**A:** The main difference is that much more data is collected by the 10 workers before the DQN is updated (10 times in a row instead of 1). Distributed learning can also be used in DQN-like algorithms ans speed up learning. Collecting 1000 transitions instead of 100 at each iteration is even better.\n", "\n", "`estimation_step` allows to use n-step returns instead of the vanilla 1-step return r + gamma * V(s'). Choosing a value of `n=3` (and double Q-learning) stabilizes learning a lot.\n", "\n", "There are many ways to implement exploration scheduling. Here is one that works OK. " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGwCAYAAACD0J42AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAClsUlEQVR4nO2deZwU1dX+n+ru6Z59hgGGYWBYXUBZBUXUKCpB0WiMmDcaNLjEJQETJYuSGNfXYMxmTIwmvxg1iUZjXjURExNEBRdAQVFBRUE2hWGffabX+v3Rfatu3bq3qnqmmu6B8/Xjh16qqqure7qeOuc552i6rusgCIIgCIIoIAL53gGCIAiCIAgREigEQRAEQRQcJFAIgiAIgig4SKAQBEEQBFFwkEAhCIIgCKLgIIFCEARBEETBQQKFIAiCIIiCI5TvHegOqVQK27dvR0VFBTRNy/fuEARBEAThAV3X0draivr6egQCzjGSXilQtm/fjoaGhnzvBkEQBEEQ3WDbtm0YPHiw4zK9UqBUVFQASL/BysrKPO8NQRAEQRBeaGlpQUNDg3Eed6JXChSW1qmsrCSBQhAEQRC9DC/2DDLJEgRBEARRcJBAIQiCIAii4CCBQhAEQRBEwUEChSAIgiCIgoMECkEQBEEQBQcJFIIgCIIgCg4SKARBEARBFBwkUAiCIAiCKDhIoBAEQRAEUXCQQCEIgiAIouDISqAsXLgQxx57LCoqKlBbW4vzzjsP69evtyzT1dWFuXPnom/fvigvL8esWbOwc+dOyzJbt27F2WefjdLSUtTW1uJ73/seEolEz98NQRAEQRAHBVkJlKVLl2Lu3LlYsWIFFi9ejHg8jhkzZqC9vd1Y5vrrr8ezzz6LJ598EkuXLsX27dtx/vnnG88nk0mcffbZiMVieP311/HII4/g4Ycfxs033+zfuyIIgiAIolej6bqud3fl3bt3o7a2FkuXLsXJJ5+M5uZm9O/fH4899hguuOACAMCHH36I0aNHY/ny5Tj++OPx73//G1/4whewfft2DBgwAADwwAMP4IYbbsDu3bsRDoddX7elpQVVVVVobm6mYYFEr6EzlkRJONijbcSTKYQCmqdBWzKaO+PojCVRV1VsezwSCqC4SL1/LV1xtHTGUVIURN/yCAAgkUyhsaULADCwqgTBgH2/YokUdrWmlykLhxAKamjujAMABlQWoyhYmJnmVEpHNJFCRyyBiuIihEPe9lPXdWxv7gL7aQ1oGgZWFTt+ZrquY0dzF1K6jj6lYezviCmPp0gypWNHc6dxPxjQUFfp/Hrs/W3n1gOAiuIiVJUUub5mIcL+vjpjSXTGk0gkUwgFAyiPhBDQ0sclm7+bzlgSe9ujAIBwKIDaimKXNdJ0xZNo7owjpeuePocDCf/d7F8RQSTUs9+j7pDN+btH04ybm5sBADU1NQCA1atXIx6PY/r06cYyo0aNwpAhQwyBsnz5cowdO9YQJwBwxhln4Bvf+AbWrVuHiRMn2l4nGo0iGo1a3iBB9Cbue2kDfvqf9Xjk8uNwyhH9Lc/d9e8P8frGPXj8quNRGlb/SUYTSZz769cQDGhYdO1JCAgnr9+8+DGefWcHfv4/4zFmUJVt/T+v2IJb/rEWKR24+pQRWDBzNABg9Zb9+J/fLUef0iL8+YopGD3Q/qPx/vYWnHffa4glUwCAX104AeeMq8esB5bjnW1NAIBjh/XBk9ecYFlv0552fPmB5djTFhU3CQA4vLYc/7nuZNt7ySdPvfUpfvyvD9HUEUMilRYZI/qX4b/XnYyQBzH1jb+8hefXNVoeu2DSYPzsy+OV63znb+/gqbc/szw2dURf/PWq4x1fS9d1zP7DCqz4ZJ/l8WlH9scfvjbZcX+veORNvLR+t+WxYEDDbecejYuPH+r4uvlgX3sMT67ahklD++C9z5qxqzWK759xJDRNwysf78alD72Jq08egb+s2IKWLtMyUFwUQFEggNNG1+JXF6bPL7qu45WP92BE/zIM7lMqfa3P/2Ip9rbHjMcWzByFq08Zqdy/PW1RnHnPMuxpM9e58NgG3DVrnB9v3xfE7+Y/552IcYOr87dDLnRboKRSKVx33XU48cQTMWbMGABAY2MjwuEwqqurLcsOGDAAjY2NxjK8OGHPs+dkLFy4ELfddlt3d5Ug8s5P/5P2av3gqffw2o2nGY/ruo4Hlm4EALzwwS6cO75euY2X1+/G+p2tAIC2WAKVxdYr3fte2ojOeBJff2QVVvzgdNv6f3tzGzLnW6zZ2mQ8/ty7O5BM6djTFsM1f1mNF78zzXbl/va2/YY4AYBVm/ejf3nEECcA8Obm/YglUpZIw8vrdxnipCioIZ5kUQUgpQMf72pDazRRUFft8//2ju2xT3a3451PmzBpaI3r+qu2pMVCOCMOYskU3ty8T7n83rYo/vHOdtvjyz/Zi+1NnaivLlGuu2ZbkyFOioLpCEEskcLL63dj+Sd78bnD+0vXS6Z0vLphT3o/QwFoAKKJFJIpHTc9sxaThvaRCtV80doVxzF3LLY9vvj9nThtVC2eXLUNyZSO37680bZMVzyFLqTwjzXbsXF3G0qKgnhz834AwKShffB/3zjBts6Tq7YZ4iQU0JBI6Vi9Zb/jPj7x5jaLOAGAJ1d/iu/MOBL9KyKe32suWblpr+X+nD++gbdvnpGnvXGn27HVuXPnYu3atXj88cf93B8pCxYsQHNzs/H/tm3bcv6aBJELUkJGlf9BCwedowgf7mg1bscSKdvznfEkABgpF55YIoX1jeb6yZS5H6u3mCfPLXs70JJJv/Dsb7f+8DZ1xvFhZntnHl2Hosy+i5GSLXs7AABXnTwCL8w/xXj86lNGgmmgaGa/C52HXtvsabnWzNX7ku+cgue+dRIAGCktGR/vakMypWNITSneuWUG7r5gHEb0LwMAR2EDwPhMTzmiPz6+8yx89L8zcdbYOgDW74vIzpYuxJM6QgENH9x+Jtb/70zLifrrj6zy8E4PHB/tlL+XDbva8Ptln2B/h/r48qz9rMUQJ0A6eihzOazdno7SL5g5CndfkI6AdLp8T/m/keULTsNhteVIpnR82Ogt4r95Tzt2t8ojjX6QTOnG9/Dbpx8OAJZIUyHSLYEyb948LFq0CC+99BIGDx5sPF5XV4dYLIampibL8jt37kRdXZ2xjFjVw+6zZUQikQgqKyst/xNEb4QXBgCwZa9pMI9KRId1XfN5t2VFPmvqtERAktyPcqvwIxVP2rfNTgDsSrCpI4Z9mR/k2soI+mc8KTsFcbRpT/r9De9XhqF9y/D1k4ajPBLCBZMGG36Xrnh27yWXdAknoSevmYpn5p4IAFj07g5s2KU+6QNpTw77bMojIVSVpiNDzZ1xpFJyux8Tm2WREKpKivA/kxtwVCZ6sVe4IhfZ15F+vl+5eYV+xIAKADCibTK27UsLx/pq0+cyuI8ZqfmsqVO6HmNHcyd+sfgj7JKI4Vywozm713ns61PwyvdPxZyp7qmqJR/ssj3WmPHmDOpTghLje+osUNpj6efnf/4IDKwqQUPmeH663/lYvrl5H0b/6HlM+9nLOPveV9AZy41gb+mMGxHU/zm2AUA6gtsDG2rOyUqg6LqOefPm4emnn8aLL76I4cOHW56fNGkSioqKsGTJEuOx9evXY+vWrZg6dSoAYOrUqXjvvfewa5f5pVi8eDEqKytx1FFH9eS9EETBI0ZQdnFXTKJQEOFFhSyC4oQoOnihFE9Zn4tLTqTs6nBEv/SVfVNH3AiB15SF0b8ybSDcJVwBbs+c6Boyef4fnj0a794yAyP7l5sCJZHdD/J/1jXiv+vk6eCesqvFuv/D+pZh/OAqIwU1/RfLLJEokfao+V6Y4AAAXQdao/LPl32WfAStmhM2TjRlhGNNmZkiY54KUSzysBP+IC591L/cexriiodX4d4lH+OqP6/2vE5P+MzlJM8zrG8pTjisHxpqSo0Tsci3Tj8cFcVph8MN//eu7fntTabxuzhjbHeLoLRnPt/SzPKDMgLFbd9venqtse1drVH89/3cfLeZmK2IhFAeSb/3lA4j7VqIZOVBmTt3Lh577DH84x//QEVFheEZqaqqQklJCaqqqnDFFVdg/vz5qKmpQWVlJa699lpMnToVxx+fNnvNmDEDRx11FC655BLcfffdaGxsxE033YS5c+ciEimMPB1B5Arx3M8LljbFCUy2brYCJSH8CPH34wnrc3HJttmP24j+ZVi5aR+aOmPYm0nn9C03Iyhiiqcpc4LtkzmBapoGVtRQnPGquF2Z8jR3xnF15qT44R1nOlYddQdRLPUrD0PTNBxdX4nXN6bz94vfb8SRdRW2dXVdx46W9MkoHAoYXpziogC64im0dMalXhsmHnnvDlvOTaCwKFafMrP6sbgo431x+I4wscTvTyCg4ZQj+mPpR7st0RQZ7+9Ipy3WcB6kXLGrpQsL//0hgLSgOrKuAsP6luGPr22SLj+g0qy2Obq+Crd/8Wi8tmEP4kkdt517NPa2x3DUwEoUBTT8fPFH2NuejgbWZI5hKqUb4m5gVbHx+bhFNphAYSf/QdVpobjdJRol+r1EH4tfNHWY3xX2HQHSwstrhdqBJqu9uv/++9Hc3Ixp06Zh4MCBxv9PPPGEscwvf/lLfOELX8CsWbNw8skno66uDk899ZTxfDAYxKJFixAMBjF16lRcfPHF+NrXvobbb7/dv3dFEAWKGEHhRUdrl/PJiE8RyE4+fDWjeNIXX5ePoCTECIosxZM5EQ7rm4mgtMeNk2PfsrBx1cj/iOu6jubMFX51qb19QKQbKR4+R3/bs++7irps4Y/bBZMGGyWiwzKRIwCoKZNfSP1qycc4855XAJgnKcAUAU0Kn0RMIlCqS8KZdZxPVuxzqeGOL2/OVdGWidaVRazXqDefk45iuwmjA8l3/25GOC6ZOhR/vPRYfP/MIy3LnMOZy/sJhtSvTR2G310yGX+89Fg01JRiQkM1wqEArj39cNRmluVFRGc8aVRv9SkNcyke5+8p+y6yY1oeSa/X4SJshnPfLcD8bPxmXzu7WAgjHAz0Cg9YVhEUL7mq4uJi3HfffbjvvvuUywwdOhT/+te/snlpgjgoEH0I/N+U2w8TLypiSfuPSiQUMH5EW7riluhCQnhdPl0khnhlJza2XRa2bo0msDPT26RPqXlFxntjOuNJY1vVkshBJIsISiKZwsOvb7b0lPjrG1tRXVqEG84c5bq+V9j7HNq31FIW/I1TRuKxlVsBmFfKIve88LFxuyxiHvuqkiLsbIkqT/rsmPH9YJh3pcktgpIRMLwAZMIv6nBCbYumt8vSHIy+mShCa1fCVpGlYtiNz+HWc47CpScOd122O7zFVc8w4cd/t4f3K8NPZo3Fs5lKqLIseg2xY55QiP9wKGD0LnJL8TAhwj57duychCJgvbAAzM/Gb/azCEppETRNQ3FREB2ZnjGFSmHGdQjiIMUpxePmQeHXlZ18+EBIS6d1W8mU2oOSSIoRFPuFCIuq8D6FbfvSV52VJSGj4RN/NcYiBkVBzYiw8BR7NB8CwNNvf4b/fe4D3LHofcvjH+9sc103G9i+lAipo4aaUsyeMgSAe6QLAMojpiBjV9TtMfnna6R4gnwExTnqYu4vM9ia+8uEX9TB28O8MuVCBIW/n41Z89Zn33dNO27Y1Yr/eWA5XsuUN3uFP4GLggpIH7+SoiAGVKa/m+cfM9i2jApWfcb/DTBBEQxoCAY047uwrz1mmL5lMOFalullZAgUl+PCjjPzA7VFcyMYxGib18hQPiGBQhAHEFuKh/ttcAsF8+tGBVGh67rlSk1M29g8KNzzzBTLflBFwQLI0xCMyuIiI4LSxf0Ys5NrVUlY2k1Tto6KjbvlJ4Z+5e6dp7OBRTNk3pY+mR92L6WZ/ImUnbA6FALFMMlyx5Y17HMTCUzc8NGXcMgezRJpVaR4QnzoP0vz8iqXkui5j76NNzbvw+w/rMxquwHuuyMKKiB9DDRNw9+vOQEvzD8Zx4/o63nbrJEdL8pN0zLzEJnfhW8++pZyW2KKJxwMWrangv3dswo5v9OWjH0dVr9SNhcI+YIECkEcQMQyY150uIWCUw5VPOJ2RUGSFD0oSXsEhUU5ZPshOxEyyiMh48fOEkHpZOkHeRO2bH4g+yi2IfO2qPBSUsn2hTcRMipL0iceWZ8YIN3QizGyv+krYMdVJUDFEyIAhDJX9mKFlYjsc4l4uHJnaYRySUTCiIZlacTm+4vI2K3oJiwjldLxo2fW4vE3tloiKDKBwr7rDTWlOKzWbl52gn1mvGCPGcc0/Rw/nuKDHfKeJqmUjpZMZI3toyEUXf6uOzLfOeaHefad7crX6QlNzIOS+Vti33FK8RAEAcAeQeHvul1pJR1MsqLHxMkUC5iCJZnSjdRRaUYwyFM86ccikghKeXGI85OY+2UYZBVdYoslaSEVqjC0mLpSkUrpOO+3r+PLDyx3FCmmQLFHUFjn3hZFioev7Dmq3hw1wK6oOxShe1kVj+GNcCkBjUvEjReBwVI8FZITvpcIjIx125sdn89mmsGyj3fjzyu24Man3rOkuWSCyk3YOyE7zubnkT6OfLpPlqoE0r1OuuIphIMBw6flPcWTjpjw1Ufn//b1rN6HFyiCQhCEI6IHJZveJk4RFLHyRhQs7D47kTHBwq/H+j3IyoxjnJHz4cuOtTxXFDSHDPJpAWbwVEdQ7KJGhUoUeM2fN7Z04Z1tTVi1Zb+yHwkAo/17sWSIWmVGaIn+HgY7pkcNrMRXJpv9N9hJTeVBkaV4ZN4I6bqZE2tRyDz7e/GgtEblKR5+/WxL2d1SE9kMzVP5sfgICtsca0zXHYxIFe9BEfrS8GXARyna/7Oy68MHlBuix6imcvgcXl6/Cx9lfFTMQwPkJqphlBmXkkAhCEKCU4rHLRTMBwvEq1tbSkeMmCQFn0nmeV7IsBOp6F8BuLB3KIBpR9bizKOtXZ9lERTegyIjmx9IlTG1M560vVcZ/PHqUqRa4sn0vBZA3jyO+UpUYomd5G455yjBT+Kc4olK0jShQMYb4fLe3FI8qmhRm5CO4Al7EDgy2l38MsEsBEpIEW7hI1vPzjsJX5o4CPd8ZYLn7apeR1bFw3+GXz8pXaHE95vhYemrBm7woJcqnjuf+8C4feww9zlPIlv3dnj6/gPm+2IXBmSSJQjCEf635Z1tTdi4W12VYo22WE8GTp1iAfMHmJ28mGDhr9CZMTMmiB1d17kTYfoHvTRijTBEpBEUZw9KxGMZJqCOWvx99ac48a4XXStr+OdVJ1L+qlU2E4WleFRX93EjmmH9WS11McmyRnndiqA4pHhSuj2SxjBSPFIPSnYRFKYlOlwjKJ42B8DevAwApo+utaRBxgyqwi+/MgENNfZpxF4xhKCkiocXfWxwoirtFZV4l7wcx493mX/vI2vL8dClZnTSTbj/ZcUWnPzTl/Dw65sdl2Ow72cw8567Gyk7kJBAIYg8Il7hXuxQ4WBp1CaWBosRE0Wkhp0Emdjh/Sbs6lRM8SRTuuGVYSfCc8alG2OxsLRsro6bB0XldfjPukas/czqZ1BFLYB0+ubf7zm3B+dFhaqPCV+6LRMhZorHOYISFozErARY5UFhPW2sJlmPHhRZBIU7SapOqGLFCU84S5NsbUVaNLhVoQWyUCji9/fw2nL8Yc6xiqW7T8gQgs4RFHZMVX6pqGQdLx6UgVXpY/fdGUegX3kE047sbwgHt8GBNz2zFgBwx6L3cf0TaxyXBcxjyqJG7DvjFrnNJyRQCMInlm/ci/+37JOshm+JjduchqI5eVDEK22VByViS/GYkZGwJB+fvm9ui/3onjqqFo9dOQWLrv2cZbuWCEqHsweFXenz7+XDxhZc/efV+MKvX7UsK05o/tWFEyz33c59fARFmWrh9l2c3gwAlZloQ2s0IQ2rqyqdWAQlGw8KO4k4VfGkUrrxORZxc3x4oSM7OaZSuiFQZCmebK+sWdpDFSFiZBNBEVNsX1HM1OkpZqM2ruw+aY9ouRmP2bGKcN4l04OiPo7s8zt99AAAaZ8OKzcW51rx7BO+n0+//ZlyWQb7LhkCJfP+ZJ6zQiGrTrIEQai56P+tAJDubDn9qAGe1vGYPgYA8BfTogARK29E4cOqXYwIChMoSXZVFTB+rEWBwkdr+JPvCSP7GbdZBGXrPrNlODshy67S+X3hf/Q37+kwbieSKYSCAei6bhu4xkLzDFlKgKfFQwSFP5HMHGufrF5RbAqttq6E0e1VXJ8XC4C7B8U4IXLHtkjSn8O2HndS5dNKgYCGoqCGeFKX+kg6uCiALMWTbRUPK1t186BkE0ERGxHK9tMPDCHIHWcmZmW+HmWKxxAokgiKQ4SC/R3y39/+FRF8ur8Tu1vVFytihDH9HlLSNgDia7GoUZHigqSQoAgKQfjMln0d7gtlEMuBHZflRIe4mq0xm6IvCrvCS6dtzOZuoaCmPCnyJ26VeZH9MO9pi+L97emKBnaSUQ30k5kx+ZP7G5v24Su/W47fL/vEOAFMGV6DB+dMNkomGW4CxZLiUVzp//y/Hxm3f/QF+2T1cChgGAtlKSfDgxKUiyeVmVEmbNhJhH1OMiyRLeE12ecsM0CykQrBgCYtGze9QXLBIYpfdkxiiZSjZ4b/iK780ypbFIBH9F/k6hxqlhmbLzD/b+8AsApZN+Mwe1yW4oknddsxY7DX5b+/rB+KU4pn815740K3URkJ8qAQBJEN4rnH6SKTFzPiSUucSCz2B2Hr8ieklG7+aBUF1REU3luhKhU9vLbcuL0hY/SNKjwZDNkPJF+q+uCrm7By0z5jkm1tRQRPXD0Vp48eYCv3dNN5lhSPwgvy3Hs7jNt8tISHXcnL5urI+pkAZrRHZVg1/QumkONFjiqKwofmRVHkZLI1mrRFQtLP04gWKKo7nl+n9vvsdDip8hGUxe/vxE/+/SE6Y0n8853ttuMpdhf2u2Mww/CgSD6bddvNZmluJ3Npiof7HqiiKOxlQ0IEBXAWKMzkPItr6+/k0wLMixjRg0IRFIIgpIgRFFn/DQZ/BS7+nIpeBfE3x+iDwv1oJlIpS3WOKuQrVvDI6FsewemjagGYV3LMUBiRdGXl94X/0ecjHUs+3GVZvg/XNXbS0D6Wq063igc+kiCLoHjtBcEEimiitfpBsougyBu1me9NVvbNrxfQ7BEkWQt3RptiDg/DLTXxlxVbrPvBva9fcFEonlRKxyfCHJv1O1vxk+c/xLf++jbmPfaW5TleHI1vqMb00d5SptliVvE4K1w3D4rUJBt0Fyjss+XFW//ytHFW5UFJpnRs25+O0paGzRlEbrO87Cke9jlnkWc+wJBAIYgcowrvAnYPipMI4MWMKGycZu0A5o8T/wOa5E6qvAdF1QROLJ8VYSdvFhqXXVXyGCbCpFygiLBW84xpR/Q3brsJDD40L/OC7O9Qpxt4KoxSY+vVqsUPInyGsl4bPNIUT8A9giIrhxVfUyaKjKF2EfnnYg5+lJ9Uxw6ustzny4vf3rpfus4Tq7bZHutTWoQn3kw//srH1gGCzCQ7ZXgNnvnmCQhk04Y2C7yWc3ut4omoBIpC2IiiAQBqypwHRS546l1jsnZpOOja4ZjB/60DFEEhiEMSMf3Cn5jEK11RaDj9EPPnGvG8I/7AqmbzRCwRFN1Yryho+hHEssMY69PhYMADzDbkrEup7KqSx+idwp0InX5kxRMxn55wGzjIX/nKTLL7272NuFdFUGSVToxg5uSjEqpMaERUERTFCURWbcIwUxf2dZlYU3qDJMKRR3wfHbEkvn364QDSBnEZz0iqTEojIaX5lQnOY4b2yaoDbbaYM4/S70n1GbmZZFlfIj5aGOA8PqqhjzKTrFHOrvhb+NuqT43bxUVBs4GgolcQwzTEp1/Layv+fEIChSByDC8WRP0hihmnn2JLisdmklWLIoD3oJgnpVRKN05yIa5dvdhp1WlQIA+r1mkXBIrMiAmoIihqoSD+kPLnLdcICieCOmJJfNjYgjN+uQzPr037TrxGUCpVERTeDyJUGLlFUGQpHk3TjJOW63rSCIra98KOhfJzCTlHC0Tjbf+KCA4fkPYgqcYIyMzgG3e1WQTKf9c14jt/ewedsaQ5Vdoh5ekHxnHKHEv+ePEDKvnKJplp2RDjwmfB3p9sDECKm4PFd9lViWAZpeGgEdW754WPHLvKMrHKvleqtgKFBAkUgsgxfPhfLLUUf0+crhadTLLiCUC8EpT5IxIps0NsKKAZU1vFE5DK/ClSYRMo9soGHpnx0OlHWbyiP2FkX+O2W7tuPsXz8Oub8e2/rsH6na245i9p74P3FI8qgmIeRzEKZnpQnA2WogA0S2CzW49fV9borcslgmJUECmcx+xYfu7wfjj1yP743/PGGH6Wt7bsx/YmsyQ8kUzh64+skk46/rCxFdu4svSr/rwa//fWp3jw1U8MkazyL/lFkWCS5U/wj379eON2WPi7ETFEn3BM2XG5/+WNtnX448un9LymbIBMiicTcfmwsdWxH0pS+A0ocomUFQIkUAgixyS5k4RbiqchMwlVhkWgCM+5RVT4LpK8aTPOpReMeTpCKSU7EapKjBksgtJq86A4CxRePDgKFCGCcsnxQ9E30yTM3YNiXXf9zlbLfbfwOKNCSGOJ2w9JPERuVTyydvWA+0Rj0xtkf00z+iJJ8bhEUJzEDWCKwWlH1uKhy45DQ02pcVwSKR0n3PWisezSj3bjhQ92SrcDyE+On+7vxN5MCXJNaW6qdxghwYfBX0yMrDXTVbzolEUpZGk6ANif8ZH8853tkt5E3O9CkI+gOI9U4OFTPADw8a5W6XLpcRXWdJLKc1ZIkEAhCJ8RxQL/oyeeSsTfuhrFMDLA+oMm/tjpUP/4AVwPhKA1ddDFXfmxK2oxX26Y69w8KFwERdd1LsXj3AfFGkExrxrHDKrEY1dOMe6LkYRQMIBLTxgGwHmw3YZdbVj8vvokCbh3QWWoTLJOnVld+6CoypMdfCQA13clYP9czA6p6nSE6nMJupZFZ6Ib3P6WR+Rl2W7VMTIef3Mbln60GwDQryK3AqVIEGP8xQQf1eBTMLJ0lSpayH9P7JV2nEDhts/M4C2dcVuk1JYS1jSLQFH5xPiPUvSgUIqHIA4hRLHA/y7ZIx/pRyq5K1AVTiZZ8Rxmm2bM9UBgP1CplG5EHoqLguZ0U7GNPtcO3wneg8JfGatSPE5lxnd88Wg8cdVUnDCyH76VMWDedu4Y2zaYqHKaA/PdJ99x3O/Ne9rxyPLNjsswKhWGRNbHo0oyd8gtIqFK1ZhXuM5NvmRRm6BTisel/Lso6CyouiQN+MoVZlfZd2buqSNx27lHS5cX6Vce8bRcdzFmHqWsERRNKN0OukRQVFEpflFRrKnM80wE8xcQDPF+LJFCaZF57NsyFwcivAhh35ewET0q3DJjanVPEDnGqTyY3WcnI6fOspZOsqII0kVBItzPPB8QzJdd3NWwOfBPiKAI7n8VvBmUFx3qFI+9twQTKOMGVxuCZ/7nj8DXPzfcyM3z1GWGrYmt8Bm/X7YRa7Y1Oe73Nx99y+KFcOq5UaHwB5hzh+xX/F4jKOJxKnJI0wDmZxqURlDUvhf3CIrz68ojKNZTia7r0DRNGnWbOqKf44whHta0LFcYVTwsgpKSf9d575hs190M4UDGSM29nZRCoJSFgwhoaXHT0hU3vGGAvYePGP146LXNKCkK4vtnjrI8zn/3jDLjEJlkCeKQxypQrM+x3wZxquqfV2zBgqfetfyIJS0mWfE1rPdVVT1WD0rK0o6+WNHrQeyfoIJvz86LDlXYWRZBYSd+sfxUJk4AYET/tE9g4+426fOPvL5F+jjP+zvMjqFjBlXi3osmKJdVmWSXfZxOScgiKK7VOIpybKdma/z2ZMKRvaZsXZnAsLyuRw+KJYIiCBT2VmVX8wENGDeoSrptkb5luRUoJUJak59NxWOJoDikeFSiD7CnePjvA/8RptM28lSi2AX55CP62y5Ofisx5MqiNcY0Y/KgEMShC//7ocopixGUHz2zFn99w8zFp59z3465rBBB4TwoIUkEpZiLoHTG5WXGslQCD7uST3IRlHAwoOztYpRuZrbPT9lVtZoXGdEvXd66vyMune3itTqHccZRdcb0YRmyE0cskTIaZ8kiYKGAOpoBqBuuhVyaiCUF0yMP25YsamP6juQ//+4eFHu0QNwHFn2RGTCTuo6+HlI3VSVFrpVjPYV5vth3RyX6+LtyDwoTbdb95adui2KRj9aI1XvMh9IspBLZ30dZOIhXvn8qhvcr8zQ9nf8OGR4UatRGEIce9uiGU+Qj/UBYcULhT7B8NEU8d4g/UfbOsvIqHv5kZaZ45GY+t4F8fPdSLyFvfhy9rutojyWM4+N1em1JOIj6TJrH1i49kXT0pqi254QsgsKLlY932iM5ThEUvheNeDIuchEKCYfPxYygODRqU1ztO3WhBcwIm6pMmV9XFsFh380KRat9Rl8Hw7hf2ASKQoxrmmb03ZE1czM9KNZj8sUJg4zbcUGsGWlXyedXkTEd/331NsvkYuZ1GlBZjIaaUgDAl7h5PCrY5xHQzNcrIpMsQRBWc6sY6Uj/G1IYE3mhYanicYuYKIYFBjXNCF+nhYR5smJXfyoPilujNv5E/OamfQCA/pXqK2X+Cj6WTBlhdtWUXRVsQN3rG/da9n1Xi32WyaNfn4LZU4agsjiE0cLAwfQ+OQuUSkkJKN+Ea9Yk+8mCHW/Zic3aIt/6ngMuQsH4TCUnOP4zFnGLoDgN0NvTFjVm6jj6LZJMoNhPfuyxSkk67EDDBMpeIYIi8/WwShsxxZOuWFMbj1nDN9v4CQdvFxPCf31jG77w61eNx+/+T3pw5sDqYuOx4f3KcMVJw+VvMIOsEi9MZcYEcegh/qw7eVBEk2xS1y3h2IRClNhfw3pf/BHlf3jZb28ipQseFHmKx+lKnYe/8n5zc1qgnD12oHJ5cVYJe92SomBW7c1PywwpBNL9Jhqbu5BK6YY5dkjmShNIh+Dv/NJYvH3zDIwXZsoA9qtcEXbi6IwnjRMtL1a+OW2kbZ2gwwnfyUzMDo+qYZqTB8Vo8taNMmPTg2I/FkvXmylHLyXxspMf26X5nz/C9txFxw0xbn/WJDc++wnzuOzviKUHPhpi3H5MVYIxwXWElR3TkKIaS+zsyiMTb9ubOvH21iYA1sGZAFBfre6fBMiN7sz469R9Nt+QQCGIHOPkF2E3Q0aKx9q8io+EOHWSdZr/A3BXa0FrBMUoOeU8KKJpzmuZMV/9wd6DzDTKUAkUp9SBjDvPM8uPv//3d3H8wiW49vG38e9MG/svTqg3nmdh8WBAk1bcuHXV5FNPbGozi6CM6F8m3XenlAl/ArdNQdacZ/gkHU5wRkROluIxjnP2HhSWYggHA8axlGF4UITXP3JABaYdmR7yeP4xg/DsvJOMXjYAcNFxDRiUOdmO8Wik7QlMZCVTOlq64o6iwfw8rI/zUTtZVEnl9XBKncpSnHykbrcw6bjUJTUpe1/sO1LA+oTKjAki19j8Iro5R8aIoEgqawBrDt9pFo94P6nwoAQFD4pp7gtyTasEgaKobBAxtpvUPbXHDwQ0hIMBxJIpRBNmiqcknN11U21lMUYPrMQHXDXOc+/uwFGZFM6Ehmq88v1T0RZNoLbCDI2XSMSEW7g7FAwYJaDsPbLW/ipPBZ/6YuW3DL67p3iickvxJBxMsk7zf7o8RlBkr8uiRbJUFo/pQTGP5yvfP9UiajRNw9jBVXhp/S7jseKiIB79+hT89uUNuOYUezTKb8KhAEIBzeg5IhsJwTC+38IfW9SlpF7VcI9tRxYBk1Wt8UKI9QZiyBoEWl5L8r5YBEVVTl4IkEAhiBwj84sEoFme46su+KtO/seP34yrB0XxfFDTzFy6pVFbwPgBTunpq/ZAwPoD5tYHhT8Rs3C2m28lHEoLFEsEpRsD4mSGym37OwCkoyayq33ZVedXjm1wfa1QMIBYImWkT4wusgpjL3/cUjrAB6JUbe4B7opdkeJR9exg+wio5sa4lBk7pKTaoukISqWLiZmJJ/b+zj9mkDLiUsadXCOhAIb2LcPdF4x33L6fhEMBJGJJRBNJR9HHHhKFGz+1W5aaVDXcc3otWfrMEPBFQZx4WD/Lc587vJ9teR6xzT1gfm8KWJ9Qiocg/MZWqSP8AMg6wlp6iFgm75ph3Wz6oMjy5ID1Sj3Bp3iKghYDHX9yinPpISf41JHXCchGL5RkytgXt0oaGbL0E7vaH6yYb1QsvM7rN56GAZXF0mUtryVEmtjrlCnKk/mTgni1GndIn5kRFPl+mI3aHCIo0ioeex8T2f7K1mXvVXbFfgPXHOzdT9OVJ04TlxnlEXM/sk3v+QHfj8dJjJsCXhAoLoKvSJHieW3DHuN1RQZIzOX/75VPAKRNsSLVpWHcdPZoANYpzAyZmHVryFcIkEAhiBwj/qDxXWDFPihJriIAANq5xkzWKh7n17APCzTLJ0Ncl1G+iyl/kuR/tMxcufcqHlOgOIsaY2BgPCVtAOaVmQozbnkkpOxrIqZ4vFaViA3UtuxNV7WoIihObdKN9JlDBEVlkjVPOk5pBVkVj1ujNnX0pdXoU2N/r9/gDMJzMyXfThOXGXyqqTvRs55iDq00UzxO4wNsAsVF8BUpUjwL/52uyGEDBXn4VCSQTiO+8EE6FaaaG3XG0XUA5JO949zfPyPoIoALARIoBOEzYht6J78I+81iP2KplDWt084Z47w0fDO3qw4n8z9M7PEAV34MWL0vCY9ig5/x4+XKGeAjKElLCDtbvjxpsPS1yiLqbYmv47W0WTzhPPdu2ow7qq5Curw1giKv5HC8Yu+WByVgWYbHbcq0U2t+I4LisZFeTNHjxbKv3PdKVfqcS/gonpPfSlX14tbzx22mkgyxxT/fhFDVF4efDi7+HjRl+ilVcEMd+Y7ShQoJFILIMU5+EXabH1rGCxQ+mmIxyQqvIYog8UdM7kFJWXppWCIoklJnrybZREo3TkyuKR6u3XanS3WJE5qm4atThtgeL3MwD4pmXDePDUMcxMeugNkVrAh/3GzmZYdeGIGAOgoCOFeBOJlk3a74ndZtVYwiUOEl1ce/92z63/iF8R2MpyxDNUVMwWh93C0ixf6usmmINrDKGkHhBYpqO6yHT9rAbf3s2Kyphhoz3ek2I6oQyPrbsGzZMpxzzjmor6+Hpml45plnLM+nO+7Z///pT39qLDNs2DDb83fddVeP3wxBFAK2ihpFczb+dhEX1eBFCS9WnIcOCq/p0YNidphM/x3Kup46hb15eB8N6ydS5HLCYVfM0QTnQemmD0F2le5U3SCeoL32XmGCg50oWBREFSXgz3XKGUmSE3jILYLi1EnWoU2+2zRjpxb7bBaMym/D09oVx1/fSI8AcIqg8KnDbPrf+EU4k1aKJZ1TPAFFys2tr0wRd/HBEJshioijAPZ18ALFOYKS3ifr9rftyxjG+5hGZbeOwYVA1gKlvb0d48ePx3333Sd9fseOHZb///jHP0LTNMyaNcuy3O23325Z7tprr+3eOyCIAse5D4o10pDSdSGCouqDYn0Ndw+K+cPLCwn2W8dOcjKBYszi8VzFk/LsQeG7WZplxt0UKLIUj8OJtLtCqIg7frputqpXRZg0zZx/ZPucHI6t6oTIMKJvslk8Dj4S90Zt6i60RudhD1GuG//vPWPUQNjhezChoTq9zy7flVwR4U2yTike1jhPTPG4CD7DJMulePZK5kaJnH+M2SZ/b5u5vKoUPhIKGEJwp9BFeUdzFwBrQze371chkHWZ8cyZMzFz5kzl83V11jDnP/7xD5x66qkYMWKE5fGKigrbsgRxMGI3tPK3hRRP0toHRZXisRlvPXtQAmYjrqRuLMdOckUBDTFYr56dyll5zLJYrvTSqwclkTLKdbtbySG7Shf9QDzdFUK8SZb/TJxOsMFMrw2VcJTP07EuI+LkQXGq0HATGU6zg9zEDc9z7+0wbjtFUPpXRPD6jac5puNySZjzbjiaZBVl3+4elPR6fPuAvW32MQwid50/Dk+99RkA4LtPvmM8rhIomqZhYkM1Vm7ahxWf7MVhteW2feS/86rxGoVEThN+O3fuxHPPPYcrrrjC9txdd92Fvn37YuLEifjpT3+KRELuTAaAaDSKlpYWy/8E0VsQxYJuSdWk/2VXmC1dCUvDMV6s8JuxGW+F17Q1hWInQk0z+nAkdS7FkzkpiRUq/G1ZGoKHv+pkIWw3Dwo70UUTKXy0sxWAvIzSC7KToGzCMYOvGFkwc5RyOZEQd/LnT+JOx8cIpwvh+XhKfWxVVSMMJ7+EYeSVTNBln6cyguIhPZStT8hpQjSQvrJ36jqcS/gICjv5yyMobiZZ1fE0Lz4YfFdYFeFQQCp6nLodTxhSDQDYuNs6tFLWdTioMP0WEjmVrI888ggqKipw/vnnWx7/1re+hWOOOQY1NTV4/fXXsWDBAuzYsQO/+MUvpNtZuHAhbrvttlzuKkHkDFknWUZSt5+gfrXkY+O2JcVjMck6R0zEHx2+Z0aQC+Hz5llAXhLp1FKdJ8hddbITmZtvhY+gfLAjLVBkQ/y8IIvW7GlTCxR+3+Zw7dbdMDwFXMdcwDnCpIpoOIkMt1kpCUFc8rCTpXi1zUfk1GXG/kRQeJyqqfINn2Zs6kx/X6olvURUowfcIlJhiejv4NoH3PmlMbZ1GEXBgG30xEXHqZsJVpekG7zx86EAmKlczuOjasxYSORUoPzxj3/E7NmzUVxsdSTPnz/fuD1u3DiEw2FcffXVWLhwISIRe4OaBQsWWNZpaWlBQ4N7x0eCKARs6ReJB0V1IrekePjIi0PzN0Btkg0FNMsJyIygIPO8vTzVaYAaD3+SZb0YvKZ49nfE0NiSzpMfUSsv13XDKY0gY3CfUlz5ueGoLg1nlVYKcVUZ/HF2EijmrCUhguLQBM+tysIp9caa0ImDH/mInFsfFLkHJSNQfI6g5BP2XmLJFJoyFVmyZmeqVvfGdGiX0QF85KM908vkxMP6YvaUocp9E78XFx7bgJu/cLRy+cqS9HFu6bT2Vkk69EEB0u+JdbcuJHL2rXnllVewfv16PPHEE67LTpkyBYlEAps3b8aRRx5pez4SiUiFC0EUCjLjK8PRg5L5zVKdyJUmWTGC4rFRWzCoGZGOZDJlRlY0q0k23oMyY8D8MXZN8WSe/2R3utlZRXEIVZKTgxd4gXLaqFq8+2kTfvmVCY7r/PDso7J+Hd6Ayl8VO0WYVL4OJ5HhtdW9rIEeMwAzkyqjKyN4QwFNmZJS7SsvyLItB/ZS9ZMv+DJjlhKUDZIMcB4rHnYRofSghOyinzVgdBNu4t/crece7Sim2Qyfli5RoNj9SmIDwTw08XUlZ9+aBx98EJMmTcL48eNdl12zZg0CgQBqa2tdlyWIQsTJCO9kaBVn8YhYPChc1ESVNgoGtHR1jsoky/dB0c1wNfuxKpIY5xKSqy8ZQUmJqNcy40/2pHPmfBlktvAniC+MG4gH50zOSdkqH0HhJz07vZYq3+8k/txa3SccxA0TKKoIipPAUHlQeLGcrZG5u4bkAwHfqI01NJPNwlE1zjOOqWuKxz7CoszluPCfraa5C0PWDbmlU0jxpKwXIoBzh+NCIWuB0tbWhg0bNhj3N23ahDVr1qCmpgZDhqQbJbW0tODJJ5/Ez3/+c9v6y5cvx8qVK3HqqaeioqICy5cvx/XXX4+LL74Yffr06cFbIYjc8ffVn+K9T5twyzlHS3O1/J+3Wwmwbnku/a/q4luV4rGVLsM8WckECl+SavY/MCMonkyyLhGUQEAzJv0yvJYZb96TjqAMUszN8QIfhVINbvMD9sO++P2dOGZI+jcrmyZ2PEaZcRZVIwynCiDWhE7st8EiW859SeQG2yi3LbfUnUghe1B4o/Z+hxRPt02ykqikEUFxqVyydNn18J1mDfTECIqsZ46Y4ilEsq7iWbVqFSZOnIiJEycCSPtJJk6ciJtvvtlY5vHHH4eu67jooots60ciETz++OM45ZRTcPTRR+POO+/E9ddfj9///vc9eBsEkVu+++Q7eGT5FstoeB5RMFifs96XeVDE2RsM731Q0v+GFV4HVaM2FpVhJ0O+QoXhtcw4vYz1J8W7ByW7DqVO2wLcU0s9YVemx8Sid3eYPWI8N7GzRiWcoiCqE6K4rkygsAhHp5Di8TIbh0VfWqMJQzgC1tLxbA2VBZ3iyXxv9rZFsXrLfgBAVYkkgsJVv/F4TfFYTLIeIygBTpDIZuyIGCkewYMiRkoB5w7HhULW35pp06Y5/hgDwFVXXYWrrrpK+twxxxyDFStWZPuyBFEQiFcmDKe/CKeur+w5TQNuOeco3Pbs+5Zl2UlB13XrDB+F8bYoFACiHhu1cX08zBSPPV/OTmpeTKjiBZ7XMmPzfveFRT+u+2Yu7X57uB4WUaMs1SX9pYhK8P1p7Ouk/1WbZNXVVaoUD/teOH0uA6uKceSACqzf2YpXPt6NYZmyb7cOtE6U9oIUz6MrtxqPMbMpjzLF4zos0J7iYSZZt94vWzMdYL3C9rstmrBU5sjELP+1OWgiKARxKKOa6OvsQRHuWxqupf8NaJo0RJxM6UgI1SKAehaPumOpbrwO30HSSPE4mGSzEShiSaTrLB5hm9mWr/KMHVSl3K6f8MeWXQm794iRV4AkOQ+LiHuKx7ptHma+FFM8Xjr8apqGo+rTpd6yrsbd+YwKuYpHFuXjh+oxNE3+GbrO4pH8TWUzMiAbWAQlpZsiCJCnA/nRFoXqQSGBQhBZUKS4UrZW2KifU60XCKh/4KKJlGMlEGBvmS9eqZselIBlBocXk2zUY0WOyDemjXQVCuLzPREWgYCGx74+BddNPxwnHdav29txgz82s+5fDkD9vWAEFaW7cYdusG4pHk8RlJhcoLgJKn6II8OtYyoA/Ony43DcsBrb490ZAHmgkEWEZKlGldHZtZOsJMXDxEOpz96c4qKg8TfU0uUsUIDCb9ZWuN8agihA3JqVyXDug5L+N6BpypNzWqCoK4HS20z/KxMYgOhBMctkzQZu6eWMQXjc+tlEUHhuONO9O6v4o97TabYnHNYP100/wvUE3BNkw9q8RlBUqTeZ+AsqrtgZTv6V4oxJtjOetHxX4kn16/EYQxy5CIyXFM/JR/THo1dOsTz28GXH5mUIoFekM5wkqRdVZ1+ziieLFE+OIigAUFls74Vi+sis75UiKARRwCRTOl54f6fjbAz+B14lUPjfLJWBVXbf9KBoypMz3/FV/RrMY5K5UrelEiQelJR1mjF7HrCWmMYSbOCb/z8X9hRP4f8kdQmTYgEPJdjKVvfqKIjK88BwrOLJnCxTurVBWMJDigcwPwc+gmIaO51PqqL4cZooXQjIvnMyMR4wPg/r464mWa4snWFEUHLgzWFpnh3NncZj4oUIgwQKQRQwf1q+GV//0yqcfe+rymX4P16lQHGwyXrpgxLQ1FdgKV1dNmzeT/9bpKriyfw4Wjwo0hSPPUXErrpz4esQRU8uvSN+IQtouJlklREUhy69XvugyHxRvGGzK2ZuwHMEhSu9ZTRnrsiznZmTy4oqP/D6nVNV8ZgeFLcICuddYhGUHIg3Nufn8odXGY+pzNhOgyELgcL+5hBEjvnvup0AYLRZl8H/8ap+bC0RFFuXVyjvW02yDhEU4SRlGxaoW090qmGB1j4oDibZlN0km4vohijKemKSzSde+6DYPCgOURCvfVDkwwIDxnehI256EQwPiougkkVQmju6J1Dcokv5xut3ThXRasuIjXJFiXxPqnh+89WJGDe4CueOr8czc0/0tJ+7Ws1oMGs8J87cYrgNpMw3JFCIgmd3axTffHQ1Xtuwx/dte0mN8z8s6giKGqdpxjoXQVGF3WUpHlXKx4igiFNsLcMCTRFj9EHJPCbrn+GluVd36Y0RlC+MG2h7zC1l4j4sUJ1SENdhuHmDZJ8l25bbcTY8KFw6qznTnVRWgivCD8Ar0HOfgdfvXEDhCWqLpoWbKpUlS/GwEQRufVC+MK4e/5x3Eu69aCImNFR72k9+m+982gxA3TNHVf5eKBT+rwFxyHPLP9fiX+81YvYfVubl9fk/XtWVp1NvIOc+KOl/NU2ztadm8JEO4/UUr6EuZ+U8KJIICvuhqs5cHTdzBrsY16DLjcosG635bZI9EPxk1jj7gy5KVxVBSSTVUZCQS4qHiQfVCVbWCyWeUL8ej6yKh30nKj1EUL4y2RzmWvACRfhe//Ir46XLKSMomWoZVZNBWdq0PcqqePxP8Tx02XHGbdZoT9aoDXCP0uWbwv81IA55Pt3f6b5QN/EUQeGuYFUdNB0jKGI6hls6xaVYWNt0ABhUbbZ8T+q6YxSGfw12slJW8WiapXxVNMmyFt/7M6FhILsqnrmnHgYAOH2Ut7ladoFS+CmeskgIRw6wTlwW59aIqDwoxufiMM1YZZJ1S72x+Td8L5S40XfFLYKS8aDEu+dB4auaasrtXVkLCbEq6bjhfaXLqcq+WUWOKoIiTjNOJFOG8HOLoHSH44bX4IqThgMAPt2fbvTmGkEpUA9KYdurCSLHaB56jvJXPqoLDZ07P7nN4pEN/QtoQFVpEd65eQZKwkEUBTVMvGMxmjriSKV0z31QZCfCVMrsQhvkPCiySbxsiitrPa/relYpnq9/bgTGDKrCeI/haDFv3xtSPID9hz7uIlBUfVCMqhpZiselzDjqIhzNXiicSdZDq3vAFD18BVC2JtlHvz4FzZ1xi9guRGxpRtWUZ27IJiOaSBrHSOlBMfqgpJfj/W65amDHjvlnTemLO1XFV0jRlqBQIIFCFDy5jD56iaBY87PyneGjIuISTn1QUkIEo4obUhaUdHxVbZPdC0mqePh1Q4GA8SMV48L37LX6lKVfn5nr+BOUl2qMYEDDiVk0SRtVV2m53xtSPADw/o4Wy31ZbxSekCrF42SSzRwKZQQl6Zx6YxEUPsVjtrp3M8myCIq5bkuWAiWb70E+sTULVAkUSUSrjWuGpiq/Dgspnj+8sglA+qIkV4KcDd38bL9VoIipPWrURhA9xKmE90CgMinyZNPqXjZTRyaU+JCyeJJStc+XDQu0lElzHhRefLALeCOC0p4+GfEiJhfiIRwKWKIthV7xocJtPllQ1UDPqczYJYLilnqTeVBiHjvJSqt4ullmXOiIx68oJP8Osr9R/vNgJb1l4aDSQC9OM2Yp6+NHyFNJfjA4I1A+3e8cQaE+KATRQ3IbQfGQ4uH+eJUpHssdl4obiwcl/W9Ash/GXJ2UbBtWxE6y/D7zt4Pc/A1LBCXzGGvy1Bq1C5RcNGoDgCevnmrcZv0hehtH11c5Pu/qQZEOC3Q+eXhN8XTxVTzZdpLlPn82KPNgEyii70n1PS+W+HJaMxEUVXoHMFM8MUOgpH0hV548opt77M7g6lIAwN72GDpjSZsZnkEChSB6SL4N5ry/QLUrzlU86vu64ocDsF5B24YFKkQQuzLmIy4JS5rGbHXPiw/2WmI4Os5VmagMwj0lHArgG9NGYuKQapx0eO9IC4gsOMu5rb95IpCXGUuHBbr0qIi5DO8rlqR4vAwL5LdpLTPOVPEUH2wCxTwNapq6lQAzwfJD+FgExalbLv83peu6EdVo6FPasx13oLIkhIrMPn3W1KGMoDChGksW5oUBeVCIgieX+sTLKddLjwAnl4qjB8UhxcOf1Nz7oFhTBYlU+sdQ0zTjyo39+MpSPOy1RFd/d+fwZIuXuT2FyqxjBmOwy8lGFUFxatQWcPEHuFbxyMqMPUZQyjJD7JjHIp5MGb07DrYISglXSaPr6qgqEyGtnO+kzYigqI9JkVGynURTR9wQNSwNkws0TcOAqmK07mrDjmbTlCs2ajOibHH3NHY+oAgKUfC45fd7gieTLHfVq0zx6PLbgKxRG/ecQ4rHFCjOPhbAFEX8iYetY5yUAgHLiPXVW/abr5V5/SJhFk+Xy5yRQ5WHLjvWuO3l+2lU8YgN9FiESjYs0KUPilt1FTv5dFhSPMyD4vzFZ16kpo44dF239MXx0gelN9G3zFsZNBMoTGDwt8sdphLzImBbJr1TWxGxjCPIBayEmTfyiuXsRpRNmHq9avM+/PblDVj20e6c7qMb9KtDHNJ4iaDELWXGiioevjusS6t7+Swe2RV0+l++X4lqm2ajNvNPmq0jDoiTXq0rIiiGCbDAB74daE490uzz0sfDCY7pD7sHRd163qmJFv+dcKvi6ZKkeNz8RKwfTiwTOWECpSIS6tZE70JG0zTUVRa7Lsd8Ju2cQGn1kOLhJ0vf9MxaALmNnjDY59/K7a89gmLuG8/rG/fi7ufX4/l1jTneS2dIoBCHNJ5MsrxAUSzDP64SD7LndSOCYt8m70Fw96Ck/+UrENg6hu8gc6WtMsoB9mGD7R5+gA9Vfn/JJJxx9ABce9phrsuGVH1QHObpODXRspiXXfugpE8+733ajEeWb7Hsj4qSoqCx3f0dsay6yPZGrj4lbVgd0a9MuYwRQZGleCLq48L3Onk303reLSXoB+zz51NS4t++Gd2xCpRsukfnEvrVIQqenFbxeFgmnmWZsX2Qn/W+3IPi7EFw26bhQeFOPOmr8yBiCavvQNULAbDP5jB/gOmnQmTG0XWYcXSdp2VVYsNodS9JubBKGvHkAVjNq26dZNnV8ZV/MqfbFhc5n3g0TUN1SRF2tUbx+BvbMGlYusvxweY/YVx6wjAMrCpGQ41aOBgeFC4iwQS8qs09ABRLPp++B6C7LhNGzVxXaPFvXxZlA7ybqXMNRVCIgieXfVCybdSmLjNWp4FUg/3452QRFKPLo6RRm6o7LX+iY4+JYX3xKoq/mDbNnOl1jBx7ljN2CCshzvDMk3AYFlgsaZbGYFe4AU3d00Q0ye7jTlReIiF9Mj6U37y0AUs+SE/9PlgFiqZpOHPMQMdycVmKx0sVj+zzqS7JvUBhHpd7X9xgPCb+7RdLjNSA6W/y0pwxl5BAIQqe3JYZuysU/qSiEkuWCIrwnKMHJbNpqUmWeRAk04xtlUKwV2ewk59xElR4UPgIilGmrKdflzwo/qAyvDoNCzT6biTsETy3HiiAvQ9KCWfK9FIqzEcFXt+4F8DBK1C8IDPJeumDIqPKw0TonlIqmfMjRmqLJeMQAD6CQgKFIBzJdwshSxtzZQTFxN711cmD4mCSdewkK99mMKBZzLUAbCkeewTFnuIB0ukIw4OSo5khhwrqCIq6iqfYIcXj1uYesPdB4U9YlR5OkLz4ae5gHpRD93sgenoAoC3T0DDbFCg/0iJXyASKiKwUHTCnXud7NhYJFKLgyXeZsZdR5NYqHvVz6ed5D4p6P/g5GbZeGApfS0Cz+x3EqyExncCLEv5KPpnSzSoFSvH0CFZmnE0VT7FDjwqzP417eSs7+WQbQeG9LXvbY57XO1jh0yHsb7rNgwdFxoFM8TihMsmSB4UgCgAvf368OFBW8Uh8Jeb61mVTkmUdIyi6bJqx3NeiQePSCVaBEs782Ih2B2uKx7wdT6WMCAqleHqGamqsk0nWiKAkZCZZ5yZtgP2Kv8QSQXEXGrKr577lEdf1Dlb448eOf3dN5AciMiFGUKolURtVHxTyoBCER3KZ4vEWQTFvd6tRm2Mn2fS/sqpPPoLi6kHhOtKKJa1i91AxgsKfHC19VJK60eTLS7iYUGPMP0rKUzyy3iKs3bw0xeNFoIStKSL+pFjp4YpfZtydNWmQ63oHK3w1Djuhe+mDImO4QzmzX5QIf7P3XjjRvowqxUMChSA8ktMyY3eFohp3z+NUxeN038mDwoSDrA+KyoMS0EwPipjiUZlk+THx/FOJlG6c3Eig9AyWAuB7aACmiJRW8XApHvE75GUEQUlR+jWZyOTTMxVeUjXCV7JfeRi1Fe4NzQ5WQsGA4flhJ/S2bphkR9VVoL46943aBlZZX6NM0u02bEytFgUKeVAIwhP5j6DwKR4/qnjM20nOOyJi9kFRlxWL+8iXnbLH7B4UeS8EIO3yN+f5pIwrxVy35T7YYdUvTVzLeMAuHnn4XiViJQ8b7uYoUASTLFv2kuOHduvEs6ct5r7QQU6x0HnVS5kxAPzp8uOM28cM7ZOjvbNyyhH9LVVXsjQtE1yxhLyKJ9+N2kigEAVPLk2yXkhaIh7yZSxVPG49S3gPSkrdqM3oJCtJ8dhe3zDbarZma259UMqECh1+fZnBksgeZopk1TAMM4KiNskCQFQwynrp9ClWAbHvwbjB6l4fPAdXQ3t/KOE8G8mUmQJ1EygnH9HfuH2gTvrhUACH15Yb98W/c8BsBiimHtn3i1I8BOFC/iMo5m21SVYtYpwMrmare4dOsrpuGG3ZiUwlggKa1bsC2D0otnbXQvqmiPOwsAoSEig9gxkUm20RFHWKpygYMD4r0SibTR+UeFJHIpkyBGu+Tzq9Gb7qhe+Hkk2K50AO3uRn/sgiKBFJBCWWSGHlpn0AqIqHIFzJbat7vzwo3PIOjdnE+0xYiEO8AHPAHG+SZScsVet7PoKSFFI8LI0gXq2LuWk28XRnSxfWbGsCYBcxRHaYKR5rmiSZUqd4ANOYKRplvVTx8BGYrkSqYCozejN8qXFHLC1QioKaYWh24qLjGlAWDuLSE4flchctjBmUjpb1Kw9Lm+wxgcsLlD++tsm4XZRnDwrVDhKHNtl6UJTTjC33lOuLyzr2QZE0agsFNEQl2+TLlc2S1vSPjluKh5kpGUzAfOfJd4zH3Ga3EM6wCEpXPIWueNI40Tl1kgWASFEQ7bGkrReKF5NsRKg6STj4XWR4GaR5qMGneFjazYs4AYAff2ksbj33aM/L+8HsKUPRtzyMkw7rL60UkwmUl9fvMp+nYYEE4UxOZ/F4WMZLHxT+GdHA6qkPinSarZlqSQoRFHvaKP0vn+IxPSjOKR4xgsLSDZ/u7zQeI5NszyiPhKBpaXHa2pUwBYpDFQ9gniDiCo+AU6M2TdNQXBQwRJFRmdHNk86kA2TuLGTYfKTOeNJTmo1H07xFWvykJBzElyYOVj5vCBTu+8V7VfIdbaPLIqLgyWmKx8NVopfXt/pU3EyyMg+KfZvsQjfFNWozW6LL00Ya10lWleIRBUqpwiTLQx6UnpE+OdnTNYZJVhHVKAqlH7cJFI9VFrxnwqliSLrPwv37Zx/jab2DGeY1aYsmPPWiKXTY94evEiuN8AKll3lQli1bhnPOOQf19fXQNA3PPPOM5flLL70UmqZZ/j/zzDMty+zbtw+zZ89GZWUlqqurccUVV6Ctra1Hb4Q4eMmtB8UdyyRhD43axOhGQgipyCp+vM7iCSkiKOyuxYOSWUg0R4pX62KPE9kJjDwoPccc/mcKlLhDq3vA/Mws86DgLcXDv2ZXPNWj5luPXTkFtZWHbg8URjXzEnXEjc8x371CeoIsxVPG/a33ughKe3s7xo8fj/vuu0+5zJlnnokdO3YY///1r3+1PD979mysW7cOixcvxqJFi7Bs2TJcddVV2e89QfQQ3/qgwB4VYYgpHplJ1nEWj26aZEOGSVbtQRFn8YjpITFCUieceGQnS4qg9JzikCkWgLT5WrdFxqyoUjzs5Oh29c53CmXfh+5cFZ8wsl/W6xyMMC9RU0fs4IigcCke9pvCR1Rlk7QPJFl7UGbOnImZM2c6LhOJRFBXVyd97oMPPsDzzz+PN998E5MnTwYA/PrXv8ZZZ52Fn/3sZ6ivr892lwgip/BVPN5a3YtN1dQpHr4DrAjfB4VFQ4JBeQSF96CEuPUAe68NXqB8dcoQzJpkzVE7dTUluo+tLwkXWZOl1QDzClbVp8Lt5BjhUzxZ9rY4ur4S/3xnu6dlDxWqS9P9bJo64ogms/OgFCKRYPr7oevpC5qioAb+66H6Xh4ocnJkX375ZdTW1uLII4/EN77xDezdu9d4bvny5aiurjbECQBMnz4dgUAAK1eulG4vGo2ipaXF8j9x6JDTacYelvFQZezYSVacYMvOS/z7ck7xmKKG9Sixd5a1e1ASQopHFkH5yuQG248QS/Gw1E+/8jAJFB8QpxMnuLSNOsWT8aCInWQ9p3jMzqdxF0OuyGUnDsd3ZxyBZ+ed5Gn5QwEjgtIZz7qKpxDhvz/sO8Wne8Z7bOqXK3yv4jnzzDNx/vnnY/jw4di4cSN+8IMfYObMmVi+fDmCwSAaGxtRW1tr3YlQCDU1NWhsbJRuc+HChbjtttv83lWC8GSSTXqJoPBVPC6dZHXjcfMxuUnWLBdmF9CqPii8v0AcFpgUfA6qkyGDPc+6ZH7vjCMdlye8ERFG2/MCRSU0lB4Uj34S3iSbMK74vV0Vh0MBzDvtcE/LHiqwjsBvbdmPxe/vBJD/UtyeIAqUsggQy3zX5n/+iLyXmvsuUC688ELj9tixYzFu3DiMHDkSL7/8Mk4//fRubXPBggWYP3++cb+lpQUNDQ093leid5BDj6ynCIqlS6xyGfltAMpBf7xwcWp1n9TNMmPmVRCjSny5KrtANkyyKWspMx8xkf3+iH6I3hzCLiSMpmsZ/0g0aZplVaLR9AhYG7UZ/VNc/CTFliqe7CIohJ3+FREAwN52s+Feb/77CAbSnrVkSjdEr9fo3IEg53swYsQI9OvXDxs2bAAA1NXVYdeuXZZlEokE9u3bp/StRCIRVFZWWv4nDh1yOorHg0JJZrkDYsREFCi6RKDIzk/G0L6kWcUTVnhQEsYVtSaJoKg9KLJOusP6WkfBh4O9N4RdSIgpHr4viepK1YigJOTfKbdoGJui3NwZN+erFMCJp7cyvqHKUuUC9G6TLGAfGNiTai+/yfkefPrpp9i7dy8GDhwIAJg6dSqampqwevVqY5kXX3wRqVQKU6ZMyfXuEL2Q3DZq89Dq3sEAay6jjrLYIyhsW+ZjMg8Ky21HEymuAkPuQTGujrn5LQkhgsIav/Ft9WXnxQkN1rxzIVxJHQzYTLIerlSZSBVNsuJnqmJgVXoWy69e+NjYhniCJbwTCQUtg/+A3v/3wfZ/2/4OAL08gtLW1oY1a9ZgzZo1AIBNmzZhzZo12Lp1K9ra2vC9730PK1aswObNm7FkyRJ88YtfxGGHHYYzzjgDADB69GiceeaZuPLKK/HGG2/gtddew7x583DhhRdSBQ8hJc/DjK1VPIplLH1QBEEiRmDY3ZSLSZZdmUUTSdvMFvGYsF4raRe+tdW9eLXNd60dWGXvbTGyf7nlfiH8UB0MFAselDgX9VJRpCgz9hpBGVSd/nzbY2aKSGzMR2TH6aMHWO73ZpMsAAypKQUA/Gdt2gPKhGykACIoWX9TV61ahVNPPdW4z7whc+bMwf333493330XjzzyCJqamlBfX48ZM2bgjjvuQCQSMdZ59NFHMW/ePJx++ukIBAKYNWsW7r33Xh/eDnEwktMMT9azeOTL8A+LERO1B8V5P9go9GjcNMmyE5Yoevh29kGu+gfgr7bNF1l8/cnoiCXRtzwCkdpK62O92QRYSBRzETH+X6dQuqoPChOkbmWgA4QeN+FggARnDznxsL6W+739eJ5x9AC891kz9nWkJ20bwtmjmTqXZC1Qpk2b5lj2+Z///Md1GzU1NXjssceyfWniECXvnWQ99CqSNV8z15d7UPjH3VI8LBrCoir2FI9ZqRMSIihGF1ruSv3wARXK9yJ2DO3tP8CFgi3F4yHXr6riYd9JtwjKhCHVlvulkd59tV8IsLQZo7d7UFhvl7ggnAvBe9a7jyxB9JBsIyiqeA7/qK3viaLMWHcxyfInNHGonK5b1+fb2QdsHhRvV9uMikjIMr2YIij+YE/xpD8fpxMcu4qNJcQUj7fPtLaiGEu+c4pxv4zSO77wyOXHGbd7u0AJC80ADTN1nufwACRQiF5Bvk2yHlI83ONixCQhXP2yiAa/mOxEw0dQ2Dp8BQb/mgnOo2JGUKyRmqDHngaaplna31MExR8itiqebCIocpOsl8F/Mp8R0TPKuYF6ZZHeLfrEgZTxAuqQm/89IAgXcjvN2H0ZMQIix1zG5kER5uiwp936oPAmWaOKhxMy/Poxzs9gGxYo8aC4MYIzyhbCD9XBgJjiiXnI9as8KEmPVTyAdY6SOLiS6B58hLGyuHcLFJbKETvJFsLfff73gCBcyLdJlj83eKnisZlihatdsVGbSjfwfTPYNvmr7ZQlgsIETMCIlCSECIqXq21GVWZqK2BOcCV6hjEsUDwRdMODkvBYxQNYxW9zZzyLPSZU8JU7VaW9++9DLGWPFtAQxPzvAUG4kMtZPF5ssrqHFE/KQaCI/hFxWzKDLCCWGbMUjzyCwncWNYSQbRaP9z/3cyekS/5nHTMYfcrCntcj1KjLjB2qeEJWfwAjmWVUjM1VOtLBHE14hz95Vxb3coESskbpOmIJAIWRusr/HhCEC/mPoPB9UBQmWU4siCXAxqA/ZQRFIVCMlIC9URu/vq7rxmTcENcHxRZBySLFc+qRtXjxO6dgqNBVlug+7KQmChSnUHpYWIeRTQQFAP5z3cn4yfMf4uLjh2a304QUfnhmZS+PMEaETrLt0fR3rRAM1fnfA4JwId9lxp6mGXO3VX1QQkYXWOt2VSLJNMmaEZSwJMWTTOnGMQoHA9yQwe5V8TBGCA3biJ7BTmpsCi5rX+8UQWGt6tu6EpbHvVbxMBpqSvGbrx6T3Q4TSiKcB6Wkl0/6NiMoOpIpHZ0ZMVxaAB2HKcVDHNL41qjNIcUjRjCMCErKOYLCjHjRRMqIysgiKHxZc7rVvbWZG7MvZBNBIfzHMMkawwLdPSgsfdDSZfWOJGjwX14p5jwoxb1doHARlE4uUkcpHoLwQG49KO44zdkxH3eo4hFMqmxzpgdFvk0WQemKJ5GUTK/VM7YEvsIjFDA9KGaZcfr5AAmUvGLzoHgY3sfSBy2dYgQl+8oswj/CoQDOm1CPlq4ERvbv3WlQvg9KRzT9PQtohWGSJYFCFDy5tchyJ3xdl5b7ioJDilMERYh+iNOMVREUwyDpFkHhKjyKggFje+xx82qbTmb5pJjzFAHeZvGwElYxgtKdyizCX+65cGK+d8EXwlwfFDazqSwcUk7YPpDkXyIRhBsHqA+Kl/SNKprj5EExmqwFRA+KbtsHniDXNyUhSQew9dmJLqCl1xFTSXS1XRjwETGAM8k6pXiMCIqQ4nFJDxKEV1gflHgihfZMBKVQRiKQQCEKngNVZKx6HS8RFIsHRRAxYqM0c5px+l9V6oWPeLAy02BAMwQNWz8umHDNKh5xmjH9uecTMcXTyk4GDtUSzIPSGk1YpmR3pzKLIGQY4xSSKXTECqeCByCBQvQCculB4cOYqo6xXkyysp4kxnNCm3q+PBhQXwXzEQ9WAhgKasbyZoonkyrILO9HJ1nCf0rC1lb3+9tjAIC+5eo+M6yKR9eBtpjpQ+luZRZBiPDNAFkEpaQAKngAEihEL+BAWWS9iA/lug7LG/4RI4LCPCjp51XnGD7iwbo7BgOasbyZ4lFEUJLd74NC+A/ryBtLptAWTWBfRqD0KVULlOKioGFWbO5Ip3naownsbIkCIA8K0XP4PjyfNXUCAPqWR/K1OxZIoBBEBnUExbztpVGbOM1YNDTaPSjuERTWOyOoacbybDvsapqZLY1hgUYZMl1tFwJlkZAxZG5XS5chUGpcOvUaPpSMUfbmf6wzniPRSfQU3gO1dV8HAKC+QAZMkkAhCp58Dwu0dJJVGWm52ymVQMlERJjISRpGR/k2+ZNPNCmJoNha2ctTPGSSLRxqK9JXpjtbot4FCqvkyZQa/99bnxrPZTO+gCBk8JWBn+xuBwDUkUAhCG+oohZ+wJcZe/GgKOEWUUVQioQIitssngBniI1mjJW8B0UXIjFMAPnR6p7IDf0zAmVXKx9BcW6VLkZQeOgzJXpKMKChXyal88IHOwEA9VUl+dwlAxIoRMFzoCIoXjwo6giK+YQyghK0mmTd+qAA5gmImWQDmma0shc7ybKLaaPMmEyyBUe/jEDZ2xYzek5UuAybM7rJSiYR02dK+AFLAzMGVlMEhSA8caDKjJURFO5vV9lJ1imCogsRDCHy4RSlZycgZpINBQKGqDJa2QsppIAQQaG26IUDm9vS0hU3Pje3igkzgpKwPUcRFMIPmjqs4ncgpXgIwiMHKoKiWCZpiaCoTLLy5QFT4BTZIijpx50jKOY8HgAIBkwBogsChYmZEOdB0XXdmP1SXER/7vmGfQYsvQMApS6zXJgHpZkiKESOuPuCcZb7dZTiIQhv5NSDwokDPSVfxksfFj76IjZ2Y+FTsYqHbdfpFGNGUJKZ+wGuD4r19VjqxxgWmNIRTaQM8VRcIL0NDmXYkLl/vdcIIF1BEXLoJAvAqPxhc1J4KCpG+MGXJw223C8vgEGBAAkU4hDHS4rHUsWj2A7/OItcAGkRwlYXW9AnBG+KDNGDEpL0QREjKGxzyZRudC0Fev9Y+IMB1k12T1u6j0ksqVDFHCwF1MF9lowg9UEhfKAQ5u7IIIFCEBlU4sMSEFGZZIXHxegGwA8LTN/3MsRP9KCkK3uYCdb6GqZASb9OImWOTw8FNEs5IZEfupNmK80IlM6YRKAU6ImF6H1874wjoWnAH742Od+7YlAYcRyCyBfc77unVveeYihp0RAMaBbDLOvYqBsRFPcGaqJ4CQXUVTyiByWVMk9qFD0pDNjAwGwoycxF6Yil5/EEtLQAvvqUEQXTkpzo/Xxz2khcPGUoqkqdq8oOJHRJRRAZVFaTbIcF8uvw64YNk6x1GacUjxjC5xu16eJ2hEZtfASF/CeFQbciKBlx2RFLorUrYXx/5n/+CD93jTjE0TStoMQJQAKFONTh29h78aB46CQL8G3mzWdMk6zgQXGMoASE+2aKRywzZtU9LMLCe1AoglIYRLrxOZRFzBTP/o509U9pONitaAxB9CZIoBBEBrUHxYNJVoygJO0RFMODwpbx0EBNfC4SChp9UwyTrNBnhUVdkrqOzlg6jVRKEZSCoLgbAsVM8ZgCxWnAIEEcLJBAIQ5pLDN0PERQVKgmGCcyVRqaZgoIPasIivW54qIA1+qeiSCrl4Wtk0jqZoqHIigFQXHI+pP7xFXHu67DxGVHLGE01KousFA8QeQCEijEIQ2f1lG3uofrMuLDzADLREhRIGCrvkkk3U2y4nPFRUFbHxTbsEAuxdNJKZ6Cgk/xHDesBlNG9HVdp4TzoFAEhTiUIIFCFDyaYysz//DWB0XVSVbePZZP47CKULaN7kRQIkUB2zTjlJjiCZgpni5WxUMpnoKAj6B4NS6zCMqu1ijm/+0dAEBlCRVgEgc/JFCIQxrdQ3TEi0lWJJ60RlDSDdbkHWAdq3i8RFCYSTbzODPjUgSl8OCFYonHih42i4enLEwChTj4IYFCHNJYerApUzzZlxmz1vSGPyRo7wDbnSqe4lDQ5kFJGUIn/Th7njwohUcpJyy8isaa0jCKhHLzsgJpRU4QuYQEClHw5HIWj5fXSVhSPN7WZdUzcW6SMEtVGf1LsvSgBDSgKKjZphnbIijcLB6jUVuY/tQLAX7Gide0WyCgobbCOl2WqrKIQ4Gsf7WWLVuGc845B/X19dA0Dc8884zxXDwexw033ICxY8eirKwM9fX1+NrXvobt27dbtjFs2DBommb5/6677urxmyEOfrwM7stue+ZtVbFOyoNLVny4I5Ye7MY3UTM8KNlEULgr5+KiIDTNIVUk86BQiqegKC/mIx/evVVimociKMShQNYCpb29HePHj8d9991ne66jowNvvfUWfvSjH+Gtt97CU089hfXr1+Pcc8+1LXv77bdjx44dxv/XXntt994BcUjhsz6xRD6Ujdq6keJhg934NvQq70jQYSItH0FhaRpbHxRhO4ZAIQ9KwVHKfQ4plSKWMKjaGkGhlB1xKJC1DJ85cyZmzpwpfa6qqgqLFy+2PPab3/wGxx13HLZu3YohQ4YYj1dUVKCuri7blycOcXKZ7FGdL7xMMxZ9Kqx6hnlQQlxqRhQWor+Ah/egsAqQoOBBMYUOMv8yD0oKb2zal16XUgIFQYATnF6EL+POL43FCx8sycUuEUTBkvPEdHNzMzRNQ3V1teXxu+66C3379sXEiRPx05/+FIlEQrmNaDSKlpYWy//EoUkuUzyqbac8VPGID3dkBAo/sdg0t8LynNdhgeyqWeynkhIiKGydlq4EPmxsBUARlEIkmwjKgMpinHJEf+O+338HBFGI5FSgdHV14YYbbsBFF12EyspK4/FvfetbePzxx/HSSy/h6quvxo9//GN8//vfV25n4cKFqKqqMv5vaGjI5W4TBQbfByWL3/SsUW2av9JVnhiEhzuFFE8oYPYvYWklI7riZJLloitsGrJbNZBM8JBAKTyyiaAAwO+/Nsm4TfqEOBTImdMqHo/jf/7nf6DrOu6//37Lc/Pnzzdujxs3DuFwGFdffTUWLlyISCRi29aCBQss67S0tJBIOUTJZUWPssw45WFdWxWP3YNi6yTrwYMS4XqksKoPlUmWCROZP4EatRUeXkYo8PDDAUsj9HkSBz85iaAwcbJlyxYsXrzYEj2RMWXKFCQSCWzevFn6fCQSQWVlpeV/4tDEd5OsZRCgu0nW67BAFkFx8qAkhP4lMnhhwZpzmQLFOiyQCZR+5WFEQlRWXKiwVM0lxw/Net0fnjUanzu8H2YdM9jv3SKIgsP3CAoTJx9//DFeeukl9O3rPmtizZo1CAQCqK2t9Xt3CMIzMvGj67qnTrJuHhS+ioct68WDwve7YLdVZlu2HU3TMLhPCTbubrftD5F/HpwzGbvbohhYVZL1uleePAJXnjwiB3tFEIVH1gKlra0NGzZsMO5v2rQJa9asQU1NDQYOHIgLLrgAb731FhYtWoRkMonGxkYAQE1NDcLhMJYvX46VK1fi1FNPRUVFBZYvX47rr78eF198Mfr06ePfOyMOSvwvM3bettcovLhulxFBMYcFGh4UYQpxkYNAKeE6j7LeF0yIqIYFAsCgPqUWgVItaZdO5IdQMNAtcUIQhxpZC5RVq1bh1FNPNe4zb8icOXNw66234p///CcAYMKECZb1XnrpJUybNg2RSASPP/44br31VkSjUQwfPhzXX3+9xWNCECr89qBYqngk2xZ9Al47yUYTmU6yMg9KFn1QyiQRFFure2FYIACcdFhfLPtot3F/xtFU0k8QRO8ia4Eybdo0xxI3t/K3Y445BitWrMj2ZQkCQG6rF+QRFEGgqEqRhYdjGYFi8aAI20x68KDwKR4WQbF7WdKvwYQLAFwwqQE//teHAIBvnXaYYxqJIAiiECEnHVH4cOdWL4P7ssEtIuO50kLYrxibZizzoNgiKN5SPGIEhVUDZV7KEkHhUzpdCQ9lSARBEAUGCRSiV+F3AMXaqM3+vNdeFeJSsYTVgxIKBGwt6hNJ9z4opdIqHut2WJSG71LK3+4kgyxBEL0QEihE4eM+qy9niN0+lVU8ihQP30RNnGbsLYJi730hlhnz3WplsJJngiCI3gQJFKJ3kcsqHk8mWfkOiN4UM8WT/jfoMIsnFFT/GfLD5YwIijEMMP04EyClQjO2UXUVAIBzxtcrt08QBFGo0MxuoleR0yoeH1I8oYCGREqXRlBED0rcJfIBAPXVZjnqccNrAMBowhbNpJHaY0ygWP+c//6NE7B5TzuOrqfGhgRB9D5IoBC9ipxW8UgeE9vcu6V4IqEAErEkYhnxwSIpRcGAPTXjYRZPQ00pnv7mCaitLDbECmt5zkqZO6LpQZtlQvvz8kgIYwZVKbdNEARRyJBAIXoV/usT50GAYgRF3QclTaQoiPZY0oigdMbM9AtL8ZgRlPQyYZe29BOHWBsYRooyEZR4en1VBIUgCKI3Qx4Uolfhe5mxpVGbnWTSq0k2/QRLv7AqHtZiviQctE0zNlM82f0ZFhsRFPYa8ggKQRBEb4YEClHw6JYoRw5fpyceFC7FA5ipHSZQysIheydZI/2TXRM1FkHpYhGUKEVQCII4+CCBQhQ8bu3o/dq2LIbiuYoHLIKSjmKYKZ50dKM0HLR5UFgEpcihikeGaJI1IigkUAiCOIgggUIUPC4awr/X8dTq3nldFt1gAoVP8Zit7tP/Mg+KU6t7GbxJNpXSjTLjkjCleAiCOHgggUIUPLx51fdOsnDettdW94ZJNmQVKHyPEmY10Y0Iilnhkw3FzCSbSKErkTTEEXlQCII4mCCBQhQ8lmZqfjdqc+uD4lWgGB6UTIonyfwh6fRLSVEI4aA1/cN6pGTtQcm8Rlc8aWljz8yzBEEQBwMkUIiCJ5ceFOvr2LftdZqx6UFJ/0nFkzp0XTdSPKXhoBH56MpEVeKJ7kVQTA9KyvCxBAOaZf4OQRBEb4cECtGr8Dpc2Cu64jbDZpJ18aAUc63po4mUJcUjNliLc4MEs8Hsg5I0fSwkTgiCOMgg2z9R0IgRC1UEo/vbl99meO27wvaLn4fTFk1YSoBZKodFULpbZlzMCR0zTUTXGgRBHFzQrxpR0Ij6ILet7mVlxuIyinUzT4SCmiFS2roSaO6MAQD6lBUZ0RXWv6TbZcZcqijRzUoggiCIQocEClHQ+N05VsQiSiQvlRCG8ShTPMYtDeWRdGByR3OXIUL6lIYt/Ut0XfelzLi73WgJgiAKHfpVIwoaUQ/4rldcWt0nkl5TPOl/NQ2GQNm2vwMAM8gGEclEUFJ6OnpizOLJMoLCIjQdsYQhoLJNExEEQRQ6JFCIgsaW4slpFY/9sbiQ43HrJKsBKMsIlE/3pQVKn9IwALN/CZAeIpgy0kLZ/Rmy7XdEk2YEhQQKQRAHGSRQiIJGFAS+90FxeC3A9Im4vb4sgnLvixsApP0nQDpSwiYat2Xa0wPZi4uyTASlPZYwjbaU4iEI4iCDftWIgkYUBP5PM3YeRGjzoKi2k/lXg4a2aEK6jKZphg+lrctcJusUT0YApXQYr0URFIIgDjZIoBC9ilxaZmXbFlM86pUzKR4NeO+zZstTN5452rjNKnlau+LGY9n2MCnheq20ZLZDJlmCIA426FeNKGhyXWZsbaPvnuJR7QDzk2gA7jhvjOW5kbVlxm0WQWnNRD40Ld0FNhuCAc3wszR3pAUKmWQJgjjYIIFCFDR2X0gOTbKSx+wmWdW6LIKi4bwJ9ZbnmEkWMCMoLMVTFAhA07IXF2XhdJqnuZOleOhPmSCIgwv6VSMKmpxHUFx63Ytlxl5Msny7e8B6n3WBbe5MRz7Coe79CbKBg7984SMA1OqeIIiDDxIoREGT6/iJexWPNw8Kb5LlO8OKwRHWBXZXaxQAUFVS5HlfeVoFIy61uicI4mCDftWIgsY+iyeXr2V/zFZmrOqDwkVQeErEaEomgrKzuQtA9wVKTVnYcp+qeAiCONgggUIUNKIcOOBlxqIHRdnq3mzUxiOme1gEZWdrWqBUl3ZPoPzqwgmW+1TFQxDEwQb9qhEFzQGt4pE8773MOP2PLaUjeEzYHJ2dLT1L8ZSGrYPIqYqHIIiDDRIoRGFzQFvdSzwoKTHFo1g3869YkWNL8bAISkvPIijidqmKhyCIgw36VSMKGjGlc6CHBcYT3lI8qZQ8xRNRVPTsa48BACq7GUHh5/rwr08QBHGwQAKFKGhyfdrlIzLyVvceTbKZf8UIiigkxJTP8L5l6A6it4UJHoIgiIOFrAXKsmXLcM4556C+vh6apuGZZ56xPK/rOm6++WYMHDgQJSUlmD59Oj7++GPLMvv27cPs2bNRWVmJ6upqXHHFFWhra+vRGyEOTg5kFY9MDnkuMxY8KF+dMgQA8N0ZR1qWE4XF5GE1We6jfDufNnV0azsEQRCFStYCpb29HePHj8d9990nff7uu+/GvffeiwceeAArV65EWVkZzjjjDHR1dRnLzJ49G+vWrcPixYuxaNEiLFu2DFdddVX33wVx0GLvg+J3FY/8NsMmUDxW8dx53hisvmk6Tjysn2U5PqIS0IDh/bobQbH+6W7b19mt7RAEQRQqIfdFrMycORMzZ86UPqfrOu655x7cdNNN+OIXvwgA+NOf/oQBAwbgmWeewYUXXogPPvgAzz//PN58801MnjwZAPDrX/8aZ511Fn72s5+hvr5eum3i0MQ+zTh325dt2tZJ1mU7LIKiaRr6lkdsy7EqHgCoKYtkPYeHURyyRlC+OIH+bgiCOLjw1YOyadMmNDY2Yvr06cZjVVVVmDJlCpYvXw4AWL58Oaqrqw1xAgDTp09HIBDAypUrpduNRqNoaWmx/E8cGogRE1mlTS6JZSIoYY9VMprNJmuFj3z0Kw87LOlMgBM2k4b2we1fHOOwNEEQRO/DV4HS2NgIABgwYIDl8QEDBhjPNTY2ora21vJ8KBRCTU2NsYzIwoULUVVVZfzf0NDg524ThYytzNjvzbs1aks/yGbmqAQSe9xt7h/vHeknibB0h/OPGdTtfioEQRCFSq+o4lmwYAGam5uN/7dt25bvXSIOEDYPSk5TPPaNJ1LpCAprhKYsM2YpHpfX46t4ehJB4SFxQhDEwUjWHhQn6urqAAA7d+7EwIEDjcd37tyJCRMmGMvs2rXLsl4ikcC+ffuM9UUikQgiEX+uNonehV0Q5LJRm/2xWCaCwobxqRu1sQiKW4rHjKBUFPdMWNzxxaOxast+nHm0/O+GIAiiN+NrBGX48OGoq6vDkiVLjMdaWlqwcuVKTJ06FQAwdepUNDU1YfXq1cYyL774IlKpFKZMmeLn7hAHAXYPit/bl99msFk8btOCVcMCRXiTbEk46LCkO5dMHYZfXTiRusgSBHFQknUEpa2tDRs2bDDub9q0CWvWrEFNTQ2GDBmC6667Dv/7v/+Lww8/HMOHD8ePfvQj1NfX47zzzgMAjB49GmeeeSauvPJKPPDAA4jH45g3bx4uvPBCquAhbNhm8eT0tdR9UEwPimLdzL9uJtkIZ5ItDpGwIAiCUJG1QFm1ahVOPfVU4/78+fMBAHPmzMHDDz+M73//+2hvb8dVV12FpqYmnHTSSXj++edRXFxsrPPoo49i3rx5OP300xEIBDBr1izce++9Prwd4mDDNs3Y5zpjt4hMnJlkjRSPyiSb/tfVJMtFUMQ2+ARBEIRJ1gJl2rRpjqWemqbh9ttvx+23365cpqamBo899li2L00cgtg6yfr/Ctxr2Z81TLIhN/urfBaPCF9mLHaDJQiCIEwoxkwUNLYUTw5zPLLoSDwhmGRVKZ7ueFBIoBAEQSghgUIUNHYPygFudZ/K1iTrvVGb2K6eIAiCMKFfSKKgsQmSXFbxOMziMTwoihBKymNoh0/rUIqHIAhCDQkUoqA5oFU8kscSRh8U58gIWzfgEkHhG7V5bZ9PEARxKEK/kERBk/tOsrr0NiMu9EHxOixQBR81cVuWIAjiUIYEClHQiKLBayrF8/YVtxlx2ywe1Xa8VfHwERQSKARBEGpIoBAFjS2CciBfDGYnWdd0jMcICt/11a2pG0EQxKEMCRSioLGXGeewikdWZmybxaNo1Jb5NxvR0b+C5ksRBEGo8HVYIEH4T24btTlV8ei6bpYZh5ynGTPh5CVt88DFk/Dp/g6MGVSV7e4SBEEcMpBAIQqCeDIFDbANvrMJgpw2arOSTOnG67tPM/bOmWNo+jBBEIQblOIh8k4imcIJd72IU3/+smtre/8btalb3Se4uT9uHhS2qFuZMUEQBOENiqAQeeezpk7sbo0CAKKJlKUUN5+t7lmJMeCl1b33FA9BEAThDkVQiLzjJDpE0eDzMGPH/WAGWYBvde9mkiUIgiD8gAQKkXecjari/VxW8VhhJcYBDXBt+upxFg9BEAThDRIoRN7hRYfYiO1AtroXXyyWtA8KdG3URvqEIAjCF0igEHmHP+eLAsVNsPT8tTmTrPBcguuBwiIjag9K+l/SJwRBEP5AAoUoKNw9Jt4VCm9yVW5Nl98GgGTmgQCnOpSN2gwTCkkUgiAIPyCBQuQdq0hwSfF41Cd3Pvc+Rv/oeWzY1ZbFfoivnREoAc1Vd3idxUMQBEF4gwQKUQDwHhTxme51kv1/r2xCIqXjV0s+dn5lB5OsLG2jEkjUB4UgCMJfSKAQeYc/6bt5TvyfZqxu1MbuBjTNmLGj7CTrcVggQRAE4Q0SKETeSTkJFGHZbPVJNnpB3DTbF00zhYf69SnFQxAE4SckUIi8k0w5RDFcBEtP8eZ/cZcdFEEhCILwFxIoRN7hoybuEZTsJIq7udXhOU50sM0oq3jY61EMhSAIwhdIoBB5h4+g2EyyPQyZZJXiURh0A1yKR6VodJmjliAIgug2JFCIvGOJoNgaoeS2URu/eVvFkKE5NNfICM3iIQiC8BcSKETe4QWK6yyeLF0obrNxHKt4JL4StyoeKjMmCILwBxIoRN7hG766eVBS7s1hu409dmNW5phVPHKJwlf8EARBED2HBAqRd6weFH+HBbrpBaeUUcqIoGjmLB631yOBQhAE4QskUIi8Y51mrH5Odr/Hr23Ztvy1Nc2cx6OaFcT7VQiCIIieQwKFyDtJiwdFnGZsXdZvj6x12/L0UlqgaJn9UZUZU4qHIAjCT0igEHnHscxYlCQ+53h0DwbdgKa5elB8ry4iCII4xCGBQuQdp0Ztdn1y4JSAkeKBWQ2kMumaFT8UQiEIgvADEihE3kllUcWT/Sweb/1L0ttWpXg0w4Oi7iRLs3gIgiD8xHeBMmzYMKPqgf9/7ty5AIBp06bZnrvmmmv83g2iF5H0kGYx7vv82tZZPPLnNPAeFOftUB8UgiAIfwj5vcE333wTyWTSuL927Vp8/vOfx5e//GXjsSuvvBK33367cb+0tNTv3SB6ESmnMmNBkqhMqiqy0QuquT98FY+bB4X0CUEQhD/4LlD69+9vuX/XXXdh5MiROOWUU4zHSktLUVdX5/dLE72UpMWDYn3OLaLihmsfFIdtW/qgwCWCQikegiAIX8mpByUWi+Evf/kLLr/8cot58NFHH0W/fv0wZswYLFiwAB0dHY7biUajaGlpsfxPHDw4NmoTlvXdIsunl2xlxvZOssoyY4qgEARB+IrvERSeZ555Bk1NTbj00kuNx7761a9i6NChqK+vx7vvvosbbrgB69evx1NPPaXczsKFC3HbbbflcleJPGL1gbg0ZpMIhNauOIqLgigK9kxv2zbNiQ7mLVFFcMyHSaEQBEH4QU4FyoMPPoiZM2eivr7eeOyqq64ybo8dOxYDBw7E6aefjo0bN2LkyJHS7SxYsADz58837re0tKChoSF3O04cUJz7oDjf39cewzF3LMaIfmV48bvTsn5tXXGb35eApiEQYI+pIijUqI0gCMJPciZQtmzZghdeeMExMgIAU6ZMAQBs2LBBKVAikQgikYjv+0gUBhYPiotCEfXBqxv2AAA+2dMu3babYLBsz8Gg6xZBSWYeD5JCIQiC8IWceVAeeugh1NbW4uyzz3Zcbs2aNQCAgQMH5mpXiAInlUUn2Wxn82QzG0fVc4X3T6kiKOw9BAMkUAiCIPwgJxGUVCqFhx56CHPmzEEoZL7Exo0b8dhjj+Gss85C37598e677+L666/HySefjHHjxuViV4heAC9KRMGRSOrKZdPL9+y1eQGk6rkS8DCLh6WpAiRQCIIgfCEnAuWFF17A1q1bcfnll1seD4fDeOGFF3DPPfegvb0dDQ0NmDVrFm666aZc7AbRS3AqM+6MJy337T4RlwhKFikeVc8VLyZZtiyleAiCIPwhJwJlxowZ0tB7Q0MDli5dmouXJHoxfIpnxSd7cdLh/Yz7XaJAsaV4/NsPZRUPuFb3LgKFAigEQRD+QLN4iLzDV/H85qUNluc6Y0lxcQv+RlCE52BGUDRK8RAEQRxQSKAQecdJZHTGreODs+8sm4VJVjWLR9NcG7UxjUUmWYIgCH8ggULkHWeBInpQnGf1ZIu1D4rckOtlWKARQSEPCkEQhC+QQCHyTjKlfs7uQbE+rxIMXtEdcjzSYYGK7ZAHhSAIwl9IoBB5xzGCInhQ/C4ztmxLcZ+PoKj6rlAfFIIgCH8hgULkHVv3WA63FE9PTbKWbSuawAU0zbCyKE2yOqV4CIIg/IQECpF3ksJJ/7Zn1xm3bQLFoRO+LLrhJhesgwrlz/F9UFKKdBRLU1EEhSAIwh9IoBB5R4ygPPTaZuN2l0uZMS9KeuxHUdzn+6C4DQskgUIQBOEPJFCIvCNGUHhYBKU0HARgj5Lw4iYpUSiufVCcWt1LIijqYYFkkiUIgvATEihE3hHn7fDYBYr1ef6umx9FhmUOkEOre02TL8OgMmOCIAh/IYFC5J2YQ50xq+IpDaenMqgmDou3GW7TjHlRoxI/Wua/9PKK7VAVD0EQhK+QQCHyTtxBoHQJERQxSuIWQcmm1b39OXsfFLdOshRBIQiC8AcSKETeiSfcUzwlqhSPxSTbnRQPH0GRrx/QNGPGjqsHhSIoBEEQvkAChcg7ThEUM8WTESjC8ylJFY9KaMiwCBTFc54iKCzFQxEUgiAIXyCBQuQdJw9KV2ZYYElR2oMihjCsHhRdtogjfF8Tp0GEmtcqHvqLIgiC8AX6OSXyjiqCkkzphnhRR1Dst/koh3ujNj6CIhc/Ac202soiKLquG8tSBIUgCMIfSKAQeSeuKDPmBwWqyoxTEg9KShL5UGEpM1Zs260PCr8NquIhCILwBxIoRN5RRVD4NvfFRWmB8vfVn1qW4ZuzMUGh6lUiw8lYKxsWKFue3wc3QUQQBEF4gwQKkXdiCYVAyRhkS4qCRjSlsaULrV1xY5kEJw6YdsjKg2Jp1CZgdJLVjHJlmUDhH6MICkEQhD+QQCHyjiqC0sWVGHdwM3n4lFBKFkHJQqBYPCi2Rm1m+3qnFA8fQSEPCkEQhD+QQCHyjtqDkhYukVBAOa8nYREo7F8+5eL82taIiNjq3tgKF0Fx3gZV8RAEQfgD/ZwSeUcVQUlkaoCDAQ1JTsTIfCeAGU3JpmFbUpIiEu9bTbKSFA+3+9RJliAIwh9IoBB5RyVQmNAIBTTLMryo4AcNGh6ULF7baZYPS/GkTbLWfeLhozuU4iEIgvAHEihE3lGleJj4CAQ0SzO3BBeykJUZ65zeyWZYoG3OD98HxajisW+DF0zU6p4gCMIfSKAQecepURuQjqDwlT6WCIpErGRXZsy9nk2g2FvdS1M8Ok0yJgiC8BsSKETekQkUXdfN9vGaNcXDG2OTUpOsuZ1sTLL2FI+5DadW9yljP51fiyAIgvAOCRQi78j6oOi6KURCQWuKJ6kQKCy6kY1Jll80mZKneDRNc/agpEwhRRAEQfgDCRQi78g8KEldNycEBwKIJ+TGWFmZsZPxVYQXHGKKx2h1D76TrGQbGe1EKR6CIAj/IIFC5B3eR8JI6bohPoKadeIxLyoSkvJjpwGAstcxbjtEUJw6yTJhQxU8BEEQ/kEChcg7CUlYIpUyBUMoELCkgRKuJlkT9wgKf1sQKJl/+QiKbHOGB4UiKARBEL5BAoXIK6mUrjSeMiESCEDwoHBiRdIHxasHRazIEb26Omd+1ZyqeFLmcgRBEIQ/kEAh8oosegKkRYbZqC1greJRelCYSdbcjkxQmMvbX5PHapJ16INCZcYEQRC+QwKFyCu8KJhx1ADz8ZS1UVtc0QclLvGm8KJEoX9sry1uF7B2knX0oFAVD0EQhO/4LlBuvfXWjKnQ/H/UqFHG811dXZg7dy769u2L8vJyzJo1Czt37vR7N4heAh8B+eVXJhi3U1wflFBAw/D+ZdJ1ZH1QLFU8DiZZUWyoIigQphmLURl2lyIoBEEQ/pGTCMrRRx+NHTt2GP+/+uqrxnPXX389nn32WTz55JNYunQptm/fjvPPPz8Xu0H0AvghgEVB8+uY1HVLZOK3X51kPPfQa5uM21YPCouggHtM/dricyqTbECzNswX16MICkEQhP+EcrLRUAh1dXW2x5ubm/Hggw/isccew2mnnQYAeOihhzB69GisWLECxx9/fC52hyhg+CqcUCDdEC2lW02yoYCGIX1LoWlpcfDS+t3GOnFLFQ/7ly8zVuOW4pH1QZFt0+h4SwlTgiAI38jJT+rHH3+M+vp6jBgxArNnz8bWrVsBAKtXr0Y8Hsf06dONZUeNGoUhQ4Zg+fLlyu1Fo1G0tLRY/icODtjJXdPSXhPDjMqVGQeD9jbzTEwkpSZZdft6HptJ1lbFA3PfOIGiEjYhUigEQRC+4fsv6pQpU/Dwww/j+eefx/33349Nmzbhc5/7HFpbW9HY2IhwOIzq6mrLOgMGDEBjY6NymwsXLkRVVZXxf0NDg9+7TeQJfiAgwHds5Ru12VMnnfEkAGsXWlkfFKcYipsHhRHQNGgB9XIszRQiDwpBEIRv+J7imTlzpnF73LhxmDJlCoYOHYq//e1vKCkp6dY2FyxYgPnz5xv3W1paSKQcJLCTOzOYBgIAkpkyY5t4MaMeHdEEyiMhJLgqHqYbdI8RFF2ImNha3ae46A6f4lF4UMgkSxAE4R85j0lXV1fjiCOOwIYNG1BXV4dYLIampibLMjt37pR6VhiRSASVlZWW/4mDAzE9wqd4zEZt9h4k7bGkZf3089mZZG0RFFuZMcNqkhXXYz6YUJAECkEQhF/kXKC0tbVh48aNGDhwICZNmoSioiIsWbLEeH79+vXYunUrpk6dmutdIQoQw2CaObcHuRSP2ajNfuJvjyYAqEyy5nLZlBmLERSVB8UWQUlaRRZBEATRc3xP8Xz3u9/FOeecg6FDh2L79u245ZZbEAwGcdFFF6GqqgpXXHEF5s+fj5qaGlRWVuLaa6/F1KlTqYLnEMWIoGRKjJkOSOq6pVGbSEcmgpKQeFAsAwB7YpKFvdW9uH3ArEQiDwpBEIR/+C5QPv30U1x00UXYu3cv+vfvj5NOOgkrVqxA//79AQC//OUvEQgEMGvWLESjUZxxxhn47W9/6/duEL0E0YPC/k2ldPzyhY8AKCIosUwEpUd9UJxNskzAaNCEKh7hPZAHhSAIwnd8FyiPP/644/PFxcW47777cN999/n90kQvRFXF89HONmMZWQO0tZ82o76qBHvaosZjLAJi7YPifRaP2AeFqZt0iod/WF5mzDeaIwiCIHpGThq1EYRXxEF7WkaM8A3cYuKYYQA/X/wRHnp9s+UxaZlwNrN4FJ1kNW6/0utZtxNPUgSFIAjCb+iSj8gryYwQMVM89mW6Mj1PRPa1xyz3e9pJVtX6Pj3NmF9OjKCk30MRVfEQBEH4BgkUIqfouo5b/rEW92T8JCK2PiiZSEWMm14cTdgjKKrXAoQqHgcTiqqfifi8plkjKGKkhTwoBEEQ/kMChcgpG3a14ZHlW3DPCx9Ln1d5UHhREvMoUJhwSHLpoZ7M4jFTPOl9YhESvnKIv09lxgRBEP5Bv6hETmnujBu3xUZoAO9ByTRqy3wj+bQOu33ehHrH12Kbt044dl/evC+PoLDACDPBxgVPjDHUkFI8BEEQvkEChcgpfCRETI0AsEwsBsxGbVFJiucnF4zDk9dMxaxjBktfi6Vzkt30oNgFilnFAwDhkEKgJK0+GoIgCKLnkEAhcko0YUZCbGW8MLuwBsQUDxdBYQIlEgri2GE1qC4tkr6WbMKxswdFTPEIz2f+Zf4TFkGJJeQeFGrURhAE4R8kUIic0hXnIigSgSKe3JlQsURQhCqe8oi8Ot5I8aT8SvFkIiiZ+2EmUAQlI3bDJQiCIHoO/aISvrGrtQtb9rZbHrNEUCRqQZwEzIIQvAdFNMlWFMsFCjPH8l4Xp0ZtomBSV/Gkd8otxUMRFIIgCP8ggUL4xnF3LsEpP33Z0p/EEkFJSgSKMBBQVsVz9wXjLOvwAuXa0w7DjKMGADAjJ94jKB5b3Rsm2fSNeEJukiUPCkEQhH+QQCF8gY948FEU/nF5BMVqMI0UBQEALV3p6p+pI/pi8rAayzpVJWHjdm1FxPCGsOqdpEeBYmvMZisztqZ4ilxSPNTqniAIwj/oF5XwBT5qwlIhgDl1GJCXGYuN2morIgCAT/d3AgBKw0HbOkfWVRi3q0vDxroJmUnWcRaPS6t7WwSFmWStAoVa3RMEQfgPCRTCF3iBwp/AOzJThwFr6oUhNmobUGkVKLKoxNCaUuN2aTho9B9h0RivERQxhZMSq3gyKwdsHhRFq3sSKARBEL5BAoXwBX6qMO87caviEYcF1lUWAzAFT1HI/hUNBDQsPH8sLpg0GKcc0d8QN/IIihoWQSnKdIezVfFk/hWreESTbNzwoNCfE0EQhF/QNGPCF/gICl+5w5/MZdOGxSqe2opiy/OqAXwXHTcEFx03JLNuWhgks/agmB1gY0l1igdGH5TMnCDRg5I0t0MQBEH4A13yEb7Q1GG2tOcrcHiBIkvxxLgmbABsTdjCHoynYgTF+jpOHhTr+rpubd7G/Cssc6MsM6ZGbQRBEL5DAoXwhbao6TWx9jAxT/gyk6zZJTb9VawotgoUL5Uxpkk240HRvUVQUpLqGz76YnhUjGGBcpNsQqhEIgiCIHoOCRTCF3iBooqgyMqMWZdYU6BYs45eBIrNg+KSVjKfy6zPpWZ4DSVW8ag8KBRBIQiC8B8SKIQvtHZxAoWLoCS40piEpFGbEUHJ9D+pFCMoIfeTPmsxz7wgCY8mWcODwplbrYJG3gdFrOJhERWZoZcgCILoHvSLSviCKoJiSfHIIihCiqeyxBpB6VsWtq0jIkZQUl5TPJIICp/iMXq0BK1lxmKKpzXTVE4UVwRBEET3IYFC+EJbl7tJVlZmzCp+mEARBwEOqCy2rSPCvB9JiUnWS5kx7x3h01DsfRRnDLyqTrItnWlxVllCAoUgCMIvSKAQvqAyyboKlLi1ikecCCyWHcsICSZZy7BAhxAKi4SUhU1RxKehDPFUlN4nVQSlxYigUNU+QRCEX5BAIXzB4kHJKoLCPCjmV5E3yvYtd0/xsBRMQuJBcaIrI0BKw0FDfPCdb6NCCTSL8vACDABaOjMChSIoBEEQvkEChfCFdu7E3snN34lxEQlpFY+Q4gGA1248DeWREAZVl2BY3zLX12adYJOyTrIOWoV1uS0uCqIsM/OHnx1kRncCxnLpfeaqhFI6WjPRI/KgEARB+AfFpAlfiHIt7XmxEk94jKCEzKGAlcVFeOtHn0dK1y2DB1V0d1ggi4QUFwVQGg5hf0cc7Razr1U8FRfZIyhtsYQhgsQSaYIgCKL7UASFkKLrOn749Hv446ubPC3PG0f5kzxfZuzsQbF+FcOhgBGxcMMcFphtBIUJlCDKIpIISkY8MZHERBQfQWHpnWz2lyAIgnCHLvkIKW9s2odHV24FAFx+0nDX5S0RlChvknUrM7YaUbuD2Ek2kXJ+TfO1zSqd0oxRlhcoYht+WQSF3S4hcUIQBOErFEEhpPBpGi/wERS+ooeveHFs1Bbq/gneqOLJclggn+IxIygSk2yR4EGRTGsmgUIQBOEvJFCIHpNIpiyigE/xuE0zFhu1dQc2zVjuQVHDzLzFRWYEhY/+qDwo/LTmTk7kEARBEP5Bv6pEjxEbl6kEirAYADPaUhLufgSiKKhu1OakULqM9BJfxeNUZpz+t8sSQTFFDkEQBOEfJFAIV2RTiHnExmVtFoHCtY5P2RVKc8ZkWtWDHiKiByXluYqHlRkHUJrpYGtp2W8rM854UBJJyTZIoBAEQfgJCRTCFbfGZ6JAaY8ljQ6uTimernjSWLcnAiXk1OreiwclFERFhKV4Epn1dJuB16ji4SIolOIhCILIDfSrSkjhT+yyyAdPVBAoyZSOaCIFXdcdUzwsehIMaLYZPNnAPCgsWmMZFuiwHh/9YF1g2VydREo3hgnaqngS9ioeiqAQBEH4i+8CZeHChTj22GNRUVGB2tpanHfeeVi/fr1lmWnTpkHTNMv/11xzjd+7QviEWwSFCRS+UVlbNIFoIgV+VTFVxARKZXEImqahuzhHUJzKjM3oB5ujw+bq8KXEEbEPChdBiXJRGIIgCMI/fBcoS5cuxdy5c7FixQosXrwY8XgcM2bMQHt7u2W5K6+8Ejt27DD+v/vuu/3eFaIH8HohKSkP5mFpmnQ1TPpE3R5NWFreA3ah0+KD/wQwPSgsWsOLC+cIihn9MCIoGYHCvCihgGYIFGbk7UokDbHFUjw9MfkSBEEQdnxv1Pb8889b7j/88MOora3F6tWrcfLJJxuPl5aWoq6uztM2o9EootGocb+lpcWfnSWU8FmduGuKJ32SDgcDQCSEjlgSbdGEIRwY4iyeZp+G7LEeJqx3C19FJOtey+BNsqyPCUvxsOGHFVx0h6WhdD3d4r6yuMiyDYIgCMI/cv6r2tzcDACoqamxPP7oo4+iX79+GDNmDBYsWICOjg7lNhYuXIiqqirj/4aGhpzuM+Heop4nxjU0Yyfxti57BEVM8exqTYvOfuWRHu1rRWZIHxMVqkZxIrxJtrLEmuJpzfxbwQ0ALC4KGtEUFv1h2+hJozmCIAjCTk5b3adSKVx33XU48cQTMWbMGOPxr371qxg6dCjq6+vx7rvv4oYbbsD69evx1FNPSbezYMECzJ8/37jf0tJCIiXH8OkYWQdYHtYHJRwMGHNx2mMJRATjqJji2dHcBQAYWFXco31l3pe2rgR0XbeWCjsJFK4PiuFByQiPFi6CwlNZUoTdrVE0d8YxuA9fxUMChSAIwk9yKlDmzp2LtWvX4tVXX7U8ftVVVxm3x44di4EDB+L000/Hxo0bMXLkSNt2IpEIIpGeXWUT2cFHTdwiKEa/kKIgijMRhrZoEsUha7v8jqj1/o6mTgB+CJR0lCOR0tEVT6Gty2sExUzPVJWmt9HcGUcypVtSPDxVGYHCUkFNHf74aAiCIAgrORMo8+bNw6JFi7Bs2TIMHjzYcdkpU6YAADZs2CAVKMSBx63BGo8RRQgFDLPot/76tm05lj5hmBGUkh7ta1k4iIAGpPR0aqbdawSFi370LYsY29jbHjVSPOURq/BgkRbmn9meEVn11T0TWQRBEIQV3z0ouq5j3rx5ePrpp/Hiiy9i+HD3Sbhr1qwBAAwcONDv3SG6SZITJW5lxma0oQhvbNqnXI6d1Bmb9qQruxpqSru7mwAATTP7qDR3xtFumUicVK1mRH6Ki4IIBjTUlKWjdLtbo8Z7qpREUABTbDW2+COyCIIgCCu+C5S5c+fiL3/5Cx577DFUVFSgsbERjY2N6OxMX2lu3LgRd9xxB1avXo3Nmzfjn//8J772ta/h5JNPxrhx4/zeHaKbZONBYdGGyuIQvjlNHQFjaREgPfPms0z04bDa8p7sKgAzzbOzJWp5XJwTxEimdOM5lpbqX5EWKLtao9iZER59ysKW9dj9dZ81Q9d133w0BEEQhBXfBcr999+P5uZmTJs2DQMHDjT+f+KJJwAA4XAYL7zwAmbMmIFRo0bhO9/5DmbNmoVnn33W710hegDvO3GLoDBTanlxCN+YdphyOT6C8snudPSkpiyMGkEEdAdWqvzpfms1mMqDwk8kZgbX2oxA+X/LPsFDr20GAIwZVGlZ7+yx6SjfP9/Zjs17OxBLpBAOBVBHAoUgCMJXfPegOHXuBICGhgYsXbrU75clfIb3oCRdPCi8oVTsfQIA8049DL95aYPFg7JhVxsAf6InQDr68cEO4MPGVgDpiqJYMt3JNpFMIRS0anF+IjETKCP7l2PpR7vx+sa9xnPHDOljWe/Ew/ohoAH7O+L499odAICj6ytRFKQ+KARBEH5Cv6qEFIsHxTXFY3pQAODv10xFfVUxzptQj/X/eyZOG10LANjZ0mX0QvFboAzIRD8+bEw38RvImVZlaR5mkC0Kaoaomjqyr2WZW845CkP7llkeKy4KGo/d/Xx6hMOEhmof3gFBEATBQwKFkJLIIsVjVrykA3KTh9Xg9QWn454LJyISCuKIARWoiISwpy2GVzfsAQB8vCsd6Tisvz8CpbYyLVA+2JHebl2lKVD42TmMLskMnRMPMwVKVUkRLjtRbvA+9chay30SKARBEP5DAoWQkkh6Eyi6rmPj7nQ0ROwZwiiPhDBlRPrkv2Vf2iPidwSltiItSJjPpX9FumwYkEdQjNJoboZOaTiE66YfjuKiAK78nLr67OZzjsLR9aY3RUwDEQRBED0np43aiN5LIuXNg7Juews2Zgyv4wdXK5frX5E2wu5riyGeTGHL3rRQ8UugDKq2lvn2K48gHAqgK56SGmWbFYMKr5t+BK6bfoTr643oX45129PppJ6WSRMEQRB2SKAQUnhREnfwoGzLREQmNFRjWL8y5XJ9Mz1G9rVH8cGOFiRSOioiId/Kc4f3t772qLoKhINpgSJr1tbcww6wN5x5JLbsbcdlJw7r1voEQRCEM5TiIQz4CqyEx1b3rFGZWydVVkq8pz2GBU+9ByBtSmWTgnvKECGK8aVjBhmzgJwiKNXdFCiD+5Tin/NOwpcmOndJJgiCILoHCRQCQLqXyWk/X4qb/7EWgDcPypa97bjt2fcBmB4QFX3L0wLluXd3GKmRr00d1tPdNigKBhDOlPr2K48gEgoa92UelCaW4imlGToEQRCFCAkUAgDw7/d2YNOedvxp+RYAQqM2RTfWq/602rjt1qhsVJ214VlFJISTDu/X3d2V8n/fOAFjBlXi7gvGAkgPAQSAzpi93T0b8ldd0vMmcQRBEIT/kEAhAAChoJlqSaZ0y4BAVR+U9TtbjdsNfZyNokfWVeA3X51o3G8VJhv7wdjBVVh07edw2qgBAIDyTF+WNslrNXfGAADVFEEhCIIoSEig+EwqpeNn/1mPlz7cle9dyQq+H0hTRwwdXNShPeYuJo6sq3Bd5gvj6ru3c92EDfprFaYoA8DetrRA6UMChSAIoiChKh6fWfHJXvzmpQ0AgN98dSJ2tURxzNA+Bd/MK86ldPa1x7C71Ry6t7/DfoIXGdbXW6ntaaNq8eKHuzBucFX2O5klFYZAsQusve1pgdKvPJLz/SAIgiCyhwSKQDKlI55MGfNZsuXTzIReAJj32NvG7RtnjsI1p6gn/eabLi5isrc9ZkzzBYD9mZM5z3ufNhu3771oom3WjYqffXk8Hn59M748KffVL6yzrSzFs6ctLcD6VZBAIQiCKEQoxcPxjzWf4eS7X8IfXvmk29v46xtbpY/f9e8P0Z4D34VfdHHTfZ95+zN8tLPNuL+/wypQ2qMJXPXnVQCAc8fX49zx3lM3NWVhzP/8EQekuRmbDdQiSfHsyUSI+vowSZkgCILwHxIoHCldx2dNnfjLiq3KyhUnNu5uw9tbmwCk+3LM//wRePTrU4znf7n4I2MGTKHBV7o8/uY2y3Nvbdlv3G6LJnD+b1/HjuZ0hOXC4xoOzA52A1WKpzOWRHvm/VIEhSAIojAhgcJx1tiBqIiE0NjShY93tbmvIPDGpn3G7TknDMO3Tj8cJx7Wz4gw/OHVTbjpmbXGMnxjtHzT6SCctjd34b/rGgEA/3p3B9bvbEVVSRFuOns0po7oq1wv37AUzx7OTwMAb29NC65+5WFURCjLSRAEUYiQQOGIhIIYkWmZvjXTwt0riWQKv315g3F/9pQhxu2BXJfVv6/+FIvf34k7Fr2PkT/4F75432tY+1kz8o1MoPzha5ON2398bRMAYPknewEAlxw/FF//3AjfOsHmgsMHpCuLXl6/G/s4H83iD3YCSBt2C3n/CYIgDmVIoAgwb8SqzftclrTy7mfN2LavE6GAhtdvPM1ishX7iFz5p1V48NVNSOnAO9ua8IVfv4phNz6H/130Pl5evwsbdrWKm8+arngyqwgNM8mePXYg+pQWoTwSwrHDa/DPeScCAN7a0oTzf/sann77MwDACSMLN3LCOPnwfhjRrwyxZAorM8Jq274OPPTaZgDA6aMH5HHvCIIgCCcovi0wIjPw7qHXNqOmLIIpI2pwzJA+AIC9bVGs296Czx3ez3bl/fL63QCA00fXol6YrPuliYPw4KubEApo6FceMebXiPzh1U34w6ubjPsVxSGcckR/3POVCZ6rZF7fsAd3Pf8h3s1U2Zw9diAG9SnBSx/uwhlH1+G7ZxwpXa8rnvbcHFVfiZ9cMA5d8SSqSoowPHM8YskU3sr4azQNOGZoH0/7k080TcOkoX3wyZ52fOPRt3DbuUfjln+uAwAMqIzgcz53siUIgiD8gwSKwOzjh+L3r3yCrngKP3n+QwDAh3eciTc27cNvX96AFZ/sw01nj8amPe14dOVWnD12IBKpFJZ+lBYoZ40daNvmmEFVeGH+KRhYVYxIKIDtTV14cvU2jOxfjuueWGMsN6JfGT7Z027cb+1KYNG7O7By0z58/aThqK8uwZ+Wb0YskcL/HNuAzx81APvb41j60S78e20jPtjRYggNxnPv7TBuf7xrA+acMAz/fb8Rz727AxXFIRw3vC/6V0TQGk1XupQUBVEeCRn+jYriIvQpLbL0QvnhWaO7XYZ9oDn/mMF4cvWnAGCIEwC4+4LxKA3T158gCKJQ0fRCcmp6pKWlBVVVVWhubkZlZaX7Clny8GubcGtmCF42DO9Xhv9cdzLCIe+Zs3+s+Qy7W6O44qTh0HXg7W37ccUjqxAKBIxeHd1laN9SHDOkj5GW8cJjX5+CEw6zRhbe+7QZdyx6H6cc2R/fnDay1/k2Xv14Dy5+cKVx/7szjsC80w7P4x4RBEEcmmRz/iaBIiGV0nHpw29iWSYq4oV+5RE8cfXxGNm/3Lf92NMWxZ+Wb8GKT/YaFUKTh/ZBY0sXuuJJ7Mm0ay+PhBAMaGjujKO+qhh3nj8WRw+sRG1l2pzb2hXHLf9Yh6cyQiUU0KQTiofUlOLF75ziOZ3Um0imdHy6vwNDakp7ncAiCII4WCCB4gPxZAqb97Qjlkzhr29sRWVxEQZUFmPysD54c9M+6Eg3KXtraxMSyRTGNVRjkOA98ZNdrV2IxlOGibc9msDzaxtx3PAaT03PtuxtxzV/eQvHDKnGrecejaKMCIknU/jXezvQpzSMsYOq0IcalxEEQRA5ggQKQRAEQRAFRzbn74Mvlk8QBEEQRK+HBApBEARBEAUHCRSCIAiCIAoOEigEQRAEQRQcJFAIgiAIgig4SKAQBEEQBFFwkEAhCIIgCKLgIIFCEARBEETBQQKFIAiCIIiCgwQKQRAEQRAFR14Fyn333Ydhw4ahuLgYU6ZMwRtvvJHP3SEIgiAIokDIm0B54oknMH/+fNxyyy146623MH78eJxxxhnYtWtXvnaJIAiCIIgCIW/DAqdMmYJjjz0Wv/nNbwAAqVQKDQ0NuPbaa3HjjTdalo1Go4hGo8b9lpYWNDQ00LBAgiAIguhFFPywwFgshtWrV2P69OnmjgQCmD59OpYvX25bfuHChaiqqjL+b2hoOJC7SxAEQRDEASaUjxfds2cPkskkBgwYYHl8wIAB+PDDD23LL1iwAPPnzzfuNzc3Y8iQIWhpacn5vhIEQRAE4Q/svO0leZMXgZItkUgEkUjEuM/eIEVSCIIgCKL30draiqqqKsdl8iJQ+vXrh2AwiJ07d1oe37lzJ+rq6lzXr6+vx7Zt21BRUQFN03zdN+Zv2bZtG/lbcggd5wMDHecDAx3nAwcd6wNDro6zrutobW1FfX2967J5ESjhcBiTJk3CkiVLcN555wFIm2SXLFmCefPmua4fCAQwePDgnO5jZWUlffkPAHScDwx0nA8MdJwPHHSsDwy5OM5ukRNG3lI88+fPx5w5czB58mQcd9xxuOeee9De3o7LLrssX7tEEARBEESBkDeB8pWvfAW7d+/GzTffjMbGRkyYMAHPP/+8zThLEARBEMShR15NsvPmzfOU0jmQRCIR3HLLLRZTLuE/dJwPDHScDwx0nA8cdKwPDIVwnPPWqI0gCIIgCEIFDQskCIIgCKLgIIFCEARBEETBQQKFIAiCIIiCgwQKQRAEQRAFBwkUjvvuuw/Dhg1DcXExpkyZgjfeeCPfu9SrWLhwIY499lhUVFSgtrYW5513HtavX29ZpqurC3PnzkXfvn1RXl6OWbNm2ToKb926FWeffTZKS0tRW1uL733ve0gkEgfyrfQq7rrrLmiahuuuu854jI6zP3z22We4+OKL0bdvX5SUlGDs2LFYtWqV8byu67j55psxcOBAlJSUYPr06fj4448t29i3bx9mz56NyspKVFdX44orrkBbW9uBfisFTTKZxI9+9CMMHz4cJSUlGDlyJO644w7LvBY61tmzbNkynHPOOaivr4emaXjmmWcsz/t1TN9991187nOfQ3FxMRoaGnD33Xf78wZ0Qtd1XX/88cf1cDis//GPf9TXrVunX3nllXp1dbW+c+fOfO9ar+GMM87QH3roIX3t2rX6mjVr9LPOOksfMmSI3tbWZixzzTXX6A0NDfqSJUv0VatW6ccff7x+wgknGM8nEgl9zJgx+vTp0/W3335b/9e//qX369dPX7BgQT7eUsHzxhtv6MOGDdPHjRunf/vb3zYep+Pcc/bt26cPHTpUv/TSS/WVK1fqn3zyif6f//xH37Bhg7HMXXfdpVdVVenPPPOM/s477+jnnnuuPnz4cL2zs9NY5swzz9THjx+vr1ixQn/llVf0ww47TL/ooovy8ZYKljvvvFPv27evvmjRIn3Tpk36k08+qZeXl+u/+tWvjGXoWGfPv/71L/2HP/yh/tRTT+kA9KefftryvB/HtLm5WR8wYIA+e/Zsfe3atfpf//pXvaSkRP/d737X4/0ngZLhuOOO0+fOnWvcTyaTen19vb5w4cI87lXvZteuXToAfenSpbqu63pTU5NeVFSkP/nkk8YyH3zwgQ5AX758ua7r6T+oQCCgNzY2Gsvcf//9emVlpR6NRg/sGyhwWltb9cMPP1xfvHixfsoppxgChY6zP9xwww36SSedpHw+lUrpdXV1+k9/+lPjsaamJj0Sieh//etfdV3X9ffff18HoL/55pvGMv/+9791TdP0zz77LHc738s4++yz9csvv9zy2Pnnn6/Pnj1b13U61n4gChS/julvf/tbvU+fPpbfjRtuuEE/8sgje7zPlOIBEIvFsHr1akyfPt14LBAIYPr06Vi+fHke96x309zcDACoqakBAKxevRrxeNxynEeNGoUhQ4YYx3n58uUYO3aspaPwGWecgZaWFqxbt+4A7n3hM3fuXJx99tmW4wnQcfaLf/7zn5g8eTK+/OUvo7a2FhMnTsT/+3//z3h+06ZNaGxstBznqqoqTJkyxXKcq6urMXnyZGOZ6dOnIxAIYOXKlQfuzRQ4J5xwApYsWYKPPvoIAPDOO+/g1VdfxcyZMwHQsc4Ffh3T5cuX4+STT0Y4HDaWOeOMM7B+/Xrs37+/R/uY106yhcKePXuQTCZtbfYHDBiADz/8ME971btJpVK47rrrcOKJJ2LMmDEAgMbGRoTDYVRXV1uWHTBgABobG41lZJ8De45I8/jjj+Ott97Cm2++aXuOjrM/fPLJJ7j//vsxf/58/OAHP8Cbb76Jb33rWwiHw5gzZ45xnGTHkT/OtbW1ludDoRBqamroOHPceOONaGlpwahRoxAMBpFMJnHnnXdi9uzZAEDHOgf4dUwbGxsxfPhw2zbYc3369On2PpJAIXLC3LlzsXbtWrz66qv53pWDjm3btuHb3/42Fi9ejOLi4nzvzkFLKpXC5MmT8eMf/xgAMHHiRKxduxYPPPAA5syZk+e9O7j429/+hkcffRSPPfYYjj76aKxZswbXXXcd6uvr6VgfwlCKB0C/fv0QDAZtVQ47d+5EXV1dnvaq9zJv3jwsWrQIL730EgYPHmw8XldXh1gshqamJsvy/HGuq6uTfg7sOSKdwtm1axeOOeYYhEIhhEIhLF26FPfeey9CoRAGDBhAx9kHBg4ciKOOOsry2OjRo7F161YA5nFy+t2oq6vDrl27LM8nEgns27ePjjPH9773Pdx444248MILMXbsWFxyySW4/vrrsXDhQgB0rHOBX8c0l78lJFAAhMNhTJo0CUuWLDEeS6VSWLJkCaZOnZrHPetd6LqOefPm4emnn8aLL75oC/tNmjQJRUVFluO8fv16bN261TjOU6dOxXvvvWf5o1i8eDEqKyttJ4tDldNPPx3vvfce1qxZY/w/efJkzJ4927hNx7nnnHjiibYy+Y8++ghDhw4FAAwfPhx1dXWW49zS0oKVK1dajnNTUxNWr15tLPPiiy8ilUphypQpB+Bd9A46OjoQCFhPR8FgEKlUCgAd61zg1zGdOnUqli1bhng8biyzePFiHHnkkT1K7wCgMmPG448/rkciEf3hhx/W33//ff2qq67Sq6urLVUOhDPf+MY39KqqKv3ll1/Wd+zYYfzf0dFhLHPNNdfoQ4YM0V988UV91apV+tSpU/WpU6caz7Py1xkzZuhr1qzRn3/+eb1///5U/uoCX8Wj63Sc/eCNN97QQ6GQfuedd+off/yx/uijj+qlpaX6X/7yF2OZu+66S6+urtb/8Y9/6O+++67+xS9+UVqmOXHiRH3lypX6q6++qh9++OGHdOmrjDlz5uiDBg0yyoyfeuopvV+/fvr3v/99Yxk61tnT2tqqv/322/rbb7+tA9B/8Ytf6G+//ba+ZcsWXdf9OaZNTU36gAED9EsuuURfu3at/vjjj+ulpaVUZuw3v/71r/UhQ4bo4XBYP+644/QVK1bke5d6FQCk/z/00EPGMp2dnfo3v/lNvU+fPnppaan+pS99Sd+xY4dlO5s3b9Znzpypl5SU6P369dO/853v6PF4/AC/m96FKFDoOPvDs88+q48ZM0aPRCL6qFGj9N///veW51OplP6jH/1IHzBggB6JRPTTTz9dX79+vWWZvXv36hdddJFeXl6uV1ZW6pdddpne2tp6IN9GwdPS0qJ/+9vf1ocMGaIXFxfrI0aM0H/4wx9aSlfpWGfPSy+9JP1NnjNnjq7r/h3Td955Rz/ppJP0SCSiDxo0SL/rrrt82X9N17lWfQRBEARBEAUAeVAIgiAIgig4SKAQBEEQBFFwkEAhCIIgCKLgIIFCEARBEETBQQKFIAiCIIiCgwQKQRAEQRAFBwkUgiAIgiAKDhIoBEEQBEEUHCRQCIIgCIIoOEigEASRFZdeeik0TYOmaSgqKsLw4cPx/e9/H11dXZ7Wf/nll6Fpmm3aMkEQBE8o3ztAEETv48wzz8RDDz2EeDyO1atXY86cOdA0DT/5yU8O6H7E43EUFRUd0NckCOLAQBEUgiCyJhKJoK6uDg0NDTjvvPMwffp0LF68GACQSqWwcOFCDB8+HCUlJRg/fjz+/ve/AwA2b96MU089FQDQp08faJqGSy+9FAAwbNgw3HPPPZbXmTBhAm699VbjvqZpuP/++3HuueeirKwMd955J2699VZMmDABf/7znzFs2DBUVVXhwgsvRGtrq7He3//+d4wdOxYlJSXo27cvpk+fjvb29twdIIIgegwJFIIgesTatWvx+uuvIxwOAwAWLlyIP/3pT3jggQewbt06XH/99bj44ouxdOlSNDQ04P/+7/8AAOvXr8eOHTvwq1/9KqvXu/XWW/GlL30J7733Hi6//HIAwMaNG/HMM89g0aJFWLRoEZYuXYq77roLALBjxw5cdNFFuPzyy/HBBx/g5Zdfxvnnnw+ak0oQhQ2leAiCyJpFixahvLwciUQC0WgUgUAAv/nNbxCNRvHjH/8YL7zwAqZOnQoAGDFiBF599VX87ne/wymnnIKamhoAQG1tLaqrq7N+7a9+9au47LLLLI+lUik8/PDDqKioAABccsklWLJkCe68807s2LEDiUQC559/PoYOHQoAGDt2bA/ePUEQBwISKARBZM2pp56K+++/H+3t7fjlL3+JUCiEWbNmYd26dejo6MDnP/95y/KxWAwTJ0705bUnT55se2zYsGGGOAGAgQMHYteuXQCA8ePH4/TTT8fYsWNxxhlnYMaMGbjgggvQp08fX/aHIIjcQAKFIIisKSsrw2GHHQYA+OMf/4jx48fjwQcfxJgxYwAAzz33HAYNGmRZJxKJOG4zEAjY0i7xeFz62iKiUVbTNKRSKQBAMBjE4sWL8frrr+O///0vfv3rX+OHP/whVq5cieHDh7u8U4Ig8gV5UAiC6BGBQAA/+MEPcNNNN+Goo45CJBLB1q1bcdhhh1n+b2hoAADDq5JMJi3b6d+/P3bs2GHcb2lpwaZNm3zZR03TcOKJJ+K2227D22+/jXA4jKefftqXbRMEkRsogkIQRI/58pe/jO9973v43e9+h+9+97u4/vrrkUqlcNJJJ6G5uRmvvfYaKisrMWfOHAwdOhSapmHRokU466yzUFJSgvLycpx22ml4+OGHcc4556C6uho333wzgsFgj/dt5cqVWLJkCWbMmIHa2lqsXLkSu3fvxujRo3145wRB5AoSKARB9JhQKIR58+bh7rvvxqZNm9C/f38sXLgQn3zyCaqrq3HMMcfgBz/4AQBg0KBBuO2223DjjTfisssuw9e+9jU8/PDDWLBgATZt2oQvfOELqKqqwh133OFLBKWyshLLli3DPffcg5aWFgwdOhQ///nPMXPmzB5vmyCI3KHpVGtHEARBEESBQR4UgiAIgiAKDhIoBEEQBEEUHCRQCIIgCIIoOEigEARBEARRcJBAIQiCIAii4CCBQhAEQRBEwUEChSAIgiCIgoMECkEQBEEQBQcJFIIgCIIgCg4SKARBEARBFBwkUAiCIAiCKDj+P0mbkEMtyBSLAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Model\n", "net = ts.utils.net.common.Net(\n", " env.observation_space.shape,\n", " env.action_space.n,\n", " hidden_sizes=[128, 128],\n", " device=device,\n", ").to(device)\n", "\n", "optim = torch.optim.Adam(net.parameters(), lr=0.001)\n", "\n", "# Policy\n", "policy = ts.policy.DQNPolicy(\n", " model=net, # value network\n", " optim=optim, # optimizer\n", " discount_factor=0.99, # gamma\n", " estimation_step=3, # n-step returns\n", " is_double=True, # double Q-learning\n", " target_update_freq=120, # how often to update the target network\n", " action_space=env.action_space, # action space\n", ")\n", "policy.set_eps(0.1) # epsilon-greedy action selection\n", "\n", "# Collector for training\n", "collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " buffer=ts.data.VectorReplayBuffer(20000, 10),\n", " exploration_noise=True\n", ")\n", "# Collector for testing (without exploration). No need for a buffer\n", "test_collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " exploration_noise=False\n", ")\n", "# Pre-fill the training buffer with random transitions\n", "collector.collect(n_step=1000, random=True)\n", "\n", "# Interaction\n", "returns = []\n", "for iteration in range(1000):\n", "\n", " # Training mode\n", " policy.train()\n", "\n", " # Exploration schedule\n", " if iteration <= 100:\n", " eps = 0.5\n", " elif iteration <= 900:\n", " eps = 0.5 - (iteration - 100) / (900 - 100) * (0.5 - 0.05)\n", " else:\n", " eps = 0.05\n", " policy.set_eps(eps)\n", " \n", " # Collect transitions\n", " result = collector.collect(n_step=1000)\n", "\n", " # Train DQN network on minibatch\n", " policy.update(\n", " buffer=collector.buffer,\n", " sample_size=0, # use the whole buffer\n", " batch_size=64,\n", " repeat=10,\n", " )\n", "\n", " # Test 10 episodes\n", " policy.eval()\n", " result = test_collector.collect(n_episode=10)\n", " mean_reward = result['rew']\n", " #print(iteration, \":\", mean_reward, eps)\n", " returns.append(mean_reward)\n", "\n", "plt.figure()\n", "plt.plot(np.array(returns))\n", "plt.xlabel(\"Epochs\")\n", "plt.ylabel(\"Returns\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## PPO\n", "\n", "Now that DQN works on Cartpole, let's use PPO and compare its performance to DQN.\n", "\n", "You will need to use the PPO policy, obviously:\n", "\n", "```python\n", "policy = ts.policy.PPOPolicy(\n", " actor=actor, \n", " critic=critic, \n", " optim=optim,\n", " dist_fn=torch.distributions.Categorical, \n", " action_space=env.action_space,\n", " discount_factor=0.99,\n", " max_grad_norm=0.5,\n", " eps_clip=0.2,\n", " gae_lambda=0.95,\n", " deterministic_eval=True,\n", " action_scaling=False,\n", ")\n", "```\n", "\n", "It has many more hyperparameters, which can be let at their default value (or not, depending on the time you have). Check the doc for their meaning. The important thing is that you now need an actor and a critic, not a single network.\n", "\n", "One way to do it is to use the actor/critic specifications provided by tianshou:\n", "\n", "```python\n", "features = ts.utils.net.common.Net(\n", " state_shape=env.observation_space.shape, \n", " hidden_sizes=[64, 64], \n", " device=device)\n", "\n", "actor = ts.utils.net.discrete.Actor(\n", " preprocess_net=features, \n", " action_shape=env.action_space.n, \n", " device=device).to(device)\n", "\n", "critic = ts.utils.net.discrete.Critic(\n", " preprocess_net=features, \n", " device=device).to(device)\n", "\n", "actor_critic = ts.utils.net.common.ActorCritic(actor=actor, critic=critic)\n", "\n", "optim = torch.optim.Adam(actor_critic.parameters(), lr=0.001)\n", "```\n", "\n", "``features`` is the shared feature extractor between the actor and the critic. ``actor`` is the policy head (one neuron per discrete action), ``critic`` is a single output neuron for the value V(s). ``actor_critic`` is the combined two-headed network. \n", "\n", "When defining the PPO policy, `dist_fn=torch.distributions.Categorical` specifies how exploration is performed, here a softmax over the two actions left and right. \n", "\n", "**Q:** Implement PPO on Cartpole. You will need to find the right hyperparameters for the task. \n", "\n", "Remember that learning is **on-policy**, so the transition buffer must be emptied after training the network. Your interaction loop must therefore look like this:\n", "\n", "```python\n", "for iteration in range(N):\n", "\n", " # Collect enough on-policy steps\n", " result = collector.collect(...)\n", "\n", " # Update the PP0 network by learning the on-policy buffer\n", " policy.update(...)\n", "\n", " # Empty the buffer as we are on-policy\n", " collector.reset_buffer(keep_statistics=False)\n", "```\n", "\n", "Do not hesitate to collect many on-policy steps before training the network." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGwCAYAAACD0J42AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzfUlEQVR4nO3df3xU1Z3/8fdMJpmEH0lINImRBPAnUClSUIiyCpICAfEH2F1cSqPykNVNVEgXFX+tbpfGurb+oCi2q7BWWb9rK1TTlhZBg6whQpBWKUXRKKyQoMYkJJDJJHO/f8BcMhBIZubemQl5PR+PPB7M3JvJmdN9eN577vmc4zAMwxAAAEAMcUa7AQAAAMcjoAAAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzXNFuQCh8Pp/27dun/v37y+FwRLs5AACgGwzD0MGDB5WdnS2n89RzJD0yoOzbt085OTnRbgYAAAjB3r17NXDgwFPe0yMDSv/+/SUd+YLJyclRbg0AAOiOxsZG5eTkmOP4qfTIgOJ/rJOcnExAAQCgh+nO8gwWyQIAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzCCgAACDmEFAAAEDMIaAAAICYQ0ABAAAxJ6iAUlpaqksuuUT9+/dXRkaGrrvuOu3atSvgnpaWFhUVFSk9PV39+vXTrFmzVFtbG3DPnj17NH36dPXp00cZGRlatGiR2trawv82AADgtBBUQCkvL1dRUZE2b96sdevWyev1avLkyWpubjbvWbhwod544w29+uqrKi8v1759+zRz5kzzent7u6ZPn67W1la9++67+q//+i+tXLlSDz30kHXfCgAA9GgOwzCMUH/5yy+/VEZGhsrLy3XFFVeooaFBZ555platWqUbbrhBkvS3v/1Nw4YNU0VFhcaNG6c//OEPuvrqq7Vv3z5lZmZKkpYvX6577rlHX375pRISErr8u42NjUpJSVFDQ0OPPiyw3WfI2+5TYnxct3/HMAzVH/IqJSleTufJD1tq9rTpm0OtVjQTANALuV1xOrO/29LPDGb8Dus044aGBklSWlqaJKmqqkper1f5+fnmPUOHDlVubq4ZUCoqKjRixAgznEjSlClTdPvtt2vHjh0aNWrUCX/H4/HI4/EEfMHTwT/+crOqv2rW24smqE9C5/9T/OX/6vXq1v/T3m8OaW/dIf3fN4flafNpcHofLfzuBbr629mK6xBUvm7yaHn5J3qx4nN52nyR+ioAgNPMFRecqRdvuTRqfz/kgOLz+bRgwQJdfvnluuiiiyRJNTU1SkhIUGpqasC9mZmZqqmpMe/pGE781/3XOlNaWqpHHnkk1KbGJMMwtOWzOvkM6bOvDml4dudJ8t7ffKC/7j8xkH329SHd9cp2LXtrt0q+e4HGnZOu/3ynWi/8b7UOtbZLkhJcTnV9oDUAACdKiIvuCBJyQCkqKtKHH36oTZs2WdmeTi1evFglJSXm68bGRuXk5Nj+d+3U3Nou39GHa/WneBRT09giSVo05UJdnJOqnAF9lJIUr5cqP9dz5Z/oo9om3fbSNrmcDrUd/cCLzk7WDydfqAkXnCmHg4gCAOh5QgooxcXFKisr08aNGzVw4EDz/aysLLW2tqq+vj5gFqW2tlZZWVnmPe+9917A5/mrfPz3HM/tdsvttvY5WLQ1Hvaa/647SUBp9xlmePn7MTkBzwKLJp6n748bpP9851O9sKlaza3tuiCzn0q+e6GmfCuTYAIA6NGCquIxDEPFxcVavXq1NmzYoCFDhgRcHz16tOLj47V+/XrzvV27dmnPnj3Ky8uTJOXl5emDDz7QgQMHzHvWrVun5ORkDR8+PJzv0qMcbDlWVv1Nc+cBpfGw15xlSe0Tf8L1lKR4/XDyhXrnnqv0yvxx+sNdV2jqRVmEEwBAjxfUDEpRUZFWrVql3/72t+rfv7+5ZiQlJUVJSUlKSUnRvHnzVFJSorS0NCUnJ+uOO+5QXl6exo0bJ0maPHmyhg8frrlz5+qxxx5TTU2NHnjgARUVFZ12sySn0tjSYQal2dvpPf6ZleREl+LjTp4l0/omaNw56dY2EACAKAoqoDz77LOSpAkTJgS8v2LFCt10002SpCeeeEJOp1OzZs2Sx+PRlClT9Mwzz5j3xsXFqaysTLfffrvy8vLUt29fFRYW6t/+7d/C+yY9TMdHPCcrB/bPrKT17br0GgCA00lQAaU7W6YkJiZq2bJlWrZs2UnvGTRokH7/+98H86dPO4EzKJ0HFP/7AwgoAIBehrN4oqTxcIc1KCebQTn6flofAgoAoHchoERJdx7x+NemMIMCAOhtCChR0vERzzcnWSRrzqAQUAAAvQwBJUo6PuLpcg0Kj3gAAL0MASVKOs6gHPa26/DR7ek7OlbFc+IeKAAAnM4IKFHSMaBIna9D8e+DwgwKAKC3IaBEScdHPFLnAYV9UAAAvRUBJUpOmEHpZKEs+6AAAHorAkqU+M/iST8aPo4/MNDb7lPj0XvYBwUA0NsQUKLAMAxzH5Tc9D6STjwwsP7QketOh5ScxCJZAEDvQkCJgsPedrUdPaZ4cHpfSSeWGvvXpKT2SVCck9OJAQC9CwElCvwLZF1Oh7JTEyVJ9cc94jm2BwqzJwCA3oeAEgX+BbLJSfFK6+uWJNUdOn7RLBU8AIDei4ASBf71J8mJLnMTtuPXoLAHCgCgNyOgREHHGRR/ADlhDQozKACAXoyAEgX+NSjJiccCyvEbtXGSMQCgNyOgRIF/BqV/osucITlZFQ97oAAAeiMCShQcW4MSb86QeNp8AQcGsossAKA3I6BEgX+H2OQkl/omxCkh7sj/DB13kzVnUDjJGADQCxFQoqDjDIrD4dCATip5ju2DwgwKAKD3IaBEwUFzBuVIMOmskocqHgBAb0ZAiYJjZcYuSTqhkqfF267mo+tRWIMCAOiNCChR0PERj6QTKnn8BwW6nA71d7ui0EIAAKKLgBIFjcc/4vGvQTkaTL5u9hx9P0EOBwcFAgB6HwJKFJwwg+J/xHN0BuWbo5u0sQcKAKC3IqBEmGEYJ65B8T/iOboGxTyHhxJjAEAvRUCJsBavT952Q5LU/7g1KMdmUKjgAQD0bgSUCPPPnjgdUt+EOElS6nFlxuyBAgDo7QgoEWauP0mKNxfAph1XZnxsF1kCCgCgdyKgRJi5/iTx2PqSjlU8hmEwgwIA6PUIKBHWePjYOTx+/pmS1jafDrW2M4MCAOj1CCgR1tkMSlJ8nNyuowcGNreq7miZMbvIAgB6KwJKhJmbtHUIKA6H41glz6HWY1U8POIBAPRSBJQIO7ZINnALe38lz9fNreyDAgDo9QgoEdbZIx5JSjsaRr745rBa23xH32MGBQDQOxFQIuzYItnAgOKv2Pn0y2ZJktvlVFJ8XGQbBwBAjCCgRJh/BqV/YuAjHv9sySdfNpmvOSgQANBbEVAi7PiDAv38Myj+gMIeKACA3oyAEmFmFU/S8WtQjgSSL+oPB7wGAKA3IqBE2EFzBuX4Kp4jgcU4co4ge6AAAHo1AkqEmVU8J5lBMV/3ocQYANB7EVAiyDCMLqt4zNfMoAAAejECigUMw9DeukMy/M9nTsLT5lNr+5E9To5/xHPCDAoBBQDQixFQLPDM25/o7x57S6//ed8p7/NX8DgdUt+EwIBywgwKVTwAgF6MgGKBTw4cKQ1+628HTnmfv4Knf2K8nM7APU6SEuICNmZjBgUA0JsRUCzgOfrY5v299ae879gCWVen1zuGEmZQAAC9GQHFAh7vkYDy+deHVHf0JOLO+B/x9Hd3XqGT2qFyhxkUAEBvRkCxgH/hqyT9+RSzKMc2aet6BiWVMmMAQC8WdEDZuHGjZsyYoezsbDkcDq1ZsybgelNTk4qLizVw4EAlJSVp+PDhWr58ecA9LS0tKioqUnp6uvr166dZs2aptrY2rC8STa1t7ea/T/WY52Tb3Pv5H+v0TYhTIgcFAgB6saADSnNzs0aOHKlly5Z1er2kpERr167VSy+9pJ07d2rBggUqLi7W66+/bt6zcOFCvfHGG3r11VdVXl6uffv2aebMmaF/iyjztB2bQdl+yhmUzjdp8/PPoKT14/EOAKB36/xZwykUFBSooKDgpNffffddFRYWasKECZKk+fPn67nnntN7772na665Rg0NDXr++ee1atUqXXXVVZKkFStWaNiwYdq8ebPGjRsX2jeJota2wEc8hmF0ehKxuUlbFzMoaSyQBQD0cpavQbnsssv0+uuv64svvpBhGHrrrbf00UcfafLkyZKkqqoqeb1e5efnm78zdOhQ5ebmqqKiotPP9Hg8amxsDPiJJR0DSsNhr6q/au70vi6reI7OnLCLLACgt7M8oCxdulTDhw/XwIEDlZCQoKlTp2rZsmW64oorJEk1NTVKSEhQampqwO9lZmaqpqam088sLS1VSkqK+ZOTk2N1s8Pif8ST4DrSnSd7zNPVGpTJwzN11dAMFV422PI2AgDQk9gSUDZv3qzXX39dVVVV+ulPf6qioiK9+eabIX/m4sWL1dDQYP7s3bvXwhaHzz+DcnFOqqRTBJSWzs/h8ctMTtQLN12iiRdmWN5GAAB6kqDXoJzK4cOHdd9992n16tWaPn26JOnb3/62tm/frscff1z5+fnKyspSa2ur6uvrA2ZRamtrlZWV1ennut1uud1uK5tqKX+Z8bghaXqvuq4bMyiWdjsAAKcdS2dQvF6vvF6vnM7Aj42Li5PPd2QQHz16tOLj47V+/Xrz+q5du7Rnzx7l5eVZ2ZyI8c+gXDokXZK0c3+jWrztJ9zXVRUPAAA4Iuj/V76pqUm7d+82X1dXV2v79u1KS0tTbm6urrzySi1atEhJSUkaNGiQysvL9eKLL+pnP/uZJCklJUXz5s1TSUmJ0tLSlJycrDvuuEN5eXk9soJHkjxH90E558y+OqNfgr5qatWOfY0aPWhAwH0HzbN4mEEBAOBUgh4pt27dqokTJ5qvS0pKJEmFhYVauXKlXnnlFS1evFhz5sxRXV2dBg0apCVLlui2224zf+eJJ56Q0+nUrFmz5PF4NGXKFD3zzDMWfJ3I8/kMedsNSZLb5dTFOal6c+cBbd9bf0JA6WqRLAAAOCLogDJhwgQZhnHS61lZWVqxYsUpPyMxMVHLli076WZvPUnHbe4TjgsoHbV4281qHx7xAABwapzFE6aOu8i6XXG6OOfIrMn2vd8E3Od/vONwSP3dPOIBAOBUCChh6rhJW3ycQ9/OSZHDIe2tO6yvmzzmNf8C2X5ul5zOE3eZBQAAxxBQwuR/xJPgcsrhcCg5MV7nntlPUuB+KKw/AQCg+wgoYfIcLSd2xx3rSv+Gbe/vqTff62qTNgAAcAwBJUz+GRR3/IkBpfMZFNafAADQFUbLMPnXoCR0MoNSWf21/ulXW3XZuWeoprFFEjMoAAB0BwElTMcfFChJQ7P668LM/tpVe1B/3FGrP+6oNa+xBgUAgK4RUMLkn0Fxu+LM91xxTv3uzvH64IsGvfvJ16r45Gtt+axOnjafLjo7OVpNBQCgxyCghKm1kxkU6UhIGZU7QKNyB6ho4nnytLWrtsGjgQOSotFMAAB6FAJKmPzn8BwfUI7ndsUpN71PJJoEAECPRxVPmDzmIx66EgAAqzCqhulkj3gAAEDoGFXDZO4kG0dXAgBgFUbVMHm8/o3a4rq4EwAAdBcBJUzMoAAAYD1G1TCxBgUAAOsxqobJX2ZMFQ8AANZhVA1TK2XGAABYjlE1TDziAQDAeoyqYfJ0cpoxAAAID6NqmMxHPPF0JQAAVmFUDZOHMmMAACzHqBom/0ZtCS42agMAwCoElDD5N2qjigcAAOswqoap9eg+KFTxAABgHUbVMFFmDACA9RhVw+RhozYAACzHqBomZlAAALAeo2qYWCQLAID1GFXD5C8zdlNmDACAZQgoYfLPoPCIBwAA6zCqhqmVs3gAALAco2qYPOyDAgCA5RhVw+DzGfK2G5JYJAsAgJUYVcPgX38iMYMCAICVGFXD4N+kTSKgAABgJUbVMLR2DCgskgUAwDKMqmHoWGLscDii3BoAAE4fBJQweLxHKnjczJ4AAGApRtYwmNvcx9ONAABYiZE1DGzSBgCAPRhZw8BJxgAA2IORNQz+MmMOCgQAwFoElDAwgwIAgD0YWcPgIaAAAGALRtYw+A8K5BweAACsxcgaBh7xAABgD0bWMJg7yVJmDACApYIeWTdu3KgZM2YoOztbDodDa9asOeGenTt36pprrlFKSor69u2rSy65RHv27DGvt7S0qKioSOnp6erXr59mzZql2trasL5INHi8zKAAAGCHoEfW5uZmjRw5UsuWLev0+ieffKLx48dr6NChevvtt/WXv/xFDz74oBITE817Fi5cqDfeeEOvvvqqysvLtW/fPs2cOTP0bxEl5k6ylBkDAGApV7C/UFBQoIKCgpNev//++zVt2jQ99thj5nvnnnuu+e+GhgY9//zzWrVqla666ipJ0ooVKzRs2DBt3rxZ48aNC7ZJUcMaFAAA7GHpyOrz+fS73/1OF1xwgaZMmaKMjAyNHTs24DFQVVWVvF6v8vPzzfeGDh2q3NxcVVRUdPq5Ho9HjY2NAT+xgCoeAADsYenIeuDAATU1NenRRx/V1KlT9ac//UnXX3+9Zs6cqfLycklSTU2NEhISlJqaGvC7mZmZqqmp6fRzS0tLlZKSYv7k5ORY2eyQtZo7yRJQAACwkuUzKJJ07bXXauHChbr44ot177336uqrr9by5ctD/tzFixeroaHB/Nm7d69VTQ4Lj3gAALBH0GtQTuWMM86Qy+XS8OHDA94fNmyYNm3aJEnKyspSa2ur6uvrA2ZRamtrlZWV1ennut1uud1uK5tqCQ+nGQMAYAtLR9aEhARdcskl2rVrV8D7H330kQYNGiRJGj16tOLj47V+/Xrz+q5du7Rnzx7l5eVZ2RzbmY944gkoAABYKegZlKamJu3evdt8XV1dre3btystLU25ublatGiR/uEf/kFXXHGFJk6cqLVr1+qNN97Q22+/LUlKSUnRvHnzVFJSorS0NCUnJ+uOO+5QXl5ej6rgkSQPG7UBAGCLoAPK1q1bNXHiRPN1SUmJJKmwsFArV67U9ddfr+XLl6u0tFR33nmnLrzwQv3mN7/R+PHjzd954okn5HQ6NWvWLHk8Hk2ZMkXPPPOMBV8nso6tQWEfFAAArOQwDMOIdiOC1djYqJSUFDU0NCg5OTlq7fjBC+9p40df6qffG6lZowdGrR0AAPQEwYzfPJsIQ+vRfVCo4gEAwFqMrGGgzBgAAHswsobBw0ZtAADYgpE1DMygAABgD0bWMBw7zZhuBADASoysYfB4/fugUGYMAICVCChhMGdQ2EkWAABLMbKGoZWzeAAAsAUjaxg87IMCAIAtGFlD5PMZ8rYf2YSXRbIAAFiLkTVE/vUnEjMoAABYjZE1RP5N2iQCCgAAVmNkDVFrx4DCIlkAACzFyBoi/yOeBJdTDocjyq0BAOD0QkAJkX8Gxc3sCQAAlmN0DZG/xJhN2gAAsB6ja4jYpA0AAPswuoaIk4wBALAPo2uI/GXGbhcHBQIAYDUCSoiYQQEAwD6MriHyEFAAALANo2uIzIMCWSQLAIDlGF1DZO6DQpkxAACWY3QNkbmTLDMoAABYjtE1RB4va1AAALALo2uI/DMolBkDAGA9AkqIKDMGAMA+jK4hMs/iIaAAAGA5RtcQmVU8BBQAACzH6BoiHvEAAGAfRtcQUWYMAIB9GF1D5C8zZqM2AACsx+gaIg8zKAAA2IbRNUTH1qCwDwoAAFYjoITIQxUPAAC2YXQNUav/NGMCCgAAlmN0DRFlxgAA2IfRNUQeAgoAALZhdA0RO8kCAGAfRtcQHTvNmC4EAMBqjK4h8m/UlhBHmTEAAFYjoITInEFhJ1kAACzH6Bois4qHnWQBALAco2uIPOyDAgCAbRhdQ+DzGfK2G5JYJAsAgB0YXUPgX38iMYMCAIAdGF1DQEABAMBejK4h8JcYSyySBQDADkGPrhs3btSMGTOUnZ0th8OhNWvWnPTe2267TQ6HQ08++WTA+3V1dZozZ46Sk5OVmpqqefPmqampKdimRI1/BiXB5ZTD4YhyawAAOP0EHVCam5s1cuRILVu27JT3rV69Wps3b1Z2dvYJ1+bMmaMdO3Zo3bp1Kisr08aNGzV//vxgmxI15jb3zJ4AAGALV7C/UFBQoIKCglPe88UXX+iOO+7QH//4R02fPj3g2s6dO7V27Vpt2bJFY8aMkSQtXbpU06ZN0+OPP95poPF4PPJ4PObrxsbGYJttKX+JMZu0AQBgD8tHWJ/Pp7lz52rRokX61re+dcL1iooKpaammuFEkvLz8+V0OlVZWdnpZ5aWliolJcX8ycnJsbrZQWGTNgAA7GX5CPuTn/xELpdLd955Z6fXa2pqlJGREfCey+VSWlqaampqOv2dxYsXq6GhwfzZu3ev1c0OihlQqOABAMAWQT/iOZWqqio99dRT2rZtm6WLR91ut9xut2WfFy4PAQUAAFtZOsK+8847OnDggHJzc+VyueRyufT555/rhz/8oQYPHixJysrK0oEDBwJ+r62tTXV1dcrKyrKyObYxF8m6OMkYAAA7WDqDMnfuXOXn5we8N2XKFM2dO1c333yzJCkvL0/19fWqqqrS6NGjJUkbNmyQz+fT2LFjrWyObZhBAQDAXkEHlKamJu3evdt8XV1dre3btystLU25ublKT08PuD8+Pl5ZWVm68MILJUnDhg3T1KlTdeutt2r58uXyer0qLi7W7NmzO63giUXmQYEskgUAwBZBj7Bbt27VqFGjNGrUKElSSUmJRo0apYceeqjbn/Hyyy9r6NChmjRpkqZNm6bx48frF7/4RbBNiRrzEQ9lxgAA2CLoGZQJEybIMIxu3//ZZ5+d8F5aWppWrVoV7J+OGeZOssygAABgC0bYEPjP4mENCgAA9mCEDYF/BoUqHgAA7EFACQEbtQEAYC9G2BAc2weF7gMAwA6MsCEwDwskoAAAYAtG2BDwiAcAAHsxwoaAMmMAAOzFCBsCf5kxG7UBAGAPRtgQeJhBAQDAVoywITi2BoV9UAAAsAMBJQScZgwAgL0YYUPQSpkxAAC2YoQNAWXGAADYixE2BDziAQDAXoywIWCrewAA7MUIG4JjpxnTfQAA2IERNgT+jdoS4igzBgDADgSUEJgzKOwkCwCALRhhQ2BW8bCTLAAAtmCEDQFlxgAA2IsRNkg+n8EiWQAAbMYIGyR/OJGYQQEAwC6MsEEioAAAYD9G2CD5S4wlFskCAGAXRtgg+WdQElxOORyOKLcGAIDTEwElSOY298yeAABgG0bZIHna2iWxSRsAAHZilA3S4dYjASUxnm3uAQCwCwElSC1HF8kSUAAAsA8BJUgt3iMzKEkEFAAAbENACdJhAgoAALYjoATJXIOSQEABAMAuBJQgtRyt4klkF1kAAGzDKBsk/wxKEjMoAADYhoASJBbJAgBgPwJKkCgzBgDAfgSUIPmreAgoAADYh4ASJMqMAQCwHwElSC3mIlm6DgAAuzDKBsksM2YGBQAA2xBQgsRhgQAA2I+AEiTWoAAAYD8CSpAoMwYAwH4ElCCxURsAAPYjoATJfMRDFQ8AALZhlA0Si2QBALAfASVILewkCwCA7QgoQfIvkmUNCgAA9gk6oGzcuFEzZsxQdna2HA6H1qxZY17zer265557NGLECPXt21fZ2dn6wQ9+oH379gV8Rl1dnebMmaPk5GSlpqZq3rx5ampqCvvL2K2t3afWdgIKAAB2CzqgNDc3a+TIkVq2bNkJ1w4dOqRt27bpwQcf1LZt2/Taa69p165duuaaawLumzNnjnbs2KF169aprKxMGzdu1Pz580P/FhHS0uYz/80jHgAA7OMK9hcKCgpUUFDQ6bWUlBStW7cu4L2f//znuvTSS7Vnzx7l5uZq586dWrt2rbZs2aIxY8ZIkpYuXapp06bp8ccfV3Z29gmf6/F45PF4zNeNjY3BNtsS/vUnkuR28XQMAAC72D7KNjQ0yOFwKDU1VZJUUVGh1NRUM5xIUn5+vpxOpyorKzv9jNLSUqWkpJg/OTk5dje7U8cqeJxyOh1RaQMAAL2BrQGlpaVF99xzj2688UYlJydLkmpqapSRkRFwn8vlUlpammpqajr9nMWLF6uhocH82bt3r53NPik2aQMAIDKCfsTTXV6vV3//938vwzD07LPPhvVZbrdbbrfbopaFjm3uAQCIDFsCij+cfP7559qwYYM5eyJJWVlZOnDgQMD9bW1tqqurU1ZWlh3NsQwHBQIAEBmWP+Lxh5OPP/5Yb775ptLT0wOu5+Xlqb6+XlVVVeZ7GzZskM/n09ixY61ujqUOs0kbAAAREfQMSlNTk3bv3m2+rq6u1vbt25WWlqazzjpLN9xwg7Zt26aysjK1t7eb60rS0tKUkJCgYcOGaerUqbr11lu1fPlyeb1eFRcXa/bs2Z1W8MSSjotkAQCAfYIOKFu3btXEiRPN1yUlJZKkwsJCPfzww3r99dclSRdffHHA77311luaMGGCJOnll19WcXGxJk2aJKfTqVmzZunpp58O8StEjqfNf1AgMygAANgp6IAyYcIEGYZx0uunuuaXlpamVatWBfuno84/g8IaFAAA7MWziiCwBgUAgMggoASBMmMAACKDgBIEyowBAIgMAkoQzJ1kWSQLAICtCChBMMuMOSgQAABbMdIGwT+DksgMCgAAtiKgBIE1KAAARAYBJQicZgwAQGQQUIJAmTEAAJFBQAkCG7UBABAZBJQgmFvds0gWAABbEVCCYFbxUGYMAICtGGmDwEZtAABEBgElCJQZAwAQGQSUILBIFgCAyCCgdJNhGJQZAwAQIQSUbvK0+cx/swYFAAB7EVC6yV9iLFHFAwCA3Rhpu8m//iQ+ziFXHN0GAICdGGm7qYUFsgAARAwBpZsoMQYAIHIIKN3EJm0AAEQOAaWbzBJjFwEFAAC7EVC6yV/Fk8gMCgAAtiOgdNOxNSh0GQAAdmO07Sa2uQcAIHIIKN3koYoHAICIIaB0E2XGAABEDgGlmw63Hq3iYZEsAAC2I6B0U0vb0TUolBkDAGA7Ako3+cuMkxLoMgAA7MZo200trEEBACBiCCjdRJkxAACRQ0DpJk4zBgAgcggo3XT46Fk8POIBAMB+BJRuamnlNGMAACKFgNJNZpkxZ/EAAGA7RttuMk8z5hEPAAC2I6B0E1vdAwAQOQSUbqKKBwCAyCGgdFMLVTwAAEQMAaUbDMM49oiHKh4AAGxHQOkGb7uhdp8hiUc8AABEAgGlG/wlxhJlxgAARAKjbTf4N2lzOqSEOLoMAAC7Mdp2Q8cSY4fDEeXWAABw+iOgdAMnGQMAEFkElG7wlxgTUAAAiIygA8rGjRs1Y8YMZWdny+FwaM2aNQHXDcPQQw89pLPOOktJSUnKz8/Xxx9/HHBPXV2d5syZo+TkZKWmpmrevHlqamoK64vY6TAHBQIAEFFBB5Tm5maNHDlSy5Yt6/T6Y489pqefflrLly9XZWWl+vbtqylTpqilpcW8Z86cOdqxY4fWrVunsrIybdy4UfPnzw/9W9ishW3uAQCIKFewv1BQUKCCgoJOrxmGoSeffFIPPPCArr32WknSiy++qMzMTK1Zs0azZ8/Wzp07tXbtWm3ZskVjxoyRJC1dulTTpk3T448/ruzs7BM+1+PxyOPxmK8bGxuDbXZYjq1B4YkYAACRYOmIW11drZqaGuXn55vvpaSkaOzYsaqoqJAkVVRUKDU11QwnkpSfny+n06nKyspOP7e0tFQpKSnmT05OjpXN7hLn8AAAEFmWBpSamhpJUmZmZsD7mZmZ5rWamhplZGQEXHe5XEpLSzPvOd7ixYvV0NBg/uzdu9fKZneJk4wBAIisoB/xRIPb7Zbb7Y7a3/cvkmUGBQCAyLB0BiUrK0uSVFtbG/B+bW2teS0rK0sHDhwIuN7W1qa6ujrznljjaeMkYwAAIsnSgDJkyBBlZWVp/fr15nuNjY2qrKxUXl6eJCkvL0/19fWqqqoy79mwYYN8Pp/Gjh1rZXMsQ5kxAACRFfQjnqamJu3evdt8XV1dre3btystLU25ublasGCB/v3f/13nn3++hgwZogcffFDZ2dm67rrrJEnDhg3T1KlTdeutt2r58uXyer0qLi7W7NmzO63giQXsJAsAQGQFHVC2bt2qiRMnmq9LSkokSYWFhVq5cqXuvvtuNTc3a/78+aqvr9f48eO1du1aJSYmmr/z8ssvq7i4WJMmTZLT6dSsWbP09NNPW/B17EGZMQAAkeUwDMOIdiOC1djYqJSUFDU0NCg5Odn2v1fyP9v12rYvtLhgqP7pynNt/3sAAJyOghm/mRLoBnMnWdagAAAQEQSUbjDLjF0EFAAAIoGA0g3macbMoAAAEBEElG5gJ1kAACKLgNINLVTxAAAQUYy43cAMCgAAkUVA6QZOMwYAILIIKN3AVvcAAEQWAaUbzCoeZlAAAIgIAkoX2n2GWts5zRgAgEgioHTBv/5EIqAAABApBJQuHO4QUNwuugsAgEhgxO2Cf4Gs2+WU0+mIcmsAAOgdCChd8LRRwQMAQKQRULpwuJUFsgAARBoBpQuH2aQNAICII6B0gV1kAQCIPAJKF46dw0NXAQAQKYy6XWAGBQCAyCOgdME8h4eAAgBAxBBQumDOoFBmDABAxBBQunDYS5kxAACRRkDpwrEyY7oKAIBIYdTtgsfLGhQAACKNgNKFwwQUAAAijoDSBfOwQAIKAAARQ0DpAjMoAABEHgGlC/4yY04zBgAgcggop/CX/6vXOx9/JUnKTHZHuTUAAPQeBJSTOHCwRf/0qyp52ny6amiGrrwgI9pNAgCg1yCgdMLT1q7bX9qm/Q0tOvfMvnpy9sWKczqi3SwAAHoNAspxDMPQQ2t2qOrzb9Q/0aVf/mCMkhPjo90sAAB6FQLKcV6s+Fz/b+teOR3S0htH6Zwz+0W7SQAA9DoElA7e3f2V/q3sr5KkewuGasKFrDsBACAaXNFuQCxxOh1KTnRpwoUZuvXvzol2cwAA6LUIKB2MOyddb9wxXmf0c8vhYFEsAADRQkA5zsABfaLdBAAAej3WoAAAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzCCgAACDmEFAAAEDMIaAAAICYQ0ABAAAxh4ACAABiDgEFAADEHAIKAACIOT3yNGPDMCRJjY2NUW4JAADoLv+47R/HT6VHBpSDBw9KknJycqLcEgAAEKyDBw8qJSXllPc4jO7EmBjj8/m0b98+9e/fXw6Hw9LPbmxsVE5Ojvbu3avk5GRLPxuB6OvIoa8jh76OHPo6cqzqa8MwdPDgQWVnZ8vpPPUqkx45g+J0OjVw4EBb/0ZycjL/Bx8h9HXk0NeRQ19HDn0dOVb0dVczJ34skgUAADGHgAIAAGIOAeU4brdb//qv/yq32x3tppz26OvIoa8jh76OHPo6cqLR1z1ykSwAADi9MYMCAABiDgEFAADEHAIKAACIOQQUAAAQcwgoHSxbtkyDBw9WYmKixo4dq/feey/aTerxSktLdckll6h///7KyMjQddddp127dgXc09LSoqKiIqWnp6tfv36aNWuWamtro9Ti08ejjz4qh8OhBQsWmO/R19b54osv9P3vf1/p6elKSkrSiBEjtHXrVvO6YRh66KGHdNZZZykpKUn5+fn6+OOPo9jinqm9vV0PPvighgwZoqSkJJ177rn60Y9+FHCWC30duo0bN2rGjBnKzs6Ww+HQmjVrAq53p2/r6uo0Z84cJScnKzU1VfPmzVNTU1P4jTNgGIZhvPLKK0ZCQoLxwgsvGDt27DBuvfVWIzU11aitrY1203q0KVOmGCtWrDA+/PBDY/v27ca0adOM3Nxco6mpybzntttuM3Jycoz169cbW7duNcaNG2dcdtllUWx1z/fee+8ZgwcPNr797W8bd911l/k+fW2Nuro6Y9CgQcZNN91kVFZWGp9++qnxxz/+0di9e7d5z6OPPmqkpKQYa9asMf785z8b11xzjTFkyBDj8OHDUWx5z7NkyRIjPT3dKCsrM6qrq41XX33V6Nevn/HUU0+Z99DXofv9739v3H///cZrr71mSDJWr14dcL07fTt16lRj5MiRxubNm4133nnHOO+884wbb7wx7LYRUI669NJLjaKiIvN1e3u7kZ2dbZSWlkaxVaefAwcOGJKM8vJywzAMo76+3oiPjzdeffVV856dO3cakoyKiopoNbNHO3jwoHH++ecb69atM6688kozoNDX1rnnnnuM8ePHn/S6z+czsrKyjP/4j/8w36uvrzfcbrfx3//935Fo4mlj+vTpxi233BLw3syZM405c+YYhkFfW+n4gNKdvv3rX/9qSDK2bNli3vOHP/zBcDgcxhdffBFWe3jEI6m1tVVVVVXKz88333M6ncrPz1dFRUUUW3b6aWhokCSlpaVJkqqqquT1egP6fujQocrNzaXvQ1RUVKTp06cH9KlEX1vp9ddf15gxY/S9731PGRkZGjVqlH75y1+a16urq1VTUxPQ1ykpKRo7dix9HaTLLrtM69ev10cffSRJ+vOf/6xNmzapoKBAEn1tp+70bUVFhVJTUzVmzBjznvz8fDmdTlVWVob193vkYYFW++qrr9Te3q7MzMyA9zMzM/W3v/0tSq06/fh8Pi1YsECXX365LrroIklSTU2NEhISlJqaGnBvZmamampqotDKnu2VV17Rtm3btGXLlhOu0dfW+fTTT/Xss8+qpKRE9913n7Zs2aI777xTCQkJKiwsNPuzs/+m0NfBuffee9XY2KihQ4cqLi5O7e3tWrJkiebMmSNJ9LWNutO3NTU1ysjICLjucrmUlpYWdv8TUBAxRUVF+vDDD7Vp06ZoN+W0tHfvXt11111at26dEhMTo92c05rP59OYMWP04x//WJI0atQoffjhh1q+fLkKCwuj3LrTy//8z//o5Zdf1qpVq/Stb31L27dv14IFC5SdnU1fn+Z4xCPpjDPOUFxc3AnVDLW1tcrKyopSq04vxcXFKisr01tvvaWBAwea72dlZam1tVX19fUB99P3wauqqtKBAwf0ne98Ry6XSy6XS+Xl5Xr66aflcrmUmZlJX1vkrLPO0vDhwwPeGzZsmPbs2SNJZn/y35TwLVq0SPfee69mz56tESNGaO7cuVq4cKFKS0sl0dd26k7fZmVl6cCBAwHX29raVFdXF3b/E1AkJSQkaPTo0Vq/fr35ns/n0/r165WXlxfFlvV8hmGouLhYq1ev1oYNGzRkyJCA66NHj1Z8fHxA3+/atUt79uyh74M0adIkffDBB9q+fbv5M2bMGM2ZM8f8N31tjcsvv/yEcvmPPvpIgwYNkiQNGTJEWVlZAX3d2NioyspK+jpIhw4dktMZOFTFxcXJ5/NJoq/t1J2+zcvLU319vaqqqsx7NmzYIJ/Pp7Fjx4bXgLCW2J5GXnnlFcPtdhsrV640/vrXvxrz5883UlNTjZqammg3rUe7/fbbjZSUFOPtt9829u/fb/4cOnTIvOe2224zcnNzjQ0bNhhbt2418vLyjLy8vCi2+vTRsYrHMOhrq7z33nuGy+UylixZYnz88cfGyy+/bPTp08d46aWXzHseffRRIzU11fjtb39r/OUvfzGuvfZaSl9DUFhYaJx99tlmmfFrr71mnHHGGcbdd99t3kNfh+7gwYPG+++/b7z//vuGJONnP/uZ8f777xuff/65YRjd69upU6cao0aNMiorK41NmzYZ559/PmXGVlu6dKmRm5trJCQkGJdeeqmxefPmaDepx5PU6c+KFSvMew4fPmz88z//szFgwACjT58+xvXXX2/s378/eo0+jRwfUOhr67zxxhvGRRddZLjdbmPo0KHGL37xi4DrPp/PePDBB43MzEzD7XYbkyZNMnbt2hWl1vZcjY2Nxl133WXk5uYaiYmJxjnnnGPcf//9hsfjMe+hr0P31ltvdfrf6MLCQsMwute3X3/9tXHjjTca/fr1M5KTk42bb77ZOHjwYNhtcxhGh+34AAAAYgBrUAAAQMwhoAAAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzCCgAACDmEFAAAEDMIaAACMpNN90kh8Mhh8Oh+Ph4DRkyRHfffbdaWlq69ftvv/22HA7HCacqA0BHrmg3AEDPM3XqVK1YsUJer1dVVVUqLCyUw+HQT37yk4i2w+v1Kj4+PqJ/E0BkMIMCIGhut1tZWVnKycnRddddp/z8fK1bt06S5PP5VFpaqiFDhigpKUkjR47Ur3/9a0nSZ599pokTJ0qSBgwYIIfDoZtuukmSNHjwYD355JMBf+fiiy/Www8/bL52OBx69tlndc0116hv375asmSJHn74YV188cX61a9+pcGDByslJUWzZ8/WwYMHzd/79a9/rREjRigpKUnp6enKz89Xc3OzfR0EIGwEFABh+fDDD/Xuu+8qISFBklRaWqoXX3xRy5cv144dO7Rw4UJ9//vfV3l5uXJycvSb3/xGkrRr1y7t379fTz31VFB/7+GHH9b111+vDz74QLfccosk6ZNPPtGaNWtUVlamsrIylZeX69FHH5Uk7d+/XzfeeKNuueUW7dy5U2+//bZmzpwpzkkFYhuPeAAEraysTP369VNbW5s8Ho+cTqd+/vOfy+Px6Mc//rHefPNN5eXlSZLOOeccbdq0Sc8995yuvPJKpaWlSZIyMjKUmpoa9N/+x3/8R918880B7/l8Pq1cuVL9+/eXJM2dO1fr16/XkiVLtH//frW1tWnmzJkaNGiQJGnEiBFhfHsAkUBAARC0iRMn6tlnn1Vzc7OeeOIJuVwuzZo1Szt27NChQ4f03e9+N+D+1tZWjRo1ypK/PWbMmBPeGzx4sBlOJOmss87SgQMHJEkjR47UpEmTNGLECE2ZMkWTJ0/WDTfcoAEDBljSHgD2IKAACFrfvn113nnnSZJeeOEFjRw5Us8//7wuuugiSdLvfvc7nX322QG/43a7T/mZTqfzhMcuXq+30799vOMXyjocDvl8PklSXFyc1q1bp3fffVd/+tOftHTpUt1///2qrKzUkCFDuvimAKKFNSgAwuJ0OnXffffpgQce0PDhw+V2u7Vnzx6dd955AT85OTmSZK5VaW9vD/icM888U/v37zdfNzY2qrq62pI2OhwOXX755XrkkUf0/vvvKyEhQatXr7bkswHYgxkUAGH73ve+p0WLFum5557Tv/zLv2jhwoXy+XwaP368Ghoa9L//+79KTk5WYWGhBg0aJIfDobKyMk2bNk1JSUnq16+frrrqKq1cuVIzZsxQamqqHnroIcXFxYXdtsrKSq1fv16TJ09WRkaGKisr9eWXX2rYsGEWfHMAdiGgAAiby+VScXGxHnvsMVVXV+vMM89UaWmpPv30U6Wmpuo73/mO7rvvPknS2WefrUceeUT33nuvbr75Zv3gBz/QypUrtXjxYlVXV+vqq69WSkqKfvSjH1kyg5KcnKyNGzfqySefVGNjowYNGqSf/vSnKigoCPuzAdjHYVBrBwAAYgxrUAAAQMwhoAAAgJhDQAEAADGHgAIAAGIOAQUAAMQcAgoAAIg5BBQAABBzCCgAACDmEFAAAEDMIaAAAICYQ0ABAAAx5/8DJW9hIVRGyIYAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Model\n", "features = ts.utils.net.common.Net(\n", " state_shape=env.observation_space.shape, \n", " hidden_sizes=[64, 64], \n", " device=device)\n", "\n", "actor = ts.utils.net.discrete.Actor(\n", " preprocess_net=features, \n", " action_shape=env.action_space.n, \n", " device=device).to(device)\n", "\n", "critic = ts.utils.net.discrete.Critic(\n", " preprocess_net=features, \n", " device=device).to(device)\n", "\n", "actor_critic = ts.utils.net.common.ActorCritic(actor=actor, critic=critic)\n", "\n", "optim = torch.optim.Adam(actor_critic.parameters(), lr=0.003)\n", "\n", "# Policy\n", "policy = ts.policy.PPOPolicy(\n", " actor=actor, \n", " critic=critic, \n", " optim=optim,\n", " dist_fn=torch.distributions.Categorical, \n", " action_space=env.action_space,\n", " deterministic_eval=True,\n", " action_scaling=False,\n", " discount_factor=0.99,\n", " max_grad_norm=0.5,\n", " eps_clip=0.2,\n", " gae_lambda=0.95,\n", ")\n", "\n", "# Collector\n", "collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]), \n", " buffer=ts.data.VectorReplayBuffer(20000, 10),\n", " exploration_noise=True\n", ")\n", "test_collector = ts.data.Collector(\n", " policy=policy, \n", " env=ts.env.DummyVectorEnv([lambda: gym.make('CartPole-v0') for _ in range(10)]),\n", " exploration_noise=False\n", ")\n", "\n", "# On-policy trainer\n", "returns = []\n", "for iteration in range(100):\n", "\n", " policy.train() \n", "\n", " # Collect enough on-policy steps\n", " result = collector.collect(n_step=3000)\n", "\n", " # Update the PP0 network by learning the on-policy buffer 10 times\n", " policy.update(\n", " buffer=collector.buffer,\n", " sample_size=0, # use the whole buffer\n", " batch_size=256,\n", " repeat=10,\n", " )\n", "\n", " # Empty the buffer as we are on-policy\n", " collector.reset_buffer(keep_statistics=False)\n", "\n", " # Test 10 episodes\n", " policy.eval()\n", " result = test_collector.collect(n_episode=10)\n", " mean_reward = result['rew']\n", " #print(iteration, \":\", mean_reward)\n", " returns.append(mean_reward)\n", "\n", "plt.figure()\n", "plt.plot(np.array(returns))\n", "plt.xlabel(\"Epochs\")\n", "plt.ylabel(\"Returns\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Return: 200.0\n", "MoviePy - Building file videos/cartpole-ppo.gif with imageio.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ " \r" ] }, { "data": { "text/html": [ "
" ], "text/plain": [ "" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" }, { "ename": "", "evalue": "", "output_type": "error", "traceback": [ "\u001b[1;31mThe Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details." ] } ], "source": [ "# Evaluation mode\n", "policy.eval() \n", "\n", "# Create a recordable environment\n", "env = gym.make('CartPole-v0', render_mode=\"rgb_array_list\")\n", "recorder = GymRecorder(env)\n", "\n", "# Sample the initial state\n", "state, info = env.reset()\n", "\n", "# One episode:\n", "done = False\n", "return_episode = 0\n", "while not done:\n", " # Select an action from the learned policy\n", " action = policy.forward(ts.data.Batch(obs=[state], info=None)).act[0].numpy()\n", " # Sample a single transition\n", " next_state, reward, terminal, truncated, info = env.step(action)\n", " # End of the episode\n", " done = terminal or truncated\n", " # Update undiscounted return\n", " return_episode += reward\n", " # Go in the next state\n", " state = next_state\n", "\n", "print(\"Return:\", return_episode)\n", "\n", "recorder.record(env.render())\n", "video = \"videos/cartpole-ppo.gif\"\n", "recorder.make_video(video)\n", "ipython_display(video)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q:** How does it compare to DQN once the right hyperparameters are found? What is their influence? Play especially with `eps_clip`, the $\\epsilon$ threshold used to clip the IS weight in the PPO loss.\n", "\n", "**Q:** Apply PPO to more complex environments available in gymnasium. Beware that for continuous action spaces, you will need to use continuous actor/critic networks, i.e.:\n", "\n", "```python\n", "actor = ts.utils.net.continuous.Actor(...)\n", "critic = ts.utils.net.continuous.Critic(...)\n", "```\n", "\n", "instead of:\n", "\n", "```python\n", "actor = ts.utils.net.discrete.Actor(...)\n", "critic = ts.utils.net.discrete.Critic(...)\n", "```\n", "\n", "The `dist_fn` argument of `PPOPolicy` must also be set accordingly. Check the doc!" ] } ], "metadata": { "kernelspec": { "display_name": "tianshou", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.7" } }, "nbformat": 4, "nbformat_minor": 2 }