{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this first exercise, we will investigate how to evaluate the Q-value of each action available in a 5-armed bandit. It is mostly to give you intuition about the limits of sampling and the central limit theorem.\n",
    "\n",
    "Let's start with importing numpy and matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling a n-armed bandit\n",
    "\n",
    "Let's now create the n-armed bandit. The only thing we need to do is to randomly choose 5 true Q-values $Q^*(a)$.\n",
    "\n",
    "![](../img/bandit-example.png)\n",
    "\n",
    "To be generic, let's define `nb_actions=5` and create an array corresponding to the index of each action (0, 1, 2, 3, 4) for plotting purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_actions = 5\n",
    "actions = np.arange(nb_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Create a numpy array `Q_star` with `nb_actions` values, normally distributed with a mean of 0 and standard deviation of 1 (as in the lecture).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng()\n",
    "Q_star = rng.normal(0, 1, nb_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Plot the Q-values. Identify the optimal action $a^*$.\n",
    "\n",
    "*Tip:* you could plot the array `Q_star` with `plt.plot`, but that would be ugly. Check the documentation of the `plt.bar` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal action: 0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAFzCAYAAACHCIXLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAATiUlEQVR4nO3df6zld13n8debmXaXX1kWZ5DSTpmanexaMaJOKiy6KwimP4gFdElLVGTJTjBUJW5ixhj3VzabJhh1N6LNqA0gLpWkVhs7Wn5s12pE6LS0pUNbndTRTqbZDlYQBIoD7/3jnoHbO3dm7rT3nu/59D4eyc39/uq575vT6Tz7Pd/zPdXdAQBg8T1j6gEAAFgb4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwiK1TDzAP27Zt6507d049BgDAGd15552f7u7tq+1buHCrquuTvDbJo939klX2V5L/meTyJF9I8mPdfdfpHnPnzp05cODARowLALCuquqvT7VvEV8qfXeSS0+z/7Iku2Zfe5L82hxmAgCY3MKFW3ffnuSx0xxyZZL39pI/T/K8qjpvPtMBAExn4cJtDc5P8vCy9SOzbU9QVXuq6kBVHTh27NjchgMA2Cgjhlutsu2kD1zt7n3dvbu7d2/fvur1fQAAQxkx3I4k2bFs/YIkRyeaBQBgbkYMt5uT/GgteVmSz3b3I1MPBQCw0RbxdiDvT/K9SbZV1ZEk/znJOUnS3dcl2Z+lW4EcytLtQN4yzaQAAPO1cOHW3VefYX8nefucxgEAWBgjvlQKALApCTcAgEEINwCAQQg3AIBBLNybE0a2c+8tU4/wtHL42iumHgEAFoozbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAINYuHCrqkur6sGqOlRVe1fZ/71V9dmqunv29Z+mmBMAYN62Tj3AclW1Jcm7krwmyZEkd1TVzd39qRWH/kl3v3buAwIATGjRzrhdkuRQdz/U3V9OckOSKyeeCQBgISxauJ2f5OFl60dm21Z6eVXdU1V/WFXfMp/RAACmtVAvlSapVbb1ivW7kry4uz9fVZcn+b0ku056oKo9SfYkyYUXXrjOYwIAzN+inXE7kmTHsvULkhxdfkB3/313f362vD/JOVW1beUDdfe+7t7d3bu3b9++kTMDAMzFooXbHUl2VdVFVXVukquS3Lz8gKp6YVXVbPmSLP0Ofzv3SQEA5myhXirt7uNVdU2SW5NsSXJ9dx+sqrfN9l+X5IeS/HhVHU/yxSRXdffKl1MBAJ52Firckq+9/Ll/xbbrli3/SpJfmfdcAABTW7SXSgEAOAXhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADCIrVMPAMCYdu69ZeoRnlYOX3vF1CMwAGfcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAaxcOFWVZdW1YNVdaiq9q6yv6rqf83231tV3zHFnAAA87ZQ4VZVW5K8K8llSS5OcnVVXbzisMuS7Jp97Unya3MdEgBgIgsVbkkuSXKoux/q7i8nuSHJlSuOuTLJe3vJnyd5XlWdN+9BAQDmbdHC7fwkDy9bPzLbdrbHAAA87WydeoAVapVt/SSOSVXtydJLqbnwwguf+mRrcPjaK+byc3jydu69ZeoRnnY24t97z9P624jnyX/zFp8/S+tv6n/vn9QZt6p69ux6tPV2JMmOZesXJDn6JI5Jd+/r7t3dvXv79u3rPigAwLytKdyq6hlV9aaquqWqHk3yQJJHqupgVb2zqnat0zx3JNlVVRdV1blJrkpy84pjbk7yo7N3l74syWe7+5F1+vkAAAtrrS+V3pbkw0l+Nsl93f3VJKmq5yd5ZZJrq+qm7n7fUxmmu49X1TVJbk2yJcn13X2wqt42239dkv1JLk9yKMkXkrzlqfxMAIBRrDXcXt3d/7hyY3c/luTGJDdW1TnrMVB3789SnC3fdt2y5U7y9vX4WQAAI1lTuJ2Itqr6hiRvTPKlJAeTfLK7v7j8GAAANsbZvjnhpiTbk/yPJO9M8tmqemDdpwIA4CRnezuQ53b3f6uqN3T3v62qH0zyLzZiMAAAnuhsz7h9afb98ap6ZnffmKU3CgAAsMHO9ozbL8zeSfo7Sa6vqj+LTy0AAJiLszrj1t03dvdj3f2LWXrn546c/FmiAABsgDWdcauqmt2G42u6+7fOdAwAAOtnrWfcbquqn6iqJ3zoZ1WdW1Wvqqr3JHnz+o8HAMAJa73G7dIk/z7J+6vqoiSfSfLMLIXfB5P8UnffvREDAgCwZK034P1Skl9N8quzT0jYluSL3f2ZDZwNAIBlzvZdpSc+IcGHugMAzNkZr3GrqmfPvj9n48cBAOBU1vLmhH9eVdck+e6NHgYAgFNbS7h9X5IfS/JNVfWCjR0HAIBTWcs1bh/P0jtKd3T3oxs8DwAAp3DGcOvu+2eL927wLAAAnMbZfsg8AAATWfPtQKrqRVm63u1ZSR7o7j/esKlggxy+9oqpRwCAJ21NZ9yq6vuT3Jnk8iQvT/LLVfVgVf3rjRwOAICvW+sZt/+e5Hu6+9CJDVX18iS/XlVvTfIP3X3fRgwIAMCStYbbucujLUm6+6NV9YYkf5Dk8STfut7DAQDwdWt9c8KXqmr7yo3d/RdJvpKla98AANhAaw23dyb5vdkbFL6mqrYledz93QAANt6aXirt7hur6p8k+WhV3ZnkniTnJnljlq5/AwBgg635Pm7d/b+TfHOWrmn7Z0n+Mcmbuvs9GzQbAADLrPk+bknS3V9Icv0GzQIAwGn45AQAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQWydegAAYGMcvvaKqUdgnTnjBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwiK1TD3BCVT0/ye8k2ZnkcJI3dvffrXLc4SSfS/KVJMe7e/f8pgQAmM4inXHbm+Qj3b0ryUdm66fyyu5+qWgDADaTRQq3K5O8Z7b8niSvm24UAIDFs0jh9o3d/UiSzL6/4BTHdZIPVtWdVbXnVA9WVXuq6kBVHTh27NgGjAsAMF9zvcatqj6c5IWr7Pq5s3iYV3T30ap6QZIPVdUD3X37yoO6e1+SfUmye/fuflIDAwAskLmGW3e/+lT7qur/VdV53f1IVZ2X5NFTPMbR2fdHq+qmJJckOSncAACebhbppdKbk7x5tvzmJL+/8oCqenZVPffEcpLvT3Lf3CYEAJjQIoXbtUleU1V/meQ1s/VU1Yuqav/smG9M8qdVdU+Sjye5pbv/aJJpAQDmbGHu49bdf5vk+1bZfjTJ5bPlh5J825xHAwBYCIt0xg0AgNMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDWJhwq6p/V1UHq+qrVbX7NMddWlUPVtWhqto7zxkBAKa0MOGW5L4kb0hy+6kOqKotSd6V5LIkFye5uqouns94AADT2jr1ACd09/1JUlWnO+ySJIe6+6HZsTckuTLJpzZ8QACAiS3SGbe1OD/Jw8vWj8y2naSq9lTVgao6cOzYsbkMBwCwkeZ6xq2qPpzkhavs+rnu/v21PMQq23q1A7t7X5J9SbJ79+5VjwEAGMlcw627X/0UH+JIkh3L1i9IcvQpPiYAwBBGe6n0jiS7quqiqjo3yVVJbp54JgCAuViYcKuq11fVkSQvT3JLVd062/6iqtqfJN19PMk1SW5Ncn+SD3T3walmBgCYp0V6V+lNSW5aZfvRJJcvW9+fZP8cRwMAWAgLc8YNAIDTE24AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIPYOvUAACsdvvaKqUcAWEjOuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMorp76hk2XFUdS/LXU8+xQLYl+fTUQ3BGnqfF5zkag+dpDJ6nr3txd29fbcemCDeeqKoOdPfuqefg9DxPi89zNAbP0xg8T2vjpVIAgEEINwCAQQi3zWnf1AOwJp6nxec5GoPnaQyepzVwjRsAwCCccQMAGIRw20Sq6tKqerCqDlXV3qnnYXVVdX1VPVpV9009C6urqh1VdVtV3V9VB6vqp6aeiZNV1T+tqo9X1T2z5+m/Tj0Tq6uqLVX1iar6g6lnWXTCbZOoqi1J3pXksiQXJ7m6qi6edipO4d1JLp16CE7reJL/2N3fnORlSd7uz9NCejzJq7r725K8NMmlVfWyaUfiFH4qyf1TDzEC4bZ5XJLkUHc/1N1fTnJDkisnnolVdPftSR6beg5Orbsf6e67Zsufy9JfOOdPOxUr9ZLPz1bPmX25sHvBVNUFSa5I8htTzzIC4bZ5nJ/k4WXrR+IvGnjKqmpnkm9P8rGJR2EVs5fg7k7yaJIPdbfnafH8cpKfSfLViecYgnDbPGqVbf7PE56CqnpOkhuTvKO7/37qeThZd3+lu1+a5IIkl1TVSyYeiWWq6rVJHu3uO6eeZRTCbfM4kmTHsvULkhydaBYYXlWdk6Vo++3u/t2p5+H0uvszSf5vXD+6aF6R5Aeq6nCWLuF5VVW9b9qRFptw2zzuSLKrqi6qqnOTXJXk5olngiFVVSX5zST3d/cvTj0Pq6uq7VX1vNnyM5O8OskDkw7FE3T3z3b3Bd29M0t/L/2f7v7hicdaaMJtk+ju40muSXJrli6k/kB3H5x2KlZTVe9P8tEk/7KqjlTVW6eeiZO8IsmPZOnswN2zr8unHoqTnJfktqq6N0v/8/qh7na7CYbmkxMAAAbhjBsAwCCEGwDAIIQbAMAghBsAwCCEGwDAIIQbsGlV1eurqqvqX53huHdU1bOWre8/cX8wgHlyOxBg06qqD2TpXl8f6e7/cprjDifZ3d2fntNoAKtyxg3YlGafM/qKJG/N0h3bT3wg+S9U1Ser6t6q+omq+skkL8rSjVxvmx13uKq2zZZ/uqrum329Y7ZtZ1XdX1W/XlUHq+qDszv3p6p+sqo+NXv8G+b/mwMj2zr1AAATeV2SP+ruv6iqx6rqO5J8V5KLknx7dx+vqud392NV9dNJXrnyjFtVfWeSt8z+uUrysar64yR/l2RXkqu7+z/Mzuz9YJL3Jdmb5KLuftzLrcDZcsYN2KyuztKHWmf2/eosfZbldbOPiEt3P3aGx/juJDd19z909+eT/G6S75nt+6vuvnu2fGeSnbPle5P8dlX9cJLj6/B7AJuIM27AplNV35DkVUleUlWdZEuSzlJgnc2Fv3WafY8vW/5KkmfOlq9I8m+S/ECSn6+qbzkRigBn4owbsBn9UJL3dveLu3tnd+9I8ldJ7krytqramiRV9fzZ8Z9L8txVHuf2JK+rqmdV1bOTvD7Jn5zqh1bVM5Ls6O7bkvxMkuclec46/U7AJiDcgM3o6iQ3rdh2Y5behPA3Se6tqnuSvGm2b1+SPzzx5oQTuvuuJO9O8vEkH0vyG939idP83C1J3ldVn0zyiSS/1N2feWq/CrCZuB0IAMAgnHEDABiEcAMAGIRwAwAYhHADABiEcAMAGIRwAwAYhHADABiEcAMAGMT/B0SA8CHCVTUCAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Optimal action:\", Q_star.argmax())\n",
    "\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.bar(actions, Q_star)\n",
    "plt.xlabel('Actions')\n",
    "plt.ylabel('$Q^*(a)$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now let's start evaluating these Q-values with random sampling.\n",
    "\n",
    "**Q:** Define an action sampling method `get_reward` taking as arguments:\n",
    "* The array `Q_star`.\n",
    "* The index `a` of the action you want to sample (between 0 and 4).\n",
    "* An optional variance argument `var`, which should have the value 1.0 by default.\n",
    "   \n",
    "It should return a single value, sampled from the normal distribution with mean `Q_star[a]` and variance `var`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_reward(Q_star, a, var=1.0):\n",
    "    return float(rng.normal(Q_star[a], var, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** For each possible action `a`, take `nb_samples=10` out of the reward distribution and store them in a numpy array. Compute the mean of the samples for each action separately in a new array `Q_t`. Make a bar plot of these estimated Q-values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnMAAAFzCAYAAABVWI+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYjklEQVR4nO3df6yld10n8PfHGboLSMTSAUt/OI07UasrFScVU38VKOkPdWA1pnXFhsWdsKEqataM2azrbjabRlQ2RqQZtVIXpOumVCYwArWLi66KnWLpD0plttZlnAkdqICIgGU/+8c9g8fbO3fubefe83xvX6/k5DzP98c5nydPp/Oe73POc6q7AwDAmL5k0QUAAPD4CXMAAAMT5gAABibMAQAMTJgDABiYMAcAMLDtiy5gkc4666zeuXPnossAADilO++882PdvWN5+5M6zO3cuTOHDh1adBkAAKdUVX+5UrvLrAAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAY2PZFF7DV7dz3jkWXsKU8dP1Viy4BACbFyhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGCTCXNVdWNVPVxV956kv6rql6rqcFXdXVXPn+u7vKoemPXt27yqAQAWazJhLskbk1y+Sv8VSXbNHnuTvCFJqmpbktfP+i9Mck1VXbihlQIATMRkwlx3vzfJI6sM2ZPkN3vJnyR5ZlWdneTiJIe7+8Hu/nySm2djAQC2vMmEuTU4J8lH5vaPzNpO1g4AsOWNFOZqhbZepX3lF6naW1WHqurQ8ePHT1txAACLMFKYO5LkvLn9c5McXaV9Rd29v7t3d/fuHTt2bEihAACbZaQwdyDJD82+1fqCJJ/s7mNJ7kiyq6ouqKozklw9GwsAsOVtX3QBJ1TVW5J8Z5KzqupIkv+Q5ClJ0t03JDmY5Mokh5N8JskrZn2PVtV1Sd6VZFuSG7v7vk0/AACABZhMmOvua07R30lefZK+g1kKewAATyojXWYFAGAZYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABjapMFdVl1fVA1V1uKr2rdD/b6vqrtnj3qr6QlWdOet7qKrumfUd2vzqAQA23/ZFF3BCVW1L8voklyU5kuSOqjrQ3R88Maa7X5vktbPx353kx7v7kbmXubS7P7aJZQMALNSUVuYuTnK4ux/s7s8nuTnJnlXGX5PkLZtSGQDARE0pzJ2T5CNz+0dmbY9RVU9LcnmSW+aaO8m7q+rOqtp7sjepqr1VdaiqDh0/fvw0lA0AsDhTCnO1QlufZOx3J/nfyy6xXtLdz09yRZJXV9W3rzSxu/d39+7u3r1jx44nVjEAwIJNKcwdSXLe3P65SY6eZOzVWXaJtbuPzp4fTnJrli7bAgBsaVMKc3ck2VVVF1TVGVkKbAeWD6qqL0vyHUneNtf29Kp6xontJC9Jcu+mVA0AsECT+TZrdz9aVdcleVeSbUlu7O77qupVs/4bZkNfluTd3f23c9Ofk+TWqkqWjum3uvudm1c9AMBiTCbMJUl3H0xycFnbDcv235jkjcvaHkzyvA0uDwBgcqZ0mRUAgHUS5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDbF10AAFvHzn3vWHQJW85D11+16BKYOCtzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGCTCnNVdXlVPVBVh6tq3wr931lVn6yqu2aPn1nrXACArWgy95mrqm1JXp/ksiRHktxRVQe6+4PLhv5Bd3/X45wLALClTGll7uIkh7v7we7+fJKbk+zZhLkAAMOaUpg7J8lH5vaPzNqW+5aq+kBV/W5Vfd065wIAbCmTucyapFZo62X770/yld396aq6MsnvJNm1xrlLb1K1N8neJDn//PMfd7EAAFMwpZW5I0nOm9s/N8nR+QHd/anu/vRs+2CSp1TVWWuZO/ca+7t7d3fv3rFjx+msHwBg000pzN2RZFdVXVBVZyS5OsmB+QFV9RVVVbPti7NU/8fXMhcAYCuazGXW7n60qq5L8q4k25Lc2N33VdWrZv03JPm+JP+mqh5N8ndJru7uTrLi3IUcCADAJppMmEu+eOn04LK2G+a2fznJL691LgDAVjely6wAAKyTMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAA5tUmKuqy6vqgao6XFX7Vuj/l1V19+zxR1X1vLm+h6rqnqq6q6oObW7lAACLsX3RBZxQVduSvD7JZUmOJLmjqg509wfnhv1Fku/o7r+uqiuS7E/yzXP9l3b3xzataACABZvSytzFSQ5394Pd/fkkNyfZMz+gu/+ou/96tvsnSc7d5BoBACZlSmHunCQfmds/Mms7mVcm+d25/U7y7qq6s6r2bkB9AACTM5nLrElqhbZecWDVpVkKc98613xJdx+tqmcnua2qPtTd711h7t4ke5Pk/PPPf+JVAwAs0JRW5o4kOW9u/9wkR5cPqqpvSPJrSfZ098dPtHf30dnzw0luzdJl28fo7v3dvbu7d+/YseM0lg8AsPmmFObuSLKrqi6oqjOSXJ3kwPyAqjo/yVuTvLy7/3yu/elV9YwT20lekuTeTascAGBBJnOZtbsfrarrkrwrybYkN3b3fVX1qln/DUl+JsmzkvxKVSXJo929O8lzktw6a9ue5Le6+50LOAwAgE01mTCXJN19MMnBZW03zG3/cJIfXmHeg0met7wdAGCrm9JlVgAA1kmYAwAYmDAHADAwYQ4AYGDrDnOz24Bs24hiAABYn1OGuar6kqr6gap6R1U9nORDSY5V1X1V9dqq2rXxZQIAsJK1rMy9J8lXJfnpJF/R3ed197OTfFuWfuz++qr6wQ2sEQCAk1jLfeZe3N1/v7yxux9JckuSW6rqKae9MgAATumUYe5EkKuqZyX5/iSfTXJfknu6++/mxwAAsLnW8wWIW5PsSPJfkrw2ySer6kMbUhUAAGuynp/zekZ3/6eq+hfd/R1V9b1J/tlGFQYAwKmtZ2Xus7Pnz1XVU7v7liRXbkBNAACs0XpW5n6+qs5M8t+T3FhVf5TknI0pCwCAtVjzylx339Ldj3T3LyY5mOS8JHs2rDIAAE7plCtzVVXd3fNt3f3fTjUGAICNt6abBlfVj1TV+fONVXVGVb2wqm5Kcu3GlAcAwGrW8pm5y5P8qyRvqaoLknwiyVOzFATfneR13X3XRhUIAMDJreWmwZ9N8itJfmX2Sw9nJfm77v7EBtcGAMAprOfWJOnuv+/uY1laqUuSVNVXn/aqAABYk/XcmiRV9cwkr0vyNVX12SR3J3llklec/tIAADiVNYW5qnpukhcleVqSN2bpM3PHk3xDkrduVHEAAKxuLbcmeUmSm5L8fpLPJXlVlkLdK7r7Nza0OgAAVrWWlbn/nOTbuvvwiYaq+pYk+6vqlUk+0933blSBAACc3FrC3BnzQS5JuvuPq+p7k7w9S6t1/3wjigMAYHVr+TbrZ6tqx/LG7v7zJF/I0mfpAABYgLWEudcm+Z3ZlyC+qKrOSvK57n54QyoDAOCU1nLT4Fuq6p8k+eOqujPJB5KckeT7s/R5OgAAFmRNNw3u7t9K8rVZ+ozclyX5+yQ/0N03bWBtAACcwppvGtzdn0ly4wbWAgDAOq3r57wAAJiWSYW5qrq8qh6oqsNVtW+F/qqqX5r1311Vz1/rXACArWgyYa6qtiV5fZIrklyY5JqqunDZsCuS7Jo99iZ5wzrmAgBsOZMJc0kuTnK4ux/s7s8nuTnJnmVj9iT5zV7yJ0meWVVnr3EuAMCWs+YvQGyCc5J8ZG7/SJJvXsOYc9Y4N0lSVXuztKqX888//4lVvAYPXX/Vhr8HT9zOfe9YdAlbykb8d+8cnX4bcZ78P28M/jydXov+735KK3O1Qluvccxa5i41du/v7t3dvXvHjsf8sAUAwFCmtDJ3JMl5c/vnJjm6xjFnrGEuAMCWM6WVuTuS7KqqC6rqjCRXJzmwbMyBJD80+1brC5J8sruPrXEuAMCWM5mVue5+tKquS/KuJNuS3Njd91XVq2b9NyQ5mOTKJIeTfCbJK1abu4DDAADYVJMJc0nS3QezFNjm226Y2+4kr17rXACArW5Kl1kBAFgnYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIFtX3QBAMDmeuj6qxZdAqeRlTkAgIEJcwAAAxPmAAAGJswBAAzMFyAgPgwMwLiszAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMLBJhLmqOrOqbquqD8+ev3yFMedV1Xuq6v6quq+qfmyu72er6q+q6q7Z48rNPQIAgMWYRJhLsi/J7d29K8nts/3lHk3yk939tUlekOTVVXXhXP/ruvui2ePgxpcMALB4Uwlze5LcNNu+KclLlw/o7mPd/f7Z9t8kuT/JOZtVIADAFE0lzD2nu48lS6EtybNXG1xVO5N8Y5L3zTVfV1V3V9WNK12mBQDYijYtzFXV71XVvSs89qzzdb40yS1JXtPdn5o1vyHJVyW5KMmxJL+wyvy9VXWoqg4dP3788R0MAMBEbN+sN+ruF5+sr6o+WlVnd/exqjo7ycMnGfeULAW5N3f3W+de+6NzY341ydtXqWN/kv1Jsnv37l73gQAATMhULrMeSHLtbPvaJG9bPqCqKsmvJ7m/u39xWd/Zc7svS3LvBtUJADApUwlz1ye5rKo+nOSy2X6q6rlVdeKbqZckeXmSF65wC5Kfq6p7quruJJcm+fFNrh8AYCE27TLrarr740letEL70SRXzrb/MEmdZP7LN7RAAICJmsrKHAAAj4MwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADm0SYq6ozq+q2qvrw7PnLTzLuoaq6p6ruqqpD650PALDVTCLMJdmX5Pbu3pXk9tn+yVza3Rd19+7HOR8AYMuYSpjbk+Sm2fZNSV66yfMBAIY0lTD3nO4+liSz52efZFwneXdV3VlVex/H/FTV3qo6VFWHjh8/fprKBwBYjO2b9UZV9XtJvmKFrn+3jpe5pLuPVtWzk9xWVR/q7veup47u3p9kf5Ls3r271zMXAGBqNi3MdfeLT9ZXVR+tqrO7+1hVnZ3k4ZO8xtHZ88NVdWuSi5O8N8ma5gMAbDVTucx6IMm1s+1rk7xt+YCqenpVPePEdpKXJLl3rfMBALaiqYS565NcVlUfTnLZbD9V9dyqOjgb85wkf1hVH0jyp0ne0d3vXG0+AMBWt2mXWVfT3R9P8qIV2o8muXK2/WCS561nPgDAVjeVlTkAAB4HYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABibMAQAMTJgDABiYMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICBCXMAAAMT5gAABjaJMFdVZ1bVbVX14dnzl68w5qur6q65x6eq6jWzvp+tqr+a67ty0w8CAGABJhHmkuxLcnt370py+2z/H+nuB7r7ou6+KMk3JflMklvnhrzuRH93H9yMogEAFm0qYW5Pkptm2zcleekpxr8oyf/p7r/cyKIAAKZuKmHuOd19LElmz88+xfirk7xlWdt1VXV3Vd240mXaE6pqb1UdqqpDx48ff2JVAwAs2KaFuar6vaq6d4XHnnW+zhlJvifJ/5hrfkOSr0pyUZJjSX7hZPO7e3937+7u3Tt27Fj/gQAATMj2zXqj7n7xyfqq6qNVdXZ3H6uqs5M8vMpLXZHk/d390bnX/uJ2Vf1qkrefjpoBAKZuKpdZDyS5drZ9bZK3rTL2miy7xDoLgCe8LMm9p7U6AICJmkqYuz7JZVX14SSXzfZTVc+tqi9+M7Wqnjbrf+uy+T9XVfdU1d1JLk3y45tTNgDAYm3aZdbVdPfHs/QN1eXtR5NcObf/mSTPWmHcyze0QACAiZrKyhwAAI+DMAcAMDBhDgBgYMIcAMDAhDkAgIEJcwAAAxPmAAAGJswBAAxsEjcNBjiVh66/atElAEySlTkAgIEJcwAAAxPmAAAGJswBAAxMmAMAGJgwBwAwMGEOAGBgwhwAwMCEOQCAgQlzAAADE+YAAAYmzAEADEyYAwAYmDAHADCw6u5F17AwVXU8yV8uuo6JOCvJxxZdBKfkPI3BeZo+52gMztM/9pXdvWN545M6zPEPqupQd+9edB2sznkag/M0fc7RGJyntXGZFQBgYMIcAMDAhDlO2L/oAlgT52kMztP0OUdjcJ7WwGfmAAAGZmUOAGBgwhypqsur6oGqOlxV+xZdD49VVTdW1cNVde+ia2FlVXVeVb2nqu6vqvuq6scWXROPVVX/tKr+tKo+MDtP/3HRNbGyqtpWVX9WVW9fdC1TJ8w9yVXVtiSvT3JFkguTXFNVFy62KlbwxiSXL7oIVvVokp/s7q9N8oIkr/ZnaZI+l+SF3f28JBclubyqXrDYkjiJH0ty/6KLGIEwx8VJDnf3g939+SQ3J9mz4JpYprvfm+SRRdfByXX3se5+/2z7b7L0l9A5i62K5XrJp2e7T5k9fHh8Yqrq3CRXJfm1RdcyAmGOc5J8ZG7/SPwFBE9IVe1M8o1J3rfgUljB7PLdXUkeTnJbdztP0/Nfk/xUkv+34DqGIMxRK7T5Vyo8TlX1pUluSfKa7v7Uouvhsbr7C919UZJzk1xcVV+/4JKYU1XfleTh7r5z0bWMQpjjSJLz5vbPTXJ0QbXA0KrqKVkKcm/u7rcuuh5W192fSPL78XnUqbkkyfdU1UNZ+ujPC6vqTYstadqEOe5IsquqLqiqM5JcneTAgmuC4VRVJfn1JPd39y8uuh5WVlU7quqZs+2nJnlxkg8ttCj+ke7+6e4+t7t3ZunvpP/Z3T+44LImTZh7kuvuR5Ncl+RdWfrA9m93932LrYrlquotSf44yVdX1ZGqeuWia+IxLkny8iytItw1e1y56KJ4jLOTvKeq7s7SP2Zv6263vmBofgECAGBgVuYAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHMCcqnpZVXVVfc0pxr2mqp42t3/wxP3LADaTW5MAzKmq387Svchu7+6fXWXcQ0l2d/fHNqk0gBVZmQOYmf2u6iVJXpmlO8+f+FH2n6+qe6rq7qr6kar60STPzdLNZ98zG/dQVZ012/6Jqrp39njNrG1nVd1fVb9aVfdV1btnv0CQqvrRqvrg7PVv3vwjB0a2fdEFAEzIS5O8s7v/vKoeqarnJ/nmJBck+cbufrSqzuzuR6rqJ5Jcunxlrqq+KckrZvMqyfuq6n8l+esku5Jc093/erYC+L1J3pRkX5ILuvtzLtUC62VlDuAfXJOlH/bO7PmaLP125w2zn75Ldz9yitf41iS3dvffdvenk7w1ybfN+v6iu++abd+ZZOds++4kb66qH0zy6Gk4DuBJxMocQJKqelaSFyb5+qrqJNuSdJZC13o+XFyr9H1ubvsLSZ46274qybcn+Z4k/76qvu5EeAQ4FStzAEu+L8lvdvdXdvfO7j4vyV8keX+SV1XV9iSpqjNn4/8myTNWeJ33JnlpVT2tqp6e5GVJ/uBkb1pVX5LkvO5+T5KfSvLMJF96mo4JeBIQ5gCWXJPk1mVtt2Tpiw7/N8ndVfWBJD8w69uf5HdPfAHihO5+f5I3JvnTJO9L8mvd/WervO+2JG+qqnuS/FmS13X3J57YoQBPJm5NAgAwMCtzAAADE+YAAAYmzAEADEyYAwAYmDAHADAwYQ4AYGDCHADAwIQ5AICB/X8YAizx7AozBgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nb_samples = 10\n",
    "rewards = np.zeros((nb_actions, nb_samples))\n",
    "\n",
    "for a in actions:\n",
    "    for play in range(nb_samples):\n",
    "        rewards[a, play] = get_reward(Q_star, a, var=1.0)\n",
    "\n",
    "Q_t = np.mean(rewards, axis=1)\n",
    "    \n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.bar(actions, Q_t)\n",
    "plt.xlabel('Actions')\n",
    "plt.ylabel('$Q_t(a)$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Make a bar plot of the difference between the true values `Q_star` and the estimates `Q_t`. Conclude. Re-run the sampling cell with different numbers of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAAFzCAYAAABsPz7IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAY0ElEQVR4nO3df7DlZ10f8PeHDWtDpFLIIpps3FQzYqpE6TaIgAwoND9ag9CxCaIWQzNxiMhYx65/1GnVduLAiONMcLvSjPgDozMhTsasBGqpdIagu9EQCCR0J6TNNthsDFERkrD46R/nLD25uZs9d7n3nPvsfb1m7tzv9/k+zzmfO9/J7DvP98dT3R0AADa3py27AAAATkxoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABnLbsAjbamWee2bt27Vp2GQAAJ3T77bc/1N07Vjt2yoe2Xbt25eDBg8suAwDghKrqfx3vmMujAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABLDy0VdVFVXVPVR2qqj2rHL+squ6sqjuq6mBVvXTesQAAp6qFhraq2pbkuiQXJzk/yRVVdf6Kbn+U5ILu/vYkP5rkXWsYCwBwSlr0TNuFSQ51973d/XiSG5JcNtuhuz/X3T3dPSNJzzsWAOBUtejQdlaS+2f2D0/bnqCqvr+q7k5ySyazbXOPnY6/anpp9eCRI0fWpXAAgGVadGirVdr6SQ3dN3X385O8JsnPr2XsdPy+7t7d3bt37NhxsrUCAGwapy34+w4n2Tmzf3aSB47Xubs/VFXfWFVnrnUsMKZde25ZdgmnlPuuvXTZJQDrZNEzbQeSnFdV51bV9iSXJ7l5tkNVfVNV1XT7hUm2J/nLecYCAJyqFjrT1t1Hq+qaJLcm2Zbk+u6+q6qunh7fm+R1SX64qr6Y5AtJ/uX0wYRVxy6yfgCAZVn05dF09/4k+1e07Z3Z/sUkvzjvWACArcCKCAAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMYOGhraouqqp7qupQVe1Z5fgPVtWd058PV9UFM8fuq6qPVdUdVXVwsZUDACzPaYv8sqraluS6JK9KcjjJgaq6ubs/MdPt00le3t2fraqLk+xL8qKZ46/o7ocWVjQAwCaw6Jm2C5Mc6u57u/vxJDckuWy2Q3d/uLs/O939SJKzF1wjAMCms+jQdlaS+2f2D0/bjufKJH84s99J3l9Vt1fVVccbVFVXVdXBqjp45MiRr6hgAIDNYKGXR5PUKm29aseqV2QS2l460/yS7n6gqp6b5ANVdXd3f+hJH9i9L5PLqtm9e/eqnw8AMJJFz7QdTrJzZv/sJA+s7FRVL0jyriSXdfdfHmvv7gemvx9MclMml1sBAE55i55pO5DkvKo6N8n/SXJ5ktfPdqiqc5K8N8kPdfenZtrPSPK07v6b6fark/zcwioHgIHs2nPLsks45dx37aVL/f6FhrbuPlpV1yS5Ncm2JNd3911VdfX0+N4kP5vkOUneWVVJcrS7dyf52iQ3TdtOS/Ke7n7fIusHAFiWRc+0pbv3J9m/om3vzPabkrxplXH3JrlgZTsAwFZgRQQAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYACnLbsAAMaya88tyy7hlHPftZcuuwQGYKYNAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAASw8tFXVRVV1T1Udqqo9qxz/waq6c/rz4aq6YN6xAACnqoWGtqraluS6JBcnOT/JFVV1/opun07y8u5+QZKfT7JvDWMBAE5Ji55puzDJoe6+t7sfT3JDkstmO3T3h7v7s9PdjyQ5e96xAACnqkWHtrOS3D+zf3jadjxXJvnDtY6tqquq6mBVHTxy5MhXUC4AwOaw6NBWq7T1qh2rXpFJaPu3ax3b3fu6e3d3796xY8dJFQoAsJmctuDvO5xk58z+2UkeWNmpql6Q5F1JLu7uv1zLWACAU9GiZ9oOJDmvqs6tqu1JLk9y82yHqjonyXuT/FB3f2otYwEATlULnWnr7qNVdU2SW5NsS3J9d99VVVdPj+9N8rNJnpPknVWVJEenlzpXHbvI+gEAlmXRl0fT3fuT7F/Rtndm+01J3jTvWACArcCKCAAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAZxUaKuqM6pq23oXAwDA6uYKbVX1tKp6fVXdUlUPJrk7yWeq6q6qeltVnbexZQIAbG3zzrR9MMk3JvmZJM/r7p3d/dwkL0vykSTXVtUbNqhGAIAtb961R7+3u7+4srG7H05yY5Ibq+rp61oZAABfNldoOxbYquo5SX4gyaNJ7kryse7+wmwfAADW31ofRLgpyY4k/ynJ25L8VVXdve5VAQDwBPNeHj3mmd39c1X12u5+eVW9Lsk3bURhAAD8f2udaXt0+vuxqjq9u29Mcsk61wQAwAprnWl7e1U9O8nvJrm+qj6c5Kz1LwsAgFlrmmnr7hu7++Hu/qUk+5PsTHLZhlQGAMCXzTXTVlXV3T3b1t2/eaI+AACsj7lfrltVP15V58w2VtX2qnplVb07yY+sf3kAACTz39N2UZIfTfI7VXVukkeSnJ5J6Ht/knd09x0bUSAAAPO/XPfRJO9M8s7pygdnJvlCdz+ygbUBADC11ld+pLu/2N2f6e5Hquonj7VX1Tevb2kAAByz1ld+JEmq6llJ3pHk+VX1aJI7k1yZ5I3rVxoAAMecVGibXhZ9Y1X90yQPJXlBkveuY10AAMyYO7RV1dcn+Z4kz0hyd3f/cXffOj18+0YUBwDAxFz3tFXVqzMJZpckeXGSX66qe6rquzayOAAAJuadafuFJC/r7kPHGqrqxUl+raquTPK33f3xjSgQAID5Q9v22cCWJN19W1W9NskfJHksybetd3EAAEzM+8qPR6tqx8rG7v5Uki9lcq8bAAAbZN7Q9rYkvz99GOHLqurMJI9194PrXhkAAF8274oIN1bVVyW5rapuT/LRJNuT/EAm97sBALCB5l4Robvfk+RbMrmH7WuSfDHJ67v73RtUGwAAU2t6uW53fz7J9RtUCwAAx7HmtUcBAFi8kwptVfXP17sQAACO72Rn2v7julYBAMBTOtnQVutaBQAAT+lkQ1uvaxUAADwlDyIAAAxAaAMAGMDJhrb/u65VAADwlE4qtHX3q9a7EAAAjm/hl0er6qKquqeqDlXVnlWOP7+qbquqx6rqp1Ycu6+qPlZVd1TVwcVVDQCwXGtaxuorVVXbklyX5FVJDic5UFU3d/cnZro9nOQtSV5znI95RXc/tKGFAgBsMie7IsIZ0wC2VhcmOdTd93b340luSHLZbIfufrC7D2SyID0AAJlzpq2qnpbk8iQ/mOSfJHksyVdV1ZEk+5Ps6+7/OcdHnZXk/pn9w0letIZ6O8n7q6qT/Ofu3reGsRtm155bll3CKee+ay9ddgkAsKnMO9P2wSTfmORnkjyvu3d293OTvCzJR5JcW1VvmONzVltJYS0v6n1Jd78wycVJ3lxV373ql1RdVVUHq+rgkSNH1vDxAACb07z3tH1vdz/pcmV3P5zkxiQ3VtXT5/icw0l2zuyfneSBOWtIdz8w/f1gVd2UyeXWD63Sb1+SfUmye/duqzcAAMOba6btWGCrqudU1Y9V1Rur6sKqOn1lnxM4kOS8qjq3qrZncsn15nlqmN5H98xj20leneTj84wFABjdWp8evSnJf03yY0k+leTFVXVvdz9/nsHdfbSqrklya5JtSa7v7ruq6urp8b1V9bwkB5P8/SR/V1VvTXJ+kjOT3FRVx+p+T3e/b431AwAMaa2h7Znd/XNV9drufnlVvS7JN63lA7p7fyYPL8y27Z3Z/otMLpuu9NdJLlhjvQAAp4S1vvLj0envx6rq9O6+Mckl61wTAAArrHWm7e1V9ewkv5vk+qr6cCav8QAAYAOtaaatu2/s7oe7+5cyucS5MytejgsAwPqb9+W61d1PeHVGd//mifoAALA+5n65blX9eFWdM9tYVdur6pVV9e4kP7L+5QEAkMx/T9tFSX40ye9U1blJHklyeiah7/1J3tHdd2xEgQAAzBnauvvRJO9M8s7pygdnJvlCdz+ygbUBADC11qdHj6188JkNqAUAgOM44T1t0yWjUlVfvfHlAACwmnkeRPgH06WnXrrRxQAAsLp5Qtv3JPlXSf5hVT13Y8sBAGA189zT9qeZPDm6s7sf3OB6AABYxQlDW3d/crp55wbXAgDAcax1wXgAAJZg7ld+VNXXZ3J/2zOS3N3df7xhVQEA8ARzzbRV1auT3J7kkiQvTvLLVXVPVX3XRhYHAMDEvDNtv5DkZd196FhDVb04ya9V1ZVJ/ra7P74RBQIAMH9o2z4b2JKku2+rqtcm+YMkjyX5tvUuDgCAiXkfRHi0qnasbOzuTyX5Uib3ugEAsEHmDW1vS/L704cRvqyqzkzymPe3AQBsrLkuj3b3jVX1VUluq6rbk3w0yfYkP5DJ/W4AAGygud/T1t3vSfItmdzD9jVJvpjk9d397g2qDQCAqbnf05Yk3f35JNdvUC0AAByHFREAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwgIWHtqq6qKruqapDVbVnlePPr6rbquqxqvqptYwFADhVLTS0VdW2JNcluTjJ+UmuqKrzV3R7OMlbkrz9JMYCAJySFj3TdmGSQ919b3c/nuSGJJfNdujuB7v7QJIvrnUsAMCpatGh7awk98/sH562bfRYAIChLTq01Sptvd5jq+qqqjpYVQePHDkyd3EAAJvVokPb4SQ7Z/bPTvLAeo/t7n3dvbu7d+/YseOkCgUA2EwWHdoOJDmvqs6tqu1JLk9y8wLGAgAM7bRFfll3H62qa5LcmmRbkuu7+66qunp6fG9VPS/JwSR/P8nfVdVbk5zf3X+92thF1g8AsCwLDW1J0t37k+xf0bZ3ZvsvMrn0OddYAICtwIoIAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxg4aGtqi6qqnuq6lBV7VnleFXVr0yP31lVL5w5dl9Vfayq7qiqg4utHABgeU5b5JdV1bYk1yV5VZLDSQ5U1c3d/YmZbhcnOW/686Ikvzr9fcwruvuhBZUMALApLHqm7cIkh7r73u5+PMkNSS5b0eeyJL/REx9J8qyq+roF1wkAsKksOrSdleT+mf3D07Z5+3SS91fV7VV11YZVCQCwySz08miSWqWt19DnJd39QFU9N8kHquru7v7Qk75kEuiuSpJzzjnnK6kXAGBTWPRM2+EkO2f2z07ywLx9uvvY7weT3JTJ5dYn6e593b27u3fv2LFjnUoHAFieRYe2A0nOq6pzq2p7ksuT3Lyiz81Jfnj6FOl3Jvmr7v5MVZ1RVc9Mkqo6I8mrk3x8kcUDACzLQi+PdvfRqromya1JtiW5vrvvqqqrp8f3Jtmf5JIkh5J8Pskbp8O/NslNVXWs7vd09/sWWT8AwLIs+p62dPf+TILZbNveme1O8uZVxt2b5IINLxAAYBOyIgIAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDaAAAGILQBAAxAaAMAGIDQBgAwAKENAGAAQhsAwACENgCAAQhtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIABCG0AAAMQ2gAABiC0AQAMQGgDABiA0AYAMAChDQBgAEIbAMAAhDYAgAEIbQAAA1h4aKuqi6rqnqo6VFV7VjleVfUr0+N3VtUL5x0LAHCqOm2RX1ZV25Jcl+RVSQ4nOVBVN3f3J2a6XZzkvOnPi5L8apIXzTkWjmvXnluWXcIp575rL112CQBbxqJn2i5Mcqi77+3ux5PckOSyFX0uS/IbPfGRJM+qqq+bcywAwClp0aHtrCT3z+wfnrbN02eesQAAp6SFXh5NUqu09Zx95hk7+YCqq5JcNd39XFXdM3eFp74zkzy07CJOpH5x2RUs1RDnKHGeMsB52uLnKHGeRjDEOUoWdp6+4XgHFh3aDifZObN/dpIH5uyzfY6xSZLu3pdk31da7Kmoqg529+5l18HxOUdjcJ7G4Dxtfs7R/BZ9efRAkvOq6tyq2p7k8iQ3r+hzc5Ifnj5F+p1J/qq7PzPnWACAU9JCZ9q6+2hVXZPk1iTbklzf3XdV1dXT43uT7E9ySZJDST6f5I1PNXaR9QMALMuiL4+mu/dnEsxm2/bObHeSN887ljVz2Xjzc47G4DyNwXna/JyjOdUkIwEAsJlZxgoAYABC2xZhCbDNr6qur6oHq+rjy66F46uqnVX1war6ZFXdVVU/seyaeKKq+ntV9adV9dHpOfoPy66J46uqbVX151X1B8uuZbMT2raAmSXALk5yfpIrqur85VbFKn49yUXLLoITOprk33T3tyT5ziRv9t/TpvNYkld29wVJvj3JRdO3EbA5/USSTy67iBEIbVuDJcAG0N0fSvLwsuvgqXX3Z7r7z6bbf5PJPzZWZ9lEpssgfm66+/Tpjxu4N6GqOjvJpUnetexaRiC0bQ2WAIMNUFW7knxHkj9ZcimsML3kdkeSB5N8oLudo83pl5P8dJK/W3IdQxDatoa5lwAD5lNVX53kxiRv7e6/XnY9PFF3f6m7vz2T1XMurKpvXXJJrFBV/yzJg919+7JrGYXQtjXMs3wYMKeqenomge23u/u9y66H4+vuR5L897hfdDN6SZLvq6r7Mrlt55VV9VvLLWlzE9q2BkuAwTqpqkryX5J8srt/adn18GRVtaOqnjXdPj3J9ya5e6lF8STd/TPdfXZ378rk36X/1t1vWHJZm5rQtgV099Ekx5YA+2SS37ME2OZTVb+T5LYk31xVh6vqymXXxKpekuSHMpkVuGP6c8myi+IJvi7JB6vqzkz+p/UD3e11EgzPiggAAAMw0wYAMAChDQBgAEIbAMAAhDYAgAEIbQAAAxDagC2pqr6/qrqqnn+Cfm+tqmfM7O8/9g4wgEXyyg9gS6qq38vkfV5/1N3//in63Zdkd3c/tKDSAFZlpg3Ycqbrhr4kyZWZvIn92ALjb6+qj1XVnVX141X1liRfn8mLWj847XdfVZ053f7Jqvr49Oet07ZdVfXJqvq1qrqrqt4/fSt/quotVfWJ6effsPi/HBjZacsuAGAJXpPkfd39qap6uKpemORFSc5N8h3dfbSqnt3dD1fVTyZ5xcqZtqr6x0neOB1XSf6kqv44yWeTnJfkiu7+19MZvdcl+a0ke5Kc292PucQKrJWZNmAruiKTBaoz/X1FJutT7p0u+5bufvgEn/HSJDd199929+eSvDfJy6bHPt3dd0y3b0+ya7p9Z5Lfrqo3JDm6Dn8HsIWYaQO2lKp6TpJXJvnWquok25J0JuFqLTf51lMce2xm+0tJTp9uX5rku5N8X5J/V1X/6FhIBDgRM23AVvMvkvxGd39Dd+/q7p1JPp3kz5JcXVWnJUlVPXva/2+SPHOVz/lQktdU1TOq6owk35/kfxzvS6vqaUl2dvcHk/x0kmcl+ep1+puALUBoA7aaK5LctKLtxkweOPjfSe6sqo8mef302L4kf3jsQYRjuvvPkvx6kj9N8idJ3tXdf/4U37styW9V1ceS/HmSd3T3I1/ZnwJsJV75AQAwADNtAAADENoAAAYgtAEADEBoAwAYgNAGADAAoQ0AYABCGwDAAIQ2AIAB/D8k820TqOjz0wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "plt.bar(actions, Q_t - Q_star)\n",
    "plt.xlabel('Actions')\n",
    "plt.ylabel('$Q^*(a) - Q_t(a)$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** To better understand the influence of the number of samples on the accuracy of the sample average, create a `for` loop over the preceding code, with a number of samples increasing from 1 to 100. For each value, compute the **mean square error** (mse) between the estimates `Q_t` and the true values `Q^*`.\n",
    "\n",
    "The mean square error is simply defined over the `N = nb_actions` actions as:\n",
    "\n",
    "$$\\epsilon = \\frac{1}{N} \\, \\sum_{a=0}^{N-1} (Q_t(a) - Q^*(a))^2$$\n",
    "\n",
    "At the end of the loop, plot the evolution of the mean square error with the number of samples. You can append each value of the mse in an empty list and then plot it with `plt.plot`, for example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFlCAYAAADPim3FAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA48klEQVR4nO3dd3xcV5n/8e8zRZLVbMmS3OVux07s2I7TSHFCgDhlUyCQBAhsCkno8GOXzbILLLCwlF0IkEYW0ggksCSBAE5CeiW2ZTvuTe5ykWSrS5Y0ozm/P2YkqxdrrqXMfN6vl1/WzL2jOfK1pa+fc+5zzDknAAAAHB/fUA8AAADg3YwwBQAAMAiEKQAAgEEgTAEAAAwCYQoAAGAQCFMAAACDEBiqN87Ly3NTpkwZqrcHAADot1WrVh12zuV3d2zIwtSUKVNUVFQ0VG8PAADQb2a2p6djTPMBAAAMAmEKAABgEAhTAAAAg0CYAgAAGATCFAAAwCAQpgAAAAaBMAUAADAIhCkAAIBBIEwBAAAMAmEKAABgEAhTAAAAg5CwYaqmMaSXt5SpvLZpqIcCAAASWMKGqb1HGnTjQyu1em/lUA8FAAAksIQNU0F/9EsLt7ghHgkAAEhkCRumAn6TJIUjkSEeCQAASGQJG6ZSYpWp5jBhCgAAeCdhw9SxyhTTfAAAwDuJG6Z8rWumqEwBAADvJGyYCsYqUyEWoAMAAA8lbJgKtN7NxwJ0AADgoT7DlJk9YGZlZrahh+MfM7N1sV9vmdmp8R/mwAV8VKYAAID3+lOZekjS0l6O75K0xDk3X9J3JN0fh3ENWmufqRBrpgAAgIcCfZ3gnHvNzKb0cvytdg/fljQxDuMaNL/P5DOadgIAAG/Fe83UzZKe6emgmd1qZkVmVlReXh7nt+4q4PcpxJopAADgobiFKTO7UNEw9S89neOcu985t9g5tzg/Pz9eb92joM+oTAEAAE/1Oc3XH2Y2X9IvJV3inDsSj88ZDwG/jz5TAADAU4OuTJlZoaQnJd3gnNs2+CHFT9DvUzOVKQAA4KE+K1Nm9pikCyTlmVmJpG9KCkqSc+4+Sd+QNFrSPWYmSWHn3GKvBjwQQb9RmQIAAJ7qz9181/dx/BZJt8RtRHEU8Bt78wEAAE8lbAd0SQr6fPSZAgAAnkroMBXwczcfAADwVmKHKZ+PvfkAAICnEjpMBQPczQcAALyV2GHKx918AADAWwkdplgzBQAAvJbQYSrI3nwAAMBjCR2mAuzNBwAAPJbYYcpPnykAAOCthA5TKYQpAADgsYQOU2wnAwAAvJbYYcrnY80UAADwVEKHqaDfmOYDAACeSugwxTQfAADwWkKHqaDfp1CYyhQAAPBO4ocpmnYCAAAPJXSYomknAADwWmKHKb9P4YiTcwQqAADgjYQOU0GfSRKL0AEAgGcSOkwF/NEvj/YIAADAKwkdpoL+aGUqxLopAADgkQQPU9EvL0xlCgAAeCShw1TAz5opAADgrYQOU0Efa6YAAIC3EjpMtVWmWDMFAAA8kuBhisoUAADwVkKHqRTu5gMAAB5L6DAViK2ZCrM/HwAA8EhihykqUwAAwGMJHaboMwUAALyW0GEq4KMyBQAAvJXQYSoYiN3Nx5opAADgkcQOU60L0KlMAQAAjyR0mDrWtJPKFAAA8EZCh6lg69187M0HAAA8ktBhqq3PFJUpAADgkcQOU219pghTAADAGwkdplLa9uZjmg8AAHijzzBlZg+YWZmZbejhuJnZz8ys2MzWmdmi+A/z+ARo2gkAADzWn8rUQ5KW9nL8EkkzY79ulXTv4IcVH21387EAHQAAeKTPMOWce01SRS+nXCnpERf1tqRRZjYuXgMcjNY+U0zzAQAAr8RjzdQESfvaPS6JPTfk6DMFAAC8Fo8wZd08120pyMxuNbMiMysqLy+Pw1v37tjefIQpAADgjXiEqRJJk9o9nijpQHcnOufud84tds4tzs/Pj8Nb987MFPQbTTsBAIBn4hGmnpb0idhdfWdJqnbOHYzD542LgM/HNB8AAPBMoK8TzOwxSRdIyjOzEknflBSUJOfcfZKWSbpUUrGkBkk3ejXY4xHwGwvQAQCAZ/oMU8656/s47iR9Nm4jirOg36dwhMoUAADwRkJ3QJeii9BDYSpTAADAGwkfpoJ+n0JUpgAAgEeSIEyZwqyZAgAAHkn4MBVgzRQAAPBQ4ocpH3fzAQAA7yR8mAr66TMFAAC8k/Bhij5TAADASwkfpoJ+H3vzAQAAzyRBmDKF2ZsPAAB4JOHDFHvzAQAALyV8mAqyZgoAAHgo4cNUwMeaKQAA4J2ED1PBgI81UwAAwDOJH6Z8RmUKAAB4JuHDVIC9+QAAgIeSIEyxNx8AAPBOwoepIHvzAQAADyV8mArQAR0AAHgo4cNUdKNjKlMAAMAbSRCmTCHWTAEAAI8kfJgK+HxyTmqh1xQAAPBA4ocpv0kS66YAAIAnEj5MBQlTAADAQwkfpgK+6JfIInQAAOCFhA9TwUD0S2QROgAA8ELihylfdJqPyhQAAPBCwoepgJ9pPgAA4J2ED1NtC9CZ5gMAAB5I+DDVugCdu/kAAIAXEj5MtVammOYDAABeSIIwRWUKAAB4J+HDVGsH9DDbyQAAAA8kfphizRQAAPBQwocp1kwBAAAvJXyYCrBmCgAAeCjhw9SxjY6pTAEAgPhLgjAV64BO004AAOCBhA9TAfbmAwAAHkr4MEWfKQAA4KV+hSkzW2pmW82s2Mzu6Ob4SDP7s5mtNbONZnZj/Id6fAKsmQIAAB7qM0yZmV/S3ZIukTRX0vVmNrfTaZ+VtMk5d6qkCyT9j5mlxHmsx6W1zxRrpgAAgBf6U5k6Q1Kxc26nc65Z0uOSrux0jpOUZWYmKVNShaRwXEd6nFLapvmoTAEAgPjrT5iaIGlfu8clsefau0vSHEkHJK2X9EXn3LAoBbVtJ8OaKQAA4IH+hCnr5rnOZZ6LJb0jabykBZLuMrPsLp/I7FYzKzKzovLy8gEO9fiwNx8AAPBSf8JUiaRJ7R5PVLQC1d6Nkp50UcWSdkk6qfMncs7d75xb7JxbnJ+ff7xjHpAge/MBAAAP9SdMrZQ008ymxhaVXyfp6U7n7JV0kSSZ2RhJsyXtjOdAj5fPZ/IZYQoAAHgj0NcJzrmwmX1O0nOS/JIecM5tNLPbY8fvk/QdSQ+Z2XpFpwX/xTl32MNxD0jQ76NpJwAA8ESfYUqSnHPLJC3r9Nx97T4+IOkD8R1a/AT9Pu7mAwAAnkj4DuhSdBE6faYAAIAXkiNM+ahMAQAAbyRFmAr6jQXoAADAE0kRpgJ+o2knAADwRFKEqaDfpxBNOwEAgAeSI0z5fFSmAACAJ5IiTEWn+ahMAQCA+EuSMMU0HwAA8EZShKmgzxQKM80HAADiLynCFE07AQCAV5IiTLGdDAAA8ErShCkqUwAAwAtJEaYCPu7mAwAA3kiKMBX0+9RMnykAAOCBpAhT9JkCAABeSYowFfTTAR0AAHgjScKU0bQTAAB4IinCVIC9+QAAgEeSI0yxZgoAAHgkKcIUd/MBAACvJEWYCvhMYdZMAQAADyRFmAr6fWqJODlHoAIAAPGVJGHKJIn9+QAAQNwlRZgK+KNfJvvzAQCAeEuOMOWjMgUAALyRFGEqGKtMhbijDwAAxFlShKlAbM0UvaYAAEC8JUWYojIFAAC8kiRhKlaZotcUAACIs6QIUwFf7G4+KlMAACDOkiJMtVam2FIGAADEW1KEqWOVKab5AABAfCVFmAoGaNoJAAC8kRxhiqadAADAI0kRptq2kyFMAQCAOEuSMBWrTDHNBwAA4iwpwlQwtgA9FCZMAQCA+EqKMBWgaScAAPBIUoQptpMBAABe6VeYMrOlZrbVzIrN7I4ezrnAzN4xs41m9mp8hzk4QTY6BgAAHgn0dYKZ+SXdLen9kkokrTSzp51zm9qdM0rSPZKWOuf2mlmBR+M9Lm1387EAHQAAxFl/KlNnSCp2zu10zjVLelzSlZ3O+aikJ51zeyXJOVcW32EOTmufqWYqUwAAIM76E6YmSNrX7nFJ7Ln2ZknKMbNXzGyVmX2iu09kZreaWZGZFZWXlx/fiI/DsT5TVKYAAEB89SdMWTfPdS7xBCSdJukySRdL+rqZzeryIufud84tds4tzs/PH/Bgj1eANVMAAMAjfa6ZUrQSNand44mSDnRzzmHnXL2kejN7TdKpkrbFZZSDlNJ6Nx9rpgAAQJz1pzK1UtJMM5tqZimSrpP0dKdz/iTpPDMLmFm6pDMlbY7vUI9fwEdlCgAAeKPPypRzLmxmn5P0nCS/pAeccxvN7PbY8fucc5vN7FlJ6yRFJP3SObfBy4EPhL8tTFGZAgAA8dWfaT4555ZJWtbpufs6Pf6RpB/Fb2jxY2YK+o27+QAAQNwlRQd0SQr4fFSmAABA3CVNmAr6jb35AABA3CVRmPKxNx8AAIi7pAlTAb9xNx8AAIi75AlTPipTAAAg/pImTAX9phBrpgAAQJwlTZgK+LmbDwAAxF/ShKnoAnQqUwAAIL6SKEyZwuzNBwAA4ixpwlTAx918AAAg/pInTPl9ambNFAAAiLOkCVNBv7EAHQAAxF3ShKmAz8d2MgAAIO6SJkxxNx8AAPBCEoUppvkAAED8JU2YCrDRMQAA8EDShKmgz5jmAwAAcZc0YSpA004AAOCBpAlTQb+Ppp0AACDukipMsWYKAADEW9KEqYDP6DMFAADiLnnCFJUpAADggaQJU0F/9G4+56hOAQCA+EmaMBXwRb/UFqb6AABAHCVNmAoGTJJYNwUAAOIqecJUrDLFuikAABBPSROmAv5YZYpeUwAAII6SKExRmQIAAPGXNGEq6ItWpkKsmQIAAHGUNGGqtTIVpjIFAADiKGnCVDC2ZirEmikAABBHSRSmYpWpCJUpAAAQP0kTpgKta6bCVKYAAED8JE2Yaq1MhahMAQCAOEqaMEWfKQAA4IWkCVNB7uYDAAAeSKIwRZ8pAAAQf0kTpgI+KlMAACD+kidMtfWZIkwBAID46VeYMrOlZrbVzIrN7I5ezjvdzFrM7Jr4DTE+2u7mYwE6AACIoz7DlJn5Jd0t6RJJcyVdb2ZzezjvB5Kei/cg46G1zxRNOwEAQDz1pzJ1hqRi59xO51yzpMclXdnNeZ+X9ISksjiOL256q0x96pEi/eDZLSd6SAAAIAEE+nHOBEn72j0ukXRm+xPMbIKkqyW9V9LpPX0iM7tV0q2SVFhYONCxDsqx1ghdw9SqPZWqbQyd0PEAAIDE0J/KlHXzXOdEcqekf3HOtfT2iZxz9zvnFjvnFufn5/dziPHR0wL0UEtEFfXNKq9tOqHjAQAAiaE/lakSSZPaPZ4o6UCncxZLetzMJClP0qVmFnbO/TEeg4yHoK91mq9jmDpS1yxJhCkAAHBc+hOmVkqaaWZTJe2XdJ2kj7Y/wTk3tfVjM3tI0l+GU5CS2m0n06lpZ2uIqmkMqzHUorSg/4SPDQAAvHv1Oc3nnAtL+pyid+ltlvR759xGM7vdzG73eoDxcmxvvo6VqfK6xmMfU50CAAAD1J/KlJxzyyQt6/TcfT2c+4+DH1b8HZvm61iZKqs5FqDK65o0KTf9hI4LAAC8uyVNB3Sfz+T3WZc+U+2rUVSmAADAQCVNmJKijTs7V6bK65pksfsVCVMAAGCgkipMBf2+Lnfzldc2acroDJlJZYQpAAAwQP1aM5UoAn7r0rSzvLZJY7PTVNsYojIFAAAGLOkqU13WTNU1KT8rVXmZqYQpAAAwYElVmQp2WjPlnFNZTTRMVTY0q7yOMAUAAAYmqSpTgU5rpuqbW3Q01KL8rFQVZKWpvKaxl1cDAAB0lVSVqc5rplqn9QqyUlXVEFJ5XZOcczLrbjtCAACArpKqMhX0daxMtYap/KxU5WelKtTiVH00NFTDAwAA70JJFaYCfuuwN1/7MFWQldrhOQAAgP5IqjDVuc9UWW10jVR+ZrQyFX2OMAUAAPovycJU1zVTAZ8pJz2lLUxRmQIAAAORVGEq0M2aqbzMVPl8RpgCAADHJbnClN8Uar9mKtawU5KyUgNKC/roNQUAAAYkqcJU0O9TuFNlqjVMmUWrU2X0mgIAAAOQVGEq4Ou4Zqqstkn5maltj/MzU6lMAQCAAUmqMBUM+BSK7c3XEnE6Utekgux2YSqL/fkAAMDAJFeYaleZqqhvVsSpbZpPUnRLmV7CVHltk/628ZDn4wQAAO8eSRWm2u/N19awM7NjZaqyIaTmcKTb1//qjV267dFVqm8Kez9YAADwrpBUYSroN4VilanWtVHtK1OtHx/uYd3U1kM1co7GngAA4JikClMBn0/h2Jqp1rv2OoSpzN57TW0rrevwWgAAgKQKU9HWCD1XploXo3dXeaptDGl/1VFJUimVKQAAEJNkYco6rJnKTA0oPSXQdry3LuitVSmJyhQAADgmqcJUoFOYal+VkqTRGb2Fqdq2j2mfAAAAWgX6PiVxBHw+RZwUibhomMrsGKZSAj7lpAdVXte18rT1UK0yUvwalZ6iUipTAAAgJqnCVNBvkqRQJKLyuibNGZvd5ZyCrDSV1XRfmZo5Jks+424+AABwTJJN80W/3HCLU3lN12k+KdYFvZvWCNtKazV7TFY0bBGmAABATFKFqWAsTNU1hVXbFO45THUKS4frmnS4rlmzxmZpTDabIQMAgGOScprvQKzFQXdhqiAWppxzMouev+1QdPH57DFZagy1qKYxrMZQi9KC/hM0cgAAMFwlVWUq4It+uQequjbsbJWflaqmcEQ1jce2jNkau5Nv1tjMttd0t64KAAAkn+QKU7HK1MHqWGUqs/swJXVsf7CttFY56UHlZ6aqoDVM1TLVBwAAkixMHZvmiwah1o7n7XW3pczWQ7WaNSZLZqYx2WmSpFIqUwAAQEkWpo5N8x2Vz4416WyvNWC13tHnnNO20jrNHpsVPU5lCgAAtJNUYar1br6D1UeVm5Eqv8+6nJOfGa08td6xd6C6UXVN4bYwlZOeooDPaI8AAAAkJV2Yioan/VWN3S4+l6TsEQGl+H1tlan2d/JJks9nKshKZQE6AACQlGRhqrVp5+G67ht2SpKZdeg11Xon38xYmJKk/Ow0pvkAAICkJAtTwXbTegU9hCmpY+PObYdqNW5kmkaOCHZ4LZUpAAAgJVmYaq1MSd33mGp/rH1lala7qpSkaBd0KlMAAED9DFNmttTMtppZsZnd0c3xj5nZutivt8zs1PgPdfBa10xJ3feYajsWC1MtEaftZcfu5GtVkJWmyoaQmsItno0VAAC8O/QZpszML+luSZdImivpejOb2+m0XZKWOOfmS/qOpPvjPdB4CPa3MpWZqoqGZu0or1NzONKlMlXQTWNPAACQnPpTmTpDUrFzbqdzrlnS45KubH+Cc+4t51xl7OHbkibGd5jxEWhfmeolTBVkp8o56e87jkg6didfq9bGnbRHAAAA/QlTEyTta/e4JPZcT26W9MxgBuWV1qadUh8L0GNTgK9vPywzaUZBZsfjbfvzsW4KAIBkF+jHOV07W0qu2xPNLlQ0TJ3bw/FbJd0qSYWFhf0cYvwE+1mZaj329s4jmpybrhEp/g7HW7ukU5kCAAD9qUyVSJrU7vFESQc6n2Rm8yX9UtKVzrkj3X0i59z9zrnFzrnF+fn5xzPeQWm9my8t6FNmas85siA2jVfXFO6yXkqKbkPjM9EeAQAA9CtMrZQ008ymmlmKpOskPd3+BDMrlPSkpBucc9viP8z4aK1M5Welyqy7gltUXmZK28ed7+STJL8v2tiT9ggAAKDPaT7nXNjMPifpOUl+SQ845zaa2e2x4/dJ+oak0ZLuiYWUsHNusXfDPj7B2Jqp3toiSFJqwK+RI4KqPhrqtjIlRdsjlFKZAgAg6fVnzZScc8skLev03H3tPr5F0i3xHVr8td7NV5CV1ue5+Vmpqj4a6rYyFf0cqTpQTWUKAIBkl1Qd0Fv7TPW2+LxVQVaqgn7TlNEZ3R/PTlM503wAACS9flWmEkXQ71NOerBLq4PunDJhpCQpJdB93izIStXhumaFWiIdmoECAIDkklRhyu8zvfJPFyoj1d/nuV+7dI6c67YDhKRj7REO1zVp3MgRcRsjAAB4d0m6ksrI9GCHDY9709sdf2Ni665ojwAAQHJLujAVL62VqVK6oAMAkNQIU8ep9Y5AuqADAJDcCFPHKS8zRWaEKQAAkh1h6jgF/D6NzkilPQIAAEmOMDUIBVmpdEEHACDJEaYGYUw2+/MBAJDsCFODUJCVlnCtEVbtqdS9r+wY6mEAAPCuQZgahILsVB2ua1JLpOfmnu8mzjl98+kN+sGzW1RR3zzUwwEA4F2BMDUIBVmpijjpSF3v1amWiNNvl+/Ve/7rRT345q4TNLqBW7GrQhv210iKVqgAAEDfCFODUJDdd6+pN4sP67Kfva6vPbVe5XVNevTtPb1uUzOUfvXGLo1KDyrF71PR7oqhHg4AAO8KhKlBKMiKdkHvbhH6zvI63fLwSn3sl8tV1xTW3R9dpK9fPlc7yuu1vazuRA+1T3uO1Ov5zaX62JmFmj9xpFYSpgAA6BfC1CC0VqY6t0fYdKBGS3/6ut7eWaGvLp2tF/7fEl02f5yWnjJWZtJf1x0ciuH26sE3dyvgM33i7Ck6bUqO1u+vVmOoZaiHBQDAsEeYGoT8zFhlql2YCrdE9NUn1io7LagXv7JEn7lghtKCfknRu/9On5KrZeuHV5iqaQzp/4r26fL54zUmO02nT85VqMVpXUn1UA8NAIBhjzA1CCkBn3IzUjpM8/3v67u0YX+NvnPlyRoTq1y1d9m8cdpeVqftpbUncqh6cXOpNh2o6fbY71bsU31zi24+d6ok6bTJOZLEVB8AAP1AmBqk9l3Qd5TX6ScvbNPSk8fqknnjuj3/kthU37L1h07YGF/dVq5bHinSB+99U3/b2PF9wy0RPfTWbp0xNVenTBgpScrJSNGMgkwWoQMA0A+EqUEqyE5TeW2jIhGnO55Yp7SAT9++6uRezz998omb6jtU3agv/+4dzSrI0uyx2brt0VV6+K3dbcef21iq/VVH26pSrU6fkqNVeyoVSZAeWgAAeIUwNUgFWakqq23Sb5bv0crdlfr65XNVkNV1eq+9S+aN1dbSWhV7fFdfuCWiLzy+Ro2hFt39sUV67FNn6qKTxuibT2/U95ZtViTi9Ks3dqowN13vmzOmw2sXT85VTWN4WN55CADAcEKYGqTWMPX9Z7bovJl5uua0iX2+5pJTolOAz3hcnbrzhe1asatC3736FM0oyFR6SkC/uOE03XDWZN3/2k5d/79va/XeKt10zhT5fdbhtYunsG4KAID+IEwN0pjsNLVEnJyk7109T2bW52vGjkzT4sk5+quHYerVbeW6+5ViXbt4kq5eeCzg+X2mb195sv71kpO0fFeFstIC+vDiSV1eX5ibrvysVNZNAQDQh8BQD+DdbvyoEZKkr148W5Ny0/v9ukvmjdN3/rJJO8vrNC0/M65jar9O6j+u6Lp+y8x025LpOmlctnwmZaR2/WtgZjp9So6K2FYGAIBeUZkapAtm5+uRm87QJ86eMqDXXTpvrCTpmQ3xvasvEnEd1kmNSPH3eO6SWfk6b2Z+j8cXT85VSeVRHaw+GtcxAgCQSAhTgxT0+3T+rHz5fH1P77U3buQILSocFfdu6M9vLtWKXRX6xuVzNaNgcBWv1nVTRbupTgEA0BPC1BC6dN44bTpYo92H6+Py+ZxzuueVHSrMTe/XQvi+zB2XrfQUv1b1MNUXaokM+j0wMDvK61TbGBrqYQAA2iFMDaHWxp7LNsSnOvX3HUe0dl+VblsyTQH/4C9twO/TwsJR3d7R9+yGg5r3H8/pvld3DPp90D8llQ26+CevacmPXtGv396jMGEWAIYFwtQQmjBqhE6fkqNfvb5L+6sGvy7p3ld3KD8rVR9aNPiqVKvTJudq88GaDtWQN4sP6wuPvaOg36fvP7NFd720PW7vh5498vc9cpKm52fo63/coKU/fV0vby2TczRWBYChRJgaYt//0Hw1hyO69ZEiHW1uOe7Ps76kWq9vP6ybz53atrFyPJw+JUcRJ63ZWyVJWruvSrc+UqSpeRl69Z8v1NULJ+i//7ZNd76wLW7via4amsN6fMVeLT15rH5/29n6xQ2nKdwS0Y0PrtQnHlihXXGaKgYADBxhaohNz8/Uz65fqE0Ha/TVJ9Ydd5XhnleKlZ0W0MfOLIzr+BYW5shnUtGeShWX1eofH1yh3MwUPXLzGcrNSNF/f/hUXXPaRN35wnb9+G9bqZL04bmNh3TLwyv1k+e36eUtZaqsb+7X655YvV81jWHddO4UmZkuPnms/vblJfr65XO1dl+VbnpopRpDxx/GAQDHjz5Tw8CFJxXoqxefpB88u0Vzx2Xr0xdM73LOupIqrdlbpevOmKTUQMfKU3FZnZ7deEifvWCGstKCcR1bZmpAc8Zl6/lNpfq/on3y+3z69U1nakx2dMscv8/0ww/NV8Bn+tlLxQpFnL568ex+NS9NNgerj+qf/m+tJOnFLWVqzZ2TR6frPdPz9I3L53bbyiIScXrwzV2aP3GkFhXmtD2fEvDp5nOnavaYLH38V8v185e2658vPumEfC0AgGMIU8PE7UumadPBGv3wuS06aWyWLjypQJK0ak+FfvZisV7dVi5JemJ1ie66fpEKRx9rEHr/azuUGvDpxnOmeDK206fk6qG3disrLaDf3Xq2puRldDju85m+d/U8+Xyme1/Zoey0YLeBMJk553THE+sVbnF65ovnKS8rVetLqvXOviqt2Vupx1bsld8n/edV87q89rXt5dpZXq87r13QbUg9N7aN0S9e3anL5o3X3PHZJ+JLAgDEEKaGCbNohWdneZ2+8Ngaffuqk/V/RSV6a8cR5Wak6KtLZ2tSTrr+7an1uuznr+tH15yqpaeM1cHqo3pqzX599IxCjc5M9WRsF80p0J/e2a/7P7G4xx/UPp/pu1edoqqGZv3P37bq3Bl5mjdxpCfjOZEaQy1xWYP2+6J9enVbub51xcltYfTs6aN19vTRkqTv/nWT/vf1XVoyq0Dvn9tx0+kH39ytgqxUXRq7+7M7/3bpHL2ytUx3PLlOT33mnC57LQ5HoZaIlu+s0NnTRw/5eA9WH9XqPVVat79KclL2iKBGjgi2/R70m5yTIs4pEvt9Rn7mgHY9GKjlO4/oB89uUUFWmn5y7YJeG/ACGFo2VGtcFi9e7IqKiobkvYez/VVHdcXP39CR+mblZ6XqtvOn6aNnFio9JZp791U06HO/Xa21JdW68Zwpaok4/Xb5Xr3yzxdoYo5339idc/2auqtqaNbFd76mzNSA/vL584b0B0BdU1g/eX5bdIH/+dN6/cEXaolo7b4qbS2t1fbSOm09VKttpbU6Ut+sBZNG6bJ543TJvLHH9We8v+qoLv7Ja5o3YaR+c8uZ3TZ4bQq36Oq739KhmkY9+8XzVBCbRi0uq9P7fvyq/t/7Z+kLF83s9X3+vPaAPv/YGv37ZXN0y3nTBjzOE6k5HNHnH1ut5zaW6vPvnaGvfGD2CX3/+qawnlhdouW7KrRmT6UOVDdKklL8Psmi4+tLWtCnn1+/qEv4Haw9R+r1X8u26NmNh5SflaojdU1aWJijBz55ukamx3caH0D/mdkq59zibo8RpoafTQdqtGF/ta5YML7bqkhzOKLvLdush97aLUn64KIJ+vFHFpzYQfbije2H9fFfLdcnz56sb115ypCMYdWeSn35d++opLJBAb9PkYjTBxdN0OcunNlhinR7aa3+b1WJnlxdosN10cXgGSl+zRyTpVljMpWXmapXt5Vr44EaSdKpk0bpsnlj9ZHFkzQqPaXPcTjndMOvVmjN3ko9+6Xzew10xWW1uuxnb+iMqbl6+MYz5POZ/v2P6/X7ohK9dcd7lddH5dE5p1seLtJbO47ob1/u/b1a7T3SoJe3lik3I0UTckZoYs4I5WemHveat30VDXp+U6kk6WNnFXZZ3ydF//5+7rer9bdNpZozLltbDtXosU+dpbOmjT6u9xyISMTpyTX79cNnt6istkkTRo3QwsJRWlSYo0WTczR3XLZSAj41hlpUfTTU9ivc4uSzaAXWZ1LESf/5l01av79a37riZN0wwO2kulPTGNJdLxXroTd3K+A3fXrJdN1y3jS9vLVMX3r8HU3Lz9AjN53RFrTjJdQS0V0vFWtB4ShdOLsgrp97ODlQdVS5GSlxvdsZyYUwlaCeWX9Q97++Uz/+yAJN7bSOaah9+8+b9MCbu/TQjafrghP4DTrcEtFdLxfr5y8Va2x2mu68boEKc9N17ys79NsVe9UScfrgwgmaP2mUnlxdojV7qxTwmS6aU6CrF07QKRNGasKoEV3CxO7D9Vq24aD+uu6gNh6oUVZaQJ+5YIZuPGdKr9+cf7N8j/7tqQ367tWn6GNnTu5z/L9+e4++/scN+vrlc3XNook6679e1OXzx+lHHz61X1//gaqjev+PX9WiyTl65KYzegxFq/ZU6Jev79JzGw8p0ulbQErAp0k5I/SRxZP0yff0/vVFIk7r9lfrhU2len5TqbaW1rYdm1mQqR9cM7/DovnmcESf/e1qPb+pVP/xD3P14cWTdPnP31BjqEXPfPG8fgXUVofrmiSpz5DZqmh3hb79l01aV1KtUyeN0jcun6vTJuf0/cIeNDSH9fnfrtGLW8p0+5Lp+urFswe8rVSr5zeV6l+fXKcj9c26ZtFE/dPFs9tu8pCi/0G59ddFGp2ZokdvPlOTR/f+770l4vSb5Xv0+Ip9+vQF0/UPp47v9rzqoyF95jer9GbxEaX4fXr0ljN1xtTc4/oahivnnB58c7e+t2yzpudn6t6PL4r75vLov9bM8W68SYkwhROuMdSiK+56Q5UNIT33pfOVm9H7D8lVeyp1z8vFen37YQX9prSgX6kBn1KDfmWnBfS+OWN01cIJvVZb9hyp15d/945W763S1Qsn6FtXnqzsdnc3ltY0toWq5nBEMwsyde3pk3TVwgn9/oEsSZsP1ui/n9uqF7eUaWx2mr78/pn60KKJXbrO76to0MV3vqZFhTn69c09B5v2nHP61COr9Nq2cl2xYLz+sKpEy75w3oAWlT/y9936xp826p8vnq2Fk0YpJeBTasCvlIBPxWV1+uUbO7Vmb1W0lcZZk3Xt4klqCkdUUtmg/VVHtb/yqNaVVOvvO49obHaavvi+mfrwaR2/vl2H6/XEqhI9tWa/9lcdlc+iNyq8f+4YvW/OGO06XK+vPbVeh2oaddM5U/WVD8yS32f67G9W64XNZfrWFSfrk++ZIinaI+2D976pi04ao3s/vqjHP6fGUIuKdlfq9e3lem37YW0+WCMzacGkUfrA3LF6/9wxHfajDLVEtPtwvbaW1uqZDYf013UHNTY7TXdccpKuOHX8cQef9sItEX3z6Y36zfK9unLBeP3wmvndVuN6UtsY0nf+skm/LyrRnHHZ+tE183XKhO7XGr6zr0o3PrhCfp9PD/7j6T2uSVy1p0Jf/+NGbTpYo9EZKTpS36wPLZqob115sjJTjy2TLals0E0PrdTO8nr9+2Vz9Ou39+hwXbOe+PTZmlGQ1eXzOuf06Nt7tHxXhS6fP17vm1MQl50WvFTfFNYdT67Xn9ce0Lkz8rTxQLXCLU7/85FT9YGTx3r63g3NYa0vqdaafVVaX1KtqXkZ+vhZkzV2ZHwqi42hFu050qBdh+u163C9dh+u196KBk3Jy9D5M/P0nul5nk0LH6puVFlto2qOhlXbGFJNY0i1jWFNyk3X4sk5XdbvOue05VCtnl57QE+/c0ANzWFdf0ahPnH2lG7/PFoiTit2VWhdSZUuP3W8Jowa4cnXMVCEKQyJTQdqdNXdb+rCk/J138dP6/JD0jmnt3Yc0V0vFevvO48oJz2oKxdMkN9nagq3qDEUUVM4ogNVR9v2B1w8OUdXLZygS04Zq/K6Jq3eU6XVeyu1ek+ldh6uV1ZaQN+9ep6u6OF/4pJUVtuoivpmzR6TNaj/HS3feUTff3aL1uyt0oyCTC2YNEpltU0qq2lUeW2TKhqalZES0HNfPn9A3wyO1DVp6U9fV3ltk86alqvHbz17QOOKRJyuvf/vWtnDBtWFuem6+dypuua0icpI7fkelLd3HtEPn92i1XurNC0vQ196/yzVN4X1h1UlWrWnUj6TzpuZrysXjNeFswuU0ykw1zaG9INnt+jRt/dqUu4ITcpJ11s7jujbV56sT3SaFvvFqzv0X89s0feunqePduqVtnZfle5+OXpHa1M4oqDftHhyrs6blaeWFqfnN5dqXUm1JGlafoZOGpulHWX12nm4TqGW6Pe3EUG/blsyTbeeP61t/WG8OOd076s79MNnt+qUCdm6YFaB5o7P1snjszUpJ73H0Pb2ziP6yu/X6mD1UX36gun64kWzlBLoPZwUl9Xqhl+t0MHqRk3Lz9B5M/J07sx8nTUtV03hiL7/zBb9YVWJxman6euXz9UHTh6jn7+4XXe9XKyJOen66XULtLAwR+tKqnTzw0VqDLXovo+fpnNm5GlfRYOuvuctpQZ8euqz71FB1rEfcnVNYf3LH9bpr+sPKiPFr/rmFhVkpera0yfp2tMn9WstYXltk55aU6LC3HRdeFJBr6GzJeIUcU7BQYS1neV1uv3RVSouq9NXPjBbn14yXQeqj+ozv1mtdSXV+swF0/WVD8zucPPDkbomrdhVoVDE6QNzxwxoStA5p9V7K/XntQe1cneFthyqVUus7Dth1AgdqD4qv5mWnjJWN54zRYsKcwb8/aemMaTnN5bqz+sO6I3thxVuV1bOy0zVhJwR2lFWp7qmsHwWXZZw3sx8LZmVrwWTRg36Ro/islr9+PltWrb+UK/nTcvP0OmTc7V4So5Kaxr1p3cOaHtZnfw+07kz8pQa8OmFzaXymenSeeN007lTNX/CSK3aW6m/rD2gZRsOqbw2WnlubQHz6Qumd/jP8VAYdJgys6WSfirJL+mXzrnvdzpuseOXSmqQ9I/OudW9fU7CVHJo/SF56byxystMVcDnU9BvCvhNbxRH9xIck52qT53XcaF9ZyWVDXp67QE9tXq/tpfVdTg2OiNFCwtztGjyKF21YILGn8D/xTjn9NzGUt35wjZVHw2pICtV+VlpKshOVX5mqi6aU6D5E0cN+PO+tq1cn350le75+GlaMit/wK9vCrdo44EaNYejgbQ59isrLaBzZuT1+5uqc04vbC7Tj57bom2l0T/3mQWZ+tBpE3X1wgkdpqJ6snznEd3x5HrtOlyv71x1im44q+t0ZyTi9MkHV2jl7gr9+XPnauaYLL2zr0o/fWGbXt5arlHpQV21YIKWzMrXmdNyu/w9OVh9VC9sKtXfNpVqX0WDpudnatbY6Lq3WWOyND0/0/O1Mk+vPaB7Xi7W9rK6th+imakBzRyTqZz0FGWlBWK/gqqsb9bvivapMDddP/7IqTptcv+n1spqG/X0Owf0+vbDWr7riBpDEQV8ppSAT6GWiG4+d5o+/94ZHYLyil0V+vLv3tGhmkZ9ZPEk/XHNfuVmpOihG0/XzDHHqlDrSqp07S/e1vSCDP3u1rOVkRrQttJa3f7oKu0+XK+vLj1JN587VS9vKdNjK/bqlVjLlvNn5uuy+eP0vjljulShD1Yf1S9e3anHVuxVU2xh/8gRQV0+f5w+uGhCW6jYV9Gg17cf1hvF5Xqz+Ijqm8KxcJytk8Zlac7YbOVmpGhHeZ22ldZpe2mttpXV6lB1oybmpGt6foamF2Rqen6mIhGn//zrZqUEfPrZdQt17sy8tvE0hlr0rT9v1GMr9uncGXn68OKJWrm7Qst3VnT43pKdFtCHTpuoj51Z2G2lrtXuw/V6as1+/fGd/dpzpEFpQZ8WT87VwsJRWlg4SqdOHKXRmanaV9GgR/6+W4+v3KfaxrDmTRip82flyRT9t+gU/TvjN1NGakAZqdG/LxkpAdU1hbVs/UG9srVczS0RTRg1QpfOG6tTJozUtLxMTclLb+sx2HozzWvbD+v17eVau69KESflZqTogtn5eu9JBTp/Vv6Agsm+igbd+cJ2PbWmRCOCft14zlQtmDRKWWkBZY8Ito1zR3mdVu6uVNHuChXtqVT10eg2ZKdPydEVCybo0lPGtlWt9lU06OG3dut3K/eptims7LSAahrDSg349N6TCnTZ/HE6aWy27nm5WE/G/r5+8aKZ+uiZhYMK2YMxqDBlZn5J2yS9X1KJpJWSrnfObWp3zqWSPq9omDpT0k+dc2f29nkJU8mhJeL0ld+/ozd3HFG4JaJQi1OoJaJQS0QTc9J125Jpuua0if2eGnHOadPBGr28pUzjR43QaZNzVJib/q6cf+9LczjSZ6XiRGmJOL2ytUx5mamaP3HkgP+8G0Mt2lfR0OEHd2dlNY1a+tPXlZ+ZqvGj0tpC1KfOm6ZPvmdKhymq4awx1KLtpXXaeKBamw7WaHtpXds0SG3s94hzuv6MQn3t0jm9Vgf70hRu0ao9lXpj+2EdrmvSbUuma3oP64Gqj4b073/coD+vPaD5E0fql59c3KH61OqlLaW65eEinT8rX1ecOl7/9tQGZaQG9PPrF7a18mhVUtmg36/cpydWH5vuPXPqaF188hgtKMzR74v26Q9FJYo4p6sXTtBtS6bpQFWjnlxdomc3HlJjKKLJo9NlknYfaZAkjRuZpvNm5ikvM1XbSmu1+WBtl71Lg37TtLxMzRyTqfGjRqikskE7yqLTXc2xDcBPnTRK935sUY//ufrdyr36+p82qjkcUUaKX4un5OrMabk6c+poNYVb9NiKfXp2w0GFWpzOmJKrS+aNVaglouqjIVU1RG9M2FfRoLUl1TKT3jN9tK5eOFFLTxnb69/VhuawnlqzXw+/tVvby+pkOrZ+yKQO1ab2CrJSddn8cfqHU8dr4aRR/f43WNXQrNe2H9ZLm0v1yrZyVTWEFPBZhxtxYjlOAb9pVHqKctNTlJORotyMoCrqm/WHVSXymekTZ0/W7Uum96sNTyTitKO8ThmpgV7/g1vXFNYfivZpXUm1lszO10VzxnT581tfUq3vLtukt3dWaGpehhZPztHIWNuSkenR32cUZOrk8d624xlsmDpb0n845y6OPf5XSXLO/Ve7c34h6RXn3GOxx1slXeCcO9jT5yVMAejspS2luumhondliOov55xaIm5I1hs557S2pFqzx2T12rbkt8v36mtPrZcUrSrc9dFFvVYhnXPasL9Gz208pOc2Hmqr8KT4ffrI6RN12/nTu6x3rGsK65n1B/XndQcV8JnOm5mn82bma3p+RpegUH00FG1VUtesGQUZmjw6o9vqRLglopLKozpU06iFhaP6/E/avooGVTY0a+647G6vx+G6Jv1hVYkeW7FXe2Jhz++zth/kozNS9L65Y3TlgvEaNzI+FfFIxOloqEV1TWHVNYVV3xSWyTR3fPagp+laIk5r9lbqxS1l2nukQTKp9TOamULhiCobmlXZ0KyK+pAqG5plkq49fZI+/96ZcVvvdTycc3ppS5nufrlYB6oaVX00pKPtttC64azJ+s5V3t49PtgwdY2kpc65W2KPb5B0pnPuc+3O+Yuk7zvn3og9flHSvzjnijp9rlsl3SpJhYWFp+3Zs+f4vyoACWnTgRoVjk5PuBD1bvPgm7tUfTSkz144Y8DTKjvK67RqT6WWzMrv11TwcBeJOJXVNikzLaCMFH9CVsK7E4k4hSNu2FTIO2sKt6jmaFjVR0NKT/F7vsSjtzDVn+9W3f2t6ZzA+nOOnHP3S7pfilam+vHeAJIM2+EMDzeeM/W4Xzs9P7PH6cZ3I5/PhrQqM1R8PlPKMN5NITXgV36WX/lZ3uz+MRD9iZslkia1ezxR0oHjOAcAACDh9CdMrZQ008ymmlmKpOskPd3pnKclfcKizpJU3dt6KQAAgETR5zSfcy5sZp+T9JyirREecM5tNLPbY8fvk7RM0Tv5ihVtjXCjd0MGAAAYPvq1wtM5t0zRwNT+ufvafewkfTa+QwMAABj+hucSfQAAgHcJwhQAAMAgEKYAAAAGgTAFAAAwCIQpAACAQSBMAQAADAJhCgAAYBAIUwAAAINAmAIAABgEizYvH4I3NiuXtOcEvFWepMMn4H0wcFyb4Y3rM3xxbYY3rs/wNZhrM9k5l9/dgSELUyeKmRU55xYP9TjQFddmeOP6DF9cm+GN6zN8eXVtmOYDAAAYBMIUAADAICRDmLp/qAeAHnFthjeuz/DFtRneuD7DlyfXJuHXTAEAAHgpGSpTAAAAnknYMGVmS81sq5kVm9kdQz2eZGZmk8zsZTPbbGYbzeyLsedzzex5M9se+z1nqMeazMzMb2ZrzOwvscdcn2HCzEaZ2R/MbEvs39HZXJ/hwcy+HPu+tsHMHjOzNK7N0DGzB8yszMw2tHuux+thZv8aywlbzezi433fhAxTZuaXdLekSyTNlXS9mc0d2lEltbCkrzjn5kg6S9JnY9fjDkkvOudmSnox9hhD54uSNrd7zPUZPn4q6Vnn3EmSTlX0OnF9hpiZTZD0BUmLnXOnSPJLuk5cm6H0kKSlnZ7r9nrEfg5dJ+nk2GvuieWHAUvIMCXpDEnFzrmdzrlmSY9LunKIx5S0nHMHnXOrYx/XKvqDYIKi1+Th2GkPS7pqSAYImdlESZdJ+mW7p7k+w4CZZUs6X9KvJMk51+ycqxLXZ7gISBphZgFJ6ZIOiGszZJxzr0mq6PR0T9fjSkmPO+eanHO7JBUrmh8GLFHD1ARJ+9o9Lok9hyFmZlMkLZS0XNIY59xBKRq4JBUM4dCS3Z2Sviop0u45rs/wME1SuaQHY9OwvzSzDHF9hpxzbr+k/5a0V9JBSdXOub+JazPc9HQ94pYVEjVMWTfPcdviEDOzTElPSPqSc65mqMeDKDO7XFKZc27VUI8F3QpIWiTpXufcQkn1YtpoWIitvblS0lRJ4yVlmNnHh3ZUGIC4ZYVEDVMlkia1ezxR0dIrhoiZBRUNUr9xzj0Ze7rUzMbFjo+TVDZU40ty50i6wsx2Kzol/l4ze1Rcn+GiRFKJc2557PEfFA1XXJ+h9z5Ju5xz5c65kKQnJb1HXJvhpqfrEbeskKhhaqWkmWY21cxSFF1g9vQQjylpmZkput5js3Pux+0OPS3pk7GPPynpTyd6bJCcc//qnJvonJui6L+Vl5xzHxfXZ1hwzh2StM/MZseeukjSJnF9hoO9ks4ys/TY97mLFF0TyrUZXnq6Hk9Lus7MUs1sqqSZklYczxskbNNOM7tU0XUgfkkPOOe+O7QjSl5mdq6k1yWt17E1OV9TdN3U7yUVKvpN6cPOuc4LB3ECmdkFkv7JOXe5mY0W12dYMLMFit4ckCJpp6QbFf3PMNdniJnZtyRdq+hdy2sk3SIpU1ybIWFmj0m6QFKepFJJ35T0R/VwPczs3yTdpOj1+5Jz7pnjet9EDVMAAAAnQqJO8wEAAJwQhCkAAIBBIEwBAAAMAmEKAABgEAhTAAAAg0CYAgAAGATCFAAAwCAQpgAAAAbh/wO1XxH/bimehAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "errors = []\n",
    "for nb_sample in range(1, 100):\n",
    "    \n",
    "    rewards = np.zeros((nb_actions, nb_sample))\n",
    "\n",
    "    for a in actions:\n",
    "        for play in range(nb_sample):\n",
    "            rewards[a, play] = get_reward(Q_star, a, var=1.0)\n",
    "\n",
    "    Q_t = np.mean(rewards, axis=1)\n",
    "    error = np.mean((Q_star - Q_t)**2)\n",
    "    errors.append(error)\n",
    "    \n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(errors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot should give you an indication of how many samples you at least need to correctly estimate each action (30 or so). But according to the central limit theorem (CLT), the variance of the sample average also varies with the variance of the distribution itself.\n",
    "\n",
    "> The distribution of sample averages is normally distributed with mean $\\mu$ and variance $\\frac{\\sigma^2}{N}$.\n",
    "\n",
    "$$S_N \\sim \\mathcal{N}(\\mu, \\frac{\\sigma}{\\sqrt{N}})$$\n",
    "\n",
    "**Q:** Vary the variance of the reward distribution (as an argument to `get_reward`) and re-run the previous experiment. Do not hesitate to take more samples.  Conclude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.18921383192673544\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFlCAYAAAApo6aBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAstklEQVR4nO3deZxcZ33n+8+vqnpvLd1SS9ZqSbYw3m0sjIGEmBhiTyDYyQsSc4fEyZBLkiEJk8lkBpLMJZO5JGRuklcYLiTxAMEsQ+KwxL7sRhhsFhvkBSNb1mJbslpra+lW71s994+qbrXklrV0HbWi83m/Xn6dqlPndP26H3X7W8/znOdESglJkiTNXGG2C5AkSTpfGKwkSZJqxGAlSZJUIwYrSZKkGjFYSZIk1YjBSpIkqUZKs10AwMKFC9OqVatmuwxJkqSTeuSRRw6klDqme+2cCFarVq1iw4YNs12GJEnSSUXEjhO95lCgJElSjRisJEmSasRgJUmSVCMGK0mSpBoxWEmSJNWIwUqSJKlGDFaSJEk1YrCSJEmqEYOVJElSjRisJEmSasRgJUmSVCO5CFaj42Xuf3o/Ow8NzHYpkiTpPJaLYDUwPM6vffyHfP2pfbNdiiRJOo/lIlgRlU1KaXbrkCRJ57VcBKtCzHYFkiQpD3IRrCIqyapsj5UkScpQPoJVdWuukiRJWcpHsJqYYzW7ZUiSpPNcLoJVoZqs7LGSJElZykWwmuAcK0mSlKVcBKvwqkBJknQW5CJYHR0KtMdKkiRlJxfBaqLDqmyukiRJGcpHsHLyuiRJOgvyEayq2+SCC5IkKUP5CFaT9wqc3TokSdL5LSfBysnrkiQpeycNVhHxsYjYHxEbp+z7fyLi6Yh4IiK+EBHzp7z2nojYFhGbI+LmjOo+bRGuvC5JkrJ1Kj1WHwduOW7ffcAVKaWrgC3AewAi4jLgduDy6jkfjohizaqdgcChQEmSlK2TBquU0gPAoeP2fT2lNFZ9+hCwvPr4VuAfU0rDKaXngG3A9TWs94wVIpy8LkmSMlWLOVb/DvhK9fEyYOeU1zqr+14gIt4RERsiYkNXV1cNynhxEa5jJUmSsjWjYBURfwSMAZ+e2DXNYdPGmZTSnSmldSmldR0dHTMp45QE4VCgJEnKVOlMT4yIO4A3Ajelo5fbdQIrphy2HNh95uXVULiOlSRJytYZ9VhFxC3AfwHelFIamPLSvcDtEdEQEauBtcAPZl7mzBUCLwuUJEmZOmmPVUR8BrgRWBgRncB7qVwF2ADcV10j6qGU0m+mlJ6MiLuBp6gMEb4zpTSeVfGnIwjKjgVKkqQMnTRYpZTeOs3uj77I8e8D3jeTorIQ4XILkiQpW7lYeR0mlluQJEnKTm6CVYBDgZIkKVO5CVY4FChJkjKWm2A13QJbkiRJtZSbYFUoBMkuK0mSlKHcBKvKHKvZrkKSJJ3P8hOsvAmzJEnKWH6CFU5elyRJ2cpPsHIdK0mSlLEcBSucvC5JkjKVn2CFQ4GSJClb+QlWLhAqSZIylptgVYjwljaSJClTuQlWAU5elyRJmcpPsIpwKFCSJGUqR8EKFwiVJEmZylewMldJkqQM5SdY4U2YJUlStvITrMLJ65IkKVu5CVYFJ69LkqSM5SZYBbiOlSRJylRughUOBUqSpIzlJlgFmKwkSVKmchOsChGuYyVJkjKVm2AVAeXybFchSZLOZ/kJVthjJUmSspWfYOXK65IkKWM5ClZB2WAlSZIylJ9gBXhZoCRJylJ+gpVDgZIkKWO5CVaV5RYkSZKyk5tgFeEtbSRJUrbyE6xwKFCSJGUrN8EKhwIlSVLGchOsCgHJLitJkpSh3AQrhwIlSVLW8hOsvAmzJEnK2EmDVUR8LCL2R8TGKfvaI+K+iNha3bZNee09EbEtIjZHxM1ZFX667LGSJElZO5Ueq48Dtxy3793A+pTSWmB99TkRcRlwO3B59ZwPR0SxZtXOQCHCYCVJkjJ10mCVUnoAOHTc7luBu6qP7wJum7L/H1NKwyml54BtwPW1KXWGXMdKkiRl7EznWC1OKe0BqG4XVfcvA3ZOOa6zuu8FIuIdEbEhIjZ0dXWdYRmnLvBOgZIkKVu1nrwe0+ybNs+klO5MKa1LKa3r6OiocRkvVIgwWUmSpEydabDaFxFLAKrb/dX9ncCKKcctB3afeXm14y1tJElS1s40WN0L3FF9fAdwz5T9t0dEQ0SsBtYCP5hZibVhh5UkScpa6WQHRMRngBuBhRHRCbwXeD9wd0S8HXgeeAtASunJiLgbeAoYA96ZUhrPqPbTEoQrr0uSpEydNFillN56gpduOsHx7wPeN5OismCPlSRJylquVl4vm6wkSVKG8hOswKXXJUlSpvITrBwKlCRJGctNsPKWNpIkKWu5CVaB61hJkqRs5SdYhVOsJElStnITrCCcYyVJkjKVm2BVCFwgVJIkZSo3wcqhQEmSlLX8BCuC5GCgJEnKUG6CVaFgj5UkScpWboJVEC63IEmSMpWbYIUrr0uSpIzlJlhV7hU421VIkqTzWW6CVSFcx0qSJGUrN8EqwlvaSJKkbOUnWOFVgZIkKVv5CVbhOlaSJClbOQpW9lhJkqRs5SdYEQYrSZKUqfwEK2/CLEmSMpafYIXLWEmSpGzlJlgVwqFASZKUrdwEK9exkiRJWctVsDJWSZKkLOUoWDkUKEmSspWfYIVXBUqSpGzlJ1g5FChJkjKWn2BF2GMlSZIylZtgVbDHSpIkZSw3wSoiKJeNVpIkKTu5CVZgj5UkScpWboJVeE8bSZKUsdwEq0KEuUqSJGUqN8Eq8JY2kiQpW/kJVoErr0uSpEzlKFgFycFASZKUoRwFK3C1BUmSlKUZBauI+L2IeDIiNkbEZyKiMSLaI+K+iNha3bbVqtiZCFwhVJIkZeuMg1VELAN+F1iXUroCKAK3A+8G1qeU1gLrq89nXeVegSYrSZKUnZkOBZaApogoAc3AbuBW4K7q63cBt83wPWqi4OR1SZKUsTMOVimlXcBfAs8De4CelNLXgcUppT3VY/YAi6Y7PyLeEREbImJDV1fXmZZxyoJwuQVJkpSpmQwFtlHpnVoNLAVaIuJtp3p+SunOlNK6lNK6jo6OMy3jlIVTrCRJUsZmMhT4OuC5lFJXSmkU+DzwKmBfRCwBqG73z7zMmQscCpQkSdmaSbB6HrghIpojIoCbgE3AvcAd1WPuAO6ZWYm1USkRkulKkiRlpHSmJ6aUHo6IzwKPAmPAY8CdQCtwd0S8nUr4ekstCp2paq4ipaOPJUmSaumMgxVASum9wHuP2z1MpffqnBJUe6xmuQ5JknT+ytXK6+BQoCRJyk5uglVhIljNbhmSJOk8lptgNTF53bWsJElSVnITrCaYqyRJUlZyE6y8ElCSJGUtN8Gq4FCgJEnKWG6C1USHlblKkiRlJT/ByqsCJUlSxnITrAre0kaSJGUsN8FqQtlcJUmSMpKbYBWOBUqSpIzlJ1hVt8lkJUmSMpKbYDV5SxtzlSRJykhugpW3tJEkSVnLUbCqbI1VkiQpK/kJVtWtHVaSJCkr+QlWE+tY2WclSZIykqNgVdnaYyVJkrKSn2DFxMrrs1yIJEk6b+UmWE0ut+BQoCRJykhugtXEUKC3tJEkSVnJT7DCmzBLkqRs5SZY4eR1SZKUsdwEq8LEWKAkSVJGchOsJmKVt7SRJElZyU+wcihQkiRlLH/BanbLkCRJ57HcBKuJOVZeFShJkrKSm2A1wXWsJElSVnITrGLyqkCTlSRJykZ+glV160igJEnKSm6C1eQcq1muQ5Iknb9yE6yO3ivQaCVJkrKRn2BV3ZqrJElSVvITrCaXW5jlQiRJ0nkrR8GqsnUoUJIkZSU/wWq2C5AkSee9GQWriJgfEZ+NiKcjYlNEvDIi2iPivojYWt221arYmXAoUJIkZW2mPVYfAL6aUnopcDWwCXg3sD6ltBZYX30+6wqT9wo0WUmSpGyccbCKiLnAa4CPAqSURlJK3cCtwF3Vw+4CbptZibVxdI7V7NYhSZLOXzPpsVoDdAH/EBGPRcRHIqIFWJxS2gNQ3S6a7uSIeEdEbIiIDV1dXTMo49QE3oRZkiRlaybBqgS8DPjblNK1QD+nMeyXUrozpbQupbSuo6NjBmWcosmhQEmSpGzMJFh1Ap0ppYerzz9LJWjti4glANXt/pmVWBsFJ69LkqSMnXGwSintBXZGxCXVXTcBTwH3AndU990B3DOjCmvk6MrrJitJkpSN0gzP/x3g0xFRDzwL/BqVsHZ3RLwdeB54ywzfoybCoUBJkpSxGQWrlNLjwLppXrppJl83C0cnr89yIZIk6byVm5XXJ9exMllJkqSM5CZY4TpWkiQpY7kJVpNDgc6ykiRJGclNsCpMXhY4q2VIkqTzWG6C1cRNmB0KlCRJWclRsKpsHQqUJElZyU+wqm69KFCSJGUlP8Fq4pY2s1yHJEk6f+UoWFW2ZbusJElSRvITrCYemKskSVJG8hOswnWsJElStnITrI7e0mZ265AkSeev3ASriZXXXcdKkiRlJT/BypswS5KkjOUmWE0wVkmSpKzkJlgVJiav22MlSZIykptgFU5elyRJGctfsJrdMiRJ0nksN8Hq6FDgLBciSZLOW7kJVhMrr3tLG0mSlJX8BCuHAiVJUsZyE6wm+qy8KlCSJGUlN8GqECc/RpIkaSZyE6wmbsLsHCtJkpSV/ASr6tZcJUmSspKfYOUCoZIkKWO5CVaT61jNch2SJOn8lZtgNcE5VpIkKSu5CVYxOclqVsuQJEnnsRwFq4mhQJOVJEnKRm6C1cQ6VmVzlSRJykhuglXgTZglSVK28hOsJu8VaLKSJEnZyF+wMldJkqSM5CdYeRNmSZKUsfwEq8mhQEmSpGzkJ1hVt3ZYSZKkrOQmWE3e0sZkJUmSMjLjYBURxYh4LCK+WH3eHhH3RcTW6rZt5mXOXLiOlSRJylgteqzeBWya8vzdwPqU0lpgffX5rJucvD7LdUiSpPPXjIJVRCwH3gB8ZMruW4G7qo/vAm6byXvUzORyC0YrSZKUjZn2WP0N8J+B8pR9i1NKewCq20XTnRgR74iIDRGxoaura4ZlnNzELW0kSZKycsbBKiLeCOxPKT1yJuenlO5MKa1LKa3r6Og40zJO2cRNmMv2WEmSpIyUZnDuq4E3RcTPAo3A3Ij4FLAvIpaklPZExBJgfy0KnamJHqvx8osfJ0mSdKbOuMcqpfSelNLylNIq4HbgmymltwH3AndUD7sDuGfGVdZAfbHyrY6arCRJUkayWMfq/cDrI2Ir8Prq81lXKhYoFoLhsfHZLkWSJJ2nZjIUOCml9C3gW9XHB4GbavF1a62+WGBkzB4rSZKUjdysvA7QUFdg2GAlSZIykq9gVbLHSpIkZSdXwaq+ZI+VJEnKTq6CVUOpaI+VJEnKTK6CVX2x4FWBkiQpM7kKVk5elyRJWcpVsKr0WBmsJElSNnIVrBrqnGMlSZKyk6tgZY+VJEnKUq6CVUNdgREnr0uSpIzkK1jZYyVJkjKUr2BV58rrkiQpO7kKVs6xkiRJWcpXsCq5QKgkScpOroJVqVhgvJxmuwxJknSeylWwKkYYrCRJUmZyFawKhaCcICXDlSRJqr1cBatiBAB2WkmSpCzkK1hVv1uHAyVJUhZyFawKhYkeK4OVJEmqvVwFq4mhQHusJElSFvIVrKo9VuP2WEmSpAzkKlgVJiav22MlSZIykKtgNdljZbCSJEkZyGewcihQkiRlIJ/Byh4rSZKUgXwFK68KlCRJGcpVsJpcx6o8y4VIkqTzUq6C1eTK686xkiRJGchVsCo4FChJkjKUq2BV9JY2kiQpQ/kKVvZYSZKkDOUqWBVcbkGSJGUoV8FqosfKoUBJkpSFfAUre6wkSVKGDFaSJEk1YrCSJEmqkTMOVhGxIiLuj4hNEfFkRLyrur89Iu6LiK3VbVvtyp2ZyXWsnGMlSZIyMJMeqzHg91NKlwI3AO+MiMuAdwPrU0prgfXV5+eEore0kSRJGTrjYJVS2pNSerT6uBfYBCwDbgXuqh52F3DbDGusGW9pI0mSslSTOVYRsQq4FngYWJxS2gOV8AUsOsE574iIDRGxoaurqxZlnNTEUGDZOVaSJCkDMw5WEdEKfA74DymlI6d6XkrpzpTSupTSuo6OjpmWcUqcvC5JkrI0o2AVEXVUQtWnU0qfr+7eFxFLqq8vAfbPrMTameix+vVPbJjlSiRJ0vloJlcFBvBRYFNK6a+nvHQvcEf18R3APWdeXm1N9FhJkiRloTSDc18N/DLw44h4vLrvD4H3A3dHxNuB54G3zKjCGpoarMrlNHnvQEmSpFo442CVUvoOcKJkctOZft0sTQwFQuXKwMIJy5ckSTp9uVp5vTSlh8oJ7JIkqdZyFayKBitJkpShXAWrqXOqxgxWkiSpxnIVrIphj5UkScpOroJVYcp3O+YNAyVJUo3lKlhN7bEyV0mSpFrLV7A6Zo6VyUqSJNVWroJVwasCJUlShnIVrKYOBXpVoCRJqrV8BavjbmkjSZJUS7kKVo11RX7llRcC9lhJkqTay1WwAvjJtR2Ac6wkSVLt5S5YFavf8XQ9Vvt7hxgYGWNsvExKBi9JknR6chisKt/ybR/6Lh/7znPHvHb9+9bzi3//fS7+o6/wf35iw2yUJ0mS/hXLXbAqTZnA/qdffIq/+vrmY17fuOsIAN/YtP+s1iVJkv71y12wmnplIMAHv7ltliqRJEnnm9wFq9JxwWqCc6okSdJM5S5YFaYJVof6RxgZP/YWN23NdWerJEmSdJ7IXbCarsfqZf/9Pv7XA88es29uk8FKkiSdntwFq+PnWE14fGfPMc+nP0qSJOnEchesSoXpv+XeodGzXIkkSTrf5C5YFU/wHXcPHBusnMouSZJOVw6D1fTf8uGBkWOee5GgJEk6XbkLVidabuH4YOW9BCVJ0unKXbA60eT10fFjg9TwWHna4yRJkk4kd8HqRD1WxxsZG8+4EkmSdL7JXbCaboHQ6Ry/YKgkSdLJ5C5YnWqP1fFDg5IkSSeTu2B1ojlWxxsvp8kJ7N0DI/QNjwFwsG+Yd3xiA4f6R17sdEmSlEO5C1ZTFwh9/WWLX/TYkeoE9mv+9D5e8z/uB+Afvrudrz+1j089tCO7IiVJ0r9KuQtWU5ex6pjTAMDC1oZpj/3so52Tjw/1j7B1Xy//7/3bANe5kiRJL5S7YFVXTVZ/cPMl1FeXYb9hTTvzm1940+X/+i8bSVMS1Ocf2zX5uGyykiRJx8ldsCoUgu3vfwPvfO3F1BUr861a6kt88/dv5M9+/soXHN8/cnTZhdaG0uTjnsGjt8D51EM7+Kuvb37BuSNj5WOOkyRJ57fcBaup6qo9Vo11Bdpb6lnW1jT52tzGSoj6px/unNzX1Tt89HHf0cd//C8b+eA3t73g6//GJzdw9X/7es3rliRJ56ZcB6uJ1dUb6ooAtE0ZDvz4v7segP/+xaeO7vve9snHXUeOBqsJ//VfNvJLf//9yef3b+4CoGcg+16rPT2DxwxbSpKksy/Xwaq+VPn2b7niAgDamusnX1u9oGXacxbPbeB1ly5i5+EBvvHUPn7vnx6ffO2TD+3g4ecOveCc5w72A3DP47vY2zN0zGvvvWcjdz7wDP/XPRsZnWZR0t6hUQZHxvng+q387beembamLft6eeWff9MrFSVJmmWlkx9y/nrnay/mZ69YwpXL5wHQ1nI0WE19PNXqhS1cuWw+39i0n1//xIZpjzkyNMqH7j86NPi/H97BnQ88w5d/vJfXXtLBP/za9YyMlXl67xHu+v7RMHTL5RfQ3lpPW3M985rq+PaWLn7jk49wUUcLz3RVwtm/vWElcxuPnWj/5O4eANY/vZ9ffuWq0/9BTPHIjkO0NdezpqMVgKHRce7esJO3Xr9ycuj0eCklIk5tfTBJks5nuQ5WrQ2lyVAF0FJfPOb1VQua2X5wgFeuWcD3nz0IwEUdrZPLNJzIuz/3BF/+8d7J53dvOLpsw0B1Mvx7793IZ36w85jzNuw4zF/ft4X2lnreddNa3nvvkwCToQrgG0/t47ZrlvFX923mQO8Iy9ua+Kv7tgBMLmhaeZ8xugdGWTq/Mm/soWcP8tWNe7lhTTsfuv8ZPvR/vIyVC5oBGB0vE9X3v/3OhwB47s9/lojgL7+2mY985zkWtDTwhquWTH79Xd2DLJ3XyB//y0a+u+0A9/+nG08arroHRniis4fXvKRjcl+5nI65zdCOg/001RdZNKfxRb8WwINbu0gJ9h0Z4i3rVpz0eEmSshZZzcuJiFuADwBF4CMppfef6Nh169alDRum7/0521a9+0tcvWI+97zz1eztGeLj39vOwMgYn6j2LG3449cxMDzOGz74IH/3tut41UUL+PKP9/LO//3otF/vr3/xav7j3T8CoKFUYHisTH2pMLn46FQvvWAOT+/tfdH66osFyikxVp6+3S5ZPIcPv+1l/Kd//hGPPd/Nv7niAh7Y0jV5dWNjXYGh0TIr2pt4+YXt3HbtMn7lYz94wde57Zql/MWbr+KWv3mQ5w7086arl/L2n1jNS5fM4T/+04/40o/38Hdvu47f/NQjAHz0jnXcdGllwdXd3YP85dc282uvXj0ZXIfHxvnNTz7C/Zu7+M5/eS0PbDnApj1H+ORDO/jCv38VD249wJuvW86r3v9NAB7+w5sYGSvzTz/cycDIOFv29fIXb76KJXMbKad0TAgEeP8vXMnLLmzj8Z3dvOW65S8IeePlxJ0PPMtrXrKQy5fOY2SszOcf7WTdqnbWLGxhZLzM3p4hPv3wDtYunsNPv3QRe3uG6Bseo6W+xCUXzJlst9HxMnuPDLFmYcsJw2Tv0Cg7Dg5wxbJ5x+z//jMHuXBB82Tg7Rse40tP7OYt16140ftYppT49pYuXn3xwml7DofHxnnrnQ/xGz91ETdffgFb9/Xy7S1dvP0nVp9xb+LoePmEvZQTNd37o928Zm3HCXt4a2l39yBL5jW+4Pv54fZDLG9rYsm8phOcqROxt1k6MxHxSEpp3bSvZRGsIqIIbAFeD3QCPwTemlJ6arrjz6Vgdbh/hKb6Io11R3uv/vq+LfzP9Vv5wO3XcOs1y6Y978Pf2sa3nu7iB9src6xe85IOnjvQx1fe9RqueO/X+L9vu4L7ntrHt7d0TZ7zhquW8KUn9rxoPa+/bDHPdvXx0iVzeeOVS/jOtgN8+uHnX3Dc373tOjbu6uGTD+140SUeIk68uOntL1/BbdcumwwsC1rqOXjcrXsWtjZwoO+FE/eb6oosmd9I56HByRtYr1nYwtUr5vOD5w6xq3tw8tifuWwxX39q3+TzicB5qlrqi8csgzGhrbmOwwOjzGks8ZNrF7KivZlt+/oYK6djfu5//IZLeWxnN196Yg+L5zZwZHCMwdEXfr1TsaajhYWtDdx8+QXsPDRASomh0TL/tKHSG/kbP7WGlvoS/9+PdvOL61bwvi9vAuB1ly7mtmuX8rlHOrl/cxe3XrOUXYcH2XtkiJGxMteunM+8pjquWj5/MuB9/HvbqS8V+NM3Xc7K9mb+x9c2s7ytic17e3n+0MDkz/CJP/kZrvqTytWo7/v5K7h86TxWLWjm21u6+OrGvbzmJR1sP9jPl57Yw9pFrVy7so36UoEL5jby4NYD3HhJB9v29/Hhb23j565eyobth3nzdctZ2d7M1v29jI0nnujs4Y1XL+GPvrARqAT2V6xewO7uQV6xpp0t+/r49zdeRM/gKK0NJeY11fH03l56Bkd5+ap2RsfLk8PdN1++mF3dQ3T1DrOne5CLF7Vyzcr5/GhnD1csm8unHnqehlKBD6zfyhuvWsLgyDg/sXYhrQ0lvvbkPr6xaR/FQvAHN1/Clcvm8ZWNe/jutoP8zS9dw56eIeqKwYr2ZgoRbNvfy0Udraxob2Z4rMwTnd1ccsEcRscT39naRXtLAxd1tPDk7iNccsEcntnfR6lYYNn8JgoFuLijlQN9I2zae4QL25vZd2SY+57axxuuWsIF8xp5Zn8fP9rZzUuXzOULj3XyzP5+lrc1cfHiVm5YvYBXX7yQPT2DbNnXR0tDpWf2/V/ZxB/c/FK27OtlQUs9DXUFChGsWdhK9+AI7S31bNnXx4ULmhkcGWf7wX5WLWihZ3CUNR0t7O4e4p7Hd3HhghZ+7uoldB4epPPwIPXFAtu6+uhoradjTgPtLQ2sWtDMnp4hFs1p4EedPfzuZx7jjlddyMLWBjbv6+XBLQfY3zvEP/zq9Syd38jmfb0cGRzlZRe28eCWA/zk2oUsmtvIwb5h+ocrtVyxbB6tDSV2V3/HR8bL3PP4Lq5cNp+ffukifrj9EAMj46zpaKEYwQXzGnlydw8r2pr56pN7ueWKC2iuL5FSYnB0nPFyYuehQeqKQXN9idXVDz47DvZTVyxwZHCUlQsq7dlSX2J4bJzWhhL7eoc53D9CW0s9m3Yf4fo17cxtrGO8nNh+sJ/GuiJNdUXmN9WxcXcPly+dx+7uQZbNb5r8YDWnscT/evA5Ll86l9ddupim+uJkr/qh/hHmNdVN3hatZ2CULzzWyaVL5rJqYeXvQCGYDKqj42XGy4neoTFSShwZGqXz8CDf3tLFr//kGuY31XGgb5jF1Q+LhQgaSgX6R8bpHhhhblMdnYcGWdPRwtN7eykVgm37+7h4USuL5jawftN+2lvq6RkY5RdetoxiISbfO6XEwf4RdhzsZ9n8ZvpHxtjdPcgli+cwMDLO0vlNlArBswf6GR0v85LFc9i05wgpwZXL57H9QD/PdPVx4yWL2HtkiCVzKyMIj+08zKI5jaxor4x2lMuJzz7ayfymOq5d2caClnoefu4QbS11XLJ4DgBHBsdobSxNezu5sfEy5VT50NRUX2RkrExrQ4mewcpoy8Q86HI5sat7kPFy4rvPHOCtL19JoRCMjpcpFYJygp2HBljR3sy+I0P0D4+xtvr+WZmNYPVK4E9SSjdXn78HIKX059Mdfy4Fq+kMjY7zuUc7JxvzRFJKXPLHX2VkvDw5lDbVtv29/MVXN7NpzxGuXdnGB37pGvpGxvizL23i4ecO8dyBfq5f1U7HnAZ2Hh7gtZcs4vde/5IXvM/3njnAY89301hX5GDfMH9w8yWT7/XD7Yf40P3b6Dw8yLb9fQDMaShxzcr5fOxXX04xgp7BUT6wfisPbu2aHGb8hWuX8We/cCWNdUU++p3n+OA3t9I9MMqbr1tOIWD9pv2TIetXX7WKwwMj3PP4bhpKBT73W6/ijR/8zjE1zm+uo/sUr4ZcPLeB+U31bN5X6a179cUL+O62gyc8fmqwuvGSDr61+WhoaqorMjpefkGPXntLPeWUpq3pJ9cupBDBt7d0MbexxJGhyn0hf+On1vD33372lL6H01EIOEGH41lXV4x/NTccn9o2/1rNa6o7J9a2m+i5PhP1xcLkh6eTKRbimCkKcPr//lsbSpP3aj3eyb5WS32RgdHxyQ+TJxotOJH5zXWTHw56h8aoLxWoKwSL5jayt2fomA9kcxtLjI4nIqAQccKaofJzKac0WVepEIyVE831xcnpIqf6PUKlPUfHK+cHMDg6ftLf66m/+1N/LlM/6E7Uc/zfiZXtzYyMlekbHjvm+5z6b6OprsjQ2NGf/cS6kR2tDfQOj9FYV+RQ/8gL/n1MaK4vMrexEmSnfjCf+FqNpSK9w2MUonKrupHx8uT7v3xVG//8m6968R/aDM1GsHozcEtK6derz38ZeEVK6benHPMO4B0AK1euvG7HjvPjiraDfcPVT5Ktp3Xe7u5BPvtIJ79140UvOvxyOk6lm39odJyNu3pYt6r9Befu7618mpqwcVcPo+Nlrl3ZxshYufrLW2ZhawPfe+YAOw4OMLexjusubGNeUx1fe3Iv166cz7L5TRzqH6FncJQFrQ3884advOqihfSPjLG8rYnlbc2Tn1a/unEvP3/tMsbKia37+ti87whvunoZW/b1snlvL69Y087iOY1895kDvHxVO411RdZv2kdjXZFHdxzml66vzLXqGxpjw/bD3LBmAY93dvOzV1zA0FiZHzx3kIs6Wtm8t5eXr2rnW1v288arllIqBAf7R1jY2sCTu3torCtyUUcru7oH2X9kiCXzmhgZK9M7PMrlS+cxPFap9cedPby2Omw4v7nyR6C1ocRFHa10D47SPzzG/U/vZ+n8Jrr6hvm3r1jJnMZKT01KiWIh2H9kmNbGEpctmUupGIyNp8qn3wI8s7+f9pZ6Og8PsLK9mR9sP8SOgwMsmtPAtSvns793mOHRMg9uPcDK9ibaWurZvLeXK5bN49qV8/ncI53UlwqUU2XOIMCPOnuY01ji8qXzeM3ahXx1414KhWBuYx2XLpnDtv19fO+Zg/zqq1fRMzBKe0s9X/rxHhrrilzc0cojOw5x7co27nl8F33D41y7cj4/d9VSunqH+fTDOxgcHedtN1zIA1u66OobprmuxLK2JtYuauWBLV0cHhjl6hXzONA3wjNdfcxpKHHpkrn0DI6ydlErX9m4l2Ih6B4YoRDBG69ewrNd/bzpmqV88Ud7uGblfAoRtDXXsaC1gdaGEp2HBwB4ZMdhVi9soW9ojMd2dnPDmgXs6h7kcP8IzfVFDvaPsKd7kJaGEnXFAqsXtnCgb5im+iKXLZnL9oP9PNtVmedXXyzQUFdk0ZwGth/op39knP1HhpjbVMcr1yygs3uQXYcrw5N9w2M0lAp0zGlgbDxRKgY7Dg6wbH4TPYOjLG9rYnf3II/v7GZNRyvXr26nd2iM7z1zgPpSgVIhWN5WGSLeuKuHp3Yf4eoV82hpKNE9MMqahS08e6CfumKwsr2FB7Z28djz3fzUSzoYHa98ym+uL9LVO8yajlYWz22ge2CUK5bN48GtXTz6/GHKZVje1kRrY4kjg2OUU+KyJXN5+LlDFAtw4YKWyv+w6ops3NXD/OZ6LupoYf+RYUbHy/QOjzE6XqalocTcxjr2Hxmiq2+YK5fNp3twhGe7+rly2bxKL8rgKEcGRykUgquXz2NBawNb9/UxMDLGj3f1cKh/hI45DRzqH+GyJXMZT4liBHOb6hgZK9NYV2CsnNiyt5dCIVi9oIVlbU089nw3zx8a4NIlc5nTWGJgZIy25np2HhqgY04DS+c3sfPQIMUCFAsFdncPMq+pjo45DTy99wgH+irvOzJWpq4YDI+WuWBeI831RQ71V35fO+Y00FRf5MjQKH1DYzTXFzk8MMrK9mYOD4yQUmUdw4ZigZ+5/AL6h8fo6htm464eFrY2TPY+tTXX0z9SCR0TYadYCG68ZBHfeGof5ZRYvbCFzsODDI6MUywG9cUCgyPjHB4Y4aJFrWzac4SFrQ0UI1je1sS2rj66eod5xeoFXLFsLlv399HWXMfjO7sZHa/8DOtLBZrqi9QVgyuWzuPZA/1EwJJ5jRzuH6VveIxNe44wv7mea1fMp1gInujs5iUXzGFgeJx9R4ZobSxxuH+E8ZToaG1k+8F+Fs1p4MjQGE11RQ71D1NfKtBcX2LVgmbGyolySnQeHqSxrsieniHmNpZoqqsE2/piYfJv/MhYmfnN9QyNjrOgtX4yjA6Plif/7RzsGyYiKFZ7pZrqiixva6JveJzd3YMUAuZXL/IaLycGRsY50DfMeEqsbG/m9pev4MITXNlfK7MRrN4C3HxcsLo+pfQ70x1/rvdYSZIkTXixYJXVOladwNTLtJYDuzN6L0mSpHNCVsHqh8DaiFgdEfXA7cC9Gb2XJEnSOSGTdaxSSmMR8dvA16gst/CxlNKTWbyXJEnSuSKzBUJTSl8GvpzV15ckSTrX5PpegZIkSbVksJIkSaoRg5UkSVKNGKwkSZJqxGAlSZJUIwYrSZKkGjFYSZIk1YjBSpIkqUYMVpIkSTUSKaXZroGI6AJ2nIW3WggcOAvvo1Nnm5ybbJdzk+1y7rFNzk1Zt8uFKaWO6V44J4LV2RIRG1JK62a7Dh1lm5ybbJdzk+1y7rFNzk2z2S4OBUqSJNWIwUqSJKlG8has7pztAvQCtsm5yXY5N9ku5x7b5Nw0a+2SqzlWkiRJWcpbj5UkSVJmchGsIuKWiNgcEdsi4t2zXU+eRMSKiLg/IjZFxJMR8a7q/vaIuC8itla3bVPOeU+1rTZHxM2zV/35LSKKEfFYRHyx+tw2mWURMT8iPhsRT1d/Z15pu8yuiPi96t+ujRHxmYhotE3Ovoj4WETsj4iNU/addjtExHUR8ePqa/8zIqLWtZ73wSoiisCHgH8DXAa8NSIum92qcmUM+P2U0qXADcA7qz//dwPrU0prgfXV51Rfux24HLgF+HC1DVV77wI2TXlum8y+DwBfTSm9FLiaSvvYLrMkIpYBvwusSyldARSp/Mxtk7Pv41R+plOdSTv8LfAOYG31v+O/5oyd98EKuB7YllJ6NqU0AvwjcOss15QbKaU9KaVHq497qfyPYhmVNrirethdwG3Vx7cC/5hSGk4pPQdso9KGqqGIWA68AfjIlN22ySyKiLnAa4CPAqSURlJK3dgus60ENEVECWgGdmObnHUppQeAQ8ftPq12iIglwNyU0vdTZYL5J6acUzN5CFbLgJ1TnndW9+ksi4hVwLXAw8DilNIeqIQvYFH1MNvr7Pgb4D8D5Sn7bJPZtQboAv6hOkT7kYhowXaZNSmlXcBfAs8De4CelNLXsU3OFafbDsuqj4/fX1N5CFbTjZ96KeRZFhGtwOeA/5BSOvJih06zz/aqoYh4I7A/pfTIqZ4yzT7bpPZKwMuAv00pXQv0Ux3aOAHbJWPVOTu3AquBpUBLRLztxU6ZZp9tcvadqB3OSvvkIVh1AiumPF9OpStXZ0lE1FEJVZ9OKX2+untftVuW6nZ/db/tlb1XA2+KiO1UhsZ/OiI+hW0y2zqBzpTSw9Xnn6UStGyX2fM64LmUUldKaRT4PPAqbJNzxem2Q2f18fH7ayoPweqHwNqIWB0R9VQmtN07yzXlRvWKi48Cm1JKfz3lpXuBO6qP7wDumbL/9ohoiIjVVCYX/uBs1ZsHKaX3pJSWp5RWUfl9+GZK6W3YJrMqpbQX2BkRl1R33QQ8he0ym54HboiI5urfspuozBO1Tc4Np9UO1eHC3oi4odqevzLlnJop1foLnmtSSmMR8dvA16hc0fGxlNKTs1xWnrwa+GXgxxHxeHXfHwLvB+6OiLdT+eP1FoCU0pMRcTeV/6GMAe9MKY2f9arzyTaZfb8DfLr6IfBZ4NeofAC2XWZBSunhiPgs8CiVn/FjVFb0bsU2Oasi4jPAjcDCiOgE3suZ/c36LSpXGDYBX6n+V9taXXldkiSpNvIwFChJknRWGKwkSZJqxGAlSZJUIwYrSZKkGjFYSZIk1YjBSpIkqUYMVpIkSTVisJIkSaqR/x9s32BZR+R1cAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "errors = []\n",
    "for nb_sample in range(1, 1000):\n",
    "    \n",
    "    rewards = np.zeros((nb_actions, nb_sample))\n",
    "\n",
    "    for a in actions:\n",
    "        for play in range(nb_sample):\n",
    "            rewards[a, play] = get_reward(Q_star, a, var=10.0)\n",
    "\n",
    "    Q_t = np.mean(rewards, axis=1)\n",
    "    error = np.mean((Q_star - Q_t)**2)\n",
    "    errors.append(error)\n",
    "    \n",
    "print(error)\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(errors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A:** the higher the variance of the distribution, the more samples we need to get correct estimates.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bandit environment\n",
    "\n",
    "In order to prepare the next exercise, let's now implement the n-armed bandit in a Python class. As reminded in the tutorial on Python, a class is defined using this structure:\n",
    "\n",
    "```python\n",
    "class MyClass:\n",
    "    \"\"\"\n",
    "    Documentation of the class.\n",
    "    \"\"\"\n",
    "    def __init__(self, param1, param2):\n",
    "        \"\"\"\n",
    "        Constructor of the class.\n",
    "        \n",
    "        :param param1: first parameter.\n",
    "        :param param2: second parameter.\n",
    "        \"\"\"\n",
    "        self.param1 = param1\n",
    "        self.param2 = param2\n",
    "        \n",
    "    def method(self, another_param):\n",
    "        \"\"\"\n",
    "        Method to do something.\n",
    "        \n",
    "        :param another_param: another parameter.\n",
    "        \"\"\"\n",
    "        return (another_param + self.param1)/self.param2\n",
    "```\n",
    "\n",
    "You can then create an object of the type `MyClass`:\n",
    "\n",
    "```python\n",
    "my_object = MyClass(param1= 1.0, param2=2.0)\n",
    "```\n",
    "\n",
    "and call any method of the class on the object:\n",
    "\n",
    "```python\n",
    "result = my_object.method(3.0)\n",
    "```\n",
    "\n",
    "**Q:** Create a `Bandit` class taking as arguments:\n",
    "\n",
    "* nb_actions: number of arms.\n",
    "* mean: mean of the normal distribution for $Q^*$.\n",
    "* std_Q: standard deviation of the normal distribution for $Q^*$.\n",
    "* std_r: standard deviation of the normal distribution for the sampled rewards.\n",
    "\n",
    "The constructor should initialize a `Q_star` array accordingly and store it as an attribute. It should also store the optimal action.\n",
    "\n",
    "Add a method `step(action)` that samples a reward for a particular action and returns it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Bandit:\n",
    "    \"\"\"\n",
    "    n-armed bandit.\n",
    "    \"\"\"\n",
    "    def __init__(self, nb_actions, mean=0.0, std_Q=1.0, std_r=1.0):\n",
    "        \"\"\"\n",
    "        :param nb_actions: number of arms.\n",
    "        :param mean: mean of the normal distribution for $Q^*$.\n",
    "        :param std_Q: standard deviation of the normal distribution for $Q^*$.\n",
    "        :param std_r: standard deviation of the normal distribution for the sampled rewards.\n",
    "        \"\"\"\n",
    "        # Store parameters\n",
    "        self.nb_actions = nb_actions\n",
    "        self.mean = mean\n",
    "        self.std_Q = std_Q\n",
    "        self.std_r = std_r\n",
    "        \n",
    "        # Initialize the true Q-values\n",
    "        self.Q_star = rng.normal(self.mean, self.std_Q, self.nb_actions)\n",
    "        \n",
    "        # Optimal action\n",
    "        self.a_star = self.Q_star.argmax()\n",
    "        \n",
    "    def step(self, action):\n",
    "        \"\"\"\n",
    "        Sampled a single reward from the bandit.\n",
    "        \n",
    "        :param action: the selected action.\n",
    "        :return: a reward.\n",
    "        \"\"\"\n",
    "        return float(rng.normal(self.Q_star[action], self.std_r, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q:** Create a 5-armed bandits and sample each action multiple times. Compare the mean reward to the ground truth as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJwAAAFzCAYAAAB7BBMsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAApTUlEQVR4nO3dfbBl9Vkn+u9DA2NMMnKVjiFA29TcHhUzNzHTl6D4ksQYecmIMSYXMhplUtODAxpHp7yodccxNXWLKb1xjGJIJ2LC+IKZmxAZaSWJQ4LOmIQmAoEQnC5EaeFKR0byQl7s5Ll/7N2pM4fT3adhnb32PvvzqdrVe/3Wb+3zrOpDP6zvetnV3QEAAACAoRw3dgEAAAAAbC4CJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGdfzYBczCySef3Nu3bx+7DIC5dNttt328u7eOXceY9AmAtekRE/oEwNqO1CeWInDavn179u7dO3YZAHOpqv5i7BrGpk8ArE2PmNAnANZ2pD4xV7fUVdXpVXVzVd1TVXdX1WvXmFNV9Yaq2ldVd1bV88aoFQAAAIC1zVXglORgkp/o7q9PcnaSy6rqzFVzzkuyY/raleSNsy0RgLE4MQEAAIthrgKn7n6ouz88ff/JJPckOXXVtAuTXNsTH0hyUlWdMuNSARiHExMAALAA5ipwWqmqtif5xiQfXLXq1CQPrFjen8eHUqmqXVW1t6r2HjhwYMPqBGB2nJgAAIDFMJeBU1U9Lck7kvxYd39i9eo1NunHDXTv7u6d3b1z69al/2INgE3nyZ6YAAAANs7cBU5VdUImYdNvdvc715iyP8npK5ZPS/LgLGoDYD4McWLClbAAALBx5ipwqqpK8mtJ7unu1x9m2g1JXj19KOzZSR7t7odmViQAoxrqxIQrYQEAYOMcP3YBq5yT5AeSfKSqbp+O/XSSbUnS3Vcn2ZPk/CT7kjyW5JLZlwnAGI7hxMTlVXVdkufHiQkAAJi5uQqcuvuPs/atECvndJLLZlMRAHPGiQkAAFgAcxU4AcCRODEBAACLYa6e4QQAAADA4hM4AQAAADAot9QBS237FTeOXcIg7r/ygrFLANiU9AkAeGIETgAAAMCTshkCeuH8sNxSBwAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADErgBAAAAMCgBE4AAAAADGruAqequqaqHq6quw6z/gVV9WhV3T59/ZtZ1wgAAADA4c1d4JTkrUnOPcqcP+ru505fr5tBTQDMASclAABgMRw/dgGrdfctVbV97DoAmEtvTfIrSa49wpw/6u6XzqYcAABgLfN4hdN6fFNV3VFVv19V3zB2MQDMRnffkuSRsesAAACObBEDpw8n+Zrufk6SX07yrrUmVdWuqtpbVXsPHDgwy/oAGJeTEgAAMLKFC5y6+xPd/anp+z1JTqiqk9eYt7u7d3b3zq1bt868TgBGsa6TEokTEwCLrKrOrap7q2pfVV2xxvqqqjdM199ZVc9btX5LVf1pVf3e7KoGWC4LFzhV1TOrqqbvz8pkH/5m3KoAmAfrPSkxXe/EBMACqqotSa5Kcl6SM5NcXFVnrpp2XpId09euJG9ctf61Se7Z4FIBltrcBU5V9dtJ/iTJ11bV/qp6TVVdWlWXTqd8X5K7quqOJG9IclF391j1AjA/nJQAWApnJdnX3fd19+eTXJfkwlVzLkxybU98IMlJVXVKklTVaUkuSPKWWRYNsGzm8VvqLj7K+l/J5BuKAFgy05MSL0hyclXtT/KzSU5Iku6+OpOTEj9cVQeTfCZOSgBsRqcmeWDF8v4kz1/HnFOTPJTkPyT5ySRPP9IPqapdmVwdlW3btj2pggGW0dwFTgBwOE5KAJCk1hhbfXJhzTlV9dIkD3f3bVX1giP9kO7enWR3kuzcudPJC4BjNHe31AEAABzB/iSnr1g+LcmD65xzTpLvrqr7M7kV70VV9RsbVyrA8hI4AQAAi+TWJDuq6oyqOjHJRUluWDXnhiSvnn5b3dlJHu3uh7r7p7r7tO7ePt3uv3T398+0eoAl4ZY6AABgYXT3waq6PMlNSbYkuaa77z70JUPTZ/rtSXJ+kn1JHktyyVj1AiwrgRMAALBQuntPJqHSyrGrV7zvJJcd5TPel+R9G1AeAHFLHQAAAAADEzgBAAAAMCiBEwAAAACDEjgBAAAAMCiBEwAAAACDEjgBAAAAMCiBEwAAAACDEjgBAAAAMKjjxy4AAFg826+4cewSBnH/lReMXQIAwKbkCicAAAAABuUKJ9bkzDUAAADwRLnCCQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGJTACQAAAIBBCZwAAAAAGNTcBU5VdU1VPVxVdx1mfVXVG6pqX1XdWVXPm3WNAIxDjwAAgMUwd4FTkrcmOfcI689LsmP62pXkjTOoCYD58NboEQAAMPfmLnDq7luSPHKEKRcmubYnPpDkpKo6ZTbVATAmPQIAABbD8WMX8AScmuSBFcv7p2MPrZxUVbsyObudbdu2zaw4AEa1rh4BALARtl9x49glDOL+Ky8YuwQ2gbm7wmkdao2xftxA9+7u3tndO7du3TqDsgCYA+vqEcnkxERV7a2qvQcOHNjgsgAAYLksYuC0P8npK5ZPS/LgSLUAMF/W3SOcmAAAgI2ziIHTDUlePf0morOTPNrdbpUAINEjAABgLszdM5yq6reTvCDJyVW1P8nPJjkhSbr76iR7kpyfZF+Sx5JcMk6lAMyaHgEAAIth7gKn7r74KOs7yWUzKgeAOaJHAADAYljEW+oAAAAAmGMCJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAAAAGJXACAAAAYFACJwAAYKFU1blVdW9V7auqK9ZYX1X1hun6O6vqedPxL6uqD1XVHVV1d1X93OyrB1gOAicAAGBhVNWWJFclOS/JmUkurqozV007L8mO6WtXkjdOxz+X5EXd/Zwkz01yblWdPYu6AZaNwAkAAFgkZyXZ1933dffnk1yX5MJVcy5Mcm1PfCDJSVV1ynT5U9M5J0xfPbPKAZaIwAkAAFgkpyZ5YMXy/unYuuZU1Zaquj3Jw0ne090fXOuHVNWuqtpbVXsPHDgwVO0AS0PgBAAALJJaY2z1VUqHndPdX+ju5yY5LclZVfXstX5Id+/u7p3dvXPr1q1Ppl6ApSRwAgAAFsn+JKevWD4tyYPHOqe7/zbJ+5KcO3iFAAicAACAhXJrkh1VdUZVnZjkoiQ3rJpzQ5JXT7+t7uwkj3b3Q1W1tapOSpKqekqSFyf52AxrB1gax49dAAAAwHp198GqujzJTUm2JLmmu++uqkun669OsifJ+Un2JXksySXTzU9J8rbpN90dl+Tt3f17s94HgGUgcAIAABZKd+/JJFRaOXb1ived5LI1trszyTdueIEAuKUOAAAAgGEJnAAAgJmrqi1V9d6x6wBgYwicAACAmevuLyR5rKq+YuxaABieZzgBAABj+WySj1TVe5J8+tBgd//oeCUBMASBEwAAMJYbpy8ANhmBEwAAMIrufltVnZjkH06H7u3uvxuzJgCGIXACAABGUVUvSPK2JPcnqSSnV9UPdvctI5YFwADm7qHhVXVuVd1bVfuq6oo11r+gqh6tqtunr38zRp0AjEOfANhU/p8kL+nub+/ub0vyXUl+ceSaABjAXF3hVFVbklyV5DuT7E9ya1Xd0N0fXTX1j7r7pTMvEIBR6RMAm84J3X3voYXu/rOqOmHMggAYxrxd4XRWkn3dfV93fz7JdUkuHLkmAOaHPgGwudxWVb82vTr1BVX15iS3jV0UAE/evAVOpyZ5YMXy/unYat9UVXdU1e9X1Tes9UFVtauq9lbV3gMHDmxErQDM3mB9AoC5cGmSu5P8aJLXJvnodAyABTdXt9Rl8qDA1XrV8oeTfE13f6qqzk/yriQ7HrdR9+4ku5Nk586dqz8DgMU0WJ+oql1JdiXJtm3bBi4TgKOpquOS3Nbdz07y+rHrAWBY83aF0/4kp69YPi3JgysndPcnuvtT0/d7kpxQVSfPrkQARjRYn+ju3d29s7t3bt26dSNrBmAN3f3FJHdUldQfYBOatyucbk2yo6rOSPJXSS5K8qqVE6rqmUn+uru7qs7KJDT7m5lXCsAY9AmAzeWUJHdX1YeSfPrQYHd/93glATCEuQqcuvtgVV2e5KYkW5Jc0913V9Wl0/VXJ/m+JD9cVQeTfCbJRd3tljmAJaBPAGw6Pzd2AQBsjLkKnJIv3f6wZ9XY1Sve/0qSX5l1XQDMB30CYHOYPsPpqukznADYZObtGU4AAMAS8AwngM1t7q5wAgAAloZnOAFsUgInAABgLJ7hBLBJCZwAAIBRdPf7q+prkuzo7vdW1Zdn8qUQACw4z3ACAABGUVX/PMn/m+RN06FTk7xrtIIAGMwTCpyq6qlV5cwDAADwZFyW5Jwkn0iS7v7vSZ4xakUADGJdgVNVHVdVr6qqG6vq4SQfS/JQVd1dVT9fVTs2tkwAAGAT+lx3f/7QQlUdn6RHrAeAgaz3Cqebk/yDJD+V5JndfXp3PyPJtyb5QJIrq+r7N6hGAABgc3p/Vf10kqdU1Xcm+U9J/vPINQEwgPU+NPzF3f13qwe7+5Ek70jyjqo6YdDKAFhYVfXUJJ/t7i+MXQsAc+2KJK9J8pEk/yLJniRvGbUiAAaxrsDpUNhUVV+V5JVJPpvk7iQf6e7PrJwDwPKpquOSXJTknyb535N8Lsnfq6oDmRw87J4+lwMAvqS7v5jkzdMXAJvIsT40/PokW5P830l+PsmjVfWxwasCYNG49RoAAPiS9d5Sd8jTu/t1VfW93f3tVfXyJP/rRhQGwEJx6zUAAPAlx3qF02enf36uqp7S3e9Icv7ANQGwYFbeel1VP1xVl1TVWVX1lNVzAGC16bP/ANhEjjVw+oWq+sokv5Pkmqr6kSSnDl8WAAvKrdcArFtVfXNVfTTJPdPl51TVr45cFgADOKbAqbvf0d2PdPfrM3kI7OlJLtyQygBYRE/v7tcl+evu/vYkFyf59ZFrAmB+/WKS70ryN0nS3Xck+bZRKwJgEOt6hlNVVXf3yrHu/o9HmwPA0nncrddV9f4k/37MogCYX939QFWtHPrCWLUAMJz1XuF0c1X9SFVtWzlYVSdW1Yuq6m1JfnD48gBYMG69BuBYPFBV35ykp8cW/zrT2+sAWGzr/Za6c5P8syS/XVVnJPnbJE/JJLB6d5Jf7O7bN6JAABbH9MskkuT1VfUDSf5R3HoNwOFdmuSXMjk5sT+TY4t/OWpFAAxiXYFTd382ya8m+dXp11qfnOQz3f23G1gbAAvCrdcAPEFf293/dOVAVZ2T5L+OVA8AA1nvFU5fMv1a64c2oBYAFtfNVfWOJL/b3X95aLCqTkzyLZncdn1zkreOUx4Ac+qXkzxvHWMsmO1X3Dh2CYO4/8oLxi4BFtZRA6eqemp3f7qqntbdn5pFUQAsHLdeA7BuVfVNSb45ydaq+vEVq/5+ki3jVAXAkNZzhdP/UlWXJNmX5A82uB4AFpBbrwE4RicmeVomxyNPXzH+iSTfN0pFAAxqPYHTdyT5oUy+begZ3f3wxpYEwCI7dOv19Iz165Okqr62u+8dtzIA5kV3vz/J+6vqrd39F2PXA8Dw1hM4fSiT2yROFzYBcDRVdVKSX0zydVX12SR3JnlNkkvGrAuAufTWqnrcF0p094vGKAaA4Rw1cOrue6Zv79zgWgBYYFX1rEyuiv3yTB4O/pQkB5L8b0neOV5lAMyxf73i/ZcleXmSgyPVAsCAjvlb6gBgtap6SZK3JXlfks8luTST4OmS7v71EUsDYI51922rhv5rVb1/lGIAGNS6A6dVZ64/Nr3vGgCS5N8l+dbu3ndoYPoNRLur6jVJHuvuu0arDoC5VFVfuWLxuCT/OMkzRyoHgAGtK3Ba68x1VR06c/3fNq48ABbEiSvDpiTp7j+pqpcn+b1Mesc/GqUyAObZbUk6SWVyK92fZ/LcPwAW3HqvcDrcmes3T89cf9qZa4Cl9tmq2trdB1YOdvefVdUXMrlCFgD+J919xtg1ALAx1hs4He7M9ffGmWsAkp9P8q6qekV3P3hosKpOTvI533IKwErT44jD6m5fNgGw4NYbODlzDcBhdfc7qurvJfmTqrotyR1JTkzyykyukgWAlf7JEdZ1fLspwMJbb+DkzDUAR9Tdv1VV70pyUZJnJ/lEkld1962jFgbA3OnuS8auAYCNta7AyZlrANajux9Lcs3YdQCwGKrqK5L8bJJvmw69P8nruvvR8aoCYAjHrXdid/9Wkq/P5JlNX5Hk7zI5c/22DaoNAADY3K5J8slMTmS/MpOrY3991IoAGMR6b6lL4sw1AAAwqH/Q3S9fsfxzVXX7WMUAMJx1X+E0K1V1blXdW1X7quqKNdZXVb1huv7OqnreGHUCMA59AmBT+UxVfcuhhao6J8lnRqwHgIEc0xVOG62qtiS5Ksl3Jtmf5NaquqG7P7pi2nlJdkxfz0/yxumfAGxy+gTApvPDSd42fZZTJXkkyQ+NWhEAg5irwCnJWUn2dfd9SVJV1yW5MMnKA4kLk1zb3Z3kA1V1UlWd0t0Pzb5cAGZMnwDYRLr79iTPqaq/P13+xLgVATCUebul7tQkD6xY3j8dO9Y5AGxO+gTAJlJVr52GTZ9M8vqq+nBVvWTsugB48ubtCqdaY6yfwJxU1a4ku5Jk27ZtT7ig7Vfc+IS3nSf3X3nBhs7fTDbD3/kT+fvbDPud+F1fAvrEBvHfzvpthr/zZe0TT2S/l/l3fUb+WXf/UlV9V5JnJLkkk2+pe/e4ZQHwZM3bFU77k5y+Yvm0JA8+gTnp7t3dvbO7d27dunXwQgEYhT4BsLkcOklwfpJf7+47svaJAwAWzLwFTrcm2VFVZ1TViUkuSnLDqjk3JHn19FuIzk7yqOdyACwNfQJgc7mtqt6dSeB0U1U9PckXR64JgAHM1S113X2wqi5PclOSLUmu6e67q+rS6fqrk+zJpCHtS/JYJpfdArAE9AmATec1SZ6b5L7ufqyqvir+3QbYFOYqcEqS7t6TycHCyrGrV7zvJJfNui4A5oM+AbB5dPcXq2p7ku+vqk7yx919/dG2q6pzk/xSJicf3tLdV65aX9P152dy8uGHuvvDVXV6kmuTPDOTK6l2d/cvDblPAEzM2y11AADAkqiqX01yaZKPJLkryb+oqquOss2WJFclOS/JmUkurqozV007L8mO6WtXkjdOxw8m+Ynu/vokZye5bI1tARjA3F3hBAAALI1vT/Ls6dWpqaq3ZRI+HclZSfZ1933Tba5LcmGSj66Yc2GSa6ef+4GqOqmqTpk+0++hJOnuT1bVPUlOXbUtAANwhRMAADCWe5NsW7F8epI7j7LNqUkeWLG8fzp2THOmt/J9Y5IPrvVDqmpXVe2tqr0HDhw4SkkArOYKJwAAYKaq6j8n6SRfkeSeqvrQdPn5Sf7b0TZfY6yPZU5VPS3JO5L8WHd/Yq0f0t27k+xOkp07d67+fACOQuAEAADM2i8cYd3Rwp39mVwJdchpSR5c75yqOiGTsOk3u/ud66oWgGMmcAIAAGaqu9+/1nhVnZPkVUluOcLmtybZUVVnJPmrJBdNt1nphiSXT5/v9Pwkj3b3Q9Nvr/u1JPd09+uf5G4AcAQCJwAAYDRV9dxMAqNXJvnzTK4+OqzuPlhVlye5KcmWJNd0991Vdel0/dVJ9iQ5P8m+JI8luWS6+TlJfiDJR6rq9unYT3f3niH3CQCBEwAAMGNV9Q8zuTLp4iR/k+R3klR3v3A9208Doj2rxq5e8b6TXLbGdn+ctZ/vBMDABE4AAMCsfSzJHyX5J929L0mq6l+NWxIAQzpu7AIAAICl8/Ik/1+Sm6vqzVX1HXHlEcCmInACAABmqruv7+7/I8nXJXlfkn+V5Kur6o1V9ZJRiwNgEAInAABgFN396e7+ze5+aZLTktye5IpxqwJgCAInAABgdN39SHe/qbtfNHYtADx5AicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABnX82AUAAAAALKLtV9w4dglP2v1XXrAhnytwAgAA2CCb4WA02bgDUmDzcksdAAAAAIMSOAEAAAAwKIETAAAAAIPyDCcAFkJVfWWS30myPcn9SV7Z3f9jjXn3J/lkki8kOdjdO2dXJQAAkMxR4ORAAoCjuCLJH3b3lVV1xXT5/zzM3Bd298dnVxpsfh4YDAAci3m6pe7QgcSOJH84XT6cF3b3c4VNAEvlwiRvm75/W5LvGa8UAADgSOYpcHIgAcCRfHV3P5Qk0z+fcZh5neTdVXVbVe063IdV1a6q2ltVew8cOLAB5QIAwPKam1vqsupAoqqOdiDRSd7U3bvXmjQ9yNiVJNu2bduIegEYWFW9N8kz11j1M8fwMed094PTPvKeqvpYd9+yetK0f+xOkp07d/YTKhgAAFjTTAMnBxIAHEl3v/hw66rqr6vqlOlJiVOSPHyYz3hw+ufDVXV9krOSPK5PAAAAG2emgZMDCZhfHgbLArghyQ8muXL65++unlBVT01yXHd/cvr+JUleN9MqAQCAuXqG06EDieQIBxJV9fRD7zM5kLhrZhUCMKYrk3xnVf33JN85XU5VPauq9kznfHWSP66qO5J8KMmN3f0Ho1QLAABLbJ6e4XRlkrdX1WuS/GWSVySTA4kkb+nu8zM5kLi+qpJJ7b/lQAJgOXT33yT5jjXGH0xy/vT9fUmeM+PSAACAVeYmcHIgAQAAALA5zNMtdQAAAABsAgInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAAABgUAInAAAAAAYlcAIAABZKVZ1bVfdW1b6qumKN9VVVb5iuv7Oqnrdi3TVV9XBV3TXbqgGWi8AJAABYGFW1JclVSc5LcmaSi6vqzFXTzkuyY/raleSNK9a9Ncm5G18pwHITOAEAAIvkrCT7uvu+7v58kuuSXLhqzoVJru2JDyQ5qapOSZLuviXJIzOtGGAJCZwAAIBFcmqSB1Ys75+OHeucI6qqXVW1t6r2Hjhw4AkVCrDMBE4AAMAiqTXG+gnMOaLu3t3dO7t759atW49lUwAicAIAABbL/iSnr1g+LcmDT2AOABtI4AQAACySW5PsqKozqurEJBcluWHVnBuSvHr6bXVnJ3m0ux+adaEAy0zgBAAALIzuPpjk8iQ3Jbknydu7++6qurSqLp1O25PkviT7krw5yb88tH1V/XaSP0nytVW1v6peM9MdAFgSx49dAAAAwLHo7j2ZhEorx65e8b6TXHaYbS/e2OoASFzhBAAAAMDABE4AAAAADErgBAAAAMCgBE4ALISqekVV3V1VX6yqnUeYd25V3VtV+6rqilnWCAAATMxN4ORAAoCjuCvJ9ya55XATqmpLkquSnJfkzCQXV9WZsykPAAA4ZG4CpziQAOAIuvue7r73KNPOSrKvu+/r7s8nuS7JhRtfHQAAsNLcBE4OJAAYwKlJHlixvH869jhVtauq9lbV3gMHDsykOAAAWBZzEzitkwMJgE2sqt5bVXet8VrvyYVaY6zXmtjdu7t7Z3fv3Lp16xMvGgAAeJzjZ/nDquq9SZ65xqqf6e7fXc9HrDF22AOJJLuTZOfOnWvOAWC+dPeLn+RH7E9y+orl05I8+CQ/E/4n9195wdglAADMvZkGTg4kANhgtybZUVVnJPmrJBcledW4JQEAwPJZtFvqvnQgUVUnZnIgccPINQEwA1X1sqran+SbktxYVTdNx59VVXuSpLsPJrk8yU1J7kny9u6+e6yaAQBgWc30CqcjqaqXJfnlJFszOZC4vbu/q6qeleQt3X1+dx+sqkMHEluSXONAAmA5dPf1Sa5fY/zBJOevWN6TZM8MSwMAAFaZm8DJgQTzwrM5AAAA4MlZtFvqAAAAAJhzAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABiVwAgAAAGBQAicAAAAABnX82AXMu/uvvGDsEgBIUlWvSPJvk3x9krO6e+9h5t2f5JNJvpDkYHfv3Mi69AkAAHi8ubnCqapeUVV3V9UXq+qwBwdVdX9VfaSqbq+qNQ82ANiU7kryvUluWcfcF3b3czc6bAIAANY2T1c4HTqQeNM65r6wuz++wfUAMEe6+54kqaqxSwEAAI5ibq5w6u57uvvesesAYOF1kndX1W1Vtetwk6pqV1Xtraq9Bw4cmGF5AACw+c3TFU7rdehAopO8qbt3j10QAMOoqvcmeeYaq36mu393nR9zTnc/WFXPSPKeqvpYdz/uNrxp/9idJDt37uwnXDQAAPA4Mw2cZnkgMT2rvStJtm3b9oRrBmB2uvvFA3zGg9M/H66q65OclfU99wkAABjITAOnWR5IOHMNsHyq6qlJjuvuT07fvyTJ60YuCwAAls7cPMNpParqqVX19EPvMzmQuGvcqgCYhap6WVXtT/JNSW6sqpum48+qqj3TaV+d5I+r6o4kH0pyY3f/wTgVAwDA8pqbZzhV1cuS/HKSrZkcSNze3d9VVc9K8pbuPj+TA4nrp99QdHyS33IgAbAcuvv6JNevMf5gkvOn7+9L8pwZlwYAAKwyN4GTAwkAAACAzWGhbqkDAAAAYP4JnAAAgIVSVedW1b1Vta+qrlhjfVXVG6br76yq5613WwCGIXACAAAWRlVtSXJVkvOSnJnk4qo6c9W085LsmL52JXnjMWwLwAAETgAAwCI5K8m+7r6vuz+f5LokF66ac2GSa3viA0lOqqpT1rktAAMQOAEAAIvk1CQPrFjePx1bz5z1bAvAAObmW+o20m233fbxqvqLses4gpOTfHzsIkZgv5fPsu77vO/314xdwNj0ibllv5fPsu77PO/3PPaIWmOs1zlnPdtOPqBqVya34yXJp6rq3nVXOHsb/jtU/34jP/0Js98bxH7PnQ3d9ye534ftE0sROHX31rFrOJKq2tvdO8euY9bs9/JZ1n1f1v1eJPrEfLLfy2dZ931Z9/tJ2J/k9BXLpyV5cJ1zTlzHtkmS7t6dZPeTLXYWlvV3yH4vl2Xd72Rx990tdQAAwCK5NcmOqjqjqk5MclGSG1bNuSHJq6ffVnd2kke7+6F1bgvAAJbiCicAAGBz6O6DVXV5kpuSbElyTXffXVWXTtdfnWRPkvOT7EvyWJJLjrTtCLsBsOkJnObDQlyquwHs9/JZ1n1f1v1mOMv6O2S/l8+y7vuy7vcT1t17MgmVVo5dveJ9J7lsvdtuAsv6O2S/l8uy7neyoPtek3+LAQAAAGAYnuEEAAAAwKAETiOrqnOr6t6q2ldVV4xdzyxU1TVV9XBV3TV2LbNUVadX1c1VdU9V3V1Vrx27plmoqi+rqg9V1R3T/f65sWuaparaUlV/WlW/N3YtLJ5l7BGJPqFP6BOwXvqEPjF2TbOgTyxunxA4jaiqtiS5Ksl5Sc5McnFVnTluVTPx1iTnjl3ECA4m+Ynu/vokZye5bEn+vj+X5EXd/Zwkz01y7vTbYpbFa5PcM3YRLJ4l7hGJPqFP6BNwVPqEPhF9YlksbJ8QOI3rrCT7uvu+7v58kuuSXDhyTRuuu29J8sjYdcxadz/U3R+evv9kJv9onDpuVRuvJz41XTxh+lqKh8dV1WlJLkjylrFrYSEtZY9I9Inpe31iCegTPEn6xJLRJ5LoEwtF4DSuU5M8sGJ5f5bgHwySqtqe5BuTfHDkUmZiehno7UkeTvKe7l6K/U7yH5L8ZJIvjlwHi0mPWGL6hD4B66BPLDF9Qp9YBAKncdUaY0uR1C6zqnpaknck+bHu/sTY9cxCd3+hu5+b5LQkZ1XVs0cuacNV1UuTPNzdt41dCwtLj1hS+oQ+AeukTywpfUKfWBQCp3HtT3L6iuXTkjw4Ui3MQFWdkElz+M3ufufY9cxad/9tkvdlOe65PyfJd1fV/Zlc4v6iqvqNcUtiwegRS0if0CfGLYkFo08sIX1Cnxi3pGMjcBrXrUl2VNUZVXVikouS3DByTWyQqqokv5bknu5+/dj1zEpVba2qk6bvn5LkxUk+NmpRM9DdP9Xdp3X39kz+2/4v3f39I5fFYtEjlow+oU/oExwjfWLJ6BP6xKL1CYHTiLr7YJLLk9yUyQPf3t7dd49b1carqt9O8idJvraq9lfVa8auaUbOSfIDmSTTt09f549d1AyckuTmqrozk/8xek93L9xXesKsLWuPSPSJ6BP6BKyDPqFP6BPMu+p2my8AAAAAw3GFEwAAAACDEjgBAAAAMCiBEwAAAACDEjgBAAAAMCiBEwAAAACDEjjBMaqql1VVV9XXHWXej1XVl69Y3lNVJ214gQCMSp8A4Ej0CZZFdffYNcBCqaq3JzklyR929789wrz7k+zs7o/PqDQA5oA+AcCR6BMsC1c4wTGoqqclOSfJa5JcNB3bUlW/UFUfqao7q+pHqupHkzwryc1VdfN03v1VdfL0/Y9X1V3T149Nx7ZX1T1V9eaquruq3l1VT5mu+9Gq+uj086+b/Z4DsB76BABHok+wTI4fuwBYMN+T5A+6+8+q6pGqel6S5yc5I8k3dvfBqvrK7n6kqn48yQtXn5Goqn+c5JLpdpXkg1X1/iT/I8mOJBd39z+fnvl4eZLfSHJFkjO6+3MuowWYa98TfQKAw/ue6BMsCVc4wbG5OMmhMwLXTZdfnOTq7j6YJN39yFE+41uSXN/dn+7uTyV5Z5Jvna778+6+ffr+tiTbp+/vTPKbVfX9SQ4OsB8AbAx9AoAj0SdYGq5wgnWqqq9K8qIkz66qTrIlSWfyD/mxPAytjrDucyvefyHJU6bvL0jybUm+O8n/VVXfcKghATAf9AkAjkSfYNm4wgnW7/uSXNvdX9Pd27v79CR/nuTDSS6tquOTpKq+cjr/k0mevsbn3JLke6rqy6vqqUleluSPDvdDq+q4JKd3981JfjLJSUmeNtA+ATAcfQKAI9EnWCoCJ1i/i5Ncv2rsHZk8zO8vk9xZVXckedV03e4kv3/oIX+HdPeHk7w1yYeSfDDJW7r7T4/wc7ck+Y2q+kiSP03yi939t09uVwDYAPoEAEeiT7BUqvtYrtwDAAAAgCNzhRMAAAAAgxI4AQAAADAogRMAAAAAgxI4AQAAADAogRMAAAAAgxI4AQAAADAogRMAAAAAgxI4AQAAADCo/x+HjgSvj8IOCgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1440x432 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "nb_actions = 5\n",
    "bandit = Bandit(nb_actions)\n",
    "\n",
    "all_rewards = []\n",
    "for t in range(1000):\n",
    "    rewards = []\n",
    "    for a in range(nb_actions):\n",
    "        rewards.append(bandit.step(a))\n",
    "    all_rewards.append(rewards)\n",
    "    \n",
    "mean_reward = np.mean(all_rewards, axis=0)\n",
    "\n",
    "plt.figure(figsize=(20, 6))\n",
    "plt.subplot(131)\n",
    "plt.bar(range(nb_actions), bandit.Q_star)\n",
    "plt.xlabel(\"Actions\")\n",
    "plt.ylabel(\"$Q^*(a)$\")\n",
    "plt.subplot(132)\n",
    "plt.bar(range(nb_actions), mean_reward)\n",
    "plt.xlabel(\"Actions\")\n",
    "plt.ylabel(\"$Q_t(a)$\")\n",
    "plt.subplot(133)\n",
    "plt.bar(range(nb_actions), np.abs(bandit.Q_star - mean_reward))\n",
    "plt.xlabel(\"Actions\")\n",
    "plt.ylabel(\"Absolute error\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "3d24234067c217f49dc985cbc60012ce72928059d528f330ba9cb23ce737906d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}