{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Sampling" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this first exercise, we will investigate how to evaluate the Q-value of each action available in a 5-armed bandit. It is mostly to give you intuition about the limits of sampling and the central limit theorem.\n", "\n", "Let's start with importing numpy and matplotlib:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sampling a n-armed bandit\n", "\n", "Let's now create the n-armed bandit. The only thing we need to do is to randomly choose 5 true Q-values $Q^*(a)$.\n", "\n", "![](../img/bandit-example.png)\n", "\n", "To be generic, let's define nb_actions=5 and create an array corresponding to the index of each action (0, 1, 2, 3, 4) for plotting purpose." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "nb_actions = 5\n", "actions = np.arange(nb_actions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q:** Create a numpy array Q_star with nb_actions values, normally distributed with a mean of 0 and standard deviation of 1 (as in the lecture). " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng()\n", "Q_star = rng.normal(0, 1, nb_actions)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q:** Plot the Q-values. Identify the optimal action $a^*$.\n", "\n", "*Tip:* you could plot the array Q_star with plt.plot, but that would be ugly. Check the documentation of the plt.bar method." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Optimal action: 0\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAFzCAYAAACHCIXLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAATiUlEQVR4nO3df6zld13n8debmXaXX1kWZ5DSTpmanexaMaJOKiy6KwimP4gFdElLVGTJTjBUJW5ixhj3VzabJhh1N6LNqA0gLpWkVhs7Wn5s12pE6LS0pUNbndTRTqbZDlYQBIoD7/3jnoHbO3dm7rT3nu/59D4eyc39/uq575vT6Tz7Pd/zPdXdAQBg8T1j6gEAAFgb4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwiK1TDzAP27Zt6507d049BgDAGd15552f7u7tq+1buHCrquuTvDbJo939klX2V5L/meTyJF9I8mPdfdfpHnPnzp05cODARowLALCuquqvT7VvEV8qfXeSS0+z/7Iku2Zfe5L82hxmAgCY3MKFW3ffnuSx0xxyZZL39pI/T/K8qjpvPtMBAExn4cJtDc5P8vCy9SOzbU9QVXuq6kBVHTh27NjchgMA2Cgjhlutsu2kD1zt7n3dvbu7d2/fvur1fQAAQxkx3I4k2bFs/YIkRyeaBQBgbkYMt5uT/GgteVmSz3b3I1MPBQCw0RbxdiDvT/K9SbZV1ZEk/znJOUnS3dcl2Z+lW4EcytLtQN4yzaQAAPO1cOHW3VefYX8nefucxgEAWBgjvlQKALApCTcAgEEINwCAQQg3AIBBLNybE0a2c+8tU4/wtHL42iumHgEAFoozbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAINYuHCrqkur6sGqOlRVe1fZ/71V9dmqunv29Z+mmBMAYN62Tj3AclW1Jcm7krwmyZEkd1TVzd39qRWH/kl3v3buAwIATGjRzrhdkuRQdz/U3V9OckOSKyeeCQBgISxauJ2f5OFl60dm21Z6eVXdU1V/WFXfMp/RAACmtVAvlSapVbb1ivW7kry4uz9fVZcn+b0ku056oKo9SfYkyYUXXrjOYwIAzN+inXE7kmTHsvULkhxdfkB3/313f362vD/JOVW1beUDdfe+7t7d3bu3b9++kTMDAMzFooXbHUl2VdVFVXVukquS3Lz8gKp6YVXVbPmSLP0Ofzv3SQEA5myhXirt7uNVdU2SW5NsSXJ9dx+sqrfN9l+X5IeS/HhVHU/yxSRXdffKl1MBAJ52Firckq+9/Ll/xbbrli3/SpJfmfdcAABTW7SXSgEAOAXhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADCIrVMPAMCYdu69ZeoRnlYOX3vF1CMwAGfcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAYh3AAABiHcAAAGIdwAAAaxcOFWVZdW1YNVdaiq9q6yv6rqf83231tV3zHFnAAA87ZQ4VZVW5K8K8llSS5OcnVVXbzisMuS7Jp97Unya3MdEgBgIgsVbkkuSXKoux/q7i8nuSHJlSuOuTLJe3vJnyd5XlWdN+9BAQDmbdHC7fwkDy9bPzLbdrbHAAA87WydeoAVapVt/SSOSVXtydJLqbnwwguf+mRrcPjaK+byc3jydu69ZeoRnnY24t97z9P624jnyX/zFp8/S+tv6n/vn9QZt6p69ux6tPV2JMmOZesXJDn6JI5Jd+/r7t3dvXv79u3rPigAwLytKdyq6hlV9aaquqWqHk3yQJJHqupgVb2zqnat0zx3JNlVVRdV1blJrkpy84pjbk7yo7N3l74syWe7+5F1+vkAAAtrrS+V3pbkw0l+Nsl93f3VJKmq5yd5ZZJrq+qm7n7fUxmmu49X1TVJbk2yJcn13X2wqt42239dkv1JLk9yKMkXkrzlqfxMAIBRrDXcXt3d/7hyY3c/luTGJDdW1TnrMVB3789SnC3fdt2y5U7y9vX4WQAAI1lTuJ2Itqr6hiRvTPKlJAeTfLK7v7j8GAAANsbZvjnhpiTbk/yPJO9M8tmqemDdpwIA4CRnezuQ53b3f6uqN3T3v62qH0zyLzZiMAAAnuhsz7h9afb98ap6ZnffmKU3CgAAsMHO9ozbL8zeSfo7Sa6vqj+LTy0AAJiLszrj1t03dvdj3f2LWXrn546c/FmiAABsgDWdcauqmt2G42u6+7fOdAwAAOtnrWfcbquqn6iqJ3zoZ1WdW1Wvqqr3JHnz+o8HAMAJa73G7dIk/z7J+6vqoiSfSfLMLIXfB5P8UnffvREDAgCwZK034P1Skl9N8quzT0jYluSL3f2ZDZwNAIBlzvZdpSc+IcGHugMAzNkZr3GrqmfPvj9n48cBAOBU1vLmhH9eVdck+e6NHgYAgFNbS7h9X5IfS/JNVfWCjR0HAIBTWcs1bh/P0jtKd3T3oxs8DwAAp3DGcOvu+2eL927wLAAAnMbZfsg8AAATWfPtQKrqRVm63u1ZSR7o7j/esKlggxy+9oqpRwCAJ21NZ9yq6vuT3Jnk8iQvT/LLVfVgVf3rjRwOAICvW+sZt/+e5Hu6+9CJDVX18iS/XlVvTfIP3X3fRgwIAMCStYbbucujLUm6+6NV9YYkf5Dk8STfut7DAQDwdWt9c8KXqmr7yo3d/RdJvpKla98AANhAaw23dyb5vdkbFL6mqrYledz93QAANt6aXirt7hur6p8k+WhV3ZnkniTnJnljlq5/AwBgg635Pm7d/b+TfHOWrmn7Z0n+Mcmbuvs9GzQbAADLrPk+bknS3V9Icv0GzQIAwGn45AQAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQQg3AIBBCDcAgEEINwCAQWydegAAYGMcvvaKqUdgnTnjBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwCOEGADAI4QYAMAjhBgAwiK1TD3BCVT0/ye8k2ZnkcJI3dvffrXLc4SSfS/KVJMe7e/f8pgQAmM4inXHbm+Qj3b0ryUdm66fyyu5+qWgDADaTRQq3K5O8Z7b8niSvm24UAIDFs0jh9o3d/UiSzL6/4BTHdZIPVtWdVbXnVA9WVXuq6kBVHTh27NgGjAsAMF9zvcatqj6c5IWr7Pq5s3iYV3T30ap6QZIPVdUD3X37yoO6e1+SfUmye/fuflIDAwAskLmGW3e/+lT7qur/VdV53f1IVZ2X5NFTPMbR2fdHq+qmJJckOSncAACebhbppdKbk7x5tvzmJL+/8oCqenZVPffEcpLvT3Lf3CYEAJjQIoXbtUleU1V/meQ1s/VU1Yuqav/smG9M8qdVdU+Sjye5pbv/aJJpAQDmbGHu49bdf5vk+1bZfjTJ5bPlh5J825xHAwBYCIt0xg0AgNMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDWJhwq6p/V1UHq+qrVbX7NMddWlUPVtWhqto7zxkBAKa0MOGW5L4kb0hy+6kOqKotSd6V5LIkFye5uqouns94AADT2jr1ACd09/1JUlWnO+ySJIe6+6HZsTckuTLJpzZ8QACAiS3SGbe1OD/Jw8vWj8y2naSq9lTVgao6cOzYsbkMBwCwkeZ6xq2qPpzkhavs+rnu/v21PMQq23q1A7t7X5J9SbJ79+5VjwEAGMlcw627X/0UH+JIkh3L1i9IcvQpPiYAwBBGe6n0jiS7quqiqjo3yVVJbp54JgCAuViYcKuq11fVkSQvT3JLVd062/6iqtqfJN19PMk1SW5Ncn+SD3T3walmBgCYp0V6V+lNSW5aZfvRJJcvW9+fZP8cRwMAWAgLc8YNAIDTE24AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIMQbgAAgxBuAACDEG4AAIPYOvUAACsdvvaKqUcAWEjOuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMQrgBAAxCuAEADEK4AQAMorp76hk2XFUdS/LXU8+xQLYl+fTUQ3BGnqfF5zkag+dpDJ6nr3txd29fbcemCDeeqKoOdPfuqefg9DxPi89zNAbP0xg8T2vjpVIAgEEINwCAQQi3zWnf1AOwJp6nxec5GoPnaQyepzVwjRsAwCCccQMAGIRw20Sq6tKqerCqDlXV3qnnYXVVdX1VPVpV9009C6urqh1VdVtV3V9VB6vqp6aeiZNV1T+tqo9X1T2z5+m/Tj0Tq6uqLVX1iar6g6lnWXTCbZOoqi1J3pXksiQXJ7m6qi6edipO4d1JLp16CE7reJL/2N3fnORlSd7uz9NCejzJq7r725K8NMmlVfWyaUfiFH4qyf1TDzEC4bZ5XJLkUHc/1N1fTnJDkisnnolVdPftSR6beg5Orbsf6e67Zsufy9JfOOdPOxUr9ZLPz1bPmX25sHvBVNUFSa5I8htTzzIC4bZ5nJ/k4WXrR+IvGnjKqmpnkm9P8rGJR2EVs5fg7k7yaJIPdbfnafH8cpKfSfLViecYgnDbPGqVbf7PE56CqnpOkhuTvKO7/37qeThZd3+lu1+a5IIkl1TVSyYeiWWq6rVJHu3uO6eeZRTCbfM4kmTHsvULkhydaBYYXlWdk6Vo++3u/t2p5+H0uvszSf5vXD+6aF6R5Aeq6nCWLuF5VVW9b9qRFptw2zzuSLKrqi6qqnOTXJXk5olngiFVVSX5zST3d/cvTj0Pq6uq7VX1vNnyM5O8OskDkw7FE3T3z3b3Bd29M0t/L/2f7v7hicdaaMJtk+ju40muSXJrli6k/kB3H5x2KlZTVe9P8tEk/7KqjlTVW6eeiZO8IsmPZOnswN2zr8unHoqTnJfktqq6N0v/8/qh7na7CYbmkxMAAAbhjBsAwCCEGwDAIIQbAMAghBsAwCCEGwDAIIQbsGlV1eurqqvqX53huHdU1bOWre8/cX8wgHlyOxBg06qqD2TpXl8f6e7/cprjDifZ3d2fntNoAKtyxg3YlGafM/qKJG/N0h3bT3wg+S9U1Ser6t6q+omq+skkL8rSjVxvmx13uKq2zZZ/uqrum329Y7ZtZ1XdX1W/XlUHq+qDszv3p6p+sqo+NXv8G+b/mwMj2zr1AAATeV2SP+ruv6iqx6rqO5J8V5KLknx7dx+vqud392NV9dNJXrnyjFtVfWeSt8z+uUrysar64yR/l2RXkqu7+z/Mzuz9YJL3Jdmb5KLuftzLrcDZcsYN2KyuztKHWmf2/eosfZbldbOPiEt3P3aGx/juJDd19z909+eT/G6S75nt+6vuvnu2fGeSnbPle5P8dlX9cJLj6/B7AJuIM27AplNV35DkVUleUlWdZEuSzlJgnc2Fv3WafY8vW/5KkmfOlq9I8m+S/ECSn6+qbzkRigBn4owbsBn9UJL3dveLu3tnd+9I8ldJ7krytqramiRV9fzZ8Z9L8txVHuf2JK+rqmdV1bOTvD7Jn5zqh1bVM5Ls6O7bkvxMkuclec46/U7AJiDcgM3o6iQ3rdh2Y5behPA3Se6tqnuSvGm2b1+SPzzx5oQTuvuuJO9O8vEkH0vyG939idP83C1J3ldVn0zyiSS/1N2feWq/CrCZuB0IAMAgnHEDABiEcAMAGIRwAwAYhHADABiEcAMAGIRwAwAYhHADABiEcAMAGMT/B0SA8CHCVTUCAAAAAElFTkSuQmCC", "text/plain": [ "