{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"2022-01-21-vw.ipynb","provenance":[{"file_id":"https://github.com/recohut/nbs/blob/main/raw/T119194%20%7C%20Contextual%20RL%20Product%20Recommender.ipynb","timestamp":1644659711070}],"collapsed_sections":[],"authorship_tag":"ABX9TyNC5ke2GT+lT8iBhdN8FkK/"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","metadata":{"id":"nG-P5CDvH7WG"},"source":["# Contextual Recommender with Vowpal Wabbit"]},{"cell_type":"markdown","metadata":{"id":"4aI3KjKLVsnn"},"source":["We will simulate the scenario of personalizing news content on a site, using CB, to users. The goal is to maximize user engagement quantified by measuring click through rate (CTR).\n","\n","Let's recall that in a CB setting, a data point has four components,\n","- Context\n","- Action\n","- Probability of choosing action\n","- Reward/cost for chosen action\n","\n","We will need to generate a context, get an action/decision for the given context and also simulate generating a reward. We have two website visitors: 'Tom' and 'Anna' Each of them may visit the website either in the morning or in the afternoon. The context is therefore (user, time_of_day). We have the option of recommending a variety of articles to Tom and Anna. Therefore, actions are the different choices of articles: \"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"cheese\". The reward is whether they click on the article or not: 'click' or 'no click'."]},{"cell_type":"markdown","metadata":{"id":"0X6NWsRBjT5f"},"source":["## Setup"]},{"cell_type":"markdown","metadata":{"id":"y0uVpw69jU4F"},"source":["### Installations"]},{"cell_type":"code","metadata":{"id":"YqOCKRRAjR6T"},"source":["!pip install vowpalwabbit"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NtQy4OVzjWh3"},"source":["### Imports"]},{"cell_type":"code","metadata":{"id":"_m3opUkTjX7M"},"source":["from vowpalwabbit import pyvw\n","import random\n","import matplotlib.pyplot as plt\n","import pandas as pd\n","import itertools\n","from itertools import product\n","import numpy as np\n","import scipy\n","import scipy.stats as stats"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yBX5eCYOOc_8"},"source":["## Simulate reward\n","\n","In the real world, we will have to learn Tom and Anna's preferences for articles as we observe their interactions. Since this is a simulation, we will have to define Tom and Anna's preference profile. The reward that we provide to the learner will follow this preference profile. Our hope is to see if the learner can take better and better decisions as we see more samples which in turn means we are maximizing the reward.\n","\n","We will also modify the reward function in a few different ways and see if the CB learner picks up the changes. We will compare the CTR with and without learning.\n","\n","VW optimizes to minimize **cost which is negative of reward**. Therefore, we will always pass negative of reward as cost to VW."]},{"cell_type":"code","metadata":{"id":"NOzs-MMwOc_9"},"source":["# VW tries to minimize loss/cost, therefore we will pass cost as -reward\n","USER_LIKED_ARTICLE = -1.0\n","USER_DISLIKED_ARTICLE = 0.0"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"XQTwsgNMOc_-"},"source":["The reward function below specifies that Tom likes politics in the morning and music in the afternoon whereas Anna likes sports in the morning and politics in the afternoon. It looks dense but we are just simulating our hypothetical world in the format of the feedback the learner understands: cost. If the learner recommends an article that aligns with the reward function, we give a positive reward. In our simulated world this is a click."]},{"cell_type":"code","metadata":{"id":"xpVW_5juOc__"},"source":["def get_cost(context,action):\n","    if context['user'] == \"Tom\":\n","        if context['time_of_day'] == \"morning\" and action == 'politics':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'music':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE\n","    elif context['user'] == \"Anna\":\n","        if context['time_of_day'] == \"morning\" and action == 'sports':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'politics':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Hh0MYzfeOdAB"},"source":["\n","## Understanding VW format\n","\n","There are some things we need to do to get our input into a format VW understands. This function handles converting from our context as a dictionary, list of articles and the cost if there is one into the text format VW understands.\n"]},{"cell_type":"code","metadata":{"id":"Wn_sVCzxOdAC"},"source":["# This function modifies (context, action, cost, probability) to VW friendly format\n","def to_vw_example_format(context, actions, cb_label = None):\n","    if cb_label is not None:\n","        chosen_action, cost, prob = cb_label\n","    example_string = \"\"\n","    example_string += \"shared |User user={} time_of_day={}\\n\".format(context[\"user\"], context[\"time_of_day\"])\n","    for action in actions:\n","        if cb_label is not None and action == chosen_action:\n","            example_string += \"0:{}:{} \".format(cost, prob)\n","        example_string += \"|Action article={} \\n\".format(action)\n","    #Strip the last newline\n","    return example_string[:-1]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"el20qcr1OdAD"},"source":["To understand what's going on here let's go through an example. Here, it's the morning and the user is Tom. There are four possible articles. So in the VW format there is one line that starts with shared, this is the shared context, followed by four lines each corresponding to an article."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"_Yk4gaXWOdAE","executionInfo":{"status":"ok","timestamp":1634747734282,"user_tz":-330,"elapsed":9,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"0bcbd55e-95d5-4c70-b4a2-cf3c11a63638"},"source":["context = {\"user\":\"Tom\",\"time_of_day\":\"morning\"}\n","actions = [\"politics\", \"sports\", \"music\", \"food\"]\n","\n","print(to_vw_example_format(context,actions))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["shared |User user=Tom time_of_day=morning\n","|Action article=politics \n","|Action article=sports \n","|Action article=music \n","|Action article=food \n"]}]},{"cell_type":"markdown","metadata":{"id":"XNOSmbNgOdAE"},"source":["## Getting a decision\n","\n","When we call VW we get a _pmf_, [probability mass function](https://en.wikipedia.org/wiki/Probability_mass_function), as the output. Since we are incorporating exploration into our strategy, VW will give us a list of probabilities over the set of actions. This means that the probability at a given index in the list corresponds to the likelihood of picking that specific action. In order to arrive at a decision/action, we will have to sample from this list.\n","\n","So, given a list `[0.7, 0.1, 0.1, 0.1]`, we would choose the first item with a 70% chance. `sample_custom_pmf` takes such a list and gives us the index it chose and what the probability of choosing that index was."]},{"cell_type":"code","metadata":{"id":"TFpPKrU4OdAF"},"source":["def sample_custom_pmf(pmf):\n","    total = sum(pmf)\n","    scale = 1/total\n","    pmf = [x * scale for x in pmf]\n","    draw = random.random()\n","    sum_prob = 0.0\n","    for index, prob in enumerate(pmf):\n","        sum_prob += prob\n","        if(sum_prob > draw):\n","            return index, prob"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FGyYUeeVOdAG"},"source":["We have all of the information we need to choose an action for a specific user and context. To use VW to achieve this, we will do the following:\n","\n","1. We convert our context and actions into the text format we need\n","2. We pass this example to vw and get the pmf out\n","3. Now, we sample this pmf to get what article we will end up showing\n","4. Finally we return the article chosen, and the probability of choosing it (we are going to need the probability when we learn form this example)"]},{"cell_type":"code","metadata":{"id":"W9GHrLJjOdAG"},"source":["def get_action(vw, context, actions):\n","    vw_text_example = to_vw_example_format(context,actions)\n","    pmf = vw.predict(vw_text_example)\n","    chosen_action_index, prob = sample_custom_pmf(pmf)\n","    return actions[chosen_action_index], prob"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"pRt4MxDnOdAH"},"source":["\n","\n","## Simulation set up\n","\n","Now that we have done all of the setup work and know how to interface with VW, let's simulate the world of Tom and Anna. The scenario is they go to a website and are shown an article. Remember that the reward function allows us to define the worlds reaction to what VW recommends.\n","\n","\n","We will choose between Tom and Anna uniformly at random and also choose their time of visit uniformly at random. You can think of this as us tossing a coin to choose between Tom and Anna (Anna if heads and Tom if tails) and another coin toss for choosing time of day.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"fd_g9N5UOdAH","executionInfo":{"status":"ok","timestamp":1634747751750,"user_tz":-330,"elapsed":12,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"9d5106ce-b0ff-4401-968c-46a99915693f"},"source":["users = ['Tom', 'Anna']\n","times_of_day = ['morning', 'afternoon']\n","actions = [\"politics\", \"sports\", \"music\", \"food\", \"finance\", \"health\", \"camping\"]\n","\n","def choose_user(users):\n","    return random.choice(users)\n","\n","def choose_time_of_day(times_of_day):\n","    return random.choice(times_of_day)\n","\n","# display preference matrix\n","def get_preference_matrix(cost_fun):\n","    def expand_grid(data_dict):\n","        rows = itertools.product(*data_dict.values())\n","        return pd.DataFrame.from_records(rows, columns=data_dict.keys())\n","\n","    df = expand_grid({'users':users, 'times_of_day': times_of_day, 'actions': actions})\n","    df['cost'] = df.apply(lambda r: cost_fun({'user': r[0], 'time_of_day': r[1]}, r[2]), axis=1)\n","\n","    return df.pivot_table(index=['users', 'times_of_day'], \n","            columns='actions', \n","            values='cost')\n","\n","get_preference_matrix(get_cost)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>actions</th>\n","      <th>camping</th>\n","      <th>finance</th>\n","      <th>food</th>\n","      <th>health</th>\n","      <th>music</th>\n","      <th>politics</th>\n","      <th>sports</th>\n","    </tr>\n","    <tr>\n","      <th>users</th>\n","      <th>times_of_day</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Anna</th>\n","      <th>afternoon</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>morning</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","    </tr>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Tom</th>\n","      <th>afternoon</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","    </tr>\n","    <tr>\n","      <th>morning</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["actions             camping  finance  food  health  music  politics  sports\n","users times_of_day                                                         \n","Anna  afternoon         0.0      0.0   0.0     0.0    0.0      -1.0     0.0\n","      morning           0.0      0.0   0.0     0.0    0.0       0.0    -1.0\n","Tom   afternoon         0.0      0.0   0.0     0.0   -1.0       0.0     0.0\n","      morning           0.0      0.0   0.0     0.0    0.0      -1.0     0.0"]},"metadata":{},"execution_count":9}]},{"cell_type":"markdown","metadata":{"id":"0Z4F7pRzOdAI"},"source":["We will instantiate a CB learner in VW and then simulate Tom and Anna's website visits `num_iterations` number of times. In each visit, we:\n","\n","1. Decide between Tom and Anna\n","2. Decide time of day\n","3. Pass context i.e. (user, time of day) to learner to get action i.e. article recommendation and probability of choosing action\n","4. Receive reward i.e. see if user clicked or not. Remember that cost is just negative reward.\n","5. Format context, action, probability, reward in VW format\n","6. Learn from the example\n","    - VW _reduces_ a CB problem to a cost sensitive multiclass classification problem.\n","\n","This is the same for every one of our simulations, so we define the process in the `run_simulation` function. The cost function must be supplied as this is essentially us simulating how the world works.\n"]},{"cell_type":"code","metadata":{"id":"7Vd-9I-KOdAI"},"source":["def run_simulation(vw, num_iterations, users, times_of_day, actions, cost_function, do_learn = True):\n","    cost_sum = 0.\n","    ctr = []\n","\n","    for i in range(1, num_iterations+1):\n","        # 1. In each simulation choose a user\n","        user = choose_user(users)\n","        # 2. Choose time of day for a given user\n","        time_of_day = choose_time_of_day(times_of_day)\n","\n","        # 3. Pass context to vw to get an action\n","        context = {'user': user, 'time_of_day': time_of_day}\n","        action, prob = get_action(vw, context, actions)\n","\n","        # 4. Get cost of the action we chose\n","        cost = cost_function(context, action)\n","        cost_sum += cost\n","\n","        if do_learn:\n","            # 5. Inform VW of what happened so we can learn from it\n","            vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n","            # 6. Learn\n","            vw.learn(vw_format)\n","\n","        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n","        ctr.append(-1*cost_sum/i)\n","\n","    return ctr"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"STiQn3KWOdAJ"},"source":["\n","We want to be able to visualize what is occurring, so we are going to plot the click through rate over each iteration of the simulation. If VW is showing actions the get rewards the ctr will be higher. Below is a little utility function to make showing the plot easier.\n"]},{"cell_type":"code","metadata":{"id":"wJxpv72YOdAK"},"source":["def plot_ctr(num_iterations, ctr):\n","    plt.plot(range(1,num_iterations+1), ctr)\n","    plt.xlabel('num_iterations', fontsize=14)\n","    plt.ylabel('ctr', fontsize=14)\n","    plt.ylim([0,1])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"AVR_H4StOdAK"},"source":["## Scenario 1\n","\n","We will use the first reward function `get_cost` and assume that Tom and Anna do not change their preferences over time and see what happens to user engagement as we learn. We will also see what happens when there is no learning. We will use the \"no learning\" case as our baseline to compare to.\n","\n","### With learning\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"KiWbwzoXOdAL","executionInfo":{"status":"ok","timestamp":1634747758919,"user_tz":-330,"elapsed":1096,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"ceed5a34-03a8-44c1-eac4-2831bec215f2"},"source":["# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xcdX3/8ddnZ++3bC6be4AEEkhokcsKUe7lIhcLtrYC1mqF/vjJT1qp+OOHimiprYpaqxVErBYvVERrayrIRS6FtlyyAUJIIJAbJCFkd7PJZu+zM/P5/XHObmaSTbIbds7M7nk/H499ZM5lzny+85jMe873nPM95u6IiIgMKil0ASIiUlwUDCIikkPBICIiORQMIiKSQ8EgIiI5FAwiIpIjsmAwsx+YWYuZvbSf5WZm3zKzdWb2opmdGFVtIiKyR5R7DHcBFxxg+YXAwvDvauA7EdQkIiJ7iSwY3P0JoP0Aq1wK/MgDTwMNZjYrmupERGRQaaELyDIH2Jw1vSWct23vFc3saoK9Cmpqak465phjIilQRGSiWLFiRZu7Nw63rJiCYcTc/U7gToCmpiZvbm4ucEUiIuOLmb2+v2XFdFbSVmBe1vTccJ6IiESomIJhGfDh8OykpUCHu+/TjSQiIvkVWVeSmf0UOAuYZmZbgM8DZQDufgdwP3ARsA7oAT4aVW0iIrJHZMHg7lccZLkDH4+oHBER2Y9i6koSEZEioGAQEZEcCgYREcmhYBARkRwKBhERyaFgEBGRHAoGERHJoWAQEZEcCgYREcmhYBARkRwKBhERyaFgEBGRHAoGERHJoWAQEZEcCgYREcmhYBARkRwKBhERyaFgEBGRHAoGERHJoWAQEZEcCgYREcmhYBARkRwKBhERyaFgEBGRHAoGERHJoWAQEZEcCgYREcmhYBARkRwKBhERyaFgEBGRHAoGERHJoWAQEZEcCgYREcmhYBARkRwKBhERyRFpMJjZBWa21szWmdmNwyw/zMweM7PnzexFM7soyvpERCTCYDCzBHAbcCGwBLjCzJbstdpNwL3ufgJwOXB7VPWJiEggyj2Gk4F17r7B3ZPAPcCle63jQH34eBLwZoT1iYgI0QbDHGBz1vSWcF62LwAfMrMtwP3AXwy3ITO72syazay5tbU1H7WKiMRWsR18vgK4y93nAhcBPzazfWp09zvdvcndmxobGyMvUkSk0NyddMbzsu3SvGx1eFuBeVnTc8N52a4CLgBw96fMrBKYBrREUqGISB64O539KTp6BtjZk6S1s5+u/hSlJSWkMhlKzDCDRTPq6OwboLMvRTKVYfvuPnb2DNCdDJ7b2tlPfypDW1c/W3f2cvPvL+GPm+YdvIBRijIYlgMLzWw+QSBcDnxwr3XeAM4B7jKzxUAloL4iEYmEu7OzZ4BdPUkSJUbvQDr8Mh/gsCnV1FQk6OxLsa2jj01t3Wxo6w6/5AeoKE2QTGWYXFNGKu3s6hlgy84eOvtSdCdTvJ0f9+WJEuqrymisq6CqrIQ5DVUsXTCVBY21Y9f4LJEFg7unzOxa4EEgAfzA3Veb2S1As7svA64Hvmdmf0VwIPrP3D0/+0oiMuGkM05n3wAtnf3s7E7S2ZeiL5WmZXc/O7r7SaUdDHqTaXZ0J+nuT9GTTNPZl2J37wA7uvvpG8iM+PWm1JQzva6CmopSuvr7KTFj++4+yhIlTKoqY+mCqdRVllJbWUpDVTn1VaVMri6nMXyOOyRKjHTG6U+leXV7F1Nry6mvLKW0pIRZkyqZVF1GeaIEM8vjO5fLxvv3blNTkzc3Nxe6DBEZgb6BNG/u6qW9O8mbHX3UViToH8gM/fLe3TtAa1c/APWVZdRXlZEogc6+FF19KSrLErzR3sOu3iQA7lBZlmBabTm7egbY2NZNf2r4L/aS8Hu1tKSEqvIEU2rKqa0opaosQU1FgobqcqbUlDO7oYrJ1WWkM05VeYKGqnJqKhK8uauP3oE0tRWlzKivYP60GhqqyyN53/LBzFa4e9Nwy6LsShKRCS6Tcdq6+9nU1sPzb+zktZYu0hmnuz/F1l29rH2rk9R++lSqyhJMrg7CwB1e3d5JMpUhnXEaqsupLk/Qn8owd3IV86fVYAYJM7r6U7R3J5nTUMXpC6cxrbaCmZMqmVRVxuTq8qHgmFJTjjuUlBzaL+8TDns778z4omAQkWG5Oxvbumnp7KcnmWLLzl7aupL0p9L09KepryqlJ5lmc3svLZ19tOzup70nSTLrF/vUmnKqyhOUl5Ywe1IVV5+xgMOnVjO9vpJZkyrZ2T1AdXmCo6bXUlOR/6+jCHtjxjUFg0hMpTPOupYuXt/RzfbdffSnMry6vZMdXUl29QbdMu3dyX2elygxKktL6BlIU1maYN6UKmbUV7Jweh1TasqYUV/J/Gk1HD+vgam1FQVombxdCgaRGEilM6zd3klbV5JnN+5g5eYOVm7eRWd/Kme9huoyptVW0FBVxjnHTOekwyczvb6C6vJS5k2pZkZdBWZGosRIpYPTLA+1a0aKl4JBZIJIZ4Kun1e3d/La9i5ebelke0ewJ7CprXsoBBIlxqIZdfz+8bM56bDJzG+sYU5DFWWJEiZXl4347JfSRLFdHytjRcEgUqQG0hk2t/eQcejoTbKxrYflG9tZ39rFQDrD9PpKKkpLeKujj4FwjyD7VMs5DVVMrglOdXzvO2ZxyvypTK+r4B3zGiLpz5fxS58OkQj1p9KsfauT9u4kW3f10rK7n5bOfl7YvAt3J5nO8OauXmorStnVM7DPGTx1FaUcPq2a2opSVm/twMyY3VBJdXkplzXNY8nsehbPqmdBYy21+vKXQ6RPjsgYWvtWJ4+vbaE7maY/lWZHV5LpdRW0dfXz4pYO1m7vZO9Lh6rLExw9sw53aKyr4JT5U+nqTzGjroIjp9dSliihqizBYVOqOXZ2vfr0Je8UDCKjNJDOsLM7SVtXkqc27OCp9W280d7DznAsm2zlpSUkUxnKS0s4YV4Df37afJbMrqehupwjptYwo76CskQJZeqvlyKiYBAZRibjrNm2m67+FK9t76S1s5/WriTrW7tYtaWD3oH00LqNdRUc2VjD3MnVnLmokfOPnUFdZRmVpSWUmNHZl6K+qjTSIQ1E3g4FgwjBXsCqrR0s39jOf6/fwUtbO/Y5h7+yrIQjptbwvhPmcGRjDXWVpSxdMJXDp9YccNuTqsvyWbrImFMwSKy0dvbz7MZ2Vm7ZRWVpCd3JNMs3tbNqa8dQ3//cyVW8a8FU3n3UVBqqyjl6Zh1zJ1dRURrtQGYihaJgkAmrqz/FU+t3sKsnybMb2/m357fuc5aPGSyeWc8HTprHSUdM5vh5DSycXqsAkFhTMMi4l844GXfKEiVkMs7TG3fwk6df55GXW4ZG2qwqS3DiYZM5ef4Uli6YStMRkxlIZyhLlFBZlihwC0SKi4JBxpVUOsOmHd2seH0nD69p4bk3dtLenaS0xDh6Zh0tnf20dvZTW1HKJe+YzfnHzmRSVRkLp9cyuSZ3iGQFgsjwFAxS9HqSKX71wpv88H828cpbnUPzJ1WVsaCxhsWz6ihLlLChtZvFs+r55HkzufT42VSX6+Mtcij0P0eKVkfPAPc2b+a7T2ygraufRTNq+ZNTDuPIxlreMa+B4+ZO0vn/InmgYJCi0d6d5LdrtvPMxnaeWt/Gtt19uMPSBVP46h8fx5kLG3XVr0gEFAxSMG919PHW7j7uX7WNx15p4bWWrqFltRWlXP7Ow/jDE+fQdPhknSUkEiEFg+TVY6+08OgrLfzP+jY2t/eSTO//RuuNdRV88rxFXHzcLOordVGYSKEoGGTM7exOcs3dK3h6Q/tB1126YArXnr2Qo2fW0Vinu32JFAMFg4yp7zy+nq888MrQdEVpCTPqKzlvyQym1JRz7uIZzGqo1B6BSBFTMMjbtqOrn5uXrea+F7cNzbv1j47jA03zCliViBwqBYOM2r3LN3PDv7447LL3HT+br3/geBI6e0hk3FIwyEGl0hm6+9O845aH9rvONWcdySXvmM3iWfURViYi+aBgkP3q6Bngqh8up/n1nfssW3btqSyaUcfrO3o4emZdAaoTkXxRMEiOe5s3c8Mvhu8m+t6Hmzhj0TQqSveMMaRQEJl4FAwy5J5n3+DGX67KmXflqfP53HsX6wIzkRhRMAjuzrX/8jz3rQrOKnr2M+eAwfS6ygJXJiKFoGCIMXfn+ntX8svntwJQWmK88Pnzqa3Qx0IkzvQNEDM9yRTPbmzn7mfeoLGuYigUJleX8cQNZysURETBECePr23hz/55ec686vIEz33uPN20RkSGaDD7mHh52+59QgGg+aZzFQoikkN7DDHwN79ew/f/ayMAP//Yu3jnEVMKXJGIFLNI9xjM7AIzW2tm68zsxv2s8wEzW2Nmq83sX6KsbyK659k3hkLh1vcfp1AQkYOKbI/BzBLAbcB5wBZguZktc/c1WessBD4NnOruO81selT1TUTrW7uGrkt49rPn6PRTERmRKLuSTgbWufsGADO7B7gUWJO1zv8CbnP3nQDu3hJhfRNGKp3hqM/+Zmj6getOVyiIyIhF2ZU0B9icNb0lnJdtEbDIzP7bzJ42swuG25CZXW1mzWbW3Nramqdyxyd3zwmFGy88hmNmamA7ERm5Yjv4XAosBM4C5gJPmNnvuvuu7JXc/U7gToCmpiaPushidesDr3D74+uB4J7Jq75wvoayEJFRizIYtgLZd26ZG87LtgV4xt0HgI1m9ipBUOx7nqXk+NiPV/DA6reGpptvOlehICKHJMqupOXAQjObb2blwOXAsr3W+XeCvQXMbBpB19KGCGscl/70+88MhcKpR03l5Vsu0LUJInLIIttjcPeUmV0LPAgkgB+4+2ozuwVodvdl4bLzzWwNkAb+r7vviKrG8ejrD63lydfaAHj60+cwc5IOMovI22Pu47uLvqmpyZubmwtdRkH8YsUWPvXzlQCs/Pz5TKoqK3BFIjJemNkKd28ablmxHXyWEfrWI6/x9w+/CsB9f3maQkFExozGShqHXt3eORQK3/9IE8fOnlTgikRkItEewzjTn0pz/jeeAOD+vzydJbN1jYKIjC3tMYwzR9/0AAAnHT5ZoSAieaFgGEceyrpO4Rcfe1cBKxGRiUzBME5sauvm6h+vAIIB8XTxmojki4JhHBhIZzjra48D8JfnLNSAeCKSVwqGItfRM8DCcFC8uspSPnneogJXJCITnYKhyL37y48MPX7x8+cXsBIRiYsRna5qZmUEQ2af4+6r81uSAPzVz17g357fM8bg8587T8cVRCQSI9pjCEc7HQDG9/gZ48Rzb+zMCYXmm85lck15ASsSkTgZzQVu/wh82sw+6u6pfBUUd+7OH97+PwCcsaiR3zu6kWm1FQWuSkTiZDTBcDpwJrDVzF4CurMXuvslY1lYXF1259MAzJpUyY+uPLnA1YhIHI0mGNqAf81XIXH36vbOoaEuAB69/qzCFSMisTbiYHD3j+azkDjrTaZzQuGmixdTVa4b7YhIYYz4dFUze9TMGoaZX29mj45tWfmXyTj/5+4VPLOh8PcBWnzzA0OPv/unJ/Hnpy8oYDUiEnej6Uo6Cxju1JhKguMP40pnX4r7V73Fk6+1seoL78n76/UNpPnUz1fymYsWM7uhitd3dHP3M2/QsrtvaJ2X/vo91FZowFsRKayDfguZ2YlZk8eZWXvWdAJ4D7CV8WbwkoCITsC95icreGxtK79+cRtP3nA2Z3718Zzld//5KQoFESkKI/kmaib4+nTgoWGW9wJ/MZZFRWHwWrEocsHdae9ODk2ffutjOctPXziNU4+aFkElIiIHN5JgmE/w+3oD8E6Cs5MGJYEWd0/noba8KgmTIYp7Xn/zkddYuaWD+dNq2Ni25yzfB647nYrSBPOn1eS9BhGRkTpoMLj76wBm9mXgne5+R/ZyM/uYmc1x98/lqca8GOxJyuQ5FzIZ5x9++xoAH1p6OFedNp/bHlvHRb87S4EgIkVpNIPofQh4fpj5zwEfHptyouND/+Y3GRZ85v6hx1edNh+Aj599lEJBRIrWaIJhOtA6zPw2YMbYlBOdwS6kfPYkPZh1x7UHrht3J26JSEyN5jSYNwhOS92w1/wzgC1jVlHE8pEL7/7SI2zb3TcUOqcvnMYxM3V/ZhEZH0YTDN8FvmFm5cDgBW3nAF8CvjLWheWb7/NgbOzqSfJmR1/OvDv/tGlsX0REJI9GMyTG181sGvAt9lzolgS+6e635qO4fBr8NT8Wxxjauvpp+uJvOevoRq4+I/eq5SdvOFvDW4jIuDKqK6rc/dNm9kVgSTjrZXfvGvuyonOoxxj+9r41nLawkTMXNXLxt54E4PG1rTy7Mbj+74OnHMb/u+AYJlWVjVWpIiKRGPWltu7eDSzPQy3RCgMhcwjJ0Lypne89uZHvPbmRY2bWsX13/9CynmRwScff/cHvjkmZIiJRi+09nwe7kA5lh+GP7nhq6PErb3WOUUUiIsUhtoPzDB1jGKODz3917iI+fvaR3LdqG+89bvbYbFREpABiu8cwWp19A1x513IefWU7AMfNncSchioAbrzwGD5x7kJKEyVcevwcEiV2oE2JiBS1+O4xjHL9u595g0dfaeHRV1oAuPLU+bzvhDljX5iISIHFdo9hNIPnDaQzfPk3r+TMO2p67ViXJCJSFOIbDKNYd/mm9n3mLWjUWEciMjHFtitppNIZ54PfewaAqrIEv73+TGbUVVCaiG2misgEF+m3m5ldYGZrzWydmd14gPXeb2ZuZnkbS2KkPUmtnXuuUXjxC+czp6FKoSAiE1pk33BmlgBuAy4kuHL6CjNbMsx6dcAngGfyWc9Ih8LY2bPnzmtlCgQRiYEov+lOBta5+wZ3TwL3AJcOs97fEAzK1zfMssgNHnT+0ZUnF7gSEZFoRBkMc4DNWdNbwnlDzOxEYJ6733egDZnZ1WbWbGbNra3D3SJiBEbYlfSfrwbbr9ZAeCISE0XTN2JmJcDfA9cfbF13v9Pdm9y9qbGx8ZBeb6RnJV16fHAV80mHTz6k1xERGW+iDIatwLys6bnhvEF1wO8Aj5vZJmApsCxfB6BHevD5Vy+8CYCZrmYWkXiIMhiWAwvNbH54s5/LgWWDC929w92nufsR7n4E8DRwibs3R1ijiEjsRRYM7p4CrgUeBF4G7nX31WZ2i5ldElUdQ/WMoDOpPxUMoX3BsTPzXY6ISNGI9AI3d78fuH+veTfvZ92z8lvLwdd5bXtwD6KmI3R8QUTio2gOPhej9/7jfwEM3ZVNRCQOYhsMoxkr6aaL97kOT0RkwopvMIxidNXDplbnsRIRkeIS20H0RpILRzbWcMzM+vwXIyJSRGK7xzASHb0p6qtim50iElMKhv1wdzp6kzRUlxe6FBGRSMU2GA7WldSTTDOQdhqqyqIpSESkSMQ2GLr6UwdcPjjc9mTtMYhIzMQ2GL5435oDLt/VMwDApGrtMYhIvMQ2GHaGX/z7853H1wNQV6mDzyISL7ENhoO5b9U2AEo0qqqIxIyC4SBOPmJKoUsQEYlUbIPhQPsBA+nM0OOSEu0xiEi8xDYYDnS26sa27sjqEBEpNrENhgNJpoI9hm9/8IQCVyIiEj0FA7kD6r20tWNouO2pNRWFKklEpGAUDEA6sycYBkMB0DhJIhJLsQ2G7L2EVGb4Iw71lbq4TUTiJ7bBkC37LKRsCgYRiaPYBoNlXbg2kB5+j0HDYYhIHMU2GLIN7jGM5q5uIiITlYKBPaen9g3s6VJa+fnzC1WOiEhBKRjYs8ewqzcYavumixczSfdhEJGYim0wDHdW0i+f2wrAf7y4rSA1iYgUg9gGQ7bBrqRjZ9cD8MnzFhWyHBGRglIwsKcrafAYQ2OtrngWkfiKbTAMd7pq30AagKryREFqEhEpBrENhmwD6QxbdvZw3c9eAKCqTMEgIvGlYACS6Qz//vzWoWkFg4jEWWyDIfv2O6m0U591empleWzfFhGRGAdDVjIMpDM54yKVJ2L7toiIxDcYsg2kM6zd3jk0nX1gWkQkbhQMBNcxfOfx9YUuQ0SkKMQ2GHK7kpxrzjoSgCdvOLtAFYmIFIdIg8HMLjCztWa2zsxuHGb5J81sjZm9aGaPmNnhUdS1raOXV7btpixhzJtSHcVLiogUrcjuXWlmCeA24DxgC7DczJa5+5qs1Z4Hmty9x8yuAW4FLst3bf/46Lp8v4SIyLgR5R7DycA6d9/g7kngHuDS7BXc/TF37wknnwbm5qsYQweYRUSGE2UwzAE2Z01vCeftz1XAb4ZbYGZXm1mzmTW3trYeUjEX/M7MQ3qeiMhEV5QHn83sQ0AT8NXhlrv7ne7e5O5NjY2Nh/QasyZVvo0KRUQmrsiOMQBbgXlZ03PDeTnM7Fzgs8CZ7t4fUW0iIhKKco9hObDQzOabWTlwObAsewUzOwH4LnCJu7dEWBsAf3DCgXq2RETiIbJgcPcUcC3wIPAycK+7rzazW8zsknC1rwK1wM/N7AUzW7afzeXF4LUMIiJxFmVXEu5+P3D/XvNuznp8bnS17Dtv0Yy6qF5eRKRoFeXBZxERKRwFg4iI5FAwiIhIjtgGwzCHGEREhBgHw6CScGSMn129tLCFiIgUiUjPSipGv7jm3fT0pzllwdRClyIiUhRiHwyNtRXMO0xDbYuIDIp9V5KIiOSKbTD4cFe4iYhIfINBRESGp2AQEZEcCgYREcmhYBARkRyxDQYdehYRGV5sg2GQWaErEBEpLrEPBhERyaVgEBGRHAoGERHJEd9g0NFnEZFhxTcYQqajzyIiOWIfDCIikkvBICIiORQMIiKSI7bB4Dr6LCIyrNgGwyAdehYRyRX7YBARkVwKBhERyRHbYNCdPUVEhhfbYBik69tERHLFPhhERCSXgkFERHIoGEREJEdsg0HHnkVEhhfbYBhkusRNRCRH7INBRERyRRoMZnaBma01s3VmduMwyyvM7Gfh8mfM7Igo6xMRkQiDwcwSwG3AhcAS4AozW7LXalcBO939KOAbwFfyVU9PMp2vTYuIjGtR7jGcDKxz9w3ungTuAS7da51LgR+Gj38BnGN5usXaqi27AChN6BiDiEi20ghfaw6wOWt6C3DK/tZx95SZdQBTgbbslczsauDqcLLLzNYeYk3TGr+Su+0YmAZqcwyozfHwdtp8+P4WRBkMY8bd7wTufLvbMbNmd28ag5LGDbU5HtTmeMhXm6PsStoKzMuanhvOG3YdMysFJgE7IqlORESAaINhObDQzOabWTlwObBsr3WWAR8JH/8R8Ki7xkEVEYlSZF1J4TGDa4EHgQTwA3dfbWa3AM3uvgz4PvBjM1sHtBOERz697e6ocUhtjge1OR7y0mbTD3IREcmmK59FRCSHgkFERHLENhgONjzHeGJmPzCzFjN7KWveFDN72MxeC/+dHM43M/tW2O4XzezErOd8JFz/NTP7yHCvVQzMbJ6ZPWZma8xstZl9Ipw/kdtcaWbPmtnKsM1/Hc6fHw4fsy4cTqY8nL/f4WXM7NPh/LVm9p7CtGjkzCxhZs+b2a/D6QndZjPbZGarzOwFM2sO50X72Xb32P0RHPxeDywAyoGVwJJC1/U22nMGcCLwUta8W4Ebw8c3Al8JH18E/AYwYCnwTDh/CrAh/Hdy+Hhyodu2n/bOAk4MH9cBrxIMszKR22xAbfi4DHgmbMu9wOXh/DuAa8LH/we4I3x8OfCz8PGS8PNeAcwP/x8kCt2+g7T9k8C/AL8Opyd0m4FNwLS95kX62Y7rHsNIhucYN9z9CYKzuLJlDy/yQ+B9WfN/5IGngQYzmwW8B3jY3dvdfSfwMHBB/qsfPXff5u7PhY87gZcJrpqfyG12d+8KJ8vCPwd+j2D4GNi3zcMNL3MpcI+797v7RmAdwf+HomRmc4GLgX8Kp40J3ub9iPSzHddgGG54jjkFqiVfZrj7tvDxW8CM8PH+2j4u35Owu+AEgl/QE7rNYZfKC0ALwX/09cAud0+Fq2TXnzO8DDA4vMy4ajPwD8ANQCacnsrEb7MDD5nZCguG/4GIP9vjckgMGR13dzObcOclm1kt8K/Ade6+27LGW5yIbXb3NHC8mTUA/wYcU+CS8srM3gu0uPsKMzur0PVE6DR332pm04GHzeyV7IVRfLbjuscwkuE5xrvt4S4l4b8t4fz9tX1cvSdmVkYQCne7+y/D2RO6zYPcfRfwGPAugq6DwR942fXvb3iZ8dTmU4FLzGwTQXfv7wHfZGK3GXffGv7bQvAD4GQi/mzHNRhGMjzHeJc9vMhHgF9lzf9weDbDUqAj3EV9EDjfzCaHZzycH84rOmG/8feBl93977MWTeQ2N4Z7CphZFXAewbGVxwiGj4F92zzc8DLLgMvDM3jmAwuBZ6Npxei4+6fdfa67H0Hwf/RRd/8TJnCbzazGzOoGHxN8Jl8i6s92oY/AF+qP4Gj+qwT9tJ8tdD1vsy0/BbYBAwR9iVcR9K0+ArwG/BaYEq5rBDdMWg+sApqytnMlwYG5dcBHC92uA7T3NIJ+2BeBF8K/iyZ4m48Dng/b/BJwczh/AcGX3Drg50BFOL8ynF4XLl+Qta3Phu/FWuDCQrdthO0/iz1nJU3YNodtWxn+rR78bor6s60hMUREJEdcu5JERGQ/FAwiIpJDwSAiIjkUDCIikkPBICIiORQMIgdhZkeYmZtZUd1ovljrkvFPwSBycJsJRnR9AcDMzgq/kKdFVYCZPW5m3z5QXSJjRWMliRyEB2MUvZWPbZtZmbsPHMpz81mXxJv2GKSohL+MbzezvzOzNgtuQPQ1MysJl28ys08N85xvZ01vMrObzewuM+s0s81mdpmZNZjZPWbWFd685PwR1jTUZROO5vpYuKg1nH9XuJ6Z2Q1mtt7Mei242cqHhtnOFWb2qJn1Av/bzKaa2U/NbEv4vNVm9tGs590FnAl8PHy+h9vapyvJzM6w4CY1fWa23cy+EQ77MqL3N1znDy246UuvmbWb2X+a2eBonhIDCgYpRn8CpIB3A9cC1wGXjXIb1xEMi3AiwY1dfkhws5f7geOBJ4CfmFnlKLe7GXiF9ewAAAMnSURBVHh/+PhYgq6cT4TTXyQYjuTjBDeH+RLwXTO7eK9tfAm4PVzn3wmGcngOeG+4zW+GzzsnXP8TwFPAP4evN4vcIZUBMLM5BDdteZ5gKPKrgCvC18u23/fXzGYSDFj3Q2AxwU2gfjyC90UmkkKPDaI//WX/AY8DT+0172Hgn8LHm4BPDfOcb2dNbwJ+mjVdSzC20rey5h0RzmsaQU056xKM2+Nk3WULqAF6gdP3eu4/APfvtZ3rR/Ca9wy2ebg27qeuvyUYS6cka50/A/qB6hG+vyeG2zy80J8F/RXuT8cYpBi9uNf0m8D0Q92Gu3eZWQ/BIGODtof/jna7+7OE4Jf/A5Y7Vn4ZQVBla86eMLMEwe0aLyO4mUoFwS1nHx9lDYuBp909kzXvv8JtHcWe9+RA7+9KgkHaXjKzh8LHv3D31lHWIuOYgkGK0d4HY5093Z4ZghEls5WNcBsDe03D2HWnDm7n94E3DlJL917TnwKuJ+gyWgV0AX/H2IUW7GnvcPUMvb/ung6PvSwlGKr5KuBLZnamu68cw3qkiCkYZLxpJehjByA8RnAMQb96VJLhv4mseWsIumwOd/dHR7m904D/cPcfw9D9JhYBu/Z6zcQwz832MvABMyvJ2ms4LXzu+pEW4+5OcEzjKTO7hWD458sI9iYkBhQMMt48ClxpZssIQuKzRP85fp3gV/bFZvYfQK+7d5rZ14CvhV/sTxAc21gKZNz9zgNs71XgMjM7DWgD/gKYT27YbQJODs+K6gLah9nO7QQHkm83s28SjO3/ZYJjEz0jaVh4s5dzCW7qsp3gIPY8guCTmNBZSTLefIkgHH4FPETQhx7l3gIe3Hrx8wQHe7cDg6fKfg74AkHX0GqCg7rvBzYeZJNfJDiD6jcEgdIN3L3XOl8j+OW/hiAQD9tPXRcSfJm/APyA4CZOnxlF8zoIbqn5a4ID2V8H/sbdfzKKbcg4pxv1iIhIDu0xiIhIDgWDxJ6Z3RFeDT3c3x2Frk8kaupKktgzs+lA/X4W73b3lijrESk0BYOIiORQV5KIiORQMIiISA4Fg4iI5FAwiIhIjv8P8/I6ISL3f3AAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"M-s1_N1fOdAL"},"source":["Aside: interactions\n","\n","You'll notice in the arguments we supply to VW, **we include `-q UA`**. This is telling VW to create additional features which are the features in the (U)ser namespace and (A)ction namespaces multiplied together. This allows us to learn the interaction between when certain actions are good in certain times of days and for particular users. If we didn't do that, the learning wouldn't really work. We can see that in action below."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"voXxj6dwOdAM","executionInfo":{"status":"ok","timestamp":1634747763631,"user_tz":-330,"elapsed":1765,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"22ddf753-545e-4238-fb6f-be63e5d4c8fc"},"source":["# Instantiate learner in VW but without -q\n","vw = pyvw.vw(\"--cb_explore_adf --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfv0lEQVR4nO3deZhcdZ3v8fe3lt476U7SCSELBAhL2CS2ES87omwK4/IMxPG6DI9cR3F0xuXCOKKDDpsrKAwyLriCDDNXg0YQZRMUTEcCksRAE0I6CUk6S3d6ra7u+t4/zummTqc66Yb06aU+r+epp+ssder7q1TqU+d3Tv2OuTsiIiL9EmNdgIiIjC8KBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkYjYgsHMvmdm283s2SGWm5ndbGaNZvaMmS2OqzYREXlFnHsMdwDn7WP5+cDC8HY58B8x1CQiIoPEFgzu/iiwax+rXAz80ANPADVmNjue6kREpF9qrAvIMwdoypveFM57efCKZnY5wV4FlZWVrz/66KNjKVBEZLJYuXLlDnevK7RsPAXDsLn77cDtAPX19d7Q0DDGFYmITCxm9tJQy8bTWUmbgXl503PDeSIiEqPxFAzLgPeFZyedDLS6+17dSCIiMrpi60oyszuBM4EZZrYJ+DyQBnD324DlwAVAI9AJfDCu2kRE5BWxBYO7L93Pcgc+GlM5IiIyhPHUlSQiIuOAgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhIRazCY2Xlmts7MGs3sygLL55vZQ2b2lJk9Y2YXxFmfiIjEGAxmlgRuAc4HFgFLzWzRoNX+Fbjb3U8CLgVujas+EREJxLnHsARodPf17t4D3AVcPGgdB6aE96cCW2KsT0REiDcY5gBNedObwnn5vgC818w2AcuBjxXakJldbmYNZtbQ3Nw8GrWKiBSt8XbweSlwh7vPBS4AfmRme9Xo7re7e72719fV1cVepIjIZBZnMGwG5uVNzw3n5bsMuBvA3f8IlAEzYqlORESAeINhBbDQzBaYWQnBweVlg9bZCLwZwMyOIQgG9RWJiMQotmBw917gCuB+YC3B2UerzewaM7soXO2TwIfM7GngTuAD7u5x1SgiIpCK88ncfTnBQeX8eVfn3V8DnBJnTSIiEjXeDj6LiMgYUzCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIRKzBYGbnmdk6M2s0syuHWOdvzWyNma02s5/GWZ+IiEAqricysyRwC/AWYBOwwsyWufuavHUWAlcBp7j7bjObGVd9IiISiHOPYQnQ6O7r3b0HuAu4eNA6HwJucffdAO6+Pcb6RESEeINhDtCUN70pnJfvSOBIM3vczJ4ws/MKbcjMLjezBjNraG5uHqVyRUSK03g7+JwCFgJnAkuB/zSzmsErufvt7l7v7vV1dXUxlygiMrnFGQybgXl503PDefk2AcvcPevuLwLPEQSFiIjEJM5gWAEsNLMFZlYCXAosG7TOzwn2FjCzGQRdS+tjrFFEpOjFFgzu3gtcAdwPrAXudvfVZnaNmV0UrnY/sNPM1gAPAZ92951x1SgiImDuPtY1vCb19fXe0NDwqh777OZWFs6qojSVPMBViYiMb2a20t3rCy0bbwefY7O5pYu3ffMxPv+L1WNdiojIuFK0wdDamQVgVVPLGFciIjK+FG0wiIhIYUUbDGbB379ubRvbQkRExpmiDQYRESlMwSAiIhFFGwz9XUkiIhJVtMEgIiKFKRhERCRiWMFgZmkz22pmx452QSIiMraGFQzungWywMQePyPPBB8JRERk1IykK+mbwFVmFtvlQEVEJH4j+ZA/DTgD2GxmzwId+Qvd/aKCjxqntMcgIlLYSIJhB/Dfo1VI3HJKBhGRgoYdDO7+wdEsRERExodhH2MwswcLXX/ZzKaY2YMHtqzRt/blPWNdgojIuDSSg89nAiUF5pcRHH+YUL70q7VjXYKIyLi0364kM1ucN3mCme3Km04C5wKbD3RhIiIyNoZzjKGB4PcLDvymwPIu4GMHsigRERk7wwmGBYAB64E3EJyd1K8H2O7ufaNQ26ia6Ne6FhEZLfsNBnd/CcDMrgfe4O635S83sw+b2Rx3/9wo1SgiIjEaycHn9wJPFZj/Z+B9B6YcEREZayMJhplAc4H5O4BZB6ac+KgjSUSksJEEw0YKn5Z6OrDpwJQTIyWDiEhBIxkS49vA182sBOj/QdubgeuAGw50YSIiMjZGMiTGV81sBnAzr/zQrQe4yd1vHI3iREQkfiMaQtvdrzKzLwGLwllr3b39wJclIiJjZcTXVnD3DmDFKNQSq/xDDLmck0jYmNUiIjKe6KI7wPodHRwxs2rUn6dxezu3PfICW1u7SSWN73/gDZgpkERkfBnJWUmTSvSXz8M/Rem7j73IUxt3D2zjY3c+xQU3/Z6e3tw+H7fypd2c87VHuGflJh5r3MHD65pZcNXyyDrLnt7C2V99mJ7eHM1tGe5e0aRfaItI7Gyif/DU19d7Q0PDiB937NX30dETjOTxwD+dTnN7hp88uZFb3rN4yMf09uU44rO/BuCuy0/m0tufiCz/+iUnMqOqlNMW1kXm53LOYf8SDYF893/idKaUp3jTdUOPXr7+2gtIJIzevhx3rWjiPUvm77P7y921NzLO9Pbl+OvWNjp7+jhyVhXtmV6e2tjC+uYOdnVkaMv0cuLcGk45YjqtXb08u7mVbF+OTG+OhBnzp1VQXpIgl4OpFWkWzqyipiI64HFzW4atrd3MqS1nWmUJ7Zleqkr37hjoyPTiUHCZFAczW+nu9YWWFe27YvD+wnv+80kA3vG6bZyzqPDv9Z588ZWBZQeHAsA//ezpyPQ7F8/hy+8+kbd/87GBeauufgtt3b1k+3Kc/dVHADj3G49y5Kx9d2UNDpZv/PZ5drRnAPjsBcfwodMPA+DpphYuvuXxgfVeuPYC2rqzfOD7K7jm4mM57uCpE+J4ytNNLdy/eitnHz2TP7ywk9rKEg6ZVkGfOy+3dNOV7ePtJ8xm5pQyNu7sxCzoElyzZQ8HTS3l8Loq6qpLWfnSbnp6c+xoz9CR6WPetAreuGAac2rKB16HvpyzbmsbB9eUDXzQtnZladiwi+a2DNmcs7W1i0efC4YJK00lWL+jg/J0koNryujNBe+mhBlGcHXAnR097GzvYUZVCdv2ZEgljLZM75DtTSaMdNL4nz8Pf6DiZMIoSyXo7s1x0JSygbr6laUTdGeDPdnplSWUphJsb8uQTiboygZfiipKkhgwvaqU2soSMtk+qstSHF5XxakLZ3DawjqmlqeHXZNMDkW7x7Do6vvoDPcY7v/E6Zz7jUcBeOuiWXz8nIVMKUszb1rFwPpPbdzNO279w17bufD42XzkrMO58ObH9lo22Kqr3xL5htfameW0Gx9kT/crHxiP/d+zOPWGhzi8rpKff/QU7nh8A1994LkRt29fbnvv6znvuINe83Yeea6Zx55vZt22dt61eA4XHj+bVDLonXR39nT1MrUizeotrbR2ZVk8v5afrWjijj9s4MXwA6wkmeDgmjK6szmOmV1NRWmKXz3z8rBrmFFVwo72nhHXXl2a4qRDamluy0Qu2jSjqpTykgRNu7r2esyUshQzqkvp7unjqIOqMTM27e4kmUjQl8vR2dNHZUmKnr4cU8pSzJxSRqY3R3k6QVVpmoqSJAtnVTG1PM2m3V0kzFg4s4oT5k1lZnUZ7s7qLXtYsWEXlSUpjjqomrJ0ksrSJCXJBC/u6GB3Zw/lJSlaOntYs2UPm3Z3UV2WYkd7Dzs7MixZMI2FM6vZ0tJF065OKkqS9PQ5bd1Z3IMgSKcSlKYSVJWmaNrVSWkqyfa2brqzOfpyTnuml5d2dtDR00dZOsEbF0zn9CPrOOuoOhbMqNSe6DjR/9n9av899rXHoGAAfv3x0zj/pt/vtc6G6y8cuH/olb+KLDtsRiU3Lz2J4+ZMBeCOx1/kC/euGfL5Pn3uUXz0rCP2mr99TzdLrv0dAL/6x1M59uCpBR+f7cuxqqmF367dxtI3zOfMrzzMnJpyvnDRsXzoh9H2f+a8o/iHMw7nxH/7zUDoJBNGX27//9aPX3k2c2rK2dme4fybfs/2tszAshPn1fB0U8t+t3GgXPuO4/lZQxPHz5nCCXNqMINU0phSliaVTPDk+p08v72duupSZk8pY05tOSfNr2Xd1ja2t3WzuyPL/OnlTC1PM7c2CPkNOzp4blsbf97YwqqmFmor0iw6eCpHzqyipSvLSzs7aevOUltRwrnHzeKIumrMoLI0xdzactLJ4jgs153t45lNrfx81WYeb9zBSzs7gSBQp5SnmV5VwtEHVVN/6DROWziD2VPLx7ji8a+1K0t7ppe27iwbd3ays6OHbXu62d0RhH1tRZqaijTtmT62tnbRl4NdHRmyfT7QpTi1PI0ZbG3tZtPuLr70juM466iZr6oeBUMBx3zuvoHd6eX/eBof/vFKNu7qjKyz4foLWd/czsE15Rz9ufsG5jf++/kD34wHa27L0J0Nuiy+8/v13HDfX/nyu0/kb06aM+IaR6I728fz29pZOKuKsnQSCN6Itz7UyIfPOJzaymBPpXF7O+d87ZED8pxVpSmuOPsIDplWwRd/uYYtrd0F18v/Vn/mUXXc+K4TmDmljFzO6XNnT1eWPneeXL+L+dMqOHHeXleQlTH20s4OHl7XzJote2jLZNm2J0Pj9nZau7IAHDK9gkOmV7JgegXHzpnKvNoKaiqCD7GjZlUfkL2M7mwfzW0ZVm/Zw56uLCWpBDvaMzzV1MLO9gyN24O90JqKNLmc09KVpSyVoKwkSVkqyeypZfT05aipKKGrp4893VlyOWdXZw97unqpqUhz2IxKevpyzKwuJWHGjvYeMr19mBktnT3s7uyht88pTSVwoLfPmV5VwszqUjK9ObJ9OZp2dQ18trRnejGC7sf8noF8VaUpurJ9kS9uJakEqYRRVZqivCRJKmGUpZO0Z3pxh9qKNLOmlPGeN87nzIkeDGZ2HnATwZXfvuPu1w+x3ruAewiG+d7np/6BCIZffuxUrrl3DX/asCuyTv8H2vxpFQOhcdGJB3Pz0pNG/Hzjyfrmdta+3Mbv1m7jI2cdTsOG3Vx4wmy+9VAj335k/cB6SxZM40OnHcbRB1Xz4o4OfvrkRsrSCW5894mUpPYORndnw85OSlMJZlaXDhmeMjnkcs6zW1r5wws7eWZTcBB9467OgT3xfgdPLeP1h04b+KDryvYxvbIEM8NxDp1eSWtXlmxvjo6ePnZ3BB/GXdk+dndm6enN0Z7ppXF74d/SzqgqZU5tOYfNqKQ0laC5LUMyYUyvKiWT7aM900t7ppcd7RlSiQStXVkqS5OUp5OUlySZWp5mWmUJzW09rN/RTkkyOBaTMKO2Ik15SfBFq7aihCnladIJo6cvh1lwXGj7ngzNbRlSyeDD++CacqrLUuRyTkVJioRBpjfH7JoyaitKqChJMn9aBbOmlDGtsoSydBJ3py3Ty56uLGXpJDOqSkf9329cBIOZJYHngLcQDLq3Aljq7msGrVcN/Ipg2I0r4giGe684lWuXr+WP63fu93F//eJ5A9/IJ6v+fmYddJSRyuWc57e3s7mlky0t3aQSxsPrmnlmUwud2T56enOUp5O0dmUxg96ck/8RlE4alaUpEhaEyMzqUspLgg/xhbOqqKsq5YR5NdRVldLaleXgmnJqK9I67vEqjJezkpYAje6+PizqLuBiYHDH/BcJBuX79GgW43nnJV1/31r6hhGQK//1nEkfChAcj1AoyKuRSBhHHVTNUQdVD8y7dMn8guu6O305Z3NLF7WVJZQkg4Piw/2Qn3dAKpZC4tzXnwM05U1vCucNMLPFwDx3jx7pHcTMLjezBjNraG4udImIkXm8cWfkB2qzp5YVXG96DLt3IsXCzEglExwyvZIpZWnK0kl98x8nxk0nsJklgK8Bn9zfuu5+u7vXu3t9XV3d/lYfYhvR6VVNLaSTwZty8Lflz799ES9ed8Greh4RkYkmzmDYTHTvb244r181cBzwsJltAE4GlplZwT6w0ZDtC9KivCTJde88HjP49v9+PR88ZYG+yYhI0YjzGMMKYKGZLSAIhEuB9/QvdPdWYEb/tJk9DHxqfwefR8MJc6aydMl8lg7RNyoiMpnFFgzu3mtmVwD3E5yu+j13X21m1wAN7r4srlr25d4rTo0cOBMRKTaxjpXk7suB5YPmXT3EumeOai1DzD9+buFfHouIFItxc/A5dgWS4Z4Pvyn+OkRExpniDYYCdDqqiEgRB4MX2GXoP11VRKSYFW0wFFIsI2eKiOyLPgnzKBhERBQMEepKEhEp4mAoNGZeKlG0L4eIyICi/SQcnAv/csHRA+Oui4gUs6INhnxvXTSLy08/fKzLEBEZF2L95fN4dMVZR/Cpc48a6zJERMaNot9jKPR7BhGRYla0waDzj0RECivaYNB+gohIYUUbDCIiUljRBoO6kkRECivaYFBXkohIYUUbDCIiUljRBoO6kkRECivaYBARkcIUDCIiEqFgEBGRCAWDiIhEFH0wFLoug4hIMSv6YBARkaiiDwbTeasiIhFFHwzqShIRiSraYDj3uIMAOHXhjDGuRERkfCnaYKg/pBaARbOnjHElIiLjS9EGg4iIFKZgEBGRCAWDiIhEKBhERCRCwSAiIhFFGwz6/YKISGGxBoOZnWdm68ys0cyuLLD8n81sjZk9Y2a/M7NDRr0mXbJHRCQitmAwsyRwC3A+sAhYamaLBq32FFDv7icA9wA3xlWfiIgE4txjWAI0uvt6d+8B7gIuzl/B3R9y985w8glgboz1iYgI8QbDHKApb3pTOG8olwG/LrTAzC43swYza2hubj6AJYqIyLg8+Gxm7wXqgS8XWu7ut7t7vbvX19XVxVuciMgkl4rxuTYD8/Km54bzIszsHOCzwBnunompNhERCcW5x7ACWGhmC8ysBLgUWJa/gpmdBHwbuMjdt49mMTpbVUSksNiCwd17gSuA+4G1wN3uvtrMrjGzi8LVvgxUAf9lZqvMbNkQmztwdLaqiEhEnF1JuPtyYPmgeVfn3T8nznpERGRv4/Lgs4iIjB0Fg4iIRCgYREQkomiDwTWKnohIQUUbDP1MZyWJiEQUfTCIiEiUgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCSi6INBZ6uKiEQVfTCIiEiUgkFERCIUDCIiEqFgEBGRCAWDiIhEFG0waHBVEZHCijYY+pmGVxURiSj6YBARkSgFg4iIRCgYREQkQsEgIiIRCgYREYko2mBwdL6qiEghRRsM/XSyqohIVNEHg4iIRCkYREQkQsEgIiIRCgYREYlQMIiISETRBoNGVxURKaxog6GfBlcVEYkq+mAQEZEoBYOIiETEGgxmdp6ZrTOzRjO7ssDyUjP7Wbj8STM7NM76REQkxmAwsyRwC3A+sAhYamaLBq12GbDb3Y8Avg7cEFd9IiISiHOPYQnQ6O7r3b0HuAu4eNA6FwM/CO/fA7zZdO1NEZFYpWJ8rjlAU970JuCNQ63j7r1m1gpMB3bkr2RmlwOXh5PtZrbuVdY0o/KG6LaLwAxQm4uA2lwcXkubDxlqQZzBcMC4++3A7a91O2bW4O71B6CkCUNtLg5qc3EYrTbH2ZW0GZiXNz03nFdwHTNLAVOBnbFUJyIiQLzBsAJYaGYLzKwEuBRYNmidZcD7w/vvBh5012+URUTiFFtXUnjM4ArgfiAJfM/dV5vZNUCDuy8Dvgv8yMwagV0E4TGaXnN31ASkNhcHtbk4jEqbTV/IRUQkn375LCIiEQoGERGJKNpg2N/wHBOJmX3PzLab2bN586aZ2QNm9nz4tzacb2Z2c9juZ8xscd5j3h+u/7yZvb/Qc40HZjbPzB4yszVmttrMPh7On8xtLjOzP5nZ02Gb/y2cvyAcPqYxHE6mJJw/5PAyZnZVOH+dmZ07Ni0aPjNLmtlTZvbLcHpSt9nMNpjZX8xslZk1hPPifW+7e9HdCA5+vwAcBpQATwOLxrqu19Ce04HFwLN5824ErgzvXwncEN6/APg1YMDJwJPh/GnA+vBvbXi/dqzbNkR7ZwOLw/vVwHMEw6xM5jYbUBXeTwNPhm25G7g0nH8b8A/h/Y8At4X3LwV+Ft5fFL7fS4EF4f+D5Fi3bz9t/2fgp8Avw+lJ3WZgAzBj0LxY39vFuscwnOE5Jgx3f5TgLK58+cOL/AD4m7z5P/TAE0CNmc0GzgUecPdd7r4beAA4b/SrHzl3f9nd/xzebwPWEvxqfjK32d29PZxMhzcHziYYPgb2bnOh4WUuBu5y94y7vwg0Evx/GJfMbC5wIfCdcNqY5G0eQqzv7WINhkLDc8wZo1pGyyx3fzm8vxWYFd4fqu0T8jUJuwtOIvgGPanbHHaprAK2E/xHfwFocffecJX8+iPDywD9w8tMqDYD3wA+A+TC6elM/jY78BszW2nB8D8Q83t7Qg6JISPj7m5mk+68ZDOrAv4b+IS777G88RYnY5vdvQ94nZnVAP8POHqMSxpVZvY2YLu7rzSzM8e6nhid6u6bzWwm8ICZ/TV/YRzv7WLdYxjO8BwT3bZwl5Lw7/Zw/lBtn1CviZmlCULhJ+7+P+HsSd3mfu7eAjwEvImg66D/C15+/UMNLzOR2nwKcJGZbSDo7j0buInJ3WbcfXP4dzvBF4AlxPzeLtZgGM7wHBNd/vAi7wd+kTf/feHZDCcDreEu6v3AW82sNjzj4a3hvHEn7Df+LrDW3b+Wt2gyt7ku3FPAzMqBtxAcW3mIYPgY2LvNhYaXWQZcGp7BswBYCPwpnlaMjLtf5e5z3f1Qgv+jD7r73zGJ22xmlWZW3X+f4D35LHG/t8f6CPxY3QiO5j9H0E/72bGu5zW25U7gZSBL0Jd4GUHf6u+A54HfAtPCdY3ggkkvAH8B6vO28/cEB+YagQ+Odbv20d5TCfphnwFWhbcLJnmbTwCeCtv8LHB1OP8wgg+5RuC/gNJwflk43RguPyxvW58NX4t1wPlj3bZhtv9MXjkradK2OWzb0+Ftdf9nU9zvbQ2JISIiEcXalSQiIkNQMIiISISCQUREIhQMIiISoWAQEZEIBYPIfpjZoWbmZjauLjQ/XuuSiU/BILJ/TQQjuq4CMLMzww/kGXEVYGYPm9m39lWXyIGisZJE9sODMYq2jsa2zSzt7tlX89jRrEuKm/YYZFwJvxnfambXmtkOCy5A9BUzS4TLN5jZpwo85lt50xvM7Gozu8PM2sysycwuMbMaM7vLzNrDi5e8dZg1DXTZhKO5PhQuag7n3xGuZ2b2GTN7wcy6LLjYynsLbGepmT1oZl3A/zGz6WZ2p5ltCh+32sw+mPe4O4AzgI+Gj/dwW3t1JZnZ6RZcpKbbzLaZ2dfDYV+G9fqG67zTgou+dJnZLjN7xMz6R/OUIqBgkPHo74Be4H8BVwCfAC4Z4TY+QTAswmKCC7v8gOBiL8uB1wGPAj82s7IRbrcJeFd4/1iCrpyPh9NfIhiO5KMEF4e5Dvi2mV04aBvXAbeG6/ycYCiHPwNvC7d5U/i4N4frfxz4I/D98PlmEx1SGQAzm0Nw0ZanCIYivwxYGj5fviFfXzM7iGDAuh8AxxBcBOpHw3hdZDIZ67FBdNMt/wY8DPxx0LwHgO+E9zcAnyrwmG/lTW8A7sybriIYW+nmvHmHhvPqh1FTZF2CcXucvKtsAZVAF3DaoMd+A1g+aDufHMZz3tXf5kJtHKKufycYSyeRt84HgAxQMczXd3G4zUPG+r2g29jddIxBxqNnBk1vAWa+2m24e7uZdRIMMtZvW/h3pNsdyiKCb/73WXSs/DRBUOVryJ8wsyTB5RovIbiYSinBJWcfHmENxwBPuHsub95j4baO4JXXZF+v79MEg7Q9a2a/Ce/f4+7NI6xFJjAFg4xHgw/GOq90e+YIRpTMlx7mNrKDpuHAdaf2b+ftwMb91NIxaPpTwCcJuoz+ArQD13LgQgteaW+hegZeX3fvC4+9nEwwVPNlwHVmdoa7P30A65FxTMEgE00zQR87AOExgqMJ+tXj0hP+TebNW0PQZXOIuz84wu2dCtzr7j+CgetNHAm0DHrOZIHH5lsL/K2ZJfL2Gk4NH/vCcItxdyc4pvFHM7uGYPjnSwj2JqQIKBhkonkQ+HszW0YQEp8l/vfxSwTfsi80s3uBLndvM7OvAF8JP9gfJTi2cTKQc/fb97G954BLzOxUYAfwMWAB0bDbACwJz4pqB3YV2M6tBAeSbzWzmwjG9r+e4NhE53AaFl7s5RyCi7psIziIPY8g+KRI6KwkmWiuIwiHXwC/IehDj3NvAQ8uvfh5goO924D+U2U/B3yBoGtoNcFB3XcBL+5nk18iOIPq1wSB0gH8ZNA6XyH45r+GIBDnD1HX+QQf5quA7xFcxOlfRtC8VoJLav6S4ED2V4EvuvuPR7ANmeB0oR4REYnQHoOIiEQoGKTomdlt4a+hC91uG+v6ROKmriQpemY2E5gyxOI97r49znpExpqCQUREItSVJCIiEQoGERGJUDCIiEiEgkFERCL+P6SIOrlIiRvSAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"7RpLurdyOdAN"},"source":["\n","### Without learning\n","Let's do the same thing again (but with `-q`, but this time show the effect if we don't learn from what happens. The ctr never improves are we just hover around 0.2."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"-agneYrYOdAN","executionInfo":{"status":"ok","timestamp":1634747765089,"user_tz":-330,"elapsed":696,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"046fe1b2-0801-4db6-d15c-ab9609b60b08"},"source":["# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, times_of_day, actions, get_cost, do_learn=False)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAe1klEQVR4nO3deZhcdb3n8fe39l7TCekESCIJGIzBBWKLKCgoiix3yKjzXGBcuFweuaPi6Fy8Doxz0QEVcbkuIwxy3RC9cHGPThQZQRnvEEyHQCAgEEJ2QrZOJ+mu7tq+88c53anTVJPukD691Of1PPV0naVOfX+VSn3q/M6p3zF3R0REZEBivAsQEZGJRcEgIiIRCgYREYlQMIiISISCQUREIhQMIiISEVswmNl3zWyHmT02zHIzs2+Y2TozW2NmS+KqTUREDopzj+H7wLkvsvw8YGF4uwL4XzHUJCIiQ8QWDO5+P7DnRVZZCvzAAyuANjM7Jp7qRERkQGq8C6gyB9hcNb0lnPfc0BXN7AqCvQqamppet2jRolgKFBGZKlatWrXL3dtrLZtIwTBi7n4rcCtAR0eHd3Z2jnNFIiKTi5ltHG7ZRDoraSswr2p6bjhPRERiNJGCYRnwgfDspNOAbnd/QTeSiIiMrdi6kszsDuAsYKaZbQE+DaQB3P0WYDlwPrAO6AUui6s2ERE5KLZgcPdLDrHcgY/EVI6IiAxjInUliYjIBKBgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRMQaDGZ2rpk9aWbrzOzqGstfZmb3mdlqM1tjZufHWZ+IiMQYDGaWBG4CzgMWA5eY2eIhq/134C53PwW4GLg5rvpERCQQ5x7DqcA6d1/v7gXgTmDpkHUcaA3vTwO2xVifiIgQbzDMATZXTW8J51X7DPA+M9sCLAc+WmtDZnaFmXWaWefOnTvHolYRkbo10Q4+XwJ8393nAucDt5vZC2p091vdvcPdO9rb22MvUkRkKoszGLYC86qm54bzql0O3AXg7g8AOWBmLNWJiAgQbzCsBBaa2QIzyxAcXF42ZJ1NwNkAZvZKgmBQX5GISIxiCwZ3LwFXAncDTxCcfbTWzK4zswvD1a4CPmhmjwB3AH/j7h5XjSIiAqk4n8zdlxMcVK6ed23V/ceB0+OsSUREoibawWcRERlnCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJiDUYzOxcM3vSzNaZ2dXDrPPXZva4ma01s3+Jsz4REYFUXE9kZkngJuAdwBZgpZktc/fHq9ZZCFwDnO7uXWY2K676REQkEOcew6nAOndf7+4F4E5g6ZB1Pgjc5O5dAO6+I8b6RESEeINhDrC5anpLOK/aicCJZvZvZrbCzM6ttSEzu8LMOs2sc+fOnWNUrohIfZpoB59TwELgLOAS4J/NrG3oSu5+q7t3uHtHe3t7zCWKiExtcQbDVmBe1fTccF61LcAydy+6+7PAUwRBISIiMYkzGFYCC81sgZllgIuBZUPW+QXB3gJmNpOga2l9jDWKiNS92ILB3UvAlcDdwBPAXe6+1syuM7MLw9XuBnab2ePAfcA/uPvuuGoUEREwdx/vGl6Sjo4O7+zsHO8yREQmFTNb5e4dtZZNtIPPIiIyzhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCJGFAxmljaz7WZ20lgXJCIi42tEweDuRaAITO7xM0RE5JBG05X0P4FrzCy2y4GKiEj8RvMh/2bgTGCrmT0G9FQvdPcLaz5KREQmldEEwy7gp2NViIiITAwjDgZ3v2wsCxERkYlhxMcYzOzeWtdfNrNWM7v3yJYlIiLjZTQHn88CMjXm5wiOP4iIyBRwyK4kM1tSNfkaM9tTNZ0E3glsPdKFiYjI+BjJMYZOgt8vOPC7GsvzwEePZFEiIjJ+RhIMCwAD1gOvJzg7aUAB2OHu5TGoTURExsEhg8HdNwKY2ReA17v7LdXLzew/mdkcd//HMapRRERiNJqDz+8DVteY/xDwgSNTjoiIjLfRBMMsYGeN+buA2UemHBERGW+jCYZN1D4t9S3AliNTjoiIjLfRDInxLeCrZpYBBn7QdjZwA3DjkS5MRETGx2iGxPiKmc0EvsHBH7oVgK+7+xfHojgREYnfqIbQdvdrzOyzwOJw1hPufuDIlyUiIuNl1NdWcPceYOUY1CIiIhOArvksIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiETEGgxmdq6ZPWlm68zs6hdZ7z1m5mbWEWd9IiISYzCYWRK4CTiP4JfTl5jZ4hrrtQAfAx6MqzYRETkozj2GU4F17r7e3QvAncDSGutdTzAoX1+MtYmISCjOYJgDbK6a3hLOG2RmS4B57v6/X2xDZnaFmXWaWefOnbUuESEiIodrwhx8NrME8E/AVYda191vdfcOd+9ob28f++JEROpInMGwFZhXNT03nDegBXgV8Acz2wCcBizTAWgRkXjFGQwrgYVmtiC82M/FwLKBhe7e7e4z3X2+u88HVgAXuntnjDWKiNS92ILB3UvAlcDdwBPAXe6+1syuM7ML46pDRERe3Kivx/BSuPtyYPmQedcOs+5ZcdQkIiJRE+bgs4iITAwKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRNRtMBzoL/HdPz2Lu493KSIiE0qsP3CbSD7768e5c+VmFsxs4q2LZo13OSIiE0bd7jHs7S0CkC+Wx7kSEZGJpW6DwSz4q54kEZEoBQNKBhGRavUbDATJoD0GEZGoug2GMBdERGSI+g2GkHYYRESi6jYYtMMgIlJb3QbDAP3ATUQkqm6DwUz7DCIitdRtMIiISG11GwwD+wvqSRIRiarfYNAP3EREaqrfYAj/ao9BRCSqfoNBB59FRGqq22AQEZHa6j4Y1JUkIhJVt8EweIxhXKsQEZl46jcYwmMMFe0yiIhE1HEwhHeUCyIiEXUbDIkwGLTHICISVbfBkEwMdCWNcyEiIhNM3QaDjjGIiNRWt8GgriQRkdrqOBjCPQb1JYmIRNR9MCgWRESiYg0GMzvXzJ40s3VmdnWN5X9vZo+b2Roz+72ZHTd2tQR/tcMgIhIVWzCYWRK4CTgPWAxcYmaLh6y2Guhw99cAPwG+OFb1qCtJRKS2OPcYTgXWuft6dy8AdwJLq1dw9/vcvTecXAHMHatiDp6uqmAQEakWZzDMATZXTW8J5w3ncuA3tRaY2RVm1mlmnTt37jysYgZ++KwdBhGRqAl58NnM3gd0AF+qtdzdb3X3DnfvaG9vP9znCLalw88iIhGpGJ9rKzCvanpuOC/CzN4OfAo40937x6qYwd8xaJdBRCQizj2GlcBCM1tgZhngYmBZ9QpmdgrwLeBCd98xlsVoSAwRkdpiCwZ3LwFXAncDTwB3uftaM7vOzC4MV/sS0Az82MweNrNlw2zuJdOQGCIitcXZlYS7LweWD5l3bdX9t8dVS0K/YxARqWlCHnyOw+Avn7XHICISUcfBEPxVV5KISFTdBsPBYwzjXIiIyARTt8GQ0MFnEZGa6jgYgr/KBRGRqFjPSppINIieyPC680XyhTL7+4rki2WSCWPu9EYyyQR9xTLPdffRWyiRTSVJJY1sKkG+WGZ/X4lKxSlWnP19RQDaGjI0ZZPsPlDgQH+JllyKaQ1pmnMpmrMp3CGXTjKjKUOpUiGbSo64znyhTH+pTE+hzO4D/fQVK6SSRmMmSb5Q5sFn99DVWyCbSpJNJcgkE+zq6adQqlCuOG0NabLpJK0NaRrTSaY1pJnelAaMtsY00xszpJNGf6lCb38Zx0knE2zpylMqV0inEiQMtnTlqbhjGOlkgoZMYrAd6WSCijvN2RQzmjJkUgnSiQSOUyw7uXSC5myKbd19dPcWMYPmbIpcOom7s6+vSF+xgju0NaZpa0zjQEM6STo5Nt/t6zYYBobdLmuXYUQ27OphZkuW7nyRbCrBzObs4LJlj2zjq/c8RTJhVCrO9KYMJ85u5vXzZ3B0a45XHtPK9KbM4PqlcoVkwgaP80iUu1OqOKmEsS9fYu1z3WztylOuOLNbczTnUuEHYoVdB/pZvamLnv4yAI2ZJE3ZFHt7C+zuKdDWmKEll2LHvn7ASSaMYtnpzhfpLZTJphIkE0ap4hRKFfbli+zuCT5gx0tLNsUxbTmyqSSNmSRmUK44u3sK9PSX2N9XwoDeYnlEe/yZZIJCuRKZTiWDD/B9fcUJ0Wtgdni9F9cvPYn3v3H+Ea+nboNh4JfP3/u3DXzorBOY1ZIb0+dz98P6ICyUKvx89RZmNGV5x+LZ5AtlLvnnFTy8eS/vP+04rjrnRNoaM4fe0Cis3dbNqo1d7O8r8aW7nxx2vdfOa2Pt1m5KQ/a6Grr7WLWxizv+fHDMxHkzGsIPnhL5YvAhlkklmN6Yplh29vYWSCcTNGaSnHliO688ppWjmrPMP6qRr//+aUplpymbZN6MRo5uDf6tpjWkaW1IU644+UKZYqXCpj29tDdnKVccM5jdmuPkeW3Mbs2xc38/FXcO9JeoVKAxm2ROWwP7+ors2l8gl06QSwcfrNu7+wBYMLOJTGp038qK5Qrbu/t4aFMXpbJTrjgPberiwWf3MKMpw9GtOfLFMn3FMjv299NXLFMoVciE32i3dOUplCukwg/sQ2nNpZjZnMWBA/0luvNFmrMpjm3LsXF3L935ItMa0sG36GKZpkyKhkySWS3ZYG/AjHTSaW/O8PJZzcxqydLekqUpk2RaY4ZKJfiWvG1vUFc2lWBGU4bpTRmKpQqlitNfKpNNJWnOpkgljEQi+MZdKjv7+0rkiyXSyQTHTMuxvy+o8UB/iQN9Jcygr1hhx/4+EmbsyxfZ1t1HvlAmXyxjQCJhLJzVTHM2TUsuhVkQgo2ZFA3pJJlUgpnNGbLp4Ft6vlAmYcaS49qY1ZKjXHGK5Qr9xQotuRSJxMEeg0K5wr6+Ir39ZbrzRfb0FAbfJ3t6CpTKQaA251IkzCiWK0xrSDOjKUOxXKGnv8zx7U2kwm0WypXB4AaC94A7fcUyXT0FegtlnOAzIZNK0F8Mnn9WS3bwC9eB/oP/T6Y1pMmmEiTM2NtbpKu3gFnw2XDyvOmjem+OVN0GQ/VH9HW/epxv/sclR/w5Bv6z3712O393+6rIslfNaeXG97yGZY9s46nt+1m/q4fLz1jA+087jr5ihS/85glue2Bj5DHzZjSweU9+cPr2FRv51ZptXPrG+Vxy6ss4etoLw83d+a8/XYM7fOKdr2B2a45i+KGzbscB3vHV+zl2Wo43HH8UP1/9gqGrBk1vTPPGE47i+X39zGlr4Mnt+9m0u4dSxckkE/z8I2/ipGOnDQZgf6lM54Yunt3Vw7a9eVZv2ssD63dzzLQcr2ufzrFtOfb2Ftm+r4/n9/Wx6OhWGjJJShXn/qd38YuHt73EV//ISCWMo6flaM2lMYN8sUyp7DSkk/SXgg/06U0Znt/XR2+hTC6dpDtfpDzkA90MTj9hJqVKhce2dZM0I5dOMnd6A7lUkuZcKvyQKXH6y2fSnEtRqTitDWlOnN3CwlnNJBPG9n19HOgrkUklaMqmaM4mWTCzefCLjtSWTBjJRJJcOtpNlUgYuYH5LeNU3ARkk/0HXh0dHd7Z2Tnqx/3ggQ1c+8u1ALz1Fe1877JTj1hNK9bv5uJbVxyx7QGcvWgWv//LweGjfvbhN9GQTvKZZWt58Nk9ACw6uoWPnb2QJcdN566Vm/nKPU8d9vMtOrqFt5zYzhtPOIq3vmJWzXXKFcfdSR3hfs5KxVm/q4fufJF1O/YzuzXHWWENfcUyz+w8wIymDFu78hRKFfb0FpjZnCWdNF49p40D/SVKlQqpRILNe3pZs7Wbjbt6mNWapSEdfMsse/BN/vl9fTRlUoP9233FCr2FMk3ZJJlkgk17elm/s4eeQolyxWnMpAa7JbKpBNlUgq7eArNbg/Do6S/RlE2xYGYTJx3bSmtDmmTCmNGYiXSniYw3M1vl7h21ltXtHkOiqltnf18psuw3jz7HkuOmM7u1dvfSvr4iH/7hQ1z42mPZ11fk+PYm3rZoNn3FMvf9ZQcf+tFDNR/3+Xe9mjNf0c7sliw/W72VT/5kDemk8e5T5g72o/541ZbB9a9fehLvXjKXpmzwz7R1b57la57j4lPn0ZJLA3DnFafxy4e38cMVG+nc2DXsc9/4nlfz8OZu7vjzpsj88199NN+4+BSeev4Az+7q4YLXHHOIV+6g4Fvqkf+mmkgYL5/VDMDrjovuKufSSU46dhoAx0xrqPn4GamDH8AzmjK8dl7bEa9RZCqr2z2Gu1Zu5pM/XQPAK2a3cPd/eQsAqzd18a6b/x8AG75wQeQx5YqzeU8v//CTR1i5oeuQz3HHB08jXywxqyXHoqNbRvTNulzxw+4WKJQq3PHnTXz7T+vZvCfP1y46maUnH6uDvCLyAtpjqKH6s3JfeFodwLf+uH7w/vu/8yC3XXYqiYTh7lzzszXc1bmFkXj6c+cd1qlkL6WvOJNKcOmb5nPpm+Yf9jZEROo2GKr3k54Lz0CB4Eyb367dDsD/fXoX3/7TevblS9y+YiPd+WJkG59716tob86y+NhWzrjxPt77hpfxmQtPGrNzi0VE4lC3wTDUTfet47ePbefRrd2R+Z9f/pea6y//z29m8bGtg9NDu51ERCYrBUNo6Pn6v7/qTM7+yh9fsN6zN5xPV2+RGTrDRESmKPV5DOOE9mZ++ZHTB6fPWTybRz59DmamUBCRKU17DC/itfPa2PCFCw77V8siIpNR/QbDMGfpfvisE7j8jAWReQoFEakn6koa4l2nzOGoqgHiRETqjYJhCB0/EJF6p2AIDYyM2Jip3941ERGo52MMVf5y/blkUwm680UaMiO/SIiIyFRUt3sMXnX0OWHBRWOO9HUNREQmo7oNhmo66UhE5KC6D4YPnXWCxjYSEalS95+I7z/tuPEuQURkQqnbYJjkl6EQERkzdRsMA3R8QUQkqu6DQUREohQMIiISoWAQEZGIug0GHXsWEamtboNhgKGjzyIi1eo+GEREJCrWYDCzc83sSTNbZ2ZX11ieNbN/DZc/aGbz46xPRERiDAYzSwI3AecBi4FLzGzxkNUuB7rc/eXAV4Ebx6qe3kJ5rDYtIjKpxbnHcCqwzt3Xu3sBuBNYOmSdpcBt4f2fAGfbGF1X89EtewFIJXWMQUSkWpzXY5gDbK6a3gK8Ybh13L1kZt3AUcCu6pXM7ArginDygJk9eZg1zWy/MbrtOjAT1OY6oDbXh5fS5mEHipuUF+px91uBW1/qdsys0907jkBJk4baXB/U5vowVm2OsytpKzCvanpuOK/mOmaWAqYBu2OpTkREgHiDYSWw0MwWmFkGuBhYNmSdZcCl4f3/ANzrrnFQRUTiFFtXUnjM4ErgbiAJfNfd15rZdUCnuy8DvgPcbmbrgD0E4TGWXnJ31CSkNtcHtbk+jEmbTV/IRUSkmn75LCIiEQoGERGJqNtgONTwHJOJmX3XzHaY2WNV82aY2T1m9nT4d3o438zsG2G715jZkqrHXBqu/7SZXVrruSYCM5tnZveZ2eNmttbMPhbOn8ptzpnZn83skbDN/yOcvyAcPmZdOJxMJpw/7PAyZnZNOP9JM3vn+LRo5MwsaWarzezX4fSUbrOZbTCzR83sYTPrDOfF+95297q7ERz8fgY4HsgAjwCLx7uul9CetwBLgMeq5n0RuDq8fzVwY3j/fOA3gAGnAQ+G82cA68O/08P708e7bcO09xhgSXi/BXiKYJiVqdxmA5rD+2ngwbAtdwEXh/NvAT4U3v8wcEt4/2LgX8P7i8P3exZYEP4/SI53+w7R9r8H/gX4dTg9pdsMbABmDpkX63u7XvcYRjI8x6Th7vcTnMVVrXp4kduAf181/wceWAG0mdkxwDuBe9x9j7t3AfcA54599aPn7s+5+0Ph/f3AEwS/mp/KbXZ3PxBOpsObA28jGD4GXtjmWsPLLAXudPd+d38WWEfw/2FCMrO5wAXAt8NpY4q3eRixvrfrNRhqDc8xZ5xqGSuz3f258P52YHZ4f7i2T8rXJOwuOIXgG/SUbnPYpfIwsIPgP/ozwF53L4WrVNcfGV4GGBheZlK1Gfga8EmgEk4fxdRvswO/M7NVFgz/AzG/tyflkBgyOu7uZjblzks2s2bgp8DH3X2fVY23OBXb7O5l4GQzawN+Diwa55LGlJn9FbDD3VeZ2VnjXU+MznD3rWY2C7jHzP5SvTCO93a97jGMZHiOye75cJeS8O+OcP5wbZ9Ur4mZpQlC4Ufu/rNw9pRu8wB33wvcB7yRoOtg4Atedf3DDS8zmdp8OnChmW0g6O59G/B1pnabcfet4d8dBF8ATiXm93a9BsNIhueY7KqHF7kU+GXV/A+EZzOcBnSHu6h3A+eY2fTwjIdzwnkTTthv/B3gCXf/p6pFU7nN7eGeAmbWALyD4NjKfQTDx8AL21xreJllwMXhGTwLgIXAn+Npxei4+zXuPtfd5xP8H73X3d/LFG6zmTWZWcvAfYL35GPE/d4e7yPw43UjOJr/FEE/7afGu56X2JY7gOeAIkFf4uUEfau/B54G/g8wI1zXCC6Y9AzwKNBRtZ2/JTgwtw64bLzb9SLtPYOgH3YN8HB4O3+Kt/k1wOqwzY8B14bzjyf4kFsH/BjIhvNz4fS6cPnxVdv6VPhaPAmcN95tG2H7z+LgWUlTts1h2x4Jb2sHPpvifm9rSAwREYmo164kEREZhoJBREQiFAwiIhKhYBARkQgFg4iIRCgYRA7BzOabmZvZhLrQ/EStSyY/BYPIoW0mGNH1YQAzOyv8QJ4ZVwFm9gcz++aL1SVypGisJJFD8GCMou1jsW0zS7t78XAeO5Z1SX3THoNMKOE345vN7PNmtsuCCxB92cwS4fINZvaJGo/5ZtX0BjO71sy+b2b7zWyzmV1kZm1mdqeZHQgvXnLOCGsa7LIJR3O9L1y0M5z//XA9M7NPmtkzZpa34GIr76uxnUvM7F4zywN/Z2ZHmdkdZrYlfNxaM7us6nHfB84EPhI+3sNtvaAryczeYsFFavrM7Hkz+2o47MuIXt9wnXdbcNGXvJntMbM/mtnAaJ5SBxQMMhG9FygBbwKuBD4OXDTKbXycYFiEJQQXdrmN4GIvy4GTgfuBH5pZbpTb3Qy8J7x/EkFXzsfC6c8SDEfyEYKLw9wAfMvMLhiyjRuAm8N1fkEwlMNDwF+F2/x6+Lizw/U/BjwAfC98vmOIDqkMgJnNIbhoy2qCocgvBy4Jn6/asK+vmR1NMGDdbcArCS4CdfsIXheZSsZ7bBDddKu+AX8AHhgy7x7g2+H9DcAnajzmm1XTG4A7qqabCcZW+kbVvPnhvI4R1BRZl2DcHqfqKltAE5AH3jzksV8Dlg/ZzlUjeM47B9pcq43D1PU5grF0ElXr/A3QDzSO8PVdEm7zuPF+L+g2fjcdY5CJaM2Q6W3ArMPdhrsfMLNegkHGBjwf/h3tdoezmOCb/28tOlZ+miCoqnVWT5hZkuByjRcRXEwlS3DJ2T+MsoZXAivcvVI170/htl7OwdfkxV7fRwgGaXvMzH4X3v+Ju+8cZS0yiSkYZCIaejDWOdjtWSEYUbJaeoTbKA6ZhiPXnTqwnX8HbDpELT1Dpj8BXEXQZfQocAD4PEcutOBge2vVM/j6uns5PPZyGsFQzZcDN5jZme7+yBGsRyYwBYNMNjsJ+tgBCI8RLCLoV49LIfybrJr3OEGXzXHufu8ot3cG8Ct3vx0GrzdxIrB3yHMmazy22hPAX5tZomqv4Yzwsc+MtBh3d4JjGg+Y2XUEwz9fRLA3IXVAwSCTzb3A35rZMoKQ+BTxv483EnzLvsDMfgXk3X2/mX0Z+HL4wX4/wbGN04CKu9/6Itt7CrjIzM4AdgEfBRYQDbsNwKnhWVEHgD01tnMzwYHkm83s6wRj+3+B4NhE70gaFl7s5e0EF3V5nuAg9jyC4JM6obOSZLK5gSAcfgn8jqAPPc69BTy49OKnCQ72Pg8MnCr7j8BnCLqG1hIc1H0P8OwhNvlZgjOofkMQKD3Aj4as82WCb/6PEwTiy4ap6zyCD/OHge8SXMTpv42ied0El9T8NcGB7K8A17v7D0exDZnkdKEeERGJ0B6DiIhEKBik7pnZLeGvoWvdbhnv+kTipq4kqXtmNgtoHWbxPnffEWc9IuNNwSAiIhHqShIRkQgFg4iIRCgYREQkQsEgIiIR/x+9kPk7Rk6NygAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"I6ZyisDnOdAO"},"source":["## Scenario 2\n","\n","In the real world people's preferences change over time. So now in the simulation we are going to incorporate two different cost functions, and swap over to the second one halfway through. Below is a a table of the new reward function we are going to use, `get_cost_1`:\n","\n","### Tom\n","\n","| | `get_cost` | `get_cost_new1` |\n","|:---|:---:|:---:|\n","| **Morning** | Politics | Politics |\n","| **Afternoon** | Music | Sports |\n","\n","### Anna\n","\n","| | `get_cost` | `get_cost_new1`  |\n","|:---|:---:|:---:|\n","| **Morning** | Sports | Sports |\n","| **Afternoon** | Politics | Sports |\n","\n","This reward function is still working with actions that the learner has seen previously.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":204},"id":"5EY5x6FGOdAO","executionInfo":{"status":"ok","timestamp":1634747766985,"user_tz":-330,"elapsed":14,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"434a2190-9dc9-4eae-8e15-c22555ea2ec1"},"source":["def get_cost_new1(context,action):\n","    if context['user'] == \"Tom\":\n","        if context['time_of_day'] == \"morning\" and action == 'politics':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE\n","    elif context['user'] == \"Anna\":\n","        if context['time_of_day'] == \"morning\" and action == 'sports':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'sports':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE\n","        \n","get_preference_matrix(get_cost_new1)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>actions</th>\n","      <th>camping</th>\n","      <th>finance</th>\n","      <th>food</th>\n","      <th>health</th>\n","      <th>music</th>\n","      <th>politics</th>\n","      <th>sports</th>\n","    </tr>\n","    <tr>\n","      <th>users</th>\n","      <th>times_of_day</th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","      <th></th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Anna</th>\n","      <th>afternoon</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","    </tr>\n","    <tr>\n","      <th>morning</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","    </tr>\n","    <tr>\n","      <th rowspan=\"2\" valign=\"top\">Tom</th>\n","      <th>afternoon</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","    </tr>\n","    <tr>\n","      <th>morning</th>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>0.0</td>\n","      <td>-1.0</td>\n","      <td>0.0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["actions             camping  finance  food  health  music  politics  sports\n","users times_of_day                                                         \n","Anna  afternoon         0.0      0.0   0.0     0.0    0.0       0.0    -1.0\n","      morning           0.0      0.0   0.0     0.0    0.0       0.0    -1.0\n","Tom   afternoon         0.0      0.0   0.0     0.0    0.0       0.0    -1.0\n","      morning           0.0      0.0   0.0     0.0    0.0      -1.0     0.0"]},"metadata":{},"execution_count":15}]},{"cell_type":"markdown","metadata":{"id":"sap2wFagOdAP"},"source":["To make it easy to show the effect of the cost function changing we are going to modify the `run_simulation` function. It is a little less readable now, but it supports accepting a list of cost functions and it will operate over each cost function in turn. This is perfect for what we need."]},{"cell_type":"code","metadata":{"id":"nVmE_UkFOdAP"},"source":["def run_simulation_multiple_cost_functions(vw, num_iterations, users, times_of_day, actions, cost_functions, do_learn = True):\n","    cost_sum = 0.\n","    ctr = []\n","\n","    start_counter = 1\n","    end_counter = start_counter + num_iterations\n","    for cost_function in cost_functions:\n","        for i in range(start_counter, end_counter):\n","            # 1. in each simulation choose a user\n","            user = choose_user(users)\n","            # 2. choose time of day for a given user\n","            time_of_day = choose_time_of_day(times_of_day)\n","\n","            # Construct context based on chosen user and time of day\n","            context = {'user': user, 'time_of_day': time_of_day}\n","\n","            # 3. Use the get_action function we defined earlier\n","            action, prob = get_action(vw, context, actions)\n","\n","            # 4. Get cost of the action we chose\n","            cost = cost_function(context, action)\n","            cost_sum += cost\n","\n","            if do_learn:\n","                # 5. Inform VW of what happened so we can learn from it\n","                vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n","                # 6. Learn\n","                vw.learn(vw_format)\n","\n","            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n","            ctr.append(-1*cost_sum/i)\n","        start_counter = end_counter\n","        end_counter = start_counter + num_iterations\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0O--0JpdOdAQ"},"source":["def run_simulation_multiple_cost_functions(vw, num_iterations, users, times_of_day, actions, cost_functions, do_learn = True):\n","    cost_sum = 0.\n","    ctr = []\n","\n","    start_counter = 1\n","    end_counter = start_counter + num_iterations\n","    for cost_function in cost_functions:\n","        for i in range(start_counter, end_counter):\n","            # 1. in each simulation choose a user\n","            user = choose_user(users)\n","            # 2. choose time of day for a given user\n","            time_of_day = choose_time_of_day(times_of_day)\n","\n","            # Construct context based on chosen user and time of day\n","            context = {'user': user, 'time_of_day': time_of_day}\n","\n","            # 3. Use the get_action function we defined earlier\n","            action, prob = get_action(vw, context, actions)\n","\n","            # 4. Get cost of the action we chose\n","            cost = cost_function(context, action)\n","            cost_sum += cost\n","\n","            if do_learn:\n","                # 5. Inform VW of what happened so we can learn from it\n","                vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n","                # 6. Learn\n","                vw.learn(vw_format)\n","\n","            # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n","            ctr.append(-1*cost_sum/i)\n","        start_counter = end_counter\n","        end_counter = start_counter + num_iterations\n","\n","    return ctr"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FqGuu9aiOdAQ"},"source":["### With learning\n","Let us now switch to the second reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the web users but it is still working with the same action space as before. We should see the learner pick up these changes and optimize towards the new preferences.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"F1pAQ0ecOdAR","executionInfo":{"status":"ok","timestamp":1634747775296,"user_tz":-330,"elapsed":2310,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"645f59c4-a4e4-41a2-a3c4-d44bb46795ef"},"source":["# use first reward function initially and then switch to second reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new1]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n","\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxddX3/8dfnbrMnkz0hJCRAAkSgEIcISAXKYgCF/qotUP2pVEut6E9Lq4UianGhWKyKoIAWcavUpdUUESiLRS0gAdlCCGQBMiEhkz2z3+Xz++Ocmdw7uZPMkJlz7sx9Px+P+8jZ7pnPuWdy33O+55zvMXdHRESkTyLuAkREpLIoGEREpISCQURESigYRESkhIJBRERKKBhERKREZMFgZreZ2WYze3aQ+WZmN5jZajN72swWR1WbiIjsEeURw+3A0n3MPwdYEL4uBb4RQU0iIjJAZMHg7g8B2/axyAXAdz3wCNBsZrOiqU5ERPqk4i6gyGxgfdF4azht48AFzexSgqMKGhoa3njkkUdGUqCIyHjx+OOPb3H3aeXmVVIwDJm73wrcCtDS0uLLly+PuSIRkbHFzF4ebF4lXZW0AZhTNH5wOE1ERCJUScGwDHhPeHXSicBOd9+rGUlEREZXZE1JZvZD4DRgqpm1Ap8G0gDufjNwF3AusBroBC6JqjYREdkjsmBw94v3M9+ByyIqR0REBlFJTUkiIlIBFAwiIlJCwSAiIiUUDCIiUkLBICIiJRQMIiJSQsEgIiIlFAwiIlJCwSAiIiUUDCIiUkLBICIiJRQMIiJSQsEgIiIlFAwiIlJCwSAiIiUUDCIiUkLBICIiJRQMIiJSQsEgIiIlFAwiIlJCwSAiIiUUDCIiUkLBICIiJRQMIiJSQsEgIiIlFAwiIlJCwSAiIiUUDCIiUkLBICIiJRQMIiJSQsEgIiIlFAwiIlJCwSAiIiUUDCIiUkLBICIiJSINBjNbamarzGy1mV1RZv5cM3vQzH5vZk+b2blR1iciIpCK6geZWRK4CTgLaAUeM7Nl7v5c0WKfBH7k7t8ws0XAXcC8qGqU6pLNF0glDDPba97u7izd2QLbO3tp3d7Jqk3tJAzSyQQFdzp78yQTRl06ySvbOlm9uZ2GmiT5gtOTK1CXTnLIlHouPGEuh09vjGHrZLwrFJy8O+nkyP99H1kwAEuA1e6+FsDM7gAuAIqDwYEJ4fBE4NUI65NxJpsv0Lq9i11dWVZu3MXTG3ayaWc3rds72daRZUt7D5lUgrp0kmy+wNzJ9dSkErRu72JrR++Qf05dOsm8qQ2s25LFDJrr07zSneO+la/xrd+s491vOoQrzjmShpoo/7vJSOjszbG7O0dHT46CO7XpJED/HwC7u4PpdekkZjCpPkO+4NRlkhhQm07Snc2TzQfL7OzKsnFnF53ZPLm8ky84u7qy5N3Z3Z1le2eWtt095PIFtnVmyRcK9IR/oPTkCnRn8+QKwfs6enJc945j+dOWOSO+3VH+ps4G1heNtwJvGrDMZ4B7zewjQANwZrkVmdmlwKUAc+fOHfFCZWzY0t7Dy1s7WbVpNz25PFvae3hs3XY27OiiszdHd7ZAVzbfv3wmlWB6Uw1Hzmzi+Dk1TJ9QQ2dvnt3dWfIFeHHzbmrSSc5aNIOZE2uZVJ9hQl2KmRPqOGJmEwV3cnknlbT+//AFd6Y01JBM7H3UsXlXNzc9uJrvPPwy96zYxAf+cD5/1jKH5vpMlB/TuFIoOB29ObZ19FJwSBhs3t0TfvkW6OjJ09GTw4GebJ72nhyv7ephS3sPuYKTzRfI5Z3dPTm6e4P3JMJ919GTo70nR8KMnlwed+jJFSLdvnTS+n+fpjZmSCUTZJIJjpjZRG06SW06SSphJMxoqEmO2tGoufuorHivH2T2TmCpu38gHP+/wJvc/cNFy1we1vQlMzsJ+FfgaHcfdO+0tLT48uXLR7l6iVtnb45fv7iFFRt28uyru1i3pYN1WzpKlkkYHD17IjMn1JJKGtMaa1gwo4mJdWmOmNnE4dMa+78EovTwmq1cd/fzPLl+B8mEcdSsJhbPncRxc5pZdNAEFkxvKhssY1mh4Gzr7GVrey+9uQIdvcEXbjppbOvoZVd3lq7e4L91Q02SXV1ZNu/uobM3z4btXWzc1U1nTw4Ivpw7e/PkCwV2dmUpDPMra2JdmqmNGdLJBJlUglTCaKhJUZdOkk4lyOYKJMNpDZkkDtSkEpgZk+ozNNWmaKhJkkwk6O7N4zipRIKadIKGmhSphNHRk8fd2dGVJWlGdy44IsjmC9SmkyQTRk+uQFNtilkTa6nPJMkkg6OMiXVpkgmjqTZFY02qbNPmaDCzx929pdy8KI8YNgDFxzwHh9OKvR9YCuDuD5tZLTAV2BxJhRIrd2fdlg5at3fx6o4uzKCjJ88Tr2znnhWbyOaDb4RDpzZw2LRGLjxhDgumNzJ3cj1NtWkm1qWpyyRj3oq9nXTYFP7zQyfz+MvbuW/lZpa/tI0fLV/Pdx9+GYCFMxr52JkLWfqGmbEEV7Gu3jzppNHek2PDji56cgXcoTdXwHF6ssG/m3b2sLMrS08uz/ptXWzc2UVHT3CUtq2zl+0dveSG+Q2eMKhJJTmouZaDmuuY0VQDQF0mSX0m+HJtrguO4iaFR10OzJgQfNGmkwnqw2UTZtSkgi/uvuYfGboog+ExYIGZzScIhIuAPx+wzCvAGcDtZnYUUAu0RVijjKD12zpZ3dbOL57eSNKMzbu7yTu07e7h1R1ddGXzFAqOAw2ZJNm8lzT99JlQm+KiE+ZyztEzOXZOM41jsK3ezGiZN5mWeZMByOULvLi5nYfXbOW7D7/Eh37wBMmEsXBGExccdxB/fNxsZk6sHbGf39cE09WbZ1tnL6s3t7OzK8uOzixr2zp4eWsHG3Z0sXFn97DXPWNCDQc119FcnyGTSnDcnGamNGaY1lTD1MYaMqngC7vgwXZPaaxhYl2adNJIJozd3Tma69JMbgiaTiR+kTUlAYSXn34FSAK3ufvnzewaYLm7LwuvRPom0Ejwx8An3P3efa1TTUnx6+zNsbW9l1d3dPHytk5u+Z81rGkrbeapTSeYWJcmlUgwc2IttekENakkE+vS9OYK9OQK1KQTHD+nmUWzJjCtqYb2nhx1mSTzpjSM67/6cvkCd6/YxJOv7OB/12zluY27yCQTLDpoAsfMnsiS+ZM5ZEo9kxsyNNdn2LK7h60dPbRu72JL2FSTzRdY9dpudnZmyRec1h2dbO/I4u5k8062EPzlX86MCTUcMqWB2c11HDq1gd58gYl1aQ5qriOdDJpe+ppWatIJDJjWVMOk+gw1qYS+zMeofTUlRRoMo0HBEK1Cwfn5Uxv42gOryeWDKyPKXcEzZ3Idh01rZPHcSfyf42czZ3J9DNWOTWvb2rntt+t4cv0OXtjUTm9+aCdAZzfXMbkhQzppzJ5Uz5SGDGaQCdvWm2pTJBMJpjXVcNi0BqY01DChLkV9ZuwdgcmBq5RzDDIG7ezKsratnQdXtfFvj77ClvaevZY5evYE0skE86c2cNKhU3jbsQdVZFv/WHHotEY+98fHAMHRxBOv7Ajb8PPs7MoytTHDlMYMcyfXM6k+QzoV/BXfVJuOt3AZNxQMUsLdeX7TblZt2s2tD63luY279lrmg6cexvtOntd/OZ2MnlQywZL5k+MuQ6qMgqGKfe/hl8CMM4+azqs7urjsB79n067Sk4/zptTjwDGzJ/LXpx3GolkTIrucTkTioWCoMu7OV+9/ka/c92L/tKt/VrrM3Mn1HD17Ah89YyFHzGyKuEIRiZuCoYp841druO7u5/vHF89t5uBJ9dz97CbOOGo6l7x5PifMm6QjApEqp2CoAk+t38EFN/22f/z/nbGAy04/jJqUThCLyN4UDOPYyo27OOerv+4fP+/YWXzxHceqMzcR2Sd9Q4xDNz7wItff+0L/+NGzJ3D7JUuY2lgTY1UiMlYoGMYRd+cvv7uc+1bu6VrqPz90MsfPnRRjVSIy1igYxokfL1/Px3/yNAAnzJvEty9ZMib7FBKR+OmbY4zb1Z3l2M/s6U7qA6fM56rzjtKVRSLyuikYxqjNu7o55boH+/vRyaQS/ObvT2d608j1yCki1UnBMAY99EIb77ntd/3jl51+GB9/65ExViQi44mCYQzp7M3xtht+w9rwyWV/9ZZDufzshbofQURGlIJhDLjwlod5dN22kmn3Xf4WDp+u7ipEZOQpGCrYl+5dxdceWF0ybfHcZn761yfr5LKIjBoFQwUqFJxjPnMPHb3BYy4TBo9/8iwmNWRirkxEqoGCocLs6OzluGv+u3/8/r89lcOmNcZYkYhUGwVDBenO5vtDIZNMsOpzS9VkJCKR0+O3KsSmnd0cefXdAPzpGw/mhc+fo1AQkVgoGCqAu3PeDUEvqPOm1PPFdx4bc0UiUs3UlBSznZ1Z/uCaoEuL846ZxU3vWhxzRSJS7XTEEKNsvtAfCgBfvvC4GKsREQkoGGKybksHC676Zf/4U58+m0xKu0NE4qempBhs3t3N6df/qn987RfOJZHQiWYRqQz6EzViHT05lnz+fgBOXTiNZ//xrQoFEakoOmKIkLvzhk/fAwR3M3/nL5bEXJGIyN50xBChz/9iZf/wmi+cG2MlIiKDUzBEpKMnx7d+sw6AF3XzmohUMAVDRM76l/8B4ONvPYJ0Uh+7iFQufUNF4NG1W3l1ZzcAl51+eMzViIjsm4JhlOULzoW3PgLA7ZecEHM1IiL7p2AYRQ+90MZh/3AXAIdMqee0I6bHXJGIyP4pGEbJ9x95mffc9rv+8R984E0xViMiMnSRBoOZLTWzVWa22syuGGSZPzOz58xshZn9W5T1jZSeXJ5P/uzZ/vHHrjqTgyfVx1iRiMjQRXaDm5klgZuAs4BW4DEzW+buzxUtswC4Enizu283szHZ9nLH79YD8Adzmvn5ZW+OuRoRkeGJ8ohhCbDa3de6ey9wB3DBgGX+ErjJ3bcDuPvmCOsbMZ9etgKAn37wpJgrEREZviiDYTawvmi8NZxWbCGw0Mx+a2aPmNnScisys0vNbLmZLW9raxulcl+fE79wf/9wSvcriMgYVGnfXClgAXAacDHwTTNrHriQu9/q7i3u3jJt2rSIS9y3TbuC+xV+/YnTY65EROT1iTIYNgBzisYPDqcVawWWuXvW3dcBLxAExZjwytZOAD7z9kXMmayTzSIyNkUZDI8BC8xsvpllgIuAZQOW+RnB0QJmNpWgaWlthDUekIu/GdzI9ubDp8ZciYjI6xdZMLh7DvgwcA+wEviRu68ws2vM7PxwsXuArWb2HPAg8HF33xpVjQeiUHA27OgC4PDpjTFXIyLy+kX6PAZ3vwu4a8C0TxUNO3B5+BpTfvC7VwC4eMlc9ZwqImNapZ18HrOuDm9o+/ulR8RciYjIgVEwjIDeXKF/uLk+E2MlIiIHTsEwAr507yoA/urUQ2OuRETkwCkYDlBPLs8tDwUXTl1+1sKYqxEROXAKhgN00wOr+4drUskYKxERGRkKhgN0QxgMj3/yzJgrEREZGQqGA7B+W2f/8JTGmhgrEREZOQqGA/BPv3wegJv+fHHMlYiIjBwFwwH4xTMbATj3mJkxVyIiMnKGFAxmljazTWb2htEuaKzI5ffcu6A7nUVkPBlSMLh7FsgCPrrljB0Prw26cHrnGw+OuRIRkZE1nKakrwFXmlmk/StVqm/+eh0AF54wZz9LioiMLcP5kv9D4FRgg5k9C3QUz3T388u+a5ya1lhDOmmcMG9y3KWIiIyo4QTDFuCno1XIWPLqji5++kQrqYTOLYjI+DPkYHD3S0azkLEgX3A+e+dz3P6/LwGQK+iUi4iMP0MOBjN7APgTd98xYPoE4Gfu/kcjXVwl+NK9q1g4o4mTD5vCGz93X8m8Oz9ySkxViYiMnuE0JZ0GlOtTupbg/MO4ky84XyvqC6nYrz9xup7rLCLj0n6DwcyKb+s91sy2FY0ngbcCG0a6sErwX0+9ute0unSSr118vEJBRMatoRwxLCe4f8GBe8vM7wI+MpJFVYob7n+xZHxqYw3L1VmeiIxzQwmG+YABa4ETCK5O6tMLbHb3/CjUFru1W/ZckXvzuxez9OhZMVYjIhKN/QaDu78MYGb/BJzg7jcXzzezD5rZbHe/epRqjEXx4zrv/Zu3sHBGU4zViIhEZzh3Pr8b+H2Z6U8A7xmZcirH4y9v7x9WKIhINRlOMEwH2spM3wLMGJlyKsfF33wEgE+ed1TMlYiIRGs4wfAK5S9LfQvQOjLlVJ5zj9F5BRGpLsO5j+EW4MtmlgEeCKedAVwLXDfShcVtQm2KXd05Dmqui7sUEZFIDadLjC+Z2VTgBvbc6NYLfNXdvzgaxcVpV3cu7hJERGIxrC603f1KM/scsCictNLd20e+rHit2rQ77hJERGIz7GcruHsH8Ngo1FIxenLBbRnX/skxMVciIhI9PfO5jK0dvQAsnNEYcyUiItFTMJTR15Q0vak25kpERKKnx3QO8K5vPcJvVwfPc57WVBNzNSIi0VMwhE657gGmNGR4qnVn/7TadDLGikRE4qFgAAoFp3V7F63bu+IuRUQkdjrHAGzYsXcg/PiDJ8VQiYhI/BQMBE9qK/a+k+dxwrzJMVUjIhKvSIPBzJaa2SozW21mV+xjuXeYmZtZSxR17R5wl/Pt//tSFD9WRKQiRRYMZpYEbgLOIbhz+mIzW1RmuSbgo8CjUdS1raOXt9/4m5Jph0zRYztFpHpFecSwBFjt7mvdvRe4A7igzHKfJeiUrzuKorZ19Ow1TecXRKSaRRkMs4H1ReOt4bR+ZrYYmOPuv9jXiszsUjNbbmbL29rKPSJiaLqzeV7btScYpjfV8MxnztaNbSJS1SrmclUzSwD/Arxvf8u6+63ArQAtLS2+n8UHdeTVd5eM3/jni2mqTb/e1YmIjAtRHjFsAOYUjR8cTuvTBBwN/MrMXgJOBJZFdQIaIJPSRVoiIlF+Ez4GLDCz+eHDfi4ClvXNdPed7j7V3ee5+zzgEeB8d18eVYGphEX1o0REKlZkweDuOeDDwD3ASuBH7r7CzK4xs/OjqmNfBt7PICJSjSI9x+DudwF3DZj2qUGWPS2Kmoo11KhvJBERNaqHPv32RRw+vSnuMkREYqdgCJ26cFrcJYiIVAQFQ0hXJImIBPRtGFIwiIgE9G0Y0kN5REQCCobQBN3xLCICKBhERGQABQNQm9bHICLSR9+IwNVv2+uxECIiVUvBACRNfSSJiPRRMADtPbn9LyQiUiUUDEBPrhB3CSIiFUPBANy/8rW4SxARqRgKBiCv3rZFRPopGIB8QU1JIiJ9FAxATocMIiL9FAzAkvmT4y5BRKRiKBiAhO5jEBHpV7XB8MtnNvYPL5o1IcZKREQqS9UGwxOvbO8f/tOWg2OsRESkslRtMFjYfJRJJfqHRUSkmoMh7gJERCpU1QaDkkFEpLyqDQZTMoiIlFW1wSAiIuUpGHTTs4hICQWDiIiUUDCIiEiJqg0G3bogIlJe1QaDiIiUp2AQEZESVR8MrsuSRERKVG0w6BSDiEh5VRsMfVwHDCIiJSINBjNbamarzGy1mV1RZv7lZvacmT1tZveb2SFR1iciIhEGg5klgZuAc4BFwMVmtmjAYr8HWtz9WOAnwBejqk9ERAJRHjEsAVa7+1p37wXuAC4oXsDdH3T3znD0EWDUnqCj+xhERMqLMhhmA+uLxlvDaYN5P/DLcjPM7FIzW25my9va2kawRBERqciTz2b2bqAF+Ody8939VndvcfeWadOmRVuciMg4l4rwZ20A5hSNHxxOK2FmZwJXAae6e89oF6WLkkRESkV5xPAYsMDM5ptZBrgIWFa8gJkdD9wCnO/umyOsTUREQpEFg7vngA8D9wArgR+5+wozu8bMzg8X+2egEfixmT1pZssGWZ2IiIySKJuScPe7gLsGTPtU0fCZUdWiR3uKiJRXkSefRUQkPgoGEREpUfXB4OosSUSkRNUHg4iIlKraYJg7pR6Ay89aGHMlIiKVpWqDoT6TBODsN8yMuRIRkcpStcHQRxetioiUqtpg0DlnEZHyqjYY+qj7bRGRUlUbDDpgEBEpr2qDYQ8dMoiIFKvaYNCNbSIi5VVtMPTROQYRkVJVHwwiIlKq6oNBBwwiIqWqNhh0ikFEpLyqDYY+ppMMIiIlqjYYXHcyiIiUVbXB0EfHCyIipao2GHSOQUSkvKoNhj46xSAiUqpqg0FHDCIi5VVtMPQxnWUQESlRtcGgAwYRkfKqNhj66ByDiEipqg0G9a4qIlJe1QaDiIiUV7XBoOMFEZHyqjYY+ugcg4hIqeoNBh0yiIiUVb3BEFLvqiIipao2GNS7qohIeVUbDH10vCAiUqpqg0G3MYiIlFe1wdBHpxhEREpFGgxmttTMVpnZajO7osz8GjP793D+o2Y2b7Rq0QGDiEh5kQWDmSWBm4BzgEXAxWa2aMBi7we2u/vhwJeB60arns7efFCXzjKIiJSI8ohhCbDa3de6ey9wB3DBgGUuAL4TDv8EOMNG6XrSZ1p3AJBKKhhERIqlIvxZs4H1ReOtwJsGW8bdc2a2E5gCbCleyMwuBS4NR9vNbNXrrGnqtOtK110FpoK2uQpom6vDgWzzIYPNiDIYRoy73wrceqDrMbPl7t4yAiWNGdrm6qBtrg6jtc1RNiVtAOYUjR8cTiu7jJmlgInA1kiqExERINpgeAxYYGbzzSwDXAQsG7DMMuC94fA7gQdcD04QEYlUZE1J4TmDDwP3AEngNndfYWbXAMvdfRnwr8D3zGw1sI0gPEbTATdHjUHa5uqgba4Oo7LNpj/IRUSkWNXf+SwiIqUUDCIiUqJqg2F/3XOMFWY2x8weNLPnzGyFmX00nD7ZzP7bzF4M/50UTjczuyHc7qfNbHHRut4bLv+imb13sJ9ZKcwsaWa/N7M7w/H5YVcqq8OuVTLh9EG7WjGzK8Ppq8zsrfFsydCYWbOZ/cTMnjezlWZ20njfz2b2N+Hv9bNm9kMzqx1v+9nMbjOzzWb2bNG0EduvZvZGM3smfM8NQ7pp2N2r7kVw8nsNcCiQAZ4CFsVd1+vcllnA4nC4CXiBoMuRLwJXhNOvAK4Lh88FfknQ4/iJwKPh9MnA2vDfSeHwpLi3bz/bfjnwb8Cd4fiPgIvC4ZuBvw6HPwTcHA5fBPx7OLwo3Pc1wPzwdyIZ93btY3u/A3wgHM4AzeN5PxPc8LoOqCvav+8bb/sZeAuwGHi2aNqI7Vfgd+GyFr73nP3WFPeHEtOOOAm4p2j8SuDKuOsaoW37OXAWsAqYFU6bBawKh28BLi5aflU4/2LglqLpJctV2ovgPpj7gT8C7gx/6bcAqYH7mOBKuJPC4VS4nA3c78XLVdqL4J6edYQXjAzcf+NxP7OnJ4TJ4X67E3jreNzPwLwBwTAi+zWc93zR9JLlBntVa1NSue45ZsdUy4gJD52PBx4FZrj7xnDWJmBGODzYto+1z+QrwCeAQjg+Bdjh7rlwvLj+kq5WgL6uVsbSNs8H2oBvh81n3zKzBsbxfnb3DcD1wCvARoL99jjjez/3Gan9OjscHjh9n6o1GMYdM2sEfgp8zN13Fc/z4E+FcXNdspm9Ddjs7o/HXUuEUgTNDd9w9+OBDoImhn7jcD9PIuhYcz5wENAALI21qBjEsV+rNRiG0j3HmGFmaYJQ+IG7/0c4+TUzmxXOnwVsDqcPtu1j6TN5M3C+mb1E0EvvHwFfBZot6EoFSusfrKuVsbTNrUCruz8ajv+EICjG834+E1jn7m3ungX+g2Dfj+f93Gek9uuGcHjg9H2q1mAYSvccY0J4hcG/Aivd/V+KZhV3L/JegnMPfdPfE17dcCKwMzxkvQc428wmhX+pnR1OqzjufqW7H+zu8wj23QPu/i7gQYKuVGDvbS7X1coy4KLwapb5wAKCE3UVx903AevN7Ihw0hnAc4zj/UzQhHSimdWHv+d92zxu93OREdmv4bxdZnZi+Bm+p2hdg4v7pEuMJ3vOJbiCZw1wVdz1HMB2nEJwmPk08GT4OpegbfV+4EXgPmByuLwRPDBpDfAM0FK0rr8AVoevS+LetiFu/2nsuSrpUIL/8KuBHwM14fTacHx1OP/QovdfFX4WqxjC1Roxb+txwPJwX/+M4OqTcb2fgX8EngeeBb5HcGXRuNrPwA8JzqFkCY4M3z+S+xVoCT+/NcCNDLiAodxLXWKIiEiJam1KEhGRQSgYRESkhIJBRERKKBhERKSEgkFEREooGET2w8zmmZmbWUU9aL5S65KxT8Egsn/rCTojexLAzE4Lv5CnRlWAmf3KzG7cV10iIyWyZz6LjFXunifoyGzEmVnag+4ehm0065LqpiMGqSjhX8ZfN7MvmNmW8AEm15tZIpz/kpn9XZn33Fg0/pKZfcrMbjez3Wa23swutOBBN3eYWXv4MJOzh1hTf5NN2IPtg+GstnD67eFyZmafMLM1ZtYVPhzl3WXWc7GZPWBmXcBfmdkUCx5C0xq+b4WZXVL0vtuBU4HLwvd7uK69mpLM7C0WPKSm28xeM7Mvh92+DOnzDZf5EwseAtNlZtvM7H/MrK93T6kCCgapRO8CcsDJwIeBjwEXDnMdHyPoFmExwYNdvkPwUJ+7CLqWeAj4vpnVDnO964F3hMNvIGjK+Wg4/jmC7gwuI3g4zLXALWZ23oB1XAt8PVzmZwRdOTwBvC1c51fD950RLv9R4GHg2+HPm0VpF8sAmNlsggex/J6g+/X3E/S/f+2ARQf9fM1sJkHHhN8BjiJ4iMz3hvC5yHgSdz8heulV/AJ+BTw8YNp/A98Kh18C/q7Me24sGn8J+GHReCNBf1I3FE2bF05rGUJNJcsS9M/kwNSiZRqALuAPB7z3K8BdA9bzt0P4mXf0bXO5bRykrs8T9K2TKFrmfUAPUD/Ez3dxuM5D4v5d0Cu+l84xSCV6esD4q8D017sOd283s06CTsf6vBb+O9z1DmYRwV/+d5tZcQdkaYKgKra8eMTMkgTPVriQ4CEqNQSP7vzVMGs4CnjE3QtF034Trutw9nwm+/p8nyLotHf2dFMAAAH1SURBVO1ZM7s3HP6Ju7cNsxYZwxQMUokGnox19jR7Fgh6mCyWHuI6sgPGYeSaU/vW83aC7qL3VUvHgPG/A/6WoMnoGaAd+AIjF1pQ+qCXQT9fd8+H515OJOi6+f3AtWZ2qrs/NYL1SAVTMMhY00bQxg5AeI7gSIJ29aj0hv8mi6Y9R9Bkc4i7PzDM9Z0C/Je7fw/6n7GxENgx4Gcmy7y32Ergz8wsUXTUcEr43jVDLcbdneCcxsNmdg2wguBoRsFQJRQMMtY8APyFmS0jCImriP73+GWCv7LPM7P/ArrcfbeZXQ9cH36xP0RwbuNEoODut+5jfS8AF5rZKQQPsP8IweMsi8PuJWBJeFVUO7CtzHq+TnAi+etm9lWC5xb8E8G5ic6hbFj48JczCR788hrBSew5BMEnVUJXJclYcy1BOPwcuJegDT3KowU8eEj9pwlO9r5G8PATgKuBzxA0Da0gOKn7DmDdflb5OYIrqH5JECgdwA8GLHM9wV/+zxEE4txB6jqH4Mv8SeA2gofA/MMwNm8nweMz7yQ4kf0l4LPu/v1hrEPGOD2oR0RESuiIQURESigYpOqZ2c3h3dDlXjfHXZ9I1NSUJFXPzKYDEwaZvcvdN0dZj0jcFAwiIlJCTUkiIlJCwSAiIiUUDCIiUkLBICIiJf4/I02ftZ9uopgAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"rv_kVQU_OdAR"},"source":["**Note:** The initial spike in CTR depends on the rewards received for the first few examples. When you run on your own, you may see something different initially because our simulator is designed to have randomness.\n","\n","### Without learning"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"KPXXayj8OdAR","executionInfo":{"status":"ok","timestamp":1634747776303,"user_tz":-330,"elapsed":1016,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"256a29bd-17bd-4e94-ad02-aa79b525f293"},"source":["# Do not learn\n","# use first reward function initially and then switch to second reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new1]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, do_learn=False)\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAc/UlEQVR4nO3de5hkdX3n8fenrn2bnjs4zHAZZYxOoivYQVxFMRAFTOBZ3Y0QfbyEyGZXfDTR9YGYqEEjXki8rCCyajQkgaDx0ZGgmAisyUaRRhCZgZHhOjMCc7/2pW7f/eOcGeo03TPT0F3V3fV5PU89XeecX536njrV9alzqd9RRGBmZnZArt0FmJnZzOJgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy2hZMEj6qqQtku6dYLokfV7SBkn3SDq5VbWZmdlTWrnF8DXgrENMPxtYld4uAr7YgprMzGyMlgVDRPwI2HGIJucBfxuJnwALJC1rTXVmZnZAod0FNFkObGwa3pSOe3xsQ0kXkWxV0Nvb+9IXvOAFLSnQzGyuuPPOO7dFxNLxps2kYDhiEXENcA3AwMBADA4OtrkiM7PZRdKjE02bSWclbQaObRpekY4zM7MWmknBsAZ4a3p20qnA7oh42m4kMzObXi3blSTpOuB0YImkTcCHgSJARFwN3AScA2wAhoB3tKo2MzN7SsuCISIuOMz0AN7VonLMzGwCM2lXkpmZzQAOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWUZLg0HSWZLWS9og6ZJxph8n6VZJd0m6R9I5razPzMxaGAyS8sCVwNnAauACSavHNPsz4IaIOAk4H7iqVfWZmVmilVsMpwAbIuKhiKgA1wPnjWkTQH96fz7wqxbWZ2ZmtDYYlgMbm4Y3peOafQR4i6RNwE3Au8ebkaSLJA1KGty6det01Gpm1rFm2sHnC4CvRcQK4BzgWklPqzEiromIgYgYWLp0acuLNDOby1oZDJuBY5uGV6Tjml0I3AAQET8GuoAlLanOzMyA1gbDHcAqSSsllUgOLq8Z0+Yx4AwASS8kCQbvKzIza6GWBUNE1ICLgZuB+0jOPlor6TJJ56bN3ge8U9LPgeuAt0dEtKpGMzODQiufLCJuIjmo3DzuQ0331wGvaGVNZmaWNdMOPpuZWZs5GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCyjpcEg6SxJ6yVtkHTJBG1+T9I6SWsl/UMr6zMzMyi06okk5YErgd8GNgF3SFoTEeua2qwCLgVeERE7JR3VqvrMzCzRyi2GU4ANEfFQRFSA64HzxrR5J3BlROwEiIgtLazPzMxobTAsBzY2DW9KxzV7PvB8Sf9P0k8knTXejCRdJGlQ0uDWrVunqVwzs8400w4+F4BVwOnABcD/kbRgbKOIuCYiBiJiYOnSpS0u0cxsbmtlMGwGjm0aXpGOa7YJWBMR1Yh4GPglSVCYmVmLtDIY7gBWSVopqQScD6wZ0+bbJFsLSFpCsmvpoRbWaGbW8VoWDBFRAy4GbgbuA26IiLWSLpN0btrsZmC7pHXArcD/iojtrarRzMxAEdHuGp6VgYGBGBwcbHcZZmaziqQ7I2JgvGkz7eCzmZm1mYPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWcYRBYOkoqQnJP36dBdkZmbtdUTBEBFVoArM7v4zzMzssCazK+l/A5dKatnlQM3MrPUm8yF/GvBqYLOke4H9zRMj4txxH2VmZrPKZIJhG/BP01WImZnNDEccDBHxjuksxMzMZoYjPsYg6Zbxrr8sqV/SLVNblpmZtctkDj6fDpTGGd9FcvzBzMzmgMPuSpJ0ctPgiyXtaBrOA68DNk91YWZm1h5HcoxhkOT3CwH8YJzpw8C7p7IoMzNrnyMJhpWAgIeA3yQ5O+mACrAlIurTUJuZmbXBYYMhIh4FkPQJ4Dcj4urm6ZL+SNLyiPjzaarRzMxaaDIHn98C3DXO+J8Bb52acszMrN0mEwxHAVvHGb8NOHpqyjEzs3abTDA8xvinpb4K2DQ15ZiZWbtNpkuMLwGfkVQCDvyg7QzgcuCTU12YmZm1x2S6xPgrSUuAz/PUD90qwOci4lPTUZyZmbXepLrQjohLJX0MWJ2Oui8i9k19WWZm1i6TvrZCROwH7piGWszMbAbwNZ/NzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzs4yWBoOksyStl7RB0iWHaPdGSSFpoJX1mZlZC4NBUh64Ejib5JfTF0haPU67ecB7gNtbVZuZmT2llVsMpwAbIuKhiKgA1wPnjdPuoySd8o20sDYzM0u1MhiWAxubhjel4w6SdDJwbET886FmJOkiSYOSBrduHe8SEWZm9kzNmIPPknLAXwPvO1zbiLgmIgYiYmDp0qXTX5yZWQdpZTBsBo5tGl6RjjtgHvAbwG2SHgFOBdb4ALSZWWu1MhjuAFZJWple7Od8YM2BiRGxOyKWRMQJEXEC8BPg3IgYbGGNZmYdr2XBEBE14GLgZuA+4IaIWCvpMknntqoOMzM7tElfj+HZiIibgJvGjPvQBG1Pb0VNZmaWNWMOPpuZ2czgYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7OMlgaDpLMkrZe0QdIl40z/E0nrJN0j6YeSjm9lfWZm1sJgkJQHrgTOBlYDF0haPabZXcBARLwY+CbwqVbVZ2ZmiVZuMZwCbIiIhyKiAlwPnNfcICJujYihdPAnwIoW1mdmZrQ2GJYDG5uGN6XjJnIh8L3xJki6SNKgpMGtW7dOYYlmZjYjDz5LegswAHx6vOkRcU1EDETEwNKlS1tbnJnZHFdo4XNtBo5tGl6RjsuQdCbwQeDVETHaotrMzCzVyi2GO4BVklZKKgHnA2uaG0g6CfgScG5EbGlhbWZmlmpZMEREDbgYuBm4D7ghItZKukzSuWmzTwN9wDck3S1pzQSzMzOzadLKXUlExE3ATWPGfajp/pmtrMfMzJ5uRh58NjOz9nEwmJlZhoPBzMwyHAxTKCIO3v/ibQ/yR9feyXCl3saKzMwmr6UHn+eqEy755wmnff9D3+fev3gdEcHmXcPsHanx367+Ma9/8TK+cMFJSHrWz//knhE++6+/5KJXPY/jFvVQrTfoKuaf9XzNrDOp+VvubDQwMBCDg4Mtf97RWp1f+7PvT8u833vmKt5zxqqDobFl7wj1RhABH71xHfds2s1zl/bybw9sO+R8yoUc/d1Fdg9XqdQavOLExXQXC/R3FcjnRD4nVizs5qj+Lp63tI/+rgLLF3bTU8p+X9g7UqWYTzYumwMnItgzUuPW+7fw6PYh+rsLVGoNtu0bZfu+Ckvmleku5hmq1Fi+oJuVS/tYvayfiGDpvPKUhOJsV28Ej+0YYs9w8hovnVdmtFZnuFJHSl7vRb0livkcAvI50Yjkb7XeYO9IjdFand5ygXIhR7mQXT/D1Tr7R+vUGknbWj3YN1pj+75RJHHMgi6Wputp+/4K9UYgkucdqdbZM1IFRCOCh7ftp1xI3gd7hqsgkRP0lgoc3d9FMS+6innmdxfJ50Sl1iCAnGBJX5meUp6RaoMn94ywd6RGqZAjCPaP1mlEUMrnOKo/qaW7lKeUz2XeI7V6g1ojGKrU2TVUYd9oDSFyOdg3UmO4WmfzrmGGK3XyOVHI58gJirkcC3tLzOsqoPQ17y7lKeZzRMBQpYYkduwfpdZI6ugrF1jQU6JSb7BvpEY+J3pKefI5Uczn6Osq0FXIsWekRm8pTy4nyoUchVyOSr1BKZ+jEUGkz1fK5yjkRV6i2mgcXCahdJ0G+0Zq5HJitFonl9PB//lyIUd3Kc9otcHWfaOM1pLlm9dVZGlfmVLhme34kXRnRAyMO83BMLHLvruO4xf38OE1awH4wR+/KnnD5cSrP31bpu3FrzmRvq4C3xjcyKadw1TrDR78+Dlc+PVBbrl/+n+rd9JxC3jx8vncvPZJntgz8qznN69cYO9o7WnjTzyqj0YET+4eYf8Eu8kk6CsVGK7WqTXGf3/1lPIsm9/FvK7iwQ+UYj7HyiW9HN1fJp/L0V3Ms3HnECPVOkfN6+L5R/dx3KIejp7flfwTSQRBI+CY+V2HDJpGI8jlREQwUm0kH0oR5HNipNqgq5hj085h9ldqzOsqUqklHwh7R6o8vH0/o+ljuop5uoo5tu0d5dEdQ0RATqKQF/O7i/SW8uwarlJvBNv2VdgzXCWXS+rq7yowUm2wcecQDzy5l2I+x479lQlfo/GU8skHT3cxz3D16a9/qZCjnH4oVetBpd4YZy6zQ6mQo7+rQAQ00i8h9Um8Vp3gI7+7mre/YuUzeqyDYZJGa3XW/moPb7jqPw7b9hNveBEvOW4BL3hO/4Rt9oxUue72x3jjS1ewpK+cmRaR/PPWG0GtEXzn7l/xrZ9t4q7Hdo07r+MW9fCHp63k1vu3MHDCIt5y6vHM7y4e0XLtGanyHxu2s3pZP3dv2sVvHNPPUKXOV/79YRb2lPjpI9tZ2FPizkd30lMqIMHWvaM8d2kvlVqDlUt6aaQfrL3lAscu7KZcyHPy8Qs47cSlbN41THcpzzELuigX8ozW6jQa0FXM8avdIzy4ZR/rn9jLaK3O47tH2LRzmF3DVRrpso9U62zeOfyMPsy6i3mWzivTiKC/q0ipkGPjjiEqtQaFvNg5VD34LW4yH8SHcmB+wMFvhs0W9BRZ0F2kWk/G76/UKOZzrFjYzcolvYgkTJ5/dB8Le0vUG8Hju0foKuYo5nOUCzn2jdbYNVSlWm8gxP5Kja5Cjv2VZCuhv6tAT6nA3pGkzZ6RGpVa8vqVCznmdRWY310kl0ueq5AT5WKeJb1lJNi8a5jt+yoMVWos7itRyCXfPoerdQQsmVemkX5zPX5xD1ISrgt6SgRBPf0G/8TuESq1BpV6g91DVeoR5AT5XI5GI9i+v8JItU65mGNJX5n+riK19JtzX7lATmK01mDL3hGGRutU6g32jFTZO1JDJOHb350sa08pz4KeIn3lIhFBI4K+cpHuUo7nzO9mXleBWj2opu+jSq3BzqEK+0ZqBMnW1nClfvB91lsqEAQLe5Its0qtkb7uFQr5JJzqEQyN1qlHpFtdVYYrdeZ1FZMvQPVG+n8MxbwOrvPkNUiGD2zxFPNKvlCkX1TqjaRdT7lARNBVzNNIv7QIJVuQ1TqldIuyXMjRiGSr7aXHL2TV0fOe0fvXwTBJr/rUrTy2Y+iw7X76p2dwVH/XlD53sz0jVW5bv5XXv2gZ+Vxn7Hap1hts2TtKMSe27htl+YJu5ncX2TVU5f4n9rJxxxDb91foK+cJoFYPJHh0ezJewPb9o9TqyT/60nllRqr1gx+O+ZzoKeap1hs0AmqNYF5XgZFqnYU9JY7qL7N3JPkA7ysX6C3nOWFxL73lAkOVGiPVBpVag/7uAssXdGdq31+ps2+kxoKeIqV87uCWgtlMdKhg8MHncYwXCv/2gddw2qduPTh82qol0xoKAP1dRc79T8dM63PMNMV87uAHbvPru7C3xMuft5iXP29xu0pjUW/pkNP7ygX6yv6XstnP7+ImP/rlVnYOVTLjrr/oVF62chGSuOcjr6WvVGDXcJXess/6MbO5qeOD4eFt+9kzXGXL3lHe+bfZXVL5nBg4fuHBg5r9Xcm+/MN9czQzm806Phhec8VtABw1L3tQ+CtvG+CMFx7dhorMzNrLv3xOjT1baLQ2e0/zMzN7NhwMqXWP7wHg4//lRcwrFzjpuAVtrsjMrD06flfSWL//suP4/Zcd1+4yzMzapqO3GLZMwS+Ezczmmo4Ohjd/+fZ2l2BmNuN0dDA8sGVfZviLbz65TZWYmc0cHXuMYeOYXzc/8onXt6kSM7OZpWO3GLbs9fEFM7PxdGwwHLi+gJmZZXXsp2NzMBy/uKeNlZiZzSwdGwzlpqseHbfIwWBmdkDHBsOBC5IAfOZNL2ljJWZmM0vHBkOzsf0kmZl1so4NhuQy3WZmNlbHBoOZmY2vY4Nhll/q2sxs2nRsMJiZ2fg6NhgObDC898xVba3DzGym6dhgOOCExb3tLsHMbEbp+GAwM7Osjg2G8NFnM7NxdWwwHCC1uwIzs5mlpcEg6SxJ6yVtkHTJONPLkv4xnX67pBOmqxZvL5iZja9lwSApD1wJnA2sBi6QtHpMswuBnRFxIvAZ4JPTVc9wpT5dszYzm9VaucVwCrAhIh6KiApwPXDemDbnAV9P738TOEOanp0937l7MwAlX5fBzCyjlZf2XA5sbBreBLxsojYRUZO0G1gMbGtuJOki4KJ0cJ+k9c+wpiXnfDI77w6wBLzMHcDL3BmezTIfP9GEWXnN54i4Brjm2c5H0mBEDExBSbOGl7kzeJk7w3Qtcyv3o2wGjm0aXpGOG7eNpAIwH9jekurMzAxobTDcAayStFJSCTgfWDOmzRrgben9/wrcEv7BgZlZS7VsV1J6zOBi4GYgD3w1ItZKugwYjIg1wFeAayVtAHaQhMd0eta7o2YhL3Nn8DJ3hmlZZvkLuZmZNfO5mmZmluFgMDOzjI4NhsN1zzFbSDpW0q2S1klaK+k96fhFkv5F0gPp34XpeEn6fLrc90g6uWleb0vbPyDpbRM950whKS/pLkk3psMr065UNqRdq5TS8RN2tSLp0nT8ekmva8+SHBlJCyR9U9L9ku6T9PK5vp4l/XH6vr5X0nWSuubaepb0VUlbJN3bNG7K1qukl0r6RfqYzx/Rj4YjouNuJAe/HwSeC5SAnwOr213XM1yWZcDJ6f15wC9Juhz5FHBJOv4S4JPp/XOA7wECTgVuT8cvAh5K/y5M7y9s9/IdZtn/BPgH4MZ0+Abg/PT+1cD/SO//T+Dq9P75wD+m91en674MrEzfE/l2L9chlvfrwB+m90vAgrm8nkl+8Pow0N20ft8+19Yz8CrgZODepnFTtl6Bn6ZtlT727MPW1O4XpU0r4uXAzU3DlwKXtruuKVq27wC/DawHlqXjlgHr0/tfAi5oar8+nX4B8KWm8Zl2M+1G8juYHwK/BdyYvum3AYWx65jkTLiXp/cLaTuNXe/N7WbajeQ3PQ+TnjAydv3NxfXMUz0hLErX243A6+biegZOGBMMU7Je02n3N43PtJvo1qm7ksbrnmN5m2qZMumm80nA7cDREfF4OukJ4Oj0/kTLPttek88CHwAa6fBiYFdE1NLh5vozXa0AB7pamU3LvBLYCvxNuvvsy5J6mcPrOSI2A1cAjwGPk6y3O5nb6/mAqVqvy9P7Y8cfUqcGw5wjqQ/4J+C9EbGneVokXxXmzHnJkn4H2BIRd7a7lhYqkOxu+GJEnATsJ9nFcNAcXM8LSTrWXAkcA/QCZ7W1qDZox3rt1GA4ku45Zg1JRZJQ+PuI+FY6+klJy9Lpy4At6fiJln02vSavAM6V9AhJL72/BXwOWKCkKxXI1j9RVyuzaZk3AZsi4vZ0+JskQTGX1/OZwMMRsTUiqsC3SNb9XF7PB0zVet2c3h87/pA6NRiOpHuOWSE9w+ArwH0R8ddNk5q7F3kbybGHA+Pfmp7dcCqwO91kvRl4raSF6Te116bjZpyIuDQiVkTECSTr7paIeDNwK0lXKvD0ZR6vq5U1wPnp2SwrgVUkB+pmnIh4Atgo6dfSUWcA65jD65lkF9KpknrS9/mBZZ6z67nJlKzXdNoeSaemr+Fbm+Y1sXYfdGnjwZ5zSM7geRD4YLvreRbL8UqSzcx7gLvT2zkk+1Z/CDwA/CuwKG0vkgsmPQj8AhhomtcfABvS2zvavWxHuPyn89RZSc8l+YffAHwDKKfju9LhDen05zY9/oPpa7GeIzhbo83L+hJgMF3X3yY5+2ROr2fgL4D7gXuBa0nOLJpT6xm4juQYSpVky/DCqVyvwED6+j0IfIExJzCMd3OXGGZmltGpu5LMzGwCDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwOwxJJ0gKSTPqQvMztS6b/RwMZoe3kaQzsrsBJJ2efiAvaVUBkm6T9IVD1WU2VVp2zWez2Soi6iQdmU05ScVIunuYtOmsyzqbtxhsRkm/GV8l6eOStqUXMLlCUi6d/oik94/zmC80DT8i6UOSviZpr6SNkt6k5EI310val17M5LVHWNPBXTZpD7a3ppO2puO/lraTpA9IelDScHpxlLeMM58LJN0iaRj475IWK7kIzab0cWslvaPpcV8DXg28K318pPN62q4kSa9ScpGaEUlPSvpM2u3LEb2+aZs3KLkIzLCkHZL+r6QDvXtaB3Aw2Ez0ZqAG/GfgYuC9wJsmOY/3knSLcDLJhV2+TnJRn5tIupb4EfB3kromOd+NwBvT+79OsivnPenwx0i6M3gXycVhLge+JOn1Y+ZxOXBV2ubbJF05/Az4nXSen0sfd0ba/j3Aj4G/SZ9vGdkulgGQtJzkQix3kXS/fiFJ//uXj2k64esr6TkkHRN+HXghyUVkrj2C18Xmknb3E+Kbb8034Dbgx2PG/Qvw5fT+I8D7x3nMF5qGHwGuaxruI+lP6vNN405Ixw0cQU2ZtiT9MwWwpKlNLzAMnDbmsZ8Fbhozn/cdwXNef2CZx1vGCer6S5K+dXJNbd4OjAI9R/j6npzO8/h2vxd8a9/NxxhsJrpnzPCvgKOe6TwiYp+kIZJOxw54Mv072flOZDXJN//vS2rugKxIElTNBpsHJOVJrq3wJpKLqJRJLt152yRreCHwk4hoNI3793ReJ/LUa3Ko1/fnJJ223SvpB+n9b0bE1knWYrOYg8FmorEHY4Ondns2SHqYbFY8wnlUxwzD1O1OPTCf3yXpLvpQtewfM/x+4H0ku4x+AewDPs7UhRZkL/Qy4esbEfX02MupJF03XwhcLunVEfHzKazHZjAHg802W0n2sQOQHiN4Acl+9VappH/zTePWkeyyOT4ibpnk/F4JfDciroWD19h4PrBrzHPmx3lss/uA35OUa9pqeGX62AePtJiICJJjGj+WdBmwlmRrxsHQIRwMNtvcAvyBpDUkIfFBWv8+fpTkW/brJX0XGI6IvZKuAK5IP9h/RHJs41SgERHXHGJ+vwTeJOmVJBewfzfJ5Sybw+4R4JT0rKh9wI5x5nMVyYHkqyR9juS6BZ8gOTYxdCQLll785UySC788SXIQ+1iS4LMO4bOSbLa5nCQcvgP8gGQfeiu3FojkIvUfJjnY+yTJxU8A/hz4CMmuobUkB3XfCDx8mFl+jOQMqu+RBMp+4O/HtLmC5Jv/OpJAPG6Cus4m+TC/G/gqyUVg/nQSi7eb5PKZN5IcyP4r4KMR8XeTmIfNcr5Qj5mZZXiLwczMMhwM1vEkXZ3+Gnq829Xtrs+s1bwryTqepKOA/gkm74mILa2sx6zdHAxmZpbhXUlmZpbhYDAzswwHg5mZZTgYzMws4/8Dd9J4pRXiIHQAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"O8zluZecOdAS"},"source":["## Scenario 3\n","In this scenario we are going to start rewarding actions that have never seen a reward previously when we change the cost function.\n","\n","### Tom\n","\n","| | `get_cost` | `get_cost_new2` |\n","|:---|:---:|:---:|\n","| **Morning** | Politics |  Politics|\n","| **Afternoon** | Music |   Food |\n","\n","### Anna\n","\n","| | `get_cost` | `get_cost_new2` |\n","|:---|:---:|:---:|\n","| **Morning** | Sports | Food|\n","| **Afternoon** | Politics |  Food |\n"]},{"cell_type":"code","metadata":{"id":"AzmkXtSOOdAS"},"source":["def get_cost_new2(context,action):\n","    if context['user'] == \"Tom\":\n","        if context['time_of_day'] == \"morning\" and action == 'politics':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE\n","    elif context['user'] == \"Anna\":\n","        if context['time_of_day'] == \"morning\" and action == 'food':\n","            return USER_LIKED_ARTICLE\n","        elif context['time_of_day'] == \"afternoon\" and action == 'food':\n","            return USER_LIKED_ARTICLE\n","        else:\n","            return USER_DISLIKED_ARTICLE"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hyqhA3WFOdAS"},"source":["\n","### With learning\n","Let us now switch to the third reward function after a few samples (running the first reward function). Recall that this reward function changes the preferences of the users and is working with a **different** action space than before. We should see the learner pick up these changes and optimize towards the new preferences\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"VIh8qkMcOdAT","executionInfo":{"status":"ok","timestamp":1634747780582,"user_tz":-330,"elapsed":1899,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"9666899d-9a7c-4cb5-aad3-f0c2de5c8918"},"source":["# use first reward function initially and then switch to third reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new2]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions)\n","\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxddX3/8dfnbnMns2SbSSALJEBYgqJCQBZFFJWgFazaAtWHivijv1Z84EL7A7WoKRbcqlJRiJZCqRUR0aYUBGXR2gISQJYEAiEBkgjJZJ/1rp/fH+fM5J5hJpmBmXPvzH0/H4/7yD3LnPmceybnfc/2/Zq7IyIi0i9R7QJERKS2KBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiYgsGM7vWzLaY2RPDTDczu9LM1prZY2Z2dFy1iYjIHnEeMVwHLN3L9NOBReHrfOD7MdQkIiKDxBYM7v5bYPteZjkT+FcP3A9MM7P946lORET6papdQIW5wIaK4Y3huBcHz2hm5xMcVdDU1HTM4YcfPupf1tGZ46XdfRw5p5WE2SurWERkgnrooYe2unv7UNNqKRhGzN2XA8sBlixZ4itXrhz1Mq75zbNcfvtT/G7ZaUzJTMiPQUTkFTOz54ebVkt3JW0C5lcMzwvHiYhIjGopGFYAHw7vTjoe2OXuLzuNJCIi4yu2cyhm9mPgFKDNzDYCXwTSAO5+NXAb8C5gLdADnBtXbSIiskdsweDu5+xjugOfiKkcEREZRi2dShIRkRqgYBARkQgFg4iIRNR9MKhnUxGRqLoNBj3sLCIytLoNBhERGZqCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEXUfDHq+TUQkqm6DwdATbiIiQ6nbYBARkaEpGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiElH3weDqwk1EJKJug0E9uImIDK1ug0FERIamYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISETdB4MebxMRiar7YBARkSgFg4iIRKSqXYDISLk7prZMZILryRfp7CtSKJXpzpXoyRfJpBJMyaToK5TozhXJF8vkS2X6CiVKZSiUykxtTJNKGgkzzCCdTHBIezPTmzJjXqOCQWLl7ry0u4/efIlnO7rZ2pXDHebPaGRHT4HefJF1Hd30FUqs39bDju4827vz9BVK9BZKLJjZRCIBs1uyNGeD/0ibdvayq7dArlAmk0qwsK2JUjm4ejR9SoZS2VnQ1kRLOH+uWCaZMNZ3dNPWkmH/qY0cOHMKZYeWhhRNDSnK7iTMyBVLPLeth007epmSSTK9KUNrNkVzQ4opmRS7+wp0dOZoCcc1ppM0ZpKUys6u3gInH9pOOqkD81djqC8EvfkSuWKJ1myavmIJd8gVy+SLZbZ35wd2nN25YCfcnS8CsGV3H1u78uRLZXKFMmV3sukkjpMrlNndV8AwGjMJevIltuzO0Zkr0pMrUiw7fYUSyYSRSSbY3VfELPibKbtTKDkN6QT5YpliyUkY9BRKFIplzCycp0yhNHZXNpedeSQfPmHBmC2vX6zBYGZLge8ASeCH7n7FoOkHANcD08J5Lnb32+KsUV6Z/lZqX9zVx9ObO+krlHh80y62dubZ1p1nR0+eKZkkq/+4m23d+b0uK5kIdgIHtTWx/7RG2lsamNGUwYC1HV3s6inwVHcnuWKJqY1p5kxrZL/WLK2NaXZ059mwo5eWbAp3eH7bTjr7Cvxy1UsAJAwSZpTc2a81S2dfka5ccZ/rlzAov4L/z6+fP40r3v9aFs1qGViviahUdrZ159i4o5d8Mdih5otldvYU6MwV6eorsr07B0BDKklfocS27jy5YonuXIliuUwykQh2lrkS3fngG3MqEYSmGRwyqzncGffRlSuSLwXL78oVSScTuDvZVJK+Ymlg52oGr6SB5FTCaEglSJjRVyxhFgy3ZtOUPQiAxnSSWa1ZWrMp9m/Nkk4lyKYS5EtlSmWnJZumXHa680XMjFQi+CKRTSVJJY1SGZoakqSTCUplJ5Uw0uHvaG1MkUoYTQ0ppmSS5AplevIlGjNJmhtSZFIJ0kmjMZ0imTBSSWN7dx73YFuYBUF4yKzmMdvGkc9nXJY6BDNLAlcB7wA2Ag+a2Qp3X10x2xeAm9z9+2a2GLgNWBBXjbJ3fYUSv1q9mavuWcuCmU2sfnE3CYOOzhzd+dKQP5NNB4fILdngT+3EQ9o4uL2J1myag2c1M3daI735Elu7c7RmU6STCQ6d3UJDKjGmp426w51/Np0kYdBXKNOYSQKwozvPxh29lN3JFcts68qRzSQxgp3cATOnMGdqls5cke1debZ05tjeHQRdU0OKWS0N7OotsLOngBnhjszYuKOXK25/iqXf/m/aWxr40zfM5eRF7Rw5p3XMDv978yXMgvWqVCoH307NYNOOXjo6c3SG33CTCSNfDHZEW7ty7Owp0Fsosb07T2dfkZ09ebb35MkVgtMZ3bkivYXSPnfADalgJ18olWlIJZnZnKEhlaCpIdgJlspFSu40ZVLMbs0C0L+FS+7cv24bUxvTzG7NMnd6I5lkgqmNaVqyaQqlMhjki8GyWxtTZJIJdvUWmJJJkTDCnWmCmU0Z8qUyAE3h315TQ/D3N6u1gZlNDRMypA9uj+93xXnEcByw1t3XAZjZjcCZQGUwONAavp8K/DHG+iRUKjvrt3bx6IZd/Ov9z/Pohp0vm+eplzojw4fv18KMpgz7Tc1ywkEzaWtp4NgFM2jKJGviukD/jqFffygATG/KjGhH3ZpN05pNs6Ct6WXT5g/zM6ceMZvbH3+R3zzdwQ/+ex3Lf7sOgHTSmNWS5Y0HzeD4hTNZNLuZhlRwSgMY2EG/tKuPl3bnWP3ibrZ35+jJldjRk6crV8QdimUnmTBmNmUoOxTLwWmMkRwFVWpIJWhrbqAlm2L6lAyH79dCNp0MduyZFI2ZJO0tDcyb3khDKgjNdCrBjKZM5DSamela0CQQZzDMBTZUDG8E3jhoni8Bd5rZJ4Em4O1DLcjMzgfOBzjggAPGvNB60VcokS+VueG+5+nozPHzRzaxq7cw7PxHHzCNEw9uY9HsZt5yaDsNqSTFcpnmhpR2BMOYO62Rj7/5ID7+5oPY0tnH4xt38cyWLjbt6OXFXb3c9eQWbnl4016XkUoYi2a3MLu1gaYZKaZNCb5FA7RkU/Tki2ztzA+cL08ljZZseuC02f5Ts8xszjCjqYGyO+5OJhlcC2lrztCSTWNAYoy+RetvYeKrtYvP5wDXufs3zewE4AYze427lytncvflwHKAJUuWvKorOZO5A7cXtvVw88MbaW/OMG1KhqPmTeWAGVP4+SOb+OKKVXT2Df2tclZLA6+fP42TD23nmAOnc/h+LXv5z54cZrwMNqsly6lHZDn1iNkD40pl56mXdrN+azeGhefijeaGFFMb08yd3sj0KRldwJZYxRkMm4gecc8Lx1U6D1gK4O73mVkWaAO2jHUxk/FbTV+hxM8f2cRNKzfwyAsvP/0z2JRMciAs3vO6OZx0cNuYfWuUkUkmjCPnTOXIOVOrXYrIgDiD4UFgkZktJAiEs4G/GDTPC8CpwHVmdgSQBTpirHFC2rC9h9+t3coltzweGT+7tYG3HT6LHd0F5k1v5L512wZOLVz6nsXMmz6lShWLSC2LLRjcvWhmFwB3EJx/uNbdV5nZMmClu68APgv8wMw+TXAh+qPuk/lkz8iVyk6xXGZ3b5HjL79r4D79wf7smHl84q2HsN/U7MvuVBERGYlYrzGEzyTcNmjcpRXvVwMnxVlTrdvVU+B1y+7c53zHLZjBDz6yhKmN6RiqEpHJrNYuPguwrSvH3U9t4W9ufmzI6V949xGcdez8gTtTRETGkoKhhuSKJT53yxP87OGNkfGXvfc1/MVxB+jCsIjEQsFQI8pl57Av/DIy7qJ3Hspfn3KIAkFEYqVgqLKfP7KRT//k0ci4Bz536kCTASIicVMwVOmep6c3d/LOb/02Mu7kQ9u5/txjJ+UzFiIycdRtMFRj11sqO3c9uZnfr9/OD3+3fmD81z5wFKcePouZzQ1VqEpEJKpug6EajvvKryNNTr/v6Ll8889epyMEEakpCoZxtGF7D/tNzbLo87e/bNq0KWm+9v6jFAoiUnMUDOPg2t+tZ9mtq4ec9uSypZEmn0VEao2CYQz8+PcvDLRTtGhWM89s6XrZPHd86mQOnd2sIwQRqXkKhldpy+6+SON1/aHw+vnTuP5jx6mJChGZcBQMr0JfocQXV6x62fj3vn4O3z77DVWoSETk1VMwvELuzuF/t+dJ5eeueDfFUpkXd/Uxf4aasxaRiavuu4XyV/iE28JL9jQS++UzjgQglUwoFERkwqvbI4ZXcw345of2NHJ370WnDNk5vIjIRFX3RwyvxEU/Ddo2Oue4+QoFEZl0FAyjtODi/xp4f/n7jqpiJSIi40PBMAq7egsD7z/4xgOqWImIyPhRMIzCQ89vH3j/lT99bRUrEREZPwqGUXjo+R0kE8aTy5ZWuxQRkXGjYBiFR17YyRH7t6itIxGZ1BQMI9SbL/G/z25j+pRMtUsRERlXdR8MPsLn2/7p7mcAmNmkYBCRya1ug2E0z7eVy8737n0WgC+FTzmLiExWdRsMo/H75/bcjTRNp5JEZJJTMIzA2cvvB+BHH39jlSsRERl/CoZROOmQtmqXICIy7uq2Eb2RKpTKmMG5Jy6sdikiIrHQEcM+vLC9B3c4ck5rtUsREYmFgmEfvnr7UwDMbs1WuRIRkXgoGPbhztWbAThq/tQqVyIiEo+6D4aR9t/Wmk2Pax0iIrWiboPBRtCFm4ePRR82u2W8yxERqRl1Gwwjsb07D8BZx86vciUiIvGJNRjMbKmZrTGztWZ28TDz/LmZrTazVWb273HWN9gxl/0agIa08lNE6kdszzGYWRK4CngHsBF40MxWuPvqinkWAZcAJ7n7DjObFVd9e3P4frpVVUTqR5xfhY8D1rr7OnfPAzcCZw6a5/8AV7n7DgB33xJjfRHl8p7L0sccOL1aZYiIxC7OYJgLbKgY3hiOq3QocKiZ/Y+Z3W9mQ3aVZmbnm9lKM1vZ0dExLsWufH4HALNaGsZl+SIitarWTp6ngEXAKcA5wA/MbNrgmdx9ubsvcfcl7e3t41JIvlgG4KsfOGpcli8iUqviDIZNQOXtPfPCcZU2AivcveDu64GnCYIidi/u6gXg4Lbmavx6EZGqiTMYHgQWmdlCM8sAZwMrBs3zC4KjBcysjeDU0rrxLMqH6cJtw/YeAGa16lSSiNSX2ILB3YvABcAdwJPATe6+ysyWmdkZ4Wx3ANvMbDVwD/A37r5tPOrZ1/NtV969FoBsOjkev15EpGbF2uy2u98G3DZo3KUV7x34TPgSEZEqqLWLzzWhO1esdgkiIlWjYBhCR2cOgP+39PAqVyIiEj8FwxC2dgXBsFid84hIHVIwDJIrlvj6HWsAaG/WHUkiUn8UDIN87pYneGD9dgDamjNVrkZEJH4KhkF+9vDGgfczdcQgInWo7oNhbz24JRP77sxHRGSyqdtg0C5fRGRodRsM+5JV5zwiUqdGtPczs7SZvWRmR453QdW0u68w8P6xL55WxUpERKpnRMHg7gWgwN5PyU94XX17nnjOpHTEICL1aTR7v38CLjGzWNtXilNX2BTGZe99TZUrERGpntHs5N8MvAXYZGZPAN2VE939jCF/agLpDE8lzZveWOVKRESqZzTBsBX42XgVUgs6w1NJLdl0lSsREameEQeDu587noXUgv5TSS3ZSXu2TERkn0Z8jcHM7h6q/2UzazWzu8e2rPG3aWcfALt799yJ1NWnYBARGc3F51OAoRoPyhJcf5hQrv7NswDc/sRLA+PuWbMFgOYGBYOI1K997gHN7OiKwaPMbHvFcBI4Ddg01oXFpbLP5ztWbQagKaNgEJH6NZI94EqC5xccuHOI6b3AJ8eyqGpLqI0kEaljIwmGhQRNC60DjiW4O6lfHtji7qVxqC12Jx48k3yxXO0yRESqap/B4O7PA5jZFcCx7n515XQz+79mNtfd/26caoxNV67IjCb1wSAi9W00F58/BDwyxPiHgQ+PTTnVtX5rN6mEmsIQkfo2mr3gLKBjiPFbgdljU078Kq4909lX5NdPbq5eMSIiNWA0wfACQ9+WejKwcYjxE0r/3UlqbltE6t1o7su8BviWmWWA/gfaTgUuB7461oXFpf+AIRdedL7w1EOrV4yISA0YTZMY3zSzNuBK9jzolge+4+5fG4/i4tSTD26satQRg4jUuVE9yeXul5jZZcDicNST7t419mXFrycfNIcxRQ+3iUidG/Ve0N27gQfHoZaq2NlT4PbHX+SvfvQwAI2ZZJUrEhGprrr/enzt/6zn3qe3DAw3phUMIlLfdEIdIh2WTtERg4jUOQUDsG7rns7osgoGEalzCoZBdCpJROqdggGwisZUN+7orV4hIiI1QMEATGvc08fzKYe1V7ESEZHqUzAALdk9wZBO6iMRkfoW617QzJaa2RozW2tmF+9lvvebmZvZkjjqWtjWFMevERGZEGILBjNLAlcBpxM8OX2OmS0eYr4W4ELggbhqK5V93zOJiNSJOI8YjgPWuvs6d88DNwJnDjHf3xM0ytcXV2H5Upk5U7M88eXT4vqVIiI1K85gmAtsqBjeGI4bYGZHA/Pd/b/2tiAzO9/MVprZyo6OobqIGJ1CqczBs5ppbqj7B8FFRGrn4rOZJYB/BD67r3ndfbm7L3H3Je3tr/4uokKpTEYXnUVEgHiDYRMwv2J4XjiuXwvwGuBeM3sOOB5YEccF6ELRdTeSiEgozr3hg8AiM1sYdvZzNrCif6K773L3Nndf4O4LgPuBM9x95XgXViiVSacUDCIiEGMwuHsRuAC4A3gSuMndV5nZMjM7I646hpIvlUknbd8ziojUgVivtrr7bcBtg8ZdOsy8p8RR01sPa2f1i7t1jUFEJFS3e8O25qB30ulNGTbvzvHA+u1VrkhEpDbUbTD0e3ZL0DPp+oqmt0VE6lndB0OkaVUREVEw6JqziEhU3QfDwy/sBGDZmUdWuRIRkdpQ98HQr625odoliIjUBAVDSC2siogEFAwhxYKISKBug+EvTz44Mjy1ontPEZF6VrfBcNIhbZHh186dWqVKRERqS90Gw+DHF2Y0ZapTiIhIjVEwiIhIRN0GQyWFhIjIHnUbDA2p5MD7tx42q4qViIjUlroNhoVtTQPvUwkdMoiI9KvbYKikbj1FRPbQHhEolsvVLkFEpGYoGICefKnaJYiI1AwFA7Crt1DtEkREaoaCAXhs465qlyAiUjMUDCIiEqFgEBGRCAWDiIhEKBiANw1qaVVEpJ4pGIDFc1qrXYKISM1QMKBuPUVEKikYgLIrGERE+ikYAOWCiMgeCgYgk9LHICLSr673iH+49B189MQFXHjqomqXIiJSM1LVLqCapk3J8KUzjqx2GSIiNaWujxhEROTlFAwiIhKhYBARkYhYg8HMlprZGjNba2YXDzH9M2a22sweM7O7zOzAOOsTEZEYg8HMksBVwOnAYuAcM1s8aLZHgCXufhRwM/C1uOoTEZFAnEcMxwFr3X2du+eBG4EzK2dw93vcvSccvB+YF2N9IiJCvMEwF9hQMbwxHDec84Dbh5pgZueb2UozW9nR0TGGJYqISE1efDazDwFLgK8PNd3dl7v7Endf0t7eHm9xIiKTXJwPuG0C5lcMzwvHRZjZ24HPA29x91xMtYmISCjOI4YHgUVmttDMMsDZwIrKGczsDcA1wBnuviXG2kREJBRbMLh7EbgAuAN4ErjJ3VeZ2TIzOyOc7etAM/BTM/uDma0YZnEiIjJOYm0ryd1vA24bNO7Sivdvj7MeERF5uZq8+CwiItWjYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiETEGgxmttTM1pjZWjO7eIjpDWb2k3D6A2a2IM76REQkxmAwsyRwFXA6sBg4x8wWD5rtPGCHux8CfAv4alz1iYhIIM4jhuOAte6+zt3zwI3AmYPmORO4Pnx/M3CqmVmMNYqI1L1UjL9rLrChYngj8Mbh5nH3opntAmYCWytnMrPzgfPDwS4zW/MKa2obvOw6oHWuD1rn+vBq1vnA4SbEGQxjxt2XA8tf7XLMbKW7LxmDkiYMrXN90DrXh/Fa5zhPJW0C5lcMzwvHDTmPmaWAqcC2WKoTEREg3mB4EFhkZgvNLAOcDawYNM8K4CPh+w8Ad7u7x1ijiEjdi+1UUnjN4ALgDiAJXOvuq8xsGbDS3VcA/wzcYGZrge0E4TGeXvXpqAlI61wftM71YVzW2fSFXEREKunJZxERiVAwiIhIRN0Gw76a55gozGy+md1jZqvNbJWZXRiOn2FmvzKzZ8J/p4fjzcyuDNf7MTM7umJZHwnnf8bMPjLc76wVZpY0s0fM7NZweGHYlMrasGmVTDh+2KZWzOyScPwaMzutOmsyMmY2zcxuNrOnzOxJMzthsm9nM/t0+Hf9hJn92Myyk207m9m1ZrbFzJ6oGDdm29XMjjGzx8OfuXJEDw27e929CC5+PwscBGSAR4HF1a7rFa7L/sDR4fsW4GmCJke+Blwcjr8Y+Gr4/l3A7YABxwMPhONnAOvCf6eH76dXe/32se6fAf4duDUcvgk4O3x/NfBX4fu/Bq4O358N/CR8vzjc9g3AwvBvIlnt9drL+l4PfDx8nwGmTebtTPDA63qgsWL7fnSybWfgZOBo4ImKcWO2XYHfh/Na+LOn77Oman8oVdoQJwB3VAxfAlxS7brGaN3+A3gHsAbYPxy3P7AmfH8NcE7F/GvC6ecA11SMj8xXay+C52DuAt4G3Br+0W8FUoO3McGdcCeE71PhfDZ4u1fOV2svgmd61hPeMDJ4+03G7cyelhBmhNvtVuC0ybidgQWDgmFMtms47amK8ZH5hnvV66mkoZrnmFulWsZMeOj8BuABYLa7vxhOegmYHb4fbt0n2mfybeBvgXI4PBPY6e7FcLiy/khTK0B/UysTaZ0XAh3Av4Snz35oZk1M4u3s7puAbwAvAC8SbLeHmNzbud9Ybde54fvB4/eqXoNh0jGzZuBnwKfcfXflNA++Kkya+5LN7E+ALe7+ULVriVGK4HTD9939DUA3wSmGAZNwO08naFhzITAHaAKWVrWoKqjGdq3XYBhJ8xwThpmlCULhR+5+Szh6s5ntH07fH9gSjh9u3SfSZ3IScIaZPUfQSu/bgO8A0yxoSgWi9Q/X1MpEWueNwEZ3fyAcvpkgKCbzdn47sN7dO9y9ANxCsO0n83buN1bbdVP4fvD4varXYBhJ8xwTQniHwT8DT7r7P1ZMqmxe5CME1x76x384vLvheGBXeMh6B/BOM5seflN7Zziu5rj7Je4+z90XEGy7u939g8A9BE2pwMvXeaimVlYAZ4d3sywEFhFcqKs57v4SsMHMDgtHnQqsZhJvZ4JTSMeb2ZTw77x/nSftdq4wJts1nLbbzI4PP8MPVyxreNW+6FLFiz3vIriD51ng89Wu51Wsx5sIDjMfA/4Qvt5FcG71LuAZ4NfAjHB+I+gw6VngcWBJxbI+BqwNX+dWe91GuP6nsOeupIMI/sOvBX4KNITjs+Hw2nD6QRU///nws1jDCO7WqPK6vh5YGW7rXxDcfTKptzPwZeAp4AngBoI7iybVdgZ+THANpUBwZHjeWG5XYEn4+T0LfJdBNzAM9VKTGCIiElGvp5JERGQYCgYREYlQMIiISISCQUREIhQMIiISoWAQ2QczW2BmbmY11dF8rdYlE5+CQWTfNhA0RvYHADM7Jdwht8VVgJnda2bf3VtdImMltj6fRSYqdy8RNGQ25sws7UFzD6M2nnVJfdMRg9SU8Jvx98zsH8xsa9iByTfMLBFOf87MLhriZ75bMfycmV1qZteZWaeZbTCzsyzo6OZGM+sKOzN55whrGjhlE7Zge084qSMcf104n5nZ35rZs2bWG3aO8qEhlnOOmd1tZr3AX5rZTAs6odkY/twqMzu34ueuA94CfCL8eQ+X9bJTSWZ2sgWd1PSZ2WYz+1bY7MuIPt9wnvdZ0AlMr5ltN7PfmFl/655SBxQMUos+CBSBE4ELgE8BZ41yGZ8iaBbhaIKOXa4n6NTnNoKmJX4L/JuZZUe53A3A+8P3RxKcyrkwHL6MoDmDTxB0DnM5cI2ZvXvQMi4HvhfO8wuCphweBv4kXOZ3wp87NZz/QuA+4F/C37c/0SaWATCzuQQdsTxC0Pz6eQTt718+aNZhP18z24+gYcLrgSMIOpG5YQSfi0wm1W4nRC+9Kl/AvcB9g8b9Cvhh+P454KIhfua7FcPPAT+uGG4maE/qyopxC8JxS0ZQU2RegvaZHGirmKcJ6AXePOhnvw3cNmg5nx3B77yxf52HWsdh6voKQds6iYp5PgrkgCkj/HyPDpd5YLX/FvSq3kvXGKQWPTZo+I/ArFe6DHfvMrMegkbH+m0O/x3tcoezmOCb/y/NrLIBsjRBUFVaWTlgZkmCvhXOIuhEpYGg6857R1nDEcD97l6uGPe7cFmHsOcz2dvn+yhBo21PmNmd4fub3b1jlLXIBKZgkFo0+GKss+e0Z5mghclK6REuozBoGMbudGr/ct5D0Fz03mrpHjR8EfBZglNGjwNdwD8wdqEF0Y5ehv183b0UXns5nqDp5vOAy83sLe7+6BjWIzVMwSATTQfBOXYAwmsEhxOcV49LPvw3WTFuNcEpmwPd/e5RLu9NwH+6+w0w0MfGocDOQb8zOcTPVnoS+HMzS1QcNbwp/NlnR1qMuzvBNY37zGwZsIrgaEbBUCcUDDLR3A18zMxWEITE54n/7/h5gp1DZgAAAAEjSURBVG/Z7zaz/wR63b3TzL4BfCPcsf+W4NrG8UDZ3ZfvZXlPA2eZ2ZsIOrD/JEF3lpVh9xxwXHhXVBewfYjlfI/gQvL3zOw7BP0WXEFwbaJnJCsWdv7ydoKOXzYTXMSeTxB8Uid0V5JMNJcThMN/AHcSnEOP82gBDzqp/yLBxd7NBJ2fAPwd8CWCU0OrCC7qvh9Yv49FXkZwB9XtBIHSDfxo0DzfIPjmv5ogEA8Ypq7TCXbmfwCuJegE5nOjWL1dBN1n3kpwIfubwN+7+7+NYhkywamjHhERidARg4iIRCgYpO6Z2dXh09BDva6udn0icdOpJKl7ZjYLaB1m8m533xJnPSLVpmAQEZEInUoSEZEIBYOIiEQoGEREJELBICIiEf8f0etmi2bGDKIAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"a-PpzgP_OdAT"},"source":["### Without Learning"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"XuE1kVH6OdAU","executionInfo":{"status":"ok","timestamp":1634747782889,"user_tz":-330,"elapsed":830,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"bcc22ee6-f8e0-40f3-97e9-ec24e9ab7b43"},"source":["# Do not learn\n","# use first reward function initially and then switch to third reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new2]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, times_of_day, actions, cost_functions, do_learn=False)\n","\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeNElEQVR4nO3de5hcdZ3n8fenLn1NJyEkgUCCJCMCEbwwLeJ6gREvAWdgV/cZYfURHUbcWXFx1Z0H1hnHZVTUQR0ZQWQcB5dREJ1ZjICLjsD6zCwyNHINGAzhkgvkQm6dvtXtu3+c002dppN0oLuqk/q8nqeenPM7p059T51Ofepc6ncUEZiZmY3KNbsAMzObWRwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ0LBknfkbRZ0sN7mC5JV0haI+lBSSc1qjYzM3teI/cYrgVW7GX6GcAx6eMC4JsNqMnMzMZpWDBExC+BbXuZ5Wzgf0XiV8BcSYsaU52ZmY0qNLuAOkcC6+rG16dtz4yfUdIFJHsVdHd3/+5xxx3XkALNzA4W995779aIWDDRtJkUDJMWEdcA1wD09vZGX19fkysyMzuwSHpqT9Nm0lVJG4AldeOL0zYzM2ugmRQMK4EPpFcnnQLsjIgXHEYyM7Pp1bBDSZKuB04D5ktaD/wFUASIiKuBW4EzgTXAIPChRtVmZmbPa1gwRMS5+5gewEcbVI6Zme3BTDqUZGZmM4CDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmltHQYJC0QtJqSWskXTzB9KMk3SHpPkkPSjqzkfWZmVkDg0FSHrgSOANYDpwrafm42f4MuDEiXgucA1zVqPrMzCzRyD2Gk4E1EbE2IkrADcDZ4+YJYHY6PAfY2MD6zMyMxgbDkcC6uvH1aVu9zwLvl7QeuBX42EQLknSBpD5JfVu2bJmOWs3MWtZMO/l8LnBtRCwGzgSuk/SCGiPimojojYjeBQsWNLxIM7ODWSODYQOwpG58cdpW73zgRoCIuAvoAOY3pDozMwMaGwz3AMdIWiqpjeTk8spx8zwNnA4g6XiSYPCxIjOzBmpYMEREBbgQuA14lOTqo1WSLpV0VjrbJ4EPS3oAuB74YEREo2o0MzMoNPLFIuJWkpPK9W2fqRt+BHhjI2syM7OsmXby2czMmszBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkNDQZJKyStlrRG0sV7mOcPJT0iaZWk7zeyPjMzg0KjXkhSHrgSeDuwHrhH0sqIeKRunmOAS4A3RsR2SQsbVZ+ZmSUaucdwMrAmItZGRAm4ATh73DwfBq6MiO0AEbG5gfWZmRmNDYYjgXV14+vTtnqvAF4h6V8l/UrSiokWJOkCSX2S+rZs2TJN5ZqZtaaZdvK5ABwDnAacC/ytpLnjZ4qIayKiNyJ6FyxY0OASzcwObo0Mhg3AkrrxxWlbvfXAyogoR8QTwGMkQWFmZg3SyGC4BzhG0lJJbcA5wMpx89xEsreApPkkh5bWNrBGM7OW17BgiIgKcCFwG/AocGNErJJ0qaSz0tluA56T9AhwB/DfI+K5RtVoZmagiGh2DS9Jb29v9PX1NbsMM7MDiqR7I6J3omkz7eSzmZk1mYPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWcakgkFSUdKzkl453QWZmVlzTSoYIqIMlIEDu/8MMzPbp/05lPQ3wCWSGnY7UDMza7z9+ZB/M3AqsEHSw8BA/cSIOGvCZ5mZ2QFlf4JhK/CP01WImZnNDJMOhoj40HQWYmZmM8OkzzFIun2i+y9Lmi3p9qkty8zMmmV/Tj6fBrRN0N5Bcv7BzMwOAvs8lCTppLrRV0naVjeeB94JbJjqwszMrDkmc46hj+T3CwH8bILpQ8DHprIoMzNrnskEw1JAwFrgdSRXJ40qAZsjojoNtZmZWRPsMxgi4ikASV8EXhcRV9dPl/SfJR0ZEX8+TTWamVkD7c/J5/cD903Q/mvgA1NTjpmZNdv+BMNCYMsE7VuBw6amHDMza7b9CYanmfiy1LcA66emHDMza7b96RLjW8DXJLUBoz9oOx24DPjSVBdmZmbNsT9dYnxF0nzgCp7/oVsJ+HpEfHk6ijMzs8bbry60I+ISSZ8DlqdNj0bE7qkvy8zMmmW/760QEQPAPdNQi5mZzQC+57OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmltHQYJC0QtJqSWskXbyX+d4jKST1NrI+MzNrYDBIygNXAmeQ/HL6XEnLJ5ivB7gIuLtRtZmZ2fMaucdwMrAmItZGRAm4ATh7gvn+kqRTvuEG1mZmZqlGBsORwLq68fVp2xhJJwFLIuKWvS1I0gWS+iT1bdky0S0izMzsxZoxJ58l5YCvAp/c17wRcU1E9EZE74IFC6a/ODOzFtLIYNgALKkbX5y2jeoBTgDulPQkcAqw0iegzcwaq5HBcA9wjKSl6c1+zgFWjk6MiJ0RMT8ijo6Io4FfAWdFRF8DazQza3kNC4aIqAAXArcBjwI3RsQqSZdKOqtRdZiZ2d7t9/0YXoqIuBW4dVzbZ/Yw72mNqMnMzLJmzMlnMzObGRwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMlo2GGq14KcPPUOtFs0uxcxsRmnZYPjhvev4k+/9mu/d/VSzSzEzm1FaNhg27xoB4Nldw02uxMxsZmnZYMjlBICPJJmZZbVsMCjJBWrhZDAzq9eywZAbTQbngplZRssGQxoL3mMwMxunZYNhdI/B5xjMzLJaNhh8jsHMbGINDQZJKyStlrRG0sUTTP+EpEckPSjpF5JeNo21AOBcMDPLalgwSMoDVwJnAMuBcyUtHzfbfUBvRLwK+BHw5emqJzd67tnJYGaW0cg9hpOBNRGxNiJKwA3A2fUzRMQdETGYjv4KWDxdxfgcg5nZxBoZDEcC6+rG16dte3I+8NOJJki6QFKfpL4tW7a8qGLG9hh8vaqZWcaMPPks6f1AL/BXE02PiGsiojciehcsWPBiXwTwHoOZ2XiFBr7WBmBJ3fjitC1D0tuATwOnRsTIdBXz/DmG6XoFM7MDUyP3GO4BjpG0VFIbcA6wsn4GSa8FvgWcFRGbp7OY3NhVSU4GM7N6DQuGiKgAFwK3AY8CN0bEKkmXSjorne2vgFnADyXdL2nlHhb3kvmXz2ZmE2vkoSQi4lbg1nFtn6kbflujavFVSWZmE5uRJ58bYfwvn5/YOjCpm/bcdN8Gjr74Fo6++Ba++vPHprNEM7OmaOFgeL531Wv/9Ql+7/I7+fT/fpiBkQoAz+0eYc3m/sxzIoKP/+D+sfErfvFbdg6WJ/2az+wc4tmdvjGQmc1sDT2UNJPk6nrd/uxPHhlrX7tlgFK1ynu+eVcy/oUzyeVERLD0kltfsJxXX/ozjprXxdPbkt/lff+PX8+dj23h+EU9vH7poazdMsAtD21kxQmLOO87/5Z57uf+/QmcfvxCFs3pHGvbtGuYLf0jvHzhLFY/28+iOR286ct3UKrUWNjTznXnv55jD+/JLCci+PXTO5jbVaR/uMKcziJHzesiP7qSL1L/cJm7126jXK2xbvsgZ5ywiCXzusZec/tgmZ1DZeZ0FpnX3Tbh89sKOdoL+ZdUx97UakGpWqOjmH2NkUqVCCjmcy/5fahf5nSui9lMoQP9qpze3t7o6+vb7+fddN8GPv6D+zn7NUfw4/s3jrXnlD3vcOV/Ool3vWoRR198y1jbMQtn8fNPnJppm4nmz2pnxQmHccGbf4dStcrnb3mUvqe2885XHs7ND27k0O52NuwYYvEhnazfPsTcriI7JrEHNH9WO1t3Z68kXja/m2ULZrFsQTedxTwbdwzx4wc2Uq7WmNfVRjGfY7BU4cTFczju8Nkce1gP87rbaC/mqFSDrrY82wdL5HM5KtUaA6Uq87qL3P/0Djb3jzBQqjK3s0gtgiefG2D1s/3sHCpTriYbq6ejQLUWzO4osn2wxEilNlbbnM4ibYUcbfkcEizsaWfJvC46i3m29I8wUqmxcyhZ72otqEWQk5jX3cZAqUJOYsP2IZ7dNUx7IQmaQ7ramNfdRnd7niPndjFUTubrKOY5dFYbu4bKFHI5FvS0c8TcThb2tLNtoET/SIWFPe0cNa+LI+Z0MlKpsnOozLaBEvN72ilXaxRyOXaPVBgcqVCuBZt2DjOnq0heIoBCTuwaLrN7pEKpkoRiZzFPqVqjs5inuz1PrQbDlSrbBkoMlarM6Sxy2JwODp/dwcKedmZ1FBxyLU7SvRHRO+G0Vg2GH9+/gYtuuJ8/ePUR/OSBjXudd1Z7gd3pIaZzT17CF/7DiUhiqFTlY9f/mn9+NLmytq2Qo1T3gTSRNZ8/g6e3DfJnNz3M/3v8uUnXe9m7T2Rr/whf2ct5jYU97dQiWNjTwSPP7Jr0svfm7csP45CuIg9t2MVbXjGfa365luWLkg/2xfO6mN1RYMdgmYc37uTJrQM8tW2QiOTD68TFczjhiDkMlauUqzXyEqs27mL1pv59v3CdjmKOYi5HqVqjrZCjp73AiYvncMTcTiJgbleR53aXqEUwWKpyaHcbnW3JB2VeYsdQmaFSlYGRCsVCjq39I6zfPkT/cJmFszso5MShs9rISWPhUakFz+0u0VnMU41gwax2li7oZqRcY/dImVKlxtbdJQZLFTbuSA4PFvKiUg029w8zp7NItRbsGq5MyXaYDl1teSq1oJDTWCDO7igyt6tIZ1uetnwuDfQqI5UqHcU8AyOVsef0D1cop3trHcU8OSV7aLM7ixBQLIiutgI5QaUaSEmgVWvBnM4i82e10dWW/N8a/TJQqtSo1oK2Qo6cYG5XG22FHIMjFYr5HJ1tebraCnQUc/R0FOlqy1PIiXI1yOegu71AMZ/8P6xF0F7IAaK9mHwxmN1RpKOYo7s9+SJRqQVDpSq1CLbuHmG4XGOkUqVcDYr55H3pLObpai/QP1wmJzGrvUB7IXn9tvSLQjX9NlnMi5yefz+3DZQYKFUBGBipIKB/uEIhn+zF7h6pUK4GOZGGeoGhUpVyrUalGgyVqzy7c5h8TkSQ/D/KKd0Tz3H68YfxmiVzX9T231swtOyhJI1dlbTvYBwNhdkdBS5796vG2jvb8nz7vNcREWPLe3bnMLtHKnS15Xlw/Q5WnLDoBctbtmAW3//wKWPjlWqNxzbtpn+4zOuXHbrXWj52+jFjwxFB/0iF2R3FCectVWrctfY5rrpjDas39XPxiuN463ELufuJbczqKPB7xy7c57qPd8kZx+91+nC5ynC5ytyuFx5aGlWp1nh8ywDrtg2Sy0F7Ic+2gRI5ibldRcrVGvNntbN9sMSxh/ewsKdjbH3Hzg0dIIbLVTbsGGLTrmFmdxRZ0NPOMzuHWbdtkE27hskpCaXZnUW29I9QzItyJZjTVWRWe4GcxBFzO9gxWKaSfvhUqjVmdxaZ3Vmko5BjuFJjqFSlLZ9juFJl90iy91LMi4U9HbQXc/QPV3hmxxCbdo2waVfyN7p9sDT2IZp8ECcf3NsHSmNhXq4kH+Ld7R0Ml2scMadz7AvQnM4ixYIYLtcYLicfrqVKjR2DZfK55IvTc7tLlCo12ot5IoKejuRDddOuYVZt3En/cIWejgJthRxdxeTfYl5sH0wCYtXGXZSrNdoLeSq1GqVKbezDtFV0teURyWdWIS+q1aBcqzFSqbHkkK4XHQx707J7DDc/uJELv38fZ554OLc+9OwLpn/k1GWc9eojeNcV/zLW9uQX3/WSajWzqVGtBbuHKwyWK1SqQTGfo1KrMTCSBFpybomxQ4rD5RrlanLIsFSpjX1rL+ZFRyFPLj082N2ep72QTwK6muwZDVeq9A8nX8CC5HVHKjV2DZUpVWtEPH+VY6UaVCN5nsTYHlgtksOdEckRiGot6aVtVnuBYl7UAobKVXYPJ18q2wo5Crnk0GRXW37CL0S1dK+kkH9x1xB5j2ECY79jSI/8/Ne3vpyj53fziRsfAOB9J7+MJfM6ueSM4xgqV7mo7pu6mTVXPifmdBWZw8R7y60glxM5pmcPumWDYfTtHN09L+RzvOOVhwNJMBx1aHL1zUdO/Z0mVGdm1jwt/zuGcjXZZSjkk5NKZmatrmU/CUcvbR+pJFcMFNIGn0cws1bXsnsMo4bK6R5DruXfCjMzoIWDYfRarKFScinq6HXFZmatrnWDIU2GwdLooaSWfSvMzDJa/tNwLBi8x2BmBrR0MCS7DKO9qRamqKM1M7MDXQsHQ2L0l5F9T21vciVmZjNDywbD+J5A+mdwZ2dmZo3UusEwbvzdJx3ZlDrMzGaalg2G8Y49rGffM5mZtYCWDYbxh5KOmNs58YxmZi2mZYPBzMwm1rLBEC84y2BmZtDCwWBmZhNr2WCoP8ewbH538woxM5thWjcY6oYveMuyptVhZjbTtGww1Ns+WG52CWZmM0bLBkOkx5KWze/mva9b0uRqzMxmjpYNhlF/e14v87rbml2GmdmM0fLBYGZmWS0bDKNXJbmzbTOzrJYNhlGSo8HMrF7LBoN/+WxmNrGWDYZR3l8wM8tq2WAY37uqmZklGhoMklZIWi1pjaSLJ5jeLukH6fS7JR09XbWMnXz2LoOZWUbDgkFSHrgSOANYDpwrafm42c4HtkfEy4GvAV+arnoGy9WkLh9MMjPLaOQew8nAmohYGxEl4Abg7HHznA18Nx3+EXC6pumyofue2g5AseBgMDOrV2jgax0JrKsbXw+8fk/zRERF0k7gUGBr/UySLgAuSEd3S1r9Imuaf8SXsstuAfPB69wCvM6t4aWs88v2NKGRwTBlIuIa4JqXuhxJfRHROwUlHTC8zq3B69wapmudG3koaQNQ31vd4rRtwnkkFYA5wHMNqc7MzIDGBsM9wDGSlkpqA84BVo6bZyVwXjr8H4HbI3xhqZlZIzXsUFJ6zuBC4DYgD3wnIlZJuhToi4iVwN8B10laA2wjCY/p9JIPRx2AvM6twevcGqZlneUv5GZmVq9lf/lsZmYTczCYmVlGywbDvrrnOFBIWiLpDkmPSFol6aK0fZ6kn0v6bfrvIWm7JF2RrveDkk6qW9Z56fy/lXTenl5zppCUl3SfpJvT8aVpVypr0q5V2tL2PXa1IumStH21pHc2Z00mR9JcST+S9BtJj0p6w8G+nSX9t/Tv+mFJ10vqONi2s6TvSNos6eG6tinbrpJ+V9JD6XOumNSPhiOi5R4kJ78fB5YBbcADwPJm1/Ui12URcFI63AM8RtLlyJeBi9P2i4EvpcNnAj8l6Vj2FODutH0esDb995B0+JBmr98+1v0TwPeBm9PxG4Fz0uGrgT9Jh/8LcHU6fA7wg3R4ebrt24Gl6d9EvtnrtZf1/S7wx+lwGzD3YN7OJD94fQLorNu+HzzYtjPwFuAk4OG6tinbrsC/pfMqfe4Z+6yp2W9KkzbEG4Db6sYvAS5pdl1TtG4/Bt4OrAYWpW2LgNXp8LeAc+vmX51OPxf4Vl17Zr6Z9iD5HcwvgLcCN6d/9FuBwvhtTHIl3BvS4UI6n8Zv9/r5ZtqD5Dc9T5BeMDJ++x2M25nne0KYl263m4F3HozbGTh6XDBMyXZNp/2mrj0z354erXooaaLuOY5sUi1TJt11fi1wN3BYRDyTTnoWOCwd3tO6H2jvyV8DfwrU0vFDgR0RUUnH6+vPdLUCjHa1ciCt81JgC/D36eGzb0vq5iDezhGxAbgceBp4hmS73cvBvZ1HTdV2PTIdHt++V60aDAcdSbOAfwQ+HhG76qdF8lXhoLkuWdLvA5sj4t5m19JABZLDDd+MiNcCAySHGMYchNv5EJKONZcCRwDdwIqmFtUEzdiurRoMk+me44AhqUgSCt+LiH9KmzdJWpROXwRsTtv3tO4H0nvyRuAsSU+S9NL7VuDrwFwlXalAtv49dbVyIK3zemB9RNydjv+IJCgO5u38NuCJiNgSEWXgn0i2/cG8nUdN1XbdkA6Pb9+rVg2GyXTPcUBIrzD4O+DRiPhq3aT67kXOIzn3MNr+gfTqhlOAneku623AOyQdkn5Te0faNuNExCURsTgijibZdrdHxPuAO0i6UoEXrvNEXa2sBM5Jr2ZZChxDcqJuxomIZ4F1ko5Nm04HHuEg3s4kh5BOkdSV/p2PrvNBu53rTMl2TaftknRK+h5+oG5Ze9bsky5NPNlzJskVPI8Dn252PS9hPd5Espv5IHB/+jiT5NjqL4DfAv8MzEvnF8kNkx4HHgJ665b1R8Ca9PGhZq/bJNf/NJ6/KmkZyX/4NcAPgfa0vSMdX5NOX1b3/E+n78VqJnG1RpPX9TVAX7qtbyK5+uSg3s7A/wR+AzwMXEdyZdFBtZ2B60nOoZRJ9gzPn8rtCvSm79/jwDcYdwHDRA93iWFmZhmteijJzMz2wMFgZmYZDgYzM8twMJiZWYaDwczMMhwMZvsg6WhJIWlG3Wh+ptZlBz4Hg9m+rSPpjOx+AEmnpR/I8xtVgKQ7JX1jb3WZTZWG3fPZ7EAVEVWSjsymnKRiJN097LfprMtam/cYbEZJvxlfJekLkramNzC5XFIunf6kpE9N8Jxv1I0/Kekzkq6V1C9pnaT3KrnRzQ2Sdqc3M3nHJGsaO2ST9mB7RzppS9p+bTqfJP2ppMclDaU3R3n/BMs5V9LtkoaAj0g6VMlNaNanz1sl6UN1z7sWOBX4aPr8SJf1gkNJkt6i5CY1w5I2Sfpa2u3LpN7fdJ53K7kJzJCkbZL+r6TR3j2tBTgYbCZ6H1AB/h1wIfBx4L37uYyPk3SLcBLJjV2+S3JTn1tJupb4JfAPkjr2c7nrgPekw68kOZRzUTr+OZLuDD5KcnOYy4BvSXrXuGVcBlyVznMTSVcOvwZ+P13m19PnnZ7OfxFwF/D36estItvFMgCSjiS5Ect9JN2vn0/S//5l42bd4/sr6XCSjgm/CxxPchOZ6ybxvtjBpNn9hPjhR/0DuBO4a1zbz4Fvp8NPAp+a4DnfqBt/Eri+bnwWSX9SV9S1HZ229U6ipsy8JP0zBTC/bp5uYAh487jn/jVw67jlfHISr3nD6DpPtI57qOvzJH3r5Orm+SAwAnRN8v09KV3my5r9t+BH8x4+x2Az0YPjxjcCC1/sMiJit6RBkk7HRm1K/93f5e7JcpJv/v9HUn0HZEWSoKrXVz8iKU9yb4X3ktxEpZ3k1p137mcNxwO/iohaXdu/pMt6Oc+/J3t7fx8g6bTtYUk/S4d/FBFb9rMWO4A5GGwmGn8yNnj+sGeNpIfJesVJLqM8bhym7nDq6HL+gKS76L3VMjBu/FPAJ0kOGT0E7Aa+wNSFFmRv9LLH9zciqum5l1NIum4+H7hM0qkR8cAU1mMzmIPBDjRbSI6xA5CeIziO5Lh6o5TSf/N1bY+QHLJ5WUTcvp/LexPwk4i4DsbusfEKYMe418xP8Nx6jwJ/KClXt9fwpvS5j0+2mIgIknMad0m6FFhFsjfjYGgRDgY70NwO/JGklSQh8Wka/3f8FMm37HdJ+gkwFBH9ki4HLk8/2H9Jcm7jFKAWEdfsZXmPAe+V9CaSG9h/jOR2lvVh9yRwcnpV1G5g2wTLuYrkRPJVkr5Oct+CL5KcmxiczIqlN395G8mNXzaRnMReQhJ81iJ8VZIdaC4jCYcfAz8jOYbeyL0FIrlJ/V+QnOzdRHLzE4A/Bz5LcmhoFclJ3fcAT+xjkZ8juYLqpySBMgB8b9w8l5N883+EJBCP2kNdZ5B8mN8PfIfkJjD/Yz9WbyfJ7TNvJjmR/RXgLyPiH/ZjGXaA8416zMwsw3sMZmaW4WCwlifp6vTX0BM9rm52fWaN5kNJ1vIkLQRm72HyrojY3Mh6zJrNwWBmZhk+lGRmZhkOBjMzy3AwmJlZhoPBzMwy/j8ykgRCE20jPQAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"wDoMeJ8rOdAU"},"source":["This section aimed at showcasing a real world scenario where contextual bandit algorithms can be used. We were able to take a context and set of actions and learn what actions worked best for a given context. We saw that the learner was able to respond rapidly to changes in the world.  We showed that allowing the learner to interact with the world resulted in higher rewards than the no learning baseline. We worked with simplistic features. VW supports high dimensional sparse features, different exploration algorithms and policy evaluation approaches."]},{"cell_type":"markdown","metadata":{"id":"CJk-yZDQjZMX"},"source":["## Contextual bandit with changing context\n","\n","> Customizing the context and changing it midway to see how fast the agent can adapt to the new context and start recommending better products as per the context"]},{"cell_type":"markdown","metadata":{"id":"V9D8ePsikWAv"},"source":["### Setting the context"]},{"cell_type":"markdown","metadata":{"id":"-9UvUHeflfz5"},"source":["We have 3 users and 6 items. Context 1 is time of day - morning and evening. Context 2 is season - summer and winter.\n","\n","Ground truth rules:\n","\n","1. User 1 likes Item 1 in morning, and Item 6 in summer\n","2. User 2 likes Item 2 in winter, and Item 5 in summer morning\n","3. User 3 likes Item 2 in morning, Item 3 in evening, and item 4 in winter morning"]},{"cell_type":"code","metadata":{"id":"uP6Qy9upkX9x"},"source":["USER_LIKED_ARTICLE = -1.0\n","USER_DISLIKED_ARTICLE = 0.0"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"lyLnQ1ERkZhh"},"source":["users = ['A','B','C']\n","items = ['Item1','Item2','Item3','Item4','Item5','Item6']\n","context1 = ['morning','evening']\n","context2 = ['summer','winter']\n","\n","context = pd.DataFrame(list(product(users, context1, context2, items)), columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = 0\n","\n","#user 1 likes Item 1 in morning, and Item 6 in summer\n","context.loc[(context.users=='A') & \\\n","            (context.context1=='morning') & \\\n","            (context['items']=='Item1'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='A') & \\\n","            (context.context2=='summer') & \\\n","            (context['items']=='Item6'), \\\n","            'reward'] = 1\n","\n","#user 2 likes Item 2 in winter, and Item 5 in summer morning\n","context.loc[(context.users=='B') & \\\n","            (context.context2=='winter') & \\\n","            (context['items']=='Item2'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='B') & \\\n","            (context.context1=='morning') & \\\n","            (context.context2=='summer') & \\\n","            (context['items']=='Item5'), \\\n","            'reward'] = 1\n","\n","\n","#user 3 likes Item 2 in morning, Item 3 in evening, and item 4 in winter morning\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='morning') & \\\n","            (context['items']=='Item2'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='evening') & \\\n","            (context['items']=='Item3'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='morning') & \\\n","            (context.context2=='winter') & \\\n","            (context['items']=='Item4'), \\\n","            'reward'] = 1\n","\n","context['cost'] = context['reward']*-1\n","\n","contextdf = context.copy()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"gGiJWfhtkc9_","executionInfo":{"status":"ok","timestamp":1634747969054,"user_tz":-330,"elapsed":1929,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"8740bf12-79d9-42d2-a96f-494395009429"},"source":["contextdf.cost.value_counts()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" 0    60\n","-1    12\n","Name: cost, dtype: int64"]},"metadata":{},"execution_count":27}]},{"cell_type":"markdown","metadata":{"id":"Jodzh1ofkl5X"},"source":["### Cost function util"]},{"cell_type":"code","metadata":{"id":"MSfJOXwMktkA"},"source":["def get_cost(context,action):\n","    return contextdf.loc[(contextdf['users']==context['user']) & \\\n","            (contextdf.context1==context['context1']) & \\\n","            (contextdf.context2==context['context2']) & \\\n","            (contextdf['items']==action), \\\n","            'cost'].values[0]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Yy89o4afkvYy","executionInfo":{"status":"ok","timestamp":1634748013910,"user_tz":-330,"elapsed":702,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"36cc4b90-f560-4dcc-b41e-8b664437cec6"},"source":["get_cost({'user':'A','context1':'morning','context2':'summer'},'Item2')"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0"]},"metadata":{},"execution_count":29}]},{"cell_type":"markdown","metadata":{"id":"apj4zt0_kw00"},"source":["### Vowpalwabbit format util"]},{"cell_type":"code","metadata":{"id":"xJ9laRJGkzSD"},"source":["# This function modifies (context, action, cost, probability) to VW friendly format\n","def to_vw_example_format(context, actions, cb_label = None):\n","    if cb_label is not None:\n","        chosen_action, cost, prob = cb_label\n","    example_string = \"\"\n","    example_string += \"shared |User users={} context1={} context2={}\\n\".format(context[\"user\"], context[\"context1\"], context[\"context2\"])\n","    for action in actions:\n","        if cb_label is not None and action == chosen_action:\n","            example_string += \"0:{}:{} \".format(cost, prob)\n","        example_string += \"|Action items={} \\n\".format(action)\n","    #Strip the last newline\n","    return example_string[:-1]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"BrASk6IPk007","executionInfo":{"status":"ok","timestamp":1634748035616,"user_tz":-330,"elapsed":693,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"ab6dbbbd-b935-4e9a-846b-c572b5906279"},"source":["context = {\"user\":\"A\",\"context1\":\"morning\",\"context2\":\"summer\"}\n","\n","print(to_vw_example_format(context,items))"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["shared |User users=A context1=morning context2=summer\n","|Action items=Item1 \n","|Action items=Item2 \n","|Action items=Item3 \n","|Action items=Item4 \n","|Action items=Item5 \n","|Action items=Item6 \n"]}]},{"cell_type":"code","metadata":{"id":"ez2lyHNCk2u2"},"source":["def sample_custom_pmf(pmf):\n","    total = sum(pmf)\n","    scale = 1 / total\n","    pmf = [x * scale for x in pmf]\n","    draw = random.random()\n","    sum_prob = 0.0\n","    for index, prob in enumerate(pmf):\n","        sum_prob += prob\n","        if(sum_prob > draw):\n","            return index, prob"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"ZflJA3J0k8hP"},"source":["def get_action(vw, context, actions):\n","    vw_text_example = to_vw_example_format(context,actions)\n","    pmf = vw.predict(vw_text_example)\n","    chosen_action_index, prob = sample_custom_pmf(pmf)\n","    return actions[chosen_action_index], prob\n","\n","\n","def choose_user(users):\n","    return random.choice(users)\n","\n","def choose_context1(context1):\n","    return random.choice(context1)\n","\n","def choose_context2(context2):\n","    return random.choice(context2)\n","\n","\n","def run_simulation(vw, num_iterations, users, contexts1, contexts2, actions, cost_function, do_learn = True):\n","    cost_sum = 0.\n","    ctr = []\n","\n","    for i in range(1, num_iterations+1):\n","        user = choose_user(users)\n","        context1 = choose_context1(contexts1)\n","        context2 = choose_context2(contexts2)\n","\n","        context = {'user': user, 'context1': context1, 'context2': context2}\n","        # print(context)\n","        action, prob = get_action(vw, context, actions)\n","        # print(action, prob)\n","\n","        cost = cost_function(context, action)\n","        # print(cost)\n","        cost_sum += cost\n","\n","        if do_learn:\n","            # 5. Inform VW of what happened so we can learn from it\n","            vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n","            # 6. Learn\n","            vw.learn(vw_format)\n","            # 7. Let VW know you're done with these objects\n","            vw.finish_example(vw_format)\n","\n","        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n","        ctr.append(-1*cost_sum/i)\n","\n","    return ctr\n","\n","\n","def plot_ctr(num_iterations, ctr):\n","    plt.plot(range(1,num_iterations+1), ctr)\n","    plt.xlabel('num_iterations', fontsize=14)\n","    plt.ylabel('ctr', fontsize=14)\n","    plt.ylim([0,1])"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"LFnJ5I2XlFR0","executionInfo":{"status":"ok","timestamp":1634748123342,"user_tz":-330,"elapsed":5744,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"0e809d4d-f955-46af-aec6-d4ae11bb0e73"},"source":["# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, context1, context2, items, get_cost)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcdb3/8ddnZjLZl7ZJF5rSFii3rYAsEfACgoAIiPC7boDLVS9Xfl73n3j9gVxxF/ddLvJTxO2Cu/YqigugP5UtbJUC1VJKm9IloW2adTKZ+dw/zkk6J03apDRnksz7+Xjk0bPlzOc7j3Tec77nnO8xd0dERGRIotgFiIjI1KJgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiYgtGMzsRjPbbmaPjLHezOxLZrbOzFab2fFx1SYiInvEecRwE3DuPtafBywLfy4H/jOGmkREZITYgsHd/wjs2McmFwHf9sDdQIOZLYinOhERGZIqdgEFFgKbCubbwmVbRm5oZpcTHFVQXV19wvLly2MpUERkprj//vs73L1ptHVTKRjGzd1vAG4AaGlp8dbW1iJXJCIyvZjZU2Otm0pXJW0GFhXMN4fLREQkRlMpGFYB/xxenXQy0Onue3UjiYjI5IqtK8nMbgbOABrNrA34AFAG4O7XA7cC5wPrgF7gjXHVJiIie8QWDO5+6X7WO/DWmMoREZExTKWuJBERmQIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkYhYg8HMzjWztWa2zsyuHGX9oWZ2h5k9aGarzez8OOsTEZEYg8HMksBXgfOAlcClZrZyxGb/AfzA3Y8DLgGui6s+EREJxHnEcCKwzt3Xu/sAcAtw0YhtHKgLp+uBp2OsT0REiDcYFgKbCubbwmWFPgi81szagFuBt4+2IzO73Mxazay1vb19MmoVESlZU+3k86XATe7eDJwPfMfM9qrR3W9w9xZ3b2lqaoq9SBGRmSzOYNgMLCqYbw6XFboM+AGAu98FVACNsVQnIiJAvMFwH7DMzJaaWZrg5PKqEdtsBM4CMLMVBMGgviIRkRjFFgzuPgi8DbgNeIzg6qM1ZvZhM7sw3OwK4E1m9jBwM/AGd/e4ahQREUjF+WLufivBSeXCZdcUTD8KnBJnTSIiEjXVTj6LiEiRKRhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCQiVewCRERKhbsPT+fyTm82x47uATKDebZ09rFpZx8bn+lhXl0FzbOqSCUMB3b3ZRnI5ensy9JYU04yAdt2ZzhtWSPPOaT+oNepYBARGYd8PvhQTyQMCD7kO7oHeLKjhx09GbZ3ZRjMOcmEkcs727syPNOdoaM7w87eLLt6B+joHhj+3b5sjrzv/TplSSObG2XFKKrSz1EwiIgcKHdnV2+Wde3dtHdlAKitSNGTydHenaEsYTRUlbG1s58n2nvIDObYtjvDuu3d7O7P0pMZJO+QTBgJA8MYyOXHfL2ypFFfmWZ2dRmNNeUsqK9jTk2ahBnJhFGdTlFVnmRebQWppLGwoZL59RUsbKhkR88AT+3oJWlG3p2GqjTlqQRV6SQ7e7O4O3NqyqmvLJuU90rBICKxyeedbD7oEunoGiDvTlkygeNUliVpqEqztbOfzr4s3Zksq9s6GRjM40DenbadfQDUlqeorUhx6Owq0qkEfQO5YNtcnp29A2RzzjPdGVKJBKmksW13hmd6Mvj4vohTW5GisixJY005JyyeRV1liobKNMlEEAaDuTzJRILGmjRHzqtldnWaxppy0qkEg/k8CTPmVKcxswN6n+bUlDOnpnzUdQ1V6QPa50QoGERkL+7OE+3dPLqliz+sbWfzrl6SCaOzL0vvQI7DGmtIp4za8jLm11eQy3vwoRj2g+/qy1JdniKfd7Z09pPLOwODeR7bupuu/sEDqilhsGh2Fe6wuz+oY2Bwzzf2ubXllCWDD2sHDmuqIZ93BnJ5jl5YT11lGbOr0xw5r4a5tRW4Q182R1nSmFdXQT48ophTk2Z+XcUBf6jPBAoGkRIwMJhn445euvqzONDRlSFhxvauDOlUgifauxnM5Xm4rZP17d109Q+SCT90G6rKWDKnmlzeqasoY1ZVmvXt3Qzk8nR0Z+jP5kkYw/3lFWUJ6irK6MvmMKB5VhV5dxJmnHfUfBY2VFGZTjC3toLKdJLegUFSiQT92RxbOvtpnlXJrOo01ekUyxfUUlWWJO/BydrKdHK4Tbl8cFSQGcxjBgsbKp/1h3nzrGf16zOGgkFkmujP5nh8axcbOnroGRikP5unP/zGG/Q7B10gf9/WRSqZIGGwpbOf/myONU/vpncgt8/9JxPG8vm1nH7kXOoqU6yYX8fhc6s5dtEskonRP3AHc3myOSeVNDKDecqSRnkqOeq2B1syYcytq4jltUqNgkEkBv3ZHO1dGXb1ZmmeVcnW3f1UpZNs252hO5NldnU5c2vLqa1I8WRHD6vbOnl0y2529Q6wqzdLR3eG9e09DI52GcsIZUkLT5AGJzSTCePC5x7CcxbW01idZiCXZ051Odl8nkPqK+nsy7LykDqq08kJf+NOJRMM5UBZUrdFzRSxBoOZnQt8EUgCX3f3T4yyzauADwIOPOzur46zRpEDkRnM8dDGXeQdmmdV8kzPAOvbu/nbtm42dPRw+9rtkf7w8ahKJ6lKp2isSbOgvpJTj2ji2EMbWDKniuryFAkz6iuDLpu5teXkw8sn59dVjPkNX2Q8YgsGM0sCXwVeBLQB95nZKnd/tGCbZcBVwCnuvtPM5sZVn8hE5PLOX57o4NePbOV3j21j2+7MmNvOrk5zwTELWLmgjlTC6M4MUlEWfM2eU5NmTnU57V0ZtnT24Q4LGipZsaCWlQvqJvwNfmFD5bNqlwjEe8RwIrDO3dcDmNktwEXAowXbvAn4qrvvBHD37THWJzKqrZ39tD61g589uJnVbZ1UlwfdPUOet2QWZ62Yx/OWzKIsmWDb7gy15SkOa6rmOYfUU1GWKOkrXGT6iTMYFgKbCubbgJNGbHMkgJn9maC76YPu/uuROzKzy4HLAQ499NBJKVamn1zeyebybN7VR09mkKd39ZPN5Tm8qYaVh9Tx1DM9/GbNNubUpBkYzA9foz6rOnpd+K7eAe7bsJOfPtjG3et3sKMnuFu1oaqMoxcGd5meuGQ2Jx02m7NWzJu0m4xEimWqnXxOAcuAM4Bm4I9mdrS77yrcyN1vAG4AaGlpGectKzId9Wdz3LdhB8c0N1BbnuKpHb1s2dXH0c31/KC1jW/ftYH+bI6FDZU8snn3mHeijjXMQFnSWDynmmOa66ktT/HYli4ebttFZjBPZVmSU45o5JjmepbPr+X5h8+htkIhIDNfnMGwGVhUMN8cLivUBtzj7lngSTP7G0FQ3BdPiVIM3ZlBtuzqY/Gcaj6wag0337sRgJryFN2Z/d8M1VhTTk8mxwmLZ3FIQyXV5UlWLKhjQX0F2Zzz6NO7+fO6Do49tIFXntDM7v5BMtkcG57p5bEtu7lr/TP85IHNpBLGYU3VXHDMIZzznHmcvHQO9VUKAik9cQbDfcAyM1tKEAiXACOvOPoZcCnwTTNrJOhaWh9jjRKD/myOH93fxn/87JF9bpdMGC2LZ7G7P8vKBXV09mU5Ym4N8+oq6OzLctTCes5aPpfUfi6TfNHKebzz7GV7Lf/HI4J/PRxqoam2fPiksEgpiy0Y3H3QzN4G3EZw/uBGd19jZh8GWt19VbjuHDN7FMgB/+7uz8RVo0yuddu7ed037mFLZ39keUVZgiVzqunP5njf+Ss4/R+aMIx0Kp7r4s2MRbOrYnktkenAfLyjSk1RLS0t3traWuwyZAw7ewb4fusmPvGrx4eXpVMJ/u30w3nz6Yfrih2RIjGz+929ZbR1U+3ks0xD+bzT0ZOhsbqcju4Mq9s6yebyvPW/HoiMN/+CI5u44kVH8txFDcUrVkT2S8Eg47J2axcDg3kWzQ5uoCpLJvjMb9Zy870b6c+OfUdvOpXgNScdyrvOPlKXdYpMEwoGGdWTHT3k8nm6+ge56S8b+PlDT4/r95pqy0knExzTXM/bz1zGykPqJrlSETnYFAwlrG8gN9zHf+fa7bzhm+O/KnhubTkrD6njPef8A0sbq6ku15+SyEyh/80laGfPAMd95Lfj2raxppwlc6r43ptOim04ZREpLgVDCenszbJmSyev/n/3jLr+h29+PgAnHDpr+IHnIlJ6FAwloDszyFEfuC2y7IJjFvC5Vx3LTx9so6EqzTkr5+myUREBFAwz1vbd/TywcSdv/u4De6179UmH8tGLjiKRMC5+ngYhFJEoBcMMsKGjhzM+c+c+t2meVckf//2F6iISkf0aVzCYWRnBkNlnufuayS1JJuIHrZt4749Wj7n+a687gdOWNVKV1ncAERmfcX1auHvWzLIEj9uUKWAwl+eCL/+Jx7d2DS/7P2cfyVteeDhPPdPLotmVuopIRA7IRL5Gfhm4ysze6O77HwtZJs1f2zp56Vf+NDz//gtWctmpS4fnj5hbU4yyRGSGmEgwnAacDmw2s0eAnsKV7n7hwSxMRtfVn42Ewt1XncX8+ooiViQiM81EgqED+PFkFSL7NvKmtBUL6vjVO08rYkUiMlONOxjc/Y2TWYjs24kf/11k/tZ3nFqkSkRkpht3MJjZ7cDLRj5/2czqgJ+5+5kHuziBJVf+MjJ/4xtaOHP5vCJVIyKlYCJdSWcA6VGWVxCcf5BxWHnNr+kdyPHkteePeadxfzbH8vf/eq/lv3v3Czhibu1klygiJW6/wWBmxxfMHmNmOwrmk8CLCZ7hLCPk805nX5b27gyzqtJc+ePV9A7kAFh61a389YPn8GRHD9lcnhMWz+bbd23gmp/vfZvIn688k4UNlTFXLyKlajxHDK0E9y848JtR1vcBbz+YRc0EO3oGOH4/I5ge/cHR3s49vvevJ7F8fi1zasoPZmkiIvs0nmBYChiwHngewdVJQwaA7e6em4Tapq0HNu7kZdf9Zcz19159Fid+7Pdjrl+xoI7v/etJzK4eredORGRy7TcY3P0pADP7BPA8d7++cL2ZvdnMFrr7+yepxmnnE7fuefD98w+bw4L6Cn7y4Gbued9ZzKsL7jm49mVHc9VP/grADa87gV29WU5Z1qguIxEpOnMf3ygXZrYReKW73zNi+YnAD9198STUt18tLS3e2tpajJceVTaXZ9nVvwLg9itO57CmGtydXN5JJRORbTv7stSUp0hqYDsRiZmZ3e/uLaOtm8hVSXOB9lGWdwC6fjL0+8e2D08f1hQMTWFmpJJ7f/jXV5bFVpeIyHgl9r/JsI2MflnqC4C2g1PO9Ne2sxeA+64+u8iViIgcmIkcMXwN+LyZpYHbw2VnAdcCnzzYhU1H+bzz0V8+RkNVGU21upJIRKaniQyJ8VkzawS+xJ4b3QaAL7r7pyajuOli045eTvvUHcPzu3qzRaxGROTZmdDTW9z9KjP7KLAyXPSYu3cf/LKml8JQALjuNcePsaWIyNQ34cd6uXsPcN8k1DItveGb90bmT1g8i/OPXlCkakREnj097/EAbdrRyztueZAHN+4ZU/D3V5zO4U16SI6ITG8KhgM0svvo7x87j7LkRC7yEhGZmhQMB+CpZyIPr2Pdx87b6+Y1EZHpSp9mE9Q3kOP0T985PP/nK89UKIjIjKJPtAk65wt/GJ7+/+99ocY2EpEZR8EwQZt29A1PL5pdVcRKREQmh4LhAH32lc8tdgkiIpMi1mAws3PNbK2ZrTOzK/ex3cvNzM1s1JH/imVoJNp/Om4hLz+hucjViIhMjtiCwcySwFeB8wjunL7UzFaOsl0t8E7gnpHriu1792wE4KcP6kmmIjJzxXnEcCKwzt3Xu/sAcAtw0SjbfYRgUL7+GGsbl/f//BEAvnzpcUWuRERk8sQZDAuBTQXzbeGyYWZ2PLDI3X+5rx2Z2eVm1mpmre3toz0i4uC6/6mdLLnylww90+ilzz1k0l9TRKRYpszJZzNLAJ8Drtjftu5+g7u3uHtLU1PTpNaVzzsv/8+xn98sIjLTxBkMm4FFBfPN4bIhtcBRwJ1mtgE4GVhV7BPQ37prQzFfXkQkdnEGw33AMjNbGj7s5xJg1dBKd+9090Z3X+LuS4C7gQvdvagPdF67tSsyf/9/6MlsIjKzxTZWkrsPmtnbgNuAJHCju68xsw8Dre6+at97KI6hJ7H97K2nkDCYU6Mns4nIzBbrIHrufitw64hl14yx7Rlx1DSaJ9q7SZixtLGaL9++DoBjFzUUqxwRkVhpdNVRnPXZP+x/IxGRGWrKXJUkIiJTg4JhhE07evda9ut3nVaESkREikPBMMLIJ7MBLJ9fV4RKRESKQ+cYxnBxyyIe39bFFy4+ttiliIjESsEwhqOa6/nkK44pdhkiIrFTV9IY/um4hfvfSERkBlIwFFjf3j08XVOugykRKU0KhgIDuTwA5x89v8iViIgUj4KhQH82CIZX6OlsIlLCFAwFejKDAFSn1Y0kIqVLwVBgd18WgGqdXxCREqZgKPB/f7wagGTCilyJiEjxKBgKmAWBMK+uosiViIgUj4KhwKtamkmnEsyuThe7FBGRolEwFOjP5nX/goiUPAVDgb5sjsqyZLHLEBEpKgVDgb5sjooyvSUiUtr0KVigfyBHZVpHDCJS2hQMBdSVJCKiYIgIupIUDCJS2hQMBfoGdMQgIqJrMws8vrWLx7d2FbsMEZGi0hGDiIhEKBhCg+GzGN5x5hFFrkREpLgUDKH+wSAYairUuyYipU3BEOrP5gB0VZKIlDwFQygTHjFUpBQMIlLaFAyhr/3hCQDKNSSGiJQ4fQqGvn3XUwCU64hBREqcggFw9+FpDaInIqVOn4LArt7s8HQ6qbdEREqbPgWBfMERw9Od/UWsRESk+BQMQDa3JxhWLKgtYiUiIsUXazCY2blmttbM1pnZlaOsf7eZPWpmq83s92a2OI66suFdzwArF9TF8ZIiIlNWbMFgZkngq8B5wErgUjNbOWKzB4EWdz8G+BHwqThqG7qH4T3nHImZxfGSIiJTVpxHDCcC69x9vbsPALcAFxVu4O53uHtvOHs30BxHYUNHDIc31cTxciIiU1qcwbAQ2FQw3xYuG8tlwK9GW2Fml5tZq5m1tre3P+vChoKhTFckiYhMzZPPZvZaoAX49Gjr3f0Gd29x95ampqZn/XrDwZCakm+HiEis4hxKdDOwqGC+OVwWYWZnA1cDp7t7Jo7CBgaDq5J0D4OISLxHDPcBy8xsqZmlgUuAVYUbmNlxwNeAC919e1yFDR0xpFM68SwiElswuPsg8DbgNuAx4AfuvsbMPmxmF4abfRqoAX5oZg+Z2aoxdndQ6RyDiMgesT6Vxt1vBW4dseyagumz46xniIJBRGQPfRIC7V3BqQwFg4iIggGA9/98DaCTzyIioGCISOjdEBFRMBRK6z4GEREFA8B5R80HYG5tRZErEREpPgUD4A5HztM4SSIioGAAYDCfJ6UTDCIigIIBCB7UU5bUXc8iIqBgACCXd5IJBYOICCgYgODO55TuYRARARQMAAzm1ZUkIjJEwQAM5nTyWURkiD4N0clnEZFCCgZ0uaqISKFYh92ein50fxt/29ZNTyZX7FJERKaEkv+a/J4fPgzA5l19Ra5ERGRqKPlgEBGRqJIOhqt+snp4Wve3iYgESjoYbr530/D05y8+toiViIhMHSUdDIV292WLXYKIyJSgYAhddNzCYpcgIjIlKBiAs1fMo66irNhliIhMCQoGdOJZRKSQggFImJJBRGSIggHIuxe7BBGRKUPBAHRnBotdgojIlKFgALZ29he7BBGRKUPBgMZJEhEpVNLBUFMeDC6bGcwXuRIRkamjpIOhtiIIhkWzK4tciYjI1FHSweAOLYtn8dO3nFLsUkREpozSDgacw5tqaKwpL3YpIiJTRmkHg4PubRMRiSrtYEDBICIyUkkHQ0DJICJSKNZgMLNzzWytma0zsytHWV9uZt8P199jZksmsx6NhCEisrfYgsHMksBXgfOAlcClZrZyxGaXATvd/Qjg88AnJ6ue3oFBdvUOqCtJRGSEOI8YTgTWuft6dx8AbgEuGrHNRcC3wukfAWeZTc5H93fueorBvJNOqjdNRKRQKsbXWghsKphvA04aaxt3HzSzTmAO0FG4kZldDlweznab2doDrKnxQ9DxoQP85WmqkRHvZwlQm0uD2jwxi8daEWcwHDTufgNww7Pdj5m1unvLQShp2lCbS4PaXBomq81x9qNsBhYVzDeHy0bdxsxSQD3wTCzViYgIEG8w3AcsM7OlZpYGLgFWjdhmFfD6cPoVwO3uunZIRCROsXUlhecM3gbcBiSBG919jZl9GGh191XAN4DvmNk6YAdBeEymZ90dNQ2pzaVBbS4Nk9Jm0xdyEREppGs1RUQkQsEgIiIRJRsM+xueYzoxsxvNbLuZPVKwbLaZ/dbM/h7+Oytcbmb2pbDdq83s+ILfeX24/d/N7PWjvdZUYGaLzOwOM3vUzNaY2TvD5TO5zRVmdq+ZPRy2+UPh8qXh8DHrwuFk0uHyMYeXMbOrwuVrzezFxWnR+JlZ0sweNLNfhPMzus1mtsHM/mpmD5lZa7gs3r9tdy+5H4KT308AhwFp4GFgZbHrehbteQFwPPBIwbJPAVeG01cCnwynzwd+RTB64MnAPeHy2cD68N9Z4fSsYrdtjPYuAI4Pp2uBvxEMszKT22xATThdBtwTtuUHwCXh8uuBfwun3wJcH05fAnw/nF4Z/r2XA0vD/wfJYrdvP21/N/BfwC/C+RndZmAD0DhiWax/26V6xDCe4TmmDXf/I8FVXIUKhxf5FvC/CpZ/2wN3Aw1mtgB4MfBbd9/h7juB3wLnTn71E+fuW9z9gXC6C3iM4K75mdxmd/fucLYs/HHgTILhY2DvNo82vMxFwC3unnH3J4F1BP8fpiQzawZeAnw9nDdmeJvHEOvfdqkGw2jDcywsUi2TZZ67bwmntwLzwumx2j4t35Owu+A4gm/QM7rNYZfKQ8B2gv/oTwC73H0w3KSw/sjwMsDQ8DLTqs3AF4D3Avlwfg4zv80O/MbM7rdg+B+I+W97Wg6JIRPj7m5mM+66ZDOrAX4MvMvdd1vBeIszsc3ungOONbMG4KfA8iKXNKnM7AJgu7vfb2ZnFLueGJ3q7pvNbC7wWzN7vHBlHH/bpXrEMJ7hOaa7beEhJeG/28PlY7V9Wr0nZlZGEArfc/efhItndJuHuPsu4A7g+QRdB0Nf8ArrH2t4menU5lOAC81sA0F375nAF5nZbcbdN4f/bif4AnAiMf9tl2owjGd4jumucHiR1wM/L1j+z+HVDCcDneEh6m3AOWY2K7zi4Zxw2ZQT9ht/A3jM3T9XsGomt7kpPFLAzCqBFxGcW7mDYPgY2LvNow0vswq4JLyCZymwDLg3nlZMjLtf5e7N7r6E4P/o7e7+GmZwm82s2sxqh6YJ/iYfIe6/7WKfgS/WD8HZ/L8R9NNeXex6nmVbbga2AFmCvsTLCPpWfw/8HfgdMDvc1ggemPQE8FegpWA//0JwYm4d8MZit2sf7T2VoB92NfBQ+HP+DG/zMcCDYZsfAa4Jlx9G8CG3DvghUB4urwjn14XrDyvY19Xhe7EWOK/YbRtn+89gz1VJM7bNYdseDn/WDH02xf23rSExREQkolS7kkREZAwKBhERiVAwiIhIhIJBREQiFAwiIhKhYBDZDzNbYmZuZlPqQfNTtS6Z/hQMIvu3iWBE14cAzOyM8AO5Ma4CzOxOM/vKvuoSOVg0VpLIfngwRtHWydi3mZW5e/ZAfncy65LSpiMGmVLCb8bXmdnHzazDggcQfcbMEuH6DWb2nlF+5ysF8xvM7Bozu8nMusxsk5ldbGYNZnaLmXWHDy85Z5w1DXfZhKO53hGuag+X3xRuZ2b2XjN7wsz6LHjYymtH2c+lZna7mfUB/9vM5pjZzWbWFv7eGjN7Y8Hv3QScDrw1/H0P97VXV5KZvcCCh9T0m9k2M/t8OOzLuN7fcJuXWfDQlz4z22FmfzCzodE8pQQoGGQqeg0wCPwj8DbgXcDFE9zHuwiGRTie4MEu3yJ42MutwLHAH4HvmlnFBPe7CXh5OP0cgq6cd4bzHyUYjuStBA+HuRb4mpm9ZMQ+rgWuC7f5GcFQDg8AF4T7/GL4e2eF278TuAv4Zvh6C4gOqQyAmS0keGjLgwRDkV8GXBq+XqEx318zm08wYN23gBUED4H6zjjeF5lJij02iH70U/gD3AncNWLZb4Gvh9MbgPeM8jtfKZjfANxcMF9DMLbSlwqWLQmXtYyjpsi2BOP2OAVP2QKqgT7gtBG/+wXg1hH7uWIcr3nLUJtHa+MYdX2MYCydRME2bwAyQNU439/jw30uLvbfgn6K96NzDDIVrR4x/zQw90D34e7dZtZLMMjYkG3hvxPd71hWEnzz/7VFx8ovIwiqQq2FM2aWJHhc48UED1MpJ3jk7J0TrGEFcLe75wuW/Snc1xHseU/29f4+TDBI2yNm9ptw+kfu3j7BWmQaUzDIVDTyZKyzp9szTzCiZKGyce4jO2IeDl536tB+Xgps3E8tPSPm3wNcQdBl9FegG/g4By+0YE97R6tn+P1191x47uVkgqGaLwOuNbPT3f3hg1iPTGEKBplu2gn62AEIzxEsJ+hXj8tA+G+yYNmjBF02i9399gnu71Tgv939OzD8vGx/QUoAAAFjSURBVIkjgV0jXjM5yu8Wegx4lZklCo4aTg1/94nxFuPuTnBO4y4z+zDB8M8XExxNSAlQMMh0czvwL2a2iiAkrib+v+OnCL5lv8TM/hvoc/cuM/sM8Jnwg/2PBOc2Tgby7n7DPvb3N+BiMzsV6ADeDiwlGnYbgBPDq6K6gR2j7Oc6ghPJ15nZFwnG9v8EwbmJ3vE0LHzYy9kED3XZRnASexFB8EmJ0FVJMt1cSxAOPwd+Q9CHHufRAh48evEDBCd7twFDl8q+H/ggQdfQGoKTui8HntzPLj9KcAXVrwgCpQf43ohtPkPwzf9RgkA8dIy6ziP4MH8IuJHgIU7vm0DzOgkeqfkLghPZnwU+4u7fncA+ZJrTg3pERCRCRwwiIhKhYJCSZ2bXh3dDj/ZzfbHrE4mbupKk5JnZXKBujNW73X17nPWIFJuCQUREItSVJCIiEQoGERGJUDCIiEiEgkFERCL+B86EkVwp3qEaAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"UIhoP-rjlK0G","executionInfo":{"status":"ok","timestamp":1634748136550,"user_tz":-330,"elapsed":6353,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"11dc0681-d65d-4e14-b8a9-56a48ec7c5ed"},"source":["# Instantiate learner in VW but without -q\n","vw = pyvw.vw(\"--cb_explore_adf --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, context1, context2, items, get_cost)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxcZZ3v8c+vtt6TztIJZIEECEtAhNwmMrKOOBBgBO8wKlwVQV4yd644eNXxwvWKM6DirjDCAOPG6AyIyzjRQYEBFOWCEAhbQgJJCCQhS2frTnqtrvrNH+d0p06nOumG7lPdOd/361Wv1DnnqVPPU6mub53nOfUcc3dERET6pCpdARERGVsUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhGxBYOZfc/MtpjZC4NsNzO72cxWmdlzZrYgrrqJiMgecR4x/ABYtI/t5wLzwtuVwD/GUCcRERkgtmBw90eA7fsociHwzx54HGg0s4PjqZ2IiPTJVLoCJWYC60qW14frNg4saGZXEhxVUFdX99+OPvroYT9Zy65uNrV1ceyMCaTM3liNRUTGqaeeemqruzeV2zaWgmHI3P0O4A6A5uZmX7JkybD3cfvvVnPjr1fwh+vPoTY3Ll8GEZE3zMxeHWzbWDoraQMwu2R5VrhORERiNJaCYTFwaXh20slAq7vv1Y0kIiKjK7Y+FDO7CzgTmGpm64HPAVkAd78NuBc4D1gFdACXx1U3ERHZI7ZgcPdL9rPdgY/GVB0RERnEWOpKqghdjkJEJCqxwaAzVEVEyktsMIiISHkKBhERiVAwiIhIhIJBREQiFAwiIhKR+GDQ2aoiIlGJDQZD56uKiJST2GAQEZHyFAwiIhKhYBARkQgFg4iIRCgYREQkIvHB4JpeVUQkIrHBoNlVRUTKS2wwiIhIeQoGERGJUDCIiEiEgkFERCIUDCIiEpH4YNDJqiIiUYkPBhERiVIwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCQ+GDS5qohIVGKDwTS9qohIWYkNBhERKU/BICIiEQoGERGJUDCIiEiEgkFnJYmIRMQaDGa2yMxWmtkqM7umzPZDzOxhM1tqZs+Z2XmjVpfR2rGIyDgXWzCYWRq4BTgXmA9cYmbzBxT7f8A97n4icDFwa1z1ExGRQJxHDAuBVe6+xt17gLuBCweUcWBCeH8i8HqM9RMREeINhpnAupLl9eG6Un8HfMDM1gP3Ah8rtyMzu9LMlpjZkpaWltGoq4hIYo21wedLgB+4+yzgPOCHZrZXHd39Dndvdvfmpqam2CspInIgizMYNgCzS5ZnhetKXQHcA+DujwHVwNRYaiciIkC8wfAkMM/M5ppZjmBwefGAMq8BZwGY2TEEwTCqfUWu81VFRCJiCwZ37wWuAu4DXiQ4+2iZmV1vZheExT4JfMTMngXuAi5zH535TzWHnohIeZk4n8zd7yUYVC5dd13J/eXAKXHWSUREosba4LOIiFSYgkFERCIUDCIiEqFgEBGRiMQHg675LCISldhg0NmqIiLlJTYYRESkPAWDiIhEKBhERCRCwSAiIhEKBhERiUh8MOhsVRGRqMQGg2l6VRGRshIbDCIiUp6CQUREIhQMIiISoWAQEZEIBYOIiEQkPhhG6ZLSIiLjVmKDQWerioiUl9hgEBGR8hQMIiISkdhg+M8XtwCwfkdnhWsiIjK2JDYYHnmpBYDnN7RWuCYiImNLYoOhj85JEhGJSnwwoNNVRUQiEh8MigURkajEB4OIiEQpGEREJELBICIiEYkPBo09i4hEKRiUDCIiEQqGSldARGSMSXwwiIhIVKzBYGaLzGylma0ys2sGKfNeM1tuZsvM7F/jrJ+IiEAmricyszRwC/BnwHrgSTNb7O7LS8rMA64FTnH3HWY2bbTrpSEGEZGoOI8YFgKr3H2Nu/cAdwMXDijzEeAWd98B4O5bYqyfiIgQbzDMBNaVLK8P15U6EjjSzB41s8fNbFG5HZnZlWa2xMyWtLS0jFJ1RUSSaawNPmeAecCZwCXAP5lZ48BC7n6Huze7e3NTU9ObekL1JImIRMUZDBuA2SXLs8J1pdYDi9097+6vAC8RBMWo0e8YRESi4gyGJ4F5ZjbXzHLAxcDiAWV+QXC0gJlNJehaWhNjHUVEEi+2YHD3XuAq4D7gReAed19mZteb2QVhsfuAbWa2HHgY+Ft33xZXHUVEJMbTVQHc/V7g3gHrriu578AnwpuIiFTAWBt8jp2GGEREohIfDCIiEqVgEBGRiMQHg+uXDCIiEQoG5YKISISCodIVEBEZYxIfDCIiEqVgEBGRCAWDiIhEDCkYzCxrZpvM7NjRrlDcNPgsIhI1pGBw9zyQ5wAcq9XpqiIiUcPpSvoH4Fozi3V+JRERiddwPuRPA84ANpjZC0B76UZ3v6Dso8Y4dSWJiEQNJxi2Aj8brYqIiMjYMORgcPfLR7MiIiIyNgx5jMHMHip3/WUzm2BmD41stUREpFKGM/h8JpArs76aYPxhXLn5khMBOG7mxArXRERkbNlvV5KZLShZPN7Mtpcsp4FzgA0jXbHRNrOxutJVEBEZk4YyxrCE4PcLDtxfZnsn8LGRrFQ8rNIVEBEZk4YSDHMJPkXXACcRnJ3UpwfY4u6FUahbLFznq4qIROw3GNz9VQAz+xJwkrvfVrrdzP6nmc1098+OUh1HhYUHDIoFEZGo4Qw+fwBYWmb908ClI1Od+PR3JCkZREQihhMM04CWMuu3AtNHpjrxMdMYg4hIOcMJhtcof1rq6cD6kalO/DSJnohI1HCmxLgd+KaZ5YC+H7SdBdwIfHmkKzba+o4XNPYsIhI1nCkxvm5mU4Gb2fNDtx7gJnf/ymhUbjT1Dz4rGEREIoY1hba7X2tmnwfmh6tedPfdI1+t0Wf6HYOISFnDvraCu7cDT45CXSpCBwwiIlGJvebznq4kRYOISKnEBkMfxYKISFRig0GDzyIi5SU3GDT4LCJSVmKDYQ8dMoiIlEpsMKgrSUSkPAVDZashIjLmxBoMZrbIzFaa2Sozu2Yf5S4yMzez5lGri8YYRETKii0YzCwN3AKcS/DL6UvMbH6Zcg3A1cAf46iXupJERKLiPGJYCKxy9zXu3gPcDVxYptwNBJPydY1mZfZ0JSkZRERKxRkMM4F1Jcvrw3X9zGwBMNvd/2NfOzKzK81siZktaWkpd4mI/dPsqiIi5Y2ZwWczSwHfAD65v7Lufoe7N7t7c1NT0xt8vjf0MBGRA16cwbABmF2yPCtc16cBOA74rZmtBU4GFo/mADTorCQRkYHiDIYngXlmNje82M/FwOK+je7e6u5T3X2Ou88BHgcucPclo1Md63ve0dm9iMg4FVswuHsvcBVwH/AicI+7LzOz683sgrjq0UddSSIi5Q37egxvhrvfC9w7YN11g5Q9czTrolwQESlvzAw+V4p6kkREohIbDBb2Jel3DCIiUckNhvBfHTGIiEQlNxg0yCAiUlZig6GPjhhERKISGwx9s6sqF0REopIbDP0X6lE0iIiUSmwwiIhIeYkPBh0viIhEJTYY+s9KUjKIiEQkOBj0AzcRkXKSGwzhvxp7FhGJSm4w6AduIiJlJTYY+uiAYfzpLRTZ1NrVf6rxxtZOnn5tB1t3d1e4ZiIHhlin3R5L+n/gpmR4Q3Z29NBYm2NHew+ZtNFQnd2rTKHo3P7Iah5esYV/urSZiTVZ3CGVMu5ftokv/WYF0xqq+Mhph/Efz2/k1COmcuiUWtbv6GR1SzuvbWtnSn0Vz69vpa0rz/QJ1aRTxvLX29jU1sVhTXVMb6jmsTXb+p9z3rR6JtRk2bCjk+kTqpg2oZotu7o5dHItJ8xuZFt7N71FJ5dO0dNbpK2rFzMoFp2aXJqGqgxtXb1UZVJUZdMAVGVSrNi0i/buXhYc0shRB02g+dBJFN154fU2GmuyzJlSRy6TojqbYv2OTtZt72DmpBom1mRpqM6STukQVcaP5AZD3w/cEnbMUCw6qZTR3t3Lkld3MK2hijlT6simjWfXt9JYm+Vd//AHCkWnoTrD7R9s5rLvP0FXvkC+MLTXqqmhipZd0W/vJ1z/QNmya1raeXzNdgB+/vSGsmVSBkdMq+fRVVvpLTrplLHo2IN4cVMbT7+2gzOPamLBIZPYuru7/8jh4MZqdnbkeXb9ZgBefL2Nxc++PtSXaS+5dIqG6gwPrdgy7MemU8ZBE6qZPbmGGY01zJ5USyZl1Fdn2N7eQ77gdPcWyBeKnDRnMsfPamRGYzVVmfQbrm+lFIvOprYuJtZkyaZT9BSK1FdlcHdadnezastuMqkUu7ryZNMpsukU6ZSxuztPR0+BzW3dVGWC17qtqxeALW1dbNjRSX11hq58AcOYUp8jkzKm1FcxuS7Huh0dpMxIGRQ9+P8qurNuewdmxozGauZNb+CYgyYwtT5HJl2+s6QrX6Cjp0AmbdTlMqRT1n9kagP6n7t7C6zd2kG+UKRQdDa2dtLRU6BQdNZua6c6k6axNktHT4HlG9vo7CkwtaGKhqoMuUyKQtEpFJ22rjz5grO5rYvd3b30FpzqbAp3qK3KkEkZk2pzTKzJUnRnR0cP+UKRmmyGixbM5O1HTB3x/8fkBkOlKxDa1NrF2m3tvGXmROqq9vx37GjvYcPOTl7Z2s7H7lrKTRefwJlHTaM2lyYbvql7eot8++FV3PzgywAcNrWONVvbAbj+wmN5/9sOJWXw1ftWcutvVw+7bt27e7joH///sB9XGgrHzpjAWcdM57u/X0N7TyFS7n+/80hqc2l+vnQDk2qzXLzwENydmmya42ZOZEZjDbu68tTlMqTCP9B8wenqLTChOkuh6PT0FqnJDf4B6h788aVTxopNu2iszdJYk6O1M8+Emgw14VFB0YMjnK7eAsWiYxi7uvOYGdl08IeZSRmb27p5YUMrKzfvAuDwpnp2d/fy+s5O2nt66c4XmVCT5dDJtXT09NLW1cv29h7W7+hg7dYOnlm3k658sWxds2njR4+/BgQfbHOn1rG7u5cp9bn+D5nu3iITa7LUZNNs7+hhc1sX7sGH1IzGGlo78/QWnPqqDIdMrqUqm+Kogxpo7cxTnUkzd2od29t72N7ew7b2Hja2dlJXlaGtM0/RnZmNNWTSKeqrMlRlUrR25tnY2kVnT4FcJsWm1i7yhSIL5wYBlk0bHT0FtoYf+o+83MK67Z2Rdk2sydLTW6QzXyjX7CGZ1lBFR0+BbNpIp1JD7jasy6VxoKPkvZdLp5hSn6OtM09NLk1VJk1bV56e3iI9hWJ/L0I6FfzfFz34W6vJpplUm6Xo0NaVpzNfGLTHoS+g+kytr2JKXY4n1m6noyf4EgCQNqOxNksmFdRpYk2WqpoUu7uDI9nWjh4K7ry4sY22zjwAk+tzZNMpunoKnDZv5EMBwMb7lBDNzc2+ZMnwLwu9pa2LhV98kM+/+zg+cPKho1Cz/Vv+ehvn3fz7vda/t3kW9yxZP+jj5kyp5fzjD+aWh4f/YT8U73rrDI45uIGVm3bx0IotXLRgFtm0ceq8Js44sqnsY7a0dfHwyi2cfNgUfv/yVt5x9DRmNNaMSv3Gu7auPGkzXtnazuzJtdRXZdjd1UtDdYanXtvBsg2trN3WwfMbWmmoztDamWdXVy/t3b1k0kZbZ3C/sTbHwROrSaWMmmyKLW3dNFRnwIzeQpHXd3bSmS8MGkTplNFUX0V7dy+T63Pke4u83tq1V7ls2pjRWENvwZlSnyOdMp5b30qh6HuVO3H2JM4+djr5gtPamSebNrbu7iaXTjFrUi1HHdRAoejU5tIUwqDv7CkwJfywm9FYTW/42Cl1ORyYUrf3N/x8oUjRnZ0deba39zCjsQYz6C04abNw30WmT6jGPdjfik27WLGxjY2tXbTs6qauKkNv0enKF2iozpBLp6ivzjChOku+UGRnZ56ufIFMyqjOpunoKbCzI49ZEHZ1YfjWV6VJhx/sk2pzpAxmNNZQDJ+3Kp2moTr4ctPH3XEPei4GHonExcyecvfmstsSGwy7ulj4hQe54d3H8cGYg6G3UGTpup2857bHRmR/n150FBt2dPKbFzZx+SlzKDp844GXImW+c2kz75w/nXz4jSiXSfx5B4lQLDqvbGtnUm2OfKHIS5t30dRQxdT6qrDLJjo25O7BN+LOPLu7e2lqqAKgOhs9KmvryvNKSzsFD8ZrJtZkOWhidf/RrIx9+wqGxHYlpcOULhb3H4zbdnfjBIeDq1t209lT4LiZE8uW/d1LLWzc2cnFCw8pu723UOSGXy3nzsde7V+3+ovnsasrzy+WbuDR1dt4YPlm/vrMw/k/i46OPLZYdHoKRa6+eyn3LdvM5941n8tPmdu//Qv//S399//mrHm8tHkXy19v48ITZvR/K9EfbrKkUsbhTfX9y9MnVO+zvJmRNphUl2NSXW7QchOqs7x1duOI1VPGluQGQ3hYV3o4XCg6j6/ZxtvmTu4/dH3ile289/bgm/3RBzWwYlPQt/ybj5/G0QdNiOzzZ0+t55M/eRaAw6fVc9Kcyf3bmj//AFt39+xVj1duPA8zo7E2x2WnzOWykg/6gVIpozqV5vYPlg35vRw5vYEjpzcMqayISJ/EBkNff18x7EpbsnY7fxl27fzVGYdx+rwm3v+dP0Ye0xcKAIu+9Xtqsmne2zwr8u2/z3tue4wnPnMW0xqq+f6jr+wVCheeMIObLj5xRNskIjISEhsMfV1Jy19vA+DxknPhl766k9t/t2a/++jMF8qGQn1Vht3dvSz8woOsuGERf//L5QCcf/zB5NIpvv6et0YGokRExpLkBkP4wfzzpRs4/cgmanJ7XoqdnXt3+fT5zqXNPLp6K99/dG3Z7Wu+eB5dvQXmX3cfAEd/9jf92275HwtGoOYiIqMrsSORqZJTxD7+42eoLTkXfmdHnsOb6iLl/+YdRwDwtsMm87l3HcsFb53Bnx6159TN5defw9ovnU8qZdTmMjx73dmRxz9+7Vmj0QwRkRGX+COGPqVLW3Z1syX8kdYlCw/hwhNmcPJhU/jE2Uf1l7n5kn2PD0yszfLKjefxtftXMmdKHQdN3PfZICIiY0Vig2FgF3/XIL/KvPEv3lJ2/VCYGX97ztH7LygiMoYktitp4K8N/23p3vP0nD7Ir3xFRA5kiQ2GgZ5d37rXujsvP6kCNRERqSwFwwC/vvq0/vuVmsNERKSSFAwDHHNw8Gvmy94+p7IVERGpkMQOPpdz0pxJAKz90vkVromISOXoiKHEk2t3VLoKIiIVp2AQEZGIWIPBzBaZ2UozW2Vm15TZ/gkzW25mz5nZg2YW64USPvQnlblgj4jIWBJbMJhZGrgFOBeYD1xiZvMHFFsKNLv78cBPga/EVT+g/6IkIiJJFucRw0Jglbuvcfce4G7gwtIC7v6wu3eEi48Ds2KsHz9esi7OpxMRGZPiDIaZQOkn7/pw3WCuAH5dboOZXWlmS8xsSUtLy4hVcOBFzEVEkmhMDj6b2QeAZuCr5ba7+x3u3uzuzU1NIzdtxafOPnLE9iUiMl7F+TuGDcDskuVZ4boIM3sn8BngDHfvjqluABw7o/x1nEVEkiTOI4YngXlmNtfMcsDFwOLSAmZ2InA7cIG7b4mxbrz98Cn86dHT4nxKEZExKbZgcPde4CrgPuBF4B53X2Zm15vZBWGxrwL1wE/M7BkzWzzI7kbcyYdNieupRETGtFinxHD3e4F7B6y7ruT+O+OsT6nBrscgIpI0Y3LwuRI6FQwiIoCCoZ+OGEREAokOhhU3LOLLFwWX7uzsUTCIiEDCg6E6m2ZiTRaADgWDiAiQ8GAAOGnOZMzgw6fOrXRVRETGhMRfqGdKfRWv3KgL84iI9En8EYOIiEQpGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEhErMFgZovMbKWZrTKza8psrzKzH4fb/2hmc+Ksn4iIxBgMZpYGbgHOBeYDl5jZ/AHFrgB2uPsRwDeBL8dVPxERCcR5xLAQWOXua9y9B7gbuHBAmQuBO8P7PwXOMjOLsY4iIomXifG5ZgLrSpbXA28brIy795pZKzAF2FpayMyuBK4MF3eb2co3WKepA/edAGpzMqjNyfBm2nzoYBviDIYR4+53AHe82f2Y2RJ3bx6BKo0banMyqM3JMFptjrMraQMwu2R5VriubBkzywATgW2x1E5ERIB4g+FJYJ6ZzTWzHHAxsHhAmcXAh8L7fwk85O4eYx1FRBIvtq6kcMzgKuA+IA18z92Xmdn1wBJ3Xwx8F/ihma0CthOEx2h6091R45DanAxqczKMSptNX8hFRKSUfvksIiIRCgYREYlIbDDsb3qO8cTMvmdmW8zshZJ1k83sATN7Ofx3UrjezOzmsN3PmdmCksd8KCz/spl9qNxzjQVmNtvMHjaz5Wa2zMyuDtcfyG2uNrMnzOzZsM1/H66fG04fsyqcTiYXrh90ehkzuzZcv9LMzqlMi4bOzNJmttTMfhUuH9BtNrO1Zva8mT1jZkvCdfG+t909cTeCwe/VwGFADngWmF/per2J9pwOLABeKFn3FeCa8P41wJfD++cBvwYMOBn4Y7h+MrAm/HdSeH9Spds2SHsPBhaE9xuAlwimWTmQ22xAfXg/C/wxbMs9wMXh+tuAvw7v/y/gtvD+xcCPw/vzw/d7FTA3/DtIV7p9+2n7J4B/BX4VLh/QbQbWAlMHrIv1vZ3UI4ahTM8xbrj7IwRncZUqnV7kTuDdJev/2QOPA41mdjBwDvCAu2939x3AA8Ci0a/98Ln7Rnd/Ory/C3iR4FfzB3Kb3d13h4vZ8ObAOwimj4G921xuepkLgbvdvdvdXwFWEfw9jElmNgs4H/hOuGwc4G0eRKzv7aQGQ7npOWZWqC6jZbq7bwzvbwKmh/cHa/u4fE3C7oITCb5BH9BtDrtUngG2EPyhrwZ2untvWKS0/pHpZYC+6WXGVZuBbwGfBorh8hQO/DY7cL+ZPWXB9D8Q83t7XE6JIcPj7m5mB9x5yWZWD/wM+Li7t1nJfIsHYpvdvQCcYGaNwL8BR1e4SqPKzP4c2OLuT5nZmZWuT4xOdfcNZjYNeMDMVpRujOO9ndQjhqFMzzHebQ4PKQn/3RKuH6zt4+o1MbMsQSj8i7v/PFx9QLe5j7vvBB4G/oSg66DvC15p/QebXmY8tfkU4AIzW0vQ3fsO4CYO7Dbj7hvCf7cQfAFYSMzv7aQGw1Cm5xjvSqcX+RDw7yXrLw3PZjgZaA0PUe8DzjazSeEZD2eH68acsN/4u8CL7v6Nkk0HcpubwiMFzKwG+DOCsZWHCaaPgb3bXG56mcXAxeEZPHOBecAT8bRieNz9Wnef5e5zCP5GH3L393MAt9nM6sysoe8+wXvyBeJ+b1d6BL5SN4LR/JcI+mk/U+n6vMm23AVsBPIEfYlXEPStPgi8DPwnMDksawQXTFoNPA80l+znwwQDc6uAyyvdrn2091SCftjngGfC23kHeJuPB5aGbX4BuC5cfxjBh9wq4CdAVbi+OlxeFW4/rGRfnwlfi5XAuZVu2xDbfyZ7zko6YNsctu3Z8Las77Mp7ve2psQQEZGIpHYliYjIIBQMIiISoWAQEZEIBYOIiEQoGEREJELBILIfZjbHzNzMxtSF5sdqvWT8UzCI7N86ghldnwEwszPDD+SpcVXAzH5rZt/eV71ERormShLZDw/mKNo0Gvs2s6y759/IY0ezXpJsOmKQMSX8ZnyrmX3RzLZacAGir5lZKty+1sw+VeYx3y5ZXmtm15nZD8xsl5mtM7P3mVmjmd1tZrvDi5ecPcQ69XfZhLO5PhxuagnX/yAsZ2b2aTNbbWadFlxs5QNl9nOJmT1kZp3AX5nZFDO7y8zWh49bZmaXlzzuB8AZwEfDx3u4r726kszsdAsuUtNlZpvN7JvhtC9Den3DMn9hwUVfOs1su5n9zsz6ZvOUBFAwyFj0fqAXeDtwFfBx4H3D3MfHCaZFWEBwYZc7CS72ci9wAvAI8CMzqx7mftcBF4X3jyXoyrk6XP48wXQkHyW4OMyNwO1mdv6AfdwI3BqW+QXBVA5PA38e7vOm8HFnheWvBh4Dvh8+38FEp1QGwMxmEly0ZSnBVORXAJeEz1dq0NfXzA4imLDuTuAYgotA/XAIr4scSCo9N4huupXegN8Cjw1Y9wDwnfD+WuBTZR7z7ZLltcBdJcv1BHMr3Vyybk64rnkIdYqUJZi3xym5yhZQB3QCpw147LeAewfs55NDeM67+9pcro2D1OsLBHPppErKXAZ0A7VDfH0XhPs8tNLvBd0qd9MYg4xFzw1Yfh2Y9kb34e67zayDYJKxPpvDf4e738HMJ/jm/xuLzpWfJQiqUktKF8wsTXC5xvcRXEyliuCSs78dZh2OAR5392LJuj+E+zqCPa/Jvl7fZwkmaXvBzO4P7//U3VuGWRcZxxQMMhYNHIx19nR7FglmlCyVHeI+8gOWYeS6U/v28y7gtf3UpX3A8qeATxJ0GT0P7Aa+yMiFFuxpb7n69L++7l4Ix15OJpiq+QrgRjM7w92fHcH6yBimYJDxpoWgjx2AcIzgaIJ+9bj0hP+mS9YtJ+iyOdTdHxrm/k4FfunuP4T+600cCewc8JzpMo8t9SLwXjNLlRw1nBo+dvVQK+PuTjCm8ZiZXU8w/fP7CI4mJAEUDDLePAR82MwWE4TEZ4j/ffwqwbfs883sl0Cnu+8ys68BXws/2B8hGNs4GSi6+x372N9LwPvM7FRgK/AxYC7RsFsLLAzPitoNbC+zn1sJBpJvNbObCOb2/xLB2ETHUBoWXuzlnQQXddlMMIg9myD4JCF0VpKMNzcShMO/A/cT9KHHebSAB5de/BzBYO9moO9U2c8Cf0fQNbSMYFD3IuCV/ezy8wRnUP2aIFDagX8ZUOZrBN/8lxME4iGD1OtcggTha0cAAAB1SURBVA/zZ4DvEVzE6f8Oo3mtBJfU/BXBQPbXgRvc/UfD2IeMc7pQj4iIROiIQUREIhQMknhmdlv4a+hyt9sqXT+RuKkrSRLPzKYBEwbZ3ObuW+Ksj0ilKRhERCRCXUkiIhKhYBARkQgFg4iIRCgYREQk4r8AIMBqhP+OZI8AAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"11Z97qY4lN90","executionInfo":{"status":"ok","timestamp":1634748145725,"user_tz":-330,"elapsed":2205,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"5999b5bc-b17a-4d65-e1b0-fb2017be1728"},"source":["# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations = 5000\n","ctr = run_simulation(vw, num_iterations, users, context1, context2, items, get_cost, do_learn=False)\n","\n","plot_ctr(num_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfmklEQVR4nO3de3hddZ3v8fd3X3Nrmja90NLSFihCQUSMDCACAmopCp6jDnTkAZEjM0dhZNTxwDAyM3hBPDoqCodhRgZFBe9aHARRcBzmoUi4U6ASSqEthbZpmzbXffueP9ZKule60yY0WUm6P6/nydO9Lnvl+8uzuz97/X5r/5a5OyIiIv0S412AiIhMLAoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRiNiCwcxuMbNNZvb0ENvNzK43szYze9LMjo2rNhER2SXOM4ZbgaV72H4msDj8uQT4fzHUJCIig8QWDO7+B2DrHnY5B/iuB1YCTWY2J57qRESkX2q8CyhzILCubHl9uG7j4B3N7BKCswrq6+vfcvjhh8dSoIjI/uKRRx7Z4u4zK22bSMEwbO5+M3AzQEtLi7e2to5zRSIik4uZvTTUtol0VdIGYH7Z8rxwnYiIxGgiBcMK4ILw6qTjgQ53360bSURExlZsXUlmdjtwKjDDzNYD/wCkAdz9JuAuYBnQBnQDF8VVm4iI7BJbMLj78r1sd+DjMZUjIiJDmEhdSSIiMgEoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhGxBoOZLTWz1WbWZmZXVNh+kJndb2aPmdmTZrYszvpERCTGYDCzJHADcCawBFhuZksG7fb3wI/c/c3AecCNcdUnIiKBOM8YjgPa3H2Nu+eAO4BzBu3jQGP4eCrwSoz1iYgI8QbDgcC6suX14bpy/wicb2brgbuAyyodyMwuMbNWM2vdvHnzWNQqIlK1Jtrg83LgVnefBywDbjOz3Wp095vdvcXdW2bOnBl7kSIi+7M4g2EDML9seV64rtzFwI8A3P1BoAaYEUt1IiICxBsMDwOLzWyRmWUIBpdXDNrnZeB0ADM7giAY1FckIhKj2ILB3QvApcA9wLMEVx+tMrNrzOzscLdPAR81syeA24EPu7vHVaOIiEAqzl/m7ncRDCqXr7u67PEzwNvirElERKIm2uCziIiMMwWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRMQaDGa21MxWm1mbmV0xxD5/bmbPmNkqM/tBnPWJiAik4vpFZpYEbgDeCawHHjazFe7+TNk+i4Ergbe5+zYzmxVXfSIiEojzjOE4oM3d17h7DrgDOGfQPh8FbnD3bQDuvinG+kREhHiD4UBgXdny+nBducOAw8zsv81spZktrXQgM7vEzFrNrHXz5s1jVK6ISHWaaIPPKWAxcCqwHPhXM2savJO73+zuLe7eMnPmzJhLFBHZv8UZDBuA+WXL88J15dYDK9w97+4vAn8iCAoREYlJnMHwMLDYzBaZWQY4D1gxaJ9fEJwtYGYzCLqW1sRYo4hI1YstGNy9AFwK3AM8C/zI3VeZ2TVmdna42z1Au5k9A9wP/K27t8dVo4iIgLn7eNewT1paWry1tXW8yxARmVTM7BF3b6m0baINPouIyDhTMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJGFYwmFnazF41syPHuiARERlfwwoGd88DeWByz58hIiJ7NZKupG8CV5pZbLcDFRGR+I3kTf7twCnABjN7Gugq3+juZ1d8loiITCojCYYtwE/HqhAREZkYhh0M7n7RWBYiIiITw7DHGMzsvkr3XzazRjO7b3TLGnvPvLKDi/79j+QKpfEuRURkQhnJ4POpQKbC+hqC8YdJ5YqfPcn9qzfz7MYd412KiMiEsteuJDM7tmzxaDPbWracBN4NbBjtwkREZHwMZ4yhleD7Cw78psL2HuCy0SwqTvpihohI1HCCYRFgwBrgrQRXJ/XLAZvcvTgGtY0pG+8CREQmqL0Gg7u/BGBmXwLe6u43lW83s78yswPd/bNjVOOYctc5g4hIuZEMPp8PPFZh/aPABaNTToxM5wwiIpWMJBhmAZsrrN8CzB6dcmKkMwURkYpGEgwvU/my1JOB9aNTjoiIjLeRTInxL8DXzCwD9H+h7XTgWuC60S5szKkrSUSkopFMifFVM5sBXM+uL7rlgG+4+5fHorg4qENJRCRqRFNou/uVZvZ5YEm46ll37xz9ssaezhdERCob8b0V3L0LeHgMahERkQmg6u/5rIuTRESiqj4YREQkSsEgIiIRVRsMulpVRKSyqg2GXTTIICJSrmqDQScMIiKVVW0wiIhIZbEGg5ktNbPVZtZmZlfsYb/3m5mbWctY16TLVUVEomILBjNLAjcAZxJ8c3q5mS2psN8U4BPAQ3HVJiIiu8R5xnAc0Obua9w9B9wBnFNhv88RTMrXG2NtIiISijMYDgTWlS2vD9cNMLNjgfnu/h97OpCZXWJmrWbWunlzpVtE7J3pelURkYomzOCzmSWAfwY+tbd93f1md29x95aZM2eOfXEiIlUkzmDYAMwvW54Xrus3BTgK+L2ZrQWOB1aM9QC0xp5FRKLiDIaHgcVmtii82c95wIr+je7e4e4z3H2huy8EVgJnu3vrWBSjjiQRkcpiCwZ3LwCXAvcAzwI/cvdVZnaNmZ0dVx271zVev1lEZGIa8f0Y9oW73wXcNWjd1UPse2ocNYmISNSEGXwWEZGJoWqDQVeriohUVrXBICIilVV9MLhGn0VEIqo2GEwXrIqIVFS1wdBP5wsiIlFVHwwiIhJV9cGgDiURkajqDYYwEUrqSxIRiajaYOg/U3CNMoiIRFRtMCT6v+GmXBARiajaYDB1JYmIVFT1waCuJBGRqOoNhnCUQV98FhGJqt5g0BCDiEhFVRwMQTKUdMogIhJRtcGQ2HW9qoiIlKnaYOjPBZ0xiIhEVW0w9H+PQbkgIhJVtcGgwWcRkcqqNhgYuFxV0SAiUq5qgyGhbz6LiFRUtcEw0JWkMwYRkYiqDYaBwedxrkNEZKKp2mDYNYmeokFEpFwVB0OQDNu68+NciYjIxFK1wdCTKwLw2V88Pc6ViIhMLFUbDOpCEhGprGqDITUwWRKs39Y9jpWIiEwsVRsMA7f2BG5b+dI4ViIiMrFUbTCU5QKForqVRET6VW8wsCsZ8sXSOFYiIjKxVG8wlJ0x5HXGICIyoGqDoVxBZwwiIgNiDQYzW2pmq82szcyuqLD9k2b2jJk9aWa/M7MFY1fLrsfqShIR2SW2YDCzJHADcCawBFhuZksG7fYY0OLuRwM/Ab4cR215TbEqIjIgzjOG44A2d1/j7jngDuCc8h3c/X537/9SwUpg3lgVExl8LuiMQUSkX5zBcCCwrmx5fbhuKBcDv660wcwuMbNWM2vdvHnz66um/HJVnTGIiAxIjXcBlZjZ+UALcEql7e5+M3AzQEtLy+t6Vy/LhUk7xvCDh17m737+1MDyO5fM5hvnHUNdJoW7D0wUOJraNu0knUywoLl+1I8NkCuUuO7u55jRkGXWlCwHz6xnztRaatIJNmzvYUo2zas7eln92k5WvtDO2vYuCkVn8ewGFjTX8aZ5TZxwSDNTatJjUp9INYgzGDYA88uW54XrIszsDOAq4BR37xurYsrfNP/r+S20bdrJobOmjNWvG1V9hSJnXf8AbZs6I+vvfeY1llx9D8cfPJ2Va7YCcOIhzXz4xIU8v6mT/3vPav723W9g+XEHMb0+AwRXZB161a/5iz87iL8+bTGPvryNlWvaWX7cQcydWksiAf/rO62cdfQcrv7lqt1qOWRmPfOn1/HRtx/MCQc3k0gYuUKJkjs16eRe27KjN88ja7fx3QfXkjDjd89tGtHf4tBZDTTWpHjoxa386smNACQTxrxptcybVsuSOY28aX4TtekksxtrqMskaa7PkkjAk+s7WLe1m5JD0Z2uvgILm+uZWpumvauPedPqWNRcT2Nt8N9kc2cfhaKzvTvPzClZmuszJBKjH74i483iuoOZmaWAPwGnEwTCw8BfuPuqsn3eTDDovNTdnx/OcVtaWry1tXXE9Vx2+2Pc+cQrkXUvfHEZ27tzTK/PjMmn7cFWrmnHHR59eRsXn7RoWG+kvfkih3/27oHlz7/vKM4/fgG9+SI/bl3HZyu8eY+2N8yewurXdu51vyPnNnLU3Kk80LaFbd05GmvSHDC1hs07+9jc2UduiLGdD5+4kA+2zOORl7bR3pmjr1Bi/bZuDmyqpSadZG17Fy0LpvGuIw9gdmPNwPP6CkUefWk7D7Rt5rmNO1m3rZsXNndR3Meuwpp00OPam4/WW5tOMq0uTVeuyAGNNdRmkixsruPgmQ0cPW8qh82ewtr2Lh57eTuZZIJXd/QC0J0rsrC5jmPmN7F49hS2dvUxrS5DQ02KhBnp5O49vO5OoeS81N5FZ1+R9s4+tnblmNGQZeaULI01aabWpsGgWHKyqQT12RQdPXkKxRIvbO4CYGNHD7lCCTPj1Y4eevMlzKAmnaS5PqghlUiQTSdIJxIkE8ZBzXU012fIphKYGe5OseSkyurs6MmTK5RoqktH6h+rM1fZd2b2iLu3VNwW560tzWwZ8HUgCdzi7l8ws2uAVndfYWa/Bd4IbAyf8rK7n72nY45mMGRTCfoKJS4/YzGXn3EYpZLjBJ9AR9PTGzp4zzcfqLjtc+87inOOmctpX/lPjpk/ldMOn82X73mO7d15/usz7+DtX75/YN8HrzyNOVNrdzvGnU+8QsKMs46eQ9umTi75bitduQL/ekELz23cyWd++mRk/6a6NN/9yHFccMsf2d6dZ8mcRhprUwNnHW+a38QH3jKPmQ0Zlh41J/Lc3nyRB57fwu+e28TdT2+kqS5DTTqJu7O1K8emncFJXzJhkTfobCpBbSbJKYfNpLk+y7I3HsAx85sws1H9e/fkijy1oYNcocSmnb1s787TnSvQky9yyMwGDphaw+zGGoolZ2ptmpfau+noyTOjIcP6bT28vLWbjR09bO/Oc8ScRuoySabUpOnoyfNSexevdvTSkE2xpStHd1+Bte3dbOmsfKJrBkkzsqkEXeG075VMr8+QMKM3X6Q2k6S7r0DRfbdg2puadGLEz9mTVMKoz6bozhXIF52mujQzGrIkzSIfFGrDDziFUgl3mNGQJVcsMX9aLdPCgMkXHSMIlNpMko6ePDWpJHXZJNPrMsybXkc2lRj4G3R059neHew7oyHDtPoML2/tZu2WLgyjJp1ge0+wz5SaFCV3sqkk2VSC9s4cAI21aTIp49WOXnb2FmioCc4Es6kEzfXZIFSLTtGDYG3IpjCDprrMwHF29uZpqEkxe0oNmVSChBm1mSR9hRKZVILGmhSdfQWKJadUcnryJZrrMzTWpig5bOzoJWlQl00xvS5Dc0OGnb0FuvoKZNMJGmvSTK/PUJdJUZdNUp9JDfx/cHc6evK8tqOPKTUpptdnhvVhcigTJhjGwusNhr++/TFWDAqGftPq0jx45ekDn8zXfumsfaqx3OBxgddrzReX7VM3Rntn8CkVGNPukBe3dJFNJZjbVFsVnx7dna5ckSfXbWf1azuZXp/hhEOacYdpdRnSScPM2NLZxxPrttO2qZPp9Rl29BboyRXCAOsL3xBT9BWK1GdSpJMJ6rNJptVlmDetlqa6DLXpJL2FIlt29rGjNz/wqT2TStCXL7G1OzijMGPgbGv+tDqy6QQGzG6sIRu+uXXlCmzvDo4B0FcoUSiW6CuUWLetm21dObpyRbZ356nLJKlJJ9jWnae9s4980TlybiMzGrJs7coNvDFmUwkc2NLZRzoRjBFt78nRkyuSSQUfHuoySXrzJZobMuQKJTr7CmzryrFxRy/lb039b7p9+RI7+woApJPGwuZ6zKAnXySTTDBnai2dfQXcnXzR6c0XaW4IgrajJ0+uWKKpNk1zQ5bO3gLJhNFXKNLelaNUcpIJI5VIkCuW2Nmbxz0Ir0LJmZJN0VibpjtXiPUGX5lk8CEqVyjRk49+oLjmnCO54ISFr+u4ewqGCTn4HIc9vT9t687zvbIZV0slJ5Ew3vvNBzj/+IM4960HRfb/wn88w8wpWS45+ZCBdb35Is+9upNj5jcNrFv1SsdAKDTVpfn2hS0cM38ayUTwor3ziVf4+7IbB114wgJuW/kS9dkU173/aD72/UcBePaapfv8Zt7ckN2n5w/Xohm7Bqn391CAoI0N2RQnHjqDEw+dMeR+MxqynH7EbE4/YnaM1Q1tSk2aKTXpyCDgeOrNFwc+9ffki9RnkgOvn958EFDT6zNkUvFcWFkolkiYDfy/6ysUyRedkjs9uSLZVIJcocSO3gKphFEX1ptNJ+jozrOjN49hzJlagwOdvQW2dufYtCM442ysTZMvltjZW6C9q4+eXImuvgKdfQV6C0W6+gpkkknmTK1h9tQauvsKtHflOPagaWPS3uoNhr1sL7+RzyMvb+PFLV08taGD//PTp/gfbw6+XvGLxzfwmZ/s6pY56dCZLJnbSKFYGjjb+MtTDuaFTV389tnXBvZ744FTufOykyK/b2ptmvOPX8D5xy/A3XEPPsn/0zlHDewzmmcuIhNZeRdJQza127YDpr7+LpTXIzVo3CfopgoeN5ZdATercffnNla4Qm56fYaDmutGtcbRVLVdSX/zw8f5+WO7XRS1z940bypPrO/Y4z4vXrusKj49i8jEtaeupKqdRO/1vi1X6sH5wFvm8eETFwJEQuHj79jVtXTGEbP47SdP5uGrzlAoiMiEVrVdSeWWHnkAd696FYBfXXZS5IqhGz907EDffjDolmRrV25g+z2Xn8wbDgi+/3Dk3Eb+NuxaWnHp2zh6XhN/c8Zhu52GiohMZNUbDGUf2t9x+EzuXvUqzfUZjpzbyFsXTuPhtdv4/PuOYtkbd12e+dtPnkKh6Lzjq79nxaVv44gDGiODwB9smc8HW6LDdwoFEZlsqjYYmmozA4/f+6a5/HdbO1eceThmxo//6sTIvrde9FZe2d7L3KbgOwMvfHFZrLWKiMSpaoNhblPwjdmffexE6jIprl/+5iH3PfUNs+IqS0Rk3FV9P8ehsxrGuwQRkQml6oNBRESiFAwiIhJR9cGgbxSIiERVfTCIiEiUgkFERCIUDCIiEqFgEBGRCAWDiIhEVG0wTPLZxkVExkzVBkM/TYEtIhJV9cEgIiJRCgYREYlQMIiISISCQUREIhQMIiISUbXB4Oh6VRGRSqo2GPrpYlURkaiqDwYREYlSMIiISISCQUREIhQMIiISoWAQEZGIqg0Gza4qIlJZ1QZDP02uKiISVfXBICIiUQoGERGJiDUYzGypma02szYzu6LC9qyZ/TDc/pCZLYyzPhERiTEYzCwJ3ACcCSwBlpvZkkG7XQxsc/dDga8B18VVn4iIBOI8YzgOaHP3Ne6eA+4Azhm0zznAd8LHPwFON917U0QkVqkYf9eBwLqy5fXAnw21j7sXzKwDaAa2lO9kZpcAl4SLnWa2+nXWNKP+uuixq8AMUJurgNpcHfalzQuG2hBnMIwad78ZuHlfj2Nmre7eMgolTRpqc3VQm6vDWLU5zq6kDcD8suV54bqK+5hZCpgKtMdSnYiIAPEGw8PAYjNbZGYZ4DxgxaB9VgAXho8/ANznru8oi4jEKbaupHDM4FLgHiAJ3OLuq8zsGqDV3VcA3wZuM7M2YCtBeIylfe6OmoTU5uqgNleHMWmz6QO5iIiU0zefRUQkQsEgIiIRVRsMe5ueYzIxs1vMbJOZPV22brqZ3Wtmz4f/TgvXm5ldH7b7STM7tuw5F4b7P29mF1b6XROBmc03s/vN7BkzW2VmnwjX789trjGzP5rZE2Gb/ylcvyicPqYtnE4mE64fcnoZM7syXL/azN49Pi0aPjNLmtljZvarcHm/brOZrTWzp8zscTNrDdfF+9p296r7IRj8fgE4GMgATwBLxruufWjPycCxwNNl674MXBE+vgK4Lny8DPg1YMDxwEPh+unAmvDfaeHjaePdtiHaOwc4Nnw8BfgTwTQr+3ObDWgIH6eBh8K2/Ag4L1x/E/C/w8cfA24KH58H/DB8vCR8vWeBReH/g+R4t28vbf8k8APgV+Hyft1mYC0wY9C6WF/b1XrGMJzpOSYNd/8DwVVc5cqnF/kO8L6y9d/1wEqgyczmAO8G7nX3re6+DbgXWDr21Y+cu29090fDxzuBZwm+Nb8/t9ndvTNcTIc/DpxGMH0M7N7mStPLnAPc4e597v4i0Ebw/2FCMrN5wFnAv4XLxn7e5iHE+tqu1mCoND3HgeNUy1iZ7e4bw8evArPDx0O1fVL+TcLugjcTfILer9scdqk8Dmwi+I/+ArDd3QvhLuX1R6aXAfqnl5lUbQa+DnwGKIXLzez/bXbgN2b2iAXT/0DMr+1JOSWGjIy7u5ntd9clm1kD8FPgcnffYWXzLe6PbXb3InCMmTUBPwcOH+eSxpSZvQfY5O6PmNmp411PjE5y9w1mNgu418yeK98Yx2u7Ws8YhjM9x2T3WnhKSfjvpnD9UG2fVH8TM0sThML33f1n4er9us393H07cD9wAkHXQf8HvPL6h5peZjK1+W3A2Wa2lqC79zTgG+zfbcbdN4T/biL4AHAcMb+2qzUYhjM9x2RXPr3IhcAvy9ZfEF7NcDzQEZ6i3gO8y8ymhVc8vCtcN+GE/cbfBp51938u27Q/t3lmeKaAmdUC7yQYW7mfYPoY2L3NlaaXWQGcF17BswhYDPwxnlaMjLtf6e7z3H0hwf/R+9z9Q+zHbTazejOb0v+Y4DX5NHG/tsd7BH68fghG8/9E0E971XjXs49tuR3YCOQJ+hIvJuhb/R3wPPBbYHq4rxHcMOkF4Cmgpew4HyEYmGsDLhrvdu2hvScR9MM+CTwe/izbz9t8NPBY2OangavD9QcTvMm1AT8GsuH6mnC5Ldx+cNmxrgr/FquBM8e7bcNs/6nsuippv21z2LYnwp9V/e9Ncb+2NSWGiIhEVGtXkoiIDEHBICIiEQoGERGJUDCIiEiEgkFERCIUDCJ7YWYLzczNbELdaH6i1iWTn4JBZO/WEczo+jiAmZ0aviHPiKsAM/u9mX1rT3WJjBbNlSSyFx7MUfTqWBzbzNLunn89zx3LuqS66YxBJpTwk/GNZvZFM9tiwQ2IvmJmiXD7WjP7dIXnfKtsea2ZXW1mt5rZTjNbZ2bnmlmTmd1hZp3hzUveNcyaBrpswtlc7w83bQ7X3xruZ2b2GTN7wcx6LLjZyvkVjrPczO4zsx7gL82s2cxuN7P14fNWmdlFZc+7FTgF+Hj4fA+PtVtXkpmdbMFNanrN7DUz+1o47cuw/r7hPv/Tgpu+9JjZVjP7TzPrn81TqoCCQSaiDwEF4ETgUuBy4NwRHuNygmkRjiW4sct3CG72chdwDPAH4HtmVjPC464D3h8+PpKgK+cT4fLnCaYj+TjBzWGuBf7FzM4adIxrgRvDfX5BMJXDo8B7wmN+I3ze6eH+nwAeBP49/H1ziE6pDICZHUhw05bHCKYivxhYHv6+ckP+fc3sAIIJ674DHEFwE6jbhvF3kf3JeM8Noh/9lP8AvwceHLTuXuDfwsdrgU9XeM63ypbXAreXLTcQzK10fdm6heG6lmHUFNmXYN4ep+wuW0A90AO8fdBzvw7cNeg4nxrG77yjv82V2jhEXV8gmEsnUbbPh4E+oG6Yf99jw2MuGO/Xgn7G70djDDIRPTlo+RVg1us9hrt3mlk3wSRj/V4L/x3pcYeyhOCT/90WnSs/TRBU5VrLF8wsSXC7xnMJbqaSJbjl7O9HWMMRwEp3L5WteyA81qHs+pvs6e/7BMEkbU+b2W/Cxz9x980jrEUmMQWDTESDB2OdXd2eJYIZJculh3mM/KBlGL3u1P7jvBd4eS+1dA1a/jTwKYIuo6eATuCLjF5owa72Vqpn4O/r7sVw7OV4gqmaLwauNbNT3P2JUaxHJjAFg0w2mwn62AEIxwgOJ+hXj0su/DdZtu4Zgi6bBe5+3wiPdxJwp7vfBgP3mzgM2D7odyYrPLfcs8Cfm1mi7KzhpPC5Lwy3GHd3gjGNB83sGoLpn88lOJuQKqBgkMnmPuAjZraCICSuIv7X8UsEn7LPMrM7gR5332lmXwG+Er6x/4FgbON4oOTuN+/heH8CzjWzk4AtwGXAIqJhtxY4LrwqqhPYWuE4NxIMJN9oZt8gmNv/SwRjE93DaVh4s5czCG7q8hrBIPZ8guCTKqGrkmSyuZYgHH4J/IagDz3OswU8uPXiPxAM9r4G9F8q+1ngHwm6hlYRDOq+H3hxL4f8PMEVVL8mCJQu4PuD9vkKwSf/ZwgC8aAh6jqT4M38ceAWgps4/d0ImtdBcEvNXxEMZH8V+Jy7f28Ex5BJTjfqERGRCJ0xiIhIhIJBqp6Z3RR+G7rSz03jXZ9I3NSVJFXPzGYBjUNs3uHum+KsR2S8KRhERCRCXUkiIhKhYBARkQgFg4iIRCgYREQk4v8DkOIUHhifKYIAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"RwaRHglwlow_"},"source":["### Changing the context\n","\n","Updated ground truth rules:\n","\n","1. User 1 likes Item 2 in morning, and Item 5 in summer\n","2. User 2 likes Item 2 in summer, and Item 5 in morning\n","3. User 3 likes Item 4 in morning, Item 3 in evening, and item 4 in winter evening"]},{"cell_type":"code","metadata":{"id":"y_PbP3ZMlQQR"},"source":["users = ['A','B','C']\n","items = ['Item1','Item2','Item3','Item4','Item5','Item6']\n","context1 = ['morning','evening']\n","context2 = ['summer','winter']\n","\n","context = pd.DataFrame(list(product(users, context1, context2, items)), columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = 0\n","\n","#user 1 likes Item 2 in morning, and Item 5 in summer\n","context.loc[(context.users=='A') & \\\n","            (context.context1=='morning') & \\\n","            (context['items']=='Item2'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='A') & \\\n","            (context.context2=='summer') & \\\n","            (context['items']=='Item5'), \\\n","            'reward'] = 1\n","\n","#user 2 likes Item 2 in summer, and Item 5 in morning\n","context.loc[(context.users=='B') & \\\n","            (context.context2=='summer') & \\\n","            (context['items']=='Item2'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='B') & \\\n","            (context.context1=='morning') & \\\n","            (context['items']=='Item5'), \\\n","            'reward'] = 1\n","\n","\n","#user 3 likes Item 4 in morning, Item 3 in evening, and item 4 in winter evening\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='morning') & \\\n","            (context['items']=='Item4'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='evening') & \\\n","            (context['items']=='Item3'), \\\n","            'reward'] = 1\n","context.loc[(context.users=='C') & \\\n","            (context.context1=='evening') & \\\n","            (context.context2=='winter') & \\\n","            (context['items']=='Item4'), \\\n","            'reward'] = 1\n","\n","context['cost'] = context['reward']*-1\n","\n","contextdf_new = context.copy()\n","\n","def get_cost_new1(context,action):\n","    return contextdf_new.loc[(contextdf_new['users']==context['user']) & \\\n","            (contextdf_new.context1==context['context1']) & \\\n","            (contextdf_new.context2==context['context2']) & \\\n","            (contextdf_new['items']==action), \\\n","            'cost'].values[0]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"giMJGBc_lTPH"},"source":["def run_simulation_multiple_cost_functions(vw, num_iterations, users, contexts1, contexts2, actions, cost_functions, do_learn = True):\n","    cost_sum = 0.\n","    ctr = []\n","\n","    start_counter = 1\n","    end_counter = start_counter + num_iterations\n","    for cost_function in cost_functions:\n","        for i in range(start_counter, end_counter):\n","          user = choose_user(users)\n","          context1 = choose_context1(contexts1)\n","          context2 = choose_context2(contexts2)\n","\n","          context = {'user': user, 'context1': context1, 'context2': context2}\n","          \n","          action, prob = get_action(vw, context, actions)\n","          cost = cost_function(context, action)\n","          cost_sum += cost\n","\n","          if do_learn:\n","              vw_format = vw.parse(to_vw_example_format(context, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n","              vw.learn(vw_format)\n","\n","          ctr.append(-1*cost_sum/i)\n","        start_counter = end_counter\n","        end_counter = start_counter + num_iterations\n","\n","    return ctr"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"oer_Vp8SlUsu","executionInfo":{"status":"ok","timestamp":1634748177369,"user_tz":-330,"elapsed":12390,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"c77cba28-8ad5-40e7-8edb-6d22a46986ab"},"source":["# use first reward function initially and then switch to second reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new1]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, context1, context2, items, cost_functions)\n","\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxddX3/8dfnbrNmmckkIWQhARIhiAgGCC0iFYUALfjQVsFSl1KpbfFhrbY/KJYqqKjFVi0q8Kjbj1qoxf5qasNWAakVkCBbFoJZCEkgyWSbZNa7fX5/nDOTeyZ3kpkwc+6due/n4zEP7lnumc+5J9z3nO/3nO8xd0dERKRfotIFiIhIdVEwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRMQWDGb2HTPbaWarhlhuZvZ1M1tvZs+b2Rlx1SYiIgfFecbwPWDZYZZfDCwMf64BvhVDTSIiMkhsweDujwF7DrPK5cD/9cATwFQzmxVPdSIi0i9V6QJKzAa2lExvDee9NnhFM7uG4KyCpqamt5x00kmxFCgiMlE8/fTTu9x9erll1RQMw+budwJ3AixZssRXrlxZ4YpERMYXM9s81LJquippGzC3ZHpOOE9ERGJUTcGwHPhAeHXSUqDD3Q9pRhIRkbEVW1OSmd0NnA+0mdlW4G+BNIC73w6sAC4B1gPdwIfjqk1ERA6KLRjc/cojLHfgz2IqR0REhlBNTUkiIlIFFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZGIWIPBzJaZ2TozW29m15VZPs/MHjGzZ8zseTO7JM76REQkxmAwsyTwDeBiYDFwpZktHrTap4EfuvvpwBXAN+OqT0REAnGeMZwFrHf3je6eBe4BLh+0jgOTw9dTgFdjrE9ERIg3GGYDW0qmt4bzSn0GuMrMtgIrgI+V25CZXWNmK81sZXt7+1jUKiJSs6qt8/lK4HvuPge4BLjLzA6p0d3vdPcl7r5k+vTpsRcpIjKRxRkM24C5JdNzwnmlrgZ+CODujwP1QFss1YmICBBvMDwFLDSzBWaWIehcXj5onVeACwDM7GSCYFBbkYhIjGILBnfPA9cCDwBrCa4+Wm1mN5nZZeFqnwQ+YmbPAXcDH3J3j6tGERGBVJy/zN1XEHQql867seT1GuA346xJRESiqq3zWUREKkzBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIsPg7odM9+YKdPXl2d3Zx5pX99ObKwwsLxadPV3ZuMsUGRWpShcgUi3yhSKrXt3Ptr099OQKdGfzdPbl+fmvd/HExt3MmtIAQH06wWsdvXRnC4dsY15rI6mE0d7Zx4HePL/9pll8+tLFHDOlPu7dETlqCgaZsHpzBZ7f2sHe7ixb9/awZU836aTRkE7SnS2wbV8P+7pzvLKnm0QCXtvXS77oh2xnQVsTVy09jvU7O6lLJcgXnSXHtTJzch0Fd5rr0rQ2pdnY3sWG9k6yBWfJ/BZSyQT3Pr2VR9e1c8OlJ3PFmXMxswp8EiIjY4NPkcebJUuW+MqVKytdhsQgXyjy4vYDPL+1g+0dPQAU3OnqK5AtFOnNFdi5v4+iO+lkgme37KOjJzfwfjMo/ec+rSnD9El1zG1tJJNMMKelgcXHTmb+tCYSZjTXp2jKJJkx+ej/2t+8u4v/86PneWLjHs5e0Mot7z6V46c3H/X2REaLmT3t7kvKLlMwSKVl80W+9egGbv/ZBnrCdvpJ9SnOf8MMEgZ7u3O4O8+8so/OvnzZbZhBOpkYaMpxhxNmNPGuN8/mmCn1TKpPM39aI+7Qlc3TkE6SSsbTxVYsOj94cjNffmAdPdkC7zp9NhedcgznLWqjLpWMpQaRwRQMEov+f0ulzSW5QpG7f/kK+YLz+MbdbNjZydZ9PWTzxSNur625jv09OaY1Z2iqS3HC9CbOXTidxbMms2hmM3WpJMmEsburj9bGTGxf9Edr54Febn1gHfe9sJ0DfXmmNKQ5c34Lv3XSDC477Vgm1acrXaLUEAWDjDp35/5V2/mTH/xqxO9taUzTlS2QzRdpyiQ5be5Uzls0nfefPY/JNfDl2Jsr8NhL7Sx/7lWe3ryX1zp6qUsl+K03zOC8RdM598Q2Zrc0kEyoP0LGzuGCQZ3PclgHenP86OmtHNfWxKqtHXzloZdG9P7muhRtzRn+4Jz5nDZnCqfNnUq6yv+yH2v16SQXnnIMF55yDO7Os1v2cdfjm3l43U7uX70dCJrGTpszlfnTGjl2agMnTG9mTksDJx87uSbCUypLwSCH6M0VuOLOJ3h2y77DrnfWgla+8nunMbe1cWBeNl/k5d1dzArb9eXwzIzT57Vw+rwWCkXnpR0H+OWmPWza1cWzW/bx+Mbd7DzQN9BpbgbTm+toyCRprksxY1Id+aKzbW8Pi2ZOYsH0Jk6dPYWzFrTS1lxX2Z2TcUvBIADs2N/L9f/+Ag+/uLPs8kwywUffdjyfeOeiw15ymUklWDRz0liVOaElE8bJsyZz8qzJkfkHenO81tHLpl1drHl1P1v2dNNXKLK/J8f2/X0YwSW1q1/r4L/X7hi45HZqYxoDpjSkmd/WxIb2TupSSWZOrqNYhBmT62hpzDClIU1TXZKGdJI5rY20NmaYNaWenlyBproU9ekghKR26GjXAHfHzMjmi6x9bT/vveNx+g7T+Tu5PsUfvfV4/vT8E6q+Q7cWTKpPM6k+zaKZk7jolGMOu25fvsCqbft5ctNuXt3XQ6EIW/Z0s72jl1NnTyGbd3Ye6MXM+MWG3XT35ekqc6PeYMdPb+LYKQ0016XozhXI5YvMaWmgtSlDJpVgb3eWhBlNdSnmtDQwa0o986c1saCtSfdujEMKhglqT1eW+nSCpzfv5Q++/cthveeqpfP4q2UnqQ17HKtLJXnLcS285biWYb/nQG+OfMHp7Mvz6r4e9vXk2Ly7i6mNGTp783T05HhhWwe7u7Js399LKmE0ZJI8sq6dvd1ZCkVnamOaQtHpyRYiNwm2NmU4vq2Jea2NzG1tpKMnx77uLImE0RdefDBrSgO9+QJ9uSLZQpFjp9TTmEmRMGiqSzGpPk1fvsDWvT0kzEgnjcn1aVqaMuzq7GN6cx25QpGXd3ezq7OPrr489ekk0yfVUZdK0JBJclxrE4kENGZSTGvK0J9VmVSCTDLBjv19dGXzdPXl2dedY39vjv09efZ2Z+nozlGXDtbrzhUwIGGG4/TlinRnCzRkgjOuTCpBYyb43ZlkgpmT65nWnGFacx1tzZnI5cn9f7BVIwXDOFYoOk9u2s2b504lmTBe2t7J79z28yO+72tXvJlFMydxwvRm0kmr2n+cEo/+vqCWpkykv2i48oXiwJllsei82tHDzgN9rH1tP7/avI8te7v5n/W7aD/QR306QWtjhnzRyaQSdPTkONCbJ5NMkE4a6VSCfd25I/zGoTWkkzTXp+jNFTjQW/6el5HIJBNMqk+RD0OvqS5JoRh8oScTQUg1ZVL05Ar05gr05ooD9+IMlkwYdakEdakEvWEIppPGlIY0CTOM4IbNdDJBoeg0ZpI0ZlI0pJO0NKVpyqRwgkBpyKRozCS5+I3HsGR+6+vez8FiDQYzWwZ8DUgC/+TuXyyzznuBzwAOPOfu74+zxmqWzRdZ9On7ALjmvOO587GNR3xPUybJv/7xObxx9pSxLk9qVGlzYyJhzGlpZE5LI2fMa+H3zz5uYFmx6JhF73MpFn3gy7BfXz74UjfgQG8wXlUmFdyZ7h7cG9PRk2NPV5ZpTXW0d/ZRl0owt7WRyfWpge139AQ3Rh7ozbN1bw+Os78nz/6eHEUPaunLF8kVnLbmDI2ZFM11KVqbMkxtTNNcl6LpKPpW8oUie7qzZPNFtnf0sqcry67OLK/u66E7W6AvXxi4wTKbLwZnbEUnYUbCCAMjQU82GK+rO1tgQ3sXvbkC7sEFCN3ZAn25Am84ZtKYBENs9zGYWRJ4CXgnsBV4CrjS3deUrLMQ+CHwdnffa2Yz3L18b2ioVu5jeK2jh3NuefiI6333Q2fyWyfNiKEiEam019McVS33MZwFrHf3jWFR9wCXA2tK1vkI8A133wtwpFCoFe4+EArHtzVx5Vnz+PyKtdz6e6fxu2+Zg7tTdHRDlEiNGatm4DiDYTawpWR6K3D2oHUWAZjZ/xI0N33G3e8fvCEzuwa4BmDevHljUmylFYrOpl1dfODbT/JqRy8Al546i9vefzpmxkfOO35gXTMjqUwQkVFSbZ3PKWAhcD4wB3jMzE5198idVu5+J3AnBE1JcRc51tZtP8BFX33skPn9oSAiMpbiDIZtwNyS6TnhvFJbgSfdPQdsMrOXCILiqXhKrLyd+3sPCYX+JiMRkTjEGQxPAQvNbAFBIFwBDL7i6D+AK4HvmlkbQdPSkS+9mQAKRedfntzM3/x4NRCMMfTCZy7UGYKIxC62YHD3vJldCzxA0H/wHXdfbWY3ASvdfXm47EIzWwMUgL90991x1VgpHd05Trvpwcg8hYKIVIqG3a6wXKHIwhvuG5g2g023XFrBikSkFlTL5apSRml/wvOfuVDDUYhIxSkYKuhXr+xlY3sXAC/evIz6tB7zKCKVp6EzKyRfKPLub/4CgEc/db5CQUSqhoKhAl7cvp8TS/oV5rc1VbAaEZEoBUPMCkVn2Vf/Z2B69WcvqmA1IiKHUh9DjNyd37v9FwPTL39RVx+JSPVRMMTE3Vlw/YqB6bU3LatgNSIiQ1NTUky+eN+LA6+/8f4zaMios1lEqpPOGGJw/6rXuCN8qM4vb7iAGZPqK1yRiMjQhnXGYGZpM9tuZqeMdUETzY79vXz0n38FwB8sPU6hICJVb1jBEI52miN43KYMU75Q5Owv/HRg+uZ3vbGC1YiIDM9I+hj+EbjezNT8NEx/9+C6gdfrP39xBSsRERm+kXzJvxV4G7DNzFYBXaUL3f2y0SxsIrjjZ0G/wos3L4s8MF1EpJqNJBh2AT8aq0Immo3tnQCcOb9Fw12IyLgy7GBw9w+PZSETSW+uwNu/8jMAbnn3mypcjYjIyAy7fcPMHjazqWXmTzazh0e3rPHtY3c/M/D6xBnNFaxERGTkRtLwfT6QKTO/nqD/QYC9XVkeWrMDUIeziIxPR2xKMrMzSibfZGZ7SqaTwEUEz3AW4PSbHwLgrQvb1OEsIuPScPoYVhLcv+DAg2WW9wAfG82ixqtfbjqYmd//8FkVrERE5OgNJxgWAAZsBM4kuDqpXxbY6e6FMaht3Pn8irUA3P2RpSQSVuFqRESOzhGDwd03A5jZF4Ez3f320uVm9lEzm+3ufzNGNY4Le7uyPLdlH8mEcc4J0ypdjojIURtJI/hVwDNl5v8K+MDolDN+9fctzJ7aUOFKRERen5EEwwygvcz8XcDM0SlnfHp518GbwO/96DkVrERE5PUbSTC8QvnLUs8Dto5OOePTY78O8vIvL3oDMyZr9FQRGd9GMiTGHcA/mFkG6L+h7QLgFuBLo13YeNGdzXPjj1cDcPW5CypcjYjI6zeSITG+YmZtwNc5eKNbFviau395LIqrdlv2dPPWLz8yMK0xkURkIhjRENrufr2ZfQ5YHM5a6+6do1/W+FAaCt//Q923ICITw4ifreDuXcBTY1BL1XN3Fly/AoDnbrxwYP6Z81t426LplSpLRGRUacyGEejoyQ28Pu2mgzeB/9tHf6MS5YiIjAkFwzBt2tXFm2966JD5557YVoFqRETGjh7TOQynffbByNlCqb9/72kxVyMiMrYUDMMwOBQe+sR5vLKnmxOmN+u+BRGZcBQMh5EvFLl/9fbIvE23XIKZsXDmpApVJSIythQMh7H0lofZ1dkHQFtzhpWffmeFKxIRGXsKhjJ6sgVSSRsIBYAnrr+gghWJiMQn1quSzGyZma0zs/Vmdt1h1nuPmbmZLYmzPoDdnX2cfOP9LLzhvoF5p86eoqexiUjNiO3bzsySwDeAiwnunL7SzBaXWW8S8HHgybhqK/Xy7u5D5i2/9jcrUImISGXE+WfwWcB6d9/o7lngHuDyMuvdTDAoX2+MtQ14z7d+EZl+6BPnYaansYlI7YgzGGYDW0qmt4bzBpjZGcBcd/+vw23IzK4xs5VmtrK9vdwjIo7O81v3RaZXffYiXX0kIjWnahrOzSwB/D3wySOt6+53uvsSd18yffrojVF02W3/G5lurlPfvIjUnji/+bYBc0um54Tz+k0C3gg8GjbdHAMsN7PL3H3lWBVVKDq7Ovu46/HNkfkv3rxsrH6liEhVizMYngIWmtkCgkC4Anh//0J37wAGBh4ys0eBT41lKACc8NcrDpn3//70N/RsBRGpWbE1Jbl7HrgWeABYC/zQ3Veb2U1mdllcdQzH6fNaKl2CiEjFxNqI7u4rgBWD5t04xLrnx1HTYO8/e14lfq2ISNWoms7navH5d72x0iWIiFSUgmEQ3bMgIrVOwSAiIhE1Hwz16YMfwU8+dm4FKxERqQ41HwwnzmgG4IvvPpU3zp5S4WpERCqv5oMhl3fecfJMrjhLVyOJiECNP4+hO5tn3Y4DrNtxoNKliIhUjZo+Y9jTla10CSIiVaemgyGVqOndFxEpq6a/GYvulS5BRKTq1HQw5AsKBhGRwWo6GHLFIgBXLdUVSSIi/Wo6GIrF4IzhzPmtFa5ERKR61HQwPLhmBwDf+8XLlS1ERKSK1HQwtB/oA2B7R2+FKxERqR41HQwXnXIMADdcenKFKxERqR41HQz9I2y3NmYqW4iISBWp6WDo73xOJPQMBhGRfrUdDOFtDAk9nEdEZEBNB0MhvPM5WdOfgohIVE1/JQ40JemMQURkQE0HQ6HYf8agYBAR6VfTwdA/iJ7OGEREDlIwoGAQESlV08FQCMbQU1OSiEiJ2g4GXZUkInKImv5K1FVJIiKHqu1gcF2VJCIyWE0HQ0FnDCIih6jpYBi4KklnDCIiA2o6GAauStIZg4jIgNoOhoEzhgoXIiJSRWr6K9H7O591xiAiMqCmgyGbD9qS1PksInJQrMFgZsvMbJ2ZrTez68os/wszW2Nmz5vZT83suLGs53P/tRYAH8tfIiIyzsQWDGaWBL4BXAwsBq40s8WDVnsGWOLubwLuBb4cR239TUoiIhLvGcNZwHp33+juWeAe4PLSFdz9EXfvDiefAOaMVTGdffmB19Oa68bq14iIjDtxBsNsYEvJ9NZw3lCuBu4rt8DMrjGzlWa2sr29/aiKue5Hzx/V+0REJrqq7Hw2s6uAJcDflVvu7ne6+xJ3XzJ9+vSj+h0dPbnXUaGIyMSVivF3bQPmlkzPCedFmNk7gBuAt7l731gVk9LdziIiZcV5xvAUsNDMFphZBrgCWF66gpmdDtwBXObuO2OsTUREQrEFg7vngWuBB4C1wA/dfbWZ3WRml4Wr/R3QDPybmT1rZsuH2Nzr9vP1u8Zq0yIi41qcTUm4+wpgxaB5N5a8fkdcteSLukRVRKScqux8joOGwRARKa9mg0HDYIiIlFe7wVCzey4icng1+/WopiQRkfJqNhj01DYRkfJqNhiSCgYRkbJqNhgm1cd6pa6IyLhRs8EwrUkjqoqIlFOzwaC+ZxGR8mo3GCpdgIhIlardYNApg4hIWTUbDP0XJb1pzpTKFiIiUmVqNhj6zxgaM8kKVyIiUl1qNhj6zxg0yKqISFQNB0OQDEUlg4hIRM0GQyYV7HoqqU5oEZFSNRsMl546C4DrLz65wpWIiFSXmg2GdDLY9ckN6QpXIiJSXWo2GEREpDwFg4iIRCgYREQkomaDwdFlqiIi5dRsMPTTxaoiIlE1HwwiIhKlYBARkQgFg4iIRNRsMLj6nkVEyqrZYOin5/WIiETVfDCIiEiUgkFERCIUDCIiEqFgEBGRiJoNBl2VJCJSXs0GQz/ToBgiIhE1HwwiIhIVazCY2TIzW2dm683sujLL68zsX8PlT5rZ/DjrExGRGIPBzJLAN4CLgcXAlWa2eNBqVwN73f1E4B+AL41VPd25wlhtWkRkXIvzjOEsYL27b3T3LHAPcPmgdS4Hvh++vhe4wGxs7k1+ZvNeANIp9TGIiJRKxfi7ZgNbSqa3AmcPtY67582sA5gG7CpdycyuAa4JJzvNbN1R1tR27Jei264BbaB9rgHa59rwevb5uKEWxBkMo8bd7wTufL3bMbOV7r5kFEoaN7TPtUH7XBvGap/jbEraBswtmZ4Tziu7jpmlgCnA7liqExERIN5geApYaGYLzCwDXAEsH7TOcuCD4evfBR52161oIiJxiq0pKewzuBZ4AEgC33H31WZ2E7DS3ZcD3wbuMrP1wB6C8BhLr7s5ahzSPtcG7XNtGJN9Nv1BLiIipXTns4iIRCgYREQkomaD4UjDc4wXZjbXzB4xszVmttrMPh7ObzWzh8zs1+F/W8L5ZmZfD/f7eTM7o2RbHwzX/7WZfXCo31ktzCxpZs+Y2U/C6QXhUCrrw6FVMuH8IYdaMbPrw/nrzOyiyuzJ8JjZVDO718xeNLO1ZnbORD/OZvaJ8N/1KjO728zqJ9pxNrPvmNlOM1tVMm/UjquZvcXMXgjf8/Vh3TTs7jX3Q9D5vQE4HsgAzwGLK13XUe7LLOCM8PUk4CWCIUe+DFwXzr8O+FL4+hLgPsCApcCT4fxWYGP435bwdUul9+8I+/4XwL8APwmnfwhcEb6+HfiT8PWfAreHr68A/jV8vTg89nXAgvDfRLLS+3WY/f0+8Efh6wwwdSIfZ4IbXjcBDSXH90MT7TgD5wFnAKtK5o3acQV+Ga5r4XsvPmJNlf5QKnQgzgEeKJm+Hri+0nWN0r79GHgnsA6YFc6bBawLX98BXFmy/rpw+ZXAHSXzI+tV2w/BfTA/Bd4O/CT8R78LSA0+xgRXwp0Tvk6F69ng4166XrX9ENzTs4nwgpHBx28iHmcOjoTQGh63nwAXTcTjDMwfFAyjclzDZS+WzI+sN9RPrTYllRueY3aFahk14anz6cCTwEx3fy1ctB2YGb4eat/H22fyVeCvgGI4PQ3Y5+75cLq0/shQK0D/UCvjaZ8XAO3Ad8Pms38ysyYm8HF2923ArcArwGsEx+1pJvZx7jdax3V2+Hrw/MOq1WCYcMysGfgR8Ofuvr90mQd/KkyY65LN7LeBne7+dKVriVGKoLnhW+5+OtBF0MQwYAIe5xaCgTUXAMcCTcCyihZVAZU4rrUaDMMZnmPcMLM0QSj8wN3/PZy9w8xmhctnATvD+UPt+3j6TH4TuMzMXiYYpfftwNeAqRYMpQLR+ocaamU87fNWYKu7PxlO30sQFBP5OL8D2OTu7e6eA/6d4NhP5OPcb7SO67bw9eD5h1WrwTCc4TnGhfAKg28Da93970sWlQ4v8kGCvof++R8Ir25YCnSEp6wPABeaWUv4l9qF4byq4+7Xu/scd59PcOwedvffBx4hGEoFDt3nckOtLAeuCK9mWQAsJOioqzruvh3YYmZvCGddAKxhAh9ngiakpWbWGP4779/nCXucS4zKcQ2X7TezpeFn+IGSbQ2t0p0uFezsuYTgCp4NwA2Vrud17Me5BKeZzwPPhj+XELSt/hT4NfDfQGu4vhE8MGkD8AKwpGRbfwisD38+XOl9G+b+n8/Bq5KOJ/gffj3wb0BdOL8+nF4fLj++5P03hJ/FOoZxtUaF9/XNwMrwWP8HwdUnE/o4A58FXgRWAXcRXFk0oY4zcDdBH0qO4Mzw6tE8rsCS8PPbANzGoAsYyv1oSAwREYmo1aYkEREZgoJBREQiFAwiIhKhYBARkQgFg4iIRCgYRI7AzOabmZtZVT1ovlrrkvFPwSByZFsIBiN7FsDMzg+/kNviKsDMHjWz2w5Xl8hoie2ZzyLjlbsXCAYyG3VmlvZguIcRG8u6pLbpjEGqSviX8TfN7Atmtit8gMmtZpYIl79sZp8q857bSqZfNrMbzex7ZnbAzLaY2fsseNDNPWbWGT7M5MJh1jTQZBOOYPtIuKg9nP+9cD0zs78ysw1m1hM+HOWqMtu50sweNrMe4I/NbJoFD6HZGr5vtZl9uOR93wPeBvxZ+H4Pt3VIU5KZnWfBQ2p6zWyHmf1DOOzLsD7fcJ13W/AQmB4z22NmPzOz/tE9pQYoGKQa/T6QB34DuBb4c+B9I9zGnxMMi3AGwYNdvk/wUJ8VBENLPAb8s5nVj3C7W4D3hK9PIWjK+Xg4/TmC4Qz+jODhMLcAd5jZpYO2cQvwzXCd/yAYyuFXwG+H2/xa+L4LwvU/DjwOfDf8fbOIDrEMgJnNJngQyzMEw69fTTD+/i2DVh3y8zWzYwgGJvw+cDLBQ2TuGsbnIhNJpccJ0Y9+Sn+AR4HHB817CPin8PXLwKfKvOe2kumXgbtLppsJxpP6esm8+eG8JcOoKbIuwfhMDrSVrNME9ABvHfTerwIrBm3nk8P4nff073O5fRyirs8TjK2TKFnnQ0Af0DjMz/eMcJvHVfrfgn4q96M+BqlGzw+afhWYcbTbcPdOM+smGHSs347wvyPd7lAWE/zlf7+ZlQ5AliYIqlIrSyfMLEnwbIX3ETxEpY7g0Z2PjrCGk4En3L1YMu/n4bZO5OBncrjP9zmCQdtWmdmD4et73b19hLXIOKZgkGo0uDPWOdjsWSQYYbJUepjbyA2ahtFrTu3fzu8QDBd9uFq6Bk1/CvgkQZPRC0An8AVGL7Qg+qCXIT9fdy+EfS9LCYZuvhq4xcze5u7PjWI9UsUUDDLetBO0sQMQ9hGcRNCuHpds+N9kybw1BE02x7n7wyPc3rnAf7r7XTDwjI1FwL5BvzvHqIsAAAFdSURBVDNZ5r2l1gLvNbNEyVnDueF7Nwy3GHd3gj6Nx83sJmA1wdmMgqFGKBhkvHkY+EMzW04QEjcQ/7/jzQR/ZV9qZv8J9Lj7ATO7Fbg1/GJ/jKBvYylQdPc7D7O9l4D3mdm5BA+w/xjB4yxLw+5l4KzwqqhOYE+Z7XyToCP5m2b2NYLnFnyRoG+iezg7Fj785R0ED37ZQdCJPZcg+KRG6KokGW9uIQiHHwMPErShx3m2gAcPqf9bgs7eHQQPPwH4G+AzBE1Dqwk6dd8DbDrCJj9HcAXVfQSB0gX8YNA6txL85b+GIBDnDVHXxQRf5s8C3yF4CMxfj2D3Oggen/kTgo7srwA3u/s/j2AbMs7pQT0iIhKhMwYREYlQMEjNM7Pbw7uhy/3cXun6ROKmpiSpeWY2A5g8xOL97r4zznpEKk3BICIiEWpKEhGRCAWDiIhEKBhERCRCwSAiIhH/H2EcqTXj571GAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":287},"id":"p7LbRqzElWgc","executionInfo":{"status":"ok","timestamp":1634748188707,"user_tz":-330,"elapsed":11393,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"8fc30f58-d9ab-4eac-ab68-a02c050c0d1b"},"source":["# Do not learn\n","# use first reward function initially and then switch to second reward function\n","\n","# Instantiate learner in VW\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","\n","num_iterations_per_cost_func = 5000\n","cost_functions = [get_cost, get_cost_new1]\n","total_iterations = num_iterations_per_cost_func * len(cost_functions)\n","\n","ctr = run_simulation_multiple_cost_functions(vw, num_iterations_per_cost_func, users, context1, context2, items, cost_functions, do_learn=False)\n","plot_ctr(total_iterations, ctr)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEOCAYAAACNY7BQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAd7UlEQVR4nO3de5hcdZ3n8fe37n3NtRNiEklYgphZUGOLcbzACgpEB55d51FYXRXZQXfE8bouDDteGFbUwQssILCO4qIjIs46GYyLl4DOusrSiEASiDQByQWTDrn0va7f/eOc7tRpqpNu6D7V3fV5PU89qfM7p059T53O+dS51O+YuyMiIjIiUe8CRERkZlEwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRMQWDGb2DTPbZ2ZbxhlvZnadmXWb2cNmti6u2kRE5Ig49xhuBc45yvhzgTXh4xLgazHUJCIiY8QWDO7+S+DAUSY5H/ifHvgNMN/MlsVTnYiIjEjVu4Aqy4GdVcO7wrZnxk5oZpcQ7FXQ0tLyypNPPjmWAkVE5ooHHnhgv7t31Bo3k4Jhwtz9FuAWgM7OTu/q6qpzRSIis4uZ/WG8cTPpqqTdwMqq4RVhm4iIxGgmBcNG4N3h1UnrgcPu/pzDSCIiMr1iO5RkZt8FzgAWm9ku4NNAGsDdbwI2ARuAbmAQuCiu2kRE5IjYgsHdLzzGeAc+GFM5IiIyjpl0KElERGYABYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCQi1mAws3PMbLuZdZvZZTXGv9jM7jGzB83sYTPbEGd9IiISYzCYWRK4ATgXWAtcaGZrx0z2X4E73P0VwAXAjXHVJyIigTj3GE4Dut19h7sXgNuB88dM40B7+HwesCfG+kREhHiDYTmws2p4V9hW7TPAu8xsF7AJ+FCtGZnZJWbWZWZdPT0901GriEjDmmknny8EbnX3FcAG4DYze06N7n6Lu3e6e2dHR0fsRYqIzGVxBsNuYGXV8IqwrdrFwB0A7v5rIAcsjqU6EREB4g2G+4E1ZrbazDIEJ5c3jpnmaeBMADN7KUEw6FiRiEiMYgsGdy8BlwJ3A48SXH201cyuNLPzwsk+DvyFmT0EfBd4r7t7XDWKiAik4nwzd99EcFK5uu1TVc+3Aa+NsyYREYmaaSefRUSkzhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEhFrMJjZOWa23cy6zeyycaZ5u5ltM7OtZvYPcdYnIiKQiuuNzCwJ3AC8CdgF3G9mG919W9U0a4DLgde6+0EzWxJXfSIiEohzj+E0oNvdd7h7AbgdOH/MNH8B3ODuBwHcfV+M9YmICPEGw3JgZ9XwrrCt2knASWb2KzP7jZmdU2tGZnaJmXWZWVdPT880lSsi0phm2snnFLAGOAO4EPgfZjZ/7ETufou7d7p7Z0dHR8wliojMbXEGw25gZdXwirCt2i5go7sX3f1J4PcEQSEiIjGJMxjuB9aY2WozywAXABvHTPNDgr0FzGwxwaGlHTHWKCLS8GILBncvAZcCdwOPAne4+1Yzu9LMzgsnuxt41sy2AfcA/9ndn42rRhERAXP3etfwgnR2dnpXV1e9yxARmVXM7AF376w1bqadfBYRkTpTMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJmFAwmFnazP5oZn8y3QWJiEh9TSgY3L0IFIHZ3X+GiIgc02QOJf134HIzi+12oCIiEr/JbORfD5wO7DazLcBA9Uh3P6/mq0REZFaZTDDsB34wXYWIiMjMMOFgcPeLprMQERGZGSZ8jsHMNte6/7KZtZvZ5qktS0RE6mUyJ5/PADI12nME5x9ERGQOOOahJDNbVzV4qpkdqBpOAmcDu6e6MBERqY+JnGPoIvj9ggM/qTF+CPjQVBYlIiL1M5FgWA0YsAN4FcHVSSMKwD53L09DbSIiUgfHDAZ3/wOAmX0eeJW731Q93sw+YGbL3f1vpqlGERGJ0WROPr8LeLBG+2+Bd09NOSIiUm+TCYYlQE+N9v3A0qkpR0RE6m0ywfA0tS9LfQOwa2rKERGReptMlxg3A18xswww8oO2M4GrgS9MdWEiIlIfk+kS40tmthi4jiM/dCsA17r7F6ejOBERid+kutB298vN7Cpgbdj0qLv3T31ZIiJSL5O+t4K7DwD3T0MtIiIyA+iezyIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIRKzBYGbnmNl2M+s2s8uOMt3bzMzNrDPO+kREJMZgMLMkcANwLsEvpy80s7U1pmsDPgzcF1dtIiJyRJx7DKcB3e6+w90LwO3A+TWm+1uCTvmGY6xNRERCcQbDcmBn1fCusG2Uma0DVrr7j442IzO7xMy6zKyrp6fWLSJEROT5mjEnn80sAXwZ+PixpnX3W9y90907Ozo6pr84EZEGEmcw7AZWVg2vCNtGtAH/GrjXzJ4C1gMbdQJaRCRecQbD/cAaM1sd3uznAmDjyEh3P+zui919lbuvAn4DnOfuXTHWKCLS8GILBncvAZcCdwOPAne4+1Yzu9LMzourDhERObpJ34/hhXD3TcCmMW2fGmfaM+KoSUREombMyWcREZkZFAwiIhKhYBARkQgFg4iIRCgYREQkQsEgIiIRCgYREYlQMIiISISCQUREIhQMIiISoWAQEZEIBYOIiEQoGEREJELBICIiEQoGERGJUDCIiEiEgkFERCIUDCIiEqFgEBGRCAWDiIhEKBhERCRCwSAiIhEKBhERiVAwiIhIhIJBREQiFAwiIhKhYBARkQgFg4iIRDR0MOztHWa4WK53GSIiM0pDB8OrP/dzLvrm/fUuQ0RkRmnYYNjbOwzAr3c8W+dKRERmloYNhgMDhXqXICIyIzVsMIiISG0NGwwJs3qXICIyI8UaDGZ2jpltN7NuM7usxviPmdk2M3vYzH5uZsdPVy0J5YKISE2xBYOZJYEbgHOBtcCFZrZ2zGQPAp3ufipwJ/DFaaxnumYtIjKrxbnHcBrQ7e473L0A3A6cXz2Bu9/j7oPh4G+AFdNVjPYYRERqizMYlgM7q4Z3hW3juRj4ca0RZnaJmXWZWVdPT8/zKkZ7DCIitc3Ik89m9i6gE/i7WuPd/RZ373T3zo6Ojuf1HtpjEBGpLRXje+0GVlYNrwjbIszsLOAK4HR3z09XMboqSUSktjj3GO4H1pjZajPLABcAG6snMLNXADcD57n7vhhrExGRUGzB4O4l4FLgbuBR4A5332pmV5rZeeFkfwe0At83s9+Z2cZxZiciItMkzkNJuPsmYNOYtk9VPT8rvlqOPD84UGBBSyautxYRmdFiDYaZxDmSDPc9+Sw79g/wllOWcfyiljpWJTL7FEoVDg8VGS6WSSSMhEHFoW+4yHCxQlsuRSphFMuOuzOvOU1TOslAvkwqabgHF4PMa0pTccikEgwVyjxzeIhyxTk8VKRQrpAvVkgnE/TnS/TnSxwYyFOqOO25NAuaM6STRsKMUqVCUyZFazZJf75M/3CJQrlMsezMb0ozrylNUyaJYRQrFQ4PFtnfn8fMyKUTtOXSVDyodSBfpliuUK4E2wsHiuUKFYeBfInWbIq2XIp8qUK+VCGbTGAWTJcwo1iu0JxJ0p4L3jNhRqFUoT9fApy9vXmeOTyM4+RSSUqVCgmz0eUuV5xsKkFfvkShVKEtlw7qcCeXTvLWU5fRuWrhlK/Txg2Gqj2GD3z7twBcv7mbbVeeM2Xv8eDTB/m3N/7f0eFNf/V61r6ofcrmL3NHpeJU3CmWnYd2HWLPoSGSCcPM2HNoiIF8iaFCmVw6SbFcoXc42DgeHChQqlQolZ1FrRnmNaVpzwUb2P58kT2HhkkljcWtWYrlCrlUkv39eQrlCr1DRfYcHmZRS4b2XBozODxUpG+4RDppZFIJDg4WyaYSHBosjm60Ugkjlw42crl0gp0Hh0Y3nFOhJZNkqFhmCmc5o6USQaAVyhUSYagkzWjNpUgnE+SLZdpyaTKpBH3DJdydRMLIF8usfVG7gmEq1fqbGyyUcffIbxwe2nmI82/4FQA/+9gb6GjL8bLP/oSvvXMd556yDICevjzXb36cj73pJcxrTvPk/gF+t/MgH/3eQ5H5b7juX2jLpdj0V69n5cLmaVs2mT7FcoVUwthzeJhn+/Mc156jWHFK5Qp/eHaQ3uEig4UyfcMlevqCi+pGvlH2DhXpz5dY1JoBh3ypwoGBAnsODfHwrsOUKpWjbgwzyQSFcgWAhS0Z2nMp5jWlMTPM4ImegXDDXiRhwYZ9xYIm3OGxZ/pIp4zhYoX54TfmeU1pTuhopXeoSF++hDusWNBEUybYLOSLZV62Ik2xXGF++I28XIF8KfgW7Q79+RIbTlnGkrYsTZkkEOwtGNCUSdKWS9E7VKJQrpBOGslEgkODBYaLZZoyqdFABOgdKgFwcLBAey7F6o4WUokErbkU2VSCbCpBqey0ZFO059LMa06TSSboHS5ycLBAseQ4TjJhDBXK9OVLtGVTtOXSpJNGKpHg0FCBvuESA/kSTrB30p5L0dGaA2C4VKZ3qBju+RjNmSSZZIJE+Bm7QzadwICWbIr+fIm+4SLZVJJsOkG+WBldXxV30skEg4UyvcNFhgplKu5kkglasinMYFFLlqXtWSzcS0jYzPiNlbnP7lju7Oz0rq6uSb/uyf0D/Jtr7n1O+/1XnEVHW5Ytuw+zdc9h/ssPHhl3Hl/881Npzab4y+/8drTt1otexXvH3Pzn2gteTn++xBX/a0uk/RNvPonXr+ngpcvayaSi1wF07+vj6QODPPj0IXb0DHDBaSt5ydI2fr+3nwf+cJA/e9kyVixoZvNj+8ilE6xa1EJ/vsSS9ixL2nKT/jzGenxvHy3ZFMvm5TAzBgsldh0cYl9vnqeeHRj9D1muOC9e2EypEhyc6x8ONnwvPa6d4VKZR3YdHt31BliztJVcKslwqcxAvsTBwQL9+eA/Y2s2xZP7BxgslFjQnGFhS4a+4RLN2SQtmRR7Dg/x+N5++vMlsqkEmWSCxa1Z2nIpkkljcUuW/f3BxjibDjZSuw8Osbd3mKFimXypzHHtTSxpz5JJJmjOJCm7Bxv6Q8McGCiE39YY3bDuDr+tL2jOUCxX2Lqnl0QiOBwwESMbE4CmdJL2phTP9gfvk00lWNiSoTWbYv0Ji8imE6QTCV6+cj6rFrcw8n9zfnOGRS0ZEgnD3SlVgg2OyAthZg+4e2fNcY0aDDt6+nnjl37xnPa3nrqM6//9OlZd9qOpKI+tnz2blmzwDezwUJGr7trG9x/Y9ZzpPnLWGr76s8en5D0BTj6ujRM6WsgXKyxoybCoNcOpy+fzhpMWkzBj655eSuUKp6yYR6nsfPNXT3L9Pd0sbc/xzOHhKavj+cimEpQrwQZwrBULmljUkqFUcQ4NBt/ADw8VSSaCb1zJcOM58tLj2nPMa0rTmkuRTBj7eod5dqAADoPF8ughkPZcimXzmkgljYoHx3AHCiVWzG+mvSnFwcEiqYRx4pJW0skES9qyrFjQTE/fMKlkglTCWLGgmYUtGZoySVqzKdqbUqPHlLOpBKlwY14qV3DQxl3qSsFQwxM9/ZxZIxgAbv4Pr+T9tz0wOvy2dSvoHS7y0217R9ve3rmCO7qObOAf+vSb+dq9T3DTL54AYMtnz6Y1O/6Ruju6dvLJOx+eUK0vXtjM0wcGR4ebM0kGC0fuVb3hlOPY/Ng+hosVTlzSSve+/gnN92iaM0nOe9mLePDpQ2zf28fy+U2YwdL2HB896ySWtGdxh0WtGbr39TNUKFMoV1jSlqVvuMSOnn4c+FcdrRw3LxecmAyPlxfLTqFcpi2bpi2XYml7jqZMkr7hEisWNJFLJxkqlHn6wCDHL2pmqFCmP1+ioy1LLtwTGOHhcflUwjgwWGBhc4ZKGAzFcmU0lGsplSuUwjDRRloajYKhhq//yw6u+tGjx5zuvr8+k6XtOdwdd/jcpkdZd/wCNpyyjN2Hhnjt5zfzvUvW8+oTFgHw+719ZJIJVi2e+NVNA/kS51z7S/qHS5x+UgdffvvLSbzAPjvcPTj00zdMf75MSybJd+57muFimR9v+SPrT1jISUvbRkPkxCWtXPy61bw4PPcxE45zisj0UTDU8L5b72fzY0f/cfXPPnY6Jy5pfb6liYjMWEcLhobdfx77ffi9f7qKO97/mkhbe65hL9oSkQbWuMEw5lDJh89cw2mrF3LnB46EQ3tTOu6yRETqrmG/Eo89hD7SJUbnqoXs+NwG+vKl55zoFBFpBA27x1B9bjedjKZEImHM096CiDSohg0GC88yrFrUzCOfObvO1YiIzBwNGwyJcMnTyYQOGYmIVGnYYBjZY5jdF+uKiEy9hg2GkfMK+sWriEhUw24V33LqiwC47NyT61yJiMjM0rDBkAr3GPQjNhGRqIYNBp1cEBGprXGDIaTO4kREoho+GEREJErBICIiEQ0bDK6TDCIiNTVsMIzQGQYRkaiGDwYREYlSMIiISETDBsMsv6OpiMi0adhgGKGfMYiIRDV8MIiISFTDBoMOJYmI1NawwTDCdMGqiEhEwweDiIhEKRhERCSiYYNBpxhERGpr2GAYoctVRUSiYg0GMzvHzLabWbeZXVZjfNbMvheOv8/MVsVZn4iIxBgMZpYEbgDOBdYCF5rZ2jGTXQwcdPcTga8AX5iuegYLpematYjIrBbnHsNpQLe773D3AnA7cP6Yac4HvhU+vxM406bpFmu/6t4PQDrZ8EfTREQiUjG+13JgZ9XwLuDV403j7iUzOwwsAvZXT2RmlwCXhIP9Zrb9eda0+OQvROfdABaDlrkBaJkbwwtZ5uPHGxFnMEwZd78FuOWFzsfMuty9cwpKmjW0zI1By9wYpmuZ4zyOshtYWTW8ImyrOY2ZpYB5wLOxVCciIkC8wXA/sMbMVptZBrgA2Dhmmo3Ae8Lnfw5sdlevRiIicYrtUFJ4zuBS4G4gCXzD3bea2ZVAl7tvBP4euM3MuoEDBOExnV7w4ahZSMvcGLTMjWFaltn0hVxERKrpWk0REYlQMIiISETDBsOxuueYLcxspZndY2bbzGyrmX04bF9oZj81s8fDfxeE7WZm14XL/bCZraua13vC6R83s/eM954zhZklzexBM7srHF4ddqXSHXatkgnbx+1qxcwuD9u3m9nZ9VmSiTGz+WZ2p5k9ZmaPmtlr5vp6NrOPhn/XW8zsu2aWm2vr2cy+YWb7zGxLVduUrVcze6WZPRK+5roJ/WjY3RvuQXDy+wngBCADPASsrXddz3NZlgHrwudtwO8Juhz5InBZ2H4Z8IXw+Qbgx4AB64H7wvaFwI7w3wXh8wX1Xr5jLPvHgH8A7gqH7wAuCJ/fBPyn8PlfAjeFzy8Avhc+Xxuu+yywOvybSNZ7uY6yvN8C/mP4PAPMn8vrmeAHr08CTVXr971zbT0DbwDWAVuq2qZsvQL/L5zWwteee8ya6v2h1GlFvAa4u2r4cuDyetc1Rcv2T8CbgO3AsrBtGbA9fH4zcGHV9NvD8RcCN1e1R6abaQ+C38H8HHgjcFf4R78fSI1dxwRXwr0mfJ4Kp7Ox6716upn2IPhNz5OEF4yMXX9zcT1zpCeEheF6uws4ey6uZ2DVmGCYkvUajnusqj0y3XiPRj2UVKt7juV1qmXKhLvOrwDuA5a6+zPhqD8CS8Pn4y37bPtMvgp8EqiEw4uAQ+4+0jtidf2RrlaAka5WZtMyrwZ6gG+Gh8++bmYtzOH17O67gWuAp4FnCNbbA8zt9Txiqtbr8vD52PajatRgmHPMrBX4AfARd++tHufBV4U5c12ymb0V2OfuD9S7lhilCA43fM3dXwEMEBxiGDUH1/MCgo41VwMvAlqAc+paVB3UY702ajBMpHuOWcPM0gSh8B13/8ewea+ZLQvHLwP2he3jLfts+kxeC5xnZk8R9NL7RuBaYL4FXalAtP7xulqZTcu8C9jl7veFw3cSBMVcXs9nAU+6e4+7F4F/JFj3c3k9j5iq9bo7fD62/agaNRgm0j3HrBBeYfD3wKPu/uWqUdXdi7yH4NzDSPu7w6sb1gOHw13Wu4E3m9mC8Jvam8O2GcfdL3f3Fe6+imDdbXb3dwL3EHSlAs9d5lpdrWwELgivZlkNrCE4UTfjuPsfgZ1m9pKw6UxgG3N4PRMcQlpvZs3h3/nIMs/Z9VxlStZrOK7XzNaHn+G7q+Y1vnqfdKnjyZ4NBFfwPAFcUe96XsByvI5gN/Nh4HfhYwPBsdWfA48DPwMWhtMbwQ2TngAeATqr5vU+oDt8XFTvZZvg8p/BkauSTiD4D98NfB/Ihu25cLg7HH9C1euvCD+L7Uzgao06L+vLga5wXf+Q4OqTOb2egc8CjwFbgNsIriyaU+sZ+C7BOZQiwZ7hxVO5XoHO8PN7ArieMRcw1HqoSwwREYlo1ENJIiIyDgWDiIhEKBhERCRCwSAiIhEKBhERiVAwiByDma0yMzezGXWj+Zlal8x+CgaRY9tJ0BnZ7wDM7Ixwg7w4rgLM7F4zu/5odYlMldju+SwyW7l7maAjsylnZmkPunuYtOmsSxqb9hhkRgm/Gd9oZp8zs/3hDUyuMbNEOP4pM/tEjddcXzX8lJl9ysxuNbM+M9tpZu+w4EY3t5tZf3gzkzdPsKbRQzZhD7b3hKN6wvZbw+nMzD5pZk+Y2VB4c5R31ZjPhWa22cyGgPeb2SILbkKzK3zdVjO7qOp1twKnAx8MX+/hvJ5zKMnM3mDBTWqGzWyvmX0l7PZlQp9vOM2/s+AmMENmdsDMfmFmI717SgNQMMhM9E6gBPwpcCnwEeAdk5zHRwi6RVhHcGOXbxHc1GcTQdcSvwS+bWa5Sc53J/C28PmfEBzK+XA4fBVBdwYfJLg5zNXAzWb2ljHzuBq4MZzmhwRdOfwWeGs4z2vD150ZTv9h4NfAN8P3W0a0i2UAzGw5wY1YHiTofv1igv73rx4z6bifr5kdR9Ax4beAlxLcROa2CXwuMpfUu58QPfSofgD3Ar8e0/ZT4Ovh86eAT9R4zfVVw08B360abiXoT+q6qrZVYVvnBGqKTEvQP5MDi6umaQGGgNePee1XgU1j5vPxCbzn7SPLXGsZx6nrvxH0rZOomua9QB5onuDnuy6c5/H1/lvQo34PnWOQmejhMcN7gCXPdx7u3m9mgwSdjo3YG/472fmOZy3BN///bWbVHZClCYKqWlf1gJklCe6t8A6Cm6hkCW7dee8ka3gp8Bt3r1S1/Z9wXidy5DM52uf7EEGnbVvM7Cfh8zvdvWeStcgspmCQmWjsyVjnyGHPCkEPk9XSE5xHccwwTN3h1JH5/BlBd9FHq2VgzPAngI8THDJ6BOgHPsfUhRZEb/Qy7ufr7uXw3Mt6gq6bLwauNrPT3f2hKaxHZjAFg8w2PQTH2AEIzxGcTHBcPS6F8N9kVds2gkM2x7v75knO73XAP7v7bTB6j42TgENj3jNZ47XVHgXebmaJqr2G14WvfWKixbi7E5zT+LWZXQlsJdibUTA0CAWDzDabgfeZ2UaCkLiC+P+O/0DwLfstZvbPwJC795nZNcA14Yb9lwTnNtYDFXe/5Sjz+z3wDjN7HcEN7D9EcDvL6rB7CjgtvCqqHzhQYz43EpxIvtHMriW4b8HnCc5NDE5kwcKbv5xFcOOXvQQnsVcSBJ80CF2VJLPN1QTh8E/ATwiOoce5t4AHN6n/NMHJ3r0ENz8B+BvgMwSHhrYSnNR9G/DkMWZ5FcEVVD8mCJQB4DtjprmG4Jv/NoJAfPE4dZ1LsDH/HfANgpvA/PUkFu8wwe0z7yI4kf0l4G/d/duTmIfMcrpRj4iIRGiPQUREIhQM0vDM7Kbw19C1HjfVuz6RuOlQkjQ8M1sCtI8zutfd98VZj0i9KRhERCRCh5JERCRCwSAiIhEKBhERiVAwiIhIxP8HFEPPfqS+bfAAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"id":"vNgiCkfGlYAi"},"source":["mapping_users = {\n","    'Alex':'usera',\n","    'Ben':'userb',\n","    'Cindy': 'userc'\n","}\n","    \n","mapping_context1 = {\n","    'Morning':'ctx11',\n","    'Evening':'ctx12',\n","}\n","\n","mapping_context2 = {\n","    'Summer':'ctx21',\n","    'Winter':'ctx22'\n","}\n","\n","mapping_items = {\n","    'Politics':'item1',\n","    'Economics':'item2',\n","    'Technology':'item3',\n","    'Movies':'item4',\n","    'Business':'item5',\n","    'History':'item6'\n","}"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"UTu524YinWig"},"source":["users = list(mapping_users.values())\n","items = list(mapping_items.values())\n","context1 = list(mapping_context1.values())\n","context2 = list(mapping_context2.values())"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":419},"id":"Fj5L4k_UnZkr","executionInfo":{"status":"ok","timestamp":1634748746746,"user_tz":-330,"elapsed":16,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"8802dc91-faa9-401c-db4d-1678e3c5b1fb"},"source":["context = pd.DataFrame(list(product(users, context1, context2, items)), columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = np.random.choice([0,1],len(context))\n","context['cost'] = context['reward']*-1\n","contextdf = context.copy()\n","contextdf"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>users</th>\n","      <th>context1</th>\n","      <th>context2</th>\n","      <th>items</th>\n","      <th>reward</th>\n","      <th>cost</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item1</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item2</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item4</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item5</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>67</th>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item2</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>68</th>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>69</th>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item4</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>70</th>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item5</td>\n","      <td>1</td>\n","      <td>-1</td>\n","    </tr>\n","    <tr>\n","      <th>71</th>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item6</td>\n","      <td>0</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>72 rows × 6 columns</p>\n","</div>"],"text/plain":["    users context1 context2  items  reward  cost\n","0   usera    ctx11    ctx21  item1       1    -1\n","1   usera    ctx11    ctx21  item2       1    -1\n","2   usera    ctx11    ctx21  item3       1    -1\n","3   usera    ctx11    ctx21  item4       0     0\n","4   usera    ctx11    ctx21  item5       1    -1\n","..    ...      ...      ...    ...     ...   ...\n","67  userc    ctx12    ctx22  item2       1    -1\n","68  userc    ctx12    ctx22  item3       0     0\n","69  userc    ctx12    ctx22  item4       1    -1\n","70  userc    ctx12    ctx22  item5       1    -1\n","71  userc    ctx12    ctx22  item6       0     0\n","\n","[72 rows x 6 columns]"]},"metadata":{},"execution_count":45}]},{"cell_type":"code","metadata":{"id":"wZ66OoF_nb5c"},"source":["# This function modifies (context, action, cost, probability) to VW friendly format\n","def to_vw_example_format(context, actions, cb_label=None):\n","    if cb_label is not None:\n","        chosen_action, cost, prob = cb_label\n","    example_string = \"\"\n","    example_string += \"shared |User users={} context1={} context2={}\\n\".format(context[\"user\"], context[\"context1\"], context[\"context2\"])\n","    for action in actions:\n","        if cb_label is not None and action == chosen_action:\n","            example_string += \"0:{}:{} \".format(cost, prob)\n","        example_string += \"|Action items={} \\n\".format(action)\n","    #Strip the last newline\n","    return example_string[:-1]\n","\n","\n","def sample_custom_pmf(pmf):\n","    total = sum(pmf)\n","    scale = 1 / total\n","    pmf = [x * scale for x in pmf]\n","    draw = random.random()\n","    sum_prob = 0.0\n","    for index, prob in enumerate(pmf):\n","        sum_prob += prob\n","        if(sum_prob > draw):\n","            return index, prob\n","\n","\n","def get_action(vw, context, actions):\n","    vw_text_example = to_vw_example_format(context, actions)\n","    pmf = vw.predict(vw_text_example)\n","    chosen_action_index, prob = sample_custom_pmf(pmf)\n","    return actions[chosen_action_index], prob\n","\n","\n","def choose_user(users):\n","    return random.choice(users)\n","\n","\n","def choose_context1(context1):\n","    return random.choice(context1)\n","\n","    \n","def choose_context2(context2):\n","    return random.choice(context2)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"xIvHNcNmnmUa"},"source":["class VWCSimulation():\n","    def __init__(self, vw, ictxt, n=100000):\n","        self.vw = vw\n","        self.users = ictxt['users'].unique().tolist()\n","        self.contexts1 = ictxt['context1'].unique().tolist()\n","        self.contexts2 = ictxt['context2'].unique().tolist()\n","        self.actions = ictxt['items'].unique().tolist()\n","        self.contextdf = ictxt.copy()\n","        self.contextdf['cost'] = self.contextdf['reward']*-1\n","        \n","    def get_cost(self, context, action):\n","        return self.contextdf.loc[(self.contextdf['users']==context['user']) & \\\n","                (self.contextdf.context1==context['context1']) & \\\n","                (self.contextdf.context2==context['context2']) & \\\n","                (self.contextdf['items']==action), \\\n","                'cost'].values[0]\n","    \n","    def update_context(self, new_ctxt):\n","        self.contextdf = new_ctxt.copy()\n","        self.contextdf['cost'] = self.contextdf['reward']*-1\n","    \n","    def step(self):\n","        user = choose_user(self.users)\n","        context1 = choose_context1(self.contexts1)\n","        context2 = choose_context2(self.contexts2)\n","        context = {'user': user, 'context1': context1, 'context2': context2}\n","        action, prob = get_action(self.vw, context, self.actions)\n","        cost = self.get_cost(context, action)\n","        vw_format = self.vw.parse(to_vw_example_format(context, self.actions, (action, cost, prob)), pyvw.vw.lContextualBandit)\n","        self.vw.learn(vw_format)\n","        self.vw.finish_example(vw_format)\n","        return (context['user'], context['context1'], context['context2'], action, cost, prob)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"R3--k-qNntVk","executionInfo":{"status":"ok","timestamp":1634748792419,"user_tz":-330,"elapsed":18,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"2b1bed0b-6042-480e-c1b5-a9ca7b6da1dc"},"source":["context = pd.DataFrame(list(product(users, context1, context2, items)), columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = np.random.choice([0,1],len(context),p=[0.8,0.2])\n","contextdf = context.copy()\n","contextdf.reward.value_counts()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    56\n","1    16\n","Name: reward, dtype: int64"]},"metadata":{},"execution_count":49}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Jr39uZXOnvVV","executionInfo":{"status":"ok","timestamp":1634748808175,"user_tz":-330,"elapsed":559,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"18ab736c-93b8-464c-970c-7958413cae4a"},"source":["vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","vws = VWCSimulation(vw, contextdf)\n","\n","vws.step()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["('usera', 'ctx12', 'ctx22', 'item2', 0, 0.16666666666666666)"]},"metadata":{},"execution_count":50}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":359},"id":"nrcPxEkynzPC","executionInfo":{"status":"ok","timestamp":1634748831964,"user_tz":-330,"elapsed":5874,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"73616137-7c78-4f3b-8818-ba55a91ea67c"},"source":["_temp = []\n","for i in range(5000):\n","    _temp.append(vws.step())\n","\n","x = pd.DataFrame.from_records(_temp, columns=['user','context1','context2','item','cost','prob'])\n","\n","\n","xx = x.copy()\n","xx['ccost'] = xx['cost'].cumsum()\n","xx = xx.fillna(0)\n","xx = xx.rename_axis('iter').reset_index()\n","xx['ctr'] = -1*xx['ccost']/xx['iter']\n","xx.sample(10)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>iter</th>\n","      <th>user</th>\n","      <th>context1</th>\n","      <th>context2</th>\n","      <th>item</th>\n","      <th>cost</th>\n","      <th>prob</th>\n","      <th>ccost</th>\n","      <th>ctr</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>1352</th>\n","      <td>1352</td>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item6</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-772</td>\n","      <td>0.571006</td>\n","    </tr>\n","    <tr>\n","      <th>2530</th>\n","      <td>2530</td>\n","      <td>usera</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item1</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1507</td>\n","      <td>0.595652</td>\n","    </tr>\n","    <tr>\n","      <th>581</th>\n","      <td>581</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-283</td>\n","      <td>0.487091</td>\n","    </tr>\n","    <tr>\n","      <th>4116</th>\n","      <td>4116</td>\n","      <td>userb</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item5</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2585</td>\n","      <td>0.628037</td>\n","    </tr>\n","    <tr>\n","      <th>2425</th>\n","      <td>2425</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item6</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1432</td>\n","      <td>0.590515</td>\n","    </tr>\n","    <tr>\n","      <th>3175</th>\n","      <td>3175</td>\n","      <td>usera</td>\n","      <td>ctx12</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1950</td>\n","      <td>0.614173</td>\n","    </tr>\n","    <tr>\n","      <th>1654</th>\n","      <td>1654</td>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>0</td>\n","      <td>0.833333</td>\n","      <td>-968</td>\n","      <td>0.585248</td>\n","    </tr>\n","    <tr>\n","      <th>4755</th>\n","      <td>4755</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item6</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-3018</td>\n","      <td>0.634700</td>\n","    </tr>\n","    <tr>\n","      <th>1857</th>\n","      <td>1857</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item6</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1093</td>\n","      <td>0.588584</td>\n","    </tr>\n","    <tr>\n","      <th>3771</th>\n","      <td>3771</td>\n","      <td>userb</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item1</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2367</td>\n","      <td>0.627685</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["      iter   user context1 context2   item  cost      prob  ccost       ctr\n","1352  1352  usera    ctx11    ctx21  item6     0  0.033333   -772  0.571006\n","2530  2530  usera    ctx12    ctx22  item1    -1  0.833333  -1507  0.595652\n","581    581  userc    ctx11    ctx22  item3    -1  0.833333   -283  0.487091\n","4116  4116  userb    ctx12    ctx22  item5    -1  0.833333  -2585  0.628037\n","2425  2425  userb    ctx11    ctx21  item6    -1  0.833333  -1432  0.590515\n","3175  3175  usera    ctx12    ctx21  item3    -1  0.833333  -1950  0.614173\n","1654  1654  usera    ctx11    ctx22  item3     0  0.833333   -968  0.585248\n","4755  4755  userc    ctx11    ctx21  item6     0  0.033333  -3018  0.634700\n","1857  1857  userb    ctx11    ctx21  item6    -1  0.833333  -1093  0.588584\n","3771  3771  userb    ctx12    ctx22  item1    -1  0.833333  -2367  0.627685"]},"metadata":{},"execution_count":51}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":282},"id":"W2wHOA4wn3rr","executionInfo":{"status":"ok","timestamp":1634748834940,"user_tz":-330,"elapsed":1025,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"30b46d13-41f2-4a8b-fcd2-957809c50871"},"source":["xx['ccost'].plot()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.axes._subplots.AxesSubplot at 0x7f31db251f50>"]},"metadata":{},"execution_count":52},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dZ3hVZdr28f+VEJpEQgk1dBCIdLZIFwQRQYwwqGABUUEsCOIz1nHGZxxnFEUBaTKKgqMiggKigID0HqT30ARECVKll/v9kEWevAw9ZWXvnL/jWAdrX2vtva8bg2dWN+ccIiIiAGF+NyAiIpmHQkFERJIpFEREJJlCQUREkikUREQkWTa/G0itggULutKlS/vdhohIUFm6dOle51z0+fWgD4XSpUsTHx/vdxsiIkHFzLZfqK7dRyIikkyhICIiyRQKIiKSTKEgIiLJFAoiIpIs04WCmbU0sw1mlmBmL/rdj4hIVpKpQsHMwoFBwB1ALNDRzGL97UpEJOvIbNcp1AESnHNbAMxsFBAHrE3rL/pm2U6OnDhDxSKR1C6Zj7AwS+uvEBEJOpktFIoDO1K83gncfP5KZtYN6AZQsmTJa/qib1fs5sf1ewCoVTKKP99eiXrlClzTZ4mIhIpMtfvoSjnnhjnnAs65QHT0f12lfUU+6hxg6rONef3uKmz7/Sgd/72QtoPnsWLHgTTuVkQkeGS2LYVdQIkUr2O8WpozMyoUjqRC4UjuqR3DfxZuZ/DMzcQNmked0vlpWKEgLasUoXx0Hu1aEpEswzLT4zjNLBuwEWhGUhgsAe53zq252HsCgYBLq3sfJR4+wej4HYxcsI3fDp0AICZfLlrEFqFRhYI0qRiNmQJCRIKfmS11zgX+q56ZQgHAzFoB/YBwYLhz7o1LrZ+WoZDSrgPH+GHNr0xa9SvLdx7g5OmzVCoSyQt3VKJpxUJp/n0iIhkpaELhaqVXKKR04vQZvv5pF+9P38QvB49Tq2QU9wZK0LZWcXJkC0/X7xYRSQ8KhTRw8vRZRi7YxpdLdrBpzx8Uj8rFq3fG0iK2sI47iEhQUSikIeccMzbs4Z/frydhzx/EFr2ef7arSo0SURnah4jItbpYKATlKal+MzNurVSYiT0a8nrcjfx+5ATth8zn9YlrWbnzAMEetCKSdWlLIQ0cPHqKl8et4ruVu5N6KpWP1+66kSrF8/ral4jIxWj3UQZIPHyC8ct3MXTWFvYfPUmPW8vT49YKhOt4g4hkMtp9lAGiI3PwWKOyTOvdmFZVi9Jv2ibaDp7Hut2H/G5NROSKKBTSQVTu7LzfsSbv3VedXfuPcdfAubw5aT17/zjhd2siIpekUEhHbWvGMOXZxrSuWpShszbT4M0feeWbVSQeVjiISOakYwoZZNNvhxk2ewvjlu8iPMx4vHE5ujUuy3U5Mtvtp0QkK9CB5kwiYc9h+kzewA9rf6PAddn5Z7uqtIgtrHsqiUiG0oHmTKJ8oUiGdQow9ol6REfm4PFPl/LQR4vZ/vsRv1sTEVEo+KV2qfyMe6oBf70zlqXb93NH/zl8unA7p8+c9bs1EcnCFAo+yhkRziMNyzD9uVuoWjwvr45bTesBcxm/fBfHT53xuz0RyYIUCplAsahcfN61Lu/eW52Dx07Rc9RyWrw3m7mb9vrdmohkMQqFTCI8zGhXK4Y5LzRl2EO1MYOHhi/iHxPXsu/ISb/bE5EsQqGQyUSEh9HixiJM7tmYe2rH8OHcrdzadyazNibqRnsiku4UCplUruzh9GlfnYk9GpIvd3Y6D1/MoyPiWf+rbpkhIulHoZDJVSmel++facRLd1Ri/ua9tOw3h5e+XqkD0SKSLhQKQSBX9nAev6UcC19qxqMNy/DF4h20HjCHNb8c9Ls1EQkxCoUgEpU7O6/eGcvHXW7i8PHTdPhgITM27PG7LREJIQqFINS0YiHGP92AEvlz89iIeD5dsM3vlkQkRCgUglTRvLkY3b0eTW6I5tXxa3h7ynoOHT/ld1siEuTSLRTM7DUz22Vmy72pVYplL5lZgpltMLPbU9RberUEM3sxvXoLFXlyZOODh2rTvnYMg2Yk3Zr70wXbOHNWp66KyLVJ7y2F95xzNbzpewAziwU6ADcCLYHBZhZuZuHAIOAOIBbo6K0rl5AtPIy321fjy251qVz0el4dv4ZHRyxhvy54E5Fr4MfuozhglHPuhHNuK5AA1PGmBOfcFufcSWCUt65chplxc9kCfNmtLq/H3cj8hN+5te9MRi/Zoa0GEbkq6R0KT5vZSjMbbmb5vFpxYEeKdXZ6tYvV/4uZdTOzeDOLT0xMTI++g5KZ8VC90nzzVH3KRufh+bEraTt4Hit3HvC7NREJEqkKBTObZmarLzDFAUOAckANYDfQNw36BcA5N8w5F3DOBaKjo9PqY0PGjcXyMvrxerzdvhq/HTrOn4bM59OF23WbDBG5rFQ9C9I51/xK1jOzfwMTvZe7gBIpFsd4NS5Rl6sUHmbcEyhBs8qF6TlqGa+OW820tb/xQstKxBa73u/2RCSTSs+zj4qmeNkWWO3NTwA6mFkOMysDVAAWA0uACmZWxsyyk3QwekJ69ZdV5L8uOyO61OG5227gp+37ufP9Ofzr+3UcPXna79ZEJBNKz6fG9zGzGoADtgGPAzjn1pjZaGAtcBp4yjl3BsDMngamAOHAcOfcmnTsL8sICzN6NKvAQ/VK8cZ36/hg9ha+XfELvW67gXtqx+j50CKSzIJ9P3MgEHDx8fF+txFU5m/ey1uT1rNi50GqxeTln22rUqV4Xr/bEpEMZGZLnXOB8+u6ojkLql+uIOOeasBbf6rK7oPHuWvgXPpN28iJ07rzqkhWp1DIosyM+24qydRnG9OmejH6TdtEgzd/ZMqaX/1uTUR8pFDI4qJyZ6d/h5p83OUmoiNz8vinS+k5ahk79x/1uzUR8YFCQYCkO6+Oe6o+3W8px3crd9P0nZkMmL5JV0SLZDEKBUmWI1s4L95RidnPN+X2G4vw7tSNdBsZz55Dx/1uTUQyiEJB/kuxqFy837Emr7WJZebGRJq8M5Ov4ndw+sxZv1sTkXSmUJALMjMeblCGH55tTMUikfx5zEruG7aQrXuP+N2aiKQjhYJcUrnoPIzpXp8+7auxOfEP7ho4l1kbdRNCkVClUJDLCg8z7g2U4NunG1I8KhcPf7yYv45fzd4/TvjdmoikMYWCXLES+XMz5on6dLipBJ8v+pmGb/3Il0t+9rstEUlDCgW5KnlyZONf7arx3TONuKl0fl4Yu4rnRq9g98FjfrcmImlAoSDXpGKRSIY/fBOPNCjD+OW7aPHebL6K33H5N4pIpqZQkGsWER7GX9vEMrlXI8oXysOfx6zk1XGrOX5K91ASCVYKBUm18oUiGdO9Pl0bleHThdtpN3g+CXsO+92WiFwDhYKkifAw45XWsXzUOcDug8do8/48hs7arDuvigQZhYKkqWaVCzO5V2MCpfPx5qT13NFvDvM37/W7LRG5QgoFSXOFr8/Jp4/ezIedApxxjgc/XMQ/v1/HsZPaahDJ7BQKkm6axxZmYo+GtK5WjGGzt9Bp+CL2Hznpd1sicgkKBUlXkTkjeL9jTQbeX5PlOw7QasAcvlzys441iGRSCgXJEHdWK8bYJ+oTmTMbL4xdRbvB85mxYY/fbYnIeRQKkmGqxUQxpVdj+neoQeLhE3T5eAlvTV6vB/mIZCIKBclQZkZcjeLMeaEpHeuUYMjMzdzadyaTVu3W8xpEMoFUhYKZ3WNma8zsrJkFzlv2kpklmNkGM7s9Rb2lV0swsxdT1MuY2SKv/qWZZU9Nb5K55cgWzr/aVWPg/TU5efosT3z2Ew9+tIiff9ezoUX8lNothdVAO2B2yqKZxQIdgBuBlsBgMws3s3BgEHAHEAt09NYFeAt4zzlXHtgPPJrK3iQI3FmtGLOfb8obbauwcudBWg+Yw7crfvG7LZEsK1Wh4Jxb55zbcIFFccAo59wJ59xWIAGo400JzrktzrmTwCggzswMuBUY471/BHB3anqT4BERHsYDN5diSq/GlI2+jh5fLOOJ/yzV8xpEfJBexxSKAylvmbnTq12sXgA44Jw7fV79gsysm5nFm1l8YqKeAhYqSuTPzVfd6/Ns8xuYvn4PjfvM4NMF2zirA9EiGeayoWBm08xs9QWmuIxo8EKcc8OccwHnXCA6OtqvNiQdZM8WRs/mFfj+mYbULBnFq+PX0Pr9ucRv2+d3ayJZQrbLreCca34Nn7sLKJHidYxX4yL134EoM8vmbS2kXF+yoPKFIvnPozczfvkvvDlpPe2HLqB1taK80qoyxaJy+d2eSMhKr91HE4AOZpbDzMoAFYDFwBKggnemUXaSDkZPcM45YAbQ3nt/Z2B8OvUmQcLMuLtmcaY825jHG5flhzW/0uCtH3nlm1UcPn7K7/ZEQlJqT0lta2Y7gXrAd2Y2BcA5twYYDawFJgNPOefOeFsBTwNTgHXAaG9dgBeA3maWQNIxho9S05uEjry5InipVWWm927CQ3VL8fnin2nx3mx++nm/362JhBxL+iU9eAUCARcfH+93G5KBFm/dR89Ry9hz+ARdG5WlV/MK5IwI97stkaBiZkudc4Hz67qiWYJOnTL5mdyrMW2qFWXorM3c98ECPelNJI0oFCQo5c0VQb8ONRl0fy02Jx7hjv5zeOO7tezcryuiRVJDoSBBrXW1ovz43C20qVaMD+dupek7M3l7ynrdmlvkGikUJOgVuj4n795Xg2m9b+H2G4swaMZmOn20mD2HjvvdmkjQUShIyCgXnYeB99fivfuqs+znA7QdPJ9lOkNJ5KooFCTktK0Zw9dP1sc5x5+GzOfdqRv1zAaRK6RQkJBUpXheJvVsTJvqxRgwfRMdhy1k4286Q0nkchQKErLy5o6gf4ea9GlfjXW7D3FH/zkMmblZD/MRuQSFgoS8ewMl+PF/mnBb5cK8NXk9Ld6bzYj52zh5WuEgcj6FgmQJ0ZE5GPJgLd7vWJPIXBH8bcIaHvpoEct3HPC7NZFMRaEgWYaZ0aZ6McY9WZ8+f6rG8h0HuHvQPPr+sIF9R0763Z5IpqBQkCzHzLj3phIsfrk5d1Uvxvs/JhD4x1T+NWmddilJlqdQkCwrb+4I+t1Xg1Hd6nJ3jeJ8MGsLTd+ZydilO/1uTcQ3CgXJ0sLCjLplC/DufTX4uMtN5L8uO899tYIn/rOUoydPX/4DREKMQkHE07RiIb55sj7P3XYDk9f8yl0D5zF17W+c0imskoUoFERSyBYeRo9mFfikSx3+OH6ariPjadxnBh/O2cKOfboDq4Q+PWRH5CKOnjzNpFW/MnLhdlbsOEBEuPHqnbF0rFOSiHD9PiXB7WIP2VEoiFyGc46EPX/w94lrmbNpL5WKRPLOPdWpUjyv362JXDM9eU3kGpkZFQpHMqJLHQZ0rMneP07QbvB8Ri7YRrD/UiVyPoWCyBUKCzPuql6MH569hZvL5uev49dwz9AFLNj8u8JBQoZCQeQq5b8uOyO61OGNtlVISPyDjv9eSLN3ZzE/Ya/frYmkmkJB5BqEhRkP3FyKBS8241/tqnL6jOOBjxYxdNZmbTVIUEtVKJjZPWa2xszOmlkgRb20mR0zs+XeNDTFstpmtsrMEsxsgJmZV89vZlPNbJP3Z77U9CaSEXJlD6djnZJM7tWIVlWK8uak9XQavpid+3X6qgSn1G4prAbaAbMvsGyzc66GN3VPUR8CdAUqeFNLr/4iMN05VwGY7r0WCQq5s2dj4P01ea1NLIu37qNZ31mMX77L77ZErlqqQsE5t845t+FK1zezosD1zrmFLmkbeyRwt7c4DhjhzY9IURcJCmbGww3KMPXZW6haPC89Ry3nudErOHT8lN+tiVyx9DymUMbMlpnZLDNr5NWKAynvNrbTqwEUds7t9uZ/BQpf7IPNrJuZxZtZfGJiYpo3LpIaJQvk5otudXmqaTm+XraT5n1n8VX8Dt2BVYLCZUPBzKaZ2eoLTHGXeNtuoKRzribQG/jczK6/0qa8rYiLHq1zzg1zzgWcc4Ho6Ogr/ViRDBMRHsafb6/EmO71iI7MwZ/HrOTWvjOZvu43HYiWTC3b5VZwzjW/2g91zp0ATnjzS81sM3ADsAuISbFqjFcD+M3Mijrndnu7mfZc7feKZDa1S+VnwtMNmbF+D/+ctI5HR8TTqmoR3rmnOrmzX/afn0iGS5fdR2YWbWbh3nxZkg4ob/F2Dx0ys7reWUedgPHe2yYAnb35zinqIkEtPMxoHluY759pRM9mFZi0+lfiBs5j5gb93iOZT2pPSW1rZjuBesB3ZjbFW9QYWGlmy4ExQHfn3D5v2ZPAh0ACsBmY5NXfBG4zs01Ac++1SMjIGRHOs7fdwMhH6nDs1Bke/ngJg2YkaHeSZCq6IZ6ID46fOsOLY1cybvkv1Cmdn/+Nu5HKRa/4sJtIqumGeCKZSM6IcN69twZ/j7uR9b8eou3geXyx+GdtNYjvFAoiPgkLMzrVK820526hekwUL329iqc/X8a+Iyf9bk2yMIWCiM8KRebki651eb5lRaas+ZVmfWcye6OuvxF/KBREMoGwMOPJJuX57plGFMyTg07DF/PS16s4cuK0361JFqNQEMlEKhaJ5NseDenWuCyjlvzMfcMWkLDnsN9tSRaiUBDJZHJGhPNyq8p82CnA1sQjtBowl88X6SC0ZAyFgkgm1axyYWb8uQk3l8nPy9+s4rER8ez944TfbUmIUyiIZGKFInMyoksdXrqjEnM27aX9kPls3XvE77YkhCkURDK5sDDj8VvK8UW3uhw4doo2789l0qrdl3+jyDVQKIgEidql8jGxR0PKFcrDU5//xJdLfva7JQlBCgWRIBKTLzdfdL2ZBuUL8sLYVfz927V6ToOkKYWCSJDJnT0bH3W+iQfrlmT4vK00e3emzk6SNKNQEAlC2bOF8Y+7q/LvTgGicmXn5W9W0Wn4Yp2dJKmmUBAJYrfFFmbC0w14Pe5GlmzbR8t+s/l+1W7OntVWg1wbhYJIkDMzHqpXmjHd6xOZM4InP/uJbp/G6xYZck0UCiIhokrxvEzq2YhX74xl+vo9BP4xjRHzt+lYg1wVhYJICMkZEc6jDcvw1eP1CJTOx98mrOGRT5aw++Axv1uTIKFQEAlBgdL5GflIHV5rE8v8zb/T5v15rPnloN9tSRBQKIiEKDPj4QZl+LZHQyLCjbaD5jNk5mbtTpJLUiiIhLgbCkcy7qkGNKkYzVuT1/PAh4vYc+i4321JJqVQEMkCCl+fkw8eqs0/7q7CTz/v57b3ZjN5te6fJP9NoSCSRZgZD9YtxcQeDSmZPzfd//MTz3yxjANH9Uxo+T+pCgUze9vM1pvZSjP7xsyiUix7ycwSzGyDmd2eot7SqyWY2Ysp6mXMbJFX/9LMsqemNxG5sPKFIvmqez2eaVaBb1f+QuM+M/jPwu061iBA6rcUpgJVnHPVgI3ASwBmFgt0AG4EWgKDzSzczMKBQcAdQCzQ0VsX4C3gPedceWA/8GgqexORi8gZEU7v225g3JMNqFz0ev4ybjVdRy4l8bBuk5HVpSoUnHM/OOfOXTa5EIjx5uOAUc65E865rUACUMebEpxzW5xzJ4FRQJyZGXArMMZ7/wjg7tT0JiKXV71EFF90rctfWldm1sY93N5vNku37/e7LfFRWh5TeASY5M0XB3akWLbTq12sXgA4kCJgztUvyMy6mVm8mcUnJiamUfsiWVNYmPFYo7JM7NGIPDmy0WHYAobP3ardSVnUZUPBzKaZ2eoLTHEp1nkFOA18lp7NnuOcG+acCzjnAtHR0RnxlSIhr2KRSL59uiENyhfk7xPX8vTnyzh0/JTfbUkGy3a5FZxzzS+13MweBu4Emrn/+9ViF1AixWoxXo2L1H8Hoswsm7e1kHJ9EckgeXNH8PHDNzF01hbe+WEDq3Yd5OMuN1EuOo/frUkGSe3ZRy2B54G7nHNHUyyaAHQwsxxmVgaoACwGlgAVvDONspN0MHqCFyYzgPbe+zsD41PTm4hcGzPjiSblGP14Xf44cZpW/ecwbtku7U7KIlJ7TGEgEAlMNbPlZjYUwDm3BhgNrAUmA0855854WwFPA1OAdcBob12AF4DeZpZA0jGGj1LZm4ikQu1S+Rn/VAOqFs9Lry+X0+WTJboddxZgwZ7+gUDAxcfH+92GSMg6efosH8/byluT11O6wHX0vbc6NUvm87stSSUzW+qcC5xf1xXNInJJ2bOF8fgt5fikSx2OnzpDuyHz6TdtI2f0dLeQpFAQkSvS+IZoJvVsTOuqRek3bROPjVjCz78fvfwbJagoFETkiuXNHcHA+2vxtzaxLNyyj5b9ZzNoRgInTp/xuzVJIwoFEblqXRqUYdpzt1CvbAHenrKBlv3msGqnHuITChQKInJNikfl4sPOAT5++CaOnTxD3KC5DPxxEydPn/W7NUkFhYKIXDMzo2mlQkzu1YhWVYvyzg8b6TBsATv361hDsFIoiEiqReXOzsD7a9G/Qw3W7T7MrX1n8f70TZw+o62GYKNQEJE0E1ejOFN6NaZpxWj6Tt1I548Xs/+IHuITTBQKIpKmShbIzQcPBejTvhpLtu6n4Vs/Mm6ZbmUWLBQKIpIu7g2UYMwT9ahU9Hp6fbmcftM26v5JQUChICLpplpMFKO61aVdreL0m7aJF8eu4pSOM2Rql711tohIakSEh9H3nurEROViwI8JrNl9kA8eClA8KpffrckFaEtBRNKdmdG7RUUGdKzJ1sQj3P7ebL5ftdvvtuQCFAoikmHuql6MST0bU75QHp787CdeGLOSg8f0dLfMRKEgIhmqZIHcjH68Ht0al2X00h20eG8WC7f87ndb4lEoiEiGy54tjJdbVeabJxsQER5Gx38v5NVxqzl+SjfW85tCQUR8U6NEFFN6NaZT3VJ8unA7cQPnsXqXbqznJ4WCiPjquhzZ+N+4Kgx/OMDvR07SZuBc+kxerxvr+UShICKZwq2VCjOlVyPiqhdj8MzNNH1nJlPW/Op3W1mOQkFEMo0CeXLw3n01+LBTgGzhxuOfLqXXqGXsOXzc79ayDIWCiGQqZkbz2MJM630LTzYpx/erfqVlvzlM0nUNGUKhICKZUkR4GM+3rMTEZxpSLConT3z2Ez2+WMaBo7rranpKVSiY2dtmtt7MVprZN2YW5dVLm9kxM1vuTUNTvKe2ma0yswQzG2Bm5tXzm9lUM9vk/ZkvdUMTkVBwQ+FIxj5Rn2eaVWDy6t20HjCXBZt1XUN6Se2WwlSginOuGrAReCnFss3OuRre1D1FfQjQFajgTS29+ovAdOdcBWC691pEhBzZwul92w2MfrweYWFw/4cLeWHMSo6cOO13ayEnVaHgnPvBOXfuv8pCIOZS65tZUeB659xCl3QP3ZHA3d7iOGCENz8iRV1EBICaJfMxuWfSdQ2jl+6g/dAFbN17xO+2QkpaHlN4BJiU4nUZM1tmZrPMrJFXKw7sTLHOTq8GUNg5d+5I0q9A4Yt9kZl1M7N4M4tPTExMo/ZFJBicu67h44dv4pcDx2jVfw7vTd3IsZO6GjotXDYUzGyama2+wBSXYp1XgNPAZ15pN1DSOVcT6A18bmbXX2lT3lbERZ/G4Zwb5pwLOOcC0dHRV/qxIhJCmlQsxKSejbi5bH76T99E83dn6WroNHDZ5yk455pfarmZPQzcCTTz/meOc+4EcMKbX2pmm4EbgF38/7uYYrwawG9mVtQ5t9vbzbTnKsciIllMsahcfNKlDjPW7+HFr1cSN2geneqV4qmm5SmYJ4ff7QWl1J591BJ4HrjLOXc0RT3azMK9+bIkHVDe4u0eOmRmdb2zjjoB4723TQA6e/OdU9RFRC6paaVCTOnVmLgaxRgxfxsN3/qRftM2cvCobst9tSw1z0w1swQgB3Du/LCFzrnuZvYn4O/AKeAs8Dfn3LfeewLAJ0Auko5B9HDOOTMrAIwGSgLbgXudc/su10MgEHDx8fHXPAYRCS0Jew7zj+/WMXNDInlyZOPlVpXpWKcE3tnv4jGzpc65wH/Vg/1B2goFEbmQVTsP8sb3a1m4ZR8P1y/NX++MJSxMwXDOxUJBVzSLSEiqGpOXzx6ry2MNy/DJ/G10HRmvq6GvgEJBREJWeJjxSuvKvNYmltmbEokbNI+l2/f73VamplAQkZBmZjzcoAyjutXl5Omz3PfBAj6au5Vg33WeXhQKIpIl1C6Vn8m9GtP4hmhen7iWriOX8tsh3ZL7fAoFEcky8uaK4KPOAf7SujIzN+zhzvfnanfSeRQKIpKlmBmPNSrLxGcakjt7OB2GLeC1CWvYffCY361lCgoFEcmSKhW5nrFP1KdNtWKMXLCNRm/N4MEPF5F4+ITfrflKoSAiWVbBPDl4974azPifJjxYtxTx2/fR4r1ZjF6yI8seiFYoiEiWV6rAdbx2142M6V6f0gWv4/mxK7l70DxW7cx6N9hTKIiIeKoUz8vY7vXp074a2/cd5a5Bc/nr+NWcOJ11bsutUBARSSEszLg3UII5zzflobqlGLlgO20HzWflzgN+t5YhFAoiIhcQmTOCv8dVYfADtdhz+Dh3DZzHX8atCvlbZSgUREQuoVXVokx/rgkP3FyS/yz8mabvzGT88l2cPRuaB6IVCiIil5E3VwRvtK3KhKcbUPj6nPQctZyuI+ND8nkNCgURkStULSaK755pxF9aV2b2pkRu7zebCSt+CanTVxUKIiJXITws6YroMd3rE5U7gme+WEbv0Ss4fio0zlBSKIiIXIPqJaL4tkdDetxannHLd9Fp+OKQ2J2kUBARuUYR4WE816Ii/TvU5Kft+2nZfzZLt1/2KcKZmkJBRCSV7qpejLFP1Cc8zPjTkAX0/WFD0B5nUCiIiKSB6iWSDkLfUzuG939MoOvIpezcf9Tvtq6aQkFEJI3kzRVBn/bVeKVVZeZsSqRZ31lMWPGL321dFYWCiEgaMjO6Ni7LtN63UD0mime+WMY/v1/H0ZOn/W7tiqQ6FMzsdTNbaWbLzewHMyvm1c3MBphZgre8Vor3dDazTd7UOUW9tpmt8t4zwMwstf2JiEJ+Wp4AAAfxSURBVPihRP7cjHy0DvcGYhg2ewvtBs/nlwOZ/0E+abGl8LZzrppzrgYwEfirV78DqOBN3YAhAGaWH/gbcDNQB/ibmeXz3jME6JrifS3ToD8REV/kjAinT/vqjHykDjv3H6P5u7N4f/qmTH3X1VSHgnPuUIqX1wHnDrnHASNdkoVAlJkVBW4Hpjrn9jnn9gNTgZbesuudcwtd0mH7kcDdqe1PRMRvjW+I5tseDalfriB9p27kjn5zmL95r99tXVCaHFMwszfMbAfwAP+3pVAc2JFitZ1e7VL1nReoX+j7uplZvJnFJyYmpsUQRETSVZmC1/Fh5wDDHqrNidNnuf/fi+gzeX2mO3X1ikLBzKaZ2eoLTHEAzrlXnHMlgM+Ap9OzYe/7hjnnAs65QHR0dHp/nYhImmlxYxGm9b6FDjeVYPDMzfT4Yhm/HTrud1vJsl3JSs655lf4eZ8B35N0zGAXUCLFshivtgtocl59plePucD6IiIhJVf2cP7Vriol8uem7w8bmL0xkf4datK0UiG/W0uTs48qpHgZB6z35icAnbyzkOoCB51zu4EpQAszy+cdYG4BTPGWHTKzut5ZR52A8antT0QkMzIznmpanqm9b6FYVC66fLKENyet9/05DVe0pXAZb5pZReAssB3o7tW/B1oBCcBRoAuAc26fmb0OLPHW+7tz7tzNQp4EPgFyAZO8SUQkZJWLzsM3TzbgtQlrGDprM8t37KfffTUpkjenL/1YZjvIcbUCgYCLj4/3uw0RkVRxzvFV/E7+NmENubKH83KryvypVnHS63ItM1vqnAucX9cVzSIimYCZce9NJZjwdAOKReXkf75aQbsh89m290iG9qFQEBHJRCoUjmT8Uw35x91V2Lb3CG0Hz2Pupoy7pkGhICKSyYSHGQ/WLcU3TzYgOjIHDw1fxLtTN3L6zNl0/26FgohIJlW64HWMfaI+baoVY8D0TbQfuiDdtxoUCiIimVhkzgj6d6hB/w41+HnfUR78aBHPjV7BHyfS566rCgURkUzOzIirUZwFL93KE03KMfannbR4dxZb0+EgtEJBRCRI5MgWzgstK/Flt7qULxxJ3lwRaf4daXHxmoiIZKCbyxbg5rIF0uWztaUgIiLJFAoiIpJMoSAiIskUCiIikkyhICIiyRQKIiKSTKEgIiLJFAoiIpIs6B+yY2aJJD3x7VoUBDLunrSZg8acNWjMoS+14y3lnIs+vxj0oZAaZhZ/oScPhTKNOWvQmENfeo1Xu49ERCSZQkFERJJl9VAY5ncDPtCYswaNOfSly3iz9DEFERH5/2X1LQUREUlBoSAiIsmybCiYWUsz22BmCWb2ot/9XCszG25me8xsdYpafjObamabvD/zeXUzswHemFeaWa0U7+nsrb/JzDr7MZYrZWYlzGyGma01szVm1tOrh+y4zSynmS02sxXemP/Xq5cxs0Xe2L40s+xePYf3OsFbXjrFZ73k1TeY2e3+jOjKmVm4mS0zs4ne65Aes5ltM7NVZrbczOK9Wsb9bDvnstwEhAObgbJAdmAFEOt3X9c4lsZALWB1ilof4EVv/kXgLW++FTAJMKAusMir5we2eH/m8+bz+T22S4y5KFDLm48ENgKxoTxur/c83nwEsMgby2igg1cfCjzhzT8JDPXmOwBfevOx3s97DqCM9+8g3O/xXWbsvYHPgYne65AeM7ANKHheLcN+trPqlkIdIME5t8U5dxIYBcT53NM1cc7NBvadV44DRnjzI4C7U9RHuiQLgSgzKwrcDkx1zu1zzu0HpgIt07/7a+Oc2+2c+8mbPwysA4oTwuP2ev/DexnhTQ64FRjj1c8f87m/izFAMzMzrz7KOXfCObcVSCDp30OmZGYxQGvgQ++1EeJjvogM+9nOqqFQHNiR4vVOrxYqCjvndnvzvwKFvfmLjTto/z68XQQ1SfrNOaTH7e1GWQ7sIekf+WbggHPutLdKyv6Tx+YtPwgUIMjGDPQDngfOeq8LEPpjdsAPZrbUzLp5tQz72c52rV1LcHDOOTMLyfOOzSwPMBbo5Zw7lPRLYZJQHLdz7gxQw8yigG+ASj63lK7M7E5gj3NuqZk18bufDNTQObfLzAoBU81sfcqF6f2znVW3FHYBJVK8jvFqoeI3bxMS7889Xv1i4w66vw8ziyApED5zzn3tlUN+3ADOuQPADKAeSbsLzv1yl7L/5LF5y/MCvxNcY24A3GVm20jaxXsr0J/QHjPOuV3en3tICv86ZODPdlYNhSVABe8shuwkHZSa4HNPaWkCcO5sg87A+BT1Tt4ZC3WBg94m6RSghZnl885qaOHVMiVvP/FHwDrn3LspFoXsuM0s2ttCwMxyAbeRdCxlBtDeW+38MZ/7u2gP/OiSjkBOADp4Z+qUASoAizNmFFfHOfeScy7GOVeapH+jPzrnHiCEx2xm15lZ5Ll5kn4mV5ORP9t+H2n3ayLpqP1GkvbLvuJ3P6kYxxfAbuAUSfsNHyVpP+p0YBMwDcjvrWvAIG/Mq4BAis95hKQDcAlAF7/HdZkxNyRpv+tKYLk3tQrlcQPVgGXemFcDf/XqZUn6H1wC8BWQw6vn9F4neMvLpvisV7y/iw3AHX6P7QrH34T/O/soZMfsjW2FN6059/+mjPzZ1m0uREQkWVbdfSQiIhegUBARkWQKBRERSaZQEBGRZAoFERFJplAQEZFkCgUREUn2/wAssPyy5WpfcgAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":282},"id":"x7A81yOWn5yF","executionInfo":{"status":"ok","timestamp":1634748842396,"user_tz":-330,"elapsed":706,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"173e5ecc-b7a7-4c06-b224-365ae10cde79"},"source":["xx['ctr'].plot()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.axes._subplots.AxesSubplot at 0x7f31db1cfb90>"]},"metadata":{},"execution_count":53},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAejElEQVR4nO3deXhc1Znn8e8rlUq7LS/yvsjGBmKMMSCMCQRowmIgg3sSQiAb0Ok4mW4ndCeZDHR6CIHuQKCbJJ1xT8IQMkl6CCHpdOIGJ2YJZCEsNovBCwZhG9vyJi+yLFtLleqdP6okl+SSVbakKt3r3+d5/Pjec4+q3qOn+Plw7r11zd0REZHgK8h3ASIiMjAU6CIiIaFAFxEJCQW6iEhIKNBFREIikq83Hj16tNfU1OTr7UVEAunll1/e7e7VmY7lLdBrampYuXJlvt5eRCSQzOzd3o5pyUVEJCQU6CIiIaFAFxEJCQW6iEhIKNBFREJCgS4iEhIKdBGRkMjbdegiImHh7rTEOtjW2EJJUSFVZVG2N7awac8hdh1opS2WYEJVCZGCAnYeaOXiU8Ywsap0wOtQoIuIpBxqj9PS3kFRpIDV9ftpaomx71CM7Y0tjCiPUlRYQLSwgDXb9rNxzyEKDbbvb2XTnoO0xhJZv89tV57KZy46acDrV6CLSM51PljHzLq1pe+nt9c3tlC3q5ltja3sbGpl78F21u84QFlxIbMnDOeMyVUUGJQWFRKNFFBVFiVSYGzee4jy4kI6EjCsNEJ5NMLB9ji7mtooKSpk78F2nqvbzcbdB9nW2EJ9Ywtt8b6DuaSogKkjy4klEowfXsKZU6oYP7yUURVRYvEEB9s7GF0RZcaYSsZUFlMcKeCdhoMAjBtewtSRZQP0m+xOgS4iWUsknH2H2tmRCtUDrXHcYcu+Q7TGOmhqiVNRkoyVaKFRXhxhx/5W2uIJdje30dyWnAFv3ZcMzwKDypIiDrTGiEYKmDa6golVpZw8toKEw/Pv7OaN+v0kejxYLRopYMrIMjbvjfPs+oZ+jamkqICTqiuYXl3OhSdXM3lkGe3xBFNGljG+qoSyaCGTR5SxvyVGR8KJJ5wJVSUURwqP6X3GDCvpV53ZUKCLSK/a4wle2riXp9bt5HdvNbBpz0GO96mVoyuKGVYSIVJozBhTwSWnjqGxJUZ7vINxw0poao2zoaGZ17c28tS6nQC8Z/wwbj5/GpNHlDKhqpTJI8uYMrKMkqJCCguSs/m9B9v5w9sNDCspwnH2t8Q41N5BwmFiVQnt8QTu0NgSI96RIBopoLqymI4EVJZEOH3icMqL+47CbPrk29CvUET6zd1pjSV4ZfM+6ve1UFyUvMBtRFmU3c1tbN3Xwo6mVva3xBhWUsS67U1s3nuI5rY47fEEhQXG3MlVzJ8+kkkjyhhZHqUsWkh1ZTHl0QjjUyf8yqKFtMY6aI0lf6Y11sGYYcVECwsyLqf0pqk1hjsMLy3qs+/I8igL50487t9NmCjQRULmYFucN3cc4E91u1m/8wC7m9t45d1G2juOvjZcWGBUlkRoiyWYXl3OqeMqmTqqjAtnVjN/+ihGlEezev+SomNbishkWEnfQS5HUqCLBExTa4xdTW24O8NKk7PphgNtyZN7ew7xxtbGrjXnSIExojzKB84YT3VlMVNGljF7wnDiCae5LU6hGVVlRYwbXkJFcWRAwljyR4EuMoS4Ox0JJ1KYXBJ5c0cTy97YwTu7mtmw+yDb97fQeCiW8WcriyPMGFvBTe+dxqnjKzlv+igmjSg9pqUOCTYFukgOuTvL1+zkibU7ONTWwWkThnEo1sHz7+yhqTVGoRkbdx9k6qgytuxt6VomiUYKqBlVxnnTRzG9upyyaIRIgdEeT1AzupyJI0qZM3F41z8EcmLKKtDNbAHwbaAQeNDd78nQ5zrgDsCBVe7+0QGsUyTQGg608ctX6/nJis1sSF2PDPCbNTsAGFUexcwYVhrhlHGVjKoo5uypI5g5ppKFcyfk5JI3Cb4+A93MCoElwGXAVmCFmS1197VpfWYCtwHnu/s+MxszWAWLDHXxjgSPv7Gd/S0x6nY18+b2A7y2tZH2eILJI0u579o5fPCsSbg7m/ceYkRZNOsTjiJHk80MfR5Q5+4bAMzsEWAhsDatz6eBJe6+D8Dddw10oSJDSXs8geNs2dvCixv3sLq+ifrGFjbubmbL3paufmYwYXgpV58+nhvfW8OcicMpKOhc0zamV1fkZwASStkE+kRgS9r+VuDcHn1OBjCz50guy9zh7r/p+UJmtghYBDBlypTjqVdOIE+u3ck/LV9PUcRYXd/E1FFlfP2/ns4f63YzdWQZq7ft5xPza6gZXUaBGUWFBby5o4kf/uldRpQV8fn3zzyuqzaaWmM8/84etuw9xMljK5k3bSSRAmPt9iaeq9tDc1uMh1/czL4eJyeLCo1Z44dxxqQqLjy5mpljKpg9cThFWteWHDHv47YvM7sWWODuf5na/wRwrrsvTuvzGBADrgMmAb8HTnf3xt5et7a21leuXNn/EUhguXvXXYeds9aGA2189P+8wNu7mgfkPS48uZr//bGz2NnUSjRSwL8++w5Pr9vJzqY2bj6/hsV/NoN9h2L83X+8wUsb92b9uqeMrWTm2ArGDSvhvJNG8b6Z1UQjCm4ZfGb2srvXZjqWzQy9Hpictj8p1ZZuK/Ciu8eAjWb2FjATWHEc9UpAxToStMY6eGtnM798tZ5JI0qJRgqYO7mK6aMrGFYaYePug3zpZ6t4ZXOv/9Z3qSyJ8PlLZjJzbAXzp49ixaa9fO0/1/KR2sms29HEVbPH862n32J1fVPXz0wZWcbfXDqTt3c188DvN/D7txo47avLM77+D57bxA+e23RE+1Wnj+OK08YxoaqUbY0tvLBhL+3xBKeOq2TB7HEAuhxQhqRsZugR4C3g/SSDfAXwUXdfk9ZnAXCDu99oZqOBV4G57r6nt9fVDD043J2lq7YxZ1IVlSURhpUUcdF9z7B9fysXnVzNjDEVfP+PG4/rtUuKCo742tGHP30u7z1p9ECUzi9frefOx9ay92A706vLuf6cyXzknClUFEd4cu0O/vanq5gysowlHzuLGWO0ni1D39Fm6H0GeuoFrgK+RXJ9/CF3/0czuxNY6e5LLTlV+WdgAdAB/KO7P3K011SgB0NzW5zZvcxwj9cN86bwgTnjOX3S8K5bvHc2tdLUEmPm2MoBfS+RsOl3oA8GBfrQ4+7JZYq3G4gWFvDMUb6WdF7NSP76khnc9IOXcIe7P3g6N8zTiW6RwdbfNXQ5Afzrs3Xc+5v1GY99+n3T+MrVs7r24x2JrjsSN959dU7qE5G+KdBPUK2xDt7dc4jiSAEX/9OzGft85sLpfHnBqV3fO91Jt5eLDE0K9BPIY69v40/v7OEvzq/h0vt/f8TxJ//2Qq1hiwSYAv0Ecd/yN1nyzDsAPPzi5m7HFs6dwDevm5t2B6OIBJECPaTcnVVb9/PZH7/MjqbWI46/b+Zofvypnjf8ikiQKdBDaHX9fj7wnT8e0f7crZcwsao0DxWJSC4o0AOuqTVGeTTC4odf4derd2Tsc970Ufzg5nP0NBqRkFOgB0jdrmbcnRHlUXbsb804C+/0pctPZvElM3NYnYjkmwJ9iDva9eE9/cOfz+ZHz2/iy1ecyqWzxg5uYSIy5CjQh7Dbf7WaHz3/bq/HiyMFrPrq5bgnvxPFzPj4/Kk5rFBEhhIF+hCSSDirt+3ni4+u6vb1sWdNqaKypIiHbjrniJt8REQ6KdCHkOl/t+yItm9fP5eFcyfmoRoRCRrdw51HzW1xzr7rSVZtaeSi+57pdmzSiFK+94mzFeYikjXN0PPo+3/YyJ6D7Sxc8lxX2/O3XcL44bpWXESOnWboefTwS91PeC7+sxkKcxE5bpqh58kdS9ews6mN68+ZzD0fmpPvckQkBDRDz4PWWAf/90+bAPjvV5yS32JEJDQU6Hnww1SYX3xKNaMqivNbjIiEhgI9xw60xrj7128C8OAnMz5FSkTkuCjQc+y3b+4CYM6k4Xryj4gMKJ0UzaF7fv0m3/1d8iETv/yr8/NcjYiEjaaIg8zdefz17WzafbArzKORAj0dSEQGXFYzdDNbAHwbKAQedPd7ehy/CbgPqE81/S93f3AA6wykX7yylS88uqpb2+iKKCu+cmmeKhKRMOsz0M2sEFgCXAZsBVaY2VJ3X9uj60/dffEg1BhYPcMcYMVXLsVMs3MRGXjZLLnMA+rcfYO7twOPAAsHt6ze/Wb1dmpufZz6xpZ8ldCnf35iPTW3Pt61/8EzJ/LUFy5k491XKcxFZNBkE+gTgS1p+1tTbT19yMxeN7Ofm9nkTC9kZovMbKWZrWxoaDiOcuGz//YKAHcvW3dcPz9YOhJOR8Jpi3fwnd/WdbX/dNF87v/IXGaMqVSYi8igGqirXP4T+Im7t5nZZ4AfApf07OTuDwAPANTW1np/3rBfPzyAEgnnxh+8xB/e3n3EsYc/fS7nTh+Vh6pE5ESUTaDXA+kz7kkcPvkJgLvvSdt9ELi3/6UFw+d+8mrGMF975xWURXVVqIjkTjZLLiuAmWY2zcyiwPXA0vQOZjY+bfcaYGithwySfQfbefyN7Ue03/TeGoW5iORcn6nj7nEzWwwsJ3nZ4kPuvsbM7gRWuvtS4PNmdg0QB/YCNw1izanCBv0djqo9nuDMu57s2t90z9V8/48bmTa6jEtO1QOaRST3sppGuvsyYFmPttvTtm8DbhvY0oauptYYc+54omt/zdeuAOBTF0zLV0kiIsG9U9TzOEVPD/PHPncB5cVaXhGR/AtsoOfDlr2Hul1fvujC6cyeODyPFYmIHBbYqaXnaIIe60hQt6uZW3/xBqu2NHa1lxQV8HdXvSc3RYiIZCGwgZ6Le3RufOglfvfWkTdARQqMN+64YvALEBE5BoEN9MG2affBjGEOUPf1q3JcjYhI3wIb6IO95PK5n7zatX3NGRPYsLuZuxbO5swpIwb3jUVEjlNgA30wtbR38Eb9fgDW/8MCiiOFea5IRKRvgb3KZTBn6O+5/Tdd2wpzEQmK4AZ62nXoq7Y08p2n3x7w97hz4WkD/poiIoMlFEsuC5c8B0BxUQEfPnsyI8qjx/1anjb1/+R5Nf0tTUQkZwI7Q8/k68ve5Mv//nq/XuMzP34ZgGvPnjQQJYmI5ExgA723NfR9B9v79bpPrN0JwCfPm9qv1xERybXgBnov7bub247pdRoOtHH/E+uJdyTYsb+1q33OpKp+VCciknuBXUPv7UbR3c3HNkP/2IMv8NbOZv70zh6qyooAuK5Wyy0iEjyBDfTetMcTWfc91B7nrZ3NAKx8d19X+x3X6OoWEQmewC65bNnXkrH97KnZ38m5oeFgxnY9bUhEgiiwgb5ue1PG9mP5bvJtjcl/FCYML+lqKy3SjUQiEkyBDfR0w0uLurZf27LvKD27W/JMHQDLbnkfVWVFXDBjNK/8z8sGvD4RkVwIxdpCWbSQ/S0x4NhOiq7amvy+lqqyKK/dfvmg1CYikiuhmKHHOrqfCPUsvujl3154d7DKERHJi1AEels8wQUzRhONJIfTEus4av+6XQf4+1+uBuCMybreXETCIfCB/vbOAxxojXPahGHc/oFZADS3xo/6M797a3fX9vxpIwe1PhGRXMkq0M1sgZmtN7M6M7v1KP0+ZGZuZrUDV+LRXfbN3wPJO0crS5KnBDbvPXTUn7nrsbVd21+8/JRBq01EJJf6DHQzKwSWAFcCs4AbzGxWhn6VwC3AiwNdZG/S184fW7WtK9Cv/e7zva6jp7dvuufqrmUaEZGgyybN5gF17r7B3duBR4CFGfrdBXwDaM1wbFA0Hop1bR9s76C57fDa+Tefyvz96Lc88tqg1yUikg/ZBPpEYEva/tZUWxczOwuY7O6PH+2FzGyRma00s5UNDZkfwHwsLO0LXQoLjKKCww3/0uOBF3/5wxXU3Po4S1dtA+D+687o9/uLiAwl/V5vMLMC4H7gi331dfcH3L3W3Wurq6v7+9YkEoeXT2ZPHM5ls8Z27X/03Cnd+j61ble3/amjyvv9/iIiQ0k2gV4PTE7bn5Rq61QJzAaeNbNNwHxgaS5OjMbSAt3diRQW8PQXLwLg4Rc3H/W70Y/lO19ERIIgm0BfAcw0s2lmFgWuB5Z2HnT3/e4+2t1r3L0GeAG4xt1XDkrFaWJp36y4cG5yFeik6oqutjPvehKAX71W3+3nnv3SxYNdmohIzvUZ6O4eBxYDy4F1wKPuvsbM7jSzawa7wKNpT7vK5WiPjEs/EXr3B0+nZrSWW0QkfLL6Lhd3XwYs69F2ey99L+5/WdnJ5rvPD7Ufvsnohdvez7i0b1YUEQmTQF+E3fM7XDrNn3747s/Ob1Q0Q2EuIqEW6EBf9OOXAfhwj+WWBz5Z2/UdLUueeQeAez80J7fFiYjkWKADveFA8oHQ150zuVv7sJIifnTzvG5trcfwaDoRkSAKdKB3ihYeOYxhpd1PD5RH9SQiEQm3UAR6UYZAt7TbSKdXl3dd1igiElYhCXQ76vFHPj2fwoKj9xERCbpQBHpv3jdzNAAjyqN5rkREZPCF4pmivT1w7rsfP5ut+1oyLsmIiIRNKJKuqqwoY3t5cYRTxlXmuBoRkfwIRaCPqdQNQyIigQ/0m8+vyXcJIiJDQuADfcaYir47iYicAAIf6IWmyxFFRCAEgV6g68tFRIAQBLpm6CIiSYEL9K/+l1kAzJ44DICCwI1ARGRwBC4OO2/h99TdRAWaoYuIAAEM9E7xjmSi6ztaRESSAhvosUTy+821hi4ikhTYQO9IJGfouspFRCQpsIHeteSiGbqICBDAQO88GRrvXHLRDF1EBMgy0M1sgZmtN7M6M7s1w/HPmtkbZvaamf3RzGYNfKnddS65aIIuIpLUZ6CbWSGwBLgSmAXckCGwH3b30919LnAvcP+AV9pDPKGrXERE0mUzQ58H1Ln7BndvBx4BFqZ3cPemtN1yen/mxIDRGrqISHfZPLFoIrAlbX8rcG7PTmb218AXgChwSaYXMrNFwCKAKVOmHGut3cQ6kmvouspFRCRpwE6KuvsSdz8J+B/A3/fS5wF3r3X32urq6n69X1s8GeidM3URkRNdNoFeD0xO25+UauvNI8Cf96eoY9HY0p6rtxIRGdKyCfQVwEwzm2ZmUeB6YGl6BzObmbZ7NfD2wJV4dK4JuogIkMUaurvHzWwxsBwoBB5y9zVmdiew0t2XAovN7FIgBuwDbhzMotMllOgiIkB2J0Vx92XAsh5tt6dt3zLAdYmIyDEK4J2i3WfkpUWFeapERGRoCVyg93Tpe8bmuwQRkSEh0IFupuvQRUQ6BTrQdT5UROSwQAe6iIgcpkAXEQkJBbqISEgo0EVEQkKBLiISEoELdF3YIiKSWeACXUREMlOgi4iEhAJdRCQkAh3oV88Zn+8SRESGjEAHus6QiogcFuhA70go0UVEOgU60PW0IhGRwxToIiIhEehA15KLiMhhgQv09El5h/JcRKRL4AI9XUIzdBGRLsEOdK2hi4h0ySrQzWyBma03szozuzXD8S+Y2Voze93MnjazqQNf6pG0hi4iclifgW5mhcAS4EpgFnCDmc3q0e1VoNbd5wA/B+4d6EIz0QxdROSwbGbo84A6d9/g7u3AI8DC9A7u/oy7H0rtvgBMGtgyM9MMXUTksGwCfSKwJW1/a6qtN58Cfp3pgJktMrOVZrayoaEh+yp7oatcREQOG9CTomb2caAWuC/TcXd/wN1r3b22urq63+/nWnIREekSyaJPPTA5bX9Sqq0bM7sU+Apwkbu3DUx5R6clFxGRw7KZoa8AZprZNDOLAtcDS9M7mNmZwPeAa9x918CXmZkCXUTksD4D3d3jwGJgObAOeNTd15jZnWZ2TarbfUAF8DMze83Mlvbycv2WHuFacREROSybJRfcfRmwrEfb7Wnblw5wXVnpUKKLiHQJ9p2iWnIREekS6EDXDF1E5LBgB7pm6CIiXQId6Jqgi4gcFuhAv+X9M/NdgojIkJHVVS5D0aqvXs7w0qJ8lyEiMmQEeoYuIiKHBS7Q9f0tIiKZBS7QO5nluwIRkaElsIEuIiLdKdBFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQkFOgiIiGhQBcRCQkFuohISAQ20HWjqIhId4ENdBER6U6BLiISEgp0EZGQyCrQzWyBma03szozuzXD8QvN7BUzi5vZtQNfpoiI9KXPQDezQmAJcCUwC7jBzGb16LYZuAl4eKALFBGR7GTzCLp5QJ27bwAws0eAhcDazg7uvil1LDEINYqISBayWXKZCGxJ29+aajtmZrbIzFaa2cqGhobjeQkREelFTk+KuvsD7l7r7rXV1dW5fGsRkdDLJtDrgclp+5NSbXmhR4qKiGSWTaCvAGaa2TQziwLXA0sHt6y+mR4qKiLSTZ+B7u5xYDGwHFgHPOrua8zsTjO7BsDMzjGzrcCHge+Z2ZrBLFpERI6UzVUuuPsyYFmPttvTtleQXIoREZE80Z2iIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQCFygO7pVVEQkk8AFeifdJyoi0l1gA11ERLpToIuIhIQCXUQkJBToIiIhoUAXEQkJBbqISEgo0EVEQiJwga5H0ImIZBa4QO+kJ9CJiHQX2EAXEZHuFOgiIiGhQBcRCQkFuohISCjQRURCIqtAN7MFZrbezOrM7NYMx4vN7Kep4y+aWc1AFyoiIkfXZ6CbWSGwBLgSmAXcYGazenT7FLDP3WcA3wS+MdCFiojI0WUzQ58H1Ln7BndvBx4BFvbosxD4YWr758D7zXSluIhILmUT6BOBLWn7W1NtGfu4exzYD4zq+UJmtsjMVprZyoaGhuMqeNrocq46fRwF+vdCRKSbSC7fzN0fAB4AqK2tPa6b+C8/bRyXnzZuQOsSEQmDbGbo9cDktP1JqbaMfcwsAgwH9gxEgSIikp1sAn0FMNPMpplZFLgeWNqjz1LgxtT2tcBv3fU1WiIiudTnkou7x81sMbAcKAQecvc1ZnYnsNLdlwLfB35sZnXAXpKhLyIiOZTVGrq7LwOW9Wi7PW27FfjwwJYmIiLHQneKioiEhAJdRCQkFOgiIiGhQBcRCQnL19WFZtYAvHucPz4a2D2A5QSBxnxi0JhPDP0Z81R3r850IG+B3h9mttLda/NdRy5pzCcGjfnEMFhj1pKLiEhIKNBFREIiqIH+QL4LyAON+cSgMZ8YBmXMgVxDFxGRIwV1hi4iIj0o0EVEQiJwgd7XA6uDxMweMrNdZrY6rW2kmT1pZm+n/h6Rajcz+5fUuF83s7PSfubGVP+3zezGTO81FJjZZDN7xszWmtkaM7sl1R7mMZeY2Utmtio15q+l2qelHqhel3rAejTV3usD183stlT7ejO7Ij8jyp6ZFZrZq2b2WGo/1GM2s01m9oaZvWZmK1Ntuf1su3tg/pD8+t53gOlAFFgFzMp3Xf0Yz4XAWcDqtLZ7gVtT27cC30htXwX8GjBgPvBiqn0ksCH194jU9oh8j62X8Y4HzkptVwJvkXzweJjHbEBFarsIeDE1lkeB61Pt3wX+W2r7r4DvpravB36a2p6V+rwXA9NS/x0U5nt8fYz9C8DDwGOp/VCPGdgEjO7RltPPdt5/Ccf4CzsPWJ62fxtwW77r6ueYanoE+npgfGp7PLA+tf094Iae/YAbgO+ltXfrN5T/AL8CLjtRxgyUAa8A55K8SzCSau/6XJN87sB5qe1Iqp/1/Kyn9xuKf0g+2exp4BLgsdQYwj7mTIGe08920JZcsnlgddCNdfftqe0dwNjUdm9jD+TvJPW/1WeSnLGGesyppYfXgF3AkyRnmo2efKA6dK+/tweuB2rMwLeALwOJ1P4owj9mB54ws5fNbFGqLaef7Zw+JFqOjbu7mYXuulIzqwD+Hfgbd28ys65jYRyzu3cAc82sCvgP4NQ8lzSozOwDwC53f9nMLs53PTl0gbvXm9kY4EkzezP9YC4+20GboWfzwOqg22lm4wFSf+9Ktfc29kD9TsysiGSY/z93/0WqOdRj7uTujcAzJJcbqiz5QHXoXn9vD1wP0pjPB64xs03AIySXXb5NuMeMu9en/t5F8h/ueeT4sx20QM/mgdVBl/7A7RtJrjN3tn8ydXZ8PrA/9b9yy4HLzWxE6gz65am2IceSU/HvA+vc/f60Q2Eec3VqZo6ZlZI8Z7COZLBfm+rWc8yZHri+FLg+dUXINGAm8FJuRnFs3P02d5/k7jUk/xv9rbt/jBCP2czKzayyc5vkZ3I1uf5s5/tEwnGceLiK5NUR7wBfyXc9/RzLT4DtQIzkWtmnSK4dPg28DTwFjEz1NWBJatxvALVpr/MXQF3qz835HtdRxnsByXXG14HXUn+uCvmY5wCvpsa8Grg91T6dZDjVAT8DilPtJan9utTx6Wmv9ZXU72I9cGW+x5bl+C/m8FUuoR1zamyrUn/WdGZTrj/buvVfRCQkgrbkIiIivVCgi4iEhAJdRCQkFOgiIiGhQBcRCQkFuohISCjQRURC4v8DrAOtiLHqYyIAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":359},"id":"JtF5tbhOn7jf","executionInfo":{"status":"ok","timestamp":1634748887359,"user_tz":-330,"elapsed":7447,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"a531cd9b-56d9-4cc8-c804-80ff4df2d0cf"},"source":["tempdf1 = xx.copy()\n","\n","context = pd.DataFrame(list(product(users, context1, context2, items)), columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = 0\n","X = context.copy()\n","X.loc[(X['users']=='usera')&(X['items']=='item1'),'reward']=1\n","X.loc[(X['users']=='userb')&(X['items']=='item2'),'reward']=1\n","X.loc[(X['users']=='userc')&(X['items']=='item3'),'reward']=1\n","X.reward.value_counts()\n","\n","vws.update_context(X)\n","\n","_temp = []\n","for i in range(5000):\n","    _temp.append(vws.step())\n","\n","x = pd.DataFrame.from_records(_temp, columns=['user','context1','context2','item','cost','prob'])\n","xx = x.copy()\n","xx['ccost'] = xx['cost'].cumsum()\n","xx = xx.fillna(0)\n","xx = xx.rename_axis('iter').reset_index()\n","xx['ctr'] = -1*xx['ccost']/xx['iter']\n","xx.sample(10)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>iter</th>\n","      <th>user</th>\n","      <th>context1</th>\n","      <th>context2</th>\n","      <th>item</th>\n","      <th>cost</th>\n","      <th>prob</th>\n","      <th>ccost</th>\n","      <th>ctr</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>4316</th>\n","      <td>4316</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item2</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-3412</td>\n","      <td>0.790547</td>\n","    </tr>\n","    <tr>\n","      <th>4798</th>\n","      <td>4798</td>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3827</td>\n","      <td>0.797624</td>\n","    </tr>\n","    <tr>\n","      <th>4154</th>\n","      <td>4154</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3284</td>\n","      <td>0.790563</td>\n","    </tr>\n","    <tr>\n","      <th>4720</th>\n","      <td>4720</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3757</td>\n","      <td>0.795975</td>\n","    </tr>\n","    <tr>\n","      <th>4362</th>\n","      <td>4362</td>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3448</td>\n","      <td>0.790463</td>\n","    </tr>\n","    <tr>\n","      <th>3520</th>\n","      <td>3520</td>\n","      <td>usera</td>\n","      <td>ctx12</td>\n","      <td>ctx21</td>\n","      <td>item1</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2759</td>\n","      <td>0.783807</td>\n","    </tr>\n","    <tr>\n","      <th>2614</th>\n","      <td>2614</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2025</td>\n","      <td>0.774675</td>\n","    </tr>\n","    <tr>\n","      <th>4840</th>\n","      <td>4840</td>\n","      <td>userb</td>\n","      <td>ctx12</td>\n","      <td>ctx21</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3863</td>\n","      <td>0.798140</td>\n","    </tr>\n","    <tr>\n","      <th>3084</th>\n","      <td>3084</td>\n","      <td>userc</td>\n","      <td>ctx12</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2414</td>\n","      <td>0.782750</td>\n","    </tr>\n","    <tr>\n","      <th>4921</th>\n","      <td>4921</td>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item4</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-3932</td>\n","      <td>0.799025</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["      iter   user context1 context2   item  cost      prob  ccost       ctr\n","4316  4316  userc    ctx11    ctx21  item2     0  0.033333  -3412  0.790547\n","4798  4798  userc    ctx12    ctx22  item3    -1  0.833333  -3827  0.797624\n","4154  4154  userb    ctx11    ctx22  item2    -1  0.833333  -3284  0.790563\n","4720  4720  userc    ctx11    ctx22  item3    -1  0.833333  -3757  0.795975\n","4362  4362  userc    ctx12    ctx21  item3    -1  0.833333  -3448  0.790463\n","3520  3520  usera    ctx12    ctx21  item1    -1  0.833333  -2759  0.783807\n","2614  2614  userc    ctx11    ctx21  item3    -1  0.833333  -2025  0.774675\n","4840  4840  userb    ctx12    ctx21  item2    -1  0.833333  -3863  0.798140\n","3084  3084  userc    ctx12    ctx21  item3    -1  0.833333  -2414  0.782750\n","4921  4921  usera    ctx11    ctx22  item4     0  0.033333  -3932  0.799025"]},"metadata":{},"execution_count":54}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":359},"id":"m6Ej5cMHoE-O","executionInfo":{"status":"ok","timestamp":1634748887362,"user_tz":-330,"elapsed":28,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"f3d2ce8f-a9ba-48c9-c264-37ab4ddd7948"},"source":["tempdf2 = tempdf1.append(xx, ignore_index=True)\n","tempdf2.sample(10)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>iter</th>\n","      <th>user</th>\n","      <th>context1</th>\n","      <th>context2</th>\n","      <th>item</th>\n","      <th>cost</th>\n","      <th>prob</th>\n","      <th>ccost</th>\n","      <th>ctr</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>974</th>\n","      <td>974</td>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item1</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-534</td>\n","      <td>0.548255</td>\n","    </tr>\n","    <tr>\n","      <th>5527</th>\n","      <td>527</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-292</td>\n","      <td>0.554080</td>\n","    </tr>\n","    <tr>\n","      <th>9055</th>\n","      <td>4055</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3199</td>\n","      <td>0.788903</td>\n","    </tr>\n","    <tr>\n","      <th>3984</th>\n","      <td>3984</td>\n","      <td>usera</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-2508</td>\n","      <td>0.629518</td>\n","    </tr>\n","    <tr>\n","      <th>2327</th>\n","      <td>2327</td>\n","      <td>userc</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item3</td>\n","      <td>0</td>\n","      <td>0.833333</td>\n","      <td>-1375</td>\n","      <td>0.590890</td>\n","    </tr>\n","    <tr>\n","      <th>6606</th>\n","      <td>1606</td>\n","      <td>usera</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item5</td>\n","      <td>0</td>\n","      <td>0.033333</td>\n","      <td>-1182</td>\n","      <td>0.735990</td>\n","    </tr>\n","    <tr>\n","      <th>7398</th>\n","      <td>2398</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx22</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1852</td>\n","      <td>0.772310</td>\n","    </tr>\n","    <tr>\n","      <th>9051</th>\n","      <td>4051</td>\n","      <td>userb</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-3195</td>\n","      <td>0.788694</td>\n","    </tr>\n","    <tr>\n","      <th>7105</th>\n","      <td>2105</td>\n","      <td>userb</td>\n","      <td>ctx11</td>\n","      <td>ctx21</td>\n","      <td>item2</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1603</td>\n","      <td>0.761520</td>\n","    </tr>\n","    <tr>\n","      <th>1866</th>\n","      <td>1866</td>\n","      <td>usera</td>\n","      <td>ctx12</td>\n","      <td>ctx22</td>\n","      <td>item1</td>\n","      <td>-1</td>\n","      <td>0.833333</td>\n","      <td>-1099</td>\n","      <td>0.588960</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["      iter   user context1 context2   item  cost      prob  ccost       ctr\n","974    974  usera    ctx11    ctx21  item1     0  0.033333   -534  0.548255\n","5527   527  userc    ctx11    ctx22  item3    -1  0.833333   -292  0.554080\n","9055  4055  userb    ctx11    ctx21  item2    -1  0.833333  -3199  0.788903\n","3984  3984  usera    ctx11    ctx21  item3    -1  0.833333  -2508  0.629518\n","2327  2327  userc    ctx11    ctx21  item3     0  0.833333  -1375  0.590890\n","6606  1606  usera    ctx12    ctx22  item5     0  0.033333  -1182  0.735990\n","7398  2398  userb    ctx11    ctx22  item2    -1  0.833333  -1852  0.772310\n","9051  4051  userb    ctx12    ctx22  item2    -1  0.833333  -3195  0.788694\n","7105  2105  userb    ctx11    ctx21  item2    -1  0.833333  -1603  0.761520\n","1866  1866  usera    ctx12    ctx22  item1    -1  0.833333  -1099  0.588960"]},"metadata":{},"execution_count":55}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":282},"id":"WluzTjjgoGgJ","executionInfo":{"status":"ok","timestamp":1634748894913,"user_tz":-330,"elapsed":55,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"766bf74f-6313-4f7c-eba3-1ef7709493b2"},"source":["tempdf2['ccost'].plot()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.axes._subplots.AxesSubplot at 0x7f31db191050>"]},"metadata":{},"execution_count":56},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXgUVdbA4d/JvgFJIKwBEiAsAUICYXNXENBRwQFkEQFBUBEVwQXU0VGHEXRcPhRBBlBAkEVQUBEEdcaVJUAIYQ97WMMa2Qnc748utGEStu6kOt3nfZ566D5V1X0qBZxU3Vv3ijEGpZRS6jw/uxNQSinlWbQwKKWUuoAWBqWUUhfQwqCUUuoCWhiUUkpdIMDuBFxVpkwZExcXZ3caSilVrCxbtmy/MSYmv3XFvjDExcWRlpZmdxpKKVWsiMi2gtbprSSllFIX0MKglFLqAloYlFJKXUALg1JKqQtoYVBKKXUBjysMItJGRNaLSJaIDLY7H6WU8jUeVRhExB8YCdwBJAJdRCTR3qyUUsq3eFRhAJoAWcaYzcaY08BUoG1hfNHs9J1MWbyd30+eKYyPVz5uz5GTfLJoG+k7DqND26vixtMecKsE7HB6nw00vXgjEekL9AWoUqXKNX3RFyt28sP6HMb/soXxPRpTpXTYNX2OUvmZsngbI77PAqBpfDSP3lKdm2vGICI2Z6bU5XnaFcMVMcaMMcakGmNSY2LyfaL7ssb1aMy4Hqnsyz3J3e//zPfr9upvdsptzll/lYbcUZs1u3Lp+dFSHvx4KUeO6xWq8nyeVhh2ApWd3sdaMbfz8xNa1CnHl4/fQFRYIL0+TuOu937mq4xdnD2nBUK5zt9PePjm6ix6vgXPtqnFTxv30+b/fuS/G3LsTk2pS/K0wrAUSBCReBEJAjoDcwrzC6uWDufrJ27klXvqknvyDP2nrKDH+CUcOaG/2Sn3CA8OoN8tNZj+cHP8ROj50RLmr95jd1pKFcijCoMxJg/oD8wH1gLTjTGrC/t7w4MD6HFdHN8PuoUX/1KH3zYf4O73fmb59kOF/dXKhzSqGsW8ATeSWKEkz83MYOPe3+1OSal8eVRhADDGzDXG1DTGVDfGDC3K7w709+OhG6vxSe+mnDOGLmMW8c2q3UWZgvJyJUICea9LCgF+Qucxi1i0+YDdKSn1PzyuMHiC5tVLM/ux66ldvgT9pixn1vJsbZhWblMtJoJP+zQjPDiAbmMXMy9Tbyspz6KFoQClI4KZ2rc5DatEMXD6SvpMTOPkmbN2p6W8REK5Eszpfz11K5XiiU9X8MO6fXanpNQftDBcQmiQP5/2acZzbWqzcO0+BkxNJ+/sObvTUl4iMiyIj3s2pkbZCPpPWc76PdrmoDyDFobLCArw49FbqvO3uxKZt3oPT01fqcVBuU1UeBBje6QSFhzAo58s4+ipPLtTUkoLw5XqfUM8z7SuxZcrd9FrQhr7fj9pd0rKS1SMDOW9LilsPXCMRyYt43Se/uKh7KWF4So8dmsN/nlvfRZtOsC9I38la99Ru1NSXqJZtdIMvbc+P2ft56XZmdrZQdlKC8NV6tq0ClMfbsbvJ8/QcfSvrNuTa3dKykt0aVKFx26tztSlO/j3T5vtTkf5MC0M16BhlShm9buOQH8/uoxZxJpdWhyUewy6vRZt6pbn9W/WaTdWZRstDNeoRtkSTHu4OUEBfvT8aIk+Ja3cws9PeLdzMkmxkQyank7WPu2ppIqeFgYXxJcJZ1Jvx6jg7Uf9yhcrCmW8P+VjQgL9Gd2tIUEBfgyYlq7Pz6gip4XBRTXLlWDBUzfTND6aZz/LYF6mDqGhXFehVCjD2ieRuTOXN+attzsd5WO0MLhBqbBARndrRK3yJeg/ZYXeG1Zu0bpuebo3r8r4X7bwa9Z+u9NRPkQLg5tEhgUxuU9TalcowaOTlzFp0Ta7U1JeYPAdtalWJpxBM1aSq9PQqiKihcGNSoYE8mmfZtyUEMNLszP5OkNvKynXhAUF8HanZPbmnuTvs1fr8w2qSGhhcLMSIYGM6taQpNhInpuZwY6Dx+1OSRVzyZUj6X9rDWat2MkH/9lkdzrKB2hhKARhQQGM7JqCCPT/dAVndGwl5aKnbq/JPQ0q8q9v1zNn5S6701FertAKg4j8XUR2iki6tdzptG6IiGSJyHoRae0Ub2PFskRkcGHlVhRio8IY3j6JlTsO89CENI7p4Gg+xeDeWz4iwrD29WlYJYqnpqWTkX3YrZ+vlLPCvmJ4xxiTbC1zAUQkEcdcznWBNsAHIuIvIv7ASOAOIBHoYm1bbN1ZvwKvta3Lz1n7eXTycs6e0/vDvkTc/HlhQQGM79mYMhFBPD1jJafy9PkGVTjsuJXUFphqjDlljNkCZAFNrCXLGLPZGHMamGptW6w90DyOV9vW5ccNOfSfslxHzlQuKRUayLC/JrFh71He/naD3ekoL1XYhaG/iGSIyHgRibJilYAdTttkW7GC4v9DRPqKSJqIpOXk5BRG3m51f9OqvPiXOnyTuYf+U5ZzTq8clAturV2WLk2qMOanzSzWOaNVIXCpMIjIQhHJzGdpC4wCqgPJwG7gLTfkC4AxZowxJtUYkxoTE+Oujy1UD91Yjb/dlci3a/byypertTgol7z4lzpUiQ5jwLR0Dh07bXc6ysu4VBiMMS2NMfXyWWYbY/YaY84aY84B/8ZxqwhgJ1DZ6WNirVhBca/R6/o4ujSpwoTftjFk1iotDuqahQcH8H6Xhuw/eoonpmrPN+VehdkrqYLT23uBTOv1HKCziASLSDyQACwBlgIJIhIvIkE4GqjnFFZ+dhARXv9rfR6/rQbT0nbw1HQdIE1du/qxpRjarj4/bdzPsG/W2Z2O8iIBhfjZb4hIMmCArcDDAMaY1SIyHVgD5AGPGWPOAohIf2A+4A+MN8asLsT8bDPw9pqEBPrz5vz1HDuVx6hujQj010dK1NW7r3FlVu86wrift5BYoSTtG8XanZLyAoVWGIwxD1xi3VBgaD7xucDcwsrJU4gIj91ag4jgAF6es5q+E9MY/UAjggP87U5NFUMv3pXI2j2/M3hWBhUjQ2levbTdKaliTn9NtVGP6xxdWX9Yn8Njk5frbSV1TQL9/fj3A6lUjgrjyakrOHxcG6OVa7Qw2Kx78zhevjuR79bt48GPlpKnjYjqGpQKC2RElxQOHjvN0zMy9GFK5RItDB7gwevjGf7XJH7bfIBBM1bqP2p1TepVKsXzd9Zh4dq9vPf9RrvTUcVYYTY+q6twX+PK7Mk9ydsLNhDg58ebHZLw83P3oArK2/W6IZ6M7MO8/30WrRLLk1ixpN0pqWJIrxg8yBMtEniiRQIzl2cz+kcdXlldm5fvrktkWBCDZqzUdit1TbQweJinWiZwd4OKvDFvPeN+3mJ3OqoYigoPYnj7+qzdnavPN6hrooXBw4gIb7RPomWdcrz21Rq+ytCx99XVa1GnHD2vi+PjX7fy3w2eP56Y8ixaGDxQaJA/73dNcczcNWUFs9O9amQQVUSea1ObhLIRPPvZSo4c1/mi1ZXTwuChQgL9mdq3GU3io3luZgYrd+jELOrqhAb58/Z9yew/epqB09N1bC51xbQweLCQQH9Gdm1ImYhgek9II/uQzh+trk792FIMblOb79bt46Nft9qdjiomtDB4uJgSwXz8YGNOnTlLx9G/sSnnqN0pqWLmoRvjaVmnLMO+WatXnuqKaGEoBmqULcGnfZtx4sxZuo9bolcO6qqICP/q2IAyEcE8NS2d3JPa3qAuTQtDMVGvUik+6tmYQ8dP88gny3S+X3VVIsOCeOu+Bmw9cIy/fZF5+R2UT9PCUIykVIninU7JZO7M5aEJaRw/nWd3SqoYua56GZ5sUZPZ6buYl7nb7nSUB9PCUMy0rlue1//qmJylw6jftDh4KOOhHYD63VqdepVK8tzMVew4qLckVf60MBRDXZpU4cMHGrF2Ty59Jy7TOX89lHjgUFeB/n6M7NqQc8bQ86Ml+nyDypdLhUFEOorIahE5JyKpF60bIiJZIrJeRFo7xdtYsSwRGewUjxeRxVZ8mjW9pypA67rlebVtPX7ZtJ9u4xbzuzYoqitUtXQ4Y7unsu3AcV6cre0N6n+5esWQCfwV+NE5KCKJOOZsrgu0AT4QEX8R8QdGAncAiUAXa1uA4cA7xpgawCGgt4u5eb0HmlXlw26NWLfnd56cms7pPJ3LQV2ZptVKM6BlAl+u3MUni7bZnY7yMC4VBmPMWmPM+nxWtQWmGmNOGWO2AFlAE2vJMsZsNsacBqYCbUVEgNuAz6z9JwDtXMnNV7SqW55X7qnL9+v20W/ycs7oRD/qCj16Sw1urhnDy3NW80vWfrvTUR6ksNoYKgE7nN5nW7GC4qWBw8aYvIvi+RKRviKSJiJpOTk6QFi3ZlV56a5EFq7dy6OfLNOhltUV8fcTRnRJoWrpMJ6eoeMpqT9dtjCIyEIRycxnaVsUCebHGDPGGJNqjEmNiYmxKw2P0uuGeF65py7frdtH93FLOKgN0uoKlAoN5J37ksn5/RQDpq3Q8ZQUcAWFwRjT0hhTL59l9iV22wlUdnofa8UKih8AIkUk4KK4ugo9rovj3U7JpO84TIfRv5Lz+ym7U1LFQIPKkbx0dyI/rM9hzE+b7U5HeYDCupU0B+gsIsEiEg8kAEuApUCC1QMpCEcD9RxjjAF+ADpY+/cALlV4VAHaJlfik4easuvwCR6elKa9ldQVeaBZVe6oV55/zV/Pos0H7E5H2czV7qr3ikg20Bz4WkTmAxhjVgPTgTXAPOAxY8xZqw2hPzAfWAtMt7YFeA4YKCJZONocxrmSmy9rEh/N8PZJpO84zMOTlmmDtLosEWFY+yTiyoTTb/Jy9uWetDslZSNXeyV9boyJNcYEG2PKGWNaO60baoypboypZYz5xik+1xhT01o31Cm+2RjTxBhTwxjT0Rij90Fc0Da5EsPbJ/HrpgP0mZjG4ePa5qAurVRoIKO7NeT46Tye0vkbfJo++ezFOqZW5rV29fhp4366/nsxx07p8Bnq0mqULcHf767LL1kH+PBHbW/wVVoYvNwDzaoytkcq6/bkMmTWKoynDuKjPEanxpUd7Q3frtf5G3yUFgYfcGutsgxqVYs5K3fxf99ttDsd5eHOtzfERAQzaMZKTpzW52J8jRYGH9Hvlup0aBTLuws38tEvW+xOR3m4UqGBvNEhiU05R3nhC73S9DVaGHyEiPDPe+tze2I5XvlyDZMXb9N/7OqSbqoZw+O3JTBr+U4+X6GPFfkSLQw+JCjAj/e6pHBd9dK88Hkm//h6rRYHdUlPtkigcVwUL89Zza7DJ+xORxURLQw+JiTQn4m9mtCtWRXG/byFF7/IJE+fc1AF8PdzzBd97pzhoQlpOg6Xj9DC4IMC/P149Z56PNCsKpMXb6fPRP0HrwpWtXQ473VNYc3uXF6fu9budFQR0MLgo/z8hFfb1uVvdznGyOk2djGn8rQ4qPzdVrscva6PZ8Jv25iXucfudFQh08Lgw0SE3jfE826nZNK2HWLwzFWc1addVQEG31GbepVK8vznq9h9RNsbvJkWBkW7lEoMaJnA5yt20nbkz6TrQ00qH0EBfrzbKZlTZ84ycNpKHTLDi2lhUAAMaFmTdzsls+3AcbqMWcSvOqOXS7z1v8waZUvw8t11+W3zAUZ8rw9LeistDOoP7VIq8d2gm6lQKoReE5aSka1XDq4QxO4UCkXH1Fj+mlKJEd9t1ClBvZQWBnWBsiVCmP5Ic8pEBPPgR0tZtyfX7pSUhxERXmtXj2oxEQyYlq4TQnkhLQzqf5SJCGZiryb4+wn3vPcLX+hTr+oi4cEBvN81hdwTZxgyK0MflPQyrk7U01FEVovIORFJdYrHicgJEUm3ltFO6xqJyCoRyRKRESIiVjxaRBaIyEbrzyhXclOuqRYTwez+15NcOZJnPlvJsm0H7U5JeZja5UvyTOtaLFy7j/G/bLU7HeVGrl4xZAJ/BX7MZ90mY0yytTziFB8F9MEx3WcC0MaKDwa+M8YkAN9Z75WNKpQKZUz3RlSMDKX3hDQydx6xOyXlYXpdH8/tieV47as1/Gf9PrvTUW7i6gxua40x6690exGpAJQ0xiyy5nmeCLSzVrcFJlivJzjFlY0iw4KY1KspYYH+9JmYpv3X1QX8/IQRnVOoXb4ET05NZ69OCeoVCrONIV5EVojIf0XkRitWCch22ibbigGUM8bstl7vAcoV9MEi0ldE0kQkLScnx+2JqwtVKR3Gv3ukknviDN3GLmbbgWN2p6Q8SGiQPx/c35BTeWd55rMMfUjSC1y2MIjIQhHJzGdpe4nddgNVjDEpwEBgioiUvNKkrKuJAv92GWPGGGNSjTGpMTExV/qxygV1K5ZifM/G7M09RbuRv7B6l95WUn+qFhPBC3fW4ccNOXz44ya701EuumxhMMa0NMbUy2eZfYl9ThljDlivlwGbgJrATiDWadNYKwaw17rVdP6Wk96w9DBNq5VmTv/rCfT3o+dHS9lzRG8bqD890DyONnXL8+7CjWTtO2p3OsoFhXIrSURiRMTfel0NRyPzZutWUa6INLN6I3UHzheYOUAP63UPp7jyINViIpjQqwlHT+Zx34e/kXvyjN0pKQ/yWrt6hAX58/SMlZzO0+HciytXu6veKyLZQHPgaxGZb626CcgQkXTgM+ARY8z5/o79gLFAFo4riW+s+DDgdhHZCLS03isPVKdCSSb1bsLOwyfo/fFSjp/Oszsl5SFiSgTzatt6pO84zAidX7zYCnBlZ2PM58Dn+cRnAjML2CcNqJdP/ADQwpV8VNFJjYvmjfZJPPPZSp6als6o+xvh5+edQ0Coq3NPg4r8uCGHD/6Txa21Y2hUNdrulNRV0ief1TVr3yiW5++sw/zVexk+b53d6SgP8tLdiVSMDGXAtHSOntIryuJGC4NySe8b4unatAof/riZD/+rvVGUQ8mQQN7plEz2oRO8+Pkqu9NRV0kLg3KJiPDqPXX5S1IFhs1bx/zVOruXcmgcF82TLRL4In0Xc1busjsddRW0MCiXBfj78WaHJOpWLMnjn65gyRYdV0k59L+1BilVInl+1io25WgX1uJCC4Nyi7CgACb1akpsZCgPTVjKZv1PQOH4peH9rg0JCvDjkUnLtAdbMaGFQblNVHgQE6zhuh+bsoKTZ87anZLyAJUiQxnROYWsnKO8NHu13emoK6CFQblV5egw3r4vmbW7c7WnkvrDDQll6H9rDT5blq3tDcWAFgbldrfWLkvP6+L46JetOsmP+sMTLRJoVDWKITMztL3Bw2lhUIVi8B21aVglkgHT0pnw61a701EeINDfj/e6pBAc6E+fCWkcPHba7pRUAbQwqEIREujPlD7NuD2xHH//crV2Y1UAVIwMZWTXhmw/eJxXvtT2Bk+lhUEVmpBAf/6vczL1K5Wi3+Tl/LjBd+bO0CmQC9a8emn631aD2em7+G7tXrvTUfnQwqAK1flurFWjw+g9YalPFQd06KgC9bulBrXKleCFzzN1hF4PpIVBFbpSYYHMfPQ6qsdE8Njk5WTt+93ulJTNggL8eKNDEvt+P8mQmasweonlUbQwqCIRFR7EuJ6NCQ70o/eENA4cPWV3SspmDSpHMqhVLb5etZtPl+ywOx3lRAuDKjKVIkP58IFU9hw5Se8JaZw4rQ/A+bpHbq7OjQlleOXL1Wzcq1eSnkILgypSjapG8W6nZFZmH+bxT1dw5qzO8uXL/P2Et+9LJiI4gEc+0SEzPIWrM7i9KSLrRCRDRD4XkUindUNEJEtE1otIa6d4GyuWJSKDneLxIrLYik8TkSBXclOe6476FXj5rkQWrt3LMzNWanHwcTElghnRJYXN+4/x/Cxtb/AErl4xLADqGWOSgA3AEAARSQQ6A3WBNsAHIuJvzQM9ErgDSAS6WNsCDAfeMcbUAA4BvV3MTXmwntfH83SrmnyRvosXP8/U/wx83PU1yvD4bY4humcsy7Y7HZ/nUmEwxnxrjDl/7bcIiLVetwWmGmNOGWO24JjfuYm1ZBljNhtjTgNTgbYiIsBtOOaHBpgAtHMlN+X5+t+WQP9bazAtbQejdJIfn/dkiwSaVyvNS7MzSd9x2O50fJo72xh6Ad9YrysBzt0Msq1YQfHSwGGnInM+ni8R6SsiaSKSlpPjQ/3ivdCgVjX5S/0KvPXtBv6zfp/d6Sgb+fsJI7qkUCYimO7jFrN1/zG7U/JZly0MIrJQRDLzWdo6bfMCkAdMLsxkzzPGjDHGpBpjUmNiYoriK1UhERGGd0iiZrkS9J+ygvV7tGeKL4spEcynfZoB8OS0dB263SaXLQzGmJbGmHr5LLMBRKQncBdwv/nzRvFOoLLTx8RasYLiB4BIEQm4KK58QERwAON6pBIa5M9jU5brk7A+rnJ0GMPbJ7Fyx2HeXbjR7nR8kqu9ktoAzwL3GGOOO62aA3QWkWARiQcSgCXAUiDB6oEUhKOBeo5VUH4AOlj79wBmu5KbKl4qRobybqdktu4/Rrexizmdpz2VfNkd9SvQKbUyY37cxIrth+xOx+e42sbwPlACWCAi6SIyGsAYsxqYDqwB5gGPGWPOWm0I/YH5wFpgurUtwHPAQBHJwtHmMM7F3FQxc32NMozokkJG9hFenqM9lXzdC3fVoXzJEJ6cmq5XkUUs4PKbFMzqWlrQuqHA0Hzic4G5+cQ34+i1pHzYnfUr8MjN1Rn9303ElAhh4O017U5J2aRkSCDvdU3hvg8X8dxnGXxwf0McHRhVYdMnn5XHebZ1Le5NqcSI7zYy6j+b9MrBhzWqGs2zrWvxTeYeJv62ze50fIZLVwxKFQY/P+HNDkmcPnuO4fPWEeAn9Lmpmt1pKZv0ubEaS7YcZOjXa0mpEklSbOTld1Iu0SsG5ZEC/P0Y0TmFO+uXZ+jctczUp2F9lp+f8GbHBpSJCKLvxGXszT1pd0peTwuD8lj+fsK7nVK4rnpphsxaxcI1OtuXr4oOD2L0A404fOI0Q3Q8pUKnhUF5tKAAP97v2pCEchH0m7ycjGwdKsFXJcVG8mzr2ny/bh+zlutjToVJC4PyeNHhQXzSuymlI4LoNnYx2w7oUAm+qud1cTSOi+LFLzLZokNmFBotDKpYiAoP+mOohEc/Wa5DJfgoP2s8pUB/YYAOmVFotDCoYiOuTDjvdEpmze5c/vH1GrvTuSSD3gMvLBVKhTLMGjLjBR2yvVBoYVDFSos65eh7UzU+WbSdSYs8u1+7PopVeO6sX4EnWyQwc3k2H/+61e50vI4WBlXsPNemNrfVLsvf56zm+3XaU8lXDWiZQIvaZRn2zTqy9h21Ox2vooVBFTvnx+2vU6EE/SYv59dN++1OSdlARHi9fX1Cg/zpP2U5J05re4O7aGFQxVJEcADjezSmUmQoD09apvM4+KiyJUJ4t1My6/f+zotfZNqdjtfQwqCKrbIlQ/j4wSYE+fvR9d+LtPuij7qlVlkev83R3jB5sWe3OxUXWhhUsVY5OozpjzTnnDF0/fcisvbplYMverJFAjfXjOFvX2SyaPMBu9Mp9rQwqGKvekwEE3s15VTeOTqP0bmCfZG/n/B+1xSqRIcxcFo6h46dtjulYk0Lg/IK9WNLMaVPU86eO8eDHy9lnw605nNKhATyf51T2H/0NE9MXcHZc/p8w7VydWrPN0VknYhkiMjnIhJpxeNE5IQ1q9sfM7tZ6xqJyCoRyRKREWLNvCEi0SKyQEQ2Wn9GuXZoytfULl+Sf3dPZc+Rk3Qfv4Qjx3XWL1/ToHIkr7aty08b9/PWt+v14bdr5OoVwwKgnjEmCdgADHFat8kYk2wtjzjFRwF9cMwDnQC0seKDge+MMQnAd9Z7pa5Kalw0ox9oxKacozz48RJO5WkXRl/TqXFlOjaK5YP/bGLMj5vtTqdYcqkwGGO+teZxBlgExF5qexGpAJQ0xiwyjlI+EWhnrW4LTLBeT3CKK3VVbq4Zw9v3JbN8+2EGz1xF3tlzdqekipCIMLx9En9JqsDweev0OZdr4M42hl7AN07v40VkhYj8V0RutGKVAOcZV7KtGEA5Y8xu6/UeoFxBXyQifUUkTUTScnJy3JS+8iZ3N6jIwNtr8vmKnTw5LZ1zer/Zp/j5CW+0TyKuTDgDp63kyAm9rXg1LlsYRGShiGTms7R12uYFIA+YbIV2A1WMMSnAQGCKiJS80qSsq4kC/yUbY8YYY1KNMakxMTFX+rHKxzzRIoFn29Ti64zdDJ6VoSNx+pjw4ADe7ZRMztFTvDRbB9u7Gped89kY0/JS60WkJ3AX0ML6Dx1jzCnglPV6mYhsAmoCO7nwdlOsFQPYKyIVjDG7rVtO+67yWJT6H4/eXJ0Tp8/y3vdZZB86wQf3NyQyLMjutFQRSYqNZECLBN5asIH6lUrx0I06d/iVcLVXUhvgWeAeY8xxp3iMiPhbr6vhaGTebN0qyhWRZlZvpO7AbGu3OUAP63UPp7hS10xEGNSqFv/q2IClWw/Se0IaZ7TNwac8dmsNWiWWY/i8dfySpe0NV8LVNob3gRLAgou6pd4EZIhIOvAZ8Igx5qC1rh8wFsgCNvFnu8Qw4HYR2Qi0tN4r5RYdGsXy1n3JLNt2iH/OXWt3OqoI+fkJb3RIolqZCPpMTGPt7ly7U/J4l72VdCnGmBoFxGcCMwtYlwbUyyd+AGjhSj5KXco9DSqyYvshPvplK/UqlqJ9o0t2olNeJDIsiAm9mnD3+z/T6+OlfNqnGXFlwu1Oy2Ppk8/Kpzx/Zx2aVyvNkM9X6Zg6PqZ8qRDG9Ujl6Mk8Hhi/mN9Pak+lgmhhUD4l0N+Pkfc3JDYqlEc+WabjKvmYpNhIPu7VhJ2HTvD6N+vsTsdjaWFQPic6PIhxPRpjDDw0MY2dh0/YnZIqQo2qRtHr+nimLN7Ob5v0qjE/WhiUT4ovE84H9zdkz5GTtH7nR35Yr72jfcmgVrWoWjqMwbMy9JZSPrQwKJ91fY0yzO5/PbFRoTw2eTm/aldGnxEa5M/w9klkHzpB34nLdCTWi2hhUD7NMZdDE2KjQuk1Yan7rhz0/xmP16xaaV6/tz6/bce7HA0AABD4SURBVD6gg+1dRAuD8nllS4YwpU8zqsdE8Njk5aTvOOyWz3UMKK88WcfUWO6sX563vl3Pml36fMN5WhiUAspEBDOuR2Oiw4PoNnYxy7cfsjslVQREhKHt6hMZFsSgGSt1PC2LFgalLOVLhTDt4eZEhgXywNjFLN168PI7qWIvKjyIf95bj7W7c3nlyzV2p+MRtDAo5aRSZCgzH72OsiVDeHjSMrL2/W53SqoItKpbnodvqsanS7YzZ+Uuu9OxnRYGpS5SrqTjCVk/Ee77cBG79DkHnzCoVS1Sq0YxeGYGG/f69i8EWhiUyke1mAim9m3GyTNneWpauo7I6gOCAhxPxYcF+fPYlOUcPZV3+Z28lBYGpQpQo2wEQ++tx+ItB3l+1iqd6MUHlCsZwjudksnad5RX5qy2Ox3baGFQ6hLuTYnliRYJzFiWzdifttidjioCNybE8Ogt1ZmxLJsJv261Ox1baGFQ6jKeaplAm7rlGTp3LbPTd15+B1XsDby9Fi1ql+UfX68hc+cRu9Mpci4XBhF5TUQyrIl6vhWRilZcRGSEiGRZ6xs67dNDRDZaSw+neCMRWWXtM8Ka5U0pW4kI73ZOpnFcFM/NzGDJFu3G6u38/YR/dWxAdHgQj3+6wufGU3LHFcObxpgkY0wy8BXwkhW/A8eUnglAX2AUgIhEAy8DTYEmwMsiEmXtMwro47RfGzfkp5TLQgL9GXl/QypGhvLwpDS2HdDhur1dVHgQ73VpyPaDx3lmRgbnfGg8JZcLgzHG+TnycP4cJaYtMNE4LAIiRaQC0BpYYIw5aIw5BCwA2ljrShpjFhlHK99EoJ2r+SnlLmVLhDC2eyp5Zw0dR/+mzzj4gCbx0Qy5ozbzVu/hhS9W2Z1OkXFLG4OIDBWRHcD9/HnFUAnY4bRZthW7VDw7n7hSHqNaTATTHm7OOQPdxi5hb+5Ju1NShaz3DfH0vakany7ZwVcZvvHw2xUVBhFZKCKZ+SxtAYwxLxhjKgOTgf6FmbCVT18RSRORtJycnML+OqUukFixJJN6N+HIiTM8PmUFx0/7bn93XyAiPNO6FsmVI3lmRoZPDLZ3RYXBGNPSGFMvn2X2RZtOBtpbr3cClZ3WxVqxS8Vj84nnl88YY0yqMSY1JibmSg5BKbeqU6Ekw9rXZ+m2gzz7mW/df/ZFgf5+jOneiBIhAQyYtoJTed492J47eiUlOL1tC5yfSHUO0N3qndQMOGKM2Q3MB1qJSJTV6NwKmG+tyxWRZlZvpO7AxYVHKY/RNrkST7eqxVcZuxn5Q5bd6ahCVrZECMM7JLFh71He+naD3ekUqgA3fMYwEakFnAO2AY9Y8bnAnUAWcBx4EMAYc1BEXgOWWtu9aow53/+vH/AxEAp8Yy1Keax+t1Rn497feWvBBsqVDOG+xpUvv5Mqtm6tVZb7m1ZhzI+buSkhhhsSytidUqFwuTAYY9oXEDfAYwWsGw+MzyeeBtRzNSelioqIMKx9ErsOn+S5WRmEBPlzT4OKdqelCtGLf0lk8ZaDPDU9nfkDbiI6PMjulNxOn3xWykUhgf5M6NWE1KpRPD19pdtmgFOeKTTIn/e6pHDk+BmvHUNLC4NSbhAa5M+YB1IpVyqYvhPT2JRz1O6UVCGqU6EkT7euybzVe5ixLPvyOxQzWhiUcpOo8CDG9WjMmbPnWLh2HyfP6FDd3uyhG6rRND6a175cQ/ah43an41ZaGJRyo5rlSvB5v+vtTkMVAT9rPCWA+8cu5tCx0zZn5D5aGJRys7gy4Xw/6GZGd2t4+Y1VsVY5OoxxPRuz6/AJXvwi02vaG7QwKFUIqsVE0KZeBbvTUEWgSXw0T91ek69X7Wbmcu8Yll0Lg1JKueiRm6rTJC6aV75c7RVzhGthUEopF/n5CW92TOLsOcPA6enFfogULQxKKeUGVUuH8/e767Jo80HG/1K8p4HVwqCUUm7SMTWWVonlGPbNOhZvPmB3OtdMC4NSSrmJiPBmxwZUiQ7j0cnLi217gxYGpZRyo1KhgYzpnsqpM2cZMC2ds8WwvUELg1JKuVmNshG80rYeS7YcZOjXa+1O56q5Y9htpZRSF+nQKJbMnUcY/8sW6lYsSftGsZffyUPoFYNSShWSF/5Sh6bx0QyZtYrl2w/Znc4V08KglFKFJNDfj9HdGlmj7i5jb+5Ju1O6Ii4VBhF5TUQyRCRdRL4VkYpW/BYROWLF00XkJad92ojIehHJEpHBTvF4EVlsxaeJiPfNfqGU8jnnR909diqPp2esLBaN0a5eMbxpjEkyxiQDXwEvOa37yRiTbC2vAoiIPzASuANIBLqISKK1/XDgHWNMDeAQ0NvF3JRSyiPULFeCv92VyE8b9/OPr9fYnc5luVQYjDG5Tm/DgcuVwiZAljFmszHmNDAVaCsiAtwGfGZtNwFo50puSinlSbo2rULP6+L46JetLFiz1+50LsnlNgYRGSoiO4D7ufCKobmIrBSRb0SkrhWrBOxw2ibbipUGDhtj8i6KF/SdfUUkTUTScnJyXD0EpZQqEkPurE1ihZIMnpnBgaOn7E6nQJctDCKyUEQy81naAhhjXjDGVAYmA/2t3ZYDVY0xDYD3gC/cmbQxZowxJtUYkxoTE+POj1ZKqUITHODP250akHvyDH+b7bnzN1y2MBhjWhpj6uWzzL5o08lAe2ufXGPMUev1XCBQRMoAO4HKTvvEWrEDQKSIBFwUV0opr1K7fEkGtKzJ3FV7mJHmmfNFu9orKcHpbVtgnRUvb7UbICJNrO85ACwFEqweSEFAZ2COcZTNH4AO1mf1AC4uPEop5RUevbm6Y77or9ew+4jnjafkahvDMOu2UgbQCnjSincAMkVkJTAC6Gwc8nDcbpoPrAWmG2NWW/s8BwwUkSwcbQ7jXMxNKaU8kp+f8EaHJPLOGp79LMPjbimJpyV0tVJTU01aWprdaSil1FWb9NtW/jZ7Nf9oV49uzaoW6XeLyDJjTGp+6/TJZ6WUskm3ZlW5oUYZ/jl3LdsPHLc7nT9oYVBKKZuICMPa18ffT3hy2gryzp6zOyVAC4NSStkqNiqMf95bnxXbD/P2gg12pwNoYVBKKdvd3aAinVIrM+q/m/h5436709HCoJRSnuDlexJJKBvBgGkrbB+FVQuDUkp5gLCgAEZ2bcixU2cZON3eKUG1MCillIdIKFeCl+9O5JesA4z9abNteWhhUEopD9KpcWVa1y3HWws2sHHv77bkoIVBKaU8iIgw9N76lAgO4Imp6ZzOK/ourFoYlFLKw5SJCOb1v9Zn7e5cPvhPVpF/vxYGpZTyQK3qlqddckXe/z6L5dsPFel3a2FQSikP9fd76lKuZAgDp6Vz/HTe5XdwEy0MSinloSLDgvhXxwZsPXCc1+euK7Lv1cKglFIerHn10jx0QzyTFm1jXubuIvlOLQxKKeXhnm1TmwaxpXh6Rgabco4W+vdpYVBKKQ8XFODHqG6NCArw4/EpKzh55myhfp/bCoOIDBIRY83tjDiMEJEsEckQkYZO2/YQkY3W0sMp3khEVln7jDg/PahSSvm6ipGhvNkhiTW7c3l+1qpC/S63FAYRqYxjas/tTuE7gARr6QuMsraNBl4GmgJNgJdFJMraZxTQx2m/Nu7ITymlvEGLOuV4okUCs1bs5IsVOwvte9x1xfAO8CzgPOpTW2CiNdfzIiBSRCoArYEFxpiDxphDwAKgjbWupDFmkXHMNzoRaOem/JRSyis8cVsNGlWN4qXZmYU2CqvLhUFE2gI7jTErL1pVCdjh9D7bil0qnp1PPL/v7CsiaSKSlpOT4+IRKKVU8RHg78ebHZJIqRJFXiGNwBpwJRuJyEKgfD6rXgCex3EbqcgYY8YAYwBSU1PtG5tWKaVsUC0mggm9mhTa519RYTDGtMwvLiL1gXhgpdVOHAssF5EmwE6gstPmsVZsJ3DLRfH/WPHYfLZXSilVhFy6lWSMWWWMKWuMiTPGxOG4/dPQGLMHmAN0t3onNQOOGGN2A/OBViISZTU6twLmW+tyRaSZ1RupOzDblfyUUkpdvSu6YrhGc4E7gSzgOPAggDHmoIi8Biy1tnvVGHPQet0P+BgIBb6xFqWUUkVIHB2Aiq/U1FSTlpZmdxpKKVWsiMgyY0xqfuv0yWellFIX0MKglFLqAloYlFJKXUALg1JKqQsU+8ZnEckBtl3j7mWA/W5MpzjQY/YNeszez9XjrWqMiclvRbEvDK4QkbSCWuW9lR6zb9Bj9n6Febx6K0kppdQFtDAopZS6gK8XhjF2J2ADPWbfoMfs/QrteH26jUEppdT/8vUrBqWUUhfRwqCUUuoCPlsYRKSNiKwXkSwRGWx3PtdKRCqLyA8iskZEVovIk1Y8WkQWiMhG688oKy4iMsI67gwRaej0WT2s7TeKSA+7julKiYi/iKwQka+s9/Eistg6tmkiEmTFg633Wdb6OKfPGGLF14tIa3uO5MqISKSIfCYi60RkrYg09/bzLCJPWX+vM0XkUxEJ8bbzLCLjRWSfiGQ6xdx2XkWkkYissvYZYU1rcGnGGJ9bAH9gE1ANCAJWAol253WNx1IBxxwYACWADUAi8AYw2IoPBoZbr+/EMZy5AM2AxVY8Gths/RllvY6y+/guc+wDgSnAV9b76UBn6/Vo4FHrdT9gtPW6MzDNep1onftgHBNObQL87T6uSxzvBOAh63UQEOnN5xnH1L5bgFCn89vT284zcBPQEMh0irntvAJLrG3F2veOy+Zk9w/FphPRHMfkQOffDwGG2J2Xm45tNnA7sB6oYMUqAOut1x8CXZy2X2+t7wJ86BS/YDtPW3DM8PcdcBvwlfWXfj8QcPE5xjE5VHPrdYC1nVx83p2387QFKGX9JykXxb32PPPn/PDR1nn7CmjtjecZiLuoMLjlvFrr1jnFL9iuoMVXbyWd/wt3XrYVK9asS+cUYDFQzjhmxQPYA5SzXhd07MXtZ/Iu8CxwznpfGjhsjMmz3jvn/8exWeuPWNsXp2OOB3KAj6zbZ2NFJBwvPs/GmJ3Av4DtwG4c520Z3n2ez3PXea1kvb44fkm+Whi8johEADOBAcaYXOd1xvGrgtf0SxaRu4B9xphldudShAJw3G4YZYxJAY7huMXwBy88z1FAWxxFsSIQDrSxNSkb2HFefbUw7AQqO72PtWLFkogE4igKk40xs6zwXhGpYK2vAOyz4gUde3H6mVwP3CMiW4GpOG4n/R8QKSLnp6t1zv+PY7PWlwIOULyOORvINsYstt5/hqNQePN5bglsMcbkGGPOALNwnHtvPs/nueu87rReXxy/JF8tDEuBBKt3QxCOhqo5Nud0TaweBuOAtcaYt51WzQHO90zogaPt4Xy8u9W7oRlwxLpknQ+0EpEo6ze1VlbM4xhjhhhjYo0xcTjO3ffGmPuBH4AO1mYXH/P5n0UHa3tjxTtbvVnigQQcDXUexxizB9ghIrWsUAtgDV58nnHcQmomImHW3/Pzx+y159mJW86rtS5XRJpZP8PuTp9VMLsbXWxs7LkTRw+eTcALdufjwnHcgOMyMwNIt5Y7cdxb/Q7YCCwEoq3tBRhpHfcqINXps3oBWdbyoN3HdoXHfwt/9kqqhuMffBYwAwi24iHW+yxrfTWn/V+wfhbruYLeGjYfazKQZp3rL3D0PvHq8wy8AqwDMoFJOHoWedV5Bj7F0YZyBseVYW93nlcg1fr5bQLe56IODPktOiSGUkqpC/jqrSSllFIF0MKglFLqAloYlFJKXUALg1JKqQtoYVBKKXUBLQxKKaUuoIVBKaXUBf4f4V2fp5FSgzwAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":282},"id":"264SEH2zoH1M","executionInfo":{"status":"ok","timestamp":1634748915710,"user_tz":-330,"elapsed":998,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"e15f0fcd-49fb-49d2-d481-1d4dc24dd0b2"},"source":["tempdf2['ctr'].plot()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.axes._subplots.AxesSubplot at 0x7f31e9365790>"]},"metadata":{},"execution_count":57},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8dcnN2uzNW3TdG+6QVvKUhqWWgSUAgUEdBhHyriOyugIo8L8RhB+qDiM4Cgqwk+tCszgwuZCBwvIpigI0sra0iUtLU23JG2z77nf3x/3JL1JbpKb9Cb3npP38/HIo+eec3LzOTkn7577Pd9zvuacQ0RE/C8t2QWIiEhiKNBFRAJCgS4iEhAKdBGRgFCgi4gERHqyfvCkSZNcaWlpsn68iIgvbdiwodo5VxxrWdICvbS0lPXr1yfrx4uI+JKZ7epvmZpcREQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIAYNdDO728wqzezNfpabmd1hZuVm9rqZnZz4MkVEZDDxnKHfC6waYPkFwALv60rgB0dfloiIDNWgge6cew44NMAqlwL/4yJeBMab2dREFSgymnYfauL232/hnYNNyS5FZMgS0YY+Hdgd9brCm9eHmV1pZuvNbH1VVVUCfrRIYu2paeaOZ8qpqFGgi/+M6kVR59wa51yZc66suDjmnasiIjJMiQj0PcDMqNczvHkiIjKKEhHoa4GPer1dTgdqnXP7EvC+IiIyBIM+nMvMfgmcDUwyswrgK0AGgHPuh8A64EKgHGgCPjFSxYqISP8GDXTn3OpBljvgcwmrSCQVaOx08SHdKSoSxZJdgMhRUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6SAzq5CJ+pEAXiWKmfi7iXwp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6SAxO/RbFhxToIlHUa1H8TIEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIjE4PZ5LfEiBLhJFnVzEzxToIiIBoUAXEQkIBbqISEAo0EVEAkKBLhKDnuUifqRAF4miZ7mInynQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXSQGdXIRP4or0M1slZltMbNyM7suxvJZZvasmb1iZq+b2YWJL1VkNKibi/jXoIFuZiHgLuACYDGw2swW91rtRuBB59xS4HLg/yW6UBERGVh6HOucCpQ753YAmNn9wKXApqh1HFDgTRcCexNZpIhIKguHHa0dYfbXtVDd0Mremma2HqjnYEMblfWt1Le0AxBKM/bWtPCv5yzg75fNSHgd8QT6dGB31OsK4LRe63wV+L2ZXQ3kAitjvZGZXQlcCTBr1qyh1ioiMiJaOzqpbW5n96FmcrNCtLaHMYNxmekUjcugqa2T7IwQuw42sqO6keqGVt7aV8/GPbUcamqjvqWDznDPKy9mMD4ng5KCbAqyM8CgpT3M/Ml5HFOSNyLbEU+gx2M1cK9z7ttmthy4z8yWOOfC0Ss559YAawDKysp03UlEjlpzWyf1Le10hB1NbZ1UN7QSSjMaWzvIzghRmJNBRiiNQ41tbNpbS1VDK9srG9laWU+aGelpRnllAx3hoUXStMJsFk8r5LT8LMaPyyAvK53ivCxKCrMpKciidGIu2RmhEdrq2OIJ9D3AzKjXM7x50T4JrAJwzv3FzLKBSUBlIooUGW1OD3NJuI7OMK9V1LLtQD1FuZm8c7CJHdWNZKWnUdvcTpoZdS3tOAenz51AXUsHO6sbyUxPIzczxJLphSyaWsAbe2p5vryaUJqxvaqBTXvrGGIWM6Moh8VTCzjU2EZBTgZnHVNMSUE2syeOo6U9TFZ6GmHnqGvpoK65nYyQ0dbpmFmUw9ziXIrGZTIxL2tkflFHIZ5AfxlYYGZziAT55cAVvdZ5BzgHuNfMFgHZQFUiCxUZDXqWy9B1hh1V9a28sL2a5fMmEkoz7vvLLu59YSfvXjCJ3YeacTj2HG7mcFN7j+8NpRkGdIQd+dnpGFDX0sFTbx0AID8rnY6wo7m9s8f35Wenk5MRYl5xHleeOY+JuZlkZ6SREUojNyudjJAxMS+LhpYO9tQ0YwbFeVksmlrAlMJsMkLB7LE9aKA75zrM7CrgCSAE3O2c22hmNwPrnXNrgWuBH5vZF4lcIP240ymOSCCVVzZw7wtv87MX3xl03afeqmR8TgYT87JYPm8i5ywsYd7kPFraO5lamM3sibk453AO0tIi/5sebmyjtrmdgpwMJuRmApFmlU37atm0r56y2UUsnJKP6X/fPuJqQ3fOrQPW9Zp3U9T0JmBFYksTkWTaW9PMb1/dw7ObK9lf18LuQ8191snOSCNkxkUnTKWlPczBxlZKJ+bykeWzWTilIMa79mVmPT4ZFeVmUuQFeZeczBDLZk9g2ewJR7VNQZeoi6Ii4nPOOfbWtrDu9X18+8kttLSHY653aukE/vPvjmf+5JHpqSHDp0AXGeM6w457X9jJujf2sWHX4e75KxdN5vS5EynOz2LlohJysxQXqU57SGSMqW1q57YnNvOLl/q2gednp3PjRYv4h7KZaqP2IQW6SAxBu6K/v7aF07/xdMxlReMyWDF/EjdctIiphTmjXJkkkgJdJEqQzkkPNbZx6i1Pxbxh5vJTZnLtecdSnJ96fall+BToIgGz9UA9533nuR7zJuVlcffHyzhhxvgkVSWjQYEuEhA7qxs5+1t/6DHv4c8sZ9nsIrWHjxEKdBEf232oic/+fANv7qnrMf/Rq89gyfTCJFUlyaJAF/EZ5xwX3/nnPiEO8NOPlXHOopIkVCWpQIEuEksKdnPZU9PMiluf6TP//ONK+M6HTmJcpv6cxzodASJRUrGt2TnHPc/v5OZHN/WY/+t/eRcnzhhPKC31apbkUKCLpLDK+hZOveVI//EffvhkVi2ZmsSKJJUp0EVSVHllAytv/2P364c/s5yyUj2cSvqnQBdJQb97fR+f+8XfAPj0u+fw5QsXpWRzkKQWBbpIinlua1V3mF93wUL++cy5CnOJiwJdJAaXpG4uVfWtfPTuvwLwrQ+eOCIjw0twKdBFoiT7PPiUW54C4Purl3LxidOSXI34TTAH1hPxmea2Tkqv+x0AC6fkK8xlWBToIilg0U2Pd08/cpVGc5ThUaCLJNFP/rSj+8wc4KlrziIrPZTEisTP1IYukgQNrR3c9thm7ntxV/e8zV9fRXaGwlyGT4EuEoMbwU4uzjmWfOWJ7tfnLi7h+6uXKszlqCnQRaKMRnfv+Tc81j39g388mQuO1638khgKdJFRVFnXQqc3JJyaWCTRdFFUZBQ98upeAJ6+9iyFuSScztBlTDjU2EbF4SYMI+wcWw7UU9vUzvaqBto6w9Q0tbO3ppm3qxsBqG5oTXgNzjluWfcWx5bkM684L+HvL6JAl5RS19JOZV0r+2qbMYzcrBClE3MJhYy/7jjE1sp6Kg43s+tgI7MmjGNG0TjGj8sgHHaEHeRkhGjtDNPS1snLOw+xeX89B+paaO0Ix/x5E3MzMTOKxmUwbXwO2yobADjU2J7wbXt2SyUAS2dpoGYZGQp0GRXOOTbsOszjb+5ny4F66lo62FfTzKHGNrLS00gPpdHU1kF75+DdS/Kz0ykpyGb9zsP9BjVERrpfOCWfM4+ZxKS8LKYV5pCblU5HOMwxJfmUFGQzITezx/e8XlHDJXc+z4LJiT2D3lPTzLd/vxWAr1x8XELfW6SLAl0Soqapjdcratl6oJ5dB5swg/qWDnYebKS5rZPtVQ20dzoyQsb08TmUFGRzypwJTBiXSUc4TGNrJ0XjMigpzCY/O4MZ43MIO0drR5gt++tpbu/kvQsns2hKAYXjMoDI7fKNbR00tXZiBqE0o6mtk6z0NLIzQkzKyxzyUwptBJ7mUl5Zz8rbnwMgPc3IyVTbuYwMBboMSTjs2FPTzMHGNmqa2vjTtmr+sKWS7VWNfdYtyE6ndFIuuVnpXHzCNE6fO5GLTphKbtbQDrsL++nWl5MZioRjCjdHRwapeK779QP/vDyJ1UjQxfWXZWargO8BIeAnzrlbY6zzD8BXiQyv+5pz7ooE1imjaEdVA/trW6hpbmfz/npqm9poaQ/zwPrdMdc/fnohn1hRypJphZw2dwLTx+cAqTk+52j70R+3d0/vvPWiJFYiY8GggW5mIeAu4FygAnjZzNY65zZFrbMAuB5Y4Zw7bGaTR6pgGTlb9tdz7UOv8uaeun7XOWnmeM47roT87AymFGRzaumE7iYQ6WvrgXqOKcnjsc+fmexSZAyI5wz9VKDcObcDwMzuBy4Foocg/zRwl3PuMIBzrjLRhUpihcOOJzbu598eeo3Gts4ey46bVsCE3EzmT85j2ewixmWGOG5aIZPzs3TWPQTllQ28VlHLFafNIpSm35uMvHgCfToQ/Vm7Ajit1zrHAJjZ80SaZb7qnHu81zqY2ZXAlQCzZs0aTr1j0q6DjXzxgVf52zs13fNmFOXw/pOmc/yMQu54ehtXv3d+92jwnWHXHSDbqxq49bHNTB+fQyjNeHHHQXZWN/YJ8cz0NM5bXMJnzprHkumFo7dxAXb7k1sAeP9J05NciYwViboomg4sAM4GZgDPmdnxzrma6JWcc2uANQBlZWXJGeMrxXSGHW9XN1Ld0Mr8yXnkZ6fz9Uc38bMX3xnw+yoON3Pns+Xdrz/zs78N6efmZaWzeFoBH1g6nUtPmsa4TF0fj5aIg3PdG/sBOHXOhAS8m8jg4vkr3gPMjHo9w5sXrQJ4yTnXDrxtZluJBPzLCakyIL78mzd45JU9fc6O4/G598zjqvcs4M29tRxqbKOprYMvPvAaZvDzT57GFT95qc/3FGSnM7UwhytOm8X5x03h8Tf3sWz2BBZPK1ATQD8S1aJU3xK5Mekjp89OzBuKxCGeQH8ZWGBmc4gE+eVA7x4svwVWA/eY2SQiTTA7ElmoX71RUcu//+p13trX/4XG3uYV53LjRYspKcjm+fJqPvXuOd1t16eUHjnb+8DSIwMI77z1IpxzA7Zxf3zFnGFsgQxH101Eq5ZMSXIlMpYMGujOuQ4zuwp4gkj7+N3OuY1mdjOw3jm31lt2npltAjqB/+OcOziShftB9Eg0XT6xopRJeVlccuI0apvbB22vXjytIO6fpwuWqeNXGyoAWDa7KMmVyFgSV8Opc24dsK7XvJuiph1wjfc1ZoXDjnd/81n21DT3WXbzpcfxoVNm9hhebGaftSQIqupbqW/tANATFWVU6UpYgrxRUcvFd/65z/yt/3EBmel6SvFY8uKOyIfTGy9alORKZKxRoB+llvZOFv7fPj00ee0r51GYoxtu/ModxRh0V//yFQA+WKbPYDK6FOhD1NDaQUbIOPbGviEOGoVmrOvoPPL0R/2HLqNNgR6nmx55k//5y65+lyvIBeCyH7wAwFnHFCe5EhmLFOiDKK9s4PzvPtc9DmS0Z649i7kaeUaivFZRC8CdVyxNciUyFinQ+1HT1MamfXVc8eMjN+z88MPLOP+4EnUPlJhe3X3kxuj8bDW3yOhToMfgnOOkm5/sMW/NR5Zx3nG6SUT69/67ngfgCysXJLkSGasU6DFEPwRrXGaITTevSmI1kgxD7eMS3SvmCyuPSWwxInFSoMfQdWHrrzecw+T87CRXI6NpuK1pc65fN/hKIiNMd7z0Ut3Q2j2tMJehevTqM5JdgoxhCvRe7vO6Jq75yLIkVyJ+0dYR6Xs+tzhXz5KXpFKTiyccdjy+cT/fe3obAOcuLklyReIXV/8y8iz6hVPyk1yJjHUKdCKDTMz78pE2UHVNlKF4YuMBAL6oi6GSZL5rcqmsb+Gbj28mHONGn+FwrmeYQ6S/uYxt8T7KJfpW/wUlOkOX5PLdGfqXHn6dZ7dUccaCSbxr3qSjeq/fvFLBn7ZVd79+8fpzmFKoC6FjmTG0T2Z7a1oAuO2y40eiHJEh8V2gt3oXoI7iYXgA3PjbN3qM2/mdD52oMJchu+eFtwE0JqukBN81uXQ5mkAPh12fQZijh3MTidcb3rNbzj5WD+OS5PPdaUUirlXOjWozf+RzKzheXc1kmNbvOgzo2S2SGnwX6Efb1PK+7/+pe/rlG1ZSnJ91lBXJWJWoC/MiieK7QB+uB15+hy/96o3u12cfW6wwl6Py4PrdyS5BpAfftaEPp8ll497aHmEOcM/HT0lQRRJMg5997z7cBMD3V+vZ55IafBfoXdwQnod30R09B28uv+UC3TgkMQ3lsMhKj4xQpbuKJVUEvskleqSht79xoYJcEqKlvZPbn9wKoKEHJWX49gwdoPS63/GZ+zYMuM6ew83d0wpzSZSv/e/GZJcg0ofvAr3rTr4O78z78Y37eW5rVb/rf/+ZyMO2fvXZd418cTJmnDRzPACfe8+8JFcicoTvAr1Le8eRZ2hsPVDfZ3ljawd/2FLJQxsqAFg8tWDUapPg++mfI3eIfv4cPZBLUodv29Dboh6K1Nja2Wf5h3/6Eq9EDSWXk6l2TonfQPc7/OiP29l6oAGAzHTfnhNJAPnuaOwK8pb2I4E+e+K4PutFh3muwlziNNhllpb2Tr7x2ObRKUZkiHwX6H99+xAAP39pV/e85va+Z+jR/vPv9CQ8SYzKuiNDFN7zCd3LIKnFd4HepbapvXu6sbWjx7Kq+iN/dCvmT+SSE6eNWl0SbBU1Td3T7zl2chIrEenLt23o7eEjTS4NvQJ9+TeeBuDT757DDRctHtW6JNiu+PFLAHzzshOSXIlIX3GdoZvZKjPbYmblZnbdAOtdZmbOzMoSV2Jsuw8d6V/e+wy9q0vjiV7XMpFEu+QkfeqT1DNooJtZCLgLuABYDKw2sz6nvWaWD3weeCnRRQ7mx396u3u6uuFIc8uFS6aOdikSELE6ubR2HLlWo7tDJRXFc4Z+KlDunNvhnGsD7gcujbHe14HbgJYE1he3TXvrONzYRtl/PAXA8rkTSUvTnaEyNF1dYLsuvkfbX5uUQ1skbvEE+nQg+jmhFd68bmZ2MjDTOfe7gd7IzK40s/Vmtr6qqv+7OweyclHsC1HbKutZ+vUnu19ff+HCYb2/jG0b90ZGILr3hZ19ltV4F+J/8tERb1EUGZaj7uViZmnA7cC1g63rnFvjnCtzzpUVFw9vyK7+Rob51u+39HitUYhkOAb6THeosQ2AiXmZo1OMyBDFE+h7gJlRr2d487rkA0uAP5jZTuB0YO1oXBgFeOqaM4GeF0kXTy3Qg7hkeAY4bv6y4yAAE3IV6JKa4gn0l4EFZjbHzDKBy4G1XQudc7XOuUnOuVLnXCnwInCJc279iFTcy4yivneJPnr1GaPxoyWABjoN+M0rkfOYAo0fKilq0EB3znUAVwFPAG8BDzrnNprZzWZ2yUgX2FvvP7jevQ1+8anTdDFUhm2gD3YfWDqdzFAaRTpDlxQV141Fzrl1wLpe827qZ92zj76sAcT4g7vyzLmseW4HABq3V45G2gCJXt/STuE4nZ1L6vLdrf8WI9H/acWc7umTZulmIhm+gT7b1bV0kJ/l25urZQzwXaDHEt3rIE9/cHIUBmpyaWjpID9bx5ekLt8dnbH+4DJCaXxp1ULOmD9p9AuSMaO+pb3fbrMiqcB/gd7P/M+eraHA5OjFatLrUt/SweT87FGsRmRofNfkEn2Grjv2JNEGanKpV5OLpDjfBXq04vysZJcgAdPfDWl7a5rZX9fCc9uG98gKkdHgu0CP/kgcUn9zSbD+jqhrHnwVgANRIxaJpBr/BXrUX9xAfYZFhiPWIRUOO17cEXn64tcuOW6UKxKJn+8CPZrO0CXRYgV6Y9uRAVQuWzZjFKsRGRqfB3qyK5CxoK7lSKDnZmpgC0ldvotENbnISIp1TH33ya3d03qKp6Qy3wU6uigqIyhWYD+0oQKA952gIQ0ltfku0KP/3hTokmgDHVFfWHnMqNUhMhz+C/SoaQW6jKb5k/OSXYLIgHx321uPM3S1Z0qCxTqksjPSuOLU2aNfjMgQ+e4MPfqilQaykETr/SyXyroWWtrD7KhuSFJFIvHzdaDrDF0Srfch9dRblQD8YYtu+ZfU57tAj6YzdEm03odUSUHkeUG3fGBJEqoRGRrfBXqPM3QFuiRcz2NqW2WkqeWU0gnJKEZkSHwY6Eem1eQiiRZ9SNW3tHPrY5sB6NRgteID/gv0NJ2hy+g4/qu/754+piQ/iZWIxMd3ga5+6DKS+juidKyJH/gv0KO7LepvTESkmw8DPXpaiS6JNWviuGSXIDJs/gv0ZBcggbZwSkGySxAZNv8FuhJdRli62vLEp/wX6DpHlxHW+6ShbHZRcgoRGSL/BbryXEZY75OGhz/7riRVIjI0/gv0ZBcggRd90nDZyRpDVPzDd4GuU3QZadGH2KUnTUteISJDFFegm9kqM9tiZuVmdl2M5deY2SYze93MnjazEXt4tOJcRlp0k8uK+ZOSWInI0Awa6GYWAu4CLgAWA6vNbHGv1V4BypxzJwAPA99MdKFH6hmpdxaJ6DkQefLqEBmqeM7QTwXKnXM7nHNtwP3ApdErOOeedc41eS9fBEas4VG9XGSkRR9hunlN/CSeQJ8O7I56XeHN688ngcdiLTCzK81svZmtr6oa3oAB4zJDw/o+kXgpxMWvEnpR1Mw+DJQB/xVruXNujXOuzDlXVlxcPKyfkRHSH5uMLOW5+FU8g0TvAWZGvZ7hzevBzFYCNwBnOedaE1OeyOhTnotfxXOG/jKwwMzmmFkmcDmwNnoFM1sK/Ai4xDlXmfgyRUZPXUsHAOPHZSS5EpGhGTTQnXMdwFXAE8BbwIPOuY1mdrOZXeKt9l9AHvCQmb1qZmv7eTsR35hamJPsEkSGJJ4mF5xz64B1vebdFDW9MsF1iSTdW/vqkl2CyJD47k5RjewoIhKb7wK9y9ffvyTZJYiIpBTfBvrFJ0xNdgkiIinFt4EuIiI9+S7QnRrRRURi8l2gd9EzXWSkffuDJya7BJEh8W2gi4y0jHT9eYi/6IgV6UdHZzjZJYgMie8CXU3oMlo6OnW0ib/4LtC7qQldRtgZCzRakfiLfwNdZIRNG69nuYi/KNBFRAJCgS4iEhBxPW0xlTjdWSQj7N/OO4bqhrZklyEyZL4L9C4aJkxGylXvXZDsEkSGRU0uIiIBoUAXEQkIBbqISED4NtDVhC4i0pNvA11ERHpSoIuIBITvAl3d0EVEYvNdoHcxdUQXEenBt4EuIiI9KdBFRAJCgS4iEhC+C3SnMYtERGLyXaB30SVREZGefBvoIiLSkwJdRCQgfBfourFIRCS2uALdzFaZ2RYzKzez62IszzKzB7zlL5lZaaIL7fszR/oniIj4y6CBbmYh4C7gAmAxsNrMFvda7ZPAYefcfOA7wG2JLlRERAYWzxn6qUC5c26Hc64NuB+4tNc6lwL/7U0/DJxjujdfRGRUxRPo04HdUa8rvHkx13HOdQC1wMTeb2RmV5rZejNbX1VVNayC5xbncdHxU0nT/xciIj2M6iDRzrk1wBqAsrKyYV3ePHdxCecuLkloXSIiQRDPGfoeYGbU6xnevJjrmFk6UAgcTESBIiISn3gC/WVggZnNMbNM4HJgba911gIf86b/HnjGOXUwFBEZTYM2uTjnOszsKuAJIATc7ZzbaGY3A+udc2uBnwL3mVk5cIhI6IuIyCiKqw3dObcOWNdr3k1R0y3ABxNbmoiIDIXv7hQVEZHYFOgiIgGhQBcRCQgFuohIQFiyeheaWRWwa5jfPgmoTmA5fqBtHhu0zWPD0WzzbOdccawFSQv0o2Fm651zZcmuYzRpm8cGbfPYMFLbrCYXEZGAUKCLiASEXwN9TbILSAJt89igbR4bRmSbfdmGLiIiffn1DF1ERHpRoIuIBITvAn2wAav9wsxmmtmzZrbJzDaa2ee9+RPM7Ekz2+b9W+TNNzO7w9vu183s5Kj3+pi3/jYz+1h/PzNVmFnIzF4xs0e913O8wcXLvcHGM735/Q4+bmbXe/O3mNn5ydmS+JjZeDN72Mw2m9lbZrY86PvZzL7oHddvmtkvzSw7aPvZzO42s0ozezNqXsL2q5ktM7M3vO+5wyyOYdqcc775IvL43u3AXCATeA1YnOy6hrktU4GTvel8YCuRQbi/CVznzb8OuM2bvhB4DDDgdOAlb/4EYIf3b5E3XZTs7Rtk268BfgE86r1+ELjcm/4h8Flv+l+AH3rTlwMPeNOLvX2fBczxjolQsrdrgO39b+BT3nQmMD7I+5nIkJRvAzlR+/fjQdvPwJnAycCbUfMStl+Bv3rrmve9FwxaU7J/KUP8BS4Hnoh6fT1wfbLrStC2PQKcC2wBpnrzpgJbvOkfAauj1t/iLV8N/Chqfo/1Uu2LyIhXTwPvBR71DtZqIL33PibyDP7l3nS6t5713u/R66XaF5HRu97G64DQe/8FcT9zZIzhCd5+exQ4P4j7GSjtFegJ2a/ess1R83us19+X35pc4hmw2ne8j5hLgZeAEufcPm/RfqBrANX+tt1vv5PvAv8OhL3XE4EaFxlcHHrW39/g437a5jlAFXCP18z0EzPLJcD72Tm3B/gW8A6wj8h+20Cw93OXRO3X6d507/kD8lugB46Z5QG/Ar7gnKuLXuYi/zUHpl+pmb0PqHTObUh2LaMoncjH8h8455YCjUQ+incL4H4uAi4l8p/ZNCAXWJXUopIgGfvVb4Eez4DVvmFmGUTC/OfOuV97sw+Y2VRv+VSg0pvf37b76XeyArjEzHYC9xNpdvkeMN4ig4tDz/r7G3zcT9tcAVQ4517yXj9MJOCDvJ9XAm8756qcc+3Ar4ns+yDv5y6J2q97vOne8wfkt0CPZ8BqX/CuWP8UeMs5d3vUougBtz9GpG29a/5HvavlpwO13ke7J4DzzKzIOzM6z5uXcpxz1zvnZjjnSonsu2ecc/8IPEtkcHHou82xBh9fC1zu9Y6YAywgcgEp5Tjn9gO7zexYb9Y5wCYCvJ+JNLWcbmbjvOO8a5sDu5+jJGS/esvqzOx073f40aj36l+yLyoM4yLEhUR6hGwHbkh2PUexHWcQ+Tj2OvCq93UhkbbDp4FtwFPABG99A+7ytvsNoCzqvf4JKPe+PpHsbYtz+9oz+sQAAACPSURBVM/mSC+XuUT+UMuBh4Asb36297rcWz436vtv8H4XW4jj6n+St/UkYL23r39LpDdDoPcz8DVgM/AmcB+RniqB2s/AL4lcI2gn8knsk4ncr0CZ9/vbDtxJrwvrsb5067+ISED4rclFRET6oUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiATE/wd39yu5Ok7vsAAAAABJRU5ErkJggg==\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"Kxcubne-oNic"},"source":["## Contextual bandit dash app\n","\n","> Building a dash app of contextual bandit based recommender system"]},{"cell_type":"markdown","metadata":{"id":"pI_Y3hYnorHh"},"source":["The objective of this app is to apply the contextual bandit algorithms to recommendation problem under a simulated environment. The recommender agent is able to quickly adapt the changing behavior of users and change the recommendation strategy accordingly.\n","\n","There are 3 users: Alex, Ben and Cindy. There are 6 news topics and 2 types of context. That means, Alex, Ben, and Cindy might prefer to read news reated to any of the 6 topics on morning/evening and weekday/weekends. Eg. Alex might prefer business related news on weekday mornings and entertainment related news on weekend evenings. And it is also possible that in future, Alex starts reading politics on weekday mornings. These situations reflect the real-world scenarios and the job of our contextual agent is to automatically detect these preferences and changes and recommend the items accordingly to maximize the reward like user satisfaction.\n","\n","[https://www.youtube.com/watch?v=9t0-FZIWMRQ](https://www.youtube.com/watch?v=9t0-FZIWMRQ)\n","\n","In the example, agent initialized with random preferences and starts recommending news to the users. We added 2 context: \"Cindy prefers economy news on weekday mornings\" and \"Ben prefers weather news on weekday mornings\" and starts rewarding agent for correctly recommending as per these preferences. At the moment, agent knows that Ben prefers business news and Cindy prefers history news. With time, agent started recommending weather news to Ben. Similar case we will see for Cindy and in fact, for all users.\n","\n","Note: Interval is 1 sec but we are not seeing updates every second because we are looking at a particular context only: Weekday mornings but agent is recommending globally.\n","\n","It is important to note that agent do not know the ground truth. It just taking action and receiving reward and the objective is to estimate this ground truth preferences."]},{"cell_type":"code","metadata":{"id":"YVP4fTkR3duW"},"source":["!pip install -q dash dash-html-components dash-core-components dash_bootstrap_components jupyter-dash"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Wq_hAT_t5AGN","executionInfo":{"status":"ok","timestamp":1634749093998,"user_tz":-330,"elapsed":831,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"62d27a73-ab02-48b1-c246-d6fb66bcfbd8"},"source":["!mkdir -p assets\n","!wget -q --show-progress -O assets/image.jpg https://moodle.com/wp-content/uploads/2020/04/Moodle_General_news.png"],"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["\rassets/image.jpg      0%[                    ]       0  --.-KB/s               \rassets/image.jpg    100%[===================>]  26.00K  --.-KB/s    in 0s      \n"]}]},{"cell_type":"code","metadata":{"id":"pcv55-nd3WWy"},"source":["import dash\n","from dash import dcc\n","import dash_html_components as html\n","import dash_bootstrap_components as dbc\n","from dash.dependencies import Input, Output, State\n","from jupyter_dash import JupyterDash\n","import plotly.graph_objs as go\n","import plotly.express as px\n","\n","from vowpalwabbit import pyvw\n","\n","import numpy as np\n","import pandas as pd\n","import itertools\n","import pathlib\n","from copy import deepcopy\n","from itertools import product\n","import scipy\n","import scipy.stats as stats\n","import random"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"8mLLJQJn5Wws"},"source":["# This function modifies (context, action, cost, probability) to VW friendly format\n","def to_vw_example_format(context, actions, cb_label=None):\n","    if cb_label is not None:\n","        chosen_action, cost, prob = cb_label\n","    example_string = \"\"\n","    example_string += \"shared |User users={} context1={} context2={}\\n\".format(context[\"user\"], context[\"context1\"], context[\"context2\"])\n","    for action in actions:\n","        if cb_label is not None and action == chosen_action:\n","            example_string += \"0:{}:{} \".format(cost, prob)\n","        example_string += \"|Action items={} \\n\".format(action)\n","    #Strip the last newline\n","    return example_string[:-1]\n","\n","\n","def sample_custom_pmf(pmf):\n","    total = sum(pmf)\n","    scale = 1 / total\n","    pmf = [x * scale for x in pmf]\n","    draw = random.random()\n","    sum_prob = 0.0\n","    for index, prob in enumerate(pmf):\n","        sum_prob += prob\n","        if(sum_prob > draw):\n","            return index, prob\n","\n","\n","def get_action(vw, context, actions):\n","    vw_text_example = to_vw_example_format(context, actions)\n","    pmf = vw.predict(vw_text_example)\n","    chosen_action_index, prob = sample_custom_pmf(pmf)\n","    return actions[chosen_action_index], prob\n","\n","\n","def choose_user(users):\n","    return random.choice(users)\n","\n","\n","def choose_context1(context1):\n","    return random.choice(context1)\n","\n","\n","def choose_context2(context2):\n","    return random.choice(context2)\n","    \n","\n","class VWCSimulation():\n","    def __init__(self, vw, ictxt):\n","        self.vw = vw\n","        self.users = ictxt['users'].unique().tolist()\n","        self.contexts1 = ictxt['context1'].unique().tolist()\n","        self.contexts2 = ictxt['context2'].unique().tolist()\n","        self.actions = ictxt['items'].unique().tolist()\n","        self.contextdf = ictxt.copy()\n","        self.contextdf['cost'] = self.contextdf['reward']*-1\n","        \n","    def get_cost(self, context, action):\n","        return self.contextdf.loc[(self.contextdf['users']==context['user']) & \\\n","                (self.contextdf.context1==context['context1']) & \\\n","                (self.contextdf.context2==context['context2']) & \\\n","                (self.contextdf['items']==action), \\\n","                'cost'].values[0]\n","    \n","    def update_context(self, new_ctxt):\n","        self.contextdf = new_ctxt.copy()\n","        self.contextdf['cost'] = self.contextdf['reward']*-1\n","    \n","    def step(self):\n","        user = choose_user(self.users)\n","        context1 = choose_context1(self.contexts1)\n","        context2 = choose_context2(self.contexts2)\n","        context = {'user': user, 'context1': context1, 'context2': context2}\n","        action, prob = get_action(self.vw, context, self.actions)\n","        cost = self.get_cost(context, action)\n","        vw_format = self.vw.parse(to_vw_example_format(context, self.actions, (action, cost, prob)), pyvw.vw.lContextualBandit)\n","        self.vw.learn(vw_format)\n","        self.vw.finish_example(vw_format)\n","        return (context['user'], context['context1'], context['context2'], action, cost, prob)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"IEd6IKmv3NLq"},"source":["app = JupyterDash(__name__, external_stylesheets = [dbc.themes.BOOTSTRAP])\n","\n","def generate_input_cards(preference='Random'):\n","    card_content = [\n","    dbc.CardImg(src=\"assets/image.jpg\", top=True),\n","    dbc.CardBody([html.P(preference, className=\"card-title\")])\n","    ]\n","    card = dbc.Card(card_content, color=\"primary\", outline=True)\n","    return dbc.Col([card], width={\"size\": 2})\n","\n","pref_grid = []\n","\n","mapping_users = {\n","    'Alex':'usera',\n","    'Ben':'userb',\n","    'Cindy': 'userc'\n","}\n","    \n","mapping_context1 = {\n","    'Morning':'ctx11',\n","    'Evening':'ctx12',\n","}\n","\n","mapping_context2 = {\n","    'Weekday':'ctx21',\n","    'Weekend':'ctx22'\n","}\n","\n","mapping_items = {\n","    'Politics':'item1',\n","    'Economics':'item2',\n","    'Technology':'item3',\n","    'Weather':'item4',\n","    'Business':'item5',\n","    'History':'item6'\n","}\n","\n","mapping_users_reverse = {v:k for k,v in mapping_users.items()}\n","mapping_context1_reverse = {v:k for k,v in mapping_context1.items()}\n","mapping_context2_reverse = {v:k for k,v in mapping_context2.items()}\n","mapping_items_reverse = {v:k for k,v in mapping_items.items()}\n","\n","users = list(mapping_users.values())\n","items = list(mapping_items.values())\n","context1 = list(mapping_context1.values())\n","context2 = list(mapping_context2.values())\n","\n","context = pd.DataFrame(list(product(users, context1, context2, items)),\n","                       columns=['users', 'context1', 'context2', 'items'])\n","context['reward'] = np.random.choice([0,1],len(context),p=[0.8,0.2])\n","\n","vw = pyvw.vw(\"--cb_explore_adf -q UA --quiet --epsilon 0.2\")\n","vws = VWCSimulation(vw, context)\n","last_update = vws.step()\n","\n","contextdf = context.copy()\n","countDF = contextdf.copy()\n","countDF['prob'] = 0\n","\n","def generate_input_boxes():\n","    dropdown_users = dcc.Dropdown(\n","        id='ddown_users',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_users.items()],\n","        clearable=False,\n","        value=\"usera\",\n","        className=\"m-1\",\n","    )\n","    dropdown_context1 = dcc.Dropdown(\n","        id='ddown_ctx1',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_context1.items()],\n","        clearable=False,\n","        value=\"ctx11\",\n","        className=\"m-1\",\n","    )\n","    dropdown_context2 = dcc.Dropdown(\n","        id='ddown_ctx2',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_context2.items()],\n","        clearable=False,\n","        value=\"ctx21\",\n","        className=\"m-1\",\n","    )\n","    dropdown_items = dcc.Dropdown(\n","        id='ddown_items',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_items.items()],\n","        clearable=False,\n","        value=\"item1\",\n","        className=\"m-1\",\n","    )\n","    return html.Div(\n","        [\n","            dropdown_users,\n","            dropdown_context1,\n","            dropdown_context2,\n","            dropdown_items,\n","        ],\n","        style={\"display\": \"flex\", \"flex-direction\": \"column\"},\n","    )\n","\n","def generate_context_boxes():\n","    dropdown_outcontext1 = dcc.Dropdown(\n","        id='ddown_outctx1',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_context1.items()],\n","        clearable=False,\n","        value=\"ctx11\",\n","        className=\"m-1\",\n","    )\n","    dropdown_outcontext2 = dcc.Dropdown(\n","        id='ddown_outctx2',\n","        options=[{\"label\":k, \"value\":v} for k,v in mapping_context2.items()],\n","        clearable=False,\n","        value=\"ctx21\",\n","        className=\"m-1\",\n","    )\n","    return html.Div(\n","        [\n","            dropdown_outcontext1,\n","            dropdown_outcontext2\n","        ],\n","        style={\"display\": \"flex\", \"flex-direction\": \"column\"},\n","    )\n","\n","app.layout = html.Div([\n","        generate_input_boxes(),\n","        dbc.Button(\"Register your Preference\", color=\"primary\", className=\"m-1\", \n","                   id='pref-button'),\n","        html.Div(id='pref-grid'),\n","        dbc.Button(\"Clear the context\", color=\"secondary\", \n","                   className=\"m-1\", id='clr-button'),\n","        dbc.Button(\"Start rewarding Agent for these Preferences\", color=\"success\", \n","                   className=\"m-1\", id='updt-button'),\n","        generate_context_boxes(),\n","        dcc.Interval(\n","            id='interval-component',\n","            interval=100, # in milliseconds\n","            n_intervals=0),\n","        html.Div(id='placeholder'),\n","        html.Div(id='placeholder2'),\n","\n","])\n","\n","@app.callback(\n","    Output(\"pref-grid\", \"children\"),\n","    Input(\"pref-button\", \"n_clicks\"),   \n","    Input(\"clr-button\", \"n_clicks\"),\n","    State('ddown_users', 'value'),\n","    State('ddown_items', 'value'),\n","    State('ddown_ctx1', 'value'), \n","    State('ddown_ctx2', 'value'),\n",")\n","def update_pref_grid(nclick_pref, nclick_clr, pref_user, pref_item, pref_ctx1, pref_ctx2):\n","    global pref_grid\n","    changed_id = [p['prop_id'] for p in dash.callback_context.triggered][0]\n","    if \"pref-button\" in changed_id:\n","        global contextdf\n","        card_text = '{} prefers {} related news in {} {}s'.format(mapping_users_reverse[pref_user],\n","                                                  mapping_items_reverse[pref_item],\n","                                                  mapping_context2_reverse[pref_ctx2],\n","                                                  mapping_context1_reverse[pref_ctx1])\n","        \n","        contextdf.loc[(contextdf.users==pref_user) & \\\n","            (contextdf.context1==pref_ctx1) & \\\n","            (contextdf.context2==pref_ctx2), \\\n","            'reward'] = 0\n","        contextdf.loc[(contextdf.users==pref_user) & \\\n","            (contextdf.context1==pref_ctx1) & \\\n","            (contextdf.context2==pref_ctx2) & \\\n","            (contextdf['items']==pref_item), \\\n","            'reward'] = 1\n","        pref_grid.append(generate_input_cards(card_text))\n","        return dbc.Row(children=pref_grid,\n","                      style={'max-width': '100%',\n","                             'display': 'flex',\n","                             'align-items': 'center',\n","                             'padding': '2rem 5rem',\n","                             'overflow': 'auto',\n","                             'height': 'fit-content',\n","                             'flex-direction': 'row',\n","                            })\n","    elif \"clr-button\" in changed_id:\n","        pref_grid = []\n","        return dbc.Row(children=pref_grid)\n","\n","@app.callback(\n","    Output(\"placeholder2\", \"children\"),\n","    Input(\"updt-button\", \"n_clicks\")\n",")\n","def update_context(nclick):\n","    if nclick:\n","        global vws\n","        global contextdf\n","        vws.update_context(contextdf)\n","    return ''\n","\n","\n","@app.callback(\n","    Output(\"placeholder\", \"children\"),\n","    Input('interval-component', 'n_intervals'),\n","    Input('ddown_outctx1', 'value'), \n","    Input('ddown_outctx2', 'value'),\n",")\n","def update_metrics(n, octx1, octx2):\n","    global countDF\n","    countDF = countDF.append(pd.Series(vws.step(),countDF.columns),ignore_index=True)\n","    _x = countDF.copy()\n","    _x = _x[(_x.context1==octx1) & (_x.context2==octx2)]\n","    _x['reward']*=-1\n","    pv = pd.pivot_table(_x, index=['users'], columns=[\"items\"], values=['reward'], aggfunc=sum, fill_value=0)\n","    pv.index = [mapping_users_reverse[x] for x in pv.index]\n","    pv.columns = pv.columns.droplevel(0)\n","    pv = pv.rename_axis('User').reset_index().rename_axis(None, axis=1).set_index('User').T.reset_index()\n","    pv['index'] = pv['index'].map(mapping_items_reverse)\n","    pv = pv.rename(columns={\"index\": \"Preferences\"})\n","    out = html.Div([\n","        dbc.Table.from_dataframe(pv, striped=True, bordered=True, hover=True, responsive=True)\n","    ])\n","    return out"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":671},"id":"Cm6QckQy4BeO","executionInfo":{"status":"ok","timestamp":1634749335337,"user_tz":-330,"elapsed":12,"user":{"displayName":"Sparsh Agarwal","photoUrl":"https://lh3.googleusercontent.com/a/default-user=s64","userId":"13037694610922482904"}},"outputId":"8c426484-039f-4403-8546-b017fa284e4f"},"source":["app.run_server(mode='inline', port=8081)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"application/javascript":["(async (port, path, width, height, cache, element) => {\n","    if (!google.colab.kernel.accessAllowed && !cache) {\n","      return;\n","    }\n","    element.appendChild(document.createTextNode(''));\n","    const url = await google.colab.kernel.proxyPort(port, {cache});\n","    const iframe = document.createElement('iframe');\n","    iframe.src = new URL(path, url).toString();\n","    iframe.height = height;\n","    iframe.width = width;\n","    iframe.style.border = 0;\n","    element.appendChild(iframe);\n","  })(8081, \"/\", \"100%\", 650, false, window.element)"],"text/plain":["<IPython.core.display.Javascript object>"]},"metadata":{}}]},{"cell_type":"code","metadata":{"id":"Lg14PTYT4CuG"},"source":["# !kill -9 $(lsof -t -i:8081) # command to kill the dash once done"],"execution_count":null,"outputs":[]}]}