{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "2022-01-07-ncf.ipynb", "provenance": [], "collapsed_sections": [], "authorship_tag": "ABX9TyO+EzcZLxWCVB+iE5tVY1BN" }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "iVkUysixCyk6" }, "source": [ "# Neural Collaborative Filtering Recommenders" ] }, { "cell_type": "code", "metadata": { "id": "-xzeGn4mvtvd" }, "source": [ "!pip install -q pytorch-lightning" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "xNgD2QXOkq4g" }, "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import pandas as pd\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "from torch.utils.data import Dataset, DataLoader\n", "from torch.utils.data import TensorDataset\n", "from tqdm.notebook import tqdm\n", "import pytorch_lightning as pl\n", "\n", "np.random.seed(123)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8fqqbAgmrxoT" }, "source": [ "## NCF with PyTorch Lightning on ML-25m\n", "\n", "In this section, we will build a simple yet accurate model using movielens-25m dataset and pytorch lightning library. This will be a retrieval model where the objective is to maximize recall over precision." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "MrlkpJITv8Rw", "outputId": "4fd97487-1f04-4563-8adf-5aa306c13d2b" }, "source": [ "!wget -q --show-progress https://files.grouplens.org/datasets/movielens/ml-25m.zip\n", "!unzip ml-25m.zip" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ml-25m.zip.1 100%[===================>] 249.84M 45.7MB/s in 5.9s \n", "Archive: ml-25m.zip\n", "replace ml-25m/tags.csv? [y]es, [n]o, [A]ll, [N]one, [r]ename: N\n" ] } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "5-juuyKOwCmL", "outputId": "93ff3c63-a959-4e2d-fdff-c5e72a475160" }, "source": [ "ratings = pd.read_csv('ml-25m/ratings.csv', infer_datetime_format=True)\n", "ratings.head()" ], "execution_count": null, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
012965.01147880044
113063.51147868817
213075.01147868828
316655.01147878820
418993.51147868510
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "0 1 296 5.0 1147880044\n", "1 1 306 3.5 1147868817\n", "2 1 307 5.0 1147868828\n", "3 1 665 5.0 1147878820\n", "4 1 899 3.5 1147868510" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ] }, { "cell_type": "markdown", "metadata": { "id": "A2bKYi9WwGyP" }, "source": [ "### Subset\n", "\n", "In order to keep memory usage manageable, we will only use data from 20% of the users in this dataset. Let's randomly select 30% of the users and only use data from the selected users." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "HsYAMEXqwZoH", "outputId": "434b4209-8c53-4474-edd8-6c5d109c3184" }, "source": [ "rand_userIds = np.random.choice(ratings['userId'].unique(), \n", " size=int(len(ratings['userId'].unique())*0.2), \n", " replace=False)\n", "\n", "ratings = ratings.loc[ratings['userId'].isin(rand_userIds)]\n", "\n", "print('There are {} rows of data from {} users'.format(len(ratings), len(rand_userIds)))" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There are 5015129 rows of data from 32508 users\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "w7OS6UqDpXdi" }, "source": [ "### Train/Test Split\n", "**Chronological Leave-One-Out Split**" ] }, { "cell_type": "markdown", "metadata": { "id": "ZaqAMrH-gn_i" }, "source": [ "Along with the rating, there is also a timestamp column that shows the date and time the review was submitted. Using the timestamp column, we will implement our train-test split strategy using the leave-one-out methodology. For each user, the most recent review is used as the test set (i.e. leave one out), while the rest will be used as training data ." ] }, { "cell_type": "markdown", "metadata": { "id": "A4COa9yVguUO" }, "source": [ "> Note: Doing a random split would not be fair, as we could potentially be using a user's recent reviews for training and earlier reviews for testing. This introduces data leakage with a look-ahead bias, and the performance of the trained model would not be generalizable to real-world performance." ] }, { "cell_type": "code", "metadata": { "id": "WtdtS0FMgTez" }, "source": [ "ratings['rank_latest'] = ratings.groupby(['userId'])['timestamp'] \\\n", " .rank(method='first', ascending=False)\n", "\n", "train_ratings = ratings[ratings['rank_latest'] != 1]\n", "test_ratings = ratings[ratings['rank_latest'] == 1]\n", "\n", "# drop columns that we no longer need\n", "train_ratings = train_ratings[['userId', 'movieId', 'rating']]\n", "test_ratings = test_ratings[['userId', 'movieId', 'rating']]" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "XFoYAbhqpnPD" }, "source": [ "### Implicit Conversion" ] }, { "cell_type": "markdown", "metadata": { "id": "HwYvXqJ6hz2u" }, "source": [ "We will train a recommender system using implicit feedback. However, the MovieLens dataset that we're using is based on explicit feedback. To convert this dataset into an implicit feedback dataset, we'll simply binarize the ratings such that they are are '1' (i.e. positive class). The value of '1' represents that the user has interacted with the item.\n", "\n", "> Note: Using implicit feedback reframes the problem that our recommender is trying to solve. Instead of trying to predict movie ratings (when using explicit feedback), we are trying to predict whether the user will interact (i.e. click/buy/watch) with each movie, with the aim of presenting to users the movies with the highest interaction likelihood.\n", "\n", "> Tip: This setting is suitable at retrieval stage where the objective is to maximize recall by identifying items that user will at least interact with." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "sQYuW1Otg_Cg", "outputId": "096aec13-d21e-4690-e88d-258fe9a681a0" }, "source": [ "train_ratings.loc[:, 'rating'] = 1\n", "\n", "train_ratings.sample(5)" ], "execution_count": null, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdrating
98655406404320191
1764897511439826711
190457581235279861
312501220593864871
45403492980345711
\n", "
" ], "text/plain": [ " userId movieId rating\n", "9865540 64043 2019 1\n", "17648975 114398 2671 1\n", "19045758 123527 986 1\n", "3125012 20593 86487 1\n", "4540349 29803 4571 1" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ] }, { "cell_type": "markdown", "metadata": { "id": "uhwZiaBPpsQl" }, "source": [ "### Negative Sampling" ] }, { "cell_type": "markdown", "metadata": { "id": "ngZdyoMjizlw" }, "source": [ "We do have a problem now though. After binarizing our dataset, we see that every sample in the dataset now belongs to the positive class. However we also require negative samples to train our models, to indicate movies that the user has not interacted with. We assume that such movies are those that the user are not interested in - even though this is a sweeping assumption that may not be true, it usually works out rather well in practice.\n", "\n", "The code below generates 4 negative samples for each row of data. In other words, the ratio of negative to positive samples is 4:1. This ratio is chosen arbitrarily but I found that it works rather well (feel free to find the best ratio yourself!)" ] }, { "cell_type": "code", "metadata": { "id": "4T0_UVhTizVn" }, "source": [ "# Get a list of all movie IDs\n", "all_movieIds = ratings['movieId'].unique()\n", "\n", "# Placeholders that will hold the training data\n", "users, items, labels = [], [], []\n", "\n", "# This is the set of items that each user has interaction with\n", "user_item_set = set(zip(train_ratings['userId'], train_ratings['movieId']))\n", "\n", "# 4:1 ratio of negative to positive samples\n", "num_negatives = 4\n", "\n", "for (u, i) in tqdm(user_item_set):\n", " users.append(u)\n", " items.append(i)\n", " labels.append(1) # items that the user has interacted with are positive\n", " for _ in range(num_negatives):\n", " # randomly select an item\n", " negative_item = np.random.choice(all_movieIds) \n", " # check that the user has not interacted with this item\n", " while (u, negative_item) in user_item_set:\n", " negative_item = np.random.choice(all_movieIds)\n", " users.append(u)\n", " items.append(negative_item)\n", " labels.append(0) # items not interacted with are negative" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "9brxnZqlpvXD" }, "source": [ "### PyTorch Dataset" ] }, { "cell_type": "markdown", "metadata": { "id": "Br2u5nn5jAy1" }, "source": [ "Great! We now have the data in the format required by our model. Before we move on, let's define a PyTorch Dataset to facilitate training. The class below simply encapsulates the code we have written above into a PyTorch Dataset class." ] }, { "cell_type": "code", "metadata": { "id": "pCn0M346i6Z8" }, "source": [ "class MovieLensTrainDataset(Dataset):\n", " \"\"\"MovieLens PyTorch Dataset for Training\n", " \n", " Args:\n", " ratings (pd.DataFrame): Dataframe containing the movie ratings\n", " all_movieIds (list): List containing all movieIds\n", " \n", " \"\"\"\n", "\n", " def __init__(self, ratings, all_movieIds):\n", " self.users, self.items, self.labels = self.get_dataset(ratings, all_movieIds)\n", "\n", " def __len__(self):\n", " return len(self.users)\n", " \n", " def __getitem__(self, idx):\n", " return self.users[idx], self.items[idx], self.labels[idx]\n", "\n", " def get_dataset(self, ratings, all_movieIds):\n", " users, items, labels = [], [], []\n", " user_item_set = set(zip(ratings['userId'], ratings['movieId']))\n", "\n", " num_negatives = 4\n", " for u, i in user_item_set:\n", " users.append(u)\n", " items.append(i)\n", " labels.append(1)\n", " for _ in range(num_negatives):\n", " negative_item = np.random.choice(all_movieIds)\n", " while (u, negative_item) in user_item_set:\n", " negative_item = np.random.choice(all_movieIds)\n", " users.append(u)\n", " items.append(negative_item)\n", " labels.append(0)\n", "\n", " return torch.tensor(users), torch.tensor(items), torch.tensor(labels)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "SqMDcEYRjOZN" }, "source": [ "### Model\n", "\n", "While there are many deep learning based architecture for recommendation systems, I find that the framework proposed by He et al. is the most straightforward and it is simple enough to be implemented in a tutorial such as this." ] }, { "cell_type": "code", "metadata": { "id": "xwlBJpqljJvS" }, "source": [ "class NCF(pl.LightningModule):\n", " \"\"\" Neural Collaborative Filtering (NCF)\n", " \n", " Args:\n", " num_users (int): Number of unique users\n", " num_items (int): Number of unique items\n", " ratings (pd.DataFrame): Dataframe containing the movie ratings for training\n", " all_movieIds (list): List containing all movieIds (train + test)\n", " \"\"\"\n", " \n", " def __init__(self, num_users, num_items, ratings, all_movieIds):\n", " super().__init__()\n", " self.user_embedding = nn.Embedding(num_embeddings=num_users, embedding_dim=8)\n", " self.item_embedding = nn.Embedding(num_embeddings=num_items, embedding_dim=8)\n", " self.fc1 = nn.Linear(in_features=16, out_features=64)\n", " self.fc2 = nn.Linear(in_features=64, out_features=32)\n", " self.output = nn.Linear(in_features=32, out_features=1)\n", " self.ratings = ratings\n", " self.all_movieIds = all_movieIds\n", " \n", " def forward(self, user_input, item_input):\n", " \n", " # Pass through embedding layers\n", " user_embedded = self.user_embedding(user_input)\n", " item_embedded = self.item_embedding(item_input)\n", "\n", " # Concat the two embedding layers\n", " vector = torch.cat([user_embedded, item_embedded], dim=-1)\n", "\n", " # Pass through dense layer\n", " vector = nn.ReLU()(self.fc1(vector))\n", " vector = nn.ReLU()(self.fc2(vector))\n", "\n", " # Output layer\n", " pred = nn.Sigmoid()(self.output(vector))\n", "\n", " return pred\n", " \n", " def training_step(self, batch, batch_idx):\n", " user_input, item_input, labels = batch\n", " predicted_labels = self(user_input, item_input)\n", " loss = nn.BCELoss()(predicted_labels, labels.view(-1, 1).float())\n", " return loss\n", "\n", " def configure_optimizers(self):\n", " return torch.optim.Adam(self.parameters())\n", "\n", " def train_dataloader(self):\n", " return DataLoader(MovieLensTrainDataset(self.ratings, self.all_movieIds),\n", " batch_size=512, num_workers=2)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "I8wT1WK9jzeJ" }, "source": [ "We instantiate the NCF model using the class that we have defined above." ] }, { "cell_type": "code", "metadata": { "id": "F3Bh9dorjww7" }, "source": [ "num_users = ratings['userId'].max()+1\n", "num_items = ratings['movieId'].max()+1\n", "\n", "all_movieIds = ratings['movieId'].unique()\n", "\n", "model = NCF(num_users, num_items, train_ratings, all_movieIds)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "K4Mw8CdVp5lF" }, "source": [ "### Model Training" ] }, { "cell_type": "markdown", "metadata": { "id": "xnYHNWe3kRRD" }, "source": [ "> Note: One advantage of PyTorch Lightning over vanilla PyTorch is that you don't need to write your own boiler plate training code. Notice how the Trainer class allows us to train our model with just a few lines of code.\n", "\n", "Let's train our NCF model for 5 epochs using the GPU. " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 375, "referenced_widgets": [ "68bcd7bfc32f4d9ebaba5c08437bca28" ] }, "id": "0JganCIMj2EW", "outputId": "6fa64b89-c835-4f39-d6ad-bac6f2369c64" }, "source": [ "trainer = pl.Trainer(max_epochs=5, gpus=1, reload_dataloaders_every_epoch=True,\n", " progress_bar_refresh_rate=50, logger=False, checkpoint_callback=False)\n", "\n", "trainer.fit(model)" ], "execution_count": null, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "---------------------------------------------\n", "0 | user_embedding | Embedding | 1.3 M \n", "1 | item_embedding | Embedding | 1.7 M \n", "2 | fc1 | Linear | 1.1 K \n", "3 | fc2 | Linear | 2.1 K \n", "4 | output | Linear | 33 \n", "---------------------------------------------\n", "3.0 M Trainable params\n", "0 Non-trainable params\n", "3.0 M Total params\n", "11.907 Total estimated model params size (MB)\n", "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "68bcd7bfc32f4d9ebaba5c08437bca28", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ] }, { "cell_type": "markdown", "metadata": { "id": "R6V1Tiw1kIxk" }, "source": [ "> Note: We are using the argument reload_dataloaders_every_epoch=True. This creates a new randomly chosen set of negative samples for each epoch, which ensures that our model is not biased by the selection of negative samples." ] }, { "cell_type": "markdown", "metadata": { "id": "I3BXx1YzlAUq" }, "source": [ "### Evaluating our Recommender System\n", "\n", "Now that our model is trained, we are ready to evaluate it using the test data. In traditional Machine Learning projects, we evaluate our models using metrics such as Accuracy (for classification problems) and RMSE (for regression problems). However, such metrics are too simplistic for evaluating recommender systems.\n", "\n", "The key here is that we don't need the user to interact on every single item in the list of recommendations. Instead, we just need the user to interact with at least one item on the list - as long as the user does that, the recommendations have worked.\n", "\n", "To simulate this, let's run the following evaluation protocol to generate a list of 10 recommended items for each user.\n", "- For each user, randomly select 99 items that the user has not interacted with\n", "- Combine these 99 items with the test item (the actual item that the user interacted with). We now have 100 items.\n", "- Run the model on these 100 items, and rank them according to their predicted probabilities\n", "- Select the top 10 items from the list of 100 items. If the test item is present within the top 10 items, then we say that this is a hit.\n", "- Repeat the process for all users. The Hit Ratio is then the average hits." ] }, { "cell_type": "markdown", "metadata": { "id": "B2PVVpUflN34" }, "source": [ "> Note: This evaluation protocol is known as Hit Ratio @ 10, and it is commonly used to evaluate recommender systems." ] }, { "cell_type": "code", "metadata": { "id": "uSLTYZuhlNEV" }, "source": [ "# User-item pairs for testing\n", "test_user_item_set = set(zip(test_ratings['userId'], test_ratings['movieId']))\n", "\n", "# Dict of all items that are interacted with by each user\n", "user_interacted_items = ratings.groupby('userId')['movieId'].apply(list).to_dict()\n", "\n", "hits = []\n", "for (u,i) in tqdm(test_user_item_set):\n", " interacted_items = user_interacted_items[u]\n", " not_interacted_items = set(all_movieIds) - set(interacted_items)\n", " selected_not_interacted = list(np.random.choice(list(not_interacted_items), 99))\n", " test_items = selected_not_interacted + [i]\n", " \n", " predicted_labels = np.squeeze(model(torch.tensor([u]*100), \n", " torch.tensor(test_items)).detach().numpy())\n", " \n", " top10_items = [test_items[i] for i in np.argsort(predicted_labels)[::-1][0:10].tolist()]\n", " \n", " if i in top10_items:\n", " hits.append(1)\n", " else:\n", " hits.append(0)\n", " \n", "print(\"The Hit Ratio @ 10 is {:.2f}\".format(np.average(hits)))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "s1XtzBFsllfN" }, "source": [ "We got a pretty good Hit Ratio @ 10 score! To put this into context, what this means is that 86% of the users were recommended the actual item (among a list of 10 items) that they eventually interacted with. Not bad!" ] }, { "cell_type": "markdown", "metadata": { "id": "_xofdqRI29zl" }, "source": [ "## NMF with PyTorch on ML-1m" ] }, { "cell_type": "code", "metadata": { "id": "3wo7aehx3AyG" }, "source": [ "import os\n", "import time\n", "import random\n", "import argparse\n", "import numpy as np \n", "import pandas as pd \n", "import torch\n", "import torch.nn as nn\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "from torch.utils.tensorboard import SummaryWriter" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "y1iNipgl3JhO", "outputId": "363cf5f9-b062-4930-b122-d7573d824ab0" }, "source": [ "DATA_URL = \"https://raw.githubusercontent.com/sparsh-ai/rec-data-public/master/ml-1m-dat/ratings.dat\"\n", "MAIN_PATH = '/content/'\n", "DATA_PATH = MAIN_PATH + 'ratings.dat'\n", "MODEL_PATH = MAIN_PATH + 'models/'\n", "MODEL = 'ml-1m_Neu_MF'\n", "\n", "!wget -q --show-progress https://raw.githubusercontent.com/sparsh-ai/rec-data-public/master/ml-1m-dat/ratings.dat" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\rratings.dat 0%[ ] 0 --.-KB/s \rratings.dat 100%[===================>] 23.45M 128MB/s in 0.2s \n" ] } ] }, { "cell_type": "code", "metadata": { "id": "EXFnsMFy3YTE" }, "source": [ "def seed_everything(seed):\n", " random.seed(seed)\n", " os.environ['PYTHONHASHSEED'] = str(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed(seed)\n", " torch.backends.cudnn.deterministic = True\n", " torch.backends.cudnn.benchmark = True" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "KvTX81Z23bFs" }, "source": [ "### Dataset" ] }, { "cell_type": "code", "metadata": { "id": "NN5GjJCf3rI8" }, "source": [ "class Rating_Datset(torch.utils.data.Dataset):\n", "\tdef __init__(self, user_list, item_list, rating_list):\n", "\t\tsuper(Rating_Datset, self).__init__()\n", "\t\tself.user_list = user_list\n", "\t\tself.item_list = item_list\n", "\t\tself.rating_list = rating_list\n", "\n", "\tdef __len__(self):\n", "\t\treturn len(self.user_list)\n", "\n", "\tdef __getitem__(self, idx):\n", "\t\tuser = self.user_list[idx]\n", "\t\titem = self.item_list[idx]\n", "\t\trating = self.rating_list[idx]\n", "\t\t\n", "\t\treturn (\n", "\t\t\ttorch.tensor(user, dtype=torch.long),\n", "\t\t\ttorch.tensor(item, dtype=torch.long),\n", "\t\t\ttorch.tensor(rating, dtype=torch.float)\n", "\t\t\t)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "d4xgxyBsfoJM" }, "source": [ "- *_reindex*: process dataset to reindex userID and itemID, also set rating as binary feedback\n", "- *_leave_one_out*: leave-one-out evaluation protocol in paper https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf\n", "- *negative_sampling*: randomly selects n negative examples for each positive one" ] }, { "cell_type": "code", "metadata": { "id": "HggfgX_8Oqmq" }, "source": [ "class NCF_Data(object):\n", "\t\"\"\"\n", "\tConstruct Dataset for NCF\n", "\t\"\"\"\n", "\tdef __init__(self, args, ratings):\n", "\t\tself.ratings = ratings\n", "\t\tself.num_ng = args.num_ng\n", "\t\tself.num_ng_test = args.num_ng_test\n", "\t\tself.batch_size = args.batch_size\n", "\n", "\t\tself.preprocess_ratings = self._reindex(self.ratings)\n", "\n", "\t\tself.user_pool = set(self.ratings['user_id'].unique())\n", "\t\tself.item_pool = set(self.ratings['item_id'].unique())\n", "\n", "\t\tself.train_ratings, self.test_ratings = self._leave_one_out(self.preprocess_ratings)\n", "\t\tself.negatives = self._negative_sampling(self.preprocess_ratings)\n", "\t\trandom.seed(args.seed)\n", "\t\n", "\tdef _reindex(self, ratings):\n", "\t\t\"\"\"\n", "\t\tProcess dataset to reindex userID and itemID, also set rating as binary feedback\n", "\t\t\"\"\"\n", "\t\tuser_list = list(ratings['user_id'].drop_duplicates())\n", "\t\tuser2id = {w: i for i, w in enumerate(user_list)}\n", "\n", "\t\titem_list = list(ratings['item_id'].drop_duplicates())\n", "\t\titem2id = {w: i for i, w in enumerate(item_list)}\n", "\n", "\t\tratings['user_id'] = ratings['user_id'].apply(lambda x: user2id[x])\n", "\t\tratings['item_id'] = ratings['item_id'].apply(lambda x: item2id[x])\n", "\t\tratings['rating'] = ratings['rating'].apply(lambda x: float(x > 0))\n", "\t\treturn ratings\n", "\n", "\tdef _leave_one_out(self, ratings):\n", "\t\t\"\"\"\n", "\t\tleave-one-out evaluation protocol in paper https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf\n", "\t\t\"\"\"\n", "\t\tratings['rank_latest'] = ratings.groupby(['user_id'])['timestamp'].rank(method='first', ascending=False)\n", "\t\ttest = ratings.loc[ratings['rank_latest'] == 1]\n", "\t\ttrain = ratings.loc[ratings['rank_latest'] > 1]\n", "\t\tassert train['user_id'].nunique()==test['user_id'].nunique(), 'Not Match Train User with Test User'\n", "\t\treturn train[['user_id', 'item_id', 'rating']], test[['user_id', 'item_id', 'rating']]\n", "\n", "\tdef _negative_sampling(self, ratings):\n", "\t\tinteract_status = (\n", "\t\t\tratings.groupby('user_id')['item_id']\n", "\t\t\t.apply(set)\n", "\t\t\t.reset_index()\n", "\t\t\t.rename(columns={'item_id': 'interacted_items'}))\n", "\t\tinteract_status['negative_items'] = interact_status['interacted_items'].apply(lambda x: self.item_pool - x)\n", "\t\tinteract_status['negative_samples'] = interact_status['negative_items'].apply(lambda x: random.sample(x, self.num_ng_test))\n", "\t\treturn interact_status[['user_id', 'negative_items', 'negative_samples']]\n", "\n", "\tdef get_train_instance(self):\n", "\t\tusers, items, ratings = [], [], []\n", "\t\ttrain_ratings = pd.merge(self.train_ratings, self.negatives[['user_id', 'negative_items']], on='user_id')\n", "\t\ttrain_ratings['negatives'] = train_ratings['negative_items'].apply(lambda x: random.sample(x, self.num_ng))\n", "\t\tfor row in train_ratings.itertuples():\n", "\t\t\tusers.append(int(row.user_id))\n", "\t\t\titems.append(int(row.item_id))\n", "\t\t\tratings.append(float(row.rating))\n", "\t\t\tfor i in range(self.num_ng):\n", "\t\t\t\tusers.append(int(row.user_id))\n", "\t\t\t\titems.append(int(row.negatives[i]))\n", "\t\t\t\tratings.append(float(0)) # negative samples get 0 rating\n", "\t\tdataset = Rating_Datset(\n", "\t\t\tuser_list=users,\n", "\t\t\titem_list=items,\n", "\t\t\trating_list=ratings)\n", "\t\treturn torch.utils.data.DataLoader(dataset, batch_size=self.batch_size, shuffle=True, num_workers=2)\n", "\n", "\tdef get_test_instance(self):\n", "\t\tusers, items, ratings = [], [], []\n", "\t\ttest_ratings = pd.merge(self.test_ratings, self.negatives[['user_id', 'negative_samples']], on='user_id')\n", "\t\tfor row in test_ratings.itertuples():\n", "\t\t\tusers.append(int(row.user_id))\n", "\t\t\titems.append(int(row.item_id))\n", "\t\t\tratings.append(float(row.rating))\n", "\t\t\tfor i in getattr(row, 'negative_samples'):\n", "\t\t\t\tusers.append(int(row.user_id))\n", "\t\t\t\titems.append(int(i))\n", "\t\t\t\tratings.append(float(0))\n", "\t\tdataset = Rating_Datset(\n", "\t\t\tuser_list=users,\n", "\t\t\titem_list=items,\n", "\t\t\trating_list=ratings)\n", "\t\treturn torch.utils.data.DataLoader(dataset, batch_size=self.num_ng_test+1, shuffle=False, num_workers=2)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "hfXyLsgVOsBs" }, "source": [ "### Metrics\n", "Using Hit Rate and NDCG as our evaluation metrics" ] }, { "cell_type": "code", "metadata": { "id": "KM4B7r12OvnS" }, "source": [ "def hit(ng_item, pred_items):\n", "\tif ng_item in pred_items:\n", "\t\treturn 1\n", "\treturn 0\n", "\n", "\n", "def ndcg(ng_item, pred_items):\n", "\tif ng_item in pred_items:\n", "\t\tindex = pred_items.index(ng_item)\n", "\t\treturn np.reciprocal(np.log2(index+2))\n", "\treturn 0\n", "\n", "\n", "def metrics(model, test_loader, top_k, device):\n", "\tHR, NDCG = [], []\n", "\n", "\tfor user, item, label in test_loader:\n", "\t\tuser = user.to(device)\n", "\t\titem = item.to(device)\n", "\n", "\t\tpredictions = model(user, item)\n", "\t\t_, indices = torch.topk(predictions, top_k)\n", "\t\trecommends = torch.take(\n", "\t\t\t\titem, indices).cpu().numpy().tolist()\n", "\n", "\t\tng_item = item[0].item() # leave one-out evaluation has only one item per user\n", "\t\tHR.append(hit(ng_item, recommends))\n", "\t\tNDCG.append(ndcg(ng_item, recommends))\n", "\n", "\treturn np.mean(HR), np.mean(NDCG)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "HWyCD7pLOxjq" }, "source": [ "### Models\n", "- Generalized Matrix Factorization\n", "- Multi Layer Perceptron\n", "- Neural Matrix Factorization" ] }, { "cell_type": "code", "metadata": { "id": "aTQaitu7d1R3" }, "source": [ "class Generalized_Matrix_Factorization(nn.Module):\n", " def __init__(self, args, num_users, num_items):\n", " super(Generalized_Matrix_Factorization, self).__init__()\n", " self.num_users = num_users\n", " self.num_items = num_items\n", " self.factor_num = args.factor_num\n", "\n", " self.embedding_user = nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.factor_num)\n", " self.embedding_item = nn.Embedding(num_embeddings=self.num_items, embedding_dim=self.factor_num)\n", "\n", " self.affine_output = nn.Linear(in_features=self.factor_num, out_features=1)\n", " self.logistic = nn.Sigmoid()\n", "\n", " def forward(self, user_indices, item_indices):\n", " user_embedding = self.embedding_user(user_indices)\n", " item_embedding = self.embedding_item(item_indices)\n", " element_product = torch.mul(user_embedding, item_embedding)\n", " logits = self.affine_output(element_product)\n", " rating = self.logistic(logits)\n", " return rating\n", "\n", " def init_weight(self):\n", " pass" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7kSFzPlNd50f" }, "source": [ "class Multi_Layer_Perceptron(nn.Module):\n", " def __init__(self, args, num_users, num_items):\n", " super(Multi_Layer_Perceptron, self).__init__()\n", " self.num_users = num_users\n", " self.num_items = num_items\n", " self.factor_num = args.factor_num\n", " self.layers = args.layers\n", "\n", " self.embedding_user = nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.factor_num)\n", " self.embedding_item = nn.Embedding(num_embeddings=self.num_items, embedding_dim=self.factor_num)\n", "\n", " self.fc_layers = nn.ModuleList()\n", " for idx, (in_size, out_size) in enumerate(zip(self.layers[:-1], self.layers[1:])):\n", " self.fc_layers.append(nn.Linear(in_size, out_size))\n", "\n", " self.affine_output = nn.Linear(in_features=self.layers[-1], out_features=1)\n", " self.logistic = nn.Sigmoid()\n", "\n", " def forward(self, user_indices, item_indices):\n", " user_embedding = self.embedding_user(user_indices)\n", " item_embedding = self.embedding_item(item_indices)\n", " vector = torch.cat([user_embedding, item_embedding], dim=-1) # the concat latent vector\n", " for idx, _ in enumerate(range(len(self.fc_layers))):\n", " vector = self.fc_layers[idx](vector)\n", " vector = nn.ReLU()(vector)\n", " # vector = nn.BatchNorm1d()(vector)\n", " # vector = nn.Dropout(p=0.5)(vector)\n", " logits = self.affine_output(vector)\n", " rating = self.logistic(logits)\n", " return rating\n", "\n", " def init_weight(self):\n", " pass" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7DQpVuaV9cF0" }, "source": [ "class NeuMF(nn.Module):\n", " def __init__(self, args, num_users, num_items):\n", " super(NeuMF, self).__init__()\n", " self.num_users = num_users\n", " self.num_items = num_items\n", " self.factor_num_mf = args.factor_num\n", " self.factor_num_mlp = int(args.layers[0]/2)\n", " self.layers = args.layers\n", " self.dropout = args.dropout\n", "\n", " self.embedding_user_mlp = nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.factor_num_mlp)\n", " self.embedding_item_mlp = nn.Embedding(num_embeddings=self.num_items, embedding_dim=self.factor_num_mlp)\n", "\n", " self.embedding_user_mf = nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.factor_num_mf)\n", " self.embedding_item_mf = nn.Embedding(num_embeddings=self.num_items, embedding_dim=self.factor_num_mf)\n", "\n", " self.fc_layers = nn.ModuleList()\n", " for idx, (in_size, out_size) in enumerate(zip(args.layers[:-1], args.layers[1:])):\n", " self.fc_layers.append(torch.nn.Linear(in_size, out_size))\n", " self.fc_layers.append(nn.ReLU())\n", "\n", " self.affine_output = nn.Linear(in_features=args.layers[-1] + self.factor_num_mf, out_features=1)\n", " self.logistic = nn.Sigmoid()\n", " self.init_weight()\n", "\n", " def init_weight(self):\n", " nn.init.normal_(self.embedding_user_mlp.weight, std=0.01)\n", " nn.init.normal_(self.embedding_item_mlp.weight, std=0.01)\n", " nn.init.normal_(self.embedding_user_mf.weight, std=0.01)\n", " nn.init.normal_(self.embedding_item_mf.weight, std=0.01)\n", " \n", " for m in self.fc_layers:\n", " if isinstance(m, nn.Linear):\n", " nn.init.xavier_uniform_(m.weight)\n", " \n", " nn.init.xavier_uniform_(self.affine_output.weight)\n", "\n", " for m in self.modules():\n", " if isinstance(m, nn.Linear) and m.bias is not None:\n", " m.bias.data.zero_()\n", "\n", " def forward(self, user_indices, item_indices):\n", " user_embedding_mlp = self.embedding_user_mlp(user_indices)\n", " item_embedding_mlp = self.embedding_item_mlp(item_indices)\n", "\n", " user_embedding_mf = self.embedding_user_mf(user_indices)\n", " item_embedding_mf = self.embedding_item_mf(item_indices)\n", "\n", " mlp_vector = torch.cat([user_embedding_mlp, item_embedding_mlp], dim=-1)\n", " mf_vector =torch.mul(user_embedding_mf, item_embedding_mf)\n", "\n", " for idx, _ in enumerate(range(len(self.fc_layers))):\n", " mlp_vector = self.fc_layers[idx](mlp_vector)\n", "\n", " vector = torch.cat([mlp_vector, mf_vector], dim=-1)\n", " logits = self.affine_output(vector)\n", " rating = self.logistic(logits)\n", " return rating.squeeze()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "tpBX6rqNfSc9" }, "source": [ "### Setting Arguments\n", "\n", "Here is the brief description of important ones:\n", "- Learning rate is 0.001\n", "- Dropout rate is 0.2\n", "- Running for 10 epochs\n", "- HitRate@10 and NDCG@10\n", "- 4 negative samples for each positive one" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Bc5Vg1Ik_gnF", "outputId": "072e970d-c6d2-413c-d6f4-2f25e13ee4bf" }, "source": [ "parser = argparse.ArgumentParser()\n", "parser.add_argument(\"--seed\", \n", "\ttype=int, \n", "\tdefault=42, \n", "\thelp=\"Seed\")\n", "parser.add_argument(\"--lr\", \n", "\ttype=float, \n", "\tdefault=0.001, \n", "\thelp=\"learning rate\")\n", "parser.add_argument(\"--dropout\", \n", "\ttype=float,\n", "\tdefault=0.2, \n", "\thelp=\"dropout rate\")\n", "parser.add_argument(\"--batch_size\", \n", "\ttype=int, \n", "\tdefault=256, \n", "\thelp=\"batch size for training\")\n", "parser.add_argument(\"--epochs\", \n", "\ttype=int,\n", "\tdefault=10, \n", "\thelp=\"training epoches\")\n", "parser.add_argument(\"--top_k\", \n", "\ttype=int, \n", "\tdefault=10, \n", "\thelp=\"compute metrics@top_k\")\n", "parser.add_argument(\"--factor_num\", \n", "\ttype=int,\n", "\tdefault=32, \n", "\thelp=\"predictive factors numbers in the model\")\n", "parser.add_argument(\"--layers\",\n", " nargs='+', \n", " default=[64,32,16,8],\n", " help=\"MLP layers. Note that the first layer is the concatenation of user \\\n", " and item embeddings. So layers[0]/2 is the embedding size.\")\n", "parser.add_argument(\"--num_ng\", \n", "\ttype=int,\n", "\tdefault=4, \n", "\thelp=\"Number of negative samples for training set\")\n", "parser.add_argument(\"--num_ng_test\", \n", "\ttype=int,\n", "\tdefault=100, \n", "\thelp=\"Number of negative samples for test set\")\n", "parser.add_argument(\"--out\", \n", "\tdefault=True,\n", "\thelp=\"save model or not\")" ], "execution_count": null, "outputs": [ { "data": { "text/plain": [ "_StoreAction(option_strings=['--out'], dest='out', nargs=None, const=None, default=True, type=None, choices=None, help='save model or not', metavar=None)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ] }, { "cell_type": "markdown", "metadata": { "id": "RnaRWy2gg_Nw" }, "source": [ "### Training" ] }, { "cell_type": "code", "metadata": { "colab": { "background_save": true, "base_uri": "https://localhost:8080/" }, "id": "VyWquJG893CV", "outputId": "61938a61-f7f5-4885-85d1-1e3e2d2a06f6" }, "source": [ "# set device and parameters\n", "args = parser.parse_args(args={})\n", "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", "writer = SummaryWriter()\n", "\n", "# seed for Reproducibility\n", "seed_everything(args.seed)\n", "\n", "# load data\n", "ml_1m = pd.read_csv(\n", "\tDATA_PATH, \n", "\tsep=\"::\", \n", "\tnames = ['user_id', 'item_id', 'rating', 'timestamp'], \n", "\tengine='python')\n", "\n", "# set the num_users, items\n", "num_users = ml_1m['user_id'].nunique()+1\n", "num_items = ml_1m['item_id'].nunique()+1\n", "\n", "# construct the train and test datasets\n", "data = NCF_Data(args, ml_1m)\n", "train_loader = data.get_train_instance()\n", "test_loader = data.get_test_instance()\n", "\n", "# set model and loss, optimizer\n", "model = NeuMF(args, num_users, num_items)\n", "model = model.to(device)\n", "loss_function = nn.BCELoss()\n", "optimizer = optim.Adam(model.parameters(), lr=args.lr)\n", "\n", "# train, evaluation\n", "best_hr = 0\n", "for epoch in range(1, args.epochs+1):\n", "\tmodel.train() # Enable dropout (if have).\n", "\tstart_time = time.time()\n", "\n", "\tfor user, item, label in train_loader:\n", "\t\tuser = user.to(device)\n", "\t\titem = item.to(device)\n", "\t\tlabel = label.to(device)\n", "\n", "\t\toptimizer.zero_grad()\n", "\t\tprediction = model(user, item)\n", "\t\tloss = loss_function(prediction, label)\n", "\t\tloss.backward()\n", "\t\toptimizer.step()\n", "\t\twriter.add_scalar('loss/Train_loss', loss.item(), epoch)\n", "\n", "\tmodel.eval()\n", "\tHR, NDCG = metrics(model, test_loader, args.top_k, device)\n", "\twriter.add_scalar('Perfomance/HR@10', HR, epoch)\n", "\twriter.add_scalar('Perfomance/NDCG@10', NDCG, epoch)\n", "\n", "\telapsed_time = time.time() - start_time\n", "\tprint(\"The time elapse of epoch {:03d}\".format(epoch) + \" is: \" + \n", "\t\t\ttime.strftime(\"%H: %M: %S\", time.gmtime(elapsed_time)))\n", "\tprint(\"HR: {:.3f}\\tNDCG: {:.3f}\".format(np.mean(HR), np.mean(NDCG)))\n", "\n", "\tif HR > best_hr:\n", "\t\tbest_hr, best_ndcg, best_epoch = HR, NDCG, epoch\n", "\t\tif args.out:\n", "\t\t\tif not os.path.exists(MODEL_PATH):\n", "\t\t\t\tos.mkdir(MODEL_PATH)\n", "\t\t\ttorch.save(model, \n", "\t\t\t\t'{}{}.pth'.format(MODEL_PATH, MODEL))\n", "\n", "writer.close()" ], "execution_count": null, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "The time elapse of epoch 001 is: 00: 05: 41\n", "HR: 0.626\tNDCG: 0.359\n", "The time elapse of epoch 002 is: 00: 05: 42\n", "HR: 0.658\tNDCG: 0.389\n", "The time elapse of epoch 003 is: 00: 05: 47\n", "HR: 0.664\tNDCG: 0.396\n", "The time elapse of epoch 004 is: 00: 05: 34\n", "HR: 0.669\tNDCG: 0.400\n", "The time elapse of epoch 005 is: 00: 05: 44\n", "HR: 0.671\tNDCG: 0.401\n", "The time elapse of epoch 006 is: 00: 05: 44\n", "HR: 0.672\tNDCG: 0.402\n", "The time elapse of epoch 007 is: 00: 05: 39\n", "HR: 0.668\tNDCG: 0.396\n", "The time elapse of epoch 008 is: 00: 05: 34\n", "HR: 0.667\tNDCG: 0.396\n", "The time elapse of epoch 009 is: 00: 05: 41\n", "HR: 0.668\tNDCG: 0.397\n", "The time elapse of epoch 010 is: 00: 05: 37\n", "HR: 0.664\tNDCG: 0.395\n" ] } ] }, { "cell_type": "code", "metadata": { "colab": { "background_save": true, "base_uri": "https://localhost:8080/" }, "id": "fkiRJWeD_trR", "outputId": "d3efcab5-fa0b-4938-d5ff-7967f38dab4d" }, "source": [ "print(\"Best epoch {:03d}: HR = {:.3f}, NDCG = {:.3f}\".format(\n", "\t\t\t\t\t\t\t\t\tbest_epoch, best_hr, best_ndcg))" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best epoch 006: HR = 0.672, NDCG = 0.402\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "WOTaSMGnPoAG" }, "source": [ "## MF with PyTorch on ML-100k\n", "\n", "Training Pytorch MLP model on movielens-100k dataset and visualizing factors by decomposing using PCA" ] }, { "cell_type": "code", "metadata": { "id": "I2f_R0Yo6BUp" }, "source": [ "!pip install -U -q git+https://github.com/sparsh-ai/recochef.git" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "KXT07lHDBzAQ" }, "source": [ "import torch\n", "from torch import nn\n", "from torch import optim\n", "from torch.nn import functional as F \n", "from torch.optim.lr_scheduler import _LRScheduler\n", "\n", "from recochef.datasets.movielens import MovieLens\n", "from recochef.preprocessing.encode import label_encode\n", "from recochef.utils.iterators import batch_generator\n", "from recochef.models.embedding import EmbeddingNet\n", "\n", "import math\n", "import copy\n", "import pickle\n", "import numpy as np\n", "import pandas as pd\n", "from textwrap import wrap\n", "from sklearn.decomposition import PCA\n", "from sklearn.model_selection import train_test_split\n", "\n", "import matplotlib.pyplot as plt\n", "plt.style.use('ggplot')" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "NINhOhYAxt5n" }, "source": [ "### Data loading and preprocessing" ] }, { "cell_type": "code", "metadata": { "id": "3Z4R3bXNjaNP" }, "source": [ "data = MovieLens()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "A2Xgw-sXk7Ac", "outputId": "399404ea-51bd-476f-f74e-0dc4807ebaa9" }, "source": [ "ratings_df = data.load_interactions()\n", "ratings_df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
01962423.0881250949
11863023.0891717742
2223771.0878887116
3244512.0880606923
41663461.0886397596
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 196 242 3.0 881250949\n", "1 186 302 3.0 891717742\n", "2 22 377 1.0 878887116\n", "3 244 51 2.0 880606923\n", "4 166 346 1.0 886397596" ] }, "metadata": {}, "execution_count": 16 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 343 }, "id": "wIUc-Ba_6xBK", "outputId": "3f1382b8-d132-48b8-f85f-fae21140922d" }, "source": [ "movies_df = data.load_items()\n", "movies_df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ITEMIDTITLERELEASEVIDRELEASEURLUNKNOWNACTIONADVENTUREANIMATIONCHILDRENCOMEDYCRIMEDOCUMENTARYDRAMAFANTASYFILMNOIRHORRORMUSICALMYSTERYROMANCESCIFITHRILLERWARWESTERN
01Toy Story (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Toy%20Story%2...0001110000000000000
12GoldenEye (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?GoldenEye%20(...0110000000000000100
23Four Rooms (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Four%20Rooms%...0000000000000000100
34Get Shorty (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Get%20Shorty%...0100010010000000000
45Copycat (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Copycat%20(1995)0000001010000000100
\n", "
" ], "text/plain": [ " ITEMID TITLE RELEASE ... THRILLER WAR WESTERN\n", "0 1 Toy Story (1995) 01-Jan-1995 ... 0 0 0\n", "1 2 GoldenEye (1995) 01-Jan-1995 ... 1 0 0\n", "2 3 Four Rooms (1995) 01-Jan-1995 ... 1 0 0\n", "3 4 Get Shorty (1995) 01-Jan-1995 ... 0 0 0\n", "4 5 Copycat (1995) 01-Jan-1995 ... 1 0 0\n", "\n", "[5 rows x 24 columns]" ] }, "metadata": {}, "execution_count": 17 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "X-IcaBzgmOrN", "outputId": "7a6afd50-b93a-498b-f9ea-96d7db363795" }, "source": [ "ratings_df, umap = label_encode(ratings_df, 'USERID')\n", "ratings_df, imap = label_encode(ratings_df, 'ITEMID')\n", "ratings_df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
0003.0881250949
1113.0891717742
2221.0878887116
3332.0880606923
4441.0886397596
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 0 0 3.0 881250949\n", "1 1 1 3.0 891717742\n", "2 2 2 1.0 878887116\n", "3 3 3 2.0 880606923\n", "4 4 4 1.0 886397596" ] }, "metadata": {}, "execution_count": 50 } ] }, { "cell_type": "code", "metadata": { "id": "36dsiSqWwNRz" }, "source": [ "X = ratings_df[['USERID','ITEMID']]\n", "y = ratings_df[['RATING']]" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Mp-OqA2jyNAS", "outputId": "969eca89-0f0e-464d-d98e-70aeb215b99b" }, "source": [ "for _x_batch, _y_batch in batch_generator(X, y, bs=4):\n", " print(_x_batch)\n", " print(_y_batch)\n", " break" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "tensor([[873, 377],\n", " [808, 601],\n", " [ 90, 354],\n", " [409, 570]])\n", "tensor([[4.],\n", " [3.],\n", " [4.],\n", " [2.]])\n" ] } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "oQlnTmST0cx6", "outputId": "96db7cc7-9614-4215-f0e8-be69eb17e4e0" }, "source": [ "_x_batch[:, 1]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "tensor([377, 601, 354, 570])" ] }, "metadata": {}, "execution_count": 24 } ] }, { "cell_type": "markdown", "metadata": { "id": "mVavggU2_WUY" }, "source": [ "### Embedding Net" ] }, { "cell_type": "markdown", "metadata": { "id": "D39WDxT5_f3l" }, "source": [ "The PyTorch is a framework that allows to build various computational graphs (not only neural networks) and run them on GPU. The conception of tensors, neural networks, and computational graphs is outside the scope of this article but briefly speaking, one could treat the library as a set of tools to create highly computationally efficient and flexible machine learning models. In our case, we want to create a neural network that could help us to infer the similarities between users and predict their ratings based on available data." ] }, { "cell_type": "markdown", "metadata": { "id": "9Gve8w7f_l8i" }, "source": [ "The picture above schematically shows the model we're going to build. At the very beginning, we put our embeddings matrices, or look-ups, which convert integer IDs into arrays of floating-point numbers. Next, we put a bunch of fully-connected layers with dropouts. Finally, we need to return a list of predicted ratings. For this purpose, we use a layer with sigmoid activation function and rescale it to the original range of values (in case of MovieLens dataset, it is usually from 1 to 5)." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9zDzhH2c0Cv-", "outputId": "134739d6-e9ff-4cd7-a0b5-0aac999500fa" }, "source": [ "netx = EmbeddingNet(\n", " n_users=50, n_items=20, \n", " n_factors=10, hidden=[500], \n", " embedding_dropout=0.05, dropouts=[0.5])\n", "netx" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "EmbeddingNet(\n", " (u): Embedding(50, 10)\n", " (m): Embedding(20, 10)\n", " (drop): Dropout(p=0.05, inplace=False)\n", " (hidden): Sequential(\n", " (0): Linear(in_features=20, out_features=500, bias=True)\n", " (1): ReLU()\n", " (2): Dropout(p=0.5, inplace=False)\n", " )\n", " (fc): Linear(in_features=500, out_features=1, bias=True)\n", ")" ] }, "metadata": {}, "execution_count": 25 } ] }, { "cell_type": "markdown", "metadata": { "id": "o4vQFZ6iwiyM" }, "source": [ "### Cyclical Learning Rate (CLR)" ] }, { "cell_type": "markdown", "metadata": { "id": "6RIaav5rwk66" }, "source": [ "One of the `fastai` library features is the cyclical learning rate scheduler. We can implement something similar inheriting the `_LRScheduler` class from the `torch` library. Following the [original paper's](https://arxiv.org/abs/1506.01186) pseudocode, this [CLR Keras callback implementation](https://github.com/bckenstler/CLR), and making a couple of adjustments to support [cosine annealing](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.CosineAnnealingLR) with restarts, let's create our own CLR scheduler.\n", "\n", "The implementation of this idea is quite simple. The [base PyTorch scheduler class](https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html) has the `get_lr()` method that is invoked each time when we call the `step()` method. The method should return a list of learning rates depending on the current training epoch. In our case, we have the same learning rate for all of the layers, and therefore, we return a list with a single value. \n", "\n", "The next cell defines a `CyclicLR` class that expectes a single callback function. This function should accept the current training epoch and the base value of learning rate, and return a new learning rate value." ] }, { "cell_type": "code", "metadata": { "id": "eYQh4ZCmmgW9" }, "source": [ "class CyclicLR(_LRScheduler):\n", " \n", " def __init__(self, optimizer, schedule, last_epoch=-1):\n", " assert callable(schedule)\n", " self.schedule = schedule\n", " super().__init__(optimizer, last_epoch)\n", "\n", " def get_lr(self):\n", " return [self.schedule(self.last_epoch, lr) for lr in self.base_lrs]" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "1bpK5hOvw7Hg" }, "source": [ "Our scheduler is very similar to [LambdaLR](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR) one but expects a bit different callback signature. \n", "\n", "So now we only need to define appropriate scheduling functions. We're createing a couple of functions that accept scheduling parameters and return a _new function_ with the appropriate signature:" ] }, { "cell_type": "code", "metadata": { "id": "I6st2zPctj1T" }, "source": [ "def triangular(step_size, max_lr, method='triangular', gamma=0.99):\n", " \n", " def scheduler(epoch, base_lr):\n", " period = 2 * step_size\n", " cycle = math.floor(1 + epoch/period)\n", " x = abs(epoch/step_size - 2*cycle + 1)\n", " delta = (max_lr - base_lr)*max(0, (1 - x))\n", "\n", " if method == 'triangular':\n", " pass # we've already done\n", " elif method == 'triangular2':\n", " delta /= float(2 ** (cycle - 1))\n", " elif method == 'exp_range':\n", " delta *= (gamma**epoch)\n", " else:\n", " raise ValueError('unexpected method: %s' % method)\n", " \n", " return base_lr + delta\n", " \n", " return scheduler" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "k-CjYin0toWa" }, "source": [ "def cosine(t_max, eta_min=0):\n", " \n", " def scheduler(epoch, base_lr):\n", " t = epoch % t_max\n", " return eta_min + (base_lr - eta_min)*(1 + math.cos(math.pi*t/t_max))/2\n", " \n", " return scheduler" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "oB_zTLW-wwdM" }, "source": [ "To understand how the created functions work, and to check the correctness of our implementation, let's create a couple of plots visualizing learning rates changes depending on the number of epoch:" ] }, { "cell_type": "code", "metadata": { "id": "Dl-TWx4OwwdN" }, "source": [ "def plot_lr(schedule):\n", " ts = list(range(1000))\n", " y = [schedule(t, 0.001) for t in ts]\n", " plt.plot(ts, y)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "id": "wCfhKAoMwwdN", "outputId": "52bd67ed-e05a-4a07-9747-c8d68b1eae5a" }, "source": [ "plot_lr(triangular(250, 0.005))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzda2BU13no/f/aM+KiC4KRkGSwwEbC2AiMjAbQzUaXUZIGtyU9rtvYcRqgdfvGIUfmvDlxoW/cNuWUE2JwjEid06qkqXlTUjeQxrm4GoTA0iCQABkLjI0MGMsICzRCFyQkzex1PmwskCXQbUZ7Luv3xRaz9t7Pmj0zz8zaaz9LSCkliqIoinKTZnYAiqIoSmBRiUFRFEUZQCUGRVEUZQCVGBRFUZQBVGJQFEVRBlCJQVEURRnAanYAvnLp0qUxbRcfH8/Vq1d9HE1gU30OD6rP4WE8fZ41a9aQ/65+MSiKoigDqMSgKIqiDKASg6IoijKASgyKoijKACoxKIqiKAOMaFZSXV0du3btQtd1CgsLWb169YDH+/r6KCkp4dy5c8TExFBcXExCQgIAe/fupby8HE3TWLNmDenp6QA899xzTJkyBU3TsFgsbNmyBYDOzk62b9/OlStXmDlzJs8//zzR0dG+7LOiKIpyF8P+YtB1ndLSUjZu3Mj27dupqqqisbFxQJvy8nKioqLYsWMHq1atYvfu3QA0NjbicrnYtm0bmzZtorS0FF3X+7d78cUX2bp1a39SANi3bx+LFy/mlVdeYfHixezbt89XfVUURVFGYNjE0NDQQFJSEomJiVitVrKzs6mpqRnQpra2lry8PAAyMzOpr69HSklNTQ3Z2dlERESQkJBAUlISDQ0Ndz1eTU0NK1euBGDlypWDjqWYR3a0oVftR1VqV/xBSmm8vjrazA4l7A07lOR2u4mLi+v/Oy4ujrNnz96xjcViITIyko6ODtxuN/Pnz+9vZ7PZcLvd/X9v3rwZgKKiIhwOBwBtbW3MmDEDgOnTp9PWNvSLxOl04nQ6AdiyZQvx8fHD93YIVqt1zNsGq7H2uX3fT+j+1etMn7+ASQvT/RCZ/6jzHPh6T9fR+uMfMOWLTzDtzzaMaR/B1mdf8EefTbvz+bvf/S42m422tjb+7u/+jlmzZrFw4cIBbYQQCCGG3N7hcPQnE2DMd/6pOyVHRvb1oh/4LQDXfvUfaAn3+iM0v1HnOfDpv/oPALorfkvP43+MiJg06n0EW599wZQ7n202Gy0tLf1/t7S0YLPZ7tjG6/XS1dVFTEzMoG3dbnf/tp/+NzY2lmXLlvUPMcXGxtLa2gpAa2sr06ZNG3EnFf+RdUegqxNmzUHWViK7u8wOSQkhsrsLWVsJs+ZAVyfyRLXZIYW1YRNDSkoKTU1NNDc34/F4cLlc2O32AW0yMjKoqKgAoLq6mrS0NIQQ2O12XC4XfX19NDc309TURGpqKjdu3KC7uxuAGzducPLkSebMmQOA3W7n4MGDABw8eJBly5b5sr/KGMnKMrDNRPvqN6C3B1nzltkhKSFE1rwFvT3G68s203i9KaYZdijJYrGwdu1aNm/ejK7r5Ofnk5yczJ49e0hJScFut1NQUEBJSQnr168nOjqa4uJiAJKTk8nKymLDhg1omsa6devQNI22tja+//3vA8YvjNzc3P5prKtXr2b79u2Ul5f3T1dVzCVbmuHdtxGP/xHMWwD3JCOrnPDY580OTQkRssoJ9yTDvAWInELkG3uQVz9BxCeaHVpYEjJEppio6qojN9o+6//5U+Qb/4b29/+IiEtA/699yH//Z7S/KUHMmuPHSH1HnefAJS9dRH/xG4g/XIv2udXIlmb0v/wzxON/hPZ7T41qX8HSZ19S1VWVCSd1HenaDw8tQcQZNy2KrHywWNTPfcUnZGUZWCzG6wqM19lDS5BV+5G61+TowpNKDMrdnTkJLc2InFszwERMLCxZjqyuQHr6TAxOCXbS04esroAly43X1U0ixwHuK8brT5lwKjEodyUryyAyGvFI5oB/13KLoKMNTqobEJVxOFkDHW3G6+k24pFMiIxGVjpNCiy8qcSg3JG83oE8UY3IzBs8pzztEZgeh67euMo46JVOmB5nvJ5uIyImITLzkCcOIzvbTYoufKnEoNyRPHIQPH0DhpE+JTQLIrsA6o8jW1uG2FpR7k62tkD9cUR2AUKzDHpc5DjA40EeOWRCdOFNJQbljmRlGcxJQcyZN+TjIscB8ubFaUUZJenaD1If8osHYLzu5qQgK8tUfa4JphKDMiT54Qfw0XnEZ8Z+bycS7oEFi5FVTuRtVXMVZThSSuPehQWLjdfRHYjcImg8DxfPTWB0ikoMypBkVRlYIxDLH7trO5HjgCuX4ezpCYpMCQnvn4Irl+/4a+FTYvljYI1QU6MnmEoMyiCytwd55CBiaTYi6u6LJIml2TA1Ur1xlVGRlWUwNdJ4/dyFiIpGLM1GHjmI7O2ZoOgUlRiUQeSJaui6jsi9+7c5ADF5MmLZY8jjVciu6xMQnRLsZNd15PEqxLLHEJMnD9te5Dqg+7oqrDeBVGJQBpFVTohPhAWLR9Re5BZBb68qrKeMiFEwr/eu168GWLAY4hPVr9IJpBKDMoC8ctkomJdTiNBG+PK4LxVmz1VvXGVEZGUZzJ5rvG5GQGgaIqcQzpw0Xp+K36nEoAwgXeUgBCKrcMTbCCGMn/sXziIbL/gvOCXoycYLcOEsItdxx0W4hiKyCkEINTV6gqjEoPSTuhfpcsLCdETczFFtK1bkg8VqDEMpyh3IKidYrMbrZRRE3ExYmK4K600QlRiUW06/De6rg+rWjISImYZIX4GsPoDsU4X1lMFkXx+y+gAifQUiZvQrM2q5RdB6FU7X+SE65XYqMSj9ZJUTomNgyYoxbS9yHdDZASeP+jgyJSScPAqdHSOa7TakJSsgOkYV1psAw67gBlBXV8euXbvQdZ3CwkJWr1494PG+vj5KSko4d+4cMTExFBcXk5Bg1O7fu3cv5eXlaJrGmjVr+ldqA9B1nRdeeAGbzcYLL7wAwM6dOzl9+jSRkZEAPPfcc9x3332+6KtyF7KzHVlXjVj5O4iIiLHtZGE62OLRK8uwZOT4NkAl6OmVZWCLN14nYyAiIhAr8pAVv0F2tI/pV4cyMsP+YtB1ndLSUjZu3Mj27dupqqqisbFxQJvy8nKioqLYsWMHq1atYvfu3QA0NjbicrnYtm0bmzZtorS0FP220gm//vWvmT179qBjPvPMM2zdupWtW7eqpDBBZHUFeDwjn0I4BKOwXiGcOoF0X/FdcErQk+4rcOoEIrtwyIJ5IyVyi8DrQR6p8F1wyiDDJoaGhgaSkpJITEzEarWSnZ1NTc3AGvy1tbXk5eUBkJmZSX19PVJKampqyM7OJiIigoSEBJKSkmhoaACgpaWF48ePU1g48tkvin9IKY0phHNTEffeN659iexCkNKY3aQoN0lXOUhpvD7GQdx7H8xNVYX1/GzYoSS3201cXFz/33FxcZw9e/aObSwWC5GRkXR0dOB2u5k/f35/O5vNhtvtBuDHP/4xX/nKV+ju7h50zJ/+9Ke8/vrrLFq0iKeffpqIIYY2nE4nTqcx1rhlyxbi4+NH0t9BrFbrmLcNVp/tc1/Du7g//pCYP/8WkeN9LuLjaV2cgbf6AHFf/X9Gfi+En6nzbB6p67RUH8CyOIMZDy0a9/66vvAlOn60lenXrhAxf+GAxwKlzxPJH30e0TUGXzt27BixsbHMmzePU6dODXjsqaeeYvr06Xg8Hn70ox/xi1/8gieeeGLQPhwOBw7HrYtYY10MWy0eDvob/w4Rk7i+cCldPngu9OUrkaXbuFp1APHQknHvzxfUeTaPPHMS/ZNL6I//sU/ikQuXQsQkWn/1OtpXvj7gsUDp80QaT59nzZo15L8P+3XOZrPR0nJrIZaWlhZsNtsd23i9Xrq6uoiJiRm0rdvtxmaz8d5771FbW8tzzz3Hyy+/TH19Pa+88goAM2bMQAhBREQE+fn5/UNPin/Inh7k0UOIjGxEZJRP9imWZsHUKHVPgwJ8WjAvynhd+ICIjEJkZCOPHkL2qMJ6/jBsYkhJSaGpqYnm5mY8Hg8ulwu73T6gTUZGBhUVFQBUV1eTlpaGEAK73Y7L5aKvr4/m5maamppITU3lqaee4tVXX2Xnzp0UFxezaNEivvnNbwLQ2toK0H+NIjk52cddVm4nT7igu2tcF50/S0yajFixEnn8MLKr02f7VYKP7OpEHj+MWLESMWn4gnkjJXKLoLsLedzls30qtww7lGSxWFi7di2bN29G13Xy8/NJTk5mz549pKSkYLfbKSgooKSkhPXr1xMdHU1xcTEAycnJZGVlsWHDBjRNY926dWjDjDm/8sortLcba7zOnTuXZ5991gfdVO5EVjphZhLMT/PpfkWuA1nxa+PXSN4XfbpvJXjIo4egr3fs9y7cyQOLYGaS8as0a3R3USvDEzJELu1funRpTNuF85ikbG5C3/TniNVfQVv1pE+PIaVE/9tisFiw/NU2n+57LML5PJvJ+3cbwOtF+87Lo6qNNBL6r36G3Pca2uYf9a8CFwh9nmimXGNQQpes2g9CQ2QV+HzfRmG9IviwAfnReZ/vXwl88qPz8GEDIrfI50kBMF63QlPXsvxAJYYwZRTM2w9pjyBs/pneJ1Y8BlZVWC9cySonWK3G68APhC0e0h5BulRhPV9TiSFcnaqDay1jKpg3UiJ6GuKRLGR1hSqsF2aMgnkViEeyENH+K12h5RbBNTecOuG3Y4QjlRjClF5ZBtHTYMkyvx5H5DrgegeyTi3LGE5k3RG4Po6CeSO1ZBlETzNez4rPqMQQhvS2Vnj7KCIzH2EdY8G8kXpwCdhmqoqYYUZWloFtpnH+/UhYIxCZ+fD2UWRHm1+PFU5UYghD3QffBO/4CuaNVP+yjO/WIVua/X48xXyypRnerRvd8rDjYBTW8yIPH/D7scKFSgxhRkpJt/OXcP8DiNlzJuSYnxZOk1VqWcZw8GkBxfEWzBspMXsO3P+AKqznQyoxhJsLZ/F+dN7/Y7+3EfGJ8ODDN2eP6MNvoAQtqevGbKQHHzbO+wQRuQ5o+gjP2dMTdsxQphJDmJGVZTB5CmKZf6YQ3onILYKWZjhzckKPq0ywMyehpXlChilvJ5Y9BpMmG7+GlXFTiSGMfFowb0p2AWJq5IQeWzySCZHRRmJSQpasckJktHG+J5CYGonIyOFGpRPZc2NCjx2KVGIII/JYFdzoZmrhqgk/toiYZBTWO1GNvN4x4cdX/E9ev61gXsSkCT++yHUgu7uM17kyLioxhBFZVQYJ9xAxxjV3x0vkFoGnD3nkoCnHV/xLHqkAT9+EDyP1m5+G5Z5k9avUB1RiCBPyk0vw/ilEjsMvdWtGQsyZB3PmqRIZIUpWOWHOPOM8m0AIYfwaPnsaefljU2IIFSoxhAlZ5TQK5mX7vmDeaIjcIrh4DnnxA1PjUHxLXvwALp4z79fCTVPyf8corOdSXz7GQyWGMCC9XmNu+eIMxPS44TfwI7F8JVgj1M/9ECMry8AaYZxfE1lsM2FxBtJ1AOlVhfXGSiWGcHDqOLS50XIm7t6FOxFR0YilWcgjB5F9vWaHo/iA7OtFHjmIWJqFiIo2Oxzjdd7mhvrjZocStIZdwQ2grq6OXbt2oes6hYWFrF69esDjfX19lJSUcO7cOWJiYiguLiYhIQGAvXv3Ul5ejqZprFmzhvT0Wxc+dV3nhRdewGaz8cILLwDQ3NzMyy+/TEdHB/PmzWP9+vVYrSMKU7kDvbIMYmLhYf8WzBspkVtkrNd7cwaLEtzk8cPQdd30YaR+Dy+DmFj0yjIsfi4SGaqG/cWg6zqlpaVs3LiR7du3U1VVRWNj44A25eXlREVFsWPHDlatWsXu3bsBaGxsxOVysW3bNjZt2kRpaSn6bXe+/vrXv2b27NkD9vXaa6+xatUqduzYQVRUFOXl5b7oZ9iS7a1wsgaRVYAIlAS7YDHEJaiL0CFCVjkhLsE4rwFAWK3GIj7v1Bivf2XUhk0MDQ0NJCUlkZiYiNVqJTs7m5qamgFtamtrycvLAyAzM5P6+nqklNTU1JCdnU1ERAQJCQkkJSXR0NAAQEtLC8ePH6ew8FY9FSklp06dIjPTuDkmLy9v0LGU0ZHVFeD1TmgJjOEYhfUc8O7byKufmB2OMg7y6ifw7tvGbLcJKJg3UiLXcbOwXoXZoQSlYb9Cut1u4uJuXbCMi4vj7Nmzd2xjsViIjIyko6MDt9vN/Pnz+9vZbDbcbjcAP/7xj/nKV75Cd3d3/+MdHR1ERkZisVgGtf8sp9OJ02l849yyZQvx8WNbhcxqtY5520AnpaTl8AG0BYuwLX6k/98Doc/ex5/g6i9/ytQTh4n+8p/6/XiB0OeJNhF97izbx3UhiHv8CSwB8Pz29zk+HveCReiHy4l76k9Nm6I9Efxxnk0ZWzh27BixsbHMmzePU6dOjWkfDocDh+PWt+CxLoYdyouHyw/OoDdeQP/qNwb0MSD6LKzwUDrXnf9Jd+HvIjSLXw8XEH2eYP7us9S96M7/hIfSaRVWCIDn9/Y+6yvykD8p4erRKkTKgyZH5j/jOc+zZs0a8t+H/e1ns9loaWnp/7ulpQWbzXbHNl6vl66uLmJiYgZt63a7sdlsvPfee9TW1vLcc8/x8ssvU19fzyuvvEJMTAxdXV14b04z+7S9MjayynmzYF6u2aEMSeQWgfsqvKsK6wWld0+C+2rgXHT+DLEsFyZPUVOjx2DYxJCSkkJTUxPNzc14PB5cLhd2u31Am4yMDCoqKgCorq4mLS0NIQR2ux2Xy0VfXx/Nzc00NTWRmprKU089xauvvsrOnTspLi5m0aJFfPOb30QIQVpaGtXVxjKQFRUVg46ljIy80Y08+hbCnoOYMrEF80ZKpK+AqBj1xg1SsrIMomKM8xiAxJRIhD0HWVOJvNE9/AZKv2ETg8ViYe3atWzevJnnn3+erKwskpOT2bNnD7W1tQAUFBTQ2dnJ+vXreeONN3j66acBSE5OJisriw0bNrB582bWrVuHNswFqqeffpo33niD9evX09nZSUGBuXfqBit5zAU93QH7bQ5AREQgMvOQddXIznazw1FGQXa2I+uqEZl5iAg/Lw87DiK3CHq6VWG9URIyRJY8unTp0pi2C9WxZ+//fgE629D+9oeDLrwFUp9l43n0v/nviD/+M7TC3/XbcQKpzxPFn33W9/8S+W//iPbiDxD33u+XY4zFZ/sspUT/ztchehqWb/9vEyPzH1OuMSjBR15uhIbTphbMGylx7/0wN1UtyxhEpJTGMNLc1IBKCkMRQhhToxveRTY1Dr+BAqjEEJJk1X7QNOMmnyAgch3QeAFUYb3gcPEDaLwQUPfG3I3IKgBNUzdUjoJKDCFGer3Iw+Ww2I6InWF2OCMilj8GEZPUReggISvLIGKScd6CgIidAYvtyMPlSI/H7HCCgkoMoeadWmhrRQvgi86fJSI/Lax3CNnbY3Y4yl3I3h7kkUNGwbxI8wvmjZSWWwTt16D+mNmhBAWVGEKMXuWEadNhUYbZoYyKyC2C7utGQTYlYMnjh6E7gArmjdSiDJg23SgoqQxLJYYQItsCsGDeSD2wCGYmqeGkACcry2BmknG+gsitwnq1yGtDl9lRblGJIYTIw+Wg60FzUfB2QtMQ2YXw3jvI5iazw1GGIK9chvfeQWQXBlTBvJESuQ7QdWT1AbNDCXjBd3aVIUkpjVkXqQ8hku41O5wxEdmFN5dl3G92KMoQbi0PWzh84wAkku6F1IeQlU41NXoYKjGEig/ehcsfB9/Y722ELR7SHkG6ypG6WpYxkEj95vKwaY8Y5ylIidwi+ORjaHjX7FACmkoMIUJWlsHkqYiMHLNDGRct1wGtV+FUndmhKLc7XQetV43zE8RERg5MnoqsUtey7kYlhhAgb3Qha6sQy3IRU6aaHc74LFkO0dPQ1Rs3oOiVZRA9zTg/QUxMmYpYlousrULe6DI7nIClEkMIkDWV0HMjqIeRPiWsEYjMfKg7iuxoMzscBYzzUHcUkZmPsAZuwbyRMgrr3TDeN8qQVGIIAbLKCfckw7wFZofiE8ayjB5jWVLFdPJIBXg9QTnbbUjzFsA9yapExl2oxBDkZNNH8MGZoCiYN1Ji9ly4/wFVWC8AGAXznHD/A8Z5CQH9hfU+OGO8f5RBVGIIcrLSCRYLIivP7FB8SuQ44NJFuHB2+MaK/1xogI8/NM5HCBFZ+WCxGO8fZZAR3R5bV1fHrl270HWdwsJCVq9ePeDxvr4+SkpKOHfuHDExMRQXF5OQkADA3r17KS8vR9M01qxZQ3p6Or29vbz44ot4PB68Xi+ZmZk8+eSTAOzcuZPTp08TGWmsOvbcc89x3333+bDLoUN6PDcL5i1DTAuOgnkjJZY9ivzZPyErnYj7HzA7nLAlK8tg0iTEskfNDsWnxLTp8PAyo7Del54JvkoBfjbss6HrOqWlpfzVX/0VcXFx/OVf/iV2u5177711E1V5eTlRUVHs2LGDqqoqdu/ezfPPP09jYyMul4tt27bR2trKd7/7XX7wgx8QERHBiy++yJQpU/B4PHznO98hPT2dBx4wPgCeeeYZMjMz/dfrUPFOLXS0BVXBvJESkVGIjBxkzSHkk+sQkyebHVLYkT09yJpDiIwcRGSU2eH4nJZThH6iGk7WwNIss8MJKMMOJTU0NJCUlERiYiJWq5Xs7GxqamoGtKmtrSUvLw+AzMxM6uvrkVJSU1NDdnY2ERERJCQkkJSURENDA0IIpkyZAoDX68Xr9YbM+PhE0ivLINYGi5aaHYpfiJwi6O5SyzKaRB53QXeXcR5C0aKlEGszCk8qAwz7i8HtdhMXF9f/d1xcHGfPnr1jG4vFQmRkJB0dHbjdbubPn9/fzmaz4XYbBax0Xefb3/42ly9f5vOf//yAdj/96U95/fXXWbRoEU8//TQRQ6wp63Q6cTqNE7plyxbi48d2N6bVah3ztmbyuq9wtf4YkaufJiYxcVTbBkufZdxKWu65F+3oQWy/9+S49hUsffal8fbZfaQC/Z57icteGTRf3Ebb547CVXTt280MTWKxzfRjZP7jj9e2aQNrmqaxdetWrl+/zve//30uXrzInDlzeOqpp5g+fToej4cf/ehH/OIXv+CJJ54YtL3D4cDhuHVBbKxrngbrWsD6b/4DdJ0bS3PoGWX8wdRnPTMf795/5crpk4iEodenHYlg6rOvjKfPsvkS+qkTiC89Q0tLi48j85/R9lkuzYGf/ystb7yO9sU/9GNk/mPKms82m23AC6OlpQWbzXbHNl6vl66uLmJiYgZt63a7B20bFRVFWloadXVGCYQZM2YghCAiIoL8/HwaGhpG2MXw0T+F8IE0ROLYPyyDgcgqMArrqdkjE0pW7TcK5gXJ8rBjJRJnwQNpyCpVWO92wyaGlJQUmpqaaG5uxuPx4HK5sNvtA9pkZGRQUVEBQHV1NWlpaQghsNvtuFwu+vr6aG5upqmpidTUVNrb27l+/ToAvb29nDx5ktmzZwPQ2toK0H+NIjk52Zf9DQ1nT0PzpZCbQjgUMSMOFi01Zo94VWG9iSC9XqPC7aKlxvMf4kSOA5qb4Owps0MJGMMOJVksFtauXcvmzZvRdZ38/HySk5PZs2cPKSkp2O12CgoKKCkpYf369URHR1NcXAxAcnIyWVlZbNiwAU3TWLduHZqm0drays6dO9F1HSklWVlZZGQYK4698sortLe3AzB37lyeffZZP3Y/OMnKMpgS/AXzRkrLLUL/h7+HU8fh4WVmhxP6Th2Ha260L/+52ZFMCJGRg/zp/0FWliGCbAEifxEyRH4/Xbp0aUzbBdvYs+zuQv9//wSRmYf2zHNj2kfQ9dnTh/4/10LqQ1i+vnFM+wi2PvvCWPvs/Ye/h7On0b73z0FXG2msfdb/dSey+gDa93+CmBrph8j8x5RrDEpgkTVvQW9PWAwjfUpYI4w7VU/WINuvmR1OSJPt1+Dto4is0CiYN1IixwG9vciaQ2aHEhBUYggysrIMZs2BMLsbWOQ4wOtVyzL6mayuAK83rL54AMb7adYcNcnhJpUYgoj8+CKcfx+RWxQ088p9RcyaA/MWqGUZ/ciY7VYG8xYYz3cYEUIY5bjPv2+8z8KcSgxBRFaVgcWKyMwzOxRTiNwiaPoIzr1ndiih6dx70PRRSKzrMRYiMw8sViM5hjmVGIKE9PQhDx+AJcsRMbFmh2MKYc+FSZNVHX0/kVVOmDTZeJ7DkIiJhSXLkdUHkJ4+s8MxlUoMweJkDXS2B/2au+MhpkYi7LnIo28he26YHU5IkT03kDVvIey5QTcrx5e03CLobIe3a4ZvHMJUYggSeqUTpsdB2iNmh2IqY1nGbmStKqznS7K2Cm50h+0wUr+0dJgeZxSoDGMqMQQB2doC9ccR2YUIzWJ2OOZKfQgSZ6txYB+TVWWQONt4fsOY0CyI7EI4dQLpDq/7Xm6nEkMQkK79IHVETqHZoZiuf1nGhtPIyx+bHU5IkJc/hrOnQ2p52PEQOYUgdWMRrDClEkOAk7puXBRcsBiRcI/Z4QQEkZUPmqYuQvuIrHKCphnPq2K8zxYsNgrr6brZ4ZhCJYZAd/YUXLmMCOOLzp8lpttgsV0V1vMB6fXeXB7WbjyvCoDxfrtyOWwL66nEEOBkpROmRiIeyTY7lICi5TqgrRXqj5kdSnCrPwZtrWE9220o4pFsmBoZtteyVGIIYLLrOvJ4FWL5Y2rN489aZIdp08N+9sh46ZVOmDbdeD6VfmLyZMTyx5DHXMiu62aHM+FUYghgRsG8XjWFcAjCar1VWK+t1exwgpJsb4V3am4WzDNtMceAJXKLoK8XeTT8CuupxBDAZGUZzJ4Lc1PNDiUgiZwi0HVVWG+M5OEDNwvmqS8eQ5qbCrPnhuVwkkoMAUo2XoALZ8OyYN5IiXvuhZQHkZVlqrDeKPUvD5vyoPE8KoP0F9b7sAHZeN7scCbUiH4/1tXVsWvXLnRdp7CwkNWrVw94vK+vj5KSEs6dO0dMTAzFxcUkJNweGwgAACAASURBVCQAsHfvXsrLy9E0jTVr1pCenk5vby8vvvgiHo8Hr9dLZmYmTz75JADNzc28/PLLdHR0MG/ePNavX481DH/myionWMO3YN5Iidwi5L/sgA/OhP3NWaPywRm43Ij4k/VmRxLQRGYe8j9+jKx0Iv74z8wOZ8IM+4tB13VKS0vZuHEj27dvp6qqisbGxgFtysvLiYqKYseOHaxatYrdu3cD0NjYiMvlYtu2bWzatInS0lJ0XSciIoIXX3yRrVu38r3vfY+6ujref/99AF577TVWrVrFjh07iIqKorw8/G4ykX19yOoDiCUrENHTzA4noAl7LkyeEpY/98dDVpbB5ClhWzBvpET0NMSSFcjqCmRf+BTWGzYxNDQ0kJSURGJiIlarlezsbGpqBhaYqq2tJS8vD4DMzEzq6+uRUlJTU0N2djYREREkJCSQlJREQ0MDQgimTJkCgNfrxev1IoRASsmpU6fIzMwEIC8vb9CxwsLbR6CzQ110HgExZapRWK+2Enmjy+xwgoK80Y2srTQK5k2ZanY4AU/kFsH1DuN9GSaGHaNxu93ExcX1/x0XF8fZs2fv2MZisRAZGUlHRwdut5v58+f3t7PZbLjdbsD4JfLtb3+by5cv8/nPf5758+fT3t5OZGQkFotlUPvPcjqdOJ3Gna9btmwhPj5+NP3uZ7Vax7ytv7QePYQnPpH4RwsRFt/XRgrEPo9H7+N/SGuVk+gzJ5nqeHzINqHW55G4U5+7nW/Q3nOD6Y//IZNC7Dnxx3mWjxZydfc/YD16kBlfWD38BhPMH302bfBe0zS2bt3K9evX+f73v8/FixeZPn36iLd3OBw4HLduyhnrYtiBtki8dF9BrzuCWPUkLa3+mYYZaH0eLxmXBEn30v7bn3M9PXPINqHW55G4U5+9v/05JN1LW1wSIsSeE3+dZ5mZR++vfsaV995FxM30+f7HYzx9njVr1pD/PuxQks1mo6Wlpf/vlpYWbDbbHdt4vV66urqIiYkZtK3b7R60bVRUFGlpadTV1RETE0NXVxfem2UOhmof6oyCedKo8KiMiDF7xAEfnEE2fWR2OAFNNjXCB2cQuapg3miI7EKQEnl4v9mhTIhhE0NKSgpNTU00Nzfj8XhwuVzY7QPvkszIyKCiogKA6upq0tLSEEJgt9txuVz09fXR3NxMU1MTqamptLe3c/26cTdhb28vJ0+eZPbs2QghSEtLo7q6GoCKiopBxwplRsG8/fDgw4iZSWaHE1RUYb2RkVVlqmDeGIiZSfDgw8aa42FQWG/YoSSLxcLatWvZvHkzuq6Tn59PcnIye/bsISUlBbvdTkFBASUlJaxfv57o6GiKi4sBSE5OJisriw0bNqBpGuvWrUPTNFpbW9m5cye6riOlJCsri4yMDACefvppXn75Zf7t3/6N+++/n4KCAv8+A4HkvXfg6ieI1V8xO5KgI6bNgIeXIV3lyNXPqDt5hyA9HuOmtoeXGc+XMioitwj5Ty8Z79OHlpgdjl8JGSJ3Bl26dGlM2wXS2LP+jy8h62vRtv4YMcl/tZECqc++JN8+il7yd2hf34h4ZOC1hlDt8918ts+yrhp95/9C+8ZfIZYsNzEy//HneZa9Pejf+hpikR3tz/6HX44xFqZcY1AmhrzeiTzuQixf6dekENIWZUDsDHQ1nDQkvdIJsTOM50kZNTFpMmL5SuRxF/J6p9nh+JVKDAFCHj0Enj5178I4CIsFkVUA79Qirw09zTlcyWtueKcWkVXglynQ4ULkFoGnL+QL66nEECBklROS70fMTTE7lKAmchxGYb0wXpZxKPLwAdB14/lRxkzMTYHk+0P+TnuVGAKA/Og8fNigqlz6gEiaDfMXGrNHQuPy2bhJKY0vHvMXGs+PMi4ipwgufoC8eM7sUPxGJYYAICvLwBqByFxpdighQeQUQfMlOHva7FACQ8O78MnH6ouHj4jMlWCNCOmp0SoxmEz29SKrKxCPZCKiYswOJyQIew5MmRrSb9zRkJVlMGWq8bwo4yaiYhCPZN4srNdrdjh+oRKDyWTdEejqNO7cVXxCTJ6CWPaoUVivO7wL68nuLqNg3rJHEZOnmB1OyBC5DujqRJ6oNjsUv1CJwWSysgxsM+HB0L5hZqKJHAf09hjLo4YxWVsJvT3qorOvPbgE4hJC9lepSgwmki3N8O7biJxChKZOhU/NWwD3JIfsG3ekZGUZ3JNsPB+KzwhNM+onvfu28T4OMerTyESyyijIpb7N+V5/Yb1z7yEvXTQ7HFN4PjoP595TBfP8ROQYhS5D8cuHSgwmkbpuVFJ9aAkiLsHscEKSyMwHiyXk55zfSff+N8BiMZ4HxedEXAI8tARZtT/kCuupxGCWMyehpVn9WvAjMW06LFkedssyAkhPH90HfgNLlhvPg+IXIscB7itw5m2zQ/EplRhMIivLIDJ6ULE3xbe0HAd0tNFTW2V2KBPrZC2y/ZrRf8VvxCOZEBmNrAyt4SSVGEwgr3cgT1QjMvMQEZPMDie0pS2F6Ta69//S7EgmlF5ZhmaLN/qv+I2ImITIzEOeqEZe7zA7HJ9RicEE8shBo2Ce+jbnd8JiQWQX0nviCLK1ZfgNQoBsbYH640zN/6IqmDcBRI7DKKxXfdDsUHxGJQYTyMoymJOCmDPP7FDCgsgpNArrucJjWUZ5uBykzpTCx80OJSyIOfNgToqxOl6IGNEyV3V1dezatQtd1yksLGT16tUDHu/r66OkpIRz584RExNDcXExCQnGTJu9e/dSXl6OpmmsWbOG9PR0rl69ys6dO7l27RpCCBwOB1/84hcB+NnPfsb+/fuZNm0aAF/+8pdZujR0fg7Lix/AR+cRT/2F2aGEDZEwi4i0R+irciJ/54mQvmekv2DeA4uw3nMvhNniRGYRuUXI//9V5IcfhESF5GHfIbquU1paysaNG9m+fTtVVVU0NjYOaFNeXk5UVBQ7duxg1apV7N69G4DGxkZcLhfbtm1j06ZNlJaWous6FouFZ555hu3bt7N582befPPNAftctWoVW7duZevWrSGVFOC2gnnLHzM7lLAy1fE4XLkc+oX13j8FzU1qXY8JJpY/ZhTWC5Gp0cMmhoaGBpKSkkhMTMRqtZKdnU1NTc2ANrW1teTl5QGQmZlJfX09UkpqamrIzs4mIiKChIQEkpKSaGhoYMaMGcybZwyjTJ06ldmzZ+N2h/7CKrK3B3nkIGJpNiIq2uxwwsqUrHyYGhkyb9w7kVVlMDUSsTTb7FDCioiKRizNRh49iOztMTuccRt2KMntdhMXF9f/d1xcHGfPnr1jG4vFQmRkJB0dHbjdbubPn9/fzmazDUoAzc3NnD9/ntTU1P5/e/PNNzl06BDz5s3jq1/9KtHRgz9EnU4nTqcxRWzLli3Ex8ePpL+DWK3WMW87Wt1v/RftXdeJXfXfmDxBxxzKRPY5UFitVqY++jm6K36D7Rt/iRaCiVm/3smVYy6m5v0O02bPDtvzbFafe1f9N1qPHiS64RRTH/vchB3XH30e0TUGf7lx4wYvvfQSX/va14iMjATgc5/7HE888QQAe/bs4Sc/+Qlf//rXB23rcDhwOG7N6hnrYtgTuUi89zd7IT6R9qQ5CBPHfieyz4EiPj6eHnsu/Nc+rv52H9rKL5gdks/ph34LvT302HO5evVq2J5ns/osk+YY7+/f/JzrCyduCHw8fZ41a9aQ/z7sUJLNZqOl5dY0v5aWFmw22x3beL1eurq6iImJGbSt2+3u39bj8fDSSy/x6KOPsmLFiv4206dPR9M0NE2jsLCQDz74YBTdDFzyymVVMM9s982H2XNDdjhJVjph9lyjn8qEE5pmzIA7c9J4vwexYT+hUlJSaGpqorm5GY/Hg8vlwm63D2iTkZFBRUUFANXV1aSlpSGEwG6343K56Ovro7m5maamJlJTU5FS8uqrrzJ79mwef3zglLrW1tb+/z969CjJyck+6Kb5pKschEBkFZodStjqL6x34Syy8YLZ4fiU/PhDOP++KphnMpFdCEIE/dToYYeSLBYLa9euZfPmzei6Tn5+PsnJyezZs4eUlBTsdjsFBQWUlJSwfv16oqOjKS4uBiA5OZmsrCw2bNiApmmsW7cOTdM4c+YMhw4dYs6cOXzrW98Cbk1Lfe2117hw4QJCCGbOnMmzzz7r32dgAkjdi3Q5YWE6Im6m2eGENbEiH/n6vyCrnIg/+lOzw/EZWekEixWxQhXMM5OwzYSF6UjXfuTv/jFCC84bDIUMkRXTL126NKbtJmJMUtYfR//BX6P9+f9E2HP9eqyRCPexZ++rW+C9d9C+92NERITJkY2f9PShf2sNLFiE5S9e6P/3cD/PZpG1leg/+h7af/9rxCL/X2sw5RqDMn6yygnRMbBkxfCNFb/TcougswNOHjU7FN94+yh0thv9Usy3ZAVExwT1tSyVGPxMdrYj66oRK/JC4ttpSFiYDjPi0YP4jXs7vdIJM+KNfimmExERiBV5yLojyI52s8MZE5UY/ExWV4DHo+5EDSBCsyCyC+DUCaT7itnhjIt0X4VTJxDZBUE7nh2KRG4ReD3IIxVmhzImKjH4kZTS+Dk5NxVx731mh6PcRuQ4QEpjtlgQk679IHVVqTfAiHvvg7mpyMoygvEyrkoM/vRhA3z8ofq1EIDEzCRYsBhZ5QzaZRn7l4ddsNjojxJQRG4RfPyh8TkQZFRi8CNZ5YSISapgXoASuUVw9RN47x2zQxmb9+vhymX1xSNAieWPQcSkoLwIrRKDnxgF8w4hMrIRkVFmh6MMQSzNgqlRRgIPQrLKCVOjjH4oAUdERiEyspFHDyF7gquwnkoMfiKPu6D7uvo2F8DEpMmIFY8hjx9GdnWaHc6oyK5O5DEXYsVjiEmTzQ5HuQORWwTdXcgTLrNDGRWVGPxEVjphZhLMTzM7FOUuRG4R9PUijx4yO5RRkUcPQV+v+uIR6B5YBDOTjM+DIKISgx/I5iZ47x1EjkMVzAt0c1Lg3vuC7o0rK51w731G/ErAEkIYM8bee8f4XAgS6lPLD2TVfhAaIqvA7FCUYRiF9YrgwwbkR+fNDmdEZON5+LABkVukCuYFAaOwnmZ8LgQJlRh8zCiYtx/SHkHYwmuRlGAlVqwEqzVoLkLLSidYrUbcSsATM+Jg0VKjsJ7uNTucEVGJwddO1cG1FlW3JoiI6GmI9ExkdQWyr8/scO5K9vUhqysQ6ZmI6Glmh6OMkJbjgGstxudDEFCJwcf0yjKIngZLlpkdijIKIrcIrncg66rNDuWuZN0RuN6hLjoHmyXLICY2aOpzqcTgQ7KjDd4+isjMR1hVwbyg8tDDYJsZ8BehZWUZ2GYa8SpBQ1gjEJl58PZR43MiwI1ozee6ujp27dqFrusUFhayevXqAY/39fVRUlLCuXPniImJobi4mISEBAD27t1LeXk5mqaxZs0a0tPTuXr1Kjt37uTatWsIIXA4HHzxi18EoLOzk+3bt3PlyhVmzpzJ888/T3R0cCzcLqsrwKsK5gUjo7BeIfJXe5AtzYi4BLNDGkS2XIF36xCr/kgVzAtCIqcIWfYLYyiw6PfNDueuhv3FoOs6paWlbNy4ke3bt1NVVUVjY+OANuXl5URFRbFjxw5WrVrF7t27AWhsbMTlcrFt2zY2bdpEaWkpuq5jsVh45pln2L59O5s3b+bNN9/s3+e+fftYvHgxr7zyCosXL2bfvn1+6Lbv9RfMu/8BxOw5ZoejjIHIMZZdDdTZI58uF/lpnEpwEbPnwP0PBEVhvWETQ0NDA0lJSSQmJmK1WsnOzqampmZAm9raWvLy8gDIzMykvr4eKSU1NTVkZ2cTERFBQkICSUlJNDQ0MGPGDObNmwfA1KlTmT17Nm63G4CamhpWrjRmW6xcuXLQsQLWhbNw6aKxprASlER8Ijz48M3ZI4FVWE/qujFr6sGHjTiVoCRyHXDpIpx/3+xQ7mrYoSS3201cXFz/33FxcZw9e/aObSwWC5GRkXR0dOB2u5k/f35/O5vN1p8APtXc3Mz58+dJTU0FoK2tjRkzZgAwffp02tqGHo9zOp04ncZ48JYtW4iPH9vUUKvVOuZtb9f+76V0T55C/Be+hBbgtZF81edgMtI+d//Ol2jf9tdMa/qQyQE0gaDnZC3XWpqZ9idfZ+oIz506z4FH/8KXuPKzf2bysUqmLc/xyT790ecRXWPwlxs3bvDSSy/xta99jcjIyEGPCyHueAOPw+HA4bj17Xysa576Yo1Y2dODfui/EEuzcXd1Q1f3uPbnb4GwLu5EG2mfZeoiiIyi7Vevo82+fwIiGxn9V69DZBSdqYu4PsJzp85zYBJLs+k+9F/0/N7TiMlTxr0/U9Z8ttlstLS09P/d0tKCzWa7Yxuv10tXVxcxMTGDtnW73f3bejweXnrpJR599FFWrLi1FnJsbCytra0AtLa2Mm1a4M/Vlseq4Ea3GkYKASJiEmLFSuSJauT1DrPDAUBe70QeP4xYsRIRMcnscJRxErkOuNGNPBa4hfWGTQwpKSk0NTXR3NyMx+PB5XJht9sHtMnIyKCiogKA6upq0tLSEEJgt9txuVz09fXR3NxMU1MTqampSCl59dVXmT17No8//viAfdntdg4ePAjAwYMHWbYscH7O34msKoOEe1TBvBAhcovA04c8ctDsUACQRw+Cp0/NdgsV89MgYZbxuRGghh1KslgsrF27ls2bN6PrOvn5+SQnJ7Nnzx5SUlKw2+0UFBRQUlLC+vXriY6Opri4GIDk5GSysrLYsGEDmqaxbt06NE3jzJkzHDp0iDlz5vCtb30LgC9/+cssXbqU1atXs337dsrLy/unqwYy+ckleP8U4kvPqLo1IULMSYE584yLvQWPD7+Bn8nKMpgzz4hLCXpGfS4H8uc/QX5yCZE49HCOmYQM9HlTI3Tp0qUxbTfeMUn95z9B/vbnaN8rRUyPG36DABAM47C+Nto+6+VvIH/6f9D+v+2mfiDLix+gf/d5xJefRRtlklLnOXDJay3o/3Md4gt/gPYHXx3Xvky5xqDcmfR6jcXkF2cETVJQRkasyANrhOnLMhoF8yKMeJSQIabHweIMpKsc6Q28wnoqMYzHqePQ5jYKZCkhRURFI5ZmIY8cRPb1mhKD7OtFHjmIWJqFiAqOu/+VkdNyi6DNDfXHzQ5lEJUYxkGvLIOYWHg48C+QK6MnchzQdR15/LApx5cnqqGr04hDCT2L7UZhvQC8CK0SwxjJ9lY4WYPIKkBYTb0dRPGXBx+GuATT1mmQlWUQl2DEoYQcYbUai3mdrDE+TwKISgxjZBTM86p7F0KY0DTj2/q7byOvfjKhx5ZXP4EzJ9XysCFO5DrA60UerjA7lAHUK24MjIJ5Tkh5EHFPstnhKH5kLMsoJrywXn/BvGxVMC+UiXuSIeVBZJUzoArrqcQwFufeg6aP1NhvGBBxM+GhdKTLOWHLMkrdaySih9KN4yshTeQ4oOkj43MlQKjEMAayygmTpyCW5ZodijIBRK4D3Ffh3ZMTc8AzJ8F9RQ1ThgmxLBcmTwmoNcdVYhgleaMbefQthD0HMWVw4T8l9Ij0TIiKmbB7GmSlE6JijOMqIU9MiUTYc5BH30LeCIwCnCoxjJI85oKeblW3JoyICGNZRllXjexs9+uxZGc78sRhRGYeIkItDxsuRG4R9HQbBTkDgEoMoyQryyBpNqQ8ZHYoygQSOQ7wePxeWE8eOQQej7p+FW5SHoKk2QGz5rhKDKMgLzdCw2ljCqEqmBdWRPL9MDfVr8sy9i8POzfVOJ4SNoQQxpeBhtPG54zJVGIYBVm1HzTNuClFCTsi1wGNF+DiB/45wMUPoPG8uugcpkRWAWhaQPxqUIlhhKTXizxcDovtiNgZZoejmEAsfwwiJvntIrSsdELEJOM4StgRsTNgsR15uBzp8Zgai0oMI/VOLbS1GoWvlLAkIj8trHcI2dvj033L3p5bBfMiVcG8cKXlFkH7Nag/Zm4cph49iOhVTpg2HRZlmB2KYiKRWwTdvi+sJ48fhu7rarZbuFtsh9gZRoFOE42o+ltdXR27du1C13UKCwtZvXr1gMf7+vooKSnh3LlzxMTEUFxcTEJCAgB79+6lvLwcTdNYs2YN6enpAPzwhz/k+PHjxMbG8tJLL/Xv62c/+xn79+/vX+v505XdzCTbbhbMK1qtCuaFuwcWQXyiMZyUmeez3coqJ8QnGvtXwpawWBCZ+ciyfci2VtOGrYf9xaDrOqWlpWzcuJHt27dTVVVFY+PAq+bl5eVERUWxY8cOVq1axe7duwFobGzE5XKxbds2Nm3aRGlpKbquA5CXl8fGjRuHPOaqVavYunUrW7duNT0pAMa1BV1XFwWVW4X13nsH2dzkk33KK5dVwTyln8h1gK4bnzsmGfZV2NDQQFJSEomJiVitVrKzs6mpqRnQpra2lry8PAAyMzOpr69HSklNTQ3Z2dlERESQkJBAUlISDQ0NACxcuJDo6MAfS5VSGt/mUh9CJN1rdjhKABDZBUZhPZdvCutJ134QwtivEvZE0r2QutDUwnrDjou43W7i4m4tWxkXF8fZs2fv2MZisRAZGUlHRwdut5v58+f3t7PZbLjd7mGDevPNNzl06BDz5s3jq1/96pAJxOl04nQa07q2bNlCfHz8sPsditVqveu2ve+epPXyx0z7xp8wdYzHCDTD9TkU+bTP8fG0pq/AU11B3Jr1CItlzLuSXi9XDx8gIn0FMx7w7U2T6jwHr+4vrKa95H8Re7WJSQ/dfT0Of/Q54AbMP/e5z/HEE08AsGfPHn7yk5/w9a9/fVA7h8OBw3FraGesi2EPt5C2/qvXYfJUOhcs4XoQLDI+EsGyYLov+brPcsVK9BPVXD3kRCwe+4QEWX8MvaUZzx+u8fk5Uec5eMkFS2DyVK796t/RZs66a9vx9HnWrKH3PexQks1mo6Wlpf/vlpYWbDbbHdt4vV66urqIiYkZtK3b7R607WdNnz4dTdPQNI3CwkI++MBPNxONgLzRhaytRCzLRUyZalocSgB6eDlEx4x7WUZZ6YToGGN/inKTmDIVsSwXWVuFvNE14ccfNjGkpKTQ1NREc3MzHo8Hl8uF3W4f0CYjI4OKigoAqqurSUtLQwiB3W7H5XLR19dHc3MzTU1NpKam3vV4ra23lrg7evQoycnmLYQjayqh54aaQqgMYhTWy4e6o8iOtjHtQ3a0I+uOIDLzVcE8ZRCjsN4N43Nogg07lGSxWFi7di2bN29G13Xy8/NJTk5mz549pKSkYLfbKSgooKSkhPXr1xMdHU1xcTEAycnJZGVlsWHDBjRNY926dWg3Z128/PLLnD59mo6ODv7iL/6CJ598koKCAl577TUuXLiAEIKZM2fy7LPP+vcZuAtZ5YR7kmHeAtNiUAKXyC1COv8TWV2BKPr9UW8vjxwAr0d98VCGNm8B3JNsfA49+rkJPbSQgbSe3DhcunRpTNvdaXxONn2E/p3nEE+sQfv8l8YbXkAJlXHY0fBXn72b/wf09qD99Y5RFVaUUqL/zTchYhKWTS8Nv8EYqPMc/PQ39yJf34X2tzvvuIywKdcYwpWsdILFgsjKMzsUJYCJ3CK4dBEunB2+8e0uNMDHH6pfC8pdiax8sFgmvLCeSgxDkB7PzYJ5yxDTVME85c7Eskdh0qRRv3FlZRlMmmRsryh3IKZNh4eXTXhhPZUYhvJOLXS0qYJ5yrBEZBRiaQ6y5hCyZ2SF9WRPD7LmEGJpDiIyys8RKsFOyymCjjbjc2mijjlhRwoiemUZxNpgkfnlOJTAZxTW6xrxsozyuAu6u9QwkjIyi5ZCrG1CC+upxPAZ8loLvHMMkZ0/rjtalTDyQBrMTDJmj4yArHLCzCRjO0UZhrBYjHIp7xwzPp8mgEoMnyEPHwCpI3LUtzllZPqXZXy/Htl899lxsrkJ3ntHLQ+rjIrIcYDUjc+nCaASw22MNXed8EAaIvHut6Eryu1EdiGI4ZdllFVOEJrRXlFGSCTOggfSkJUTU1hPJYbbnT0NzZeM7KwooyBmxMGipcbsEa93yDZS9xqVVBctNdoryiiInCJovmR8TvmZSgy3kZVlMGUqIiPH7FCUIKTlOuCaG04dH7rBqRNwzW20U5RREhnZMGWq39Ycv51KDDfJm7NKxPLHEJOnmB2OEoweXgYxsXecPaJXlkFMrNFOUUZJTJ6CWP4Y8lgVstu/hfVUYrhJ1rwFvT1qGEkZM2GNQGTmwckaZPu1AY/JjjZ4+ygiMw9hVQXzlLEROQ7o7TE+r/xIJYabZGUZzJoD9z9gdihKEBO5ReD1IqsHzh6Rhw+A16vuXVDG5/4HYNYcvw8nqcQAyI8vwvn3EblFagqhMi5i1hyYt2DA7BFjtlsZzFtgPK4oYySEML5cnH/f+NzyE5UYAFlVBharMQygKOMkchzQ9BGce8/4h/PvQ9NHaphS8QmRmQcWq/G55SdhnxhkX5/xM3/JckRMrNnhKCHAKKw3uf9OaKNg3mRVME/xCRETC0uWIw8fQHr6/HKMsE8MPbVV0NmuphAqPiOmRiIycpBH30K2X0PWvIXIyEFMjTQ7NCVEaLlF0NkOb9f4Zf/DruAGUFdXx65du9B1ncLCQlavXj3g8b6+PkpKSjh37hwxMTEUFxeTkJAAwN69eykvL0fTNNasWUN6ejoAP/zhDzl+/DixsbG89NKthUo6OzvZvn07V65cYebMmTz//PNER0f7qr+DdO//JUyPg7RH/HYMJfyI3CLk4XL0f3oJbnSri86Kb6Wlw/Q49ConfP73fL77YX8x6LpOaWkpGzduZPv27VRVVdHY2DigTXl5OVFRUezYsYNVq1axe/duABobG3G5XGzbto1NmzZRWlqKrusA5OXlsXHjxkHH27dvH4sXL+aVV15h8eLF7Nu3zxf9HJJsbaH3Xe3qiwAACSBJREFUxBFEdiFCUwXzFB+avxASZsG7bxv/nb/Q7IiUECI0i1FWpf443pYrPt//sImhoaGBpKQkEhMTsVqtZGdnU1Mz8OdLbW0teXl5AGRmZlJfX4+UkpqaGrKzs4mIiCAhIYGkpCQaGhoAWLhw4ZC/BGpqali5ciUAK1euHHQsX5Ku/aDriBxVt0bxLWP2iDE8KXJVwTzF90ROIUTH4Gm84PN9DzuU5Ha7iYu7VdclLi6Os2fP3rGNxWIhMjKSjo4O3G438+fP729ns9lwu913PV5bWxszZhirpk2fPp22trYh2zmdTpxO4+Leli1biI+PH64rg3TfOwdP0e8Ss3DxqLcNZlardUzPVzAzo8/6l57ieu8Nor70FFr0tAk9NqjzHPLi45G7fknE5ClM9vHqbiO6xmAWIcQdv2k5HA4cjlsXjMe0GPaSTOILHw+pxcNHItQWTB8J0/r8+1+h50Yv3Jj4Y6vzHB7G0+dZs4auIj3sUJLNZqOl5dbiEC0tLdhstju28Xq9dHV1ERMTM2hbt9s9aNvPio2NpbW1FYDW1lamTZv4b1qKoijhbNjEkJKSQlNTE83NzXg8HlwuF3a7fUCbjIwMKioqAKiuriYtLQ0hBHa7HZfLRV9fH83NzTQ1NZGamnrX49ntdg4ePAjAwYMHWbZMFRxTFEWZSEKOYNWH48eP8y//8i/ouk5+fj5/8Ad/wJ49e0hJScFut9Pb20tJSQnnz58nOjqa4uJiEhMTAfj5z3/OgQMH0DSNr33tazzyiDEt9OWX/2979xfSVP/HAfx9mCktbfOc0MiKmtmFhgkpSlCmRhdREF4IRRde5kqR6GLdRDcRBMtBTraL0PCuixTsoiBMQ0SYfzFXy8xCeKrljsqZU6fb57kQx3OenueHS5/Obzuf15U7Dvx89mZ+zvnu6NcBr9cLRVFgMplQU1ODyspKKIqCpqYmzM7OxnW76h9//O+ds/4NX3rqA/esD9xzfP5tKWlTgyER8GDYPO5ZH7hnfdDkMwbGGGP6woOBMcaYCg8GxhhjKjwYGGOMqSTNh8+MMca2h+6vGGw2m9Yl/Hbcsz5wz/rwX/Ss+8HAGGNMjQcDY4wxFcPdu3fval2E1iwWi9Yl/Hbcsz5wz/qw3T3zh8+MMcZUeCmJMcaYCg8GxhhjKv/XG/X810ZHR9Ha2opoNIqqqipcunRJ65K2bHZ2Fk6nE/Pz8xAEAWfPnsX58+cRDAbR1NSEHz9+qP5rLRGhtbUVIyMjSEtLg9VqTdg12mg0CpvNBlEUYbPZ4Pf74XA4oCgKLBYL6uvrkZKSgtXVVTQ3N+PTp0/IyMhAY2MjsrKytC4/bouLi3C5XJiZmYEgCKirq8O+ffuSOufnz5+ju7sbgiDgwIEDsFqtmJ+fT6qcW1paMDw8DJPJBLvdDgC/9P7t6enBs2fPAADV1dWx7Zc3hXQqEonQjRs36Nu3b7S6ukq3bt2imZkZrcvaMlmWaWpqioiIQqEQNTQ00MzMDLW3t1NHRwcREXV0dFB7ezsREQ0NDdG9e/coGo2Sz+ej27dva1b7VnV1dZHD4aD79+8TEZHdbqe+vj4iInK73fTy5UsiInrx4gW53W4iIurr66OHDx9qU/AWPXr0iF69ekVERKurqxQMBpM650AgQFarlVZWVohoPd/Xr18nXc4TExM0NTVFN2/ejB2LN1dFUej69eukKIrq683S7VLSx48fsXfvXmRnZyMlJQUnT56Ex+PRuqwty8zMjJ0x7Ny5Ezk5OZBlGR6PB+Xl5QCA8vLyWK+Dg4M4ffo0BEHA0aNHsbi4GNtBL5EEAgEMDw+jqqoKAEBEmJiYQFlZGQDgzJkzqp43zp7Kysrw9u1bUILdgxEKhfDu3TtUVlYCWN/reNeuXUmfczQaRTgcRiQSQTgchtlsTrqc8/Pzf9qDJt5cR0dHUVhYiPT0dKSnp6OwsBCjo6ObrkG3S0myLEOSpNhjSZIwOTmpYUXbz+/3Y3p6GkeOHMHCwgIyMzMBAGazGQsLCwDWX4e/bp4uSRJkWY49N1G0tbXh6tWrWFpaAgAoigKj0QiDwQBgfftZWZYBqLM3GAwwGo1QFCWhtpH1+/3YvXs3Wlpa8OXLF1gsFtTW1iZ1zqIo4uLFi6irq0NqaiqOHz8Oi8WS1DlviDfXv/9+++vrshm6vWJIdsvLy7Db7aitrYXRaFR9TxAECIKgUWXbb2hoCCaTKSHXzH9VJBLB9PQ0zp07hwcPHiAtLQ2dnZ2q5yRbzsFgEB6PB06nE263G8vLy3GdBSeL35Grbq8YRFFEIBCIPQ4EAhBFUcOKts/a2hrsdjtOnTqF0tJSAIDJZMLc3BwyMzMxNzcXO2sSRVG1+1Mivg4+nw+Dg4MYGRlBOBzG0tIS2traEAqFEIlEYDAYIMtyrK+N7CVJQiQSQSgUQkZGhsZdxEeSJEiShLy8PADrSyWdnZ1JnfP4+DiysrJiPZWWlsLn8yV1zhvizVUURXi93thxWZaRn5+/6Z+n2yuG3NxcfP36FX6/H2tra+jv70dxcbHWZW0ZEcHlciEnJwcXLlyIHS8uLkZvby8AoLe3FyUlJbHjb968ARHhw4cPMBqNCbW8AABXrlyBy+WC0+lEY2Mjjh07hoaGBhQUFGBgYADA+h0aG/meOHECPT09AICBgQEUFBQk3Jm12WyGJEmxLW3Hx8exf//+pM55z549mJycxMrKCogo1nMy57wh3lyLioowNjaGYDCIYDCIsbExFBUVbfrn6fovn4eHh/HkyRNEo1FUVFSgurpa65K27P3797hz5w4OHjwYexNcvnwZeXl5aGpqwuzs7E+3uz1+/BhjY2NITU2F1WpFbm6uxl38uomJCXR1dcFms+H79+9wOBwIBoM4fPgw6uvrsWPHDoTDYTQ3N2N6ehrp6elobGxEdna21qXH7fPnz3C5XFhbW0NWVhasViuIKKlzfvr0Kfr7+2EwGHDo0CFcu3YNsiwnVc4OhwNerxeKosBkMqGmpgYlJSVx59rd3Y2Ojg4A67erVlRUbLoGXQ8GxhhjP9PtUhJjjLF/xoOBMcaYCg8GxhhjKjwYGGOMqfBgYIwxpsKDgTHGmAoPBsYYYyp/AkyDqgXSPnXJAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "id": "UPXizw35wwdO", "outputId": "674c0c79-c01c-41d7-a10d-466fb3dbefd1" }, "source": [ "plot_lr(triangular(250, 0.005, 'triangular2'))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3xU1bnw8d/aM+GSC4FJSCIYVBK8EFAkgyQhSm5oK/aUtlaPolagpSrFE6hWBN9jz/HwaVqEIASL2hRvVLFWsNpWyxADJmMkASMGFAkXIRIMZEIIBMhk9nr/GA1GLrnNZM9lff/RZNbe+1mZkGf22ms9S0gpJYqiKIryNc3oABRFURTfohKDoiiK0o5KDIqiKEo7KjEoiqIo7ajEoCiKorSjEoOiKIrSjtnoADzl4MGD3TouOjqaI0eOeDga36b6HBxUn4NDT/o8ZMiQc35f3TEoiqIo7ajEoCiKorSjEoOiKIrSjkoMiqIoSjsqMSiKoijtdGpWUmVlJatWrULXdbKzs5kyZUq7151OJwUFBezZs4eIiAhyc3OJiYkBYO3atRQVFaFpGtOmTWPMmDEAzJo1i379+qFpGiaTiby8PACOHz9Ofn4+hw8fZvDgwcyZM4fw8HBP9llRFEW5gA7vGHRdp7CwkPnz55Ofn09paSk1NTXt2hQVFREWFsby5cuZPHkyq1evBqCmpga73c6SJUtYsGABhYWF6Lredtzjjz/OokWL2pICwLp16xg9ejTLli1j9OjRrFu3zlN9VRRFUTqhw8RQXV1NXFwcsbGxmM1m0tLSKC8vb9emoqKCjIwMAFJSUqiqqkJKSXl5OWlpaYSEhBATE0NcXBzV1dUXvF55eTkTJ04EYOLEiWddSzGObGpEL92AqtSuKIGtw6Ekh8NBVFRU29dRUVHs2rXrvG1MJhOhoaE0NTXhcDgYMWJEWzuLxYLD4Wj7euHChQBMmjSJnJwcABobGxk0aBAAAwcOpLGx8Zxx2Ww2bDYbAHl5eURHR3fc23Mwm83dPtZfdbfPx9a9yMl/vM7AEVfQZ+QYL0TmPep9Dg6qzx46p0fP1gVPPPEEFouFxsZG/u///o8hQ4YwcuTIdm2EEAghznl8Tk5OWzIBur3yT62U7BzpbEF/7x0Ajv7jb2gxF3sjNK9R73NwUH3umm6vfLZYLNTX17d9XV9fj8ViOW8bl8tFc3MzERERZx3rcDjajv3mv5GRkYwbN65tiCkyMpKGhgYAGhoaGDBgQKc7qXiPrPwQmo/DkGHIihLkyWajQ1IUxUs6TAwJCQnU1tZSV1dHa2srdrsdq9Xark1ycjLFxcUAlJWVkZSUhBACq9WK3W7H6XRSV1dHbW0tiYmJnDp1ipMnTwJw6tQptm3bxrBhwwCwWq1s3LgRgI0bNzJu3DhP9lfpJlmyHiyD0e75FbScRpa/b3RIiqJ4SYdDSSaTienTp7Nw4UJ0XSczM5P4+HjWrFlDQkICVquVrKwsCgoKmD17NuHh4eTm5gIQHx9Pamoqc+fORdM0ZsyYgaZpNDY28uSTTwLuO4z09PS2aaxTpkwhPz+foqKitumqirFkfR18+jHiltth+BVwUTyy1AY33GR0aIqieIGQATLFRFVX7byu9ln/+yvIt19F+91ziKgY9H+vQ/71z2j/U4AYMsyLkXqOep+Dg+pz16jqqkq3SF1H2jfAVdcgotyLFkVqJphM7uElRVECjkoMyoV9tg3q6xATzswAExGRcM11yLJiZKvTwOAURfEGlRiUC5Il6yE0HHFtSrvva+mToKkRtqkFiIoSaFRiUM5LnmhCflSGSMlAhPRp/2LStTAwCr3EZkxwiqJ4jUoMynnJDzdCq7PdMNI3hGZCpGVB1VZkQ/05jlYUxV+pxKCclyxZD8MSEMOGn/N1MSEH5NcPpxVFCRgqMSjnJL/YDQf2ItInnbeNiLkIrhiNLLUhv1U1V1EU/6YSg3JOsnQ9mEMQ191wwXZiQg4cPgS7dvRSZIqieJtKDMpZZMtp5IcbEWPTEGEX3iRJjE2D/qFqTYOiBBCVGJSzyI/KoPkEIv3sh87fJfr2RYy7Abm1FNl8oheiUxTF21RiUM4iS20QHQtXjO5Ue5E+CVpaVGE9RQkQKjEo7cjDh9wF8yZkI7RO/npcmghDL1HDSYoSIFRiUNqR9iIQApGa3eljhBDuYad9u5A1+7wXnKIovUIlBqWN1F1Iuw1GjkFEDe7SsWJ8JpjM7mEoRVH8mkoMyhk7PgbHEXcdpC4SEQMQY8Yjy95DOlVhPUXxZyoxKG1kqQ3CI+Ca8d06XqTnwPEm2LbZw5EpitKbOtzBDaCyspJVq1ah6zrZ2dlMmTKl3etOp5OCggL27NlDREQEubm5xMS4a/evXbuWoqIiNE1j2rRpbTu1Aei6zrx587BYLMybNw+AFStWsGPHDkJDQwGYNWsWl156qSf6qlyAPH4MWVmGmPh9REhI904ycgxYotFL1mNKnuDZABVF6TUdJgZd1yksLOSxxx4jKiqKRx99FKvVysUXX9zWpqioiLCwMJYvX05paSmrV69mzpw51NTUYLfbWbJkCQ0NDTzxxBM89dRTaF/PdvnnP//J0KFD2/Z//sbdd99NSkr7Ms+Kd8myYmhtvWAJjI64C+tlI//xGtJxGGHp2nMKRVF8Q4dDSdXV1cTFxREbG4vZbCYtLY3y8vY1+CsqKsjIyAAgJSWFqqoqpJSUl5eTlpZGSEgIMTExxMXFUV1dDUB9fT1bt24lO7vzs18U75BSuqeaXpKIuPjSHp1LpGWDlO7ZTYqi+KUO7xgcDgdRUVFtX0dFRbFr167ztjGZTISGhtLU1ITD4WDEiBFt7SwWCw6HA4Dnn3+eu+6666y7BYBXXnmF119/nVGjRjF16lRCzjG0YbPZsNncM2Dy8vKIjo7uTH/PYjabu32sv/pun53Vn+L48gsifvkwoT39WURH0zA6GVfZe0Tdc3/n10J4mXqfg4Pqs4fO6dGzddKWLVuIjIxk+PDhbN++vd1rd955JwMHDqS1tZVnnnmGN998k1tvvfWsc+Tk5JCTc6ZkQ3c3w1abh4P+9l8hpA8nRo6l2QM/C/26icjCJRwpfQ9x1TU9Pp8nqPc5OKg+d82QIUPO+f0OP85ZLBbq689sxFJfX4/FYjlvG5fLRXNzMxEREWcd63A4sFgs7Ny5k4qKCmbNmsXSpUupqqpi2bJlAAwaNAghBCEhIWRmZrYNPSneIU+fRm7ehEhOQ4SGeeScYmwq9A9TaxoUxU91mBgSEhKora2lrq6O1tZW7HY7Vqu1XZvk5GSKi4sBKCsrIykpCSEEVqsVu92O0+mkrq6O2tpaEhMTufPOO1m5ciUrVqwgNzeXUaNG8eCDDwLQ0NAA0PaMIj4+3sNdVr5NfmSHk809euj8XaJPX8T4icitHyCbj3vsvIqi9I4Oh5JMJhPTp09n4cKF6LpOZmYm8fHxrFmzhoSEBKxWK1lZWRQUFDB79mzCw8PJzc0FID4+ntTUVObOnYumacyYMaNtRtL5LFu2jGPHjgFwySWXMHPmTA90UzkfWWKDwXEwIsmj5xXpOcjif7rvRjJu9ui5FUXxLiGllEYH4QkHDx7s1nHBPCYp62rRF/wSMeUutMm3efQaUkr0/80FkwnTY0s8eu7uCOb3OZioPndNt58xKIFLlm4AoSFSszx+bndhvUnwRTXywF6Pn19RFO9RiSFIuQvmbYCkaxEW70zvE+NvALMqrKco/kYlhmC1vRKO1nerYF5nifABiGtTkWXFqrCeovgRlRiClF6yHsIHwDXjvHodkZ4DJ5qQlWVevY6iKJ6jEkMQ0hsb4OPNiJRMhLmbBfM668prwDLYPftJURS/oBJDEDq58V1w9axgXmcJTUNMyIZPK5H1dV6/nqIoPacSQ5CRUnLS9hZcdjli6LBeuaZIcxdKlKUbeuV6iqL0jEoMwWbfLlwH9rrH/nuJiI6FK69G2jcgdb3XrqsoSveoxBBkZMl66NsPMe6GXr2uSJ8E9XXw2bZeva6iKF2nEkMQ+aZgXr+0LET/0F69trg2BULD1ZoGRfEDKjEEEbmlFE6dpH/25F6/tgjpc6aw3glVWE9RfJlKDEFElq6HmIsIGTmm48ZeINInQasT+WGxIddXFKVzVGIIEvKrg/D5dsSEHIQQhsQghg2HYcPVcJKi+DiVGIKELLW5C+aleb5gXleI9Emwfw9y/25D41AU5fxUYggC0uVC2otgdDJiYFTHB3iRuG4imEPcs6MURfFJKjEEg+1bodGBNqH31i6cjwgLR4xNRX64EelsMTocRVHOocMd3AAqKytZtWoVuq6TnZ3NlClT2r3udDopKChgz549REREkJubS0xMDABr166lqKgITdOYNm0aY8acefCp6zrz5s3DYrEwb948AOrq6li6dClNTU0MHz6c2bNnYzZ3KkzlPPSS9RARCVd7t2BeZ4n0ScjNm5BbP0CMn2h0OIqifEeHdwy6rlNYWMj8+fPJz8+ntLSUmpqadm2KiooICwtj+fLlTJ48mdWrVwNQU1OD3W5nyZIlLFiwgMLCQvRvrXz95z//ydChQ9ud6+WXX2by5MksX76csLAwioqKPNHPoCWPNcC2ckRqFsJXEuwVoyEqRj2EVhQf1WFiqK6uJi4ujtjYWMxmM2lpaZSXl7drU1FRQUZGBgApKSlUVVUhpaS8vJy0tDRCQkKIiYkhLi6O6upqAOrr69m6dSvZ2dlt55FSsn37dlJSUgDIyMg461pK18iyYnC5erUERkfchfVy4NOPkUe+MjocRVG+o8OPkA6Hg6ioMw8so6Ki2LVr13nbmEwmQkNDaWpqwuFwMGLEiLZ2FosFh8MBwPPPP89dd93FyZMn215vamoiNDQUk8l0Vvvvstls2GzuT5x5eXlER3dvFzKz2dztY32dlJL6D95Du2IUltHXtn3fF/rsuuVWjrz1Cv0/+oDwO37u9ev5Qp97m+pzcPBGnw0ZW9iyZQuRkZEMHz6c7du3d+scOTk55OSc+RTc3c2wA3nzcLn7M/Safej3/KpdH32iz8IMV43hhO3vnMz+AUIzefVyPtHnXqb6HBx60uchQ4ac8/sdDiVZLBbq6+vbvq6vr8disZy3jcvlorm5mYiIiLOOdTgcWCwWdu7cSUVFBbNmzWLp0qVUVVWxbNkyIiIiaG5uxuVytWuvdI8stX1dMC/d6FDOSaRPAscR+FQV1lMUX9JhYkhISKC2tpa6ujpaW1ux2+1YrdZ2bZKTkykuLgagrKyMpKQkhBBYrVbsdjtOp5O6ujpqa2tJTEzkzjvvZOXKlaxYsYLc3FxGjRrFgw8+iBCCpKQkysrc20AWFxefdS2lc+Spk8jN7yOsExD9erdgXmeJMeMhLEKtaVAUH9PhUJLJZGL69OksXLgQXdfJzMwkPj6eNWvWkJCQgNVqJSsri4KCAmbPnk14eDi5ubkAxMfHk5qayty5c9E0jRkzZqBpF85FU6dOZenSpbz66qtcdtllZGUZu1LXX8ktdjh9sld2aesuERKCSMlAbvwX8vgxRPgAo0NSFAUQUkppdBCecPDgwW4dF6hjkq7fz4PjjWj/+/RZtZF8qc+yZi/6//wX4j9/gZb9A69dx5f63FtUn4ODIc8YFP8jD9VA9Q5DC+Z1lrj4MrgkEVmyngD5jKIofk8lhgAkSzeApiFS/WMYTqTnQM0+UIX1FMUnqMQQYKTLhfygCEZbEZGDjA6nU8R1N0BIH/UQWlF8hEoMgeaTCmhsQPPhh87fJUK/Kay3Cdly2uhwFCXoqcQQYPRSGwwYCKOSjQ6lS0T6JDh5Arn1A6NDUZSgpxJDAJGNPlgwr7MuHwWD49RwkqL4AJUYAoj8oAh03acK5nWW0DREWjbs/ARZV2t0OIoS1FRiCBBSSncJjMSrEHEXGx1Ot4i0bBAa0r7B6FAUJaipxBAodn8Kh7706ZXOHRGWaEi6FmkvQuouo8NRlKClEkOAkCXroW9/RPIEo0PpES09BxqOwI5Ko0NRlKClEkMAkKeakRWliHHpiH79jQ6nZ665DsIHuLcjVRTFECoxBABZXgKnT/n1MNI3hDkEkZIJlZuRTY1Gh6MoQUklhgAgS21wUTwMv8LoUDxCpOeAqxX5YbHRoShKUFKJwc/J2gOw+zO/KJjXWWLoJXDZ5cgSmyqspygGUInBz8kSG5hMiNQMo0PxKDEhB778Avbt6rixoige1anlsZWVlaxatQpd18nOzmbKlCntXnc6nRQUFLBnzx4iIiLIzc0lJiYGgLVr11JUVISmaUybNo0xY8bQ0tLC448/TmtrKy6Xi5SUFG677TYAVqxYwY4dOwgNde86NmvWLC699FIPdjlwyNbWrwvmjUMM8I+CeZ0lxl2PfO1PyBIb4rLLjQ5HUYJKh4lB13UKCwt57LHHiIqK4tFHH8VqtXLxxWcWURUVFREWFsby5cspLS1l9erVzJkzh5qaGux2O0uWLKGhoYEnnniCp556ipCQEB5//HH69etHa2sr//3f/82YMWO4/HL3H4C7776blJQU7/U6UHxSAU2NflUwr7NEaBgieQKyfBPythmIvn2NDklRgkaHQ0nV1dXExcURGxuL2WwmLS2N8vLydm0qKirIyMgAICUlhaqqKqSUlJeXk5aWRkhICDExMcTFxVFdXY0Qgn79+gHgcrlwuVwBMz7em/SS9RBpgVFjjQ7FK8SESXCyGbml1OhQFCWodHjH4HA4iIqKavs6KiqKXbt2nbeNyWQiNDSUpqYmHA4HI0aMaGtnsVhwOByA+07kkUce4dChQ9x0003t2r3yyiu8/vrrjBo1iqlTpxISEnJWXDabDZvNBkBeXh7R0dFd6Xcbs9nc7WON5HIc5kjVFkKnTCUiNrZLx/pLn2XUROovuhht80Ys/3Fbj87lL332JNXn4OCNPhtWglPTNBYtWsSJEyd48skn2b9/P8OGDePOO+9k4MCBtLa28swzz/Dmm29y6623nnV8Tk4OOTlnisV1d89Tf90jVv/X30DXOTV2Aqe7GL8/9VlPycS19iUO79iGiDn3/rSd4U999hTV5+BgyJ7PFouF+vr6tq/r6+uxWCznbeNyuWhubiYiIuKsYx0Ox1nHhoWFkZSURGWluwTCoEGDEEIQEhJCZmYm1dXVnexi8JBSumcjXZ6EiO3+H0t/IFKz3IX1SmxGh6IoQaPDxJCQkEBtbS11dXW0trZit9uxWq3t2iQnJ1NcXAxAWVkZSUlJCCGwWq3Y7XacTid1dXXU1taSmJjIsWPHOHHiBAAtLS1s27aNoUOHAtDQ0ADQ9owiPj7ek/0NDLt2QN1B95TOACcGRcGoscgPipAuVVhPUXpDh0NJJpOJ6dOns3DhQnRdJzMzk/j4eNasWUNCQgJWq5WsrCwKCgqYPXs24eHh5ObmAhAfH09qaipz585F0zRmzJiBpmk0NDSwYsUKdF1HSklqairJye4dx5YtW8axY8cAuOSSS5g5c6YXu++fZMl66Of/BfM6S0ufhP7H38H2rXD1OKPDUZSAJ2SALC09ePBgt47ztzFJebIZ/aGfIVIy0O6e1a1z+F2fW53ov5kOI0Ziuv/Rbp3D3/rsCarPwcGQZwyKb5Hl70PL6aAYRvqGMIcgUjPh483IY0eNDkdRAp5KDH5GlqyHIcMgyFYDiwk54HIhy94zOhRFCXgqMfgR+eV+2Ps5In1S0C0IFEOGwfArVGE9RekFKjH4EVm6HkxmREqG0aEYQqRPgtoDsGen0aEoSkBTicFPyFYn8oP34JrrEBGRRodjCGFNhz593ftPKIriNSox+Itt5XD8mHtP5CAl+ocirOnIze8jT58yOhxFCVgqMfgJvcQGA6Mg6VqjQzGUSJ8Ep08iK1RhPUXxFpUY/IBsqIeqrYi0bIRmMjocYyVeBbFD3bOzFEXxCpUY/IC0bwCpIyZkGx2K4YQQ7qmr1TuQh740OhxFCUgqMfg4qevuh61XjEbEXGR0OD5BpGaCpqmH0IriJSox+Lpd2+HwIUQQP3T+LjHQAqOtqrCeoniJSgw+TpbYoH8o4to0o0PxKVp6DjQ2QNUWo0NRlICjEoMPk80nkFtLEdfdoPY8/q5RVhgw0L29qaIoHqUSgw9zF8xrcU/RVNoRZrP7WcO2cmRjg9HhKEpAUYnBh8mS9TD0Ergk0ehQfJKYMAl0XRXWUxQPU4nBR8mafbBvV1AWzOsscdHFkHAlsmS9KqynKB7U4Q5uAJWVlaxatQpd18nOzmbKlCntXnc6nRQUFLBnzx4iIiLIzc0lJiYGgLVr11JUVISmaUybNo0xY8bQ0tLC448/TmtrKy6Xi5SUFG677TYA6urqWLp0KU1NTQwfPpzZs2djNncqzIAiS21gDt6CeZ0l0ichX1gOuz9zL35TFKXHOrxj0HWdwsJC5s+fT35+PqWlpdTU1LRrU1RURFhYGMuXL2fy5MmsXr0agJqaGux2O0uWLGHBggUUFhai6zohISE8/vjjLFq0iD/84Q9UVlby+eefA/Dyyy8zefJkli9fTlhYGEVFRV7otm+TTiey7D3ENeMR4QOMDsenCWs69O2nVkIrigd1mBiqq6uJi4sjNjYWs9lMWloa5eXl7dpUVFSQkZEBQEpKClVVVUgpKS8vJy0tjZCQEGJiYoiLi6O6uhohBP369QPA5XLhcrkQQiClZPv27aSkpACQkZFx1rWCwscfwvEm9dC5E0S//u7CehUlyFPNRoej9IB0tqC//jzyyy+MDiXodThG43A4iIqKavs6KiqKXbt2nbeNyWQiNDSUpqYmHA4HI0aMaGtnsVhwOByA+07kkUce4dChQ9x0002MGDGCY8eOERoaislkOqv9d9lsNmw298rXvLw8oqOju9LvNmazudvHekvD5k20RscSfX02wuT52ki+2OeeaLnlpzSU2gj/bBv9c245Z5tA63Nn+Fufj634HSdtbyE++gDLk6vQIrp+t+xvffYEb/TZsMF7TdNYtGgRJ06c4Mknn2T//v0MHDiw08fn5OSQk3NmNXB3N8P2tc3DpeMweuWHiMm3Ud/gnWmYvtbnnpJRcRB3McfeeYMTY1LO2SbQ+twZ/tRnvdSGtL2FsKajf1TG4UWPof3qMYTWtfkx/tRnT+lJn4cMGXLO73f4U7dYLNTX17d9XV9fj8ViOW8bl8tFc3MzERERZx3rcDjOOjYsLIykpCQqKyuJiIigubkZ19dlDs7VPtC5C+ZJRJoqmNdZQgh3yZDdnyFrDxgdjtJFcv8e5OqVcOXViF/8GnH7DPikAvmv140OLWh1mBgSEhKora2lrq6O1tZW7HY7Vqu1XZvk5GSKi4sBKCsrIykpCSEEVqsVu92O0+mkrq6O2tpaEhMTOXbsGCdOnACgpaWFbdu2MXToUIQQJCUlUVZWBkBxcfFZ1wpk7oJ5G9z/QAbHGR2OX1GF9fyTbD6OvjIPwsLRfvEQQjMhMm5GXHcD8s2/ID/92OgQg1KHQ0kmk4np06ezcOFCdF0nMzOT+Ph41qxZQ0JCAlarlaysLAoKCpg9ezbh4eHk5uYCEB8fT2pqKnPnzkXTNGbMmIGmaTQ0NLBixQp0XUdKSWpqKsnJyQBMnTqVpUuX8uqrr3LZZZeRlZXl3Z+AL9n5CRz5CjHlLqMj8TtiwCC4ehzSXoSccjciCKc4+xspJfqqp8BxGO2hhYgB7qFkIQTcPQt5YC/6c0+iPZaPsATXcwOjCRkgK4MOHjzYreN8aUxSf24xsqoCbdHziD7eq43kS332JPnxZvSC/0ObNR/xnWcNgdrnC/H1Puvv/A35txcQt89Ay/nhWa/L2hr0hb+Giy9xJw5zSIfn9PU+e4MhzxiU3iFPHEdutSOum+jVpBDQRiVD5CD3NqiKT5M7q5BvvIRInoDI/o9zthEXXYz42Wz3s6PXn+/dAIOcSgw+Qm7eBK1OtXahB4TJhEjNcj+4PHruac6K8eRRB/qzf4DYixA/m33Bki/auHRE9g+QG95CLy/pxSiDm0oMPkKW2iD+MsQlCUaH4tfEhBx3Yb0PVGE9XyRbW91J4dRJtPseRfQP7fAYceu97ppYLyxH1tZ02F7pOZUYfIA8sBe+qHZXC1V6RMQNhREjkaU2VVjPB8m1L8GuHYi7ZyGGDuvUMcIcgjbzNxASgv7H3yFPnfRylIpKDD5AlqwHcwgiZaLRoQQEMWESfPUl7NphdCjKt8itHyD/vRaR8X20LhaHFJZotF88BIdqkC89rZK+l6nEYDDpbEGWFSOuTUGERRgdTkAQ1gnQr79a0+BD5FcH0Z9/Ci4dgbjt5906hxg5BvHDqcjNG5HF//JwhMq3qcRgMFn5ITQfd6/cVTxC9O2HGHe9u7DeSVVYz2jy9Gn0P/4ONBPafY8gQjqedno+4vu3wmgrcs2fkHt2ejBK5dtUYjCYLFkPlsFw5TVGhxJQxIQcaDnt3h5VMYyUErn6j3BwP9rP5yKiYnp0PqFpaDPmwEAL+jO/RzYd81CkyrepxGAgWV8Hn36MmJDd5WJhSgeGXwEXxavhJIPJ9/+N/KAIMfl2xKhkj5xThEWg3T8Pjh1F/9NipO7yyHmVM9RfIwPJ0g3A159uFY9qK6y3Zyfy4H6jwwlK8otq5CvPwshrET+43aPnFpckIu6YCTs+Qr69xqPnVlRiMIzUdXcl1auu6fHttXJuIiUTTCa1u5sB5Ikm9D/mwYBItJ//GqF5fl8Rcf1NiNRM5NtrkFVbPH7+YKYSg1E+2wb1depuwYvEgIFwzXXIsmKk02l0OEFD6jp6YT4cdaD98hFENzbc6QwhBGLqAzBkGPqfliDrD3vlOsFIJQaDyJL1EBqOuPbcG8sonqFNyIGmRk5XlBodStCQ/3odPqlA3D4DMfwKr15L9O2Ldv+joLvcD6OdLV69XrBQicEA8kQT8qMyREoGIqSP0eEEtqSxMNDCyQ1vGx1JUJCffox88y+I625AZNzcK9cUsUPQ7n0Q9n5O05+X9co1A51KDAaQH250F8xTw0heJ0wmRFo2LR+VIRvqOz5A6TbZUI/+3JMQN9Rd8uICxfE8TYxNQ9w4hZPvvJ9NuAoAACAASURBVIH+4cZeu26gUonBALJkPQxLQAwbbnQoQUFMyHYX1rNvMDqUgCVbW9Gf+T20nEa7fx6iX/9ej0H86B5CRl6DfLEA+aWaidYTndrmqrKyklWrVqHrOtnZ2UyZMqXd606nk4KCAvbs2UNERAS5ubnExLhn2qxdu5aioiI0TWPatGmMGTOGI0eOsGLFCo4ePYoQgpycHG6+2X3b+dprr7FhwwYGDHA/sLrjjjsYO3asJ/tsKLl/NxzYi7jzPqNDCRoiZgghSdfiLLUhb/5pr36SDRbyb8/D7s8QMx9GXBRvSAzCbCby109wZM496Ct/h7ZgMaJfx9VblbN1eMeg6zqFhYXMnz+f/Px8SktLqalpX/q2qKiIsLAwli9fzuTJk1m9ejUANTU12O12lixZwoIFCygsLETXdUwmE3fffTf5+fksXLiQd999t905J0+ezKJFi1i0aFFAJQX4VsG8624wOpSg0j/nFjh8CD7fbnQoAUcvL0Ha/o7I/gHauOsNjcVkiUab+TB8VYt8frkqttdNHSaG6upq4uLiiI2NxWw2k5aWRnl5ebs2FRUVZGRkAJCSkkJVVRVSSsrLy0lLSyMkJISYmBji4uKorq5m0KBBDB/uHkbp378/Q4cOxeEI/I1VZMtp5Icb3eOhYeFGhxNU+qVmQv9QtabBw2RtDfKF5ZBwpXvfBB8grhiN+NHdyC2lyA1vGR2OX+pwKMnhcBAVFdX2dVRUFLt27TpvG5PJRGhoKE1NTTgcDkaMGNHWzmKxnJUA6urq2Lt3L4mJiW3fe/fdd9m0aRPDhw/nnnvuITz87D+iNpsNm81d7iAvL4/o6O5tFm42m7t9bFedfP/fHGs+QeTkn9C3l655Lr3ZZ19hNpvpf/2NnCz+F5ZfPYoWBInZ2++zPHWS+ucWIfr2JWpeHqZo4xdqftNneddMGmv2cPr1VQwYM44+V442OjSv8cb73KlnDN5y6tQpFi9ezL333ktoqHss8MYbb+TWW28FYM2aNbz44os88MADZx2bk5NDTs6ZWT3d3Qy7NzcPd/1rLUTHcixuGMLADcuDdcP009Z0+Pc6jryzDm3i94wOyeu8+T5LKZF/WoKs2YeW+z80oIEP/E59u89y6v2wdxcNv38U7f8tdS94DEA9eZ+HDBlyzu93OJRksViorz8zza++vh6LxXLeNi6Xi+bmZiIiIs461uFwtB3b2trK4sWLuf766xk/fnxbm4EDB6JpGpqmkZ2dze7du7vQTd8lDx9SBfOMdukIGHqJGk7yAFn8L+TmjYj/uBMxcozR4ZyTCA1Hu28enDiuiu11UYd/oRISEqitraWuro7W1lbsdjtWq7Vdm+TkZIqLiwEoKysjKSkJIQRWqxW73Y7T6aSuro7a2loSExORUrJy5UqGDh3KLbfc0u5cDQ0Nbf+/efNm4uONmeHgadJeBEIgUrONDiVotRXW27cLWbPP6HD8ltz7OXLNn2C0FXHzT40O54LEsOGIqffBpx8j33zF6HD8RodDSSaTienTp7Nw4UJ0XSczM5P4+HjWrFlDQkICVquVrKwsCgoKmD17NuHh4eTm5gIQHx9Pamoqc+fORdM0ZsyYgaZpfPbZZ2zatIlhw4bx8MMPA2empb788svs27cPIQSDBw9m5syZ3v0J9AKpu5B2G4wcg4gabHQ4QU2Mz0S+/gKy1Ia4vXs7iQUz2XQMfeXvYaAFbcYcv7j71SbkoFd/ivzna8iEKxBXjzM6JJ8nZIDM5zp48GC3juuN8XZZtRX9qd+i/fI3CGu6V6/VGcH6jOGbPrtW5sHOT9D+8HyPdhPzdZ5+n6XuQl/2v+6f3SO/R1w6osNjetv5+ixbTqPn/Qbq69Aey0cMjjMgOu8w5BmD0nOy1AbhEXDN+I4bK16npU+C402wbbPRofgV+fZrsP0jxH/OxBeTwoWIPl8X25Ogr1TF9jqiEoOXyePHkJVliPEZAf3p1K+MHAODotHVQ+hOk1VbkW+/ikjJRNxwk9HhdIsYHIc2PRf270a++pzR4fg0lRi8TJYVQ2srIn2S0aEoXxOaCZGWBds/QjpUDf+OyPrD6IWLYcgwxF0P+HVJETFmPOL7P0Fuehdd1c46L5UYvEhK6Z4aeUki4uJLjQ5H+RYxIQekdM8WU85LOp3u4nitrWj3zUP07Wt0SD0mfngXXDEa+fIfkTV7jQ7HJ6nE4E1fVMOXX6i7BR8kBse5/ziU2pC6bnQ4Pkv+tRD2fo427b8QcUONDscjhMmENvMhCA1H/2MesvmE0SH5HJUYvEiW2iCkjyqY56NE+iQ48hV8XmV0KD5J/3Aj8r1/Im6cghibZnQ4HiUGDEL75W/gyFfozz+liu19h0oMXuIumLcJkZyGCA0zOhzlHMTYVOgfplZCn4P8cj/yxQJIHIn40T1Gh+MVYsRIxE/uhY/KkP9eZ3Q4PkUlBi+RW+1w8oQaRvJhok9fxPgbkFs/QDYfNzocnyFPNaOv/B3064/2y4cRZkNLqnmVmPRDGJuGfOMFpLpzbKMSg5fIEhsMjoMRSUaHolyASJ8Ezhbk5k1Gh+ITpJTIFwrgq1q0mQ8jBkZ1fJAfE0K494uOjkN/dhGysaHjg4KASgxeIOtqYecniAk5flEyIKgNS4CLL3UncgVZ9DayogTxo7sRVwRuqepvE/1D0e6fBydPoD/7B6RLFdtTf7W8QJZuAKEhUrOMDkXpgLuw3iT4ohp5ILinLsrqT5F//TNccx3iez82OpxeJS6+FHHXLPh8O3LtS0aHYziVGDzMXTBvAyRdi7AE12Y4/kqMnwhms3sWWZCSx46iP/MHsAxGm57r14vYuktLzUTc8D3ku28gK8uMDsdQKjF42vZKOFrvrsej+AURPgAxJgVZVox0Oo0Op9dJ3YX+p8Vw/BjafY8gQgN/d7vzEf/5c7gkEf3PTyHruleYMxCoxOBhesl6CB8A16jSvv5EpE+CE03Iyg+NDqXXyb+/4t5Eaup9iGEJRodjKBHSB+2+R0AI9D/+Htly2uiQDKESgwfJpkb4eLO70JhZFczzK1ddDZbBQbemQW4rR/7jNcSEHHWX+zURHYv287lQsxf5l5VGh2OITk1QrqysZNWqVei6TnZ2NlOmTGn3utPppKCggD179hAREUFubi4xMe6NwdeuXUtRURGapjFt2jTGjBnDkSNHWLFiBUePHkUIQU5ODjfffDMAx48fJz8/n8OHDzN48GDmzJlDeLh/3NrKsmJwqYJ5/shdWC8b+Y81yPo6RJTxG9t7mzzyFXphPsRfhrjzl0aH41PEaCviltuRb69BT7gK7fobjQ6pV3V4x6DrOoWFhcyfP5/8/HxKS0upqalp16aoqIiwsDCWL1/O5MmTWb16NQA1NTXY7XaWLFnCggULKCwsRNd1TCYTd999N/n5+SxcuJB333237Zzr1q1j9OjRLFu2jNGjR7NunX+sSGwrmHfZ5Yihw4wOR+kGMcG97WowFNaTzhb3TmxSuovj9fH/4nieJn7wnzByDPIvzyC/CIy95zurw8RQXV1NXFwcsbGxmM1m0tLSKC8vb9emoqKCjIwMAFJSUqiqqkJKSXl5OWlpaYSEhBATE0NcXBzV1dUMGjSI4cOHA9C/f3+GDh2Kw+EAoLy8nIkTJwIwceLEs67ls/btgoP73XsKK35JRMfClVcHRWE9+epz8EW1ewZSzEVGh+OThGZC+/mvISISfWUe8kTwrI7vcCjJ4XAQFXVm9WNUVBS7du06bxuTyURoaChNTU04HA5GjDiz05PFYmlLAN+oq6tj7969JCYmAtDY2MigQYMAGDhwII2NjeeMy2azYbO5pxfm5eURHd29qaFms7nbx37bsb8WcrJvP6K/9yM0H6+N5Kk++5PO9vnk93/EsSW/ZUDtF/T18wkE5+vzyff+xbFN7xL6o7uIyJlsQGTe4/Hf7ehoWn6zkIbHHsC8+mkGzsvzuUWr3vj3bGgRlFOnTrF48WLuvfdeQkNDz3pdCHHe+dQ5OTnk5Jz5dN7dPU89sS+uPH0afdO/EWPTcDSfhOaTPTqftwX7ns8XIhNHQWgYjf/8G9rQy3ohMu85V59lzT73ENIVozl10084HWC/B1753Y6+CPHTGbS8+iyHVz+L9v1bPXv+HjJkz2eLxUJ9fX3b1/X19VgslvO2cblcNDc3ExERcdaxDoej7djW1lYWL17M9ddfz/jxZ/ZCjoyMpKHBXa+koaGBAQMGdLaPhpFbSuHUSTWMFABESB/E+InuwnoBNnQgm0+g/zEP+oej/eIhhMlkdEh+Q2RNRoy7Hrn2ZeRn24wOx+s6TAwJCQnU1tZSV1dHa2srdrsdq9Xark1ycjLFxcUAlJWVkZSUhBACq9WK3W7H6XRSV1dHbW0tiYmJSClZuXIlQ4cO5ZZbbml3LqvVysaNGwHYuHEj48b5/u28LF0PMRepgnkBQqRPglYn8sNio0PxGCkl+gvL4Mghd3G8yEFGh+RXhBCIe34FsUPcxfYa6js+yI91mBhMJhPTp09n4cKFzJkzh9TUVOLj41mzZg0VFRUAZGVlcfz4cWbPns3bb7/N1KlTAYiPjyc1NZW5c+eycOFCZsyYgaZp7Ny5k02bNlFVVcXDDz/Mww8/zNatWwGYMmUK27Zt48EHH+STTz45a2qsr5FfHYTPt7sL5gVhGYFAJIYlwLDhAVUiQ65fB1s/QPzkZ4jL1QeY7hD9+ruL7bWcdhfba201OiSvETJAti46eLB7y9d7Oiapv/Ei8p030P5Q6DclitUzho7pRW8jX3kW7f/l++1q4G/6LD/fjr54AYwZ756aGsAfYHrjd1vfvAn53JOInB+i3T7Dq9fqDEOeMSjnJ10u95z30cl+kxSUzhHjM8Ac4vcroWVjA/qziyA6Du1nDwZ0Uugt2nU3IDInI21vup8vBiCVGHpi+1ZodKBNUA+dA40IC0eMTUV+uBHpbDE6nG6RrlZ3Ujh5HO3+eWqLWQ8St02Hyy5Hf34Z8tCXRofjcSox9IBesh4iIuFq339ArnSdmJADzSeQWz8wOpRuOb76Wfi8CjH1AcTFlxodTkAR5hB3sT2z2b347fQpo0PyKJUYukkea4Bt5YjUrIDeEzeoXXk1RMX45UNoWVlG89qXETd8Dy1NbRjlDcIyGO0XD8HB/ciXnyZAHtcCKjF0m7tgnkutXQhgQtPcdw2ffow88pXR4XSarKtF//NTmBOudO8voHiNGHkt4gd3uPfy2PiO0eF4jEoM3eAumGeDhCsRF8UbHY7iRSItG4Rwb9fqB2TLafciNiEY+JuFiJA+RocU8MTk22BUMnLNc8i9uzo+wA+oxNAde3ZC7QH3p0kloImowXDVGKTdhtR9f5N4+ZdnoGYv2ow5mFRxvF4hNA1txhwYMMj9vOH4MaND6jGVGLpBltqgbz/EuHSjQ1F6gUjPAccR+NS3SyHo7/8bWWpDTL4NoSZE9CoRPgDtvnlwrAG9MN/vq/OqxNBF8tRJ5Ob3EdYJiH5nF/5TAo8YkwJhET69pkHu3+2+W7jqGsR/3GF0OEFJXDYCcfvPoWoL8p+vGR1Oj6jE0EVyix1On1S7tAURERKCSMlAVpb55DCBbD7urpgaPsBdHE9TxfGMIiZ+3/278vdXkDs+MjqcblOJoYtkyXqIGwoJVxkditKLxIQcaG1FfrjR6FDakbqO/uel4DiMdt8jiIhIo0MKakIIxF0PwEXx6M89iXQcNjqkblGJoQvkoRqo3qEK5gUhEX8ZXJKILFnvU/PV5btvwMebET+djki40uhwFED07ecuttfair7y98hWp9EhdZlKDF0gSzeApiFS1YKhYCTSc6BmH+z3jf1/5WfbkGtfRljTEVm3dHyA0mtE3MVo9z4Iez9H/nWV0eF0mUoMnSRdLuQHRTDaqmrZBylx3Q0Q0scnHkLLo/XuOkixQxA/+5W6g/VBInkCIueHyKK30TdvMjqcLlGJobM+qYDGBjT10DloidBvCuttQracNiwO2dqK/swiaDntLo6nZsf5LPGTn0HiVcgXC5AH9xsdTqepxNBJeqkNBgyEUclGh6IYSKRPgpPGFtaTb7zgftZ19yzEkGGGxaF0TJjNaDN/A336up83nPLt/eC/0anqb5WVlaxatQpd18nOzj5rVzWn00lBQQF79uwhIiKC3NxcYmJiAFi7di1FRUVomsa0adMYM2YMAE8//TRbt24lMjKSxYsXt53rtddeY8OGDW17Pd9xxx2MHTvWI53tLtn4dcG8SVNUwbxgd/koiI51DyelZPT65eUWO3L9m4jMm9HGT+z16ytdJwZFof3iIfT8x5EvFsAvHvL5ob8O7xh0XaewsJD58+eTn59PaWkpNTU17doUFRURFhbG8uXLmTx5MqtXrwagpqYGu93OkiVLWLBgAYWFhehfrwjMyMhg/vz557zm5MmTWbRoEYsWLTI8KQDuZwu6rgrmKWcK6+38BHn4UK9eWx76Ev35p+CyyxE/NX7nMKXzxFXXIKZMRZa/jyz6h9HhdKjDxFBdXU1cXByxsbGYzWbS0tIoLy9v16aiooKMjAwAUlJSqKqqQkpJeXk5aWlphISEEBMTQ1xcHNXV1QCMHDmS8PBwz/fIw6SU7hIYiVch4i42OhzFB4i0rK8L6/VeOW55+hT6yjwwm9F++QgiJKTXrq14hvjeT+Dqcci//hm5+zOjw7mgDsdFHA4HUVFntq2Miopi165d521jMpkIDQ2lqakJh8PBiBEj2tpZLBYcDkeHQb377rts2rSJ4cOHc88995wzgdhsNmw29z/MvLw8oqOjOzzvuZjN5gse2/LpNhoOfcmAX/2M/t28hq/pqM+ByKN9jo6mYcx4WsuKiZo2G2Hy7kpjKSXHlj3BqYP7GfjfS+h7RecWV6r32ffoDz+B46HpyOeexLJ4FZoHZjh6o88+N2B+4403cuuttwKwZs0aXnzxRR544IGz2uXk5JCTc2Zop7ubYXe0kbb+j9ehb3+OX3ENJ7y8yXhv6Y0N032Np/ssx09E/6iMI+/bEF6ekKBvfAdZ/A7iB3fQdHECTZ3sh3qffZP8xcPoeb/h8O/no+X+tsclTHrS5yFDhpzz+x0OJVksFurr69u+rq+vx2KxnLeNy+WiubmZiIiIs451OBxnHftdAwcORNM0NE0jOzub3buNW0wkTzUjK0oQ49IR/fobFofig66+DsIj3Nu7epHctwv56rOQdC3iltu9ei2ld4hLEhB3/tK9AdRbrxodzjl1mBgSEhKora2lrq6O1tZW7HY7Vqu1XZvk5GSKi4sBKCsrIykpCSEEVqsVu92O0+mkrq6O2tpaEhMTL3i9hoaGtv/fvHkz8fHGbYQjy0vg9ClVME85i7uwXiZUbkY2NXrlGvL4MXdxvAGD0Gb8GqGp2eWBQqRPQqRlI99eg/xki9HhnKXDoSSTycT06dNZuHAhuq6TmZlJfHw8a9asISEhAavVSlZWFgUFBcyePZvw8HByc3MBiI+PJzU1lblz56JpGjNmzED7+pd76dKl7Nixg6amJu677z5uu+02srKyePnll9m3bx9CCAYPHszMmTO9+xO4AFlqg4viYfgVhsWg+C6RPglp+zvyw2JEzg89em6p6+iF+XDUgfZIHiJigEfPrxhLCAF33ofcvwe9cAnaY0sQ0bFGh9VGSF+qCNYDBw8e7NZx5xufk7UH0P97FuLWaWg3/ain4fkUfxiH9TRv9dm18NfgbEF7fJlH56brb69Bvrkaced9aJk3d+sc6n32fbLuIPr/zYWYIWiP/L5bs80MecYQrGSJDUwmRGqG0aEoPkykT4Ivv4B9ntvrV+74CPn3vyDGT0RkfN9j51V8j4gZgjYtF76oRq55zuhw2qjEcA6ytfXrgnnjEANUwTzl/MS466FPH/cHCQ+QjsPozy2Gi+LdJS98fIWs0nPi2hTETT9CbnwHvew9o8MBVGI4t08qoKlRFcxTOiRCwxBjJyDLNyFP96ywnmx1oj/zB3A63cXx+vbzUJSKrxM/ugcuT0K+tAJZs8/ocFRiOBe9ZD1EWmCU8eU4FN/nLqzXjNxq79F55OvPw56daPfOVqvsg4wwmdzF9vqHof8xD3my2dB4VGL4Dnm0Hj7ZgkjL9PqKViVAXJ4Eg+N6tE+DvnkTcsNbiJz/QFjTPRic4i9E5CC0mQ/DkUPozy8zdKdAlRi+Q37wHkgdMUENIymdI4RwF9b7vApZ1/XZcbL2gLvqZsKViJ/c6/kAFb8hLh+F+PHPYKsdafu7YXGoxPAtUkr3Q8TLkxCx557GpSjnItKyQWju7V+7QJ46if7HPOjT110cT5V1D3rixilwbQry9VXIXTsMiUElhm/btQPqDro//SlKF4hBUTBqLNK+AelydeoYKSXypRVw6Eu0XzzkPocS9IQQaPf+F0THoj/zB+Sxho4P8jCVGL5FlqyHfv0RyROMDkXxQ1p6Dhx1wPatnWov3/sHcvMmxA/vRFx1jZejU/yJCA1Du28eNB9Hf/bJTn/Y8BSVGL4mTzYjt5QirrtBTRNUuufqcRAR6d4GtgNy92fI1/4MV49DfP/WXghO8Tci/jLEXfe7N4V6c3WvXlslhq/J8veh5bQaRlK6TZhDECkZ8PFm5LGj520nm46hP/sHGGhBmz5HFcdTzktLy0ZcfyPyX68jKz/svev22pV8nCxZD0OGwWWXGx2K4sdE+iRwuZDnWcEqdRf6nxbDsUb3IrYw39/FUDGWuGMmDEtA//PSXttOViUGQH65H/Z+7i6Fq0oQKD0ghgyD4VcgS2znnIcu31oDOz5C3DETccmFS9ArCoAI6YN23yMgQF+Zh2zp2Qr7zlCJAZCl68Fkdg8DKEoPiQk5UHsA9uxs931ZtQX5jzWI1CzE9TcaFJ3ij8TgOLQZc2H/HuQrz3r9ekGfGKTT6V7Uds11iIhIo8NRAoC7sF5f934eX5P1deh/WgJDL0FMvV/dmSpdJq4eh7j5p8iS9Z2a4NATQZ8YTleUwvFj7qmGiuIBon8oInkCcvP7yNOnkE6nexGb7kK7bx6ib1+jQ1T8lPjhnXDl1cjVK5H793jtOp1aZllZWcmqVavQdZ3s7GymTJnS7nWn00lBQQF79uwhIiKC3NxcYmJiAFi7di1FRUVomsa0adMYM2YMAE8//TRbt24lMjKSxYsXt53r+PHj5Ofnc/jwYQYPHsycOXMID/feA7qTG96CgVGQdK3XrqEEH5E+CflBEbKiFPZ9Dl9Uo93/qFpRr/SI0Exov3gI/Ylc9JV5aI8tAaI9fp0O7xh0XaewsJD58+eTn59PaWkpNTU17doUFRURFhbG8uXLmTx5MqtXu+fc1tTUYLfbWbJkCQsWLKCwsBBd1wHIyMhg/vz5Z11v3bp1jB49mmXLljF69GjWrVvniX6ek2yop+WjDxFp2QhNFcxTPGjESIgZgvzb88jifyFu/BFibKrRUSkBQAwYiPbLR8BxGH3VU14pttdhYqiuriYuLo7Y2FjMZjNpaWmUl5e3a1NRUUFGRgYAKSkpVFVVIaWkvLyctLQ0QkJCiImJIS4ujurqagBGjhx5zjuB8vJyJk6cCMDEiRPPupYnSfsG0HXEhGyvXUMJTkIIRHoONDW6a2/9+B6jQ1ICiEi8CnHrvVD5Iae7WJ+rMzocSnI4HERFnanhEhUVxa5du87bxmQyERoaSlNTEw6HgxEjRrS1s1gsOByOC16vsbGRQYPcu6YNHDiQxsbGc7az2WzYbO4HMHl5eURHd/126uTFw2id9AMiRo7u8rH+zGw2d+vn5c+M6LP+46mcEBA6+aeYDKiDpN7nwCZvn87p+MsIuz6Hfl+PxHiKT5dyFEKcd/ZGTk4OOTlnHhh3azPsa1KIzr7FrzYP9wR/2zDdEwzr8/du5bRLggHXVu9zELjiavrperf7PGTIuZ95dTiUZLFYqK+vb/u6vr4ei8Vy3jYul4vm5mYiIiLOOtbhcJx17HdFRkbS0OCuJtjQ0MCAAQM6ClFRFEXxoA4TQ0JCArW1tdTV1dHa2ordbsdqtbZrk5ycTHFxMQBlZWUkJSUhhMBqtWK323E6ndTV1VFbW0ti4oVXe1qtVjZu3AjAxo0bGTduXDe7piiKonSHkJ14pL1161ZeeOEFdF0nMzOTH//4x6xZs4aEhASsVistLS0UFBSwd+9ewsPDyc3NJTY2FoA33niD9957D03TuPfee7n2Wve00KVLl7Jjxw6ampqIjIzktttuIysri6amJvLz8zly5EiXpqsePNj1nbMgCG89UX0OFqrPwaEnfT7fUFKnEoM/UImh81Sfg4Pqc3DwRmII+pXPiqIoSnsqMSiKoijtqMSgKIqitKMSg6IoitJOwDx8VhRFUTwj6O8Y5s2bZ3QIvU71OTioPgcHb/Q56BODoiiK0p5KDIqiKEo7pt/+9re/NToIow0fPtzoEHqd6nNwUH0ODp7us3r4rCiKorSjhpIURVGUdlRiUBRFUdrx6Y16vK2yspJVq1ah6zrZ2dlMmTLF6JB67MiRI6xYsYKjR48ihCAnJ4ebb76Z48ePk5+fz+HDh9tVrZVSsmrVKj766CP69u3LAw884LdjtLquM2/ePCwWC/PmzaOuro6lS5fS1NTE8OHDmT17NmazGafTSUFBAXv27CEiIoLc3FxiYmKMDr/LTpw4wcqVKzlw4ABCCO6//36GDBkS0O/z22+/TVFREUII4uPjeeCBBzh69GhAvc9PP/00W7duJTIyksWLFwN0699vcXExb7zxBgA//vGP27Zf7hQZpFwul/zVr34lDx06JJ1Op3zooYfkgQMHjA6rxxwOh9y9e7eUUsrm5mb54IMPygMHDsiXXnpJrl27Vkop5dq1a+VLL70kpZRyy5YtcuHChVLXdblz50756KOPGhZ7T7311lty6dKl8ne/WK3qeQAABNVJREFU+52UUsrFixfLkpISKaWUzzzzjHz33XellFK+88478plnnpFSSllSUiKXLFliTMA9tHz5cmmz2aSUUjqdTnn8+PGAfp/r6+vlAw88IE+fPi2ldL+/7733XsC9z9u3b5e7d++Wc+fObfteV9/XpqYmOWvWLNnU1NTu/zsraIeSqquriYuLIzY2FrPZTFpaGuXl5UaH1WODBg1q+8TQv39/hg4disPhoLy8nIkTJwIwceLEtr5WVFRwww03IITg8ssv58SJE2076PmT+vp6tm7dSnZ2NgBSSrZv305KSgoAGRkZ7fr8zaenlJQUqqqqkH42B6O5ufn/t3f3Lq2zYRzHvxJUqNW2ieigiK9LK7Wg4uSgg5MuDoLi4KgFxdHJP0AQHazUQdDVRcFdrEMRfC3i+1CdxKIppUVraJtnKPY5fXwGe46c0vT+bE0Due78aO/kJuHi+vqa/v5+IN3ruKKiwvA5p1IpNE0jmUyiaRpWq9VwOdvt9i89aHLN9fz8HKfTidlsxmw243Q6OT8//3YNRbuUpKoqivJvg3ZFUbi/v89jRT8vFAoRDAZpbW0lEolgs9kAsFqtRCIRIH0efm2erigKqqpm9i0UGxsbjI+P8/7+DkA0GsVkMiFJEpBuP6uqKpCdvSRJmEwmotFoQbWRDYVCVFVVsbq6yuPjI83NzUxMTBg6Z1mWGRoaYmpqirKyMjo6OmhubjZ0zp9yzfW//2+/npfvKNo7BqOLx+MsLi4yMTGByWTK+q6kpISSkpI8VfbzTk5OsFgsBblm/ruSySTBYJCBgQEWFhYoLy9nZ2cnax+j5RyLxTg6OsLj8bC2tkY8Hs/pKtgo/kauRXvHIMsyr6+vmc+vr6/IspzHin5OIpFgcXGR3t5eenp6ALBYLITDYWw2G+FwOHPVJMtyVvenQjwPt7e3HB8fc3Z2hqZpvL+/s7GxwdvbG8lkEkmSUFU1M67P7BVFIZlM8vb2RmVlZZ5HkRtFUVAUhba2NiC9VLKzs2PonC8uLqipqcmMqaenh9vbW0Pn/CnXXGVZ5urqKrNdVVXsdvu3j1e0dwwtLS08PT0RCoVIJBL4/X66urryXdYf03Udr9dLXV0dg4ODme1dXV34fD4AfD4f3d3dme0HBwfous7d3R0mk6mglhcAxsbG8Hq9eDweZmdnaW9vZ2ZmBofDweHhIZB+QuMz387OTvb39wE4PDzE4XAU3JW11WpFUZRMS9uLiwvq6+sNnXN1dTX39/d8fHyg63pmzEbO+VOuubpcLgKBALFYjFgsRiAQwOVyfft4Rf3m8+npKZubm6RSKfr6+hgeHs53SX/s5uaG+fl5GhoaMj+C0dFR2traWFpa4uXl5cvjbuvr6wQCAcrKynC73bS0tOR5FL/v8vKS3d1d5ubmeH5+Znl5mVgsRlNTE9PT05SWlqJpGisrKwSDQcxmM7Ozs9TW1ua79Jw9PDzg9XpJJBLU1NTgdrvRdd3QOW9tbeH3+5EkicbGRiYnJ1FV1VA5Ly8vc3V1RTQaxWKxMDIyQnd3d8657u3tsb29DaQfV+3r6/t2DUU9MQiCIAhfFe1SkiAIgvD/xMQgCIIgZBETgyAIgpBFTAyCIAhCFjExCIIgCFnExCAIgiBkERODIAiCkOUfC6ZQWC23jhoAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "id": "2MMOua0WwwdO", "outputId": "5b3a0bc1-b1f8-4157-b292-4f81795f459c" }, "source": [ "plot_lr(triangular(250, 0.005, 'exp_range', gamma=0.999))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deUBV1733//faBxwYRA+IRMVE0AwOEeWoQDSiEk1iBjPZzFVzmza25pI8t7c26W36u328zdPEoRFT015qJtuYSTOZGAnihAOoOCYqDkmIGISDCqJMe/3+OIaEODCdwz7D9/WXwB4+i4182WuvvZbSWmuEEEKIcwyrAwghhPAuUhiEEEI0IoVBCCFEI1IYhBBCNCKFQQghRCNSGIQQQjQSZHUAdzl69Gir9ouKiqK0tNTNabybtDkwSJsDQ1va3LNnzwt+Xu4YhBBCNCKFQQghRCNSGIQQQjQihUEIIUQjUhiEEEI0IoVBCCFEI1IYhBBCNCKFQTSbPnUCc0MWMlO7EP7Nb15wE55nvvDf8GUhqtflcEV/q+MIITxE7hhEs+jiIviy0PXv7ZstTiOE8CQpDKJZ9IdLoWMn6HU5umCT1XGEEB4khUE0SRcXofPWosZOQo2eAEe/Qn/burmphBDer1nPGAoKCli8eDGmaTJ+/HgmT57c6Ou1tbVkZGRw6NAhwsPDSU9PJzo6GoBly5aRnZ2NYRhMmzaNhISEhv1M02TWrFnY7XZmzZoFQElJCfPnz6eiooK4uDhmzpxJUJA8CrGS/vAN6NARNWEy1FSj3/g7umATauKdVkcTQnhAk3cMpmmSmZnJU089xbx589iwYQNFRUWNtsnOziY0NJQFCxYwadIklixZAkBRURG5ubnMnTuXp59+mszMTEzTbNhvxYoV9OrVq9GxXn/9dSZNmsSCBQsIDQ0lOzvbHe0UraSLv0bnrXPdLYRHoCKjoU88ert0Jwnhr5osDIWFhcTExNCjRw+CgoJISUkhLy+v0Tb5+fmkpqYCkJSUxO7du9Fak5eXR0pKCsHBwURHRxMTE0NhoesBZllZGdu2bWP8+PENx9Fas2fPHpKSkgBITU0971yifekPl567W7ij4XNq6Eg4tA99wmlhMiGEpzRZGJxOJ5GRkQ0fR0ZG4nQ6L7qNzWYjJCSEioqK8/a12+0N+7788ss8+OCDKKUavl5RUUFISAg2m+287UX700e/ct0tjJuECu/S8Hk1NBm0Ru/cYmE6IYSnWNJ5v3XrViIiIoiLi2PPnj2tOkZWVhZZWVkAPPvss0RFRbXqOEFBQa3e11c1t80nXnmBmo6dibr3EYwuXRs+ryMjKYvphW33Nrrd+aAno7qNXOfAIG120zGb2sBut1NWVtbwcVlZGXa7/YLbREZGUl9fT1VVFeHh4eft63Q6sdvt5Ofnk5+fz/bt26mpqeHMmTO88MILzJw5k6qqKurr67HZbA3bX0haWhppaWkNH7d2BSNZ8enC9NGvMDd8hrrxLpw1dfCj7c1rh1P/2Ycc//orVOcQT8Z1C7nOgUHa3DKtXsEtPj6e4uJiSkpKqKurIzc3F4fD0WibxMREcnJyANi0aRMDBw5EKYXD4SA3N5fa2lpKSkooLi6mX79+3H///SxatIiFCxeSnp7OoEGDePzxx1FKMXDgQDZtcj3YzMnJOe9con24ni10co1EugA1NAnq69C78ts5mRDC05q8Y7DZbEyfPp3Zs2djmiZjx44lNjaWpUuXEh8fj8PhYNy4cWRkZDBz5kzCwsJIT08HIDY2luTkZJ588kkMw+CRRx7BMC5dix544AHmz5/PG2+8Qd++fRk3bpx7WiqaTX/zFTp/Peqmu1FhXS68UdxVEB4BBZthxPXtG1AI4VFK+8mMaEePtu6FK7n1PJ/59+fRO/Iwnv37xQsDYL6agc5bhzH3dVRwsCeiuo1c58AgbW6ZVnclicCiS46i89ajUm+6ZFGAc91JZ8/AFzvbKZ0Qoj1IYRCN6I/fgaAg1ITbm9746iHQsTN6+0bPBxNCtBspDKKBdh5Hb1yNGnUDqku3JrdXwcGowYnogs1os74dEgoh2oMUBtFAr1wG6JbNgZQwEipOwqH9HsslhGhfUhgEAPpUOXrdp6ikVFRk92bvpwY7wBYkcycJ4UekMAgA9Kr3oa4OdePdLdpPhYTC1YPR2zfKkp9C+AkpDAJ9uhKdswLluA4V06vpHX5EDU2G48fg6FceSCeEaG9SGAR69Ydw9gzq5pbdLXxHDRnhOo6MThLCL0hhCHD67Bl01gcwZASqd99WHUN1tUP81eitUhiE8AdSGAKcXvsJnK7AuKl1dwvfUYnXQdFhdIks+SmEr5PCEMB0bQ360+Vw9bWo+KvbdCw1LNl1TLlrEMLnSWEIYDo3G06WY9x8T5uPpSKj4Yr+6K0b3JBMCGElKQwBSpv1rruFy/vB1de65ZhqWAp8WYguK3HL8YQQ1pDCEKgKNkPJUYwb72y0vGpbqMTvupNy3XI8IYQ1pDAEIK015ifvQvcYOPdswB1UdE/o3Re9TQqDEL5MCkMAqt1bAIf3oyZMRhk2tx5bJabAwS/Q5WVNbyyE8EpSGALQ6eX/hLAuqJTxbj+2SkwB5GU3IXyZFIYAo7/5ipr8Dahxt6A6dHT78dVlsXBZrDxnEMKHSWEIMPrTZdCxE2rszR47h0pMgQN70afKPXYOIYTnSGEIILq8DL15DZ3Tbmly2c62UIkpoE309s0eO4cQwnOkMAQQ/dn7YJqE3nqvZ0/U6wqIvkxGJwnho6QwBAhddRq95hOU4zpsPXp69FxKKdddwxc70ZWnPHouIYT7SWEIEHrdStfU2i1ZtrMN1LAUME30ji3tcj4hhPsENWejgoICFi9ejGmajB8/nsmTJzf6em1tLRkZGRw6dIjw8HDS09OJjo4GYNmyZWRnZ2MYBtOmTSMhIYGamhqeeeYZ6urqqK+vJykpiSlTpgCwcOFC9u7dS0hICAC//OUvueKKK9zY5MCj62rRWe/DNUNQl8e3z0kv7weR0a7RSdeltc85hRBu0WRhME2TzMxMfve73xEZGclvf/tbHA4HvXv3btgmOzub0NBQFixYwIYNG1iyZAlPPPEERUVF5ObmMnfuXMrLy/njH//IX/7yF4KDg3nmmWfo1KkTdXV1/P73vychIYErr7wSgIceeoikpCTPtTrA6Lz1cMKJ8dOZ7XZOpRRqWDI6+yN01WnXEqBCCJ/QZFdSYWEhMTEx9OjRg6CgIFJSUsjLy2u0TX5+PqmpqQAkJSWxe/dutNbk5eWRkpJCcHAw0dHRxMTEUFhYiFKKTp06AVBfX099fb3b5usRjWmt0VnvwWWxMHBYu55bJV4H9XXondKdJIQvafKOwel0EhkZ2fBxZGQkBw4cuOg2NpuNkJAQKioqcDqd9O/fv2E7u92O0+kEXHciv/nNbzh27BgTJ05stN2//vUv3n77bQYNGsQDDzxAcHDwebmysrLIysoC4NlnnyUqKqol7W4QFBTU6n19Qc3u7ZR/dYjwx35DSPfuQPu1WdtTKLVHEbwrn663tH1q77bw9+t8IdLmwOCJNjfrGYMnGIbBc889x+nTp3n++ef56quv6NOnD/fffz9du3alrq6Ol156iffee4+77z5/dbG0tDTS0r7vuy4tLW1VjqioqFbv6wvq33kNwsI5PchB1bl2tmebdUIS1WtXcvzrr1CdQ9rlnBfi79f5QqTNgaEtbe7Z88IjFJvsSrLb7ZSVfT8hWllZGXa7/aLb1NfXU1VVRXh4+Hn7Op3O8/YNDQ1l4MCBFBQUANCtWzeUUgQHBzN27FgKCwub2UTxY7qkGHZsRl1/k0emv2gONXwU1NXK6CQhfEiThSE+Pp7i4mJKSkqoq6sjNzcXh8PRaJvExERycnIA2LRpEwMHDkQphcPhIDc3l9raWkpKSiguLqZfv36cOnWK06dPA1BTU8POnTvp1asXAOXlrmkUvntGERsb6872BhSd/SEYNo9Of9GkuKuhayQ6f711GYQQLdJkV5LNZmP69OnMnj0b0zQZO3YssbGxLF26lPj4eBwOB+PGjSMjI4OZM2cSFhZGeno6ALGxsSQnJ/Pkk09iGAaPPPIIhmFQXl7OwoULMU0TrTXJyckkJiYC8MILL3DqlOulqMsvv5xHH33Ug833X7rqNHp9Fmr4aFRXe9M7eIgyDJRjFHr1R+iqSlRImGVZhBDNo7TW2uoQ7nD06NFW7eevfZLmp8vQby3G+K95qD6N311o7zbrQ/sw//Rr1NR/x7jO/VN9N4e/XudLkTYHBkueMQjfo+vr0dkfwZWDzisKluh7petlt/x1VicRQjSDFAZ/VLAJykow0m6zOglw7mU3xyj4fIfMnSSED5DC4IfMVe+51nMeMtzqKA3U8NFQX4/evsnqKEKIJkhh8DP68H44+AVq/K1uX8+5TfrEQfcYdJ50Jwnh7aQw+Bmd9T50DkFZ9JD3YpRSrruGL3ahT52wOo4Q4hKkMPgRfaIMvXUDatQNqE7WvWV8MWr4KNfKbrKAjxBeTQqDH9FrV4JpolItfKHtUnpdATG90fkbrE4ihLgEKQx+QtfVugrDoERU9GVWx7kgV3fSKNi/G33CaXUcIcRFSGHwE3prLpwsxxg3yeool6Qco0BrV14hhFeSwuAn9OqPIPoyGDDU6iiXpHr2gV6Xy8tuQngxKQx+QH950DVEdezNKMP7L6lyjILCz9HO41ZHEUJcgPf/FhFN0qs/gg4dUSneNUT1YtTw0QDSnSSEl5LC4ON05Sn0lrWo5LE+M3Op6tET+sTJy25CeCkpDD5Ob8iC2hrUWO9+6PxjyjEaDu9Hl35rdRQhxI9IYfBh2qxHr14BVw1G9brc6jgtohzXAchdgxBeSAqDL9uZ75pF1cfuFgBU9xiIvxq9eY3VUYQQPyKFwYeZqz+CblGQMNLqKK2iRo6Bb75EFx2xOooQ4gekMPgoXVwEewtQY25E2bxoFtUWUInXgWGgt6y1OooQ4gekMPgonbMCgoJQoydYHaXVVJeuMCABvWUt2jStjiOEOEcKgw/S1WfRG1ejhl3n+uXqw9SIMVBWAoe+sDqKEOIcKQw+SOetgzOnUWNutDpKm6mhI6FDB/Rm6U4SwltIYfBBes0ncFks9B9gdZQ2U51CUENGovPXo+vqrI4jhEAKg8/RXx6EIwdQY25CKWV1HLdQI66HylPweYHVUYQQQFBzNiooKGDx4sWYpsn48eOZPHlyo6/X1taSkZHBoUOHCA8PJz09nejoaACWLVtGdnY2hmEwbdo0EhISqKmp4ZlnnqGuro76+nqSkpKYMmUKACUlJcyfP5+Kigri4uKYOXMmQUHNihkQ9NpPoEMHVHKq1VHcZ9AwCAlDb16DGuywOo0QAa/JOwbTNMnMzOSpp55i3rx5bNiwgaKiokbbZGdnExoayoIFC5g0aRJLliwBoKioiNzcXObOncvTTz9NZmYmpmkSHBzMM888w3PPPcef//xnCgoK2L9/PwCvv/46kyZNYsGCBYSGhpKdne2BZvsmfabK9ctz+GifmRepOVRQMMpxHbpgM7r6rNVxhAh4TRaGwsJCYmJi6NGjB0FBQaSkpJCXl9dom/z8fFJTUwFISkpi9+7daK3Jy8sjJSWF4OBgoqOjiYmJobCwEKUUnTp1AqC+vp76+nqUUmit2bNnD0lJSQCkpqaed65ApjfnQPVZ1JibrI7idmrEGKg+i96xxeooQgS8JvtonE4nkZGRDR9HRkZy4MCBi25js9kICQmhoqICp9NJ//79G7az2+04na4lHU3T5De/+Q3Hjh1j4sSJ9O/fn1OnThESEoLt3AtbP9z+x7KyssjKygLg2WefJSoqqiXtbhAUFNTqfduT1hrn+lUQdyV2R3Kbni94Y5u1/XpKI7sTVLCJbjff6fbje2ObPU3aHBg80WbLOu8Nw+C5557j9OnTPP/883z11Vd07dr8MflpaWmkpaU1fFxaWtqqHFFRUa3etz3pg19gfnkQ9dAMysrK2nQsb22zThxFzWfvc/zIIVRYF7ce21vb7EnS5sDQljb37Nnzgp9vsivJbrc3+kVUVlaG3W6/6Db19fVUVVURHh5+3r5Op/O8fUNDQxk4cCAFBQWEh4dTVVVFfX39RbcPVHrNJ9Cxs2sEj59SI8dAfb0s4COExZosDPHx8RQXF1NSUkJdXR25ubk4HI1HjiQmJpKTkwPApk2bGDhwIEopHA4Hubm51NbWUlJSQnFxMf369ePUqVOcPn0agJqaGnbu3EmvXr1QSjFw4EA2bdoEQE5OznnnCkT6dAU6fz0qaQyqU4jVcTwnti9cFoveIjOuCmGlJruSbDYb06dPZ/bs2ZimydixY4mNjWXp0qXEx8fjcDgYN24cGRkZzJw5k7CwMNLT0wGIjY0lOTmZJ598EsMweOSRRzAMg/LychYuXIhpmmitSU5OJjExEYAHHniA+fPn88Ybb9C3b1/GjRvn2e+AD9Abs12L8fjhQ+cfUkqhRlyPfm8J2nkcZe9udSQhApLSWmurQ7jD0aNHW7Wft/dJaq0xfz8DQsKw/fY5txzTm9usS4oxn/456q6fYtx4l9uO681t9hRpc2Cw5BmDsNj+PXDsG7+YF6k5VPRlEHcVelOO1VFEO9Nnz2Au/V908ddWRwl4Uhi8nF67EjqHohJHWR2l3ajksa4FfL4+bHUU0U60aWJmzkNnvY+5ZBF+0pHhs6QweDF9uhK9LRc1cgyqY0er47Qb5RgFtiDXsxUREPTy16Bgk2tiyH274IudVkcKaFIYvJjenAN1tajRN1gdpV2psC5wrQO9eQ363NBl4b/M3Gz0x++grp+I8cR/Q7cozGWvyV2DhaQweCmtNXrdKugTj+oTb3Wcdmckj4NTJ2CvzLjqz/TBL9CvZcDV16Lu+zkquAPqlp/A4f2wU6bDsYoUBm/11UEoOowaFVh3Cw0GJ0JouHQn+TFdXob51z9BtyiMX/wGdW4WZZUyHrrHYC5fIku+WkQKg5fS6z6F4A6okf77pvOlqKBg1PDRrhlXz1RZHUe4ma6tcRWFs2cwfvk0KjS84WsqKAh1231QdFjegreIFAYvpKur0VvWohJT/Gp67ZZSyWOhtga9dYPVUYQbaa3Rr/8VDu/HmP4Eqtfl522jRlzvegv+/SXynMkCUhi8kN66Ac5UoUZNsDqKtfpeCT16oTeutjqJcCOd/SE69zPULfeihiVfcBtl2DBufwCOfeMahCHalRQGL6Q3rILoy+DKgVZHsZRSynXXsH83uvRbq+MIN9Cf70C/mQkJI1G33nvpjYclQ5949Pv/QtfVtk9AAUhh8Dr62Dewfw9q1A1+s6ZzW6ikVAB5E9oP6OPHMP/2Z+jRy9WFZFz6149SCmPyg1BWgl6f1U4pBUhh8Dp6QxYYBipZJg8EUJHRcOUg9MbVMq7dh+nqs5gv/glM0/WwuXMzZwkeNAzir0Z/tBRdU+3ZkKKBFAYvouvq0LmfwWAHqqusQ/EdlTwWSo66xrYLn+N62PwifHME42f/gepx4YnbLkQphXHHQ3DC6VqTRLQLKQzeZFc+nDqBMTrAHzr/iEq8DoI7yENoH6XXfILelIO67T7UoMQW76+uGgzXDEGveEuGLrcTKQxexFy/CiLs0Ir/PP5MdQ5BDU1C561D18pDSF+iD+9HL/07DEpE3Tyl1ccx7ngYKk+hP13mxnTiYqQweAldXga7tqJSxqFsNqvjeB2VPBZOV7juqoRP0BWnMBf9P4iwY/zbk00+bL4U1bc/yjEK/ely9MlyN6YUFyKFwUvoTatBm6jr0qyO4p2uSYAuXTGlO8knaLMe83/nwKlyjMdmNXqzubXU5Aehvg794VI3JBSXIoXBC2it0bnZ0G9Aix7MBRJls6FGjoFd+eiKU1bHEU3QHy6FvdtR9z2KuryfW46pevREjZ6IXrcS/W3rVmwUzSOFwRsc3g/HilApMkT1UlTKeNdfjFvWWB1FXILetRX94VJU8jjU6IluPba65ScQFIxe/rpbjysak8LgBXTuZ9Chg2v0jbgo1fsKuLyfvOzkxXRZCWbmXOh1OeqBx9z+kqaK6Ia64XZ0/nr04QNuPbb4nhQGi+naGnTeOtTQZFRIqNVxvJ66Ls016+ZXB62OIn5E19Vh/u05qK9zPVfw0KqDasIdENYF891X5KVHD5HCYDFdsAWqTks3UjOpEde7uhLkrsHr6PeWwKF9qId/hYr23LMy1TnE1aX0xU5ZyMlDpDBYTG/Mhm5RcPW1VkfxCSo0zPVOw+Y16Noaq+OIc/TubehP3kGNnoAxfLTHz6euvxGiemC+87Is5uMBQc3ZqKCggMWLF2OaJuPHj2fy5MmNvl5bW0tGRgaHDh0iPDyc9PR0oqOjAVi2bBnZ2dkYhsG0adNISEigtLSUhQsXcuLECZRSpKWlcfPNNwPw5ptv8tlnn9GlSxcA7rvvPoYNG+bONnsNfcIJu7ehbroLZci7C82lrktzvexWsAU1fJTVcQKePuHE/Mc813OFn/ysXc6pgoNRkx9E/+8cV1fsyDHtct5A0WRhME2TzMxMfve73xEZGclvf/tbHA4HvXv3btgmOzub0NBQFixYwIYNG1iyZAlPPPEERUVF5ObmMnfuXMrLy/njH//IX/7yF2w2Gw899BBxcXGcOXOGWbNmce211zYcc9KkSdx2222ea7WX0JtzXO8uJI+1OopvueZasEehc7NACoOltFnvethcfQbj0dkee65wIWr4aPTKd9HLXkMPS0EFB7fbuf1dk11JhYWFxMTE0KNHD4KCgkhJSSEvr/Ei3fn5+aSmpgKQlJTE7t270VqTl5dHSkoKwcHBREdHExMTQ2FhId26dSMuLg6Azp0706tXL5xOp/tb58Ua3l2IvxoV07vpHUQDZdhcs8/u2Y52llodJ6DpFW/BFztR9/0c1bNPu55bGQbGXVNd03LnrGjXc/u7Ju8YnE4nkZGRDR9HRkZy4MCBi25js9kICQmhoqICp9NJ//79G7az2+3nFYCSkhIOHz5Mv37fvwSzcuVK1q5dS1xcHA8//DBhYecvb5mVlUVWlusB5LPPPktUVFRz2nueoKCgVu/bFrWFn+M8+hXhv/hPQtr5/Fa12Z3qJt1N2UdvErJzM6F3/7TJ7f2hzS3l6TbX7NlO+Qdv0On6CXS5/V5r1g8ZcwPlOR9Ru+JN7LfcI9fZXcd069Fa6OzZs8yZM4epU6cSEuKan33ChAncfffdACxdupRXX32VGTNmnLdvWloaaWnfTx9RWtq6vxyjoqJavW9bmCvegaBgTl+TQFU7n9+qNrtVcCe4chCVq96naszNTf5S8os2t5An26wrT2E+/3uI6kHNPdMpKyvzyHmaleX2B9H/nU7pqy8S/ctZcp1boGfPC48ea7IryW63N7roZWVl2O32i25TX19PVVUV4eHh5+3rdDob9q2rq2POnDmMHj2akSNHNmzTtWtXDMPAMAzGjx/PwYP+N15d19ait6xDDU1ChZx/NySaR12XBiXFcGCv1VECitYa85UMqDyJ8eh/ojo1c9EdD1G9r0CNSkOvXkHd0a8tzeIvmiwM8fHxFBcXU1JSQl1dHbm5uTgcjkbbJCYmkpOTA8CmTZsYOHAgSikcDge5ubnU1tZSUlJCcXEx/fr1Q2vNokWL6NWrF7fcckujY5WXfz9z4pYtW4iNjXVDM73Mzjw4XSHvLrSRSkyBTp1dq96JdqPXr4KCTag7HkJdHm91HADU7Q9AUDCVr75odRS/0GRXks1mY/r06cyePRvTNBk7diyxsbEsXbqU+Ph4HA4H48aNIyMjg5kzZxIWFkZ6ejoAsbGxJCcn8+STT2IYBo888giGYfDFF1+wdu1a+vTpw69//Wvg+2Gpr7/+OkeOHEEpRffu3Xn00Uc9+x2wgJn7GXS1w4AEq6P4NNWxk2tkypa16Pt+ZvlfroFAf3sU/cbf4ZohqLTbrY7TQEV0Q914J9XvLcG4fiLqykFWR/JpSvvJO+VHj7ZutsX27nvWp05g/noq6obJGHdPbbfz/pA/9bfrg19gPvufqJ/OxBh1w0W386c2N5e726zr6jD/32+gpBjjmRdQdu96yKurq+GZGZhhERhPPd+m9R98iSXPGIR76bz1YJqu4Zai7eKugphe6A2fWZ3E7+kP34AjBzAemuF1RQFAdexI2AO/gC8L0VvWWh3Hp0lhaGd6cw7E9kX1at8x3/5KKeV6CF24F32syOo4fksf2Ite8bZrKm2H975U2GnMROgTj172Krqm2uo4PksKQzvSx76Bw/tRI1OtjuJXVPI4sNnQ61ZZHcUv6arTrrebI7uj7vPuZ37KMDCmTAdnKTrrfavj+CwpDO1Ib14DSrlmCBVuoyK6wbXD0Ruz0XW1VsfxO/pffwNnKcYjT6I6e/8DfnXVYBgyAv3x2+hTsj50a0hhaCdaa1c30tXXorpFNrm9aBlj9ESoOAkFm62O4lfMvPXoTatRk6ag+l1jdZxmM+6eCrU16Pf+aXUUnySFob0c2gfHj0k3kqcMTAB7d8x1n1qdxG/ok+XoJX+FK/qjJk2xOk6LqJjeqNSb0es+RX91yOo4PkcKQzvRm3MguANqWLLVUfySMmyoUTfA3gL08WNWx/F5WmvM1xZC9VmM6emoIEtnz2kVdet9EBqO+cbfZKW3FpLC0A50XZ1rzviEkT7RR+ur1HVpoAzXm7miTfTG1bBjC+qOB1GX+ebsAyo0DDX5QTiwF52/3uo4PkUKQ3vYsx0qK6QbycOUPQoGDUNv+AxdX291HJ+lnaWut5v7DUCl+fa6KGr0DRDbF/32YtcLcKJZpDC0A705B8LCYeBQq6P4PeP6iXDSCbvymt5YnEdrjfnqAqivw5j27z6/sqAybBj3PuoavrryHavj+AwpDB6mz1ShCzajHKN9sp/W5wx2QFc75lp5CN0aet2nsGc76u6pqOjLrI7jFurKga45tT55F11WYnUcnyCFwcP09o1QW4NKSrU6SkBQNhsqJQ12b0M7j1sdx6fo0m/Rb/7DNaR6zE1Wx3ErdfdUUKDfWmx1FNipEk0AACAASURBVJ8ghcHD9KYc6B7jmtNHtAs1Kg20iV4v03E3lzZNzJdfAAXG1Mf9bgI6Ze+OuvFu9NYN6H27rY7j9fzr6nsZfaLMtR7uyFRrlj0MUKp7DAwYit6wCm3KQ+jm0DkrYN8u1JRHUJHRVsfxCDXhDte7Lm/8XX4umiCFwYP0lrWgNWrkGKujBBzj+gngLHWNCBOXpEu/Rb/7Kgwc6noXxE+pjh0x7pkGRYfR8gzqkqQweJDevMb11mhML6ujBJ4hIyA8Qh5CN8H1ItuLgMJ46Jf+f2ebeB1cOQj93uvoylNWp/FaUhg8RB8rgq8OoUbKhHlWUEHBrqVTd25Bn3BaHcdr6Y2rYe921F0P+20X0g8ppTDuexSqTqOXvW51HK8lhcFD9JZ1rplUvXjuen+nRk8E05Q3oS9CnypHL/1f6HeN341CuhTV+wrUuFvR61aiDx+wOo5XksLgAVprdN46uHIQqqvMpGoV1aMnXDPE9QtA3oQ+j/7n36CmGuOnM/1uFFJT1G33QZdumEv+Kg+iLyCwfhray9eH4VgRavhoq5MEPCP1JnCWUr011+ooXkVv24jeugF1y09QMb2tjtPuVOcQ1D3TXMuAynOo80hh8ACdtw5sNtSwFKujiCEjoaudM5+8a3USr6FPV2L+cxH07ouaeKfVcSyjRlwPVw1GL3sNXXHS6jheRQqDmzV0I12TgArvYnWcgKdsNtToidRs34wuKbY6jlfQby+GipOuF9kCeJoWpRTG/T+H6jPod162Oo5XadZPRUFBAYsXL8Y0TcaPH8/kyZMbfb22tpaMjAwOHTpEeHg46enpREe7RjgsW7aM7OxsDMNg2rRpJCQkUFpaysKFCzlx4gRKKdLS0rj55psBqKysZN68eRw/fpzu3bvzxBNPEBYW5uZme9ChfVBWgrrtfquTiHPU6Anoj95Er/0Edfc0q+NYSn++A71+FerGu1CXx1sdx3KqZx9U2u3ole+iR92A6jfA6kheock7BtM0yczM5KmnnmLevHls2LCBoqKiRttkZ2cTGhrKggULmDRpEkuWLAGgqKiI3Nxc5s6dy9NPP01mZiamaWKz2XjooYeYN28es2fPZuXKlQ3HXL58OYMHD+aFF15g8ODBLF++3APN9hydtw6CglFDk6yOIs5R3SLpOGI0ekMWurbG6jiW0TXVrsV3oi9D3Xqv1XG8hrrlJ9AtCnPJSzJI4ZwmC0NhYSExMTH06NGDoKAgUlJSyMtrPKVxfn4+qampACQlJbF792601uTl5ZGSkkJwcDDR0dHExMRQWFhIt27diIuLA6Bz58706tULp9M11jwvL48xY1xvCo8ZM+a8c3kzbda7FgS51iEL8niZzjfeAZUV6K0brI5iGb3iLTh+DOPBGagOHa2O4zVUp84YP3nE9UZ0zgqr43iFJruSnE4nkZHfD7mMjIzkwIEDF93GZrMREhJCRUUFTqeT/v37N2xnt9sbCsB3SkpKOHz4MP369QPg5MmTdOvWDYCuXbty8uSFHwplZWWRleWaJO3ZZ58lKiqqycZeSFBQUKv3/bGanfmUnywnYtwkOrnpmJ7gzjb7Clt0NBU9+2CsX4X9lnusjtMufnid674+QtnKd+k0ZiIRo8dbnMxzWvuzrSfcxolNq6l9/590u+FWbHbf+f/hif/Plj55Onv2LHPmzGHq1KmEhJz/F7ZS6qKv6KelpZGWltbwcWlpaasyREVFtXrfHzOzPoSOnam44ioq3XRMT3Bnm31FVFQU5qgbqH8zk+Pb81Cxfa2O5HHfXWetNWbG/0CHTtTc9oBfX/u2/Gzru6ej//AryhY9h/Hor92czHPa0uaePXte8PNNdiXZ7XbKysoaPi4rK8Nut190m/r6eqqqqggPDz9vX6fT2bBvXV0dc+bMYfTo0YwcObJhm4iICMrLywEoLy+nSxffGNmj62rR2zaiEkagOsptujdSKeMguAM652Oro7QrnZsN+3ej7vopqktXq+N4LdWjJ+qme9B569C7t1odx1JNFob4+HiKi4spKSmhrq6O3NxcHA5Ho20SExPJyckBYNOmTQwcOBClFA6Hg9zcXGpraykpKaG4uJh+/fqhtWbRokX06tWLW265pdGxHA4Ha9asAWDNmjUMHz7cTU31sL0FcLoCNVzmRvJWKjTctZLX5hz0mSqr47QLXXEK/fY/XNNe+PHMqe6ibrobYnpjvv5XdPVZq+NYxvaHP/zhD5fawDAMYmJiWLBgAZ988gmjR48mKSmJpUuXcvbsWXr27EmfPn1Yv349//znPzly5AiPPvooYWFhREREUFlZyUsvvcT69euZPn06PXv2ZN++fbz88stUV1ezatUqVq1aRVRUFJdddhlxcXG89957vPPOO1RWVjJt2jQ6dOjQZEMqKipa9Q0ICQmhqqrtvyT0h0vBWYp68DGvXyfXXW32JQ1t7mp3PWDsFoXq27/pHX1YSEgIpzPnwpEDGDP/CxXRzepIHtfWn21ls6F6X47Oeh/q61ADvH+d9ra0OTw8/IKfV1pr3ZZQ3uLo0aOt2s8d/e26phrzyYdRI0ZjPPyrNh2rPQTqM4aG/vb/+yTU1WL8YYFfTzPd5duvKf/dL1E33oVx10+tjtMu3PWzbb6agd6QhfH0XFSfODck8xxLnjGIZtiVD9VnZG4kH6CUQqXeBEe/ggN7rY7jMbqullOLnoPIaNQt8s5CS6m7pkJouKtABOAke1IY3MDMWwfhEXDlIKujiGZQI8ZASCh69UdWR/EYvXIZ9UVHMB74hQyGaAUVGob6yb+5JtlbHXjvNkhhaCNdfRZ25aMSU1A27362IFxUx46oUTegt+Winf7XpabLStAr3qRjcipqsKPpHcQFqRHXw8Ch6GWvo53HrY7TrqQwtNWufKipQSVeZ3US0QIq9WbQGr3mE6ujuJ35ZiagCJ/2uNVRfJpSCuOBx0DXY/7rb1bHaVdSGNpI528414000OooogVU9xi4drhrER8/mj9J79kO2zaibr4HW/cYq+P4PNU9BnXrfVCwGb1to9Vx2o0UhjbQ1dXoXfmoYcleP0RVnM8YdwtUnHRNfOgHdF0t5ht/c02SN+EOq+P4DZV2O/S+AvNfL6GrTlsdp11IYWiL3flQUy3dSL7qmiFwWSw6+yP8YdS2znofjn2Dce+jqOBgq+P4DRUU5BqGfvJEwKzbIIWhDb7vRpLRSL5IKYUaNwm+LHSto+HDdHmZ6yXLISNQgxOtjuN3VN8rURNuR69dif58h9VxPE4KQyvp6mr0zjzU0GQZjeTDVNJY6ByK/uwDq6O0iX7rH1Bfj/GTf7M6it9St90P0T1d7zacPWN1HI+SwtBau7e6upEc0o3ky1Snzqjr0lxDV0+UNb2DF9L7dqHz1qFuusv1UF14hOrQEWPq41BWgl7+utVxPEoKQyvprRsgrIt0I/kBNfZmME30mpVWR2kxXVfnGkoZGY268S6r4/g91X8AauwkdPaHaD9+c14KQyvomnPdSMOkG8kfqOjLYFAieu0n6Npaq+O0iM75CL75EuPef5NV2dqJuuMhsHfHfGUBuqba6jgeIYWhNXZvheqzMhrJjxjjb4VTJ9Bb11sdpdl0xUn0+/+CgUNhyMimdxBuoTp1do1S+vYb1/ffD0lhaAWdf64b6arBVkcR7nLNEIjphc72nfmT9HtLoPosxk9+5tezxHojNSABNXoC+tPl6MMHmt7Bx0hhaKGGbqShSdKN5EeUYaDGToLD+9E+MHRVFx1Br/0UNXYS6rLeVscJSOruaRDRDfOVF9B1vtUF2RQpDC21e5urG0lGI/kdlTIOOod4/dBVrbVrPqTOIahbZUptq6iQUIwHZ8A3X6I/esvqOG4lhaGF9NZcCAuHq661OopwM9UpxDXrav56755Nc2cefL4Dddv9qNALr8Al2ocaMhyVNBb98VvoLwutjuM2UhhaQNfWoHdukZfa/JgafytovPZZg66rxXzzHxDTGzXmRqvjCEDd+zMI74qZOc9vJmSUwtASe7bB2TMyGsmPqchoVGKKa+oDL3y7Va9eASVHMaY8ggoKsjqOwLWoj/HTmVD8td+8+CaFoQX0to0QEiajkfycSrsNzpxGb/jM6iiN6IpT6A/egEHDZD4kL6MGDUONuRG96j30/t1Wx2kzKQzNpOvq0Du2uPoU5S81v6bir4b4q9Gfve9V6/3q9/8J1Wcw7pludRRxAeruaRDVA3PxX9Bnq6yO0yZSGJpr/26oOo0almx1EtEOjBtuh+PHYEee1VEA0N98iV7zCSr1ZlTPPlbHERegOnXGmJ7umkvpzX9YHadNpDA0k96+ETp0hAFDrY4i2kNCEkRGY65abnWSc8NT/yHDU32A6jcANeEO9LpP0bvyrY7Tas3qEykoKGDx4sWYpsn48eOZPHlyo6/X1taSkZHBoUOHCA8PJz09nejoaACWLVtGdnY2hmEwbdo0EhISAHjxxRfZtm0bERERzJkzp+FYb775Jp999hldunQB4L777mPYsGFuaWxradNEb98MgxJlPpoAoWw21Phb0W9moo8cQF3R37owe7bD3u2onzyCCutiXQ7RLOr2B9C7t2K+koHx/y3wySHFTd4xmKZJZmYmTz31FPPmzWPDhg0UFRU12iY7O5vQ0FAWLFjApEmTWLJkCQBFRUXk5uYyd+5cnn76aTIzMzFNE4DU1FSeeuqpC55z0qRJPPfcczz33HOWFwUADu+Hk07pRgowatQN0KkzetX7lmXQZj3mOy9D9xhU6s2W5RDNp4KDXV1KlSfRSxZZHadVmiwMhYWFxMTE0KNHD4KCgkhJSSEvr3G/a35+PqmpqQAkJSWxe/dutNbk5eWRkpJCcHAw0dHRxMTEUFjoeglkwIABhIWFub9FHqC3bQRbEGqww+oooh2pziGoURPQW9ejnaWWZNAbc6DoCOqOh1FBslynr1B94lG33IvOW4fpg2uKN9mV5HQ6iYyMbPg4MjKSAwcOXHQbm81GSEgIFRUVOJ1O+vf//hbcbrfjdDqbDLVy5UrWrl1LXFwcDz/88AULSFZWFllZWQA8++yzREVFNXncCwkKCrrkvlprynZuwXatg259Lm/VObxNU232R61tc/09D1Oa/QGdNmUT/vAMDyS7OF1dTekH/8TW7xrsN97e4ony5DpbSz/0c8o/L6BuySK6OZKxeWgRJU+02evGXU6YMIG7774bgKVLl/Lqq68yY8b5/yHT0tJIS0tr+Li0tHV/0UVFRV1yX110GPPYN5g3TG71ObxNU232R61usxEMQ5OoWrmMs+NuRXXq7P5wF2F+/Da67DhMe4KyspavLifX2Xr6p4+j/zud0uf/C+P//F+U4f4ZE9rS5p49e17w8012Jdnt9kY/lGVlZdjt9otuU19fT1VVFeHh4eft63Q6z9v3x7p27YphGBiGwfjx4zl48GBTET1Kb9sESqESRliaQ1jHuGEyVJ1G57bfC2+64hT647dhyAjUVbJKoK9S0Zeh7n8U9u9Bf/yO1XGarcnCEB8fT3FxMSUlJdTV1ZGbm4vD0bivPTExkZycHAA2bdrEwIEDUUrhcDjIzc2ltraWkpISiouL6dev3yXPV15e3vDvLVu2EBsb24pmuY/evhH6XYPq0s3SHMI6DS+8rXoPXd8+L7zpj5bC2bMYdz7cLucTnqOSx6GGj0Z/8C/04f1Wx2mWJruSbDYb06dPZ/bs2ZimydixY4mNjWXp0qXEx8fjcDgYN24cGRkZzJw5k7CwMNLT0wGIjY0lOTmZJ598EsMweOSRRzAMVy2aP38+e/fupaKigl/84hdMmTKFcePG8frrr3PkyBGUUnTv3p1HH33Us9+BS9Alxa4Hf1MesSyD8A7GxDsxX/wf9LZc1PDRHj2XLjmKzlmBGn2DvMzmB5RS8OBj6INfYP79eYzf/6VduyRbQ2mttdUh3OHo0aOt2u9S/XPmymXotxdj/OnvqKgebYnnVbytH7Y9tLXN2jQxf/9L6NgJ43dzPbpimvnSn9E78zBmv4Tqeumu10uR6+xd9P7dmM//DpUyFmPqv7vtuJY8YwhkevtG6BPnV0VBtI4yDNTEO+Crg/DFTo+dRx/ah85fj5pwR5uKgvA+6spBqJvuRm/4DJ3v3WuLS2G4CH3CCQe/QA2Vl9qEi0oa61rK8RPPPETUWrteZguPQE2c3OT2wveoW++FvldivrbQNeLMS0lhuAhdsBlA3nYWDVRwMGr8bbC3AP2VB0bL7cqH/XtQt96H6hTi/uMLy6mgIIx/exLqTcx/zPWq2Xt/SArDRejtG6FHL7jM2lFRwruoMRNd02SsXObW42rTxFz2umvqi9E3uPXYwruo6J6o+84NYfXStaKlMFyAPl0J+3ahhiV59CGj8D0qJAx1/Y2udaGPH3PbcXX+eig67FrHWaa+8HsqZRxq5Bj0B2+g9+2yOs55pDBcgN6ZB/X18nxBXJAafysoA53lnsn1dF0d+r0l0Oty1AjPDoUV3kEphXrwMYi+DPPvc9CnTlgdqREpDBegCzZD10i4/NIv44nApOxRrr/21n+KrjjV5uPp3M+gpBhj8oMemTJBeCfVKQTj5/8JpyswM+ehz8087Q2kMPyIrq2BPdtQCSNQhnx7xIWpiXdATQ06Z0WbjqNra1zrOMddBUNk2pVAo2L7ou79Gezdjl75rtVxGshvvh/7YhdUn0XJf1JxCapnHxgyAp39Ibq6utXH0atXwIkyjDsekudZAUpdP9E1Zcby19EH9lodB5DCcB5dsBk6doarrrU6ivByxsQ7ofIUesOqVu2vz1ShP34LBiSgrpaft0CllEI99EvXUrJ/fx5d2fbuybaSwvAD2jTRO7fAoKGoYBkZIprQ7xrX5HqfLkfX1bV4d73qPaiswJj8kAfCCV+iOodg/Pw3UHEC8x/zsXqmIikMP/TVQTjhRA0ZaXUS4QOUUhg33wNlJegta1q0r644hV61HIYlo/pauJ608Brq8njUPdNhV77rZ8NCUhh+QBdsBsNADU60OorwFYMdENsX/fHbLXqLVX/yNlRXY9z+gAfDCV+jxk6CYSnod1+19HmDFIYf0Du2QL8BqLAuVkcRPqLhruHYN7BtY7P20eVl6OyPUEmpMq22aEQphfHTmRDZwzXL7snypnfyACkM5+jjx1xrL8hoJNFSw5IhphfmR281q29Yf/wWaNM1oZoQP6JCQjEemwVnKjH/9udWPb9qKykM5+ideQCoBHm+IFpGGTbUTXdD0WHXRHiXoJ3H0es+RaWMR3locXjh+1TvK1AP/co1n9KyV9v9/FIYztE7tsBlsajoy6yOInyQGjHGNdxwxaXvGvTHb4MGNWlKO6YTvshISkWNneQa9bZ1Q/ueu13P5qV0VSXs341KkG4k0ToqKAg18U44+AVcZFI0XXYcvW4ValQaKjK6nRMKX6SmTIf4qzEXv4AuLmq380phAPSura5J82SYqmgDNSrNtZDPigtPpaxXvAkK1M33tHMy4atUUDDGo/8JHTpg/vVP6LNn2uW8UhgAdmyB8Ajoe6XVSYQPU8EdUDdMhs93oA/ta/Q1XfotekMWatQElL27RQmFL1L2KIyf/Qcc+wb9yoJ2efkt4AuDrq1F796KGiKT5om2U2NuhNDw8+4a9Iq3QCnXQ2ohWkhdMwR1x0OudUDcNN37pQT8b8KavQVwpkpGIwm3UJ06u9Zr2LEFXXQYcA2F1rmfoa6/EWWPsjih8FXqxjthaBL67cXoz3d49FwBXxiqt6yDDh3g6iFWRxF+Qo27xbX854q3AdAfLQXDhrrpLouTCV+mlMKYng49ernebyj91mPnCmrORgUFBSxevBjTNBk/fjyTJ09u9PXa2loyMjI4dOgQ4eHhpKenEx3tGnWxbNkysrOzMQyDadOmkZCQAMCLL77Itm3biIiIYM6cOQ3HqqysZN68eRw/fpzu3bvzxBNPEBYW5q72NqK1pjpvHQwYiurY0SPnEIFHhYa5hhl+8g56ZCp642rU2EmorpFWRxM+TnUKwfjV05iz/w/mwv/BmPVnj5ynyTsG0zTJzMzkqaeeYt68eWzYsIGiosbDprKzswkNDWXBggVMmjSJJUuWAFBUVERubi5z587l6aefJjMzE/PcKkWpqak89dRT551v+fLlDB48mBdeeIHBgwezfLkHJ5P6+jDm8W/lbWfhduqGydChI+aiP4EtCHWj3C0I91DRPV0Po785gn7lBY88jG6yMBQWFhITE0OPHj0ICgoiJSWFvLy8Rtvk5+eTmpoKQFJSErt370ZrTV5eHikpKQQHBxMdHU1MTAyFhYUADBgw4IJ3Anl5eYwZMwaAMWPGnHcud9I7trgeCF7r8Ng5RGBS4V1QqTdBXR0q9SZUV7vVkYQfUYMSXQ+j89ZRveEztx+/ya4kp9NJZOT3t8CRkZEcOHDgotvYbDZCQkKoqKjA6XTSv//3Uwrb7XacTuclz3fy5Em6desGQNeuXTl58uQFt8vKyiIrKwuAZ599lqiolj/UOxN7OXVptxIeF1jTHgcFBbXq++XLrGizef+jVAbZCJvyCEaXiHY9N8h19nf6wZ9z9vI4QkffQCc33zU06xmDVZRSF13uMC0tjbS0tIaPS0tLW36ChGSi0m5t3b4+LCoqStrcXiY/jLOmFiw4t1znADBgGJ21bnWbe/bsecHPN9mVZLfbKSsra/i4rKwMu91+0W3q6+upqqoiPDz8vH2dTud5+/5YREQE5eWuqWbLy8vp0kWmwBZCiPbUZGGIj4+nuLiYkpIS6urqyM3NxeFo3CefmJhITk4OAJs2bWLgwIEopXA4HOTm5lJbW0tJSQnFxcX069fvkudzOBysWeNaDWvNmjUMHz68lU0TQgjRGko345H2tm3beOWVVzBNk7Fjx3LnnXeydOlS4uPjcTgc1NTUkJGRweHDhwkLCyM9PZ0ePXoA8O6777J69WoMw2Dq1KkMHToUgPnz57N3714qKiqIiIhgypQpjBs3joqKCubNm0dpaWmLhqsePXq0Vd+AgLv1RNocKKTNgaEtbb5YV1KzCoMvkMLQfNLmwCBtDgyeKAwB/+azEEKIxqQwCCGEaEQKgxBCiEakMAghhGjEbx4+CyGEcI+Av2OYNWuW1RHanbQ5MEibA4Mn2hzwhUEIIURjUhiEEEI0YvvDH/7wB6tDWC0uLs7qCO1O2hwYpM2Bwd1tlofPQgghGpGuJCGEEI1IYRBCCNGIVy/U42kFBQUsXrwY0zQZP348kydPtjpSm5WWlrJw4UJOnDiBUoq0tDRuvvlmKisrmTdvHsePH280a63WmsWLF7N9+3Y6duzIjBkzfLaP1jRNZs2ahd1uZ9asWZSUlDB//nwqKiqIi4tj5syZBAUFUVtbS0ZGBocOHSI8PJz09HSio6Otjt9ip0+fZtGiRXz99dcopXjsscfo2bOnX1/nDz/8kOzsbJRSxMbGMmPGDE6cOOFX1/nFF19k27ZtREREMGfOHIBW/f/Nycnh3XffBeDOO+9sWH65WXSAqq+v17/61a/0sWPHdG1trf6P//gP/fXXX1sdq82cTqc+ePCg1lrrqqoq/fjjj+uvv/5av/baa3rZsmVaa62XLVumX3vtNa211lu3btWzZ8/Wpmnqffv26d/+9reWZW+rDz74QM+fP1//6U9/0lprPWfOHL1+/XqttdYvvfSSXrlypdZa608++US/9NJLWmut169fr+fOnWtN4DZasGCBzsrK0lprXVtbqysrK/36OpeVlekZM2bo6upqrbXr+q5evdrvrvOePXv0wYMH9ZNPPtnwuZZe14qKCv3LX/5SV1RUNPp3cwVsV1JhYSExMTH06NGDoKAgUlJSyMvLszpWm3Xr1q3hL4bOnTvTq1cvnE4neXl5jBkzBoAxY8Y0tDU/P5/rr78epRRXXnklp0+fblhBz5eUlZWxbds2xo8fD4DWmj179pCUlARAampqozZ/99dTUlISu3fvRvvYGIyqqio+//xzxo0bB7jWOg4NDfX762yaJjU1NdTX11NTU0PXrl397joPGDDgvDVoWnpdCwoKuPbaawkLCyMsLIxrr72WgoKCZmcI2K4kp9NJZGRkw8eRkZEcOHDAwkTuV1JSwuHDh+nXrx8nT56kW7duAHTt2pWTJ08Cru/DDxdPj4yMxOl0NmzrK15++WUefPBBzpw5A0BFRQUhISHYbDbAtfys0+kEGl97m81GSEgIFRUVPrWMbElJCV26dOHFF1/kyy+/JC4ujqlTp/r1dbbb7dx666089thjdOjQgSFDhhAXF+fX1/k7Lb2uP/799sPvS3ME7B2Dvzt79ixz5sxh6tSphISENPqaUgqllEXJ3G/r1q1ERET4ZJ95a9XX13P48GEmTJjAn//8Zzp27Mjy5csbbeNv17myspK8vDwWLlzISy+9xNmzZ1v0V7C/aI/rGrB3DHa7nbKysoaPy8rKsNvtFiZyn7q6OubMmcPo0aMZOXIkABEREZSXl9OtWzfKy8sb/mqy2+2NVn/yxe/Dvn37yM/PZ/v27dTU1HDmzBlefvllqqqqqK+vx2az4XQ6G9r13bWPjIykvr6eqqoqwsPDLW5Fy0RGRhIZGUn//v0BV1fJ8uXL/fo679q1i+jo6IY2jRw5kn379vn1df5OS6+r3W5n7969DZ93Op0MGDCg2ecL2DuG+Ph4iouLKSkpoa6ujtzcXBwOh9Wx2kxrzaJFi+jVqxe33HJLw+cdDgdr1qwBYM2aNQwfPrzh82vXrkVrzf79+wkJCfGp7gWA+++/n0WLFrFw4ULS09MZNGgQjz/+OAMHDmTTpk2Aa4TGd9c3MTGRnJwcADZt2sTAgQN97i/rrl27EhkZ2bCk7a5du+jdu7dfX+eoqCgOHDhAdXU1WuuGNvvzdf5OS69rQkICO3bsoLKyUQmWAAAAAShJREFUksrKSnbs2EFCQkKzzxfQbz5v27aNV155BdM0GTt2LHfeeafVkdrsiy++4Pe//z19+vRp+E9w33330b9/f+bNm0dpael5w90yMzPZsWMHHTp0YMaMGcTHx1vcitbbs2cPH3zwAbNmzeLbb79l/vz5VFZW0rdvX2bOnElwcDA1NTVkZGRw+PBhwsLCSE9Pp0ePHlZHb7EjR46waNEi6urqiI6OZsaMGWit/fo6v/nmm+Tm5mKz2bjiiiv4xS9+gdPp9KvrPH/+fPbu3UtFRQURERFMmTKF4cOHt/i6Zmdns2zZMsA1XHXs2LHNzhDQhUEIIcT5ArYrSQghxIVJYRBCCNGIFAYhhBCNSGEQQgjRiBQGIYQQjUhhEEII0YgUBiGEEI38/6cUaXkSH5aaAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "id": "gdwJu6i3wwdP", "outputId": "3501bcf8-9895-44e1-e6bd-c6b2051e69f7" }, "source": [ "plot_lr(cosine(t_max=500, eta_min=0.0005))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU1f3H8fe5kwWyEJgJIYBRIcSFTSpBQhQJJKCCKCJal7oArQuKjdRfq9haa0ulWhYlWKiNiIiCVXYVNcZgYYwEEWRRJOAWDQYyCAlhSXLP74+R1JQl2yR35s739Tx9nk5y753PyR38zjn33nOU1lojhBBC/MiwOoAQQgj/IoVBCCFELVIYhBBC1CKFQQghRC1SGIQQQtQihUEIIUQtIVYH8JXvvvuuUfvFxsayb98+H6fxb9Lm4CBtDg5NaXOnTp1O+nPpMQghhKhFCoMQQohapDAIIYSoRQqDEEKIWqQwCCGEqKVedyVt2rSJefPmYZom6enpjBo1qtbvKysrycrKYvfu3URHR5OZmUlcXBwAS5cuJTc3F8MwGDt2LH369AHgmWeeYePGjcTExDBt2rSaY5WXlzNjxgz27t1L+/btuf/++4mKivJVe4UQQtShzh6DaZpkZ2czefJkZsyYwbp16ygqKqq1TW5uLpGRkcyaNYsRI0awcOFCAIqKinC73UyfPp2HH36Y7OxsTNMEIC0tjcmTJ5/wfsuWLaNXr148/fTT9OrVi2XLlvminUIIIeqpzsJQWFhIfHw8HTp0ICQkhNTUVAoKCmpts2HDBtLS0gBISUlh69ataK0pKCggNTWV0NBQ4uLiiI+Pp7CwEIDu3buftCdQUFDAoEGDABg0aNAJ7+VL5gfvUf7KPMycFZjud9Gfb0Mf/AGZiVz4gj5Ujvnmq5hvLcV8fzV6ywZ0STHarLY6mhCnVedQksfjweVy1bx2uVzs3LnzlNs4HA4iIiIoKyvD4/GQlJRUs53T6cTj8Zz2/Q4cOEC7du0AaNu2LQcOHDjpdjk5OeTk5AAwdepUYmNj62rKCfZv/pBDH7lrXh8vB4YrjtBeFxLWux/hFw3EiLTXUFZISEij/l6BzIo2H95awMElL9S8Pv75Uq1aE3J+b8J69SW8/yBCOiU0y/vLeQ4OzdFmv37yWSmFUuqkv8vIyCAjI6PmdaOe/LvrQeLatmXft9/AoTL4vhi9pwi96zOObHBzJG81hITCBf0wLr0Mzu9zyjyBRJ4ObRnmgYMAGI/OgtaRUFqC3lME3+zm2GdbOPbxh5S/8AycnYRKTUelDkGFt/LZ+8t5Dg7N8eRznYXB6XRSWlpa87q0tBSn03nSbVwuF9XV1VRUVBAdHX3Cvh6P54R9/1dMTAz79++nXbt27N+/nzZt2tQVsUlUSAgqMhoioyGuE6pXXxh6Ndo04cud6PXvo9e/j/mRGxK6oC6/FpV8CcqQG7pEXX7sI7RqjXLGgjMWldT9v7/17ENv+A86Pw/90hz08oWowcNRGVejbNZLFYGlzv+6JSYmUlxcTElJCVVVVbjdbpKTk2tt07dvX/Ly8gDIz8+nR48eKKVITk7G7XZTWVlJSUkJxcXFdOvW7bTvl5yczJo1awBYs2YN/fr1a2TTmkYZBqrruRg3/Arjb8+hbpsIVVXoZ/+OOeU36B1bLMkl7EM5YzGGXYPjkacwfjcVkrqjX38F8+E7MXNWoKsqrY4ogpTj0UcfffR0GxiGQXx8PLNmzWL16tUMHDiQlJQUFi9ezJEjR+jUqRNnnnkma9eu5aWXXuLLL7/kjjvuICoqipiYGMrLy5k7dy5r165l3LhxNV2XmTNnsnjxYkpLS8nJySEiIoIuXbrQtWtXli9fzmuvvUZ5eTljx44lLCyszoaUlZU16g8QERFBRUXFabdRDgfqzETUoCugQ0f4ZD363VXw/bdwTg+fdv9bQn3abDdWtFl/sxs2fejtAUREnnZb5WyPcdGlqD4p6G+/hLw30R+5UWclensbjSDnOTg0pc3R0dEn/bnSNrkFpyVnV9XHjqJXv4Z+41VoHYFx812o5Esa9f5WkHHYlmGufQc9fxbG1GyUq32999Naw5YNmAvnwP59qPSRqNG3okLr/oL0U3Keg4PMruonVFg4xlU3YfxhBrjiMOc+gbnwH+jKY1ZHEzaglEL17ofxp1moQVegc1ZgPv5/6JLGffkRoqGkMDSB6nwWxoNPoC67Bp33JubU36L37rE6lvAXxzvjjbyRTbXy9kaNiX+A0r2Yf74f/dE63+UT4hSkMDSRCgnBGDMW497fw74SzL8+gC781OpYwkZU734YjzwFnc7EnPM3zNdfkYcwRbOSwuAj6oKLMCb/HSIiMaf9HrNgrdWRhN9o+rMvytUe44G/olLS0MteRM9/Wu5aEs1GCoMPqQ6dMB58Es5OQv/zCcx3V1kdSdiICg1FjbsfNfIG9Lp3MbP+gj561OpYwoakMPiYim6DMenP8LMU9KJ/Yq5+zepIwkaUUhhX3eR9rmb7JsxZj6GPHLY6lrAZKQzNQIWGYtzxW1S/gejX5mOueMnqSMIKNReffT+NinHJUNT4SbBzG+bMP6IrDvn8PUTwksLQTFRICOqXk1Cp6eiVizDffNXqSMJmjP6DMO78LXy5E3PWn2VYSfiMFIZmpAwH6raJqIsuRS95ATPvTasjCSs047yL6sJUjF/+BnZ9hjnncbkgLXxCCkMzU4aBGpsJvZLRL83B/HCN1ZFEi2mZW0pV8iWoWybA1o3of02X9R5Ek0lhaAEqJATjrt9BUg/0vJnoTzdbHUnYjDFwGOq6seiP1qEXPWt1HBHgpDC0EBUWjnHPw9ChM+Y/pqKLv7E6kmhuNR2GllnDwxh2DWrYKPR7b8it0qJJpDC0IBUR6Z3eICTEe7Gw7KDVkYTNqGtvgwsuQi/+F0d/sjqhEA0hhaGFqdgO3p7DDx7MZ6agK+Vioe214Kp/ynB4L0YnnM2Bvz+CLvqyxd5b2IcUBguoxPO8F6QLP0UvlvFg27JoPiPVqjXGvX9AtY7AfOav6IpyS3KIwCWFwSJGv0u8s7KuWY3pzrU6jrAZ1c5FzG+ngGcv5nMzvUvVClFPUhgspK65Fc7thX7xGfQ3X1gdRzSXlhtJqiXsvF6o68bD5vVoecBSNIAUBgsphwPjjgcgMgrzH49Ll992rJ8aWw0Z4X3AcvlC9PaPrY4jAoQUBoupNu0w7noQPPsw58+SefaFTymlULfeCx0TMJ+dhv7BY3UkEQCkMPgBlXge6ppbYOMH6LXvWB1H+EoLP8dwKiq8lXdOpWNHMOc9JdcbRJ2kMPgJNfRqOP8C9KJn0cVFVscRNqM6nYm6/pew/WN0znKr4wg/J4XBTyjDwBh3P4SFYz77pDzfYCct+BzD6ahLL4M+KeglC9Bf7bI6jvBjUhj8iGrrxLj9PvjmC/TSF6yOI5rMv64XKaUwbrsXottgPvt39NEjVkcSfkoKg59RF1yEShuOfmc5+vOtVscRNqOi2nh7pt9/i14iXz7EyUlh8ENqzO3QPh7z+aflW10gq7nDzD+Gko5T51+AGnIlOncVeod8+RAnksLgh1R4K++Q0t496NfmWx1H2JAafav3y8f8p2XNaHECKQx+Sp3TE5U+Ev3e6+gdW6yOI5rCvzoMwPEvH7+Gfd+jl8iXD1GbFAY/pq65BeI6eu89l291gce/rj2fQJ3Twzuk9N4b6M8+sTqO8CNSGPxYzbc6z1700gVWxxE2pK651fvlY/4s9NGjVscRfkIKg59TSd1RaVd4h5S+2Gl1HNEgP3YZ/OQ5hpNR4eEYt070Dim9vsjqOMJPSGEIAGrULRDTDnNBFrpaFnoXvqXO7Ym6OAP99jJZ2EcAEFKfjTZt2sS8efMwTZP09HRGjRpV6/eVlZVkZWWxe/duoqOjyczMJC4uDoClS5eSm5uLYRiMHTuWPn36nPaYW7duZcGCBVRVVdGlSxfuvvtuHA6HL9sccFREJMYNd2DOmYp+dyVq2Ki6dxJ+xH97DMepMbejN6/HfPEZjN9ORRnynTGY1Xn2TdMkOzubyZMnM2PGDNatW0dRUe25fHJzc4mMjGTWrFmMGDGChQsXAlBUVITb7Wb69Ok8/PDDZGdnY5rmKY9pmiazZ8/m17/+NdOmTaN9+/asWbOmeVoeaC4cAL37eadPLi2xOo2ojwCaKVdFtUFdPx52fYZ+/y2r4wiL1VkYCgsLiY+Pp0OHDoSEhJCamkpBQUGtbTZs2EBaWhoAKSkpbN26Fa01BQUFpKamEhoaSlxcHPHx8RQWFp7ymOXl5YSEhNCpUycAevfuzYcffuj7VgcgpRTGTXcCYL40V6bnFj6nUtLgvN7oJS/I9NxBrs6hJI/Hg8vlqnntcrnYuXPnKbdxOBxERERQVlaGx+MhKSmpZjun04nH46k5zv8eMzo6murqanbt2kViYiL5+fns27fvpLlycnLIyckBYOrUqcTGxta3zbWEhIQ0et8WFxvLoZt+RfnzWUQXbqPVgLRGHSag2uwjVrT5UGQk5YAr1oURGd2i7w2Na3PVvZMpvf9WQpcvoO1v/txMyZqPfLZ9dEyfHq2JlFJkZmYyf/58KisrueCCCzBOMdaZkZFBRkZGzetTFZC6xMbGNnpfK+iUdMhZxYF/zaDszCRUeHiDjxFobfYFK9psHvKuyFdaWoo63PK3gjaqzeERqMuv5ejKl9nbfzDqvN7NE66ZyGe7YY6PzvyvOoeSnE4npaWlNa9LS0txOp2n3Ka6upqKigqio6NP2Nfj8eB0Ok97zHPOOYfHHnuMxx9/nPPPP5+OHTs2oJn2pxwOjBvv8D7bsPo1q+OIevH/i88/pS4fDa44zEXPyl1wQarOwpCYmEhxcTElJSVUVVXhdrtJTk6utU3fvn3Jy8sDID8/nx49eqCUIjk5GbfbTWVlJSUlJRQXF9OtW7fTHvPAgQOA906n5cuXM2zYMB83OfCpc3qi+g1Er34NvXeP1XHEqQToZSAVFo5x/Xj49it03ptWxxEWqHMoyeFwMG7cOKZMmYJpmgwePJiEhAQWL15MYmIiycnJDBkyhKysLCZOnEhUVBSZmZkAJCQkMGDAACZNmoRhGIwfP75maOhkxwRYsWIFGzduxDRNhg0bRs+ePZux+YFLjRnrvb3w38/hmDDZ6jjCbn6WAt37oFcsRF80EBUdY3Ui0YKUtsntLd99912j9gvkMUnzjX+jly7AuP9PqO4/q/d+gdzmxrLkGsPby9D/fg7j6UWo1hEt+t7Q9Dbr4m8w/3QfKjUd49Z7fZis+chnu2EafY1B+C81dJR36uSXn0VXVVkdR9iM6pjgnWRv7TvoL2U6lmAihSGAqdBQjJ//CvYUoXNXWR1H2JC68gaIjsF8+Z9o07Q6jmghUhgCnLqgH/Tsi161GF120Oo4opbjk+hZm6IpVESkd1Gf3TvQBf+xOo5oIVIYbMC4biwcPYxeJbNjCt9TA4ZAQhf00gXoymNWxxEtQAqDDahOZ6IGDkOveRO951ur44jjam7rCOAuA6AMA+O6cVBagn53pdVxRAuQwmAT6qqbIDQM87XnrY4ibEidf4F3Esc3/o0uO2B1HNHMpDDYhGrTFnX5tbDpQ/SOrVbHETZkjLkdjh5Br5QhS7uTwmAjaujV4IzF/PdzcgeJX/D/FdwaQnVMQF16Gfr91eg9RXXvIAKWFAYbUWHhqGtuga8K0etlHQvhe2rkjd4hy1eftzqKaEZSGGxGXTQIzurmvYPkmCzubqmaSQXs0WOAH4csrxgDm9ejd2yxOo5oJlIYbKbmDhLPPvS78tCb8D2VcdWPQ5bzZMEom5LCYEPq3J7QKxm9+lX0j2sCCOErKiwcddXN8FUhbPzA6jiiGUhhsCnjmlug4hD6rSVWRwle9htJqqEGpEHHBMxlC2TNBhuSwmBTKqEL6qJB6HdXyPq9wueU4fB++djzLdr9rtVxhI9JYbAxdfVNUF2Nfn2x1VGCm01uVz1Bn/7Q5Rz0ykVyo4PNSGGwMRXXETXwMvR/3kaXNG69CtEU9r4wq5TCuPY22L8PnfeG1XGED0lhsDk14npwhKCXv2R1FGFD6txe0ONn6DdeRVccsjqO8BEpDDan2jpRGVeh17+P/nq31XGCiw2fYzgZ45pb4VAZ+u2lVkcRPiKFIQioy66BiCjMZS9aHUXYkDorEdVvIPqd5eiD+62OI3xACkMQUBFRqOFjYMsG9OcywV6Ls3eHAQB19c1QVYleJTc62IEUhiChBo+Atk7MZS/K06otJYj+zqpDJ9QlQ9Hvv40uLbE6jmgiKQxBQoWFo4ZfDzu3c2xzgdVxhA2pEdeDAv36K1ZHEU0khSGIqEuGgjOWQy8/K72GFhUEY0mAcrb33h7tfhe9d4/VcUQTSGEIIio0FDXi51R+vg22fmR1HGFDavgYMBxyrSHASWEIMio1HUeHTpjLX5Jeg/A51daFGnQF+oP3ZP3xACaFIciokBAirx/rnRlz84dWx7E3ba8V3OpLXTEaQkPRq2QJ0EAlhSEItRp0GcR19PYaZAlQ4WOqTTvU4BHehyq/+9rqOKIRpDAEIeUIQY28AYq+hI9lPv3mc7zHYG0KK6jLRkNYK/RK6TUEIikMQUpddCnEn/Fjr0Hm0xe+paLboDJGojesRRd9YXUc0UBSGIKUMhyoq26E4m/QBWutjiNsSA0dBa0jMVe8bHUU0UAh9dlo06ZNzJs3D9M0SU9PZ9SoUbV+X1lZSVZWFrt37yY6OprMzEzi4uIAWLp0Kbm5uRiGwdixY+nTp89pj7llyxZefPFFTNOkVatW3HPPPcTHx/uyzeJHqu/F6M6veOfTT74E5XBYHcleam76CsKxJEBFRqGGXo1e8RL6q12osxKtjiTqqc4eg2maZGdnM3nyZGbMmMG6desoKiqqtU1ubi6RkZHMmjWLESNGsHDhQgCKiopwu91Mnz6dhx9+mOzsbEzTPO0x//WvfzFx4kSefPJJLrnkEl577bVmaLYAUIaBcdWN8P236A/XWB1H2JBKH+mdwHH5QqujiAaoszAUFhYSHx9Phw4dCAkJITU1lYKC2lMqbNiwgbS0NABSUlLYunUrWmsKCgpITU0lNDSUuLg44uPjKSwsrPOYhw8fBqCiooJ27dr5sLniBD8bAGd2Ra9ahK6qsjqNvQTp7ao/pSIivbP7btmA3r3D6jiinuocSvJ4PLhcrprXLpeLnTt3nnIbh8NBREQEZWVleDwekpKSarZzOp14PJ6a45zsmHfddRePP/44YWFhtG7dmilTppw0V05ODjk5OQBMnTqV2NjYejX4f4WEhDR630D1v20+cvOdHHj8d0Rt30jrIcMtTNZ8rDjP5ZERHAJiY2NRFhQHf/lsm9fdxr6cFYS8vZR2v/97s76Xv7S5JTVHm+t1jaElvf766zz00EMkJSWxYsUKXnjhBe66664TtsvIyCAjI6Pm9b59+xr1frGxsY3eN1D9b5t1l/MgoQsHF2dT3qOvLa81WHGezUMVgPezaUVh8KvPdvpIji17kb0bPkCdnVTn5o3lV21uIU1pc6dOnU768zqHkpxOJ6WlpTWvS0tLcTqdp9ymurqaiooKoqOjT9jX4/HgdDpPecyDBw/y1Vdf1fQyUlNT2bFDup/NTSmFceUNUFKMLnjf6jg24h1KsqIo+Bs15ErvtQaZQykg1FkYEhMTKS4upqSkhKqqKtxuN8nJybW26du3L3l5eQDk5+fTo0cPlFIkJyfjdruprKykpKSE4uJiunXrdspjRkZGUlFRwXffeReu/+STT+jcubPvWy1O1Kc/dD4L/for8lyD8DnVOgI19CrYvB799S6r44g61DmU5HA4GDduHFOmTME0TQYPHkxCQgKLFy8mMTGR5ORkhgwZQlZWFhMnTiQqKorMzEwAEhISGDBgAJMmTcIwDMaPH49heGvRyY4JcOeddzJt2jQMwyAyMpK77767GZsvjlOGgTHyBsw5f0MXrEX1H2R1pMAncxTWooaMRL+9HHPVYhwTJlsdR5yG0jaZYvN4L6OhZEzyv7RpYv7pPtAa49GnUYZ9rjVYco1h+UvoVYtwPLuiRd/3OH/8bJsrXkKvXITxx6dQZ3Tx+fH9sc3NzZJrDCJ4KMNAXflz79PQH7mtjiNsSKVfBa0jMFfKtQZ/JoVB1KL6pkLHBPSqxTLzapPpoH6G4WRUZJT3obeNbvS3X1kdR5yCFAZRizIc3rV7v/taZl4VzUJlXAWtWssqb35MCoM4gep3CcR3xpReQ9NoTbDOk3Q6KjIaNeRK9EfrZL0GPyWFQZygptdQ9CVsklXehO+pjKshLBz9+itWRxEnIYVBnJTqd6l3lbdVi2RtaOFzKrqNd5W3gv+gi4vq3kG0KCkM4qSU48dewzdfwOb1VscJTDKSdFpq2CgIDUO/Ib0GfyOFQZyS6p8G7eMxV0qvQfieio5BpQ1Hf/g+es+3VscRPyGFQZyScjhQw6+Dr3fBlg1WxxE2pC4bBaEh6Df+bXUU8RNSGMRpqZTBENtBeg2NIs8x1EW1aYe69Ar0h3nokmKr44gfSWEQp6VCQry9hi93wraNVscRNqQuuwYc0mvwJ1IYRJ3UgMHgbC+9hoaS5xjqRbV1oi69DJ3/HnrvHqvjCKQwiHpQIaHeXsPuHbB9k9VxhA2py0eDMtBvvmp1FIEUBlFPKjUd2sVirnxZeg3C51RbF2rgULT7XfS+762OE/SkMIh6UaGhqCvGwK7P4LNPrI4TILSMJDWAunwMKIV+8zWrowQ9KQyi3tQlQ6GtS3oNolkoZyzqkqHodTno0r1WxwlqUhhEvXl7DdfCzu2wY4vVcfyfBukyNIy6fAwAerVca7CSFAbRIGrgMIhxYq5cZHUUYUPK1R51cQZ67TtoT3CtxOZPpDCIBlGhYd47SD7fit6x1eo4wobUFdeC1tJrsJAUBtFg6tLLIKYd5irpNZyWliefG0PFdkClpqP/8zZ6f6nVcYKSFAbRYCosHHXZaPjsE/TO7VbHETakrhjj7TW8tcTqKEFJCoNoFHXp5RAdg7nyZauj+DG5XbWxVPt4VMpg9PtvoX/wWB0n6EhhEI2iwn/sNXy6GV34qdVxhA2p4ddBdZX0GiwghUE0mkq7wttrkGsNohmouI6o/mnoNavRB/ZbHSeoSGEQjabCW3lX4dr2MXr3Dqvj+B95jqHJ1IjroUp6DS1NCoNoEpU2HKKi5bkG0SxUh06o/oPQa95EH/zB6jhBQwqDaBLVqjVq6CjY+hH6i8+tjuNn5HZVX1AjroPKKvTbS62OEjSkMIgmU0NGQGQ05qrFVkcRNqTiz0D1G4h+7w102QGr4wQFKQyiyVSrCNTQq+GTAvRXhVbHETakrrweKo+h315mdZSgIIVB+IQaPAIiIuVaw0/JCm4+ozomoJIvQb/3OrrsoNVxbE8Kg/AJFRGJyrgaNq9Hf73L6jjChtSVP4djR9E5y62OYnsh9dlo06ZNzJs3D9M0SU9PZ9SoUbV+X1lZSVZWFrt37yY6OprMzEzi4uIAWLp0Kbm5uRiGwdixY+nTp89pj/nII49w+PBhAA4ePEhiYiK//e1vfdZg0XxU+pXod5ZjrlqMY8Jkq+NYT5as8CnV6UxU34vRuavQw0ahIqOtjmRbdfYYTNMkOzubyZMnM2PGDNatW0dRUVGtbXJzc4mMjGTWrFmMGDGChQsXAlBUVITb7Wb69Ok8/PDDZGdnY5rmaY/52GOP8eSTT/Lkk0+SlJRE//79m6HZojmoiChUxkj4OB9d9IXVcfyDjCT5lBpxPRw5jH5Heg3Nqc7CUFhYSHx8PB06dCAkJITU1FQKCgpqbbNhwwbS0tIASElJYevWrWitKSgoIDU1ldDQUOLi4oiPj6ewsLBex6yoqGDbtm3069fPd60VzU6lXwWtIzBXyh1KwvfUGWfDhaneXsOhcqvj2FadQ0kejweXy1Xz2uVysXPnzlNu43A4iIiIoKysDI/HQ1JSUs12TqcTj8dTc5zTHbOgoICePXsSERFx0lw5OTnk5OQAMHXqVGJjY+tqykmFhIQ0et9A1axtjo2l/MrrOfTv54k5dIDQsxKb530ayIrzXNa6FYcNw7LPl10/25W33IXn/ltp7c4h6sZf1vqdXdt8Os3R5npdY7DCunXrGDJkyCl/n5GRQUZGRs3rffsat9pTbGxso/cNVM3dZn3xUFi5mP0L5mDc9btme5+GsOI8m4cPo7W27PNl2892VFv4WQqHVi7m8MUZqIioml/Zts2n0ZQ2d+rU6aQ/r3Moyel0Ulr638UySktLcTqdp9ymurqaiooKoqOjT9jX4/HgdDrrPObBgwcpLCzkwgsvrGfzhD9RkdGoIVeiN7rR335tdRzraLn63FyMK38Ohw+h311ldRRbqrMwJCYmUlxcTElJCVVVVbjdbpKTk2tt07dvX/Ly8gDIz8+nR48eKKVITk7G7XZTWVlJSUkJxcXFdOvWrc5j5ufnc+GFFxIWFubb1ooWozKuhrBw9OvBfq1Brj43B3VmIlxwETpnObrikNVxbKfOoSSHw8G4ceOYMmUKpmkyePBgEhISWLx4MYmJiSQnJzNkyBCysrKYOHEiUVFRZGZmApCQkMCAAQOYNGkShmEwfvx4DMNbi052zOPcbvcJt8SKwKKi26CGjECvXoIefp33oqEQPmSMvBHzL/ejc5ajrrrJ6ji2orS2R3/3u+++a9R+MibZfPShMsyHfgXn9bb8uQZLrjEsehb9QS6Op6xZ5S4YPtvV/3gctm/CePxZVFSboGjz/7LkGoMQjaUio71DSh/nyxxKolkYV90ER4+g35KZV31JCoNoVirjKu/Mq8tfsjpKy7NHZ9yvqc5nofpd6n2u4aCs8uYrUhhEs1IRkd61obdsCNK1oeXic3NTI2+Aqkr0m69ZHcU2pDCIZqeGjPCuDb18odVRhA2p+M6oAUPQeW9Sva/E6ji2IIVBNDsV3go1/Dr47BP0p5utjtNytKzg1lLUlT8HrTn06nyro9iCFAbRItSgy6GtC3P5QmxyI5zwIyq2A2rgMA7nrEDv3WN1nIAnhUG0CKO8+P8AABcTSURBVBUa5v1Wt+sz2LrR6jjChtSI68DhQMsSs00mhUG0GHVxOsR2wFz2YpD0GrRce25Bqq2LiMtHoz94D72nqO4dxClJYRAtRoWEokbeCF/vgo/zrY4jbCjyml9AWBh6hTUPFdqFFAbRolTKIIg/w3utway2Ok7z0iBdhpZltHWi0keiC/4ji0U1gRQG0aKU4fDOa/Pd1+j171sdR9iQGnYNtI7EXCa3RzeWFAbR4lTfVDgzEb1sIbqy0uo4wmZUZBTq8tGweT36821WxwlIUhhEi1OGgXHtbVBagl7zhtVxmpE8x2AVlX4VtHViLpkfJDc6+JYUBmEJ1b0PdP8Z+vVXZD594XMqPNw7ZLnrM9j0odVxAo4UBmEZ49pbobwM/dYSq6M0D/mmaimVmu690WHJC+hqm9/o4GNSGIRl1JmJqIsGeVfh+qG07h2EaADlcGCMvhX2FKHX5VgdJ6BIYRCWUqNuhmoTvXKR1VGEHfXpD4nnoVe8jD561Oo0AUMKg7CUah+PSrsCvfYddLHNnlbVyMVniymlMK69HQ540DnLrY4TMKQwCMupEddDWDjm0hesjiJsSCV1hwsuQr+1BF120Oo4AUEKg7Ccio7xLubzcb7NFvORi8/+whh9Kxw5gn7jFaujBAQpDMIvqKFXQ4wT85Vse913LkNJfkF1OhN1SQb6vTfQ339ndRy/J4VB+AUV3go1+hb44nOZKkM0C3X1zRASivnqPKuj+D0pDMJvqJTBcFY39Gvz7XEHidbIJHr+Q8W0867ZsOnD4FpJsBGkMAi/oQwD4/rxsH8f+u2lVscRNqQyrgJXnHfI0u6z+zaBFAbhV9Q5PVB9L0avfg29Xx56E76lQsMwxtwORV+i18pDb6cihUH4HXXtbWCaaDvcviojSf6n78XQrTt62YsyT9cpSGEQfke1j0cNvcq7ROMXO62OI2xGKYXx8/FQdgD9xr+tjuOXpDAIv6SuuA7atMVc/Gzg3r4qF5/9ljo7CTVgCPrdFeiSYqvj+B0pDMIvqdYRqGtugV2fofPzrI4jbEiNvgUcoZiL/2V1FL8jhUH4LZWaDl3PRf/7OXRFudVxGi5QezpBQrV1oUbeAJ8UoGXNhlpC6rPRpk2bmDdvHqZpkp6ezqhRo2r9vrKykqysLHbv3k10dDSZmZnExcUBsHTpUnJzczEMg7Fjx9KnT5/THlNrzaJFi8jPz8cwDIYOHcrw4cN92WYRIJRhYNx0F+aU36CXLUTddKfVkRpOnnz2ayp9JHpdDuaiZzHO74MKD7c6kl+os8dgmibZ2dlMnjyZGTNmsG7dOoqKas+CmZubS2RkJLNmzWLEiBEsXOhdhLuoqAi328306dN5+OGHyc7OxjTN0x4zLy+P0tJSZsyYwYwZM7j44oubodkiUKizElFpl6Pz3kR/vcvqOMJmVEgIxs13eZeZfVMuRB9XZ2EoLCwkPj6eDh06EBISQmpqKgUFBbW22bBhA2lpaQCkpKSwdetWtNYUFBSQmppKaGgocXFxxMfHU1hYeNpjvv3224wZMwbD8EaLiYnxcZNFoFGjfgFR0ZgL56BN0+o4DSBDSYFAndsL1X+Qd/ZVmUcJqMdQksfjweVy1bx2uVzs3LnzlNs4HA4iIiIoKyvD4/GQlJRUs53T6cTj8dQc52TH/P7773G73axfv542bdowduxYOnbseEKunJwccnK8D6hMnTqV2NjYejf6p0JCQhq9b6AKvDbHcvj2iRyc9ReiPvmQ1hkjG3wEK9p8ILwVxxwOy/7WgXeem66xba6+8wFK791AyKvP0faRGagAGgJsjvNcr2sMLamyspLQ0FCmTp3Khx9+yD/+8Q8ee+yxE7bLyMggIyOj5vW+ffsa9X6xsbGN3jdQBWKbda9+0K07B+dnUd6tByqqTYP2t6LN5pEjaLPasr91IJ7npmpSm0feyLHF/2Lf2ytQfQNnCLspbe7UqdNJf17nUJLT6aS09L9TE5SWluJ0Ok+5TXV1NRUVFURHR5+wr8fjwel0nvaYLpeL/v37A3DRRRfx1Vdf1beNwsaUUhg33wkVh9CvPGd1nAYInG+ewU4NHgEJXTBffjYw74LzoToLQ2JiIsXFxZSUlFBVVYXb7SY5ObnWNn379iUvLw+A/Px8evTogVKK5ORk3G43lZWVlJSUUFxcTLdu3U57zH79+rF161YAtm/ffsqKJoKPOqML6vJr0R/kord9bHUcYTPK4cC4bSIc/AH96vNWx7FUnUNJDoeDcePGMWXKFEzTZPDgwSQkJLB48WISExNJTk5myJAhZGVlMXHiRKKiosjMzAQgISGBAQMGMGnSJAzDYPz48TUXlU92TIBRo0bx9NNP8/rrr9OqVSvuvDMAb1EUzUZd+XP0RjfmgtkYj85CtWptdaRTk+cYAo46qxtq2NXot5aiL7oUdV5vqyNZQumAnW+gtu++a9zdBDIOG3j0zu2YTzyISh+JccOv6rWPJdcYnpuJ/nwrjqnWPFkb6Oe5MXzRZn30KOafJgJg/HGW3z/bYMk1BiH8jUrqjkobjs5dhd71mdVxTsMW37mCjgoPx7j1Xti7B73yJavjWEIKgwhIavSt0NaFOX8WurLS6jjCZtR5vVEDh6HfXo7+Mvhm+JXCIAKSah2BccsEKP4maL/ViealxtwOMW29Q4KVx6yO06KkMIiApXole7/VrV6C3rnd6jgnkpGkgKYiojBuu8/75WPJAqvjtCgpDCKgqevHedfwfW4G+kiF1XFOFEBP0IoTqZ4XogYPR+csR3+62eo4LUYKgwhoqlUExvj7oXQvenG21XGEDalrx0KHzpjPPxU0D75JYRABT3Xrjrr8GvTad9Cb8q2O8xNaegw2oMLDvV8+fvCgX/6n1XFahBQGYQvqqpu80xnMz0LvL617ByEaQHU5BzXi5+j8PMwgWFFQCoOwBRUSivGr/4PKY5jPPomurrY6kjz5bDNqxPXQrTv6xWfQe4rq3iGASWEQtqE6noH6xd2wczt6hdzCKnxLORwYv3oAQsMw5/wNfeyo1ZGajRQGYStGymDUJUPRb76K3rrR6jjCZpQz1nu94duv0IuetTpOs5HCIGxH3XAHdDoTM3s62rPXuiAaufhsQ6pnX9QVY9D/eRvzg/esjtMspDAI21Hh4Rh3/g6qKjGfedzWXX5hDXX1zXBOT/SC2bacMkMKg7Al1fEMjF/+Br7ehZ4/C2smEZaLz3alHA6Mu34Hbdpizv4r+geP1ZF8SgqDsC11wUWoUb9Ar3+fCsumNJChJLtS0TEY9zwMFeWYc6baajJHKQzC1tQVY1D9BlK+cC5683qr4wibUQldMMbdD7s+897GapNblKUwCFtTSqFuu4+Qrudi/vMJ9O4dLffmNvmPhDg91TcVNfJGtPtd9IqXrY7jE1IYhO2p8HDa/v7vEOPEnPVYyz6cJHclBQU18gbUxRnoVYsw8960Ok6TSWEQQcHR1omR+SgoA3Pmo+gfZNoM4TtKKdQt90DvfuiX5qI3fmB1pCaRwiCChorrhHHfI1B+EHP6I+iDP7TAmzb/Wwj/oBwOjDv+D87uhvns39FbP7I6UqNJYRBBRZ2dhDHxD1D6Pea037dMcRBBQ4W3wvj1H6FTgvc21gB9+l4Kgwg66txeGBMfgX17MKf/AV12oHneSC4+ByUVGY0x6c/Q8QzM2VMCsjhIYRBBSZ3XG+PeP8DeYswnHkKXljTXOzXTcYU/O6E4fOS2OlKDSGEQQUudfwHGr/8EB/djTv0tuugL376B9BiCmopqg/Gbv8BZiZhz/4b53utWR6o3KQwiqKlzemD8diqgvD2HbR/7+A2kxxDManoOP96tZL76PNr0g7VC6iCFQQQ91fksjIeeAGd7zKf+hPnmq7Z5glVYT4WFY9z9ECrtCvRbSzCffgx9qMzqWKclhUEIQDnbYzz0JCr5YvSSF7wLsRxq2sLvWibREz9SDgfGzXejbr0XdmzB/Mskv56VVQqDED9S4a1Qv3oAdd1Y2JSP+ehE3w8tiaBmDByG8X+PQ3U15uP/h7niJXRVldWxTiCFQYifUEphDLsG46EnoXUE5sw/Yi6YjS4/2PCDSYdBnITqei7GH59GXXQpeuUi740PftZ7kMIgxEmos5Mwfj8dNWwUeu07mA/fhZm7quHf7uTiszgJFRmFMX4Sxt0Pwv59mH99APP5p/xmXYcQqwMI4a9UWDjqunHo1HTMRc+iX/4n+q2lqMtHoy7OQIWFWx1RBDh1YSrG+X3Qqxaj312JXv8f1CVDUZeNRrnaW5arXoVh06ZNzJs3D9M0SU9PZ9SoUbV+X1lZSVZWFrt37yY6OprMzEzi4uIAWLp0Kbm5uRiGwdixY+nTp89pjzl79my2b99OREQEAPfccw9nn322r9orRIOpzmd5bznc+hHm6694J0lb/hIqJQ2VOgQSuqJO2jOQsSRRN9U6AnXdWPSgy9BvvoZ+fzV6zZvQKxnj4gzo2RcVGtqimeosDKZpkp2dze9//3tcLhcPPfQQycnJnHHGGTXb5ObmEhkZyaxZs1i3bh0LFy7k/vvvp6ioCLfbzfTp09m/fz9//vOfeeqppwBOe8xbbrmFlJSUZmqyEA2nlPL+Q+3ZFz7fil7j/cer310J7WJRPS+EbuejzjgbOiagQsOO72hpbhE4VFwn1G0T0VfegM57A/3Be5ib10N4Kzi3F+r83qiErtD5LIiMPsWXEd+oszAUFhYSHx9Phw4dAEhNTaWgoKBWYdiwYQPXXXcdACkpKTz33HNorSkoKCA1NZXQ0FDi4uKIj4+nsLAQoM5jCuGPlFLef6Tn9kIfKkNv/AC9dSN6w1r4z9v/7SO0joDDFdDpTCvjigCkXO1R196GHvUL2L4JvaXA+xn7pOC/n6+QUIhqA60jqPrDNAht5dMMdRYGj8eDy+Wqee1yudi5c+cpt3E4HERERFBWVobH4yEpKalmO6fTicfjqTnOqY758ssv8+qrr9KzZ09uvvlmQk/SjcrJySEnJweAqVOnEhsbW68G/6+QkJBG7xuopM0+EhsLZ3WBa25CV1dTvaeIqi93UfXtV5gHf0CXHSCsV19aW/S3lvNsAx0ug8GXAVDt2UfV17uo+moX5g/7vZ+xwxWERkQSG9POp2/rdxefb7rpJtq2bUtVVRVz585l+fLljBkz5oTtMjIyyMjIqHm9b9++Rr1fbGxso/cNVNLmZhIeCef29v7vR5XAIYv+1nKebeiMRO//fkLHtGt0mzt16nTSn9d5u6rT6aS09L+rXZWWluJ0Ok+5TXV1NRUVFURHR5+wr8fjwel0nvaY7dq1QylFaGgogwcPrhl6EkII0TLqLAyJiYkUFxdTUlJCVVUVbreb5OTkWtv07duXvLw8APLz8+nRowdKKZKTk3G73VRWVlJSUkJxcTHdunU77TH3798PUHONIiEhwcdNFkIIcTp1DiU5HA7GjRvHlClTME2TwYMHk5CQwOLFi0lMTCQ5OZkhQ4aQlZXFxIkTiYqKIjMzE4CEhAQGDBjApEmTMAyD8ePHYxjeWnSyYwI8/fTTHDzofcr0rLPO4o477miutgshhDgJpW0yjeR3333XqP1sPyZ5EtLm4CBtDg5NaXOjrzEIIYQILlIYhBBC1CKFQQghRC1SGIQQQtRim4vPQgghfCPoewwPPvig1RFanLQ5OEibg0NztDnoC4MQQojapDAIIYSoxfHoo48+anUIq3Xt2tXqCC1O2hwcpM3BwddtlovPQgghapGhJCGEELVIYRBCCFGL3y3U05I2bdrEvHnzME2T9PR0Ro0aZXWkJtu3bx+zZ8/mhx9+QClFRkYGw4cPp7y8nBkzZrB3717at2/P/fffT1RUFFpr5s2bx8cff0x4eDgTJkwI2DFa0zR58MEHcTqdPPjgg5SUlDBz5kzKysro2rUrEydOJCQkhMrKSrKysti9ezfR0dFkZmYSFxdndfwGO3ToEHPmzOGbb75BKcXdd99Np06dbH2eV61aRW5uLkopEhISmDBhAj/88IOtzvMzzzzDxo0biYmJYdq0aQCN+vebl5fHkiVLABg9ejRpaWn1D6GDVHV1tb733nv1nj17dGVlpX7ggQf0N998Y3WsJvN4PHrXrl1aa60rKir0fffdp7/55hu9YMECvXTpUq211kuXLtULFizQWmv90Ucf6SlTpmjTNPWOHTv0Qw89ZFn2plq5cqWeOXOmfvzxx7XWWk+bNk2vXbtWa6313Llz9VtvvaW11nr16tV67ty5Wmut165dq6dPn25N4CaaNWuWzsnJ0VprXVlZqcvLy219nktLS/WECRP00aNHtdbe8/vee+/Z7jxv27ZN79q1S0+aNKnmZw09r2VlZfqee+7RZWVltf5/fQXtUFJhYSHx8fF06NCBkJAQUlNTKSgosDpWk7Vr167mG0Pr1q3p3LkzHo+HgoICBg0aBMCgQYNq2rphwwYuvfRSlFKcc845HDp0qGaxpEBSWlrKxo0bSU9PB7wLPW3bto2UlBQA0tLSarX5+LenlJQUtm7dig6wezAqKir49NNPGTJkCOBd6zgyMtL259k0TY4dO0Z1dTXHjh2jbdu2tjvP3bt3JyoqqtbPGnpeN23aRO/evYmKiiIqKorevXuzadOmemcI2qEkj8eDy+Wqee1yudi5c6eFiXyvpKSEL774gm7dunHgwAHatfMuGN62bVsOHDgAeP8OP1083eVy4fF4arYNFM8//zy/+MUvOHz4MABlZWVERETgcDgA7/KzHo8HqH3uHQ4HERERlJWV0aZNG2vCN0JJSQlt2rThmWee4auvvqJr167cfvvttj7PTqeTkSNHcvfddxMWFsYFF1xA165dbX2ej2voef3f/7799O9SH0HbY7C7I0eOMG3aNG6//XYiIiJq/U4phVLKomS+99FHHxETExOQY+aNVV1dzRdffMGwYcN44oknCA8PZ9myZbW2sdt5Li8vp6CggNmzZzN37lyOHDnSoG/BdtES5zVoewxOp5PS0tKa16WlpTidTgsT+U5VVRXTpk1j4MCB9O/fH4CYmBj2799Pu3bt2L9/f823JqfTWWv1p0D8O+zYsYMNGzbw8ccfc+zYMQ4fPszzzz9PRUUF1dXVOBwOPB5PTbuOn3uXy0V1dTUVFRVER0db3IqGcblcuFwukpKSAO9QybJly2x9nrds2UJcXFxNm/r378+OHTtsfZ6Pa+h5dTqdbN++vebnHo+H7t271/v9grbHkJiYSHFxMSUlJVRVVeF2u0lOTrY6VpNprZkzZw6dO3fmyiuvrPl5cnIya9asAWDNmjX069ev5ufvv/8+Wms+//xzIiIiAmp4AeCmm25izpw5zJ49m8zMTHr27Ml9991Hjx49yM/PB7x3aBw/v3379iUvLw+A/Px8evToEXDfrNu2bYvL5apZ0nbLli2cccYZtj7PsbGx7Ny5k6NHj6K1rmmznc/zcQ09r3369GHz5s2Ul5dTXl7O5s2b6dOnT73fL6iffN64cSPz58/HNE0GDx7M6NGjrY7UZJ999hmPPPIIZ555Zs0/ghtvvJGkpCRmzJjBvn37TrjdLTs7m82bNxMWFsaECRNITEy0uBWNt23bNlauXMmDDz7I999/z8yZMykvL6dLly5MnDiR0NBQjh07RlZWFl988QVRUVFkZmbSoUMHq6M32JdffsmcOXOoqqoiLi6OCRMmoLW29Xl+5ZVXcLvdOBwOzj77bO666y48Ho+tzvPMmTPZvn07ZWVlxMTEcP3119OvX78Gn9fc3FyWLl0KeG9XHTx4cL0zBHVhEEIIcaKgHUoSQghxclIYhBBC1CKFQQghRC1SGIQQQtQihUEIIUQtUhiEEELUIoVBCCFELf8PrUnKrhV08RQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "metadata": { "id": "agK_98cWwwdQ" }, "source": [ "### Training Loop\n", "\n", "Now we're ready to start the training process. First of all, let's split the original dataset using [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function from the `scikit-learn` library. (Though you can use anything else instead, like, [get_cv_idxs](https://github.com/fastai/fastai/blob/921777feb46f215ed2b5f5dcfcf3e6edd299ea92/fastai/dataset.py#L6-L22) from `fastai`)." ] }, { "cell_type": "code", "metadata": { "id": "216J9e39wwdR" }, "source": [ "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2, random_state=42)\n", "datasets = {'train': (X_train, y_train), 'val': (X_valid, y_valid)}\n", "dataset_sizes = {'train': len(X_train), 'val': len(X_valid)}" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fFgH5tuLwwdR", "outputId": "4e64cc2c-8377-46a1-d220-96fbee9a59bb" }, "source": [ "minmax = ratings_df.RATING.astype(float).min(), ratings_df.RATING.astype(float).max()\n", "minmax" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1.0, 5.0)" ] }, "metadata": {}, "execution_count": 35 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WlAmBo9Jys0m", "outputId": "59234538-cd74-426f-8543-72cac11d7ee0" }, "source": [ "n_users = ratings_df.USERID.nunique()\n", "n_movies = ratings_df.ITEMID.nunique()\n", "n_users, n_movies" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(943, 1682)" ] }, "metadata": {}, "execution_count": 36 } ] }, { "cell_type": "code", "metadata": { "id": "KHeaux79wwdS" }, "source": [ "net = EmbeddingNet(\n", " n_users=n_users, n_items=n_movies, \n", " n_factors=150, hidden=[500, 500, 500], \n", " embedding_dropout=0.05, dropouts=[0.5, 0.5, 0.25])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "onmMNylKwwdS" }, "source": [ "The next cell is preparing and running the training loop with cyclical learning rate, validation and early stopping. We use `Adam` optimizer with cosine-annealing learnign rate. The rate is decreased on each batch during `2` epochs, and then is reset to the original value.\n", "\n", "Note that our loop has two phases. One of them is called `train`. During this phase, we update our network's weights and change the learning rate. The another one is called `val` and is used to check the model's performence. When the loss value decreases, we save model parameters to restore them later. If there is no improvements after `10` sequential training epochs, we exit from the loop." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4mRf0N9kwwdT", "scrolled": false, "outputId": "50f21317-af76-4dd8-a8f0-5ab7b61cc726" }, "source": [ "lr = 1e-3\n", "wd = 1e-5\n", "bs = 50\n", "n_epochs = 100\n", "patience = 10\n", "no_improvements = 0\n", "best_loss = np.inf\n", "best_weights = None\n", "history = []\n", "lr_history = []\n", "\n", "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", "\n", "net.to(device)\n", "criterion = nn.MSELoss(reduction='sum')\n", "optimizer = optim.Adam(net.parameters(), lr=lr, weight_decay=wd)\n", "iterations_per_epoch = int(math.ceil(dataset_sizes['train'] // bs))\n", "scheduler = CyclicLR(optimizer, cosine(t_max=iterations_per_epoch * 2, eta_min=lr/10))\n", "\n", "for epoch in range(n_epochs):\n", " stats = {'epoch': epoch + 1, 'total': n_epochs}\n", " \n", " for phase in ('train', 'val'):\n", " training = phase == 'train'\n", " running_loss = 0.0\n", " n_batches = 0\n", " \n", " for batch in batch_generator(*datasets[phase], shuffle=training, bs=bs):\n", " x_batch, y_batch = [b.to(device) for b in batch]\n", " optimizer.zero_grad()\n", " \n", " # compute gradients only during 'train' phase\n", " with torch.set_grad_enabled(training):\n", " outputs = net(x_batch[:, 0], x_batch[:, 1], minmax)\n", " loss = criterion(outputs, y_batch)\n", " \n", " # don't update weights and rates when in 'val' phase\n", " if training:\n", " scheduler.step()\n", " loss.backward()\n", " optimizer.step()\n", " lr_history.extend(scheduler.get_lr())\n", " \n", " running_loss += loss.item()\n", " \n", " epoch_loss = running_loss / dataset_sizes[phase]\n", " stats[phase] = epoch_loss\n", " \n", " # early stopping: save weights of the best model so far\n", " if phase == 'val':\n", " if epoch_loss < best_loss:\n", " print('loss improvement on epoch: %d' % (epoch + 1))\n", " best_loss = epoch_loss\n", " best_weights = copy.deepcopy(net.state_dict())\n", " no_improvements = 0\n", " else:\n", " no_improvements += 1\n", " \n", " history.append(stats)\n", " print('[{epoch:03d}/{total:03d}] train: {train:.4f} - val: {val:.4f}'.format(**stats))\n", " if no_improvements >= patience:\n", " print('early stopping after epoch {epoch:03d}'.format(**stats))\n", " break" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "loss improvement on epoch: 1\n", "[001/100] train: 0.9756 - val: 0.8986\n", "loss improvement on epoch: 2\n", "[002/100] train: 0.8512 - val: 0.8780\n", "[003/100] train: 0.8775 - val: 0.8851\n", "loss improvement on epoch: 4\n", "[004/100] train: 0.8071 - val: 0.8705\n", "loss improvement on epoch: 5\n", "[005/100] train: 0.8338 - val: 0.8697\n", "loss improvement on epoch: 6\n", "[006/100] train: 0.7598 - val: 0.8624\n", "[007/100] train: 0.7931 - val: 0.8698\n", "[008/100] train: 0.7192 - val: 0.8733\n", "[009/100] train: 0.7555 - val: 0.8743\n", "[010/100] train: 0.6720 - val: 0.8844\n", "[011/100] train: 0.7104 - val: 0.8882\n", "[012/100] train: 0.6229 - val: 0.9149\n", "[013/100] train: 0.6686 - val: 0.8936\n", "[014/100] train: 0.5796 - val: 0.9359\n", "[015/100] train: 0.6257 - val: 0.9201\n", "[016/100] train: 0.5433 - val: 0.9525\n", "early stopping after epoch 016\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "EGrWJQGnwwdT" }, "source": [ "### Metrics\n", "\n", "To visualize the training process and to check the correctness of the learning rate scheduling, let's create a couple of plots using collected stats:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 282 }, "id": "y3vAtcy9wwdU", "outputId": "612b10fe-440e-4422-90ca-cef19cc5ce36" }, "source": [ "ax = pd.DataFrame(history).drop(columns='total').plot(x='epoch')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEJCAYAAACE39xMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUVfrA8e+5M+mVSUhCIHQQEAExdOkoCoigAiuC8mOLiKu7rmXtnbWytrUDKqwoFkBREEXpUQywKFVAQwkESIFUUib3/P4YCEQS0qZl8n6exyeZmTv3fWcw75w559xzlNZaI4QQot4zPJ2AEEII55CCLoQQPkIKuhBC+Agp6EII4SOkoAshhI+Qgi6EED7C6snghw8f9mT4cqKjo8nIyPB0GpXy9vzA+3P09vzA+3P09vzA93OMj4+v9DFpoQshhI+Qgi6EED5CCroQQvgIKehCCOEjpKALIYSPkIIuhBA+Qgq6EEL4iHpX0PVvv2AufM/TaQghhNepfwV9/6/oZZ+iU1M8nYoQQniVelfQVeKlYLGgf1jt6VSEEMKr1L+CHhYOF3ZH/7gGbZqeTkcIIbxGvSvoAKrXQDieAXu2ezoVIYTwGvWzoHftBQFB6A3S7SKEEKfVz4IeEIDq3hu9cT26pNjT6QghhFeolwUdQPUaBCfzYetGT6cihBBeod4WdDp0gfBITOl2EUIIoB4XdGWxoHoOgJ+T0QV5nk5HCCE8rt4WdDg128VuR29K8nQqQghRLXlFpS47d70u6LRoC7FNZbaLEKJeSE7NY9qS31i286hLzl+vC7pSytFK370NnZXu6XSEEKJCJaUmszYe5cnVqUQHW+kUG+aSOPW6oMOpbhet0T+u8XQqQghxjkM5xdyzfD9LfjnOqAsa8ezwFrSwBbskltUlZ3UjFdMEWl/g6Ha54lpPpyOEEGW++y2bN5OP4GcxeGBgU3o2c03L/LR630KHU6301H3oQ/s9nYoQQlBQUsoL6w/z0vdptLUF8tKIli4v5uArBb1HfzAM9IZVnk5FCNHA7ck8yR1L97Fmfw4Tu0Tz+NDmRAX7uSW2bxT0sAjHCowbVssKjEIIjzC1ZvHOTO79ej92UzNjWHMmXBSNxVBuy8EnCjqc6nbJyoC9OzydihDCQ9Jyi9mceoJSU7s17olCO0+sTOWdzekkNg3lxRGt6BTjmoHP86n3g6KnqW690AGB6A2rUe07ezodIYSb7c44yWMrD5JXbNIoyMqAFmEMbBVB60YBKOW6VvKWtHxeTDpMXrHJtB6xXNEu0qXxzsd3CnpAIOri3uiN69B/+AvKzz19VkIIz9t+tIDHV6USGWjh9oFt+HZnGl/uPs5nu47TLNyfgS3DGdgqnNhQf6fFtJua+T+ls3BHFk3D/Xl0SAItGwU67fy14TMFHRzdLvqHVbBtE1zc29PpCCHcYPPhPJ5ac4iYED8eH5rABc1j6RVjIbeolPUHclidksP7P2fw/s8ZdIgOYmCrcC5tHkZ4YO3L39G8YmauP8wvGYVc3jaCP10SS4DV8z3YPlXQ6dgNwiIwN6zCIgVdCJ/3/cFcnl93iISIAB4bkkDEWUU6LMDCFe0acUW7RhzLK2HNvhxW78vmzeSjzNp4lO7xIQxoGUGvZqE1Ksbr9ufw6oYjKODuS+O5tEW4C15Z7fhUQT+9AqNe/RW6IB8VHOLplIQQLrIqJZuXvk+jXVQgDw9OINTfUumxMaF+XNc5imsvtLHvRBGrUnJYuy+H5EOHCbQa9EkIZWCrCLrEBlc6K6XIbvL2xqN882s2F0QHcme/eKd24TiDTxV0ONXt8u0S9OYk1KWXeTodIYQLLN9zgtd/PELn2GAeGNiMIL/qtbCVUrRqFEirRoHc2K0x248VsHpfDkkHclmZkkNkoIX+LcMZ2DKctrbAssHNfccLeW7dYQ7lFHPdhVFc3yUaqxunI1aXzxV0WraDmCaOpQCkoAvhcz7bmcWczce4JD6Ef/ZvWuu+a4uh6BIXQpe4EG7uEcvGQ3ms3pfDst0nWLLrOE3D/RnQMpwgq8G8LemE+hs8OiSBbk2895u/zxV0xwqMg9BffIg+nolqFOXplIQQTqC15qNtmcz/OYO+zcP4R994/CzOaSX7Wwz6Ng+nb/Nw8opKSTqYy+qUbD74OQOAS+JDuL1PEyLrMJDqDt6dXS2p3gPRSz5A/7gGNXysp9MRQtSR1pr3/pfOop1ZDGkdzl97NXHZFZihARYubxvJ5W0jSc8vIS23mM6xwRgemlteE56fZ+MCKiYeWrWXtV2EcKKNh/J4eNkuVqVku/VKTFNr3kw+yqKdWVzZLpLberuumP9e4xA/usSF1ItiDj7aQgcc3S4fvoU+dADVtLmn0xGi3jqaV8zsTcfYkJpHoNXg290m83/O4JpONoa2jsDP4rp2YampeeWHNFam5HBNJxs3dmvssasw6wOfbKEDqB6XygqMQtRBcanJgq0Z/PWLFH46ks9N3Rqz7Obe3D+gKeEBFl7/8Sh/+ew3PtuZRaHd+YvilZRqnl9/mJUpOdzQJVqKeTX4bgs9PBI6dUP/uAY9ZhLK8NnPLiGcbuOhPN7eeJQjeSX0ax7G1EtiiA72w99q0CshjJ7NQvnpSAGfbM9kzuZjfLw9k9EXNGLEBY3OOx+8uorsJs+sPcSmw/lM7R7D1R1tTnhVvs9nCzqc6naZ/W/YuxPaX+jpdITwemd3rzQL9+exSqbpKaXo1iSEbk1C2JV+kk+2Oy6tX7gjixHtIxndwUZkUO3KS0FJKf9afYhtRwuY3jOO4e0i6/qyGoxqveNbtmzhnXfewTRNhg4dypgxY8o9np6ezuuvv05OTg6hoaHcdtttREV5frqg6tYL7R9wagVGKehCVKa41GTRjiw+2Z6JoeCmbo25qoOtWtMCOzQO4sFBCaQcL+TjbZks3JHFkl+Oc1nbSMZ2tNE4pPoL5eUVlfL4qoPsySzk732bMKhVRF1eVoNTZUE3TZPZs2fz4IMPEhUVxX333UdiYiLNmjUrO2bevHkMGDCAQYMGsW3bNubPn89tt93m0sSrQwUGobr1Rm9aj77+zyirrMAoxO9V1r1SU60aBXJP/6Ycyinm0+2ZfLX7OMv3HGdQqwiu7RRFfPj5L5PPLrTzyHcHOZhdzD39m9InwfVbtvmaKjuW9+7dS1xcHLGxsVitVvr27UtycnK5Y1JTU+nc2bEG+YUXXsjGjRtdky2QWVDC5sN5aF29aVOq90DIz4Vtm12WkxD10dG8Yv61OpUnVqViMRSPDUngnv5Na1XMz9Y03J/b+zThzavbMLxtJGv25XDrF7/x3LpD7DteWOFzMgtKuP+bAxzKKeaBgVLMa6vKFnpWVla57pOoqCj27NlT7pgWLVrw448/MmLECH788UdOnjxJbm4uYWHl/1FWrFjBihUrAHj66aeJjo6uccILk/bzXnIq7RqHcMMlzRjc7vxrKuj+w0h/92X8/vc9kcNGVnqc1WqtVT7u4u35gffn6O35gXtyLLKbzN+UytzkVCwGTO/XkvEXx1dr+mFN8ouOho4tmjAtv5gF/zvMwq1prNufS79WjbixRwKdmzhWKUzLKeTBL7ZxorCUF8Z2plvTunWzNOR/Z6cMik6ePJk5c+awatUqOnbsiM1mw6hgVsmwYcMYNmxY2e2MjIwax7qqTRDhljgW7cji0a9+4bW1vzGmo41hbSIqX9Phkn4UrfuG9IMHUEEVbwsVHR1dq3zcxdvzA+/P0dvzA9fn+Pvulf/rHkPjED+yj2e5NL9xHUK5slVrvtx9nCW7srg55ThdYoMZ2iaCuVvSKbKbPDY4gWYBJXV+/b7+7xwfH1/pY1UWdJvNRmZmZtntzMxMbDbbOcfcddddABQWFrJhwwZCQlyzgI2fxWBYm0iGtI4gOTWPT3dk8dbGo3ywNYNR7R3TpsIDyk+bUr0Gold+id78ParfUJfkJYQ3O3v2StPzzF5xpdAACxMuimZ0Bxtf7z3Bop1ZvJCURkSghRnDmnt8tx9fUGVBb9OmDWlpaRw7dgybzUZSUhK33357uWNOz24xDINFixYxePBglyV8mqFU2XzYneknWbgjkw+2ZrBwRybD2kYypoONmNBTfYGtL4DGcY6LjKSgiwbk7NkrCrixW2NGV3P2iqsE+Rlc3dHGiPaRJB3I5YLoIOLCvGtd8fqqyoJusViYOnUqM2bMwDRNBg8eTEJCAgsWLKBNmzYkJiayY8cO5s+fj1KKjh078sc//tEduQOO+bCdYoLpFBPM/hNFLN7pGF1ftvs4/VuEc00nGy0bBTrmpH+5AH0iExXp+SmVQjhLqanJLS7lxEk72UWlZBeWcqLQTnZhKev255zTveIt/CwGA2VaolMpXd3pIi5w+PBhl5w3Pb+EJbuyWL43m0K7SfcmIYxtYtLp37dhjJuKcfmYc57jrH43rTW5RaUE+RlOXePC1/sF3cHb84MzORbZzbKifHaBPudnkZ3colIqWivLoqB5ZABTLo5xWvdKfXoPvZnH+tDro8Yhfky9JJbxnaNZuuc4X+w6zkNppbTrcxdjtv2PPsN0nVdrM7UmPb+Eg9nFpOYUcTC7mEM5xaRmF5FbbBLsZ9CzaSh9m4fRrUmIV2wgK7xPXnEpuzNOsjP9JL9knCTj5D4y84srXRslyGoQEWghMtBKXJgfHRoHld2ODLQQEWgtux3ib9SbVQKFc/hkQT8tNMDC+M7RXN3Bxne/ZbN4czHPBQwnfvFuxnaJZVCrcPyraEWXlGrScos5mFNEanYxqdmO3w/lFFNceqZZFBFgoVmEP32bh9M03J8D2UVsOJjLqn05BFoNejQNoW/zMC6Jr9mGtMJ3aK05nFvCrvQCdmWcZFf6SQ5mF6MBQ0HLyAA6xYYTaNjPFOgAK5FBjp8RgRb5f0ecl08X9NMCrAZXtm/EZTGa7599gUUXjeHVDZr5P6UzqoONK9pFElRsZ0+m4w8sNbuI1JxiDmYXcySvuNzX2ZgQK83CA7goNpiEiACahfvTLCLgnJk1ALf0jGPb0QKSDuTy/cFc1u7PJcCiSDzVcr8kPrTaeyGK+qfIbrIns5Bd6SfZlVHAroxCcotKAQjxN+gQHUT/FuF0aBxEu6gggvyMetFdILxXgyjop1kjbfSNMejz0+ts/9uLLNx5nHlb0vng53Ts5pmLpSwKmoT50yLSn37Nw2gW4U9CRABNw/0JrKKFpPNz0csXodp2xNqlR9kCRjf3iGX7sTPFff2BXPwtikviQ+jbPJzEpiEE+9V9lTrhOen5JaeKt6P1nXK8kNNf4pqF+9OrWSgdooPo0DiIpuH+0h0inK5BFXRwrMDInBfokn+ArkM68VtWIatSsomzhWOz2mkW4U9cqH+Nd/TWpole9w160VzIy0UHhWA8/ioq0jFn/+wNaf+cGMuu9JOsP5jL9wdy+f5gHn6G4uL4EPo1D6NH01BCnLAEqag9U2vs5un/cPwsPfs+zUm7ya9Zp1rg6SfJPGkHIMCiaBcdxNhOUXRsHET76KAKv8EJ4WwNr6Bf3Avt7+9YgbFtJ1rbAmltC6zTV12dshtz/puwbw+064Qx7GrMt59HL5iFuvmec463GIoLY4O5MDaYP10Swy+ninvSgVx+TM3DakC3OEefe69mYYRKMTgvrTVFpZrsQjs5RaXkFJY6fpb955gRUqwPU1BUUq4o/75Ql5z6WZMd1mJCrFwYE0yHxo7Wd8vIALdtkSbE2RpeQQ8MRnXthU5eh57wpzqtwKhzs9EL56LXfQMRNtSf7kT1HIBSCjVyPPqz99F9h6AuSqz0HIZSdIwJpmNMMFO7x7Ans5CkA7kkHchh4w/5WNQRusaF0Kt1ISWFBRhKYTUUFoMzvyvHh4RFOe63GAqrUhgGpx5Xpx4/c5y/VRERYPHaHWAKSkrJyHcU6Owi+7lFuvD0Y6XkFpWWG6A+m6EgPODUoGJIAIFWhdUwsBqO987PUFgtqux2ufsNhdVCpY/5WRQtIgOIquNiVkI4S4Mr6ACq9yB08lrY/j/o2rPGz9elpejVy9CfvQ9FhajLx6KumoAKPLNOjLriGvSPazDffwPj0VdQgUFVntdQiguig7ggOogpFzdmb9bp4p7L6+v31TjPqgT7GSScGh9IiPAnITyAhIgAokOsbuvfzSsuJTW7mAPZRRzMdkz/PJhdREaBvcLjQ/wMwgIshAdYiAq20rJR4KmCbSE80HF/eIDV8TPQQoifUfahJQOOwtc1yIJOp4shNNzR7VLDgq737HB0r6SmQMeuGNf/BdUk4ZzjlNUPY/KtmM/ei/58Pmp8za6eVUrRLsox++HGbo0JibBxLD2DUn2mS8Buakq1ptR0XC34+9/Pd9zJEpPDuY6ZPBsP5bHi19Ky2IFWRbPwgPLFPiKAmBC/Wncl5BaVlivYB079nnXyTOH2tygSIvzpHBNMQmQAsSF+RJwu0oFWwvwtHr1kXQhv1yALurJaUYmXotevQJ8sqHQFxrPpE1noT99F/7AKbNEY0+6F7n3O22Wh2nVCDRiOXrEE3WsQqkWb2uWrFMH+Fpf2pecUlZJ6VsE9mF3Ez0cKWJmSU3aMv0XRNPx0S/5MsY8LOzOInFNo52AFLe7jhed+YHRrElz2raB5pD+NQ/xk5ocQddAgCzqc6nZZtRT9v+9RfStfsEvb7ejvlqCXfAj2EtTI8agrr0MFVG9lOHXtTeiffsSc9yrGfc+hLN45wBkeYClbE+ds+cWlp+bknynOuzIKWLP/TKG3GhAX6k9eya+cOFlSdn+Q1dGl0z0+tOwDoLmbu3SEaEgabEE/swLjaqikoOudP2F+8BakHYSLEjH+8CdUTOXrKFREBYeiJvwZ/daz6O++QF12tTOyd5sQf0tZv/7ZTpaYHMo505pPzSmmcUQIMQG6rHhHB1u9dtBVCF/UYAu6UgrVcwB66SfoE1mO7VVO0Vnp6I/moDeth8ZxGH99CNW1R+1jJfZDf5/omPXSvS8qqrEzXoJHBfkZtI0KpG3UmW8qMugohGc16OvOVa9BoE3HjBdAl5RgfvkR5kPT0T8no66eiPHYf+pUzMHx4WHcMA20xpz/RrX3QxVCiJposC10ANWkGbRoi96wmqL2nTDfeh6OpUH3PhjjpqKiY50XKyoGdfUN6I/nwOYkuKSf084thBDQwFvo4Niejv17OfHknWAYGH9/DMst9zm1mJfFGnoVNG+D+cFb6II8p59fCNGwSUHvPQjadiT0xukYj7yMuvBi18WyWDBuvBVyHFeYCiGEM0lBD4vA8s9nCBk7qU7LAFQ7Xou2qKFXoVd/hd67w+XxhBANR4Mv6J6grp4ItsaYc19F20uqfoIQQlSDFHQPUIFBGBOnQdpB9PJFnk5HCOEjpKB7iOraA3VJP/QXC9BHXbNZthCiYZGC7kHqD38GP3/Mea/K3HQhRJ1JQfcgFWlDXXMj/LIV/f13nk5HCFHPSUH3MDVgOLTpgP54Djo32y0x9e5tmO+/js7PdUs8IYR7SEH3MGUYGJP/CidPoj+a49JYOvs45ux/Yz53P3rVMvTCeS6NJ4RwLynoXkA1bY4afg36h5XoHVucfn5dWoq54nPMh25Bb1znWAJ40JXotcvR+/Y4PZ4QwjOkoHsJNXIcxDTB/O9r6OIip51X79mB+eQd6AWzoPUFGI/+B2PMJNTYGyEsAnP+m2jTdFo8IYTnSEH3Eso/AGPSdEg/gv7yozqfT+ccx5zzIuaz90JBHsYt92L87VFUrGM9dxUcghr3f5CyG71+RZ3jCSE8r0GvtuhtVMeuqD5D0MsXonv0RzVrWeNzlG1gvfh9KC5y7K40cnyFOyypXoPQa5ajF76H7t4HFRLmhFchhPAUaaF7GTVuKgQFO7peatgVovfuxJzxD/QHb0GrdhiPvoxxzY2VbpenlMKYeDMU5KMXyQCpEPWdFHQvo8LCUeP+CL/uQq/5qlrP0TknMN99CfOZf0JeLsa0f2L8/TFUXLOq4zVrhRo80tFSlwFSIeo1KeheSPUZDB27ohfORZ/IrPQ4bZZirlrqmL3ywyrUFddiPP4q6pJ+NdrLU42eKAOkQvgAKeheSCmFMekWsNsxP3i7wmP0b79gzrgL/f4b0LwNxiMvY1x7EyowqMLjzxsvOAR1nQyQClHfSUH3UiomHjVyPGxOQm/ZUHa/zs3BnPsfzKfuhpzjqL/cg/GPJ1BNEuoWr/cgaNvJMUAqV5AKUS9JQfdiavhYaNoCc/6bmPl5mKu/wnxwGjrpW9TlYzGeeA2jx6U16l6pNJZSGDfIAKkQ9ZkUdC+mrH4Yk2+FE5lkTLsO/d/XIKEVxkMvYYz7P1RgsHPjnT1Aun+vU88thHC9as1D37JlC++88w6maTJ06FDGjBlT7vGMjAxeffVV8vPzMU2TiRMn0r17d5ck3NCoNh1Ql41BbVoPf/gzqucAp7TIK403eiI6eS3m+29g3PssypDPfCHqiyr/Wk3TZPbs2dx///288MILrF+/ntTU1HLHfPrpp/Tp04dnn32Wv//978yePdtlCTdE6ropRL+9CKPXQJcWc5ABUiHqsyoL+t69e4mLiyM2Nhar1Urfvn1JTk4ud4xSioKCAgAKCgpo1KiRa7JtoJRSLi/k5eLJAKkQ9VKVBT0rK4uoqKiy21FRUWRlZZU7Zty4caxdu5Zp06bx1FNPMXXqVOdnKtxGBkiFqJ+cspbL+vXrGTRoEFdddRW7d+/mlVdeYebMmRi/639dsWIFK1Y4vsY//fTTREdHOyO8U1itVq/K5/fcnl90NLkjrqPgy4+JuGo8fm06VPkUeQ/rzttz9Pb8oGHnWGVBt9lsZGaeuVoxMzMTm81W7pjvvvuO+++/H4D27dtTUlJCbm4uERER5Y4bNmwYw4YNK7udkZFRp+SdKTo62qvy+T1P5KcvGwtrvibr1aerNUDqzBy11qC1Uwdlvf3fGLw/R2/PD3w/x/j4+Eofq/KvpU2bNqSlpXHs2DHsdjtJSUkkJiaek9y2bdsASE1NpaSkhPDw8FolK7yHpwZI9dHDmDPuxJz5gCxFIEQNVNlCt1gsTJ06lRkzZmCaJoMHDyYhIYEFCxbQpk0bEhMTufHGG3nzzTf58ssvAZg+fbpbB/GE66je7l1i19ywGj3vNdAmFBeh133j2HdVCFElpbXWngp++PBhT4U+h7d/TfNkfjo1BfOJO1ADhmPccEulx9UlR11chF4w27HCZNuOGH++C3P2C3BoP8aTr6NC6/6Nz9v/jcH7c/T2/MD3c6xTl4sQZVeQrv7KJVeQ6iOHMJ+6B73mK9SV12LcOQNla+xYq/2kzLQRorqkoItqUaOvdyyx+/4bTu3XNjesxnzyH3AiA+P2RzCuuQlldfQEqqYtUEOvQq/9Gp0ia7ULURUp6KJaVHAo6topThsg1cVFmPNeRc+aCQktMR56CXXRJefGvep6CG+E+f7raLO0znGF8GVS0EW1qT6DT11BOrdOV5DqI6mYT92NXrPc0cVy179Qtorn5KqgYMdm1vv3otd9U+uYQjQEUtBFtZXtQZqfh17831qdw9HFciecyDzTxWKxnD9uzwHQvjN64Tx0bk6t4grREEhBFzWiElqhhtR8gLR8F0urSrtYKoypFMbEaacGSOfWNnUhfJ4UdFFjNR0gLd/Fch3GXTMq7WKpNGbT5qhho9HrvkGn7K5t6kL4NCnoosbKDZAmfXveY8/MYsnE+NsjGNfcWGUXS6Vxr/oDRDQ69UEiA6RC/J4UdFErZQOkn1a8xK4uLsKc+59TXSytHV0snavXxVJpzMBgx1IE+/ei18oAqRC/JwVd1Mr5Bkj1kVTMf92FXvt1rbtYKo3bcwBccJFjpo0MkApRjhR0UWsVDZCaP6xydLFkH69zF0uFMZXCuP5mKDopA6RC/I4UdFEnavT1EBqO+f4b5Lz2NHr2v091sbxY5y6WSmM2bY4aemqA9LdfXBJDiPpICrqoExUcWrbE7slvPnd6F0ulca+a4Bggnf+mDJAKcYpTdiwSDZvqMxiyjhHRrQe5zdq4J2ZgMGrcVPTbz6PXfI0adKVb4grhzaSFLupMKYUx6g8EdOvl3rg9+jsGSBe5/wpS/esuSp++B713h1vjCnE+UtBFvVU208bNA6Rm0reYz98Pv+7C/OBt2VVJeA0p6KJeU/GnBkjXfo3+dZdLY+nSUsyPZqPfeQnadkJN+BMc+BW9KcmlcYWoLinoot5TV02ASJtLB0h1fh7mK4+jv/kMNWQUxt8eRQ0ZCU1boBf/F223uySuEDUhBV3UeyowGDX+j47W8prlTj+/TnNcKMWuragb/4px/V9QVivKsGCMnQzHDqOT3LeJthCVkYIufIJKvBQ6dDk1QJrttPPqrZswn7oLTuZj3PkkRv/Lyx/QpQe06YBe8iG6qMhpcYWoDSnowiecGSAtRC+s+wCp1hpz+ULMVx6H6FiMB/6Natep4rjX3AQnstArv6hzXCHqQgq68BmqScKZJXbrMECqi4vQc15Af/IuqntfjH8+g4pqXHnc9hfCRYnoZZ+g8/NqHVeIupKCLnyKGjUBIqMw59duiV19PBPzufvRP6xCjZmEuvkeVEBglc8zxk6GkwXo5Qtrk7YQTiEFXfiUMwOkv6FX12yAVP/2C+aMOyEtFePW+zFGjkcpVb24Ca1QPQegv/0cfSKrNqkLUWdS0IXPUYn9HAOki6s/QGomfYf53P3g54dx37Oobr1rHvfqG6C0FP3FhzV+rhDOIAVd+JxyA6SfvnfeY7VZivnxHPQ7L0KbDhgPzEQ1bVG7uI3jUAOGOy5yOnq4VucQoi6koAuf5BggvRq9fkWlA6S6IA/zlSfQXy9GDR6J8ffHUKHhdYs7cgJY/dCfvV+n8whRG1LQhc8qGyB9//VzBkgduyrdDTt/Rk2+FWPizShr3RcfVRGNHB8kyWvRB36t8/mEqAkp6MJnqcAgxwDpwRT06q/K7tdbNzmKeUEexj+ewBgw3Llxh4+FkDDMRfOcel4hqiIFXfg0ldgPOnZFL/4v5okszOWLMF95AqJjHP3l7S90fszgENSV18G2zehftjr9/EJURpEiG10AAByQSURBVAq68Gln9iAtIvOu/0N/8g50733qYqEY18UdPMLR3bNwLlprl8UR4mxS0IXPU02aoS4fg5mZjrp6IsbN/6zWxUJ1iukf4Nhv9bdf4KcNLo0lxGlS0EWDoMZMIvqthRij/lDti4XqHLPvUIhrirlwnux7KtxCCrpoEJRhYGkc596YFgvGmMmQdhD9/Sq3xhYNkxR0IVypex9o0Rb9+Xx0SbFbQ+uiInRWultjCs+Sgi6ECymlMK69CbLS0auXuS2uTj+C+eQdmA/fis4+7ra4wrOkoAvhYqpjV8fUyS8/Rp8scHk8/dsvmE/dDTnHoaQY/ZWsANlQVOvSuC1btvDOO+9gmiZDhw5lzJgx5R5/99132b59OwDFxcVkZ2fz7rvvOj1ZIeorY+yNmP+6E/3NYtToiS6LozcnYc76N0Q0wrj9EfTyT9Grl6GHj0FFRrksrvAOVRZ00zSZPXs2Dz74IFFRUdx3330kJibSrFmzsmOmTJlS9vuyZctISUlxSbJC1FeqVTu4pC/668/Qg0agwiOden6tNfqbzxzz7Fu2w/jrg44YIyegf1iFXvYp6vq/ODWm8D5Vdrns3buXuLg4YmNjsVqt9O3bl+Tk5EqPX79+PZdeeqlTkxTCFxhjJkFJEXrpx049ry4tRc9/E/3xHLi4D8ZdM8o+MFTjOFTfoeg1X6GzMpwaV3ifKlvoWVlZREWd+aoWFRXFnj17Kjw2PT2dY8eO0blz5wofX7FiBStWOHZHf/rpp4mOjq5Nzi5htVq9Kp/f8/b8wPtz9Hh+0dHkDBnJyVVf0Wj8FCwxTc45pKY5micLyJ75MMWbkgi+eiKhN05HGeXbaaWTbibj++8IWLmE8JvvrtNL8Ph7WA0NOce6Ly93lvXr19O7d28Mo+KG/7Bhwxg2bFjZ7YwM72kxREdHe1U+v+ft+YH35+gN+enLxsLq5WS++x+MqXec83hNctQnMh3r0hzch7phGkWDRlCUVcFuSYYfqt8wTn6zhKJBo867P2pVvOE9rIqv5xgfH1/pY1V2udhsNjIzM8tuZ2ZmYrPZKjw2KSmJfv361SJFIRoGZYtGDRnp6Nc+tL/W59Gp+xwrRh49jHHbgxiDRpw/7ohxoEAv/ajWMYX3q7Kgt2nThrS0NI4dO4bdbicpKYnExMRzjjt06BD5+fm0b9/eJYkK4SvUlddBYHCtl9fV2/+H+cw/QZsY9zyFuujcv8dzYtoao/oPd2z4kX6kVnGF96uyoFssFqZOncqMGTO444476NOnDwkJCSxYsICNGzeWHbd+/Xr69u3rtnUyhKivVEiYY830n35E791Zo+eaa5ZjvvwYRMdi3Pc8qnmb6scdcR0oA/2ltNJ9VbX60Lt370737t3L3TdhwoRyt8ePH++8rITwcWrYaPR3X2AufA/j7qeqbAhp03Rser3sU7jwYseKkUHBNYsZGYUadCX6uy/QI65DxVTeFyvqJ7lSVAgPUAGBqFF/gD07YNvm8x6rS4rRs2Y65pIPGI7x14dqXMzL4l5xLVit6C8W1Or5wrtJQRfCQ1T/y6BxnGMTDNOs8Bidm4P574fQyWtR196EmjS9TnufqohGqEEj0D+sRh9JrfV5hHeSgi6EhyirH+rqGyA1BZ289pzH9dHDmE/fDfv2ov5yD8YV1zpljEoNvwb8/NBLpJXua6SgC+FBqkd/aNYK/dn7aHtJ2f167w5HMS/Iw7jzSYwezrv6WoVHooaMQievQacddNp5hedJQRfCg5RhYFxzI6QfQa/9BgAzeS3mzIcgOAzjvudQbTs6P+7lY8E/EL3kQ6efW3iOU68UFULUQufu0P5C9Bcfkmfa0R/OgradMG69HxUa7pKQKiwcNfQq9LKP0SPGoZq1dEkc4V7SQhfCw5RSGGNvhJwT5H84C9VzAMY/HndZMS+Le/nVEBiEKa10nyEtdCG8gGrbETVyPCGNbBT0v+KcBbZcEjMkzDEffsmH6AO/oZq3dnnM39NFReDvLxckOom00IXwEsaYSYRce6NbivlpathoCArBXPKB22Kepn/dhXn3TehP3nV7bF8lBV2IBkwFhzq6XrZsQO/f67a4ev9ezJceg8KTjitXT2RW/SRRJSnoQjRwauhoCA7F/Nw9rXR9MAXz3w9DcAjGXTNAm7LvqZNIQReigVNBwY7Fwn5ORqfsdmksffgA5r8fgoBAjDufRLXvjOozBL36K2mlO4EUdCEEashICA3H/Hy+y2LoI4ccxdxidRTzxnGO2CPGSSvdSaSgCyFQgcGoK66BbZtrvKRvdehjaZgzHwTTxLjzCVTsmZUeVeM4VO/B0kp3AinoQggA1KAREBbh9Fa6zjzmKOYlxRj/eALVJOHc2CPHg1kqrfQ6koIuhABOLel7xbWw8yf07m1OOac+nuko5oUFGHc8XukVqapxnKMvfc1y9IkK9kUV1SIFXQhRRg28EiIaOWXGi84+7ijmudkYf38M1eL8uyupkeOh1I7+6tM6x26opKALIcqogADHnqe/bEXv+rnW59G52Y5ifiIT42+PoFpVvdewo5U+WFrpdSAFXQhRjhowHCJtmJ/PR2td4+fr/FzHPPOMoxi3PYRq26n6sUecaqUvl7702pCCLoQoR/n5Owrrnh2w86caPVcX5GO+8AgcOYhx6wOoCy6qWeyYJmfNeJFWek1JQRdCnENdehnYomvUSteFBZgvPQqp+zBuuQ914cW1iz1y3KlW+qJaPb8hk4IuhDiH8vNzDFL+ugu2n38TawBdVIj58uOwbw/GX+5GdelR+9gx8ada6cvQ2cdrfZ6GSAq6EKJCqu9QiIrB/Oz8rXRdXIT5nydh7y7Un+5Ede9T99inW+kyL71GpKALISqkrKda6fv2wM8bKzxGl5Rgvv4U/LIV9X9/w+jR3zmxY+JRvQZJK72GpKALISql+gyBxnEV9qVrewnmm8/Ats2oybdi9Bns3NijZMZLTUlBF0JUSlmtqFET4MCv8NOGsvt1aSnm2zPhpx9RE6dh9L/c+bFj4lG9BkorvQakoAshzkv1GgQx8ZiffYA2TbRZip7zImxOQo3/I8bgEa6LPXIClEgrvbqkoAshzktZLKir/gCpKRR9vwr93n/QP65GXXMjxmVXuzZ2bDyq96lWeo77W+n6RBbm+hVos9TtsWtDCroQokqqZ3+Ia0b2S4+jk75FXXU9xpXXuSd2WSvdvfPSdfZxzOcfQL/7MnrDGrfGri0p6EKIKinDgnH1RCgpRl15naPF7q7Ysaf60lctdVsrXefmODbjOJEJMU3Qn72PLilxS+y6kIIuhKgWlXgp0W8vQo2djFLKvbFHjj/VSl/s8lg6Pw/zxYch/QjGXx/EuGEaZB5Dr17q8th1JQVdCFFtluhYtxdzABXX9FQr/Ut0zgmXxSlbvuDQAYzp96E6dEF1uhg6dkV/+RG6IN9lsZ1BCroQol4400p3TV+6LipyLF+wfy/GzfegOl9S9phx7U2Ql+v168tIQRdC1AuOVvoAl7TSdUkx5qtnLV9wce/ysVu0RfXoj16x2KtXgZSCLoSoN8pa6V87r6Ws7SWYrz8NO39CTbmt0uUL1JhJUFqKXvKh02I7mxR0IUS9oeKaoXr2R69c6pRWetkVr1s3oiZNx+g7tPLYMU1QA65Ar/safeRQnWO7ghR0IUS94piXXlLnVnq5K14n/BFj4BVVxx41Hvz8MRfPq1NsV7FW56AtW7bwzjvvYJomQ4cOZcyYMecck5SUxMcff4xSihYtWvC3v/3N6ckKIYRqclYrffg1qLCIGp9DmyZ63muOK17HTsYYVr0rXlV4I9TlY9BLPkT/9guq9QU1ju1KVRZ00zSZPXs2Dz74IFFRUdx3330kJibSrFmzsmPS0tJYvHgxTzzxBKGhoWRnZ9cqGa01hYWFmKbp9qlRR48epaioyOVxtNYYhkFgYKBHpn8J4QvUyAnoH9egly9CXTelRs/VWqM/fAu97hvUqAkYI8bVLPblY9CrlmF++h7GXTO86u+4yoK+d+9e4uLiiI2NBaBv374kJyeXK+jffvstw4cPJzQ0FICIiJp/YgIUFhbi5+eH1VqtLw5OZbVasVgsbollt9spLCwkKCjILfGE8DWqSTNUjwHolV+ih4+tditda43+5F30yqWoy8eiRk+seezAYNSoCegP3oJtm+GiS6p+kptUWTmzsrKIiooqux0VFcWePXvKHXP48GEAHnroIUzTZNy4cXTr1u2cc61YsYIVK1YA8PTTTxMdHV3u8aNHjxIQEFDzV+Ek7vogsVqtKKXOef1VPacmx3uCt+fo7fmB9+foTfnZJ99MZvIaAtcuJ+zG6WX3ny/HvA9mkf/1IoJGXEfYn+6odetaj72BzO++QH3+PraBl6GMmg1Huup9dEoFM02TtLQ0HnnkEbKysnjkkUd4/vnnCQkJKXfcsGHDGDZsWNntjIyMco8XFRW5rZX8e1arFbvd7rZ4RUVF57z+84mOjq7R8Z7g7Tl6e37g/Tl6VX6Boage/SlY+gmF/YeXtdIry9Fc9gl64VzUpZdRdPUkijMz6xTeHD0R/fbzpC/9FKN3zTb3qMv7GB8fX+ljVX6s2Gw2Ms964ZmZmdhstnOOSUxMxGq1EhMTQ5MmTUhLS6tVskIIUV1q1AQoLkJ/ff41XswVnzmKec+BqMnTa9yirjB24qXQvDV6sfcs3FXlq2rTpg1paWkcO3YMu91OUlISiYmJ5Y7p2bMn27dvByAnJ4e0tLSyPvf6JDs7m3fffbfGz5s8eXKtB4KFELWnmiQ4ruBc+SU6N6fCY8zVX6EXzIbufVFT/44ynNMLoAzDsSSAFy3cVWVBt1gsTJ06lRkzZnDHHXfQp08fEhISWLBgARs3OjaO7dq1K2FhYdxxxx089thjTJo0ibCwMJcn72w5OTnMnTv3nPur6oqZN29erQeChRB1U9ZK/+bceelm0rfo/74GXXpg/PlOlJO7dL1t4S6lf7/zqxudHkw9raCggODgYADMD99GH0xxajyV0ArjD3+u8DGr1cqf//xnvv76a1q3bo2fnx8BAQFERESwd+9e1q1bx9SpUzl8+DBFRUX88Y9/ZNKkSQD06tWLZcuWkZ+fz6RJk+jZsycbN24kLi6OOXPmVDib5ezXWh1e1XdZCW/P0dvzA+/P0VvzM996Dv1zMsZTs2jcqjUZGRmYyWvRb8+EDhdh3PYQys/fJbH1/r2YT/4DNWI8xthJ1XqOx/rQG5L777+fFi1a8M033/Dggw+ydetWHn/8cdatWwfAzJkz+eqrr1i6dClz5swhK+vcRXpSUlK46aabWLlyJeHh4Sxd6h1fxYTwZb9vpestP6BnzYS2HTBufcBlxRzOXrjrM48v3OX+Cd/VVFlL2p26detG8+bNy27PmTOHZcuWAY5vFykpKecMECckJNC5c2cAunTpwsGDB92XsBANlIpvjkq8FP3dUk5e0BnzzWehRVuM2x9GBQS6Pv6YSejNSegvPkRNml71E1xEWujncXaXSFJSEmvXrmXJkiWsWLGCzp07V3hl6dnz6C0WC6Wl9WNzWSHqO0crvZCcFx+D+OYYf38UFVj9bs06xY5pghowHL3Wswt3SUE/S0hICHl5eRU+lpubS0REBEFBQezdu5fNmze7OTshxPmo+OaoAcOxtuuE8ffHUcGh7o0/aoLHF+7y2i4XT7DZbPTo0YMhQ4YQGBhY7kquQYMGMW/ePAYOHEibNm3o3r27BzMVQlRE3XALtujoctfOuC22Fyzc5bWzXNzN3VeKyiwX9/P2/MD7c/T2/MCzOerCAsz7b3Z0+dz5ZKVLC8gsFyGE8HKnF+7il62w3f3dslLQhRDCidSA4dA4DvPT99Cm6dbYUtCFEMKJlNXPsf9o6j70j6vdGlsKuhBCOJmnFu6Sgi6EEE7mqYW7pKALIYQLeGLhLinoddCuXTtPpyCE8GLGtTdBXi7663NXgnRJPLdEEUKIBqhs4a5v3LNwl9deKTpr41FSjhc69ZytGgXyp8TKN97417/+RXx8PFOmTAEcqytaLBaSkpLIzs7Gbrdzzz33MHz4cKfmJYTwXe5cuEta6GcZPXo0S5YsKbu9ZMkSxo0bx+zZs1m+fDkff/wxjz/+OB68uFYIUc+4c+Eur22hn68l7SqdO3cmIyODI0eOkJmZSUREBDExMTz66KNs2LABpRRHjhwhPT2dmJgYt+cnhKif1KgJ6KTvMBfPwzLtXpfF8dqC7imjRo3iyy+/5NixY4wePZqFCxeSmZnJsmXL8PPzo1evXhUumyuEEJUpt3BXym44a+E/Z5Iul98ZPXo0n332GV9++SWjRo0iNzeX6Oho/Pz8WL9+PampqZ5OUQhRD6nLx0BYhGNJABd120pB/50LLriA/Px84uLiiI2N5ZprruGnn35i6NChfPLJJ7Rt29bTKQoh6qGzF+4q/t8Gl8SQLpcKfPvtt2W/22y2cgOlZ9uzZ4+7UhJC+AA1YDh622aU1TWlVwq6EEK4ibL6Ybn9Yfyjo8EFa7ZLl4sQQvgIryroDWl+d0N6rUII9/Cqgm4Yhlu3gfMUu92OYXjVWy+E8AFe1YceGBhIYWEhRUVFle7F5yoBAQFumV+utcYwDAIDA10eSwjRsHhVQVdKERQU5JHY9WHzWyGEOB/53i+EED5CCroQQvgIKehCCOEjlJb5c0II4ROkhX7Kvfe6bklLZ/D2/MD7c/T2/MD7c/T2/KBh5ygFXQghfIQUdCGE8BGWRx999FFPJ+EtWrdu7ekUzsvb8wPvz9Hb8wPvz9Hb84OGm6MMigohhI+QLhchhPARUtCFEMJHeNVaLu6WkZHBq6++yokTJ1BKMWzYMEaMGOHptM5hmib33nsvNpvNK6dk5efn88Ybb3Dw4EGUUtxyyy20b9/e02mV88UXX/Ddd9+hlCIhIYHp06fj7+/v0Zxee+01Nm/eTEREBDNnzgQgLy+PF154gfT0dBo3bswdd9xBaGio1+Q3b948Nm3ahNVqJTY2lunTpxMSEuKR/CrL8bQlS5Ywb948Zs2aRXh4uFflt2zZMpYvX45hGHTv3p1JkyY5J6BuwLKysvSvv/6qtda6oKBA33777frgwYMezupcS5Ys0S+++KJ+6qmnPJ1KhV555RW9YsUKrbXWJSUlOi8vz8MZlZeZmamnT5+ui4qKtNZaz5w5U69cudKzSWmtt2/frn/99Vf9j3/8o+y+efPm6UWLFmmttV60aJGeN2+ep9KrML8tW7Zou92utXbk6sn8tK44R621Tk9P108++aS+5ZZbdHZ2toeyqzi/rVu36scff1wXFxdrrbU+ceKE0+I16C6XRo0alY00BwUF0bRpU7KysjycVXmZmZls3ryZoUOHejqVChUUFLBz506GDBkCgNVq9WiLrTKmaVJcXExpaSnFxcU0atTI0ynRqVOnc1rfycnJDBw4EICBAweSnJzsidSAivPr2rUrFosFgPbt23v876WiHAHee+89brjhBrcvw/17FeX39ddfc/XVV+Pn5wdARESE0+I16C6Xsx07doyUlBTatm3r6VTKeffdd5k0aRInT570dCoVOnbsGOHh4bz22mvs37+f1q1bM2XKFK9a791ms3HVVVdxyy234O/vT9euXenataun06pQdnZ22YdNZGQk2dnZHs6oct999x19+/b1dBrnSE5Oxmaz0bJlS0+nUqG0tDR27drFhx9+iJ+fH5MnT3Za3WnQLfTTCgsLmTlzJlOmTCE4ONjT6ZTZtGkTERERXj2ntrS0lJSUFC6//HKeffZZAgICWLx4safTKicvL4/k5GReffVV3nzzTQoLC1mzZo2n06qSUsrjLczKLFy4EIvFQv/+/T2dSjlFRUUsWrSICRMmeDqVSpmmSV5eHjNmzGDy5Mm88MILTtuSssEXdLvdzsyZM+nfvz+9evXydDrl/PLLL2zcuJFbb72VF198kW3btvHyyy97Oq1yoqKiiIqKol27dgD07t2blJQUD2dV3tatW4mJiSE8PByr1UqvXr3YvXu3p9OqUEREBMePHwfg+PHjHhvMO59Vq1axadMmbr/9dq/7wDl69CjHjh3j7rvv5tZbbyUzM5N//vOfnDhxwtOplbHZbPTs2ROlFG3btsUwDHJzc51y7gbd5aK15o033qBp06aMGjXK0+mcY+LEiUycOBGA7du3s2TJEm6//XYPZ1VeZGQkUVFRHD58mPj4eLZu3UqzZs08nVY50dHR7Nmzh6KiIvz9/dm6dStt2rTxdFoVSkxMZPXq1YwZM4bVq1fTo0cPT6dUzpYtW/jss8947LHHCAgI8HQ652jevDmzZs0qu33rrbfy1FNPedUHY48ePdi+fTudO3fm8OHD2O12wsLCnHLuBn2l6K5du3j44Ydp3rx5WUvj+uuvp3v37h7O7FynC7o3Tlvct28fb7zxBna7nZiYGKZPn+6xqXaV+eijj0hKSsJisdCyZUumTZtWNijlKS+++CI7duwgNzeXiIgIxo8fT48ePXjhhRfIyMjw+LTFivJbtGgRdru9LKd27drxl7/8xSP5VZbj6QF68HxBryi/AQMGlI05Wa1WJk+eTOfOnZ0Sr0EXdCGE8CUNvg9dCCF8hRR0IYTwEVLQhRDCR0hBF0IIHyEFXQghfIQUdCHq4NixY4wfP57S0lJPpyKEFHQhhPAVUtCFEMJHNOhL/4VvysrKYs6cOezcuZPAwEBGjhzJiBEj+Oijjzh48CCGYfC///2PJk2acMstt5StypeamsqsWbPYt28fNpuNiRMnkpiYCEBxcTEffvghP/zwA/n5+TRv3pyHHnqoLObatWtZsGABxcXFjBw5kmuuucYTL100cNJCFz7FNE2eeeYZWrZsyZtvvsnDDz/M0qVL2bJlCwAbN26kT58+zJkzh379+vHcc89ht9ux2+0888wzdOnShVmzZjF16lRefvllDh8+DMDcuXP57bffePLJJ3nnnXeYNGlSuYWpdu3axUsvvcRDDz3EJ598Qmpqqkdev2jYpKALn/Lrr7+Sk5PDddddV7ZN2tChQ0lKSgKgdevW9O7dG6vVyqhRoygpKWHPnj3s2bOHwsJCxowZg9VqpXPnznTv3p1169ZhmiYrV65kypQp2Gw2DMPgggsuKLcWzLhx4/D396dly5a0aNGC/fv3e+otEA2YdLkIn5Kens7x48eZMmVK2X2madKxY0eio6OJiooqu98wDKKiosqWq42OjsYwzrRxGjduTFZWFrm5uZSUlBAXF1dp3MjIyLLfAwICKCwsdOKrEqJ6pKALnxIdHU1MTEyF68Z/9NFHZGZmlt02TZPMzMyyHYIyMjIwTbOsqGdkZNCkSRPCwsLw8/PjyJEjXrsLjhAgXS7Cx7Rt25agoCAWL15McXExpmly4MAB9u7dC8Bvv/3Ghg0bKC0tZenSpfj5+dGuXTvatWtHQEAAn3/+OXa7ne3bt7Np0yb69euHYRgMHjyYuXPnkpWVhWma7N69m5KSEg+/WiHKk+Vzhc/Jyspi7ty5bN++HbvdTnx8PBMmTGDXrl3lZrnExcUxbdq0si3+Dh48WG6Wy/XXX0/Pnj0BxyyX+fPn8/3331NYWEjLli154IEHOHHiBH/961/54IMPyjZPfvTRR+nfv7/XbuwtfJcUdNFgfPTRRxw5csTrdn0Swlmky0UIIXyEFHQhhPAR0uUihBA+QlroQgjhI6SgCyGEj5CCLoQQPkIKuhBC+Agp6EII4SP+H1yyjenPc+ZxAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 265 }, "id": "8SNXU2CKwwdU", "outputId": "c4ba4191-0a03-4cff-8a2c-afc478a1658e" }, "source": [ "_ = plt.plot(lr_history[:2*iterations_per_epoch])" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVxVdf7H8df3ey8uLC4XBNKhJhFnCi0TTKRFFKopmxlqrKlfy7g0Zm6R2ZQyU7ZQtiiOYjtZmc1YU1RTTTZItkgUZtjYKtrGiJJcUxRL4Hx/f9xivAmieOHcA5/n49GjLpx7eJ/T1Tfn+z2LMsYYhBBCiB9ouwMIIYQILlIMQggh/EgxCCGE8CPFIIQQwo8UgxBCCD9SDEIIIfy47Q4QKFu2bGnV+6Kioti+fXuA07QfJ+d3cnZwdn4nZwdn5w+m7H379m3y63LEIIQQwo8UgxBCCD9SDEIIIfxIMQghhPAjxSCEEMLPIZ2VVFZWxtKlS7Esi/T0dDIzM/2+X1dXR15eHps3byYiIoKsrCyio6MBKCgooKioCK0148ePZ8iQIQDce++9rFu3jp49ezJ//vzGde3evZvc3Fy++eYb+vTpwzXXXEN4eHigtlcIIUQLWjxisCyL/Px85syZQ25uLmvWrKGiosJvmaKiIsLCwli8eDFjxoxh+fLlAFRUVFBcXMyCBQvIzs4mPz8fy7IASEtLY86cOQf8vOeee47BgwezaNEiBg8ezHPPPReI7RRCCHGIWiyG8vJyYmNjiYmJwe12k5qaSmlpqd8ya9euJS0tDYCUlBQ2bNiAMYbS0lJSU1MJCQkhOjqa2NhYysvLATj++OObPBIoLS1l5MiRAIwcOfKAnxVIVslr7F6Rj/Xy01ivPodV9CLWGysxa9/CfLwe89VmjPcbTN2+NssghBCtYSq+wHp+OaZmZ8DX3eJQktfrJTIysvF1ZGQkGzdubHYZl8tFaGgoNTU1eL1eEhISGpfzeDx4vd6D/rydO3fSu3dvAHr16sXOnU1vdGFhIYWFhQDMmzePqKioljblADvKStjz3tsHfL2pB1ToXh50n1hcP/7TNw730f1xxx2LDrNvqMvtdrdq24OBk7ODs/M7OTs4O3+gsu/9eB27XlyB51eZuAO8L4L6ymelFEqpJr+XkZFBRkZG4+tWXUk4eTbRHg/bt22D+jqor4e6fbB3D+yugT27MLtrYNe3GO831FdXUV/+Cbz7pm/5H3mioO8xqJ8noPr/AvoPRIVFHH6eVgimqygPl5Ozg7PzOzk7ODt/oLJbu2oA2LFjB6prWKvW0dyVzy0Wg8fjobq6uvF1dXU1Ho+nyWUiIyNpaGigtraWiIiIA97r9XoPeO9P9ezZkx07dtC7d2927NhBjx49Wop4RJTWqJAQCAnZ76v/a9+maslYFlRXwZavMFu+8v37688xH76PMb45FGL6oeJ/CcediDruRFTP3m26HUIIESgtFkN8fDyVlZVUVVXh8XgoLi5mxowZfsskJSWxevVqBg4cSElJCYmJiSilSE5OZtGiRZx77rns2LGDyspKBgwYcNCfl5yczOuvv05mZiavv/46w4YNO7ItbANKa+gTC31iUSee3Ph1891e+LIcs/lTzKZPMOvfheJVvqGpfsf4CmJQEvxiEMod0uz6hRDCTi0Wg8vlYsKECeTk5GBZFqNGjSIuLo4VK1YQHx9PcnIyo0ePJi8vj+nTpxMeHk5WVhYAcXFxjBgxgpkzZ6K1ZuLEiWjtm+9euHAhH330ETU1NUyePJkLL7yQ0aNHk5mZSW5uLkVFRY2nqzqF6tYdfjEY9YvBwA9HFl9/jvm4zDeZ/formMIXoHsYanAy6qThMCjJ9z4hhAgSyhjT1Fyr4zjh7qpm3/fw8XrM+yW+o4ndu8AdAicko1NG+Uoi5PCOJGSs1T5Ozu/k7ODs/AGbY3jndczD89G33ouK/Vmr1tHqOQYROKpLVzjxZNSJJ2MaGmDTx5h1b2PefQNr3dsQGo4adioqJQ3ij2t24l0IIdqSFINNlMsFAwehBg7CXDABPirDlKzGvF2Eef0VOCoOlXY2KmUUKrR1ZxwIIURrSDEEAeVyweAk1OAkzHe1mLVrfPMRf3sQ88xjqOEjfSVxdLzdUYUQQSfwIwtSDEFGdQtFnXoGnHoG5styzOp/Yd5ZjXnzVRg4CH3Web65CC33PxRCtA0phiCmjhmA+sN0zAXjMW8VYla9gLX4Vt8w01nnoU4eaXdEIUQHJL92OoAKDUefmYnOeRA18RpwuTCPLsKa80dq/7nCd7aTEEIEiBwxOIhyu1EpozDD0+CjMqx//YOaR/4KzyxDnTMWddqZqJAudscUQjicFIMDKaUg8SRciSfRY+tX7Fh2n2+i+l/PoM65AHXaGXJltRCi1WQoyeG6DBqKnnU7euatEBWDefJ+rBunYpW+RQe5dlEI0c7kiKEDUErBcSeif3kCfPg+1jOPYh68C/PvgegLJqASjrc7ohAi0NrwFz8phg5EKQWDhqKPPxHz9mrMc8uw7roBhqSgf/cHVGw/uyMKIRxAhpI6IKVd6FPS0bc9gMq8FD5ejzV3Otazj2G+/87ueEKIQGqDW+dIMXRgqmtX9JgL0bffjzr5dMy/nsH6yxTfo0tl/kEI0Qwphk5A9eiNnpCFvn4ehEdgPXAXVu6NmMoKu6MJIYKQFEMnogYcj85egLp4EnxRjnXzDKznn8TU1bX8ZiFEpyHF0Mkolws9+lz0bfehkk/BvPh3rFuzMJs+sTuaECJISDF0UqpHL/QV16Jn3ATf78W683qsvz3oezypEKJTk2Lo5NTgJPTNeai0czCvvYR10zTMhnV2xxJC2EiKQaC6haL/70r0n+ZB125Yf52L9cS9cmqrEEGt7c4slGIQjdSA49B/yUWdmYl5YyXWzTNk7kGIYNcGTwCWYhB+VEgX9AUT0NfmgGVh3XkDVsEyTL2cuSREZyHFIJqkfjEIfdMiVOpozMtPY90+C/Pfr+yOJYRoB1IMolmqeyh63Az01DnwrRcrZybWGyvlqmkhOjgpBtEiNSQFfdMiSDges2wJ5sG7MbV77I4lhGgjUgzikKievdFXz0WdfzlmXTHWLVdjNn9qdywhRBuQYhCHTGmNPnus77RWwLrrBqyVz2Isy+ZkQohAkmIQh03F/xJ940I4cTjmH49iLb4Vs6fG7lhCdC5tONUnxSBaRYWGoydfj7pksu95D7deg/lyk92xhOiE5HkMIogopdBp56D/dIfvmod5f8J66992xxJCHCEpBnHEVP9foP+S6ztr6bHFWI/nYer22R1LCNFKUgwiIFRET3TWXNQ5F2DefBXrzhsw27fZHUsI0QpSDCJglHahz7sMPTUbqiqxbpuJ+Xi93bGEEIdJikEEnBoyHP3n+dCjF9bCm7Bee0mulhbCQaQYRJtQ0X3Rs++GwcmYJx/APHGv3IhPCIeQYhBtRnUPRU+Z45t3eGMlVu6NmJqddscSomNow6Nw96EsVFZWxtKlS7Esi/T0dDIzM/2+X1dXR15eHps3byYiIoKsrCyio6MBKCgooKioCK0148ePZ8iQIQdd53/+8x+eeOIJLMuiW7duTJ06ldjY2EBus2hHSmvUeZdh9T3ad8ZSzrXoadmonx1rdzQhOgZlw3UMlmWRn5/PnDlzyM3NZc2aNVRUVPgtU1RURFhYGIsXL2bMmDEsX74cgIqKCoqLi1mwYAHZ2dnk5+djWdZB1/nwww8zffp07r77bk499VSeeeaZgG+0aH96+Ejf9Q4NDVjzrsese9vuSEKIZrRYDOXl5cTGxhITE4Pb7SY1NZXS0lK/ZdauXUtaWhoAKSkpbNiwAWMMpaWlpKamEhISQnR0NLGxsZSXl7e4zr17fQ+kr62tpXfv3gHcXGEn9fMEdPZ86HcM1v3z2PP8kzIpLUQQanEoyev1EhkZ2fg6MjKSjRs3NruMy+UiNDSUmpoavF4vCQkJjct5PB68Xm/jeppa5+TJk7njjjvo0qUL3bt3Jycnp8lchYWFFBYWAjBv3jyioqIOaYN/yu12t/q9wcBx+aOiMLffz85Ft7L70Ty6V/6XiD9eg3Id0qhmUHHcvt+Pk7ODs/MHKvveiAh2Ab1798Yd4H0RdH8aX3rpJWbPnk1CQgIvvPACjz/+OJMnTz5guYyMDDIyMhpfb9++vVU/LyoqqtXvDQZOzW/+MIPQmL7UFjzB3v9+hb7yOlS3ULtjHRan7ntwdnZwdv5AZbdqfDeu3LFjB8rdtVXr6Nu3b5Nfb3EoyePxUF1d3fi6uroaj8fT7DINDQ3U1tYSERFxwHu9Xi8ej6fZde7atYsvv/yy8SgjNTWVTz+Ve/53REprIi6fgrpsKnz0vu9Kaa8z/6AL0dG0WAzx8fFUVlZSVVVFfX09xcXFJCcn+y2TlJTE6tWrASgpKSExMRGlFMnJyRQXF1NXV0dVVRWVlZUMGDCg2XWGhYVRW1vLli1bAPjggw/o169f4LdaBA19+lnoGTfB9m1Yd8zCfCV3aBXCbi0OJblcLiZMmEBOTg6WZTFq1Cji4uJYsWIF8fHxJCcnM3r0aPLy8pg+fTrh4eFkZWUBEBcXx4gRI5g5cyZaayZOnIjWvi5qap0AV155JfPnz0drTVhYGFdddVUbbr4IBirxJPT1d2ItvgXrrtnoSdehThhmdywhglzbnbihTAc5LeTHo4zD5eSxSnB2/p9mN99WYy2+DSo+R102FX3qGTama1lH2vdO4+T8AZtjKF6FWfpX9O0Povq07lqvVs8xCNFeVK9I9HW3w3En+i6Ge+kpOZ1VCBtIMYigorp1R0/7MyolDfPcE5i/PYCxGuyOJUSnEnSnqwqh3CEwPgt69sasLMDs+hY9cSYqpIvd0YToFKQYRFBSWqPGjsfq6cE8lY9Vsws9NRsVGmZ3NCE6PBlKEkFNn/Fb1BXXwqZPsO6ejfm2uuU3CSGOiBSDCHp6+Ej0jBvhm22+G/BtrWj5TUKIVpNiEI6gjh/iO2Np3/dYd82WC+GEaENSDMIx1DHx6OvvhJAuWPdkYzZ+ZHckIezThmdySzEIR1ExfdHXz4OevbEW3ojZ8J7dkYSwlx0P6hEi2ChPH/R1d0BMP6y8HMzat+yOJESHIsUgHEn16IWelQPHJmA9eA/Wm6/aHUmIDkOKQTiWCg1HZ90CiUMwj+dhvfqc3ZGE6BCkGISjqa5dfRe+JZ2CefoRrOeXy/2VhDhCcuWzcDzlDoFJs2BZKObFFVC7B35/BUrL7z1CtIYUg+gQlHbB5dOgeyjm389D3T64dIqUgxCtIMUgOgylFFwwAUK6Yl5+CurrYNwMX2kI0eG03ZCpFIPoUJRSqPMuxQoJwTy/HOrrYcI1KLd81EUH1QbXMcifFtEh6XN/7yuHfzyKqa/zPS7UHWJ3LCEcQQZgRYelzzofddEf4f0SrHvvwNTtszuSEI4gxSA6NJ3+a9SlU+A/a7HybsN8/73dkYQIelIMosPTI3+FGnc1fLwea9HNmO/22h1JiKAmxSA6BX1KOmriTCj/CGvhTZjaPXZHEiJoSTGITkMPH4me9Cf4YiPWgr9g9tTYHUmIoCTFIDoVlZSKvmo2/PcLrAU3SjkI52rDW79IMYhOR514MnrKHNjypZSD6ADkeQxCBIQanLxfOciwkhD7k2IQnZavHLJhy1dSDkLsR4pBdGpqcNIP5fC1lIMQP5BiEJ2eGpyEnjpHykGIH0gxCAGoQfuVw/w/Y3bvsjuSELaRYhDiB43lUFnhO3KQchCdlBSDEPvxlUO2lIMIfnIdgxDtRw0aKuUgOjUpBiGa4FcO86UcRBAL/PVtUgxCNEcNGoqe9mfYKuUgOpdDeoJbWVkZS5cuxbIs0tPTyczM9Pt+XV0deXl5bN68mYiICLKysoiOjgagoKCAoqIitNaMHz+eIUOGHHSdxhj+/ve/U1JSgtaaM844g3POOSeQ2yzEIVOJJ6Gn/Rkr7zas3BvRM29DhYXbHUuINtXiEYNlWeTn5zNnzhxyc3NZs2YNFRUVfssUFRURFhbG4sWLGTNmDMuXLwegoqKC4uJiFixYQHZ2Nvn5+ViWddB1rl69murqanJzc8nNzeWUU05pg80W4tCpxJN+uH3GV1i5N2Jqd9sdSYg21WIxlJeXExsbS0xMDG63m9TUVEpLS/2WWbt2LWlpaQCkpKSwYcMGjDGUlpaSmppKSEgI0dHRxMbGUl5eftB1vvrqq4wdOxatfdF69uwZ4E0W4vCpwUnoybOh4gushXMxe2vtjiREm2lxKMnr9RIZGdn4OjIyko0bNza7jMvlIjQ0lJqaGrxeLwkJCY3LeTwevF5v43qaWue2bdsoLi7m3XffpUePHowfP56jjjrqgFyFhYUUFhYCMG/ePKKiog55o/fndrtb/d5g4OT8jsuefjbfhYex8+5sXPfmoG9e5Kz8+3Hcvv8JJ+cPVPa9ERHswvf3qivA++KQ5hjaU11dHSEhIcybN4933nmH++67j1tuueWA5TIyMsjIyGh8vX379lb9vKioqFa/Nxg4Ob8js8cfj/7jddQ9eBfbb86iYUo2qms3u1MdNkfu+/04OX+gsls1vlu3eL1eFK5WraNv375Nfr3FoSSPx0N1dXXj6+rqajweT7PLNDQ0UFtbS0RExAHv9Xq9eDyeg64zMjKS4cOHA3DyySfz5ZdfHuo2CtEuVFIq6oprqfvkP1iLb8V8/73dkYQIqBaLIT4+nsrKSqqqqqivr6e4uJjk5GS/ZZKSkli9ejUAJSUlJCYmopQiOTmZ4uJi6urqqKqqorKykgEDBhx0ncOGDWPDhg0AfPTRR802mhB20sNOo8fVN8JnH2ItuQ2zT8pB2CXwFzK0OJTkcrmYMGECOTk5WJbFqFGjiIuLY8WKFcTHx5OcnMzo0aPJy8tj+vTphIeHk5WVBUBcXBwjRoxg5syZaK2ZOHFi46RyU+sEyMzMZNGiRbz00kt069aNK6+8MuAbLUQgdD/9TGq+/Rbz6F+x7r0dPTUbFdLF7lhCHDFlTBvecKMdbdmypVXvc/JYJTg7v5Ozw//yW2/9G/PYYhicjL5qNiokxO5oLeoo+96JAjbH8OarmMfz0Hc+gvK0bvK51XMMQoiD06eegbpsCvxnLdYDd2Lq6+yOJMQRkWIQIgD06b9CXTIZ1r+L9eDdmPp6uyMJ0WpSDEIEiE47B3XRJHi/BOvhezANDXZHEqJVpBiECCCdfi7qwonwXjEmf4GUg3CkoLvATQin02f8FstqwPzjUXC5YPzVKN26C5CEaFYbnjckxSBEG9BnnY9VX4957glQGsbNQGk5QBdtQNlwHYMQonX0mAuxLAvzwpO+I4fLpko5CEeQYhCiDelfX4TVUI956SnQLrj0KlQb/IYnRCBJMQjRxtRvLwGrAfOvZ8Cl4eIrpRxEUJNiEKKNKaXgvMuhwcK8WuA7cvj9FVIOImhJMQjRDpRSMHYcNNRjVv3TN+cwdryUgwhKUgxCtBOlFPz+Ct+w0qvP+c5W+t0fpBxE0JFiEKIdKaXg4ivBGMzKZ0H7hpmkHMThk+sYhOgwGsvBMr4JaaUh81IpB9E6bfCxkWIQwgZKa7hkMhgL8/LTvouUfnuJlIMIClIMQthEaQ2XTvENK730FGiN+s3/2R1LCCkGIeyktIbLpvqOHP75dywU+jcX2x1LdHJSDELYTGkNl08HA+aff8PSCn3uRXbHEp2YFIMQQUBpDX+Y5jtyeP5JLKXRYy60O5bopKQYhAgSSrtg3AzfnMNzT2AphT7nArtjiU5IikGIIKK07/kNWAZTsMx35HD27+yOJYJR213GIMUgRLBR2gUTsgCDefYx35zDWefbHUt0IlIMQgQh5XLBhGt8w0r/eNQ3rHTmeXbHEkFJHtQjRKehXC6YOBMsC/P0Ut+w0hm/tTuW6ASkGIQIYsrlgiuuxWAwT+X7jhwyfmN3LNHByXMGhQhyyu1GXzELho7ArHgYa9WLdkcSHZwUgxAOoNxu9B+vg5NSMH9/EKtIykG0HSkGIRxCud3oSdfBkOGYvz2IteqfdkcSHZQUgxAOotwh6Cv/9MORw0NYrxbYHUl0QFIMQjiMcoegJ/0JlXSK72yll5+2O5Kwg5EH9Qgh9qPcbvjjLHC5fVdINzSgfy033uuU2uAZHlIMQjiU7zqHLHC5MC88idVQj5KH/YgAkGIQwsEab7zndvse9lNfD7/7g5SDOCJSDEI4XOOT4FwuzMpnoaEeLpwo5SBaTYpBiA5AaQ3/N9k351D4gq8cLprk+7oQh+mQiqGsrIylS5diWRbp6elkZmb6fb+uro68vDw2b95MREQEWVlZREdHA1BQUEBRURFaa8aPH8+QIUMOaZ2PPPIIr732GsuWLQvEdgrR4Sml4PdX+I4cXn0OGhrgkqukHMRha/ETY1kW+fn5zJkzh9zcXNasWUNFRYXfMkVFRYSFhbF48WLGjBnD8uXLAaioqKC4uJgFCxaQnZ1Nfn4+lmW1uM5NmzaxZ8+eAG+qEB2fUgo1djzq7LGYN1ZiHl+MsRrsjiUcpsViKC8vJzY2lpiYGNxuN6mpqZSWlvots3btWtLS0gBISUlhw4YNGGMoLS0lNTWVkJAQoqOjiY2Npby8/KDrtCyLJ554gksvvTTwWytEJ6CUQp13GercizBrVmEeWYhpkHLoeGy8jsHr9RIZGdn4OjIyko0bNza7jMvlIjQ0lJqaGrxeLwkJCY3LeTwevF5v43qaWucrr7xCUlISvXv3PmiuwsJCCgsLAZg3bx5RUVEtbUqT3G53q98bDJyc38nZwQH5J85gT8+e7F7+AF0w9Lz2FlRIF8AB2Vvg5PyByl4bHk4N4ImMxNXLc+TB9hNUk89er5e3336buXPntrhsRkYGGRkZja+3b9/eqp8ZFRXV6vcGAyfnd3J2cEj+tDGoBsP3f3+QqrlZ6ClzUF27OSP7QTg5f6CyW7t3A+D1VqPqrVato2/fvk1+vcWhJI/HQ3V1dePr6upqPB5Ps8s0NDRQW1tLRETEAe/1er14PJ5m1/nFF1+wdetWZsyYwdSpU9m3bx/Tp08/vC0VQvjR6eeixl0NH3+AlXsjpna33ZFEkGuxGOLj46msrKSqqor6+nqKi4tJTk72WyYpKYnVq1cDUFJSQmJiIkopkpOTKS4upq6ujqqqKiorKxkwYECz6xw6dCgPPfQQS5YsYcmSJXTp0oXFixe3yYYL0ZnoU9LRV14HX5Rjzf8z1s4ddkcSQazFoSSXy8WECRPIycnBsixGjRpFXFwcK1asID4+nuTkZEaPHk1eXh7Tp08nPDycrKwsAOLi4hgxYgQzZ85Ea83EiRPRP5w619Q6hRBtRyWdgu7SDeu+O/BmT8FcPRfVO7LlN4pORxnThrfoa0dbtmxp1fucPFYJzs7v5Ozg3Pzmsw2YvNswYRHombei+sTaHemwOXXfQwDnGFa/jFl+P3r+Y6geBz9ZpzmtnmMQQnQsauAget+yGPbWYt15A2bLV3ZHEkFGikGITihkwHHo624HDNbdszFfltsdSRyuNhzrkWIQopNS/Y5B/+kO6Nod655szMfr7Y4kgoQUgxCdmIrui77+ToiMxlp0M2btW3ZHEoct8HfRlWIQopNTvSPR190BPx+I9eDdWK+9bHckYTMpBiEEKiwcfc3NcMIwzJP3Yz2/nA5ywqJoBSkGIQQAqktX9FWzUaeegXlxBWbZErn5XicVVPdKEkLYS7lccPk06NEb8/JTmJpd6EmzGm++JzoHOWIQQvhRSqHPuxR10SRY/w7Wwpvk/kqdjBSDEKJJOv1c1BXXwqZPse6ajfF+Y3ck4aft5oCkGIQQzdInn46++ibwfoN1+3WYrzbZHUm0AykGIcRBqeNO9F3r4NK+I4f/rLU7ktifkusYhBA2UP2OQc++G2L6YS2+DWv1v+yOJNqQFIMQ4pCoXpG++ysNGopZfh/WP5ZirNY9OUwENykGIcQhU926o6dmo9LOxqwswDx4N2bf93bHEgEm1zEIIQ6Lcrng/yZDn1jM00sx31b7niXdo5fd0USAyBGDEOKwKaXQZ56Hnnw9fLUZ6/ZZmIrP7Y4lAkSKQQjRairpFN8N+BrqseZdj3m/xO5InUcb3stKikEIcUTUsQno7PnQ92ise2/HenGF3IDP4aQYhBBHTPWKRM/KQQ0fiXl+OeahezDfy6R0+wj8dQwy+SyECAjVpStMnAn9jsEULMNUVfrOYOodaXc0cZjkiEEIETBKKfTZY9FTs2Hrf7FyZmLKP7Y7ljhMUgxCiIBTJ56Mnn0XdOmKdc8crKIXZd7BQaQYhBBtQvU7Bv3nBZA4FPO3BzH5CzDff2d3LHEIpBiEEG1GhYb75hl+ewnm3Tew7rgOU7XF7liiBVIMQog2pbRGn/t79Iyb4Fsv1m3XYta/a3cscRBSDEKIdqEGDfUNLUUfhZV3G1bBMnmm9JFowykbKQYhRLtRUTHo6+ehTjsT8/LTWPfMkSfDHanAX8YgxSCEaF8qpAv68mm+x4Z+/QXWzVdjyuRWGsFEikEIYQs9fCT6L7kQFYO15Hasvz2IqauzO5ZAikEIYSMV0xd9w12o9F9jil7EmncdZpuctWQ3KQYhhK1USAj6oj/6rpau/gbr1iysN1bKBXE2kmIQQgQFNWQ4+sa/Qv9fYJYtwVp8K2bnDrtjdUpSDEKIoKE8Ueism1G/vwI++QBr7jTMe8V2x+p0pBiEEEFFaY3O+I1vYjoyBuv+eViP5GJq99gdLci03VDbId12u6ysjKVLl2JZFunp6WRmZvp9v66ujry8PDZv3kxERARZWVlER0cDUFBQQFFREVprxo8fz5AhQw66zkWLFrFp0ybcbjfx8fFMmjQJt1vuDi5EZ6OOikPfcBfmpRWYl5/GfPof9KVTUIOT7Y7W4bV4xGBZFvn5+cyZM4fc3FzWrFlDRUWF3zJFRUWEhYWxePFixowZw/LlywGoqKiguLiYBQsWkJ2dTX5+PpZlHXSdp556KgsXLuSee+5h3759FBUVtcFmCyGcQLnd6N9egr7+TujaHWvRLVgPzceSuYf/UYG/wgiKZFAAAA0PSURBVK3FYigvLyc2NpaYmBjcbjepqamUlpb6LbN27VrS0tIASElJYcOGDRhjKC0tJTU1lZCQEKKjo4mNjaW8vPyg6xw6dChKKZRSDBgwgOrq6oBvtBDCWVT/X6D/shD164sw761h+4xLsEpWy5lLbaTFMRqv10tk5P+ewBQZGcnGjRubXcblchEaGkpNTQ1er5eEhITG5TweD16vt3E9B1tnfX09b775JuPGjWsyV2FhIYWFhQDMmzePqKioljalSW63u9XvDQZOzu/k7ODs/I7NPmEG9elj2HXfndTlLyCk7G16XHkdrj6xdic7ZIHa97VhYdQAkZ5IdESPIw+2n6AdvH/44Yc57rjjOO6445r8fkZGBhkZGY2vt2/f3qqfExUV1er3BgMn53dydnB2fidnJ6wnkTn38c3Tj7GvYBnbp12MOucC1JnnoUJC7E7XokDte2uPbzK+2luN+n5fq9bRt2/fJr/e4lCSx+PxG86prq7G4/E0u0xDQwO1tbVEREQc8F6v14vH42lxnU8//TS7du3i8ssvP8TNE0J0Jsrl8p25dMsSGDQU89wTWHOnYza8Z3e0DqHFYoiPj6eyspKqqirq6+spLi4mOdn/rICkpCRWr14NQElJCYmJiSilSE5Opri4mLq6OqqqqqisrGTAgAEHXeeqVatYv349WVlZaC1n0wohmqcio3FdNRt99VwArL/eTMO9t2Oqq+wN5nAtDiW5XC4mTJhATk4OlmUxatQo4uLiWLFiBfHx8SQnJzN69Gjy8vKYPn064eHhZGVlARAXF8eIESOYOXMmWmsmTpzY+Jd9U+sEeOihh+jTpw/Z2dkADB8+nLFjx7bV9gshOgA1aCh67mLMv5/DvPQU1o1TUBmZqF+dj+oeane8ttGGE+/KdJBp/S1bWnfjLUePteLs/E7ODs7O7+TscPD8pvobzLOPYd59AyJ6on59Eeq0s1BBcj1UwOYYCl/ArHgYvXA5KiyiVeto9RyDEEI4iYrsg/7jLHT2fDgqDvPkA1g3TcOsK+6gp7facB2DEEI4kfp5AnpWDnraX8DlwrpvHtYd12E2rOugBRE4wXFsJYQQbUApBScOQw8aillT6Jt/+OtciP8l+tcXw/FDfMsIP1IMQogOT7lcqNPPwqSOxqxZhXn5KayFN/kK4tyLIPEkKYj9SDEIIToN5Q5BjfwVJjXddwTx8tO+I4if/dx3gdyw04JmktpOMscghOh0VEgIOu1sdM4DqHEzoKEB80gu1pxJWCsLOv0tvqUahRCdlgoJQZ2SgUlNhw3rsFY+i/nHUsw//45KGYk6/Veoo/vbHbMZNj+PQQghOjKlFAxOwjU4CfNlOWbVi5jiIszrr8CxA1Ejz0Yln4rq2tXuqO1ChpKEEGI/6pgB6AlZ6Lsf9T1idG8t5tG/Yl03DuuxxZhPPsBYlt0x/6cN5szliEEIIZqgwsJRGb/BpP8aNn6IeevfmNK3MG/9G3pFok4+HTX8dIjr3+HOaJJiEEKIg1BKwcBBqIGDMJd8j/ngXUzJasyqFzCvFkBkNOrEk1FDhkNCYoc4q8n5WyCEEO1Ede2KGnYaDDsNU7MLU1aCWf8u5s1XMUUvQmgYKnEo/PIE1C9PgD6xjjyakGIQQohWUBE9UKedCaedifn+O/ioDFP2DubD96H0Td85Q5HRqF8O9h1J/HwgHNXP5tSHRopBCCGOkOraDU5KQZ2U4rsP09b/Yj5Z75uofv8dWLPKVxRdu+NNOA6r3zHQ7+eoo34GsT9Ddetu9yb4kWIQQogAUkrBUT/z/aU/aozvDKaqLZjPN8Lnn2EqPses+ifU1//vSgRPH4jth/L0gd5R4IlCeaKgRy8IDff907Vbuw1LSTEIIUQbUlr7jgpifwYjRhEZFcU3W7fCN1uh8itMZQVs+RpTtQXz3y9h5w6gicvXXC7oHgruEHC54fu9bZZZikEIIdqZcrvhqJ/5jix+8j1TXwc7qmHHdqjZhandDbW7Yc9u2FsLDfVQXwcNDdArErqHBTyfFIMQQgQR5Q6BPrG+f2iT69daJFc+CyGE8CPFIIQQwo8UgxBCCD9SDEIIIfxIMQghhPAjxSCEEMKPFIMQQgg/UgxCCCH8KGNM2z04VAghhON0+iOGG264we4IR8TJ+Z2cHZyd38nZwdn5nZC90xeDEEIIf1IMQggh/Ljmzp071+4Qduvfv7/dEY6Ik/M7OTs4O7+Ts4Oz8wd7dpl8FkII4UeGkoQQQviRYhBCCOGnUz+op6ysjKVLl2JZFunp6WRmZtod6QBTp06lW7duaK1xuVzMmzeP3bt3k5ubyzfffEOfPn245pprCA8PxxjD0qVLef/99+natStTpkxp97HMe++9l3Xr1tGzZ0/mz58P0Kq8q1ev5tlnnwXg/PPPJy0tzZbsTz31FKtWraJHjx4AXHzxxQwdOhSAgoICioqK0Fozfvx4hgwZAtj3udq+fTtLlizh22+/RSlFRkYG55xzjiP2f3PZnbL/9+3bx0033UR9fT0NDQ2kpKRw4YUXUlVVxcKFC6mpqaF///5Mnz4dt9tNXV0deXl5bN68mYiICLKysoiOjj7odrUr00k1NDSYadOmma1bt5q6ujoza9Ys8/XXX9sd6wBTpkwxO3fu9PvasmXLTEFBgTHGmIKCArNs2TJjjDHvvfeeycnJMZZlmU8//dTMnj273fN++OGHZtOmTWbmzJmtzltTU2OmTp1qampq/P7bjuwrVqwwzz///AHLfv3112bWrFlm3759Ztu2bWbatGmmoaHB1s+V1+s1mzZtMsYYU1tba2bMmGG+/vprR+z/5rI7Zf9blmX27t1rjDGmrq7OzJ4923z66adm/vz55q233jLGGPPAAw+YlStXGmOMeeWVV8wDDzxgjDHmrbfeMgsWLDjodrW3TjuUVF5eTmxsLDExMbjdblJTUyktLbU71iEpLS1l5MiRAIwcObIx99q1azn99NNRSjFw4ED27NnDjh072jXb8ccfT3h4+BHlLSsr44QTTiA8PJzw8HBOOOEEysrKbMnenNLSUlJTUwkJCSE6OprY2FjKy8tt/Vz17t278Tf+7t27069fP7xeryP2f3PZmxNs+18pRbdu3QBoaGigoaEBpRQffvghKSkpAKSlpfnt+x+PwlJSUtiwYQPGmGa3q7112qEkr9dLZGRk4+vIyEg2btxoY6Lm5eTkAHDGGWeQkZHBzp076d27NwC9evVi586dgG+boqKiGt8XGRmJ1+ttXNYuh5v3p/9vPB7PQf+SaGsrV67kjTfeoH///lx++eWEh4fj9XpJSEhoMmMwfK6qqqr4/PPPGTBggOP2//7ZP/nkE8fsf8uyuP7669m6dStnnXUWMTExhIaG4nK5Dsi4/z52uVyEhoZSU1Nz0O1qT522GJzi1ltvxePxsHPnTm677Tb69u3r932lFErZ8bjw1nFa3jPPPJOxY8cCsGLFCh5//HGmTJlic6qD++6775g/fz7jxo0jNDTU73vBvv9/mt1J+19rzd13382ePXu455572LJli92RWq3TDiV5PB6qq6sbX1dXV+PxeGxM1LQfM/Xs2ZNhw4ZRXl5Oz549G4eIduzY0Tgx5/F42L59e+N7g2WbDjfvT//feL1e27ajV69eaK3RWpOens6mTZuAAz8/P2a0+3NVX1/P/PnzOe200xg+fDjgnP3fVHan7X+AsLAwEhMT+eyzz6itraWhocEv40/zNzQ0UFtbS0RERNB89jttMcTHx1NZWUlVVRX19fUUFxeTnJxsdyw/3333HXv37m387w8++ICjjz6a5ORkXn/9dQBef/11hg0bBkBycjJvvPEGxhg+++wzQkNDbR9G+jHX4eQdMmQI69evZ/fu3ezevZv169fbc2YG+M3RvPvuu8TFxTVmLy4upq6ujqqqKiorKxkwYICtnytjDPfffz/9+vXj3HPPbfy6E/Z/c9mdsv937drFnj17AN8ZSh988AH9+vUjMTGRkpISwHem149ZkpKSWL16NQAlJSUkJiailGp2u9pbp77yed26dTz22GNYlsWoUaM4//zz7Y7kZ9u2bdxzzz2A77eKU089lfPPP5+amhpyc3PZvn37Aacf5ufns379erp06cKUKVOIj49v18wLFy7ko48+oqamhp49e3LhhRcybNiww85bVFREQUEB4DtdctSoUbZk//DDD/niiy9QStGnTx8mTZrUWLbPPvssr732Glprxo0bx0knnQTY97n65JNPuPHGGzn66KMbh4suvvhiEhISgn7/N5d9zZo1jtj/X375JUuWLMGyLIwxjBgxgrFjx7Jt2zYWLlzI7t27OfbYY5k+fTohISHs27ePvLw8Pv/8c8LDw8nKyiImJuag29WeOnUxCCGEOFCnHUoSQgjRNCkGIYQQfqQYhBBC+JFiEEII4UeKQQghhB8pBiGEEH6kGIQQQvj5f/eGqvJ09q1oAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "metadata": { "id": "Yy8lkzpGwwdV" }, "source": [ "As expected, the learning rate is updated in accordance with cosine annealing schedule." ] }, { "cell_type": "markdown", "metadata": { "id": "t_4cZUXgwwdV" }, "source": [ "The training process was terminated after _16 epochs_. Now we're going to restore the best weights saved during training, and apply the model to the validation subset of the data to see the final model's performance:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "b9WGkwZ7wwdV", "outputId": "90510a76-d643-4682-a7b3-aaf17de15494" }, "source": [ "net.load_state_dict(best_weights)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": {}, "execution_count": 41 } ] }, { "cell_type": "code", "metadata": { "id": "lI_DqmEIwwda" }, "source": [ "# groud_truth, predictions = [], []\n", "\n", "# with torch.no_grad():\n", "# for batch in batch_generator(*datasets['val'], shuffle=False, bs=bs):\n", "# x_batch, y_batch = [b.to(device) for b in batch]\n", "# outputs = net(x_batch[:, 1], x_batch[:, 0], minmax)\n", "# groud_truth.extend(y_batch.tolist())\n", "# predictions.extend(outputs.tolist())\n", "\n", "# groud_truth = np.asarray(groud_truth).ravel()\n", "# predictions = np.asarray(predictions).ravel()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "YdXslUMBwwda" }, "source": [ "# final_loss = np.sqrt(np.mean((predictions - groud_truth)**2))\n", "# print(f'Final RMSE: {final_loss:.4f}')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "hB9t-ARGwwdb" }, "source": [ "with open('best.weights', 'wb') as file:\n", " pickle.dump(best_weights, file)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "8Qcj-GoNwwdb" }, "source": [ "### Embeddings Visualization\n", "\n", "Finally, we can create a couple of visualizations to show how various movies are encoded in embeddings space. Again, we're repeting the approach shown in the original post and apply the [Principal Components Analysis](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) to reduce the dimentionality of embeddings and show some of them with bar plots." ] }, { "cell_type": "markdown", "metadata": { "id": "OqwMsqWHwwdc" }, "source": [ "Loading previously saved weights:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JKlP4S1zwwdc", "outputId": "ed020588-98cd-4d38-9ba8-4fee5dcd31b1" }, "source": [ "with open('best.weights', 'rb') as file:\n", " best_weights = pickle.load(file)\n", "net.load_state_dict(best_weights)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": {}, "execution_count": 45 } ] }, { "cell_type": "code", "metadata": { "id": "omN9TxzFwwdd" }, "source": [ "def to_numpy(tensor):\n", " return tensor.cpu().numpy()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "_8x4DFcbwwdd" }, "source": [ "Creating the mappings between original users's and movies's IDs, and new contiguous values:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6TOmrBbbg9WA", "outputId": "6b34ad31-b1a9-4dbe-b12f-040dcb3a2c2b" }, "source": [ "maps.keys()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "dict_keys(['ITEMID_TO_IDX', 'IDX_TO_ITEMID'])" ] }, "metadata": {}, "execution_count": 49 } ] }, { "cell_type": "code", "metadata": { "id": "qdHe7Nekwwdd", "scrolled": true }, "source": [ "user_id_map = umap['USERID_TO_IDX']\n", "movie_id_map = imap['ITEMID_TO_IDX']\n", "embed_to_original = imap['IDX_TO_ITEMID']\n", "\n", "popular_movies = ratings_df.groupby('ITEMID').ITEMID.count().sort_values(ascending=False).values[:1000]" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "L7cJFbDXwwde" }, "source": [ "Reducing the dimensionality of movie embeddings vectors:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "A0My7Epuwwde", "outputId": "44f167aa-656f-4fbc-ee20-898672fd5ee4" }, "source": [ "embed = to_numpy(net.m.weight.data)\n", "pca = PCA(n_components=5)\n", "components = pca.fit(embed[popular_movies].T).components_\n", "components.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(5, 1000)" ] }, "metadata": {}, "execution_count": 52 } ] }, { "cell_type": "markdown", "metadata": { "id": "MeHZJsE3wwdf" }, "source": [ "Finally, creating a joined data frame with projected embeddings and movies they represent:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "g4_xKa2p7bIO", "outputId": "1f7539cf-8346-4f83-9a33-9d9abcd739e2" }, "source": [ "movies = movies_df[['ITEMID','TITLE']].dropna()\n", "movies.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1682, 2)" ] }, "metadata": {}, "execution_count": 53 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "YkGwvIUqwwdf", "outputId": "c99192b1-7bd9-41da-efff-dd2e5a379e5a" }, "source": [ "components_df = pd.DataFrame(components.T, columns=[f'fc{i}' for i in range(pca.n_components_)])\n", "components_df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fc0fc1fc2fc3fc4
0-0.013240-0.0391030.061148-0.050387-0.020236
1-0.020302-0.040640-0.0052810.0136270.022263
2-0.046928-0.006252-0.027646-0.0124380.033915
30.068046-0.001613-0.0422350.006097-0.029498
4-0.056585-0.006064-0.015407-0.017673-0.018524
\n", "
" ], "text/plain": [ " fc0 fc1 fc2 fc3 fc4\n", "0 -0.013240 -0.039103 0.061148 -0.050387 -0.020236\n", "1 -0.020302 -0.040640 -0.005281 0.013627 0.022263\n", "2 -0.046928 -0.006252 -0.027646 -0.012438 0.033915\n", "3 0.068046 -0.001613 -0.042235 0.006097 -0.029498\n", "4 -0.056585 -0.006064 -0.015407 -0.017673 -0.018524" ] }, "metadata": {}, "execution_count": 54 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "chcYgtkw8ij6", "outputId": "4e996939-45d5-4031-ca55-5cef20b87a6a" }, "source": [ "components_df.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(1000, 5)" ] }, "metadata": {}, "execution_count": 55 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 173 }, "id": "lTu7ZbLL8NaO", "outputId": "820a8b21-2a52-4c36-983b-cc0c9123efff" }, "source": [ "movie_ids = [embed_to_original[idx] for idx in components_df.index]\n", "meta = movies.set_index('ITEMID')\n", "components_df['ITEMID'] = movie_ids\n", "components_df['TITLE'] = meta.reindex(movie_ids).TITLE.values\n", "components_df.sample(4)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
fc0fc1fc2fc3fc4ITEMIDTITLE
667-0.0117020.0078130.044108-0.036211-0.002407667Audrey Rose (1977)
703-0.0073030.0268040.0735950.0660340.041713703Widows' Peak (1994)
879-0.0255450.039012-0.001066-0.0167790.006389879Peacemaker, The (1997)
197-0.0386260.002753-0.045264-0.0338070.004753197Graduate, The (1967)
\n", "
" ], "text/plain": [ " fc0 fc1 fc2 ... fc4 ITEMID TITLE\n", "667 -0.011702 0.007813 0.044108 ... -0.002407 667 Audrey Rose (1977)\n", "703 -0.007303 0.026804 0.073595 ... 0.041713 703 Widows' Peak (1994)\n", "879 -0.025545 0.039012 -0.001066 ... 0.006389 879 Peacemaker, The (1997)\n", "197 -0.038626 0.002753 -0.045264 ... 0.004753 197 Graduate, The (1967)\n", "\n", "[4 rows x 7 columns]" ] }, "metadata": {}, "execution_count": 60 } ] }, { "cell_type": "code", "metadata": { "id": "rW-l-0Izwwdg" }, "source": [ "def plot_components(components, component, ascending=False):\n", " fig, ax = plt.subplots(figsize=(18, 12))\n", " \n", " subset = components.sort_values(by=component, ascending=ascending).iloc[:12]\n", " columns = components_df.columns\n", " features = columns[columns.str.startswith('fc')].tolist()\n", " \n", " fc = subset[features]\n", " labels = ['\\n'.join(wrap(t, width=10)) for t in subset.TITLE]\n", " \n", " fc.plot(ax=ax, kind='bar')\n", " y_ticks = [f'{t:2.2f}' for t in ax.get_yticks()]\n", " ax.set_xticklabels(labels, rotation=0, fontsize=14)\n", " ax.set_yticklabels(y_ticks, fontsize=14)\n", " ax.legend(loc='best', fontsize=14)\n", " \n", " plot_title = f\"Movies with {['highest', 'lowest'][ascending]} '{component}' component values\" \n", " ax.set_title(plot_title, fontsize=20)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 704 }, "id": "WpJkrTX9-Asp", "outputId": "6724cabe-23d6-4ffe-8a21-1c3ba6cb3066" }, "source": [ "plot_components(components_df, 'fc0', ascending=False)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABCUAAAMFCAYAAABDA0wqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde1yUZf7/8fcwI6ggMEqjKQieT5snQAwtPKBUnovS1S0NO9A3Navtt9JhZctSt93Nylr9WnksTVrLY2lSVmqaluX5UEqGmngAz2LA9fujL7NODIjD6Ki9no9Hj+K6r/u+P3PNfU/Mm/u+bosxxggAAAAAAOAy8/N1AQAAAAAA4PeJUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAwDVqyJAhslgsysrK8nUpFebJa1mxYoUsFovS09MrvP+srCxZLBYNGTKk3OtMmzZNFotF06ZNu+z7vtrMnj1bbdq0UbVq1WSxWDRy5EhflwR4ncViUadOnXxdBgBccQglAMADFotFFotFfn5++uGHH0rt17lzZ2ffin45havfw5f1K5GnYVdxSLNixQqX9i+//FKDBg3SiRMn9NBDD2n06NG65ZZbPKrtzJkzGj16tJo0aaLKlSvL4XDorrvu0rZt29z25/i59qSnp7s9zgAAVy5CCQDwkM1mkzFGb775ptvlu3bt0ooVK2Sz2S5zZb8aO3astm3bpjp16vhk/950Nb6Wfv36adu2berXr5+vS7miLV68WMYYzZgxQ3//+9+Vnp7uUSiRn5+vbt266dlnn1VwcLAeeeQRJSYm6v3331dMTIzWrl17CaoHAAAVRSgBAB6qWbOmYmJiNHXqVBUUFJRY/sYbb0iSevXqdblLkyRdf/31atq0qSpVquST/XvT1fhaQkJC1LRpU4WEhPi6lCva/v37JUm1a9eu0Hb+9a9/adWqVUpOTtbatWs1fvx4vfPOO3rvvfd0+vRppaSkqKioyBslAwAALyKUAIAKuP/++/Xzzz9r0aJFLu2//PKLpk2bpvj4eDVv3rzU9Xft2qV77rlHderUkb+/v2rXrq177rlHu3btcumXmpoqi8Wi+fPnu93O2rVrZbFYlJyc7Gwr6zL7tWvXKjk5WbVq1ZK/v78iIiL04IMPOr8gnm/37t164IEH1LBhQ1WpUkXVq1fXDTfcoNTUVB05cqSs4ZH065dNd1c4REZGymKx6LnnnnNp//DDD2WxWPTXv/611NeSnp6uevXqSZKmT5/uvEWmtNtkvv32W/Xo0UOhoaGqWrWqEhIStHr16gvW7k5WVpYGDBigsLAwVa5cWTExMSXef6nsOSWWLl2qDh06KDAwUNWrV1ffvn21ffv2C94aUd59F5s9e7Y6d+6s0NBQVa5cWc2aNdOYMWOUn59fou8XX3yhXr16KTw8XAEBAapVq5bat2+vv/3tb84+FotF06dPlyTVq1fPOeZRUVFlD5obxeMzderUEts7//VnZ2drxIgRatSokfP4a9eunctxY4zRpEmTJEl///vf5ef3319v+vTpo5tuuklbt27VZ599dtF1/tayZcvUq1cvORwOBQQEKCIiQn369NHy5ctd+hUVFWnSpEmKjY1VUFCQAgMDFRsbq3//+99uw5Hi+QYOHjyolJQU1axZU4GBgYqPj9cXX3whSTp16pSeeOIJRUZGKiAgQC1atFBGRkaJbZ1/7C1evFjx8fEKDAyU3W5XcnJyic+XYgcOHNDDDz+sqKgo+fv767rrrtPtt9+ur7/+usx9fPrpp+rUqZOqVaum4OBg9ejRo9RbZk6fPq2xY8eqdevWCgwMVFBQkG688UbNnj27RN/z54UpzzkcFRXlPF7Pv3XOYrG4raXYnDlzZLFY9Oijj7pdnp+fL7vdruuvv94ZQB87dkwvvviiunTpovDwcOd49e7dW19++WWZ+ztfWed8WfPiHD16VGlpaWrWrJmqVKmikJAQde3aVcuWLSvR99y5c3rllVfUtm1b2e12Va1aVVFRUW6PWwDwBUIJAKiAP/7xjwoMDHReFVFswYIFysnJ0f3331/quuvWrVNMTIxmzZql2NhY/fnPf1b79u01a9YsxcTEaN26dc6+gwcPliTNmDHD7baKvyiW5/74t956Sx06dNCHH36ozp07a+TIkYqJidEbb7yhmJgY7d2719n3wIEDio2N1dSpU9WiRQuNGDFCd999t+rVq6eZM2fqwIEDF9xfly5dtH//fm3fvt3Z9v333zv3k5mZ6dK/+OeuXbuWus1OnTrpkUcekSS1atVKo0ePdv7TunVrl77r169XfHy8zp49q/vuu089e/bUypUr1bVrV+3YseOC9Z/vxx9/VLt27ZSVlaW7775b/fv31+bNm9WnTx99+umn5drGnDlzdOutt2rDhg2688479eCDDyo3N1c33nhjmfM0XOy+U1JSNHDgQH3//fe644479PDDD6t69ep65plndMstt7hc3fPRRx+pU6dOznF5/PHH1bdvXwUEBOj111939hs9erRatWolSXrkkUecY+7JxJStW7cudXuhoaGSfn3vWrVqpVdffVW1a9fWiBEjNGjQIFWrVs3li9oPP/ygvXv3qnHjxs6w6ny33nqrJOmTTz656DrPN3r0aCUlJWnFihVKSkrS448/rq5du2rbtm2aNWuWS9+7775bDz30kA4ePKj77rtPDzzwgA4dOqT/+Z//0d133+12+3l5eerQoYM2bNigP/7xj7rjjju0fv16JSUl6bvvvlPXrl01f/589ezZU4MHD9bevXvVv39/rVmzxu325s2bp759+yo8PFyPPPKIbrzxRv3nP/9R+/btSxz7e/bsUUxMjF5//XU1aNBAjz/+uJKSkpyhRmnh16JFi9S9e3cFBwcrNTVVN910k5YsWaKEhAQdPny4xOvr2LGjnnzySVmtVqWkpGjw4ME6dOiQBg4cqKefftrtPsp7Do8cOVIJCQmSfv3MPP9zoSx9+/ZVSEiI3nnnHbdXvc2fP195eXkaNGiQ83a8bdu26amnnpKfn5969Oihxx57TN26ddMnn3yim2++WR999FGZ+6yIH3/8UdHR0Ro3bpyuu+46paamqn///tq2bZtuueUWTZkyxaX/kCFD9Mgjj+iXX37RPffcoxEjRujmm2/Wpk2bLmmdAFBuBgBw0SSZOnXqGGOMGTp0qLFareann35yLk9KSjLBwcHm1KlT5qmnnjKSzNSpU53Li4qKTNOmTY0kM2vWLJdtz5kzx0gyTZo0MYWFhc72xo0bG39/f3PkyBGX/mfPnjV2u904HA7zyy+/ONsHDx5sJJk9e/Y423bs2GEqVapkGjRoYLKzs122s3z5cuPn52f69u3rbHvllVeMJDNhwoQSY3Dy5Elz+vTpC47Vm2++aSSZiRMnOtsmTZpkJJlu3boZf39/c+rUKeey1q1bmypVqpj8/PwyX8uePXuMJDN48GC3+/3000+NpBJjf/7+H3rooQvWf/6+JJn09HSXZR999JGRZG699VaX9qlTp5bY9/Hjx01oaKjx9/c33377rUv/v/zlL859uHudnuy7X79+Jd6j0aNHl3hPb7/9diOpRE3GGHPo0CGXn929FxVR2vby8/NNVFSUkWTefvvtEuudf74tWrTISDI9e/Z0u4+MjAwjydx1110e17l06VIjydSrV6/EufPbet555x0jybRp08acOHHC2X7y5EkTHR3t9jUVv8cPPvigy3k/Y8YMI8nY7XbTs2dPc+bMGeeyzz//3EhyOWeN+e/7L8ksXLjQZdmECROMJNOlSxeX9u7duxtJZsyYMS7tq1atMlar1VSvXt3ltRTvw2q1muXLl7usM2rUKCPJjB8/3qW9+L3+bfuZM2dMUlKSsVgsZsOGDc52T87h4uP7008/NRfjgQcecDtexhhz2223GUlm48aNzra8vLwS54Yxvx4H119/vWnatGmJZZJMQkKCS1tZ51Px6x89erRLe0JCgrFYLGb27Nku7bm5uaZVq1amcuXK5ueff3bWabFYTHR0tCkoKCixj8OHD5doA4DLjVACADxwfiixZs0aI8n87W9/M8YYk5WVZfz8/Jy/LLsLJVauXGkkmRtvvNHt9jt27Ggkmc8++8zZ9vzzz5f4cm/Mf79wPfrooy7t7n7ZHTlypJFkFi1a5Ha/ffv2NVar1Rw/ftwY899QYvLkyeUYFfeysrKcX5CL3XnnnaZmzZpm4cKFRpJZunSpMebXX5AtFovp1q3bBV9LeUOJDh06lFh27tw5Y7PZTHR0dLleQ/G+IiMj3f5iX7duXVOjRg2XNnehxMyZM40kc++995bYxokTJ0xoaGipr/Ni9t26dWtjs9lMbm5uif4FBQWmRo0aJjY21tlWHErs2LGj1DEodrlCiffee89IMr17977gNt5++20jyQwaNMjt8mXLlhlJpnv37h7X2bNnTyPJzJs374J9ExMTXY7r8y1fvtxIMp07d3Zpl2SqVq3qPPeKFRQUGJvNZiSZH374ocT2oqKiTFRUlEtb8bH32+CheHsNGjQwkkxWVpYx5tcv0pJM3bp1zblz50qs86c//clIMtOnTy+xD3djvnv3biPJ3HHHHc62w4cPG6vVamJiYkr0N8aYb7/91kgyTzzxhLPNk3PY01Bi1apVRpJJTk52aT9w4ICxWq2mTZs25d7W8OHDjSTz448/urR7I5QoHqff1lnsgw8+MJLMa6+9Zowx5tixY0aSiY+PN0VFReV+DQBwOflmSngAuIbExcXphhtu0FtvvaWnn35ab7zxhoqKisq8deObb76R9OutDe506dJFK1eu1IYNG3TzzTdLku655x4988wzmj59uh5++GFn34u5daP4XufPPvvM5faQYjk5OSosLNTOnTsVHR2t3r1768knn9TDDz+spUuXKikpSR06dFDz5s0veJ92scjISNWvX18rVqxQUVGR83F9iYmJSkhIkM1mU2Zmprp3765PP/1UxphSx8UTMTExJdoqVaqkmjVrKjc396K21bp1a1mt1hLtERER5bqPfMOGDZKkjh07llgWFBSk1q1bl/oow/Lu+/Tp0/ruu+8UFhamCRMmuN1WQECAyz3/gwYN0rx58xQXF6f+/furc+fO6tChg8LDwy/4mi6V4lsSim+98LU1a9bIYrGU68kg33zzjfz8/NSpU6cSyxISEmS1Wp3HwvkaN26satWqubRZrVbVrFlTp06dUv369UusU6dOnVKfLFJ8K8Nvt9exY0f98MMP2rBhgyIjI5213HTTTW4nk+3SpYtmzZqlDRs26J577nFZ5u78ioiIkCSX82vdunUqLCwsdY6EX375RZLczkXhzXO4NPHx8WrcuLEWLlyo3Nxc2e12SdLbb7+twsJCt5+vq1at0ssvv6wvv/xSOTk5OnfunMvyffv2qW7dul6pr1jxuX7s2DG343jo0CFJ/x3H4OBg9erVSwsXLlTr1q11xx136KabblJcXJyqVq3q1doAwFOEEgDgBffff79GjBihDz/8UFOnTlV0dLTatGlTav9jx45J+vWpEu4Ut+fl5TnbwsPD1bVrV3388cfatm2bmjVrppycHH300Udq3bq1WrZsecE6iyemfPHFF8vsd/LkSUm/BgpfffWV0tPT9dFHH2nevHmSfv3S8ec//1kjRoy44D6lX+eHmDJlir755htVqlRJhw4dUteuXVWtWjXFxsY655Eoz3wSF6t4foLfstlsKiws9Nq2yvNkh+L3vWbNmm6Xl9Z+MfvOzc2VMUaHDh1ymaSyLLfffrsWLVqkf/7zn3rrrbc0efJkSVJ0dLTGjh2rbt26lWs73lR87JfnMbDFTzgpHt/fKm4vbQzLW4/dbleVKlUu2PfYsWOqXr26/P39Syyz2WwKCwtTTk5OiWWlPanFZrOVuczdPAhS6cdTrVq1nHWe/++L+Twq5m5Mi+ddOP/8Kv7sWbdundtAtFjxZ8+F9lG8n4s9h8syePBgPfXUU5ozZ44eeughSb+GvpUqVdLAgQNd+r7//vtKTk5W5cqV1a1bNzVo0ECBgYHy8/PTihUr9Nlnn7mdULaiisfx448/1scff1xqv/PH8d1333U+jaZ4fo3KlSsrOTlZ//jHP8r83AGAy4GJLgHAC+6++25VqVJFqamp2rdvnx544IEy+xd/wfj555/dLi+eQPK3X0SKJ7wsvjri7bffVkFBgbP9Qs7/8mZ+vYXP7T/n/4W1WbNmevfdd3XkyBGtX79e48aNU1FRkR555BG9+eab5dpv8ZUPy5cvLxE8dOnSRRs2bNDRo0eVmZmpkJAQtW3btlzbvdoEBwdLkg4ePOh2eWntF6P4PW7Tpk2Z77ExxmW9Hj166JNPPlFubq4yMzP16KOPasuWLerZs6e2bt1a4bouVvEX0X379l2wb5MmTSRJO3fudLu8+GkTjRs3rlA9ubm5OnPmzAX7hoSE6OjRo86//p+voKBAhw8fdh4Ll1Jpx1Px507xseLp59HFKF730UcfLfOYLO+EsZfC3XffLT8/P+fn64YNG7Rp0ybddtttCgsLc+n7zDPPyN/fX+vXr9cHH3ygf/7zn3r22WeVnp7uPB7Lo/hJMe6CJXchUPE4vvzyy2WOY/FTbSSpSpUqSk9P186dO7V3717NmjVLHTt21KxZs1ye2AQAvkIoAQBeEBoaquTkZGVnZyswMFB//OMfy+xffBVFaZfqF/9i/tsv57fffruCg4M1a9YsFRUVafr06bLZbCX+ilea9u3bS5LzEYMXw2azKTo6Wn/5y1+cj+/74IMPyrVuly5dZLFYlJmZqU8++UT169d3Pkaya9euKioq0owZM7Rr1y516tTJ7W0Kv1Xcx5t/Kb3Uit/3lStXllh28uRJffvttxXeR1BQkFq0aKEtW7bo6NGjF71+YGCgunTpon/961968sknde7cOX344YfO5Zdr3IuP1fP3XZoGDRqobt262rlzp/bs2VNiefE2KnJbUPv27WWMKdfTCtq0aaOioiJ9/vnnJZZ9/vnnKiwsvCzBm7tHoBYWFjqPv+Lj8fzj0t2X49I+jy5Gu3bt5Ofn59Fnz8WoyPEZERGhLl26aO3atdqxY4cznHAX+n7//fdq3ry5mjVr5tJeVFTk9vwuTfFtIj/99FOJZevXry/RVpHPcOnX1zho0CAtXbpUDRs21MqVK8v1aGcAuJQIJQDAS8aMGaP3339fS5cuLXFf+G916NBBTZo00cqVK/Xee++5LHvvvff0xRdfqHHjxiXmHqhSpYruuusu7du3Ty+99JK+++473XbbbXI4HOWqcdiwYapUqZIeffRRt39VPnfunMsvu19//bXbS+KL/wJb3nuSHQ6HWrRooVWrVunzzz93uT0jPj5elStX1tixYyWV/4uj3W6XxWJxeYTpla5Pnz4KCQnR22+/re+++85l2ZgxY9z+ZdQTjz32mM6dO6eUlBS328zNzXXOayL9+kXZ3ZdRd+9zjRo1JOmSj3uvXr0UFRWlBQsWOEOw82VnZzv/22KxKDU1VZL0//7f/3O5nWX+/Pn64osv1Lx5c7dzLJTX8OHDJUmPP/6426s3zm9LSUmRJKWlpen06dPO9tOnT2vUqFGSpKFDh3pcS3l98sknJR7lOXHiRP3www/q3LmzIiMjJf16a1i3bt2UlZVVYh6StWvX6p133pHdble/fv08rsXhcGjQoEFav369nnvuObehwQ8//OA2VLoYFT0+i+eOePPNNzV79myFhYWpZ8+eJfpFRUVp165d2r9/v7PNGKP09PSLurKoXbt2klTiMZ6bNm3Syy+/XKJ/TEyMbrrpJs2bN09vvfWW221u2rTJeXvQoUOHtGnTphJ9Tp06pZMnT8pms7m9zQgALifmlAAAL6lbt265JzWzWCyaPn26unXrpv79+6tPnz5q2rSpduzYoQ8++EDVqlXTjBkznJf2nm/w4MF64403lJaW5vy5vJo2baq33npLKSkpatGihW655RY1btxYv/zyi/bu3asvvvhC1113nbZv3y5JmjlzpiZPnqyOHTuqQYMGstvt+uGHH7Rw4UIFBARo5MiR5d53165dtXnzZud/FwsICFCHDh0uej6JoKAgxcXF6YsvvtCgQYPUuHFjWa1W9e7du1zza/hCcHCwXnvtNd19992Kj4/XXXfdpeuvv16rV6/Wd999p4SEBH322Wdu3/eLkZKSoq+//lqvv/66GjRooKSkJNWtW1dHjx7Vnj179Pnnn+vee+/VpEmTJEkjRozQvn371KFDB0VFRcnf319ff/21PvnkE0VGRmrAgAHObXft2lUvvvii7r//ft1xxx2qVq2aQkNDNWzYsArV/Fv+/v7KyMhQ9+7dNXDgQE2ePFnt27fX2bNntW3bNmVmZroEKY899pgWLVqk9957T3Fxceratav27t2rjIwMVa1aVW+99VaFxrV79+56+umnNWbMGDVr1kx9+/ZVRESEDh48qJUrV6p9+/aaNm2aJGngwIGaP3++5s6dqxYtWqhv376yWCz64IMPtGfPHvXv31+DBg2q6BBdUK9evdSvXz/169dPDRs21LfffqsPP/xQ1atX1+uvv+7Sd9KkSerQoYOeeOIJLVu2TDExMfrpp5+UkZEhPz8/TZ069YJh64VMnDhRu3bt0l//+lfNnDlTHTt2VM2aNbV//35t27ZN69at0+zZs1WvXj2P99G5c2f5+fkpLS1Nmzdvdl6J8PTTT5dr/X79+ik4OFgTJkzQL7/8ouHDh7ud/PPRRx9Vamqq2rRpozvuuEOVKlXSqlWrtHXrVufEkuXRp08fNWrUSLNnz1Z2drbi4uK0d+9ezZ8/X3369NHcuXNLrPPOO++oS5cuGjp0qF555RXFxcUpNDRU2dnZ2rhxozZv3qwvv/xSDodD+/btU5s2bXTDDTeoZcuWioiI0PHjx7Vo0SL9/PPPGjFiRIXfVwCosEv+fA8AuAbpvEeCXoi7R4IW2759u/nTn/5katWqZWw2m6lVq5YZNGiQ2b59e5nbbNiwoZFkqlevbvLz8932KetRcxs3bjSDBw82devWNf7+/sZut5sWLVqYBx54wGRmZjr7rVmzxqSmppqWLVsau91uKleubBo0aGCGDBliNm3aVK7XX2zBggVGkrFYLObgwYMuy1544QUjydSsWfOiXsuuXbtMz549TfXq1Y3FYnEZZ3eP0ztfZGSkiYyMLFftF3r8aEJCgvnt/1LdPRK02JIlS8yNN95oqlSpYkJDQ03v3r3Ntm3bTI8ePYwkl0d5erLvYgsXLjQ9evQw1113nalUqZKpWbOmiY2NNU899ZTZtm2bs9+7775rBgwYYBo2bGgCAwNNtWrVTIsWLcyTTz5pcnJySmz3n//8p2natKnx9/d3Pq7UUxd6xOiPP/5oHnroIRMVFWUqVapkqlevbtq1a2eef/75En1PnTplnnnmGdOwYUPj7+9vwsLCTHJystmyZYvH9f3W4sWLTVJSkrHb7cbf39+Eh4ebvn37upw3xhhTWFhoXnvtNRMdHW2qVKliqlSpYtq2bWsmTpxoCgsLS2xXbh4XWaysY/VCx97ChQtN+/btTdWqVU1ISIi5/fbbS330a3Z2tklNTTV169Y1lSpVMjVq1DB9+vQxX331VYm+ZR3fZb2e/Px88+qrr5obb7zRBAcHG39/fxMREWG6dOliXnrpJXP48GFnX0/P4ZkzZ5pWrVqZypUrG0mlnh+lGTp0qHO99evXl9pv6tSpplWrVqZq1aqmRo0apm/fvmbjxo2lPpa0tDHZu3evueuuu5yfsTExMeY///lPma//+PHj5vnnnzdt27Y1gYGBpnLlyiYqKsrcdtttZvLkyebkyZPGGGNyc3PN3/72N9O5c2dTu3Zt4+/vb2rVqmUSEhLMO++8w2NCAVwRLMb8ZqYrAABw2RUWFqp+/fo6d+6cc2JBwBPTpk3Tvffeq6lTp5brUcEAAPgSc0oAAHAZ5eXlucwzIP16L/qYMWO0d+/eCt23DwAAcLVhTgkAAC6jNWvWqH///urevbuioqJ08uRJrVmzRt9++60iIiKUnp7u6xIBAAAuG0IJAAAuoyZNmqhnz55atWqVlixZooKCAoWHh2vEiBF68skny/0kFQAAgGsBc0oAAAAAAACfYE4JAAAAAADgE4QSAAAAAADAJ66pOSX279/v6xIuKCwsTIcPH/Z1GdcMxtN7GEvvYjy9i/H0HsbSuxhP72I8vYex9C7G07sYT++6Gsazdu3apS7jSgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+cU09fQMAAAAAgEvll19+0blz5yRJFovFx9X86uDBg8rPz/fZ/o0x8vPzU+XKlT0aE0IJAAAAAAAu4OzZs5KkqlWrXjGBhCTZbDZZrVaf1lBQUKCzZ8+qSpUqF70ut28AAAAAAHABhYWFHl8NcK2z2WwqKiryaF1CCQAAAAAALoAwomyejg+hBAAAAAAA8IkKzSmxdOlSLViwQHl5eQoPD9eQIUPUrFmzUvtv3bpV06dPV3Z2tux2u3r37q3u3bs7lxcVFWnu3Ln64osvlJeXp9DQUN1000268847fX6PDAAAAAAA8C6PQ4nVq1dr2rRpGjp0qJo2baply5bphRde0EsvvaSwsLAS/XNycjR27Fh17txZw4cP1/bt2/Xmm28qODhY7du3lyR98MEHWrp0qR5++GHVrVtXe/fu1WuvvSabzabk5GTPXyUAAAAAAJdA4f29L+v+rFMWXPQ6RUVFGjVqlBYvXqy8vDxlZGQoPj7+ElR38Ty+fWPRokVKSEhQYmKiwsPDlZKSIrvdrmXLlrntv2zZMtntdqWkpCg8PFyJiYlKSEjQwoULnX127typ6OhoxcTEyOFwKCYmRtHR0fr+++89LRMAAAAAgN+1zMxMzZ07V9OmTdOGDRsUExNTZv+8vDwNHz5cTZs2VdOmTTV8+HAdO3bsktTmUShRUFCg3bt3q1WrVi7tLVu21I4dO9yus2vXLrVs2dKlrVWrVtq9e7cKCgokSU2bNtWWLWUpeXoAACAASURBVFu0b98+SVJ2dra2bNmiNm3aeFImAAAAAAC/e1lZWXI4HIqNjZXD4ZC/v3+Z/YcNG6bNmzdr1qxZmjVrljZv3qwRI0Zckto8un3j+PHjKioqUkhIiEt7aGioNm3a5HadvLw83XDDDS5tISEhKiws1IkTJ2S329WnTx+dOXNGjz32mPz8/FRYWKjbb79dSUlJbre5fPlyLV++XJI0btw4t7eNXGlsNttVUefVgvH0HsbSuxhP72I8vYex9C7G07sYT+9hLL2L8fSuq3U8Dx48KJut5Ffowstch7sa3LUVGzFihN59911JUp06dRQREaF169Zp0qRJmj59uvbt26caNWooOTlZTz/9tHbu3KlPP/1UCxcuVLt27SRJ//jHP9S7d29lZWWpYcOGbvcTEBDg0ftaoYkuvW316tX6/PPPNWLECEVERCgrK0tTp06Vw+FQly5dSvRPTExUYmKi8+fDhw9fznI9EhYWdlXUebVgPL2HsfQuxtO7GE/vYSy9i/H0LsbTexhL72I8vetqHc/8/Pwr4gEMxXcaFLPZbCXazpeenq7atWtrzpw5WrJkiaxWq8aMGaMZM2Zo9OjRiouL05EjR7R582YVFBToq6++UmBgoNq0aePcbtu2bVW1alWtXbtWUVFRbveTn59f6vtau3btUuvzKJQIDg6Wn59fiXtKip+Y4U5oaKjy8vJc2o4dOyar1apq1apJkmbNmqVevXqpQ4cOkqS6devq0KFDev/9992GEgAAAAAAoHTBwcEKCgqS1WqVw+HQqVOnNGXKFKWnp2vAgAGSpHr16jnnmcjJyVGNGjVksVic27BYLAoLC1NOTo7X6/NoTgmbzab69etr48aNLu2bNm1SkyZN3K7TqFGjErd2bNy4UfXr13deapKfny8/P9eS/Pz8ZIzxpEwAAAAAAHCenTt3Kj8/Xx07dvR1KZIq8PSNnj17asWKFcrMzFR2dramTp2qo0ePqlu3bpKkiRMnauLEic7+3bt319GjRzVt2jRlZ2crMzNTK1asUK9evZx9oqOj9cEHH+ibb75RTk6OvvrqKy1atMh5HwsAAAAAALh0HA6Hjhw54nJxgDFGhw8flsPh8Pr+PJ5TIj4+XidOnNC8efOUm5uriIgIpaWl6brrrpNUcn4Hh8OhtLQ0TZ8+3fl40HvvvVft27d39klJSdG7776rN954Q8eOHZPdblfXrl2VnJzsaZkAAAAAAOD/NGrUSAEBAVq5cqXq169fYnl0dLROnTql9evXKzY2VpK0fv16nT59WtHR0V6vp0ITXSYlJZX6ZIz09PQSbc2bN9f48eNL3V6VKlU0ZMgQDRkypCJlAQAAAAAAN4KCgjR06FCNGzdOAQEBiouLU25urjZu3KjBgwerUaNG6ty5s0aNGuX8/j5q1CglJiaW+uSNiriinr4BAAAAAMDVxDplga9LuGhpaWkKCQnRhAkTdODAAYWFhbncoTBx4kQ988wzGjRokKRfp2MYM2bMJanFYq6hWST379/v6xIu6Gp9/M2VivH0HsbSuxhP72I8vYex9C7G07sYT+9hLL2L8fSuq3U8T58+rapVq/q6jBIu9EjQy6Ws8SnrkaAeT3QJAAAAAABQEYQSAAAAAADAJwglAAAAAACATxBKAAAAAAAAn+DpG2UovL93mcuvxllWfYnx9J4LjaXEeAIAAAC48nGlBAAAAAAA8AmulADwu8dVPN7FeHoPV0V5F8emdzGe3sO57l2MJ3B14UoJAAAAAADgE4QSAAAAAADAJ7h9AwAAAAAAD/V5e/tl3d/8QU0vep2ioiKNGjVKixcvVl5enjIyMhQfH38Jqrt4hBIAAAAAAFzDMjMzNXfuXGVkZCgyMlKhoaFl9n/55Zf1ySefaMuWLTpz5oz27dt3yWrj9g0AAAAAAK5hWVlZcjgcio2NlcPhkL+/f5n9z507p1tvvVX33XffJa+NKyUAAAAAALhGjRw5UhkZGZKkOnXqKDw8XGvWrNHkyZM1c+ZM7d+/X9WrV1dycrLS0tIkSU888YQkadGiRZe8PkIJAAAAAACuUc8++6zCw8M1Z84cLVmyRFarVePGjdOMGTM0evRoxcXF6ciRI9q8ebNP6iOUAAAAAADgGhUcHKygoCBZrVY5HA6dOnVKU6ZMUXp6ugYMGCBJqlevnmJiYnxSH3NKAAAAAADwO7Fz507l5+erY8eOvi5FEqEEAAAAAADwEUIJAAAAAAB+Jxo1aqSAgACtXLnS16VIYk4JAAAAAAB+N4KCgjR06FCNGzdOAQEBiouLU25urjZu3KjBgwdLkvbt26fc3FxlZ2dLknMSzHr16ikwMNCr9RBKAAAAAADcKry/d5nLrVMWXKZKrlzzBzX1dQkXLS0tTSEhIZowYYIOHDigsLAwJScnO5e/+OKLzseISlJSUpIkKSMjQ/Hx8V6thVACAAAAAIBrWGpqqlJTU50/+/n5adiwYRo2bJjb/hMmTNCECRMuS23MKQEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+QSgBAAAAAAB8wubrAgAAAAAAuFotfDfvsu6vV//Qi16nqKhIo0aN0uLFi5WXl6eMjAzFx8dfguouHqEEAAAAAADXsMzMTM2dO1cZGRmKjIxUaGjpwcZPP/2kCRMmaPXq1crJyZHD4VDv3r01cuRIValSxeu1EUoAAAAAAHANy8rKksPhUGxs7AX7fv/99yosLNTYsWNVr1497dq1S3/5y1+Um5urv//9716vjTklAAAAAAC4Ro0cOVLp6enat2+f6tSpo7i4OBljNGnSJHXo0EH16tVTdHS0xo4dK0nq3LmzJkyYoE6dOikyMlKJiYkaPny4Fi9efEnq40oJAAAAAACuUc8++6zCw8M1Z84cLVmyRFarVePGjdOMGTM0evRoxcXF6ciRI9q8eXOp2zh58mSZt3xUBKEEAAAAAADXqODgYAUFBclqtcrhcOjUqVOaMmWK0tPTNWDAAElSvXr1FBMT43b97OxsTZo0ScOHD78k9XH7BgAAAAAAvxM7d+5Ufn6+OnbseMG+hw4d0qBBg3TzzTfrgQceuCT1EEoAAAAAAAAXOTk5uvPOO9WkSRO98sorslgsl2Q/hBIAAAAAAPxONGrUSAEBAVq5cmWpfQ4ePKjk5GQ1atRIr7/+umy2SzfzA3NKAAAAAADwOxEUFKShQ4dq3LhxCggIUFxcnHJzc7Vx40YNHjxYP//8s5KTk1WrVi2lp6fr6NGjznVr1Kghq9Xq1XoIJQAAAAAA8FCv/uV/KoXJ2lXmcktUowpWUz5paWkKCQnRhAkTdODAAYWFhSk5OVmS9Nlnn2nPnj3as2eP2rVr57LemjVrFBER4dVaCCUAAAAAALiGpaamKjU11fmzn5+fhg0bpmHDhpXo279/f/Xv3/+y1cacEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnbL4uAAAAAACAq9Urr7zixa19eMEeI0aMuOitFhUVadSoUVq8eLHy8vKUkZGh+Ph4Twr0OkIJAAAAAACuYZmZmZo7d64yMjIUGRmp0NDQUvsWFRUpJSVFW7Zs0ZEjRxQSEqKOHTvqySef1PXXX+/12rh9AwAAAACAa1hWVpYcDodiY2PlcDjk7+9fZv8OHTpo0qRJ+vzzz/W///u/+vHHH3Xfffddktq4UgIAAAAAgGvUyJEjlZGRIUmqU6eOwsPDtWbNGk2ePFkzZ87U/v37Vb16dSUnJystLU1+fn66//77neuHh4dr2LBhuvfee3X27FlVrlzZq/URSgAAAAAAcI169tlnFR4erjlz5mjJkiWyWq0aN26cZsyYodGjRysuLk5HjhzR5s2b3a6fm5urefPmqU2bNl4PJCRCCQAAAAAArlnBwcEKCgqS1WqVw+HQqVOnNGXKFKWnp2vAgAGSpHr16ikmJsZlveeff15Tp07VmTNn1LZtW82YMeOS1MecEgAAAAAA/E7s3LlT+fn56tixY5n9HnroIS1dulSzZ8+W1WrV8OHDZYzxej1cKQEAAAAAAFxUr15d1atXV4MGDdSwYUPFxsbqq6++UlxcnFf3w5USAAAAAAD8TjRq1EgBAQFauXJludcpvkIiPz/f6/VwpQQAAAAAAL8TQUFBGjp0qMaNG6eAgADFxcUpNzdXGzdu1ODBg7V+/Xpt3rxZsbGxCgkJUVZWll588UVFRESoXbt2Xq+HUAIAAAAAAA+NGDGi3H1N1q4yl1uiGlWwmvJJS0tTSEiIJkyYoAMHDigsLEzJycmSpMqVK2vRokV68cUXdebMGTkcDnXq1En//ve/efoGAAAAAAC4OKmpqUpNTXX+7Ofnp2HDhmnYsGEl+v7hD3/Qe++9d9lqq1AosXTpUi1YsEB5eXkKDw/XkCFD1KxZs1L7b926VdOnT1d2drbsdrt69+6t7t27u/TJzc3V22+/rQ0bNujs2bNyOBy6//771bx584qUCgAAAAAArjAehxKrV6/WtGnTNHToUDVt2lTLli3TCy+8oJdeeklhYWEl+ufk5Gjs2LHq3Lmzhg8fru3bt+vNN99UcHCw2rdvL0k6deqUnnnmGTVt2lRpaWkKDg7WwYMHFRwc7PkrBAAAAAAAVySPQ4lFixYpISFBiYmJkqSUlBR9++23WrZsmQYOHFii/7Jly2S325WSkiJJCg8P1/fff6+FCxc6Q4n58+fLbre7XELicDg8LREAAAAAAFzBPAolCgoKtHv3bvXq1culvWXLltqxY4fbdXbt2qWWLVu6tLVq1UqfffaZCgoKZLPZtG7dOrVu3VovvfSStmzZIrvdrq5duyopKUkWi8WTUgEAAAAAwBXKo1Di+PHjKioqUkhIiEt7aGioNm3a5HadvLw83XDDDS5tISEhKiws1IkTJ2S325WTk6Nly5apR48e6tu3r7KysvTWW29Jkm655ZYS21y+fLmWL18uSRo3bpzb20Yq4uAFlnuyP5vN5vU6rxaMp/dcaCylix/P3+tYShyb3sZ4eg/nundxbHoX4+k9nOvexXh6F+f6fx08eFA2W8WeFfHLBZZ7uv2K1uUNAQEBnh0Pl6AWjxUVFalBgwbO2z/q1aunAwcOaOnSpW5DicTEROftI5J0+PDhy1arp/sLCwu77HVeLRhP77rYcWEsS8ex6V2Mp3dxrnsPx6Z3MZ7exbnuXYyn9/yezvX8/HxZrdZLuo+CgoKLXsdms3m0nrfl5+eX+r7Wrl271PX8PNlZcHCw/Pz8dOzYMZf2vLw8hYaGul0nNDRUeXl5Lm3Hjh2T1WpVtWrVJEl2u13h4eEufcLDw6/KAxYAAAAAAJTNo1DCZrOpfv362rhxo0v7pk2b1KRJE7frNGrUqMStHRs3blT9+vWdl5o0adJE+/fvd+mzf//+q/LSHgAAAAAAUDaPQglJ6tmzp1asWKHMzExlZ2dr6tSpOnr0qLp16yZJmjhxoiZOnOjs3717dx09elTTpk1Tdna2MjMztWLFCpfJMnv06KFdu3Zp3rx5+vnnn/Xll1/qww8/VFJSUgVeIgAAAAAAuBJ5PKdEfHy8Tpw4oXnz5ik3N1cRERFKS0vTddddJ6nkvUUOh0NpaWmaPn268/Gg9957r/NxoJLUsGFDPfHEE5o9e7b+85//KCwsTP379yeUAAAAAABckRzfp3lvY99fuEtOw7EXvdmioiKNGjVKixcvVl5enjIyMhQfH+9Bgd5XoYkuk5KSSg0M0tPTS7Q1b95c48ePL3Obbdu2Vdu2bStSFgAAAAAA+D+ZmZmaO3euMjIyFBkZWepckL919uxZ9ezZU9u2bdOSJUvUqlUrr9fm8e0bAAAAAADgypeVlSWHw6HY2Fg5HA75+/uXa73nnntO119//SWtjVACAAAAAIBr1MiRI5Wenq59+/apTp06iouLkzFGkyZNUocOHVSvXj1FR0dr7FjX20KWLl2q1atX669//eslra9Ct28AAAAAAIAr17PPPqvw8HDNmTNHS5YskdVq1bhx4zRjxgyNHj1acXFxOnLkiDZv3uxcZ//+/UpLS9PMmTNVuXLlS1ofoQQAAAAAANeo4OBgBQUFyWq1yuFw6NSpU5oyZYrS09M1YMAASVK9evUUExMjSSosLNTw4cP1wAMPqEWLFvrpp58uaX3cvgEAAAAAwO/Ezp07lZ+fr44dO7pd/sorr6hSpUp68MEHL0s9XCkBAAAAAAAkSatWrdLatWsVGRnp0t6rVy/17t1bEydO9Or+CCUAAAAAAPidaNSokQICArRy5UrVr1+/xPJ//etfOn36tPPngwcPauDAgXr11VcVGxvr9XoIJQAAAAAA+J0ICgrS0KFDNW7cOAUEBCguLk65ubnauHGjBg8erLp167r0DwwMlCRFRUWpdu3aXq+HUAIAAAAAAA/lNBx74U7/x2TtKnO5JapRBaspn7S0NIWEhGjChAk6cOCAwsLClJycfFn2/VuEEgAAAAAAXMNSU1OVmprq/NnPz0/Dhg3TsGHDLrhuRESE9u3bd8lq4+kbAAAAAADAJwglAAAAAACATxBKAAAAAAAAnyCUAAAAAAAAPkEoAQAAAADABRhjfF3CFc3T8SGUAAAAAACgHAgm3KvIuBBKAAAAAABwAZUrV9apU6cIJtzIz8+Xv7+/R+vavFwLAAAAAADXHKvVqipVquj06dOSJIvFctHbKPphZ5nL/Rx1LnqbAQEBys/Pv+j1vMUYI6vVqkqVKnm0PqEEAAAAAADlYLVaFRgY6PH6hXMml739rj0uepthYWE6fPiwpyX5HLdvAAAAAAAAnyCUAAAAAAAAPkEoAQAAAAAAfIJQAgAAAAAA+AShBAAAAAAA8AlCCQAAAAAA4BOEEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+QSgBAAAAAAB8glACAAAAAAD4BKEEAAAAAADwCUIJAAAAAADgE4QSAAAAAADAJwglAAAAAACATxBKAAAAAAAAnyCUAAAAAAAAPkEoAQAAAAAAfIJQAgAAAAAA+AShBAAAAAAA8AlCCQAAAAAA4BOEEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+QSgBAAAAAAB8glACAAAAAAD4BKEEAAAAAADwCVtFVl66dKkWLFigvLw8hYeHa8iQIWrWrFmp/bdu3arp06crOztbdrtdvXv3Vvfu3d32ff/99zV79mwlJSVp6NChFSkTAAAAAABcgTy+UmL16tWaNm2a+vXrp/Hjx6tJkyZ64YUXdPjwYbf9c3JyNHbsWDVp0kTjx49X3759NXXqVK1Zs6ZE3507d2r58uWKjIz0tDwAAAAAAHCF8ziUWLRokRISEpSYmKjw8HClpKTIbrdr2bJlbvsvW7ZMdrtdKSkpCg8PV2JiohISErRw4UKXfqdPn9arr76qhx56SIGBgZ6WBwAAAAAArnAehRIFBQXavXu3WrVq5dLesmVL7dixw+06u3btUsuWLV3aWrVqpd27d6ugoMDZNnnyZMXFxekPf/iDJ6UBAAAAAICrhEdzShw/flxFRUUKCQlxaQ8NDdWmTZvcrpOXl6cbbrjBpS0kJESFhYU6ceKE7Ha7li9frp9//lnDhw8vVx3Lly/X8uXLJUnjxo1TWFiYB6+mdAcvsNyT/dlsNq/XebVgPL3nQmMpXfx4/l7HUuLY9DbG03s4172LY9O7GE/v4Vz3LsbTuzjXvYvxLKlCE1160/79+zV79mw999xzstnKV1ZiYqISExOdP5c2n8Wl4sn+wsLCLnudVwvG07sudlwYy9JxbHoX4+ldnOvew7HpXYynd3Guexfj6T2c6951rY5n7dq1S13mUSgRHBwsPz8/HTt2zKU9Ly9PoaGhbtcJDQ1VXl6eS9uxY8dktVpVrVo1fffddzpx4oQee+wx5/KioiJt27ZNH3/8sWbOnKlKlSp5Ui4AAAAAALgCeRRK2Gw21a9fXxs3btSNN97obN+0aZPi4uLcrtOoUSOtW7fOpW3jxo2qX7++bDabYmNj9Y9//MNl+b///W/VqlVL/fr1K/fVEwAAAAAA4Org8Tf9nj176tVXX1XDhg3VpEkTffzxxzp69Ki6desmSZo4caIkadiwYZKk7t27a+nSpZo2bZoSExO1Y8cOrVixQo888ogkKTAwsMTTNgICAhQUFKS6det6WiYAAAAAALhCeRxKxMfH68SJE5o3b55yc3MVERGhtLQ0XXfddZJK3gvjcDiUlpam6dOnOx8Peu+996p9+/YVewUAAAAAAOCqVKF7IpKSkpSUlOR2WXp6eom25s2ba/z48eXevrttAAAAAACAa4OfrwsAAAAAAAC/T4QSAAAAAADAJwglAAAAAACATxBKAAAAAAAAnyCUAAAAAAAAPkEoAQAAAAAAfIJQAgAAAAAA+AShBAAAAAAA8AlCCQAAAAAA4BOEEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+QSgBAAAAAAB8glACAAAAAAD4BKEEAAAAAADwCUIJAAAAAADgE4QSAAAAAADAJwglAAAAAACATxBKAAAAAAAAnyCUAAAAAAAAPkEoAQAAAAAAfIJQAgAAAAAA+AShBAAAAAAA8AlCCQAAAAAA4BOEEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJQgkAAAAAAOAThBIAAAAAAMAnCCUAAAAAAIBPEEoAAAAAAACfIJQAAAAAAAA+QSgBAAAAAAB8glACAAAAAAD4BKEEAAAAAADwCUIJAAAAAADgE4QSAAAAAADAJwglAAAAAACATxBKAAAAAAAAnyCUAAAAAAAAPkEoAQAAAAAAfIJQAgAAAAAA+AShBAAAAAAA8AlCCQAAAAAA4BOEEgAAAAAAwCcIJQAAAAAAgE8QSgAAAAAAAJ8glAAAAAAAAD5BKAEAAAAAAHyCUAIAAAAAAPgEoQQAAAAAAPAJW0VWXrp0qRYsWKC8vDyFh4dryJAhatasWan9t27dqunTpys7O1t2u129e/dW9+7dncvff/99ffXVV9q/f79sNpsaNWqkgQMHqm7duhUpEwAAAAAAXIE8vlJi9erVmjZtmvr166fx48erSZMmeuGFF3T48GG3/XNycjR27Fg1adJE48ePV9++fTV16lStWbPG2Wfr1q3q3r27nnvuOY0ePVpWq1XPPfecTp486WmZAAAAAADgCuVxKLFo0SIlJCQoMTFR4eHhSklJkd1u17Jly9z2X7Zsmex2u1JSUhQeHq7ExEQlJCRo4cKFzj5PPfWUOnfurLp166pu3boaPny4jh8/ru3bt3taJgAAAAAAuEJ5FEoUFBRo9+7datWqlUt7y5YttWPHDrfr7Nq1Sy1btnRpa9WqlXbv3q2CggK365w5c0bGGAUFBXlSJgAAAAAAuIJ5NKfE8ePHVVRUpJCQEJf20NBQbdq0ye06eXl5/5+9O4+3qq73x/9ikIAUDsrsAf0igwJCGfo9maAZDg/MIW4dHMAEynKMvDbw8GsaV0O8XTUTLczQmxIOOaRYGSQ30Yt+r6UIKmg5fOEGhHqOoiAy/P7w576eGA7nsHGBPp+PBw9da6+19me99xpf+7PXyf77719nXNu2bbNu3bq88cYbadeu3UbzTJ06NXvvvXd69+69yWXOnDkzM2fOTJJcdtllad++fWNWZ7OW1fN6Y96vefPmZW/nzkI9y6e+WiYNr+dHtZaJbbPc1LN87OvlZdssL/UsH/t6ealnednXy0s9N7ZND7rcnm666aYsXLgwEyZMSNOmm+7QMXTo0AwdOrQ0vLnnWWwvjXm/9u3bf+Dt3FmoZ3k1tC5quXm2zfJSz/Kyr5ePbbO81LO87OvlpZ7lY18vrw9rPbt27brZ1xr18402bdqkadOmqa2trTO+pqYmFRUVm5ynoqIiNTU1dcbV1tamWbNm2W233eqMv/HGG/Pwww/ne9/7Xjp16tSYJgIAAAA7uEaFEs2bN0+PHj0yb968OuOfeuqp9OnTZ5Pz9OrVa6OfdsybNy89evRI8+b/02Fj6tSppUBizz33bEzzAAAAgJ1Ao//6xuc///nMnj07s2bNyuLFizN16tS8+uqrOeKII5Ik11xzTa655prS9EceeWReffXV3HjjjVm8eHFmzZqV2bNn59hjjy1N87Of/SyzZ8/ON77xjey6666pqalJTU1NVq9evQ2rCAAAAOyIGv1MiYMPPjhvvPFG7rzzzrz22mvp1q1bxo8fnw4dOiTZ+LcwHTt2zPjx43PTTTeV/jzo6NGjU1VVVZrmvT8nOmHChDrzfvGLX0x1dXVjmwoAAADsgLbpQZdHHXVUjjrqqE2+dvHFF280rm/fvpk0adJml3fbbbdtS3MAAACAnUijf74BAAAAsC2EEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAqqE3eQAAIABJREFUAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIYQSAAAAQCGEEgAAAEAhhBIAAABAIZoX3QAAAACgPK6++uotvn7uued+QC3ZOnpKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIXYpgdd/u53v8uvf/3r1NTUpLKyMqeddlr222+/zU7/9NNP56abbsrixYvTrl27HHfccTnyyCO3aZkAAADAzqnRPSUeeeSR3HjjjfnCF76QSZMmpU+fPvnBD36QFStWbHL65cuXZ+LEienTp08mTZqUE044IVOnTs3cuXMbvUwAAABg59XoUOK+++7LoYcemqFDh6aysjJjxoxJu3bt8sADD2xy+gceeCDt2rXLmDFjUllZmaFDh+bQQw/Nvffe2+hlAgAAADuvRoUSa9euzV//+tcMHDiwzvgBAwZk4cKFm5znueeey4ABA+qMGzhwYP76179m7dq1jVomAAAAsPNq1DMlXn/99axfvz5t27atM76ioiJPPfXUJuepqanJ/vvvX2dc27Zts27durzxxhvZsGFDg5c5c+bMzJw5M0ly2WWXpX379o1Znc2765EtvvyZH82pdxFfad75H8bU1Bl6Ydm/b3H+S4b9rd73WF91fb3T7BC2sZ4b1zIpdz0/LLVMGlPPhtUy+ejU077eQDvBvp7sJPW0r5fXDrCvJ+r5fo6d/78PYF9PXCe9n2NnA3wA+/ros3pucf6mc79a73t8VOq5NddJEyZM2OIy6qvnB13LbXrQZdGGDh2aoUOHloY/qs+e+Kiu9/agluWlnuWlnuWlnuWjluWlnuWlnuWjluWlnv+jvlp0LMMyPkq2tZ7bo5Zdu3bd7GuNCiXatGmTpk2bpra2ts74mpqaVFRUbHKeioqK1NTUTXBqa2vTrFmz7LbbbknS4GUCAAAAO69GPVOiefPm6dGjR+bNm1dn/FNPPZU+ffpscp5evXpt9DOMefPmpUePHmnevHmjlgkAAADsvBr91zc+//nPZ/bs2Zk1a1YWL16cqVOn5tVXX80RRxyRJLnmmmtyzTXXlKY/8sgj8+qrr+bGG2/M4sWLM2vWrMyePTvHHnvsVi8TAAAA+PBo9DMlDj744Lzxxhu5884789prr6Vbt24ZP358OnTokGTj36F07Ngx48ePz0033VT686CjR49OVVXVVi8TAAAA+PDYpgddHnXUUTnqqKM2+drFF1+80bi+fftm0qRJjV4mAAAA8OHR6J9vAAAAAGwLoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQiOZFNwAAAABI7jll3y2+fu+tNR9QSz44ekoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFaF50AwA+Cs4999wtT/D8+A+mIR8S6lk+9dYyUc8GUM/ysq+Xl3qWj30dykdPCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQHnQJAAAAHxHLe04sugl16CkBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFKJ50Q0AAACAzVnec2LRTWA70lMCAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAAChE88bMtGHDhtx+++2ZNWtWVq5cmV69emXs2LHp1q3bFuebO3dubr311ixbtiydOnXKSSedlIMOOihJsnbt2kyfPj1PPPFEli1bllatWqVfv3455ZRT0r59+8Y0EwAAANiBNaqnxD333JP77rsvo0ePzsSJE9OmTZtccsklWbVq1WbnWbRoUa666qoMHjw4l19+eQYPHpwrrrgizz33XJJkzZo1eeGFFzJ8+PBMmjQp3/72t/PKK6/k0ksvzbp16xq3dgAAAMAOq8GhxIYNG3L//ffnhBNOSFVVVbp3756zzz47q1atypw5czY734wZM9KvX78MHz48lZWVGT58ePr165cZM2YkSVq3bp0LL7wwBx98cLp27ZqePXvm9NNPz5IlS7JkyZLGryEAAACwQ2pwKLF8+fLU1NRkwIABpXEtWrTIfvvtl4ULF252vkWLFmXgwIF1xg0cODCLFi3a7DxvvfVWkuTjH/94Q5sJAAAA7OAa/EyJmpqaJElFRUWd8W3bts1rr722xfnatm270TzvLe8frV27Nr/4xS/yqU99Knvssccmp5k5c2ZmzpyZJLnssst2ymdPTJgwYcsTzP1qvcvYGdd7e9nWeqrl/6i3lol6NkC9tXi+DMv4CFHP8tmqOtRTT7X8H+pZXvb18trWeqrl/7Cvl5daNMSm75/fb2erZ72hxEMPPZQpU6aUhsePH79dG5Qk69aty9VXX50333wz3/72tzc73dChQzN06NDS8IoVK7Z728qtvjZ3LMMyPkq2tZ5q+T+2phbqufXs6+WlnuVjXy8v9Swv+3p5uU4qH/t6ealFee2I9ezatetmX6s3lBg0aFB69epVGn7nnXeSvNvz4f0JTG1t7UY9Id6voqIitbW1dcbV1tZu1ONi3bp1+dGPfpSXX345F198cXbbbbf6mggAAADshOp9pkSrVq3SuXPn0r/KyspUVFRk3rx5pWnWrFmTZ599Nn369Nnscnr37l1nniSZN29eevfuXRpeu3Ztrrzyyrz00ku56KKLNgosAAAAgA+PBj9TokmTJhk2bFjuuuuu7LnnnunSpUvuvPPOtGzZMoccckhpugkTJqRnz545+eSTkyTDhg3LRRddlLvvvjsHHnhgHnvssSxYsKD0u/V169bliiuuyF/+8pd85zvfSZMmTUrPm2jdunVatGhRjvUF2C6OHVE3RG3fvv0O2XVuZ/CPtUzUc1vYNstLPctLPctHLctLPeGD0+BQIkmOP/74rFmzJjfccEPefPPN9OzZMxdccEFatWpVmmbZsmV1HlDZp0+fjBs3LtOnT8+tt96azp07Z9y4caWfhrzyyiv5r//6ryTJd7/73Trvd+aZZ+awww5rTFMBAACAHVSjQokmTZqkuro61dXVm51m8uTJG42rqqpKVVXVJqfv2LFjbrvttsY0BwAAANgJ1ftMCQAAAIDtQSgBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFKJRf32DD87ynhOLbsKHinqWl3oCAADbQk8JAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQzYtuAMCO7p5T9i26CR8q6lle6lk+alle6lle6lle6gk7Dj0lAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQjQvugE7s3tO2bfeae69teYDaMmHQ331VMuGUU8AAGBHp6cEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQiOaNmWnDhg25/fbbM2vWrKxcuTK9evXK2LFj061bty3ON3fu3Nx6661ZtmxZOnXqlJNOOikHHXTQJqedMmVKZs6cmZEjR+a4445rTDMBAACAHVijekrcc889ue+++zJ69OhMnDgxbdq0ySWXXJJVq1Ztdp5FixblqquuyuDBg3P55Zdn8ODBueKKK/Lcc89tNO3cuXPz/PPPp127do1pHgAAALATaHAosWHDhtx///054YQTUlVVle7du+fss8/OqlWrMmfOnM3ON2PGjPTr1y/Dhw9PZWVlhg8fnn79+mXGjBl1pvv73/+eqVOn5txzz03z5o3qyAEAAADsBBocSixfvjw1NTUZMGBAaVyLFi2y3377ZeHChZudb9GiRRk4cGCdcQMHDsyiRYtKw+vWrcuPfvSj/NM//VMqKysb2jQAAABgJ9Lgrgg1NTVJkoqKijrj27Ztm9dee22L87Vt23ajed5bXpLcdttt2W233XLkkUduVVtmzpyZmTNnJkkuu+yytG/ffqvm+2DVbPHVHbPNO6ot1zJRz4axbW5PzZs3b1gNn69/ko/yZ6Ke5dPgWib11vOjWstEPcvNvl4+ts3yUs/tSy0a4sN3T1RvKPHQQw9lypQppeHx48dvl4YsWLAgs2fPzr/+679u9TxDhw7N0KFDS8MrVqzYHk3brnbGNu/I1LN81HLbtG/fvkE17LgV03yUPxP1LJ+G1jKpv54f1Vom6llu9vXysW2Wl3puX2pRXjtiPbt27brZ1+oNJQYNGpRevXqVht95550k7/Z8eH8CU1tbu1FPiPerqKhIbW1tnXG1tbWlHhcLFixITU1NTj/99NLr69evzy233JL7778/P/nJT+prKgAAALATqTeUaNWqVVq1alUa3rBhQyoqKjJv3rz07NkzSbJmzZo8++yzGTly5GaX07t378ybN6/On/ecN29eevfunSQ56qijUlVVVWeeSy+9NJ/5zGfq9IYAAAAAPhwa/KDLJk2aZNiwYbnnnnvy6KOP5uWXX861116bli1b5pBDDilNN2HChEybNq00PGzYsMyfPz933313lixZkrvuuisLFizIMccck+Td50t07969zr/mzZunoqJii109AAAAgJ1To/7m5vHHH581a9bkhhtuyJtvvpmePXvmggsuqNOjYtmyZdljjz1Kw3369Mm4ceMyffr03HrrrencuXPGjRtX56chAAAAwEdHo0KJJk2apLq6OtXV1ZudZvLkyRuNq6qq2ugnGluyqWUAAAAAHw4N/vkGAAAAQDkIJQAAAIBCCCUAAACAQjTqmRIAAABwzyn71jvNvbfWfAAtYWelpwQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQCKEEAAAAUAihBAAAAFAIoQQAAABQiOaNmWnDhg25/fbbM2vWrKxcuTK9evXK2LFj061bty3ON3fu3Nx6661ZtmxZOnXqlJNOOikHHXRQnWn++7//O9OmTcv8+fOzdu3a7LnnnjnnnHNSWVnZmKYCAAAAO6hG9ZS45557ct9992X06NGZOHFi2rRpk0suuSSrVq3a7DyLFi3KVVddlcGDB+fyyy/P4MGDc8UVV+S5554rTbN8+fJceOGF6dixY773ve/l3/7t3zJixIi0bNmyMc0EAAAAdmANDiU2bNiQ+++/PyeccEKqqqrSvXv3nH322Vm1alXmzJmz2flmzJiRfv36Zfjw4amsrMzw4cPTr1+/zJgxozTNL3/5ywwcODCnnnpqevTokU6dOuWAAw5I+/btG7d2AAAAwA6rwaHE8uXLU1NTkwEDBpTGtWjRIvvtt18WLly42fkWLVqUgQMH1hk3cODALFq0KEmyfv36PP7446msrMyll16asWPHZvz48XnkkUca2kQAAABgJ9DgZ0rU1NQkSSoqKuqMb9u2bV577bUtzte2bduN5nlvea+//npWr16du+66KyNGjMgpp5yS+fPn5+qrr07Lli1zwAEHbLTMmTNnZubMmUmSyy67bAftUVGzxVd3zDbvqLZcy0Q9G8a2uT01b968YTV8vv5JPsqfiXqWT4NrmdRbz49qLRP1LDf7evnYNstLPbeV687y+fDdE9UbSjz00EOZMmVKaXj8+PHbpSHr169PkgwaNCif//znkyR77713/vKXv+S3v/3tJkOJoUOHZujQoaXhFStWbJe2bU87Y5t3ZOpZPmq5bdq3b9+gGnbcimk+yp+JepZPQ2uZ1F/Pj2otE/UsN/t6+dg2y0s9ty+1KK8dsZ5du3bd7Gv1hhKDBg1Kr169SsPvvPNOknd7Prw/gamtrd2oJ8T7VVRUpLa2ts642traUo+LNm3apFmzZhv9lY0999zTTzgAAADgQ6jeZ0q0atUqnTt3Lv2rrKxMRUVF5s2bV5pmzZo1efbZZ9OnT5/NLqd379515kmSefPmpXfv3kne7RK1zz775L//+7/rTPO3v/0tHTp0aNBKAQAAADu+Bj9TokmTJhk2bFjuuuuu7LnnnunSpUvuvPPOtGzZMoccckhpugkTJqRnz545+eSTkyTDhg3LRRddlLvvvjsHHnhgHnvssSxYsCATJkwozXPcccflyiuvzH777Zf+/ftn/vz5eeSRR/Ktb32rDKsKAADAB+3YEXWfR9iYn8Pw4dXgUCJJjj/++KxZsyY33HBD3nzzzfTs2TMXXHBBWrVqVZpm2bJl2WOPPUrDffr0ybhx4zJ9+vTceuut6dy5c8aNG1fnpyEHHXRQvva1r+Wuu+7K1KlT06VLl5x11lmbfJ4EAAAAsHNrVCjRpEmTVFdXp7q6erPTTJ48eaNxVVVVqaqq2uKyDzvssBx22GGNaRYAAACwE6n3mRIAAAAA24NQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAAChE86Ib8GF37IiKOsPt27fPihUrCmrNzu0fa5mo57awbQIAAEXTUwIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAoRPOiGwAAAADU79gRFRuNa9++fVasWFFAa8pDTwkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBDNGzPThg0bcvvtt2fWrFlZuXJlevXqlbFjx6Zbt25bnG/u3Lm59dZbs2zZsnTq1CknnXRSDjrooNLrq1evzrRp0/LYY4/ljTfeSPv27XPEEUfk85//fGOaCQAAAOzAGtVT4p577sl9992X0aNHZ+LEiWnTpk0uueSSrFq1arPzLFq0KFdddVUGDx6cyy+/PIMHD84VV1yR5557rjTNTTfdlD/96U85++yzc+WVV2b48OGZNm1a/vjHPzammQAAAMAOrMGhxIYNG3L//ffnhBNOSFVVVbp3756zzz47q1atypw5czY734wZM9KvX78MHz48lZWVGT58ePr165cZM2aUplm0aFGGDBmS/v37p2PHjjn00EPTq1evOsEFAAAA8OHQ4FBi+fLlqampyYABA0rjWrRokf322y8LFy7c7HyLFi3KwIED64wbOHBgFi1aVBru06dPHn/88axYsSJJsnDhwrz44ov5xCc+0dBmAgAAADu4Bj9ToqamJklSUVFRZ3zbtm3z2muvbXG+tm3bbjTPe8tLkjFjxmTKlCk588wz06xZsyTJ6NGj86lPfWqTy5w5c2ZmzpyZJLnsssvSvn37hq7OB6558+Y7RTt3FupZPmpZXg2u5/P1T/JR/nzUs3wata/XU8+Pai0T9Sw3+3r52DbLSz3Ly3Vnee3s9aw3lHjooYcyZcqU0vD48eO3W2N+85vfZOHChfn2t7+dDh065JlnnskvfvGLdOzYcZO9JYYOHZqhQ4eWht/rYbEja9++/U7Rzp2FepaPWpZXQ+vZcSum+Sh/PupZPo3Z1+ur50e1lol6lpt9vXxsm+WlnuXlurO8doZ6du3adbOv1RtKDBo0KL169SoNv/POO0ne7fnw/jSmtrZ2o54Q71dRUZHa2to642pra0s9LtasWZNp06blvPPOy6BBg5Ike+21V1588cXce++9fsIBAAAAHzL1hhKtWrVKq1atSsMbNmxIRUVF5s2bl549eyZ5N1B49tlnM3LkyM0up3fv3pk3b16OO+640rh58+ald+/eSZK1a9dm3bp1adq07mMumjZtmvXr1zdsrQB2Mst7Tiy6CR8q6lle6lle6lk+alle6lle6glbp8EPumzSpEmGDRuWe+65J48++mhefvnlXHvttWnZsmUOOeSQ0nQTJkzItGnTSsPDhg3L/Pnzc/fdd2fJkiW56667smDBghxzzDFJktatW6dv376ZNm1aFixYkOXLl2f27Nn5j//4jxx00EFlWFUAAABgR9LgB10myfHHH581a9bkhhtuyJtvvpmePXvmggsuqNOjYtmyZdljjz1Kw3369Mm4ceMyffr03HrrrencuXPGjRtX56ch48aNy7Rp03L11Vdn5cqV6dChQ0aMGJGjjz56G1YRAAAA2BE1KpRo0qRJqqurU11dvdlpJk+evNG4qqqqVFVVbXaeioqKnHnmmY1pEgAAALCTafDPNwAAAADKQSgBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFKLJhg0bNhTdCAAAAOCjR0+JD9h3v/vdopvwoaKe5aOW5aWe5aWe5aOW5aWe5aWe5aOW5aWe5aWe5bWz11MoAQAAABRCKAEAAAAUotnFF198cdGN+Kjp0aNH0U34UFHP8lHL8lLP8lLP8lHL8lLP8lLP8lHL8lLP8lLP8tqZ6+lBlwAAAEAh/HwDAAAAKIRQAgAAACiEUKIAZ511Vn79618X3Qw+BNavX58pU6ZkzJgxqa6uzoIFC7br+1188cW54YYbtut7fFip3cYmT56cyy67rOhmsA2WL1+e6urq/OUvfym6KdvNggULUl1dnddff32Tw1tr9uzZGTVq1PZo4nZVXV2duXPnbtW0jnPsDG677bb88z//c9HN+FD5KJwLdlRbc2559tlnc/755+ekk07Kjvo4SaHE+0yePDnV1dWlf2PHjs1ll12WJUuWFNquxl4A7UjeX9uTTjopZ599dv793/89q1evrnfeza3/znzxs7m2N/Si9c9//nMefPDBfOc738mUKVPSp0+fcjZzI+eff35OPvnk7foeRSrHTfKHYX/dGpMmTcqECRM2+drixYtTXV2dJ5988gNu1Y6tpqYmU6dOzTnnnJOTTz45X/va1/KDH/wgf/rTn4pu2lbZGUOkTbX58ccfz8iRIzN9+vTt9r474pcPv//97zNq1KisXbu2NG7t2rUZOXLkRjdoS5fZCLt7AAAgAElEQVQuTXV1dZ566qlMmTIln/rUpz7o5pbVjn5c3pp96x/DofXr1+faa6/NmDFj8txzz23vJn5g3n+9eOKJJ+aMM87I9ddfn5UrVxbdtJKt3Z7eu1EfMWJEVqxYUee1lStX5pRTTtkhb+Qbc65q3759pkyZkr333vuDa+gOZEe9h3zPjTfemL322is//vGPc/7552/z8rbHMbV52Zb0IbH//vvnnHPOSZK8+uqrufnmm/PDH/4wV155ZcEt2/m9V9u1a9fm2WefzU9+8pO8/fbb+epXv1p003ZaS5cuTbt27bY5jFi7dm2aN6//cLDrrrtu0/vw4XH44Yfnhz/8YZYvX56OHTvWee0Pf/hDOnTokP3337+g1u14li9fngsvvDCtWrXKSSedlL333jvr16/P/Pnzc/311+e6664ruokfCX/84x/zk5/8JCNHjsywYcOKbs4Hql+/fnn77bfz/PPPZ999902SPPfcc2ndunX+9re/5fXXX0+bNm2SJPPnz88uu+ySPn36pEWLFkU2e4eztefL7WnNmjW56qqr8sILL2TChAmprKwstD3l9t714rp167J48eJcd911efPNNzNu3Liim9You+++ex588MF86UtfKo2bM2dO2rZtu1FYUbTGnKve2ycqKioKaPGOo6H3kB/ksWTp0qU56qij0r59+w/k/RpDKPEPdtlll9JOVVFRkWOOOSaTJk3KmjVrSifmW265JY899lhWrFiRioqKfPrTn051dXWdE/ef/vSn3HHHHXnppZfysY99LL1798555523yZP7H//4x9xwww0555xzMmjQoDqvLV++PN///veTJF/5yleSJIceemjOOuusvPPOO7nlllvy8MMP56233sree++dUaNGlS42djTvr+0hhxyS+fPn5//+3/+bPn365De/+U2WLFmSFi1apG/fvjnttNOy++67b3b9k+Tpp5/O008/nd/97ndJkmuuuSYdO3bM008/nZtvvjkvvfRSWrdunc985jMZOXJkace/+OKLU1lZmdatW2fWrFlp0qRJhgwZkpEjR6Zp0x2n89DkyZPzxhtvZMCAAbnnnnuyZs2aHHjggRk7dmw+9rGPZfLkyfmP//iPJO9+g9KhQ4dMnjy53u1iwYIF+f73v5/vfve7uf322/Piiy/m/PPPT79+/fKzn/0sjz76aFq2bJlhw4Zl4cKF2W233XLWWWclebd23bp1y9ixYzNt2rQ8+eSTmTRpUp12/5//83/So0ePjBkzJkny4IMP5te//nWWL1+e9u3b54gjjsiwYcNKta6urs7pp5+eefPm5c9//nPatm2b6urqDBky5IMq9WbV9xn8oy3tr0myYcOGTJs2bbPb3dq1azN9+vTMmTMnK1euTLdu3TJixIh84hOf+IDWeOsdcMABadu2bWbPnp3q6urS+LVr1+ahhx7KUUcdlWeffXaL++I/2rBhQ+677778/ve/z4oVK9KmTZsMGTKk1DunvmPvbbfdlkcffTTDhw/P9OnTU1tbm/79++frX/966WZr/fr1ufPOOzNr1qzU1tamS5cuOfHEE3PggQdu13q91zPqsssuS8uWLUvjKysrM3jw4NLwihUrMnXq1Dz11FNJkgEDBmT06NHZY489tnodn3/++UyfPj0vvPBC1q5dm+7du2fUqFHp3bt36X0aut/ddtttdY43SXLRRRelQ4cOSZK///3vmTZtWhYuXJgOHTpk9OjRGTBgQGn+xYsX5xe/+EWeeeaZtGjRIv37989pp532gV7EzpgxI7fccku+/vWvl9Zz9uzZueGGG/KLX/yiNN17x8if/exnpZo2xMUXX5y///3vufnmm3PzzTcnebd+73nqqady4403Zvny5enZs2fOOOOMOsHef/3Xf+X222/P4sWLU1FRkUMOOSRf+tKXtvnitWvXrmnXrl3mz59f53zQv3///P3vf8+CBQvy6U9/ujS+d+/eadGiRaqrq3PeeeelqqoqSXLHHXfkD3/4Q2pqavLxj388AwcOzNlnn116n/qOc2eddVY++9nPZtmyZXn00Ufz8Y9/PKNGjcrAgQNz/fXX5/HHH0+7du0yduzYDBw4MMm7++1Pf/rTzJ8/PzU1Ndljjz3yuc99Lscee2xpuS+//HJuvPHG/OUvf8n69evTuXPnfPnLX07Hjh03e1zesGFDfv3rX2fmzJl59dVX07lz5xx//PGl7WP58uU5++yzc+6552bWrFlZtGhRRo0alaOPPnqbPott8dZbb+Xyyy9PTU1N/uVf/qXOTcbW7Ncvv/xybrrppjz77LNp0aJFBg0alNGjR6d169ZZsmRJvvnNb2bKlCmpqKjI22+/ndGjR6dfv3654IILkiSzZs3K3XffnR//+MfbbR3ff724xx575OCDD87s2bNLr2/Ncfy9m8Innngia9asSZcuXfLlL385/fv33+j9VqxYkUsuuaS0P27YsGGz5+L6zvObcthhh2X27Nn54he/mCZNmiR5N7w/7LDDcscdd9SZthznuW2xNeeq6urqjBkzJvPnz8+TTz6ZI444IkcffXTOPvvsTJw4Mfvss0/pODp+/PhMnz49ixcvzj777JNvfOMbWbZsWaZOnZqlS5emX79+Oeuss7Lbbrsl2fx+/N7ntiNf42/pHrKmpmazx5L6rpPvu+++zJ49O8uWLUvr1q3zyU9+MqNGjcrHP/7xTbZj5cqVmTRpUlq2bJlTTz211BPuuuuuy3XXXZczzzwzQ4YM2S7H1G0hlNiCVatW5ZFHHkn37t3rhAkf+9jHcsYZZ2T33XfP4sWLc/3116d58+Y58cQTkyRPPPFELr/88pxwwgk588wzs27dujz55JPZ1F9fvf/++3P77bfnO9/5Tvr27bvR6+3bt88///M/59/+7d9yxRVXZNdddy215eabb85//ud/li5o7rvvvlx66aW5+uqr065du+1UlfJp0aJF1q1bl7Vr1+ZLX/pS9txzz7zxxhu55ZZb8qMf/Sjf//73t7j+f/vb39K1a9fSDUubNm3y6quvZuLEiRk8eHDOPPPMLFu2LD/5yU/StGnTnHrqqaX3fuihhzJs2LD8y7/8S1588cVcffXV6dGjRw455JBCarE5zzzzTCoqKnLhhRfmlVdeyZVXXpkuXbrkC1/4QkaPHp0OHTrkwQcfzMSJE0sHka3dLm655Zaceuqp6dy5c1q1apV///d/z9NPP51vfetbadeuXX71q1/lmWeeyUEHHbTJtg0ZMiR33313lixZkj333DNJsmzZsixatCinnXZakmTmzJm57bbbMmbMmPTo0SMvv/xyfvrTn6Z58+Z1LuruuOOOnHzyyTn55JPzhz/8Idddd1369u27QyS6W/oM/tGWttek/u3u2muvzbJly3Luuedmjz32yJ///OdMmjQpEydO3OG6RDZr1iyHHnpo6ULrve3v8ccfz+uvv55DDz005513Xr374vv98pe/zAMPPJBTTz01ffv2zeuvv54XXnih9Hp9x97k3ZuIRx55JOeff37efvvtXHXVVZk+fXpOP/30JO8ec++999589atfTY8ePfLQQw/lhz/8YSZNmrTdarxy5co88cQTGTFiRJ2LvPe8d2Gxfv36XH755WnRokUuuuiiJMnPf/7z/Ou//msmTpxYupitbx1Xr16dIUOG5LTTTkuTJk3y29/+NhMnTszVV19duvBLGrbfHXfccVmyZElWrlxZ+iZo1113zauvvpokmT59ekaOHJmvfOUr+dWvfpWrrroq1157bVq2bJnXXnstF110UT772c9m1KhRWbduXX75y1/m8ssvzyWXXPKBhMHTp0/Pfffdl/PPPz8HHHDAdn2v888/P9/61rfy2c9+NkceeWSd19auXZu77747Z5xxRnbZZZdMnjw5119/femG74knnsiPf/zjnHbaadlvv/2yYsWKXH/99XnnnXc2u980RL9+/bJgwYJ88YtfTPJu+HDIIYdk+fLldUKJp59+OkccccRG88+dOzf33ntvvvGNb6R79+6pra3d6KcDW3N+nTFjRk488cQMHz48v//97zN58uT0798/Bx98cE488cTcdddd+fGPf5xrr702LVq0yPr167P77rvnm9/8Ztq0aZPnn38+U6ZMyW677ZbDDz88SfKjH/0oe+21V37wgx+kWbNmefnll9OiRYstHpenT5+euXPnZuzYsenatWsWLVqUn/70p9l1113rbCe//OUvM2rUqJxxxhlp1qzZNn8OjVVbW5vvf//7adKkSSZMmLDJm9At7derV6/OpZdemn322ScTJ07MypUr89Of/jTXXnttzj///Oy5556pqKjIggUL8pnPfCYLFy5Mq1atsnDhwqxbty7NmjXLggUL0q9fvw9snZctW5YnnniiTt3rO46vXr06F198cdq2bVu6pnnppZc2ufzFixfn0ksvTVVVVU499dQ0adIkV1999WbPxd27d9/ieX5TPvnJT2bWrFmZP39+9t9//7zwwgtZtmxZPv3pT28USpTjPNdYW3uuSt7dzk466aSMGjWqdG7alNtuuy2nnXZaWrdunauvvjpXXXVVdtlll5x++ulp2rRprrjiitx+++2lL7I2tx8n2amu8Td3D/mPx5KtuU5u0qRJTjvttHTs2DErVqzIz3/+8/z85z8vnYvf79VXX82ll16aysrKnHPOOWnatGmmTJmSc845JyeddFIOPvjgtG7dersdU7fFjvO18A7iiSeeyKhRozJq1Kh8+ctfztNPP51zzz23zjRf/OIXs++++6Zjx4454IAD8oUvfCEPP/xw6fVf/epXqaqqyoknnpjKysrstddeOe644zb6ZnX69Om566678r3vfW+TgUSSNG3atNRlvk2bNqmoqEjr1q2zevXqPPDAAznllFNywAEHpLKyMqeffnoqKipKPQd2ZM8//3wefvjh9O/fP4cffngOOOCAdOrUKT179sxXvvKVPPPMM3nllVc2u/6tW7dO8+bN87GPfSwVFRWpqKhI06ZN87vf/S7t2rXLV77ylVRWVuZTn/pUTjnllPz2t7/N22+/XXr/ysrKjBgxIl27ds3BBx+cfv36Zf78+UWVY7Nat26d008/PZWVlRk4cGCqqqpK7WzdunVatmyZpk2bpqKiIm3atGnQdvGlL30pAwcOTKdOndKiRYs8+OCDOeWUUzJgwIB069YtX//617d4s1BZWZn/9b/+Vx566KHSuDlz5qRLly7p2bNnknf3hZEjR6aqqiodO3bMoEGDcsIJJ2zUliFDhmTIkCHp3LlzRowYkWbNmuXpp58uVxm3yZY+g3+0ue31PVva7pYuXZqHH3443/zmN9O3b9906tQpRx99dD75yU9m5syZ239FG+Hwww/PihUrSt/qJ+9++zNw4MDMnDlzq/bF96xevTozZszIySefnMMPPzydO3dO7969c9RRR5Wmqe/Ym7x7Y3/WWWdlr732Su/evTN06NA67bv33ntz7LHH5pBDDknXrl0zYsSI7Lffftv19/9Lly7Nhg0b6u1iPX/+/Lz00ks599xzs88++2SfffbJueeemxdeeKHOOtS3jv3798+QIUNSWVmZPffcM2PGjMkuu+ySP//5z3XeryH7XcuWLdOiRYvSN0EVFRV1vrk/5phjMmjQoHTp0iUnn3xyVq5cmRdffDFJ8sADD2SvvfbKyJEjS+fEs88+O88//3z++te/NrScDTZv3rzceeedOe+887Z7IJG8G9Y0bdo0LVu2LNXqPevWrcvYsWPTs2fP7LXXXjn22GOzYMGC0pcWd911V4499th89rOfTefOndO/f/+ccsop+f3vf7/JLzYaqn///lm0aFHeeeedrFmzJosWLUq/fv3St2/f0oOSlyxZktdee22z3yhXVFRkwIABad++ffbZZ5+Neg1szfl14MCBOeqoo9KlS5dUV1fnnXfeSadOnXLooYemc+fO+ad/+qe8/vrr+X//7/8lSZo3b54RI0akZ8+e6dixYw4++OAcccQRdfb/FStWZMCAAdlzzz3TuXPnHHTQQendu/cWr6Puu+++fP3rX88nPvGJdOzYMYccckg+97nPbXSOOvroo0vnsfd6LRXhpptuyltvvZWLLrpos9+Kb2m/njNnTlavXp1zzjkn3bt3T9++fXP66afnsccey9KlS5Okzrbw9NNPp6qqKrvttlvpuQfPPPPMZq9Zy+W9a/FTTjkl55xzThYvXpzjjz++9Hp9x/E5c+akpqYm3/rWt7Lffvulc+fO+d//+39vtE0/99xzueiii3LEEUfky1/+cpo0aVLvubi+8/ymNGvWLEOGDMmDDz6Y5N3z5Kc//elN9rgsx3musbb2XJUkBx98cD73uc+lU6dOG/2E8/3e+2z22muvHHHEEVm4cGFGjhyZXr16ZZ999smhhx5a5yHtm9uPk+zw1/hbcw/5j8eSrblOPuaYY9K/f/907Ngxffv2zciRI/Of//mfWb9+fZ1lL126NBdeeGH69OmTb3zjG2nevHnpHiF593q2oqIiLVq02C7H1G2lp8Q/2G+//fK1r30tybuJ4QMPPJBLL700l156aenbo7lz52bGjBlZunRpVq9enfXr19fZMF544YXSTww25ze/+U1WrVqViRMnpkuXLg1u57Jly7Ju3bo6zxJo2rRpevXqlcWLFzd4eR+E93bW9evXZ+3atTnwwAMzZsyY/PWvf80dd9yRF198MStXrixdeK1YsaLBJ/8lS5akV69edW6m991336xduzZLly7NXnvtlSSl/76nXbt2qa2t3cY1LL/Kyso667L77rvn+eef3+z0Ddku9tlnn9L/L126NOvWrSuFCcm7NyHdunXbYvsGDx6c3/3ud6UEf86cOaXufa+//npeeeWVTJkyJddff31pnvXr1290cd29e/fS/zdr1ixt2rTZYR5I1tDPYEu2tN298MIL2bBhQ775zW/WmWbt2rWbvDnYEXTp0iV9+/bNgw8+mIEDB+bVV1/Nk08+mXHjxmXOnDlbtS++Z/HixXnnnXe2+ByK+o69ybu9Vd5/cmzXrl1pW3rrrbfy2muvbfQMln333XejG/Zy2tqbycWL/z/2zjysqmp9/B+mwyTDQWYQkAAnFERucv1apJSaXrWMtFQcL1pidUtv2i0TyuFqZVbiFcccSM2BB8RSLjij1jVnQAWVZBBQ4QAOeBjO7w+es38cDigoCOj6PA/PA2uvvfbai7XWu/a73vW+2VhZWWks8Ozs7JDL5WRnZ0vHIR70jlC9m7p161ZSUlJQKBRUVVWhVCq1zi435bir+f9UW2Sp+/aVK1dIS0ur04lvXl6exrzTHHTo0IG7d++ybds2OnXqVK/J65PAwMAAR0dH6W+5XE5FRQV37tyhXbt2XLlyhYyMDGJjY6U8KpVKMv99XCtIb29vysvLuXTpEiqVCnNzc+zt7bG0tCQvLw+FQkFKSgqGhoZ1/l8CAgL45ZdfmD59Oj4+Pvj6+uLv74+BgYGUpyHytWYeIyMjDA0NNfqjehFd876EhAT27dvHjRs3UCqVVFZWSseHoHrhHhUVxcGDB+nevTu9e/eWrPjqQj3nLFiwQCO9drmgKS9bEj8/P06cOMGvv/7KiBEj6szzoHGdk5ODq6srxsbGUp5OnTqho6NDdnY29vb2dO3ald27dwPVljSvvvoqSqWSlJQUzM3NuXXrVrNbSqjX4kqlksTERPLz8yUfMA2ZxzMzM3F1dX3gcYbCwkK+/PJLgoODGTZsmJTeXLK4f//+fPzxxygUCo4cOcInn3xSZ77HlXOPQ2MUn+7u7g3KV3OsW1hYAGiN9Zrj/EHjuLWv8R/0Damm5lzS0HXy+fPniYmJIScnh7t370rfUQqFAisrK6C6f86ZM4fevXtLRyoeRlPPqY+LUErUwtDQEHt7e+lvd3d3xo8fT2JiIm+99RaXLl1i6dKlBAcHM378eExNTTlx4oTGedSG0KlTJ86cOUNycrJkRvm0ox6senp6yOVy9PX1JVPC7t27M336dCwsLCgtLeXzzz/X8BDeFNQ0L6ttfqmjo9Mku1ANxdjYmLt372ql37lzR0PQ1GUm2lT1rEtD31j+7//+j02bNnHp0iX09fXJycmRlBJqIRoaGvpQR5y1z0rr6OhoCeGWoin/Bw/qdyqVCh0dHRYuXKjVHq3Z0Vz//v2Jiori9u3bHDhwgHbt2uHv78+RI0fqvedBpp710dC5tzX2JQcHB2nRX99xqIdRs80e9o6RkZEUFxczfvx4bGxsMDAw4IsvvtCaU5uyrWr2bXVda/btnj171nn8QL1IbU7kcjmzZs0iIiKCL7/8ks8++0za5amrL1ZWVjZbXWpbn6mfr273qqoqgoODpWMUNWmK8+K2trbY2NhIO5NdunQBqhUD7u7upKSkkJKSQufOnev0YWFtbc3SpUs5f/48Z8+eZcOGDWzfvp358+dL5t4Nka91zat1PU9939GjR1m/fr3kG8XExIQ9e/bwv//9T8o7cuRIXnjhBU6dOsWZM2fYtm0boaGhkilyfWXPmjVL68hS7fo1hbxsCvr27UtAQADLli2T+kptHndcq/1L5eXlceXKFbp164ZSqeTIkSOYm5tjZ2fX7NYiNdfikyZNIiIigu3bt2v4L3pczMzMsLGxITk5mf79+0tzQnPJYkdHRzp27Mh3332HpaUlXl5eFBQUaORpaTnXGFlV1/GOuqhLNtSuf835obHjuHbZtZ+pvvYk1vgP+oZU17/mXNKQdfKNGzdYuHAhQUFBjBo1inbt2nH16lW+++47DZmur6+Pj48Pp06d4saNG1qK1do0x5z6uIjjGw1AV1cXpVIJwMWLF7GysiI4OBgPDw8cHBy4ceOGRv6OHTs+1EzI3d2dTz/9lPj4eK3zZLVRD96aE46dnR36+vpcvHhRSquqqiI9Pb3VemFWD1YbGxvpnXJzcyktLWX06NF07doVJycnLW1mXe+vTq+d5uTkRHp6ukb6hQsX0NfXx87Orjle65FwdHSUtPE1uXr1qsYuWmN51H5hb2+Pnp6eRliq+/fvS6az9SGXy/H29ubw4cMcOXIELy8vqZ0tLS2Ry+Xk5+djb2+v9fO0Ul9/fRhubm6oVCoUCoVWW6k14a2RgIAADAwMOHToEPv37+fFF19EX1+/0WPRyckJAwODek1QGzL3PgwTExPkcrnG+FDXqznnzXbt2uHj48PevXvrDIN8584doNoqp7CwUGOhmp+fT1FRUaPqd+HCBQYNGoSfnx8dOnSQ/Do8LnXNuQ2hY8eOZGdnY21trdW3a+7YNidWVlaEh4dz//59vvzyS0pLS4HqD/379+9rKInVx04eh0dtK3d3d3JycuqcM5vKl4Har0Rt3wBqE+fU1NQH7oTLZDL8/PyYMGECCxcuJCsrS2tMNTUXLlzAw8ODQYMG4e7ujr29Pfn5+Vr5HBwcGDx4MJ988gn9+/dn3759QN3zsrOzMwYGBty4cUOrrR+2oG9J+vbtywcffMCOHTs0HKg2BCcnJ65du8a9e/ektIsXL2qY7Kv9SuzcuRM7OzssLCzo2rUrFy9e5OzZs0/Un4Sa4OBgYmNjKSwsbNA87ubmxp9//vlA6wEDAwNmzZpFu3btmDdvnjQPN0QWP6qc79+/PykpKfTr16/O600h5x6Hhsqq5qa+cdxW1vg1qfkNWZuGrJMvX75MRUUFEyZMwMvLC0dHx3rl+bRp0+jcuTMREREPjerSHHPq4yKUErUoLy9HoVCgUCjIzs5m7dq1lJWVSTG6HRwcKCws5PDhw+Tn55OQkKB11uv111/n2LFjkrfZrKws4uPjtc5Re3h48NlnnxEfH8+OHTvqrZONjQ06OjqcPHmSkpISysrKMDIyYsCAAURHR3Py5EnJGY5CodA4g93asba2xsDAgD179pCfn8/JkyfZunWrRp663l+dnpGRQUFBASUlJVRVVTFw4ECKiopYvXo12dnZnDx5kujoaAYNGtRqdjoABgwYQH5+PmvXriUzM5Pc3Fzi4+NJTk7WMCNsLI/aL4yMjOjXrx/R0dGcO3eO7OxsVqxYQVVV1UN3tV944QWOHj1KcnKyRhQBqNayxsbGEh8fT25uLteuXePgwYPExMQ88ju2durrrw/D0dGRvn37snz5co4fP05+fj6XL18mLi6O3377rZlr/ejIZDL69u3Ltm3byM/PlzTojR2LxsbGvPrqq2zevJn9+/eTl5dHRkYGCQkJQMPm3oYwbNgwdu3axZEjR8jNzWXr1q2kpaUxdOjQx2uIhzB58mRUKhWzZ8/m2LFj5ObmkpOTQ0JCghQzvHv37lIc8cuXL3P58mW+//57Onbs2CizYQcHBw4fPkx2djYZGRl89913TRJ2zMbGhqysLHJzcykpKWmwNdvAgQO5e/cuS5cuJT09nfz8fM6ePUtUVJTGx1FzI5fLmTt3LhUVFXzxxReUlJTg6emJoaEhP/30E3l5eRw/frxJ/DLZ2Nhw4cIFCgsLG2VW/cYbb5CcnMzWrVu5du0aOTk5HD9+XIri0RR069aN9PR00tPTNT4wu3btytGjRyVv/nVx4MABkpKSuHbtGgUFBRw4cAA9Pb1HOobaGBwcHLh69SqnTp3i+vXrbN++XcP/iVKpZPXq1aSkpFBQUEB6errGR2pd87KxsTFDhw5l48aN7Nu3j7y8PDIzM0lISGi1fnzU/PWvf+XDDz8kJiaGzZs3N/i+F154AUNDQ5YtW8a1a9dITU1l5cqVPP/88xqbBV27duXw4cNS/7C1tcXc3Jzff/+92f1J1EW3bt1wdnZm586dwMPn8b59+2JhYcFXX31FWloa+fn5nDhxQmvDUCaTMWvWLExMTCTFRENk8aPK+cDAQFavXs2QIUPqvN5Ucu5xaIisai4eNo5b+xr/Yd+QdfGwdbKDgwMqlb9ooLEAACAASURBVIrdu3dTUFDAkSNHpONVtdHV1SUsLAwvLy/Cw8MfqJhojjn1cRHHN2px7tw5yXutsbExjo6OfPjhh9LE7O/vz7Bhw/jxxx9RKpX4+PgwatQoVq9eLZXh5+fHP//5T7Zt20ZcXBzGxsZ4eXlpeeGG/6+YmDdvHlC9IKmNlZUVb775Jlu2bCEqKooXX3yRsLAwxowZAyDFb+7YsSOffvppm4i8ocbc3JywsDA2b97M3r17cXFxYdy4cRpnPOt7/6FDhxIZGclHH32EUqmUQoJ+8sknbNq0iY8//hhTU1P+7//+j7fffrsF31IbOzs7IiIi2Lp1K/Pnz0epVOLk5MSHH35Iz549H6vsR+0X48aNY9WqVSxevBgjIyOGDBlCcXGxxlnhuujduzerV6/m7t279OnTR+NaUFAQhoaG7Nq1i82bNyOTyXB2dm7RcGrNTX39tSFMmzaNnTt3smnTJm7dukW7du3w8PBotT4l1PTv35+EhAQ6deokCSwrK6tGj8XRo0fTrl07duzYwa1bt7C0tJRC2TVk7m0Ir776Kvfu3SM6OhqFQoGjoyMzZsxo9ugmdnZ2LFq0iJiYGKKjoyksLMTMzAxXV1fpDKqOjg4ff/wxa9eulcJtde/enUmTJjXqyMu7777LypUrmTVrltQfm+K88csvv0xqaiqzZ8+mrKxMIyTog7CysuLLL7/kp59+YsGCBSiVSqytrfHx8Xno/NLUWFpaMnfuXL788ksiIiL4/PPPef/999m0aRP79++na9eujBo1imXLlj3Wc0aOHMmqVat47733KC8vb/COtq+vL7Nnz2bHjh3s2rVL+uB/6aWXHqs+NenWrRsVFRW0b99e40O0c+fOKJVKjI2N6z0vbmJiQmxsLBs3bqSyshJnZ2dmzpz5QEd3TcErr7wiedFXqVT07t2boUOHSo4DdXV1uXPnDsuXL6eoqAgzMzP8/PwkPyb1zcujRo3CwsKCXbt2sXr1aoyNjXFzc9Nwqthaef7555kxYwZLliyhsrKSsWPHPvQeQ0NDPv30U3788Uc++eQTjZCgNVErqGorrQ4ePNgilhIAQ4cOZfny5QwfPvyh87iRkRHh4eFs2LCBRYsWUVFRgaOjI+PHj9cqVyaTMXv2bP79738zb948Pvvss4fK4keV87q6ug88htVUcu5xaIisai4aMo5b8xr/Qd+QtY/qqHnYOtnV1ZUJEyYQGxvLli1b6NSpEyEhISxdurTO8nR1dZk+fTrLli0jIiKCuXPn1hlRq7nm1MdBR/UkD9ILBII2Q3l5OdOmTWPYsGHNvossEAgEAoFAIBAInk2EpYRAIACq/Vnk5OTg4eHBvXv3iI2NpaysTMv6QSAQCAQCgUAgEAiaCqGUEAgEEuozbXp6eri5uREREdGiMdkFAoFAIBAIBALB0404viEQCAQCgUAgEAgEAoGgRRDRNwQCgUAgEAgEAoFAIBC0CM+sUiIlJYWRI0c2iUdyQcO4ffs2oaGh5OXltXRVtFiyZAm7du1q6Wo0GNGWT4awsDDi4uJauhoAJCYm8u677zJq1KhGx6Z/0rTm/rlx40bWrl3b0tVoFsLDw1mzZk2zlH3gwAHJ67ZAoEaM9eajNcmftojom62HhsiPuLi4x47eIGjbPBM+JcLDw+nQoQOTJ09u9mcVFBQwffp0rXR/f38+/vjjBpUxcuRIPvroIwICApq6ei1KTEwMPXv2lEKQrVu3josXL5KVlYWlpSWRkZFa9xw9epSYmBiuX7+Oubk5gwYNYtiwYdL1yMhIDh48qHWfoaEhGzdulP5OTU1l/fr1ZGdnI5fLGTZsmEaI1uDgYObOnUtQUBAmJiZN+drNQnO0JcCePXvYu3cvBQUFWFtbM2LECAIDA6XriYmJHDp0iKysLFQqFR07dmTUqFF07txZytOW2jIyMpLS0lJmz55d5/WFCxe2itjXt2/fZs2aNYwbN46AgACMjY1bukoPpKX6Z3h4uEacbTXOzs4sWbIEgOHDh/Pee+8xZMgQ7OzsmvK1G0TtOcvMzAxPT09CQkJwcnJ64vVpKH369HnscMVNjUKhICYmhpMnT3Lr1i0pbN2gQYPw8/Nr6eo9E4ix/mD++9//smHDBtatW4e+fvWSu6KiggkTJmBnZ8c333wj5c3Ly+P9999nzpw5dO/evUmef+DAAdasWaOxHqrN2bNnWbhwIREREXh5eUnpSqWSf/7zn3h7exMaGtok9XmSiL75ZBg5cuQDrwcGBhIaGtrq5MfTivo7dOHChTz33HMtXZ1G8UwoJVqCf/3rXxpx7590LHaAyspKdHV1GxXfvrm4f/8++/btY9asWVKaSqUiMDCQa9eucfbsWa17Tp06xffff8/EiRPx9fUlJyeHqKgoZDKZFL934sSJjBkzRuO+OXPm0KVLF+nvgoICFi5cSL9+/Xjvvfe4cOECa9aswdzcXFL8uLi4YGdnx6FDh6SyWyvN1ZYJCQlER0czdepUPD09ycjIICoqClNTU/z9/YFq5U6fPn3o1KkThoaGxMfHM3/+fBYvXoyDgwPQttryYTwonviT5ObNm1RWVtKrVy/kcnlLV+eBtGT/nDlzJhUVFVK55eXlzJw5k7/+9a9Smrm5OT169CAhIaHFdv67d+/Oe++9B0BhYSGbNm3i66+/5ttvv32k8tRzfXNRUVGBTCZDJpM12zMaS0FBAXPmzMHY2Ji3334bNzc3qqqqOH/+PKtWreI///lPS1fxsamoqJA+ZJ/EfY1FjPWH061bN+7fv09GRoakvE9PT8fExITr169TUlIiyZnz589jYGBAp06dnmgde/TowSuvvEJkZCSLFy+WFPE//fQTKpWqTVpIib755Fi5cqX0+x9//EFUVJRGmlp2tCb5IWidPPVKicjISFJTU0lNTWXv3r0ALFu2TLr+559/snnzZq5du4azszNTpkzB3d1dun7x4kV++uknLl++LE06Y8aMeegOsJmZGZaWllrpdVlBhIWFMXDgQIYNGyaZLqm1qTY2NkRGRvLzzz/z22+/aWjVa2vA1XmGDh3Kjh07KCgoYP369VRVVbFx40b+97//oVQq6dixI+PGjXuiGrRTp04BaAjbSZMmAdUmW3UJiEOHDtGrVy8GDhwIgJ2dHa+99hqxsbEMHDgQHR0dTExMNP4XFy5cID8/X8NaJSEhAblcLj3P2dmZjIwMdu3apfF/8Pf3Jzk5udV/SDdXWx46dIigoCD69u0r5bl8+TKxsbGSsH3//fc1yg0NDeV///sfp0+flpQS0Hba8mHUHJtQPX6nTJnC2bNnOXXqFBYWFowcOZIXX3xRuqewsJANGzZw5swZALy8vJgwYYJG+9Tm5s2brFu3jnPnzgHVi8SJEyfSvn17Dhw4wPLlywGkfr1s2TJsbW2b5Z0fl5bsn+3atdMo9/Dhw9y/f59+/fpppPv7+7N58+YWWwwaGBhI8sHS0pIhQ4awaNEilEolMpmM6Ohofv/9d27evImlpSV//etfGTlypLSoq2+uh2oFxbp16zh06BAA/fv3Z8yYMZLS4tChQ/z666/k5OQgk8no2rUrEyZMwMrKCqg+2hgREcHs2bPZtm0bmZmZzJw5k9LSUg15k5eXx4YNG0hPT6esrAxHR0dGjhxJr169pPcMCwujf//+3Lp1i+TkZIyNjRk8eLDWzuOjoD6m8u9//xsjIyMp3dnZmRdeeAF48Liq2Y4jRoxgy5YtFBcX4+3tzTvvvCN9KFZVVbFp0yb2798PVO/6lZeXk5OTQ3h4OFC3NWZtKyyVSkVcXByJiYkUFhZib2/P8OHDpblDvbv1/vvvk5SUxKVLlwgJCWHAgAHs3LmTpKQkiouLcXBw4K233uIvf/nLA+97EnOvGOsPx9HREblczvnz5yWlREpKCt7e3ty4cYOUlBTpYzUlJQUvLy+Nj7fy8nJWrlxZ7/iJj4/nwIED5OfnY2JiQs+ePQkJCcHU1JSUlBRJdqh3s4ODg+vc2R47dixnzpwhOjqaSZMmSWvmiIgI9PT0+PHHH0lOTubu3bu4ubkREhKi8T4RERGsXr1aGjctvVsr+uaTo+a3jqmpqVYa1G2xExsbS3x8PGVlZfTu3bvONc3+/fuJi4uTrFJeeeUVBg8e3KxK+LbA6dOn2blzJ1lZWQB4eHgwfvx4nJ2dpXXiJ598AkDXrl0lWdXaeer/qxMnTsTLy4uXXnqJlStXsnLlSqytraXrP/30E6NHj2bRokWYmZnxww8/oA5Icu3aNebNm4e/vz9fffUVM2fOJDMzs1l3YBYuXAjA1KlTWblypfR3QykoKODIkSN8+OGHfPXVV+jr67Nw4UIKCwuZPXs2ixcvpkuXLnzxxRcUFRU1xyvUSVpaGu7u7o2y2igvL9eyMJHJZNy6dYsbN27UeU9SUhIdOnTQEETp6en06NFDI5+Pjw9XrlzR0GZ7eHiQkZGBUqlscB1bguZqy/Lyci1NtkwmIyMjQ6OdalJRUUF5ebkkiNS0lbZ8FLZv3y7NCX369OE///kPN2/eBKp3ZyIiIjAwMCA8PJx58+Yhl8v58ssvuX//fp3lVVVVsXjxYoqLi5k7dy5z586lqKiIr776CpVKRZ8+ffjXv/4FwIIFC7TmsNZGa+qfSUlJ+Pr6arWXh4cHhYWFreKs8b179zh69CguLi7S+xkaGvLuu+/y7bffMnnyZJKTk9m5c6fGfbXnenX7HTlyBJVKxbx58wgNDSUxMZFffvlFuq+iooI333yTr776itmzZ1NaWsp3332nVa/o6Gjeeustli5diqenp9b1srIyfH19mTNnDl999RW9e/fm66+/JicnRyPf7t27cXFxYdGiRQwfPpxNmzZx6dKlx2qz27dvc/r0aQYOHKihkFBjamr60HGlpqCggKNHjzJz5kw+++wzMjMz2bJli3R9165dJCUlERoayrx586iqquLIkSONrvOWLVvYt28fkydP5ttvv+X1119n1apVnDx5UiPf5s2bGThwIN9++y1/+ctf+OWXX9i1axdjxozh66+/5vnnn+frr78mMzPzgfc9CcRYbxjdunUjJSVF+jslJYVu3brRtWtXjfTU1FS6deumce/Dxo+Ojg4TJkzgm2++4YMPPiAjI0PyVdCpUycmTJiAoaGhtP6tTyEok8l47733+O9//8uJEydYvnw5w4YNw8vLi02bNnH06FHeffddFi1aRIcOHZg/f/4TXUM2FtE3WzdHjx5ly5YtjBw5kkWLFuHo6Mju3bs18iQmJrJ582ZGjRrFt99+S0hICLGxsSQkJLRQrVsPZWVlDB48mAULFhAeHo6xsTGLFi2ioqKCBQsWANUW+ytXrmTmzJktXNuG89QrJUxMTNDX18fQ0BBLS0ssLS01NGyjRo3C29sbJycn3njjDXJycigsLASqtal9+vRh6NChODg44OnpSWhoKL/99hvFxcUPfO7cuXMJCQmRftLS0hpUX7WW2dTUFEtLy0abj1dUVDB9+nTc3d1xcXHhwoULZGZmMmPGDDw8PLC3t+ett97C1tZW2kl7Ety4caPRZue+vr6cOHGCM2fOUFVVRW5uLvHx8UD1WeLa3L17l2PHjhEUFKSRrlAotLS2FhYWVFZWUlpaKqXJ5XIqKyul/39rpbna0sfHh/3795ORkYFKpeLy5cskJSVptVNNtmzZgpGRkbRDoKattOWj8OKLL/Liiy9ib2/PqFGj0NPTk86PJicno1KpmDZtGq6urjg5OTFlyhTKysr4448/6izv/Pnz/Pnnn7z//vs899xzPPfcc7z//vtcvXqVc+fOIZPJMDMzA6rnh9pzWGujtfTP3NxcUlNTteYDQKpffcrN5ub06dOSbBg/fjypqakaVkjBwcF07twZW1tb/Pz8eP3110lOTtYoo/Zcr6enB1S/28SJE3FycqJPnz4MGzZMakuotpzw8/PDzs4ODw8P/v73v5OWlsatW7c0yn/zzTfx8fHBzs6uTjnk5ubGgAEDcHFxwd7enhEjRuDu7s7x48c18vXo0YNBgwZhb2/Pq6++ir29vWS58Kjk5eWhUqlwdnauN8/DxpWaqqoqwsLCcHV1xcvLi5dfflnj+i+//MLw4cPp06cPTk5OTJgwoU4ryAdRVlZGfHw877zzDr6+vtja2tK3b1+CgoIkC041gwYNIiAgAFtbW9q3b8+uXbsYOnQoffv2xdHRkVGjRtGlSxctB4i173sSiLHeMLy9vbl06RLl5eUolUouXbqkpZTIycmhqKgIb29vjXsfNn6GDBmCt7c3tra2dO3albFjx3Ls2DGqqqrQ19eXLEnV69+6lHhqPDw8eO211/jqq68wMTHhzTffpKysjISEBMaMGYOfn59kUWxpaanVd1sTom+2bn755RcCAwN55ZVXcHR0ZMSIEXh4eGjk2bFjB2PHjpXmNX9/f1577bVW3e+eFAEBAQQEBODg4ICrqyvTpk2joKCAjIwMSV6rLfZrW+60Zp764xsPw9XVVfpdbb5aXFxM+/btuXLlCnl5eRw9elTrvvz8fCwsLOot9/3338fFxUWr7ObGyspKY8F05coVlEqllpPP8vJy8vPzn0id1M9r7HmyoKAg8vLyWLx4MZWVlZLp4rZt2+rUfh86dAiVSqVhSt8Y1PVr7bv7zdWWwcHBKBQK5syZg0qlwsLCgsDAQOLi4ups719++YXExETmzJmjdZyprbTlo1BzXOvp6WFubi5F8bly5QoFBQWMGzdO4x6lUlnveMvOzsbKykrDdNHOzg65XE52draWlU9rp7X0z6SkJORyeZ0OD1u6f3bp0oWpU6cC1bv+CQkJzJ8/n/nz52Ntbc3x48fZvXs3eXl5lJWVUVVVRVVVlUYZted6NZ6enhrt4eXlxdatW7l79y4mJiZcuXKF7du3k5mZye3btyWrgZs3b2p8zD7M5LqsrIzt27fzxx9/oFAoJKupmuMDNGUsVC/EH6bUfxg1LR3qo6HjytraWmP+ksvl0ni+e/cuRUVFGs7/dHV18fDw0FLiPKwu5eXl0g6WmsrKSmxsbDTSara7+vm1fQx07txZMk+v674nhRjrDcPb25vy8nIuXbqESqXC3Nwce3t7LC0tycvLQ6FQkJKSgqGhodaH2cPGz/nz54mJiSEnJ4e7d+9SVVVFRUUFCoXikdadwcHB7Nixg+HDh6Ovr09OTg6VlZUafVBXVxdPT0+ys7MbXf6TQvTN1k1OTg79+/fXSPP09JSsRkpKSrh16xYrV65k1apVUp6qqqoGzf9PO3l5eWzdupWMjAxKSkqkdrl58+YT+95sDp55pYR6d6km6g6vUqno378/f/vb37TyPOyf3r59e8njb010dHS0BlR9Jl810dXVbdB9tbXgVVVVWFhY8MUXX2jlfZIe/M3MzLh9+3aj7tHR0WHs2LGMHj0ahUKBubm5tENQl7fipKQkevfuraUVtLS01LKsKC4uRk9PT9qBBqT6tRbnhvXRXG0pk8mYNm0aU6ZMobi4GLlcTmJiIsbGxlptsnv3brZu3cq//vUvrUUUtJ22fBRqO5DT0dGRPhhVKhVubm784x//0LrvUbTVrcFJbWNpDf2zoqKCgwcPEhQUVOcc39L909DQUEM+uLu7M378eBITE/Hz82Pp0qUEBwczfvx4TE1NOXHihJb3/AfteNZHWVkZ8+fPp3v37kyfPh0LCwtKS0v5/PPPteTJw6LObNy4UbL4cHBwwNDQkGXLlmmVU7v965KBjcXBwQEdHR2ys7N5/vnnG31/zXH1oPHcmPJqv1NlZaX0u/rarFmztEy4a7fPo0b7aYkoQWKsNwxbW1tsbGwkqwi1I24jIyPc3d1JSUkhJSWFzp07a/XHB42fGzdusHDhQoKCghg1ahTt2rXj6tWrfPfddw1aV9aF+nl1tWV91CWnavb/lkD0zbaNeg4ODQ194o5f2wKLFi3CysqK0NBQrKys0NPT46OPPnrkcd9aaL02wE2Ivr5+oxcZAB07diQ7Oxt7e3utn0f1Imtubq5xDk+hUGh9MOvp6WnV19zcnOLiYo2FT+0zpXXh7u5OcXExOjo6Wu/wIEuPpsbNzU3rrHFD0dXVxcrKCn19fZKTk/Hy8tKaxDMyMvjzzz/rNJHz9PTUMhc+e/Ys7u7uGguArKysencfWxPN3Zb6+vq0b98eXV1dkpOT8fPz0zguEB8fz9atW5k9e7ZGKNCatJW2bGo6duxIXl4eZmZmWuOtPqWEs7MzhYWFFBQUSGn5+fkUFRU90Dy9tdLS/RPg999/p7S0VGsnRk1WVhZ6enpau/otia6uLkqlkosXL2JlZUVwcDAeHh44ODg0yrw3PT1dQ06kp6cjl8sxMTEhNzeX0tJSRo8eTdeuXXFycnpkq4ULFy4QGBhIQEAArq6uWFlZPTHru3bt2uHj48PevXspKyvTun7nzp0mGVcmJibI5XKNM/wqlYqMjAyNfObm5lpy/M8//5R+d3Z2xsDAgBs3bmjNC7UtJep6/sWLFzXSL1y40CrmBjHWG47ar4Tan0TN9PPnz9fpT+JhXL58WQov6uXlhaOjo5afh0dd/6qxs7NDX19fow9WVVWRnp4u9UH1/63msxuyPm1ORN9s3Tg5OZGenq6RVvNvS0tL5HI5+fn5dX6DPcuUlpaSk5PD66+/To8ePXB2dubevXuSIlD9XfM4476leCaUEjY2NmRkZFBQUCCZuTSE4cOHk5GRwcqVK7l69Sp5eXn88ccfGqFuGku3bt3Yu3cvly9f5urVqyxfvlzLsY6trS3nzp1DoVBImtSuXbty+/ZtYmJiyMvLY9++ffz2228PfV737t3p1KkTixcv5tSpUxQUFHDp0iV+/vnnBvu5aAp8fX3Jzs7WOHOXl5dHZmYmRUVFVFRUkJmZSWZmpqTpKykpISEhgezsbDIzM1m3bh3Hjh1jwoQJWuUnJibi4OBQp1AfMGAAhYWF/Pjjj2RnZ5OUlMSBAwcYOnSoRr60tDR8fHya9sWbgeZqy9zcXA4dOsT169fJyMhg6dKlZGVl8fbbb0t54uLiiI6O5p133sHR0VFSqt29e1ejjm2lLaHa0aC6vdQ/NT9kGsMLL7yAhYUFixcvJjU1lYKCAlJTU9mwYQPXr1+v857u3bvj6urKDz/8wOXLl7l8+TLff/89HTt21Dpf3BZoyf6pJikpCW9v73rjv6elpdGlS5cW2V2GatNi9djJzs5m7dq1lJWV0atXLxwcHCgsLOTw4cPk5+eTkJCg5U/iQRQVFfHjjz+Sm5vL8ePHiYuLY8iQIUD1UQUDAwP27NlDfn4+J0+eZOvWrY/0Dg4ODvz+++9cuXKFa9eu8cMPPzxRM+TJkyejUqmYPXs2x44dIzc3l5ycHBISEpg5c2aTjatXX32VuLg4jh8/Tm5uLj/++KOWAsLb25tTp05x4sQJcnNzWb9+veT8FqqtEocOHcrGjRvZt2+fNB4SEhJITEx84POHDRvGrl27OHLkCLm5uWzdupW0tDQt+dUSiLHecLp160Z6ejrp6eka65SuXbty9OhRKfJLY3BwcEClUrF7927J8W1tZ4E2NjaUl5dz9uxZSkpK6nW4XB9GRkYMGDCA6OhoTp48SXZ2NqtWrUKhUEhRKuzt7Wnfvj3btm0jNzeXM2fOaDnmfdKIvtm6GTx4MAcPHiQxMZHr168TExOjpewdOXKkFKEjNzeXa9eucfDgQWJiYlqo1q0DU1NTzMzMSEpKIi8vj9TUVFatWiVZ41hYWCCTyThz5kyd6/PWzDNxfGPo0KFERkby0UcfoVQqNUKCPghXV1ciIiLYsmUL4eHhVFVVYWtr+0jmomrGjRvHihUrCA8Px9LSkjFjxmhpc0NCQtiwYQPvvvsuVlZWREZG4uzszN///ndiYmKIiYmhV69evP7662zevPmBz9PR0eGTTz5hy5YtREVFUVxcjKWlJZ06dXpk3wuPgouLCx4eHhphIlesWCE5CAT4+OOPAc1whwcPHpTMlr28vAgPD9c6LnDv3j2Sk5MJDg6u89m2trZ88sknrF+/XgoPOnHiRI1woEqlkt9//51PP/206V66mWiutqyqqpImfz09Pbp168a8efM0zmTv3buXyspKli5dqlGnwMBAKZxtW2pLqF4YqNtLTe/evZkxY0ajyzI0NCQiIoKffvqJJUuWcPfuXeRyOd26ddOKUKJGR0eHjz/+mLVr1xIREQFUKyomTZrUJo9vtGT/hOrd8PPnz/PBBx/UW8fk5OQ6w+I9Kc6dO8eUKVOA6g9WR0dHPvzwQ+ljZdiwYfz4448olUp8fHwYNWoUq1evblDZffv2paqqin/961/o6OhoHEE0NzcnLCyMzZs3s3fvXlxcXBg3bpyWr4OGMH78eFasWMHcuXMxNTVl8ODBlJeXN7qcR8XOzo5FixYRExNDdHQ0hYWFmJmZ4erqytSpU5tsXA0dOhSFQsGKFSuAake3ffv21ZDb/fr1488//5Qicw0cOJDnn39e44No1KhRWFhYsGvXLlavXo2xsTFubm4MHz78gc9/9dVXuXfvHtHR0SgUChwdHZkxYwZubm4NfofmQoz1htOtWzcqKiq0jvZ27twZpVKJsbGxRjj6huDq6sqECROIjY1ly5YtdOrUiZCQEA353KlTJ1555RW+++47SktL6w0J+iDGjBkDwH/+8x/u3LlDx44d+fTTTyVHjfr6+vzjH/9g9erV/POf/8TNzY23336bf//73416TlMi+mbrpk+fPuTn57Nlyxbu37+Pv78/Q4YM4eDBg1KeoKAgDA0N2bVrF5s3b0Ymk+Hs7NzmQ80/Lrq6unz44YesW7eOGTNmYG9vT0hICN988w1QbW0/ceJEtm/fzrZt2+jSpUubCQmqoxIeQwRPiNOnT7Nu3Tq+/fbbVhc9YM+ePZw4cYLPPvuspavSIERbx/7NHQAAIABJREFUClozrbl/njx5ko0bN/L111836ty0QKBmzZo1ZGVltZmFXnMixrqgtSL6pkDQttALF1JV8ISwt7dHpVIhl8vr3TVuKTIzMxkwYICG48vWjGhLQWumNffPq1ev8tJLL2k5HBQIGsqpU6coKSnhpZdeaumqtDhirAtaK6JvCgRtC2EpIRAIBAKBQNBAhKWEQCAQCARNi1BKCAQCgUAgEAgEAoFAIGgRWtchK4FAIBAIBAKBQCAQCATPDEIp8QBu375NaGgoeXl5LV0VLZYsWcKuXbtauhoCgQCIjIxsUU/jgqcXIYealtbcnhs3bmTt2rUtXQ1BKyI8PJw1a9Y8MM+MGTP4+eefn1CNBM8qrXnubIuy6MCBA4SEhGikJSYm8u677zJq1ChpTNeV9rTyTIQEfVRiYmLo2bOnFL5p3bp1XLx4kaysLCwtLYmMjNS65+jRo8TExHD9+nXMzc0ZNGgQw4YN08hz5MgRYmNjuX79OsbGxnTv3p1x48ZhaWkJVHfU5cuXa5W9adMmZDIZAMHBwcydO5egoCBMTEya+tUFAkEdREZGaoSsUhMeHo6rq2sL1EjwtNNScigrK4uff/6Zq1evUlBQUGcowbYoh5qrPffs2cPevXspKCjA2tqaESNGEBgYKF0PDw/XCEeoxtnZmSVLlgAwfPhw3nvvPYYMGYKdnV1TvragFVFTjujp6WFqakqHDh3o3bs3L7/8Mvr6/39pPnPmzMeOznDgwAHWrFkjhbpsasLDw+nQoQOTJ09ulvIFrYOWmjvb4jfRw0K9BgYGEhoaSs+ePaW027dvs2bNGsaNG0dAQADGxsZ1pj0pIiMjsbGxeaJha4VSoh7u37/Pvn37mDVrlpSmUqkIDAzk2rVrnD17VuueU6dO8f333zNx4kR8fX3JyckhKioKmUwmxdW9cOECP/zwAyEhITz//PMoFArWrFnD999/z+effy6VZWhoyA8//KBRvnrwQXUMZjs7Ow4dOvTMx+wVCJ4k3bt357333tNIMzMze+DCsaKiQmOhKRA0hJaUQ/fv38fGxobevXuzZcuWOuvX1uRQc7VnQkIC0dHRTJ06FU9PTzIyMoiKisLU1BR/f3+g+uOyoqJCKre8vJyZM2fy17/+VUozNzenR48eJCQkaO2gCZ4u1HKkqqqKkpISzp8/z7Zt2zh8+DBz5szByMgIgHbt2rVwTQWClp07oe19E61cuVL6/Y8//iAqKkojTSaTST9qbt68SWVlJb169UIulwPV0exqpz3NiFVyPZw6dQqATp06SWmTJk0CIC4urs4BeOjQIXr16sXAgQMBsLOz47XXXiM2NpaBAweio6PDpUuXaN++PX/7298AsLW1ZdCgQXWabKp3rOrD39+f5OTkVjEABYJnBQMDA62xGRkZSWlpKbNnzwaqd46cnJwwNDTk4MGD2NrasnDhQrKzs9m4cSNpaWnIZDK8vb2ZMGHCQ8e64NmkJeWQh4cHHh4eQPUOWX20JTnUXO156NAhgoKC6Nu3r5Tn8uXLxMbGSgvr2h+Xhw8f5v79+/Tr108j3d/fn82bNwulxFNOTTliZWWFm5sbPXr0YNasWcTFxUm7k7WtEIqLi4mKiuLMmTNYWFgQHBz82HU5ffo0O3fuJCsrC6ge++PHj8fZ2VnKs337dvbt24dCocDU1BQfHx+mT59OZGQkqamppKamsnfvXgCWLVuGra3tY9dL0HpoyblTTVv6JqpZV3U42tr1r2nBVNMaZPr06QBMmzZNK23ZsmVUVVWxYcMG0tPTKSsrw9HRkZEjR9KrVy+p7LCwMPr378+tW7dITk7G2NiYwYMHS1Yqy5cvp6SkRFqzAlRVVREWFsaQIUOktUFNfvvtN7Zt28b169eRyWS4uLjw4YcfNun6VSgl6iEtLQ13d3d0dHQafE95eTkGBgYaaTKZjFu3bnHjxg1sbW3p3Lkzmzdv5sSJE/Tq1YvS0lKOHj2qYcIDoFQqmTZtGlVVVbi5uTFq1Cg6duyokcfDw4MdO3agVCo1tG0CgaDlOXz4MC+//DJffPEFKpWKoqIi5s6dS79+/QgJCaGyspLNmzezePFi5s2bh66ucPEj0KSl5VBDaEtyqLnas7y8XOvdZTIZGRkZ9VpJJSUl4evri7W1tUa6h4cHhYWF5OXlSWbSgmcDFxcXfH19+e233+o1mV6+fDk3btxgzpw5GBoasn79egoKCh7ruWVlZQwePBhXV1eUSiU7duxg0aJFfPvtt+jr63P8+HF27drFBx98gIuLC8XFxaSnpwMwceJErl+/jqOjI6NHjwaqLX4ETxctPXc+7d9Effr0wdLSkgULFrBgwQKsra0xMjLSSjM3N+fatWv4+vry1ltvIZPJOHr0KF9//TVff/01Tk5OUpm7d+9m5MiRDBs2jFOnTrFu3To6d+6Ml5cXL7/8Mp9//jlFRUWSBcbZs2dRKBS8+OKLWvVTKBQsXbqU0aNH07t3b8rKyqQ5oCkRq+B6uHHjRqNNZXx9fTlx4gRnzpyhqqqK3Nxc4uPjgep/KICXlxf/+Mc/+OGHHxg9ejR///vfUalUkhYMwNHRkXfffZePP/6YDz74AAMDA+bMmcP169c1nieXy6msrKSwsPAx31YgEDSU06dPExISIv0sWLCgzny2traMGzcOJycnnJ2dSUhIwNXVlbFjx+Ls7IyrqyvTp08nIyODK1euPOG3ELQFWlIONZS2JIeaqz19fHzYv38/GRkZqFQqLl++TFJSEpWVlZSWlmqVmZubS2pqKkFBQVrX1PW7ceNGY19P8BTg7OxMfn5+nddyc3M5deoUU6ZMoXPnznTs2JGwsDCUSuVjPTMgIICAgAAcHBxwdXVl2rRpFBQUkJGRAVSblVtaWtKjRw+sra157rnnpN1oExMT9PX1MTQ0xNLSEktLS6FgfwppybnzWfgmkslkmJmZAdVKPUtLS4yMjLTSdHV1cXNzY8CAAbi4uGBvb8+IESNwd3fn+PHjGmX26NGDQYMGYW9vz6uvvoq9vT3nzp0DqtcATk5OGj7S9u/fj7+/v6RUDAsLk5SjhYWFVFZWEhAQgK2tLS4uLgQFBTW5la+wlKiHurR3DyMoKIi8vDwWL15MZWWlZC6zbds2SbuYnZ3N2rVreeONN/Dx8aGoqIhNmzaxcuVKaUHo5eWFl5eXVG6nTp345z//ya+//iqZS8H/P0/1uAJJIBA0nC5dujB16lTpb5lMxubNm7Xyubu7a/x95coV0tLS6jTLzsvLk0zlBQI1LSmHGkpbkkPN1Z7BwcEoFArmzJmDSqXCwsKCwMBA4uLi6txZTEpKQi6X4+fnp3WtLbWnoOlRqVT17kbn5OSgo6OjIStsbGywsrJ6rGfm5eWxdetWMjIyKCkpoaqqCpVKxc2bN4FqpcUvv/zC9OnT8fHxwdfXF39/f61dcMHTS0vOneKbSJOysjK2b9/OH3/8gUKhoKKigvLyclxcXDTy1Xa+LpfLKS4ulv4OCgpi7969vPbaa9y+fZsTJ04wc+bMOp/p5uZG9+7dmTFjBj169KBHjx4EBAQ0uVWUUErUg5mZGbdv327UPTo6OowdO5bRo0ejUCgwNzeXtFJqT9oxMTF4eHhI53pcXV0xMjLi888/5+2336Z9+/Za5erq6vLcc89pheFR10+YygkETw5DQ8MGmVUbGhpq/K1SqejZsyfjxo3TymthYdFk9RM8PbQmOVQfbUkONVd7ymQypk2bxpQpUyguLkYul5OYmIixsbFWu1RUVHDw4EGCgoLqdI7bltpT0PRkZ2c/1B9DY0zoG8KiRYuwsrIiNDQUKysr9PT0+OijjyTHrNbW1ixdupTz589z9uxZNmzYwPbt25k/f77kkFPwdNMa5k41z/o30caNGyWLXQcHBwwNDVm2bJmGI2VAS77o6OigUqmkv1988UWio6O5cOECV69exdzcHB8fnzqfqaury2effUZ6ejpnzpxh3759/PTTT4SHh+Pm5tZk7yZsrOrBzc2NnJycR7pXV1cXKysr9PX1SU5OxsvLSxok9+/f1zJtU/9ds7PURKVS8eeff2qZyWRlZWFlZSWc5AkEbYCOHTuSnZ2NtbU19vb2Gj9PMsyToO3QmuRQfbQlOdRc7alGX1+f9u3bo6urS3JyMn5+flrt/Pvvv1NaWkr//v3rfE5WVhZ6enpau16Cp59r165x5swZAgIC6rzu5OSESqWSjlVA9dGKxzFXLy0tJScnh9dff50ePXrg7OzMvXv3qKys1Mgnk8nw8/NjwoQJLFy4kKysLC5evAhU9/uqqqpHroOg9dMa5k41z/o30YULFwgMDCQgIABXV1esrKzqPfL1INq1a8fzzz/Pvn372L9/P4GBgQ88eqWjo4OXlxdvvvkmCxcuRC6Xc/To0cd5FS2EpUQ9+Pr6Eh0dTWlpqXSmJy8vj7KyMoqKiqioqCAzMxOoPgOor69PSUkJx48fp2vXrlRUVLB//36OHTtGRESEVK6/vz9RUVEkJCRIZrPr16+nY8eOksOrbdu24enpiYODA/fu3eOXX37h2rVrhIaGatQxLS2tXq2WQCBoXQwcOJCkpCSWLl3K8OHDMTc3Jz8/n2PHjjFu3DihmBBo0ZJyqKKiguzsbKDaHFahUJCZmYmRkZGGpVBbkkPN1Z65ublkZGTg6enJnTt3iI+PJysri7CwMK06JCUl4e3tLe0U1iYtLY0uXbpoWVoJni7Ky8tRKBQaIUFjYmJwd3dn6NChdd7j6OiIr68vK1euZOrUqchkMtavX98gs3qVSiX1bTW6uro4OztjZmZGUlIS1tbWFBYWsnHjRo1d1gMHDlBZWYmnpydGRkYcPXoUPT09HBwcgOojJBkZGRQUFGBkZES7du2EX4mnjJacO8U3kSYODg78/vvv+Pv7o6+vz7Zt2x75yEpQUBALFiygsrKSGTNm1Jvv0qVLnDt3Dh8fHywtLbl69Sq3bt3SiNDTFAilRD24uLjg4eGhEV5mxYoVpKamSnk+/vhjQDP80cGDB9m4cSNQfQ4qPDxc4/zfSy+9xL1799izZw8bNmzAxMQEb29vxowZI+W5c+cOK1euRKFQYGJiQseOHYmIiNAoR6lU8vvvv/Ppp582XyMIBIImw8rKii+//JKffvqJBQsWoFQqsba2xsfHR5zNFdRJS8qhwsJCqWyA/Px8EhMT6dq1K+Hh4UDbk0PN1Z5VVVXEx8eTm5uLnp4e3bp1Y968eVpm+Pn5+Zw/f54PPvig3jomJyfXG3lB8PRw7tw5pkyZgq6uLqampnTo0IE333yTl19+uc5oLWqmTZtGVFQUERERmJubExwcTElJyUOfp1QqNcYzVJvkr1mzhg8//JB169YxY8YM7O3tCQkJ4ZtvvpHymZiYEBsby8aNG6msrMTZ2ZmZM2dK/Xvo0KFERkby0UcfoVQqRUjQp5CWnDvFN5Em48ePZ8WKFcydOxdTU1MGDx5MeXn5I5XVrVs32rdvj7W1db2KcqieAy5evMiePXu4c+cO7du354033qgzUsfjoKNqrK3mM8Tp06dZt24d3377bavT+u7Zs4cTJ07w2WeftXRVBAKBQNBMCDnUtLTm9jx58iQbN27k66+/rtPfhEAgELQUrXnubIuyqDWgVCqZOnUqkyZN4oUXXmjp6qAXrt7yEGhhb2+PSqVCLpdjamra0tXRIDMzkwEDBkhmVAKBQCB4+hByqGlpze159epVXnrpJekIjUAgELQWWvPc2RZlUUuiPjYWFxdHVlYWU6dObRWKJmEpIRAIBAKBQCAQCAQCwVNOQUEB06dPp3379rzzzjutxheHUEoIBAKBQCAQCAQCgUAgaBFa3lZDIBAIBAKBQCAQCAQCwTOJUEoIBAKBQCAQCAQCgUAgaBGEUkIgEDzz3L59m9DQUPLy8lq6KlosWbKEXbt2tXQ1BAKBQNAElJSUsHr1asLCwhg9ejShoaF88cUXnD17tqWrJhAInhBi3alN/cGQBQKB4BkhJiaGnj17Ym9vD8C6deu4ePEiWVlZWFpaEhkZqXXP0aNHiYmJ4fr165ibmzNo0CCGDRumkWfPnj3s3buXgoICrK2tGTFiBIGBgdL1Y8eOERsbS15eHpWVldjb2zNkyBBeeuklKU9wcDBz584lKCgIExOT5mkAgUAgEDwRvvnmG+7fv88777yDvb09xcXFpKamUlpa2tJVazIqKirQ1xefGAJBfTTXurOiooIdO3Zw6NAhioqKsLCwYOjQoQwePBiArKwsfv75Z65evUpBQQHBwcGMHDlSo4yWWneKGUMgEDzT3L9/n3379jFr1iwpTaVSERgYyLVr1+rcvTp16hTff/89EydOxNfXl5ycHKKiopDJZAwaNAiAhIQEoqOjmTp1Kp6enmRkZBAVFYWpqSn+/v4AmJmZMWLECJycnNDT0+PkyZOsWLECc3Nz/Pz8AHBxccHOzo5Dhw5JZQsEAoGg7XHnzh3S0tL47LPP6N69OwA2NjZ4eHhIecLCwhg4cKDGx0Z4eDgdOnRg8uTJUp5+/fqRn5/Pb7/9hqmpKSEhIfj4+LBq1Sr++OMP5HI5kydPljzrp6SkEBERwSeffMKWLVvIzs7mueee44MPPiA/P59169aRl5dHt27dCAsL0wivuH//fuLi4iQF+yuvvMLgwYOlMIIjR45k0qRJnD9/njNnzvDKK68wbty4Zm9PgaAt0lzrToClS5dy69Ytpk6dKik9lUqlxrNtbGzo3bs3W7ZsqbN+LbXuFEoJgUDwTHPq1CkAOnXqJKVNmjQJgLi4uDqFw6FDh+jVqxcDBw4EwM7Ojtdee43Y2FgGDhyIjo4Ohw4dIigoiL59+0p5Ll++TGxsrKSU8Pb21ih38ODBHDx4kAsXLkhKCQB/f3+Sk5OFUkIgEAjaMEZGRhgZGXHixAk6d+6MTCZ75LJ2797NW2+9xYgRI/jvf/9LZGQk3t7e9OnTh7feeouYmBh++OEHli9frvGcn3/+mQkTJmBiYsL333/P0qVLMTAwYMqUKejq6rJkyRK2bdsmycHExER+/vlnJk2ahLu7O9euXSMqKgp9fX0NmbR9+3befvttQkJC0NHRefRGEgiecppr3XnmzBnOnTvHDz/8gLm5OQC2trYa5Xh4eEhK0JiYmHrr2BLrTuFTQiAQPNOkpaXh7u7eqEVUeXk5BgYGGmkymYxbt25x48YNKU/tBadMJiMjI4OKigqtMlUqFefOnSM3N5cuXbpoXPPw8CAjI0ND2y0QCASCtoWenh7Tpk3j8OHDTJw4kU8//ZQNGzaQnp7e6LJ8fHwYOHAgDg4OjBw5kvLycuzs7AgMDMTe3p433niDkpISsrKyNO4bNWoUXbp0wdXVlVdeeYWLFy8yduxYPD09ee655wgMDCQlJUXKv2PHDsaOHUtAQAC2trb4+/vz2muvsXfvXo1y+/TpQ1BQEHZ2dlofQgKB4P/TXOvO//3vf3h4eBAfH88777zD+++/z9q1aykrK2t0HVti3SksJQQCwTPNjRs3kMvljbrH19eXH3/8kTNnztC9e3fy8vKIj48HQKFQYGtri4+PD/v37+f555/nueee48qVKyQlJVFZWUlpaan0zLt37zJ16lQqKirQ1dVl8uTJ9OzZU+N5crmcyspKCgsLpfOHAoFAIGh7BAQE4Ofnx4ULF7h06RKnT58mPj5esnpoKK6urtLvRkZGGBoa4uLiIqVZWloCUFxcXO99FhYWAFr3qe8pKSnh1q1brFy5klWrVkl5qqqqUKlUGuW6u7s3uO4CwbNMc6078/PzuXDhAvr6+syYMYM7d+6wbt06ioqKmDFjRqOe1xLrTqGUEAgEzzR1WTQ8jKCgIPLy8li8eDGVlZUYGxszePBgtm3bJmm+g4ODUSgUzJkzB5VKhYWFBYGBgcTFxWlox42MjPjqq68oKyvj3LlzrF+/HhsbG+m8MSDVT1hKCAQCQdtHJpPRo0cPevToQXBwMCtWrGDbtm0MGzaszt3TyspKrTQ9PT2ttLqcS9ZWHtS8T/2s2vep76mqqgIgNDRUw9S8LoyMjB54XSAQVNNc6071uP3ggw8kB5WTJk1i/vz5KBQKSVHZEFpi3SmUEgKB4JnGzMyM27dvN+oeHR0dxo4dy+jRo1EoFJibm3Pu3Dmg+pwfVE/o06ZNY8qUKRQXFyOXy0lMTMTY2Fg66wegq6sraaHd3NzIyckhJiZGQymhrl/N+wQCgUDwdODs7ExVVRVKpRJzc3OKioqka0qlkpycHNzc3J54vSwtLZHL5eTn52tEjhIIBI9Oc607LS0tsbKy0oiY4eTkBMDNmzcbpZRoiXWnUEoIBIJnGjc3Nw4ePPhI9+rq6mJlZQVAcnIyXl5eWhO4vr4+7du3l/L4+flJHsvroqqqivLyco20rKwsrKysGiVQBAKBQNC6KC0tZcmSJfTr1w9XV1eMjY0lB8je3t6YmJjg7e3N/v378ff3x9zcnJ07d9ZpKfGkGDlyJGvXrsXExAQ/Pz8qKiq4evUqhYWFvP766y1WL4GgrdJc687OnTtz/PhxysrKJMul69evA9VRfhpDS6w7hVJCIBA80/j6+hIdHU1paakUAi0vL4+ysjKKioqoqKggMzMTqN7N0tfXp6SkhOPHj9O1a1cqKirYv38/x44dIyIiQio3NzeXjIwMPD09uXPnDvHx8WRlZREWFibl2blzJx4eHtjZ2VFeXs6pU6ckB2g1SUtLk8K6CQQCgaBtYmRkhKenJ7/++it5eXmUl5djZWVF3759eeONNwB47bXXKCgoYPHixRgZGTFixAgNy4knTVBQEIaGhuzatYvNmzcjk8lwdnYW0aAEgkekudadffv2ZceOHf+vvbsLiWrr4zj+05mmxJy0RHsRidCJNGoyg6JCagKngoqoiLKLBAus6EbqwgKDbopIEIu80UDspgsxvDBpEoUpCUl7QQ0mEF8mU6YmX0BzdJ6LcHhsPD3P6djZneP3A3Oz9lpr//e+GJf/+e+1dffuXR09elSjo6O6f/++tm7dGto/JhAIqLe3V9K3Kiy/36+uri4tWrRoxt4RRqw7I4LfP2wGAPNMYWGhdu7cGVpkFRUVqb29PaxfaWmpEhISNDQ0pBs3bqi7u1uSZLPZdPz4caWmpob69vb2qqSkRF6vVyaTSenp6crJydHKlStDfR48eKDm5mb5fD5ZLBatWrVKTqcz9BpR6dsfjby8PBUWFspms/2qWwAAAIC/wa9Yd0rffhArLy9XZ2enoqOjtWXLFp08eVJRUVGSpIGBAZ0/fz7sPGlpaSoqKpJk3LqTpASAea+trU0VFRUqLi7+4aMVRqirq1NLS4uuXLlidCgAAAD4i1h3hvu97gIAGMButys7O1s+n8/oUMKYzWbl5uYaHQYAAADmAOvOcFRKAAAAAAAAQ1ApAQAAAAAADEFSAgAAAAAAGIKkBAAAAGCgkZER5eXlqb+/3+hQwlRWVqq8vNzoMAD8i5mNDgAAAACYz6qrq7Vp0yYtX75cklRRUaF3796pp6dHsbGxunPnTtiYZ8+eqbq6Wh8+fJDVapXT6dSBAwdm9Kmrq9Pjx481MDCg+Ph4HT58WFlZWaHjf/QqwqSkJN2+fVuSdPDgQV24cEH79+9XYmLiXF42AEgiKQEAAAAYZnx8XE+fPtXly5dDbcFgUFlZWeru7tbr16/DxrS2tqqkpESnT5+W3W5XX1+fysrKZLFY5HQ6JUn19fWqqqrS2bNnlZqaKo/Ho7KyMkVHRyszM1OSVFBQoEAgEJp3YmJCBQUF2rZtW6jNarVqw4YNqq+v16lTp37VbQAwj/H4BgAAAGCQ1tZWSdLatWtDbbm5udq7d69WrFgx65impiZt3rxZ2dnZSkxMVEZGhg4dOqSamhpNv1ivqalJDodDO3bsUGJiorZv3649e/aopqYmNM/ixYsVGxsb+nR2dmp8fFy7du2acb7MzEy53e65vnQAkERSAgAAADBMR0eH1qxZo4iIiP97zMTEhBYsWDCjzWKxyOfzaXBwMNTHYrGE9fF4PDOqI/6by+WS3W5XfHz8jPaUlBR9+vTpt9zzAsA/H0kJAAAAwCCDg4OKi4v7U2PsdrtaWlr06tUrTU1Nyev1qra2VpLk9/slSRs3blRDQ4M8Ho+CwaDev38vl8ulyclJDQ8Ph83p9XrV3t4uh8MRdmw6vumEBwDMJfaUAAAAAAwyW0XD/+JwONTf36+bN29qcnJSUVFR2rdvnx4+fBiquDhy5Ij8fr+uXr2qYDCoJUuWKCsrS48ePZq1KsPlcikuLk4ZGRlhx6bj+/r1609cIQD8GEkJAAAAwCAxMTEaGRn5U2MiIiKUk5OjEydOyO/3y2q16s2bN5IUekOGxWJRfn6+zpw5oy9fviguLk5PnjxRVFSUrFbrjPkCgYAaGxvlcDhkMpnCzjcd3/fjAGAu8PgGAAAAYJDVq1err6/vp8ZGRkZq6dKlMpvNcrvdstlsYYkDs9msZcuWKTIyUm63WxkZGYqMnPkvwIsXLzQ8PKzdu3fPep6enh6ZTCYlJyf/VJwA8CNUSgAAAAAGsdvtqqqq0vDwsGJiYiRJ/f39Ghsb0+fPnxUIBNTV1SVJSkpKktls1tDQkJqbm5WWlqZAIKCGhgY9f/5c165dC83r9Xrl8XiUmpqq0dFR1dbWqqenR+fOnQuLweVyaf369aEqi+91dHRo3bp1Wrhw4dzfAADzHkkJAAAAwCDJyclKSUmR2+2W0+mUJN27d0/t7e2hPpcuXZIklZaWKiEhQZLU2NioyspKSZLNZlNRUZFSUlJCY6amplRbWyuv1yuTyaT09HRdv349NH7ax48f9fbtW128ePEPY3S73Tp27NjcXDAAfCciOP0yYwAAAAB/u7a2NlVUVKi4uDjs0QqjvXz5UpWVlbp169as+011BXLuAAAAa0lEQVQAwF/1e33rAQAAAPOM3W5Xdna2fD6f0aGEGRsbU35+PgkJAL8MlRIAAAAAAMAQVEoAAAAAAABDkJQAAAAAAACGICkBAAAAAAAMQVICAAAAAAAYgqQEAAAAAAAwBEkJAAAAAABgiP8Ay6vDS8y6EY0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 690 }, "id": "VCnP_uOT-ApN", "outputId": "af46a636-923f-4a9b-81e1-418a70319878" }, "source": [ "plot_components(components_df, 'fc0', ascending=True)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABCUAAAL1CAYAAADw5l6HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZQV5Z038G9DAyICzSIYZYuyqLxRCS4oGFQQJyqoqGFcIkiirzGOS0YzGscBjVGJM1l8jTETFUmicRsVMYkLKFFj1GhU3FAScQETiUATXJCt3j883WPbDULTWmg+n3NyYj/1VNVTv1t1D/d7q55bURRFEQAAAICPWbOyBwAAAAD8YxJKAAAAAKUQSgAAAAClEEoAAAAApRBKAAAAAKUQSgAAAAClEEoA/IMYN25cKioq8tJLL5U9lA3WmGOZOXNmKioqMnHixA3e/0svvZSKioqMGzdug7f1j2TOnDk55JBDssUWW6SioiJVVVVlDwma3KfpvRbg4yCUAGgCFRUVqaioSLNmzfLnP/95jf323nvv2r5XX331xzfAfwCCgqa1ISFOQ6/DqlWrcvDBB+fXv/51DjzwwEyYMCFnnnlmo8d3++23Z6+99kr79u2z2WabZbfddsuUKVMa7FvzIZFPD9c7wKeHUAKgiVRWVqYoilx55ZUNLp8zZ05mzpyZysrKj3lk77nwwgvz3HPPZauttipl/03p03Qs/yjmzp2bZ599NmPHjs0VV1yRiRMnNjqUuPTSSzNy5Mg8/fTTOfroo3Pcccfltddey7hx43L66ac38cgBgI+SUAKgiXTt2jU777xzJk+enJUrV9ZbfsUVVyRJRo4c+XEPLUnymc98Jttuu21atGhRyv6b0qfpWP5RvPbaa0mSLbfccoO289JLL+X0009Px44d8+ijj+ZHP/pRvv/972fWrFnZZptt8l//9V/5/e9/3xRDBgA+BkIJgCZ03HHH5a9//Wtuv/32Ou0rVqzI1VdfnT322CPbb7/9GtefM2dOjjnmmGy11VZp2bJlttxyyxxzzDGZM2dOnX4nnHBCKioqMnXq1Aa38/DDD6eioiKHHXZYbdvannN++OGHc9hhh2WLLbZIy5Yt07179/zf//t/az9Ivt+LL76Y448/Pr17907r1q3TsWPHfO5zn8sJJ5yQhQsXrq08Sd77UNrQHQ49e/ZMRUVFvv3tb9dp/81vfpOKior8x3/8xxqPZeLEifnsZz+bJJkyZUrtIzJrekzmiSeeyAEHHJCqqqpsuummGTp0aB588MEPHfu6+Mtf/pKvf/3r6dWrV1q2bJnNN988o0ePzmOPPVan35133pmKioqcffbZddrvvffe2rG/+uqrdZaNGTMmFRUVefHFF+u0z549O+PGjUv37t3TsmXLdO3aNUceeWSef/75euN7/fXXc/rpp6dfv35p06ZNqqqq0q9fv4wbN652u+PGjcvee++dJDn33HPr1HPmzJnrXZOKiooMHTq03vbe/2jIqlWrcvnll2fw4MFp3759Wrdund69e+erX/1qnfP/qquuyrvvvpuTTjopvXr1qm3v0KFDvvWtbyVJLr/88vUe4wfNmzcvJ598cvr06VN7nu+66671zs8keeyxx3LooYemS5cuadWqVXr27JkTTzwxf/nLX+r1rTl3586dm0svvTTbb799Ntlkk/Tq1SsXXHBBiqJIktx4443Zdddd06ZNm3Tp0iUnnXRS3nnnnXrbq6ioyF577ZXXXnstX/7yl9OlS5e0bt06AwcOzLXXXtvgsa1evTqXX355dtlll2y22WZp06ZNdtlll/z4xz/O6tWr17iPN954I8cff3w+85nPpFWrVunfv38mT568xhreeeed2X///dO5c+e0atUq22yzTc4444xUV1fX69urV6/06tUrb731Vs4444z06NEjrVq1Su/evTNp0qTauiTrf73XWLZsWaqqqtKlS5cGg+Mk+drXvpaKioo67+G33nprjj766PTt2zdt2rRJmzZtMnDgwFxyySUN1qshH/Y4VM3xN+SXv/xl9t5771RVVWWTTTbJdtttl/PPPz/vvvtuvb73339/Ro4cmW7duqVVq1bZYostMmjQoJx77rnrNE6AMpRzDzHAp9QRRxyRb3zjG7niiity8MEH17bfdtttWbBgQSZNmpQ//elPDa77hz/8IcOHD8/SpUszatSobL/99pk9e3Z+8YtfZOrUqZk+fXp22WWXJMnYsWPzk5/8JD/72c9y0EEH1dtWzbP16/K89VVXXZXjjz8+rVq1yqhRo9K9e/fMmTMnV1xxRaZNm5aHHnooPXr0SPLeB+5ddtklf//737P//vvn0EMPzbJlyzJ37tz8/Oc/z0knnZROnTqtdX/77LNPrrnmmsyePTvbbrttkuRPf/pTXnnllSTJjBkzcs4559T2nzFjRpJk2LBha9zmXnvtlerq6vzwhz/MjjvuWKf2O+20U52+jz76aL773e9m9913z1e/+tW88sor+Z//+Z8MGzYsTzzxRPr16/ehNVuTuXPnZsiQIXnttdeyzz775Igjjsirr76aG2+8Mb/61a/yP//zPznwwAOTJHvuuWdatmyZGTNm5Dvf+U69463575rXsCiK3HvvvenVq1e23nrr2j533HFHRo8enRUrVmTkyJHp3bt35s2bl5tvvjm/+tWvcu+99+bzn/98kuTtt9/O4MGD8+c//zn77rtvRo4cmaIo8vLLL2fq1Kk57LDDsvXWW9fWb8qUKRk6dGj22muv2v2t6YPT2kyYMCEvvfRSve3V/P/y5ctz4IEH5u6770737t1z5JFHpl27dnnppZdyyy23ZMiQIenTp0+S5J577kmS/NM//VO9/Xzxi1+s06exHn300ey3335ZtGhRvvCFL2T06NF5++238+yzz2bixIl1zs/bb789hx56aIqiyGGHHZaePXvmsccey49//ONMnTo1DzzwQO0H6Pc7/fTTM3PmzIwcOTIjRozIbbfdlrPPPjvLly9Px44dc+aZZ+bggw/Onnvumbvvvjs/+tGPsmrVqvz4xz+ut63Fixdnjz32SFVVVY499thUV1fnhhtuyFFHHZX58+fnjDPOqNP/y1/+cq699tp07949X/3qV1NRUZFbbrklJ554Yh544IFcc8019fZRXV2dwYMHp2XLljnssMPy7rvv5sYbb8z48ePTrFmzjB07tk7/c889NxMnTkzHjh1z4IEHpkuXLpk1a1b+8z//M7/+9a/z+9//Pu3atauzzooVK7Lffvvltddeyxe/+MVUVlbm1ltvzZlnnplly5ZlwoQJSdbven+/TTbZJGPGjMl///d/5ze/+U29u9befffdXH/99enatWud8+vMM89Ms2bNsttuu2WrrbbKkiVLcs899+SUU07JH/7wh/z85z9f4z431Pjx4zN58uR069Ythx56aKqqqvLQQw/lnHPOyYwZM3L33XfXPhJ4xx135IADDki7du0yatSobLXVVlm0aFGee+65XHbZZbX1A9joFABssCTFVlttVRRFUXzlK18pmjdvXrz66qu1y/fbb7+iXbt2xVtvvVWcffbZRZJi8uTJtctXr15dbLvttkWS4he/+EWdbV933XVFkqJfv37FqlWratv79u1btGzZsli4cGGd/suWLSs6dOhQdOnSpVixYkVt+9ixY4skxdy5c2vbnn/++aJFixbFNttsU8ybN6/OdqZPn140a9asOPjgg2vbLrnkkiJJ8YMf/KBeDd58883i7bff/tBaXXnllUWS4tJLL61tu/zyy4skxb777lu0bNmyeOutt2qX7bTTTkXr1q2Ld999d63HMnfu3CJJMXbs2Ab3e++99xZJ6tX+/fv/2te+9qHjX9u+RowYUSQpzj///Drtv/vd74rmzZsXHTt2LJYuXVrbvueeexbNmzcvqqura9sGDRpUDBgwoOjUqVNx9NFH17Y/8cQTRZJi/PjxtW2LFi0qqqqqik6dOhXPPPNMnX0+9dRTRZs2bYoBAwbUtt12221FkuLUU0+td0zvvvtu8fe//73275p6TZgwYZ1q8mHWtr2zzjqrSFKMHDmyWLZsWZ1ly5YtKxYsWFD7d+fOnYskxRtvvNHgftq0aVMkqXMOrY9333236NWrV5GkuOaaa+otf/91vXTp0qJjx45Fs2bNivvuu69Ov4suuqj2nH6/mnO3Z8+eda65xYsXF506dSo23XTTonPnzsWzzz5bu2zZsmXFdtttV7Rs2bJ4/fXX62yv5pw+/PDD67w/vPjii0WHDh2KFi1aFH/+859r26+99toiSTFgwIA65+Kbb75ZDBw4sMHjrtnHV77ylWLlypW17c8880zRvHnzYrvttqvT/5577imSFLvvvnuxePHiOssmT57c4DnYs2fPIknxxS9+sc77yOuvv160b9++aN++fbF8+fLa9g+73tfkwQcfLJIUhx56aL1lN9xwQ5Gk+MY3vlGn/U9/+lO9vqtWrSqOOeaYIknx0EMP1VnW0PvTh11PPXv2LHr27FmnraZWhxxySL331gkTJtR7Lx49enSRpHjiiSfqbf9vf/tbg/sF2Bh4fAOgiR133HFZtWpVrrrqqiTJyy+/nLvvvjtHHXVUNt100wbXefDBBzN79uzsvvvuOeqoo+osGzNmTIYMGZLnn38+DzzwQG372LFjs3z58vzyl7+s03/atGlZvHhxjjrqqA+dVPPHP/5xVqxYkR/+8If1HqkYNmxYRo0alWnTpmXp0qV1lrVu3brettq0adNg+wfV3PHwwTsCunbtmpNPPjnLly+vPc6FCxfmySefzJAhQ9KyZcsP3fa6GDx4cL07SMaPH5/Kyso88sgjjd7uvHnzctddd6VHjx755je/WWfZHnvskSOOOCKLFi3KzTffXNs+bNiwrFq1Kr/97W+TJEuXLs2jjz6afffdN3vvvXedb/wbumPkZz/7Waqrq3PuuefWeyzo//yf/5Pjjjsujz/+eJ599tk6yxp6nVq2bJm2bds28ugbb9WqVbnsssvSunXrXH755WnVqlWd5a1atcrmm29e+/eSJUuSJO3bt29wezXtNf3W17Rp0/LSSy9l1KhROfLII+st79atW+1/T506NYsWLcqYMWOy55571un3r//6r+nVq1fuvvvu2ruA3u+cc86pc81VVVVl1KhRefvtt/O1r30t2223Xe2yVq1aZcyYMVm+fHmee+65ettq3rx5Jk2alGbN/vefdZ/97Gdz8sknZ8WKFXW+ya95X7rooouy2Wab1ba3adMmkyZNSvK/89+836abbprvfe97ad68eW3b9ttvn8GDB+e5557Lm2++Wdt+ySWXJEl++tOf1vvZ13HjxmWnnXZq8G6MmnXff3526dIlBx10UJYsWdLg40jra/fdd0/fvn0zbdq0LFq0qM6ymjvMPnjXxzbbbFNvO82aNcspp5yS5L3HVD4KP/zhD1NZWZmrrrqq3jV7zjnnpFOnTg3WsaHru3Pnzh/JGAGagsc3AJrYbrvtls997nO56qqr8u///u+54oorsnr16hx33HFrXOePf/xjkvcebWjIPvvskwceeCCPP/54vvCFLyRJjjnmmJxzzjmZMmVKvv71r9f2XZ9HN2omBPztb3+bP/zhD/WWL1iwIKtWrcoLL7yQgQMHZtSoUfnWt76Vr3/967nzzjuz3377ZfDgwdl+++3X+ScXe/bsma233jozZ87M6tWra+cpGD58eIYOHZrKysrMmDEjI0aMyL333puiKNZYl8bYeeed67W1aNEiXbt2zeLFixu93ccffzzJe49lNDQB5z777JNf/OIXefzxx3PMMcfUtk2cODEzZszIqFGj8tvf/jYrV67MsGHD0qtXr9x000157rnnst1229UGFO+vRc3r9+STTzb4rPoLL7yQJHnuueey/fbbZ+jQodlqq61y0UUX5Y9//GP233//DB48ODvttFOdD5sfp9mzZ2fJkiXZbbfdNngSzKbw0EMPJfnfR0HWZm3XbWVlZb7whS/kpZdeyuOPP177CFSNhs7DmuMfOHBgvWU1Aca8efPqLevRo0eDj4jstddeOffcc2vPzZoxN2vWrM4jOTWGDh2a5s2b1+lfo0+fPvUet0iS7t27J3nvEZKakOP3v/99WrRokRtvvDE33nhjvXWWL1+ev/3tb1m4cGGdx73at2+f3r17r3UfTWHs2LE5++yzc9111+XEE09M8t5cK3feeWcGDBiQHXbYoU7/hQsX5uKLL86vf/3rvPjii3nrrbfqLJ8/f36TjOv93n777Tz55JPp3LlzfvCDHzTYp1WrVnVCqqOOOio333xzdtttt4wZMyZ77713Bg8eXCdIA9gYCSUAPgLHHXdcTj755PzmN7/J5MmTM3DgwAwYMGCN/Wu+1f3MZz7T4PKa9vdPENetW7cMGzYsd999d+0H1wULFuSOO+7ITjvtVO8f1g2pmZjy4osvXmu/mm9Be/bsmUceeSQTJ07MHXfcUfutf/fu3XP66afn5JNP/tB9Ju992//Tn/40f/zjH9OiRYv87W9/y7Bhw9K2bdvssssutXcFrMt8Euvrg9/c1qisrMyqVasavd3GvIaDBg1KmzZt6hxvy5YtM2TIkNq5G2bMmJE+ffrkvvvuy/bbb58tttiidv2a1++nP/3pWsdW8/q1a9cuDz30UCZMmJDbbrut9hvezp0758QTT8y///u/f+y/aFJTj3X9edf27dvnjTfeyJIlSxqcv+TD7qRoyvE05jWv0dD4au5sWtuyFStW1FvWtWvXBvdfc668/66RJUuWpGPHjg3eeVRZWZnOnTtnwYIF9Zat7bpJUufaWbhwYVauXPmhkyu++eabdV7D9dnHhnh/oFsTSlxzzTVZuXJlvbskqqurs8suu2Tu3LnZddddc8wxx6Rjx46prKysndeioQknN9TixYtTFEX+9re/rfMklaNHj87tt9+e//qv/8pVV12Vn/zkJ0neC7kuvPDC7Lvvvk0+ToCm4PENgI/Al7/85bRu3TonnHBC5s+fn+OPP36t/Ws+hPz1r39tcHnNLP4f/LBS8w/omrsj1vQP6w/b75IlS1IUxRr/V/PLCUmy3Xbb5frrr8/ChQvz6KOP5qKLLsrq1atzyimn5Morr1yn/dZ8szx9+vR6wcM+++yTxx9/PIsWLcqMGTPSvn372okaN2aNeQ1btGiRIUOG5Jlnnslf//rXzJgxI7vvvns23XTT9O3bN926dcv06dPzyCOPZOnSpfW+ka/Z1pNPPrnW1+/950O3bt1y5ZVXZsGCBXn66adzySWXpFOnTjnvvPNy3nnnNWlN1kXNB9F1/ba5ZiLSmrtA3u8vf/lL3nrrrXTr1m2Nj0o15Xgae902tddff73B9ppxvX//7du3z6JFixoMN1auXJk33nijwTsi1kf79u3ToUOHtZ6TRVGkZ8+eG7SfxurWrVv22WefPPLII5k9e3aS995DW7RoUe+RnSuuuCJz587NhAkT8vDDD+eyyy7L+eefn4kTJ2bMmDHrvM+aR2vW9KsfHwyual6zAQMGfGgd3++AAw7IPffck8WLF2fGjBk57bTT8swzz+TAAw+s9xgXwMZCKAHwEaiqqsphhx2WefPmpU2bNjniiCPW2r/mLoo1/dzivffemyT1PpyPHj067dq1yy9+8YusXr06U6ZMSWVlZYPPwjdk0KBBSd77Gbn1VVlZmYEDB+bf/u3faue1uPXWW9dp3X322ScVFRWZMWNG7rnnnmy99da1dwYMGzYsq1evzs9+9rPMmTMne+211zo9WlDTp6m+TV1fNa/hAw880OAHjzW9hjVhzC9/+cs8/fTTde4K2WeffTJz5szcfffddfrW2JDXr6KiIv3798+//Mu/1G7//a/fx1XPbbfdNlVVVZk1a1aDP0H7QTXBzB133FFv2W9+85s6fRqjpqY121qbtV23K1eurH1dPupQ7ZVXXmnwp35rxvX+u7QGDBiQ1atX57777qvX/7777suqVas2eLyDBg3K4sWL88wzz2zQdtZmQ8/PmsfbpkyZkieeeCKzZs3KF7/4xTrzlySp/bWkQw89tN42auaCWRcdOnRIkno/81uzjw/OgbLZZpulf//+eeaZZ+rNfbEu2rRpk3322Sff+9738q1vfSvLly9fp3MaoAxCCYCPyPnnn59bbrkld95554dOIDh48OD069cvDzzwQG666aY6y2666abcf//96du3b4YMGVJnWevWrfOlL30p8+fPz/e///08+eST2X///dOlS5d1GuNJJ52UFi1a5LTTTmvwm+fly5fX+cD72GOPNTiBYM03tev67XSXLl3Sv3///O53v8t9991X58P2HnvskU022SQXXnhhknX/gNmhQ4dUVFQ0OKngx6Fbt27Zd99989JLL9V7Bvzhhx/Otddemw4dOuSQQw6ps6zm+C666KIURVEvlFiyZEkuu+yyBucBOPbYY1NVVZVzzz23wUk6V69eXecD8zPPPNPgt+oNvX41t9V/1PVs3rx5TjzxxLzzzjs54YQT6t0KXzP/QI1jjz02rVq1yqWXXlrng/jixYtzwQUXJElOOOGERo9n5MiR6dWrV2677bZ6k8gmded0OPjgg9OxY8f88pe/rJ2LosYPfvCDzJ07N8OHD683n0RTW7VqVf7t3/4tq1evrm2bO3duLrnkklRWVuboo4+ubR8/fnyS5Kyzzsrbb79d2/7222/nzDPPTJJ85Stf2aDxnHbaaUnee4ytoaDprbfeqlev9bWh1/v7A92rr746ScPz8NSEpR8Mnh5//PHa96h1se2226Zdu3aZOnVqncdj3nnnnTU+9vaNb3wjy5cvz/jx4xt8BGjx4sW185ok74VKDQWi6/v+DPBxM6cEwEekR48e6/xhpKKiIlOmTMm+++6bMWPG5KCDDsq2226b559/Prfeemvatm2bn/3sZ3Vm168xduzYXHHFFTnrrLNq/15X2267ba666qqMHz8+/fv3zz/90z+lb9++WbFiRV555ZXcf//92XzzzWtvcf75z3+en/zkJxkyZEi22WabdOjQIX/+858zbdq0tGrVKqeeeuo673vYsGF5+umna/+7RqtWrTJ48OD1nk9is802y2677Zb7778/Rx11VPr27ZvmzZtn1KhR6zS/RlO4/PLLM3jw4Jxxxhm56667svPOO+fVV1/NjTfemGbNmmXy5Mn1AqoBAwakQ4cOWbBgQdq2bZtdd921dlnNsS9YsCA777xzvWfuO3XqlJtuuimHHHJIBg0alGHDhqV///6pqKjIq6++mt///vdZuHBhli1bliS5++67c8YZZ9T+AkGXLl0yb968TJ06Nc2aNcsZZ5xRu+1+/fplq622ynXXXZcWLVqkZ8+eqaioyJe//OUmv+2+5tb4adOmpW/fvjnwwAPTtm3bvPrqq7nrrrty8cUX135g/OxnP5uLL744J598cnbeeeeMGTMmLVu2zE033ZR58+blX//1X7P77rs3eiwtW7bMjTfemBEjRuTII4/MT37ykwwaNCjLli3Lc889lxkzZtR+8Ntss81y1VVX5fDDD8/QoUNz+OGHp0ePHnnsscdy1113ZYsttqh9rv+jtMMOO+Thhx/OwIEDM2LEiFRXV+eGG25IdXV1vvvd79b59YgjjzwyU6dOzQ033JD+/fvn4IMPTkVFRW699dbMnTs3Y8aMqfcLQOtr2LBhueiii3LWWWelT58+2X///fPZz342b775Zl5++eX89re/zZAhQxq822Vdbej13rp16xx++OG58sorc9lll6VTp0454IAD6vU75phjcvHFF+fUU0/Nvffemz59+mTOnDm5/fbbM3r06Fx//fXrNN4WLVrklFNOybe//e0MGDAghxxySFauXJm77747W265ZYOTvI4fPz6PPfZYLrvssmyzzTbZb7/90qNHjyxatChz587Nfffdl2OPPTaXX355kuTkk0/O/PnzM3jw4PTq1SstW7bMY489lnvuuSc9e/bMP//zP6/TWAE+dh/5j44C/ANIUmy11Vbr1Pfss88ukhSTJ0+ut2z27NnF0UcfXWyxxRZFZWVlscUWWxRHHXVUMXv27LVus3fv3kWSomPHjsW7777bYJ+xY8cWSYq5c+fWWzZr1qxi7NixRY8ePYqWLVsWHTp0KPr3718cf/zxxYwZM2r7PfTQQ8UJJ5xQ7LDDDkWHDh2KTTbZpNhmm22KcePGFU899dQ6HX+N2267rUhSVFRUFK+//nqdZRdccEGRpOjatet6HcucOXOKAw88sOjYsWNRUVFRp8733ntvkaSYMGFCg9vs2bNn0bNnz3Ua+9y5c4skxdixY+stmzdvXnHCCScUPXr0KFq0aFF06tSpOOigg4pHHnlkjdsbPXp0kaTYf//96y3r27dvkaT45je/udbxfP3rXy969+5dtGrVqmjbtm3Rr1+/4uijjy5uueWW2n7PPvtscdpppxUDBw4sOnfuXLRs2bLo2bNnceihhxa/+93v6m33kUceKfbZZ5+iXbt2tfW89957116cNfiw+q9YsaL4f//v/xW77LJL0aZNm2LTTTctevfuXRx33HHFnDlz6vW/7bbbii984QvFZpttVmy66abFzjvvXFx99dWNGltDXn755eJrX/ta0atXr6JFixZFx44di1133bX4zne+U6/vI488Uhx88MFF586dixYtWhTdu3cvTjjhhGL+/Pn1+q7tOpwwYcIaazx58uQG3zeSFEOHDi3mz59fHHXUUcXmm29etGrVqhgwYEBxzTXXNHhsq1atKn70ox8VAwcOLFq3bl20bt26+PznP19ceumlxapVq+r1r9lHQ9Z2PPfff39x+OGHF5/5zGeKFi1aFJ07dy523HHH4rTTTiv+8Ic/1Om7tutvTXVZ2/W+Lu6///4iSZGkOOmkk9bY75lnnilGjhxZbL755sWmm25afP7zny9++tOfrvF9YE01Wb16dXHhhRcWW2+9de15csYZZxRvvfXWWo9/2rRpxQEHHFBsvvnmRYsWLYquXbsWu+yyS3H22WcXzz33XG2/66+/vvjnf/7nonfv3kWbNm2Ktm3bFv3798mFA2YAACAASURBVC++9a1vFQsWLFjnugB83CqK4gMz5AAA8IlQUVGRoUOHrnE+GgDY2JlTAgAAACiFUAIAAAAohVACAAAAKIVf3wAA+IQyNRgAn3TulAAAAABKIZQAAAAASvGpenzjtddeK3sIH6pz58554403yh7Gp4Z6Nh21bFrq2bTUs+moZdNSz6alnk1HLZuWejYt9Wxan4R6brnllmtc5k4JAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBRCCQAAAKAUQgkAAACgFEIJAAAAoBSVZQ/gH90ll1yy1uUnn3zyxzSSTwf1bDofVstEPdeHc7NpqWfTca03LfVsWq71pqWeTce13rScm03rk1ZPd0oAAAAApRBKAAAAAKUQSgAAAAClEEoAAAAApRBKAAAAAKUQSgAAAAClEEoAAAAApRBKAAAAAKUQSgAAAAClEEoAAAAApRBKAAAAAKUQSgAAAAClqNyQle+8887cdtttqa6uTrdu3TJu3Lhst912a+z/7LPPZsqUKZk3b146dOiQUaNGZcSIEbXLb7nlljzyyCN57bXXUllZmT59+uTII49Mjx49NmSYAAAAwEao0XdKPPjgg7n66qtzyCGHZNKkSenXr18uuOCCvPHGGw32X7BgQS688ML069cvkyZNysEHH5zJkyfnoYcequ3z7LPPZsSIEfn2t7+dCRMmpHnz5vn2t7+dN998s7HDBAAAADZSjQ4lbr/99gwdOjTDhw9Pt27dMn78+HTo0CF33XVXg/3vuuuudOjQIePHj0+3bt0yfPjwDB06NNOmTavtc/bZZ2fvvfdOjx490qNHj/zLv/xL/v73v2f27NmNHSYAAACwkWpUKLFy5cq8+OKL2XHHHeu077DDDnn++ecbXGfOnDnZYYcd6rTtuOOOefHFF7Ny5coG13nnnXdSFEU222yzxgwTAAAA2Ig1ak6Jv//971m9enXat29fp72qqipPPfVUg+tUV1fnc5/7XJ229u3bZ9WqVVm6dGk6dOhQb53JkyenV69e6du3b4PbnD59eqZPn54kueiii9K5c+fGHM7HqrKycr3G+Uk4pjKpZ9NZ31om6rk2zs2mpZ5Nx7XetNSzabnWm45zs2mpZ9NyrTetT3o9N2iiy4/SlClT8vzzz+e8885Ls2YN39AxfPjwDB8+vPbvNc1nsTHp3Lnzeo3zk3BMZVLPprO+tUzUc22cm01LPZuOa71pqWfTcq03Hedm01LPpuVab1qfhHpuueWWa1zWqMc32rVrl2bNmmXJkiV12qurq1NVVdXgOlVVVamurq7TtmTJkjRv3jxt27at03711Vfnd7/7Xf7jP/4jXbt2bcwQAQAAgI1co0KJysrKbL311pk1a1ad9qeeeir9+vVrcJ0+ffrUe7Rj1qxZ2XrrrVNZ+b83bEyePLk2kNhqq60aMzwAAADgE6DRv75x4IEHZubMmZkxY0bmzZuXyZMnZ9GiRdl3332TJJdeemkuvfTS2v4jRozIokWLcvXVV2fevHmZMWNGZs6cmZEjR9b2ueKKKzJz5syccsop2WyzzVJdXZ3q6uosW7ZsAw4RAAAA2Bg1ek6JPfbYI0uXLs3NN9+cxYsXp3v37jnrrLOy+eabJ6n/nEqXLl1y1llnZcqUKbU/D3rsscdm0KBBtX1qfk70vPPOq7PuYYcdli996UuNHSoAAACwEdqgiS7322+/7Lfffg0umzhxYr227bffPpMmTVrj9m644YYNGQ4AAADwCdLoxzcAAAAANoRQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAohVACAAAAKIVQAgAAACiFUAIAAAAoRWXZA/i0m3Z99QdaPvg366p+LRP1bDznZtNSz6bjWm9azs2mpZ5NSz2bjlo2LfVsWurZdD6N/05ypwQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUAqhBAAAAFAKoQQAAABQCqEEAAAAUIrKDVn5zjvvzG233Zbq6up069Yt48aNy3bbbbfG/s8++2ymTJmSefPmpUOHDhk1alRGjBixQdsEAAAAPpkafafEgw8+mKuvvjqHHHJIJk2alH79+uWCCy7IG2+80WD/BQsW5MILL0y/fv0yadKkHHzwwZk8eXIeeuihRm8TAAAA+ORqdChx++23Z+jQoRk+fHi6deuW8ePHp0OHDrnrrrsa7H/XXXelQ4cOGT9+fLp165bhw4dn6NChmTZtWqO3CQAAAHxyNSqUWLlyZV588cXsuOOOddp32GGHPP/88w2uM2fOnOywww512nbccce8+OKLWblyZaO2CQAAAHxyNWpOib///e9ZvXp12rdvX6e9qqoqTz31VIPrVFdX53Of+1ydtvbt22fVqlVZunRpiqJY721Onz4906dPT5JcdNFF6dy5c2MOZ41eP2SPtS4fvdd3P3QbvztlSJ2/Kysrs3Llyve1nLfW9Zs9dNyH7mP1oJ9+aJ+NwYbW84O1TJq+np+WWibrX8/1rWXyj1NP1/r6+SRc68kno56u9aa1MVzriXq+n/fO93w813ri30n/y3vnunOtN61Pwr+TPu5abtBEl2UbPnx4hg8fXvv3xjj3xAfH1Llz5/UaZ5dG7OPTqqHjbOp6/qPUMtnwczNRz/dzrTedj+NaX9N+Po1c601LPZuW986m49xsWurZtFzrTeeT+ployy23XOOyRoUS7dq1S7NmzbJkyZI67dXV1amqqmpwnaqqqlRXV9dpW7JkSZo3b562bdsmyXpvEwAAAPjkatScEpWVldl6660za9asOu1PPfVU+vXr1+A6ffr0qfcYxqxZs7L11lunsrKyUdsEAAAAPrka/esbBx54YGbOnJkZM2Zk3rx5mTx5chYtWpR99903SXLppZfm0ksvre0/YsSILFq0KFdffXXmzZuXGTNmZObMmRk5cuQ6bxMAAAD49Gj0nBJ77LFHli5dmptvvjmLFy9O9+7dc9ZZZ2XzzTdPUv85lC5duuSss87KlClTan8e9Nhjj82gQYPWeZsAAADAp8cGTXS53377Zb/99mtw2cSJE+u1bb/99pk0aVKjtwkAAAB8ejT68Q0AAACADSGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASlFZ9gA+yaYetW3ZQ/hUUc+mpZ5NRy2blno2LfVsOmrZtNSzaaln01LPpqOWTesfsZ7ulAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEohlAAAAABKIZQAAAAASiGUAAAAAEpR2ZiViqLIjTfemBkzZuTNN99Mnz598pWvfCXdu3df63oPPfRQrr/++rz++uvp2rVrjjjiiOy6665JkpUrV+a6667LE088kddffz2tW7dO//79c9RRR6Vz586NGSYAAACwEWvUnRJTp07N7bffnmOPPTYXXnhh2rVrl/PPPz/vvPPOGtd54YUX8oMf/CB77rlnvvvd72bPPffM9773vcyZMydJsnz58sydOzejR4/OpEmT8s1vfjMLFy7Md77znaxatapxRwcAAABstNY7lCiKIr/+9a9z8MEHZ9CgQenRo0dOOumkvPPOO3nggQfWuN6vfvWr9O/fP6NHj063bt0yevTo9O/fP7/61a+SJJtuumnOOeec7LHHHtlyyy3Tu3fvHH/88Zk/f37mz5/f+CMEAAAANkrrHUosWLAg1dXV2WGHHWrbWrZsme222y7PP//8Gtd74YUXsuOOO9Zp23HHHfPCCy+scZ233347SdKmTZv1HSYAAACwkVvvOSWqq6uTJFVVVXXa27dvn8WLF691vfbt29dbp2Z7H7Ry5cr8/Oc/z8CBA9OpU6cG+0yfPj3Tp09Pklx00UVNPvfE6x+yvDH7q6ysXL/1/vThXT4pc258Eur5aallsv7Hst61TP5h6rkxnJuNHUcZ1LPpuNab1kZxbibquRau9TVzra879WxarvWm9Umo58ddyw8NJe6///7893//d+3fZ5111kc6oCRZtWpVLrnkkrz11lv55je/ucZ+w4cPz/Dhw2v/fuONNz7ysb1fY/bXuXPn9Vqvy0c0jo3RxlDPT0stk/U/lvWtZfKPU8+N4dxs7Dg2RurZtFzrTefjODcT9Vwb1/qaudablno2Hdd609oY6vlR1HLLLbdc47IPDSV23nnn9OnTp/bvFStWJHnvzof3JyhLliypdyfE+1VVVWXJkiV12pYsWVLvjotVq1blhz/8YV555ZVMnDgxbdu2/bAhAgAAAJ9AHzqnROvWrbPFFlvU/q9bt26pqqrKrFmzavssX748s2fPTr9+/da4nb59+9ZZJ0lmzZqVvn371v69cuXKfP/738/LL7+cCRMm1AssAAAAgE+P9Z7osqKiIvvvv3+mTp2ahx9+OK+88kouu+yybLLJJhkyZEhtv/POOy/XXntt7d/7779/nn766dx6662ZP39+brnlljzzzDM54IADkrx3h0TNT4SecsopqaioSHV1daqrq7N8+fImOFQAAABgY7LeE10myUEHHZTly5fnyiuvzFtvvZXevXvn7LPPTuvWrWv7vP7663UmqOzXr19OPfXUXHfddbn++uuzxRZb5NRTT619NGThwoV59NFHkyRnnnlmnf2deOKJ2WuvvRozVAAAAGAj1ahQoqKiIl/60pfypS99aY19fvSjH9VrGzRoUAYNGtRg/y5duuSGG25ozHAAAACAT6D1fnwDAAAAoCkIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCUAAACAUgglAAAAgFIIJQAAAIBSCCWA/8/enYdlUe//H3+xyCYCIqICrom5pYkLuJWeSIvSzAzNJTOtNLE81fFIy9Hjt0KrY5ZpLqnlLlhqKm2uRyItPZqiEmqauYEgmKKALL8/vJgfyI43DOrzcV1ded/3zNyf+82sr/nMDAAAAACYglACAAAAAACYglACAAAAAACYwrY8I+Xk5CgiIkKbN2/W5cuX5evrq5EjR6p+/frFjrdz506tWrVK8fHxqlOnjp566il16tSp0GHnzZunTZs2aejQoerbt295mgkAAAAAAKqwcvWUWLdunTZs2KARI0YoLCxMLi4uevvtt3X16tUix4mLi9OMGTPUvXt3vffee+revbumT5+uI0eOFBh2586dOnr0qGrWrFme5gEAAAAAgFtAmUOJnJwcRUZGql+/fgoICFCDBg0UEhKiq1evKioqqsjxNm7cqFatWql///7y8fFR//791apVK23cuDHfcOfPn9eiRYv00ksvyda2XB05AAAAAADALaDMoURCQoJSUlLUpk0b4z07Ozu1aNFCv/32W5HjxcXFqW3btvnea9u2reLi4ozXWVlZ+uijj/TEE0/Ix8enrE0DAAAAAAC3kDJ3RUhJSZEkubm55Xvf1dVVycnJxY7n6upaYJzc6UlSeHi4atSooV69epWqLZs2bdKmTZskSVOnTpWHh0epxiut+BI+L8/32dralm28oyUPYunfXVFuhXreLrWUyv5bylxL6Y6pZ1WYN8vbDjNQT8thWbesKjFvStSzGCzrRWNZLz3qaVks65Z1K9SzsmtZYiixY8cOzZs3z3gdGhpaIQ05ePCgtm3bpvfff7/U4wQGBiowMNB4nZiYWBFNK1J5vs/Dw6NM43lWUDuqoqpQz9ulllLZf0tZayndOfWsCvNmedtRFVFPy2JZt5zKmDcl6lkclvWisaxbFvW0HJZ1y6oK9ayIWnp5eRX5WYmhRIcOHeTr62u8vnbtmqTrPR/yJigXL14s0BMiLzc3N128eDHfexcvXjR6XBw8eFApKSl6/vnnjc+zs7O1bNkyRUZGas6cOSU1FQAAAAAA3EJKDCUcHR3l6OhovM7JyZGbm5v279+vpk2bSpIyMjIUGxuroUOHFjmdZs2aaf/+/fke77l//341a9ZMktS7d28FBATkG+edd95R165d8/WGAAAAAAAAt4cy3+jSyspKQUFBWrdunXbt2qWTJ09q9uzZcnBwULdu3YzhpkyZouXLlxuvg4KCFBMTo7Vr1+r06dNas2aNDh48qEceeUTS9ftLNGjQIN9/tra2cnNzK7arBwAAAAAAuDWV65mbjz32mDIyMrRgwQKlpqaqadOmeuONN/L1qIiPj1etWrWM13fffbfGjx+vlStXatWqVapbt67Gjx+f79IQAAAAAABw5yhXKGFlZaXg4GAFBwcXOcysWbMKvBcQEFDgEo3iFDYNAAAAAABweyjz5RsAAAAAAACWQCgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMQSgBAAAAAABMYWt2A1C8hKZhZjfhtkI9LYt6Wg61tCzqaVnU07Kop+VQS8uinpZFPS2HWlpWVasnPSUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApbM1uQGXJyspSWlqaJMnKyqpU42QPeqHYz62vXClzO+Lj45Wenl7m8SwtJydHkuTg4CAbGxuTWwMAAAAAuBPdEaFEVlaWrl69qurVq5c6kJCknLuaFfu5lZNTmdtia2tbZUKAnJwcpaamytHRscq0CQAAAABw57gjLt9IS0srcyBxJ7CyslL16tWNHiQAAAAAAFSmOyKUkEp/ycadhroAAAAAAMxSrss3cnJyFBERoc2bN+vy5cvy9fXVyJEjVb9+/WLH27lzp1atWqX4+HjVqVNHTz31lDp16pRvmDNnzmj58uWKiYlRZmamvL29NW7cOPn4+JSnqZI48C4J9QEAAAAAmKFcPSXWTa/vggAAIABJREFUrVunDRs2aMSIEQoLC5OLi4vefvttXb16tchx4uLiNGPGDHXv3l3vvfeeunfvrunTp+vIkSPGMAkJCXrrrbfk6empf/3rX/rPf/6jgQMHysHBoTzNBAAAAAAAVViZQ4mcnBxFRkaqX79+CggIUIMGDRQSEqKrV68qKiqqyPE2btyoVq1aqX///vLx8VH//v3VqlUrbdy40RhmxYoVatu2rZ5++mk1adJEderUkZ+fnzw8PMr3625x2dnZmjBhglq1aiVvb29FR0eb3SQAAAAAACymzJdvJCQkKCUlRW3atDHes7OzU4sWLfTbb7/pwQcfLHS8uLg4Pfzww/nea9u2rb799ltJ1w/A9+zZo379+umdd97R77//Lk9PT/Xp00ddunQpazNLJeu5vhUy3aLYzP+6TMNv3rxZ4eHhioiIUMOGDeXm5lbs8CkpKXrrrbf0ww8/SJIefPBBvf3223J1dS13mwEAAAAAqChlDiVSUlIkqcABsqurq5KTk4sd78aDY1dXV2N6f/31l9LS0rRmzRoNHDhQQ4YMUUxMjD7++GM5ODjIz8+vwDQ3bdqkTZs2SZKmTp1aZI+K+Ph42doW/KlZxfzOipDbhsLaUpiTJ0+qTp066ty5c6mGHzdunE6fPq0VK1ZIkl599VW9/PLLWrp0abHj2dvbV0pvlPgSPi9PG2xtbe/InjQl1VIqez3v1FpKzJuWRj0th2Xdspg3LYt6Wg7LumVRT8tiWbcs6llQiUfHO3bs0Lx584zXoaGhFdKQ7OxsSVKHDh306KOPSpIaNWqkY8eO6dtvvy00lAgMDFRgYKDxOjExsdBpp6eny8bGpgJaXTaZmZmytbVVZmZmicOOHz9eERERkqQ6derIx8dHO3fu1Ny5c7VkyRKdOXNG7u7uGjBggEJDQ3XkyBFt2bJFa9euVbt27SRdD2oef/xxxcbGqmnTpkV+V3p6epG1q0zlaYOHh0eVaHtVVNa6UMuiMW9aFvW0LJZ1y2HetCzqaVks65ZFPS2HZd2ybtd6enl5FflZiaFEhw4d5Ovra7y+du2apOs9H/KmMRcvXiz2MgE3NzddvHgx33sXL140ely4uLjIxsamwFM27tR7KUyZMkU+Pj5auXKlIiMjZWNjo6lTp2rx4sWaNGmS/P39lZSUpJiYGEnSnj17VL16dXXo0MGYRseOHeXk5KQ9e/YUG0oAAAAAAGCGEkMJR0dHOTo6Gq9zcnLk5uam/fv3Gwe6GRkZio2N1dChQ4ucTrNmzbR//3717fv/7+Owf/9+NWvW7HpDbG1111136cyZM/nGO3v2rGrXrl22X3UbcHFxkbOzs2xsbOTp6anU1FTNnz9fkydP1qBBgyRJjRs3NkKIhIQE1apVK9/jPa2srOTh4aGEhARTfgMAAAAAAMUp89M3rKysFBQUpHXr1mnXrl06efKkZs+eLQcHB3Xr1s0YbsqUKVq+fLnxOigoSDExMVq7dq1Onz6tNWvW6ODBg3rkkUeMYfr27avo6Ght2rRJ586d06ZNmxQdHa3evXvf5M+89cXFxSk9PT1fjQEAAAAAuJWV+UaXkvTYY48pIyNDCxYsUGpqqpo2bao33ngjX4+K+Ph41apVy3h99913a/z48Vq5cqVWrVqlunXravz48fkuDenUqZNeeOEFrVmzRosWLVK9evU0duzYQu8ngfw8PT2VlJSknJwco7dETk6OEhMT5enpaXLrAAAAAAAoqFyhhJWVlYKDgxUcHFzkMLNmzSrwXkBAgAICAoqddo8ePdSjR4/yNOu25uvrK3t7e0VFRalJkyYFPm/fvr1SU1O1e/dudezYUZK0e/duXblyRe3bt6/s5gIAAAAAUKJyhRKofM7Ozho5cqSmTp0qe3t7+fv7Kzk5Wfv379fw4cPl6+urnj17auLEiZo2bZokaeLEiQoMDOQmlwAAAACAKolQ4hYSGhoqV1dXzZgxQ2fPnpWHh4cGDBhgfP7JJ5/orbfe0pAhQyRJvXr10ttvv21WcwEAAAAAKNYdHUrYzP+62M9zThwp9nOrRr7Ffn6zRo8erdGjRxuvra2tFRISopCQkEKHd3Nz08yZMyu0TQAAAAAAWEqZn74BAAAAAABgCYQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSVVh2drYmTJigVq1aydvbW9HR0WY3CQAAAAAAi7E1uwFmemxZ7M1N4Meyjb9uSPMyDb9582aFh4crIiJCDRs2lJubW7HDf/TRR9qyZYsOHjyoq1ev6vTp02X6PgAAAAAAKhM9JaqwEydOyNPTUx07dpSnp6fs7OyKHT4jI0MPP/ywRo0aVUktBAAAAACg/O7onhJV2fjx4xURESFJ8vb2lo+Pj3bu3Km5c+dqyZIlOnPmjNzd3TVgwACFhoZKkv7xj39IkjZs2GBauwEAAAAAKC1CiSpqypQp8vHx0cqVKxUZGSkbGxtNnTpVixcv1qRJk+Tv76+kpCTFxMSY3VQAAAAAAMqFUKKKcnFxkbOzs2xsbOTp6anU1FTNnz9fkydP1qBBgyRJjRs3VocOHUxuKQAAAAAA5cM9JW4RcXFxSk9PV7du3cxuCgAAAAAAFkEoAQAAAAAATEEocYvw9fWVvb29oqKizG4KAAAAAAAWwT0lbhHOzs4aOXKkpk6dKnt7e/n7+ys5OVn79+/X8OHDJUmnT59WcnKyTp06JUnGTTAbN26s6tWrm9Z2AAAAAAAKQyhxCwkNDZWrq6tmzJihs2fPysPDQwMGDDA+f//9943HiEpS7969JUkRERHq0qVLpbcXAAAAAIDi3NGhxLohzYv9POfEkWI/t2rka8HWFDR69GiNHj3aeG1tba2QkBCFhIQUOvyMGTM0Y8aMCm0TAAAAAACWwj0lAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglqrDs7GxNmDBBrVq1kre3t6Kjo81uEgAAAAAAFmNrdgPMtH5VSglD1C7+410ljZ9fn4FuZRp+8+bNCg8PV0REhBo2bCg3t6LH//PPPzVjxgxFR0crISFBnp6e6tu3r8aPHy9HR8cyfS8AAAAAAJXhjg4lqroTJ07I09NTHTt2LHHYo0ePKisrS2FhYWrcuLGOHDmif/7zn0pOTtZ7771XCa0FAAAAAKBsuHyjiho/frwmT56s06dPy9vbW/7+/srJydGcOXPUtWtXNW7cWO3bt1dYWJgkqWfPnpoxY4Z69Oihhg0bKjAwUOPGjdPGjRtN/iUAAAAAABSOnhJV1JQpU+Tj46OVK1cqMjJSNjY2mjp1qhYvXqxJkybJ399fSUlJiomJKXIaly9fLvaSDwAAAAAAzEQoUUW5uLjI2dlZNjY28vT0VGpqqubPn6/Jkydr0KBBkqTGjRurQ4cOhY5/6tQpzZkzR+PGjavMZgMAAAAAUGpcvnGLiIuLU3p6urp161bisOfPn9eQIUN033336fnnn6+E1gEAAAAAUHaEEreZhIQEPfnkk7r77rv18ccfy8rKyuwmAQAAAABQKEKJW4Svr6/s7e0VFRVV5DDx8fEaMGCAfH19NXv2bNnacnUOAAAAAKDq4qj1FuHs7KyRI0dq6tSpsre3l7+/v5KTk7V//34NHz5c586d04ABA1S3bl1NnjxZFy5cMMatVauWbGxsTGw9AAAAAAAFEUrcQkJDQ+Xq6qoZM2bo7Nmz8vDw0IABAyRJ27dv1/Hjx3X8+HF16tQp33g7d+5U/fr1zWgyAAAAAABFuqNDiT4Di39cZs6JI8V+btXI14KtKWj06NEaPXq08dra2lohISEKCQkpMOzAgQM1cODACm0PAAAAAACWxD0lAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglqrDs7GxNmDBBrVq1kre3t6Kjo81uEgAAAAAAFmNrdgPM9PHHH9/kFL4p09AvvfRSmYbfvHmzwsPDFRERoYYNG8rNza3IYbOzs/Xss8/q4MGDSkpKkqurq7p166bXX39d9erVK9P3AgAAAABQGegpUYWdOHFCnp6e6tixozw9PWVnZ1fs8F27dtWcOXP03//+V/PmzdMff/yhUaNGVVJrAQAAAAAomzu6p0RVNn78eEVEREiSvL295ePjo507d2ru3LlasmSJzpw5I3d3dw0YMEChoaGytrbWc889Z4zv4+OjkJAQjRgxQmlpaXJwcDDrpwAAAAAAUChCiSpqypQp8vHx0cqVKxUZGSkbGxtNnTpVixcv1qRJk+Tv76+kpCTFxMQUOn5ycrK++uortWvXjkACAAAAAFAlEUpUUS4uLnJ2dpaNjY08PT2Vmpqq+fPna/LkyRo0aJAkqXHjxurQoUO+8d555x0tWrRIV69elZ+fnxYvXmxG8wEAAAAAKBH3lLhFxMXFKT09Xd26dSt2uDFjxui7777TihUrZGNjo3HjxiknJ6eSWgkAAAAAQOnRU+I24+7uLnd3d911111q2rSpOnbsqJ9//ln+/v5mNw0AAAAAgHzoKXGL8PX1lb29vaKioko9Tm4PifT09IpqFgAAAAAA5UZPiVuEs7OzRo4cqalTp8re3l7+/v5KTk7W/v37NXz4cO3evVsxMTHq2LGjXF1ddeLECb3//vuqX7++OnXqZHbzAQAAAAAogFDiFhIaGipXV1fNmDFDZ8+elYeHhwYMGCBJcnBw0IYNG/T+++/r6tWr8vT0VI8ePfTpp5/y9A0AAAAAQJV0R4cSL730UrGf55w4UuznVo18LdiagkaPHq3Ro0cbr62trRUSEqKQkJACw7Zu3VqrV6+u0PYAAAAAAGBJ3FMCAAAAAACYglACAAAAAACYglACAAAAAACYglACAAAAAACYglACAAAAAACYglACAAAAAACYwionJyfH7EZYypkzZwp9/8qVK3Jycqrk1hTO1tZWmZmZZjcjn6pUn7Ly8PBQYmKi2c24LVBLy6KelkU9LYdaWhb1tCzqaTnU0rKop2VRT8u6Ferp5eVV5Gf0lAAAAAAAAKYglAAAAAAAAKYglKjCsrOzNWHCBLVq1Ure3t6Kjo42u0kAAAAAAFiMrdkNMJPn0dBK/b6EpmFlGn7z5s0KDw9XRESEGjZsKDc3t1KNl5aWpkcffVSHDx9WZGSk2rZtW57mAgAAAABQoegpUYWdOHFCnp6e6tixozw9PWVnZ1eq8f7v//5P9erVq+DWAQAAAABwcwglqqjx48dr8uTJOn36tLy9veXv76+cnBzNmTNHXbt2VePGjdW+fXuFheXvffHdd98pOjpa//rXv0xqOQAAAAAApXNHX75RlU2ZMkU+Pj5auXKlIiMjZWNjo6lTp2rx4sWaNGmS/P39lZSUpJiYGGOcM2fOKDQ0VEuWLJGDg4OJrQcAAAAAoGSEElWUi4uLnJ2dZWNjI09PT6Wmpmr+/PmaPHmyBg0aJElq3LixOnToIEnKysrSuHHj9Pzzz6tVq1b6888/zWw+AAAAAAAl4vKNW0RcXJzS09PVrVu3Qj//+OOPVa1aNb3wwguV3DIAAAAAAMqHnhK3iR9//FG7du1Sw4YN873fp08f9e3bV5988olJLQMAAAAAoHCEErcIX19f2dvbKyoqSk2aNCnw+fTp03XlyhXjdXx8vAYPHqyZM2eqY8eOldlUAAAAAABKhVDiFuHs7KyRI0dq6tSpsre3l7+/v5KTk7V//34NHz5cDRo0yDd89erVJUmNGjWSl5eXGU0GAAAAAKBYhBK3kNDQULm6umrGjBk6e/asPDw8NGDAALObBQAAAABAuVjl5OTkmN0ISzlz5kyh71+5ckVOTk6V3JrC2draKjMz0+xm5FOV6lNWHh4eSkxMNLsZtwVqaVnU07Kop+VQS8uinpZFPS2HWloW9bQs6mlZt0I9i+u9z9M3AAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAACAKe6IUOI2updnhaA+AAAAAAAz3BGhhMSBd1GoCwAAAADALHdEKOHg4KDU1FQOwG+Qk5Oj1NRUOTg4mN0UAAAAAMAdyNbsBlQGGxsbOTo66sqVK5IkKysr09pib2+v9PR0074/V25A4+joKBsbG5NbAwAAAAC4E90RoYR0PZioXr262c2Qh4eHEhMTzW4GAAAAAACmuyMu3wAAAAAAAFUPoQQAAAAAADAFoQQAAAAAADAFoQQAAAAAADAFoQQAAAAAADCFVU7usyEBAAAAAAAqET0lKtnEiRPNbsJthXpaDrW0LOppWdTTcqilZVFPy6KelkMtLYt6Whb1tKxbvZ6EEgAAAAAAwBSEEgAAAAAAwBQ2kydPnmx2I+40TZo0MbsJtxXqaTnU0rKop2VRT8uhlpZFPS2LeloOtbQs6mlZ1NOybuV6cqNLAAAAAABgCi7fAAAAAAAApiCUAAAAAAAApiCUwG1n7Nix+vrrr81uhmnCw8P13HPPKTg4WNu2bStyuMmTJ2vBggWV1zCgAh07dkzBwcFKSEgo03jbtm3TsGHDKqhVFSshIUHBwcE6duxYhX9XeHi4Xn311RKHY70CMwUHB2vnzp1mN8PiDh48qODgYP31118WnW5pl+tbQWnWh5W5zryV7dy5U8HBwWY3AybbtGmTxowZo4EDByo8PLzCv8+2wr/hFvXXX38pPDxce/fuVXJysqpXr6769eurX79+atOmjdnNMwQHB+uVV15RQEBApX/3rFmztH37dgUHB2vAgAHG+wcPHtS///1vffbZZ3JxcSlxOpMnT1b9+vU1cuRIi7QrLCxM9vb2FplWVfX7778rNDRUzZo10//93/8Z7588eVKrV6/Wa6+9pmbNmsnJyanIabz22muysbGpjOZWWbnzsCRZW1urZs2a8vPz01NPPSVnZ2eTW2euouaxW9nYsWPVu3dv9e3b1+ymFGvWrFm6dOlSgWeOHzt2TKGhofrkk0/k6ekpDw8PzZs3TzVq1Cj3d+Xk5Gjr1q3aunWrTp48qezsbHl4eKhVq1Z6+OGH5e3tLUnq27evHn744Zv6XVXVjfUuqv7BwcFG7W8XJR143H///Ro7dmyZppmZmanIyEhFRUXpzJkzqlatmry8vNSjRw/16NFD1apVK3EaCQkJCgkJUVhYmO66664yfX9VlnebI0k1atSQr6+vhg0bZixrhbn77rtvelkvzK2yXJdmPn3yySdLnI4l1pnS9f3WQ4cOFdj/laQPP/xQP/30k3r37m2x/VpLuBW26ZY+Hqhqblz+c/n6+uqdd96xyHdY6rjw8uXLWrBggZ5++mkFBATI0dHRIu0rDqFEEf7zn/8oPT1do0ePVt26dXXx4kUdOnRIly5dMrtpVUq1atW0fv169erVq1QBREXKzMyUra2t6e2oDFu2bFHv3r21fft2nTp1Sj4+PpKkc+fOSZI6duwoKyurQsfNrdOdftCd65577tG4ceOUlZWlU6dO6dNPP1VqaqrGjx9vdtNMVdQ8VpTc+QqVx9raWm5ubuUePycnRzNnztSuXbv0+OOPa9iwYXJ3d1dycrL27Nmj8PBw/f3vf5ckOTg4yMHBochp8fe/Nc2bN8/49549ezR37tx879nZ2ZVpepmZmXrnnXd0/PhxBQcHq0WLFqpevbqOHj2qjRs3ysvLS61atbJY+29FudscSbpw4YKWLl2qDz74QB9++GGhw+cuWzezrBelpOW6qijNfHr58uUSp3Oz68y8atWqpe3bt+uJJ54w9rcuXbqk3bt3q1atWhb5Dksq6zb9VldVt0l5l/9cVbGdiYmJysrKUvv27VWzZs1K+c6qV4UqIDU1VYcPH9abb76pe+65R5JUu3ZtNW3a1BimsDNuNyZ8Y8eO1d/+9jclJSXpxx9/lKOjo4KCgvKN88MPP2jDhg1KTEyUg4ODmjRpookTJxpnsLdu3aqvv/5aCQkJ8vDw0IMPPqigoCBZW1sbZy+mT59utHHWrFkVW5wbtG7dWklJSVq9erWeffbZQoc5dOiQli5dqj/++ENOTk7q2rWrhg4dKltbW82aNUuHDh3SoUOH9N1330mScSaquPGk6/X29vaWvb29tm/fLk9PT4WFhRX42yQmJmrRokU6cOCAJKlNmzYaMWKEsdEIDw/Xrl279J///Mdo87Zt27RgwQItWbLEmMbChQt1+PBhXbt2TR4eHnryySfVtWvXiilsMTIyMhQVFaUpU6YoPT1dW7Zs0dNPP63w8HCtXr1akjRw4EDjt+We9WvevLm+/fZbZWZm6rPPPiswv2ZmZio8PFxRUVFKSUmRu7u7goKCFBQUpOzsbM2dO1cxMTFKSUlRrVq19MADD6hPnz6ytr5+FVju97Rp00br1q1TRkaGOnbsqJEjR1bpnivVqlUzdlJq1aqlLl26GJe9ZGdn66uvvtLmzZt18eJF1atXT4MGDVLHjh0l/f+zeS+//LK+//57HT16VN7e3ho7dqysrKw0b948/fHHH2rUqJHGjRtnnGE9d+6cFi9erCNHjigtLU1eXl4KDg5W+/btjXaVZv1RUYqax3Ll/u6XXnpJmzdvVlxcnIYNG6ZevXqVql43nv3Mm+znDvPKK6/ohx9+0G+//abatWtrxIgR+Xqp7du3T59//rnOnz+vu+66S7169Sr2N02ePFnnz5/X0qVLtXTpUknK1x3xwIED+vzzz5WQkKCmTZtqzJgx+c6I7969WxERETp16pTc3NzUrVs3Pfnkk6buTNxYz8zMTC1evFi7du3SpUuX5Orqqm7dumnIkCGFjv/TTz8pKipKEyZMUIcOHYz3PTw85Ovrq7wP57pxPVnUeuVWFx4ebpzJyj07O2nSpAIH0mWt9aVLl7RgwQLFxsbq0qVLqlOnjvr06aOePXtW7A8qQd4DtOrVqxd474cfftDXX3+txMREeXh46LHHHlNgYGCR09u4caMOHTqkd999N98y7unpqYCAAKWlpUm6vvx+9dVX+vPPPyVJTZs21fDhw40DpZCQEElSaGioJKlly5bKfYL9tm3btH79ep09e1bVq1dX27ZtjeGl62f4pk+frr1798rV1VXBwcG67777imzz0aNHtXLlSh0/flyZmZlq0KCBhg0bpmbNmpVcwHLIu81xc3PTI488omnTpikjI0MpKSmFrlvr16+frwdq7j7KhAkTil1vrVmzRpGRkUpLS5O/v7/q1Kmjbdu2GfuKRS3XxW3H09LS9Nlnn2nXrl1ycHBQUFCQfvvtN9WoUaPMvWpKq6T5VJIRSpw/f17Lly8vdNtx4zozt2fvW2+9pRUrVujkyZPy8fHR888/X+KjFdu1a6dffvlFBw8eVOvWrSVJO3bsUNOmTQucFCppfi/tdq+8Stqmb9++XatWrdJff/2l1q1b69577zU+O3PmjMaPH68PPvhADRo0MN7ftGmTVqxYoblz58rW1lanTp3SkiVLdPjwYdnZ2al169Z65plnjL9TSfNWUccD58+fL9D7uqi/48SJExUREaETJ07otddek5+fn77++mtt2rRJFy5cUN26dfXYY4/lWx+sXr1aW7ZsUUpKSqHrE0vLu/wXZsOGDdq2bZvi4+Pl5OSkdu3aadiwYcZ8f+XKFS1YsEC//vqrrl69qpo1a+rhhx/WI488UqbjwuKOjbZt26bZs2dL+v/r4sroJUgoUYjc5Hj37t1q3rx5mc8U5LVx40YFBwerb9++2rt3rxYtWqTmzZurWbNmOnbsmBYsWKCxY8eqefPmSk1NVUxMjDHupk2bFB4ermeffVZNmjTRyZMnjYX/oYceUlhYmEaNGqUXXnhB7du3Nw4OK5OVlZUGDx6s999/X0FBQapbt26+zy9cuKCwsDB1795dL774ouLj4zVnzhxZW1vr6aef1ogRI3T27Fl5eXlp8ODBkiQXF5cSx8u1Y8cOBQYGasqUKSrs6bbZ2dl67733ZGdnp0mTJkmSFi5cqPfff19hYWFF9ia40WeffaZr165p0qRJcnJy0pkzZ8pbspu2c+dO1a5dWw0aNNB9992nDz/8UIMHD1bfvn1Vq1atAmcQpOvBkJOTk15//fUip/vJJ58oNjZWzzzzjBo3bqzz588rKSlJ0vU6uru76+9//7tcXFx09OhRowvk3/72N2Mahw8flpubm9566y0lJSXpww8/VL169fT4449XTDEsLD4+Xvv27TNCwcjISK1fv17PPfecmjRpoh07duiDDz7QtGnT1KhRI2O88PBwDR8+XHXq1NFnn32mjz76SK6urho0aJBcXV01a9YsLVy40OgOnpaWpnvvvVeDBg2SnZ2doqOj9cEHH+iDDz7I14W3uPVHRSpqHrvxAHzFihUaNmyYxowZIxsbm1LXqzRWrlypoUOHatSoUfryyy81Y8YMzZ49Ww4ODkpMTNT777+vBx54QL1799Yff/yhxYsXFzu91157Tf/4xz/Us2fPAgFGZmam1q5dqzFjxqhatWqaNWuW5s+frzfeeEPS9R3KmTNn6plnnlGLFi2UmJio+fPn69q1a/nWR2b75ptv9Msvv+jll1+Wp6enkpKSil1XRUVFycvLK18gkVdJ68fSrFduNX379tXp06d1+fJl42xWYb3Kylrra9euqUmTJurXr58cHR114MABzZs3Tx4eHsbJj6rm559/1sKFCzV8+HC1adNGv/76qxYsWCA3N7ci55moqCjdc889hV5yYW1tbVxSmJaWpqCgIDVs2FAZGRn68ssvNW3aNH344YeytbXVu+++q9dff12vv/66GjVqZKwQniVoAAAgAElEQVR7fvjhB33++ed66qmn5Ofnp7S0tHz7TdL1g4zBgwdr8ODB2rJliz799FO1bNlSHh4ehbY5LS1N9913n5555hlZWVnp22+/VVhYmD7++GOLXy5xo6tXryo6OloNGjTIt69547o1txdkXiWtt3788UfjhFGLFi20a9curV27tsRekiVtxxcvXqxDhw7pH//4h2rWrKkvv/xShw8fVqdOnSxYmfIrbttRlOXLl2vIkCGqWbOmPv/8c82cOVPTp08vdh1oY2Oj++67T1u3bjVCia1bt+rRRx/V1q1b8w1b0vx+M20vjeK26UeOHNHs2bM1cOBABQQE6ODBg1qxYoUxrpeXl+666y7t2LEjX+i6Y8cOde7cWba2tkpOTtakSZPUs2dPDRs2TFlZWVqxYoXee+89vf3228bxSXHzVlHHA+fPny/171y2bJmefvpp1a1bV46Ojlq5cqV27typkSNHysvLS3FxcZo7d66cnZ3l5+ennTt3av369Xr55ZfVoEEDXbx4UUeOHLmpWt8sKysrPfPMM/L09DROiC5cuNDYHq1cuVInT57UxIkT5erqqoSEBONeM6U9Lizp2KhLly5yc3PTu+++q3fffVceHh6V0gudUKIQNjY2evHFFzV37lxt3rxZjRo10t13363OnTvL19e3TNNq06aNHnroIUnSww8/rG+++UYHDhxQs2bNlJiYKHt7e3Xo0EGOjo6qXbt2vh33L7/8UkOHDjWuC/L09FR8fLy+++47PfTQQ8YMUr169Qrp1ldafn5+at68uVauXFmgy/t3332nmjVratSoUbK2tpaPj4+GDBmiefPmaeDAgXJycpKtra3s7e3z/YaSxstN7D09PYs9KIiJidEff/yhmTNnGgnfSy+9pJdeekkHDhwodQKdmJgof39/4+9j5jXFW7ZsUffu3SVdP3tkb2+v3bt3KyAgoMgzCNWqVTN2XApz9uxZRUdH6/XXXzcS8jp16hif29raGr0vpOu///jx4/rxxx/zhRJOTk56/vnnjb9ZQECAYmJiqnQosW/fPg0bNkzZ2dm6du2aJBnz1Pr169WnTx9169ZN0vUeKIcPH9bXX3+tl156yZjGo48+Kj8/P+Pf06ZN08CBA40dlYceeijfzf8aNWqUb1nv37+/9uzZo507d+qJJ54w3i9u/VGRipvH8nrooYfyvVfaepXGI488Yhz4DB48WP/973914sQJNW/eXN9//708PDw0YsQIWVlZydvbW2fPntWqVauKnJ6zs7Osra3l4OBQYPnIysoydlokqU+fPvr000+Vk5MjKysrrVmzJt9Z7bp162rIkCGaOXOmhg0bVupwsyxy58u8Cgte8zp//rzq1aunFi1ayMrKSh4eHrr77ruLHD53BzCvpUuXGmepJBm9xQpT0nrlVuTg4CA7O7tCz2bl7VlT1lq7u7vn6+VUp04dxcTE6Mcff6yyocT69evVvXt3Yx3k5eWl33//XevWrSsylDh79qxatmxZ4rRvXJe8+OKLGj58uI4eParmzZsb+zc1atTI93f48ssvFRQUpEcffdR478Yz2vfdd59xJnTgwIGKjIzUoUOHiuwtkbuezvXss89q165d2rt3b7E9LMor77Kdnp6uWrVqGT1Cct24bi0slChpvRUZGan7779fDzzwgCTp8ccf18GDB3X27Nli21fcdjwtLU1bt25VSEiIsf80evRojRkzpvwFsbDith1Fybu9fuKJJ/Svf/1LFy5cKPEyjJ49eyo0NFRXrlzR2bNnlZCQoICAgAKhREnz+820vTSK26ZHRkaqdevW6t+/v6Try/mxY8e0ZcsWY/zu3btrw4YNGjx4sKysrJSYmKjY2FgjPPj+++/VsGFDDR061BgnJCREzz77rH7//Xejp3lx81ZRxwNl8eSTT6pt27aSrgdBGzZs0JtvvqkWLVpIur7vevToUX333Xfy8/NTYmKi3Nzc1KZNG9na2srDw6PC72FT2La9d+/eRu0eeeQR431PT08NHTpU7733nsaOHStra2udP39ejRs3Nmpau3ZtY/jSHheW5tgoN5B1cXGptGNMQokiBAQEyM/PT7GxsYqLi9O+ffu0YcMGDRo0yFhwS6Nhw4b5XtesWVMXL16UdP2Ao3bt2goJCVHbtm3Vpk0b+fv7y9HRUX/99ZeSkpI0b948zZ8/3xg/Ozu7xB1TMwwZMkRvvPFGga7lp0+flq+vb760rnnz5srMzNS5c+cK1Kes45XUve7UqVNyd3fPFyLUqVNHNWvW1KlTp0odSgQFBWn+/Pnat2+f7rnnHnXq1KnE764I586dU2xsrHGAZ2VlpW7dumnLli3F3tSmQYMGxR44HD9+XFZWVsVe6/v9999ry5YtOn/+vDIyMpSVlZVvZShJPj4++f5m7u7uOnr0aGl/nilatGihF154QRkZGdq0aZPi4+MVFBSkK1euKDk5ucCBRvPmzbV379587+Wdj11dXSUpXzdHV1dXpaenKz09Xfb29kpLS9Pq1au1Z88epaSkKDMzU9euXcs3zo3TlfKvPypKWeaxvBvvstSrNPL+9tzrGXN/e+76IW8YcDNBTe6N+PJ+X2ZmplJTU+Xs7Kzff/9dR48e1bp164xhcnJyjO7WFXG9Ze58mdfJkyf1wQcfFDlOjx499Pbbb+vll19WmzZt5Ofnp3vvvbdMvej69u2rwMBA7du3TwsXLix22JLWK7ezstY6Oztba9euVXR0tC5cuKBr164pMzOzSt9f4dSpUwUuL2nevLl2795d5Dil3T85d+6cVq1apaNHj+qvv/4y9m0SExOLHOfixYu6cOFCiSFO3vWojY2NXFxcin1qxcWLF7Vq1SodPHhQKSkpys7OVkZGRrFtuRl5l+3Lly/r+++/1zvvvJPvRnelOTAqab115swZI5DI1bRp0xJDieK24+fOnVNWVla+y5kdHBxUv379EttbWYrbdpRmHHd3d2OckkIJHx8fNWzYUD/++KNOnDihrl27Fnq5amnn9/K0vSQlbdNPnz6d79JR6fr2NG8o0bVrV+PSjJYtWyoqKkqenp7G9v7333/X4cOHC32S1blz54z5paL3EfMuN6dOndK1a9f07rvv5hsm775rbiiTexx27733qkOHDhW6XSts2573pvQxMTFas2aNTp8+rStXrig7O1uZmZnGZdW9evXS9OnTdfz4cd1zzz3q0KFDqYLgvCx1bGRphBLFsLOzU5s2bdSmTRsNGDBAc+bMUUREhPr27VvombGsrKwC7934dAMrKytjo+3o6Khp06bp8OHD2r9/v9auXasVK1YoLCzMWGife+65Ys++VBVNmzaVv7+/li5dmu9Mb3HKe3Yx73g3c6+C3OlYW1sX2JHKzMzM9/pvf/ub2rZtq71792r//v1688031a9fv0p/ZNLmzZuVnZ2tF1980Xgvt+3F7UDd7D0doqOj9cUXXxjX2To5Oenbb7/VL7/8km+4wp7mURVDtLzs7e2Ny46effZZ/fvf/9bq1avznYkrSd7fnTtfFfZebi2WLFlipOX16tWTvb29PvnkkwLzXXHrj4pS0jyWtwt0Weer3PVa3t9w42/OVVz9LO3GA8nc78vOzjb+P2DAAHXu3LnAuBXVpTHvfJkrNTW12HGaNGmiWbNm6ddff9WBAwc0a9YsNWzYUG+++WahB8v16tXT6dOn873n4uJS6jMjVfleMRWtrLX++uuvtX79eo0YMUINGjSQg4ODli9fbvFHPFaG4rbdXl5eBeapwkybNk3u7u567rnn5O7uLhsbG73yyitFrg/K4sbLzKysrIxluTCzZs3SxYsXNXz4cNWuXVvVqlXTlClTLNKWwty4bDdp0kTDhw/Xpk2bjJ6HpVm2SlpvldetuB3Pqzzbjpv5zT179tT333+v+Ph449KZG5V2fq+I7V559xvzcnV1VZs2bRQVFWWEErk9InOn165du0J7LueeqJHKV+fSHm9J+Zeb3On+85//LHDpVm47PDw8NGPGDMXExGj//v1avHixVq9erXfeeafCbgBb2LY91/nz5xUWFqYHHnhAAwcOlLOzs44fP66PPvrImFfatWunWbNmad++fTpw4IDCwsLUuXPnfH/fm1ERPT9Li1CiDHx8fIwE3cXFRcnJycZnGRkZOn36dJmvm7axsVHr1q3VunVrBQcHa9SoUfrf//6nwMBA1axZU/Hx8br//vuLHf9mN0CWMnjwYP3973/Xvn37jPe8vb31008/KTs729iAxsbGytbW1rg8wNbWtsBvKM14peHj46MLFy4oISHBSATj4+OVnJxs3GDIxcVFFy9eNLo8StKJEycKTKtWrVoKDAxUYGCg1q5dq2+++aZSQ4msrCxt375dgwcPNi4VyPXJJ59o27Zt5b6bcqNGjZSTk6ODBw/mu8FRrtjYWDVt2tToxitdr+PtaMCAAXr33XeNZfC3337Ld2YuNjb2pu9aHRsbq/vvv9/oeZCRkaH4+HjVq1fvpqZ7s0ozj934+LNcTk5OJdYr9wA+JSXF+LywZa0k3t7e2rVrV75ltjTXgRa2rimNJk2a6PTp00XuSFQljo6OCggIUEBAgHr06KE33nhD586dK3CZhnT97NdHH32kXbt2yd/f34TWVk2lnU/KUuvY2Fi1b9/euBwgJyfHuFFjVeXj46PY2Nh8l+iVtP7r2rWrVqxYoWPHjhU425+dna20tDRlZWXp9OnTGjlypNFl/vfff893oJEbLOT9O7i6usrd3b1Ml16WRmxsrEaMGGGs81JSUvLt31UGa2trZWRkWHSaud3w8/79jh07dlPTrFu3rmxsbHTs2DFjXyw9PV1//vlnmfbNbiddunTR559/Lk9Pz0Iv8b506VKJ83tFKc023dvbu8D2My4ursC0unfvrgULFigwMFAnT57UK6+8YnzWuHFj/fTTT/Lw8Lipmz8Xtu7N3W9ITk42/l2a/QYfHx9Vq1ZN58+fL3CJVl52dnby8/OTn5+f+vXrp+eff16//fabcRlIZTp27JgyMzP1zDPPGMc+//vf/woM5+LiYlym1q5dO3300Ud67rnnVK1atVIdF5bm2MgMhBKFuHTpkqZPn66ePXuqYcOGcnR01LFjx7Ru3Tq1bt1aTk5Oat26tbZu3aoOHTrIxcVFX331VZlXMHv27FF8fLxatGghZ2dnHTx4UFevXjVudBccHKyFCxfKyclJfn5+yszM1PHjx3XhwgXjGn1PT08dOHBALVu2NP0xj3Xr1lVgYKAiIyON93r37q3IyEh99tlnCgoKUkJCgpYtW6aHHnrISDRr166to0ePKiEhQQ4ODnJ2di7VeKVxzz33qGHDhsZN6qTrN3Np3LixsZJq2bKlLl++rDVr1qhLly46dOiQdu3alW86ixYtUrt27VSvXj1dvXpVv/76a6UvuP/73/906dIlPfDAAwVuvtWlSxf98MMP+a7nKwsvLy917txZc+bM0TPPPKMmTZooKSlJ58+f13333ad69epp27Zt2rt3r+rWrasff/xRhw4dui0fK9qqVSv5+Pjoq6++Ut++fRUeHq66desaN248fPiwpk2bdlPfUa9ePf3888/q0KGDbG1tFRERYfEd0vIozTxWXE+okuplZ2cnX19frVu3TnXq1NGVK1e0fPnyMrezV69e2rBhgz7//HP17t1bJ0+e1A8//FDieLVr11ZsbKwuXLhQpscHP/HEE5o2bZpq166tzp07y8bGRn/++aeOHj1a7mWuImzYsEFubm7GjQGjoqLk6OhYZBfkLl266JdfftHHH3+sxx57TPfee6/c3NyUlJSk//73v6aeMTFT7dq1tW/fPp05c0bOzs7Gtc55lbXWXl5eio6OVmxsrGrUqKFvvvlGCQkJaty4cWX8pHLp06ePPvzwQzVp0kRt27bVvn37FBUVpVdffbXIcR555BHt3btXb7/9tp588km1bNlSTk5OOn78uNavX6+nnnpKLVq0UI0aNbR582Z5eHjowoULWrJkSb6zqK6urrKzs9Ovv/6q2rVry87OTk5OTurfv7+++OILubq6ys/PTxkZGTpw4ID69OlT7t9Zr1497dixQ76+vkpLS9OyZcsq9Kk6165dM4LZy5cv69tvv1VaWlqBLvQ3KygoSLNnz9Zdd92lFi1a6Oeff9aRI0duKghzcHBQz549tWzZMtWoUcO40WV2dvYdu75wdHTU3Llzi7x0q3r16iXO7xWlNNv0l19+WW+99ZbWrFlj3Ojyxl6w0vVHzc+bN0+ffvqp7rrrrnzha+/evbV582bNmDFDjz32mFxcXBQfH6+ffvpJTz/9tBwdHUvV3sKOB+rWratatWopIiJCgwcP1vnz5/XVV1+VOC1HR0f16dNHS5YsUU5Ojlq2bKm0tDTFxcXJ2tpagYGB2rZtm7KysuTr6ysHBwdFR0fLxsamQk8Q5V3+c1lbW8vFxUX16tVTTk6ONm7cKH9/f8XFxWnjxo35hl21apUaN26s+vXrKysrS7t27ZKnp6dxyUlpjgtLc2xkBkKJQjg4OMjX11fffPONzp07p2vXrsnd3V3dunUzdsj79eunhIQEvffee3JwcFD//v3LnKxXr15dv/zyi1avXq309HTVrVtXo0ePNm7I8sADD8je3l7r16/XihUrZGdnJx8fn3xnq4cNG6bFixdrzJgxcnd3r/RHgt5owIAB2r59u3HDQHd3d4WGhmrp0qWaMGGCqlevrq5du+qpp54yxunTp49mzZqlV155RRkZGcZjZ0oarzSsrKw0YcIELVy4UP/+978lXV8Yn332WWMD6uPjo1GjRmnNmjVas2aN2rdvr8cffzzf3YdzcnK0cOFCJSUlycHBQffcc0+l33V/y5YtatWqVaF3A+/cubOWL19+U938QkJCtGrVKi1atEiXLl1SrVq1jBvuPPjggzpx4oQ+/vhj5eTkyN/fX3369ClwM6fbRZ8+fTR79mx99NFHunr1qpYtW6aUlBR5eXnp1VdfLXOPqBsNHz5cc+bM0aRJk1S9enUFBQUZy4yZSjOP7d+/v8gN9sMPP1xivcaMGaO5c+cqNDRUderU0ahRo4y7P5eWh4eHXnvtNX3xxRfatGmTmjRposGDB2vmzJnFjhccHKz58+dr3LhxunbtWr4bFxbn3nvv1cSJE/Xll19q/fr1xk5Ljx49ytTuiubg4GA8KtHKykqNGjXS66+/XmSQa2VlpZdfflmbN2/W1q1btWHDBmN717p165sO325VgYGBOnTokCZOnKi0tLRCHwla1lr3799fCQkJevfdd2VnZ6cePXqoe/fuOnXqVGX8pHLp1KmTRowYofXr1+uLL76Qh4eHRo4cWeRNLqXr9zl48803tXHjRm3dulXLli2TnZ2dvLy81KNHD919992ytrbW3//+dy1atEivvvqq6tatq2HDhuV7LLeNjY1GjBih1atXKyIiQi1atNDkyZPVq1cv2draav369Vq2bJmcnZ3Vrl27m/qdY8aM0bx58/TPf/5T7u7uevLJJyv0spoDBw7o+eef/3/t3XlYVdXi8PEvM5ogEDKDqIA445CamFylxOEtu4mzlOZwVdSbaSXXvIITiSVWTpiFJRKoaY7liKIYDiUOiAxXkUEZFEFQGQ6c9w+es38cDygoeA62Ps/j83j22XuftRdrr7X22msAKh+cbGxsmDNnDh06dCAnJ6fefsfd3Z3s7GzCw8MpKSmhV69evPXWW0+cE6Q23n//fb777jup/jt06FAKCgr+tvPLgPKcAI+rTXqvLV9fX9q3b1/rpVdrU6Y/evSIadOmScvKd+jQgREjRqjMKWRgYEDPnj2Jjo6WHmQVzMzMWLJkCeHh4SxfvpzS0lLMzc3p0qVLndJFTc8DH330EZs2beKTTz7B0dGRMWPG8MUXXzz1fKNGjaJ58+bs3buXTZs20aRJExwdHRk2bBhQ+XfbvXs3W7Zsoby8HDs7O+bNm9egk9lXvf8VzMzM2LBhAy1btmTChAns3r2biIgI2rZti4+PD6tXr5b21dPTIyIigpycHPT09HBxceGzzz6Tvq/Nc2Ftno3UQUvemAaKCYIgCIIgCIJQZytXrqS8vFxanro+lJWVMWPGDN55553n6rEiPFlJSQkffvgh06dPV5rPQRBeFqKnhCAIgiAIgiC8REpKSjh06BBubm7o6OgQGxvL+fPnnzj8pjZu3LhBZmYmTk5OPHr0iN27d1NcXEyfPn3qKeRCdeLj43F2dhYNEsJLS/SUEARBEARBEISXSGlpKStWrODGjRuUlpZibW3NsGHDnvuh9saNG4SEhHDr1i10dHRwdHTEx8dHLcukC4Lw8hCNEoIgCIIgCIIgCIIgqEX1U8UKgiAIgiAIgiAIgiA0MNEo0QBGjhxJbGzsc51j7ty5tZ4ZvrEoKipiypQpZGVlqTsoKlatWsXevXvVHYyXXn3cG5pk7dq1SjNAP/65Ol988YXaV8lpbI4fP46Pj88T99mzZ89TZyR/2dJfTXx9fdmzZ88T9/Hx8eH48eMvJkAvudjYWEaOHFkv58rMzGTBggWMGzeu1jPsQ+NJ2zk5OYwcOZL//e9/z7VPdbZt28aUKVMYOXKkSNvPqTGkp2dNJ7V1/PhxaWUCTVNQUMCkSZO4e/fuC//ta9euMW/ePMaMGYO/v3+D/U5D5hUvi5etHBcTXdbR2rVrOXHihPTZyMgIZ2dnfHx8sLW1VWPINN+uXbvo2rUrVlZWAISGhpKYmEh6ejomJibVPqidPn2aXbt2cfv2bYyNjRk0aBDvvPNOtee/du0a/v7+2Nraqiy19PDhQyIiIjhz5oy03OWYMWOkiZm8vb1ZtGgRnp6eT1zaqT7k5eWxfft2Lly4QEFBAcbGxnTt2pURI0bUuMa9pvP19SU3N7fG79u3b9+ghVddrV27lsLCQpUZyP/3v//h5+cnLUNVVxMnTnyuZVkbi8OHD/PTTz8RGhqKrm5lMSKTyZgwYQKWlpZK919WVhazZ89m4cKFdOrUSV1BBmDjxo288sorag3Ds3i83FEICgqqdnnawMDAGpemfFk01jT4NBERERgYGBAcHIyhoaG6g1MnT2uY8fDwYMSIEU89j7m5ORs3bqx2GcOapKWlsWPHDubNm4eLi0uDl+PPyt/fH3t7eyZNmqS0/fjx43z//fds2bLlmc+dk5PDzJkzpc+6urq0aNECT0/PGutNmqq+0tKzkslkREREMHv2bGlbeno627Zt48aNG+Tk5ODt7a0SzkePHhEZGcnZs2cpKCigVatWTJgwAScnJ2mfiIgIYmNjuXv3Lrq6urRq1YpRo0bRtm1blXDI5XICAwOJi4vj448/pnfv3gA0b94cDw8Ptm3bxvTp0xsoFqq3efNmWrZsyfz582vMo6qWWdra2piamtKtWzfGjBlDs2bN6i0sz5JXaKqaynlnZ2eWLVumhhC9eKJR4hl06tSJWbNmAZUPmGFhYXz55ZcEBwerOWSaq6SkhGPHjimtpSuXy/Hw8CAtLY1Lly6pHHPhwgW++eYbJk6ciJubG5mZmYSEhKCvr8+gQYOU9i0qKmLNmjV06tSJvLw8pe9kMhlLly6lWbNmzJkzBzMzM/Ly8qSKLICDgwOWlpZER0ernLs+5eTk8Pnnn2NhYYGvry/W1tZkZWURERGBn58fS5cubdD1kRtKYGAgFRUVAKSmprJ8+XKWL1+Oubk5gFJcv8w0tSJc3zp06EBJSQkpKSm4uroCkJycTNOmTbl9+zb379/H2NgYgCtXrqCnp1dthas2ZDJZvYXbxMSk3s71olUtdxQer4jJZDJ0dXWluH+Zvcg0+CIo/nZZWVm89tprjbIc2Lhxo/T/P//8k5CQEKVt+vr6FBUVPfU82tradb5XFT0wX3vtNbS0tOp07MvmP//5D46OjpSVlXHlyhU2btyIubl5o1odo77S0rOKjY1FX1+f9u3bS9tKSkpo0aIFvXr1IiIiotrjNmzYQFpaGr6+vrz66qtER0ezZMkSgoODMTMzA8DGxoZJkyZhYWFBaWkp+/fvZ/ny5Xz99dcq6X7v3r01pud//OMf+Pn54ePjU68P+k+TlZWFl5eXVL+riaLMKi8vJyMjg/Xr1/PgwQM++uijegvLs+QVmqy6cv7vUn8G0SjxTPT09KSbwMTEhKFDh7JixQpKS0vR19dX2vfzzz/HxcWF999/X9r28OFDpkyZwuzZs+nVqxcFBQWEhIRw8eJFmjdvjre39wu9nhfhwoULAEqVwg8//BCo7HZdXaNEdHQ03bt3x8vLCwBLS0veffdddu/ejZeXl1JGvWHDBjw8PJDL5Zw5c0bpPMePH+f+/fssXrxYurmrq/D16NGDmJiYBm2U+P7779HS0mLhwoXSm0xzc3MWLlzI7Nmz+f777/Hz8wOqf6Py+Bt+uVzOnj17OHLkCHl5eVhZWTFs2DD69esnHZOXl8dPP/3ExYsXAXBxcWHChAlYW1sDlV1ez5w5w3vvvUdERAQFBQV07NiRadOm1frhpup+igclY2PjaguLoqIiVq1axYULF2jevDkjR46sU3hfpKtXrxIWFsbNmzdp2rQp7u7ujB8/vsZC4vG/T0lJCZs2bSI2NhZDQ0MGDx6sckx0dDS//fYbmZmZUiVowoQJmJmZIZfLmT17Nm+99ZbSm67bt2/z73//my+++EItM57b2NhgamrKlStXpAfC+Ph4OnbsSG5uLvHx8bz++uvSdhcXF/T19SkrK2Pr1q3ExMTw8OFDadb2qucICAhg/vz5bN++ndTUVObNm1dtGHbv3s2+ffsoLi6mV69etXqIGzlypPS2SfFW8eOPP+bw4cMkJibSokULJk6cSOfOnesppupP1XJHQdEzzMDAgBMnTmBhYUFgYCC+vr54eXlJaSYrK4sNGzaQnJyMubm5UnmksHXrVs6ePTlWV3wAACAASURBVMudO3cwMTHh9ddfZ+TIkejr65OTk8OsWbNYvnw5bdq0kY45cuQIP//8MyEhIS+84vQsaVBLS4vNmzc/Nf0tXLiQn3/+mbS0NOzs7Jg6darSfXbixAkiIyO5f/8+HTt2xM3NTSV858+fZ/v27WRkZGBiYkLfvn0ZMWKEFE++vr54eHhw584dzp49S+fOnaXu8jdv3mTHjh14e3vzj3/8g5kzZxIYGKgU91XTsqaomj4VPZIeT7OKB8nc3FzCw8Orve8U92bVa87IyGDLli0kJCSgr69Px44dmTBhAiYmJmzbto0dO3YAMGrUKIBGP/xVUZY4Ozvz+++/U1xczOuvv87kyZNV6pmPMzIykuK9f//+HDx4kOvXr0uNEikpKURERHDjxg1kMhkODg74+Pjg4uKidJ78/HwCAwOJj4/H2NiY0aNHS2V1QEAAdnZ2SvWThw8fMnXqVGbNmkWvXr2e6/rrKy3Bk9NOTU6dOkW3bt2Utjk5OUk9Hnbt2qVyTGlpKWfOnGHu3Ll06NABqLxP//zzTw4dOsTo0aMBlOo7AO+//z7Hjh0jNTVVKS9JSUnht99+44svvmDKlCkqv+fg4ICpqSlnzpzB09OzxmupiyeV0VV74qxfv57169czY8YM/vGPf1R7rqpl1quvvkqfPn2UhhpUVFSwc+dOjh49SkFBAdbW1owePZrXXntN6Ty3b99m8+bNXL9+Xfr7dunSBVDNK2qbh2uq6sp5hdqU4y+ivt+QxJwSz+nRo0ecPn0aBweHaguKN954g5iYGOktMsCZM2fQ19eXMrx169aRlZXFwoUL+eSTT4iOjiYnJ+eFXcOLkJCQQOvWrev0BqOsrAw9PT2lbfr6+ty9e1dpqMDBgwcpKChg+PDh1Z7n3LlztG3blh9++IEpU6YwZ84ctm3bpvIG1snJiZSUFEpLS+twZbVXVFREXFwcXl5eKl2rDQwM8PLyIi4urk6t/xERERw7doxJkyYRHBzMP//5T7777jv++usvoPKhOCAgAD09Pfz9/Vm6dCmmpqYsWbKEkpIS6Tw5OTmcPn2aefPm8fnnn5Oamlrjm4DntWPHDnr06MHKlSvp06cP69ev586dO3UK74uQl5dHYGAgjo6OrFixgmnTphETE0N4eHitz7FlyxYuXbrE3LlzWbhwIampqSQkJCjtI5PJGDFiBCtXrmT+/PkUFhby9ddfA6ClpcWAAQNUxgxGRUXh6Oio1kK2Q4cOxMfHS5/j4+Pp0KED7du3V9p+9epVqYIWFhbG6dOnmT59OitWrMDe3p5ly5Zx7949pXNv3bqV0aNHs3r1apydnVV++/Tp00RERDBy5EhWrFiBjY0N+/fvf6briIiIYPDgwaxcuZI2bdqwevVqiouLn+lc6nDy5EkAFi9eXO0cBBUVFaxcuRK5XM7SpUuZPn0627dvV8n/DAwMmD59OsHBwUyaNImYmBh27twJVDbidu7cmaioKKVjoqKieOONN9T2JqeuabC26S88PJyxY8eyYsUKjIyM+Pbbb6VhWcnJyaxbt44333yToKAgunfvrvIAHBcXx7fffsugQYP46quvmD59OrGxsSp5x/79+7G1teWLL75gzJgxbNy4ERsbG/7f//t/bNy4sdF1ua+Lutx39+7dY9GiRdjb27N8+XIWLlxIcXExQUFBVFRU8M477/Cvf/0LqHzDXvWNemN29epVbt68ycKFC5k7dy4XL14kLCys1sfL5XKuXbtGZmamUj5aXFxMv379CAgIYPny5Tg6OhIYGEhhYaHS8du2baNHjx4EBQXx5ptvsnbtWmnsvqenJ6dOnaKsrEzaPyYmBkNDQ7p37/6cV143T0pLT0s7Nbl27ZpSI2BtlJeXU1FRUW299dq1a9UeI5PJOHLkCE2aNFEaivfo0SO++eYbpk6dSvPmzWv8TScnJ65evVqncD7Jk/JIxVAJAwMDJkyYwMaNG2vd+yY7O5u4uDh0dHSkbQcOHGDv3r2MGzeOL7/8kp49e/Lll1+SmpqqEqbBgwcTFBRE586dCQoKUukR/bgn5eGNUW3KcU2s79eVaJR4BnFxcfj4+ODj48MHH3zA1atXlcadVdWnTx/u37+vVEE6deoUvXv3Rk9Pj1u3bnHhwgWmTp2Kq6srrVq1wtfXt8EejNUlNzcXU1PTOh3j5ubG+fPnuXjxIhUVFdy6dYt9+/YBlS348H/jSGfNmoW2dvXJOTs7m9jYWGQyGX5+fowaNYrDhw+rVBBNTU0pLy9/amb3rG7fvo1cLsfOzq7a7+3s7JDL5bWeCLS4uJh9+/Yxbdo03NzcsLCwoG/fvnh6enLw4EGgspIgl8uZMWMGLVu2xNbWlqlTp1JcXMyff/4pnauiogJfX19atmyJi4sLb775JpcvX37+i65Gv3796NevH1ZWVowaNQodHR2pUK1teOtD1ftY8W/RokXS9wcPHsTU1JTJkydjZ2dH9+7dGTduHL///nutGkiKi4s5duwY48ePx83NDQcHB2bMmKHSMDdgwAC6deuGpaUlTk5OTJ48mYSEBGkCq/79+3P79m2SkpKAyr/ViRMnGDBgQD3GRt117NiRpKQkysrKKC0tJSkpSeWBMDMzk3v37tGxY0eKi4s5dOgQ48aNo1u3btLbCxMTEym9KowYMYIuXbpgaWlZbev9gQMH8PDw4K233sLGxob33ntPacxuXQwdOpQePXpgbW3N2LFjKSoqUqkUaYLH0+vy5cuBygaD999/H1tb22rzlsuXL5ORkcGsWbNo1aoVrq6uTJgwgfLycqX9vL29cXV1xcLCgm7duvHPf/6TmJgY6XtPT09iYmKksikjI4Pk5GS1psO6pMH27dvXOv2NGjWKjh07Ymtry/Dhw8nMzJTKhQMHDtCxY0fee+89bGxseOutt+jZs6fS8bt27eLtt9+mf//+WFlZ0bFjR8aNG8fhw4eVKsbt2rVj2LBhWFlZYW1tjYmJCTo6OhgaGmJiYtLo5pSoi7rcd4cOHaJly5aMHz8eOzs7WrZsycyZM0lJSeH69esYGhoqvU1/Wbpza2trM2PGDBwcHHBzc2PcuHEcOXLkqY2mixYtwsfHh7Fjx/Lf//4XT09PpZ4LHTt2pF+/ftjZ2WFra8uHH36Inp6e1KNVoWfPnkp5bMeOHaXG3169eqGtrc3Zs2el/aOioujXr98Lb6R8Ulp6WtqpzoMHD3j48GGd66xNmjTBxcWFnTt3kpeXR0VFBdHR0SQlJak0fP7555/4+Pgwbtw49u/fz8KFC5XS7XfffYebmxtdu3Z94m+ampo+cT6vunhaGV11qETTpk0xMTF5Yq8dRZk1btw4Zs2aRUZGBsOGDZO+37t3L2+//TZ9+/bFxsaGUaNG0a5dO5VJmgcOHEifPn2wtbVlwoQJmJubc+jQoSdey5PycE1WXb00LCysVuW4Jtb360oM33gG7dq1k1rli4qKOHToEMuWLWPZsmUqY6yMjIxwc3Pj5MmT0nwHV65ckR5+MjMz0dLSUqpQt2jRQhp79rIoKyt7apfDx3l6epKVlUVQUBDl5eU0adKEIUOGsH37drS0tCgrKyM4OBgfH58ndt2Wy+UYGxszbdo0tLW1ad26NUVFRfz444/4+PhID4mK8Km7Qai2BXpGRgZlZWXSw4lCeXk5LVq0AOD69evk5OSodPMqLS0lOztb+mxubq40H4KpqSn3799/1kt4IgcHB+n/Ojo6GBsbS79V2/DWh6r3sUJaWhpffvklgPR2qWpjl6urKzKZjKysLFq2bPnE82dlZSGTyZS6xBoaGipdP1Re844dO0hNTaWoqEh6aLlz5w6vvvoqJiYmdOvWjaioKFxcXKTeNH379n2u639eHTt2pKysjKSkJOkes7KywsTEhKysLPLz84mPj8fAwAAnJycyMzMpLy9XGsKlra2Ns7MzGRkZSud+2huqzMxMlYdhZ2fnZ1rZp+rfUVEJLSgoqPN5Gtrj6VVfX59vvvnmqb1lMjMzMTMzUyqbnJycVBrHYmNj2b9/P1lZWRQXF1NRUaH0JrFHjx58//33nD17lr59+xIVFYWTk5NKen6R6pIGmzZtWuv0VzVNKMrigoICXn31VTIzM1XeBLu4uHDs2DHp8/Xr10lJSWH37t3SNrlcTmlpKfn5+VI6q+ub2JdJXe6769evk5CQUO0KPFlZWc/cIKnpWrZsqdQw5eLigkwmIzs7+4nlz+zZs3FwcEAmk5Gens4PP/yAoaGhNHygoKCAyMhI4uPjyc/Pp6KigtLSUqnHYtXfq8rZ2VlquNDT0+ONN94gKioKd3d30tPTSUlJYcaMGfV1+bX2pLT0LGlHUQesa50VYObMmaxfv16qb7Zq1Qp3d3du3LihtF+HDh1YuXIl9+/f5+jRowQHB0tvtqOjo7l58yaBgYFP/T19ff16q7NmZ2fXOo+sDUWZVVpaypEjR8jOzmbIkCFA5VCfe/fuqczz4+rqqtI4VjUdamtr4+Tk9NTwPCkP12TV1UubNm3KqVOnnlqOa2J9v65Eo8QzMDAwkFaQAGjdujUffPABR44ckTL9qt544w1CQkKYPHkyp0+fxtzcnHbt2int87JPzGRkZFTnSYm0tLQYP348Y8eOJT8/H2NjY6k1z9LSknv37pGZmcm6detYt24dUFnxk8vljB49Gj8/P7p06YKJiQm6urpKD5e2traUlJRQWFgovYlVhK+hxlVZW1ujpaVFRkaGyps1qGxk0NHRkRpYtLS0VLqbVW0VVXz32WefqTSGKbrIyeVyHB0dq51YqOrESI83hGhpaT2xa+PzeNJv1Ta89eHx+xgq35DURn3dr8XFxSxbtoxOnToxc+ZMmjdvTmFhIf/973+VuuV5enry9ddfM2HCBI4dO0bPnj1f6MRW1bGwsKBFixbSG2lFnmZoaEjr1q2Jj48nPj4eV1fXOr85e5ErR1TtTqr4u2piN8/q0qti+/NKSkpi9erVeHt788EHH/DKK69w/vx5pZUAdHV16devH1FRUbz++utER0dL4/fVpS5psC73bNU0oVCXNFFRUYG3t7c0p0VVVcuX2vztFOVW1d+vz8lf1aUu951cLqdr167VjqF+Utd2TdSkSRMePnyosv3Bgwf1NlHyq6++KuUVdnZ2ZGdnExkZyXvvvYe+vj5r166loKCADz74gBYtWqCnp8fixYvrnK48PT2ZN28ed+7ckRrNa+oJ2pCelJaeJe0YGRmhpaX1TBNpWllZERAQQHFxMY8ePcLU1JTg4GCVF2eGhoZYWVlhZWWFi4sLs2fP5ujRo3h7e0tvxR8Pc3BwMC4uLixZskTaVlRUpBFzAVSnapn14YcfEhAQwI4dO+pt+eQned48XF1qKudrQxPr+3UlGiXqiba2do2tlT169CAkJIS//vqLkydP4u7uLmWctra2yOVyUlJSpBbDO3fuNIpuRnXh6OhY7VI3taGtrS21dMbExODi4oKxsTFNmzaV3morHDp0iEuXLjFv3jypEGjbtq00r4eignf79m0MDAyUZq9PT0/HzMyswbp+NmvWDDc3Nw4ePMjQoUOVKqQlJSUcPHiQ1157TaqYGBsbS8NUFG7evCn1grCzs0NPT4/c3Fw6duxY7W+2atWKmJgYjIyMGsVSiJoUXltbW/744w+ldHPt2jV0dXWxtLR86vFWVlbo6OiQnJws7V9cXEx6err0+datWxQWFjJ27FgpvT4+UStUDmVq2rQphw8f5s8//5QmQ1W3qmP6q07e1aFDB65cucLVq1cZOnQoUNmQqKurS2JiolToVlRUkJycjLu7e51+19bWVmXoQHJy8vNezkvJ1taWvLw87ty5IzVepqSkKFXQEhMTMTMzU5pkubouwZ6ensyZM4eDBw9SXFysEbP51zYN1lf6U6S9qhRDqxRat25NZmbmM1cuq1I8cFQtCzRxeFFDatWqFX/88Qfm5uaNfiZ6GxsbLly4gFwuV2oou3HjBjY2Nkr7pqWlUVxcLPWWSE5OrnX5U5W2tjbl5eXIZDJpfoOJEydK85rl5+erDC9Q/N7jeaytra302d7eHmdnZ44cOcLJkyerfSmnbs+SdnR1dbGzsyMjI0NlssvaMjQ0xNDQkKKiIi5evMj48eOfuL9cLpcahcaMGcPbb7+t9P28efPw8fFRmQQyPT1d5SXns6rPMro63t7eLF++nDfffBMzMzNMTU1JTExUWqb52rVrKg1bSUlJUh1X8bykSRP8vgi1Kcc1qf78rMScEs+grKyM/Px88vPzycjI4IcffqC4uLjGyX309fXp1asXv/zyCzdu3FCqONnY2ODm5sbGjRtJSkoiNTWVtWvXqnQbW7NmDWvWrGnQ62pIbm5uZGRkKE2klJWVRWpqKvfu3UMmk5GamkpqaqqUMd+/f59Dhw6RkZFBamoqoaGh/PHHH0yYMAGoLDgcHByU/hkbG6Onp4eDg4NUkA8cOJCioiI2b97MrVu3iIuLY9u2bQwcOFCpUpCQkCDN6NtQJk2aREVFBUuWLOHKlSvcuXOH+Ph4li5dio6OjrQiCVR2Tb5w4QLnz5/n1q1b/Pjjj0rdK5s0acLbb7/Nli1bOHbsmBSfhw4d4siRI0BlL53mzZsTFBTE1atXycnJ4erVq/z000/cvn27Qa/1WWhSeL28vLh37x6bNm0iIyODv/76i61btzJo0KBaveE0NDRkwIABbN26lUuXLpGens769euVWqTNzc3R09Pj999/Jzs7m7/++ovIyEiVc2lra9O/f3/Cw8MxMzNTKsTVqUOHDiQnJ5OcnCxNZgnQvn17Tp8+Lc3sDJXxMXDgQLZu3cpff/1FRkYG3333Hfn5+dIKO7U1ZMgQTpw4wZEjR7h9+za7du0iJSWlXq/tZdGpUydsbW1Zu3YtqampJCUl8eOPPyq9SbK2tiYvL4+TJ0+SnZ3NoUOHlOaTULCxscHV1ZWwsDB69eqlEUvg1jYN1lf6Gzx4MJcvX2bXrl3cvn2bI0eOcO7cOaV9hg8fTkxMDJGRkaSlpZGZmUlsbGydJilU0NfXx9nZmd27d5Oenk5iYqJSD5a/Ay8vLx4+fMjq1atJTk4mOzubS5cuERISwqNHj9QdvDoZOHAg2dnZ/PDDD6SmpkpzZcXExKhMbFpeXs769etJT0/n0qVLhIeH4+np+dS5RgoLC8nPz+fu3btcuHCBAwcO0KFDB+l+tba25uTJk2RkZJCSksLXX39d7QP72bNnlfLYK1euSN3vFTw9PdmzZ4/GNFI+7lnTTpcuXVQmp6xaT1UMxUpNTVUaNhgXF8eFCxfIycnh0qVLBAQEYGtrK61Q8fDhQyIiIkhOTubOnTtcv36ddevWcffuXalnlZmZmUrdFirrC1UbpEpKSrh+/Xq1q/88i/oso6vToUMH7OzspAmU33nnHfbu3cupU6e4desWkZGRJCQkqDTIHD58mNjYWG7dusXmzZu5c+cOAwcOfO7waKKqz5eKf/fv369VOa5J9edn1bibnNXk8uXLTJ06Fah8MLSxsWHOnDlKFaLH9evXj+PHj9OqVSuVVsAZM2YQEhJCQEAAxsbGeHt7q4zveXysX2Pj4OCAk5OT0pKbGzZsUJo1+NNPPwUqG2AUb41PnDghVcBcXFzw9/ev8/hRc3NzFixYwE8//cQnn3yCiYkJ/fv3V1qto7S0lLNnz7JgwYLnus6nsbCw4IsvvmD79u18++235OfnI5fLcXV1JSgoSKmLVf/+/bl58ybr168HKgvXnj17KjXsjBo1iubNm7N37142bdokzeCsmEzIwMCAgIAAwsPDWbVqlTR5U4cOHerUkqpYZmnRokVPTOfPq77CWx/MzMzw8/MjLCyMTz/9lFdeeQV3d3fGjBlT63P4+PhQUlLCypUrMTAwYNCgQUqTZBobG+Pr68vPP//MwYMHcXBw4P3331eZJwQq08OOHTvo37+/xgz36tChAzKZTKm7MFSOCy0tLaVJkyZKcx6MGzcOQFqvvFWrVixYsKDOE4r16dOH7OxsIiIiKCkpoUePHgwdOvSZe2O9zLS1tZk3bx4hISH85z//kZYSU6zwApW9+d555x02b95MaWkpXbp0YdSoUWzatEnlfAMGDCAhIUHtE60q1CUN1kf6c3FxYdq0adIylB06dGDEiBH88MMP0j5ubm7Mnz+fX375hb1796Kjo4O1tXWNS+c9zfTp0wkJCcHPzw9LS0smT56sNCnvy87MzIwlS5YQHh7O8uXLKS0txdzcnC5duqisdKDpLC0tCQgIIDIykmXLllFaWoqtrS1z5sxRmdSwffv22NvbExAQQElJCb169XrqG3dAKj+0tbUxNTWla9euSuXW9OnT2bhxI5999hlmZmaMGDGi2jHlI0aM4MyZM4SGhmJsbMz06dNV6l99+vQhNDSU3r1706RJk2eJkgb1rGnH09OTTz/9lKKiIqlelpeXJ9VToXIOhiNHjtC+fXv8/f2BykaHn3/+mbt379KsWTN69erFmDFjpEYfHR0d0tPTiYqKorCwECMjI9q0aUNAQMBT56l63Llz56odDv486quMrsnbb7/NunXrGDZsGIMHD+bRo0ds3bqV/Px8bGxsmDt3rtIqJABjx45l37593LhxA3Nzc+bNm6fxc0M8q6rPlwpmZmZs2LDhqeW4JtWfn5WWvDEMshFeCnFxcYSGhhIcHFzjShnq8vvvv3P+/Hk+//xztfz2Tz/9xMcff0yPHj1e+O/XRlRUFOHh4axevbrRZG4vm+TkZBYuXMiaNWtU5hARhBfl119/JSoqSqkyJAhC/Vq7di2FhYXMnz9f3UF5ory8PGbMmIG/vz+urq7qDk69Wr16NXZ2dkrD2jSJn58fQ4cOVfuk14JQXzTryVB4qbm5ueHl5SUtdahJdHV1lYZOvEiDBg1i5syZpKenq33lj5pcuHCBcePGiQYJNSgrK+Pu3btERkbSs2dP0SAhqIViPpTffvuNwYMHqzs4giCokUwmIz8/n59//llaovBlM378eI0YoladgoICevfuXS9zPQiCphA9JQRBEDTY8ePHWb9+PY6OjnzyySeiUUJQi7Vr1xITE0OPHj3497//Xe3s5oIg1A9N7ymhGNJpbW3NnDlzVLrcC4Ig1JVolBAEQRAEQRAEQRAEQS3E8A1BEARBEARBEARBENRCNEo8QVFREVOmTFFa7kdT+Pn5ERsbq+5gCGqiyWlz1apV7N27V93BqBMRn4Km0uS0+Xcth3x9fdmzZ4/0OT8/n6VLl+Lj48PIkSPVGLIXT5PTZ2PLO0VcCn8H//vf/xg5ciQ5OTn1el5/f3++//77ej1nQxL3uyqxJOgT7Nq1i65du0rLjIWGhpKYmEh6ejomJiasXbtW5ZjTp09L65cbGxszaNAgpbWnFePwHhccHIytra30OTY2lsjISLKzs7G0tGTMmDH07NlT+n748OH89NNP9OzZU+NWshAanrrSZnp6Otu2bePGjRvk5OTg7e2tUgn39vZm0aJFeHp6auwkUY9TV3z+8ccf7N69m6ysLMrLy7GysmLo0KFKSwdqanxWVFSwbds2Tp48SX5+PiYmJrzxxhuMGDFCmm9ALpezfft2jh49SlFREc7OzkyaNAl7e/saz6tYahEql7Rr0qQJtra2dO/encGDB2NoaPhCrk9TNETarOratWv4+/tja2vLV199pfTd37kcetKY/sDAQAwMDKTPe/bs4d69ewQFBWnksogNSV155/Hjx1m3bp3KPmFhYejr6wOam3fWpCHi8urVq4SHh3Pr1i1KSkpo0aIFAwYMUMkPDhw4wKFDh8jNzcXIyIgePXowfvx4Kb9tbHEp1KymvO1///sffn5+rFmzBgsLCzWFTtXatWufutz3tm3bXlBo6o/IO1WJRokalJSUcOzYMT777DNpm1wux8PDg7S0NC5duqRyzIULF/jmm2+YOHEibm5uZGZmEhISgr6+PoMGDVLad9WqVdLaxwDGxsbS/5OSkli9ejUjR46kZ8+enD17llWrVrFkyRKcnZ0B6NatGyEhIcTFxdGtW7f6vnxBg6kzbSoqNb169SIiIqLa8Dk4OGBpaUl0dLTKuTWROuPTyMiI9957D1tbW3R0dPjrr7/YsGEDxsbG0n2tqfH566+/cvDgQXx9fXFwcCAtLY21a9eiq6srLaG2e/du9u3bx4wZM7CxsWHHjh0sXbqU1atXP/HhzcbGBn9/f+RyOUVFRVy7dk1ainLx4sWYmJi8qMtUq4ZOm0VFRaxZs4ZOnTqRl5en9J0oh2pW9R4GyMrKolWrVlhbW6spROqhzrwTwMDAgG+//VZpm6JSDZqbd1anoeLS0NCQwYMH4+DggIGBAdeuXeO7777DwMAALy8vAE6dOkVYWBjTpk3D1dWVnJwc1q9fT1lZGdOnTwcaV1wKL5eJEycybtw46fOsWbMYM2YMffr0UWOono/IO6snGiVqcOHCBQDatm0rbVMsGblnz55qE0x0dDTdu3eXMnpLS0veffdddu/ejZeXF1paWtK+xsbGKolEYf/+/XTo0IH33nsPADs7O+Lj49m/fz8fffQRUPkGsWvXrpw6depvVRkU1Js2nZyccHJyAipbeWvSo0cPYmJiGkXlRZ3x2bFjR6XPQ4YM4cSJE1y7dk3pvtbE+ExKSqJ79+706NEDAAsLC7p3705KSgpQWcAeOHCAd999l969ewMwc+ZMJk+ezKlTp3jrrbdqPLeOjo7U8GBqaoq9vT09evRg7ty5hIWFMXPmTADi4uLYuXMn6enpQGX6/OCDD7CzswMgICAAOzs7Jk2aJJ374cOHTJ06lVmzZtGrVy/OnDnD9u3buX37Nvr6+jg4ODBnzhyNaPho6LS5YcMGPDw8kMvlnDlzRuk8ohyqma+vL15eXrzzzjv4+vqSm5sLVMa9h4cHvr6+PHz4kC1bHeumAQAAF5tJREFUtnDu3DlKS0tp1aoV77//Pm3atFFz6OuPOvNOhafdp5qYd1anoeKydevWtG7dWjrGwsKCs2fPkpCQIB2XmJiIs7Mz/fr1k/bx8PBQyRMaS1wK9efq1auEhYVx8+ZNmjZtiru7O+PHj0dXt/LxsaysjK1btxITE8PDhw9xdHTEx8dHaYnYuLg4Nm/eTG5uLm3atGHgwIF1CkPTpk1V3tY3bdq02ntfLpcTHh7O0aNH0dLSol+/fowfP17qySeTyYiIiODUqVMUFRVhb2/PqFGjcHNzq2vUPBeRd1bv5epvWY8SEhJo3bq10h/5acrKytDT01Papq+vz927d6VKi4Kfnx9Tp05l8eLFXLlyRem7pKQkunTporStS5cuJCUlKW1zcnIiISGh1uETXg7qTJu15eTkREpKCqWlpc90/IukKfEpl8u5fPkyt27dol27dkrfaWJ8urq6Eh8fT2ZmJgAZGRnEx8fTtWtXAHJycsjPz6dz587SMfr6+rRr147ExMQ6/56pqSl9+/bl3LlzVFRUAFBcXMyQIUNYvnw5/v7+NGnShBUrViCTyQDw9PTk1KlTlJWVSeeJiYnB0NCQ7t27k5+fz+rVq/Hw8CA4OJiAgACpYq4JGjJtHjx4kIKCAoYPH17teUQ5VDuBgYF06tSJ119/nY0bNzJx4kTkcjmBgYHk5eUxf/58goKCaNeuHYsXL+bevXvqDnK9UXfeWVpayowZM5g2bRpffPEFN27cUNlHE/PO6jR0XCrcuHGDxMRE2rdvL21zdXUlNTVVurfv3LnD+fPnpbxcobHEpVA/8vLyCAwMxNHRkRUrVjBt2jRiYmIIDw+X9gkLC+P06dNMnz6dFStWYG9vz7Jly6R87s6dO6xcuZLOnTsTFBTEoEGDCAsLa7Awnzx5Eh0dHZYsWcKHH37IgQMHOH36tPT9unXrSEhIYPbs2Xz11Vd4eHiwYsUKUlNTGyxM1RF5Z/VET4ka5ObmYmpqWqdj3Nzc2Lx5MxcvXqRTp05kZWWxb98+oHIiLAsLC0xNTZk8eTJOTk7IZDKio6NZsmQJ/v7+0oNIfn4+zZs3Vzp38+bNyc/PV9pmZmZGXl4e5eXlYs34vxF1ps3aMjU1pby8nLy8PGm8nKZSd3w+fPiQf/3rX8hkMrS1tZk0aZJKZVAT43PYsGE8evSIjz/+GG1tbcrLy3nvvfekVnxFfvV4a3zz5s2f+cHMzs6OR48eUVhYSPPmzaUeGAozZszggw8+ICUlBVdXV3r16kVoaChnz57F3d0dgKioKPr164eurq6Uf/bu3ZsWLVoAld0WNUVDpc20tDR27NjBsmXLapwLQpRDtWNsbIyenh76+vpSWr9y5Qqpqal8//33UpfY0aNH8+effxIdHc2wYcPUGeR6o86808bGhunTp+Po6MijR484cOAACxcuZOXKlUrDaDQx76xOQ8WlwrRp07h//z7l5eWMGDFC6W21u7s7hYWFLFq0CIDy8nL69eun1GUeGk9cCk8XFxeHj4+P0ja5XK70+eDBg9K9qK2tjZ2dHePGjWPjxo2MGjUKuVzOoUOHmDZtmtRTburUqcTHx3Pw4EFGjx7NoUOHMDc3Z+LEiWhpaWFra8vt27eJjIxskOuys7Nj1KhRQGUecfToUa5cuULfvn3JysoiJiaGtWvXYm5uDsCgQYO4dOkSR44cYfLkyQ0SpuqIvLN6olGiBmVlZUrja2rD09OTrKwsgoKCKC8vp0mTJgwZMoTt27dLrWE2NjbY2NhIx7i4uJCbm8uePXvq/OCnr6+PXC6nrKzsb1kZ/LtqLGkTaBRvVNQdn4aGhqxcuZLi4mIuX77Mjz/+SIsWLejUqZO0jybG5+nTp4mOjmb27NnY29uTmppKaGgoFhYWDBgwoEF/WxHHWVlZREZGkpKSwv3796moqEAul3Pnzh0A9PT0eOONN4iKisLd3Z309HRSUlKYMWMGAI6OjnTq1Im5c+fSuXNnOnfuTO/evZ/a7fFFaYi0WVZWRnBwMD4+PvUymZkoh1Rdv36d0tJSpWFDUPn3zM7OVlOo6p86804XFxdcXFykfdq2bcsnn3zCb7/9JnWDBs3MO6vTUHGpsHjxYoqLi0lKSmLr1q1YWFhIvcKuXr3KL7/8wuTJk3F2diYrK4vQ0FC2bdsmPeBB44lL4enatWvHv/71L6VtaWlpfPnll9LnzMxMnJ2dlRquXV1dkclk0ooR5eXlSkMQtLW1cXZ2JiMjQ+kcVdNj1fu2vrVs2VLps6mpKQUFBUBlLyG5XM6cOXOU9pHJZCpDaRuayDurJxolamBkZERRUVGdjtHS0mL8+PGMHTuW/Px8jI2NuXz5MlA59qcmTk5OSt2LTExMpJtIoaCgQOWNY1FREXp6en+72ej/7tSZNmtLET5Nebh7EnXHp7a2ttQK7ejoSGZmJrt27VJqlNDE+AwLC+Ptt9+WeiA4ODiQm5vLrl27GDBggJRf5efnS28loDIve/wNfG1lZGTQpEkTaQKnFStWYGZmxpQpUzAzM0NHR4ePP/5YGr4BlQX5vHnzuHPnDlFRUbi4uEhzTmhra/P555+TnJzMxYsXOXbsGOHh4fj7++Po6PhMYaxPDZE27927R2ZmJuvWrZNm4JbL5cjlckaPHo2fnx9dunQR5dBzqKiooHnz5ixevFjlu5dpdQ51551VaWtr06ZNG5Xl9TQx76xOQ8elogHSwcGBgoICtm/fLjVKRERE4O7ujqenp7RPcXExISEheHt7S42NjSUuhaczMDBQefv94MGDWh+vpaWl0rNCEzzeMF41nHK5HC0tLQIDA6U5MRTq2kDwvETeWUNYXtgvNTKKh4Nnoa2tjZmZGbq6usTExODi4vLEP2pqaqpSRc/FxUVlkpNLly6ptC6mpaUpTWAk/D2oM23WVnp6OmZmZhoxWeDTaFp8VlRUKM2BAJoZnyUlJSpd/7W1taUKgIWFBSYmJkp5WWlpKdeuXVN6s1Jb9+7d49SpU/Tq1QttbW0KCwvJzMzkn//8J507d5aGdpSXlysdZ29vj7OzM0eOHOHkyZP0799f6XstLS1cXFwYMWIEgYGBmJqaPlNDXENoiLRpZmbGl19+SVBQkPTvrbfewsrKiqCgIOlvI8qhZ9e6dWsKCgrQ0tLCyspK6d+zNshpIk3KO+VyOTdv3lTZRxPzzuq8yLhU9GxSeFpertBY4lKoH7a2tiQnJ0tzOEHlEtK6urpYWlpiaWmJrq6u0hxRFRUVJCcnSw3/inNUTUvJyckv7iKqcHR0RC6Xk5+fr5Ivm5mZvfCwiLxTlegpUQM3Nze2bt1KYWEhRkZGQGVX4eLiYu7du4dMJpMmRrGzs0NXV5f79+8TGxtL+/btkclkREVF8ccffyitGbt//35atGiBvb09MpmMkydPcu7cOebOnSvtM2TIEBYtWsSvv/7Ka6+9xtmzZ4mPj1d563Lt2jWViciEl58606ZMJpO65ZWWlpKfn09qaiqGhoZKre4JCQmNJm2qMz537tyJk5MTlpaWlJWVceHCBU6ePMnEiROVwqiJ8dm9e3d+/fVXLCwssLOzIzU1lX379uHh4QFUPuwPGTKEXbt2YWtri7W1NTt37sTQ0JC+ffs+8dzl5eXk5+dLS4ImJiaya9cumjVrxtixYwF45ZVXMDIy4ujRo5ibm5OXl8eWLVuqHULg6enJd999h46OjtIyYklJSVy+fFnqGXDjxg3u3r0rVajUrSHSpq6ursq8GYp5EapuF+UQPHr0SGUCtNqs2d6pUyfatm1LUFAQ48ePx9bWlvz8fOLi4ujUqVOdh8NpKnXmndu3b8fZ2Rlra2tpXHRaWhpTpkxRCqMm5p3Vaai4/O2337CwsJC6dCckJLB3716lOSW6d+/O/v37adOmjTR8IzIykm7duinlp40lLoX64eXlxYEDB9i0aRNDhgwhJyeHrVu3MmjQIAwMDAAYOHAgW7duxcjICAsLC/bv309+fr40t9TAgQPZt28fmzdvxsvLi7S0NA4fPqyW67GxsaFv376sW7eO999/n1atWlFUVER8fDyWlpb06tXrhYVF5J3VE40SNXBwcMDJyUlpOZQNGzZw9epVaZ9PP/0UgDVr1khd406cOMGWLVuAyjdN/v7+0hKKUPlQFxYWxt27d9HX18fe3p758+crLafWtm1bPvroIyIiIoiMjMTKyoqPPvpIWhseKmfFTUxMZNasWQ0XCYJGUmfazMvLk84NkJ2dzZEjR2jfvj3+/v5AZWPF2bNnWbBgQcNEQD1TZ3wWFxezadMmaR9bW1t8fX2VHto1NT4//PBDIiMj2bRpEwUFBZiamuLp6Ym3t7e0z7BhwygtLeX777/nwYMHODk5sWDBgqd2Yb916xZTp05FS0uLpk2bYmNjg6enJ4MHD5aO1dbWZs6cOYSGhjJ37lysrKzw8fHhq6++Ujlfnz59CA0NpXfv3kq/3bRpUxITE/n999958OABr776KsOHD9eYFTgaKm3WhiiHKitlVfM7oFYVVy0tLfz8/IiIiCAkJEQa9tK2bVuNSVv1QZ1554MHD9i4cSP5+fk0bdqUVq1aERAQoHQeTc07q9NQcVlRUcHWrVvJzc2VhgqOHTtWaUnm4cOHo6WlRWRkJHfv3sXY2Jju3bszevRoaZ/GFJdC/TAzM8PPz4+wsDA+/fRTXnnlFdzd3RkzZoy0j2Iy1PXr1/PgwQNatWrFggULpEkczc3NmTdvHj/++CNHjhyhdevWjB07lm+//Vbpt0aOHIm3tzcjR45s0GuaMWMGO3fulPKXZs2a4eTk9MLnlBB5Z/W05Jo4KEhDxMXFERoaSnBwcI0zlKvLli1bpFn7hb8fTU6bv//+O+fPn+fzzz9Xd1BqTcTnyy0vL48ZM2bg7++vtH56Y6DJaVOUQ4Imp8/GlneKuBT+jnJycpg1axYBAQGNrnx+HuJ+V6Xjr3i9KaiwsrJCLpdjamrKK6+8ou7gKLl58yZDhw4Vk4v9TWly2kxNTWXgwIFSl7TGQMTny0kmk3H//n3Cw8PR0dFRevPXWGhy2hTlkKDJ6bOx5Z0iLoW/o5MnT2JsbMyQIUPUHZQXStzvqkRPCUEQBOGlFB8fT0BAANbW1syZM0cjVtQQBEEQBEEQlIlGCUEQBEEQBEEQBEEQ1EKzBrEIgiAIgiAIgiAIgvC3IRolBEEQBEEQBEEQBEFQC9EoIQiCINSroqIipkyZQlZWlrqDomLVqlXs3btX3cEQ1ESkTUEQhLoTeafQ0HTVHQBBEATh5bJr1y66du2KlZUVAKGhoSQmJpKeno6JiQlr165VOeb06dPs2rWL27dvY2xszKBBg3jnnXek7xWTVj4uODgYW1tb6fPDhw+JiIjgzJkzFBYW8uqrrzJmzBj69OkDgLe3N4sWLcLT05OmTZvW96ULGq4h0mZV165dw9/fH1tbW7766iul70TaFAShsWqIvPPq1auEh4dz69YtSkpKaNGiBQMGDFDJXw8cOMChQ4fIzc3FyMiIHj16MH78eGnlJ5F3vhxEo4QgCIJQb0pKSjh27BifffaZtE0ul+Ph4UFaWhqXLl1SOebChQt88803TJw4ETc3NzIzMwkJCUFfX59BgwYp7btq1SqaNWsmfTY2Npb+L5PJWLp0Kc2aNWPOnDmYmZmRl5eHru7/FXUODg5YWloSHR2tcm7h5dbQabOoqIg1a9bQqVMn8vLylL4TaVMQhMaqofJOQ0NDBg8ejIODAwYGBly7do3vvvsOAwMDvLy8ADh16hRhYWFMmzYNV1dXcnJyWL9+PWVlZUyfPh0QeefLQgzfEARBEOrNhQsXAGjbtq207cMPP2Tw4MFYW1tXe0x0dDTdu3fHy8sLS0tLunXrxrvvvsvu3bt5fIEoY2NjTExMpH/a2v9XjB0/fpz79+/z6aef4urqioWFBa6urjg5OSmdo0ePHsTExNTXJQuNREOnzQ0bNuDh4YGzs7PKeUTaFAShsWqovLN169a4u7tjb2+PhYUF/fr1o0uXLiQkJEjnSUxMxNnZmX79+mFhYUHHjh3x8PAgJSVF6fdE3tn4iUYJQRAEod4kJCTQunVrtLS0an1MWVkZenp6Stv09fW5e/cuubm5Stv9/PyYOnUqixcv5sqVK0rfnTt3jrZt2/LDDz8wZcoU5syZw7Zt25DJZEr7OTk5kZKSQmlpaR2vTmjMGjJtHjx4kIKCAoYPH17teUTaFAShsWrocl3hxo0bJCYm0r59e2mbq6srqampJCUlAXDnzh3Onz9P165dlY4VeWfjJxolBEEQhHqTm5uLqalpnY5xc3Pj/PnzXLx4kYqKCm7dusW+ffsAyM/PB8DU1JTJkyczd+5c5s2bh42NDUuWLFF6o5KdnU1sbCwymQw/Pz9GjRrF4cOHCQ8PV/o9U1NTysvLVbrYCy+3hkqbaWlp7Nixg1mzZin13KlKpE1BEBqrhso7FaZNm8bYsWOZP38+Xl5eDBw4UPrO3d2dMWPGsGjRIsaMGcOMGTNwcHBg3LhxSucQeWfjJ+aUEARBEOpNWVkZ+vr6dTrG09OTrKwsgoKCKC8vp0mTJgwZMoTt27dLb2ZsbGywsbGRjnFxcSE3N5c9e/bQrl07oHKMq7GxMdOmTUNbW5vWrVtTVFTEjz/+iI+Pj3QuRfjEG5W/l4ZIm2VlZQQHB+Pj44OFhUWN5xFpUxCExqqhynWFxYsXU1xcTFJSElu3bpWGckDlZJi//PILkydPxtnZmaysLEJDQ9m2bRujRo2SziHyzsZPNEoIgiAI9cbIyIiioqI6HaOlpcX48eMZO3Ys+fn5GBsbc/nyZQAsLS1rPM7JyYnTp09Ln01MTNDV1VV6W21ra0tJSQmFhYXSpJiK8FWdJFN4+TVE2rx37x6ZmZmsW7eOdevWAZUNEHK5nNGjR+Pn50eXLl1E2hQEodFq6HJd0aDr4OBAQUEB27dvlxolIiIicHd3x9PTU9qnuLiYkJAQvL290dHRAUTe+TIQjRKCIAhCvXF0dOTEiRPPdKy2tjZmZmYAxMTE4OLi8sQKRmpqKiYmJtLntm3bEhMTQ0VFhfTwd/v2bQwMDDAyMpL2S09Px8zMTOlY4eXXEGmzadOmfPnll0r7Hjp0iEuXLjFv3jypsi3SpiAIjdWLLNflcjllZWXS55KSEpVhcdra2ioTDYu8s/ETjRKCIAhCvXFzc2Pr1q0UFhZKD1tZWVkUFxdz7949ZDIZqampANjZ2aGrq8v9+/eJjY2lffv2yGQyoqKi+OOPPwgICJDOu3//flq0aIG9vT0ymYyTJ09y7tw55s6dK+0zcOBADh48yObNmxk0aBA5OTls27aNgQMHKnUXTUhIoEuXLi8mQgSN0RBpU1dXFwcHB6XfMTY2Rk9PT2m7SJuCIDRWDVWu//bbb1hYWEhDMxMSEti7d6/SnBLdu3dn//79tGnTRhq+ERkZSbdu3aReEopjRd7ZuGnJH29qEgRBEITnsGDBAt544w1pvXB/f3+uXr2qst+aNWuwsLDg/v37rFixgrS0NKByvojRo0crLa24e/dujh49yt27d9HX18fe3p53332Xbt26KZ0zKSmJn376iRs3bmBiYkK/fv0YPnw4urqVbfClpaVMmTKFBQsW4OLi0lBRIGiohkibj9u2bRtnzpzhq6++Utou0qYgCI1VQ+Sd+/fv5+jRo+Tm5qKtrY2VlRUDBgzgrbfeknpHlJeXs3PnTk6ePMndu3cxNjame/fujB49mmbNmgEi73xZiEYJQRAEoV7FxcURGhpKcHBwjasRqMvvv//O+fPn+fzzz9UdFEENRNoUBEGoO5F3Cg1Nx9/f31/dgRAEQRBeHlZWVsjlckxNTXnllVfUHRwlqampDBw4UGkcv/D3IdKmIAhC3Ym8U2hooqeEIAiCIAiCIAiCIAhqoVn9bwRBEARBEARBEARB+NsQjRKCIAiCIAiCIAiCIKiFaJQQBEEQBEEQBEEQBEEtRKOEIAiCIAiCIAiCIAhqIRolBEEQBEEQBEEQBEFQC9EoIQiCIAiCIAiCIAiCWvx/UODBlE7aE4YAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 676 }, "id": "Z8sdZC7-wwdh", "outputId": "2288028a-5149-4e5b-9380-9c12e72378ad" }, "source": [ "plot_components(components_df, 'fc1', ascending=False)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABCUAAALlCAYAAADzMFwcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVyVdf7//+eRI6Agi+JxAcU1t3JJEVMLV3AL0/ik6TSYptlt0JxuMxY1JqkljjOl5TT5s8Q117Fc0lxwKTMrLUVzTy0RNwTcRYXr90dfTh45oByOXi6P++3WrXhf7+u6Xud9ruvEeXJd78tiGIYhAAAAAACAO6yE2QUAAAAAAIAHE6EEAAAAAAAwBaEEAAAAAAAwBaEEAAAAAAAwBaEEAAAAAAAwBaEEAAAAAAAwBaEEANyn+vXrJ4vFosOHD5tdSrG58lrWr18vi8WihISEYu//8OHDslgs6tev3y2vM23aNFksFk2bNu2O7/teM2fOHDVp0kRlypSRxWLRsGHDzC4JcKsH4TwGAFcRSgCACywWiywWi0qUKKFffvmlwH5t27a19y3ul1M44pd8c7gaduWFNOvXr3do//bbb9W3b1+dO3dOL730kkaOHKlOnToVua7vv/9e8fHx6ty5sypWrCiLxaKQkJAC++cdP+4IrXD3uJ/CWAB4UFjNLgAA7lVWq1XXrl3TJ598onfeeSff8v3792v9+vX2fnfa2LFj9dprryk4OPiO79vd7sXX0qNHD7Vo0UKVKlUyu5S72hdffCHDMDRjxgy1bNnS5e18+umnmjhxokqWLKn69evrxIkTbqwSAADcLlwpAQAuqlChgpo1a6akpCSnocPHH38sSXryySfvdGmSpEqVKqlu3boqWbKkKft3p3vxtfj7+6tu3bry9/c3u5S7WlpamiSpcuXKxdpOv3799OOPP+r8+fPatm2bO0oDAAB3AKEEABTDwIEDdfz4cS1btsyh/erVq5o2bZpatmyp+vXrF7j+/v379ec//1nBwcHy9PRU5cqV9ec//1n79+936Dd48GBZLBYtXrzY6Xa+++47WSwWxcTE2NsKu4z5u+++U0xMjCpWrChPT09VqVJFL774ov0L4vUOHjyoQYMGqVatWipVqpTKli2rRx55RIMHD9bp06cLGx5Jv3/ZdHaFQ2hoqCwWi0aPHu3QvmLFClksFr355psFvpaEhARVr15dkjR9+nT7LTIF3Sazbds2de3aVQEBASpdurQiIiK0adOmm9buzOHDh9W7d28FBQXJ29tbzZo1y/f+S4XPKbFy5Uq1atVKPj4+Klu2rJ566int2bPnppee3+q+88yZM0dt27ZVQECAvL29Va9ePY0ZM0bZ2dn5+n799dd68sknFRISIi8vL1WsWFEtWrTQW2+9Ze9jsVg0ffp0SVL16tXtY16tWrXCB82JvPFJSkrKt73rX39qaqqGDh2q2rVr24+/5s2b5ztuGjdurCZNmsjT07PItRTFvHnz1L59e5UtW1be3t6qVq2ann32WW3ZssWhX3Z2thITE/XII4+odOnS8vPz0+OPP6758+fn2+b1tyL98ssviomJUbly5VSmTBlFRkZq586dkqRTp05p0KBBqlSpkry9vRUWFqZ169bl215CQoL9Vpnp06erSZMmKlWqlGw2m/r376/jx487fW23+nl04z4WLlyo5s2bq3Tp0ipbtqx69+6to0ePOt1HRkaG4uPjVa9ePZUqVUr+/v5q3769Vq1ala/v9efQunXr1KZNG5UpU0Z+fn7q2rWrdu/e7dDf1eMzMTFRFotFEydOdLo8LS1NVqtVzZo1c2gbNWqUWrVqZf8crVy5svr06aNdu3YVur/rtWnTRhaLxemywj5DUlNTFRcXpxo1asjLy0vlypVTdHS0fvjhh3x9z507p9GjR+vhhx+Wn5+fypQpo5o1a6pXr17aunXrLdcKALcLt28AQDE8++yzeuWVV/Txxx/rqaeesrcvWbJEJ0+e1Lhx43TgwAGn6/7www/q0KGDzp07p+joaNWvX1979uzRrFmztHjxYq1Zs0ZhYWGSpNjYWE2ePFkzZsxQ9+7d820r7xfxW5lfYerUqRo0aJC8vLwUHR2tKlWqaP/+/fr444+1dOlSbd68WVWrVpUkHTt2TGFhYTp79qy6dOmip59+WpcvX9ahQ4c0c+ZMxcXFqVy5coXur127dpo9e7b27NmjunXrSpIOHDig3377TZKUnJysESNG2PsnJydLktq3b1/gNtu0aaOsrCxNnDhRjRo1chj7xo0bO/TdsmWL/vnPf+qxxx7TCy+8oN9++03/+9//1L59e23btk116tS56Zjl+fXXX9W8eXPVqFFDzz33nDIyMjRv3jx1795da9asUdu2bW+6jblz56pPnz7y9vbWM888o0qVKmnTpk167LHH1KhRI7ftu3///kpKSlJISIiefvppBQQEaPPmzRoxYoSSk5O1evVqWa2//xrw5ZdfqmvXrvLz81N0dLSCg4OVkZGh3bt368MPP9TIkSMlSSNHjtTnn3+u7du36+WXX1ZAQIAk2f9dFI0bN77p9rZs2aKoqChlZGToiSeeUM+ePXXx4kXt2rVLCQkJDsfN7WYYhp5//nlNnz5dQUFB6tmzp8qXL6/U1FStW7dOderUsX9pvXLliqKiorRhwwbVrVtXf/nLX3Tx4kUtXLhQvXr10rZt25ze8nX48GGFh4erXr166tevnw4fPqzPPvtMbdq00bfffqtOnTrJz89PvXr1UkZGhubOnavOnTtr37599nP2eu+9955WrVqlXr16qVOnTtq4caOSkpK0fv16fffddypfvry9b1E+j6734YcfasmSJYqOjlZERIS+++47zZs3T9u3b9e2bdvk5eVl7/vrr7+qTZs2Onz4sB5//HF16tRJFy5c0LJly9SpUydNnjxZAwcOzLePZcuWafHixercubMGDx6sXbt2afny5frhhx+0a9cuBQUFSXL9+Hzuuef0xhtvaMaMGXr55ZfzLZ81a5ZycnIcPl+/+uorJSYmqm3btnr66afl6+ur/fv3a+HChVqyZIm++eabQs/n4vjxxx8VGRmpjIwMRUVFqWfPnkpPT9fnn3+u1q1b67PPPlOXLl0k/X7cdurUyf4Z88ILL8hqtdqP28cff1xNmza9LXUCwC0zAABFJskIDg42DMMwBgwYYHh4eBhHjhyxL4+KijL8/PyMCxcuGG+88YYhyUhKSrIvz83NNerWrWtIMmbNmuWw7blz5xqSjDp16hg5OTn29oceesjw9PQ0Tp8+7dD/8uXLRmBgoGGz2YyrV6/a22NjYw1JxqFDh+xte/fuNUqWLGnUrFnTSE1NddjOmjVrjBIlShhPPfWUve399983JBkTJkzINwbnz583Ll68eNOx+uSTTwxJxqRJk+xtH330kSHJ6Nixo+Hp6WlcuHDBvqxx48ZGqVKljOzs7EJfy6FDhwxJRmxsrNP9rlu3zpCUb+yv3/9LL7100/qv35ckIyEhwWHZl19+aUgyOnfu7NCelJSUb99nz541AgICDE9PT2Pbtm0O/V999VX7Ppy9Tlf23aNHj3zv0ciRI/O9pz179jQk5avJMAzj1KlTDj87ey+Ko6DtZWdnG9WqVTMkGbNnz8633vXnmzPXn6PuMHnyZEOSERYWZmRlZTksu3btmpGWlmb/+Z133rG/L9efkydOnDBCQ0MNScY333xjb7/+PR4zZozDtkeNGmVIMgIDA40XX3zR4TNhxowZhiRj2LBhDuvkvcclS5Y0fvzxR4dlw4YNMyQZ/fv3t7e58nmUt48yZcoYKSkpDus8++yzhiRj3rx5Du0RERGGxWIx5syZ49CemZlpNGrUyPD29jaOHz9ub887jj08PIw1a9Y4rPPaa68Zkoxx48Y5tLt6fEZGRhqSjB07duRbVr9+fcPT09NIT0+3t504ccI4e/Zsvr7btm0zfHx8jE6dOjm0F/R5FRERYRT067izz5CrV68aNWvWNLy8vIz169c79D969KhRuXJlo2LFisbly5cNwzCMlJQUQ5LD53qenJwcIyMjw+m+AeBO4vYNACimgQMHKicnR1OnTpX0+18DV69erb59+6p06dJO19m0aZP27Nmjxx57TH379nVY1qtXL7Vu3Vp79+7Vxo0b7e2xsbG6cuWK5syZ49B/6dKlyszMVN++fe1/+S7If//7X129elUTJ07Md0tF+/btFR0draVLl+rcuXMOy0qVKpVvWz4+Pk7bb5R3xUPeFRB5/12hQgUNHTpUV65csb/O06dPa/v27WrdurXbLsNv1apVvitI+vfvL6vVqu+//75I2woNDdU//vEPh7aoqChVrVr1lra1ePFiZWVlqW/fvvn+ivqPf/yj0L/oFmXfEydOlNVq1dSpU/O9RyNGjFC5cuU0e/bsfPtw9n7m/RX6Tlu6dKkOHz6s6Oho9enTJ9/ywp6scTt88MEHkqTJkyfnmyfEw8PDYULTqVOnymKx6N1333U4J202m/3qjrw5Z65XrVo1vfbaaw5tsbGxkn6/HWT8+PEqUeKPX9369Okjq9Va4Bwazz33nJo0aeLQlpCQIH9/f3366af223hc+TzKM3ToUD3yyCMObXlXO1x/XG7fvl0bNmzQ008/rd69ezv0DwgI0FtvvaXLly/rf//7X7599O7dO9+VU4MGDcq3j+LIG+e8q87ybNmyRbt27VLXrl0drgqz2WwqU6ZMvu00atRI7dq107p163T16lW31Ha9L774Qr/88ouGDBmiiIgIh2WVK1fW8OHDdfz4cYfPW8n5uV2iRAkFBga6vUYAKCpu3wCAYgoPD9cjjzyiqVOn6h//+Ic+/vhj5ebmOr0MOc+PP/4o6fdbG5xp166dNm7cqJ9++klPPPGEJOnPf/6zRowYoenTp+svf/mLvW9Rbt349ttvJUkbNmxweu/xyZMnlZOTo3379qlp06aKjo7W66+/rr/85S9auXKloqKi1KpVK9WvX7/A+6BvFBoaqho1amj9+vXKzc2134feoUMHRUREyGq1Kjk5WZGRkVq3bp0MwyhwXFxx/X3geUqWLKkKFSooMzOzSNtq3LixPDw88rVXqVLFPraF+emnnyRJrVu3zrfM19dXjRs3zvfIzKLu++LFi9q+fbuCgoI0YcIEp9vy8vJyuB+/b9++WrRokcLDw9WrVy+1bdtWrVq1uuNf/K+3efNmSVLnzp1NqyHPhQsXtHPnTlWoUCHfl/wbnTt3TgcOHFBwcLD9dqXr5R3becfC9Zy9x3kTgD700EP5vgR7eHioQoUKSk1NdVrLjV9apd8nYG3cuLE2bNig3bt3q3Hjxi59HuVxdn5VqVJFkhzOr7xj9MyZM04fw3rq1ClJyjdPRFH2URw9evSQv7+/Zs+ercTERPv7UNjn6xdffKGPPvpIW7ZsUXp6er4Jj9PT093+9J28cfz111+djmPe/B+7d+9Wly5dVL9+fTVu3Fhz5szRr7/+qu7du6t169Zq1qzZbZ9/BQBuFaEEALjBwIEDNXToUK1YsUJJSUlq2rRpoV9ezpw5I0kF/sKa156VlWVvCwkJUfv27bV69Wrt3r1b9erV08mTJ/Xll1+qcePGatiw4U3rzJuYcvz48YX2O3/+vKTfA4Xvv/9eCQkJ+vLLL7Vo0SJJv38h+Nvf/qahQ4fedJ/S71dLTJkyRT/++KNKliypU6dOqX379ipTpozCwsLsf9W7lfkkiqqgqw+sVqtycnLctq3c3Nybrp/3vleoUMHp8oLai7LvzMxMGYahU6dOOUxSWZiePXtq2bJl+ve//62pU6dq8uTJkqSmTZtq7Nix6tix4y1tx53yjv274TGwRanFlXM7j7MnteRdaVHQU1ysVmuBf5Ev6HiqWLGiQ63FqdnZcZlX8/XnV95nz+rVq7V69Wqn+5H++OxxZR/FUapUKT3zzDOaMmWKVq1apc6dO9uvTCtfvny+cGzixIkaNmyYAgMD1bFjR1WtWlWlS5eWxWKxz2vhbELZ4sobxwULFhTaL28cPTw8tHbtWo0aNUoLFy7Uq6++KkkqU6aMYmNjNXbsWPn6+rq9TgAoCm7fAAA3eO6551SqVCkNHjxYR48etV9aXJC8LxgFzYJ/7Ngxh355brzEePbs2bp27Zq9/WbytnfmzBkZhlHgP9f/hbVevXqaN2+eTp8+rS1btigxMVG5ubl6+eWX9cknn9zSfvP+ArtmzZp8wUO7du30008/KSMjQ8nJyfL399ejjz56S9u91/j5+UmSTpw44XR5Qe1FkfceN2nSpND32DAMh/W6du2qtWvXKjMzU8nJyfrrX/+qn3/+Wd26dSvS0wTcJe+LaEFPcbiTilKLq+f27VDQ8ZRXW14Nd6LmvHUnTpxY6DGZ9zQWM9z4+frFF1/o9OnT6tOnj8PjiK9du6aEhARVrFhRP//8s+bNm6fx48frrbfeUkJCQqHh4o3ybsdx9ljpwoKrxYsXFzqOeZPTSlJgYKDee+89HTlyxD6pcd26dTVp0iS99NJLt1wrANwuhBIA4AYBAQGKiYlRamqqfHx89OyzzxbaP+8qioIu1c97zN+NX8579uwpPz8/zZo1S7m5uZo+fbqsVqvTe+6dadGihaTfH/9YVFarVU2bNtWrr75qn9fi888/v6V127VrJ4vFouTkZK1du1Y1atSwP6avffv2ys3N1YwZM7R//361adPG6W0KN8rr466/lN4Jee+7s3vzz58/X+DcAEXh6+urBg0a6Oeff1ZGRkaR1/fx8VG7du307rvv6vXXX9eVK1e0YsUK+/I7Ne55x+r1+zaLj4+PHn74YZ04ccLpbRfXy3vc4tGjR50+SrOgc/t22LBhQ762M2fOaNu2bfZHxEqufx4VRXE+e4qiOMdnq1atVLt2bS1evFhnzpyxhxM3hr7p6enKyspSy5Yt811dcv78efvtMLcib06HI0eO5Ft242NmpeKPY61atTRgwABt2LBBvr6+BT5mGgDuJEIJAHCTMWPG6LPPPtPKlSudToB2vVatWqlOnTrauHGjFi5c6LBs4cKF+vrrr/XQQw/lm3sg7xLjo0eP6r333tP27dvVpUsX2Wy2W6oxLi5OJUuW1F//+lft27cv3/IrV644/LK7detW+6Xd18v7C2xBE3neyGazqUGDBvrmm2/01VdfOdye0bJlS3l7e2vs2LGSCr6v/UaBgYGyWCz2R4veC7p3726/b3379u0Oy8aMGeP0L6OueOWVV3TlyhX179/f6TYzMzMdvjh99dVXTv9S6+x9zpvs73aP+5NPPqlq1appyZIl+SZ3lVTgPAq3S96tSi+++GK+cyI3N9d+NYH0+0SqhmHo73//u8OX4/T0dI0ePdre53abOXNmvhAlISFBZ86c0bPPPmt/XKern0dF0axZMz3++ONatGiRfVLgG+3YsUMnT550eR9S8Y/P2NhYXb58WR9++KGWL1+uhg0b5rsVz2azqXTp0tq6davD7SZXr17Vyy+/rPT09FveX/PmzSVJU6ZMcWhPTk52etx3795dNWvW1H/+8x8tX77c6Ta//fZbXbx4UZJ06NAhHTx4MF+fzMxMZWdn39JkxQBwuzGnBAC4SdWqVVW1atVb6muxWDR9+nR17NhRvXr1Uvfu3VW3bl3t3btXn3/+ucqUKaMZM2Y4zLSfJzY2Vh9//LHi4+PtP9+qunXraurUqerfv78aNGigTp066aGHHtLVq1f122+/6euvv1b58uW1Z88eSb9/qZk8ebJat26tmjVrKjAwUL/88ouWLl0qLy8vDRs27Jb33b59e+3cudP+33m8vLzUqlWrIs8n4evrq/DwcH399dfq27evHnroIXl4eCg6OvqW5tcwg5+fn/7zn//oueeeU8uWLfXMM8+oUqVK2rRpk7Zv366IiAht2LDB6fteFP3799fWrVv14YcfqmbNmvandGRkZOjQoUP66quv9Pzzz+ujjz6S9PsX7qNHj6pVq1aqVq2aPD09tXXrVq1du1ahoaEOT0to3769xo8fr4EDB+rpp59WmTJlFBAQoLi4uGLVfCNPT08tWLBAkZGR6tOnjyZPnqwWLVro8uXL2r17t5KTkx2ClD179igxMdFhG5mZmQ4TFP7rX/9y+WkiL7zwgr7++mvNnDlTtWvXVvfu3VW+fHmlpaVp7dq16t+/v33iwb/97W9asWKFFi9erEaNGqlLly66ePGiFixYoJMnT2r48OHF+oJ/qzp37qxWrVrZj7ONGzdq48aNqlatmsNYFefzqCg+/fRTtWvXTgMGDND777+v8PBwBQQEKDU1VSkpKdq5c6e+/fbbWw5ZnSnu8fncc8/pzTff1MiRI3X16lWnn68lSpTQ0KFDlZiYqEceeUTdu3fXlStXtG7dOmVkZKht27b2q0tu5vnnn9f48eM1duxYbd++XfXr19e+ffu0YsUK9ejRI9/TSEqWLKlFixYpKipKXbt2VcuWLdW4cWOVLl1aR44c0Q8//KCDBw/q2LFjKl26tLZv366ePXsqLCxM9erVU+XKlXXq1CktXrxYV69etc8xAQCmuv1PHQWA+48kIzg4+Jb6vvHGG/meNZ9nz549xp/+9CejYsWKhtVqNSpWrGj07dvX2LNnT6HbrFWrliHJKFu2rJGdne20T2xsrCHJOHToUL5lKSkpRmxsrFG1alXD09PTCAwMNBo0aGAMGjTISE5OtvfbvHmzMXjwYKNhw4ZGYGCg4e3tbdSsWdPo16+fsWPHjlt6/XmWLFliSDIsFotx4sQJh2XvvPOOIcmoUKFCkV7L/v37jW7duhlly5Y1LBaLwzivW7fOkGSMHDnS6TZDQ0ON0NDQW6r90KFDhiQjNjbW6fKIiAjjxv+lJiUlFfi+L1++3HjssceMUqVKGQEBAUZ0dLSxe/duo2vXroYkIzMzs1j7zrN06VKja9euRvny5Y2SJUsaFSpUMMLCwow33njD2L17t73fvHnzjN69exu1atUyfHx8jDJlyhgNGjQwXn/9dePkyZP5tvvvf//bqFu3ruHp6WlIuuVxdKaw49QwDOPXX381XnrpJaNatWpGyZIljbJlyxrNmzc33n77bYd+ee93Yf8UtI+imDVrlvHEE08Yfn5+hpeXl1GtWjWjT58+xtatWx36Xbp0yXj77beNBg0aGN7e3oavr6/RqlUr49NPP823zZu9x5KMiIgIp8ucHccjR440JBnr1q0zkpKSjEaNGhne3t5GUFCQ0a9fPyMtLc3ptoryeXT9Poryes6ePWu8/fbbxqOPPmr4+PgY3t7eRrVq1YwuXboYkydPNs6fP2/vW9g5VNi4FPf4bN++vSHJsFqtxvHjx532uXr1qvHvf//bqFevnuHt7W1UqFDB+NOf/mQcPnzY6TFd2Jjs3LnT6Ny5s+Hr62v4+PgYERERxvr16wt9/SdOnDBeffVVo0GDBkapUqUMHx8fo1atWsbTTz9tzJw507h69aphGIZx5MgRIz4+3mjZsqVRoUIFw9PT0wgODjY6depkLF++vEjjAgC3i8UwbpjpCgAA3HE5OTmqUaOGrly54nArAFBUCQkJeuutt7Ru3Tq1adPG7HIAACgUc0oAAHAHZWVl2e/3zmMYhsaMGaPffvtNPXr0MKkyAACAO485JQAAuIM2b96sXr16KTIyUtWqVdP58+e1efNmbdu2TVWqVLHPSwAAAPAgIJQAAOAOqlOnjrp166ZvvvlGy5cv17Vr1xQSEqKhQ4fq9ddfL9YkfwAAAPca5pQAAAAAAACmYE4JAAAAAABgCkIJAAAAAABgivtqTom0tDSzS7ipoKAgpaenm13GfYPxdB/G0r0YT/diPN2HsXQvxtO9GE/3YSzdi/F0L8bTve6F8axcuXKBy7hSAgAAAAAAmIJQAgAAAAAAmIJQAgAAAAAAmIJQAgAAAAAAmIJQAgAAAAAAmOK+evoGAAAAAAC3S05Oji5fvixJslgsJlfzuxMnTig7O9u0/RuGIQ8PD3l7e7u0PqEEAAAAAAA3kZOTo0uXLsnHx+euCSQkyWq1ysPDw9QaLl++rKtXr6pkyZJFXpfbNwAAAAAAuInLly/fdYHE3cLLy0tXrlxxaV1CCQAAAAAAbgGBhHPFGRdCCQAAAAAAboJAonCujg+hBAAAAAAAMAWhBAAAAAAA97Hc3FwNHz5cDRo0UHBwsDZt2mR2SXY8fQMAAAAAABflDIy+o/vzmLKkyOskJydr/vz5WrBggUJDQxUQEFBo/6ysLI0YMUKrV6+WJHXs2FFjxoyRv7+/SzUXhislAAAAAAC4jx0+fFg2m01hYWGy2Wzy9PQstH9cXJx27typWbNmadasWdq5c6eGDh16W2rjSgkAAAAAAO5Tw4YN04IFCyRJwcHBCgkJ0ebNmzV58mTNnDlTaWlpKlu2rGJiYhQfH6/9+/dr3bp1+vzzz9WsWTNJ0rhx49SjRw8dOHBAtWrVcmt9hBIAAAAAANynRo0apZCQEM2dO1fLly+Xh4eHEhMTNWPGDI0cOVLh4eE6ffq0du7cKUnaunWrfHx87IGEJIWFhal06dLaunUroQQAAAAAALg1fn5+8vX1lYeHh2w2my5cuKApU6YoISFBvXv3liRVr17dHkKcPHlS5cqVc3jEp8ViUVBQkE6ePOn2+phTAgAAAACAB8S+ffuUnZ2t1q1bm12KJEIJAAAAAADw/9hsNp0+fVqGYdjbDMNQenq6bDab2/dHKAEAAAAAwAOidu3a8vLy0saNG50ub9q0qS5cuKAtW7bY27Zs2aKLFy+qadOmbq+HOSUAAAAAAHhA+Pr6asCAAUpMTJSXl5fCw8OVmZmplJQUxcbGqnbt2mrbtq1ee+01jRs3TpL02muvqUOHDm6f5FIilAAAAAAA4IESHx8vf39/TZgwQceOHVNQUJBiYmLsyydNmqQRI0aob9++kqTIyEiNGTPmttRiMa6/UeQel5aWZnYJNxUUFKT09HSzy7hvMJ7uw1i6F+PpXoyn+zCW7sV4uhfj6T6MpXsxnu51r47nxYsXVbp0abPLyMdqteratWtml1Ho+FSuXLnA9ZhTAgAAAAAAmIJQAgJu4lwAACAASURBVAAAAAAAmIJQAgAAAAAAmIJQAgAAAAAAmIJQAgAAAAAAmIJHghZD99l7btpncd+6d6CS+8PNxpOxLBrG03041wEAAIDbgyslAAAAAACAKQglAAAAAACAKQglAAAAAACAKQglAAAAAAC4j+Xm5mr48OFq0KCBgoODtWnTJrNLsmOiSwAAAAAAXHQrk6K7kysTrCcnJ2v+/PlasGCBQkNDFRAQUGj/iRMnau3atfr555916dIlHT161NVyb4orJQAAAAAAuI8dPnxYNptNYWFhstls8vT0LLT/lStX1LlzZ73wwgu3vTaulAAAAAAA4D41bNgwLViwQJIUHByskJAQbd68WZMnT9bMmTOVlpamsmXLKiYmRvHx8ZKkv//975KkZcuW3fb6CCUAAAAAALhPjRo1SiEhIZo7d66WL18uDw8PJSYmasaMGRo5cqTCw8N1+vRp7dy505T6CCUAAAAAALhP+fn5ydfXVx4eHrLZbLpw4YKmTJmihIQE9e7dW5JUvXp1NWvWzJT6mFMCAAAAAIAHxL59+5Sdna3WrVubXYokQgkAAAAAAGASQgkAAAAAAB4QtWvXlpeXlzZu3Gh2KZKYUwIAAAAAgAeGr6+vBgwYoMTERHl5eSk8PFyZmZlKSUlRbGysJOno0aPKzMxUamqqJNknwaxevbp8fHzcWg+hBAAAAAAAD5D4+Hj5+/trwoQJOnbsmIKCghQTE2NfPn78ePtjRCUpKipKkrRgwQK1bNnSrbUQSgAAAAAA4KLFfeuaXcJNDR48WIMHD7b/XKJECcXFxSkuLs5p/wkTJmjChAl3pDbmlAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAAAKYglAAAAAAA4D6Wm5ur4cOHq0GDBgoODtamTZvMLsnOanYBAAAAAADcq5bOy7qj+3uyV0CR10lOTtb8+fO1YMEChYaGKiCg4G0cOXJEEyZM0KZNm3Ty5EnZbDZFR0dr2LBhKlWqVHFKd4pQAgBwR3WfvafQ5Yv71r1DlQAAADwYDh8+LJvNprCwsJv2PXDggHJycjR27FhVr15d+/fv16uvvqrMzEz985//dHtthBIAANzDCHnc52ZjKTGeRcF4uhfnunsxnu7DuX73GzZsmBYsWCBJCg4OVkhIiDZv3qzJkydr5syZSktLU9myZRUTE6P4+Hi1bdtWbdu2ta8fGhqqIUOGaPz48YQSAAAAAADg1o0aNUohISGaO3euli9fLg8PDyUmJmrGjBkaOXKkwsPDdfr0ae3cubPAbZw/f77QWz6Kg1ACAAAAAID7lJ+fn3x9feXh4SGbzaYLFy5oypQpSkhIUO/evSVJ1atXV7NmzZyun5qaqo8++khDhgy5LfXx9A0AAAAAAB4Q+/btU3Z2tlq3bn3TvqdOnVLfvn31xBNPaNCgQbelHkIJAAAAAADg4OTJk/q///s/1alTR++//74sFstt2Q+hBAAAAAAAD4jatWvLy8tLGzduLLDPiRMnFBMTo9q1a+vDDz+U1Xr7Zn5gTgkAAAAAAB4Qvr6+GjBggBITE+Xl5aXw8HBlZmYqJSVFsbGxOn78uGJiYlSxYkUlJCQoIyPDvm65cuXk4eHh1noIJQAAAAAAeIDEx8fL399fEyZM0LFjxxQUFKSYmBhJ0oYNG3To0CEdOnRIzZs3d1hv8+bNqlKliltrIZQAAAAAAMBFT/Zy36MyD5y+VOjyWuVKubTdwYMHa/DgwfafS5Qoobi4OMXFxeXr26tXL/Xq1cul/biCOSUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApCCUAAAAAAIApivX0jZUrV2rJkiXKyspSSEiI+vXrp3r16hXYf9euXZo+fbpSU1MVGBio6OhoRUZGOvTJzMzU7Nmz9dNPP+ny5cuy2WwaOHCg6tevX5xSAQAAAADAXcblUGLTpk2aNm2aBgwYoLp162rVqlV655139N577ykoKChf/5MnT2rs2LFq27athgwZoj179uiTTz6Rn5+fWrRoIUm6cOGCRowYobp16yo+Pl5+fn46ceKE/Pz8XH+FAAAAAADgruRyKLFs2TJFRESoQ4cOkqT+/ftr27ZtWrVqlfr06ZOv/6pVqxQYGKj+/ftLkkJCQnTgwAEtXbrUHkosXrxYgYGBDs9KtdlsrpYIAAAAAADuYi6FEteuXdPBgwf15JNPOrQ3bNhQe/fudbrO/v371bBhQ4e2Ro0aacOGDbp27ZqsVqt++OEHNW7cWO+9955+/vlnBQYGqn379oqKipLFYnGlVAAAAAAAcJdyaaLLs2fPKjc3V/7+/g7tAQEBysrKcrpOVlaWAgICHNr8/f2Vk5Ojc+fOSfr9Fo9Vq1apQoUKeuONN9SlSxfNnj1bK1eudKVMAAAAAAAeeLm5uRo+fLgaNGig4OBgbdq0yeyS7Io10aW75ebmqmbNmvbbP6pXr65jx45p5cqV6tSpU77+a9as0Zo1ayRJiYmJTueyMNuNNVmt1ruyznuBs3FjPF3HselejKf7cK67F8emezGe7sV4ug9j6V6Mp3vdL+N54sQJWa35v0K/++67d7SOV155JV+bs7qut2rVKs2fP1+fffaZQkNDFRAQUOA6ubm5io2N1c8//6z09HT5+/vr8ccf14gRI1SpUqUC9+Hl5eXS++pSKOHn56cSJUrozJkzDu3OrobI4+wqijNnzsjDw0NlypSRJAUGBiokJMShT0hIiFasWOF0mx06dLDPaSFJ6enpRX4tt9uNNQUFBd2Vdd4LnI0b4+k6jk33Yjzdh3PdvTg23YvxdC/G030YS/diPN3rfhnP7OxseXh4mF2Grl275vCz1WrN13ajX375RTabTU2aNClwO3lyc3PVsmVLxcXFqUKFCjp27JhGjx6tfv366YsvvihwH9nZ2QW+r5UrVy5wPZdu37BarapRo4ZSUlIc2nfs2KE6deo4Xad27drasWOHQ1tKSopq1KhhT2jq1KmjtLQ0hz5paWn3ZIoGAAAAAIDZhg0bpoSEBB09elTBwcEKDw+XYRj66KOP1KpVK1WvXl1NmzbV2LFjJUklSpTQwIED1bRpU4WEhCgsLExxcXHatm2bLl++7Pb6XAolJKlbt25av369kpOTlZqaqqSkJGVkZKhjx46SpEmTJmnSpEn2/pGRkcrIyNC0adOUmpqq5ORkrV+/3mGyzK5du2r//v1atGiRjh8/rm+//VYrVqxQVFRUMV4iAAAAAAAPplGjRumvf/2rKlWqpJ9++knLly9XYmKiJk6cqCFDhmjt2rWaPHlygbdmZGZmatGiRWrSpIm8vb3dXp/Lc0q0bNlS586d06JFi5SZmakqVaooPj5e5cuXl5T/Eh2bzab4+HhNnz7d/njQ559/3v44UEmqVauW/v73v2vOnDn63//+p6CgIPXq1YtQAgAAAAAAF/j5+cnX11ceHh6y2Wy6cOGCpkyZooSEBPXu3VvS7/M5NmvWzGG9t99+W0lJSbp06ZIeffRRzZgx47bUV6yJLqOiogoMDBISEvK11a9fX+PGjSt0m48++qgeffTR4pQFAAAAAACc2Ldvn7Kzs9W6detC+7300kvq3bu3jh49qnfffVdDhgzRrFmzZLFY3FrPXfX0DQAAAAAAYL6yZcuqbNmyqlmzpmrVqqWwsDB9//33Cg8Pd+t+XJ5TAgAAAAAA3Ftq164tLy8vbdy48ZbXMQxD0u9P2HA3rpQAAAAAAOAB4evrqwEDBigxMVFeXl4KDw9XZmamUlJSFBsbqy1btmjnzp0KCwuTv7+/Dh8+rPHjx6tKlSpq3ry52+shlAAAAAAA4AESHx8vf39/TZgwQceOHVNQUJBiYmIkSd7e3lq2bJnGjx+vS5cuyWazqU2bNvrvf/97dz19AwAAAACAB93QoUPdtq0Dpy8VurxWuVIubXfw4MEaPHiw/ecSJUooLi5OcXFx+fo+/PDDWrhwoUv7cQVzSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAAFMQSgAAAAAAcB/Lzc3V8OHD1aBBAwUHB2vTpk1ml2RnNbsAAAAAAADuVbYD8e7b1s06ZEona40t8naTk5M1f/58LViwQKGhoQoICLil9S5fvqxu3bpp9+7dWr58uRo1alTkfd8MV0oAAAAAAHAfO3z4sGw2m8LCwmSz2eTp6XlL640ePVqVKlW6rbURSgAAAAAAcJ8aNmyYEhISdPToUQUHBys8PFyGYeijjz5Sq1atVL16dTVt2lRjxzpegbFy5Upt2rRJb7755m2tj9s3AAAAAAC4T40aNUohISGaO3euli9fLg8PDyUmJmrGjBkaOXKkwsPDdfr0ae3cudO+TlpamuLj4zVz5kx5e3vf1voIJQAAAAAAuE/5+fnJ19dXHh4estlsunDhgqZMmaKEhAT17t1bklS9enU1a9ZMkpSTk6MhQ4Zo0KBBatCggY4cOXJb6+P2DQAAAAAAHhD79u1Tdna2Wrdu7XT5+++/r5IlS+rFF1+8I/VwpQQAAAAAAJAkffPNN/ruu+8UGhrq0P7kk08qOjpakyZNcuv+CCUAAAAAAHhA1K5dW15eXtq4caNq1KiRb/m7776rixcv2n8+ceKE+vTpow8++EBhYWFur4dQAgAAAACAB4Svr68GDBigxMREeXl5KTw8XJmZmUpJSVFsbKyqVq3q0N/Hx0eSVK1aNVWuXNnt9RBKAAAAAADwAImPj5e/v78mTJigY8eOKSgoSDExMabUQigBAAAAAICLTtYa67ZtHTh9qdDltcqVcmm7gwcP1uDBg+0/lyhRQnFxcYqLi7vpulWqVNHRo0dd2u+tIJQw2fvvv1/o8qFDh96hSu4PjKf73GwsJcazKDg2AQAAgPx4JCgAAAAAADAFoQQAAAAAADAFoQQAAAAAADAFoQQAAAAAADdhGIbZJdzVXB0fQgkAAAAAAG4BwYRzxRkXQgkAAAAAAG7C29tbFy5cIJhwIjs7W56eni6tyyNBAQAAAAC4CQ8PD5UqVUoXL16UJFksFrfvY++xM4Uur1wqfyDi5eWl7Oxst9dyqwzDkIeHh0qWLOnS+oQSAAAAAADcAg8PD/n4+Ny27f9/238rdHnXhyvlawsKClJ6evrtKum24/YNAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCkIJAAAAAABgCmtxVl65cqWWLFmirKwshYSEqF+/fqpXr16B/Xft2qXp06crNTVVgYGBio6OVmRkpNO+n332mebMmaOoqCgNGDCgOGUCAAAAAIC7kMtXSmzatEnTpk1Tjx49NG7cONWpU0fvvPOO0tPTnfY/efKkxo4dqzp16mjcuHF66qmnlJSUpM2bN+fru2/fPq1Zs0ahoaGulgcAAAAAAO5yLocSy5YtU0REhDp06KCQkBD1799fgYGBWrVqldP+q1atUmBgoPr376+QkBB16NBBERERWrp0qUO/ixcv6oMPPtBLL70kHx8fV8sDAAAAAAB3OZdCiWvXrungwYNq1KiRQ3vDhg21d+9ep+vs379fDRs2dGhr1KiRDh48qGvXrtnbJk+erPDwcD388MOulAYAAAAAAO4RLoUSZ8+eVW5urvz9/R3aAwIClJWV5XSdrKwsBQQEOLT5+/srJydH586dkyStWbNGx48fV+/evV0pCwAAAAAA3EOKNdGlO6WlpWnOnDkaPXq0rNZbK2vNmjVas2aNJCkxMVFBQUG3s0SX3FiT1WotUp1342syi7OxYDxdV9xj09k2HmSc6+7jjnMdf3DHuY4/MJ7uxXi6D2PpXoynezGe7nM//p7kUijh5+enEiVK6MyZMw7tzq6GyOPsKoozZ87Iw8NDZcqU0fbt23Xu3Dm98sor9uW5ubnavXu3Vq9erZkzZ6pkyZIO63fo0EEdOnSw/1zQJJtmurGmoKCgItV5N74mszgbC8bTdcU9Np1t40HGue4+7jjX8Qd3nOv4A+PpXoyn+zCW7sV4uhfj6T736u9JlStXLnCZS6GE1WpVjRo1lJKSoscee8zevmPHDoWHhztdp3bt2vrhhx8c2lJSUlSjRg1ZrVaFhYXpX//6l8Py//73v6pYsaJ69Ohxy1dPAAAAAACAe4PL3/S7deumDz74QLVq1VKdOnW0evVqZWRkqGPHjpKkSZMmSZLi4uIkSZGRkVq5cqWmTZumDh06aO/evVq/fr1efvllSZKPj0++p214eXnJ19dXVatWdbVMAAAAAABwl3I5lGjZsqXOnTunRYsWKTMzU1WqVFF8fLzKly8vKf9lJTabTfHx8Zo+fbr98aDPP/+8WrRoUbxXAAAAAAAA7knFuiciKipKUVFRTpclJCTka6tfv77GjRt3y9t3tg0AAAAAAHB/cOmRoAAAAAAAAMVFKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExBKAEAAAAAAExhLc7KK1eu1JIlS5SVlaWQkBD169dP9erVK7D/rl27NH36dKWmpiowMFDR0dGKjIy0L//ss8/0/fffKy0tTVarVbVr11afPn1UtWrV4pQJAAAAAADuQi5fKbFp0yZNmzZNPXr00Lhx41SnTh298847Sk9Pd9r/5MmTGjt2rOrUqaNx48bpqaeeUlJSkjZv3mzvs2vXLkVGRmr06NEaOXKkPDw8NHr0aJ0/f97VMgEAAAAAwF3K5VBi2bJlioiIUIcOHRQSEqL+/fsrMDBQq1atctp/1apVCgwMVP/+/RUSEqIOHTooIiJCS5cutfd544031LZtW1WtWlVVq1bVkCFDdPbsWe3Zs8fVMgEAAAAAwF3KpVDi2rVrOnjwoBo1auTQ3rBhQ+3du9fpOvv371fDhg0d2ho1aqSDBw/q2rVrTte5dOmSDMOQr6+vK2UCAAAAAIC7mEtzSpw9e1a5ubny9/d3aA8ICNCOHTucrpOVlaVHHnnEoc3f3185OTk6d+6cAgMD862TlJSkatWq6aGHHnK6zTVr1mjNmjWSpMTERAUFBbnycm6rG2uyWq1FqvNufE1mcTYWjKfrintsOtvGg4xz3X3cca7jD+441/EHxtO9GE/3YSzdi/F0L8bTfe7H35OKNdHl7TR9+nTt3btXo0aNUokSzi/o6NChgzp06GD/uaD5LMx0Y01BQUFFqvNufE1mcTYWjKfrintsOtvGg4xz3X3cca7jD+441/EHxtO9GE/3YSzdi/F0L8bTfe7V35MqV65c4DKXbt/w8/NTiRIldObMGYf2rKwsBQQEOF0nICBAWVlZDm1nzpyRh4eHypQp49A+bdo0ffPNN3rzzTdVoUIFV0oEAAAAAAB3OZdCCavVqho1aiglJcWhfceOHapTp47TdWrXrp3v1o6UlBTVqFFDVusfF2wkJSXZA4ng4GBXygMAAAAAAPcAl5++0a1bN61fv17JyclKTU1VUlKSMjIy1LFjR0nSpEmTNGnSJHv/yMhIZWRkaNq0aUpNTVVycrLWr1+vJ5980t7n448/1vr16/Xyyy/L19dXWVlZysrK0uXLl4vxEgEAAAAAwN3I5TklWrZsqXPnzmnRokXKzMxUlSpVFB8fr/Lly0vKf6+LzWZTfHy8pk+fbn886PPPP68WLVrY++Q9TnTUqFEO68bExOiZZ55xtVQAAAAAAHAXKtZEl1FRUYqKinK6LCEhIV9b/fr1NW7cuAK3N3/+/OKUAwAAAAAA7iEu374BAAAAAABQHIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAADAFIQSAAAAAAD8/+zdeVhWdf7/8ReKJKYCioh6u47KqAmT+SvGvaKNcskKs7RxKS11HGua0qux1CzMLq00LDUzdTRNG3NNu9Asl6/W5MLighsxoIKIN+IKCL8/vDjDLesNt35An4/r8qr75pxzf86bcz7nnNf9OQcYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGCEu+kGAACQ35pl9kLedXyvZz/vm9MYAAAA3FCMlAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGuJtuAAAAzpoxY0axPx89evRNagkAAADKg5ESAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABghHt5Zt64caNWr14tu90um82mQYMGqU2bNkVOv3//fi1YsECJiYny8fFRr1699PDDD5drmQAAAAAAoHIq80iJHTt26KuvvtKTTz6pDz74QAEBAXr//feVmppa6PQpKSkKDw9XQECAPvjgA/Xp00fz58/Xzp07y7xMAAAAAABQeZU5lFi7dq26d++ukJAQ2Ww2DRkyRD4+Pvrhhx8Knf6HH36Qj4+PhgwZIpvNppCQEHXv3l1r1qwp8zIBAAAAAEDlVaZQIjs7W8eOHVNQUJDD+4GBgTp06FCh8xw+fFiBgYEO7wUFBenYsWPKzs4u0zIBAAAAAEDlVaZQ4ty5c8rJyZGXl5fD+97e3rLb7YXOY7fb5e3t7fCel5eXrl69qoyMjDItEwAAAAAAVF7letClaZGRkYqMjJQkTZkyRb6+vi5dfvKTnYr9+faVO0pcxvyII9e94xiwTJo0qdj5q+x8qcTPyAmeW+I0FUF561mwlpKr63mr1FIqSz2dq6V0+9STfd05rqjn9dzd3ZWdnZ3vndujnjdjXz+evLDEz5gcerLYn1eGWko3Z1+nnv9zM+pZUi2lylHPm7GvS+WvZ2WopUTf6Wrs6651M66JKtu+XqZQonbt2qpSpYrS09Md3i9suo9XygAAIABJREFUNESewkY8pKenq2rVqqpVq5YkOb3MkJAQhYSEWK9v9gMxXfF5JS3D7ya1oyKoCPW8VWoplX9dSjP/7VLPirBtuqodFUFZ1sPX19ep+ajn7dcGV6go61FR2lFeFWU9Kko7yqsirEdFaIOrVIR1qQhtcIWKsh4VpR3lVRHW40a0oWHDhkX+rEy3b7i7u6tFixaKiopyeD86OloBAQGFztOqVStFR0c7vBcVFaUWLVrI3d29TMsEAAAAAACVV5n/+sYTTzyhLVu2aNOmTUpMTNT8+fOVlpamhx56SJL06aef6tNPP7Wmf/jhh5WWlqavvvpKiYmJ2rRpk7Zs2aKePXuWepkAAAAAAODWUeZnSnTq1EkZGRn697//rbNnz6px48YaN26c6tWrJ6ngkA8/Pz+NGzdOCxYssP486ODBgxUcHFzqZQIAAAAAgFtHuR50+cgjj+iRRx4p9GcTJkwo8F7btm31wQcflHmZAAAAAADg1lHm2zcAAAAAAADKg1ACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMMLddANudT37eTu89vX1VWpqqqHWVG7X11KinuXBtula1BMAAABwHiMlAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADDCvSwz5ebmavny5dq0aZPOnz+vVq1aaejQoWrcuHGx8+3cuVPLli1TcnKy6tevr/79++vee++VJGVnZ2vp0qXau3evkpOT5enpqXbt2un555+Xr69vWZoJAAAAAAAqsDKNlFi1apXWrl2rwYMHKzw8XLVr19bkyZN16dKlIueJi4vTxx9/rK5du2rq1Knq2rWrpk+frsOHD0uSMjMzdfz4cfXt21cffPCB3njjDZ05c0bvvfeerl69Wra1AwAAAAAAFZbToURubq7Wr1+vPn36KDg4WE2aNNGoUaN06dIlbdu2rcj51q1bp3bt2qlv376y2Wzq27ev2rVrp3Xr1kmSatSoofHjx6tTp05q2LChWrZsqWHDhikpKUlJSUllX0MAAAAAAFAhOR1KpKSkyG63KzAw0HrPw8NDbdq00aFDh4qcLy4uTkFBQQ7vBQUFKS4ursh5Ll68KEm68847nW0mAAAAAACo4JwOJex2uyTJ29vb4X0vLy+lp6cXO5+Xl1eBefKWd73s7GwtWrRI99xzj+rWretsMwEAAAAAQAVX4oMut27dqjlz5livx40bd0MbJElXr17VjBkzdOHCBb3xxhtFThcZGanIyEhJ0pQpU1z+QMzkEn5els9zd3d3br4jJU9SWR4EWhnqeavUUnJ+XZyupXTb1LMibJtlbYcJ1NN1XLOvFx7+u1JlqKXkqm2Teuahnq7Dvu5a1NO12Nddq/z1vPVqWWIo0bFjR7Vq1cp6nZWVJenayIf8jU1PTy8wEiI/b2/vAiMp0tPTC4y4uHr1qj755BMlJCRowoQJqlWrVpHLDAkJUUhIiPU6NTW1pNVxqbJ8nq+vr1Pz+d2gdlREFaGet0otJefXxdlaSrdPPSvCtlnWdlRE1NO1KsJ6VIQ2uEJFWY+K0o7yqijrUVHaUV4VYT0qQhtcpSKsS0VogytUlPWoKO0or4qwHjeiDQ0bNizyZyXevuHp6Sl/f3/rn81mk7e3t6KioqxpMjMzdfDgQQUEBBS5nNatWzvMI0lRUVFq3bq19To7O1sfffSRfv/9d73zzjsFAgsAAAAAAHDrcPqZEm5ubgoNDdWqVau0a9cuJSQkaNasWapevbq6dOliTTdp0iQtWbLEeh0aGqqYmBh99913SkpK0sqVKxUbG6vHH39c0rUREnl/IvRvf/ub3NzcZLfbZbfblZmZ6YJVBQAAAAAAFUmJt28Upnfv3srMzNS8efN04cIFtWzZUm+99ZY8PT2taZKTkx0eUBkQEKAxY8Zo6dKlWrZsmfz9/TVmzBjr1pAzZ87oP//5jyRp7NixDp83YsQI9ejRoyxNBQAAAAAAFVSZQgk3NzeFhYUpLCysyGkiIiIKvBccHKzg4OBCp/fz89M333xTluYAAAAAAIBKyOnbNwAAAAAAAFyBUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwwt10AwAAgDmjR48ueaIj4258Q24R1NO1SqwntXQK9XQd9nXXYtt0rcpWT0ZKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABghLvpBlRkVeeuNt2EWwr1dB1q6VrUEwAAADCDkRIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACPcTTcAAABUbCktw0034ZZCPV2HWroW9XQt6uk61NK1Klo9GSkBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADDC3XQDAAC3lqpzV5tuAgAAACoJRkoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjHAvy0y5ublavny5Nm3apPPnz6tVq1YaOnSoGjduXOx8O3fu1LJly5ScnKz69eurf//+uvfeewudds6cOYqMjNSAAQPUq1evsjQTAAAAAABUYGUaKbFq1SqtXbtWgwcPVnh4uGrXrq3Jkyfr0qVLRc4TFxenjz/+WF27dtXUqVPVtWtXTZ8+XYcPHy4w7c6dO3XkyBH5+PiUpXkAAAAAAKAScDqUyM3N1fr169WnTx8FBwerSZMmGjVqlC5duqRt27YVOd+6devUrl079e3bVzabTX379lW7du20bt06h+lOnz6t+fPna/To0XJ3L9NADgAAAAAAUAk4HUqkpKTIbrcrMDDQes/Dw0Nt2rTRoUOHipwvLi5OQUFBDu8FBQUpLi7Oen316lV98skneuqpp2Sz2ZxtGgAAAAAAqEScHopgt9slSd7e3g7ve3l56ezZs8XO5+XlVWCevOVJ0jfffKNatWrp4YcfLlVbIiMjFRkZKUmaMmWKfH19SzWfSe7u7s6180jJk1SG9b5RXF1Paunk+lPPIrGvu9btWs/kUkxT8nrYi/1pZaiDq5RUz9LVgnrmoZ6uczP29dIt49ZA3+la7OuuVf563nr7eomhxNatWzVnzhzr9bhx425IQ2JjY7VlyxZ9+OGHpZ4nJCREISEh1uvU1NQb0TSX8vX1daqdfqWYpjKs943i6npSS+fWn3oWjX3dtahn0cq7HrdKHVzBFbWgnv9DPV2LeroWfafrsG261q1az4YNGxb5sxJDiY4dO6pVq1bW66ysLEnXRj7kT2DS09MLjITIz9vbW+np6Q7vpaenWyMuYmNjZbfbNWzYMOvnOTk5Wrx4sdavX6/PP/+8pKYCAAAAAIBKpMRQwtPTU56entbr3NxceXt7KyoqSi1btpQkZWZm6uDBgxowYECRy2ndurWioqIc/rxnVFSUWrduLUl65JFHFBwc7DDPe++9p86dOzuMhgAAAKXXs5/j7ZZlGRWF/6GerkU9XYdauhb1dC3q6TrX11Kq/PV0+kGXbm5uCg0N1apVq7Rr1y4lJCRo1qxZql69urp06WJNN2nSJC1ZssR6HRoaqpiYGH333XdKSkrSypUrFRsbq8cff1zStedLNGnSxOGfu7u7vL29ix3qAQAAAAAAKqcy/c3N3r17KzMzU/PmzdOFCxfUsmVLvfXWWw4jKpKTk1W3bl3rdUBAgMaMGaOlS5dq2bJl8vf315gxYxxuDQEAAAAAALePMoUSbm5uCgsLU1hYWJHTREREFHgvODi4wC0axSlsGQAAAAAA4Nbg9O0bAAAAAAAArkAoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBHuphsAAAAKV3XuatNNuKVQT9einq5DLV2LeroW9XQt6lkQIyUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACHfTDQAAwNVSWoabbgIAAABKgZESAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABjhXpaZcnNztXz5cm3atEnnz59Xq1atNHToUDVu3LjY+Xbu3Klly5YpOTlZ9evXV//+/XXvvfc6THPixAktWbJEMTExys7OVqNGjfTXv/5VNputLE0FAAAAAAAVVJlGSqxatUpr167V4MGDFR4ertq1a2vy5Mm6dOlSkfPExcXp448/VteuXTV16lR17dpV06dP1+HDh61pUlJSNH78ePn5+entt9/WtGnT1K9fP1WvXr0szQQAAAAAABWY06FEbm6u1q9frz59+ig4OFhNmjTRqFGjdOnSJW3btq3I+datW6d27dqpb9++stls6tu3r9q1a6d169ZZ03z99dcKCgrSCy+8oBYtWqh+/frq0KGDfH19y7Z2AAAAAACgwnI6lEhJSZHdbldgYKD1noeHh9q0aaNDhw4VOV9cXJyCgoIc3gsKClJcXJwkKScnR7/99ptsNpvee+89DR06VOPGjdOOHTucbSIAAAAAAKgEnA4l7Ha7JMnb29vhfS8vL6Wnpxc7n5eXV4F58pZ37tw5Xb58WStXrlRQUJDGjx+vzp07a8aMGdq9e7ezzQQAAAAAABVciQ+63Lp1q+bMmWO9Hjdu3A1pSE5OjiSpY8eOeuKJJyRJzZo109GjR7VhwwZ16NChwDyRkZGKjIyUJE2ZMqVS3Obh7u7uXDuPlDxJZVjvG8XV9aSWTq4/9SwS+7prlWn7RKGopWtRT9einq5DLV2LeroW9XStyl7PEkOJjh07qlWrVtbrrKwsSddGPuRf8fT09AIjIfLz9vYuMJIiPT3dGnFRu3ZtVa1atcBf2WjUqFGRt3CEhIQoJCTEep2amlrS6hjn6+vrVDv9SjFNZVjvG8XV9aSWzq0/9Swa+7prlWX7ROGopWtRT9einq5DLV2LeroW9XStylDPhg0bFvmzEm/f8PT0lL+/v/XPZrPJ29tbUVFR1jSZmZk6ePCgAgICilxO69atHeaRpKioKLVu3VrStXTnD3/4g06cOOEwzcmTJ1WvXr2SmgkAAAAAACoZp58p4ebmptDQUK1atUq7du1SQkKCZs2aperVq6tLly7WdJMmTdKSJUus16GhoYqJidF3332npKQkrVy5UrGxsXr88cetaXr16qUdO3YoMjJSp06dUmRkpHbs2KFHHnmknKsJAAAAAAAqmhJv3yhM7969lZmZqXnz5unChQtq2bKl3nrrLXl6elrTJCcnq27dutbrgIAAjRkzRkuXLtWyZcvk7++vMWPGONwacu+992r48OFauXKl5s+frwYNGmjkyJGFPk8CAAAAAABUbmUKJdzc3BQWFqawsLAip4mIiCjwXnBwsIKDg4tddo8ePdSjR4+yNAsAAAAAAFQiTt++AQAAAAAA4AqEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAI9xNNwDFS2kZbroJtxTq6VrU03WoJQAAAG5HjJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIQgkAAAAAAGAEoQQAAAAAADCCUAIAAAAAABhBKAEAAAAAAIwglAAAAAAAAEYQSgAAAAAAACMIJQAAAAAAgBGEEgAAAAAAwAhCCQAAAAAAYAShBAAAAAAAMIJQAgAAAAAAGEEoAQAAAAAAjCCUAAAAAAAARhBKAAAAAAAAIwglAAAAAACAEYQSAAAAAADACEIJAAAAAABgBKEEAAAAAAAwglACAAAAAAAYQSgBAAAAAACMIJQAAAAAAABGEEoAAAAAAAAjCCUAAAAAAIARhBIAAAAAAMAIt9zc3FzTjQAAAAAAALcfRkrcZGPHjjXdhFsK9XQdaula1NO1qKfrUEvXop6uRT1dh1q6FvV0LerpWpW9noQSAAAAAADACEIJAAAAAABgRNUJEyZMMN2I202LFi1MN+GWQj1dh1q6FvV0LerpOtTStaina1FP16GWrkU9XYt6ulZlricPugQAAAAAAEZw+wYAAAAAADCCUAIAAAAAABhBKIHbRmxsrMLCwnTu3DnTTQFQDhEREZoyZYrpZtxS6B9hwsSJE/XTTz+Zbkahdu/erX/84x/Kycm5aZ85cOBAbdmy5aZ9njPCwsK0c+dO080ok2+++UZ///vfb+pnjhw5UqtXr76pnwkzv+vb3d///nd988035V6OuwvaUqnZ7XZ999132r17t86cOSNPT0/5+/urc+fOuv/++1W9evVC59uyZYvmzZunRYsWlfqzYmNjNXHiRH3xxReqXbu2q1ahUrLb7Vq5cqVV91q1aqlp06Z69NFH1aFDB9PNq/QiIiKUkZHh8r9ZXJbtvqKJiIiwToKrVq2qO++8U40bN9Z9992nkJAQubu7rlssa71SUlI0atQol3TyN1P+2lapUkU+Pj7q0KGD+vfvr5o1a7rscwYPHqzK9DikytDfBQQEaM6cOapVq5bpppTKzdyPTQoLC9Nrr72m4OBgly97//79WrNmjY4dO6azZ89qxIgR6tGjh8M0ly9f1pIlS/TLL78oIyNDvr6+euihh/TEE09I+l9fVZgBAwaoV69eRX7+7t27lZqaqq5du1rvRUZGavv27Tp+/LguXryoTz/9VH5+fg7zHTt2TIsXL9bRo0dVpUoV3XffffrLX/7icL4WHR2tZcuWKSEhQXfccYe6d++u/v37q2rVqpKuXbisWLGi0HbNnTtXXl5e6tChg5YtW6Zt27apW7duRReyEPm3z/xatWql9957z6ll3WgjR47U6dOni/x527ZtVZGfiV/Y+c5vv/2mjz76SE888YSeffZZ9erVS4899tgN+fzKcl5U1uudimLLli2aNWtWsdO88847N6k1RTN5vL+Rx4ub4dY4apdRSkqKxo8frxo1aqhfv35q2rSpPDw89N///lebNm1SrVq11KVLF9PNvOXk1d3T01P9+/dXs2bNlJOTo5iYGM2dO1efffZZgXmys7NvmZNMmNe+fXv99a9/VU5Ojs6dO6eYmBgtX75cW7du1fjx4yv8wbkiy6vt1atXlZiYqM8++0wXLlzQmDFjXPYZNWrUcNmybrSy9HcmuLu7y9vbu8if531bXKVKxRlgWVn344pSy8uXL6tx48bq3r27Pv3000KnWbBggaKjozVq1Cj5+fnpwIEDmj17tmrXrq1u3brJ19dXc+bMcZjnl19+0bx580o8MV6/fr169OjhUIcrV64oMDBQHTt21IIFCwrMk5aWpnfffVd//vOfNXToUF28eFELFixQRESE9e1ofHy8wsPD1adPH40aNUppaWmaO3eucnJy9MILL0iSevXqpYcffthh2R9//LHc3Nzk5eVlvXf//ffr+++/dzqUkP63feZXEc9jwsPDrW0yPj5e77//vt5//335+vpKqphtLs7PP/+szz//XAMGDFBoaKgkqXr16hW2P7gZboXrnU6dOulPf/qT9XrmzJmqWbOmBg8ebL1Xs2ZNxcbGmmiepMpzvK+oKldP42JffPGFqlSpovDwcIfOys/PT/fcc88N/yYuJydHs2fPVkxMjOx2u+rWrasHH3xQPXv2tA7SeQlwYGCgVq1apczMTP2///f/NHToUN1xxx2SpNzcXK1evVqRkZFKS0uTv7+/evfuXaaD6M0wb948SdKUKVMc6m6z2axvTMLCwjRkyBDFxMRo3759euihhzRgwIAS65WQkKCvvvpKR48eVU5Ojvz9/fWXv/xFd911l/U5v//+u77++mslJCTIZrNp2LBhlfpP6Dhr//79+te//qXff/9dNWrUUOfOnTVgwADrxGP//v1avHixEhISVKVKFTVs2FCvvPKKMjIyrJQ6LCxMkvT0009b/1+ZVKtWzboAq1Onjpo1a6bAwEC9+eabWr16tcLCwvTzzz/r+++/V1JSkjw8PNS2bVsNGjRIderUkfS/kU/jx48vdHuKjY0tsl7Z2dlaunSptm3bpvPnz6tx48bq16+fwwE3v4sXL2revHnat2+fLl26JB8fHz322GN6/PHHb0K1nJO/tnXr1lWnTp2s4cil6fOuXr2qRYsWWd8ydu/eXVlZWUpKSrK+rbv+m7EJEybIZrOpRo0a2rRpk9zc3NStWzcNGDDAWq7dbtfs2bMVFRUlLy8vPfPMM1q7dq3uu+++G7oNl6a/S01N1fz58xUdHS1JCgwM1ODBg1W3bl1J177V3bVrl/r27aulS5cqPT1dd911l15++WVr1F1xfV/etvrmm29q6dKlOnHihGw2m4YPH271fdeP5Mv79u/VV1/V4sWLlZSUpKlTpyozM1NLly7V8ePHlZ2drSZNmmjgwIFq3bq1tW4XL17U4sWL9euvv+rChQvy8/PTM888ow4dOmj48OF65ZVXHC5Yo6KiFB4ers8++6zYYOR6pdmPS9rXsrOztXDhQu3atUsZGRny8vJSly5d9Pzzz0u69i1y9+7dderUKf3666+qXr26evbs6TAC4OLFi1q0aJF+/fVXZWZmqnnz5nrhhRf0hz/8QZLKXMuRI0dKkqZPny5JqlevniIiIiRJ//nPf7R8+XIlJibK29tbXbp00TPPPOPUBWSHDh2sb+7ylnu9uLg4devWzTqG+vn5afPmzTp8+LC6deumKlWqFPid7dq1S+3bty8wwiG/c+fOKTo6WgMGDHB4P69PO3r0aKHz7d69W1WqVNGLL75o7dsvvfSSXn/9dZ06dUr+/v7asWOHbDabtV/7+/vr+eef10cffaRnnnlGnp6eBS7V4T7dAAAgAElEQVRSU1NTdeDAgQIhQseOHfXll19ay3ZG/u2zMKdOndLnn3+uw4cPy9fX1wpM8jt8+LC++OILJSYmqlGjRnr22Wc1ZcoUvfPOO2rXrp0kKTExUYsWLdKBAwfk4eGhu+66S4MGDSr1vpR/5G7eSKnatWsXOv/58+c1ffp07dmzR15eXgoLC3M410xLS9PChQu1b98+SVLr1q01aNAgNWjQoFRtKa9169Zp8eLFevnllx3aldeHTps2TVLx/WVJfUJ+xR3nJSkrK0tz5szR9u3b5enpqdDQUKf6Dldx5npn7dq12rJli5KTk1WjRg3dfffdGjhwoO68805J/+vPXnvtNS1YsECpqalWABcVFaUlS5YoPT1dHTt21PDhw+Xh4SGp/NcqHh4e1rKka/uXh4dHkdv59u3bizxeStKPP/6o1atXKyUlxRoBFhoaWq6wuDTHe6n0x/zQ0FCtWLFC586dU1BQUIF1cEZGRobmzZungwcPKiMjQ/Xr11fPnj11//33W9OU5lwqPT1ds2fP1r59++Tl5aWnn366TO0pzG0bSmRkZGjfvn3q379/kempm5vbDW1DTk6O6tSpo1dffVW1a9fWkSNHrOGzDzzwgDXdgQMH5O3trfHjx+vMmTP66KOP1KBBAz355JOSpKVLl2rnzp0aOnSoGjZsqLi4OM2ePVs1a9asMEOD85w/f1579+5Vv379Cq17XqcnSStWrFD//v01cOBAubm5lapen3zyiZo2bar3339fVatWVUJCgkMnJklLlizR888/Lx8fH3311VeaOXOmpk+ffsN/3xVBWlqawsPD1bVrV40YMULJycn6/PPPVaVKFb3wwgu6evWqPvzwQ91///3Wt93Hjx9XlSpVFBAQoEGDBunrr7/WzJkzJemW+uahSZMm+tOf/qRdu3ZZFzPPPPOMGjVqpIyMDC1evFiffPKJJk6c6DBfUdtTcfWaNWuWkpOTNXr0aNWtW1d79uzRBx98oPDwcDVr1qxA25YuXaqEhASNHTtWXl5eSklJqRT3/icnJ2vv3r3WkOnS7MNr1qzRTz/9pOHDh6tJkybauHGjtm3bpubNmxf7WVu3blVoaKjeffddxcfHa8aMGWrRooX17U9ERITsdrveeecdeXh4aOHChcUOWXaF0vR3OTk5mjp1qjw8PKyhp19++aU+/PBDhYeHW/1SSkqKduzYoddff11XrlzRxx9/rKVLl2rYsGGSStf3LVq0yArWVqxYoSlTpmjmzJlWwH29rKwsffvtt3rppZdUu3Zt+fj46OjRo+rWrZsGDRokNzc3bdiwQeHh4ZoxY4Zq1aql3NxchYeH6/z58xoxYoQaNGigEydOKCsrS9WrV1fnzp31448/OoQSmzdvVocOHZwKJIpy/X5c0r72/fff69dff9Xf/vY3+fn56cyZMzpx4oTDMtetW6fevXvr6aefVmxsrL788kvVr19f9913n7W+NWrU0NixY1WzZk1t2bJFkyZN0scffywfH58y1zI8PFwvvviihg8frnvuucc6Kdy7d69mzpypQYMGqU2bNkpNTdXcuXOVlZVV6IVteQQEBOi3337TAw88IF9fXx06dEjx8fFF3paRnJysmJgYvfrqq8Uu9+DBg3J3d1eTJk2cak9WVpaqVq3qcOGQt50fPHhQ/v7+ys7OVrVq1Rzm8/DwUFZWlo4dO2ZdzOe3efNm1axZU/fdd5/D+76+vvLy8tL+/fudDiWKk5OTow8//FA1a9bU5MmTdeXKFX311VfKzs62prl8+bKmTJmiwMBAjRo1SmfPntVXX33lsJyzZ8/qnXfe0f3336+BAwfq6tWr+vrrrzV16lRNnjzZ5aNxVqxYoeeee07PPfecNm/erM8++0xt27aVr6+vrly5ookTJ6p169aaMGGC3N3dtWbNGr377rv66KOPiuxnXGXp0qVau3atXn/99RLPfYvrL0vTJ+Qp6bxo3bp1CgsLU69evbRnzx7Nnz9ff/zjH9W6detS9x3l5ez1jpubmwYNGiQ/Pz+lpqbqyy+/1JdffukQ2GVnZ2vt2rUaPXq0srOzNW3aNE2bNk3VqlXT3//+d2VkZGjatGnauHGjevbsKenmXquUdLyMjIzUN998oyFDhqhFixZKSEjQ7Nmz5e7urkcffbRMn1na6xtnjvlbt27VG2+8oStXrmjOnDn67LPP9Oabb5apfVlZWWrRooX69OkjT09PRUdHa86cOfL19VX79u2t6Uo6l5o1a5ZOnz6t8ePH64477tCCBQuUkpJSpjZd77YNJU6dOqXc3Fw1bNjQ4f2XX35ZFy5ckCR17drV2oBvBHd3d/Xr18967efnp+PHj2v79u0OoUSNGjU0bNgwValSRTabTcHBwYqJidGTTz6py5cva+3atfrnP/+pNm3aWMs5cuSINm7cWOFCiby622y2Eqft1KmTHnzwQYf3SqpXamqqevbsqUaNGklSoScR/fr1s771eeqpp/T2228rLS3NSihvZRs3bpSPj4/1LZPNZtPzzz+vOXPmqF+/fsrKytKFCxfUsWNHq3Z5tZT+N2zeFRcPFZHNZrOS6/z7YP369fXiiy/q1Vdf1ZkzZxy2leK2p8LqderUKW3fvl0RERHW8NhHH31UUVFRioyM1Isvvig/Pz+H50mcPn1azZs3V8uWLSVd+8a0otq7d68GDhyonJwcZWVlSZJ1oVSaPm/9+vXq3bu3ddE6aNAg7d27t8TPtdls1rIbNmyoTZs2KSYmRl26dNGJEye0b98+TZ482foWesSIEdY30TdKafq7mJgY/f7775o5c6b1zfLo0aM1evRoRUdHKzAwUNK1E5mRI0da21RISIh+/PFHazml6fueeuopa4TAiBEj9PLLL2vbtm0F+tk8OTk5Gjp0qMNIsvyjziRpyJAh2rVrl/bs2aNu3bopOjpacXFxmjZtmrXe9evXt6Z/8MEH9dZbbyktLU116tTR+fPn9euvv+q1114rskbOytuPS7OvnT59Wg0aNFCbNm3k5uYmX19fBQQEOCyvZcuW6tu3r6Rr29bRo0etUTaxsbGKj4/XvHnzrIuaZ599Vr/99pt+/vln9e7du8y1zPtG7M4773ToQ1auXOnwDVfeSICZM2daIb6rDBkyRHPmzNGIESOscHHw4MG65557Cp1+06ZNql27tjp27Fjsck+fPi0vLy+nL5rvuusuLVy4UN99952eeOIJXb58WYsXL5Z07QJdkoKCgrRu3Tr9/PPP6ty5s9LT0/Xtt986TJNfTk6OfvzxR3Xt2rVAmCFdG4VTlpPuvL4wv0ceeUQDBgxQdHS0EhMTHbbNQYMG6e2337am3bp1q3JycvTKK6/Iw8NDjRs3Vt++fTVjxgxrmh9++EFNmzZ1GHEyatQoDRkyRMeOHbOOGa7SrVs365vtfv36af369dq/f7+6deum7du3Kzc3VyNGjLC2wWHDhunFF1/Ub7/9pk6dOrm0LflFRUVp9+7dGjt2bKnOe4vrL0vTJ+Rxd3cv9rwoMDDQush97LHH9P333ys6OlqtW7cudd9RXs5e7+Qfgenn56cBAwZo6tSpGjlypMOIxrxwQZI6d+6sdevWae7cuVa/1bFjR8XGxqpnz543/VqlpOPlt99+qwEDBljnGX5+fkpOTtbGjRvLHEqU9vqmtMf8zMxMjRo1yuofhg0bprffflsnT54s08ijOnXqOITJ9evXV0xMjLZv3+4QSpR0LrVnzx5NmjRJf/zjHyVdG9FX1HOFnHXbhhJFmTRpkjXEOO+E+kb64YcftHnzZp0+fVqZmZm6evVqgQsOm83mcOCuU6eOjhw5IunasL2srCy9//77DvMUtpyKwJlbYgq7paKkej3++OOaPXu2fvrpJ7Vv31733Xefw0W1JDVt2tT6/7yh+Onp6bdFKJGUlKRWrVo5bE9//OMflZ2drVOnTqlp06bq0aOH3nvvPd11111q3769goODrU7xVpebm2udTB07dkwrVqxQfHy8zp8/b227qampDtuKs9vT8ePHlZubW+CbxOzs7AIXKXkefvhhTZ8+XcePH1f79u3VsWNHtW3btuwregO1adNGw4cPV2ZmpiIjI5WcnGzd1ysVvw9fvHhRdrvd4UTazc1NLVu21JkzZ4r93Py/B0ny8fFRenq6pGvbvZubm8OQWF9fX+v3daOUpr9LTExUnTp1HIa6169fXz4+PkpMTLROUHx9fR2epeHj4+MwWqY0fV/+WyyqV6+uJk2aKDExsci2Va1atcDInfT0dC1btkyxsbGy2+3KyclRZmamUlNTJV3bvr29vYs8MfvDH/6gJk2aaMuWLerbt6+2bdummjVr6u677y6hUqWXtx+XZl/r0aOHJk+erL/97W8KDAxUhw4d9Kc//cmhj8xft7zXu3btknStn8jMzNTQoUMdpsnKylJycrL1uiy1LMqxY8d05MgRrVq1ymGdMzMzZbfbXfYNq3TtW+NDhw7pjTfeUL169XTgwAEtWrRIfn5+BW43u3r1qrZs2aLu3buXeBtJZmZmoQFASRo3bqyRI0dqwYIF+vrrr1W1alU99thj8vLysvruoKAgDRw4UPPmzdOsWbNUrVo1PfXUUzpw4EChIcjevXt15swZhYSEFPqZHh4eyszMdLqteX1hfnn7cFJSkurUqeNwbG3ZsqVDoJSUlKQmTZo4jHi6PmQ4duyYDhw4UCD8kK5dJLk6lMg/sqVq1aqqXbu21Q8dO3ZMKSkpBUbrZGZmOuwLN0Ljxo118eJFLV++XAEBAQ6jbgtTXH9Zmj6htIo7LpW277hRirreiYmJ0cqVK5WUlKSLFy8qJydH2dnZstvt1jGzWrVqDiGHt7e3vL29HW4t8PLyso4vN/tapbjj5blz53TmzBnNmTNHc+fOtabJyckp1237pZ23tMf8ovqHpKSkMoUSOTk5+u6777Rjxw6lpaUpKytL2dnZBUaOleZcKn+/Uq9ePZedS922oYS/v7/1y80vbyO50cPMJGnHjh1asGCBdQ9pjRo1tGHDBv36668O0+V9O5Ff3saf998333yzwIVjYfOZ1qBBA7m5uSkxMVH33ntvsdNeP/ypNPUKCwtT165dtWfPHu3bt0/Lly/XSy+95PCtd3H1vJ3lnQyNGDFCoaGh2rt3r/7zn//o66+/1j/+8Y8in3dwK0lMTJSfn58uX76s9957T+3bt9eoUaPk5eWljIwMvf322w7DayXnt6e8C6bw8PACJ+7XD7fPc/fddysiIkJ79+5VdHS0wsPD9ec//1kjRowow1reWHfccYf1rdOQIUM0ceJErVixQmFhYaXu88ri+t+Dm5ub8f3amf6uMPkvUK7fVvJuactTmr7PWe7u7gVOxCMiIpSenq6//OUvqlevnqpVq6ZJkyYV2C+K88ADD+j7779X37599eOPP6p79+4uHWaetx+XZl9r0aKFIiIitG/fPkVHRysiIkJNmzbVP//5z1K1KScnR15eXpo0aVKBn3l6elr/78pa5uTk6Omnn9af//znAj9z5V/2yszM1JIlS/Taa69ZIx+aNm2q+Ph4rVmzpsAx4bfffpPdbi/VNlerVi3rW1pndenSRV26dJHdbrfOE9auXeswIueJJ57Q448/rrNnz6pmzZpKSUnRkiVLCn3ORWRkpAICAooM0s6fP1+muubvC2+U3Nxc3X333YXetpP/gZ2uUlw/lJubq2bNmhX6UGNX/vWlwvj4+OjNN9/UxIkT9e677+qf//xnsZ9ZXH9Z3j4hv+KOS6XtO8rLmeud06dPKzw8XA8++KD69eunmjVr6vjx4/rkk08c+qXC6lDYtmHqWqW47TTvvy+99FKRI2DKorzH+7x23iirV6/WmjVrNHjwYDVp0kTVq1fXkiVLCtwKXJpzqRvVzorzGO2brFatWgoMDNSGDRt0+fJlI204ePCgWrZsqUcffVQtWrSQv7+/0+mozWZTtWrVdPr0afn7+zv8q4gjJWrWrKmgoCBt3Lix0LoXd5JS2no1aNBAoaGhGjdunB544AFt3rzZpetQmTVq1EiHDx92uJjJu7c3/wlds2bN1KdPH02YMEHt2rWzHjro7u5+U/9m+82UkJCgffv2KTg4WCdOnFBGRoaee+45tW3bVo0aNbKSYmcUVq9mzZopNzdXdru9wD5bXNqc97T7kSNH6pVXXtFPP/10U0ZzldfTTz+tVatWKS0trcR9uEaNGvL29rZGgknXTmaKeuhdaTVq1Ei5ubk6duyY9d6ZM2eUlpZWruWWpDT9nc1mU1pamsPw8OTkZJ09e7ZUt7nlV1Lfd/jwYev/L1++rP/+978FRlOU5ODBg9afNmvcuLGqV6/uMCS+efPmstvtxY7A6Nq1q86cOaMNGzbo+PHjDg/aKq/8+3Fp9zVPT08FBwfrpZde0tixYxUTE6NTp05ZP89fN+nawx/zfjctWrRQenq63NzcCnxGSReFJdVSunaCeH0f0qJFCyUlJRX4PH9/f5ee4GdnZ+vq1asFLkCqVKlS6HFg06ZNatu2bYFh4oVp3ry5zp07V65n43h7e6t69erasWOHPDw8rG8Y87i5ualOnTry8PDQ9u3bVbdu3QIjMNPS0rR79+4ib2HKzMzUqVOnXP4w7EaNGiktLc1hVMyRI0ccTv4bNWqkhIQEh1Ea+ftG6VodExMT5evrW2BbcOWFbWk0b95cp06dUq1atQq05UaHEtK1b5YnTJigK1eu6N1331VGRkax0xfXX5bUJ+RX1vOi8vQdznDmeufo0aPKzs7WoEGD1Lp1azVs2LDQW56cVZGuVby9veXj46Pk5ORC+9CyKu31TWmP+UX1D84es/McPHhQ99xzj7p166ZmzZqpfv36OnnypFPLyDuXyt8Ppaamuuxc6rYNJSTpxRdfVG5urt58801t27ZNiYmJOnHihLZt26bff//d4UD86aefFvkns6RrG8+YMWP0yy+/lPrzGzRooOPHj2vPnj06efKkVqxYof379zu1Dp6enurZs6cWLVqkzZs369SpU4qPj9cPP/ygyMhIp5Z1swwdOlS5ubkaO3as/u///k8nTpxQUlKSfvjhB73++utFzldSvTIzM/XFF18oNjZWKSkpOnz4sA4ePOj0if2t4tKlS4qPj3f4d/fdd+vs2bPW07x3796txYsX69FHH9Udd9yhlJQULV68WIcOHdLp06ete9/yalivXj1lZWUpKipK586d05UrVwyvZdlkZWXJbrcrLS1N8fHxWrt2rSZOnKgWLVqoZ8+e8vX1VbVq1bRhwwYlJydr9+7dWrZsmdOfU1i9GjZsqC5dumjWrFnauXOnkpOTdfToUa1evdoaEn69ZcuW6ZdfftHJkyeVmJioXbt2yc/Pr0zDn2+2du3ayWaz6d///nep+rzQ0FCtXr1av/zyi06cOKGFCxfq7Nmz5UrmGzZsqKCgIM2dO1dxcXGKj4/XrFmzdMcdd9zwB9yW1N+1b99eTZs21cyZM3X06FEdPXpUM2bMUPPmzYu8ned6pe37vv32W0VFRem///2vPvvsM7m7uzv9Z+AaNGigrVu3KjExUUeOHNEnn3zi8K3UXXfdpZYtW2ratGnau3evUlJSFBUV5XBsvPPOOxUcHKyFCxeqTZs2ZX4yf0n7cWn2tbVr11rH/1OnTmnbtm3y9PR0uP3q8OHDWrlypU6ePKnIyEj9/PPP1n3X7du3V0BAgKZOnao9e/YoJSVFcXFx+uabb3TgwIFy1VK69m1mdHS07Ha7zp8/L+nas0G2b9+uZcuWKSEhQUlJSdq5c6f+9a9/OVW/y5cvW8eG3NxcpaamKj4+3joRrlGjhtq2baslS5ZY29aWLVv0008/FfgmMDU1VXv37i3y4v56zZs3l5eXlw4ePOjwvt1uV3x8vHWynJiYaN1Cl2fD/2/v3uN6vP/Hjz86vVU6RwdatdaJHBIzPrIkp/E1G6k5Hz7MYj528BG2fTA24+trbGsTFsohmbWUz+IjYcLwRQ5FChNJKaXQ4V3v3x9+XV9v75riXe9387rfbt3oel/X9X5dz17X67qu1/U6JCZy5coVcnJySExMJCIiglGjRik12d+1axfXr18nOzubn376iV9++YVJkyapVLAkJyfTokWLWludwKO/vYGBwTO9Ua3Jn4//1FTCdOzYkbZt2xIWFsa1a9fIyMhg06ZNSpVKvr6+6OrqsmbNGm7cuMHZs2eJjY0F/u9N5cCBA3nw4AGrVq3i8uXL3L59m7NnzxIeHs7Dhw8bnObn0bt3b8zNzVm+fDlpaWnk5eWRlpZGZGRkgx9+npWlpSULFixALpfz+eef11rp9bTysj5lwuOe9b7oecqOhqrv8469vT0KhYLdu3eTl5fH4cOH2b1793N/v7Y9qwQFBREXF0dCQgI5OTlcv36dgwcPSufXs6rP8019r/kymUypfFi3bh0+Pj5PvV7m5eWp3Ps/ePCANm3acP78eS5evMjNmzf58ccfGzxWTps2bfD29mbt2rXSvVRYWFidrXwb6oXtvgGP+vAsX76c2NhYtm/fTkFBAXp6ejg4ODBgwAClwU6e1sdTLpeTk5PDgwcP6lynpga85qLTv39/aWRThULBa6+9xtChQ5UGY6mP4OBgzM3NiY+PZ/369RgZGeHs7Ky2QXLUzdbWlmXLlhEbG8uWLVsoLCzE1NQUJycnlf6Xj3tavHR1dbl//z7ff/89d+/exdTUFB8fn1r7Wr4I0tPTmTNnjtKy1157jXnz5rF582bmzJlDy5Yt6dWrF6NGjQIeFYK3bt1i5cqV0lRYvXv3lvKSh4cH/fv3Z/Xq1ZSUlDTbKUHPnTsnDR7bsmVLXnrpJUaOHEm/fv3Q19fH0NCQGTNmsG3bNvbs2YOjoyPjx49X6Q/5NHXFa/r06fz8889s3ryZgoICTExMcHV1rfMh1MDAgOjoaPLy8jAwMMDd3f2ZR2DWhKFDh/L999+zevXqp5Z5Q4cOpaioiO+//x4dHR369OlD9+7dn6mlyuNmzJjBmjVrWLRoEWZmZgQHB0vxbExPK+90dHSYM2cOERER0swuHTt2ZPLkyfWuMKlv2TdmzBgiIyPJycnhpZdeIjQ0tMEz6ISEhLB27VpCQ0OxsrJi5MiRSjf+urq6zJ8/n6ioKL799lvKysqkKUEf17dvXw4dOvRc3Uuedh4DTz3XDA0NiY+P59atW+jo6ODs7Mz8+fOVmjQPGTKEP/74g59//hlDQ0OCgoKkAdJ0dHSYN28e0dHRhIeHU1xcjIWFBR4eHk+d6u5psQQYN24ckZGRhISEYGVlRVhYGN7e3sydO5edO3cSHx+Pnp4e9vb29OnTp0Hxy8rKUppNKCYmhpiYGPz8/KRBYD/44AO2bt3KN998Q2lpKa1btyY4OFhlMLj9+/djbGysMntFXXR1dfH39+fw4cNKFRx79+7lp59+kn7/6quvgEd/x5rjy8zMJCYmhrKyMtq2bcu7776rEuvTp0/z888/U1lZibOzM3PmzFEZt0ShULB//3569+5dZ5fdlJQUfH19n6lLb03+fJyVlZU049Xs2bMJDw9n/vz50pSgq1evltY1MjIiNDSU9evXM2fOHBwcHBg5ciQrV66Uyi0rKysWL17M1q1b+fLLL6moqKBVq1Z07txZWqdmut/HpxFtDC1atGDRokVs3bqVlStX8uDBAywtLfHy8nrqGA/qZGFhwYIFC1i8eDGLFi1SGjwUnl5e1qdMeNyz3hc9T9nRUPV93nFycmLixInExcURHR2Nh4cH48aNY9WqVc+dhvo8q9RM+13zb2MJCAigRYsWxMfHs23bNmQyGQ4ODs88yGWN+jzf1Peab2NjQ69evVi2bJnSlKBPU1vldGhoKMOHDycvL48vv/wSmUxGnz596N2795+2aqzN9OnTCQ8Pl+6lAgMD1TYbnI5C051uXyCHDx9mzZo1DX6bIQiC8CKbM2cOnp6eTJ48WW37vHfvHtOmTWPWrFlK01P+FdU8lKxfv16tYw48jyNHjrB27VrCw8ObZAynZzVjxgwGDhxY5xSYwrMrLi7mo48+YunSpbWO9aBpxcXFfPjhh3z11Vdak74TJ06wYsUKpVkOniY5OZmtW7eyatWqJq0cEISGmj59Ov379+ftt9/WdFI0KiYmht9//53/+Z//0XRSmtQL3VKiqdT0Sfz111+Vpl0RBEEQlOXn55Oamkr79u2Ry+UkJSXxxx9//Gkrqvo4f/48Dx8+xNHRkeLiYqKjozEzM3shBnDVJuXl5RQVFREbGyu9rRJeTObm5oSEhHDnzh2teeh/XH5+vjRFs6YcOHAAW1tbrK2tyc7OZuPGjXTt2rVBlYunT59mzJgxokJC0GrZ2dkYGBgwdOhQTSdF0BBRKdEEjhw5woYNG/Dw8FCZ+kcQBEH4Pzo6Ohw8eJCoqChpzu/58+crTef5LORyudQFRiaT4ebmxqJFixrcfUF4PnFxccTGxuLp6cmIESM0nRxBw2pm9dBGrq6uap9Ss6GKi4vZsWMHd+/excLCAh8fH8aMGdOgfXz00UeNlDpBUJ+XXnpJqfuS8OIR3TcEQRAEQRAEQRAEQdCIF3r2DUEQBEEQBEEQBEEQNEdUSvyJ0tJSpk6dWufcxJoUFRVFRESEppPRINocz5UrVxIfH6/pZDRIXl4eQUFBZGVl1XubAwcOvLCzkWijmJgYPv74Y00no1kLCgri2LFjdf7+V/Us578gCNpLm++RmuM9pzrMmDGDXbt2/ek6L0pZLPKn+ohY1k6MKfEnYmNj6dKlC3Z2dgBs2LCBS5cukZ2djYWFBWFhYSrbHDlyRJrP3MzMjEGDBqmM2p2YmMiePXvIy8ujVatWDB8+HD8/P+nzhQsXkpaWprJvBwcHVq5cCcCwYcOYOXMmQ4YMwdbWVp2H3Wg0FU+ABw8eEB0dze+//05JSUsDXlgAAB+2SURBVAnW1taMGjWKv/3tbwAEBgayYMECAgICMDY2bqQI1F9YWBglJSXMnTtXaXlWVhbz5s3ju+++o1WrVqxduxZTU1MNpfKvLywsjIMHDwKPpvK1trame/fuBAUFibEIntPjsQUwNTXFzc2NcePG0bZtWw2mTPOeNp2cn5+fyhSbL7KavOTv709ISIjSZ5s3b2bXrl34+PiolKeCoE3EPWfTKioq4pdffuHUqVMUFBRgZGSEnZ0dvXr1wt/fH0NDQ5YuXSoG4/3/RP5UHxHL2olKiTqUl5ezf/9+QkNDpWUKhQI/Pz+uX7/O2bNnVbY5ffo033zzDZMmTcLb25ubN28SHh6OTCaT5r7du3cvW7ZsYdq0abi5uZGZmUl4eDgtW7aUBnyaPXs2crlc2m9lZSWzZ8+mZ8+e0jIzMzM6derE3r17m8Wbb03GUy6Xs2TJEkxMTPjwww+xsrKisLBQmscewNHREVtbWw4dOvTc8xQ3FV1dXSwsLDSdjL+8jh07MnPmTORyORcvXmTNmjWUl5czdepUTSet2auJLUBhYSGbN29mxYoVfP311xpOmWatXbtW+v///u//Eh4errRMJpNRWlqqiaQBj8rUx8tPbWBtbc3Ro0eZNGmSVGFYVVXFoUOHaNWqlYZT93TaGFOh6Yh7zqaVl5fHZ599hrGxMcHBwTg5OSGTycjOziYpKQlTU1N8fX2fOsvJ43H7KxP5U31ELOsmroB1OH36NAAeHh7SssmTJwOwa9euWjPNoUOH6Nq1KwMHDgTA1taWt956i7i4OAYOHIiOjg6HDh0iICAAX19faZ2srCzi4uKkTGNiYqK0399++43y8nL8/f2Vlnfr1o1t27Zp/QkImo3ngQMHuHfvHp9//rl001fbFF/dunUjJSWl2VRK5OXl8f7777N06VJpZoJTp06xadMm7ty5g6urKwMGDGD16tV89913Ssd87tw5Nm7cSF5eHq6uroSEhGBjY0NZWRmTJk1i0aJFuLu7AxASEkKLFi1YtWoVAGfPnuW///u/2bBhA/r6+iQkJHDgwAFu376NsbExXbp0Ydy4cbRs2ZKysjKmTZtGSEgIPXr0kL7/7NmzLF26lB9++EHrK1YMDAykNPr6+nL+/HlOnDjBlClT2LVrF/v27aOwsBA7OzuGDRvG66+/Lm27ZcsWjh8/zp07d7CwsKBnz54EBQUhk8lq/a47d+6wZMkS6W+ip6fXJMeoKY/H1sLCgiFDhrBs2TIqKiooKipSyd/wqBXBRx99pJSf/moePydqpvF78jypqZTIz89n69atXLp0idatWzNp0iQ6deokrXfjxg2ioqJIT09HJpPRoUMHJk6cKO2vurqan3/+maSkJIqLi7G3t+edd97h1VdfBf6vnPnHP/5BUlISGRkZjBkzhu3bt2vVee3k5MTdu3c5evSodK08deoUBgYGtGvXTqkSJzk5mV27dklvk/r378/gwYPR1X3Uo/XPyjR41PLuxx9/JDU1lYcPH2Jpackbb7zBkCFDgNrz6IwZMxg4cKD0ZisoKIjJkydz/vx5UlNT6d+/P+PHj+fkyZPs2LGDGzduYGFhga+vLyNHjpSuXb///js7duzg1q1byGQyHB0d+fDDD7W+HBX+nLjnbFrr169HV1eXpUuXKrV6tLGxoWvXrtTMAVCf8/bJe0a5XE5kZKTUMtfc3BxfX98Gz5qiTUT+VB8Ry7qJSok6pKen4+Ligo6OTr23qaysxMDAQGmZTCajoKCA/Px8bGxsqKysVHkgkclkZGZm1vmmJCkpCW9vb5W3Pa6urhQWFpKbmys1AdJWmozniRMn8PDwICIighMnTmBiYkLPnj0ZPny4UrxdXV3ZuXMnFRUVdT40arM7d+6wYsUKBg4cSP/+/bl+/TqbNm1SWU8ul/PLL78QEhKCgYEBYWFhrFu3jk8++QRDQ0NcXFxIS0vD3d2d3Nxc7t+/z7179ygqKsLCwkL6rCZ2Ojo6TJw4ERsbG+7cuUNERAQRERHMnDkTQ0NDevXqRXJystIN+v79+/Hx8WmWN9IymYyqqiqio6M5duwYf//732nTpg0ZGRmEh4djYmKCj48PAC1atCAkJAQrKytu3LjBunXr0NfX55133lHZ740bN/jiiy/o0aMH48ePb9C58lfw8OFDjhw5gqOjY7M8/zQlOjqasWPHMmXKFHbu3MmqVav4/vvvMTQ05O7duyxYsAB/f3/GjRtHVVUV27ZtY/ny5SxZsgRdXV3+/e9/Ex8fz9SpU3FxceG3335jxYoVLFu2DGdnZ+l7am5QairLbty4oXXntb+/P8nJydINVs3/b9++La2zb98+YmJimDx5Mi4uLly/fp3w8HD09fWlh4s/K9PgUcyvX7/O3LlzMTc3Jy8vj3v37jU4vT/99BOjRo1i3Lhx6OjocObMGb799lsmTpxIu3btuHPnDuvWraOyspLx48dTVFTEqlWrGD16NK+99hplZWVcvnxZDZETNE3cczadkpISUlNTGTVqVJ3dMP/s7/DkefukX3/9lRMnTjBr1ixsbGwoKCggJydHbenXBJE/1UfEsm5ioMs65OfnY2lp2aBtvL29OXnyJKmpqVRXV5OTk0NCQgLwqO8aQOfOnUlOTiYzMxOFQkFWVhZJSUlUVVVRUlKiss+cnBzS0tIICAhQ+awmffn5+Q09vCanyXjevn2bY8eOIZfLmTdvHsHBwfznP/9h69atSt9naWlJVVUVhYWFajji53fmzBnGjRun9LNgwYI619+7dy+2trZMmDCBNm3a0KNHD/r376+yXlVVFX//+99xdXXFycmJoUOHcuHCBenNQPv27blw4QIAFy5cwNPTEzc3N86fPy8ta9++vbS/IUOG0KFDB2xsbGjfvj1jx47l6NGjVFdXAxAQEEBqaqoU19LSUk6cOEHfvn3VE6gmlJmZSUpKCl5eXiQkJPDee+/h7e2NjY0Nvr6+BAQEsGfPHmn9wMBAPD09sbGxwcfHh7fffpuUlBSV/V6+fJkFCxbQv39/JkyY8MJUSDyexydMmEBaWhr/+Mc/NJ2sZmXIkCF069YNe3t7Ro8eTWlpKdeuXQMelQlOTk6MHTsWBwcHnJyceP/998nMzOTKlSsAxMfHM3ToUHx9fWnTpg3BwcG0a9dOZXC3QYMG0aNHD2xsbLC2ttbK89rX15esrCxu3bpFUVERZ86coU+fPkrr7Ny5k7Fjx0rH0q1bN9566y2l8/ZpZVp+fj4vv/wyrq6utG7dGi8vL6Xmr/X1t7/9jYCAAGxtbbGxsSE2NpahQ4fi7++PnZ0dHTp0YMyYMfznP/9BoVBQWFhIVVWVlHZHR0cCAgKaZeWuoEzcczad3NxcFAoFbdq0UVr+3nvvSdejx7vLPenJ8/ZJ+fn52Nvb065dO1q1aoWHh4fKm+jmRuRP9RGxrJtoKVGH2mqcniYgIIDc3FyWL19OVVUVRkZGDB48mB07dkgPGYGBgRQVFfHZZ5+hUCgwNzfHz8+PXbt21fogkpSUhKWlpfTm9XE16auoqHiGI2xamoynQqHAzMyM9957D11dXVxcXCgtLWXTpk1KNd3aFs927doxbdo0pWXXr19nxYoVta5/8+ZNpWbuAG5ubirrGRgYKF2MLS0tkcvl3L9/HxMTE7y8vEhMTEQul3PhwgW8vLwoLy8nLS2NV199laysLKVmiOfPnyc2NpabN2/y4MEDqqurkcvlFBUVYWVlxSuvvIKjoyMHDhxg+PDhHD58GBMTE7p06fI84WkyNQ/ONcf16quvMnToUI4dO8aXX36ptG5VVRWtW7eWfj927Bi7d+8mNzeXsrIyqqurpQebGoWFhSxevJjAwECVQYv+6h7P46Wlpezdu5cvvviCL774QsMpaz6cnJyk/9fcSBQXFwNw5coV0tPTa22CmZubS5s2bbh7965SM1IAT09PqYlpjSfLFm08r01MTOjevTvJyckYGxvj5eWl9Abo3r17FBQUsHbtWtatWyctr66ulipl4ell2oABA1i5ciVXr16lY8eOdOvWTamitr5cXFyUfr9y5QqZmZnExcVJyxQKhdSdydnZmY4dO/Lxxx/TqVMnOnXqRI8ePZ7a713QfuKeU/M+//xzqqurCQ8Pp7Kyss71njxvn9SnTx+WLFnCrFmz6NSpEz4+Pnh7e0vdw5ojkT/VR8SybqJSog6mpqYNHkhMR0eHsWPHMnr0aIqKijAzM+PcuXMA0gimMpmM6dOn8+6771JcXIylpSX79u3DyMhI5cZCLpdz8OBBAgICau1bXpO+5nBDosl4WlhYoK+vr3RBaNu2LeXl5ZSUlEjraVs8W7RoodJs6v79+8+93ycvjDWFVc3DsqenJ3K5nKysLNLT0xk8eDDl5eWsXbuWS5cuoaenh6urK/CoFnXp0qUEBAQQHByMiYkJV69eZfXq1UqD6fTt25dff/2V4cOHk5ycjJ+fX7O5QNc8OOvp6WFpaYm+vr7UZDo0NFSl2VvNuZqRkcGqVasIDAxkwoQJtGzZkpMnTxIVFaW0vqmpKa1btyYlJYW+ffuq9Pn7K3syj7u4uDBhwgT27dtHv379AJQeFl+UQcUa4vFrw+MVsTX/dunShfHjx6tsZ25urhTbp6ltBHptPK/9/f0JCwvD0NCQ4OBgpc9qyripU6eqVMTUqE+Z1qVLF8LCwjhz5gznzp1j6dKl9OzZk+nTpwOP/g5Pxra2vPtk0/Hq6moCAwNrbXVhZmaGrq4un376KZcvXyY1NZX9+/ezdetWFi5cqNTVRmh+xD1n07Gzs0NHR4ebN28qLa9p9fC02TaeNvOWi4sLYWFhpKamcu7cOcLCwnBycuLTTz/VePn4rET+VB8Ry7o1z7OjCTg7O6sUWPWlq6uLlZUV+vr6pKSk4O7urvKH1dfXx9raGl1dXVJSUvDx8VEprI4fP05JSUmdzWGzs7PR09PD0dHxmdLZlDQZTw8PD3Jzc5XeUN+6dYsWLVooTaeZnZ2NlZVVs20K27ZtW5V5sjMzMxu8n5pxJZKSknjw4AEuLi64ublx584dDh8+rDSeRFZWFnK5nIkTJ+Lu7i69eX1S7969KSgoIDExkatXrzarpow1D86tW7eWjtvBwQEDAwPy8/Oxs7NT+qlpKXHp0iWsrKwIDAzE1dUVe3v7WpvCGRgYEBoaiomJCUuWLFFLxVNzpqurS0VFhXSO1zRNBKRuCUL9vPzyy9y4cYNWrVqp5FMjIyOMjY2xtLTk0qVLSttdvHgRBweHp+5fG8/rjh07oq+vT0lJiTRYZw0LCwssLS25ffu2SjxqKsfqW6aZmZnx+uuvM2PGDEJCQjh48KD0dtXMzExpm6KiIqV8XBcXFxdu3rxZa9pqbhx1dHRwd3dn5MiRLF26FEtLS44cOfLM8RK0g7jnbDqmpqZ06tSJxMREysrKGuU7jIyM6NGjB1OnTmXu3LmcP3+e3NzcRvmupiDyp/qIWNZNtJSog7e3N1u2bKGkpER6cK1pgn337l3kcrl0g+zg4IC+vj737t3j2LFjtG/fHrlcTnJyMkePHmXRokXSfnNycsjMzMTNzY379++TkJBAdnY2M2bMUElDUlISHTp0qHOe2PT0dNq1a9cs5lDWZDwHDBjAnj172LhxI4MGDSIvL4+YmBgGDBig1KQpPT2dzp07N01AGkH//v1JSEggMjKSfv36kZ2dzb59+4A/H7SpNu3btychIYHOnTujq6uLTCbDzc2N3377jcDAQGk9e3t7FAoFu3fv5rXXXiMjI4Pdu3er7K9ly5b06NGDyMhI2rVrh729/fMdrIYZGRkxdOhQoqKiUCgUtG/fnrKyMjIyMtDV1aVfv37Y29tTWFjIb7/9hru7O6mpqbWOJwGParhDQ0P56quvWLJkCZ9++qk00v9fWWVlpfSwVlpaKt0kdu3aVcpzcXFx2Nra8uDBA5VxYIQ/N3DgQJKSkli1ahXDhg3DzMyM27dvc/ToUcaPH4+RkRFvvvkmMTEx2NnZSQNdpqens2zZsqfuXxvPax0dHVasWIFCoVAZGAwejZ4fERGBsbExPj4+yOVyrl69SmFhIW+//Xa9yrTt27fz8ssv89JLL1FVVcXvv/+OjY2N9H1eXl7s2bMHDw8PdHV12bZtW61pedKIESNYtmwZrVu3pmfPnujp6ZGdnU1mZiZjx44lIyODc+fO0blzZywsLLh69SoFBQX1qkAStJu452xaU6ZM4bPPPiM0NJSRI0fi7OyMrq4uV65c4Y8//lCawaihEhISsLCwwNnZGX19fQ4fPoyRkRHW1tZqPIKmJfKn+ohY1k1UStTB0dERV1dXpSki16xZQ1pamrTOnDlzAJSmWzx48KDUPNvd3Z2FCxdKTd3hUfPMhIQEcnJy0NPTw8vLiyVLlqgMlnP79m3Onz/PrFmz6kxjSkoKQUFB6jngRqbJeLZq1YpPPvmEyMhI/vnPf2JhYYG/vz8jRoyQ1qmoqOD48eN88sknjReERta6dWs+/vhjIiMj2bNnD6+88gqBgYH88MMP9bohfpyXlxdxcXFK/aTbt29PWloaXl5e0jInJycmTpxIXFwc0dHReHh4MG7cOGn60Mf17duXQ4cONcsBLmsTHByMubk58fHxrF+/HiMjI5ydnRk2bBjwaEqlN998k40bN1JRUUHnzp0JDg5m/fr1te5PJpMxd+7cF6pi4ty5c7z77rvAo4qeNm3a8OGHH0p5LCQkhPDwcObNm4etrS1Tpkz508FeBWVWVlYsXryYrVu38uWXX1JRUUGrVq3o3LmzVCa88cYbPHz4kC1btlBUVESbNm34+OOP690dQBvPayMjozo/CwgIoEWLFsTHx7Nt2zZkMhkODg7Sdak+ZZqBgQHR0dHk5eVhYGCAu7u70pzz48ePZ82aNSxcuBALCwvGjBlTrzdj3t7ezJ07l507dxIfH4+enh729vbSYJ3GxsZcunSJxMRE7t+/j7W1NSNGjFCahlhonsQ9Z9OytbVl+fLlxMbGsn37dgoKCtDT08PBwYEBAwY819TwhoaGxMfHc+vWLXR0dHB2dmb+/Pla/7D8Z0T+VB8Ry7rpKBrSqfQFc+bMGTZs2MDXX3+tdf3ATp06RVRUFCtWrKi1P5A20uZ4JiYmcvLkST799FNNJ0Wt/v3vf7N9+3Y2btyo8Rkdjhw5wtq1awkPD2/WF2dBEP6POK8FQT20+R6pOd5zCuol8qf6iFjWTm/hwoULm/QbmxE7OzsUCgWWlpZa98by6tWr9OnTR2WQPW2mzfG8du0aAwYMUBpjojlKTEwEHg1ic/r0abZu3crrr7+u0RHxy8vLKSgoICIigp49e9Y60q8gCM2LOK8FQb20+R6pOd5zCuol8qf6iFjWTrSUEIS/kI0bN3L06FFKS0uxsrKiV69eBAYGSgM0akJMTAyxsbF4enryz3/+E2NjY42lRRAE9RDntSAIgiAI6iIqJQRBEARBEARBEARB0Ajt6sgiCIIgCIIgCIIgCMILQ1RKCEIzVFpaytSpU7Vy3uuoqCgiIiI0nYwGEfFUL22O57x58zh27Jimk1Fv2hxLkTfVa+XKlcTHx2s6GYLwl6DN53pzLDsFobGJKUEFoRmKjY2lS5cu2NnZAbBhwwYuXbpEdnY2FhYWhIWFqWxz5MgRYmNjuXXrFmZmZgwaNIg333xTaZ3ExET27NlDXl4erVq1Yvjw4fj5+UmfL1y4UGnaohoODg6sXLkSgGHDhjFz5kyGDBlS5xzI2kbEU70aK56HDx8mLi6OW7duYWRkRMeOHRk/fjwWFhbSOseOHWP79u3cvn0bW1tbRo0aRffu3aXPR4wYQWRkJN27d9e6Ua9rI/KmemkqnvBoNqS9e/eSn5+Pqakp3bp1Y+zYsRgaGgIQGBjIggULCAgIEGN0CMJzEmWnIDQvolJCEJqZ8vJy9u/fT2hoqLRMoVDg5+fH9evXOXv2rMo2p0+f5ptvvmHSpEl4e3tz8+ZNwsPDkclk0jzJe/fuZcuWLUybNg03NzcyMzMJDw+nZcuWdOvWDYDZs2cjl8ul/VZWVjJ79mx69uwpLTMzM6NTp07s3buXcePGNVYY1EbEU70aK54XL17k22+/Zdy4cXTv3p2ioiJ+/PFHvvnmG/71r38BkJGRwapVqwgKCqJ79+4cP36clStXsnjxYtzc3ADw8fEhPDycM2fOaP2MESJvqpcm43n48GE2b97Me++9h6enJ3l5efzwww9UVlYSEhICPJq/3tbWlkOHDkn7FgSh4UTZKQjNj/a/JhIEQcnp06cB8PDwkJZNnjyZN954A3t7+1q3OXToEF27dmXgwIHY2tri4+PDW2+9RVxcHDVj3R46dIiAgAB8fX2xtbWlV69e9OvXj7i4OGk/JiYmWFhYSD8XL16kvLwcf39/pe/r1q0bKSkp6j70RiHiqV6NFc+MjAysra35r//6L2xsbHB3d2fQoEFcvnxZ2s/u3bvx8vJi+PDhODg4MHz4cLy8vNi9e7e0jq6uLl26dOHw4cONcfhqJfKmemkynpcuXcLNzY3XX38dGxsbOnTogJ+fH5mZmUrf15ziKQjaSpSdgtD8iEoJQWhm0tPTcXFxQUdHp97bVFZWYmBgoLRMJpNRUFBAfn6+tI5MJlNZJzMzU6nW/3FJSUl4e3urzGfs6upKYWGhVvblfJKIp3o1Vjw9PT25e/cuJ0+eRKFQcO/ePY4cOUKXLl2kbTIyMujcubPSfjp37kxGRobSMldXV9LT0xt6aE1O5E310mQ8PT09uXbtmpQX79y5w8mTJ5XyLzyKZ2ZmJhUVFQ0+PkEQHhFlpyA0P6JSQhCamfz8fCwtLRu0jbe3NydPniQ1NZXq6mpycnJISEgAoKioCHj08JacnExmZiYKhYKsrCySkpKoqqqipKREZZ85OTmkpaUREBCg8llN+mou5NpMxFO9Giue7u7ufPDBB3z77beMHj2aKVOmoFAoeP/996X9FBUVYW5urrRvc3NzaR81rKysKCwspKqq6lkOscmIvKlemoxnr169GDVqFAsWLGDUqFFMnz4dR0dHxowZo/R9lpaWVFVVUVhYqIYjFoQXkyg7BaH5EWNKCEIzU1tN/dMEBASQm5vL8uXLqaqqwsjIiMGDB7Njxw7pTUJgYCBFRUV89tlnKBQKzM3N8fPzY9euXbW+bUhKSsLS0rLWfvk16WsOb/tEPNWrseJ548YNIiIiGDFiBJ07d+bu3bts3ryZtWvXKlVM1IdMJkOhUFBZWYmenl6Dtm1KIm+qlybjmZaWxs6dO5kyZQpubm7k5uayYcMGYmJiCA4Olr6vOcVTELSVKDsFofkRlRKC0MyYmppSWlraoG10dHQYO3Yso0ePpqioCDMzM86dOwcgjfwsk8mYPn067777LsXFxVhaWrJv3z6MjIwwMzNT2p9cLufgwYMEBATU+lBXk74nt9NGIp7q1VjxjI2NxdXVVRoJ3cnJCUNDQ/71r38xatQorK2tsbCwoLi4WGnfxcXFSrNzwKN4GhgYSLMeaCuRN9VLk/GMjo6mV69e0htTR0dHysrKCA8PJzAwUIptc4qnIGgrUXYKQvMjum8IQjPj7OzMzZs3n2lbXV1drKys0NfXJyUlBXd3d5ULor6+PtbW1ujq6pKSkoKPj4/K1InHjx+npKSEvn371vo92dnZ6Onp4ejo+EzpbEoinurVWPEsLy9XiVvN7zWDkLm7u6uMqn727Fnc3d2Vll2/fh0XF5dnSmNTEnlTvTQZz7ryb03erZGdnY2VlZVKRZogCPUnyk5BaH5ESwlBaGa8vb3ZsmULJSUlmJqaApCbm0tZWRl3795FLpdz7do14NG82Pr6+ty7d49jx47Rvn175HI5ycnJHD16lEWLFkn7zcnJITMzEzc3N+7fv09CQgLZ2dnMmDFDJQ1JSUl06NChzvm109PTadeuHS1atFB/ANRMxFO9Giue3bp1Izw8nL1790rdNzZt2sTLL78sDSA2ePBgFixYwC+//MKrr77K8ePHuXDhAp9//rlSGi9evKgyIKY2EnlTvTQZz65du7J7925eeeUVqfvG9u3b8fHxUXqLmp6e3izypiBoM1F2CkLzIyolBKGZcXR0xNXVlZSUFGnu7DVr1pCWliatM2fOHAC+++47bGxsADh48CBRUVHAozfKCxcuxNXVVdqmurqahIQEcnJy0NPTw8vLiyVLlkjb17h9+zbnz59n1qxZdaYxJSWFoKAg9RxwIxPxVK/GimefPn14+PAhiYmJREZGYmxsTIcOHZQGCvTw8OCDDz4gOjqa7du3Y2dnxwcffICbm5u0TmFhIZcuXWLmzJmNFwQ1EXlTvTQZzxEjRqCjo8P27dspKCjAzMyMrl278s4770jrVFRUcPz4cT755JPGC4IgvABE2SkIzY+O4sm2g4IgaL0zZ86wYcMGvv76a5Umg5p26tQpoqKiWLFihVYPIvg4EU/10uZ4RkVF8eDBA6ZNm6bppNSLNsdS5E31SkxM5OTJk3z66aeaToogNHvafK43x7JTEBqb3sKFCxdqOhGCIDSMnZ0dCoUCS0tLWrZsqenkKLl69Sp9+vRRmZNbm4l4qpc2x/OPP/5gyJAhWj/IZQ1tjqXIm+p17do1BgwYIDU3FwTh2Wnzud4cy05BaGyipYQgCIIgCIIgCIIgCBqhXe2ZBEEQBEEQBEEQBEF4YYhKCUEQBEEQBEEQBEEQNEJUSgiCIAiCIAiCIAiCoBGiUkIQBEEQBEEQBEEQBI0QlRKCIAiCIAiCIAiCIGiEqJQQBEEQBEEQBEEQBEEjRKWEIAiCIAiCIAiCIAga8f8AJ8lyv5c7sbAAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 732 }, "id": "8xp79OV8wwdh", "outputId": "b8a17815-aa7f-4e6e-d818-4f9d3b13ad12" }, "source": [ "plot_components(components_df, 'fc1', ascending=True)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAABCUAAAMlCAYAAABEr0kcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3yMd97/8fdkJgdEMiFCJcQp4rCoolFJG4egVZQ2LaVtiFItuna33ZW7a5PqQdR9d1XdSrWNoAfSbYvSIlm0KD2pOBRRshVUKsmgSIhcvz/8MrfpJEgyMg6v5+Oxj+18r+/3e32uz8yVh/nMdX0vk2EYhgAAAAAAAKqZh7sDAAAAAAAANyeKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAwE1ixIgRMplMys7OdncoVVaZY1m3bp1MJpOSkpKqvP/s7GyZTCaNGDGiynPdTLKysjR48GA1aNBAJpNJVqvV3SEBLte9e3eZTCZ3hwEA1w2KEgDgAiaTSSaTSR4eHvrpp5/K7dejRw973/nz51dfgDcBCgWuVZUiTlnvw/nz5zVo0CCtXLlS/fv3V2JioiZNmlThuW02m6ZPn67hw4erTZs2slgsMplMSk9PL3dM9+7d1aRJkwrvC9cuVxYZAQDuZXF3AABwo7BYLCouLtbbb7+tl19+2Wl7VlaW1q1bZ+9X3aZOnapJkyYpODi42vftajfSsdwsDhw4oF27dmn06NF68803Kz1Pdna2/vrXv0qSQkJCFBgYqKNHj7oqTAAAUM24UgIAXKR+/frq3LmzUlJSyiw6vPXWW5KkAQMGVHdokqRbbrlFrVq1kqenp1v270o30rHcLA4fPixJatiwYZXmCQ0NVXp6uvLy8nTw4EHdfffdrggPAAC4CUUJAHCh0aNH65dfftGnn37q0H7u3DnNnz9f3bp1U5s2bcodn5WVpccee0zBwcHy8vJSw4YN9dhjjykrK8uh39ixY2UymbR06dIy59myZYtMJpNiY2PtbZdah2HLli2KjY1VgwYN5OXlpUaNGumJJ56wf5G82P79+zVmzBi1aNFCNWrUUJ06ddSuXTuNHTtWeXl5l0qPpAtfSsu6wiE0NFQmk0kvvPCCQ/tnn30mk8mkf/zjH+UeS1JSkpo2bSpJSk1Ntd8iU95tMj/88IPuvfdeWa1W1axZU9HR0dq0adNlY78SR44c0bhx49SkSRN5eXmpXr16uv/++/Xdd9859Fu1apVMJpOee+45h/a1a9faYz948KDDtiFDhshkMmn//v0O7bt379aIESPUqFEjeXl5qX79+ho2bJj27NnjFN/Ro0f1zDPPKDw8XLVq1ZLValV4eLhGjBhhn3fEiBHq0aOHJOn55593yOe6desqnBOTyaTo6Gin+S6+9P78+fOaM2eOIiMj5e/vrxo1aqhFixZ6/PHHHT7/AQEB6tWrl+rUqVPhOCpi9+7dio+PV5MmTeTt7a2goCDdeeedeuONN5z6ZmRk6O6771adOnXk7e2tli1batKkSTp+/LhT39L1Bs6dO6cpU6aoefPm8vHxUXh4uObNm2fvN2fOHLVr1041atRQSEiIEhMTVVJS4jDXxbcs7d69W4MGDVKdOnVUq1YtRUVFafXq1WUeW1FRkZKTk9WuXTvVrFlTfn5+uvPOO7VkyRKnvhfvIzs7W0OHDlVgYKB8fHzUuXNnp791F3v//ffVo0cPWa1W+fj4qHXr1nrxxRdVVFTk1NdkMql79+46duyYxowZo1tuuUXe3t5q27atUlJSHPpW9vN56NAhmc1mdezYsdw+99xzj0wmk3bs2GFvmz9/vh544AE1a9ZMNWrUkJ+fnyIjI7Vo0aJy5/m9+fPnX/K2vdLj/73i4mLNnj1bXbt2lZ+fn2rWrKmOHTtq1qxZTp8HSVq2bJl69eplz1/Dhg0VHR2t2bNnX3GsAFDduH0DAFzo4Ycf1p///Ge99dZbGjRokL192bJlys3N1bRp07Rv374yx37zzTeKiYnRyZMnNXDgQLVp00a7d+/WokWLtHTpUqWnp6tLly6SpLi4OM2dO1cLFizQfffd5zRXamqqJF3R+grvvPOOxowZI29vbw0cOFCNGjVSVlaW3nrrLS1fvlybN29W48aNJV34wt2lSxedOHFC/fr10wMPPKDCwkIdOHBACxcu1Pjx41W3bt1L7q9nz5569913tXv3brVq1UqStG/fPv3888+SLnzBmzx5sr1/RkaGJKlXr17lztm9e3fZbDa99tpr6tChg0Pub731Voe+3377rV555RXdcccdevzxx/Xzzz/rX//6l3r16qUffvhB4eHhl81ZeQ4cOKCoqCgdPnxYPXv21MMPP6yDBw8qLS1NK1as0L/+9S/1799fknTnnXfKy8tLGRkZeumll5yOt/S/S99DwzC0du1aNWnSRM2aNbP3+fzzz3X//ffr3LlzGjBggFq0aKGcnBx99NFHWrFihdauXavbbrtNknT69GlFRkbqp59+Uu/evTVgwAAZhqH//Oc/Wrp0qWJjY9WsWTN7/lJTUxUdHe3wZakyazMkJiYqOzvbab7S/z979qz69++vNWvWqFGjRho2bJj8/PyUnZ2tjz/+WFFRUQoLC6vwfitrxYoVevDBB1VUVKS7775bDz/8sGw2m7Zt26ZXXnlFTz75pL3v3Llz9eSTT6pWrVp68MEHFRQUpHXr1mnatGlavny5Nm7cWOaCnkOHDtWWLVvUr18/eXp66sMPP9SYMWPk6empzMxMpaamqn///urVq5eWLVumKVOmqGbNmvrb3/7mNNeBAwd0xx13qF27dnriiSd05MgRLV68WPfcc4/ee+89DRkyxN737Nmz6tu3r9avX69WrVpp3LhxOn36tD788EMNGTJEP/zwQ5m3n/3nP//R7bffrmbNmunRRx9Vfn6+Fi9erPvuu0/p6en2IkGp+Ph4paSkKCQkRA888ICsVqs2b96syZMnKyMjQ2vWrJHF4vjPUJvNpsjISHl5eSk2NlZFRUVKS0tTfHy8PDw8FBcXJ0mV/nwGBwcrJiZGq1ev1vbt29WuXTuH7UeOHNGaNWvUqVMn/eEPf7C3P/nkk2rbtq3uuusu3XLLLcrLy9PKlSv16KOPas+ePU6FVFcpPadXrVql8PBwDRs2TD4+Plq7dq0mTJigLVu2aOHChfb+b775pp544gk1aNBAAwYMUGBgoHJzc5WZmamUlBQ99dRTVyVOAKgyAwBQZZKM4OBgwzAMY9SoUYbZbDYOHjxo3963b1/Dz8/POHXqlPHcc88ZkoyUlBT79pKSEqNVq1aGJGPRokUOc3/wwQeGJCM8PNw4f/68vb1ly5aGl5eXkZeX59C/sLDQCAgIMIKCgoxz587Z2+Pi4gxJxoEDB+xte/bsMTw9PY3mzZsbOTk5DvOkp6cbHh4exqBBg+xtM2fONCQZM2bMcMrBb7/9Zpw+ffqyuXr77bcNScasWbPsbXPmzDEkGb179za8vLyMU6dO2bfdeuutRo0aNYyioqJLHsuBAwcMSUZcXFyZ+127dq0hySn3F+//ySefvGz8l9pXnz59DEnGiy++6NC+ceNGw2w2G3Xq1DFOnjxpb7/zzjsNs9ls2Gw2e1vXrl2Njh07GnXr1jUeeeQRe/sPP/xgSDLi4+Ptbfn5+YbVajXq1q1r7Ny502Gf27dvN2rVqmV07NjR3rZs2TJDkjFx4kSnYyoqKjJOnDhhf12ar8TExCvKyeVcar6EhARDkjFgwACjsLDQYVthYaGRm5tb7ryln4U1a9a4JM5ff/3V8PPzMzw9PY1169Y5bb/4vM7Ozja8vLyM2rVrGz/++KNDvyeffNKQZIwePdqhPTo62pBkdO7c2SgoKLC3//TTT4anp6dhtVqNJk2aOJyPBQUFRt26dY3AwECHc7r0cyjJeOaZZxz288033xgWi8WwWq3G8ePH7e0vv/yyIcm45557HOY6evSoERoaakgyNm7cWOY+kpKSHPbx+eef2+e6WEpKiiHJGDx4sNPfhMTExDL/hpTuY9SoUUZxcbG9fefOnYbZbDZat27t0L+yn8/33nvPkGT85S9/cdr2yiuvGJKMmTNnOrTv27fPqW9RUZHRs2dPw2KxOP3tLH2PL1aak9//7SklyYiOjnZoK83V+PHjHXJSXFxsxMfHG5KMTz75xN5+2223GV5eXsbRo0ed5v/111/L3C8AXAu4fQMAXGz06NE6f/683nnnHUkXfmFcs2aNhg8frpo1a5Y5ZtOmTdq9e7fuuOMODR8+3GHbkCFDFBUVpT179mjDhg329ri4OJ09e1bvv/++Q//ly5eroKBAw4cPd/ol8vfeeOMNnTt3Tq+99prTLRW9evXSwIEDtXz5cp08edJhW40aNZzmqlWrVpntv1d6xcPvrwioX7++nn76aZ09e9Z+nHl5edq2bZuioqLk5eV12bmvRGRkpNMVJPHx8bJYLPr6668rPW9OTo5Wr16txo0b2xdiLNWtWzc9/PDDys/P10cffWRv79Wrl86fP6/169dLkk6ePKlvv/1WvXv3Vo8ePfTvf//b3resK0YWLFggm82m559/3um2oD/84Q8aPXq0tm7dql27djlsK+t98vLyUu3atSt59JV3/vx5zZ49WzVq1NCcOXPk7e3tsN3b21v16tWrtnhSU1N14sQJPfnkk/ZbTi4WEhJi/+9Fixbp7NmzGj9+vP2qn1IvvfSSateurYULF5Z5u0JycrLDFRTNmjVTVFSUbDabJk+e7HA+Wq1WDRgwQMeOHdOhQ4ec5vL393e4vUmSOnfurOHDh8tms+njjz+2t7/zzjsymUx69dVXHf4+BAUF2a9QKl3/5mKhoaH6+9//7tDWt29fNW7c2Om8ee2112SxWPTOO+84fdYmT56sunXr6t1333XaR82aNfXqq6/KbDbb29q0aaPIyEj9+OOP+u2335zGVNSgQYPk7++vd999V+fPn3fYlpqaKk9PTz388MMO7c2bN3eax8vLS+PGjVNxcbHD3zJXKSkp0euvv64GDRron//8p0NOzGaz/ud//kcmk8kpjxaLpcy1dgIDA10eIwC4CrdvAICLRUREqF27dnrnnXf097//XW+99ZZKSko0evTocsd8//33ki7c2lCWnj17asOGDdq6davuuusuSdJjjz2myZMnKzU1VePGjbP3rcitG1999ZUkaf369frmm2+ctufm5ur8+fPau3evOnXqpIEDB+q//uu/NG7cOK1atUp9+/ZVZGSk2rRpI5PJdNn9SRe+3DRr1kzr1q1TSUmJ/T7wmJgYRUdHy2KxKCMjQ3369NHatWtlGEa5eamMzp07O7V5enqqfv36KigoqPS8W7dulXThtoyyvhT07NlTixYt0tatW/XYY4/Z25KSkpSRkaGBAwdq/fr1Ki4uVq9evdSkSRN9+OGH+vHHH9W6dWt7geLiXJS+f9u2bSvz0Yh79+6VJP34449q06aNoqOjFRwcrOTkZH3//ffq16+fIiMjdeuttzp86alOu3fv1vHjxxUREVHlRTBdYfPmzZIurC1wOZc6bwMCAtSxY0d98cUX2r17tzp06OCwvazPYenxd+rUyWlbaZEiJydHoaGhDttuu+22MgtK3bt3V2pqqrZu3aq4uDidPHlS+/btU3BwsFMR5eLjKP0sX6y8z0ijRo3sn0Ppwi1C27ZtU2BgoGbMmOHUX7pQaPrxxx+d2sPCwuTn51fmPiSpoKBAvr6+Zc55pWrUqKGHHnpI8+bN06pVq9SvXz9J0nfffaedO3dq8ODBTl/gf/75Z02bNk0ZGRn6+eefdebMGYftZRWKqmrv3r3Kz89XWFiYXnzxxXKP5eI8Dh8+XH/5y1/Upk0bDR06VNHR0YqMjKzWoh4AVAZFCQC4CkaPHq2nn35an332mVJSUtSpU6dLLq5WuiDeLbfcUub20nabzWZvCwkJUa9evbRmzRr7F9fc3Fx9/vnnuvXWW9W+ffvLxlm6MOX06dMv2a/0F8rQ0FB9/fXXSkpK0ueff27/1b9Ro0Z65pln9PTTT192n9KFX/vnzZun77//Xp6envr111/Vq1cv1a5dW126dLH/8ngl60lUVFn390sXfmH8/S+nFVGZ97Br166qVauWw/F6eXkpKirKfm98RkaGwsLC9MUXX6hNmzZq0KCBfXzp+3fxAollKX3//Pz8tHnzZiUmJmrZsmVatWqVpAu/oj711FP6+9//Xu1PNCnNx7XyeNeKxFOZ97yUv7+/U1vplQuX2nbu3DmnbfXr1y9z/6WfldI4qxLvpc6bixdcLCgokGEY+vXXX/X888+XOaY8l9qHpCqdnxcbMWKE5s2bp9TUVHtRorSYW7puRan9+/fr9ttvV0FBge6880716dNH/v7+MpvN9nVSyroSpqpKz+2srKxL5vHiq0f+/Oc/KzAwULNnz9bMmTM1Y8YM+yKz06dPL7MQBgDXAm7fAICr4NFHH1WNGjU0duxYHTp0SGPGjLlk/9IvIb/88kuZ248cOeLQr1TpP6BL/0H97rvvqri42Okf1pfb7/Hjx2UYRrn/u/gy9tatW2vx4sXKy8vTt99+q+TkZJWUlOiPf/yj3n777Svab+kvsunp6U6Fh549e2rr1q3Kz89XRkaG/P397Qs1Xssq8x56enoqKipKO3fu1C+//KKMjAzdcccdqlmzplq2bKmQkBClp6fr66+/1smTJ51+kS+da9u2bZd8/y7+PISEhOjtt99Wbm6uduzYoZkzZ6pu3bqaMmWKpkyZ4tKcXInSL6JX49fmyqhIPJU9b13t6NGjZbaXxlW6/+qIt3Rsx44dL/mZNAyj0vuoqm7duiksLEzLli2TzWbTuXPn9P777yswMNBepCj16quvKi8vT2+//bbWrVunmTNn6oUXXlBSUpL69u17xfv08LjwT+6yHhd9qaLV4MGDL5nDAwcOOIx77LHHtHnzZuXl5WnFihUaNWqUvvjiC/Xt21e//vrrFccLANWJogQAXAVWq1WxsbHKyclRrVq1nO5R/r3SqyjKe5zd2rVrJcnpy/n9998vPz8/LVq0SCUlJUpNTZXFYtGwYcOuKM6uXbtKkr788ssr6n8xi8WiTp066W9/+5t9XYtPPvnkisb27NlTJpNJGRkZ+ve//61mzZrZrwzo1auXSkpKtGDBAmVlZal79+5XdGtBaR9X/ZpaUaXv4YYNG8r84lHee1hajHn//fe1Y8cOh6tCevbsqXXr1mnNmjUOfUtV5f0zmUxq27atJkyYYJ//4vevuvLZqlUrWa1WZWZmlvkI2upWmtPPPvvssn0vdd7abDb98MMP9kdhXk3ff/+907ovF8dVGmft2rXVvHlzHTp0yOkxw1L5n9GK8PX1Vdu2bbVz507l5+dXep7LqernMy4uToWFhVq8eLFWrFihY8eOadiwYU5XCpU+LemBBx5wmqN0LZgrERAQIElOj/mVLjwR6PdKz4vNmzeXeXXM5VitVvXr10/z5s3TiBEjlJ+fry+++KLC8wBAdaAoAQBXyYsvvqiPP/5Yq1atuuwCgpGRkQoPD9eGDRv04YcfOmz78MMP9eWXX6ply5aKiopy2FZ6f/ShQ4f0z3/+U9u2bVO/fv0UFBR0RTGOHz9enp6e+tOf/mRff+BiZ8+edfjC+91339kvAb9Y6S+15S3k+XtBQUFq27atNm7cqC+++MLhy3a3bt3k4+OjqVOnSip/nY3fCwgIkMlksj9atLqFhISod+/eys7OdrqXfsuWLXrvvfcUEBCgwYMHO2wrPb7k5GQZhuFUlDh+/Lhmz54tDw8Ph0cfStLIkSNltVr1/PPPl7lIZ0lJicMX5p07d5b5q3pZ71/po12vdj7NZrOeeuopnTlzRmPHjnW6FP7s2bPV+gtvXFyc/Pz89MYbb5T5JS4nJ8f+34888og8PT31+uuvOz3qd/LkyTpx4oQeeeQRp8U7Xe348eNOV7l8++23evfdd+Xv7+/wmYuPj5dhGHr22WcdvtAfO3bM/mjL+Pj4KsXz5z//WWfPnlV8fHyZVwEUFBTY1+OorKp+Ph977DF5eHhowYIFWrBggaSy1+EpLZb+vvC0atWqMhcELU/nzp3l4eGh9957T6dPn7a35+fnOy2MK10o+k6YMEFHjhzR008/7bSOhXThypaLF7EtXYPn93JzcyVd+d9nAKhurCkBAFdJ48aN1bhx4yvqazKZlJqaqt69e2vIkCG677771KpVK+3Zs0effPKJateurQULFtgvAb5YXFyc3nrrLSUkJNhfX6lWrVrpnXfeUXx8vNq2bau7775bLVu21Llz5/Tzzz/ryy+/VL169bR7925J0sKFCzV37lxFRUWpefPmCggI0E8//aTly5fL29tbEydOvOJ99+rVSzt27LD/dylvb29FRkZWeD0JX19fRURE6Msvv9Tw4cPVsmVLmc1mDRw48IrW13CFOXPmKDIyUs8++6xWr16tzp076+DBg0pLS5OHh4dSUlKcClQdO3ZUQECAcnNzVbt2bd1+++32baXHnpubq86dOzvdc1+3bl19+OGHGjx4sLp27apevXqpbdu2MplMOnjwoL766ivl5eWpsLBQkrRmzRo9++yzuuOOO9SyZUsFBQUpJydHS5culYeHh5599ln73OHh4QoODtYHH3wgT09PhYaGymQy6dFHH3VaaLGqEhMTtWXLFi1fvlwtW7ZU//79Vbt2bR08eFCrV6/W9OnTHb4wPvPMMzp27Jgk2Z/UMn36dC1atEjShScsDBo0qFKxBAYG6r333lNsbKx69Oihe+65R+3bt9eJEyeUmZmpgwcP2i+Zb9KkiWbMmKFx48bptttu00MPPaR69epp/fr1+uqrr9SqVStNmzatCpm5MnfddZfeeustbdmyRZGRkTpy5IgWL16skpISzZ0712HxyGeeeUafffaZli5dqg4dOqhfv346ffq00tLSlJubq7/+9a9Oxc+Kio+P13fffafZs2erefPm9qd05Ofn68CBA/riiy80cuRIzZkzp9L7qOrns1GjRurRo4cyMjJksVjUrl27Mtf9eeqpp5SSkqIHH3xQsbGxatiwoXbs2KHPP/9cDz30kBYvXnxF8d5yyy0aPny4Fi5cqFtvvVX33nuvTpw4oZUrV+quu+4qc3HRyZMna9u2bZozZ46WL1+unj17Kjg4WLm5ucrKytLGjRv10ksv2Z+8M3jwYPn6+qpr165q0qSJDMPQl19+qW+++UadOnVSTEzMFcUKANXuKj9yFABuCpKM4ODgK+r73HPPlfu8+t27dxuPPPKI0aBBA8NisRgNGjQwhg8fbuzevfuSc7Zo0cKQZNSpU8coKioqs09cXJwhyThw4IDTtszMTCMuLs5o3Lix4eXlZQQEBBht27Y1xowZY2RkZNj7bd682Rg7dqzRvn17IyAgwPDx8TGaN29ujBgxwti+ffsVHX+pZcuWGZIMk8lkHD161GHbyy+/bEgy6tevX6FjycrKMvr372/UqVPHMJlMDnleu3atIclITEwsc87Q0FAjNDT0imI/cOCAIcmIi4tz2paTk2OMHTvWaNy4seHp6WnUrVvXuO+++4yvv/663Pnuv/9+Q5LRr18/p20tW7Y0JBl//etfLxnPuHHjjBYtWhje3t5G7dq1jfDwcOORRx4xPv74Y3u/Xbt2GX/605+MTp06GYGBgYaXl5cRGhpqPPDAA8bGjRud5v3666+Nnj17Gn5+fvZ8rl279tLJKcfl8n/u3Dnj9ddfN7p06WLUqlXLqFmzptGiRQtj9OjRRlZWlkPf0NBQQ1K5/ytvHxWxY8cO49FHHzUaNmxoeHp6GkFBQcZdd91lzJ0716nvqlWrjN69extWq9Xw8vIymjdvbjz77LNGQUGBU9/o6GijvH9+XeocTUxMdMr/xZ/DXbt2GQMHDjSsVqtRo0YNo1u3bsbnn39e5n7OnDljvPTSS0bbtm0NHx8fw9fX14iMjDTee+89p76X+qxf7niWL19u3HvvvUa9evUMT09Po379+kaXLl2M5557zvjxxx8d+koyoqOjK5SXqn4+Fy5caP/M/Pd//3e5/TZu3Gj06NHDsFqt9lx9/PHH5X6my8tJYWGh8cwzzxjBwcGGp6en0bx5c+Pll182zp07V+7xl5SUGAsWLDB69uxpBAQEGJ6enkbDhg2NyMhI46WXXjJ+/vlne9833njDGDRokNG0aVOjRo0aRkBAgHHrrbca06ZNM06cOHHFeQGA6mYyDDeuNAQAAIBKyc7OVtOmTRUXF6f58+e7OxwAACqFNSUAAAAAAIBbUJQAAAAAAABuQVECAAAAAAC4BWtKAAAAAAAAt+BKCQAAAAAA4BYWdwfgSocPH3Z3CJcVGBhof7Y6qo58ug65dC3y6Vrk03XIpWuRT9cin65DLl2LfLoW+XSt6yGfDRs2LHcbV0oAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt7ihnr4BAAAAAMDVcv78eRUWFkqSTCaTm6O54OjRoyoqKnLb/g3DkNlslo+PT6XGV6kosWrVKi1btkw2m00hISEaMWKEWrduXW7/Xbt2KTU1VTk5OQoICNDAgQPVp08f+/aPP/5YX3/9tQ4fPiyLxaKwsDANGzZMjRs3rkqYAAAAAABUyfnz53XmzBnVqlXrmilISJLFYpHZbHZrDIWFhTp37pw8PT0rPLbSt29s2rRJ8+fP1+DBgzVt2jSFh4fr5ZdfLvf5qLm5uZo6darCw8M1bdo0DRo0SCkpKdq8ebO9z65du9SnTx+98MILSkxMlNls1gsvvKDffvutsmECAAAAAFBlhYWF11xB4lrh7e2ts2fPVmpspYsSn376qaKjoxUTE6OQkBDFx8crICBAq1evLrP/6tWrFRAQoPj4eIWEhCgmJkbR0dFavny5vc9zzz2nHj16qHHjxmrcuLEmTJigEydOaPfu3ZUNEwAAAAAAl6AgUbaq5KVSRYni4mLt379fHTp0cGhv37699uzZU+aYrKwstW/f3qGtQ4cO2r9/v4qLi8scczhSEC8AACAASURBVObMGRmGIV9f38qECQAAAACAS1CQuLTK5qdSa0qcOHFCJSUl8vf3d2i3Wq3avn17mWNsNpvatWvn0Obv76/z58/r5MmTCggIcBqTkpKiJk2aqGXLlmXOmZ6ervT0dElScnKyAgMDK3M41cpisVwXcV4vyKfrkEvXIp+uRT5dh1y6Fvl0LfLpOuTStcina12v+Tx69KgslmvzWRHXQlze3t6Vel/dH3k5UlNTtWfPHk2ZMkUeHmVf0BETE6OYmBj76/LWs7iWBAYGXhdxXi/Ip+uQS9cin65FPl2HXLoW+XQt8uk65NK1yKdrXa/5LCoqcvuCkmWxWCzl3n1QnYqKisp9Xxs2bFjuuEoVJfz8/OTh4aHjx487tNtsNlmt1jLHWK1W2Ww2h7bjx4/LbDardu3aDu3z58/Xpk2blJiYqPr161cmRAAAAAAArrrzowdW6/7M85ZVeExJSYkmTZqkFStWyGazKS0tTd26dbsK0VVcpdaUsFgsatasmTIzMx3at2/frvDw8DLHhIWFOd3akZmZqWbNmjlcapKSkqKNGzfqH//4h4KDgysTHgAAAAAA+P8yMjK0ZMkSzZ8/X1u3blXnzp0v2d9ms2nChAlq1aqVWrVqpQkTJjhdlOAqlX76Rv/+/bVu3TplZGQoJydHKSkpys/PV+/evSVJs2bN0qxZs+z9+/Tpo/z8fM2fP185OTnKyMjQunXrNGDAAHuft956S+vWrdMf//hH+fr6ymazyWazqbCwsAqHCAAAAADAzSs7O1tBQUHq0qWLgoKC5OXldcn+48eP144dO7Ro0SItWrRIO3bs0NNPP31VYqv0mhLdunXTyZMn9dFHH6mgoECNGjVSQkKC6tWrJ8l5fYegoCAlJCQoNTXV/njQkSNHqmvXrvY+pY8TnTJlisPY2NhYPfTQQ5UNFQAAAACAm9LEiROVlpYmSQoODlZISIg2b96suXPnauHChTp8+LDq1Kmj2NhYJSQkKCsrS2vXrtUnn3xiv6Ji2rRpGjx4sPbt26cWLVq4NL4qLXTZt29f9e3bt8xtSUlJTm1t2rTRtGnTyp1vyZIlVQkHAAAAAABcZMqUKQoJCdEHH3yglStXymw2Kzk5WQsWLFBiYqIiIiKUl5enHTt2SJK+++471apVy+EWjy5duqhmzZr67rvvrq2iBAAAAAAAuHb5+fnJ19dXZrNZQUFBOnXqlObNm6ekpCQNHTpUktS0aVN7ESI3N1d169aVyWSyz2EymRQYGKjc3FyXx1fpNSUAAAAAAMD1Ze/evSoqKlJUVJS7Q5FEUQIAAAAAAPx/QUFBysvLk2EY9jbDMHTs2DEFBQW5fH8UJQAAAAAAuEmEhYXJ29tbGzZsKHN7p06ddOrUKX377bf2tm+//VanT59Wp06dXB4Pa0oAAAAAAHCT8PX11ahRo5ScnCxvb29FRESooKBAmZmZiouLU1hYmHr06KFJkybZH1QxadIkxcTEuHyRS4miBAAAAAAAlWaet8zdIVRYQkKC/P39NWPGDB05ckSBgYGKjY21b581a5YmT56s4cOHS5L69OmjF1988arEYjIuvlHkOnf48GF3h3BZgYGBOnbsmLvDuGGQT9chl65FPl2LfLoOuXQt8ula5NN1yKVrkU/Xul7zefr0adWsWdPdYTixWCwqLi52dxiXzE/Dhg3LHceaEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALnr5xjQval3DZPrktplZDJDeGy+WTXFYM+XQdznXXIp+uxbnuWuTTdTjXXYvPpmuRT+DKcKUEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKFLgEAAAAAqKT73t1drftbOrxVhceUlJRo0qRJWrFihWw2m9LS0tStW7erEF3FUZQAAAAAAOAGlpGRoSVLligtLU2hoaGyWq2X7P/aa6/p3//+t3bu3KkzZ87o0KFDVy02bt8AAAAAAOAGlp2draCgIHXp0kVBQUHy8vK6ZP+zZ8/qnnvu0eOPP37VY+NKCQAAAAAAblATJ05UWlqaJCk4OFghISHavHmz5s6dq4ULF+rw4cOqU6eOYmNjlZCQIEl69tlnJUmffvrpVY+PogQAAAAAADeoKVOmKCQkRB988IFWrlwps9ms5ORkLViwQImJiYqIiFBeXp527NjhlvgoSgAAAAAAcIPy8/OTr6+vzGazgoKCdOrUKc2bN09JSUkaOnSoJKlp06bq3LmzW+JjTQkAAAAAAG4Se/fuVVFRkaKiotwdiiSKEgAAAAAAwE0oSgAAAAAAcJMICwuTt7e3NmzY4O5QJLGmBAAAAAAANw1fX1+NGjVKycnJ8vb2VkREhAoKCpSZmam4uDhJ0qFDh1RQUKCcnBxJsi+C2bRpU9WqVcul8VCUAAAAAABcNcsX237X4vh6wBBr9QVzFSwd3qra9mXLL5Ytv/h3rY6vrXUu/zU/ISFB/v7+mjFjho4cOaLAwEDFxsbat0+fPt3+GFFJ6tu3ryQpLS1N3bp1q/wBlIGiBAAAAAAAN7CxY8dq7Nix9tceHh4aP368xo8fX2b/GTNmaMaMGdUSG2tKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3sLg7AAAAAAAArlfLF9uqdX939vat8JiSkhJNmjRJK1askM1mU1pamrp163YVoqs4ihIAAAAAANzAMjIytGTJEqWlpSk0NFRWq7XcvgcPHtSMGTO0adMm5ebmKigoSAMHDtTEiRNVo0YNl8dGUQIAAAAAgBtYdna2goKC1KVLl8v23bdvn86fP6+pU6eqadOmysrK0t/+9jcVFBTolVdecXlsrCkBAAAAAMANauLEiUpKStKhQ4cUHBysiIgIGYahOXPmKDIyUk2bNlWnTp00depUSVKPHj00Y8YMde/eXaGhoYqJidGECRO0YsWKqxIfV0oAAAAAAHCDmjJlikJCQvTBBx9o5cqVMpvNSk5O1oIFC5SYmKiIiAjl5eVpx44d5c7x22+/XfKWj6qgKAEAAAAAwA3Kz89Pvr6+MpvNCgoK0qlTpzRv3jwlJSVp6NChkqSmTZuqc+fOZY7PycnRnDlzNGHChKsSH7dvAAAAAABwk9i7d6+KiooUFRV12b6//vqrhg8frrvuuktjxoy5KvFQlAAAAAAAAA5yc3P14IMPKjw8XDNnzpTJZLoq+6EoAQAAAADATSIsLEze3t7asGFDuX2OHj2q2NhYhYWFafbs2bJYrt7KD6wpAQAAAADATcLX11ejRo1ScnKyvL29FRERoYKCAmVmZiouLk6//PKLYmNj1aBBAyUlJSk/P98+tm7dujKbzS6Nh6IEAAAAAACVNGDI1XkqRVls+cUumSchIUH+/v6aMWOGjhw5osDAQMXGxkqS1q9frwMHDujAgQO6/fbbHcZt3rxZjRo1ckkMpShKAAAAAABwAxs7dqzGjh1rf+3h4aHx48dr/PjxTn2HDBmiIUOGVFtsrCkBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANzC4u4AAAAAAAC4Xs2cObNa9/fYI09VeExJSYkmTZqkFStWyGazKS0tTd26dbsK0VUcRQkAAAAAAG5gGRkZWrJkidLS0hQaGiqr1Vpu35KSEsXHx2vnzp3Ky8uTv7+/oqKi9F//9V+65ZZbXB4bt28AAAAAAHADy87OVlBQkLp06aKgoCB5eXldsn9kZKTmzJmjL774Qm+++ab+85//6PHHH78qsXGlBAAAAAAAN6iJEycqLS1NkhQcHKyQkBBt3rxZc+fO1cKFC3X48GHVqVNHsbGxSkhIkIeHh0aPHm0fHxISovHjx2vkyJEqLCyUj4+PS+OjKAEAAAAAwA1qypQpCgkJ0QcffKCVK1fKbDYrOTlZCxYsUGJioiIiIpSXl6cdO3aUOb6goEAfffSROnbs6PKChFTFosSqVau0bNky2Ww2hYSEaMSIEWrdunW5/Xft2qXU1FTl5OQoICBAAwcOVJ8+fRy2L1++XPv371dBQYGeeuopde/evSohAgAAAABw0/Lz85Ovr6/MZrOCgoJ06tQpzZs3T0lJSRo6dKgkqWnTpurcubPDuJdeekkpKSk6c+aMbrvtNi1YsOCqxFfpNSU2bdqk+fPna/DgwZo2bZrCw8P18ssv69ixY2X2z83N1dSpUxUeHq5p06Zp0KBBSklJ0ebNm+19CgsL1ahRI40cOfKy97gAAAAAAICK2bt3r4qKihQVFXXJfk8++aRWrVql999/X2azWRMmTJBhGC6Pp9JXSnz66aeKjo5WTEyMJCk+Pl4//PCDVq9erWHDhjn1X716tQICAhQfHy/pwn0p+/bt0/Lly9W1a1dJ0m233abbbrtNkvS///u/lQ0NAAAAAABUQZ06dVSnTh01b95cLVq0UJcuXfT1118rIiLCpfupVFGiuLhY+/fv14ABAxza27dvrz179pQ5JisrS+3bt3do69Chg9avX6/i4mJZLBUPJT09Xenp6ZKk5ORkBQYGVniO6maxWCoW577Ld7kejvtqcXU+yWUFj598lotz3bXIp+twrrsW+XQtznXX4bPpWuSzqmyX3Hq95OLo0aOV+t5a3cqK0cPDQyaTSRaLRa1bt5a3t7e++uortWzZ8orm9PC4cJPFpb67e3t7V+q9rFRGT5w4oZKSEvn7+zu0W61Wbd++vcwxNptN7dq1c2jz9/fX+fPndfLkSQUEBFQ4jpiYGPuVGpLKvXXkWhIYGFihOIOuoM/1cNxXi6vzSS4rdvzks3yc665FPl2Hc921yKdrca67Dp9N1yKfV9f1kouioiKZzWZ3h3FZxcXFTm0lJSUyDEPFxcXy8fHRqFGj9NJLL8lisSgiIkIFBQXKzMxUXFycvv32W+3YsUNdunSRv7+/srOzNX36dDVq1EidOnUqc37pQn7Key8bNmxYbrzXfpkHAAAAAIBr1NNPP11t+7Lll10QqKiEhAT5+/trxowZOnLkiAIDAxUbGytJ8vHx0aeffqrp06frzJkzCgoKUvfu3fXGG29cO0/f8PPzk4eHh44fP+7QbrPZZLVayxxjtVplszletnP8+HGZzWbVrl27MmEAAAAAAIDLGDt2rMaOHWt/7eHhofHjx2v8+PFOff/whz/oww8/rLbYKvX0DYvFombNmikzM9Ohffv27QoPDy9zTFhYmNOtHZmZmWrWrNl1cV8OAAAAAABwrUo/ErR///5at26dMjIylJOTo5SUFOXn56t3796SpFmzZmnWrFn2/n369FF+fr7mz5+vnJwcZWRkaN26dQ6LZRYWFio7O1vZ2dkyDEPHjh1Tdnb2dXOPEQAAAAAAuHKVvkShW7duOnnypD766CMVFBSoUaNGSkhIUL169SQ5L1YSFBSkhIQEpaam2h8POnLkSPvjQCXpp59+0vPPP29/vWTJEi1ZskTR0dEaN25cZUMFAAAAAADXoCrdN9G3b1/17du3zG1JSUlObW3atNG0adPKna9t27ZasmRJVUICAAAAAADXiUrfvgEAAAAAAFAVFCUAAAAAAIBbUJQAAAAAAABuQVECAAAAAAC4BUUJAAAAAADgFlV6+gYAAAAAADezoH0J1bcvSXvrvFDhcSUlJZo0aZJWrFghm82mtLQ0devWzfUBVgJXSgAAAAAAcAPLyMjQkiVLNH/+fG3dulWdO3e+onGFhYWKiYlRcHCwtm3bdlVioygBAAAAAMANLDs7W0FBQerSpYuCgoLk5eV1ReNeeOEF3XLLLVc1NooSAAAAAADcoCZOnKikpCQdOnRIwcHBioiIkGEYmjNnjiIjI9W0aVN16tRJU6dOdRi3atUqbdq0Sf/4xz+uanysKQEAAAAAwA1qypQpCgkJ0QcffKCVK1fKbDYrOTlZCxYsUGJioiIiIpSXl6cdO3bYxxw+fFgJCQlauHChfHx8rmp8FCUAAAAAALhB+fn5ydfXV2azWUFBQTp16pTmzZunpKQkDR06VJLUtGlT+zoT58+f14QJEzRmzBi1bdtWBw8evKrxcfsGAAAAAAA3ib1796qoqEhRUVFlbp85c6Y8PT31xBNPVEs8XCkBAAAAAAAkSRs3btSWLVsUGhrq0D5gwAANHDhQs2bNcun+KEoAAAAAAHCTCAsLk7e3tzZs2KBmzZo5bX/11Vd1+vRp++ujR49q2LBhev3119WlSxeXx0NRAgAAAACAm4Svr69GjRql5ORkeXt7KyIiQgUFBcrMzFRcXJwaN27s0L9WrVqSpCZNmqhhw4Yuj4eiBAAAAAAAlZTbYurlO7mILb/YJfMkJCTI399fM2bM0JEjRxQYGKjY2FiXzF1RFCUAAAAAALiBjR07VmPHjrW/9vDw0Pjx4zV+/PjLjm3UqJEOHTp01WLj6RsAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAABchmEY7g7hmlbZ/FCUAAAAAADgMsxmswoLCylOlKG4uFgeHpUrL/BIUAAAAAAALsPHx0fnzp3T6dOnJUkmk6naYzj6y9nL9vHy8aqGSP6PYRjy8PCQj49PpcZTlAAAAAAA4Ap4enrK09PTbfvfu/3yRYnwNjWrIRLX4fYNAAAAAADgFhQlAAAAAACAW1CUAAAAAAAAbkFRAgAAAAAAuAVFCQAAAAAA4BY8fQMAAAAAcM0K2pdw2T65LaZWQyQ3hsvls7pzyZUSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANzCUpXBq1at0rJly2Sz2RQSEqIRI0aodevW5fbftWuXUlNTlZOTo4CAAA0cOFB9+vSp0pwAAAAAAOD6VOkrJTZt2qT58+dr8ODBmjZtmsLDw/Xyyy/r2LFjZfbPzc3V1KlTFR4ermnTpmnQoEFKSUnR5s2bKz0nAAAAAAC4flW6KPHpp58qOjpaMTExCgkJUXx8vAICArR69eoy+69evVoBAQGKj49XSEiIYmJiFB0dreXLl1d6TgAAAAAAcP2q1O0bxcXF2r9/vwYMGODQ3r59e+3Zs6fMMVlZWWrfvr1DW4cOHbR+/XoVFxdLUoXnTE9PV3p6uiQpOTlZgYGBlTmcch0d3O2S2+/v/spl53jc0uB3LTaHVweOLrjk+ClT5l12H6496qunqvl0zqXk6nzeKLmUKpPPiuVSunnyybleMdfDuS5dH/nkXHeta+Fcl8jnxfjbeUF1nOsS/066GH87r5wrzvWNf4xyeG2xWOzfASXpH//4xyXH3yjnulT1fP4+l5Lr81nduaxUUeLEiRMqKSmRv7+/Q7vVatX27dvLHGOz2dSuXTuHNn9/f50/f14nT56UYRgVnjMmJkYxMTH21zfibR434jG5E/l0LfLpOuTStcina5FP1yKfrkMuXYt8uhb5/D+/z0VgYGCF8kMu/09Zubge8tmwYcNyt/H0DQAAAAAA4BaVulLCz89PHh4eOn78uEO7zWaT1Wotc4zVapXN5njZ0/Hjx2U2m1W7dm1JqvCcAAAAAADg+lWpKyUsFouaNWumzMxMh/bt27crPDy8zDFhYWFOt2FkZmaqWbNmslgslZoTAAAAAABcvyp9+0b//v21bt06ZWRkKCcnRykpKcrPz1fv3r0lSbNmzdKsWbPs/fv06aP8/HzNnz9fOTk5ysjI0Lp16xwWtrzcnAAAAAAA4MZRqds3JKlbt246efKkPvroIxUUFKhRo0ZKSEhQvXr1JDkvnhEUFKSEhASlpqbaHw86cuRIde3a9YrnBAAAAAAAN45KFyUkqW/fvurbt2+Z25KSkpza2rRpo2nTplV6TgAAAAAAcOPg6RsAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANzC4u4AbnZPP/20u0O4oZBP1yGXrkU+XYt8ug65dC3y6Vrk07XIp+uQS8B1uFICAAAAAAC4BUUJAAAAAADgFhQlAAAAAACAW1CUAAAAAAAAbkFRAgAAAAAAuAVFCQAAAAAA4BY8EvQSzPOWXbrDu7urJ5AbBPl0ncvmUiKfFcBn07XIp+twrrsWn03XIp+uw7nuWuQTuL5wpQQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt6AoAQAAAAAA3IKiBAAAAAAAcAuKEgAAAAAAwC0oSgAAAAAAALegKAEAAAAAANyCogQAAAAAAHALihIAAAAAAMAtKEoAAAAAAAC3oCgBAAAAAADcgqIEAAAAAABwC4oSAAAAAADALShKAAAAAAAAt7BUZpBhGEpLS1NGRoZ+++03hYWFadSoUWrUqNElx23evFmLFy/W0aNHVb9+fT388MO6/fbb7du3bNmi9PR07d+/XydPnlRiYqLatm1bmRABAAAAAMA1rlJXSixdulSffvqpRo4cqalTp8rPz08vvviizpw5U+6YvXv3asaMGbrzzjv1yiuv6M4779Srr76qrKwse5+ioiK1bNlScXFxlQkLAAAAAABcRypclDAMQytXrtSgQYPUtWtXNW7cWOPHj9eZM2e0YcOGcsetWLFCbdu21f3336+QkBDdf//9atu2rVasWGHvc9ddd+nBBx/UrbfeWrmjAQAAAAAA140K376Rm5srm82m9u3b29u8vLzUunVr7dmzR7179y5z3N69e3XPPfc4tHXo0EGff/55RUOwS09PV3p6uiQpOTlZgYGBlZ7LXa7HmK9l5NN1yKVrkU/XIp+uQy5di3y6Fvl0LfLpOjdTLo+6YI7f58tisVQohzdSvquaz7Jycb3ns8JFCZvNJkmyWq0O7f7+/iooKLjkOH9/f6cxpfNVRkxMjGJiYuyvjx07Vum53OV6jPlaRj5dh1y6Fvl0LfLpOuTStcina5FP1yKfrkMuK+b3+QoMDKxQDsn3/ykrF9dDPhs2bFjutssWJb788ku9+eab9tcJCQmuiQoAAAAAANzULluU6Ny5s8LCwuyvz507J+nClQ8XX/Zx/PhxpyshLma1WnX8+HGHtuPHjztdcQEAAAAAAG4Ol13oskaNGmrQoIH9fyEhIbJarcrMzLT3OXv2rHbv3q3w8PBy52nZsqXDGEnKzMxUy5YtqxA+AAAAAAC4XlX46Rsmk0n9+vXT0qVLtWXLFv3888+aPXu2fHx8FBUVZe83ZcoUvffee/bX/fr1044dO/TJJ5/o0KFD+vjjj7Vz507de++99j6//fabsrOzdfDgQUnSL7/8ouzs7CqtOwEAAAAAAK5NFV7oUpLuu+8+nT17Vm+//bZOnTqlFi1a6LnnntP/Y+/uw60q67yBf4+8CKRwkCMgHMGUFxUFcxDPGJgapkEqD9VxUEkNtVQ0pqaufBzNCPOlUtIBk9RpnkpBbRQTHscANc2w0VQeUMSXDEWFUM9BEEVenj+82OPhnePGBfb5XBfXxVp73Wvf+7fvvfba332vfVq2bFnaZuHChWnXrl1puWfPnhk1alQmTpyYSZMmpWPHjhk1alSDS0MeffTRjB8/vrR8/fXXJ0m+9KUvpba2tjFdBQAAALZTjQolKioqUltbu8mgYNy4ceutq6mpSU1NzUbbHHHEETniiCMa0yUAAABgB7PVl28AAAAAlINQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAoRNOiOwAAAMDfr/PPP7/oLnys7Gj1NFMCAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAFS62QAAAIABJREFUKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKETTxjRas2ZNbrvttkyfPj1Lly5N9+7dM2LEiOy5556bbDdz5sxMmjQpCxcuTIcOHTJs2LD069cvSbJy5cpMnDgxTzzxRBYuXJiWLVumV69eOfnkk1NVVdWYbgIAAADbsUbNlJg8eXLuvvvunH766bnsssvSunXrjBkzJsuXL99om3nz5mXs2LEZMGBArrzyygwYMCBXXXVVnn322STJihUr8pe//CVDhw7NFVdcke985zt5/fXXc+mll2bVqlWNe3QAAADAdmurQ4k1a9Zk6tSpGTJkSGpqatKlS5eMHDkyy5cvz0MPPbTRdlOmTEmvXr0ydOjQVFdXZ+jQoenVq1emTJmSJGnVqlUuuuiiHHbYYenUqVO6deuWs846KwsWLMiCBQsa/wgBAACA7dJWX76xaNGi1NXVpXfv3qV1zZs3z3777ZdnnnkmRx999AbbzZs3L5///OcbrOvTp0/uueeejd7X22+/nST5xCc+scHbp02blmnTpiVJLr/88u3yMo/Tz+3WYLlp06ZZuXJlQb3Zsa1by0Q9Pwxjs7zUs3y81svL2Cwv9Swv9SwftSwv9fwfC8uwj3U/pzVt2nS7/Oz2Ufiw9dxQ3Xb0em51KFFXV5ckqaysbLC+TZs2efPNNzfZrk2bNuu1Wbu/da1cuTK//OUv8w//8A9p167dBrcZOHBgBg4cWFpevHjxFj2Gj9K6faqqqtou+7kj2FDd1LPxjM3yUs/y8VovL2OzvNSzvNSzfNSyvNSzvNSzfHbU86ROnTpt9LbNhhIPPvhgJkyYUFq+4IILytOrTVi1alWuueaaLFu2LN/5zne2+f0BAAAAH73NhhJ9+/ZN9+7dS8vvvfdekvdnPnxwikh9ff16MyE+qLKyMvX19Q3W1dfXrzfjYtWqVfnpT3+a+fPn55JLLsmuu+66ZY8EAAAA2KFs9ocuW7ZsmY4dO5b+VVdXp7KyMrNmzSpts2LFisydOzc9e/bc6H569OjRoE2SzJo1Kz169Cgtr1y5MldffXX++te/5nvf+956gQUAAADw8bHVf32joqIigwYNyuTJk/PII49k/vz5GT9+fFq0aJH+/fuXths9enRuvvnm0vKgQYMye/bs3HnnnVmwYEHuuOOOzJkzJ4MHD07y/gyJtX8i9Bvf+EYqKipSV1eXurq6rFixogwPFQAAANiebPUPXSbJCSeckBUrVuTGG2/MsmXL0q1bt1x44YVp2bJlaZuFCxc2+IHKnj17ZtSoUZk4cWImTZqUjh07ZtSoUaVLQ15//fU8+uijSZLvfve7De7vnHPOyRFHHNGYrgIAAADbqUaFEhUVFamtrU1tbe1Gtxk3btx662pqalJTU7PB7du3b59bb721Md0BAAAAdkBbffkGAAAAQDkIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCCCUAAACAQgglAAAAgEIIJQAAAIBCNG1MozVr1uS2227L9OnTs3Tp0nTv3j0jRozInnvuucl2M2fOzKRJk7Jw4cJ06NAhw4YNS79+/Uq3T5w4MTNnzszrr7+epk2b5pOf/GROPPHE9OzZszHdBAAAALZjjZopMXny5Nx99905/fTTc9lll6V169YZM2ZMli9fvtE28+bNy9ixYzNgwIBceeWVGTBgQK666qo8++yzpW06deqUESNG5Mc//nFGjx6d9u3b54c//GHq6uoa000AAABgO7bVocSaNWsyderUDBkyJDU1NenSpUtGjhyZ5cuX56GHHtpouylTpqRXr14ZOnRoqqurM3To0PTq1StTpkwpbXP44YfnwAMPTIcOHbLnnnvmK1/5SpYvX54XX3yxUQ8OAAAA2H5tdSixaNGi1NXVpXfv3qV1zZs3z3777Zdnnnlmo+3mzZuXPn36NFjXp0+fzJs3b4Pbr1y5MtOmTUvLli2z1157bW03AQAAgO3cVv+mxNpLKSorKxusb9OmTd58881NtmvTps16bda9NOOxxx7L2LFjs2LFilRWVuaiiy5a777WmjZtWqZNm5Ykufzyy1NVVbW1D2ebW7dPTZs23S77uSPYUN3Us/GMzfJSz/LxWi8vY7O81LO81LN81LK81PN/LCzDPtTzf3zYen4cz5M2G0o8+OCDmTBhQmn5ggsu2KYd6tWrV370ox9lyZIlmT59eq6++uqMGTMmbdu2XW/bgQMHZuDAgaXlxYsXb9O+Nca6faqqqtou+7kj2FDd1LPxjM3yUs/y8VovL2OzvNSzvNSzfNSyvNSzvNSzfHbU86ROnTpt9LbNhhJ9+/ZN9+7dS8vvvfdekvdnPnwwjamvr19vJsQHVVZWpr6+vsG6+vr69WZBtGjRIh07dkzHjh3To0ePnH/++Zk+fXq+9KUvba6rAAAAwA5ks78p0bJly1JI0LFjx1RXV6eysjKzZs0qbbNixYrMnTt3k3+6s0ePHg3aJMmsWbPSo0ePTd7/mjVrsnLlys11EwAAANjBbPUPXVZUVGTQoEGZPHlyHnnkkcyfPz/jx49PixYt0r9//9J2o0ePzs0331xaHjRoUGbPnp0777wzCxYsyB133JE5c+Zk8ODBSZK33347EydOzLPPPpvFixfnhRdeyPjx4/P666/nH//xH8vwUAEAAIDtyVb/0GWSnHDCCVmxYkVuvPHGLFu2LN26dcuFF16Yli1blrZZuHBh2rVrV1ru2bNnRo0alYkTJ2bSpEnp2LFjRo0aVbo0pEmTJnnppZdy33335a233squu+6affbZJ9///vfTtWvXD/kwAQAAgO1No0KJioqK1NbWpra2dqPbjBs3br11NTU1qamp2eD2O++8c7797W83pjsAAADADmirL98AAAAAKAehBAAAAFCIRl2+AQAAwMdfk5/ftekNfj33o+nIx4R6rs9MCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEEIJAAAAoBBCCQAAAKAQQgkAAACgEE0b02jNmjW57bbbMn369CxdujTdu3fPiBEjsueee26y3cyZMzNp0qQsXLgwHTp0yLBhw9KvX78NbjthwoRMmzYtp5xySo4//vjGdBMAAADYjjVqpsTkyZNz99135/TTT89ll12W1q1bZ8yYMVm+fPlG28ybNy9jx47NgAEDcuWVV2bAgAG56qqr8uyzz6637cyZM/Pcc8+lbdu2jekeAAAAsAPY6lBizZo1mTp1aoYMGZKampp06dIlI0eOzPLly/PQQw9ttN2UKVPSq1evDB06NNXV1Rk6dGh69eqVKVOmNNjub3/7W/793/89559/fpo2bdREDgAAAGAHsNWhxKJFi1JXV5fevXuX1jVv3jz77bdfnnnmmY22mzdvXvr06dNgXZ8+fTJv3rzS8qpVq/LTn/40X/ziF1NdXb21XQMAAAB2IFs9FaGuri5JUllZ2WB9mzZt8uabb26yXZs2bdZrs3Z/SXLrrbdm1113zec+97kt6su0adMybdq0JMnll1+eqqqqLWr3UVq3T02bNt0u+7kj2FDd1LPxjM3yUs/y8VovL2OzvNSzvNSzfNSyvNSzvNSzfD6O50mbDSUefPDBTJgwobR8wQUXbJOOzJkzJ/fff39+9KMfbXGbgQMHZuDAgaXlxYsXb4uufSjr9qmqqmq77OeOYEN1U8/GMzbLSz3Lx2u9vIzN8lLP8lLP8lHL8lLP8lLP8tlRz5M6deq00ds2G0r07ds33bt3Ly2/9957Sd6f+fDBNKa+vn69mRAfVFlZmfr6+gbr6uvrSzMu5syZk7q6upx11lml21evXp1f//rXmTp1an72s59trqsAAADADmSzoUTLli3TsmXL0vKaNWtSWVmZWbNmpVu3bkmSFStWZO7cuTnllFM2up8ePXpk1qxZDf6856xZs9KjR48kyTHHHJOampoGbS699NJ8+tOfbjAbAgAAAPh42OofuqyoqMigQYMyefLkPPLII5k/f37Gjx+fFi1apH///qXtRo8enZtvvrm0PGjQoMyePTt33nlnFixYkDvuuCNz5szJ4MGDk7z/+xJdunRp8K9p06aprKzc5FQPAAAAYMfUqL+5ecIJJ2TFihW58cYbs2zZsnTr1i0XXnhhgxkVCxcuTLt27UrLPXv2zKhRozJx4sRMmjQpHTt2zKhRoxpcGgIAAAD8/WhUKFFRUZHa2trU1tZudJtx48att66mpma9SzQ2ZUP7AAAAAD4etvryDQAAAIByEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhWjamEZr1qzJbbfdlunTp2fp0qXp3r17RowYkT333HOT7WbOnJlJkyZl4cKF6dChQ4YNG5Z+/fqVbh83blweeOCBBm26d++eSy+9tDHdBAAAALZjjQolJk+enLvvvjvnnHNOOnXqlNtvvz1jxozJ2LFj07Jlyw22mTdvXsaOHZva2tr069cvf/rTn3LVVVflBz/4Qbp3717a7sADD8x55533Px1s2qguAgAAANu5rb58Y82aNZk6dWqGDBmSmpqadOnSJSNHjszy5cvz0EMPbbTdlClT0qtXrwwdOjTV1dUZOnRoevXqlSlTpjTYrlmzZqmsrCz922WXXbb+UQEAAADbva0OJRYtWpS6urr07t27tK558+bZb7/98swzz2y03bx589KnT58G6/r06ZN58+Y1WDd37tycccYZ+cY3vpGf/exnqa+v39ouAgAAADuArb42oq6uLklSWVnZYH2bNm3y5ptvbrJdmzZt1muzdn9JctBBB+XQQw9N+/bts2jRokyaNCmjR4/O5ZdfnmbNmq23z2nTpmXatGlJkssvvzxVVVVb+3C2uXX71LRp0+2ynzuCDdVNPRvP2Cwv9Swfr/XyMjbLSz3LSz3LRy3LSz3LSz3L5+N4nrTZUOLBBx/MhAkTSssXXHDBNuvMpz/96dL/u3Tpkr333jvnnntu/vznP+fQQw9db/uBAwdm4MCBpeXFixdvs7411rp9qqqq2i77uSPYUN3Us/GMzfJSz/LxWi8vY7O81LO81LN81LK81LO81LN8dtTzpE6dOm30ts2GEn379m3wQ5TvvfdekvdnPnwwjamvr19vJsQHVVZWrncpRn19/XozLj5ot912y2677ZZXX311c90EAAAAdjCb/U2Jli1bpmPHjqV/1dXVqayszKxZs0rbrFixInPnzk3Pnj03up8ePXo0aJMks2bNSo8ePTbaZsmSJXnjjTfStm3bLXksAAAAwA5kq3/osqKiIoMGDcrkyZPzyCOPZP78+Rk/fnxatGiR/v37l7YbPXp0br755tLyoEGDMnv27Nx5551ZsGBB7rjjjsyZMyeDBw9Okrzzzjv5P//n/2TevHlZtGhR5syZkyuuuCJt2rRJv379yvBQAQAAgO3JVv/QZZKccMIJWbFiRW688cYsW7Ys3bp1y4UXXpiWLVuWtlm4cGHatWtXWu7Zs2dGjRqViRMnZtKkSenYsWNGjRpVujRkp512yksvvZTf//73WbZsWdq2bZtevXrln//5nxvsFwAAAPh4aFQoUVFRkdra2tTW1m50m3Hjxq23rqamJjU1NRvcvnnz5rnwwgsb0x0AAABgB7TVl28AAAAAlINQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAACiEUAIAAAAohFACAAAAKIRQAgAAAChE08Y0WrNmTW677bZMnz49S5cuTffu3TNixIjsueeem2w3c+bMTJo0KQsXLkyHDh0ybNiw9OvXr8E2r7zySm6++ebMnj07K1euTOfOnXPeeeelurq6MV0FAAAAtlONmikxefLk3H333Tn99NNz2WWXpXXr1hkzZkyWL1++0Tbz5s3L2LFjM2DAgFx55ZUZMGBArrrqqjz77LOlbRYtWpSLLroo7du3z8UXX5yf/OQnOfHEE9OiRYvGdBMAAADYjm11KLFmzZpMnTo1Q4YMSU1NTbp06ZKRI0dm+fLleeihhzbabsqUKenVq1eGDh2a6urqDB06NL169cqUKVNK29xyyy3p06dPvvKVr2TvvfdOhw4dcvDBB6eqqqpxjw4AAADYbm11KLFo0aLU1dWld+/epXXNmzfPfvvtl2eeeWaj7ebNm5c+ffo0WNenT5/MmzcvSbJ69eo89thjqa6uzqWXXpoRI0bkggsuyMMPP7y1XQQAAAB2AFv9mxJ1dXVJksrKygbr27RpkzfffHOT7dq0abNem7X7W7JkSd55553ccccdOfHEE3PyySdn9uzZueaaa9KiRYscfPDB6+1z2rRpmTZtWpLk8ssv3y5nVKzbp6ZNm26X/dwRbKhu6tl4xmZ5qWf5eK2Xl7FZXupZXupZPmpZXupZXupZPh/H86TNhhIPPvhgJkyYUFq+4IILtklHVq9enSTp27dvvvCFLyRJ9tprrzz//PO55557NhhKDBw4MAMHDiwtL168eJv07cNYt09VVVXbZT93BBuqm3o2nrFZXupZPl7r5WVslpd6lpd6lo9alpd6lpd6ls+Oep7UqVOnjd622VCib9++6d69e2n5vffeS/L+zIcPpjH19fXrzYT4oMrKytTX1zdYV19fX5px0bp16zRp0mS9v7LRuXNnl3AAAADAx9Bmf1OiZcuW6dixY+lfdXV1KisrM2vWrNI2K1asyNy5c9OzZ8+N7qdHjx4N2iTJrFmz0qNHjyTvTznZZ5998sorrzTY5tVXX83uu+++VQ8KAAAA2P5t9Q9dVlRUZNCgQZk8eXIeeeSRzJ8/P+PHj0+LFi3Sv3//0najR4/OzTffXFoeNGhQZs+enTvvvDMLFizIHXfckTlz5mTw4MGlbY4//vg8/PDDmTZtWl577bVMmzYtDz/8cI455pgP+TABAACA7c1W/9BlkpxwwglZsWJFbrzxxixbtizdunXLhRdemJYtW5a2WbhwYdq1a1da7tmzZ0aNGpWJEydm0qRJ6dixY0aNGtXg0pB+/frla1/7Wu644478+7//e/bYY4+ce+65G/w9CQAAAGDH1qhQoqKiIrW1tamtrd3oNuPGjVtvXU1NTWpqaja57yOOOCJHHHFEY7oFAAAA7EC2+vINAAAAgHIQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIUQSgAAAACFEEoAAAAAhRBKAAAAAIVoWnQHAAAA2DFNPnnforvwsfL3WE8zJQAAAIBCCCUAAACAQlSsWbNmTdGdKJdXXnml6C5sVlVVVRYvXlx0Nz421LN81LK81LO81LN81LK81LO81LN81LK81LO81LO8doR6durUaaO3mSkBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABRCKAEAAAAUQigBAAAAFEIoAQAAABSiYs2aNWuK7gQAAADw98dMiY/Yd7/73aK78LGinuWjluWlnuWlnuWjluWlnuWlnuWjluWlnuWlnuW1o9dTKAEAAAAUQigBAAAAFKLJJZdccknRnfh7s/feexfdhY8V9SwftSwv9Swv9SwftSwv9Swv9SwftSwv9Swv9SyvHbmefugSAAAAKITLNwAAAIBCCCXgY27+/Pm58847s3LlyqK7AgAA0IBQokC1tbWZOXNm2fd7//33Z/jw4WXfb2MsWrQotbW1ef755z/UNn/vbr311nzrW9/a6nZvv/12fvKTn6RDhw5p2rTph+7HnDlzUltbmyVLlnyobT4uttVrmG3H8Qb+/owbNy6XX3550d0APkJ33XVXzj333KK7URZbcu5y7rnn5q677voIe1VeH/5TynbqiiuuyLvvvpuLL754vdtefvnlfPOb38yFF16YPn36FNC7902YMCGf+MQnyr7fww47LJ/61KfKvt911dbWbvL2z3zmM/nyl7+82f1UVVVlwoQJ2XXXXcvVtUYZN25c3nrrrY/s7/yOGzcuDzzwQI488sicffbZDW771a9+lbvuuisHH3xwvvvd7+b444/P5z//+a2+j/Hjx+eYY47JP/7jP5ar29uVtTVMkiZNmqRdu3bp169famtr06JFi4J7Vz6XXHJJ9txzz4wYMaLormyRrRnbjXX//ffnxhtvzC9/+csP293t1kdRx+3dx6UGt956ax555JH85Cc/abB+yZIlOeOMM/K9730vvXr1Ktv9zZkzJ9///vdzww03pHXr1lvV9oUXXsgFF1yQHj165Ac/+EHZ+rQlPky/N+X000/P9vITah/VmF6wYEFuu+22zJkzJ8uWLUvbtm1z6KGHZujQodlll10+1L63R9vD+cCG3qu31Zj+MOrq6nLHHXfkz3/+c15//fXsuuuu6dq1a4499tgcfPDBRXfv79qHOf5edtll2XnnnbdRz7a9j20ocdRRR+XHP/5xFi1alPbt2ze4bcaMGdl9991z4IEHFtS791VWVm7y9pUrVzbq2+3mzZunefPmje3WFpswYULp/4899liuv/76BuuaN2+epUuXbnY/O+2002Zr8XHVrl27/PGPf8zpp59eetO2lcVAAAAgAElEQVRctWpVfv/736eqqqq0XYsWLRr1pvov//IvW7RdY8fa9uDAAw/Meeedl5UrV2bu3Ln52c9+lnfffTdnnnlm0V37u7alY7sx/p4uRdqWddxRqMFHa8aMGTnmmGPywAMP5OWXX051dXXRXfrQWrVqVXQXGtjWY/q5557L6NGjs//+++fb3/52dtttt/z1r3/Nr371qzz++OMZM2bMNvlSrGjOBzZv0aJFueiii9KyZcsMGzYse+21V1avXp3Zs2fn5z//ea677rpG7XdHPo/cnnyY4+/mQq/t/Tnafnv2IR188MFp06ZN7r///gbf6K9cuTIPPvhgjjnmmCTJddddl9mzZ6euri7t2rXLZz/72Rx33HHZaaf3r2xZ++35vvvumylTpmTFihX53Oc+l2HDhuX222/Pvffem4qKigwePDhDhgwp3U9tbW2++tWv5vHHH8+cOXPSunXr/NM//VMOP/zwBtt885vfTE1NTRYtWpSRI0fm/PPPz/Tp0zNv3rwMHz48xx57bO67777cddddWbRoUaqqqnL00Udn0KBBpT6ua91vENd+QzN06NBMnDgx9fX1OeCAA/L1r3/9Q6W2HwwS1r65rRsurA0l/va3v+Xmm2/OM888k9133z2nn356evfunSSlx37ZZZdln332KaXKF110UW655ZbMnz8/1dXVOeussxr8qZsZM2bktttuy1tvvZUDDjggn/rUp3LjjTfm1ltvTZIsXrw4N910U55++um89957qaqqype//OV8+tOfXu+x3HrrraWEfe14WfvN1fz58/Mf//EfmTt3bpo3b56+ffvm9NNPL8tJTteuXfPmm2/mj3/8Y4488sgkyZ///Oc0a9Ys++23X6l+637LNn/+/PziF7/I888/n9WrV6djx4459dRTc8ABByR5fzbQL3/5yzz99NNp3rx5DjjggJx22mml5+eD4/qee+7JypUrc8MNN+T3v/99/u///b9ZsGBBmjdvnv333z+nnXZadttttwb9fvbZZzNx4sS88sorqa6uzte+9rVN/hmiZ555JjfffHOef/75fOITn0jfvn1z8sknl2r41FNP5de//nXmz5+fnXbaKZ06dcrZZ5+dLl26bLaGzZo1Kz2u/v37Z/bs2fnv//7vnHbaafn1r3+dP/zhD3n77bez1157Zfjw4dl3331LbRcsWJBf/epXeeqpp7J69ep06dIlX/va19KlS5c899xzmThxYv7yl79k5cqV6dKlS4YPH54ePXo0uP+lS5fmqquuyuOPP542bdqktra2wet8c+Nn7XPRu3fvTJ48OStWrMghhxySESNGZOedd864cePy1FNP5amnnsp//dd/JUn+7d/+LVVVVbn++us3efza3DjZlrZ0bK9evTr/+Z//menTp6e+vj577LFH/umf/imHHHJIkmz02HjTTTcl+Z/X65e+9KXU1tZu8RjeUZS7jt/85jfzu9/9boPH4mTTx46nnnoqP/jBD3Ldddc1ONbfcssteeyxx/LjH/+40BpsTR3Wvt+s9cH34yS5/fbbM2PGjNTV1eUTn/hE+vTpk5EjRyZJ1qxZk7vuuivTpk3LG2+8kY4dO+aEE05o8LpvrC3p3+aey0WLFuX73/9+kuSMM85I8v7MxS2ZxrxixYo89NBDGT16dN59993MmDEjX/nKVxr0bVNjaPXq1Y0+LrVv336j/X7iiSfyn//5n3nppZeSJN26dcupp57a4IR9U8/ZujMht2R/29KWjunGvObWrFmT6667LnvssUe+853vlOpeVVWVT37ykzn//PNzyy235Iwzzsi9996bqVOnZuzYsUmSWbNmZcyYMTnppJNK57TXXHNNmjdvnq9//eul88vvfOc7+cUvfpFFixalW7duOfvss9f7ArAIjT0f2NLzzk2dy2zsvXpjY3pbHkc25cYbb0ySXH755Q2+7Kqurs6AAQOSvD/LdsmSJQ1m66xevTrnnntuBg8enC984Qu55JJL0rlz5+y888554IEH0r59+1x22WW5++67c//992fhwoVp1apVPvWpT2X48OGlzwlbOoZ+97vf5a677srixYtTVVWVE044IQMHDtzkY5s8eXLuvvvuvPPOOzn00EM3OCa39jPVR2lTx991rV69OjfddFMef/zx/Ou//mv22GOPnHvuuTnmmGNy/PHHJ/mfz6KzZ8/Ok08+maOPPjqnnHLKZo/RRfnYhhJNmjTJZz7zmdx///350pe+VCr0Y489liVLluSII47I6tWrs9tuu+Wf//mf07p16zz33HOlywiOOuqo0r6efvrp7Lbbbrnkkkvyl7/8Jddee21efPHFfPKTn8zo0aMze/bs3HDDDendu3eDg9ett96aYcOG5dRTT83MmTMzbty4dO7cucGJxrpuueWWDB8+PGeffXaaNGmSadOm5dZbb81Xv/rV7L333pk/f36uv/76NG3aNMcee+wW12PRokV5+OGH8y//8i959913M3bs2EycODFnnXVWI6q79SZOnJhTTjklZ5xxRn7zm99k7NixGT9+/Ca//b/55ptz8sknp23btvnFL36Ra6+9NldddVUqKioyb968XH/99Rk2bFj69euXp556KrfcckuD9jfccEPee++9fO9730urVq3yyiuvbPS+jj/++CxYsCBLly7NeeedlyTZZZdd8s477+TSSy/NPvvsk8suuyxLly7N9ddfn/Hjx2/xLITNOfLII3PfffeVTkzW/n/hwoUbbfPTn/40Xbt2zQ9/+MM0adIk8+fPL82OefPNN/O9730vRx55ZIYPH55Vq1bllltuyZVXXpkxY8aUXgtPPfVUWrVqlf/9v/93ab8rV67Ml7/85XTu3DlvvfVWfv3rX+enP/1p6U11rV/+8pelD3q33357Lr/88lx77bUbnDY2f/78jBkzJrW1tfn617+epUuX5he/+EWuu+66fOtb38qqVavyox/9KEceeWTOO++8rFq1Kn/5y18afXBs3rx5Vq1alV/96lf54x//WHqju/vuu3PppZfmmmuuSdu2bfPGG2/k4osvTs+ePXPRRRelVatWee6557J69eokyTvvvJPDDz88p512WioqKnLPPffksssuyzXXXNPgUqPbb789J510Uk466aTMmDEj1113Xfbff/9UVVVt8fh5+umnU1lZmYsuuiivv/56rr766uyxxx75X//rf+X000/Pq6++mk6dOuWkk05K8n4aviXHr02Nk4/CloztqVOn5re//W3OPPPM7L333nnwwQfz4x//OFdccUX22muv0nYfPDbutNNOWb16dW655ZZce+21SVI6lmzpGN6RlLOOmzoWb+7Ysf/++6dDhw554IEHcsIJJyR5/8To97//fY477rjCa7A1ddiUmTNn5re//W2+8Y1vpEuXLqmvr8+zzz5bun3ixImZOXNmRowYkU6dOpXej3bZZZePdOrzxp7LqqqqfOtb38pPfvKTXHXVVdlll122+HU/c+bM7L777unSpUsOP/zwXH311TnppJMafLu2qTH0YY5Lm+r3O++8k0GDBqVr165ZsWJFfvOb3+SKK67I1VdfnaZNm272OVvX5vb3UdiSMd2Y19yLL76Yl156Keeff/5676O77bZb+vfvnz/84Q8ZMWJEevXqlRtuuCF1dXWl4HHXXXfNnDlzSqHE008/nWHDhpX2sXLlytx55505++yz06xZs4wbNy4///nPc+GFF5a1PuWwpecDa23qvHNz5zIbe6/e2Jgu4jiydOnSPPHEEznxxBM3eP69NjgYOHBgLr744rz55pul+syaNSt1dXUNQpMHH3wwAwcOzOjRo0uXR1VUVOS0005L+/btS18O3nTTTaVz62TzY+hPf/pTbrrpppx66qnp3bt3nnzyydx4442prKxM3759N/jYHn744UycODFf/epX06tXr8ycOTOTJ09ucKlSuT5TbStbcvxN3q/fv/3bv+Wll17KD37wg01+6XL77bdn2LBhGT58eCoqKrb4s28Rio+FtqGjjjoqixcvzv/7f/+vtG7GjBnp06dPqqqq0rRp05x44onp1q1b2rdvn8MOOyxHH310/vCHPzTYT6tWrXLGGWekc+fO6d+/fz75yU+mrq4uJ510Ujp16pTPfe5z2X333TN79uwG7fr165ejjz46nTp1ytChQ3PAAQdkypQpm+zzsccem5qamrRv3z7t2rXLb37zm5xyyimldX379s2QIUNKKeyWWptwdu3aNT169MjAgQMb1GVbGzx4cPr27Zs99tgjJ510UpYuXZoXX3xxk21OPPHEHHDAAencuXO++MUvZsGCBXnjjTeSvH/i2bt37wwZMiSdOnXKwIED069fvwbtFy9enH333Td77bVX2rdvn4MOOigHHXTQBu+rRYsWad68eSllr6ysTNOmTfPQQw/lnXfeyXnnnZcuXbpk//33z1lnnZU//elPee2118pSm/79++f555/Pq6++mrq6ujzxxBM54ogjNtlm8eLF6d27dzp37pyOHTumX79+pW/w77333nTt2jWnnHJKqqur07Vr14wcOTLPPfdcXnjhhdI+mjVrVpqNsHZGwlFHHZWDDz44HTp0SLdu3XLGGWfk6aefzuuvv97g/r/4xS/moIMOSpcuXXLOOeeU0t0Nueuuu3LYYYfluOOOyx577JHu3bvnzDPPzCOPPJL6+vosX748y5YtS9++fdOxY8fS66wx31g999xz+cMf/pBevXrl3nvvzcknn5yDDz649I1HZWVl6bXzX//1X9l5553zzW9+M926dUunTp1y+OGHlz64HHDAATn88MNTXV2dzp0756tf/WqaNWuWxx9/vMF9Hn744Tn88MPTsWPHnHjiiWnSpEmeeuqpJNni8dOqVaucddZZqa6uTp8+fVJTU1M6nrRq1SpNmzbNzjvvXBqbO+200xYdvzY1Tj4KWzK2f/vb3+a4445L//7906lTp5x44onZb7/91vuxpg8eG6uqqkozTdbWZO0J1paO4R1JOeu4qWPxlhw7jjrqqNx///2l/T355JOpr68vfcO2rWzpcXJL67ApixcvTmVlZXr37p2qqqrss88+pRPWd955J3fffXe+/vWv56CDDkr79u3Tv3//fPazn93s+/LLL7+c4cOHN/j3YX6EbWPP5U477VQ6EW/dunUqKyu3eGbfjBkzSs/l/vvvn5133jmPPvroFt1vkg91XNpUv2tqalJTU5M99tgjXbt2zTnnnJNFixblueeeK+1zY8/Zhmxufx+FLR3TW/uae/XVV5MknTt33uDt1dXVWbZsWZYsWZLOnTunsrKy9H4zZ86cHHfccZk7d25WrVqV1157La+//nqD3ztZtWpVRowYkW7duqVr16457rjjMmfOnO3mNzvW2przgbU2dd65uXOZjb1Xb2hMf5jjyIfx2muvZc2aNZs9v+rRo0c6d+5cmkGcvB+a9e3bt8EM6/bt2+crX/lKOnfuXNrn4MGDc8ABB6R9+/bZf//9c8opp+SPf/xj6QufZPNj6Le//W0GDBiQY489Np06dcrnP//59O/fP5MnT95on6dOnZrPfOYzDT53devWrcE25fpMta1syfH33XffzRVXXJG//e1v+f73v7/ZWaCHHXZYPvvZz6ZDhw5p3779Fn/2LcLHdqZEkuyxxx7Zf//9c99996VPnz5544038uSTT2bUqFGlbe69997MmDEjf/vb37JixYqsWrUqu+++e4P9VFdXN0ib27Rps961eG3atFnvrw2se/LfvXv39T7QrOuDsyiWLFmS119/PRMmTMjPf/7z0vrVq1dv9cH/gyfxSdK2bduP9K8jdO3atcF9J0l9ff0Wt1n7oquvr0+7du3yyiuv5B/+4R8abN+9e/dMnz69tDxo0KD8/Oc/zxNPPJEDDzww/fr12+QlBhuyYMGCdO3aNS1btiyt69mzZyoqKvLyyy+nY8eOW7W/Ddlll13Sr1+/3HfffWnVqlV69eq12WtKBw8enOuvvz4PPPBADjzwwBx66KGlE5AXXnghTz/99Ab/Astrr71WOkh36dIlzZo1a3D7Cy+8kNtvvz0vvvhili5dWhpnixcvTrt27UrbfXBst2jRIl26dMnLL7+8wb6+8MILee211/Lwww+vd9vChQvTo0ePHHHEEbn00ktzwAEH5MADD0xNTc0WX1f7xBNPZPjw4Vm9enVWrlyZQw45JMcee2xmzpyZnj17lrbbaaed0r1791I/X3zxxey7774b/Vasvr4+kyZNypw5c1JXV5fVq1dnxYoVWbx4cYPtPniJSZMmTdK6devSa2tLx8+6x5jddttti06ON3f82tQ4+Shsbmy//fbbefPNNxs8T0my7777rnes3NQMsw/a0jG8IylnHTd1LN6SY8cRRxyRiRMn5plnnknPnj1z33335ZBDDtnmP1S8JcfJranDptTU1GTq1KkZOXJk+vTpk4MOOih9+/ZNs2bN8vLLL+e9997LD3/4wwZtNnTusK6OHTvmggsuaLBu6dKlDWarbY3GvK9uymuvvZa5c+fm/PPPT/L+N579+/fPjBkzSpe1bMn9bovj0muvvZZJkyblueeey5IlS0rnQWuPx5t6zhqzv4/Clr73b+vX3P7775+nnnoqhxxySJ5//vl861vfyu9+97s8//zzeemll9KhQ4cGx85mzZqlU6dOpeW2bdtm5cqVWbZsWeE/oNnY84G1NnXeublzmTZt2mxxPz/MceTD2JrPDmsDkiFDhmTp0qV59NFH15shvKFz6tmzZ+eOO+7IggUL8vbbb5eei7q6ulJNNzeGXn755dIMorX23Xff9T6gf9CCBQvW+6a/e/fupS+AyvmZalvY0uPvtddem8rKynzve9/bot+a29BztCWffYvwsQ4lkvcT5uuvvz5Lly7N/fffn1122aU09efhhx/Of/zHf5SuE2/VqlXuueee/Pd//3eDfTRp0qTBckVFxQbXfTAFbKwPTn9fu78zzzxzvZOsrbXuB69y9XdLfbBeFRUVSTZ/cFy3xlvS5oOOOuqo9OnTJ48//nhmzZqVf/3Xf82QIUM2+1dDinDkkUdm3LhxadGiRU488cTNbl9bW5sBAwbk8ccfz5NPPpnbbrstZ555Zo466qisWbMmn/rUpzZ4HdoH3zTXvdRi7aUGBx54YEaOHJk2bdrkrbfeysUXX/yhflhwzZo1Oeqoo/KFL3xhvdvWvkGdc845GTRoUJ544ok8+uijueWWW/Ltb397ozNbPmi//fbL1772tTRp0iRt27ZN06ZN89e//rXR/V1r3Lhxqa+vz6mnnprdd989zZo1y/9n787jqqj+x4+/WAQF2ZFdQAI03FBJzVRKTM2iTcNyt1JzqyzN+vrxI6RpLtmiWFimuaJW5lKpHxEVsDTNnUWvimxeUBFFERC4vz94MD+vgIAC96rv5+Ph46Fzz5w5czwzZ+bMWT799NNyeVEb19a9lPXq3L/uVk7qS03LdmWqM6N0XZVhfVBb+Xi3e3F17h2WlpYEBAQQHR2Ni4sLBw8eZMqUKfecnpqojTwoa/y7/fq6s2zY29vz1VdfceLECY4dO8aKFSv4+eef+eyzz5T9pkyZUu4FsqLr+HbGxsblGrLv/DhQnfRVdLzq1qt3ExUVRUlJCWPHjlW23d6oV53j1tV9ac6cOdja2jJy5EhsbW0xMjLigw8+UPLmbv9nFT24VxVffalOma7pNefs7AyUvvg2a9as3O9paWmYm5srX7z9/Pz4/fffSUpKwsnJCWtra/z8/Dhx4gRpaWnlVoW5c0hIWRmoz2fKytzv88Dd6uLqPMtU1/3cR+6Hs7Oz8lHkzt7Fd+revTurV68mMTGRc+fOYWlpWW7Fwjvr5YsXLzJ79myCgoIYMGAAjRs35ty5c3z99dda19a9lqGycPeiNt+p6kJ177/t2rVj7969JCUlVWsFyTvvf9V999WFh75RonPnzvz444/s3buX6OhounfvrrxEJCYm4u3trdXF727j+Gvq9OnTWpXs6dOna/SV0traGhsbGzIzMwkMDKy1dD0MXFxcyq3VW9GXZTs7O3r27EnPnj357bff+PPPPyttlDA2Ni53Q3R1dSU6OpqbN28qX7uTkpKq1f2tJlq3bo2xsTG5ubnKhGxVcXZ2xtnZWekRsmvXLnr06EGzZs3466+/lCFK1ZWRkUFubi4DBw5UJgfav39/hWFPnz6No6MjUPoimJqaWunkTM2aNatWrxJPT088PT15+eWXmTVrFnv27KlWo4SpqWm5uB0dHTE2NlYesqC0Qjp9+rQy0amnpycxMTGVzkacmJjIiBEjlLGdOTk5XLlypcr03K62yk9FZbO696/Kykl9uVvZNjMzw8bGhqSkJK3VkBITE6vMn4rypCZl+EFTV/l4u+reO4KCgliwYAEODg5YW1vX20pWVd0nq5MPZS9iOTk5yu8VDSU0MTGhffv2tG/fnpdffplRo0aRlJSEr68vDRo04OLFi3UyYWx101eVsv+/6r4oFhcXs2fPHgYOHFhuPPuiRYvYvXt3tSbgu9/7UkXpzs3NJT09nbfeekvJ87Nnz1JcXKwVZ2X/Z3c+uFc3vvpQ3bq/Jtecp6cnrq6ubN26laeeekrrBTA7O5vY2Fiefvpp5QWvbF6J2NhY/Pz8lG0xMTFkZGRozSeh7+71eaA6qvMsU1G9VFGZdnNzq9P7SGUaN25M27Zt2b59O3379i33wnrjxg2lJ3hZT55du3aRnJxMYGBglXN9nTlzhqKiIoYPH66E/ffff2ucTjc3NxITE7WeVaqqz1xdXSt87yqjz+9UNbn/BgUF4eXlxbx58/joo4+0Jqqujrp+970fD/WcElBaSXXt2pUNGzaQmZmpVVidnZ05d+4chw8f5sKFC/z888/KWPDacODAAXbu3MmFCxfYuHEjJ06coG/fvjWKIyQkRJlNNiMjg5SUFPbs2cPGjRtrLZ0Por59+3L06FE2b97MhQsX2LVrFwcOHNAKs2zZMo4cOUJmZibJyckcPXr0rje0Jk2akJqaSkZGBteuXaOoqIhu3bphamrKokWLSElJIT4+niVLltCxY8daGbpRxsDAgPnz57No0aJKu5uWKSws5IcffuDkyZNkZWVx+vRprZt17969ycvL46uvvuL06dNkZmZy7NgxIiIiuHnzZqXx2tvb06BBA7Zt20ZmZib//vsv69atqzDsL7/8wrFjx0hNTeXbb7/F2NiYrl27Vhj2pZdeUibSOXfuHGq1mkOHDinLx2ZlZbF69WqSkpK4ePEiJ06c4Pz58/fV6NOwYUN69erF6tWr+ffff0lLS+P7778nJydHWXmnd+/e5Ofns2DBAlQqFWq1mtjYWOUFwNnZmZiYGNLS0lCpVHz99dc1ngCttspPkyZNUKlUZGVlKV2Nq7p/VVVO6ktVZfvFF19ky5YtxMbGkpGRwbp160hISKhy4sQmTZpw69Ytjh07xrVr1ygoKKhRGX7Q1FU+3q669442bdrQuHFjfv75Z55++ul6m7G7OvfJqvLBxMQEHx8fNm3aRGpqKklJScpKVWV2795NVFQUKSkpZGVlsXv3boyMjHB2dqZRo0YEBwezcuVKdu3ahVqtJjk5mR07drBz5877PsfqpK86mjRpgoGBAf/++y/Xrl0jPz//ruH//fdfcnNzCQoKUuYZKvvTpUsXoqOjq3Xc+70vVZRuc3NzLCwsiIqKQq1WEx8fz/fff6/1Rflu/2d3qk589aW6dX9NrjkDAwPGjBlDRkYGc+fO5dSpU1y6dIl///2XGTNm0KRJE15//XUlfNm8EjExMcoLctmQjjvnk3gQVed5oDqqepaBiuvqisp0Xd9H7uatt95Co9Hw8ccf89dff5GRkUF6ejo7duwoNzwjKCiI2NhYzp8/X244RUWcnZ3RaDT8/vvvZGVlERsbW+VcehUJDg4mJiaGbdu2ceHCBf78809iY2OVVSUq0rdvX/bs2aP13nXnx0p9faeq6f23Z8+eDBs2jHnz5nHs2LEaHauu333vx0PfUwJKu/Hv2LGD5s2baz2QP/vssyQnJ/PNN9+g0Wjo1KkTwcHB1a58q/Laa6+xf/9+li1bhqWlJWPGjCk36UpVgoKCMDU1ZcuWLaxduxYTExPc3Nz0YpZYXfL19WX06NFs2LCBdevW0bp1a1566SUiIyOVMBqNhh9//JHLly/TsGFDWrduXenSOlB6kcfHx/Pxxx+Tn5+vLAk6depUli9fzieffKK1pGNtu33egbsxNDTkxo0bLF68mCtXrmBhYUH79u2VceC2trbMmDGDNWvWMGvWLAoLC7G3t6dt27Z3feixtLRk3LhxrF27lu3bt+Pu7s7QoUPLjXkEGDRoECtWrCAjI4OmTZsyZcqUSse2eXh4EBYWRmRkJKGhoZSUlODg4KB0HTQxMeHChQssWLCA3NxcrKys6NatmzLT+L0aNGgQULrs740bN2jWrBlTp05VxkDb2toSFhbGqlWrCAsLw8DAAHd3d2VFmjFjxrBkyRKmTJmCra0tr732Wo3nYTE1Na2V8hMcHEx4eDgffPABhYWFLFq0qMr7V1XlpD7drWw/99xz3Lx5k9WrV5OTk4OLiwsffvhhlSslNG/enGeffZavv/6a3NxcZUnQ6pbhB1Fd5OPtqnvvMDAw4JlnnmHDhg1VTspb26q6T1YnH8aMGUNERASffPIJjo6OvP3220yfPl353czMjE2bNrFy5UqKi4txc3Nj0qRJSu+bAQMGYGVlxZYtW/jhhx9o1KgRnp6e933Pqm76qqPsnhUZGUlERATdu3e/66Sau3btomXLlhXOU/Dkk0+yZs2aaj383u99qbJ0T5w4kWXLlvHhhx/i5OTEkCFDlCWyoer/s9sZGhpWGV99qk7dX9NrztfXl9mzZ/Pzzz8zd+5cbty4gUfpTeMAACAASURBVK2tLR07dqRfv37l5n7w8/Pjr7/+UnpKODg4YGtri6Gh4QM7F8/tqnoeqI6qnmWg4rrawcGhwjJd1/eRyjg6OjJnzhw2btzI6tWryc7OxsLCAg8PD0aPHq0VtmXLltjZ2WFvb6/0jr0bDw8Phg8fzqZNm4iMjKR58+YMGTJEWXK2ujp27MiIESPYsmULP/30E/b29rz11luVrrwBpRM6ZmZmEhkZSUFBAQEBATz//PNak3Xq6zvVvdx/n332WTQaDfPmzWPy5MnV7jFR1+++98NAow+zezyE7lzzXNS95cuXc/z4cZ09WAghxKPi+++/R61WM23aNF0nRYhHglxzor4VFhYyevRo3nzzzTpfYUmIh374hnh4bd68meTkZNRqNTt27OB///tftbqXCSGEuDd5eXmcOnWKvXv38vzzz+s6OUI89OSaE/WtpKSEq1ev8ssvv2BiYsKTTz6p6ySJR8AjMXxDPJzOnDnDli1byMvLw8HBgYEDB9Z4zg4hhBDVN3fuXFQqFT169Cg3IZcQovbJNSfq26VLlxg/fjx2dnaMHTu2xvNpCXEvZPiGEEIIIYQQQgghdEKGbwghhBBCCCGEEEInpFFCB8LDw/n8888r/X337t06mSH/YRIaGsrSpUt1nQzxgLh+/TojR45ErVbrOinlLFiwgC1btug6GXrl5MmThISE1Hg1FFG5quqlh4U+X+srV67kxx9/1HUyakTyUzcelev1fuhz2ZR6XUj5LO+RHSQUHh6utUyMhYUFPj4+DBkyBFdXVx2m7NFwe/4bGRlhbm5O06ZN6dSpEz179rzv8WuTJk3SyZrjD4Lw8HByc3P5+OOPdZ0UvbFx40batWuHk5MTAMuWLSMpKYnU1FSsra0JDw8vt8++ffvYuHEjFy5cwNLSkj59+pRbQ3vbtm1s376drKws7O3tefXVVwkMDFR+3717N4sXLy4X96pVqzAxMQGgf//+TJ8+naCgIMzMzGrztGtFReXp0KFDfPnll7zwwgu8/vrrOkzd3YWGhtK0aVPeeustXSel2kJCQu76e2Bg4F2XfqzMiBEjeBRGc+rqWg8NDa1wLXg3NzcWLFgAwEsvvcSECRN4/vnnq7X8nj6Q/Lx/dz4P2dnZ0bFjR0JCQipdarsurtdx48bRu3fvcv8XDypdlU0onZw0MjKS/fv3k5ubi52dHW+88QZdunQB9K9el3ei+ldX5TM2NpZNmzZx4cIFGjVqROvWrRk6dCjW1tZKmD/++IMdO3Zw8eJFLCwsCAgIYPDgwcr9Rlfl85FtlABo3bo1EyZMACA7O5tVq1Yxf/58vvzyywrDFxUVyWQvtags/0tKSrh27RonTpxgw4YNxMTEMG3atEor4+q4cw1uISpTUFDArl27mDJlirJNo9EQGBhISkpKubWhAQ4fPsw333zDiBEj8Pf3Jz09nYiICExMTJT1rnfs2MHq1asZPXo0Pj4+qFQqIiIiMDc311pr29TUlIULF2rFX9YgAeDu7o6joyN79+7V+Vra1bF3716+++47Bg8eLBPP1oElS5Yofz906BARERFa224vO1D9eksfHozrmi6v9UmTJlFUVKTEe+vWLSZNmqQ1q72lpSVt2rRhx44dD0RvScnP2lP2PFRUVERiYiLfffcdBQUFjBw5UitccXExhoaGj8T1ej90WTaLioqYOXMmjRs3ZuLEidja2pKdna11H9bHer2m70RVKSurBgYGtZnMOo+7PtRV+UxMTGThwoUMGTKEjh07kpOTw9KlS/nmm2/473//C5Q2WqxatYp33nmHFi1akJWVxbfffsutW7cYM2YMoLvy+Ui/YTdo0EBpObK2tub5559nzpw5FBYWkpOTw/jx43n33XeJiori1KlTDBkyhF69evHrr78SFRXF1atXcXZ25vXXX+eJJ55Q4k1JSeGnn34iMTERExMTAgICGDFiRKWVSHJyMrNmzeKZZ57hjTfe0PotKyuLCRMmMGvWLB577DFl+86dO1m7di0REREkJSURFhbGtGnTWLt2LSkpKbi5uTFq1Ci8vLzqIOdqx+35b2tri6enJ23atGHKlCls3rxZ+SJ4/fp1li9fzqFDhygsLKRFixYMHz6cpk2bVhr3nV9Ax40bR48ePbh8+TJxcXE0atSIvn37PjRfBGpTWloaK1euJCEhARMTE1q1asXw4cOV/6uUlBSWL1/OmTNnKCkpwcnJiWHDhtGqVSsdp/zeHD58GIDmzZsr2958802gdNnZiiqHvXv30qFDB3r37g2Ao6MjL7/8Mps2baJ3794YGBiwd+9egoKC6Nq1qxLmzJkzbNq0SatRAtBqwa5IQEAAcXFxevPwUpnff/+d1atX884779C9e3cA9u/fz/r167lw4QJWVlY8++yzvPLKK8rDxLhx43jmmWfIzMxk//79mJubM2TIENq2bcv333/PoUOHsLGx4a233qJt27Zaxzt9+jSRkZFkZGTg5ubG6NGjlXtebm4uS5cuJTExkdzcXBwdHQkODlaWDQ4PDyc+Pp74+Hi2b98OwKJFi3BwcKiv7Lont5cVc3NzrW1ZWVmMGjWqXL311FNP3TUvoHyPl9DQUNzc3DAzMyMqKgoDAwO6d+/O4MGDMTR8MEd+6vJav7OhPCYmhoKCgnLLWAcEBLB27Vq9f4kGyc/adPvzUNeuXTlx4gT//PMPVlZW7N+/n+DgYH755ReysrL46aefWLp0qXK97ty5k3Xr1hEREaF1bX799dfk5+czZcoU1Go1K1as4PTp0+Tn5+Pi4kJISAgdOnQASq/3ixcvsmrVKlatWgXA+vXrAUhKSmLNmjWcOXNGefkeNGiQXjeM6LJs7t69m2vXrvHpp58qDREV1Sv6Vq/f7Z3IxMSE7OxsVqxYwdGjRwHw9fVl+PDhODs7A6XlpaKymp2dTUREBCqVCnt7e4YNG8aXX37JW2+9xdNPPw1wz3EnJiby66+/kpqaCoC3tzfDhg3Dzc1NOa+yBpYjR45QWFiIs7Ozzp9Z66p8njp1Cjs7O1544QWgtNz16dNHaxhbUlISPj4+yjOag4MDgYGB7N+/X+t4uiifD+aTRR24efMm+/btw93dXetL09q1a+nduzdffvklTzzxBH/88Qdbtmxh0KBBzJ8/n44dOzJ//nySk5MByM/P57PPPsPU1JTZs2czefJkTp06VWEXbYCEhATCwsJ48cUXyzVIQGlhadOmDdHR0Vrbo6Oj6datm1bL65o1axg4cCBz5szBwsKChQsXPnDdcd3d3fH399e6OBYvXoxKpWLy5MnMnj0bExMTZs2aRWFhYY3i/v3333F3d2fOnDm89NJLrFq1ilOnTtX2KTzQrly5wvTp02natCmzZs1i2rRp5OfnM3fuXEpKSoDSBx1ra2tmzZrFvHnzeO2118p9nX2QJCQk4OXlVaMW91u3btGgQQOtbSYmJly+fJmLFy8qYe7MFxMTE1QqldYXvsLCQsaOHcs777zD559/zrlz58odz9vbG5VKVeMyX58iIyNZu3YtkyZNUiq7s2fPsmDBAjp16sT8+fMZOHAgGzduZNu2bVr7/v7773h7ezNnzhyefPJJwsPD+eabb2jXrh3z5s3j8ccfZ+HCheXOf+XKlQwaNIjZs2fj6OjI559/TkFBAVCa/15eXnz88ccsWLCAvn37smTJEo4fPw6Udn/29fXl6aefZsmSJSxZsgR7e/t6yKm6d2e9VVVeVCYmJgYjIyNmzJjBm2++yR9//MG+ffvq6Sxqn66v9dtFRUXh7+9frsx5e3uTnZ2tl+OM7yT5WXdMTEwoLi4GShsbY2NjmThxIvPmzSuXf507dyYvL0/rRSY/P5+DBw/SrVs35d/+/v5MmzaNefPmKffk9PR0oLTniZ2dHf3791fuh1D6EWLmzJkEBAQwb948Jk2aRHJyMt9++219ZMM902XZ/Oeff2jevDk//vgjI0eOZOLEiaxfv75c2dXnev3Od6KCggLCwsJo0KABoaGhzJw5ExsbG2bMmKHUuVC+rBobGzN//nyMjIz47LPPGDduHD///LNWXtxr3A0aNCA/P5++ffsya9YsQkNDadSoEXPmzFHiz8/PVxrcJk+ezPz58+nfv3/9ZWQl6qp8tmjRgitXrnDw4EE0Gg3Xrl1j3759tGvXTtmnRYsWJCcnK+8/ly5d4uDBg1phQDfl85FulDhy5AhDhgxhyJAhDBs2jPj4eN59912tMH369KFz5844ODhgZ2fHli1bCA4OpmvXrri4uDBgwAAef/xxNm/eDJR2i8nPz2fChAm4u7vj5+fHqFGjOHDgQLlK8dChQ3z++ecMHz5cadWqSFBQEHFxcUrBSEtL4/Tp0/To0UMr3IABA2jVqhWurq7069eP9PR0srOzayOr6pWbmxuZmZkAXLhwgYMHDzJq1Cj8/Pxwd3dnwoQJ5OXlERMTU6N427RpQ58+fXBycuK5557DycmpyofyR82OHTvw8PBg8ODBuLm54eHhwfjx41GpVJw9exYovYG1adMGV1dXnJyc6NixI76+vjpO+b27ePEiNjY2NdrH39+fgwcPcvToUUpKSsjIyGDr1q0A5OTkANC2bVuio6NRqVRoNBrOnDlDVFQUxcXF5ObmAuDi4sKYMWP46KOPeO+992jQoAHTpk3jwoULWsezsbGhuLhYb6/nY8eO8euvv/LBBx/Qvn17ZfvWrVvx8/MjJCQEFxcXunXrRnBwMJs2bdLav23btvTu3RtnZ2dCQkK4desWjo6OBAYG4uTkRL9+/bh27ZryNaRMv3798Pf3x93dnbFjx1JYWEhsbCxQ2vvqxRdfxNPTE0dHR3r27EmnTp2Ii4sDSocrGBsbY2pqirW1NdbW1g9sD4A73VlvVZUXlXFzc2PAgAG4uLjQpUsXWrZsyYkTJ+rpLGqfLq/122VkZBAfH09QUFC538rSV/aQqc8kP+uGSqUiLi5O+ZJbVFTE+PHj8fLywt3dvdx8WY0bN6Zdu3Zaz0QHDhzA0NBQ+Xrv6elJr169cHd3x8nJiVdffRUvLy/+/vtvJQ5DQ0MaNmyo3A+h9Kttly5dCA4OxtnZGR8fH0aOHMn+/fu5evVqfWTHPdFl2czMzOTvv/+mqKiITz75hAEDBvC///2PNWvWaB1P3+r1u70TxcXFodFoGDt2LB4eHri6ujJq1Cjy8/M5dOiQEsedZfXEiRNkZGQwfvx4PD098fX1ZdiwYUqD2/3EbWRkROfOnencuTPOzs54eHgwduxYsrKyUKlUQOk7WU5ODpMnT+bxxx/HycmJTp066bxnb12VT19fX95//30WLlzIwIEDefvtt9FoNIwfP16J56mnnuKNN95g+vTpvPHGG4wdOxZ3d3cGDRqkdTxdlM9HevjG448/zujRo4HSIQI7duzgs88+47PPPlPC3D5kIi8vjytXrmh1t4HSVqeyrjjp6el4eHjQqFEj5ffmzZtjYGBAWlqaMqHJ2bNnmT9/Pu+++67WGMiKBAQEsHTpUg4cOEDXrl2Jjo7G29sbd3d3rXAeHh7K321tbQG4evUqdnZ21c4TfaDRaJTWw/T0dAwMDLRees3MzHB3dyctLa1G8d6eP1B6welzpaoLZ8+eJSEhocKurmq1Gm9vb55//nkiIiLYs2cPrVu3plOnTg/0REgVffmoSlBQEGq1mrlz51JcXKwMB9qwYYNSdvv3709OTg7Tpk1Do9FgZWVFYGAgmzdvVsL4+vpqle3mzZszefJk/vzzT6UrH/z/eQL08YsKQNOmTcnLy2PDhg00b95cGVaQnp5ervW9RYsW/Pzzz+Tl5Sndf2+/Nhs2bIipqanW/a3sAfnO6/X2vGvYsKHWfaGkpITffvuNffv2kZ2dza1btygqKqJly5a1eOb66fZ6C+49Lx62e6Yur/XbRUVFYWNjo9WAV0bfr/XbSX7WnrIXwpKSEoqKinjiiSd488032b59O7a2tlUO8evWrRvh4eEUFBRgampKbGwsnTp1Us4/Pz+fn3/+mUOHDpGTk0NRURG3bt0q9xx5p7Nnz6JWqyvsIZWZmYmVldW9n3Qd0mXZ1Gg0WFpa8s4772BoaIiXlxfXr1/np59+YsiQIUo4fSubd3snOnv2LFlZWQwdOlRrn8LCQuUjIlCurGZkZGBjY6O8k0Bp/XT7dXyvcUPpc+m6detQqVRcu3aNkpISNBoNly5dAkqHx3t4eGBpaXmv2VIn6qp8pqWl8eOPP9KvXz/atm3LlStXWLVqFUuWLFEaJuLj4/nll194++238fHxQa1Ws2zZMtavX8+AAQOU4+mifD7SjRKmpqZKIwGAl5cXw4YNY+fOnUovBFNT0zo5toODA1ZWVuzevZuAgIByXXJuZ2xsTPfu3YmOjubJJ59k7969WgWnTEWrTTxowzeg9KKqzrjumk5wc2f+GBgYPJD5U5c0Gg3t2rUrVzkAysNHSEgI3bp14/Dhwxw9epQNGzYwcuTIcj13HhQWFhZcv369RvsYGBgwePBgBg4cSE5ODpaWlkqvm7JZ3k1MTBg7diyjRo3i6tWr2NjYsHPnTho1alRpBWloaMhjjz1WrldVWfr0rWItY2Njw5QpUwgLC2PGjBn85z//qXKy2duv34ruXRVNzliT63Xz5s1s2bKFESNG4O7uTsOGDVmzZs0jsYzonfXWvebFw3bP1IdrvaioiD179hAUFFRhudf3a/12kp+1p+yF0MjICBsbG637X3Um/W7fvj1GRkb8888/tG7dmuPHjzN16lTl95UrVyoNH87OzpiamrJo0aJKh8OU0Wg09OjRo8LevLe/aOobXZZNa2trjI2NtXreubq6UlBQQG5urhJO38rm3d6JNBoNnp6evP/+++X2u72uv5cJ6u8n7jlz5mBra8vIkSOxtbXFyMiIDz74oMpyrWt1VT43btyIt7e3Ml+eh4cHDRs25L///S9vvPEGdnZ2REZG8tRTTyk9y9zd3cnPzyciIoL+/fsr91FdlM+Ho69qLTI0NKy0VcjMzAwbGxuSkpK0ticmJiqTqri6upKSksLNmzeV35OSktBoNFoTrzRu3Jhp06aRnZ3N/PnzuXXr1l3TFRQUxIkTJ9i+fTv5+fnKskIPm5SUFI4ePUrnzp2B0vzUaDRacz/k5eUpk3mK2tWsWTPS0tKwt7fHyclJ68/tvX+cnZ3p27cvn3zyCT169GDXrl06TPX98fT0VMbV1pShoSG2trYYGxsTFxeHr69vuRu4sbExdnZ2GBoaEhcXR/v27SsdJqDRaDh//ny5rwGpqanV+lqmS7a2toSGhlJQUMCMGTPIzc3F1dW1wvulnZ2dVnm6V6dPn1b+np+fT2pqqtJrJzExkQ4dOtC9e3dl2MKdw2KMjY2VuVIeZtXJi0eBPlzrBw4cIDc3t9JG3NTUVIyMjKr8gq0PJD9rT9kLYZMmTe5plbcGDRrQuXNnYmNj2bdvH9bW1vj5+Sm/JyYmEhgYSOfOnfHw8MDW1lbrKzRUfD8seya483nAyclJr+eS0mXZbN68OWq1WisvL1y4gKmpKRYWFsq2B6FeL3snatasGWq1GgsLi3Ll4G4fIFxcXLhy5YrWEICzZ89qNW7fa9y5ubmkp6fzyiuv0KZNG9zc3Lh586bW0BBPT0/Onz+vdx8j6qp8FhQUlLtHlv27LM8rC3PnBwddlM9HulHi1q1b5OTkkJOTo3R5yc/PV2YjrsiLL77Ili1biI2NJSMjg3Xr1pGQkEBwcDBQ2oWurAU6JSWF+Ph4lixZQseOHbVaIKG09WnatGlcvny5yoYJFxcXWrRowapVq+jUqZNez3pcXWX5n52dTXJyMlu3biUsLAwvLy8lP52dnQkICOD7778nISGBlJQUFi5ciJmZmTL7sai5mzdvkpycrPUnKyuL3r17k5eXx1dffcXp06fJzMzk2LFjREREcPPmTQoLC/nhhx84efIkWVlZnD59WqtR7kHk7+9PWlqa1lhltVpNcnIyV65coaioSMmjstb3a9eusWPHDtLS0khOTmbZsmX89ddfDB8+XIkjIyODvXv3cuHCBVQqFV999RWpqalaE9pu2LCBI0eOkJmZqUwelpKSQq9evbTSmJCQUG7lCX1kY2PD9OnTKSoq4tNPP6Vv377Ex8ezfv16MjIyiImJYevWrbW26s0vv/zCsWPHSE1N5dtvv8XY2Fi5L7i4uHDixAkSExNJT09n6dKlZGVlae3fpEkTVCoVWVlZStfPh1F18uJRoMtrvUxUVBStWrVSvmzdKSEhgccff7zOemnWJslP/dKtWzeOHj3K//73P5566imtFw9nZ2cOHDjA2bNnleeoOz/ANWnShMTERLKzs5WXuJdeegmVSsWSJUs4d+4carWaQ4cOaS1DrI90WTZ79eqlrBqXkZHBkSNHWL9+Pb169dLqIahv9frd3om6deuGlZUVc+fOJT4+nqysLOLj41mxYsVdG7jbtGmDi4sL4eHhyuSKP/30k1avpnuN29zcHAsLC6KiolCr1cTHx/P9999rxd21a1esrKyYN28eCQkJZGZmcvDgQZ3PjVRX5TMgIICDBw+yY8cOMjMzSUxMZNmyZTRr1kyZBLhDhw5ERUURFxdHVlYWx44dY926dUpvqzK6KJ+P9PCN48ePM2rUKAAaNWqEi4sLEydOpGXLlpU+sD333HPcvHmT1atXk5OTg4uLCx9++CGenp5AaWv31KlTWb58OZ988onWkqAVsbS05L///S+ffvopX3zxBR9++GGl6e3RowcJCQkPbDf5O5Xlv6GhIebm5jRt2pTXXnuNnj17an0pGDt2LMuXL2fu3LnKkqD/93//p9et9Lqye/duFi9eXOXShgkJCXz00Uda2zp16sSHH37IjBkzWLNmjbLCib29PW3btlWGGN24cYPFixdz5coVLCwsaN++vd4vt3Y37u7ueHt7ay199N133xEfH6+EKcur2/N1z549rFy5Eiid2yA0NBRvb29ln5KSErZu3UpGRgZGRka0bNmSmTNnav2/3LhxgyVLlpCTk4OZmRnNmjUjLCxMK57CwkIOHDig1RVXn1lbWzN9+nRmzJjBt99+y3vvvccvv/zCxo0bsba25uWXX661JaYGDRrEihUryMjIoGnTpkyZMkXp4vnqq6+SlZXFrFmzMDEx4emnn6Zbt25ac9EEBwcTHh7OBx98QGFh4QOxJOi9qE5ePAp0ea1D6Rj8EydO8N5771Waxri4OGU5bH0n+alfHn/8cWxtbUlLSyuXJ8OGDeO7775j+vTpmJub07dv33IfwkJCQvj++++ZMGECt27dYv369Xh4eBAWFkZkZCShoaGUlJTg4OBAx44d6/PUakyXZdPe3p6pU6eyYsUKJk+ejLW1Nc888wz9+vVTwuhjvX63dyKAsLAw1qxZw4IFC8jLy8PGxoaWLVsqc0hVxNDQkEmTJhEREcH//d//0aRJE4YOHcr8+fOVZ3hTU9N7jnvixIksW7aMDz/8ECcnJ4YMGcIXX3yhhGnYsCGhoaGsWLFCWZXDxcWFYcOG1UaW3bO6Kp9PP/00N2/eZNu2baxYsQIzMzNatWqlNYllv379MDAwYN26dVy+fBlLS0s6dOjA66+/roTRVfk00DzIA0QfMb/99hvR0dF8/fXXuk6K0FPr16/n77//Zt68eRWOrxUVO3LkCMuWLePLL7/UuxUYtm3bxsGDB/nPf/6j66QI8cDT52v933//ZeXKlcoSeg8CyU+hr/S5bD7K9XpycjIfffQRn3/+OV5eXrpOjs5I+SzPKDQ0NLRejyhqLD8/nwsXLvDjjz8SHBys1SomxO1WrVrF8OHDK+3KKirm5OSERqPBxsbmri3zupCcnEyvXr20xqIKIe6NPl/r586d4+mnn1a62T4IJD+FvtLnsvko1esHDhzg0qVLGBoacu7cOX744QesrKwICQmp8YT1DxMpn+VJT4kHQHh4OHFxcQQEBPDee+9Ji78QQgghhBBCr+3Zs4dff/2VS5cu0bhxY/z8/Bg2bJheT/ApdEMaJYQQQgghhBBCCKET+jWIRQghhBBCCCGEEI8MaZS4i+vXrzNy5EjUarWuk1LOggUL2LJli66TIXREyqbQZ1I+a4/kpRDiTkuXLqUmU8JlZWUREhLCmTNn6i5R4qGmz3XRJ598wt9//63rZIj79EgvCVqVjRs30q5dO5ycnABYtmwZSUlJpKamYm1tTXh4eLl99u3bx8aNG7lw4QKWlpb06dOHF198USvMtm3b2L59O1lZWdjb2/Pqq68SGBio/F62rOOdVq1apSyh079/f6ZPn05QUBBmZma1edq1Jjw8nD179gBgZGSkLPvZqVOncst+ipqRsin0mZTP2iN5KcSjJzw8nNzcXD7++GNdJ0UIQHd1EcDff//NunXryMzMxNHRkTfeeENrWdp+/fqxYsUKOnbsqHcrWYjqk7fCShQUFLBr1y6mTJmibNNoNAQGBpKSksKxY8fK7XP48GG++eYbRowYgb+/P+np6URERGBiYqKsQ7tjxw5Wr17N6NGj8fHxQaVSERERgbm5OQEBAUpcpqamLFy4UCv+sgdBKF3j1tHRkb179ypx66PWrVszYcIESkpKuHbtGidOnGDDhg3ExMQwbdo0GjZsqOskVqq4uBhDQ0O9mx1YyqbQZ1I+a4/kpRBCCF3TZV106tQpvvrqK0JCQujYsSMHDhxgwYIFzJgxAx8fHwDat29PREQER44coX379vWQI6IuSKNEJQ4fPgxA8+bNlW1vvvkmAJs3b67wAty7dy8dOnSgd+/eADg6OvLyyy+zadMmevfujYGBAXv37iUoKIiuXbsqYc6cOcOmTZu0HgaBKmemDQgIIC4uTq8fBhs0aKCch62tLZ6enrRp04YpU6awefNmQkJCKCoqIjIyktjYWK5fv07Tpk0ZMGAAoHpbUgAAHjpJREFU/v7+AJw8eZKwsDCmTZvG2rVrSUlJwc3NjVGjRuHl5UVeXh4jR45k4sSJWnl49OhRPv/8c7777jusrKzIzs5mxYoVHD16FABfX1+GDx+Os7MzAOvXr2f//v0EBwfzyy+/kJWVxU8//aR3DSdSNoU+k/JZeyQvhRAlJSWsWrWK6OhoAAIDAykpKdEKc+TIEX799VdSU1MB8Pb2ZtiwYbi5uWmFu3jxImvWrCEpKYkmTZowYsQI2rRpo/weHx/PqlWrOH/+PGZmZjz11FMMHjxYerY+4nRZF/3++++0bNmSV199FQA3NzdOnjzJ77//zvvvvw+AoaEh7dq1IzY2VholHmDSx6USCQkJeHl51egr+a1bt2jQoIHWNhMTEy5fvszFixeVMLd/aSoLo1KpKCoqUrYVFhYyduxY3nnnHT7//HPOnTtX7nje3t6oVCoKCwtrcmo65+7ujr+/P/v37wdg8eLFJCQk8O677/LFF18QGBjInDlzSE5O1tpvzZo1DBw4kDlz5mBhYcHChQvRaDSYmZnRoUMHYmNjtcLHxMTQpk0brKysKCgoICwsjAYNGhAaGsrMmTOxsbFhxowZFBQUKPtkZWURGxvLxIkTmTdvXrn/T30gZVPoMymftUfyUgixZcsWoqKiGDlyJDNnzqSkpKTc805+fj59+/Zl1qxZhIaG0qhRI+bMmaN1PQNERkby3HPPMW/ePB577DG++uor8vPzAcjOzmb27Nl4enoyZ84c3nnnHeLi4lizZk29navQT7qsi06dOkXbtm21wrRt25ZTp05pbfP29iYhIaHa6RP6RxolKnHx4kVsbGxqtI+/vz8HDx7k6NGjlJSUkJGRwdatWwHIyckBSi+k6OhoVCoVGo2GM2fOEBUVRXFxMbm5uQC4uLgwZswYPvroI9577z0aNGjAtGnTuHDhgtbxbGxsKC4uJjs7uxbOuH65ubmRmZmJWq0mLi6OiRMn4ufnh6OjI3369KFdu3bs3LlTa58BAwbQqlUrXF1d6devH+np6cq5d+/enYMHD3Lz5k2g9GH6n3/+oVu3bgDExcWh0WgYO3YsHh4euLq6MmrUKPLz8zl06JByjKKiIsaPH4+Xlxfu7u4YGRnVU45Un5RNoc+kfNYeyUshxB9//MFLL71Ely5dcHV1Zfjw4eV6MHXu3JnOnTvj7OyMh4cHY8eOJSsrC5VKpRXu+eefJyAgAGdnZwYOHMj169eVD0Dbt2/HxsaGt99+Gzc3Nzp06MCgQYPYtm2b1scb8ejRZV2Uk5ODlZWVVtxWVlZKHGVsbW3Jzs6muLj4Xk9T6Jj0x6pERa13VQkKCkKtVjN37lyKi4tp1KgRffv2ZcOGDUrrYv/+/cnJyWHatGloNBqsrKwIDAxk8+bNShhfX198fX2VeJs3b87kyZP5888/le5S8P/H9j6IX6g0Gg0GBgacO3cOjUbDxIkTtX4vKiqiVatWWts8PDyUv9va2gJw9epV7Ozs8Pf3x9TUlAMHDhAYGMjBgwfRaDQ88cQTAJw9e5asrCyGDh2qFWdhYSGZmZla8VbVXVnXpGwKfSbls/ZIXgrxaMvLy+PKlSta16KhoSHe3t5cvnxZ2aZWq1m3bh0qlYpr165RUlKCRqPh0qVLWvHd/hxV9pJ59epVANLT0/Hx8dGaKLBFixYUFRWhVqu19hWPFl3WRdVlYmKCRqPh1q1bevlBUVRNGiUqYWFhwfXr12u0j4GBAYMHD2bgwIHk5ORgaWnJ8ePHgdJxUlB60YwdO5ZRo0Zx9epVbGxs2LlzJ40aNcLS0rLCeA0NDXnsscfKLcNTlr7K9tNnaWlpODg4KI0Ts2fPLjdm8c4bYEU3GY1GA4CxsTFPPvkksbGxBAYGEhMTQ8eOHTE1NVXCeXp6KuPPbte4cWPl7/o2f0RFpGwKfSbls/ZIXgohqmPOnDnY2toycuRIbG1tMTIy4oMPPig3fOP256iyl76y56i70bcJv0X90mVdZG1trTSclbl69Wq5D4jXr1+nQYMGD8RzvKiYDN+ohKenJ+np6fe0r6GhIba2thgbGxMXF4evr2+5BzZjY2Ps7OwwNDQkLi6O9u3bV7qMjUaj4fz58+UuwNTU1Afiy/6dUlJSOHr0KJ07d8bT0xONRkNOTg5OTk5af8p6Q1RXt27dOH78OGlpaRw5ckQZugHQrFkz1Go1FhYW5Y5ze6PEg0DKptBnUj5rj+SlEI82MzMzbGxstMbPazQarWEZubm5pKen88orr9CmTRvc3Ny4efNmjbuxu7q6cvr0aa1JNBMTEzE2NlZeIsWjSZd1ka+vb7mJNI8dO6bVewhK3y28vLzuKY1CP0hPiUr4+/uzevVqcnNzsbCwAEq7x+Xn53PlyhWKioqUcXhubm4YGxtz7do1/v77b/z8/CgqKiI6Opq//vqLsLAwJd6MjAxUKhU+Pj7cuHGDrVu3kpqayrhx45QwGzZswMfHB2dnZ27evMkff/xBSkoKI0eO1EpjQkJCuclf9M2tW7fIycnRWhJ048aNeHl5ERwcTMOGDenatSuLFy9m6NChNGvWjOvXr3Py5EkcHR3p1KlTtY/VvHlzmjRpwtdff42lpSWtW7dWfuvWrRtbtmxh7ty5DBgwAHt7ey5dusTBgwd59tlnlRU4HgRSNoU+k/JZeyQvhRDPPfccv/32Gy4uLri7u7N9+3ZycnKU4Rfm5uZYWFgQFRWFvb092dnZrFy5ssZd2Hv37s0ff/zBDz/8QN++fcnKymL16tX06dNH6XUqHk26rIv69u3L9OnT+e2333jiiSc4cOAAJ0+e5NNPP9VKY2JiotRFDzhplKiEu7s73t7eWkudfffdd8THxythPvroIwAWLVqEg4MDAHv27GHlypVAaeteaGgo3t7eyj4lJSVs3bqVjIwMjIyMaNmyJTNnzlT2B7hx4wZLliwhJycHMzMzmjVrRlhYmFY8hYWFHDhwgKlTp9ZdJtSC48ePM2rUKAwNDTE3N6dp06a89tpr9OzZUxmuMXbsWH799VdWrVrF5cuXady4Md7e3uXmlKiOrl278ssvv/D8889rffEzNTUlLCyMNWvWsGDBAvLy8rCxsaFly5aYm5vX2vnWBymbQp9J+aw9kpdCiODgYHJycvjuu++A0om9u3btqny5NjQ0ZOLEiSxbtowPP/wQJycnhgwZwhdffFGj49ja2vLJJ5+watUqPvroI8zNzXnqqad44403av2cxINFl3VR8+bNef/994mMjGTdunU4OTnx/vvv4+Pjo4TJzs4mKSmJCRMm1F0miDpnoKnOYLJH1JEjR1i2bBlffvllpV1adWXbtm0cPHiQ//znP7pOitABKZtCn0n5rD2Sl0IIIXRNn+uilStXkpeXx+jRo3WdFHEfjEJDQ0N1nQh95eTkhEajwcbGRu++picnJ9OrVy+lG5V4tEjZFPpMymftkbwUQgiha/pcF50/f57nn39eJrl8wElPCSGEEEIIIYQQQuiEfvW/EUIIIYQQQgghxCNDGiWEEEIIIYQQQgihE9IoIfTayZMnCQkJ4dq1a7pOihBCCCGEEEKIWiaNEkIIIYQQQujQ9evXGTlyJGq1WtdJKWfBggVs2bJF18kQQjzEjHWdACGEEEIIIR5lGzdupF27djg5OQGwbNkykpKSSE1NxdramvDw8HL77Nu3j40bN3LhwgUsLS3p06cPL774olaYbdu2sX37drKysrC3t+fVV18lMDBQ+T01NZX169dz7tw5srKy6N+/PyEhIVpx9O/fn+nTpxMUFISZmVkdnL0Q4lEnjRKi3h05coQFCxawbNkyjIyMUKvVvPvuu/Ts2ZNRo0YBEBkZyenTp3n11VeB0uV+1q5dS0pKCm5ubowaNQovLy8lzqSkJNasWcOZM2cwNzcnICCAQYMGKZVnaGgobm5umJmZERUVhYGBAd27d2fw4MF6t96yEEIIIR4dBQUF7Nq1iylTpijbNBoNgYGBpKSkcOzYsXL7HD58mG+++YYRI0bg7+9Peno6ERERmJiY0KdPHwB27NjB6tWrGT16ND4+PqhUKiIiIpTnpLJjN2nShE6dOhEZGVlh+tzd3XF0dGTv3r1K3EIIUZvkbUzUuxYtWnDr1i3OnDkDlM4bYWFhQXx8vBLm5MmT+Pn5Kf9es2YNAwcOZM6cOVhYWLBw4ULKVrNNSUlh5syZBAQEMG/ePCZNmkRycjLffvut1nFjYmIwMjJixowZvPnmm/zxxx/s27evHs5YCCGEEKJihw8fBqB58+bKtjfffJPnnnsOZ2fnCvfZu3cvHTp0oHfv3jg6OtK+fXtefvllNm3apDwf7d27l6CgILp27YqjoyNPPfUUPXv2ZNOmTUo83t7eDB06lK5du2JqalppGgMCAoiLi6uN0xVCiHKkUULUu4YNG+Ll5cXJkyeB0gaIPn36cPHiRa5cuUJBQQFnzpyhZcuWyj4DBgygVatWuLq60q9fP9LT08nOzgZg8+bNdOnSheDgYJydnfHx8WHkyJHs37+fq1evKnG4ubkxYMAAXFxc6NKlCy1btuTEiRP1e/JCCCGEELdJSEjAy8sLAwODau9z69YtGjRooLXNxMSEy5cvc/HiRSWMiYlJuTAqlYqioqIapdHb2xuVSkVhYWGN9hNCiOqQ4RtCJ/z8/IiPj+eVV14hISGBvn37cvLkSU6ePImlpSVGRkZ4e3uTlJQEgIeHh7Kvra0tAFevXsXOzo6zZ8+iVqsr7PWQmZmJlZVVuTgAbGxstBothBBCCCHq28WLF7GxsanRPv7+/ixfvpyjR4/SunVr1Go1W7duBSAnJwcHBwfatm1LdHQ0HTt25LHHHuPs2bNERUVRXFxMbm5ujY5pY2NDcXEx2dnZyrwXQghRW6RRQuhEy5Yt2bZtG2lpaeTl5eHl5YWfnx8nT57EysoKX19fjI3/f/E0MjIqF0dZ90SNRkOPHj144YUXyoUpa8CoKA4DAwMlDiGEEEIIXaioR0NVgoKCUKvVzJ07l+LiYho1akTfvn3ZsGGD0uOif//+5OTkMG3aNDQaDVZWVgQGBrJ58+Ya9coAlPRJTwkhRF2QRgmhEy1atKCoqIjNmzfTokULDA0NadmyJREREVhZWeHv71/tuJo1a0ZaWpq03AshhBDigWNhYcH169drtI+BgQGDBw9m4MCB5OTkYGlpyfHjxwFwdHQEShsSxo4dy6hRo7h69So2Njbs3LmTRo0aYWlpWaPjlaWvpvsJIUR1yJwSQifK5pWIiYlR5o7w8fHh8uXLnD59Wms+iaq89NJLqFQqlixZwrlz51Cr1Rw6dIglS5bUVfKFEEIIIWqFp6cn6enp97SvoaEhtra2GBsbExcXh6+vb7mGA2NjY+zs7DA0NCQuLo727dvXeOWx1NRUbG1tsba2vqd0CiHE3UhPCaEzfn5+Wg0QJiYm+Pj4cObMGby9vasdj4eHB2FhYURGRhIaGkpJSQkODg507NixrpIuhBBCCFEr/P39Wb16Nbm5uVhYWACgVqvJz8/nypUrFBUVkZycDJRO2m1sbMy1a9f4+++/8fPzo6ioiOjoaP766y/CwsKUeDMyMlCpVPj4+HDjxg22bt1Kamoq48aNU8IUFRWRlpYGlA7NyMnJITk5mYYNG2r1QE1ISKBt27b1kBtCiEeRgUYG1QshhBBCCKEzU6dOpVu3bvTp0weA0NBQraXSyyxatAgHBweuXbvGnDlzSElJAcDX15fXX38dHx8fJWxaWhrffPMNGRkZGBkZ0bJlSwYPHoyLi4sSJisri/Hjx5c7jp+fH6GhoUBpY8XIkSOZOnUqvr6+tXnaQggBSKOEEEIIIYQQOnXkyBGWLVvGl19+WeOhFXVt27ZtHDx4kP/85z+6TooQ4iGlX3c9IYQQQgghHjH+/v707t2by5cv6zop5RgbG/Pmm2/qOhlCiIeY9JQQQgghhBBCCCGETkhPCSGEEEIIIYQQQuiENEoIIYQQQgghhBBCJ6RRQtSLb775hsmTJ1NUVKS1/fjx47zxxhskJSXdV/zr168nPDz8vuIQQgghhBBCCFG/pFFC1Iu33nqL69evs2HDBmVbXl4e3377LcHBwTRv3rxOj39nY4gQQgghhBBCCN0z1nUCxKPB3NycMWPGMHv2bJ544gm8vb356aefMDc3p0uXLsyYMYPExERMTEwICAhgxIgRmJmZARAeHk5ubi4ff/yxEt/69evZv38/X3zxRYXHCw0NxdXVFVNTU/bs2YODgwOzZ88mLS2NlStXkpCQgImJCa1atWL48OFYW1vXSz4IIYQQQgghhPj/pKeEqDdt2rTh2WefJTw8nL///pvY2FjGjRvH7NmzMTU1Zfbs2UyePJlTp06xePHi+z5eTEwMAJ9++injxo3jypUrTJ8+naZNmzJr1iymTZtGfn4+c+fOpaSk5L6PJ4QQQgghhBCiZqRRQtSrwYMHo9Fo+PLLLxkwYAAqlYr8/HwmTJiAu7s7fn5+jBo1igMHDqBWq6sdb0hICOPGjdPa5uDgwNChQ3F1dcXNzY0dO3bg4eHB4MGDcXNzw8PDg/Hjx6NSqTh79mxtn6oQQgghhBBCiCrI8A1Rr0xMTAgODmbZsmW88MILrFy5Eg8PDxo1aqSEad68OQYGBqSlpeHk5HTPx/Ly8tL699mzZ0lISGDIkCHlwqrVary9ve/5WEIIIYQQQgghak4aJUS9MzIywsDAAEPD6nXUMTAwQKPRaG0rLi6ucj9TU1Otf2s0Gtq1a8fQoUPLhbWysqpWWoQQQgghhBBC1B5plBA65erqSnR0NDdv3lR6SyQlJaHRaHBzcwPA0tKS8+fPa+2XnJxc42M1a9aMv/76C3t7e4yNpegLIYQQQgghhK7JnBJCp7p164apqSmLFi0iJSWF+Ph4lixZQseOHZWhG61ateLcuXPs2rULtVrNpk2bSEpKqvGxevfuTV5eHl999RWnT58mMzOTY8eOERERwc2bN2v71IQQQgghhBBCVEE+FwudMjU1ZerUqSxfvpxPPvlEa0nQMv7+/vTv35/IyEgKCgro1q0bvXr14tChQzU6lq2tLTNmzGDNmjXMmjWLwsJC7O3tadu2LQ0aNKjtUxNCCCGEEEIIUQUDzZ2D9YUQQgghhBBCCCHqgQzfEEIIIYQQQgghhE5Io4QQQgghhBBCCCF0QholhBBCCCGEEEIIoRPSKCGEEEIIIYQQQgidkEYJIYQQQgghhBBC6IQ0SohH0tKlSwkNDdV1MoQQQgghhBDikWas6wSIR1N4eDh79uwBwMjICHNzc5o2bUqnTp3o2bMnxsZSNIUQQgghhBDiYSdvfkJnWrduzYQJEygpKeHatWucOHGCDRs2EBMTw7Rp02jYsKGukyiEEEIIIYQQog5Jo4TQmQYNGmBtbQ2Ara0tnp6etGnThilTprB582ZCQkIoKioiMjKS2NhYrl+/TtOmTRkwYAD+/v4AlJSUEBERwYkTJ8jJycHOzo6goCCCg4MxNDRUwqxatYro6GgAAgMDKSkp0c1JCyGEEEIIIYRQSKOE0Cvu7u74+/uzf/9+QkJCWLx4MZmZmbz77rvY2dlx+PBh5syZw+zZs/H09KSkpARbW1smTpyIpaUlKpWKJUuWYGFhQY8ePQDYsmULUVFRjB49Gg8PD7Zv305sbCzNmjXT8dkKIYQQQgghxKNNJroUesfNzY3MzEzUajVxcXFMnDgRPz8/HB0d6dOnD+3atWPnzp0AGBsbM2DAALy9vXFwcKBLly48++yzxMXFKfH98ccfvPTSS3Tp0gVXV1eGDx+u9NAQQgghhBBCCKE70lNC6B2NRoOBgQHnzp1Do9EwceJErd+Liopo1aqV8u8dO3awa9cuLl68SGFhIcXFxTRp0gSAvLw8rly5gq+vrxLe0NAQb29vLl++XD8nJIQQQgghhBCiQtIoIfROWloaDg4OSuPE7Nmzy63GYWJiAsC+ffv46aefGDJkCL6+vpiZmbFt2zb++ecfXSRdCCGEEEIIIUQNSKOE0CspKSkcPXqUV199FU9PTzQaDTk5OVo9I26XmJiIt7c3ffr0UbZlZmYqfzczM8PGxoZTp04pcWg0GlQqFTY2NnV7MkIIIYQQQggh7koaJYTO3Lp1i5ycHK0lQTdu3IiXlxfBwcE0bNiQrl27snjxYoYOHUqzZs24fv06J0+exNHRkU6dOuHs7Mzu3bs5fPgwTk5OxMXFER8fT+PGjZXjPPfcc/z222+4uLjg7u7O9u3bycnJkUYJIYQQQgghhNAxA41Go9F1IsSjJzw8nD179gClczyYm5vTtGlTOnfuTM+ePZXhGkVFRfz666/s3buXy5cv07hxY7y9vXnttdfw8vKiqKiI77//ngMHDqDRaOjUqRNNmjQhOjqa8PBwAIqLi1m5ciW7d+8GoHv37hQXF5Oenk5oaKguTl8IIYT4f+3dX0hTfRzH8Y9rLjRczsJlhGhtE+rCZd4V7GKBq4uIEILoJsEiu+gquygpoasuCioiu/GivJIQa5RFZQmrCMH+QBIsKIdzJNZKpZrbznMRHZ49M+p5Htt5eHq/YDdnv+9333M4N/vyPecHAABEUwIAAAAAAFiELUEBAAAAAIAlaEoAAAAAAABL0JQAAAAAAACWoCkBAAAAAAAsQVMCBTMzM6PW1lYlEgmrS8lz6tQpXbt2zeoyAAAAAOC3Yre6APw++vr6tH79eq1YsUKS1N3drZcvXyoWi6m8vNzcwvPPHjx4oL6+Pk1MTMjpdCoUCmnbtm05awYGBnTz5k29fftWy5cv144dOxQIBMzv7927p/Pnz+flvnz5shwOhySpublZx44dUzAYVGlp6UKeNgAAAADgO2hKoCC+fPmiu3fv6vDhw+YxwzAUCAQ0NjamZ8+e5cWMjIzozJkz2rNnj/x+v8bHx9XV1SWHw6FQKCRJunXrlnp6erRv3z55vV5Fo1F1dXVpyZIlamxsNHMtXrxYZ8+ezcn/rSEhSdXV1XK73RoaGjJzAwAAAAB+LR7fQEGMjIxIkurq6sxjLS0t2rJli6qqquaNGRoa0oYNG9TU1CS3262GhgZt375d/f39MgzDXBMMBrVp0ya53W5t3LhRmzdvVn9/f16+8vLynM9fNTY2KhKJLMTpAgAAAAB+ApMSKIjR0VGtXr1aRUVFPx0zNzen4uLinGMOh0NTU1OanJxUZWWl5ubmciYevq2JRqNKp9Oy27/e4qlUSm1tbcpms6qpqdHOnTtVW1ubE+fxeHTlyhWlUqm8nAAAAACAhcekBApicnJSLpfrb8X4/X4NDw/r6dOnymazisfjCofDkqRkMilJqq+v1+DgoKLRqAzD0KtXr3Tnzh1lMhlNT09LklauXKn9+/ervb1dBw8eVHFxsTo6OjQxMZHzey6XS5lMRu/evVuAMwYAAAAA/AiTEiiI+SYafiQYDCqRSOjkyZPKZDIqKSnR1q1b1dvba05cNDc3K5lMqqOjQ4ZhaOnSpQoEArp69aq5xufzyefzmXnr6up06NAh3bhxQy0tLebxb/WlUql/e7oAAAAAgJ9AUwIFUVZWppmZmb8VU1RUpN27d2vXrl1KJpNyOp16/vy5JMntdkv62khoa2vT3r179eHDB7lcLt2+fVslJSVyOp3z5rXZbFqzZk3e1qTf6vteHAAAAABgYfH4BgqipqZG4+Pj/yjWZrOpoqJCdrtdkUhEPp8vr3Fgt9u1bNky2Ww2RSIRNTQ0yGab//Y2DENv3rzJe9llLBZTRUXFvC/BBAAAAAAsPCYlUBB+v189PT2anp5WWVmZJCmRSOjz5896//690um0Xr9+LUlatWqV7Ha7Pn78qEePHmnt2rVKp9MaHBzUw4cP1dnZaeaNx+OKRqPyer2anZ1VOBxWLBbTgQMHzDW9vb3yer2qqqrSp0+fdP36dY2Njam1tTWnxtHRUdXX1//6iwEAAAAAkERTAgVSXV0tj8ejSCSiUCgkSbpw4YJevHhhrmlvb5cknTt3TpWVlZKk+/fv69KlS5K+vhvi+PHj8ng8Zkw2m1U4HFY8HteiRYu0bt06nThxwoyXpNnZWV28eFHJZFKlpaWqra1VZ2dnTp5UKqXHjx/ryJEjv+4iAAAAAAByFBmGYVhdBH4PT548UXd3t06fPv3dRyusMjAwoOHhYR09etTqUgAAAADgt/Hf+meI/zW/36+mpiZNTU1ZXUoeu92esxMHAAAAAODXY1ICAAAAAABYgkkJAAAAAABgCZoSAAAAAADAEjQlAAAAAACAJWhKAAAAAAAAS9CUAAAAAAAAlqApAQAAAAAALPEHH8VjFbSZjLsAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "metadata": { "id": "6zcqiJUEwwdQ" }, "source": [ "Note that cosine annealing scheduler is a bit different from other schedules as soon as it starts with `base_lr` and gradually decreases it to the minimal value while triangle schedulers increase the original rate." ] }, { "cell_type": "markdown", "metadata": { "id": "27Y_3GcPgTfS" }, "source": [ "## MF with PyTorch on ML-100k" ] }, { "cell_type": "markdown", "metadata": { "id": "2jqJNaQhK8ji" }, "source": [ "### Utils" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "vsby4vwWGlWJ", "outputId": "52308560-91b9-4db7-daea-3dc502d34bb5" }, "source": [ "%%writefile utils.py\n", "\n", "import os\n", "import requests\n", "import zipfile\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import scipy.sparse as sp\n", "\n", "\"\"\"\n", "Shamelessly stolen from\n", "https://github.com/maciejkula/triplet_recommendations_keras\n", "\"\"\"\n", "\n", "\n", "def train_test_split(interactions, n=10):\n", " \"\"\"\n", " Split an interactions matrix into training and test sets.\n", " Parameters\n", " ----------\n", " interactions : np.ndarray\n", " n : int (default=10)\n", " Number of items to select / row to place into test.\n", "\n", " Returns\n", " -------\n", " train : np.ndarray\n", " test : np.ndarray\n", " \"\"\"\n", " test = np.zeros(interactions.shape)\n", " train = interactions.copy()\n", " for user in range(interactions.shape[0]):\n", " if interactions[user, :].nonzero()[0].shape[0] > n:\n", " test_interactions = np.random.choice(interactions[user, :].nonzero()[0],\n", " size=n,\n", " replace=False)\n", " train[user, test_interactions] = 0.\n", " test[user, test_interactions] = interactions[user, test_interactions]\n", "\n", " # Test and training are truly disjoint\n", " assert(np.all((train * test) == 0))\n", " return train, test\n", "\n", "\n", "def _get_data_path():\n", " \"\"\"\n", " Get path to the movielens dataset file.\n", " \"\"\"\n", " data_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),\n", " 'data')\n", " if not os.path.exists(data_path):\n", " print('Making data path')\n", " os.mkdir(data_path)\n", " return data_path\n", "\n", "\n", "def _download_movielens(dest_path):\n", " \"\"\"\n", " Download the dataset.\n", " \"\"\"\n", "\n", " url = 'http://files.grouplens.org/datasets/movielens/ml-100k.zip'\n", " req = requests.get(url, stream=True)\n", "\n", " print('Downloading MovieLens data')\n", "\n", " with open(os.path.join(dest_path, 'ml-100k.zip'), 'wb') as fd:\n", " for chunk in req.iter_content(chunk_size=None):\n", " fd.write(chunk)\n", "\n", " with zipfile.ZipFile(os.path.join(dest_path, 'ml-100k.zip'), 'r') as z:\n", " z.extractall(dest_path)\n", "\n", "\n", "def read_movielens_df():\n", " path = _get_data_path()\n", " zipfile = os.path.join(path, 'ml-100k.zip')\n", " if not os.path.isfile(zipfile):\n", " _download_movielens(path)\n", " fname = os.path.join(path, 'ml-100k', 'u.data')\n", " names = ['user_id', 'item_id', 'rating', 'timestamp']\n", " df = pd.read_csv(fname, sep='\\t', names=names)\n", " return df\n", "\n", "\n", "def get_movielens_interactions():\n", " df = read_movielens_df()\n", "\n", " n_users = df.user_id.unique().shape[0]\n", " n_items = df.item_id.unique().shape[0]\n", "\n", " interactions = np.zeros((n_users, n_items))\n", " for row in df.itertuples():\n", " interactions[row[1] - 1, row[2] - 1] = row[3]\n", " return interactions\n", "\n", "\n", "def get_movielens_train_test_split(implicit=False):\n", " interactions = get_movielens_interactions()\n", " if implicit:\n", " interactions = (interactions >= 4).astype(np.float32)\n", " train, test = train_test_split(interactions)\n", " train = sp.coo_matrix(train)\n", " test = sp.coo_matrix(test)\n", " return train, test" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing utils.py\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "d9wghzkqLR2i" }, "source": [ "### Metrics" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "EWmOo0AqKiEN", "outputId": "5b4b9d36-218e-42ed-da99-f5af0b4b93f9" }, "source": [ "%%writefile metrics.py\n", "\n", "import numpy as np\n", "from sklearn.metrics import roc_auc_score\n", "from torch import multiprocessing as mp\n", "import torch\n", "\n", "\n", "def get_row_indices(row, interactions):\n", " start = interactions.indptr[row]\n", " end = interactions.indptr[row + 1]\n", " return interactions.indices[start:end]\n", "\n", "\n", "def auc(model, interactions, num_workers=1):\n", " aucs = []\n", " processes = []\n", " n_users = interactions.shape[0]\n", " mp_batch = int(np.ceil(n_users / num_workers))\n", "\n", " queue = mp.Queue()\n", " rows = np.arange(n_users)\n", " np.random.shuffle(rows)\n", " for rank in range(num_workers):\n", " start = rank * mp_batch\n", " end = np.min((start + mp_batch, n_users))\n", " p = mp.Process(target=batch_auc,\n", " args=(queue, rows[start:end], interactions, model))\n", " p.start()\n", " processes.append(p)\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " aucs.append(queue.get())\n", "\n", " queue.close()\n", " for p in processes:\n", " p.join()\n", " return np.mean(aucs)\n", "\n", "\n", "def batch_auc(queue, rows, interactions, model):\n", " n_items = interactions.shape[1]\n", " items = torch.arange(0, n_items).long()\n", " users_init = torch.ones(n_items).long()\n", " for row in rows:\n", " row = int(row)\n", " users = users_init.fill_(row)\n", "\n", " preds = model.predict(users, items)\n", " actuals = get_row_indices(row, interactions)\n", "\n", " if len(actuals) == 0:\n", " continue\n", " y_test = np.zeros(n_items)\n", " y_test[actuals] = 1\n", " queue.put(roc_auc_score(y_test, preds.data.numpy()))\n", "\n", "\n", "def patk(model, interactions, num_workers=1, k=5):\n", " patks = []\n", " processes = []\n", " n_users = interactions.shape[0]\n", " mp_batch = int(np.ceil(n_users / num_workers))\n", "\n", " queue = mp.Queue()\n", " rows = np.arange(n_users)\n", " np.random.shuffle(rows)\n", " for rank in range(num_workers):\n", " start = rank * mp_batch\n", " end = np.min((start + mp_batch, n_users))\n", " p = mp.Process(target=batch_patk,\n", " args=(queue, rows[start:end], interactions, model),\n", " kwargs={'k': k})\n", " p.start()\n", " processes.append(p)\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " patks.append(queue.get())\n", "\n", " queue.close()\n", " for p in processes:\n", " p.join()\n", " return np.mean(patks)\n", "\n", "\n", "def batch_patk(queue, rows, interactions, model, k=5):\n", " n_items = interactions.shape[1]\n", "\n", " items = torch.arange(0, n_items).long()\n", " users_init = torch.ones(n_items).long()\n", " for row in rows:\n", " row = int(row)\n", " users = users_init.fill_(row)\n", "\n", " preds = model.predict(users, items)\n", " actuals = get_row_indices(row, interactions)\n", "\n", " if len(actuals) == 0:\n", " continue\n", "\n", " top_k = np.argpartition(-np.squeeze(preds.data.numpy()), k)\n", " top_k = set(top_k[:k])\n", " true_pids = set(actuals)\n", " if true_pids:\n", " queue.put(len(top_k & true_pids) / float(k))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing metrics.py\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "7N1Rl15-LJAj" }, "source": [ "### Model" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "NUlsa6LeLKqu", "outputId": "6326b89c-3659-4008-8801-f8d3b991b6fc" }, "source": [ "%%writefile torchmf.py\n", "\n", "import collections\n", "import os\n", "\n", "import numpy as np\n", "from sklearn.metrics import roc_auc_score\n", "import torch\n", "from torch import nn\n", "import torch.multiprocessing as mp\n", "import torch.utils.data as data\n", "from tqdm import tqdm\n", "\n", "import metrics\n", "\n", "\n", "# Models\n", "# Interactions Dataset => Singular Iter => Singular Loss\n", "# Pairwise Datasets => Pairwise Iter => Pairwise Loss\n", "# Pairwise Iters\n", "# Loss Functions\n", "# Optimizers\n", "# Metric callbacks\n", "\n", "# Serve up users, items (and items could be pos_items, neg_items)\n", "# In this case, the iteration remains the same. Pass both items into a model\n", "# which is a concat of the base model. it handles the pos and neg_items\n", "# accordingly. define the loss after.\n", "\n", "\n", "class Interactions(data.Dataset):\n", " \"\"\"\n", " Hold data in the form of an interactions matrix.\n", " Typical use-case is like a ratings matrix:\n", " - Users are the rows\n", " - Items are the columns\n", " - Elements of the matrix are the ratings given by a user for an item.\n", " \"\"\"\n", "\n", " def __init__(self, mat):\n", " self.mat = mat.astype(np.float32).tocoo()\n", " self.n_users = self.mat.shape[0]\n", " self.n_items = self.mat.shape[1]\n", "\n", " def __getitem__(self, index):\n", " row = self.mat.row[index]\n", " col = self.mat.col[index]\n", " val = self.mat.data[index]\n", " return (row, col), val\n", "\n", " def __len__(self):\n", " return self.mat.nnz\n", "\n", "\n", "class PairwiseInteractions(data.Dataset):\n", " \"\"\"\n", " Sample data from an interactions matrix in a pairwise fashion. The row is\n", " treated as the main dimension, and the columns are sampled pairwise.\n", " \"\"\"\n", "\n", " def __init__(self, mat):\n", " self.mat = mat.astype(np.float32).tocoo()\n", "\n", " self.n_users = self.mat.shape[0]\n", " self.n_items = self.mat.shape[1]\n", "\n", " self.mat_csr = self.mat.tocsr()\n", " if not self.mat_csr.has_sorted_indices:\n", " self.mat_csr.sort_indices()\n", "\n", " def __getitem__(self, index):\n", " row = self.mat.row[index]\n", " found = False\n", "\n", " while not found:\n", " neg_col = np.random.randint(self.n_items)\n", " if self.not_rated(row, neg_col, self.mat_csr.indptr,\n", " self.mat_csr.indices):\n", " found = True\n", "\n", " pos_col = self.mat.col[index]\n", " val = self.mat.data[index]\n", "\n", " return (row, (pos_col, neg_col)), val\n", "\n", " def __len__(self):\n", " return self.mat.nnz\n", "\n", " @staticmethod\n", " def not_rated(row, col, indptr, indices):\n", " # similar to use of bsearch in lightfm\n", " start = indptr[row]\n", " end = indptr[row + 1]\n", " searched = np.searchsorted(indices[start:end], col, 'right')\n", " if searched >= (end - start):\n", " # After the array\n", " return False\n", " return col != indices[searched] # Not found\n", "\n", " def get_row_indices(self, row):\n", " start = self.mat_csr.indptr[row]\n", " end = self.mat_csr.indptr[row + 1]\n", " return self.mat_csr.indices[start:end]\n", "\n", "\n", "class BaseModule(nn.Module):\n", " \"\"\"\n", " Base module for explicit matrix factorization.\n", " \"\"\"\n", " \n", " def __init__(self,\n", " n_users,\n", " n_items,\n", " n_factors=40,\n", " dropout_p=0,\n", " sparse=False):\n", " \"\"\"\n", "\n", " Parameters\n", " ----------\n", " n_users : int\n", " Number of users\n", " n_items : int\n", " Number of items\n", " n_factors : int\n", " Number of latent factors (or embeddings or whatever you want to\n", " call it).\n", " dropout_p : float\n", " p in nn.Dropout module. Probability of dropout.\n", " sparse : bool\n", " Whether or not to treat embeddings as sparse. NOTE: cannot use\n", " weight decay on the optimizer if sparse=True. Also, can only use\n", " Adagrad.\n", " \"\"\"\n", " super(BaseModule, self).__init__()\n", " self.n_users = n_users\n", " self.n_items = n_items\n", " self.n_factors = n_factors\n", " self.user_biases = nn.Embedding(n_users, 1, sparse=sparse)\n", " self.item_biases = nn.Embedding(n_items, 1, sparse=sparse)\n", " self.user_embeddings = nn.Embedding(n_users, n_factors, sparse=sparse)\n", " self.item_embeddings = nn.Embedding(n_items, n_factors, sparse=sparse)\n", " \n", " self.dropout_p = dropout_p\n", " self.dropout = nn.Dropout(p=self.dropout_p)\n", "\n", " self.sparse = sparse\n", " \n", " def forward(self, users, items):\n", " \"\"\"\n", " Forward pass through the model. For a single user and item, this\n", " looks like:\n", "\n", " user_bias + item_bias + user_embeddings.dot(item_embeddings)\n", "\n", " Parameters\n", " ----------\n", " users : np.ndarray\n", " Array of user indices\n", " items : np.ndarray\n", " Array of item indices\n", "\n", " Returns\n", " -------\n", " preds : np.ndarray\n", " Predicted ratings.\n", "\n", " \"\"\"\n", " ues = self.user_embeddings(users)\n", " uis = self.item_embeddings(items)\n", "\n", " preds = self.user_biases(users)\n", " preds += self.item_biases(items)\n", " preds += (self.dropout(ues) * self.dropout(uis)).sum(dim=1, keepdim=True)\n", "\n", " return preds.squeeze()\n", " \n", " def __call__(self, *args):\n", " return self.forward(*args)\n", "\n", " def predict(self, users, items):\n", " return self.forward(users, items)\n", "\n", "\n", "def bpr_loss(preds, vals):\n", " sig = nn.Sigmoid()\n", " return (1.0 - sig(preds)).pow(2).sum()\n", "\n", "\n", "class BPRModule(nn.Module):\n", " \n", " def __init__(self,\n", " n_users,\n", " n_items,\n", " n_factors=40,\n", " dropout_p=0,\n", " sparse=False,\n", " model=BaseModule):\n", " super(BPRModule, self).__init__()\n", "\n", " self.n_users = n_users\n", " self.n_items = n_items\n", " self.n_factors = n_factors\n", " self.dropout_p = dropout_p\n", " self.sparse = sparse\n", " self.pred_model = model(\n", " self.n_users,\n", " self.n_items,\n", " n_factors=n_factors,\n", " dropout_p=dropout_p,\n", " sparse=sparse\n", " )\n", "\n", " def forward(self, users, items):\n", " assert isinstance(items, tuple), \\\n", " 'Must pass in items as (pos_items, neg_items)'\n", " # Unpack\n", " (pos_items, neg_items) = items\n", " pos_preds = self.pred_model(users, pos_items)\n", " neg_preds = self.pred_model(users, neg_items)\n", " return pos_preds - neg_preds\n", "\n", " def predict(self, users, items):\n", " return self.pred_model(users, items)\n", "\n", "\n", "class BasePipeline:\n", " \"\"\"\n", " Class defining a training pipeline. Instantiates data loaders, model,\n", " and optimizer. Handles training for multiple epochs and keeping track of\n", " train and test loss.\n", " \"\"\"\n", "\n", " def __init__(self,\n", " train,\n", " test=None,\n", " model=BaseModule,\n", " n_factors=40,\n", " batch_size=32,\n", " dropout_p=0.02,\n", " sparse=False,\n", " lr=0.01,\n", " weight_decay=0.,\n", " optimizer=torch.optim.Adam,\n", " loss_function=nn.MSELoss(reduction='sum'),\n", " n_epochs=10,\n", " verbose=False,\n", " random_seed=None,\n", " interaction_class=Interactions,\n", " hogwild=False,\n", " num_workers=0,\n", " eval_metrics=None,\n", " k=5):\n", " self.train = train\n", " self.test = test\n", "\n", " if hogwild:\n", " num_loader_workers = 0\n", " else:\n", " num_loader_workers = num_workers\n", " self.train_loader = data.DataLoader(\n", " interaction_class(train), batch_size=batch_size, shuffle=True,\n", " num_workers=num_loader_workers)\n", " if self.test is not None:\n", " self.test_loader = data.DataLoader(\n", " interaction_class(test), batch_size=batch_size, shuffle=True,\n", " num_workers=num_loader_workers)\n", " self.num_workers = num_workers\n", " self.n_users = self.train.shape[0]\n", " self.n_items = self.train.shape[1]\n", " self.n_factors = n_factors\n", " self.batch_size = batch_size\n", " self.dropout_p = dropout_p\n", " self.lr = lr\n", " self.weight_decay = weight_decay\n", " self.loss_function = loss_function\n", " self.n_epochs = n_epochs\n", " if sparse:\n", " assert weight_decay == 0.0\n", " self.model = model(self.n_users,\n", " self.n_items,\n", " n_factors=self.n_factors,\n", " dropout_p=self.dropout_p,\n", " sparse=sparse)\n", " self.optimizer = optimizer(self.model.parameters(),\n", " lr=self.lr,\n", " weight_decay=self.weight_decay)\n", " self.warm_start = False\n", " self.losses = collections.defaultdict(list)\n", " self.verbose = verbose\n", " self.hogwild = hogwild\n", " if random_seed is not None:\n", " if self.hogwild:\n", " random_seed += os.getpid()\n", " torch.manual_seed(random_seed)\n", " np.random.seed(random_seed)\n", "\n", " if eval_metrics is None:\n", " eval_metrics = []\n", " self.eval_metrics = eval_metrics\n", " self.k = k\n", "\n", " def break_grads(self):\n", " for param in self.model.parameters():\n", " # Break gradient sharing\n", " if param.grad is not None:\n", " param.grad.data = param.grad.data.clone()\n", "\n", " def fit(self):\n", " for epoch in range(1, self.n_epochs + 1):\n", "\n", " if self.hogwild:\n", " self.model.share_memory()\n", " processes = []\n", " train_losses = []\n", " queue = mp.Queue()\n", " for rank in range(self.num_workers):\n", " p = mp.Process(target=self._fit_epoch,\n", " kwargs={'epoch': epoch,\n", " 'queue': queue})\n", " p.start()\n", " processes.append(p)\n", " for p in processes:\n", " p.join()\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " train_losses.append(queue.get())\n", " queue.close()\n", " train_loss = np.mean(train_losses)\n", " else:\n", " train_loss = self._fit_epoch(epoch)\n", "\n", " self.losses['train'].append(train_loss)\n", " row = 'Epoch: {0:^3} train: {1:^10.5f}'.format(epoch, self.losses['train'][-1])\n", " if self.test is not None:\n", " self.losses['test'].append(self._validation_loss())\n", " row += 'val: {0:^10.5f}'.format(self.losses['test'][-1])\n", " for metric in self.eval_metrics:\n", " func = getattr(metrics, metric)\n", " res = func(self.model, self.test_loader.dataset.mat_csr,\n", " num_workers=self.num_workers)\n", " self.losses['eval-{}'.format(metric)].append(res)\n", " row += 'eval-{0}: {1:^10.5f}'.format(metric, res)\n", " self.losses['epoch'].append(epoch)\n", " if self.verbose:\n", " print(row)\n", "\n", " def _fit_epoch(self, epoch=1, queue=None):\n", " if self.hogwild:\n", " self.break_grads()\n", "\n", " self.model.train()\n", " total_loss = torch.Tensor([0])\n", " pbar = tqdm(enumerate(self.train_loader),\n", " total=len(self.train_loader),\n", " desc='({0:^3})'.format(epoch))\n", " for batch_idx, ((row, col), val) in pbar:\n", " self.optimizer.zero_grad()\n", "\n", " row = row.long()\n", " # TODO: turn this into a collate_fn like the data_loader\n", " if isinstance(col, list):\n", " col = tuple(c.long() for c in col)\n", " else:\n", " col = col.long()\n", " val = val.float()\n", "\n", " preds = self.model(row, col)\n", " loss = self.loss_function(preds, val)\n", " loss.backward()\n", "\n", " self.optimizer.step()\n", "\n", " total_loss += loss.item()\n", " batch_loss = loss.item() / row.size()[0]\n", " pbar.set_postfix(train_loss=batch_loss)\n", " total_loss /= self.train.nnz\n", " if queue is not None:\n", " queue.put(total_loss[0])\n", " else:\n", " return total_loss[0]\n", "\n", " def _validation_loss(self):\n", " self.model.eval()\n", " total_loss = torch.Tensor([0])\n", " for batch_idx, ((row, col), val) in enumerate(self.test_loader):\n", " row = row.long()\n", " if isinstance(col, list):\n", " col = tuple(c.long() for c in col)\n", " else:\n", " col = col.long()\n", " val = val.float()\n", "\n", " preds = self.model(row, col)\n", " loss = self.loss_function(preds, val)\n", " total_loss += loss.item()\n", "\n", " total_loss /= self.test.nnz\n", " return total_loss[0]" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing torchmf.py\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "5KpaDgMwLNI5" }, "source": [ "### Trainer" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "HxKI9a2dLDDy", "outputId": "538238ee-aa22-49e9-aa44-68eb30d5e92a" }, "source": [ "%%writefile run.py\n", "\n", "import argparse\n", "import pickle\n", "\n", "import torch\n", "\n", "from torchmf import (BaseModule, BPRModule, BasePipeline,\n", " bpr_loss, PairwiseInteractions)\n", "import utils\n", "\n", "\n", "def explicit():\n", " train, test = utils.get_movielens_train_test_split()\n", " pipeline = BasePipeline(train, test=test, model=BaseModule,\n", " n_factors=10, batch_size=1024, dropout_p=0.02,\n", " lr=0.02, weight_decay=0.1,\n", " optimizer=torch.optim.Adam, n_epochs=40,\n", " verbose=True, random_seed=2017)\n", " pipeline.fit()\n", "\n", "\n", "def implicit():\n", " train, test = utils.get_movielens_train_test_split(implicit=True)\n", "\n", " pipeline = BasePipeline(train, test=test, verbose=True,\n", " batch_size=1024, num_workers=4,\n", " n_factors=20, weight_decay=0,\n", " dropout_p=0., lr=.2, sparse=True,\n", " optimizer=torch.optim.SGD, n_epochs=40,\n", " random_seed=2017, loss_function=bpr_loss,\n", " model=BPRModule,\n", " interaction_class=PairwiseInteractions,\n", " eval_metrics=('auc', 'patk'))\n", " pipeline.fit()\n", "\n", "\n", "def hogwild():\n", " train, test = utils.get_movielens_train_test_split(implicit=True)\n", "\n", " pipeline = BasePipeline(train, test=test, verbose=True,\n", " batch_size=1024, num_workers=4,\n", " n_factors=20, weight_decay=0,\n", " dropout_p=0., lr=.2, sparse=True,\n", " optimizer=torch.optim.SGD, n_epochs=40,\n", " random_seed=2017, loss_function=bpr_loss,\n", " model=BPRModule, hogwild=True,\n", " interaction_class=PairwiseInteractions,\n", " eval_metrics=('auc', 'patk'))\n", " pipeline.fit()\n", "\n", "\n", "if __name__ == '__main__':\n", " parser = argparse.ArgumentParser(description='torchmf')\n", " parser.add_argument('--example',\n", " help='explicit, implicit, or hogwild')\n", " args = parser.parse_args()\n", " if args.example == 'explicit':\n", " explicit()\n", " elif args.example == 'implicit':\n", " implicit()\n", " elif args.example == 'hogwild':\n", " hogwild()\n", " else:\n", " print('example must be explicit, implicit, or hogwild')" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Writing run.py\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "40lNybzWLtRP" }, "source": [ "### Explicit Model Training" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0i4BoW9HLHSb", "outputId": "b2a2a2bc-c5c8-4c6e-f576-6a68326d5a30" }, "source": [ "!python run.py --example explicit" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Making data path\n", "Downloading MovieLens data\n", "( 1 ): 100% 89/89 [00:01<00:00, 74.87it/s, train_loss=7.64] \n", "Epoch: 1 train: 14.61587 val: 8.83048 \n", "( 2 ): 100% 89/89 [00:00<00:00, 112.92it/s, train_loss=2.5]\n", "Epoch: 2 train: 4.20514 val: 4.05539 \n", "( 3 ): 100% 89/89 [00:00<00:00, 112.48it/s, train_loss=1.57]\n", "Epoch: 3 train: 1.86044 val: 2.45105 \n", "( 4 ): 100% 89/89 [00:00<00:00, 113.76it/s, train_loss=1.15]\n", "Epoch: 4 train: 1.20612 val: 1.82121 \n", "( 5 ): 100% 89/89 [00:00<00:00, 110.97it/s, train_loss=0.966]\n", "Epoch: 5 train: 0.98724 val: 1.51758 \n", "( 6 ): 100% 89/89 [00:00<00:00, 112.02it/s, train_loss=0.89]\n", "Epoch: 6 train: 0.89150 val: 1.35180 \n", "( 7 ): 100% 89/89 [00:00<00:00, 112.91it/s, train_loss=0.906]\n", "Epoch: 7 train: 0.83810 val: 1.25295 \n", "( 8 ): 100% 89/89 [00:00<00:00, 108.44it/s, train_loss=0.873]\n", "Epoch: 8 train: 0.80769 val: 1.18821 \n", "( 9 ): 100% 89/89 [00:00<00:00, 113.67it/s, train_loss=0.83]\n", "Epoch: 9 train: 0.78222 val: 1.15017 \n", "(10 ): 100% 89/89 [00:00<00:00, 113.89it/s, train_loss=0.777]\n", "Epoch: 10 train: 0.76105 val: 1.11414 \n", "(11 ): 100% 89/89 [00:00<00:00, 113.23it/s, train_loss=0.73]\n", "Epoch: 11 train: 0.74182 val: 1.08541 \n", "(12 ): 100% 89/89 [00:00<00:00, 112.17it/s, train_loss=0.64]\n", "Epoch: 12 train: 0.72437 val: 1.06774 \n", "(13 ): 100% 89/89 [00:00<00:00, 107.08it/s, train_loss=0.733]\n", "Epoch: 13 train: 0.70896 val: 1.05505 \n", "(14 ): 100% 89/89 [00:00<00:00, 111.47it/s, train_loss=0.702]\n", "Epoch: 14 train: 0.69648 val: 1.03989 \n", "(15 ): 100% 89/89 [00:00<00:00, 114.82it/s, train_loss=0.69]\n", "Epoch: 15 train: 0.68401 val: 1.03105 \n", "(16 ): 100% 89/89 [00:00<00:00, 113.08it/s, train_loss=0.772]\n", "Epoch: 16 train: 0.67320 val: 1.02541 \n", "(17 ): 100% 89/89 [00:00<00:00, 112.42it/s, train_loss=0.624]\n", "Epoch: 17 train: 0.66667 val: 1.01918 \n", "(18 ): 100% 89/89 [00:00<00:00, 113.76it/s, train_loss=0.671]\n", "Epoch: 18 train: 0.65996 val: 1.01878 \n", "(19 ): 100% 89/89 [00:00<00:00, 113.38it/s, train_loss=0.667]\n", "Epoch: 19 train: 0.65364 val: 1.01307 \n", "(20 ): 100% 89/89 [00:00<00:00, 110.37it/s, train_loss=0.745]\n", "Epoch: 20 train: 0.64888 val: 1.01569 \n", "(21 ): 100% 89/89 [00:00<00:00, 113.61it/s, train_loss=0.671]\n", "Epoch: 21 train: 0.64512 val: 1.01603 \n", "(22 ): 100% 89/89 [00:00<00:00, 108.82it/s, train_loss=0.623]\n", "Epoch: 22 train: 0.64155 val: 1.01564 \n", "(23 ): 100% 89/89 [00:00<00:00, 112.19it/s, train_loss=0.677]\n", "Epoch: 23 train: 0.63771 val: 1.01452 \n", "(24 ): 100% 89/89 [00:00<00:00, 114.23it/s, train_loss=0.739]\n", "Epoch: 24 train: 0.63746 val: 1.00893 \n", "(25 ): 100% 89/89 [00:00<00:00, 114.42it/s, train_loss=0.766]\n", "Epoch: 25 train: 0.63591 val: 1.01990 \n", "(26 ): 100% 89/89 [00:00<00:00, 112.95it/s, train_loss=0.586]\n", "Epoch: 26 train: 0.63194 val: 1.01370 \n", "(27 ): 100% 89/89 [00:00<00:00, 111.70it/s, train_loss=0.734]\n", "Epoch: 27 train: 0.63205 val: 1.01533 \n", "(28 ): 100% 89/89 [00:00<00:00, 112.88it/s, train_loss=0.733]\n", "Epoch: 28 train: 0.63321 val: 1.01158 \n", "(29 ): 100% 89/89 [00:00<00:00, 107.37it/s, train_loss=0.645]\n", "Epoch: 29 train: 0.63266 val: 1.01819 \n", "(30 ): 100% 89/89 [00:00<00:00, 112.43it/s, train_loss=0.683]\n", "Epoch: 30 train: 0.63357 val: 1.01789 \n", "(31 ): 100% 89/89 [00:00<00:00, 109.35it/s, train_loss=0.7]\n", "Epoch: 31 train: 0.63155 val: 1.01247 \n", "(32 ): 100% 89/89 [00:00<00:00, 113.49it/s, train_loss=0.68]\n", "Epoch: 32 train: 0.63328 val: 1.01842 \n", "(33 ): 100% 89/89 [00:00<00:00, 112.89it/s, train_loss=0.68]\n", "Epoch: 33 train: 0.63136 val: 1.01667 \n", "(34 ): 100% 89/89 [00:00<00:00, 113.90it/s, train_loss=0.752]\n", "Epoch: 34 train: 0.63255 val: 1.01864 \n", "(35 ): 100% 89/89 [00:00<00:00, 113.94it/s, train_loss=0.716]\n", "Epoch: 35 train: 0.63282 val: 1.01362 \n", "(36 ): 100% 89/89 [00:00<00:00, 112.76it/s, train_loss=0.618]\n", "Epoch: 36 train: 0.63292 val: 1.01480 \n", "(37 ): 100% 89/89 [00:00<00:00, 113.63it/s, train_loss=0.666]\n", "Epoch: 37 train: 0.63206 val: 1.02341 \n", "(38 ): 100% 89/89 [00:00<00:00, 107.43it/s, train_loss=0.652]\n", "Epoch: 38 train: 0.63254 val: 1.02066 \n", "(39 ): 100% 89/89 [00:00<00:00, 112.98it/s, train_loss=0.65]\n", "Epoch: 39 train: 0.63397 val: 1.01905 \n", "(40 ): 100% 89/89 [00:00<00:00, 109.15it/s, train_loss=0.732]\n", "Epoch: 40 train: 0.63401 val: 1.01783 \n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "JxqWyDE4LxPb" }, "source": [ "### Implicit Model Training" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IBXdHOUXLdPE", "outputId": "62bd667d-56d6-4969-e390-8cab4842353e" }, "source": [ "!python run.py --example implicit" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n", "( 1 ): 100% 46/46 [00:01<00:00, 28.55it/s, train_loss=0.382]\n", "Epoch: 1 train: 0.41578 val: 0.39289 eval-auc: 0.55840 eval-patk: 0.00913 \n", "( 2 ): 100% 46/46 [00:01<00:00, 28.86it/s, train_loss=0.323]\n", "Epoch: 2 train: 0.34652 val: 0.34228 eval-auc: 0.61282 eval-patk: 0.01507 \n", "( 3 ): 100% 46/46 [00:01<00:00, 30.01it/s, train_loss=0.273]\n", "Epoch: 3 train: 0.27728 val: 0.31357 eval-auc: 0.65768 eval-patk: 0.02215 \n", "( 4 ): 100% 46/46 [00:01<00:00, 29.36it/s, train_loss=0.226]\n", "Epoch: 4 train: 0.23051 val: 0.29723 eval-auc: 0.69258 eval-patk: 0.02991 \n", "( 5 ): 100% 46/46 [00:01<00:00, 29.57it/s, train_loss=0.198]\n", "Epoch: 5 train: 0.20115 val: 0.28018 eval-auc: 0.71729 eval-patk: 0.03539 \n", "( 6 ): 100% 46/46 [00:01<00:00, 28.66it/s, train_loss=0.152]\n", "Epoch: 6 train: 0.17812 val: 0.26524 eval-auc: 0.73440 eval-patk: 0.03607 \n", "( 7 ): 100% 46/46 [00:01<00:00, 30.65it/s, train_loss=0.15]\n", "Epoch: 7 train: 0.16726 val: 0.25652 eval-auc: 0.74640 eval-patk: 0.03813 \n", "( 8 ): 100% 46/46 [00:01<00:00, 29.89it/s, train_loss=0.172]\n", "Epoch: 8 train: 0.15538 val: 0.24975 eval-auc: 0.75780 eval-patk: 0.03950 \n", "( 9 ): 100% 46/46 [00:01<00:00, 29.88it/s, train_loss=0.133]\n", "Epoch: 9 train: 0.14574 val: 0.24520 eval-auc: 0.76651 eval-patk: 0.04498 \n", "(10 ): 100% 46/46 [00:01<00:00, 30.02it/s, train_loss=0.14]\n", "Epoch: 10 train: 0.13953 val: 0.22739 eval-auc: 0.77529 eval-patk: 0.04749 \n", "(11 ): 100% 46/46 [00:01<00:00, 29.70it/s, train_loss=0.151]\n", "Epoch: 11 train: 0.13218 val: 0.22872 eval-auc: 0.78306 eval-patk: 0.04749 \n", "(12 ): 100% 46/46 [00:01<00:00, 29.81it/s, train_loss=0.13]\n", "Epoch: 12 train: 0.12857 val: 0.22756 eval-auc: 0.78880 eval-patk: 0.04840 \n", "(13 ): 100% 46/46 [00:01<00:00, 30.26it/s, train_loss=0.13]\n", "Epoch: 13 train: 0.12364 val: 0.21565 eval-auc: 0.79382 eval-patk: 0.05114 \n", "(14 ): 100% 46/46 [00:01<00:00, 30.80it/s, train_loss=0.0979]\n", "Epoch: 14 train: 0.11943 val: 0.21567 eval-auc: 0.79833 eval-patk: 0.05479 \n", "(15 ): 100% 46/46 [00:01<00:00, 30.27it/s, train_loss=0.109]\n", "Epoch: 15 train: 0.11619 val: 0.21074 eval-auc: 0.80249 eval-patk: 0.05548 \n", "(16 ): 100% 46/46 [00:01<00:00, 29.81it/s, train_loss=0.129]\n", "Epoch: 16 train: 0.11254 val: 0.21105 eval-auc: 0.80617 eval-patk: 0.05890 \n", "(17 ): 100% 46/46 [00:01<00:00, 30.27it/s, train_loss=0.111]\n", "Epoch: 17 train: 0.10796 val: 0.20284 eval-auc: 0.80958 eval-patk: 0.05890 \n", "(18 ): 100% 46/46 [00:01<00:00, 30.48it/s, train_loss=0.1]\n", "Epoch: 18 train: 0.10627 val: 0.19820 eval-auc: 0.81167 eval-patk: 0.06119 \n", "(19 ): 100% 46/46 [00:01<00:00, 29.63it/s, train_loss=0.132]\n", "Epoch: 19 train: 0.10392 val: 0.20573 eval-auc: 0.81511 eval-patk: 0.06370 \n", "(20 ): 100% 46/46 [00:01<00:00, 29.22it/s, train_loss=0.106]\n", "Epoch: 20 train: 0.10310 val: 0.20031 eval-auc: 0.81784 eval-patk: 0.06393 \n", "(21 ): 100% 46/46 [00:01<00:00, 29.44it/s, train_loss=0.084]\n", "Epoch: 21 train: 0.10323 val: 0.19672 eval-auc: 0.82062 eval-patk: 0.06530 \n", "(22 ): 100% 46/46 [00:01<00:00, 28.61it/s, train_loss=0.123]\n", "Epoch: 22 train: 0.10163 val: 0.19164 eval-auc: 0.82266 eval-patk: 0.06986 \n", "(23 ): 100% 46/46 [00:01<00:00, 29.98it/s, train_loss=0.109]\n", "Epoch: 23 train: 0.09932 val: 0.18622 eval-auc: 0.82489 eval-patk: 0.06849 \n", "(24 ): 100% 46/46 [00:01<00:00, 30.33it/s, train_loss=0.125]\n", "Epoch: 24 train: 0.09856 val: 0.18985 eval-auc: 0.82689 eval-patk: 0.06941 \n", "(25 ): 100% 46/46 [00:01<00:00, 30.46it/s, train_loss=0.0867]\n", "Epoch: 25 train: 0.09591 val: 0.18680 eval-auc: 0.82851 eval-patk: 0.07100 \n", "(26 ): 100% 46/46 [00:01<00:00, 29.23it/s, train_loss=0.0945]\n", "Epoch: 26 train: 0.09670 val: 0.18181 eval-auc: 0.83038 eval-patk: 0.07009 \n", "(27 ): 100% 46/46 [00:01<00:00, 29.79it/s, train_loss=0.0699]\n", "Epoch: 27 train: 0.09253 val: 0.18122 eval-auc: 0.83169 eval-patk: 0.06667 \n", "(28 ): 100% 46/46 [00:01<00:00, 30.00it/s, train_loss=0.0759]\n", "Epoch: 28 train: 0.09226 val: 0.18196 eval-auc: 0.83282 eval-patk: 0.06826 \n", "(29 ): 100% 46/46 [00:01<00:00, 29.22it/s, train_loss=0.0822]\n", "Epoch: 29 train: 0.09307 val: 0.18249 eval-auc: 0.83441 eval-patk: 0.07648 \n", "(30 ): 100% 46/46 [00:01<00:00, 30.18it/s, train_loss=0.114]\n", "Epoch: 30 train: 0.09162 val: 0.18411 eval-auc: 0.83504 eval-patk: 0.07648 \n", "(31 ): 100% 46/46 [00:01<00:00, 29.39it/s, train_loss=0.086]\n", "Epoch: 31 train: 0.08987 val: 0.17815 eval-auc: 0.83631 eval-patk: 0.07374 \n", "(32 ): 100% 46/46 [00:01<00:00, 29.27it/s, train_loss=0.0911]\n", "Epoch: 32 train: 0.08841 val: 0.18399 eval-auc: 0.83683 eval-patk: 0.07306 \n", "(33 ): 100% 46/46 [00:01<00:00, 29.72it/s, train_loss=0.0876]\n", "Epoch: 33 train: 0.09061 val: 0.17719 eval-auc: 0.83845 eval-patk: 0.07489 \n", "(34 ): 100% 46/46 [00:01<00:00, 29.93it/s, train_loss=0.0647]\n", "Epoch: 34 train: 0.08688 val: 0.18095 eval-auc: 0.83955 eval-patk: 0.06918 \n", "(35 ): 100% 46/46 [00:01<00:00, 30.05it/s, train_loss=0.0928]\n", "Epoch: 35 train: 0.08915 val: 0.17626 eval-auc: 0.84050 eval-patk: 0.07215 \n", "(36 ): 100% 46/46 [00:01<00:00, 29.89it/s, train_loss=0.111]\n", "Epoch: 36 train: 0.08683 val: 0.17530 eval-auc: 0.84146 eval-patk: 0.07420 \n", "(37 ): 100% 46/46 [00:01<00:00, 29.70it/s, train_loss=0.0915]\n", "Epoch: 37 train: 0.08663 val: 0.16717 eval-auc: 0.84286 eval-patk: 0.07215 \n", "(38 ): 100% 46/46 [00:01<00:00, 29.93it/s, train_loss=0.0765]\n", "Epoch: 38 train: 0.08452 val: 0.16749 eval-auc: 0.84417 eval-patk: 0.07763 \n", "(39 ): 100% 46/46 [00:01<00:00, 30.19it/s, train_loss=0.0737]\n", "Epoch: 39 train: 0.08514 val: 0.16763 eval-auc: 0.84427 eval-patk: 0.07443 \n", "(40 ): 100% 46/46 [00:01<00:00, 29.84it/s, train_loss=0.0934]\n", "Epoch: 40 train: 0.08454 val: 0.16994 eval-auc: 0.84553 eval-patk: 0.07283 \n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "Y2yIPrmu98bL" }, "source": [ "## Hybrid Model with PyTorch on ML-100k" ] }, { "cell_type": "markdown", "metadata": { "id": "kImzuHXJ9_kV" }, "source": [ "Testing out the features of Collie Recs library on MovieLens-100K. Training Factorization and Hybrid models with Pytorch Lightning." ] }, { "cell_type": "code", "metadata": { "id": "P43fS4H27gCt" }, "source": [ "!pip install -q collie_recs\n", "!pip install -q git+https://github.com/sparsh-ai/recochef.git" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "PD0n8kefAb67" }, "source": [ "import os\n", "import joblib\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from collie_recs.interactions import Interactions\n", "from collie_recs.interactions import ApproximateNegativeSamplingInteractionsDataLoader\n", "from collie_recs.cross_validation import stratified_split\n", "from collie_recs.metrics import auc, evaluate_in_batches, mapk, mrr\n", "from collie_recs.model import CollieTrainer, MatrixFactorizationModel, HybridPretrainedModel\n", "from collie_recs.movielens import get_recommendation_visualizations\n", "\n", "import torch\n", "from pytorch_lightning.utilities.seed import seed_everything\n", "\n", "from recochef.datasets.movielens import MovieLens\n", "from recochef.preprocessing.encode import label_encode as le\n", "\n", "from IPython.display import HTML" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ClWobbREN4VU", "outputId": "3f4c1125-a726-444e-d6f5-bc231df2a167" }, "source": [ "# this handy PyTorch Lightning function fixes random seeds across all the libraries used here\n", "seed_everything(22)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Global seed set to 22\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "22" ] }, "metadata": {}, "execution_count": 10 } ] }, { "cell_type": "markdown", "metadata": { "id": "T0N-kmIrcDZr" }, "source": [ "### Data Loading" ] }, { "cell_type": "code", "metadata": { "id": "LEuHGsYgGv-e" }, "source": [ "data_object = MovieLens()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "g3lLxQE1G120", "outputId": "2c6b310f-b237-4f1f-a69e-bee2ac458172" }, "source": [ "df = data_object.load_interactions()\n", "df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
01962423.0881250949
11863023.0891717742
2223771.0878887116
3244512.0880606923
41663461.0886397596
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 196 242 3.0 881250949\n", "1 186 302 3.0 891717742\n", "2 22 377 1.0 878887116\n", "3 244 51 2.0 880606923\n", "4 166 346 1.0 886397596" ] }, "metadata": {}, "execution_count": 12 } ] }, { "cell_type": "markdown", "metadata": { "id": "3X8-bE9YcGBg" }, "source": [ "### Preprocessing" ] }, { "cell_type": "code", "metadata": { "id": "tJvifZdrLsGK" }, "source": [ "# drop duplicate user-item pair records, keeping recent ratings only\n", "df.drop_duplicates(subset=['USERID','ITEMID'], keep='last', inplace=True)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "eq7GY0lEINgA" }, "source": [ "# convert the explicit data to implicit by only keeping interactions with a rating ``>= 4``\n", "df = df[df.RATING>=4].reset_index(drop=True)\n", "df['RATING'] = 1" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Jo4FnRzSHPzs" }, "source": [ "# label encode\n", "df, umap = le(df, col='USERID')\n", "df, imap = le(df, col='ITEMID')\n", "\n", "df = df.astype('int64')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "LA9BtiDrJ_wf", "outputId": "9044fa8b-53c4-421f-cbfa-8ce6b374d666" }, "source": [ "df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
0001884182806
1111891628467
2221879781125
3331876042340
4441879270459
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 0 0 1 884182806\n", "1 1 1 1 891628467\n", "2 2 2 1 879781125\n", "3 3 3 1 876042340\n", "4 4 4 1 879270459" ] }, "metadata": {}, "execution_count": 16 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "W3hUWib-MyjB", "outputId": "148d3ca2-17fb-4622-db52-122f045d1563" }, "source": [ "user_counts = df.groupby(by='USERID')['ITEMID'].count()\n", "user_list = user_counts[user_counts>=3].index.tolist()\n", "df = df[df.USERID.isin(user_list)]\n", "\n", "df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
0001884182806
1111891628467
2221879781125
3331876042340
4441879270459
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 0 0 1 884182806\n", "1 1 1 1 891628467\n", "2 2 2 1 879781125\n", "3 3 3 1 876042340\n", "4 4 4 1 879270459" ] }, "metadata": {}, "execution_count": 17 } ] }, { "cell_type": "markdown", "metadata": { "id": "37Y7qEEJNiE4" }, "source": [ "### Interactions\n", "While we have chosen to represent the data as a ``pandas.DataFrame`` for easy viewing now, Collie uses a custom ``torch.utils.data.Dataset`` called ``Interactions``. This class stores a sparse representation of the data and offers some handy benefits, including: \n", "\n", "* The ability to index the data with a ``__getitem__`` method \n", "* The ability to sample many negative items (we will get to this later!) \n", "* Nice quality checks to ensure data is free of errors before model training \n", "\n", "Instantiating the object is simple! " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "dLcqt57TI-6l", "outputId": "b399e7e8-fec0-460a-8048-e7a1e9395106" }, "source": [ "interactions = Interactions(\n", " users=df['USERID'],\n", " items=df['ITEMID'],\n", " ratings=df['RATING'],\n", " allow_missing_ids=True,\n", ")\n", "\n", "interactions" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Checking for and removing duplicate user, item ID pairs...\n", "Checking ``num_negative_samples`` is valid...\n", "Maximum number of items a user has interacted with: 378\n", "Generating positive items set...\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "Interactions object with 55375 interactions between 942 users and 1447 items, returning 10 negative samples per interaction." ] }, "metadata": {}, "execution_count": 18 } ] }, { "cell_type": "markdown", "metadata": { "id": "4TaWM_OFNZzn" }, "source": [ "### Data Splits \n", "With an ``Interactions`` dataset, Collie supports two types of data splits. \n", "\n", "1. **Random split**: This code randomly assigns an interaction to a ``train``, ``validation``, or ``test`` dataset. While this is significantly faster to perform than a stratified split, it does not guarantee any balance, meaning a scenario where a user will have no interactions in the ``train`` dataset and all in the ``test`` dataset is possible. \n", "2. **Stratified split**: While this code runs slower than a random split, this guarantees that each user will be represented in the ``train``, ``validation``, and ``test`` dataset. This is by far the most fair way to train and evaluate a recommendation model. \n", "\n", "Since this is a small dataset and we have time, we will go ahead and use ``stratified_split``. If you're short on time, a ``random_split`` can easily be swapped in, since both functions share the same API! " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "U_4IPy2aNLVE", "outputId": "5db1d12d-5d17-46f5-c4d1-05cfe4d458e4" }, "source": [ "train_interactions, val_interactions = stratified_split(interactions, test_p=0.1, seed=42)\n", "train_interactions, val_interactions" ], "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Generating positive items set...\n", "Generating positive items set...\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "(Interactions object with 49426 interactions between 942 users and 1447 items, returning 10 negative samples per interaction.,\n", " Interactions object with 5949 interactions between 942 users and 1447 items, returning 10 negative samples per interaction.)" ] }, "metadata": {}, "execution_count": 19 } ] }, { "cell_type": "markdown", "metadata": { "id": "ZO5rLYzH-imu" }, "source": [ "### Model Architecture \n", "With our data ready-to-go, we can now start training a recommendation model. While Collie has several model architectures built-in, the simplest by far is the ``MatrixFactorizationModel``, which use ``torch.nn.Embedding`` layers and a dot-product operation to perform matrix factorization via collaborative filtering." ] }, { "cell_type": "markdown", "metadata": { "id": "ZWi4VUjeghUA" }, "source": [ "Digging through the code of [``collie_recs.model.MatrixFactorizationModel``](../collie_recs/model.py) shows the architecture is as simple as we might think. For simplicity, we will include relevant portions below so we know exactly what we are building: \n", "\n", "````python\n", "def _setup_model(self, **kwargs) -> None:\n", " self.user_biases = ZeroEmbedding(num_embeddings=self.hparams.num_users,\n", " embedding_dim=1,\n", " sparse=self.hparams.sparse)\n", " self.item_biases = ZeroEmbedding(num_embeddings=self.hparams.num_items,\n", " embedding_dim=1,\n", " sparse=self.hparams.sparse)\n", " self.user_embeddings = ScaledEmbedding(num_embeddings=self.hparams.num_users,\n", " embedding_dim=self.hparams.embedding_dim,\n", " sparse=self.hparams.sparse)\n", " self.item_embeddings = ScaledEmbedding(num_embeddings=self.hparams.num_items,\n", " embedding_dim=self.hparams.embedding_dim,\n", " sparse=self.hparams.sparse)\n", "\n", " \n", "def forward(self, users: torch.tensor, items: torch.tensor) -> torch.tensor:\n", " user_embeddings = self.user_embeddings(users)\n", " item_embeddings = self.item_embeddings(items)\n", "\n", " preds = (\n", " torch.mul(user_embeddings, item_embeddings).sum(axis=1)\n", " + self.user_biases(users).squeeze(1)\n", " + self.item_biases(items).squeeze(1)\n", " )\n", "\n", " if self.hparams.y_range is not None:\n", " preds = (\n", " torch.sigmoid(preds)\n", " * (self.hparams.y_range[1] - self.hparams.y_range[0])\n", " + self.hparams.y_range[0]\n", " )\n", "\n", " return preds\n", "````\n", "\n", "Let's go ahead and instantiate the model and start training! Note that even if you are running this model on a CPU instead of a GPU, this will still be relatively quick to fully train. " ] }, { "cell_type": "markdown", "metadata": { "id": "L7o3t9vNN-Lt" }, "source": [ "Collie is built with PyTorch Lightning, so all the model classes and the ``CollieTrainer`` class accept all the training options available in PyTorch Lightning. Here, we're going to set the embedding dimension and learning rate differently, and go with the defaults for everything else" ] }, { "cell_type": "code", "metadata": { "id": "LNfxzlruN1xx" }, "source": [ "model = MatrixFactorizationModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " embedding_dim=10,\n", " lr=1e-2,\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "TKxeyMsMN1vg" }, "source": [ "trainer = CollieTrainer(model, max_epochs=10, deterministic=True)\n", "\n", "trainer.fit(model)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "dx8EoEhC_Cjh" }, "source": [ "```text\n", "GPU available: False, used: False\n", "TPU available: False, using: 0 TPU cores\n", "IPU available: False, using: 0 IPUs\n", "\n", " | Name | Type | Params\n", "----------------------------------------------------\n", "0 | user_biases | ZeroEmbedding | 942 \n", "1 | item_biases | ZeroEmbedding | 1.4 K \n", "2 | user_embeddings | ScaledEmbedding | 9.4 K \n", "3 | item_embeddings | ScaledEmbedding | 14.5 K\n", "4 | dropout | Dropout | 0 \n", "----------------------------------------------------\n", "26.3 K Trainable params\n", "0 Non-trainable params\n", "26.3 K Total params\n", "0.105 Total estimated model params size (MB)\n", "Validation sanity check: 0%\n", "0/2 [00:00User 895:\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Willy Wonka and the Chocolate Factory (1971)Mighty Aphrodite (1995)Conspiracy Theory (1997)Sense and Sensibility (1995)Liar Liar (1997)In & Out (1997)Return of the Jedi (1983)Ransom (1996)Emma (1996)Toy Story (1995)
Some loved films:
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Princess Bride, The (1987)Graduate, The (1967)Cold Comfort Farm (1995)Apartment, The (1960)Jerry Maguire (1996)Sleeper (1973)Independence Day (ID4) (1996)Desperado (1995)Three Colors: Red (1994)Lawnmower Man, The (1992)
Recommended films:
-----

User 895 has rated 12 films with a 4 or 5

User 895 has rated 8 films with a 1, 2, or 3

% of these films rated 5 or 4 appearing in the first 10 recommendations:0.0%

% of these films rated 1, 2, or 3 appearing in the first 10 recommendations: 0.0%

" ], "text/plain": [ "" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "metadata": { "id": "DbZ9ufGvghUE" }, "source": [ "### Save and Load a Standard Model " ] }, { "cell_type": "code", "metadata": { "id": "WqJbXHXgghUG" }, "source": [ "# we can save the model with...\n", "os.makedirs('models', exist_ok=True)\n", "model.save_model('models/matrix_factorization_model.pth')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Dz_8miLPghUG", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "1f2f8c07-be29-4fb9-ca8c-168ac1515f16" }, "source": [ "# ... and if we wanted to load that model back in, we can do that easily...\n", "model_loaded_in = MatrixFactorizationModel(load_model_path='models/matrix_factorization_model.pth')\n", "\n", "model_loaded_in" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "MatrixFactorizationModel(\n", " (user_biases): ZeroEmbedding(942, 1)\n", " (item_biases): ZeroEmbedding(1447, 1)\n", " (user_embeddings): ScaledEmbedding(942, 10)\n", " (item_embeddings): ScaledEmbedding(1447, 10)\n", " (dropout): Dropout(p=0.0, inplace=False)\n", ")" ] }, "metadata": {}, "execution_count": 25 } ] }, { "cell_type": "markdown", "metadata": { "id": "cQpWlCuughUH" }, "source": [ "Now that we've built our first model and gotten some baseline metrics, we now will be looking at some more advanced features in Collie's ``MatrixFactorizationModel``. " ] }, { "cell_type": "markdown", "metadata": { "id": "Q3Ne8ETsgzLe" }, "source": [ "### Faster Data Loading Through Approximate Negative Sampling " ] }, { "cell_type": "markdown", "metadata": { "id": "8nBu6PZhgzLe" }, "source": [ "With sufficiently large enough data, verifying that each negative sample is one a user has *not* interacted with becomes expensive. With many items, this can soon become a bottleneck in the training process. \n", "\n", "Yet, when we have many items, the chances a user has interacted with most is increasingly rare. Say we have ``1,000,000`` items and we want to sample ``10`` negative items for a user that has positively interacted with ``200`` items. The chance that we accidentally select a positive item in a random sample of ``10`` items is just ``0.2%``. At that point, it might be worth it to forgo the expensive check to assert our negative sample is true, and instead just randomly sample negative items with the hope that most of the time, they will happen to be negative. \n", "\n", "This is the theory behind the ``ApproximateNegativeSamplingInteractionsDataLoader``, an alternate DataLoader built into Collie. Let's train a model with this below, noting how similar this procedure looks to that in the previous tutorial. " ] }, { "cell_type": "code", "metadata": { "id": "MgSijz04gzLf" }, "source": [ "train_loader = ApproximateNegativeSamplingInteractionsDataLoader(train_interactions, batch_size=1024, shuffle=True)\n", "val_loader = ApproximateNegativeSamplingInteractionsDataLoader(val_interactions, batch_size=1024, shuffle=False)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "n9wQLd9-gzLf" }, "source": [ "model = MatrixFactorizationModel(\n", " train=train_loader,\n", " val=val_loader,\n", " embedding_dim=10,\n", " lr=1e-2,\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "AbYirCSNgzLg" }, "source": [ "trainer = CollieTrainer(model, max_epochs=10, deterministic=True)\n", "\n", "trainer.fit(model)\n", "model.eval()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "uvykQiW2_Oof" }, "source": [ "```text\n", "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "----------------------------------------------------\n", "0 | user_biases | ZeroEmbedding | 941 \n", "1 | item_biases | ZeroEmbedding | 1.4 K \n", "2 | user_embeddings | ScaledEmbedding | 9.4 K \n", "3 | item_embeddings | ScaledEmbedding | 14.5 K\n", "4 | dropout | Dropout | 0 \n", "----------------------------------------------------\n", "26.3 K Trainable params\n", "0 Non-trainable params\n", "26.3 K Total params\n", "0.105 Total estimated model params size (MB)\n", "Detected GPU. Setting ``gpus`` to 1.\n", "Global seed set to 22\n", "\n", "MatrixFactorizationModel(\n", " (user_biases): ZeroEmbedding(941, 1)\n", " (item_biases): ZeroEmbedding(1447, 1)\n", " (user_embeddings): ScaledEmbedding(941, 10)\n", " (item_embeddings): ScaledEmbedding(1447, 10)\n", " (dropout): Dropout(p=0.0, inplace=False)\n", ")\n", "```" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "d7c34c7be74246acb1a7da702b490029", "8ca15bf2e3be4ba88b7dbbf4ed26820d", "3e63ae601740455e81c35d4fbab2db6e", "b289ad3cdd3f4a25a9e92ed22c37aeed", "3d7d55e46c7d4e2caa6680229fe73b4e", "e841d95562314124b242c2e4225afbdb", "7b9d13b53df04e2082d4e236f50b807d", "9a4137c369cc4f26817ed3ab568a5ef7" ] }, "id": "xLKDSq-hgzLg", "outputId": "44183d74-a567-418d-a85e-946bf1443713" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc], val_interactions, model)\n", "\n", "print(f'MAP@10 Score: {mapk_score}')\n", "print(f'MRR Score: {mrr_score}')\n", "print(f'AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d7c34c7be74246acb1a7da702b490029", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "MAP@10 Score: 0.027979833367276323\n", "MRR Score: 0.1703751336709069\n", "AUC Score: 0.8517987786322347\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "qipOCQzxgzLh" }, "source": [ "We're seeing a small hit on performance and only a marginal improvement in training time compared to the standard ``MatrixFactorizationModel`` model because MovieLens 100K has so few items. ``ApproximateNegativeSamplingInteractionsDataLoader`` is especially recommended for when we have more items in our data and training times need to be optimized. \n", "\n", "For more details on this and other DataLoaders in Collie (including those for out-of-memory datasets), check out the [docs](https://collie.readthedocs.io/en/latest/index.html)! " ] }, { "cell_type": "markdown", "metadata": { "id": "iGgM-FaegzLi" }, "source": [ "### Multiple Optimizers " ] }, { "cell_type": "markdown", "metadata": { "id": "4xcFspsbgzLi" }, "source": [ "Training recommendation models at ShopRunner, we have encountered something we call \"the curse of popularity.\" \n", "\n", "This is best thought of in the viewpoint of a model optimizer - say we have a user, a positive item, and several negative items that we hope have recommendation scores that score lower than the positive item. As an optimizer, you can either optimize every single embedding dimension (hundreds of parameters) to achieve this, or instead choose to score a quick win by optimizing the bias terms for the items (just add a positive constant to the positive item and a negative constant to each negative item). \n", "\n", "While we clearly want to have varied embedding layers that reflect each user and item's taste profiles, some models learn to settle for popularity as a recommendation score proxy by over-optimizing the bias terms, essentially just returning the same set of recommendations for every user. Worst of all, since popular items are... well, popular, **the loss of this model will actually be decent, solidifying the model getting stuck in a local loss minima**. \n", "\n", "To counteract this, Collie supports multiple optimizers in a ``MatrixFactorizationModel``. With this, we can have a faster optimizer work to optimize the embedding layers for users and items, and a slower optimizer work to optimize the bias terms. With this, we impel the model to do the work actually coming up with varied, personalized recommendations for users while still taking into account the necessity of the bias (popularity) terms on recommendations. \n", "\n", "At ShopRunner, we have seen significantly better metrics and results from this type of model. With Collie, this is simple to do, as shown below. " ] }, { "cell_type": "code", "metadata": { "id": "GxUMsr61gzLj" }, "source": [ "model = MatrixFactorizationModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " embedding_dim=10,\n", " lr=1e-2,\n", " bias_lr=1e-1,\n", " optimizer='adam',\n", " bias_optimizer='sgd',\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 557, "referenced_widgets": [ "49f97e2e5f55454bab313b1570b4b9c4", "b3c7b293fe8b44e0920b060f00382401", "80b4510b6a7e4d77879516c59662180c", "f1e818c6ef934704ae0ef1f80eaa7b5b", "695c6402e0a1475eb2965bee589c2bf4", "06f180111dfe4f57a12723bccb253f98", "316eaa17f9cd4558b8c7f6951b98d6f0", "75fc6c2f339b4e47bb22790d37e7ebab", "8d714e45f0d64e85b637961f4a25f3d0", "b5a4c2dd499c4d5c8d3ab2e890b44642", "d1dd8920ddfc4c97beffe89ece6b982e", "fc7d1f4bea4145af93d9059177a477fd", "ef66bb3e20eb4d25a6dc3af0fccf2e2a", "9275275798974477a5f5858880d1b429", "2bcffbf7f05b41688936d51d6c3ffcac", "28198fbab991465583818c47d8ddbb6d", "83658275a6c346d7822fd1368345e09a", "3389371180f64e088e7c6549a25b3ee1", "54d47349fff54cd0bca2e4903b06e5d8", "ec37483292aa4715a97c6f0de133cb24", "78845118227b40adac2503e57e1f38a7", "ce20fc80baf0494f976d7ebb54303833", "b0606438031a4d0fbe1b3b1bc4d49e9e", "38cb1183eb294363af2d2762d8a49a13", "b9a2d6afed7641d995fba91c5a94f8ec", "6f42c45ff32f46deb83569801ab976c8", "58d6593d8810421bb8bd39099e82feb9", "7cdc87e9c27f48e2b081bdbf2d6228e8", "45b86b0dc3394da4b2f87191d23dfed7", "585898b88c044e2e8d6fd0c46c3b575d", "a0259f8bb2264993a6d27acd097b6c05", "5b0ed212d376472987768ea06e314f9d", "a59ea6641fca44bbaa553a1624deab51", "671b80df47dc444d817c51101c7adf49", "d56ba50fc09c4502a1055ff9a3199062", "1bd1fa4ccb564778be34c5fe9036e731", "1b3ff709422c4864acff0ca42ec89e07", "5b6f8c0d04be4ec19c55259caf622c02", "b8903c665b7c4a3695101aa7d6e8f929", "518b4b044b4d411fb43dac5fc993c02c", "36d600f8e9be46a6b05ec580025aac20", "f14194bdbf724d20bdd7d5b572374608", "b62d3801dd3d4f45b53de6d6c25e26bd", "066ce24dcd2e46acb54f1fd437963b85", "bd80f8f37cb247c69f969d96b146ea8c", "4100968fe609430f8cd1f73c48a356f2", "9e6c6c48df824c548aa0425888adac9b", "27866404913f4820bb46d1641f63dc1f", "d75b9faa61b242279706a77458e55276", "ba792412f171494d937051219e01607d", "f43bba18ba134b269006e100f8de55d1", "63c6fb5bfa3147a89962641d1a7d215b", "6d7a19d43c4846d99d3d470fd5d7d66e", "90af7f79f905435f8c81ca21e59cab59", "fad0632c95f74aca92b6e72751da0632", "7eb49fb403564170b47e0c6f867f81e1", "57322a343ddd4ad2bc408a840eca0e98", "18b3142ac5c24401941cd4f79bac783b", "6d730313b939447bbd396526cdee3fad", "73cccc23b004406e8521b0cc26401c9c", "7520241c3b2c4591bc0532d20dafccda", "5dd8ebbab9ce418cae4eaf9f568ab5c9", "0afd2fe607b04590813dc297c3bfc4da", "c14405b6297a435eaab8032dbbb9966f", "0351e40f8c0f46708aeea7233565800a", "d7e152362cd54b4380a6cd3e506e7b86", "73dff796a59d444a96016b2578f277d7", "89a5fbcb3a4e4732a72c8cee79a346b6", "bcde36e5444444568f7a6fa34d85ccaa", "53e3249bbef446208f54a6216420062f", "235ef9871d5443a9ab3b31f1c231f53d", "b329742fdb3e44e2974d3a31700072a3", "93fbdf1f9df1464888cdecc6285ef575", "7ef0973e658740528843af596067fe8e", "dad5c7f76e20437ca651d4d39ac68660", "3a61c8ddd8b74af78c6ceffb7001da02", "fec250de7d6f45688eb48b51d837b53a", "54d415a20e204d53bbe9baddd4598e2f", "136520e97ebe4b04b9b46a44e46297fc", "c4d1f1810116465b9dcb9282eed0f113", "2c8df37c614447d5b5064dbbaa837081", "dc9b62bd3eb94dc8afd9c21491faff0b", "2416bc3ad43149ff9e8d10572693f115", "93054dc57a344b9daae9e6e36e14b594", "4d9cf1e48cb74afb91410b46b2d601b5", "897727bf519648e0bc3f204771ded957", "795429863893411cab25dfe5771a2e4a", "93a527dd2045487393fb8280ff3945b1", "3d3ca53ac0cb41d1bbb58aa754a79619", "600e766c4f4c4136875db902954f9ac7", "2cf5bd88c0c74729a89a70f6ba9dd02e", "a247748059184bbaac041b8fffd99e3f", "010c2bf6e2af470d950542394f8ceef5", "5bb0a358fcd64106acdac950f82d12a7", "16202361258644f6ab7b168990f73136", "c2fffba8678b48f9957b9a581aae9e2e" ] }, "id": "dQ_tTRfOgzLj", "outputId": "4d3af437-c62d-4b9b-ddb5-eaf3731556da" }, "source": [ "trainer = CollieTrainer(model, max_epochs=10, deterministic=True)\n", "\n", "trainer.fit(model)\n", "model.eval()" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "----------------------------------------------------\n", "0 | user_biases | ZeroEmbedding | 941 \n", "1 | item_biases | ZeroEmbedding | 1.4 K \n", "2 | user_embeddings | ScaledEmbedding | 9.4 K \n", "3 | item_embeddings | ScaledEmbedding | 14.5 K\n", "4 | dropout | Dropout | 0 \n", "----------------------------------------------------\n", "26.3 K Trainable params\n", "0 Non-trainable params\n", "26.3 K Total params\n", "0.105 Total estimated model params size (MB)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Detected GPU. Setting ``gpus`` to 1.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "49f97e2e5f55454bab313b1570b4b9c4", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validation sanity check', layout=Layout…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Global seed set to 22\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\r" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8d714e45f0d64e85b637961f4a25f3d0", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "83658275a6c346d7822fd1368345e09a", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b9a2d6afed7641d995fba91c5a94f8ec", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a59ea6641fca44bbaa553a1624deab51", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n", "Epoch 3: reducing learning rate of group 0 to 1.0000e-02.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "36d600f8e9be46a6b05ec580025aac20", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d75b9faa61b242279706a77458e55276", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "57322a343ddd4ad2bc408a840eca0e98", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0351e40f8c0f46708aeea7233565800a", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "93fbdf1f9df1464888cdecc6285ef575", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2c8df37c614447d5b5064dbbaa837081", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3d3ca53ac0cb41d1bbb58aa754a79619", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "MatrixFactorizationModel(\n", " (user_biases): ZeroEmbedding(941, 1)\n", " (item_biases): ZeroEmbedding(1447, 1)\n", " (user_embeddings): ScaledEmbedding(941, 10)\n", " (item_embeddings): ScaledEmbedding(1447, 10)\n", " (dropout): Dropout(p=0.0, inplace=False)\n", ")" ] }, "metadata": { "tags": [] }, "execution_count": 55 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "5c2218b13d6345ff91c4d8980b2b2ca0", "309d24f05b1b4bc5b7d0148795fd5cf4", "ccfb5dc3eef14f1f8324bc48b3dc0e7f", "a84e5372c5c24417a08487dd3efcebe8", "c28c07424b7f45088e73ac25f2bb712a", "5d01bda6f6354f25bb6dcb15ef4596a0", "758cba83e9414f60bf87799a71b3cd7f", "48d71d4847be4a25a237568371ae4493" ] }, "id": "ENZSOd1DgzLk", "outputId": "7b1844fb-b81a-43cb-8a98-5206b87b4506" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc], val_interactions, model)\n", "\n", "print(f'MAP@10 Score: {mapk_score}')\n", "print(f'MRR Score: {mrr_score}')\n", "print(f'AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5c2218b13d6345ff91c4d8980b2b2ca0", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "MAP@10 Score: 0.03243186201880122\n", "MRR Score: 0.19819369246580287\n", "AUC Score: 0.8617710409716284\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "blXcVpg3gzLk" }, "source": [ "Again, we're not seeing as much performance increase here compared to the standard model because MovieLens 100K has so few items. For a more dramatic difference, try training this model on a larger dataset, such as MovieLens 10M, adjusting the architecture-specific hyperparameters, or train longer. " ] }, { "cell_type": "markdown", "metadata": { "id": "1Dt0IsWJgzLk" }, "source": [ "### Item-Item Similarity " ] }, { "cell_type": "markdown", "metadata": { "id": "sqQJpyKVgzLl" }, "source": [ "While we've trained every model thus far to work for member-item recommendations (given a *member*, recommend *items* - think of this best as \"Personalized recommendations for you\"), we also have access to item-item recommendations for free (given a seed *item*, recommend similar *items* - think of this more like \"People who interacted with this item also interacted with...\"). \n", "\n", "With Collie, accessing this is simple! " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 343 }, "id": "RRMouFweUOhw", "outputId": "02e6281a-f64e-43c4-a151-84601d91ab50" }, "source": [ "df_item = data_object.load_items()\n", "df_item.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ITEMIDTITLERELEASEVIDRELEASEURLUNKNOWNACTIONADVENTUREANIMATIONCHILDRENCOMEDYCRIMEDOCUMENTARYDRAMAFANTASYFILMNOIRHORRORMUSICALMYSTERYROMANCESCIFITHRILLERWARWESTERN
01Toy Story (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Toy%20Story%2...0001110000000000000
12GoldenEye (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?GoldenEye%20(...0110000000000000100
23Four Rooms (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Four%20Rooms%...0000000000000000100
34Get Shorty (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Get%20Shorty%...0100010010000000000
45Copycat (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Copycat%20(1995)0000001010000000100
\n", "
" ], "text/plain": [ " ITEMID TITLE RELEASE ... THRILLER WAR WESTERN\n", "0 1 Toy Story (1995) 01-Jan-1995 ... 0 0 0\n", "1 2 GoldenEye (1995) 01-Jan-1995 ... 1 0 0\n", "2 3 Four Rooms (1995) 01-Jan-1995 ... 1 0 0\n", "3 4 Get Shorty (1995) 01-Jan-1995 ... 0 0 0\n", "4 5 Copycat (1995) 01-Jan-1995 ... 1 0 0\n", "\n", "[5 rows x 24 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 68 } ] }, { "cell_type": "code", "metadata": { "id": "Ycl8PcLIgzLl", "colab": { "base_uri": "https://localhost:8080/", "height": 343 }, "outputId": "27f2b6aa-9b6f-4fe4-9d8f-3637cadd4bbe" }, "source": [ "df_item = le(df_item, col='ITEMID', maps=imap)\n", "df_item.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ITEMIDTITLERELEASEVIDRELEASEURLUNKNOWNACTIONADVENTUREANIMATIONCHILDRENCOMEDYCRIMEDOCUMENTARYDRAMAFANTASYFILMNOIRHORRORMUSICALMYSTERYROMANCESCIFITHRILLERWARWESTERN
09.0Toy Story (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Toy%20Story%2...0001110000000000000
1160.0GoldenEye (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?GoldenEye%20(...0110000000000000100
2579.0Four Rooms (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Four%20Rooms%...0000000000000000100
325.0Get Shorty (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Get%20Shorty%...0100010010000000000
4436.0Copycat (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?Copycat%20(1995)0000001010000000100
\n", "
" ], "text/plain": [ " ITEMID TITLE RELEASE ... THRILLER WAR WESTERN\n", "0 9.0 Toy Story (1995) 01-Jan-1995 ... 0 0 0\n", "1 160.0 GoldenEye (1995) 01-Jan-1995 ... 1 0 0\n", "2 579.0 Four Rooms (1995) 01-Jan-1995 ... 1 0 0\n", "3 25.0 Get Shorty (1995) 01-Jan-1995 ... 0 0 0\n", "4 436.0 Copycat (1995) 01-Jan-1995 ... 1 0 0\n", "\n", "[5 rows x 24 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 69 } ] }, { "cell_type": "code", "metadata": { "id": "TvtbOrbtgzLl", "colab": { "base_uri": "https://localhost:8080/", "height": 117 }, "outputId": "67ef757a-91e9-4b32-c2ac-5f43c4742634" }, "source": [ "df_item.loc[df_item['TITLE'] == 'GoldenEye (1995)']" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ITEMIDTITLERELEASEVIDRELEASEURLUNKNOWNACTIONADVENTUREANIMATIONCHILDRENCOMEDYCRIMEDOCUMENTARYDRAMAFANTASYFILMNOIRHORRORMUSICALMYSTERYROMANCESCIFITHRILLERWARWESTERN
1160.0GoldenEye (1995)01-Jan-1995NaNhttp://us.imdb.com/M/title-exact?GoldenEye%20(...0110000000000000100
\n", "
" ], "text/plain": [ " ITEMID TITLE RELEASE ... THRILLER WAR WESTERN\n", "1 160.0 GoldenEye (1995) 01-Jan-1995 ... 1 0 0\n", "\n", "[1 rows x 24 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 70 } ] }, { "cell_type": "code", "metadata": { "id": "Uom6EVG8gzLm", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "a4f6c8a4-c472-48c2-f240-14763b5d1d8e" }, "source": [ "# let's start by finding movies similar to GoldenEye (1995)\n", "item_similarities = model.item_item_similarity(item_id=160)\n", "\n", "item_similarities" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "160 1.000000\n", "123 0.842003\n", "948 0.828162\n", "398 0.827030\n", "197 0.826931\n", " ... \n", "26 -0.654127\n", "88 -0.680429\n", "165 -0.697536\n", "499 -0.729313\n", "312 -0.780792\n", "Length: 1447, dtype: float64" ] }, "metadata": { "tags": [] }, "execution_count": 71 } ] }, { "cell_type": "code", "metadata": { "id": "MW741iOcgzLm", "colab": { "base_uri": "https://localhost:8080/", "height": 428 }, "outputId": "ad4b1617-bc85-4698-ebf6-65b70161e7ce" }, "source": [ "df_item.iloc[item_similarities.index][:5]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ITEMIDTITLERELEASEVIDRELEASEURLUNKNOWNACTIONADVENTUREANIMATIONCHILDRENCOMEDYCRIMEDOCUMENTARYDRAMAFANTASYFILMNOIRHORRORMUSICALMYSTERYROMANCESCIFITHRILLERWARWESTERN
162182.0Return of the Pink Panther, The (1974)01-Jan-1974NaNhttp://us.imdb.com/M/title-exact?Return%20of%2...0000010000000000000
125229.0Spitfire Grill, The (1996)06-Sep-1996NaNhttp://us.imdb.com/M/title-exact?Spitfire%20Gr...0000000010000000000
975933.0Solo (1996)23-Aug-1996NaNhttp://us.imdb.com/M/title-exact?Solo%20(1996)0100000000000001100
401175.0Ghost (1990)01-Jan-1990NaNhttp://us.imdb.com/M/title-exact?Ghost%20(1990)0000010000000010100
19972.0Shining, The (1980)01-Jan-1980NaNhttp://us.imdb.com/M/title-exact?Shining,%20Th...0000000000010000000
\n", "
" ], "text/plain": [ " ITEMID TITLE ... WAR WESTERN\n", "162 182.0 Return of the Pink Panther, The (1974) ... 0 0\n", "125 229.0 Spitfire Grill, The (1996) ... 0 0\n", "975 933.0 Solo (1996) ... 0 0\n", "401 175.0 Ghost (1990) ... 0 0\n", "199 72.0 Shining, The (1980) ... 0 0\n", "\n", "[5 rows x 24 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 72 } ] }, { "cell_type": "markdown", "metadata": { "id": "_py39oQ8gzLm" }, "source": [ "Unfortunately, not seen these movies. Can't say if these are relevant.\n", "\n", "``item_item_similarity`` method is available in all Collie models, not just ``MatrixFactorizationModel``! \n", "\n", "Next, we will incorporate item metadata into recommendations for even better results." ] }, { "cell_type": "markdown", "metadata": { "id": "8hkoWyfVg9AK" }, "source": [ "### Partial Credit Loss\n", "Most of the time, we don't *only* have user-item interactions, but also side-data about our items that we are recommending. These next two notebooks will focus on incorporating this into the model training process. \n", "\n", "In this notebook, we're going to add a new component to our loss function - \"partial credit\". Specifically, we're going to use the genre information to give our model \"partial credit\" for predicting that a user would like a movie that they haven't interacted with, but is in the same genre as one that they liked. The goal is to help our model learn faster from these similarities. " ] }, { "cell_type": "markdown", "metadata": { "id": "4iFhjr7eg9AK" }, "source": [ "### Read in Data" ] }, { "cell_type": "markdown", "metadata": { "id": "bK4bGSUEWe9F" }, "source": [ "To do the partial credit calculation, we need this data in a slightly different form. Instead of the one-hot-encoded version above, we're going to make a ``1 x n_items`` tensor with a number representing the first genre associated with the film, for simplicity. Note that with Collie, we could instead make a metadata tensor for each genre" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "mi7IZRHbWwsc", "outputId": "20906809-3682-4bee-eee3-1d98fe9b03ca" }, "source": [ "df_item.columns[5:]" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "Index(['UNKNOWN', 'ACTION', 'ADVENTURE', 'ANIMATION', 'CHILDREN', 'COMEDY',\n", " 'CRIME', 'DOCUMENTARY', 'DRAMA', 'FANTASY', 'FILMNOIR', 'HORROR',\n", " 'MUSICAL', 'MYSTERY', 'ROMANCE', 'SCIFI', 'THRILLER', 'WAR', 'WESTERN'],\n", " dtype='object')" ] }, "metadata": { "tags": [] }, "execution_count": 76 } ] }, { "cell_type": "code", "metadata": { "id": "bWBxAUUXYvZ3" }, "source": [ "metadata_df = df_item[df_item.columns[5:]]" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "LSy_-Jsxg9AL", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "a1d9aa32-d10c-454a-dc7b-9cfcda410071" }, "source": [ "genres = (\n", " torch.tensor(metadata_df.values)\n", " .topk(1)\n", " .indices\n", " .view(-1)\n", ")\n", "\n", "genres" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "tensor([ 5, 1, 16, ..., 14, 5, 8])" ] }, "metadata": { "tags": [] }, "execution_count": 77 } ] }, { "cell_type": "markdown", "metadata": { "id": "LhaQLQQig9AM" }, "source": [ "### Train a model with our new loss" ] }, { "cell_type": "markdown", "metadata": { "id": "5NrYwTFVXCco" }, "source": [ "now, we will pass in ``metadata_for_loss`` and ``metadata_for_loss_weights`` into the model ``metadata_for_loss`` should have a tensor containing the integer representations for metadata we created above for every item ID in our dataset ``metadata_for_loss_weights`` should have the weights for each of the keys in ``metadata_for_loss``" ] }, { "cell_type": "code", "metadata": { "id": "Sysr04kSg9AN" }, "source": [ "model = MatrixFactorizationModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " embedding_dim=10,\n", " lr=1e-2,\n", " metadata_for_loss={'genre': genres},\n", " metadata_for_loss_weights={'genre': 0.4},\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 472, "referenced_widgets": [ "7f9d29d6c1f04d4b9adcda6292a698fb", "4d95aa15fb8e45168224c12f033db89e", "567edd5b84854cc3ba9d8b34702716ce", "3ccc6ebee63d473aba28adc868b5b5c0", "5db3013f6fca47428b7f05d89d611441", "650154b034e843648fdd31198e6b5c8b", "b85c146244634db5a8ec5bb8c37a6a3f", "71e7704f60a44170988aed05dc8a4788", "71fb17475171409ab7b30f0915d3c3f9", "ec82867f02de401faf33bdeb69a1bf2e", "4ffc6e6cdbbb47239b7da5a07f16b5fb", "cce5557f779742e3932bb8ef2016a17c", "668011e6db324325974aa6a6b1fdc782", "ebcdd1df171847a38f7655716f2d6061", "9d11e361b37a4849a83531e249455615", "30fefc2060004b2bb028ed54ea9383b6", "85b96bc221984d08815cf3c09fde7c9f", "1abf2c736a91486eb8621d1cd679f9b2", "89b238d17aad4ef389c20de16ed1e55c", "70d5a3c3cf41478b83813ec40298cd22", "bdf376ea2f8e4b9e9d495a1fe44b2b59", "c6213cfdba3d4228aad387361e062fc7", "a28aa408fe444c179080e2ef9433a877", "6e3313408a184dcca85da1f7514d8bfe", "b6f3004c83574b379bc3520a410b5df5", "57e0d47b70b744898e4e548c28d27095", "d5f9ad23ba314382a2ffa29ecc311b5c", "75bcb22fb4514a288fdd43cdb2084b11", "d7bf1b026d10490fa70221af127986e5", "bef2ed39c203462a80a1f217adfc301b", "1038bcbdb10a4da08c7b2ffbc08c7e81", "2a435432b9914e3faf2d49a0645a9fd5", "127ad96af11c41519ded336bbc246c42", "b5fe20560cc64837af9e13936fd702d3", "2260ec6d57724ecca381bf65a151e97f", "add9b2d1bfeb4b3baaa97d39ca4eb8e3", "2baf39d33c2a4f28bfd557ba7b9f8c78", "bdf0ea6c2c8042d68c50cd5e19b1c15b", "143ecbd32aa14efeb1d2eae6873609ce", "b4eee80b60f748b4886ede6073b37a7c", "1997f9f7a4d04e5ebec4b8907e32b797", "21380c3a63df4a7592b0e580535603df", "0bd610aa3a55452dae1e3fbc721cb87a", "07e2308d6aff45c5bd0ad77e246e9981", "2354778e0ce6427385e2e107ea30332b", "717568c66bc444ddb60b415c5169588c", "13a5b3039ac04a2dbd6049992f822c89", "96773fc1d61d4b9faf8f8e7f0d0c3786", "2bb338780a1a40318d91fc689692c9ae", "869cd72009244cd997fc6b8bca19cd53", "8f341ddecce34a2e9967972e901123b6", "4f53235cc013444c8a71141df07ccc4b", "d76f878252a54a79bd98b1e093285964", "69b2d3fe2f5d4f73ba34142ccea3dbf8", "818427165e3c40d6b9c60f5fc26ea0db", "a62b8b75b5bb4b1a9a35a98b9f87edc0", "b066cece2abe4a74b1aed6c238cd82b3", "5a5b267987ed4ad0a143744fe30f27b1", "2666a159d7f1406887a6bff3efdd1dd6", "7b3b88973bc74728a3ea7eb04a122371", "7e684d1ba34a49e1b108909220e0a487", "a96a161717124b54ba9473f5d4476177", "2137633428de4deb8056902144e7551d", "2454cc03bb5342eca1ab02501d3ac39e", "2da2765da24841709884a7cca16f1107", "9d9f37ef8c9844a9ad5f4022ce904af3", "b7febc3da8714283b798aafaab3a8d8f", "f44ea695c0b2485faff83a6bbe12407e", "1e1888b6dc3e4c47aadf843a62f03233", "d75a669ab0a147ddb4185d6950c8267e", "a1eab9ad9a6147c2a3a13ed1d837f94b", "dd40e90d90a34827bb07d9d86c4234ce", "2391e70d0de74d71ab5372f39a4fa49b", "05fa6eea049e4b41a40604eed4c798d0", "e271494364c24f4e901c42b64e464056", "69b72f3ef4ae40a0a7b060be0849a354", "3e756777c21b4c56a06b537ca232ffc3", "0d4e9b5e9a3241dd920146b8be70d1ce", "d6e7bf43dc2346c0a2278d6b19078c90", "852593a171a048bc81b0cd3466366bd0", "13c2754aca2a42c2ac06d3640062b6d4", "1ac7aa00d4b24eccaf9b0cb622a65b05", "50691a81b511451aa9f78cbd0803a652", "43a743d7e9f14787a606cea5d0176931", "8561cb62512f47d9929c3abd557ef739", "52ac26d1722c40b68b18e0d92c6ac994", "cbf2e61a7dea4b91a8f86774bcfc8ff7", "8e58b5a80dca4cca8b03635c8d529072", "9adf92080dd2458a942dbd76a285ce48", "9a6a243411e84f049dfaeb99cccbc562", "5458638e910744509182e8c4ace883ec", "e692df9e6fb949c6a345dbaf9235d195", "7409891326ad4a259842211e306e980a", "0f57dd28b6954df693a160125075b506", "ae151ed064b547b6a8bca5181668f491", "5e8575da45094823bce0d16b9fe01f09" ] }, "id": "ZAk1C815g9AN", "outputId": "0c5600e3-d81d-4f6e-d621-c7882a767fb1" }, "source": [ "trainer = CollieTrainer(model=model, max_epochs=10, deterministic=True)\n", "\n", "trainer.fit(model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "----------------------------------------------------\n", "0 | user_biases | ZeroEmbedding | 941 \n", "1 | item_biases | ZeroEmbedding | 1.4 K \n", "2 | user_embeddings | ScaledEmbedding | 9.4 K \n", "3 | item_embeddings | ScaledEmbedding | 14.5 K\n", "4 | dropout | Dropout | 0 \n", "----------------------------------------------------\n", "26.3 K Trainable params\n", "0 Non-trainable params\n", "26.3 K Total params\n", "0.105 Total estimated model params size (MB)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Detected GPU. Setting ``gpus`` to 1.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7f9d29d6c1f04d4b9adcda6292a698fb", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validation sanity check', layout=Layout…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Global seed set to 22\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\r" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "71fb17475171409ab7b30f0915d3c3f9", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "85b96bc221984d08815cf3c09fde7c9f", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b6f3004c83574b379bc3520a410b5df5", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "127ad96af11c41519ded336bbc246c42", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n", "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1997f9f7a4d04e5ebec4b8907e32b797", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2bb338780a1a40318d91fc689692c9ae", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b066cece2abe4a74b1aed6c238cd82b3", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 6: reducing learning rate of group 0 to 1.0000e-04.\n", "Epoch 6: reducing learning rate of group 0 to 1.0000e-04.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2da2765da24841709884a7cca16f1107", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2391e70d0de74d71ab5372f39a4fa49b", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "13c2754aca2a42c2ac06d3640062b6d4", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9adf92080dd2458a942dbd76a285ce48", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "NQdGTCPvg9AN" }, "source": [ "### Evaluate the Model " ] }, { "cell_type": "markdown", "metadata": { "id": "ISQzTUnVg9AO" }, "source": [ "Again, we'll evaluate the model and look at some particular users' recommendations to get a sense of what these recommendations look like using a partial credit loss function during model training. " ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "b1c126acba99422c9aad4bc9b1b0293f", "3e7789b77e3342acabd132d924906d5e", "a9db4a7b06d34e2eaee4933580e8e28e", "2e28af584886415c869f079215546e5e", "e078e83f8216456997e974acfc7f4816", "12b2182ba29547a8845685618988224e", "6d1261287d6042b486877c29f1056305", "6715a50cf1074e98b4e8e8f9aca4932e" ] }, "id": "6STWH4Ozg9AO", "outputId": "861755ec-75be-40fe-975b-17accf21477f" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc], val_interactions, model)\n", "\n", "print(f'MAP@10 Score: {mapk_score}')\n", "print(f'MRR Score: {mrr_score}')\n", "print(f'AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b1c126acba99422c9aad4bc9b1b0293f", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "MAP@10 Score: 0.02882727154889818\n", "MRR Score: 0.1829242957435939\n", "AUC Score: 0.8585049499223719\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "Bk75mVQWg9AP" }, "source": [ "Broken record alert: we're not seeing as much performance increase here compared to the standard model because MovieLens 100K has so few items. For a more dramatic difference, try training this model on a larger dataset, such as MovieLens 10M, adjusting the architecture-specific hyperparameters, or train longer. " ] }, { "cell_type": "markdown", "metadata": { "id": "9X25yfucbbKx" }, "source": [ "### Inference" ] }, { "cell_type": "code", "metadata": { "id": "dB6eeXWfg9AP", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "805ffd4f-38fd-4f4b-d9d1-b9ddfbdda60c" }, "source": [ "user_id = np.random.randint(10, train_interactions.num_users)\n", "\n", "display(\n", " HTML(\n", " get_recommendation_visualizations(\n", " model=model,\n", " user_id=user_id,\n", " filter_films=True,\n", " shuffle=True,\n", " detailed=True,\n", " )\n", " )\n", ")" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "

User 895:

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Willy Wonka and the Chocolate Factory (1971)Mighty Aphrodite (1995)Conspiracy Theory (1997)Sense and Sensibility (1995)Liar Liar (1997)In & Out (1997)Return of the Jedi (1983)Ransom (1996)Emma (1996)Toy Story (1995)
Some loved films:
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Cold Comfort Farm (1995)Apartment, The (1960)Blown Away (1994)Star Wars (1977)Star Trek: First Contact (1996)Sex, Lies, and Videotape (1989)Big Squeeze, The (1996)Client, The (1994)Jerry Maguire (1996)Ghost and the Darkness, The (1996)
Recommended films:
-----

User 895 has rated 12 films with a 4 or 5

User 895 has rated 8 films with a 1, 2, or 3

% of these films rated 5 or 4 appearing in the first 10 recommendations:10.0%

% of these films rated 1, 2, or 3 appearing in the first 10 recommendations: 10.0%

" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "ZoIh0oHfg9AQ" }, "source": [ "Partial credit loss is useful when we want an easy way to boost performance of any implicit model architecture, hybrid or not. When tuned properly, partial credit loss more fairly penalizes the model for more egregious mistakes and relaxes the loss applied when items are more similar. \n", "\n", "Of course, the loss function isn't the only place we can incorporate this metadata - we can also directly use this in the model (and even use a hybrid model combined with partial credit loss). Next, we will train a hybrid Collie model! " ] }, { "cell_type": "markdown", "metadata": { "id": "Laxa0vh1hE3o" }, "source": [ "### Train a ``MatrixFactorizationModel`` " ] }, { "cell_type": "markdown", "metadata": { "id": "Fj3tJg-1hE3o" }, "source": [ "The first step towards training a Collie Hybrid model is to train a regular ``MatrixFactorizationModel`` to generate rich user and item embeddings. We'll use these embeddings in a ``HybridPretrainedModel`` a bit later. " ] }, { "cell_type": "code", "metadata": { "id": "m75xWkQLhE3o" }, "source": [ "model = MatrixFactorizationModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " embedding_dim=30,\n", " lr=1e-2,\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 438, "referenced_widgets": [ "62986af6e0394e969bb8942274ba6784", "4e51bc72f9fe41ca82096d01e26a5c23", "a3b4a6f432f7406b97c59bdd31eec848", "d6b7dc0b58874a93927b9182fe92ae63", "2528199682404110adb35e51bf1c39ff", "16dc9e31d8bb483faa575fd6db3915d7", "d9b41c7acb1741d98f5edf4e942e3b77", "7aabb759b56a4fc295687e0a24b6cb62", "b1ccac573c5c4c79b4357c14f99a08b6", "1c45f04d9cdc4876969a8a85d9087f18", "84065182993c4cdcbc3e5a37c769447b", "13ef2e644cae4a48ad8ab142e8b6435e", "191a266c1f034c9fa6e7b4138805d60c", "b8f64ba5faab4995a8b6f676b5a8e5d5", "22cb3e6197d34688914565f07d26ecef", "cc2768fdc75942c4a894bcf6006ad11f", "952b7b171ddb4f4eb96cc91f1257ca9a", "660daa560af241f2911156e2b7187c7e", "847fd69ae9924b03929b1a0e59e80f87", "e1cfc5099d1b4cc7bedcc7a53da34453", "9b62c6c326784d799f790398e7fa22e9", "65115b3c64504cc09ed3459b778995bf", "069b286ddde94f77a425f288e6e14d09", "f051d817821744d78a30577f69178e1e", "c0632ad487cf400e906da9a52efa239a", "12183c2f615f435a9cd920100f2820d4", "839eb1025574493ca1cf19f5a2f2010e", "ca5087b11f79495e8d32cfe8cca64f7b", "949ae8d78ed644cebf9b8fd86a6be7fc", "38f2d3b8e5ad41d1af054ef4c2ea47be", "195cc632420d4e85aadf55741dafe166", "e6ee962e992e44cca46b92e93b9f2c37", "b06e9ed8cacb428ca888b8c7e0558e54", "77d14fa2a785407181e4a9da9d40a3d0", "fd4cec7630304a159031e309fdbd715c", "cffb8b1a826b429488fc7d6af390df80", "c58a9370d7484bf8bf50c8e1fef3da12", "8d8c48b368c943aca716058d8a6c7485", "4ff6be2f8fe8422fa6ed67aa4e4bb2cc", "741722e693c344cca086aacee15eb874", "d227738e76a3420ba0a79a6684de0dcf", "0057a0667a434d1e8ac207b121049b2c", "c6316fd570b741628ea906f553c52679", "845c79c672984d0bab35df090dc632e5", "52d654208b6b4001a56c287a822354d3", "c47d7500fff14be39a2184e2d30b3795", "f6273966027146e9b345c02f935e65be", "72d633d88bd6413cad4cd3d2717dc7ca", "b423c474bb5d4ba98c14502c6b02d95d", "75241cec097b4fba877099eede78e3e7", "2cef863fdeb0413a8a3ce9bc68d1f985", "be5f89ff1dbc41b2a8263817d4a59f1f", "9459414718e44c6a9b88760f1fb1b46a", "1c8e2a3fde7942e697e111b07aefe26b", "637168904e464ac9a6782ada14784eb1", "a069be3b5a6d46e083524d14053bcb9f", "71907183c5584961bfd232d68e685d3d", "9e35548cd1714a0c9bf3ccc52d268fb1", "5a131f872f8d4fcb96b713b60e56656b", "7e72190f3fd44d1489badb8d28b6bb29", "82dc472697914b1aaf5866e1573c662c", "dd1af81809254587aa2a3d0deacff0a0", "b61d4544c6a54060a2dc120eee5362f3", "3b096679a00843148f3b874b9c70e901", "2e2eb1edad7748ee92249d555840d612", "8b2752894a8b4a69af0ad6be380c8c23", "09a7ea71de884bcc94aa330697ab728b", "29c1f761f5634be2bf0ff25736fd7d61", "be4983b296534b4ba8991262f81d6c0a", "058111735ae7416fab7f7adb8c981d14", "1e7ed32472134c508325266efe272c40", "d46475c0add24876b714a409bc9933b2", "51757d5291844c6981c0258a245ad5be", "36090e9a0cc94f56a377ff368eca2b64", "cb77a4c364704cdca658427246c1a3fb", "921e0906b18841d4b68b0b72dc911121", "86439582332e43cea3d1eea6062510f2", "edb0b8167858489c8f94c95337647d9f", "92b45c3467f44cdfb4b12fc7d9f25f5a", "1b13814df36f42969c8795c40b351ccf", "d1a11fbd7c3a4dadbdee14b744937c30", "248753435c404995812b93422ca8c062", "2b3ae44914934e31ab2229c3b7d1b9a7", "08c6b30a70024da28e546ff11dce4f94", "ce736cb44cec467e8d6db28528e30780", "d0ea236e247c46c78d6ee489433b76cf", "33e822baba3b4f0bbec237b48deba7f4", "fb8e91ab0228428487907e96827a91b9", "c7fec0dc1e8648ca869c65caefa05751", "acaa2bdd5eef41a49a99c02f384a8173", "2a62b154d02740988eaf6df9ce39cd93", "369e4fbc8a1b4c6d91f502a7fe5b18f4", "f2ff83d1d99d40c7bb9d563ad623e164", "e359213b91d841e3a28cdc293668a104", "86a3bee8f0604c9a922c94e4161fc24b", "f260359ad48e4f4c99e7aec1fc5cd09f" ] }, "id": "bjh0jE0yhE3p", "outputId": "34e09142-f66f-4d32-9052-5e41d3e7d166" }, "source": [ "trainer = CollieTrainer(model=model, max_epochs=10, deterministic=True)\n", "\n", "trainer.fit(model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "----------------------------------------------------\n", "0 | user_biases | ZeroEmbedding | 941 \n", "1 | item_biases | ZeroEmbedding | 1.4 K \n", "2 | user_embeddings | ScaledEmbedding | 28.2 K\n", "3 | item_embeddings | ScaledEmbedding | 43.4 K\n", "4 | dropout | Dropout | 0 \n", "----------------------------------------------------\n", "74.0 K Trainable params\n", "0 Non-trainable params\n", "74.0 K Total params\n", "0.296 Total estimated model params size (MB)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Detected GPU. Setting ``gpus`` to 1.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "62986af6e0394e969bb8942274ba6784", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validation sanity check', layout=Layout…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Global seed set to 22\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\r" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b1ccac573c5c4c79b4357c14f99a08b6", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "952b7b171ddb4f4eb96cc91f1257ca9a", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c0632ad487cf400e906da9a52efa239a", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b06e9ed8cacb428ca888b8c7e0558e54", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n", "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d227738e76a3420ba0a79a6684de0dcf", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b423c474bb5d4ba98c14502c6b02d95d", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "71907183c5584961bfd232d68e685d3d", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2e2eb1edad7748ee92249d555840d612", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "51757d5291844c6981c0258a245ad5be", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d1a11fbd7c3a4dadbdee14b744937c30", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c7fec0dc1e8648ca869c65caefa05751", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "9b69ca9af02e4dcead166064746dfcb2", "1748fb86e5824e739a39350e4777a4eb", "27157458047e4a769ae205d23b8e6221", "2c7ac2735ed649cfafc52ca48eb38d1a", "ed61762550634ce6a906b6da4550f4c7", "b1ab812f3fae427db2b8cef8fe572f83", "08103e370a6047068d6e7d54653c1f3a", "1d603bf9ea684001b97a63be7f446e3b" ] }, "id": "6tvE66cfhE3p", "outputId": "1424a753-c468-48c2-dfc9-3b2118195955" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc], val_interactions, model)\n", "\n", "print(f'Standard MAP@10 Score: {mapk_score}')\n", "print(f'Standard MRR Score: {mrr_score}')\n", "print(f'Standard AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9b69ca9af02e4dcead166064746dfcb2", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "Standard MAP@10 Score: 0.024415062120220127\n", "Standard MRR Score: 0.1551878337645617\n", "Standard AUC Score: 0.8575152364604943\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "CRq29RVfhE3q" }, "source": [ "### Train a ``HybridPretrainedModel`` " ] }, { "cell_type": "markdown", "metadata": { "id": "lFh1LEcChE3q" }, "source": [ "With our trained ``model`` above, we can now use these embeddings and additional side data directly in a hybrid model. The architecture essentially takes our user embedding, item embedding, and item metadata for each user-item interaction, concatenates them, and sends it through a simple feedforward network to output a recommendation score. \n", "\n", "We can initially freeze the user and item embeddings from our previously-trained ``model``, train for a few epochs only optimizing our newly-added linear layers, and then train a model with everything unfrozen at a lower learning rate. We will show this process below. " ] }, { "cell_type": "code", "metadata": { "id": "RPgUTdR1hE3r" }, "source": [ "# we will apply a linear layer to the metadata with ``metadata_layers_dims`` and\n", "# a linear layer to the combined embeddings and metadata data with ``combined_layers_dims``\n", "hybrid_model = HybridPretrainedModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " item_metadata=metadata_df,\n", " trained_model=model,\n", " metadata_layers_dims=[8],\n", " combined_layers_dims=[16],\n", " lr=1e-2,\n", " freeze_embeddings=True,\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 438, "referenced_widgets": [ "1b0f046d63ec46909920000d416a956b", "2010b6b4500e4b139e93077008383a3c", "3a31cc9116d7465bbeea6471c6e93968", "560c2c1f4ca94b73a47aeb344ee81ab1", "cf1581e1b6614caa92b7845465c9de39", "346d17d614ee426fbe2be794a34737f3", "52271d9aa0d64d0dacb8d2d0d827adc0", "eb62d6f5333a4cad90fe00ecab4f4de3", "73d516b143a94e0a9ff2115a975dfe0b", "cabcae1934154c03a2ddbfa47b2b4501", "726b226d57474815ac786d93f4039902", "1a2825490a2043aca84c13d316d11ed2", "293790e04a5a44bb9f14edf7ac0ab8cc", "5ccacb85b1214ee1a0e5665364c235b7", "fb3d481dfea04ea3b6d1f8d57682826c", "044c05d2a4824890a595a2f196d64459", "ad5390af70b14da0aa0d3a0b5d20a90a", "0f93d0dc68984a279b6382fbedda6630", "e20d6099034e4567a726f6a7b038395c", "76f0fc24f91d40e4869be6e1b5b4b0d3", "a0666c429914487faf1fac59402e7364", "d6c7474eb9114cde9ff9adde036f3bc2", "83e1559c03c543c88418b270c16c88f6", "01c61537bfe842649a60c70fde9a51a3", "32b3c6d930f94d56b9f6e652303cc1ca", "27b7e22b72b94b9da9d22190c338ac84", "e39a8caba8854f5696d3ff03277bfaa0", "ad367116739e4f4c8717c5b5b741b2ee", "916dfdca43f74c058da5b45dff1eb621", "564e25c0cc9d4f8bb9a00e8f480eaf7b", "0cfb4e7dc348480d9cae85167df7a73e", "24f32e83468d4a4eb03b91130a526df5", "cf3dcff653a3443184d0942defe61230", "7f75f89935e84173bf5d914bc8f50150", "8c86d4c10827436b90b7f62885c57ef6", "3971fc6570fc4f3ca85df75030b1a984", "b5be05a8a7c045c1bce4eda3852a9256", "5802083d9af4473684eadd22aa17d5c0", "00a30c0b9eac4a38b0bd3be2cc05476b", "dc0817dca939450caa82a751377451d4", "aa23f91202f7432ab937a60fd0cc69c2", "fa12cd0edffc4459aadb255b2f85309c", "cfa25ee4254c479498e117189651d8c4", "7e6c87609e254706b7e6ee6ac7bf78b4", "a50062a20b2a475fb760c5ddf6266197", "82c055fff9914a1d8a6c838062046b99", "c16130a0a9624f848af1b5553264b78d", "1f2cb0a8875946439822315ad3a7e1e5", "22fbff68d90c43d6960fea7ddd420810", "6f02c441db1a45488e41b29c656de74e", "1408cb730c964bd39fd32f56916b9701", "095d3de7d1e1464eb81d927e6edc3b0c", "e5379796148347bc8f6ab506e1f5b1cd", "bb09a545620c4dd1af632339d6fdd04a", "08ee77ef97744d6d92a86674b9247b25", "ef2fb16bb26f4a489fe456a33a9aaabf", "c4588fb79e384aeabc04a175fe8d6548", "d519f414b012498bafdd6ebd8503bbf0", "a571a202b33d4a0dae7428d244d64cf6", "f8841a4feab24ea1ace015833bbf0891", "e0eb2bb0242e44aba710350121762fad", "2cbdcb710a39477494596ee8e941152a", "7ef288048daf401699f47fcf24ae2b2a", "7927e1af2f644102b87c7c13ad22546a", "c560a29789ca48129d338f88feddfeee", "a2dc9ff9d3b6413489e37645e5a81969", "e23fffe8236f4f9ab8b76858b371ed8c", "7a1cdaf584e44e3392d963cd39f65cc0", "0303b557637e45078eb791c8e6091f61", "7025cdb08e7646169e13713fc442b5e3", "c3216b05839d47d781220cc4dc762e46", "41b780dbe1394396a24bb180a7945c3d", "c58dc7499eef43f39a406d873aa9bf2f", "3a7fe9e14ff94e14a3ea9be6a3685299", "65d8dc2103ba4794a94ddbf4e1ef9825", "16a6b8dc6f36458da47ce830fac15d26", "5be8f16d753c47a6a68248c561a9605e", "c85c795ddd734662b1277c3b4dd4bb6f", "37aacb2f901c46c4bc7b9c580890a3c6", "f837d58642d247cf9ffb657002d83ae6", "109fb55f40334538966bbdb2961cfdf9", "1a1fa94f37524d4cb93f5a2bfde53b12", "7978b82dcd874c2782f4821d859ef918", "e4f3bb71ec4645649a3e81bfd8b3e310", "0374e23a55c249bb955a605926214583", "14acab5f63754fd0970ca0a6e204d47e", "720887ecd42045539a4f03a57b8a2d5e", "94f0ec5f8e5e43de8397102e899dde43", "fcbf9f9aecf54c1f97c3e2e3915c36c2", "2b71910b6bf74406a468ef78e914db9f", "7bfe5a6f9f2f4787acb15eedd6e23341", "365f4e59afcb443f8c3cad99137dcd40", "d9068f85980247bc89628635dd221f3c", "c98453288d7e416c8ae6067db3220b5f", "6657854d04e446339437228998a63325", "27a1d1d028474aab8ee3cdc1c00166ad" ] }, "id": "vyyUg5ilhE3r", "outputId": "77ffdf06-0964-4a7d-f491-a83d53d3e210" }, "source": [ "hybrid_trainer = CollieTrainer(model=hybrid_model, max_epochs=10, deterministic=True)\n", "\n", "hybrid_trainer.fit(hybrid_model)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "--------------------------------------------------------------\n", "0 | _trained_model | MatrixFactorizationModel | 74.0 K\n", "1 | embeddings | Sequential | 71.6 K\n", "2 | dropout | Dropout | 0 \n", "3 | metadata_layer_0 | Linear | 160 \n", "4 | combined_layer_0 | Linear | 1.1 K \n", "5 | combined_layer_1 | Linear | 17 \n", "--------------------------------------------------------------\n", "75.3 K Trainable params\n", "71.6 K Non-trainable params\n", "146 K Total params\n", "0.588 Total estimated model params size (MB)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Detected GPU. Setting ``gpus`` to 1.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1b0f046d63ec46909920000d416a956b", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validation sanity check', layout=Layout…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Global seed set to 22\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\r" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "73d516b143a94e0a9ff2115a975dfe0b", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ad5390af70b14da0aa0d3a0b5d20a90a", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "32b3c6d930f94d56b9f6e652303cc1ca", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "cf3dcff653a3443184d0942defe61230", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "aa23f91202f7432ab937a60fd0cc69c2", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "22fbff68d90c43d6960fea7ddd420810", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c4588fb79e384aeabc04a175fe8d6548", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c560a29789ca48129d338f88feddfeee", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "c58dc7499eef43f39a406d873aa9bf2f", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "109fb55f40334538966bbdb2961cfdf9", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fcbf9f9aecf54c1f97c3e2e3915c36c2", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 10: reducing learning rate of group 0 to 1.0000e-03.\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "f3842767c3b9491ca0e39c7d83c3bba7", "42e789d0121c43df87d03eeb4e58d3fe", "bdebaa528fe345f58ed0e85d0015187b", "08455bd7b312416a8828902e10a0a6de", "4a1d091431614ff2a9a94baa79f4ebdd", "78f224d19e054aa3a6787b9c3aa536eb", "b6cff5f6b6004d6999a277ef1ed64fe1", "9c1fd029256144a68b8c996e201c3e08" ] }, "id": "I8eEYwcfhE3s", "outputId": "9babb089-060c-4636-f3dd-ebbf91d939e6" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc], val_interactions, hybrid_model)\n", "\n", "print(f'Hybrid MAP@10 Score: {mapk_score}')\n", "print(f'Hybrid MRR Score: {mrr_score}')\n", "print(f'Hybrid AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f3842767c3b9491ca0e39c7d83c3bba7", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "Hybrid MAP@10 Score: 0.02650305521043056\n", "Hybrid MRR Score: 0.15837650977843062\n", "Hybrid AUC Score: 0.780685132170672\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "EEw83cTUhE3s" }, "source": [ "hybrid_model_unfrozen = HybridPretrainedModel(\n", " train=train_interactions,\n", " val=val_interactions,\n", " item_metadata=metadata_df,\n", " trained_model=model,\n", " metadata_layers_dims=[8],\n", " combined_layers_dims=[16],\n", " lr=1e-4,\n", " freeze_embeddings=False,\n", ")\n", "\n", "hybrid_model.unfreeze_embeddings()\n", "hybrid_model_unfrozen.load_from_hybrid_model(hybrid_model)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 472, "referenced_widgets": [ "b776627648254a919b1a18b4025fbcb3", "2b4a275a432241ec91b5fe2a67fff267", "b9fed3cbac434c5e8e8eccc37fdc7909", "518844d0b3cc4deabe8b0ebb657eac8f", "20a1726ffbf9408b8a00ccd5c44b81a8", "f9e3ccff7f41406eb5ec4f9658b24718", "d354c4d6b73d412f87a41f8cbb7bb9eb", "01f11efae5cc4bc48726f6554c346655", "bede57139cc14b599167088250548b43", "512af7cae0c84552b736431802ed4402", "2a5f13e1cce34787ba032800b8a59ef6", "2dd137e524324334b5f7c56e7d4d8877", "e7f0ad3f8d204c1a9c611a897655abbe", "dbaaabf67d1640d5b988ca6e095577c2", "9814e7c9d49245e7802828d385800a6d", "7efc774340274141a6a6d4c3e5bfcf12", "fe7f069fe1f94fe3a98c44c3cf5c3ed1", "82de05cc4d4f4b32bf7127007ba17bef", "3e5ca461bd1f4d6ba6e3f5d2a5c708b4", "4d565f9c62d84ba487c79a1e2053770d", "7beb2d7bb4b8421fa6a3fdc5a0a578b1", "b56b52a8d05c4c2496e18f32df63ee5b", "55a7ecf02b1f492e82b95e5460e04f22", "7d93dfd6d22843108d76c8a2416bee21", "3b0d7093adde4f64bfddffdb4444e1f2", "aa9c6996aa3c40d18f0b3f0b8cd1705d", "7dc6a6399f604fef8dda1b5a1d7b2920", "c1ab4cda390d4a99b92162421e86718a", "c894ed9774704d88bff3e9d1ad542900", "4968d6ba2303488c9256042f3a7f8206", "27da441d4d55492ebc526ca00dd7d01e", "ad048e4075c847f1b911080e51548cdf", "84cd0ec866694431a1bc3f6eb7686107", "8b4924ef271d4097abd6e57303794327", "200eca6c62424921ab682d2ab8a0785d", "80117a933a694fd9914f88917466a00f", "47c26c18cccf44f4b6a5caf3ccdd4e83", "4fdf8e0a36244ff1bdf34e9060e3f035", "7478d08325084ef28fa9f1c5a6ab18f6", "1b0c6cff674c4930b18748d9fa4f9090", "73232374fc7f4880a87dec552959d3a4", "03dad58a3f004920aff305a2357ad121", "fe02f58536de4d49b132a067fc065671", "7805d39a07414d199a012bad80e90acb", "094a99863ecc425dae15237f990ffb3e", "b91817bbe07449b2a9a8d1b6be2ce378", "82c6bf962cd04ee8bf02d24b03952b28", "3e392b5c457a4ef2b7c883982f60c36b", "5ebee37d75004546b316d10399b69431", "b7fbdb82ca614754a12635170518e0cb", "6cb3f43666734c869fcc960645e129e9", "ad4885e1e6ab41f48ac113c30aa13b39", "87093d6b3c924e6087e9d2b78ba0c6eb", "0a4702427f524e768b8d1b2379e65499", "3c9b57c8d7e24cb181e50d3b5e9a89d4", "1f6a03090e2f4bd5a08b973a2e31a48e", "43ff46cfff674d37a04bf926feca9048", "7fb80bf0ab9549989de36323648126cd", "34c2e6b19152468e8e8bbdcbf1e7d87e", "521b635587644d588e28f0efba61aea2", "6408b9869970482d949d6a794700716b", "61a742fb4fde49fb9e4026e330cf2159", "05bdc0f323064d359a32a0b8d345dd78", "424e11f8cff442ee8a7643e56ffb36af", "2ac6cf2e1f304f06bdc354d04507fecd", "44ac79c4dfec4caa9bd2e4f987aeca9d", "ebe6a36807834fb38ff46654e27075c1", "ac257e025a9545d5b924d2946b422735", "4606964ad6b5447db1d7498178cb5a78", "e823a52deb434d5f81deb90ae34adc5e", "9dc20f131e8642ddae5a90537309d835", "44eea8a841af4ddb92d93bfe06c4c6fc", "5a147c6e4b59428ebd7e2e3412cc52fc", "49bf717484f84a25a7ac27b01b2606a3", "c91bb1aa56074b398362c4326c9b9b13", "1aad34c1efeb4af6a9ec6809a12a3569", "7214617bec6645ba891a2071f6bc6442", "ab827148c2884c728fe50dd09ce55912", "d48633090c514ddd9e9fc2baa4bf347d", "6315ef0a9eb14469a4de8063b9e745ca", "afe817a0babe466cbbf8e4b802ef3360", "60f26b29709e462381c221393c45e76b", "cd7d3ce3534447b98fafe4d5aab0194b", "f44f6872eb08434ea56d62708770cd25", "cb38f538344b43928074a1184cd05997", "286f2daf07f745d891110f78c116823c", "c456a6bba5804e8e911a89abf34e5670", "b4b847b559944ce2ba7a4ce34c472120", "975f803a94994f91891c3fb7f187a135", "47d4f8168a3049b09471968aca76fa08", "0af18a7ade304878bdcb8dbca2ef1074", "a521fbb49c5946b1afca37cbc4052b52", "dac2cbf4f3f2477ab18adce6db8e77a8", "5e9949db3b97478a905b23c0a437dd45", "544a63b30c714580be37d69eb8669328", "33fbfb7b8c2d4a5bb0e979c626862b13" ] }, "id": "yiA-EylqhE3t", "outputId": "aefcb665-c88d-43ab-eb7a-322cbc58a262" }, "source": [ "hybrid_trainer_unfrozen = CollieTrainer(model=hybrid_model_unfrozen, max_epochs=10, deterministic=True)\n", "\n", "hybrid_trainer_unfrozen.fit(hybrid_model_unfrozen)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "GPU available: True, used: True\n", "TPU available: False, using: 0 TPU cores\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "\n", " | Name | Type | Params\n", "--------------------------------------------------------------\n", "0 | _trained_model | MatrixFactorizationModel | 74.0 K\n", "1 | embeddings | Sequential | 71.6 K\n", "2 | dropout | Dropout | 0 \n", "3 | metadata_layer_0 | Linear | 160 \n", "4 | combined_layer_0 | Linear | 1.1 K \n", "5 | combined_layer_1 | Linear | 17 \n", "--------------------------------------------------------------\n", "75.3 K Trainable params\n", "71.6 K Non-trainable params\n", "146 K Total params\n", "0.588 Total estimated model params size (MB)\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Detected GPU. Setting ``gpus`` to 1.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "b776627648254a919b1a18b4025fbcb3", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validation sanity check', layout=Layout…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Global seed set to 22\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "\r" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "bede57139cc14b599167088250548b43", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Training', layout=Layout(flex='2'), max…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fe7f069fe1f94fe3a98c44c3cf5c3ed1", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3b0d7093adde4f64bfddffdb4444e1f2", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "84cd0ec866694431a1bc3f6eb7686107", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 3: reducing learning rate of group 0 to 1.0000e-03.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "73232374fc7f4880a87dec552959d3a4", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5ebee37d75004546b316d10399b69431", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "43ff46cfff674d37a04bf926feca9048", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "2ac6cf2e1f304f06bdc354d04507fecd", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 7: reducing learning rate of group 0 to 1.0000e-04.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5a147c6e4b59428ebd7e2e3412cc52fc", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "afe817a0babe466cbbf8e4b802ef3360", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "Epoch 9: reducing learning rate of group 0 to 1.0000e-05.\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "975f803a94994f91891c3fb7f187a135", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Validating', layout=Layout(flex='2'), m…" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "2Txbqf3fbqvD" }, "source": [ "### Evaluate the Model" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 117, "referenced_widgets": [ "0d9f2c4528634f63a26527a755b33773", "6d196fda01ee4b63a7731b169c2f36b7", "20d9caea7c744c09a8efd05793d2c6db", "1760eeca2a084b1c8e57a9989139cdf6", "8ccaa38836fd4fa588723ba2516f635b", "113a8ae2c462494abf3ca97f65e51d06", "c0282a49c81f4dce83322f9734eb0efd", "9848dcb0b0e048c88f9c6243dc5ede33" ] }, "id": "sof4rqMbhE3u", "outputId": "a344f1bb-0627-4e46-968e-f89bc81a5224" }, "source": [ "mapk_score, mrr_score, auc_score = evaluate_in_batches([mapk, mrr, auc],\n", " val_interactions,\n", " hybrid_model_unfrozen)\n", "\n", "print(f'Hybrid Unfrozen MAP@10 Score: {mapk_score}')\n", "print(f'Hybrid Unfrozen MRR Score: {mrr_score}')\n", "print(f'Hybrid Unfrozen AUC Score: {auc_score}')" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0d9f2c4528634f63a26527a755b33773", "version_minor": 0, "version_major": 2 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=28.0), HTML(value='')))" ] }, "metadata": { "tags": [] } }, { "output_type": "stream", "text": [ "\n", "Hybrid Unfrozen MAP@10 Score: 0.02789580198163252\n", "Hybrid Unfrozen MRR Score: 0.17139103232628614\n", "Hybrid Unfrozen AUC Score: 0.8118089364191508\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "0FzTQc6WbtJA" }, "source": [ "### Inference" ] }, { "cell_type": "code", "metadata": { "id": "EwM1pkf_hE3v", "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "outputId": "650e4d07-8566-484c-c7b7-543e7c7428a9" }, "source": [ "user_id = np.random.randint(10, train_interactions.num_users)\n", "\n", "display(\n", " HTML(\n", " get_recommendation_visualizations(\n", " model=hybrid_model_unfrozen,\n", " user_id=user_id,\n", " filter_films=True,\n", " shuffle=True,\n", " detailed=True,\n", " )\n", " )\n", ")" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "

User 895:

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Willy Wonka and the Chocolate Factory (1971)Mighty Aphrodite (1995)Conspiracy Theory (1997)Sense and Sensibility (1995)Liar Liar (1997)In & Out (1997)Return of the Jedi (1983)Ransom (1996)Emma (1996)Toy Story (1995)
Some loved films:
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Jerry Maguire (1996)Bad Boys (1995)Blown Away (1994)Santa Clause, The (1994)Tin Cup (1996)Graduate, The (1967)Cold Comfort Farm (1995)Princess Bride, The (1987)Private Benjamin (1980)True Romance (1993)
Recommended films:
-----

User 895 has rated 12 films with a 4 or 5

User 895 has rated 8 films with a 1, 2, or 3

% of these films rated 5 or 4 appearing in the first 10 recommendations:0.0%

% of these films rated 1, 2, or 3 appearing in the first 10 recommendations: 10.0%

" ], "text/plain": [ "" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "fNwj-u-AhE3w" }, "source": [ "The metrics and results look great, and we should only see a larger difference compared to a standard model as our data becomes more nuanced and complex (such as with MovieLens 10M data). \n", "\n", "If we're happy with this model, we can go ahead and save it for later! " ] }, { "cell_type": "markdown", "metadata": { "id": "xYmFQZEhhE3w" }, "source": [ "### Save and Load a Hybrid Model " ] }, { "cell_type": "code", "metadata": { "id": "2ZDlfmAVhE3w" }, "source": [ "# we can save the model with...\n", "os.makedirs('models', exist_ok=True)\n", "hybrid_model_unfrozen.save_model('models/hybrid_model_unfrozen')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "qW3kPpenhE3x", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "da7c6d72-d7ca-4913-98fc-e85691a771f4" }, "source": [ "# ... and if we wanted to load that model back in, we can do that easily...\n", "hybrid_model_loaded_in = HybridPretrainedModel(load_model_path='models/hybrid_model_unfrozen')\n", "\n", "\n", "hybrid_model_loaded_in" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "HybridPretrainedModel(\n", " (embeddings): Sequential(\n", " (0): ScaledEmbedding(941, 30)\n", " (1): ScaledEmbedding(1447, 30)\n", " )\n", " (dropout): Dropout(p=0.0, inplace=False)\n", " (metadata_layer_0): Linear(in_features=19, out_features=8, bias=True)\n", " (combined_layer_0): Linear(in_features=68, out_features=16, bias=True)\n", " (combined_layer_1): Linear(in_features=16, out_features=1, bias=True)\n", ")" ] }, "metadata": { "tags": [] }, "execution_count": 98 } ] }, { "cell_type": "markdown", "metadata": { "id": "qKn2XvzuSaqU" }, "source": [ "## Yet another Movie Recommender from scratch\n", "> Building and training Item-popularity and MLP model on movielens dataset in pure pytorch." ] }, { "cell_type": "markdown", "metadata": { "id": "GUoQMgjCPmkp" }, "source": [ "### Setup" ] }, { "cell_type": "code", "metadata": { "id": "Q6wvep55K6of" }, "source": [ "import math\n", "import torch\n", "import heapq\n", "import pickle\n", "import argparse\n", "import numpy as np\n", "import pandas as pd\n", "from torch import nn\n", "import seaborn as sns\n", "from time import time\n", "import scipy.sparse as sp\n", "import matplotlib.pyplot as plt\n", "import torch.nn.functional as F\n", "from torch.autograd import Variable\n", "from torch.utils.data import Dataset, DataLoader" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "jz4ocKNnLN4P" }, "source": [ "np.random.seed(7)\n", "torch.manual_seed(0)\n", "\n", "_model = None\n", "_testRatings = None\n", "_testNegatives = None\n", "_topk = None\n", "\n", "use_cuda = torch.cuda.is_available()\n", "device = torch.device(\"cuda:0\" if use_cuda else \"cpu\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "srnZMdMoPh9V" }, "source": [ "### Data Loading" ] }, { "cell_type": "code", "metadata": { "id": "J52BdmTvKvUv" }, "source": [ "!wget https://github.com/HarshdeepGupta/recommender_pytorch/raw/master/Data/movielens.train.rating\n", "!wget https://github.com/HarshdeepGupta/recommender_pytorch/raw/master/Data/movielens.test.rating\n", "!wget https://github.com/HarshdeepGupta/recommender_pytorch/raw/master/Data/u.data" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "csjr7o5wPd7n" }, "source": [ "### Eval Methods" ] }, { "cell_type": "code", "metadata": { "id": "et-6h-pkLLMk" }, "source": [ "def evaluate_model(model, full_dataset: MovieLensDataset, topK: int):\n", " \"\"\"\n", " Evaluate the performance (Hit_Ratio, NDCG) of top-K recommendation\n", " Return: score of each test rating.\n", " \"\"\"\n", " global _model\n", " global _testRatings\n", " global _testNegatives\n", " global _topk\n", " _model = model\n", " _testRatings = full_dataset.testRatings\n", " _testNegatives = full_dataset.testNegatives\n", " _topk = topK\n", "\n", " hits, ndcgs = [], []\n", " for idx in range(len(_testRatings)):\n", " (hr, ndcg) = eval_one_rating(idx, full_dataset)\n", " hits.append(hr)\n", " ndcgs.append(ndcg)\n", " return (hits, ndcgs)\n", "\n", "\n", "def eval_one_rating(idx, full_dataset: MovieLensDataset):\n", " rating = _testRatings[idx]\n", " items = _testNegatives[idx]\n", " u = rating[0]\n", "\n", " gtItem = rating[1]\n", " items.append(gtItem)\n", " # Get prediction scores\n", " map_item_score = {}\n", " users = np.full(len(items), u, dtype='int32')\n", "\n", " feed_dict = {\n", " 'user_id': users,\n", " 'item_id': np.array(items),\n", " }\n", " predictions = _model.predict(feed_dict)\n", " for i in range(len(items)):\n", " item = items[i]\n", " map_item_score[item] = predictions[i]\n", "\n", " # Evaluate top rank list\n", " ranklist = heapq.nlargest(_topk, map_item_score, key=map_item_score.get)\n", " hr = getHitRatio(ranklist, gtItem)\n", " ndcg = getNDCG(ranklist, gtItem)\n", " return (hr, ndcg)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "SvpXLWBpPYKk" }, "source": [ "### Eval Metrics" ] }, { "cell_type": "code", "metadata": { "id": "vvU46669Mnmz" }, "source": [ "def getHitRatio(ranklist, gtItem):\n", " for item in ranklist:\n", " if item == gtItem:\n", " return 1\n", " return 0\n", "\n", "\n", "def getNDCG(ranklist, gtItem):\n", " for i in range(len(ranklist)):\n", " item = ranklist[i]\n", " if item == gtItem:\n", " return math.log(2) / math.log(i+2)\n", " return 0" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Rv_-b2rnPQXI" }, "source": [ "### Pytorch Dataset" ] }, { "cell_type": "code", "metadata": { "id": "grg5RywRK1H8" }, "source": [ "class MovieLensDataset(Dataset):\n", " 'Characterizes the dataset for PyTorch, and feeds the (user,item) pairs for training'\n", "\n", " def __init__(self, file_name, num_negatives_train=5, num_negatives_test=100):\n", " 'Load the datasets from disk, and store them in appropriate structures'\n", "\n", " self.trainMatrix = self.load_rating_file_as_matrix(\n", " file_name + \".train.rating\")\n", " self.num_users, self.num_items = self.trainMatrix.shape\n", " # make training set with negative sampling\n", " self.user_input, self.item_input, self.ratings = self.get_train_instances(\n", " self.trainMatrix, num_negatives_train)\n", " # make testing set with negative sampling\n", " self.testRatings = self.load_rating_file_as_list(\n", " file_name + \".test.rating\")\n", " self.testNegatives = self.create_negative_file(\n", " num_samples=num_negatives_test)\n", " assert len(self.testRatings) == len(self.testNegatives)\n", "\n", " def __len__(self):\n", " 'Denotes the total number of rating in test set'\n", " return len(self.user_input)\n", "\n", " def __getitem__(self, index):\n", " 'Generates one sample of data'\n", "\n", " # get the train data\n", " user_id = self.user_input[index]\n", " item_id = self.item_input[index]\n", " rating = self.ratings[index]\n", "\n", " return {'user_id': user_id,\n", " 'item_id': item_id,\n", " 'rating': rating}\n", "\n", " def get_train_instances(self, train, num_negatives):\n", " user_input, item_input, ratings = [], [], []\n", " num_users, num_items = train.shape\n", " for (u, i) in train.keys():\n", " # positive instance\n", " user_input.append(u)\n", " item_input.append(i)\n", " ratings.append(1)\n", " # negative instances\n", " for _ in range(num_negatives):\n", " j = np.random.randint(1, num_items)\n", " # while train.has_key((u, j)):\n", " while (u, j) in train:\n", " j = np.random.randint(1, num_items)\n", " user_input.append(u)\n", " item_input.append(j)\n", " ratings.append(0)\n", " return user_input, item_input, ratings\n", "\n", " def load_rating_file_as_list(self, filename):\n", " ratingList = []\n", " with open(filename, \"r\") as f:\n", " line = f.readline()\n", " while line != None and line != \"\":\n", " arr = line.split(\"\\t\")\n", " user, item = int(arr[0]), int(arr[1])\n", " ratingList.append([user, item])\n", " line = f.readline()\n", " return ratingList\n", "\n", " def create_negative_file(self, num_samples=100):\n", " negativeList = []\n", " for user_item_pair in self.testRatings:\n", " user = user_item_pair[0]\n", " item = user_item_pair[1]\n", " negatives = []\n", " for t in range(num_samples):\n", " j = np.random.randint(1, self.num_items)\n", " while (user, j) in self.trainMatrix or j == item:\n", " j = np.random.randint(1, self.num_items)\n", " negatives.append(j)\n", " negativeList.append(negatives)\n", " return negativeList\n", "\n", " def load_rating_file_as_matrix(self, filename):\n", " '''\n", " Read .rating file and Return dok matrix.\n", " The first line of .rating file is: num_users\\t num_items\n", " '''\n", " # Get number of users and items\n", " num_users, num_items = 0, 0\n", " with open(filename, \"r\") as f:\n", " line = f.readline()\n", " while line != None and line != \"\":\n", " arr = line.split(\"\\t\")\n", " u, i = int(arr[0]), int(arr[1])\n", " num_users = max(num_users, u)\n", " num_items = max(num_items, i)\n", " line = f.readline()\n", " # Construct matrix\n", " mat = sp.dok_matrix((num_users+1, num_items+1), dtype=np.float32)\n", " with open(filename, \"r\") as f:\n", " line = f.readline()\n", " while line != None and line != \"\":\n", " arr = line.split(\"\\t\")\n", " user, item, rating = int(arr[0]), int(arr[1]), float(arr[2])\n", " if (rating > 0):\n", " mat[user, item] = 1.0\n", " line = f.readline()\n", " return mat" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "M_CJ2wlKPS-w" }, "source": [ "### Utils" ] }, { "cell_type": "code", "metadata": { "id": "YhQs5tfvK9Pf" }, "source": [ "def train_one_epoch(model, data_loader, loss_fn, optimizer, epoch_no, device, verbose = 1):\n", " 'trains the model for one epoch and returns the loss'\n", " print(\"Epoch = {}\".format(epoch_no))\n", " # Training\n", " # get user, item and rating data\n", " t1 = time()\n", " epoch_loss = []\n", " # put the model in train mode before training\n", " model.train()\n", " # transfer the data to GPU\n", " for feed_dict in data_loader:\n", " for key in feed_dict:\n", " if type(feed_dict[key]) != type(None):\n", " feed_dict[key] = feed_dict[key].to(dtype = torch.long, device = device)\n", " # get the predictions\n", " prediction = model(feed_dict)\n", " # print(prediction.shape)\n", " # get the actual targets\n", " rating = feed_dict['rating']\n", " \n", " \n", " # convert to float and change dim from [batch_size] to [batch_size,1]\n", " rating = rating.float().view(prediction.size()) \n", " loss = loss_fn(prediction, rating)\n", " # clear the gradients\n", " optimizer.zero_grad()\n", " # backpropagate\n", " loss.backward()\n", " # update weights\n", " optimizer.step()\n", " # accumulate the loss for monitoring\n", " epoch_loss.append(loss.item())\n", " epoch_loss = np.mean(epoch_loss)\n", " if verbose:\n", " print(\"Epoch completed {:.1f} s\".format(time() - t1))\n", " print(\"Train Loss: {}\".format(epoch_loss))\n", " return epoch_loss\n", " \n", "\n", "def test(model, full_dataset : MovieLensDataset, topK):\n", " 'Test the HR and NDCG for the model @topK'\n", " # put the model in eval mode before testing\n", " if hasattr(model,'eval'):\n", " # print(\"Putting the model in eval mode\")\n", " model.eval()\n", " t1 = time()\n", " (hits, ndcgs) = evaluate_model(model, full_dataset, topK)\n", " hr, ndcg = np.array(hits).mean(), np.array(ndcgs).mean()\n", " print('Eval: HR = %.4f, NDCG = %.4f [%.1f s]' % (hr, ndcg, time()-t1))\n", " return hr, ndcg\n", " \n", "\n", "def plot_statistics(hr_list, ndcg_list, loss_list, model_alias, path):\n", " 'plots and saves the figures to a local directory'\n", " plt.figure()\n", " hr = np.vstack([np.arange(len(hr_list)),np.array(hr_list)]).T\n", " ndcg = np.vstack([np.arange(len(ndcg_list)),np.array(ndcg_list)]).T\n", " loss = np.vstack([np.arange(len(loss_list)),np.array(loss_list)]).T\n", " plt.plot(hr[:,0], hr[:,1],linestyle='-', marker='o', label = \"HR\")\n", " plt.plot(ndcg[:,0], ndcg[:,1],linestyle='-', marker='v', label = \"NDCG\")\n", " plt.plot(loss[:,0], loss[:,1],linestyle='-', marker='s', label = \"Loss\")\n", "\n", " plt.xlabel(\"Epochs\")\n", " plt.ylabel(\"Value\")\n", " plt.legend()\n", " plt.savefig(path+model_alias+\".jpg\")\n", " return\n", "\n", "\n", "def get_items_interacted(user_id, interaction_df):\n", " # returns a set of items the user has interacted with\n", " userid_mask = interaction_df['userid'] == user_id\n", " interacted_items = interaction_df.loc[userid_mask].courseid\n", " return set(interacted_items if type(interacted_items) == pd.Series else [interacted_items])\n", "\n", "\n", "def save_to_csv(df,path, header = False, index = False, sep = '\\t', verbose = False):\n", " if verbose:\n", " print(\"Saving df to path: {}\".format(path))\n", " print(\"Columns in df are: {}\".format(df.columns.tolist()))\n", "\n", " df.to_csv(path, header = header, index = index, sep = sep)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "qEl945qyM1FB" }, "source": [ "### Item Popularity Model" ] }, { "cell_type": "code", "metadata": { "id": "dzjxYN-mM3uv" }, "source": [ "def parse_args():\n", " parser = argparse.ArgumentParser(description=\"Run ItemPop\")\n", " parser.add_argument('--path', nargs='?', default='/content/',\n", " help='Input data path.')\n", " parser.add_argument('--dataset', nargs='?', default='movielens',\n", " help='Choose a dataset.')\n", " parser.add_argument('--num_neg_test', type=int, default=100,\n", " help='Number of negative instances to pair with a positive instance while testing')\n", " \n", " return parser.parse_args(args={})" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "xAr3XZ4sM73x" }, "source": [ "class ItemPop():\n", " def __init__(self, train_interaction_matrix: sp.dok_matrix):\n", " \"\"\"\n", " Simple popularity based recommender system\n", " \"\"\"\n", " self.__alias__ = \"Item Popularity without metadata\"\n", " # Sum the occurences of each item to get is popularity, convert to array and \n", " # lose the extra dimension\n", " self.item_ratings = np.array(train_interaction_matrix.sum(axis=0, dtype=int)).flatten()\n", "\n", " def forward(self):\n", " pass\n", "\n", " def predict(self, feeddict) -> np.array:\n", " # returns the prediction score for each (user,item) pair in the input\n", " items = feeddict['item_id']\n", " output_scores = [self.item_ratings[itemid] for itemid in items]\n", " return np.array(output_scores)\n", "\n", " def get_alias(self):\n", " return self.__alias__" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0EDavlooLWT0", "outputId": "a1f13e86-030e-4530-e2d9-34b0d676570a" }, "source": [ "args = parse_args()\n", "path = args.path\n", "dataset = args.dataset\n", "num_negatives_test = args.num_neg_test\n", "print(\"Model arguments: %s \" %(args))\n", "\n", "topK = 10\n", "\n", "# Load data\n", "\n", "t1 = time()\n", "full_dataset = MovieLensDataset(path + dataset, num_negatives_test=num_negatives_test)\n", "train, testRatings, testNegatives = full_dataset.trainMatrix, full_dataset.testRatings, full_dataset.testNegatives\n", "num_users, num_items = train.shape\n", "print(\"Load data done [%.1f s]. #user=%d, #item=%d, #train=%d, #test=%d\"\n", " % (time()-t1, num_users, num_items, train.nnz, len(testRatings)))\n", "\n", "model = ItemPop(train)\n", "test(model, full_dataset, topK)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Model arguments: Namespace(dataset='movielens', num_neg_test=100, path='/content/') \n", "Load data done [4.3 s]. #user=944, #item=1683, #train=99057, #test=943\n", "Eval: HR = 0.4062, NDCG = 0.2199 [0.1 s]\n" ], "name": "stdout" }, { "output_type": "execute_result", "data": { "text/plain": [ "(0.4061505832449629, 0.21988638109018463)" ] }, "metadata": { "tags": [] }, "execution_count": 20 } ] }, { "cell_type": "markdown", "metadata": { "id": "IaJQ8h8kNWmr" }, "source": [ "### MLP Model" ] }, { "cell_type": "code", "metadata": { "id": "F_GYre42NhDX" }, "source": [ "def parse_args():\n", " parser = argparse.ArgumentParser(description=\"Run MLP.\")\n", " parser.add_argument('--path', nargs='?', default='/content/',\n", " help='Input data path.')\n", " parser.add_argument('--dataset', nargs='?', default='movielens',\n", " help='Choose a dataset.')\n", " parser.add_argument('--epochs', type=int, default=30,\n", " help='Number of epochs.')\n", " parser.add_argument('--batch_size', type=int, default=256,\n", " help='Batch size.')\n", " parser.add_argument('--layers', nargs='?', default='[16,32,16,8]',\n", " help=\"Size of each layer. Note that the first layer is the concatenation of user and item embeddings. So layers[0]/2 is the embedding size.\")\n", " parser.add_argument('--weight_decay', type=float, default=0.00001,\n", " help=\"Regularization for each layer\")\n", " parser.add_argument('--num_neg_train', type=int, default=4,\n", " help='Number of negative instances to pair with a positive instance while training')\n", " parser.add_argument('--num_neg_test', type=int, default=100,\n", " help='Number of negative instances to pair with a positive instance while testing')\n", " parser.add_argument('--lr', type=float, default=0.001,\n", " help='Learning rate.')\n", " parser.add_argument('--dropout', type=float, default=0,\n", " help='Add dropout layer after each dense layer, with p = dropout_prob')\n", " parser.add_argument('--learner', nargs='?', default='adam',\n", " help='Specify an optimizer: adagrad, adam, rmsprop, sgd')\n", " parser.add_argument('--verbose', type=int, default=1,\n", " help='Show performance per X iterations')\n", " parser.add_argument('--out', type=int, default=1,\n", " help='Whether to save the trained model.')\n", " return parser.parse_args(args={})" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "BJVmkqWGNoXM" }, "source": [ "class MLP(nn.Module):\n", "\n", " def __init__(self, n_users, n_items, layers=[16, 8], dropout=False):\n", " \"\"\"\n", " Simple Feedforward network with Embeddings for users and items\n", " \"\"\"\n", " super().__init__()\n", " assert (layers[0] % 2 == 0), \"layers[0] must be an even number\"\n", " self.__alias__ = \"MLP {}\".format(layers)\n", " self.__dropout__ = dropout\n", "\n", " # user and item embedding layers\n", " embedding_dim = int(layers[0]/2)\n", " self.user_embedding = torch.nn.Embedding(n_users, embedding_dim)\n", " self.item_embedding = torch.nn.Embedding(n_items, embedding_dim)\n", "\n", " # list of weight matrices\n", " self.fc_layers = torch.nn.ModuleList()\n", " # hidden dense layers\n", " for _, (in_size, out_size) in enumerate(zip(layers[:-1], layers[1:])):\n", " self.fc_layers.append(torch.nn.Linear(in_size, out_size))\n", " # final prediction layer\n", " self.output_layer = torch.nn.Linear(layers[-1], 1)\n", "\n", " def forward(self, feed_dict):\n", " users = feed_dict['user_id']\n", " items = feed_dict['item_id']\n", " user_embedding = self.user_embedding(users)\n", " item_embedding = self.item_embedding(items)\n", " # concatenate user and item embeddings to form input\n", " x = torch.cat([user_embedding, item_embedding], 1)\n", " for idx, _ in enumerate(range(len(self.fc_layers))):\n", " x = self.fc_layers[idx](x)\n", " x = F.relu(x)\n", " x = F.dropout(x, p=self.__dropout__, training=self.training)\n", " logit = self.output_layer(x)\n", " rating = torch.sigmoid(logit)\n", " return rating\n", "\n", " def predict(self, feed_dict):\n", " # return the score, inputs and outputs are numpy arrays\n", " for key in feed_dict:\n", " if type(feed_dict[key]) != type(None):\n", " feed_dict[key] = torch.from_numpy(\n", " feed_dict[key]).to(dtype=torch.long, device=device)\n", " output_scores = self.forward(feed_dict)\n", " return output_scores.cpu().detach().numpy()\n", "\n", " def get_alias(self):\n", " return self.__alias__" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "lA9s7rv0LspV", "outputId": "57d64091-be5b-493d-d20a-5f8d991e69ef" }, "source": [ "print(\"Device available: {}\".format(device))\n", "\n", "args = parse_args()\n", "path = args.path\n", "dataset = args.dataset\n", "layers = eval(args.layers)\n", "weight_decay = args.weight_decay\n", "num_negatives_train = args.num_neg_train\n", "num_negatives_test = args.num_neg_test\n", "dropout = args.dropout\n", "learner = args.learner\n", "learning_rate = args.lr\n", "batch_size = args.batch_size\n", "epochs = args.epochs\n", "verbose = args.verbose\n", "\n", "topK = 10\n", "print(\"MLP arguments: %s \" % (args))\n", "model_out_file = '%s_MLP_%s_%d.h5' %(args.dataset, args.layers, time())\n", "\n", "# Load data\n", "\n", "t1 = time()\n", "full_dataset = MovieLensDataset(\n", " path + dataset, num_negatives_train=num_negatives_train, num_negatives_test=num_negatives_test)\n", "train, testRatings, testNegatives = full_dataset.trainMatrix, full_dataset.testRatings, full_dataset.testNegatives\n", "num_users, num_items = train.shape\n", "print(\"Load data done [%.1f s]. #user=%d, #item=%d, #train=%d, #test=%d\"\n", " % (time()-t1, num_users, num_items, train.nnz, len(testRatings)))\n", "\n", "training_data_generator = DataLoader(\n", " full_dataset, batch_size=batch_size, shuffle=True, num_workers=0)\n", "\n", "# Build model\n", "model = MLP(num_users, num_items, layers=layers, dropout=dropout)\n", "# Transfer the model to GPU, if one is available\n", "model.to(device)\n", "if verbose:\n", " print(model)\n", "\n", "loss_fn = torch.nn.BCELoss()\n", "# Use Adam optimizer\n", "optimizer = torch.optim.Adam(model.parameters(), weight_decay=weight_decay)\n", "\n", "# Record performance\n", "hr_list = []\n", "ndcg_list = []\n", "BCE_loss_list = []\n", "\n", "# Check Init performance\n", "hr, ndcg = test(model, full_dataset, topK)\n", "hr_list.append(hr)\n", "ndcg_list.append(ndcg)\n", "BCE_loss_list.append(1)\n", "\n", "# do the epochs now\n", "\n", "for epoch in range(epochs):\n", " epoch_loss = train_one_epoch( model, training_data_generator, loss_fn, optimizer, epoch, device)\n", "\n", " if epoch % verbose == 0:\n", " hr, ndcg = test(model, full_dataset, topK)\n", " hr_list.append(hr)\n", " ndcg_list.append(ndcg)\n", " BCE_loss_list.append(epoch_loss)\n", " if hr > max(hr_list):\n", " if args.out > 0:\n", " model.save(model_out_file, overwrite=True)\n", "\n", "print(\"hr for epochs: \", hr_list)\n", "print(\"ndcg for epochs: \", ndcg_list)\n", "print(\"loss for epochs: \", BCE_loss_list)\n", "plot_statistics(hr_list, ndcg_list, BCE_loss_list, model.get_alias(), \"/content\")\n", "with open(\"metrics\", 'wb') as fp:\n", " pickle.dump(hr_list, fp)\n", " pickle.dump(ndcg_list, fp)\n", "\n", "best_iter = np.argmax(np.array(hr_list))\n", "best_hr = hr_list[best_iter]\n", "best_ndcg = ndcg_list[best_iter]\n", "print(\"End. Best Iteration %d: HR = %.4f, NDCG = %.4f. \" %\n", " (best_iter, best_hr, best_ndcg))\n", "if args.out > 0:\n", " print(\"The best MLP model is saved to %s\" %(model_out_file))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Device available: cpu\n", "MLP arguments: Namespace(batch_size=256, dataset='movielens', dropout=0, epochs=30, layers='[16,32,16,8]', learner='adam', lr=0.001, num_neg_test=100, num_neg_train=4, out=1, path='/content/', verbose=1, weight_decay=1e-05) \n", "Load data done [3.8 s]. #user=944, #item=1683, #train=99057, #test=943\n", "MLP(\n", " (user_embedding): Embedding(944, 8)\n", " (item_embedding): Embedding(1683, 8)\n", " (fc_layers): ModuleList(\n", " (0): Linear(in_features=16, out_features=32, bias=True)\n", " (1): Linear(in_features=32, out_features=16, bias=True)\n", " (2): Linear(in_features=16, out_features=8, bias=True)\n", " )\n", " (output_layer): Linear(in_features=8, out_features=1, bias=True)\n", ")\n", "Eval: HR = 0.0848, NDCG = 0.0386 [0.6 s]\n", "Epoch = 0\n", "Epoch completed 5.8 s\n", "Train Loss: 0.4429853802195507\n", "Eval: HR = 0.3945, NDCG = 0.2187 [0.6 s]\n", "Epoch = 1\n", "Epoch completed 5.6 s\n", "Train Loss: 0.3646208482657292\n", "Eval: HR = 0.3818, NDCG = 0.2133 [0.6 s]\n", "Epoch = 2\n", "Epoch completed 5.6 s\n", "Train Loss: 0.35764367812979747\n", "Eval: HR = 0.3924, NDCG = 0.2137 [0.6 s]\n", "Epoch = 3\n", "Epoch completed 5.7 s\n", "Train Loss: 0.35384849094297227\n", "Eval: HR = 0.3796, NDCG = 0.2103 [0.6 s]\n", "Epoch = 4\n", "Epoch completed 5.7 s\n", "Train Loss: 0.35072445729290175\n", "Eval: HR = 0.3818, NDCG = 0.2143 [0.6 s]\n", "Epoch = 5\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3481164647319212\n", "Eval: HR = 0.3881, NDCG = 0.2171 [0.7 s]\n", "Epoch = 6\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3454590990638856\n", "Eval: HR = 0.4157, NDCG = 0.2292 [0.6 s]\n", "Epoch = 7\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3422531268162321\n", "Eval: HR = 0.4231, NDCG = 0.2371 [0.6 s]\n", "Epoch = 8\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3384355346053762\n", "Eval: HR = 0.4443, NDCG = 0.2508 [0.6 s]\n", "Epoch = 9\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3335341374156395\n", "Eval: HR = 0.4677, NDCG = 0.2598 [0.6 s]\n", "Epoch = 10\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3280563016347491\n", "Eval: HR = 0.4719, NDCG = 0.2652 [0.6 s]\n", "Epoch = 11\n", "Epoch completed 5.7 s\n", "Train Loss: 0.3223747977760719\n", "Eval: HR = 0.4995, NDCG = 0.2748 [0.6 s]\n", "Epoch = 12\n", "Epoch completed 5.8 s\n", "Train Loss: 0.3164166678753934\n", "Eval: HR = 0.5090, NDCG = 0.2817 [0.6 s]\n", "Epoch = 13\n", "Epoch completed 5.7 s\n", "Train Loss: 0.31102338709726507\n", "Eval: HR = 0.5143, NDCG = 0.2829 [0.6 s]\n", "Epoch = 14\n", "Epoch completed 5.7 s\n", "Train Loss: 0.30582732322604156\n", "Eval: HR = 0.5175, NDCG = 0.2908 [0.6 s]\n", "Epoch = 15\n", "Epoch completed 5.6 s\n", "Train Loss: 0.3016319169092548\n", "Eval: HR = 0.5429, NDCG = 0.2963 [0.6 s]\n", "Epoch = 16\n", "Epoch completed 5.7 s\n", "Train Loss: 0.2980319341254789\n", "Eval: HR = 0.5493, NDCG = 0.2978 [0.6 s]\n", "Epoch = 17\n", "Epoch completed 5.7 s\n", "Train Loss: 0.29476294266469105\n", "Eval: HR = 0.5504, NDCG = 0.3014 [0.6 s]\n", "Epoch = 18\n", "Epoch completed 5.6 s\n", "Train Loss: 0.2921119521985682\n", "Eval: HR = 0.5589, NDCG = 0.3108 [0.6 s]\n", "Epoch = 19\n", "Epoch completed 5.8 s\n", "Train Loss: 0.28990745035406845\n", "Eval: HR = 0.5620, NDCG = 0.3092 [0.6 s]\n", "Epoch = 20\n", "Epoch completed 5.7 s\n", "Train Loss: 0.2876521824250234\n", "Eval: HR = 0.5514, NDCG = 0.3097 [0.6 s]\n", "Epoch = 21\n", "Epoch completed 5.6 s\n", "Train Loss: 0.2858751243245078\n", "Eval: HR = 0.5578, NDCG = 0.3122 [0.6 s]\n", "Epoch = 22\n", "Epoch completed 5.6 s\n", "Train Loss: 0.2843063232125546\n", "Eval: HR = 0.5567, NDCG = 0.3043 [0.6 s]\n", "Epoch = 23\n", "Epoch completed 5.6 s\n", "Train Loss: 0.28271066885277896\n", "Eval: HR = 0.5663, NDCG = 0.3141 [0.6 s]\n", "Epoch = 24\n", "Epoch completed 5.6 s\n", "Train Loss: 0.2813221255630178\n", "Eval: HR = 0.5610, NDCG = 0.3070 [0.6 s]\n", "Epoch = 25\n", "Epoch completed 5.7 s\n", "Train Loss: 0.28002421261420235\n", "Eval: HR = 0.5610, NDCG = 0.3110 [0.6 s]\n", "Epoch = 26\n", "Epoch completed 5.9 s\n", "Train Loss: 0.27882074906998516\n", "Eval: HR = 0.5610, NDCG = 0.3095 [0.6 s]\n", "Epoch = 27\n", "Epoch completed 5.8 s\n", "Train Loss: 0.27783915350449484\n", "Eval: HR = 0.5663, NDCG = 0.3115 [0.6 s]\n", "Epoch = 28\n", "Epoch completed 5.7 s\n", "Train Loss: 0.2768868865122783\n", "Eval: HR = 0.5631, NDCG = 0.3109 [0.6 s]\n", "Epoch = 29\n", "Epoch completed 5.8 s\n", "Train Loss: 0.2760479487343968\n", "Eval: HR = 0.5631, NDCG = 0.3092 [0.6 s]\n", "hr for epochs: [0.08483563096500531, 0.3944856839872747, 0.38176033934252385, 0.39236479321314954, 0.3796394485683987, 0.38176033934252385, 0.38812301166489926, 0.41569459172852596, 0.42311770943796395, 0.4443266171792153, 0.4676564156945917, 0.471898197242842, 0.49946977730646874, 0.5090137857900318, 0.5143160127253447, 0.5174973488865323, 0.542948038176034, 0.5493107104984093, 0.5503711558854719, 0.5588547189819725, 0.5620360551431601, 0.5514316012725344, 0.5577942735949099, 0.5567338282078473, 0.5662778366914104, 0.5609756097560976, 0.5609756097560976, 0.5609756097560976, 0.5662778366914104, 0.5630965005302226, 0.5630965005302226]\n", "ndcg for epochs: [0.03855482836637224, 0.2186689741068423, 0.21325592738572174, 0.21374918741658008, 0.21033736603276898, 0.21431768576892837, 0.21714573069782853, 0.2292039485312514, 0.23708514689275148, 0.2507826695009706, 0.2598176007060155, 0.2652029648171546, 0.2747717153150814, 0.2817258947342069, 0.28289172403583096, 0.2907608027818361, 0.29626902860751664, 0.29775495439534627, 0.3014327139896777, 0.31075028453364517, 0.30917060839326094, 0.3096903348455541, 0.31217614966561463, 0.3043410687051171, 0.314059797472155, 0.3070033682048637, 0.31104383409268926, 0.3094572048871119, 0.3115140344405953, 0.31090220293994014, 0.3092050624323008]\n", "loss for epochs: [1, 0.4429853802195507, 0.3646208482657292, 0.35764367812979747, 0.35384849094297227, 0.35072445729290175, 0.3481164647319212, 0.3454590990638856, 0.3422531268162321, 0.3384355346053762, 0.3335341374156395, 0.3280563016347491, 0.3223747977760719, 0.3164166678753934, 0.31102338709726507, 0.30582732322604156, 0.3016319169092548, 0.2980319341254789, 0.29476294266469105, 0.2921119521985682, 0.28990745035406845, 0.2876521824250234, 0.2858751243245078, 0.2843063232125546, 0.28271066885277896, 0.2813221255630178, 0.28002421261420235, 0.27882074906998516, 0.27783915350449484, 0.2768868865122783, 0.2760479487343968]\n", "End. Best Iteration 24: HR = 0.5663, NDCG = 0.3141. \n", "The best MLP model is saved to movielens_MLP_[16,32,16,8]_1626069383.h5\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU1fnA8e87SzKTkIUsEAggKKAiu5FWUVFaXKq11FoU7aK1om3pz4ob1mrRat1aF6zVutO6rxQrVlsBl7ogCMjmkoCyCCQkJCH7Muf3x50kk2RmMgkzmST3/TwPT2bunNx7bkbPe8+5575HjDEopZSyL0e8K6CUUiq+NBAopZTNaSBQSimb00CglFI2p4FAKaVszhXvCnRWVlaWGT58eLyroZRSvcrq1av3GmOyg33W6wLB8OHDWbVqVbyroZRSvYqIfBXqMx0aUkopm9NAoJRSNqeBQCmlbK7X3SNQSqlQ6uvr2bFjBzU1NfGuStx4PB6GDBmC2+2O+Hc0ECil+owdO3aQkpLC8OHDEZF4V6fbGWMoLi5mx44djBgxIuLfi1kgEJFHgdOBQmPM2CCfC3AP8B2gCjjfGPNxtOtxwrMnUFxT3G57pieTFWeviPbhlFJxVFNTY9sgACAiZGZmUlRU1Knfi+U9gseBU8J8fiowyv9vDnB/LCoRLAiE266U6t3sGgSadOX8YxYIjDFvAyVhinwP+LuxfACki8igWNVHKaVUcPGcNZQLbA94v8O/rR0RmSMiq0RkVWe7PEop1Z369evX6v3jjz/O3LlzAViwYAG5ublMnDiRMWPG8PTTT8ejiu30iumjxpgHjTF5xpi87OygT0grpVSnLV6zk6m3LmPE/FeZeusyFq/ZGfNjXnbZZaxdu5Z//vOfXHzxxdTX18f8mB2JZyDYCQwNeD/Ev00ppWJu8ZqdXPPSenaWVmOAnaXVXPPS+m4JBgCjRo0iKSmJffv2dcvxwonn9NElwFwReQb4BlBmjNkV7YNkejJDzhpSSvVdN7yykU1fl4f8fM22Uuoafa22Vdc3ctULn/D0ym1Bf2fM4FR+/90jwh63urqaiRMnNr8vKSnhjDPOaFfu448/ZtSoUQwYMCDs/rpDLKePPg2cAGSJyA7g94AbwBjzALAUa+poPtb00QtiUY+mKaLF1cWc8NwJXDPlGs49/NxYHEop1Yu0DQIdbY+U1+tl7dq1ze8ff/zxVoky77rrLh577DE+//xzXnnllQM6VrTELBAYY2Z38LkBfhWr47eVnpiOQxw6bVQpm+joyn3qrcvYWVrdbntuupdnLz46VtXisssu44orrmDJkiVceOGFFBQU4PF4Yna8SPSKm8XR4HQ4SU9Mp6Qm3IxWpZRdXHnyoXjdzlbbvG4nV558aLcc/4wzziAvL49FixZ1y/HCsU0gAMj0ZlJcrT0CpRTMnJTLLWeOIzfdi2D1BG45cxwzJwWdxR4T119/PXfeeSc+34ENRx0osUZoeo+8vDzT1YVpLnrjIqoaqnjyO09GuVZKqZ5g8+bNHH744fGuRtwF+zuIyGpjTF6w8rbrEZRU69CQUkoFslUgyPBk6M1ipZRqw1aBINOTSXVDNVX1VfGuilJK9Rj2CgRe6yEynTmklFItbBUIMjwZgKagVkqpQLYKBE09Ap1CqpRSLewVCDw6NKSUii0R4fLLL29+/6c//YkFCxYArdNQjxo1ijPPPJNNmzY1l62vr2f+/PmMGjWKyZMnc/TRR/Paa68BUFFRwS9+8QsOOeQQJk+ezJFHHslDDz0UlTrbas3i5qEh7REopR44Fnavb789Zxxc8m6Xd5uYmMhLL73ENddcQ1ZWVrvPm1JMADz77LNMnz6d9evXk52dzXXXXceuXbvYsGEDiYmJ7Nmzh7feeguAn//85xx88MF88cUXOBwOioqKePTRR7tcz0C26hEkOBNISUjRewRKKRgyBZwJrbc5E6ztB8DlcjFnzhzuuuuuDsueffbZnHTSSTz11FNUVVXx0EMPce+995KYmAjAwIEDmTVrFgUFBaxcuZKbbroJh8NqtrOzs7n66qsPqK7NdY7KXnqRTE+mDg0pZQevzQ9+xd+koQ58Da23+Rqs33nstOC/kzMOTr21w0P/6le/Yvz48Vx11VUdlp08eTKffvop+fn5DBs2jNTU1HZlNm7cyIQJE5qDQLTZqkcA/ofKdGhIKeVKgOQBQNNi72K9b9tL6ILU1FR+8pOfsHDhwg7LdiXNz80338zEiRMZPHhwV6rXjv16BN5M8kvz410NpVSsRXDlzv7dcM8EaKgBVyJc/DakDIzK4X/zm98wefJkLrgg/FIra9asIS8vj5EjR7Jt2zbKy8vb9QrGjBnDunXr8Pl8OBwOrr32Wq699tp26yN3le16BDo0pJRqlpIDE88DcVg/oxQEADIyMpg1axaPPPJIyDIvvvgib7zxBrNnzyYpKYkLL7yQSy+9lLq6OgCKiop4/vnnGTlyJHl5efzud7+jsbERgJqami71JoKxXSDI8GZQVltGvS/+C0YrpXqAaVfBsG/CtOjceA10+eWXs3fv3lbb7rrrrubpo0888QTLli0jOzsbgJtuuons7GzGjBnD2LFjOf3005t7Bw8//DDFxcXNQWHGjBncfvvtUamnrdJQAzz32XP84YM/8N+z/svA5OhFf6VU/Gkaaoumoe6A5htSSqnW7BcI/E8X67MESillsW8g0CmkSikF2DEQ6NCQUkq1YrtA4HV58Tg92iNQSik/2wUCESHTm6n3CJRSys92gQD0oTKlVOxE62nf7mS7FBNg5RvaVbkr3tVQSsXRCc+eEHRkINOTyYqzV3R/heLInj0CHRpSyvZCtQGxaBvWrl3LN7/5TcaPH8/3v/999u3bB8DChQsZM2YM48eP55xzzgHgrbfeYuLEiUycOJFJkyaxf//+qNenLdv2CPbV7MNnfDjElrFQqT7vtpW38WnJp1363Qv+HTxR3GEZh3H1lM6novjJT37Cvffey7Rp07j++uu54YYbuPvuu7n11lvZunUriYmJlJaWAtaKZvfddx9Tp06loqICj8fTpXPoDFu2gpneTBpNI2W1ZfGuilKqjysrK6O0tJRp06YB8NOf/pS3334bgPHjx3PeeefxxBNP4HJZ1+VTp05l3rx5LFy4kNLS0ubtsWTLHkHgQ2X9Pf3jXBulVCx0dOU+btG4kJ89dspj0a5OUK+++ipvv/02r7zyCjfffDPr169n/vz5nHbaaSxdupSpU6fy+uuvc9hhh8W0HrbtEYCmmVBKxV5aWhr9+/fnnXfeAeAf//gH06ZNw+fzsX37dk488URuu+02ysrKqKiooKCggHHjxnH11Vdz1FFH8emnXRve6gxb9giaFrHXKaRK2VemJ/ikkaYRg66qqqpiyJAhze/nzZvHokWLuOSSS6iqquLggw/mscceo7GxkR/96EeUlZVhjOH//u//SE9P57rrrmP58uU4HA6OOOIITj311AOqTyRiGghE5BTgHsAJPGyMubXN58OARUC6v8x8Y8zSWNYJNN+QUoqYTRH1+XxBt3/wwQfttr377rvttt17771Rr1NHYjY0JCJO4D7gVGAMMFtExrQp9jvgOWPMJOAc4K+xqk+g1MRUXOLSoSGllCK29wimAPnGmC3GmDrgGeB7bcoYoGlxzjTg6xjWp5lDHGR4MnRoSCmliG0gyAW2B7zf4d8WaAHwIxHZASwFfh1sRyIyR0RWiciqoqKiqFQuw5uhQ0NK9UG9bdXFaOvK+cd71tBs4HFjzBDgO8A/RNo/4WWMedAYk2eMyWta2/NAZXoyNRAo1cd4PB6Ki4ttGwyMMRQXF3f6IbRY3izeCQwNeD/Evy3QhcApAMaY90XEA2QBhTGsF2BNId1atjXWh1FKdaMhQ4awY8cOojVy0Bt5PJ5Ws5YiEctA8BEwSkRGYAWAc4Bz25TZBnwLeFxEDgc8QLd8gxmeDIprrCsHEemOQyqlYsztdjNixIh4V6PXidnQkDGmAZgLvA5sxpodtFFEbhSRM/zFLgcuEpF1wNPA+aab+nSZnkxqG2uprK/sjsMppVSPFdPnCPzPBCxts+36gNebgKmxrEMogUtW9kvoffnDlVIqWuJ9szhump4u1mcJlFJ2Z9tA0JxvSGcOKaVszr6BwNMyNKSUUnZm20CQ7kkHtEeglFK2DQRuh5v0xHS9R6CUsj3bBgKwhod0aEgpZXe2DgSab0gppWweCEItTKGUUnZi70DgzaSkWoeGlFL2ZutAkOHJYH/9fmoba+NdFaWUihtbB4LmZwm0V6CUsjF7BwKvPlSmlFK2DgSab0gppWweCDTfkFJK2TwQaI9AKaVsHgi8Li9JriTtESilbM3WgQCs4SHtESil7EwDgeYbUkrZnO0DQYZH8w0ppezN9oEg06s9AqWUvWkg8Gayr2YfDb6GeFdFKaXiwvaBIMOTgcFQWlsa76oopVRc2D4QNOUb0vsESim70kDQ9HSxTiFVStmUBgKPJp5TStmb7QNBhtefZkKHhpRSNmX7QJDiTsHtcOvQkFLKtmwfCEREl6xUStma7QMB+J8u1h6BUsqmNBBg3TDWewRKKbvSQICmmVBK2ZsGAqyhoZKaEowx8a6KUkp1u5gGAhE5RUQ+E5F8EZkfoswsEdkkIhtF5KlY1ieUTE8m9b56yuvK43F4pZSKK1esdiwiTuA+YAawA/hIRJYYYzYFlBkFXANMNcbsE5EBsapPOE1PF5fUlJCWmBaPKiilVNzEskcwBcg3xmwxxtQBzwDfa1PmIuA+Y8w+AGNMYQzrE1Lz2sV6w1gpZUOxDAS5wPaA9zv82wKNBkaLyP9E5AMROSXYjkRkjoisEpFVRUVFUa+o5htSStlZvG8Wu4BRwAnAbOAhEUlvW8gY86AxJs8Yk5ednR31Smi+IaWUncUyEOwEhga8H+LfFmgHsMQYU2+M2Qp8jhUYulV6YjoOcejQkFLKlmIZCD4CRonICBFJAM4BlrQpsxirN4CIZGENFW2JYZ2CcjqcpCem69CQUsqWYjZryBjTICJzgdcBJ/CoMWajiNwIrDLGLPF/dpKIbAIagSuNMXFpjTXfkFK9w+I1O7nj9c/4urSawelerjz5UGZOanv7MX56ev2CiVkgADDGLAWWttl2fcBrA8zz/4srzTekVM9vxBav2ck1L62nur4RgJ2l1Vzz0nqAHlHPztYv0r93rL+XiAOBiCQZY6qiduQeJtOTySdFn8S7GkrFRCQNSSwa2c40YJGUveP1T5vr16S6vpE7Xv8s6PlEs5GNpNzt/w5ev1te28wJh2aT4nHjdEjz/iL5e3dH8JOO0iqIyDHAw0A/Y8wwEZkAXGyM+WVUatBJeXl5ZtWqVVHf7+0f3c6Ln7/Ih+d9GPV9KxVPbRsSAK/byY3fO4Jpo7Mpq66nvKaeOX9fTXFlXbvfz0338r/509vts7OBpem4t5w5LsKyDq446VAGpXvZsLOMjV+X89bnoaePjxrQj5w0DzmpHsqq61j+WRH1jS3tW4LLwZzjRnDcqJaZh+98UcSD72ylrsHX6XIuh3DU8P4kup3sLqthd3kNpVX1IevXJCXRRarXzZ7yGhp87dtfj9vBtNEtx37r8yJq6n3tygX7XsIRkdXGmLygn0UQCD4EzsKa3TPJv22DMWZsxDWIolgFgofXP8w9H9/DyvNW4nV5o75/pTojGleplbUNfFFYwfmPrqS0uuMGKpyfTR3BxGHpTBqazqovS/jtyxtCNvDGGIor6zj17ncoqqhtt69+iS5+NnU4bqcDt8uByyHcuyyfsjB1dDmEkQP6sa2kiqq6xnafJyc6OXZkFrvLa9ldVs2e8vbHjQWHwJjBqeSkeslJS2TJ2q8pr2loV65/kptfTx9FeU29FXirG3jx4x0h93tYTkrz60937w9aRoCtt54WcV3DBYKIhoaMMdtFJHBT+2+il2t6lqC4upghKUPiXBvVV0VriMYYw8sf7+C3izc0Xy3uLK3miufX8dA7BZTXNLC9pLrD+tw0cyxpXjepXjdXPLcuaMOd4HTw5Idf8ej/tgJW49f2Qra6vpH5L37CX1fks2NfddDGuklFbQMLl+V3WLcmS+ZOZfTAFDxuZ8hexs0zW/cyRsx/lWCXuAI8+fNvNL8/9+HgIwCRljMG/vXr45rf5x2UEbR+v//uEe2+5w+2FLOztP13lJvu5d+/Ob75/dRblwUtNzg9eheskQSC7f7hISMibuBSYHPUatBDBOYb0kCgYiFUA19b38iEYelsL6lme0kVf/7PZ0HHmS97bi3Xvrye+kZDvc9HsM58g8/w2e4KThmbw6wjhzI6J4Xr/7kh6BVybrqXH33zoOb31552eMihnNPGD+Kz3ftZs72U6xZvCHp+NQ0+hmUkM3VkFsMykrh3WT4lIYaa3r36RBp9pvlcTrrrbXaX1QQtO35IyzOmTY1pR8F0cLo3ZON5zMisVvs/0HKBIq0fwJUnHxr0733lyYd2qdyBiCQQXALcg5UeYifwBvCrqNWghwjsESgVC3e8HryBv9p/td8RY2D2lGG4nA4SnBLyqrrRZ/jLuZNbjlHXGFFD0lEjNjY3jbG5aTywoiDklezDP20ZeeiflBDyuCKCyym4nODFyfxTDou4sZs5KbfDm6TRbmQ70xhHUr+mctBx0OhMcOmqDgOBMWYvcF7UjthDab4hFUsllXVBG88m986exNCMJIb293LGX95lZ2nwq+PfnT6m+f2LH++M+lVqNBvZzh430rKRiHYjG6vGuDNBI5bTYyO5WfwYtB9uM8b8LFaVCidWN4vrGus48okjmTtxLhdPuDjq+1f2VFZdzyPvbOGRd7dSGWLcvO3sj0hn23RmVk609fTnDVR7B3qz+F8Brz3A94Gvo1GxniTBmUCKO0UTz6lOC9YozhgzkMff+5K/vWXduD1t3CDG5aZyz5v5UbuS7o4hg1BifYWqulckQ0MvBr4XkaeBd2NWozjK9Gbq0JBq1tUZPle+sI4Ep4PKuka+ffgALpsxmiMGWwse5aR5ozZE05lySoXTlRQTo4C4rCQWaxmeDL1ZbAPRmMJpjKGitoFbXtvc7gZwfaPBIYaXf3kMk4b1b/WZNtyqJ+owEIjIfqx7BOL/uRu4Osb1iotMbyYFpQXxroaKoXAN/Cljc9hTXsOushpueGVj0Bk+lz+/jgWvbKS8ur7dXPpAdQ2+dkFAqZ4qkqGhlI7K9BUZngxW1qyMdzVUF0VypX9biFww855by2+e7fgYjT7Dd8cPJtXrIs3r5q/LC4I+tRvNh32UirWQgUBEJof6DMAY83H0qxNfmd5MymrLqPfV43a4410d1QnBrvTnv/gJm3aVkeZNYOPXZWzYWc6uIA8tgfWk7BUnjSYnzUtOqod5z62lcH/wh7D+MLMlu8qAFE/MH/ZRKtbC9Qj+HOYzA0Se7aiXaHqobF/NPgYk9cnbIH1WsIe1ahp8PPi2lRbhoMwkxuamUlpVFzQXTG66l7nTWxbH++13gj9leyBz5ZXqqUIGAmPMid1ZkZ4g8OliDQS9hzEm5MNaAqxbcBKpHquHF2ru/YE+DKUNv+rNIpo1JCJjgTFYzxEAYIz5e6wqFS/6dHHvs/HrMm5+NXTqq8Hp3uYgANrAKxVMJLOGfo+1rvAYrNXGTsV6jqDvBQJPS+I51bPtLqvhT298xosf7yDN6+bMSYNZumF3q7ztB5oLRim7iKRHcBYwAVhjjLlARAYCT8S2WvGR4c0ANPFcTxM4GygnzcP43DTe/mIvDT4fPz92BHNPHEVakpvjR2vaA6W6IpJAUGOM8YlIg4ikAoXA0BjXKy6SXEl4nB4NBN2kKw927Sqz5vlPHJLGwtmTGZaZ1FxWr/SV6ppw00fvA54GVopIOvAQsBqoAN7vnup1LxEh05upQ0PdINh0z6te+IT3t+xlWEYyxRV1FFfW8u8Nu6ltaL9MX1FFXasgoJTqunA9gs+BO4DBQCVWUJgBpBpj+uwq7xmeDL1Z3A1ufa39g111jT6e/chavi85wUlmv8SgQQDg6zApnZVSnRNu+ug9wD0ichBwDvAo4AWeFpFqY8wX3VTHbpXpyWRX5a54V6PP2rq3kr+9VcDu8uAPdgmw6cZT8CY4ge5Zpk8pu4skxcRXwG3AbSIyCSsgXA84Y1y3uMj0ZrKxeGO8q9GrBRv7HzmgH/e/VcBr63fhcjpITnAGzc8/ON3bHASge5bpU8ruIpk+6sKaMnoO8C1gBbAgprWKgxOePaHVkNC4ReMAq4ew4uwVcapV7xNs7H/ec2vxGeiX6GLO8Yfws2OH815+sT65q1QPEe5m8QxgNvAdYCXwDDDHGFPZTXXrVqHuC+j9gs4JlurBZyDV4+Kdq6eT5rUe7tIHu5TqOcL1CK4BngIuN8bs66b6qF5sf019yFQP+2samoNAE23gleoZwt0s7nNJ5VRsbN1byaL3vuSF1TtCltGbu0r1XF1ZoUzZUNsbwFfMGE1GSiKP/28ryz8rwu0UTh8/mBFZSdy/Yove3FWqF9FAYHNdXbZx3vPrMEBWv0R+8+1RnPuNYQxIsXISDstI1pu7SvUiGgj8Mj3BF65PcCTEoTbdI9yyjaeOy2FbcRVb9lZy/T83tLsBbID+SW7emz+dBJej1Wc69q9U76KBwC/YFNGH1z/MPR/fw/JtyzlxWN9bniHYDJ+mdXmbpnyGU1pV3y4IKKV6n5j+Xywip4jIZyKSLyLzw5T7gYgYEcmLZX0666dH/JRR/Udx84c3U1nf92bNhkrT0OgzzJ0+irvPnsiSuVMZlOYJWk5vACvVN8QsEIiIE7gP62G0McBsERkTpFwKcCnwYazq0lVuh5vfH/17CqsKuXfNvfGuTtT4fIaX1+zAIcE/z033Mm/GaGZOymX8kHSuPuUwvO7WD5LrDWCl+o5YDg1NAfKNMVsAROQZ4HvApjbl/oCVwuLKGNYlIsFvnE5g1qGzeGrzU5x+8OmMzRrb8Y463Gf8xs9Xbi3hplc38cmOMoakeyisqKOuIfxiLvp0r1J9WywDQS6wPeD9DuAbgQVEZDIw1BjzqojENRCEu3F66eRLWb5tOTe8fwNPn/Y0Lkdkf7Zw+4x1I9o2AJ1/zHBWfVXC6xv3MCjNw52zJjBzYi5L1n2tT/cqZXNxu1ksIg7gTuD8CMrOAeYADBs2LCb1CXXj9KZXN3HK2Olc841ruGzFZTyx6QnOH3t+h1f6xhhueW1z0H3e8fpnXW5Uuzrd8+alm0lwClecNJoLjz24ObGbNvBKqVgGgp20XslsiH9bkxRgLLBCRABygCUicoYxZlXgjowxDwIPAuTl5XUwl6VrQt043VtRx4Qb3uDI4ekMTz2Ke9f8hcb9R/CnpcWtGtr5L33C53v2k5zoYu32UtZuL6Vof23Qfe4sreb5Vds5ZmQWuf4brl1t4Jt6GN+dMJg95TVsL6liwSsb2wUggIzkROZOH9W1P5BSqs8SY2LSrjZlLf0cK2PpTuAj4FxjTNAczyKyAriibRBoKy8vz6xaFbZIl4TKe5+ZnMDMSbm8V1DMp0XbSD74Thqrh1O9/QKs7PntjchKZuLQdJZ9WkhZdX27zx1C89TMgzKTGJzmYdVX+6hvbPkuPG4HV518KMeNyqaqrpGqukbmPvUxxZV17fbndAgOodXvByPA1ltPC1tGKdU3ichqY0zQmZkx6xEYYxpEZC7wOtbaBY8aYzaKyI3AKmPMklgduytC5b2/7vQxzVfmxRW1/PnDMl7ZcT+u1E9oKJ/Qbj9rr59BepL1EFrbK/imff5x5lgOz03lvfxi3iso5s3Ne2jbhNfU+7jxX5uBzR3WvdFnuGjaIQzN8DK0fxJXvrCOPeXteyM63VMpFUzMegSxEqseAVgNd9ODVLkhhmcafY1M/PtkkPZLKEpjCp/87L12++xoyGfE/FfbBYIm986eRFKCE2+Ck0ufXktRRfsGPjfdy//mt+QIDBWAbjlznN4PUMqm4tIj6I2OG5WFz8DvTjucnx93cNAyToczaBAAMM79rd43L3YzCPoNgnLguk/g7s9bL3YzON0bdFgqN93LdycMbn5/7WmH62IuSqmo00AQIL+wAoCRA/p1eR+flnzK0JShJLuTI17sxjd0ASmD2i/54HP1B1qu9HUxF6VULGggCJBfdOCB4Iev/BCALG9W2HKlNaWkJKTgdDipaAi+7k/b7ZH2MNouu9lEl91USgWjgSBAfmEFXreTwWldv6n652l/Ztv+bWwr38bL+S+HLHfcs8chCGmJaWH398/8f5KakEpKQkrEPYxIy0UaMDSwKNW3aSAIUFBUySEDknGESsITgZOGn9T8OlwgmD9lPvtq9lFaW8qznz0bstzv/ve7iI573qvnkeROop87fG/mo90fkexOpp+7X9QDC2jQUKo30kAQoKCwgqOG9++wXKi1CzI9mREf67zDz2t+HS4QLP3+Usrry9lft5+L3rgoZLl+Cf2orK9kb/XesMf92es/i6h+81bMIzUhldSE1LDljDH4HwgEot8bUUrFngYCv8raBnaWVnNO9tAOy0baUEUjYAxN7bg+AH+b8bfm1+MWjQtZ7pGTHqGivoLK+kp+++5vQ5bLL82nvLac8rrysMc96smjSE9Mp7+nf4fDXA2+huY8TRowlOo5NBD4bSmy1hs4kBvFbXVnwIjUlEFTml+HCwRLZrY87xcusJx72Lnsq91HaU0ppbWlYY89+R+T6e/pz4CkAWHLBfYyOjMspZTqGg0EfvlF1jMA0QwEkYp2wOjOwDIvb16r9+GCxiUTLqGwqpC91Xv5tOTTkOWmPDmFgckDyUnKibge2nNQqus0EPgVFFbidAgHZSbHuyohRdqg9dTA8suJv2x+HS5gzDp0Fnuq9rC7cnfY/c1cPJOc5BxyknN0qMkuHjgWdq9vvz1nHFzybvfXp4/QQOCXX1jBQZlJtlqDN9qBBaITNK48qmVpinABY3jacHZV7mJzSfh8TFe+dSWDkgcxqN8gewaMSBvPeJXrjCFToOgzaAxIvuhMsLZ3RTzPOZ5/xzY0EPjlF1VwSHb3Dwv1Nd15Xw7+qXIAABcSSURBVOTuE+9ufh0uYGwq3sSb296k3tc+E2yg+9fez8DkgQxMGti3AkakjWe8ykHkjd03fwFr/tG6jAhMu6pr+4vnOcdin12kgQCob/Tx5d5KZowZGO+q2EZ3BoxXz3wVn/FRXF3M9Oenhyx3/7r7MSHT/7V4avNTDEwayICkAfF9xqKjxs7ng31brfemzfoUvgbY9yU8fwG4POBKBF9j+3LGB64EePPGls+Nz3rddn+7N8CDJ0BtBdRVQE1568YLoLEeqkrgnTsh+1DIOhT6Dw/R2LkhbQj87x7YtQ6+XgslBe3Pt6EWFk6GjIMhYwRkHgKeNHC4rHo1cbggeQB89AhUFUPlXti/q3WZpjruXg9P/KDlb2NMkHNuhLLt8MRZUF8FdZVQuz/IOdfBxpdg8yvgcII4rZ8Y61ht/44lW+H588Hl9R/b1/57EQdMu7r936KLNBAAXxVX0eAzjNQeQY8TratqhzjITsoOW2b1j1ZTVF3Enqo9/OS1n4Qsd8vKWyI65qKNi8jwZDT/i7iX8fh4iqV9QMo0worzP2kpl1RN8Yj2K/Zl+spY8cjJsGeD1SC3I5CUCRWFULrNakgbqq2fvjbH9TXABw/4GzBHSyMmjtaNkyfdujJPyoT0gyCxHySkwLb3rUbVNFq/02+gtW1TwMOWDrf1O8Ea5M9es/6lDYVBE2DCbEgfBkt+DY211pXxCfOtRr24AAo3W+WD9f58DVDwpvUPIDENkjMhKQsqiwBj/W1SB1tBqHqf/29TY/10uqEh4JzdXuvv506y/vUbYAU1ESjZYjXg4oDsw2DY0dbfwBcQSH0N1t9z35ctx07KgqoiK8A01EB9tfXTF5Do0pkAE8+DlOhduGogAAqikGNIxVc0eg5up5vB/QYzuN/gsOWWz1pOYVUhhVWF/HrZr0OW+9OqP0V87Fs+vIX0xDRSXclBgwBgbX/rDqgugaoSik37RYoAih2AD5h4LuSM44TPHqS4rqxduUyPixVnt6R071SPZf9uuGeC1Ui5PPDLD4I3TM3lGq0GbM5bVrmaMtj7hdUL2Pu59a+yEGqbnlsRyD0STvytFQCS2+Tu2v4hrH4MJv0Yjru89WeNDVZD+trVkP9fqwF2OGHkDJj+O6uxTcq0ejrtziURLlrewbn4z/nXH3dczpkAP14cutFue+xL3u14n1HuDYAGAqAl6+ghGgh6re4casryZpHlzWJM5piw5d47+UlKSj5n376tFJdv5zeFy0OWfWXTE+x3dDxR4eQvHiXVQKq4wB263PunLmhJJbL+jqBlDjhH1dCA50Fe+nboHFWhyg3Js/41lR2cDqQHHKmQzNV/CP7dTrsKijYHbxCdLmuI6IyFLUHI4YbvLgzeyKbkWFfYqx8Lf6Ud7XKx2mcXaCDASi2Rk+qhX6L+Ofq6FfsaYPe29h/ktH4qOtNIyOGZZtXBs8Y2SXngOFKAg5o2BBnGafLekLNocLopdwjTvnomZLkjR59BeX2l9cR34ZqQ5eb8Z07YujX58dIf43V5SXInhS3336/+S5IriSR3UkxyVHXp5rwDeOnbQJhEiZEGK4DhQ6D4P7BoXOj7NuECUFfKxWqfnaQtH9aMIR0W6uWiPEtkRea3YO0TrcuJE7JGweOnW/uoLCRzaC7FLme7w2Y6PDDzAUjLhdRca9z5qTCzPGbcgAvIAFgUOhD88fjbml+Hmyn1+CmPU1lfSWV9JVe9fVXIcomuRCobKimqLgpdN+CyFZeF/bzJic+dSKIzEY/TE7bcbStvI9GZaP1zJYYtu2Hvhuay0Q5CMQtAHZSL1T67yvaBwBhDQWEFP8yLLKeP6qHCNfB1ldZMjJIt4O3ffvZHYx1sXAwbXrBuUDbWtb9xCdZYc/lOSEyF0SdB1qGsyBoNSf1h0RktY8eXfhK0696dT3wfOfDI5tfhAsHDJz3c/DpcYHnhuy9Q1VBFVX0Vl/z3kpDlpg2ZRm1jLTUNNRSUBZnh4/dy/svUNtbSEOzv3MbsV2d3WAasIJTgSCDBmRC23LXvXovb4cbtCDO2BizOX4zb4SbBmRA2YGwt24rL4cLtcMekF9QdaVZsHwh2l9dQWdeo9wd6qkiv9I+/EtY80bqMrwE2LYFVj7Te7kyERh/WTA0HZI6EEcdagcPp9v9MgC/esKYsmkZr6uG4WTDzr9askLYiGL9tvnrbvxteuADOevyAAkZ3BpZDMw7tuBCw4JgFza/DBZYPzv0AsNYAr/PVMeXJ0L2lv0z/C7WNtdQ21obNjzVtyDTqffXUNdbxZfmXIcut2r3KKucLfrO9yXX/uy7s503OWHxGROWmPDkFl7hwOVzNyRdDOf/f51vlpHuaaNsHgublKXXqaPc60KGcrMNg/QvW/PLdn1g/G2tb7ys525opkjHCP8fcP8+8vjpgpkYCnP+v4I33kecH3Gx0wbcXBA8C0Lnx25QcuOC1kB/39lQineF0OPE6wi8ENW3otObX4QJBYBD695f/Dlnu9bNeb34dLli9duZr1PvqqffV84MlPwhZ7pbjbqHB10CDr4Eb3r8hZLlZo2fRYBqay774xYshywpCXWMdVb6qkGWiSQNB84yhnptjqE/qaKzeGOtm7OFnwJq/t/7dxjrY8Lz1z5kAA8bA4d+F/iNgxS3W5y4PXPxO8Abekxab2R9hGvd46g2BJR5BqCNDUoZEVO70g09vfh0uEFxx1BWt3ocLBI+d8ljz63DBKlo0EBRWkOpxkd0v/A0rFaEDGspptJ5OXTgZyr+2HnJqR2DA4XD0ryBnvPWwjitgTLhsR2QNdw+YqdFbxSJHVbyCUE8MQPGggaDQmjEkobr8qnNCXelnHGw1/Ls3WE+87tnQfijH7bWGXgaNh0NPtWbapA62ru6f+6lV3pUY/gGdSBvuSK/ge/CVvh1FOwj1hl5QdwQrMabj3Co9SV5enlm1alXHBSPd303/Zfph2dx+1oSo7dPWyr6GhRPa51tp4k6yhnJyxkLaMHjr1pahnBCzbQD41zzrSv/IC+D0O2NXf6X6KBFZbYzJC/aZrXsEZVX17K2o1WcIOtLRcE9FIRQsg/w3YcvyNkHAf4V/7GUwcJx1s9YRMO++fGd0h3KUUp1m60DQtCqZpp/uQLDhHocLkNZBIikTDpkOuXnwn+tbhnLOfb77hnKUUp1m70BQaPNkc5Hc2K2rgsNOg4/bzNzxNcCejTDsmzD9Ohj5LciZAE35cvZ+3utn2yhlF7YPBAkuB0P6h8+z0mcFu9IXpzV184kfQNHnUBYkL484rCv/sx4DT2rwfetQjlK9hq0DQUFRJQdnJeN0SN9aCzWSc9n3FWSNbp9uwTTC3nxr9s7QKTD5x1Y5bwY89cOW1Lrf+2voIAB6pa9UL2LrQJBfWMH4If6sk7FYVi/aweVAnsZ1uK2FOF6aA1+9Z+VrB+scm9ItOFww9kyY+beWIZ5AMUyDq5SKH9sGgpr6Rrbvq+LMybnWhmlXwdonWxfyNVpXxh/+rWVOe2ou5B4V3bVGD7SBzxwF2z6wlgasLYf0odYqSK3OpR6+etdKu3DQVJh6KRx0DHj6w72TrCt9hwtm3BQ8CDT9jXS4R6k+x7aBYEtRJcYEzBhqTifwuH8JPrEecPr47+3nxIuz/RqijfVWY/zAcf41Rn3W8nbB1iSt2w//XWBlwvSkW41z2/VVxWmtWfqf31vrq1YVW0/btttfvbUe6saXwpytAw46Gr57j5Vgre3Dc30gjYJSqutiGghE5BTgHsAJPGyMubXN5/OAnwMNQBHwM2PMV7GsU5P8YMtTTrsKVj1qvXYlWkvR9Rvgb4R3Wg1x089N/7TWSA1c5zQtt2VdVxFrvrzxWePx+KxyCSlQsNzKoxMuBa9phB0r4es11jJ9SVmQlAEZw639GZ91nIOOhqMugsQUK4dOYqr1uqEa/np0S2K1sx478CmcSqk+KWaBQEScwH3ADGAH8JGILDHGbAootgbIM8ZUicgvgNuBs2NVp0AFhRU4BEZkBSSbcybQ3LAHXh0nZ1n/BgU8fTxlThfWOU2EuR9Z5Yyx8uRX77P+rfgjfPEfKzg4XHDEmXDan61GPfAKvtV6qG74waOhG3i90ldKRaDjRVK7bgqQb4zZYoypA54BvhdYwBiz3BjTlGf1AyCydH9RkF9UwdCMJDzugKdct75l/cwZG9kDThPPs3oAkWSwbFtOBBL7WeP5g8bD6Xf7H9LC+nnSTdasnLbDOJEeF6wr/WHf1Ct9pVRYsQwEucD2gPc7/NtCuRAIelkqInNEZJWIrCoqCr+kXqQKCivar0FQsMyaWXPRishmxUTa0EZSLhYNfNOVvs7wUUqF0SNuFovIj4A8YFqwz40xDwIPgpV07kCP1+gzbNlbyfGjswMPYo3dH3w8OCP8s0Q7g6WmW1BKxUEsewQ7gcCFgIf4t7UiIt8GrgXOMMbUtv08FraXVFHX4GvdIyjOt+bWHzK9O6oQnF7BK6XiIJaB4CNglIiMEJEE4BxgSWABEZkE/A0rCBTGsC6tFBQ1rUoWEAjy37R+xjMQKKVUHMQsEBhjGoC5wOvAZuA5Y8xGEblRRJpWe74D6Ac8LyJrRWRJiN1FVdBkcwXLIOMQ6D+8O6qglFI9RkzvERhjlgJL22y7PuD1t2N5/FDyCyvITkkkzeu2NjTUwpfvWDdplVLKZmI5NNRj5RdVcEh2wPMD21dCfZUOCymlbMl2gcAY07xOcbOCZdbc/eHHxq9iSikVJ7YLBEUVteyvaWg9Y6hgmZXQLVxaZaWU6qNsFwhabhSnWBsq98KudTospJSyLdsFgoK2M4a2rACMBgKllG3ZLhDkF1bQL9HFwNREa0PBMisd9OCJ8a2YUkrFif0CgX/GkIj400osg4NPsFJGK6WUDdkvEBRWtDxRXPQp7N+lw0JKKVuzVSDYX1PPnvLalvsDBcusnwefGL9KKaVUnNkqEBQUVQK0TB0tWAZZo601AZRSyqZsFQha5Riqr4Ev/6fDQkop27NdIHA7hWEZSbD9A2td30O+Fe9qKaVUXNkuEAzPTMbldFhppx1uGD413tVSSqm4slUg2FIUkGOoYLm13GNCcvhfUkqpPs4WgWDxmp0cc8ubbNlbybtf7OW199fCnvV6f0AppbBBIFi8ZifXvLSer8tqANhf28Dy1563PtRAoJRSfT8Q3PH6Z1TXN7ba9g2zln2kQs74ONVKKaV6jj4fCL4urW6zxXC8Yz1vN44FR58/faWU6lCfbwkHp3tbvT9MtpMtZaz3HBmnGimlVM/S5wPBlScfitfdklDueMc6APJO/EG8qqSUUj1KTBev7wlmTsoFrHsFX5dWMyNxE2XJozjlmElxrplSSvUMfT4QgBUMZk7KhboquO0CGHtRvKuklFI9Rp8fGmpl23vQWAuHaLZRpZRqYq9AULAcnIkw7Jh410QppXqMvj809MCxsHt9621/HAQ54+CSd+NTJ6WU6kH6fo9gyBRwJrTe5kywtiullLJBIJh2FUib0xQHTLs6PvVRSqkepu8HgpQcmHgeOPyjYM4E633KwPjWSymleoi+HwjA6hU0BQLtDSilVCv2CARNvQJxaG9AKaXa6PuzhppMuwqKNmtvQCml2rBPIEjJgQtei3ctlFKqx4np0JCInCIin4lIvojMD/J5oog86//8QxEZHsv6KKWUai9mgUBEnMB9wKnAGGC2iIxpU+xCYJ8xZiRwF3BbrOqjlFIquFj2CKYA+caYLcaYOuAZ4HttynwPWOR//QLwLRGRGNZJKaVUG7EMBLnA9oD3O/zbgpYxxjQAZUBm2x2JyBwRWSUiq4qKimJUXaWUsqdeMX3UGPOgMSbPGJOXnZ0d7+oopVSfEstZQzuBoQHvh/i3BSuzQ0RcQBpQHG6nq1ev3isiX3WxTlnA3i7+bk+j59Lz9JXzAD2XnupAzuWgUB/EMhB8BIwSkRFYDf45wLltyiwBfgq8D5wFLDPGmHA7NcZ0uUsgIquMMXld/f2eRM+l5+kr5wF6Lj1VrM4lZoHAGNMgInOB1wEn8KgxZqOI3AisMsYsAR4B/iEi+UAJVrBQSinVjWL6QJkxZimwtM226wNe1wA/jGUdlFJKhdcrbhZH0YPxrkAU6bn0PH3lPEDPpaeKyblIB0PySiml+ji79QiUUkq1oYFAKaVszjaBoKMEeL2JiHwpIutFZK2IrIp3fTpDRB4VkUIR2RCwLUNE/iMiX/h/9o9nHSMR4jwWiMhO//eyVkS+E886RkpEhorIchHZJCIbReRS//Ze9b2EOY9e972IiEdEVorIOv+53ODfPsKfoDPfn7AzoaN9RXQ8O9wj8CfA+xyYgZXq4iNgtjFmU1wr1kUi8iWQZ4zpdQ/JiMjxQAXwd2PMWP+224ESY8yt/iDd3xjToxeOCHEeC4AKY8yf4lm3zhKRQcAgY8zHIpICrAZmAufTi76XMOcxi172vfhzriUbYypExA28C1wKzANeMsY8IyIPAOuMMfcf6PHs0iOIJAGe6gbGmLexnhkJFJh8cBHW/7w9Wojz6JWMMbuMMR/7X+8HNmPlAetV30uY8+h1jKXC/9bt/2eA6VgJOiGK34ldAkEkCfB6EwO8ISKrRWROvCsTBQONMbv8r3cDvXkt0bki8ol/6KhHD6UE418TZBLwIb34e2lzHtALvxcRcYrIWqAQ+A9QAJT6E3RCFNsxuwSCvuZYY8xkrLUefuUfpugT/ClGeut45f3AIcBEYBfw5/hWp3NEpB/wIvAbY0x54Ge96XsJch698nsxxjQaYyZi5WmbAhwWq2PZJRBEkgCv1zDG7PT/LARexvqPpDfb4x/fbRrnLYxzfbrEGLPH/z+vD3iIXvS9+MehXwSeNMa85N/c676XYOfRm78XAGNMKbAcOBpI9yfohCi2Y3YJBM0J8Px32c/BSnjX64hIsv9GGCKSDJwEbAj/Wz1eU/JB/D//Gce6dFlTo+n3fXrJ9+K/MfkIsNkYc2fAR73qewl1Hr3xexGRbBFJ97/2Yk102YwVEM7yF4vad2KLWUMA/iljd9OSAO/mOFepS0TkYKxeAFi5op7qTeciIk8DJ2Cl090D/B5YDDwHDAO+AmYZY3r0jdgQ53EC1vCDAb4ELg4YY++xRORY4B1gPeDzb/4t1vh6r/lewpzHbHrZ9yIi47FuBjuxLtifM8bc6P///xkgA1gD/MgYU3vAx7NLIFBKKRWcXYaGlFJKhaCBQCmlbE4DgVJK2ZwGAqWUsjkNBEopZXMaCJTyE5HGgAyVa6OZpVZEhgdmKlWqJ4npmsVK9TLV/kf6lbIV7REo1QH/+g+3+9eAWCkiI/3bh4vIMn8yszdFZJh/+0ARedmfS36diBzj35VTRB7y55d/w//EKCLyf/4c+p+IyDNxOk1lYxoIlGrhbTM0dHbAZ2XGmHHAX7CeUAe4F1hkjBkPPAks9G9fCLxljJkATAY2+rePAu4zxhwBlAI/8G+fD0zy7+eSWJ2cUqHok8VK+YlIhTGmX5DtXwLTjTFb/EnNdhtjMkVkL9ZCKPX+7buMMVkiUgQMCXz0358W+T/GmFH+91cDbmPMTSLyb6xFbhYDiwPy0CvVLbRHoFRkTIjXnRGYE6aRlnt0pwH3YfUePgrILqlUt9BAoFRkzg74+b7/9XtYmWwBzsNKeAbwJvALaF5cJC3UTkXEAQw1xiwHrgbSgHa9EqViSa88lGrh9a8I1eTfxpimKaT9ReQTrKv62f5tvwYeE5ErgSLgAv/2S4EHReRCrCv/X2AtiBKME3jCHywEWOjPP69Ut9F7BEp1wH+PIM8YszfedVEqFnRoSCmlbE57BEopZXPaI1BKKZvTQKCUUjangUAppWxOA4FSStmcBgKllLK5/weA/uHgFsh5DAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "1rDDCMkehE3x" }, "source": [ "Thus far, we keep our focus only on the implicit feedback based matrix factorization model on small movielens dataset. In future, we will be expanding this MVP in the following directions:\n", "1. Large scale industrial datasets - Yoochoose, Trivago\n", "2. Other available models in [this](https://github.com/ShopRunner/collie_recs/tree/main/collie_recs/model) repo\n", "3. Really liked the poster carousel. Put it in dash/streamlit app." ] }, { "cell_type": "markdown", "metadata": { "id": "thC-jHYLJKkz" }, "source": [ "## Training neural factorization model on movielens dataset\n", "> Training MF, MF+bias, and MLP model on movielens-100k dataset in PyTorch." ] }, { "cell_type": "code", "metadata": { "id": "U9XYsONJClRh" }, "source": [ "!pip install -q git+https://github.com/sparsh-ai/recochef.git" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "2LS69WtgCuxJ" }, "source": [ "import torch\n", "import torch.nn.functional as F\n", "\n", "from recochef.datasets.synthetic import Synthetic\n", "from recochef.datasets.movielens import MovieLens\n", "from recochef.preprocessing.split import chrono_split\n", "from recochef.preprocessing.encode import label_encode as le\n", "from recochef.models.factorization import MF, MF_bias\n", "from recochef.models.dnn import CollabFNet" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "X7-2sy7dDJte" }, "source": [ "# # generate synthetic implicit data\n", "# synt = Synthetic()\n", "# df = synt.implicit()\n", "\n", "movielens = MovieLens()\n", "df = movielens.load_interactions()\n", "\n", "# changing rating colname to event following implicit naming conventions\n", "df = df.rename(columns={'RATING': 'EVENT'})" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "EGLNfBJBCw38", "outputId": "06429212-3b1c-4a95-df70-927b8e8a3e43" }, "source": [ "# drop duplicates\n", "df = df.drop_duplicates()\n", "\n", "# chronological split\n", "df_train, df_valid = chrono_split(df, ratio=0.8, min_rating=10)\n", "print(f\"Train set:\\n\\n{df_train}\\n{'='*100}\\n\")\n", "print(f\"Validation set:\\n\\n{df_valid}\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Train set:\n", "\n", " USERID ITEMID EVENT TIMESTAMP\n", "59972 1 168 5.0 874965478\n", "92487 1 172 5.0 874965478\n", "74577 1 165 5.0 874965518\n", "48214 1 156 4.0 874965556\n", "22971 1 166 5.0 874965677\n", "... ... ... ... ...\n", "98752 943 139 1.0 888640027\n", "89336 943 426 4.0 888640027\n", "80660 943 720 1.0 888640048\n", "93177 943 80 2.0 888640048\n", "87415 943 53 3.0 888640067\n", "\n", "[80000 rows x 4 columns]\n", "====================================================================================================\n", "\n", "Validation set:\n", "\n", " USERID ITEMID EVENT TIMESTAMP\n", "10508 1 208 5.0 878542960\n", "83307 1 3 4.0 878542960\n", "8976 1 12 5.0 878542960\n", "78171 1 58 4.0 878542960\n", "9811 1 201 3.0 878542960\n", "... ... ... ... ...\n", "81005 943 450 1.0 888693158\n", "92536 943 227 1.0 888693158\n", "95003 943 230 1.0 888693158\n", "94914 943 229 2.0 888693158\n", "92880 943 234 3.0 888693184\n", "\n", "[20000 rows x 4 columns]\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "68zLUPlvC5LK", "outputId": "46c0f8b6-dd84-4c54-8d55-8eb61fb3fc47" }, "source": [ "# label encoding\n", "df_train, uid_maps = le(df_train, col='USERID')\n", "df_train, iid_maps = le(df_train, col='ITEMID')\n", "df_valid = le(df_valid, col='USERID', maps=uid_maps)\n", "df_valid = le(df_valid, col='ITEMID', maps=iid_maps)\n", "\n", "# # event implicit to rating conversion\n", "# event_weights = {'click':1, 'add':2, 'purchase':4}\n", "# event_maps = dict({'EVENT_TO_IDX':event_weights})\n", "# df_train = le(df_train, col='EVENT', maps=event_maps)\n", "# df_valid = le(df_valid, col='EVENT', maps=event_maps)\n", "\n", "print(f\"Processed Train set:\\n\\n{df_train}\\n{'='*100}\\n\")\n", "print(f\"Processed Validation set:\\n\\n{df_valid}\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Processed Train set:\n", "\n", " USERID ITEMID EVENT TIMESTAMP\n", "59972 0 0 5.0 874965478\n", "92487 0 1 5.0 874965478\n", "74577 0 2 5.0 874965518\n", "48214 0 3 4.0 874965556\n", "22971 0 4 5.0 874965677\n", "... ... ... ... ...\n", "98752 942 933 1.0 888640027\n", "89336 942 990 4.0 888640027\n", "80660 942 643 1.0 888640048\n", "93177 942 155 2.0 888640048\n", "87415 942 166 3.0 888640067\n", "\n", "[80000 rows x 4 columns]\n", "====================================================================================================\n", "\n", "Processed Validation set:\n", "\n", " USERID ITEMID EVENT TIMESTAMP\n", "10508 0 341.0 5.0 878542960\n", "83307 0 983.0 4.0 878542960\n", "8976 0 425.0 5.0 878542960\n", "78171 0 639.0 4.0 878542960\n", "9811 0 490.0 3.0 878542960\n", "... ... ... ... ...\n", "81005 942 314.0 1.0 888693158\n", "92536 942 154.0 1.0 888693158\n", "95003 942 183.0 1.0 888693158\n", "94914 942 176.0 2.0 888693158\n", "92880 942 132.0 3.0 888693184\n", "\n", "[19917 rows x 4 columns]\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VnhEaj5QC8j1", "outputId": "f15ef434-6b1d-4f51-f11c-4bfbe4af649b" }, "source": [ "# get number of unique users and items\n", "num_users = len(df_train.USERID.unique())\n", "num_items = len(df_train.ITEMID.unique())\n", "\n", "num_users_t = len(df_valid.USERID.unique())\n", "num_items_t = len(df_valid.ITEMID.unique())\n", "\n", "print(f\"There are {num_users} users and {num_items} items in the train set.\\n{'='*100}\\n\")\n", "print(f\"There are {num_users_t} users and {num_items_t} items in the validation set.\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "There are 943 users and 1613 items in the train set.\n", "====================================================================================================\n", "\n", "There are 943 users and 1429 items in the validation set.\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "xTiGbb5UCpwM" }, "source": [ "# training and testing related helper functions\n", "def train_epocs(model, epochs=10, lr=0.01, wd=0.0, unsqueeze=False):\n", " optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=wd)\n", " model.train()\n", " for i in range(epochs):\n", " users = torch.LongTensor(df_train.USERID.values) # .cuda()\n", " items = torch.LongTensor(df_train.ITEMID.values) #.cuda()\n", " ratings = torch.FloatTensor(df_train.EVENT.values) #.cuda()\n", " if unsqueeze:\n", " ratings = ratings.unsqueeze(1)\n", " y_hat = model(users, items)\n", " loss = F.mse_loss(y_hat, ratings)\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " print(loss.item()) \n", " test_loss(model, unsqueeze)\n", "\n", "def test_loss(model, unsqueeze=False):\n", " model.eval()\n", " users = torch.LongTensor(df_valid.USERID.values) #.cuda()\n", " items = torch.LongTensor(df_valid.ITEMID.values) #.cuda()\n", " ratings = torch.FloatTensor(df_valid.EVENT.values) #.cuda()\n", " if unsqueeze:\n", " ratings = ratings.unsqueeze(1)\n", " y_hat = model(users, items)\n", " loss = F.mse_loss(y_hat, ratings)\n", " print(\"test loss %.3f \" % loss.item())" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "LxhbI4ECC_Jb", "outputId": "fa326841-1e15-4900-c0e4-fc7790beb762" }, "source": [ "# training MF model\n", "model = MF(num_users, num_items, emb_size=100) # .cuda() if you have a GPU\n", "print(f\"Training MF model:\\n\")\n", "train_epocs(model, epochs=10, lr=0.1)\n", "print(f\"\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Training MF model:\n", "\n", "13.594555854797363\n", "5.292399883270264\n", "2.558849573135376\n", "3.584117889404297\n", "1.0360910892486572\n", "1.9875222444534302\n", "2.920832633972168\n", "2.4130148887634277\n", "1.2886441946029663\n", "1.112807273864746\n", "test loss 2.085 \n", "\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fnbkknGIDAs6", "outputId": "e8466582-7078-49ab-dda7-eeffaa65c8de" }, "source": [ "# training MF with bias model\n", "model = MF_bias(num_users, num_items, emb_size=100) #.cuda()\n", "print(f\"Training MF+bias model:\\n\")\n", "train_epocs(model, epochs=10, lr=0.05, wd=1e-5)\n", "print(f\"\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Training MF+bias model:\n", "\n", "13.59664535522461\n", "9.730958938598633\n", "4.798837184906006\n", "1.3603413105010986\n", "2.697232723236084\n", "4.214857578277588\n", "2.871798276901245\n", "1.3329992294311523\n", "0.9624974727630615\n", "1.459389328956604\n", "test loss 2.269 \n", "\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "N9ltu-ISDCUY", "outputId": "06c01140-05a4-4b06-9819-546a7ecdba66" }, "source": [ "# training MLP model\n", "model = CollabFNet(num_users, num_items, emb_size=100) #.cuda()\n", "print(f\"Training MLP model:\\n\")\n", "train_epocs(model, epochs=15, lr=0.05, wd=1e-6, unsqueeze=True)\n", "print(f\"\\n{'='*100}\\n\")" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Training MLP model:\n", "\n", "12.962654113769531\n", "1.4028953313827515\n", "15.373563766479492\n", "2.177295207977295\n", "2.6291019916534424\n", "5.752542495727539\n", "6.88251256942749\n", "6.2746357917785645\n", "4.8090314865112305\n", "3.095308303833008\n", "1.6791961193084717\n", "1.1257785558700562\n", "1.678966760635376\n", "2.615834951400757\n", "2.80102276802063\n", "test loss 2.559 \n", "\n", "====================================================================================================\n", "\n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "UxWgUAp7vnIM" }, "source": [ "## Neural Matrix Factorization on Movielens\n", "> Experiments with different variations of Neural matrix factorization model in PyTorch on movielens dataset." ] }, { "cell_type": "code", "metadata": { "id": "hfac3W-Z4yEs" }, "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "74OaJCfn5H4Z", "outputId": "9ec26cd4-a004-49cf-aefb-9a24c670bf11" }, "source": [ "data = pd.read_csv(\"https://raw.githubusercontent.com/sparsh-ai/reco-data/master/MovieLens_LatestSmall_ratings.csv\")\n", "data.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
0114.0964982703
1134.0964981247
2164.0964982224
31475.0964983815
41505.0964982931
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "0 1 1 4.0 964982703\n", "1 1 3 4.0 964981247\n", "2 1 6 4.0 964982224\n", "3 1 47 5.0 964983815\n", "4 1 50 5.0 964982931" ] }, "metadata": { "tags": [] }, "execution_count": 2 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "F0Irmpk2oUna", "outputId": "082a3deb-3f36-4d93-8917-77c27db5fc55" }, "source": [ "data.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(100836, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 3 } ] }, { "cell_type": "markdown", "metadata": { "id": "AsSMbrTG6LQr" }, "source": [ "Data encoding" ] }, { "cell_type": "code", "metadata": { "id": "m_wEgrHx5U93" }, "source": [ "np.random.seed(3)\n", "msk = np.random.rand(len(data)) < 0.8\n", "train = data[msk].copy()\n", "valid = data[~msk].copy()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "fmeQCQXP6Vtv" }, "source": [ "# here is a handy function modified from fast.ai\n", "def proc_col(col, train_col=None):\n", " \"\"\"Encodes a pandas column with continous ids. \n", " \"\"\"\n", " if train_col is not None:\n", " uniq = train_col.unique()\n", " else:\n", " uniq = col.unique()\n", " name2idx = {o:i for i,o in enumerate(uniq)}\n", " return name2idx, np.array([name2idx.get(x, -1) for x in col]), len(uniq)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "OUfpCvFJ6W72" }, "source": [ "def encode_data(df, train=None):\n", " \"\"\" Encodes rating data with continous user and movie ids. \n", " If train is provided, encodes df with the same encoding as train.\n", " \"\"\"\n", " df = df.copy()\n", " for col_name in [\"userId\", \"movieId\"]:\n", " train_col = None\n", " if train is not None:\n", " train_col = train[col_name]\n", " _,col,_ = proc_col(df[col_name], train_col)\n", " df[col_name] = col\n", " df = df[df[col_name] >= 0]\n", " return df" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "TZDUY2rt6Z9B" }, "source": [ "# encoding the train and validation data\n", "df_train = encode_data(train)\n", "df_valid = encode_data(valid, train)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "c9VIAfR6otTe", "outputId": "dc5ad891-2004-4fda-acfa-22d04525df3b" }, "source": [ "df_train.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
0004.0964982703
1014.0964981247
2024.0964982224
3035.0964983815
6045.0964980868
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "0 0 0 4.0 964982703\n", "1 0 1 4.0 964981247\n", "2 0 2 4.0 964982224\n", "3 0 3 5.0 964983815\n", "6 0 4 5.0 964980868" ] }, "metadata": { "tags": [] }, "execution_count": 8 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "iy77qSeCo2WY", "outputId": "c4ce1748-6c39-44f6-c094-476873135fdb" }, "source": [ "df_train.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(80450, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 9 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "sOSEDMQPo2Tq", "outputId": "4327e4a5-b01b-4d03-c829-0220e7c8b36c" }, "source": [ "df_valid.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
403885.0964982931
509953.0964982400
2908414.0964981179
3005674.0964982653
3204024.0964982546
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "4 0 388 5.0 964982931\n", "5 0 995 3.0 964982400\n", "29 0 841 4.0 964981179\n", "30 0 567 4.0 964982653\n", "32 0 402 4.0 964982546" ] }, "metadata": { "tags": [] }, "execution_count": 10 } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "2uiymy6po2Lo", "outputId": "f1ebfdf2-a139-46d4-d8e7-850701cb57a2" }, "source": [ "df_valid.shape" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(19591, 4)" ] }, "metadata": { "tags": [] }, "execution_count": 11 } ] }, { "cell_type": "markdown", "metadata": { "id": "y3QSDyZj61Iy" }, "source": [ "Matrix factorization model" ] }, { "cell_type": "code", "metadata": { "id": "HBPnUZl-6z1g" }, "source": [ "class MF(nn.Module):\n", " def __init__(self, num_users, num_items, emb_size=100):\n", " super(MF, self).__init__()\n", " self.user_emb = nn.Embedding(num_users, emb_size)\n", " self.item_emb = nn.Embedding(num_items, emb_size)\n", " self.user_emb.weight.data.uniform_(0, 0.05)\n", " self.item_emb.weight.data.uniform_(0, 0.05)\n", " \n", " def forward(self, u, v):\n", " u = self.user_emb(u)\n", " v = self.item_emb(v)\n", " return (u*v).sum(1)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "dBQhfy2l7AAn", "outputId": "d667d3a3-2baa-467b-e8ab-905c3282780f" }, "source": [ "# unit testing the architecture\n", "sample = encode_data(train.sample(5))\n", "display(sample)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
63234003.0961596392
96012113.0840875700
31417222.0955944735
17473330.51516141230
66983444.01328232721
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "63234 0 0 3.0 961596392\n", "96012 1 1 3.0 840875700\n", "31417 2 2 2.0 955944735\n", "17473 3 3 0.5 1516141230\n", "66983 4 4 4.0 1328232721" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "code", "metadata": { "id": "9tmmtDTuqnIB" }, "source": [ "num_users = 5\n", "num_items = 5\n", "emb_size = 3\n", "\n", "user_emb = nn.Embedding(num_users, emb_size)\n", "item_emb = nn.Embedding(num_items, emb_size)\n", "\n", "users = torch.LongTensor(sample.userId.values)\n", "items = torch.LongTensor(sample.movieId.values)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 102 }, "id": "H557JwqSqimK", "outputId": "b45ad07e-0cf7-4cb0-f9ad-5de2f8a86c08" }, "source": [ "U = user_emb(users)\n", "display(U)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "tensor([[ 2.2694, 0.9679, 0.3305],\n", " [-1.1478, -0.7004, -0.8113],\n", " [-1.2287, -0.7210, 0.3875],\n", " [ 0.9106, 0.0427, -0.7128],\n", " [-1.0396, -0.2739, 0.7271]], grad_fn=)" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 102 }, "id": "IsdtlmtBq3cj", "outputId": "418f529d-1481-475e-ad2c-39578baa4cdf" }, "source": [ "V = item_emb(items)\n", "display(V)" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "tensor([[-1.9371, -1.1172, -1.5967],\n", " [-2.4336, -1.1177, 0.6197],\n", " [ 0.5889, 1.4830, -1.0103],\n", " [-0.8294, 0.5744, -1.7315],\n", " [-1.6733, -0.2447, -0.2630]], grad_fn=)" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 102 }, "id": "TKtMvkNfq0q2", "outputId": "c7bb19a7-3a7b-45db-f4c1-b95f0994750f" }, "source": [ "display(U*V) # element wise multiplication" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "tensor([[-4.3959, -1.0813, -0.5278],\n", " [ 2.7932, 0.7828, -0.5027],\n", " [-0.7236, -1.0693, -0.3915],\n", " [-0.7552, 0.0246, 1.2343],\n", " [ 1.7397, 0.0670, -0.1912]], grad_fn=)" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 34 }, "id": "5_AN_dhQq0nE", "outputId": "7cc9d053-c9f3-49d0-e2a6-f0ea809ecd8f" }, "source": [ "display((U*V).sum(1))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/plain": [ "tensor([-6.0050, 3.0733, -2.1844, 0.5036, 1.6155], grad_fn=)" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "W01e58dr86WY" }, "source": [ "Model training" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "VC5vARcP7QAc", "outputId": "b4cea803-bbac-4301-bb1c-12683689948d" }, "source": [ "num_users = len(df_train.userId.unique())\n", "num_items = len(df_train.movieId.unique())\n", "print(\"{} users and {} items in the training set\".format(num_users, num_items))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "610 users and 8998 items in the training set\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "yRi5sy-K8-fr" }, "source": [ "model = MF(num_users, num_items, emb_size=100) # .cuda() if you have a GPU" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "4nAGJ4l08_83" }, "source": [ "def train_epocs(model, epochs=10, lr=0.01, wd=0.0, unsqueeze=False):\n", " optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=wd)\n", " model.train()\n", " for i in range(epochs):\n", " users = torch.LongTensor(df_train.userId.values) # .cuda()\n", " items = torch.LongTensor(df_train.movieId.values) #.cuda()\n", " ratings = torch.FloatTensor(df_train.rating.values) #.cuda()\n", " if unsqueeze:\n", " ratings = ratings.unsqueeze(1)\n", " y_hat = model(users, items)\n", " loss = F.mse_loss(y_hat, ratings)\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " print(loss.item()) \n", " test_loss(model, unsqueeze)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "7l_3G5gn9GH3" }, "source": [ "def test_loss(model, unsqueeze=False):\n", " model.eval()\n", " users = torch.LongTensor(df_valid.userId.values) #.cuda()\n", " items = torch.LongTensor(df_valid.movieId.values) #.cuda()\n", " ratings = torch.FloatTensor(df_valid.rating.values) #.cuda()\n", " if unsqueeze:\n", " ratings = ratings.unsqueeze(1)\n", " y_hat = model(users, items)\n", " loss = F.mse_loss(y_hat, ratings)\n", " print(\"test loss %.3f \" % loss.item())" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "EztQtZKl9M53", "outputId": "25713f3c-edff-4333-e45b-ffb2c79375fb" }, "source": [ "train_epocs(model, epochs=10, lr=0.1)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "12.914263725280762\n", "4.8582916259765625\n", "2.5804786682128906\n", "3.109440565109253\n", "0.850287139415741\n", "1.819737195968628\n", "2.657919406890869\n", "2.138274908065796\n", "1.0904945135116577\n", "0.9722878932952881\n", "test loss 1.851 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AoSgUhWV9O1q", "outputId": "48e887fa-b55c-4465-e2d0-05789b5a7419" }, "source": [ "train_epocs(model, epochs=10, lr=0.01)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "1.6430705785751343\n", "1.0046814680099487\n", "0.712002694606781\n", "0.6611021757125854\n", "0.7258523106575012\n", "0.803934633731842\n", "0.843424379825592\n", "0.8351688981056213\n", "0.7928505539894104\n", "0.737376868724823\n", "test loss 0.827 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "erRnsApY9Q7e", "outputId": "1e6a68f3-13b1-4963-b187-5d1b1bc3e438" }, "source": [ "train_epocs(model, epochs=10, lr=0.01)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0.6877127289772034\n", "0.6256141066551208\n", "0.6374999284744263\n", "0.6272100210189819\n", "0.6171814799308777\n", "0.614914059638977\n", "0.6113061308860779\n", "0.6033822298049927\n", "0.595890998840332\n", "0.592114269733429\n", "test loss 0.764 \n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "HAwA9Rts9UI1" }, "source": [ "MF with bias" ] }, { "cell_type": "code", "metadata": { "id": "Dur1n3lo9S3C" }, "source": [ "class MF_bias(nn.Module):\n", " def __init__(self, num_users, num_items, emb_size=100):\n", " super(MF_bias, self).__init__()\n", " self.user_emb = nn.Embedding(num_users, emb_size)\n", " self.user_bias = nn.Embedding(num_users, 1)\n", " self.item_emb = nn.Embedding(num_items, emb_size)\n", " self.item_bias = nn.Embedding(num_items, 1)\n", " self.user_emb.weight.data.uniform_(0,0.05)\n", " self.item_emb.weight.data.uniform_(0,0.05)\n", " self.user_bias.weight.data.uniform_(-0.01,0.01)\n", " self.item_bias.weight.data.uniform_(-0.01,0.01)\n", " \n", " def forward(self, u, v):\n", " U = self.user_emb(u)\n", " V = self.item_emb(v)\n", " b_u = self.user_bias(u).squeeze()\n", " b_v = self.item_bias(v).squeeze()\n", " return (U*V).sum(1) + b_u + b_v" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "WAyaietL9ZAq" }, "source": [ "model = MF_bias(num_users, num_items, emb_size=100) #.cuda()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5nEO-IVp9acn", "outputId": "773ee576-ea2e-40ac-97c7-7d78606595e1" }, "source": [ "train_epocs(model, epochs=10, lr=0.05, wd=1e-5)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "12.91020393371582\n", "9.150527954101562\n", "4.3840012550354\n", "1.1575191020965576\n", "2.46807861328125\n", "3.7430803775787354\n", "2.4481022357940674\n", "1.0781667232513428\n", "0.816169023513794\n", "1.3183783292770386\n", "test loss 2.069 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "nD2fD5A59cK4", "outputId": "03df230c-70ff-4767-e088-928d54f6cd37" }, "source": [ "train_epocs(model, epochs=10, lr=0.01, wd=1e-5)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "1.8935126066207886\n", "1.3250681161880493\n", "0.9350242614746094\n", "0.7446779012680054\n", "0.722224235534668\n", "0.7774652242660522\n", "0.8231741189956665\n", "0.8222126364707947\n", "0.7816660404205322\n", "0.727698802947998\n", "test loss 0.798 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Os53hZxr9e_T", "outputId": "d9bf93d3-8411-4535-dcc7-ebcf4a3fbc48" }, "source": [ "train_epocs(model, epochs=10, lr=0.001, wd=1e-5)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0.6853442788124084\n", "0.6711287498474121\n", "0.6592414975166321\n", "0.6495122909545898\n", "0.6417150497436523\n", "0.6356027722358704\n", "0.6309247612953186\n", "0.6274365186691284\n", "0.6249085068702698\n", "0.6231329441070557\n", "test loss 0.751 \n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "-XhFy6bU9h48" }, "source": [ "Note that these models are susceptible to weight initialization, optimization algorithm and regularization.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "NugoowzF9kCk" }, "source": [ "### Neural Network Model\n", "Note here there is no matrix multiplication, we could potentially make the embeddings of different sizes. Here we could get better results by keep playing with regularization." ] }, { "cell_type": "code", "metadata": { "id": "qLVWHOxQ9fVX" }, "source": [ "class CollabFNet(nn.Module):\n", " def __init__(self, num_users, num_items, emb_size=100, n_hidden=10):\n", " super(CollabFNet, self).__init__()\n", " self.user_emb = nn.Embedding(num_users, emb_size)\n", " self.item_emb = nn.Embedding(num_items, emb_size)\n", " self.lin1 = nn.Linear(emb_size*2, n_hidden)\n", " self.lin2 = nn.Linear(n_hidden, 1)\n", " self.drop1 = nn.Dropout(0.1)\n", " \n", " def forward(self, u, v):\n", " U = self.user_emb(u)\n", " V = self.item_emb(v)\n", " x = F.relu(torch.cat([U, V], dim=1))\n", " x = self.drop1(x)\n", " x = F.relu(self.lin1(x))\n", " x = self.lin2(x)\n", " return x" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ljjju7Yy9x7b" }, "source": [ "model = CollabFNet(num_users, num_items, emb_size=100) #.cuda()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "YuG2Hz5e9yyl", "outputId": "67b716ed-84e4-4dcf-eaf0-f06609542d0d" }, "source": [ "train_epocs(model, epochs=15, lr=0.05, wd=1e-6, unsqueeze=True)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "14.657201766967773\n", "2.586819648742676\n", "6.025796890258789\n", "2.89852237701416\n", "1.1256697177886963\n", "2.0860772132873535\n", "2.9243881702423096\n", "2.806140422821045\n", "1.9981783628463745\n", "1.1265769004821777\n", "0.8947575092315674\n", "1.4373805522918701\n", "1.795198678970337\n", "1.4024922847747803\n", "0.8697773218154907\n", "test loss 0.797 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JYyXb1qO90vb", "outputId": "b3bcc463-51a8-4625-f547-38ebbf105a7f" }, "source": [ "train_epocs(model, epochs=10, lr=0.001, wd=1e-6, unsqueeze=True)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0.7495059967041016\n", "0.7382366061210632\n", "0.731941282749176\n", "0.7295416593551636\n", "0.7321946024894714\n", "0.7312469482421875\n", "0.731982409954071\n", "0.7298287153244019\n", "0.7264290452003479\n", "0.7244617938995361\n", "test loss 0.774 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "o0d-WRvW92h-", "outputId": "20f81203-7fbb-44db-b421-c6c20239cd22" }, "source": [ "train_epocs(model, epochs=10, lr=0.001, wd=1e-6, unsqueeze=True)" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0.7242854833602905\n", "0.7213587760925293\n", "0.7197834849357605\n", "0.7182263135910034\n", "0.7177621722221375\n", "0.7155387997627258\n", "0.7147852182388306\n", "0.7143447995185852\n", "0.7133223414421082\n", "0.712261974811554\n", "test loss 0.766 \n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "6fpmHCaYAn2d" }, "source": [ "### Neural network model - different approach\n", "> youtube: https://youtu.be/MVB1cbe923A" ] }, { "cell_type": "markdown", "metadata": { "id": "SMuCwuPWPfGz" }, "source": [ "### Ethan Rosenthal\n", "\n", "Ref - https://github.com/EthanRosenthal/torchmf" ] }, { "cell_type": "code", "metadata": { "id": "J31f-camBorB" }, "source": [ "import os\n", "import requests\n", "import zipfile\n", "import collections\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import scipy.sparse as sp\n", "from sklearn.metrics import roc_auc_score\n", "\n", "import torch\n", "from torch import nn\n", "import torch.multiprocessing as mp\n", "import torch.utils.data as data\n", "from tqdm import tqdm" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "ahu8EWCGQJkI" }, "source": [ "def _get_data_path():\n", " \"\"\"\n", " Get path to the movielens dataset file.\n", " \"\"\"\n", " data_path = '/content/data'\n", " if not os.path.exists(data_path):\n", " print('Making data path')\n", " os.mkdir(data_path)\n", " return data_path\n", "\n", "\n", "def _download_movielens(dest_path):\n", " \"\"\"\n", " Download the dataset.\n", " \"\"\"\n", "\n", " url = 'http://files.grouplens.org/datasets/movielens/ml-100k.zip'\n", " req = requests.get(url, stream=True)\n", "\n", " print('Downloading MovieLens data')\n", "\n", " with open(os.path.join(dest_path, 'ml-100k.zip'), 'wb') as fd:\n", " for chunk in req.iter_content(chunk_size=None):\n", " fd.write(chunk)\n", "\n", " with zipfile.ZipFile(os.path.join(dest_path, 'ml-100k.zip'), 'r') as z:\n", " z.extractall(dest_path)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "DouK7x1nPsNb" }, "source": [ "def read_movielens_df():\n", " path = _get_data_path()\n", " zipfile = os.path.join(path, 'ml-100k.zip')\n", " if not os.path.isfile(zipfile):\n", " _download_movielens(path)\n", " fname = os.path.join(path, 'ml-100k', 'u.data')\n", " names = ['user_id', 'item_id', 'rating', 'timestamp']\n", " df = pd.read_csv(fname, sep='\\t', names=names)\n", " return df\n", "\n", "\n", "def get_movielens_interactions():\n", " df = read_movielens_df()\n", "\n", " n_users = df.user_id.unique().shape[0]\n", " n_items = df.item_id.unique().shape[0]\n", "\n", " interactions = np.zeros((n_users, n_items))\n", " for row in df.itertuples():\n", " interactions[row[1] - 1, row[2] - 1] = row[3]\n", " return interactions\n", "\n", "\n", "def train_test_split(interactions, n=10):\n", " \"\"\"\n", " Split an interactions matrix into training and test sets.\n", " Parameters\n", " ----------\n", " interactions : np.ndarray\n", " n : int (default=10)\n", " Number of items to select / row to place into test.\n", "\n", " Returns\n", " -------\n", " train : np.ndarray\n", " test : np.ndarray\n", " \"\"\"\n", " test = np.zeros(interactions.shape)\n", " train = interactions.copy()\n", " for user in range(interactions.shape[0]):\n", " if interactions[user, :].nonzero()[0].shape[0] > n:\n", " test_interactions = np.random.choice(interactions[user, :].nonzero()[0],\n", " size=n,\n", " replace=False)\n", " train[user, test_interactions] = 0.\n", " test[user, test_interactions] = interactions[user, test_interactions]\n", "\n", " # Test and training are truly disjoint\n", " assert(np.all((train * test) == 0))\n", " return train, test\n", "\n", "\n", "def get_movielens_train_test_split(implicit=False):\n", " interactions = get_movielens_interactions()\n", " if implicit:\n", " interactions = (interactions >= 4).astype(np.float32)\n", " train, test = train_test_split(interactions)\n", " train = sp.coo_matrix(train)\n", " test = sp.coo_matrix(test)\n", " return train, test" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0x9xMyW6PsK6", "outputId": "00096a84-7542-4d07-920d-48830410244e" }, "source": [ "%%writefile metrics.py\n", "\n", "import numpy as np\n", "from sklearn.metrics import roc_auc_score\n", "from torch import multiprocessing as mp\n", "import torch\n", "\n", "def get_row_indices(row, interactions):\n", " start = interactions.indptr[row]\n", " end = interactions.indptr[row + 1]\n", " return interactions.indices[start:end]\n", "\n", "\n", "def auc(model, interactions, num_workers=1):\n", " aucs = []\n", " processes = []\n", " n_users = interactions.shape[0]\n", " mp_batch = int(np.ceil(n_users / num_workers))\n", "\n", " queue = mp.Queue()\n", " rows = np.arange(n_users)\n", " np.random.shuffle(rows)\n", " for rank in range(num_workers):\n", " start = rank * mp_batch\n", " end = np.min((start + mp_batch, n_users))\n", " p = mp.Process(target=batch_auc,\n", " args=(queue, rows[start:end], interactions, model))\n", " p.start()\n", " processes.append(p)\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " aucs.append(queue.get())\n", "\n", " queue.close()\n", " for p in processes:\n", " p.join()\n", " return np.mean(aucs)\n", "\n", "\n", "def batch_auc(queue, rows, interactions, model):\n", " n_items = interactions.shape[1]\n", " items = torch.arange(0, n_items).long()\n", " users_init = torch.ones(n_items).long()\n", " for row in rows:\n", " row = int(row)\n", " users = users_init.fill_(row)\n", "\n", " preds = model.predict(users, items)\n", " actuals = get_row_indices(row, interactions)\n", "\n", " if len(actuals) == 0:\n", " continue\n", " y_test = np.zeros(n_items)\n", " y_test[actuals] = 1\n", " queue.put(roc_auc_score(y_test, preds.data.numpy()))\n", "\n", "\n", "def patk(model, interactions, num_workers=1, k=5):\n", " patks = []\n", " processes = []\n", " n_users = interactions.shape[0]\n", " mp_batch = int(np.ceil(n_users / num_workers))\n", "\n", " queue = mp.Queue()\n", " rows = np.arange(n_users)\n", " np.random.shuffle(rows)\n", " for rank in range(num_workers):\n", " start = rank * mp_batch\n", " end = np.min((start + mp_batch, n_users))\n", " p = mp.Process(target=batch_patk,\n", " args=(queue, rows[start:end], interactions, model),\n", " kwargs={'k': k})\n", " p.start()\n", " processes.append(p)\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " patks.append(queue.get())\n", "\n", " queue.close()\n", " for p in processes:\n", " p.join()\n", " return np.mean(patks)\n", "\n", "\n", "def batch_patk(queue, rows, interactions, model, k=5):\n", " n_items = interactions.shape[1]\n", "\n", " items = torch.arange(0, n_items).long()\n", " users_init = torch.ones(n_items).long()\n", " for row in rows:\n", " row = int(row)\n", " users = users_init.fill_(row)\n", "\n", " preds = model.predict(users, items)\n", " actuals = get_row_indices(row, interactions)\n", "\n", " if len(actuals) == 0:\n", " continue\n", "\n", " top_k = np.argpartition(-np.squeeze(preds.data.numpy()), k)\n", " top_k = set(top_k[:k])\n", " true_pids = set(actuals)\n", " if true_pids:\n", " queue.put(len(top_k & true_pids) / float(k))" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Writing metrics.py\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "I6IqWKWFhAQh", "outputId": "3bef0869-34b5-488c-ede7-bf9be08c115f" }, "source": [ "import metrics\n", "import importlib\n", "importlib.reload(metrics)" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "" ] }, "metadata": { "tags": [] }, "execution_count": 55 } ] }, { "cell_type": "code", "metadata": { "id": "UEhEE_GhPsH2" }, "source": [ "class Interactions(data.Dataset):\n", " \"\"\"\n", " Hold data in the form of an interactions matrix.\n", " Typical use-case is like a ratings matrix:\n", " - Users are the rows\n", " - Items are the columns\n", " - Elements of the matrix are the ratings given by a user for an item.\n", " \"\"\"\n", "\n", " def __init__(self, mat):\n", " self.mat = mat.astype(np.float32).tocoo()\n", " self.n_users = self.mat.shape[0]\n", " self.n_items = self.mat.shape[1]\n", "\n", " def __getitem__(self, index):\n", " row = self.mat.row[index]\n", " col = self.mat.col[index]\n", " val = self.mat.data[index]\n", " return (row, col), val\n", "\n", " def __len__(self):\n", " return self.mat.nnz\n", "\n", "\n", "class PairwiseInteractions(data.Dataset):\n", " \"\"\"\n", " Sample data from an interactions matrix in a pairwise fashion. The row is\n", " treated as the main dimension, and the columns are sampled pairwise.\n", " \"\"\"\n", "\n", " def __init__(self, mat):\n", " self.mat = mat.astype(np.float32).tocoo()\n", "\n", " self.n_users = self.mat.shape[0]\n", " self.n_items = self.mat.shape[1]\n", "\n", " self.mat_csr = self.mat.tocsr()\n", " if not self.mat_csr.has_sorted_indices:\n", " self.mat_csr.sort_indices()\n", "\n", " def __getitem__(self, index):\n", " row = self.mat.row[index]\n", " found = False\n", "\n", " while not found:\n", " neg_col = np.random.randint(self.n_items)\n", " if self.not_rated(row, neg_col, self.mat_csr.indptr,\n", " self.mat_csr.indices):\n", " found = True\n", "\n", " pos_col = self.mat.col[index]\n", " val = self.mat.data[index]\n", "\n", " return (row, (pos_col, neg_col)), val\n", "\n", " def __len__(self):\n", " return self.mat.nnz\n", "\n", " @staticmethod\n", " def not_rated(row, col, indptr, indices):\n", " # similar to use of bsearch in lightfm\n", " start = indptr[row]\n", " end = indptr[row + 1]\n", " searched = np.searchsorted(indices[start:end], col, 'right')\n", " if searched >= (end - start):\n", " # After the array\n", " return False\n", " return col != indices[searched] # Not found\n", "\n", " def get_row_indices(self, row):\n", " start = self.mat_csr.indptr[row]\n", " end = self.mat_csr.indptr[row + 1]\n", " return self.mat_csr.indices[start:end]\n", "\n", "\n", "class BaseModule(nn.Module):\n", " \"\"\"\n", " Base module for explicit matrix factorization.\n", " \"\"\"\n", " \n", " def __init__(self,\n", " n_users,\n", " n_items,\n", " n_factors=40,\n", " dropout_p=0,\n", " sparse=False):\n", " \"\"\"\n", "\n", " Parameters\n", " ----------\n", " n_users : int\n", " Number of users\n", " n_items : int\n", " Number of items\n", " n_factors : int\n", " Number of latent factors (or embeddings or whatever you want to\n", " call it).\n", " dropout_p : float\n", " p in nn.Dropout module. Probability of dropout.\n", " sparse : bool\n", " Whether or not to treat embeddings as sparse. NOTE: cannot use\n", " weight decay on the optimizer if sparse=True. Also, can only use\n", " Adagrad.\n", " \"\"\"\n", " super(BaseModule, self).__init__()\n", " self.n_users = n_users\n", " self.n_items = n_items\n", " self.n_factors = n_factors\n", " self.user_biases = nn.Embedding(n_users, 1, sparse=sparse)\n", " self.item_biases = nn.Embedding(n_items, 1, sparse=sparse)\n", " self.user_embeddings = nn.Embedding(n_users, n_factors, sparse=sparse)\n", " self.item_embeddings = nn.Embedding(n_items, n_factors, sparse=sparse)\n", " \n", " self.dropout_p = dropout_p\n", " self.dropout = nn.Dropout(p=self.dropout_p)\n", "\n", " self.sparse = sparse\n", " \n", " def forward(self, users, items):\n", " \"\"\"\n", " Forward pass through the model. For a single user and item, this\n", " looks like:\n", "\n", " user_bias + item_bias + user_embeddings.dot(item_embeddings)\n", "\n", " Parameters\n", " ----------\n", " users : np.ndarray\n", " Array of user indices\n", " items : np.ndarray\n", " Array of item indices\n", "\n", " Returns\n", " -------\n", " preds : np.ndarray\n", " Predicted ratings.\n", "\n", " \"\"\"\n", " ues = self.user_embeddings(users)\n", " uis = self.item_embeddings(items)\n", "\n", " preds = self.user_biases(users)\n", " preds += self.item_biases(items)\n", " preds += (self.dropout(ues) * self.dropout(uis)).sum(dim=1, keepdim=True)\n", "\n", " return preds.squeeze()\n", " \n", " def __call__(self, *args):\n", " return self.forward(*args)\n", "\n", " def predict(self, users, items):\n", " return self.forward(users, items)\n", "\n", "\n", "def bpr_loss(preds, vals):\n", " sig = nn.Sigmoid()\n", " return (1.0 - sig(preds)).pow(2).sum()\n", "\n", "\n", "class BPRModule(nn.Module):\n", " \n", " def __init__(self,\n", " n_users,\n", " n_items,\n", " n_factors=40,\n", " dropout_p=0,\n", " sparse=False,\n", " model=BaseModule):\n", " super(BPRModule, self).__init__()\n", "\n", " self.n_users = n_users\n", " self.n_items = n_items\n", " self.n_factors = n_factors\n", " self.dropout_p = dropout_p\n", " self.sparse = sparse\n", " self.pred_model = model(\n", " self.n_users,\n", " self.n_items,\n", " n_factors=n_factors,\n", " dropout_p=dropout_p,\n", " sparse=sparse\n", " )\n", "\n", " def forward(self, users, items):\n", " assert isinstance(items, tuple), \\\n", " 'Must pass in items as (pos_items, neg_items)'\n", " # Unpack\n", " (pos_items, neg_items) = items\n", " pos_preds = self.pred_model(users, pos_items)\n", " neg_preds = self.pred_model(users, neg_items)\n", " return pos_preds - neg_preds\n", "\n", " def predict(self, users, items):\n", " return self.pred_model(users, items)\n", "\n", "\n", "class BasePipeline:\n", " \"\"\"\n", " Class defining a training pipeline. Instantiates data loaders, model,\n", " and optimizer. Handles training for multiple epochs and keeping track of\n", " train and test loss.\n", " \"\"\"\n", "\n", " def __init__(self,\n", " train,\n", " test=None,\n", " model=BaseModule,\n", " n_factors=40,\n", " batch_size=32,\n", " dropout_p=0.02,\n", " sparse=False,\n", " lr=0.01,\n", " weight_decay=0.,\n", " optimizer=torch.optim.Adam,\n", " loss_function=nn.MSELoss(reduction='sum'),\n", " n_epochs=10,\n", " verbose=False,\n", " random_seed=None,\n", " interaction_class=Interactions,\n", " hogwild=False,\n", " num_workers=0,\n", " eval_metrics=None,\n", " k=5):\n", " self.train = train\n", " self.test = test\n", "\n", " if hogwild:\n", " num_loader_workers = 0\n", " else:\n", " num_loader_workers = num_workers\n", " self.train_loader = data.DataLoader(\n", " interaction_class(train), batch_size=batch_size, shuffle=True,\n", " num_workers=num_loader_workers)\n", " if self.test is not None:\n", " self.test_loader = data.DataLoader(\n", " interaction_class(test), batch_size=batch_size, shuffle=True,\n", " num_workers=num_loader_workers)\n", " self.num_workers = num_workers\n", " self.n_users = self.train.shape[0]\n", " self.n_items = self.train.shape[1]\n", " self.n_factors = n_factors\n", " self.batch_size = batch_size\n", " self.dropout_p = dropout_p\n", " self.lr = lr\n", " self.weight_decay = weight_decay\n", " self.loss_function = loss_function\n", " self.n_epochs = n_epochs\n", " if sparse:\n", " assert weight_decay == 0.0\n", " self.model = model(self.n_users,\n", " self.n_items,\n", " n_factors=self.n_factors,\n", " dropout_p=self.dropout_p,\n", " sparse=sparse)\n", " self.optimizer = optimizer(self.model.parameters(),\n", " lr=self.lr,\n", " weight_decay=self.weight_decay)\n", " self.warm_start = False\n", " self.losses = collections.defaultdict(list)\n", " self.verbose = verbose\n", " self.hogwild = hogwild\n", " if random_seed is not None:\n", " if self.hogwild:\n", " random_seed += os.getpid()\n", " torch.manual_seed(random_seed)\n", " np.random.seed(random_seed)\n", "\n", " if eval_metrics is None:\n", " eval_metrics = []\n", " self.eval_metrics = eval_metrics\n", " self.k = k\n", "\n", " def break_grads(self):\n", " for param in self.model.parameters():\n", " # Break gradient sharing\n", " if param.grad is not None:\n", " param.grad.data = param.grad.data.clone()\n", "\n", " def fit(self):\n", " for epoch in range(1, self.n_epochs + 1):\n", "\n", " if self.hogwild:\n", " self.model.share_memory()\n", " processes = []\n", " train_losses = []\n", " queue = mp.Queue()\n", " for rank in range(self.num_workers):\n", " p = mp.Process(target=self._fit_epoch,\n", " kwargs={'epoch': epoch,\n", " 'queue': queue})\n", " p.start()\n", " processes.append(p)\n", " for p in processes:\n", " p.join()\n", "\n", " while True:\n", " is_alive = False\n", " for p in processes:\n", " if p.is_alive():\n", " is_alive = True\n", " break\n", " if not is_alive and queue.empty():\n", " break\n", "\n", " while not queue.empty():\n", " train_losses.append(queue.get())\n", " queue.close()\n", " train_loss = np.mean(train_losses)\n", " else:\n", " train_loss = self._fit_epoch(epoch)\n", "\n", " self.losses['train'].append(train_loss)\n", " row = 'Epoch: {0:^3} train: {1:^10.5f}'.format(epoch, self.losses['train'][-1])\n", " if self.test is not None:\n", " self.losses['test'].append(self._validation_loss())\n", " row += 'val: {0:^10.5f}'.format(self.losses['test'][-1])\n", " for metric in self.eval_metrics:\n", " func = getattr(metrics, metric)\n", " res = func(self.model, self.test_loader.dataset.mat_csr,\n", " num_workers=self.num_workers)\n", " self.losses['eval-{}'.format(metric)].append(res)\n", " row += 'eval-{0}: {1:^10.5f}'.format(metric, res)\n", " self.losses['epoch'].append(epoch)\n", " if self.verbose:\n", " print(row)\n", "\n", " def _fit_epoch(self, epoch=1, queue=None):\n", " if self.hogwild:\n", " self.break_grads()\n", "\n", " self.model.train()\n", " total_loss = torch.Tensor([0])\n", " pbar = tqdm(enumerate(self.train_loader),\n", " total=len(self.train_loader),\n", " desc='({0:^3})'.format(epoch))\n", " for batch_idx, ((row, col), val) in pbar:\n", " self.optimizer.zero_grad()\n", "\n", " row = row.long()\n", " # TODO: turn this into a collate_fn like the data_loader\n", " if isinstance(col, list):\n", " col = tuple(c.long() for c in col)\n", " else:\n", " col = col.long()\n", " val = val.float()\n", "\n", " preds = self.model(row, col)\n", " loss = self.loss_function(preds, val)\n", " loss.backward()\n", "\n", " self.optimizer.step()\n", "\n", " total_loss += loss.item()\n", " batch_loss = loss.item() / row.size()[0]\n", " pbar.set_postfix(train_loss=batch_loss)\n", " total_loss /= self.train.nnz\n", " if queue is not None:\n", " queue.put(total_loss[0])\n", " else:\n", " return total_loss[0]\n", "\n", " def _validation_loss(self):\n", " self.model.eval()\n", " total_loss = torch.Tensor([0])\n", " for batch_idx, ((row, col), val) in enumerate(self.test_loader):\n", " row = row.long()\n", " if isinstance(col, list):\n", " col = tuple(c.long() for c in col)\n", " else:\n", " col = col.long()\n", " val = val.float()\n", "\n", " preds = self.model(row, col)\n", " loss = self.loss_function(preds, val)\n", " total_loss += loss.item()\n", "\n", " total_loss /= self.test.nnz\n", " return total_loss[0]" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "iKeUFiRNPsFY" }, "source": [ "def explicit():\n", " train, test = get_movielens_train_test_split()\n", " pipeline = BasePipeline(train, test=test, model=BaseModule,\n", " n_factors=10, batch_size=1024, dropout_p=0.02,\n", " lr=0.02, weight_decay=0.1,\n", " optimizer=torch.optim.Adam, n_epochs=40,\n", " verbose=True, random_seed=2017)\n", " pipeline.fit()\n", "\n", "\n", "def implicit():\n", " train, test = get_movielens_train_test_split(implicit=True)\n", "\n", " pipeline = BasePipeline(train, test=test, verbose=True,\n", " batch_size=1024, num_workers=4,\n", " n_factors=20, weight_decay=0,\n", " dropout_p=0., lr=.2, sparse=True,\n", " optimizer=torch.optim.SGD, n_epochs=40,\n", " random_seed=2017, loss_function=bpr_loss,\n", " model=BPRModule,\n", " interaction_class=PairwiseInteractions,\n", " eval_metrics=('auc', 'patk'))\n", " pipeline.fit()\n", "\n", "\n", "def hogwild():\n", " train, test = get_movielens_train_test_split(implicit=True)\n", "\n", " pipeline = BasePipeline(train, test=test, verbose=True,\n", " batch_size=1024, num_workers=4,\n", " n_factors=20, weight_decay=0,\n", " dropout_p=0., lr=.2, sparse=True,\n", " optimizer=torch.optim.SGD, n_epochs=40,\n", " random_seed=2017, loss_function=bpr_loss,\n", " model=BPRModule, hogwild=True,\n", " interaction_class=PairwiseInteractions,\n", " eval_metrics=('auc', 'patk'))\n", " pipeline.fit()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "dPQaj1FjPsCo", "outputId": "42275f83-f4e9-43cc-a105-3ecde82efaa4" }, "source": [ "explicit()" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "Making data path\n", "Downloading MovieLens data\n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 1 ): 100%|██████████| 89/89 [00:01<00:00, 53.63it/s, train_loss=6.88]\n", "( 2 ): 7%|▋ | 6/89 [00:00<00:01, 57.03it/s, train_loss=6.06]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 1 train: 14.42120 val: 8.68083 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 2 ): 100%|██████████| 89/89 [00:01<00:00, 63.13it/s, train_loss=2.27]\n", "( 3 ): 8%|▊ | 7/89 [00:00<00:01, 62.84it/s, train_loss=2.23]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 2 train: 4.15028 val: 3.99969 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 3 ): 100%|██████████| 89/89 [00:01<00:00, 59.57it/s, train_loss=1.67]\n", "( 4 ): 7%|▋ | 6/89 [00:00<00:01, 59.43it/s, train_loss=1.33]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 3 train: 1.84903 val: 2.41240 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 4 ): 100%|██████████| 89/89 [00:01<00:00, 59.96it/s, train_loss=1.05]\n", "( 5 ): 8%|▊ | 7/89 [00:00<00:01, 61.59it/s, train_loss=0.982]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 4 train: 1.20266 val: 1.78271 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 5 ): 100%|██████████| 89/89 [00:01<00:00, 57.47it/s, train_loss=0.917]\n", "( 6 ): 8%|▊ | 7/89 [00:00<00:01, 62.99it/s, train_loss=0.861]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 5 train: 0.98022 val: 1.48147 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 6 ): 100%|██████████| 89/89 [00:01<00:00, 61.39it/s, train_loss=0.9]\n", "( 7 ): 8%|▊ | 7/89 [00:00<00:01, 65.11it/s, train_loss=0.77] " ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 6 train: 0.88477 val: 1.32482 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 7 ): 100%|██████████| 89/89 [00:01<00:00, 62.83it/s, train_loss=0.806]\n", "( 8 ): 7%|▋ | 6/89 [00:00<00:01, 54.86it/s, train_loss=0.766]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 7 train: 0.83306 val: 1.22818 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 8 ): 100%|██████████| 89/89 [00:01<00:00, 58.63it/s, train_loss=0.776]\n", "( 9 ): 3%|▎ | 3/89 [00:00<00:03, 25.32it/s, train_loss=0.722]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 8 train: 0.80015 val: 1.16457 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 9 ): 100%|██████████| 89/89 [00:01<00:00, 59.21it/s, train_loss=0.871]\n", "(10 ): 2%|▏ | 2/89 [00:00<00:04, 19.07it/s, train_loss=0.708]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 9 train: 0.77529 val: 1.12250 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(10 ): 100%|██████████| 89/89 [00:01<00:00, 60.45it/s, train_loss=0.749]\n", "(11 ): 2%|▏ | 2/89 [00:00<00:04, 19.87it/s, train_loss=0.735]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 10 train: 0.75322 val: 1.09408 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(11 ): 100%|██████████| 89/89 [00:01<00:00, 60.82it/s, train_loss=0.728]\n", "(12 ): 8%|▊ | 7/89 [00:00<00:01, 62.74it/s, train_loss=0.655]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 11 train: 0.73431 val: 1.06755 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(12 ): 100%|██████████| 89/89 [00:01<00:00, 64.48it/s, train_loss=0.729]\n", "(13 ): 8%|▊ | 7/89 [00:00<00:01, 61.52it/s, train_loss=0.706]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 12 train: 0.71816 val: 1.05441 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(13 ): 100%|██████████| 89/89 [00:01<00:00, 63.59it/s, train_loss=0.804]\n", "(14 ): 7%|▋ | 6/89 [00:00<00:01, 57.44it/s, train_loss=0.658]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 13 train: 0.70331 val: 1.04291 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(14 ): 100%|██████████| 89/89 [00:01<00:00, 62.10it/s, train_loss=0.648]\n", "(15 ): 7%|▋ | 6/89 [00:00<00:01, 55.63it/s, train_loss=0.662]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 14 train: 0.69230 val: 1.03409 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(15 ): 100%|██████████| 89/89 [00:01<00:00, 59.82it/s, train_loss=0.71]\n", "(16 ): 8%|▊ | 7/89 [00:00<00:01, 63.50it/s, train_loss=0.648]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 15 train: 0.68174 val: 1.02946 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(16 ): 100%|██████████| 89/89 [00:01<00:00, 63.41it/s, train_loss=0.762]\n", "(17 ): 8%|▊ | 7/89 [00:00<00:01, 66.62it/s, train_loss=0.6] " ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 16 train: 0.67185 val: 1.02574 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(17 ): 100%|██████████| 89/89 [00:01<00:00, 61.57it/s, train_loss=0.709]\n", "(18 ): 7%|▋ | 6/89 [00:00<00:01, 59.98it/s, train_loss=0.647]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 17 train: 0.66559 val: 1.01690 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(18 ): 100%|██████████| 89/89 [00:01<00:00, 59.60it/s, train_loss=0.657]\n", "(19 ): 7%|▋ | 6/89 [00:00<00:01, 58.13it/s, train_loss=0.609]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 18 train: 0.65754 val: 1.01814 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(19 ): 100%|██████████| 89/89 [00:01<00:00, 58.23it/s, train_loss=0.609]\n", "(20 ): 8%|▊ | 7/89 [00:00<00:01, 64.70it/s, train_loss=0.636]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 19 train: 0.65179 val: 1.01196 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(20 ): 100%|██████████| 89/89 [00:01<00:00, 58.38it/s, train_loss=0.693]\n", "(21 ): 8%|▊ | 7/89 [00:00<00:01, 68.79it/s, train_loss=0.607]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 20 train: 0.64911 val: 1.00926 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(21 ): 100%|██████████| 89/89 [00:01<00:00, 60.85it/s, train_loss=0.75]\n", "(22 ): 7%|▋ | 6/89 [00:00<00:01, 52.77it/s, train_loss=0.635]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 21 train: 0.64537 val: 1.01296 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(22 ): 100%|██████████| 89/89 [00:01<00:00, 59.46it/s, train_loss=0.702]\n", "(23 ): 4%|▍ | 4/89 [00:00<00:02, 39.91it/s, train_loss=0.588]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 22 train: 0.64303 val: 1.00838 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(23 ): 100%|██████████| 89/89 [00:01<00:00, 56.49it/s, train_loss=0.683]\n", "(24 ): 7%|▋ | 6/89 [00:00<00:01, 59.61it/s, train_loss=0.633]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 23 train: 0.63932 val: 0.99910 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(24 ): 100%|██████████| 89/89 [00:01<00:00, 58.42it/s, train_loss=0.709]\n", "(25 ): 7%|▋ | 6/89 [00:00<00:01, 52.67it/s, train_loss=0.594]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 24 train: 0.63549 val: 1.01004 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(25 ): 100%|██████████| 89/89 [00:01<00:00, 57.48it/s, train_loss=0.786]\n", "(26 ): 7%|▋ | 6/89 [00:00<00:01, 58.84it/s, train_loss=0.59] " ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 25 train: 0.63468 val: 1.00146 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(26 ): 100%|██████████| 89/89 [00:01<00:00, 55.84it/s, train_loss=0.64]\n", "(27 ): 7%|▋ | 6/89 [00:00<00:01, 58.98it/s, train_loss=0.603]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 26 train: 0.63316 val: 1.00257 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(27 ): 100%|██████████| 89/89 [00:01<00:00, 60.23it/s, train_loss=0.682]\n", "(28 ): 8%|▊ | 7/89 [00:00<00:01, 67.37it/s, train_loss=0.584]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 27 train: 0.63269 val: 1.00099 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(28 ): 100%|██████████| 89/89 [00:01<00:00, 59.51it/s, train_loss=0.721]\n", "(29 ): 7%|▋ | 6/89 [00:00<00:01, 57.41it/s, train_loss=0.573]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 28 train: 0.63194 val: 0.99549 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(29 ): 100%|██████████| 89/89 [00:01<00:00, 58.52it/s, train_loss=0.759]\n", "(30 ): 7%|▋ | 6/89 [00:00<00:01, 58.95it/s, train_loss=0.564]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 29 train: 0.63050 val: 1.00029 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(30 ): 100%|██████████| 89/89 [00:01<00:00, 59.03it/s, train_loss=0.718]\n", "(31 ): 8%|▊ | 7/89 [00:00<00:01, 65.42it/s, train_loss=0.563]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 30 train: 0.63016 val: 0.99232 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(31 ): 100%|██████████| 89/89 [00:01<00:00, 57.36it/s, train_loss=0.699]\n", "(32 ): 8%|▊ | 7/89 [00:00<00:01, 62.85it/s, train_loss=0.58] " ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 31 train: 0.63022 val: 0.99609 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(32 ): 100%|██████████| 89/89 [00:01<00:00, 56.56it/s, train_loss=0.743]\n", "(33 ): 7%|▋ | 6/89 [00:00<00:01, 59.53it/s, train_loss=0.576]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 32 train: 0.63043 val: 0.99635 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(33 ): 100%|██████████| 89/89 [00:01<00:00, 57.91it/s, train_loss=0.643]\n", "(34 ): 8%|▊ | 7/89 [00:00<00:01, 64.98it/s, train_loss=0.625]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 33 train: 0.63210 val: 0.99697 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(34 ): 100%|██████████| 89/89 [00:01<00:00, 58.12it/s, train_loss=0.641]\n", "(35 ): 6%|▌ | 5/89 [00:00<00:01, 49.84it/s, train_loss=0.546]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 34 train: 0.63177 val: 0.99458 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(35 ): 100%|██████████| 89/89 [00:01<00:00, 54.93it/s, train_loss=0.654]\n", "(36 ): 7%|▋ | 6/89 [00:00<00:01, 57.96it/s, train_loss=0.543]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 35 train: 0.63137 val: 1.00267 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(36 ): 100%|██████████| 89/89 [00:01<00:00, 58.59it/s, train_loss=0.742]\n", "(37 ): 7%|▋ | 6/89 [00:00<00:01, 59.93it/s, train_loss=0.553]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 36 train: 0.63002 val: 0.99718 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(37 ): 100%|██████████| 89/89 [00:01<00:00, 58.76it/s, train_loss=0.733]\n", "(38 ): 7%|▋ | 6/89 [00:00<00:01, 57.61it/s, train_loss=0.56]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 37 train: 0.62959 val: 0.99938 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(38 ): 100%|██████████| 89/89 [00:01<00:00, 59.98it/s, train_loss=0.638]\n", "(39 ): 8%|▊ | 7/89 [00:00<00:01, 61.75it/s, train_loss=0.599]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 38 train: 0.63083 val: 1.00133 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(39 ): 100%|██████████| 89/89 [00:01<00:00, 61.77it/s, train_loss=0.724]\n", "(40 ): 8%|▊ | 7/89 [00:00<00:01, 60.35it/s, train_loss=0.573]" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 39 train: 0.63185 val: 0.99541 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(40 ): 100%|██████████| 89/89 [00:01<00:00, 61.02it/s, train_loss=0.69]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 40 train: 0.63168 val: 0.99467 \n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "mtI0DewsPr_0", "outputId": "6cc428e1-211f-4e7e-8264-e8332ad47e8b" }, "source": [ "implicit()" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:477: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n", "( 1 ): 100%|██████████| 46/46 [00:02<00:00, 21.50it/s, train_loss=0.361]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 1 train: 0.42040 val: 0.40008 eval-auc: 0.55278 eval-patk: 0.00776 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 2 ): 100%|██████████| 46/46 [00:02<00:00, 22.72it/s, train_loss=0.298]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 2 train: 0.34066 val: 0.35044 eval-auc: 0.60807 eval-patk: 0.01164 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 3 ): 100%|██████████| 46/46 [00:02<00:00, 22.89it/s, train_loss=0.303]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 3 train: 0.27492 val: 0.31180 eval-auc: 0.65543 eval-patk: 0.01804 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 4 ): 100%|██████████| 46/46 [00:01<00:00, 23.75it/s, train_loss=0.192]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 4 train: 0.22703 val: 0.29160 eval-auc: 0.69006 eval-patk: 0.02694 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 5 ): 100%|██████████| 46/46 [00:02<00:00, 21.58it/s, train_loss=0.17]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 5 train: 0.19465 val: 0.27365 eval-auc: 0.71412 eval-patk: 0.03265 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 6 ): 100%|██████████| 46/46 [00:02<00:00, 22.30it/s, train_loss=0.176]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 6 train: 0.17487 val: 0.25775 eval-auc: 0.73276 eval-patk: 0.03973 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 7 ): 100%|██████████| 46/46 [00:02<00:00, 22.14it/s, train_loss=0.202]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 7 train: 0.16267 val: 0.25430 eval-auc: 0.74666 eval-patk: 0.04201 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 8 ): 100%|██████████| 46/46 [00:02<00:00, 22.22it/s, train_loss=0.17]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 8 train: 0.15176 val: 0.24547 eval-auc: 0.75858 eval-patk: 0.04429 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "( 9 ): 100%|██████████| 46/46 [00:02<00:00, 22.55it/s, train_loss=0.141]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 9 train: 0.14359 val: 0.23771 eval-auc: 0.76822 eval-patk: 0.04589 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(10 ): 100%|██████████| 46/46 [00:01<00:00, 23.32it/s, train_loss=0.151]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 10 train: 0.13715 val: 0.22593 eval-auc: 0.77713 eval-patk: 0.04361 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(11 ): 100%|██████████| 46/46 [00:01<00:00, 23.04it/s, train_loss=0.115]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 11 train: 0.13167 val: 0.22131 eval-auc: 0.78402 eval-patk: 0.04772 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(12 ): 100%|██████████| 46/46 [00:02<00:00, 22.63it/s, train_loss=0.134]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 12 train: 0.12781 val: 0.22118 eval-auc: 0.79055 eval-patk: 0.04749 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(13 ): 100%|██████████| 46/46 [00:01<00:00, 23.33it/s, train_loss=0.128]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 13 train: 0.12185 val: 0.21263 eval-auc: 0.79726 eval-patk: 0.05228 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(14 ): 100%|██████████| 46/46 [00:02<00:00, 22.32it/s, train_loss=0.109]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 14 train: 0.11865 val: 0.20135 eval-auc: 0.80326 eval-patk: 0.04977 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(15 ): 100%|██████████| 46/46 [00:01<00:00, 23.13it/s, train_loss=0.117]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 15 train: 0.11352 val: 0.20501 eval-auc: 0.80805 eval-patk: 0.05434 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(16 ): 100%|██████████| 46/46 [00:01<00:00, 23.17it/s, train_loss=0.113]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 16 train: 0.11156 val: 0.20189 eval-auc: 0.81208 eval-patk: 0.05753 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(17 ): 100%|██████████| 46/46 [00:02<00:00, 22.15it/s, train_loss=0.127]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 17 train: 0.10898 val: 0.19678 eval-auc: 0.81534 eval-patk: 0.05936 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(18 ): 100%|██████████| 46/46 [00:01<00:00, 23.03it/s, train_loss=0.13]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 18 train: 0.10363 val: 0.19250 eval-auc: 0.81967 eval-patk: 0.05890 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(19 ): 100%|██████████| 46/46 [00:02<00:00, 22.78it/s, train_loss=0.121]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 19 train: 0.10260 val: 0.18791 eval-auc: 0.82216 eval-patk: 0.06416 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(20 ): 100%|██████████| 46/46 [00:02<00:00, 22.97it/s, train_loss=0.121]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 20 train: 0.10081 val: 0.18382 eval-auc: 0.82357 eval-patk: 0.06370 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(21 ): 100%|██████████| 46/46 [00:02<00:00, 22.89it/s, train_loss=0.0978]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 21 train: 0.09957 val: 0.18360 eval-auc: 0.82604 eval-patk: 0.06667 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(22 ): 100%|██████████| 46/46 [00:02<00:00, 22.88it/s, train_loss=0.105]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 22 train: 0.09936 val: 0.17989 eval-auc: 0.82805 eval-patk: 0.06667 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(23 ): 100%|██████████| 46/46 [00:01<00:00, 23.03it/s, train_loss=0.102]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 23 train: 0.09896 val: 0.17684 eval-auc: 0.83031 eval-patk: 0.07123 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(24 ): 100%|██████████| 46/46 [00:01<00:00, 23.09it/s, train_loss=0.116]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 24 train: 0.09503 val: 0.18290 eval-auc: 0.83277 eval-patk: 0.06758 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(25 ): 100%|██████████| 46/46 [00:02<00:00, 22.64it/s, train_loss=0.081]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 25 train: 0.09565 val: 0.17506 eval-auc: 0.83462 eval-patk: 0.07511 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(26 ): 100%|██████████| 46/46 [00:02<00:00, 22.48it/s, train_loss=0.102]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 26 train: 0.09337 val: 0.17530 eval-auc: 0.83571 eval-patk: 0.07169 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(27 ): 100%|██████████| 46/46 [00:02<00:00, 21.46it/s, train_loss=0.0837]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 27 train: 0.09035 val: 0.17689 eval-auc: 0.83655 eval-patk: 0.07420 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(28 ): 100%|██████████| 46/46 [00:02<00:00, 20.81it/s, train_loss=0.0846]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 28 train: 0.08635 val: 0.17874 eval-auc: 0.83849 eval-patk: 0.07420 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(29 ): 100%|██████████| 46/46 [00:02<00:00, 21.13it/s, train_loss=0.107]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 29 train: 0.08961 val: 0.17910 eval-auc: 0.83905 eval-patk: 0.07237 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(30 ): 100%|██████████| 46/46 [00:02<00:00, 21.09it/s, train_loss=0.0935]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 30 train: 0.08822 val: 0.17294 eval-auc: 0.84065 eval-patk: 0.07717 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(31 ): 100%|██████████| 46/46 [00:02<00:00, 21.52it/s, train_loss=0.0926]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 31 train: 0.08964 val: 0.16762 eval-auc: 0.84098 eval-patk: 0.07466 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(32 ): 100%|██████████| 46/46 [00:02<00:00, 21.57it/s, train_loss=0.0708]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 32 train: 0.08982 val: 0.16215 eval-auc: 0.84217 eval-patk: 0.07055 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(33 ): 100%|██████████| 46/46 [00:02<00:00, 20.14it/s, train_loss=0.106]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 33 train: 0.08753 val: 0.16941 eval-auc: 0.84282 eval-patk: 0.07352 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(34 ): 100%|██████████| 46/46 [00:02<00:00, 20.73it/s, train_loss=0.0781]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 34 train: 0.08659 val: 0.17334 eval-auc: 0.84284 eval-patk: 0.07489 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(35 ): 100%|██████████| 46/46 [00:02<00:00, 20.66it/s, train_loss=0.0971]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 35 train: 0.08623 val: 0.17476 eval-auc: 0.84393 eval-patk: 0.07443 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(36 ): 100%|██████████| 46/46 [00:02<00:00, 20.77it/s, train_loss=0.0864]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 36 train: 0.08559 val: 0.17291 eval-auc: 0.84470 eval-patk: 0.07397 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(37 ): 100%|██████████| 46/46 [00:02<00:00, 20.11it/s, train_loss=0.0751]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 37 train: 0.08506 val: 0.16872 eval-auc: 0.84690 eval-patk: 0.07648 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(38 ): 100%|██████████| 46/46 [00:02<00:00, 18.27it/s, train_loss=0.0964]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 38 train: 0.08522 val: 0.16541 eval-auc: 0.84715 eval-patk: 0.07991 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(39 ): 100%|██████████| 46/46 [00:02<00:00, 19.55it/s, train_loss=0.0962]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 39 train: 0.08316 val: 0.16021 eval-auc: 0.84812 eval-patk: 0.07991 \n" ], "name": "stdout" }, { "output_type": "stream", "text": [ "(40 ): 100%|██████████| 46/46 [00:02<00:00, 19.17it/s, train_loss=0.0943]\n" ], "name": "stderr" }, { "output_type": "stream", "text": [ "Epoch: 40 train: 0.08459 val: 0.16542 eval-auc: 0.84809 eval-patk: 0.07237 \n" ], "name": "stdout" } ] }, { "cell_type": "markdown", "metadata": { "id": "tbXR1BvPWXKO" }, "source": [ "## Neural Graph Collaborative Filtering on MovieLens\n", "> Applying NGCF PyTorch version on Movielens-100k." ] }, { "cell_type": "markdown", "metadata": { "id": "sG-h_5yQQEvQ" }, "source": [ "### Libraries" ] }, { "cell_type": "code", "metadata": { "id": "3QJEhOlDwIR7" }, "source": [ "!pip install -q git+https://github.com/sparsh-ai/recochef" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "08tq9wC8pdD1" }, "source": [ "import os\n", "import csv \n", "import argparse\n", "import numpy as np\n", "import pandas as pd\n", "import random as rd\n", "from time import time\n", "from pathlib import Path\n", "import scipy.sparse as sp\n", "from datetime import datetime\n", "\n", "import torch\n", "from torch import nn\n", "import torch.nn.functional as F\n", "\n", "from recochef.preprocessing.split import chrono_split" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "F_-NBXzHS0_o" }, "source": [ "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "use_cuda = torch.cuda.is_available()\n", "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", "torch.cuda.set_device(0)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "uuM8Q9GlP8RN" }, "source": [ "### Data Loading\n", "\n", "The MovieLens 100K data set consists of 100,000 ratings from 1000 users on 1700 movies as described on [their website](https://grouplens.org/datasets/movielens/100k/)." ] }, { "cell_type": "code", "metadata": { "id": "Q-yPokpipXsY" }, "source": [ "!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "6CLWhSTXpbvW" }, "source": [ "!unzip ml-100k.zip" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "KS9OJY75pgwq", "outputId": "0c1ec91e-2ef6-4b5a-e613-fa4501e2737f" }, "source": [ "df = pd.read_csv('ml-100k/u.data', sep='\\t', header=None, names=['USERID','ITEMID','RATING','TIMESTAMP'])\n", "df.head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
01962423881250949
11863023891717742
2223771878887116
3244512880606923
41663461886397596
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "0 196 242 3 881250949\n", "1 186 302 3 891717742\n", "2 22 377 1 878887116\n", "3 244 51 2 880606923\n", "4 166 346 1 886397596" ] }, "metadata": { "tags": [] }, "execution_count": 5 } ] }, { "cell_type": "markdown", "metadata": { "id": "Mo6kTBolQYca" }, "source": [ "### Train/Test Split\n", "\n", "We split the data chronologically in 80:20 ratio. Validated the split for user 4." ] }, { "cell_type": "code", "metadata": { "id": "XTJpJj2uvpD2" }, "source": [ "df_train, df_test = chrono_split(df, ratio=0.8)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "Dpeu1H5xxXcR", "outputId": "d94c244d-ad25-47c1-d084-e51f6b015645" }, "source": [ "userid = 4\n", "\n", "query = \"USERID==@userid\"\n", "display(df.query(query))\n", "display(df_train.query(query))\n", "display(df_test.query(query))" ], "execution_count": null, "outputs": [ { "output_type": "display_data", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
125042643892004275
132943035892002352
220443615892002353
252643574892003525
327742604892004275
596043563892003459
1215142945892004409
1389342884892001445
163054505892003526
1893043545892002353
2008242714892001690
2038343005892001445
2451943283892001537
2474342585892001374
2486642103892003374
3531343295892002352
488264114892004520
5120343275892002352
6409143245892002353
6827343595892002352
7105543625892002352
7672243582892004275
8681543605892002352
8889143015892002353
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "1250 4 264 3 892004275\n", "1329 4 303 5 892002352\n", "2204 4 361 5 892002353\n", "2526 4 357 4 892003525\n", "3277 4 260 4 892004275\n", "5960 4 356 3 892003459\n", "12151 4 294 5 892004409\n", "13893 4 288 4 892001445\n", "16305 4 50 5 892003526\n", "18930 4 354 5 892002353\n", "20082 4 271 4 892001690\n", "20383 4 300 5 892001445\n", "24519 4 328 3 892001537\n", "24743 4 258 5 892001374\n", "24866 4 210 3 892003374\n", "35313 4 329 5 892002352\n", "48826 4 11 4 892004520\n", "51203 4 327 5 892002352\n", "64091 4 324 5 892002353\n", "68273 4 359 5 892002352\n", "71055 4 362 5 892002352\n", "76722 4 358 2 892004275\n", "86815 4 360 5 892002352\n", "88891 4 301 5 892002353" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
2474342585892001374
1389342884892001445
2038343005892001445
2451943283892001537
2008242714892001690
6827343595892002352
7105543625892002352
132943035892002352
5120343275892002352
3531343295892002352
8681543605892002352
220443615892002353
1893043545892002353
8889143015892002353
6409143245892002353
2486642103892003374
596043563892003459
252643574892003525
163054505892003526
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "24743 4 258 5 892001374\n", "13893 4 288 4 892001445\n", "20383 4 300 5 892001445\n", "24519 4 328 3 892001537\n", "20082 4 271 4 892001690\n", "68273 4 359 5 892002352\n", "71055 4 362 5 892002352\n", "1329 4 303 5 892002352\n", "51203 4 327 5 892002352\n", "35313 4 329 5 892002352\n", "86815 4 360 5 892002352\n", "2204 4 361 5 892002353\n", "18930 4 354 5 892002353\n", "88891 4 301 5 892002353\n", "64091 4 324 5 892002353\n", "24866 4 210 3 892003374\n", "5960 4 356 3 892003459\n", "2526 4 357 4 892003525\n", "16305 4 50 5 892003526" ] }, "metadata": { "tags": [] } }, { "output_type": "display_data", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMIDRATINGTIMESTAMP
7672243582892004275
327742604892004275
125042643892004275
1215142945892004409
488264114892004520
\n", "
" ], "text/plain": [ " USERID ITEMID RATING TIMESTAMP\n", "76722 4 358 2 892004275\n", "3277 4 260 4 892004275\n", "1250 4 264 3 892004275\n", "12151 4 294 5 892004409\n", "48826 4 11 4 892004520" ] }, "metadata": { "tags": [] } } ] }, { "cell_type": "markdown", "metadata": { "id": "qny2aVoWQknP" }, "source": [ "### Preprocessing\n", "\n", "1. Sort by User ID and Timestamp\n", "2. Label encode user and item id - in this case, already label encoded starting from 1, so decreasing ids by 1 as a proxy for label encode\n", "3. Remove Timestamp and Rating column. The reason is that we are training a recall-maximing model where the objective is to correctly retrieve the items that users can interact with. We can select a rating threshold also\n", "4. Convert Item IDs into list format\n", "5. Store as a space-seperated txt file" ] }, { "cell_type": "code", "metadata": { "id": "1iSOiyCqpmYE" }, "source": [ "def preprocess(data):\n", " data = data.copy()\n", " data = data.sort_values(by=['USERID','TIMESTAMP'])\n", " data['USERID'] = data['USERID'] - 1\n", " data['ITEMID'] = data['ITEMID'] - 1\n", " data.drop(['TIMESTAMP','RATING'], axis=1, inplace=True)\n", " data = data.groupby('USERID')['ITEMID'].apply(list).reset_index(name='ITEMID')\n", " return data" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "D7ZtrUPp22dO", "outputId": "56290e63-dcf3-448b-b3a5-ce60bd2c23db" }, "source": [ "preprocess(df_train).head()" ], "execution_count": null, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
USERIDITEMID
00[167, 171, 164, 155, 165, 195, 186, 13, 249, 1...
11[285, 257, 304, 306, 287, 311, 300, 305, 291, ...
22[301, 332, 343, 299, 267, 336, 302, 344, 353, ...
33[257, 287, 299, 327, 270, 358, 361, 302, 326, ...
44[266, 454, 221, 120, 404, 362, 256, 249, 24, 2...
\n", "
" ], "text/plain": [ " USERID ITEMID\n", "0 0 [167, 171, 164, 155, 165, 195, 186, 13, 249, 1...\n", "1 1 [285, 257, 304, 306, 287, 311, 300, 305, 291, ...\n", "2 2 [301, 332, 343, 299, 267, 336, 302, 344, 353, ...\n", "3 3 [257, 287, 299, 327, 270, 358, 361, 302, 326, ...\n", "4 4 [266, 454, 221, 120, 404, 362, 256, 249, 24, 2..." ] }, "metadata": { "tags": [] }, "execution_count": 9 } ] }, { "cell_type": "code", "metadata": { "id": "yDMAhrig1Lde" }, "source": [ "def store(data, target_file='./data/movielens/train.txt'):\n", " Path(target_file).parent.mkdir(parents=True, exist_ok=True)\n", " with open(target_file, 'w+') as f:\n", " writer = csv.writer(f, delimiter=' ')\n", " for USERID, row in zip(data.USERID.values,data.ITEMID.values):\n", " row = [USERID] + row\n", " writer.writerow(row)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "XUIFrKsavRzV" }, "source": [ "store(preprocess(df_train), '/content/data/ml-100k/train.txt')\n", "store(preprocess(df_test), '/content/data/ml-100k/test.txt')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "vq2IwCkJtUTy", "outputId": "bdd5df6b-213f-4e87-deae-b8f29e42ec87" }, "source": [ "!head /content/data/ml-100k/train.txt" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0 167 171 164 155 165 195 186 13 249 126 180 116 108 0 245 256 247 49 248 252 261 92 223 123 18 122 136 145 6 234 14 244 259 23 263 125 236 12 24 120 250 235 239 117 129 64 189 46 30 27 113 38 51 237 198 182 10 68 160 94 59 82 178 21 97 63 134 162 25 201 88 7 213 181 47 98 159 174 191 179 127 142 184 67 54 203 55 95 80 78 150 211 22 69 83 93 196 190 183 133 206 144 187 185 96 84 35 143 158 16 173 251 104 147 107 146 219 105 242 121 106 103 246 119 44 267 266 258 260 262 9 149 233 91 70 41 175 90 192 216 176 215 193 72 58 132 40 194 217 169 212 156 222 26 226 79 230 66 118 199 3 214 163 1 205 76 52 135 45 39 152 268 253 114 172 210 228 154 202 61 89 218 166 229 34 161 60 264 111 56 48 29 232 130 151 81 140 71 32 157 197 224 112 20 148 87 100 109 102 238 33 28 42 131 209 204 115 124\r\n", "1 285 257 304 306 287 311 300 305 291 302 268 298 314 295 0 18 296 292 274 256 294 276 286 254 297 289 279 273 275 272 290 277 293 24 278 13 110 9 281 12 236 283 99 126 312 284 301 282 250 310\r\n", "2 301 332 343 299 267 336 302 344 353 257 287 318 340 351 271 349 352 333 342 338 341 335 298 325 293 306 331 270 244 354 323 348 322 321 334 263 324 337 329 350 346 339 328\r\n", "3 257 287 299 327 270 358 361 302 326 328 359 360 353 300 323 209 355 356 49\r\n", "4 266 454 221 120 404 362 256 249 24 20 99 108 368 234 411 406 410 104 367 224 150 0 180 49 405 423 412 78 396 372 230 398 228 225 175 449 182 434 88 1 227 229 226 448 209 430 173 171 143 402 397 390 384 16 371 385 392 395 166 366 89 400 389 41 152 185 455 69 383 109 79 380 363 208 450 381 427 382 429 210 432 238 172 207 203 413 167 153 421 431 422 418 142 416 414 373 28 433 364 365 379 391 386 428 424 213 134 61 374 97 447 184 233 435 199 442 444 446 218 443 378 369 440 144 445 401 240 370 215 65 420 426 377 94 419 101 415 417 98 403\r\n", "5 285 241 301 268 305 257 339 302 303 320 309 258 267 308 537 260 181 247 407 274 6 296 126 275 99 8 458 123 13 514 14 136 292 533 535 12 116 284 220 474 0 256 110 476 245 507 470 150 297 283 124 409 532 236 457 293 471 459 534 404 531 472 20 475 300 307 63 523 7 513 97 426 164 222 134 78 530 509 177 486 176 135 88 49 526 204 512 480 461 186 191 46 168 173 519 317 483 488 497 142 70 11 478 190 496 479 491 503 511 495 210 468 524 196 481 198 529 55 536 499 473 68 520 203 506 31 179 182 518 489 188 22 193 463 494 510 460 184 165 174 487 485 69 132 528 482 215 521 522 434 192 462 431 492 237 493 58 208 130 21 94 502 527 86 490 316 467 505 155\r\n", "6 268 677 681 258 680 306 265 285 267 682 299 287 263 679 308 63 173 186 602 514 175 179 85 366 264 227 522 434 617 185 418 215 446 529 177 642 31 650 171 649 181 100 473 172 615 428 97 233 487 49 92 525 196 88 99 481 495 513 21 611 203 610 652 658 96 143 483 190 402 656 131 170 180 633 7 494 222 95 22 645 654 635 55 8 490 195 435 81 200 498 655 632 167 422 603 134 272 430 67 204 384 165 614 660 510 182 214 155 607 647 643 197 212 670 68 126 189 43 595 355 542 236 526 512 3 284 135 163 497 237 151 484 592 193 482 612 91 478 491 191 317 392 156 381 583 479 509 420 496 202 429 486 590 6 426 662 152 207 501 567 587 631 78 460 178 629 504 480 27 506 130 228 213 503 433 657 10 588 24 646 160 613 549 469 628 206 210 98 69 626 601 194 528 80 555 673 527 651 70 26 46 547 608 150 536 187 508 216 667 618 274 518 431 627 606 634 9 470 176 120 401 605 209 403 674 201 50 630 89 621 377 211 659 415 454 609 548 153 139 520 678 464 124 462 132 419 125 648 442 543 669 404 604 76 378 229 471 565 117 500 162 161 545 140 580 488 199 636 639 505 644 38 225 591 638 280 383 546 600 596 672 364 231 594 597 51 28 572 447 451 598 90 105 623 619 450 570 586 502 663 71 240 379 561 141 577 599 388 593 77 622 440 400 395 443 571 398 79 414 664 563 676\r\n", "7 257 293 300 258 335 259 687 242 357 456 340 686 337 688 650 171 186 126 49 384 88 21 55 189 181 180 95 173 510 509 567 176 10 434 175 182 402 143 54 227 78 272 209 6 194 228 685\r\n", "8 339 241 478 520 401 506 614 526 689 275 293 6 370 49 384 5 297 200\r\n", "9 301 285 268 288 318 244 333 332 653 526 429 55 512 662 63 31 173 126 492 193 152 557 58 185 517 701 610 628 417 473 384 692 602 706 155 504 655 587 485 663 22 11 403 530 685 370 81 222 123 474 49 708 190 272 609 487 220 705 174 274 696 10 177 21 710 700 167 204 650 184 605 181 233 15 510 697 0 479 196 460 159 115 477 134 178 156 8 508 495 197 601 47 194 210 651 175 3 98 133 68 136 284 356 154 462 97 434 501 496 217 691 199 481 482 215 179 273 163 169 497 69 446 461 99 469 275 132 466 654 483 588 191 478 413 128 202 704 12 198 703 160 694 518 702 656 520 603\r\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "E7YYuu2XuVQa", "outputId": "fd6fc81c-8ee7-4a34-cb7f-ae5c9ffb27d6" }, "source": [ "!head /content/data/ml-100k/test.txt" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "0 207 2 11 57 200 137 65 36 37 139 240 75 225 77 62 231 138 141 74 50 53 43 86 99 8 227 153 85 168 15 177 221 257 265 254 271 270 19 128 220 243 5 17 269 208 31 188 241 170 110 4 255 101 73\r\n", "1 49 241 271 309 303 299 288 315 307 308 313 280\r\n", "2 320 327 326 345 347 259 330 317 316 319 180\r\n", "3 357 259 263 293 10\r\n", "4 138 388 453 68 161 232 242 258 451 439 437 436 438 188 168 407 100 425 376 62 399 93 408 193 162 393 375 39 409 23 441 387 394 452 456\r\n", "5 516 80 133 498 194 466 418 131 504 207 356 465 199 172 187 212 273 422 169 525 484 508 515 501 201 469 366 185 500 153 477 202 167 424 18 85 152 27 517 538 464 271\r\n", "6 675 61 144 551 616 560 569 553 585 540 52 138 589 448 544 558 333 293 259 624 417 541 557 218 671 439 568 562 550 564 566 668 445 637 556 579 665 641 226 582 53 573 449 142 416 625 620 559 230 575 390 539 427 174 72 584 574 576 385 578 519 386 316 661 552 554 30 133 323 257 11 192 640 184 198 356 666 653 432 581 340\r\n", "7 549 81 187 221 430 683 517 226 240 232 565 684\r\n", "8 690 285 482 486\r\n", "9 143 503 59 616 709 431 494 92 509 524 488 229 529 161 6 237 614 695 581 282 699 693 366 84 39 419 711 707 528 182 698 131 32 498 320 293 339\r\n" ], "name": "stdout" } ] }, { "cell_type": "code", "metadata": { "id": "C-f9mzEf4Ow6" }, "source": [ "Path('/content/results').mkdir(parents=True, exist_ok=True)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "Bxz7aV0Sws1S" }, "source": [ "def parse_args():\n", " parser = argparse.ArgumentParser(description=\"Run NGCF.\")\n", " parser.add_argument('--data_dir', type=str,\n", " default='./data/',\n", " help='Input data path.')\n", " parser.add_argument('--dataset', type=str, default='ml-100k',\n", " help='Dataset name: Amazond-book, Gowella, ml-100k')\n", " parser.add_argument('--results_dir', type=str, default='results',\n", " help='Store model to path.')\n", " parser.add_argument('--n_epochs', type=int, default=400,\n", " help='Number of epoch.')\n", " parser.add_argument('--reg', type=float, default=1e-5,\n", " help='l2 reg.')\n", " parser.add_argument('--lr', type=float, default=0.0001,\n", " help='Learning rate.')\n", " parser.add_argument('--emb_dim', type=int, default=64,\n", " help='number of embeddings.')\n", " parser.add_argument('--layers', type=str, default='[64,64]',\n", " help='Output sizes of every layer')\n", " parser.add_argument('--batch_size', type=int, default=512,\n", " help='Batch size.')\n", " parser.add_argument('--node_dropout', type=float, default=0.,\n", " help='Graph Node dropout.')\n", " parser.add_argument('--mess_dropout', type=float, default=0.1,\n", " help='Message dropout.')\n", " parser.add_argument('--k', type=str, default=20,\n", " help='k order of metric evaluation (e.g. NDCG@k)')\n", " parser.add_argument('--eval_N', type=int, default=5,\n", " help='Evaluate every N epochs')\n", " parser.add_argument('--save_results', type=int, default=1,\n", " help='Save model and results')\n", "\n", " return parser.parse_args(args={})" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "twi1ZIucR0ga" }, "source": [ "### Helper Functions\n", "\n", "- early_stopping()\n", "- train()\n", "- split_matrix()\n", "- ndcg_k()\n", "- eval_model" ] }, { "cell_type": "markdown", "metadata": { "id": "aCShFbsCTPzw" }, "source": [ "#### Early Stopping\n", "Premature stopping is applied if *recall@20* on the test set does not increase for 5 successive epochs." ] }, { "cell_type": "code", "metadata": { "id": "tHVTudWxTVZo" }, "source": [ "def early_stopping(log_value, best_value, stopping_step, flag_step, expected_order='asc'):\n", " \"\"\"\n", " Check if early_stopping is needed\n", " Function copied from original code\n", " \"\"\"\n", " assert expected_order in ['asc', 'des']\n", " if (expected_order == 'asc' and log_value >= best_value) or (expected_order == 'des' and log_value <= best_value):\n", " stopping_step = 0\n", " best_value = log_value\n", " else:\n", " stopping_step += 1\n", "\n", " if stopping_step >= flag_step:\n", " print(\"Early stopping at step: {} log:{}\".format(flag_step, log_value))\n", " should_stop = True\n", " else:\n", " should_stop = False\n", "\n", " return best_value, stopping_step, should_stop" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "6JEG5Jlpw3Nw" }, "source": [ "def train(model, data_generator, optimizer):\n", " \"\"\"\n", " Train the model PyTorch style\n", " Arguments:\n", " ---------\n", " model: PyTorch model\n", " data_generator: Data object\n", " optimizer: PyTorch optimizer\n", " \"\"\"\n", " model.train()\n", " n_batch = data_generator.n_train // data_generator.batch_size + 1\n", " running_loss=0\n", " for _ in range(n_batch):\n", " u, i, j = data_generator.sample()\n", " optimizer.zero_grad()\n", " loss = model(u,i,j)\n", " loss.backward()\n", " optimizer.step()\n", " running_loss += loss.item()\n", " return running_loss\n", "\n", "def split_matrix(X, n_splits=100):\n", " \"\"\"\n", " Split a matrix/Tensor into n_folds (for the user embeddings and the R matrices)\n", " Arguments:\n", " ---------\n", " X: matrix to be split\n", " n_folds: number of folds\n", " Returns:\n", " -------\n", " splits: split matrices\n", " \"\"\"\n", " splits = []\n", " chunk_size = X.shape[0] // n_splits\n", " for i in range(n_splits):\n", " start = i * chunk_size\n", " end = X.shape[0] if i == n_splits - 1 else (i + 1) * chunk_size\n", " splits.append(X[start:end])\n", " return splits\n", "\n", "def compute_ndcg_k(pred_items, test_items, test_indices, k):\n", " \"\"\"\n", " Compute NDCG@k\n", " \n", " Arguments:\n", " ---------\n", " pred_items: binary tensor with 1s in those locations corresponding to the predicted item interactions\n", " test_items: binary tensor with 1s in locations corresponding to the real test interactions\n", " test_indices: tensor with the location of the top-k predicted items\n", " k: k'th-order \n", " Returns:\n", " -------\n", " NDCG@k\n", " \"\"\"\n", " r = (test_items * pred_items).gather(1, test_indices)\n", " f = torch.from_numpy(np.log2(np.arange(2, k+2))).float().cuda()\n", " dcg = (r[:, :k]/f).sum(1)\n", " dcg_max = (torch.sort(r, dim=1, descending=True)[0][:, :k]/f).sum(1)\n", " ndcg = dcg/dcg_max\n", " ndcg[torch.isnan(ndcg)] = 0\n", " return ndcg" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "sx-Vzl2vTeWN" }, "source": [ "#### Eval Model\n", "\n", "At every N epoch, the model is evaluated on the test set. From this evaluation, we compute the recall and normal discounted cumulative gain (ndcg) at the top-20 predictions. It is important to note that in order to evaluate the model on the test set we have to ‘unpack’ the sparse matrix (torch.sparse.todense()), and thus load a bunch of ‘zeros’ on memory. In order to prevent memory overload, we split the sparse matrices into 100 chunks, unpack the sparse chunks one by one, compute the metrics we need, and compute the mean value of all chunks." ] }, { "cell_type": "code", "metadata": { "id": "1dysqVKGTjm6" }, "source": [ "def eval_model(u_emb, i_emb, Rtr, Rte, k):\n", " \"\"\"\n", " Evaluate the model\n", " \n", " Arguments:\n", " ---------\n", " u_emb: User embeddings\n", " i_emb: Item embeddings\n", " Rtr: Sparse matrix with the training interactions\n", " Rte: Sparse matrix with the testing interactions\n", " k : kth-order for metrics\n", " \n", " Returns:\n", " --------\n", " result: Dictionary with lists correponding to the metrics at order k for k in Ks\n", " \"\"\"\n", " # split matrices\n", " ue_splits = split_matrix(u_emb)\n", " tr_splits = split_matrix(Rtr)\n", " te_splits = split_matrix(Rte)\n", "\n", " recall_k, ndcg_k= [], []\n", " # compute results for split matrices\n", " for ue_f, tr_f, te_f in zip(ue_splits, tr_splits, te_splits):\n", "\n", " scores = torch.mm(ue_f, i_emb.t())\n", "\n", " test_items = torch.from_numpy(te_f.todense()).float().cuda()\n", " non_train_items = torch.from_numpy(1-(tr_f.todense())).float().cuda()\n", " scores = scores * non_train_items\n", "\n", " _, test_indices = torch.topk(scores, dim=1, k=k)\n", "\n", " # If you want to use a as the index in dim1 for t, this code should work:\n", " #t[torch.arange(t.size(0)), a]\n", "\n", " pred_items = torch.zeros_like(scores).float()\n", " # pred_items.scatter_(dim=1,index=test_indices,src=torch.tensor(1.0).cuda())\n", " pred_items.scatter_(dim=1,index=test_indices,src=torch.ones_like(test_indices, dtype=torch.float).cuda())\n", "\n", " topk_preds = torch.zeros_like(scores).float()\n", " # topk_preds.scatter_(dim=1,index=test_indices[:, :k],src=torch.tensor(1.0))\n", " _idx = test_indices[:, :k]\n", " topk_preds.scatter_(dim=1,index=_idx,src=torch.ones_like(_idx, dtype=torch.float))\n", "\n", " TP = (test_items * topk_preds).sum(1)\n", " rec = TP/test_items.sum(1)\n", " ndcg = compute_ndcg_k(pred_items, test_items, test_indices, k)\n", "\n", " recall_k.append(rec)\n", " ndcg_k.append(ndcg)\n", "\n", " return torch.cat(recall_k).mean(), torch.cat(ndcg_k).mean()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "mvvKJOlpSLn4" }, "source": [ "### Dataset Class" ] }, { "cell_type": "markdown", "metadata": { "id": "AzoIqHHuUADD" }, "source": [ "#### Laplacian matrix\n", "\n", "The components of the Laplacian matrix are as follows,\n", "\n", "- **D**: a diagonal degree matrix, where D{t,t} is |N{t}|, which is the amount of first-hop neighbors for either item or user t,\n", "- **R**: the user-item interaction matrix,\n", "- **0**: an all-zero matrix,\n", "- **A**: the adjacency matrix,\n", "\n", "#### Interaction and Adjacency Matrix\n", "\n", "We create the sparse interaction matrix R, the adjacency matrix A, the degree matrix D, and the Laplacian matrix L, using the SciPy library. The adjacency matrix A is then transferred onto PyTorch tensor objects." ] }, { "cell_type": "code", "metadata": { "id": "s0w9GTdKw7Vj" }, "source": [ "class Data(object):\n", " def __init__(self, path, batch_size):\n", " self.path = path\n", " self.batch_size = batch_size\n", "\n", " train_file = path + '/train.txt'\n", " test_file = path + '/test.txt'\n", "\n", " #get number of users and items\n", " self.n_users, self.n_items = 0, 0\n", " self.n_train, self.n_test = 0, 0\n", " self.neg_pools = {}\n", "\n", " self.exist_users = []\n", "\n", " # search train_file for max user_id/item_id\n", " with open(train_file) as f:\n", " for l in f.readlines():\n", " if len(l) > 0:\n", " l = l.strip('\\n').split(' ')\n", " items = [int(i) for i in l[1:]]\n", " # first element is the user_id, rest are items\n", " uid = int(l[0])\n", " self.exist_users.append(uid)\n", " # item/user with highest number is number of items/users\n", " self.n_items = max(self.n_items, max(items))\n", " self.n_users = max(self.n_users, uid)\n", " # number of interactions\n", " self.n_train += len(items)\n", "\n", " # search test_file for max item_id\n", " with open(test_file) as f:\n", " for l in f.readlines():\n", " if len(l) > 0:\n", " l = l.strip('\\n')\n", " try:\n", " items = [int(i) for i in l.split(' ')[1:]]\n", " except Exception:\n", " continue\n", " if not items:\n", " print(\"empyt test exists\")\n", " pass\n", " else:\n", " self.n_items = max(self.n_items, max(items))\n", " self.n_test += len(items)\n", " # adjust counters: user_id/item_id starts at 0\n", " self.n_items += 1\n", " self.n_users += 1\n", "\n", " self.print_statistics()\n", "\n", " # create interactions/ratings matrix 'R' # dok = dictionary of keys\n", " print('Creating interaction matrices R_train and R_test...')\n", " t1 = time()\n", " self.R_train = sp.dok_matrix((self.n_users, self.n_items), dtype=np.float32) \n", " self.R_test = sp.dok_matrix((self.n_users, self.n_items), dtype=np.float32)\n", "\n", " self.train_items, self.test_set = {}, {}\n", " with open(train_file) as f_train:\n", " with open(test_file) as f_test:\n", " for l in f_train.readlines():\n", " if len(l) == 0: break\n", " l = l.strip('\\n')\n", " items = [int(i) for i in l.split(' ')]\n", " uid, train_items = items[0], items[1:]\n", " # enter 1 if user interacted with item\n", " for i in train_items:\n", " self.R_train[uid, i] = 1.\n", " self.train_items[uid] = train_items\n", "\n", " for l in f_test.readlines():\n", " if len(l) == 0: break\n", " l = l.strip('\\n')\n", " try:\n", " items = [int(i) for i in l.split(' ')]\n", " except Exception:\n", " continue\n", " uid, test_items = items[0], items[1:]\n", " for i in test_items:\n", " self.R_test[uid, i] = 1.0\n", " self.test_set[uid] = test_items\n", " print('Complete. Interaction matrices R_train and R_test created in', time() - t1, 'sec')\n", "\n", " # if exist, get adjacency matrix\n", " def get_adj_mat(self):\n", " try:\n", " t1 = time()\n", " adj_mat = sp.load_npz(self.path + '/s_adj_mat.npz')\n", " print('Loaded adjacency-matrix (shape:', adj_mat.shape,') in', time() - t1, 'sec.')\n", "\n", " except Exception:\n", " print('Creating adjacency-matrix...')\n", " adj_mat = self.create_adj_mat()\n", " sp.save_npz(self.path + '/s_adj_mat.npz', adj_mat)\n", " return adj_mat\n", " \n", " # create adjancency matrix\n", " def create_adj_mat(self):\n", " t1 = time()\n", " \n", " adj_mat = sp.dok_matrix((self.n_users + self.n_items, self.n_users + self.n_items), dtype=np.float32)\n", " adj_mat = adj_mat.tolil()\n", " R = self.R_train.tolil() # to list of lists\n", "\n", " adj_mat[:self.n_users, self.n_users:] = R\n", " adj_mat[self.n_users:, :self.n_users] = R.T\n", " adj_mat = adj_mat.todok()\n", " print('Complete. Adjacency-matrix created in', adj_mat.shape, time() - t1, 'sec.')\n", "\n", " t2 = time()\n", "\n", " # normalize adjacency matrix\n", " def normalized_adj_single(adj):\n", " rowsum = np.array(adj.sum(1))\n", "\n", " d_inv = np.power(rowsum, -.5).flatten()\n", " d_inv[np.isinf(d_inv)] = 0.\n", " d_mat_inv = sp.diags(d_inv)\n", "\n", " norm_adj = d_mat_inv.dot(adj).dot(d_mat_inv)\n", " return norm_adj.tocoo()\n", "\n", " print('Transforming adjacency-matrix to NGCF-adjacency matrix...')\n", " ngcf_adj_mat = normalized_adj_single(adj_mat) + sp.eye(adj_mat.shape[0])\n", "\n", " print('Complete. Transformed adjacency-matrix to NGCF-adjacency matrix in', time() - t2, 'sec.')\n", " return ngcf_adj_mat.tocsr()\n", "\n", " # create collections of N items that users never interacted with\n", " def negative_pool(self):\n", " t1 = time()\n", " for u in self.train_items.keys():\n", " neg_items = list(set(range(self.n_items)) - set(self.train_items[u]))\n", " pools = [rd.choice(neg_items) for _ in range(100)]\n", " self.neg_pools[u] = pools\n", " print('refresh negative pools', time() - t1)\n", "\n", " # sample data for mini-batches\n", " def sample(self):\n", " if self.batch_size <= self.n_users:\n", " users = rd.sample(self.exist_users, self.batch_size)\n", " else:\n", " users = [rd.choice(self.exist_users) for _ in range(self.batch_size)]\n", "\n", " def sample_pos_items_for_u(u, num):\n", " pos_items = self.train_items[u]\n", " n_pos_items = len(pos_items)\n", " pos_batch = []\n", " while True:\n", " if len(pos_batch) == num: break\n", " pos_id = np.random.randint(low=0, high=n_pos_items, size=1)[0]\n", " pos_i_id = pos_items[pos_id]\n", "\n", " if pos_i_id not in pos_batch:\n", " pos_batch.append(pos_i_id)\n", " return pos_batch\n", "\n", " def sample_neg_items_for_u(u, num):\n", " neg_items = []\n", " while True:\n", " if len(neg_items) == num: break\n", " neg_id = np.random.randint(low=0, high=self.n_items,size=1)[0]\n", " if neg_id not in self.train_items[u] and neg_id not in neg_items:\n", " neg_items.append(neg_id)\n", " return neg_items\n", "\n", " def sample_neg_items_for_u_from_pools(u, num):\n", " neg_items = list(set(self.neg_pools[u]) - set(self.train_items[u]))\n", " return rd.sample(neg_items, num)\n", "\n", " pos_items, neg_items = [], []\n", " for u in users:\n", " pos_items += sample_pos_items_for_u(u, 1)\n", " neg_items += sample_neg_items_for_u(u, 1)\n", "\n", " return users, pos_items, neg_items\n", "\n", " def get_num_users_items(self):\n", " return self.n_users, self.n_items\n", "\n", " def print_statistics(self):\n", " print('n_users=%d, n_items=%d' % (self.n_users, self.n_items))\n", " print('n_interactions=%d' % (self.n_train + self.n_test))\n", " print('n_train=%d, n_test=%d, sparsity=%.5f' % (self.n_train, self.n_test, (self.n_train + self.n_test)/(self.n_users * self.n_items)))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "J2RxhIxmSYvl" }, "source": [ "### NGCF Model" ] }, { "cell_type": "markdown", "metadata": { "id": "P4vz1IOwTvED" }, "source": [ "#### Weight initialization\n", "\n", "We then create tensors for the user embeddings and item embeddings with the proper dimensions. The weights are initialized using [Xavier uniform initialization](https://pytorch.org/docs/stable/nn.init.html).\n", "\n", "For each layer, the weight matrices and corresponding biases are initialized using the same procedure." ] }, { "cell_type": "markdown", "metadata": { "id": "wo9vgNvJUWvR" }, "source": [ "#### Embedding Layer\n", "\n", "The initial user and item embeddings are concatenated in an embedding lookup table as shown in the figure below. This embedding table is initialized using the user and item embeddings and will be optimized in an end-to-end fashion by the network." ] }, { "cell_type": "markdown", "metadata": { "id": "ntRUCJGMUeNf" }, "source": [ "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASEAAABICAYAAAC5kUM9AAAgAElEQVR4Ae2daVhUV7rvz5f74T733A99zj3dT/p0x3S0p6QTTxJN2k6cYsc4xChx1hijxiHGeR4jzjOKAwqIoOIMIiAKIiLzXMxTMRbFUExFUUVBjft3n12AIuJAGoLEzfPUQ9Xae7/rXb937f9ea+219/o3pD+JgERAItCDBP6tB/OWspYISAQkAkgiJFUCiYBEoEcJSCLUo/ilzCUCEgFJhKQ6IBGQCPQoAUmEehS/lLlEQCIgiZBUByQCEoEeJSCJUI/ilzKXCEgEJBGS6oBEQCLQowQkEepR/FLmEgGJgCRCUh2QCEgEepSAJEI9il/KXCIgEZBESKoDEgGJQI8SkESoR/FLmUsEJAKSCEl1QCIgEehRApII9Sh+KXOJgERAEiGpDkgEJAI9SuAXK0KNufe45O7CqVOncLkWhUJrwNKjqKXMJQJdRECoIyPoEmdPn+D4cSe84srRGoQuMv7zm/mZRchCScRFju/ZyqaNG9mwYUObz2ZOBBZS12TtAgoCNV7LGPb5fA55R5BRUkOj2UrvDVMXIJFM/IIImNBVK8mJvcLG8R/xtWM8Sk3vvcT+zCIk0FirIGjbaN7u898MXX2Fe1EyZDF3cPp+BKM236ao1tAFYmGl6sI3/M+Uw0QV1GISJPn5BZ2BUlFsBAQsBhVXFw5g8qFoitXmXsvlZxYhkZOZ9MNj+MOv/p0hOxNQ1pmwmBqpTjrGUns/FNX6LhOhAXPOkFWh6wJ7vTa+kuO/aAJNBK7+mGkOkgh1MswWMh2+4A+/+r8M3S2jrN5MVbQ/IVmFZBepMZq7ojvW3BIaOOcM2S8qQnoFMT5nObpvHw6ufsjKGjB1hSudpNM1u+tRxN7E/eg+9jm44isrQ2d8iQujLyH2pnsLe1+SSnW8zO52FCN9SRy+7o7s3+eAq28SpVoj3U+8icA1kgh1FI/npLWK0P/hrck7OHLKma3TZ7A3XIna2HG3yVKVSfjd2/j7+eHX9nM7jkKdgScbop0TIaEugTPLJjJ1zRn8A1xZMXYQw+eeIq5C14Ht5xSvpzcLdSScWc6kqWtw9buF68ov+MfwOTjFlqM19bRzT+YvsndbMZlpa1zwu+XKyi8+Zvick8SUa7G5K9SSGXqPpJJ6ml7KYQ+BusSzrJwyjdXOvvi7rmLcx8OZcyKa0vpm4II6i9B7SSjqm7r45ogkQk/WqBdKaRWhf+e9+c5c9fPn0MxJbL+voPYpItSU/4Cr587gfPo0p9t8nD1Dye8wsJ0RITM5Z2cz8I13mLT7An53b2L/xRv812tjOJRYhrbLKr6AWuaFs1sgWbX6x8VNUCPzcsYtMIta/eOSKqhleDm7EZhVS7tNHdI257gz56M/8O7EXZz3DeLm9nG8+evXGH0wntL6LitMh3l3PtFMjvtcPnrzXSbuPIdv0E22j3uTX782moNxSmzuWvLxd9jHxdgKNJ29AySoSfZ2wS0wk5qGtlwF1MneuLgFklnT8EQsHh3zAqptzsFj3iD6vjuRHR43Cbq5gy/7/obfjtpPbInGJjqWglsc2X+RmHINnS3Cs5lKIvRsPk/d+kiEBttHkVdeR+E9b+7laZ56klkbNVRXqqioqHj8UyVeHTu669UJEbIquTD7bX7z6w+Zt/8kbp4XcD68k207zxJWWo/BakKjSCejWIPBLIBVTV5mIeomUyeb2xbk5xYxetxGfApqaGzb6LPIObdoNOM2+lBQ0/jYGJZFfo5Fo8ex0aeAmscO6giwlVLPb3nntV/z4dx9nHTz5ILzYXba78QtTEl9kxWTRkF6ZjF1BhMCVtT5mRSpG3um62ktxXPOO7z2m4HM3XsCtwvN7O13uvFAqaHJrEOVl0Cw/30yVA2d76JZ5JxfPJYvN3qTV92WqwX5+cWM/XIj3nnVT8Si9Rh51fPHJ62lF5nb/7f8ZuAc9hw/w4ULzhzeac9OtweUaJow61TkJQbjfz8Dla6ru2iSCHV0FrxAWqsItY4JWbE0atEZm8gODCBRpcPQrjOtT77EznXL+X7RIha1+Xy//gzRKi1PNqA6IUKWPE7a9eX//WoIW++mkl9eSWVlJZVVdehNGrJDLnNgwWi+PRZnG78y557l+0XHiFLWdZDvs4ovoK/IJS2tgOrGdgIm6KnITSOtoJrGdgNRgr6C3LQ0CqpfRCgs5DlN5I//9SsGbwkiJb+8uSyVVdTpjWiyQ7h8cCFjvj1GbKkGkzkX9x++51hkCXVPQnxWYbpmmyUPp0l/5L9+NZgtgSnktWUvTqkwqUg5u5yxM/YTolB3vhUhNKKSp5FeUNWOq0CjSk5aegFVT8Tiacd0XGRL/ikm//nX/Mcnm7kty6NcrDuVlVTV6TFbBUyqVNxXfMHM/cFddOe3rR+SCLWl0YnvFjIcxrYZmG5WHGtVMNumreZKfhUNbVsJgKW+FHlWBmlpaY9/shSoDeYOWiSdECFBS+imwbz+q9cY/uNt5GoDVksNyQ/iUGh1aCqC2frZxyzyzKRSb0bpOZdPF54jraIBq9BASYKMwoam5vELkYK5gqRb1wlIVKI1dF/3x6xK4tb1ABJLtDzKRkAbupkhff6D14ZtISCnFoPVQk1yGHEKLVpNBfd+HMkniy6QXtGAufQi8/65CI/UcnRiGAxKQo+vYpH9NdKqmrsp5qo0Lu3bgZNPDIU64zPHNMwqWQc+PaNqCFoebBnKG//xGsO23CKntpl9Slg8Cm0jZque5MPjGTT3DLIyHRYE9MpEkgt1ND7sKZlRyQLwCkigpL4bJ6SaVcgCvAhIUFD/CDiC9gFbh/+B/3xtKJv9sqlpsmKpSSEsXkG93oxVn4LDhI+Z6yoOVjfXB0GdzbWdi9jokUh5y7gR+kwubV3L3nP3yalrQmx0P//vSRFqUsYT/CCNcq2hg/Pi+RZ7Yo+f+Ra9hZJIT7aO/yO/+t//i//+eCaLlq5izeplzB33AW8MXIF/UU3nr3hPkOuECCGgzbnBjxPfp2/f9/nn5DksXLiY9c5hKOsNmEvO8c2grzmRVIZGncjRyX9nylFxfEVLScJ17L9Zy9XCR8JpVfmy5rMP6D96O8GKzraWnijIUxKsqHzXMvKD/ozefpdStfHhfoI2Bx/7SXzQty/v/XMycxYsZPF6Zx6UiN3JEs7P/pivjydQWqcm0XEKg6YeJU6paR4XsTaQcHg2Qz4azKLzMlQNFoSmbFx2OxKWVYL2mRM+m336fEB/RtvfRak2PPTp6V8EtLk+2E/+gH793mPE5DkssLEPRaExYLWWcXn+YGY4xlKiMdsEyGvHt6y7kodK13KWWivxWzeKAf1HYx9UhLprB10eum6t9GPdqIH8z6htBBbWPqqjgpbcm9uZMqAf/d4bwaRvF7Bg8XpOhxbbJt5ay66wYMgMjkYrqGsdlmqIwnHqQN6Z5oSsTIsVPTlBB/j673bsCcmkwtDREMNDV9p8aS9CZjJd5jDi0yVcyFI9cTFvc+BL9fVnFiFxsmIpeWnxxERHEZuUSnpGJpmZ6aQkxhKTUkSdwfzYmMhPo9UsQuI8ocwXuUVvaUCVn0b0XT9u3n5AnCwNeXk9RouALnA1g8bYE5KfTbS3PXZ/G4n9nTRKGo00qNM4NusHzudVoG05JwRjFTmRV9iw7BARpW0q608ryFOOEjBW5RB1ZSPLDkVQUdv2hLfQoMonPeYufjdv8yBWRpq8nHqjBUEXxJp/jMU+WE52jDfbv3qHz+0DSFXosYgtIaEaP6eTeDnNZ/QXq7iYUklDfQjHHO9SUq1/zpW1xaerok/hlD/m01OKISbb2KcT08I+tg17oe4Oq4ZMYGdAMkU6C0a9mvQTs1lyPpey+pY+u2CkKieKq5uWcyi8hJpuEiExrrlRV9m0/BDhJY9fKC0NKgrSYwj28+X2g1hkaXLKNQbMgoDmzmqGTdjJLVkhWovYlgNDsidO+xYwcNBKvHJUVGffw+/WQSZ/8gOXMspo1ddnUGvZ1H6ekIBOEUfI/ZTubRU+37FO7fEzi1CnfPsXdhZFaBb9Jx62XeVftFNkNRtoMpiwPJxhLT7+sYj3PhjDoq0n8b64hqF/Gcqc/TdIVYvdhVLc5y/DM/+RCCHoyA2+yMXAdFQN4uBv9/wJulyCL10kMF1Fg6mDXKxmjE0GTBbhoQ9CjTffvz+AMYu2csL7ImuH/ZVhc/bhnVJLo9j+b4zE5fR9FMVpeG+wY/yaSySGHOHAZTmVL3BnTfTp3qVLT/fpGSisZmM79mBKOcjYd4cxa7snceLAtGCl7NwiVnjKH4kQAjr5PS5fCiStQtd9A+yCDvm9y1wKTKOiwwFmK2ZjEwaThYfVBxOph8fRf/gs7C/E2o4TMCC74MqdJFe+HTSenX4BXPF+QEbYASZ+tYt7eVWPWlnP4GXbJNRwdf4AJh+MejRjWrBgElusHVSJ55nrqe2/WBGqvraYQQNHMH3ZNhw8Iyj+iQ+wmmpyiQmPJiVHQXVlNtERiWQVVqIzWRE6EiGMaKsqqNYam1sX3RVZo5aqimq0RstzWihtHDDVII+JIDolh+KqSrJjIkjKLKRSJwovmDPdOXGjgGqtCV2BL5snjmfO9DnsDC2l9kVaGD/Fpzbutf8qaOTERCSQkd/SksNKqUd7EQKjtoqKai1GW3OuvZWu+v1T4ipQL48lIiGDfFvL2gpNSZx3CaSgIpkTUwcyct5eriSVUnNvM6O/c0bWclv/mV6LD7AGenJy1xLGDXyLiYfjUda96KX2mZZ7ZOMvVITAVFNAqkxGSmYuBUrxSv+i/ez2cRCwmC1YxUuLYMEsTgmwXWWs1GRdZ+Wno1h5ORalvrXVI/w8VyHhUQunvcfP+i1YzFis4rFtygXoy1Lw2z6PZacjya8zYrU2UOi/lQkjFnMuQ9U8cP0sw+K2n+jTU80+xhusNdl4rf6MMSsvEqVooLUBKPwsl/2fFtdHvIGmClKubmLFofvkVdcStnsu6zyikCvyuL97PAMm7+eeXP0CUxFMaCuVFMkzSE5KIlucYf5iI9lPRd2TG36xItT9UAWM9eXkpqSQW6a2zVfq/jy7LwdLYx0V8kxyleKkyGbBtjaUkibLpUrf1fNbflo5BKOWcnkqKbllqJs60QL8adl1/VHWRupK88gtUdNosqArK0BZ14jJ2EBVQTpJGcXUiHfUuj7nl9qiJEIvdXh+TufE5p3Y2ns8T8HaLuHxzdKvLiLQQr+LrPUuM5II9a54Sd5KBH5xBCQR+sWFVCqQRKB3EZBEqHfFq+e9lXpnP1MMXh3Qr7wImYrCCYwvRftyvifiZ6rwL5aNtUZGUFgu6i6a/2QqjiAoXkl9Y++9vfxi5Dq5l7WG5Lth5NbqOvl8YifzeUl2f+VFSNAreHB2P/tdbyEr7c0vMuvGGmWppzDyMg67jnMzpcw2R6orchP0JYS5H+CAqz9Jyt73IrOuYPC4DQv1RVFcObKb4z4ySrXtHnR+fOdfzK9XXoTAgrYslUDXbSyYNoVZi1azdY8DTm6eXPP2wdfPhwvOLpy/4YuPpzPO573wPneKo04e3PQ6h7PzeXy9z3H6qBMe17045+zMeS8vzp06ipPHdbzONR/j1XKMj+8NLnleITDoHuExSaTLlagbzbzYTSgTuqpictMSiA67z51rl7ly0x8fz9M2f3xa/fF6cX+uu59sPtbm1yX8fH3xu+mD95ULnDlxmN2bVzB36iRmLtvHhfuZVDR05Ykh3qZOJfCMPQunN7Pfsvvwk+y9fbnh6YLLeS+8zp3G0ckDsawuLufxFcvq6ITHNZG9SzP70444eVyzsReP8b3hiYvrJe4EhRKdlI5cqX7xeWNmHVUKOWkJ0YQHXcfNzZMbvjfwdOk4r1b/rrmffMxPW12x+XmTG5c8uXTDl5u+vvh4X+HCmRMc3r2FFfOmMunrZew9H0JGuXhBfDW6ZJII2a4nZhoqC8mIDuDC8Z2sWTiLSWM/Y9jHA3nv3bfo26cPb/T7I/3eeJ3X3+hL3z6/47e/60O/vn14/fU3bP9//9vf0efNvvT5/eu80bcvfX73W37X50369vn948f060fft/rx57/+hbfefof+7w1k0NBRTJq7mr1ng8hQ6R9/9EDQUxJ/E+ddK5ht9xlDBg3gvf5/4+2//pU/9fsbffv9mX5v/P4n+/Nma1lEv97ux1/efpv+H3zIJ8NGMm7ybBav38upq0HEZiiofWGx7MxF2kxDVREZ0bfxPL6TtYu+YdIXIvsPeb+Vfd9+/xL7fn3f4PXX/8Cf/vxX3n6nP+8NHMSQUZOYu3oPboHpTwiroFcS7+vC7pXfYjdyCIMGvEf/v73NW39+kz593qBvv3688fqz4/yQ6xN1pB99+75Fvz+9xdvvvs+Hnwxj5LjJzF68nj1OVwiKSae4Vnwk6NUQILGmSCLU9nyxGtDWlFOcl01GiozE+FiioyKJiIggPDy8yz4RURGEh90nJDiI277XOO96kqMHdrJ55QJmfbsap/sFaAxWrJp0vPYsZd53i1mzdRcHjp7E7aIXfrcDCQ4JJSw8iojwrvNN9CsiMpKomFjiE2WkZuSQr6iwjQF1+znRyj4/p9vYP7gfQnDQbXyvncfV6SgHdm5m5YJZfLvaiXv5dbYJp5p0b/Yum8f8xavZsusAR0+64Xndj9uBwYSEhnVNHYiIIiIiksioGGLjE5GlZpCTr6CiVnxx26sjPq2nniRCrSR+7v+CgNVqwWxqQq/VUFtVjiI/i+RIP9wOOHA9IQGv4w64XA0kSpZFQUkFVbUadHoDJvExklewsv6rIRIEK1aLGVOTHq2mlqpyBflZyUT6n+WgwzXi4704ccSFq3cikWUVUFJeRa1Gh95gsj2uIyH/VyPQ8fGSCHXMpedSzQ1UFSUSdOM6vuGvXtO8J8Cb9dUUJQVxw8uX8LQiarql29kTJesdeUoi9DLGSTBSX1XViQHrl7EQvcsnwVhPVZU4YN38zp/e5X3v9lYSod4dP8l7iUCvJyCJUK8PoVQAiUDvJiCJUO+On+S9RKDXE5BEqNeHUCqARKB3E5BEqHfHT/JeItDrCUgi1OtDKBVAItC7CUgi9JLGz1BTTHr0fRIKdTS1rlf1Qr4K6CvzSI4IJ6Ws4bGFETtOfyGjv+CdDNQWpxMdmkChtunxdemfV2pBT1VeChHhKZTq2i68KKCvyiNFjEGprk0Mnmfw1dwuidBLGXcBbUEQe7/+kg3eBVTrOzOV30pt1jW2TJrBvhBxhdrWY9unWykLOYXDNRmV2keLJ3YLDms5IaePcC2pkuasftpL47vFN0FH4d39fDN+A9fzKukc6lqyr29l8sy9BD+2aKeV2uzrbJ08k73B4uTHMu47H+Vakor6bkbdLYy62agkQt0M+Keat+jlOE39mO89c1C1Xxf7OUZN9bHsGTuaLXcKqH4oQtA+XZt+m5tRxWi6+30+gpb0275EFWlotJjJvb6NfdezqWxZFvk5xenmzRb0eaeYPngxFzLLO7HwoOiWCW3cPsaN2UKAvIqmVr0Xt2jj2DduDFsC5FQ11ZNxRyx/HfpOtWq7uegviflXRoQsdUpyszLJF59SN6lR5maRma9CLy5WJ1aamgJkMdHEpRVT27K+vaBXkZucSHpxLU1mK1ZTPaVZOSjraynOzLItVWwWTNQUyIiJjiOtuBajuf1aCQJ6VS7JieIjGE2YrVYMagUZuWXotdUUpSWSodRgMDWgypWRkKFsXi3VquL87KEsORdNVGQ8KYU1Nh+a6017my21SWigLDOemJibbPxsJJtbRaij9CYj9cUyotNKaWjUU1eSSW6ZnobqYtLiE5FXtnma31BNXlI00SlysiNvcMEnltL6JgwPmRVR0/T0VSIEUz0KWTRppQ00akrwWjmYLzddJTiuCLV4nFWPKjeZxPRiam2/DagVGTZ/tNVFpCVmoNQYMDWoyJUlkFFSj6GjJW6EdnawYqgrIbMta3E57IesS6g3mLGoLjBn+BI8xIdK41MorGls8yYDkbWclKR0imxPt7dEoKGcrPgYYm5u4vNRmx+JkNBAeVZzDDZ9PorNAXIqtfUokmNIU+poMhoestZVF5OekIhcXNix9b1uImtZjI11VtQNPH1iUWqaMNQWkhwbTVzqs1m/JLrSKTdeHRGqkuGydDxL3NJQaSpIPrMcu6VnSVfpMFWGcNj+OL7BAZxcu4sbRTXUFd/F+fgFbgf7c3L9chzvF5KfcJFNU6byw/ZdbFq6BMdQJfKAQ9gf9yX4lhNrd3mjqm5oEwAzirsunLgQQLD/SdYvdySkMIdI15VMmLmGw8dcOOu4iunTV7D32GnczzqwfNo8jkWUomlScf6bAYz94QDO513ZtmgBu29mU6M3oAhutenEhhWOhBSpaTIUE+R0kNNeQYT4HmLa/wxl7a18qnUdp1dVZOD14wymbg+gIDsKj7UTmbHqEMdOueNsP5dJK86RotJhacjg0p5DXLr3AP9jy/lu6yku306lqiiQwzuOc/PuLZzW7ca7oOopa5MJqDO92DZzKttvFaIqknHq2/cZsfgY5/xklNblEehyggsBd/F32sAKx3vkZ0fiusqOmWsOcczlLI6rZjBjxV4cT7tz9sgKpn/nSFhJHY/1bMwlBLuefMxOYU0x8W6rsft6NYdaWM+YsYK9jiLrI6yY/h2OYSXUllzg2w+/4Id9pzjnas+ihbvxyaymwWKi5J4rJy8EcNffiY0rHblXUIu26C6nDp3meuA9fA/N4P2ha/ETW0JGBXdPHeL09UDu+R5ixvtDWeuXS26SF/ZfT2O7fwGViiTOr5/EzFUHcTzljov9PCav8EBWrsXckMHlfYe5GByK//EVzN96iosBKVQWBXJk5wl8gvxxWr8H7/xKtO2vdW1qXW/7+sqIkGCqxWfJh0w+kkippola32X8fepRZKUajFknmPz5Qk4GJpMSFkuupoLAHRP5av4W9h89wvoJHzB0gz/5BeHs/eJDZh0JJDohjcKaBtKOTebzhScJlKUQFpuLvvHRqSFo7rNz0kTmb9nHkSMbsPtgKOv9Msnx28Cwf67iYmQahfJLfP+PiewKiCNbkcuZbwYx3yOLCm0F57/5mDnHH5BWpCT51ByGzTxKRIYP20Sbm1tsDhBt5lFwZyfTFh/jQVYp6spwto8axeY7eRQE7uogXVxltQ7ZQTuGr/Imt6SU0G0j+XSZB2EpBRRH72PCP9fhK69Cm3SESVN3EpRdSnnoNsZN3MXtzAp0mSeZOnoRJ+4kkSwyq2986qtITRoZh+w+ZZV3Dqq6GgLWDGbS3hDSiuqovLeDSRPns3nfEY5ssGPAsPX4ZmTjv/FTPlvlSXhqAfLLi/lk0i78Y7JQ5Lrx7ccLcM8oQ/uw+yOgCd3NlEnz2by3jR25EuXdTYwYuQrP8FQK5JdZPHgSu/xjyFLk4vbtxyxwz0CZf545g+dy7H4KhcoUTs/9lK+PhFNcfJfdUyYxf9NeW/y+GjiMdT4JXP9xBj843idDqaYyYgdjRostoUoqH+xhxg+O3M9Qoq6MYMeY0baWUKlKxuGJI1h1PZsKdQ1h20fzz2XuhMryKY7ez1cj1+GTU0m9zJGp03dyJ7OE8gfbGT95F7fSy9BlOjF97CKOBySSHB5Hrkb/VNa9TYBEf18ZEQIDwWs/ZrqjjDKNBcP9DQyecZyUsnosulx8j6zju8ni1fcsMco4DowfwkKXMGQ5hchTY4jLrqSpSY7TlBGsuSGnqlE8AwR0ub4cWfcdk+1msvpsNNW6RyJkznRkwpCFuDxIIqdQTmpsHNmVehri9zB6wi5Ci2oxGh6wedg0jiQo0VgaubX0I75xzqBMU9HcHbvUPCbUFL6F4SM34ee/lTFDF+Lc1qaqntRjExmy5BLZqgYESxaOdl/wY2AOkYc7ShfHiiwUu85g5LqbyKt0pDlM4IttQRTUGLCWuTNr+HKu5qjQlnixxO57XMJTkXmuYPLKi6RU6DDrcvE7uoH5U75i5mo3osq0Tz8xLApcZ37Oupu5VDY28WDzcGY6JqGsM5B57CuGLXQmNDGbQnkqsXHZqPQNxO8di92u+xTWGDGEbeHT6UeIU6gxNwWw/B+zOZ2qpP5ha8BM5rGJDFt4mvtt7TQYaUrcxxd2uwgpqMFoCGPLiOkciVOgNjcRsPwfzD6dSon8vK075pkljgk1EfHjCEZtukXWg4NMHL6I0yEJZLfGrzSJIxOHscQzg/IGAUv2cSaN+5Hb8nJSTk5h2BJP21sRBUs2xyeN48fbcqr0xZz5ejTrbohlM5N+9Cu+3HaHvKomrGXnmD1iOVcyy6kv8Wb5pMU4hyYj81zF1FWeyMq0mHVybjlubGF9hsjS+qez7oUq9EqJ0P0Ng5l4MBZlnRl94Cr+McWBRGUdalk44ZkZJEVcZt34Cey4Hc6hiR/wxfYgCtQGrIIVo9GMYM7j5OSRbPDLp1ochRS0yMLDycxIIuLyOsbb7aBEWfuwGlgUZ5k5YBzbg/JRG8S164225XqNCbsZNWE3YcW1mAwP2CSKUHwJdZZG/JcM5JtT6ZRpytuIkJXK64sYtfgcyXHHmD5wHNsD86h9aNNMifssPpx6hBilBrMplUPjRrHplpzk0193kC76b6HIZTqfrfVBXllP6uHxjNt2t1mESt2ZNXQpl7JV6KpvsXPdIc6JL1PzDyIyR0WDyUy9LIKIzHQSI66wfsJX7LhTSE3bkdmHFABLES4zRrL2RqsIDWOqQxyKOiMK0e9x9tzJq8VgFbAajZgFIwl7xmC3O/ShCA2f5kBssRpz4y2W/X02TilKNA9FyGIr/0fj7Lktr2ljR8CYuIexX+0itFWEPp2OQ2wxanMjt5b9ndlOKSjaipC1Eq8fxrDYI5mSTDe++ehL7G/nUtPK2lCCx+xBTHWIpLjOjCnVgfFjNuEnryDv3GwGTaDKfY0AAAZUSURBVHUgsrjOFgOH8WPY5CensqHQJsJrvUURMpJ2xI7x2wLJqxIF/xyzP13KxcxytNUB7Nl4CA/P6/j5BxKZXUGD0Ux9cmQz68grbPhqIjsCWupfW8a9+PsrJEIWiq8uZtT4H9hx5DSu2yfz3oAp7LiRSvGVTSze7cHVq8dZOn0p7okKMm7aM33UBOZvPcgJFw/8EgvJeXCcOe//kU9/OM29fC0GSx2+Py5mt8dVrh5fxvSlZylVaR9VB6OCW9tnMNpuPlsPnsDFw4/EwixCDkzh7b+NZ5tvCmn3dmH3l/eY5nCXjJQANn3+Jh/OOkFIfg6uc4Zgt+Io5y66sn/zDs6EF1GnLeLWjmabW1psJpRq0RX4sHHyBOZtPsBJF0cWDH6X0SvdeRDlyZqJT6ZHJIZy8rsP+eOnS3C+fo3dU9/lrbFbuJFSgMxzCZ/0G8zCM+EoCq+wZtYCVq9Zy4YtOzns4kO8UkulzzZ+2OPOlSsnWDZjKWcTytB2eOfHQnm8Kws++hMjlrgRpqgj8fAEhtgtZqtTMJmp3mybORq7+Vs4cMIFD78ECrNCODj1Hf725Y/cTE7j3u6v+Ov7UzkUmEZywBZG9/uQWceCydc8Wp7aqAhg58wxLXac8fBNoLQmn/DD03jnnS/50UdG6r3dTHzrfaYeCiQtOYAto/vx4axj3I08wewhX7HCwQNP1wNs2XGGsAI1TU0Kbu/8mjF237H5wHGcPXxJUNaQc2MTU+zmsWn/SVwcFzKk/2hWuEVQmn6dzVPsmLdpvy0GC4f0Z/SKM/j4HmXu3//MiB/OEBwVxMGv3+PtsZvwkuUju7iMIX8awgKXBxQVXGXd7IWsXt2GdUk9lTe3s3TPWS5fOcHymctwi1NSb3rYF31U33rpt1dIhMBUW4AsKpKE9FwKcuIJj0gmp6yexopckpNTbJ+U9Hyq9CZMunJyEsIJDY8hOauAUrWeevEuSUwk8akFqLQmLIKZKnkyySniJ4X0/CqMpoeXZ7EJgK48h4TwUMJjkskqKEXdoKFCnkx0VCLZZXWoK3KRRccgy1OhqS0lKzGa2OR8Khu0FKbEIpMlkZqZTXZuScs7npttJra1Kb6Ey1yPMjOB6LhksvPzkUU9ID5TQW2DGkXGk+k1tRXkpcYSGZ9GgUJBtiyaqIQsyup01BRnEB8ZR2pRNeoUd/YccOeK9y1u+3vjcWAZ83b4I89ORfYYMyOWDs8LAX11AamxkcSnFdnY6pRJhIXHk55Xga6pnrKcRMJDw4lJzqKgVE2DpgJ5cjTRidmU1ampyE0mOkZGnkpDbWk2SdGxJOdX0tByZ9N27ll0VOQkEvHQTi2NRi2VeclERz9indyGdXZSM2tVbT7JsTJkSSlkZmeTW1Lb8iJ8C7qKHBIjWuKXX2qLgbG+lMyEaOKSs8jLlxEVFk+GeGfUoKG0JQZZeWIMwojPKKJUmUuKrfyFqKrLyE2JsbEurdM2s46KI7WwCnWqB/sOunP5MdZ+5GSl2uqXWEfF+lnZ8DTWvVOFXikREkMkWMRXdYqT5SxY2uiFYDFiMIjC0uZMEiyYDEbbS8dtqYJ4XOunNeACFqMBg3hCtDm0dWtrngajuKJG8w6PbLTaavuf5jwQsFqsNn/F2/7tbYvlaGuzOT8Bi7k5H4ul7cu5Okh/WI62ebf/biL52BxWuMWTX16PTquhLPooq/YGoqhuxCr60J5Z24K3fH+svLY0K2bx5WEPeQlYTGJXtbWc7f1o95vm309m1d5OC8unlrV1uxWLtflY8QXzD91q9d9isnXHW+NnSxYsmG3+WrDY6lOrN+1YPzXvdmUSTKQc/46VZ2KRl7WwjnFkzd47FFXqX5h1qxe96f8rJ0K9KTg976uAKsSBlUs3sOekB5e9buDjd4eonEoaH2vx9bynvd8DAdX9o6xetoHdJ9y5fL2FdbY4l63N1bL3F/SJEkgi9AQSKaEtAUuDivysLHILiilRllJWWffia3a1NSR9fy4BkXVBtsi66JViLYnQc6uGtINEQCLQnQQkEepOupJtiYBE4LkEJBF6LiJpB4mARKA7CUgi1J10JdsSAYnAcwlIIvRcRNIOEgGJQHcSkESoO+lKtiUCEoHnEpBE6LmIpB0kAhKB7iQgiVB30pVsSwQkAs8l8P8BQIzxTVPvPikAAAAASUVORK5CYII=)" ] }, { "cell_type": "markdown", "metadata": { "id": "HKqah7XFUhan" }, "source": [ "#### Embedding propagation\n", "\n", "The embedding table is propagated through the network using the formula shown in the figure below." ] }, { "cell_type": "markdown", "metadata": { "id": "ZvrR6lvUUuIm" }, "source": [ "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXUAAAApCAYAAADKxrnwAAAgAElEQVR4Ae2dd3iUVdr/33/23d/qirvrvjZcxV1R0bXRV6SDFOlSVYr0IiBShFBCCb2ELtJbSEJIMKRDAmlAAiG99zLpk2QyM5k+z+d3PZOZZBKSEAUbO3Ndc81TznPK99zP99znvu9z5n+wfWwI2BCwIWBD4LFB4H8em5bYGmJDwIaADQEbAthI3SYENgRsCNgQeIwQsJH6Y9SZtqbYELAhYEPARuo2GbAhYEPAhsBjhICN1B+jzrQ1xYaADQEbAjZSt8mADQEbAjYEHiMEbKT+GHWmrSk2BGwI2BCwkbpNBmwI2BCwIfAYIWAj9ceoM21NsSFgQ8CGgI3UH1MZEBRJ3LpXgk5nfExbaGuWDYH/PgSM0hgikirR6IRmG28j9Wah+R3fEGREnT/EpXgpGkPznf87buFvt+qGMlJTi9DqDaCWkp0QQVBYEpqaXBKTS9HobYPsb7fzfqWaGcpJSy1Co6uVmZzESK6HJaFW5pCUXIpavG7+CKoMvI6e56ZEjrYZUXpoUjeUpZBSpMNgBHV5NgkRQYQlV6DOjSepVGu6bqmQ7feXQUCTeJLVmz1Jq9BgfJw4vY4wRWGzEGYyWq3+JwCrpyAxmVKNnmbejZ+Qp5KEi4dxjiqhRicgVKdzafUYhix1QyorIeb8AS5ElaLS/5Y7xUB5aipFWh0G1EhzEom8Hk6SVM1PqbZekkRyqZpfbcLYFGGGJ6HW/DSZkSQlU6rWPVKZSbx0BJe7RSh0RoTqDDzWjuOTpa6UVhQTe+EQzneLUVo0c0FDYeAu1h6NoFihoylJejhSV8bjcuAC98o0GASB6rSLrBw5iGUehShK73F2nxPRUvHeT3g/bI/8NAQEKQGrp7HOLw+59nECvp4wRVIUCdPNbjRDll6iQq5pJVZ6KgsKkekNCBipijnPgQv3KFPpm3w5WplpXTJ18hm+XX2G6OIaEwEK2kLOftGBARtvIlVoUaScY+U3R7lbXnu/7sHf0IEywY3vnKMoVuoQhGoy3Nfw6dCluOXIULdSnPRVEgplepNCZ6yKxengBaJKarDw0i/XXCWJbkdwiRJJsZYw3deO5ZOlFymtUreyGnqqJIXIdAaMoszEOnHowl1KRHxamUNLyTQp57Bbe4YoicIsM0Wcn/oOH28IoaRaiyLVCbtlR4ksUdThZ5TdZNtsO9zTK1A1oZE8BKmrSTq5lJVn4yhTiy+JgFZymomv9cMhUopKJyf5zHIWH42mSlM/fWipgbZ7D4+AUeLKrGHfcCVPzuPE6SbCtDtDjDVhfv4mAzbdpEL5IK1LQJl7k0v7lzLpS0duiaQKGBUpnPt2CUejpI9Ae1YStmkC888kU6Eyy7vmNut6fsRS3wKqxc7Q53N+3qes9c3/bQ646hTOrlzDmXtFKEW1XNBSeG4ybw/cSFiR3IRZSxIqKHO55X6AZZ9NxzG8rFa7NCpIOb+KpUfvUqZ8NINnS3WwvqdOOcuqNWe4V6i0Isx/8/GGUErlrZGZW3gcXM7n0x0JM5OqKDNOdks5eqe0Xnu2LvRHHSsJ3/IZX52Op6zGIjORbOjbm6VeOVRpRJkp4MJX41nnbT4X8xfkhK77lJknYihT3s+tP53UFSHYj5nL2fTqOtuO5tZqundfytXiWhD1eWeZNXINAcW/xij9o9B9TBIbkVyYyn+mnCJDpn6EU8RfGx4FYRvHmwizUm1WTUTC/KiHiTDlD1QBBdRlOST5rqBnj2+5WixHZ2qSnvzzcxmz1o8ChfbhNC91OGv7jsUxphzLe6ZP3c/wLjNwyqiittp6Mg98Sj+7q5TLtb82qPeVrwx3YOKC0yRKVdRShYaI9b3pudSHPNmD8RHUZeQk+fFt756s8C+k2txEfb4T88euxTevGpGnfpmPkvDNk1hwOpHyOsKMYEOfXvWE2WJFzDLjt5K+vVbgVyAzK0l6Ci4sYNw6H3JlmoeTGc1N1g8cz56oEixjjD7tEKO7z+RsitSshevJPDSegXZ+FFdZZqQClT/Mp8/n33GvVGnuq/rG/GRSV4euoseovSTWkYeelL1D+GC6M7lyswDoMtg7ohd2QeXUPGhgrK/TL3sk6FCU5ZEYcYvkMu3v21QklHFpxrsM2BxZL8h1aBqpTIwmQ6H7ddtoKCQuOhelTpzdtfKjDmdNn09xjJVieT9FwhzWeQZOmVVompiCNs7ZaDCgy9jH8D6rCCyxkDroMw8wpp+dFdE3frJ158aCY4zrOAvnvCqzmcJIyYWpdBx7kHhpjfnFE1D+MIf3JxyhtFLVuox/sVRqbq7rz7g90fXanz6VAyO7MvN8OhWWwbSl+hgNGHSZ7B/Zj1VXi+pIHX0mB8f2xy5AJPpW93pLJT34nvom9v3HsedeKQqzMqtPO8iobjM5l2ohzJazMclM5gFG97cjQCLDMgzrMw8xbqCdFdG3nE9zd42SE0zqOhunLCkqEyxGSlym03XcfqJLFPUyc2U+XSYdRlKmrMvKkHWI0Z2+4GhcWd07YbnZNKkbCrjpcoQ9Wzfj4ODQ8Lv1NGElNWR/N4p3Zlystb2JuRmLOf/Zu4w5lILM8pYJSjxmvM247zORP/QQLaBMucrpgzvZtnUrW3cdxPlmAaqHjSZQZRJ0eAmjeo1l5+1KLDNnC0At/grVJPmf4uCubWxx2MTWPYc4cngfjoddCc+SoX+Al1JQpnD1zCF2bdvK1q27OOh8k3yVxXFnpDTqEkf2bGfr1q3s3H+Sq6k1aA1gLLvH5WP72LFtKzv2X+ROsdmJpQ5mRae3+PJCHtUN8BZQJLqw64A/mXJtixq8TlZASnQU8VllKPXG1hNvU0Dp8wi7cATH7dtMbdj93WWiSsqIPr+DAwFZKEzuewMFty7yveNWNjeWNYetnAopQpH9PWM7zsQ5X2bW9ETCnELHTw+SUKEy2TpLo/3xcHPB2dnZ6uvCpdDMuugBQ9Z+RjQidUF5mdnvj+dIagWK3DDOHdrFdlG+dh7ENUKCojCKH47vZ6elj9wiKVTpMBrLiblykgM7t7P74BkCA7cx+P25uBVY6qjg6pKufLwpnBJFvUaj8VnAu6P2UVJR0xRij+aaXkZBSjT34jMpVepMznKD5DZuRx3Ztvn+d3rrqRAk8iyOje/MrAs5VJqN58YSZ6Z1HsuB2NrZh7E0hoDLbrg0wNgZl0uhZCi1tSRkyOLgqH7YWZO6oMRzbkcmfJeMtKYVI/ADUTAgibjEsb3b2LK5ET85bOFksAR51jEmdJmFU3Zl/SDr/CVdxu0nxqTdGimNCeBykzKTgcpsMhbJc0wjUheUnszrPJHDicWkhDrx3Z7tJk7aedCV2wVyCu95cvJAPU9djJCg1BoxSmPxPnWQXdt3c+B0EMlxexjRdR6uORXmOioIXPYhQzaGIKmut9lrfBfRcYwjklJFHTKC8ipLunRinmt2XX9ZbjZN6oKC4hQPlvdpz8sv9+dbt2vcCAsj0H030z4axMbwMhJ3DeadOZdNQmPKTO7Pwo792RQhtSJGDV5zOjBifxrVrRnpLbVq8ldAV5lPwqVl9HnjHcbvvEZcvvyBxNlkVtYX9dVIbm6k3/NdWRn8I2cUgpbKvHhcv+nNG50msz/wDrEJUfjtm8vIT1fhnt582JFYBUFXSX6CO8v7vsm743dyLS4feR2RCqhK0/G3H0KHt0ayyS+KzEq96QUVVGVk3TvDvL5j2eQTQ76y9rohfT9DX/6INWEiIVs1siae44sXsfdGDtXNhSEISlI9d7JyzT5cvLxw3raAmfbupMg1903vrHJu+dAopyjJjaV93uTVfw5j49V4JDU6FFme2M9ehVtKNRqjgKI4hR++7c+b7V6m/3IX/K+HERbozp4ZvRmyIZii+D0M6ziXSxLL9F3B1a+78vHGm5SYGmqkLD4Ib8/LeHh4WH0vc+V2Tm2oGNAUqaPxZv47o9iXVIGyuog412/o3+FVXum3Ep80KRplGVmhe5jU8TVe6fAZ+25nUyX2kaCmPN2Zxf1HseZSJBnZznzZ8QtOZFbWal26KDb17cESj2jSS8RBwNTjSM9PpuscV0pllql0yxD+uLsCytQr7F61ln3OV/By3s7CWfZcSpKhqi4m1XMVA01tW4aTbxBhYYF4OM6k75D13JDE4TiyM3Mv5lFlJnXFtW/oPqjenm4siyfIx5PLDTD24PKVW2TXtEDqaPD56n1G741HqmyB1I1qqgpSiIqMIjmvApVB9NQ19RFQlqTiafcxb736Cv2WOuEbJMqMB44z+zJ0/XUK4hwZ2WUuF3MtpK7g2tIPGbwxhEITYRopT7iOT1Mycysblag9mWTmflJH48vCjmPYG1dKsSQet2UDefuf7ei74grJZWqUZVmE7f2cLq+3o8PEPYRlVKAzCgjqctJdlvDx6NW43kqnvNSN2d2mcDy1HNNYp7vHloG9WOJ2h9RiS+SgQMWFaXw415lCqdXsTp/K3qHt6W8fbGqPNUpNk7qYQh/Dph7P8Kc/9WN3ehU1ej0aeQHBm6Zi51uO5PwU3vnsNPlmW6Tu7np6dl+CZ3wWpRZiEqScnfgBs9wkKB7RtEuftpsBz73JbA8xBKjpLrduYGuOjSXHGf3Sh9iF/khSN2WuJ2lHf55/bzG+JUrEYKeazMOMefU1Jh1Pr5+1NFcRfRp7Pn6BDrPdKbwvREmg6OhoXnz1c87myxvaI7VRbBw+mwt1miGoAxbx9osj2J9useGKhRrIdZnHoGmHiS1XNWt6UUYd5IsP+zLn5G3SC4tI/P5LBs89RUyl+qeTuli8NhL77v/Hk20+4XC22TRhrCTUfgTDN1xFYooi0BO7pTfPPfkEfXckUa7QodfKkYRuYcZqH0rynJje8QtOZpmftxDm5WgyTIQpoKuRUVkhRSq1/laYImOMQq2cGDL2MqzXylpTi1l0BOl5vugyG9fcWg3bUO7D4k7P8OQLoziSWGGyhQs1iewa/CJt2nRi5Y0SLIq3LmE3I4evIyhPjk6fzfGJg/jWbGIwFp/ls/admbjxLOESlTmkT0nIqmFMP51MVVNhC83JSGuvK+9xeMpH9Jt9nPBUCUWJR5kxdC4n74kECfq4rfR74c882Wc78aXV6PRa5JJQts1cg3dRLk4zOzP5RAYVJluAjqjN/floiTtRaSXoDEYEXQ2yyopGGEupqJCjFknLJG4Z7B/Rh5UBknpTi1CB05TuzHHJuk+rrG2agYo4D/Y7bGTr7gMcOXqEg7s3s9Z+Lz/El6FuMpZST9y2/rz41JP02RpHSbUoMwokYduYtdabwhwnZnWZzPF0aUPCvHSX1GIxBLslmVFTLzMHGNVvJf51NnUQKi4w9T9zcc6sQCUYKPdbQtdnn+KFEYeJKxdNbQKqpD0Me/kvPN1xBYGSarMPR0ei4xhGrbtKdpUWwZDDyS8+YaVvnskpaiw5z+QOXZm4/hSh+TVmX6WS0DUjmXkqAanF9iiCJlThPO113px2hqwKVYPBrwVSj2Nzz2f40xMD2JstR2Mo57Z3MOlZcaSU6tBmfM/Y/t8SWCaGZxkpPjWOdh9MZLPTbYo0Zi+34gbLB00zOVMtFpnWymdz6Qy5Bxn6wjss9Ba9z02n0pUlcsPTBScXbyJyquu0eWN1DhG+bpw7c54rkfkozIOPsfQEYyykrsoj0qSNuHP5WhzZCTfw8ryMu7s7/jElaPQGSmIC8LzsgYdvFBK1hvR9g3mx03KulSlrIwRkrkx+5QWGH0hDVjdD0VGWeIMrLk64eEeQU12rXWPI5dCwF3l3oTclivsbJD0znrbtp+NS1CiaRZ/IzgmL8CixkL2RohOf8tI/J3OuTqMFdEkcGNmJ8Yfjm7eLCpX4Le3CC69O4FSGzKQ91+THEpUkQS6GgjUNM+gz8DjhTaaiBW1ed5f1H4qkPpwjuZawOAHFrbX0776QS9nVaAU98dv68tyTT9J/dxpVKgPlEb6EpGUSm1KKVpPJMTNhik5RY/EZJr3WiYmbRMJU0xoLnLEkFq9d43mz3UCWnQkj12zmUoas5JPpZ0ipNDuWhQr8v+7E3//cltHfJ2JyzBqzOTi8LW3++BTd1gabZwc6EhxHM3J9MBK5OFXWke/xDZPXX0MimrhUGQR7BnArPotytRgOB0J1MJtm2OFpwrg5UMXrejIun8QnQ476/uCGZh4UqPRfzn9e+icTTqSa+lqoKSDuXpJpKi9aAvUJ2xnwwlM82W8nyVIl+vJI/ELTyIhNoVSjIfP4ZwxZ6U+B6BQ1FnPu8zfoMnEDZ8IKUDU3w7OujbGEWO/dTOjwKgOXniI0u6Z2MFOGYDd8BqeTrGfx9Q+qUy/isNYRJ79wohJSycrOIi0xhoiAs2z6dhte6bImcNCTsGMgbds8ST9REVDqkUb6EZqWYZIZjSaTE58PYaVfPjKtgLHkHF+8KRLmaRNhtqo5pXH47JnI2/8cyDcnQ8gWTVmAMnQ1I2eeJqGs1lciVATwTdfnaNN2FIfjyk02bmPOYUa/8hf+31NdsAuSUC2+2rpE9o8bw4agPFOdQEfBD8uZtiGAPJkGoyqTkCtXuRWXSbnKUDsrrw5h8+w1XE6taISBhoAl7/OPwVuJqrO/12L6YFL/03t8efA058468MXnO4mSmTU3XR6XFk3CPkhcTGFElXGdy/63SMiWmkgBBKqv2zN11RWyFc0E6xvKSA4L4lpAAAENvkHESDRNLlxqmdRF27ErDvb78bx1lzBXe6ZMXMeVLAWa6miOLV7Ooev3uBd8gqXjZ3AgotykBTQgdW0RXqsmMnnVYTxuZ1OWF8GpxYPp0ncBZ+LL0RiMyDPdWfXZXPb4JFCu1ZJhTeqaMqKOTeXDj2ZzOq4Sk2lOUJB40QH7/Z7cuhuGq/1UJq3zJEuhxfiQpH65jtR1JGzrw/Md5uBhNQAYso8y+rV+bIy436FS91opA1n6wTM83WMj9ypqBdVYay+oS9LkgeY6qyZv47ZIEE0mEAW5KVIHQfYDc97pzNdeRch1FlL/E+9O3ceJs2fZPGUyOyOlZjNSPWEWiphZEabUFE7bXOH11wVlKZmxYQT4BXMntZBqkwmlmuCN07HzzLCaURmR+i2m49+fou2YoyRVqtEXujD9o7d45Yk/8HS3dYSUKNHrEnAcPYL1N8SBr3bYM8oScd22Ddf4StR6IzrRGVw3IqpJubSD3W6xlDwwLl7DjdVT2X6rDPn943x9oxocKQla3pln/9KD9ZFltc5BoxhZXf+xkPoT70xhz7EznN08lSk7IigzTz10BZdZOmUDVwvEiDYVmcFXCLgVR1a5usl3sT5n85GgpDQzjvAAf4LvpFAoE+35AtUhDsxc/QNploHT+kFjKX6blrLLK44i0YxodU/QlJPotpbFjqFUKixuSksCC6k/wTuT93DszFk2T53CjtulZsx0FFxextQNV8mvFmVGJMwAbsVl1RGmJafmfk0yExfOVf9g7qSIseqi2a2akM2zWP1D7cBpetZYgf+SLjzXpi2jv4unvEZPoessev+7HU/84Wm6rg4yDay6xP2MG72BwDxLJA0YZUlc3LEd11jRSmBE30hmUj12scctxoRNnSiZCtUSvvo/vNR9mcmJay0mrSD1Tsw/484VDwdGf7KBiEpLuJORqvgLbN5ykaRq0f6jQ2e9ykidzMWtO3EzEWHD6tSBqE4n8NxJjn5/hCNHrL5HnQjJVTWpgbVI6oYcLsztxZBFR/EOvUWY1waGtGvPl075VBdeZev8bfgVKtDUZHJwZHsGbLmFVBwR6zT1YnLCT7N563H8Y7IpU2gx6BTk+3xDt3/0xD60DJUetAkH+Xq1OwlF4oIBA5n7BvNC+09Yum0DS6YMY8DYZRy7mkS5pjbCw5DjzNzeQ1j0vRcht8Lw2jiUdq9/iZMYS67/6Zr6jgmLsCb1iLXdeO69RfiKMbVmkNXXvubf7SZwrIXFI7q4rfR9rg2vzbps1jrreqjJA72inJLCQiQ5LswZvoIr8enkSYoorlLVTVvrHmyG1NHfY1PPVxi2J4Eqldasqf+JjnNP4vaDB5vHjmBDeFmdqUMkTBcLYRoaE2Zdac0fCIKpboIgYPoC6pRL7NjtRlypyhTHbHnYKPVl4Qd/589tx3A0qZws5xl8PHsziz78P576W3fWhZRQFePIqBGiLVpetyhENHVVZUcSHNOEZqsvJDb0LtktLcbTKygvKaJQkoPLvJF86xlLWq6EouJKDA8aZPXxbB/wIn9pP4NLVqRhaZP4W0fqHWdzzPUyHlvGMXJDGCWWkcMoI9F1O9td46gQZzI6MVKqmXfXOuO6YwFBNMNYMBYfVafgvnM3brHF1DRlRlFeZdXcfUSJcfCmsOwwzh+7xJ1CpUnLN1YEsGb+Xqoq5HWl1B7Uk/oHs47hctmDLeNGsiG0uFYrFmM3ZIlc3L4d1zhpE4TZKLumTpuSmVQPdu1xI6bIegGZkQq/r+nybBvajv6O+NJMXGYPYY7DQno8/zTPdFtNUKGUmH1jGb0+kFzR9FJXngFZ9h1CYvKpMdvx627pi4gLizINqvfPRrVEOfThH+/NwyOvqoFp9sGk/kRftsVJKJUm88MZX7JEz7+lVEMVWRE3iJZo7iNgfWEMIXezqdS2MH03KCgryCcvN5dc629eYe300VKO1W9LpC5UXuLL1//F8DWncHFzx93tJHvs13EiXIq6pozMu8F4XzzDiZPfs6jXc7y1wMu0MsxE6m3fZvSir/j044k4BOYhtxJAQR7NjsGv02OFL4WKKiIOrOVIZIlZSGtJ/cV3ZnAiJBTvHWPo8OZYDkVbpksClZem88Zrw1l90hk3d3fcTu7Bft0JwsvV6B9A6pXnJvDSa9O4UNjI/KKLZvOk5fiWWq5rCV7+Ps92Wc51ixkIqHGfTrt/TeW8tUnGCk8wUnzuM9r9tR0TTltrrA0SWZ3UkODuyNaN67FfM5Ue/+7PzOWrWGu/gT0/JFPTeKFZc6RuSGHXwJfos/42FUqNmdSfoM+We+QVS0m+cg7fTEXtTMdUuoGqrFrCVLdm7mxV46YP9RTGhnI3u+L+/XGMUny++sBkgvn0+xsc+WIA810TCN8ymJfa/I3u9kEEOQxjxPob9w+Cgha5vInBzahCLvpM6t/m+6qlSvBg37ZNrLdfw9Qe79B/xjJWrbVnw54fkCtbXgFpLHHii389Q7txJ+pNSY1KsJD6k70duJNdhDTFi/N+GcitbDwGWTZ3QmIoUInbBDz8R18US1hUFlKzgtM4R9GnMXfhBQoqakxEVxO+g7EDp3Ag0uzs1yfhOGcd1VJZo0ctpP4kvTZFklUkJcXLCb8GJqsWCLNRbq071VMUF0ZUltQ0E7N+RlQEFnd+ljZtR3M48DumDFqAc2woWz9px1+f6caagKs4jBzF+sBcqhr7Fy0y0zhaziwzxsbXTQXrSdg5iJc7TMe5LnqmtkatIHWzTV3QI6+UodGkEuAba5pailkIWjnVKnFaYt1EMNbITY7MRpcbJqqJwXnLGlauWM7y5VbfFQ64xstN4XsNH4CmSV1LVlQMhXnfMeKFfzB6byTpuRIKCwspLCxCqtRjKL7B7vkL2HIxlHsJsRwe346353maQjJrNfXOzDt6lMX93uOjOSe4Z61RCRqyTk3i3+9N59ztizhsdCOt0rL9gZnUzTZ1VclNdoz6gB4LXUgVp30iaR4bzYsvj8YxMp0ciVinQgqLRNOCEaEZ84s26x4xxVqqL8+hfdtRHGzg/BTZOoBlU3bVmUtAS8iKD3i28zKCSuvNIZqAhbz1ykRONLvwQ8X1ZR/w/PuzcM2oX0jWGPf6cz0VOQnExUQTHXGILwbM51RQOHeiY0jMq0JvPVsTH2qO1HXROPT8B0N3JVBZp6lbbOpG9IpKZBoNadf8ia3Q1GrSFuFvLGz1lfsRR0ZUcnHpdVMSakTq/RXv//3PtB0wlv7dZuCSUYksbjdDXmrD37p/zuc9+rPGyvTyIwpuNqm+IofEuBiioyM4PPlj5p8MJOxONDGJuejEDcJa+KiCV9C57QfMvJBKZTMOrDpSN9vUjXoFlTI16rRAAuKkqE1mJAGtXI7K2IIy1kI9Gt8yqkQeuJ8fLOmEmhusmbWD2+LaAQEM5ancDr1DhrnPhUpvls3ZT2VFfThf7bP1pF5rUzfLjFpNWmAAcVJV7QzKIjNNEqOlFq39NcuMxSls/ZioCCzqxLNt2tL/0/50m3GBlPJK4h2H0+6vz9Bt0iR6DFhDYG7VI1rprSd+x8f8460ZuORYInxqK9QCqcc2dJSaZN9IWeAaPv3GA3mNZYJv3bIfeawvJ+3ubW6GhxNu/b15h3Rp0wuBDBmOfPxcB+Z5ltQt09UVXDHtj5BQHMCyLs/z5tiD3DHtRyM6p3LJKlZR6DKLdzvPxjlDtHNX4jL5Vd6c7W7aFMdYeJRRbbuxMjCHZP+tjOnUhfHbAimoqV/WbCi+woLOr/PR8Cms9co1CWptaw2k7hrAC+8txk+MfjFtuGPPkA8G8K1nlildTchKuj7/JmMPRFJq3lJBnptFscaA0ZDB3kEv8NbcH+o36NEVcMVuGUcTZSgzTzHh9bcYd/iueTsGsVQ9xf5rmbstpH6dAFoi1nbn2XcX4mNlfhHjbIe/2p/Nd+sX7lj3kiALYXWfbkw+eIuiB9p6rZ8ENP4sGbuJ8HJFCzb1O2ZH6TC+q3OUAgpv5r3diYWeEuRaMZLB2lFaOxc0lgdhP2EZ7s3OMhrV5xGeGsu9mP/+Mzz59F958/OzpFWpMaoT2DXkJdr85Rme77aMawXWphdL4ea9Qkz7y1iu/dhfDQFLx7MprKR1NnWhmtC1/fnP5P2EFdSu5m6qRH38dvpbOUpNKBvLub5hEssv5SNr7eYuTWX+U68ZKwjdtRC7CzGUNpY/oYp7Rxbz9fd3qGwQoysWJpKataPUIjM32PjZcsCxo3EAAAqgSURBVC7VLQL7qRX7sc+JisBCOj37FE//9Q0+O51sMu2qEx0Z1u6v/OVvz9P1G/9GphexDBVFSbe4ERROYnGtyal1Jeu4u6k3L703n8u5teYXfWks/p7XaJrUDfncdFrLsFee4A9/eJVBi+xYu34961cvZmL3V+m07Doy0bj8i34ElGmBnF49hFeefJp/j7dj+5597N2xlrkjO9FhwnFyZGXEn/uawZ06M3jaCjbt3M3u/ecIzVcivbqSD9/4iOlbDnDk++9ZN/o1nusyhc2n3XHdPJY327xA76WXSSrPI8xxLO926MUXdke5nmO27RurCFn5H/45wJ4gy4sjyEm5eoqVg17mz3/rxLTtF7hdqEZXk8fV7V8wcNh8HL0TKS+P5dySIXTuPIhpKzaxY/du9p8LJbc0kaunVzO03Z95+t/jsNu+h317d7B27kg6dZjAsexqNNpibn63mPEjxzNn9S6+O3GCI/u247DlMD4p1qYDPYk7+vH8m7MbOEoFdSK7h77LxKPJtZEcpj7TUp6ZREp6Itf2LmDG6nPckVjbCM0da9TfZ1Zr0OX6VDydrlNQ08yCJlGOLqxl6Ct/5n//+Abjt7gQWVq7UEp9ex29us7FLbOSrJvOrB/xT5763z/Q7uMFrFxjz/r1q1k86UP+1fkbrlqtAG1Q/s95YizHd3Fn/q9NeyafT6NK1H4FNQn7R9Du6b/R7dtrFJiiXiyVEPeXMe8VMmNv3f4ylrs/7ldP2pUL3MhXtrhaVlueRVJKOonX9rFw5mrORhSYZn8NyzKi1xsxFNzGZcNI/tXmj/yh3UDmf7sa+/XrWb14Ej1e68ISq2X9DZ//uc8EavJucNLBjvUH3bgek0VpRSk5iaG47V3DCoeThIgRUnU2X7E+RiQRrmwc1Z6n//gH2g2Yx4rVFpnpwWtdluBntQL0526BJX9juR/fdHuOv7T/nDPJtZttie/fwTGv8bdnurLcL9cc9WJ+QlAQd2E/x/3CCQ84zIr5W/BuPCO3ZH7fr5Ywu24mR+lVicwUSl16ZQWDu3drhtQFBUVp9wj198bLy5/rN28TERFJxK0Qrvn6EJpSef80+75CH/UFAV1F7da+Pt5e+AdHEh0TS0zUba77euJzJ99k59LLcokOvsJFZzeuBN7krujtVhswyDK57eOOu28wkbFJJNz24ryLL7eT00i5G4K/jzeBkVlU6Axoy+K57uNL4M148mTm0EOMFHz/JZ8fiqFMZV7iLmiRZscTEeSDt5c/oXeTKZSL6QXUJUmE+XoTGF2AQq9FlhtDsNdFnN2uEHjzLnFZ5ajU5WQnRBLk4423fzCR0THExtzj9nVfPH3ukKcWowEEtBXZxIZfw8/fD/9rwdyMvEt0SlFdSGYt0gJlp8fz0iufcybfskhHtI9pyDw3jb5TTpIsapvokVzbyayR/ejdqwfvt+9ArylrOODkTUhUIpkSKQpFJXkJN3D5/hLRYvRHUxYKsVBBg6xSURcyel+PCzIkSXe44evFlSt+BN9JpbhGxKeGmxtGMGbTNXIVWuTF6USHBuDj5YX/9XBuR0QQGXGLkGu++IQkU9GMPfa+8h7pBSOy1Ks4O/sRZx6IxIguTWEkHucvEpJRhbaBqUlAXZpNos9yevVY8ZDbDghoZJW1Zotm2qQvDGT3nNEM6N2LHh+8Todek7Hbdx7vkLskZEqQyhVU5icS7HKUS9EV1FQVkR4TRoCPN15+1wm/HUFEZAS3Qq/h6xNCstRs4mqmvJ/1sqCmLO0OgZfPc/zwXnbu2MneQ0c54+bPrVQxOKEBo4uCh6I4nZiwAEQu8LOSmVCzzEgtYdU/a8UbZW6UkXbNBWc/McLJ7HsUNBTd+QGni8Gki2s+rN8lQw6u367k5L0CyktDsR/Uk28u18asN8q5iVM1fovf4R9DtxNTKm4pIKDKv4v3kQXNkHoTWfwmLgnG2giGRpURzNfrLgsGNDVKasSOtQJRMGioqRHjmsUd6HSoVaKJxxINYfkVc2nkwRcvGSV4bNiAR4a4CrKuJATR9igWIuZjOra6p9egEonZXAdT+coaNOLWsaZkAqITxLqOtZfN1+uzMmknek0NSqWqEZnUJ1KJkS4vfsLelMoGW3IayiM5OG8mu0KLqdHKyY0OJcjfCw/nExzYsoZlC6bz2eih9OvZk979BjJo8ED69h3KbMcgcpsLR60vtoUjCya1eIphkmK79blurJi9Ec8GM40Wsvm1bokyoqldZl9fBQMaVe1W0/XXao+Mer1pfxlxK4JrdZuGNU71KM4FFHnRhAX5433ZmZMHtrJ22QJmfDaGof160rN3XwYOGszAvn0ZOnsPgdktr2x+FDV6+DyMaGTF5GYkExcbR1JaDkVVFr/Vw+f+S+Ug6NRoGvsQDBrT/u33v+cK8pNSKVLq0MuusWzAaHaEiiG+raitIMVpcnve+vJs3eIjQZHBdeezvzNSb0VbH30SNaneh9i8ahFLD4VRaGVnf/RlPVyOhpzDDH+5OytvlDbcJkDQUhbjzoE9rtyTKlCIWoQYrqVTUV1eSHZKHHfDb+Dnfp7jB3axedNW9p2+wp3catPy5oerVaOnDRKuH9/P2esZVGrNM55GSX7Pp4asA4zs+/OTul4t/vGEOGjqUMnLKcxOJe7uTW74eeB0/AC7tjiwde8pPO/kmPYCt9Jtfs/wPp5115cSfnQNdnu9SRLXBLSms/SJ7Br0Lz7edJNi8xaPgqaSgpxCG6k/WErUJHvswsHxPGE58kdPcg+uQKtTCKowVnVtz+SzuY029BJ3C5BTkJZtip23mmg0yNs0k6iuRFouRaZqOfyuwYM/5kRQIEnPQVw01Fw9fkx2v7W0vwypt9BqwYC2Rk6VtBypTIwAaQ1DtJCf7dbPi4CxjIjz+znkFEC06Hi3TOsfVKrSl0UfdOGrS1lUWjbvE60FYCP1B2EnOmWUZXnkFovOmt/4CyJIuTL3A/rYh5n+kODBbbOleNQIGMTtfXuv5GpRU5Exj7o0W36/bwS0ZF3ewZZj/kQXKFCl+eJzT0ZN4zj2JhqpT3Hkk87TOJlo3gzMKk3T0S9WCWyHvycEjJS4Taf7xKOkmpyiv6e6//7raiyJw2f3BDpY9pcRncK//2bZWvBzIaCJZMcn3en96XTmL1rMrFFjsL9W1DBCpsmyBcouzqTnlGPEPso/yWiyLNvFXx0BY8kVvho8j4s5ln29f/Uq/ddUQFCWkBETip/vdSLFvUIsu5X+1yBga+iPQsBQTNz1QIJuBBMcEkyQ3w0Syyw7eraQkyDDf+kI5p2z+hs8q+Q2Td0KjMfi0FhJuMM0ll7Koto6TOexaNxvvBFN7BXyG6+xrXq/KgKNIt/MNvEHVUmQBrBm5np80ysbROJZnrORugWJx+ZXQJXlwcZ154h/2L3QHxtMbA2xIfC4IKAj230zDs4xFDfzR942Un9c+tq6HeIiiIATnLtRwKPZ/Mo6c9uxDQEbAr8WAsaK27ie9iHWvItlU/WwkXpTqDwG14yKArKLlKZ/eHkMmmNrgg0BGwLiskh1MTmSarTNLvO2hTTaBMWGgA0BGwKPFQI2Tf2x6k5bY2wI2BD4b0fARur/7RJga78NARsCjxUC/x+3ILT73sRliQAAAABJRU5ErkJggg==)" ] }, { "cell_type": "markdown", "metadata": { "id": "Co9D_oNXUlgj" }, "source": [ "The components of the formula are as follows,\n", "\n", "- **E⁽ˡ⁾**: the embedding table after l steps of embedding propagation, where E⁽⁰⁾ is the initial embedding table,\n", "- **LeakyReLU**: the rectified linear unit used as activation function,\n", "- **W**: the weights trained by the network,\n", "- **I**: an identity matrix,\n", "- **L**: the Laplacian matrix for the user-item graph, which is formulated as" ] }, { "cell_type": "markdown", "metadata": { "id": "SLrIvZowUyyI" }, "source": [ "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAOkAAAA6CAYAAABVnX5iAAAVDklEQVR4Ae2dd1hU176Gz1/35mhEjLElGsWSeKPx5CRGb2JNMWJLjp4cjSZqComJxxKjRlEsUQRF7EFRYjkWUETpKggoUqSpiBVp0oQBBAZmmGGG2e99hjqUGRgFIbnb59mPM7utb31rvXu13x7+gvhPdEB0oE078Jc2rU4UJzogOoAIqVgJRAfauAMipG28gER5ogMipGIdEB1o4w6IkLbxAhLliQ6IkIp1QHSgjTsgQtrGC0iUJzogQirWAdGBNu6ACGkbL6D/F/IU0RxatQLrfW74+gUSlVSIqkz4w2VdKE7hWshlArycsF62HMfgHGSlT58PEdI/XFX4EwqWnsZi8EgWHI/kblIa2YWlaJ6+bj9zowSVFElmOqnx7vw8YhBzDiXzuOTpMyJC+syLUkywngNSF2b3G8WqoCyKVPWO/vF2qKLZMHoAM50SRUj/eKUnKm7QgSeEVC2Xo9QIPH1b1aCqmp3qRPwPbMdmrRWrVqxi3aYt2G3ZjN1vJ/C/lU2JWlNzrvaTCGltP8RvfwIHjIRUkD3AZ8dqLNdasWShJQ7+CRSWlrWcEZpCUq8fY+GI/vTsac4azyDCIv3YN38iYz7+ip1XHiFT6TwqREhbrizEO7eSA8ZAKhQQtmM2Iz+wYO/5AJyXT2ToB//mP7fyUbQgp5RGsn5Edzp0nIRDYh5yjYL0kxYM7tKVt5b6kiktrWnRRUifsCIJctJjYnmoUNGSZWm0uraqy+iMPMUFRkCqyfZgwdsv0Xf679x7XEJh0C8Mf6kPE+2jyJG1YMmqYtg4qjsmVZAKUHrFkmHdOmL2tSupBUoR0qeoAkAJ6TFn2Pi1JR7ZxSh1eiZPd9+nvVqr6yzWX6/CI7uoDel62nwZeb0RkCqClvN2144MXRlMVpGaslRH/tGzEy9PP0xSvqIGFCMlNHp6XUhLJYRYm9P7xVeZ6RRHXonOA0JsSRu1s4ET1BRlRWI71YLjmYUo2gyklbqmWXA8oy3pasDCltzVZEgF8o7Poo9pB0bbxJIrK0PIPcrM3qaYvL+FuBwZdaZwmk91FaTtBjJ5/nwsvpjCuwMGYf7LYcLTZKh0ExYhfULfNens/2Iezo/aGAxaXV/Ow7lNPTye0OMnvcwISAtc5mDWyYRR1tfLu7eajAP8s5cppuN2cC9P3vKQdhjBspOn2DbnTTo/34Px2yKR1O1mi5A+YU3QpOHYJiFNEyFtMqSgjFrPqO6mDPnJj0ypitLoXxnVvSMDf/AkvVBn8uYJq4ney6paUu2YNCGb9IjtTB3QmZdHreJ8ahG1AotESPXaaOCAhsf33Fg08mNW+twmt7Ss5cYuBlTUP6Th8X03Fo8azwqfW89IlwalNIukm+FcSypqG+F3RkAqSK+yaXxveo63IyqniMTfP6PvS0NZeDaRQqVun7O+20+1Rwte5cTRb9rZXWUm51eM5uUuA5i2M5xHcnVNnWpTkKoLSb93g+u3ksnRimyWsZ6a1LCTHNhpx2ZbW2y32LNtmz3b9zjh7BtBcoFuyJgGyQ0/3N1cOXXqlM7mytnQJJSqisF8ZmYGweec2WOzGQcXH2JuxqFWqxsuM8Vdzmzdwsnrj1GodTNkQJdPBEkFpdSEm1bo8tCjS1FLl0ulLm9i4m7p19WwWuP3Co+55W3PDxPG8MORJAoVjVRsrR/2dpy89piSWn4Yn7TeK4yAFKGEh4Hb+Xr8JL5bu5bvJnzI9NXOXM8uoaXkoU4j7IQVk/qY8NxzA/ls4wmuZskpiD/L6smDMHtjIt9v9qSgWFmRxbYBqYAs3pttq9ay+5Q3Pi52LPp+Ax7xxZQ2UuZ6C6r6gIaizFucWjKG18zMmGDlQXBYOEEeh7BZMIOpXy5n36UUZOUjdQ05Ny/i7XGWM2fO6Gxn8QxPQVk5mg8MDGTZsmUsXLSQhQsXsmPHDuRyeXWKNR8E8gNW88GAPgxb4kVasW73qVLXzzW6Lmt1eeroCkqhuFpXgB5dydUPj4CAgFq6du3apUdXjcKn/iQoyEs8zfy/9WDKrgc8lus+iOreXSA/0IoPXzNj2E8epBbp+lH33Kf4bgykgKDM4X7EJfz8zuPrd4Wb6VJULRnsqyki/W4Ul3y98PQ8T3BMPFkyNZqyYtJuXMb7lCveYQ9QVAVUtAlIZdfYO3sE73//O2HxGWTGOTL343kcji1AqTMT/eTFVkrk2v/lxfYdmLQ3mUJFGcqiLB5EOrPso0EMHjmXPWHZ5S2dSlbA47xccnN1tzzypAo0lU37o0cZ+Ls4st1uE1uc3Im5qafF0mTg9u+PeaPXc7Qb8A0nkwprjzWo0NWlfQcmOiRRUKkrQatrnFbXHHaHZpWHiTVNV3ptXXG3W74l1RaKKoYNI3sxdU8C+YYCwLV+LBjPkFe0fnyNS2IBLdKjNBLSinoloFYqy+E09Jh58jpY+0pBbzdRQKMqLR82VOtofUgF8i8sZVgPM2YcTigfBwjyVK5H3iK9SNVMby+oiF7/Ll3amzBlfyqFVWsmgoxE528Z3LkLA2cdIK5Q0bTZPFkov20/w+27l9j9sw1+WbI68FUUiDreiW/m/MqOb4fwgokZnx++T0GtWqnV9V6FLseH9XS9odU1cz838xVNC5iQheFQpWupDRceFTeoq6q6aIoeEnXhDM7HXPCJTKVIpUEbuVoiiScq8CxeERkUZd8nzNeVYye8iEwr1lkaUJN7Jwh3l2Mcd9nBnMHd+GS3YUjV8b/z7dxf2f7t3+jc0YwZh+6S31j3uEqsMf8/EaTGJPCMz219SGUELv07nTuOYIN2nKJtOTUag7CU5d4j7FIAF/398dfZLgbFkqFUN3CtHki1SeW48tWrprTrMoW9dwtoUp1RpRMbm4FCFsnG6YtxT5c2EDggJ8rucywcr5McZMXobib0/Mc+4vJ1HwR6IK3U9fVrnWjfZQoOt/MpaUq3X6vrZoUu6+k/6dFVUcGE4lgOL7Nkb0AUUUEHWT7rexyuSpCryyiM98Nu7ije/8oKuy27+f3QXlbN/ADzFWdIkSoRBBl33WxYZXMUv8hoQs6swbx3ZybtMgSpnOits7DYd42kwDWM7dGRXp86cPNxif4HkCaX++GXCbxYu5z9LwYRm6HQeWDUgUaEtI4htb8a/6qaOo7N73fDpL8F7pnF6MYV1751zTfFgwCOH3Ziv6MjjjrbgROXSZGrjIIU1XU2jupC+78OYfnlHGR65n9qUtd+EhA0Um5772P3sVDSZPXTFPIvsuKjCSw95MPFcw7MGWRKu27j2RZT+SAqv6F+SFFrdXWl/V/fYFmQhGIjdN3xdizXlVpcX1dVPjS5AWxZYItvmpQSWQIOUwcy3iYUiUyNqigd1+9ep9c4S9yCb5CYlkr0ZnN6j1lHiKSY0rTTLPpgKlY+d5Eo1CiyfVj81kt8aqAl1fqxctxElh70xl/rxxudaN/tY+yjc5HrG9IoEghyPozT/trl7HjgOJeTZVQN2aryVP2/CGm1FQ19MBpSTfYxZvU2pff0wyRIlQ0AVj+ZsiIJaQ9TSElJJjm5ZktJz6OkTNtlq/tPf0uqhdRaC2m74ayPytNfYWrdUkGCvzOufpHcu5NAWnGZzkys9kQNGad/wPyTRWzcuRfH/XvY8MXfebHdi4zZFF4+c11xOwOQqm5gPborz7cbzrrIHOqub9eSU/1FQcJFFwO6qk8snyxJiLqMr9sxDh9x4qexPRg035OM8smcUoJ/eRuzf+0jIb+k3E+Z61f0HbaU85mF5Hj+wKBXZ3MkuaAi2ko7Jh1laEyqIcNtPhM+XcSGnQ6VfrxFl3YvMtq64sFQo0znU1kxkrRUUlJSapVzckp6edic3rkdEVIdE+t/NBrSkkvL+Hv3v2Fx8gHSJk7lltx0xW69FZYrV7JSZ1tl40qcVNlA90k/pJosZ2b360jHNxbhm9G0llyT7c3KWbNZsm4L1kvt8Ko7JlXHc+CLqaxwi+L2gySSkpJIDLdn8ismdB5uRVC2jIqGUT+kWl1z+pvScfBCfNKKmtTD0Ei8sZw1hyVrN1fr0hdXrJEEs3vxYmxPXiIq9hp7P+/L4B/dySh/+6IC0n6zDpJaUBG/WuL2Df3eWYJvZiGZR2bwSp+ZHExpIqTqeJy+nMaK05Hciq/xY0rvjnQevprALD2+l8ThZv8rayxrl/NKSxtOxRag0Ne7eGJI1TwMPMhOO1s2bdpUZ7Nlt9ddZC36akx9oMr3tOqYVJByZfVYhn2xh7BHTV+XUufcJzIshODg4Frblav3yWkwsEBF1LqKiaPJjjUTR2XSBHw3TKRfz7eYuzeC7JKmBSUI8mQigsOJjo4hIuQGqSW642AB6VUbPpm2gcsZxTVrbapEnKb3peMLb7LEN53i8n69iqjKiaPJ+2omjjTluibRX6vL4SpZugvbespRu1urK7JKV2iFroZ7khqy3X7krWHfcfxeLoqyAly/6s/r37tWQxq09E3MPncipRLS4lNzMXtrEV4ZUgrD1jKi11AWeCRTWCqgyfVh4Zvd+dD6WgNvjmj9sOUf/9zApXSdh40qkd9n9Mf0hTf5ySeVolohNpWZVOcQHxVGSJ1yDr5ylXsSJXXfja625okh1SBNiSXqahihoaEE2M1hrq03F4JCCQ0NI/pBHiq9iVan3vwfDEGqlnDTz4uAhl4W16OkSS1paW4Sd+8ncCdgNwu+seQ/ERnI660ca1A3iyFq0kJPYDXhFZ7/r/9mwKeW2Npvx36rLWt+XsD8BcvYdMSfOzmKOl1WPTk0uFtDRtgRVk8dhNno+ewLSEJeuc4pueaO9dR+dHiuPWYTlrLTPYbYoKNYTazRZVNX12F/bkuaQ1dd0QLSAEtGvj6ab21/44CTE+v/+So9hs/BxjmCm0GH+GlMd0wHzcD+/AMKU0JxshjKC12HY7E3gMTsaxxe8DHvmX/LKtsd7Nm/A4t3XmLIVEuORjzSCVKo8MNq2mD6jv6RvRcTq9ejJdfd2TStPyZ/bY+Z+c/sOHOdXKVOlE1dycZ8bwBSddJFft+5mfXaX0NY8yu2W+3ZamfPXueAco+rq5ogoF0e0W7Sk/P44Vg6eTJN5T5jRBhzrhq5XFm9xFfvSgOQaiRe/GL+Lu/O2MFVSVUPrd4dau1oFFJ1ZgDb5k1l3PvvM/rt13h9zFesdXDhXMg17iRl8rhYRn7abYJPOXFWu05aD95a6TXhiwZpxl2ignzwdHfHJzCcyOhooiIjCAkOIUob3aRdSK4/kG3CveueIlCccYswf298gyK5kyZFXX5jAXl2AjdDL+Dl6YGnXwjXHmQhSb2joyusWldocAiRzaqrrk7QFCYQ5uXKaZ8gwq/FERvqwVFnb0LvZpKdcoNQP088fAKJSc5HWZDKjSvn8fI4x+XYhxSolDxOjOSi51m8AsKIjrtLuO9pfC9HcC9LVplnbZo1fpwLiuR2WmFlkECVH37VfsTEZ+uZT6ivvdE9DUCqKUgh5ugC3uvfk57ma/AMCuXqeQfmTxqL+Te7Cc2S1xtSFJVDmtFIgEajagycICBP8GXXmtWsX7uUxVZ7uZhQWH9CzACkQslDIjwc+OHbzYRJimjKTzo1AqlA8cMYLvv74HH6GAd2bGTVz/P5ZtY0Jn00lrEfjMN80iQmfPQRk+ftIPCh7rqcgbz+IQ7VfgroXct+hnkRypTI5dqlDAEEFYoSJWXlrUhVy1HRomjjM8sDOapamUqN2uu1UTHaVket0q5p186j4azUPteoSw3fGBqAVHtJaeQ63uv2PCaT95KUX4JGkYbLN6/TpevbLDv3qF6Xu6UhFQrC2TV3NB9a/Iav/wmWTXyHjxYc5VZ+nWUpA5AiFJMQ5MJ/vK6TXaJqYNK0vlmNQqoqKamI6tCUIpfmkJ54hxsRwVz0duWI4062bFiP9baDuEekIC1fXK+fiLhHdMCgA3ogVcVsYGT3DnSshFSglCsr36GriRlfn06jsM4sW8tCqiHbcyFDX+7LdKc75MkLCfplOC/3mYR9pKT2bL5BSJU8Tk0iPV9JWVPW0uEp/j6pUIZSJiU/N4fcQm3Xo/aT1mChiAdFB3QdaCKkpZIQrM178+KrM3GK012/rrhZy0KqIGj523QzHcqKy48oUpeR6jiVXp16Mv1wEvlVUXFaKYYg1a7ZG4lKIy2prpPiZ9GBFnKgEUjbDZzCggXfMfvTdxkwaDzLDoY1GJCizrjBjbSS+mPE5pAt5HH8iz50MhnNphs5FJcJ5B6dRe9OJozdEkeuTKdZNAip8WJESI33TLyiuR1oBNLnRyzj5MmtfDnkBdr3MGd7VE4Tg1iaUahQgMscM14wGcXGaxKKyzRkOH3GK6amjNt+l1y5CGkzui3eqs050Aik5WNSSRpXt35C/849GWN1gbTipk26NF9elUSuH0kP0yEs9stAqiolZsNoepgOZJ5HWvn6c3VaYktabYX44c/igD5Io3+tmDiapJ3dlaPI8GX5qJfo8upn7L6a1cBafUsaIiANt2Z8n16Y20UhkSZxcHo/Xhr6b9wS6rzCJ0LakgUh3rtVHGgAUnVaOM5rJtHH5Dme+5/p2LhEIJEXcN/NkkmvmzFk8o/Yed1H1jwvMDcp24I8hYv2X2E+eR7r1s9j4of/wvJYDFkldYI6REib5Kd40h/JgQYg1UjTuRMRiLeHO+6+l4i+r205NZQVpXI9yJOTJz0Jjc+ltOY3a55BjgWUOfcID7zAhXM+nA+OJa1Q92dzKiWIkD6DshCTeLYONABp+euF+pYqBA2qUlV5IMezFVqZmqBGqTQQDCJC2irFIibakg40CGlLJtjC9xYhbWGDxds/ewdESA16Lq6TGrRHPPhMHNBCavY60zae4nxQKLGpbeT3gI3MvCBL51bUVUIubGfGwF78a7/4R4SNtFA8vc06oHpIqPtZzofe4Pa9B6TmKShrntecnmmWhdLHpCc/IP72VS6ccePyg+Jm+fVMsSV9psUoJtawAwJlahUqlXZT67w61/DZbXavoKnJh1rd5AD6xvIjQtqYQ+Jx0YFWdkCEtJULQExedKAxB0RIG3NIPC460MoOiJC2cgGIyYsONOaACGljDonHRQda2QER0lYuADF50YHGHBAhbcwh8bjoQCs7IELaygUgJi860JgDIqSNOSQeFx1oZQdESFu5AMTkRQcac0CEtDGHxOOiA63swP8B060KAVwSYHwAAAAASUVORK5CYII=)" ] }, { "cell_type": "markdown", "metadata": { "id": "TcNXtDMFVLii" }, "source": [ "#### Architecture" ] }, { "cell_type": "markdown", "metadata": { "id": "kyKDfxNfBWl-" }, "source": [ "![](https://github.com/recohut/reco-static/raw/master/media/images/120222_ncf.png)" ] }, { "cell_type": "code", "metadata": { "id": "4i1YYbJB4oGQ" }, "source": [ "class NGCF(nn.Module):\n", " def __init__(self, n_users, n_items, emb_dim, layers, reg, node_dropout, mess_dropout,\n", " adj_mtx):\n", " super().__init__()\n", "\n", " # initialize Class attributes\n", " self.n_users = n_users\n", " self.n_items = n_items\n", " self.emb_dim = emb_dim\n", " self.adj_mtx = adj_mtx\n", " self.laplacian = adj_mtx - sp.eye(adj_mtx.shape[0])\n", " self.reg = reg\n", " self.layers = layers\n", " self.n_layers = len(self.layers)\n", " self.node_dropout = node_dropout\n", " self.mess_dropout = mess_dropout\n", "\n", " #self.u_g_embeddings = nn.Parameter(torch.empty(n_users, emb_dim+np.sum(self.layers)))\n", " #self.i_g_embeddings = nn.Parameter(torch.empty(n_items, emb_dim+np.sum(self.layers)))\n", "\n", " # Initialize weights\n", " self.weight_dict = self._init_weights()\n", " print(\"Weights initialized.\")\n", "\n", " # Create Matrix 'A', PyTorch sparse tensor of SP adjacency_mtx\n", " self.A = self._convert_sp_mat_to_sp_tensor(self.adj_mtx)\n", " self.L = self._convert_sp_mat_to_sp_tensor(self.laplacian)\n", "\n", " # initialize weights\n", " def _init_weights(self):\n", " print(\"Initializing weights...\")\n", " weight_dict = nn.ParameterDict()\n", "\n", " initializer = torch.nn.init.xavier_uniform_\n", " \n", " weight_dict['user_embedding'] = nn.Parameter(initializer(torch.empty(self.n_users, self.emb_dim).to(device)))\n", " weight_dict['item_embedding'] = nn.Parameter(initializer(torch.empty(self.n_items, self.emb_dim).to(device)))\n", "\n", " weight_size_list = [self.emb_dim] + self.layers\n", "\n", " for k in range(self.n_layers):\n", " weight_dict['W_gc_%d' %k] = nn.Parameter(initializer(torch.empty(weight_size_list[k], weight_size_list[k+1]).to(device)))\n", " weight_dict['b_gc_%d' %k] = nn.Parameter(initializer(torch.empty(1, weight_size_list[k+1]).to(device)))\n", " \n", " weight_dict['W_bi_%d' %k] = nn.Parameter(initializer(torch.empty(weight_size_list[k], weight_size_list[k+1]).to(device)))\n", " weight_dict['b_bi_%d' %k] = nn.Parameter(initializer(torch.empty(1, weight_size_list[k+1]).to(device)))\n", " \n", " return weight_dict\n", "\n", " # convert sparse matrix into sparse PyTorch tensor\n", " def _convert_sp_mat_to_sp_tensor(self, X):\n", " \"\"\"\n", " Convert scipy sparse matrix to PyTorch sparse matrix\n", " Arguments:\n", " ----------\n", " X = Adjacency matrix, scipy sparse matrix\n", " \"\"\"\n", " coo = X.tocoo().astype(np.float32)\n", " i = torch.LongTensor(np.mat([coo.row, coo.col]))\n", " v = torch.FloatTensor(coo.data)\n", " res = torch.sparse.FloatTensor(i, v, coo.shape).to(device)\n", " return res\n", "\n", " # apply node_dropout\n", " def _droupout_sparse(self, X):\n", " \"\"\"\n", " Drop individual locations in X\n", " \n", " Arguments:\n", " ---------\n", " X = adjacency matrix (PyTorch sparse tensor)\n", " dropout = fraction of nodes to drop\n", " noise_shape = number of non non-zero entries of X\n", " \"\"\"\n", " \n", " node_dropout_mask = ((self.node_dropout) + torch.rand(X._nnz())).floor().bool().to(device)\n", " i = X.coalesce().indices()\n", " v = X.coalesce()._values()\n", " i[:,node_dropout_mask] = 0\n", " v[node_dropout_mask] = 0\n", " X_dropout = torch.sparse.FloatTensor(i, v, X.shape).to(X.device)\n", "\n", " return X_dropout.mul(1/(1-self.node_dropout))\n", "\n", " def forward(self, u, i, j):\n", " \"\"\"\n", " Computes the forward pass\n", " \n", " Arguments:\n", " ---------\n", " u = user\n", " i = positive item (user interacted with item)\n", " j = negative item (user did not interact with item)\n", " \"\"\"\n", " # apply drop-out mask\n", " A_hat = self._droupout_sparse(self.A) if self.node_dropout > 0 else self.A\n", " L_hat = self._droupout_sparse(self.L) if self.node_dropout > 0 else self.L\n", "\n", " ego_embeddings = torch.cat([self.weight_dict['user_embedding'], self.weight_dict['item_embedding']], 0)\n", "\n", " all_embeddings = [ego_embeddings]\n", "\n", " # forward pass for 'n' propagation layers\n", " for k in range(self.n_layers):\n", "\n", " # weighted sum messages of neighbours\n", " side_embeddings = torch.sparse.mm(A_hat, ego_embeddings)\n", " side_L_embeddings = torch.sparse.mm(L_hat, ego_embeddings)\n", "\n", " # transformed sum weighted sum messages of neighbours\n", " sum_embeddings = torch.matmul(side_embeddings, self.weight_dict['W_gc_%d' % k]) + self.weight_dict['b_gc_%d' % k]\n", "\n", " # bi messages of neighbours\n", " bi_embeddings = torch.mul(ego_embeddings, side_L_embeddings)\n", " # transformed bi messages of neighbours\n", " bi_embeddings = torch.matmul(bi_embeddings, self.weight_dict['W_bi_%d' % k]) + self.weight_dict['b_bi_%d' % k]\n", "\n", " # non-linear activation \n", " ego_embeddings = F.leaky_relu(sum_embeddings + bi_embeddings)\n", " # + message dropout\n", " mess_dropout_mask = nn.Dropout(self.mess_dropout)\n", " ego_embeddings = mess_dropout_mask(ego_embeddings)\n", "\n", " # normalize activation\n", " norm_embeddings = F.normalize(ego_embeddings, p=2, dim=1)\n", "\n", " all_embeddings.append(norm_embeddings)\n", "\n", " all_embeddings = torch.cat(all_embeddings, 1)\n", " \n", " # back to user/item dimension\n", " u_g_embeddings, i_g_embeddings = all_embeddings.split([self.n_users, self.n_items], 0)\n", "\n", " self.u_g_embeddings = nn.Parameter(u_g_embeddings)\n", " self.i_g_embeddings = nn.Parameter(i_g_embeddings)\n", " \n", " u_emb = u_g_embeddings[u] # user embeddings\n", " p_emb = i_g_embeddings[i] # positive item embeddings\n", " n_emb = i_g_embeddings[j] # negative item embeddings\n", "\n", " y_ui = torch.mul(u_emb, p_emb).sum(dim=1)\n", " y_uj = torch.mul(u_emb, n_emb).sum(dim=1)\n", " log_prob = (torch.log(torch.sigmoid(y_ui-y_uj))).mean()\n", "\n", " # compute bpr-loss\n", " bpr_loss = -log_prob\n", " if self.reg > 0.:\n", " l2norm = (torch.sum(u_emb**2)/2. + torch.sum(p_emb**2)/2. + torch.sum(n_emb**2)/2.) / u_emb.shape[0]\n", " l2reg = self.reg*l2norm\n", " bpr_loss = -log_prob + l2reg\n", "\n", " return bpr_loss" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "5xbcqHLUSowG" }, "source": [ "### Training and Evaluation" ] }, { "cell_type": "markdown", "metadata": { "id": "N6S7uZU3Tpht" }, "source": [ "Training is done using the standard PyTorch method. If you are already familiar with PyTorch, the following code should look familiar.\n", "\n", "One of the most useful functions of PyTorch is the torch.nn.Sequential() function, that takes existing and custom torch.nn modules. This makes it very easy to build and train complete networks. However, due to the nature of NCGF model structure, usage of torch.nn.Sequential() is not possible and the forward pass of the network has to be implemented ‘manually’. Using the Bayesian personalized ranking (BPR) pairwise loss, the forward pass is implemented as follows:" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 1000 }, "id": "LEMcstCz4vSm", "outputId": "4e06cd7c-8e69-4f6b-d4b8-054d373f01cb" }, "source": [ "# read parsed arguments\n", "args = parse_args()\n", "data_dir = args.data_dir\n", "dataset = args.dataset\n", "batch_size = args.batch_size\n", "layers = eval(args.layers)\n", "emb_dim = args.emb_dim\n", "lr = args.lr\n", "reg = args.reg\n", "mess_dropout = args.mess_dropout\n", "node_dropout = args.node_dropout\n", "k = args.k\n", "\n", "# generate the NGCF-adjacency matrix\n", "data_generator = Data(path=data_dir + dataset, batch_size=batch_size)\n", "adj_mtx = data_generator.get_adj_mat()\n", "\n", "# create model name and save\n", "modelname = \"NGCF\" + \\\n", " \"_bs_\" + str(batch_size) + \\\n", " \"_nemb_\" + str(emb_dim) + \\\n", " \"_layers_\" + str(layers) + \\\n", " \"_nodedr_\" + str(node_dropout) + \\\n", " \"_messdr_\" + str(mess_dropout) + \\\n", " \"_reg_\" + str(reg) + \\\n", " \"_lr_\" + str(lr)\n", "\n", "# create NGCF model\n", "model = NGCF(data_generator.n_users, \n", " data_generator.n_items,\n", " emb_dim,\n", " layers,\n", " reg,\n", " node_dropout,\n", " mess_dropout,\n", " adj_mtx)\n", "if use_cuda:\n", " model = model.cuda()\n", "\n", "# current best metric\n", "cur_best_metric = 0\n", "\n", "# Adam optimizer\n", "optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)\n", "\n", "# Set values for early stopping\n", "cur_best_loss, stopping_step, should_stop = 1e3, 0, False\n", "today = datetime.now()\n", "\n", "print(\"Start at \" + str(today))\n", "print(\"Using \" + str(device) + \" for computations\")\n", "print(\"Params on CUDA: \" + str(next(model.parameters()).is_cuda))\n", "\n", "results = {\"Epoch\": [],\n", " \"Loss\": [],\n", " \"Recall\": [],\n", " \"NDCG\": [],\n", " \"Training Time\": []}\n", "\n", "for epoch in range(args.n_epochs):\n", "\n", " t1 = time()\n", " loss = train(model, data_generator, optimizer)\n", " training_time = time()-t1\n", " print(\"Epoch: {}, Training time: {:.2f}s, Loss: {:.4f}\".\n", " format(epoch, training_time, loss))\n", "\n", " # print test evaluation metrics every N epochs (provided by args.eval_N)\n", " if epoch % args.eval_N == (args.eval_N - 1):\n", " with torch.no_grad():\n", " t2 = time()\n", " recall, ndcg = eval_model(model.u_g_embeddings.detach(),\n", " model.i_g_embeddings.detach(),\n", " data_generator.R_train,\n", " data_generator.R_test,\n", " k)\n", " print(\n", " \"Evaluate current model:\\n\",\n", " \"Epoch: {}, Validation time: {:.2f}s\".format(epoch, time()-t2),\"\\n\",\n", " \"Loss: {:.4f}:\".format(loss), \"\\n\",\n", " \"Recall@{}: {:.4f}\".format(k, recall), \"\\n\",\n", " \"NDCG@{}: {:.4f}\".format(k, ndcg)\n", " )\n", "\n", " cur_best_metric, stopping_step, should_stop = \\\n", " early_stopping(recall, cur_best_metric, stopping_step, flag_step=5)\n", "\n", " # save results in dict\n", " results['Epoch'].append(epoch)\n", " results['Loss'].append(loss)\n", " results['Recall'].append(recall.item())\n", " results['NDCG'].append(ndcg.item())\n", " results['Training Time'].append(training_time)\n", " else:\n", " # save results in dict\n", " results['Epoch'].append(epoch)\n", " results['Loss'].append(loss)\n", " results['Recall'].append(None)\n", " results['NDCG'].append(None)\n", " results['Training Time'].append(training_time)\n", "\n", " if should_stop == True: break\n", "\n", "# save\n", "if args.save_results:\n", " date = today.strftime(\"%d%m%Y_%H%M\")\n", "\n", " # save model as .pt file\n", " if os.path.isdir(\"./models\"):\n", " torch.save(model.state_dict(), \"./models/\" + str(date) + \"_\" + modelname + \"_\" + dataset + \".pt\")\n", " else:\n", " os.mkdir(\"./models\")\n", " torch.save(model.state_dict(), \"./models/\" + str(date) + \"_\" + modelname + \"_\" + dataset + \".pt\")\n", "\n", " # save results as pandas dataframe\n", " results_df = pd.DataFrame(results)\n", " results_df.set_index('Epoch', inplace=True)\n", " if os.path.isdir(\"./results\"):\n", " results_df.to_csv(\"./results/\" + str(date) + \"_\" + modelname + \"_\" + dataset + \".csv\")\n", " else:\n", " os.mkdir(\"./results\")\n", " results_df.to_csv(\"./results/\" + str(date) + \"_\" + modelname + \"_\" + dataset + \".csv\")\n", " # plot loss\n", " results_df['Loss'].plot(figsize=(12,8), title='Loss')" ], "execution_count": null, "outputs": [ { "output_type": "stream", "text": [ "n_users=943, n_items=1682\n", "n_interactions=100000\n", "n_train=80000, n_test=20000, sparsity=0.06305\n", "Creating interaction matrices R_train and R_test...\n", "Complete. Interaction matrices R_train and R_test created in 1.4850668907165527 sec\n", "Loaded adjacency-matrix (shape: (2625, 2625) ) in 0.018111467361450195 sec.\n", "Initializing weights...\n", "Weights initialized.\n", "Start at 2021-07-12 09:57:58.311285\n", "Using cuda for computations\n", "Params on CUDA: True\n", "Epoch: 0, Training time: 9.11s, Loss: 107.9355\n", "Epoch: 1, Training time: 8.88s, Loss: 101.6095\n", "Epoch: 2, Training time: 8.75s, Loss: 80.7764\n", "Epoch: 3, Training time: 8.76s, Loss: 76.1915\n", "Epoch: 4, Training time: 8.58s, Loss: 73.0698\n", "Evaluate current model:\n", " Epoch: 4, Validation time: 1.51s \n", " Loss: 73.0698: \n", " Recall@20: 0.0623 \n", " NDCG@20: 0.2352\n", "Epoch: 5, Training time: 8.84s, Loss: 69.3378\n", "Epoch: 6, Training time: 8.71s, Loss: 64.4498\n", "Epoch: 7, Training time: 8.67s, Loss: 60.1440\n", "Epoch: 8, Training time: 8.76s, Loss: 56.8538\n", "Epoch: 9, Training time: 8.78s, Loss: 52.3951\n", "Evaluate current model:\n", " Epoch: 9, Validation time: 1.54s \n", " Loss: 52.3951: \n", " Recall@20: 0.0837 \n", " NDCG@20: 0.2559\n", "Epoch: 10, Training time: 8.72s, Loss: 50.5261\n", "Epoch: 11, Training time: 8.73s, Loss: 49.2488\n", "Epoch: 12, Training time: 8.72s, Loss: 48.5012\n", "Epoch: 13, Training time: 8.75s, Loss: 47.5585\n", "Epoch: 14, Training time: 8.82s, Loss: 47.0483\n", "Evaluate current model:\n", " Epoch: 14, Validation time: 1.51s \n", " Loss: 47.0483: \n", " Recall@20: 0.0926 \n", " NDCG@20: 0.2676\n", "Epoch: 15, Training time: 8.84s, Loss: 46.4847\n", "Epoch: 16, Training time: 8.98s, Loss: 46.2644\n", "Epoch: 17, Training time: 8.99s, Loss: 45.5963\n", "Epoch: 18, Training time: 8.78s, Loss: 45.0955\n", "Epoch: 19, Training time: 8.84s, Loss: 44.9321\n", "Evaluate current model:\n", " Epoch: 19, Validation time: 1.55s \n", " Loss: 44.9321: \n", " Recall@20: 0.1102 \n", " NDCG@20: 0.2934\n", "Epoch: 20, Training time: 8.61s, Loss: 44.4621\n", "Epoch: 21, Training time: 9.02s, Loss: 44.1910\n", "Epoch: 22, Training time: 8.94s, Loss: 43.7996\n", "Epoch: 23, Training time: 8.83s, Loss: 43.1078\n", "Epoch: 24, Training time: 9.01s, Loss: 43.1549\n", "Evaluate current model:\n", " Epoch: 24, Validation time: 1.54s \n", " Loss: 43.1549: \n", " Recall@20: 0.1217 \n", " NDCG@20: 0.3255\n", "Epoch: 25, Training time: 9.08s, Loss: 42.8759\n", "Epoch: 26, Training time: 8.92s, Loss: 42.4126\n", "Epoch: 27, Training time: 8.82s, Loss: 42.0810\n", "Epoch: 28, Training time: 8.97s, Loss: 41.7865\n", "Epoch: 29, Training time: 8.89s, Loss: 41.3096\n", "Evaluate current model:\n", " Epoch: 29, Validation time: 1.57s \n", " Loss: 41.3096: \n", " Recall@20: 0.1257 \n", " NDCG@20: 0.3217\n", "Epoch: 30, Training time: 9.15s, Loss: 40.9893\n", "Epoch: 31, Training time: 9.11s, Loss: 40.8605\n", "Epoch: 32, Training time: 9.06s, Loss: 40.3089\n", "Epoch: 33, Training time: 8.87s, Loss: 40.1379\n", "Epoch: 34, Training time: 8.89s, Loss: 39.6859\n", "Evaluate current model:\n", " Epoch: 34, Validation time: 1.51s \n", " Loss: 39.6859: \n", " Recall@20: 0.1293 \n", " NDCG@20: 0.3432\n", "Epoch: 35, Training time: 9.12s, Loss: 39.9238\n", "Epoch: 36, Training time: 9.12s, Loss: 39.4329\n", "Epoch: 37, Training time: 9.20s, Loss: 38.9671\n", "Epoch: 38, Training time: 8.79s, Loss: 38.7849\n", "Epoch: 39, Training time: 8.78s, Loss: 38.3410\n", "Evaluate current model:\n", " Epoch: 39, Validation time: 1.54s \n", " Loss: 38.3410: \n", " Recall@20: 0.1365 \n", " NDCG@20: 0.3411\n", "Epoch: 40, Training time: 8.85s, Loss: 38.6723\n", "Epoch: 41, Training time: 8.78s, Loss: 37.9243\n", "Epoch: 42, Training time: 9.07s, Loss: 37.8358\n", "Epoch: 43, Training time: 8.85s, Loss: 37.2368\n", "Epoch: 44, Training time: 8.97s, Loss: 37.4086\n", "Evaluate current model:\n", " Epoch: 44, Validation time: 1.51s \n", " Loss: 37.4086: \n", " Recall@20: 0.1383 \n", " NDCG@20: 0.3554\n", "Epoch: 45, Training time: 8.94s, Loss: 37.1695\n", "Epoch: 46, Training time: 9.05s, Loss: 36.9502\n", "Epoch: 47, Training time: 8.75s, Loss: 36.5551\n", "Epoch: 48, Training time: 9.08s, Loss: 36.4953\n", "Epoch: 49, Training time: 9.13s, Loss: 35.9976\n", "Evaluate current model:\n", " Epoch: 49, Validation time: 1.54s \n", " Loss: 35.9976: \n", " Recall@20: 0.1397 \n", " NDCG@20: 0.3541\n", "Epoch: 50, Training time: 8.79s, Loss: 35.8774\n", "Epoch: 51, Training time: 9.03s, Loss: 36.0130\n", "Epoch: 52, Training time: 9.00s, Loss: 35.4460\n", "Epoch: 53, Training time: 8.76s, Loss: 35.2867\n", "Epoch: 54, Training time: 9.11s, Loss: 35.4907\n", "Evaluate current model:\n", " Epoch: 54, Validation time: 1.53s \n", " Loss: 35.4907: \n", " Recall@20: 0.1435 \n", " NDCG@20: 0.3563\n", "Epoch: 55, Training time: 8.97s, Loss: 35.1628\n", "Epoch: 56, Training time: 8.86s, Loss: 34.5842\n", "Epoch: 57, Training time: 8.83s, Loss: 34.1935\n", "Epoch: 58, Training time: 8.88s, Loss: 34.3039\n", "Epoch: 59, Training time: 8.79s, Loss: 34.2499\n", "Evaluate current model:\n", " Epoch: 59, Validation time: 1.49s \n", " Loss: 34.2499: \n", " Recall@20: 0.1495 \n", " NDCG@20: 0.3704\n", "Epoch: 60, Training time: 8.84s, Loss: 33.9897\n", "Epoch: 61, Training time: 8.66s, Loss: 33.2779\n", "Epoch: 62, Training time: 8.86s, Loss: 33.2062\n", "Epoch: 63, Training time: 8.78s, Loss: 32.9654\n", "Epoch: 64, Training time: 9.03s, Loss: 32.2721\n", "Evaluate current model:\n", " Epoch: 64, Validation time: 1.51s \n", " Loss: 32.2721: \n", " Recall@20: 0.1497 \n", " NDCG@20: 0.3725\n", "Epoch: 65, Training time: 8.90s, Loss: 32.5445\n", "Epoch: 66, Training time: 8.85s, Loss: 32.1805\n", "Epoch: 67, Training time: 8.81s, Loss: 32.1525\n", "Epoch: 68, Training time: 8.80s, Loss: 31.7560\n", "Epoch: 69, Training time: 8.81s, Loss: 31.3688\n", "Evaluate current model:\n", " Epoch: 69, Validation time: 1.51s \n", " Loss: 31.3688: \n", " Recall@20: 0.1536 \n", " NDCG@20: 0.3816\n", "Epoch: 70, Training time: 8.55s, Loss: 31.3098\n", "Epoch: 71, Training time: 8.87s, Loss: 31.3700\n", "Epoch: 72, Training time: 8.72s, Loss: 31.1579\n", "Epoch: 73, Training time: 8.76s, Loss: 30.1733\n", "Epoch: 74, Training time: 8.76s, Loss: 30.5201\n", "Evaluate current model:\n", " Epoch: 74, Validation time: 1.50s \n", " Loss: 30.5201: \n", " Recall@20: 0.1581 \n", " NDCG@20: 0.3809\n", "Epoch: 75, Training time: 8.70s, Loss: 30.2994\n", "Epoch: 76, Training time: 8.76s, Loss: 29.8949\n", "Epoch: 77, Training time: 8.77s, Loss: 29.7122\n", "Epoch: 78, Training time: 8.74s, Loss: 29.7030\n", "Epoch: 79, Training time: 8.64s, Loss: 29.6655\n", "Evaluate current model:\n", " Epoch: 79, Validation time: 1.49s \n", " Loss: 29.6655: \n", " Recall@20: 0.1609 \n", " NDCG@20: 0.3873\n", "Epoch: 80, Training time: 8.94s, Loss: 29.6567\n", "Epoch: 81, Training time: 8.87s, Loss: 29.5109\n", "Epoch: 82, Training time: 8.91s, Loss: 29.1704\n", "Epoch: 83, Training time: 8.82s, Loss: 28.6625\n", "Epoch: 84, Training time: 8.79s, Loss: 28.7304\n", "Evaluate current model:\n", " Epoch: 84, Validation time: 1.48s \n", " Loss: 28.7304: \n", " Recall@20: 0.1613 \n", " NDCG@20: 0.3908\n", "Epoch: 85, Training time: 8.85s, Loss: 29.0495\n", "Epoch: 86, Training time: 8.76s, Loss: 28.4390\n", "Epoch: 87, Training time: 8.81s, Loss: 28.5633\n", "Epoch: 88, Training time: 8.83s, Loss: 28.3275\n", "Epoch: 89, Training time: 8.96s, Loss: 27.8343\n", "Evaluate current model:\n", " Epoch: 89, Validation time: 1.52s \n", " Loss: 27.8343: \n", " Recall@20: 0.1591 \n", " NDCG@20: 0.3895\n", "Epoch: 90, Training time: 8.92s, Loss: 28.3271\n", "Epoch: 91, Training time: 8.85s, Loss: 28.0346\n", "Epoch: 92, Training time: 8.69s, Loss: 27.7937\n", "Epoch: 93, Training time: 8.93s, Loss: 27.5649\n", "Epoch: 94, Training time: 9.08s, Loss: 27.9189\n", "Evaluate current model:\n", " Epoch: 94, Validation time: 1.50s \n", " Loss: 27.9189: \n", " Recall@20: 0.1611 \n", " NDCG@20: 0.3912\n", "Epoch: 95, Training time: 8.86s, Loss: 27.9343\n", "Epoch: 96, Training time: 8.83s, Loss: 27.2735\n", "Epoch: 97, Training time: 8.92s, Loss: 27.3794\n", "Epoch: 98, Training time: 8.84s, Loss: 27.2788\n", "Epoch: 99, Training time: 8.86s, Loss: 27.4216\n", "Evaluate current model:\n", " Epoch: 99, Validation time: 1.50s \n", " Loss: 27.4216: \n", " Recall@20: 0.1656 \n", " NDCG@20: 0.3922\n", "Epoch: 100, Training time: 8.71s, Loss: 26.6066\n", "Epoch: 101, Training time: 8.88s, Loss: 27.1389\n", "Epoch: 102, Training time: 9.04s, Loss: 26.6459\n", "Epoch: 103, Training time: 8.71s, Loss: 26.8171\n", "Epoch: 104, Training time: 8.91s, Loss: 26.7730\n", "Evaluate current model:\n", " Epoch: 104, Validation time: 1.49s \n", " Loss: 26.7730: \n", " Recall@20: 0.1627 \n", " NDCG@20: 0.3926\n", "Epoch: 105, Training time: 9.06s, Loss: 26.4580\n", "Epoch: 106, Training time: 9.12s, Loss: 25.9192\n", "Epoch: 107, Training time: 8.93s, Loss: 26.4427\n", "Epoch: 108, Training time: 8.77s, Loss: 26.3804\n", "Epoch: 109, Training time: 8.86s, Loss: 26.1349\n", "Evaluate current model:\n", " Epoch: 109, Validation time: 1.52s \n", " Loss: 26.1349: \n", " Recall@20: 0.1691 \n", " NDCG@20: 0.3950\n", "Epoch: 110, Training time: 8.81s, Loss: 25.8410\n", "Epoch: 111, Training time: 8.84s, Loss: 25.9275\n", "Epoch: 112, Training time: 8.77s, Loss: 25.9278\n", "Epoch: 113, Training time: 8.92s, Loss: 26.2235\n", "Epoch: 114, Training time: 8.90s, Loss: 25.4737\n", "Evaluate current model:\n", " Epoch: 114, Validation time: 1.50s \n", " Loss: 25.4737: \n", " Recall@20: 0.1673 \n", " NDCG@20: 0.3995\n", "Epoch: 115, Training time: 8.78s, Loss: 25.7582\n", "Epoch: 116, Training time: 8.77s, Loss: 25.3173\n", "Epoch: 117, Training time: 8.63s, Loss: 25.4568\n", "Epoch: 118, Training time: 8.63s, Loss: 25.3934\n", "Epoch: 119, Training time: 8.63s, Loss: 25.2544\n", "Evaluate current model:\n", " Epoch: 119, Validation time: 1.50s \n", " Loss: 25.2544: \n", " Recall@20: 0.1689 \n", " NDCG@20: 0.4028\n", "Epoch: 120, Training time: 8.77s, Loss: 24.9747\n", "Epoch: 121, Training time: 8.93s, Loss: 24.7825\n", "Epoch: 122, Training time: 8.92s, Loss: 25.2147\n", "Epoch: 123, Training time: 8.79s, Loss: 24.5176\n", "Epoch: 124, Training time: 8.72s, Loss: 24.7453\n", "Evaluate current model:\n", " Epoch: 124, Validation time: 1.48s \n", " Loss: 24.7453: \n", " Recall@20: 0.1682 \n", " NDCG@20: 0.3954\n", "Epoch: 125, Training time: 8.78s, Loss: 24.9444\n", "Epoch: 126, Training time: 8.81s, Loss: 24.9258\n", "Epoch: 127, Training time: 8.77s, Loss: 24.5360\n", "Epoch: 128, Training time: 8.70s, Loss: 24.4527\n", "Epoch: 129, Training time: 8.65s, Loss: 24.5864\n", "Evaluate current model:\n", " Epoch: 129, Validation time: 1.48s \n", " Loss: 24.5864: \n", " Recall@20: 0.1689 \n", " NDCG@20: 0.3977\n", "Epoch: 130, Training time: 8.66s, Loss: 24.2351\n", "Epoch: 131, Training time: 8.84s, Loss: 24.4298\n", "Epoch: 132, Training time: 8.57s, Loss: 24.3624\n", "Epoch: 133, Training time: 8.74s, Loss: 24.1980\n", "Epoch: 134, Training time: 8.84s, Loss: 24.0672\n", "Evaluate current model:\n", " Epoch: 134, Validation time: 1.47s \n", " Loss: 24.0672: \n", " Recall@20: 0.1735 \n", " NDCG@20: 0.4069\n", "Epoch: 135, Training time: 8.75s, Loss: 24.4691\n", "Epoch: 136, Training time: 8.67s, Loss: 23.9019\n", "Epoch: 137, Training time: 8.77s, Loss: 24.1378\n", "Epoch: 138, Training time: 8.68s, Loss: 23.8090\n", "Epoch: 139, Training time: 8.81s, Loss: 23.9487\n", "Evaluate current model:\n", " Epoch: 139, Validation time: 1.48s \n", " Loss: 23.9487: \n", " Recall@20: 0.1687 \n", " NDCG@20: 0.4037\n", "Epoch: 140, Training time: 8.64s, Loss: 23.8015\n", "Epoch: 141, Training time: 8.57s, Loss: 24.0985\n", "Epoch: 142, Training time: 8.70s, Loss: 23.8640\n", "Epoch: 143, Training time: 8.77s, Loss: 23.5799\n", "Epoch: 144, Training time: 8.77s, Loss: 23.7568\n", "Evaluate current model:\n", " Epoch: 144, Validation time: 1.48s \n", " Loss: 23.7568: \n", " Recall@20: 0.1708 \n", " NDCG@20: 0.4068\n", "Epoch: 145, Training time: 8.75s, Loss: 23.6537\n", "Epoch: 146, Training time: 8.77s, Loss: 23.8114\n", "Epoch: 147, Training time: 8.64s, Loss: 23.5442\n", "Epoch: 148, Training time: 8.51s, Loss: 23.2413\n", "Epoch: 149, Training time: 8.77s, Loss: 23.5159\n", "Evaluate current model:\n", " Epoch: 149, Validation time: 1.49s \n", " Loss: 23.5159: \n", " Recall@20: 0.1698 \n", " NDCG@20: 0.4052\n", "Epoch: 150, Training time: 8.67s, Loss: 23.4435\n", "Epoch: 151, Training time: 8.54s, Loss: 23.5388\n", "Epoch: 152, Training time: 8.54s, Loss: 23.2494\n", "Epoch: 153, Training time: 8.60s, Loss: 23.1259\n", "Epoch: 154, Training time: 8.68s, Loss: 23.1326\n", "Evaluate current model:\n", " Epoch: 154, Validation time: 1.49s \n", " Loss: 23.1326: \n", " Recall@20: 0.1709 \n", " NDCG@20: 0.4059\n", "Epoch: 155, Training time: 8.69s, Loss: 22.8828\n", "Epoch: 156, Training time: 8.60s, Loss: 23.0292\n", "Epoch: 157, Training time: 8.54s, Loss: 22.9355\n", "Epoch: 158, Training time: 8.61s, Loss: 22.7000\n", "Epoch: 159, Training time: 8.83s, Loss: 22.9723\n", "Evaluate current model:\n", " Epoch: 159, Validation time: 1.47s \n", " Loss: 22.9723: \n", " Recall@20: 0.1731 \n", " NDCG@20: 0.4118\n", "Early stopping at step: 5 log:0.1731223464012146\n" ], "name": "stdout" }, { "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAHwCAYAAABOlBKbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3zd5X33//d1ls7R3tOSlzzkiY0xZhMgCZvQ7EFIQ0Lvlmb3kdk2d9M0bXqnd0hzN00hJCG/JCQpGSYJGZhA2QYbG+8hW7b23jo60hnX7w/JRhgZrPk90vf1fDz8sM/3DH30fQjz5uJzfS5jrRUAAADgZh6nCwAAAACcRigGAACA6xGKAQAA4HqEYgAAALgeoRgAAACuRygGAACA6xGKAQAA4HqEYgBIEsaYE8aYa5yuAwDciFAMAAAA1yMUA0ASM8akGGPuNsY0jv662xiTMvpcvjHmN8aYbmNMpzHmSWOMZ/S5zxhjGowxfcaYw8aYq539TgAgufmcLgAA8Jq+IGmLpPMkWUlbJf2tpL+T9ClJ9ZIKRl+7RZI1xqyQ9NeSLrDWNhpjFknyzm7ZADC3sFIMAMntvZK+ZK1ttda2SfoHSbeNPheVVCJpobU2aq190lprJcUlpUhaZYzxW2tPWGuPOVI9AMwRhGIASG6lkk6OeXxy9Jok/R9J1ZL+aIw5boz5rCRZa6slfVzS/5bUaoz5iTGmVACAsyIUA0Bya5S0cMzjitFrstb2WWs/Za1dIulmSZ881Ttsrf2xtfbS0fdaSV+d3bIBYG4hFANAcvEbY4Knfkl6QNLfGmMKjDH5kv5e0g8lyRhzozGm0hhjJPVopG0iYYxZYYy5anRDXkTSoKSEM98OAMwNhGIASC4PayTEnvoVlLRD0h5JeyW9KOnLo69dJmmbpH5Jz0r6lrX2MY30E/+LpHZJzZIKJX1u9r4FAJh7zMieDAAAAMC9WCkGAACA6xGKAQAA4HqEYgAAALgeoRgAAACuRygGAACA6/mcLkCS8vPz7aJFi5wuAwAAAPPczp072621BWdeT4pQvGjRIu3YscPpMgAAADDPGWNOjned9gkAAAC4HqEYAAAArkcoBgAAgOsRigEAAOB6hGIAAAC4HqEYAAAArkcoBgAAgOsRigEAAOB6hGIAAAC4HqEYAAAArkcoBgAAgOsRigEAAOB6hGIAAAC4HqEYAAAArkcoBgAAgOsRigEAAOB6rg3F1lp1h4c1FIs7XQoAAAAc5tpQ/HR1h8770iPaXdvtdCkAAABwmGtD8YKckCSprmvQ4UoAAADgNNeG4tLskIyR6jrDTpcCAAAAh7k2FAd8HpVmhQjFAAAAcG8olkZaKOq6CMUAAABu5+pQXJ6bqlpWigEAAFzP1aG4IjdVLb1DikQZywYAAOBmrg7F5bkjEygauplAAQAA4GauDsUVuamSRAsFAACAy7k6FJfnjITiekIxAACAq7k6FBdkpCjF52GlGAAAwOVcHYqNMSrPTVVdJz3FAAAAbubqUCxJ5cwqBgAAcD1CMbOKAQAAXM/1obgiN1V9kZh6wlGnSwEAAIBDXB+KF+Qwlg0AAMDtXB+KT80qpq8YAADAvVwfik+dalfHSjEAAIBruT4UZwT9yk710z4BAADgYq4PxdJIC0VdF7OKAQAA3IpQrJHjnmmfAAAAcC9CsaQFuSE1dA0qkbBOlwIAAAAHEIo10j4xHE+opS/idCkAAABwAKFYI+0TklTbQQsFAACAGxGKNXZWMZvtAAAA3IhQLKk0OyRjONUOAADArQjFkgI+j0oyg6onFAMAALgSoXhUeW4qK8UAAAAuRSgeVZ6bqrouQjEAAIAbEYpHleekqqV3SJFo3OlSAAAAMMsIxaMKMlIkSV3hYYcrAQAAwGwjFI9KDXglSYPDrBQDAAC4DaF4VNA/GoppnwAAAHAdQvGo0OhKMT3FAAAA7vO6odgY811jTKsxZt+Ya7nGmEeMMUdHf88ZvW6MMf9ujKk2xuwxxmycyeKnU+jUSvFwwuFKAAAAMNvOZaX4+5KuPePaZyU9aq1dJunR0ceSdJ2kZaO/7pT0n9NT5swL0T4BAADgWq8biq21T0jqPOPyLZLuH/3z/ZLeMub6D+yI5yRlG2NKpqvYmRQKjNwKQjEAAID7TLanuMha2zT652ZJRaN/LpNUN+Z19aPXXsUYc6cxZocxZkdbW9sky5g+oYBPkjQ4HHO4EgAAAMy2KW+0s9ZaSXYS77vHWrvJWrupoKBgqmVM2cs9xawUAwAAuM1kQ3HLqbaI0d9bR683SCof87oFo9eS3ss9xWy0AwAAcJvJhuKHJN0++ufbJW0dc/39o1MotkjqGdNmkdRSfPQUAwAAuJXv9V5gjHlA0pWS8o0x9ZK+KOlfJP3MGHOHpJOS3jH68oclXS+pWlJY0p/PQM0zwuMxCvo9zCkGAABwodcNxdbad5/lqavHea2VdNdUi3JKyO+lpxgAAMCFONFujJDfS/sEAACACxGKxwgFWCkGAABwI0LxGKEAK8UAAABuRCgeg55iAAAAdyIUjxGkpxgAAMCVCMVjhPxeRrIBAAC4EKF4DHqKAQAA3IlQPAY9xQAAAO5EKB6DkWwAAADuRCgeg8M7AAAA3IlQPEbI71UsYRWNJ5wuBQAAALOIUDxGKOCVJFaLAQAAXIZQPEbQPxKKI/QVAwAAuAqheIyQn5ViAAAANyIUj0H7BAAAgDsRisc4HYppnwAAAHAVQvEYp9snCMUAAACuQigeg55iAAAAdyIUj0FPMQAAgDsRisegfQIAAMCdCMVjnJ5TzEoxAACAqxCKx6B9AgAAwJ0IxWO83D6RcLgSAAAAzCZC8Rhej1HA51E4GnO6FAAAAMwiQvEZQn6vImy0AwAAcBVC8RlCfi89xQAAAC5DKD5DKODVYJSeYgAAADchFJ8h6PcypxgAAMBlCMVnSA14mVMMAADgMoTiM9BTDAAA4D6E4jME/V6FaZ8AAABwFULxGUK0TwAAALgOofgMIb+HjXYAAAAuQyg+Az3FAAAA7kMoPkMwQCgGAABwG0LxGVL9Pg3HEoonrNOlAAAAYJYQis8QCozcEjbbAQAAuAeh+Awhv1eSaKEAAABwEULxGYKnQjETKAAAAFyDUHyGUICVYgAAALchFJ8hxEoxAACA6xCKz0BPMQAAgPsQis9A+wQAAID7EIrPcCoUR2ifAAAAcA1C8RlonwAAAHAfQvEZToXiMCvFAAAArkEoPkPwVPsEK8UAAACuQSg+AyPZAAAA3IdQfAa/1yOfx9BTDAAA4CKE4nGEAl5CMQAAgIsQiscR8nvpKQYAAHARQvE4QgEvPcUAAAAuQigeR8jvZSQbAACAixCKxxH001MMAADgJoTicdBTDAAA4C6E4nEwfQIAAMBdCMXjYKMdAACAuxCKxzHSPpFwugwAAADMEkLxOEJstAMAAHAVQvE4aJ8AAABwF0LxOE6NZEskrNOlAAAAYBYQiscR8nslSUMx+ooBAADcgFA8jpB/5LbQVwwAAOAOhOJxpAZ8kgjFAAAAbkEoHkcwMNI+wWY7AAAAdyAUj+NUTzFHPQMAALgDoXgcp0Ix7RMAAADuQCgeRygwclvCtE8AAAC4AqF4HEE/PcUAAABuQigeBz3FAAAA7kIoHgcj2QAAANyFUDyOEO0TAAAArkIoHkcwwIl2AAAAbkIoHkfA65HH0FMMAADgFoTicRhjFPJ7GckGAADgEoTiswgFvLRPAAAAuASh+CxCAa8irBQDAAC4AqH4LEJ+VooBAADcglB8FoRiAAAA9yAUn0XQ72VOMQAAgEsQis8iFPAykg0AAMAlCMVnwUg2AAAA9yAUnwU9xQAAAO5BKD6L1BSv+odiTpcBAACAWTClUGyM+YQxZr8xZp8x5gFjTNAYs9gYs90YU22M+akxJjBdxc6mRXlp6g5H1dE/5HQpAAAAmGGTDsXGmDJJH5W0yVq7RpJX0rskfVXS1621lZK6JN0xHYXOtpXFmZKkw819DlcCAACAmTbV9gmfpJAxxicpVVKTpKskPTj6/P2S3jLFr+GIlSUZkqSDhGIAAIB5b9Kh2FrbIOlrkmo1EoZ7JO2U1G2tPdWMWy+pbLz3G2PuNMbsMMbsaGtrm2wZMyY/PUX56Sk61NTrdCkAAACYYVNpn8iRdIukxZJKJaVJuvZc32+tvcdau8lau6mgoGCyZcyoqpIMHWwmFAMAAMx3U2mfuEZSjbW2zVoblfQLSZdIyh5tp5CkBZIaplijY6pKMnWkpV+xeMLpUgAAADCDphKKayVtMcakGmOMpKslHZD0mKS3jb7mdklbp1aic1YWZ2g4ltCJjgGnSwEAAMAMmkpP8XaNbKh7UdLe0c+6R9JnJH3SGFMtKU/SfdNQpyNOTaA42MRmOwAAgPnM9/ovOTtr7RclffGMy8clbZ7K5yaLpYVp8nmMDjX36qb1pU6XAwAAgBnCiXavIcXn1dKCdB1ipRgAAGBeIxS/jpUlGTrErGIAAIB5jVD8OlYWZ6qhe1A9g1GnSwEAAMAMIRS/jlMn23GIBwAAwPxFKH4dq0pGJlDQQgEAADB/EYpfR2FGinJS/TrEyXYAAADzFqH4dRhjtLI4k1nFAAAA8xih+BysLMnQ4eY+JRLW6VIAAAAwAwjF56CqOFOD0bhqO8NOlwIAAIAZQCg+B6cnUNBXDAAAMC8Ris/BssIMeYzoKwYAAJinCMXnIBTwalF+mg4wqxgAAGBeIhSfo/PKs7XjRCeb7QAAAOYhQvE5urQyX13hqA7SVwwAADDvEIrP0SWV+ZKkp6vbHa4EAAAA041QfI6KMoOqLEzXU9UdTpcCAACAaUYonoBLK/P1Qk2nhmJxp0sBAADANCIUT8AllfkajMa1q7bb6VIAAAAwjQjFE3Dhklx5DH3FAAAA8w2heAIyg36tL88mFAMAAMwzhOIJurQyXy/V96g3EnW6FAAAAEwTQvEEXbw0X/GE1fbjnU6XAgAAgGlCKJ6gjQuzFfR7aKEAAACYRwjFE5Ti82rz4jxCMQAAwDxCKJ6ES5bm6Whrv1p6I06XAgAAgGlAKJ6EU0c+P3OM1WIAAID5gFA8CatKMpWd6tezxzjyGQAAYD4gFE+Cx2O0sSJHL3KyHQAAwLxAKJ6kjRXZqm7tV0+YecUAAABzHaF4kjZU5EiSdtezWgwAADDXEYonaX15tjxGevFkl9OlAAAAYIoIxZOUnuLT8qIM7apjpRgAAGCuIxRPwYaKHO2q7VIiYZ0uBQAAAFNAKJ6CjRXZ6ovEdKyt3+lSAAAAMAWE4ik4tdluF6PZAAAA5jRC8RQsyU9TVsivF2vZbAcAADCXEYqnwOMx2lCRzUoxAADAHEconqIN5Tk60tqn3giHeAAAAMxVhOIp2rgwW9ZKLzGaDQAAYM4iFE/R+vJsGcNmOwAAgLmMUDxFmUG/lhWms9kOAABgDiMUT4ONFTnaVdvNIR4AAABzFKF4GmyoyFbPYFQ1HQNOlwIAAIBJIBRPg40c4gEAADCnEYqnwZKCdKX4PDrc3Ot0KQAAAJgEQvE08HqMlhak60hLv9OlAAAAYBIIxdNkeVG6jrb0OV0GAAAAJoFQPE2WF2eosSfCyXYAAABzEKF4miwvzJAkHaWFAgAAYM4hFE+T5UWnQjEtFAAAAHMNoXiaLMgJKeT3stkOAABgDiIUTxOPx6iyMF1HW1kpBgAAmGsIxdNoeVGGDjcTigEAAOYaQvE0Wl6Urta+IfWEmUABAAAwlxCKp9GpzXZHaKEAAACYUwjF02hZUbok6QgTKAAAAOYUQvE0KssOKS3g1RH6igEAAOYUQvE0MsaosiiDsWwAAABzDKF4mq0oYiwbAADAXEMonmbLizLU3j+szoFhp0sBAADAOSIUT7NlpyZQsNkOAABgziAUT7PlTKAAAACYcwjF06w4M6iMFB+hGAAAYA4hFE8zY4yWFzOBAgAAYC4hFM+A5UXpOtrSJ2ut06UAAADgHBCKZ8Cywgx1haNq72cCBQAAwFxAKJ4BK4pHJlAcau51uBIAAACcC0LxDFhVkilJOtBIKAYAAJgLCMUzICctoNKsoPYTigEAAOYEQvEMWVWapf2NPU6XAQAAgHNAKJ4hq0szdbx9QOHhmNOlAAAA4HUQimfImrIsWSsdbKKFAgAAINkRimfI6tKRzXb0FQMAACQ/QvEMKckKKifVr/0NhGIAAIBkRyieIcYYrS7N0v4mNtsBAAAkO0LxDFpdmqkjzf2KxhNOlwIAAIDXQCieQatKMzUcT+hoS7/TpQAAAOA1EIpn0OrSLEliXjEAAECSIxTPoMX5aQr5vUygAAAASHKE4hnk9RhVlWToAKEYAAAgqRGKZ9jq0iwdaOpVImGdLgUAAABnQSieYWvKMtU/FFNtZ9jpUgAAAHAWhOIZdmqz3T422wEAACStKYViY0y2MeZBY8whY8xBY8xFxphcY8wjxpijo7/nTFexc9GyonT5PIbNdgAAAElsqivF35D0e2vtSknrJR2U9FlJj1prl0l6dPSxa6X4vFpWlEEoBgAASGKTDsXGmCxJl0u6T5KstcPW2m5Jt0i6f/Rl90t6y1SLnOtWl2bqQGOPrGWzHQAAQDKaykrxYkltkr5njNlljPmOMSZNUpG1tmn0Nc2SiqZa5Fy3tixL7f3DaugedLoUAAAAjGMqodgnaaOk/7TWbpA0oDNaJezI0ui4y6PGmDuNMTuMMTva2tqmUEby27w4V5K0/Xinw5UAAABgPFMJxfWS6q2120cfP6iRkNxijCmRpNHfW8d7s7X2HmvtJmvtpoKCgimUkfxWFGUoJ9WvZ493OF0KAAAAxjHpUGytbZZUZ4xZMXrpakkHJD0k6fbRa7dL2jqlCucBj8fowsV5eo5QDAAAkJR8U3z/RyT9yBgTkHRc0p9rJGj/zBhzh6STkt4xxa8xL2xZkqvf729WXWdY5bmpTpcDAACAMaYUiq21uyVtGuepq6fyufPRlqV5kqTtNZ2EYgAAgCTDiXazZHlhhnLTArRQAAAAJCFC8SwZ6SvO1bPHCMUAAADJhlA8i7YsyVND96DqOsNOlwIAAIAxCMWzaMuSkb5iWigAAACSC6F4Fi0rTB/tK+YQDwAAgGRCKJ5FHo/RliW5eu54h0YO+wMAAEAyIBTPslN9xfVdg06XAgAAgFGE4ll2qq+YI58BAACSB6F4li0rTFce84oBAACSCqF4lhljdOGSXD1fw2Y7AACAZEEodsDasmzVdw2qJxx1uhQAAACIUOyIVaWZkqQDTb0OVwIAAACJUOyIqpIMSdJBQjEAAEBSIBQ7oDAjqPz0FEIxAABAkiAUO6SqJIP2CQAAgCRBKHbIqpJMHW3pVzSecLoUAAAA1yMUO2RVaaaG4wkda+t3uhQAAADXIxQ7pKpkZAIFfcUAAADOIxQ7ZEl+mgI+jw429TldCgAAgOsRih3i83q0oihDBxpZKQYAAHAaodhBVSUZOtjUK2ut06UAAAC4GqHYQatKMtUxMKy2viGnSwEAAHA1QrGDTm22289mOwAAAEcRih1UVcoECgAAgGRAKHZQZtCvBTkhNtsBAAA4jFDssKqSTFaKAQAAHEYodtiqkkzVtA9ocDjudCkAAACuRSh2WFVJphJWOtzCIR4AAABOIRQ7bDWb7QAAABxHKHbYgpyQMlJ8bLYDAABwEKHYYcYYrS/P1uNHWhVPcLIdAACAEwjFSeC9F1aornNQjxxocboUAAAAVyIUJ4E3rS7WgpyQvvtUjdOlAAAAuBKhOAl4PUYfuHiRnj/Rqb31PU6XAwAA4DqE4iTxzgvKlZ7i031PHXe6FAAAANchFCeJjKBf79hUrt/saVJzT8TpcgAAAFyFUJxE/vySRUpYqx88e8LpUgAAAFyFUJxEynNT9aZVxfrR9lqFh2NOlwMAAOAahOIkc8dli9UzGNXPXqhzuhQAAADXIBQnmU0Lc3TRkjz92yNH1NpLbzEAAMBsIBQnGWOMvvJnazUcS+jvt+53uhwAAABXIBQnocX5afrEG5fr9/ub9bu9TU6XAwAAMO8RipPUhy5drDVlmfq7rfvVE446XQ4AAMC8RihOUj6vR1996zp1hYf15d8ecLocAACAeY1QnMRWl2bpLy5fov/eWa+tuxucLgcAAGDeIhQnuY9evUybF+Xq4z/drfufOeF0OQAAAPMSoTjJBf1e/eCOzbqmqkhffGi/vvaHw7LWOl0WAADAvEIongOCfq/+870b9e7N5fp/j1Xrsz/fq3iCYAwAADBdfE4XgHPj83r0lVvXKj89Rd/8U7VCAa++eNMqGWOcLg0AAGDOIxTPIcYYfepNKxSJxnXvkzVakBPShy5b4nRZAAAAcx6heA763HVVauge1Jd/e1AlWSHdsK7E6ZIAAADmNHqK5yCPx+j/vuM8bVqYo0/8bLdeONHpdEkAAABzGqF4jgr6vbr3/Zu0IDukD37vBf1o+0kl2HwHAAAwKYTiOSwnLaAf3LFZq0oz9YVf7tPbvv2MDjb1Ol0WAADAnEMonuMW5KTqJ3du0b+9fb1OdIR14zef0j8/fFCRaNzp0gAAAOYMQvE8YIzRW89foEc/eYXetnGB/uuJ47rpm09pb32P06UBAADMCYTieSQnLaCvvm2d7v/gZvVGorr1W0/r7m1HFI0nnC4NAAAgqZlkODJ406ZNdseOHU6XMa/0hKP64kP79KvdjcpLC+iSynxdWpmvS5flqzQ75HR5AAAAjjDG7LTWbjrzOnOK56msVL/uftcG3XxeqX79UpOeqm7XQy81SpL+bGOZvnLrWgX9XoerBAAASA6E4nnuqpVFumplkay1OtLSr1/uatC3/+eYjrb0679uO59VYwAAANFT7BrGGK0oztBnr1upe9+/STXtA7r5/z3FwR8AAAAiFLvSG1cV6Vd3XayMoF/vufc53b3tCCPcAACAqxGKXaqyMEO/uusSXbemRHdvO6o33/2EHjvc6nRZAAAAjmD6BPR0dbv+bus+HW8b0DVVhbpqZZHWlmVpeXG6UnxsxgMAAPPH2aZPEIohSRqOJXTvk8d175PH1R2OSpL8XqMN5Tn61JuW68IleQ5XCAAAMHWEYpwTa63qOge1r7FHext6tHVXgxp7IrpuTbE+d12VKvJSnS4RAABg0gjFmJTB4bi+8+RxfevxY4onrN61uVw3ry/VxooceTzG6fIAAAAmhFCMKWnuiehrfzysh3Y3ajieUFFmiq5dXay3nV+utQuynC4PAADgnBCKMS36IlH96VCrHt7bpMcPt2koltBly/L1l1cu1UVL8mQMq8cAACB5EYox7XojUf3ouVrd91SN2vuHdF55tr50y2qtW5DtdGkAAADjOlsoZk4xJi0z6NdfXrlUT33mDfrHt6xRc09Eb//2s/r1S41OlwYAADAhhGJMWdDv1W1bFuq3H71U6xZk6SMP7NLd244oGf4vBAAAwLnwOV0A5o+89BT98EMX6vO/2Ke7tx3V4eY+XbAoV/1DMfUPxWSt1RXLC7VlSa58Xv57DAAAJA96ijHtrLW654nj+pffH9KpH6+Q36u4tRqOJZSXFtC1a4r1ZxsX6PyFOc4WCwAAXIWNdph13eFhWSulB33yez2KRON6/HCrfr2nSX862KrBaFzXVBXpc9ev1NKCdKfLBQAALkAoRlIJD8d0/zMn9R+PVSsSjet9Wxbqo1cvU25awOnSAADAPEYoRlJq7x/S1x85ogeer5Xf69EN60r03gsrtLEih5nHAABg2hGKkdSqW/v0vadPaOvuRvUPxbSiKEPvuKBcN60vUWFG0OnyAADAPEEoxpwwMBTTQy816oHna7WnvkceI11Sma9bzivTjetKFPR7nS4RAADMYYRizDnVrX3aurtRv9rdoLrOQZVlh/SFG6p03ZpiWisAAMCkEIoxZ1lr9VR1u/7ptwd1qLlPFy7O1RdvWq1VpZlOlwYAAOYYQjHmvFg8oZ+8UKd/++NhdQ9GddmyAr3rgnJdU1WkgI/DQAAAwOsjFGPe6AlHdd/TNfrvHXVq6okoNy2gG9eVaE1plpYWpmlpQbqyUxntBgAAXo1QjHknnrB68mibfvpCnR491KrhWOL0c5WF6frKrWu1eXGugxUCAIBkM2Oh2BjjlbRDUoO19kZjzGJJP5GUJ2mnpNustcOv9RmEYkxVPGFV3xXWsbZ+Vbf264fP1aquK6wPXLxIn37zSoUCTK0AAABnD8XT0Yj5MUkHxzz+qqSvW2srJXVJumMavgbwmrweo4V5abpqZZHuvHypfvexy/T+LQv1vadP6NpvPKHf7GlUc0/E6TIBAECSmtJKsTFmgaT7Jf2TpE9KuklSm6Ria23MGHORpP9trX3za30OK8WYKc8e69Cnf/6S6joHJUmFGSlatyBbN60v0U3rSuXxMNoNAAA3OdtKsW+Kn3u3pE9Lyhh9nCep21obG31cL6lsil8DmLSLluZp2yev0L6GXu2p79ae+h7tONmpbT9p0XeerNHnrl+pi5fmS5K6Bob1xNE2HWzq01s3lmlZUcbrfDoAAJgvJh2KjTE3Smq11u40xlw5ifffKelOSaqoqJhsGcDrSvF5df7CHJ2/MEeSlEhY/Wp3g772h8N6z73bdWllvsLDMe2u61Zi9H+c3Pvkcd22ZaE+cc1yZaX6HaweAADMhkm3Txhj/lnSbZJikoKSMiX9UtKbRfsE5oBINK77nzmh7zxVo9KsoK5cUagrVxSoPDdVd287oh9vr1VWyK9PvmmF3n1BuXxeZiEDADDXzehIttGV4r8ZnT7x35J+bq39iTHm25L2WGu/9VrvJxQjGR1s6tU//Hq/njveqZXFGfr7m1adbrUAAABz00xOnzjTZyR90hhTrZEe4/tm4GsAM66qJFMPfHiL/vO9G9UXiek9927XX/5wp4639SsZ5nsDAIDpw+EdwDmIROO694nj+tbjxzQYjasoM0UbK3K0oSJbV60sUmVhutMlAgCAc8CJdsA0aO6J6A/7m/VibZd21XartjMsj5HeunGBPvHG5SrNDkmS+odiemh3o363r0m3X7RI16wqcrhyAAAgEYqBGdHSG9G9T2+gTucAAB9TSURBVBzXD549KRnp9osWqn8orod2N2hgOK70FJ/CwzF95da1etdmpqwAAOC0mZpTDLhaUWZQf3vjKn3gkkX6+iNH9Z2napTi8+imdaV694UVWlGUob/60Yv67C/2qr1/SHe9oVLGcGAIAADJhpViYBo190SUmuJVZvDl2cbReEKfeXCPfrGrQe+9sEK3bihTcVZQhRlBBXyMeQMAYDaxUgzMguKs4Kuu+b0efe3t65WfkaJ7njiuH22vPf3ckoI0feXWtdqyJG82ywQAAGdgpRiYRcfb+lXbGVZzT0TNvRFt3d2okx0D+us3VOqjVy/jgBAAAGYYK8VAElhSkK4lBS+Pb/vwZUv0xYf269//VK2nj3XoI1dVqrq1X7vrurW3oUchv1eXLcvXFcsLtWlRjoJ+r4PVAwAwf7FSDCSBrbsb9IVf7lP/UEySVJYd0tqyLPVGotpxokvD8YSCfo+2LMnT5csKdPnyAi0tSGPTHgAAE8RKMZDEbjmvTJsX5+pwc59Wl2apICPl9HPh4ZieO96hJ46064kjbfrS4QOSRoLzuzeX67aLFikr5D/bRwMAgHPASjEwx9R1hvXE0Tb9fl+znjzarowUn95/8UJ98JLFyktPef0PAADAxTi8A5iH9jX06FuPV+t3+5rlMUY5qX5lhfzKSQ1oUX6aPnb1MpXnpjpdJgAASYNQDMxj1a39emh3g9oHhtUdHlbXQFQv1XcrnrD6qysr9RdXLGGTHgAAIhQDrtPUM6gv//agfrunSQvzUnXXlZVaWZKhRflpygz6Za1Vc29ER1v6dbJjQBdX5mvpmMkYAADMR4RiwKWeOtquv39on463DZy+lpcW0HAsob7RaReSlOLz6DPXrtQHLl4kj4epFgCA+YlQDLhYLJ7QsbYB1bQP6ETHgGraBhTwebS8KF2VhRkqyAjonx8+pEcPteqiJXn62jvWqyw7NO5nWWtlrQjOAIA5iVAM4DVZa/WzHXX60q8PyBijSyrztLo0S2vKMlWSFdJLdd167niHttd0qmcwqpvXl+rdmyu0bkEW85IBAHMGoRjAOanrDOvrjxzRrrpu1bQPvOK5/PSALlycp6Dfq4f3NmkwGldVSaY+fNli3bqhjHAMAEh6hGIAE9YXiepgU58auwe1pizrFafo9UWi2rq7UT/aXquDTb26pqpI//LWtcpnVjIAIIkRigHMiETC6rtP1+hf/3BYmUGf/vVt63TVyiKnywIAYFxnC8UeJ4oBMH94PEYfumyJfv3Xlyo/PUUf/P4OffSBXTrc3Od0aQAAnDNWigFMm6FYXN98tFrffbpG4eG4rqkq1F9euVR5aSmq7QyrtjOspp5BBX1eZaX6lRn0qzAjRRsX5nC4CABgVtA+AWDWdA0M6/5nT+j7z5xQdzj6iuc8Rkqc8ddOWsCrK1cU6k2ri3TVykJlBP2zVywAwFUIxQBm3cBQTL/d0yRjpIrcVFXkpaooI6hoIqHewZh6I1HVdoT1yMEWPXKgRW19Q8pO9evb7ztfW5bkOV0+AGAeIhQDSGqJhNXO2i597hd7dbJjQP/6tnW6dcMCp8sCAMwzbLQDkNQ8HqMLFuXq5//rYm1amKtP/PQl3b3tiJLhP9wBAPOfz+kCAGCsrFS/7v/gZn3uF3t197aj+tOhVpXnpio/LaC89BStKsnUxZV5Sg3w1xcAYPrwbxUASSfg8+hrb1+nqpIM/XF/iw429qq9f0i9kdjp5y9akqc3rCjQW89fwMY8AMCU0VMMYM6IROPacaJLjx1u1WOHWnW8fUCVhem67/ZNWpiX5nR5AIA5gI12AOadZ6rb9Vc/flFG0n/dtkmbF+eefu5kx4COtvTr0mX5zEAGAJxGKAYwL51oH9AH739BdZ1hff76KoWH43p4b5P2N/ZKknLTAnrfhRV630ULVZgRdLhaAIDTCMUA5q2ecFR3/fhFPVXdLknaWJGt69eWaElBmn68vU6PHmqR3+PRJZV5ygj6FfR7lOLzamVJht66cQEryQDgIoRiAPNaNJ7Q09XtWl6UodLs0Cueq2kf0PefrtH2mk4NxRKKROMKD8fVMxhVfnqK7rh0sd63pYINewDgAoRiABjDWqvtNZ36j8eq9eTRdmUEfbr9okX680sWKS89xenyAAAzhFAMAGext75H33q8Wr/f36wUn0fvuqBCd16+5FUrzgCAuY9QDACvo7q1X9/+n2P61a4GWUklWUFlp/qVkxpQeopPfZGYugeH1TUQVcJanb8wRxcvzdfFS/O0MC9VxhinvwUAwOsgFAPAOWroHtQD22vV0D2o7vCwugej6o/ElB70KSc1oOxUv2Jxq+01HWrpHZIkrS/P1g/v2ExfMgAkubOFYk60A4AzlGWH9DdvXvG6r7PWqqZ9QI8dbtM/P3xQH3lgl77z/k3yeT2zUCUAYDrxNzcATJIxRksK0nXHpYv1pVvW6PHDbfqnhw86XRYAYBJYKQaAafCeCytU3dqv7z5do8rCdL33woVq6Y3ox9tr9d876pSa4tOllfm6tDJfFy7Jpc0CAJIMoRgApskXbqjS8fZ+/f3W/XrsUKseP9ymuLW6fFmBrKSfvFCr7z9zQj6P0U3rS3XXGypVWZjudNkAALHRDgCmVV8kqrd/+1k1dg/qnReU631bFmphXpokKRKN68XaLv1xf4t++kKdIrG4blhbor++qlIrizMdrhwA3IHpEwAwS4ZicUlSiu/sx0e39w/pO0/W6P979oQGhuMqzQpqw8IcbSjPVlVJpgI+jzzGyOsx8nuN0gI+paZ4lRbwnX7OY8QYOACYIEIxACShroFh/Wp3g3ae7NKu2m41dA9O6P0eI5VkhbSyOEMrSzK0sjhTV64ooGcZAM6CUAwAc0BLb0THWvsVt1bxhFXCWg3HEgoPx0d/xTQcSyhhpXjCKpZIqLZzUIeaenW8fUDxhFVqwKu3bCjT+y5cqFWltGUAwFjMKQaAOaAoM6iizOCk3jsUi2tfQ49+8nydfr6zXj/eXqtNC3P0+RuqtLEi53Xf3z8UU4rPIz9zlgG4ECvFADAPdYeH9eDOen3nyRq19EX0rgsq9JlrVyg7NfCK18UTVk8cbdPPXqjTtoMtKsoM6pvv3qAN5xCiAWAuon0CAFyofyimux85ou89c0LZIb/uuGyxEgmrjoFhdQ4M6/maTjX1RJSbFtDN60v1yIEWtfRG9OlrV+hDly6Rx8NGPgDzC6EYAFzsQGOvvvCrvdpV2y1JSk/xKTctoGWF6Xrb+Qt0dVWRAj6PesJRfebne/T7/c26ckWBtizJU03bgI6396uxO6LVpZm6ckWhrlxRoNLskMPfFQBMHKEYAFzOWqu2/iFlBv0K+s8+Ls5aqx8+d1L/+JuDGo4nlJ8e0JL8dBVlBfXiya7TEzKqSjL1qTcu1zWril7x/s6BYX1j2xE1dEd0w7pivWlVsdJS2MICIDkQigEAE9IdHpYxRlmhl8e7WWtV3dqvxw+36Scv1OpY24CuWlmoL960SgtyUvXj52v1tT8cVv9QTIUZKWrqiSjo9+iNq4p1/ZpiXbIsX5mMiwPgIEIxAGBaDccS+v4zNfrGtqOKJqwqclNV3dqvLUty9aVb1qiyIF07a7u0dXeDfrunSV3hqHweo40Lc3TligJdsbxAq0oyX3EAyf7GHt3/zAk9erBVkWhc0YRVLJ5QWU5I33z3Rp1Xnu3gdwxgPiAUAwBmRHNPRP/8u4M61NSnu66q1E3rSl510l40ntCu2m49frhVjx9u04GmXklSYUaKrlheoLULsvSbl5r0/IlOhfxeXbumWLlpAfm8Rj6P0dbdjWrtHdKXb12jd2wqd+LbBDBPEIoBAEmjtTei/znSpsePtOnJI23qjcS0ICek2y9apHdsKldW6itbLLoGhvWRB3bpqep2vf+ihfr89VVq6xtSXVdY9V2DCng9KssJqSw7pKLMoFr7Ijrc3KcjLX3q6B/WBy5ZpJIsNgYCIBQDAJJULJ7QiY6wFuenyfsaI+Bi8YT+9Q+Hdc8Tx1/z84yRxv6rzWOk8txUPfDhLa+amNE5MKzmnggn/wEuQigGAMwLjxxo0Z76bi3ICWlBTqoW5IQUjSdU3zWo+q5BNfdEVJiZohVFGVpelKGajgHdft/zyk7z64EPb9GCnFRZa/WLFxv0j789oO5wVDevL9UXbqia9GmCAOYOQjEAwLVequvWbfdtV2bIr397+3r9x+PH9MSRNm2syNbmxXn67tM18nuMPnbNMl1TVaS9DT3aU9+jvQ09Go4llJ7iU3qKTxlBny5dlq83ry5+zbF2AJIXoRgA4Gp763v0vvu2q2cwqtSAV59+8wrddtEieT1GJzsG9KVfH9Cjh1pPvz7F59Gq0kylp/g0MBRT/1BM7f0jJwFmBn26+bxSvXNThdYuyBr361lrFR6OKzXgfdXGQwDOIRQDAFzvQGOvfrajTh+6bLEW5KS+6vknj7apoWtQaxdkaXlRhvxezyueTySsnjveoZ/tqNPv9jVrKJbQLeeV6os3rVZuWuAVX+dzv9ijl+p75PcaZYUCyk3zq7IwXW9YUag3rCxUfnrKpL8Pay1BG5gkQjEAANOoZzCq7z5Vo289Xq3MoF//cMtqXVNVpLu3HdW9Tx5XTqpft21ZpKFYXF3hkRXm3XXdaukdkjHSeeXZunZ1sW4+r/ScJ2McaenTr3Y1aOvuRqX4PLrn/eersjBjhr9TYH4hFAMAMAMONffq0w/u0Z76HmUGfeqNxPTOTeX63PUrlZ0aeMVrrbXa39irPx1q1baDLdpT3yNjpC2L83Tj+hL5PR4190bU3BtRR/+QEmP+FV3fNaiDTb3yeowuW5avfQ29iiUS+t4HLtCGipxZ/q6BuYtQDADADInFE7rvqRo9erBVn3jjcl20NO+c3lfTPqCtuxv0q10NOtERPn09Ny2g/PSAvJ6X2zcygj5dv6ZYN64vVX56ik52DOi2+55XW9+QvvW+jXrDikJVt/bpod2N2nawVatKM/Xxa5aN2yYCuBmhGACAJGWt1bG2fqX4vCrMTFGK79wmW7T1DekD33teh5v7tKQgTUda+k+3Zuxv7JWsdNtFC3XXGyqVGvCqurVfR1v7VN85qPSgT7lpAeWkBlSYmaJFeWlM1IArEIoBAJiH+iJRffYXe9XaG9ENa0t0/doSFWYG1dA9qG9sO6IHd9bL5/UoFk+8oh3jTB4jLcpL07KidK0vz9YNa0u0MC/tNb92JBrXk0fbdUllnlIDvmn+zoCZQSgGAMCFjrb06Ufba5UZ8o8eaJKu8txUhYfj6hwYVnd4WE09ER1t6dORln4daenT8fYBSdLasizduK5EV1cVaWlB2umJF7F4Qg/urNc3Hj2qpp6I1pRl6jvvv0DFWa99+EkkGld7/5ACXo98Xo/8XqP0FB+TNDCrCMUAAOCc1HeF9bu9zfrNnka9VN8jScpJ9ev8hblaVZqpX7/UqJr2AZ1Xnq0b15Xo648cUXrQp/tuv0Bryl6e29w/FNMLNZ3aXtOp52s6tLehR9H4K3PHssJ0/a8rlurm80pfNQIPmAmEYgAAMGF1nWE9e6xDL5zo1I6TXappH9CKogz9zZtX6JqqQhljdKi5V3d8f4c6B4b1tzdWqWtgWE8cbdeLJ7sUS1j5vUZry7K0eXGeFuenKpawisYSGowmtHV3gw4196ksO6Q7L1+iSyrzVJgZVMboCvJQLK4T7WEdbe1TdziqN68uVkHG5Gc8A4RiAAAwZb2RqNIDPnk8r2x5aO2L6M4f7NTuum5J0urSTF2+vECXVuZrY0WOQoHxN/FZa/XY4VZ967Fj2nGy6/T1kN+r7FS/WvuGFB/TDB3wenTDuhLdfvEirSvL0rG2fj1/olM7TnSpOzysnLSA8tICyk1L0eL8VK0py1JZduisLRp1nWH95/8c0976Hn38mmW6uqpoqrcISY5QDAAAZlQkGtfOk11aUZwxqRP79jX06Hj7gFp7I2rpjahjYFhl2SFVFqarsjBdXo/RA9tr9eDOeg0Mx5UW8GpgOC5Jyk9PUXFWiroGouoYGFIkmjj9uTmpfq0py9KKogxVFqZraWG6UgNefe/pE/rlrgZ5jVFxVlC1nWG9c1O5/vbGKmUE/ae/pxdOdCrF59UFi3Im3P88FIurqTuihXmp9E4nCUIxAACYF/oiUf18Z72OtvbrvPJsXbAo91Whc2AopqOt/dpb3629DT3a19CrY239Goq9HJZTfB6958IK/cXlS5WT5tc3th3Vt//nmEqzQ3rvhQv1wolOPXusQ4PRkeC9JD9N79pcrrduXKCskF8nOsI60tKnmvYBFWakqKokU5WF6Qp4Pdpxsku/3NWg3+5pVG8kpoKMFF2xvEBXrijQZcsKlBXyz/p9wwhCMQAAcLV4wqqxe1DVrf1q6Y3o6qqiV/Un7zzZqU/+7CWd7AirIjdVV64YCbLd4ageeL5WL5zokt9rZGQ0HE+86mt4jJQZ8qs7HFXI79W1a4q1sSJb22s69eTRdvUMRuX3Gl2xvEA3rS/VNVVFSg14daIjrN11XdpT36OA16MFOSEtyElVYWaK6jrD2t/Yq/2NvWroGtTmxbm6bk2xNi/OlY/NiRNGKAYAADgHQ7G4OvqHVZIVfFXLw9GWPj34Yr0kaXlhhlYUZ2hxfppaeiM63Nyng819auga1GXL8vXGVUVKS3l5fnMsntBL9d36w/4W/fqlRjX1RBT0exT0e9Udjkoa6aWOW6vh2CsDt9djVFmQrsLMFL1wolORaEK5aQG9aVWRbt1QpgsW5b6iz7tzYFiPHGhWVziqvLSA8jNSlJcWUH8kpsaeiBq7B9XRP6TzF+XqmqpCV82ZJhQDAAAkiUTC6sXaLv1mT5Mi0bjOK8/W+vJsLS/KkJHUPjCk+q5BNfdEVJod0srijNMnDoaHY3riSJt+t69Z2w60aGA4rgU5If3ZhjIVZQX1u73NevZ4xys2KI4n5PdqMBpXasCrN60q0g3rSrWqNFMlmcFXbaScTwjFAAAA80x4OKY/7m/Rz1+s19PV7UpYaVFeqq5fW6Ib1o2cStjRP6T2/mF19A8pPehTaVZIxVlBBbwePX+iU1t3N+rhvU3qGRxZrQ76PVqUl6bS7JAkKWGtElbKTw/oqpWFunx5gTKDL/dEDwzFdKytX0G/VyVZwdObFF9LXyR6Tq+bCYRiAACAeaylN6KewaiWFaZPeNLFcCyhF2u7dKytXzVtAzrePqCW3og8xshjJGOMTnYMqCs80hN94eI8pQa8OtzSp5Md4Vd8VkaKT2U5IV2wKFdXrijQRUtHjgE/dSjMw/uadLIjrO2fv9qRA1sIxQAAAJi0+GjLx7aDLfrTwVbFrVVVcaZWFI8cHz4ct2rqHlRj96BOdIT1fE2nBqNxBXweVeSmqrq1X5K0pixT160p0Z9fssiRXmZCMQAAAGbNqRnPjx9u05GWPl1Sma/r1hRrYV6ao3WdLRS7Z6shAAAAZk3Q79Vly0bmMs8FDLcDAACA6xGKAQAA4HqEYgAAALgeoRgAAACuRygGAACA6xGKAQAA4HqEYgAAALjepEOxMabcGPOYMeaAMWa/MeZjo9dzjTGPGGOOjv6eM33lAgAAANNvKivFMUmfstaukrRF0l3GmFWSPivpUWvtMkmPjj4GAAAAktakQ7G1tsla++Lon/skHZRUJukWSfePvux+SW+ZapEAAADATJqWnmJjzCJJGyRtl1RkrW0afapZUtF0fA0AAABgpkw5FBtj0iX9XNLHrbW9Y5+z1lpJ9izvu9MYs8MYs6OtrW2qZQAAAACTNqVQbIzxayQQ/8ha+4vRyy3GmJLR50sktY73XmvtPdbaTdbaTQUFBVMpAwAAAJiSqUyfMJLuk3TQWvt/xzz1kKTbR/98u6Stky8PAAAAmHm+Kbz3Ekm3SdprjNk9eu3zkv5F0s+MMXdIOinpHVMrEQAAAJhZkw7F1tqnJJmzPH31ZD8XAAAAmG2caAcAAADXIxQDAADA9czI1DSHizCmTSP9x07Il9Tu0Neei7hfE8c9mxju18RxzyaG+zVx3LOJ4X5N3Gzes4XW2leNPkuKUOwkY8wOa+0mp+uYK7hfE8c9mxju18RxzyaG+zVx3LOJ4X5NXDLcM9onAAAA4HqEYgAAALgeoVi6x+kC5hju18RxzyaG+zVx3LOJ4X5NHPdsYrhfE+f4PXN9TzEAAADASjEAAABcz7Wh2BhzrTHmsDGm2hjzWafrSUbGmHJjzGPGmAPGmP3GmI+NXs81xjxijDk6+nuO07UmE2OM1xizyxjzm9HHi40x20d/1n5qjAk4XWMyMcZkG2MeNMYcMsYcNMZcxM/Y2RljPjH6z+M+Y8wDxpggP2OvZIz5rjGm1Rizb8y1cX+mzIh/H713e4wxG52r3DlnuWf/Z/Sfyz3GmF8aY7LHPPe50Xt22BjzZmeqds5492vMc58yxlhjTP7oY37GdPZ7Zoz5yOjP2X5jzL+OuT7rP2OuDMXGGK+k/5B0naRVkt5tjFnlbFVJKSbpU9baVZK2SLpr9D59VtKj1tplkh4dfYyXfUzSwTGPvyrp69baSkldku5wpKrk9Q1Jv7fWrpS0Xv9/e/caYsdZx3H8+yObhKSF3oKxZitbNfVFai+hSvGGjSJtLV3BQiMBqxaEvPDypmoNCIIvRERLvVS0pYkaLFpjDYLSmpYqaBttyKX1mrYh3bAxCZJ4JY3154vniZ2e3bObgMnMZn4fGHbmmTmH5/z3f878z8wzc0rskmPTkLQM+Ahwle1LgXnAapJjg9YD1w60Dcup64DldfoQcNdp6mPXrGdqzB4CLrV9GfBH4HaAuh9YDayoj/la3a/2yXqmxgtJFwHvBPY2mpNjxXoGYibpGmAcuNz2CuALtb2VHOtlUQy8Adht+xnbzwP3Uf4p0WB70va2Ov83SrGyjBKrDXWzDcC72+lh90gaBd4F3F2XBawC7q+bJF4Nks4B3grcA2D7eduHSY7NZARYJGkEWAxMkhx7Cds/B/4y0Dwsp8aBb7l4DDhX0oWnp6fdMV3MbD9o+9918TFgtM6PA/fZPmr7WWA3Zb/aG0NyDOBLwMeB5gVbyTGGxmwt8DnbR+s2B2p7KznW16J4GfBcY3mitsUQksaAK4HHgaW2J+uq/cDSlrrVRXdQPhD/U5cvAA43dizJtZe6GDgI3FuHnNwt6SySY9OyvY9yJGUvpRg+AjxBcuxEDMup7A9OzAeBn9T5xGwaksaBfbZ3DKxKvIa7BHhLHf71qKTX1/ZWYtbXojhOgqSzgR8AH7P91+Y6l9uX5BYmgKQbgAO2n2i7L3PICLASuMv2lcA/GBgqkRx7UR0HO075MvEK4CymOYUbM0tOnRxJ6yjD6Ta23ZeukrQY+BTw6bb7MseMAOdThmjeBnyvnmFtRV+L4n3ARY3l0doWAyTNpxTEG21vqs1/Pn7qp/49MOzxPfMm4EZJeyhDclZRxsueW091Q3Jt0AQwYfvxunw/pUhOjk3vHcCztg/aPgZsouRdcmx2w3Iq+4MZSHo/cAOwxi/ewzUxm+rVlC+rO+o+YBTYJunlJF4zmQA21aElWylnWZfQUsz6WhT/Glher9heQBnMvbnlPnVO/bZ2D/A7219srNoM3FLnbwF+dLr71kW2b7c9anuMklMP214DPALcVDdLvBps7week/Ta2vR24Lckx4bZC1wtaXF9fx6PV3JsdsNyajPwvnqHgKuBI41hFr0m6VrKcLAbbf+zsWozsFrSQkkXUy4g29pGH7vC9i7bL7M9VvcBE8DK+hmXHBvuAeAaAEmXAAuAQ7SVY7Z7OQHXU66mfRpY13Z/ujgBb6acYtwJbK/T9ZRxsluAPwE/A85vu69dm4C3AT+u86+qb+bdwPeBhW33r0sTcAXwm5pnDwDnJcdmjNdngN8DTwLfBhYmx6bE6LuUMdfHKMXJrcNyChDlbkRPA7sod/Zo/TV0JGa7KeM6j3/+f72x/boasz8A17Xd/y7Ea2D9HmBJcmzWHFsAfKd+nm0DVrWZY/lFu4iIiIjovb4On4iIiIiI+J8UxRERERHReymKIyIiIqL3UhRHRERERO+lKI6IiIiI3ktRHBHRMkkvSNremD45+6NO+LnHJD35/3q+iIgz1cjsm0RExCn2L9tXtN2JiIg+y5HiiIiOkrRH0ucl7ZK0VdJravuYpIcl7ZS0RdIra/tSST+UtKNOb6xPNU/SNyU9JelBSYtae1ERER2Vojgion2LBoZP3NxYd8T264CvAHfUti8DG2xfBmwE7qztdwKP2r4cWAk8VduXA1+1vQI4DLznFL+eiIg5J79oFxHRMkl/t332NO17KD97+oyk+cB+2xdIOgRcaPtYbZ+0vUTSQWDU9tHGc4wBD9leXpc/Acy3/dlT/8oiIuaOHCmOiOg2D5k/GUcb8y+Q60kiIqZIURwR0W03N/7+qs7/Elhd59cAv6jzW4C1AJLmSTrndHUyImKuy9GCiIj2LZK0vbH8U9vHb8t2nqSdlKO9761tHwbulXQbcBD4QG3/KPANSbdSjgivBSZPee8jIs4AGVMcEdFRdUzxVbYPtd2XiIgzXYZPRERERETv5UhxRERERPRejhRHRERERO+lKI6IiIiI3ktRHBERERG9l6I4IiIiInovRXFERERE9F6K4oiIiIjovf8CyBMWCmhaCugAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "tags": [], "needs_background": "light" } } ] }, { "cell_type": "markdown", "metadata": { "id": "wBD0wM3JVXol" }, "source": [ "### Appendix" ] }, { "cell_type": "markdown", "metadata": { "id": "Iz0QI1P7VaDP" }, "source": [ "#### References\n", "1. [https://medium.com/@yusufnoor_88274/implementing-neural-graph-collaborative-filtering-in-pytorch-4d021dff25f3](https://medium.com/@yusufnoor_88274/implementing-neural-graph-collaborative-filtering-in-pytorch-4d021dff25f3)\n", "2. [https://github.com/xiangwang1223/neural_graph_collaborative_filtering](https://github.com/xiangwang1223/neural_graph_collaborative_filtering)\n", "3. [https://arxiv.org/pdf/1905.08108.pdf](https://arxiv.org/pdf/1905.08108.pdf)\n", "4. [https://github.com/metahexane/ngcf_pytorch_g61](https://github.com/metahexane/ngcf_pytorch_g61)" ] }, { "cell_type": "markdown", "metadata": { "id": "_JywmGguVbNU" }, "source": [ "#### Next\n", "\n", "Try out this notebook on the following datasets:\n", "\n", "![](https://github.com/recohut/reco-static/raw/master/media/images/120222_data.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "MppKYNXJVswT" }, "source": [ "Compare out the performance with these baselines:\n", "\n", "1. MF: This is matrix factorization optimized by the Bayesian\n", "personalized ranking (BPR) loss, which exploits the user-item\n", "direct interactions only as the target value of interaction function.\n", "2. NeuMF: The method is a state-of-the-art neural CF model\n", "which uses multiple hidden layers above the element-wise and\n", "concatenation of user and item embeddings to capture their nonlinear feature interactions. Especially, we employ two-layered\n", "plain architecture, where the dimension of each hidden layer\n", "keeps the same.\n", "3. CMN: It is a state-of-the-art memory-based model, where\n", "the user representation attentively combines the memory slots\n", "of neighboring users via the memory layers. Note that the firstorder connections are used to find similar users who interacted\n", "with the same items.\n", "4. HOP-Rec: This is a state-of-the-art graph-based model,\n", "where the high-order neighbors derived from random walks\n", "are exploited to enrich the user-item interaction data.\n", "5. PinSage: PinSage is designed to employ GraphSAGE\n", "on item-item graph. In this work, we apply it on user-item interaction graph. Especially, we employ two graph convolution\n", "layers, and the hidden dimension is set equal\n", "to the embedding size.\n", "6. GC-MC: This model adopts GCN encoder to generate\n", "the representations for users and items, where only the first-order\n", "neighbors are considered. Hence one graph convolution layer,\n", "where the hidden dimension is set as the embedding size, is used." ] }, { "cell_type": "markdown", "metadata": { "id": "e7RRc2UQBuc9" }, "source": [ "## A simple recommender with tensorflow\n", "> A tutorial on how to build a simple deep learning based movie recommender using tensorflow library." ] }, { "cell_type": "code", "metadata": { "id": "hLtJPt_5idKN" }, "source": [ "import numpy as np\n", "import pandas as pd\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "from tensorflow.keras import layers\n", "from tensorflow.keras import models\n", "\n", "tf.random.set_seed(343)" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "id": "DNLlAwKUihC1" }, "source": [ "# Clean up the logdir if it exists\n", "import shutil\n", "shutil.rmtree('logs', ignore_errors=True)\n", "\n", "# Load TensorBoard extension for notebooks\n", "%load_ext tensorboard" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "8IRTF0EVjQuX", "outputId": "932eaa43-725c-4fb8-e9d4-dca92ced4cf0" }, "source": [ "movielens_ratings_file = 'https://github.com/sparsh-ai/reco-data/blob/master/MovieLens_100K_ratings.csv?raw=true'\n", "df_raw = pd.read_csv(movielens_ratings_file)\n", "df_raw.head()" ], "execution_count": null, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
UserIdMovieIdRatingTimestamp
01962423.0881250949
11863023.0891717742
2223771.0878887116
3244512.0880606923
41663461.0886397596
\n", "
" ], "text/plain": [ " UserId MovieId Rating Timestamp\n", "0 196 242 3.0 881250949\n", "1 186 302 3.0 891717742\n", "2 22 377 1.0 878887116\n", "3 244 51 2.0 880606923\n", "4 166 346 1.0 886397596" ] }, "execution_count": 22, "metadata": { "tags": [] }, "output_type": "execute_result" } ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "El1C8OwWjhxk", "outputId": "f8ed06e7-8554-45f2-982d-987f153a5cc7" }, "source": [ "df = df_raw.copy()\n", "df.columns = ['userId', 'movieId', 'rating', 'timestamp']\n", "user_ids = df['userId'].unique()\n", "user_encoding = {x: i for i, x in enumerate(user_ids)} # {user_id: index}\n", "movie_ids = df['movieId'].unique()\n", "movie_encoding = {x: i for i, x in enumerate(movie_ids)} # {movie_id: index}\n", "\n", "df['user'] = df['userId'].map(user_encoding) # Map from IDs to indices\n", "df['movie'] = df['movieId'].map(movie_encoding)\n", "\n", "n_users = len(user_ids)\n", "n_movies = len(movie_ids)\n", "\n", "min_rating = min(df['rating'])\n", "max_rating = max(df['rating'])\n", "\n", "print(f'Number of users: {n_users}\\nNumber of movies: {n_movies}\\nMin rating: {min_rating}\\nMax rating: {max_rating}')\n", "\n", "# Shuffle the data\n", "df = df.sample(frac=1, random_state=42)" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of users: 943\n", "Number of movies: 1682\n", "Min rating: 1.0\n", "Max rating: 5.0\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "1W5V8T-C8Gpv" }, "source": [ "### Scheme of the model\n", "\n", "![](https://github.com/recohut/reco-static/raw/master/media/images/120222_scheme.png)" ] }, { "cell_type": "code", "metadata": { "id": "G-iv9rijkaBf" }, "source": [ "class MatrixFactorization(models.Model):\n", " def __init__(self, n_users, n_movies, n_factors, **kwargs):\n", " super(MatrixFactorization, self).__init__(**kwargs)\n", " self.n_users = n_users\n", " self.n_movies = n_movies\n", " self.n_factors = n_factors\n", " \n", " # We specify the size of the matrix,\n", " # the initializer (truncated normal distribution)\n", " # and the regularization type and strength (L2 with lambda = 1e-6)\n", " self.user_emb = layers.Embedding(n_users, \n", " n_factors, \n", " embeddings_initializer='he_normal',\n", " embeddings_regularizer=keras.regularizers.l2(1e-6),\n", " name='user_embedding')\n", " self.movie_emb = layers.Embedding(n_movies, \n", " n_factors, \n", " embeddings_initializer='he_normal',\n", " embeddings_regularizer=keras.regularizers.l2(1e-6),\n", " name='movie_embedding')\n", " \n", " # Embedding returns a 3D tensor with one dimension = 1, so we reshape it to a 2D tensor\n", " self.reshape = layers.Reshape((self.n_factors,))\n", " \n", " # Dot product of the latent vectors\n", " self.dot = layers.Dot(axes=1)\n", "\n", " def call(self, inputs):\n", " # Two inputs\n", " user, movie = inputs\n", " u = self.user_emb(user)\n", " u = self.reshape(u)\n", " \n", " m = self.movie_emb(movie)\n", " m = self.reshape(m)\n", " \n", " return self.dot([u, m])\n", "\n", "n_factors = 50\n", "model = MatrixFactorization(n_users, n_movies, n_factors)\n", "model.compile(\n", " optimizer=keras.optimizers.Adam(learning_rate=0.001),\n", " loss=keras.losses.MeanSquaredError()\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Bac1w7u49Ddx", "outputId": "bb733033-9aba-446b-a56d-f971897221d0" }, "source": [ "try:\n", " model.summary()\n", "except ValueError as e:\n", " print(e, type(e))" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build. \n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "o-JSFnJA-1dz" }, "source": [ "This is why building models via subclassing is a bit annoying - you can run into errors such as this. We'll fix it by calling the model with some fake data so it knows the shapes of the inputs." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7wkIhqmO92Ca", "outputId": "6825be87-3d5e-4d25-e276-5043ff3a3bb9" }, "source": [ "_ = model([np.array([1, 2, 3]), np.array([2, 88, 5])])\n", "model.summary()" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"matrix_factorization_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "user_embedding (Embedding) multiple 47150 \n", "_________________________________________________________________\n", "movie_embedding (Embedding) multiple 84100 \n", "_________________________________________________________________\n", "reshape_1 (Reshape) multiple 0 \n", "_________________________________________________________________\n", "dot_1 (Dot) multiple 0 \n", "=================================================================\n", "Total params: 131,250\n", "Trainable params: 131,250\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "9Nxdrz7b_HOq" }, "source": [ "We're going to expand our toolbox by introducing callbacks. Callbacks can be used to monitor our training progress, decay the learning rate, periodically save the weights or even stop early in case of detected overfitting. In Keras, they are really easy to use: you just create a list of desired callbacks and pass it to the model.fit method. It's also really easy to define your own by subclassing the Callback class. You can also specify when they will be triggered - the default is at the end of every epoch.\n", "\n", "We'll use two: an early stopping callback which will monitor our loss and stop the training early if needed and TensorBoard, a utility for visualizing models, monitoring the training progress and much more." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6N_Y7u5o-QpY", "outputId": "b4f299bc-25e2-4e38-b349-dc07184fb488" }, "source": [ "callbacks = [\n", " keras.callbacks.EarlyStopping(\n", " # Stop training when `val_loss` is no longer improving\n", " monitor='val_loss',\n", " # \"no longer improving\" being defined as \"no better than 1e-2 less\"\n", " min_delta=1e-2,\n", " # \"no longer improving\" being further defined as \"for at least 2 epochs\"\n", " patience=2,\n", " verbose=1,\n", " ),\n", " keras.callbacks.TensorBoard(log_dir='logs')\n", "]\n", "\n", "history = model.fit(\n", " x=(df['user'].values, df['movie'].values), # The model has two inputs!\n", " y=df['rating'],\n", " batch_size=128,\n", " epochs=20,\n", " verbose=1,\n", " validation_split=0.1,\n", " callbacks=callbacks\n", ")" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/20\n", "704/704 [==============================] - 3s 3ms/step - loss: 12.0905 - val_loss: 5.5121\n", "Epoch 2/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 2.1751 - val_loss: 1.2149\n", "Epoch 3/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 1.0271 - val_loss: 0.9839\n", "Epoch 4/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.9003 - val_loss: 0.9266\n", "Epoch 5/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.8470 - val_loss: 0.8996\n", "Epoch 6/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.8046 - val_loss: 0.8786\n", "Epoch 7/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.7667 - val_loss: 0.8680\n", "Epoch 8/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.7329 - val_loss: 0.8618\n", "Epoch 9/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.6999 - val_loss: 0.8558\n", "Epoch 10/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.6688 - val_loss: 0.8558\n", "Epoch 11/20\n", "704/704 [==============================] - 2s 3ms/step - loss: 0.6381 - val_loss: 0.8560\n", "Epoch 00011: early stopping\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "5QUGLmtw_eWA" }, "source": [ "We see that we stopped early because the validation loss was not improving. Now, we'll open TensorBoard (it's a separate program called via command-line) to read the written logs and visualize the loss over all epochs. We will also look at how to visualize the model as a computational graph." ] }, { "cell_type": "code", "metadata": { "id": "_J-v9Hua_SV8" }, "source": [ "# Run TensorBoard and specify the log dir\n", "%tensorboard --logdir logs" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Ldq0DwgI_lWC" }, "source": [ "We've seen how easy it is to implement a recommender system with Keras and use a few utilities to make it easier to experiment. Note that this model is still quite basic and we could easily improve it: we could try adding a bias for each user and movie or adding non-linearity by using a sigmoid function and then rescaling the output. It could also be extended to use other features of a user or movie." ] }, { "cell_type": "markdown", "metadata": { "id": "-dpCn5hm_nUM" }, "source": [ "Next, we'll try a bigger, more state-of-the-art model: a deep autoencoder." ] }, { "cell_type": "markdown", "metadata": { "id": "zTGdZ0b4_4rl" }, "source": [ "We'll apply a more advanced algorithm to the same dataset as before, taking a different approach. We'll use a deep autoencoder network, which attempts to reconstruct its input and with that gives us ratings for unseen user / movie pairs." ] }, { "cell_type": "markdown", "metadata": { "id": "yIf926SkCOEp" }, "source": [ "![](https://github.com/recohut/reco-static/raw/master/media/images/120222_algo.png)" ] }, { "cell_type": "markdown", "metadata": { "id": "rSdY5NKQAYfI" }, "source": [ "Preprocessing will be a bit different due to the difference in our model. Our autoencoder will take a vector of all ratings for a movie and attempt to reconstruct it. However, our input vector will have a lot of zeroes due to the sparsity of our data. We'll modify our loss so our model won't predict zeroes for those combinations - it will actually predict unseen ratings.\n", "\n", "To facilitate this, we'll use the sparse tensor that TF supports. Note: to make training easier, we'll transform it to dense form, which would not work in larger datasets - we would have to preprocess the data in a different way or stream it into the model." ] }, { "cell_type": "markdown", "metadata": { "id": "HBcKm55rCVkk" }, "source": [ "### Sparse representation and autoencoder reconstruction\n", "\n", "![](https://github.com/recohut/reco-static/raw/master/media/images/120222_ae.png)" ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 204 }, "id": "OWybs9LyE8bB", "outputId": "1e108c11-a007-4942-bf0d-50409e4bbb1d" }, "source": [ "df_raw.head()" ], "execution_count": null, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
userIdmovieIdratingtimestamp
01962423.0881250949
11863023.0891717742
2223771.0878887116
3244512.0880606923
41663461.0886397596
\n", "
" ], "text/plain": [ " userId movieId rating timestamp\n", "0 196 242 3.0 881250949\n", "1 186 302 3.0 891717742\n", "2 22 377 1.0 878887116\n", "3 244 51 2.0 880606923\n", "4 166 346 1.0 886397596" ] }, "execution_count": 21, "metadata": { "tags": [] }, "output_type": "execute_result" } ] }, { "cell_type": "code", "metadata": { "id": "M9jASOsh_gvU" }, "source": [ "# Create a sparse tensor: at each user, movie location, we have a value, the rest is 0\n", "sparse_x = tf.sparse.SparseTensor(indices=df[['movie', 'user']].values, values=df['rating'], dense_shape=(n_movies, n_users))\n", "\n", "# Transform it to dense form and to float32 (good enough precision)\n", "dense_x = tf.cast(tf.sparse.to_dense(tf.sparse.reorder(sparse_x)), tf.float32)\n", "\n", "# Shuffle the data\n", "x = tf.random.shuffle(dense_x, seed=42)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "1j2-lFANEp8t" }, "source": [ "Now, let's create the model. We'll have to specify the input shape. Because we have 9724 movies and only 610 users, we'll prefer to predict ratings for movies instead of users - this way, our dataset is larger." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9s4qXdbuEpuX", "outputId": "45ac7925-b8f3-44dd-8e35-53fa7192b538" }, "source": [ "class Encoder(layers.Layer):\n", " def __init__(self, **kwargs):\n", " super(Encoder, self).__init__(**kwargs)\n", " self.dense1 = layers.Dense(28, activation='selu', kernel_initializer='glorot_uniform')\n", " self.dense2 = layers.Dense(56, activation='selu', kernel_initializer='glorot_uniform')\n", " self.dense3 = layers.Dense(56, activation='selu', kernel_initializer='glorot_uniform')\n", " self.dropout = layers.Dropout(0.3)\n", " \n", " def call(self, x):\n", " d1 = self.dense1(x)\n", " d2 = self.dense2(d1)\n", " d3 = self.dense3(d2)\n", " return self.dropout(d3)\n", " \n", " \n", "class Decoder(layers.Layer):\n", " def __init__(self, n, **kwargs):\n", " super(Decoder, self).__init__(**kwargs)\n", " self.dense1 = layers.Dense(56, activation='selu', kernel_initializer='glorot_uniform')\n", " self.dense2 = layers.Dense(28, activation='selu', kernel_initializer='glorot_uniform')\n", " self.dense3 = layers.Dense(n, activation='selu', kernel_initializer='glorot_uniform')\n", "\n", " def call(self, x):\n", " d1 = self.dense1(x)\n", " d2 = self.dense2(d1)\n", " return self.dense3(d2)\n", "\n", "n = n_users\n", "inputs = layers.Input(shape=(n,))\n", "\n", "encoder = Encoder()\n", "decoder = Decoder(n)\n", "\n", "enc1 = encoder(inputs)\n", "dec1 = decoder(enc1)\n", "enc2 = encoder(dec1)\n", "dec2 = decoder(enc2)\n", "\n", "model = models.Model(inputs=inputs, outputs=dec2, name='DeepAutoencoder')\n", "model.summary()" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"DeepAutoencoder\"\n", "__________________________________________________________________________________________________\n", "Layer (type) Output Shape Param # Connected to \n", "==================================================================================================\n", "input_1 (InputLayer) [(None, 943)] 0 \n", "__________________________________________________________________________________________________\n", "encoder (Encoder) (None, 56) 31248 input_1[0][0] \n", " decoder[0][0] \n", "__________________________________________________________________________________________________\n", "decoder (Decoder) (None, 943) 32135 encoder[0][0] \n", " encoder[1][0] \n", "==================================================================================================\n", "Total params: 63,383\n", "Trainable params: 63,383\n", "Non-trainable params: 0\n", "__________________________________________________________________________________________________\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "aqXWA_TQGMDa" }, "source": [ "Because our inputs are sparse, we'll need to create a modified mean squared error function. We have to look at which ratings are zero in the ground truth and remove them from our loss calculation (if we didn't, our model would quickly learn to predict zeros almost everywhere). We'll use masking - first get a boolean mask of non-zero values and then extract them from the result." ] }, { "cell_type": "code", "metadata": { "id": "G7AyGH8IFXAj" }, "source": [ "def masked_mse(y_true, y_pred):\n", " mask = tf.not_equal(y_true, 0)\n", " se = tf.boolean_mask(tf.square(y_true - y_pred), mask)\n", " return tf.reduce_mean(se)\n", "\n", "model.compile(\n", " loss=masked_mse,\n", " optimizer=keras.optimizers.Adam()\n", ")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "6O-Hqm_FGTmz" }, "source": [ "The model training will be similar as before - we'll use early stopping and TensorBoard. Our batch size will be smaller due to the lower number of examples. Note that we are passing the same array for both x and y, because the autoencoder reconstructs its input." ] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OHoZ3IuJGSrL", "outputId": "7923e0b0-7bc6-42ba-b3e6-c2bfe025291d" }, "source": [ "callbacks = [\n", " keras.callbacks.EarlyStopping(\n", " monitor='val_loss',\n", " min_delta=1e-2,\n", " patience=5,\n", " verbose=1,\n", " ),\n", " keras.callbacks.TensorBoard(log_dir='logs')\n", "]\n", "\n", "model.fit(\n", " x, \n", " x, \n", " batch_size=16, \n", " epochs=100, \n", " validation_split=0.1,\n", " callbacks=callbacks\n", ")" ], "execution_count": null, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:Model failed to serialize as JSON. Ignoring... Layer Decoder has arguments in `__init__` and therefore must override `get_config`.\n", "Epoch 1/100\n", "95/95 [==============================] - 2s 7ms/step - loss: 4.6136 - val_loss: 1.1074\n", "Epoch 2/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 1.1491 - val_loss: 1.0088\n", "Epoch 3/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 1.0577 - val_loss: 0.9768\n", "Epoch 4/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 1.0257 - val_loss: 0.9758\n", "Epoch 5/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.9971 - val_loss: 0.9774\n", "Epoch 6/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.9812 - val_loss: 0.9604\n", "Epoch 7/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.9598 - val_loss: 0.9275\n", "Epoch 8/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.9501 - val_loss: 0.9253\n", "Epoch 9/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.9177 - val_loss: 0.9159\n", "Epoch 10/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.9193 - val_loss: 0.9189\n", "Epoch 11/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.9016 - val_loss: 0.9040\n", "Epoch 12/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.9119 - val_loss: 0.9108\n", "Epoch 13/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.8917 - val_loss: 0.9192\n", "Epoch 14/100\n", "95/95 [==============================] - 0s 5ms/step - loss: 0.8855 - val_loss: 0.9166\n", "Epoch 15/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.8843 - val_loss: 0.9067\n", "Epoch 16/100\n", "95/95 [==============================] - 0s 4ms/step - loss: 0.8851 - val_loss: 0.9034\n", "Epoch 00016: early stopping\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 27, "metadata": { "tags": [] }, "output_type": "execute_result" } ] }, { "cell_type": "markdown", "metadata": { "id": "kkkhIjHhGhP5" }, "source": [ "Let's visualize our loss and the model itself with TensorBoard." ] }, { "cell_type": "code", "metadata": { "id": "MMVp_HbwGdGQ" }, "source": [ "%tensorboard --logdir logs" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "eSBFppW8Gkih" }, "source": [ "That's it! We've seen how to use TensorFlow to implement recommender systems in a few different ways. I hope this short introduction has been informative and has prepared you to use TF on new problems. Thank you for your attention!" ] } ] }