{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Doc2vec from scratch in PyTorch\n",
"===============================\n",
"\n",
"Here we are implementing this useful algorithm with a library we know and trust. With luck this will be more accessible than reading the papers but more in-depth than typical \"install gensim and just do what I say\" tutorials, and still easy to understand for anyone whose maths skills have atrophied to nothing (like me). This is all based on the great work by [Nejc Ilenic](https://github.com/inejc/paragraph-vectors) and reading the referenced papers and gensim's source.\n",
"\n",
"`doc2vec` descends from `word2vec`, the basic form of which is that it is a model trained to predict the missing word in a context. Given sentences like \"the cat ___ on the mat\" it should predict \"sat\", and in doing so learn a useful representation of words. We can then extract the internal weights and re-use them as \"word embeddings\", vectors giving each word a position in N-dimensional space that is hopefully close to similar words and an appropriate distance from related words. \n",
"\n",
"`doc2vec` or \"Paragraph vectors\" extends the `word2vec` idea by simply adding a document id to each context. This helps the network learn associations between contexts and produces vectors that position each paragraph (document) in space."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we need to load the data. We'll begin by overfitting on a tiny dataset just to check all the parts fit together."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>text</th>\n",
" <th>tokens</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
" <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
" <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
" <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
" <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" text \\\n",
"0 In the week before their departure to Arrakis, when all the final scurrying about had reached a ... \n",
"1 It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide... \n",
"2 The old woman was let in by the side door down the vaulted passage by Paul's room and she was al... \n",
"3 By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could... \n",
"\n",
" tokens \n",
"0 [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ... \n",
"1 [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se... \n",
"2 [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room... \n",
"3 [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak... "
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"import spacy\n",
"\n",
"nlp = spacy.load(\"en_core_web_sm\")\n",
"\n",
"pd.set_option(\"display.max_colwidth\", 100)\n",
"\n",
"example_df = pd.read_csv(\"data/example.csv\")\n",
"\n",
"def tokenize_text(df):\n",
" df[\"tokens\"] = df.text.str.lower().str.strip().apply(lambda x: [token.text.strip() for token in nlp(x) if token.text.isalnum()])\n",
" return df\n",
"\n",
"example_df = tokenize_text(example_df)\n",
"\n",
"example_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will need to construct a vocabulary so we can reference every word by an ID."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset comprises 4 documents and 106 unique words (over the limit of 1 occurrences)\n"
]
}
],
"source": [
"from collections import Counter\n",
"\n",
"class Vocab:\n",
" def __init__(self, all_tokens, min_count=2):\n",
" self.min_count = min_count\n",
" self.freqs = {t:n for t, n in Counter(all_tokens).items() if n >= min_count}\n",
" self.words = sorted(self.freqs.keys())\n",
" self.word2idx = {w: i for i, w in enumerate(self.words)}\n",
" \n",
"vocab = Vocab([tok for tokens in example_df.tokens for tok in tokens], min_count=1)\n",
"\n",
"print(f\"Dataset comprises {len(example_df)} documents and {len(vocab.words)} unique words (over the limit of {vocab.min_count} occurrences)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Words that appear extremely rarely can harm performance, so we add a simple mechanism to strip those out."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>text</th>\n",
" <th>tokens</th>\n",
" <th>length</th>\n",
" <th>clean_tokens</th>\n",
" <th>clean_length</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
" <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
" <td>32</td>\n",
" <td>[in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ...</td>\n",
" <td>32</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
" <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
" <td>39</td>\n",
" <td>[it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se...</td>\n",
" <td>39</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
" <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
" <td>34</td>\n",
" <td>[the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room...</td>\n",
" <td>34</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
" <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
" <td>53</td>\n",
" <td>[by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak...</td>\n",
" <td>53</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" text \\\n",
"0 In the week before their departure to Arrakis, when all the final scurrying about had reached a ... \n",
"1 It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide... \n",
"2 The old woman was let in by the side door down the vaulted passage by Paul's room and she was al... \n",
"3 By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could... \n",
"\n",
" tokens \\\n",
"0 [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ... \n",
"1 [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se... \n",
"2 [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room... \n",
"3 [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak... \n",
"\n",
" length \\\n",
"0 32 \n",
"1 39 \n",
"2 34 \n",
"3 53 \n",
"\n",
" clean_tokens \\\n",
"0 [in, the, week, before, their, departure, to, arrakis, when, all, the, final, scurrying, about, ... \n",
"1 [it, was, a, warm, night, at, castle, caladan, and, the, ancient, pile, of, stone, that, had, se... \n",
"2 [the, old, woman, was, let, in, by, the, side, door, down, the, vaulted, passage, by, paul, room... \n",
"3 [by, the, half, light, of, a, suspensor, lamp, dimmed, and, hanging, near, the, floor, the, awak... \n",
"\n",
" clean_length \n",
"0 32 \n",
"1 39 \n",
"2 34 \n",
"3 53 "
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def clean_tokens(df, vocab):\n",
" df[\"length\"] = df.tokens.apply(len)\n",
" df[\"clean_tokens\"] = df.tokens.apply(lambda x: [t for t in x if t in vocab.freqs.keys()])\n",
" df[\"clean_length\"] = df.clean_tokens.apply(len)\n",
" return df\n",
"\n",
"example_df = clean_tokens(example_df, vocab)\n",
"example_df[:5]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The difficulty with our \"the cat _ on the mat\" problem is that the missing word could be any one in the vocabulary V and so the network would have |V| outputs for each input e.g. a huge vector containing zero for every word in the vocabulary and some positive number for \"sat\" if the network was perfectly trained. For calculating loss we need to turn that into a probabilty distribution, i.e. _softmax_ it. Computing the softmax for such a large vector is expensive.\n",
"\n",
"So the trick (one of many possible) we will use is _Noise Contrastive Estimation (NCE)_. We change our \"the cat _ on the mat\" problem into a multiple choice problem, asking the network to choose between \"sat\" and some random wrong answers like \"hopscotch\" and \"luxuriated\". This is easier to compute the softmax for since it's now a binary classifier (right or wrong answer) and the output is simply of a vector of size 1 + k where k is the number of random incorrect options.\n",
"\n",
"Happily, this alternative problem still learns equally useful word representations. We just need to adjust the examples and the loss function. There is a simplified version of the NCE loss function called _Negative Sampling (NEG)_ that we can use here.\n",
"\n",
"[Notes on Noise Contrastive Estimation and Negative Sampling (C. Dyer)](https://arxiv.org/abs/1410.8251) explains the derivation of the NCE and NEG loss functions.\n",
"\n",
"When we implement the loss function, we assume that the first element in a samples/scores vector is the score for the positive sample and the rest are negative samples. This convention saves us from having to pass around an auxiliary vector indicating which sample was positive."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"import torch.nn as nn\n",
"\n",
"class NegativeSampling(nn.Module):\n",
" def __init__(self):\n",
" super(NegativeSampling, self).__init__()\n",
" self.log_sigmoid = nn.LogSigmoid()\n",
" def forward(self, scores):\n",
" batch_size = scores.shape[0]\n",
" n_negative_samples = scores.shape[1] - 1 # TODO average or sum the negative samples? Summing seems to be correct by the paper\n",
" positive = self.log_sigmoid(scores[:,0])\n",
" negatives = torch.sum(self.log_sigmoid(-scores[:,1:]), dim=1)\n",
" return -torch.sum(positive + negatives) / batch_size # average for batch\n",
"\n",
"loss = NegativeSampling()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's helpful to play with some values to reassure ourselves that this function does the right thing."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>scores</th>\n",
" <th>loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>[1, -1, -1, -1]</td>\n",
" <td>tensor(1.2530)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>[0.5, -1, -1, -1]</td>\n",
" <td>tensor(1.4139)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>[0, -1, -1, -1]</td>\n",
" <td>tensor(1.6329)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>[0, 0, 0, 0]</td>\n",
" <td>tensor(2.7726)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>[0, 0, 0, 1]</td>\n",
" <td>tensor(3.3927)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>[0, 1, 1, 1]</td>\n",
" <td>tensor(4.6329)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>[0.5, 1, 1, 1]</td>\n",
" <td>tensor(4.4139)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>[1, 1, 1, 1]</td>\n",
" <td>tensor(4.2530)</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" scores loss\n",
"0 [1, -1, -1, -1] tensor(1.2530)\n",
"1 [0.5, -1, -1, -1] tensor(1.4139)\n",
"2 [0, -1, -1, -1] tensor(1.6329)\n",
"3 [0, 0, 0, 0] tensor(2.7726)\n",
"4 [0, 0, 0, 1] tensor(3.3927)\n",
"5 [0, 1, 1, 1] tensor(4.6329)\n",
"6 [0.5, 1, 1, 1] tensor(4.4139)\n",
"7 [1, 1, 1, 1] tensor(4.2530)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch \n",
"\n",
"data = [[[1, -1, -1, -1]], # this dummy data uses -1 to 1, but the real model is unconstrained\n",
" [[0.5, -1, -1, -1]],\n",
" [[0, -1, -1, -1]],\n",
" [[0, 0, 0, 0]],\n",
" [[0, 0, 0, 1]],\n",
" [[0, 1, 1, 1]],\n",
" [[0.5, 1, 1, 1]],\n",
" [[1, 1, 1, 1]]]\n",
"\n",
"loss_df = pd.DataFrame(data, columns=[\"scores\"])\n",
"loss_df[\"loss\"] = loss_df.scores.apply(lambda x: loss(torch.FloatTensor([x])))\n",
"\n",
"loss_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Higher scores for the positive sample (always the first element) reduce the loss but higher scores for the negative samples increase the loss. This looks like the right behaviour."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With that in the bag, let's look at creating training data. The general idea is to create a set of examples where each example has:\n",
"\n",
"- doc id\n",
"- sample ids - a collection of the target token and some noise tokens\n",
"- context ids - tokens before and after the target token\n",
"\n",
"e.g. If our context size was 2, the first example from the above dataset would be:\n",
"\n",
"```\n",
"{\"doc_id\": 0,\n",
" \"sample_ids\": [word2idx[x] for x in [\"week\", \"random-word-from-vocab\", \"random-word-from-vocab\"...],\n",
" \"context_ids\": [word2idx[x] for x in [\"in\", \"the\", \"before\", \"their\"]]}\n",
" ```\n",
" \n",
" The random words are chosen according to a probability distribution:\n",
" \n",
" > a unigram distribution raised to the 3/4rd power, as proposed by T. Mikolov et al. in Distributed Representations of Words and Phrases and their Compositionality\n",
"\n",
"This has the effect of slightly increasing the relative probability of rare words (look at the graph of `y=x^0.75` below and see how the lower end is raised above `y=x`)."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.vegalite.v2+json": {
"$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
"config": {
"view": {
"height": 300,
"width": 400
}
},
"data": {
"name": "data-afc3dc0951a9e12875119a5af86e52b5"
},
"datasets": {
"data-afc3dc0951a9e12875119a5af86e52b5": [
{
"x": 0,
"y": 0
},
{
"x": 0.01,
"y": 0.03162277660168379
},
{
"x": 0.02,
"y": 0.053182958969449884
},
{
"x": 0.03,
"y": 0.07208434242404263
},
{
"x": 0.04,
"y": 0.08944271909999159
},
{
"x": 0.05,
"y": 0.10573712634405642
},
{
"x": 0.06,
"y": 0.12123093028059741
},
{
"x": 0.07,
"y": 0.13608915892697748
},
{
"x": 0.08,
"y": 0.15042412372345573
},
{
"x": 0.09,
"y": 0.16431676725154984
},
{
"x": 0.1,
"y": 0.1778279410038923
},
{
"x": 0.11,
"y": 0.19100490227716513
},
{
"x": 0.12,
"y": 0.2038853093816547
},
{
"x": 0.13,
"y": 0.2164998073464082
},
{
"x": 0.14,
"y": 0.22887377179317683
},
{
"x": 0.15,
"y": 0.2410285256833955
},
{
"x": 0.16,
"y": 0.25298221281347033
},
{
"x": 0.17,
"y": 0.26475044029330763
},
{
"x": 0.18,
"y": 0.2763467610958144
},
{
"x": 0.19,
"y": 0.28778304315451386
},
{
"x": 0.2,
"y": 0.29906975624424414
},
{
"x": 0.21,
"y": 0.3102161981490854
},
{
"x": 0.22,
"y": 0.32123067524150845
},
{
"x": 0.23,
"y": 0.33212064831351956
},
{
"x": 0.24,
"y": 0.34289285156385596
},
{
"x": 0.25,
"y": 0.3535533905932738
},
{
"x": 0.26,
"y": 0.3641078238014289
},
{
"x": 0.27,
"y": 0.37456123052590357
},
{
"x": 0.28,
"y": 0.38491826849295824
},
{
"x": 0.29,
"y": 0.39518322257770583
},
{
"x": 0.3,
"y": 0.4053600464421103
},
{
"x": 0.31,
"y": 0.41545239829339137
},
{
"x": 0.32,
"y": 0.42546367175559907
},
{
"x": 0.33,
"y": 0.43539702265375557
},
{
"x": 0.34,
"y": 0.4452553923589699
},
{
"x": 0.35000000000000003,
"y": 0.45504152822405847
},
{
"x": 0.36,
"y": 0.46475800154489
},
{
"x": 0.37,
"y": 0.47440722340731084
},
{
"x": 0.38,
"y": 0.4839914587188715
},
{
"x": 0.39,
"y": 0.4935128386754873
},
{
"x": 0.4,
"y": 0.5029733718731741
},
{
"x": 0.41000000000000003,
"y": 0.5123749542422491
},
{
"x": 0.42,
"y": 0.5217193779544038
},
{
"x": 0.43,
"y": 0.5310083394307343
},
{
"x": 0.44,
"y": 0.5402434465602292
},
{
"x": 0.45,
"y": 0.549426225222706
},
{
"x": 0.46,
"y": 0.5585581251971565
},
{
"x": 0.47000000000000003,
"y": 0.5676405255254853
},
{
"x": 0.48,
"y": 0.576674739392341
},
{
"x": 0.49,
"y": 0.5856620185738529
},
{
"x": 0.5,
"y": 0.5946035575013605
},
{
"x": 0.51,
"y": 0.6035004969804791
},
{
"x": 0.52,
"y": 0.6123539276009055
},
{
"x": 0.53,
"y": 0.6211648928681236
},
{
"x": 0.54,
"y": 0.629934392084505
},
{
"x": 0.55,
"y": 0.6386633830041155
},
{
"x": 0.56,
"y": 0.6473527842827909
},
{
"x": 0.5700000000000001,
"y": 0.6560034777426358
},
{
"x": 0.58,
"y": 0.6646163104680073
},
{
"x": 0.59,
"y": 0.6731920967482075
},
{
"x": 0.6,
"y": 0.6817316198804996
},
{
"x": 0.61,
"y": 0.6902356338456498
},
{
"x": 0.62,
"y": 0.6987048648669424
},
{
"x": 0.63,
"y": 0.7071400128625219
},
{
"x": 0.64,
"y": 0.7155417527999327
},
{
"x": 0.65,
"y": 0.7239107359608682
},
{
"x": 0.66,
"y": 0.7322475911233668
},
{
"x": 0.67,
"y": 0.7405529256680135
},
{
"x": 0.68,
"y": 0.7488273266140879
},
{
"x": 0.6900000000000001,
"y": 0.7570713615910638
},
{
"x": 0.7000000000000001,
"y": 0.7652855797503655
},
{
"x": 0.71,
"y": 0.7734705126218591
},
{
"x": 0.72,
"y": 0.7816266749191567
},
{
"x": 0.73,
"y": 0.7897545652974598
},
{
"x": 0.74,
"y": 0.7978546670673515
},
{
"x": 0.75,
"y": 0.8059274488676564
},
{
"x": 0.76,
"y": 0.8139733653002305
},
{
"x": 0.77,
"y": 0.8219928575293057
},
{
"x": 0.78,
"y": 0.829986353847804
},
{
"x": 0.79,
"y": 0.837954270212839
},
{
"x": 0.8,
"y": 0.8458970107524514
},
{
"x": 0.81,
"y": 0.8538149682454624
},
{
"x": 0.8200000000000001,
"y": 0.8617085245761865
},
{
"x": 0.8300000000000001,
"y": 0.869578051165608
},
{
"x": 0.84,
"y": 0.8774239093805121
},
{
"x": 0.85,
"y": 0.8852464509219427
},
{
"x": 0.86,
"y": 0.8930460181942644
},
{
"x": 0.87,
"y": 0.9008229446560111
},
{
"x": 0.88,
"y": 0.9085775551536168
},
{
"x": 0.89,
"y": 0.9163101662390513
},
{
"x": 0.9,
"y": 0.9240210864723069
},
{
"x": 0.91,
"y": 0.9317106167096201
},
{
"x": 0.92,
"y": 0.9393790503782488
},
{
"x": 0.93,
"y": 0.9470266737385726
},
{
"x": 0.9400000000000001,
"y": 0.9546537661342305
},
{
"x": 0.9500000000000001,
"y": 0.9622606002309622
},
{
"x": 0.96,
"y": 0.9698474422447793
},
{
"x": 0.97,
"y": 0.9774145521600454
},
{
"x": 0.98,
"y": 0.9849621839380145
},
{
"x": 0.99,
"y": 0.992490585716335
}
]
},
"encoding": {
"x": {
"field": "x",
"type": "quantitative"
},
"y": {
"field": "y",
"type": "quantitative"
}
},
"mark": "line",
"title": "x^0.75"
},
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFxCAYAAADzkehqAAAgAElEQVR4Xu2dC5RlVXnnf6e6oRpXjGSUiMYHUMD4ttGAkvHRLiX46JZqtYFSg9UYzSwSQmkMFI5OV+tI3cAk3QxJBkYTaBND4wMaq2Am0SAsW4zigwZCOkYHmlHA+IBePqCBrjNrV52ib926j7PPt8+9d9/zv2uxatF3//fZ+/d95/zvPmefvRP0EQEREAEREAERaEkgERsREAEREAEREIHWBGSUyg4REAEREAERaENARqn0EAEREAEREAEZpXJABERABERABIoR0IiyGDepREAEREAEKkJARlmRQKubIiACIiACxQjIKItxk0oEfAm8H/hT4BfAbwB76yp4ObAFeAHwfeC/AZ9qcoCJrFzjV9uBs4CfNtGMAe57fURABAoSkFEWBCeZCHgQcOfZvwL/ALwDmAIuzvSrgLuA7wDrgY8A/zkzzd0djjEKXAOcDnwd+L/AVuB9Hm1TUREQgQ4EZJRKERGwE/gj4L8DbtToRoavBm4ErgTeDvw28H+A/wicDZwMPAdIgTcA12cjwv8JvAz4J2BTZpqtWvdE4F8yk30lcBzwLWBzZsT2XqkGERCBeQIySiWCCNgJuPPoi8DxwIsy4/tV4IXAA8AO4FeA1wHPA/4ZOCnTnJONAt8KXA08G7gb+Lts9NmqdR8FPgS8Cvgy8BrgBuBrwDOBg4DPAc7Ef2nvomoQgeoSkFFWN/bqeVgCzpxuA/YBv56NGr+QmZa7teqeFX4mO+RN2fNEd6v1fOAC4M3ADPA04F7gWsDdWm32eUo2knQjSDd6dZ/F27BuJOvqdc8sPwbUsmOE7a1qE4EKEZBRVijY6mrpBNxzxz/Mnhe6W6ju40Z+7wWeATya/Zt7pvi3wJGZqTld44jSTeZ5Z4sWL07q2Qhc0aLMIdnEoVuBl5Tecx1ABAaYgIxygIOrrnWVgJuxeks20ntudtvU3T7t9HHPL/++7hnlicDNwAeB6Rbir2TPMp8K/KRFmSdkRunq+k+dGqHvRUAEWhOQUSo7RMBOYDgbRT45m63qbpuuBl6cPW9sd4SDge9l/7nbp38CvCub7ON07ratM9LX11Xinjn+KHueufjP48BfABcCFwEfyCb2uGeg/8PeRdUgAtUlIKOsbuzV83AE/ix7JeMU4PPAscAu4JvZM8T9HQ71m5nJPT97xcNN0nH1HNHEKA8D/j0bddaPFFdkt3ndLNvDgf8HfCIzzblwXVVNIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SMgo6xezNVjERABERABDwIySg9YKioCIiACIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SMgo6xezNVjERABERABDwIySg9YKioCIiACIlA9AjLK6sVcPRYBERABEfAgIKP0gKWiIiACIiAC1SPQD0bpFoHeBrhlwC5tEgLXRrcZrlvL8n7gzGz5rupFSz0WAREQARHoOoFeG6VbENqZoNvc9vstjNLtree2FXJ/3S7uzkyP6TopHVAEREAERKCSBHptlIvQ3SLQP25hlG4nhDuyUacr7zbHPRm4r5IRU6dFQAREQAS6SiAGo7ws2/l9NiNzQ7Z33+7p6empJEk21RM77LDDWLNmTVch6mAiIAIiIAKDQWBkZGSZL8ZglG5/vTvrdnK/HTgpe165LDK1Wi2dnJzsl34ta9/3vve9tFkg+iXF1D5bJMRP/GwEbGrlXz5+o5uuOXT/vodHZ2pjV9QrWvHrF0Npd+t1HXAu4Pb6Ox6oZc8qmxKRUeZLlFaldKKJn42ATa38Ez8bgfZqZ5CP7XvkHNJ0goRDV5IeuaM2dveiql+N8hXAlxu65jamXQnsBI4GFme9bgTuBc4AbmmFQ0ZpSzNdqMTPRsCmVv6Jn41Aa/Wbzr/qXaTpVLKwITopXHsQ6UQMRhmciYzShlQXKvGzEbCplX/iZyOwXL128so1MD+XZXHyyi5IJ2ZrYzc2lu7XEWVoJsgobUh1oRI/GwGbWvknfjYCB9Sjk1ce8SjJpmThHXz32ZOQTjU+l6w/nowyFH1jPboQ2ACKn/jZCNjUyr/+5/f4c0jSqay1eyHdunJ41dYdm9c/2K4HMkpbfIOpdaLZUIqf+NkI2NTKv/7m555DJnPpVjdRx7U0Id22Aqbqn0PKKN2UWL0eYspkXQhM+BA/8bMRsKmrmn/zzyHTZAsJqxcIpjcBU82eQ8ooZZS2swx0oTcSrOqFyojtcbn42UhWjZ97DvnYgkGOZuQ6PoeUUcoobWeZjFL8zARsFVTtQm+jtVxdFX6tnkPO1sYWn0sWQqtnlIWwhRdVJZHDk1uoUfxsZMVP/GwEbOoQ+df0OeTwqolOE3XytFxGmYdSF8qESJQym6n22eiKn/jZCNjUg5x/zZ5Dzg2tmLj+glNvtVE7oJZRhiJprGeQE9mIJpdc/HJhallI/MTPRsCmLpJ/TZ9DJsnEzPRpO2ytyX/rul/Weg3WX816taEsksi2I/qp1T4/Xo2lxU/8bARsap/8K+s5ZLseaERpi28wtU+iBDuoR0VqnwesJkXFT/xsBGzqQcm/Mp9Dyig169V2lmmyjPiZCdgqGJQLvY1CcXXs/LrxHFJGKaMsfoZlythPNDMAYwXiZwMoftXk183nkDJKGaXtLNOIUvzMBGwVyCirxa8XzyFllDJK21kmoxQ/MwFbBTLK6vDr1XNIGaWM0naWySjFz0zAVoGMcvD5nfPxr7+mcV3W0O9DFqWoWa9FyQXW6UJgAyp+4mcjYFMr/4rzc88hX3rsYXd97Ts/XqzEtC5r8Za0Vsooy6BaoE6daAWg1UnET/xsBGxq5Z8/v8efQ6bpRLb9Ve79If2PZlPIKG38gql1otlQip/42QjY1Mo/P37uOSRpOpXAEU55wrFP4Vvf+dGRefeH9DuavbSM0s4wSA060WwYxU/8bARsauVfPn5v/OCnVw/NzW0B1mSKXZBOXPyeE740MjLStyvCySjzxbf0UjrRbIjFT/xsBGxq5V97fu4266P79m1JYDwruTchnZipjV3h/j9Wfn3r7EXTWWu9FiW3oIs1kW29DqcWPxtL8YuX39rztp8DTGXPIYF088rhVVvrt7+KNb69Nkp3fDc8d78+7gfOBG5uSJUVwMeBtwL3AL8DtNxWRUYZ74mWp+Wxnmh5+taNMuJnoyx+y/m5ZedSkssXn0NCetNKGG/2HDJWfr02yvXABOD+HgdcChzTEIpTgXcB7wDeAJwN/FardJdR6kJgI2BTx3ohsPU6nFr8bCy7yW9+2TmSy+ueQ+6BdHy2NnZjq150s31FSPbrM8qLgDuAbVmnbgNOBu6r6+Qm4AfAJ7J/+yHwdGB/MxAyyiLpcUATayLbeh1OLX42luLX//wsy87FGt9ejygvA2aA2Sw9bgDOAnbXpcvbs9HkGHBiVv7XgQenp6enkiRxRrrks2HDBlu2SS0CIiACIrCMwNe/82Ou/uo9PPTIwjjFve7xlhOfxSEHuydkg/FpNiu310Z5IXAnMD8jCrgdOCl7XrlI/SDgU8AbgauBVwBHtQqJRpS2ZI31F5+t1+HU4mdjKX79yW9++yvmByXZ6x7pTW7iTrvbrM16Emt8e22U64BzgVOA491SrdmzynrGzwBeCnwRcKNL95zyLTJK2wnVSh1rIpdDw79W8fNnVq8Qv/7i1+l1D9/WxhrfXhvl4qzXjcC9wBnALcDhwE7gaOBXgc8DLwO+kd2GdbNfm340ovRN3aXlY01kW6/DqcXPxlL8+odfntc9fFsba3x7bZS+nDuWl1F2RNS2QKyJbOt1OLX42ViKX+/5+bzu4dvaWOMro/SNtLF8rIli7HYwufjZUIqf+LUiMP+6R5psIWE0K9PxdQ9fmrHmn4zSN9LG8rEmirHbweTiZ0MpfuLXSKDV7h6ztbEpG63l6ljzT0YZOhM61BdronQZU8vDiZ8tEuInfvUE1p1/1ehcmrq1Wed390hIt60YXjVRv+ycjdhSdaz5J6MMmQU56oo1UXJ0rStFxM+GWfzEzxFotbuH7+sevjRjzT8ZpW+kjeVjTRRjt4PJxc+GUvyqzc/dZt3/8L5NaTK/dKj7LNndw0anszrW/JNRdo5t0BKxJkpQCIbKxM8AT7vD2OBFzs9topzMpVsP7O7BxSuHh6fKus3aDHas56+M0nzq+FUQa6L49bK80uJnYyt+1ePXbFWduaEVE9dfcGrLXZhslFqrY80/GWVZGdGi3lgTpcuYWh5O/GyREL/q8Guyqs6ehHRqcRNlG4li6ljzT0ZZLN6FVbEmSuEOBxaKnw2o+FWDXxmr6tjILahjzT8ZZYjoe9QRa6J4dLHUouJnwyt+g83v73feml4ys/tWElYv9LT1Jso2EsXUseafjLJYvAurYk2Uwh0OLBQ/G1DxG0x+TW+zJsnEzPRpO2w9DquONf9klGHzoGNtsSZKx451qYD42UCL3+Dxq7/N6vaFfOiRxzavHF61tZuzWfNSjTX/ZJR5IxyoXKyJEqj75mrEz4ZQ/AaHX+Pi5SlcO3X6i045fvXz+va6Hmv+9S3Qoums3UOKklvQxZrItl6HU4ufjaX4debXbvFy8evMr12JVvxklDau3molsjeyJQLxEz8bAZu61/m3dvKqTaTpRLZowF5It9YvXt7r9nWiG2v7ZJSdIhv4+1gTJTCGwtWJX2F0umNgQ9dTfs1usx5EOrGjNnZ3fbd0ftiCrBGljV8wtRLZhlL8xM9GwKbudv757hHZ7fb50oy1fRpR+kbaWD7WRDF2O5hc/GwoxS8efp1uszbrieJbTnxllDau3molsjeyJQLxEz8bAZu6G/mX9zarjNIWSx9+MsrwrNvW2I0TzdIltc9CT7OGbfSqzc/3NqvPhd4al1D6WK8vMspQGZCznlgTJWf3Si8mfjbE4tef/IrcZpVR2mLpw09GGZ61RpQlMtWF3gZX/PqLn+U2q8+F3tbrcOpY86/XRumOvwUYB+4HzgRubgjLQcDlwJuBH8H8ztwzrUKnBQdsSR1rItt6HU4tfjaWVeHn1mZ97OF9l5MwmhHbA+n4bG3sRgvBqvCzMGqn7dfXQ9Znxuf+HgdcChzT0JG3Ab8DvAM4FvgscJSMspxU0Ylm4yp+4teJQMMWWMsWDeikL3Kht9QZUhvr+dHrEeVFwB3AtiwYtwEnA/fVBec1wPuBtwNHA5cBJ8goQ6bvgbpiTeRyaPjXKn7+zOoVg8zP3WYlTbYsboHl1mZttmiAheAg87Nwyavt1xGlMz13G3U268gNwFnA7rqOOTN3W8W4W6/7gLcA17vvp6enp5Ik2dQIYcOGDXm5qJwIiIAIlErgoUf2c81X7+Fr3/nx/HH+w68czDvWHMXRT3tiqcdV5cUIjIyMLBtA9npEeSFwJ3BF1qXbgZOy55WLvXxPdlv2A8DTMpN8MfBwMwx6RlksORZV+kUqfjYCNvWg5V/DbVa3kXKpW2ANGj9bNvmr+3VEuQ44FzgFOB6oZaZY38MPAU8HnFEenk32cc8xfyaj9E+ETgqdaJ0Itf9e/MTPEXjjBz+9emhuzk1UXLNAJL1pJYw3rs1qo7VcrfyzEe1Xo1yc9boRuBc4A7glM8Sd2TPJpwB/C7wS2AtcAPx5KxwaUZaTKLZaw6l1IbCxFL9y+bnZrPsf3rcpTeZn57vPniRJJmamT3OPj0r/KL42xP1qlLZeNVHLKG1IdaKJn42ATR1z/r3p/KvelcylW7MtsEq/zdqMdMz8bJkTRi2jDMPRXIsS2YZQ/MTPRsCmbpZ/80vPkbh3vR+/zTo3tGLi+gtOvdV2NH+1zg9/ZvUKGaWNXzC1EtmGUvzEz0bApq7Pv/lFA/Y9cg6kU1mtexPSiZna2OLkRNvBCqh1fhSAVieRUdr4BVMrkW0oxU/8bARs6sX8a1x6Drh45fDw1I7N6x+0HcGm1vlRDr9evx5i61UTtZ5R2pDqRBM/GwGbut/z75Zb70w3X3nbjrql53ZBOmFdes5G7YC63/nF2j4ZZagMzVlPrImSs3ulFxM/G2LxK87P7fBxyMFDU24BATcDP02Yum769K3FawyvVHxtTHXr1cYvmFqJbEMpfuJnI+Cv7sbSc/6taq7Q+WEjKaO08QumViLbUIqf+NkI5Fe7yTqP7tu3JVnY3ch99py99jnPPvkVq/v2TpzOj/zxbVZSRmnjF0ytRLahFD/xsxHIp272TuRsbWxK+ZePX6tSsfLr219GRcOhyTxFyS3oYk1kW6/DqcXPxrLX/Jq9E1m/9Fyv29eJrtrXiVD77zWitPELplYi21CKn/jZCDRX530nUvlnox8rP40obXH3VseaKN4dLUkgfjaw4recn887keJXzfyTUdri7q3WieaNbIlA/MTPRuCAen4U+fC+y33eiVT+2ejHyk9GaYu7tzrWRPHuaEkC8bOBFb8Ffg37RO6FdKubrNOJrvh1ItT++1j5yShtcfdWx5oo3h0tSSB+NrBV59e4T2QK1x5EOpF3n8iq87NlX7yTBWWU1sh76nWieQJrKC5+4leEQJPJOoX2iVT+FaF/QBMrPxmlLe7e6lgTxbujJQnEzwa2ivx8Jut0oltFfp2Y+HwfKz8ZpU+UA5SNNVECdD1IFeJnw1glfs0m68wNDY1b9omsEj9bpjVXx8pPRllGNrSpM9ZE6TKmlocTP1skqsKvcbJOqAXMq8LPlmWt1bHyk1GWlREt6o01UbqMSUZZEvBBzz/rZJ1O2AedX6f+W7+PlZ+M0hp5T32sieLZzdKKi58N7aDyCzVZpxPdQeXXqd+hvo+Vn4wyVAbkrCfWRMnZvdKLiZ8N8SDyCzlZpxPdQeTXqc8hv4+Vn4wyZBbkqCvWRMnRta4UET8b5kHiV8ZknU50B4lfp76W8X2s/HptlO74W1jY7+1+4Ezg5oYAfQj4aMO/Hep2GG8WSO0eYkvvWBPZ1utwavGzsczLr2EbrNwr69haF+8L89Z+h9LnjW+o4/nW06p9vTbK9cAE4P4eB1wKHNOmc88B/gQ4pVUZGaVvaiwtH2si23odTi1+Npad+HXaBst29M7qTu3rXEO5JdQ+G99+NcqLgDuAbVn3bgNOBu5r0d0vAO8F7pJR2hKilVonmo2r+JXHb+3kVZsgXVyPdW+SJOMz06ftsB3RT634+vFqLB0rv16PKC8DZoDZDOgNwFnA7ibhWAu8LhuBzn89PT09lSTJpsayGzZssEVTahEQgb4h8N37fsbVX72HH/zkl/NtOuHYp/CWE5/FIQev6Js2qiGDQ2BkZGSZL/baKC8E7gSuyDDfDpyUPa9sJH898P4WJvp4Wd16tSVsrL/4bL0OpxY/G8t6fm6yzv6H921Kk/nHM+6zB9Lx2drYjbajFFcrvsXZOWWs/HptlOuAc7NnjscDtexZZWM0ngDs6vD8cl4jo6xmItt6HU4d64UgHAFbTYv81p1/1ehcmm5J4IiFGtPNebbBsh29s1rx7cyoXYlY+fXaKBdnvW4E7gXOAG4BDgd2Akdn0J+bTfR5dacwySg7EWr/fayJbOt1OLX42Vje8S/fSSev+NaOA5sppzfNDa2YsKzPamvRUrXia6MZK79eG6WNehO1jNKGNNZEtvU6nFr8irN067MeMrxi60OP7HeV7A21PmvxFi1XKr42mrHyk1Ha4u6tjjVRvDtakkD8bGD7kV/jKx++mynbiPip+5FffQ/UPr94Npbu19dDbL3SiDI4P51oNqTi58dv/pWPNJ0gYX4Rkd896egnjb72N/v2B7zi6xffvEZkqzWcWkYZjqWpJp1oJnzRzpqz9Tqcul/yb36Xj/1zl5Ow2vUuId22YnjVxJ+e8aIHmk3PD0fAVlO/8GvVC7WvnPj27S+3ot3VM8qi5BZ0OtHEz0agvbrZLh/1r3wo/2z0xa8cfjJKG1dvtRLZG9kSgfjFyy/PLh+Kb7zxzdPyWOMro8wT3YBlYk2UgAhMVYmfCV9P7hg0WThg19zQ0HizVz4U3/ji69PiWOMro/SJcoCysSZKgK4HqUL8bBi7zc8tHJDOpe5ZpJus03HhgG63z5em2udLbGn5WPnJKG1x91bHmijeHS1JIH42sN3it3yvyPSmlTC+ozZ2d7sedKt9RSmqfUXJLehi5SejtMXdWx1ronh3tCSB+NnAdoNf416RPgsHdKN9FoJqn4WejNJGL6Bas15tMHUhEL+iBELsFan8K0o/7hGbrdfh1K3yTyPKcIxz1aQLQS5MLQuJX3/yc8vPAVOLCwcU3StS8e3P+NpadUAda3xllKEyIGc9sSZKzu6VXkz8bIhD82u6/Nzw8PiOzesfLNLS0O0r0oZ2GrXPRjRWfjJKW9y91bEmindHSxKInw1sSH7zy8+RTmUt2lt0FFnfo5Dts5Fqrlb7bFRj5SejtMXdWx1ronh3tCSB+NnAhuDXuPwccPHK4eGpoqNIGaUtpuJXPj8ZZTjGuWoKcaHKdaCChdS+guAy2aDzaxhF7qlffs5GbkE96PxCMNKt4fIoajJPeWy9ataFwAvXssLi1xt+ZY4iNSKyxVT8yuenEWU4xrlq0oU+F6aWhcSvu/w6LWJua81yteJrIyp+5fCTUdq4equVyN7IlgjEr3v8li9inm5eObxqa4hnka16ofh2L762IxVTxxpfGWWxeBdWxZoohTscWCh+NqB5+PksYm5rjUaU4heagK0+PaO08QumznOhCnawAhWpfQWg1Uli59dsFDlbG1t8BcQGJ4c6dn45ulhqEfGz4ZVR2vgFUyuRbSjFrxx+vRxF1vdI8S0nvrZaw6ljja9uvYbLgVw1xZoouTrXhULiZ4PcjF+vR5EySltMxa98fr02Snf8LcA4cD9wJnBzk26/C/gY8ARgE3BJKzRaFN2WNDKi6vDrl1GkLvS2nBO/8vn12ijXAxOA+3sccClwTEO3Dwe+AKwDfgEcBXxNRhkuOXSihWMZyw+NfhpFKv+ql3/hehy2pn59RnkRcAewLevubcDJwH113X8n8Cbg5cAq4GzgszLKsAmyWFssF/pyem+vtd/53fEv30nPv+JbW9Nk/geq++yaGxoav/6CU2+1995eQ7/zU/tsMY6VX69HlJcBM8Bshv8G4Cxgd104PgCsBd4GHAnsAJ4JzE1PT08lSeJuxS75bNiwwRZNqUVgAAl8976f8amb7uKnP9s337vXv+TpvOGlvzGAPVWXRKA4gZGRkWW+2GujvBC4E7gi69btwEnZ88rFnv4e8Azgw9k/fMud48C/N0OhZ5TFE8QpY/3FZ+t1OHU/8uvHZ5GtiPcjv/q2qn22cyVWfr02Svfc8VzgFOB4oJY9q6yPxvOAzwGvA54KXJONLOdklLakbaaONZHDkyhWY7/xa3wW6UaRf3Dqq3p93reE22/8Ghuq9hU7LxZVsfLr9QmzOOt1I3AvcAZwC+Am8OwEjs4Au9uvzlAfAn6/7lbtsqhpRFnNRLb1Opy6Xy4ErUaRl7z7pd9udmspHAFbTf3CTyNeWxwHjV+vjTJ4NGSUNqS6UMXPr92MVsU3/vi264HiW058ZZQ2rt5qJbI3siUC8WvNL8+zSPFT/tkI2NSx5p+M0hZ3b3WsieLd0ZIE4tcc7PL9ItPNzdZoFT9bYopfNfnJKG1x91brRPNGphFlB2RrJ6/aBOniwuVt34tU/in/bARs6ljzT0Zpi7u3OtZE8e5oSQLxOwA27yiyPhTiZ0tM8asmPxmlLe7eap1o3sg0omyCrGEUuQfS8dna2I2d6Cr/OhFq/734VZOfjNIWd2+1TjRvZDLKOgLLR5FcvHJ4eGrH5vUP5iGr/MtDqXUZ8asmPxmlLe7eap1o3shklBmBtedtPweYIuFQIPcoUrdebTknfuInowyXA7lqklHmwtSyUBX5jU5eecRjJJcDazIwXqNIXehtOSd+4iejDJcDuWqq4oU+F5icharGr2EUuTdJkvGZ6dPcxgCFPlXjVwhSG5H42YjGyk9GaYu7tzrWRPHuaEmCqvBziwc89vC+y0kYdShTuPag4eHxvM8iW+GvCr+S0k+bBhjBxpp/Mkpj4H3lsSaKbz/LKl8FfuvOv2o0nUudSbpnkeZRpG4dhsvGKuRfOFrLa4qVn4yyzKxoUnesidJlTC0PN8j8GkeRkN60EsZ31MbuDsV/kPmFYtSuHvGzUY6Vn4zSFndvdayJ4t3RkgSDyq9hIfO9acLUddOnbw2NcVD5heakW9flEI01/2SU5eRDJUdE3UAZ64nWik2zhcxXko6GHEXq1mu4zBy0/AtHJl9NsfKTUeaLb7BSsSZKMADGigaJX5El6Iz4NBnFCHCQ8s+IopA8Vn4yykLhLi6KNVGK9zisclD4+SxkHpLgoPALycSnLvHzobW8bKz8ZJS2uHurY00U746WJIidX8jFA4ogjp1fkT6H1IifjWas/GSUtrh7q2NNFO+OliSImV+IJeisWGPmZ+17CL342SjGyk9GaYu7tzrWRPHuaEmCGPmVtXhAEcQx8ivSz7I04mcjGys/GaUt7t7qWBPFu6MlCWLjV+biAUUQx8avSB/L1IifjW6s/GSUtrh7q2NNFO+OliSIhd/y1z7CLx5QBHEs/Ir0rRsa8bNRjpWfjNIWd291rIni3dGSBDHwO/uvvnnc0P45twTdaochTXhfGYsHFEEcA7+RkZG+vS6JX5GsO6CJlZ9vQrryqQ3VErWrbwswDtwPnAnc3FD/8cDX6/7tH4HXtWpDrVZLJycnffsVsEvtq4o1UboGqMOB+p3fX3zmy+n//uYPFnuxa25oaPz6C069VfzyEej3+Kp9+eLYqlSs/HwN5XvAFcA24B4bsnn1emAi+3sccClwTEO9vw28GfiDPMeTUeah1LpMrIls67Vd3ey1j9na6S63++qj+NrCIX7V5OdrlJ8GTgZ+BbgBcJvJXg08XBDfRcAdmfG6Ku76u5MAACAASURBVG7L6r+vrr7TgI8CTwZ+kRnm5zWiLEg88hFbP16o6ifsHHLwCh565LHXzNbGbiwnQrZa+5FffY/UPsXXRsCmbpV/vkbpWnEQ8CpgLfDO7P//F/AR4OeezbwMmAFmM50z37OA3XX1vARw/20H3KjzM8AzgMemp6enkiTZ1HjMDRs2eDZDxUXAn8BDj+zn7266i9vufmBe/MJnH8o71hyFM0t9REAE4iTQ7Bl5EaN8PvAW4K3u2gC4X87u77eA13uiuRC4M7ud66S3AydlzytbVfVt4I1A/ajz8bK69eoZgYbi+kWfj1+r3T7ELx+/VqXET/xsBGzqUCPKfwWOBe7Nbrt+HNgDPBP4N2CVZzPXAecCpwBu0k4tGzXWV/NymN/A1k3iORH4JHAUMNfsWDJKzwjIKL2BNa7TWr/bhy703jiXCMRP/GwEbOpQRnkd4MzR3S7dX9ckd6/p94C/9Gzm4qzXjZn5ngHcAhwO7ASOBo4E/gZ4KeA2sP397Plo00PJKD0jIKPMDWx+wk6aXLP42gekm2drY1P1FehCnxtn04LiJ342AjZ1KKO0taILahmlDbIuVM35NaywswfS8WYTdsRP+WcjYFMr/8rhV+QZpa0lJatllDbAOtGW8nMr7Dy6b9+WZOFdX/cS8bUHDQ+P79i8/sFmpMVP+WcjYFMr/8rhJ6O0cfVWK5G9kS0RdJOf21g5mZu7JoEjgL0J6cRMbcy9R9zy0832FSGp9hWhdkAjftXkJ6O0xd1brRPNG1lPjLLoxsqKbxzxLdpKxbcouQVdrPxklLa4e6tjTRTvjpYkKJvf/JZY+/ZdA6zJunCxzwo7ZbfPilXtsxEUv2ryk1Ha4u6t1onmjaxrI8rGLbEgHfVdYUfx7d/42loW94goRN9D1BHr+SGjDBF9jzpiTRSPLpZatCx+687bviVN5tcddlN2blo5vGq01YSddh0sq32hoKp9NpLiV01+Mkpb3L3VOtG8kZU6onQTdkJuiaX49ld8ba1ZrlZ8bURj5SejtMXdWx1ronh3tCRBSH7rJq8cT9NkC8n8yk9BtsQK2b4yEKp9NqriV01+Mkpb3L3VOtG8kQUfUc5P2Hl4n9tYedRVnpBuWzG8aqLIrdbG3ii+vY+vrQXt1YqvjW6s/GSUtrh7q2NNFO+OliSw8ivybqRPV6zt8zlWkbJqXxFqBzTiV01+Mkpb3L3VOtG8kQUbUa49b/s5JGzNKtxVv5i5rVW6kIpfKAK2enR9KYefjNLG1VutRPZGZjbKxlutgNe7kT4tVnx9aC0vK37iZyNgU7fKPxmljau3WhcCb2Qmo3T7RrKw44ebsLM3SZLxmenTdtha0Vqt+NrIip/42QjY1DJKG79gal0IbCh9+C1dhq74u5E+LfZpn0+9ocqqfTaS4ldNfhpR2uLurdaJ5o3Me0S5fBm65ftG2lqhEaX4lUXAVq+uL+Xwk1HauHqrlcjeyLyMsvFWa5Fl6CwtVHwt9OJdNNvW63Bq5Z+NpW692vgFUyuRbSjb8evFrdbG3ii+5cXXVnMYteJr4xgrP40obXH3VseaKN4dLUnQjF8vb7XKKMMGWueHjaf4lcNPRmnj6q1WInsja3vrtde3WmWUtniKn/iFJWCrTbdebfyCqWWUNpT1/PrhVqsu9LZ4ip/4hSVgq01GaeMXTC2jtKF0/P7ok7f92tLNlbs3q7VT6xXfToTafy9+4mcjYFPLKG38gql1IbCh/Pudt6aXzOx+cHEBgW7Pau3UesW3EyEZpY2Q+PWCX6+fUbrjbwHGgfuBM4GbW4B4ArAbOAuYbQWrVqulk5OTve5Xy1jqQlo8zfvxVqtuHRaPZzOlzg8bT/Erh1+vDWU9zO8q7/4eB1wKHNOiqx8DXp4Zq4zSlg9RGXk/zWrthF0Xqk6ENCKyERK/XvDrtVFeBNwBbMs6fxtwMnBfA4xjgT8Efg7s1IiyvFTptwt947ZYZ699zpNOfsXqXudtVD806hvbb/HViDzsuaz42nj26zPKy4CZOuO7Ibu16m6x1n8+lRnlH9cb5fT09FSSJJsa0WzYsMFGS+q+IHDTHT/k6q/eM9+W33jyEzh77XM45OAVfdE2NUIERGAwCYyMjCz7Id7rX+YXAncCV2TIbwdOyp5XLkbhbcATgcuBmkaU5SZnP/wibbctVj+0r10E1D5bfoqf+NkI2NT9OqJcB5wLnAIcnxmhe1ZZ//kSsKbh35yZfrEZEk3mKSdRbLXmV7tbrUP75y4nYXWzbbF0Ic3PsllJ8RM/GwGbOtb86/WIcnHW60bgXuAM4Bbg8GzkeHRDWDSitOVpR3UvE3nd5JXjaZpsyV792DU3NDR+/QWn3lrf6F62ryM8tKh3HkYakVsptdbr/LCx7dcRpa1XTdQaUdqQ9upEW3fe9i1pMj8DmoR024rhVRM7Nq9/sLE3vWpfXqpqX15SzcuJn/jZCNjUMkobv2BqXQiWohydvPKIx9LkmuxWqzPJjTO1scVn1su4i58tFcVP/GwEbOpY86/Xt15t1DWiDM6vm4ncsKD5nrmhodHGW60aUYYNcTfjW6Tlal8Ragc04lcOPxmljau3Wom8gGztedvPIWHrwv+lN60cXjXa7FarjNI7xdoKlH82nuJXTX4ySlvcvdVVP9GWv/rht6B51fl5J1yDQPxsBMWvmvxklLa4e6urfKJ1evUjD8wq88vDp1MZ8etEqP334ldNfjJKW9y91VU90dadf9VoOpe69yMPBXatJB3dURu72xdgVfn5cmpVXvxsJMWvmvxklLa4e6ureKLlffUjD8wq8svDJW8Z8ctLqnk58asmPxmlLe7e6iqdaI27fqQJ77tu+vRsAo83unlBlfgVI6Rbh2VwW6xT+WejGys/GaUt7t7qWBPFt6ONu36E2mC5Kvx8eectL355SWlEaSM1WPxklGVkQ5s6q3ChalyKbuXw8Jo8r37kCUUV+OXhULSM+BUlt6ATv2ryk1Ha4u6tHvQTrfF55ExtbNwbUsV/aITk1VjXoOdfmexklHa6seafjNIee68aYk2UTp1sfB7ZaSm6TvW1+n5Q+RXl4asTP19iS8uLXzX5yShtcfdWD+KJ1vg8cm5oaE2npei8wWWCQeRXlEURnfgVoXZAI37V5CejtMXdWz1oJ1qZzyObwR00ft4JZBSInw2g+FWTn4zSFndv9SCdaGsnr9oE6ZSD4LbGCv08UkbpnV4dBYOUfx07W0IB8bNBjZWfjNIWd291rIlS39HG9VpDvB+ZF+Qg8Mvb1zLKiZ+NqvhVk5+M0hZ3b3XsJ1rjeq2h3o/MCzJ2fnn7WVY58bORFb9q8pNR2uLurY75RGvYP7Lweq3e0OoEMfOz9DuUVvxsJMWvmvxklLa4e6tjPdHmJ+2QXO46nMK1Bw0Pj4daRMAHYqz8fPpYZlnxs9EVv2ryk1Ha4u6tjvFEe9Pk9ssTyBYO8Ns/0htQB0GM/EIzsNQnfhZ6WpnHRi9efjJKa+Q99TFdqLJJO18iYTWwNyGdmKmNXeHZ5aDFY+IXtOOBKhM/G0jxqyY/GaUt7t7qWE60bi4i4AMxFn4+fepmWfGz0Ra/avLrtVG6429h4bbe/cCZwM0NoXgq8DfAK4D7gPcD17YKV61WSycnJ3vdr5bZFMOJNvGJb6xfsslywEXNbadZvLdurP0OpY8h/0ZGRnT+Fgy44lsQXCZrxa/XCbkemADc3+OAS4FjGrrq/v1ZwBeAFwOfAo6SUdoSopX6r6+9Ob36q/fMf+0WEVgxvGqiF5N2WrVPFwJb3MVP/GwEbOpY86/XRnkRcAewLcN/G3ByNnJsjMgQ8Frgw8CrZJS2hG2m7qdJOzLK8PF1NcZ6oSqHhn+t4ufPrF4RK79eG+VlwAwwm8G8ATgL2N0kHA+7CSXAKPBV9/309PRUkiSbGstu2LDBFs2KqR96ZD+f+Id/47v3/YxDDl7B+hOfxcuOfUrFKKi7IiACIgDNbv332igvBO4EFmdS3g6clD2vbDaifBHwWeAFgDPOZR89o/RL9caVds59y/Of9KoTXtjrvGjZiVh/kfpFpbzS4mdjK37V5NfrC+I64FzgFOB4oJY9q6yPxsuBYeArwAiwEzgS+LmM0pa0mUm61z8OBXatHB5e86dnvOgBTaYozlUX0uLsdGvYxk78yuPXa6NcnPW6EbgXOAO4BTg8M8SjgecBbkUYN5p8APhINumnKRWNKPMlS6uVdnShz8evVSnxEz8bAZta+VcOv14bpa1XuvVaiF/99ljAxbO1093M4/mPTrRCSB8XiZ/42QjY1Mq/cvjJKG1cvdW9TuT6ma0J6cbGlXZ63b5OQNW+ToTafy9+4mcjYFPHmn8ySlvcvdW9SpTG5ehabY/Vq/blBan25SXVvJz4iZ+NgE0da/7JKG1x91b3IlEaZrbumRsaGr3+glNvbdb4XrTPB6La50NreVnxEz8bAZs61vyTUdri7q3udqI0m9nabqWdbrfPF6Da50tsaXnxEz8bAZs61vyTUdri7q3uZqLMz2xNky3u9Y+8e0h2s33e8DTZqAiyJRrF14ZQ/KrJT0Zpi7u3ulsnWv3rH27N1pnaWLafZPsmd6t93uAygdpXlNyCTvzEz0bApo41/2SUtrh7q7uRKPUzW9OE9103ffrWvA3tRvvytqVZObXPQk9GaaMnflXlJ6O0Rt5TX/aFvtPrH52aW3b7Oh2/0/dqXydCumNgIyR+4rd8mzcZZZlZ0aTusi70ja9/zA0NrWk1s7Vdl8tqXyjMap+NpPiJn42ATR1r/skobXH3VpeRKKFMUs+wvMO5TFBGfO2tOlCD2mejKX7V5CejtMXdWx36RPN9/aNTg0O3r9PxfL9X+3yJLS0vfuJnI2BTx5p/Mkpb3L3VIRMltElqROkdTo0o7ciW1BDy/AjctPnq1D4b1Vj5yShtcfdWh0qU+nckfV7/6NTgUO3rdJyi36t9Rckt6MRP/GwEbOpY809GaYu7tzpEohR9RzJPY0O0L89xipZR+4qSk1HayIlflfnJKENE36MO64V+7XnbzyEhey8y3TxbG5vyOHzHotb2dTyAsYDaZwMofuJnI2BTx5p/Mkpb3L3VlkSxviOZp7GW9uWp31pG7bMRFD/xsxGwqWPNPxmlLe7e6qKJ0g2TdJ0p2j5vEAUFal9BcJlM/MTPRsCmjjX/ZJS2uHurfRPFvSP56L59WxJwa7XuLbqQQN6G+rYvb72hyql9NpLiJ342AjZ1rPkno7TF3VvtkyghFxLI21Cf9uWtM2Q5tc9GU/zEz0bApo41/2SUtrh7q/MmSi9MUrdevcO5TJA3vvYjFatB7SvGbVElftXkJ6O0xd1bnedEazDJXStJR3fUxu72PlgBQZ72Fag2mETts6EUP/GzEbCpY80/GaUt7t7qTokyOnnlEY+lyTUkrAZ2rRweXrNj8/oHvQ9UUNCpfQWrDSZT+2woxU/8bARs6ljzr9dG6Y6/hYWJKvcDZwI3N4TiScAngdcCPwU+AHy6VbhqtVo6OTnZ6361zKZ2iVLGknS+aR1rIvv2s6zy4mcjK37iZyNgU7fKv14bynpgAnB/jwMuBY5p6OoLAfff54DnADuAIwfNKPvBJB1TXajKOdFstYZTK742luJXTX69NsqLgDuAbRn+24CTgftahONFwJ8Drxoko6w3yRSuPWh4eLybt1vrWepCUM0Lga3X4dTKPxtL8SuHX6+N8jJgBpjNuncDcBawu0l3nwv8JfC7buDjvp+enp5KkmRTY9kNGzbYaHVR/YOf/JJLZnfz0CP7OeHYp/COV7ccLHexVTqUCIiACFSTwMjIyDJf7LVRXgjcCVyRheR24KTseWV9lNYAk8DGNqPN+fIxPaOsH0mG3AHEkt76RWqhp1vXNnriJ35WAjZ9vz6jXAecC5wCHO98LntWWd/bZwN/DawFHuqEIRaj7EeTdGxllJ0yrP334id+NgI2tfKvHH69HlEuznp1I8V7gTOAW4DDgZ3A0cCHgY80dP/XgKavTMRglGf/1TePG9o/9yUSDu2XkeQiX51o5ZxotlrDqRVfG0vxqya/XhuljXoTdb8b5Y5//Eb6iX/47oP9aJIaUdrTURdSG0PxEz8bAZu6X2+92noVmVGWueFyKJC6UNlIip/42QjY1Mq/cvhpRGnjmlsdg0lqRJk7nC0L6kJlYyh+4mcjYFNrRGnjZ1LPT9yZm/u2q8S9AvJfz3xd3/5A0YXKFGpNhrLhEz/xMxKwyWWUNn6F1Y2zW7e+54R3NXtPp/ABAgtllDag4id+NgI2tfKvHH59O7Ip2t1+mszT7BUQJXLRyC7oxE/8bARsauVfNfnJKG1xb6lu9Z6kTjQbcPETPxsBm1r5V01+Mkpb3Juq2y0moBPNBlz8xM9GwKZW/lWTn4zSFvdl6k4r7uhEswEXP/GzEbCplX/V5CejtMV9ibqTSeoZmx22LlQ2huInfjYCNnWs+SejtMX9cfXo5JVHPJYm33Yr7ritsq6rnT7arOpYEyUQJnM14mdDKH7iZyNgU8eafzJKW9zn1aObrjn0sYf3ubVbVwO7Vg4Pr2m1n2SsiRIAU5AqxM+GUfzEz0bApo41/2SUtrh7maRuvRph6/UQM8BYL1TmjgeqQPxsIGPlJ6M0xN1nJLl4mFgTxYApqFT8bDjFT/xsBGzqWPNPRmmI+9rJ7V8C3KbSe1YOD69udbu1/hCxJooBU1Cp+Nlwip/42QjY1LHmn4yyYNzfNLn98gTGgb1zQ0Nrrr/g1FvzVBVrouTpWzfKiJ+NsviJn42ATR1r/skoC8S9qEnqGWUB2A2SWE80e8/D1CB+No7iV01+MkrPuNdvlwXpa2ZrYzf6VKETzYfW8rLiJ342Aja18q+a/GSUHnFv2FNy40xt7AoP+XxRnWi+xJaWFz/xsxGwqZV/1eQno8wZ9/o9JRPSQiYpo8wJu00xXahsDMVP/GwEbOpY809GmSPueZamy1GNRpR5IckoA5BqXkWsF6rSgHhWLH6ewBqKx8pPRtkh7vXvSrZbmi5v+sSaKHn7V3Y58bMRFj/xsxGwqWPNPxllh7ivPW+7W7+149J0edMn1kTJ27+yy4mfjbD4iZ+NgE0da/712ijd8bew8D7i/cCZwM1NQnEisA34M+DSdqGq1Wrp5ORkkH7Vvwaycnj4iDwLCnRKo1gTpVO/uvW9+NlIi5/42QjY1LHmXxBDMaBbD0wA7u9xmQke01DfwZmZPgB8v1tGuXbyyilINvkuKNCJRayJ0qlf3fpe/GykxU/8bARs6ljzr9dGeRFwRzZadBG4DTgZuK9JOD4E/LgbRrnu/KtG0zS9xrUhSZL1M9On7bClxwF1rIkSqv/WesTPRlD8xM9GwKaONf96bZSXATPAbIb/BuAsYHceo5yenp5KkvlR35LPhg0bCkfzBz/5JZfM7uahR/az/sRnseYFTy1cl4QiIAIiIAJxERgZGVnmi702yguBO4HFF/dvB07Knlc20i19ROlmuD66b9+3EzgiId02Uxtzz06DfmL9RRUUgqEy8TPA04IXNnjiV1l+vTbKdcC5wCnA8UAte1bZLCClG2XdbiBtN1+2ZIsu9BZ6WtnIRk/8xM9KwKaP9frXa6NcnPW6EbgXOAO4BTgc2AkcDbwC+HJDeA7Lnlcui1rRWa/1k3dCzXBtllKxJort9AinFj8bS/ETPxsBmzrW/Ou1UdqoN1EXMcr6yTtFFjr36USsieLTxzLLip+NrviJn42ATR1r/lXeKEcnrzzisTRxiwocmia877rp07faUqG9OtZEKZOJT93i50NreVnxEz8bAZs61vyrvFEurrwTYnm6PCkUa6Lk6Vs3yoifjbL4iZ+NgE0da/5V2ijXTm53o8dzgD0rh4dXh1h5p1MaxZoonfrVre/Fz0Za/MTPRsCmjjX/KmuUdc8l984NDa25/oJTb7WlQD51rImSr3fllxI/G2PxEz8bAZs61vyrpFFmO4Lc1a3nkvWpFWui2E6PcGrxs7EUP/GzEbCpY82/Shrlgfcl05tma2NrbKH3U8eaKH69LK+0+NnYip/42QjY1LHmX+WM8k3nb59I0vkdS/aW+b5kq3SKNVFsp0c4tfjZWIqf+NkI2NSx5l+ljPKNH/z06qH9c19yt1xDL3aeN31iTZS8/Su7nPjZCIuf+NkI2NSx5l+ljLJuE+aLZ2unu+29uv6JNVG6DqrFAcXPFgnxEz8bAZs61vyrjFHWLVHXtVdBmqVUrIliOz3CqcXPxlL8xM9GwKaONf8qYZT1q++UvURdpzSKNVE69atb34ufjbT4iZ+NgE0da/5VwijXnrf9GhJGu7X6TrtUijVRbKdHOLX42ViKn/jZCNjUsebfwBtl/cICvZjl2phWsSaK7fQIpxY/G0vxEz8bAZs61vwbaKOs34i5Gwue50mhWBMlT9+6UUb8bJTFT/xsBGzqWPNvoI2ybi3XXbO101fbQhxGHWuihOm9vRbxszEUP/GzEbCpY82/gTXK+Qk8JHe5sM4NDR3XrbVcO6VRrInSqV/d+l78bKTFT/xsBGzqWPNvYI1y8XWQhHTbTG1s3BbecOpYEyUcAVtN4id+NgI2tfKvmvwG1yjP2/6AW4Gn16+DNKaVTrRqnmi2XodTK/9sLMWvmvwG0ii/8uCzN6YklwN982xyMb10olXzRLP1Opxa+WdjKX7V5DeQRrnzwSNuBNYkpBtnamNX2EIbVq0TzcZT/MTPRsCmVv5Vk9+gGqWLZk92B+mURjrROhFq/734iZ+NgE2t/Ksmv0E2yp4tfN4ulXSiVfNEs/U6nFr5Z2MpftXk12ujdMd3e0O6Wan3A2cCNzeEIk+ZeYlbYODlh/zrAzsfPIKVpEfuqI3dbQtreLVONBtT8RM/GwGbWvlXTX69Nsr1gNvuyv09DrgUOKYhFHnKzEvcpsyvfNLdW7784BHXXlc7fdQW0nLUOtFsXMVP/GwEbGrlXzX59dooLwLuALZl+G8DTgbuqwtHnjILRjm5fccrD737lK/sPXL9zPRpO2whLUetE83GVfzEz0bAplb+VZNfr43yMmAGmM3w3wCcBeyuC0fLMtPT01NJkmyqD92KlSvZ/9hjtmhKLQIiIAIiUDkChx12GO9+97uX+WKvjfJC4E5g8RWO24GTsueVi0HKU+bxgNZqtXRycrLX/WqZYGqf7dwTP/GzEbCplX/V5NdrQ1kHnAucAhwP1LJnlfXRyFNGRmnLX/ETv0AEbNXIiMTPRsCmbpV/vTbKxRmtG4F7gTOAW4DDgZ3A0UCrMk2J6EQrJ1FstYZTK742luInfjYCNnWs+ddro7RRb6KONRDBQRSsUPwKgstk4id+NgI2tfKvHH4DZ5Rugs/5558/ZcNVnlrts7EVP/GzEbCplX/V5DdwRmkLo9QiIAIiIAIisJSAjFIZIQIiIAIiIAJtCMgolR4iIAIiIAIiUEGjzLM+bJ4yRZMnb90nZqsS/Vm2fF/R4/nq8rTvScAngdcCPwU+AHza90AFy+dp31OBvwFeka3k9H7g2oLH85Xlad9inU/IFtBwC2ksLqzhezzf8nna517H+npdxf8IvM73QAXL52mfq/pdwMcAx9AtLHJJweP5yvK070PARxsqPtTtWuR7sALl87TvIMDtyftm4EfZUqFucZdufPK0bwXwceCtwD3A7wC3Bm5cp+trnnbON2lQR5R51ofNU6Zo3PLUfXC2IPwDwPe7bJR52vdCwP33OeA5gFsS8MiiQDx1edrn1gZ+FvAF4MXAp4CjPI9TtHie9i3W7S70L89i3S2jzNO+384uon9QFIJBl6d97hUxF1v3HvUvsth+zXBMH2me9tXX586PP8neB/c5TtGyedr3tsx83gEcC3y2z86PU7MfQq59bwDOBn6rKJAmujzX1zwcB9oo86wPm6dM0bj51O1+mf64y0bp0z7H4EXAnwOvKgrEU+fTvqFs1PvhPmyfu0D9IfDz7L3gbhllHn6nZSOiJ2dG5Azz855xKlo8T/ve6ZZvzn5krMoupO5i341PnvbVt8MZ+nuBu7rROCBP+14DuLssb8/eR3dLgZ7QR+1zdwh+AHwia9MPgacD+wO3sd31NQ/HgTZK0xqyAQKV5/iLh+mFUfq077nAXwK/C3wvAJs8Vfi07+HsdpfbLeareSoPUCZv+9wo1xnlH3fZKPO07yWA+297thrWZ4BnAN1YKDlP+9yt/rWAGxm5OxnujsYzgbkA8etURZ72Ldbh2uhuWbtdkLr1ydM+d7fQMXO3XvcBbwGu71ID87TPGbgbTY4B7hapuy3868CDgdvY7vqap50DbZR51ofNU6ZozHzq7oVR5m3fGmAScCsn1e/oUpRLXl3e9rn63IjSjXjdaOMFgDPOsj952ucu8E/MnhO5pRndSlPdGlHmaV8jo28Db+xSnPO07/cy43Z3CtznW8DrgX8vO7hAnvYtNsOZjxu51W/kUHYT87TvPdkPIPeD42mZSbpHFP1yfrhnqO6HpMu5q7O5BmU8Oml3fc3DcaCNMs/6sHnKFE14n7p7YZR52vds4K+zX/UPFQVRUJenfe653zDwFWAkMyI38nC3Ocv+5GnflwD3Q6P+4xb8/2LZjcue63VaQ9nxc5NP3CQe94veTdxyF6pujNjy8Hte9nzcjdbcxK1rspFlv7TPhdFNMtrVZA/dskOch5+7rrhbmc4o3fPem7N2/qzsxuXMP3f34qXZ+eBGl+45pRv1hv60u77m4TjQRhl8DVnP6OU5vput+eWGeg/Lnld6Hs67eJ72uV/yH2mo+ddKuDXSrPF52ucupG5WnxtNuglRrq1u4+9ufPK0r74d3R5R5mmf+1HhZg27i9XdwO8Dbpu7bnzytM+1w13kneG7H2qufd0akedtn3ss4XLu1d2AVneMPO17CvC3wCuzRxMXFRdFSwAAAdlJREFUZPMMutHUPO371eyZ+MuAb2S3Yd3s11CfVtfXlUXWER/UWa+hYKseERABERCBihOQUVY8AdR9ERABERCB9gRklMoQERABERABEWhDQEap9BABERABERABGaVyQAREQAREQASKEdCIshg3qURABERABCpCQEZZkUCrmyIgAiIgAsUIyCiLcZNKBERABESgIgRklBUJtLopAiIgAiJQjICMshg3qURABERABCpCQEZZkUCrm5Un8F+B/5ItWed2aPjnbONct0ycPiIgAm0IyCiVHiJQDQJujct/yrr6k2xBdLce5qPV6L56KQLFCcgoi7OTUgRiI+AWknfbVa3Idr3v1kbDsXFSe0VgCQEZpRJCBKpDwO208nXgYOC9dbvLV4eAeioCBQjIKAtAk0QEIiSwCrgFuB34YbYZ9/OBH0TYFzVZBLpKQEbZVdw6mAj0jMDF2Z5/bg/FXwJ3ZP+5zWv1EQERaENARqn0EAEREAEREAEZpXJABERABERABIoR0IiyGDepREAEREAEKkJARlmRQKubIiACIiACxQjIKItxk0oEREAERKAiBGSUFQm0uikCIiACIlCMgIyyGDepREAEREAEKkJARlmRQKubIiACIiACxQj8f1fZJzKUyb8GAAAAAElFTkSuQmCC",
"text/plain": [
"<VegaLite 2 object>\n",
"\n",
"If you see this message, it means the renderer has not been properly enabled\n",
"for the frontend that you are using. For more information, see\n",
"https://altair-viz.github.io/user_guide/troubleshooting.html\n"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import altair as alt\n",
"import numpy as np\n",
"\n",
"data = pd.DataFrame(zip(np.arange(0,1,0.01), np.power(np.arange(0,1,0.01), 0.75)), columns=[\"x\", \"y\"])\n",
"alt.Chart(data, title=\"x^0.75\").mark_line().encode(x=\"x\", y=\"y\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"class NoiseDistribution:\n",
" def __init__(self, vocab):\n",
" self.probs = np.array([vocab.freqs[w] for w in vocab.words])\n",
" self.probs = np.power(self.probs, 0.75)\n",
" self.probs /= np.sum(self.probs)\n",
" def sample(self, n):\n",
" \"Returns the indices of n words randomly sampled from the vocabulary.\"\n",
" return np.random.choice(a=self.probs.shape[0], size=n, p=self.probs)\n",
" \n",
"noise = NoiseDistribution(vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With this distribution, we advance through the documents creating examples. Note that we are always putting the positive sample first in the samples vector, following the convention the loss function expects."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"def example_generator(df, context_size, noise, n_negative_samples, vocab):\n",
" for doc_id, doc in df.iterrows():\n",
" for i in range(context_size, len(doc.clean_tokens) - context_size):\n",
" positive_sample = vocab.word2idx[doc.clean_tokens[i]]\n",
" sample_ids = noise.sample(n_negative_samples).tolist()\n",
" # Fix a wee bug - ensure negative samples don't accidentally include the positive\n",
" sample_ids = [sample_id if sample_id != positive_sample else -1 for sample_id in sample_ids]\n",
" sample_ids.insert(0, positive_sample) \n",
" context = doc.clean_tokens[i - context_size:i] + doc.clean_tokens[i + 1:i + context_size + 1]\n",
" context_ids = [vocab.word2idx[w] for w in context]\n",
" yield {\"doc_ids\": torch.tensor(doc_id), # we use plural here because it will be batched\n",
" \"sample_ids\": torch.tensor(sample_ids), \n",
" \"context_ids\": torch.tensor(context_ids)}\n",
" \n",
"examples = example_generator(example_df, context_size=5, noise=noise, n_negative_samples=5, vocab=vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we package this up as a good old PyTorch dataset and dataloader."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"from torch.utils.data import Dataset, DataLoader\n",
"\n",
"class NCEDataset(Dataset):\n",
" def __init__(self, examples):\n",
" self.examples = list(examples) # just naively evaluate the whole damn thing - suboptimal!\n",
" def __len__(self):\n",
" return len(self.examples)\n",
" def __getitem__(self, index):\n",
" return self.examples[index]\n",
" \n",
"dataset = NCEDataset(examples)\n",
"dataloader = DataLoader(dataset, batch_size=2, drop_last=True, shuffle=True) # TODO bigger batch size when not dummy data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's going to also be useful to have a way to convert batches back to a readable form for debugging, so we add a helper function."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'doc_id': tensor(2),\n",
" 'context': 'was allowed a moment to ____ in at him where he',\n",
" 'context_ids': tensor([ 99, 5, 0, 61, 93, 52, 11, 48, 103, 47]),\n",
" 'samples': ['peer', 'moment', 'atreides', 'a', 'caladan', 'night'],\n",
" 'sample_ids': tensor([71, 61, 12, 0, 20, 65])},\n",
" {'doc_id': tensor(3),\n",
" 'context': 'mother the old woman was ____ witch shadow hair like matted',\n",
" 'context_ids': tensor([ 62, 91, 67, 105, 99, 104, 79, 44, 59, 60]),\n",
" 'samples': ['a', 'where', 'in', 'cooled', 'an', 'woman'],\n",
" 'sample_ids': tensor([ 0, 103, 52, 24, 6, -1])}]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def describe_batch(batch, vocab):\n",
" results = []\n",
" for doc_id, context_ids, sample_ids in zip(batch[\"doc_ids\"], batch[\"context_ids\"], batch[\"sample_ids\"]):\n",
" context = [vocab.words[i] for i in context_ids]\n",
" context.insert(len(context_ids) // 2, \"____\")\n",
" samples = [vocab.words[i] for i in sample_ids]\n",
" result = {\"doc_id\": doc_id,\n",
" \"context\": \" \".join(context), \n",
" \"context_ids\": context_ids, \n",
" \"samples\": samples, \n",
" \"sample_ids\": sample_ids}\n",
" results.append(result)\n",
" return results\n",
"\n",
"describe_batch(next(iter(dataloader)), vocab)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's jump into creating the model itself. There isn't much to it - we multiply the input paragraph and word matrices by the output layer. Combining the paragraph and word matrices is done by summing here, but it could also be done by concatenating the inputs. The original paper actually found concatenation works better, perhaps because summing loses word order information."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import torch.nn as nn\n",
"\n",
"class DistributedMemory(nn.Module):\n",
" def __init__(self, vec_dim, n_docs, n_words):\n",
" super(DistributedMemory, self).__init__()\n",
" self.paragraph_matrix = nn.Parameter(torch.randn(n_docs, vec_dim))\n",
" self.word_matrix = nn.Parameter(torch.randn(n_words, vec_dim))\n",
" self.outputs = nn.Parameter(torch.zeros(vec_dim, n_words))\n",
" \n",
" def forward(self, doc_ids, context_ids, sample_ids):\n",
" # first add doc ids to context word ids to make the inputs\n",
" inputs = torch.add(self.paragraph_matrix[doc_ids,:], # (batch_size, vec_dim)\n",
" torch.sum(self.word_matrix[context_ids,:], dim=1)) # (batch_size, 2x context, vec_dim) -> sum to (batch_size, vec_dim)\n",
" #\n",
" # select the subset of the output layer for the NCE test\n",
" outputs = self.outputs[:,sample_ids] # (vec_dim, batch_size, n_negative_samples + 1)\n",
" #\n",
" return torch.bmm(inputs.unsqueeze(dim=1), # then multiply with some munging to make the tensor shapes line up \n",
" outputs.permute(1, 0, 2)).squeeze() # -> (batch_size, n_negative_samples + 1)\n",
"\n",
"model = DistributedMemory(vec_dim=50,\n",
" n_docs=len(example_df),\n",
" n_words=len(vocab.words))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take it for a spin!"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 0., 0., 0., 0.]])"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with torch.no_grad():\n",
" logits = model.forward(**next(iter(dataloader)))\n",
"logits"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Oh yeah, the output layer was initialized with zeros. Time to bash out a standard issue PyTorch training loop."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"from tqdm import tqdm, trange\n",
"from torch.optim import Adam # ilenic uses Adam, but gensim uses plain SGD\n",
"import numpy as np\n",
"\n",
"def train(model, dataloader, epochs=40, lr=1e-3):\n",
" optimizer = Adam(model.parameters(), lr=lr)\n",
" training_losses = []\n",
" try:\n",
" for epoch in trange(epochs, desc=\"Epochs\"):\n",
" epoch_losses = []\n",
" for batch in dataloader:\n",
" model.zero_grad()\n",
" logits = model.forward(**batch)\n",
" batch_loss = loss(logits)\n",
" epoch_losses.append(batch_loss.item())\n",
" batch_loss.backward()\n",
" optimizer.step()\n",
" training_losses.append(np.mean(epoch_losses))\n",
" except KeyboardInterrupt:\n",
" print(f\"Interrupted on epoch {epoch}!\")\n",
" finally:\n",
" return training_losses"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we'll sanity check by overfitting the example data. Training loss should drop from untrained loss to something close to the minimum possible."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Epochs: 100%|██████████| 40/40 [00:02<00:00, 21.83it/s]\n"
]
}
],
"source": [
"training_losses = train(model, dataloader, epochs=40, lr=1e-3)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.vegalite.v2+json": {
"$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
"config": {
"view": {
"height": 300,
"width": 400
}
},
"data": {
"name": "data-b4211a284b320247988d74ee339a35c7"
},
"datasets": {
"data-b4211a284b320247988d74ee339a35c7": [
{
"epoch": 0,
"training_loss": 3.873234655897496
},
{
"epoch": 1,
"training_loss": 2.4964269541077693
},
{
"epoch": 2,
"training_loss": 1.8195302142935283
},
{
"epoch": 3,
"training_loss": 1.41878628124625
},
{
"epoch": 4,
"training_loss": 1.1586913345223766
},
{
"epoch": 5,
"training_loss": 0.9543892801818201
},
{
"epoch": 6,
"training_loss": 0.8089911760920185
},
{
"epoch": 7,
"training_loss": 0.690002828836441
},
{
"epoch": 8,
"training_loss": 0.6072199561838376
},
{
"epoch": 9,
"training_loss": 0.5273456401744131
},
{
"epoch": 10,
"training_loss": 0.4652211883310544
},
{
"epoch": 11,
"training_loss": 0.4105674152152013
},
{
"epoch": 12,
"training_loss": 0.36498250900688817
},
{
"epoch": 13,
"training_loss": 0.3294624433679096
},
{
"epoch": 14,
"training_loss": 0.29689174738980956
},
{
"epoch": 15,
"training_loss": 0.2668858468532562
},
{
"epoch": 16,
"training_loss": 0.24272207208609176
},
{
"epoch": 17,
"training_loss": 0.21813622003389618
},
{
"epoch": 18,
"training_loss": 0.1999257556715254
},
{
"epoch": 19,
"training_loss": 0.18043559644434412
},
{
"epoch": 20,
"training_loss": 0.16454930458281
},
{
"epoch": 21,
"training_loss": 0.14988945260391398
},
{
"epoch": 22,
"training_loss": 0.1378944904496104
},
{
"epoch": 23,
"training_loss": 0.12689736037183616
},
{
"epoch": 24,
"training_loss": 0.11709884914048647
},
{
"epoch": 25,
"training_loss": 0.10717140561190702
},
{
"epoch": 26,
"training_loss": 0.09873407258320663
},
{
"epoch": 27,
"training_loss": 0.09167827893111666
},
{
"epoch": 28,
"training_loss": 0.08547388257111534
},
{
"epoch": 29,
"training_loss": 0.0792784820610689
},
{
"epoch": 30,
"training_loss": 0.07406510791536104
},
{
"epoch": 31,
"training_loss": 0.06805220107405872
},
{
"epoch": 32,
"training_loss": 0.064012377917514
},
{
"epoch": 33,
"training_loss": 0.06023278630385965
},
{
"epoch": 34,
"training_loss": 0.05617725763911918
},
{
"epoch": 35,
"training_loss": 0.05255285915681871
},
{
"epoch": 36,
"training_loss": 0.04945897159434981
},
{
"epoch": 37,
"training_loss": 0.047309281130842235
},
{
"epoch": 38,
"training_loss": 0.04321649504857043
},
{
"epoch": 39,
"training_loss": 0.040696044799761244
}
]
},
"encoding": {
"x": {
"field": "epoch",
"type": "quantitative"
},
"y": {
"field": "training_loss",
"scale": {
"type": "log"
},
"type": "quantitative"
}
},
"mark": "bar"
},
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFfCAYAAADON4wsAAAgAElEQVR4Xu2dDbRdZXnn/yfBAHZKpKyMKVBFr3wo8YMgo3QxDK3aJOXix1qkXBjkS6iDtdVlLbnpkkliV3tvtCPt1FGsjBJQuDBxquYyCeMMuvADK0OsCC5Q6TAzELLGjwhjJUTCnfXk7nOzz8k5Z3++Z+9nn99eiwWc+348+/8+7/Pfz/O+7/O2xAMCIAACIAACINAXgRbYgAAIgAAIgAAI9EcAokQ7QAAEQAAEQGAAAhAl6gECIAACIAACECU6AAIgAAIgAAL5EMCjzIcbtUAABEAABEYEAYhyRAaa1wQBEAABEMiHAESZDzdqgQAIgAAIjAgCdSPKMyVtkfQRSddHY2AyXifpMkm7JV0h6RsjMj68JgiAAAiAQMUI1Ikol0SEuEfSYzGifJuk90qyf58W/X5ixbjRPQiAAAiAwIggUCeibEP+AUk/jhHlhyU9EHmaVuZ+SaskPTEiY8RrggAIgAAIVIiAB6L8hKRtkmYjnO6S9C5JD01NTW1stVob4vgtW7ZM55xzToWQ0jUIgAAIgIBXBMbGxg7hRQ9E+SFJ35N0YwT8dyW9KVqvPGQspqen5yYnJ+v4Xj315pFHHpnrNTB1VjJvMnuT18bem8ze5AXj4VgYb3rRT946Ekp36PU8SddIeoukMyRNR2uVPUcaogw/AZqi/OGRyt8DGOfHLm1NME6LVP5yTcG4TkR5lqSvdg3JMkk/iTb5XC5pl6RLJN3bb+ggyvxKnbZmU5Q/7ftWUQ6Mw6MOxmDcjYAnj7LQ6EGUheBLVRkDkwqmQoXAuBB8qSqDcSqYChVqCsZ18igzD0ivzTzWyNq1azO3RQUQAAEQAAEQ8LKZp9BI4VEWgi9V5aZ8JaZ62YoKgXF44MEYjAm9hteBUnrwNlntpb3J7E1eMC5laiU24k0vvMnbJD12HXrtNRPwKBPtQ+EC3iasN3mbZGAKK1vABrzphTd5m6THEGXAiZimaZQ/DUrFyoBxMfzS1AbjNCgVKwPGxfBLU7uRu17ZzJNm6CkDAiAAAiCQFgE286RFaojl+EoMDzYYg3EvBLzphTd5Cb2Gn3e5e2CNMjd0qSt6m7De5G2SgUmtVBUU9KYX3uRtkh6zRlnBBI13ifKHHwAwBmM8yvA60GSMIcpq9GehV4x4+AEAYzBushEPP7r5e/A299jMk3+sqQkCIAACIDAiCLCZp4YD7e2Lq0nrDjVUB7eRBvQ4vDaBcXUYE3oNj/3AHlD+8AMAxmBM6DW8DjQZY4iyGv1x6zngUQ5HYbyRuzd50WP0OAuxQ5TD0Ze+vWBgwg8AGINxFqMYHq18PaDH+XDLUquRm3l6AcA5yixqka+stwnrTV68nXx6mbWWN73wJm+T9BiPMuvsKrk8yl8yoD2aA2MwxqMMrwNNxtg1UZLrtRrlp1cQAAEQaCoCHA+p4cji7YQfFDAG4yZ7O+FHN38P3uYea5T5xzpoTW+K1KR1h6ADW7Bxb3rhTV70uKCCpqzuTS8gypQDO+xi3hQJAzMcDfGmF97kRY/R4yxRBtdrlL1elF2v4SeAN6PoTV6MeHgdBmMwhignJ918AGDEw09YMAbjLEYxPFr5ekCP8+GWpRah1yxoDbEsyh8ebDAGY4gyvA40GWM3nlfaYSb0mhap/OW8EY83eQkL5tfNLDW96YU3eZukx66JknOUWcwCZUEABEAABJIQ4BxlEkIV/J2vxPCggzEYNzksGH508/fgbe6xRpl/rIPW9KZITQqnBB3Ygo170wtv8qLHBRU0ZXVvegFRphzYYRfzpkgYmOFoiDe98CYveoweZ4kyuF6j7PWibOYJPwG8GUVv8mLEw+swGIMxRMk5yqCzwBvxeJMXIx5UfRca96YX3uRtkh7jUQ5nTvbtBeUPPwBgDMZZvIfwaOXrAT3Oh1uWWqxRZkFriGVR/vBggzEYQ5ThdaDJGONRVqM/bsM/TQqnVDz0A7v3Ru7e5EWPh6P93vQCj3I4epG5F2+KhIHJPMS5KnjTC2/yose51DJzJW960UiiJDNPZr2lAgiAAAiAwAAEyMxTE/VYs+EzRy1+5rCvmDjHH/P80x77yS++bf89Oz2xsiYiEhaseCCa8iVeMYzoccUD0BQ9Zo2yAkWKiPLJrq6fmp2eWFqBOJm7bIryZ37xIVYA4/BggzEYdyPQyNBrr2H2kHAAogw/QeM9eDOIrJ8NRz+86YU3eZukx3iUw5mTHb1AlMMFHQMTHm8wBuNeCHjTCzzK8HqcugeIMjVUpRT0Nlmb9CVeygAGasSbXniTt0l6jEcZaBIOahaiHC7oGJjweIMxGONRhteB0npgjbI0KPs25M0oepO3SV/i4bUxfw/e9MKbvE3SYzzK/PMsd008ytzQ5aqIgckFW6ZKYJwJrlyFwTgXbJkqeV6jfKGkmyWdJekJSe+T9IV+b49HmUkvchX2NmG9ydukL/FcCjakSt70wpu8TdJjDx7laZJeJOlLkl4t6bOSXgpRDsma9OjG24T1Jm+TDEx1Wprcsze98CZvk/TYA1G2NX6RpDdIulbS2RBlsiEIVcLbhPUmb5MMTCgdLKNdb3rhTd4m6bEnotwrybLZvFXSPTYIXnO97t23X+u27OyY60csWazNl7rIYFeGjaINEAABEKglAt5zvZpH+SpJWyWtkGTEecjDGmV43fP2ZetN3iZ9iYfXxvw9eNMLb/I2SY89eJSvl3S4pK9LGpP0NUkvkfRziDK/kShS09uE9SZvkwxMET0LXdebXniTt0l67IEoXyHp05E3uUfSByVdzxplaDPSv31vE9abvE0yMNVpaXLP3vTCm7xN0mMPRJms8bEShF4zwZWrsLcJ603eJhmYXAo2pEre9MKbvE3SY4hySJMy3g0JB4YLOgYmPN5gDMa9EPCmF54TDmTSQDzKTHDlKtwU5c/18kOqBMbhgQZjMO5GAKIMrxOpe8CjTA1VKQW9GcQmhaxKGcBAjXjTC2/yNkmPCb0GmoSDmk1DlOetu+VULVp8XXc726Yu+J0KRO7o0tuE9SZvkwxM1bo6qH9veuFN3ibpMURZwUxOS5RzrUUPdIn34Oz0hJ0hrfTxNmG9ydskA1OpoiZ07k0vvMnbJD2GKCuYyRDlcEHHwITHG4zBuBcC3vSCNcrwepy6B4gyNVSlFPQ2WZv0JV7KAAZqxJteeJO3SXqMRxloEpaxRknotZzBwcCUg2OT1vuaZMTDj27+HrzNPTzK/GNdek08ytIhHdigt8mKER+OfnjTC2/yNkmPPXiUSyXdFF2x9VNJ75d0e7+p1JRzlLbrFY+yHIOJgSkHRzzK8DiCcT0x9kCUr5Rk/3xO0imSPh8lRe+JKEQZXtG8EY83eZv0JR5eG/P34E0vvMnbJD32QJTxmWDXbH10FC5uxqPMbwC7a2JgysOyX0tgDMa9EPCmF01Yo3y5pI9JulLSI4Rew0/MphhFb5O1SV/i1Wlpcs/e9MKbvE3SYy8e5TmSJiVdLumJ9hSYmpra2Gq1NnRPibVr1ybPkgpL7N23X+u27OyQ4Igli7X50pULv+3e87SmtnbmG1h+9JFaf37l+QYqRI6uQQAEQCAsAmNjY4fwogeifLGkT0kal/R0EkSsUSYhVPzv3r5svcnbpC/x4toWrgVveuFN3ibpsQeivDa6rDk+Y46W9LNeUwiiDGdY2i17m7De5G2SgQmvjfl78KYX3uRtkh57IMpMMwGizARXrsLeJqw3eZtkYHIp2JAqedMLb/I2SY8hyiFNyng3ZSUcOHfdbatarbntHa/Q0n+dnZpYHfK1vE1Yb/I2ycCE1MOibXvTC2/yNkmPIcqisy1H/ZKJckcXUd4JUXYOCgYmh5JmrALGGQHLURyMc4CWsUoTjoekeuVRCr1GHiVEmaAZGJhUU6dQITAuBF+qymCcCqZChSDKQvCVWxmPslw8k1rDwCQhVPzvYFwcw6QWwDgJoeJ/hyiLY1haCxBlaVCmaggDkwqmQoXAuBB8qSqDcSqYChWCKAvBV25liLJcPJNaw8AkIVT872BcHMOkFsA4CaHif4coi2NYWgsQZWlQpmoIA5MKpkKFwLgQfKkqg3EqmAoVgigLwVduZYiyXDyTWsPAJCFU/O9gXBzDpBbAOAmh4n+HKItjWFoLEGVpUKZqCAOTCqZChcC4EHypKoNxKpgKFYIoC8FXbmWIslw8k1rDwCQhVPzvYFwcw6QWwDgJoeJ/hyiLY1haC8MkyjdP3npaL8G/OH3ht/O+kLcJ601eGxdvMnuTF4zzzv5s9bzpBUSZbXyDlh4mUY5Pztj9nZ/seqEbZqcnrsr7kk1R/rzvP4x6YBweZTAG424EIMrwOpG6B4gyNVSlFPRmEPF2Shn2xEa86YU3eZukx+R6TZxO5ReAKMvHdFCLGJjweIMxGPdCwJte4FGG1+PUPUCUqaEqpaC3ydqkL/FSBjBQI970wpu8TdJjPMpAk3BQsxDlcEHHwITHG4zBGI8yvA6U1gO3h6jjmi028/jbQdqkL/HSJnaAhryRuzd5m6THeJQBJmBSk3iUSQiV+3cMTLl4NsFzaJIRDz+6+XvwNvdYo8w/1qXXhChLh3Rgg94mK0Z8OPrhTS+8ydskPcajHM6c7OilbkR57uStv98LhjumL/zbJngPGJjwSg7GYNxkWwFRhtfvQ3qoG1GOT966UWpt6BR0btPs9IUbm6z8FQx96i69EY83eZvk7aRWqgoKetMLQq8VKEm/LiHK4Q6Gt8mKER+OfnjTC2/yNkmP8SiHMydrHXrFo6xACRK69GYUvcnbJCNeP+09KJE3vfDuUZ4paYukj0i6fpBicDwk+/EQiLJ+pqYpBqZ+yDbPiINxeQh4Jsolkq6TtEfSYxDlxIq2Wpy77rZVrdbcjg41aUGU3dPGG+ng7ZRn+Aa15E0vvMnbJD32FHr9gKQfQ5QQZVYzioHJilj28mCcHbOsNcA4K2LZy3v2KNtvewhRTk1NbWy1undrSmvXrs2O0BBr7N23X+u27Ozo8Ygli7X50pULv+3e87Smtj7QUWb50Udq/fkLDqUeeuxJfXz79zvKnHL8Ul295qSF3+55+EeaufvRjjJnnrxME2efsPDbjp27tP2+xzvKrDn9OK1eeewQUaErEAABEKgegbGxsUMcSDzKCsaFXa/DBZ0v8fB4gzEY90LAm1400qPsNTBs5gmzRjk+OWOXP7/dMD9scevwZ/fPPSPp3bPTEzeENxHFevA2We1tvcnsTV4wLjan0tb2pheeifIsSV/tGphl0XrlIeMFUQYlyiu7AL8KokxrMrKVa4qByfbWwy0NxuHxbgrGnkKvqUYVooQouxXF22TF20k11QsX8qYX3uRtkh67Jko28wxvM49tCLKNQfHHNgTZxiAeEAABEGgKAt4386QaBzxKPEo8ylRTpdRCeDulwtmzMTCuDmPXHmUv2CBKiBKiDG9QwBiM0yDgjdw9b+ZJMx4LZSBKiBIjnmnKlFLYm0Fs0vpZKQMYqBFvegFRBlKEPM16PEcZHQ8ZuOt1fP3MDs3p1+KYzM21rr1j8wV35sGprDreJitGvKyRH9yON73wJm+T9Nh16JXNPPXazGNZgixbUPyxLEGWLYgHBEAABDwgwGaemoxSwz3KVV0e5Wo8yuyK58178CZvk7yd7No1vBre9ILQ6/B0I7EniDIRolILeJusGPFSh79vY970wpu8TdJj16HXXjOAzTzVbeaJ1ijxKEuw896Mojd5m2TES1C3YE140ws8ymCqkL1hPMrsmBWp4W2yYsSLjHb6ut70wpu8TdJj1x4lm3nYzJPeLFISBEAABJIRKGMzz6sk3S/JCPa9kuziwz+X9Fhy98MpQei13qHX8ckZu2TzqLg2tOaeW7Nt80UPhtIQvsRDIXuwXTAG414IeNOLskKv35L0LySdL+k2SQ9L+j+SOtalwqtM/x4gShdEeWoXUa6AKDt1uikGpkpbkNQ3GCchVPzvTcE4a+h1n6Qlkm6UdIykiyTtkvSrxSEtpwWIEqLs1iRvk7VJazvlzOowrXjTC2/yNkmPsxLlzyS9VtJdkv69pL+UZBf4Hh5GlbO3ClFClBBl9nlTtAZGvCiCyfXBOBmjoiXKCr3eLOliSU9LeoWkRyXtlLSyqIBl1YcoIUqIsqzZlL4djHh6rPKWBOO8yKWvVxZRWth1dbQ2aeuT9rxe0jfTixK2JETpnyjPW3fL73VrybNHPLdj+6aLn8qjPRiYPKhlqwPG2fDKUxqM86CWrU5ZRFmrXa8cD/F3PGRq6wPavccCEgef9eev0PKjj1z4Yd2Wndq7b39Hmc2XrtQRSxZn03pKgwAIgEBGBMo4HsKu14yg9yo+ygkHouMhA3e9jk/OWGb1jiMk+w9/dikeZQnKF6gJvJ1AwMaaBePqMM66mYddryWMFUQpiDJBj7wZRW/yGvzeZPYmb5MwzkqU7HqFKOMIXDU7PXFD+4c0uV7xKNMpkDej6E3eJhnxdBpVTSlvelHWGiW7XkvQNzxKPMokNWqKgUl6zyr/Dsbh0W8Kxlk9Sna9lqBbEGUxojyA39OtU7qHYvZD/9rW0A95vE1WvJ0SJlmKJrzphTd5m6THWYnS3v35kl4taU7Sd6IzlSnUcjhFOB7i/3hI0maeNB8acW3DwISfe2AMxk34SC0r9Pqbkv5O0j+PQLFk6G+W9O3wapKuB4gSouzWFIx4urlTpBQYF0EvXV0wTodTkVJlEaWFtp6QdKukRZLeHuV5PauIcHnrco5yNM9R2hlLO2sZf+yMpZ215AEBEACBIgiUcY7y55J+XdL/iwR5QXR7CEnRM4xMmtDheetuOXWutciupIo/D85OT6xo/3DuuttWtVpzOzpKtPAo8SgzKGNJRfF2SgJyQDNgXB3GWdcoLW3dJyR9KrqT8ipJ75B0cvhXSNcDoVeIEqJMN1fKLIURLxPN3m2BcXUYZyXKyyOSbEtsG3ouk3RT+FdI1wNECVGapsx75Iuvt/9+6fJfOesfd//T1+y/Z6cv+JfpNKnaUt6Mojd5bXS9yexN3iZhnJUo7d1fF13U/Jyk7ZLuq9akdPYOUUKUB4lycOi6Tnrr3QvGiIfXJjCuDuO0RGk3hgx6OtfJwr9P3x4gSogSohz+BMSIh8ccjKvDOC1RWoh10JO2neBvClFClBBl8Gl2SAcY8fCYg3F1GKcluKTjH7b+U4t7KSFKiBKiDG9QvIeKm7R+NvzRTt+jN3Iv6xzlIITM60xLvOmRzlgSooQo0xLl/PGa5+yIU8czO33hbRnVrvTiTTEwpQNTYoNgXCKYfZpqCsZlEhtEmVLvOEdZQq7XZw6zOyvjz1Oz0xNL2z+UdQ415ZCWXqwpBqZ0YEpsEIxLBBOiTA3m0ImSzDxk5mlrZ3dmnt17ntbU1s58DcuPPlLrz1/I16CHHkvGL7X2UxAEQKARCJSRmWcQEEMnyl7CEHol9Jot9Do4s1FVMx9vJzzyYAzG3QiwRhleJ1L3QOiV0GuSsmDEkxAq/ncwLo5hUgtNwZg1yqSRDvB3iNIPUZ677la7MeeQ547NF34jgGosNNkUAxMSo6Jtg3FRBJPrNwXjMonyeEl27ValD6FXQq9lhl7HJ2eulPTJLqW+YXZ6wvIcB3uaYmCCAVRCw2BcAogJTTQF46xE+QtJlrqu+7FbRewL+72S/nd4+Pv3AFFClBDl8GegN4NoCHmT2Zu8TcI4K1F+WtI5UY5Xq7tGkt1R+RNJb5L0g+i34c/UqEeIEqKEKIc//TDi4TEH4+owzkqUfy/pLZJ2RyIfK+l2SZa5Z5l9pEk6Kvzr4FEaAtxH2aEHQc5REnpNN5sx4ulwKlIKjIugl65uWbte90j6t5Juibq9SNKfSbLsJn8q6Q8kHZdOpDCl8CjxKIftUY5P3rpRaq3q1Oi5O2enL9yYV8u9GUVv8jYpLJhXx4ZRz5telEWUfyPp3V0A/0dJtuHBvMyPSPpQhgEwj/a66E5Lq39FtNYZb+KFkm6OvNYnJL1P0hf69QFRQpQVEeWGLqLcBFFmsAQVFG2KEa8AutRdNgXjrKFXK/+7kl4bIWV3Uf6XaIPPyyT9MDWC8wXfFm0Asn+fJsku2j2xqw37/UWSviTp1ZI+a3fxQpSEXrt0oLLQa+RRQpRjY1ntSUZzUW7xphjxclEpt7WmYJxXsZdIWhSDdG9OeD8syfKMbYnq3x9dCm2eY/dj/b1B0rWSzoYoIUqIMuesC1DNm0E0CLzJ7E3eJmGclSgvl7Q52rgTn25Z22nX/YSkbZJmox/ukvQuSQ/1mMtGxpYI+62S7rG/k+s1OVfpPQ//SDN3P9oB55knL9PE2Scs/LZj5y5tv+/xjjJrTj9Oq1faXq35x9qwtuKPtWFttZ+Pb//+gfyp8efqNSfplOMXcpUfyL9qeVjjj+VftTys7Wfdlp3au29/R5nNl66U5XO1x/5mZeJPqFyvZeH3/V1P9aSnk46tdO9bT5n4EQRGGYEycr3aMZBvS/qKpGdjYE7nBNbWM78n6cao/nejYybtXbXxZs2jfJWkrZIss3VPL5Y1StYoTWnKuj0kza7XNKHX8ckZS1pga/nx56rZ6Ykbes0db96DN3kNc28ye5O3SRhn9QT/V0RW3Vcc5eRJnSfpmujIyRmSjHBtTTL+2IXQh0v6uqQxSXZJ9EskWZKDQx6IEqKEKPNOx/z1MOL5sUtbE4zTIpW/XFm7Xv9IkhGaeXXPxMTZkVO09q5XC+nuknSJpHslLY8I0TYIvUKSJTowb9KOp3ww2vTTs0uIEqKEKHPOxgLVMOIFwEtZFYxTAlWgWFlEaVdp9XqyeqYFXmVwVYgSooQog02vvg1jxMNjDsbVYZyV4Cx9XXxtsi25hUNr8UCUEKVXohyfnGnv/tYZJx5zyb0/+MlNc3OtW+7YfMGdtZhcA4TAiIcfITCuDuO0RLlakoVX7d+9nryh19LfHKKEKN0S5fqZHZpTR4afubnWaoiydDNxoEFvxONN3iZhnJYoLeRqZQm9ljBnuY/Sz32Uw9z1Og5RljC70jfhjXi8yTuKRGlJzy28av/u9RB6TT8/BVFClJG6dBwPgSgzTKISinojHm/yjiJRdqtlWZl5SlD3ziYIvRJ6bXLodXzyNtv13fG05vbftm3zRQ+WPpkyNIgRzwBWzqJgnBO4DNXK2vVadmaeDK+QrihECVE2myhnLOXjqfHZ0Jp7bgVEmc4+xEt5Ix5v8o6yR1l2Zp7s2p1QA6KEKCHK0qdVYoMY8USIChcA48IQJjZQlkdZdmaeRMGzFoAoIUqIMuusKV4eI14cw6QWwDgJoeJ/L4soy87MU/zNulqAKCFKiLL0aZXYIEY8EaLCBcC4MISJDZRFlGUfD0lzcbNdPXFTdMXWTyW9X9Lt/d4YooQoR50ox9fP/GP3/Ni/5NnXbN90ce8rTBLNR3IBjHgyRkVLgHFRBJPrl0WUZWfmSXNx8ysl2T+fk3SKpM9HSdF7vjVECVGOPFFOztilBR33d+0//NmlEGWnyfBGPN7kNbS9yVyUKENl5slycbPhbonRP8rFzfMT/tx1t61qteY6syK1IEqIEqJM9h2aY8TTvGtVZUaNKENl5slycfPLJX0sutPvEUKvEGWXDjw1Oz2xcEO0x/so0yQcGJ9MPh4ynuBRWsKLI/7puSO659Df/eUl/zevQfVmEJvk7eQds2HU86YXRT3KQZl5jpc0kxP0tBc3W8h3UpKd43yi3dfU1NTGVqu1obvvtWvX5hRnONX27tuvdVt2dnR2xJLF2nzpyoXfdu95WlNb7cjcwWf50Udq/fl2Z/X889BjT+rj27/fUeaU45fq6jUnLfx2z8M/0szdj3aUOfPkZZo4+4SF33bs3KXt9z3eUWbN6cdp9cpjF36zNqyt+GNtWFvtx2QxmeKPyWIyLYzZ1gdk7xZ/7J3s3dqPYWMYxR/DxjCyB/zC49cBPv8DAiOEwNjY2CGpXdPmem3DZOV/W5KRo/23fZVeK+m4nDimubj5xZI+JWlcUqeF6NEpa5SEXk0t8Cj7r1GmSaGYdT578xzs/bzJ7E3eJmGclSivl/TO2CSyz367VPmqrBMrKp/m4mYj4u60XUdL+lmvPiFKiBKiHLxGCVHOWw5vxONN3iZhnJUobXv5myV9WdIrItL8YbTBJidXllsNooQoIUqIMo1V8UY83uQdZaK00KctJu2TZJtrnpFkyZgPLkKl0dCAZSBKiBKiLE6UUej6mu6pOjs9cWmv6YsRD2jUoqbBuDqMs3qUd0dHM+xarddGYtsukLHwr5CuB4gSooQoSyPKzt1k0oOz0xMHd5PFpiRGPJ19KlIKjIugl65u0V2v7V7OlPRNSb8RZcixzTz/QdJ30okRvhRECVFClBBlGkvjjXi8yTvKodefS7LzAIm7T9MoaogyECVECVFClGlsizfi8SbvKBPlJkl2XOOLkn4RU8bO7DBptDRQGYgSooQoh0OUlhlqUUsXGt5nnHjMpff+4Cdb7L+3TV9wWaDpXWqz3ojHm7yjTJRlJ0UvVfGtMYgSooQoh0eUSSkUS5/gJTbojXi8yTvKRPlbkn7ZQ1dtc08tHogSooQoIco0xsgb8XiTd5SJ0tLITceU0HKK/UnXb2l0NFgZiBKihCghyjQGxhvxeJN3VInyYkk3S3p7TAlfJmm9pMPTKOYwykCUECVEWR+iHJ+cuVLSq7vn/uz0xB8Owx4M6sMb8XiTd1SJ8tuSXtND8e6KLlWuWu8P9A9RQpQQZe2I8pNdxuGG2emJvGkvS7Mz3ojHm7yjSpT23pau7o0xTbX1SrvN47nStLdgQxAlRAlRQpRpzIg34vEm7ygTZRr9q7QMRAlRQpQQZRoj5I14vMkLUabRworKQJQQJUTpiyjHJ2/dKLX+uNNkzP272ekLN4Y0I96Ix5u8EGVI7S3YNkQJUUKULlHVI/kAABeySURBVImy6wL2uU0QZacxhCgLkkOK6mXlek3RVbVFIEqIEqKEKNNYIW/E401ePMo0WlhRGYgSooQom0eU566bmehlUu7YPDGT19R4Ix5v8kKUeTVzCPUgSogSomweUY5PztgREzuTGX+ump2euCGvWfFGPN7khSjzauYQ6kGUECVECVGmMTXeiMebvBBlGi2sqAxECVFClBBlGvPjjXi8yQtRptHCispAlBAlRDmaRDm+fmaH5vTSuOmZm2v94R2bL7izlznyRjze5IUoKyLBNN1ClBAlRDnSRLmqiyhXQ5RpLGeYMt7IneMhYfQgV6trNnzmqMXPHPZkV+WnZqcnlrZ/O2/dLafOtRY90FXmwdnpiRXt3+zi3KT7AKOk1ANzbUYHvgeeY0uzmSL6oh9oqMYnZ+ydTo2/V2vuuRXbNl/0YPu38UkMfRuLublWh6EHv44Z0bGZJ43+xWs3xYjnMkJDqtQUjFtDwmto3eBR4lHiUfKh0e9DY8OGDYvaf3v72y/Zf/PNN9lVgdq0aVNt8lX3M5beSIfQa3m0Z0R9naTLJO2WdIWkb/Ro/kxJWyR9RNL1g7qHKCFKiBKiLOKRl2feym0JoiwXz16t1TX0+jZJ75Vk/z4tIsETu15gSUSmeyQ9BlESeiV0Teg6shGZQ69pQtfhzXG+HiDKfLhlqVVXovywJFuzMm/Rnvsl2RqXXd3V/XxA0o8hSogSooQoIcos5r+6st7Iva5E+QlJ2yTNRkNpl0C/S9JDaYhyampqY6vV6tqEIq1du7Y6zUjR8959+7Vuy86OkkcsWazNl65c+G33nqc1tbVzL8/yo4/U+vMX9vLoocee1Me3f7+jnVOOX6qr15y08Ns9D/9IM3c/2lHmzJOXaeLsExZ+27Fzl7bf93hHmTWnH6fVK49d+M3asLbij7VhbbUfk8Vkij8mi8nUfuyd7N3ij72TvVv7MWwMo/hj2BhG9oAf+JkehNK/L3/XVoE6H9Pztv6lmOIUcYzA2NjYIXt3qt7M8yFJ35N0Y4TrdyW9KVqv7IYaj1Ji16skPEo8ysg4BAm9Ju26rooDvHlnhpM3mevqUZ4n6RpJb5F0hqTpaK2yly5ClBDlU6YYECVECVFWRdfZ+oUos+HVr3R71+vlknZJukTSvZKWS/qapJdJOkvSV7sasHifrVce8rDrlV2vphScQ9VR8cmx//Bnl27fdDEfGiWd4120d/HBNZAI6Ds2X9hrx345ltKhd4ZHWdrQl98QRAlRQpQcD2lblqoSNpRv2fyFMSHKEFqQo00287CZp602bIZiM1SdNpPlMGdUqQkCddzMUzo0eJR4lHiUeJRVe5QHQv9adFi3gZvdPPGdvEbP23ofHmXekR5CPYgSooQoIcpaEGVCruas5hCizIpY9vJ13fWa/U0SakCUECVECVFClKWb1lwNeiN3iDLXMIepxPEGjjdEmlXJOUD0rx76l9W6eCMdQq9ZRzhQeTbzsJmHzTw6kK2JzEY6kKXKW2aom7/yPzus4/HHPL8js1Yg00mzAxBgM09N1IMv+np80XOfZ8eE4D7UCI6kzDzDnL/xEcKjDG/ACb2Gxzh1D8OcaFzc3DEspAAks5Gri8MhytRmtZSCEGUpMJbTCESJR8ka5cRV7dk0PnnrRqn7coO5TbPTF248WGbmk5Ku7JqBo7LG+9f23ksOW/T8fc8+9wtJt8xOH8SvHKsUphVvXjBEGUYPcrUKUUKUECVEWVboP5cRGlIliHJIQA/qhs08bOZhMw+bedo64HEzTxnX5NXAFDdKBDbz1GQ48SjxKPEo8SjL8ijPWzezppdp27Z5YnvVJg+PsuoR6NM/CQdIOGCqwe0h3B5ielBVUvS66V9V5hqirAr5hH4hSogSoiQzT9tMQJTVGmqIslr8+/YOUUKUECVECVEeumt4TupI0t6Svj47PXFDSFMOUYZEt0DbECVECVFClBBl9uM1Bcxu36oQZQhUS2gTooQoIUqIEqKEKPPQSSPPUXI8hOMh7cnAxc1c3Dyqx0N27Nyl7fc93sELa04/TqtXHrvw28zdj+qeh3/UUWbi7BN05snLFn4z/L6/66mOMu9cdaJOOX5pHs5xW4fjITUZOo6HcDwkUsVRySxjWXXizw3xzDJk5olB06p3RCirCSX0mhWxIZUn9FrviTY+OeMq1ya5cjsmLrlyRzxXblYzDlFmRWxI5SFKiJI1StYoWaPMvkY5vn5mh+a0Km6q+xyvOSFepjX33Ou2bb7owV4mHqIcEvFl7QaihCghSogSogxKlKd2EeUKiDIrU1VcHqKEKCFKiBKirI4o37ju9oXdP3992Wt+9p4b/+EFzzti39z2TRd37hSqmCuyeMCtGspaSCSIEqKEKCFKiLI6oky6+LqQgQ9cuZHHQ3phBlFClBAlRAlRQpR5OLWRRMk5Ss5RticD5yg5R8k5yoPUkPcc5UOPPdnBL1evOanjHOXU1ge0e8/THWXWn79Cy4/Opn/f+sGPD+Gxs099YR5uK70O5yhLhzRfg5yj5BxlpDmco5TEOcqYHan5OcphHe/KZ1mL12qkR0no9bZVrdbcjg4cmGgH4KjbNUfjkzN26P7KLp2FKCFK5u/8pHhqdnpiYRPQ/Pxtnd9t42enL9xUnA77twBRhkQ3Y9t4lHiUeJRc3FzWxc2j6pFnNLupikOUqWAaTiGIEqKEKCFKiLLD3vbwKBdZFq/405EZKoS1hihDoJqzTYgSooQoIUqIEqLMSSHFq3E8hOMhrFFyPKRtSfqkYBuYWSbpHCAfurX60F1z6DrmxHvzMgkeZV7kAtRjotVqorGZou9misGhLzyiYh4R+IXHL6v5hiizIhawPEQJURJ6JfQKUUKUAWlmcNOEXgm9Enol9Erotb6ZeSpwFP6qizVujd+HGv9bIz1KMvOQmaet5GTmyZYZZd2WnR22A/zAb1QzG3W7XmTmqczP7ey4gi8qbpifHwIuHh7xi4eHlVmGhBc6Km719h/+7NL27SF1s3+N9yh78R6hV0KvhF4JvRJ6JfRqOjA+OWPZsAY6ChBlTTzIbjHq9kU1qpk92EwRfjNFGkOF/sXGgRSUB8AYpkcOUUKUqb6oMFQYqt65Njke0taMOocOmb/F5i9ECVFClJ06wBola5SWFo2EA/PzgqT8XRzRyF2vrFFye0hMBzLniiT0SujVEGjNPbdi2+aLHmyjQWaeg3oxCpmN8CjxKPEo8SgPIEDocF4R2PXaMSHwKPEoa8qShL4IfR1UTQwV91GSQnF+PlQWEaqTR9mSdJ2kyyTtlnSFpG90UVm/Mi+UdLOksyQ9Iel9kr7QjwY5HsLxENONYe6aYzNFbDaya/MAGOgf5yjzuGpvk2QZ3e3fp0m6XtKJXQ31K2PlXyTpS5JeLemzkl4KUUqssbHGxhrbxA0La4vrZ3ZoTqviWjEKa2x8qBX7UKuTR/lhSRZy2xIJdb90QKHNQ2w/SWUWSXqDpGslnQ1RQpRdOlBZ6AZDVcxQgR/4EXqd14FPSNomaTZSibskvUvSQzEVSSqzV9KTkt4q6R6rR65Xcr229YdcpeQqHdVcpTt27tL2+x7v+G5cc/pxWr3y2IXfZu5+VPc8/KOOMhNnn6AzT1628Nuo4tftdFWZ6/VDkr4n6cZIqO9KelO0XtmWM6mMeZSvkrRV0gpJRpyHPKxRskZpSsEakZ81IjxKPEo8ynkdOE/SNZLeIukMSdPRWmWc6PqVeb2kwyV9XdKYpK9Jeomkn0OUnKOM6QCh1wgMzgEe1ArWKOexGJ+csXynlvc0/rDruguQqhMOtHe0Xi5pl6RLJN0raXlEfC+zc7/RztjuMq+Q9OnIm9wj6YPRZqBePCk8SjxKPEqSoreNA0QJUZIUndDrqlZrjnNY83qAR4lHya7XBZvI7SGRJ83tIYReCb0SeiUFW1sHxjkeEpsOECVE2TPoOv8joVdCr4ReCb0SeoUoIUqI8gACJBzoUARCr4ReCb0Ser1zdmpi9UKUgYube7MlHiUeJR4lHiUeJR4lHiUeJR7lM4dZ8on4g0eJR4lHiUeJRzmAHxf+hEeJR4lHiUeJR4lHiUeJR4lHiUe5MAu4eHgeCna9xg0jRAlRQpQQJUQJUUpkluE+z8LnyON0UnVmnjRR01LKEHol9EroldAroVc8So8eZZGLm5dKuim6Yuunkt4v6fZ+rApRQpQQJUQJUUKUHomyyMXNr5Rk/3xO0imSPh8lRe/JlRAlRAlRQpQQJUTpkSiTLmW2d0pTxq7Z+igXN8+bARIOdHwrcTwkgoPbQw7qBUnRo41O3B4SNxY3zE5PXNXL06p6jTLpUmaTOanMyyV9LLoq5hFCrxBllw5AlBAl5ygXJgUepUePMulSZnunQWXOkTQpya7geqKtC1NTUxtbrdaGbtJcu3ZtKRuDQjWyd99+rduys6P5I5Ys1uZLVy78tnvP05ra+kBHmeVHH6n159ud1fPPQ489KbuVPP6ccvxSXb3mpIWf7FZzu908/tit5na7efvhhvSD6IDfPBbo3/6OOWNz0+aoPczfpzuwMZtktqn9mG0zjOJPnfHrtvNjY2O2p6bjOeSH7gIl/X+Ri5tfLOlTtgtaUucI9RCONUrWKE0tzlt3y6lzrUWdXxrSg7PTEwtfGoSuOyYQHnkEB6Hrg3oxCqHr+CyoOvRa5OLma6PLmuPvc7Skn/UicYgSooQo2czTtg2jYOjHJ2/dKHVH1gi9mg6MkxS9t68LUUKUECVECVFClBDlgHgwRAlRQpQQJUQJUUKUEOUBBFhjY43NECDX67wekOs1Ph8gSogSooQoyfW6MAsgSoiSNcpoOrSyR9TqtJlnALWV+ydCr9kVhc0AMR3MMdHAD/wksWs4UgNvu4YhynI5uLTW1mz4zFGL8YjwiLj9Yj5kyq7Ng7aFD7UDWAzzeBdEWRq1ldsQRKlT44gSOiR0SOgwf+iQD41iEQ2Islx+K601iBKijJSJ+xTxKAvfpwhRQpSZyYk1StYohx26wVAVM1TgB35VrvHiUWam2eFUwKPEo8SjPHh7A0QJUUKUw+GehV7wKPEo8ShJONA2CKSwi9a/uWYrzkS1vWZraHQJUUKUECVECVGScMB0gFyvfagXooQoIUqIEqKEKCHKAf4pRAlRQpQQJUQJUUKUEOUBBMj12qEIZEaJ4PCWGYXNPGzmYTPP0FYn5zvCo8SjxKPEo8SjxKPEo8SjxKMkBeDCLCCzUbSzc/3MDs1pVdw8sOuVXa+SPtlFGex6xaPEo8SjxKPEo8SjxKPEo8SjxKPEoySp/AEdYI232BpvnE4eeeSRubGxsVY3xRzyw5CXFEvvDo8SjxKPEo8SjxKPEo8SjxKPEo8SjxKPEo+yNVc4qTweZen+ajkNkuuVXK+RJnF7CKHDwoae0Cuh18zMROiV0CuhV0KvhF4JvRJ6JfRK6JXQK6FXQq+EXhsWerVNQ9dJukzSbklXSPpGF98NKnOmpC2SPiLp+kFuJh4lHiUeJR4lHiUepUeP8m2S3ivJ/n1aRHYndhFevzJLIpLdI+kxiHJiRRs3Uth1aBAp7CI4SGF3UC9IODCPxTjXbMWNRW0TDnxY0gORV2gC3y8dyKDxREz6pDIfkPRjiBKiZDMUm6HYDMXF12U5CnXa9foJSdskzUZC3SXpXZIeigmZVOYQopyamtrYarU2xF/0ec97nn75y19m3gREBRAAARAAgdFGYNmyZXrHO95RWcKBD0n6nqQbo2H4rqQ3ReuV7ZFJKtMYjzKuih7WVLunjjeZvclreHuT2Zu8YDwcQvSmF/3kHVZmnvMkXSPpLZLOMDsQrVXGRyupDEQ5HN1O7KUpyp/4ohUWAOPw4IMxGKd1AoZFlO0drZdL2iXpEkn3Slou6WuSXiapX5mzJH2164WWReuVh4w0yo/yp1X+8Ejl7wE9zo9d2ppgnBap/OWagvGwiDI/0hlrNmVgMr72UIuDcXi4wRiM+eALrwNpMW4cUdoGn/Xr128cPsT5evQmr72lN5m9yQvG+eZS1lre9MKbvE3S48YRZdbJQnkQAAEQAAEQGIQARIl+gAAIgAAIgMAABCBK1AMEQAAEQAAERpwo7TjKt2IY/HdJb6yBVqTJf1sDMRdEqCuOvTDqlRu4znj3krfOeC+VdJOkN0j6qaT3S7o9tnN9UE7nKnS6n7x1xviFkm6WZLv+LYPZ+yR9ocYY95O3Thg/P0pyY8luLPlNapswCh7l70h6s6R3VzFDB/SZJv9tnUSuK47dGPXLDVxXvPvJW2e8XynJ/vmcpFMkfV7SS6Jczkk5navQ6X7y1hljy4n9IklfkvRqSZ+V9NIaY9xP3jph/OeSXh/lDjeiTG0TRoEoL5D0Z5KOkfRPEWF+sYrZ2tVnUm7bGojYIUJdceyHU3eCirrj3S2vF7xfJemjks6WVHeMTVfi8nrAeFHkuV/rBONueeuC8UmS/kjSz6Oz+0aUqfV1FIhypST7ZybKBvSfJB0v6dmKmSgpt23F4h3SfV1xTEuUdce7myg94P1ySR+TdKWkRyTVHeNueT1gvFfSk5LeKukeBxh3y1sXjM0jN6L8kxhRptbXphLlX0l6j6THI1KMG9NvS/rdrptLqiClpNy2VciUpc+64JiWKOuOd1KKxrrhfY6kSUmWbat9C1CdMe4lb7fu1A1jk888NPOCt0qyK/Y+mCJvdpZ5XHbZbnmNOKu2v+dL+lVJn47Sp1o2OPMoU+trU4kyPjAWk36BJNvEY5smbBOCxfqfK1tDMraXlNs2Y3PBi9cVx7REWXe8u4myzni/WNKn7JpDSU/HBqCuGPeTt84Ym2yHS/q6pLHIC7J14N9KkTc7uDHo0UE/eY3cq7a/X5ZkH0rxxy7lODItlqNAlKZctnvsdEmPSvoDSXbNV9VPv9y2VcvVr/+64tgtb7/cwD+JFvG78w1XjXc/ee0LuI56a3jZepl5NvHn6ChEeF3kZcZzOleNcT95Tea6YvyKyAMyb9IurTe8rx+QE7tqjPvJWze7YRdytD3K1DZ4FIiyagWifxAAARAAAccIQJSOBw/RQQAEQAAEwiMAUYbHmB5AAARAAAQcIwBROh48RAcBEAABEAiPAEQZHmN6AAEQAAEQcIwAROl48BAdBEAABEAgPAIQZXiM6QEEQAAEQMAxAhCl48FDdBAAARAAgfAIQJThMaYHEPCIQDsRwhpJOzy+ADKDQFkIQJRlIUk7INAsBCDKZo0nb1MAAYiyAHhUBYGKELBLfP84ylVpl/leLek3o3zGfyHp9yT9iqQ/lXRjJOPbJa2X9BuSHpBk90b+ffQ3u6vVLl/+NUn/TdIVUQLur0raIOkdUXtWpt1eRa9OtyAwfAQgyuFjTo8gUASB344I8Q2SfijpXklTkv6HJCO22yT9fpQndLWkX48uV/5mlAD6b6P8pq+LLgewS4G/IenfRDcqfCdKeG53tlp7t0T5kf+zpFMl2U32PCAwUghAlCM13LxsAxAwj9E8w/jzOUl2tZwR21skGcldJMnu4DNCNMLcJMmSgP9M0oSkWyMv1AjXLjZfKumpWKPt0KvdAv95SXa7iZWzGy32NQBHXgEEUiMAUaaGioIgUAsE2jdh9CO2NlFeHHmOZ0iyDTl2+4SFVu0mijaJ2tVIRqIbo6uQ7ILg9tO9Rml3T5rnalcTdd8xWAtgEAIEQiEAUYZClnZBIAwC/0rSV6Lb2s1ztPsKzcu8P/IozVN8Z3TvqnmLFnq1K5BsPdJud/9k5E1ayPVESa+VdLekd0m6Q9K3JNm6p10/ZR5qe9crRBlmPGnVAQIQpYNBQkQQ6ELgmogo/1kUFjViNM/RiO1votDqs9EGHVtjtMfu4TSyO07SP0h6j6T7or/Z5iDb3GOhWbur9UpJJ0OU6B0IzCMAUaIJINAMBNqhUgul3tmMV+ItQKAeCECU9RgHpACBoghw7rEogtQHgT4IQJSoBgg0AwGIshnjyFvUEAGIsoaDgkggAAIgAAL1QQCirM9YIAkIgAAIgEANEYAoazgoiAQCIAACIFAfBCDK+owFkoAACIAACNQQAYiyhoOCSCAAAiAAAvVB4P8D+77+ovte0PAAAAAASUVORK5CYII=",
"text/plain": [
"<VegaLite 2 object>\n",
"\n",
"If you see this message, it means the renderer has not been properly enabled\n",
"for the frontend that you are using. For more information, see\n",
"https://altair-viz.github.io/user_guide/troubleshooting.html\n"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import altair as alt\n",
"\n",
"df_loss = pd.DataFrame(enumerate(training_losses), columns=[\"epoch\", \"training_loss\"])\n",
"alt.Chart(df_loss).mark_bar().encode(alt.X(\"epoch\"), alt.Y(\"training_loss\", scale=alt.Scale(type=\"log\")))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And because we're paranoid types, let's check a prediction."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[ 4.0063, -7.1900, -6.8099, -5.7869, -6.4321, -3.4320],\n",
" [ 4.9544, -6.3152, -7.6040, -6.6827, -7.8661, -5.0460]])"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with torch.no_grad():\n",
" logits = model.forward(**next(iter(dataloader)))\n",
"logits"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The positive sample gets a positive score and the negatives get negative scores. Super."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We should be able get the paragraph vectors for the documents and do things like check these for similarity to one another."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>doc_id</th>\n",
" <th>similarity</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1</td>\n",
" <td>1.000000</td>\n",
" <td>It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>0.177416</td>\n",
" <td>In the week before their departure to Arrakis, when all the final scurrying about had reached a ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>3</td>\n",
" <td>0.081760</td>\n",
" <td>By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2</td>\n",
" <td>-0.044768</td>\n",
" <td>The old woman was let in by the side door down the vaulted passage by Paul's room and she was al...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" doc_id similarity \\\n",
"1 1 1.000000 \n",
"0 0 0.177416 \n",
"3 3 0.081760 \n",
"2 2 -0.044768 \n",
"\n",
" text \n",
"1 It was a warm night at Castle Caladan, and the ancient pile of stone that had served the Atreide... \n",
"0 In the week before their departure to Arrakis, when all the final scurrying about had reached a ... \n",
"3 By the half-light of a suspensor lamp, dimmed and hanging near the floor, the awakened boy could... \n",
"2 The old woman was let in by the side door down the vaulted passage by Paul's room and she was al... "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.preprocessing import normalize\n",
"\n",
"def most_similar(paragraph_matrix, docs_df, index, n=None):\n",
" pm = normalize(paragraph_matrix, norm=\"l2\") # in a smarter implementation we would cache this somewhere\n",
" sims = np.dot(pm, pm[index,:])\n",
" df = pd.DataFrame(enumerate(sims), columns=[\"doc_id\", \"similarity\"])\n",
" n = n if n is not None else len(sims)\n",
" return df.merge(docs_df[[\"text\"]].reset_index(drop=True), left_index=True, right_index=True).sort_values(by=\"similarity\", ascending=False)[:n]\n",
"\n",
"most_similar(model.paragraph_matrix.data, example_df, 1, n=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's not particularly illuminating for our tiny set of dummy data though. We can also use PCA to reduce our n-dimensional paragraph vectors to 2 dimensions and see if they are clustered nicely."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2-component PCA, explains 48.18% of variance\n"
]
},
{
"data": {
"application/vnd.vegalite.v2+json": {
"$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
"config": {
"view": {
"height": 300,
"width": 400
}
},
"data": {
"name": "data-0b1e0474027b1da095c3ed8d1f5f386e"
},
"datasets": {
"data-0b1e0474027b1da095c3ed8d1f5f386e": [
{
"group": "0",
"x": -4.767539597377173,
"y": -4.756888669464686
},
{
"group": "1",
"x": -3.202457816084362,
"y": 6.363203538611249
},
{
"group": "2",
"x": 6.650774519184007,
"y": -0.034075790016020964
},
{
"group": "3",
"x": 1.3192228942775253,
"y": -1.572239079130542
}
]
},
"encoding": {
"color": {
"field": "group",
"type": "nominal"
},
"x": {
"field": "x",
"type": "quantitative"
},
"y": {
"field": "y",
"type": "quantitative"
}
},
"mark": "point"
},
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAfwAAAFZCAYAAAB9g51OAAAgAElEQVR4Xu2dC5RdVZnn//tWeBiwfXRjsHmksRCVbnSle2xsF4M4tEIgCYgWCciAoEtcOgpCgEqQR6NQxSug2LToNAIuQyoKYsLDHh/YIz7QaekR2wEGpEEeAW3FHl4tqXtm7eRWrNzcx77n3HPvP7d+tRaLBbX3Od/5/7/v++297711g/hBARRAARRAARQYeAXCwD8hD4gCKIACKIACKCCATxKgAAqgAAqgwAxQAODPAJN5RBRAARRAARQA+OQACqAACqAACswABQYO+GvXrs323nvvGWAdj4gCKIACKNBtBYaHhweOi1MaDdyDjY+PZ6Ojo319rgceeCBzSBqHOIjh9+0ILTZq4aCDSxwOWjjEgB/dXrY0vl5fwVjGIwJ8AFOfVzQ0cqJRr3HIC2IgN8vgYLNrAvwS1HYoYlbMXo0EP/CDRUfrZuvQN7sYQ2TrpyQdI+k+ST+TtFjSbElvlfQtSedJOlXSXpIOlLRM0m6SfirpZEl3Sjpc0lckzZP0z5Iul/Q+STtK+i+SvinpAklHStpB0nJJ1wD8EsDe7JJdTJpCUTvEQQxeoMMP/OAErHlb7WJ9HCLplhq4V0v6nqSdaqDeT9J3JH1B0gck7SPp+5JOl/TZ2v/fV9KrJL29BfCnrjMh6f2SPi/pYEmvlPTvjZ6SHX4hpDae3MWkKRSdQxzEAGAATE8Ak7tXONToAJ6AnSPpXEl/JOnfJF0p6dg64L9T0o2Szpb0N5JeJukpSUskXS/pzZLmJAD/MElrJB0t6YuS4mLhhwA/d0l0NpECAnIc33pDbgAB01mTmjaaflVKv0oB/nxJX5N0Vu14/+WSfjMN3G+StLOkmyT9J0n/JOkqSe+uWzhMAT++fBBPDd4o6X8B/Nwl0dlECqiUAurMBBpaQ70cctMhBoDvVaMD6MfC2q77I5K+3ORIfwr4EdDx9frTJH2utrt/g6RX14774+/i6/sR5t+uHdnH1/CnjvTjacCJkq6rvRcgHuk/A/BzI6OziTQ0r2aCH/jBiQsnLildvIu9YqgG73hsH9+E96ik+Lr+dFBPAT+GdrykUUm71N6cd1JtRx9/F0F+hKS7aguDD9be/DcF/CtqLwOsl7RU0spmz8pr+ClZ0OGYLiZNh3fefLhDHMQAbOuT2CEnBnBHmbtX4EdPavTvJL2jdkSf26u6iVPAj2/U+4eUiwL8FJU6HEMB9aSAkl3BD/xgh88OP6VhdLFXxOP4/1n7aF48po/vwv+X2sfsUkJJGTMF/OknBS3nAfwUWTsc08Wk6fDO7PBdGzs7Sq9FB37gRw96RXzdPR7Nx8/H31F7nf3hQk1988kAf6b8pb1nx+YcFar6k+h/NejJUKl8ffbo45slk8PCgxi8Git+4Ec9cBxyggVYF5cBLS7FDr8EncsuoGc+sfPaLcIOejgMhQunQ7/sOFKkIwYAA2C8j9MdahTgp3TT4mMAfnENt7hCmQUUd/bZZDg6DGUrq1U9Wc2y54ZC5UBl+svJrPr3f3DWk/Ezmxt+yowjVTZiAPgAH+Cn9At6RYpKxcYA/GL6NZxdZuI+e8HOS7Oq3hKUnT/7Y0/8IAawaRFQ0T/OXr7uEoC/uS1l+tFJ+jjEQQwswFiAeS/AOukpnY4F+J0qljC+zKb63PlzllWz8OahoXDF9sse/x8xnOfO3/nYaqaRUAm3z17++AqAD/CbpWmZuZlQGjYnT5yAeS18BtmP+aetGp5V0V7Vil4Tn7NS1b3rq7rvtouXPJBaM90aB/C7peS065TZVJ+5YM57VQ2HK4S7K8ruCZWwbrKqA5Rl+yjoyh3OXHcbwAf4AL99YZdZp+3vvnEEMXgtPLrtx8JlqxZkWThByn4nhSc2Pm02RwrbhpBdvXZsyc2pudKNcQC/GyrWXaPbSTP98s+Ov3L3bDI7Q5l2n/7/Q9Ads89cd+H0/1dmHKmyEcNgN7TUPHDLS2DrlZeD6Mei0evnVRXOCwp3rB1fvFlvXjg6cUambL+KsrPXjB8V/4Je/U9k82WS3iNpnaQTan+eN0/JbZoD8AvJ13hy2ZCL0F//wuSfzxqq7DmZZf8RpJ/teOYT8XuRN/spO44U6YjBq7HiB3449olBA/4hZ94wt7J+/SUh6Ma144vj37rf4mfh6MRRWaYjqrNmLb31/Hc+VDcg/lW+k2t/nW+epM/U/rZ+StttOgbgF5KvP8BPDZnmvlEpBx1c4nDQwiEG/PBa+AyaHwtPX3l4VqksuXl8Sfyq26Y/C0ZXrQrV6qq1Fx296dNVtcEX1/4G/7W1//6JpIMkPZ7a/xuNA/hF1Gsyl4bm1UzwAz8alapDXhDDYObmwmUTS7Msm3vz+JIPtwH+FSGEh9aOLd706ara+Pg1uPHvrUy9xv8tSfFLc+4pgqytAfjHSTq/9u1A8TuG4zcDNf2ZKX9pL8V0mgk7/Po8IScGEzAp/aDZGIecGLQd/qHLVp0csrDHzeOL45/WbbHDn/hkFrIHbxlbcnndoIsk/UzSNbX/f7ekt9Vez89ttzvwd5b0dUnxu4Xj9/u+qvb1gAA/wXKHQiYGAOO46Bg0wCS0g6ZDHGp00PxYeMaq+VnQcUlH+pmuXXvhkk2frqoZFZl3uqTDJL1R0rik+Fp+oR934B8j6VBJb5K0vaR4PPJldvhpnjsUMjEAfIDfvF6pj8Gsj/jZ+6GhcHGQ7qx/h/7UE298p772nZzMTmvwmfypd+kfL+kxScdK+lFa528+yh34SyUtkPQuSXtIim9s2C1+X0x8pLGxsXNDCPGYf7OfkZGRorowHwVQAAVQYAYqMDw83BUubvwMvk4MCivr36m/4R36yo4OQVf18rP4XXmwEnPiREm7Sjqrdo8fSzpY0pPN7slr+IO5Yi6SYw67qEE7ssSPIgpQo43Uc6jTbsew4bP4WWW5QjYp6Ze1595JWRiqhOoFTT6D353kanAVd+DvLekGSX8taY6kr9R2+ht2+I1+AD7NpD4vul3EeavRIQ5ioD6oj+YVXEZ9xM/kD73wH/M0NLTnhjtPTt4/uc12dzX47H3e1pI8zx348UHisX5888Jzkj407WMKAL+NzWUkb3Jm1QYSA4ABML0FzNZYo5yAdepavvFbA/A7ejJ2+AAGwACYlKbBYnSjSg46uMThokVK/uYZA/DzqLYV7KwpIK+FD37gR6O24QAYhxiojxJA1OCSAL8EnSkgr+aOH/gBbL1PfQB+CSAC+L0RFcAAGAADYFK6jUOvcIhhkIH/1MdfMbxNpbJXloXXxOcMIbv3hWr1vpee9eQDKTnSzTHs8LupZu1aFBDAB/gAP6W1OPQKhxgGFfjPfHzOAlV0QhbC7yoKT8TnrCqbE7JsW1V19Q5nPTH1t/JT0qXwGIBfWMItL0ABAXyAD/BTWotDr3CIYRCB/8zYH8/TZPW8EHTH7DPXXTg9H549f+czskz7aahy9g7LHrurSa78laT4bXkral+Pm5JSLccA/MISAvxWEjo0E4cYBrGh5S0d/GBBPOgL4uc+8cq51ZBdEirZjbOXPXF9o+d9dmzOUVk1HFHJwtIXfezxh+rGbCvpMkm/kfQIwG/SbfhYHs2kPjUADDkx6IDJu/hyWYi6xNGtXvHvH3/F4bOGKktmL1+3pJU3z16w86r1k9VVf3DWk/HPxjf6+ZikXwF8gN+2xruVvG1v1GIAMQBbFmDNC4T6GMz6ePaCneMfjJs7e/m6+IVvTX+evWDn+HXvD81evu4SgJ+DNOzwB7OAcqTCpikOTXXQdjD4UUQBanTQT1ye/sQrT65UtMfs5Y+f1Br4r/xktaoHd/zY45cD/Bw1BfBpJuwo2VGmtA6HhSAxDGa/eub8nednQcft2OZI/+kLdl4VMl27w5nrbgP4KVVbNwbgD2YB5UgFdvgNRAMw1AcL4vIXxBs/ex8uDgp31r9Df+ruG96pr2zfF6rZaQ0+k7+fpO/URbpT7fX83O2Qd+nnlq78pCkaGs19o4IOOrjE4aCFQwz44bXwGUQ/NnwGP4QTw1C2sv6d+hveoT8ZjlaWXdXLz+ID/KJUNd1FDWIB5bUKwHg1d/zAj0a17JAX3Y5hw2fxq9XlIWhS0i9rz71TlmlIlcoFLT6Dn7fdtZwH8EuQtdtJkzdEhziIgebOEbL3aaBDjQ7yBiV+Jv+FbHLerKHKnvE5109W798mDN3V4LP3eVt98jyAnyxV+kAKCMjNlB1MelV45cQgA6ZTT+hXfrnZqYep4wF+qlIdjKOAvAoIP/CDBZj3KQMLsA4AU2AowC8gXrOpAAbAABgAk9JaHHqFQwwAPyVbio8B+MU13OIKFBDAB/gAP6W1OPQKhxgAfkq2FB8D8ItrCPBbaOjQTBxioKF5LQLxAz96tSh/ePHi4SDtFaTXxHtm0r2ZdN/uExMPlICflpcE+CUoDmC8mgl+4Eevmnun7cQhNx1iGNQF2COLFy8IWXZCFsLvQsie2AD8LMwJWbZtFsLVu05M3NxpzhQZD/CLqNdkLgUEYAAMR/oprcWhVzjEMIjAf3TJknlZNnleCOGOXVatvnB6Pjy65MgzsizbL4Shs3dZtequBrnyEknXSTpQ0q8lxS/jWZ2SU63GAPyiCjaYTwEBfIAP8FNai0OvcIhh0ID/2NFHz62uX39JGNKNu1y/+vpGufDoUUcelU3qiMqsWUv/eOXKh+rG7CMp/nODpNdKil+fu0dKTgH8oip1OJ8CAvgAH+CntA2HXuEQw6AB/+Ejjzx8KIQlu0xMLGmVB48uXrxqMstW7b56dQR6s5/XS/q0pP1TcgrgF1Wpw/kUEMAH+AA/pW049AqHGAYN+I8uOXKpsjB3l4mJD7cB/hUK2UO7rFp9SZNxr5N0paT3xa8FSckpgF9UpQ7nU0AAH+AD/JS24dArHGIYNOA/snjxySFke+yyavVJLYG/5MhPZll4cNeJicsbjDtA0qik4yU9npJP7cbwGn47hXL8ngIC+AAf4Ke0Dode4RDDoAH/F4sXzw/Scbu2OdJ/ZPHiVZl07W4TE7fV5ctcSVdLWiDpuZRcShkD8FNU6nAMBQTwAT7AT2kbDr3CIYZBA3787P1Qll2siu6sf4f+VF7Ed+qrqn0nQzitwWfyz5J0Xl0OvUzSUyl51WwMwC+iXpO5FBDAB/gAP6W1OPQKhxgGDfjxeeJn8KXqiaESVta/U3/DO/Sr2dFS5apefhYf4KdUZYdjKCCAD/ABfkrbcOgVDjEMIvDjM234LH61ujyEMCnpl7Wc2CnLsqFQqVzQ5DP4KamTawzAzyVb60kUEMAH+AA/pbU49AqHGAYV+PG54mfy169fP2+ooj3jf09Wdf+sWbPuavDZ+5SUKTQG4BeSr/FkCgjgA3yAn9JaHHqFQwyDDPyUPOjVGIBfgtIUEMAH+AA/pbU49AqHGAB+SrYUHwPwi2u4xRUoIIAP8AF+Smtx6BUOMQD8lGwpPgbgF9cQ4LfQ0KGZOMRAQ/NaBOIHfrguyktA0qZLAvwS1AUwXs0EP/DDtbk75KZDDIO8ADvt1q8MV4eG9som9Zr4nGFI91YmJ++7+JB3FP5TuZ3iC+B3qljCeAoIwAAYjvQTWoUceoVDDIMK/FNuWbNAlewEZeF3CuGJDTmRZXMUsm1VDVevOHTRzSl50q0xAL9bSk67DgUE8AE+wE9pLQ69wiGGQQT+Kbd+dZ4UzlNFd6w4eNGF0/PhlK+tiX9lbz8pO3vFIYfd1SBX5kj6ghTHbPg7+qdI+mpKTrUaA/CLKthgPgUE8AE+wE9pLQ69wiGGQQP+qbfcMrdaWX9JReHGS+cvur5RLpx625qjqsqOqFRnLb300EMfqhszT9Lukr4u6Q2SvijpVSk5BfCLqtThfAoI4AN8gJ/SNhx6hUMMAwf8W9cenlW0ZMXBC5e0yoNTvrZ2Vahq1aWHLLypybiKpAMlxb+tv39KTgH8oip1OJ8CAvgAH+CntA2HXuEQw6AB/5Rb1yxVJcxdcfDCD7cB/hWqZg+tOGTRJU3GPS/pt5IOl/T9lJwC+EVV6nA+BQTwAT7AT2kbDr3CIYYBBP7JCmGPFfMXntQS+Let/aSy7MEVhyy6vMUO//WSvizpzyTFBUDuH17Dzy0dDS1FOodm4hDDoDW0FO+bjcEPFsSDviA+9R/WzM8mddyKQxa1PtK/dc2qMKRrLz1o0W11mrxJ0naSvitpWNIdkvaQ9HSR2ttagD9b0j2SPiip5ccYxsfHs9HR0b4+Fw2NhjboDa1I06E+qI9Br4/42fvJULlYIdxZ/w79qWff8E79LNt3KKue1uAz+XtL+rykuLv/jaTzJH2mSN3FuX0FYwfBny8prnguA/jpqjk0VmKguddnrENOcOLilZeD6MeGz+AHnRiCVta/Uz++Qz/LdLQyXdXLz+JvDcDfS9JHakcZ8ViDHX4i8x0aKzF4NVb8wA8WYL17OTZ+Fj8LWh5CZVLSL2t33inLqkMh0wVNPoOf2OE7H7Y1AD9+/jAC/7Ta6xibgD82NnZuCOGc+sceGRnpXAlmoAAKoAAKzHgFhoeHu8rF+Jl8heq8TNmeG4/Vw/3KKnc1+Ox96dp39cFKiPZdkl5cey1jvB74je7Ha/jsYNjB9G4Hk6fmHU4ZBvEIOY8XLjq4xOGSm3m9bDfPHfi3Szqg7iHeJukbzR4M4AN8gA/w2zU+AOPVJ/AjJWOLj3EH/vQnZIffod8Oq1Vi8Gqs+IEfLIi9F8QdtvmOhm9NwE96MHb4NDQamndDc1h0sKP06hP4kYS3woMAfmEJt7wADc2rmeAHfjQqc4e8IAZyswQENb0kwC9BbYciZsXs1UjwAz9YdLRutg590yGGEpC06ZIAvwR1XZLGIQ5i8AIdfuAHL3l5v+RVApIAfpmiOjRVdpRejR0/8IMdPjv8MrmTcm12+CkqdTgG4Hs1d/zAD2Drv6t1qFOHGDrETUfDAX5HcqUNdkkahziIAdhyhOwNW4ca5QQsjS1FRwH8ogo2mE8BATl2lN6QAzBeNYofJYCowSUBfgk6A3yvZoIf+MECjAVYSqt36RUpseYZA/DzqNZmjkvSOMRBDMCWI31v2DrUKDv8EkDEDr83olJAQI4dpTfkAIxXjeJHb9jEDr8EnQG+VzPBD/xgAcYCLKXVu/SKlFjzjAH4eVTjSD9ZNYcCcoiBHYzXogM/8MN1EZjcXHMMBPg5RGs3BcB4NRP8wA/X5u6Qmw4xsABrR5Xu/B7gd0fHza5CAQEYAMMRckprcegVDjEA/JRsKT4G4BfXcIsrUEAAH+AD/JTW4tArHGIA+CnZUnwMwC+uIcBvoaFDM3GIgYbmtQjED/xwXZSXgKRNlwT4JagLYLyaCX7gh2tzd8hNhxhYgJUAogaXBPgl6EwBARgAw5F+Smtx6BUOMQD8lGwpPgbgF9eQI32O9JOyyKGxEgOL0fpkdcgJgJ/UQgoPAviFJdzyAhQQTZUdPjv8lNbi0CscYgD4KdlSfAzAL64hO3x2+ElZ5NBYiYHFKDt8/8VoUkPJMQjg5xCt3RSHpsqK2aux4wd+cOrTunM69E2HGNrxpcjvAX4R9ZrMdUkahziIwQt0+IEf7PDZ4ZeAvf5ccnx8PBsdHe3rQsahqbKj9Grs+IEf7PDZ4feHir+/a1/BWMbDA3yvxuqw+HGIAeB75SV+4IfrAqwMLk5dE+CXoC6A8Wom+IEfrs3dITcdYmABVgKIGlwS4JegMwUEYACM/+ukDnVKDPSKEhDU9JIAvwS1HYqYFbNXI8EP/GARyGv4JeCmo0sC/I7kShsM8L2aO37gB7DlxCWle7v0ipRY84wB+HlUazPHJWkc4iAGYFtfLg45wYmLV17iRwkganBJgF+CzjQ0r2aCH/jBDp8dfkqrd+kVKbHmGQPw86jGDj9ZNYcCcoiBHYzXogM/8MN1EZjcXHMMBPg5RGs3BcB4NRP8wA/X5u6Qmw4xsABrR5Xu/B7gd0fHza5CAQEYAMMRckprcegVDjEA/JRsKT4G4BfXcIsrUEAAH+AD/JTW4tArHGIA+CnZUnwMwC+uIcBvoaFDM3GIgYbmtQjED/xwXZSXgKRNlwT4JagLYLyaCX7gh2tzd8hNhxhYgJUAogaXBPgl6EwBARgAw5F+Smtx6BUOMQD8lGwpPgbgF9eQI32O9JOyyKGxEgOL0fpkdcgJgJ/UQgoPAviFJdzyAhQQTZUdPjv8lNbi0CscYgD4KdlSfAzAL64hO3x2+ElZ5NBYiYHFKDt8/8VoUkPJMQjg5xCt3RSHpsqK2aux4wd+cOrTunM69E2HGNrxpcjv3YH/EknXSTpQ0q8lLZW0utUDj4+PZ6Ojo319LpekcYiDGLxAhx/4wQ6fHX6RRUOZc/eRFP+5QdJrJd0kaQ+AnyY5zX2jTg46uMThoIVDDPjhtfDBj7SeXnRUX3fCHQb/ekmflrQ/wE9TzqGxEoNXY8UP/GCHzw4/jSD9G/U6SVdKel/csE2FMTY2dm4I4Zz6sEZGRvoXKXdGARRAARTYahUYHh7emjbCHem8NTzYAZJGJR0v6fF2T8dr+Oxg2MF472AcThk4QvbqE/jRjmzd+b078OdKulrSAknPpTwywPcqZIfm7hADDc0rL/EDPxrxxKVXpLAuzxh34J8l6by6B3uZpKeaPSzA9ypkhwJyiAHAeOUlfuAHwM+zZDCbA/C9CtkBtg4xABivvMQP/AD4ZvDOEw7A9ypkB9g6xABgvPISP/AD4OchrNkcgO9VyA6wdYgBwHjlJX7gB8A3g3eecAC+VyE7wNYhBgDjlZf4gR8APw9hzeYAfK9CdoCtQwwAxisv8QM/AL4ZvPOEA/C9CtkBtg4xABivvMQP/AD4eQhrNgfgexWyA2wdYgAwXnmJH/gB8M3gnSccgO9VyA6wdYgBwHjlJX7gB8DPQ1izOQDfq5AdYOsQA4Dxykv8wA+AbwbvPOEAfK9CdoCtQwwAxisv8QM/AH4ewprNAfhehewAW4cYAIxXXuIHfgB8M3jnCQfgexWyA2wdYgAwXnmJH/gB8PMQ1mwOwPcqZAfYOsQAYLzyEj/wA+CbwTtPOADfq5AdYOsQA4Dxykv8wA+An4ewZnMAvlchO8DWIQYA45WX+IEfAN8M3nnCAfhehewAW4cYAIxXXuIHfgD8PIQ1mwPwvQrZAbYOMQAYr7zED/wA+GbwzhMOwPcqZAfYOsQAYLzyEj/wA+C3JmyQlOWBcC/nAHyvQnaArUMMAMYrL/EDPwB+azI/IOkaSddKeriXEO/kXgDfq5AdYOsQA4Dxykv8wA+A35qsqyUdJGlHSd+S9HlJN0p6vhMglz0W4HsVsgNsHWIAMF55iR/4AfDb03gbSftLWiDpGEnxvz8r6TxJT7efXv4IgO9VyA6wdYgBwHjlJX7gB8Bvz+M/lXSEpHdK2kfSt2v//rGkg9tPL38EwPcqZAfYOsQAYLzyEj/wA+C35vG9kvaS9FjtOP9zkh6StJuk/ytp+/Jx3v4OAN+rkB1g6xADgPHKS/zAD4Dfmqe3SIqQXytpctrQIUknSrqyPY7LHwHwvQrZAbYOMQAYr7zED/wA+OXzuPQ7AHyvQnaArUMMAMYrL/EDPwB+6Tgu/wYA36uQHWDrEAOA8cpL/MAPgF8+j0u/A8D3KmQH2DrEAGC88hI/8APgl47j8m8A8L0K2QG2DjEAGK+8xA/8APjl87j0OwB8r0J2gK1DDADGKy/xAz8Afuk4Lv8GAN+rkB1g6xADgPHKS/zAD4BfPo9LvwPA9ypkB9g6xABgvPISP/AD4JeO4/JvAPC9CtkBtg4xABivvMQP/AD45fO49DsAfK9CdoCtQwwAxisv8QM/AH7pOC7/BgDfq5AdYOsQA4Dxykv8wA+AXz6PS78DwPcqZAfYOsQAYLzyEj/wA+CXjuPybwDwvQrZAbYOMQAYr7zED/wA+OXzuPQ7AHyvQnaArUMMAMYrL/EDPwB+6Tgu/wYA36uQHWDrEAOA8cpL/MAPgF8+j0u/A8D3KmQH2DrEAGC88hI/8APgl47j8m8A8L0K2QG2DjEAGK+8xA/8APjl87j0OwB8r0J2gK1DDADGKy/xAz8Afuk4Lv8GAN+rkB1g6xADgPHKS/zAD4BfPo87vUOQdJmk90haJ+kESd9rdRGA71XIDrB1iAHAeOUlfgy2Hw+NjOw3FMK+CuGNku4Oleznu1y/+vp2AHLpFe3izPv7CFTnn3dIOllS/Pc8SZ+R9GqAn2aZQ/ISg1djxQ/8qO8eDjnRzQXYYyMjf1Gt6Nz65wyVsLId9F20SOvwnY9yB/7Fkn4q6drao/1E0kGSHm/2qOzwaWiD3tA6L3NywvX41gEwDjF0C/gPjIy8ZLtK5SJJ22bV6t/9Trp3O2k/VfQBKTy868TqD7WqHxctitR4q7nuwL9K0lpJN9ce4luSPijpnvjfY2Nj54YQzql/wJGRkbL04roogAIogAKmClTuvVdDd/5A1bl/osm3vGVjlM8/r21WT0hDQ3rh3ce0jXx4eNidi22fodkA9weLK7WfSbqm9gB3S3pb7fX8hs/EDp/dHDv85v3AYQfjEEO3dpS5O29tooMWDjF0y49fjIzsEyq6IITw/3ZZtfroeN1fjIzsEirx5eDw6K4Tqz/ADr9o1pY3f6Gk0yUdJim++WK89lp+0zsCfIAP8AF+SktyAB0xdL9fPbL4yL+Vst3jEX6W6Z+DNEch21eqfHPXiYnLAX5KdfRnzNS79I+X9JikYyX9qFUoAL/7BVTEehoafrAAYwGW0kO61SseO+qdr82yobOzLHvxtPvet+vEl05tF0e3Ymh3n3793v1Iv2NdAD6AATAAJqVxODR3YiinX2UjI9s+OjT0n7Nq9aUhyx7b9Utf+v7WkhMpceYdA/DzKtdinkMRx/Ac4iCGchpa3rTFD/xgQey9IM5b2ynzAH6KSh2OcWiqAHv45SQAABTzSURBVN+rseMHfjRqIw69wiEG6qNDyOQcDvBzCtdqGgXk1dzxAz+Arf+u1qFOHWIoAUmbLgnwS1DXJWkc4iAGYMsRsjdsHWqUHX4JIGpwSYBfgs4UEJBjR+kNOQDjVaP4UQKIAH5vRAX4Xs0EP/CDBRgLsJTu79IrUmLNM4Ydfh7V2sxxSRqHOIgB2HKk7w1bhxplh18CiNjh90ZUCgjIsaP0hhyA8apR/OgNm9jhl6AzwPdqJviBHyzAWICltHqXXpESa54xAD+PahzpJ6vmUEAOMbCD8Vp04Ad+uC4Ck5trjoEAP4do7aYAGK9mgh/44drcHXLTIQYWYO2o0p3fA/zu6LjZVSggAANgOEJOaS0OvcIhBoCfki3FxwD84hpucQUKCOADfICf0loceoVDDAA/JVuKjwH4xTUE+C00dGgmDjHQ0LwWgfiBH66L8hKQtOmSAL8EdQGMVzPBD/xwbe4OuekQAwuwEkDU4JIAvwSdKSAAA2A40k9pLQ69wiEGgJ+SLcXHAPziGnKkz5F+UhY5NFZiYDFan6wOOQHwk1pI4UEAv7CEW16AAqKpssNnh5/SWhx6hUMMAD8lW4qPAfjFNWSHzw4/KYscGisxsBhlh++/GE1qKDkGAfwcorWb4tBUWTF7NXb8wA9OfVp3Toe+6RBDO74U+T3AL6Jek7kuSeMQBzF4gQ4/8IMdPjv8ErDXn0uOj49no6OjfV3IODRVdpRejR0/8IMdPjv8/lDx93ftKxjLeHiA79VYHRY/DjEAfK+8xA/8cF2AlcHFqWsC/BLUBTBezQQ/8MO1uTvkpkMMLMBKAFGDSwL8EnSmgAAMgPF/ndShTomBXlECgppeEuCXoLZDEbNi9mok+IEfLAJ5Db8E3HR0SYDfkVxpgwG+V3PHD/wAtpy4pHRvl16REmueMQA/j2pt5rgkjUMcxABs68vFISc4cfHKS/woAUQNLgnwS9CZhubVTPADP9jhs8NPafUuvSIl1jxjAH4e1djhJ6vmUEAOMbCD8Vp04Ad+uC4Ck5trjoEAP4do7aYAGK9mgh/pfpxz++3b//aZ3+xRGdr21Vl4YZ30wv2XHXzkr9vlfCe/x490PzrRNe9Y/PDyI6+PKfMAfopKHY6hgLwKCD/S/PjImjW7b7ONzsgy7T495UPQhZfOX3RHh2XQdDh+pPnRLb3bXQc/vPxo51eR3wP8Iuo1mUsBeRUQfqT5ccqtay+Usr0Vws+VVX+qSthbVe0ZZ4fJcPylCxf+qhvlgh9pfnRD65Rr4IeXHyme5R0D8PMq12IeBeRVQPiR5scpX/vqtaqGl88K2dKL5h92b5x16m1rr8+ybMdu7vLxI82PElpTw0vih5cfZfoO8EtQlwLyKiD8SPPjlNvWrFamF1WGtltyyUEHPRNnffRrXz01VMMBCpXPrZi/YE03ygU/0vzohtYp18APLz9SPMs7BuDnVY4dfpJyDs3EIYYolkMcrWKYOtLPQvXqocqLvrs+e/6NlWr4QIy9Wl1/+uULjvg/Saa3GeSgw9bgRze0TrkGfgD8lDyxHMO35Xklr0MzcYhhawDMR29d8/Ygfbi+sEPQwztu/+JT/+atb32+G0WPH9RoozxyyAuHGLpRY82uwQ6/BHVdksYhDmLYupr7R2+5aZ9KZei/VlXdIwQ9q2pl3fr12d9+atGih7tVKg45sTUswLqld7vr4IdXjbbzq8jvAX4R9ZrMpYC8Cgg/8IMdZfNGR3141UcJSNp0SYBfgroUkFcB4Qd+AHyAn9LqXXpFSqx5xgD8PKq1meOSNA5xEAOwrS8Xh5zgSN8rL/GjBBA1uCTAL0FnGppXM8EP/GCHzw4/pdW79IqUWPOMcQf+SyRdJ+lASfHveS+VtLrVg/IufZo7O0rv5u7SVB3iIAb6VR5w553jDvx9JMV/bpD0Wkk3SdoD4KfZTTPZqJODDi5xOGjhEAN+eIEWP9J6etFR7sCf/nyvl/RpSfsD/DTbHRorMXg1VvzAD07AvE/A0rp7vlFbC/BfJ+lKSe+LG7apRx0bGzs3hHBO/aOPjIzkU4NZKIACKIACM1qB4eHhrYWLHfvk+GCXSzpJ0qOSdpV0gKRRScdLerzdE/IaPjsYdjDeOxiHUwaOkL36BH60I1t3fu8I/OlPNlfS1ZIWSHou5ZEBvlchOzR3hxhoaF55iR/40YgnLr0ihXV5xrgD/yxJ59U92MskPdXsYQG+VyE7FJBDDADGKy/xAz8Afp4lg9kcgO9VyA6wdYgBwHjlJX7gB8A3g3eecAC+VyE7wNYhBgDjlZf4gR8APw9hzeYAfK9CdoCtQwwAxisv8QM/AL4ZvPOEA/C9CtkBtg4xABivvMQP/AD4eQhrNgfgexWyA2wdYgAwXnmJH/gB8M3gnSccgO9VyA6wdYgBwHjlJX7gB8DPQ1izOQDfq5AdYOsQA4Dxykv8wA+AbwbvPOEAfK9CdoCtQwwAxisv8QM/AH4ewprNAfhehewAW4cYAIxXXuIHfgB8M3jnCQfgexWyA2wdYgAwXnmJH/gB8PMQ1mwOwPcqZAfYOsQAYLzyEj/wA+CbwTtPOADfq5AdYOsQA4Dxykv8wA+An4ewZnMAvlchO8DWIQYA45WX+IEfAN8M3nnCAfhehewAW4cYAIxXXuIHfgD8PIQ1mwPwvQrZAbYOMQAYr7zED/wA+GbwzhMOwPcqZAfYOsQAYLzyEj/wA+DnIazZHIDvVcgOsHWIAcB45SV+4AfAN4N3nnAAvlchO8DWIQYA45WX+IEfAD8PYc3mAHyvQnaArUMMAMYrL/EDPwC+GbzzhAPwvQrZAbYOMQAYr7zED/wA+HkIazYH4HsVsgNsHWIAMF55iR/4AfDN4J0nHIDvVcgOsHWIAcB45SV+4AfAz0NYszkA36uQHWDrEAOA8cpL/MAPgG8G7zzhAHyvQnaArUMMAMYrL/EDPwB+HsKazQH4XoXsAFuHGACMV17iB34AfDN45wkH4HsVsgNsHWIAMF55iR/4AfDzENZsDsD3KmQH2DrEAGC88hI/8APgm8E7TzgA36uQHWDrEAOA8cpL/MAPgJ+HsGZzAL5XITvA1iEGAOOVl/iBHwDfDN55wgH4XoXsAFuHGACMV17iB34A/DyENZsD8L0K2QG2DjEAGK+8xA/8APhm8M4TDsD3KmQH2DrEAGC88hI/8APg5yGs2RyA71XIDrB1iAHAeOUlfuAHwDeDd55wAL5XITvA1iEGAOOVl/iBHwA/D2HN5gB8r0J2gK1DDADGKy/xAz8Avhm884QD8L0K2QG2DjEAGK+8xA/8APh5CGs2B+B7FbIDbB1iADBeeYkf+AHwzeCdJxyA71XIDrB1iAHAeOUlfuAHwM9DWLM5AN+rkB1g6xADgPHKS/zAD4BvBu884QB8r0J2gK1DDADGKy/xAz8Afh7Cms0B+F6F7ABbhxgAjFde4gd+AHwzeOcJB+B7FbIDbB1iADBeeYkf+AHw8xDWbA7A9ypkB9g6xABgvPISP/AD4JvBe1o4syXdI+mDkm5uFSbA9ypkB9g6xABgvPISP/AD4PsC/3xJb5J0GcBPN8kBdMTg1VjxAz/qO4hDTrAAS+/rRUaGIpN7NHcvSR+R9LSkOwB+uuoOhUwMAAbANK9Z6oP6SO/oxUduDcD/Yg34p9UDf2xs7NwQwjn1MoyMjBRXhiugAAqgAArMOAWGh4e3Bi7m8sXxwS6XdJKkRyWdLOnFkj4vaZwdfmces3vYqJeDDi5xOGjhEAN+eO2s8aOz3p53tCPwpz/L7ZIOqHu4t0n6RrMH5k17XoXs0NwdYqCheeUlfuBHI4a49Iq8QG83zx340+Nnh9/OzbrfOyQvMXg1VvzAj/o24pATLMA6bO45h29NwE96RHb4NDQaWvNScWjuDjEAGK8+gR9JeCs8COAXlnDLC9DQvJoJfuCH6/GtQ246xADwSwBRg0sC/BJ0poAADIDxPmUAMF41ih8lgAjg90ZUgO/VTPADP1iAsQBL6f4uvSIl1jxj2OHnUa3NHJekcYiDGIBtfbk45AQ7Sq+8xI8SQMQOvzei0tC8mgl+4Ac7fHb4Kd3fpVekxJpnDDv8PKqxw09WzaGAHGJgB+O16MAP/HBdBCY31xwDAX4O0dpNATBezQQ/8MO1uTvkpkMMLMDaUaU7vwf43dFxs6tQQAAGwHCEnNJaHHqFQwwAPyVbio8B+MU13OIKFBDAB/gAP6W1OPQKhxgAfkq2FB8D8ItrCPBbaOjQTBxioKF5LQLxAz9cF+UlIGnTJWcE8BeeuvKPqtsOHapMu4eKfp1NVv/1lguPuqUsYQGMVzPBD/xwbe4OuekQAwuwsmi0+XUHHvjvWHbjH76Q/e6aejkzhW/fMr740jJkpoAADIDhSD+ltzj0CocYAH5KthQfM/DAXzA6sUzK3hwUVlYV7guVyZeqGo6V9PKKsrPXjB91V3EZN78CBQTwAT7AT+krDr3CIQaAn5ItxccMPPAPHb3+C0HhpdVs/fJbLzzm7ijZoaMTZwVlf6ksfPbmCxevLS4jwG+moUMzcYiBhua1CMQP/HBdlHebR9OvN/DAXzB6/d9L4RUVVT66ZvzI+2vAPzUoO6BSrX5qzUVHf73bAgMYr2aCH/jh2twdctMhBhZg3aZQ4+sNPPAXLlt1YpZpgbJwZ6bqXSFUdgvS/pmybTPNOv2W8Xf9vNtSU0AABsBwpJ/SVxx6hUMMAD8lW4qPGXjgz1+2eqdZ1eryLGjPzY82wsq144uvLy7hlleggAA+wAf4Kb3FoVc4xADwU7Kl+JiBB36U6O1Lr9thm1nbLA7S7grh34Mq/7R27Mh/LC5f4ytQQAAf4AP8lP7i0CscYgD4KdlSfMyMAH5xmTq7AgUE8AE+wE/pGg69wiEGgJ+SLcXHAPziGm5xBQoI4AN8gJ/SWhx6hUMMAD8lW4qPGTjgX3HFFdkzzzxTXBmugAIogAIoMKMU2GmnnfTe97534Lg4ZeLAPdj4+Hg2Ojra1+dyiCEa7BAHMfy+X6LFRi0cdHCJw0ELhxjwozfrqr6CsYxHdEhehxgoIC/Q4gd+NOp3Dr3CIQbqowwabnlNgF+CzhSQV3PHD/wAts0bHfXhVR8lIGnTJQF+CepSQF4FhB/4AfABfkqrd+kVKbHmGTNwwB8bGzt32bJl5+YRo1tzHGKIz+IQBzH8PqvQYqMWDjq4xOGghUMM+NEt+rS+zsABvzeycRcUQAEUQAEU2LoUAPhbl19EiwIogAIogAK5FAD4uWRjEgqgAAqgAApsXQoMOvDfKOmH0yz5pqS/7pFFsyXdI+mDkm7u0T2n32aOpC9I2k/S45JOkfTVHsfxEknXSTpQ0q8lLZW0uscxxNv9laRrJa2Q9Jke3z/W2GWS3iNpnaQTJH2vxzH0W4Opx3XIB4e6mG5/P/tEP/vjdA2Ok3S+pKjFOZKu6FF9fEzSx+vu9VJJv+3R/Xt+m0EH/tslLZL033qu7MYEflOt2fcD+PMUvyxI+rqkN0j6oqRX9ViHfSTFf26Q9FpJN0nao8cxbFvz4DeSHukD8N8h6WRJ8d/Rk7jgePUM02DqcR3ywaEuptvfzz7Rz/44pcHOtR61UFL8E6mxR93Z4/qIt4v96UJJh/Xh3j275aADf3FtBfeHtWSK4F/TA3X3kvQRSU9LuqNPO/ypx6zUdthnSdq/B8/e7Bavl/TpPsYQV/O/6gPwL5b009oJQ9TmJ5IOqp269NqOfmnQ6Dn7nQ8OddHvPtGv/jg9H46RdGhtc7S9pA9L+nKvC6O26Hi/pAf7cO+e3XLQgf/nkuI/q2q7qy9J2lXS+pIVjrvpCPzTDID/fO2I6nBJ3y/5uZtd/nWSrpT0PkkP9CmGfsHuKklrpy36vlV7mSe+3NPrn35pUP+cDvngUBf97hP96o/T8yG+zLdA0rtqp3/xFHA3SdUeFke8f3ypN57EDfTPIAL/ckknSXq0BvfpBt4l6ZASdlfT7xmT5sWSPh//bHiPgd/o2eNOJu6m4qr5zyTFRlfmT30MB0galXR8Cbo3e45GOvQLdhdJ+pmka2rB3i3pbbXX88v0odG1+6XB9Fj6kQ+NtOh1XdTHEAHXrz7RLO/K6o+t8vzEWp+OJ5Dx58eSDpb0ZA+L49bae5z6sQjv4WNKgwj86QLG19DjmzDim/XiG7fiG8jia0Rlrh5vlxSb2vSf2OC/0VNnN75/YDtJ35U0XFt4xNfP48sMvfqZK+nq2gr+uV7dtMl9+gW7+Nrk6bXXBuObpOIiML6O3I+ffmkw9awO+eBQF1EPhz7Rj/5Yn/d7197jE3fY8Q2VX6nt9Mvs0dNjiG8U/N99eF9NP+p/4IEfARffqf4Xkv5V0ockxSPVXv30eoc//bliIcVThri7j29YO68Pr1/HVXu87/Sfl0l6qlcG1D6l8J26++1Uez2/F2FMvUs/nnA8JulYST/qxY2n3SN+UqOfGkyF4pAPDnVRb3+/+kS/++OUDvFYPy6K46Yg9uhevsk5vrwU30j7lh7XZF9uN+g7/L6Iyk1RAAVQAAVQwE0BgO/mCPGgAAqgAAqgQAkKAPwSROWSKIACKIACKOCmAMB3c4R4UAAFUAAFUKAEBQB+CaJySRRAARRAARRwUwDguzlCPCiAAiiAAihQggIAvwRRuSQKoAAKoAAKuCkA8N0cIR4UQAEUQAEUKEEBgF+CqFwSBVAABVAABdwUAPhujhAPCqAACqAACpSgAMAvQVQuiQJmCpwt6czan5iOf9b4XyR9TlL8k6b8oAAKzBAFAP4MMZrHnNEKzJL0g5oC/1b7Qqn49/VfmNGq8PAoMMMUAPgzzHAed8YqEL80Jn716JCkvSQ9OGOV4MFRYIYqAPBnqPE89oxTIH5r4g8lbSvp/ZL++4xTgAdGgRmuAMCf4QnA488IBbavfSXv3ZKekBS/qvdPJT06I56eh0QBFNigAMAnEVBg8BX4pKR3S4rf/f2spJ/W/lk4+I/OE6IACkwpAPDJBRRAARRAARSYAQoA/BlgMo+IAiiAAiiAAgCfHEABFEABFECBGaAAwJ8BJvOIKIACKIACKADwyQEUQAEUQAEUmAEKAPwZYDKPiAIogAIogAIAnxxAARRAARRAgRmgAMCfASbziCiAAiiAAijw/wH+7Ig4wU4A1QAAAABJRU5ErkJggg==",
"text/plain": [
"<VegaLite 2 object>\n",
"\n",
"If you see this message, it means the renderer has not been properly enabled\n",
"for the frontend that you are using. For more information, see\n",
"https://altair-viz.github.io/user_guide/troubleshooting.html\n"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.decomposition import PCA\n",
"\n",
"def pca_2d(paragraph_matrix, groups):\n",
" pca = PCA(n_components=2)\n",
" reduced_dims = pca.fit_transform(paragraph_matrix)\n",
" print(f\"2-component PCA, explains {sum(pca.explained_variance_):.2f}% of variance\")\n",
" df = pd.DataFrame(reduced_dims, columns=[\"x\", \"y\"])\n",
" df[\"group\"] = groups\n",
" return df\n",
"\n",
"example_2d = pca_2d(model.paragraph_matrix.data, [\"0\",\"1\",\"2\",\"3\"])\n",
"alt.Chart(example_2d).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not much to see on such a tiny dataset without any labelled groups."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Running this on some bigger data\n",
"--------------------------------\n",
"\n",
"We'll use the BBC's dataset. The dataset was created by Derek Greene at UCD and all articles are copyright Auntie. I've munged it into a file per topic."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>text</th>\n",
" <th>tokens</th>\n",
" <th>group</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Claxton hunting first major medal British hurdler Sarah Claxton is confident she can win her fi...</td>\n",
" <td>[claxton, hunting, first, major, medal, british, hurdler, sarah, claxton, is, confident, she, ca...</td>\n",
" <td>sport</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>O'Sullivan could run in Worlds Sonia O'Sullivan has indicated that she would like to participat...</td>\n",
" <td>[could, run, in, worlds, sonia, has, indicated, that, she, would, like, to, participate, in, nex...</td>\n",
" <td>sport</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Greene sets sights on world title Maurice Greene aims to wipe out the pain of losing his Olympi...</td>\n",
" <td>[greene, sets, sights, on, world, title, maurice, greene, aims, to, wipe, out, the, pain, of, lo...</td>\n",
" <td>sport</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>IAAF launches fight against drugs The IAAF - athletics' world governing body - has met anti-dop...</td>\n",
" <td>[iaaf, launches, fight, against, drugs, the, iaaf, athletics, world, governing, body, has, met, ...</td>\n",
" <td>sport</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" text \\\n",
"0 Claxton hunting first major medal British hurdler Sarah Claxton is confident she can win her fi... \n",
"1 O'Sullivan could run in Worlds Sonia O'Sullivan has indicated that she would like to participat... \n",
"2 Greene sets sights on world title Maurice Greene aims to wipe out the pain of losing his Olympi... \n",
"3 IAAF launches fight against drugs The IAAF - athletics' world governing body - has met anti-dop... \n",
"\n",
" tokens \\\n",
"0 [claxton, hunting, first, major, medal, british, hurdler, sarah, claxton, is, confident, she, ca... \n",
"1 [could, run, in, worlds, sonia, has, indicated, that, she, would, like, to, participate, in, nex... \n",
"2 [greene, sets, sights, on, world, title, maurice, greene, aims, to, wipe, out, the, pain, of, lo... \n",
"3 [iaaf, launches, fight, against, drugs, the, iaaf, athletics, world, governing, body, has, met, ... \n",
"\n",
" group \n",
"0 sport \n",
"1 sport \n",
"2 sport \n",
"3 sport "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dfs = []\n",
"for document_set in (\"sport\",\n",
" \"business\",\n",
" \"politics\", \n",
" \"tech\", \n",
" \"entertainment\"):\n",
" df_ = pd.read_csv(f\"data/bbc/{document_set}.csv.bz2\", encoding=\"latin1\")\n",
" df_ = tokenize_text(df_)\n",
" df_[\"group\"] = document_set\n",
" dfs.append(df_)\n",
"\n",
"bbc_df = pd.concat(dfs)\n",
"bbc_df[:4]"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset comprises 2225 documents and 19063 unique words\n"
]
}
],
"source": [
"bbc_vocab = Vocab([tok for tokens in bbc_df.tokens for tok in tokens])\n",
"\n",
"bbc_df = clean_tokens(bbc_df, bbc_vocab)\n",
"\n",
"print(f\"Dataset comprises {len(bbc_df)} documents and {len(bbc_vocab.words)} unique words\")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"bbc_noise = NoiseDistribution(bbc_vocab)\n",
"bbc_examples = list(example_generator(bbc_df, context_size=5, noise=bbc_noise, n_negative_samples=5, vocab=bbc_vocab))"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [],
"source": [
"bbc_dataset = NCEDataset(bbc_examples)\n",
"bbc_dataloader = DataLoader(bbc_dataset, batch_size=1024, drop_last=True, shuffle=True) # TODO could tolerate a larger batch size\n",
"\n",
"bbc_model = DistributedMemory(vec_dim=50,\n",
" n_docs=len(bbc_df),\n",
" n_words=len(bbc_vocab.words))"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Epochs: 100%|██████████| 80/80 [36:14<00:00, 26.78s/it]\n"
]
}
],
"source": [
"bbc_training_losses = train(bbc_model, bbc_dataloader, epochs=80, lr=1e-3)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.vegalite.v2+json": {
"$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json",
"config": {
"view": {
"height": 300,
"width": 400
}
},
"data": {
"name": "data-1dabe6fce07510d2b0d23042635cb568"
},
"datasets": {
"data-1dabe6fce07510d2b0d23042635cb568": [
{
"epoch": 0,
"training_loss": 2.642270294331616
},
{
"epoch": 1,
"training_loss": 2.1742215334258463
},
{
"epoch": 2,
"training_loss": 1.990165346749821
},
{
"epoch": 3,
"training_loss": 1.873253081452032
},
{
"epoch": 4,
"training_loss": 1.789763425596012
},
{
"epoch": 5,
"training_loss": 1.7244694537997987
},
{
"epoch": 6,
"training_loss": 1.670895414915144
},
{
"epoch": 7,
"training_loss": 1.6244636086943727
},
{
"epoch": 8,
"training_loss": 1.5836919797873645
},
{
"epoch": 9,
"training_loss": 1.5470218249729701
},
{
"epoch": 10,
"training_loss": 1.5139503951398483
},
{
"epoch": 11,
"training_loss": 1.4832230216968134
},
{
"epoch": 12,
"training_loss": 1.4547421681954993
},
{
"epoch": 13,
"training_loss": 1.4282586559745836
},
{
"epoch": 14,
"training_loss": 1.403701515079285
},
{
"epoch": 15,
"training_loss": 1.3798937556166087
},
{
"epoch": 16,
"training_loss": 1.3577345677784511
},
{
"epoch": 17,
"training_loss": 1.3366852766238384
},
{
"epoch": 18,
"training_loss": 1.3162777810363295
},
{
"epoch": 19,
"training_loss": 1.2970764010589315
},
{
"epoch": 20,
"training_loss": 1.2788146187800058
},
{
"epoch": 21,
"training_loss": 1.2611135737496133
},
{
"epoch": 22,
"training_loss": 1.2440896054972772
},
{
"epoch": 23,
"training_loss": 1.2277287255162779
},
{
"epoch": 24,
"training_loss": 1.211776340229911
},
{
"epoch": 25,
"training_loss": 1.1965603542624053
},
{
"epoch": 26,
"training_loss": 1.1817405841365365
},
{
"epoch": 27,
"training_loss": 1.1673207131972223
},
{
"epoch": 28,
"training_loss": 1.1533892758884785
},
{
"epoch": 29,
"training_loss": 1.1397850001080436
},
{
"epoch": 30,
"training_loss": 1.12681249217217
},
{
"epoch": 31,
"training_loss": 1.1138663835407043
},
{
"epoch": 32,
"training_loss": 1.1015954906896035
},
{
"epoch": 33,
"training_loss": 1.089359670544263
},
{
"epoch": 34,
"training_loss": 1.0776376641314962
},
{
"epoch": 35,
"training_loss": 1.0659556754627584
},
{
"epoch": 36,
"training_loss": 1.0547422724480955
},
{
"epoch": 37,
"training_loss": 1.0438089944561075
},
{
"epoch": 38,
"training_loss": 1.0329133888209088
},
{
"epoch": 39,
"training_loss": 1.0227227165832282
},
{
"epoch": 40,
"training_loss": 1.0123565251042383
},
{
"epoch": 41,
"training_loss": 1.0023637906364773
},
{
"epoch": 42,
"training_loss": 0.9926695086200785
},
{
"epoch": 43,
"training_loss": 0.982954163951163
},
{
"epoch": 44,
"training_loss": 0.9737152805239517
},
{
"epoch": 45,
"training_loss": 0.9645086004867317
},
{
"epoch": 46,
"training_loss": 0.9555849193786242
},
{
"epoch": 47,
"training_loss": 0.9466561066437952
},
{
"epoch": 48,
"training_loss": 0.9380779170101474
},
{
"epoch": 49,
"training_loss": 0.929785754813911
},
{
"epoch": 50,
"training_loss": 0.9214358320147354
},
{
"epoch": 51,
"training_loss": 0.9133923515770006
},
{
"epoch": 52,
"training_loss": 0.905320764328382
},
{
"epoch": 53,
"training_loss": 0.8976051105475574
},
{
"epoch": 54,
"training_loss": 0.8899229808623746
},
{
"epoch": 55,
"training_loss": 0.8823469078318673
},
{
"epoch": 56,
"training_loss": 0.8750101461173585
},
{
"epoch": 57,
"training_loss": 0.867823110456052
},
{
"epoch": 58,
"training_loss": 0.8607556018029681
},
{
"epoch": 59,
"training_loss": 0.8537966694891083
},
{
"epoch": 60,
"training_loss": 0.8470302788367182
},
{
"epoch": 61,
"training_loss": 0.8402785332306572
},
{
"epoch": 62,
"training_loss": 0.8336316384884142
},
{
"epoch": 63,
"training_loss": 0.8271638499283642
},
{
"epoch": 64,
"training_loss": 0.8209897966118332
},
{
"epoch": 65,
"training_loss": 0.8146745335981712
},
{
"epoch": 66,
"training_loss": 0.8085972012940401
},
{
"epoch": 67,
"training_loss": 0.8024606412982348
},
{
"epoch": 68,
"training_loss": 0.7965273064856203
},
{
"epoch": 69,
"training_loss": 0.7905283134916554
},
{
"epoch": 70,
"training_loss": 0.785075603212629
},
{
"epoch": 71,
"training_loss": 0.7794511792822654
},
{
"epoch": 72,
"training_loss": 0.7739284674573389
},
{
"epoch": 73,
"training_loss": 0.7684736896745907
},
{
"epoch": 74,
"training_loss": 0.7630798336141598
},
{
"epoch": 75,
"training_loss": 0.7578280769519924
},
{
"epoch": 76,
"training_loss": 0.7526835779966035
},
{
"epoch": 77,
"training_loss": 0.7476536778189381
},
{
"epoch": 78,
"training_loss": 0.7426093139263413
},
{
"epoch": 79,
"training_loss": 0.7375516610856382
}
]
},
"encoding": {
"x": {
"field": "epoch",
"type": "quantitative"
},
"y": {
"field": "training_loss",
"type": "quantitative"
}
},
"mark": "bar"
},
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAFZCAYAAAAYbm8xAAAgAElEQVR4Xu2dC7QlVZnff6dRQaODxmFGRWOw8UWiBhAHZxlHx7dRGJNppPER8ZFJ1Dhmoq52loZuUe8FE3FFx1FxfOEIikSlm4dRGJ+oKKBg0HFkBlTAxOf4QkVuZW2oC+eevufeXfVVnburz++s1avh9v7vs+v3fbX/99tVtWuEHwlIQAISkIAEphIYyUYCEpCABCQggekENEqzQwISkIAEJLAGAY3S9JCABCQgAQlolOaABCQgAQlIoB2BPa6i3LlzZ3XQQQe1o6FKAhKQgATmmsDmzZt388U9zigXFxerbdu2FXtcV1xxRbVaIErJTMcXi4T85BcjEFObf/3wK9ZQ2h6uRtmW3E06TzT5xQjE1Oaf/GIEYupp+adRxrg2VjsRNEa2QiA/+cUIxNTm33zy0yhjcW+s9kRrjEyjjCGTn/w6JBDraqjzn0YZi3tj9VATpfGB9iSQXwys/OQXIxBTDzX/NMpY3Burh5oojQ+0J4H8YmDlJ78YgZh6qPmnUcbi3lg91ERpfKA9CeQXAys/+cUIxNRDzT+NMhb3xuqhJkrjA+1JIL8YWPnJL0Ygph5q/mmUsbg3Vg81URofaE8C+cXAyk9+MQIx9VDzT6OMxb2xeqiJ0vhAexLILwZWfvKLEYiph5p/GmUs7o3VQ02Uxgfak0B+MbDyk1+MQEw91PzTKGNxb6weaqI0PtCeBPKLgZWf/GIEYuqh5p9GGYt7Y/VQE6XxgfYkkF8MrPzkFyMQUw81/zTKWNwbq4eaKI0PtCeB/GJg5Se/GIGYeqj5p1HG4t5YPdREaXygPQnkFwMrP/nFCMTUQ80/jTIW9yz1ES879W7LDbc/4+Crt59yyf7p/888ces1WR3MsNFQE3mGiNb8KvnFIiE/+cUIxNTT8k+jjHHNUj9p26nbYXTcysbVjl2LW7dndTDDRk5UMdjyk1+MQExt/vXDT6OMcc1Sa5RZmLIaORFkYZraSH7yixGIqYeafxplLO5Zao0yC1NWo6GeaFkHN4NG8otBlt988tMoY3HPUmuUWZiyGjlRZWGyooxhkp/8VhDQKHtKiPFuNcruIGuUMZbyk1+MQEw91PzTKGNxz1JrlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lwNRuNcruiGuUMZbyk1+MQEw91PyzoozFPUutUWZhymo01BMt6+Bm0Eh+Mcjym09+GmUs7llqjTILU1YjJ6osTC4dxjDJT34uvfaUAy69zgCsRhmDLD/5xQjE1EPNPyvKWNyz1FaUWZiyGg31RMs6uBk0kl8Msvzmk99GG+W+wHuARwE/BF4CfGAiFIcBF4797Dzg0dPCtbi4WG3btm2jj2vF8DTK2Mk1rnaiirGUn/xiBGLqoebfRhvKA4D05wzgfsCHgQMmQvFY4AjghTkh0ihzKE1vM9REjh11d2r5xVjKT34xAjH1tPzbaKMcP6oHAm8CHj5xqE8FjgfuDPy8NswzrShjCTFN7UQV4yo/+cUIxNTmXz/8SjHK+wNvBp4LXDFxqIcA6c9pwMHA6cDdgd8sLCxsH40mX18FW7ZsidHqWH3uxddwzkVXr+j1CYfuz+MPufk1lR1/o91JQAISkEAbAps3b97NF0swykcA24BjgWszDuwS4InT2rr0mkFwjSb+Riq/GIGY2vyTX4xATF3q0us9gXcATwKum3KIhwN3BNJNPA+tb/65F7C0WnuNsp9EifXandqJNMZSfvKLEYiph5p/G11RvhJ41QT6OwH7AJ8BDqxv7jkFOBS4EngBcP60cGmU85nIsaPuTj3UiaA7ArGe5Ce/GIGYutSKMnZUq6g1yhhSJyr5xQjE1Oaf/GIEYmqNMsYvpPY5yhC+FWIn0hhL+ckvRiCmHmr+bfTSa4y6FWXn/IaayJ2DaNmh/FqCq2Xyk1+MQExtRRnjF1JbUYbwWVF2hw+NKAZTfvPJz4oyFvcstUaZhSmrkRNVFqapjeQnvxiBmHqo+adRxuKepdYoszBlNRrqiZZ1cDNoJL8YZPnNJz+NMhb3LLVGmYUpq5ETVRYmK8oYJvnJbwUBjbKnhBjvVqPsDrJGGWMpP/nFCMTUQ80/jTIW9yy1RpmFKavRUE+0rIObQSP5xSDLbz75aZSxuGepNcosTFmNnKiyMLl0GMMkP/m59NpTDkztVqPsjrhGGWMpP/nFCMTUQ80/K8pY3LPUGmUWpqxGQz3Rsg5uBo3kF4Msv/nkp1HG4p6l1iizMGU1cqLKwuTSYQyT/OTn0mtPOeDS6wzAapQxyPKTX4xATD3U/LOijMU9S21FmYUpq9FQT7Ssg5tBI/nFIMtvPvlplLG4Z6k1yixMWY2cqLIwuXQYwyQ/+bn02lMOuPQ6A7AaZQyy/OQXIxBTDzX/rChjcc9SW1FmYcpqNNQTLevgZtBIfjHI8ptPfhplLO5Zao0yC1NWIyeqLEwuHcYwyU9+Lr32lAMuvc4ArEYZgyw/+cUIxNRDzT8ryljcs9RWlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lgEuvMwCrUcYgy09+MQIx9VDzz4oyFvcstRVlFqasRkM90bIObgaN5BeDLL/55KdRxuKepdYoszBlNXKiysLk0mEMk/zk59JrTzng0usMwGqUMcjyk1+MQEw91PyzoozFPUttRZmFKavRUE+0rIObQSP5xSDLbz75aZSxuGepNcosTFmNnKiyMLl0GMMkP/m59NpTDrj0OgOwGmUMsvzkFyMQUw81/6woY3HPUltRZmHKajTUEy3r4GbQSH4xyPKbT34aZSzuWWqNMgtTViMnqixMLh3GMMlPfi699pQDDZde+TXc+Kf+VP9j1+LW7bMe2+T3aUSxCMhPfjECMbX51w8/K8oY1yz16hXlpLTaoVGuj9OJYH1Ga7WQn/xiBGLqoeafRhmLe5Zao8zClNVoqCda1sHNoJH8YpDlN5/8NMpY3LPUGmUWpqxGTlRZmKY2kp/8YgRi6qHmn0YZi3uWWqPMwpTVaKgnWtbBzaCR/GKQ5Tef/DTKWNyz1BplFqasRk5UWZisKGOY5Ce/FQQ0yp4SYrxbjbI7yBpljKX85BcjEFMPNf80yljcs9QaZRamrEZDPdGyDm4GjeQXgyy/+eSnUcbinqXWKLMwZTVyosrC5NJhDJP85OfSa085MLVbjbI74hpljKX85BcjEFMPNf+sKGNxz1JrlFmYshoN9UTLOrgZNJJfDLL85pOfRhmLe5Zao8zClNXIiSoLk0uHMUzyk59Lrz3lgEuvMwCrUcYgy09+MQIx9VDzz4oyFvcstRVlFqasRkM90bIObgaN5BeDLL/55KdRxuKepdYoszBlNXKiysLk0mEMk/zkV9TS677Ae4BHAT8EXgJ8YCJGycxPAp4FfBd4NnDBtDguLi5W27ZtK+oXAI2yu7NOo4yxlJ/8YgRi6qHm30YbygOA9OcM4H7Ah4EDJkLxFODFQPr7YOAtwL01yljCTlMPNZH7odG8V/k1ZzaukJ/8YgRi6mn5t9FGOX5UDwTeBDx84lBfB3wVeHf980uBxwHXrobEirKfRIn12p3aiTTGUn7yixGIqYeaf6UY5f2BNwPPBa6YCMVbgZ3Arvrn5wPPB76uUcaSdjX1UBO5exLtepRfO27LKvnJL0Ygpi65onwEsA04dkqVeCJwOfCuGsFlwGPS9cqFhYXto9HouEk0W7ZsidHqWH3uxddwzkVXr9nr4ffdj8Pv+9sr2hzwu7fveCR2JwEJSEACaxHYvHnzbgXkRleU9wTeATwJuG7K4J8MvAw4EjgMWKyvVa7afLhLr5OHU+3Ytbh1+6xT2t/oY8TlJ78YgZja/OuH30Yb5SuBV00c2p2AfYDPAAcCy3e9porzGuCZwBen4dAo+0mUWK/dqZ0IYizlJ78YgZh6qPm30UYZo76KWqOMIR1qIseOuju1/GIs5Se/GIGYuuRrlLEjm1BrlDGcTlTyixGIqc0/+cUIxNQaZYxfSJ234cDkV3iNcjXoTqShVER+8osRiKmHmn8uvcbinqXWKLMwZTUa6omWdXAzaCS/GGT5zSc/jTIW9yy1RpmFKauRE1UWpqmN5Ce/GIGYeqj5p1HG4p6l1iizMGU1GuqJlnVwM2gkvxhk+c0nP40yFvcstUaZhSmrkRNVFiYryhgm+clvBQGNsqeEGO9Wo+wOskYZYyk/+cUIxNRDzT+NMhb3LLVGmYUpq9FQT7Ssg5tBI/nFIMtvPvlplLG4Z6k1yixMWY2cqLIwuXQYwyQ/+bn02lMOTO22lVFW1d+NRpu+sdxpxdKXZrH3q0YUyw75yS9GIKY2//rhZ0UZ45qlbmWUu/U8mw0IPNGyQmrFEcMkP/n1RCDWrTvzxPiF1BplCN8KsUYeYyk/+cUIxNRDzT8ryljcs9QaZRamrEZDPdGyDm4GjeQXgyy/+eSnUcbinqXWKLMwZTVyosrC5NJmDJP85LeCgEbZU0KMd6tRdgdZo4yxlJ/8YgRi6qHmn0YZi3uWWqPMwpTVaKgnWtbBzaCR/GKQ5Tef/DTKWNyz1BplFqasRk5UWZhcOoxhkp/8XHrtKQemdqtRdkdco4yxlJ/8YgRi6qHmnxVlLO5Z6m6Mkm8BV93yhdX5fWxAMNREzgrEDBrJLwZZfvKLEYipfY4yxi+k7sgoJ8bQzwYETlShUCM/+cUIxNTmXz/8rChjXLPUGmUWpqxGTgRZmLzGFsMkP/mtIKBR9pQQ491qlN1B1ihjLOUnvxiBmHqo+adRxuKepdYoszBlNRrqiZZ1cDNoJL8YZPnNJz+NMhb3LLVGmYUpq5ETVRYmlw5jmOQnP5dee8qBqd1qlN0R1yhjLOUnvxiBmHqo+WdFGYt7lrofo0yPi4zSIyP1Z+m8Lh4XGWoiZwViBo3kF4MsP/nFCMTUXT0e8kDgUiAZ7IuB+wCvAb4TG1536sXFxWrbtm1F/QLQk1FOQOvmcREnqlguyk9+MQIxtfnXD7+mhnIh8BDgj4H3A38LfBt4XGx43ak1yhhLTzT5xQjE1Oaf/GIEYuquKspfA7cB3gXcGTgGuAa4Q2x43ak1yhhLJyr5xQjE1Oaf/GIEYuqujPLHwIOB84H/Cfx34FfA3rHhdafWKGMsnajkFyMQU5t/8osRiKm7MspTgKcD1wEHAVcCFwOHxIbXnXpujbKqXn3bffY6fpzk6TuOSisAjT5OVI1w7dZYfvKLEYipzb9++DW9RpmWXR9fX5tM1yfT53Dg87HhdaeeW6PcDWG7m3s80WK5KD/5xQjE1OZfP/yaGqV3vbaIw2zuep0cmEbZIlRhiRNVDKH85BcjEFN3tfTqXa8t4qBRtoA2ReJEGmMpP/nFCMTUQ82/phWld722yBONsgU0jbI7aGM9DXWi6gVGi07l1wLaHpB/TY3Su15b5MkGGeXCpqXbLYwP98wTj/zpesN3IliP0Nr/Lj/5xQjE1OZfP/yaGqV3vbaIw8YY5eRA865ZeqK1CPAe8Btz7Ki7U5t/MZby64dfU6P0rtcWcdAoW0Bz6bU7aBp5Zyw1ohjKofJrapSJ0u2ABwEV8JX6mcoYvQ7VPh4yDaYVZYdpNrWroU4Es2CT8x3yy6E0vY38+uHX1Ch/H/gQ8Dv1cNJm6EcAl8SG151ao9Qou8um5j05UTVnNq6Qn/xiBGLqLh8PuRY4FdgEPKPe5/VhseF1p9Yop7L82mjE15b/taqqy1Z7LZcTVSwX5Se/GIGY2vzrh1/TivJnwF2B5bsn71i/PcRN0deITxnXKCcHuPpSrCdaPydarNfu1MY3xlJ+88mvqVGmbeveCryjfifl84DnAPeN4etObUWZy1KjzCXVpJ0TaRNau7eVn/xiBGLqrpZej61N8uYVPOBZwHtiw+tOrVHmstQoc0k1aedE34SWRhmjJb9Z8WtaUaZx/V79ouYl4Bzgoq4HG+lPo8ylV10Fm9LbX+rP0ifSNUsn+lx+q7eTn/xiBGJq868ffrlGmd4Ystbn3NjwulNrlG1Z3lRheqK15XeTTn7yixGIqc2/fvjlGmV6ZnKtT24/q/XxUODdwOuBt6zS4DAgbca+/DkPePS0wWiUbRNFo2xLblznRBWjKD/5xQjE1NFrlOs9/vGZlu+lTDv9nAT8CEjPZK5mlI+tn9V8YQ4CjTKH0iptqurk0Wh08p/90UEXvv7Dlz8ktdi5ePQXW/bWm8yJNIZWfvKLEYiph5p/kUpwkliqOtv29wrg+1OM8qnA8cCdgZ8DyTDPtKKMJez66rydfNbvp9sWQz3RuqXQvjf5tWeXlPKbT35tjW01Wn0Z5SFA+nMacDBwOnB34DcLCwvbR6PRcZOD2bJlSyyaHavPvfgazrno6o577be7Jxy6P48/5G79fom9S0ACEiiMwObNm3fzxSEY5STGtF3eE4G0Q9BuH5deu8o6K8o2JK042lC7RSM/+cUIxNTRa5Q5395XRXk4kHYASjfxpBt/0jOb9wLS4ykaZU5k2rX5Rb3UXaurN6+25V27rturnEjbs0tK+ckvRiCmHmr+bXRFmW4S+vQE+v2AWwHpBqEDgQOA9B7MQ4H03N8LgPOnhcuKMpbI09VlVJhDPdH6ikrTfuXXlNjK9vKbT34bbZQx6laUnfPTKGNInUjlFyMQU5t//fDr0ijTDTbpEY8N/VhR9oY/PcKT/tSf6pSNWIp1IojFV37yixGIqYeaf02NMl23Wu3aYHqryAXAi4FvxVDG1BpljF++emOWYod6ouVz7bel/GJ85Tef/Joa5TuBR9R7vCbtE+pdc34APAb4u/pnMZoBtUYZgNdEOqpefcPP9n31uOScNz7xV026aNPWiaoNtVs08pNfjEBMPdT8a2qUXwCOBL5b40oP2n0ASDflpJtwrgB+K4YyptYoY/zaq2dTYQ71RGvPtVul/GI85Tef/JoaZbpG9d+A99W4jql3zUmPb/x5fUfq/jGUMbVGGePXXq1RJnZOpO0zSH4xdvLrj19To3xjvYXc+Ij+CnhuXWWmjc1PjA+3fQ8aZXt2MWW1q4Kd432ctbj1bbE+d1drRDGi8pNfjEBMPdT8a2qUqX3aFefBNa70Lsqz6xt80jOP34xhjKs1yjjDbnrop8Ic6onWDdN4L/KLMZTffPJrapTLlNJbPzaNIftlDF93ao2yO5axnjTKGL9+1E70Ma7ym09+TY3yWOCE+sadcWJN+4nRXkOtUfaGtmHH1Q9glN4IU3+q07p47tKJqmEYJprLT34xAjH1UPOvqcGlx0DSpuSfSG/vGEO2GMPXnVqj7I5ltz11U2EO9UTrlmX73uTXnl1Sym8++TU1yquABwL/GMPVn1qj7I9tqOeK8xgtpY3tb/7sWjxmoWmfTlRNia1sLz/5xQjE1EPNv6ZG+SLgMOCDwPjD5efG8HWn1ii7Y9lvT+0qzKGeaP2yzO9dfvmsVmspv/nk19Qo06u0Vvs07SdGew21Rtkb2q47fvvSqHr7eKdnL2xNG1qs+XGiWo+Q/GKE5Ce/+Iub0/Z149cml5mmV2IV8dEoiwhDi0HkVZgaZQu0YxL5yS9GIKYeav7lVoKPB9Lyavp7tY9Lr2vkz5O2nbodRsfFUmzPVlfwmp/vfd2KvWM/sePY3R47GuqJVkr05BeLhPzmk1+uUaYl19TWpdcWeaJRtoDG6hWmE1Ublrdo5Ce/GIGYeqj5l2uUadPztLya/l7t49KrFWXsDNpNrVF2DPTG7oY6UfXBok2f8mtDbfi/qOUa5SQdd+ZpkC9WlA1gLTetuIHR+PXwajFtWOBE1YLlmER+8osRiKmHmn9NjdKdeVrkiUbZAtruFebxP9v7Lq86+en3uP557/32rdM/f2LHI1e7sayLL2vdx1AngtYH3LFQfjGg8uuHX1OjdGeeFnHQKFtAW1eSd5fsut103MCJKgZUfvKLEYipp+VfU6N0Z54WcdAoW0BbRzKqeMMNt7rhDePNzn7N01J+bujHiT6GX37yixGIqbsySnfmaREHjbIFtMaSMipMJ/rGgVshkJ/8YgRi6q6M0sdDWsRBo2wBraGkojp102j0vnHZzoWjdzXsJtzciT6GUH7yixGIqbsySnfmaREHjbIFtLBkYypMJ/pY4OQnvxiBmDpqlO7ME+CvUQbgtZdeAHz2Fnn1iy7eh7necJzo1yO09r/LT34xAjF11CjdmSfAX6MMwOtMOpsK04k+FjD5yS9GIKaOGuVaO/PcHTgtNrzu1G6K3h3LPayn7wH/b6zC/GAfFaYTfSxr5Ce/GIGYOmqUy9+eHif5QyCZY/rvfYBXAvvHhtedWqPsjuUe3VNVnbxU8bbxYzz7xK1fih6zE32MoPzkFyMQU3dllG8B/mRsKDcA7wSeFxted2qNsjuW89VTN0uzTvSxrJGf/GIEYuqujPInwBHA3wAH1ab5TeBNseF1p9You2M5Zz1dMhpx8fIxV1X1nTZLs070sayRn/xiBGLqrozyOuC2wK+B+wO/Av4PsG9seN2pNcruWM51TxUnXv/rTSeMM/joSUf9cD0mTvTrEVr73+UnvxiBmLoro/wU8PD6lVsProd0NbA5Nrzu1BpldyztaZxA3tKsE30sa+QnvxiBmLoro3wo8HngHsBL6pt5/gL4Smx43ak1yu5Y2tMYgdHoQqrqwlt+Uv1gtaVZJ/pY1shPfjECMXVXRvkzYD8gLcEW+dEoiwzLHjeoaZuyO9HHQi0/+cUIxNRdGeUO4J7AmcAvxoZ0bmx43ak1yu5Y2lMTAjctzTrRN2G2e1v5yS9GIKbuyijdFL1FHNyZpwW0wUmqvx+NNl1xn7vd4THfuOanH6uqpQva3DXb92FrRDHC8ptPfk3fR/lI4PpVUH0mhq87tRVldyztKUTgjKrig+M9nHXC0Ru+g5UTfSimyG8++TU1ym3A4hiqvYCXTvwsRjKo1iiDAJX3Q2BUvfqGn+376vHOz3njE9PjVTP9ONHHcMtvPvk1McqnA6cAzxhDdSDwcmDvGL7u1BpldyztqU8CeY+bdD0CJ/oYUfnNJ78mRnkJ8K9WwXQ+8KgYvu7UGmV3LO2pVwLfHDFKu1rd+KlY+sIsrmk60cdiKr/55NfEKBOhdGI/egxVul55LbAUw9edWqPsjqU9zZTAx6mWPjb+jbtOOObErkfgRB8jKr/55NfUKGOUZqDWKGcA2a+YBYG3s3TDySuM88SnjW140G4ITvTtuC2r5Def/DTKWNyz1D4ekoXJRmsQqCpe9eB9vpaeY775s2PHjsYrOU70sTST33zy0yhjcc9Sa5RZmGzUiEC7m4Gc6BtB3q2x/OaTn0YZi3uWWqPMwmSjZgS+N4LvLUsqqtNzbgZyom8GebK1/OaTXwlGmTZafzfweiC9GHryk8Z4EvAs4LvAs4ELpoXLa5SxRFY9TAIVXD666ZV39ae63E3bu4+lRhljOlR+G22Ut6lN8EfAd6YY5VOAFwPp74PrNvfWKGMJq3oPJzDlbSdDnahKiZb8YpEYKr+NNspl6q8Avj/FKF8HfLWuOlP7S4HH1Y+l7BY1K8pYIqveYwl8vKL6+BEPucfimRd+O+2wxVmLW1e8mLqEIx/qRFoCuzQG+cUi0dWm6LFRTFevZZRvBXYCu2p52uDg+cDXV+tOo+wrRPa7hxHYSVWl8+rmz64Ttq54HGUjjteJPkZdfv3wG0JFmR66vhx4V43gMuAx6XrlwsLC9tFodNwkmi1btsRodaw+9+JrOOeiqzvu1e4k0B2BBx1wJx74z++0osMHH3jn7r7AniQwEAKbN2/ezReHYJRPBl4GHAkcVm/Anq5VrvqxohxINjrM0gm8H6r3r6g4F7d+qO9BWxHFCMuvH34bbZQPAz49cWj7AbcC0qu70qbry3e9HgtcAzwT+KJGGUsI1RJoQqCC00dL1QdWGOeJW1e8RqxJf9PaOtHHKMqvH34bbZSxo1pFbUXZOVI7lMDuBKrRBxktrTDKXYtbV1SgbbA50behdotGfv3w0yhjXLPUbjiQhclGgyZQnVnBR8YP4azFre9oekhO9E2JrWwvv374aZQxrllqjTILk432LALnVBXnrDDOE45+43qH6ES/HqG1/11+/fDTKGNcs9QaZRYmG+3RBEafHI2qTy4fYlVVlTsHdR9wjTLGtPTnKGNHN6b2GmVnKO1IAv0RqPgym0gvg7/pU1XfSsbpRB9DLr9++FlRxrhmqa0oszDZaI4JjEZcUVVccb+7/9Zjv/6dn/xvqD6Xs8n7rJFpRDHiQ+WnUcbinqXWKLMw2UgC4wR+NBrxw7Gl2veWYJxDnehLSa2h8tMoZ5BBGuUMIPsVc0VgVFV/ubTX6M3jB33Wa49Oe0L3+hnqRN8rlAadD5WfRtkgyG2bapRtyamTQDaBM5ZYOmO89dmLx5yarc5sONSJPvPwem82VH4aZe+pARrlDCD7FRIYI1DBZ0cVn735R6Pqui6Wboc60ZeSHEPlp1HOIIM0yhlA9isksAaBCq4cwZW3NKk+2cY4hzrRl5IcQ+WnUc4ggzTKGUD2KyTQiEC1NGJ0Q5JUjEZVVb17tGnpr8a72LVwzOcmuxzqRN8ITY+Nh8pPo+wxKZa71ihnANmvkECnBKrzq4rzxrs864Strx3qRN8pmkBnQ+WnUQaCnivVKHNJ2U4CpRKoroJNVx5419v/wTev/emnK/jUz/e+7tXjo/3EjmN/udGjH6oRbTS35e93Z54NjIRGuYHw/WoJzIbAhry/06XhboOrUXbLs1FvGmUjXDaWwPAJjPhylbbpqz8jqqva3DzUFIQVZVNiK9trlDF+IbVGGcKnWAJ7AoHfANfXB7IX6UXYFX89fmA7Tzh6xdtW2hy0RtmG2i0ajTLGL6TWKEP4FEtgDghUl8LoK7ccaJpnZzIAAA+KSURBVPX3bSpQjTKWKhpljF9IrVGG8CmWwDwSSI+u3Pj4CrAJ+EhV8cFxEGedcPRpk2A0yliqaJQxfiG1RhnCp1gCEtidwDeAv61/fNuq4qsjNp20/ZgHXLX9fZfdM/181wlHfas0cEM1ch8PmUEmaZQzgOxXSEACNxNIj6+MGN38omxYWmqzlNs1Uo2ya6It+/PFzS3BKZOABPZkAr8A0p/02Zuq+uiI6vTxA955wjEf6BuARtk34cz+NcpMUDaTgAQkcAuBq0Zw1Y3/O+L2S1X1pdENo7eOA9r1uqMvjgLTKKMEO9JrlB2BtBsJSEACNYEKLhvBpTf9b3U7qtE/cKtNr19hpK856ur1gGmU6xGa0b9rlDMC7ddIQAISuIXAF4DP1/+7F1X149vus9fx44BO33HUrzXKQlJGoywkEA5DAhKQwM0Eqh3pZiKNspCU0CgLCYTDkIAEJKBRlpkDGmWZcXFUEpDAPBOwoiwq+hplUeFwMBKQgATSDUAuvZaUBxplSdFwLBKQgAQSAY2yqDzQKIsKh4ORgAQkoFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKIvKAY2yqHA4GAlIQAJWlKXlgEZZWkQcjwQkIAEryqJyQKMsKhwORgISkIAVZWk5oFGWFhHHIwEJSMCKsqgc0CiLCoeDkYAEJGBFWVoOaJSlRcTxSEACErCiLCoHNMqiwuFgJCABCVhRBnNgBJwEPAv4LvBs4IKJPg8DLhz72XnAo6d9r0YZjIhyCUhAAp0TsKKMIH0K8GIg/X0w8Bbg3hMdPhY4AnhhzhdplDmUbCMBCUhglgQ0ygjt1wFfBd5dd3Ip8Djg2rFOnwocD9wZ+HltmGdaUUawq5WABCQwSwIaZYT2W4GdwK66k/OB5wNfH+v0ECD9Oa2uOk8H7g78ZmFhYftoNDpucgBbtmyJjKlz7bkXX8M5F13deb92KAEJSGAIBJ5w6P48/pC7DWGobN68OV0SXPHZ7QczPpITgcuBd9XfexnwmPp65bShXAI8caLqvLmtS68zjqBfJwEJSGBdAlaU6yJao8GTgZcBRwLppp3FumoclxwO3BFIN/E8FHgPcC9gabV+NcpIONRKQAIS6IOARhmhunzX67HANcAzgS8CdwE+AxwIHACcAhwKXAm8AEhLtKt+NMpIONRKQAIS6IOARtkH1dZ9apSt0SmUgAQk0BMBjbInsO261SjbcVMlAQlIoD8CGmV/bFv0rFG2gKZEAhKQQK8ENMpe8TbtXKNsSsz2EpCABPomoFH2TbhR/xplI1w2loAEJDADAhrlDCDnf4VGmc/KlhKQgARmQ0CjnA3nzG/RKDNB2UwCEpDAzAholDNDnfNFGmUOJdtIQAISmCUBjXKWtNf9Lo1yXUQ2kIAEJDBjAhrljIGv/XUaZVHhcDASkIAEfHFzaTmgUZYWEccjAQlIwIqyqBzQKIsKh4ORgAQkYEVZWg5olKVFxPFIQAISsKIsKgc0yqLC4WAkIAEJWFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKIvKAY2yqHA4GAlIQAJWlKXlgEZZWkQcjwQkIAEryqJyQKMsKhwORgISkIAVZWk5oFGWFhHHIwEJSMCKsqgc0CiLCoeDkYAEJGBFWVoOaJSlRcTxSEACErCiLCoHNMqiwuFgJCABCVhRlpYDGmVpEXE8EpCABKwoi8oBjbKocDgYCUhAAlaUpeWARllaRByPBCQgASvKonJAoywqHA5GAhKQgBVlaTmgUZYWEccjAQlIwIqyqBzQKIsKh4ORgAQkYEVZWg5olKVFxPFIQAISsKIsKgc0yqLC4WAkIAEJWFGWlgMaZWkRcTwSkIAErCiLygGNsqhwOBgJSEACVpSl5YBGWVpEHI8EJCABK8qickCjLCocDkYCEpCAFWVpOaBRlhYRxyMBCUjAirKoHNAoiwqHg5GABCRgRVlaDmiUpUXE8UhAAhKwoiwqBzTKosLhYCQgAQlYUZaWAxplaRFxPBKQgASsKCM5MAJOAp4FfBd4NnDBRIc5bW6WaJSRcKiVgAQk0AcBjTJC9SnAi4H098HAW4B7T3SY00ajjERBrQQkIIFeCWiUEbyvA74KvLvu5FLgccC1Y53mtNEoI1FQKwEJSKBXAhplBO9bgZ3ArrqT84HnA18f63Rqm4WFhe2j0ei48QHc+ta35vrrr4+MSa0EJCABCcwhgf3224/nPOc56XLfis9uP5gxmxOBy4F31d97GfCY+nrl8lBy2hRdUY4zLfEaquPrLuuNb4yl/OQXIxBTT8u/jTbKJwMvA44EDgMW62uV40eb00ajjOWH/OTXEYFYNxql/GIEYupSjXL5jtZjgWuAZwJfBO4CfAY4EJjWZlUinmj9JEqs1+7UxjfGUn7yixGIqYeafxtdUcaor6IeaiA6B9GyQ/m1BFfL5Ce/GIGY2vzrh98eZ5TpBp+Xv/zl22O4+lM7vhhb+ckvRiCmNv/mk98eZ5SxMKqWgAQkIAEJrCSgUZoREpCABCQggTUIaJSmhwQkIAEJSECj5KH17j+vr7fJS0ga7SHbQRbN+vtyh1wCm2lj3Rd4D/Ao4IfAS4APbEDspo3vd4FTgIfVu0n9GfCRgsa3PO7b1Zt4pM080uYepeRieiTswjG45wGPLmh8aWj/HngNkBimzU3eWND4XgEcP5GcdwR+krGHdu78EGl3a+CdwBHA9+rtStMGM7POv72Ak4F/B3wLeAbw5SbjmIeK8jZ10vwI+M6YUTbaQzaSLbV21t+XM+RS2Ewb6wOA9OcM4H7Ah4ED6r2B19sjOOf4o23S/sT/DPgY8CDgr4F7FTS+5eNLE/3h9XmQjLKUXHxsPYm+cCIQpYwvPaaWYpue5f55HdsvFMRvHFs6P06on0kvhd8f16b0NOA+wAc36Pw4qv6FJ43jCcB/Bn6/SRznwSiXkyn99vX9MaNstIdsdEYFZv19TYa80WxyxvpA4E3Awwtkuamuel9Z4PjSBPUi4Gf1s8nJKEvJxafWFdGdayNKhnlmQeN7OvBv6l8y9qkn2DTZl8Jv/LxJhv4fgH8oaHyPBNIqyzH1M/FpO9KHbMD40krA1cDba2D/F7hbvcHNenuN3yiZZ6PM2Wc2ZwLPbTPr78sdV2o3aZSljfX+wJuB5wJXAKWN75fAPwJ/BHyusPGlKjcZ5UvHjLIUfocA6c9p9Y5cpwN3B/4iYw/oJvndtm1a6n8SkCqjtJKRVjTuAfxlIeNbPq40xrRknVZZ0qeU+CZ/SczS0uuvgH8LnL0B40tGnarJrXDjZbi0/Ps7dQW+3l7je7RRvgH40/q3iHTirWYGjfaQbXumjelm/X1NhjxplCWN9RHANiDt3rT8VpmSxpc4p4oyVbyp2viXwKsy9jBuEp+2bdMEf4f6OlHaHjLtdpUqytL4LR/fJcATgf9SCL8/qY07rRSkz8XA4+tr5evtUd02Zm10yXxS5bb8MolS4vu8+heg9AvHXWuTTJcoZn1+pGul6RfGlFv/q76nIF0iyeY0zxVloz1k22TvhGbW39dkyJNGWcpY7wm8o/6t/rqxAyplfOm6397AZ4HNtRGlyiMtOa23h3GT+LRt+zdA+kVj/JNeOnDbQsaX+KWbT9JNPOk3/XTjVprA0nJnCfwOqq+Pp2ot3bj1obqyLGV8Ka7pJqOvTLzHt5TzI80raYkzGWW63ntBPc6Uk7OMbyqWDgU+Xi8Dp+uUqbrN5jQPRpnuSPz0xGSxH/CD+uaGyX1m205K6+ka7Vm7Xmcd/XspbKYdTvpNPv32Of65U73MeVJdZY7vEdwRluxu0kSa7upL1WS6WSyNNb18vMRYj1eUpYwv/VKR7hpOk9iVwAuA9Kq9UsaXEiFN8mlST7+opfGN3zU8q7ljrYRMlyVSzv3BWKNS+P028F7gX9fn7Gvr+wxmPb7fqq99/x7wpXoZNt39mj2OeTDK7FnPhhKQgAQkIIFJAhqlOSEBCUhAAhJYg4BGaXpIQAISkIAENEpzQAISkIAEJNCOgBVlO26qJCABCUhgTgholHMSaA9TAhKQgATaEdAo23FTJQEJSEACc0JAo5yTQHuYEpCABCTQjoBG2Y6bKgns6QSWN6NIu5icu6cfrMcngbUIaJTmhwQksBoBjdK8kEBNQKM0FSQwPAJpA+z/Wu/Zml4U/Z/q9+ulPVPTNmHp/Xv/BPhz4F314aWX1b68fvtFerVQetNEerdi+qTXW6Wt2v5pvR/ms+vN3dPWj+kVRc+p+0ttlvsbHjVHLIGWBDTKluCUSWCDCPxhvYn4o4BvAl8EFuo9LJOxvb9+L2Hagza96SK9tSG91Pfz9Z6lb6v3V037XqYNyNPbHNJm1f+x3sc0bbCdNqJP74VM/b2v3uM0vXXhX9Sbg2/Qofu1EtgYAhrlxnD3WyXQlkCqGFNlOP45A0ivlkvGdmRtcukdfOnVQskQk2HuANKG8j8GjgZOravQZLjHA/sCPxnrdHnp9Sn1OwXTmyBSu/S2lF+3Hbw6CQyRgEY5xKg55nkmsPxGlWnGtmyUT68rx8OAdENOerNJWlpNbzlZNtH0mqtkotvr112ll08vfyavUaZ3gqbKNb2iK72o2o8E5oaARjk3ofZA9xAC6XVKnwBeVFeO6V2Yqcq8tK4oU6WYXjic3u2YqsW09JpeB5auR74UOLmuJtOS672BBwOfAp4PnAVcCKTrnun1V6lCXb7rVaPcQxLIw2hOQKNszkyFBDaaQHo/YjLK29fLoskYU+WYjO2N9dLqb+obdNI1xvRJ705MZrc/8GXgT4GL6n9LNwelm3vS0mx6H+RzgftqlBsdZr+/FAIaZSmRcBwSiBFYXipNS6kfjXWlWgISGCegUZoPEtgzCPjc454RR4+iQAIaZYFBcUgSaEFAo2wBTYkEcgholDmUbCMBCUhAAnNLQKOc29B74BKQgAQkkENAo8yhZBsJSEACEphbAhrl3IbeA5eABCQggRwCGmUOJdtIQAISkMDcEvj/bAzS/QHkyUQAAAAASUVORK5CYII=",
"text/plain": [
"<VegaLite 2 object>\n",
"\n",
"If you see this message, it means the renderer has not been properly enabled\n",
"for the frontend that you are using. For more information, see\n",
"https://altair-viz.github.io/user_guide/troubleshooting.html\n"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alt.Chart(pd.DataFrame(enumerate(bbc_training_losses), columns=[\"epoch\", \"training_loss\"])).mark_bar().encode(x=\"epoch\", y=\"training_loss\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the reduced dimensionality paragraph vectors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"bbc_2d = pca_2d(bbc_model.paragraph_matrix.data, bbc_df.group.to_numpy())\n",
"chart = alt.Chart(bbc_2d).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")\n",
"# Uncomment to print chart inline, but beware it will inflate the notebook size\n",
"# chart"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`2-component PCA, explains 2.65% of variance`\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These results aren't great, but we can see the beginnings of separation. If we look at just two topics it becomes more obvious."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"chart = alt.Chart(bbc_2d[bbc_2d[\"group\"].isin([\"sport\", \"business\"])]).mark_point().encode(x=\"x\", y=\"y\", color=\"group\")\n",
"# Uncomment to print chart inline, but beware it will inflate the notebook size\n",
"# chart"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Likewise we can see sorting by similarity produces reasonable, but not ideal, results."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>doc_id</th>\n",
" <th>similarity</th>\n",
" <th>text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0</td>\n",
" <td>1.000000</td>\n",
" <td>Claxton hunting first major medal British hurdler Sarah Claxton is confident she can win her fi...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>37</th>\n",
" <td>37</td>\n",
" <td>0.504319</td>\n",
" <td>Radcliffe proves doubters wrong This won't go down as one of the greatest marathons of Paula's ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>41</th>\n",
" <td>41</td>\n",
" <td>0.499603</td>\n",
" <td>Radcliffe enjoys winning comeback Paula Radcliffe made a triumphant return to competitive runni...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1545</th>\n",
" <td>1545</td>\n",
" <td>0.499484</td>\n",
" <td>Search wars hit desktop PCs Another front in the on-going battle between Microsoft and Google i...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1266</th>\n",
" <td>1266</td>\n",
" <td>0.490500</td>\n",
" <td>Student 'inequality' exposed Teenagers from well-off backgrounds are six times more likely to g...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>19</th>\n",
" <td>19</td>\n",
" <td>0.442955</td>\n",
" <td>Edwards tips Idowu for Euro gold World outdoor triple jump record holder and BBC pundit Jonatha...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>348</th>\n",
" <td>348</td>\n",
" <td>0.430447</td>\n",
" <td>Italy aim to rattle England Italy coach John Kirwan believes his side can upset England as the ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>251</th>\n",
" <td>251</td>\n",
" <td>0.429918</td>\n",
" <td>Ferguson rues failure to cut gap Boss Sir Alex Ferguson was left ruing Manchester United's fail...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>24</th>\n",
" <td>24</td>\n",
" <td>0.429485</td>\n",
" <td>El Guerrouj targets cross country Double Olympic champion Hicham El Guerrouj is set to make a r...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>464</th>\n",
" <td>464</td>\n",
" <td>0.412518</td>\n",
" <td>Henin-Hardenne beaten on comeback Justine Henin-Hardenne lost to Elena Dementieva in a comeback...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" doc_id similarity \\\n",
"0 0 1.000000 \n",
"37 37 0.504319 \n",
"41 41 0.499603 \n",
"1545 1545 0.499484 \n",
"1266 1266 0.490500 \n",
"19 19 0.442955 \n",
"348 348 0.430447 \n",
"251 251 0.429918 \n",
"24 24 0.429485 \n",
"464 464 0.412518 \n",
"\n",
" text \n",
"0 Claxton hunting first major medal British hurdler Sarah Claxton is confident she can win her fi... \n",
"37 Radcliffe proves doubters wrong This won't go down as one of the greatest marathons of Paula's ... \n",
"41 Radcliffe enjoys winning comeback Paula Radcliffe made a triumphant return to competitive runni... \n",
"1545 Search wars hit desktop PCs Another front in the on-going battle between Microsoft and Google i... \n",
"1266 Student 'inequality' exposed Teenagers from well-off backgrounds are six times more likely to g... \n",
"19 Edwards tips Idowu for Euro gold World outdoor triple jump record holder and BBC pundit Jonatha... \n",
"348 Italy aim to rattle England Italy coach John Kirwan believes his side can upset England as the ... \n",
"251 Ferguson rues failure to cut gap Boss Sir Alex Ferguson was left ruing Manchester United's fail... \n",
"24 El Guerrouj targets cross country Double Olympic champion Hicham El Guerrouj is set to make a r... \n",
"464 Henin-Hardenne beaten on comeback Justine Henin-Hardenne lost to Elena Dementieva in a comeback... "
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"most_similar(bbc_model.paragraph_matrix.data, bbc_df, 0, n=10)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next steps\n",
"----------\n",
"\n",
"That's all for now! I honestly hope that was fun and educational (it was for me, anyway).\n",
"\n",
"But data science projects are notorious for never being finished. To carry this on, we could:\n",
"\n",
"- look for better hyperparameters, since the training loss remains quite high\n",
"- benchmark against `gensim` and Ilenic's PyTorch implementation; it should be very similar to the latter\n",
"- implement the inference step for new documents, which freezes the word and output matrices and adds a new column to the paragraph matrix\n",
"- use inferred paragraph vectors as the input for a topic classifier; looking at the business/sport plot above it could be quite successful\n",
"- try visualization with a better dimensionality reduction algorithm than PCA (I've used [LargeVis](https://arxiv.org/abs/1602.00370) in the past)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}