{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"\n",
"\n",
"***\n",
"***\n",
"\n",
"# Introduction to Word Embeddings\n",
"Analyzing Meaning through Word Embeddings\n",
"\n",
"***\n",
"***"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"**Using vectors to represent things**\n",
"- one of the most fascinating ideas in machine learning. \n",
"- Word2vec is a method to efficiently create word embeddings. \n",
" - Mikolov et al. (2013). [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)\n",
" - Mikolov et al. (2013). [Distributed representations of words and phrases and their compositionality](https://arxiv.org/pdf/1310.4546.pdf)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"***\n",
"\n",
"## The Geometry of Culture\n",
"\n",
"Analyzing Meaning through Word Embeddings\n",
"\n",
"***\n",
"***\n",
"\n",
"Austin C. Kozlowski; Matt Taddy; James A. Evans\n",
"\n",
"https://arxiv.org/abs/1803.09288\n",
"\n",
"Word embeddings represent **semantic relations** between words as **geometric relationships** between vectors in a high-dimensional space, operationalizing a relational model of meaning consistent with contemporary theories of identity and culture. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"- Dimensions induced by word differences (e.g. man - woman, rich - poor, black - white, liberal - conservative) in these vector spaces closely correspond to dimensions of cultural meaning, \n",
"- Macro-cultural investigation with a longitudinal analysis of the coevolution of gender and class associations in the United States over the 20th century \n",
"\n",
"The success of these high-dimensional models motivates a move towards \"high-dimensional theorizing\" of meanings, identities and cultural processes."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## HistWords \n",
"\n",
"HistWords is a collection of tools and datasets for analyzing language change using word vector embeddings. \n",
"\n",
"- The goal of this project is to facilitate quantitative research in diachronic linguistics, history, and the digital humanities.\n",
"\n",
"\n",
"- We used the historical word vectors in HistWords to study the semantic evolution of more than 30,000 words across 4 languages. \n",
"\n",
"- This study led us to propose two statistical laws that govern the evolution of word meaning \n",
"\n",
"\n",
"https://nlp.stanford.edu/projects/histwords/\n",
"\n",
"https://github.com/williamleif/histwords\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"**Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change**\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Word embeddings quantify 100 years of gender and ethnic stereotypes\n",
"\n",
"http://www.pnas.org/content/early/2018/03/30/1720347115\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## The Illustrated Word2vec\n",
"\n",
"Jay Alammar. https://jalammar.github.io/illustrated-word2vec/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Personality Embeddings\n",
"\n",
"> What are you like?\n",
"\n",
"**Big Five personality traits**: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism\n",
"- the five-factor model (FFM) \n",
"- **the OCEAN model**\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:08:26.103877Z",
"start_time": "2019-04-05T09:08:26.096732Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- 开放性(openness):具有想象、审美、情感丰富、求异、创造、智能等特质。\n",
"- 责任心(conscientiousness):显示胜任、公正、条理、尽职、成就、自律、谨慎、克制等特点。\n",
"- 外倾性(extraversion):表现出热情、社交、果断、活跃、冒险、乐观等特质。\n",
"- 宜人性(agreeableness):具有信任、利他、直率、依从、谦虚、移情等特质。\n",
"- 神经质或情绪稳定性(neuroticism):具有平衡焦虑、敌对、压抑、自我意识、冲动、脆弱等情绪的特质,即具有保持情绪稳定的能力。"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:37:14.866873Z",
"start_time": "2019-04-05T09:37:14.862287Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Personality Embeddings: What are you like?\n",
"jay = [-0.4, 0.8, 0.5, -0.2, 0.3]\n",
"john = [-0.3, 0.2, 0.3, -0.4, 0.9]\n",
"mike = [-0.5, -0.4, -0.2, 0.7, -0.1]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Cosine Similarity\n",
"The cosine of two non-zero vectors can be derived by using the Euclidean dot product formula:\n",
"\n",
"$$\n",
"\\mathbf{A}\\cdot\\mathbf{B}\n",
"=\\left\\|\\mathbf{A}\\right\\|\\left\\|\\mathbf{B}\\right\\|\\cos\\theta\n",
"$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\n",
"\\text{similarity} = \\cos(\\theta) = {\\mathbf{A} \\cdot \\mathbf{B} \\over \\|\\mathbf{A}\\| \\|\\mathbf{B}\\|} = \\frac{ \\sum\\limits_{i=1}^{n}{A_i B_i} }{ \\sqrt{\\sum\\limits_{i=1}^{n}{A_i^2}} \\sqrt{\\sum\\limits_{i=1}^{n}{B_i^2}} },\n",
"$$\n",
"\n",
"where $A_i$ and $B_i$ are components of vector $A$ and $B$ respectively."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:17:02.584345Z",
"start_time": "2019-04-05T09:17:02.577116Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-0.4999999999999999"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from numpy import dot\n",
"from numpy.linalg import norm\n",
"\n",
"def cos_sim(a, b):\n",
" return dot(a, b)/(norm(a)*norm(b))\n",
"\n",
"cos_sim([1, 0, -1], [-1,-1, 0])"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:14:47.865304Z",
"start_time": "2019-04-05T09:14:47.857879Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.5]])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.metrics.pairwise import cosine_similarity\n",
"\n",
"cosine_similarity([[1, 0, -1]], [[-1,-1, 0]])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$CosineDistance = 1- CosineSimilarity$$"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:25:29.120401Z",
"start_time": "2019-04-05T09:25:29.113611Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-0.5"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from scipy import spatial\n",
"# spatial.distance.cosine computes \n",
"# the Cosine distance between 1-D arrays.\n",
"1 - spatial.distance.cosine([1, 0, -1], [-1,-1, 0])"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:17:48.677658Z",
"start_time": "2019-04-05T09:17:48.672323Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.6582337075311759"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cos_sim(jay, john)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:18:04.442036Z",
"start_time": "2019-04-05T09:18:04.437385Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"-0.3683509554826695"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cos_sim(jay, mike)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Cosine similarity works for any number of dimensions. \n",
"- We can represent people (and things) as vectors of numbers (which is great for machines!).\n",
"- We can easily calculate how similar vectors are to each other."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Word Embeddings\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Google News Word2Vec\n",
"\n",
"You can download Google’s pre-trained model here.\n",
"\n",
"- It’s 1.5GB! \n",
"- It includes word vectors for a vocabulary of 3 million words and phrases \n",
"- It is trained on roughly 100 billion words from a Google News dataset. \n",
"- The vector length is 300 features.\n",
"\n",
"http://mccormickml.com/2016/04/12/googles-pretrained-word2vec-model-in-python/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Using the **Gensim** library in python, we can \n",
"- find the most similar words to the resulting vector. \n",
"- add and subtract word vectors, \n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:48:32.663015Z",
"start_time": "2019-04-05T09:46:33.523568Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import gensim\n",
"# Load Google's pre-trained Word2Vec model.\n",
"filepath = '/Users/datalab/bigdata/GoogleNews-vectors-negative300.bin'\n",
"model = gensim.models.KeyedVectors.load_word2vec_format(filepath, binary=True) "
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:54:56.388102Z",
"start_time": "2019-04-05T09:54:56.383705Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0.24316406, -0.07714844, -0.10302734, -0.10742188, 0.11816406,\n",
" -0.10742188, -0.11425781, 0.02563477, 0.11181641, 0.04858398],\n",
" dtype=float32)"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model['woman'][:10]"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:52:13.063134Z",
"start_time": "2019-04-05T09:52:12.809519Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[('man', 0.7664012312889099),\n",
" ('girl', 0.7494640946388245),\n",
" ('teenage_girl', 0.7336829900741577),\n",
" ('teenager', 0.631708562374115),\n",
" ('lady', 0.6288785934448242),\n",
" ('teenaged_girl', 0.6141784191131592),\n",
" ('mother', 0.607630729675293),\n",
" ('policewoman', 0.6069462299346924),\n",
" ('boy', 0.5975908041000366),\n",
" ('Woman', 0.5770983099937439)]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.most_similar('woman')"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:51:43.612745Z",
"start_time": "2019-04-05T09:51:43.607316Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.76640123"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.similarity('woman', 'man')"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:54:38.743342Z",
"start_time": "2019-04-05T09:54:38.738722Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.76640123"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cos_sim(model['woman'], model['man'])"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T09:49:03.690483Z",
"start_time": "2019-04-05T09:48:48.812748Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[('queen', 0.7118192911148071),\n",
" ('monarch', 0.6189674139022827),\n",
" ('princess', 0.5902431607246399),\n",
" ('crown_prince', 0.5499460697174072),\n",
" ('prince', 0.5377321243286133)]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.most_similar(positive=['woman', 'king'], negative=['man'], topn=5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$King- Queen = Man - Woman$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Now that we’ve looked at trained word embeddings, \n",
"\n",
"- let’s learn more about the training process. \n",
"- But before we get to word2vec, we need to look at a conceptual parent of word embeddings: **the neural language model**.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# The neural language model\n",
"\n",
"“You shall know a word by the company it keeps” J.R. Firth\n",
"\n",
"\n",
"\n",
"> Bengio 2003 A Neural Probabilistic Language Model. Journal of Machine Learning Research. 3:1137–1155\n",
"\n",
"After being trained, early neural language models (Bengio 2003) would calculate a prediction in three steps:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T10:08:02.288041Z",
"start_time": "2019-04-05T10:08:02.282384Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The output of the neural language model is a probability score for all the words the model knows. \n",
"- We're referring to the probability as a percentage here, \n",
"- but 40% would actually be represented as 0.4 in the output vector.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Language Model Training\n",
"\n",
"- We get a lot of text data (say, all Wikipedia articles, for example). then\n",
"- We have a window (say, of three words) that we slide against all of that text.\n",
"- The sliding window generates training samples for our model"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"As this window slides against the text, we (virtually) generate a dataset that we use to train a model. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Instead of only looking two words before the target word, we can also look at two words after it.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"If we do this, the dataset we’re virtually building and training the model against would look like this:\n",
"\n",
"\n",
"\n",
"This is called a **Continuous Bag of Words** (CBOW) https://arxiv.org/pdf/1301.3781.pdf"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Skip-gram\n",
"Instead of guessing a word based on its context (the words before and after it), this other architecture tries to guess neighboring words using the current word. \n",
"\n",
"\n",
"\n",
"https://arxiv.org/pdf/1301.3781.pdf"
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T10:48:55.984057Z",
"start_time": "2019-04-05T10:48:55.979592Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The pink boxes are in different shades because this sliding window actually creates four separate samples in our training dataset.\n",
"\n",
"\n",
"- We then slide our window to the next position:\n",
"- Which generates our next four examples:\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T10:45:28.851644Z",
"start_time": "2019-04-05T10:45:28.847193Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Negative Sampling\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"And switch it to a model that takes the input and output word, and outputs a score indicating **if they’re neighbors or not** \n",
"- 0 for “not neighbors”, 1 for “neighbors”.\n",
"\n",
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"we need to introduce negative samples to our dataset\n",
"- samples of words that are not neighbors. \n",
"- Our model needs to return 0 for those samples.\n",
"- This leads to a great tradeoff of computational and statistical efficiency.\n",
"\n",
"## Skipgram with Negative Sampling (SGNS)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-05T10:48:55.984057Z",
"start_time": "2019-04-05T10:48:55.979592Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Word2vec Training Process\n",
"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Pytorch word2vec \n",
"https://github.com/jojonki/word2vec-pytorch/blob/master/word2vec.ipynb\n",
"\n",
"https://github.com/bamtercelboo/pytorch_word2vec/blob/master/model.py"
]
},
{
"cell_type": "code",
"execution_count": 118,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T07:43:56.056098Z",
"start_time": "2019-04-07T07:43:56.045854Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 118,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# see http://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html\n",
"import torch\n",
"from torch.autograd import Variable\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"\n",
"torch.manual_seed(1)"
]
},
{
"cell_type": "code",
"execution_count": 202,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:08.645360Z",
"start_time": "2019-04-07T09:08:08.641731Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"text = \"\"\"We are about to study the idea of a computational process.\n",
"Computational processes are abstract beings that inhabit computers.\n",
"As they evolve, processes manipulate other abstract things called data.\n",
"The evolution of a process is directed by a pattern of rules\n",
"called a program. People create programs to direct processes. In effect,\n",
"we conjure the spirits of the computer with our spells.\"\"\"\n",
"\n",
"text = text.replace(',', '').replace('.', '').lower().split()"
]
},
{
"cell_type": "code",
"execution_count": 203,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:11.597040Z",
"start_time": "2019-04-07T09:08:11.590972Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"vocab_size: 44\n"
]
}
],
"source": [
"# By deriving a set from `raw_text`, we deduplicate the array\n",
"vocab = set(text)\n",
"vocab_size = len(vocab)\n",
"print('vocab_size:', vocab_size)\n",
"\n",
"w2i = {w: i for i, w in enumerate(vocab)}\n",
"i2w = {i: w for i, w in enumerate(vocab)}"
]
},
{
"cell_type": "code",
"execution_count": 204,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:13.785456Z",
"start_time": "2019-04-07T09:08:13.773647Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cbow sample (['we', 'are', 'to', 'study'], 'about')\n"
]
}
],
"source": [
"# context window size is two\n",
"def create_cbow_dataset(text):\n",
" data = []\n",
" for i in range(2, len(text) - 2):\n",
" context = [text[i - 2], text[i - 1],\n",
" text[i + 1], text[i + 2]]\n",
" target = text[i]\n",
" data.append((context, target))\n",
" return data\n",
"\n",
"cbow_train = create_cbow_dataset(text)\n",
"print('cbow sample', cbow_train[0])\n"
]
},
{
"cell_type": "code",
"execution_count": 205,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:14.703431Z",
"start_time": "2019-04-07T09:08:14.671677Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"skipgram sample ('about', 'we', 1)\n"
]
}
],
"source": [
"def create_skipgram_dataset(text):\n",
" import random\n",
" data = []\n",
" for i in range(2, len(text) - 2):\n",
" data.append((text[i], text[i-2], 1))\n",
" data.append((text[i], text[i-1], 1))\n",
" data.append((text[i], text[i+1], 1))\n",
" data.append((text[i], text[i+2], 1))\n",
" # negative sampling\n",
" for _ in range(4):\n",
" if random.random() < 0.5 or i >= len(text) - 3:\n",
" rand_id = random.randint(0, i-1)\n",
" else:\n",
" rand_id = random.randint(i+3, len(text)-1)\n",
" data.append((text[i], text[rand_id], 0))\n",
" return data\n",
"\n",
"\n",
"skipgram_train = create_skipgram_dataset(text)\n",
"print('skipgram sample', skipgram_train[0])"
]
},
{
"cell_type": "code",
"execution_count": 206,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:15.358380Z",
"start_time": "2019-04-07T09:08:15.335746Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"class CBOW(nn.Module):\n",
" def __init__(self, vocab_size, embd_size, context_size, hidden_size):\n",
" super(CBOW, self).__init__()\n",
" self.embeddings = nn.Embedding(vocab_size, embd_size)\n",
" self.linear1 = nn.Linear(2*context_size*embd_size, hidden_size)\n",
" self.linear2 = nn.Linear(hidden_size, vocab_size)\n",
" \n",
" def forward(self, inputs):\n",
" embedded = self.embeddings(inputs).view((1, -1))\n",
" hid = F.relu(self.linear1(embedded))\n",
" out = self.linear2(hid)\n",
" log_probs = F.log_softmax(out, dim = 1)\n",
" return log_probs\n",
" \n",
" def extract(self, inputs):\n",
" embeds = self.embeddings(inputs)\n",
" return embeds"
]
},
{
"cell_type": "code",
"execution_count": 207,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:15.952938Z",
"start_time": "2019-04-07T09:08:15.934330Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"class SkipGram(nn.Module):\n",
" def __init__(self, vocab_size, embd_size):\n",
" super(SkipGram, self).__init__()\n",
" self.embeddings = nn.Embedding(vocab_size, embd_size)\n",
" \n",
" def forward(self, focus, context):\n",
" embed_focus = self.embeddings(focus).view((1, -1)) # input\n",
" embed_ctx = self.embeddings(context).view((1, -1)) # output\n",
" score = torch.mm(embed_focus, torch.t(embed_ctx)) # input*output\n",
" log_probs = F.logsigmoid(score) # sigmoid\n",
" return log_probs\n",
" \n",
" def extract(self, focus):\n",
" embed_focus = self.embeddings(focus)\n",
" return embed_focus"
]
},
{
"cell_type": "markdown",
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T08:24:20.526644Z",
"start_time": "2019-04-07T08:24:20.523217Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"`torch.mm` Performs a matrix multiplication of the matrices \n",
"\n",
"`torch.t` Expects :attr:`input` to be a matrix (2-D tensor) and transposes dimensions 0\n",
"and 1. Can be seen as a short-hand function for ``transpose(input, 0, 1)``."
]
},
{
"cell_type": "code",
"execution_count": 208,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:17.249149Z",
"start_time": "2019-04-07T09:08:17.245376Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"embd_size = 100\n",
"learning_rate = 0.001\n",
"n_epoch = 30\n",
"CONTEXT_SIZE = 2 # 2 words to the left, 2 to the right"
]
},
{
"cell_type": "code",
"execution_count": 209,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:18.005531Z",
"start_time": "2019-04-07T09:08:17.973458Z"
},
"code_folding": [],
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def train_cbow():\n",
" hidden_size = 64\n",
" losses = []\n",
" loss_fn = nn.NLLLoss()\n",
" model = CBOW(vocab_size, embd_size, CONTEXT_SIZE, hidden_size)\n",
" print(model)\n",
" optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
" for epoch in range(n_epoch):\n",
" total_loss = .0\n",
" for context, target in cbow_train:\n",
" ctx_idxs = [w2i[w] for w in context]\n",
" ctx_var = Variable(torch.LongTensor(ctx_idxs))\n",
"\n",
" model.zero_grad()\n",
" log_probs = model(ctx_var)\n",
"\n",
" loss = loss_fn(log_probs, Variable(torch.LongTensor([w2i[target]])))\n",
"\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" total_loss += loss.data.item()\n",
" losses.append(total_loss)\n",
" return model, losses "
]
},
{
"cell_type": "code",
"execution_count": 210,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:18.803221Z",
"start_time": "2019-04-07T09:08:18.773964Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def train_skipgram():\n",
" losses = []\n",
" loss_fn = nn.MSELoss()\n",
" model = SkipGram(vocab_size, embd_size)\n",
" print(model)\n",
" optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
" \n",
" for epoch in range(n_epoch):\n",
" total_loss = .0\n",
" for in_w, out_w, target in skipgram_train:\n",
" in_w_var = Variable(torch.LongTensor([w2i[in_w]]))\n",
" out_w_var = Variable(torch.LongTensor([w2i[out_w]]))\n",
" \n",
" model.zero_grad()\n",
" log_probs = model(in_w_var, out_w_var)\n",
" loss = loss_fn(log_probs[0], Variable(torch.Tensor([target])))\n",
" \n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" total_loss += loss.data.item()\n",
" losses.append(total_loss)\n",
" return model, losses"
]
},
{
"cell_type": "code",
"execution_count": 211,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:24.429022Z",
"start_time": "2019-04-07T09:08:19.718142Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CBOW(\n",
" (embeddings): Embedding(44, 100)\n",
" (linear1): Linear(in_features=400, out_features=64, bias=True)\n",
" (linear2): Linear(in_features=64, out_features=44, bias=True)\n",
")\n",
"SkipGram(\n",
" (embeddings): Embedding(44, 100)\n",
")\n"
]
}
],
"source": [
"cbow_model, cbow_losses = train_cbow()\n",
"sg_model, sg_losses = train_skipgram()"
]
},
{
"cell_type": "code",
"execution_count": 212,
"metadata": {
"ExecuteTime": {
"end_time": "2019-04-07T09:08:24.968373Z",
"start_time": "2019-04-07T09:08:24.430942Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAEYCAYAAABBfQDEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzs3X2c1XP+//HHa6qZpBldSOhqSKKS6JKkXCyJLWGJqFb0Q6xYF1m+a1kti3V9GVJ2WRdLtCRiXSxKVxIqSrJNulK6IqXm9fvj85npNJ0zM82cmXM+M8/77XZu55zX530+53Vm+Myr93l/Xh9zd0REREREJJCR6gRERERERNKJCmQRERERkRgqkEVEREREYqhAFhERERGJoQJZRERERCSGCmQRERERkRgqkEVEREREYqhAFhERERGJoQJZRERERCRGzVQnALDnnnt6bm5uqtMQESm3mTNnfu/ujVKdRzLpGC0iVUVpj9FpUSDn5uYyY8aMVKchIlJuZvZtqnNINh2jRaSqKO0xWkssRERERERiqEAWEREREYmhAllEREREJEZarEEWkfh++eUX8vLy+Pnnn1OdihRRu3ZtmjZtSq1atVKdioiUgY6vVVt5j9EqkEXSWF5eHtnZ2eTm5mJmqU5HQu7O6tWrycvLY7/99kvafs2sGfAU0BhwYLS732tmDYDngFxgMXCmu/9gwX8U9wJ9gJ+AIe4+K9zXYOCGcNe3uPu4MN4RGAvsBkwELnd3T9qHEIkIHV+rrmQco7XEQiSN/fzzzzRs2FAH7zRjZjRs2LAiZp62Ar939zZAN2C4mbUBRgJvu3sr4O3wOcBJQKvwNgx4OMyvAXAj0BXoAtxoZvXD1zwMXBjzut7J/hAiUaDja9WVjGN0NAvkp5+G3FzIyAjun3461RmJVBgdvNNTRfxe3H1ZwQywu28A5gFNgH7AuHDYOODU8HE/4CkPTAXqmdk+wInAZHdf4+4/AJOB3uG2HHefGs4aPxWzr6TY+869sZtsp9ved+6dzLcRSQodX6uu8v5uo7fE4umnYdgw+Omn4Pm33wbPAQYOTF1eIiJJZGa5wGHAx0Bjd18WblpOsAQDguJ5SczL8sJYcfG8OPF47z+MYFaa5s2blzrvFT+u2KW4iEg6it4M8vXXby+OC/z0UxAXqe4q4NuV5cuXM2DAAFq2bEnHjh3p06cPX331FYsXL2a33XajQ4cOHHrooRx55JF8+eWXha/74IMP6NKlCwcddBAHHXQQo0ePBmDt2rU0bNiQgmWvU6ZMwczIywvqtnXr1tGgQQPy8/N3yONPf/oTd955Z7k/TxSYWV3gRWCEu6+P3RbO/Fb4mmF3H+3undy9U6NGVerCgCK7rKK+GRk1ahRt27alffv2dOjQgY8//hgILs7z/fff7zT+yCOPLHGfGzdu5OKLL6Zly5YcfvjhdOzYkccee6xceZbG4sWLadeuXYW/T2WJXoH8v//tWlykuij4duXbb8F9+7cr5SiS3Z3+/fvTq1cvvv76a2bOnMmtt97KihXBbGDLli2ZPXs2n376KYMHD+Yvf/kLEBTV55xzDo888gjz58/ngw8+4NFHH+W1116jXr167LPPPsybNw+Ajz76iMMOO4yPPvoIgKlTp9KlSxcyMqJ3eEoGM6tFUBw/7e4vheEV4fIIwvuVYXwp0Czm5U3DWHHxpnHiIlKMivhmZMqUKbz66qvMmjWLOXPm8NZbb9GsWbNiX1NwnCzOBRdcQP369VmwYAGzZs1i0qRJrFmzZqdxW7duLXPu1UGJf4HMrJmZvWNmc83sCzO7PIzfYWbzzWyOmY03s3oxr7nOzBaa2ZdmdmJSM070Vd+++yb1bUTSzogR0KtX4tvQofG/XRk6NPFrRowo9i3feecdatWqxUUXXVQYO/TQQ+nRo8dOY9evX0/9+sF5YA8++CBDhgzh8MMPB2DPPffk9ttv57bbbgOCWZCCA/1HH33EFVdcscPz7t27l/rHctddd9GuXTvatWvHPffcA8CPP/7IySefzKGHHkq7du147rnnABg5ciRt2rShffv2XHXVVQCsWrWK008/nc6dO9O5c2c+/PBDAN577z06dOhAhw4dOOyww9iwYUOpcyqrsCvFE8A8d78rZtMEYHD4eDDwSkx8kAW6AevCpRhvACeYWf3w5LwTgDfCbevNrFv4XoNi9iVSbY2YNIJeY3slvBUn0WtGTCr++Lps2TL23HNPsrKygOA4uW+RWmbTpk2cdNJJhTPAdevWBeDdd9/l6KOP5uSTT6Z169ZcdNFF5Ofn8/XXXzNt2jRuueWWwkmGRo0ace211xa+rkePHvTt25c2bdoAcOqpp9KxY0fatm1b+E1fwXtdffXVtG3bluOPP55p06bRq1cv9t9/fyZMmFDKnyzMnj2bbt260b59e/r3788PP/wAwH333Vd4PB4wYACQ+Lh7xx130LlzZ9q3b8+NN94IJD7OJ0tp1iAXnFU9y8yygZlmNpngpI/r3H2rmf0VuA64NjzjegDQFtgXeMvMDnT3bUnJeNSoHdcgF/j5Z5g/Hw46KClvIxI5mzfvWrwUPv/8czp27Jhw+9dff02HDh3YsGEDP/30U+HXg1988QWDBw/eYWynTp344osvAOjevTvvvfceF1xwAYsWLeI3v/kNjz76KBAUyCNHjqQ0Zs6cyZNPPsnHH3+Mu9O1a1d69uzJokWL2HfffXnttdeAYNnG6tWrGT9+PPPnz8fMWLt2LQCXX345V1xxBUcddRT/+9//OPHEE5k3bx533nknDz74IN27d2fjxo3Url171354ZdMdOA/4zMxmh7E/ALcBz5vZUOBb4Mxw20SCFm8LCdq8/RbA3deY2Z+B6eG4m929YArpEra3eXs9vIlIJTvhhBO4+eabOfDAAzn++OM566yz6NmzZ+H2jRs3MmDAAAYNGsSgQYN2ev20adOYO3cuLVq0oHfv3rz00ktkZmZy6KGHFvsN3KxZs/j8888L25+NGTOGBg0asGnTJjp37szpp59Ow4YN+fHHHzn22GO544476N+/PzfccAOTJ09m7ty5DB48mL59+5bqcw4aNIj777+fnj178sc//pGbbrqJe+65h9tuu41vvvmGrKyswuNxvOPum2++yYIFC5g2bRruTt++fXn//fdZtWrVTsf5ZCqxQA5nHJaFjzeY2Tygibu/GTNsKnBG+Lgf8Ky7bwa+MbOFBG2GpiQl44IT8a6/PlhW0bw5XHAB3H8/dO8Or74KRxyRlLcSSSvh7GhCubnBsoqiWrSAd9+tiIwKl1gAPPfccwwbNoxJkyaV+LojjzySW2+9lW+++Ybc3Fxq166Nu7Nx40ZmzpxJ165dS/X+H3zwAf3792f33XcH4LTTTuO///0vvXv35ve//z3XXnstp5xyCj169GDr1q3Url2boUOHcsopp3DKKacA8NZbbzF37tzCfa5fv56NGzfSvXt3rrzySgYOHMhpp51G06ZN4+aQTO7+AZDo1Ovj4ox3YHiCfY0BxsSJzwAqbKFg490bx/3aufHujeOMFkkP9/Qu/vhqNyXuiPDukHfL9J5169Zl5syZ/Pe//+Wdd97hrLPO4rbbbmPIkCEA9OvXj2uuuYaBCRoQdOnShf333x+As88+mw8++IBjjz12hzGjRo3ihRdeYOXKlXz33XeFr4vtDXzfffcxfvx4AJYsWcKCBQto2LAhmZmZ9O4ddIE85JBDyMrKolatWhxyyCEsXry4VJ9x3bp1rF27trDwHzx4ML/5zW8AaN++PQMHDuTUU0/l1FODZjrxjrtvvvkmb775JocddhgQ/MNhwYIF9OjRY6fjfDLt0iK/ImdVxzqf7bMQic6eLrqvYWY2w8xmrFq1alfSCIrkxYshPz+4v+EG+OgjaNAAjjsOdmHqX6TKGDUK6tTZMVanThAvo7Zt2zJz5sxSjS34Vz1AmzZtdnrdzJkzadu2LQCtWrVi7dq1/Pvf/+aI8B+0HTt25MknnyQ3N7fwa8SyOvDAA5k1axaHHHIIN9xwAzfffDM1a9Zk2rRpnHHGGbz66quFB/78/HymTp3K7NmzmT17NkuXLqVu3bqMHDmSxx9/nE2bNtG9e3fmz59frpyqi+VXLcdvdB49JfhGYOmVS/EbneVXLU9xZiLpp0aNGvTq1YubbrqJBx54gBdffLFwW/fu3Zk0aVLhCc1FFW1jZma0adOGTz/9tPAk5+uvv57Zs2ezfv32c30LJhQgWHLx1ltvMWXKFD799FMOO+ywwt7BtWrVKnyPjIyMwqUgGRkZSVm//NprrzF8+HBmzZpF586d2bp1a9zjrrtz3XXXFR6jFy5cyNChQ+Me55Op1AVyorOqzex6gmUYu3QmUNLPkG7ZEj78ENq1g/79oRLO2BRJKwMHwujRwYyxWXA/enS52h8ee+yxbN68eYd1aXPmzOG///3vTmM/+OADWrZsCcDw4cMZO3Zs4ezy6tWrufbaa7nmmmsKx3fr1o177723sEA+4ogjuOeee3Zp/XGPHj14+eWX+emnn/jxxx8ZP348PXr04LvvvqNOnTqce+65XH311cyaNYuNGzeybt06+vTpw913382nn34KBF9z3n///YX7LMj566+/5pBDDuHaa6+lc+fOKpB3UU5WDgDrN68vYaRI+kv0DUh5vhn58ssvWbBgQeHz2bNn06JFi8LnN998M/Xr12f48LhfEjFt2jS++eYb8vPzee655zjqqKM44IAD6NSpEzfccAPbtgUrW3/++eeERfa6deuoX78+derUYf78+UydOrXMnyeePfbYg/r16xf+zfj73/9Oz549yc/PZ8mSJRxzzDH89a9/Zd26dWzcuDHucffEE09kzJgxbNy4EYClS5cWzogXPc4nU6n6ICc4qxozGwKcAhwXc6nSRGdPV7y99oL//AfOPDNYpzxpEsyYAUuWBEsxRo1Sr2Sp2gYOTOp/42bG+PHjGTFiBH/961+pXbs2ubm5hSfDFaxBdncyMzN5/PHHAdhnn334xz/+wYUXXsiGDRtwd0aMGMGvf/3rwn13796diRMn0qlTJyAokBctWlRsG6Nbbrml8L0huFTskCFD6NKlCxCcvX3YYYfxxhtvcPXVV5ORkUGtWrV4+OGH2bBhA/369Sv8Y3HXXcE5cPfddx/Dhw+nffv2bN26laOPPppHHnmEe+65h3feeYeMjAzatm3LSSedlLSfa3WgAlmqkor4BmTjxo1cdtllrF27lpo1a3LAAQfsMBkBcO+993L++edzzTXXcPvtt++wrXPnzlx66aUsXLiQY445hv79+wPw+OOPc/XVV3PAAQfQsGFDdtttt51eW6B379488sgjHHzwwbRu3Zpu3bqV6zN9+eWXOyxHu/vuuxk3bhwXXXQRP/30E/vvvz9PPvkk27Zt49xzz2XdunW4O7/73e+oV68e//d//7fTcTcrK4t58+YVTqbUrVuXf/zjHyxcuHCn43wyWaJ/VRQOCObXxwFr3H1ETLw3cBfQ091XxcTbAs8QrDvel+CyqK2KO0mvU6dOPmPGjPJ8jh398kuw1KLoLFedOuWeUROpTPPmzePggw9OdRqSQLzfj5nNdPdOKUqpQpTlGP3fb//L0WOPZvJ5kzl+/+MrKDORsovy8fXdd9/lzjvv5NVXX011KmmtPMfo0iyxKDir+lgzmx3e+gAPANnA5DD2CIC7fwE8D8wFJgHDk9bBorRq1Yp/spIuKCIiUik0gywiUVaaLhaJzqqeWMxrRgFlPzMoGZYsiR/XBUVERCpcQYG8YXPF948WqW569epFr169Up1GlVZ1L1WV6IIie5fvspAila2kZVCSGvq9FE8zyBIF+v+46irv77bqFsjxWl6ZwZo18Lr64ks01K5dm9WrV+sgnmbcndWrV1fWxUMiKTsrG1CBLOlLx9eqKxnH6FJ1sYikeBcUueoqeOIJ+PWv4YEHIObyuSLpqGnTpuTl5bHLvcKlwtWuXbtSLh4SVZk1Mqlds7YKZElbOr5WbeU9RlfdAhnit7waPBgGDICLL4aFC+H226GYSzKKpFKtWrV2uOKRSJTkZOWoQJa0peOrFKf6VYbZ2fDKKzB8OPztb/Cb38CTTwaX6c3ICO6f3qVrnoiISBw5WTms36ICWUSip2rPICdSsybcf39w9b0rr4SXXw4uWw1Be7hhw4LH6pcsIlJmmkEWkaiqfjPIBczgiiugUaPtxXEB9UsWESm37MxsFcgiEknVt0Au8P338ePqlywiUi6aQRaRqFKBnKhfcrNmlZuHiEgVk5OVowuFiEgkqUCO1y8ZgvXJW7ZUfj4iIlWEZpBFJKpUIA8cCKNHQ4sWwbrk5s2hb1945x048URYvTrVGYqIRJIKZBGJKhXIEBTJixcHJ+t9+23QBu6pp+Cjj6BbN5g/P9UZiohETk5WDpu3bWbz1s2pTkVEZJeoQE7kvPOCWeR164IieeRI9UoWEdkFOVk5AGzYonXIIhItKpCLc+SRMH16cHGRv/41mF12394rWUWyiCSZmY0xs5Vm9nlM7Dkzmx3eFpvZ7DCea2abYrY9EvOajmb2mZktNLP7zMzCeAMzm2xmC8L7+hX1WQoKZC2zEJGoUYFckoK1yUWpV7KIVIyxQO/YgLuf5e4d3L0D8CLwUszmrwu2uftFMfGHgQuBVuGtYJ8jgbfdvRXwdvi8QqhAFpGoUoFcGnl58ePqlSwiSebu7wNr4m0LZ4HPBP5Z3D7MbB8gx92nursDTwGnhpv7AePCx+Ni4kmXnZkNqEAWkehRgVwaiXolN25cuXmISHXXA1jh7gtiYvuZ2Sdm9p6Z9QhjTYDYf9nnhTGAxu6+LHy8HIh7IDOzYWY2w8xmrFq1qkzJagZZRKJKBXJpxOuVbBa0gPtnsRM5IiLJdDY7zh4vA5q7+2HAlcAzZpZT2p2Fs8ueYNtod+/k7p0aNWpUpmQLT9LTxUJEJGJUIJdG0V7JLVrAAw9A165wzjlwzTWwbVuqsxSRKszMagKnAc8VxNx9s7uvDh/PBL4GDgSWAk1jXt40jAGsCJdgFCzFWFlROWsGWUSiSgVyacX2Sl68GC65BN5+Gy6+GO64A/r0gR9+SHWWIlJ1HQ/Md/fCpRNm1sjMaoSP9yc4GW9RuIRivZl1C9ctDwJeCV82ARgcPh4cE086FcgiElUqkMsjMxMeeiiYXX7nHejcOWgHp37JIlJGZvZPYArQ2szyzGxouGkAO5+cdzQwJ2z79i/gIncvOMHvEuBxYCHBzPLrYfw24FdmtoCg6L6toj5LnVp1yLAMFcgiEjk1U51AlXDhhdCmTTCLPDKmY1JBv2QIZqBFRErg7mcniA+JE3uRoO1bvPEzgHZx4quB48qXZemYmS43LSKRpBnkZOnePbigSFHqlywi1VhOVg7rt6hAFpFoUYGcTN99Fz+ufskiUk1pBllEokgFcjIl6pe8996Vm4eISJrIzsxWgSwikaMCOZkS9UtetQqeeio1OYmIpJBmkEUkilQgJ1O8fskPPghHHQWDB8Pw4bBlS6qzFBGpNDlZObpQiIhEjgrkZCvaL/nii2HyZLjqqqAlXM+ewUVG1ApORKoBzSCLSBSpzVtlqFkzuJhI165w7rkwder2bWoFJyJVmApkEYkizSBXpjPOgAYNdo6rFZyIVFE5WTls2LKBfM9PdSoiIqVWYoFsZs3M7B0zm2tmX5jZ5WG8gZlNNrMF4X39MG5mdp+ZLTSzOWZ2eEV/iEhZvjx+XK3gRKQKKrjc9MYtG1OciYhI6ZVmBnkr8Ht3bwN0A4abWRtgJPC2u7cC3g6fA5wEtApvw4CHk551lKkVnIhUIwUFspZZiEiUlFggu/syd58VPt4AzAOaAP2AceGwccCp4eN+wFMemArUM7N9kp55VMVrBQfw/ffwz39Wfj4iIhVIBbKIRNEurUE2s1zgMOBjoLG7Lws3LQcah4+bAEtiXpYXxorua5iZzTCzGatWrdrFtCMsXiu4++8PTuA75xy47DK1ghORKiM7MxtQgSwi0VLqAtnM6gIvAiPcfYcjnbs74Lvyxu4+2t07uXunRo0a7cpLo69oK7hLL4X//AeuvDJoAdezJyxZUtJeRETSnmaQRSSKSlUgm1ktguL4aXd/KQyvKFg6Ed6vDONLgWYxL28axqQ4tWrB3/4GL7wAn38Ohx8O112nfskiEmkFBbIuFiIiUVKaLhYGPAHMc/e7YjZNAAaHjwcDr8TEB4XdLLoB62KWYkhJzjgDZsyAzEy47bagT7L79n7JKpJFJEI0gywiUVSaGeTuwHnAsWY2O7z1AW4DfmVmC4Djw+cAE4FFwELgMeCS5KddxbVuDTVq7BxXv2QRiRgVyCISRSVeSc/dPwAswebj4ox3YHg585K8vPhx9UsWkQjJztJJeiISPbqSXrpK1C+5Xr1gyYWISATUzKhJnVp1VCCLSKSoQE5X8fol16gBP/wAZ58NG3TCi4hEQ05WjgpkEYkUFcjpKl6/5LFj4dZbg04XnTrBZ5+lOksRkRLlZOWwfosKZBGJDhXI6axov+Rzz4WRI4OeyevXBxcXGTcu6GyhdnAiVYKZjTGzlWb2eUzsT2a2tMiJ0gXbrjOzhWb2pZmdGBPvHcYWmtnImPh+ZvZxGH/OzDIr+jNlZ2ZrBllEIkUFchT17AmffALdusGQITB4sNrBiVQdY4HeceJ3u3uH8DYRwMzaAAOAtuFrHjKzGmZWA3gQOAloA5wdjgX4a7ivA4AfgKEV+mnQEgsRiR4VyFG1994weTLk5MC2bTtuUzs4kchy9/eBNaUc3g941t03u/s3BO01u4S3he6+yN23AM8C/cK+9scC/wpfPw44NakfII6crBxdKEREIkUFcpTVqJH4ZD21gxOpai41sznhEoz6YawJEHtd+rwwlijeEFjr7luLxHdiZsPMbIaZzVi1alW5EtcMsohEjQrkqEvUDq5p08rNQ0Qq0sNAS6ADsAz4W0W/obuPdvdO7t6pUaNG5dqXCmQRiRoVyFEXrx1cga++qtxcRKRCuPsKd9/m7vkEVyjtEm5aCjSLGdo0jCWKrwbqmVnNIvEKVVAgu3q4i0hEqECOunjt4K66KliH3LEjPPNMqjMUkXIys31invYHCjpcTAAGmFmWme0HtAKmAdOBVmHHikyCE/kmhFc6fQc4I3z9YOCVis4/JyuHX/J/YfO2zRX9ViIiSaECuSoo2g7ujjtg9mzo0CHYNmwYbNqU6ixFpBTM7J/AFKC1meWZ2VDgdjP7zMzmAMcAVwC4+xfA88BcYBIwPJxp3gpcCrwBzAOeD8cCXAtcaWYLCdYkP1HRnyknKwfQ5aZFJDpqljxEIqlpU3jnHfjjH4OLi7z+elBAL1sWrFseNSoonkUkrbj72XHCCYtYdx8FjIoTnwhMjBNfxPYlGpUitkDea/e9KvOtRUTKRAVyVVazJvzlL0EbuNtv3x4v6JUMKpJFpMJlZ2YDmkEWkejQEovq4Lnndo6pV7KIVBItsRCRqFGBXB0k6omsXskiUglUIItI1KhArg4S9Up2h4ceCu5FRCpIQYGsq+mJSFSoQK4O4vVK3m03aN8ehg+H00+HH35ITW4iUuVpBllEokYFcnUQr1fyY4/BJ5/A3/4Gr74atIT76KNUZyoiVZAKZBGJGhXI1UXRXskDB0JGBlx5JXz4YdDx4uij4cwzgwI6IwNyc+Hpp1OcuIhEXe2atamZUVMFsohEhtq8CXTuHMwmn3givPDC9rjawYlIEphZ4eWmRUSiQDPIEsjJge++2zmudnAikgQ5WTms36ICWUSiQQWybLdkSfy42sGJSDllZ2ZrBllEIkMFsmyXqB1crVrBcgsRkTLSEgsRiRIVyLJdvHZwmZlB54sOHeDll1OTl4hEngpkEYkSFciyXbx2cGPGwOefQ8uW0L8/XHYZ/PxzqjMVkYjJycrRhUJEJDJUIMuO4rWDO+CAoBXciBHwwANw5JFB/+TcXLWDE5FS0QyyiESJCmQpnawsuPtumDABvvoKrroqWJfsvr0dnIpkEUlABbKIRIkKZNk1v/411Ku3c1zt4ESkGDlZOfz4y49sy9+W6lREREqkAll2Xbx+yaB2cCKSUMHlpjds0TpkEUl/KpBl1yVqB1e/frDkQkSkiIICWcssRCQKSiyQzWyMma00s89jYh3MbKqZzTazGWbWJYybmd1nZgvNbI6ZHV6RyUuKxGsHl5EBa9YEJ/Wt1x9AEdlRdmY2oAJZRKKhNDPIY4HeRWK3Aze5ewfgj+FzgJOAVuFtGPBwctKUtBKvHdzYsXDLLfDcc9CxI8yaleosRSIpwaTEHWY2P5x4GG9m9cJ4rpltCicrZpvZIzGv6Whmn4UTFveZmYXxBmY22cwWhPf1K+NzaQZZRKKkxALZ3d8H1hQNAznh4z2AgkWp/YCnPDAVqGdm+yQrWUkjRdvBnXdecJLeu+/Cpk1wxBEwaFBQPKsVnMiuGMvOkxKTgXbu3h74CrguZtvX7t4hvF0UE38YuJDtkxYF+xwJvO3urYC3w+cVTgWyiERJWdcgjwDuMLMlwJ1sP1g3AZbEjMsLYzsxs2Hh8owZq1atKmMaknZ69IDZs6FNG/j734MT99QKTqTU4k1KuPub7r41fDoVaFrcPsKJiRx3n+ruDjwFnBpu7geMCx+Pi4lXqMKT9HSxEBGJgLIWyBcDV7h7M+AK4Ild3YG7j3b3Tu7eqVGjRmVMQ9LSnnsG65GLUis4kWQ4H3g95vl+ZvaJmb1nZj3CWBOCCYoCsZMVjd19Wfh4OdC4QrMNaQZZRKKkrAXyYOCl8PELQJfw8VKgWcy4pmFMqpslS+LH1QpOpMzM7HpgK1DwVcwyoLm7HwZcCTxjZjmJXl9UOLsct/VMsr/lU4EsIlFS1gL5O6Bn+PhYYEH4eAIwKOxm0Q1YFzNTIdVJolZwtWvDDz9Ubi4iVYCZDQFOAQaGhS3uvtndV4ePZwJfAwcSTEzELsOInaxYUXBuSHi/Mt77JftbvrqZdQEVyCISDaVp8/ZPYArQ2szyzGwowYkffzOzT4G/EHSsAJgILAIWAo8Bl1RI1pL+4rWCq1ULfv4ZDj9sjcsWAAAgAElEQVQcpk9PTV4iEWRmvYFrgL7u/lNMvJGZ1Qgf709wMt6icGJivZl1C7tXDAJeCV82geBbQML7gniFqpFRg7qZdVUgi0gk1CxpgLufnWBTxzhjHRhe3qSkChg4MLi//vpgWUXz5kHR3LIlnHUWdO8Od94Jl10WtIoTEaBwUqIXsKeZ5QE3EpwInQVMDru1TQ07VhwN3GxmvwD5wEXuXnACwCUEHTF2I1izXLBu+Tbg+XCy41vgzEr4WECwzEIFsohEQYkFskiZDRy4vVCO9cknMGQIXH45vPce9O4dFM+xhXS814lUAwkmJeKeCO3uLwIvJtg2A2gXJ74aOK48OZZVdmY267eoQBaR9KcCWSpfgwbwyitw111w9dUwfvz2S1QXtIMDFckiVYxmkEUkKsp6kp5I+ZjB738Pe+21vTguoHZwIlWSCmQRiQoVyJJaK+OeQK92cCJVUE5Wji4UIiKRoAJZUitRO7i9967cPESkwmkGWUSiQgWypFa8dnAAq1bBE0/svPxCRCJLBbKIRIUKZEmtgQNh9Gho0SJYl9yiBTz4IPTsCRdcAL/9Lfz4Y6qzFJEkKCiQXf/wFZE0pwJZUm/gQFi8GPLzg/tLLoE33oAbb4SnnoKuXWHevFRnKSLllJOVwzbfxqatm1KdiohIsVQgS3qqUQP+9CeYNAlWrIDOnWH4cMjNhYyM4P7pp1OcpIjsipysHECXmxaR9KcCWdLbCSfA7Nmw777w0ENBn2T37f2SVSSLREZ2ZjagAllE0p8KZEl/TZrA5s07x9UvWSRSNIMsIlGhAlmiYcmS+HH1SxaJDBXIIhIVKpAlGhL1S87JgW3bKjcXESmTggJZFwsRkXSnAlmiIV6/5Bo1YN066NMHvv8+NXmJSKlpBllEokIFskRDvH7J48YFsXffhY4dYfr0VGcpIsVQgSwiUaECWaKjaL/kgQPhwgvhww+Dovmoo+D884PiWa3gRNKOCmQRiYqaqU5ApNw6dYJZs+CYY+DJJ7fHC1rBQVBMi0hKZdXMIrNGpgpkEUl7mkGWqqFBA1i7due4WsGJpJWCy02LiKQzFchSdagVnEjay87MZv0WFcgikt5UIEvVkagVXHY2bN1aubmISFyaQRaRKFCBLFVHvFZwNWvC+vXBJatXrEhNXiJSSAWyiESBCmSpOuK1ghs7NmgHN2UKHH44fPRRqrMUKZaZjTGzlWb2eUysgZlNNrMF4X39MG5mdp+ZLTSzOWZ2eMxrBofjF5jZ4Jh4RzP7LHzNfWZmlfn5crJydKEQEUl7KpClaonXCm7QIJg6FXbbDXr2hPvuC9q/5eaqHZyko7FA7yKxkcDb7t4KeDt8DnAS0Cq8DQMehqCgBm4EugJdgBsLiupwzIUxryv6XhVKM8giEgVq8ybVw6GHwowZMHgwXH55cBW+gktUqx2cpBF3f9/McouE+wG9wsfjgHeBa8P4U+7uwFQzq2dm+4RjJ7v7GgAzmwz0NrN3gRx3nxrGnwJOBV6vuE+0IxXIIhIFmkGW6qNePRg/PrgvKI4LqB2cpLfG7r4sfLwcaBw+bgLEtm/JC2PFxfPixHdiZsPMbIaZzVi1alX5P0FIBbKIRIEKZKleMjJg3br429QOTiIgnC32Snif0e7eyd07NWrUKGn7zcnKYdPWTfyy7Zek7VNEJNlUIEv1k6gdXLNmlZuHSOmtCJdOEN6vDONLgdj/cJuGseLiTePEK03B5aY3bNGJeiKSvlQgS/UTrx0cQP368a/GJ5J6E4CCThSDgVdi4oPCbhbdgHXhUow3gBPMrH54ct4JwBvhtvVm1i3sXjEoZl+VIjszG0DLLEQkralAluqnaDu45s3ht7+FuXOhc2f44otUZyjVmJn9E5gCtDazPDMbCtwG/MrMFgDHh88BJgKLgIXAY8AlAOHJeX8Gpoe3mwtO2AvHPB6+5msq8QQ92D6DrAJZRNKZulhI9TRw4M4dK4YOhTPOgK5dg97Jp5+emtykWnP3sxNsOi7OWAeGJ9jPGGBMnPgMoF15ciwPFcgiEgUlziDHa1ofxi8zs/lm9oWZ3R4Tvy5sQP+lmZ1YEUmLVIju3WHmTDjkkKBQ/sMf4O9/V79kkSRSgSwiUVCaGeSxwAPAUwUBMzuGoP/moe6+2cz2CuNtgAFAW2Bf4C0zO9Ddt+20V5F0tO++8O678Lvfwa23BoVxfn6wTf2SRcqt8CQ9XU1PRNJYiTPI7v4+sKZI+GLgNnffHI4pOKO6H/Csu292928I1rh1SWK+IhUvKwsefRQaNNheHBdQv2SRctEMsohEQVlP0jsQ6GFmH5vZe2bWOYwnak4vEj0//BA/rn7JImWmAllEoqCsBXJNoAHQDbgaeD5sGVRqFXWVJpGkUb9kkaTbPXN3DFOBLCJprawFch7wkgemAfnAniRuTr+TirpKk0jSJOqX3LAhrNcfd5GyyLAMsrOyVSCLSFora4H8MnAMgJkdCGQC3xM0rR9gZllmth/QCpiWjERFKl28fsmDB8NnnwWt4L76KtUZikRSdqYKZBFJb6Vp8xavaf0YYP+w9duzwOBwNvkL4HlgLjAJGK4OFhJpAwfC4sXByXrffgtjx8LkyfD999ClC7xeqddYEKkScrJyWL9FBbKIpK/SdLE42933cfda7t7U3Z9w9y3ufq67t3P3w939PzHjR7l7S3dv7e6qHqTq6dULZsyA/faDk0+Gs84KZpnVK1mkVHKycjSDLCJpTVfSEymLFi3gww/huOPg+ee3x9UrWaREKpBFJN2VdQ2yiNSpA999t3NcvZJFipWTlaMLhYhIWlOBLFIeS5bEj6tXskhCmkEWkXSnAlmkPBL1Sq5fv3LzEIkQFcgiku5UIIuUR7xeyRkZsGYNDB8Ov/ySmrxE0lhBgezuqU5FRCQuFcgi5VG0V3KLFkEruKuvhoceghNOCFrCiUihnKwcHOfHX35MdSoiInGpi4VIeQ0cGL9jRfv2cMEF0LkzTJgAhxxS+bmJpKHszGwA1m9eT93MuinORkRkZ5pBFqko554L778PW7bAEUfAFVcEfZLVL1mquZysHACtQxaRtKUCWaQidekC06dD48Zwzz1Bn2T37f2SVSRLNaQCWUTSnQpkkYq2777xT9ZTv2SpplQgi0i6U4EsUhny8uLH1S9ZqqGCAlkXCxGRdKUCWaQyJOqX3Lhx5eYhkWVmrc1sdsxtvZmNMLM/mdnSmHifmNdcZ2YLzexLMzsxJt47jC00s5GV/Vk0gywi6U4FskhliNcv2QxWrdI6ZCkVd//S3Tu4ewegI/ATMD7cfHfBNnefCGBmbYABQFugN/CQmdUwsxrAg8BJQBvg7HBspVGBLCLpTgWySGWI1y/5oYfgqKOCbhd/+APk56c6S4mO44Cv3f3bYsb0A551983u/g2wEOgS3ha6+yJ33wI8G46tNNlZ29u8iYikIxXIIpVl4EBYvDgohBcvhosugjffhAsvhFtvhdNOg40bU52lRMMA4J8xzy81szlmNsbMCq5z3gRYEjMmL4wliu/AzIaZ2Qwzm7Fq1aqkJp9ZI5PaNWurQBaRtKUCWSSVMjPh0Ufh3nvh3/+G7t2Dx+qXLAmYWSbQF3ghDD0MtAQ6AMuAvyXjfdx9tLt3cvdOjRo1SsYud5Cdma0CWUTSlq6kJ5JqZvC730Hr1tC/P4wYsX1bQb9kiH+1PqmOTgJmufsKgIJ7ADN7DHg1fLoUaBbzuqZhjGLilSYnK4f1W1Qgi0h60gyySLo48USoX3/nuPoly47OJmZ5hZntE7OtP/B5+HgCMMDMssxsP6AVMA2YDrQys/3C2egB4dhKlZOVoxlkEUlbmkEWSSfLlsWPq1+yAGa2O/Ar4P/FhG83sw6AA4sLtrn7F2b2PDAX2AoMd/dt4X4uBd4AagBj3P2LSvsQIRXIIpLOVCCLpJPmzYNlFUU1bVr5uUjacfcfgYZFYucVM34UMCpOfCIwMekJ7oKcrBzy1ie4gI6ISIppiYVIOonXLxmCzheLFlV+PiIVRDPIIpLOVCCLpJN4/ZKvuy5Yh9ylC7z3XqozFEkKFcgiks5UIIukm6L9kv/yF/j4Y9hzTzj+eHj88VRnKFJuKpBFJJ2pQBaJglatYOpUOPbY4MIiJ50UzC6rV7JEVE5WDpu3bWbz1s2pTkVEZCc6SU8kKurVg9deg1NOgUmTtsfVK1kiKDszuNz0hi0byKqZleJsRER2pBlkkSipWRPmz985rl7JEjE5WTkAWmYhImlJBbJI1CTqiaxeyRIhKpBFJJ2pQBaJmubN48fjXYVPJE2pQBaRdKYCWSRq4vVKzsiANWvgiitg27bU5CWyCwoK5A2bN6Q4ExGRnalAFomaeL2Sx46Fyy+He+6BX/8a1mtWTtKbZpBFJJ2pi4VIFA0cuHPHivPOg4MPhksvhSOOgH//G/bfPzX5iZRABbKIpLMSZ5DNbIyZrTSzz+Ns+72ZuZntGT43M7vPzBaa2RwzO7wikhaRBP7f/4M33oBly6BrV/jjH4M+yeqXLGlGBbKIpLPSLLEYC/QuGjSzZsAJQOyp8ycBrcLbMODh8qcoIrvk2GODK+/VrAl//nPQJ9l9e79kFcmSBurUqkOGZahAFpG0VGKB7O7vA2vibLobuAbwmFg/4CkPTAXqmdk+SclUREqvVaugQC5K/ZIlTZgZ2ZnZKpBFJC2V6SQ9M+sHLHX3T4tsagIsiXmeF8bi7WOYmc0wsxmrVq0qSxoiUpylS+PH1S9Z0kROVg7rt6hAFpH0s8sFspnVAf4A/LE8b+zuo929k7t3atSoUXl2JSLxJOqXvO++lZuHSAI5WTmaQRaRtFSWGeSWwH7Ap2a2GGgKzDKzvYGlQLOYsU3DmIhUtnj9kiFoATd1auXnI1KECmQRSVe7XCC7+2fuvpe757p7LsEyisPdfTkwARgUdrPoBqxz92XJTVlESiVev+Tbb4e99oJeveCZZ1KdoVRzOVk5ulCIiKSl0rR5+ycwBWhtZnlmNrSY4ROBRcBC4DHgkqRkKSJlM3AgLF4M+fnB/dVXBx0uunYNtv3f/wXbRFJAM8gikq5K08XibHffx91ruXtTd3+iyPZcd/8+fOzuPtzdW7r7Ie4+o6ISF5EyatgQJk+G88+HW26Bs86CJ59Uv+QIMLPFZvaZmc02sxlhrIGZTTazBeF9/TCesC+9mQ0Oxy8ws8Gp+jwqkEUkXelKeiLVUWYmPP54cOW9q6+Gl17aPpNc0C8Zdr5an6SDYwomJUIjgbfd/TYzGxk+v5Yd+9J3JehL39XMGgA3Ap0I2nTONLMJ7v5DZX4IUIEsIumrTG3eRKQKMIOrroJGjXZeZqF+yVHSDxgXPh4HnBoTj9eX/kRgsruvCYviycS5GFRlyMnKYcOWDeS7lvmISHpRgSxS3X3/ffy4+iWnIwfeNLOZZhZO89M45mTo5UDj8HGivvSl6ldfGb3qszOzAdi4ZWOF7F9EpKxUIItUd4n6JTdrFj8uqXSUux9OsHxiuJkdHbvR3Z0dr25aZpXRqz4nKwdAyyxEJO2oQBap7hL1S95nn2CphaQNd18a3q8ExgNdgBXh0gnC+5Xh8ER96dOmX70KZBFJVyqQRaq7ov2SmzeHAQNg2jQ4+ujEl6yWSmVmu5tZdsFj4ATgc4L+8wWdKAYDr4SPE/WlfwM4wczqhx0vTghjlU4FsoikK3WxEJGgSC7aseKcc4Jb587wyivBvaRSY2C8mUFw7H7G3SeZ2XTg+bBH/bfAmeH4iUAfgr70PwG/BXD3NWb2Z2B6OO5md19TeR9jOxXIIpKuVCCLSHy//jV89BH07RvMJA8dCq++Gpy817x5sDRDbeAqjbsvAg6NE18NHBcn7sDwBPsaA4xJdo67qqBA1tX0RCTdqEAWkcQOOSRYatGjBzz44Pa4eiVLOe19596s+HEFAGe8cEZhvPHujVl+1fJUpSUiAmgNsoiUpFEj2LRp57h6JUs5FBTHpY2LiFQmFcgiUrIlS+LH1StZRESqIBXIIlKyRL2S99qrcvMQERGpBCqQRaRk8XolmwVX4Rs3Lv5rREREIkoFsoiUrGiv5BYt4JFHoGdPGDIErrkGtm1LdZYiIiJJoQJZREpn4EBYvBjy84P7YcNg0iQYPhzuuAP69YP16mcrpdN498a7FBcRqUwqkEWk7GrVggcegIceCorlI46Au++G3FzIyAjun3461VlKGlp+1XL8RsdvdIYeNpTszGx+vv5ntXgTkbSgAllEyu/ii2Hy5KA/8pVXBvfu2/slq0iWYvQ/qD8btmzgP9/8J9WpiIgAKpBFJFmOOQb22GPnuPolSwmO2/846mbWZfz88alORUQEUIEsIsm0bFn8uPolSzFq16xNn1Z9eOXLV9iWr5M9RST1VCCLSPIk6pfcpEnl5iGRc9pBp7Hyx5VMyZuS6lRERFQgi0gSxeuXDPDLLzB/fuXnI5FxUquTyKyRyUvzXkp1KiIiKpBFJIni9Uv+4x+D1nDdusEbb6Q6Q0lTOVk5HL//8YyfPx53T3U6IlLNqUAWkeQq2i/5pptg+vSgWO7TB+69N+hwIVJE/4P6s3jtYj5d8WmqUxGRak4FsohUvBYt4MMPoW9fGDEiaP02bpz6JcsO+rbuS4ZlMH6eulmISGqpQBaRylG3Lrz4YtDy7fHH4fzz1S9ZdrDX7nvRvVl3tXsTkZRTgSwilScjA265BfbcM1iCEUv9koVgmcVnKz/j6zVfpzoVEanGVCCLSOVbvTp+XP2Sq73+B/cH0CyyiKSUCmQRqXyJ+iU3a1a5eUjaya2XS4e9O6hAFpGUUoEsIpUvUb/khg1h/frKzycCzKyZmb1jZnPN7AszuzyM/8nMlprZ7PDWJ+Y115nZQjP70sxOjIn3DmMLzWxkKj5Pcfof1J8pS6awfOPyVKciItWUCmQRqXxF+yU3bw6DBsGcOdC1K3z1VaozTEdbgd+7exugGzDczNqE2+529w7hbSJAuG0A0BboDTxkZjXMrAbwIHAS0AY4O2Y/aaH/Qf1xnFfmv5LqVESkmiqxQDazMWa20sw+j4ndYWbzzWyOmY03s3ox2+LOWIiI7CC2X/K33wZt3956C77/Hrp0gddfT3WGacXdl7n7rPDxBmAeUNw1vPsBz7r7Znf/BlgIdAlvC919kbtvAZ4Nx6aNdnu1o2X9llpmISIpU5oZ5LEEsw+xJgPt3L098BVwHSSesUhatiJStfXqFVxUZL/94OSTYcCAYJZZvZJ3YGa5wGHAx2Ho0nDCYoyZ1Q9jTYAlMS/LC2OJ4kXfY5iZzTCzGatWrUryJyiemdH/oP7855v/sO7ndZX63iIiUIoC2d3fB9YUib3p7lvDp1OBpuHjRDMWIiKlk5sbXFSka1d47rmgs4V6JRcys7rAi8AId18PPAy0BDoAy4C/JeN93H20u3dy906NGjVKxi53Sf+D+/NL/i+8tuC1Sn9vEZFkrEE+Hyj4LrRUMxOQ2tkJEUlzderAd9/tHK/mvZLNrBZBcfy0u78E4O4r3H2bu+cDj7F9UmIpENsWpGkYSxRPK92admPvuntrmYWIpES5CmQzu57gxJFdntJJ9eyEiKS5JUvix6tpr2QzM+AJYJ673xUT3ydmWH+g4HyRCcAAM8sys/2AVsA0YDrQysz2M7NMgmVxEyrjM+yKDMvg1Nan8vqC19n0y6ZUpyMi1UyZC2QzGwKcAgx0dw/DkZiZEJEISNQruV69YMlF9dMdOA84tkhLt9vN7DMzmwMcA1wB4O5fAM8Dc4FJwPBwpnkrcCnwBsGJfs+HY9NO/4P78+MvP/LWordSnYqIVDNlKpDNrDdwDdDX3X+K2ZRoxkJEZNfE65Vcowb88AOcd16w3KIacfcP3N3cvX1sSzd3P8/dDwnjfd19WcxrRrl7S3dv7e6vx8QnuvuB4bZRqflEJeuV24s9svbQMgsRqXSlafP2T2AK0NrM8sxsKPAAkA1MDmcxHoHEMxYVlr2IVF1FeyW3aBG0ghs1Cp55Brp3D9rESZWVWSOTUw48hQlfTmBr/taSXyAikiQ1Sxrg7mfHCT9RzPhRQNrOSIhIhAwcGNyK6tABzjkHOnWC55+HY4+t/NykUvQ/qD9Pf/Y0H/zvA3rl9kp1OiJSTehKeiISPX36BP2SGzeGX/0Kzj1X/ZKroL3v3JszXjgDgGPGHYPdZNhNxt537p3izESkqitxBllEJC21agVTpwazx7EFcUG/ZIg/+yyRseLHFbsUFxFJFs0gi0h0ZWfDypU7x6t5v2QRESkfFcgiEm3qlywiIkmmAllEoi1Rv+Q99oD8/MrNRSrNhs0bUp2CiFRhKpBFJNoS9UteuxZOPx3Wr09NXlKheo7tyfKNy1OdhohUUSqQRSTaEvVLvuce+Pe/oUsXmDs31VlKGTTevXHceL3a9fhy9Zcc8cQRfPn9l5WclYhUB+piISLRV1y/5DPPhK5dYexY+Pnn4OS9//0vWJoxapQ6XaSx5VclniGevnQ6Jz9zMt3HdOfVc16lW9NulZiZiFR1KpBFpOrq2RNmzYIzzghuNWvC1vCKbGoHF2mdm3RmytApnPiPEzniiSPijmm8e+Nii2wRkUS0xEJEqrYmTeDdd6Fu3e3FcQG1g4u0lg1a8tHQjxJuV79kESkrFcgiUvVlZcGPP8bfpnZwkbbX7nulOgURqYK0xEJEqofmzYNlFUU1a1b5uUil+TjvY7o06YJZcInqeLPKWoohIkVpBllEqod47eAAGjdWK7gqrNsT3WhxTwuumHSFLl0tIqWmGWQRqR4KTsQr6GLRrBl07w7PPw+dOsG//gXt26c2R0m6caeO419z/8VDMx4q1fjSzjJrNlqkalOBLCLVR7x2cBdfDGedFbSCe/hhGDIkJalJ2TXevXHCYnXQoYMYdOgg1m9ezx637ZFwH50f68wBDQ4o9SxzaceVppBOdlGu96y495TqQwWyiFRvPXrAJ5/AOefAb38b9EtetAjy8tQrOSJKU8DkZOUUu73Bbg2YtnRasWPaPdSO+rvVp37t+sWOm7hgInVq1aFOrTqlKqSTXZTrPSvuPavDPwSinn+yqEAWEWncGN58M+iV/PLL2+PqlVxtvHHuGwDYTZZwTOs9W/PDph/437riO5+c/MzJpXrPmjfXJLNGZrFjDnv0MGpm1KRmRvF/rs94/gwyLIMaGTWKHXfZxMvIsOJPP7rxnRvJsAzMEv8sAO6aclcwjuLHPTbzsRL39Y85/wAocV//mvsvDCtxf69+9Wqx2wHeWPhGifsBeOebd4LczIotpD/43weFjxONmZo3NdgXxe9rxnczdnieaNwnyz4pccycFXNKta8vVn5R4ph5q+YF+Zfws/hq9Veles+FaxaW+LNY9MOi4D1LGJds5u5J3+mu6tSpk8+YMaPkgSIiFSk3N36nixYtYPHiUu3CzGa6e6ek5pVkZtYbuBeoATzu7rcVN76qHKNLM/tUXIHsN27/e1ncuKlDp7Jp6yZ++uWnYovl63tcz5ZtW7jjozsSjunbui9b87eyNX8rb379ZsJxbRu1ZZtvY1v+NhasWZBwXIPdGpDv+az9eW3CMSJRFPv/Z3FKe4zWDLKISIFEPZGrUK9kM6sBPAj8CsgDppvZBHefm9rMKl5lrSXt2rRrqcbdcuwtAMUWyK8MeKXwcXFF+eeXfF6qcauvWV3iGL/RcXfyPZ+af05cJqy9di1OMK7h7Q0Tjsu7Io98z6f5Pc0Tjllw2QLcHcdp/UDrhOPmXDQHJyiEDn3k0ITjpl84HXeny+NdEo758PwPAXB3jnryqITj3h38buF7HjPumITjJp83GXfnhH+ckHDM6wNfp2Biss8zfRKO+/fZ/y7Mre+zfROOe/msl3Gc/s/1TzjmxTNfLNzXGS+ckXDc82c8D8CZ/zoz4ZhnT3+28Gdx9otnJxz39GlPF77nuePPTTjuqVOfAmDQy4MSjhl36rjCn9mQV4YkHJdsKpBFRAok6pXcPPEf9gjqAix090UAZvYs0A+o8gVyaRR3wl9ZxkWVmVHDil+usUftxCc9xmqS06TEMQc0OKBU+zqk8SGlGtdp35K/xDmy2ZGl2lfP3J6lGnf8/seXOKb3Ab1Lta9TDjylVOP6HdSvxDGnHXxaqfb1m7a/CR78K/GYs9qdVfi4uAL5nEPOKXxcXIF83qHnAcUXyIMO3b5NBbKISCqMGhWsOf7pp+2xOnWCeNXRBFgS8zwP2GnK08yGAcMAmletfyAUq7SzzKUdV5pCOtlFud6z4t5Tqg8VyCIiBYr2Sq7GXSzcfTQwGoI1yClOJ7JKU0gnuyjXe1bce1aHfwhEPf9k0Ul6IiJJlO4n6ZnZEcCf3P3E8Pl1AO5+a6LX6BgtIlVFaY/RutS0iEj1Mh1oZWb7mVkmMACYkOKcRETSipZYiIhUI+6+1cwuBd4gaPM2xt2/SHFaIiJpRQWyiEg14+4TgYmpzkNEJF1piYWIiIiISAwVyCIiIiIiMVQgi4iIiIjESIs2b2a2Cohz+aoS7Ql8n+R0KlPU84fofwbln3pR/wxF82/h7o1SlUxF0DE60qL+GZR/akU9fyjjMTotCuSyMrMZ6dxvtCRRzx+i/xmUf+pF/TNEPf+KFPWfTdTzh+h/BuWfWlHPH8r+GbTEQkREREQkhgpkEREREZEYUS+QR6c6gXKKev4Q/c+g/FMv6p8h6vlXpKj/bKKeP0T/Myj/1Ip6/lDGzxDpNcgiIiIiIskW9RlkEREREZGkUoEsIiIiIhIjsgWymfU2sy/NbKGZjUx1PrvKzBab2WdmNtvMZqQ6n9IwszFmttLMPo+JNTCzyWa2ILyvn8oci5Mg/z+Z2dLw9zDbzPqkMsfimFkzM3vHzOaa2RdmdnkYj8TvoJj8I/E7MLPaZjbNzD4N878pjO9nZh+Hx6LnzMHzYlUAAAO1SURBVCwz1bmmAx2jK1fUj8+gY3Sq6RhdZH9RXINsZjWAr4BfAXnAdOBsd5+b0sR2gZktBjq5e2QacJvZ0cBG4Cl3bxfGbgfWuPtt4R/B+u5+bSrzTCRB/n8CNrr7nanMrTTMbB9gH3efZWbZwEzgVGAIEfgdFJP/mUTgd2BmBuzu7hvNrBbwAXA5cCXwkrs/a2aPAJ+6+8OpzDXVdIyufFE/PoOO0ammY/SOojqD3AVY6O6L3H0L8CzQL8U5VXnu/j6wpki4HzAufDyO4H+mtJQg/8hw92XuPit8vAGYBzQhIr+DYvKPBA9sDJ/WCm8OHAv8K4yn7c+/kukYXcmifnwGHaNTTcfoHUW1QG4CLIl5nkeEfokhB940s5lmNizVyZRDY3dfFj5eDjROZTJldKmZzQm/3kvLr76K+v/t3D1oU2EUxvH/oUWQOhTRrYoogpNUN6FDJ8FREFEQOjq4OLsIgqPi5iC6+UGhfnR1cHASBwWFToIORdKpuAnax+G+gUtpYgtJ73vK81uS3JuEQw55ONy8byLiGHAG+EDCHmyqH5L0ICImIuIzsAa8Bb4B65L+lKdkzKJxcEbXIV02DJAiH9qc0d0YZUZnHZD3gjlJZ4ELwI3y01JqatbrZFuz8xA4AcwCP4F73ZbzfxFxAFgCbkr61T6XoQdb1J+mB5L+SpoFZmiukp7quCQbnz2V0RmyYYA0+dDnjO7OKDM664C8ChxpPZ4px9KQtFpu14BXNI3MqFfWLfXXL611XM+OSOqVL9QG8IjK+1DWVS0BTyW9LIfT9GCr+rP1AEDSOvAOOAdMR8RkOZUui8bEGV2HNNkwSLZ8cEbXYRQZnXVA/gicLDsT9wFXgOWOa9q2iJgqC+CJiCngPPB1+KuqtQwslPsLwJsOa9mxfmgVF6m4D2UDwmNgRdL91qkUPRhUf5YeRMThiJgu9/fTbEBboQnhS+Vp1X7+u8wZXYcU2TBMlnwAZ3TXRp3RKf/FAqD8zcgDYAJ4IuluxyVtW0Qcp7kiATAJPMtQf0Q8B+aBQ0APuA28BhaBo8AP4LKkKjdZDKh/nuZnIwHfgeuttWJViYg54D3wBdgoh2/RrBGrvgdD6r9Kgh5ExGmaDR4TNBcXFiXdKd/nF8BB4BNwTdLv7iqtgzN6d2XPZ3BGd80Zven9sg7IZmZmZmbjkHWJhZmZmZnZWHhANjMzMzNr8YBsZmZmZtbiAdnMzMzMrMUDspmZmZlZiwdkMzMzM7MWD8hmZmZmZi3/AN+Hs3rbhjw0AAAAAElFTkSuQmCC\n",
"text/plain": [
"