{ "cells": [ { "cell_type": "markdown", "id": "a1b61aad", "metadata": {}, "source": [ "# Lesson 45: Distributed representations demonstration\n", "\n", "This notebook demonstrates using pre-trained word embeddings and extending them to document-level representations.\n", "\n", "**1. Pre-trained word embeddings**\n", "- 1.1. Loading pre-trained Word2Vec\n", "- 1.2. Word similarity and analogies\n", "\n", "**2. Document embeddings**\n", "- 2.1. Averaging word vectors\n", "- 2.2. Doc2Vec\n", "\n", "**3. Document similarity comparison**\n", "\n", "\n", "## Notebook set up\n", "\n", "**Note**: depending on your environment, you may need to install NLTK and gensim:\n", "\n", "```text\n", "pip install nltk gensim\n", "```\n", "\n", "### Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "cec35574", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "from matplotlib.patches import Patch\n", "from gensim.models import Word2Vec, KeyedVectors\n", "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n", "from sklearn.decomposition import PCA\n", "from sklearn.metrics.pairwise import cosine_similarity\n", "from nltk.tokenize import word_tokenize\n", "\n", "import gensim.downloader as api\n", "import nltk\n", "nltk.download('punkt', quiet=True)" ] }, { "cell_type": "markdown", "id": "a7c1f45b", "metadata": {}, "source": [ "### Sample documents" ] }, { "cell_type": "code", "execution_count": 2, "id": "0605a20e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sample documents:\n", " Doc 1: The king ruled his kingdom with wisdom and justice\n", " Doc 2: The queen governed her realm with grace and power\n", " Doc 3: Machine learning algorithms can learn patterns from data\n", " Doc 4: Deep neural networks are powerful for pattern recognition\n", " Doc 5: The cat sat lazily on the warm sunny windowsill\n", " Doc 6: Dogs love playing fetch in the park with their owners\n" ] } ], "source": [ "# Sample documents for demonstration\n", "documents = [\n", " 'The king ruled his kingdom with wisdom and justice',\n", " 'The queen governed her realm with grace and power',\n", " 'Machine learning algorithms can learn patterns from data',\n", " 'Deep neural networks are powerful for pattern recognition',\n", " 'The cat sat lazily on the warm sunny windowsill',\n", " 'Dogs love playing fetch in the park with their owners'\n", "]\n", "\n", "# Tokenize documents\n", "tokenized_docs = [word_tokenize(doc.lower()) for doc in documents]\n", "\n", "print('Sample documents:')\n", "for i, doc in enumerate(documents):\n", " print(f' Doc {i+1}: {doc}')" ] }, { "cell_type": "markdown", "id": "1d3cd8e2", "metadata": {}, "source": [ "## 1. Pre-trained word embeddings\n", "\n", "### 1.1. Loading pre-trained Word2Vec\n", "\n", "Pre-trained embeddings capture semantic relationships learned from large text corpora.\n", "\n", "Gensim [downloader](https://radimrehurek.com/gensim/downloader.html) documentation" ] }, { "cell_type": "code", "execution_count": 3, "id": "ebb216cf", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary size: 400000\n", "Vector dimensions: 100\n" ] } ], "source": [ "# Load pre-trained GloVe embeddings (smaller than Google News for demo)\n", "# Using glove-wiki-gigaword-100 (100-dimensional vectors)\n", "w2v_model = api.load('glove-wiki-gigaword-100')\n", "\n", "print(f'Vocabulary size: {len(w2v_model)}')\n", "print(f'Vector dimensions: {w2v_model.vector_size}')" ] }, { "cell_type": "markdown", "id": "08c4fe67", "metadata": {}, "source": [ "### 1.2. Word similarity and analogies" ] }, { "cell_type": "code", "execution_count": 4, "id": "a47d14f7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Words most similar to \"king\":\n", " prince: 0.768\n", " queen: 0.751\n", " son: 0.702\n", " brother: 0.699\n", " monarch: 0.698\n", "\n", "Words most similar to \"computer\":\n", " computers: 0.875\n", " software: 0.837\n", " technology: 0.764\n", " pc: 0.737\n", " hardware: 0.729\n" ] } ], "source": [ "# Find similar words\n", "print('Words most similar to \"king\":')\n", "for word, score in w2v_model.most_similar('king', topn=5):\n", " print(f' {word}: {score:.3f}')\n", "\n", "print('\\nWords most similar to \"computer\":')\n", "for word, score in w2v_model.most_similar('computer', topn=5):\n", " print(f' {word}: {score:.3f}')" ] }, { "cell_type": "code", "execution_count": 5, "id": "9d0beafb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Analogy: king - man + woman = ?\n", " queen: 0.770\n", " monarch: 0.684\n", " throne: 0.676\n", "\n", "Analogy: actor - man + woman = ?\n", " actress: 0.907\n", " comedian: 0.689\n", " actresses: 0.683\n", "\n", "Analogy: paris - france + germany = ?\n", " berlin: 0.885\n", " frankfurt: 0.799\n", " vienna: 0.768\n" ] } ], "source": [ "# Word analogies: king - man + woman = ?\n", "print('Analogy: king - man + woman = ?')\n", "result = w2v_model.most_similar(positive=['king', 'woman'], negative=['man'], topn=3)\n", "\n", "for word, score in result:\n", " print(f' {word}: {score:.3f}')\n", "\n", "print('\\nAnalogy: actor - man + woman = ?')\n", "result = w2v_model.most_similar(positive=['actor', 'woman'], negative=['man'], topn=3)\n", "\n", "for word, score in result:\n", " print(f' {word}: {score:.3f}')\n", "\n", "print('\\nAnalogy: paris - france + germany = ?')\n", "result = w2v_model.most_similar(positive=['paris', 'germany'], negative=['france'], topn=3)\n", "\n", "for word, score in result:\n", " print(f' {word}: {score:.3f}')" ] }, { "cell_type": "markdown", "id": "4efec5b1", "metadata": {}, "source": [ "**Understanding these classic analogies:**\n", "\n", "These demonstrate that word embeddings capture semantic relationships through vector arithmetic:\n", "\n", "1. **king - man + woman ≈ queen**\n", " - This shows the model understands gender relationships in royalty\n", " - The vector difference (king - man) captures \"royalty without gender\"\n", " - Adding woman gives \"female royalty\" = queen\n", "\n", "2. **Actor - man + woman ≈ actress**\n", " - This shows the model understands gender relationships in performers\n", " - The vector difference (actor - man) captures \"performer without gender\"\n", " - Adding woman gives \"female performer\" = actress\n", " \n", "2. **paris - france + germany ≈ berlin**\n", " - This shows the model understands country-capital relationships\n", " - The vector difference (paris - france) captures \"capital city of\"\n", " - Adding germany gives \"capital of Germany\" = berlin\n", "\n", "These algebraic properties emerge naturally from training on large text corpora, where words appearing in similar contexts develop similar vector representations." ] }, { "cell_type": "markdown", "id": "67f881e3", "metadata": {}, "source": [ "### 1.3. Analogies in 2D space\n", "\n", "To visualize word analogies, we project embeddings onto a 'gender direction' found by averaging the difference vectors between gendered word pairs (e.g., woman - man, queen - king, actress - actor). This averaged vector captures the semantic direction of gender in the embedding space.\n", "\n", "**Key operations:**\n", "- `np.linalg.norm(v)`: Normalizing the gender vector to unit length ensures the projection values represent actual distances along the direction, not scaled by the vector's arbitrary magnitude.\n", "- `A @ B`: Projects each word vector onto the gender direction via dot product. The result measures how far each word lies along the gender axis (positive = female, negative = male)." ] }, { "cell_type": "code", "execution_count": 6, "id": "97a72403", "metadata": {}, "outputs": [], "source": [ "# Select interesting words representing different semantic categories\n", "interesting_words = [\n", " 'man', 'woman', 'king', 'queen', 'actor', 'actress'\n", "]\n", "\n", "# Get Word2Vec embeddings\n", "w2v_vectors = np.array([w2v_model[w] for w in interesting_words])\n", "\n", "# Compute the gender direction by averaging difference vectors\n", "gender_vector = (\n", " (w2v_model['woman'] - w2v_model['man']) +\n", " (w2v_model['queen'] - w2v_model['king']) +\n", " (w2v_model['actress'] - w2v_model['actor'])\n", ") / 3\n", "\n", "# Normalize the gender vector\n", "gender_vector = gender_vector / np.linalg.norm(gender_vector)\n", "\n", "# Project each word onto the gender direction\n", "gender_projections = w2v_vectors @ gender_vector\n", "\n", "# Create an orthogonal direction (using PCA on residuals)\n", "residuals = w2v_vectors - np.outer(gender_projections, gender_vector)\n", "\n", "pca = PCA(n_components=1)\n", "orthogonal_projections = pca.fit_transform(residuals).flatten()\n", "\n", "# Combine into 2D coordinates\n", "w2v_2d = np.column_stack([gender_projections, orthogonal_projections])" ] }, { "cell_type": "code", "execution_count": 7, "id": "322d8db6", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAGGCAYAAABfbHkYAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAY6pJREFUeJzt3XdYFFf7N/DvsMDSQREBFalWVCxYQBMbChYUu4mJYOzRoBFNNE9iSaKkGDVqYksCeSyP0dh7r8Su2AsoisGCDRCQtnveP3jZnyvFXQR2ke/nuvbSOXNm5p7ZXe6dc87MSEIIASIiItI7BroOgIiIiArGJE1ERKSnmKSJiIj0FJM0ERGRnmKSJiIi0lNM0kRERHqKSZqIiEhPMUkTERHpKSZpIiIiPcUkTcVy8OBBSJKEgwcP6joUFUmSMHbs2FLfjjb73q5dO7Rr1041ffv2bUiShMjIyFKLryLTh89lZGQkJEnC7du3VWWvfg70gT7GRPkxSeuxNWvWQJIkbNiwId88Ly8vSJKEAwcO5JtXs2ZN+Pr6lkWIRKTHrly5gunTp6v9YKDyhUlaj7Vp0wYAcPToUbXylJQUXLp0CYaGhoiKilKbd/fuXdy9e1e1LOkXZ2dnvHjxAh9++KGuQ6EytHv3buzevbvMt3vlyhXMmDGjwCStq5hIO0zSeqxatWpwdXXNl6SPHTsGIQT69euXb17e9JsmaSEEXrx48UbroPwkSYKJiQlkMpmuQyENpKWllch6jI2NYWxsXGSdjIwMKJXKEtmeJjSJiXSPSVrPtWnTBufOnVNLmFFRUfD09ESXLl1w/PhxtS92VFQUJElC69atAQA5OTn45ptv4O7uDrlcDhcXF3zxxRfIzMxU246Liwu6d++OXbt2wdvbG6ampliyZAkA4N9//0VQUBDMzc1RtWpVfPrpp/mWL0pCQgI++ugj2NvbQy6Xw9PTE3/88Ydanby+xDVr1mDGjBmoXr06LC0t0bdvXyQnJyMzMxPjx49H1apVYWFhgSFDhhQaw8qVK1GnTh2YmJigWbNmOHz4cLFi0nbfly5dCnd3d5iamqJFixY4cuRIvjoF9UmHhITAwsICCQkJCAoKgoWFBezs7DBx4kQoFAq15Z88eYIPP/wQVlZWsLGxQXBwMM6fP59vnQ8ePMCQIUNQo0YNyOVyODo6omfPniXe7Hnw4EF4e3vDxMQE7u7uWLJkCaZPnw5JkvLVXbFiBZo1awZTU1NUrlwZAwcOxN27d9XqtGvXDg0aNMCVK1fQvn17mJmZoXr16vjhhx/yrU+b9+bEiRMICAiAtbU1zMzM0LZt23ytUHlxX7lyBe+//z4qVar02h+7ly9fRocOHWBqaooaNWrg22+/LTDRvtr/m/d5X716Nb788ktUr14dZmZmSElJ0TheIPdzPHToUFSrVg1yuRyurq4YPXo0srKyEBkZiX79+gEA2rdvD0mS1PrrC+qTTkxMxNChQ2Fvbw8TExN4eXnhzz//VKuT9xmePXu26jMvl8vRvHlznDp1qsjjRdoz1HUAVLQ2bdpg+fLlOHHihOoLFRUVBV9fX/j6+iI5ORmXLl1Co0aNVPPq1q0LW1tbAMCwYcPw559/om/fvggLC8OJEycQHh6Oq1ev5uvrvn79Ot577z2MHDkSw4cPR506dfDixQt07NgR8fHxCA0NRbVq1bB8+XLs379fo/gfPnyIVq1aqQZ12dnZYceOHRg6dChSUlIwfvx4tfrh4eEwNTXF5MmTERsbiwULFsDIyAgGBgZ49uwZpk+fjuPHjyMyMhKurq6YOnWq2vKHDh3CX3/9hdDQUMjlcvz6668ICAjAyZMn0aBBA61i0mbff//9d4wcORK+vr4YP348bt26hR49eqBy5cpwcnJ67XFSKBTw9/dHy5YtMXv2bOzduxc//fQT3N3dMXr0aACAUqlEYGAgTp48idGjR6Nu3brYtGkTgoOD862vT58+uHz5Mj755BO4uLggMTERe/bsQXx8PFxcXDR4517v3LlzCAgIgKOjI2bMmAGFQoGvv/4adnZ2+erOnDkTX331Ffr3749hw4bh0aNHWLBgAd59912cO3cONjY2qrrPnj1DQEAAevfujf79++Pvv//G559/joYNG6JLly4AtHtv9u/fjy5duqBZs2aYNm0aDAwMEBERgQ4dOuDIkSNo0aKFWv1+/fqhVq1amDVrFop6ku+DBw/Qvn175OTkYPLkyTA3N8fSpUthamqq8TH85ptvYGxsjIkTJyIzMxPGxsYax3vv3j20aNECSUlJGDFiBOrWrYuEhAT8/fffSE9Px7vvvovQ0FDMnz8fX3zxBerVqwcAqn9f9eLFC7Rr1w6xsbEYO3YsXF1dsXbtWoSEhCApKQnjxo1Tq79q1So8f/4cI0eOhCRJ+OGHH9C7d2/cunULRkZGGh8Deg1Beu3y5csCgPjmm2+EEEJkZ2cLc3Nz8eeffwohhLC3txe//PKLEEKIlJQUIZPJxPDhw4UQQkRHRwsAYtiwYWrrnDhxogAg9u/frypzdnYWAMTOnTvV6s6bN08AEGvWrFGVpaWlCQ8PDwFAHDhwoMj4hw4dKhwdHcXjx4/VygcOHCisra1Fenq6EEKIAwcOCACiQYMGIisrS1XvvffeE5IkiS5duqgt7+PjI5ydndXKAAgA4vTp06qyO3fuCBMTE9GrVy+tY9J037OyskTVqlVF48aNRWZmpqru0qVLBQDRtm1bVVlcXJwAICIiIlRlwcHBAoD4+uuv1eJp0qSJaNasmWp63bp1AoCYN2+eqkyhUIgOHTqorfPZs2cCgPjxxx9FaQoMDBRmZmYiISFBVRYTEyMMDQ3Fy39abt++LWQymZg5c6ba8hcvXhSGhoZq5W3bthUAxH//+19VWWZmpnBwcBB9+vRRlWn63iiVSlGrVi3h7+8vlEqlqm56erpwdXUVnTp1UpVNmzZNABDvvfeeRvs/fvx4AUCcOHFCVZaYmCisra0FABEXF6e2Xy9/DvI+725ubqrPm7bxDh48WBgYGIhTp07liy1v2bVr1xb6PX01prxjumLFClVZVlaW8PHxERYWFiIlJUUI8X+fYVtbW/H06VNV3U2bNgkAYsuWLUUcNdIWm7v1XL169WBra6vqaz5//jzS0tJUo7d9fX1VzWDHjh2DQqFQNdFt374dADBhwgS1dYaFhQEAtm3bplbu6uoKf39/tbLt27fD0dERffv2VZWZmZlhxIgRr41dCIF169YhMDAQQgg8fvxY9fL390dycjLOnj2rtszgwYPVfoW3bNkSQgh89NFHavVatmyJu3fvIicnR63cx8cHzZo1U03XrFkTPXv2xK5du6BQKLSKSdN9P336NBITEzFq1Ci1Pr6QkBBYW1u/9jjlGTVqlNr0O++8g1u3bqmmd+7cCSMjIwwfPlxVZmBggDFjxqgtZ2pqCmNjYxw8eBDPnj3TePvaUCgU2Lt3L4KCglCtWjVVuYeHh+psN8/69euhVCrRv39/tePt4OCAWrVq5btCwcLCAh988IFq2tjYGC1atFA7Fpq+N9HR0YiJicH777+PJ0+eqLadlpaGjh074vDhw/map199Hwqzfft2tGrVSu1M3M7ODoMGDdJoeQAIDg5WO/PWNF6lUomNGzciMDAQ3t7e+dZbUHeDJvvj4OCA9957T1VmZGSE0NBQpKam4tChQ2r1BwwYgEqVKqmm33nnHQBQe5/ozbG5W89JkgRfX1/VlzMqKgpVq1aFh4cHgNwkvXDhQgBQJeu8JH3nzh0YGBio6uZxcHCAjY0N7ty5o1bu6uqab/t37tyBh4dHvi99nTp1Xhv7o0ePkJSUhKVLl2Lp0qUF1klMTFSbrlmzptp0XpJ7tcnY2toaSqUSycnJqqZ9AKhVq1a+bdSuXRvp6el49OgRDAwMNI5J033PO46vbtvIyAhubm4FbuNVJiYm+ZqJK1WqpJZk79y5A0dHR5iZmanVe/X9lcvl+P777xEWFgZ7e3u0atUK3bt3x+DBg+Hg4FBoDC9evEBycrJaWWH1ExMT8eLFi3zbLiiemJgYCCEKfG8A5GsarVGjRr5jXqlSJVy4cEE1rel7ExMTAwAFdgnkSU5OVks2BX0PCnLnzh20bNkyX7km343CtqVpvFlZWUhJSVF14ZSEO3fuoFatWjAwUD93y2sef/Xvxavf1bxjWFo/DCsqJulyoE2bNtiyZQsuXryo6o/O4+vri0mTJiEhIQFHjx5FtWrV8iUGTX9Va9OXpom8M5QPPvig0D86eX3peQob9VxYuSiiz7CkYioLJT3ae/z48QgMDMTGjRuxa9cufPXVVwgPD8f+/fvRpEmTApf566+/MGTIELUybY9vQZRKJSRJwo4dOwrcTwsLC7Xpknqv87YNAD/++CMaN25cYJ1Xt1/S34OivLotTeN9+vRpaYf2WiX5PlHhmKTLgZevl46KilIbbNWsWTPI5XIcPHgQJ06cQNeuXVXznJ2doVQqERMTozZY5OHDh0hKSoKzs/Nrt+3s7IxLly5BCKGW7K9fv/7aZe3s7GBpaQmFQgE/Pz9NdvWN5Z2JvOzGjRswMzNTnalqGpOm+553HGNiYtChQwdVeXZ2NuLi4uDl5VXs/Xl1OwcOHEB6erra2XRsbGyB9d3d3REWFoawsDDExMSgcePG+Omnn7BixYoC6/v7+2PPnj0axVK1alWYmJgUuO1Xy9zd3SGEgKurK2rXrq3R+l9H0/fG3d0dAGBlZVXin0FnZ+cCP2+afDcKo2m8dnZ2sLKywqVLl4pcnzbN3s7Ozrhw4QKUSqXa2fS1a9dU86nssU+6HMi7xGXlypVISEhQO5OWy+Vo2rQpfvnlF6SlpaldMpKXsOfNm6e2vjlz5gAAunXr9tptd+3aFffu3cPff/+tKktPTy+0qfhlMpkMffr0wbp16wr8Y/Lo0aPXrkNbx44dU+vnvnv3LjZt2oTOnTtDJpNpFZOm++7t7Q07OzssXrwYWVlZqvLIyEgkJSWV2L75+/sjOzsby5YtU5UplUr88ssvavXS09ORkZGhVubu7g5LS8siL51zdHSEn5+f2qswMpkMfn5+2LhxI+7du6cqj42NxY4dO9Tq9u7dGzKZDDNmzMh3liWEwJMnTwrf6UJo+t40a9YM7u7umD17NlJTU/Ot500+g127dsXx48dx8uRJtfWtXLmy2OvUNF4DAwMEBQVhy5YtOH36dL56ecfZ3NwcADT6HHbt2hUPHjzAX3/9pSrLycnBggULYGFhgbZt2xZnl+gN8Uy6HDA2Nkbz5s1x5MgRyOVytYFRQG6T908//QRA/SYmXl5eCA4OxtKlS5GUlIS2bdvi5MmT+PPPPxEUFIT27du/dtvDhw/HwoULMXjwYJw5cwaOjo5Yvnx5vn7Rwnz33Xc4cOAAWrZsieHDh6N+/fp4+vQpzp49i71795Z4s12DBg3g7++vdgkWAMyYMUPrmDTddyMjI3z77bcYOXIkOnTogAEDBiAuLg4REREa90lrIigoCC1atEBYWBhiY2NRt25dbN68WRVv3lnTjRs30LFjR/Tv3x/169eHoaEhNmzYgIcPH2LgwIElFs/06dOxe/dutG7dGqNHj4ZCocDChQvRoEEDREdHq+q5u7vj22+/xZQpU3D79m0EBQXB0tIScXFx2LBhA0aMGIGJEydqtW1N3xsDAwP89ttv6NKlCzw9PTFkyBBUr14dCQkJOHDgAKysrLBly5Zi7f9nn32G5cuXIyAgAOPGjVNdgpV3Rloc2sQ7a9Ys7N69G23btsWIESNQr1493L9/H2vXrsXRo0dhY2ODxo0bQyaT4fvvv0dycjLkcjk6dOiAqlWr5tv2iBEjsGTJEoSEhODMmTNwcXHB33//jaioKMybNw+WlpbF2id6Q2U+npyKZcqUKQKA8PX1zTdv/fr1AoCwtLQUOTk5avOys7PFjBkzhKurqzAyMhJOTk5iypQpIiMjQ62es7Oz6NatW4HbvnPnjujRo4cwMzMTVapUEePGjRM7d+7U6BIsIYR4+PChGDNmjHBychJGRkbCwcFBdOzYUSxdulRVJ++SlLVr16otGxERIQDku8wk73KZR48eqcoAiDFjxogVK1aIWrVqCblcLpo0aVJgjJrEpO2+//rrr8LV1VXI5XLh7e0tDh8+nO8yl8IuwTI3N88XY94+vuzRo0fi/fffF5aWlsLa2lqEhISIqKgoAUCsXr1aCCHE48ePxZgxY0TdunWFubm5sLa2Fi1btlS7XKmk7Nu3TzRp0kQYGxsLd3d38dtvv4mwsDBhYmKSr+66detEmzZthLm5uTA3Nxd169YVY8aMEdevX1fVadu2rfD09My3bHBwcL5L7rR5b86dOyd69+4tbG1thVwuF87OzqJ///5i3759qjoFfaZe58KFC6Jt27bCxMREVK9eXXzzzTfi999/1/gSrFc/79rEm3cMBg8eLOzs7IRcLhdubm5izJgxapcCLlu2TLi5uQmZTKZ2bF6NSYjc78WQIUNElSpVhLGxsWjYsKHaZ1WI//sMF3SJHwAxbdq01x430pwkBHv5icqzjRs3olevXjh69KjqTnO6FBQUhMuXLxfYX0tE2mGfNFE58ur91BUKBRYsWAArKys0bdpU5/HExMRg+/btfAQiUQlhnzRROfLJJ5/gxYsX8PHxQWZmJtavX49//vkHs2bNKtNLh/K4ubkhJCQEbm5uuHPnDhYtWgRjY2N89tlnZR4L0duIzd1E5ciqVavw008/ITY2FhkZGfDw8MDo0aMxduxYncQzZMgQHDhwAA8ePIBcLoePjw9mzZqlk7N6orcRkzQREZGeYp80ERGRnmKSJiIi0lMVauCYUqnEvXv3YGlpWaynxBC9Tnh4OD799FOYmJjoOhQi0pAQAs+fP0e1atXyPWBE1ypUn/S///6b72lKREREQO5thGvUqKHrMNRUqDPpvNva3b17F1ZWVjqOhsqDYcOGISYmBtnZ2ahevToWLlwIe3t77Nq1C+Hh4cjOzoYkSZg3bx5WrFiBiIgI1K9fHzKZDBs2bACQ+0SqW7duQQiBESNGqJ6N3bBhQ/Tq1QtHjhyBu7s7fvvtN13uKlGFlZKSAicnJ7289WmFOpNOSUmBtbU1kpOTmaRJI48ePVI9Peu7777D7du3MWHCBLRp0waHDx9G3bp1kZ2djfT0dFhbW0OSJDx79gw2NjYAgAEDBsDNzQ3h4eFITExEs2bNsHbtWrRq1QouLi7w8/PDsmXL2P1CpEP6nBsq1Jk0kbZWrVqF5cuXIyMjAxkZGahSpQr27NmDgIAA1K1bF0DuAzasra0LXH7v3r04c+YMgNzHO/bu3Rt79+5Fq1atAAAhISFM0ERUKP3qISfSI0ePHsX8+fOxfft2XLp0CXPmzMn3CEhtvZqQLSws3mh9RPR2Y5ImKsSzZ89gaWkJW1tbZGVlYcmSJQByn+u8a9cuXLt2DQCQnZ2N5ORkALnjHvL+D0DVnA3kNp2vX78enTp1KuM9IaLyis3dVKGkZ6dj642tSEhJAADUsKqB7rW7w9Qo/32vAwICsGLFCtSpUwe2trbw8/NDQkICPDw8EBERgQ8++ADZ2dmQyWRYvHix6lnPnTp1gpmZGXbv3o358+dj9OjRaNiwIYQQ+M9//oOWLVuW9W4TUTnFgWNUIfyb8i9m/zMbv5/7HalZqZBJMgCAQihgZWyFYU2HIcw3DNUsq+k4UiIqa/qcG5ik6a135t4ZdF7RGckZyVAIRYF1ZJIMlU0rY8+He+Dl4FXGERKRLulzbmCfNL3VYp/Gwm+5X5EJGsg9o3764ik6/LcDbifdLrsAiYiKwCRNb7XJeyfjeebzIhN0HoVQICUzBf/Z/58yiIyI6PWYpOmtde/5PWy4tkGjBJ0nR5mDNZfX4FHao1KMjIj01fTp09/4UsuSxCRNb60/o/+EBO1vFKIUSiy/sLwUIiIifTdjxoxCk3ROTk4ZR8MkTW+xm89uFutuXjJJhptPb5ZCRERU1gYNGgRvb280atQI3bp1w4MHDwAA27ZtQ/PmzeHl5YU2bdoAyL3PPgC88847aNy4MRITExESEoKPPvoI7777Lho0aAAAWL58OVq2bImmTZvi3Xffxfnz5wEAx48fR7NmzdC4cWM0aNAAixYtAgD89ttvqF+/Pho3boyGDRvixIkTGsfP0d301greGIyVF1Zq1dwNAIYGhgjxCsGyHstKKTIiKiua3H//yZMnqFKlCpKTk2Ftba12//2QkBCcO3cOR48ehaWlJaKiojBz5kxs2LABcrkcR44cwahRo3D58mX07NkTAwcOxHvvvQcg94ZIlSpVgrW1Na5duwZHR0dkZ2cjMzNT47sN8mYm9NaqYlol90y6GD9D7cztSj4gIipzmt5/vyj9+vVTPSFr06ZNOH/+vNpNiZ4+fYoXL16gffv2+OabbxATE4MOHTqoztA7duyIDz/8EIGBgejSpQtq166tcfxs7qa3Vt/6fZGj1L4PKUeZg771+5ZCRERUlkrq/vsvn/UKIRAcHIzo6GjV6/79+zA1NcX48eOxbds2ODo64osvvsDHH38MAFi3bh2+++47ZGdno2vXrli9erXG22aSprdWqxqt0KBqA60GjxlIBvB29EZTx6alGBkRlQVt7r+f59X777+qR48eWLFiBeLj4wEASqUSp0+fBgBcv34drq6uGD58OL744gscP34cOTk5uHnzJry9vTFx4kT07dsXJ0+e1Hgf2NxNby1JkvB1u6/Re01vjZdRCiWmtZtWilER0ZtI+TcFZ5aeweU1l5H+OB0yYxns6tvBe5Q36vSsA5mRTFVX0/vvv+zV+++/6p133sEPP/yAXr16IScnB1lZWejWrRu8vb2xcOFC7N+/H8bGxpDJZPjpp5+gUCjw0Ucf4enTpzA0NISdnR0iIiI03l8OHKO33o9RP+KzvZ9pVHee/zyMazWulCMiIm3lZORg66itOP/f85AMJAjF/6UuSZY7bV7VHD1+74Ha3TXv8wX0OzewuZveepNaT8LyXsvhaOEIAKqHa7z8/+qW1bG6z2omaCI9lJORgxX+K3Bh+QVAQC1BA/83nfYoDf/r8T9c/N9FXYRZKngmTRVGjjIH225sw3/P/xfxyfGQJAk1rWsi2CsYXWt1hcxA9vqVEFGZ2zpqK84uOwuh1CBdSYCBzAAjzoyAfSN7jdavz7mBSZqIiPRWWmIa5lSfA2WOUuNlDAwN0PCDhgiKCNKovj7nhnLT3L1o0SI0atQIVlZWsLKygo+PD3bs2KHrsIiIqBSd/V3DM+iXKHOUuLTqEtKfpJdSVGWn3CTpGjVq4LvvvsOZM2dw+vRpdOjQAT179sTly5d1HRoREZWSaxuuaZ2kAUCRpUDc/rhSiKhslZtLsAIDA9WmZ86ciUWLFuH48ePw9PTUUVRERFSaXjx5UfxlnxZ/WX1RbpL0yxQKBdauXYu0tDT4+PjoOhwiIiolMpPiD+g0NCmXKU5NudqDixcvwsfHBxkZGbCwsMCGDRtQv379QutnZmYiMzNTNZ2SklIWYRIRUQlxaOyApzeeajVwLI9d/fJ/D/5y0ycNAHXq1EF0dDROnDiB0aNHIzg4GFeuXCm0fnh4OKytrVUvJyenMoyWiIjelPcob+0TtAFg38ge1byrlU5QZahcX4Ll5+cHd3d31f1YX1XQmbSTk5NeDrMnIqL8hBD4tf6veBLzJN9NTIoS+Fsgmg7V7B78vASrlCiVSrUk/Cq5XK66ZCvvRURE5YckSQj6bxAMDA0gGbz+YTmSTIJbZzc0Dm5c+sGVgXKTpKdMmYLDhw/j9u3buHjxIqZMmYKDBw9i0KBBug6NiIhKUfXm1TFoxyAYmRlBkhWcqPMSuFsnNwxYNwAGhuUmvRWp3AwcS0xMxODBg3H//n1YW1ujUaNG2LVrFzp16qTr0IiIqJS5tnfFx5c/xqlfT+H0ktPITFJvRa3esjpajG0Bz/6eb02CBsp5n7S29LnfgYiINJOTkYP4qHi8ePICMmMZbGvbvtFIbn3ODeXmTJqIiAjIvf7ZraObrsMoE29PmwAREdFbhkmaiIhITzFJExER6SkmaSIiIj3FJE1ERKSnmKSJiIj0FJM0ERGRnmKSJiIi0lNM0kRERHqKSZqIiEhPMUkTERHpKSZpIiIiPcUkTUREpKeYpImIiPQUkzQREZGeYpImIiLSU0zSREREeopJmoiISE8xSRMREekpJmkiIiI9xSRNRESkp5ikiYiI9BSTNBERkZ5ikiYiItJTTNJERER6ikmaiIhITzFJExER6SkmaSIiIj3FJE1ERKSnmKSJiIj0VLlJ0uHh4WjevDksLS1RtWpVBAUF4fr167oOi4iIqNSUmyR96NAhjBkzBsePH8eePXuQnZ2Nzp07Iy0tTdehERERlQpJCCF0HURxPHr0CFWrVsWhQ4fw7rvvarRMSkoKrK2tkZycDCsrq1KOkIiIygN9zg3l5kz6VcnJyQCAypUr6zgSIiKi0mGo6wCKQ6lUYvz48WjdujUaNGhQaL3MzExkZmaqplNSUsoiPCIiohJRLs+kx4wZg0uXLmH16tVF1gsPD4e1tbXq5eTkVEYREhERvbly1yc9duxYbNq0CYcPH4arq2uRdQs6k3ZyctLLfgciItINfe6TLjfN3UIIfPLJJ9iwYQMOHjz42gQNAHK5HHK5vAyiIyIiKnnlJkmPGTMGq1atwqZNm2BpaYkHDx4AAKytrWFqaqrj6IiIiEpeuWnuliSpwPKIiAiEhIRotA59btIgIiLd0OfcUG7OpMvJbwkiIqISUy5HdxMREVUETNJERER6Susk/fDhQ3z44YeoVq0aDA0NIZPJ1F5ERERUMrTukw4JCUF8fDy++uorODo6Fjqgi4iIiN6M1kn66NGjOHLkCBo3blwK4RAREVEerZu7nZycONKaiIioDGidpOfNm4fJkyfj9u3bpRAOERER5dG6uXvAgAFIT0+Hu7s7zMzMYGRkpDb/6dOnJRYcERFRRaZ1kp43b14phEFERESv0jpJBwcHl0YcRERE9Ipi3RZUoVBg48aNuHr1KgDA09MTPXr04HXSREREJUjrJB0bG4uuXbsiISEBderUAQCEh4fDyckJ27Ztg7u7e4kHSUREVBFpPbo7NDQU7u7uuHv3Ls6ePYuzZ88iPj4erq6uCA0NLY0YiYiIKiStz6QPHTqE48ePo3LlyqoyW1tbfPfdd2jdunWJBkdERFSRaX0mLZfL8fz583zlqampMDY2LpGgiIiIqBhJunv37hgxYgROnDgBIQSEEDh+/DhGjRqFHj16lEaMREREFZLWSXr+/Plwd3eHj48PTExMYGJigtatW8PDwwM///xzacRIRERUIWndJ21jY4NNmzYhJiYG165dAwDUq1cPHh4eJR4cERFRRVas66QBoFatWqhVq1ZJxkJEREQv0ShJT5gwAd988w3Mzc0xYcKEIuvOmTOnRAIjIiKq6DRK0ufOnUN2drbq/4WRJKlkoiIiIiJIogI9HDolJQXW1tZITk6GlZWVrsMhIiI9oM+5QevR3a9KSUnBxo0bVYPIiIiIqGRonaT79++PhQsXAgBevHgBb29v9O/fHw0bNsS6detKPEAiIqKKSuskffjwYbzzzjsAgA0bNkAIgaSkJMyfPx/ffvttiQdIRERUUWmdpJOTk1X37d65cyf69OkDMzMzdOvWDTExMSUeIBERUUWldZJ2cnLCsWPHkJaWhp07d6Jz584AgGfPnsHExKTEAyQiIqqotL6Zyfjx4zFo0CBYWFjA2dkZ7dq1A5DbDN6wYcOSjo+IiKjC0jpJf/zxx2jRogXu3r2LTp06wcAg92Tczc2NfdJEREQliNdJExFRhabPuUHrM2mFQoHIyEjs27cPiYmJUCqVavP3799fYsERERFVZFoPHBs3bhzGjRsHhUKBBg0awMvLS+1Vmg4fPozAwEBUq1YNkiRh48aNpbo9IiIiXdL6THr16tVYs2YNunbtWhrxFCktLQ1eXl746KOP0Lt37zLfPhERUVnSOkkbGxvr7NnRXbp0QZcuXXSybSIiorKmdXN3WFgYfv75Z5SH8WaZmZlISUlRexEREZUXWp9JHz16FAcOHMCOHTvg6ekJIyMjtfnr168vseDeVHh4OGbMmKHrMIiIiIpF6yRtY2ODXr16lUYsJW7KlCmYMGGCajolJQVOTk46jIiIiEhzWifpiIiI0oijVMjlcsjlcl2HQUREVCzFep50Tk4O9u7diyVLluD58+cAgHv37iE1NbVEgyMiIqrItD6TvnPnDgICAhAfH4/MzEx06tQJlpaW+P7775GZmYnFixeXRpwAgNTUVMTGxqqm4+LiEB0djcqVK6NmzZqltl0iIiJdKNbNTLy9vfHs2TOYmpqqynv16oV9+/aVaHCvOn36NJo0aYImTZoAACZMmIAmTZpg6tSppbpdIiIiXdD6TPrIkSP4559/YGxsrFbu4uKChISEEgusIO3atSsXl34RERGVBK3PpJVKJRQKRb7yf//9F5aWliUSFBERERUjSXfu3Bnz5s1TTUuShNTUVEybNk0ntwolIiJ6W2n9qMp///0X/v7+EEIgJiYG3t7eiImJQZUqVXD48GFUrVq1tGJ9Y/r8ODIiItINfc4NxXqedE5ODlavXo0LFy4gNTUVTZs2xaBBg9QGkukjfX4jiIhIN/Q5N2g9cAwADA0N8cEHH5R0LERERPSSYiXpe/fu4ejRo0hMTIRSqVSbFxoaWiKBERERVXRaJ+nIyEiMHDkSxsbGsLW1hSRJqnmSJDFJExERlRCt+6SdnJwwatQoTJkyBQYGxbqrqM7oc78DERHphj7nBq2zbHp6OgYOHFjuEjQREVF5o3WmHTp0KNauXVsasRAREdFLtG7uVigU6N69O168eIGGDRvCyMhIbf6cOXNKNMCSpM9NGkREpBv6nBu0HjgWHh6OXbt2oU6dOgCQb+AYERERlQytk/RPP/2EP/74AyEhIaUQDhEREeXRuk9aLpejdevWpRELERERvaRYz5NesGBBacRCREREL9G6ufvkyZPYv38/tm7dCk9Pz3wDx9avX19iwRERUdlZunQpTp8+jaVLl+LKlSvw9PTErl270LlzZ3z99dcAgK5duyI0NBSpqakwMTHB3Llz0bp1a9y+fRuNGzfGJ598gm3btuH58+eIjIzE33//jQMHDqie+dCgQQM8ePAA7733HlJSUpCRkYH27dtj/vz5MDAwQGRkJFasWAE7OztcunQJcrkca9asgZubm46Pjm5ofSZtY2OD3r17o23btqhSpQqsra3VXkREVD75+flh7969AIA9e/bAx8dHbbpdu3bo3bs3pk2bhgsXLmDOnDno06cPUlNTAQDJyclo1qwZzp49i8mTJ8Pf3x89evRAdHQ0goODMWPGDAC5eWTLli04c+YMLly4gNu3b2PNmjWqOE6dOoVZs2bh4sWL8PPzw/fff1/GR0J/aH0mHRERURpxEBGRjuWdrd66dQt79+5FeHg4wsLCkJqaiitXrqBSpUowMDCAv78/AKBNmzawt7dHdHQ0atSoARMTEwQFBQEAvL29YWFhgfbt2wMAWrRogZUrVwIAlEolPv/8cxw9ehRCCCQmJqJBgwYYOHAgAMDHxweurq6q/1fkLtZiPWCDiIjeTn5+ftixYwdiYmLQtm1bCCGwbt06+Pj4FFj/5Utv5XK56v8ymQwmJiZq0zk5OQBy76eRmJiIEydOwMTEBBMmTEBGRoaqbmHLVUQaJemmTZti3759qFSpEpo0aVLk9dBnz54tseCIiKhs+fn54bPPPsO7774LAOjQoQOmTZuG8ePHo06dOlAqldizZw86deqEf/75Bw8ePEDjxo3x+PFjjbfx7NkzODg4wMTEBA8ePMDatWvRp0+f0tqlck2jJN2zZ0/VL6S8pgwiItJvimwFrm++jtOLTuPR5UfIycyBaWVT1OtTD94jvVHJrVK+ZTp27Ij4+Hj4+fkBADp16oTZs2ejY8eOMDY2xvr16xEaGoqwsDCYmJjg77//hoWFhVZJety4cejbty88PT1RrVo11bYoP61vC1qe6fOt34iISlLsrlhsHLwRaYlpkGQShOL//tRLMglCKVC/X330/KMnjM2NdRip7ulzbuCjrIiI3jJX1l3Bqq6rkPYoDQDUErRqWgBX113Fn+3/RFZali7CJA1olKQrVaqEypUra/QiKi2SJGHmzJlo2bIlXFxcsHHjRoSHh8Pb2xu1atXCwYMHAQA5OTnw9/eHt7c3PD098f777yMtLfeP1cGDB9GgQQN8/PHH8PLygqenJ06fPq3DvSIqWY+vPca699ZBiNxEXBShELh/5j62jNhSNsGR1jRK0vPmzcPcuXMxd+5cfPnllwAAf39/TJ8+HdOnT1cNx//qq69KL1IiABYWFjhx4gR+//13fPDBB3B0dMTp06cxa9YsTJo0CUDuaNBVq1bh9OnTuHTpEqytrdUu4bh27RqCg4Nx/vx5fPLJJ/jPf/6jq90hKnEn5p/ITc4admQKpcCl/11CcnxyqcZFxaN1n3SfPn3Qvn17jB07Vq184cKF2Lt3LzZu3FiS8ZUofe53oNeTJAn379+Hg4MDkpOTYWNjgxcvXsDExAR37tyBl5cXkpKSoFQqMXXqVGzbtg05OTlITk6Gr68vVq9ejYMHD2LUqFG4du0aAOD8+fPo06cPYmNjdbx3RG8uMyUTsx1mI+eFdpcsSTIJbSa3QYdvO5RSZPpNn3OD1n3Su3btQkBAQL7ygIAA1Z1piEpL3vWTMpks33TetZSrVq3C/v37cejQIVy8eBETJ07kNZhUIdw5fEfrBA3kNntf23CtFCKiN6V1kra1tcWmTZvylW/atAm2trYlEhTRm3j27BmqVKkCKysr1f2DiSqCF89e6GRZKj1a33FsxowZGDZsGA4ePIiWLVsCAE6cOIGdO3di2bJlJR4gvb2EEIi6G4U1l9fgUfojGBoYwr2SO0Iah8DFxqXY6x08eDA2bdqEOnXqwM7ODu+88w7u3LlTcoET6SlDk+LfRPJNlqXSU6zrpE+cOIH58+fj6tWrAIB69eohNDRUlbT1lT73O1Q0666sw1cHvsLVx1dhaGAIpVBCQu6d7JRCiS4eXfBj5x9R366+jiMlKj8eXXmEXz1/1Xo5A0MD1OpWCwM3DiyFqPSfPucG3syEytyPUT/is72fQYIEUcgQVJkkg6mRKXZ9sAu+Tr5lHCFR+fW77+9IOJEAodTuT/sHuz6Ae2f3UopKv+lzbih3NzP55Zdf4OLiAhMTE7Rs2RInT57UdUikhVUXV+GzvZ8BQKEJGgAUQoH07HR0WdkFt57dKqvwiMq9lqEttUrQkoEEGxcbuPlVzOc167tylaT/+usvTJgwAdOmTcPZs2fh5eUFf39/JCYm6jo00oBCqcBnez7TuL5SKJGenY4fo34sxaiI3i6e/T1RN6guJIPCH4SURzKQIMkk9FrRS6P6VPbKVZKeM2cOhg8fjiFDhqB+/fpYvHgxzMzM8Mcff+g6NNLA9pjtSHieoNUyOcocRJ6PREpmSilFRfR2kQwk9PlfH9QNqps7LSs4+UoyCYYmhnh/2/uo2bpmWYZIWig3STorKwtnzpxRe1qKgYEB/Pz8cOzYsQKXyczMREpKitqLdGflxZWQSTKtl8vIycCma/kv+yOighmaGKLf2n7o93c/1GyTPwHLreXwmeCDjy9/DPdOFbMfurwoN2PuHz9+DIVCAXt7e7Vye3t71d2jXhUeHo4ZM2aURXikgfjkeCiEQuvlDA0Mce/5vVKIiOjtJRlIqN+nPur3qY8nMU/w5PoTZL/IhpmtGZx8nXjJVTmh0bvUu3dvjVe4fv36YgdT0qZMmYIJEyaoplNSUuDk5KTDiCo2A6l4DTdCCEgS+8uIisu2li1sa/FmU+WRRkna2tq6tON4rSpVqkAmk+Hhw4dq5Q8fPoSDg0OBy8jlcsjl8rIIjzTgYuOC4/8e1/psWiEUcLLijysiqng0StIRERGlHcdrGRsbo1mzZti3bx+CgoIAAEqlEvv27cv3sA/ST8FewVh5caXWy1kYW6Bn3Z6lEBERkX4rV50SEyZMQHBwMLy9vdGiRQvMmzcPaWlpGDJkiK5DIw10dOsIVxtX3E66XeQ10i8zNDDE0CZDYWZkVsrRERHpn2Il6b///htr1qxBfHw8srKy1OadPXu2RAIryIABA/Do0SNMnToVDx48QOPGjbFz5858g8lIPxlIBpgXMA9Bq4M0qi+TZKhkUgmTfCeVbmBERHpK65E88+fPx5AhQ2Bvb49z586hRYsWsLW1xa1bt9ClS5fSiFHN2LFjcefOHWRmZuLEiRN6f79wUtejTg8s6b4EEqQiB5IZGhjCxsQGez7cg+pW1cswQiIi/aF1kv7111+xdOlSLFiwAMbGxvjss8+wZ88ehIaGIjk5uTRipLfM8GbDsefDPWjt1BpA7hmzkYERjAyMYCAZwMjACO81eA9nRpyBl4OXjqMlItIdrR+wYWZmhqtXr8LZ2RlVq1bFnj174OXlhZiYGLRq1QpPnjwprVjfmD7fRL2iupx4GX9f+RuJaYkwkhnBrZIbBjUcBFszXi5CRGVDn3OD1n3SDg4OePr0KZydnVGzZk0cP34cXl5eiIuLQwV6oBaVEM+qnvCs6qnrMIiI9JLWzd0dOnTA5s2bAQBDhgzBp59+ik6dOmHAgAHo1atXiQdIRERUUWnd3K1UKqFUKmFomHsSvnr1avzzzz+oVasWRo4cCWNj41IJtCToc5MGERHphj7nBq2TdHmmz28EERHphj7nhmJdJ52UlISTJ08iMTERSqVSbd7gwYNLJDAiIqKKTuskvWXLFgwaNAipqamwsrJSe/CBJElM0kRERCVE64FjYWFh+Oijj5CamoqkpCQ8e/ZM9Xr69GlpxEhERFQhaZ2kExISEBoaCjMz3kuZiIioNGmdpP39/XH69OnSiIWIiIheonWfdLdu3TBp0iRcuXIFDRs2hJGRkdr8Hj16lFhwREREFZnWl2AZGBR+8i1JEhQKxRsHVVr0eZg9ERHphj7nBq3PpF+95IqIiIhKh9Z90kRERFQ2ipWkDx06hMDAQHh4eMDDwwM9evTAkSNHSjo2IiKiCk3rJL1ixQr4+fnBzMwMoaGhCA0NhampKTp27IhVq1aVRoxEREQVktYDx+rVq4cRI0bg008/VSufM2cOli1bhqtXr5ZogCVJnwcHEBGRbuhzbtD6TPrWrVsIDAzMV96jRw/ExcWVSFBERERUjCTt5OSEffv25Svfu3cvnJycSiQoIiIiKsYlWGFhYQgNDUV0dDR8fX0BAFFRUYiMjMTPP/9c4gESERFVVFon6dGjR8PBwQE//fQT1qxZAyC3n/qvv/5Cz549SzxAIiKiikrrgWPlmT4PDiAiIt3Q59zAm5kQERHpKa2buytVqgRJkvKVS5IEExMTeHh4ICQkBEOGDCmRAImIiCoqrZP01KlTMXPmTHTp0gUtWrQAAJw8eRI7d+7EmDFjEBcXh9GjRyMnJwfDhw8v8YCJiIgqCq2T9NGjR/Htt99i1KhRauVLlizB7t27sW7dOjRq1Ajz589nkiYiInoDWvdJ79q1C35+fvnKO3bsiF27dgEAunbtilu3br15dERERBWY1km6cuXK2LJlS77yLVu2oHLlygCAtLQ0WFpavnl0REREFZjWzd1fffUVRo8ejQMHDqj6pE+dOoXt27dj8eLFAIA9e/agbdu2JRspERFRBVOs66SjoqKwcOFCXL9+HQBQp04dfPLJJ6o7kJWGmTNnYtu2bYiOjoaxsTGSkpK0Xoc+XwtHRES6oc+5QeszaQBo3bo1WrduXdKxFCkrKwv9+vWDj48Pfv/99zLdNhERkS4UK0krFAps3LhR9VhKT09P9OjRAzKZrESDe9mMGTMAAJGRkaW2DSIiIn2idZKOjY1F165dkZCQgDp16gAAwsPD4eTkhG3btsHd3b3EgyQiIqqItB7dHRoaCnd3d9y9exdnz57F2bNnER8fD1dXV4SGhpZGjMWWmZmJlJQUtRcREVF5oXWSPnToEH744QfV5VYAYGtri++++w6HDh3Sal2TJ0+GJElFvq5du6ZtiCrh4eGwtrZWvfi8ayIiKk+0bu6Wy+V4/vx5vvLU1FQYGxtrta6wsDCEhIQUWcfNzU2rdb5sypQpmDBhgmo6JSWFiZqIiMoNrZN09+7dMWLECPz++++q66RPnDiBUaNGoUePHlqty87ODnZ2dtqGoDG5XA65XF5q6yciIipNWifp+fPnIzg4GD4+PjAyMgIA5OTkoEePHvj5559LPMA88fHxePr0KeLj46FQKBAdHQ0A8PDwgIWFRaltl4iISFeKdTMTAIiJiVH1F9erVw8eHh4lGtirQkJC8Oeff+YrP3DgANq1a6fROvT5gnUiItINfc4NxU7S5ZE+vxFERKQb+pwbtG7uVigUiIyMxL59+5CYmAilUqk2f//+/SUWHBERUUWmdZIeN24cIiMj0a1bNzRo0ACSJJVGXERERBWe1kl69erVWLNmDbp27Voa8RAREdH/p/XNTIyNjUt9kBgREREVI0mHhYXh559/RgUab0ZERKQTGjV39+7dW216//792LFjBzw9PVXXSudZv359yUVHRERUgWmUpK2trdWme/XqVSrBEBER0f/RKElHRESUdhxERET0Cq37pDt06ICkpKR85SkpKejQoUNJxEREREQoRpI+ePAgsrKy8pVnZGTgyJEjJRIUERERaXGd9IULF1T/v3LlCh48eKCaVigU2LlzJ6pXr16y0REREVVgGifpxo0bQ5IkSJJUYLO2qakpFixYUKLBERERVWQaJ+m4uDgIIeDm5oaTJ0+qPQfa2NgYVatWhUwmK5UgiYiIKiKNk7SzszOys7MRHBwMW1tbODs7l2ZcREREFZ5WA8eMjIywYcOG0oqFiIiIXqL16O6ePXti48aNpRAKERERvUzrp2DVqlULX3/9NaKiotCsWTOYm5urzQ8NDS2x4IiIiCoySWj5pAxXV9fCVyZJuHXr1hsHVVpSUlJgbW2N5ORkWFlZ6TocIiLSA/qcG7Q+k46LiyuNOIiIiOgVWvdJ53n8+DEeP35ckrEQERHRS7RK0klJSRgzZgyqVKkCe3t72Nvbo0qVKhg7dmyB9/MmIiKi4tO4ufvp06fw8fFBQkICBg0ahHr16gHIvUVoZGQk9u3bh3/++QeVKlUqtWCJiIgqEo2T9Ndffw1jY2PcvHkT9vb2+eZ17twZX3/9NebOnVviQRIREVVEGjd3b9y4EbNnz86XoAHAwcEBP/zwA290QkREVII0TtL379+Hp6dnofMbNGig9mQsIiIiejMaJ+kqVarg9u3bhc6Pi4tD5cqVSyImIiIighZJ2t/fH//5z3+QlZWVb15mZia++uorBAQElGhwREREFZnGdxz7999/4e3tDblcjjFjxqBu3boQQuDq1av49ddfkZmZidOnT8PJyam0Yy42fb6rDBER6YY+5waNR3fXqFEDx44dw8cff4wpU6YgL7dLkoROnTph4cKFep2giYiIyhutbgvq6uqKHTt24NmzZ4iJiQEAeHh4sC+aiIioFGh9724AqFSpElq0aFHSsRAREdFLin3v7rJ0+/ZtDB06FK6urjA1NYW7uzumTZtW4CA2IiKit0WxzqTL2rVr16BUKrFkyRJ4eHjg0qVLGD58ONLS0jB79mxdh0dERFQqtH6etL748ccfsWjRIq2eX63PI/iIiEg39Dk3lIsz6YIkJye/dsBaZmYmMjMzVdMpKSmlHRYREVGJKRd90q+KjY3FggULMHLkyCLrhYeHw9raWvXiJWJERFSe6DRJT548GZIkFfm6du2a2jIJCQkICAhAv379MHz48CLXP2XKFCQnJ6ted+/eLc3dISIiKlE67ZN+9OgRnjx5UmQdNzc3GBsbAwDu3buHdu3aoVWrVoiMjISBgXa/MfS534GIiHRDn3ODTvuk7ezsYGdnp1HdhIQEtG/fHs2aNUNERITWCZqIiKi8KRcDxxISEtCuXTs4Oztj9uzZePTokWqeg4ODDiMjIiIqPeUiSe/ZswexsbGIjY1FjRo11OaV0yvIiIiIXqtctBmHhIRACFHgi4iI6G1VLpI0ERFRRcQkTUREpKeYpImIiPQUkzQREZGeYpImIiLSU0zSREREeopJmoiISE8xSRMREempcnHHMaI3oVAokJ2dreswqBwxMjKCTCbTdRhETNL09hJC4MGDB0hKStJ1KFQO2djYwMHBAZIk6ToUqsCYpOmtlZegq1atCjMzM/6xJY0IIZCeno7ExEQAgKOjo44jooqMSZreSgqFQpWgbW1tdR0OlTOmpqYAgMTERFStWpVN36QzHDhGb6W8PmgzMzMdR0LlVd5nh+MZSJeYpOmtxiZuKi5+dkgfMEkTvcVu374NSZIQHR2t61CIqBiYpIn0SEhICCRJwqhRo/LNGzNmDCRJQkhISJnGFBkZCRsbmzLdJhHl4sAxqlCkGWXbhCmmCa2XcXJywurVqzF37lzVAKaMjAysWrUKNWvWLOkQy4xCoYAkSTAw4LkBkab4bSHSM02bNoWTkxPWr1+vKlu/fj1q1qyJJk2aqNXduXMn2rRpAxsbG9ja2qJ79+64efNmkeu/dOkSunTpAgsLC9jb2+PDDz/E48ePC6x78OBBDBkyBMnJyZAkCZIkYfr06QCAzMxMTJw4EdWrV4e5uTlatmyJgwcPqpbNOwPfvHkz6tevD7lcjvj4eLi4uODbb7/F4MGDYWFhAWdnZ2zevBmPHj1Cz549YWFhgUaNGuH06dOqdd25cweBgYGoVKkSzM3N4enpie3bt2t5ZInKHyZpIj300UcfISIiQjX9xx9/YMiQIfnqpaWlYcKECTh9+jT27dsHAwMD9OrVC0qlssD1JiUloUOHDmjSpAlOnz6NnTt34uHDh+jfv3+B9X19fTFv3jxYWVnh/v37uH//PiZOnAgAGDt2LI4dO4bVq1fjwoUL6NevHwICAhATE6NaPj09Hd9//z1+++03XL58GVWrVgUAzJ07F61bt8a5c+fQrVs3fPjhhxg8eDA++OADnD17Fu7u7hg8eDCEyG2JGDNmDDIzM3H48GFcvHgR33//PSwsLIp3cInKETZ3E+mhDz74AFOmTMGdO3cAAFFRUVi9erXamSoA9OnTR236jz/+gJ2dHa5cuYIGDRrkW+/ChQvRpEkTzJo1S20ZJycn3LhxA7Vr11arb2xsDGtra0iSBAcHB1V5fHw8IiIiEB8fj2rVqgEAJk6ciJ07dyIiIkK1/uzsbPz666/w8vJSW2/Xrl0xcuRIAMDUqVOxaNEiNG/eHP369QMAfP755/Dx8cHDhw/h4OCA+Ph49OnTBw0bNgQAuLm5aXYgico5JmkiPWRnZ4du3bohMjISQgh069YNVapUyVcvJiYGU6dOxYkTJ/D48WPVGXR8fHyBSfr8+fM4cOBAgWehN2/ezJekC3Px4kUoFIp89TMzM9VuHmNsbIxGjRrlW/7lMnt7ewBQJeCXyxITE+Hg4IDQ0FCMHj0au3fvhp+fH/r06VPgeoneNkzSRHrqo48+wtixYwEAv/zyS4F1AgMD4ezsjGXLlqFatWpQKpVo0KABsrKyCqyfmpqKwMBAfP/99/nmaXP7y9TUVMhkMpw5cybf3bhe/gFgampa4PXGRkZGqv/nzS+oLO9Hx7Bhw+Dv749t27Zh9+7dCA8Px08//YRPPvlE45iJyiMmaSI9FRAQgKysLEiSBH9//3zznzx5guvXr2PZsmV45513AABHjx4tcp1NmzbFunXr4OLiAkNDzb7+xsbGUCgUamVNmjSBQqFAYmKiatulzcnJCaNGjcKoUaMwZcoULFu2jEma3nocOEakp2QyGa5evYorV64UeO/oSpUqwdbWFkuXLkVsbCz279+PCRMmFLnOMWPG4OnTp3jvvfdw6tQp3Lx5E7t27cKQIUPyJeI8Li4uSE1Nxb59+/D48WOkp6ejdu3aGDRoEAYPHoz169cjLi4OJ0+eRHh4OLZt21Yi+/+y8ePHY9euXYiLi8PZs2dx4MAB1KtXr8S3Q6RvmKSJ9JiVlRWsrKwKnGdgYIDVq1fjzJkzaNCgAT799FP8+OOPRa6vWrVqiIqKgkKhQOfOndGwYUOMHz8eNjY2hV6/7Ovri1GjRmHAgAGws7PDDz/8AACIiIjA4MGDERYWhjp16iAoKAinTp0qlWu5FQoFxowZg3r16iEgIAC1a9fGr7/+WuLbIdI3ksi7xqECSElJgbW1NZKTkwv9w0dvh4yMDMTFxcHV1RUmJia6DofKIX6GKg59zg08kyYiItJTTNJERER6ikmaiIhITzFJExER6SkmaSIiIj1VbpJ0jx49ULNmTZiYmMDR0REffvgh7t27p+uwiIiISk25SdLt27fHmjVrcP36daxbtw43b95E3759dR0WEZFe2bx5M+rVq4dGjRrhs88+Q5UqVXD79m24uLggOjpaVc/b21v1wJYHDx6gf//+aNGiBRo2bIgvv/xSVS8mJgbdunVD8+bN0ahRIyxcuFA1T5IkzJo1Cy1atICrq6vak9uoZJSb24J++umnqv87Oztj8uTJCAoKQnZ2tto9f/WBJEl49uwZbGxs1MoXL16M58+fY9KkSboJjIjeaomJiRgyZAiOHDmC+vXrY+nSpXjy5MlrlwsODsYXX3yBtm3bIicnB927d8fatWvRu3dvvPfee1ixYgXq1q2L9PR0tGrVCi1btkTz5s0BAHK5HCdPnsS1a9fQvHlzfPjhhxrfcpZer1weyadPn2LlypXw9fUtMkFnZmYiMzNTNZ2SklIW4RVq1KhROt0+Eb3djh8/jkaNGqF+/foAgKFDh772/uZpaWnYt28fHj58qCpLTU3F9evXcf36dVy+fBkDBw5UzXv+/DmuXLmiStKDBg0CANStWxeGhoZ48OABatSoUdK7VmGVqyT9+eefY+HChapfc1u3bi2yfnh4OGbMmFFG0eUnhMDkyZNx9epVrF69Gj/88AOSkpIwb948REZGYsWKFbCzs8OlS5cgl8uxZs0a1XNyp02bhpUrV6JSpUrw9/fHihUrcPv2bZ3tC1VsLi4uGD9+PMaPH6/rUEgLLz+BzNDQUO3+7BkZGQBy/04BuQn+1TurXb58GZUrV1ZrJn/Vy8vIZDLk5OSUROj0/+m0T3ry5MmQJKnI17Vr11T1J02ahHPnzmH37t2QyWQYPHgwirqr6ZQpU5CcnKx63b17tyx2C0DuWfx7772H1NRUbNiwAWZmZvnqnDp1CrNmzcLFixfh5+enenzgtm3bsG7dOpw7dw4nT55EQkJCmcVNuhUSElLg9yA2NlbXob2R27dvQ5KkIv/YU9GUOUoosgp+CEoeHx8fXLhwQfV3848//lA9ttTDwwMnTpwAAJw8eRLXr18HkPto0fbt2+O7775TrefevXv4999/UadOHVhZWan1NcfGxuLp06clum9UOJ2eSYeFhSEkJKTIOnlnlgBQpUoVVKlSBbVr10a9evXg5OSE48ePw8fHp8Bl5XI55HJ5SYassW7duqFnz5746quvCq3j4+MDV1dX1f8XLFgAANi3bx/69esHS0tLALlNVgcOHCj9oCuAGVLZtqxME9O0XiYgICDfABw7O7uSCqnc08dxKKUl5d8UnFl2BmeXnUXq/VQAgLGlMRq+3xDNP24O+0b2avXt7Ozwxx9/oFevXjA2NkZAQABsbW0BAN9++y2Cg4OxZMkS+Pj4wNPTU7XcypUrMWHCBDRo0ACSJMHc3BxLlixBjRo1sHXrVowfPx5z586FQqFAlSpVsGrVqrI7CBWcTs+k7ezsULdu3SJfxsbGBS6b9zD4l/uc9UmHDh2wZ8+eIvvBNW0mernJit5+crkcDg4Oaq+8R1Vu2rQJTZs2hYmJCdzc3DBjxgy1z40kSViyZAm6d+8OMzMz1KtXD8eOHUNsbCzatWsHc3Nz+Pr64ubNm6plbt68iZ49e8Le3h4WFhZo3rw59u7dW2SMSUlJGDZsGOzs7GBlZYUOHTrg/PnzhdbP+zHapEkTSJKEdu3aqeb99ttvqFevHkxMTFC3bl21p1vlnYH/9ddfaNu2LUxMTLBy5UqEhIQgKCgIs2bNgr29PWxsbPD1118jJycHkyZNQuXKlVGjRg21HztZWVkYO3YsHB0dYWJiAmdnZ4SHh2v2ppQxIQQOTDuAec7zcGTmEVWCBoCs51k49/s5LPZajDV91iA7PVtt2Z49e+Lq1as4f/68qnUOyB3NffnyZZw/fx6LFy9GdHS06n2oWrUqVqxYgUuXLuHixYs4fvw4vLy8AADu7u7YsmULLly4gMuXL+PQoUOoXr26Ks6XB8g+fvwYLi4upXNQKqhycQnWiRMnsHDhQkRHR+POnTvYv38/3nvvPbi7uxd6Fq1rX3zxBXr37g0/Pz+NRle+rEOHDli3bh1SU1MhhMAff/xRSlFSeXLkyBEMHjwY48aNw5UrV7BkyRJERkZi5syZavW++eYbDB48GNHR0ahbty7ef/99jBw5ElOmTMHp06chhMDYsWNV9VNTU9G1a1fs27cP586dQ0BAAAIDAxEfH19oLP369UNiYiJ27NiBM2fOoGnTpujYsWOhzaAnT54EAOzduxf379/H+vXrAeSewU2dOhUzZ87E1atXMWvWLHz11Vf4888/1ZafPHkyxo0bh6tXr8Lf3x8AsH//fty7dw+HDx/GnDlzMG3aNHTv3h2VKlXCiRMnMGrUKIwcORL//vsvAGD+/PnYvHmz6lLOlStX6mVCEUJg5/idOPz1YQilgFDk79JT5uSepFzbeA3LOy9HTgb7gd9W5WLgmJmZGdavX49p06YhLS0Njo6OCAgIwJdffllmzdlPXzxFZHQkDt85jKSMJFjJrdCqRisMbTIU9hb2BS4zfvx4mJubo0OHDti1a5fG2+revTtOnDiBxo0bw8bGBm3bts13ORe9vbZu3QoLCwvVdJcuXbB27VrMmDEDkydPRnBwMIDcrqBvvvkGn332GaZN+79m9SFDhqB///4Acgdb+vj44KuvvlIlt3HjxmHIkCGq+l5eXqqzJiA3yW/YsAGbN29WS+Z5jh49ipMnTyIxMVH1/Zs9ezY2btyIv//+GyNGjMi3TF5zva2tLRwcHFTl06ZNw08//YTevXsDyD3jzvsBkrefQO53Ka9OnsqVK2P+/PkwMDBAnTp18MMPPyA9PR1ffPEFgNwxKd999x2OHj2KgQMHIj4+HrVq1UKbNm0gSRKcnZ0LfxN06PJfl3Fy/kmN6gqlwL/H/sW+L/bBf45/gXUeP35ckuFRGSsXSbphw4bYv3+/TradlpWGCbsnIDI6EtmK3GYlgdxftttitmHawWkY6DkQ87vMRyXTSrnzXxrMNnz4cAwfPhwAMH36dFV5SEiIWn989+7d0b17d9X0Z599hm+++QZCCISFheltiwGVvPbt22PRokWqaXNzcwDA+fPnERUVpXbmrFAokJGRgfT0dNXgxEaNGqnm29vn/oBs2LChWllGRgZSUlJgZWWF1NRUTJ8+Hdu2bcP9+/eRk5ODFy9eFHomff78eaSmpqr6OvO8ePFCrRn9ddLS0nDz5k0MHTpU9R0BgJycHFhbW6vV9fb2zre8p6cnDAz+rzHQ3t4eDRo0UE3LZDLY2toiMTERQO53rlOnTqhTpw4CAgLQvXt3dO7cWeN4y0rUD1GQDCQIZeGDYl8mlAKnF59GuxntILfUzRgcKj3lIknrSnJGMtr/2R4XHl6AQuQfVakUSiiFEv+79D+cSDiBI0OOFHpWra3Bgwfj9u3byMjIgKenJxYvXlwi6yX9Z25uDg8Pj3zlqampmDFjRr4zSkB9fMPLg6ryxjMUVJY3rmPixInYs2cPZs+eDQ8PD5iamqJv376qUcEFxeHo6Ki6W9XLtGnxSU3N7WddtmwZWrZsqTYvrw8+T94PlZe9OnhMkqQCy/L2s2nTpoiLi8OOHTuwd+9e9O/fH35+fvj77781jrm03Tt9Dw/OPdB6uZyMHFxYfgHNP25eClGRLjFJF0IIgT5r+hSaoF+mEArEJcWh68quODH8BAwN3vywbtiw4Y3XQW+Xpk2b4vr16wUm8DcRFRWFkJAQ9OrVC0Bu8izqmvymTZviwYMHMDQ01LhPN28A6MvX6drb26NatWq4deuW6oYYpc3KygoDBgzAgAED0LdvXwQEBODp06eoXLlymWz/deL2x0GSSQX2Q2uyLJP024dJuhD/3P0H++L2aVw/R5mDsw/OYtuNbehZt2cpRkYV1dSpU9G9e3fUrFkTffv2hYGBAc6fP49Lly7h22+/LfZ6a9WqhfXr1yMwMBCSJOGrr75SnX0WxM/PDz4+PggKCsIPP/yA2rVr4969e9i2bRt69epVYNN01apVYWpqip07d6JGjRowMTGBtbU1ZsyYgdDQUFhbWyMgIACZmZk4ffo0nj17hgkTJhR7nwoyZ84cODo6okmTJjAwMMDatWvh4OCgV+M9MpIzcpu6tU3SAsh4llE6QZFOlYvR3brwy6lftD4jlkkyLDy58PUViYrB398fW7duxe7du9G8eXO0atUKc+fOfeMBUHPmzEGlSpXg6+uLwMBA+Pv7o2nTpoXWlyQJ27dvx7vvvoshQ4agdu3aGDhwIO7cuaPqA3+VoaEh5s+fjyVLlqBatWro2TP3h+ywYcPw22+/ISIiAg0bNkTbtm0RGRmpumSrJFlaWuKHH36At7c3mjdvjtu3b2P79u1q/dq6ZmxuDGh/Ep27rGXBl6tS+SaJom7Z9ZZJSUmBtbU1kpOTYWVlVWi9HGUOzGaaIVuZXWidojz57Akqm+pH81lFlZGRgbi4OLi6uua71SGRJnTxGbq5+yZW+K/QejnJQELb6W3R9qu2pRDV20/T3KAL+vMTUo8kZSQVO0EDwKO0RyUYDRFVFG5+brB2tn59xVdJQNNhhbd+UPnFJF2ANx34VRIDx4io4pEMJLQc1xLQ4iaDkkxC/b71YeloWXqBkc4wSRfASm4FK3nxmjyMDIxK7DIsIqp4WoxtAbdObpAMXp+pDQwNYFXDCl0WdCmDyEgXmKQLYCAZYFiTYZBJstdXfomhgSEGNhgIC2OL11cmIiqAzEiGgRsGonZgbQCAZJg/Wecl8Mq1KmPIkSEwt8t/HTm9HZikCzHKe9Rrr49+VY4yB2OajymliIioojAyM8KA9QPwwe4PULtr7XzN347ejui1vBdGnh0Ja6di9GFTucHO00LUsq2FMJ8wzDk2R3Ub0KIYSAb4oOEHaFG9RRlER5qqQBcvUAnT9WdHMpDg3skd7p3ckfYoDSn/pkCZo4SFvQWsazIxVxRM0kX4odMPSMpIwu/nfocBDKBE/hs8SJAgIBBUNwjLeizjYyX1RN7tIdPT02FqaqrjaKg8Sk9PB5D/9qO6YG5nzibtCopJuggGkgGWBS7Du87vYvY/s3Ex8SJkkgwGkgGUQgmFUKCWbS182upTjGg2AgYSew/0hUwmg42NjerhCmZmZvwBRRoRQiA9PR2JiYmwsbHJdx9xorLEm5loSAiBkwkncfjOYaRkpsDC2AKtarTCu87v8o+/nhJC4MGDB0hKStJ1KFQO2djYwMHBgd/vCkCfb2bCJE1vPYVCgezs4t+chioeIyMjnkFXIPqcG9jcTW89mUzGP7hEVC6xE5WIiEhPMUkTERHpKSZpIiIiPVWh+qTzxsilpKToOBIiItIXeTlBH8dRV6gk/fz5cwCAk5OTjiMhIiJ98/z5c1hb69fd3CrUJVhKpRL37t2DpaWl6trHlJQUODk54e7du3o39F7f8di9GR6/N8Pj92Z4/P6PEALPnz9HtWrVYGCgX73AFepM2sDAADVq1ChwnpWVVYX/oBYXj92b4fF7Mzx+b4bHL5e+nUHn0a+fDERERKTCJE1ERKSnKnySlsvlmDZtGuRyua5DKXd47N4Mj9+b4fF7Mzx+5UOFGjhGRERUnlT4M2kiIiJ9xSRNRESkp5ikiYiI9BST9P93+/ZtDB06FK6urjA1NYW7uzumTZuGrKwsXYdWbsycORO+vr4wMzODjY2NrsPRe7/88gtcXFxgYmKCli1b4uTJk7oOqVw4fPgwAgMDUa1aNUiShI0bN+o6pHIjPDwczZs3h6WlJapWrYqgoCBcv35d12FREZik/79r165BqVRiyZIluHz5MubOnYvFixfjiy++0HVo5UZWVhb69euH0aNH6zoUvffXX39hwoQJmDZtGs6ePQsvLy/4+/sjMTFR16HpvbS0NHh5eeGXX37RdSjlzqFDhzBmzBgcP34ce/bsQXZ2Njp37oy0tDRdh0aF4OjuIvz4449YtGgRbt26petQypXIyEiMHz8eSUlJug5Fb7Vs2RLNmzfHwoULAeTestbJyQmffPIJJk+erOPoyg9JkrBhwwYEBQXpOpRy6dGjR6hatSoOHTqEd999V9fhUAF4Jl2E5ORkVK5cWddh0FsmKysLZ86cgZ+fn6rMwMAAfn5+OHbsmA4jo4omOTkZAPh3To8xSRciNjYWCxYswMiRI3UdCr1lHj9+DIVCAXt7e7Vye3t7PHjwQEdRUUWjVCoxfvx4tG7dGg0aNNB1OFSItz5JT548GZIkFfm6du2a2jIJCQkICAhAv379MHz4cB1Frh+Kc/yISP+NGTMGly5dwurVq3UdChXhrX8KVlhYGEJCQoqs4+bmpvr/vXv30L59e/j6+mLp0qWlHJ3+0/b40etVqVIFMpkMDx8+VCt/+PAhHBwcdBQVVSRjx47F1q1bcfjw4UKfDEj64a1P0nZ2drCzs9OobkJCAtq3b49mzZohIiJC754rqgvaHD/SjLGxMZo1a4Z9+/apBjwplUrs27cPY8eO1W1w9FYTQuCTTz7Bhg0bcPDgQbi6uuo6JHqNtz5JayohIQHt2rWDs7MzZs+ejUePHqnm8exGM/Hx8Xj69Cni4+OhUCgQHR0NAPDw8ICFhYVug9MzEyZMQHBwMLy9vdGiRQvMmzcPaWlpGDJkiK5D03upqamIjY1VTcfFxSE6OhqVK1dGzZo1dRiZ/hszZgxWrVqFTZs2wdLSUjUGwtraGqampjqOjgokSAghREREhABQ4Is0ExwcXODxO3DggK5D00sLFiwQNWvWFMbGxqJFixbi+PHjug6pXDhw4ECBn7Pg4GBdh6b3CvsbFxERoevQqBC8TpqIiEhPsdOViIhITzFJExER6SkmaSIiIj3FJE1ERKSnmKSJiIj0FJM0ERGRnmKSJiIi0lNM0kRERHqKSZr0Srt27TB+/Pgy2dbBgwchSRKSkpIAAJGRkbCxsSmTbb+qLLd9/fp1ODg44Pnz56W6nVePb3m1dOlSODk5wcDAAPPmzdNpLC4uLlrFsHjxYgQGBpZeQFTqmKRJzYMHDzBu3Dh4eHjAxMQE9vb2aN26NRYtWoT09HRdh1eqBgwYgBs3bpT6dgr6Q1tW2waAKVOm4JNPPoGlpWWZbK88S0lJwdixY/H5558jISEBI0aM0HVIWvnoo49w9uxZHDlyRNehUDHxARukcuvWLbRu3Ro2NjaYNWsWGjZsCLlcjosXL2Lp0qWoXr06evTooeswi6RQKCBJUrGeYGZqalrkQwaysrJgbGz8JuEVe9slJT4+Hlu3bsWCBQtKfVtvg/j4eGRnZ6Nbt25wdHTUdThaMzY2xvvvv4/58+fjnXfe0XU4VAw8kyaVjz/+GIaGhjh9+jT69++PevXqwc3NDT179sS2bdvUms2SkpIwbNgw2NnZwcrKCh06dMD58+dV86dPn47GjRtj+fLlcHFxgbW1NQYOHKjWxJqWlobBgwfDwsICjo6O+Omnn/LFlJmZiYkTJ6J69eowNzdHy5YtcfDgQdX8vGbizZs3o379+pDL5YiPjy9w/7Zv347atWvD1NQU7du3x+3bt9Xmv9rknLcPv/32G1xdXWFiYqLRvgPAli1b0Lx5c5iYmKBKlSro1asXgNzm/Dt37uDTTz+FJEmQJKnAbQPAokWL4O7uDmNjY9SpUwfLly9Xmy9JEn777Tf06tULZmZmqFWrFjZv3lzgvudZs2YNvLy8UL169Xz7vXXrVtSpUwdmZmbo27cv0tPT8eeff8LFxQWVKlVCaGgoFAqFarnly5fD29sblpaWcHBwwPvvv4/ExMQit3/06FG88847MDU1hZOTE0JDQ5GWllbkMgBw8eLF19bR1qVLl4qcHxkZiYYNGwLIfWa6JEmqz8ymTZvQtGlTmJiYwM3NDTNmzEBOTo5qWUmSsGTJEnTv3h1mZmaoV68ejh07htjYWLRr1w7m5ubw9fXFzZs3VcvcvHkTPXv2hL29PSwsLNC8eXPs3bu3yBg1+SwGBgZi8+bNePHihTaHh/SFrp/wQfrh8ePHQpIkER4erlF9Pz8/ERgYKE6dOiVu3LghwsLChK2trXjy5IkQQohp06YJCwsL0bt3b3Hx4kVx+PBh4eDgIL744gvVOkaPHi1q1qwp9u7dKy5cuCC6d+8uLC0txbhx41R1hg0bJnx9fcXhw4dFbGys+PHHH4VcLhc3btwQQuQ+vczIyEj4+vqKqKgoce3aNZGWlpYv3vj4eCGXy8WECRPEtWvXxIoVK4S9vb0AIJ49e6Zal7W1tWqZadOmCXNzcxEQECDOnj0rzp8/r9G+b926VchkMjF16lRx5coVER0dLWbNmiWEEOLJkyeiRo0a4uuvvxb3798X9+/fL3Db69evF0ZGRuKXX34R169fFz/99JOQyWRi//79qjoARI0aNcSqVatETEyMCA0NFRYWFqo4CtKjRw8xatQotbK8Y9ipUydx9uxZcejQIWFrays6d+4s+vfvLy5fviy2bNkijI2NxerVq1XL/f7772L79u3i5s2b4tixY8LHx0d06dJFNT/vaVV5xzc2NlaYm5uLuXPnihs3boioqCjRpEkTERISUmi8QgixadMmYWJiIk6dOlVkPW3cu3dPmJubi19//bXQOunp6WLv3r0CgDh58qS4f/++yMnJEYcPHxZWVlYiMjJS3Lx5U+zevVu4uLiI6dOnq5YFIKpXry7++usvcf36dREUFCRcXFxEhw4dxM6dO8WVK1dEq1atREBAgGqZ6OhosXjxYnHx4kVx48YN8eWXXwoTExNx584dVR1nZ2cxd+5c1fTrPotCCJGWliYMDAz4NLpyikmahBBCHD9+XAAQ69evVyu3tbUV5ubmwtzcXHz22WdCCCGOHDkirKysREZGhlpdd3d3sWTJEiFEboIzMzMTKSkpqvmTJk0SLVu2FEII8fz5c2FsbCzWrFmjmv/kyRNhamqqStJ37twRMplMJCQkqG2nY8eOYsqUKUKI/3vEaHR0dJH7N2XKFFG/fn21ss8///y1SdrIyEgkJiaqyjTZdx8fHzFo0KBCY3n1D21B2/b19RXDhw9Xq9OvXz/RtWtX1TQA8eWXX6qmU1NTBQCxY8eOQrft5eUlvv7663zbBiBiY2NVZSNHjhRmZmbi+fPnqjJ/f38xcuTIQtd96tQpAUC1zKtJeujQoWLEiBFqyxw5ckQYGBiIFy9eFLjOLVu2CBMTE/Hnn38Wut3i2rNnjzAzM1O9bwU5d+6cACDi4uJUZR07dlT96MqzfPly4ejoqJp+9b05duyYACB+//13Vdn//vc/YWJiUmSMnp6eYsGCBarplz87mnwW81SqVElERkYWuS3ST+yTpiKdPHkSSqUSgwYNQmZmJgDg/PnzSE1Nha2trVrdFy9eqDXfubi4qA1OcnR0VDWH3rx5E1lZWWjZsqVqfuXKlVGnTh3V9MWLF6FQKFC7dm217WRmZqpt29jYGI0aNSpyP65evaq2LQDw8fEpchkAcHZ2hp2dnWpak32Pjo7G8OHDX7vu18X76iCl1q1b4+eff1Yre3m/zc3NYWVlVWST84sXL1TN9i8zMzODu7u7atre3h4uLi6wsLBQK3t53WfOnMH06dNx/vx5PHv2DEqlEkBuP279+vXzbeP8+fO4cOECVq5cqSoTQkCpVCIuLg716tVTq5+dnY2+ffsiMzMTwcHBCA4OLnS/4uLi4OLikq88rzuhKKNGjUJAQABq1qz52rp5+xEVFYWZM2eqyhQKBTIyMpCeng4zMzMA6u+Nvb09AKiaz/PKMjIykJKSAisrK6SmpmL69OnYtm0b7t+/j5ycHLx48aLQ7htNv4dA7piHt33g59uKSZoAAB4eHpAkCdevX1crd3NzAwC1QU2pqalwdHRU6xvO83K/qpGRkdo8SZJUf8g1kZqaCplMhjNnzkAmk6nNezl5mJqaavTHuDjMzc3zxfS6fS+LAWB5tD3GVapUwbNnzzRaT1HrTktLg7+/P/z9/bFy5UrY2dkhPj4e/v7+yMrKKnDbqampGDlyJEJDQ/PNKyhBGhkZ4fvvv0dYWBh++eUXtG3bttD9ermP/WVXr14tdJmrV69i0KBBGD16tMYJGsjdjxkzZqB379755r38A+jl45f3+SyoLO+YTpw4EXv27MHs2bPh4eEBU1NT9O3bt8jjqcn3EACePn2q9mOTyg8maQIA2NraolOnTli4cCE++eSTfMnpZU2bNsWDBw9gaGhY4NmLJtzd3WFkZIQTJ06o/kA+e/YMN27cUP0xbtKkCRQKBRITE994ZGq9evXyDao6fvy41uvRZN8bNWqEffv2YciQIQXONzY2VhuAVVi8UVFRamePUVFRBZ6haqNJkya4cuXKG60DAK5du4YnT57gu+++g5OTEwDg9OnTRS7TtGlTXLlyBR4eHhpvZ9y4cVAqlQgLC0NUVBS8vLy0irNu3boFlj948AAff/wxRowYUeCAxaI0bdoU169f12o/NBEVFYWQkBDVIMPU1NR8gxtfjUOT7+HNmzeRkZGBJk2alGi8VDY4uptUfv31V+Tk5MDb2xt//fUXrl69iuvXr2PFihW4du2a6mzWz88PPj4+CAoKwu7du3H79m38888/+M9//vPaP9R5LCwsMHToUEyaNAn79+/HpUuXEBISonbpVO3atTFo0CAMHjwY69evR1xcHE6ePInw8HBs27ZNq30bNWoUYmJiMGnSJFy/fh2rVq1CZGSkVusANNv3adOm4X//+x+mTZuGq1ev4uLFi/j+++9V63BxccHhw4eRkJCAx48fF7idSZMmITIyEosWLUJMTAzmzJmD9evXY+LEiVrH/DJ/f38cO3bstT8SXqdmzZowNjbGggULcOvWLWzevBnffPNNkct8/vnn+OeffzB27FhER0cjJiYGmzZtwtixY4tc7tNPP8XKlSvf+AfKyxwcHLBw4cJi3Zxk6tSp+O9//4sZM2bg8uXLuHr1KlavXo0vv/zyjWKqVasW1q9fj+joaJw/fx7vv/9+ka0imn4Pjxw5Ajc3N7XuDCo/mKRJxd3dHefOnYOfnx+mTJkCLy8veHt7Y8GCBZg4caLqj7AkSdi+fTveffddDBkyBLVr18bAgQNx584dVd+bJn788Ue88847CAwMhJ+fH9q0aYNmzZqp1YmIiMDgwYMRFhaGOnXqICgoCKdOndKqeRLITSrr1q3Dxo0b4eXlhcWLF2PWrFlarQPQbN/btWuHtWvXYvPmzWjcuDE6dOiAkydPqtbx9ddf4/bt23B3dy+0CTIoKAg///wzZs+eDU9PTyxZsgQRERFo166d1jG/rEuXLjA0NHztpT2vY2dnh8jISKxduxb169fHd999h9mzZxe5TKNGjXDo0CHcuHED77zzDpo0aYKpU6eiWrVqr91ez5498zW/v6k+ffoUazl/f39s3boVu3fvRvPmzdGqVSvMnTsXzs7ObxTPnDlzUKlSJfj6+iIwMBD+/v5o2rRpofU1/R7+73//e+MxEqQ7khBC6DoIIio7v/zyCzZv3oxdu3bpOhQqZZcvX0aHDh1w48YNWFtb6zocKgb2SRNVMCNHjkRSUhKeP3/OW4O+5e7fv4///ve/TNDlGM+kiYiI9BT7pImIiPQUkzQREZGeYpImIiLSU0zSREREeopJmoiISE8xSRMREekpJmkiIiI9xSRNRESkp5ikiYiI9BSTNBERkZ76f2osEQYdFMx+AAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Create color mapping by word category\n", "colors = []\n", "\n", "for word in interesting_words:\n", " if word in ['man', 'king', 'actor']:\n", " colors.append('green')\n", "\n", " elif word in ['woman', 'queen', 'actress']:\n", " colors.append('purple')\n", "\n", "# Create 1x1 subplot\n", "fig, ax = plt.subplots(1, 1, figsize=(5, 4))\n", "\n", "# Word2Vec\n", "ax.scatter(w2v_2d[:, 0], w2v_2d[:, 1], c=colors, s=100)\n", "\n", "for i, word in enumerate(interesting_words):\n", " ax.annotate(\n", " word, (w2v_2d[i, 0], w2v_2d[i, 1]), \n", " xytext=(5, 5), textcoords='offset points', fontsize=8\n", ")\n", " \n", "ax.set_title('Word embeddings - gender direction')\n", "ax.set_xlabel('Gender direction (male ← → female)')\n", "ax.set_ylabel('Orthogonal dimension')\n", "\n", "# Add legend\n", "legend_elements = [\n", " Patch(facecolor='green', label='Male terms'),\n", " Patch(facecolor='purple', label='Female terms')\n", "]\n", "\n", "ax.legend(handles=legend_elements, loc='best')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "f38d4670", "metadata": {}, "source": [ "## 2. Document embeddings\n", "\n", "### 2.1. Averaging word vectors\n", "\n", "A simple approach to create document embeddings is to average the word vectors of all words in the document." ] }, { "cell_type": "code", "execution_count": 8, "id": "041d40b6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Document vectors shape: (6, 100)\n" ] } ], "source": [ "# Create document vector by averaging word vectors\n", "def get_doc_vector_avg(tokens, model):\n", "\n", " vectors = []\n", " for token in tokens:\n", " if token in model:\n", " vectors.append(model[token])\n", "\n", " if vectors:\n", " return np.mean(vectors, axis=0)\n", " return np.zeros(model.vector_size)\n", "\n", "# Create document vectors\n", "doc_vectors_avg = np.array([get_doc_vector_avg(doc, w2v_model) for doc in tokenized_docs])\n", "\n", "print(f'Document vectors shape: {doc_vectors_avg.shape}')" ] }, { "cell_type": "markdown", "id": "1d57cb08", "metadata": {}, "source": [ "### 2.2. Doc2Vec\n", "\n", "Doc2Vec learns document embeddings directly, treating each document as a unique entity during training.\n", "\n", "Gensim [Doc2Vec](https://radimrehurek.com/gensim/models/doc2vec.html) documentation" ] }, { "cell_type": "code", "execution_count": 9, "id": "44caf250", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Doc2Vec vectors shape: (6, 100)\n" ] } ], "source": [ "# Prepare tagged documents for Doc2Vec\n", "tagged_docs = [TaggedDocument(words=doc, tags=[str(i)]) for i, doc in enumerate(tokenized_docs)]\n", "\n", "# Train Doc2Vec model\n", "doc2vec_model = Doc2Vec(\n", " documents=tagged_docs,\n", " vector_size=100,\n", " window=2,\n", " min_count=1,\n", " epochs=100,\n", " seed=315\n", ")\n", "\n", "# Get document vectors\n", "doc_vectors_d2v = np.array([doc2vec_model.dv[str(i)] for i in range(len(documents))])\n", "\n", "print(f'Doc2Vec vectors shape: {doc_vectors_d2v.shape}')" ] }, { "cell_type": "markdown", "id": "a81e2b31", "metadata": {}, "source": [ "## 3. Document similarity comparison\n", "\n", "Compare how averaging and Doc2Vec capture document similarity." ] }, { "cell_type": "code", "execution_count": 10, "id": "2b0457aa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Document similarity (Averaging method):\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Doc 1Doc 2Doc 3Doc 4Doc 5Doc 6
Doc 11.0000.9330.6320.7540.7010.832
Doc 20.9331.0000.6700.7740.7470.867
Doc 30.6320.6701.0000.8440.5800.731
Doc 40.7540.7740.8441.0000.6300.772
Doc 50.7010.7470.5800.6301.0000.820
Doc 60.8320.8670.7310.7720.8201.000
\n", "
" ], "text/plain": [ " Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 Doc 6\n", "Doc 1 1.000 0.933 0.632 0.754 0.701 0.832\n", "Doc 2 0.933 1.000 0.670 0.774 0.747 0.867\n", "Doc 3 0.632 0.670 1.000 0.844 0.580 0.731\n", "Doc 4 0.754 0.774 0.844 1.000 0.630 0.772\n", "Doc 5 0.701 0.747 0.580 0.630 1.000 0.820\n", "Doc 6 0.832 0.867 0.731 0.772 0.820 1.000" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Compute cosine similarity matrices\n", "sim_matrix_avg = cosine_similarity(doc_vectors_avg)\n", "sim_matrix_d2v = cosine_similarity(doc_vectors_d2v)\n", "\n", "# Create dataframes with document labels\n", "doc_labels = [f'Doc {i+1}' for i in range(len(documents))]\n", "\n", "sim_avg_df = pd.DataFrame(sim_matrix_avg.round(3), index=doc_labels, columns=doc_labels)\n", "sim_d2v_df = pd.DataFrame(sim_matrix_d2v.round(3), index=doc_labels, columns=doc_labels)\n", "\n", "print('Document similarity (Averaging method):')\n", "sim_avg_df" ] }, { "cell_type": "code", "execution_count": 11, "id": "421bb6b9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Document similarity (Doc2Vec method):\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Doc 1Doc 2Doc 3Doc 4Doc 5Doc 6
Doc 11.0000.2900.3560.2520.2740.286
Doc 20.2901.0000.3640.2500.3600.287
Doc 30.3560.3641.0000.4330.3240.336
Doc 40.2520.2500.4331.0000.3390.362
Doc 50.2740.3600.3240.3391.0000.249
Doc 60.2860.2870.3360.3620.2491.000
\n", "
" ], "text/plain": [ " Doc 1 Doc 2 Doc 3 Doc 4 Doc 5 Doc 6\n", "Doc 1 1.000 0.290 0.356 0.252 0.274 0.286\n", "Doc 2 0.290 1.000 0.364 0.250 0.360 0.287\n", "Doc 3 0.356 0.364 1.000 0.433 0.324 0.336\n", "Doc 4 0.252 0.250 0.433 1.000 0.339 0.362\n", "Doc 5 0.274 0.360 0.324 0.339 1.000 0.249\n", "Doc 6 0.286 0.287 0.336 0.362 0.249 1.000" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print('Document similarity (Doc2Vec method):')\n", "sim_d2v_df" ] }, { "cell_type": "code", "execution_count": 12, "id": "c2040442", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Expected similar pairs:\n", " - Doc 1 & Doc 2 (both about royalty/governance)\n", " - Doc 3 & Doc 4 (both about machine learning)\n", " - Doc 5 & Doc 6 (both about pets/animals)\n", "\n", "Actual most similar pairs (averaging):\n", " Doc 1 most similar to: Doc 2\n", " Doc 3 most similar to: Doc 4\n", " Doc 5 most similar to: Doc 6\n" ] } ], "source": [ "# Find most similar document pairs\n", "print('Expected similar pairs:')\n", "print(' - Doc 1 & Doc 2 (both about royalty/governance)')\n", "print(' - Doc 3 & Doc 4 (both about machine learning)')\n", "print(' - Doc 5 & Doc 6 (both about pets/animals)')\n", "\n", "print('\\nActual most similar pairs (averaging):')\n", "print(f' Doc 1 most similar to: Doc {np.argsort(sim_matrix_avg[0])[-2] + 1}')\n", "print(f' Doc 3 most similar to: Doc {np.argsort(sim_matrix_avg[2])[-2] + 1}')\n", "print(f' Doc 5 most similar to: Doc {np.argsort(sim_matrix_avg[4])[-2] + 1}')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 5 }