{
"cells": [
{
"cell_type": "code",
"execution_count": 213,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"%%capture\n",
"%load_ext autoreload\n",
"%autoreload 2\n",
"%matplotlib inline\n",
"# %cd .. \n",
"import sys\n",
"sys.path.append(\"..\")\n",
"import statnlpbook.util as util\n",
"import matplotlib\n",
"matplotlib.rcParams['figure.figsize'] = (10.0, 5.0)\n",
"import random \n",
"from collections import defaultdict\n",
"random.seed(2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"source": [
"\n",
"$$\n",
"\\newcommand{\\Xs}{\\mathcal{X}}\n",
"\\newcommand{\\Ys}{\\mathcal{Y}}\n",
"\\newcommand{\\y}{\\mathbf{y}}\n",
"\\newcommand{\\weights}{\\mathbf{w}}\n",
"\\newcommand{\\balpha}{\\boldsymbol{\\alpha}}\n",
"\\newcommand{\\bbeta}{\\boldsymbol{\\beta}}\n",
"\\newcommand{\\aligns}{\\mathbf{a}}\n",
"\\newcommand{\\align}{a}\n",
"\\newcommand{\\source}{\\mathbf{s}}\n",
"\\newcommand{\\target}{\\mathbf{t}}\n",
"\\newcommand{\\ssource}{s}\n",
"\\newcommand{\\starget}{t}\n",
"\\newcommand{\\repr}{\\mathbf{f}}\n",
"\\newcommand{\\repry}{\\mathbf{g}}\n",
"\\newcommand{\\bar}{\\,|\\,}\n",
"\\newcommand{\\x}{\\mathbf{x}}\n",
"\\newcommand{\\prob}{p}\n",
"\\newcommand{\\Pulp}{\\text{Pulp}}\n",
"\\newcommand{\\Fiction}{\\text{Fiction}}\n",
"\\newcommand{\\PulpFiction}{\\text{Pulp Fiction}}\n",
"\\newcommand{\\pnb}{\\prob^{\\text{NB}}}\n",
"\\newcommand{\\vocab}{V}\n",
"\\newcommand{\\params}{\\boldsymbol{\\theta}}\n",
"\\newcommand{\\param}{\\theta}\n",
"\\DeclareMathOperator{\\perplexity}{PP}\n",
"\\DeclareMathOperator{\\argmax}{argmax}\n",
"\\DeclareMathOperator{\\argmin}{argmin}\n",
"\\newcommand{\\train}{\\mathcal{D}}\n",
"\\newcommand{\\counts}[2]{\\#_{#1}(#2) }\n",
"\\newcommand{\\length}[1]{\\text{length}(#1) }\n",
"\\newcommand{\\indi}{\\mathbb{I}}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 214,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%HTML\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Text Classification "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Automatically classify input text into a set of **atomic classes**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Motivation\n",
"* **Information retrieval**: classify documents into topics, such as \"sport\" or \"business\"\n",
"* **Sentiment analysis**: classify tweets into being \"positive\" or \"negative\" \n",
"* **Spam filters**: distinguish between \"ham\" and \"spam\"\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Text Classification as Structured Prediction\n",
"Simplest instance of [structured prediction](/template/statnlpbook/02_methods/00_structuredprediction) \n",
"\n",
"* Input space $\\Xs$ are sequences of words\n",
"* output space $\\Ys$ is a set of labels\n",
" * $\\Ys=\\{ \\text{sports},\\text{business}\\}$ in document classification\n",
" * $\\Ys=\\{ \\text{positive},\\text{negative}, \\text{neutral}\\}$ in sentiment prediction\n",
"* model $s_{\\params}(\\x,y)$ \n",
" * scores $y$ highly if it fits text $\\x$\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Overview\n",
"\n",
"* **Generative** model: Naive Bayes\n",
" * Learn how to *explain input and output*\n",
"* **Discriminative** model: *Logistic Regression*\n",
" * Learn how to *discriminate best output based on input*"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Sentiment Analysis as Text Classification\n",
"Let us focus on a specific task: sentiment analysis\n",
"\n",
"* load data for this task from the [Movie Review dataset](https://www.cs.cornell.edu/people/pabo/movie-review-data/)"
]
},
{
"cell_type": "code",
"execution_count": 215,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"from os import listdir\n",
"from os.path import isfile, join\n",
"def load_from_dir(directory,label):\n",
" \"\"\"\n",
" Load documents from a directory, and give them all the same label `label`.\n",
" Params:\n",
" directory: the directory to load the documents from.\n",
" label: the label to assign to each document.\n",
" Returns:\n",
" a list of (x,y) pairs where x is a tokenised document (list of words), and y is the label `label`.\n",
" \"\"\"\n",
" result = []\n",
" for file in listdir(directory):\n",
" with open(directory + file, 'r') as f:\n",
" text = f.read()\n",
" tokens = [t.strip() for t in text.split()]\n",
" result.append((tokens,label))\n",
" return result\n",
" \n",
"data_pos = load_from_dir('../data/rt-2k/txt_sentoken/pos/', 'pos') \n",
"data_neg = load_from_dir('../data/rt-2k/txt_sentoken/neg/', 'neg')\n",
" \n",
"data_all = data_pos + data_neg"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let us look at some example data ..."
]
},
{
"cell_type": "code",
"execution_count": 216,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"in 1970s , many european intellectuals , especially those on the left political hemisphere , became obsessed with the rise of fascism . which wasn't so hard to expect , because the social turmoil of 1960s and economic decline of 1970s seemed to be the breeding ground for many dangerous ideologies . in such times , when political involvement could be associated with noble passion , many filmmakers tried to warn the present generations of dangers that lurk ahead by giving the look of pre-war europe and circumstances that led to phenomena like fascist italy and nazi germany . of course , there were authors who jumped on the bandwagon for other , less noble reasons . for them , moral depravity of fascism could be explained to the audience by explicitly showing sexual depravity of those era . which , naturally , made some of those films very popular among teen audience . one of such filmmakers was italian director tinto brass , who later made career shooting expensive , stylish soft porn . salon kitty , his 1976 film , is very losely based on the novel by peter nordern , book that deals with bizarre yet true\""
]
},
"execution_count": 216,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\" \".join(data_neg[11][0][:200])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Setup\n",
"\n",
"Divide dataset in **train**, **test** and **development** set"
]
},
{
"cell_type": "code",
"execution_count": 217,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"815"
]
},
"execution_count": 217,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"random.seed(0)\n",
"shuffled = list(data_all)\n",
"random.shuffle(shuffled)\n",
"train, dev, test = shuffled[:1600], shuffled[1600:1800], shuffled[1800:]\n",
"len([(x,y) for (x,y) in train if y == 'pos']) # check balance "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"The simplest and most\n",
"\n",
"## Naive Bayes Model\n",
"\n",
"uses a distribution $p^{\\mbox{NB}}_{\\params}$ for $s_\\params$: \n",
"\n",
"\\begin{equation}\n",
" s_{\\params}(\\x,\\y)\\ = p^{\\text{NB}}_{\\params}(\\x,y)\n",
"\\end{equation}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Naive Bayes is an instance of a \n",
"\n",
"### Noisy Channel Model\n",
"\n",
"\\begin{equation}\n",
"\\prob^{\\text{NB}}_{\\params}(\\x,y)= p^\\text{NB}_{\\params}(y) p^{\\text{NB}}_{\\params}(\\x|y) \n",
"\\end{equation}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Why Naive?\n",
"\n",
"Makes a naive conditional independence assumption\n",
"\n",
"\\begin{equation}\n",
" p^{\\text{NB}}_{\\params}(\\x|y) = \n",
" \\prod_i^{\\text{length}(\\x)} p^{\\text{NB}}_{\\params}(x_i|y)\n",
"\\end{equation}\n",
"\n",
"observed words are independent of each other\n",
"* when *conditioned on the label* $y$ "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"For example:\n",
"\n",
"$$\n",
"p^{\\text{NB}}_{\\params}(\\text{great,tremendous}\\bar+) = p^{\\text{NB}}_{\\params}(\\text{great}\\bar+) p^{\\text{NB}}_{\\params}(\\text{tremendous}\\bar+)\n",
"$$\n",
"\n",
"* Sometimes this is (approximately) true! \n",
"* Even when it isn't NB may still work...\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Parametrization\n",
"The NB model has the parameters $\\params=(\\balpha,\\bbeta)$ where \n",
"\n",
"\\begin{split}\n",
" p^{\\text{NB}}_{\\params}(w|y) & = \\alpha_{w,y} \\\\\\\\\n",
" p^{\\text{NB}}_{\\params}(y) & = \\beta_{y}.\n",
"\\end{split}\n",
"\n",
"* $\\balpha$ for per-class feature probabilities\n",
"* $\\bbeta$ for class priors "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Training Naive Bayes \n",
"\n",
"Maximum Likelihood estimation: \n",
"\n",
"\\begin{split}\n",
" \\alpha_{w,y} & = \\frac{\\counts{\\train}{w,y}}{\\sum_{w'}\\counts{\\train}{w',y}}\\\\\\\\\n",
" \\beta_{y} & = \\frac{\\counts{\\train}{y}}{\\left| \\train \\right|}\n",
"\\end{split}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"MLE can produce zero probabilities for unseen events in two ways:\n",
"\n",
"* Words in the training set, unseen for specific class\n",
" * address via [Laplace Smoothing](language_models.ipynb) \n",
"* Words outside of the training set\n",
" * address later ..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Laplace Smoothing\n",
"\n",
"Let $V$ be the full training set vocabulary, and $\\gamma$ a pseudo-count, then\n",
"\n",
"\\begin{split}\n",
" \\alpha^\\gamma_{w,y} & = \\frac{\\counts{\\train}{w,y} + \\gamma}{|V|\\gamma + \\sum_{w'}\\counts{\\train}{w',y}}\\\\\\\\\n",
"\\end{split}"
]
},
{
"cell_type": "code",
"execution_count": 218,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def train_nb(data, pseudo_count=0.0):\n",
" alpha = defaultdict(float)\n",
" beta = defaultdict(float)\n",
" vocab = set(w for x,_ in data for w in x)\n",
" labels = set(y for _,y in data)\n",
" norm = 0\n",
" for x,y in data:\n",
" for w in x:\n",
" beta[y] += 1.0\n",
" alpha[w,y] += 1\n",
" norm += 1\n",
" for y in labels:\n",
" for w in vocab:\n",
" alpha[w,y] = (alpha[w,y]+pseudo_count) / (beta[y] + len(vocab) * pseudo_count)\n",
"\n",
" for y in list(beta.keys()):\n",
" beta[y] = beta[y] / norm\n",
" return (alpha, beta)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Train NB on data:"
]
},
{
"cell_type": "code",
"execution_count": 219,
"metadata": {},
"outputs": [],
"source": [
"theta = (alpha, beta) = train_nb(train) "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"**Inspect** the learned parameters of the NB model! \n",
"\n",
"* they are easy to interpret"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The class prior $\\bbeta$ looks sensible:"
]
},
{
"cell_type": "code",
"execution_count": 220,
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"data": {
"text/plain": [
"defaultdict(float, {'neg': 0.46393907123093997, 'pos': 0.53606092876906})"
]
},
"execution_count": 220,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"beta"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"The per-class word distributions $\\balpha$:"
]
},
{
"cell_type": "code",
"execution_count": 221,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def plot_top_k(alpha, label='pos', k=10):\n",
" positive_words = [w for (w,y) in alpha.keys() if y == label]\n",
" sorted_positive_words = sorted(positive_words, key=lambda w:-alpha[w,label])[:k]\n",
" util.plot_bar_graph([alpha[w,label] for w in sorted_positive_words],sorted_positive_words,rotation=45)"
]
},
{
"cell_type": "code",
"execution_count": 222,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAl0AAAE9CAYAAAAxntfsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFp1JREFUeJzt3X+w5Xdd3/HXmw0BhNpYWSAkwU3HFZtqgbjG+APrL5hs\noizV8iNQoql1iSQDCLYsUnSoQw2OpRAI2UZJMYhGZqp2JTtEq6jTjtEsCIEAaXfSKMmkZhESwCgh\n8O4f57szl8vCPbt78zn33H08Zu7sPd/v92Tf38nee57ne77ne6q7AwDAg+shix4AAOBEILoAAAYQ\nXQAAA4guAIABRBcAwACiCwBgANEFADCA6AIAGEB0AQAMILoAAAY4adEDHMmjH/3o3rZt26LHAABY\n03vf+96Pd/fWtbbbkNG1bdu2HDhwYNFjAACsqar+cp7tvLwIADCA6AIAGEB0AQAMILoAAAYQXQAA\nA4guAIABRBcAwACiCwBgANEFADCA6AIAGEB0AQAMsCE/e3GEbXuuX/QIa7r98gsWPQIAsE4c6QIA\nGEB0AQAMILoAAAYQXQAAA4guAIABTth3L2423o0JABubI10AAAOILgCAAUQXAMAAogsAYADRBQAw\ngOgCABhAdAEADDBXdFXVeVV1a1UdrKo9R1hfVXXFtP7mqjp7xbrbq+qDVfX+qjqwnsMDACyLNS+O\nWlVbklyZ5GlJ7khyU1Xt6+4Pr9hsZ5Lt09e3Jblq+vOw7+3uj6/b1AAAS2aeI13nJDnY3bd19/1J\nrkuya9U2u5Jc2zM3Jjmlqk5d51kBAJbWPNF1WpKPrbh9x7Rs3m06yf+oqvdW1e5jHRQAYJmN+OzF\n7+ruO6vqMUl+v6o+2t1/snqjKch2J8kTnvCEAWMBAIwzz5GuO5OcseL26dOyubbp7sN/3p3ktzN7\nufJLdPfV3b2ju3ds3bp1vukBAJbEPNF1U5LtVXVmVZ2c5LlJ9q3aZl+Si6Z3MZ6b5N7uvquqHllV\n/yBJquqRSZ6e5EPrOD8AwFJY8+XF7n6gqi5LckOSLUmu6e5bquqSaf3eJPuTnJ/kYJL7klw83f2x\nSX67qg7/Xb/e3e9e970AANjg5jqnq7v3ZxZWK5ftXfF9J7n0CPe7LcmTjnNGAICl54r0AAADiC4A\ngAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwgugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDo\nAgAYQHQBAAwgugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwgugAABhBdAAAD\niC4AgAFEFwDAAKILAGAA0QUAMMBJix4AVtu25/pFjzCX2y+/YNEjALBEHOkCABhAdAEADCC6AAAG\nEF0AAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhgruiqqvOq6taqOlhVe46wvqrqimn9zVV19qr1\nW6rqL6rqXes1OADAMlkzuqpqS5Irk+xMclaSC6vqrFWb7UyyffraneSqVetfkuQjxz0tAMCSmudI\n1zlJDnb3bd19f5Lrkuxatc2uJNf2zI1JTqmqU5Okqk5PckGSX1nHuQEAlso80XVako+tuH3HtGze\nbd6Q5N8l+cIxzggAsPQe1BPpq+oHk9zd3e+dY9vdVXWgqg4cOnTowRwLAGC4eaLrziRnrLh9+rRs\nnm2+M8kzqur2zF6W/L6q+rUj/SXdfXV37+juHVu3bp1zfACA5TBPdN2UZHtVnVlVJyd5bpJ9q7bZ\nl+Si6V2M5ya5t7vv6u5Xdvfp3b1tut8fdve/Ws8dAABYBiettUF3P1BVlyW5IcmWJNd09y1Vdcm0\nfm+S/UnOT3IwyX1JLn7wRgYAWD5rRleSdPf+zMJq5bK9K77vJJeu8d/4oyR/dNQTAgBsAq5IDwAw\ngOgCABhAdAEADCC6AAAGEF0AAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhAdAEADCC6AAAGEF0A\nAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhAdAEADCC6AAAGEF0AAAOILgCAAUQXAMAAogsAYADR\nBQAwgOgCABhAdAEADCC6AAAGEF0AAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhAdAEADCC6AAAG\nEF0AAAOILgCAAUQXAMAAc0VXVZ1XVbdW1cGq2nOE9VVVV0zrb66qs6flD6+qP6+qD1TVLVX1mvXe\nAQCAZbBmdFXVliRXJtmZ5KwkF1bVWas225lk+/S1O8lV0/LPJvm+7n5SkicnOa+qzl2n2QEAlsY8\nR7rOSXKwu2/r7vuTXJdk16ptdiW5tmduTHJKVZ063f7MtM1Dp69er+EBAJbFPNF1WpKPrbh9x7Rs\nrm2qaktVvT/J3Ul+v7v/7NjHBQBYTg/6ifTd/fnufnKS05OcU1XfdKTtqmp3VR2oqgOHDh16sMcC\nABhqnui6M8kZK26fPi07qm26+54k70ly3pH+ku6+urt3dPeOrVu3zjEWAMDymCe6bkqyvarOrKqT\nkzw3yb5V2+xLctH0LsZzk9zb3XdV1daqOiVJquoRSZ6W5KPrOD8AwFI4aa0NuvuBqrosyQ1JtiS5\nprtvqapLpvV7k+xPcn6Sg0nuS3LxdPdTk/zq9A7IhyR5Z3e/a/13AwBgY1szupKku/dnFlYrl+1d\n8X0nufQI97s5yVOOc0YAgKXnivQAAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhAdAEADCC6AAAG\nEF0AAAOILgCAAUQXAMAAogsAYADRBQAwgOgCABhAdAEADCC6AAAGOGnRA8Bmt23P9YseYS63X37B\nokcA2NQc6QIAGEB0AQAM4OVF4Kgsw8ulXioFNiJHugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUA\nMIDoAgAYwHW6gBOa644BozjSBQAwgOgCABhAdAEADCC6AAAGEF0AAAOILgCAAUQXAMAAogsAYADR\nBQAwgOgCABjAxwABbCI+1gg2rrmOdFXVeVV1a1UdrKo9R1hfVXXFtP7mqjp7Wn5GVb2nqj5cVbdU\n1UvWewcAAJbBmtFVVVuSXJlkZ5KzklxYVWet2mxnku3T1+4kV03LH0jy8u4+K8m5SS49wn0BADa9\neY50nZPkYHff1t33J7kuya5V2+xKcm3P3JjklKo6tbvv6u73JUl3fzrJR5Kcto7zAwAshXmi67Qk\nH1tx+458aTituU1VbUvylCR/drRDAgAsuyHvXqyqRyX5b0le2t2f+jLb7K6qA1V14NChQyPGAgAY\nZp7oujPJGStunz4tm2ubqnpoZsH1ju7+rS/3l3T31d29o7t3bN26dZ7ZAQCWxjzRdVOS7VV1ZlWd\nnOS5Sfat2mZfkoumdzGem+Te7r6rqirJW5N8pLtfv66TAwAskTWv09XdD1TVZUluSLIlyTXdfUtV\nXTKt35tkf5LzkxxMcl+Si6e7f2eSFyT5YFW9f1r2M929f313AwBgY5vr4qhTJO1ftWzviu87yaVH\nuN//TFLHOSMAwNLzMUAAAAOILgCAAUQXAMAAogsAYADRBQAwwFzvXgSA0bbtuX7RI8zl9ssvWPQI\nLAlHugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwgugAABhBdAAADiC4AgAFE\nFwDAAKILAGAA0QUAMMBJix4AAE4E2/Zcv+gR1nT75RcseoRNzZEuAIABRBcAwACiCwBgANEFADCA\n6AIAGEB0AQAMILoAAAYQXQAAA4guAIABRBcAwACiCwBgANEFADCA6AIAGEB0AQAMILoAAAYQXQAA\nA4guAIABRBcAwABzRVdVnVdVt1bVwarac4T1VVVXTOtvrqqzV6y7pqrurqoPrefgAADL5KS1Nqiq\nLUmuTPK0JHckuamq9nX3h1dstjPJ9unr25JcNf2ZJG9L8uYk167f2ADAIm3bc/2iR1jT7ZdfsOgR\nvsg8R7rOSXKwu2/r7vuTXJdk16ptdiW5tmduTHJKVZ2aJN39J0k+sZ5DAwAsm3mi67QkH1tx+45p\n2dFuAwBwwtowJ9JX1e6qOlBVBw4dOrTocQAA1tU80XVnkjNW3D59Wna023xF3X11d+/o7h1bt249\nmrsCAGx480TXTUm2V9WZVXVykucm2bdqm31JLprexXhuknu7+651nhUAYGmtGV3d/UCSy5LckOQj\nSd7Z3bdU1SVVdcm02f4ktyU5mOSXk7zo8P2r6jeS/GmSJ1bVHVX14+u8DwAAG96al4xIku7en1lY\nrVy2d8X3neTSL3PfC49nQACAzWDDnEgPALCZiS4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwg\nugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwgugAABhBdAAADiC4AgAFEFwDA\nAKILAGAA0QUAMIDoAgAYQHQBAAwgugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQB\nAAwgugAABhBdAAADiC4AgAFEFwDAAKILAGAA0QUAMIDoAgAYQHQBAAwwV3RV1XlVdWtVHayqPUdY\nX1V1xbT+5qo6e977AgCcCNaMrqrakuTKJDuTnJXkwqo6a9VmO5Nsn752J7nqKO4LALDpzXOk65wk\nB7v7tu6+P8l1SXat2mZXkmt75sYkp1TVqXPeFwBg05snuk5L8rEVt++Yls2zzTz3BQDY9E5a9ACH\nVdXuzF6aTJLPVNWti5znGD06ycfX6z9Wr1uv/9IxWdd9SezPOrM/X8Fm2pfE/qwz+/MVLHhfkuXd\nn6+bZ6N5ouvOJGesuH36tGyebR46x32TJN19dZKr55hnw6qqA929Y9FzrIfNtC+J/dnoNtP+bKZ9\nSezPRmd/lss8Ly/elGR7VZ1ZVScneW6Sfau22ZfkouldjOcmube775rzvgAAm96aR7q6+4GquizJ\nDUm2JLmmu2+pqkum9XuT7E9yfpKDSe5LcvFXuu+DsicAABvYXOd0dff+zMJq5bK9K77vJJfOe99N\nbKlfHl1lM+1LYn82us20P5tpXxL7s9HZnyVSs14CAODB5GOAAAAGEF0AAAOILmBDqKpa+ecyqqrv\nqKonLnoOYGMSXbBkquoRi57hQfJPktkbc5YxvKbL5bwtyeeq6mELHmddLeP/j5WWfX42D9HFl/hy\nv6Cqyr+XBZsuwfKLVfULVfUPFz3Pepiu73dSkv9eVW9Pli+8pp+Nr0/yW0m2JXnhtE9L7XA89hK/\n46qq6vD8VfUNVfXIRc90vJbpZ2MtVfWjVXXdoucYxbsX+SKrfkE9P8kXkpzc3b+62MmoqhcleU6S\n5yV5X2bXv/v57v4/Cx3sOFXVlu7+/PT9wSS/090/Pd2ujf6Af3jG6cH8tswuxfPE7l7Xj5oZrape\nkuRJSU5N8h+SfKS771nsVPM7HCYrfp+9LMl5SX50unj3Ulr1O/r8JJXkT7v7E4ud7OhV1S8l+ZUk\nP5rklO7+yQWP9KBz5GIdVdVDFz3D8Vrxw/zSJD+R5IEkP1NVz1voYCe4qvrqJGdn9qkOP5LkL6ZV\nV1TV9oUNtg5WBNd5Sd6V5EVVdcW0bkMf8aqqh6yIwlOS/F6Su5M8e3FTHb/pwfzizGLrQGYPiv98\nWrdh/3+ssmXVE8hnJXlWd99VVY+rqsctdryjMx0RfsiKfbo4yS8meUWSn6uq717ogMfmDzN7Sf6t\nSX5hsaOMIbrWyfTywtur6qcWPcvxml62+tbu/p4k25PcmuQ3l/1coumjqJZSd38qswsQPybJv+ju\n8zJ7IPzWJC9Y5n1Lkqp6VpIrk7wlyc4kT6+qvcnGDq/u/kKSVNULk+zJLLheneQVVfXyRc52NKrq\n5Ko6a/r+u5M8Pcnvdvft3f3qzI6s7qmqr9roRx6TpKq2JvmtFf9utmQW9OdX1b9P8jtJXltV37io\nGY/BlhX/3i5I8sNJvjnJ9yf5dJIfqqqnLnC+ozZdPP1VSR7e3X+16HlGEF3rZPpheG2S51XV2Yue\n52gc4VytLUkeWVW/nGRHkudMRyOeU1VL+UGkVfXwJG9b5hOcu/uzmX3M1klV9c1JLkjyB0l+pbvv\nX+hwx+8LSd7W3f+7u/84s6Mqz6yqq5KNfU5RVf1Ikhdn9mz95CRPSHJNkh+rqmV59v6EJG+oqnck\neUlmkfX4w1HS3Vcn+eskX7e4EefX3YcyOyr8tKr6R0n+PMnjk+xO8oEkL88sVDZkzK+2MiKncwV3\nZPYzck53fy7JG5N8NsmFVfUdCxz1qHX3H3T3hxY9xyiiax119wczO9T7l4ue5WisePZ0VlU9bDo3\n4I+SPDPJnu7+u6q6KMnLkvy/xU167Lr775P8xBQuy+yvMnvG/vokr0vymmV7hvhljlr9XZJnHz5i\n191/neQ3kpxXVY/dqEe6Jk9M8l+7+/2ZPZjfl+RrklyY5KlV9ehFDjeP7j6Y5OYkP5Tk97r72sxC\n+F9W1bOnI5HfkORvFjjmUenu+5I8IrOAvLO7L01yQXf/bpKvzSxa7lvgiHNbEZE/kORR3f2aJG/K\n7OjjP5vWvymz388HFzcpa3Ei/TpbhhN/D6uqpyR5andfMZ2kfWmSO5P8l8yeDe5K8qIk1yf5riQv\n8IHlizedO/i4JF/o7jsXPc/RWHUS8E9m9kD+3iTvSPIfMwv9S5I8Jck5SV4yPaBsWFX1zMzOf3pl\nd394WvbHmb3888nDT2o2uqr6+iTfntmTq1dldnToBzM7F+pQktd3982Lm/DYVNXOJG9OsqO7P1lV\nF2b2UvDzl+0Iy/Rv7c1J/mmSTyV5ZZJvSfLa7n7fdM7XUvx7O1GJrhPUdOTgB5L8dGbPBP9xkhdm\n9gv2KZn9wv31JE/O7GT6v+nupTqCx8ZVVd+TWWTtS3JGkr9P8m+THA6xr88sYjb8g3xVnZLZz1Fl\ndoT4EUl+NsnOjR6MR1JVz8jsVIkXJ3lYZiH2hu7+5EIHOw5TeP3nJOcmeXiSk7r7jsVOdWymN5y8\nKbOXGD+V5OeTnJnkXye5f1me9J+oRNcJqKoek+RruvvW6ZyT709yR3f/8LT+BZkd2fqLJO9cxrci\ns3FN/75+KsnF3f2B6TzBZ2d2usNrp6MRJy/TeWpV9fjMjmw9I8lnMnvZ9wOLnerYTZHyuiSfT3Jh\nd390wSMdt+ko0asyOw9qqR/4pneX/qck397d91TV13b30rz0eyITXSeg6RIDb0lyR2Ynxr41swfB\na7v7immbf5Pkm5L8XHffu6hZWX6rX3KvqjMyO3/o7d394mnZtyT5scyOeL0yyeeX8YGxqr4qs9+r\nf7voWY7XdPJ2lvFo3ZdTVY/q7s8seo71MEXkqzN72XTpflZOVKLrBDVdlG53kld091XTM9sXJnlP\nd79x2uarp0sVwDFZdQ7XZZmF/AczO49rf5LXdffrpvVPzuyE503zIA8Pps0UkSeKpf+YCo7Z3sxO\nln9ZVX2iu3+zqu5O8paq+pvu/jXBxfFaEVwvyux8wedndpRrb5IfT/Lmqnpkd//s9O4/YE6Ca/mI\nrhPU9Bbxg1V1T2YXCbwnsxNM70/yvxY6HJtKffHV9J+V5KbMPp/wsUlemuTyqnpjkk94mQTYzETX\nCa67f7eqPpfkl5L8bZIf7+7/u+Cx2ES6+1NVdWmSb8zsavrfO7179p7MPmLmyd396YUOCTCA6CLd\n/e6qet/sW+fTsP66+7NVtfJq+l+X5N1J9gsu4EThRHpgiOkjmF6a2fXhHp/Zhw9/eLFTAYwjuoBh\nlvlq+gDHS3QBAAzgA68BAAYQXQAAA4guAIABRBcAwACiCwBgANEFADCA6AIAGOD/A4peOA0BzxAd\nAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_top_k(alpha, 'pos')"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Fairly uninformative! For most words \n",
"\n",
"* Clearly *any* document will contain these words with high probability\n",
"* Don't **discriminate** between document classes "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Remove such words apriori, using a so-called **stop-word** list"
]
},
{
"cell_type": "code",
"execution_count": 223,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import string\n",
"\n",
"stop_words = set(['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours',\n",
"'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers',\n",
"'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves',\n",
"'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are',\n",
"'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',\n",
"'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until',\n",
"'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into',\n",
"'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down',\n",
"'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here',\n",
"'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\n",
"'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',\n",
"'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', '\\n', 'the'] + list(string.punctuation))"
]
},
{
"cell_type": "code",
"execution_count": 224,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"def filter_dataset(data):\n",
" \"\"\"\n",
" Removes stop words from a dataset of (x,y) pairs.\n",
" \"\"\"\n",
" return [([w for w in x if w not in stop_words],y) for x,y in data]\n",
"\n",
"train_filtered = filter_dataset(train)\n",
"dev_filtered = filter_dataset(dev)\n",
"test_filtered = filter_dataset(test)\n",
"\n",
"theta_filtered = (alpha_filtered, beta_filtered) = train_nb(train_filtered,pseudo_count=0.01)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let us look at $\\balpha$ again:"
]
},
{
"cell_type": "code",
"execution_count": 225,
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFVCAYAAACjA+hQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXVV5+PHvCyEgqAQkKAIhKEHEKxgBr6BcJNyCFjWo\ngAjEYCIVUYla8YripYoXJMWKAlUpKpYUsEAp1apFARUUBY2IBcSCVrH+tCL4/v5Y6zjH42TmzJk9\ns2cm38/zzDNnn31Za5+zzt7vuuy9IzORJElSO9ZrOwOSJEnrMoMxSZKkFhmMSZIktchgTJIkqUUG\nY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUolltZ2Astthii5w/f37b2ZAkSRrVdddd9/PM\nnDvactMqGJs/fz7XXntt29mQJEkaVUT8pJ/l7KaUJElqkcGYJElSiwzGJEmSWmQwJkmS1KK+grGI\n2D8ibo6INRGxcpj5EREfqvNviIhdu+adHRF3RcR3e9Z5b0TcVJf/QkTMGf/uSJIkTS+jBmMRsT5w\nBrAI2Bk4PCJ27llsEbCg/i0Fzuya90lg/2E2fQXw2Mx8PPAD4PVjzbwkSdJ010/L2G7Amsy8JTPv\nBc4HFvcssxg4N4urgTkRsRVAZn4Z+J/ejWbm5Zl5X528Gthm0J2QJEmarvoJxrYGbuuavr2+N9Zl\nRvIy4ItjWF6SJGlGaH0Af0S8EbgP+NRa5i+NiGsj4tq77757cjMnSZI0wfoJxu4Atu2a3qa+N9Zl\n/kJEvBQ4CHhxZuZwy2TmWZm5MDMXzp076hMFJEmSppV+grFrgAURsX1EzAaWAKt7llkNHFmvqtwD\nuCcz7xxpoxGxP/A64JDM/O0AeZckSZr2Rn02ZWbeFxErgMuA9YGzM/PGiFhW568CLgUOANYAvwWO\n7qwfEZ8B9gK2iIjbgTdn5seBjwAbAldEBMDVmbmswX0byPyVlzS+zVtPO7DxbUqSpJmhrweFZ+al\nlICr+71VXa8TWL6WdQ9fy/s79J9NSZKkman1AfySJEnrMoMxSZKkFhmMSZIktchgTJIkqUUGY5Ik\nSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMk\nSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgk\nSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYk\nSZJaZDAmSZLUIoMxSZKkFhmMSZIktaivYCwi9o+ImyNiTUSsHGZ+RMSH6vwbImLXrnlnR8RdEfHd\nnnU2j4grIuKH9f9m498dSZKk6WXUYCwi1gfOABYBOwOHR8TOPYstAhbUv6XAmV3zPgnsP8ymVwJX\nZuYC4Mo6LUmStE7pp2VsN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjYCiAzvwz8zzDbXQycU1+fAxw6\nyA5IkiRNZ/0EY1sDt3VN317fG+syvR6amXfW1z8DHtpHXiRJkmaUKTGAPzMTyOHmRcTSiLg2Iq69\n++67JzlnkiRJE6ufYOwOYNuu6W3qe2Ndptd/d7oy6/+7hlsoM8/KzIWZuXDu3Ll9ZFeSJGn66CcY\nuwZYEBHbR8RsYAmwumeZ1cCR9arKPYB7urog12Y1cFR9fRRw0RjyLUmSNCOMGoxl5n3ACuAy4PvA\nBZl5Y0Qsi4hldbFLgVuANcDHgFd01o+IzwD/CTwqIm6PiGPqrNOAfSPih8A+dVqSJGmdMqufhTLz\nUkrA1f3eqq7XCSxfy7qHr+X9XwB7951TSZKkGWhKDOCXJElaVxmMSZIktchgTJIkqUUGY5IkSS0y\nGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWrR\nrLYzsK6av/KSRrd362kHNro9SZI0OWwZkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJ\nkiS1yGBMkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBM\nkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQW9RWMRcT+EXFzRKyJiJXD\nzI+I+FCdf0NE7DrauhHxxIi4OiK+HRHXRsRuzeySJEnS9DFqMBYR6wNnAIuAnYHDI2LnnsUWAQvq\n31LgzD7WfQ/w1sx8InBKnZYkSVqn9NMythuwJjNvycx7gfOBxT3LLAbOzeJqYE5EbDXKugk8uL7e\nFPjpOPdFkiRp2pnVxzJbA7d1Td8O7N7HMluPsu6rgMsi4n2UoPCpwyUeEUsprW3Mmzevj+xKkiRN\nH20O4D8eODEztwVOBD4+3EKZeVZmLszMhXPnzp3UDEqSJE20foKxO4Btu6a3qe/1s8xI6x4FXFhf\nf5bSpSlJkrRO6ScYuwZYEBHbR8RsYAmwumeZ1cCR9arKPYB7MvPOUdb9KbBnff1s4Ifj3BdJkqRp\nZ9QxY5l5X0SsAC4D1gfOzswbI2JZnb8KuBQ4AFgD/BY4eqR166aPAz4YEbOA/6OOC5MkSVqX9DOA\nn8y8lBJwdb+3qut1Asv7Xbe+/xXgSWPJrCRJ0kzTVzCm6Wv+yksa3d6tpx3Y6PYkSVrX+TgkSZKk\nFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIk\ntchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIk\nqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5Ik\nSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFvUVjEXE/hFxc0SsiYiVw8yPiPhQnX9D\nROzaz7oR8cqIuCkiboyI94x/dyRJkqaXWaMtEBHrA2cA+wK3A9dExOrM/F7XYouABfVvd+BMYPeR\n1o2IZwGLgSdk5u8jYssmd0ySJGk66KdlbDdgTWbekpn3AudTgqhui4Fzs7gamBMRW42y7vHAaZn5\ne4DMvKuB/ZEkSZpW+gnGtgZu65q+vb7XzzIjrbsj8IyI+HpEfCkinjyWjEuSJM0Eo3ZTTnDamwN7\nAE8GLoiIR2Rmdi8UEUuBpQDz5s2b9ExKkiRNpH5axu4Atu2a3qa+188yI617O3Bh7dr8BvBHYIve\nxDPzrMxcmJkL586d20d2JUmSpo9+grFrgAURsX1EzAaWAKt7llkNHFmvqtwDuCcz7xxl3X8CngUQ\nETsCs4Gfj3uPJEmSppFRuykz876IWAFcBqwPnJ2ZN0bEsjp/FXApcACwBvgtcPRI69ZNnw2cHRHf\nBe4FjurtopQkSZrp+hozlpmXUgKu7vdWdb1OYHm/69b37wVeMpbMSpIkzTTegV+SJKlFBmOSJEkt\nMhiTJElqkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFfd2BXxrJ/JWXNL7N\nW087cMLTGS4NSZImmy1jkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1\nyGBMkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSp\nRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSpRQZjkiRJ\nLeorGIuI/SPi5ohYExErh5kfEfGhOv+GiNh1DOueFBEZEVuMb1ckSZKmn1GDsYhYHzgDWATsDBwe\nETv3LLYIWFD/lgJn9rNuRGwL7Af817j3RJIkaRrqp2VsN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjY\nqo91PwC8Dsjx7ogkSdJ0NKuPZbYGbuuavh3YvY9lth5p3YhYDNyRmddHxBizLU2M+SsvaXybt552\nYOPblCTNHP0EY42LiI2BN1C6KEdbdiml65N58+ZNcM4kSZImVz/dlHcA23ZNb1Pf62eZtb3/SGB7\n4PqIuLW+/82IeFhv4pl5VmYuzMyFc+fO7SO7kiRJ00c/wdg1wIKI2D4iZgNLgNU9y6wGjqxXVe4B\n3JOZd65t3cz8TmZumZnzM3M+pfty18z8WVM7JkmSNB2M2k2ZmfdFxArgMmB94OzMvDEiltX5q4BL\ngQOANcBvgaNHWndC9kSSJGka6mvMWGZeSgm4ut9b1fU6geX9rjvMMvP7yYckSdJM4x34JUmSWmQw\nJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqUSvPppTWdT6QXJLUYcuYJElSiwzG\nJEmSWmQ3pTSDNd0daleoJDXPljFJkqQWGYxJkiS1yGBMkiSpRY4ZkzQuk3WbDse/SZqpbBmTJElq\nkS1jktTFFjhJk81gTJImmU9gkNTNbkpJkqQWGYxJkiS1yGBMkiSpRQZjkiRJLTIYkyRJapFXU0rS\nDDUZt+nwylBp/GwZkyRJapEtY5KkKc+b8Woms2VMkiSpRbaMSZJU2QKnNhiMSZI0ibzoQb0MxiRJ\nmoFm0tW0Mz2AdcyYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLWor2AsIvaPiJsjYk1ErBxm\nfkTEh+r8GyJi19HWjYj3RsRNdfkvRMScZnZJkiRp+hg1GIuI9YEzgEXAzsDhEbFzz2KLgAX1bylw\nZh/rXgE8NjMfD/wAeP2490aSJGma6adlbDdgTWbekpn3AucDi3uWWQycm8XVwJyI2GqkdTPz8sy8\nr65/NbBNA/sjSZI0rfQTjG0N3NY1fXt9r59l+lkX4GXAF4dLPCKWRsS1EXHt3Xff3Ud2JUmSpo/W\nB/BHxBuB+4BPDTc/M8/KzIWZuXDu3LmTmzlJkqQJ1s/jkO4Atu2a3qa+188yG4y0bkS8FDgI2Dsz\ns+9cS5IkzRD9tIxdAyyIiO0jYjawBFjds8xq4Mh6VeUewD2ZeedI60bE/sDrgEMy87cN7Y8kSdK0\nMmrLWGbeFxErgMuA9YGzM/PGiFhW568CLgUOANYAvwWOHmnduumPABsCV0QEwNWZuazJnZMkSZrq\n+ummJDMvpQRc3e+t6nqdwPJ+163v7zCmnEqSJM1ArQ/glyRJWpcZjEmSJLXIYEySJKlFBmOSJEkt\nMhiTJElqkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElq\nkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqkcGYJElS\niwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqkcGYJElSiwzGJEmS\nWmQwJkmS1CKDMUmSpBYZjEmSJLWor2AsIvaPiJsjYk1ErBxmfkTEh+r8GyJi19HWjYjNI+KKiPhh\n/b9ZM7skSZI0fYwajEXE+sAZwCJgZ+DwiNi5Z7FFwIL6txQ4s491VwJXZuYC4Mo6LUmStE7pp2Vs\nN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjYapR1FwPn1NfnAIeOc18kSZKmnX6Csa2B27qmb6/v9bPM\nSOs+NDPvrK9/Bjy0zzxLkiTNGJGZIy8QcRiwf2YeW6ePAHbPzBVdy1wMnJaZX6nTVwInA/PXtm5E\n/Coz53Rt45eZ+RfjxiJiKaXrE+BRwM2D7uwE2AL4+QxIY7LSmUn7MlnpzKR9max03Jd1Ox33Zd1O\nZ7L2pV/bZebc0Raa1ceG7gC27Zrepr7XzzIbjLDuf0fEVpl5Z+3SvGu4xDPzLOCsPvI56SLi2sxc\nON3TmKx0ZtK+TFY6M2lfJisd92XdTsd9WbfTmax9aVo/3ZTXAAsiYvuImA0sAVb3LLMaOLJeVbkH\ncE/tghxp3dXAUfX1UcBF49wXSZKkaWfUlrHMvC8iVgCXAesDZ2fmjRGxrM5fBVwKHACsAX4LHD3S\nunXTpwEXRMQxwE+AFzS6Z5IkSdNAP92UZOallICr+71VXa8TWN7vuvX9XwB7jyWzU9BkdJ9OVhet\n+zI105lJ+zJZ6bgv63Y67su6nc6UHNY0mlEH8EuSJGni+DgkSZKkFhmMSZrxIiKm03YlrVsMxqRp\nJCJm1f/+dsdmY5iQz+0RDW9Pwxhv0GvQrKnOA7pmnKlw4J2IPETEQ4BLIuIRmfnHpgOLif7c6q1v\nNqqvHzCRafWkOx/4akQ8vsnPLSIeCJwXEe9uYnttq88S7rzu6+KuCcxLdL3eKMc/uPnh41x/2hj0\nd1zLc+f1Q5rL0dQREdu1nYe1MRgbUFsn/LWl21R+ImJuRMwZfcmBtv0XeZyIgKJz4J7ME35Nb5eI\n2DwiNs7MnIBWmF9S7t13TkRs03Bg0f25rRcRG3ReN7V9YC9gYUS8GHh/RGzYxLZHSzczbwX+ATgr\nIh7VxOcWEetl5m+AlwBPj4iTG8hua+r3fVxEzIuIJwCndgdnk62rLK4APjqe76sGFmdExOObyt9U\nqPANp+d3PLffz61Wkg6OiEMj4kWUsjChv8/J/gxrOXh9RCyYzHT71WrtZzrrKvB7ApvUW3hMqJ4f\n2ouA3wMbZ+Z5DdQciYgTgOcAv4yI2zLz9ePdZte2u/O+P/BH4KeZ+d0JSuPVwC4RcWxm/r6pNEZI\n+xXA64F/BX4WEe/PzLvrSfuPDWw/ahDxGeCJlBaZIzLz9ibS6PrcTgAeCzwoIs7KzKvGm/fO9uuJ\n4b3Aw4ATJ/p76S4PwLXAsyiB7Esz86bxfG5d6z0BuAE4PiLWz8x3jj/nw6snk19n5h9qQLsZ8IPM\nvLxnX8esbvMW4Fbgx5TH2N3fSMYHFBH7AocAL2zgN3Q7MKdut8nfy2HATsD3getq4N+arnydBOwA\n/A3wiz5W/QNlHy4ANgQel5m/r2W68XLQc6x+Xn37nsy8cqLSAQLYElgI/LDJdJpgy9gY9TSfrwA+\nBLw9Is6OiG3Xvub49dQWXwH8DlgVEYeOd9sRsQRYTKnp/z/gcePdZreuvL8CeDOwI3B9ROw8AWks\nB54HnFIPKBNdwzsE2Bp4OvBxIIE3RcQWNYAadw2wBjMHA+cBXwZ+DXwuIrZrqoUsIo6jnPzeSjlo\nvXS826zbDYB6oP0e5WT/h4jYsontr01XeXgV8G7g8zX9T0fEY8b7uUXEkXW75wBvARZFxFvGm++1\npPUI4J3AUyPiKOAUyji4z0XEi2r5GLR7KmqAcjmlbG1JuUl3p8Vs0tWycRCwM7B5J59j3MZ8+NM9\nLa8BPtz5TTaUx5dTjmW/oTxF5tiIeHIT2x6Pmq/FwFsy8xcRsXGUJ+D8xWfY9du8nxK0/Qq4CTi4\n6/3Gdf02jwdeA8wGroiIpzWx/YiY3QnEamvvFpn5c+AjwLKI2KGJdJpkMDYGPdH8bOBBwB6Z+SRK\nYVoZEdtMQLrbRu1yi4i5wDMpN8zdCbgK+OcGAo7fUR7ufiTlAe/PrentMs7tUrcTEfFoypMa9gXu\nB74E3NQ5IIzjZPKonpPgPOC1wENq8HdFRLwwIjZsIjDqSne9iNgMOB94Smb+BPhP4AuUA/R76kGg\nqZv5HQC8KzPfQ3m02GXAJyNi24ZOMJsAR1CehvF74Jh6UBs4aOo6IO4UEfOAN1JaEF8C7FeXeURE\nNDamJyJ2jNJi3fEY4FWZeTZwHCUoOysiHj3Oz+0BwLsz82pKkPxq4PkRcco4tjmszLyF8vDjw4Gn\nAi+o5eBQSoVsySDlrOuY9pD629gPeCHw9YjYs7aY7RQRmzS3N8Pno+v17My8C3g/5bF5K2qlo++A\nM0pX67si4vKI2BW4HDibUhbG3f0epQv3qcBLM/N04HV1ViPBxBjz0v3ZzaK0AH4SeHiUHoJPUyqH\nD+wuIz3nsxWUCuV+lADz+fXYSUQsjIjtm8xrPXZuV9NbBGwK/BtwdQPfzeaUSvHCKM+9Xg78S0Q8\nE/hvyk3o/9RKOp60mjRlMjLV1Zpjp+C+BriEErgcWBc5Dngw8M6I2LrBdLemnHhn1R/aLymPnHo3\npSXmBbX2cnREPGkcST2I0sW2b2Y+px6Ej6WckMc99qp+dj8FvkY5IS8GDqgnw+OiPDR+oJMJpYVg\nbkS8vb79fzWNd1Ga379ICWQYJI0RzMrMX1K6qh4fESdl5v2ZeQ1wMeUxX42Mu6kHjdmULkSAeylB\n3xaUx4ptNJZAcy3Lbk0JkHfLzAMy8z7gGErZGmg/6gl0EfBZSuvBv1Nap/4J2C8i3gt8A2ikEhNl\n7MuLgefVgy+Usr245ud+4F8ogedHOjXoPra7tmVeXYOH+4FvA98CnhMNDYCulZhO68UbKV1JjwGe\nUU+u/0YJyD4dpctsTNvuanH9NPDPlFaKayjf1aX1JH0BMKHjbHqGF5wRERcCD6S0Ov4/SkD2iJF+\nv10n+t2AVZTg+CpKxfKzlLJ8ZE1v4CC8bn9jSnD8kigXGNxEOX4eNNGBa6+eIQZnAXdRWrZOp5wr\nLqJ8lmtb70RKK/ivM/NXwDeBMyktvRdSguL/azKvlGPZncB3Ka28BwMH19/R8oh41CDbj4gNM/N/\nKN3Sr6MMiXgL8EHgMEqZeC3wN1G6YBtpJW1EZvo3hj9KzediSuvUGyg1t/3qvI2BjwEPazjNBwGP\nB1bU6XdQWpY2rtMvAq4Dthnjdo+mFNRn1Om3U04mjwb+mjIW5jED5nmzrtdLgFMpweoVwE965n0N\nePgAaczuer0bpQbUearEE4FN6+v9KEHGQxr8Tl5GOeCfAGxPOVn9HHh11zIbjWP7nf14ArArpbVv\nHiWQOb7OeyrlQLnLONJ5CSV42ZkSoHwN+EiddwxwI/CocWx/PiXYemQtp98F5tR5e9QyuHdD30nn\nM1sAvAl4HyVw2baW5ZV1/osoz8btq8x1tltfHwYcCzyhTr8P+DplfM7RwGeAuU3uT319HHBsfX0y\n8FFgH+AB9b1n9Ps9Aet1vd6R0i21C6XC8hrgo3XeoZRunf2a2J8+8nVEPT7MAn4AvK++vyvlZHoq\npQI00jb2oQSWB3Tvb/2tnFnL98HjyOPmwIeB/etn/rfA0jrvEEolY5PJ+Lx68rVv/ew2qtMPBx7Y\n9T1eC2wxTLnanHI+25ZyfD4MWFaPBzsB7wEe3UD+tux6fSjwd/V7+TTw8655L6jHi20HSOMhlDGp\nT63Tp1AC0V3q9KaU49Cq+j3t3ft5tPnXegam+l8tkH9VXz+ZEvS8rU5vVg+SFwEHTWAeNqacMD9O\nOXnOrgfJ6+pB+RrgsWPc5uJa6D9Mab4/AdiIMuDz7yknlYF+hJQT8N93FfajgJfX11tQBk9+jFKL\nu5YyWHSsaTyY0mWzBeWigxfRFZzVZWbVA8sNg6QxQtpLKWNrnklpDTmlvv9oyoUJr2wonQMpwfG5\n9UD7KspJ/4fAJyi1vwPGuM3uA/GhlODu72rZemE9oF1Oufrw34Cdx5H/rSiB6rKug+wOdd5zqCeL\n4fI2zs/tQfXvzdRgtR6Eb6jleg0DVDLq5/8fdbtX1f3amFKJuYAS8D+hqXLWle4JlIDvcV3v/Q1w\nBiWAesAYtrU1pZt4yzq9ELi4a/6jKd3ue9XpDZven6601h/m831a3d8vdtKmnLQX0HVCH2GbL66/\nwc7xZhY1+Kxl4iTgmAHz++j6/1jgzPr6ebVMXVnLd+Pffx/52pISrN4OLOh6fxPK+eL7nfLe8/t/\nCaVF7B8px+LPUo7LX6JW/BvK3zaU88EL6vQhwBs7+aEM7fg8pWv1Gwx4rAY2AD5AOTfuVt97I6XF\nd5dOeaMML3gb8NrJ/q5GzH/bGZjqf/UAcSZwYJ1+dz1QPKJObwq8shboTWgwyq4FdU/KSTHqD/+j\nwFF1/tMpNb75Y9zuYuBqYKs6/TxKk/YKakDDKDXQEbbdORGupNRAnkLpGjiua5lNa5rP73yOY0xj\nVv1/BKVW/72ufEfPskdRA4AGv5d3UAKNoxiqyXfS3xHYsYE0HlQPik+v0ztQAsDnUioBOzHGgKLn\nQLw1cCKwU51+QT0QP7c7D4OU2fr/8ZQg77XAj4D/6vre9qAEM49s8nup234cpda7Wy1nb6a0XuxE\nOVhvTh8n9bqt7hakJ9Xf+HqU1qNv1v17edc+D9wSOkIeHkg52c/rTYNyocUHqS3kfW5vK0pA+TbK\nuJlN6nfR/fs8HVjeW2Ya3q9NgWfX13tTTpavqnn5PLBBnfcm4NQ+ytuWXb/Bwyjdak+r07O6ljuV\n0hI0ayz7RmkF+wHlIorZlApLpxK2HqUlaYuJ+KzWts/1dWeft6OcGz4AbFffewClQvcXx1hKC+K/\n19dPowRlO9bp4ygVwNlNfP+UrsLl9fdyIOX8c0LPMvvWPM0fMI1OwD27fser+POA7EJKxaMTkJ1E\nDfgnqoyPeR/azsBU/aOcMHaitMC8shb0feu804FPUU8mdZk5DaUb3f/r6/OBI+vrIylXcB5PT8tC\nn9vfGNidctXM67veP5RSe1lRDy5jKqCUYHE+cH6dfhjlpPXh+kM4h9Lltg/l4LvZWPNetzsXuKi+\n3ocyIPMyatcwXSfQCSgTe1ICoROBm4HLuuYtB17UYFqbUMYlzu9673DgtAG31x1YnEQZ3/JThloQ\n5lACsvMZCvYHOkhRxn98iRLw/wOlq+OuWh5WUFr7Fjf0OfUG34+ktKycWw++m1JO6GdRuy8GSGNr\nShC3A6U19EuUk/lbKN2uywf5zfSZ9gaUAOXYnvc73aSbj2FbnWB4C0rA+mbgoZQAfxXluPbMWraf\n0vS+9ORlPmX8zr8C36rv7VTLxjLKkw1eDFzPKK2ztbz9C+Xk+ob6XR1KGTawZ3f5p3TxPn6Mee0M\nil/N0FjFJ4WMAAAS50lEQVSk44GvUE/4bfzVz+9jlOPrYyjH9XdQuuo656bO+WQPYGF9/bR6bDlj\nmG0eXcv0wC3iXdvqPoc9jNKj8BFKS9XllOPpiykV8/njTYfaPVx/M2+hBH+dgOxtdZ8fXKePGWs5\nmPDvs+0MTMU/Ss3q7npA2LweaFdQugU6Adnf1h/n9hOUh4d1vf4r6viJOr2UMuZlTAFgPch9klKj\nfhmlVenorvkHAQ8dML/dB/pnUVoSHkBpGfk65XYGK4HP1R/FmFvEutLamHID0Y0Zapm8iNpVS+lq\nabR7pZaDj9YDyBMotetX1nkvAr7D+MZWdQ4oD2eoe+aU+tl1xoEsoXSHDVxjpXRr/XP93JbWMtBp\nfdusHhgHHvNIObl/haEWtxW1vL2d0sL8GobGWDbZiry46/X8WtY/RbngYXPKSbjfFrGnAkvq61dS\nuqI/QRkf9hbgDXXeyyhBXl/bHeP+vBA4uat8vQPYp04fTgnQxjwGspbfgyityl+hBBU71jJ9HqXy\ndEjT+9OVfnel4GTKVbsf6nrv6ZQA/px6nBhx+AV/PubtwFq+Ol2Ix1AG/8+hp0t0lG3OpgYjlPGm\nr6Yczx5af5MnUm4t9EdKT8mEVQBHyGNf4+soleTO+Wy7ug8bUQK2C6kBWl13h1qemw7EOsevDSkt\nyVfVPK+gDI84j67u1UHSoQx7OJdSMdqZcs4+hXLMfkpdpvGW+Ea/07YzMNX+KMHEJZTujs3rgeEg\nylVxnRayzliodzLAwPNRCtV6lGDiznpgWUSJ9K8HXtK1/KZj3P5fUWo8T6S0rL2aoZP9SQ18ZrdS\na+mUmuNNlK6qDerBa1X3D7+Bz+u5lK6vTet384b6vb2VMvah7xaDMaT5TuAT9fXzatm4EvgqYxyz\nt5bt708ZP3EuJZCYV7+j79eycBPlZpxj2eZTGLrwYz4lmLuya/7xdfudLqNxBUiUgO5rDHURbUBp\ncb2IOvayoe/iT7+X+v8bwFe75u9Q9/XyWg7H0iV1IKXy8HbKAONHUlpzT6Z08f2REiz8kBp0TkBZ\n243S/X4cpVJ4Ui0b/0jXGKAxflZPrev+A+Vk/KX6ub2K2iXNUDfORLTydZ+gn0HpUTiUEuC+kxpc\nUgKsoI+Wf8qxcrgxb8+q04MMBN+hlpvzKK3unStK/47Sndf5rTyHcVTAxpingcbXMfz5bBElMHp/\nLQe7dm237/GHfeb7BEo3+8cZaqVcSjkfjOlYNkIa+1IuNHp2/X1+tr63Xi1Xf88Yz5dt/LWegan2\nRxmr8y/1B/0ZSg3o7yknyvXr9LnUAa4NpTlcLaJzddaXKYN1V9Y8DRRkUIKV19TXsyktBx9gaPzO\nnPEcgCldBTdRux/rAezblBr3AyhBxQcprVlNDdTen3JC7Fw1ubQeeAa6AnQtaSyk62qyWjae3/ne\nKM3vD24gnUdQaovPoJyMXkcJ8h5IGVt3WOdgNsbtzqOME5pfp/ejtIyd1LXMiZSLQR7QxHdDCRze\nxFBL5XMoLYnnNHGw589bV+Z0vf4CdRxMnX4L5WQzyJW6+1IqL5+q0xtSTvSnUgKIo2h4LGJN5zEM\nXfX2pPobOq6WtW1qedxqgO3uVn/ne9TpHShXTH+CcgJ7NwOOEx0gL8fXsv7wOr03pafhrZQK7+kM\nM/6ufgbPpwQVe9U8P5ByjDy2a7nTGaqADBRcUlpBf83QFawPq2X4B5ShEWP+DsbxeQ08vo7hz2cf\np16JW38fZzKOK7JHyPdyyq1stqd0i99KbXWtZe/DNX+DtvIHJaB/N6U1bB9Ko8VbKcHzXpSArPHf\n6YR8z21nYCr+UbrWfsNQd8SxlCsOn0MJyI6j4dtX1HROoHR9/jtweH3vQZSWrH+iNDUPNDaNcgK5\niK4m6Ppj3mG4A9+AaSyiDNbuBGQnUK6W3JVyMpuI1qoDKLX9TqtcY92TNd+vo7RQvJZyMj6aemuJ\nBtPZilJz7dxSoNPacwbw4gG3uSVDwVBn/EznKuBFlFp+9204GhnzWLe1DaVV6YuU4OVmSjCwmgav\nNqME35+l1LIPru99gdLi8wZK1/G8cWx/MeW+fi/sem81DV45zZ8Hlo+knBhfyVAr0ZOBe4C/GWc6\nnRstd45pG1CuansXZUzcruPZ/hjysUs9Jmzb8/6elK7YbzFCKzPlCsBfUCp+ncDyubU8f4CGxrzV\n4+IRNT9Hdr1/PGWox8DDLAbIy3zGMb6O4c9nn6AEdhvXz33cXe09ZXnDmreHUCpnF1K63m9h6GK4\nQc9lveNEH0AJlq+ktMzPorSUfYxJuqiike+57QxMxT9K3/o+9QffuXz9ZfXAv88Epfk8SpfhjpQW\nnx8Br6jz1qfUAAbuEqW0fL2DcnLsPPPtmqYLK38ZkHW6dibyEvnFlBaE9Who/AalVncB5TYAp1Ja\ndU6jtJbcDuw+zu13X3W4ihL03cGfj+F7GwN2IVO6KjpdLRdSxuJcxdAVYM+hdIWu6M5Pg9/Jg2s5\nPpkSaO5OubXEQGMSh9l+pyX2SfWzO52h+z2dTFfL3DjTOaieQN5CCWq/w8RcBXpQ/V2+r+7PMur9\nyihB+dWMs6ul/k7WMFTR25NyVWhjgXgfeXgSQxf5BEOtOp3/w7YyM1RBeTgl4P4xQy1rD6G0wJ9L\nGUbS2Ji3Ws6ur5/dMymVjDFfODVg2o2Mr2Pt57PP0WAPT08524/Syr8hpXGhU5avrL+hge7FxtBx\n89mU4+MRlEry5pRK+VaUSs3lNDD2bTL/Ws/AVP6jtIz8kDIQtPOomEaap+kJGii1hnd1TT8ZuI0G\na6z1QLaiFtQLmaB74lACspsYaq0a6MrJMabZ2AGSctL9D0ot66uUIPaBlNamUyhdegMNOO1Jp/uq\nw/MoAcVtlFad51ECzL3Gsf1OV0vnQoPH1P15U53elwlo4R0mH8+q+9hIeWNorEzn5rcbU4KZTzN0\nI+QmLw44lNKqdBENtYjw50MTllCuNn0rZQzXJyitGe+ijBW8gHG08A1T5n5dT8QXMLGD9bv3sTOm\nadP6+zm+a97Lgb8d7Xur38O36sn2lfXk+8Q6b5u1pd3AfuxPqUh8kwZugDrAZ9fU+LoJOZ8NU5Z/\nRqm4XkPpSXgPpdJ5LGWc2rgqZJQxnd+ktJJeztDFC2+l9GJ8j65b9EyXv9YzMNX/KDWuX1Av95+A\n7R9KqUW8k9Id+adxO5RaXuPRfT15Tehdoim1o28xQZf8T3Dej6IEQy+tP/bOOL759f+Y7781TBq9\nVx0up9T03kEJKk5lHHcKr9vsdLV8k9rdSbk1yK003NU6Sj62ot77aMD1uw/2D66/i09RWpIf3zXv\nciZg7Evd9p6M4/L7EfZnHqUi1rkVwWH1xP/eeiK7kgZa+HrSP4QS6L+2k5+mf6M9+3hsPbYdR2nB\neDqlC3sVZbziqDd+plx4dD1dA+ZrGfjPuv2fTtR3X9OaS0NPVhhjugONrxthe42ez0Ypyy+qZfn7\n9dj2I8Z5OwnKkwJOoQyF2JcS8G3bNX9HakW56TI90X+z0Igy8/r60OFGns3V83DWJZRxDh+j/Mge\nRXlg95ejPHB8L0rtuFGZ+dumtzlMGhdFxJU5lZ791b9bKWMEf5qZz4A/Pfdtu4g4OTP/t4E07qUE\nqp1nGJ5FGS/0EMqg8c/VdP9UXsYqM9cAayLiV8CpEXEP8L+Uqx3/dZz5H0s+7hzn+p3fy86Z+b2I\nuI4yKPgbwIkR8RmGbnMyrrRGyMOXmthOz+//BMpYnwcB74+IOzLzc/UZix+gBE3nZXlGaGMyc3VE\n/B9wdkT8KDMvbHL7NY3OPi6jVAhOpgwiX0BpkXs5ZTD5hpQxWd8bZZO/p1Tu9oqIF1Jai35KGdO3\nHeWB3d9qej86MvPuidr22kTELpRWrL0z86c1H1dGxH2UQORlwBGZ2fe5qcnzWR9l+dMR8RtKN/u3\ngPcOcuzseobq07u2dQ6l7PxVZt4WEQdSLkC5qLPeoMfNthiM9SEzv9vEdnoK7zwgKfd4+lFEXENp\nFdmLMnZoH+CwzLytibTbkJm/aTsPA7qO0iX1x4jYi1LjO4pSm2zkxJiZv4yIzwPPjoh7MvO7EfFZ\nSvfbwRFxSWb+rokDSmb+c0T8gdJdcC/lAP7D8W53MkXEU4DzI+KdlCdSHEXpcr2Y0qX3a8qYsZ+1\nl8vRdf3+D6VcGXkEpWXnccAeEfGVzPxsfTD7r5oOxLrycXlEHE1prWhMzzFua8pwi4Mp3Vc/obSM\nLaMMyVg5hk3fRmlBO4rS/f45SkD2c+AL07TSN5r1gDU12AhKsPEH4GuZ+aWIeE9m/nqsG23qfNZn\nWV4dERsB1w9aia2B2O6ULtpllPGCF1Mqrf9Vg7TTKS2v01fbTXPryh9/3pzbedbc9yiFt9MNdijl\noPMUJukyc//W+n1tRam9X0IZGNzYsy270piUqw670tuSFrpaGsj37PpZfY3S5bqY0o13A6UrNpjA\nC0QmYH+2ptwj7+N1eqNaDj5MafGYlr/9nmPcCkrFcg7lCsor6vs7Um4N8ToGuM0JQ4//eTLlYppG\nHjLf9l/PZzfu8XWTmO9JKcuUoTz3M3R7piX1ePB5ygUBE/Zs6Mn6W2+tUZoalZ1fz5/XIi5mqBYx\nKzP/iVLL/++coBqx+pOZd2bm31EC5GMy8zsTkMbtlKDig5SWncMYeqxU4y08mXlXttDdMh61ReyN\nlBPTSyhXsm5GGYj8WMq4lPUy8/etZXKMMvMOyn2iFkXE4Vm6md4K/IFylevsNvM3qJ5j3LOBWzLz\nV5TupIfW1r5HUsZ5nZeZvxsgmfsj4kmUx+q8MTOvbCb37elpTTwWeG9EHEe5iv6vgUMiYlVEnEhp\n/fkkTI1uuMkqy5l5OWUc79KIWJyZ51Nua/Rm4NDMvLi2Hk5bMQW+z3VGbbb/T0ot8ZjafPtGhp57\ndpVB2LorIp5FGSP48sy8vu38TAV17OT+lO6Jj1Luj3V3Zl4YEcdQbvLaaFfbZKnjXN5F6bL7TETM\nolx5PK0C5m4RsRXlGPf1zHxhDcCS0o20G+UqviWZeeM40tiEcl+sH49nTOVUM8z4uk9TxtfdRRlf\n9ytgdY4+vm7STVZZrum8jXKLj3Oa3HbbDMYmWUQ8j1KrO6mr0L6H8oiVU3ISBtdraqonstmZ+ZO2\n8zLVRMQTKAf7B1G6WndqOUuNiIhFlIs3Xp2Zn207P2M1XDAUEQdRnlqyMjM/Wd/bgHJ7g5/V1pR1\n3jDj695G6RlZQhkM//266LumQ4VjsspyRCymHAv2ppSnGRHEGIy1YCbWiKWJFhFbUg7Af01pXbm1\n3Rw1IyL2BX6Umbe0nZex6Akmnku5Lc/3MvPbdZ/eC5zeCcg0pOezW0EZ//ZtylXC78nMfSNiR8r9\nDv8W+PCA3bqTarLKckTMnWnnS6+mbEFmXhIRfwTOioj7ai1iRhUsqWmZeRfwmYj4XJarymaEzLyi\n7TwMoiuYWA4cSRnL9B8RsaQe414NfCIi7s3MT7eY1SlnmPF1qzPzVxHR5Pi6STdZZXmmBWJgMNaa\nzPxiRLyMhi8tl2a6mRSITXcR8WTKjT73oQRkPwE+HBGbZOYFEXEk5Qpx9ajDEk6njK/7rxqAfYPy\n+KCvMjS+bkLunaepxW5KSVJf1jJGbCvgqcDyzHx2bRF7H7BfZk7azYWnOsfXaSS2jEmSRhURG9Vb\nF3TuDr9eZl6XmXfW8Xyd27/cTnmY95qWsjrljDC+7uKIOIJyOwsy85O15fe6NvOryWcwJkkaUUR0\n7of4D5TH8Pw1cGdE/DIzD6U8N3LPiPhHYCdg8Uy5wKIJjq/TaAzGJEmj2Q5YRHkG6FOA3eqA869H\nxPmZuSQi/pfyQPU3G4j9JcfXaSSOGZMkDSsi1sv63MeIOAl4DOUROMuzPIieiLga+GVmLmovp1OP\n4+s0Fj4OSZI0rK5AbBmwK/CvlEd3PSMitq3L7AHM7kzrT+PrOl2Tu9RHOFGvjHR8nf6C3ZSSpLWK\niEOA5cCB9RYMv6Z0t0VEXJWZP87MvdvN5dTh+DoNwmBMkjSShwOfqYHYrHoF4P2UQON3EXEbcP9M\neSxNAxxfpzGzm1KSNJKfAM+MiEdl5n31vfWAXwBXZeZ9BmJlfB1AZl5MuWnrE4DNgC3q+7sD8yPi\ni5n5ncz8SGbe1FqGNaUYjEmSRvJV4BrgpRFxUES8BHgz5RmKP2s3a1OH4+s0Hl5NKUkaUb0KcDFw\nCHAP8K7MvKHdXE09dXzdqQyNrzuIMr7uKkor4o9bzaCmLMeMSZJGVK8CXBURZ9fpe1vO0lTl+DoN\nxG5KSVJfMvNeA7EROb5OA7GbUpKkBkTEg4HXUnqdvgrMAU4AlmTmLW3mTVObwZgkSQ1xfJ0GYTAm\nSVLDImI2OL5O/TEYkyRJapED+CVJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSpRQZjkiRJLTIY\nkyRJatH/B027mhVP4YPeAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_top_k(alpha_filtered,'neg',k=20)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"See words that \n",
"* indicate positive movie reviews (\"good\", \"like\"), \n",
"* are generally likely (\"movie\", \"story\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"How about negative reviews?"
]
},
{
"cell_type": "code",
"execution_count": 226,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmMAAAFVCAYAAACjA+hQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYXVV5+PHvCyEgqAQkKAIhKEHEKxgBr6BcJNyCFjWo\ngAjEYCIVUYla8YripYoXJMWKAlUpKpYUsEAp1apFARUUBY2IBcSCVrH+tCL4/v5Y6zjH42TmzJk9\ns2cm38/zzDNnn31Za5+zzt7vuuy9IzORJElSO9ZrOwOSJEnrMoMxSZKkFhmMSZIktchgTJIkqUUG\nY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUolltZ2Astthii5w/f37b2ZAkSRrVdddd9/PM\nnDvactMqGJs/fz7XXntt29mQJEkaVUT8pJ/l7KaUJElqkcGYJElSiwzGJEmSWmQwJkmS1KK+grGI\n2D8ibo6INRGxcpj5EREfqvNviIhdu+adHRF3RcR3e9Z5b0TcVJf/QkTMGf/uSJIkTS+jBmMRsT5w\nBrAI2Bk4PCJ27llsEbCg/i0Fzuya90lg/2E2fQXw2Mx8PPAD4PVjzbwkSdJ010/L2G7Amsy8JTPv\nBc4HFvcssxg4N4urgTkRsRVAZn4Z+J/ejWbm5Zl5X528Gthm0J2QJEmarvoJxrYGbuuavr2+N9Zl\nRvIy4ItjWF6SJGlGaH0Af0S8EbgP+NRa5i+NiGsj4tq77757cjMnSZI0wfoJxu4Atu2a3qa+N9Zl\n/kJEvBQ4CHhxZuZwy2TmWZm5MDMXzp076hMFJEmSppV+grFrgAURsX1EzAaWAKt7llkNHFmvqtwD\nuCcz7xxpoxGxP/A64JDM/O0AeZckSZr2Rn02ZWbeFxErgMuA9YGzM/PGiFhW568CLgUOANYAvwWO\n7qwfEZ8B9gK2iIjbgTdn5seBjwAbAldEBMDVmbmswX0byPyVlzS+zVtPO7DxbUqSpJmhrweFZ+al\nlICr+71VXa8TWL6WdQ9fy/s79J9NSZKkman1AfySJEnrMoMxSZKkFhmMSZIktchgTJIkqUUGY5Ik\nSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMk\nSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgk\nSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYk\nSZJaZDAmSZLUIoMxSZKkFhmMSZIktaivYCwi9o+ImyNiTUSsHGZ+RMSH6vwbImLXrnlnR8RdEfHd\nnnU2j4grIuKH9f9m498dSZKk6WXUYCwi1gfOABYBOwOHR8TOPYstAhbUv6XAmV3zPgnsP8ymVwJX\nZuYC4Mo6LUmStE7pp2VsN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjYCiAzvwz8zzDbXQycU1+fAxw6\nyA5IkiRNZ/0EY1sDt3VN317fG+syvR6amXfW1z8DHtpHXiRJkmaUKTGAPzMTyOHmRcTSiLg2Iq69\n++67JzlnkiRJE6ufYOwOYNuu6W3qe2Ndptd/d7oy6/+7hlsoM8/KzIWZuXDu3Ll9ZFeSJGn66CcY\nuwZYEBHbR8RsYAmwumeZ1cCR9arKPYB7urog12Y1cFR9fRRw0RjyLUmSNCOMGoxl5n3ACuAy4PvA\nBZl5Y0Qsi4hldbFLgVuANcDHgFd01o+IzwD/CTwqIm6PiGPqrNOAfSPih8A+dVqSJGmdMqufhTLz\nUkrA1f3eqq7XCSxfy7qHr+X9XwB7951TSZKkGWhKDOCXJElaVxmMSZIktchgTJIkqUUGY5IkSS0y\nGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWrR\nrLYzsK6av/KSRrd362kHNro9SZI0OWwZkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJ\nkiS1yGBMkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBM\nkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQW9RWMRcT+EXFzRKyJiJXD\nzI+I+FCdf0NE7DrauhHxxIi4OiK+HRHXRsRuzeySJEnS9DFqMBYR6wNnAIuAnYHDI2LnnsUWAQvq\n31LgzD7WfQ/w1sx8InBKnZYkSVqn9NMythuwJjNvycx7gfOBxT3LLAbOzeJqYE5EbDXKugk8uL7e\nFPjpOPdFkiRp2pnVxzJbA7d1Td8O7N7HMluPsu6rgMsi4n2UoPCpwyUeEUsprW3Mmzevj+xKkiRN\nH20O4D8eODEztwVOBD4+3EKZeVZmLszMhXPnzp3UDEqSJE20foKxO4Btu6a3qe/1s8xI6x4FXFhf\nf5bSpSlJkrRO6ScYuwZYEBHbR8RsYAmwumeZ1cCR9arKPYB7MvPOUdb9KbBnff1s4Ifj3BdJkqRp\nZ9QxY5l5X0SsAC4D1gfOzswbI2JZnb8KuBQ4AFgD/BY4eqR166aPAz4YEbOA/6OOC5MkSVqX9DOA\nn8y8lBJwdb+3qut1Asv7Xbe+/xXgSWPJrCRJ0kzTVzCm6Wv+yksa3d6tpx3Y6PYkSVrX+TgkSZKk\nFhmMSZIktchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIk\ntchgTJIkqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIk\nqUUGY5IkSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFhmMSZIktchgTJIkqUUGY5Ik\nSS0yGJMkSWqRwZgkSVKLDMYkSZJaZDAmSZLUIoMxSZKkFvUVjEXE/hFxc0SsiYiVw8yPiPhQnX9D\nROzaz7oR8cqIuCkiboyI94x/dyRJkqaXWaMtEBHrA2cA+wK3A9dExOrM/F7XYouABfVvd+BMYPeR\n1o2IZwGLgSdk5u8jYssmd0ySJGk66KdlbDdgTWbekpn3AudTgqhui4Fzs7gamBMRW42y7vHAaZn5\ne4DMvKuB/ZEkSZpW+gnGtgZu65q+vb7XzzIjrbsj8IyI+HpEfCkinjyWjEuSJM0Eo3ZTTnDamwN7\nAE8GLoiIR2Rmdi8UEUuBpQDz5s2b9ExKkiRNpH5axu4Atu2a3qa+188yI617O3Bh7dr8BvBHYIve\nxDPzrMxcmJkL586d20d2JUmSpo9+grFrgAURsX1EzAaWAKt7llkNHFmvqtwDuCcz7xxl3X8CngUQ\nETsCs4Gfj3uPJEmSppFRuykz876IWAFcBqwPnJ2ZN0bEsjp/FXApcACwBvgtcPRI69ZNnw2cHRHf\nBe4FjurtopQkSZrp+hozlpmXUgKu7vdWdb1OYHm/69b37wVeMpbMSpIkzTTegV+SJKlFBmOSJEkt\nMhiTJElqkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFfd2BXxrJ/JWXNL7N\nW087cMLTGS4NSZImmy1jkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1\nyGBMkiSpRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSp\nRQZjkiRJLTIYkyRJapHBmCRJUosMxiRJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSpRQZjkiRJ\nLeorGIuI/SPi5ohYExErh5kfEfGhOv+GiNh1DOueFBEZEVuMb1ckSZKmn1GDsYhYHzgDWATsDBwe\nETv3LLYIWFD/lgJn9rNuRGwL7Af817j3RJIkaRrqp2VsN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjY\nqo91PwC8Dsjx7ogkSdJ0NKuPZbYGbuuavh3YvY9lth5p3YhYDNyRmddHxBizLU2M+SsvaXybt552\nYOPblCTNHP0EY42LiI2BN1C6KEdbdiml65N58+ZNcM4kSZImVz/dlHcA23ZNb1Pf62eZtb3/SGB7\n4PqIuLW+/82IeFhv4pl5VmYuzMyFc+fO7SO7kiRJ00c/wdg1wIKI2D4iZgNLgNU9y6wGjqxXVe4B\n3JOZd65t3cz8TmZumZnzM3M+pfty18z8WVM7JkmSNB2M2k2ZmfdFxArgMmB94OzMvDEiltX5q4BL\ngQOANcBvgaNHWndC9kSSJGka6mvMWGZeSgm4ut9b1fU6geX9rjvMMvP7yYckSdJM4x34JUmSWmQw\nJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqUSvPppTWdT6QXJLUYcuYJElSiwzG\nJEmSWmQ3pTSDNd0daleoJDXPljFJkqQWGYxJkiS1yGBMkiSpRY4ZkzQuk3WbDse/SZqpbBmTJElq\nkS1jktTFFjhJk81gTJImmU9gkNTNbkpJkqQWGYxJkiS1yGBMkiSpRQZjkiRJLTIYkyRJapFXU0rS\nDDUZt+nwylBp/GwZkyRJapEtY5KkKc+b8Woms2VMkiSpRbaMSZJU2QKnNhiMSZI0ibzoQb0MxiRJ\nmoFm0tW0Mz2AdcyYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLWor2AsIvaPiJsjYk1ErBxm\nfkTEh+r8GyJi19HWjYj3RsRNdfkvRMScZnZJkiRp+hg1GIuI9YEzgEXAzsDhEbFzz2KLgAX1bylw\nZh/rXgE8NjMfD/wAeP2490aSJGma6adlbDdgTWbekpn3AucDi3uWWQycm8XVwJyI2GqkdTPz8sy8\nr65/NbBNA/sjSZI0rfQTjG0N3NY1fXt9r59l+lkX4GXAF4dLPCKWRsS1EXHt3Xff3Ud2JUmSpo/W\nB/BHxBuB+4BPDTc/M8/KzIWZuXDu3LmTmzlJkqQJ1s/jkO4Atu2a3qa+188yG4y0bkS8FDgI2Dsz\ns+9cS5IkzRD9tIxdAyyIiO0jYjawBFjds8xq4Mh6VeUewD2ZeedI60bE/sDrgEMy87cN7Y8kSdK0\nMmrLWGbeFxErgMuA9YGzM/PGiFhW568CLgUOANYAvwWOHmnduumPABsCV0QEwNWZuazJnZMkSZrq\n+ummJDMvpQRc3e+t6nqdwPJ+163v7zCmnEqSJM1ArQ/glyRJWpcZjEmSJLXIYEySJKlFBmOSJEkt\nMhiTJElqkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElq\nkcGYJElSiwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqkcGYJElS\niwzGJEmSWmQwJkmS1CKDMUmSpBYZjEmSJLXIYEySJKlFBmOSJEktMhiTJElqkcGYJElSiwzGJEmS\nWmQwJkmS1CKDMUmSpBYZjEmSJLWor2AsIvaPiJsjYk1ErBxmfkTEh+r8GyJi19HWjYjNI+KKiPhh\n/b9ZM7skSZI0fYwajEXE+sAZwCJgZ+DwiNi5Z7FFwIL6txQ4s491VwJXZuYC4Mo6LUmStE7pp2Vs\nN2BNZt6SmfcC5wOLe5ZZDJybxdXAnIjYapR1FwPn1NfnAIeOc18kSZKmnX6Csa2B27qmb6/v9bPM\nSOs+NDPvrK9/Bjy0zzxLkiTNGJGZIy8QcRiwf2YeW6ePAHbPzBVdy1wMnJaZX6nTVwInA/PXtm5E\n/Coz53Rt45eZ+RfjxiJiKaXrE+BRwM2D7uwE2AL4+QxIY7LSmUn7MlnpzKR9max03Jd1Ox33Zd1O\nZ7L2pV/bZebc0Raa1ceG7gC27Zrepr7XzzIbjLDuf0fEVpl5Z+3SvGu4xDPzLOCsPvI56SLi2sxc\nON3TmKx0ZtK+TFY6M2lfJisd92XdTsd9WbfTmax9aVo/3ZTXAAsiYvuImA0sAVb3LLMaOLJeVbkH\ncE/tghxp3dXAUfX1UcBF49wXSZKkaWfUlrHMvC8iVgCXAesDZ2fmjRGxrM5fBVwKHACsAX4LHD3S\nunXTpwEXRMQxwE+AFzS6Z5IkSdNAP92UZOallICr+71VXa8TWN7vuvX9XwB7jyWzU9BkdJ9OVhet\n+zI105lJ+zJZ6bgv63Y67su6nc6UHNY0mlEH8EuSJGni+DgkSZKkFhmMSZrxIiKm03YlrVsMxqRp\nJCJm1f/+dsdmY5iQz+0RDW9Pwxhv0GvQrKnOA7pmnKlw4J2IPETEQ4BLIuIRmfnHpgOLif7c6q1v\nNqqvHzCRafWkOx/4akQ8vsnPLSIeCJwXEe9uYnttq88S7rzu6+KuCcxLdL3eKMc/uPnh41x/2hj0\nd1zLc+f1Q5rL0dQREdu1nYe1MRgbUFsn/LWl21R+ImJuRMwZfcmBtv0XeZyIgKJz4J7ME35Nb5eI\n2DwiNs7MnIBWmF9S7t13TkRs03Bg0f25rRcRG3ReN7V9YC9gYUS8GHh/RGzYxLZHSzczbwX+ATgr\nIh7VxOcWEetl5m+AlwBPj4iTG8hua+r3fVxEzIuIJwCndgdnk62rLK4APjqe76sGFmdExOObyt9U\nqPANp+d3PLffz61Wkg6OiEMj4kWUsjChv8/J/gxrOXh9RCyYzHT71WrtZzrrKvB7ApvUW3hMqJ4f\n2ouA3wMbZ+Z5DdQciYgTgOcAv4yI2zLz9ePdZte2u/O+P/BH4KeZ+d0JSuPVwC4RcWxm/r6pNEZI\n+xXA64F/BX4WEe/PzLvrSfuPDWw/ahDxGeCJlBaZIzLz9ibS6PrcTgAeCzwoIs7KzKvGm/fO9uuJ\n4b3Aw4ATJ/p76S4PwLXAsyiB7Esz86bxfG5d6z0BuAE4PiLWz8x3jj/nw6snk19n5h9qQLsZ8IPM\nvLxnX8esbvMW4Fbgx5TH2N3fSMYHFBH7AocAL2zgN3Q7MKdut8nfy2HATsD3getq4N+arnydBOwA\n/A3wiz5W/QNlHy4ANgQel5m/r2W68XLQc6x+Xn37nsy8cqLSAQLYElgI/LDJdJpgy9gY9TSfrwA+\nBLw9Is6OiG3Xvub49dQWXwH8DlgVEYeOd9sRsQRYTKnp/z/gcePdZreuvL8CeDOwI3B9ROw8AWks\nB54HnFIPKBNdwzsE2Bp4OvBxIIE3RcQWNYAadw2wBjMHA+cBXwZ+DXwuIrZrqoUsIo6jnPzeSjlo\nvXS826zbDYB6oP0e5WT/h4jYsontr01XeXgV8G7g8zX9T0fEY8b7uUXEkXW75wBvARZFxFvGm++1\npPUI4J3AUyPiKOAUyji4z0XEi2r5GLR7KmqAcjmlbG1JuUl3p8Vs0tWycRCwM7B5J59j3MZ8+NM9\nLa8BPtz5TTaUx5dTjmW/oTxF5tiIeHIT2x6Pmq/FwFsy8xcRsXGUJ+D8xWfY9du8nxK0/Qq4CTi4\n6/3Gdf02jwdeA8wGroiIpzWx/YiY3QnEamvvFpn5c+AjwLKI2KGJdJpkMDYGPdH8bOBBwB6Z+SRK\nYVoZEdtMQLrbRu1yi4i5wDMpN8zdCbgK+OcGAo7fUR7ufiTlAe/PrentMs7tUrcTEfFoypMa9gXu\nB74E3NQ5IIzjZPKonpPgPOC1wENq8HdFRLwwIjZsIjDqSne9iNgMOB94Smb+BPhP4AuUA/R76kGg\nqZv5HQC8KzPfQ3m02GXAJyNi24ZOMJsAR1CehvF74Jh6UBs4aOo6IO4UEfOAN1JaEF8C7FeXeURE\nNDamJyJ2jNJi3fEY4FWZeTZwHCUoOysiHj3Oz+0BwLsz82pKkPxq4PkRcco4tjmszLyF8vDjw4Gn\nAi+o5eBQSoVsySDlrOuY9pD629gPeCHw9YjYs7aY7RQRmzS3N8Pno+v17My8C3g/5bF5K2qlo++A\nM0pX67si4vKI2BW4HDibUhbG3f0epQv3qcBLM/N04HV1ViPBxBjz0v3ZzaK0AH4SeHiUHoJPUyqH\nD+wuIz3nsxWUCuV+lADz+fXYSUQsjIjtm8xrPXZuV9NbBGwK/BtwdQPfzeaUSvHCKM+9Xg78S0Q8\nE/hvyk3o/9RKOp60mjRlMjLV1Zpjp+C+BriEErgcWBc5Dngw8M6I2LrBdLemnHhn1R/aLymPnHo3\npSXmBbX2cnREPGkcST2I0sW2b2Y+px6Ej6WckMc99qp+dj8FvkY5IS8GDqgnw+OiPDR+oJMJpYVg\nbkS8vb79fzWNd1Ga379ICWQYJI0RzMrMX1K6qh4fESdl5v2ZeQ1wMeUxX42Mu6kHjdmULkSAeylB\n3xaUx4ptNJZAcy3Lbk0JkHfLzAMy8z7gGErZGmg/6gl0EfBZSuvBv1Nap/4J2C8i3gt8A2ikEhNl\n7MuLgefVgy+Usr245ud+4F8ogedHOjXoPra7tmVeXYOH+4FvA98CnhMNDYCulZhO68UbKV1JjwGe\nUU+u/0YJyD4dpctsTNvuanH9NPDPlFaKayjf1aX1JH0BMKHjbHqGF5wRERcCD6S0Ov4/SkD2iJF+\nv10n+t2AVZTg+CpKxfKzlLJ8ZE1v4CC8bn9jSnD8kigXGNxEOX4eNNGBa6+eIQZnAXdRWrZOp5wr\nLqJ8lmtb70RKK/ivM/NXwDeBMyktvRdSguL/azKvlGPZncB3Ka28BwMH19/R8oh41CDbj4gNM/N/\nKN3Sr6MMiXgL8EHgMEqZeC3wN1G6YBtpJW1EZvo3hj9KzediSuvUGyg1t/3qvI2BjwEPazjNBwGP\nB1bU6XdQWpY2rtMvAq4Dthnjdo+mFNRn1Om3U04mjwb+mjIW5jED5nmzrtdLgFMpweoVwE965n0N\nePgAaczuer0bpQbUearEE4FN6+v9KEHGQxr8Tl5GOeCfAGxPOVn9HHh11zIbjWP7nf14ArArpbVv\nHiWQOb7OeyrlQLnLONJ5CSV42ZkSoHwN+EiddwxwI/CocWx/PiXYemQtp98F5tR5e9QyuHdD30nn\nM1sAvAl4HyVw2baW5ZV1/osoz8btq8x1tltfHwYcCzyhTr8P+DplfM7RwGeAuU3uT319HHBsfX0y\n8FFgH+AB9b1n9Ps9Aet1vd6R0i21C6XC8hrgo3XeoZRunf2a2J8+8nVEPT7MAn4AvK++vyvlZHoq\npQI00jb2oQSWB3Tvb/2tnFnL98HjyOPmwIeB/etn/rfA0jrvEEolY5PJ+Lx68rVv/ew2qtMPBx7Y\n9T1eC2wxTLnanHI+25ZyfD4MWFaPBzsB7wEe3UD+tux6fSjwd/V7+TTw8655L6jHi20HSOMhlDGp\nT63Tp1AC0V3q9KaU49Cq+j3t3ft5tPnXegam+l8tkH9VXz+ZEvS8rU5vVg+SFwEHTWAeNqacMD9O\nOXnOrgfJ6+pB+RrgsWPc5uJa6D9Mab4/AdiIMuDz7yknlYF+hJQT8N93FfajgJfX11tQBk9+jFKL\nu5YyWHSsaTyY0mWzBeWigxfRFZzVZWbVA8sNg6QxQtpLKWNrnklpDTmlvv9oyoUJr2wonQMpwfG5\n9UD7KspJ/4fAJyi1vwPGuM3uA/GhlODu72rZemE9oF1Oufrw34Cdx5H/rSiB6rKug+wOdd5zqCeL\n4fI2zs/tQfXvzdRgtR6Eb6jleg0DVDLq5/8fdbtX1f3amFKJuYAS8D+hqXLWle4JlIDvcV3v/Q1w\nBiWAesAYtrU1pZt4yzq9ELi4a/6jKd3ue9XpDZven6601h/m831a3d8vdtKmnLQX0HVCH2GbL66/\nwc7xZhY1+Kxl4iTgmAHz++j6/1jgzPr6ebVMXVnLd+Pffx/52pISrN4OLOh6fxPK+eL7nfLe8/t/\nCaVF7B8px+LPUo7LX6JW/BvK3zaU88EL6vQhwBs7+aEM7fg8pWv1Gwx4rAY2AD5AOTfuVt97I6XF\nd5dOeaMML3gb8NrJ/q5GzH/bGZjqf/UAcSZwYJ1+dz1QPKJObwq8shboTWgwyq4FdU/KSTHqD/+j\nwFF1/tMpNb75Y9zuYuBqYKs6/TxKk/YKakDDKDXQEbbdORGupNRAnkLpGjiua5lNa5rP73yOY0xj\nVv1/BKVW/72ufEfPskdRA4AGv5d3UAKNoxiqyXfS3xHYsYE0HlQPik+v0ztQAsDnUioBOzHGgKLn\nQLw1cCKwU51+QT0QP7c7D4OU2fr/8ZQg77XAj4D/6vre9qAEM49s8nup234cpda7Wy1nb6a0XuxE\nOVhvTh8n9bqt7hakJ9Xf+HqU1qNv1v17edc+D9wSOkIeHkg52c/rTYNyocUHqS3kfW5vK0pA+TbK\nuJlN6nfR/fs8HVjeW2Ya3q9NgWfX13tTTpavqnn5PLBBnfcm4NQ+ytuWXb/Bwyjdak+r07O6ljuV\n0hI0ayz7RmkF+wHlIorZlApLpxK2HqUlaYuJ+KzWts/1dWeft6OcGz4AbFffewClQvcXx1hKC+K/\n19dPowRlO9bp4ygVwNlNfP+UrsLl9fdyIOX8c0LPMvvWPM0fMI1OwD27fser+POA7EJKxaMTkJ1E\nDfgnqoyPeR/azsBU/aOcMHaitMC8shb0feu804FPUU8mdZk5DaUb3f/r6/OBI+vrIylXcB5PT8tC\nn9vfGNidctXM67veP5RSe1lRDy5jKqCUYHE+cH6dfhjlpPXh+kM4h9Lltg/l4LvZWPNetzsXuKi+\n3ocyIPMyatcwXSfQCSgTe1ICoROBm4HLuuYtB17UYFqbUMYlzu9673DgtAG31x1YnEQZ3/JThloQ\n5lACsvMZCvYHOkhRxn98iRLw/wOlq+OuWh5WUFr7Fjf0OfUG34+ktKycWw++m1JO6GdRuy8GSGNr\nShC3A6U19EuUk/lbKN2uywf5zfSZ9gaUAOXYnvc73aSbj2FbnWB4C0rA+mbgoZQAfxXluPbMWraf\n0vS+9ORlPmX8zr8C36rv7VTLxjLKkw1eDFzPKK2ztbz9C+Xk+ob6XR1KGTawZ3f5p3TxPn6Mee0M\nil/N0FjFJ4WMAAAS50lEQVSk44GvUE/4bfzVz+9jlOPrYyjH9XdQuuo656bO+WQPYGF9/bR6bDlj\nmG0eXcv0wC3iXdvqPoc9jNKj8BFKS9XllOPpiykV8/njTYfaPVx/M2+hBH+dgOxtdZ8fXKePGWs5\nmPDvs+0MTMU/Ss3q7npA2LweaFdQugU6Adnf1h/n9hOUh4d1vf4r6viJOr2UMuZlTAFgPch9klKj\nfhmlVenorvkHAQ8dML/dB/pnUVoSHkBpGfk65XYGK4HP1R/FmFvEutLamHID0Y0Zapm8iNpVS+lq\nabR7pZaDj9YDyBMotetX1nkvAr7D+MZWdQ4oD2eoe+aU+tl1xoEsoXSHDVxjpXRr/XP93JbWMtBp\nfdusHhgHHvNIObl/haEWtxW1vL2d0sL8GobGWDbZiry46/X8WtY/RbngYXPKSbjfFrGnAkvq61dS\nuqI/QRkf9hbgDXXeyyhBXl/bHeP+vBA4uat8vQPYp04fTgnQxjwGspbfgyityl+hBBU71jJ9HqXy\ndEjT+9OVfnel4GTKVbsf6nrv6ZQA/px6nBhx+AV/PubtwFq+Ol2Ix1AG/8+hp0t0lG3OpgYjlPGm\nr6Yczx5af5MnUm4t9EdKT8mEVQBHyGNf4+soleTO+Wy7ug8bUQK2C6kBWl13h1qemw7EOsevDSkt\nyVfVPK+gDI84j67u1UHSoQx7OJdSMdqZcs4+hXLMfkpdpvGW+Ea/07YzMNX+KMHEJZTujs3rgeEg\nylVxnRayzliodzLAwPNRCtV6lGDiznpgWUSJ9K8HXtK1/KZj3P5fUWo8T6S0rL2aoZP9SQ18ZrdS\na+mUmuNNlK6qDerBa1X3D7+Bz+u5lK6vTet384b6vb2VMvah7xaDMaT5TuAT9fXzatm4EvgqYxyz\nt5bt708ZP3EuJZCYV7+j79eycBPlZpxj2eZTGLrwYz4lmLuya/7xdfudLqNxBUiUgO5rDHURbUBp\ncb2IOvayoe/iT7+X+v8bwFe75u9Q9/XyWg7H0iV1IKXy8HbKAONHUlpzT6Z08f2REiz8kBp0TkBZ\n243S/X4cpVJ4Ui0b/0jXGKAxflZPrev+A+Vk/KX6ub2K2iXNUDfORLTydZ+gn0HpUTiUEuC+kxpc\nUgKsoI+Wf8qxcrgxb8+q04MMBN+hlpvzKK3unStK/47Sndf5rTyHcVTAxpingcbXMfz5bBElMHp/\nLQe7dm237/GHfeb7BEo3+8cZaqVcSjkfjOlYNkIa+1IuNHp2/X1+tr63Xi1Xf88Yz5dt/LWegan2\nRxmr8y/1B/0ZSg3o7yknyvXr9LnUAa4NpTlcLaJzddaXKYN1V9Y8DRRkUIKV19TXsyktBx9gaPzO\nnPEcgCldBTdRux/rAezblBr3AyhBxQcprVlNDdTen3JC7Fw1ubQeeAa6AnQtaSyk62qyWjae3/ne\nKM3vD24gnUdQaovPoJyMXkcJ8h5IGVt3WOdgNsbtzqOME5pfp/ejtIyd1LXMiZSLQR7QxHdDCRze\nxFBL5XMoLYnnNHGw589bV+Z0vf4CdRxMnX4L5WQzyJW6+1IqL5+q0xtSTvSnUgKIo2h4LGJN5zEM\nXfX2pPobOq6WtW1qedxqgO3uVn/ne9TpHShXTH+CcgJ7NwOOEx0gL8fXsv7wOr03pafhrZQK7+kM\nM/6ufgbPpwQVe9U8P5ByjDy2a7nTGaqADBRcUlpBf83QFawPq2X4B5ShEWP+DsbxeQ08vo7hz2cf\np16JW38fZzKOK7JHyPdyyq1stqd0i99KbXWtZe/DNX+DtvIHJaB/N6U1bB9Ko8VbKcHzXpSArPHf\n6YR8z21nYCr+UbrWfsNQd8SxlCsOn0MJyI6j4dtX1HROoHR9/jtweH3vQZSWrH+iNDUPNDaNcgK5\niK4m6Ppj3mG4A9+AaSyiDNbuBGQnUK6W3JVyMpuI1qoDKLX9TqtcY92TNd+vo7RQvJZyMj6aemuJ\nBtPZilJz7dxSoNPacwbw4gG3uSVDwVBn/EznKuBFlFp+9204GhnzWLe1DaVV6YuU4OVmSjCwmgav\nNqME35+l1LIPru99gdLi8wZK1/G8cWx/MeW+fi/sem81DV45zZ8Hlo+knBhfyVAr0ZOBe4C/GWc6\nnRstd45pG1CuansXZUzcruPZ/hjysUs9Jmzb8/6elK7YbzFCKzPlCsBfUCp+ncDyubU8f4CGxrzV\n4+IRNT9Hdr1/PGWox8DDLAbIy3zGMb6O4c9nn6AEdhvXz33cXe09ZXnDmreHUCpnF1K63m9h6GK4\nQc9lveNEH0AJlq+ktMzPorSUfYxJuqiike+57QxMxT9K3/o+9QffuXz9ZfXAv88Epfk8SpfhjpQW\nnx8Br6jz1qfUAAbuEqW0fL2DcnLsPPPtmqYLK38ZkHW6dibyEvnFlBaE9Who/AalVncB5TYAp1Ja\ndU6jtJbcDuw+zu13X3W4ihL03cGfj+F7GwN2IVO6KjpdLRdSxuJcxdAVYM+hdIWu6M5Pg9/Jg2s5\nPpkSaO5OubXEQGMSh9l+pyX2SfWzO52h+z2dTFfL3DjTOaieQN5CCWq/w8RcBXpQ/V2+r+7PMur9\nyihB+dWMs6ul/k7WMFTR25NyVWhjgXgfeXgSQxf5BEOtOp3/w7YyM1RBeTgl4P4xQy1rD6G0wJ9L\nGUbS2Ji3Ws6ur5/dMymVjDFfODVg2o2Mr2Pt57PP0WAPT08524/Syr8hpXGhU5avrL+hge7FxtBx\n89mU4+MRlEry5pRK+VaUSs3lNDD2bTL/Ws/AVP6jtIz8kDIQtPOomEaap+kJGii1hnd1TT8ZuI0G\na6z1QLaiFtQLmaB74lACspsYaq0a6MrJMabZ2AGSctL9D0ot66uUIPaBlNamUyhdegMNOO1Jp/uq\nw/MoAcVtlFad51ECzL3Gsf1OV0vnQoPH1P15U53elwlo4R0mH8+q+9hIeWNorEzn5rcbU4KZTzN0\nI+QmLw44lNKqdBENtYjw50MTllCuNn0rZQzXJyitGe+ijBW8gHG08A1T5n5dT8QXMLGD9bv3sTOm\nadP6+zm+a97Lgb8d7Xur38O36sn2lfXk+8Q6b5u1pd3AfuxPqUh8kwZugDrAZ9fU+LoJOZ8NU5Z/\nRqm4XkPpSXgPpdJ5LGWc2rgqZJQxnd+ktJJeztDFC2+l9GJ8j65b9EyXv9YzMNX/KDWuX1Av95+A\n7R9KqUW8k9Id+adxO5RaXuPRfT15Tehdoim1o28xQZf8T3Dej6IEQy+tP/bOOL759f+Y7781TBq9\nVx0up9T03kEJKk5lHHcKr9vsdLV8k9rdSbk1yK003NU6Sj62ot77aMD1uw/2D66/i09RWpIf3zXv\nciZg7Evd9p6M4/L7EfZnHqUi1rkVwWH1xP/eeiK7kgZa+HrSP4QS6L+2k5+mf6M9+3hsPbYdR2nB\neDqlC3sVZbziqDd+plx4dD1dA+ZrGfjPuv2fTtR3X9OaS0NPVhhjugONrxthe42ez0Ypyy+qZfn7\n9dj2I8Z5OwnKkwJOoQyF2JcS8G3bNX9HakW56TI90X+z0Igy8/r60OFGns3V83DWJZRxDh+j/Mge\nRXlg95ejPHB8L0rtuFGZ+dumtzlMGhdFxJU5lZ791b9bKWMEf5qZz4A/Pfdtu4g4OTP/t4E07qUE\nqp1nGJ5FGS/0EMqg8c/VdP9UXsYqM9cAayLiV8CpEXEP8L+Uqx3/dZz5H0s+7hzn+p3fy86Z+b2I\nuI4yKPgbwIkR8RmGbnMyrrRGyMOXmthOz+//BMpYnwcB74+IOzLzc/UZix+gBE3nZXlGaGMyc3VE\n/B9wdkT8KDMvbHL7NY3OPi6jVAhOpgwiX0BpkXs5ZTD5hpQxWd8bZZO/p1Tu9oqIF1Jai35KGdO3\nHeWB3d9qej86MvPuidr22kTELpRWrL0z86c1H1dGxH2UQORlwBGZ2fe5qcnzWR9l+dMR8RtKN/u3\ngPcOcuzseobq07u2dQ6l7PxVZt4WEQdSLkC5qLPeoMfNthiM9SEzv9vEdnoK7zwgKfd4+lFEXENp\nFdmLMnZoH+CwzLytibTbkJm/aTsPA7qO0iX1x4jYi1LjO4pSm2zkxJiZv4yIzwPPjoh7MvO7EfFZ\nSvfbwRFxSWb+rokDSmb+c0T8gdJdcC/lAP7D8W53MkXEU4DzI+KdlCdSHEXpcr2Y0qX3a8qYsZ+1\nl8vRdf3+D6VcGXkEpWXnccAeEfGVzPxsfTD7r5oOxLrycXlEHE1prWhMzzFua8pwi4Mp3Vc/obSM\nLaMMyVg5hk3fRmlBO4rS/f45SkD2c+AL07TSN5r1gDU12AhKsPEH4GuZ+aWIeE9m/nqsG23qfNZn\nWV4dERsB1w9aia2B2O6ULtpllPGCF1Mqrf9Vg7TTKS2v01fbTXPryh9/3pzbedbc9yiFt9MNdijl\noPMUJukyc//W+n1tRam9X0IZGNzYsy270piUqw670tuSFrpaGsj37PpZfY3S5bqY0o13A6UrNpjA\nC0QmYH+2ptwj7+N1eqNaDj5MafGYlr/9nmPcCkrFcg7lCsor6vs7Um4N8ToGuM0JQ4//eTLlYppG\nHjLf9l/PZzfu8XWTmO9JKcuUoTz3M3R7piX1ePB5ygUBE/Zs6Mn6W2+tUZoalZ1fz5/XIi5mqBYx\nKzP/iVLL/++coBqx+pOZd2bm31EC5GMy8zsTkMbtlKDig5SWncMYeqxU4y08mXlXttDdMh61ReyN\nlBPTSyhXsm5GGYj8WMq4lPUy8/etZXKMMvMOyn2iFkXE4Vm6md4K/IFylevsNvM3qJ5j3LOBWzLz\nV5TupIfW1r5HUsZ5nZeZvxsgmfsj4kmUx+q8MTOvbCb37elpTTwWeG9EHEe5iv6vgUMiYlVEnEhp\n/fkkTI1uuMkqy5l5OWUc79KIWJyZ51Nua/Rm4NDMvLi2Hk5bMQW+z3VGbbb/T0ot8ZjafPtGhp57\ndpVB2LorIp5FGSP48sy8vu38TAV17OT+lO6Jj1Luj3V3Zl4YEcdQbvLaaFfbZKnjXN5F6bL7TETM\nolx5PK0C5m4RsRXlGPf1zHxhDcCS0o20G+UqviWZeeM40tiEcl+sH49nTOVUM8z4uk9TxtfdRRlf\n9ytgdY4+vm7STVZZrum8jXKLj3Oa3HbbDMYmWUQ8j1KrO6mr0L6H8oiVU3ISBtdraqonstmZ+ZO2\n8zLVRMQTKAf7B1G6WndqOUuNiIhFlIs3Xp2Zn207P2M1XDAUEQdRnlqyMjM/Wd/bgHJ7g5/V1pR1\n3jDj695G6RlZQhkM//266LumQ4VjsspyRCymHAv2ppSnGRHEGIy1YCbWiKWJFhFbUg7Af01pXbm1\n3Rw1IyL2BX6Umbe0nZex6Akmnku5Lc/3MvPbdZ/eC5zeCcg0pOezW0EZ//ZtylXC78nMfSNiR8r9\nDv8W+PCA3bqTarLKckTMnWnnS6+mbEFmXhIRfwTOioj7ai1iRhUsqWmZeRfwmYj4XJarymaEzLyi\n7TwMoiuYWA4cSRnL9B8RsaQe414NfCIi7s3MT7eY1SlnmPF1qzPzVxHR5Pi6STdZZXmmBWJgMNaa\nzPxiRLyMhi8tl2a6mRSITXcR8WTKjT73oQRkPwE+HBGbZOYFEXEk5Qpx9ajDEk6njK/7rxqAfYPy\n+KCvMjS+bkLunaepxW5KSVJf1jJGbCvgqcDyzHx2bRF7H7BfZk7azYWnOsfXaSS2jEmSRhURG9Vb\nF3TuDr9eZl6XmXfW8Xyd27/cTnmY95qWsjrljDC+7uKIOIJyOwsy85O15fe6NvOryWcwJkkaUUR0\n7of4D5TH8Pw1cGdE/DIzD6U8N3LPiPhHYCdg8Uy5wKIJjq/TaAzGJEmj2Q5YRHkG6FOA3eqA869H\nxPmZuSQi/pfyQPU3G4j9JcfXaSSOGZMkDSsi1sv63MeIOAl4DOUROMuzPIieiLga+GVmLmovp1OP\n4+s0Fj4OSZI0rK5AbBmwK/CvlEd3PSMitq3L7AHM7kzrT+PrOl2Tu9RHOFGvjHR8nf6C3ZSSpLWK\niEOA5cCB9RYMv6Z0t0VEXJWZP87MvdvN5dTh+DoNwmBMkjSShwOfqYHYrHoF4P2UQON3EXEbcP9M\neSxNAxxfpzGzm1KSNJKfAM+MiEdl5n31vfWAXwBXZeZ9BmJlfB1AZl5MuWnrE4DNgC3q+7sD8yPi\ni5n5ncz8SGbe1FqGNaUYjEmSRvJV4BrgpRFxUES8BHgz5RmKP2s3a1OH4+s0Hl5NKUkaUb0KcDFw\nCHAP8K7MvKHdXE09dXzdqQyNrzuIMr7uKkor4o9bzaCmLMeMSZJGVK8CXBURZ9fpe1vO0lTl+DoN\nxG5KSVJfMvNeA7EROb5OA7GbUpKkBkTEg4HXUnqdvgrMAU4AlmTmLW3mTVObwZgkSQ1xfJ0GYTAm\nSVLDImI2OL5O/TEYkyRJapED+CVJklpkMCZJktQigzFJkqQWGYxJkiS1yGBMkiSpRQZjkiRJLTIY\nkyRJatH/B027mhVP4YPeAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
" plot_top_k(alpha_filtered,'neg', 20)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Observations\n",
"* Negative reviews fairly similar\n",
"* Mention \"good\" just as often as the positive reviews\n",
"* Subtle differences: \n",
" * Negative reviews contain \"movie\" more than positive ones\n",
" * For \"film\" this effect is reversed\n",
"\n",
"* \"bad\" appears with high probability only in the negative reviews"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\"good\" in negative reviews?"
]
},
{
"cell_type": "code",
"execution_count": 227,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def show_context(word, label=\"neg\", index=0, window=5): \n",
" docs_with_word = [x for x,y in train if word in x and y==label]\n",
" word_index = docs_with_word[index].index(word)\n",
" return docs_with_word[index][max(word_index-window,0):min(word_index+window,len(docs_with_word[index]))]"
]
},
{
"cell_type": "code",
"execution_count": 228,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'to be censored in some manner . in an early scene of the film , for example , the lead good guy and bad guy have a minor confrontation in the streets . when the good guy spouts out a'"
]
},
"execution_count": 228,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\" \".join(show_context(\"good\", \"neg\", 2, 20))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Looking at the most likely words not that helpful:\n",
"\n",
"* Discriminative words may be rare\n",
"* Look at probability *ratios* or *differences*, e.g. $p(\\text{bad}\\bar +) - p(\\text{bad}\\bar -)$ "
]
},
{
"cell_type": "code",
"execution_count": 229,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def diff(alpha, w):\n",
" return alpha[w,'pos'] - alpha[w,'neg'] \n",
"\n",
"def plot_discriminative_features(alpha, threshold = 0.0, reverse=False):\n",
" frequent_words = {w for ((w,y),p) in alpha.items() if p > threshold}\n",
" sorted_by_ratio = sorted(frequent_words, key=lambda w: diff(alpha, w),reverse=reverse)[-20:]\n",
" util.plot_bar_graph([diff(alpha,w) for w in sorted_by_ratio],sorted_by_ratio,rotation=45)"
]
},
{
"cell_type": "code",
"execution_count": 230,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmoAAAFdCAYAAACzRUH3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xv8bfWc+PHXu3M6XSiVTkm3E05xkqgjKYXuFzlJpQyF\nSJRcxuVkhkEyZdyVmoZMuUzFkEMRwhhMo3KdqFGJSkoZ/AhJ798f78/Xd/d1zvnu8937+/2u7zmv\n5+OxH3vvdfustfbaa73X57YiM5EkSVL3rDbdKyBJkqSlM1CTJEnqKAM1SZKkjjJQkyRJ6igDNUmS\npI4yUJMkSeooAzVJkqSOMlCTJEnqKAM1SZKkjjJQkyRJ6qjZ070Cw7DhhhvmvHnzpns1JEmSxnXV\nVVfdkZlz+5l2pQjU5s2bx5VXXjndqyFJkjSuiPhJv9Na9ClJktRRBmqSJEkdZaAmSZLUUX0FahGx\nX0RcGxHXRcTipYyPiHhPG/+9iNhhvHkj4rCIuDoi7o2IhT3D946IqyLi++19j0E3UpIkaSYaN1CL\niFnAGcD+wALgyIhYMGay/YH57XUscGYf8/4PcAjw1THLugM4KDO3A44GPrTimyVJkjTz9dPqcyfg\nusy8ASAizgcWAT/omWYRcF5mJnB5RKwXEZsA85Y1b2b+sA27T2KZ+e2er1cDa0XEGpn5xwlsnyRJ\n0ozVT9HnpsBNPd9vbsP6maafeZfn6cC3DNIkSdKqqLP9qEXEtsBpwD7LGH8sVczKFltsMYVrJkmS\nNDX6yVG7Bdi85/tmbVg/0/Qz71+JiM2ATwJHZeb1S5smM8/OzIWZuXDu3L4695UkSZpR+gnUrgDm\nR8RWETEHOAJYMmaaJcBRrfXnzsCvM/PWPue9j4hYD7gYWJyZX1/B7ZEkSVppjBuoZeY9wAnApcAP\ngQsz8+qIOC4ijmuTXQLcAFwH/Avw4uXNCxART4uIm4HHAxdHxKVtWScADwNeHxHfaa+NhrO5kiRJ\nM0dUQ82ZbeHChemzPiVJWrXMW3zx0Jd546kHDn2ZY0XEVZm5cPwpfTKBJElSZxmoSZIkdZSBmiRJ\nUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJHGahJ\nkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSB\nmiRJUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJH\nGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJH9RWoRcR+EXFtRFwXEYuXMj4i\n4j1t/PciYofx5o2IwyLi6oi4NyIWjlneSW36ayNi30E2UJIkaaYaN1CLiFnAGcD+wALgyIhYMGay\n/YH57XUscGYf8/4PcAjw1THpLQCOALYF9gPe15YjSZK0SuknR20n4LrMvCEz7wbOBxaNmWYRcF6W\ny4H1ImKT5c2bmT/MzGuXkt4i4PzM/GNm/hi4ri1HkiRpldJPoLYpcFPP95vbsH6m6WfeiaQnSZK0\n0puxjQki4tiIuDIirvzFL34x3asjSZI0dP0EarcAm/d836wN62eafuadSHpk5tmZuTAzF86dO3ec\nRUqSJM08/QRqVwDzI2KriJhDVfRfMmaaJcBRrfXnzsCvM/PWPucdawlwRESsERFbUQ0UvrkC2yRJ\nkrRSmD3eBJl5T0ScAFwKzALOycyrI+K4Nv4s4BLgAKri/13Ac5c3L0BEPA14LzAXuDgivpOZ+7Zl\nXwj8ALgHOD4z/zzUrZYkSZoBIjOnex0GtnDhwrzyyiunezUkSdIUmrf44qEv88ZTDxz6MseKiKsy\nc+H4U87gxgSSJEkrOwM1SZKkjjJQkyRJ6qhxGxNIkiStqGHXH5uKumNdZI6aJElSRxmoSZIkdZSB\nmiRJUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJH\nGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHXU7OleAUmSNHXmLb546Mu88dQDh75MFXPU\nJEmSOspATZIkqaMM1CRJkjrKQE2SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspATZIkqaMM1CRJkjrK\nQE2SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspATZIkqaP6CtQiYr+IuDYirouIxUsZHxHxnjb+exGx\nw3jzRsQGEfGFiPhRe1+/DV89Is6NiO9HxA8j4qRhbKgkSdJMM26gFhGzgDOA/YEFwJERsWDMZPsD\n89vrWODMPuZdDFyWmfOBy9p3gMOANTJzO2BH4IURMW+C2ydJkjRj9ZOjthNwXWbekJl3A+cDi8ZM\nswg4L8vlwHoRsck48y4Czm2fzwUObp8TuF9EzAbWAu4GfjOxzZMkSZq5+gnUNgVu6vl+cxvWzzTL\nm3fjzLy1ff45sHH7/HHgd8CtwE+Bt2XmL/tYT0mSpJVKJxoTZGZSOWlQuXB/Bh4MbAX8bUQ8ZOw8\nEXFsRFwZEVf+4he/mLqVlSRJmiL9BGq3AJv3fN+sDetnmuXNe1srHqW9396GPxP4XGb+KTNvB74O\nLBy7Upl5dmYuzMyFc+fO7WMzJEmSZpZ+ArUrgPkRsVVEzAGOAJaMmWYJcFRr/bkz8OtWrLm8eZcA\nR7fPRwOfap9/CuwBEBH3A3YGrpnQ1kmSJM1gs8ebIDPviYgTgEuBWcA5mXl1RBzXxp8FXAIcAFwH\n3AU8d3nztkWfClwYEccAPwEOb8PPAD4YEVcDAXwwM783lK2VJEmaQcYN1AAy8xIqGOsddlbP5wSO\n73feNvxOYM+lDP8t1UWHJEmrlHmLLx7q8m489cChLk9TrxONCSRJkvTXDNQkSZI6ykBNkiSpowzU\nJEmSOspATZIkqaMM1CRJkjrKQE2SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspATZIkqaMM1CRJkjrK\nQE2SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspATZIkqaMM1CRJkjrKQE2SJKmjDNQkSZI6ykBNkiSp\nowzUJEmSOspATZIkqaMM1CRJkjrKQE2SJKmjZk/3CkiS1HXzFl881OXdeOqBQ12eVl7mqEmSJHWU\ngZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWU/ahJkmasYfdvBvZxpm7pK0ct\nIvaLiGsj4rqIWLyU8RER72njvxcRO4w3b0RsEBFfiIgftff1e8Y9KiL+KyKujojvR8Sag26oJEnS\nTDNuoBYRs4AzgP2BBcCREbFgzGT7A/Pb61jgzD7mXQxclpnzgcvadyJiNvBh4LjM3BZ4EvCniW+i\nJEnSzNRPjtpOwHWZeUNm3g2cDywaM80i4LwslwPrRcQm48y7CDi3fT4XOLh93gf4XmZ+FyAz78zM\nP09w+yRJkmasfgK1TYGber7f3Ib1M83y5t04M29tn38ObNw+bw1kRFwaEd+KiFf3sY6SJEkrnU40\nJsjMjIhsX2cDTwAeC9wFXBYRV2XmZb3zRMSxVDErW2yxxVSuriRJ0pToJ0ftFmDznu+btWH9TLO8\neW9rxaO099vb8JuBr2bmHZl5F3AJsANjZObZmbkwMxfOnTu3j82QJEmaWfoJ1K4A5kfEVhExBzgC\nWDJmmiXAUa31587Ar1ux5vLmXQIc3T4fDXyqfb4U2C4i1m4NC54I/GCC2ydJkjRjjVv0mZn3RMQJ\nVAA1CzgnM6+OiOPa+LOoXK8DgOuo4srnLm/etuhTgQsj4hjgJ8DhbZ7/i4h3UEFeApdk5vA7ypEk\nSeq4vuqoZeYlVDDWO+ysns8JHN/vvG34ncCey5jnw1QXHZKkGWrYndHaEa1WRT5CSpIkqaMM1CRJ\nkjrKQE2SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspATZIkqaMM1CRJkjrKQE2SJKmjDNQkSZI6qq9H\nSEmSVh7DfrQT+HgnabIYqElSh/h8TEm9LPqUJEnqKAM1SZKkjjJQkyRJ6igDNUmSpI4yUJMkSeoo\nAzVJkqSOMlCTJEnqKAM1SZKkjjJQkyRJ6igDNUmSpI4yUJMkSeooAzVJkqSO8qHsktQHH5YuaToY\nqEma0YYdQIFBlKTusOhTkiSpowzUJEmSOspATZIkqaOsoyZp0lgBX5IGY46aJElSRxmoSZIkdZSB\nmiRJUkcZqEmSJHWUgZokSVJH9dXqMyL2A94NzALen5mnjhkfbfwBwF3AczLzW8ubNyI2AC4A5gE3\nAodn5v/1LHML4AfAGzLzbRPfRElj2Zu/JM0M4+aoRcQs4Axgf2ABcGRELBgz2f7A/PY6Fjizj3kX\nA5dl5nzgsva91zuAz05gmyRJklYK/RR97gRcl5k3ZObdwPnAojHTLALOy3I5sF5EbDLOvIuAc9vn\nc4GDRxYWEQcDPwaunuB2SZIkzXj9FH1uCtzU8/1m4HF9TLPpOPNunJm3ts8/BzYGiIj7A68B9gZe\n2cf6SSsNiyQlSb068WSCzMyIyPb1DcA7M/O3VfVt6SLiWKqYlS222GLS11Gyl31J0lTrJ1C7Bdi8\n5/tmbVg/06y+nHlvi4hNMvPWVkx6exv+OODQiHgrsB5wb0T8ITNP700wM88GzgZYuHBhIkmStJLp\np47aFcD8iNgqIuYARwBLxkyzBDgqys7Ar1ux5vLmXQIc3T4fDXwKIDN3y8x5mTkPeBfwlrFBmiRJ\n0qpg3By1zLwnIk4ALqW62DgnM6+OiOPa+LOAS6iuOa6juud47vLmbYs+FbgwIo4BfgIcPtQtkyRJ\nmuH6qqOWmZdQwVjvsLN6PidwfL/ztuF3AnuOk+4b+lk/SZKklVEnGhNIg7ClpCRpZeUjpCRJkjrK\nQE2SJKmjLPrUpLLvMUmSJs5ArWOmqr6VAZQkSd1noLYCDG4kSdJUso6aJElSRxmoSZIkdZSBmiRJ\nUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJHGahJ\nkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSB\nmiRJUkcZqEmSJHWUgZokSVJHGahJkiR1lIGaJElSRxmoSZIkdZSBmiRJUkcZqEmSJHWUgZokSVJH\n9RWoRcR+EXFtRFwXEYuXMj4i4j1t/PciYofx5o2IDSLiCxHxo/a+fhu+d0RcFRHfb+97DGNDJUmS\nZppxA7WImAWcAewPLACOjIgFYybbH5jfXscCZ/Yx72LgssycD1zWvgPcARyUmdsBRwMfmvDWSZIk\nzWD95KjtBFyXmTdk5t3A+cCiMdMsAs7LcjmwXkRsMs68i4Bz2+dzgYMBMvPbmfmzNvxqYK2IWGOC\n2ydJkjRj9ROobQrc1PP95jasn2mWN+/GmXlr+/xzYOOlpP104FuZ+cc+1lOSJGmlMnu6VwAgMzMi\nsndYRGwLnAbss7R5IuJYqpiVLbbYYtLXUZIkaar1k6N2C7B5z/fN2rB+plnevLe14lHa++0jE0XE\nZsAngaMy8/qlrVRmnp2ZCzNz4dy5c/vYDEmSpJmln0DtCmB+RGwVEXOAI4AlY6ZZAhzVWn/uDPy6\nFWsub94lVGMB2vunACJiPeBiYHFmfn2AbZMkSZrRxi36zMx7IuIE4FJgFnBOZl4dEce18WcBlwAH\nANcBdwHPXd68bdGnAhdGxDHAT4DD2/ATgIcBr4+I17dh+2TmX3LcJEmSVgV91VHLzEuoYKx32Fk9\nnxM4vt952/A7gT2XMvzNwJv7WS9JkqSVmU8mkCRJ6igDNUmSpI4yUJMkSeooAzVJkqSOMlCTJEnq\nKAM1SZKkjjJQkyRJ6igDNUmSpI4yUJMkSeooAzVJkqSOMlCTJEnqKAM1SZKkjjJQkyRJ6igDNUmS\npI4yUJMkSeooAzVJkqSOMlCTJEnqKAM1SZKkjjJQkyRJ6igDNUmSpI4yUJMkSeooAzVJkqSOMlCT\nJEnqKAM1SZKkjjJQkyRJ6igDNUmSpI4yUJMkSeooAzVJkqSOMlCTJEnqKAM1SZKkjjJQkyRJ6igD\nNUmSpI4yUJMkSeooAzVJkqSO6itQi4j9IuLaiLguIhYvZXxExHva+O9FxA7jzRsRG0TEFyLiR+19\n/Z5xJ7Xpr42IfQfdSEmSpJlo3EAtImYBZwD7AwuAIyNiwZjJ9gfmt9exwJl9zLsYuCwz5wOXte+0\n8UcA2wL7Ae9ry5EkSVql9JOjthNwXWbekJl3A+cDi8ZMswg4L8vlwHoRsck48y4Czm2fzwUO7hl+\nfmb+MTN/DFzXliNJkrRK6SdQ2xS4qef7zW1YP9Msb96NM/PW9vnnwMYrkJ4kSdJKLzJz+RNEHArs\nl5nPb9+fDTwuM0/omeYzwKmZ+bX2/TLgNcC8Zc0bEb/KzPV6lvF/mbl+RJwOXJ6ZH27DPwB8NjM/\nPma9jqWKWQG2Aa6d6E6YBBsCd6wk6bgtq3Y6K9O2TFU6bsuqnY7bYjr92DIz5/Yz4ew+prkF2Lzn\n+2ZtWD/TrL6ceW+LiE0y89ZWTHr7CqRHZp4NnN3H+k+5iLgyMxeuDOm4Lat2OivTtkxVOm7Lqp2O\n22I6w9ZP0ecVwPyI2Coi5lAV/ZeMmWYJcFRr/bkz8OtWrLm8eZcAR7fPRwOf6hl+RESsERFbUQ0U\nvjnB7ZMkSZqxxs1Ry8x7IuIE4FJgFnBOZl4dEce18WcBlwAHUBX/7wKeu7x526JPBS6MiGOAnwCH\nt3mujogLgR8A9wDHZ+afh7XBkiRJM0U/RZ9k5iVUMNY77Kyezwkc3++8bfidwJ7LmOcU4JR+1q2j\npqpIdirScVtW7XRWpm2ZqnTcllU7HbfFdIZq3MYEkiRJmh4+QkqSJKmjDNQkSZI6ykBNmiIREdO9\nDl0WEfdr7+6nIXFfdoO/gwZhoLYK8CQxvXr2/0r1f4uI1Xo+T/gYa936bAVcGRHbZWauLMds65po\nyp9VHBHbRMSaOYmVkCNiy0lc9iMiYvtJWva6EfHA9nnziFh9MtIZ48GTteD2/9m0fd4iItafrLQm\n02T/52fyOWWlunB01XQeIBERIyfriHhgRKzbPnf6t++9uE3G/htZZkTsHhEvGPbye9NpgcfewAci\n4sSIOHjcGYeU9iQuew1gt4hYPSK2BfYYIL212nN9zwc+FBELuhqs9Rw3465bRDwHeB2wxiSv1th0\n16Va4W/fvg/9v94CnZMiYv4kLHsd4OnASyNiuyEvew6wM3BIRJwGvJnqOmrStH11RkQ8apKS2BZ4\ndkS8HjgLWHNYC56q/+CY69RjWgA9tEdHRsRqPcufExFD20dTodMX65ksIp7UDrj503XRGXPwvwr4\nOPD5iNgtM+9dkRN4RDwtIvaJiP0ma3170joeeFtEvBmq+5dh77+2zKcCZwLXj0l/aCfuls5ewNuA\nfwOeBBzcLhhD0RM8PDEiDomIw0bSHlYaS/EA6hFxF1GdVN84kfQiYhvgwy04eyMVrF3YxWCtJ+h+\nEvC69p942DKmfQbwaOAdmXnXVK4n8HvqYn0wQGbeOwlpBLARMNRe3ts+/n/AR4CfAi+KiK2HtfzM\nvJt6tvTzgWcCH8rMPwxr+ctxM7AeTErg/DPg4cArgC+OPEN7GP+dnuvHEyPigEGX10c6LwPOAN4L\nvCUiFg1p+fe25Z8IfAA4d+Q8ORMYqE2CFmi8BdgD+M+I2GKSL5pL1XPw7wbsBTwb+Gfg0xGxewvW\nxg1KojotfiWwAfDvbXmTogVPxwBfAJ4SEf8Gww/W2l3uccBBmfmliNg5Iv4+IuZk5p8HSSsiNhlz\nN7g19fSN31GPR3tdZt4dEUMpDmn7Zl/qBPdn4IKWmzNpMvN24DfAfsBV1MVvhS9CmXktdaF5fUQ8\nPDNPBT5MB4O1npzRM4EfAm8FDl9G7u+hwGHA2mOGD1VE3C9a0V1EPCQiHp2ZfwJeDzw6InYfcnpb\nRMQDM/MO4HTguGUFqwPaF9gO2A14+aC5UWP2//XAh4DPAdv1LnuYQVREzIO/9Bl6BfDeiNhwWIHz\nyDZl5i+BLwEfBbaMiH3a8IwJFuv27q92/n8PcHJEnBMRmy97zomLiAXUNWo/4GXAJ4HnRsSOAyxz\nm4jYuH0+Bngq8CpgLeo/OiMYqA1ZVL2Kp1KB0erAd4GbJ+Euqt/12QV4NfC/mXlzZn6Q+hN8IiL2\nXN5TH6JsCexNBZ2bAv8BfGOiJ4Bx1nVv6uL2lsy8JDMfDWwTER+B4eYQtZPnr4CzI+L9VHC4F5Xr\nOGhazwfW6dlH9wIXAG8HDszMm1rO5NMGzVmLiNUiYm3gWGrf/Rb4FvD5QZa7nPRGcu92BL5MHeuX\nU3e/W7fgf8Pxjo92bK0GkJknUM/zfcuYYO2jEfGo6bjJGaut75pUAPEM4MfUvv7XFtiv3SZ9AEBm\nHkYFAm+MiLUmI+CMiPtTOU9PaxejI6h9dhyV2/klYKSqw8BpR8Rc4OXApS0AvI3qzHzgnKKImA1/\nCS6eDJxI/Y8WAzdRAeGEc9Z6blpfBLwzM0+nOj/dnMrh3igqp/RxE02jV7sO/GNEfD4idqD+j+dQ\nxZQDB4QRf8ndfXTbL5/PzBdTT/k5LCJ2jCqWPmxFz9Ujy26f5wDrADtn5o7AHGBxRGw2yPqPpLOU\nwb/JzN9k5o3Af1LnhS0muPy9qKLg3n39fOp/sjrwNxExOyImtPwplZm+hvgCtgL+njrBfA5Ysw1/\nJrDBFKQfY77fHzgZ+HdgF2BWG34c9adea+w8PfOuRuUInEE9KWIJsHYb93xg6yGv+0FUTtrbgM1H\ntoe6KH5gGPsFeDxwILAjdRE7GXhcG7c1dTJdY9B9DzyICjbmUUVEFwDvbuN2pXJk9ptgOmsCD+5J\nZy0qB+V1wFeBbdq4ZwG7T8Ixtl87dh7fvi9s+/EdwFFtH27c537aqufzG6i76Ie3728CvgHMGfY2\nTOC4mdPeX0w9Eu8KYLM27BDgKVS9sHPbeo8cvx8CPg3cb8jrdf/2fnRbn/2oJ81sTz2e713AncD3\ngS2GkN4Dej4fTeWwfAC4gyr+njXAsndov/0D2/enAp/oGb8TdVPwYeARA6RzPPXc6G16hs2nzjef\noIKChwzhWNkJ+C9gE+Ck9t/4GvA9BjyPjUnvAOpRi4uB/wYe1Y6Bl7Vj7jZgzxVc5mo9n19JnY9/\nCBzahq3VjunzgE0H3Vft80HATu3z54BTe8a9A3jNim4DFYidDryx/Sd2Bf6ROm9d0DPtC9uxt/qw\nfpfJeE37CqwsL2BuOzjmUDkMt/aMexbwRWDuJK9D78E/kqs3n6ose0o7ue4KzG7TPGA5y3oC8OL2\n+YPAH3rGPbNt44OHtN5Pbn+mjdrJ5iPU82I37ZlmqyGk81TqAruYCmiePmbcd4CDh5DO1u1k8RGq\nqHlr4InURfxr7SR+0ADL3xV4AZVT+nkqGH8L8MeRfQY8pl0YHj/kY2zjtg93GzN8YTuxfwdY1Oey\nRoKetwKvbcPeRgW127bvDxzm+q/gto5ceA8C3kfllh1IBQ3PauN2oC6Wb6WCynnADVQx1MjF51PA\nx4a4XutRweCG7fuhbZ0Oad/nUBfUV1MX1We04atNML2nAJdRAdkj2rB1gIe14/siWkDAMm76xln+\nlsBXgL9ry92AChCO7pnmLCpH+kET3Ib7Af8CPLT9Ri+kgpsntN91FwYI0nrS2av99gf0DFutLf/M\ndoxM+L/fs8xt239tK+B5wP+2427kmHs4sHCA5e8KfAbYHXgtdZO+Txu3dtuXE/otxqRzIvBt2k1/\n264PUue1l7X/1sNW9P/R3jejbiRu6xn3Feq8vBZVCnE17cawy69pX4GV4UXdqV1CVYB8CVVEeE37\nfip1YdtuCtfnFVS28cntInEQFay9kcrtGMlB+quTajupzGoH8T9T2cRrUTlylwHvHub2tH33Deqi\n8jMql2sR8K/UhXyTIaQRbbkXtZPys6lgaWOqRd56VDHIomXtl3GWvz4t14K6i76Y0RyY91MXma3a\n941oAfsE0tmcCmg3pHIAfgm8pGf8BVRx15lU8WdfAVO/+7C9Pwi4uGf4Wu19JOd4/X62jcoN+Dp1\ng/MJ4NyecadTd+yrr+g+GvaLKvL/NqOByGzgNe13/RJVP+9NwDvbsXVi+5+8rZ0TRi6cQ7mp6Vmv\nB1NFQiPB2SHURehgenKEqSL99w2Qzo5te3ZmNIjevef3Xrtt/6sGPK4WAp+lzlEbUcXL51A3l89t\nx/PmK7rcMcMWU+eYf6eCgFdTAe66Q/xd/oaq6vDCnuNltfZ5HeBvgWOGkM6mwIL2W3ybOr+9Bbgd\nePIElvdw2o0r8NiR47p9X5+6MfwU8JQB1/shPZ93Aq5kNHd4V+DIlt7LqPP/ghX5zamg/3wqMN+C\nykG7GvibnuP1Eur6cumKLH86X9O+AjP9RQUyX6Gi9w8D57fhGwMvpYoYh1pEOM76PAH4TPv8j9TF\n8N+o4Gc2dde6vGKpkYBjbUaLOJ7Zhj2VylEY+M6zLW/79mdZu51Ev9Az7ilU8LTMXL8VSGekSOU8\nKpv7q7S7NKr+3da0i9vSTvDjLHsNqmj4tVSQdn/gQnqKuam7zwsH2W/tJHQw1ZIwqOD7Q+33fNKY\n/bYrsONEtmdp6bb33u25CHhrz/c92zG2Jn3k2lC5C4cB+1CBxOcZDWxHfpeNJuP/MYHtXwwc1z6P\nBKVzGM39fRRVzPgcKvf60p7f60aq+4e1hrg+vUVTL6KKig9u359G5UQdymggdSR10V1nAmltSuU+\nfLhn2Entt35yTxp/SwVZa/R7vHHf3P8XUtVFtqYuoq+hcoq2pW5y3g08aoL76wVUIPla6obzsbRz\nCpXL/VlaoDDg/2OjnmP4UOAPwK7t++ye6U6hcqr+MmwF09mUnvM3FdC8tH0+euS3mcB2vIy6wTuw\nfT+t7ZsEoYqaAAAYuElEQVSHtO8PoDIhLqCCoBU9TwZ1fvghcHIbtj6VGXB++40vpYLoF0zwtxgp\nKXpgOz5HqlBsT9UVP65n2lm0ajwz4TXtKzCTX9RF+RDqrvNY6oKzehs3lGCmj3XoPeGtSeVQbEkV\nt36Zyn15L3VH+rRxlvVg6g5k//b9fm27LqXqHs0e8rovqz7fEcP6I1E5QJdQgfSrqHoou7VxT2wn\njgldBHrS2J3Krn8V1f3G6WPGr0blEm0/YDpz2knoc1RruPtRd9GnUhe1bUZOtEP+nfZrx/Yb2zZu\n107Yn6ByJ7/LcnLvxhyj61IX39Opu///6hn3IurGYNrqpC1l3U9jTI4UVay8Me1unCp6eg0VqF3T\nxu9P1ROacD2eZe1HWt249vkIWoOC9v1wKjd9o/b9CbRi5AmkN3KzeTlwWM/wN1IX1JHc02Mm+h9q\n55V3MVqvch4VIPwdo/VhJ1pkeyJV5WQk1+nve8ad1IYN9N9vyzqo/Sc/SwWEG1A3VXcAT+zdhnac\nTHRfHUgF3edQ5/bZ7ff5MFUy8ZdzGf0HzDtTuWnrUoHY+4C927h3tWProe37urRixQms+8j2P6zt\n9zdQ58Unt//8I6lz/kuBV09g+RtSN0YbtO+L239x5/Z9D6ok6OWD/t7T8Zr2FZipLypb9uXUxeU3\nVP81I+NeQF0815zC9Xke8A+M3lW8htE77cXUndwyi1+oYoCnU1n336bVR2jjvkwVe0zoT7qUtMar\nz/dlBqjPx30Dg7nthLMzFeS8uf1hX0fVf5hwVj73zd3YhspOP73tv4uo4rC3tX0/oYtN7/YweuF6\nGVVnZCF1UTiFyi38JbDXEH6fBzFa121nqkL6o6icwYuoAPEBVJD4CkbrroxX3DkS2BzTjtWTqPo8\ne7Rh3wIeORX/l3HWcyHV+m9tKqfk+8BJbdyuVPcOz6Nual5I5QT9F5Wj8UyqKP87k7EtVNB8Q/st\njqcudke23//wNs3G/fwe42z/oxkNnl7Y0uut0/nQAY/lkQv3F6nz54Y902xJ1eVcvCLb0LPMkTT+\niTrP/C11sza7/aazqBuMCQWwY9LcmtHg/ECqnuaZPcf576iqFRNubNGWtR11rtycOkdf37ZnTtu+\nd7CC9WupnL9fAA+hziOrASdQJQQjwdrbqXPNhOsIj/0NqVzB7wKLxww/mioKnVBxJBUwX8PoDcTx\nLZ2Rqj77UqUp6030vzFdr2lfgZn4aieubzJ6MTuNunveggrgvjuMk0Af6zFyYnoBFRzM7xl3AvBr\n6u73BpaTw0fVF7qE0WLPZwL/0048T23jBq4r1pY9afX5uG/dnN5ch1dQFYdHiiae1v7UI60WJ1IB\neuRisC9VrBlUscpHqaz8V1LFkIfSikAG3G9bAdcyWun2RW0/7thOsI8GHjuEdOZTdTpG6tTt17Zh\nl/bbzGvDt1zB5T6e+wY2X6UCzEXtQvCvTGOQ1vN7PpFqZfxF6k5/Zypw+BaVc/Gtts6bUQHZt9p/\n5O1U443HUHf3y6xeMMC6bUDddDypHb/vooLd1aicqfMHTbf9569u/5lvM5oj9IJ2bB8+wLJ7b2zW\n6/n8aeCrY6bdnBWokzZm3q2ogOyT7T/ySUZLOo4FnjrE3+YRtKomPd/PpxU/TnQbxvzu67X98Szq\n3PzfjBZJ7rC0efpY9oZUXdrt2nH1Yep8NYvRnLWReplvYYJ1LLnvTfNLqOLVE6lz5Q8ZbUT0GKpr\npIHqPlM52dczGqy9hAr+RoqhZ0xx5322a7pXYKa9qHoOF1EX6A2pOmgnU339nEddtCc1SKPngkzl\nbnxwZBj3DVae3k64jxgz/xxGczeeQwVKnxkzzaFULsGXGELxQFvmpNXna8t4SdsfG7ff4iM9f9hT\nqIvN0O6kqIv6/9LTBJ5qBXgOlds6lOC2Z9n/QAUDI/W4Xtj2555jppvwNlIX/H+niswWUYHKLVTu\n40hrqn2pIKavStjteOsNbBZRuR292zLhHMch7t9dqIBxPlV/5hVtOxdSuRcbUEH+G6mi5odQQeZz\nqADgXuAfJmndnkzlnFzAaPcDe1G5tm9owwY63qhc4avadh1NBWw/pnUjQ90cDNyIqO2rj1H1kw5q\nwy6ip47qBH63I9rnl1AB5jupQOTnjFaSfw4VHEw4N5CqW7kBFSyfRlV/+Srw/J7p3gWc0D6PdIc0\n0dzNfajczH3a7/EdRuvT7kY19NhyAstdhyquPZ+q1/ZiqoHMflSw9mLqHPqkIR2/L6b64NyM6jbm\nNOpm47sj/xkGqCs4Jq2xwdqrqeoAa070d5ju17SvwEx8tRPNt6k7wbdSAcZpVBA31Hpcy0j//fQ0\njW4nvNdy3yBtL5bRbxtVT+DzVHcRX6aKAa4CThwz3QMYUv9PTHJ9PurCOp8q3nwSVWT1Aao+1Jnt\n93nzkH+HExmtaD6H0TvgkZy1CV0QxqSx1Zjf+jVU0DSS4/VihpCT1rP8Dang4w5GL6Jvbifz7ajg\n9Hv0mStB5aQtL7B5HQMWCw1pu4MKeO5h9CZmHhVw/wujlaw3o/oQvJIq/nwhoy0vjxnGb76UdduV\nupl6NdVJ8/Ft+GrURemMQdOlGgJsRlUV2J26gN6fqpP4a1q91SFsy0jx1I5te94FHNvGfRO4aALL\nPJAKKE8e+d9Rgc3LqfPbrdTN0xUM2MqPytW6k/vWf3oadQ5+Z9t31zKEbnGoVp0fAJ7Qvj+Hqte5\niAqav89g3fy8iuq0eSRX6/ltP+1LBWsvYDhdcKxLXbM2oc6Zl7bf6YNt332TIfcx2v4X1zBaZ239\nYS5/ql/TvgIz8UVF5o/tOQie2U4Ik5qt2ntBo+7yv9k+P6WdJA5o3w+j7rSWeYdN1Z36DfCi9n1/\n6q72JZOw3pNan4/Ru9YtqRyRf2I0p+bhVDHwrVRgMJ8h5d5Qd++f5b4tIp9IFVMMoyHEQ6kcszdx\n35ZeHwSuYwh9y/UscyTIfCRVP+jjVI7S/ak6ay+lclg/TgvS6OPulGkIbCawzb3FM2e17Ryp57UV\ndUEbmyu9PVWs9p/ANZO4jg+lWgke1b4/DvhRz/92NQbsa466WH6ayj3crZ0LXtPGHUEFC3sMYVt2\npS7UI+u+djt3fbT3PzzBZe9NVdf4SPu+BlUM+daWxgYMVu91pJrJg6mi1B8z2un0A9vxcB5VZDhw\n0Sp1U/PFdizu3ZP+s6mbzvfQ6qP28z9cRhpbUjf011CZDWu3/+jHGEJd1zFprdH20ZdH1pm6ATie\nCbRI7jPNRVSGymoT3UddeU37CszkVzsAjqHubCa1fs2Yi8lh7f1yWi/LVBB0AZVTdQXjFFdSuWrP\nbgfySIeYO1L1H44c4npPan0+Ri+227dlL6Zy0N5ET/2NNn6ird+iJ52tgUe3z3OpgPfE9vkxVECy\ny5C25xTqiQOfa9v1oDbuSCpI2G2i6Swjzd2pQOxh1I3Ie1r692vj78/EuzGZksBmAtv8lLaf39Yz\n7jQqQN6kfV/qTQSVa3tk+x/Om6T1fAIVGFzc8x/aiSrSG/imiqr/dFHbjhdQRWsfoxrFvJIqQnrM\nBH/zsa1930dVR/jv3vMTdc6acOesPctZBPwf7XzWhn2K1qv+EJZ/MHW+fCh1k/bDnnPBZmOmHaTe\n6/ZU10qvpc6Vf8cQcraWk+4OVPB/DFV15NkMudpGS2d++/9v1/53FzDBwHwF0hxKcep0v0YODE1A\ne77fM4DLM/OHU5TmIVQW/8LM/H1EfA34WWYe3p4ft4DqifkXfS7vIOpC9Soqp/CV1N37j4ewrmtR\nRWZnUkWrh1KNB55D5UCuCbwxM68eMJ2DqPVeA/hpS2trqrn2lzLz62Om/8uz7FYwnQOpwGzkZPN0\nqjXk/lQR35+oC/4nJ7wx3Gd71qJyzn5IBVH/TbUiOwx4dmZePdFtWUqaj6VO0Fdk5ofag8b3oOqs\n/Ak4JTP/34BpbET1ufZSqk7RjYOt9WAi4gCqovRRVFcj1wDPy8zbI+LdVO7V7sA9uZwHaUfE6lkP\nQh/GOs3KenboZsCd7T++NZXjcTf1GLJbI2JnKoD8ygBpPY66SG+YmSe3YYdSDRRuo/bJ7Zm5ZMBt\nWpCZP2gPxd6Kamn4aOrcsHZLb1Fm/nyQdFpaT6FuMM6jgs6TqdaQ1w+43EdTVUUOz8xr27CPUDlf\nH6BuDA/MzG8PmM7If39NqhHYD6j6iZcC52XmrYMsfznpbk/VR35FZp47SWmsQbVY34vKmTwsM38w\nGWmtbAzUBjSsC2WfaT2GOrm9ITPP7xn+Fapu3BMmuNz9qOLC31G9Zg8UOI1Z9rFUbt/NVMBxA3Wy\nfgPwp8y8Z8Dlb0xVfn9+Zl4TEcdTdSHupC5CP6GCp19NYNmbA6/PzBe0E/VHqcBlB+p3+Aa1v25s\nF9Y/ZeZtgxwTLZj5BFVv5wcRcQJVb+x3VL2Rdaki74smsvzlpHsCdRL9APXQ6j+0wH8fKhA9PTN/\nNKS0hhbYDLAOa1DFnO+hLhqLqe5N5lIX9tsjYpuRi/IUrM+W1Pn4xhZAnkJVX1g3M4+NiG2pG5w5\nwGmZ+bM230RvOnZhtAh9I6q+2Ncy808RcSRVd3C3zLxzwOP58VSF9bdQgcAHqE64r6SCz99QPeB/\ndyLLX0aaB1PnhM9Q/WbdMIRlPoKqHzryRJPdqKcczKVuDP8zMz8/YBpjz2Uj//0/UC2LP0v99ncP\nks5y0n8k9ajA6yZj+S2N1amqFPdm5i2Tlc5KZ7qz9Hwt+8Vf9z+zOdVa8krGZE1TRSMTfvgyPY82\nGvI2TGp9PqoRwTcYbX69OqP9fQ3cSo0qhtiSyt3ajArUvtm+X8Dw64qtTxWljXTKuzpVEfdS6kI9\n0jv+oP0yjdykPYTR/tkOoXq2730e7MB1oLry6tnm3kf6PLTt75HH2NzRftcp63SXarRwDZUruwuV\nE7QN1crzNmBJz7H4LgZ8NiGVU/iFkf8Glev0Xqp+5UgDn4E76mWc1r5UlYI1Bk1nGWk/kSEWR1PF\n/ie0bTmEKrl4IZWrPqw6r0s7l72fqo7wMlp/YL5WvddqqJN672IjYs+WpX83VUxwCfCOiNhkZPrM\nPDAzfzrR9DLz9uyzuHQFl/uHzLwC+FUr+jiJqltz15CW/3/UXegeEfHIrJyaj1PFoDtQLbBWWERE\nW/53qZPl1zPzZkYfE/R7qi7Pb6nWsUPRtudC4Mk92/MxKkftkcBzImKtzPzzgOlkROzf0jopIi6m\nWmR+hvqNnhARszPz3sy8c5C0uqJt8+7AiyPiSVlFuXdR3Y9sFRE7UHWa3puTlGsxVjvOFlL7/Saq\nTtpR1M3BM6igap2I+FQ7Fv8+M68ZMNkHUMVpe7fvb6JyE49u6ZMD5na0nLS/a2k9i8pRX5+qC/VI\n6oZttcz84yDpLEtm/kcOsWg9M3+bmadT3VV8gqrL9RLgV7mcYvEVTGNp57KPtbQWUMGuVkEGah3V\nE6S9jDqR7kndBW9CFR/8L/D+iHjQtK3kilmTanV5eGb+z5CXfQF19/5PEXEKVZz1D1QRxSMmssCR\n/d8+7w38LCK+SF1MHxgRb6JaRb4gM78z4PqPdSG1Pe9o2/NeqpjuB9QFfM6gCUTEAqp47XCq9dWG\nVJ2nd1NFbq+jcpxWGhGxkCqyngtcFBHPporIr6WCiiVU45yvjQTqk60dZ1+kepv/H6ovse9RuTan\ntWDjq8CWEbFDZv52CGl+nsoJOiYintkCgpOpBgq3D7r85qb2OpfqLudi4DeZeTbVaOEjg95sTJM/\nR8SOVGOLv8vMy4a8/GWdyx5EBWtaBVlHrcPaxfTkzHx6q3v1tMzcq43bkroD/sCgd79TZTLr80XE\nulSx0UjrwrWpItC9M/O2AZa72sgdc0R8ktH+n3amelP/xKDrvox016GKwh4JfDszv9wq+K+TE6hv\n15bZm0v7UKq/pB9T/ZwdmZnXR8QumfmNiNhikBzarhjZ5ojYlNEHWn86IvYCzqa6jfkCVU9t3cyc\n8lyLVhfwM9Rv/brMPLc1ZriFqtN5HNWlxVDqCPakewCt2DMz/3WYy+5JY6QF4zpU1YqHT0Y6Uyki\n7kc9S/XHk3FOm6xzmWYuA7UOGfunj4j1qboJm1MXmQMz855W2ffTwO9n6F3ppIqIJ1MXhxfmECop\njwnWPkEFS3u377Om4jcYVjoRsStVL+uPVK7AHVSHub9txYKvoSozT0rrsunQGsv8C1X89kvg6My8\nIyL2pCq6v2qyApUVERFbULmZp1C5a8+nfqt/zsyPT1KaT6X6MdyLai0+9GM5OtbadyYZ9rlMM5OB\nWkeMye3YBPhzVsuz91Odth6emT+LiGdROTr7ZWv5pftq+29OZv5kiMvsDdYuAv43M189rOVPpp5c\npV2oYvOrqA6A96CKWd4G/J6qm/aGzPzUtK3skLUcnaOoumerUf03/Z5q2frLlrP258z88jSu5l9E\ntez+KPD2zHx/q4/4+0nOjZ47GfVTl5LOtLf2nWkm41ymmcdArWMiYjHVseLd1IOer6D6IbsTSKoF\n5ZE5xC401J+RYC0inke1KjsmB+xeZKpExE5UR64nZeblrejzKVTx6ppU69XLMvOzkxkUTIWewHQT\nqoXx7VQx793U77Y/1eLwH0caSnRpm1sdqI9TD/a+cZpXR9I0mz3dK7CqG5OTdj+qVeHT2/snqdyA\nI6g6CxtTddZunJ61XbX1tO66gerkeEYEac0DqM5b96C6o/gp1SHw5pn5ypGJuhSwTERPkLYP1YP/\nUVSr3Rdn5tuBL0XEbOAA6tE/d8J9G49Mt8y8KiIelQN2MCxp5WCgNo3GBGnHUC0UZwG/zMyvRMQi\nqvPT17bWUuqAHKA3+OmSmV+IeqrF2yPix5n5bxHxK+CJUR1t3p7NNK/qQFqQtjuVg/a5zPxmRPwN\n8PGIuDcz35mZn4+IKzPzl9O8usszcOtOSSsHiz47oNWTeQv1fMHtqWKPf291aPagehDfnmrePpQ+\ne7RqinpEzUeo5yveC3w4B3xEUFf0FE1/juo5fn6O9uD/SKrT4Hdl5j9N53pK0oowUJtmrXHAq4Bn\nZj278SiqC4irgU+01mlrZXWwKg2stfR7E9WX1T+N9Bk2E3PTetc9ItbMzD+04ZdST1fYs2fa7agn\nZPzH9KytJK04A7UptpQuODanHqnyocw8sQ17FtVz+NeoDiNnfJGUuqXV4ToHODEnqS+4qRQRe1N1\nOe8ELmlVBz5LPVPwwDHTzuh6eJJWLQZqU2hMnbQTqA4uv091l3AJ8NbMPLWNfwbwFTs41GRpwc31\nOYSHVk+1qCdyzMrMW2L0wd8vpx7HtC7w3cz8l4i4gqrzue80rq4kTZiB2jSIiBdTz/H7Gyo37Syq\nG47TgXMy83XTuHpSp0XEfOAi4ClZvcM/C3hIZr6ptZx+AvV8yWNbH2SPz8z/ms51lqSJ8lmfU6w9\nHmQHqpjmECpAm0f1afUy4JkR8cCRujeS/srjgWuATVvjiDuAwyJi28z8XWZeSj3PcyGAQZqkmcxA\nbYpl5m+A44GNqGd37kvlrB1KPRj70Zl5p3VopGW6BFidylWLzPwc9bD1EyJi54jYhgrUutz9hiT1\nxX7UpkFm/jEi7gJmt5ZoWwKfoypB28mltBQ9dTwfBGxAdWfzsFbc+a9ULvXI80vf4tM7JK0MrKM2\nTSJiDaqocy/gwcBhmfmD6V0rqZt6njiwO3AisBhYH3g2FZi9LTPviogNgT9l5q9t3SlpZWCgNo0i\nYnUqd+DezLxlutdH6rKIeCwVmF2RmR+KiFnUI7H2A/4EnGKOtKSVjUWf0ygz/wTcNN3rIc0Qj6Oe\n0XnrSOe2EXEZ9di1/ambHgM1SSsVc9QkdVJPcedDgJ+3os1DgBcBbwD+OzPviYjVgPUz887pXF9J\nmgy2+pTUSS1I2x+4EDgpIi4GlgCfAU4CnhARszPzXoM0SSsrAzVJnRQRC4BTgMOBX1Pd16yZme8G\nLgNeB6wzfWsoSZPPok9JnTHmMWsPBfYFfgy8ETgyM6+PiF0y8xsRsUVm/nQ611eSJpuBmqROiYhd\ngYcCf2S0X7THZuZvW/ccrwGen5m3TuNqStKUsNWnpGnX03BgF+D9wFXArcBPgTnA0yPi91TdtDcY\npElaVZijJqkTImIn4DTgpMy8vBV9PoV6tueawHXAZZn5WTuzlbSqMEdNUlc8ANid6sT2cio37UZg\n88x85chEBmmSViW2+pTUCZn5BeAQ4HkRcWTrEPpXwBMjYuOIiDadQZqkVYY5apI6IzM/FRH3Ah+J\niKcD9wInZ+Zt07xqkjQtzFGT1CmZ+WngWcDDqOd6LolmmldNkqacOWqSOqcFZ38AzomI6zPzE9O9\nTpI0HWz1KamzImJv4PrMvGG610WSpoOBmiRJUkdZR02SJKmjDNQkSZI6ykBNkiSpowzUJEmSOspA\nTZIkqaMM1CRJkjrq/wNov2ouyvjFGAAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_discriminative_features(alpha_filtered,reverse=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Many of these words seem to match our intuition for negative reviews."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Naive Bayes Prediction\n",
"Given a trained NB model, how do we predict the class of a given text? \n",
"\n",
"Like in [MT](word_mt.ipynb) and [parsing](word_mt.ipynb), search for the $y\\in\\Ys$ with maximum *a posteriori* probability:\n",
"\n",
"$$\n",
"\\argmax_{y\\in\\Ys} \\prob_\\params(y|\\x) = \\argmax_{y\\in\\Ys} \\frac{\\prob(\\x|y) \\prob(y) }{ \\prob(\\x) } =\\\\ \\argmax_{y\\in\\Ys} \\prob(\\x|y) \\prob(y) \n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"What to do with words outside of the training vocabulary? \n",
"* Ignore them, assuming that unseen words are equally likely in both classes"
]
},
{
"cell_type": "code",
"execution_count": 231,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"from math import log, exp\n",
"def log_prob_nb(theta, x, y):\n",
" \"\"\"\n",
" Calculates the log probability log(p_theta(x|y)).\n",
" \"\"\"\n",
" alpha, beta = theta\n",
" result = util.safe_log(beta[y])\n",
" for w in x:\n",
" if (w,y) in alpha:\n",
" result += util.safe_log(alpha[w,y])\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": 232,
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"('neg', 'neg')"
]
},
"execution_count": 232,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def predict_nb(theta, x):\n",
" \"\"\"\n",
" Finds y^*=argmax_y p_theta(y|x)\n",
" \"\"\"\n",
" if log_prob_nb(theta, x, 'pos') > log_prob_nb(theta, x, 'neg'):\n",
" return 'pos'\n",
" else:\n",
" return 'neg'\n",
"\n",
"i = 0\n",
"predict_nb(theta_filtered,train_filtered[i][0]), train_filtered[i][1]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Evaluation\n",
"Use **accuracy**, the ratio of the \n",
"\n",
"* number of correct predictions and the \n",
"* number of all predictions \n",
"\n",
"$\\y^*$ is predicted sequence of labels, $\\y$ the true labels:"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"$$\n",
"\\mathrm{Acc}(y_1,\\ldots,y_n, y^*_1,\\ldots,y^*_n) = \\frac{\\sum_i \\indi [ y_i = y^*_i]}{n}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 233,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def accuracy(data, guess):\n",
" correct = 0\n",
" for (x,y),y_guess in zip(data,guess):\n",
" if y_guess == y:\n",
" correct += 1\n",
" return correct / len(data)\n",
"\n",
"def batch_predict_nb(theta, data):\n",
" return [predict_nb(theta, x) for x,_ in data]\n",
"\n",
"def accuracy_nb(theta,data):\n",
" return accuracy(data, batch_predict_nb(theta,data))"
]
},
{
"cell_type": "code",
"execution_count": 234,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"(0.98625, 0.805)"
]
},
"execution_count": 234,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"theta_filtered = (alpha_smoothed, beta_smoothed) = train_nb(train_filtered,pseudo_count=1.0)\n",
"accuracy_nb(theta_filtered, train_filtered), \\\n",
"accuracy_nb(theta_filtered, dev_filtered) "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Broken Assumptions\n",
"\n",
"Naive Bayes assumption can make sense, e.g.\n",
"\n",
"$\\prob(\\text{... great ... awesome}\\bar +) \\approx \\ldots \\prob(\\text{great}\\bar +) \\ldots \\prob(\\text{awesome}\\bar +) \\ldots$\n",
"\n",
"\n",
"When is it violated?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"What is $p(\\text{Fiction}\\bar \\text{Pulp},+)$ according to NB?\n",
"\n",
"* $\\approx 1$\n",
"* $p(\\text{Fiction}\\bar+)$\n",
"* $p(\\text{Fiction})$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"|$\\x$ |$y$| \n",
"|---------------------|---|\n",
"|Pulp Fiction | + |\n",
"|Pulp Fiction | + |\n",
"|Pulp Fiction | + |\n",
"|Fiction | - |\n",
"|Fiction | - |\n",
"|Pulp Fiction | - |"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"What is $p(\\text{Fiction}\\bar \\text{Pulp},+)$ according to this **data**? What about $p(\\text{Fiction}\\bar +)$?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"$1$, $\\frac{1}{2}$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Therefore NB will **underestimate** \n",
"$$\n",
"p(\\text{Pulp Fiction}\\bar +)\n",
"$$ \n",
"\n",
"which should be\n",
"\n",
"$$\n",
"p(\\text{Pulp}\\bar+,\\text{Fiction}) p(\\text{Fiction}\\bar +) = p(\\text{Pulp}\\bar+)\n",
"$$\n",
"\n",
"but is \n",
"$$\n",
"p(\\text{Pulp}\\bar +)p(\\text{Fiction}\\bar +)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"This means a positive instance with \"Pulp Fiction\" may be **misclassified**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Bigram Representation\n",
"\n",
"Problem can be partially addressed by using a **bag of bigrams**\n",
"\n",
"* Count bigrams (like Pulp Fiction) and not just single words"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"\n",
"Turn\n",
"\n",
"* like, Pulp, Fiction\n",
"\n",
"into\n",
"\n",
"* (like,Pulp),(Pulp,Fiction)\n"
]
},
{
"cell_type": "code",
"execution_count": 235,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"def bigram_dataset(data):\n",
" return [([tuple(x[i:i+2]) for i in range(0,len(x)-1)] + x,y) for x,y in data] "
]
},
{
"cell_type": "code",
"execution_count": 236,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"[('paul', 'verhoeven'),\n",
" ('verhoeven', 'dutch'),\n",
" ('dutch', 'auteur'),\n",
" ('auteur', 'dragged'),\n",
" ('dragged', 'violent'),\n",
" ('violent', 'sexually'),\n",
" ('sexually', 'aggressive'),\n",
" ('aggressive', 'aesthetic'),\n",
" ('aesthetic', 'american'),\n",
" ('american', 'film')]"
]
},
"execution_count": 236,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_bigram = bigram_dataset(train_filtered)\n",
"dev_bigram = bigram_dataset(dev_filtered)\n",
"test_bigram = bigram_dataset(test_filtered)\n",
"train_bigram[0][0][:10]"
]
},
{
"cell_type": "code",
"execution_count": 237,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"0.82"
]
},
"execution_count": 237,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"theta_bigram = (alpha_bigram, beta_bigram) = train_nb(train_bigram, 1.0)\n",
"accuracy_nb(theta_bigram, dev_bigram) "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Look at features"
]
},
{
"cell_type": "code",
"execution_count": 238,
"metadata": {
"scrolled": false,
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAngAAAF9CAYAAACXshUUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xm8lGX5x/HPBQgqbrgECKholHslhGiaphAoKpZpWCqa\nSyZkafkTXDPFfUnc0VRwCclKSXFBy8zSFFs0LZO0ctdssbJcr98f1z2e+wxzzpkzM2d7+L5fr3nN\nzLPvz/Xc22PujoiIiIgUR6+uXgARERERaSwFeCIiIiIFowBPREREpGAU4ImIiIgUjAI8ERERkYJR\ngCciIiJSMA0J8Mxsgpk9YWZLzGx6hf5mZrNS/0fMbIu2xjWz1c1skZk9mb4HZP1mpOGfMLPxWfeR\nZvZo6jfLzKwR6yciIiLSk9Qd4JlZb+AiYCdgY2BvM9u4bLCdgBHpcwhwSRXjTgfudvcRwN3pP6n/\nZGATYAJwcZoOaboHZ/OaUO/6iYiIiPQ0jUjBGw0scfen3P1NYB4wqWyYScBcDw8Aq5nZ4DbGnQTM\nSb/nALtn3ee5+xvu/jSwBBidpreKuz/g0Xrz3GwcERERkWVGnwZMYwjwTPb/WWDLKoYZ0sa4A939\nhfT7RWBgNq0HKkzrrfS7vPtSzOwQIiWR/v37j9xwww1bWLXGePS5fzZ8mpsNWbUw8yn6unTWfIq0\nzXrqunTWfLQuy/Z8ir4unTWfztpmjfbwww//1d3Xamu4RgR4Hc7d3cwa9k41d58NzAYYNWqUL168\nuFGTrmi96bc2fJqLT59YmPkUfV06az5F2mY9dV06az5al2V7PkVfl86aT2dts0Yzsz9XM1wjsmif\nA4Zl/4embtUM09q4L6VsV9L3y1VMa2gbyyEiIiJSeI0I8B4CRpjZcDPrS1SAWFA2zAJgv1Sbdgzw\nz5T92tq4C4Ap6fcU4Oas+2Qz62dmw4nKFA+m6b1mZmNS7dn9snFERERElhl1Z9G6+9tmNg24A+gN\nXOnuj5nZoan/pcBCYGeiQsTrwAGtjZsmfTow38wOBP4M7JXGeczM5gOPA28DU939nTTOYcDVwArA\nbekjIiIiskxpSBk8d19IBHF5t0uz3w5MrXbc1P1VYMcWxpkJzKzQfTGwaXuWXURERKRo9CYLERER\nkYJRgCciIiJSMArwRERERApGAZ6IiIhIwSjAExERESkYBXgiIiIiBaMAT0RERKRgFOCJiIiIFIwC\nPBEREZGCUYAnIiIiUjAK8EREREQKRgGeiIiISMEowBMREREpGAV4IiIiIgWjAE9ERESkYBTgiYiI\niBSMAjwRERGRglGAJyIiIlIwCvBERERECkYBnoiIiEjBKMATERERKRgFeCIiIiIFowBPREREpGAU\n4ImIiIgUjAI8ERERkYJRgCciIiJSMArwRERERApGAZ6IiIhIwSjAExERESkYBXgiIiIiBaMAT0RE\nRKRgFOCJiIiIFIwCPBEREZGCUYAnIiIiUjAK8EREREQKRgGeiIiISMEowBMREREpmLoCPDNb3cwW\nmdmT6XtAC8NNMLMnzGyJmU2vZnwzm5GGf8LMxmfdR5rZo6nfLDOz1P3jZvZLM3vbzD5Tz3qJiIiI\n9GT1puBNB+529xHA3el/M2bWG7gI2AnYGNjbzDZubfzUfzKwCTABuDhNB+AS4GBgRPpMSN3/AuwP\nXF/nOomIiIj0aPUGeJOAOen3HGD3CsOMBpa4+1Pu/iYwL43X2viTgHnu/oa7Pw0sAUab2WBgFXd/\nwN0dmFsax93/5O6PAO/WuU4iIiIiPVq9Ad5Ad38h/X4RGFhhmCHAM9n/Z1O31sZvaZwh6XelaVXN\nzA4xs8VmtviVV15p7+giIiIi3VqftgYws7uAQRV6HZv/cXc3M691Qeodv53zmg3MBhg1alSnzFNE\nRESks7QZ4Ln72Jb6mdlLZjbY3V9I2acvVxjsOWBY9n9o6gbQ0vgtjfNc+l1pWiIiIiJC/Vm0C4Ap\n6fcU4OYKwzwEjDCz4WbWl6g8saCN8RcAk82sn5kNJypTPJiyc18zszGp9ux+LcxTREREZJlVb4B3\nOjDOzJ4Exqb/mNnaZrYQwN3fBqYBdwC/A+a7+2OtjZ/6zwceB24Hprr7O2mcw4AriIoXfwRuS/P8\nqJk9C+wJXGZmpXmIiIiILFPazKJtjbu/CuxYofvzwM7Z/4XAwmrHT/1mAjMrdF8MbFqh+0M0z74V\nERERWSbpTRYiIiIiBaMAT0RERKRgFOCJiIiIFIwCPBEREZGCUYAnIiIiUjAK8EREREQKRgGeiIiI\nSMEowBMREREpGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBERERECkYB\nnoiIiEjBKMATERERKRgFeCIiIiIFowBPREREpGAU4ImIiIgUjAI8ERERkYJRgCciIiJSMArwRERE\nRApGAZ6IiIhIwSjAExERESkYBXgiIiIiBaMAT0RERKRgFOCJiIiIFIwCPBEREZGCUYAnIiIiUjAK\n8EREREQKRgGeiIiISMEowBMREREpGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRgqkrwDOz\n1c1skZk9mb4HtDDcBDN7wsyWmNn0asY3sxlp+CfMbHzWfaSZPZr6zTIzS92PNLPHzewRM7vbzNat\nZ91EREREeqp6U/CmA3e7+wjg7vS/GTPrDVwE7ARsDOxtZhu3Nn7qPxnYBJgAXJymA3AJcDAwIn0m\npO6/Aka5++bAjcCZda6biIiISI9Ub4A3CZiTfs8Bdq8wzGhgibs/5e5vAvPSeK2NPwmY5+5vuPvT\nwBJgtJkNBlZx9wfc3YG5pXHc/cfu/noa/wFgaJ3rJiIiItIj1RvgDXT3F9LvF4GBFYYZAjyT/X82\ndWtt/JbGGZJ+V5pW7kDgtpYW2swOMbPFZrb4lVdeaWkwERERkR6pT1sDmNldwKAKvY7N/7i7m5nX\nuiD1jl9iZvsAo4DtWpnXbGA2wKhRo+qep4iIiEh30maA5+5jW+pnZi+Z2WB3fyFln75cYbDngGHZ\n/6GpG0BL47c0znM0z3rNp4WZjSUCz+3c/Y221k1ERESkiOrNol0ATEm/pwA3VxjmIWCEmQ03s75E\n5YkFbYy/AJhsZv3MbDhRmeLBlJ37mpmNSbVn9yuNY2YfAS4DdnP3SoGmiIiIyDKh3gDvdGCcmT0J\njE3/MbO1zWwhgLu/DUwD7gB+B8x398daGz/1nw88DtwOTHX3d9I4hwFXEBUv/khTWbuzgJWA75rZ\nr82sFESKiIiILFPazKJtjbu/CuxYofvzwM7Z/4XAwmrHT/1mAjMrdF8MbFqhe4tZySIiIiLLEr3J\nQkRERKRgFOCJiIiIFIwCPBEREZGCUYAnIiIiUjAK8EREREQKRgGeiIiISMEowBMREREpGAV4IiIi\nIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBERERECqZPVy/AsuBPp08s1HxERESk\ne1MKnoiIiEjBKAVP2k0phSIiIt2bAjzplhREioiI1E5ZtCIiIiIFoxQ8WaYppVBERIpIKXgiIiIi\nBaMAT0RERKRgFOCJiIiIFIzK4Il0ApX1ExGRzqQAT6QgFESKiEiJsmhFRERECkYBnoiIiEjBKItW\nRNpFWcEiIt2fUvBERERECkYBnoiIiEjBKMATERERKRgFeCIiIiIFowBPREREpGAU4ImIiIgUjAI8\nERERkYJRgCciIiJSMArwRERERApGb7IQkW5Hb8sQEalPXSl4Zra6mS0ysyfT94AWhptgZk+Y2RIz\nm17N+GY2Iw3/hJmNz7qPNLNHU79ZZmap+6Gp+6/N7D4z27iedRMRERHpqepNwZsO3O3up6fAbTpw\ndD6AmfUGLgLGAc8CD5nZAnd/vKXxU3A2GdgEWBu4y8w+4O7vAJcABwO/ABYCE4DbgOvd/dI0z92A\nc1M/EZGKlFIoIkVVbxm8ScCc9HsOsHuFYUYDS9z9KXd/E5iXxmtt/EnAPHd/w92fBpYAo81sMLCK\nuz/g7g7MLY3j7q9l8+wPeJ3rJiIiItIj1ZuCN9DdX0i/XwQGVhhmCPBM9v9ZYMs2xh8CPFA2zhDg\nrfS7vDsAZjYVOBLoC+zQ0kKb2SHAIQDrrLNOS4OJiIiI9EhtpuCZ2V1m9tsKn0n5cClFreZUs3rH\nT9O4yN03ILKJj2tluNnuPsrdR6211lr1zFJERESk22kzBc/dx7bUz8xeMrPB7v5Cyj59ucJgzwHD\nsv9DUzeAlsZvaZzn0u9K08rNI8rqiYiIiCxz6i2DtwCYkn5PAW6uMMxDwAgzG25mfYnKEwvaGH8B\nMNnM+pnZcGAE8GDKzn3NzMak2rP7lcYxsxHZPCcCT9a5biIiIiI9Ur1l8E4H5pvZgcCfgb0AzGxt\n4Ap339nd3zazacAdQG/gSnd/rLXx3f0xM5sPPA68DUxNNWgBDgOuBlYgas/elrpPM7OxRDm9v9MU\nOIqIiIgsU+oK8Nz9VWDHCt2fB3bO/i8kmjSpavzUbyYws0L3xcCmFbp/pT3LLiIiIlJUelWZiIiI\nSMHoVWUiIh1MDSqLSGdTCp6IiIhIwSjAExERESkYBXgiIiIiBaMAT0RERKRgVMlCRKQAVJFDRHJK\nwRMREREpGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBERERECkYBnoiI\niEjBKMATERERKRgFeCIiIiIFowBPREREpGD0LloREama3nkr0jMoBU9ERESkYJSCJyIi3YpSCUXq\npxQ8ERERkYJRCp6IiCyTlFIoRaYUPBEREZGCUYAnIiIiUjAK8EREREQKRgGeiIiISMEowBMREREp\nGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBERERECkbvohUREelAeuet\ndAWl4ImIiIgUTF0BnpmtbmaLzOzJ9D2gheEmmNkTZrbEzKZXM76ZzUjDP2Fm47PuI83s0dRvlplZ\n2bz2MDM3s1H1rJuIiIhIT1VvCt504G53HwHcnf43Y2a9gYuAnYCNgb3NbOPWxk/9JwObABOAi9N0\nAC4BDgZGpM+EbF4rA18BflHneomIiIj0WPUGeJOAOen3HGD3CsOMBpa4+1Pu/iYwL43X2viTgHnu\n/oa7Pw0sAUab2WBgFXd/wN0dmFs2z5OBM4D/1bleIiIiIj1WvQHeQHd/If1+ERhYYZghwDPZ/2dT\nt9bGb2mcIen3UtMysy2AYe5+a1sLbWaHmNliM1v8yiuvtDW4iIiISI/SZi1aM7sLGFSh17H5H3d3\nM/NaF6Se8c2sF3AusH+V85oNzAYYNWpUzcssIiIi0h21GeC5+9iW+pnZS2Y22N1fSNmnL1cY7Dlg\nWPZ/aOoG0NL4LY3zXPpd3n1lYFPgnlTnYhCwwMx2c/fFba2jiIiISJHUm0W7AJiSfk8Bbq4wzEPA\nCDMbbmZ9icoTC9oYfwEw2cz6mdlwojLFgyk79zUzG5Nqz+4H3Ozu/3T3Nd19PXdfD3gAUHAnIiIi\ny6R6A7zTgXFm9iQwNv3HzNY2s4UA7v42MA24A/gdMN/dH2tt/NR/PvA4cDsw1d3fSeMcBlxBVLz4\nI3BbnesgIiIiUih1vcnC3V8FdqzQ/Xlg5+z/QmBhteOnfjOBmRW6LyayY1tbru3bWHQRERGRwtKb\nLEREREQKRu+iFRER6eH0vlsppxQ8ERERkYJRCp6IiIhURSmFPYdS8EREREQKRgGeiIiISMEowBMR\nEREpGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBERERECkYBnoiIiEjB\nKMATERERKRgFeCIiIiIFowBPREREpGAU4ImIiIgUjAI8ERERkYJRgCciIiJSMArwRERERApGAZ6I\niIhIwSjAExERESkYBXgiIiIiBaMAT0RERKRgFOCJiIiIFIwCPBEREZGCUYAnIiIiUjAK8EREREQK\nRgGeiIiISMEowBMREREpGAV4IiIiIgWjAE9ERESkYBTgiYiIiBSMAjwRERGRglGAJyIiIlIwCvBE\nRERECqauAM/MVjezRWb2ZPoe0MJwE8zsCTNbYmbTqxnfzGak4Z8ws/FZ95Fm9mjqN8vMLHXf38xe\nMbNfp89B9aybiIiISE9VbwredOBudx8B3J3+N2NmvYGLgJ2AjYG9zWzj1sZP/ScDmwATgIvTdAAu\nAQ4GRqTPhGx2N7j7h9PnijrXTURERKRHqjfAmwTMSb/nALtXGGY0sMTdn3L3N4F5abzWxp8EzHP3\nN9z9aWAJMNrMBgOruPsD7u7A3BbmKSIiIrLMqjfAG+juL6TfLwIDKwwzBHgm+/9s6tba+C2NMyT9\nrjQtgD1S9u2NZjaspYU2s0PMbLGZLX7llVdaXjsRERGRHqjNAM/M7jKz31b4TMqHSylqXuuC1Ds+\n8ENgPXffDFhEU8pgpXnNdvdR7j5qrbXWqmOWIiIiIt1Pn7YGcPexLfUzs5fMbLC7v5CyT1+uMNhz\nQJ6aNjR1A2hp/JbGeS79Xmpa7v5q1v0K4My21k1ERESkiOrNol0ATEm/pwA3VxjmIWCEmQ03s75E\n5YkFbYy/AJhsZv3MbDhRmeLBlJ37mpmNSbVn9yuNkwLEkt2A39W5biIiIiI9UpspeG04HZhvZgcC\nfwb2AjCztYEr3H1nd3/bzKYBdwC9gSvd/bHWxnf3x8xsPvA48DYw1d3fSeMcBlwNrADclj4Ah5vZ\nbmn4vwH717luIiIiIj1SXQFeyhbdsUL354Gds/8LgYXVjp/6zQRmVui+GNi0QvcZwIx2LL6IiIhI\nIelNFiIiIiIFowBPREREpGAU4ImIiIgUjAI8ERERkYJRgCciIiJSMArwRERERApGAZ6IiIhIwSjA\nExERESkYBXgiIiIiBaMAT0RERKRg6n0XrYiIiEhD/en0iV29CD2eUvBERERECkYBnoiIiEjBKMAT\nERERKRgFeCIiIiIFowBPREREpGAU4ImIiIgUjAI8ERERkYJRgCciIiJSMArwRERERApGAZ6IiIhI\nwSjAExERESkYBXgiIiIiBaMAT0RERKRgFOCJiIiIFEyfrl4AERERkc72p9MndvUidCil4ImIiIgU\njAI8ERERkYJRgCciIiJSMArwRERERApGAZ6IiIhIwSjAExERESkYBXgiIiIiBaMAT0RERKRgFOCJ\niIiIFIwCPBEREZGCUYAnIiIiUjAK8EREREQKxty9q5ehS5nZK8Cfu3o5MmsCfy3APIo2H63Lsj2f\nIq1LZ81H67Jsz0fr0nHWdfe12hpomQ/wuhszW+zuo3r6PIo2H63Lsj2fIq1LZ81H67Jsz0fr0vWU\nRSsiIiJSMArwRERERApGAV73M7sg8yjafLQuy/Z8irQunTUfrcuyPR+tSxdTGTwRERGRglEKnoiI\niEjBKMATERERKRgFeNIjmJl19TIUSVdvz66ev4gUj64rzSnAW0Z11IlgZiuY2SoNnubqVQ6n47kN\nZjbGzJbzri9827eL51+3nnAzMbM1umCeOg/bqSccS7Uws+XSd4eun5mtaWZrNPq61tP3i07EZYyZ\nHWhmawLLd8C0VwC+AZxmZl9r0DS3A64DTjaz/VsZ7lDgmEbMs7OZ2a7pu3cHz+eDwGTgVjMbZ2Yb\ndOT8WlmOlYC5ZvZZM9ushWEGmdksM9vczIambl16vTKzY83sy2a2DYC7e6OXycwmWijdGGuevpnt\nDpxkZtPNbMWO3n5mtq2Zrebu73bkfDqSmW1mZv3Sb2vpBt/oG38pMDGzT5nZmEZOu6uY2S7At8xs\nHWDFDpyPAXsBN5jZLma2RQOm+Vkz6wP0Sf8bfr/sDKpFu4wxs7OAVYEXgHvd/e4GTXcV4HWgP7AK\ncBWwGLgHuLOWi366WfzDzIYD7wO+DXwXOMfd/50N9wngWODT7v5avevSmcxsZeBq4GngeeC77v5M\nB8xnG3e/L/3eH1gfGAzMdfefNnp+rSxHb3d/x8zGAtsBawD/cvejKwx7AvAmsC1wprv/xMysq1If\nzWwTYAIwDBji7num7nUvU7pJrQh8B/g1cR6d6u6v1jL9dD6+DawGzAL+CPwO+I67v1HPsrYwv+HA\nIcS+OhtY4u6/zfr36u6BX3rguRD4FXFjv6B0LprZF4j9fi/wsLu/1qD9PgT4m7v/18zGAccDE939\nX/VMtztI6/Z54EPEsXevu9/b4Hm8z91fTr93AzYCPgwscvcra5zmcsDlwL+Ia/ILwGvu/v3S9asx\nS98J3F2fZeADHJj93gLYmzjp9mnAtFckUu4OAQalbqsC04HzgF1IDxPtmObuwCLiRlrqNhS4Azg5\n67YBMAd4AOifurVrXl24T8bmv4EZxM19owbP5wPADcC0rNv6wKHA94DRnbS+A4CfAutlx8j6wI+A\n+anbOsA3snFWAj4LvATsmrr16uT9dAbw8fS7H5G9fAPwE6BfI5YJ2Cp99yVuUjOBJ4H12zt9YGfg\nHGDD9L8/sB/wLeBrwHIN3j7jst+fBk4gHsZ2Id7huXJn7q8a12FEdowOTteZv6fttQPwIHABceM/\nEhiQhq/5WpOuZxcABwGfARYAM7p6WzRgW+4LHJP9/ygwNV1rJjRwPpaO8+uzbgPStfR+4OAapvmp\n7Pe2wDTgReDRrt6uNW2jrl4AfTppR0dq2tyybtulG+eeDZj+HsST+57ZTW9F4CjgzPZc5IkXKQN8\nE5gPrAz0Tt3WJoK5GekGMg44CTgXOAxYLQ3XrYM8YC3i5dVHlHX/GvAs8IEGzmsF4OPAFcCWWff3\nAV8lUg1W6qT1PpF4sFi7rPtPgPOJgO4JIpU27z8pHas7dsG+OpBIAduyrPu1RKpE6X9NxxwwHHgN\n+HxZ928AvwcGt2f6wCjg/9J58b7UrQ+wWzpPPtHAbTMcuCYPTIjAZV/gNuCR8n3d3T7A1sAvga3z\n7Qzcmo65n6XrjgGfStvwCGCNOudrwBTgZCIH4o50jg7s6m1Sxzr1IgKji4D/y7qvAexP5Oxs2sD5\nrUEE3WeVdd8e+AGwTTum1R/4MzAz/V8hfe8L/Ld8Hj3h0+ULoE8H72Dom/3+PimYI6UIAOPThWXz\nGqffO/s9BhiWfpcukqukE+3YKqf3GeB6YGT6v33Wr7TM6xJBwt/SxfHBdOE4Gvgi6em6u36y9diA\nCGa2ym/eRFB8E7BmnfPpXfZ/3QrDbAlcCazTweucHydfKl3kaQrchxBZ1Z9IF9rvEql5vbNjaU8i\nuN+gk/ZTn+z3ZFIwXtb9GiIrta7tAowGngI2L5v+TCI1YtUqpjU4+z0eGFPWvy8RSJzewG20HJFC\ncxGwbdZ9M+BR4GLSQ1d3/hApnHeVjrnUrV+6Nv4vv34RuQuzidSdmlJus2P6C2ketxFB+Y+Ih9f3\ndfU2qWOdliPuBbuV9R9KPOB9Oh++nnml3yuRcgWybv2BLwMHpP+t7iciMO2brsW/SufJEcDk1P8C\nIkW9Q6+TDd8nXb0A+nTgzm26UK0InNjCMP2IVJxd0v+qT7rshN4R+ApNgUupe+n/YKJs0fAqp3sa\nkeo3KptGaV16p4vwFaRsZyIofCR1OyldNLtlCl6+Hul79QrDDEgXwjENmE8vIljMU+56lV0gjweu\n6MB1Lh0PW+UXyLJl6AscnvbfYa1M42tkQX8n7Kc+pJSdFvpvRKQ0965jHqXvQfk+yn6fDExqY1pD\niJTGzxNFFnbIt122/VYEfky6yda7fbLjKQ8u103n425E0N6wFJuO2MfZ/02z319Ky34S8BDwHClY\nSP13oc6UtrSvfkWUGbsgnYfHA7cDp1DnA15Xbsuyfvl5vjsRSLf5wNLW9IBdga/l3cvmtR1RJKTV\n7Zidfx8A7iQSGO5Ox8C9RM7CXsCrwAnl69SdP6pFW2AehdkNOJXIjgOa1wDzKHD9ErB/KkDq7Zi+\np5pSFxJlFN7Nuvd293dTzb2/EzeV/7Q0rbIafmsRhdnPBsaUFWydSqTujAPGm9ny7n4jcZMdT5T5\nubk969FZsgoGvYDPmNmWpOZCyvbJ34nKBXvWOq9s398N/MHdf5H1ezfto9I2PwO4uaNqWaZ57URc\nOIfk3bPB3gLuI4LAj5Q6lrZLNuzrxH7uMKnwfGk/3QVMKK/hnB2PLwLvJ1Kx2jOP/Fi4IRWwfzX1\nK6+Q8AxRUL01/yRucLOI1J8fZfPxtA/6uPvrRNDyt/Ysb0vLbmYTiCzfF0r9iAesz7r7AqJoyDGl\nWsHdRdk67GtmI0nXJzPbkyjuMYt42Po9EeQdaql1AHe/xd1fqnMxPkiUH/s18eDyb2I/P0ik5Ha7\na1gl+fliZteb2ZkWrRq8N0jph7vfRARRH1lqQlXK7jvfBB7Lu6d+pWvGT4js2yGVp7TUebgVca/a\nFZgHXEpca94iiiKsBOxnZn274/2loq6OMPXp2A9R7uFVYLP0v2IBa+Jpu387p70ake27Ufq/HXAW\nqWIEzZ+mNqCVcl6krDgiePsmMIgoqD2LSO434ulvLvGkdSaRLbQ9KUuLKKe1fldv8za2WS/iqfAc\noobxuWRZymXbbAqwfB3z2hK4OP3emkgl+CZlT89pmTqsDB5RkeIxmrLdNyRuZPl6982OoV8An2ll\neh+vZ7u0Y7mvA05Lv1cgKietW76viPJZS6XEVjF9I8p5nZZ1W77C9JcnAo4VK0wjT0nbjLh5XkCW\nQkdZakNa3rqKMaRj5u50PP0hnYulMkt9smXvBUykhhTOTti/vYCFadkvI8pUrkQ0t/T10nFJpKr9\nFNiGeFBdrXyb1jj/3YnU9U2ybouJrNoek3qXLfu3iGv2rsBvaF7R4r1cA+L63d57TX5d7Eekrm5H\nZMXuSKR4fiibR+l7IFkxpfL9ny3bvcSD7s+JbPLfAPuXpkUEifsAXy8d5z3ho2ZSCqa8Gndqg+gM\noizcvu7+tzyFoPQ7Pd2/XcP8LiAufI8QT8BrEifVOHd/s4rxZwGXu/ujZrYakZV7vLsvtmhXbzbR\nPMFFRFD+elRrAAAgAElEQVS0yN0PTO0SHUtcbBcAP65l+TubmX2ZuECcaWaPAOe6+9VmtoK7/zcN\nU9onfavZhtm0y/f9+4FbiEobvwf+Qtxsz3f37zdyvdpYrjWJm+bLROrsaKIJgmvc/TupeYqZRNb8\nI0QAtw9xo/uHl66yHdxESnnKmZnNAN4lKvkMJQLVR4gKBTU1Y5GlGBgRaJ1HlB1dG/gckVpwi7vf\nlS9TpWZGsmn1Jh6MjiYq7kwkHnxucfebzGwP4vyoOdWuwnocB7zr7qea2f3AVe4+O+tv2X5bybNm\njbqLdC72IQKTXxLnxdUpBW8foizc79zdzeweorbrs+7+vwbNfzWivC1EULECUe5rP3d/rhHz6Ehm\ntjWw2N3fNLPziXvMNHf/j5ltRKSC3eDup9Y5n+WJh8Ofpem+jzhXjKbiAB8CfuPuR7U8pRanfzAR\nUJ9mZg8T5UZfJnK+jiHKYG7j7idaNBL/Vj3r05mURVsg6SZQSir/mpkdRqSMnUgUeD7PorXvd7Nk\n7FK2apvBUWkcM9vGzD5n0QjxdOJp6nx3P4y4ybyQ5luNfwB3m9km7v4P4kK7nZkNTgHPicTNz4my\ngjuZ2d7pInsScUMcTzd9M0KFbM+XgSFm9hPg6nRDWYsIkoFm+6TdwV3a96eb2TSihtkORLMj09z9\nTKIs0Wp1rlZby1I6TtZNQearRC3UdYkb2Y5E1udmZrY2kQ12P5ElfxZRvqk3sEq6ufaCpbJ0G73M\npSIFZmYfTDeV3xIPK0aUg/sK0bRLTcuRn5+kcpFE9uvtRO3M54gs30GlcbJjYak25LJA8UfA0+7+\npEf2/s1ESsTuZvYEUfmhruCuwnH8O+Af6Tj+gbvPTttt97Rs722j7hLcVViH/xDlgxcSDxtXp6DL\niBTnzwNjLdpX6088bDQkuANI17uLiOvlsUQZ1CN6SHD3MWBodo16jkhB3ioFQb8jmuL6qqWG3Ouw\nOrCJmf2AOLZ/TTwM3gtMT0HdMcBIizdatNoItZlNM7M8EHwFWM/M7gOudfdvE+fP92l68JwP0JOC\nO0BZtEX7EBenO4nAax5xEgwjUh/OIFJ0as6OI7JyHyLagroXOCjrtztx8n2qiulMJ2U5EU+xrxDZ\nuJsTNWKPJ1IhrgJOycabSDyx7Z3+9wHW6urt3sI65tlnpdrFmxMXj7OyfrfQgCr4xAPbLUTzGocR\nwXOpXcJhxAXr2k5a912IAuTXE9mQw2jKNhlJBPKTiayxg7JlHAP8kAh05tEJbd5ly1XKdryCaINx\nPM1roV8NzG7A/M4jteVIPLxslvW7kSxrq7XjigiCtwS+lf6PIYo0HJG25YY0b/+y1mZc8uN4VaKW\n5I7An4Cjsn43Uda8TXf5lK3DWUT5tw8CjwNXZv1uTefP2kQt2TvTefOhDl6+/nRSU0UNWt7SOfMl\nUntz6bhbQFSOe692fIPmtz+Rmn5t1q2UxfpJ4oFjl3ZO87z0PZxopuncrN8ConLFWnTT+0s1nz5I\n0XwaeMjdTzezO4Eb3f2ZlCJxKVEzs6YnajNbkSj4P444qfoBt6QC1CsSN5gT3H1BFdlpzwH/M7OV\n3f0si9fC/Iy4OJxF1Iw9nMgSOS7N39z9VjN7F5htZm+7+3eJ4LDb8abUmjuBd83sn0SqwE3Aumb2\nfeIm/VevIWuhgm2JigrnEoHeCe7+osXbMpYDHnH3b0DHvlkgZaN8jQiQtiNSv/7l7m5mHybKbZ3k\n7jdbvLZstJm9SWQpPmBmexPHwWSi7EvD3+yRy47TWcACdz/fzF4lysO9aWYDiWPR3P2QtI41ZReb\n2USiIdaFad7Pm9mLaR4XEdupxSytslTAmUTW+3gzux1YQgTGnwGedPdbUv+69nc2v/uIbH4nmiM6\nG9jVzIYRFU1edveGvKKwkczsaOAvZnYjEcSvS1wz/gWcDuxlZrOJ1LznS+cIcKGZXUns9xYriDVC\nR0+/UUo5BelcNiJBYYuUq3Je6nYicKqZ/cJTamQtx1/pHLN4g8z3iKzSDczsTCI4e8HiDSqfBA53\n90XtXJ3NzezH7v4JM7uaqNB3O1HB7e/u/pV2Tq/76eoIU5/6Pixd1f/DRK3Wh0lVyIlspulkFSyo\n4WmeeMq8mniyuZfUHhlxI/8wTZUdWpw2zdv4mkK8omv19P8YIvB7f/q/YjZsr7LpjKObVqigeWrB\nJOI1WxCpQlcRZbo+QAQwecvp7UqtIsqFlZ6k1yNSPG8kCoIflrr3JQKq/rXOp4b1H0q0QTWVaLeu\n9DaGbdP3J4nsztKy70vUdtuHrLICERi32jxIg5f7aOIB6XaaCtkPJbK0h2fDteetEuXn58rEG19+\nAOyRdd8MmFLNPIib6lHA2en/GKLcYml7ziMVEG/gcbwjkVU1IH3/mKiQMJLIXv58Ncve2R8ioJtM\nZGPvRVNKdl7B5/1Ema498/G6etm76ydt00/Q1JblHkRZ6VKbcd8AjmzQvCYQDxUfSf+3JSrYnZLm\nO5PIKm7PNL9KlEeHaH9wUfq9FpELtVNRjoMuXwB96th5WbtzRLXz9YmUtB8SBUWXT/2/Sw3tnGU3\njEFEeSiAA4ismVK7edsRtehGtnPah6fv84mUgTXS/+lEUvyQbP49os2hsm3Wi8gSP4Pm2WS3phtw\nn7Lx2hvcrUFkw36JyPItBXQLiZS60nFxI3BRJ63zdkQW9DpEkPQwTW9h2JbIWl8/XbQXAVOzaRxP\npDbtn47hwUS2S8Pe6FG2zM3accuOvT/QvG2tu6gxm5PmbRGeSaR6bZ+6HUwEtUvVFm7pWMi280lp\nWx5Y1n8YkdJRdzY8zWsYfpNI/d83638q8ZC3TqVl7A4fUhm69HsXIvt9MlFb9kiiWYxtiHe/1nwu\nLgufdB3bPv3+MdE48+3A91K3T6dj5MAGznMYUcxj27Lu26Rz4A9U8QCY7890PB+eXxOJ++VPK4zX\nbY7lWj+qZNFDpeTrUmWJB4mnptuJ5jC+TmrCJCU5/8fdDyqNV+083N9rb+gmoiLETkQTFucDp5jZ\n2cRJfaS7P9zG8vbOfq8GfMjMPu6RDP574Admtrq7n068c/Q5T2dZ6bu7S9kQpWW9hCjjsyaws0U7\nZ7j7RCKLaGo+rrcj+yLN51Ui9fM0orD4Nan3nsCrZnYb8XT6qrtPTeNVve/bIx0nE4ksxkHu/hei\nuYReREH/rxPb4wbivbe3E9nw4yxqMkIEvk8Bv3D31z3aVdvK3f/Q6OVN506pQsqlwHFmtgMRcD0N\n9DazA83su8CfPQpdv7eu1c7Hm7I2byUqAy0PXGlmE939cuJc2tvMRpeNt1Rt2bJ5n0eUg93KooJO\nyfuAx9x9nzReTdd3a6q5a8Q2WZ9IIf6YmW2WluUYorztAWXL3i3O1bTN7gXuSRU/FhFFF44CNiYa\nqN6bSE0dmY/bnnNxGfIX4HIzO4l4Pd94d58A9DOzGzxq5f+GOM6B2q832XhGVB76aeq+fOr+oLuf\nSAR+N7c1H29qMWLd9PtKoljIV1P/XdNsTywbr1scy3Xp6ghTn/Z/aJ76sB5wXPq9M9Hg6Y7p/wdJ\nLzFP/9ubSvQRIkjYiEhZuYnIjhlAZAvtQFPbZm0+7RAn7JT0+2CaF2r9NvHark7LSuygfWNEW31n\npf9DiMLalwGfbPC8tiAalb2KeIvBBlm/zWleeL/DtmU6Hn7K0q/GmkSUxTs17e8ridTZUlbOJ4lA\nZT4R5JdSW0opXw1/gqb50/xsIjA+kEgNmECkHH6FaJJnRqXx2jm/KUTRg75EQe5vETWKd0r9t69m\neYlg+VQi63sSUbno+rScS71RoY7lzbMu55BSA4liHhcQWWIf7qhjqUH7eB/iAaJUFOI7RAWAfkRW\n9k+Ia2V59nmPT7HpgG2Z32v2JYrQXJJ160OkgA2rd/vRlEK9cmne6bpydDbMWCJRoXdb88v7E6nn\nP6Ipa/YTwHH0wPYG27VNu3oB9GnnDmue7TOfKE9yDSkwAnYiXly+f9l47Tr50gX9cuCerNveRJC3\nD1U2NFt2Q92XSG2aQQSPvyQFp6l/w5L3u2KfZP+vI4LVUs3Z9VPQMJ/mr0Oq512M+xLZcasTKRDf\nIQKV9YATyMqldPSNi8jCv5+mbPbly753IFLnPklkjf2DpkZEhxJZzUu9DqyDl/nMshvVWKKSwlKv\n8KK+MnerEoHGVUTFEogswrfJAqXW9hFNNeMvJwKVRUSK/XJEdv9s6nj1UzafPYlmJ0r/j0/n6+j0\nfxhR9ONCWnjlXHf4EGVRzydu4H2ILL0LiYeNvkR27aM0f31ft1qH7vAhu9dk3SYSRXTy9w7fD3y0\nQfMcTwSMxxPtDo5M/2cTZSh/TXXZsuXn4drAx4js5ZPT+bigfLmLdhx0+QLoU8NOiwv+dKLW35HE\nk/b+NJWTm0QNzTnQ/ImnL5Fatwj4atZ9XyJVb3A7ptuLCAJWJcqI/Zwos3E50dbYdi0tR3f/0Lwc\n5A40PYVeRDwxrpr+fwAYX8d8yi9YW6SL4OVp244hmve4h1QupiOPv/S9atbtIqKczspEtvQpRJGB\n1Yin5xOyYbclAofPd+RytrH9LiJSD0fTVDloHPG6qG1rnEde/nJKulmVjo9zaEpZPwfYp9rlJbIU\nb8j+DyACvh3Ttv5Kg7fV+aSXtxONTf8S2CL9H05WGaQ7fdK1aUb6vVW6Ph5HBMLbEimQR6b/m3f1\n8nbnT9mxPJuoXPcNIlCaSDywXU7UpK65zCdZMyRpHz2azsm5RNnx/sSDxSVE7dyd8uWrctnnERWo\nRqTr0/ZEjtG7wKldva07dD929QLo046d1XTgfh/4Zdb9C0TWz0HAanVOexxRy+hgorD7HulmOC0b\nts3gjkgh2TP9PihdDHYlUpwuIbLD9iRSMvatZZm7y4cI7hamG+HVRBlCiFSDeyh7LVRrF6cq5nNE\n9n9TIvtvdtquy9P8tUcdFiini/zCdPEcTqSSnEZUmDmfaNz4/DTsAUTzI/n4c4kmPXbrhP2Tp3qP\nI1XcIJ7kryGKMpSO/01qnEeeynEbEdzOSedqLyJQupEod3ddpfFa2N+bEkHcH0gpwqnfsdSZSp+N\nV6oI05t4sJtF1H4emrp/NR3bWzZifh20j3ulY/BiUsBLBMDfoinI24ZIXZ/QHdehO37SsXx2unYf\nnc75QUQq6NtUWfO7hWlvmI6rddP/zxAPRVsRD1+l7gPLxqtqnxGNIp9J5GodRzwAj8z679DV27fD\n919XL4A+Veykyqk3/yQ1iJpuBF8gvZ+vjvnsSDxBbUO81P3rxOtnPpVuVl9Nw7V5IhMB3FNpmmuk\naT6cLhanALun4bp1eZ5q9gkRNJTKQU4jexcoUU6q1YZr25hPHjj0J9ruujjrtjWRIvo9UqPGpWOi\nA9d9FJHVsR0RuH87XZTXIiqPTCOyiWfQVHTgp0SAMyAdE1cQwek3Oml/9SKyei4gsspLZSQvILJM\nNykfvsZjYWPSwxDxgHQ1KaAjUhB2bm0flU2rdA6tlLblL2lKWbuZBjRFQaSwXkrUOL2Vptrt5xA3\n2VKQdwJZg+Pd8UMEcVvS9IBl6Vg7j7jB96WDamUX5UPzh6HVaP4wslo6Do5K/z+W9WtvESAjipR8\nhyiusw0RiD1LVNYYkIYbT5T7bO+7a9cgmiIqpaAPS9OZVmHYHlfWu+rt0NULoE8bO6h5IevDSidV\nupG8WrrIpxPm4zXOo3e6OF6SLpDbpIv7sKz/p8gK7Ve5zDvmNyHi5n5BmvbrRMBi+Tg94VO2Ty5P\nN/ELU7cViSzJi6gzC7LsZl96I8UqRIWEvPzYXDopFTRdKOcBl2XdTk3bYMtsf/ajLDWZCELnEtny\nH04X+Mvbe3OocblnEilRqxO1zg/O+l0JfKmefUTTW0R+QaRibpIdD3OJgv3NmmtoZZpGU2B8JKk5\nGeLB6DdE2aE5DdgmpbbFdiQad721rP85aVut1xnHVoP2c3l7maUg72qysludccz1tA9ZO6ZEpbAP\npuM5f5PQ3sClrW3zKuazEVEmeQMihfBvpNS0dH+YSzwsbpuuFW2+oYKyRJDU7UYiBa90vf408O2u\n3s6d+SldjKUbylryNiL1AaLyw3XEiTCCSBm50N1PKh+vHdNf3t3/Z2ZfIQLHTYkXXv/RzA4BXnH3\nH7Rz2ctf3N7H3d82s77Ei6F3cPcz2jPN7iTtk9lENsVPiRTOA9z9WjPrR6Ri/dXd55SGr2afZNN/\n7yXzxIXqfcAP3f2M9PaHh4mL71DirQVfrGU+7WVmg4jU4t2IC/8tqfvZaVm+6O7/LBunj6d3HZtZ\nf6Lg+0eJlJXJ7v5YByxnb3d/J/t/JJEl/DmimYczzWwdInB+sM55GdFW3OtEKtgxxL75obsvsXgD\nzD7uPruVabx3vpjZxcQD1ReIm+xaxOvIXjGzDYHXPZqiqfkNFWb2EaJM1RyiCYyJRBm2o939e9lw\n5xLNLB1fWteOPL4aqez6uZG7P97Vy9TdpevNV4hs+/8zs42JNueeJo7ra4nrzfE1Tn8AEdQdDPyX\neOh7ijhvriKakDqAqJD1N+JBttU3I2X7uReRQ/QqEdAPJ7KWVySKNFwB/N7dT6hl2Xukro4w9an8\nYemmUEo18LYhToSvESlrm5Fl2bVj+qXgfieifTIjCob/iZQSSDS18VtgXIPWqVKWVI9JuStb7sPJ\nytcRBXf/RlPt0D4NmIcR5fiOJ7Ll7yAuVL2Ii9YBNH8XcEc0K1I6TrYmgo6PEgWVpxJP+Xmr7y1m\nf9E89WoNopJQhxR0p3mq2sFEFuduRMWO/B3AC0hvrGjv9iNL8SNSIt8h1cgjUjKvIsrdbVhpe7ax\nrbcjXtu3H1Ee7lnKUk0asb+JVvuvAkak/6XazqXiBSc14jjuqE81618+TEecIz39QzxIDEy/9ycq\nH+yT/q9ElJW7lwia8vf21vI2pGHpnnJ4un5slo73S4gU9tI5MICmVOzWzpn8PnkNUQZzHlEOeEI6\nF68hsoIvq2fZe+KnyxdAnwo7pXkW4L3EU86fSK/uIpKu5xJP4HnN1/aWgxibTra80PGJ6cZ3I5E9\nU1UheHpooNaObVVeDnIfoiD9/9HUPMj26eKYNyHQ3n2SBw4HEy9DL726bTOi0P5xZK/06ujtT9PL\nvL+Q1q/UWPMR6WLarpd8p2l2aOCQzp17iJq9peZaDidSPqcRKeLtfrtLNv17gPnZ/28Di2kK+EcR\nTQpNbGM6u9AUYH2dphegzyUKta9NBPZ/z8/TOrdNfs3YuKzfeKIc7u3A92s9jjt4346jqcWAbrNc\nPfVDBG5XZdex04GXWbpyWN/sd83XmzT9t2mq8bwS8bAxi/QAW+V08hYMhgPfzPqdQjRsvXX6368R\ny97TPl2+APq0snMi2DqLaHz1eqIG0wqp3zgiO6ye6X+dpkZnV8i6v5/Iqt0w/W/tCSp/n2YhTxya\npwhtT9O7VXcjssoPyW7sG9Yxn+WJbN7S/82JgO4Smp6wNyWC/nYHVbWsN1Fm7Uai3MzWxCuyBhHl\nJ88ksmy6TXtiNKUATKNCAEcEVFNpHki3p0JFHhzdDHw221bnEJVPSjfK4W1MayUii/Q4Iit2/XSO\nn5VueIuI1PtBZO9JbeR2KtvXpW23Ac0bSO82QVQ6Ds8jUntW7url6cmfsmP5Apq/i/dcIjX3fel/\nr0rj1TjfKURljX+SHoCIHInPpGtdq+dNGj4vL3g/0WzQmzQ1pN+XKHd7PVnOQnc6ljvjozJ43UhZ\nOZzpRPmBs939B6lc1ywi5eRT7v7fbLz2lrlbyd3/bWYzicLgu2fDbAP80eNVUW1NbwUiZed6j1cX\nLbU85WWhepqy8h0LiHJcyxFlOU4zs12JrIA/Etlor6fxaioblcY9imgzb6yZbUKknL1JlMN6ycyG\nuPtzDVi9apfnaCLLZHuituWrRHZtf+LJ+R6igH5e5nJNd/9rJy5jeZm7KUTN3qnEzeCN9Io8c/e/\nZ8O1ez+1dEybWR8idWInIuj9d+reWvmhNYmU+O+6+09St6OJG9RBRCrhUdnwnVoGrruVuUvl6XYg\nKk780N3vz/oNIx5UG/56u6Jq7fg3s7OIN6es5e7/qmMepWvoKKLJnyfc/Wkz24NIOdzT3e9I5XP7\nu/vLVU63d1q+Qe4+3cz2JFIAT3P3G9I9c1d3v7HWZe/xujrC1Kf5h3gi+QDx1H4FkUpSysLpSzTx\ncFQt003fuxE1PAcQr9G6iNR6PVG+6rdk1d9bmd726Xs1IttrbFn/EdnvHpmyR1M7TKV3ch5FVHb4\nNdGI8Tmp/2TqaPyVpbN/VyKyPr+T/m9GPFFfTMqayvdpB6375jTVDD6eKAc2ggjqrgZeIooKjCtt\np/zYSONU3Rh2ncuap7CeRFTiGE9kc36Apqycq4G9GjTPPPUjT91YDti7ymmUlmvFCv2GEtlMV9a6\njEX6VDhHhpT970u0MvAtGvBWj2XpU3YsW9nxPLnOaZeO8Z2JxIATided7Za670EU+9i5yuntQVOt\n8klEZaZzaUrR243IZTigbLxlKuWu9FEKXjeTXoz9ReJJ5AUimfmPxBP+H/IaiTVMexwRMB7m7ven\nFLiPEzULhxOBxYnu/sNWJoOZfYBok+tr7n5ZqRZu2TCXE0He9rUsa1czs4lEObuz3f1hM9uOaN7l\nZiIb7W6i4O5P3P2wbLz21pbNUwj3A9509+vTvrkKeNfdP2dmI4EPuvv1DVvJ1pdrI6L82q/c/UQz\nuy71epfIqv0dUU7rorT876VqmdnWaV1meSfVXEzb71bgz+5+aOp2EpHd+C7xIPIfd9+7hmnXlIpV\ny7EAlV9yXscyHAX8w90vb++43UVZjfJ5wDyPl9tTdtyNJFI9Z7j7P7puibuvduT2lKeItyul28xW\ncffX0u/BRBbwEUSRg4uJBIGXUv/PEsfoHW1MczkiwPsQ8Gi6Tk4j3nwxB/iZR2sQnwU+5Fmu0rJK\nAV4XKz9xzGwDolD7x4gyPS8RT/KvEAWwX0zDtXmimtm6RBmhM9P/s4E/ExfJsUQKzNNEjaPBwFvu\n/nwbWUrLuftbKSv3KuJVL1elfuUXhSuIi/Fd7d4wXSzthz2BdYhA5fdmtj5wrLsfmLITLiIuNOfU\nOI88S/5HxJPtSOJCdbBF8xrfJipUjM/G67BsMzMb4O5/TzfTDxDNfzzm7ieZ2VZEmbz10nJuSAS5\nF7v7O2XrMw14x90v6YjlTPPY2t1/nn4fTmTVHGNmQ4hs8x8RtfbWIFLJrkvDVr39zGxVd/9nVxY1\nqGd/m9nHiDd2XOnu1+bTM7N+7v5GI5e1o6Tg9x7gp+5+XFm/PMj7InBTKXiQJp11LKes0WuI5ogu\nTN1mEE18bUWkCpayaB/ypiZ/qrmnLU8Ugfivu9+euk0nEii+Rxwf/21lEsuUXl29AMu69GRqZjYr\n/f8j0U7QvUQNzVWIbKcnS8FdGq6aC/6/gC+a2cnp/8NEMyu3EykbzxM3wBXd/c/u/nxr004XhrfS\nk9QuwGPAt83s0DRe6SLbO41yHVGzsMcoLXvaDwOJIOYoi3bDXgM+ZmYHEq2kv1QK7kqpL+2ZT7bv\n1wLudPd9iafTDc2sVJ7vICLV8D0dGNwNAn5lZmNToPYHonzYJ8zsNI/yTq8RWdU3E2/Q2Ag4Mktl\n6ZOW8cIODu5GAp83s/1Sp7eB9c3sKqIJltLr8X7n7j/Igrte7QjudgEeN7NBKYC1rF979/dK7Rk+\nV2PKXa90w/wZkRuwXD49i/JqXzezVdLw7VqfzpBdRyDOw5fc/Tgz29nMZpnZnWa2Qto3pfP2MgV3\nS2vtWG609NBwLjDJzPZJ94uViKIbU1Nw91Eid2pwNl6bx7lHTtEPS8Fd6nY6USHkYOK+JolS8LpI\nhdSuB4F/ufuO6f8HiaRsJ8oTPJO6tyuJ3aIQ97VEC+QQZcjeTCfZFkQq3Kfc/alql5soG/iWux9i\nZuPTNL7p7peWDdutCmhXK63jnUSW7O+JV+n0JWp+bUSkfi7n7jPS8O3Niis1+tyLqHUJ0RzGAe5+\nn0X27CLgeXffKxuvI1PuBnpU4DiAaJ7jUHe/J/WbRZTJ+xKRsjzU3b+RgpZPEsHUIuLds++WTbdD\nljmloI4n2rn6tkfxhROIFOqbUkrFQiKF+b465nMMUcZya0+VJlL3Vdz9tSpTHT5PVEo51Duh4kl2\n7vciCq3/K+v3XkUrIsXjfnf/RkcvU63SuXgocY15mGjL8OdE4+KfBR5w97O7bgl7jlaO5QGeVTxq\n4Pw2B95w9ycscj/OIppfKb1S7hhvozhQFfPIcw0muvut9S53oXg3KAi4rH1oXih8BqlQMNHe1Y+z\n4a4CPteA+ZQXUO5D1EJ7gjba6UrDHwTMzv6fQlaQnEjNe5c6m23p4n2yE00FdYcCC7N+mxKpQVey\ndMO1NVUgSfv+i0QKWSkrdBYwJvVfETi+g9e59ID3USLbft/0/3NE6t0uabvcRpT/K+3rR4BNs+nc\nTjxErN/J+2xFytpxS93XSstzTa37puz/DtlvI1LVH6EdTdUQDVZfSdYcUeq+KlkbXQ3cp72A76Zj\n6pKydSgVfB9ElI1auTRed/gQTcSUGo3+GDA3/V6J5k3yXEWNr5hbVj4VjuVPlP3vn87vqioG1bgM\n+XV1e2AvYFTqVvdxV2Edu82x3NUfZdF2AW96ur6FSKJ+LXUfD7xmZvea2T3EgXo91JaF4k0phOWF\nY/sQZakO8eqeeH4J9LYowwdRLnDXLFvkFiLFa0R7l7E7sGg+43giq9GIrOuVLQqo4+6/JVIPNgDG\n5ON6+woe75pSZiHaPjuFSEH5A1E7+nlgPzP7uLu/7u4np/E6JDvF3d2iMskZRPMF/2dmn03H3LFE\noHcsUZZtI4tXe91KBE9TzWzLtD4rEE0TVJUK3MDlf91TJY50PpV8nii0vW/qV/X2K2Wdp98j0nx+\nVDfpqyIAACAASURBVJqHh9eIwPxLFgXIW5rWe8vk7tOIilJ580YDiFSNLSqsQ0083eFI5ZGIIG97\n4t3ApWHeTev5InCEu/8rG69LWZQbHg/sYmbvT51L2+W/7v4LMxtkZouI62OHFQPo6cqKgYxJqbc/\nLvUDcPf/ENmpXy8d743mkVth7v6su9/j7vPdfXHq11JxoI+Wig9UMf13y/53i2O5O1AWbScys0+6\n+53p95FETZ8p6aK2G9H+3EIz25loNfymNGy12bJV13SqZliLwqu/JoK3jYmsu6c8alVeT1x4f0RU\nV/+5u8+sZt7dSZZltTaRSnemu//EosbxJOAFd59pZtcCD7v7eTXOZyWiUsaTRAO8L5rZPCILbdc0\nzMZEcPKku19d/9q1uUwDiUaMv+pRU/gQImX3B+4+L90EdgdOJVL49iEaEH6ZKCz9BSLL7Fuevb+0\nO6nlnEiB1l3E8e1EEYZ/ZMMZEdR+msgO/neFaZWySY3YVn8CXixNPwsiv0ac+zt5akOxEczsDKJN\nvuuAH7n72WY2nHiPbbcuo5aKjuwF/AP4H03tnJXa71wd+Ly7X5CGr7nNyaIqO5bvIFJpXydecflo\nKegiYoB3Lcro3e/ur9YzvwYt+4eJ4iB3A7d5HW3wLfO6OglxWfkQNfleJp6YAT5BZI/cS7TddDNR\ntm2jsvHazAIktTaefvdu4DIPSt+lt11sQrQMflz6/wUii/nUbJwelzxOU5bVylm3VYmUjx8TDRxf\nW+86EqlkF5K1GUik4n43W4a1O3G9e6fjbves2xlEELoX8bqtB4gaamOJZnvupOldpauSXpnWkfud\n7BVJVQxb86v7svFOpendz9cRFWoGVhiu1axVIkC8mXhwuJosy53mbY0dVGn67VjerSBa/ycKrvch\nUln/CXw5G+56sqza7vQpv24RlSrOBO4jGta+gXjYvIcoy7jUdtSn4nY9iyjrVjq35xLl3/qUDdfu\n7UiU556a/W9EduvY9L0bcDZZUZDUfTCwQVdv157yUQpeJ8hSidYjyjvsS2THbUYEUXNSaskPgXO9\nHc2KpCe0c4hXZe2fd/emVIKa2mZLv9cmgpxr3P0UizcrzCBS8k4oXxYvwJO0LV0B5r23MtS6jtkT\n9YpellJjZj8kyhftkG33hldOKEut7Ovuf0rZ0A7c4e6Pmtlo4DRgCXFzfZzI3ptFlIeaSlQ22R9Y\n0NH728w+Q6SWfd8jO6lDmdnBwIFE+37XpG7fJoLzQ72KN7xk0yo1SzSfSI04xd3nZf0bcr5YNIVy\nDfEA+XN3PzLtx68SwfrFROrxf9x9Sr3zazRrXinkGOC/wE+AZ4iWBEqB8m+IQLDT3pDSk5nZJKIY\nyDXe1FTWScSrKC8D7qvn+DOzDxEpgr8tTb/CMO1JQf8I8frMX7j7LDPbwKM1g1L/FYgcBCMSRd4u\nwv2mI6kMXidIN9Xe7v4n4h2Pi939eXe/IwV3KxBZYM+2J7hL036XeGp/08xOL+tOytYoNaRb9fKm\ncQ8hCpTvDuxsZse7+2NEAPARi9qB5cvS7bVVJsubmnspNTxbCu6s1nUsjedNrzKzrBzMrkTVf8+G\nb/iTVzoOdidSQ843s1OIBouHAzPM7FIipekKIrXuY0Rli8FEcyNvEDUYf00E+J2xv1cjUo53tmhu\n4T0W7W3VxZo3xQERiP0c2DIFSbj7gUQ51sltTCtvRqU3kcX4PLE953hke69t0cRL3edLdvz8jEjZ\n2pC4jkDst3OIh8hTiaIGU8qXszvwpqzsHxBtLPZKvz9CvHf2LSJFp39+LnbN0nZfFY7l+4iKPR+1\nKGuLu59ItKm6Sa3Hn5mtY2aHuvtviAe/ARZNLOXDfMrMVm/PPNz9V8Txu6mZHUa00Yo1Nb30X6I8\n+Ghg+Z5yv+lKSsHrRGUpY6UUnX7ExWsLb2p2o5rycf2JGnl/tSgc+28i2f2ZbJjtiIv7p72KcjfW\nvMHQFYks5MXufklKubuMSOk52Tr5faiNVp5K14XL0ezNJB2RcpdNe1MiJWci8Vqnz7r7yCxV9vdE\ntt6XiIo/fwRuIlLzTgL+StS4/bKnJlQ6SlkK9H5EG2h3ZP0HEw1RL0gPTrXMI085Op5IObqfCI6+\nCrxDBN6/qHZaZdPdm0g9uzQ7t29Jyzy7lmWusOx9iKZqnidqY88EDnf329Jwy7n7W9l43TKV3aLh\n9BnuPjH935HITtyZSGHeyN1v6MJF7NbKjuUTiTKfv0mfLxHta97sUSGu3nltTlTiuSCltDVrMNui\ncsRxxH471tt481LZffFaoujHmkTRlfPTeuXn14HA7T35/tNZlILXQSo9YZal0JRSdN4AbmlPcJd8\nELjYzKYST7nL58Fdci9RCWKXap54S0/SZnYEkYx/AnCEme2aUu4OAfY1s0mlk6snPUmb2eFmdhs0\nSzXoiPlU3aht+cWvo4K7ZDniNWuTiVTZvdI2WIOoGT0I2BXYw93HEKlCo4js45OJJ+ojOiG4K9X+\n6wXg7nN96dcYrU0ENOvVOp+ylKOhwBvEmzm2JN7u0odoTPm9xlMrHTPpnH3HonHh2cD1Fm/9uI8o\nA7WpmU0xs5uJihZ1BXfZspcqOY0mssnmEzf3iy0aA15EZGm9t+zdJbirkNr0EvC6ma1pZn3d/W4i\nOP6Quz9SCu560vWmM3nzlhn6EefydUSO0YVEoHeARQUGoPZt6e6PEGV09zWzr5eCuyxF+TWi8tb/\nt3fe4XZUVRv/rRQUSCC0AAlSIiCRYiAIqBGQki+hBKUqECAUNSJBKSJNVEq+CNJD6GACEQEVpAWp\nIipVRAQLqLRPBRUeiiIQWd8f7x7uvpNbTr3n5Ga9zzPPOTOzZ/aemT2z117lXW8jP9/ezlcIdxej\nfvBpRIGzGvDF7P0qzn9JCHeVIQS8JqEas6hn1AmVfoDd/ZdI23IqcK2LuHhQuQ3ISf6ZngSHkunr\n44iT7Spk3jkK2MXMRrgoKbZ093czKzRZIGko3P1s4G1T9GqntpvM5HUPICaz9eUmgumWorgWMxtr\nZouh/vJhxL+3LzIh/hjRJFyJTDcfQBQ6oD6wIvINfNDdT3T325s5yObCEnCVmZ1sZgdk11IIfQ+j\nQevT+bVWWMcu2epHEGn1Qe5+FiLPnYVS1M1GqY/e9QPqqr8nYdRQMMVLwCNI4/khJDBekc53j7sf\nmF9HnZiFfO6+DgxLWq+bUdDGrsCjnkV9t8u7mmubzGwrM9va3Z9EAvbXgBXS/fwo4jp8F+1yDW2K\nzwI/Q0L+BEST8x1TbuhZSJP8q6Jwtfcyf8eSOXUqsId10Em9K4QhU+pDXmEqPJMla0kU8f0v9N48\nhQj6j0mTk5ZbXBY2DOq9SKAamJy0x7r755sxW7bOpsX70Udxqpk94vKJ6KTy9l547sxsqLu/loS8\nld39HjPbHWl4DkJO4h9Jy/c9pTPL62h3pMHUEyaZ2d6l/cOBWWb2FXf/Yz3X5u5XJu3Nt8zsYO/M\ne7Y08J9KP3r1wDpcALZDAtx+7n6fmf0cmR63QbQJG6IP6VPIUX9ZYF8ze83df21mNwHjTTkg3yxu\nYrPanQlLRyOB9HmkRRxpZiem/YPcfb67X2Rm30/HVdQmEyVRrmF9BmmOVgRecvefmdmZKHrvUhRk\nUkl/PwSZEQ9I5f+GgjVmA1eXJhO1BuqU2/BbYG0zOxWZtEYAbyHh7n5P9C211tcsZAL8zWiSMdKU\nQeUQ1Fe/jSI0n/RSdpxAB2xBN5Nr0XhwCXCHu3/TzH6ItLwbuPtt6biqv2/FMWa2KdIQvuHuD5rZ\nVPTtfMfdv108W5dloluaL+tscjV3f9PE/TrFzP7s7k+Y2XcR0frzC8tY03bwNgjl7U8LMvVcgaJh\ny/sKKoyqqUzIGPBRovetSfQRKPLoYaSWXx1FOhYcRz2dcxKa5Q1HL9L1KCpqD+SntSEyV/0UJZRv\n+f2t4b4NzP6/v7Qvp6o4CWmz3ltjPWU29Yml9WWAC5HJZIHyDbzenOplNPKrWz+tL4eCZqYiX7Oz\ngG2Rv9moVGZ9pAF4APnRPEwF2U4a1PbCJ3ge8hcCWRm2Qr6Dx2dlaqVAKd7Bw1GkLMgR/UKkZRuE\nzFxfrKJfLQHsg7QWR2bb90MBG2Mb3I+XT7+jEPXFCantKyMqlJw2qW1oi7I+ZsCXgbnZvmtIVETp\nOsaUn1ksne5lnrFkN2RZKfadhdKRgahGptRZV5ERabv0PZmKtP0T0vaxaCL01SrPNwAF7E1HqRBX\nR1HTN6NJyrXAaeVrjqWKZ9fqBvSXBUV4FR+vVYBJpf0jkRls6RrOvVz6iG+GuNmeRnQrT5CEFpQA\n/qm0bXwV5z4eaeZA5tnPohys9wMntvq+1vlMisG88E25H2kIRmZl8oHzIGBYDfUUH6zCrDQirzsr\ndziif1iiSdc7JD3Pg9L6uigqdkvkfH8fEtjWp8O/aYEBFPm2XYBoe8YV19bs55St7wnMJ/HzIZqU\nrZHw8ska6yjzrG2KtGtHpfULkUn6TuCCKp73pqT0Wcgv6Rw687SNa3A/vh2Zgm8lcRBm5eYAlzfr\nOdV5DeNJ0clpfRdgRqnMXaQUVtm2GNS76Q/p/+2pL3yPjpRuJ6V35QEUvb3AcRXWMyL7vybwUPqd\nhAKw3kL+uiDXj49Vcw3om3wkCgR8EqVBXAMRqs8ETo5+UGdfaXUD+sOCZtLTUSqrMpFtMdMaiELK\nv1bDi7YqIv2cjnwTNkrbZyBH+LXS+qZkM98ezjcZmJW1a6vS/s3SQPc6maC6ML5kaRA+C/FrLQf8\nIN3HNbp4RoPrqKdPSG17acPiyCftW3QIR99FPGyfQdqp04DD0iAwrXSfcpLgD+X3qInPJxeWJpCI\nvpG24HXgf9J6l3lnK3022TO6EQmLA5Hm4bt0CHlLkYTe8nPr5lx3I239n9L9HI2EvAuBw8v9sAH9\neB6aJCyFBtc7gfch36WjyIS7dnpX0cTDkKb4epSabTSaSG6WlbuVlIM2lm7vZT4h/QCJyBq5VlxL\nsrQAY0jCVy39IfXvGYhOJa/vI8AjaX0qog+aVMN1HIB8vd+LLCffQxOAPbpqS6vv+8K6tLwB/WVB\nTtWvA5f0UOZjSEs0qIrzFgPKSJT/8kHkT1XsPwVpZT5QxTmXT+daKq3n6v7i/+r0kWmuyc9lb+D3\npCTbKGhgdvp4rd3Aek5HPkQrAr8hZf8oP8c+uN4lkLl99Wxbkex7TGrbuDTAPgPsk5Xr1azfpDYX\n0aDXIR66M1N/n4j88HYsla+6jena7kY5c/PtY1J/mNlbHSTzNxIOjwAuS+vLIe38yWl9X+o0i5Xb\ngISkKYhC4s5Ux8XAY0jrsWJWtm0GRJTu7yfAcmn9UKQZXg1pbO5L78684n7G0u29zE2b1yM3izuA\nNdP2ZVBE+I9Kx9XUH9CEcFXge9m2fYBT0//xSKjs1WJEF2MesBiafBcTrEsQRdGGrb7X/WWJKNo6\nUAr1fwZ9rNaxEuljhgdQwvEeeYFyuJzKN0Yf8eko2fu6ZrZV2n8MesmX7f4sC+C/yHy3ZzqHFxQK\n2f+nPQVoNCjir09gC9Iv/BIN7FPMbLSLD/BwYD3kY9govITIgZtCalspXETK81xZKgpagflmtjn6\ngB7j7ve6+2+RJvcrZjY5lfP0/Jv+vM1saLZ6APA7d/8kEpx+j+hYbkG+WjuWrtErrCO/jnWBP7j7\n0Wa2tZmdb2bnouwPM5Gg1G0dZjYJuNnMhrucw/8CvGZmi7vyd04DdjOzddCAeFk6ruaI46INZvZ5\nYNV0zkmIauU7KNfsSmiAfaGor6/6WiVwRdzfAVyaAnXmIrPiueie74V8rua6+xRYuL43fQnvoPX5\nBproHwm8DGxrZqu6+8vIQnBb6biq+kPWZ99BSoshZjYnbXsOWM3MTkHP8BR3/3FP/dzMVknfoIFm\n9kUzOzC19y0UZDMsFf0PEvIfqaa9gR7QaglzYV3omE0NROaZ96X1Y5Ggt0Ja/xQ1aEbo0KSNRWa/\nB5HpdAjyx5tBMl/V2P4PoZd1z3KdC+tC5xnufkgDNAT5RJ6IZouFCXBog+o8A/nd7YI+tnle3huB\nz7b6vmTt+WD5OaOcyM8hc8uqfdSOg+mc//ZLiOqnWN8YCQWrNaAvGBLuhgOPp/OeizTpNwOTS8d1\n+w4g/8YbkSl8PPLZG0MKzEE5i9ettc3ltqf/SyDBfGpa/zgimZ2AuMKm1Vtfk55x2Q1ky2zf8kgg\nvoEUDJDtaxvtY7stqS/vjczza6dtWwHfSfdzVLl8LXWk3+3RWLYYUh5cQ4fGeiKKct+ugvN9EmnI\n10Ua+suRSXYm0gaOR+bZR4FL62l7LAsukcmiDqSZ5o8Qz9zBadtAFAm0DwqGeNzdj6jx/Nshc9U5\nSCAbgQS7h9HL9x5kFvpnjeffBvkFfsvdL8+vy9tIC1AN0jO5Aan6l0D3bG/kp/QF5LN0tNdI92Kd\nMyy8F2kD/4x8+45FA/41SOD7uyfes1ait+dpynBxCPAv1ObpTW5PQc1zADIpvRf5Dd7l7pekMvOA\nb7r7z2s4f0HpMAA9l/vc/X/NbG3klvBQKjcn7ZvZw7kmIyFkanq3P+6J6NnMjkLP+zUU+fmau+9Z\nbXu7uwYk+N4B/BMFIByKyJO/gHyhXvGO9GNt9c6aeCC/iNgEXi2/Z2n/QYh65/RWtbPdkTTvL7j7\n783sWBTxPQuZY7d2aca2RJrvs939xw2ocxtEVXO4p9SZZrZsqvcdd/9MVrbb76cpG9JySKu4Gpq4\nTDZxcu6BlBdnoaw5H3b3eb2dM1AlWi1hLswLcly/DGmMRqEBfjdgKBrgD63yfCvQ2cn7RGDX9H9F\npJW6E2k4hpGCK+q8hnFIQJlGBTOydlzorJH6JqL3GILoXW5DvipLoY/MrnXU05XT/W5o9rkMErgn\nI03PYT0dV+f1Lo9MdUMaeM7F0z3bgoySp8HtXjr7vyTKtHIK8vPZBQWDXJeWK2usY0j2/yrguGy9\niDgfiXyY5lR4r79OyV812z8OaTumZNtqpXAZnP3fHAmOTyBt3c5I+1G04z311tfMJb0P19A5orh8\n7xrWf/vrgiYQr6NJ/fFp2xLIqnMjHVRZ6zSwzjNImu1SnyyC1Hr1kUO+l3enfjASTeCepSNAcGDa\nv1vpuNDgNnAJX4cq0IV/11+QuvxeNFsdi1TOo9z9+y5m/Ip8StKM9nPAq5ayKiD1eOEf9QIy/f4X\nqcdXdbG/1wV3vze1eT6wlZntV+85+xqevgxmNgKZA85ETu/Xufu2SEN0H/Ciu1+bylblG2VmnySl\nxTKzw83sClNKpWvQQHZEasocV8aH01PZZmhWtkRExRPTLLnuDBzu/oa7v+7uP/GMnLlRMLNRwIFm\ntrGZ3YvelV2BDwL7IwqbqWgAmePue6XjqslQcSCigCnSxQ0FHjazfczsfOCepAV5A2WAmJzK9vR+\ndumvah0ZNe5195u8w+duQNEfq0HSar5tZoOTf9I9KCL3p0jTNQ65Amyb6i3SQ7WltsPlD3YScKyZ\nle+dpfWCiDnSj5VQ3CdX5olbUFDUT9Put5E29y/AL0zE378rjmtA9UOQQiFvz3pIw7+7V+Aj5/K9\nvAspQP6OfJOvQCTqY11+rG9QSrbQhG/loo1WS5gLy0Jn/66tSOSlaKa9a1buFroI9e7l3OsgP4R1\nkannDEQlMAy9IKelcoX570Lq0ET10palWn2va2z3CSj5NUigu5DE44SoQY6s49w7kaJN0zNYFwmS\nVyKH52lIy1NxdHQDrncfRP2yTGn7FjRAs9ukNp+ItBGzs23DkZl2BinSMttXLZ3QBul3W6Qh3BkF\nP52EhKMP5nVXWgdN9lelHxOOo4nIE2SR/7U820VpycaawoVq5dQvXgV2zsqNrHas6aKuoo6NUz8f\nlb5vz5DoT5A7wB+ogKaInn0vN0CBgs8j16aLW32v+/vS8gYsDEv2EgxAAtzZyIw0IyuzIhK+uqVJ\n6eq8aAYzE5kVByMn6lnIB+KjSCNxIzLNPoVmcscCJ7T6vrT4mQwurY9AWqAJyITxQxRleB9wbvlZ\nVlHPhcBX0v+D0Ey0CFb4CAoYeB5FnO1Z6/VU0w+z9aGl9cIUsn/RX9vgOeUBA1sgh/CrURRzQTsy\nHPma1XT/SnV8FPGp7Z/er8KkOQj5L51TYx1NFVTo34TjC70bSB/eq1yR8C00OdkwbZuEtGg7IG1Y\nt+bvKuvcHuVPPgzRKG2UvqPPIMLzR4EdKjxXby4Na6Lx83PZtpZ/p/rr0vIGtPtCIntNL9xFyDy6\nDPLr+jUp3U4aAL6RHVeJZqDguNsMUWy8iNTjiyNuqNNIxMUkrqs0SP6OKnjv+tuSCQaDkWBXfEwO\nAQ5O/0cj0t9DsuOqFe6GIz+xCchReCPgVOQYvHFWbmNgrz669k8gQXNzOiYeuYAzCWmAWu6XRWdS\n4Avo8Cc9IbVxdHqGO1Cj3x9dpP1DGva5iH5lBPKN/D51EgHTYEGFRYtwfC0UHHIaJSE5lgXuVZ7l\n4eTU54qMETsgqqwzGlTXCsgXbiU0Kfo5sFLatwrS6BXMA732OyrzvVytu32xNLgvtboB7bzQkV5n\nk7T+YSR83Yr8hZZFkbJXlY6rRLhbFpn4RiJfoVeRhq4w/Q5DM7gL6UjvMwpp99Zv9b1p4TPJzVm7\noSCKM5AJayzivVuzi+OqNfcVQuM+aYC9Na0PRdrV05G2qJwCq+GzUToEuQ1RSp/LkSbx2GzfoKz8\nhFY/p7ztiI7kjPxjjsy1VyMB+rrytVZZx4D0Tl6JIk+LYJErEDnwSvSSoaKKuhomqLAIEY6Xrnuh\ndAPpw/tTzvLwXaTJ/Uzav2RWtlYS46J/LYssSPsiX/KCfmUHasxkQzcuDeV3u5Z3PZYqn0WrG9Cu\nCwum19k4bV8FmZkGIO3DOWQ586qsYzryRVgMCXoTEPlnkcR5GAoYGJ23q9X3ptULnc1ZY5Am5Ank\nCH9fEiYWq+P8k4CfIY3pTkkI+RUd2TCWTgP8JXQhTDbpmrdOAkzRD7dB+WRzIa8sbLb8A4r8S2/I\n1hfP/m9CZvqppr101lp+AZnjJ6JJ0alIEN8CuVRs3ox7Uq+gQi/aji4GxDBl9cOFyrI8XEwpy0Mt\nfTn7ViyXbbsABTyMTOvjkHtAzZG5hO9lWywtb0A7LnSfXmcz5N91GzLT3AOclB1XiQp7OHBWUR75\nPbw/27830kJtn9bf9clo9X1p8TPpzZy1IfKH+xVwfgPqOx6Z9Qoz4+5I+B6f1pchzaj76Pp3QX5+\nhQl6CST0XQZ8vdXPJ2tnWchcGmnXxmd9eUlSbtysXMX9m86m392RoLt+2rYZol45NdXdMPqIJt2v\nfkc4HktVz3+V9DsQMTEcSCIcRwFc09P/mYi/sxF1bodcjC5FgVobImHyVjRZepQa8st2UU/4XrZ4\nCaLjbmBmX0Mmvz2QNuDTaJCahviAxqFsFcem8hXTFZjZeSj35eMFjUZ+fKIqOQo5W7/sCilfpNEd\ncWqirnknu3eD3f3t9L9aEuMyqe3m7n5Xtn9P4KuIj+r6bHvDqSqy61sREej+28y2Rxqfndz9tkSn\nMw74q7v/ppH11wIzG+hKpzQAvTsD3P1+MzsuFXnM3a83s6uBp1xp9qqtY5CL3NWQL9IrSBN4lydS\naTPbFJmc7nH3q9K2tqQTgf5JOB7oHYl6aWcUQX4ySts1Arlh3A/8DZlr1wEedvf903E192Uz2wT5\n9l2ArFSbpjq/jHhW/wk86+53NeKdMbO1UFT7KOA3ef8ONB8h4GXohbV+eaR1GI8Y9h/MjqvoQ5wG\npYHI7+YFdz+n2F5+kcxshKdsCwEws2WQP+Id7n5+2lZmx8+F5Ko/Tj0Ikfl5D0K+KVULJ1W0o6h3\nRxQ44shkfAEySc9FRKQ3t4sQkE1UBiC/oV8ik+l5yLx0KDLbzEfZMoq8o9VMjIoMGIOQ5mENd59i\nZssh7cNd7n5kKvt+d/9jgy+zaTCzccAc5F7wlLvf3OImBZoIa0GWB1M2inuAJ939U2Y2GGX2+Rqy\njtxf73X1Uv9S7v5qM+sIdEYQHXfGLcALqSP+F5lpAXD3fyBn158jzRrZvooGWBfmI1+qI8xsn2J7\nmaAyhLvO8B6IU7My3tX/KtAlqW3pvBc1S7jLyHPdzN6P/MmORP5+ryGTze1I6Ls2CTYtRxK83kmC\n15WI8uR4RAJ+JnCMu5+GfBsPyYS7ikmBzWwScLOZDUeah5WBTcxsjCtV3/bA1mZ2EUAh3C0sJLre\nDwjHA5XBzHZCgUevo0nri8AWZraRu7+FJnBjkL/tPzPhrmoS7aL/m9kK7v4SCt74hJnt6u5vu/uf\nUtF1GnJxPSCEu77HoN6LLFLIB/jzywO8u//DzM7xxMBeK9z9t0lbeF46/ZxMQ9QWGpl2hLs/amZT\ngLNNWSQub+R9c/eXzewk4EYze9Xd53Yl5DUDZrYK8CkzO99lYh4KPO3uj6b9zyJT5NbuPtfM7vYa\ncxA3EknwOtLMdnH3F83sTEQfdAPyg/sp8FyaNBV5ewtNRMXPy91/ZGYfQlRFuyJN19+B/czsUnf/\ndTJhf6F03EJjonBlpnkSpO1ocXMCTYLLTWEM8p/dHWm456MsD+buD5tZ3VkeSpaAr5jZnu5+tZnN\nB2aYssvcicy0lzXg0gJthjDRlpAGkRtR9NLctK1IrdOlObCOuj6BMiKcAtzk7s/Wc75FBc02Z7XC\nJyqZoFdFs/m3EKHpDcAP3f28VOZU4P/c/Uzr8HdruW+ZmR2PhM89gDcRtcd05NLwDoo6f8PdP1fD\nubv1i0zv6iQUuHS5uz+cHdfy+xII5OiiL+cuQBug92Uycm940ZNPaQ31vPudMrMt0bfsAHd/0MyW\nBP6NaFDmAg8A0zzzB6/rIgNthTDRlpA0JlOA4wozSTKtumU5KxsxeKSBaiJSx3/JzI6u95yLrBsv\nTgAABMlJREFUApptznL321EmgRPMbJqZbZe2N/zjZ2YDTYEhL6Mo3XORKXYoouBZz8xmpcnATuiD\nTHIhaJmGyswmm9mstHoKSqf379Su/yIhdRrSEDxeCHc1mEzLbhN3FzvSu3o98B+UhoxsXwh3gXZD\nTy5Av0a0Sz9Ak/0iYKiqMTr52c02s5XSphFIOzfYlKv5x4iD8hE6AghHFc2o9cIC7YnQ4HWDvnR6\nNkVDDkSOtQ94E5K992c0y3nXmhwBZmbvSee/D0XDroyoWc5Hwt5sRHZ6OHK2vsPdb2hkG2qFdROQ\nku3fDQ0uq7n7YWlbLYEvlQTXvM/dn6v/qgKB5qHCvryauz/T1b4q6pmOUlzui8aUvZDr0QVIezcO\n5TR+zMx2RxOxicDrMTHqXwgBrwc0e4APLDxoohD5eZQVZTHgy+4+LwUSXIR82U5z91faySQL3Q9W\n0LX2rB7zT1+6TQQCzUSz+nL6Zhzr7oem800Drnf3p81sJHKReMnM1kZUS/sXLg1mNsTr9CsPtCdC\nwKsQzRrgA4smrINWZBmkqVsWcS3+w93fSB/s2Sjv8JGeuP3aCZUOVg2qK7jiAv0CzerL1gW/arZv\nIKIpOg04zjMez0D/RQh4gUAfI4tu2wLYGEXRHYyi2U5094dMPFnvAVZ390da2Nwe0ZeCV1+6TQQC\nzUQj+3KaVPXKr5rqfDMFW4S2exFACHiBQAtgZhMRR9xUd78zbfsG8p35CfpYb+ruf2hZIytEH/ur\nhttEoF+g0X3ZzEYD81CmndlpWwhyizBCwAsE+hhJOzcLUXvcZeL0eyvt2w998H/h7re0sJlVoRWC\nV7hNBPoLGtWXzWxzlD1mhrvPybaHK8MiiBDwAoEWwMyuQELczGzb2sCfPculCwsf5UcIXoFA62DB\nrxpICB68QKDJKAQ1MxtmZsPS5tuBlcxsbNo3FvmyrV4c5wl93Ny6EcJdINA6ePCrBhJCgxcINBFZ\nQMWOwBHAXxHv3UyUW3dtRAq8AfDViG4LBAKNQPCrBiIXbSDQBBQ+L0m42waxx++M0hGdDizp7keZ\n2Zoo0fezrnyq4RQdCATqRibQ/aTHgoF+ixDwAoEGI6ULOtvMjnD3vwGLI1b50SjF2ibAHWa2pLsf\nAzxVHBvCXSAQCAQagfDBCwQaDHd/CXgOuDil9LoJ+APKcTzD3R9CTtCHmdla1eabDAQCgUCgN8TA\nEgg0CGY23MzOSqvHALcBw5Kp9g3gT8AaZvYpYBiwkbs/GfQFgUAgEGg0QsALBBoEd38RGGxm6yZT\n6znu/lQRRQs8BqyJSIx/4O5PQEeUbSAQCAQCjUJE0QYCDUBP6YKgw7fOzAYBS6XE3xFQEQgEAoGm\nIIIsAoEGIAlq883sSmCemb3i7rNTFK0Vwpy7zwdeyo4JBAKBQKDhCAEvEGgg3P23ZjYZOC/JdHMy\n7V2kCwoEAoFAnyBMtIFAExDpggKBQCDQSoSAFwg0CWa2HnAI8C/g7+4+vcVNCgQCgcAighDwAoEm\nItIFBQKBQKAVCAEvEAgEAoFAoJ8hePACgUAgEAgE+hlCwAsEAoFAIBDoZwgBLxAIBAKBQKCfIQS8\nQCAQCAQCgX6GEPACgUAgEAgE+hlCwAsEAoFAIBDoZ/h/eXbsUtrr8J0AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_discriminative_features({(f,y):p for (f,y),p in alpha_bigram.items() if len(f)==2}, reverse=True)\n",
"# alpha_bigram"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Observe:\n",
"\n",
"* Presence of entities such as movies and actors\n",
"\n",
"For example\n",
"* Mentions of \"pulp fiction\" suggest positive reviews"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"How do such reviews look like?"
]
},
{
"cell_type": "code",
"execution_count": 239,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"in essence , this is the science fiction equivolence of pulp fiction . the easiest way to write a review\n",
"than anything actually shown--like most of the violence in \" pulp fiction \" . \" se7en \" is gory ,\n",
", the film is paced at half the speed of pulp fiction , which avary co-wrote with quentin tarantino .\n",
"this statement , but you need look no further than pulp fiction for an example ) . and , while\n",
"\n",
"since this is a gritty crime comedy , flashbacks of pulp fiction should arise ) . at first , i\n",
", stock & two smoking barrels ( 8/10 ) - pulp fiction ( 8/10 ) - reservoir dogs ( 9/10\n",
"incredibly fun to watch . it starts off as a pulp fiction-type crime story , with criminal brothers george clooney\n",
"can be even more graphic ( e . g . pulp fiction ) . pulp fiction did it in a\n",
"you couldn't find in any of the two dozen \" pulp fiction \" wannabes ; fortunately , the reappearance of\n",
"but if most of those movies have their roots in pulp fiction , exploring a modern myth of the doomed\n",
"the three years since the release of the groundbreaking success pulp fiction , the cinematic output from its creator ,\n",
"first movie quentin tarantino has directed since the highly touted pulp fiction . to say he has been inactive in\n",
"me give you an example . do you know in pulp fiction where jules and vincent go on brain detail\n"
]
}
],
"source": [
"docs_with_good = [x for x,y in train if 'pulp' in x and 'fiction' in x and y=='pos']\n",
"for doc_index in range(0,14):\n",
" good_index = docs_with_good[doc_index].index(\"pulp\")\n",
" print(\" \".join(docs_with_good[doc_index][good_index-10:good_index+10]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Still problems, like longer distance dependencies: \n",
"\n",
"$$\n",
"p(\\text{Quentin} \\bar \\text{Pulp},+)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Can we do more with unigrams?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let $\\hat{\\prob}$ be the empirical distribution\n",
"\n",
"The problem is: $\\pnb(\\text{Pulp Fiction}\\bar +)$ is **too small** \n",
"\n",
"\\begin{split}\n",
"\\pnb(\\text{Pulp Fiction}\\bar +) &= \\pnb(\\text{Pulp}\\bar +)\\pnb(\\text{Fiction} \\bar +) \\\\\\\\\n",
" &\\approx \\hat{\\prob}(\\text{Pulp}\\bar +) \\hat{\\prob}(\\text{Fiction} \\bar +) \\\\\\\\\n",
" & < \\hat{\\prob}(\\text{Pulp}\\bar +) \\hat{\\prob}(\\text{Fiction}\\bar \\Pulp, +) \\\\\\\\\n",
" &\\approx \\hat{\\prob}(\\text{Pulp}\\bar +) \\\\\\\\\n",
"\\end{split}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"How to fix this for a unigram model?\n",
"\n",
"* Increase $\\prob(\\text{Pulp}\\bar +)$ "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"If \n",
"\n",
"$$\n",
"\\pnb(\\text{Pulp}\\bar +)\\approx \\frac{\\hat{\\prob}(\\text{Pulp}\\bar +)}{\\hat{\\prob}(\\text{Fiction}\\bar +)}\n",
"$$ \n",
"\n",
"we'd get\n",
"\n",
"\\begin{split}\n",
"\\pnb(\\text{Pulp Fiction}\\bar +) & \\approx \\frac{\\hat{\\prob}(\\text{Pulp}\\bar +)}{\\hat{\\prob}(\\text{Fiction}\\bar +)} \\hat{\\prob}(\\text{Fiction}\\bar +) \\\\\\\\ & = \\hat{\\prob}(\\text{Pulp}\\bar +)\n",
"\\end{split}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Note: $\\pnb(\\text{Pulp}\\bar +)$ may be **larger than 1**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"But this is **not the maximum likelhood estimate**..."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"If you were to generate data from it, it could look like\n",
"\n",
"---------------------------\n",
"|$\\x$ |$y$| \n",
"|---------------------|---|\n",
"|... Pulp Pulp ... | + |\n",
"|... Pulp Pulp ... | + |\n",
"|... Pulp Pulp ... | + |\n",
"|... Fiction ... | - |\n",
"|... Fiction ... | - |\n",
"|... Pulp Fiction ... | - |"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"But it **doesn't matter** for us, because we get $\\prob(\\text{Pulp Fiction}|+)$ right and we may never encounter \"Pulp Pulp\" anyway. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Discriminative Text Classification\n",
"\n",
"Ignore generation, care about **discrimination**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"How can we set probabilities algorithmically?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Turn it into an optimisation problem and \n",
"\n",
"maximise $\\prob(+ \\bar \\text{...Pulp Fiction...})$ directly"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Maximum Conditional Log-Likelihood\n",
"\n",
"Directly optimise the **conditional likelihood** of the correct labels\n",
"\n",
"$$\n",
"\\mathit{CL}(\\params) = \\sum_{(\\x,y) \\in \\train} \\log(\\prob_\\params(y|\\x)) = \\sum_{(\\x,y) \\in \\train} \\log\\left(\\frac{\\prob_\\params(y,\\x)}{\\sum_y \\prob_\\params(y,\\x)}\\right)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Unfortunately less trivial to optimise (no closed form solution)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Log-Linear Models\n",
"Present NB (and other following models) in log-space:\n",
"* Does not require per-word normalisation\n",
"* Simplify math\n",
"* easier to optimise\n",
"* enables rich generalisations\n",
"* resemblance to linear models like linear SVMs... "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Feature Function\n",
"\n",
"For log-linear models we need **feature functions** $f_i(\\x)$ mapping input to a real value, e.g.\n",
"\n",
"$$\n",
"f_{\\text{Pulp}}(\\x) = \\counts{\\x}{\\text{Pulp}}\n",
"$$\n",
"i.e., the number of times *Pulp* appears in the input $\\x$. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Features don't have to correspond to single words\n",
"$$\n",
"f_{\\text{Pulp Fiction}}(\\x) = \\counts{\\x}{\\text{Pulp Fiction}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"or even correspond to n-grams\n",
"$$\n",
"f_{\\text{RT}}(\\x) = \\text{Lowest rotten tomatoes score of all mentioned movies in }\\x\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Joint Probability\n",
"\n",
"Assume we have a weight vector $\\weights_y$ for each class $y$, then a log linear model defines\n",
"\n",
"$$\n",
" p_{\\weights}(\\x,y)= \\frac{1}{Z} \\exp \\left( \\sum_{i \\in \\mathcal{I}} f_i(x) w_{y,i} \\right) = \\frac{1}{Z} \\exp \\langle \\mathbf{f}(\\x), \\mathbf{w}_y \\rangle \n",
"$$\n",
"\n",
"where $Z = \\sum_{y,\\x} \\exp \\langle \\mathbf{f}(\\x), \\mathbf{w}_y \\rangle$ is the partition function (or Zustandssumme) and **intractable in general**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Note: **weights do not need to normalise**"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"You can convert a Naive Bayes model into log-linear form with\n",
"$$\n",
"f_{\\text{w}}(\\x) = \\counts{\\x}{w}\n",
"$$\n",
"and \n",
"$$\n",
"\\weights_{w,y} = \\log(\\alpha_{w,y})\n",
"$$\n",
"\n",
"and a bias feature $f_0(\\x)=1$"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Conditional Distributions\n",
"\n",
"We care only about conditional probabilities: \n",
"\\begin{equation}\n",
" p_{\\params}(y|\\x)= \\frac{1}{Z_\\x} \\exp \\langle \\mathbf{f}(\\x), \\mathbf{w}_y \\rangle = \\frac{1}{Z_\\x} \\exp s_\\weights(\\x,y)\n",
"\\end{equation}\n",
"\n",
"with a tractable *conditional* normalizer: \n",
"\n",
"$$\n",
"Z_\\x=\\sum_{y\\in\\Ys} \\exp s_\\weights(\\x,y)\n",
"$$ \n",
"\n",
"where $s_\\weights(\\x,y)= \\langle \\mathbf{f}(\\x), \\mathbf{w}_y \\rangle$ is the linear score of $\\x$ and $y$.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"For binary tasks corresponds to **logistic regression model**:\n",
"\n",
"\\begin{split}\n",
"\\frac{1}{Z_\\x} \\exp s_\\weights(\\x,y) & = \\frac{\\exp s_\\weights(\\x,y)}{\\exp s_\\weights(\\x,+)+\\exp s_\\weights(\\x,-)} \\\\\\\\ & = \\operatorname{sigmoid}_y\\left(s_\\weights(\\x,y)\\right) \n",
"\\end{split}"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Conditional log-likelihood in log-linear form:\n",
"\n",
"$$\n",
"\\mathit{CL}(\\weights) = \\sum_{(\\x,y) \\in \\train} \\log(\\prob_\\params(y|\\x)) = \\sum_{(\\x,y) \\in \\train} \\log \\left (\\frac{1}{Z_\\x} \\exp s_\\weights(\\x,y) \\right) =\\\\ \\sum_{(\\x,y) \\in \\train} s_\\weights(\\x,y) - \\log Z_\\x.\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {
"slideshow": {
"slide_type": "skip"
}
},
"outputs": [],
"source": [
"p,f = \"Pulp\", \"Fiction\"\n",
"data = [((p,f),True),((p,f),True),((p,f),True), ((f,),False), ((f,),False),((p,f),False)]\n",
"from math import log, exp\n",
"\n",
"def cl(data, w_p_true, w_f_true, w_true, l=0.0):\n",
" loss = 0.0\n",
" for x,y in data:\n",
" count_p = len([w for w in x if w==p])\n",
" count_f = len([w for w in x if w==f])\n",
" score_true = w_true + count_p * w_p_true + count_f * w_f_true\n",
" log_z = log(exp(score_true) + exp(0))\n",
"# print(count_p)\n",
"# print(count_f)\n",
"# print(log_z)\n",
" if y:\n",
" loss += score_true - log_z\n",
" else:\n",
" loss += 0 - log_z\n",
" return loss - l * (w_p_true * w_p_true + w_f_true * w_f_true + w_true * w_true)\n",
"\n",
"def jl(data, w_p_true, w_f_true, w_true, w_p_false, w_f_false, w_false):\n",
" loss = 0.0\n",
" for x,y in data:\n",
" count_p = len([w for w in x if w==p])\n",
" count_f = len([w for w in x if w==f])\n",
" score_true = w_true + count_p * w_p_true + count_f * w_f_true\n",
" score_false = w_false + count_p * w_p_false + count_f * w_f_false\n",
" if y:\n",
" loss += score_true\n",
" else:\n",
" loss += score_false\n",
" return loss \n",
"\n",
"\n",
"# cl(data,log(0.5),log(0.5),log(0.5))\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import mpld3\n",
"import numpy as np\n",
"\n",
"# x = np.linspace(-10, 10, 100)\n",
"# cl_loss = np.vectorize(lambda w: cl(data,w,log(0.5),log(0.5),1))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Consider a log-linear model\n",
"$$\n",
"\\mathbf{f}(\\x) = \n",
"\\begin{bmatrix}\n",
" \\counts{\\x}{\\text{Pulp}} \\\\\n",
" \\counts{\\x}{\\text{Fiction}} \\\\\n",
" 1\n",
" \\end{bmatrix}\n",
"$$\n",
"\n",
"and loss on the following data:"
]
},
{
"cell_type": "code",
"execution_count": 241,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" x | \n",
" y | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" (Pulp, Fiction) | \n",
" True | \n",
"
\n",
" \n",
" | 1 | \n",
" (Pulp, Fiction) | \n",
" True | \n",
"
\n",
" \n",
" | 2 | \n",
" (Pulp, Fiction) | \n",
" True | \n",
"
\n",
" \n",
" | 3 | \n",
" (Fiction,) | \n",
" False | \n",
"
\n",
" \n",
" | 4 | \n",
" (Fiction,) | \n",
" False | \n",
"
\n",
" \n",
" | 5 | \n",
" (Pulp, Fiction) | \n",
" False | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" x | \n",
" y | \n",
" p(+) | \n",
" #(Pulp) * p_data(+) | \n",
" #(Pulp) * p(+) | \n",
" #(Fiction) * p_data(+) | \n",
" #(Fiction) * p(+) | \n",
" CL | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" (Pulp, Fiction) | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" -0.287335 | \n",
"
\n",
" \n",
" | 1 | \n",
" (Pulp, Fiction) | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" -0.287335 | \n",
"
\n",
" \n",
" | 2 | \n",
" (Pulp, Fiction) | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" 1.0 | \n",
" 0.750260 | \n",
" -0.287335 | \n",
"
\n",
" \n",
" | 3 | \n",
" (Fiction,) | \n",
" 0.0 | \n",
" 0.000136 | \n",
" 0.0 | \n",
" 0.000000 | \n",
" 0.0 | \n",
" 0.000136 | \n",
" -0.000136 | \n",
"
\n",
" \n",
" | 4 | \n",
" (Fiction,) | \n",
" 0.0 | \n",
" 0.000136 | \n",
" 0.0 | \n",
" 0.000000 | \n",
" 0.0 | \n",
" 0.000136 | \n",
" -0.000136 | \n",
"
\n",
" \n",
" | 5 | \n",
" (Pulp, Fiction) | \n",
" 0.0 | \n",
" 0.750260 | \n",
" 0.0 | \n",
" 0.750260 | \n",
" 0.0 | \n",
" 0.750260 | \n",
" -1.387335 | \n",
"
\n",
" \n",
" | 6 | \n",
" [Avg] | \n",
" 0.5 | \n",
" 0.500219 | \n",
" 0.5 | \n",
" 0.500173 | \n",
" 0.5 | \n",
" 0.500219 | \n",
" -0.374936 | \n",
"
\n",
" \n",
"
\n",
"