{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important: This notebook will only work with fastai-0.7.x. Do not try to run any fastai-1.x code from this path in the repository because it will load fastai-0.7.x**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai.nlp import *\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDB dataset and the sentiment classification task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [large movie review dataset](http://ai.stanford.edu/~amaas/data/sentiment/) contains a collection of 50,000 reviews from IMDB. The dataset contains an even number of positive and negative reviews. The authors considered only highly polarized reviews. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. Neutral reviews are not included in the dataset. The dataset is divided into training and test sets. The training set is the same 25,000 labeled reviews.\n",
    "\n",
    "The **sentiment classification task** consists of predicting the polarity (positive or negative) of a given text.\n",
    "\n",
    "To get the dataset, in your terminal run the following commands:\n",
    "\n",
    "`wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz`\n",
    "\n",
    "`gunzip aclImdb_v1.tar.gz`\n",
    "\n",
    "`tar -xvf aclImdb_v1.tar`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenizing and term document matrix creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH='data/aclImdb/'\n",
    "names = ['neg','pos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aclImdb_v1.tar.gz  imdbEr.txt  imdb.vocab  \u001b[0m\u001b[01;34mmodels\u001b[0m/  README  \u001b[01;34mtest\u001b[0m/  \u001b[01;34mtmp\u001b[0m/  \u001b[01;34mtrain\u001b[0m/\r\n"
     ]
    }
   ],
   "source": [
    "%ls {PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;34maclImdb\u001b[0m/  \u001b[01;34mall_val\u001b[0m/         \u001b[01;34mneg\u001b[0m/  \u001b[01;34mtmp\u001b[0m/    unsupBow.feat  urls_pos.txt\r\n",
      "\u001b[01;34mall\u001b[0m/      labeledBow.feat  \u001b[01;34mpos\u001b[0m/  \u001b[01;34munsup\u001b[0m/  urls_neg.txt   urls_unsup.txt\r\n"
     ]
    }
   ],
   "source": [
    "%ls {PATH}train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0_9.txt\r\n",
      "10000_8.txt\r\n",
      "10001_10.txt\r\n",
      "10002_7.txt\r\n",
      "10003_8.txt\r\n",
      "10004_8.txt\r\n",
      "10005_7.txt\r\n",
      "10006_7.txt\r\n",
      "10007_7.txt\r\n",
      "10008_7.txt\r\n",
      "ls: write error\r\n"
     ]
    }
   ],
   "source": [
    "%ls {PATH}train/pos | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn,trn_y = texts_labels_from_folders(f'{PATH}train',names)\n",
    "val,val_y = texts_labels_from_folders(f'{PATH}test',names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the text of the first review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Story of a man who has unnatural feelings for a pig. Starts out with a opening scene that is a terrific example of absurd comedy. A formal orchestra audience is turned into an insane, violent mob by the crazy chantings of it's singers. Unfortunately it stays absurd the WHOLE time with no general narrative eventually making it just too off putting. Even those from the era should be turned off. The cryptic dialogue would make Shakespeare seem easy to a third grader. On a technical level it's better than you might think with some good cinematography by future great Vilmos Zsigmond. Future stars Sally Kirkland and Frederic Forrest can be seen briefly.\""
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`CountVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) converts a collection of text documents to a matrix of token counts (part of `sklearn.feature_extraction.text`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "veczr = CountVectorizer(tokenizer=tokenize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fit_transform(trn)` finds the vocabulary in the training set. It also transforms the training set into a term-document matrix. Since we have to apply the *same transformation* to your validation set, the second line uses just the method `transform(val)`. `trn_term_doc` and `val_term_doc` are sparse matrices. `trn_term_doc[i]` represents training document i and it contains a count of words for each document for each word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_term_doc = veczr.fit_transform(trn)\n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<25000x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 3749745 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x75132 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 93 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aussie', 'aussies', 'austen', 'austeniana', 'austens']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = veczr.get_feature_names(); vocab[5000:5005]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a',\n",
       " 'absurd',\n",
       " 'an',\n",
       " 'and',\n",
       " 'audience',\n",
       " 'be',\n",
       " 'better',\n",
       " 'briefly.',\n",
       " 'by',\n",
       " 'can',\n",
       " 'chantings',\n",
       " 'cinematography',\n",
       " 'comedy.',\n",
       " 'crazy',\n",
       " 'cryptic',\n",
       " 'dialogue',\n",
       " 'easy',\n",
       " 'era',\n",
       " 'even',\n",
       " 'eventually',\n",
       " 'example',\n",
       " 'feelings',\n",
       " 'for',\n",
       " 'formal',\n",
       " 'forrest',\n",
       " 'frederic',\n",
       " 'from',\n",
       " 'future',\n",
       " 'general',\n",
       " 'good',\n",
       " 'grader.',\n",
       " 'great',\n",
       " 'has',\n",
       " 'insane,',\n",
       " 'into',\n",
       " 'is',\n",
       " 'it',\n",
       " \"it's\",\n",
       " 'just',\n",
       " 'kirkland',\n",
       " 'level',\n",
       " 'make',\n",
       " 'making',\n",
       " 'man',\n",
       " 'might',\n",
       " 'mob',\n",
       " 'narrative',\n",
       " 'no',\n",
       " 'of',\n",
       " 'off',\n",
       " 'off.',\n",
       " 'on',\n",
       " 'opening',\n",
       " 'orchestra',\n",
       " 'out',\n",
       " 'pig.',\n",
       " 'putting.',\n",
       " 'sally',\n",
       " 'scene',\n",
       " 'seem',\n",
       " 'seen',\n",
       " 'shakespeare',\n",
       " 'should',\n",
       " 'singers.',\n",
       " 'some',\n",
       " 'stars',\n",
       " 'starts',\n",
       " 'stays',\n",
       " 'story',\n",
       " 'technical',\n",
       " 'terrific',\n",
       " 'than',\n",
       " 'that',\n",
       " 'the',\n",
       " 'think',\n",
       " 'third',\n",
       " 'those',\n",
       " 'time',\n",
       " 'to',\n",
       " 'too',\n",
       " 'turned',\n",
       " 'unfortunately',\n",
       " 'unnatural',\n",
       " 'vilmos',\n",
       " 'violent',\n",
       " 'who',\n",
       " 'whole',\n",
       " 'with',\n",
       " 'would',\n",
       " 'you',\n",
       " 'zsigmond.'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w0 = set([o.lower() for o in trn[0].split(' ')]); w0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(w0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1297"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veczr.vocabulary_['absurd']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0,1297]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc[0,5000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the **log-count ratio** $r$ for each word $f$:\n",
    "\n",
    "$r = \\log \\frac{\\text{ratio of feature $f$ in positive documents}}{\\text{ratio of feature $f$ in negative documents}}$\n",
    "\n",
    "where ratio of feature $f$ in positive documents is the number of times a positive document has a feature divided by the number of positive documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pr(y_i):\n",
    "    p = x[y==y_i].sum(0)\n",
    "    return (p+1) / ((y==y_i).sum()+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=trn_term_doc\n",
    "y=trn_y\n",
    "\n",
    "r = np.log(pr(1)/pr(0))\n",
    "b = np.log((y==1).mean() / (y==0).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the formula for Naive Bayes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80691999999999997"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_preds = val_term_doc @ r.T + b\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and binarized Naive Bayes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.83016000000000001"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=trn_term_doc.sign()\n",
    "r = np.log(pr(1)/pr(0))\n",
    "\n",
    "pre_preds = val_term_doc.sign() @ r.T + b\n",
    "preds = pre_preds.T>0\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is how we can fit logistic regression where the features are the unigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85504000000000002"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=1e8, dual=True)\n",
    "m.fit(x, y)\n",
    "preds = m.predict(val_term_doc)\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85487999999999997"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=1e8, dual=True)\n",
    "m.fit(trn_term_doc.sign(), y)\n",
    "preds = m.predict(val_term_doc.sign())\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and the regularized version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88275999999999999"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(x, y)\n",
    "preds = m.predict(val_term_doc)\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88404000000000005"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(trn_term_doc.sign(), y)\n",
    "preds = m.predict(val_term_doc.sign())\n",
    "(preds==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigram with NB features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our next model is a version of logistic regression with Naive Bayes features described [here](https://www.aclweb.org/anthology/P12-2018). For every document we compute binarized features as described above, but this time we use bigrams and trigrams too. Each feature is a log-count ratio. A logistic regression model is then trained to predict sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "veczr =  CountVectorizer(ngram_range=(1,3), tokenizer=tokenize, max_features=800000)\n",
    "trn_term_doc = veczr.fit_transform(trn)\n",
    "val_term_doc = veczr.transform(val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 800000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_term_doc.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = veczr.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['by vast', 'by vengeance', 'by vengeance .', 'by vera', 'by vera miles']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[200000:200005]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y=trn_y\n",
    "x=trn_term_doc.sign()\n",
    "val_x = val_term_doc.sign()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = np.log(pr(1) / pr(0))\n",
    "b = np.log((y==1).mean() / (y==0).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we fit regularized logistic regression where the features are the trigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.90500000000000003"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = LogisticRegression(C=0.1, dual=True)\n",
    "m.fit(x, y);\n",
    "\n",
    "preds = m.predict(val_x)\n",
    "(preds.T==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the $\\text{log-count ratio}$ `r`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1, 800000),\n",
       " matrix([[-0.05468, -0.161  , -0.24784, ...,  1.09861, -0.69315, -0.69315]]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.shape, r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[ 0.94678,  0.85129,  0.78049, ...,  3.     ,  0.5    ,  0.5    ]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we fit regularized logistic regression where the features are the trigrams' log-count ratios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.91768000000000005"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_nb = x.multiply(r)\n",
    "m = LogisticRegression(dual=True, C=0.1)\n",
    "m.fit(x_nb, y);\n",
    "\n",
    "val_x_nb = val_x.multiply(r)\n",
    "preds = m.predict(val_x_nb)\n",
    "(preds.T==val_y).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## fastai NBSVM++"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sl=2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is how we get a model from a bag of words\n",
    "md = TextClassifierData.from_bow(trn_term_doc, trn_y, val_term_doc, val_y, sl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "075afdb7ebee432bb4917146014ebfcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.0251   0.12003  0.91552]                          \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner = md.dotprod_nb_learner()\n",
    "learner.fit(0.02, 1, wds=1e-6, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fbcc4a674a5a46fca621e6d2244d9511",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.02014  0.11387  0.92012]                         \n",
      "[ 1.       0.01275  0.11149  0.92124]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner.fit(0.02, 2, wds=1e-6, cycle_len=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6010f5e493984a6991da1c3e7079e72a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.01681  0.11089  0.92129]                           \n",
      "[ 1.       0.00949  0.10951  0.92223]                          \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learner.fit(0.02, 2, wds=1e-6, cycle_len=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Baselines and Bigrams: Simple, Good Sentiment and Topic Classification. Sida Wang and Christopher D. Manning [pdf](https://www.aclweb.org/anthology/P12-2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}