{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will briefly present *Bayesian spam filtering*, which is an established and effective technique for spam filtering.  The basic idea is to look at documents as *bags of words* (that is, as mappings of words to frequencies, disregarding ordering).  The underlying assumption is that spam and legitimate documents will have different distributions of words, and that we'll be able to rate the probability that a given set of words came from a legitimate document or a spam document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "data = pd.read_parquet(\"data/training.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by splitting our data into randomly-selected train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import model_selection\n",
    "train, test = model_selection.train_test_split(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our feature extraction pipeline is very simple:  we'll use a _bag-of-words_ model in which we represent texts as dictionaries of word counts.  Furthermore, we'll use feature hashing so that we store word counts in a array of hash buckets rather than as an explicit dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import text as text_feature\n",
    "\n",
    "hv = text_feature.HashingVectorizer(norm='l1', alternate_sign=False)\n",
    "hashed_features = hv.transform(train.text.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From there, we'll use the multinomial naive Bayes classifier in scikit-learn, which will train a model to identify which words (really, which hash values of words) are most likely to distinguish between legitimate and spam messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn import naive_bayes\n",
    "\n",
    "nb = naive_bayes.MultinomialNB()\n",
    "nb.fit(hashed_features, train.label.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've fit our model, we can evaluate its accuracy on our training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb.score(hashed_features, train.label.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_features = hv.transform(test.text.values)\n",
    "nb.score(test_features, test.label.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you know, raw accuracy isn't the most useful metric to evaluate a binary classifier.  In order to visualize how the naive Bayes classifier performs overall, we'll plot a confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlworkflows import plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df, chart = plot.binary_confusion_matrix(test.label.values, nb.predict(test_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also inspect the performance in tabular form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can produce a report showing us the precision and recall for each class, as well as an [f<sub>1</sub>-score](https://en.wikipedia.org/wiki/F1_score)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(test.label.values, nb.predict(test_features)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "0.  The preprocessing step in this notebook is virtually nonexistent.  How might you use spaCy to have a more robust preprocessing pipeline?\n",
    "1.  The feature extraction pipeline in this notebook uses hashed vectors with 2<sup>20</sup> elements.  Run some experiments to identify whether or not smaller vectors would still provide acceptable performance.\n",
    "2.  Bayesian document classification is an established technique that has worked well in practice for a long time.  A common way to fool Bayesian spam filters is to append a lot of legitimate text to the end of a spam document.  What sort of features would you extract in order to train a classifier that could identify spam messages that were using this trick?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "jupyter"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}