{
 "metadata": {
  "name": "",
  "signature": "sha256:c8c84d6b3c06f261c203b390bc25b88c613c46653ce980616d623e31480a0fd0"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Naive Bayes Classification: United States Congressional Record"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The Naive Bayes classifier is a statsitcal programing tool with diverse uses in text analysis. It is famously used for determining if an email is \"spam\" or \"ham\"(the opposite of spam). It also has uses in sentiment analysis, for example, determining whether a movie review was good or bad, determining the authorship of an article, or categorizing documents into topics. Today we will be using a naive bayes classifier however, not to classify, but to analyze competing sides of political debates on the floor of the US Congress."
     ]
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Import Necessary Tools"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pylab inline \n",
      "\n",
      "from __future__ import division\n",
      "\n",
      "from sklearn.feature_extraction.text import CountVectorizer\n",
      "from sklearn.feature_extraction.text import TfidfVectorizer\n",
      "from sklearn.naive_bayes import MultinomialNB\n",
      "from sklearn.naive_bayes import GaussianNB\n",
      "from sklearn.naive_bayes import BernoulliNB\n",
      "from sklearn.cross_validation import cross_val_score\n",
      "from sklearn.cross_validation import train_test_split\n",
      "\n",
      "from speech import Speech"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Query Database for Speeches"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Twiddle with the parameters and see how many speeches you can retireve for the subset of the congressional record you are interested in analyzing. Good machine leanring generally requires large samplesizes so try to go for a few thousand speeches. Pick a topic that will classify well (i.e. that will contrast starkly between democrats and republicans)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "phrase = \"abortion\" \n",
      "num_speeches = Speech.get(0, 0, phrase=phrase, congress=\"\", start_date=\"1995-05-04\", speaker_party=\"*\")['count']\n",
      "print \"Downloading %i speeches\" % num_speeches\n",
      "speeches = Speech.get(start=0, rows=num_speeches, phrase=phrase, speaker_party=\"*\")['speeches']\n",
      "print len(speeches), \"speeches downloaded\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Perpare the tools we will use to do our text analysis"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "naive_bayes = MultinomialNB(alpha=1.0, fit_prior=True)\n",
      "vectorizer = TfidfVectorizer(min_df=.1, max_df=.6, stop_words='english' )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Perpare data for analysis"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Make an array of text objects. Each chunk of text is just the text of a congressional speech.\n",
      "data = [\" \".join(speech['speaking']) for speech in speeches]\n",
      "\n",
      "# Transform speeches into vectors\n",
      "data = vectorizer.fit_transform(data)\n",
      "\n",
      "# Make an array of 0s and 1s that determine if each speech is democrat or republican. This is called the target vector.\n",
      "target = [speech['speaker_party'] for speech in speeches]\n",
      "target = [ 0 if x == \"D\" else 1 for x in target ]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What the data vector looks like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.shape"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What the target vector looks like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "target"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 6,
     "metadata": {},
     "source": [
      "Building a Training and Testing set"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Classifiers generally are split into two parts. The \"training set\" which the algorithm uses to learn to recognize a pattern, and the \"testing set\" which we give the algorithm as an unknown. The naive bayes algorithm uses what it learns from the training set to attempt to classify items in the testing set. In this case, even though we know the correct 'class' or categorization of each speech, we split our data 80/20 and assume for the 20% test set that we don't know which class it belongs in."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "X_train, X_test , Y_train, Y_test = train_test_split(data, target, test_size=0.2)\n",
      "print X_train.shape, X_test.shape, len(Y_train), len(Y_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Run the Classifier"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A naive bayes classifier works by adding up the conidtional probabilities of a word being in a particular class.\n",
      "\n",
      "P(party|word) = P(word|party) * P(party)\n",
      "P(Democrat   | 'choice') = P('choice' | Democrat)   * P (Democrat)   / P('choice') # Probably pretty high\n",
      "P(Republican | 'choice') = P('choice' | Republican) * P (Republican) / P('choice') # Probably fairly low\n",
      "\n",
      "When you add up the conditional probabilities P(party|word) for every word in a document, you get the probability that the document in question comes from a member of party P. In short\n",
      "\n",
      "P( party | word1 & word2 & word3 ... & word-n ) is the same as P( party | document )\n",
      "\n",
      "\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "naive_bayes.fit(X_train, Y_train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "See if the classifier is any good"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In our case, we knew the correct 'class' of 100% of our data, but we assumed that 20% was unknown and asked the classifier to figure it out using the patterns it learned. Now we can see how accurately the classifier classified the data into our two classes (Democrat and Republican). Since we're only classifying into two classes, random guesses would land us at 50%. In order to continue we must ensure that the classifier is doing a fair bit better than chance."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "naive_bayes.score(X_test, Y_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Cross validation is a more robust method of checking if a classifier is correct or not. It runs the above simulations several times on many different subsets of the data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cross_val_score(naive_bayes, data, target, scoring='accuracy', verbose=1, cv=5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 5,
     "metadata": {},
     "source": [
      "Observe words most indicative if a speech is DEM or REP speech"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "terms = vectorizer.get_feature_names()\n",
      "t1 = [(naive_bayes.feature_log_prob_[0][i] * (naive_bayes.class_count_[0] / naive_bayes.class_count_.sum())) for i in range(len(terms))]\n",
      "t2 = [(naive_bayes.feature_log_prob_[1][i] * (naive_bayes.class_count_[1] / naive_bayes.class_count_.sum())) for i in range(len(terms))]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "[(terms[i],t1[i]) for i in np.array(t1).argsort()] # Top Terms for Republicans"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "[(terms[i],t2[i]) for i in np.array(t2).argsort()] # Top Terms for Democrats"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Brainstrom"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "What other contexts might you be able to apply this methodology to? How can a classifier be useful in your work? How about the particular way in which we were able to find the words most indicative of a certain class?"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}