{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Tutorial: automatic summarization using Gensim</h1>\n",
    "\n",
    "This module automatically summarizes the given text, by extracting one or more important sentences from the text. In a similar way, it can also extract keywords. This tutorial will teach you to use this summarization module via some examples. First, we will try a small example, then we will try two larger ones, and then we will review the performance of the summarizer in terms of speed.\n",
    "\n",
    "This summarizer is based on the \"TextRank\" algorithm, from an [article](http://web.eecs.umich.edu/%7Emihalcea/papers/mihalcea.emnlp04.pdf) by Mihalcea et al. This algorithm was later improved upon by Barrios et al. in another [article](https://raw.githubusercontent.com/summanlp/docs/master/articulo/articulo-en.pdf), by introducing something called a \"BM25 ranking function\". \n",
    "\n",
    "This tutorial assumes that you are familiar with Python and have [installed Gensim](http://radimrehurek.com/gensim/install.html).\n",
    "\n",
    "<b>Note</b>: Gensim's summarization only works for English for now, because the text is pre-processed so that stopwords are removed and the words are stemmed, and these processes are language-dependent.\n",
    "\n",
    "\n",
    "<h2>Small example</h2>\n",
    "\n",
    "First of all, we import the function \"summarize\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,358 : INFO : Pattern library is not installed, lemmatization won't be available.\n",
      "2016-09-19 12:45:22,361 : INFO : Could not import Theano, will use standard float for default ShardedCorpus dtype.\n",
      "2016-09-19 12:45:22,372 : INFO : 'pattern' package not found; tag filters are not available for English\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "\n",
    "from gensim.summarization import summarize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try summarizing a small toy example; later we will use a larger piece of text. In reality, the text is too small, but it suffices as an illustrative example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input text:\n",
      "Thomas A. Anderson is a man living two lives. By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination. Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government. Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix. As a rebel against the machines, Neo must return to the Matrix and confront the agents: super-powerful computer programs devoted to snuffing out Neo and the entire human rebellion. \n"
     ]
    }
   ],
   "source": [
    "text = \"Thomas A. Anderson is a man living two lives. By day he is an \" + \\\n",
    "    \"average computer programmer and by night a hacker known as \" + \\\n",
    "    \"Neo. Neo has always questioned his reality, but the truth is \" + \\\n",
    "    \"far beyond his imagination. Neo finds himself targeted by the \" + \\\n",
    "    \"police when he is contacted by Morpheus, a legendary computer \" + \\\n",
    "    \"hacker branded a terrorist by the government. Morpheus awakens \" + \\\n",
    "    \"Neo to the real world, a ravaged wasteland where most of \" + \\\n",
    "    \"humanity have been captured by a race of machines that live \" + \\\n",
    "    \"off of the humans' body heat and electrochemical energy and \" + \\\n",
    "    \"who imprison their minds within an artificial reality known as \" + \\\n",
    "    \"the Matrix. As a rebel against the machines, Neo must return to \" + \\\n",
    "    \"the Matrix and confront the agents: super-powerful computer \" + \\\n",
    "    \"programs devoted to snuffing out Neo and the entire human \" + \\\n",
    "    \"rebellion. \"\n",
    "\n",
    "print ('Input text:')\n",
    "print (text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize this text, we pass the <b>raw string data</b> as input to the function \"summarize\", and it will return a summary.\n",
    "\n",
    "Note: make sure that the string does not contain any newlines where the line breaks in a sentence. A sentence with a newline in it (i.e. a carriage return, \"\\n\") will be treated as two sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,405 : WARNING : Input text is expected to have at least 10 sentences.\n",
      "2016-09-19 12:45:22,405 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2016-09-19 12:45:22,406 : INFO : built Dictionary(53 unique tokens: ['realiti', 'averag', 'polic', 'legendari', 'hacker']...) from 6 documents (total 68 corpus positions)\n",
      "2016-09-19 12:45:22,406 : WARNING : Input corpus is expected to have at least 10 documents.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "print ('Summary:')\n",
    "print (summarize(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the \"split\" option if you want a list of strings instead of a single string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,428 : WARNING : Input text is expected to have at least 10 sentences.\n",
      "2016-09-19 12:45:22,428 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2016-09-19 12:45:22,429 : INFO : built Dictionary(53 unique tokens: ['realiti', 'averag', 'polic', 'legendari', 'hacker']...) from 6 documents (total 68 corpus positions)\n",
      "2016-09-19 12:45:22,430 : WARNING : Input corpus is expected to have at least 10 documents.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\"]\n"
     ]
    }
   ],
   "source": [
    "print (summarize(text, split=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can adjust how much text the summarizer outputs via the \"ratio\" parameter or the \"word_count\" parameter. Using the \"ratio\" parameter, you specify what fraction of sentences in the original text should be returned as output. Below we specify that we want 50% of the original text (the default is 20%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,446 : WARNING : Input text is expected to have at least 10 sentences.\n",
      "2016-09-19 12:45:22,446 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2016-09-19 12:45:22,447 : INFO : built Dictionary(53 unique tokens: ['realiti', 'averag', 'polic', 'legendari', 'hacker']...) from 6 documents (total 68 corpus positions)\n",
      "2016-09-19 12:45:22,447 : WARNING : Input corpus is expected to have at least 10 documents.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination.\n",
      "Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government.\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "print ('Summary:')\n",
    "print (summarize(text, ratio=0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the \"word_count\" parameter, we specify the maximum amount of words we want in the summary. Below we have specified that we want no more than 50 words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,463 : WARNING : Input text is expected to have at least 10 sentences.\n",
      "2016-09-19 12:45:22,464 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2016-09-19 12:45:22,464 : INFO : built Dictionary(53 unique tokens: ['realiti', 'averag', 'polic', 'legendari', 'hacker']...) from 6 documents (total 68 corpus positions)\n",
      "2016-09-19 12:45:22,465 : WARNING : Input corpus is expected to have at least 10 documents.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
     ]
    }
   ],
   "source": [
    "print ('Summary:')\n",
    "print (summarize(text, word_count=50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned earlier, this module also supports <b>keyword</b> extraction. Keyword extraction works in the same way as summary generation (i.e. sentence extraction), in that the algorithm tries to find words that are important or seem representative of the entire text. They keywords are not always single words; in the case of multi-word keywords, they are typically all nouns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keywords:\n",
      "humanity\n",
      "human\n",
      "neo\n",
      "humans body\n",
      "super\n",
      "reality\n",
      "hacker\n"
     ]
    }
   ],
   "source": [
    "from gensim.summarization import keywords\n",
    "\n",
    "print ('Keywords:')\n",
    "print (keywords(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Larger example</h2>\n",
    "\n",
    "Let us try an example with a larger piece of text. We will be using a synopsis of the movie \"The Matrix\", which we have taken from [this](http://www.imdb.com/title/tt0133093/synopsis?ref_=ttpl_pl_syn) IMDb page.\n",
    "\n",
    "In the code below, we read the text file directly from a web-page using \"requests\". Then we produce a summary and some keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:22,510 : INFO : Starting new HTTP connection (1): rare-technologies.com\n",
      "2016-09-19 12:45:23,035 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2016-09-19 12:45:23,042 : INFO : built Dictionary(1093 unique tokens: ['realiti', 'keanu', 'miseri', 'vestig', 'massiv']...) from 416 documents (total 2985 corpus positions)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:\n",
      "Anderson, a software engineer for a Metacortex, the other life as Neo, a computer hacker \"guilty of virtually every computer crime we have a law for.\" Agent Smith asks him to help them capture Morpheus, a dangerous terrorist, in exchange for amnesty.\n",
      "Morpheus explains that he's been searching for Neo his entire life and asks if Neo feels like \"Alice in Wonderland, falling down the rabbit hole.\" He explains to Neo that they exist in the Matrix, a false reality that has been constructed for humans to hide the truth.\n",
      "Neo is introduced to Morpheus's crew including Trinity; Apoc (Julian Arahanga), a man with long, flowing black hair; Switch; Cypher (bald with a goatee); two brawny brothers, Tank (Marcus Chong) and Dozer (Anthony Ray Parker); and a young, thin man named Mouse (Matt Doran).\n",
      "Cypher cuts up a juicy steak and ruminates that he knows the steak is merely the simulation telling his brain that it is delicious and juicy, but after nine years he has discovered that \"ignorance is bliss.\" He strikes a deal for the machines to reinsert his body into a power plant, reinsert him into the Matrix, and he'll help the Agents.\n",
      "\n",
      "Keywords:\n",
      "neo\n",
      "morpheus\n",
      "trinity\n",
      "cypher\n",
      "agents\n",
      "agent\n",
      "smith\n",
      "tank\n",
      "says\n",
      "saying\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "text = requests.get('http://rare-technologies.com/the_matrix_synopsis.txt').text\n",
    "\n",
    "print ('Summary:')\n",
    "print (summarize(text, ratio=0.01))\n",
    "\n",
    "print ('\\nKeywords:')\n",
    "print (keywords(text, ratio=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you know this movie, you see that this summary is actually quite good. We also see that some of the most important characters (Neo, Morpheus, Trinity) were extracted as keywords.\n",
    "\n",
    "<h2>Another example</h2>\n",
    "\n",
    "Let's try an example similar to the one above. This time, we will use the [IMDb synopsis](http://www.imdb.com/title/tt0118715/synopsis?ref_=tt_stry_pl) of \"The Big Lebowski\".\n",
    "\n",
    "Again, we download the text and produce a summary and some keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2016-09-19 12:45:25,227 : INFO : Starting new HTTP connection (1): rare-technologies.com\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "text = requests.get('http://rare-technologies.com/the_big_lebowski_synopsis.txt').text\n",
    "\n",
    "print ('Summary:')\n",
    "print (summarize(text, ratio=0.01))\n",
    "\n",
    "print ('\\nKeywords:')\n",
    "print (keywords(text, ratio=0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time around, the summary is not of high quality, as it does not tell us much about the movie. In a way, this might not be the algorithms fault, rather this text simply doesn't contain one or two sentences that capture the essence of the text as in \"The Matrix\" synopsis.\n",
    "\n",
    "The keywords, however, managed to find some of the main characters.\n",
    "\n",
    "<h2>Performance</h2>\n",
    "\n",
    "We will test how the speed of the summarizer scales with the size of the dataset. These tests were run on an Intel Core i5 4210U CPU @ 1.70 GHz x 4 processor. Note that the summarizer does <i>not</i> support multithreading (parallel processing).\n",
    "\n",
    "The tests were run on the book \"Honest Abe\" by Alonzo Rothschild. Download the book in plain-text <a href=\"http://www.gutenberg.org/ebooks/49679\">here</a>. \n",
    "\n",
    "In the <b>plot below</b>, we see the running times together with the sizes of the datasets. To create datasets of different sizes, we have simply taken prefixes of text; in other words we take the first <i>n</i> characters of the book. The algorithm seems to be <b>quadratic in time</b>, so one needs to be careful before plugging a large dataset into the summarizer.\n",
    "\n",
    "<figure>\n",
    "<img src=\"http://rare-technologies.com/summarization_tutorial_plot.png\">\n",
    "    <figcaption></figcaption>\n",
    "</figure>\n",
    "\n",
    "<h3>Text-content dependent running times</h3>\n",
    "\n",
    "The running time is not only dependent on the size of the dataset. For example, summarizing \"The Matrix\" synopsis (about 36,000 characters) takes about 3.1 seconds, while summarizing 35,000 characters of this book takes about 8.5 seconds. So the former is <i>more than twice as fast</i>. \n",
    "\n",
    "One reason for this difference in running times is the data structure that is used. The algorithm represents the data using a graph, where vertices (nodes) are sentences, and then constructs weighted edges between the vertices that represent how the sentences relate to each other. This means that every piece of text will have a different graph, thus making the running times different. The size of this data structure is <i>quadratic in the worst case</i> (the worst case is when each vertex has an edge to every other vertex).\n",
    "\n",
    "Another possible reason for the difference in running times is that the problems converge at different rates, meaning that the error drops slower for some datasets than for others.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}