\n",
"\n",
"This module automatically summarizes the given text, by extracting one or more important sentences from the text. In a similar way, it can also extract keywords. This tutorial will teach you to use this summarization module via some examples. First, we will try a small example, then we will try two larger ones, and then we will review the performance of the summarizer in terms of speed.\n",
"\n",
"This summarizer is based on the \"TextRank\" algorithm, from an [article](http://web.eecs.umich.edu/%7Emihalcea/papers/mihalcea.emnlp04.pdf) by Mihalcea et al. This algorithm was later improved upon by Barrios et al. in another [article](https://raw.githubusercontent.com/summanlp/docs/master/articulo/articulo-en.pdf), by introducing something called a \"BM25 ranking function\". \n",
"\n",
"This tutorial assumes that you are familiar with Python and have [installed Gensim](http://radimrehurek.com/gensim/install.html).\n",
"\n",
"Note: Gensim's summarization only works for English for now, because the text is pre-processed so that stopwords are removed and the words are stemmed, and these processes are language-dependent.\n",
"\n",
"\n",
"
Small example
\n",
"\n",
"First of all, we import the function \"summarize\"."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:40,713 : INFO : 'pattern' package not found; tag filters are not available for English\n"
]
}
],
"source": [
"import logging\n",
"logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
"\n",
"from gensim.summarization import summarize"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will try summarizing a small toy example; later we will use a larger piece of text. In reality, the text is too small, but it suffices as an illustrative example."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input text:\n",
"Thomas A. Anderson is a man living two lives. By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination. Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government. Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix. As a rebel against the machines, Neo must return to the Matrix and confront the agents: super-powerful computer programs devoted to snuffing out Neo and the entire human rebellion. \n"
]
}
],
"source": [
"text = \"Thomas A. Anderson is a man living two lives. By day he is an \" + \\\n",
" \"average computer programmer and by night a hacker known as \" + \\\n",
" \"Neo. Neo has always questioned his reality, but the truth is \" + \\\n",
" \"far beyond his imagination. Neo finds himself targeted by the \" + \\\n",
" \"police when he is contacted by Morpheus, a legendary computer \" + \\\n",
" \"hacker branded a terrorist by the government. Morpheus awakens \" + \\\n",
" \"Neo to the real world, a ravaged wasteland where most of \" + \\\n",
" \"humanity have been captured by a race of machines that live \" + \\\n",
" \"off of the humans' body heat and electrochemical energy and \" + \\\n",
" \"who imprison their minds within an artificial reality known as \" + \\\n",
" \"the Matrix. As a rebel against the machines, Neo must return to \" + \\\n",
" \"the Matrix and confront the agents: super-powerful computer \" + \\\n",
" \"programs devoted to snuffing out Neo and the entire human \" + \\\n",
" \"rebellion. \"\n",
"\n",
"print ('Input text:')\n",
"print (text)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To summarize this text, we pass the raw string data as input to the function \"summarize\", and it will return a summary.\n",
"\n",
"Note: make sure that the string does not contain any newlines where the line breaks in a sentence. A sentence with a newline in it (i.e. a carriage return, \"\\n\") will be treated as two sentences."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:40,736 : WARNING : Input text is expected to have at least 10 sentences.\n",
"2017-11-28 08:32:40,737 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:40,738 : INFO : built Dictionary(53 unique tokens: [u'electrochem', u'real', u'captur', u'mind', u'agent']...) from 6 documents (total 68 corpus positions)\n",
"2017-11-28 08:32:40,738 : WARNING : Input corpus is expected to have at least 10 documents.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary:\n",
"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
]
}
],
"source": [
"print ('Summary:')\n",
"print (summarize(text))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the \"split\" option if you want a list of strings instead of a single string."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:40,748 : WARNING : Input text is expected to have at least 10 sentences.\n",
"2017-11-28 08:32:40,749 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:40,750 : INFO : built Dictionary(53 unique tokens: [u'electrochem', u'real', u'captur', u'mind', u'agent']...) from 6 documents (total 68 corpus positions)\n",
"2017-11-28 08:32:40,751 : WARNING : Input corpus is expected to have at least 10 documents.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\"]\n"
]
}
],
"source": [
"print (summarize(text, split=True))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can adjust how much text the summarizer outputs via the \"ratio\" parameter or the \"word_count\" parameter. Using the \"ratio\" parameter, you specify what fraction of sentences in the original text should be returned as output. Below we specify that we want 50% of the original text (the default is 20%)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:40,761 : WARNING : Input text is expected to have at least 10 sentences.\n",
"2017-11-28 08:32:40,761 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:40,762 : INFO : built Dictionary(53 unique tokens: [u'electrochem', u'real', u'captur', u'mind', u'agent']...) from 6 documents (total 68 corpus positions)\n",
"2017-11-28 08:32:40,763 : WARNING : Input corpus is expected to have at least 10 documents.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary:\n",
"By day he is an average computer programmer and by night a hacker known as Neo. Neo has always questioned his reality, but the truth is far beyond his imagination.\n",
"Neo finds himself targeted by the police when he is contacted by Morpheus, a legendary computer hacker branded a terrorist by the government.\n",
"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
]
}
],
"source": [
"print ('Summary:')\n",
"print (summarize(text, ratio=0.5))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the \"word_count\" parameter, we specify the maximum amount of words we want in the summary. Below we have specified that we want no more than 50 words."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:40,774 : WARNING : Input text is expected to have at least 10 sentences.\n",
"2017-11-28 08:32:40,775 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:40,776 : INFO : built Dictionary(53 unique tokens: [u'electrochem', u'real', u'captur', u'mind', u'agent']...) from 6 documents (total 68 corpus positions)\n",
"2017-11-28 08:32:40,777 : WARNING : Input corpus is expected to have at least 10 documents.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary:\n",
"Morpheus awakens Neo to the real world, a ravaged wasteland where most of humanity have been captured by a race of machines that live off of the humans' body heat and electrochemical energy and who imprison their minds within an artificial reality known as the Matrix.\n"
]
}
],
"source": [
"print ('Summary:')\n",
"print (summarize(text, word_count=50))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As mentioned earlier, this module also supports keyword extraction. Keyword extraction works in the same way as summary generation (i.e. sentence extraction), in that the algorithm tries to find words that are important or seem representative of the entire text. They keywords are not always single words; in the case of multi-word keywords, they are typically all nouns."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Keywords:\n",
"humanity\n",
"human\n",
"neo\n",
"humans body\n",
"super\n",
"hacker\n",
"reality\n"
]
}
],
"source": [
"from gensim.summarization import keywords\n",
"\n",
"print ('Keywords:')\n",
"print (keywords(text))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"
Larger example
\n",
"\n",
"Let us try an example with a larger piece of text. We will be using a synopsis of the movie \"The Matrix\", which we have taken from [this](http://www.imdb.com/title/tt0133093/synopsis?ref_=ttpl_pl_syn) IMDb page.\n",
"\n",
"In the code below, we read the text file directly from a web-page using \"requests\". Then we produce a summary and some keywords."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:41,320 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:41,326 : INFO : built Dictionary(1093 unique tokens: [u'code', u'squiddi', u'relai', u'dinosaur', u'electron']...) from 416 documents (total 2985 corpus positions)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary:\n",
"Anderson, a software engineer for a Metacortex, the other life as Neo, a computer hacker \"guilty of virtually every computer crime we have a law for.\" Agent Smith asks him to help them capture Morpheus, a dangerous terrorist, in exchange for amnesty.\n",
"Morpheus explains that he's been searching for Neo his entire life and asks if Neo feels like \"Alice in Wonderland, falling down the rabbit hole.\" He explains to Neo that they exist in the Matrix, a false reality that has been constructed for humans to hide the truth.\n",
"Neo is introduced to Morpheus's crew including Trinity; Apoc (Julian Arahanga), a man with long, flowing black hair; Switch; Cypher (bald with a goatee); two brawny brothers, Tank (Marcus Chong) and Dozer (Anthony Ray Parker); and a young, thin man named Mouse (Matt Doran).\n",
"Cypher cuts up a juicy steak and ruminates that he knows the steak is merely the simulation telling his brain that it is delicious and juicy, but after nine years he has discovered that \"ignorance is bliss.\" He strikes a deal for the machines to reinsert his body into a power plant, reinsert him into the Matrix, and he'll help the Agents.\n",
"\n",
"Keywords:\n",
"neo\n",
"morpheus\n",
"trinity\n",
"cypher\n",
"agents\n",
"agent\n",
"smith\n",
"tank\n",
"says\n",
"saying\n"
]
}
],
"source": [
"import requests\n",
"\n",
"text = requests.get('http://rare-technologies.com/the_matrix_synopsis.txt').text\n",
"\n",
"print ('Summary:')\n",
"print (summarize(text, ratio=0.01))\n",
"\n",
"print ('\\nKeywords:')\n",
"print (keywords(text, ratio=0.01))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you know this movie, you see that this summary is actually quite good. We also see that some of the most important characters (Neo, Morpheus, Trinity) were extracted as keywords.\n",
"\n",
"
Another example
\n",
"\n",
"Let's try an example similar to the one above. This time, we will use the [IMDb synopsis](http://www.imdb.com/title/tt0118715/synopsis?ref_=tt_stry_pl) of \"The Big Lebowski\".\n",
"\n",
"Again, we download the text and produce a summary and some keywords."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2017-11-28 08:32:43,682 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
"2017-11-28 08:32:43,687 : INFO : built Dictionary(1054 unique tokens: [u'fawn', u'windi', u'concept', u'doctor', u'gant']...) from 227 documents (total 2434 corpus positions)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Summary:\n",
"The answering machine records a woman introducing herself as Maude Lebowski and saying that she is the one who took his rug and has sent a car to pick Dude up at his apartment.\n",
"As he climbs out of bed to make a White Russian, Maude asks about the apartment and Dude explains that Treehorn's thugs most likely vandalized it looking for Lebowski's money.\n",
"\n",
"Keywords:\n",
"dude\n",
"dudes\n",
"walter\n",
"lebowski\n",
"brandt\n",
"maude\n",
"donny\n",
"bunny\n"
]
}
],
"source": [
"import requests\n",
"\n",
"text = requests.get('http://rare-technologies.com/the_big_lebowski_synopsis.txt').text\n",
"\n",
"print ('Summary:')\n",
"print (summarize(text, ratio=0.01))\n",
"\n",
"print ('\\nKeywords:')\n",
"print (keywords(text, ratio=0.01))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This time around, the summary is not of high quality, as it does not tell us much about the movie. In a way, this might not be the algorithms fault, rather this text simply doesn't contain one or two sentences that capture the essence of the text as in \"The Matrix\" synopsis.\n",
"\n",
"The keywords, however, managed to find some of the main characters.\n",
"\n",
"
Performance
\n",
"\n",
"We will test how the speed of the summarizer scales with the size of the dataset. These tests were run on an Intel Core i5 4210U CPU @ 1.70 GHz x 4 processor. Note that the summarizer does not support multithreading (parallel processing).\n",
"\n",
"The tests were run on the book \"Honest Abe\" by Alonzo Rothschild. Download the book in plain-text here. \n",
"\n",
"In the plot below, we see the running times together with the sizes of the datasets. To create datasets of different sizes, we have simply taken prefixes of text; in other words we take the first n characters of the book. The algorithm seems to be quadratic in time, so one needs to be careful before plugging a large dataset into the summarizer.\n",
"\n",
"\n",
"\n",
" \n",
"\n",
"\n",
"
![](\"http://rare-technologies.com/summarization_tutorial_plot.png\")
\n",
"