{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualizing the Corpus of ACL 2017 Paper Titles\n",
"\n",
"There was a significant variation topics covered in long and short papers in ACL 2017. Below is an application of Scattertext, written up in an ACL 2017 demo paper, on visualizing the differences in these paper titles.\n",
"\n",
"Jason S. Kessler. Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ. To appear in ACL Demos. Vancouver, BC. 2017.\n",
"\n",
"https://github.com/JasonKessler/scattertext\n",
"\n",
"https://arxiv.org/abs/1703.00565"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let's first, parse the accepted papers blog post using Beautiful Soup and load it into a Pandas data frame"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"import requests\n",
"from bs4 import BeautifulSoup\n",
"import re\n",
"import pygal\n",
"import scattertext as st\n",
"from IPython.display import IFrame\n",
"from IPython.core.display import display, HTML\n",
"import seaborn as sns\n",
"display(HTML(\"\"))\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"url = 'https://chairs-blog.acl2017.org/2017/04/05/accepted-papers-and-demonstrations/'\n",
"soup = BeautifulSoup(requests.get(url).text, 'lxml')\n",
"content = soup.find_all('div', class_='entry-content')[0]\n",
"\n",
"def split_authors(author_list):\n",
" return re.split('(?:\\W+and|\\,)\\W+', author_list)\n",
"\n",
"assert split_authors('Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Stefan Scherer and Louis-Philippe Morency') \\\n",
" == ['Sayan Ghosh', 'Mathieu Chollet', 'Eugene Laksana', 'Stefan Scherer', 'Louis-Philippe Morency']\n",
"assert split_authors('Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Stefan Scherer, and Louis-Philippe Morency') \\\n",
" == ['Sayan Ghosh', 'Mathieu Chollet', 'Eugene Laksana', 'Stefan Scherer', 'Louis-Philippe Morency']\n",
" \n",
"data = []\n",
"for paper_length in content.find_all('h2'):\n",
" length_name = paper_length.text.strip()\n",
" field = 'N/A'\n",
" for sib in paper_length.findNextSiblings():\n",
" if sib.name == 'h2':\n",
" break\n",
" if sib.name == 'h3':\n",
" field = sib.text.strip()\n",
" if sib.name == 'ol':\n",
" for paper in sib.find_all('li'):\n",
" try:\n",
" title, authors = paper.text.split('Authors:')\n",
" except:\n",
" continue\n",
" data.append({'Paper Type': length_name,\n",
" 'Field': field,\n",
" 'Title': title.strip(),\n",
" 'Authors': split_authors(authors.strip())})\n",
"df = pd.DataFrame(data)\n",
"df['author count'] = df.Authors.apply(len) \n",
"df['Paper Type']= df['Paper Type'].apply(lambda x: {'Accepted Long Papers':'Long', \n",
" 'Accepted Short Papers':'Short',\n",
" 'Accepted Software Demonstrations':'Demo',\n",
" 'Transactions of the ACL articles to be presented at ACL 2017':'TACL'}[x])"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['Long', 'Short', 'Demo', 'TACL'], dtype=object)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['Paper Type'].unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let's use Scattertext to see how the titles of long and short papers differ\n",
"End-to-end modeling, joint modeling, adversarial learning, and reading comprehension were particularly characteristic of long papers. \n",
"\n",
"Case studies, social media, multilingual analysis and were more prevalent in short papers. The term \"temporal\" fell victim to polysemy-- both temporal relation mining and trend-over-time analysis were common in short papers.\n",
"\n",
"Learning, neural, and translation were very common in both sets. \n",
"\n",
"Click around on the plot below to see other findings."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['Parsed'] = df['Title'].apply(st.whitespace_nlp)\n",
"long_short_df = df[df['Paper Type'].isin(['Long', 'Short'])]\n",
"long_short_corpus = (st.CorpusFromParsedDocuments(long_short_df, \n",
" category_col = 'Paper Type', \n",
" parsed_col = 'Parsed', \n",
" feats_from_spacy_doc = st.FeatsFromSpacyDoc(strip_final_period=True))\n",
" .build()\n",
" .get_stoplisted_unigram_corpus())\n",
"html = st.produce_scattertext_explorer(long_short_corpus, \n",
" category='Long', \n",
" category_name='Long Papers', \n",
" not_category_name='Short Papers',\n",
" minimum_term_frequency=1,\n",
" metadata=long_short_df.apply(lambda x: ', '.join(x['Authors']) + '('+x['Field']+')', axis=1),\n",
" use_full_doc=True,\n",
" width_in_pixels=1000)\n",
"file_name = \"ACL2017LongShort.html\"\n",
"open(file_name, 'wb').write(html.encode('utf-8'))\n",
"IFrame(src=file_name, width = 1200, height=1000)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from scattertext.termsignificance import LogOddsRatioUninformativeDirichletPrior\n",
"term_freq_df = long_short_corpus.get_term_freq_df()\n",
"frequencies = term_freq_df.sum(axis=1).values\n",
"frequencies_scaled = frequences/frequences.max()\n",
"p_vals = LogOddsRatioUninformativeDirichletPrior().get_p_vals(term_freq_df.values)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## As requested by Min-Yen Kan, let's look at how TACL differs from long papers\n",
"\n",
"Interesting finding: no TACL paper uses the word \"networks\" in its title, but lots of long papers do. \n",
"\n",
"Two TACL papers talked about \"effects\", while no long papers did:\n",
"- \"evaluating visual representations for topic understanding and their effects on manually generated labels\"\n",
"- \"winning on the merits: the joint effects of content and style on debate outcomes\"\n",
"\n",
"\"LSTMs\" appears to be a false positive for TACL, since the unmarked \"LSTM\" appeared four times in long papers. (I'm safely assuming the long papers used multiple LSTM nodes.)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df['Parsed'] = df['Title'].apply(st.whitespace_nlp)\n",
"long_short_df = df[df['Paper Type'].isin(['Long', 'TACL'])]\n",
"long_short_corpus = (st.CorpusFromParsedDocuments(long_short_df, \n",
" category_col = 'Paper Type', \n",
" parsed_col = 'Parsed', \n",
" feats_from_spacy_doc = st.FeatsFromSpacyDoc(strip_final_period=True))\n",
" .build()\n",
" .get_stoplisted_unigram_corpus())\n",
"def get_metadata(x):\n",
" meta = ', '.join(x['Authors']) \n",
" if x['Field'].strip():\n",
" meta += ' ('+x['Field']+')'\n",
" return meta\n",
" \n",
"html = st.produce_scattertext_explorer(long_short_corpus, \n",
" category='TACL', \n",
" category_name='TACL Papers', \n",
" not_category_name='Long Papers',\n",
" minimum_term_frequency=1,\n",
" metadata=long_short_df.apply(get_metadata, axis=1),\n",
" use_full_doc=True,\n",
" width_in_pixels=1000)\n",
"file_name = \"ACL2017TACLLong.html\"\n",
"open(file_name, 'wb').write(html.encode('utf-8'))\n",
"IFrame(src=file_name, width = 1200, height=1000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let's see how the papers the area chairs published differed from the rest of the set\n",
"\n",
"The 53 papers were co-authored by an area chair, and 290 were only authored by average joes. \n",
"\n",
"\"Fast and accurate\" was a phrase used by two area chairs and nobody else.\n",
"\n",
"Lots of people worked on Chinese, but no area chairs had \"Chinese\" in their paper titles. \"Dialog\" and \"adpatation\" were strongly associated with average joes. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Average Joe 291\n",
"Area Chair 53\n",
"Name: AC, dtype: int64"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import itertools\n",
"area_chairs = set(itertools.chain(*[x.split(':')[1].strip().split(', ') for x in '''Biomedical: Aurélie Névéol, Karin Verspoor\n",
"Cognitive Modeling and Psycholinguistics: Roger Levy, Anders Søgaard\n",
"Dialogue and Interactive Systems: Ron Artstein, Raquel Fernandez, Oliver Lemon\n",
"Discourse and Pragmatics: Yangfeng Ji, Sujian Li, Bonnie Webber\n",
"Information Extraction and NLP Applications: Eugene Agichtein, Chia-Hui Chang, Jing Jiang, Sarvnaz Karimi, Zornitsa Kozareva, Kang Liu, Tie-Yan Liu, Mausam, Alessandro Moschitti, Smaranda Muresan\n",
"Machine Learning: Grzegorz Chrupała, Amir Globerson, Tommi Jaakkola, Sujith Ravi, William Yang Wang\n",
"Machine Translation: Yang Liu, Minh-Thang Luong, Haitao Mi, Graham Neubig, Deyi Xiong\n",
"Multidisciplinary: Michael Piotrowski, Karën Fort\n",
"Multilinguality: Omri Abend, Mona Diab\n",
"Phonology, Morphology and Word Segmentation: Jason Eisner, Hinrich Schütze\n",
"Resources and Evaluation: Sophie Rosset, Wajdi Zaghouani\n",
"Semantics: Manaal Faruqui, Hannaneh Hajishirzi, Anna Korhonen, Preslav Nakov, Mehroosh Sadrzadeh, Aline Villavicencio\n",
"Sentiment Analysis and Opinion Mining: Alexandra Balahur, Lun-Wei Ku, Saif M Mohammad\n",
"Social Media: Zhiyuan Liu, Shimei Pan, Svitlana Volkova\n",
"Speech: Chiori Hori, Chia-ying Lee\n",
"Summarization and Generation: Wenjie Li, Alexander M Rush, Verena Rieser\n",
"Tagging, Chunking, Syntax and Parsing: Emily Pitler, Barbara Plank, Yue Zhang, Hai Zhao\n",
"Vision, Robotics and Grounding: Mohit Bansal, Nate Kushman'''.split('\\n')]))\n",
"df['AC'] = df.Authors.apply(lambda x: 'Area Chair' if set(x) & area_chairs != set() else 'Average Joe')\n",
"df['AC'].value_counts()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"area_chair_corpus = (st.CorpusFromParsedDocuments(df, \n",
" category_col = 'AC', \n",
" parsed_col = 'Parsed', \n",
" feats_from_spacy_doc = st.FeatsFromSpacyDoc(strip_final_period=True))\n",
" .build()\n",
" .get_stoplisted_unigram_corpus())\n",
"def get_metadata(x):\n",
" meta = ', '.join(x['Authors']) \n",
" meta += '('+x['Paper Type']+'; '+x['Field']+')'\n",
" return meta\n",
" \n",
"html = st.produce_scattertext_explorer(area_chair_corpus, \n",
" category='Area Chair', \n",
" category_name='Area Chair', \n",
" not_category_name='Average Joe',\n",
" minimum_term_frequency=1,\n",
" metadata=df.apply(get_metadata, axis=1),\n",
" use_full_doc=True,\n",
" width_in_pixels=1000)\n",
"file_name = \"ACL2017AC.html\"\n",
"open(file_name, 'wb').write(html.encode('utf-8'))\n",
"IFrame(src=file_name, width = 1200, height=1000)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let's look at some descriptive statistics about the set accepted papers, and look at variations in paper categories\n",
"The vast majority of papers are long papers. MT, Semantics, and Gen. Summarization were more associated with Long Papers, while ML, Social Media, and Multidisciplinary dominated short papers. "
]
},
{
"cell_type": "code",
"execution_count": 224,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"