{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualizing the LDA models from the R LDAvisData package"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import pyLDAvis as vis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `load_R_model` function loads LDA model data that ships with the [LDAvisData](https://github.com/cpsievert/LDAvisData) package. It was extracted from the R data files into JSON using [this script](https://github.com/bmabey/pyLDAvis/blob/master/tests/data/export_data.R)."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"def load_R_model(filename):\n",
" with open(filename, 'r') as j:\n",
" data_input = json.load(j)\n",
" data = {'topic_term_dists': data_input['phi'], \n",
" 'doc_topic_dists': data_input['theta'],\n",
" 'doc_lengths': data_input['doc.length'],\n",
" 'vocab': data_input['vocab'],\n",
" 'term_frequency': data_input['term.frequency']}\n",
" return data\n",
"\n",
"def vis_R_model(filename):\n",
" return vis.display(vis.prepare(**load_R_model(filename)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Movies Reivew Model\n",
"\n",
"This model was trained on a corpus of 2000 movie reviews parsed by [Pang and Lee (ACL, 2004)](http://www.cs.cornell.edu/people/pabo/movie-review-data/), originally gathered from the IMDB archive of the rec.arts.movies.reviews newsgroup."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"\n",
"\n",
""
],
"text/plain": [
""
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"movies_model_data = load_R_model('data/movie_reviews_input.json')\n",
"movies_pd = vis.prepare(**movies_model_data)\n",
"vis.display(movies_pd)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## AP Model\n",
"\n",
"This model was trained on a [corpus of 2246 documents from the Associated Press](http://www.cs.columbia.edu/~blei/lda-c/) made available by Blei. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"\n",
"\n",
""
],
"text/plain": [
""
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vis_R_model('data/ap_input.json')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Jeopardy Model\n",
"\n",
"This model was trained on a [corpus of over 200,000 Jeopardy questions](http://www.reddit.com/r/datasets/comments/1uyd0t/200000_jeopardy_questions_in_a_json_file)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"\n",
"\n",
""
],
"text/plain": [
""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vis_R_model('data/jeopardy_input.json')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 1
}