"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.scatter(X_feature_reduced[:,0],X_feature_reduced[:,1],c=target)\n",
"plt.title('LDA')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Observation\n",
"As we can see from above that LDA projected data on new axis in such a way that class are separated as much as possible. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### How does LDA achieves this?\n",
"\n",
"LDA creates new axis based on two criteria:\n",
"* Distance between means of classes\n",
"* Variation within each category\n",
"\n",
"It projects data on new axis and finds mean for each class and variance for each class. It tries to maximise the distance between class means and tries to minimise the variation with each class. Using these into consideration we get a new axis."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![alt text](../../img/shivam_panwar_data1.png \"data\")\n",
"Above is the data for two Genes, we want to project them on new axis with one dimension."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"../../img/shivam_panwar_transformed_data.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Criterion which we choose above to solve this is\n",
"\n",
"**(µ1-µ2)^2\n",
"/(s1+s2)^2**\n",
"\n",
",where µ1 and µ2 are mean of each class and 's1 and s2' are variation/scatter within a clas while making new axis.\n",
"We try to maximise this criteria while making new axis."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Topic modelling \n",
"\n",
"**Topic modelling** is a method of assigning topic to each document. Each topic is made up of certain words.\n",
"\n",
"Consider for example:\n",
"\n",
"We have two topics, topic 1 and topic 2. **'Topic1'** is represented by 'apple, banana, mango' and **topic2** is represented by 'tennis, cricket, hockey'. We can infer that topic1 is talking about fruits and topic2 is talking about sports. We can assign new incoming document into one of these topics and that can be used for **clustering** purpose too. It is used in recommendation systems and many more. \n",
"\n",
"Another example: \n",
"Consider we have 6 documents\n",
"* apple banana\n",
"* apple orange\n",
"* banana orange\n",
"* tiger cat\n",
"* tiger dog\n",
"* cat dog\n",
"\n",
"What topic modelling would do is if want to extract say two topic out of these documents, it will give two distributions, topic-word distribution and doc-topic distribution. In topic-word representation it should give word wise distribution for each topic and in doc-topic it would give for each document, it's topic representation or distribution of document for each topic.\n",
"\n",
"It's ideal topic-word distribution should be:\n",
"\n",
"| Topic | Apple | Banana | Orange | Tiger | Cat | Dog | \n",
"| --- | --- | --- | --- | --- | --- | --- | \n",
"| Topic 1 | .33 | .33 | .33 | 0 | 0 | 0 |\n",
"| Topic 2 | 0 | 0 | 0 | 0.33 | 0.33 | 0.33 |\n",
"\n",
"and it's ideal document-topic distrubution should be:\n",
"\n",
"| Topic | doc1 | doc2 | doc3 | doc4 | doc5 | doc6 | \n",
"| --- | --- | --- | --- | --- | --- | --- | \n",
"| Topic 1 | 1 | 1 | 1 | 0 | 0 | 0 |\n",
"| Topic 2 | 0 | 0 | 0 | 1 | 1 | 1 |\n",
"\n",
"and now suppose we have a new document say, ' cat dog apple', its topic wise representation should be\n",
"\n",
"\n",
"**Topic1: 0.33**\n",
"\n",
"**Topic2: 0.63**\n",
"\n",
"\n",
"LDA is highly used for this purpose.It's usage for topic modelling and has been demonstrated below. We give to it the number of topics we want to find out of the corpus. Remember it follow bow approach therefore, relationship between words are lost in this manner.\n"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [],
"source": [
"lemmatizer=WordNetLemmatizer() #For words Lemmatization\n",
"stemmer=PorterStemmer() #For stemming words\n",
"stop_words=set(stopwords.words('english'))"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"def TokenizeText(text):\n",
" ''' \n",
" Tokenizes text by removing various stopwords and lemmatizing them\n",
" '''\n",
" text=re.sub('[^A-Za-z0-9\\s]+', '', text)\n",
" word_list=word_tokenize(text)\n",
" word_list_final=[]\n",
" for word in word_list:\n",
" if word not in stop_words:\n",
" word_list_final.append(lemmatizer.lemmatize(word))\n",
" return word_list_final"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [],
"source": [
"def gettopicwords(topics,cv,n_words=10):\n",
" '''\n",
" Print top n_words for each topic.\n",
" cv=Countvectorizer\n",
" '''\n",
" for i,topic in enumerate(topics):\n",
" top_words_array=np.array(cv.get_feature_names())[np.argsort(topic)[::-1][:n_words]]\n",
" print \"For topic {} it's top {} words are \".format(str(i),str(n_words))\n",
" combined_sentence=\"\"\n",
" for word in top_words_array:\n",
" combined_sentence+=word+\" \"\n",
" print combined_sentence\n",
" print \" \""
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [],
"source": [
"df=pd.read_csv('million-headlines.zip',usecols=[1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Data link: \n",
"\n",
"[https://www.kaggle.com/therohk/million-headlines](https://www.kaggle.com/therohk/million-headlines)"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"'\n",
"\twith 5855365 stored elements in Compressed Sparse Row format>"
]
},
"execution_count": 81,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"transformed_data"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 1h 33min 20s, sys: 5.73 s, total: 1h 33min 25s\n",
"Wall time: 1h 33min 32s\n"
]
}
],
"source": [
"%%time\n",
"no_topics=10 ## We can change this, hyperparameter\n",
"lda = LatentDirichletAllocation(n_components=no_topics, max_iter=5, learning_method='online', learning_offset=50.,random_state=0).fit(transformed_data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Lda.components_** is a topic_word table, it shows representation of each word in the topic. components_[i, j] can be viewed as pseudocount that represents the number of times word j was assigned to topic i. It can also be viewed as distribution over the words for each topic after normalization"
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"For topic 0 it's top 10 words are \n",
"sydney child first found national family brisbane house record abc \n",
" \n",
"For topic 1 it's top 10 words are \n",
"court council home face final state drug could job cut \n",
" \n",
"For topic 2 it's top 10 words are \n",
"call plan sa trump farmer test accused centre go league \n",
" \n",
"For topic 3 it's top 10 words are \n",
"win queensland crash u school canberra perth set one service \n",
" \n",
"For topic 4 it's top 10 words are \n",
"australia woman day attack china show death labor tasmania business \n",
" \n",
"For topic 5 it's top 10 words are \n",
"new wa south coast cup nt price gold killed tasmanian \n",
" \n",
"For topic 6 it's top 10 words are \n",
"say government world qld country car change help dy minister \n",
" \n",
"For topic 7 it's top 10 words are \n",
"year nsw back election new adelaide north take fire west \n",
" \n",
"For topic 8 it's top 10 words are \n",
"police australian interview report say hit get time ban driver \n",
" \n",
"For topic 9 it's top 10 words are \n",
"man rural melbourne charged murder hour two open hospital road \n",
" \n"
]
}
],
"source": [
"gettopicwords(lda.components_,cv)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Assigning new topic "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that each document is a combination of each topic. Let's see topic represntation of first ten documents."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First ten documents and their topicwise representation is shown below"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {},
"outputs": [],
"source": [
"docs=df['headline_text'][:10]"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {},
"outputs": [],
"source": [
"data=[]\n",
"for doc in docs:\n",
" data.append(lda.transform(cv.transform([doc])))"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {},
"outputs": [],
"source": [
"cols=['topic'+str(i) for i in range(1,11)]\n",
"doc_topic_df=pd.DataFrame(columns=cols,data=np.array(data).reshape((10,10)))"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [],
"source": [
"doc_topic_df['major_topic']=doc_topic_df.idxmax(axis=1)\n",
"doc_topic_df['raw_doc']=docs"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Criterion which we choose above to solve this is\n",
"\n",
"**(µ1-µ2)^2\n",
"/(s1+s2)^2**\n",
"\n",
",where µ1 and µ2 are mean of each class and 's1 and s2' are variation/scatter within a clas while making new axis.\n",
"We try to maximise this criteria while making new axis."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Topic modelling \n",
"\n",
"**Topic modelling** is a method of assigning topic to each document. Each topic is made up of certain words.\n",
"\n",
"Consider for example:\n",
"\n",
"We have two topics, topic 1 and topic 2. **'Topic1'** is represented by 'apple, banana, mango' and **topic2** is represented by 'tennis, cricket, hockey'. We can infer that topic1 is talking about fruits and topic2 is talking about sports. We can assign new incoming document into one of these topics and that can be used for **clustering** purpose too. It is used in recommendation systems and many more. \n",
"\n",
"Another example: \n",
"Consider we have 6 documents\n",
"* apple banana\n",
"* apple orange\n",
"* banana orange\n",
"* tiger cat\n",
"* tiger dog\n",
"* cat dog\n",
"\n",
"What topic modelling would do is if want to extract say two topic out of these documents, it will give two distributions, topic-word distribution and doc-topic distribution. In topic-word representation it should give word wise distribution for each topic and in doc-topic it would give for each document, it's topic representation or distribution of document for each topic.\n",
"\n",
"It's ideal topic-word distribution should be:\n",
"\n",
"| Topic | Apple | Banana | Orange | Tiger | Cat | Dog | \n",
"| --- | --- | --- | --- | --- | --- | --- | \n",
"| Topic 1 | .33 | .33 | .33 | 0 | 0 | 0 |\n",
"| Topic 2 | 0 | 0 | 0 | 0.33 | 0.33 | 0.33 |\n",
"\n",
"and it's ideal document-topic distrubution should be:\n",
"\n",
"| Topic | doc1 | doc2 | doc3 | doc4 | doc5 | doc6 | \n",
"| --- | --- | --- | --- | --- | --- | --- | \n",
"| Topic 1 | 1 | 1 | 1 | 0 | 0 | 0 |\n",
"| Topic 2 | 0 | 0 | 0 | 1 | 1 | 1 |\n",
"\n",
"and now suppose we have a new document say, ' cat dog apple', its topic wise representation should be\n",
"\n",
"\n",
"**Topic1: 0.33**\n",
"\n",
"**Topic2: 0.63**\n",
"\n",
"\n",
"LDA is highly used for this purpose.It's usage for topic modelling and has been demonstrated below. We give to it the number of topics we want to find out of the corpus. Remember it follow bow approach therefore, relationship between words are lost in this manner.\n"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [],
"source": [
"lemmatizer=WordNetLemmatizer() #For words Lemmatization\n",
"stemmer=PorterStemmer() #For stemming words\n",
"stop_words=set(stopwords.words('english'))"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {},
"outputs": [],
"source": [
"def TokenizeText(text):\n",
" ''' \n",
" Tokenizes text by removing various stopwords and lemmatizing them\n",
" '''\n",
" text=re.sub('[^A-Za-z0-9\\s]+', '', text)\n",
" word_list=word_tokenize(text)\n",
" word_list_final=[]\n",
" for word in word_list:\n",
" if word not in stop_words:\n",
" word_list_final.append(lemmatizer.lemmatize(word))\n",
" return word_list_final"
]
},
{
"cell_type": "code",
"execution_count": 90,
"metadata": {},
"outputs": [],
"source": [
"def gettopicwords(topics,cv,n_words=10):\n",
" '''\n",
" Print top n_words for each topic.\n",
" cv=Countvectorizer\n",
" '''\n",
" for i,topic in enumerate(topics):\n",
" top_words_array=np.array(cv.get_feature_names())[np.argsort(topic)[::-1][:n_words]]\n",
" print \"For topic {} it's top {} words are \".format(str(i),str(n_words))\n",
" combined_sentence=\"\"\n",
" for word in top_words_array:\n",
" combined_sentence+=word+\" \"\n",
" print combined_sentence\n",
" print \" \""
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {},
"outputs": [],
"source": [
"df=pd.read_csv('million-headlines.zip',usecols=[1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Data link: \n",
"\n",
"[https://www.kaggle.com/therohk/million-headlines](https://www.kaggle.com/therohk/million-headlines)"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" headline_text\n",
"0 aba decides against community broadcasting lic...\n",
"1 act fire witnesses must be aware of defamation\n",
"2 a g calls for infrastructure protection summit\n",
"3 air nz staff in aust strike for pay rise\n",
"4 air nz strike to affect australian travellers"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 5min 6s, sys: 3.17 s, total: 5min 9s\n",
"Wall time: 5min 8s\n"
]
}
],
"source": [
"%%time \n",
"num_features=100000\n",
"# cv=CountVectorizer(min_df=0.01,max_df=0.97,tokenizer=TokenizeText,max_features=num_features)\n",
"cv=CountVectorizer(tokenizer=TokenizeText,max_features=num_features)\n",
"transformed_data=cv.fit_transform(df['headline_text'])"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<1103665x87637 sparse matrix of type '| \n", " | headline_text | \n", "
|---|---|
| 0 | \n", "aba decides against community broadcasting lic... | \n", "
| 1 | \n", "act fire witnesses must be aware of defamation | \n", "
| 2 | \n", "a g calls for infrastructure protection summit | \n", "
| 3 | \n", "air nz staff in aust strike for pay rise | \n", "
| 4 | \n", "air nz strike to affect australian travellers | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" topic1 topic2 topic3 topic4 topic5 topic6 topic7 \\\n",
"0 0.016667 0.016667 0.183332 0.016667 0.016667 0.016667 0.016670 \n",
"1 0.014286 0.014286 0.014286 0.014286 0.014286 0.014286 0.014286 \n",
"2 0.016667 0.183333 0.183333 0.016667 0.016667 0.016667 0.016667 \n",
"3 0.012500 0.012502 0.012500 0.012501 0.012500 0.012500 0.762497 \n",
"4 0.014286 0.014286 0.014286 0.014289 0.014286 0.014286 0.442854 \n",
"5 0.016667 0.016667 0.016667 0.183463 0.016667 0.683204 0.016667 \n",
"6 0.183333 0.016667 0.016667 0.016667 0.016667 0.016667 0.516667 \n",
"7 0.012500 0.012500 0.012500 0.137500 0.012500 0.137500 0.012500 \n",
"8 0.014286 0.728571 0.014286 0.014286 0.157143 0.014286 0.014286 \n",
"9 0.016667 0.516667 0.016667 0.016667 0.183334 0.183332 0.016667 \n",
"\n",
" topic8 topic9 topic10 major_topic \\\n",
"0 0.016667 0.683331 0.016667 topic9 \n",
"1 0.014289 0.871424 0.014287 topic9 \n",
"2 0.516667 0.016667 0.016667 topic8 \n",
"3 0.137500 0.012500 0.012500 topic7 \n",
"4 0.014286 0.442857 0.014286 topic9 \n",
"5 0.016667 0.016667 0.016667 topic6 \n",
"6 0.183333 0.016667 0.016667 topic7 \n",
"7 0.012500 0.637500 0.012500 topic9 \n",
"8 0.014286 0.014286 0.014286 topic2 \n",
"9 0.016667 0.016667 0.016667 topic2 \n",
"\n",
" raw_doc \n",
"0 aba decides against community broadcasting lic... \n",
"1 act fire witnesses must be aware of defamation \n",
"2 a g calls for infrastructure protection summit \n",
"3 air nz staff in aust strike for pay rise \n",
"4 air nz strike to affect australian travellers \n",
"5 ambitious olsson wins triple jump \n",
"6 antic delighted with record breaking barca \n",
"7 aussie qualifier stosur wastes four memphis match \n",
"8 aust addresses un security council over iraq \n",
"9 australia is locked into war timetable opp "
]
},
"execution_count": 88,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"doc_topic_df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We saw how LDA can be used for topic modelling. This can be used for document custering based on the doc topic representation. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### References\n",
"[Statquest LDA](https://www.youtube.com/watch?v=azXCzI57Yfc)\n",
"\n",
"[https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/](https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/)\n",
"\n",
"[https://sebastianraschka.com/faq/docs/lda-vs-pca.html](https://sebastianraschka.com/faq/docs/lda-vs-pca.html)\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
| \n", " | topic1 | \n", "topic2 | \n", "topic3 | \n", "topic4 | \n", "topic5 | \n", "topic6 | \n", "topic7 | \n", "topic8 | \n", "topic9 | \n", "topic10 | \n", "major_topic | \n", "raw_doc | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "0.016667 | \n", "0.016667 | \n", "0.183332 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016670 | \n", "0.016667 | \n", "0.683331 | \n", "0.016667 | \n", "topic9 | \n", "aba decides against community broadcasting lic... | \n", "
| 1 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014289 | \n", "0.871424 | \n", "0.014287 | \n", "topic9 | \n", "act fire witnesses must be aware of defamation | \n", "
| 2 | \n", "0.016667 | \n", "0.183333 | \n", "0.183333 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.516667 | \n", "0.016667 | \n", "0.016667 | \n", "topic8 | \n", "a g calls for infrastructure protection summit | \n", "
| 3 | \n", "0.012500 | \n", "0.012502 | \n", "0.012500 | \n", "0.012501 | \n", "0.012500 | \n", "0.012500 | \n", "0.762497 | \n", "0.137500 | \n", "0.012500 | \n", "0.012500 | \n", "topic7 | \n", "air nz staff in aust strike for pay rise | \n", "
| 4 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014289 | \n", "0.014286 | \n", "0.014286 | \n", "0.442854 | \n", "0.014286 | \n", "0.442857 | \n", "0.014286 | \n", "topic9 | \n", "air nz strike to affect australian travellers | \n", "
| 5 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.183463 | \n", "0.016667 | \n", "0.683204 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "topic6 | \n", "ambitious olsson wins triple jump | \n", "
| 6 | \n", "0.183333 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.516667 | \n", "0.183333 | \n", "0.016667 | \n", "0.016667 | \n", "topic7 | \n", "antic delighted with record breaking barca | \n", "
| 7 | \n", "0.012500 | \n", "0.012500 | \n", "0.012500 | \n", "0.137500 | \n", "0.012500 | \n", "0.137500 | \n", "0.012500 | \n", "0.012500 | \n", "0.637500 | \n", "0.012500 | \n", "topic9 | \n", "aussie qualifier stosur wastes four memphis match | \n", "
| 8 | \n", "0.014286 | \n", "0.728571 | \n", "0.014286 | \n", "0.014286 | \n", "0.157143 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "0.014286 | \n", "topic2 | \n", "aust addresses un security council over iraq | \n", "
| 9 | \n", "0.016667 | \n", "0.516667 | \n", "0.016667 | \n", "0.016667 | \n", "0.183334 | \n", "0.183332 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "0.016667 | \n", "topic2 | \n", "australia is locked into war timetable opp | \n", "