{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "fjzcvgz2NTzj"
},
"source": [
"# DH 2018 Machine Reading: Advanced Topics in Word Vectors"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "lG6sPBChKzgG"
},
"source": [
"## **Welcome to Machine Reading!**\n",
"\n",
"This is a hands-on workshop focusing on various word vectorization methods and applications for digital humanities.\n",
"The workshop will be split into 4 sections with 10 minute breaks in-between. The sections get incrementally more advanced, building on concepts and methods from the previous sections.\n",
"\n",
"To follow along, you can run the script portions piecemeal, in order, as we progress through the workshop material.\n",
"\n",
"Instructors:\n",
"- Eun Seo Jo, *eunseo@stanford.edu*, Stanford University, Stanford Literary Lab\n",
"- Javier de la Rosa, *versae@stanford.edu*, Stanford University\n",
"- Scott Bailey, *scottbailey@stanford.edu*, Stanford University\n",
"\n",
"\n",
"## 1. Understanding Word Vectors with Visualization (50 mins)\n",
"\n",
"This unit will give a brief introduction of word vectors and word embeddings. Concepts needed to understand the internal mechanics of how they work will also be explained, with the help of plots and visualizations that are commonly used when working with them.\n",
"\n",
"\n",
"- 0:00 - 0:20 Sparse and dense vectors (SVD, PMI, etc.)\n",
"- 0:20 - 0:35 What to do with vectors (cosine similarity, etc.)\n",
"- 0:35 - 0:50 Visualizations (Clustering, PCA, t-SNE) \n",
"\n",
"1. What are the limitations of these word vectors?\n",
"2. What are the different use cases between count-based vectors and word2vec? (size of corpus)\n",
"3. What are limitations?\n",
"4. Why do we use Word2Vec instead?\n",
"\n",
"## 2. Word Vectors via Word2Vec (50 mins)\n",
"\n",
"This unit will focus on Word2Vec as an example of neural net-based approaches of vector encodings, starting with a conceptual overview of the algorithm itself and ending with an activity to train participants’ own vectors.\n",
"\n",
"\n",
"- 0:00 - 0:15 Conceptual explanation of Word2Vec\n",
"- 0:15 - 0:30 Word2Vec Visualization and Vectorial Features and Math\n",
"- 0:30 - 0:50 Word2Vec Construction [using Gensim] and Visualization (from part 1)\n",
"\n",
"\n",
"## 3. Pre-trained Models and Extended Vector Algorithms (50 mins)\n",
"\n",
"This unit will explore the various flavors of word embeddings specifically tailored to sentences, word meaning, paragraph, or entire documents. We will give an overview of pre-trained embeddings, including where they can be found and how to use them.\n",
"\n",
"- 0:00 - 0:15 [Out-of-vocabulary words and pre-trained embeddings](part3.ipynb#1.-Out-of-vocabulary-words-and-pre-trained-embeddings)\n",
"- 0:15 - 0:25 [Activity] Bias in pre-trained historical word embeddings\n",
"- 0:25 - 0:40 [Extending Vector Algorithms: Text Classification](part3.ipynb#2.-Extending-Vector-Algorithms:-Text-Classification)\n",
"- 0:40 - 0:50 [Activity] Authorship attribution\n",
"\n",
"## 4. Role of Bias in Word Embeddings (50 mins)\n",
"\n",
"In this unit, we will explore an application and caveat of using word embeddings -- cultural bias. Presenting methods and results from recent articles, we will show how word embeddings can carry the historical biases of the training corpora and lead an activity that shows these human-biases on vectors. We'll also address how such bias can be mitigated.\n",
"\n",
"- 0:00 - 0:10 Algorithmic bias vs human bias \n",
"- 0:10 - 0:40 Identifying bias in corpora (occupations, gender, ...) \n",
"- 0:40 - 0:50 Towards unbiased embeddings; Examine “debiased” embeddings\n",
"- 0:50 - 0:60 Concluding remarks and debate\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-tcd-ksk4_Hp"
},
"source": [
"# 0. Setting Up "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "aBDSstL82uNa"
},
"source": [
"Before we get started, let's go ahead and set up our notebook.\n",
"\n",
"We will start by importing a few Python libraries that we will use throughout the workshop.\n",
"\n",
"## What are these libraries?\n",
"\n",
"1. NumPy: This is a package for scientific computing in python. For us, NumPy is useful for vector operations. \n",
"2. NLTK: Easy to use python package for text processing (lemmatization, tokenization, POS-tagging, etc.)\n",
"3. matplotlib: Plotting package for visualization\n",
"4. scikit-learn: Easy to use python package for machine learning algorithms and preprocessing tools\n",
"5. gensim: Builtin word2vec and other NLP algorithms\n",
"\n",
"We will be working with a few sample texts using NLTK's corpus package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 465
},
"colab_type": "code",
"id": "1gctO2uj_sJd",
"outputId": "876d9303-9c1a-4be4-f75f-9746c8e0b1bc"
},
"outputs": [],
"source": [
"%%capture --no-stderr\n",
"import sys\n",
"!pip install Cython\n",
"!pip install -r requirements.txt\n",
"!python -m nltk.downloader all\n",
"print(\"All done!\", file=sys.stderr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If all went well, we should be able now to import the next packages into our workspace"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "FuIdxn38Njrq"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import nltk\n",
"import sklearn\n",
"import matplotlib.pyplot as plt\n",
"import gensim"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Why do we need to use numpy?\n",
"#numpy arrays\n",
"integer_array_1 = np.array([0,2,3])\n",
"integer_array_2 = np.array([1,1,1])\n",
"\n",
"#normal python lists\n",
"normal_1 = [0,2,3]\n",
"normal_2 = [1,1,1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#numpy's elementwise operation\n",
"integer_array_1 + integer_array_2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#whereas if you use normal python lists, you would be adding them together\n",
"normal_1 + normal_2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#to do the same thing with normal python lists, I have to...\n",
"elementwise_add = []\n",
"for _ in range(len(normal_1)):\n",
" added = normal_1[_] + normal_2[_]\n",
" elementwise_add.append(added)\n",
"elementwise_add"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#numpy broadcasting\n",
"matrix1 = np.random.randn(3,4)\n",
"matrix1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"to_add = np.array([5,0,5,0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"matrix1 * to_add"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "8dVMpegUqNQW"
},
"source": [
"\n",
"\n",
"---\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "uBOp_mLvnB-N"
},
"source": [
"# 1. Understanding Word Vectors with Visualization\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gpEglVu64ygd"
},
"source": [
"## What is a word vector?\n",
"\n",
"A word vector or embedding is a **numerical representation** of a word within a corpus based on co-occurence with other words. Linguists have found that much of the meaning of a word can be derived from looking at its surrounding context. In this unit, we will explore a few major approaches to representing words in a numerical format."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "DmS8_eIC5SYl"
},
"source": [
"## What is a vector?\n",
"\n",
"Before anything related to words or text let's make sure we're on the same page about vectors! A vector is just a list/array of real numbers. A vector has a size/length which indicates how many numbers are in it. In Python you can make a vector using square brackets '[]'."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# \n",
"vector_one = [1, 2, 3]\n",
"vector_two = [1, 2, 34.53222222]\n",
"vector_three = [-2494, 3, 48.2934, -0.49484]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "BHdEZ6Hs5nyf"
},
"source": [
"Here is a list of 5 real numbers (represented as floating point numbers). This vector has 5 dimensions or features. Unlike formal vectors, Python lists can contain different types of elements and do not support vector operations broadly. NumPy provides a numerical engine with proper vector/array implementations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "ppctTkPC5cFy",
"outputId": "6c7aacd0-3038-4261-9550-781e780e6d1c"
},
"outputs": [],
"source": [
"# Here you can generate a vector of random floats with the random function from numpy\n",
"# You'll see that every time you run this command you get a series of different numbers - try it!\n",
"# In this instance we're making a vector of length (or size) 5\n",
"\n",
"vector_of_floats = np.random.randn(5)\n",
"vector_of_floats"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1eMEa0-_53s6"
},
"source": [
"Here is a list of 20 integers between 0 and 3 (exclusive; not including 3). Later we will go into more vector math but you can see that a vector is a multi-dimensional numerical representation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "rOqQ4XrG5zFY",
"outputId": "24868c2b-8b56-4903-fb91-4462846df357"
},
"outputs": [],
"source": [
"# You can call a vector of random integers too\n",
"# There are three inputs here: the start range of your integer, \n",
"# the end range(exclusive), and the size of the vector\n",
"# In our example, the range is [0, 3)\n",
"\n",
"vector_of_ints = np.random.randint(0, 3, size=100)\n",
"vector_of_ints"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Activity: Try making vectors of your own here!\n",
"my_vector = np.random.randint(5, 55, size=1000)\n",
"my_vector"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "TXqm9Vu26o1j"
},
"source": [
"Word vectors (and vectors in general) can be largely classified into **sparse** and **dense** vectors.\n",
"\n",
"A sparse vector is count-based vector where each element in the vector represent the integer counts of words, usually co-occurence or frequecy. Because a lot of words don't appear all the time, many elements of sparse vectors are 0, to represent 0 observations.\n",
"\n",
"There are a few examples of sparse vectors we will examine here. \n",
"\n",
"\n",
"**Sparse vectors vs. Dense Vectors**\n",
"1. Sparse have lots of zeros (usually the size of sparse vectors is the size of the vocabulary)\n",
"2. They are also big for this reason\n",
"3. Dense vectors are denser (smaller)\n",
"4. Non-count-based operations have been done on dense vectors (even if it came from counting)\n",
"5. Benefit of sparse vectors is that you can readily interpret the meaning of each element\n",
"6. Dense vectors, this isn't case. (With a sparse vector you can say something like \"oh, this means 'life' co-occurs 5 times with 'happiness'\". It's less meaningful to say, with dense vectors, \"oh, 'life's first dimension means -5\")\n",
"7. Dense vectors are smaller, easier to work with.\n",
"8. There are different types of sparse vectors. A word-word matrix is one of them, a document-term matrix gives you a set too. \n",
"9. There are different types of dense vectors too. You can use SVD to get the most informative dimensions, this can be derived from a count-based vector/sparse vector. You can also make dense vectors via prediction methods such as neural networks (eg. Word2Vec).\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "TXqm9Vu26o1j"
},
"source": [
"## Document-term matrix\n",
"\n",
"One of the simplest and most common ways of constructing a matrix is recording its occurence through out a set of documents. This creates a document-term matrix where one dimension indicates the frequency of a word in documents and the other indicates the vocabulary (all words that occur at least once in your entire corpus)."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "DD-eWXX7ts2E"
},
"source": [
"Among the many packages that help you construct your own matrix with your corpus, `scikit-learn` is one of the most heavily used within the Python scientific stack. Let's import `scikit-learn`'s `CountVectorizer()` "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "Tj9BDJRAtqmA"
},
"outputs": [],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "pSmRFOz5uCI6"
},
"outputs": [],
"source": [
"# Imagine you have a document that is just a sentence like this...\n",
"\n",
"documents = [\n",
" \"This is a piece of text. This is also some random text. Also text.\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "32OUeIINt7dz"
},
"source": [
"Let's now transform this document so that each word is given a unique identifying number."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "gy3TSca-_AI_"
},
"outputs": [],
"source": [
"example_vectorizer = CountVectorizer() #initialize your count vectorizer\n",
"example_vectorizer.fit(documents) #documents much be a vector of strings(individual documents)\n",
"print(\"Vocabulary size:\", len(example_vectorizer.vocabulary_))\n",
"example_vectorizer.vocabulary_ #We can get the unique vocabulary set and its corresponding index here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also, we add the corresponding frequency number, which gives the total number of times each word appears in each document."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 575
},
"colab_type": "code",
"id": "bw_tEfm8uZcr",
"outputId": "0b8c514d-fe48-4179-8332-c999893306b8"
},
"outputs": [],
"source": [
"counts = example_vectorizer.transform(documents)\n",
"print(counts)\n",
"print(\" ↑ ↑ ↑\\n doc word_id count\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "huVjKtqLx4qu"
},
"source": [
"Now, let's iterate through all the words that appear in our original document and print all the counts that we generated above."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 198
},
"colab_type": "code",
"id": "nI6ZmEpuubbh",
"outputId": "e093bb94-43dc-45fe-d21e-3b3df54578c0"
},
"outputs": [],
"source": [
"doc = 0 # first document\n",
"for word, word_id in example_vectorizer.vocabulary_.items():\n",
" print(word, \":\", counts[doc, word_id])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gc9OwJiI1N2k"
},
"source": [
"A **document-term matrix** is just a big table (formally, a mathematical matrix) that describes the frequency of words or terms that occur in a collection of documents. In a document-term matrix, **rows correspond to documents** in the collection and **columns correspond to terms**. In \n",
"\n",
"In our case, since we only have one document, our document-term matrix only has one row (doc `0`) and looks like this.\n",
"\n",
"| | also | is | of | piece | random | some | text | this |\n",
"| ----------- |:----:|:--:|:--:|:-----:|:------:|:----:|:----:|:----:|\n",
"| Document #0 | 2 | 2 | 1 | 1 | 1 | 1 | 3 | 2 |\n",
"\n",
"It can easily be extracted by using the `transform()` method of our `CountVectorizer()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "88JlYyYnu1aL",
"outputId": "0d0e5cf0-b56b-4759-92a5-24df0ccc96b0"
},
"outputs": [],
"source": [
"counts = example_vectorizer.transform(documents)\n",
"counts.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each element of the matrix represents vocabulary from above, with the placement corresponding to the unique identifier assigned by scikit-learn, eg. 7th placement (6th, starting from 0) is `text`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's now add a new document that looks almost identical but introduces a new word, just to see how this change reflects on the document-term matrix."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"documents = [\n",
" \"This is a piece of text. This is also some random text. Also text.\",\n",
" \"This is a piece of text. This is also some random text. Also new text.\",\n",
"]\n",
"example_vectorizer.fit(documents)\n",
"print(\"Vocabulary size:\", len(example_vectorizer.vocabulary_))\n",
"example_vectorizer.vocabulary_"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"counts = example_vectorizer.transform(documents)\n",
"print(counts)\n",
"print(\" ↑ ↑ ↑\\n doc word_id count\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"counts = example_vectorizer.transform(documents)\n",
"counts.toarray()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now with two documents our matrix looks like this. \n",
"\n",
"| | also | is | new | of | piece | random | some | text | this |\n",
"| ----------- |:----:|:--:|:---:|:--:|:-----:|:------:|:----:|:----:|:----:|\n",
"| Document #0 | 2 | 2 | 0 | 1 | 1 | 1 | 1 | 3 | 2 |\n",
"| Document #1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 3 | 2 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Apart from the fact that the vocabulary size is now bigger, just by looking at the document-term matrix we can easily spot that one of our documents has one word more than the other. Certainly, we can spot the difference at column 3 (2 in zero-index Python sequences), which corresponds to the new word introduced, `new`, in our vocabulary. You can see there is also an additional column for the additional document (document 2). You can induce what the matrix would look like with lots more documents and a bigger vocabulary!"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ENE90ssduc-m"
},
"source": [
"By now, you might have noticed that 1-letter words are being ignored. That's due to the way `CountVectorizer()` splits sentences into words. `CountVectorizer()` has options to customize this behaviour and it allows to specify your own regular expression to extract words, disregard stopwords, count ngrams instead of words, cap the max number of words to count, normalize spelling, or count terms within a frequency range. It is worth exploring the [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "32OUeIINt7dz"
},
"source": [
"Here, we have written up a temporary new regular expression that takes into account 1-letter words, so our `CountVectorizer()` can count 'a' as a vocabulary term. As such you can modify the regex to fix you needs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Lots of library functions will often have a default setting. You must check the documentation online to make sure you're using the settings that you want. \n",
"#In this case, our function's default was to ignore single chracter words.\n",
"new_regex = r\"(?u)\\b\\w+\\b\" # this regex now considers single character tokens\n",
"CountVectorizer(token_pattern=new_regex).fit(documents).vocabulary_ #this is a new, optional parameter for this function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Activity: Make your own corpus of documents and see what kind of doc-term matrix you can generate!\n",
"\n",
"my_corpus = ['this is our trial document','hello good bye random word blue green purple','i wonder if other people feel the same', 'new','new','new']\n",
"my_corpus = ['i love life i love life i love life', 'i love life i love life i love life']\n",
"new_regex = r\"(?u)\\b\\w+\\b\"\n",
"my_vectorizer = CountVectorizer(token_pattern=new_regex)\n",
"my_vectorizer.fit(my_corpus) \n",
"my_vectorizer.transform(my_corpus).toarray()\n",
"#how can I construct a corpus such that this matrix only has 3 columns; every element in the matrix is 3; no. of rows has to be 2"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "cS6G5hX01xqI"
},
"source": [
"OK, since we have vectorized lots of text humans have generated we will now turn to some canons. \n",
"\n",
"**Let's now play with three texts/documents in our corpus, taken from literature. ** \n",
"\n",
"We will use Moby Dick, Emma, and Parents as our example texts in our corpus.\n",
"Each text is treated as a document."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "pYNxZSglgtBe"
},
"outputs": [],
"source": [
"from nltk.corpus import gutenberg"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"gutenberg.fileids()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "m7VkZdNT3MEH"
},
"outputs": [],
"source": [
"#raw means plain text\n",
"\n",
"mobydick = gutenberg.raw('melville-moby_dick.txt')\n",
"emma = gutenberg.raw('austen-emma.txt')\n",
"alice = gutenberg.raw('carroll-alice.txt')\n",
"\n",
"#number of tokens in mody dick\n",
"len(mobydick.split())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#emma looks like this\n",
"print(emma)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "EcA-dIKKwqH6"
},
"outputs": [],
"source": [
"corpus = [mobydick, emma, alice]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17051
},
"colab_type": "code",
"id": "y5EJf7w_uRqV",
"outputId": "d7892b5a-2fcb-423a-dd9f-80abdc3e19ed"
},
"outputs": [],
"source": [
"# We do the same thing as above \n",
"lit_vectorizer = CountVectorizer(token_pattern=new_regex)\n",
"lit_vectorizer.fit(corpus)\n",
"print(\"Vocabulary size:\", len(lit_vectorizer.vocabulary_))\n",
"lit_vectorizer.vocabulary_"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Kha_OLYwuUlo"
},
"source": [
"To get the ID of a given vocab term:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "1podXJFeuOgg",
"outputId": "94aab82a-5f4d-4633-b613-6899bd1eaeab"
},
"outputs": [],
"source": [
"print(\"The ID of the word 'piece' is \", str(lit_vectorizer.vocabulary_.get('piece')))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "9YRv-pJjw0gi"
},
"outputs": [],
"source": [
"X = lit_vectorizer.fit_transform(corpus)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GazZdRZexTwi"
},
"source": [
"This is what the doc-term matrix looks like for our three document corpus."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"colab_type": "code",
"id": "qhgVoStVw0ea",
"outputId": "22c98800-b6fb-48d0-9ed9-78088389f0ea"
},
"outputs": [],
"source": [
"X = X.toarray()\n",
"X #Remember each row corresponds to each document (novel) and each column is each word from our combined vocabulary\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "wrijrmQH2eNx"
},
"source": [
"The dimensions of the matrix are given by the shape property."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "btnKmwIS2Sg1",
"outputId": "f44fe033-bb88-48f1-a213-32e36c2e212c"
},
"outputs": [],
"source": [
"X.shape # How many novels are there? # How big is our vocabulary?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "efcIAA4W2sxG",
"outputId": "2db82a5c-053c-4407-fbe7-2e64650eaee7"
},
"outputs": [],
"source": [
"print(\"The doc-term matrix has {} documents and {} dimensions.\".format(str(X.shape[0]), str(X.shape[1])))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 17034
},
"colab_type": "code",
"id": "MnHKIQ5Bw0bq",
"outputId": "02d4196a-3860-4aa3-c60c-26e7afe820c9"
},
"outputs": [],
"source": [
"# You can look up all the words in the vocab from the three novels\n",
"lit_vectorizer.get_feature_names() "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "mULrSjzB4T_w"
},
"source": [
"Let's get vocab IDs for 'happy', 'sad', 'angry'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
},
"colab_type": "code",
"id": "BWuN5RId38CT",
"outputId": "02757bf6-e10a-4552-ee41-b566fc871a38"
},
"outputs": [],
"source": [
"print(lit_vectorizer.vocabulary_.get('happy'))\n",
"print(lit_vectorizer.vocabulary_.get('sad'))\n",
"print(lit_vectorizer.vocabulary_.get('angry'))\n",
"print(lit_vectorizer.vocabulary_.get('queer'))\n",
"print(lit_vectorizer.vocabulary_.get('handkerchief'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Using python slicing, you can grab by columns\n",
"X[:,8860], X[:,16233], X[:,1059], X[:,15070], X[:,8823]\n",
"# before comma (first dimension; dimension of novels)\n",
" # We want the entire column (all novels)\n",
"# after comma (second dimension; dimension of the words)\n",
" # We only want one column (not all words, just the select word)\n",
"#This is one way of making word vectors. \n",
"#What kind of information do you think these vectors represent?\n",
"#Novels are in order of [moby dick, emma, alice in wonderland]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"freq_of_queer = X[:,15070]\n",
"x = [0,1,2] #you can also do np.arange(len(X[:,15070]))\n",
"plt.bar(x, freq_of_queer, color=\"pink\") #this is the most important part!\n",
"plt.xticks(x, ('Moby Dick', 'Emma', 'Alice in Wonderland'))\n",
"plt.xlabel('Novel')\n",
"plt.ylabel('Frequency')\n",
"plt.title('Frequecy of \"queer\" in Novels')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#How would you graph a chart titled : \"Frequency of words X, Y, Z in novel N\"\n",
"freq = [X[2,8172], X[2,14929], X[2,994], X[2,13872], X[2,8139]] #frequency for happy, sad, angry, queer, handkerchief only for Alice\n",
"# [2: alice, 8172: frequecy of happy]\n",
"x = [0,1,2,3,4] #you can also do np.arange(len(freq))\n",
"plt.bar(x, freq, color=\"0.7\") #this is the most important part!\n",
"plt.xticks(x, ('happy', 'sad', 'angry', 'queer', 'handkerchief'))\n",
"plt.xlabel('Word')\n",
"plt.ylabel('Frequency')\n",
"plt.title('Frequecy of Select Words in Alice in Wonderland')\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You may be wondering, well that wasn't fair since some novels are longer than others. In that case, we divide all frequencies by the length of respective novels (normalize) so that we get relative frequencies.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"length_of_novels = np.array([len(mobydick.split()), \n",
" len(emma.split()), \n",
" len(alice.split())])\n",
"\n",
"print(\"length of novels: \", length_of_novels) #How long these novels are\n",
"print(\"original: \\n\", X)\n",
"normalized_X = X/length_of_novels.reshape(3,1) #thanks broadcasting\n",
"normalized_X"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"freq_of_queer = normalized_X[:,15070]\n",
"x = [0,1,2] #you can also do np.arange(len(X[:,15070])) more generically\n",
"plt.bar(x, freq_of_queer, color=\"orange\") #this is the most important part!\n",
"plt.xticks(x, ('Moby Dick', 'Emma', 'Alice in Wonderland'))\n",
"plt.xlabel('Novel')\n",
"plt.ylabel('Normalized Frequency')\n",
"plt.title('Normalized Frequecy of \"queer\" in Novels')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#In another instance, you could treat each sentence as one document\n",
"\n",
"from nltk import sent_tokenize\n",
"sentences = []\n",
"for novel in ['melville-moby_dick.txt', 'austen-emma.txt', 'edgeworth-parents.txt']:\n",
" sentences += sent_tokenize(gutenberg.raw(novel))\n",
"len(sentences)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# sentences as documents now\n",
"lit_vectorizer = CountVectorizer(token_pattern=new_regex)\n",
"X = lit_vectorizer.fit_transform(sentences).toarray()\n",
"X.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X #Here you will notice most of these elements are zeros! Why?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#happy, sad, angry\n",
"X[:,8860], X[:,16233], X[:,1059] "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.set_printoptions(threshold=np.inf)\n",
"print(X[:,8860]) # let's see all the zeros!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print('There are '+str(len(X[:,8860]))+' total elements since there were this many sentneces total')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"... and of these \" + str(len(np.where(X[:,8860]>0)[0])) + \" have non-zero entries.\")\n",
"#What does that mean about this word?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"In total this word appears \" + str(np.sum(X[:,8860])) + \" times.\")\n",
"#What does that say about this word?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Doc-term matrices are used in information retrieval for finding documents that are most relevant to a query. If you look at each row (rather than column) you get a numerical representation of a document by the words that appear in it. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Just putting this back normal print options\n",
"np.set_printoptions(threshold=10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "774DpqAmqPOD"
},
"source": [
"## Word-word Matrix\n",
"\n",
"In the previous section we looked at representing words by their relations to a corpus of documents. What about their relation to one another? The most intuitive way of doing this is to build a word-word matrix where now both dimensions are the vocab and each element at [**k**][**l**] represents the co-occurence of the vocab **k** with vocab **l** in a window of **w**. The window of **w** indicates the number of words before and after given word **k** where we count occurrneces of **l**. **w** is usually around 4. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from nltk.tokenize import word_tokenize\n",
"import coocc #look for this file in our directory"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"a = ['a c b c b c a d d a c c a d c b a d c']\n",
"\n",
"v, m = coocc.ww_matrix(a, word_tokenize, 2) #This is not the most efficient function, sorry \n",
"m.toarray(), v\n",
"#How do you interpret this matrix? #Also, notice anything interesting?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Doing this for mobydick\n",
"mobydick = gutenberg.raw('melville-moby_dick.txt')\n",
"v, m = coocc.ww_matrix([mobydick], word_tokenize, 4) \n",
"#v is vocabulary dictionary; m is the matrix\n",
"X = m.toarray()\n",
"X"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"v #vocabulary dictionary"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X = m.toarray()\n",
"#What is the shape of this matrix?\n",
"X.shape\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Now, say we want to the word vectors for 'happy','sad','angry' again\n",
"#We need to first get the indices\n",
"happy_i = v['happy']\n",
"sad_i = v['sad']\n",
"angry_i = v['angry']\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"happy_ww = X[happy_i,:] #because this is symmetric, it doesn't matter if you get the row or the column\n",
"sad_ww = X[sad_i,:]\n",
"angry_ww = X[angry_i,:]\n",
"np.set_printoptions(threshold=np.inf)\n",
"happy_ww #looking at this vector for happy... "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#You can look up the co-occurrence of two words within a window\n",
"#How many times does dark occur with night in a window of 4?\n",
"\n",
"dark_i = v['dark']\n",
"stormy_i = v['stormy']\n",
"night_i = v['night']\n",
"\n",
"X[night_i,dark_i]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Activity: See if you can identify interesting co-occurrences!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"np.set_printoptions(threshold=10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Ota8XdGSrAXO"
},
"source": [
"## PPMI Matrix\n",
"\n",
"If we look at our word-word matrix, you'll see that because it is only recording the raw co-occurrences, it makes no adjustments for how certain words are just more frequent. For instance, words such as 'is' or 'the' are more likely to appear together with any other word **w** than other words. Pointwise mutual information introduces a weighting scheme to take into account co-occurence relative to two words' independent freqencies. \n",
"\n",
"Since these methods are tricks/engineering to improve results and there is no absolute truth to what is the best method. Here is one for $w$ as target word and $c$ as the context word:\n",
"\n",
"\n",
"\n",
"$$PPMI_{\\alpha}(w,c) = max(log_2 \\frac{P(w,c)}{P(w)P_{\\alpha}(c)})$$\n",
"\n",
"\n",
"$$P_{\\alpha}(c) = \\frac{count(c)^{\\alpha}}{\\sum_c count(c)^{\\alpha}}$$\n",
"\n",
"\n",
"Notice the PPMI is taking the log ratio of the probability of co-occurrence over the probability of individual freqencies. We only consider positive value because negative probabilties (when the co-occurence is lower than expected) fluctuate. The \\alpha is a measure correct for rare words with high PMI.\n",
"\n",
"\n",
"*PPMI: Positive Pointwise Mutual Information\n",
"\n",
"Levy, O., Goldberg, Y., and Dagan, I. (2015). http://www.aclweb.org/anthology/Q15-1016\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "eT_d-Y49rkGQ"
},
"source": [
"## Dense Vectorization\n",
"\n",
"We have so far looked at vectorization methods where each element correspondence to a discrete entity such as a term from the vocabulary or a document. We have seen that this results in a lot of 0 entries. On the other hand dense vector elements are mostly non-zero, they tend to be shorter (denser) and sometimes more effective.l\n",
"\n",
"Dense vectors have become more popularized lately due to deep-learning based vectors such as GloVe and Word2Vec. We will examine the truncated SVD, one dense vectorization method that is not deep-learning based but used widely. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "chy7ocWIrmpb"
},
"source": [
"**SVD**\n",
"\n",
"Singular Value Decomposition (SVD) is a common method for identifying dimensions with highest variance. It is a form of dimension reduction where the algorithm identifies ways of condense as much of the information of the data in with fewer dimensions. SVD factorizes a given matrix $X$ into three matrices:\n",
"\n",
"$$SVD(X) = W\\Sigma C^T $$\n",
"\n",
"where X is a word-word matrix, $W$ is a matrix of your new dense vectors, and $\\Sigma$ is a diagonal matrix of singular values that represents the importance (how much variance encoded) of the corresponding dimension. Starting from the top, the first dimension encodes the most information and the following dimensions are orthogonal the the previous and contain less information down the line. A truncated SVD is the same thing but taking only $k$ top dimensions. \n",
"\n",
"See here for more information:\n",
"https://web.stanford.edu/~jurafsky/slp3/16.pdf\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "jesnsy1iBe0_"
},
"outputs": [],
"source": [
"# You can do this super easily with sklearn!\n",
"from sklearn.decomposition import TruncatedSVD\n",
"svd = TruncatedSVD(n_components=10) #this is your K (how many top dimensions you want to keep)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "GAs3YDPPCfLi"
},
"outputs": [],
"source": [
"denseX = svd.fit_transform(X) #n_samples, n_dims"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "akYkC0mpCnyy",
"outputId": "71dff9ca-640d-4ab7-f8fe-1dc71e141bd6"
},
"outputs": [],
"source": [
"#What is the shape of this matrix?\n",
"\n",
"denseX.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#You can chart your elbow graph here\n",
"eigen = svd.fit(X).explained_variance_\n",
"eigen"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Most of your variance is encoded in the first dimension\n",
"plt.plot(eigen)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "jutSseTg4--s"
},
"outputs": [],
"source": [
"happy_vector_dense = denseX[v['happy'],:]\n",
"sad_vector_dense = denseX[v['sad'],:]\n",
"angry_vector_dense = denseX[v['angry'],:]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"id": "qmF1kLZE2ajU",
"outputId": "f3f26d7f-3391-462a-d788-e27e2c2f4a05"
},
"outputs": [],
"source": [
"happy_vector_dense\n",
"\n",
"#How does a dense vector compare to a sparse vector?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"happy_ww"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "4vq8gl8CFv2M"
},
"source": [
"## Vector Usages"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "13TWG94f-FBx"
},
"source": [
"There are several ways of working with vectors and the most useful for our purposes may be similarity. We will explore this further in the next section as well.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are several ways to measure distance between two vectors. The most widely used is **cosine similarity**. This a cosine measure of angle between two vectors. Mathematically, cosine similarity looks like this:\n",
"\n",
"$$cos(\\vec{v}, \\vec{w}) = \\frac{dot(\\vec{v},\\vec{w})}{norm(\\vec{v})norm(\\vec{w})}$$\n",
"\n",
"$$dot(\\vec{v},\\vec{w}) = \\sum_{i=0}^{n}v_iw_i$$\n",
"$$norm(\\vec{v}) = \\sqrt{\\sum_{i=0}^{n}v_i^2}$$\n",
"\n",
"We normalize here because we want to normalize out frequency so that word similarity disregards frequency. \n",
"\n",
"Cosine similarity will range from 1 to -1. Closer to 1 means closer in direction, closer to -1 opposite in direction and something close to 0 means orthogonal."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Image\n",
"Image(\"./cosine.png\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "I8dz1im5F0J5"
},
"outputs": [],
"source": [
"from sklearn.metrics.pairwise import cosine_similarity "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# So that we can see more contrast when evaluating similarity, let's add in a rather different word: 'biscuit'\n",
"\n",
"index = v['elated']\n",
"elated_vector_dense = denseX[index,:]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "iZvvjLEbGFm2"
},
"outputs": [],
"source": [
"happy_vector = happy_vector_dense.reshape(1,-1) \n",
"sad_vector = sad_vector_dense.reshape(1,-1)\n",
"angry_vector = angry_vector_dense.reshape(1,-1)\n",
"elated_vector = elated_vector_dense.reshape(1,-1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "CRN_oWSpGDzN",
"outputId": "2dd4cf18-f2f1-4195-cac4-68eb4352561b"
},
"outputs": [],
"source": [
"#angle between happy and sad\n",
"cosine_similarity(happy_vector, sad_vector)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "d3qknGoy5zRt",
"outputId": "9d67d41e-088c-425c-9df1-226b77d32197"
},
"outputs": [],
"source": [
"cosine_similarity(angry_vector, sad_vector)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cosine_similarity(happy_vector, angry_vector)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "fU0OXeJx6KFy",
"outputId": "df71d8f9-7b5d-429e-b064-35f490d3ba58"
},
"outputs": [],
"source": [
"cosine_similarity(happy_vector, elated_vector)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"whale_i = v['whale']\n",
"harpoon_i = v['harpoon']\n",
"boat_i = v['boat']\n",
"blood_i = v['blood']\n",
"cosine_similarity(denseX[boat_i,:].reshape(1,-1), denseX[harpoon_i,:].reshape(1,-1))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Activity: Compare similarity of vectors of your choice!"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "2sGJkBnkIbo7"
},
"source": [
"Using similarity as a method, we can also cluster similar vectors together. \n",
"This is called **clustering** and **k-means** is one popular clustering algorithm. \n",
"\n",
"K-means is an iterative algorithm that finds clusters of similar vectors by first assigning observations to their nearest means (initially randomly chosen) as its cluster and then calculating the new centroids of these clusters. \n",
"\n",
"It is called k-means because you are splitting all of your observations into k clusters by their means. "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "l_-8lN32_Op1"
},
"source": [
"Let's work with a small set of words\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "L0YJiABxHtmb"
},
"outputs": [],
"source": [
"selection = ['green','blue','dark','yellow','bright','round','tiny','slim','square','black','thin']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "qXtgAFfd_EhR"
},
"outputs": [],
"source": [
"#iterate through all of these words to make a matrix\n",
"select_matrix = []\n",
"for word in selection:\n",
" word_id = v[word]\n",
" select_matrix.append(denseX[word_id,:])\n",
"select_matrix = np.array(select_matrix)\n",
"select_matrix"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "MU99nlfeATbd",
"outputId": "bde748c7-0256-49ea-df9e-a4617559cad0"
},
"outputs": [],
"source": [
"np.array(select_matrix).shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "wQ7qjbNCGDxK"
},
"outputs": [],
"source": [
"from sklearn.cluster import KMeans"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "5948ycvpGDuA"
},
"outputs": [],
"source": [
"kmeans = KMeans(n_clusters=3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "BoEF5i6pJc0U",
"outputId": "ec1a9bfc-d1cf-4a71-9dd6-e80b3f662713"
},
"outputs": [],
"source": [
"np.set_printoptions(threshold=20)\n",
"predictions = kmeans.fit_predict(select_matrix)\n",
"predictions"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-U7zY5kXJ1vN"
},
"source": [
"## Visualization "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "DKY4c3g9Kav2"
},
"source": [
"You have probably heard of **t-sne** (is it TEA SNEA? or TAE SNAE..)! This is \"newer\" dimension reduction method that emphasizes visual convenience. Sometimes PCA can produce overlapping/crowding of similar points. The con of tsne is that it is not as easily interpretable as PCA (we'll use that in part2). It's also non-deterministic -- you'll get different but similar results everytime. But we thought you should play with it here since it has been widely used in machine learning today."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "tfJDByCCKeTn"
},
"outputs": [],
"source": [
"from sklearn.manifold import TSNE\n",
"tsne = TSNE(n_components=2) #n-components = reduced dimensions\n",
"X"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "d9HTginIJcp1"
},
"outputs": [],
"source": [
"#Let's make a matrix of 50 random words\n",
"random_indices = np.random.choice(len(v), 50, replace=False)\n",
"select_matrix = X[random_indices]\n",
"lookup = {val:key for key,val in v.items()}\n",
"labels = [lookup[w] for w in random_indices]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"select_matrix.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"labels"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"embed = tsne.fit_transform(select_matrix)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"embed"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 379
},
"colab_type": "code",
"id": "j7GKqDoJKpBP",
"outputId": "7ce6b2c3-0f16-4b48-de42-8e76830fbb83"
},
"outputs": [],
"source": [
"random_x, random_y = zip(*embed)\n",
"fig, ax = plt.subplots(figsize=(16, 8))\n",
"ax.scatter(random_x, random_y, alpha=.8)\n",
"\n",
"for _, lab in enumerate(labels):\n",
" ax.annotate(lab, (random_x[_]+.1, random_y[_]-.05))\n",
"\n",
"plt.title(\"random 50 embeddings\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Sx3VVV2sIYNC"
},
"source": [
"Now, let's do this with the entire set for fun..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "yDVlb1aiIYAP"
},
"outputs": [],
"source": [
"tsne = TSNE(n_components=2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 870
},
"colab_type": "code",
"id": "i8h7A5kpQKws",
"outputId": "eb779dc0-c2d0-46f4-ef1a-f547d0fc7408"
},
"outputs": [],
"source": [
"embed = tsne.fit_transform(denseX)\n",
"random_x, random_y = zip(*embed)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 870
},
"colab_type": "code",
"id": "i8h7A5kpQKws",
"outputId": "eb779dc0-c2d0-46f4-ef1a-f547d0fc7408"
},
"outputs": [],
"source": [
"plt.figure(figsize=(16, 8), dpi=80)\n",
"plt.scatter(random_x, random_y, alpha=0.3)\n",
"plt.title(\"tsne visual of all \" +str(len(v)) + \" word embeddings\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "GqVbhvK_JIA1"
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"name": "dh2018_workshop_wordvectors.ipynb",
"provenance": [],
"toc_visible": true,
"version": "0.3.2"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}