{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Programming Exercise 7:\n", "# K-means Clustering and Principal Component Analysis\n", "\n", "## Introduction\n", "\n", "In this exercise, you will implement the K-means clustering algorithm and apply it to compress an image. In the second part, you will use principal component analysis to find a low-dimensional representation of face images. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.\n", "\n", "All the information you need for solving this assignment is in this notebook, and all the code you will be implementing will take place within this notebook. The assignment can be promptly submitted to the coursera grader directly from this notebook (code and instructions are included below).\n", "\n", "Before we begin with the exercises, we need to import all libraries required for this programming exercise. Throughout the course, we will be using [`numpy`](http://www.numpy.org/) for all arrays and matrix operations, [`matplotlib`](https://matplotlib.org/) for plotting, and [`scipy`](https://docs.scipy.org/doc/scipy/reference/) for scientific and numerical computation functions and tools. You can find instructions on how to install required libraries in the README file in the [github repository](https://github.com/dibgerge/ml-coursera-python-assignments)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# used for manipulating directory paths\n", "import os\n", "\n", "# Scientific and vector computation for python\n", "import numpy as np\n", "\n", "# Import regular expressions to process emails\n", "import re\n", "\n", "# Plotting library\n", "from matplotlib import pyplot\n", "from mpl_toolkits.mplot3d import Axes3D\n", "import matplotlib as mpl\n", "\n", "from IPython.display import HTML, display, clear_output\n", "\n", "try:\n", " pyplot.rcParams[\"animation.html\"] = \"jshtml\"\n", "except ValueError:\n", " pyplot.rcParams[\"animation.html\"] = \"html5\"\n", "\n", "# Optimization module in scipy\n", "from scipy import optimize\n", "\n", "# will be used to load MATLAB mat datafile format\n", "from scipy.io import loadmat\n", "\n", "# library written for this exercise providing additional functions for assignment submission, and others\n", "import utils\n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "# define the submission/grader object for this exercise\n", "grader = utils.Grader()\n", "\n", "# tells matplotlib to embed plots within the notebook\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submission and Grading\n", "\n", "\n", "After completing each part of the assignment, be sure to submit your solutions to the grader. The following is a breakdown of how each part of this exercise is scored.\n", "\n", "\n", "| Section | Part | Submitted Function | Points |\n", "| :- |:- |:- | :-: |\n", "| 1 | [Find Closest Centroids](#section1) | [`findClosestCentroids`](#findClosestCentroids) | 30 |\n", "| 2 | [Computed Centroid Means](#section2) | [`computeCentroids`](#computeCentroids) | 30 |\n", "| 3 | [PCA](#section3) | [`pca`](#pca) | 20 |\n", "| 4 | [Project Data](#section4) | [`projectData`](#projectData) | 10 |\n", "| 5 | [Recover Data](#section5) | [`recoverData`](#recoverData) | 10 |\n", "| | Total Points | |100 |\n", "\n", "\n", "You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.\n", "\n", "
\n", "At the end of each section in this notebook, we have a cell which contains code for submitting the solutions thus far to the grader. Execute the cell to see your score up to the current section. For all your work to be submitted properly, you must execute those cells at least once.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1 K-means Clustering\n", "\n", "In this exercise, you will implement K-means algorithm and use it for image compression. You will first start on an example 2D dataset that will help you gain an intuition of how the K-means algorithm works. After\n", "that, you wil use the K-means algorithm for image compression by reducing the number of colors that occur in an image to only those that are most common in that image.\n", "\n", "### 1.1 Implementing K-means\n", "\n", "The K-means algorithm is a method to automatically cluster similar data examples together. Concretely, you are given a training set $\\{x^{(1)} , \\cdots, x^{(m)}\\}$ (where $x^{(i)} \\in \\mathbb{R}^n$), and want to group the data into a few cohesive “clusters”. The intuition behind K-means is an iterative procedure that starts by guessing the initial centroids, and then refines this guess by repeatedly assigning examples to their closest centroids and then recomputing the centroids based on the assignments.\n", "\n", "The K-means algorithm is as follows:\n", "\n", "```python\n", "centroids = kMeansInitCentroids(X, K)\n", "for i in range(iterations):\n", " # Cluster assignment step: Assign each data point to the\n", " # closest centroid. idx[i] corresponds to cˆ(i), the index\n", " # of the centroid assigned to example i\n", " idx = findClosestCentroids(X, centroids)\n", " \n", " # Move centroid step: Compute means based on centroid\n", " # assignments\n", " centroids = computeMeans(X, idx, K)\n", "```\n", "\n", "The inner-loop of the algorithm repeatedly carries out two steps: (1) Assigning each training example $x^{(i)}$ to its closest centroid, and (2) Recomputing the mean of each centroid using the points assigned to it. The K-means algorithm will always converge to some final set of means for the centroids. Note that the converged solution may not always be ideal and depends on the initial setting of the centroids. Therefore, in practice the K-means algorithm is usually run a few times with different random initializations. One way to choose between these different solutions from different random initializations is to choose the one with the lowest cost function value (distortion). You will implement the two phases of the K-means algorithm separately\n", "in the next sections." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "#### 1.1.1 Finding closest centroids\n", "\n", "In the “cluster assignment” phase of the K-means algorithm, the algorithm assigns every training example $x^{(i)}$ to its closest centroid, given the current positions of centroids. Specifically, for every example $i$ we set\n", "\n", "$$c^{(i)} := j \\quad \\text{that minimizes} \\quad \\lvert\\rvert x^{(i)} - \\mu_j \\lvert\\rvert^2, $$\n", "\n", "where $c^{(i)}$ is the index of the centroid that is closest to $x^{(i)}$, and $\\mu_j$ is the position (value) of the $j^{th}$ centroid. Note that $c^{(i)}$ corresponds to `idx[i]` in the starter code.\n", "\n", "Your task is to complete the code in the function `findClosestCentroids`. This function takes the data matrix `X` and the locations of all centroids inside `centroids` and should output a one-dimensional array `idx` that holds the index (a value in $\\{1, ..., K\\}$, where $K$ is total number of centroids) of the closest centroid to every training example.\n", "\n", "You can implement this using a loop over every training example and every centroid.\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def findClosestCentroids(X, centroids):\n", " \"\"\"\n", " Computes the centroid memberships for every example.\n", " \n", " Parameters\n", " ----------\n", " X : array_like\n", " The dataset of size (m, n) where each row is a single example. \n", " That is, we have m examples each of n dimensions.\n", " \n", " centroids : array_like\n", " The k-means centroids of size (K, n). K is the number\n", " of clusters, and n is the the data dimension.\n", " \n", " Returns\n", " -------\n", " idx : array_like\n", " A vector of size (m, ) which holds the centroids assignment for each\n", " example (row) in the dataset X.\n", " \n", " Instructions\n", " ------------\n", " Go over every example, find its closest centroid, and store\n", " the index inside `idx` at the appropriate location.\n", " Concretely, idx[i] should contain the index of the centroid\n", " closest to example i. Hence, it should be a value in the \n", " range 0..K-1\n", "\n", " Note\n", " ----\n", " You can use a for-loop over the examples to compute this.\n", " \"\"\"\n", " # Set K\n", " K = centroids.shape[0]\n", "\n", " # You need to return the following variables correctly.\n", " idx = np.zeros(X.shape[0], dtype=int)\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", " \n", " \n", " # =============================================================\n", " return idx" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you have completed the code in `findClosestCentroids`, the following cell will run your code and you should see the output `[0 2 1]` corresponding to the centroid assignments for the first 3 examples." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load an example dataset that we will be using\n", "data = loadmat(os.path.join('Data', 'ex7data2.mat'))\n", "X = data['X']\n", "\n", "# Select an initial set of centroids\n", "K = 3 # 3 Centroids\n", "initial_centroids = np.array([[3, 3], [6, 2], [8, 5]])\n", "\n", "# Find the closest centroids for the examples using the initial_centroids\n", "idx = findClosestCentroids(X, initial_centroids)\n", "\n", "print('Closest centroids for the first 3 examples:')\n", "print(idx[:3])\n", "print('(the closest centroids should be 0, 2, 1 respectively)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You should now submit your solutions.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grader[1] = findClosestCentroids\n", "grader.grade()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### 1.1.2 Computing centroid means\n", "\n", "Given assignments of every point to a centroid, the second phase of the algorithm recomputes, for each centroid, the mean of the points that were assigned to it. Specifically, for every centroid $k$ we set\n", "\n", "$$ \\mu_k := \\frac{1}{\\left| C_k\\right|} \\sum_{i \\in C_k} x^{(i)}$$\n", "\n", "where $C_k$ is the set of examples that are assigned to centroid $k$. Concretely, if two examples say $x^{(3)}$ and $x^{(5)}$ are assigned to centroid $k = 2$, then you should update $\\mu_2 = \\frac{1}{2} \\left( x^{(3)} + x^{(5)} \\right)$.\n", "\n", "You should now complete the code in the function `computeCentroids`. You can implement this function using a loop over the centroids. You can also use a loop over the examples; but if you can use a vectorized implementation that does not use such a loop, your code may run faster.\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def computeCentroids(X, idx, K):\n", " \"\"\"\n", " Returns the new centroids by computing the means of the data points\n", " assigned to each centroid.\n", " \n", " Parameters\n", " ----------\n", " X : array_like\n", " The datset where each row is a single data point. That is, it \n", " is a matrix of size (m, n) where there are m datapoints each\n", " having n dimensions. \n", " \n", " idx : array_like \n", " A vector (size m) of centroid assignments (i.e. each entry in range [0 ... K-1])\n", " for each example.\n", " \n", " K : int\n", " Number of clusters\n", " \n", " Returns\n", " -------\n", " centroids : array_like\n", " A matrix of size (K, n) where each row is the mean of the data \n", " points assigned to it.\n", " \n", " Instructions\n", " ------------\n", " Go over every centroid and compute mean of all points that\n", " belong to it. Concretely, the row vector centroids[i, :]\n", " should contain the mean of the data points assigned to\n", " cluster i.\n", "\n", " Note:\n", " -----\n", " You can use a for-loop over the centroids to compute this.\n", " \"\"\"\n", " # Useful variables\n", " m, n = X.shape\n", " # You need to return the following variables correctly.\n", " centroids = np.zeros((K, n))\n", "\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", " \n", " \n", " # =============================================================\n", " return centroids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you have completed the code in `computeCentroids`, the following cell will run your code and output the centroids after the first step of K-means." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Compute means based on the closest centroids found in the previous part.\n", "centroids = computeCentroids(X, idx, K)\n", "\n", "print('Centroids computed after initial finding of closest centroids:')\n", "print(centroids)\n", "print('\\nThe centroids should be')\n", "print(' [ 2.428301 3.157924 ]')\n", "print(' [ 5.813503 2.633656 ]')\n", "print(' [ 7.119387 3.616684 ]')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You should now submit your solutions.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grader[2] = computeCentroids\n", "grader.grade()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 K-means on example dataset \n", "\n", "After you have completed the two functions (`findClosestCentroids` and `computeCentroids`), you have all the necessary pieces to run the K-means algorithm. The next cell will run the K-means algorithm on a toy 2D dataset to help you understand how K-means works. Your functions are called from inside the `runKmeans` function (in this assignment's `utils.py` module). We encourage you to take a look at the function to understand how it works. Notice that the code calls the two functions you implemented in a loop.\n", "\n", "When you run the next step, the K-means code will produce an animation that steps you through the progress of the algorithm at each iteration. At the end, your figure should look as the one displayed below.\n", "\n", "![](Figures/kmeans_result.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": false }, "outputs": [], "source": [ "# Load an example dataset\n", "data = loadmat(os.path.join('Data', 'ex7data2.mat'))\n", "\n", "# Settings for running K-Means\n", "K = 3\n", "max_iters = 10\n", "\n", "# For consistency, here we set centroids to specific values\n", "# but in practice you want to generate them automatically, such as by\n", "# settings them to be random examples (as can be seen in\n", "# kMeansInitCentroids).\n", "initial_centroids = np.array([[3, 3], [6, 2], [8, 5]])\n", "\n", "\n", "# Run K-Means algorithm. The 'true' at the end tells our function to plot\n", "# the progress of K-Means\n", "centroids, idx, anim = utils.runkMeans(X, initial_centroids,\n", " findClosestCentroids, computeCentroids, max_iters, True)\n", "anim" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3 Random initialization \n", "\n", "The initial assignments of centroids for the example dataset in the previous cell were designed so that you will see the same figure as that shown in the cell above. In practice, a\n", "good strategy for initializing the centroids is to select random examples from the training set.\n", "\n", "In this part of the exercise, you should complete the function `kMeansInitCentroids` with the following code:\n", "\n", "```python\n", "# Initialize the centroids to be random examples\n", "\n", "# Randomly reorder the indices of examples\n", "randidx = np.random.permutation(X.shape[0])\n", "# Take the first K examples as centroids\n", "centroids = X[randidx[:K], :]\n", "```\n", "\n", "The code above first randomly permutes the indices of the examples (using `permute` within the `numpy.random` module). Then, it selects the first $K$ examples based on the random permutation of the indices. This allows the examples to be selected at random without the risk of selecting the same example twice.\n", "\n", "*You do not need to make any submission for this part of the exercise*\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def kMeansInitCentroids(X, K):\n", " \"\"\"\n", " This function initializes K centroids that are to be used in K-means on the dataset x.\n", " \n", " Parameters\n", " ----------\n", " X : array_like \n", " The dataset of size (m x n).\n", " \n", " K : int\n", " The number of clusters.\n", " \n", " Returns\n", " -------\n", " centroids : array_like\n", " Centroids of the clusters. This is a matrix of size (K x n).\n", " \n", " Instructions\n", " ------------\n", " You should set centroids to randomly chosen examples from the dataset X.\n", " \"\"\"\n", " m, n = X.shape\n", " \n", " # You should return this values correctly\n", " centroids = np.zeros((K, n))\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", "\n", " \n", " # =============================================================\n", " return centroids" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.4 Image compression with K-means\n", "\n", "In this exercise, you will apply K-means to image compression. We will use the image below as an example (property of Frank Wouters with permission to this class).\n", "\n", "![](Data/bird_small.png)\n", "\n", "In a straightforward 24-bit color representation of an image, each pixel is represented as three 8-bit unsigned integers (ranging from 0 to 255) that specify the red, green and blue intensity values. This encoding is often referred to as the RGB encoding. Our image contains thousands of colors, and in this part of the exercise, you will reduce the number of colors to 16 colors.\n", "\n", "By making this reduction, it is possible to represent (compress) the photo in an efficient way. Specifically, you only need to store the RGB values of the 16 selected colors, and for each pixel in the image you now need to only store the index of the color at that location (where only 4 bits are necessary to represent 16 possibilities).\n", "\n", "In this exercise, you will use the K-means algorithm to select the 16 colors that will be used to represent the compressed image. Concretely, you will treat every pixel in the original image as a data example and use the K-means algorithm to find the 16 colors that best group (cluster) the pixels in the 3-dimensional RGB space. Once you have computed the cluster centroids on the image, you will then use the 16 colors to replace the pixels in the original image.\n", "\n", "#### 1.4.1 K-means on pixels\n", "\n", "In python, images can be read in as follows:\n", "\n", "```python\n", "# Load 128x128 color image (bird_small.png)\n", "img = mpl.image.imread(os.path.join('Data', 'bird_small.png'))\n", "\n", "# We have already imported matplotlib as mpl at the beginning of this notebook.\n", "```\n", "This creates a three-dimensional matrix `A` whose first two indices identify a pixel position and whose last index represents red, green, or blue. For example, A[50, 33, 2] gives the blue intensity of the pixel at row 51 and column 34.\n", "\n", "The code in the following cell first loads the image, and then reshapes it to create an m x 3 matrix of pixel colors (where m = 16384 = 128 x 128), and calls your K-means function on it.\n", "\n", "After finding the top K = 16 colors to represent the image, you can now assign each pixel position to its closest centroid using the `findClosestCentroids` function. This allows you to represent the original image using the centroid assignments of each pixel. Notice that you have significantly reduced the number of bits that are required to describe the image. The original image required 24 bits for each one of the 128 x 128 pixel locations, resulting in total size of 128 x 128 x 24 = 393,216 bits. The new representation requires some overhead storage in form of a dictionary of 16 colors, each of which require 24 bits, but the image itself then only requires 4 bits per pixel location. The final number of bits used is therefore 16 x 24 + 128 x 128 x 4 = 65,920 bits, which corresponds to compressing the original image by about a factor of 6.\n", "\n", "Finally, you can view the effects of the compression by reconstructing the image based only on the centroid assignments. Specifically, you can replace each pixel location with the mean of the centroid assigned to it. The figure below shows the reconstruction we obtained. \n", "\n", "![](Figures/bird_compression.png)\n", "\n", "Even though the resulting image retains most of the characteristics of the original, we also see some compression artifacts.\n", "\n", "Run the following cell to compute the centroids and the centroid allocation of each pixel in the image." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ======= Experiment with these parameters ================\n", "# You should try different values for those parameters\n", "K = 16\n", "max_iters = 10\n", "\n", "# Load an image of a bird\n", "# Change the file name and path to experiment with your own images\n", "A = mpl.image.imread(os.path.join('Data', 'bird_small.png'))\n", "# ==========================================================\n", "\n", "# Divide by 255 so that all values are in the range 0 - 1\n", "A /= 255\n", "\n", "# Reshape the image into an Nx3 matrix where N = number of pixels.\n", "# Each row will contain the Red, Green and Blue pixel values\n", "# This gives us our dataset matrix X that we will use K-Means on.\n", "X = A.reshape(-1, 3)\n", "\n", "# When using K-Means, it is important to randomly initialize centroids\n", "# You should complete the code in kMeansInitCentroids above before proceeding\n", "initial_centroids = kMeansInitCentroids(X, K)\n", "\n", "# Run K-Means\n", "centroids, idx = utils.runkMeans(X, initial_centroids,\n", " findClosestCentroids,\n", " computeCentroids,\n", " max_iters)\n", "\n", "# We can now recover the image from the indices (idx) by mapping each pixel\n", "# (specified by its index in idx) to the centroid value\n", "# Reshape the recovered image into proper dimensions\n", "X_recovered = centroids[idx, :].reshape(A.shape)\n", "\n", "# Display the original image, rescale back by 255\n", "fig, ax = pyplot.subplots(1, 2, figsize=(8, 4))\n", "ax[0].imshow(A*255)\n", "ax[0].set_title('Original')\n", "ax[0].grid(False)\n", "\n", "# Display compressed image, rescale back by 255\n", "ax[1].imshow(X_recovered*255)\n", "ax[1].set_title('Compressed, with %d colors' % K)\n", "ax[1].grid(False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You do not need to make any submissions for this part of the exercise.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.5 Optional (ungraded) exercise: Use your own image\n", "\n", "In this exercise, modify the code we have supplied in the previous cell to run on one of your own images. Note that if your image is very large, then K-means can take a long time to run. Therefore, we recommend that you resize your images to\n", "manageable sizes before running the code. You can also try to vary $K$ to see the effects on the compression." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2 Principal Component Analysis\n", "\n", "In this exercise, you will use principal component analysis (PCA) to perform dimensionality reduction. You will first experiment with an example 2D dataset to get intuition on how PCA works, and then use it on a bigger dataset of 5000 face image dataset.\n", "\n", "### 2.1 Example Dataset\n", "\n", "To help you understand how PCA works, you will first start with a 2D dataset which has one direction of large variation and one of smaller variation. The cell below will plot the training data, also shown in here:\n", "\n", "In this part of the exercise, you will visualize what happens when you use PCA to reduce the data from 2D to 1D. In practice, you might want to reduce data from 256 to 50 dimensions, say; but using lower dimensional data in this example allows us to visualize the algorithms better." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load the dataset into the variable X \n", "data = loadmat(os.path.join('Data', 'ex7data1.mat'))\n", "X = data['X']\n", "\n", "# Visualize the example dataset\n", "pyplot.plot(X[:, 0], X[:, 1], 'bo', ms=10, mec='k', mew=1)\n", "pyplot.axis([0.5, 6.5, 2, 8])\n", "pyplot.gca().set_aspect('equal')\n", "pyplot.grid(False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### 2.2 Implementing PCA\n", "\n", "In this part of the exercise, you will implement PCA. PCA consists of two computational steps: \n", "\n", "1. Compute the covariance matrix of the data.\n", "2. Use SVD (in python we use numpy's implementation `np.linalg.svd`) to compute the eigenvectors $U_1$, $U_2$, $\\dots$, $U_n$. These will correspond to the principal components of variation in the data.\n", "\n", "First, you should compute the covariance matrix of the data, which is given by:\n", "\n", "$$ \\Sigma = \\frac{1}{m} X^T X$$\n", "\n", "where $X$ is the data matrix with examples in rows, and $m$ is the number of examples. Note that $\\Sigma$ is a $n \\times n$ matrix and not the summation operator. \n", "\n", "After computing the covariance matrix, you can run SVD on it to compute the principal components. In python and `numpy` (or `scipy`), you can run SVD with the following command: `U, S, V = np.linalg.svd(Sigma)`, where `U` will contain the principal components and `S` will contain a diagonal matrix. Note that the `scipy` library also has a similar function to compute SVD `scipy.linalg.svd`. The functions in the two libraries use the same C-based library (LAPACK) for the SVD computation, but the `scipy` version provides more options and arguments to control SVD computation. In this exercise, we will stick with the `numpy` implementation of SVD.\n", "\n", "Complete the code in the following cell to implemente PCA.\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def pca(X):\n", " \"\"\"\n", " Run principal component analysis.\n", " \n", " Parameters\n", " ----------\n", " X : array_like\n", " The dataset to be used for computing PCA. It has dimensions (m x n)\n", " where m is the number of examples (observations) and n is \n", " the number of features.\n", " \n", " Returns\n", " -------\n", " U : array_like\n", " The eigenvectors, representing the computed principal components\n", " of X. U has dimensions (n x n) where each column is a single \n", " principal component.\n", " \n", " S : array_like\n", " A vector of size n, contaning the singular values for each\n", " principal component. Note this is the diagonal of the matrix we \n", " mentioned in class.\n", " \n", " Instructions\n", " ------------\n", " You should first compute the covariance matrix. Then, you\n", " should use the \"svd\" function to compute the eigenvectors\n", " and eigenvalues of the covariance matrix. \n", "\n", " Notes\n", " -----\n", " When computing the covariance matrix, remember to divide by m (the\n", " number of examples).\n", " \"\"\"\n", " # Useful values\n", " m, n = X.shape\n", "\n", " # You need to return the following variables correctly.\n", " U = np.zeros(n)\n", " S = np.zeros(n)\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", " \n", " \n", " # ============================================================\n", " return U, S" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before using PCA, it is important to first normalize the data by subtracting the mean value of each feature from the dataset, and scaling each dimension so that they are in the same range.\n", "\n", "In the next cell, this normalization will be performed for you using the `utils.featureNormalize` function.\n", "After normalizing the data, you can run PCA to compute the principal components. Your task is to complete the code in the function `pca` to compute the principal components of the dataset. \n", "\n", "Once you have completed the function `pca`, the following cell will run PCA on the example dataset and plot the corresponding principal components found similar to the figure below. \n", "\n", "![](Figures/pca_components.png)\n", "\n", "\n", "The following cell will also output the top principal component (eigenvector) found, and you should expect to see an output of about `[-0.707 -0.707]`. (It is possible that `numpy` may instead output the negative of this, since $U_1$ and $-U_1$ are equally valid choices for the first principal component.)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Before running PCA, it is important to first normalize X\n", "X_norm, mu, sigma = utils.featureNormalize(X)\n", "\n", "# Run PCA\n", "U, S = pca(X_norm)\n", "\n", "# Draw the eigenvectors centered at mean of data. These lines show the\n", "# directions of maximum variations in the dataset.\n", "fig, ax = pyplot.subplots()\n", "ax.plot(X[:, 0], X[:, 1], 'bo', ms=10, mec='k', mew=0.25)\n", "\n", "for i in range(2):\n", " ax.arrow(mu[0], mu[1], 1.5 * S[i]*U[0, i], 1.5 * S[i]*U[1, i],\n", " head_width=0.25, head_length=0.2, fc='k', ec='k', lw=2, zorder=1000)\n", "\n", "ax.axis([0.5, 6.5, 2, 8])\n", "ax.set_aspect('equal')\n", "ax.grid(False)\n", "\n", "print('Top eigenvector: U[:, 0] = [{:.6f} {:.6f}]'.format(U[0, 0], U[1, 0]))\n", "print(' (you should expect to see [-0.707107 -0.707107])')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You should now submit your solutions.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grader[3] = pca\n", "grader.grade()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Dimensionality Reduction with PCA\n", "\n", "After computing the principal components, you can use them to reduce the feature dimension of your dataset by projecting each example onto a lower dimensional space, $x^{(i)} \\rightarrow z^{(i)}$ (e.g., projecting the data from 2D to 1D). In this part of the exercise, you will use the eigenvectors returned by PCA and\n", "project the example dataset into a 1-dimensional space. In practice, if you were using a learning algorithm such as linear regression or perhaps neural networks, you could now use the projected data instead of the original data. By using the projected data, you can train your model faster as there are less dimensions in the input.\n", "\n", "\n", "\n", "#### 2.3.1 Projecting the data onto the principal components\n", "\n", "You should now complete the code in the function `projectData`. Specifically, you are given a dataset `X`, the principal components `U`, and the desired number of dimensions to reduce to `K`. You should project each example in `X` onto the top `K` components in `U`. Note that the top `K` components in `U` are given by\n", "the first `K` columns of `U`, that is `Ureduce = U[:, :K]`.\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def projectData(X, U, K):\n", " \"\"\"\n", " Computes the reduced data representation when projecting only \n", " on to the top K eigenvectors.\n", " \n", " Parameters\n", " ----------\n", " X : array_like\n", " The input dataset of shape (m x n). The dataset is assumed to be \n", " normalized.\n", " \n", " U : array_like\n", " The computed eigenvectors using PCA. This is a matrix of \n", " shape (n x n). Each column in the matrix represents a single\n", " eigenvector (or a single principal component).\n", " \n", " K : int\n", " Number of dimensions to project onto. Must be smaller than n.\n", " \n", " Returns\n", " -------\n", " Z : array_like\n", " The projects of the dataset onto the top K eigenvectors. \n", " This will be a matrix of shape (m x k).\n", " \n", " Instructions\n", " ------------\n", " Compute the projection of the data using only the top K \n", " eigenvectors in U (first K columns). \n", " For the i-th example X[i,:], the projection on to the k-th \n", " eigenvector is given as follows:\n", " \n", " x = X[i, :]\n", " projection_k = np.dot(x, U[:, k])\n", "\n", " \"\"\"\n", " # You need to return the following variables correctly.\n", " Z = np.zeros((X.shape[0], K))\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", "\n", " \n", " # =============================================================\n", " return Z" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you have completed the code in `projectData`, the following cell will project the first example onto the first dimension and you should see a value of about 1.481 (or possibly -1.481, if you got $-U_1$ instead of $U_1$)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Project the data onto K = 1 dimension\n", "K = 1\n", "Z = projectData(X_norm, U, K)\n", "print('Projection of the first example: {:.6f}'.format(Z[0, 0]))\n", "print('(this value should be about : 1.481274)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You should now submit your solutions.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grader[4] = projectData\n", "grader.grade()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "#### 2.3.2 Reconstructing an approximation of the data\n", "\n", "After projecting the data onto the lower dimensional space, you can approximately recover the data by projecting them back onto the original high dimensional space. Your task is to complete the function `recoverData` to project each example in `Z` back onto the original space and return the recovered approximation in `Xrec`.\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def recoverData(Z, U, K):\n", " \"\"\"\n", " Recovers an approximation of the original data when using the \n", " projected data.\n", " \n", " Parameters\n", " ----------\n", " Z : array_like\n", " The reduced data after applying PCA. This is a matrix\n", " of shape (m x K).\n", " \n", " U : array_like\n", " The eigenvectors (principal components) computed by PCA.\n", " This is a matrix of shape (n x n) where each column represents\n", " a single eigenvector.\n", " \n", " K : int\n", " The number of principal components retained\n", " (should be less than n).\n", " \n", " Returns\n", " -------\n", " X_rec : array_like\n", " The recovered data after transformation back to the original \n", " dataset space. This is a matrix of shape (m x n), where m is \n", " the number of examples and n is the dimensions (number of\n", " features) of original datatset.\n", " \n", " Instructions\n", " ------------\n", " Compute the approximation of the data by projecting back\n", " onto the original space using the top K eigenvectors in U.\n", " For the i-th example Z[i,:], the (approximate)\n", " recovered data for dimension j is given as follows:\n", "\n", " v = Z[i, :]\n", " recovered_j = np.dot(v, U[j, :K])\n", "\n", " Notice that U[j, :K] is a vector of size K.\n", " \"\"\"\n", " # You need to return the following variables correctly.\n", " X_rec = np.zeros((Z.shape[0], U.shape[0]))\n", "\n", " # ====================== YOUR CODE HERE ======================\n", "\n", " \n", "\n", " # =============================================================\n", " return X_rec" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you have completed the code in `recoverData`, the following cell will recover an approximation of the first example and you should see a value of about `[-1.047 -1.047]`. The code will then plot the data in this reduced dimension space. This will show you what the data looks like when using only the corresponding eigenvectors to reconstruct it. An example of what you should get for PCA projection is shown in this figure: \n", "\n", "![](Figures/pca_reconstruction.png)\n", "\n", "In the figure above, the original data points are indicated with the blue circles, while the projected data points are indicated with the red circles. The projection effectively only retains the information in the direction given by $U_1$. The dotted lines show the distance from the data points in original space to the projected space. Those dotted lines represent the error measure due to PCA projection." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X_rec = recoverData(Z, U, K)\n", "print('Approximation of the first example: [{:.6f} {:.6f}]'.format(X_rec[0, 0], X_rec[0, 1]))\n", "print(' (this value should be about [-1.047419 -1.047419])')\n", "\n", "# Plot the normalized dataset (returned from featureNormalize)\n", "fig, ax = pyplot.subplots(figsize=(5, 5))\n", "ax.plot(X_norm[:, 0], X_norm[:, 1], 'bo', ms=8, mec='b', mew=0.5)\n", "ax.set_aspect('equal')\n", "ax.grid(False)\n", "pyplot.axis([-3, 2.75, -3, 2.75])\n", "\n", "# Draw lines connecting the projected points to the original points\n", "ax.plot(X_rec[:, 0], X_rec[:, 1], 'ro', mec='r', mew=2, mfc='none')\n", "for xnorm, xrec in zip(X_norm, X_rec):\n", " ax.plot([xnorm[0], xrec[0]], [xnorm[1], xrec[1]], '--k', lw=1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*You should now submit your solutions.*" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grader[5] = recoverData\n", "grader.grade()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4 Face Image Dataset\n", "\n", "In this part of the exercise, you will run PCA on face images to see how it can be used in practice for dimension reduction. The dataset `ex7faces.mat` contains a dataset `X` of face images, each $32 \\times 32$ in grayscale. This dataset was based on a [cropped version](http://conradsanderson.id.au/lfwcrop/) of the [labeled faces in the wild](http://vis-www.cs.umass.edu/lfw/) dataset. Each row of `X` corresponds to one face image (a row vector of length 1024). \n", "\n", "The next cell will load and visualize the first 100 of these face images similar to what is shown in this figure:\n", "\n", "![Faces](Figures/faces.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load Face dataset\n", "data = loadmat(os.path.join('Data', 'ex7faces.mat'))\n", "X = data['X']\n", "\n", "# Display the first 100 faces in the dataset\n", "utils.displayData(X[:100, :], figsize=(8, 8))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.4.1 PCA on Faces\n", "\n", "To run PCA on the face dataset, we first normalize the dataset by subtracting the mean of each feature from the data matrix `X`. After running PCA, you will obtain the principal components of the dataset. Notice that each principal component in `U` (each column) is a vector of length $n$ (where for the face dataset, $n = 1024$). It turns out that we can visualize these principal components by reshaping each of them into a $32 \\times 32$ matrix that corresponds to the pixels in the original dataset. \n", "\n", "The following cell will first normalize the dataset for you and then run your PCA code. Then, the first 36 principal components (conveniently called eigenfaces) that describe the largest variations are displayed. If you want, you can also change the code to display more principal components to see how they capture more and more details." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# normalize X by subtracting the mean value from each feature\n", "X_norm, mu, sigma = utils.featureNormalize(X)\n", "\n", "# Run PCA\n", "U, S = pca(X_norm)\n", "\n", "# Visualize the top 36 eigenvectors found\n", "utils.displayData(U[:, :36].T, figsize=(8, 8))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.4.2 Dimensionality Reduction\n", "\n", "Now that you have computed the principal components for the face dataset, you can use it to reduce the dimension of the face dataset. This allows you to use your learning algorithm with a smaller input size (e.g., 100 dimensions) instead of the original 1024 dimensions. This can help speed up your learning algorithm.\n", "\n", "The next cell will project the face dataset onto only the first 100 principal components. Concretely, each face image is now described by a vector $z^{(i)} \\in \\mathbb{R}^{100}$. To understand what is lost in the dimension reduction, you can recover the data using only the projected dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Project images to the eigen space using the top k eigenvectors \n", "# If you are applying a machine learning algorithm \n", "K = 100\n", "Z = projectData(X_norm, U, K)\n", "\n", "print('The projected data Z has a shape of: ', Z.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next cell, an approximate recovery of the data is performed and the original and projected face images\n", "are displayed similar to what is shown here:\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", "\n", "From the reconstruction, you can observe that the general structure and appearance of the face are kept while the fine details are lost. This is a remarkable reduction (more than 10x) in the dataset size that can help speed up your learning algorithm significantly. For example, if you were training a neural network to perform person recognition (given a face image, predict the identity of the person), you can use the dimension reduced input of only a 100 dimensions instead of the original pixels." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Project images to the eigen space using the top K eigen vectors and \n", "# visualize only using those K dimensions\n", "# Compare to the original input, which is also displayed\n", "K = 100\n", "X_rec = recoverData(Z, U, K)\n", "\n", "# Display normalized data\n", "utils.displayData(X_norm[:100, :], figsize=(6, 6))\n", "pyplot.gcf().suptitle('Original faces')\n", "\n", "# Display reconstructed data from only k eigenfaces\n", "utils.displayData(X_rec[:100, :], figsize=(6, 6))\n", "pyplot.gcf().suptitle('Recovered faces')\n", "pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.5 Optional (ungraded) exercise: PCA for visualization\n", "\n", "In the earlier K-means image compression exercise, you used the K-means algorithm in the 3-dimensional RGB space. We reduced each pixel of the RGB image to be represented by 16 clusters. In the next cell, we have provided code to visualize the final pixel assignments in this 3D space. Each data point is colored according to the cluster it has been assigned to. You can drag your mouse on the figure to rotate and inspect this data in 3 dimensions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# this allows to have interactive plot to rotate the 3-D plot\n", "# The double identical statement is on purpose\n", "# see: https://stackoverflow.com/questions/43545050/using-matplotlib-notebook-after-matplotlib-inline-in-jupyter-notebook-doesnt\n", "%matplotlib notebook\n", "%matplotlib notebook\n", "from matplotlib import pyplot\n", "\n", "\n", "A = mpl.image.imread(os.path.join('Data', 'bird_small.png'))\n", "A /= 255\n", "X = A.reshape(-1, 3)\n", "\n", "# perform the K-means clustering again here\n", "K = 16\n", "max_iters = 10\n", "initial_centroids = kMeansInitCentroids(X, K)\n", "centroids, idx = utils.runkMeans(X, initial_centroids,\n", " findClosestCentroids,\n", " computeCentroids, max_iters)\n", "\n", "# Sample 1000 random indexes (since working with all the data is\n", "# too expensive. If you have a fast computer, you may increase this.\n", "sel = np.random.choice(X.shape[0], size=1000)\n", "\n", "fig = pyplot.figure(figsize=(6, 6))\n", "ax = fig.add_subplot(111, projection='3d')\n", "\n", "ax.scatter(X[sel, 0], X[sel, 1], X[sel, 2], cmap='rainbow', c=idx[sel], s=8**2)\n", "ax.set_title('Pixel dataset plotted in 3D.\\nColor shows centroid memberships')\n", "pass" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It turns out that visualizing datasets in 3 dimensions or greater can be cumbersome. Therefore, it is often desirable to only display the data in 2D even at the cost of losing some information. In practice, PCA is often used to reduce the dimensionality of data for visualization purposes. \n", "\n", "In the next cell,we will apply your implementation of PCA to the 3-dimensional data to reduce it to 2 dimensions and visualize the result in a 2D scatter plot. The PCA projection can be thought of as a rotation that selects the view that maximizes the spread of the data, which often corresponds to the “best” view." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Subtract the mean to use PCA\n", "X_norm, mu, sigma = utils.featureNormalize(X)\n", "\n", "# PCA and project the data to 2D\n", "U, S = pca(X_norm)\n", "Z = projectData(X_norm, U, 2)\n", "\n", "# Reset matplotlib to non-interactive\n", "%matplotlib inline\n", "\n", "fig = pyplot.figure(figsize=(6, 6))\n", "ax = fig.add_subplot(111)\n", "\n", "ax.scatter(Z[sel, 0], Z[sel, 1], cmap='rainbow', c=idx[sel], s=64)\n", "ax.set_title('Pixel dataset plotted in 2D, using PCA for dimensionality reduction')\n", "ax.grid(False)\n", "pass" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.4" } }, "nbformat": 4, "nbformat_minor": 2 }