{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "s1", "content", "l1" ] }, "source": [ "# Linear Algebra - Basics\n", "\n", "# Introduction to Linear Algebra\n", "\n", "Linear algebra is a branch of mathematics that deals with equations of straight lines. A line is made up of multiple points. A point in 2 dimensional (2D) space is represented using two coordinates (x,y). \n", "\n", "General form of equation of straight line passing through any point (x,y) will be of the form ax+by+c=0. \n", "Eg. 2x+1y-5=0 is an equation, where x and y is different for different point.There are numerous points (x1,y1),(x2,y2),(x3,y3),(x4,y4).... lying on this straight line. (1,3),(2,1) are couple of points that lie on this line. \n", "\n", "In 3D space, a point is represented by three coordinates (x,y,z) or (x1,x2,x3).
\n", "In general, a point in n-dimensional space is represented using n coordinates (x1,x2,...xn). Coordinates of several points may be represented using superscript (x11,x21, ...,xn1), (x12,x22,...,xn2),....,(x1n, x2n,...,xnn)\n", "\n", "In 2D space, a hyperplane is 1D. This 1D hyperplane is a line. In 3D space, a hyperplane is 2D. Similarly, in an n-dimensional space, the hyperplane is n-1 dimensional.\n", "\n", "\n", "## Scalars, Vectors, Matrices and Tensors in programming parlance\n", "\n", "Scalar: A number, generally rational number, is called a 'scalar'. Eg. 2.4,5,100.9001
\n", "\n", "```python\n", "Sample code:\n", "a=5 #assigns 5 to variable a
\n", "b=numpy.pi #assigns value of pi to variable b
\n", "```\n", "\n", "Vector: The coordinates of a point is represented using an array of numbers called a 'vector'. Vector is a 1D array. A 1xm array is called a row vector and has 1 row and m columns. An mx1 array is called a column vector and has m rows of values in 1 column. Eg. A point (6,-1,2) in 3D is represented by a row vector [6,-1,2] or a column vector [[6],[-1],[2]].\n", "\n", "```python\n", "Sample code:\n", "v1= [[1,2,3]] #creates 1x3 list row vector\n", "v2= [[5],[1.5],[2]] #creates 3x1 list column vector\n", "v3= numpy.array([[8,3,2]]) #creates 1x3 ndarray row vector\n", "```\n", "\n", "Matrix: Multiple points can be represented using a matrix, which is a 2D array of numbers, that is also a collection of row/column vectors. Size of a 2D array is written as mxn. ie. m rows and n columns. A 2D matrix with same number of rows and columns is known as a square matrix ie matrix of size mxm.\n", "\n", "```python\n", "Sample code:\n", "x=[[1,2,3],[1,2,1]] #creates 2x3 matrix-like list \n", "y=numpy.ndarray([[4,2,2],[2,100,9]]) #creates 2x3 ndarray matrix\n", "```\n", "\n", "Tensor: A 'tensor' is a 3D (mxnxp) or higher dimensional array. \n", "\n", "```python\n", "Sample code:\n", "t1=numpy.ndarray([5,5,2]) #creates a 5x5x2 3D ndarray matrix with random values\n", "```\n", "\n", "\n", "\n", "
\n", "
\n", "\n", "## Matrix Operations - Addition, Subtraction, Multiplication, Transpose and Inverse\n", "\n", "### Addition and Subtraction\n", "\n", "Two matrices can be added, only if they have the same dimensions. ie. a matrix of 5x6 can be added to another matrix of 5x6 only . In general, two matrices of sizes mxn can be added. Same rule holds true for subtraction. For these operations, matrices can be treated as variables and '+' and '-' operators can be used to perform addition and subtraction.\n", " \n", "Addition\n", "\n", "$\\left[\\begin{array}{cc}x_{11} & x_{12}\\\\x_{21} & x_{22}\\\\\\end{array}\\right] + \\left[\\begin{array}{cc}y_{11} & y_{12}\\\\y_{21} & y_{22}\\\\\\end{array}\\right]$ = $\\left[\\begin{array}{cc}(x_{11}+y_{11}) & (x_{12}+y_{12})\\\\(x_{21}+y_{21}) & (x_{22}+y_{22})\\\\\\end{array}\\right]$\n", "\n", "
\n",
    "Syntax:\n",
    "a+b   - returns an ndarray matrix with individual corresponding elements added, \n",
    "        where a and b are ndarrays of same size\n",
    "
\n", "\n", "```python\n", "Sample code:\n", "a=[[4,5,2],[1,2,6]] # creates 2x3 matrix-like list\n", "b=numpy.array([[1,2,4],[100,2,3]]) # creates 2x3 ndarray matrix\n", "numpy.array(a) + b # variable 'a' has to be converted to an ndarray matrix \n", " # to perform addition\n", "```\n", "Subtraction\n", "\n", "$\\left[\\begin{array}{cc}x_{11} & x_{12}\\\\x_{21} & x_{22}\\\\\\end{array}\\right] - \\left[\\begin{array}{cc}y_{11} & y_{12}\\\\y_{21} & y_{22}\\\\\\end{array}\\right]$ = $\\left[\\begin{array}{cc}(x_{11}-y_{11}) & (x_{12}-y_{12})\\\\(x_{21}-y_{21}) & (x_{22}-y_{22})\\\\\\end{array}\\right]$\n", "
\n", "
\n",
    "Syntax: \n",
    "a-b    - returns an ndarray matrix with individual corresponding elements of b \n",
    "         subtracted from a where a and b are ndarrays of same size\n",
    "
\n", "\n", "### Mutliplication\n", "\n", "Two matrices can be multiplied with each other if the number of columns of the first matrix is equal to the number of rows of the second matrix. The product of two matrices will be another matrix with dimensions as number of rows equal to the number of rows of first matrix and number of columns equal to the number of columns of the second matrix. ie. matrix1 of size mxn can be multiplied with matrix2 of size nxp and the resulting matrix will have size mxp. \n", "\n", "\n", "\n", "$\\left[\\begin{array}{cc}x_{11} & x_{12}\\\\x_{21} & x_{22}\\\\x_{31} & x_{32}\\\\\\end{array}\\right] * \\left[\\begin{array}{cc}y_{11} & y_{12} & y_{13} & y_{14}\\\\y_{21} & y_{22} & y_{23} & y_{24}\\\\\\end{array}\\right]$ = $\\left[\\begin{array}{cc}(x_{11}*y_{11})+(x_{12}*y_{21}) & (x_{11}*y_{12})+(x_{12}*y_{22}) & (x_{11}*y_{13})+(x_{12}*y_{23}) & (x_{11}*y_{14})+(x_{12}*y_{24})\\\\(x_{21}*y_{11})+(x_{22}*y_{21}) & (x_{21}*y_{12})+(x_{22}*y_{22}) & (x_{21}*y_{13})+(x_{22}*y_{23}) & (x_{21}*y_{14})+(x_{22}*y_{24})\\\\(x_{31}*y_{11})+(x_{32}*y_{21}) & (x_{31}*y_{12})+(x_{32}*y_{22}) & (x_{31}*y_{13})+(x_{32}*y_{23}) & (x_{31}*y_{14})+(x_{32}*y_{24})\\\\\\end{array}\\right]$\n", "
\n", "
\n", "
\n",
    "Syntax:\n",
    "numpy.dot(matrix1,matrix2)      - returns mxp ndarray matrix multiplication of matrix1(mxn) \n",
    "                                  and matrix2 (nxp). matrix1 and matrix2 may be ndarray \n",
    "                                  matrix or matrix-like list\n",
    "numpy.matmul(matrix1,matrix2)   - also returns mxp ndarray matrix multiplication of \n",
    "                                  matrix1(mxn) and matrix2 (nxp), but has has added \n",
    "                                  capabilities for higher dimensional arguments. matrix1 and\n",
    "                                  matrix2 be ndarray matrix or matrix-like list\n",
    "
\n", "\n", "\n", "### Exercise\n", "\n", "Perform addition, subtraction and multiplication operations on the given matrices. Store the result in 3 variables - addition, subtraction and multiplication and print them out." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true, "tags": [ "s1", "ce", "l1" ] }, "outputs": [], "source": [ "import numpy as np\n", "\n", "a = np.array([[1,2],[3,4]])\n", "b = np.array([[1,1],[1,1]])\n", "\n", "# addition = \n", "# subtraction = \n", "# multiplication = " ] }, { "cell_type": "markdown", "metadata": { "tags": [ "s1", "l1", "hint" ] }, "source": [ "### Hint\n", "\n", "Use matmul function for matrix multiplication" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "tags": [ "s1", "l1", "ans" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[2 3]\n", " [4 5]] \n", " [[0 1]\n", " [2 3]] \n", " [[3 3]\n", " [7 7]]\n" ] } ], "source": [ "addition = a+b\n", "subtraction = a-b\n", "multiplication = np.matmul(a,b)\n", "\n", "print(addition,\"\\n\",subtraction,\"\\n\",multiplication)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false, "tags": [ "s1", "hid", "l1" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " test1 = [[2,3],[4,5]]\n", " test2 = [[0,1],[2,3]]\n", " test3 = [[3,3],[7,7]]\n", "\n", " if np.array_equal(test1,addition) and np.array_equal(test2,subtraction) and np.array_equal(test3,multiplication):\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l2", "content", "s2" ] }, "source": [ "\n", "


\n", "### Transpose\n", "\n", "A transpose of a matrix is another matrix which results from transforming all the rows of elements of the original matrix into columns. If the order or shape of the matrix is $(i,j)$ then the transpose of this matrix will have a shape of $(j,i)$. ie. A matrix of size mxn after transpose becomes nxm sized matrix\n", "\n", "\n", "\n", "$\\left(\\left[\\begin{array}{cc}x_{11} & x_{12}\\\\x_{21} & x_{22}\\\\\\end{array}\\right]\\right)^T$ = $\\left[\\begin{array}{cc}x_{11} & x_{21}\\\\x_{12} & x_{22}\\\\\\end{array}\\right]$\n", "\n", "
\n", "
\n",
    "Syntax:\n",
    "numpy.transpose(matrix1) -  returns ndarray transpose of matrix1, where matrix1 is a \n",
    "                            matrix-like or ndarray square matrix\n",
    "
\n", "\n", "### Inverse of a Matrix\n", "\n", "In mathematics, we have the concept of 'reciprocal'. If a number is multiplied by its reciprocal the result is '1'. \n", "\n", "Note that only a square matrix has an Inverse. \n", "\n", "The Inverse of a matrix can be determined by using the 'inv' function of 'linalg' sub-module of numpy ($numpy.linalg.inv()$). This function can be performed on a numpy array (matrix).\n", "\n", "
\n",
    "Syntax:\n",
    "numpy.linalg.inv(matrix1)   - returns ndarray matrix that is inverse of matrix1; matrix1 is \n",
    "                              a square matrix\n",
    "
\n", "An Identity matrix is a square matrix with ones on its principal diagonal and zero otherwise. 'Identity matrix' is analogous to the '1' among numbers.\n", "\n", "Identity Matrix $(2x2)$ = $\\left[\\begin{array}{cc}1 & 0\\\\0 & 1\\\\\\end{array}\\right]$\n", "\n", "Identity Matrix $(3x3)$ = $\\left[\\begin{array}{cc}1 & 0 & 0\\\\0 & 1 & 0\\\\0 & 0 & 1\\\\\\end{array}\\right]$\n", "\n", "A matrix multiplied with its inverse, results in an Identity matrix.\n", "\n", "**A * A-1 = I**\n", "\n", "
\n",
    "Syntax:\n",
    "numpy.dot(matrix1, numpy.linalg.inv(matrix1))  -  returns an ndarray identity matrix where\n",
    "                                                  matrix1 is a square matrix\n",
    "
\n", "Generating Identity matrices in numpy: \n", "\n", "
\n",
    "Syntax:\n",
    "numpy.eye(n)  -   returns an ndarray identity matrix of size nxn\n",
    "
\n", "\n", "```python\n", "Sample code:\n", "numpy.eye(2) # generates a 2x2 ndarray identity matrix\n", "```\n", "allclose and array_equal functions: numpy.array_equal() method can be used to test whether two arrays are equal to each other in terms of shape and elements. numpy.allclose() function performs the same operation but it has tolerance while matching elements, which enables it to compare floating point elements with varying accuracies/decimals.\n", "The above two methods can be used to verify if a matrix is indeed an inverse of another matrix.\n", "\n", "
\n",
    "Syntax:\n",
    "numpy.array_equal(matrix1,matrix2)   - returns True if matrix1 and matrix2 have exactly \n",
    "                                       same values in the corresponding positions\n",
    "numpy.allclose(matrix1,matrix2)    -  returns True if matrix1 and matrix2 have nearly close \n",
    "                                      values in the corresponding positions\n",
    "numpy.allclose(numpy.matmul(matrix1,matrix2),numpy.eye(np.size(matrix1,0))   - returns True \n",
    "                        if matrix2 is inverse of matrix1 else False; matrix1 and matrix2 are\n",
    "                        square matrices of same size nxn; matrix1 is multiplied with matrix2\n",
    "                        and the product is compared to an identity matrix using allclose.\n",
    "
\n", "\n", "```python\n", "Sample code:\n", "matrix1=[1,2.00001,3.423] # creates a list-matrix \n", "matrix2=[1,2.000011,3.423] # creates another matrix-like list with close values\n", "np.allclose(matrix1,matrix2) # returns True \n", "np.array_equal(matrix1,matrix2) # returns False\n", "```\n", "\n", "### Exercise\n", "\n", "Find out the transpose and inverse of the given matrix. Store the results in variables called 'atranspose' and 'ainverse' respectively and print them out. Also verify " ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true, "tags": [ "l2", "ce", "s2" ] }, "outputs": [], "source": [ "a = np.array([[1,2],[3,4]])\n", "\n", "# atranspose = \n", "# ainverse = " ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l2", "s2", "hint" ] }, "source": [ "### Hint\n", "\n", "Use numpy.linalg.inv() method" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false, "tags": [ "l2", "s2", "ans" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[1 3]\n", " [2 4]] \n", " [[-2. 1. ]\n", " [ 1.5 -0.5]]\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "atranspose = np.transpose(a)\n", "ainverse = np.linalg.inv(a)\n", "print(atranspose,\"\\n\",ainverse)\n", "np.allclose(np.matmul(a,ainverse),np.eye(2))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false, "tags": [ "l2", "hid", "s2" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " test1 = np.array([[-2.,1.],[1.5,-0.5]])\n", " test2 = np.array([[1,3],[2,4]])\n", "\n", " if np.allclose(test1,ainverse) and np.array_equal(atranspose,test2):\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l3", "s3", "content" ] }, "source": [ "


\n", "## Vector Operations - Dot Product and Cross Product\n", "\n", "Assume two vectors, 'a' and 'b' having elements **(x1,x2,x3)** and **(y1,y2,y3)** respectively.\n", "\n", "Dot product of a and b is:
\n", "**a.b = x1y1 + x2y2 + x3y3**
\n", "\n", "Dot product of one-dimensional arrays or vectors can be performed using numpy.dot(). If two vectors, ie 1D arrays, are passed to numpy.dot(), then it returns the inner product (dot product) of the two vectors. \n", "
\n",
    "Syntax: \n",
    "numpy.dot(a,b) - returns a scalar value, when a and b are vectors of same sizes; \n",
    "
\n", "\n", "```python\n", "Sample code:\n", "a=[1,2,3] #creates row vector\n", "b=[2,3,4] \n", "np.dot(a,b) # returns 20\n", "```\n", "\n", "Note: We saw earlier that if a and b are matrices of sizes mxn and nxp then numpy.dot(a,b) returns an ndarray matrix of size mxp.\n", "\n", "Cross product of a and b is:
\n", "**axb = $\\left|\\begin{array}{cc}x_{2} & x_{3}\\\\y_{2} & y_{3}\\\\\\end{array}\\right|i - \\left|\\begin{array}{cc}x_{1} & x_{3}\\\\y_{1} & y_{3}\\\\\\end{array}\\right|j + \\left|\\begin{array}{cc}x_{1} & x_{2}\\\\y_{1} & y_{2}\\\\\\end{array}\\right|k $**

\n", "\n", "
\n",
    "Syntax: \n",
    "numpy.cross(a,b) - returns an ndarray vector of same size, where a and b are vector arrays of \n",
    "                   same size\n",
    "
\n", "Note that dot product of two vectors returns a scalar and cross product of two vectors returns another vector.\n", "\n", "### Exercise\n", "\n", "Find the dot product and cross product for the given vectors. Store the result in variables prod_dot and prod_cross, and print them out." ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true, "tags": [ "l3", "s3", "ce" ] }, "outputs": [], "source": [ "vector_one = np.array([1,2,3])\n", "vector_two = np.array([1,1,1])\n", "\n", "# prod_dot =\n", "# prod_cross =" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l3", "s3", "hint" ] }, "source": [ "### Hint" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false, "tags": [ "l3", "s3", "ans" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "6 \n", " [-1 2 -1]\n" ] } ], "source": [ "prod_dot = np.dot(vector_one, vector_two)\n", "prod_cross = np.cross(vector_one, vector_two)\n", "\n", "print(prod_dot,\"\\n\",prod_cross)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false, "tags": [ "l3", "s3", "hid" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " var = 6\n", " test = np.array([-1,2,-1])\n", "\n", " if prod_dot == var and np.allclose(prod_cross,test):\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l4", "s4", "content" ] }, "source": [ "\n", "


\n", "## Norm of a vector and types of vectors\n", "\n", "The norm of a vector is a numeric value which represents the lenght or size of the vector. It is defined as\n", "\n", "**Lp = ||x||p = ($\\sum_{i}$|xi|p)1/p**\n", "\n", "1. L1 Norm: It is also known as least absolute error
\n", "If x = [1,-2,3], then
\n", "**L1 of x = |x1| + |x2| + |x3| = |1| + |-2| + |3| = 6**
\n", "
\n",
    "Syntax: \n",
    "numpy.linalg.norm(x,ord=1) - returns the first norm, which is the sum of absolute \n",
    "                                values of elements of vector x \n",
    "
\n", "2. L2 Norm: It is also known as least squares
\n", "For x, as given above
\n", "**L2 of x = $\\sqrt{|x_{1}|^{2} + |x_{2}|^{2} + |x_{3}|^{2}}$ = $\\sqrt{|1|^{2} + |-2|^{2} + |3|^{2}}$ = $\\sqrt{14}$ = 3.74165**
\n", "
\n",
    "Syntax: \n",
    "numpy.linalg.norm(x) - returns the second norm by default, which is squareroot of\n",
    "                          sum of squares of the elements of the vector x \n",
    "
\n", "3. L$\\infty$ Norm: It is called as max norm
\n", "For x, given above
\n", "**L$\\infty$ of x = max{|xi|: i = 1,2,3....} = max{|1|,|-2|,|3|} = 3**\n", "
\n",
    "Syntax to calculate max norm: \n",
    "max(abs(x))  - returns the maximum of absolute values of the elements of the ndarray\n",
    "                  vector x\n",
    "
\n", "\n", "Norm of a vector can be calculated using the method numpy.linalg.norm(). The function takes at least two arguments, i.e., the vector (numpy array) and the order of the norm. The default value for order of norm is 2. So when no 'order' argument is specified, the function calculates the second order norm by default.\n", "\n", "\n", "### Angle Between Vectors\n", "\n", "\n", "\n", "The L2 Norm can also be written as
\n", "**(aT)(b) = ||a||2 ||b||2 cos($\\theta$)**
\n", "where $\\theta$ is the angle between the two vectors a and b.
\n", "\n", "Calculating the angle between two vectors is not a straight forward process in Python The following steps can be followed in order to find out the angle between any two vectors.\n", "\n", "1. Normalize the given vectors: Normalizing is the process of converting the length of a vector to '1' while preserving the direction of the vector. This can be done by dividing the vector with its second order norm.

\n", "```python\n", "Sample code:\n", "x=numpy.array([1,2,3])\n", "y=numpy.array([4,1,2])\n", "xnorm2 = numpy.linalg.norm(x) # returns the second norm by default\n", "ynorm2 = numpy.linalg.norm(y)\n", "xn = x/xnorm2 # returns the normalized vector\n", "yn = y/ynorm2 \n", "```\n", "2. Dot product of Normalized vectors: Find out the dot product of the normalized vectors from above step using the numpy.dot function.\n", "```python\n", "Sample code:\n", "xydotnorm = numpy.dot(xn,yn) # returns the dot product of the normalized vectors\n", "```\n", "3. Clipping the value of the dot product: As we are trying to calculate the angle between vectors using the law of cosines, we should note that the cosine function has a maximum value of '1' and minimum value of '-1'. The dot product calculated in previous step cannot have a value beyond these bounds. Hence we use the 'numpy.clip()' function to limit the result of the dot product. If the value of the dot product of the normalized vectors falls within -1 and 1, it retains its value. If it is less than -1 it assumes a value of -1 and if greater than 1, then it assumes a value of 1.\n", "
\n",
    "Syntax: \n",
    "numpy.clip(x,min,max) -  clips the value of x to min/max if outside the range                                    where x is a scalar value  \n",
    "
\n", "```python\n", "Sample code:\n", "xydotnormclipped = numpy.clip(xydotnorm,-1,1) # returns x; -1<=x<=1\n", "```\n", "4. Calculating angle using **cos-1** function: We have a value between -1 and 1 which we need to use to calculate the possible angle using the inverse cosine function. This can be acheived using the 'numpy.arccos()' function. Note that this function returns the angle in radians. In order to convert this result into degrees, the 'numpy.degrees()' function can be used.\n", "
\n",
    "Syntax: \n",
    "numpy.arccos(x) - returns the cos-1(x) in radians; -1<=x<=1\n",
    "numpy.degrees(x) - returns the degrees equivalent of  x radians\n",
    "
\n", "```python\n", "Sample code:\n", "angle = numpy.arccos(xydotnormclipped # returns the value of the cos angle in radians\n", "degree = numpy.degrees(angle) # returns the value of the angle in degrees \n", "```\n", "\n", "### Special Vectors and Matrices\n", "\n", "If a and b are non-zero vectors and **(aT)(b) = 0**, then it implies that **cos($\\theta$) = 0**. So, the angle between a and b would be 90 degrees and in such a condition a and b are called orthogonal vectors with respect to each other.\n", "\n", "* Symmetric Matrix: If **a = aT** then 'a' is a symmetric matrix\n", "* Orthogonal Matrix: If **a-1 = aT** then 'a' is called an orthogonal matrix.\n", "\n", "\n", "### Exercise\n", "\n", "* Find the second degree norms of given vectors 'v_one' and 'v_two'. Assign the norms to two variables 'v_one_norm' and 'v_two_norm' and print them out.\n", "* Normalize 'v_one' and 'v_two' [divide them with their respective norms]. Store the values in two variables called 'v_one_normvec' and 'v_two_normvec' and print them.\n", "* Calculate the angle between 'v_one' and 'v_two' [Use numpy.arccos() and numpy.degrees()]. Assign the angle to a variable 'v_angle' and print it out.\n", "* Are 'v_one' and 'v_two' orthogonal vectors with respect to each other? If yes, store 'True' in the variable 'v_ortho' else store 'False'." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": true, "tags": [ "l4", "s4", "ce" ] }, "outputs": [], "source": [ "v_one = np.array([1,2,1])\n", "v_two = np.array([3,4,5])\n", "\n", "# Modify this code\n", "\n", "# v_one_norm = \n", "# v_two_norm =\n", "# v_one_normvec =\n", "# v_two_normvec =\n", "# v_angle = \n", "# v_ortho =" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "s4", "l4", "hint" ] }, "source": [ "### Hint\n" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false, "tags": [ "s4", "l4", "ans" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.44948974278 7.07106781187\n", "[ 0.40824829 0.81649658 0.40824829] [ 0.42426407 0.56568542 0.70710678]\n", "22.5178253582\n", "False\n" ] } ], "source": [ "v_one_norm = np.linalg.norm(v_one,2)\n", "v_two_norm = np.linalg.norm(v_two,2)\n", "print(v_one_norm,v_two_norm)\n", "\n", "v_one_normvec = v_one/v_one_norm\n", "v_two_normvec = v_two/v_two_norm\n", "print(v_one_normvec,v_two_normvec)\n", "\n", "v_angle = np.degrees(np.arccos(np.clip(np.dot(v_one_normvec,v_two_normvec),-1.0,1.0)))\n", "print(v_angle)\n", "\n", "if round(v_angle,2) == 90.00:\n", " v_ortho = True\n", "else:\n", " v_ortho = False\n", " \n", "print(v_ortho)" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false, "tags": [ "s4", "hid", "l4" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " test1 = round(2.44948974278,2)\n", " test2 = round(7.07106781187,2)\n", " test3 = [round(x,2) for x in [0.40824829,0.81649658,0.40824829]]\n", " test4 = [round(y,2) for y in [0.42426407,0.56568542,0.70710678]]\n", " test5 = round(22.5178253582,2)\n", " test6 = False\n", "\n", " if test1 == round(v_one_norm,2) and test2 == round(v_two_norm,2) and test3 == [round(a,2) for a in v_one_normvec] and test4 == [round(b,2) for b in v_two_normvec] and test5 == round(v_angle,2) and test6 == v_ortho:\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l5", "content", "s5" ] }, "source": [ "\n", "\n", "


\n", "## Matrix Decomposition \n", "\n", "Decomposition means breaking of a large entity into smaller ones. Similarly, matrix decompositon or matrix factorization means decomposing or factorizing a large matrix into smaller ones. The usefullness of decompostion is that it helps us in learning about some special properties of the matrix which we couldn't have done if we hadn't decomposed it. Here we will talk about decomposition of a matrix into **Eigen Vectors and Eigen Values**\n", "\n", "\n", "## Eigen Values and Eigen Vectors\n", "\n", "In practical applications, many problems are presented in the form of Eigenvalue problem:\n", "

\n", "**A·v = λ·v**\n", " \n", "In this equation A is an n-by-n matrix, v is a non-zero n-by-1 vector and λ is a scalar (which may be either real or complex). Any value of λ for which this equation has a solution is known as an eigenvalue of the matrix A. It is sometimes also called the characteristic value. The vector, v, which corresponds to this value is called an eigenvector. The eigenvalue problem can be rewritten as:\n", "

\n", "**(A·v) - (λ·v) = 0**\n", "
\n", "**(A·v) - (λ·I·v) = 0**\n", "
\n", "**(A - (λ·I))·v = 0**\n", "
\n", "If v is non-zero, this equation will only have a solution if \n", "\n", "**|A - (λ·I)| = 0**\n", " \n", "This equation is called the characteristic equation of A, and is an nth order polynomial in λ with n roots.\n", "\n", "**|A - (λ·I)| = (λ1-λ)(λ2-λ)…(λn-λ)**\n", "

\n", "These roots are called the eigenvalues of A. For each eigenvalue there will be an eigenvector for which the eigenvalue equation is true. In python we can find eigen values and eigen vectors using \"linalg\" package from scipy. \n", "\n", "### Exercise: \n", "\n", "Find eigen values and eigen vector of an array A = [1,2,3,4]. Store the eigen values in a variable called B." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true, "tags": [ "l5", "ce", "s5" ] }, "outputs": [], "source": [ "from scipy import linalg as LA\n", "import numpy as np\n", "\n", "A = np.array([2,3,4,6]).reshape(2,2)" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l5", "s5", "hint" ] }, "source": [ "### Hint" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "tags": [ "l5", "s5", "ans" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Eigen Values: [ 0.+0.j 8.+0.j] \n", "Eigen Vectors: [[-0.83205029 -0.4472136 ]\n", " [ 0.5547002 -0.89442719]]\n" ] } ], "source": [ "B, C= LA.eig(A)\n", "\n", "print(\"Eigen Values: \",B,\"\\nEigen Vectors: \",C)" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "tags": [ "hid", "l5", "s5" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " import numpy as np\n", "\n", " if np.all(B == [ 0.+0.j , 8.+0.j]) :\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "content", "l6", "s6" ] }, "source": [ "\n", "


\n", "## Eigen Decomposition\n", "\n", "As seen in the topics above, Eigen Decomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors.\n", "\n", "Lets see an example:\n", "\n", "Let's see the eigendecomposition for the matrix:\n", "\n", "A=$\\left[ \\begin{array}{cccc}\n", "1 & 1 \\\\\n", "0 & 3 \\\\ \\end{array} \\right]$\n", "\n", "From above we can find determinant of the matrix as:\n", "\n", "det$\\left( \\begin{array}{cccc}\n", "1-\\lambda & 0 \\\\\n", "1 & 3-\\lambda \\\\ \\end{array} \\right)$\n", "\n", "**(1−λ)(3−λ)=0**\n", "\n", "\n", "we get directly **λ1 =1 and λ2=3**. The above expression is usually referred as the **characteristic polynomial or characteristic equation of a matrix**. \n", "\n", "Putting λ1 in the equation, we get:\n", "\n", "$\\left[\\begin{array}{cc}1 & 0\\\\1 & 3\\\\\\end{array}\\right]\\left[\\begin{array}{c}v_{11}\\\\v_{12}\\\\\\end{array}\\right]$ = 1 $\\left[\\begin{array}{c}v_{11}\\\\v_{12}\\\\\\end{array}\\right]$\n", "\n", "\n", "from which we get \n", "\n", "**v11=−2v12**. \n", "\n", "That is, any vector **v1=[v11,v12] **where **v11=−2v12** is an eigenvector of A with eigenvalue 1. Putting λ2 into equation, we get:\n", "\n", "$\\left[\\begin{array}{cc}1 & 0\\\\1 & 3\\\\\\end{array}\\right]\\left[\\begin{array}{c}v_{21}\\\\v_{22}\\\\\\end{array}\\right]= 3 \\left[\\begin{array}{c}v_{21}\\\\v_{22}\\\\\\end{array}\\right]$\n", "\n", "from which we get v_21 = 0 and v_22 ∈R . That is, any vector v_2 = [v_21, v_22] where v_21 = 0 is an eigenvector of A with eigenvalue 3.\n", "\n", "### Use of Eigen Decompositon\n", "\n", "From our previous example, we can use eigen values and eigen vectors, join them in a single matrix equation.\n", "\n", " A$\\left[\\mathbf{v_1 v_2}\\right]$ = $\\left[\\begin{array}{cc}1 & 0\\\\1 & 3\\\\\\end{array}\\right]$\n", " $\\left[\\begin{array}{cc}v_{11} & v_{21}\\\\\n", " v_{12} & v_{22}\\\\\\end{array}\\right]$ = \n", " $\\left[\\begin{array}{cc}v_{11} & v_{21}\\\\v_{12} & v_{22}\\\\\\end{array}\\right]$\n", " $\\left[\\begin{array}{cc}\\lambda_1 & 0\\\\0 & \\lambda_2\\\\\\end{array}\\right]$ =\n", " $\\left[\\mathbf{v_1 v_2}\\right]$\n", " $\\left[\\begin{array}{cc}\\lambda_1 & 0\\\\0 & \\lambda_2\\\\\\end{array}\\right]$\n", " \n", " If we replace the value,\n", " \n", " λ=$\\left[\\begin{array}{cc}\\lambda_1 & 0\\\\0 & \\lambda_2\\\\\\end{array}\\right]$\n", " \n", " V = $\\left[\\mathbf{v_1 v_2}\\right]$\n", " \n", " it becomes,\n", "\n", " **AV=VΛ**\n", " \n", " **A=VΛV−1**\n", " \n", "Eigendecomposition decomposes a matrix A into a multiplication of a matrix of **eigenvectors V** and a **diagonal matrix of eigenvalues Λ** as **A=VΛV−1**\n", "\n", "### Benefits of Eigen Decomposition \n", "\n", "If all eigenvalues are positive, then the matrix is positive definite. If all eigenvalues are positive or zero-valued, then the matrix is positive semi-definite. Similar is the case negative definite and negative semi-definite.\n", "There are benefits to knowing that a matrix is positive definite, positive semi-definite, negative definite and negative semi-definite. Many useful facts can be gather using this -\n", "\n", "* A matrix is singular if and only if any of the eigenvalues is zero\n", "* It can be used to optimize quadratic expressions of the form **f(x)=xTAx subject to ||x||2=1** \n", "* The determinant of a matrix is equal to the sum of all eigenvalues of a matrix\n", "\n", "\n", "## Trace and Determinant \n", "\n", "### Trace\n", "\n", "In linear algebra, the trace of an n-by-n square matrix A is defined to be the sum of the elements on the main diagonal (the diagonal from the upper left to the lower right) of A. **INSERT PICTURE**\n", "\n", "**The sum of Eigen Values corresponds to the trace of the matrix.**\n", "\n", "### Determinant \n", "\n", "In Linear Algebra, determinant of a matrix is a special number that can be calculated from a square matrix. \n", "\n", "\n", "For a 2×2 matrix:\n", "\n", " A=$\\left[\\begin{array}{cc}a & b \\\\c & d\\\\\\end{array}\\right]$\n", " \n", " The determinant is found by,\n", " \n", " **|A| = (a \\* d) − (b \\* c)**\n", " \n", "For a 3x3 matrix:\n", "\n", " A=$\\left[\\begin{array}{cc}a & b & c \\\\d & e &f\\\\ g &h&i\\\\\\end{array}\\right]$\n", " The determinant is found by,\n", " **|A| = a((e \\* i) − (f \\* h)) − b((d \\* i) − (f \\* g)) + c((d \\* h) − (e \\* g))**\n", " \n", "**Determinant of a matrix can also be found by product of its eigenvalues.**\n", "\n", "## Exercise \n", "\n", "Find determinent of a matrix using \"numpy.linalg.det\" in python and store its value in variable called b." ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": true, "tags": [ "ce", "l6", "s6" ] }, "outputs": [], "source": [ "a = np.array([[5, 4], [1, 2]])" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "l6", "s6", "hint" ] }, "source": [ "### Hint" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "tags": [ "l6", "s6", "ans" ] }, "outputs": [ { "data": { "text/plain": [ "6.0" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "b=np.linalg.det(a)\n", "b" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "tags": [ "hid", "s6", "l6" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "ref_tmp_var = False\n", "\n", "try:\n", " import numpy as np\n", "\n", " if np.all(b == 6.0) :\n", " ref_assert_var = True\n", " ref_tmp_var = True\n", " else:\n", " ref_assert_var = False\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "except Exception:\n", " print('Please follow the instructions given and use the same variables provided in the instructions. ')\n", "\n", "assert ref_tmp_var" ] } ], "metadata": { "anaconda-cloud": {}, "executed_sections": [], "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 2 }