{ "cells": [ { "cell_type": "markdown", "metadata": { "code_folding": [ 1 ], "slideshow": { "slide_type": "slide" } }, "source": [ "# Lecture 3.2: Scalar product, matrix norms, unitary matrices" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Matrices and norms\n", "How to measure distances between matrices? \n", "\n", "A trivial answer is that there is no big differences between matrices and vectors, and here comes the **Frobenius** norm of the matrix:\n", "$$\n", " \\Vert A \\Vert_F = \\Big(\\sum_{i=1}^n \\sum_{j=1}^m |a_{ij}|^2\\Big)^{1/2}\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Matrix norms\n", "$\\Vert \\cdot \\Vert$ is called a **matrix norm** if it is a vector norm on the linear space of $n \\times m$ matrices, and it also is consistent with the matrix-by-matrix product, i.e.\n", "\n", "$$\\Vert A B \\Vert \\leq \\Vert A \\Vert \\Vert B \\Vert$$\n", "\n", "The multiplicative property is needed in many places, for example in the estimates for the error of solution of linear systems (we will cover this subject later). \n", "\n", "Can you think of some matrix norms, and is Frobenius norm a matrix norm?" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Operator norms\n", "The most important class of the norms is the class of **operator norms**. Mathematically, they are defined as\n", "\n", "$$\n", " \\Vert A \\Vert_* = \\sup_{x \\ne 0} \\frac{\\Vert A x \\Vert_*}{\\Vert x \\Vert_*},\n", "$$\n", "\n", "where $\\Vert \\cdot \\Vert_*$ is a **vector norm**." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Matrix p-norms\n", "It is not diffcult to show that operator norm is a matrix norm. Among all operator norms $p$-norms are used, where $p$-norm is used as the vector norm. Among all $p$-norms three norms are the most common ones: \n", "\n", "- $p = 2, \\quad$ spectral norm, denoted by $\\Vert A \\Vert_2$.\n", "- $p = \\infty, \\quad \\Vert A \\Vert_{\\infty} = \\max_i \\sum_j |A_{ij}|$.\n", "- $p = 1, \\quad \\Vert A \\Vert_{1} = \\max_j \\sum_i |A_{ij}|$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Spectral norm\n", "Spectral norm, $\\Vert A \\Vert_2$ is undoubtedly the most used matrix norm. It can not be computed directly from the entries using a simple formula, like the Euclidean norm, however, there are efficient algorithm to compute it. It is directly related to the **singular value decomposition** (SVD) of the matrix. It holds\n", "\n", "$$\n", " \\Vert A \\Vert_2 = \\sigma_1(A)\n", "$$\n", "\n", "where $\\sigma_1(A)$ is the largest singular value of the matrix $A$. We will soon learn all about this. Meanwhile, we can already compute the norm in Python." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Spectral: 19.4195674332 Frobenius: 99.5002249595 1-norm 94.7398844201 infinity 96.8093380047\n" ] } ], "source": [ "import numpy as np\n", "n = 100\n", "a = np.random.randn(n, n) #Random n x n matrix\n", "s1 = np.linalg.norm(a, 2) #Spectral\n", "s2 = np.linalg.norm(a, 'fro') #Frobenius\n", "s3 = np.linalg.norm(a, 1) #1-norm\n", "s4 = np.linalg.norm(a, np.inf) #It was trick to find the infinity\n", "print 'Spectral:', s1, 'Frobenius:', s2, '1-norm', s3, 'infinity', s4" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Scalar product\n", "If norm is a measure of distances, then the scalar product takes angle into account. \n", "\n", "The scalar product is defined as\n", "$$\n", " (x, y) = \\sum_{i=1}^n \\overline{x}_i y_i,\n", "$$\n", "where $\\overline{x}$ denotes the *complex conjugate* of $x$. The Euclidean norm is then\n", "\n", "$$\n", " \\Vert x \\Vert^2 = (x, x),\n", "$$\n", "\n", "or it is said the the norm is **induced** by scalar product. " ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "**Remark**. For the angle between two vectors is defined as\n", "$$\n", " \\cos \\phi = \\frac{(x, y)}{\\Vert x \\Vert \\Vert y \\Vert} \n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "An important property of the scalar product is the **Cauchy-Bunyakovski inequality**:\n", "$$\n", " |(x, y)| \\leq \\Vert x \\Vert \\Vert y \\Vert,\n", "$$\n", "and thus the angle between two vectors is defined properly." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "The scalar product can be written as a matrix-by-matrix product \n", "$$\n", " (x, y) = x^* y,\n", "$$\n", "where $^*$ is a **conjugate transpose** of the matrix: \n", "$$\n", "B = A^*, \\quad B_{ij} = \\overline{A_{ji}}.\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Norm conservation\n", "For stability it is really important that the error does not grow. Suppose that you approximately get a vector, \n", "\n", "$$\n", " \\Vert x - \\widehat{x} \\Vert \\leq \\varepsilon.\n", "$$\n", "Let final result is (some) linear transformation of $x$: \n", "$$\n", " y = Ux, \\quad \\widehat{y} = U \\widehat{x}.\n", "$$\n", "If we want to estimate a difference between $\\widehat{y}$ and $y$: \n", "\n", "$$\n", " \\Vert y - \\widehat{y} \\Vert = \\Vert U ( x - \\widehat{x}) \\Vert \\leq \\Vert U \\Vert \\varepsilon.\n", "$$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Matrices, preserving the norm\n", "The question is for which kind of matrices the norm of the vector **will not change**. \n", "\n", "For the euclidean norm this produces a very important class of matrices: **unitary** (or orthogonal in the real case) matrices." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Unitary (orthogonal) matrices\n", "Let $U$ be an $n \\times r$ matrix, and $\\Vert U z \\Vert = \\Vert z \\Vert$ for all $z$. This can happen if and only if \n", "\n", "$$\n", " U^* U = I,\n", "$$\n", "\n", "where $I$ is an **identity matrix**\n", "\n", "Indeed, $$\\Vert Uz \\Vert^2 = (Uz, Uz) = (Uz)^* Uz = z^* (U^* U) z = z^* z,$$ \n", "\n", "which can also hold if $U^* U = I$." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Unitary matrices\n", "In the real case, when $U^* = U^{\\top}$, the matrix is called orthogonal. \n", "\n", "Are there many unitary matrices? First of all, **a product of two unitary matrices is a unitary matrix:** \n", "\n", "$$(UV)^* UV = V^* (U^* U) V = V^* V = I,$$\n", "\n", "thus if we give some non-trivial examples of unitary matrices, we will be able to get any unitary transformation." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Examples of unitary matrices\n", "There are two important classes of unitary matrices, using those we can make any unitary matrix\n", "1. Householder matrices\n", "2. Givens (Jacobi) matrices" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Householder matrices\n", "Householder matrix is the matrix of the form \n", "$$H = I - 2 vv^*,$$\n", "where $u$ is an $n \\times 1$ matrix and $v^* v = 1$. Can you show that $H$ is unitary? It is also a reflection. \n", "A simple proof: $H^* H = (I - 2 vv^*)(I - 2 v v^*) = I$" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Givens (Jacobi) matrix\n", "A Givens matrix is a matrix \n", "\n", "$$\n", " A = \\begin{bmatrix}\n", " \\cos \\alpha & \\sin \\alpha \\\\\n", " -\\sin \\alpha & \\cos \\alpha\n", " \\end{bmatrix},\n", "$$\n", "\n", "which is a rotation. For a general case, we select two $(i, j)$ planes and rotate only in those: \n", "\n", "$$\n", " x'_i = \\cos \\alpha x_i + \\sin \\alpha x_j, \\quad x'_j = -\\sin \\alpha x_i + \\cos\\alpha x_j,\n", "$$\n", "\n", "with all other $x_i$ remain unchanged." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Summary on unitary matrices\n", "- Unitary matrices preserve the norm\n", "- There are two \"basic\" classes of unitary matrices, Householder and Givens.\n", "- Every unitary matrix can be represented as a product of those." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Take home message\n", "- Matrix multiplication, idea of blocking, memory hiearchy\n", "- Scalar product, unitary matrices, basic classes of unitary matrices" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Next week\n", "- TA week\n", "- You got PSet 1\n", "- Think of course projects (i.e. oil&gas, power networks, social networks)\n", "- There is a course on Mathematics of the Internet going now at Skoltech, you are welcome to visit." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "##### Questions?" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "slideshow": { "slide_type": "skip" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n" ], "text/plain": [ "" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from IPython.core.display import HTML\n", "def css_styling():\n", " styles = open(\"./styles/custom.css\", \"r\").read()\n", " return HTML(styles)\n", "css_styling()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }