{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Notes on The Matrix Calculus You Need for Deep Learning\n", "\n", "These are my notes from the article [\"The Matrix Calculus you need for Deep Learning\"](https://explained.ai/matrix-calculus/index.html) by Terence Parr and Jeremy Howard \n", "as part of a fastai study group being conducted May-Oct 2020 (ongoing).\n", "\n", "_(Status: unfinished)_\n", "\n", "#### Affine function for a Neuron\n", "\n", "$$ \n", "z(x) = \\sum \\limits _{i} ^{n} w_{i}x_{i} + b = w \\cdot x + b \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Common loss function\n", "$$\n", "\\frac {1}{N} \\sum \\limits _{x} (target(x) - activation(x))^{2} = \n", "\\frac {1}{N} \\sum \\limits _{x} (target(x) - max(0, \\sum \\limits _{i}^{\\vert x \\vert} w_{i}x_{i} + b))^2 \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Table of Derivatives\n", "\n", "![basic derivative chain rules](https://raw.githubusercontent.com/butchland/fastai_nb_explorations/master/images/basic-derivative-chain-rules.png)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Partial Derivative Example\n", "\n", "Consider the function $ f(x,y) = 3x^{2}y $\n", "\n", "The derivative of _f_ wrt to _x_ is:\n", "\n", "$\\frac{\\partial f}{\\partial x} = $\n", "$\\frac{\\partial} {\\partial x} 3x^{2}y = $\n", "$3y\\frac{\\partial}{\\partial x}x^{2} = $\n", "$ 6yx $\n", "\n", "The derivative of _f_ wrt to _y_ is:\n", "\n", "$\\frac{\\partial f}{\\partial y} = $\n", "$\\frac{\\partial} {\\partial y} 3x^{2}y = $\n", "$3x^{2}\\frac{\\partial}{\\partial y}y = $\n", "$ 3x^{2} $\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Gradient of $ f(x,y) $\n", "\n", "$ \\nabla f(x,y) = [\\frac{\\partial f(x,y)}{\\partial x},\\frac{\\partial f(x,y)}{\\partial y}] = $\n", "$ [6yx, 3x^{2}] $\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Matrix calculus\n", "\n", "Given the functions $ f(x,y) = 3x^{2}y $ and \n", "$ g(x,y) = 2x + y^{8} $\n", "\n", "\n", "$ \\frac{\\partial g(x,y)}{\\partial x} = $\n", "$ \\frac{\\partial 2x}{\\partial x} + \\frac{\\partial y^{8}}{\\partial x} = $\n", "$ 2\\frac{\\partial x}{\\partial x} + 0 = $\n", "$ 2 \\times 1 = 2 $\n", "\n", "and\n", "\n", "$ \\frac{\\partial g(x,y)}{\\partial y} = $\n", "$ \\frac{\\partial 2x}{\\partial y} + \\frac{\\partial y^{8}}{\\partial y} = $\n", "$ 0 + 8y^{7} = $\n", "$ 8y^{7} $\n", "\n", "giving us the gradient of $ g(x,y) $ as\n", "\n", "$ \\nabla g(x,y) = $\n", "$ [2,8y^{7}] $" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Jacobian matrix (aka Jacobian)\n", "\n", "**_numerator layout_(used here)** :\n", "* rows: _equations(f,g)_ , \n", "* columns: _variables(x,y)_ \n", "\n", "\n", "$$\n", "J = \n", "\\begin{bmatrix}\n", "\\nabla f(x,y) \\\\\n", "\\nabla g(x,y)\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "6yx & 3x^{2} \\\\\n", "2 & 8y^{7}\n", "\\end{bmatrix} \n", "$$\n", "\n", "_denominator layout_ - rows: _variables(x,y)_ , columns: _equations(f,g)_\n", "\n", "$$\n", "\\begin{bmatrix}\n", "6yx & 2 \\\\\n", "3x^{2} & 8y^{7}\n", "\\end{bmatrix} \n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generalization of the Jacobian\n", "\n", "Combine $ f(x,y,z) \\Rightarrow f(x) $ where **`x`** is a vector (aka $ \\vec{x} $) and _x_ are scalars. e.g. $ x_{i} $ is the $i^{th}$ element of vector **x**. \n", "\n", "Assume vector **x** is a column vector (vertical vector) by default of size $ n \\times 1 $. \n", "\n", "$$\n", "x = \n", "\\begin{bmatrix}\n", "x_{1} \\\\\n", "x_{2} \\\\\n", "\\vdots \\\\\n", "x_{n}\n", "\\end{bmatrix} \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For multiple scalar-valued functions, combine all into a vector just like the parameters. \n", "\n", "Let $ y = f(x) $ be a vector of `m` scalar-valued functions that each take a vector **x** of length $ n = \\vert x \\vert $ where $ \\vert x \\vert $ is the count of elements in **x**.\n", "\n", "Each $ f_{i} $ function within **f** returns a scalar.\n", "\n", "$$\n", "\\begin{matrix}\n", "y_{1} = f_{1}(x) \\\\\n", "y_{2} = f_{2}(x) \\\\\n", "\\vdots \\\\\n", "y_{m} = f_{m}(x)\n", "\\end{matrix} \n", "$$\n", "\n", "For instance, given $ f(x,y) = 3x^{2}y $ and $ g(x,y) = 2x + y^{8} $,\n", "then \n", "\n", "$ y_{1} = f_{1}(x) = 3x_{1}^{2}x_{2} $ (substituting $ x_{1} $ for _x_, $ x_{2} $ for _y_ )\n", "\n", "$ y_{2} = f_{2}(x) = 2x_{1} + x_{2}^{8} $\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the identity function $ y = f(x) = x $ it will be the case that $ m = n $ :\n", "$$\n", "\\begin{matrix}\n", "y_{1} = f_{1}(x) = x_{1} \\\\\n", "y_{2} = f_{2}(x) = x_{2} \\\\\n", "\\vdots \\\\\n", "y_{m} = f_{m}(x) = x_{n}\n", "\\end{matrix} \n", "$$\n", "\n", "So for the identity function, we will $ m = n $ functions and parameters.\n", "\n", "Generally, the Jacobian matrix is the collection of all $ m \\times n $ possible partial derivatives (_m_ rows and _n_ columns), which is a stack of _m_ gradients with respect to **x**:\n", "\n", "$$\n", "\\frac {\\partial y}{\\partial x} \n", "=\n", "\\begin{bmatrix}\n", "\\nabla f_{1}(x) \\\\\n", "\\nabla f_{2}(x) \\\\\n", "\\dotsb \\\\\n", "\\nabla f_{m}(x)\n", "\\end{bmatrix} \n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x}f_{1}(x) \\\\\n", "\\frac{\\partial}{\\partial x}f_{2}(x) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x}f_{m}(x) \\\\\n", "\\end{bmatrix} \n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_{1}}f_{1}(x) & \\frac{\\partial}{\\partial x_{2}}f_{1}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{1}(x) \\\\\n", "\\frac{\\partial}{\\partial x_{1}}f_{2}(x) & \\frac{\\partial}{\\partial x_{2}}f_{2}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{2}(x) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x_{1}}f_{m}(x) & \\frac{\\partial}{\\partial x_{2}}f_{m}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{m}(x) \\\\\n", "\\end{bmatrix} \n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each $ \\frac{\\partial}{\\partial x}f_{i}(x) $ is a horizontal _n_-vector b/c the partial derivative wrt to the vector **x**, whose length $ n = \\vert x \\vert $. The _width_ of the Jacobian is _n_ if we take the partial derivative with respect to **x** because there are _n_ parameters we can wiggle, each potentially changing the function's value. Therefore, the Jacobian is always _m_ rows for _m_ equations.\n", "\n", "##### Jacobian Shapes \n", "![jacobian shapes](https://raw.githubusercontent.com/butchland/fastai_nb_explorations/master/images/jacobian_shapes.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### An Example: Jacobian of the identity function\n", "Given the identity function $ \\pmb{f}(x) = \\pmb{x} $, with $ f_{i}(x) = x_{i} $, the Jacobian of the identity function has _n_ functions and each function has _n_ parameters held in a single vector **x**. The Jacobian is, therefore, a square matrix since $ m = n $:\n", "\n", "$$\n", "\\frac{\\partial y}{\\partial x} \n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x}f_{1}(x) \\\\\n", "\\frac{\\partial}{\\partial x}f_{2}(x) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x}f_{m}(x) \\\\\n", "\\end{bmatrix} \n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_{1}}f_{1}(x) & \\frac{\\partial}{\\partial x_{2}}f_{1}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{1}(x) \\\\\n", "\\frac{\\partial}{\\partial x_{1}}f_{2}(x) & \\frac{\\partial}{\\partial x_{2}}f_{2}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{2}(x) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x_{1}}f_{m}(x) & \\frac{\\partial}{\\partial x_{2}}f_{m}(x) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}f_{m}(x) \\\\\n", "\\end{bmatrix} \n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_{1}}x_{1} & \\frac{\\partial}{\\partial x_{2}}x_{1} & \\dotsb & \\frac{\\partial}{\\partial x_{n}}x_{1} \\\\\n", "\\frac{\\partial}{\\partial x_{1}}x_{2} & \\frac{\\partial}{\\partial x_{2}}x_{2} & \\dotsb & \\frac{\\partial}{\\partial x_{n}}x_{2} \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x_{1}}x_{n} & \\frac{\\partial}{\\partial x_{2}}x_{n} & \\dotsb & \\frac{\\partial}{\\partial x_{n}}x_{n} \\\\\n", "\\end{bmatrix}\n", "$$\n", "\n", "And since $ \\frac{\\partial}{\\partial x_{j}}x_{i} = 0 $ for $ j \\ne i $ and $ \\frac{\\partial}{\\partial x_{j}}x_{i} = 1 $ for $ j = i $\n", "\n", "$$\n", "=\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_{1}}x_{1} & 0 & \\dotsb & 0 \\\\\n", "0 & \\frac{\\partial}{\\partial x_{2}}x_{2} & \\dotsb & 0 \\\\\n", " & & \\ddots \\\\\n", "0 & 0 & \\dotsb & \\frac{\\partial}{\\partial x_{n}}x_{n} \\\\\n", "\\end{bmatrix} \n", "=\n", "\\begin{bmatrix}\n", "1 & 0 & \\dotsb & 0 \\\\\n", "0 & 1 & \\dotsb & 0 \\\\\n", " & & \\ddots \\\\\n", "0 & 0 & \\dotsb & 1 \\\\\n", "\\end{bmatrix}\n", "= I \n", "$$\n", "\n", "(_I_ is the identity matrix with the ones down the diagonal)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Derivatives of vector element-wise binary operators\n", "\n", "_Element-wise binary operations on vectors_ - applying an operation to the first element of each vector to get the first element of the output, then apply to the second items of each vector to get the second item of the output, and so forth.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Generalized notation for element-wise binary operations\n", "$$\n", "\\pmb{y} = \\pmb{f}(w) \\bigcirc \\pmb{g}(x) \n", "$$\n", "\n", "where $ m = n = \\vert y \\vert = \\vert w \\vert = \\vert x \\vert $\n", "\n", "Reminder: $ \\vert x \\vert $ is the number of items in _x_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Zooming in $ \\pmb{y} = \\pmb{f}(\\pmb{w}) \\bigcirc \\pmb{g}(\\pmb{x}) $\n", "gives :\n", "\n", "$$\n", "\\begin{bmatrix}\n", "y_{1} \\\\\n", "y_{2} \\\\\n", "\\vdots \\\\\n", "y_{n}\n", "\\end{bmatrix}\n", "=\n", "\\begin{bmatrix}\n", "f_{1}(w) \\bigcirc g_{1}(x) \\\\\n", "f_{2}(w) \\bigcirc g_{2}(x) \\\\\n", "\\vdots \\\\\n", "f_{n}(w) \\bigcirc g_{n}(x) \n", "\\end{bmatrix} \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Jacobian of Elementwise Binary Operations\n", "The general case for the Jacobian of **y** wrt **w** is the square matrix:\n", "$$\n", "J_{W} =\n", "\\frac{\\partial y}{\\partial w} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_{1}}(f_{1}(w) \\bigcirc g_{1}(x)) & \\frac{\\partial}{\\partial w_{2}}(f_{1}(w) \\bigcirc g_{1}(x)) & \\dotsb & \\frac{\\partial}{\\partial w_{n}}(f_{1}(w) \\bigcirc g_{1}(x)) \\\\\n", "\\frac{\\partial}{\\partial w_{1}}(f_{2}(w) \\bigcirc g_{2}(x)) & \\frac{\\partial}{\\partial w_{2}}(f_{2}(w) \\bigcirc g_{2}(x)) & \\dotsb & \\frac{\\partial}{\\partial w_{n}}(f_{2}(w) \\bigcirc g_{2}(x)) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial w_{1}}(f_{n}(w) \\bigcirc g_{n}(x)) & \\frac{\\partial}{\\partial w_{2}}(f_{n}(w) \\bigcirc g_{n}(x)) & \\dotsb & \\frac{\\partial}{\\partial w_{n}}(f_{n}(w) \\bigcirc g_{n}(x)) \n", "\\end{bmatrix} \n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The general case for the Jacobian of **y** wrt **x** is the square matrix:\n", "$$\n", "J_{X} =\n", "\\frac{\\partial y}{\\partial x} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_{1}}(f_{1}(w) \\bigcirc g_{1}(x)) & \\frac{\\partial}{\\partial x_{2}}(f_{1}(w) \\bigcirc g_{1}(x)) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}(f_{1}(w) \\bigcirc g_{1}(x)) \\\\\n", "\\frac{\\partial}{\\partial x_{1}}(f_{2}(w) \\bigcirc g_{2}(x)) & \\frac{\\partial}{\\partial x_{2}}(f_{2}(w) \\bigcirc g_{2}(x)) & \\dotsb & \\frac{\\partial}{\\partial w_{n}}(f_{2}(w) \\bigcirc g_{2}(x)) \\\\\n", "\\dotsb \\\\\n", "\\frac{\\partial}{\\partial x_{1}}(f_{n}(w) \\bigcirc g_{n}(x)) & \\frac{\\partial}{\\partial x_{2}}(f_{n}(w) \\bigcirc g_{n}(x)) & \\dotsb & \\frac{\\partial}{\\partial x_{n}}(f_{n}(w) \\bigcirc g_{n}(x)) \n", "\\end{bmatrix} \n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Diagonal Jacobians\n", "In a Diagonal Jacobian, all elements off the diagonal are zero, $ \\frac{\\partial}{\\partial w_{j}}(f_{i}(w) \\bigcirc g_{i}(x)) = 0 $ where $ j \\ne i $\n", "\n", "This will be the case when $ f_{i} $ and $ g_{i} $ are constants wrt $ w_{j} $:\n", "\n", "$$ \n", "\\frac{\\partial}{\\partial w_{j}}f_{i}(w) =\n", "\\frac{\\partial}{\\partial w_{j}}g_{i}(x) = 0\n", "$$\n", "\n", "Regardless of the operation $ \\bigcirc $, if the partial derivatives go to zero, $ 0 \\bigcirc 0 = 0 $ and the partial derivative of a constant is zero.\n", "\n", "These partial derivatives go to zero when $f_{i} $ and $ g_{i} $ are not functions of $ w_{j} $.\n", "\n", "Element-wise operations imply that $ f_{i} $ is purely a function of $ w_{i} $ and $ g_{i} $ is purely a function of $ x_{i} $.\n", "\n", "For example, $ \\pmb{w} + \\pmb{x} $ sums $ w_{i} + x_{i} $. \n", "\n", "Consequently, $ f_{i}(w) \\bigcirc g_{i}(x) $ reduces to $ f_{i}(w_{i}) \\bigcirc g_{i}(x_{i}) $ and the goal becomes $ \\frac{\\partial}{\\partial w_{j}}f_{i}(w_{i}) = 0$ and $ \\frac{\\partial}{\\partial w_{j}}g_{i}(x_{i}) = 0 $\n", "\n", "Notice that $ f_{i}(w_{i}) $ and $ g_{i}(x_{i}) $ look like constants to the partial differentiation wrt to $ w_{j} $ when $ j \\ne i $\n", "\n", "#### Element-wise diagonal condition\n", "_Element-wise diagonal condition_ refers to the constraint that $ f_{i}(w) $ and $ g_{i}(x) $ access at most only $ w_{i} $ and $ x_{i} $, respectively." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Jacobians under an element-wise diagonal condition\n", "\n", "Under this condition, the elements along the diagonal of the Jacobian are $ \\frac{\\partial}{\\partial w_{i}}(f_{i}(w_{i}) \\bigcirc g_{i}(x_{i})) $:\n", "\n", "$$\n", "\\frac{\\partial y}{\\partial w} = \n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_{1}}(f_{1}(w_{1}) \\bigcirc g_{1}(x_{1})) & & \\\\\n", "& \\frac{\\partial}{\\partial w_{2}}(f_{2}(w_{2}) \\bigcirc g_{2}(x_{2})) & \\huge0 \\\\\n", "\\dotsb \\\\\n", "\\huge0 & & \\frac{\\partial}{\\partial w_{n}}(f_{n}(w_{n}) \\bigcirc g_{n}(x_{n}))\n", "\\end{bmatrix}\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "More succinctly, we can rewrite the following as:\n", "$$\n", "\\frac{\\partial y}{\\partial w} = \n", "diag \\left(\\frac{\\partial}{\\partial w_{1}}(f_{1}(w_{1}) \\bigcirc g_{1}(x_{1})),\n", "\\frac{\\partial}{\\partial w_{2}}(f_{2}(w_{2}) \\bigcirc g_{2}(x_{2})),\n", "\\dotsb ,\n", "\\frac{\\partial}{\\partial w_{n}}(f_{n}(w_{n}) \\bigcirc g_{n}(x_{n}))\\right)\n", "$$\n", "\n", "and\n", "\n", "$$\n", "\\frac{\\partial y}{\\partial x} = \n", "diag \\left(\\frac{\\partial}{\\partial x_{1}}(f_{1}(w_{1}) \\bigcirc g_{1}(x_{1})),\n", "\\frac{\\partial}{\\partial x_{2}}(f_{2}(w_{2}) \\bigcirc g_{2}(x_{2})),\n", "\\dotsb ,\n", "\\frac{\\partial}{\\partial x_{n}}(f_{n}(w_{n}) \\bigcirc g_{n}(x_{n}))\\right)\n", "$$\n", "\n", "where $ diag(x) $ constructs a matrix whose diagonal elements are taken from the vector $ \\pmb{x} $ \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the general function $ \\pmb{f}(w) $ in the binary element-wise operation is just the vector $ \\pmb{w} $, then we know that $ f_i(w) $ reduces to $ f_i(w_i) = w_i $\n", "\n", "As an example, $ \\pmb{w} + \\pmb{x} $ fits our element-wise diagonal condtion because $ \\pmb{f}(\\pmb{w}) + \\pmb{g}(\\pmb{x}) $ has scalar equations $ y_i = f_i(w) + g_i(x) $ that reduce to just $ y_i = f_i(w_i) + g_i(x_i) = w_i + x_i $ with the partial derivatives:\n", "\n", "\n", "\n", "$$\n", "\\frac{\\partial}{\\partial w_i}(f_i(w_i) + g_i(x_i)) =\n", "\\frac{\\partial}{\\partial w_i}(w_i + x_i) = \n", "\\frac{\\partial w_i}{\\partial w_i} + \\frac{\\partial x_i}{\\partial w_i} = \n", "1 + 0 =\n", "1\n", "$$\n", "\n", "and\n", "\n", "$$\n", "\\frac{\\partial}{\\partial x_i}(f_i(w_i) + g_i(x_i)) =\n", "\\frac{\\partial}{\\partial x_i}(w_i + x_i) = \n", "\\frac{\\partial w_i}{\\partial x_i} + \\frac{\\partial x_i}{\\partial x_i} = \n", "0 + 1 =\n", "1\n", "$$\n", "Which gives us:\n", "$$\n", "\\frac{\\partial (w + x)}{\\partial w} =\n", "\\frac{\\partial (w + x)}{\\partial x} =\n", "I\n", "$$\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given the simplicity of this special case $f_i(\\pmb{w}) $ reducing to $ f_i(w_i) $, we can derive the Jacobians for common element-wise binary operations:\n", "\n", "##### Addition _(wrt to w)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} + \\pmb{x})}{\\partial \\pmb{w}} =\n", "diag \\left(\\dotsb \\frac{\\partial (w_i + x_i)}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{\\partial w_i}{\\partial w_i} + \\frac{\\partial x_i}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb 1 + 0 \\dotsb \\right) =\n", "diag \\left(\\dotsb 1 \\dotsb \\right) =\n", "diag(\\vec{1}) =\n", "I\n", "$$\n", "##### Subtraction _(wrt to w)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} - \\pmb{x})}{\\partial \\pmb{w}} =\n", "diag \\left(\\dotsb \\frac{\\partial w_i - x_i}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{\\partial w_i}{\\partial w_i} - \\frac{\\partial x_i}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb 1 - 0 \\dotsb \\right) =\n", "diag \\left(\\dotsb 1 \\dotsb \\right) =\n", "diag(\\vec{1}) =\n", "I\n", "$$\n", "\n", "##### Elementwise Multiplication (aka Hadamard product) _(wrt to w)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} \\otimes \\pmb{x})}{\\partial \\pmb{w}} =\n", "diag \\left(\\dotsb \\frac{\\partial w_i \\times x_i}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb (x_i \\times \\frac{\\partial w_i}{\\partial w_i}) + (w_i \\times \\frac{\\partial x_i}{\\partial w_i}) \\dotsb \\right) =\n", "diag \\left(\\dotsb (x_i \\times 1 ) + (w_i \\times 0) \\dotsb \\right) =\n", "diag \\left(\\dotsb x_i \\dotsb \\right) =\n", "diag(\\pmb{x})\n", "$$\n", "\n", "##### Elementwise Division _(wrt to w)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} \\oslash \\pmb{x})}{\\partial \\pmb{w}} =\n", "diag \\left(\\dotsb \\frac{\\partial (\\frac{w_i}{x_i})}{\\partial w_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb (\\frac{1}{x_i} \\times \\frac{\\partial w_i}{\\partial w_i}) + (w_i \\times \\frac{\\partial \\frac{1}{x_i}}{\\partial w_i}) \\dotsb \\right) =\n", "diag \\left(\\dotsb (\\frac{1}{x_i} \\times 1 ) + (w_i \\times 0) \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{1}{x_i} \\dotsb \\right)\n", "$$\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "##### Addition _(wrt to x)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} + \\pmb{x})}{\\partial \\pmb{x}} =\n", "diag \\left(\\dotsb \\frac{\\partial (w_i + x_i)}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{\\partial w_i}{\\partial x_i} + \\frac{\\partial x_i}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb 0 + 1 \\dotsb \\right) =\n", "diag \\left(\\dotsb 1 \\dotsb \\right) =\n", "diag(\\vec{1}) =\n", "I\n", "$$\n", "##### Subtraction _(wrt to x)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} - \\pmb{x})}{\\partial \\pmb{x}} =\n", "diag \\left(\\dotsb \\frac{\\partial (w_i - x_i)}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{\\partial w_i}{\\partial x_i} - \\frac{\\partial x_i}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb 0 - 1 \\dotsb \\right) =\n", "diag \\left(\\dotsb -1 \\dotsb \\right) =\n", "diag(-\\vec{1}) =\n", "-I\n", "$$\n", "\n", "##### Elementwise Multiplication (aka Hadamard product) _(wrt to x)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} \\otimes \\pmb{x})}{\\partial \\pmb{x}} =\n", "diag \\left(\\dotsb \\frac{\\partial (w_i \\times x_i)}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb (x_i \\times \\frac{\\partial w_i}{\\partial x_i}) + (w_i \\times \\frac{\\partial x_i}{\\partial x_i}) \\dotsb \\right) =\n", "diag \\left(\\dotsb (x_i \\times 0) + (w_i \\times 1) \\dotsb \\right) =\n", "diag \\left(\\dotsb w_i \\dotsb \\right) =\n", "diag(\\pmb{w})\n", "$$\n", "\n", "##### Elementwise Division _(wrt to x)_\n", "$$\n", "\\frac{\\partial (\\pmb{w} \\oslash \\pmb{x})}{\\partial \\pmb{x}} =\n", "diag \\left(\\dotsb \\frac{\\partial (\\frac{w_i}{x_i})}{\\partial x_i} \\dotsb \\right) =\n", "diag \\left(\\dotsb (\\frac{1}{x_i} \\times \\frac{\\partial w_i}{\\partial x_i}) + (w_i \\times \\frac{\\partial}{\\partial x_i}(\\frac{1}{x})) \\dotsb \\right) =\n", "diag \\left(\\dotsb (\\frac{1}{x_i} \\times 0) + (w_i \\times -\\frac{1}{x_{i}^{2}}) \\dotsb \\right) =\n", "diag \\left(\\dotsb \\frac{-w_i}{x_i^2} \\dotsb \\right)\n", "$$\n", "\n", "_Note_:\n", "$$\n", "\\frac{\\partial (\\frac{w_i}{x_i})}{\\partial x_i} =\n", "w_i \\times \\frac{\\partial (\\frac{1}{x_i})}{\\partial x_i} =\n", "w_i \\times \\frac{\\partial x_{i}^{-1}}{\\partial x_i} =\n", "w_i \\times (-1) \\times x_{i}^{-2} =\n", "\\frac {-w_i}{x_{i}^{2}}\n", "$$\n", "\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Derivatives involving scalar expansion\n", "\n", "To multiply or add scalars to vectors, we expand the scalar to a vector and\n", "perform an elementwise binary operation.\n", "\n", "Given a scalar $ z $ and a vector $ \\pmb{x} $, the partial derivatives for elementwise\n", "operations for the function $ \\pmb{y} = \\pmb{x} \\bigcirc z $ wrt to $ \\pmb{x} $ is given by:\n", "\n", "$$\n", "\\frac{\\partial \\pmb{y}}{\\partial \\pmb{x}} =\n", "diag \\left( \\dotsb \\frac{\\partial}{\\partial x_i}(f_i(x_i) \\bigcirc g_i(z)) \\dotsb \\right)\n", "$$\n", "\n", "where $ \\pmb{g}(z) = \\vec{1}z $ is a vector function of the appropriate length where $ g_i(z) = z $ and $ \\pmb{f}(\\pmb{x}) $ is a vector function such that $ f_i(x_i) = x_i $" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This results in the following (based on the previous derivations for elementwise binary operations)\n", "\n", "##### Addition wrt x :\n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} + z) =\n", "I\n", "$$\n", "Since \n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} + z) =\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{f}(\\pmb{x}) + \\pmb{g}(z) =\n", "diag \\left(\\dotsb \\frac{\\partial x_i + z}{\\partial x_i} \\dotsb \\right) =\n", "diag(\\vec{1}) =\n", "I\n", "$$\n", "Since for a given $ y_i = f_i(x) + g_i(z) = x_i + z $, \n", "\n", "$$\n", "\\frac{\\partial y_i}{\\partial x_i} =\n", "\\frac{\\partial}{\\partial x_{i}}(f_i(x) + g_i(z)) =\n", "\\frac{\\partial x_i}{\\partial x_i} + \\frac{\\partial z}{\\partial x_i} =\n", "1 + 0 =\n", "1\n", "$$ \n", "and $ \\frac{\\partial}{\\partial x_{i}}(f_j(x) + g_j(z)) = 0 + 0 $ for all $ i \\ne j $ \n", "\n", "##### Subtraction wrt x:\n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} - z) =\n", "I\n", "$$\n", "Since \n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} - z) =\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{f}(\\pmb{x}) - \\pmb{g}(z) =\n", "diag \\left(\\dotsb \\frac{\\partial x_i - z}{\\partial x_i} \\dotsb \\right) =\n", "diag(\\vec{1}) =\n", "I\n", "$$\n", "Since for a given $ y_i = f_i(x) - g_i(z) = x_i - z $, \n", "\n", "$$\n", "\\frac{\\partial y_i}{\\partial x_i} =\n", "\\frac{\\partial}{\\partial x_{i}}(f_i(x) - g_i(z)) =\n", "\\frac{\\partial}{\\partial x_{i}}x_i - \\frac{\\partial}{\\partial z} =\n", "1 - 0 =\n", "1\n", "$$ \n", "and $ \\frac{\\partial}{\\partial x_{i}}(f_j(x) - g_j(z)) = 0 - 0 $ for all $ i \\ne j $ \n", "\n", "##### Multiplication wrt x :\n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} \\times z) =\n", "diag(z)\n", "$$\n", "Since\n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} \\times z) =\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{f}(\\pmb{x}) \\otimes \\pmb{g}(z)) = \n", "diag \\left( \\dotsb \\frac{\\partial}{\\partial x_{i}}(x_{i} \\times z) \\dotsb \\right) =\n", "diag(z)\n", "$$\n", "Since for $ y_i = x_i \\times z $, \n", "$$\n", "\\frac{\\partial y_i}{\\partial x_i} = \n", "\\frac{\\partial}{\\partial x_{i}}(x_{i} \\times z) = \n", "(z \\times \\frac{\\partial x_{i}}{\\partial x_{i}}) + (x_i \\times \\frac {\\partial z}{\\partial x_{i}}) =\n", "(z \\times 1) + (x_i \\times 0)\n", "= z\n", "$$\n", "for all $ i = j $ and $ \\frac{\\partial y_j}{\\partial x_i} = 0 $ for all $ j \\ne i $\n", "since $ \\frac{\\partial x_j}{\\partial x_i} = 0 $ and $ \\frac{\\partial z}{\\partial x_i} = 0 $\n", "##### Division wrt x:\n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\frac{\\pmb{x}}{z}) =\n", "diag \\left( \\frac{1}{z} \\right)\n", "$$\n", "Since \n", "$$\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\frac{\\pmb{x}}{z}) =\n", "\\frac{\\partial}{\\partial \\pmb{x}}(\\pmb{x} \\times \\frac{1}{z}) =\n", "diag \\left( \\frac{1}{z} \\right)\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### WRT to Z however,\n", "\n", "##### Addition wrt z :\n", "$$ \n", "\\frac{\\partial}{\\partial z}(\\pmb{x} + z) =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}(x_1 + z) \\\\\n", "\\frac{\\partial}{\\partial z}(x_2 + z) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_n + z)\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}x_1 + \\frac{\\partial z}{\\partial z} \\\\\n", "\\frac{\\partial}{\\partial z}x_2 + \\frac{\\partial z}{\\partial z} \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}x_n + \\frac{\\partial z}{\\partial z}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "0 + 1 \\\\\n", "0 + 1 \\\\\n", "\\vdots \\\\\n", "0 + 1 \\\\\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "1 \\\\\n", "1 \\\\\n", "\\vdots \\\\\n", "1 \\\\\n", "\\end{bmatrix} =\n", "\\vec{1}\n", "$$\n", "\n", "##### Subtraction wrt z:\n", "$$ \n", "\\frac{\\partial}{\\partial z}(\\pmb{x} - z) =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}(x_1 - z) \\\\\n", "\\frac{\\partial}{\\partial z}(x_2 - z) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_n - z)\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}x_1 - \\frac{\\partial z}{\\partial z} \\\\\n", "\\frac{\\partial}{\\partial z}x_2 - \\frac{\\partial z}{\\partial z} \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}x_n - \\frac{\\partial z}{\\partial z}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "0 - 1 \\\\\n", "0 - 1 \\\\\n", "\\vdots \\\\\n", "0 - 1 \\\\\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "-1 \\\\\n", "-1 \\\\\n", "\\vdots \\\\\n", "-1 \\\\\n", "\\end{bmatrix} =\n", "\\vec{-1}\n", "$$\n", "##### Multiplication wrt z :\n", "$$\n", "\\frac{\\partial}{\\partial z}(\\pmb{x} \\times z) =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}(x_1 \\times z) \\\\\n", "\\frac{\\partial}{\\partial z}(x_2 \\times z) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_i \\times z) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_n \\times z)\n", "\\end{bmatrix} =\n", "\\pmb{x}\n", "$$\n", "Since\n", "$$\n", "\\frac{\\partial}{\\partial z}(x_i \\times z) =\n", "(x_i \\times \\frac{\\partial z}{\\partial z}) + (z \\times \\frac{\\partial x_i}{\\partial z}) =\n", "(x_i \\times 1) + (z \\times 0) =\n", "x_i\n", "$$\n", "##### Division wrt z:\n", "$$\n", "\\frac{\\partial}{\\partial z}(\\frac{\\pmb{x}}{z}) =\n", "\\frac{\\partial}{\\partial z}(\\pmb{x} \\times \\frac{1}{z}) =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial z}(x_1 \\times \\frac{1}{z}) \\\\\n", "\\frac{\\partial}{\\partial z}(x_2 \\times \\frac{1}{z}) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_i \\times \\frac{1}{z}) \\\\\n", "\\vdots \\\\\n", "\\frac{\\partial}{\\partial z}(x_n \\times \\frac{1}{z})\n", "\\end{bmatrix} =\n", "\\pmb{x}\\frac{-1}{z^2} =\n", "\\frac{- \\pmb{x}}{z^2} =\n", "$$\n", "Since \n", "$$\n", "\\frac{\\partial}{\\partial z}(x_i \\times \\frac{1}{z}) =\n", "(x_i \\times \\frac{\\partial}{\\partial z}\\frac{1}{z}) + (\\frac{1}{z} \\times \\frac{\\partial x_i}{\\partial z}) =\n", "(x_i \\times \\frac{-1}{z^2}) + (\\frac{1}{z} \\times 0) =\n", "\\frac{-x_i}{z^2}\n", "$$\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Vector Sum Reduction\n", "\n", "Let $ y = sum(f(\\pmb{x})) = \\sum \\limits_{i=1}^{n}f_i(\\pmb{x}) $, where each function $ f_i $ could use all values in the vector, not just $ x_i $. The sum is over the results of the function, not the parameter. The gradient ($ 1 \\times n $ Jacobian) of vector summation is: \n", "\n", "$$\n", "\\frac{\\partial y}{\\partial \\pmb{x}} = \n", "\\begin{bmatrix}\n", "\\frac{\\partial y}{\\partial x_1}, \\frac{\\partial y}{\\partial x_2}, \\dotsb , \\frac{\\partial y}{\\partial x_n}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_1}\\sum_i f_i(x), \\frac{\\partial}{\\partial x_2} \\sum_i f_i(x), \\dotsb , \\frac{\\partial}{\\partial x_n}\\sum_i f_i(x)\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\sum_i \\frac{\\partial}{\\partial x_1}f_i(x), \\sum_i \\frac{\\partial}{\\partial x_2} f_i(x), \\dotsb , \\sum_i \\frac{\\partial}{\\partial x_n}f_i(x)\n", "\\end{bmatrix}\n", "\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For example, given $ y = sum(x) $, the function $ f_i(x) = x_i $ and the gradient is:\n", "\n", "$$\n", "\\nabla y = \n", "\\begin{bmatrix}\n", "\\sum_i \\frac{\\partial f_i(x)}{\\partial x_1}, \\sum_i \\frac{\\partial f_i(x)}{\\partial x_2}, \\dotsb , \\sum_i \\frac{\\partial f_i(x)}{\\partial x_n}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\sum_i \\frac{\\partial x_i}{\\partial x_1}, \\sum_i \\frac{\\partial x_i}{\\partial x_2}, \\dotsb , \\sum_i \\frac{\\partial x_i}{\\partial x_n}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial x_1}{\\partial x_1}, \\frac{\\partial x_2}{\\partial x_2}, \\dotsb , \\frac{\\partial x_n}{\\partial x_n}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "1, 1, \\dotsb , 1\n", "\\end{bmatrix} =\n", "\\vec{1}^T\n", "$$\n", "Since $ \\frac{\\partial}{\\partial x_j}x_i = 0 $ for $ i \\ne j $." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As another example, given $ y = sum(\\pmb{x}z) $ then $ f_{i}(x,z) = x_{i}z $\n", "The gradient wrt to $ \\pmb{x} $ is:\n", "$$\n", "\\frac{\\partial y}{\\partial \\pmb{x}} =\n", "\\begin{bmatrix}\n", "\\sum_i\\frac{\\partial}{\\partial x_1}x_{i}z,\n", "\\sum_i\\frac{\\partial}{\\partial x_2}x_{i}z,\n", "\\dotsb,\n", "\\sum_i\\frac{\\partial}{\\partial x_n}x_{i}z,\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial x_1}x_{1}z,\n", "\\frac{\\partial}{\\partial x_2}x_{2}z,\n", "\\dotsb,\n", "\\frac{\\partial}{\\partial x_n}x_{n}z,\n", "\\end{bmatrix} =\n", "\\left[ z,z, \\dotsb , z \\right] =\n", "\\vec{z}^T\n", "$$\n", "where $ \\vert \\pmb{x} \\vert = \\vert \\vec{z} \\vert = n $.\n", "\n", "With respect to the scalar z, the resulting is a $ 1 \\times 1 $\n", "$$\n", "\\frac{\\partial y}{\\partial z} =\n", "\\frac{\\partial}{\\partial z}\\sum_{i=1}^{n}x_{i}z =\n", "\\sum_i \\frac{\\partial}{\\partial z} x_{i}z =\n", "\\sum_i x_i =\n", "sum(\\pmb{x})\n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Vector Chain Rules\n", "\n", "Three types of chain rules:\n", "\n", "* _single-variable chain rule_ - derivative of a scalar function with respect to a scalar\n", "* _single-variable total-derivative chain rule_ - derivative of a scalar function wrt to scalar _but with \"many paths\"_\n", "* _vector chain rule_ - full vector chain rules\n", "\n", "##### Single Variable Chain Rule\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given $ y = f(g(x)) $, the scalar derivation rule can be expressed by defining an intermediate variable $ u $ such that:\n", "$$\n", "y = f(u)\n", "$$\n", "and\n", "$$\n", "u = g(x)\n", "$$\n", "The derivative of\n", "$$\n", "\\frac{dy}{dx} = \\frac{dy}{du}\\frac{du}{dx}\n", "$$\n", "As an example, given:\n", "$$\n", "y = sin(x^2)\n", "$$\n", "then \n", "$$\n", "\\frac{dy}{dx} = \n", "\\frac{d}{du}sin(u) \\times \\frac{d}{dx}(x^2) =\n", "cos(u) \\times 2x =\n", "2x cos(x^2) \n", "$$\n", "where \n", "$$\n", "u = x^2\n", "$$\n", "\n", "###### Another example: \n", "\n", "Given $ y = f(x) = ln(sin(x^3)^2) $ let: \n", "$$\n", "\\begin{align*}\n", "u_1 &= f_1(x) = x^3 \\\\\n", "u_2 &= f_2(u_1) = sin(u_1) \\\\\n", "u_3 &= f_3(u_2) = u_2^2 \\\\\n", "u_4 &= f_4(u_3) = ln(u_3) \\\\\n", "y &= u_4 &\n", "\\end{align*}\n", "$$\n", "\n", "Such that\n", "$$\n", "\\begin{align*}\n", "\\frac{d}{dx}u_1 &= \\frac{d}{dx}x^3 = 3x^2 \\\\\n", "\\frac{d}{du_1}u_2 &= \\frac{d}{du_1}sin(u_1) = cos(u_1)\\\\\n", "\\frac{d}{du_2}u_3 &= \\frac{d}{du_2}u_2^2 = 2u_2\\\\\n", "\\frac{d}{du_3}u_4 &= \\frac{d}{du_3}ln(u_3) = \\frac{1}{u_3}\\\\\n", "\\frac{dy}{du_4} &= \\frac{du_4}{du_4} = 1\n", "\\end{align*}\n", "$$\n", "\n", "And combining the intermediate variables :\n", "$$\n", "\\frac{dy}{dx} = \\frac{dy}{du_4}\\frac{du_4}{du_3}\\frac{du_3}{du_2}\\frac{du_2}{du_1}\\frac{du_1}{dx} =\n", "1 \\times \\frac{1}{u_3} \\times 2u_2 \\times cos(u_1) \\times 3x^2 =\n", "\\frac{6u_2x^2cos(u_1)}{u_3}\n", "$$\n", "\n", "Substituting the derivatives results in\n", "$$\n", "\\frac{dy}{dx} = \n", "\\frac{6sin(u_1)x^2cos(x^3}{u_2^2} =\n", "\\frac{6sin(x^3)x^2cos(x^3)}{sin(u_1)^2} =\n", "\\frac{6sin(x^3)x^2cos(x^3)}{sin(x^3)^2} =\n", "\\frac{6x^2cos(x^3)}{sin(x^3)}\n", "$$\n", "\n", "##### Single variable total derivative chain rule\n", "$$\n", "\\frac{\\partial f(u_1,\\dotsb,u_{n+1})}{\\partial x} =\n", "\\sum \\limits_{i=1}^{n+1}\\frac{\\partial f}{\\partial u_i}\\frac{\\partial u_i}{\\partial x}\n", "$$\n", "\n", "As an example, given: $ y = f(x) = x + x^2 $, we can restate the function as:\n", "$$\n", "\\begin{align*}\n", "u_1(x) &= x^2 \\\\\n", "u_2(x,u_1) &= x + u_1 \\\\\n", "y &= f(x) = u_2(x,u_1)\n", "\\end{align*}\n", "$$\n", "Applying the total derivative rule:\n", "$$\n", "\\frac{dy}{dx} =\n", "\\frac{\\partial f(x)}{\\partial x} =\n", "\\frac{\\partial u_2(x,u_1)}{\\partial x} =\n", "\\frac{\\partial u_2}{\\partial x}\\frac{\\partial x}{\\partial x} + \\frac{\\partial u_2}{\\partial u_1}\\frac{\\partial u_1}{\\partial x} = \n", "$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Questions and Exercises\n", "\n", "1. Compute the Jacobian of the following function $ \\pmb{f}(\\pmb{w}) $:\n", "\n", " 1. such that $ f_{i}(w) = w_{i}^{2} $: \n", "$$\n", "\\frac{\\partial \\pmb{y}}{\\partial \\pmb{w}} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1}f_1(w) & \\frac{\\partial}{\\partial w_2}f_1(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_1(w) \\\\\n", "\\frac{\\partial}{\\partial w_1}f_2(w) & \\frac{\\partial}{\\partial w_2}f_2(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_2(w) \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1}f_n(w) & \\frac{\\partial}{\\partial w_2}f_n(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_n(w) \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1} w_{1}^{2} & \\frac{\\partial}{\\partial w_2}w_{1}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{1}^{2} \\\\\n", "\\frac{\\partial}{\\partial w_1} w_{2}^{2} & \\frac{\\partial}{\\partial w_2}w_{2}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{2}^{2} \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1} w_{n}^{2} & \\frac{\\partial}{\\partial w_2}w_{n}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{n}^{2} \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "2w_1 & 0 & \\dotsb & 0 \\\\\n", "0 & 2w_2 & \\dotsb & 0 \\\\\n", "& & \\ddots \\\\\n", "0 & 0 & \\dotsb & 2w_n\n", "\\end{bmatrix} =\n", "diag \\left( \\dotsb 2 w_i \\dotsb \\right) \n", "$$\n", "\n", " 1. such that $ f_{i}(w) = w_{i}^{i} $:\n", "$$\n", "\\frac{\\partial \\pmb{y}}{\\partial \\pmb{w}} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1}f_1(w) & \\frac{\\partial}{\\partial w_2}f_1(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_1(w) \\\\\n", "\\frac{\\partial}{\\partial w_1}f_2(w) & \\frac{\\partial}{\\partial w_2}f_2(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_2(w) \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1}f_n(w) & \\frac{\\partial}{\\partial w_2}f_n(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_n(w) \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1} w_{1}^{1} & \\frac{\\partial}{\\partial w_2}w_{1}^{1} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{1}^{1} \\\\\n", "\\frac{\\partial}{\\partial w_1} w_{2}^{2} & \\frac{\\partial}{\\partial w_2}w_{2}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{2}^{2} \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1} w_{n}^{n} & \\frac{\\partial}{\\partial w_2}w_{n}^{n} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{n}^{n} \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "1w_{1}^{0} & 0 & \\dotsb & 0 \\\\\n", "0 & 2w_{2}^{1} & \\dotsb & 0 \\\\\n", "& & \\ddots \\\\\n", "0 & 0 & \\dotsb & nw_{n}^{n-1}\n", "\\end{bmatrix} =\n", "diag \\left( \\dotsb iw_{i}^{i-1} \\dotsb \\right) \n", "$$\n", " 1. such that $ f_{i}(w) = \\sum \\limits _{j} ^{m} w_{j}^{i} $ where $ n = m = \\vert w \\vert $:\n", "$$\n", "\\frac{\\partial \\pmb{y}}{\\partial \\pmb{w}} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1}f_1(w) & \\frac{\\partial}{\\partial w_2}f_1(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_1(w) \\\\\n", "\\frac{\\partial}{\\partial w_1}f_2(w) & \\frac{\\partial}{\\partial w_2}f_2(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_2(w) \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1}f_n(w) & \\frac{\\partial}{\\partial w_2}f_n(w) & \\dotsb & \\frac{\\partial}{\\partial w_n}f_n(w) \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1} \\sum \\limits _{j} ^{n} w_{j}^{1} & \\frac{\\partial}{\\partial w_2}\\sum \\limits _{j} ^{n} w_{j}^{1} & \\dotsb & \\frac{\\partial}{\\partial w_n}\\sum \\limits _{j} ^{n} w_{j}^{1} \\\\\n", "\\frac{\\partial}{\\partial w_1} \\sum \\limits _{j} ^{n} w_{j}^{2} & \\frac{\\partial}{\\partial w_2}\\sum \\limits _{j} ^{n} w_{j}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}\\sum \\limits _{j} ^{n} w_{j}^{2} \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1} \\sum \\limits _{j} ^{n} w_{j}^{n} & \\frac{\\partial}{\\partial w_2}\\sum \\limits _{j} ^{n} w_{j}^{n} & \\dotsb & \\frac{\\partial}{\\partial w_n}\\sum \\limits _{j} ^{n} w_{j}^{n} \n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "\\frac{\\partial}{\\partial w_1}w_{1}^{1} & \\frac{\\partial}{\\partial w_2}w_{2}^{1} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{n}^{1} \\\\\n", "\\frac{\\partial}{\\partial w_1}w_{1}^{2} & \\frac{\\partial}{\\partial w_2}w_{2}^{2} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{n}^{2} \\\\\n", "& & \\ddots \\\\\n", "\\frac{\\partial}{\\partial w_1}w_{1}^{n} & \\frac{\\partial}{\\partial w_2}w_{2}^{n} & \\dotsb & \\frac{\\partial}{\\partial w_n}w_{n}^{n} \\\\\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "1w_{1}^{0} & 1w_{2}^{0} & \\dotsb & 1w_{n}^{0} \\\\\n", "2w_{1}^{1} & 2w_{2}^{1} & \\dotsb & 2w_{n}^{1} \\\\\n", "& & \\ddots \\\\\n", "nw_{1}^{n-1} & nw_{2}^{n-1} & \\dotsb & nw_{n}^{n-1}\n", "\\end{bmatrix} =\n", "\\begin{bmatrix}\n", "1 & 1 & \\dotsb & 1 \\\\\n", "2w_{1} & 2w_{2} & \\dotsb & 2w_{n} \\\\\n", "& & \\ddots \\\\\n", "nw_{1}^{n-1} & nw_{2}^{n-1} & \\dotsb & nw_{n}^{n-1}\n", "\\end{bmatrix} \n", "$$\n", "Since\n", "$$\n", "\\frac{\\partial}{\\partial w_i} \\sum \\limits _{j} ^{n} w_{j}^{i} =\n", "\\frac{\\partial}{\\partial w_i} \\left( w_{1}^{i} + w_{2}^{i} + \\dots + w_{i}^{i} + \\dots + w_{n}^{i} \\right) =\n", "\\left( \\frac{\\partial}{\\partial w_i}w_{1}^{i} + \\frac{\\partial}{\\partial w_i}w_{2}^{i} + \\dots + \\frac{\\partial}{\\partial w_i}w_{i}^{i} + \\dots + \\frac{\\partial}{\\partial w_i}w_{n}^{i} \\right) =\n", "\\left( 0 + 0 + \\dots + iw_{i}^{i-1} + \\dots + \\right) =\n", "iw_{i}^{i-1}\n", "$$\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }