{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "62040e94",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Linear Algebra\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dc36b473",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:51.529162Z",
     "iopub.status.busy": "2023-08-18T19:41:51.528467Z",
     "iopub.status.idle": "2023-08-18T19:41:53.438267Z",
     "shell.execute_reply": "2023-08-18T19:41:53.437059Z"
    },
    "origin_pos": 3,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74979942",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Scalars are implemented as tensors \n",
    "that contain only one element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4fc9ba1d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.442690Z",
     "iopub.status.busy": "2023-08-18T19:41:53.442040Z",
     "iopub.status.idle": "2023-08-18T19:41:53.472277Z",
     "shell.execute_reply": "2023-08-18T19:41:53.471491Z"
    },
    "origin_pos": 8,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(5.), tensor(6.), tensor(1.5000), tensor(9.))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor(3.0)\n",
    "y = torch.tensor(2.0)\n",
    "\n",
    "x + y, x * y, x / y, x**y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b18f82b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can think of a vector as a fixed-length array of scalars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "91cd966f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.475759Z",
     "iopub.status.busy": "2023-08-18T19:41:53.475141Z",
     "iopub.status.idle": "2023-08-18T19:41:53.481106Z",
     "shell.execute_reply": "2023-08-18T19:41:53.479872Z"
    },
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 1, 2])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.arange(3)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25d8557f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We access a tensor's elements via indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ba15a197",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.485066Z",
     "iopub.status.busy": "2023-08-18T19:41:53.484260Z",
     "iopub.status.idle": "2023-08-18T19:41:53.492710Z",
     "shell.execute_reply": "2023-08-18T19:41:53.491415Z"
    },
    "origin_pos": 17,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0da49a3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In code, this corresponds to the tensor's length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "871cd7e6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.497529Z",
     "iopub.status.busy": "2023-08-18T19:41:53.496794Z",
     "iopub.status.idle": "2023-08-18T19:41:53.502332Z",
     "shell.execute_reply": "2023-08-18T19:41:53.501510Z"
    },
    "origin_pos": 19,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7eecf0",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Tensors with just one axis have shapes with just one element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "34ea04c3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.505748Z",
     "iopub.status.busy": "2023-08-18T19:41:53.505180Z",
     "iopub.status.idle": "2023-08-18T19:41:53.510136Z",
     "shell.execute_reply": "2023-08-18T19:41:53.509337Z"
    },
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21fdf810",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can convert any appropriately sized $m \\times n$ tensor \n",
    "into an $m \\times n$ matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "80c49751",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.513569Z",
     "iopub.status.busy": "2023-08-18T19:41:53.512931Z",
     "iopub.status.idle": "2023-08-18T19:41:53.518545Z",
     "shell.execute_reply": "2023-08-18T19:41:53.517740Z"
    },
    "origin_pos": 24,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1],\n",
       "        [2, 3],\n",
       "        [4, 5]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = torch.arange(6).reshape(3, 2)\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66b87a60",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Matrix's transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7ef1e23b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.521874Z",
     "iopub.status.busy": "2023-08-18T19:41:53.521332Z",
     "iopub.status.idle": "2023-08-18T19:41:53.526566Z",
     "shell.execute_reply": "2023-08-18T19:41:53.525812Z"
    },
    "origin_pos": 28,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 2, 4],\n",
       "        [1, 3, 5]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a94dc4f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Symmetric matrices are the subset of square matrices\n",
    "that are equal to their own transposes:\n",
    "$\\mathbf{A} = \\mathbf{A}^\\top$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "028e06ed",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.529939Z",
     "iopub.status.busy": "2023-08-18T19:41:53.529400Z",
     "iopub.status.idle": "2023-08-18T19:41:53.535337Z",
     "shell.execute_reply": "2023-08-18T19:41:53.534568Z"
    },
    "origin_pos": 32,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[True, True, True],\n",
       "        [True, True, True],\n",
       "        [True, True, True]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = torch.tensor([[1, 2, 3], [2, 0, 4], [3, 4, 5]])\n",
    "A == A.T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9e3133",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Tensors\n",
    "give us a generic way of describing \n",
    "extensions to $n^{\\textrm{th}}$-order arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "306d610e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.538891Z",
     "iopub.status.busy": "2023-08-18T19:41:53.538210Z",
     "iopub.status.idle": "2023-08-18T19:41:53.546164Z",
     "shell.execute_reply": "2023-08-18T19:41:53.545027Z"
    },
    "origin_pos": 37,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2,  3],\n",
       "         [ 4,  5,  6,  7],\n",
       "         [ 8,  9, 10, 11]],\n",
       "\n",
       "        [[12, 13, 14, 15],\n",
       "         [16, 17, 18, 19],\n",
       "         [20, 21, 22, 23]]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.arange(24).reshape(2, 3, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "53a34bc0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.550917Z",
     "iopub.status.busy": "2023-08-18T19:41:53.550017Z",
     "iopub.status.idle": "2023-08-18T19:41:53.558241Z",
     "shell.execute_reply": "2023-08-18T19:41:53.557366Z"
    },
    "origin_pos": 42,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0., 1., 2.],\n",
       "         [3., 4., 5.]]),\n",
       " tensor([[ 0.,  2.,  4.],\n",
       "         [ 6.,  8., 10.]]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = torch.arange(6, dtype=torch.float32).reshape(2, 3)\n",
    "B = A.clone()\n",
    "A, A + B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44656418",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Elementwise product of two matrices\n",
    "is called their *Hadamard product*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1e2c0645",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.561758Z",
     "iopub.status.busy": "2023-08-18T19:41:53.561198Z",
     "iopub.status.idle": "2023-08-18T19:41:53.567597Z",
     "shell.execute_reply": "2023-08-18T19:41:53.566714Z"
    },
    "origin_pos": 46,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  4.],\n",
       "        [ 9., 16., 25.]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A * B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66d59a48",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Adding or multiplying a scalar and a tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c5c86fb1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.571034Z",
     "iopub.status.busy": "2023-08-18T19:41:53.570428Z",
     "iopub.status.idle": "2023-08-18T19:41:53.577301Z",
     "shell.execute_reply": "2023-08-18T19:41:53.576396Z"
    },
    "origin_pos": 49,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[[ 2,  3,  4,  5],\n",
       "          [ 6,  7,  8,  9],\n",
       "          [10, 11, 12, 13]],\n",
       " \n",
       "         [[14, 15, 16, 17],\n",
       "          [18, 19, 20, 21],\n",
       "          [22, 23, 24, 25]]]),\n",
       " torch.Size([2, 3, 4]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = 2\n",
    "X = torch.arange(24).reshape(2, 3, 4)\n",
    "a + X, (a * X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18209f29",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The sum of a tensor's elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5a3c4cbd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.580908Z",
     "iopub.status.busy": "2023-08-18T19:41:53.580306Z",
     "iopub.status.idle": "2023-08-18T19:41:53.588497Z",
     "shell.execute_reply": "2023-08-18T19:41:53.587623Z"
    },
    "origin_pos": 54,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0., 1., 2.]), tensor(3.))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.arange(3, dtype=torch.float32)\n",
    "x, x.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2da4909",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sums over the elements of tensors of arbitrary shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7594e574",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.593587Z",
     "iopub.status.busy": "2023-08-18T19:41:53.592920Z",
     "iopub.status.idle": "2023-08-18T19:41:53.602105Z",
     "shell.execute_reply": "2023-08-18T19:41:53.600812Z"
    },
    "origin_pos": 58,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 3]), tensor(15.))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape, A.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7c7d75e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Specify the axes \n",
    "along which the tensor should be reduced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "14d191be",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.606330Z",
     "iopub.status.busy": "2023-08-18T19:41:53.605594Z",
     "iopub.status.idle": "2023-08-18T19:41:53.612393Z",
     "shell.execute_reply": "2023-08-18T19:41:53.611227Z"
    },
    "origin_pos": 61,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 3]), torch.Size([3]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape, A.sum(axis=0).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ee3c0559",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.615938Z",
     "iopub.status.busy": "2023-08-18T19:41:53.615181Z",
     "iopub.status.idle": "2023-08-18T19:41:53.621919Z",
     "shell.execute_reply": "2023-08-18T19:41:53.620799Z"
    },
    "origin_pos": 64,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 3]), torch.Size([2]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape, A.sum(axis=1).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "25b99ea4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.625271Z",
     "iopub.status.busy": "2023-08-18T19:41:53.624779Z",
     "iopub.status.idle": "2023-08-18T19:41:53.631897Z",
     "shell.execute_reply": "2023-08-18T19:41:53.630699Z"
    },
    "origin_pos": 67,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.sum(axis=[0, 1]) == A.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da6cc752",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A related quantity is the *mean*, also called the *average*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9f41e037",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.635407Z",
     "iopub.status.busy": "2023-08-18T19:41:53.634799Z",
     "iopub.status.idle": "2023-08-18T19:41:53.642980Z",
     "shell.execute_reply": "2023-08-18T19:41:53.641833Z"
    },
    "origin_pos": 71,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(2.5000), tensor(2.5000))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.mean(), A.sum() / A.numel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f268d8be",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.646514Z",
     "iopub.status.busy": "2023-08-18T19:41:53.645792Z",
     "iopub.status.idle": "2023-08-18T19:41:53.653946Z",
     "shell.execute_reply": "2023-08-18T19:41:53.652643Z"
    },
    "origin_pos": 74,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1.5000, 2.5000, 3.5000]), tensor([1.5000, 2.5000, 3.5000]))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.mean(axis=0), A.sum(axis=0) / A.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a32200f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Keep the number of axes unchanged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "863c5aca",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.658423Z",
     "iopub.status.busy": "2023-08-18T19:41:53.658026Z",
     "iopub.status.idle": "2023-08-18T19:41:53.666522Z",
     "shell.execute_reply": "2023-08-18T19:41:53.665397Z"
    },
    "origin_pos": 77,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 3.],\n",
       "         [12.]]),\n",
       " torch.Size([2, 1]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_A = A.sum(axis=1, keepdims=True)\n",
    "sum_A, sum_A.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5cf23e",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Divide `A` by `sum_A` with broadcasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "93d20f26",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.669896Z",
     "iopub.status.busy": "2023-08-18T19:41:53.669623Z",
     "iopub.status.idle": "2023-08-18T19:41:53.675548Z",
     "shell.execute_reply": "2023-08-18T19:41:53.674706Z"
    },
    "origin_pos": 80,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000, 0.3333, 0.6667],\n",
       "        [0.2500, 0.3333, 0.4167]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A / sum_A"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1ccf2b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The cumulative sum of elements of `A` along some axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e2de0ae7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.678804Z",
     "iopub.status.busy": "2023-08-18T19:41:53.678536Z",
     "iopub.status.idle": "2023-08-18T19:41:53.684437Z",
     "shell.execute_reply": "2023-08-18T19:41:53.683619Z"
    },
    "origin_pos": 82,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 1., 2.],\n",
       "        [3., 5., 7.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.cumsum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00835eec",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The *dot product* of two vectors is a sum over the products of the elements at the same position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5575e11a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.687734Z",
     "iopub.status.busy": "2023-08-18T19:41:53.687178Z",
     "iopub.status.idle": "2023-08-18T19:41:53.696147Z",
     "shell.execute_reply": "2023-08-18T19:41:53.695369Z"
    },
    "origin_pos": 86,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0., 1., 2.]), tensor([1., 1., 1.]), tensor(3.))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.ones(3, dtype = torch.float32)\n",
    "x, y, torch.dot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6742562",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We can calculate the dot product of two vectors \n",
    "by performing an elementwise multiplication followed by a sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b5186254",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.699396Z",
     "iopub.status.busy": "2023-08-18T19:41:53.698843Z",
     "iopub.status.idle": "2023-08-18T19:41:53.704931Z",
     "shell.execute_reply": "2023-08-18T19:41:53.703664Z"
    },
    "origin_pos": 91,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(x * y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ff3a47",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The matrix--vector product $\\mathbf{A}\\mathbf{x}$\n",
    "is simply a column vector of length $m$,\n",
    "whose $i^\\textrm{th}$ element is the dot product \n",
    "$\\mathbf{a}^\\top_i \\mathbf{x}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1a99c29a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.710181Z",
     "iopub.status.busy": "2023-08-18T19:41:53.709041Z",
     "iopub.status.idle": "2023-08-18T19:41:53.719479Z",
     "shell.execute_reply": "2023-08-18T19:41:53.718068Z"
    },
    "origin_pos": 99,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([2, 3]), torch.Size([3]), tensor([ 5., 14.]), tensor([ 5., 14.]))"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.shape, x.shape, torch.mv(A, x), A@x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcd25fac",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can think of the matrix--matrix multiplication $\\mathbf{AB}$\n",
    "as performing $m$ matrix--vector products \n",
    "or $m \\times n$ dot products \n",
    "and stitching the results together \n",
    "to form an $n \\times m$ matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "99535047",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.723651Z",
     "iopub.status.busy": "2023-08-18T19:41:53.722762Z",
     "iopub.status.idle": "2023-08-18T19:41:53.732227Z",
     "shell.execute_reply": "2023-08-18T19:41:53.731088Z"
    },
    "origin_pos": 104,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 3.,  3.,  3.,  3.],\n",
       "         [12., 12., 12., 12.]]),\n",
       " tensor([[ 3.,  3.,  3.,  3.],\n",
       "         [12., 12., 12., 12.]]))"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B = torch.ones(3, 4)\n",
    "torch.mm(A, B), A@B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc1d74b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The $\\ell_2$ *norm*\n",
    "$$\\|\\mathbf{x}\\|_2 = \\sqrt{\\sum_{i=1}^n x_i^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e215c544",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.735890Z",
     "iopub.status.busy": "2023-08-18T19:41:53.735108Z",
     "iopub.status.idle": "2023-08-18T19:41:53.742412Z",
     "shell.execute_reply": "2023-08-18T19:41:53.741311Z"
    },
    "origin_pos": 109,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5.)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u = torch.tensor([3.0, -4.0])\n",
    "torch.norm(u)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7139805a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The $\\ell_1$ norm\n",
    "$$\\|\\mathbf{x}\\|_1 = \\sum_{i=1}^n \\left|x_i \\right|$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8a3e0562",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.746107Z",
     "iopub.status.busy": "2023-08-18T19:41:53.745359Z",
     "iopub.status.idle": "2023-08-18T19:41:53.752438Z",
     "shell.execute_reply": "2023-08-18T19:41:53.751302Z"
    },
    "origin_pos": 114,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(7.)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.abs(u).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a340cd8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The *Frobenius norm*, \n",
    "which is much easier to compute\n",
    "$$\\|\\mathbf{X}\\|_\\textrm{F} = \\sqrt{\\sum_{i=1}^m \\sum_{j=1}^n x_{ij}^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3e00a124",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:41:53.756072Z",
     "iopub.status.busy": "2023-08-18T19:41:53.755281Z",
     "iopub.status.idle": "2023-08-18T19:41:53.762237Z",
     "shell.execute_reply": "2023-08-18T19:41:53.761172Z"
    },
    "origin_pos": 119,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(6.)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.norm(torch.ones((4, 9)))"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "language_info": {
   "name": "python"
  },
  "required_libs": [],
  "rise": {
   "autolaunch": true,
   "enable_chalkboard": true,
   "overlay": "<div class='my-top-right'><img height=80px src='http://d2l.ai/_static/logo-with-text.png'/></div><div class='my-top-left'></div>",
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}