{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"\n",
"*This notebook contains an excerpt from the [Deep Learning with Tensorflow 2.0](https://www.adhiraiyan.org/DeepLearningWithTensorflow.html) by Mukesh Mithrakumar. The code is released under the [MIT license](https://opensource.org/licenses/MIT) and is available for FREE [on GitHub](https://github.com/adhiraiyan/DeepLearningWithTF2.0).*\n",
"\n",
"*Open Source runs on love, laughter and a whole lot of coffee. Consider buying me [one](https://www.buymeacoffee.com/mmukesh) if you find this content useful!*\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"< [01.00 - Preface](01.00-Introduction.ipynb) | [Contents](Index.ipynb) | [03.00 - Probability and Information Theory](03.00-Probability-and-Information-Theory.ipynb) >\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "yeA6xblhq8w8"
},
"source": [
"# 02.00 - Linear Algebra"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "F-3yQvOQHjLr"
},
"source": [
"Linear algebra is the branch of mathematics concerning linear equations and linear functions and their representations through matrices and vector spaces.\n",
"\n",
"Machine Learning relies heavily on Linear Algebra, so it is essential to understand what vectors and matrices are, what operations you can perform with them, and how they can be useful.\n",
"\n",
"If you are already familiar with linear algebra, feel free to skip this chapter but note that the implementation of certain functions are different between Tensorflow 1.0 and Tensorflow 2.0 so you should atleast skim through the code.\n",
"\n",
"If you have had no exposure at all to linear algebra, this chapter will teach you enough to read this book."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
},
"colab_type": "code",
"id": "4cvzagFXgQrd",
"outputId": "fc155a91-c258-47e4-bfa6-1607b2d2e484"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[K |████████████████████████████████| 79.9MB 1.8MB/s \n",
"\u001b[K |████████████████████████████████| 3.0MB 49.7MB/s \n",
"\u001b[K |████████████████████████████████| 419kB 53.2MB/s \n",
"\u001b[K |████████████████████████████████| 61kB 29.2MB/s \n",
"\u001b[?25h"
]
}
],
"source": [
"# Installs\n",
"!pip install --upgrade tf-nightly-2.0-preview"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "of02mDcgUuHQ"
},
"outputs": [],
"source": [
"# Imports\n",
"import tensorflow as tf\n",
"import sys\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"\n",
"\"\"\"\n",
"If you are running this notebook in Google colab, make sure to upload the helpers.py file to your \n",
"session before running it, but if you are running this in Binder, then you \n",
"don't have to worry about it. The helpers.py file will be in the notebook\n",
"folder in GitHub.\n",
"\n",
"\"\"\"\n",
"from helpers import vector_plot, plot_transform"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "JBpOlp4jrAoy"
},
"source": [
"# 02.01 - Scalars, Vectors, Matrices and Tensors"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "kWXNG-cmHrih"
},
"source": [
"__Scalars:__ are just a single number. For example temperature, which is denoted by just one number.\n",
"\n",
"\n",
"__Vectors:__ are an array of numbers. The numbers are arranged in order and we can identify each individual number by its index in that ordering. We can think of vectors as identifying points in space, with each element giving the coordinate along a different axis. In simple terms, a vector is an arrow representing a quantity that has both magnitude and direction wherein the length of the arrow represents the magnitude and the orientation tells you the direction. For example wind, which has a direction and magnitude.\n",
"\n",
"__Matrices:__ A matrix is a 2D-array of numbers, so each element is identified by two indices instead of just one. If a real valued matrix $A$ has a height of *m* and a width of *n*, then we say that $A \\in \\mathbb{R}^{m \\times n}$. We identify the elements of the matrix as $A_{m,n}$ where *m* represents the row and *n* represents the column.\n",
"\n",
"![Scalars, Vectors, Matrices and Tensors](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0201a.png)\n",
"\n",
"__Tensors:__ In the general case, are an array of numbers arranged on a regular grid with a variable number of axes is knows as a tensor. We identify the elements of a tensor $A$ at coordinates(*i, j, k*) by writing $A_{i, j, k}$. But to truly understand tensors, we need to expand the way we think of vectors as only arrows with a magnitude and direction. Remember that a vector can be represented by three components, namely the x, y and z components (basis vectors). If you have a pen and a paper, let's do a small experiment, place the pen vertically on the paper and slant it by some angle and now shine a light from top such that the shadow of the pen falls on the paper, this shadow, represents the x component of the vector \"pen\" and the height from the paper to the tip of the pen is the y component. Now, let's take these components to describe tensors, imagine, you are Indiana Jones or a treasure hunter and you are trapped in a cube and there are three arrows flying towards you from the three faces (to represent x, y, z axis) of the cube 😬, I know this will be the last thing you would think in such a situation but you can think of those three arrows as vectors pointing towards you from the three faces of the cube and you can represent those vectors (arrows) in x, y and z components, now that is a rank 2 tensor (matrix) with 9 components. Remember that this is a very very simple explanation of tensors. Following is a representation of a tensor:\n",
"\n",
"![Tensors](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0201b.PNG)\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "HdIOKDSbNBaz"
},
"source": [
"We can add matrices to each other as long as they have the same shape, just by adding their corresponding elements:\n",
"\n",
"$$\\color{orange}{C = A + B \\ where \\ C_{i,j} = A_{i,j} + B_{i,j} \\tag{1}}$$\n",
"\n",
"\n",
"In tensorflow a:\n",
"\n",
"- Rank 0 Tensor is a Scalar\n",
"- Rank 1 Tensor is a Vector\n",
"- Rank 2 Tensor is a Matrix\n",
"- Rank 3 Tensor is a 3-Tensor\n",
"- Rank n Tensor is a n-Tensor"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 255
},
"colab_type": "code",
"id": "rKermF6KHs11",
"outputId": "9964b9e9-5832-4bab-8e0e-446bf52a2026"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3x3 Rank 2 Tensor A: \n",
"[[1. 1. 1.]\n",
" [1. 1. 1.]\n",
" [1. 1. 1.]]\n",
"\n",
"3x3 Rank 2 Tensor B: \n",
"[[1. 2. 3.]\n",
" [4. 5. 6.]\n",
" [7. 8. 9.]]\n",
"\n",
"Rank 2 Tensor C with shape=(3, 3) and elements: \n",
"[[ 2. 3. 4.]\n",
" [ 5. 6. 7.]\n",
" [ 8. 9. 10.]]\n"
]
}
],
"source": [
"# let's create a ones 3x3 rank 2 tensor\n",
"rank_2_tensor_A = tf.ones([3, 3], name='MatrixA')\n",
"print(\"3x3 Rank 2 Tensor A: \\n{}\\n\".format(rank_2_tensor_A))\n",
"\n",
"# let's manually create a 3x3 rank two tensor and specify the data type as float\n",
"rank_2_tensor_B = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]], name='MatrixB', dtype=tf.float32)\n",
"print(\"3x3 Rank 2 Tensor B: \\n{}\\n\".format(rank_2_tensor_B))\n",
"\n",
"# addition of the two tensors\n",
"rank_2_tensor_C = tf.add(rank_2_tensor_A, rank_2_tensor_B, name='MatrixC')\n",
"print(\"Rank 2 Tensor C with shape={} and elements: \\n{}\".format(rank_2_tensor_C.shape, rank_2_tensor_C))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 71
},
"colab_type": "code",
"id": "C1Fm6XkQaYcn",
"outputId": "6575ea04-ebce-4b73-8d92-15a885289850"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Incompatible shapes to add with two_by_three of shape (2, 3) and 3x3 Rank 2 Tensor B of shape (3, 3)\n",
" \n"
]
}
],
"source": [
"# Let's see what happens if the shapes are not the same\n",
"two_by_three = tf.ones([2, 3])\n",
"try:\n",
" incompatible_tensor = tf.add(two_by_three, rank_2_tensor_B)\n",
"except:\n",
" print(\"\"\"Incompatible shapes to add with two_by_three of shape {0} and 3x3 Rank 2 Tensor B of shape {1}\n",
" \"\"\".format(two_by_three.shape, rank_2_tensor_B.shape))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XuJ4rvIFNafT"
},
"source": [
"We can also add a scalar to a matrix or multiply a matrix by a scalar, just by performing that operation on each element of a matrix:\n",
"\n",
"$$\\color{orange}{D = a \\cdot B + c \\ where \\ D_{i,j} = a \\cdot B_{i,j} + c \\tag{2}}$$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 306
},
"colab_type": "code",
"id": "97sSpKcbTNoY",
"outputId": "5546ce85-9392-4fc5-f6b1-a2443ef25103"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Original Rank 2 Tensor B: \n",
"[[1. 2. 3.]\n",
" [4. 5. 6.]\n",
" [7. 8. 9.]] \n",
"\n",
"Scalar a: 2.0\n",
"Rank 2 Tensor for aB: \n",
"[[ 2. 4. 6.]\n",
" [ 8. 10. 12.]\n",
" [14. 16. 18.]] \n",
"\n",
"Scalar c: 3.0 \n",
"Rank 2 Tensor D = aB + c: \n",
"[[ 5. 7. 9.]\n",
" [11. 13. 15.]\n",
" [17. 19. 21.]]\n",
"\n"
]
}
],
"source": [
"# Create scalar a, c and Matrix B\n",
"rank_0_tensor_a = tf.constant(2, name=\"scalar_a\", dtype=tf.float32)\n",
"rank_2_tensor_B = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]], name='MatrixB', dtype=tf.float32)\n",
"rank_0_tensor_c = tf.constant(3, name=\"scalar_c\", dtype=tf.float32)\n",
"\n",
"# multiplying aB\n",
"multiply_scalar = tf.multiply(rank_0_tensor_a, rank_2_tensor_B)\n",
"# adding aB + c\n",
"rank_2_tensor_D = tf.add(multiply_scalar, rank_0_tensor_c, name=\"MatrixD\")\n",
"\n",
"print(\"\"\"Original Rank 2 Tensor B: \\n{0} \\n\\nScalar a: {1}\n",
"Rank 2 Tensor for aB: \\n{2} \\n\\nScalar c: {3} \\nRank 2 Tensor D = aB + c: \\n{4}\n",
"\"\"\".format(rank_2_tensor_B, rank_0_tensor_a, multiply_scalar, rank_0_tensor_c, rank_2_tensor_D))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5uI2sKntMyj6"
},
"source": [
"One important operation on matrices is the __transpose__. The transpose of a matrix is the mirror image of the martrix across a diagonal line, called the __main diagonal__. We denote the transpose of a matrix $A$ as $A^\\top$ and is defined as such: $(A^\\top)_{i, j} = A_{j, i}$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 153
},
"colab_type": "code",
"id": "pJwqOF1hzZNn",
"outputId": "8edb6277-2eb5-40cb-e71a-f1276402d493"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Rank 2 Tensor E of shape: (2, 3) and elements: \n",
"[[1 2 3]\n",
" [4 5 6]]\n",
"\n",
"Transpose of Rank 2 Tensor E of shape: (3, 2) and elements: \n",
"[[1 4]\n",
" [2 5]\n",
" [3 6]]\n"
]
}
],
"source": [
"# Creating a Matrix E\n",
"rank_2_tensor_E = tf.constant([[1, 2, 3], [4, 5, 6]])\n",
"# Transposing Matrix E\n",
"transpose_E = tf.transpose(rank_2_tensor_E, name=\"transposeE\")\n",
"\n",
"print(\"\"\"Rank 2 Tensor E of shape: {0} and elements: \\n{1}\\n\n",
"Transpose of Rank 2 Tensor E of shape: {2} and elements: \\n{3}\"\"\".format(rank_2_tensor_E.shape, rank_2_tensor_E, \n",
" transpose_E.shape, transpose_E))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "uU6F-99XM3_v"
},
"source": [
"In deep learning we allow the addition of matrix and a vector, yielding another matrix where $C_{i, j} = A_{i, j} + b_{j}$. In other words, the vector $b$ is added to each row of the matrix. This implicit copying of $b$ to many locations is called __broadcasting__"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 255
},
"colab_type": "code",
"id": "v3s5ZyQKUSY6",
"outputId": "9fa8acd7-910c-4a63-d538-a336e25ed839"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Rank 2 tensor A: \n",
"[[1. 1. 1.]\n",
" [1. 1. 1.]\n",
" [1. 1. 1.]]\n",
" \n",
"Rank 1 Tensor b: \n",
"[[4.]\n",
" [5.]\n",
" [6.]] \n",
"\n",
"Rank 2 tensor F = A + b:\n",
"[[5. 5. 5.]\n",
" [6. 6. 6.]\n",
" [7. 7. 7.]]\n"
]
}
],
"source": [
"# Creating a vector b\n",
"rank_1_tensor_b = tf.constant([[4.], [5.], [6.]])\n",
"# Broadcasting a vector b to a matrix A such that it yields a matrix F = A + b\n",
"rank_2_tensor_F = tf.add(rank_2_tensor_A, rank_1_tensor_b, name=\"broadcastF\")\n",
"\n",
"print(\"\"\"Rank 2 tensor A: \\n{0}\\n \\nRank 1 Tensor b: \\n{1} \n",
"\\nRank 2 tensor F = A + b:\\n{2}\"\"\".format(rank_2_tensor_A, rank_1_tensor_b, rank_2_tensor_F))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ziVxUA2WHyf4"
},
"source": [
"# 02.02 - Multiplying Matrices and Vectors"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "MzuhfYTyOR9N"
},
"source": [
"To define the matrix product of matrices $A \\ \\text{and} \\ B, \\ A$ must have the same number of columns as $B$. If $A$ is of shape *m x n* and $B$ is of shape *n x p*, then $C$ is of shape *m x p*.\n",
"\n",
"$$\\color{orange}{C_{i, j} = \\displaystyle\\sum_k A_{i, k} B_{k, j} \\tag{3}}$$\n",
"\n",
"If you do not recall how matrix multiplication is performed, take a look at:\n",
"\n",
"![Multiplying Matrices](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0202a.jpg)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"id": "Rrv8DzWuHzWR",
"outputId": "977d5ffd-3709-4924-d07e-459fed5ddb28"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: shape (2, 3) \n",
"elements: \n",
"[[1. 1. 1.]\n",
" [1. 1. 1.]] \n",
"\n",
"Matrix B: shape (3, 4) \n",
"elements: \n",
"[[1. 2. 3. 4.]\n",
" [1. 2. 3. 4.]\n",
" [1. 2. 3. 4.]]\n",
"\n",
"Matrix C: shape (2, 4) \n",
"elements: \n",
"[[ 3. 6. 9. 12.]\n",
" [ 3. 6. 9. 12.]]\n"
]
}
],
"source": [
"# Matrix A and B with shapes (2, 3) and (3, 4)\n",
"mmv_matrix_A = tf.ones([2, 3], name=\"matrix_A\")\n",
"mmv_matrix_B = tf.constant([[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]], name=\"matrix_B\", dtype=tf.float32)\n",
"\n",
"# Matrix Multiplication: C = AB with C shape (2, 4)\n",
"matrix_multiply_C = tf.matmul(mmv_matrix_A, mmv_matrix_B, name=\"matrix_multiply_C\")\n",
"\n",
"print(\"\"\"Matrix A: shape {0} \\nelements: \\n{1} \\n\\nMatrix B: shape {2} \\nelements: \\n{3}\n",
"\\nMatrix C: shape {4} \\nelements: \\n{5}\"\"\".format(mmv_matrix_A.shape, mmv_matrix_A, mmv_matrix_B.shape, \n",
" mmv_matrix_B, matrix_multiply_C.shape, matrix_multiply_C))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "pz_vETHqOXpL"
},
"source": [
"To get a matrix containing the product of the individual elements, we use __element wise product__ or __Hadamard product__ and is denoted as $A \\odot B$.\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 306
},
"colab_type": "code",
"id": "mvYdsHcoahkl",
"outputId": "b72763c3-2623-41c9-c733-838ec65845af"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: shape (3, 3) \n",
"elements: \n",
"[[1. 1. 1.]\n",
" [1. 1. 1.]\n",
" [1. 1. 1.]] \n",
"\n",
"Matrix A: shape (3, 3) \n",
"elements: \n",
"[[1. 2. 3.]\n",
" [4. 5. 6.]\n",
" [7. 8. 9.]]\n",
"\n",
"Matrix C: shape (3, 3) \n",
"elements: \n",
"[[1. 2. 3.]\n",
" [4. 5. 6.]\n",
" [7. 8. 9.]]\n"
]
}
],
"source": [
"\"\"\"\n",
"Note that we use multiply to do element wise matrix multiplication and matmul\n",
"to do matrix multiplication\n",
"\"\"\"\n",
"# Creating new Matrix A and B with shapes (3, 3)\n",
"element_matrix_A = tf.ones([3, 3], name=\"element_matrix_A\")\n",
"element_matrix_B = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]], name=\"element_matrix_B\", dtype=tf.float32)\n",
"\n",
"# Element wise multiplication of Matrix A and B\n",
"element_wise_C = tf.multiply(element_matrix_A, element_matrix_B, name=\"element_wise_C\")\n",
"\n",
"print(\"\"\"Matrix A: shape {0} \\nelements: \\n{1} \\n\\nMatrix A: shape {2} \\nelements: \\n{3}\\n\n",
"Matrix C: shape {4} \\nelements: \\n{5}\"\"\".format(element_matrix_A.shape, element_matrix_A, element_matrix_B.shape, \n",
" element_matrix_B, element_wise_C.shape, element_wise_C))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "qSDm2ptGObdH"
},
"source": [
"To compute the dot product between $A$ and $B$ we compute $C_{i, j}$ as the dot product between row *i* of $A$ and column *j* of $B$.\n",
"\n",
"![Dot Product](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0202b.jpg)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 306
},
"colab_type": "code",
"id": "gLAOBe-DahA3",
"outputId": "6ae165de-a8ee-43ed-b181-d6bca0a6ab73"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: shape (3, 3) \n",
"elements: \n",
"[[1. 1. 1.]\n",
" [1. 1. 1.]\n",
" [1. 1. 1.]] \n",
"\n",
"Matrix B: shape (3, 3) \n",
"elements: \n",
"[[1. 2. 3.]\n",
" [4. 5. 6.]\n",
" [7. 8. 9.]]\n",
"\n",
"Matrix C: shape (3, 3) \n",
"elements: \n",
"[[12. 15. 18.]\n",
" [12. 15. 18.]\n",
" [12. 15. 18.]]\n"
]
}
],
"source": [
"# Creating Matrix A and B with shapes (3, 3)\n",
"dot_matrix_A = tf.ones([3, 3], name=\"dot_matrix_A\")\n",
"dot_matrix_B = tf.constant([[1, 2, 3], [4, 5, 6], [7, 8, 9]], name=\"dot_matrix_B\", dtype=tf.float32)\n",
"\n",
"# Dot product of A and B\n",
"dot_product_C = tf.tensordot(dot_matrix_A, dot_matrix_B, axes=1, name=\"dot_product_C\")\n",
"\n",
"print(\"\"\"Matrix A: shape {0} \\nelements: \\n{1} \\n\\nMatrix B: shape {2} \\nelements: \\n{3}\\n\n",
"Matrix C: shape {4} \\nelements: \\n{5}\"\"\".format(dot_matrix_A.shape, dot_matrix_A, dot_matrix_B.shape, \n",
" dot_matrix_B, dot_product_C.shape, dot_product_C))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "6jzMm9CjOgrR"
},
"source": [
"\n",
"Some properties of matrix multiplication (Distributive property):\n",
"\n",
"$$\\color{orange}{A(B +C) = AB + AC \\tag{4}}$$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "2a3T93aNlEcr"
},
"outputs": [],
"source": [
"# Common Matrices to check all the matrix Properties\n",
"matrix_A = tf.constant([[1, 2], [3, 4]], name=\"matrix_a\")\n",
"matrix_B = tf.constant([[5, 6], [7, 8]], name=\"matrix_b\")\n",
"matrix_C = tf.constant([[9, 1], [2, 3]], name=\"matrix_c\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 357
},
"colab_type": "code",
"id": "QTryvCNYcDSD",
"outputId": "0b9bb5fc-d4de-4e77-cbb8-5d651bc6457d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: \n",
"[[1 2]\n",
" [3 4]] \n",
"\n",
"Matrix B: \n",
"[[5 6]\n",
" [7 8]] \n",
"\n",
"Matrix C: \n",
"[[9 1]\n",
" [2 3]]\n",
"\n",
"Distributive property is valid\n",
"RHS: AB + AC: \n",
"[[32 29]\n",
" [78 65]] \n",
"\n",
"LHS: A(B+C): \n",
"[[32 29]\n",
" [78 65]]\n"
]
}
],
"source": [
"# Distributive Property\n",
"print(\"Matrix A: \\n{} \\n\\nMatrix B: \\n{} \\n\\nMatrix C: \\n{}\\n\".format(matrix_A, matrix_B, matrix_C))\n",
"\n",
"# AB + AC\n",
"distributive_RHS = tf.add(tf.matmul(matrix_A, matrix_B), tf.matmul(matrix_A, matrix_C), name=\"RHS\")\n",
"\n",
"# A(B+C)\n",
"distributive_LHS = tf.matmul(matrix_A, (tf.add(matrix_B, matrix_C)), name=\"LHS\")\n",
"\n",
"\"\"\"\n",
"Following is another way a conditional statement can be implemented from tensorflow\n",
"This might not seem very useful now but I want to introduce it here so you can\n",
"figure out how it works for a simple example.\n",
"\"\"\"\n",
"# To compare each element in the matrix, you need to reduce it first and check if it's equal\n",
"predictor = tf.reduce_all(tf.equal(distributive_RHS, distributive_LHS))\n",
"\n",
"# condition to act on if predictor is True\n",
"def true_print(): print(\"\"\"Distributive property is valid\n",
"RHS: AB + AC: \\n{} \\n\\nLHS: A(B+C): \\n{}\"\"\".format(distributive_RHS, distributive_LHS))\n",
" \n",
"# condition to act on if predictor is False \n",
"def false_print(): print(\"\"\"You Broke the Distributive Property of Matrix\n",
"RHS: AB + AC: \\n{} \\n\\nis NOT Equal to LHS: A(B+C): \\n{}\"\"\".format(distributive_RHS, distributive_LHS))\n",
" \n",
"tf.cond(predictor, true_print, false_print)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "TkSyrN_uOkUC"
},
"source": [
"Some properties of matrix multiplication (Associative property):\n",
"\n",
"$$\\color{orange}{A(BC) = (AB)C \\tag{5}}$$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 357
},
"colab_type": "code",
"id": "jV9y4WC8cDJ5",
"outputId": "c350ff88-e589-4ece-e707-9d70f5bd5175"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: \n",
"[[1 2]\n",
" [3 4]] \n",
"\n",
"Matrix B: \n",
"[[5 6]\n",
" [7 8]] \n",
"\n",
"Matrix C: \n",
"[[9 1]\n",
" [2 3]]\n",
"\n",
"Associative property is valid\n",
"RHS: (AB)C: \n",
"[[215 85]\n",
" [487 193]] \n",
"\n",
"LHS: A(BC): \n",
"[[215 85]\n",
" [487 193]]\n"
]
}
],
"source": [
"# Associative property\n",
"print(\"Matrix A: \\n{} \\n\\nMatrix B: \\n{} \\n\\nMatrix C: \\n{}\\n\".format(matrix_A, matrix_B, matrix_C))\n",
"\n",
"# (AB)C\n",
"associative_RHS = tf.matmul(tf.matmul(matrix_A, matrix_B), matrix_C)\n",
"\n",
"# A(BC)\n",
"associative_LHS = tf.matmul(matrix_A, tf.matmul(matrix_B, matrix_C))\n",
"\n",
"# To compare each element in the matrix, you need to reduce it first and check if it's equal\n",
"predictor = tf.reduce_all(tf.equal(associative_RHS, associative_LHS))\n",
"\n",
"# condition to act on if predictor is True\n",
"def true_print(): print(\"\"\"Associative property is valid\n",
"RHS: (AB)C: \\n{} \\n\\nLHS: A(BC): \\n{}\"\"\".format(associative_RHS, associative_LHS))\n",
"\n",
"# condition to act on if predictor is False \n",
"def false_print(): print(\"\"\"You Broke the Associative Property of Matrix\n",
"RHS: (AB)C: \\n{} \\n\\nLHS: A(BC): \\n{}\"\"\".format(associative_RHS, associative_LHS))\n",
" \n",
"tf.cond(predictor, true_print, false_print)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "hOiY8XZtOoap"
},
"source": [
"Some properties of matrix multiplication (Matrix multiplication is not commutative):\n",
"\n",
"$$\\color{orange}{AB \\neq BA \\tag{6}}$$ "
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 289
},
"colab_type": "code",
"id": "6Pv8SOxbcC-U",
"outputId": "5414ccad-1e67-4813-eb76-56407c93ee32"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: \n",
"[[1 2]\n",
" [3 4]] \n",
"\n",
"Matrix B: \n",
"[[5 6]\n",
" [7 8]]\n",
"\n",
"Matrix Multiplication is not commutative\n",
"RHS: (AB): \n",
"[[19 22]\n",
" [43 50]] \n",
"\n",
"LHS: (BA): \n",
"[[23 34]\n",
" [31 46]]\n"
]
}
],
"source": [
"# Matrix multiplication is not commutative\n",
"print(\"Matrix A: \\n{} \\n\\nMatrix B: \\n{}\\n\".format(matrix_A, matrix_B))\n",
"\n",
"# Matrix A times B\n",
"commutative_RHS = tf.matmul(matrix_A, matrix_B)\n",
"\n",
"# Matrix B times A\n",
"commutative_LHS = tf.matmul(matrix_B, matrix_A)\n",
"\n",
"predictor = tf.logical_not(tf.reduce_all(tf.equal(commutative_RHS, commutative_LHS)))\n",
"def true_print(): print(\"\"\"Matrix Multiplication is not commutative\n",
"RHS: (AB): \\n{} \\n\\nLHS: (BA): \\n{}\"\"\".format(commutative_RHS, commutative_LHS))\n",
"\n",
"def false_print(): print(\"\"\"You made Matrix Multiplication commutative\n",
"RHS: (AB): \\n{} \\n\\nLHS: (BA): \\n{}\"\"\".format(commutative_RHS, commutative_LHS))\n",
" \n",
"tf.cond(predictor, true_print, false_print)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "j1Voc_AgOtIM"
},
"source": [
"Some properties of matrix multiplication (Transpose):\n",
"\n",
"$$\\color{orange}{(AB)^\\top = B^{\\top} A^{\\top} \\tag{7}}$$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 289
},
"colab_type": "code",
"id": "Gz7cm3obcCZe",
"outputId": "60e4938e-349e-44ac-ed60-fa43bd0fd364"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: \n",
"[[1 2]\n",
" [3 4]] \n",
"\n",
"Matrix B: \n",
"[[5 6]\n",
" [7 8]]\n",
"\n",
"Transpose property is valid\n",
"RHS: (AB):^T \n",
"[[19 43]\n",
" [22 50]] \n",
"\n",
"LHS: (B^T A^T): \n",
"[[19 43]\n",
" [22 50]]\n"
]
}
],
"source": [
"# Transpose of a matrix\n",
"print(\"Matrix A: \\n{} \\n\\nMatrix B: \\n{}\\n\".format(matrix_A, matrix_B))\n",
"\n",
"# Tensorflow transpose function\n",
"transpose_RHS = tf.transpose(tf.matmul(matrix_A, matrix_B))\n",
"\n",
"# If you are doing matrix multiplication tf.matmul has a parameter to take the tranpose and then matrix multiply\n",
"transpose_LHS = tf.matmul(matrix_B, matrix_A, transpose_a=True, transpose_b=True)\n",
" \n",
"predictor = tf.reduce_all(tf.equal(transpose_RHS, transpose_LHS))\n",
"def true_print(): print(\"\"\"Transpose property is valid\n",
"RHS: (AB):^T \\n{} \\n\\nLHS: (B^T A^T): \\n{}\"\"\".format(transpose_RHS, transpose_LHS))\n",
"\n",
"def false_print(): print(\"\"\"You Broke the Transpose Property of Matrix\n",
"RHS: (AB):^T \\n{} \\n\\nLHS: (B^T A^T): \\n{}\"\"\".format(transpose_RHS, transpose_LHS))\n",
" \n",
"tf.cond(predictor, true_print, false_print)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "sltlInVEH1aU"
},
"source": [
"# 02.03 - Identity and Inverse Matrices"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Xg8Y9m_N3miE"
},
"source": [
"Linear algebra offers a powerful tool called __matrix inversion__ that enables us to analytically solve $Ax = b$ for many values of $A$.\n",
"\n",
"To describe matrix inversion, we first need to define the concept of an __identity matrix__. An identity matrix is a matrix that does not change any vector when we multiply that vector by that matrix. \n",
"\n",
"Such that:\n",
"\n",
"$$\\color{orange}{I_n \\in \\mathbb{R}^{n \\times n} \\ \\text{and} \\ \\forall x \\in \\mathbb{R}^n, I_n x = x \\tag{8}}$$\n",
"\n",
"The structure of the identity matrix is simple: all the entries along the main diagonal are 1, while all the other entries are zero."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 255
},
"colab_type": "code",
"id": "hDED6gFiH4U9",
"outputId": "c0c6ae20-c832-4431-dcbd-f859c7fa4994"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Identity matrix I: \n",
"[[1. 0. 0.]\n",
" [0. 1. 0.]\n",
" [0. 0. 1.]]\n",
"\n",
"Vector x: \n",
"[[4.]\n",
" [5.]\n",
" [6.]]\n",
"\n",
"Matrix C from Ix: \n",
"[[4.]\n",
" [5.]\n",
" [6.]]\n"
]
}
],
"source": [
"# let's create a identity matrix I\n",
"identity_matrix_I = tf.eye(3, 3, dtype=tf.float32, name='IdentityMatrixI')\n",
"print(\"Identity matrix I: \\n{}\\n\".format(identity_matrix_I))\n",
"\n",
"# let's create a 3x1 vector x\n",
"iim_vector_x = tf.constant([[4], [5], [6]], name='Vector_x', dtype=tf.float32)\n",
"print(\"Vector x: \\n{}\\n\".format(iim_vector_x))\n",
"\n",
"# Ix will result in x\n",
"iim_matrix_C = tf.matmul(identity_matrix_I, iim_vector_x, name='MatrixC')\n",
"print(\"Matrix C from Ix: \\n{}\".format(iim_matrix_C))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "44tX7v9F4Lbv"
},
"source": [
"The __matrix inverse__ of $A$ is denoted as $A^{-1}$, and it is defined as the matrix such that:\n",
"\n",
"$$\\color{orange}{A^{-1} A = I_n \\tag{9}}$$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 289
},
"colab_type": "code",
"id": "f6RR6gQB9bTM",
"outputId": "9d83b6df-27c9-4268-fed0-f28ada5972f7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A^-1 times A equals the Identity Matrix\n",
"Matrix A: \n",
"[[2. 3.]\n",
" [2. 2.]] \n",
"\n",
"Inverse of Matrix A: \n",
"[[-1. 1.5]\n",
" [ 1. -1. ]] \n",
"\n",
"RHS: I: \n",
"[[1. 0.]\n",
" [0. 1.]] \n",
"\n",
"LHS: A^(-1) A: \n",
"[[1. 0.]\n",
" [0. 1.]]\n"
]
}
],
"source": [
"iim_matrix_A = tf.constant([[2, 3], [2, 2]], name='MatrixA', dtype=tf.float32)\n",
"\n",
"try:\n",
" # Tensorflow function to take the inverse\n",
" inverse_matrix_A = tf.linalg.inv(iim_matrix_A)\n",
" \n",
" # Creating a identity matrix using tf.eye\n",
" identity_matrix = tf.eye(2, 2, dtype=tf.float32, name=\"identity\")\n",
"\n",
" iim_RHS = identity_matrix\n",
" iim_LHS = tf.matmul(inverse_matrix_A, iim_matrix_A, name=\"LHS\")\n",
" \n",
" predictor = tf.reduce_all(tf.equal(iim_RHS, iim_LHS))\n",
" def true_print(): print(\"\"\"A^-1 times A equals the Identity Matrix\n",
"Matrix A: \\n{0} \\n\\nInverse of Matrix A: \\n{1} \\n\\nRHS: I: \\n{2} \\n\\nLHS: A^(-1) A: \\n{3}\"\"\".format(iim_matrix_A,\n",
" inverse_matrix_A,\n",
" iim_RHS, \n",
" iim_LHS))\n",
" def false_print(): print(\"Condition Failed\")\n",
" tf.cond(predictor, true_print, false_print)\n",
" \n",
"except:\n",
" print(\"\"\"A^-1 doesnt exist\n",
" Matrix A: \\n{} \\n\\nInverse of Matrix A: \\n{} \\n\\nRHS: I: \\n{} \n",
" \\nLHS: (A^(-1) A): \\n{}\"\"\".format(iim_matrix_A, inverse_matrix_A, iim_RHS, iim_LHS))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1CAGKbsr9yZv"
},
"source": [
"If you try different values for Matrix A, you will see that, not all $A$ has an inverse and we will discuss the conditions for the existence of $A^{-1}$ in the following section.\n",
"\n",
"We can then solve the equation $Ax = b$ as:\n",
"\n",
"$$\n",
"\\color{Orange}{A^{-1} Ax = A^{-1} b} \\\\\n",
"\\color{Orange}{I_n x = A^{-1} b} \\\\\n",
"\\color{Orange}{x =A^{-1} b \\tag{10}}\n",
"$$\n",
"\n",
"This process depends on it being possible to find $A^{-1}$. \n",
"\n",
"We can calculate the inverse of a matrix by:\n",
"\n",
"![Matrix Inverse](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0203a.PNG)\n",
"\n",
"Lets see how we can solve a simple linear equation: 2x + 3y = 6 and 4x + 9y = 15"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 221
},
"colab_type": "code",
"id": "sizO9Oz66KpD",
"outputId": "75428b1c-3b0f-476e-a521-7d6baf988d47"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A: \n",
"[[2. 3.]\n",
" [4. 9.]] \n",
"\n",
"Vector B: \n",
"[[ 6.]\n",
" [15.]]\n",
"\n",
"Vector x is: \n",
"[[1.5]\n",
" [1. ]] \n",
"Where x = [1.5] and y = [1.]\n"
]
}
],
"source": [
"# The above system of equation can be written in the matrix format as:\n",
"sys_matrix_A = tf.constant([[2, 3], [4, 9]], dtype=tf.float32)\n",
"sys_vector_B = tf.constant([[6], [15]], dtype=tf.float32)\n",
"print(\"Matrix A: \\n{} \\n\\nVector B: \\n{}\\n\".format(sys_matrix_A, sys_vector_B))\n",
"\n",
"# now to solve for x: x = A^(-1)b\n",
"sys_x = tf.matmul(tf.linalg.inv(sys_matrix_A), sys_vector_B)\n",
"print(\"Vector x is: \\n{} \\nWhere x = {} and y = {}\".format(sys_x, sys_x[0], sys_x[1]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "CrDWCbU5H6ym"
},
"source": [
"# 02.04 - Linear Dependence and Span"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "3J_B7nao9De_"
},
"source": [
"For $A^{-1}$ to exits, $Ax = b$ must have exactly one solution for every value of $b$. It is also possible for the system of equations to have no solutions or infinitely many solutions for some values of $b$. This is simply because we are dealing with linear systems and two lines can't cross more than once. So, they can either cross once, cross never, or have infinite crossing, meaning the two lines are superimposed.\n",
"\n",
"Hence if both $x$ and $y$ are solutions then:\n",
"\n",
"$z = \\alpha x + (1 - \\alpha)y$ is also a solution for any real $\\alpha$\n",
"\n",
"![Linear Dependence](https://raw.githubusercontent.com/adhiraiyan/DeepLearningWithTF2.0/master/notebooks/figures/fig0204a.png)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "yhPzsL6JEHAu"
},
"source": [
"\n",
"The span of a set of vectors is the set of all linear combinations of the vectors. Formally, a __linear combination__ of some set of vectors \n",
"$\\{ v^1, \\cdots, v^n\\}$ is given by multiplying each vecor $v^{(i)}$ by a corresponding scalar coefficient and adding the results:\n",
"\n",
"$$\\color{Orange}{\\displaystyle\\sum_i c_i v^{(i)} \\tag{11}}$$\n",
"\n",
"Determining whether $Ax = b$ has a solution thus amounts to testing whether $b$ is in the span of the columns of $A$. This particular span is known as the __column space__ or the __range__, of $A$. \n",
"\n",
"In order for the system $Ax = b$ to have a solution for all values of $b \\in \\mathbb{R}^m$, we require that the column space of $A$ be all of $\\mathbb{R}^m$.\n",
"\n",
"A set of vectors $\\{ v^1, \\cdots, v^n\\}$ is __linearly independent__ if the only solution to the vector equation $\\lambda_1 v^1 + \\cdots \\lambda_n v^n = 0 \\ \\text{is} \\ \\lambda_i=0 \\ \\forall i$. If a set of vectors is not linearly independent, then it is __linearly dependent__.\n",
"\n",
"For the matrix to have an inverse, the matrix must be __square__, that is, we require that *m = n* and that all the columns be linearly independent. A square matrix with linearly dependent columns is known as __singular__.\n",
"\n",
"If $A$ is not square or is square but singular, solving the equation is still possible, but we cannot use the method of matrix inversion to find the solution.\n",
"\n",
"So far we have discussed matrix inverses as being multiplied on the left. It is also possible to define an inverse that is multiplied on the right. For square matrixes, the left inverse and right inverse are equal."
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 272
},
"colab_type": "code",
"id": "QsZ8dcQgWc6M",
"outputId": "048b082a-109a-44da-daa6-6d850329b11d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matrix A is successfully inverted: \n",
"[[ 0.40000004 -0.20000002]\n",
" [-0.20000002 0.6 ]]\n",
"\n",
"The two x values match, we proved that if a matrix A is invertible\n",
"Then x = A^T b, \n",
"where x: [2. 3.], \n",
"\n",
"A^T: \n",
"[[ 0.40000004 -0.20000002]\n",
" [-0.20000002 0.6 ]], \n",
"\n",
"b: \n",
"[[6. 3.]\n",
" [2. 6.]]\n"
]
}
],
"source": [
"# Lets start by finding for some value of A and x, what the result of x is\n",
"lds_matrix_A = tf.constant([[3, 1], [1, 2]], name='MatrixA', dtype=tf.float32)\n",
"lds_vector_x = tf.constant([2, 3], name='vectorX', dtype=tf.float32)\n",
"lds_b = tf.multiply(lds_matrix_A, lds_vector_x, name=\"b\")\n",
"\n",
"# Now let's see if an inverse for Matrix A exists\n",
"try:\n",
" inverse_A = tf.linalg.inv(lds_matrix_A)\n",
" print(\"Matrix A is successfully inverted: \\n{}\".format(inverse_A))\n",
"except:\n",
" print(\"Inverse of Matrix A: \\n{} \\ndoesn't exist. \".format(lds_matrix_A))\n",
"\n",
"# Let's find the value of x using x = A^(-1)b\n",
"verify_x = tf.matmul(inverse_A, lds_b, name=\"verifyX\")\n",
"predictor = tf.equal(lds_vector_x[0], verify_x[0][0]) and tf.equal(lds_vector_x[1], verify_x[1][1])\n",
"\n",
"def true_print(): print(\"\"\"\\nThe two x values match, we proved that if a matrix A is invertible\n",
"Then x = A^T b, \\nwhere x: {}, \\n\\nA^T: \\n{}, \\n\\nb: \\n{}\"\"\".format(lds_vector_x, inverse_A, lds_b))\n",
"\n",
"def false_print(): print(\"\"\"\\nThe two x values don't match.\n",
"Vector x: {} \\n\\nA^(-1)b: \\n{}\"\"\".format(lds_vector_x, verify_x))\n",
" \n",
"tf.cond(predictor, true_print, false_print)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GzrynyTagib-"
},
"source": [
"Note that, finding inverses can be a challenging process if you want to calculate it, but using tensorflow or any other library, you can easily check if the inverse of the matrix exists. If you know the conditions and know how to solve matrix equations using tensorflow, you should be good, but for the reader who wants to go deeper, check [Linear Dependence and Span\n",
"](https://math.ryerson.ca/~danziger/professor/MTH141/Handouts/depend.pdf) for further examples and definitions."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "rB0seiU_H-hx"
},
"source": [
"# 02.05 - Norms"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "p3ieNjy4oHtr"
},
"source": [
"In machine learning if we need to measure the size of vectors, we use a function called a __norm__. And norm is what is generally used to evaluate the error of a model. Formally, the $L^P$ norm is given by:\n",
"\n",
"$$\\color{Orange}{||x||_p = \\big(\\displaystyle\\sum_i |x_i|^p\\big)^{1/p}| \\tag{12}}$$\n",
"\n",
"for $p \\in \\mathbb{R}, p \\geq 1$\n",
"\n",
"On an intuitive level, the norm of a vector $x$ measures the distance from the origin to the point $x$.\n",
"\n",
"More rigorously, a norm is any function $f$ that satisfies the following properties:\n",
"\n",
"$$\n",
"\\color{Orange}{f(x) = 0 \\implies x=0} \\\\\n",
"\\color{Orange}{f(x +y) \\leq f(x) + f(y)} \\\\\n",
"\\color{Orange}{\\forall \\alpha \\in \\mathbb{R}, f(\\alpha x) = |\\alpha|f(x) \\tag{13}}\n",
"$$\n",
"\n",
"The $L^2$ norm with *p=2* is known as the __Euclidean norm__. Which is simply the Euclidean distance from the origin to the point identified by $x$. It is also common to measure the size of a vector using the squared $L^2$ norm, which can be calculated simply as $x^{\\top}x$"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
},
"colab_type": "code",
"id": "1DciOj3Hs2g-",
"outputId": "4b084d09-1d58-4a6f-a923-ba1389d3a272"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Euclidean Distance: 5.0\n",
"Vector Size: [ 9. 16.]\n"
]
}
],
"source": [
"# Euclidean distance between square root(3^2 + 4^2) calculated by setting ord='euclidean'\n",
"dist_euclidean = tf.norm([3., 4.], ord='euclidean')\n",
"print(\"Euclidean Distance: {}\".format(dist_euclidean))\n",
"\n",
"# Size of the vector [3., 4.]\n",
"vector_size = tf.multiply(tf.transpose([3., 4.]), [3., 4.])\n",
"print(\"Vector Size: {}\".format(vector_size))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1_Zg24-vs25u"
},
"source": [
"In many contexts, the squared $L^2$ norm may be undesirable, because it increases very slowly near the origin. In many machine learning applications, it is important to discriminate between elements that are exactly zero and elements that are small but nonzero. In these cases, we turn to a function that grows at the same rate in all locations, but retains mathematical simplicity: the $L^1$ norm, which can be simplified to:\n",
"\n",
"$$\\color{Orange}{||x||_1 = \\displaystyle\\sum_i |x_i| \\tag{14}}$$\n"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 281
},
"colab_type": "code",
"id": "OlkUawi5wgBn",
"outputId": "e622e6db-8fdf-4969-8cdf-5ee412371828"
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEICAYAAAC0+DhzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XdcVtUfwPHPZYM4AScKIiDIcmtW\nZu6999YyLW2XWvZLLdtDs/JXZo7MvZKcP9MclQtxAIKCioqKoqjIkvGc3x/3aqgg8Awexnm/Xs/L\n57nj3O8FfL73nnPuOYoQAkmSJEnKycLcAUiSJEnFj0wOkiRJ0iNkcpAkSZIeIZODJEmS9AiZHCRJ\nkqRHyOQgSZIkPUImh7JrBvCruYPQUyzQ3khl9QEuAslAIyOVmZ/FwCwjlVUfOAbcAV4p4D4C8DTS\n8aVSSiaH0is5x0sHpOX4PMyMcRU3XwKTAEfgqJlj0cdk4E+gPDA3l/W7geeLMiCpdJDJofRyzPG6\nAPTI8XmZGeOyMuOxc+MGRJg7CAOU9PjvKW5/F2WeTA5lmw3wC2qVRATQNMe6msA6IAE4x+OrLJyA\n34Ek4DBqlclfOdYLYCIQrb0AvkGtzkkCjgBP59h+BrAWWKXFFgoEPXTMhsAJ4La2nV0esVkA7wHn\ngWva+VYEbFHvoiyB48CZPPb3AXYAicApYGCOdd1Q7zaStHOZ8dC+TwH/ALe09aNzrKsMbNbO7yBQ\nL4/jA/RE/f3cQr0T8NWW7wKeBb7TzsX7of0+Qv253lv/XY517VF/F7eA7wElx7qxQCRwE9iOmoBy\n4476ux2FegFyHZiWY70tMAe4rL3maMsA2gBxwBQgHliUY9lk1N/VFaA30BU4jfo7eDdH+c2BENSf\n/1Xg6zzilPQhhJCv0v+KFUK0f2jZDCFEuhCiqxDCUgjxiRDigLbOQghxRAjxvhDCRgjhIYQ4K4To\nlEf5K7WXgxCigRDiohDirxzrhRBihxCiihDCXls2XAjhJISwEkK8KYSIF0LY5YgtUwjRXwhhLYR4\nSwhxTnt/73wOCSFqamVGCiEm5BHbWCFEjHYOjkKI9UKIpQ/F5pnHvuW0cxmjxdlICHFdO0eEEG2E\nEAHazytQCHFVCNFbW+cmhLgjhBiixe0khGiorVsshLghhGiulbtM+/nlFoO3ECJFCNFBK2eydj42\n2vrdQojn89g3r/VCCLFJCFFJCFFHCJEghOisreulle+rxfaeEOKfPMp218r6Sai/1yAhxF1tX4QQ\nHwj1b6qqEMJFK+fDHD+7LCHEZ0IIW23/e8ve1851nBbbciFEeSGEnxAiTQhRVytjvxBihPbeUQjR\n8jE/B/kq5MvsAchXkbxiRe7J4Y8cnxsI9T8eQogWQogLD23/jhBiUS5lWwr1i7x+jmWzxKPJoW0+\nMd4U6pfLvdgO5FhnIYS4IoR4Osf5DM+x/nMhxA95lLtTCPFSjs/1tXitcsSWV3IYJITY99CyH4UQ\n0/PYfo4QYnaOn9eGPLZbLIRYkONzVyFEVB7b/kcIsfqhn8UloX6RIvRPDk/l+LxaCDFVe79VCPHc\nQ8dLFWqye7hsd60s1xzLDgkhBmvvz2jndm9dJ+13hxZ/hvj3guDesjSh/k0h1IQghPr3eG+bI+Lf\nBLxXCDFTCOH8mPOXLz1fslqpbIvP8T4VtWrGCrUaoSZqlcO917tAtVzKcNH2uZhj2cVctnt42Vuo\nVRe3tfIrAs55bK9DrW6o+ZjYHXM5Jto+53N8Pq/Fm9u5PMwNaMGDP4dhQHVtfQvUxuAE7Twm5DiH\n2uRdVWVI/DrUn02tAsT/OHkd3w21yu/e+SaiVjk97nh5lZXbzz7n7zABSH+orBtAtvY+Tfv3ao71\naTnKfw61Ki0KtTqz+2NilApJNgJJubmI2s7gVYBtE4AswBW1XhjUL8aH5Rz+92nUeuV2qHXpOtT6\n7Zz13jnLsNDKv1yAeB52mQfrzOto8V7NffMHXAT2AB3yWL8ctR6/C+qX3Bz+TQ4XUevEDXUZCMjx\nWUH92Vwq4P6FHXb5ImpbhTE6Ldz72d9rMK/Dg79DQ4eEjgaGoP599EVtp3ICUgwsV0I2SEu5O4Ta\nUDoFsEdttPUHmuWybTawHrUx1gG1AXdkPuWXR/2CTkC9QHkfqPDQNk1Q/8NbAa8Bd4EDhT4TWAG8\nDtRFveL8GLUBO6sA+25CvTIdAVhrr2b82yBcHvXKOh01EQzNse8y1Ebfgdo5OKE2ohfWatSG73ba\n8d9E/Vn8U8D9rwIehTjeD8A7gJ/2uSIwoBD757QCtTOAC2rSfB/jPlszXCtbh3qXg/ZeMgKZHKTc\nZKPeojdEvYO4DixA/aLIzSRtXTywFPVL4e5jyt8ObEO90ziP+uX6cLXTRmAQ6h3FCNREkVn4U2Gh\nFtNe1HNJB14u4L53gI7AYNQr3njgM/7tcfMS8IG23fuoX+T3XEDtZfMmagI5xqM9rgriFOqX4Leo\nv4ce2iujgPt/A/RH/Tnm9hzEwzagnuNK1F5A4ah3RvqYhdqb6AQQhtrrzFgP/wF0Rr0rSUY9z8H8\nWxUlGUgRQk72IxndZ6j18qP03H8G6hO8w40VkCRJhSPvHCRj8AECUevDm6M2FG4wa0SSJBlENkhL\nxlAetSqpJmod91eo1UKSJJVQslpJkiRJeoSsVpIkSZIeUXKrldY5C8q5mzsKSZKkkiPxyHWGCpeC\nbFpyk0M5d+gcYu4oJEmSSo7lyvn8N1LJaiVJkiTpETI5SJIkSY+QyUGSJEl6RMltc5CKvczMTOLi\n4khPf3jgTam0sbOzw9XVFWtra3OHIhmJTA6SycTFxVG+fHnc3d1RFCX/HaQSSQjBjRs3iIuLo27d\nuuYORzKS/KuVFGUhinINRQnPsawKirIDRYnW/q2sLVdQlLkoSgyKcgJFaZxjn1Ha9tEoyqgcy5ug\nKGHaPnPlt0jpkZ6ejpOTk/yVlnKKouDk5CTvEEuZgrQ5LEYd/TCnqcBOhPACdmqfQR290Ut7vQD8\nF1CTCUxHnRylOTD9fkJRtxmXY7+HjyWVYDIxlA3y91z65J8chNiLOuRwTr2AJdr7JaiTgN9b/os2\nz9wBoBKKUgPoBOxAiESEuIk6YXtnbV0FhDiAOo7HLznKMrr0rHSWRCzh0JVDpjqEJEmSyeyN28uy\nyGVkZuszen3h6NtbqRpCXNHex/PvlIu1eHBc/jht2eOWx+WyPHeK8gKKEoKihJCQUOigrSysWBKx\nhKUnlxZ6X6lk+uijj/Dz8yMwMJCGDRty8OBBc4cEQGxsLP7+/gVe/vbbb+Pj40NgYCB9+vTh1q1b\nue6rKArffvvt/WWTJk1i8eLFRo1dMp9F4YtYFrkMKwvTNxcb3pVVveIvmtH7hJiPEE0RoikuBXoC\n/AFWFlb08uzF3kt7uZZ6zQQBSsXJ/v372bRpE6GhoZw4cYI//viD2rVzm8HUeLKzs/PfSA8dOnQg\nPDycEydO4O3tzSeffJLrdlWrVuWbb74hI6OgcwE9KCurIBPkSeZwPuk8IVdD6OvVt0iq8fRNDle1\nKiG0f+99017iwbl/XbVlj1vumstyk+nj2Qed0BF8JtiUh5GKgStXruDs7IytrTpxm7OzMzVrqvPb\nb9u2DR8fHxo3bswrr7xC9+7q3PQzZszgyy+/vF+Gv78/sbGxAPTu3ZsmTZrg5+fH/Pnz72/j6OjI\nm2++SVBQEPv37+fIkSM888wzNGnShE6dOnHlinqTfeTIEYKCgggKCuL7778v1Ll07NgRKyv1arFl\ny5bExcXlup2Liwvt2rVjyZIlj6w7duwYLVu2vH/3cfPmTQDatGnDa6+9RtOmTfnmm28YPXo0L774\nIi1btsTDw4Pdu3czduxYfH19GT16dKHiloxnQ/QGLBQLetbrWSTH0/feJBh1lq9PtX835lg+CUVZ\nidr4fBshrqAo24GPczRCdwTeQYhEFCUJRWkJHESde/hbTKhOhTo0rdaU9dHrGes/FgtFPgdYFGb+\nHsHJy0lGLbNBzQpM7+GX5/qOHTvywQcf4O3tTfv27Rk0aBDPPPMM6enpjBs3jl27duHp6cmgQYMK\ndLyFCxdSpUoV0tLSaNasGf369cPJyYmUlBRatGjBV199RWZmJs888wwbN27ExcWFVatWMW3aNBYu\nXMiYMWP47rvvaN26NW+//bbe571w4cLHxjxlyhS6dOnC2LFjH1g+cuRIvv32W5555hnef/99Zs6c\nyZw5cwDIyMggJEQdq2z06NHcvHmT/fv3ExwcTM+ePfn7779ZsGABzZo149ixYzRsqM902JK+snRZ\nbDyzkda1WlPVoWqRHLMgXVlXAPuB+ihKHIryHGpS6ICiRKNOov6ptvUW4CwQA/yEOscuCJEIfAgc\n1l4faMvQtlmg7XMG2GqME3ucvl59uXjnIkeuHjH1oSQzcnR05MiRI8yfPx8XFxcGDRrE4sWLiYqK\nom7dunh5eaEoCsOHF2w20rlz5xIUFETLli25ePEi0dHRAFhaWtKvXz8ATp06RXh4OB06dKBhw4bM\nmjWLuLg4bt26xa1bt2jdujUAI0aM0OucPvroI6ysrBg2bFie23h4eNCiRQuWL19+f9nt27e5desW\nzzzzDACjRo1i796999c/nGx69OiBoigEBARQrVo1AgICsLCwwM/P7/6dlFR09sXt43radfp49Smy\nY+Z/5yDEkDzWtMtlWwFMzKOchaiTvT+8PAR4tAXOhDq4deCTg5+wLnodzao3K8pDl1mPu8I3JUtL\nS9q0aUObNm0ICAhgyZIlj73qtbKyQqfT3f98r+/+7t27+eOPP9i/fz8ODg60adPm/jo7OzssLS0B\n9b+An58f+/fvf6Dc3BqQC2vx4sVs2rSJnTt35lvn/O6779K/f//7ySA/5cqVe+Dzvao4CwuL++/v\nfZbtEkVvffR6nO2dedr16SI7ZpmsU7GzsqOrR1d2xO7g9t3b5g5HMpFTp07dv7oHtc7dzc0NHx8f\nYmNjOXPmDAArVqy4v427uzuhoaEAhIaGcu7cOUC98q5cuTIODg5ERUVx4MCBXI9Zv359EhIS7ieH\nzMxMIiIiqFSpEpUqVeKvv/4CYNmyZYU6l23btvH5558THByMg4NDvtv7+PjQoEEDfv/9dwAqVqxI\n5cqV2bdvHwBLly4tcOKQzOta6jX2XtpLr3q9sLYouuFJymRyALVqKUOXweazm80dimQiycnJjBo1\nigYNGhAYGMjJkyeZMWMGdnZ2zJ8/n27dutG4cWOqVv23Drdfv34kJibi5+fHd999h7e3NwCdO3cm\nKysLX19fpk6dSsuWLXM9po2NDWvXrmXKlCkEBQXRsGFD/vnnHwAWLVrExIkTadiwIY+bnvfUqVO4\nurref61Zs4ZJkyZx586d+9VVEyZMyPf8p02b9kDD9ZIlS3j77bcJDAzk2LFjvP/++wX6OUrmFXwm\nGJ3QFWmVEpTkOaS3NRWGTvYz8PeBZIts1vZYK5/wNIHIyEh8fX3NHUa+du/ezZdffsmmTZvMHUqJ\nVlJ+3yWJTujotr4b1cpVY3HnxYYXuFw5wlDRtCCbltk7B4D+3v05ffM04dfD899YkiSpiB28cpC4\n5DgGeA8o8mOX6eTQtW5X7K3sWRe9ztyhSGbUpk0bedcgFUvrotdR0bYi7d3aF/mxy3RycLRxpLN7\nZ7ac20JKZoq5w5EkSbovMT2RnRd20sOjB7aWtvnvYGRlOjmAWrWUlpXGlnNbzB2KJEnSfcExwWTp\nsujv3d8sxy/zySHAOQCvyl6sPb3W3KFIkiQB6vMy66LX0ahqI+pVqmeWGMp8clAUhf5e/Tl54yQn\nb5w0dziSJEmEXA0hNinWbHcNIJMDAN3rdcfW0lbePZRCjo6Ojyzbu3cvjRs3xsrKirVr5e9cKn7W\nnl5LeZvydHTraLYYZHIAKthUoJN7Jzaf3SwbpsuAOnXqsHjxYoYOHWqyY8ghJiR93Uy/yY7zO+ju\n0R07KzuzxSGTg2Zg/YGkZqXKhukywN3dncDAQCws8v7zj42NxdfXl3HjxuHn50fHjh1JS0sD5NDX\nkmltjNlIpi6Tgd4DzRqH6acTKiECnQOpX7k+a06tob9Xf/nEtLFtnQrxYcYts3oAdPk0/+30FB0d\nzYoVK/jpp58YOHAg69atY/jw4XLoa8lkdELHmtNraFy1MZ6VPc0ai7xz0CiKwsD6A4lMjJRPTEsA\n1K1b9/6Xd5MmTYiNjZVDX0smdfDKQS7cucCA+kX/RPTD5J1DDt08uvFVyFesPr2aAJcAc4dTupjw\nCt9Ucg5VbWlpeb9a6XHk0NeSIdacXkMl20p0cOtg7lDknUNO5azL0dWjK9vObSMpw7izlkmlgxz6\nWjKVhNQEdl3YRW/P3mZ5IvphMjk8ZKD3QNKz0/n9zO/mDkUygtTU1AeGv/766685fPjw/aGwx48f\nj59f4SYikkNfS6awIWYD2SLbrM825FSmh+zOy9DNQ0nJTOG3Xr/JhmkDyCGcyxb5+9Zfti6bLuu7\nUKdCHRZ0XGC6A8khuw0zqP4gzt4+S8hV0yQfSZKknPbG7eVKyhUG1x9s7lDuk8khF53cO1HRtiIr\no1aaOxRJksqAVadWUdW+Km1qtzF3KPfJ5JALOys7+nj2YdeFXVxLvWbucCRJKsUuJF3g78t/079+\nf6wsik8HUpkc8jDAewBZIktOBCRJkkmtPrUaK8WKfl79zB3KA2RyyEOdCnV4suaTrD21lkxdprnD\nkSSpFErPSmdDzAba1mlLVYeq5g7nATI5PMag+oO4lnaNPRf3mDsUSZJKoW2x6jNVg32KT0P0PTI5\nPEZr19bUKFdDNkyXYLkN2f3111/ToEEDAgMDadeuHefPnzdDZJIEq6JW4VHRg6bVCtS7tEjJ5PAY\nlhaWDKw/kIPxBzl766y5w5GMpFGjRoSEhHDixAn69+/P5MmTjX4MOTSGlJ+whDDCb4QzqP6gYvk8\nlUwO+ejr1RcbCxuWRy03dyiSkTz77LM4ODgA0LJlS+Li4h7ZRg7ZLZna8qjllLMuRy/PXuYOJVfF\np99UMVXFrgqd63Ym+EwwrzZ+lfI25c0dUon02aHPiEqMMmqZPlV8mNJ8ikFl/Pzzz3Tp0iXXdXLI\nbslUrqddZ3vsdgZ4D6Ccdbn8dzADeedQAEN9hpKWlUbwmWBzhyIZ0a+//kpISAhvv/12ruvlkN2S\nqaw7vY5MXWaxbIi+R945FICfsx+BLoGsjFrJEJ8hWCgypxaWoVf4xvbHH3/w0UcfsWfPngeG0s5J\nDtktmUKmLpPVp1fTqmYr6lasa+5w8iS/5QpoiM8QYpNiOXD5gLlDkQx09OhRxo8fT3BwMFWrFq5v\nuRyyWzLUnxf+5FrqNYb4DDF3KI9lWHJQlNdRlAgUJRxFWYGi2KEodVGUgyhKDIqyCkWx0ba11T7H\naOvdc5Tzjrb8FIrSyaCYTKSjW0eq2FWRDdMlTG5Ddr/99tskJyczYMAAGjZsSM+ePQtVphyyWzLE\n8qjl1HKsxdO1njZ3KI8nhNDvBbUEnBNgr31eLWC09u9gbdkPAl7U3r8k4Aft/WABq7T3DQQcF2Ar\noK6AMwIs8z3+1iaiqH0b+q0IWBwgLty+UOTHLolOnjxp7hCkIiR/3/mLvBEp/Bf7i0Vhi8wTwDJC\nRAG/4w2tVrIC7FEUK8ABuAK0BdZq65cAvbX3vbTPaOvbaZ17ewErEeIuQpwDYoDmBsZlEoPqD8JS\nsZR3D5Ik6WVZ5DLsrezp49XH3KHkS//kIMQl4EvgAmpSuA0cAW4hxL2Wtjiglva+FnBR2zdL297p\ngeWP7vMgRXkBRQlBUUJISNA7dH25OLjQ0b0jG2I2kJyRXOTHlySp5EpMT2TL2S308OhBRduK5g4n\nX/onB0WpjHrVXxeoCZQDOhsnrDwIMR8hmiJEU1xcTHqovAz3HU5KZgobz2w0y/FLGlFSZxqUCkX+\nnvO39vRaMnQZDPMdZu5QCsSQaqX2wDmESECITGA98CRQSatmAnAFLmnvLwG1AbT1FYEbDyx/dJ9i\nJ8AlgECXQFZErUAndOYOp1izs7Pjxo0b8oujlBNCcOPGDezs7MwdSrGVqctkVdQqWtVshUclD3OH\nUyCGPOdwAWiJojgAaUA7IAT4E+gPrARGAfcusYO1z/u19bsQQqAowcByFOVr1DsQL+CQAXGZ3HDf\n4UzeO5m/Lv1Fa9fW5g6n2HJ1dSUuLo4EM1QBSkXLzs4OV1dXc4dRbO2I3cG1tGtMbzXd3KEUmP7J\nQYiDKMpaIBTIAo4C84HNwEoUZZa27Gdtj5+BpShKDJAIDNbKiUBRVgMntXImIkS23nEVgfZu7alq\nX5VfT/4qk8NjWFtbU7du8X3IR5KKyrLIZbhVcOOpWk+ZO5QCM+wJaSGmAw+nwrPk1ttIiHRgQB7l\nfAR8ZFAsRcjawprBPoOZe3Qu0Tej8arsZe6QJEkqpo4nHOfE9RNMbT61RI2uUHIiLWb6e/fHztKO\nXyN/NXcokiQVY0tPLqW8dXn6eBb/7qs5yeSgp8p2lelRrwebzmziRtoNc4cjSVIxdDn5MjvO76C/\nd38crB3MHU6hlKnkIITgfxHxnL56xyjlDW8wnAxdBqtPrTZKeZIklS7LI5ejoDDUd6i5Qym0MpUc\n7tzN4q01x/lsq3HmFfCo6MHTtZ5m5amV3M2+a5QyJUkqHVIyU1gXvY6Obh2pXq66ucMptDKVHCrY\nWTOhTT12Rl3jcGyiUcoc6Tfy/pOPkiRJ92yI3kByZjIjGowwdyh6KVPJAWBMq7pULW/LZ1ujjPJw\nVovqLfCu7M0vJ3+RD3tJkgRAti6bXyN/pVHVRgS4BJg7HL2UueRgb2PJK+28CDl/kz9PXTO4PEVR\nGNFgBDG3Yth/eb8RIpQkqaTbdXEXl5IvMbLBSHOHorcylxwABjWrjZuTA59vO4VOZ/jVfte6XXGx\nd2FxxGLDg5MkqUQTQrA4fDG1y9fm2drPmjscvZXJ5GBtacGbHesTFX+HjccNH8bJxtKGob5D2X9l\nP6cSTxkhQkmSSqqj145y4voJRjYYiaWFpbnD0VuZTA4A3QNq4FezAl/97zR3swwfrWOA9wDsrexZ\nErEk/40lSSq1FkcsppJtJXp59jJ3KAYps8nBwkJhahcf4m6m8euBCwaXV9G2Iv28+rH13FbiU+KN\nEKEkSSXNudvn2H1xN4N9BmNvZW/ucAxSZpMDwNNeLjzl6cx3u6JJSs80uLzhDYYjECyPlDPFSVJZ\ntPTkUnXstfqDzR2Kwcp0cgCY2sWHm6mZ/LD7jMFl1XKsRUe3jqw5vYY7GcZ5CluSpJLhRtoNgs8E\n09OzJ072TuYOx2BlPjn416pIr4Y1Wfj3OeJvpxtc3ij/USRnJrP29Nr8N5YkqdRYHrWcjOyMEt19\nNacynxwA3upYH50OZu84bXBZfk5+tKjRgqUnl5KRnWGE6CRJKu5SM1NZGbWStnXaUrdi6ZjDRCYH\noHYVB0Y84caaIxeJik8yuLyx/mNJSEtg09lNRohOkqTibu3ptSRlJDHWf6y5QzEamRw0L7f1xNHW\nik+2GD4o3xM1nsC3ii+LwhfJeaYlqZTLzM7kl5O/0LRaUwJdAs0djtHI5KCp5GDDy2292HM6gX3R\nhs15rCgKY/3HEpsUy58X/jRShJIkFUdbzm3haurVIrlriLh82+Dvp4KSySGHka3ccK1sz8dbosg2\ncFiN9m7tcXV0ZWH4QjkgnySVUjqhY1H4Irwqe5l8fmghBDOCI3h91THSMgx/cDc/MjnkYGtlyeTO\nPkReSWLDUcOG1bCysGK032hOXD/B4fjDRopQkqTiZM/FPZy5fYYxfmNQFMWkx9oeEc/h2Ju83sEb\nexvTD8shk8NDegTWIKh2Jb7YHkVqRpZBZfXy7IWTnRMLwhYYKTpJkooLIQQLwhZQy7EWnet2Numx\nMrJ0fLI1Cu9qjgxqWtukx7pHJoeHKIrCe918uZp0l/l7zxpUlp2VHSP9RrL/yn4irkcYKUJJkoqD\nw/GHOXH9BGP8xmBtYW3SY/2yP5bzN1KZ1q0BVpZF87Utk0MumrlXoVtADX7cc9bgB+MGeg+kvE15\nefcgSaXMT2E/4WTnRG+v3iY9zs2UDObujKa1twvPeLuY9Fg5yeSQhymdfcjWCb7YbtgQ3I42jgz1\nGcofF/7gzC3Dh+iQJMn8whLCOHDlAKP8RmFraWvSY32zM5rku1lM6+pr0uM8TCaHPNRxcmDMk+6s\nC40jLO62QWUN8x2GvZU9C8MXGik6SZLMaUHYAsrblGdg/YEmPU7MtWR+PXCewc3rUL96eZMe62Ey\nOTzGxLaeVClnw4ebThrUHbWyXWX6e/dn89nNxN2JM2KEkiQVtZibMey6uIthvsMoZ13OpMeatfkk\n9taWvNnB26THyY1MDo9Rwc6aNzp4cyg2kS1hhs3RMKrBKCwUC3n3IEkl3Pyw+dhb2TPMZ5hJj/Pn\nqWvsPpXAK+28cHI0bdVVbmRyyMeQ5nXwqV6ej7dEkp6p/4Mn1cpVo49nH36L+U1OBiRJJVTs7Vi2\nx25ncP3BVLKrZLLjZGbrmLXpJO5ODoxq5W6y4zyOTA75sLRQeL9HAy7dSuMnA7u2PhfwHEIIFoUv\nMlJ0kiQVpZ/CfsLGwoaRfqYdlnvZgfOcSUhhWrcG2FiZ52taJocCaFXPmS7+1Zm3+4xBXVtrOtak\nR70erIteR0Jq0YyPIkmScVy8c5HNZzfT37s/zvbOJjtOYkoGX+84zVOezrT3rWqy4+THsOSgKJVQ\nlLUoShSKEomiPIGiVEFRdqAo0dq/lbVtFRRlLooSg6KcQFEa5yhnlLZ9NIoyyqCYTOTdrr5kC8Gn\nWyMNKuf5gOfJ1GWyOGKxcQKTJKlI/Bz2MxaKBaP9Rpv0OF/97xQpGdlM79HA5ENyPI6hdw7fANsQ\nwgcIAiKBqcBOhPACdmqfAboAXtrrBeC/AChKFWA60AJoDky/n1CKkdpVHHjhaQ9+O3aZkNhEvcup\nU6EOXet2Zc3pNSSm61+OJElF50ryFTae2Uhfr75UK1fNZMcJv3Sb5YcuMPIJN7yqFW3X1YfpnxwU\npSLQGvgZACEyEOIW0AtYom0+Gcu2AAAgAElEQVS1BLj3+GAv4BeEEAhxAKiEotQAOgE7ECIRIW4C\nOwDTDlSip5eerUeNinZMD44waNTWcYHjSM9Kl3cPklRC/Byufs2ZclhuIQQzf4+gsoMNr7Uv+q6r\nDzPkzqEukAAsQlGOoigLUJRyQDWEuKJtEw/cS7O1gIs59o/TluW1/FGK8gKKEoKihJBQ9HX2DjZW\nvNvVl4jLSaw8fEHvcjwqetC5bmdWRq2Udw+SVMzFp8SzPno9fTz7UNOxpsmO8/uJKxyOvcnkTvWp\naG/asZoKwpDkYAU0Bv6LEI2AFP6tQlKpT44ZbzIDIeYjRFOEaIpL0Y0xklP3wBq09KjCl9tPcStV\n/zmiJwROkHcPklQCLAhbgEDwfMDzJjtGyt0sPt4ciX+tCgwoolFX82NIcogD4hDioPZ5LWqyuKpV\nF6H9e01bfwnIedau2rK8lhdLiqIwo6cfSelZfPW/03qX41FJ3j1IUnF3JfkK66LXmfyuYe6uaOKT\n0pnZ0x9LC/M1Quekf3IQIh64iKLU15a0A04CwcC9HkejgI3a+2BgpNZrqSVwW6t+2g50RFEqaw3R\nHbVlxZZP9QqMaOnGsoPnCb+k/7hLE4K0u4fwxcYLTpIko7k3mvK4gHEmO0bMtWR+3neOAU1caeJW\nfPriGNpb6WVgGYpyAmgIfAx8CnRAUaKB9tpngC3AWSAG+Al4CQAhEoEPgcPa6wNtWbH2egdvqpSz\n4T8bw9Hp2TjtUdGDLnW7sPLUSm6k3TByhJIkGeJy8mXWx6ynr2dfajjWMMkx7k396WBjyZQuPiY5\nhr4MSw5CHNPaAAIRojdC3ESIGwjRDiG8EKL9/S96tZfSRISohxABCBGSo5yFCOGpvUrE48MV7a15\np4svRy/cYs2Ri/nvkIcJQRO4m31XPjUtScXM/BPzAbV3oalsDY/nr5jrvNmxPs5mGD/pceQT0gbo\n27gWzdwr8+nWKL0bp+tWrEt3j+6sPLVSPjUtScXExaSLbIzZyADvAVQvV90kx0i5m8WHm07SoEYF\nhrWoY5JjGEImBwMoisIHvfxJSs8yaFKgCYETyNJl8VPYT0aMTpIkff1w4gcsLSxN2tbwzc5ortxO\n58PefkU29WdhFL+IShjfGhUY9YQ7yw9d4NjFW3qVUbtCbXp79mbN6TVcTr5s5AglSSqMs7fOsuns\nJgbXH4yLg2m6zJ+Kv8PPf51jcLPaNHGrYpJjGEomByN4vYMXVcvbMm1DGFnZOr3KGB84HgXlfj2n\nJEnmMe/4PGwtbRkbYJqnoXU6wXu/hVHBzoopnYtXI3ROMjkYQXk7a6b38CPichK/7D+vVxk1HGsw\nwHsAv8X8xoUk/Z++liRJf6cST7E9djvDfYdTxc40V/TrQuM4HHuTd7r4UrmcjUmOYQwyORhJF//q\ntKnvwlf/O6X3sN7PBzyPtYU13x/73sjRSZJUEN8d/Y7y1uUZ5WeawaFvpmTwydYomrpVpn8TV5Mc\nw1hkcjASRVH4oKc/WTrBB5si9CrDxcGFob5D2XpuK6cS9W/gliSp8I5eO8ruuN2M8R9DRduKJjnG\nx1siSUrLZFYffyyKyZPQeZHJwYjqODnwSjsvtoTFsyvqql5ljPUfi6ONI98e/dbI0UmSlBchBHOO\nzMHJzolhvqaZG3r/mRusORLHC6098KlewSTHMCaZHIxs3NMeeFdz5D+/RZByN6vQ+1e0rchY/7Hs\nidvD0WtHTRChJEkP++vSX4ReC2V80HgcrB2MXv7drGym/RZGnSoOvNzWy+jlm4JMDkZmY2XBJ30D\nuHQrja936Dcw31CfoTjbOzPnyBxtYFtJkkxFJ3TMPTqXWo616O/V3yTH+O/uM5xNSGFWb3/sbSxN\ncgxjk8nBBJq4VWF4yzos+vscYXGFH5jPwdqB8YHjCb0Wyr5L+0wQoSRJ92yP3U5UYhQTG07E2tL4\n8yjEXLvDvD/P0DOoJq29zTPVgD5kcjCRyZ19cHa0Zer6E3o9+9DPqx+ujq7MCZ1Dti7bBBFKkpSZ\nncnc0Ll4Vfaia92uRi9fpxNMXReGg60l7/doYPTyTUkmBxOpYGfNzJ7qsw8//3Wu0PtbW1rzauNX\nib4Zzaazm0wQoSRJq0+vJi45jtcav4alhfGre5YdukDI+Zu8161BsRtYLz8yOZhQZ//qdPKrxtc7\nTnPuekqh9+/o3hE/Jz++O/Yd6Vn6PTshSVLukjOS+fH4jzSv3pynaz1t9PKv3E7js61RPOXpTL/G\nuc98XJzJ5GBC9wbms7GyYOq6E4We98FCseCNJm8QnxLP8qjlJopSksqmheELuXn3Jm80eQNFMe4z\nB0II/vNbONk6wcd9AoxeflGQycHEqlWw471uvhw8l8jKw4Wf96F5DfWqZkHYAm7f1X/WOUmS/nUt\n9RpLTy6ls3tn/Jz9jF7+7yeu8EfkNd7o4E0dJ+N3jS0KMjkUgYFNa9OqnhOfbInkyu20Qu//epPX\nSclMkYPySZKRzDs2jyyRxSuNXjF62TeS7zIjOIKg2pUY+1Rdo5dfVGRyKAKKovBJ3wCydIJ314cV\n+tkFr8pe9KrXi+VRy7mYpP+sc5Ikwembp9kQs4HB9QdTu0Jto5c/PTiC5PQsvugfiGUxHyLjcWRy\nKCJuTuV4u1N9/jyVwLrQS4Xef1KjSVhbWDM7dLYJopOksuOrkK9wtHZkQtAEo5e9LTyeTSeu8Eo7\nT7yrlTd6+UVJJociNLqVO83cK/PB7xFcTSpc76OqDlUZ4z+GHed3EHo11EQRSlLp9telv/jn8j+M\nDxxv9MH1bqVm8N5v4TSoUYHxz9QzatnmIJNDEbKwUPi8fxB3s3RM21D46qVRDUZR1b4qX4Z8iU7o\nN6mQJJVVWbosvjz8JXXK12GIzxCjlz89OIJbqRl83j8Q62I47WdhlfwzKGHqOqvVS39EXmPD0cJV\nLzlYO/BK41cIux7G1nNbTRShJJVO66PXc+b2Gd5o8obRh8nYFn6FjccuM6mtJ/61TDPcd1GTycEM\nxjxZl6ZulZkeHFHo3ks96vXAt4ovs4/MJi2r8D2fJKksupNxh++PfU/jqo1pW6etUcu+kXyXaRvC\n8atZgYnPehq1bHOSycEMLC0UvhwQRFa2YMq6wlUvWSgWTG42maupV1kcvth0QUpSKfLj8R+5mX6T\nKc2nGP2BtPc3RpCUnslXA4NKRXXSPaXnTEoYd+dyvNvVh72nE1hxqHDdU5tWb0on904sDF9IfEq8\niSKUpNLh3O1zLItcRl+vvjRwMu7gd8HHL7M57AqvtfcuERP4FIZMDmY0rIUbT3k6M2vzSS7cSC3U\nvm80eQOB4OsjX5soOkkqHb4M+RJbK1smNZpk1HLjb6fz3oYwGtWpxPjWHkYtuziQycGMLCwUPusf\niKWi8MbqY2QXYuylmo41Ge03mq3ntsqurZKUh31x+9gbt5cJgRNwtnc2WrlCCN5ee5zMbMHXAxti\nVYqqk+4pfWdUwtSqZM/MXn6EnL/Jj3vPFGrfsf5jqepQlU8PfSrnfJCkh2RmZ/L54c9xq+Bm9Hmh\nlx44z77o67zbzZe6zuWMWnZxIZNDMdCnUS26BdRg9o7ThF8q+OB6DtYOvNnkTSITI1kXvc6EEUpS\nyfNr5K/EJsUyudlko3ZdPZOQzMdbImnt7cLwFnWMVm5xI5NDMaAoCh/18aeygw2vrzpGembB7wK6\n1O1C02pNmXt0LrfSb5kwSkkqOa6mXOWH4z/QxrUNrV1bG63cjCwdr608hp21JZ/3CyyRQ3EXlEwO\nxUQlBxu+HBBE9LVkPt0aVeD9FEXh3RbvkpyRzNyjc00YoSSVHF8d+YosXRaTm002arlz/jhN2KXb\nfNo3gOoV7YxadnFjeHJQFEsU5SiKskn7XBdFOYiixKAoq1AUG225rfY5RlvvnqOMd7Tlp1CUTgbH\nVEK19nZhzJPuLP4nlj+jrhV4P6/KXgzxGcLa02uJuBFhwgglqfg7HH+Yree2MsZ/jFFHXT149gb/\n3XOGQU1r09m/htHKLa6McefwKhCZ4/NnwGyE8ARuAs9py58DbmrLZ2vbgaI0AAYDfkBnYB6KYvzJ\nXEuIKZ198KlenrfXHifhzt0C7/dSw5eoYleFjw98LMddksqsTF0mHx/8mJrlavJcwHP571BAt9My\neWP1cdyqOPB+D+M+K1FcGZYcFMUV6AYs0D4rQFtgrbbFEqC39r6X9hltfTtt+17ASoS4ixDngBig\nuUFxlWB21pbMHdKIO+lZvLXmeIGnFi1vU543m77JiesnZOO0VGYtO7mMmFsxTG4+GXsre6OUKYTg\n3Q1hxCelM2dwI8rZWhml3OLO0DuHOcBk4N6lqhNwCyGytM9xwL2ZtWsB6qPA6vrb2vb/Ln90nwcp\nygsoSgiKEkJCgoGhF1/e1cozrZsve04nsOif2ALv192jO82qN2P2kdncSLthugAlqRi6knyFecfn\n0ca1DW1rG2/8pFWHL7L5xBXe7OhNw9qVjFZucad/clCU7sA1hDhivHDyIcR8hGiKEE1xcSmyw5rD\niJZutPetyqdbIwmLK1j3VkVReK/le6Rlpcknp6Uy59NDnwLwTot3jNaLKObaHWb8HsGTnk5MaF3y\n52goDEPuHJ4EeqIoscBK1Oqkb4BKKMq9+y5X4N641JcAtXVIXV8RuPHA8kf3KbMUReGL/kE4lbPl\n5RWhJN/Nyn8nwKOiB2P8xhB8JpjD8YdNHKUkFQ97Lu5h18VdjA8cT03HmkYpMz0zm0nLj1LOxorZ\nAxtiUYKn/NSH/slBiHcQwhUh3FEblHchxDDgT6C/ttUoYKP2Plj7jLZ+lzYcaTAwWOvNVBfwAg7p\nHVcpUrmcDXOHNOJCYmqhJgcaFziOWo61+PDAh2RkZ5g4Skkyr9TMVD459AmelTwZ6TfSaOV+tDmS\nqPg7fDkgiKoVSne31dyY4jmHKcAbKEoMapvCz9rynwEnbfkbwFQAhIgAVgMngW3ARISQY0Fomtet\nwmvtvdl47DJrQuIKtI+9lT3vtXyPc7fP8XPYz/nvIEkl2Lxj87iUfIn/tPwP1hbGeRJ604nLLD1w\nnnFP1+VZn6pGKbOkMU6zuxC7gd3a+7Pk1ttIiHRgQB77fwR8ZJRYSqGJz3qy/8wN3g8OJ6h2JepX\nz3/i8qdqPUWXul34KewnOrl3wqNS6Rs1UpIibkSwNHIpA7wH0LhaY6OUGXs9hanr1NFWJ3f2MUqZ\nJZF8QroEsLRQ+GZIQxxtrXlx2RFSCtj+MKXZFOyt7Jm5f6Z89kEqdbJ0Wcz8ZyZOdk681uQ1o5R5\nNyubSStCsbRQ+G5o41I1eU9hld0zL2Gqlrdj7pCGxF5P4d0Ctj842TvxVtO3CL0WKp99kEqdX0/+\nSmRiJO+0eIcKNsaZaGfWpkjCLyXx1YAgalUyznMSJZVMDiVIq3rOvK61Pyw/dKFA+/T27E3z6s2Z\nHTKbqylXTRyhJBWNi0kX+f7Y97Sp3Yb2ddobpczfjl5i6YHzvNDag/YNqhmlzJJMJocSZuKznrT2\ndmFm8EmOX8x/FFZFUZj+xHQydZl8eODDQs1XLUnFkU7omL5/OlYWVkxrMc0ozzScvnqHd9aH0dy9\nCpM71TdClCWfTA4ljIWFwjeDGuJS3paXloWSmJJ/V9U6FerwcqOX2RO3hy3nthRBlJJkOmtPr+Vw\n/GHeavoW1ctVN7i8O+mZTFh6hHK2Vnw3tFGpnNVNH/KnUAJVLmfDvGGNSbhzl1dXHi3Q9KLDfIcR\n6BLIJ4c+4Xra9SKIUpKM70ryFb4K+YoWNVrQ16uvweUJIZi89gSxN1L4dkijMvk8Q15kciihgmpX\nYmYvP/ZFX2fOH6fz3d7SwpIPW31IamYqHx/8uAgilCTjEkIwc/9MBIKZrWYapTrphz1n2Roez5TO\nPjxRz8kIUZYeMjmUYIOb1WZgU1e+3RXD9oj4fLf3qOTBSw1fYsf5HWw7t60IIpQk41kfvZ6/L//N\na41fo5Zj7mNzFsa+6AS+2B5Ft8AavNBaPgf0MJkcSjBFUfiglz9BtSvxxqpjRF+9k+8+o/1G4+/k\nz6yDs0hILb0j20qly6XkS3x++HOaV2/OYJ/BBpd3MTGVl1ccxatqeb7oX7qn+9SXTA4lnJ21JT8M\nb4y9jRUvLD3C7bTMx25vZWHFR09/RHpWunqLLnsvScWcTuj4z9//US+GnvwAC8Wwr63UjCxeWHoE\nnU7w44gmONiUjfkZCksmh1KgRkV7/ju8MRcTUwvUQO1R0YNXGr3Cnrg9/BbzWxFFKUn6WR65nMPx\nh5ncbLLB1UlCCN5ac5xT8UnMHdIId+dyRoqy9JHJoZRo5l6Fmb382H0qgc+3ReW7/fAGw2larSmf\nHf6My8mXiyBCSSq8s7fPMid0Dq1dW9PHs4/B5X27K4YtYfG808WXNvXL5oB6BVX2ksPVCMgu2NhE\nJc2wFm6MfMKNH/eeZe2Rx4/gaqFY8OGTHwLwzr53yNbJgXCl4iUzO5Ope6dib2XPjCdmGNwusD0i\nnq93nKZvo1o8/3RdI0VZxLIy4FpkkRyqbCWHzDRY2AVmN4A/ZsCNM+aOyOj+070Breo58e76MI6c\nT3zstq7lXXmn+TuEXgtlUcSiIopQkgrm+2PfE5kYyYwnZuDiYNjMj+GXbvPaymME1a7Ex30DSl4D\ndMIp2D4NvvaFX3oVyQVu2UoOFtbQ5weo2Rj+/ga+bQyLusHxVWriKAWsLS2YN6wxNSrZMX7pES4m\npj52+571etLRrSPfH/2eiOsRRRSlJD3e4fjDLAxfSF+vvrRza2dQWVeT0nl+SQiVHaz5aWQT7Kwt\njRSliWWkwNFf4eeO8H1zOPgDuD0Bvb6HIkhuSontrbKtqaBziP77J12BY8vg6FK4GQt2FSFgIDQe\nCTUCjRamucRcS6bvvL+pXtGOtS+2ooJd3pOg3L57m77BfXGwcmBV91U4WDsUYaSS9KCkjCT6B/fH\nysKKtT3WGvT3mJaRzaD5+4m5lszaCa1oUNM4o7eajBBwKRSO/gJh6yDjDjh5QeMREDQEHA1sJ1mu\nHGGoaFqQTctucrhHp4Pzf8GRJRD5O2TfhRoN1SQR0F9NGiXUPzHXGbnwEE/Uc2Lh6GaPHZv+4JWD\njPvfOPp69WVGqxlFF6Qk5SCE4K09b7Hzwk6WdFlCkEuQ3mXpdIJJK0LZGh7PTyOaFu+RVlMT4cRq\nCP0FrkWAlT349YbGo6BOS+PdKRQiOZStaqXcWFhA3dbQ/2d4Mwq6fAG6LNj8BnxZHzZMgPP/qBm9\nhGnl6cxHffzZF32d6cERj32moUWNFjwX8BzrotfJp6cls1kXvY7/nf8fkxpNMigxAHy6LYotYfFM\n6+pbPBODTgdn98Da5+ArH9g2BSytoftseOuUWgXu9kSRVCHlRj79kZNDFWjxAjQfB5ePqlk8fB0c\nXwFOntBoBDQcavitXREa1KwO566n8sOeM9Sp4sCEZ+rlue1LDV/icPxhZu6fiZ+zH7XL1y7CSKWy\nLuZmDJ8d+oyWNVoy1n+sQWUt3R/L/L1nGfmEG889Vcx6JiVdVqu0Q5fCrfNq7UST0WrVUfUAc0d3\nn6xWyk9GCpwMVhPFhX/Awgq8O6u3e57twKL4N27pdIJXVx3j9+OX+WZwQ3o1zPtBokvJlxgQPAD3\niu4s6bLEaBO2S9LjpGelM2TzEBLTE1nXcx3O9s56l/XHyau8sDSEtj5V+XFEUywtikHPpOxMOL1d\n/R6J2QFCp9ZYNBoJvt3BuohmnStEtZK8c8iPTTloOER9JZxWG7CPLYeoTVC+JjQaBo2GQ2V3c0ea\nJwsLhS8HBHItKZ231hzHpbwtrerl/p+vlmMtZrSawZt73mT2kdlMbja5iKOVyqJPD31KzK0Yfmj/\ng0GJ4eiFm7y84ij+tSoyd0gj8yeGG2fUhHBsOaRcA8fq8NTr0HAYOOV9F18cyDsHfWRlwOmt6m3h\nmZ3qVYBHG7Xayac7WBfPMeFvp2bS/4d/iE9KZ/X4J/CtkXfPjY8PfsyKqBXMbjOb9m7GmYZRknIT\nfCaYaX9N4/mA53m18at6l3MmIZn+//2H8nbWrHuxFS7lbY0YZSFkpEJksPr9cP4vUCzBu5PaycWz\nA1ia8Zpc9lYqQrfj1KuCo0vh1gWwrwyBg9Q/hGp+5o7uEZdupdFv3j/ohGDdi62oXSX3boIZ2RmM\n2jqK2KRYVnVfRZ0KdYo4UqksiL4ZzdDNQ/F39uenjj9hZaHfF+fVpHT6zvuHu1nZrJ3QyjxjJl0+\npt4lhK2Fu7ehise/7ZTlDZ+xzihkcjADnQ7O7VH/OKI2QXYG1GqiJgn/fmBb3twR3nf66h0G/LCf\nyg7WrH2xFc6OuV9hXU6+zIDfB1DTsSZLuyzFzqp43hFJJVNKZgqDNw3mTsYd1vRYo/dT0LfTMhn0\n434uJKay6oUnCHAtwu7nabcgbI36/z7+BFjZQYNe6v97tyfN1tMoTzI5mFnKDTixSv2DSYgE63Lg\n30dtxHZtViz+YI6cT2TYgoN4VnVkxbiWlM/jIbm9cXuZuHMivT1780GrD0resANSsSSE4M09b7Lz\nwk4WdFxAs+rN9ConNSOLET8f4kTcLX4e1YzW3oYNs1EgQsD5v9Vqo5O/QVa62suo8Sj12Sj7yqaP\nQV+yQdrMyjnBEy9ByxchLuTfpx2P/grO9dWriqDBUE7/hjdDNXGrwn+HNWHcLyE8tySEJWOaY2/z\naM+r1q6tGR84nh9P/Ii/kz+DfAaZIVqptFkUsYgd53fwRpM39E4MGVk6Xvw1lKMXbvLd0MamTwx3\n4rUq5F8h8QzYVlAblhuPgJqNTHtsM5B3DkXl7h0IX6+2TcQdVsd58umqJgqPZ83WJTb4+GVeXXmU\nZ7xdmD+iKTZWjz4XqRM6Ju2cxP4r+1nUaRENqzY0Q6RSafHP5X948Y8X6eDWgS9af6HX3Wi2TvDK\nyqNsPnGFT/sGMLi5idrEsrMg5g+1FuD0NhDZUKeVmhAa9AabEjbUjKxWKuauRaq3pMdXQFoiVKyt\ndodtOAwqFf2DZysPXWDq+jC6BdTIs/tfUkYSQzYNITUrldXdVxs8SqZUNl1KvsSgTYNwsXdhWddl\neo2bpNMJ3l57gnWhcbzb1YcXWpugS2jiOfUO4dgyuHMFylVVu7M3GgHOXsY/XlGRyaGEyLoLp7ao\n4zqd3a0uq9dWvZuo3xWsbIoslAX7zjJrcyR9G9XiiwFBuSaI6JvRDNsyDK9KXizsvBBbSzN1FZRK\npNTMVIZvHU58Sjwru63UqwecEIL3fgtn2cELvNbei9faexsvwMx0tTNJ6BI4txcUC7XraeORaldU\ny1LwQKhscyghrGzBr4/6unleGyX2V1gzChyc1FEYG42Aqj4mD+X5pz1Iz8zmy/+dxspS4dO+gVg8\nlCC8KnvxyVOf8Nru15j+z3Q+eeoT2UAtFYhO6Ji6bypnb51lXvt5eieGDzdFsuzgBV5sU49X2xnp\nCj4+XK3uPb4S0m9BJTdo+556J1+hpnGOUQLpnxwUpTbwC1ANEMB8hPgGRakCrALcgVhgIELc1L5F\nvgG6AqnAaIQI1coaBbynlTwLIZboHVdJVdkNnn0XnpkCZ/5Ur14O/gD7v4PaLdSrlwa9wdbRZCFM\nautFRrZg7s5orCwt+Ki3/yNf/u3c2vFyo5f59ui3eFby5PmA500Wj1R6fHf0O/68+CdTm0+lVc1W\nhd5fCMHHWyJZ+Pc5xjzpzuRO9Q27MElPUsdNC/0FLoeCpQ349lD/n7m3VgfkLOMMuXPIAt5EiFAU\npTxwBEXZAYwGdiLEpyjKVGAqMAXoAnhprxbAf4EWWjKZDjRFTTJHUJRghLhpQGwll4UleLVXX8kJ\ncGKl+ge8cSJsnQr+fdUuc7Uam6RL7OvtvcjK1jFv9xkU4MNe/o/cQYwLGEfMzRjmhs6lbsW6tKtj\n2GQsUum26ewmfgr7iX5e/RjqM7TQ+99LDD/tO8fIJ9x4v3sD/RKDEHDxoPr/KWIDZKZC1QbQ+TMI\nHKgOvCndZ7w2B0XZCHynvdogxBUUpQawGyHqoyg/au9XaNufAtrcfwkxXlv+4HZ5KQ1tDgUlBFw4\noN763v+j9lOvckzwRy2E4LNtp/hhzxmGNK/NR70DHkkQ6VnpjN0+luib0SzqvAh/Z3+jxiCVDiHx\nIbyw4wWCXIKY32E+1oWst384Mczs6Vf4xJDzIuv6abBxVB9MbTxSfVC1LFWNFvl8DoriDjQCDgLV\nEOKKtiYetdoJoBZwMcdecdqyvJbndpwXUJQQFCWEhASjhF4iKIo6rnvveeqcE91nq+0V26ao48Cv\nHas2aOt0RjqcwpTO9Zn0rCcrDl1kyroTZOsevIiws7Jjbtu5ONk7MWnnJC4lXzLKsaXS49ztc7z6\n56vUcqzFnGfnFDox6HSCmb+f1C8x6LIh+g9YNUKdd/l/76kPp/X8Dt48BT3ngmvTspUYCsnwBmlF\ncQTWAa8hRNIDP2whBIpivO5QQswH5gPqnUNZZFcRmo5VX/Hh6tXQiVVq/Wklt3/HcqmY97DcBaEo\nCm929MbSQuGbndFkZOv4ckDQA7PJOds7M6/dPIZvHc7EPybyS9dfqGBTzKdhlIrEjbQbvPTHS1hZ\nWDGv/Twq2hZuSItsnWDahjBWHr7Ic0/V5b1uvgVLDPc7diyDpDi1Y0eL8epdgkt9Pc+mbDLszkFR\nrFETwzKEWK8tvapVJ6H9e01bfgnI2YnfVVuW13IpP9X9oevn6pVQv5+hUh34cxbM8YdlA7VpTzP1\nLl5RFF7v4M3kzvXZeOwyLy0LJT0z+4FtPCp5MKfNHM7fOc+ru17lbvZdQ89KKuFSM1N5edfLJKQl\n8G3bbws9aVRWto631hxn5eGLvNzWM//EkHVXrW5d2ge+CYI9n6uJYOAv8EYUdPpIJgY96N/moP62\nlgCJCPFajuVfADdyNN4qurUAABhASURBVEhXQYjJKEo3YBJqb6UWwFyEaK41SB8BGmslhAJNECLx\nsccvS20OhZF4Vu0Oe3QZJMfneHhnJDh76l3skn9imR4cwVOezswf2QQHmwdvOjef3czUfVPvP/Vq\nWQImQZKML1OXycs7X2b/lf3MbjObtnXaFmr/9MxsXl5xlB0nr/J2p/pMfPYxf7PXotQ752LyMGmJ\nUCQPwSnKU8A+IAy4V9n9Lmq7w2qgDnAetStropZMvgM6o3ZlHYMQIVpZY7V9AT5CiEX5Hl8mh8fL\n87H/keqokXo89r8mRG1/CHStxKLRzahc7sGH9JaeXMrnhz9noPdA3mv5nnwGoozRCR3T/prGprOb\nmPHEDPp59yvU/knpmYxbEsKh2ERm9vRj5BPuj250Nxki1qt/18VoGJoSQz4hLT3gTrx6dRX6i3pn\nYVsBAgao/6FqFm6cpG3h8byy8ih1qjjwy9jm1Kz04PSGs4/MZmH4QiYETWBiw4nGPAupGBNC8EXI\nFyw9uZSXG73MC4EvFGr/68l3GbXwEKfi7/DVwKAHp7IV4t8BLMPXQ0ZysRnAssSRyUHKlZGGGj5w\n9gbjloTgaGfFL2Ob41Xt37kqhBBM/2c6G2I28FbTtxjlN8pUZyMVI/OOzeO/x//LMN9hTGk2pVB3\njeeupzBq4SGu3Unnv8Oa8KxPVXXFI0PfO4BfX2hSfIa+L3FkcpDydX+SkiUQH/bvJCWNRoD7U/n+\nx4u4fJvRiw6TnpnN/BFNeaKe0/112bpspuybwvbY7fyn5X8YWH+gqc9GMqPF4Yv56shX9PHsw4xW\nM7BQCt7P5eiFmzy3RP1//POopjRyrQjndqsXMA9PmuXXF+xkbziDyOQgFcr96Q3XwN0kbXpDrWHv\nMdMbXkxMZcziw5y/kcIX/YPo3ejfqoDM7Exe3/06e+P2MuupWfSs17MozkQqYquiVjHr4Cw6u3fm\n06c/LVRHhP9FqFWUVcvb8euAWtQ5v0HtTHH7AthVUscWazyiWE63W2LJ5CDp5f7E6L+o1U8FmBj9\ndmom438N4cDZRN7o4M3LbT3vVynczb7LxJ0TORx/mFlPzqJHvR5FfUaSCa0+tZoPD3xIG9c2fP3s\n11hbFOwhNyEEP+07y5dbwxnjHMUbzgexPbcLEODRRr179ekO1nJaWqOTyUEy2PUYdbiOY8sh5Ro4\nVodGw9Q7iioeD2x6Nyubd9aFsf7oJXoG1eTz/oHYWatXkGlZaby882UOX5UJojS5d8fwjOszfN3m\na2wsCza8fGa2jrkrN+MYuYKhtn/z//buPD6q6mzg+O9kX0lIIASysCaEEJYExaCIIBYEFUW0ggpW\nrdq6UJe6Y/XVWq28vK+1WlHfatGKW4UKyqIYQbFKwpKwBEIWIgkkgRCyLzOZOe8f9wITkyAgmZkk\nz/fzmQ8zZ+5knjn3Ms+ce849J9hWCcH9jGNr9A0QNrCDI+/mJDmIs8ZmNYbCbn0H8r4AbYcBFxqd\n2MOuOP7rTmvNqxvyeWFNDqNiQnlj7hgiehjPNTQ3cE/aPaSXpMsppi7gWGKYGD2RRRMXnVpisNRR\nu+1jir9cTIJlFzblicfQaaiUeTB4cputUtEBJDmIDlF1wFxD9x2o/ME4LzzyOuO8cOQIwBjqet8H\nmfTw92LxjWNIjjVGQDU0NzA/bT6bSjaxIHWBdFJ3Usc6n08pMWgNB7fBtnewZX2Ep7WGfTqSuuFz\nSJr2Wwju0/5rRceQ5CA6lN0OhV8bfRO7VxojSvolG30TSdeQXQF3/HMzZVVNPHPVcK4711jYpcnW\nxAPrH2BD8QbuH3M/Nyfd7OIPIk6V1ppXMl/hte2vMXXAVJ4b/1z7E+nVV5gj4d6Bsh3YPP1YaR3L\nap8p3DnvRkbFntqQadEBJDkIp6mvgO0fGoni0C5jLHriVVQnzuGub3z4Ju8Ic8bG8uQVifh5e2K1\nW3nsm8dYU7iG20bcxj3J98iV1G7Oru0szFjIP3f/k5lDZvLkuCdbj0qy2+GHjcZxkL0CbE3YI0ex\nymcKj+4dSnz/KF69MYWIYOlkdilZJlQ4TUAYpP7GmPny4FZzSOy/6JG1lLfD4/gq/lIeTK9ix4FK\n/nb9GGLDA3j+wucJ9A7kjR1vcKTxCE+kPoGXhxyK7shis7Bg4wJWF67mxmE38uC5D7a8jqG6BLKW\nGq2Eo/vANwRS5lEefx13rLOyZe9RbrlgII9OT2gxo69wf9JyEGefpQ52/dtIFEXfY1depOkUPmYy\nV86ay6UjolqcppgQPYGFExYS4H368z2JjlNjqeHer+4lvTSd+8bcx83DbzZaeTYr5H5uJITcz415\nuwZcaAxBTZzBV/k1PPBRFo1WGy9cM5LLR3bfdZjdjpxWEm7jcI7RIbltKZ4NRziow8iJnMH519yL\nb++BfJjzIc9uepbEsET+Ovmv9PKXeXLcQUltCXen3U1BZQFPX/C0MQT5SP6J4c21ZRDUxxh+mnwj\nhA/G0mxn4do9vPHNPhIig3n5+hSGRHTcmufiDEhyEG6n2YJ192fsX7eYgZWbQEFD9HgCU28hLTCQ\nR/7zBCG+Ibx88csMDZO5911p++HtzE+bT5OtiUXjn+P8ykNGK7DwG1AeEDfVGKEWN/X4ENT8w7Xc\n90Em24urmJvan8cvG3b8WhfhRiQ5CHf2/bYssla+wuW2L4lS5Wj/MPYkTuPuul3U2Bp4/sLnT3sd\nAHF2rN63mgUbFxDhG8rLvnEM3r0KGqug54ATU6r0OHGaSGvNO9//wJ9W7cbP25Pnrx7BpUl9XfcB\nxMlJchDurry2icc/zqQ+J407gzeSat3EYWVnfnR/sj2a+W3SrdyRMv+0JnETZ67Z3sxL6Qt5K2cp\nKXYvXiwqpKfyhsQZxhDl/uPBo+W+OFjZwKPLdrBh72EmxPdm4TUj6dNDRiO5NUkOojPQWrNs6wGe\nWrGLHrqK/03Yw4ijn/KMLmdlcCAXeYXxp3FP02PgBJmeuaNoTUXe5zy06Rk22ar4ZXUND3tF4XNs\nGveAsDZeonk/o4hnP9uNza55dHoCc1P7y5DkzkCGsorOQCnFrDHRpA4O57FlO/jl9kDGxE7mxfEW\nknJeYmF9LrPT7uC/rcEkjr7ZuBo7MPyn/7D4abWHIHMpmduX8KBvIxWenjwdNJyZlzxsXNDYzhd9\nYXkdC/69k4155YwbFM6fZ40kNlxGmXVF0nIQbkFrzfJtB3j602zqm2zccdEgxseX8di3D3PUWsv9\nFUe5oa4JlXCZMWRy0KRWpznET7A1Q/6XsPVt7HvX8GZwAC/3DCXSJ4RFk15keN/2f1Bamu28/nU+\nL6Xl4ePpwcPTErhhbCweHtJa6FTktJLorA7XNPGnVbtZvu0AMWH+PDQthrWH/sKG4g1M9Ingv4oK\nCKuvgJBYo4M0+QYIiXZ12O6tYp+xTkLmUqg5yKHg3izoF8t3lsNM6T+Fp85/imCf4HZfvjG3nCdX\n7CT/cB2XjejLH65IlL6FzkqSg+jsvss/whOf7CTvUC0Th/YiOWkXS/a8TLB3EH+InsLkggwoWA8o\nGDLZ6DSNnwZepzZ1dJdnbTRWUtv6NuzbAMoDPehiVg9M5tmi1VhsFh4e+zCz4ma121dQVFHPHz/L\nZu2uMmLDAnhqRiIXJ8hkeZ2aJAfRFVia7Sz5TyEvfZlLg9XGjHMVhR5/J7cyhxmDZ/BQ3GxCdq0w\nfhXXHISAXjB6DiTPg97xrg7fNcqyjYSw/X1oOGq0sFLmcmTYdJ7NfpMvfviCkb1G8uz4ZxkQMqDN\nP1HVYOVv6/N469tCPJXi7ouHcOv4gXLdQlcgyUF0JeW1TSz6fC8fZOwn0AdSRm8hq2YZIb4hPDL2\nES6N/QUqP81YD3vvGrA3Q+w4ozWReCX4BLr6I3SsphrYucxICgc2g6ePsZJayjz0gAn8u2AFi7Ys\nos5ax12j7+JXw3/V5lxWjVYb727az1/TcqlqsDIzOYrfTxlKv1B/F3wo0SEkOYiuaG9ZDS+syWHd\n7jLCww7TM/YTypryuCDqAh4b+xixPWKNUThZ7xlflEfywCfYGJKZMu+ko3A6Ha2hKB22vQ07l4O1\nDnoPMz6nOaqroLKAP276IxmlGSRHJPPkuCcZHDq41Z+yNNv5cHMRL6flUVrdyPghvXhkWgJJUSEu\n+GCiQ0lyEF3Z5sIKXlibQ/q+csL6ZkDPNaCamZs4l9tH3k6gd6Dx5bn/O2NyuF3LobkB+iQZX54j\nrm1z/H6nUFcOWe8bya88B3yCIOlq41Ra9DmgFFVNVSzOWsx7e94jwDuA+8bcx6y4Wa0uKGy02vho\nSzGvbcin+GgDKbGhPDBlKOcPDpdrFroqSQ6iO/gu/wgvrttLelEhwX2/QAdlEOYbzl3JdzIzbuaJ\nBe8bq2DHv4wv1JJM8PQ1ljhNmWfMJuruQ2LtNij4yoh/zyqwWyF6rDG/0fCrwdeY3M5is/DR3o9Y\nnLWYaks118Rdw13JdxHm1zIRVjVYeS99P3/fuI/DNU2Mjgnld5fEMTG+tySFrk6Sg+hO0vdV8NqG\nfL4q3EJA5Gco/0Ii/aO495x7mDZwWstfzCXbzTUnPnSYM2iuOWeQm80JVFkEme8aHe5VReAfBqNm\nG0ktYtjxzZrtzazMX8mrWa9SUlfCeZHn8eC5D7aawHBfeR3/+HYfH20ppt5i48K4Xvx24mDGDZKW\nQrchyUF0R7llNbz+dT4rc9PwCF+Dp18JvXyjuTP511w1ZEbLZS2tDcYSpy1mG51ifPHGTYH2lsDs\naM0WyFllxJWfZpQNnmTENXQ6ePke37TJ1sQneZ/w1s63KK4tJik8ifkp8xnXb9zxbaw2O+uyy1ia\nvp+NeeV4e3hw+ai+3HLBQOlT6I4kOYjurKLOwvvpP/CPrBXU+q/F068EfxXO1UNm85uU2YT6hbZ8\nwZH8ExeJ1ZZCYASMvt74Qg5v3YHbIQ7nGAkh6z2oPwI9ok9c5Bca2/LzNVawLHcZ7+5+l/KGcpLC\nk7ht5G1Mipl0vAWQfbCa5duKWb7tIOW1TfQN8eO6c2O4/rxYWaqzO5PkIATY7JqNuYdZnLGK7TXL\n8QgoAO1NfOAEfj1qDlOHnIuHY3+DrRnyvjC+pPeuNVY46z/eHBI7A7zP8pDOplrIPrZi3ibw8DJa\nByk3Ga0Fh3WatdZkHc7iw5wPWVO4BqvdSmrfVG4dcSvnRZ6HUor8w7Ws2VnKyqyD7CmtwctDMSkh\ngjljY7goPgJPmepCSHIQoqXKegtLNn/Hx3kfUKG+R3lY8WyOJClkEtcPv4pfxP9ojeO21kYeea2R\nKPqOOvNAtIYDW4yEsHMZWGqgV7zR7zFqDgT1brF5UXURn+37jJX5K9lfs58ArwCuHHIls4fOJjZ4\nANuKKlmfc4gvssvYW1YLQEpsKDOTo7hsZD/CAuWKceFAkoMQ7cs/Us6rGcvYWLqGOpULgG6KIsZ3\nLBNjLmJafArD+4Uav7Ttdvhho5Eksj8BWxNEjjwxJNY/9CfezVRfAds/MJLCoWzwDoDhM42kEJt6\n/PoLm93Gnoo9rC9ez5f7vyT3aC4KxdjIsVw+6HIG+KeStb+R7wuO8G1eOdWNzXh6KM7p35NpSZFM\nTYqkb4hctCba0SmTg1KXAn8BPIH/Q+vnT7q9JAdxFuwu38eSzE/5T+l6jjbngtLYmwNRjUOI8U9i\nRO8RjI8dwajocKL9LHju+hdsWQJlO8DLDxKvMhfDOb/1BXZ2uzGv0bZ3jM5vm8W4EC/lJkiaBX49\nsNgs5FTksKN8BxmlGaSXplNtqcZDeZDYcxSDAlPxaRpJ3kFvsooqqW5sBiAq1J/zB4czKSGCC4b0\nIsTfRR3oonPpdMlBKU9gL/ALoBjIAOagdXa7r5HkIM6y8oZyPstdz7rCb9lTuYVGfRQAbffC3hQJ\n1j709I4iOjCG0T4WJtZmMLo0DX9rLc2hg2DktXgdu7iuvtxoKVTux+oXSvnwGRyKm0SZXxCF1YXk\nHS0grzKfgqo8bNr4wg/06EWwHoatfgilZbHUNxgtAE8PRXyfYEbHhJAc25Nxg8KJCZM1FMQZ6ITJ\nYRzwFFpPNR8/CoDWz7X7GkkOogNprTlYd5CtpVlsLNpGdvluyhqLaLBXtNrWw+5JkN1GiLZyrNfC\nDlR5+FCnPLB52Fq9xm4Nwd4Uga2xH/bGaGwNMXjYQ4nuGUBsWACDeweREBnMUPMW4CPrcomzoBOu\nBBcFFDk8LgbOa7WVUrcDtwPwdmyrp4U4W5RSRAVFETUkiiuGTD9eXmeto6imiJLaMvKPHmB/ZSkV\nDTVUNdVS31hFs92Gza6x2xVBBBKCHx744+cRQpBnOEFe4UT4R9ErMJhgP296B/kQ0cOPiGBf+vTw\na9kpLoQLuUtyODVavw68DhgtByGcLNA7kISwBBLCEpgkv09EF+YuP1MOADEOj6PNMiGEEC7gLskh\nA4hDqYEo5QPMBla4OCYhhOi23OO0ktbNKHU3sBZjKOubaL3LxVEJIUS35R7JAUDrVcAqV4chhBDC\nfU4rCSGEcCOSHIQQQrQiyUEIIUQrkhyEEEK04j4d0qerYks5S9UPZ/TaSnoRSvlZjujnk7hOj8R1\neiSu09M14+p/qhu6x9xKzqbUZvSpzS/iVBLX6ZG4To/EdXq6eVxyWkkIIUQrkhyEEEK00l2Tw+uu\nDqAdEtfpkbhOj8R1erp1XN2zz0EIIcRJddeWgxBCiJOQ5CCEEKKVrp0clLoWpXahlB2lzvnRc4+i\nVB5K5aDUVIfyS82yPJR6xAkxfoBSmeatEKUyzfIBKNXg8NziDo+lZVxPodQBh/ef7vBc23XnnLgW\notQelNqOUstRKtQsd219GTE499hpP44YlPoKpbLN4/93Znn7+9S58RWi1A4zhs1mWRhKfYFSuea/\nPZ0c01CHeslEqWqUutcldabUmyh1CKV2OpS1XT9KKZR6yTzmtqNUylmLQ2vddW8wTMNQDes1nONQ\nnqghS4OvhoEa8jV4mrd8DYM0+JjbJDox3kUa/mDeH6Bhpwvr7ikNv2+jvO26c15cUzR4mff/rOHP\nblJfrj12WsbSV0OKeT9Yw15zv7W9T50fX6GGXj8qe0HDI+b9R47vV9fty1IN/V1SZzBBQ0qL47m9\n+oHpGlZrUBpSNWw6W3F07ZaD1rvROqeNZ64E3kfrJrTeB+QBY81bHloXoLUFeN/ctuMppYBfAu85\n5f3OXHt15xxaf47Wzeaj7zFWDXQHrjt2fkzrErTeat6vAXZjrNPuzq4Elpj3lwBXuTCWyUA+Wp/Z\nDAw/l9ZfAxU/Km2vfq4E3ja/0b8HQlGq79kIo2snh/ZFAUUOj4vNsvbKneFCoAytcx3KBqLUNpTa\ngFIXOikOR3ebTdU3HZr5rqyjH7sFWO3w2JX15U71coJSA4BkYJNZ0tY+dTYNfI5SW1DqdrOsD1qX\nmPdLgT6uCQ0wVqJ0/JHmDnXWXv102HHX+ZODUutQamcbN9f8amvLqcU4h5YHZAkQi9bJwP3AUpTq\n4cS4XgUGA6PNWBad1fc+87iObfM40Ay8a5Z0fH11NkoFAR8D96J1Na7cpy2NR+sUYBpwF0pNaPGs\n1hojgTifsUzxDOAjs8Rd6uwEJ9VP55147xitLzmDVx0AYhweR5tlnKT8zP1UjEp5AVcDYxxe0wQ0\nmfe3oFQ+EA9s/tnxnGpcJ+J7A/jUfHSyunNOXEr9CrgcmGz+R3FOfZ1cx9fL6VDKGyMxvIvWywDQ\nuszhecd96lxaHzD/PYRSyzFOyZWhVF+0LjFPixxySWxGwtp6vK7cpc7ar58OO+46f8vhzKwAZqOU\nL0oNBOKAdCADiEOpgeYviNnmth3tEmAPWhcfL1GqN0p5mvcHmTEWOCGWY+/veN5yJnBs5ER7dees\nuC4FHgJmoHW9Q7lr68t1x05rRv/V34HdaP0/DuXt7VPnUSoQpYKP34cpZhwrgJvMrW4CPnF6bIaW\nLXh3qDNDe/WzAphnjlpKBaocTj/9PC4fudCxvf4zNRRraNJQpmGtw3OPm6NLcjRMcyifbo7uyNfw\nuJPi/IeG3/yobJaGXRoyNWzVcIWT6+4dDTs0bNewQkPfn6w758SVp6HIrJdMDYvdor5cdey0Hcd4\nDdrcd8fqafpJ96nzYhtkjuTKMvfX42Z5uIYvNeRqWKchzAWxBWo4oiHEocz5dQbvaSjRYDW/v25t\nt36MUUqvmMfcDu04KvNn3mT6DCGEEK1019NKQgghTkKSgxBCiFYkOQghhGhFkoMQQohWJDkIIYRo\nRZKDEEKIViQ5CCGEaOX/AWDszscC494LAAAAAElFTkSuQmCC\n",
"text/plain": [
"