{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Logistic Regression\n", "\n", "Logistic Regression is a simple classification technique that analyzes the relationship between a quantitative variable x and a dichotomous categorical variable y. Similar to linear regression, this relationship is inferred by applying a linear transformation to the data\n", "\n", "$$\n", "Y = Xw\n", "$$\n", "\n", "# Binary-Cross-Entropy\n", "\n", "However, Logistic Regression is unique in the way that it **learns relationships.** Given that we have to classify y given x, a natural choice of performance/criterion is the Binary Cross Entropy Loss.\n", "\n", "In short, Cross Entropy is a popular choice of option to judge classification task models whose output is a probability mass function. It is computed by independently applying below function to each prediction\n", "\n", "$$\n", "L(\\hat{y},y)=y\\cdot -log(\\hat{y})^T\n", "$$\n", "\n", "And its binary re-formulation\n", "\n", "$$\n", "L(\\hat{y},y)=y\\cdot-log(\\hat{y})+(1-y)\\cdot(-log(1-\\hat{y}))\n", "$$\n", "\n", "\n", "In general, Cross-Entropy loss exponentially increases as our predictions diverge from the truth; conversely, as our predictions become infinitely close to our target, the loss equates to zero. \n", "\n", "Below graph models Binary-Cross-Entropy when target is either 1 or 0 at different levels of prediction\n", "\n", "\"image-20200508081156737\"\n", "\n", "\n", "\n", "We can derive the gradient of our Loss function w.r.t. our prediction for the backward pass by using some simple Calculus\n", "\n", "$$\n", "\\begin{split}\\frac{\\partial L(\\hat{y},y)}{\\partial \\hat{y}} & =\\frac{\\partial}{\\hat{y}}(y\\cdot-log(\\hat{y})) + \\frac{\\partial}{\\hat{y}}(1-y)\\cdot(-log(1-\\hat{y})) \\\\& =\\frac{-y}{\\hat{y}} - (\\frac{1-y}{1-\\hat{y}}* -1) \\\\& = \\frac{-y}{\\hat{y}} + \\frac{1-y}{1-\\hat{y}}\\end{split}\n", "$$\n", "\n", "Given that the Loss function is usually the last forward operation, it also becomes the first gradient we need to compute for the backward pass. For this reason, there is no incoming gradient that we need to worry about integrating.\n", "\n", "However, instead of calculating a backward pass of *each prediction* w.r.t. our weight parameters, we usually take the mean prediction confidence as a way to better gauge our model's performance. \n", "\n", "In this case, we will be calculating below gradients\n", "\n", "$$\n", "\\frac{\\partial }{\\partial \\hat{w}}avg(L(\\hat{y},y))\n", "$$\n", "\n", "\n", "Once our loss has been computed, we follow the general DL procedure of taking a \"step\" towards steepest descent by computing the gradient of our Loss/criterion function w.r.t. weight parameters\n", "\n", "$$\n", "w_j=w_j-\\alpha\\frac{\\partial }{\\partial w_j}L(w_j)\n", "$$\n", "\n", "# Sigmoid\n", "\n", "In order for us to use the Binary-Cross-Entropy as our criterion, we must first ensure that our model's output ranges between ```[0,1]```. Of coarse, once rounded, this ensures our binary prediction:\n", "\n", "* 1 = Yes\n", "* 0 = No\n", "\n", "We will do this by applying a ```Sigmoid``` layer before feeding our inputs to the Loss function.\n", "\n", "A sigmoid layers is an activation function that \"squeezes\" all our inputs to a range between ```[0,1]``` by applying below function\n", "\n", "$$\n", "\\sigma(y)=\\frac{1}{1+e^{-y}}\n", "$$\n", "\n", "\"Sigmoid\n", "\n", "One distinct property of the Sigmoid function is that its derivative can be calculated by a simple re-formulation of its forward operation\n", "\n", "$$\n", "\\frac{\\partial \\sigma}{\\partial y} = \\sigma(y)(1-\\sigma(y))\n", "$$\n", "\n", "\n", "Given that activation function is applied independently to each element, its derivative function will be equivalent for all inputs.\n", "\n", "$$\n", "\\sigma(y) = \\sigma\\begin{pmatrix}y_1 & y_2 &y_3\\end{pmatrix} = \\begin{pmatrix}\\sigma(y_1) & \\sigma(y_2) & \\sigma(y_3)\\end{pmatrix} \\\\\\frac{\\partial \\sigma}{\\partial y} = \\begin{pmatrix}\\sigma(y_1)(1-\\sigma(y_1) & \\sigma(y_2)(1-\\sigma(y_2) &\\sigma(y_3)(1-\\sigma(y_3)\\end{pmatrix}\n", "$$\n", "\n", "Further, given that Sigmoid introduces no new parameters, its backward pass classifies as an intermediate operation. As a result, we can integrate the latest incoming gradient of the chain rule ($\\frac{\\partial L}{\\partial \\sigma}$) with the partial of our sigmoid function ($\\frac{\\partial \\sigma}{\\partial y}$) by a simple Hadamard product\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial y}=\\frac{\\partial L}{\\partial \\sigma}\\odot \\frac{\\partial \\sigma}{\\partial y}\n", "$$\n", "\n", "**NOTE:** if the above statements do not make much sense, make sure to review the [Linear Layer]() and/or [ReLU]() tutorial where I expand on such concepts" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Build Logistic Regression\n", "\n", "Now that we have defined all our needed methods, we will now manually implement the forward/backward pass of each operation using PyTorch's capabilities.\n", "\n", "**NOTE:** We will not go in-depth on our Linear Layer implementation as this was done on this [tutorial](Linear Layer.ipynb)\n" ] }, { "cell_type": "code", "execution_count": 220, "metadata": {}, "outputs": [], "source": [ "import torch\n", "torch.randn((2,2)).cuda()\n", "import torch.nn as nn" ] }, { "cell_type": "code", "execution_count": 221, "metadata": {}, "outputs": [], "source": [ "####################### Linear Layer ###################\n", "\n", "\n", "class Linear_Layer(torch.autograd.Function):\n", " \"\"\"\n", " Define a Linear Layer operation\n", " \"\"\"\n", " @staticmethod\n", " def forward(ctx, input,weights, bias = None):\n", " \"\"\"\n", " In the forward pass, we feed this class all necessary objects to \n", " compute a linear layer (input, weights, and bias)\n", " \"\"\"\n", " # input.dim = (B, in_dim)\n", " # weights.dim = (in_dim, out_dim)\n", " \n", " # given that the grad(output) wrt weight parameters equals the input,\n", " # we will save it to use for backpropagation\n", " ctx.save_for_backward(input, weights, bias)\n", " \n", " \n", " # linear transformation\n", " # (B, out_dim) = (B, in_dim) * (in_dim, out_dim)\n", " output = torch.mm(input, weights)\n", " \n", " if bias is not None:\n", " # bias.shape = (out_dim)\n", " \n", " # expanded_bias.shape = (B, out_dim), repeats bias B times\n", " expanded_bias = bias.unsqueeze(0).expand_as(output)\n", " \n", " # element-wise addition\n", " output += expanded_bias\n", " \n", " return output\n", "\n", " \n", " @staticmethod\n", " def backward(ctx, incoming_grad):\n", " \"\"\"\n", " In the backward pass we receive a Tensor (output_grad) containing the \n", " gradient of the loss with respect to our f(x) output, \n", " and we now need to compute the gradient of the loss\n", " with respect to our defined function.\n", " \"\"\"\n", " # incoming_grad.shape = (B, out_dim)\n", " \n", " # extract inputs from forward pass\n", " input, weights, bias = ctx.saved_tensors \n", " \n", " # assume none of the inputs need gradients\n", " grad_input = grad_weight = grad_bias = None\n", " \n", " \n", " # if input requires grad\n", " if ctx.needs_input_grad[0]:\n", " # (B, in_dim) = (B, out_dim) * (out_dim, in_dim)\n", " grad_input = incoming_grad.mm(weights.t())\n", " \n", " # if weights require grad\n", " if ctx.needs_input_grad[1]:\n", " # (out_dim, in_dim) = (out_dim, B) * (B, in_dim) \n", " grad_weight = incoming_grad.t().mm(input)\n", " \n", " # if bias requires grad\n", " if bias is not None and ctx.needs_input_grad[2]:\n", " # torch.ones((1,B)).mm(incoming_grad) \n", " # (out) = (1,B)*(B,out_dim)\n", " grad_bias = incoming_grad.sum(0)\n", " \n", " \n", " # below, if any of the grads = None, they will simply be ignored\n", " \n", " # add grad_output.t() to match original layout of weight parameter\n", " return grad_input, grad_weight.t(), grad_bias\n", " \n", " " ] }, { "cell_type": "code", "execution_count": 222, "metadata": {}, "outputs": [], "source": [ "class Linear(nn.Module):\n", " def __init__(self, in_dim, out_dim, bias = True):\n", " super().__init__()\n", " self.in_dim = in_dim\n", " self.out_dim = out_dim\n", " \n", " # define parameters\n", " \n", " # weight parameter\n", " self.weight = nn.Parameter(torch.randn((in_dim, out_dim)))\n", " \n", " # bias parameter\n", " if bias:\n", " self.bias = nn.Parameter(torch.randn((out_dim)))\n", " else:\n", " # register parameter as None if not initialized\n", " self.register_parameter('bias',None)\n", " \n", " def forward(self, input):\n", " output = Linear_Layer.apply(input, self.weight, self.bias)\n", " return output" ] }, { "cell_type": "code", "execution_count": 223, "metadata": {}, "outputs": [], "source": [ "################## Sigmoid Layer #######################\n", "\n", "# Remember that our incoming gradient will be of equal dims as our output\n", "# b/c of this, output now becomes an intermediate variable\n", "# input.shape == out.shape == incoming_gradient.shape\n", "\n", "import torch.nn as nn\n", "import torch\n", "\n", "class sigmoid_layer(torch.autograd.Function):\n", " \n", " def __init__(self):\n", " ''\n", " \n", " def sigmoid(self,x):\n", " sig = 1 / (1 + (-1*x).exp())\n", " return sig\n", " \n", " # forward pass\n", " def forward(self, input):\n", " # save input for backward() pass \n", " self.save_for_backward(input) \n", " activated_input = self.sigmoid(input)\n", " return activated_input\n", "\n", " # integrate backward pass with incoming_grad\n", " def backward(self, incoming_grad):\n", " \"\"\"\n", " In the backward pass we receive a Tensor containing the \n", " gradient of the loss with respect to our f(x) output, \n", " and we need to compute the gradient of the loss\n", " with respect to the input.\n", " \"\"\"\n", " input, = self.saved_tensors\n", " chained_grad = (self.sigmoid(input) * (1- self.sigmoid(input))) * incoming_grad\n", " return chained_grad" ] }, { "cell_type": "code", "execution_count": 224, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0.7311], grad_fn=)" ] }, "execution_count": 224, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test forward pass\n", "\n", "weight = torch.tensor([1.], requires_grad = True)\n", "input = torch.tensor([1.])\n", "x = input * weight\n", "sig = sigmoid_layer()(x)\n", "sig" ] }, { "cell_type": "code", "execution_count": 225, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0.1966])" ] }, "execution_count": 225, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test backward pass\n", "\n", "sig.backward(torch.tensor([1.]))\n", "weight.grad" ] }, { "cell_type": "code", "execution_count": 226, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0.7311], grad_fn=)" ] }, "execution_count": 226, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# compare output with PyTorch's inherent Method\n", "\n", "weight = torch.tensor([1.], requires_grad = True)\n", "input = torch.tensor([1.])\n", "x = input * weight\n", "\n", "sig = nn.Sigmoid()(x)\n", "sig" ] }, { "cell_type": "code", "execution_count": 227, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0.1966])" ] }, "execution_count": 227, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sig.backward()\n", "weight.grad" ] }, { "cell_type": "code", "execution_count": 228, "metadata": {}, "outputs": [], "source": [ "# Wrap ReLU_layer function in nn.module\n", "\n", "class Sigmoid(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", "\n", " \n", " def forward(self, input):\n", " output = sigmoid_layer()(input)\n", " return output\n", " " ] }, { "cell_type": "code", "execution_count": 229, "metadata": {}, "outputs": [], "source": [ "#################### Binary Cross Entropy ################\n", "\n", "# inputs must all be of type .float()\n", "class BCE_loss(torch.autograd.Function):\n", " \n", "\n", " @staticmethod\n", " def forward(self, yhat, y):\n", " # save input for backward() pass \n", " self.save_for_backward(y,yhat) \n", " loss = - (y * yhat.log() + (1-y)* (1-yhat).log())\n", " return loss\n", "\n", " @staticmethod\n", " def backward(self, output_grad):\n", " y,yhat = self.saved_tensors\n", " chained_grad = ((yhat-y) / (yhat * (1- yhat)))\n", " \n", " # y does not need gradient and thus we pass None to signify this\n", " return chained_grad, None" ] }, { "cell_type": "code", "execution_count": 230, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([0.6931], grad_fn=)" ] }, "execution_count": 230, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test above method\n", "output = torch.tensor([.50], requires_grad = True)\n", "y = torch.tensor([1.])\n", "loss = BCE_loss.apply(output,y)\n", "loss" ] }, { "cell_type": "code", "execution_count": 231, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([-2.])" ] }, "execution_count": 231, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test backward() method\n", "loss.backward()\n", "output.grad" ] }, { "cell_type": "code", "execution_count": 232, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.6931, grad_fn=)" ] }, "execution_count": 232, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test with PyTorch\n", "output = torch.tensor([.50], requires_grad = True)\n", "y = torch.tensor([1.])\n", "bce = nn.BCELoss()\n", "loss = bce(output,y)\n", "loss" ] }, { "cell_type": "code", "execution_count": 233, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor([-2.])" ] }, "execution_count": 233, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# test backward() method\n", "loss.backward()\n", "output.grad" ] }, { "cell_type": "code", "execution_count": 234, "metadata": {}, "outputs": [], "source": [ "# Wrap BCELoss function in nn.module\n", "\n", "class BCELoss(nn.Module):\n", " def __init__(self, reduction = 'mean'):\n", " super().__init__()\n", "\n", " \n", " def forward(self, pred, target):\n", " output = BCE_loss.apply(pred,target)\n", " # reduce output by average\n", " output = output.mean()\n", " return output\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have all of our \"ingredients\", we can now create our Logistic model" ] }, { "cell_type": "code", "execution_count": 235, "metadata": {}, "outputs": [], "source": [ "# Create Logistic Regression function\n", "class LogisticRegression(nn.Module):\n", " def __init__(self, input_dim = 30):\n", " super().__init__()\n", " self.linear = Linear(input_dim, 1) \n", " self.sigmoid = Sigmoid()\n", " \n", " def forward(self,x):\n", " # output.shape = (B, 1)\n", " output = self.sigmoid(self.linear(x))\n", " return output.view(-1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Wisconcin Breast Cancer Dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To showcase our Linear Regression, we will train our model to differentiate between malignant and benign cancer cells, given the characteristics of the cell nuclei. \n", "\n", "Refer to [link](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)) to learn more about the data" ] }, { "cell_type": "code", "execution_count": 236, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
radius_meantexture_meanperimeter_meanarea_meansmoothness_meancompactness_meanconcavity_meanpoints_meansymmetry_meandimension_mean...radius_worsttexture_worstperimeter_worstarea_worstsmoothness_worstcompactness_worstconcavity_worstpoints_worstsymmetry_worstdimension_worst
diagnosis
B12.3212.3978.85464.10.102800.069810.039870.037000.19590.05955...13.5015.6486.97549.10.13850.12660.124200.093910.28270.06771
B10.6018.9569.28346.40.096880.114700.063870.026420.19220.06491...11.8822.9478.28424.80.12130.25150.191600.079260.29400.07587
B11.0416.8370.92373.20.107700.078040.030460.024800.17140.06340...12.4126.4479.93471.40.13690.14820.106700.074310.29980.07881
B11.2813.3973.00384.80.116400.113600.046350.047960.17710.06072...11.9215.7776.53434.00.13670.18220.086690.086110.21020.06784
B15.1913.2197.65711.80.079630.069340.033930.026570.17210.05544...16.2015.73104.50819.10.11260.17370.136200.081780.24870.06766
\n", "

5 rows × 30 columns

\n", "
" ], "text/plain": [ " radius_mean texture_mean perimeter_mean area_mean \\\n", "diagnosis \n", "B 12.32 12.39 78.85 464.1 \n", "B 10.60 18.95 69.28 346.4 \n", "B 11.04 16.83 70.92 373.2 \n", "B 11.28 13.39 73.00 384.8 \n", "B 15.19 13.21 97.65 711.8 \n", "\n", " smoothness_mean compactness_mean concavity_mean points_mean \\\n", "diagnosis \n", "B 0.10280 0.06981 0.03987 0.03700 \n", "B 0.09688 0.11470 0.06387 0.02642 \n", "B 0.10770 0.07804 0.03046 0.02480 \n", "B 0.11640 0.11360 0.04635 0.04796 \n", "B 0.07963 0.06934 0.03393 0.02657 \n", "\n", " symmetry_mean dimension_mean ... radius_worst texture_worst \\\n", "diagnosis ... \n", "B 0.1959 0.05955 ... 13.50 15.64 \n", "B 0.1922 0.06491 ... 11.88 22.94 \n", "B 0.1714 0.06340 ... 12.41 26.44 \n", "B 0.1771 0.06072 ... 11.92 15.77 \n", "B 0.1721 0.05544 ... 16.20 15.73 \n", "\n", " perimeter_worst area_worst smoothness_worst compactness_worst \\\n", "diagnosis \n", "B 86.97 549.1 0.1385 0.1266 \n", "B 78.28 424.8 0.1213 0.2515 \n", "B 79.93 471.4 0.1369 0.1482 \n", "B 76.53 434.0 0.1367 0.1822 \n", "B 104.50 819.1 0.1126 0.1737 \n", "\n", " concavity_worst points_worst symmetry_worst dimension_worst \n", "diagnosis \n", "B 0.12420 0.09391 0.2827 0.06771 \n", "B 0.19160 0.07926 0.2940 0.07587 \n", "B 0.10670 0.07431 0.2998 0.07881 \n", "B 0.08669 0.08611 0.2102 0.06784 \n", "B 0.13620 0.08178 0.2487 0.06766 \n", "\n", "[5 rows x 30 columns]" ] }, "execution_count": 236, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# import data\n", "import pandas as pd\n", "url = 'https://raw.githubusercontent.com/PacktPublishing/Machine-Learning-with-R-Second-Edition/master/Chapter%2003/wisc_bc_data.csv'\n", "df = pd.read_csv(url)\n", "df.index = df.diagnosis\n", "df.drop(columns = ['diagnosis','id'],inplace = True)\n", "df.head()" ] }, { "cell_type": "code", "execution_count": 237, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Index: 569 entries, B to M\n", "Data columns (total 30 columns):\n", "radius_mean 569 non-null float64\n", "texture_mean 569 non-null float64\n", "perimeter_mean 569 non-null float64\n", "area_mean 569 non-null float64\n", "smoothness_mean 569 non-null float64\n", "compactness_mean 569 non-null float64\n", "concavity_mean 569 non-null float64\n", "points_mean 569 non-null float64\n", "symmetry_mean 569 non-null float64\n", "dimension_mean 569 non-null float64\n", "radius_se 569 non-null float64\n", "texture_se 569 non-null float64\n", "perimeter_se 569 non-null float64\n", "area_se 569 non-null float64\n", "smoothness_se 569 non-null float64\n", "compactness_se 569 non-null float64\n", "concavity_se 569 non-null float64\n", "points_se 569 non-null float64\n", "symmetry_se 569 non-null float64\n", "dimension_se 569 non-null float64\n", "radius_worst 569 non-null float64\n", "texture_worst 569 non-null float64\n", "perimeter_worst 569 non-null float64\n", "area_worst 569 non-null float64\n", "smoothness_worst 569 non-null float64\n", "compactness_worst 569 non-null float64\n", "concavity_worst 569 non-null float64\n", "points_worst 569 non-null float64\n", "symmetry_worst 569 non-null float64\n", "dimension_worst 569 non-null float64\n", "dtypes: float64(30)\n", "memory usage: 137.8+ KB\n" ] } ], "source": [ "df.info(verbose = True)" ] }, { "cell_type": "code", "execution_count": 238, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEJCAYAAACOr7BbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAa+klEQVR4nO3de3BU5R3/8c9ubhBiks1uAANBDRAYFEsgVAJy36oFLww/S6VefohMqrFQZGSwdio4qI2GGIhEcUTwZ73grcaqtXTW1GQktYYmVIEKREWFJM1l1wQCmMvu7w9wJzEPGgnZDcn7NeNMztnznPNN5nE/PM+5WXw+n08AAHyHNdgFAAB6JgICAGBEQAAAjAgIAIARAQEAMCIgAABGocEu4GyqqKgIdgkAcE5JSEg47WeMIAAARgQEAMCIgAAAGBEQAAAjAgIAYERAAACMCAgAgBEBAQAwIiAAAEa96k7qrqpcuSTYJaAHOj9rc7BLAIIiIAHR1NSk1atXq6WlRa2trZo0aZIWLFigvLw87d27V5GRkZKkO++8UxdeeKF8Pp+2bt2qsrIyRUREKCMjQ0lJSYEoFQBwSkACIiwsTKtXr1a/fv3U0tKi++67T+PGjZMk3XzzzZo0aVK77cvKylRVVaXc3FwdOHBAmzdv1kMPPRSIUgEApwTkHITFYlG/fv0kSa2trWptbZXFYjnt9jt37tS0adNksViUnJysxsZGeTyeQJQKADglYOcgvF6vVq1apaqqKl155ZUaOXKk/v73v+vFF1/Uq6++qksuuUQ33nijwsLC5Ha75XA4/G3tdrvcbrdsNlu7fbpcLrlcLklSZmZmuzZnorJLrdFbdbVfAeeqgAWE1WpVVlaWGhsbtW7dOn355Zf61a9+pdjYWLW0tOjJJ5/UG2+8oeuvv14+n69De9OIw+l0yul0+pdra2u79XdA30S/Qm/Wox73PWDAAI0ZM0a7du2SzWaTxWJRWFiYZs6cqfLyckknRwxt/6esq6vrMHoAAHSvgAREQ0ODGhsbJZ28ounjjz/WkCFD/OcVfD6fSkpKlJiYKElKTU1VUVGRfD6f9u/fr8jISAICAAIsIFNMHo9HeXl58nq98vl8SktL04QJE3T//feroaFBknTBBRcoPT1dkpSSkqLS0lItW7ZM4eHhysjICESZAIA2LD7ThP85qquvHOVGOZhwoxx6sx51DgIAcG4gIAAARgQEAMCIgAAAGBEQAAAjAgIAYERAAACMCAgAgBEBAQAwIiAAAEYEBADAiIAAABgREAAAIwICAGBEQAAAjAgIAIARAQEAMCIgAABGBAQAwIiAAAAYhQbiIE1NTVq9erVaWlrU2tqqSZMmacGCBaqurtb69et19OhRXXTRRVq6dKlCQ0PV3NysjRs36rPPPtN5552n5cuXa+DAgYEoFQBwSkBGEGFhYVq9erWysrL0yCOPaNeuXdq/f7+ee+45zZ07V7m5uRowYIAKCgokSQUFBRowYIAee+wxzZ07V88//3wgygQAtBGQgLBYLOrXr58kqbW1Va2trbJYLNqzZ48mTZokSZoxY4ZKSkokSTt37tSMGTMkSZMmTdLu3bvl8/kCUSoA4JSATDFJktfr1apVq1RVVaUrr7xSgwYNUmRkpEJCQiRJcXFxcrvdkiS32y273S5JCgkJUWRkpI4cOaLo6Oh2+3S5XHK5XJKkzMxMORyOLtVY2aXW6K262q+Ac1XAAsJqtSorK0uNjY1at26dDh8+fNptTaMFi8XSYZ3T6ZTT6fQv19bWnp1igTboV+jNEhISTvtZwK9iGjBggMaMGaMDBw7o2LFjam1tlXRy1BAXFydJstvtqqurk3RySurYsWOKiooKdKkA0KcFJCAaGhrU2Ngo6eQVTR9//LGGDBmiiy++WB988IEk6b333lNqaqokacKECXrvvfckSR988IEuvvhi4wgCANB9AjLF5PF4lJeXJ6/XK5/Pp7S0NE2YMEFDhw7V+vXrtW3bNl100UWaNWuWJGnWrFnauHGjli5dqqioKC1fvjwQZQIA2rD4etHlQRUVFV1qX7lyyVmqBL3J+Vmbg10C0G161DkIAMC5gYAAABgREAAAIwICAGBEQAAAjAgIAIARAQEAMCIgAABGBAQAwIiAAAAYERAAACMCAgBgREAAAIwICACAEQEBADAiIAAARgQEAMCIgAAAGBEQAAAjAgIAYBQaiIPU1tYqLy9PX3/9tSwWi5xOp+bMmaOXX35Z7777rqKjoyVJCxcu1Pjx4yVJr7/+ugoKCmS1WnXrrbdq3LhxgSgVAHBKQAIiJCREN998s5KSknT8+HHdc889uvTSSyVJc+fO1bXXXttu+0OHDqm4uFiPPvqoPB6P1q5dqw0bNshqZcADAIESkG9cm82mpKQkSVL//v01ZMgQud3u025fUlKiyZMnKywsTAMHDtTgwYNVXl4eiFIBAKcEZATRVnV1tT7//HONGDFCn3zyibZv366ioiIlJSXplltuUVRUlNxut0aOHOlvExcXZwwUl8sll8slScrMzJTD4ehSbZVdao3eqqv9CjhXBTQgTpw4oezsbC1atEiRkZG64oordP3110uSXnrpJT377LPKyMiQz+fr1P6cTqecTqd/uba2tlvqRt9Gv0JvlpCQcNrPAjap39LSouzsbE2dOlWXXXaZJCk2NlZWq1VWq1WzZ8/Wp59+Kkmy2+2qq6vzt3W73YqLiwtUqQAABSggfD6fNm3apCFDhujqq6/2r/d4PP6fP/zwQyUmJkqSUlNTVVxcrObmZlVXV6uyslIjRowIRKkAgFMCMsW0b98+FRUVadiwYVq5cqWkk5e07tixQwcPHpTFYlF8fLzS09MlSYmJiUpLS9OKFStktVp12223cQUTAASYxdfZCf9zQEVFRZfaV65ccpYqQW9yftbmYJcAdJsecQ4CAHBuISAAAEYEBADAiIAAABgREAAAIwICAGBEQAAAjAgIAIARAQEAMCIgAABGBAQAwIiAAAAYERAAACMCAgBgREAAAIwICACAUaffKPeXv/xF1157bYf1b731VrvXiAI4+xb9v38GuwT0QM/837Ru3X+nRxCvvfbaj1oPADi3/eAIYvfu3ZIkr9fr//lb//vf/9S/f//uqQwAEFQ/GBBPPPGEJKmpqcn/syRZLBbFxsZq8eLF3VcdACBofjAg8vLyJEkbN27Ub37zmzM6SG1trfLy8vT111/LYrHI6XRqzpw5Onr0qHJyclRTU6P4+HjdddddioqKks/n09atW1VWVqaIiAhlZGQoKSnpjI4NADgznT4H0TYcvF5vu/9+SEhIiG6++Wbl5OTowQcf1Pbt23Xo0CHl5+dr7Nixys3N1dixY5Wfny9JKisrU1VVlXJzc5Wenq7Nmzefwa8GAOiKTl/F9Nlnn+npp5/Wl19+qaampnafvfTSS9/b1mazyWazSZL69++vIUOGyO12q6SkRGvWrJEkTZ8+XWvWrNFNN92knTt3atq0abJYLEpOTlZjY6M8Ho9/HwCA7tfpgMjLy9OECRN0xx13KCIi4owPWF1drc8//1wjRoxQfX29/0vfZrOpoaFBkuR2u+VwOPxt7Ha73G53h4BwuVxyuVySpMzMzHZtzkRll1qjt+pqvwK6S3f3zU4HRG1trRYuXCiLxXLGBztx4oSys7O1aNEiRUZGnnY7n8/XYZ3puE6nU06ns12NwNlGv0JPdTb6ZkJCwmk/6/Q5iIkTJ+o///nPGRfR0tKi7OxsTZ06VZdddpkkKSYmRh6PR5Lk8XgUHR0t6eSIoe0vXldXx/QSAARYp0cQzc3NWrdunUaPHq3Y2Nh2n/3Q1U0+n0+bNm3SkCFD2t11nZqaqsLCQs2bN0+FhYWaOHGif/3f/vY3TZkyRQcOHFBkZCQBAQAB1umAGDp0qIYOHXpGB9m3b5+Kioo0bNgwrVy5UpK0cOFCzZs3Tzk5OSooKJDD4dCKFSskSSkpKSotLdWyZcsUHh6ujIyMMzouAODMWXymCf9zVEVFRZfaV65ccpYqQW9yflbwL7PmWUwwORvPYvq+cxCdHkF89zEbbV1yySU/riIAQI/X6YBo+5gNSWpoaFBLS4vsdrs2btx41gsDAATXj7oPoi2v16vXXnuNh/UBQC91xi8Mslqtmj9/vt54442zWQ8AoIfo0hvlPvroI1mtvJQOAHqjTk8x3XHHHe2Wm5qa1NTUpCVLuPIHAHqjTgfE0qVL2y1HRETo/PPP/95HZgAAzl2dDogxY8ZIOnlyur6+XjExMUwvAUAv1umAOH78uJ5++mkVFxertbVVISEhmjx5shYvXswoAgB6oU4PAbZs2aITJ05o3bp1eu6557Ru3To1NTVpy5Yt3VkfACBIOh0Qu3bt0tKlS5WQkKCwsDAlJCQoIyOjS094BQD0XJ0OiPDwcP8Lfb7V0NCg0NBOz1IBAM4hnf52nzVrlh544AHNnTtX8fHxqqmp0dtvv63Zs2d3Z30AgCDpdEDMnz9fcXFxev/99+V2uxUXF6frrrtOs2bN6s76AABB0umA2Lp1q6ZMmaI//OEP/nX79u3TM888o0WLFnVHbQCAIOr0OYgdO3Zo+PDh7dYlJSXp/fffP+tFAQCCr9MBYbFY5PV6263zer3qRe8bAgC00emAGD16tLZt2+YPCa/Xq1deeUWjR4/utuIAAMHT6XMQt956qzIzM/XrX/9aDodDtbW1stlsWrVqVXfWBwAIkk4HhN1u18MPP6zy8nLV1dXJbrdrxIgRPI8JAHqpH3WXm9VqVXJy8o8+yOOPP67S0lLFxMQoOztbkvTyyy/r3XffVXR0tCRp4cKFGj9+vCTp9ddfV0FBgaxWq2699VaNGzfuRx8TANA1AbkNesaMGbrqqqs6vLZ07ty5uvbaa9utO3TokIqLi/Xoo4/K4/Fo7dq12rBhAyMVAAiwgHzrjhkzRlFRUZ3atqSkRJMnT1ZYWJgGDhyowYMHq7y8vJsrBAB8V1AfpLR9+3YVFRUpKSlJt9xyi6KiouR2uzVy5Ej/NnFxcXK73cb2LpdLLpdLkpSZmSmHw9Gleiq71Bq9VVf7FdBdurtvBi0grrjiCl1//fWSpJdeeknPPvusMjIyftR9FU6nU06n079cW1t71usE6Ffoqc5G30xISDjtZ0Gb2I+NjZXVapXVatXs2bP16aefSjp5tVRdXZ1/u2+f+wQACKygBYTH4/H//OGHHyoxMVGSlJqaquLiYjU3N6u6ulqVlZUaMWJEsMoEgD4rIFNM69ev1969e3XkyBHdfvvtWrBggfbs2aODBw/KYrEoPj5e6enpkqTExESlpaVpxYoVslqtuu2227iCCQCCICABsXz58g7rvu8x4fPnz9f8+fO7syQAwA/gn+YAACMCAgBgREAAAIwICACAEQEBADAiIAAARgQEAMCIgAAAGBEQAAAjAgIAYERAAACMCAgAgBEBAQAwIiAAAEYEBADAiIAAABgREAAAIwICAGBEQAAAjAgIAIBRaCAO8vjjj6u0tFQxMTHKzs6WJB09elQ5OTmqqalRfHy87rrrLkVFRcnn82nr1q0qKytTRESEMjIylJSUFIgyAQBtBGQEMWPGDN17773t1uXn52vs2LHKzc3V2LFjlZ+fL0kqKytTVVWVcnNzlZ6ers2bNweiRADAdwQkIMaMGaOoqKh260pKSjR9+nRJ0vTp01VSUiJJ2rlzp6ZNmyaLxaLk5GQ1NjbK4/EEokwAQBsBmWIyqa+vl81mkyTZbDY1NDRIktxutxwOh387u90ut9vt37Ytl8sll8slScrMzGzX7kxUdqk1equu9iugu3R33wxaQJyOz+frsM5isRi3dTqdcjqd/uXa2tpuqwt9F/0KPdXZ6JsJCQmn/SxoVzHFxMT4p448Ho+io6MlnRwxtP2l6+rqjKMHAED3ClpApKamqrCwUJJUWFioiRMn+tcXFRXJ5/Np//79ioyMJCAAIAgCMsW0fv167d27V0eOHNHtt9+uBQsWaN68ecrJyVFBQYEcDodWrFghSUpJSVFpaamWLVum8PBwZWRkBKJEAMB3BCQgli9fblx/3333dVhnsVi0ZMmS7i4JAPADuJMaAGBEQAAAjAgIAIARAQEAMCIgAABGBAQAwIiAAAAYERAAACMCAgBgREAAAIwICACAEQEBADAiIAAARgQEAMCIgAAAGBEQAAAjAgIAYERAAACMCAgAgBEBAQAwCg12AXfeeaf69esnq9WqkJAQZWZm6ujRo8rJyVFNTY3i4+N11113KSoqKtilAkCfEvSAkKTVq1crOjrav5yfn6+xY8dq3rx5ys/PV35+vm666aYgVggAfU+PnGIqKSnR9OnTJUnTp09XSUlJkCsCgL6nR4wgHnzwQUnSz372MzmdTtXX18tms0mSbDabGhoajO1cLpdcLpckKTMzUw6Ho0t1VHapNXqrrvYroLt0d98MekCsXbtWcXFxqq+v1wMPPKCEhIROt3U6nXI6nf7l2tra7igRfRz9Cj3V2eib3/edG/Qppri4OElSTEyMJk6cqPLycsXExMjj8UiSPB5Pu/MTAIDACGpAnDhxQsePH/f//NFHH2nYsGFKTU1VYWGhJKmwsFATJ04MZpkA0CcFdYqpvr5e69atkyS1trbq8ssv17hx4zR8+HDl5OSooKBADodDK1asCGaZANAnBTUgBg0apKysrA7rzzvvPN13331BqAgA8K2gn4MAAPRMBAQAwIiAAAAYERAAACMCAgBgREAAAIwICACAEQEBADAiIAAARgQEAMCIgAAAGBEQAAAjAgIAYERAAACMCAgAgBEBAQAwIiAAAEYEBADAiIAAABgREAAAo9BgF/B9du3apa1bt8rr9Wr27NmaN29esEsCgD6jx44gvF6vnn76ad17773KycnRjh07dOjQoWCXBQB9Ro8NiPLycg0ePFiDBg1SaGioJk+erJKSkmCXBQB9Ro+dYnK73bLb7f5lu92uAwcOtNvG5XLJ5XJJkjIzM5WQkNClYyY8/9cutQe6y99/93+CXQL6oB47gvD5fB3WWSyWdstOp1OZmZnKzMwMVFl9xj333BPsEgAj+mbg9NiAsNvtqqur8y/X1dXJZrMFsSIA6Ft6bEAMHz5clZWVqq6uVktLi4qLi5WamhrssgCgz+ix5yBCQkK0ePFiPfjgg/J6vZo5c6YSExODXVaf4XQ6g10CYETfDByLzzTZDwDo83rsFBMAILgICACAUY89B4HA++Uvf6lhw4ZJkqxWqxYvXqxRo0YFuSpAWrBggaZOnaqlS5dKklpbW5Wenq6RI0dy2Ws3IiDgFx4erqysLEknn4P1wgsv6P777w9yVYAUERGhr776Sk1NTQoPD9dHH32kuLi4YJfV6zHFBKPjx49rwIABwS4D8Bs3bpxKS0slSTt27NCUKVOCXFHvxwgCfk1NTVq5cqWam5vl8Xi0evXqYJcE+E2ZMkWvvvqqxo8fry+++EIzZ87UJ598EuyyejUCAn5tp5j279+vjRs3Kjs7u8MjToBguOCCC1RTU6MdO3YoJSUl2OX0CUwxwSg5OVlHjhxRQ0NDsEsB/FJTU/WnP/1Jl19+ebBL6RMYQcDo8OHD8nq9Ou+884JdCuA3c+ZMRUZGatiwYdqzZ0+wy+n1CAj4fXsO4lt33nmnrFYGmeg57Ha75syZE+wy+gwetQEAMOKfhwAAIwICAGBEQAAAjAgIAIARAQEAMCIggFPy8vK0bds2/fe//9Vvf/vbYJdzWn/+85+1adOmYJeBPoDLXIFT8vLyZLfbdcMNNwS7FKBHYAQBADDiTmr0WZ9//rk2bdqkyspKpaSk+B9KuGfPHj322GP+aZz8/Hy9++67qq+vl91u18KFC/XTn/5UkuT1evXcc8+psLBQ/fr10zXXXKMtW7boxRdfVEhIiNasWaPRo0drz549+uKLL5ScnKxly5YpOjpakrRz50698MILcrvduvDCC7VkyRINHTrUf9x33nlHx48fl81m05IlSzR27Fi9/PLLqqqq0rJly9TU1KRNmzZp165d8nq9Ov/887Vq1SrFxsYG4S+K3oaAQJ/U0tKirKwszZkzR1dddZV27typDRs26Lrrruuw7aBBg3T//fcrNjZWH3zwgR577DHl5ubKZrPJ5XKprKxMjzzyiCIiIpSTk9Oh/Y4dO/S73/1ODodDDz30kN58803deOONqqio0IYNG7Ry5UqNGTNGb7/9th5++GHl5OSourpa27dv1x//+EfFxcWpurpaXq+3w74LCwt17NgxPfHEEwoLC9PBgwcVHh7eLX8z9D1MMaFP2r9/v1pbWzV37lyFhoZq0qRJGj58uHHbtLQ0xcXFyWq1avLkyRo8eLDKy8slSf/85z81Z84c2e12RUVFGQNmxowZSkhIUHh4uNLS0nTw4EFJUnFxsVJSUnTppZcqNDRU11xzjZqamrRv3z5ZrVY1Nzfr0KFDamlp0cCBAzV48OAO+w4JCdHRo0dVVVUlq9WqpKQkRUZGnr0/FPo0RhDokzwej+Li4tq968LhcBi3LSws1FtvvaWamhpJ0okTJ3TkyBH/fux2+/fuo+10T0REhE6cOOFvGx8f7//MarXK4XDI7Xbr4osv1qJFi/TKK6/o0KFD+slPfqJbbrmlw2s2p02bprq6Oq1fv17Hjh3T1KlTdcMNNyg0lP+10XWMINAn2Ww2ud1utb2Ir66ursN2NTU1evLJJ3Xbbbdpy5YteuaZZ5SYmOhv9+1+vlVbW/ujavg2dCTJ5/OptrbWHwKXX3651q5dq7y8PEnS888/32EfoaGh+sUvfqGcnBytXbtW//73v1VUVNTpGoDvQ0CgT0pOTpbVatU777yj1tZW/etf//JPG7X1zTffyGKx+E8q/+Mf/9BXX33l/zwtLU1//etf5Xa71djYqDfeeKPTNUyePFllZWX6+OOP1dLSojfffFNhYWEaNWqUKioqtHv3bjU3Nys8PFzh4eHGR6/v3r1bX375pbxeryIjIxUaGsoj2nHWMA5FnxQaGqq7775bTz75pLZt26aUlBT/lUltDR06VFdffbV+//vfy2q1atq0aRo1apT/89mzZ6uiokJ33323+vfvr5///Ofau3dvp76kExIStHTpUm3ZssV/FdOqVasUGhqq5uZmPf/88zp8+LBCQkI0atQopaend9jH119/raeeekput1v9+vVTWlqapk6d2rU/DnAKN8oBZ1FZWZmeeuopPf7448EuBegyxqJAFzQ1Nam0tFStra1yu9169dVXjSMR4FzECALogm+++UZr1qzR4cOHFR4ervHjx2vRokVcaopegYAAABgxxQQAMCIgAABGBAQAwIiAAAAYERAAAKP/D3jF3NSPttQOAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# visualize the distribution of our binary classes\n", "\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "plt.style.use('ggplot')\n", "\n", "sns.countplot(df.index);plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given that there is about twice more data on Benign cells than Malignant, a model will become bias towards classifying Benign cells" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Preprocessing\n", "\n" ] }, { "cell_type": "code", "execution_count": 239, "metadata": { "scrolled": false }, "outputs": [], "source": [ "# Separate features (X) from target (y)\n", "import numpy as np\n", "\n", "X = df.values\n", "y = (df.index == 'M')\n", "y = y.astype(np.double)" ] }, { "cell_type": "code", "execution_count": 240, "metadata": {}, "outputs": [], "source": [ "# normalize features\n", "from sklearn.preprocessing import normalize\n", "X = normalize(X, axis = 0)" ] }, { "cell_type": "code", "execution_count": 241, "metadata": {}, "outputs": [], "source": [ "# parse data to training and testing set for evaluation\n", "\n", "from sklearn.model_selection import train_test_split\n", "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = .20, random_state = 42, shuffle = True)" ] }, { "cell_type": "code", "execution_count": 242, "metadata": {}, "outputs": [], "source": [ "# Transform data to PyTorch tensors and separate data into batches\n", "from skorch.dataset import Dataset\n", "from torch.utils.data import DataLoader\n", "\n", "# Wrap each observation with its corresponding target\n", "train = Dataset(X_train,y_train) \n", "test = Dataset(X_test,y_test) \n", "\n", "# separate data into batches of 16\n", "train_dl = DataLoader(train, batch_size = 16, pin_memory = True)\n", "test_dl = DataLoader(test, batch_size = 16, pin_memory = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have all the data formatted, let's instatiate our model, criterion, and optimizer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Instantiate Logistic Regression" ] }, { "cell_type": "code", "execution_count": 243, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LogisticRegression(\n", " (linear): Linear()\n", " (sigmoid): Sigmoid()\n", ")" ] }, "execution_count": 243, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# instantiate model and place it on GPU\n", "\n", "device = torch.device('cuda')\n", "model = LogisticRegression(30).to(device)\n", "model" ] }, { "cell_type": "code", "execution_count": 244, "metadata": {}, "outputs": [], "source": [ "# initiate loss function\n", "criterion = BCELoss()" ] }, { "cell_type": "code", "execution_count": 245, "metadata": {}, "outputs": [], "source": [ "# initiate optimizer\n", "from torch import optim\n", "optimizer = optim.SGD(model.parameters(), lr = .01)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Make one forward pass to make sure everything works as it should" ] }, { "cell_type": "code", "execution_count": 246, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "batch_X.shape: torch.Size([16, 30])\n", "-----------------------------------\n", "batch_X.shape: torch.Size([16])\n" ] } ], "source": [ "# test train_dl\n", "batch_X,batch_y = next(iter(train_dl))\n", "print(f\"batch_X.shape: {batch_X.shape}\")\n", "print('-'*35)\n", "print(f\"batch_X.shape: {batch_y.shape}\")" ] }, { "cell_type": "code", "execution_count": 247, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "torch.Size([16])" ] }, "execution_count": 247, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Assert our model makes as many predictions according to the batch\n", "# all inputs must be of type .float()\n", "\n", "output = model(batch_X.cuda().float())\n", "output.shape" ] }, { "cell_type": "code", "execution_count": 248, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tensor(0.6896, device='cuda:0', grad_fn=)" ] }, "execution_count": 248, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# average loss\n", "loss = criterion(output,batch_y.cuda().float())\n", "loss" ] }, { "cell_type": "code", "execution_count": 249, "metadata": {}, "outputs": [], "source": [ "# compute gradients by calling .backward()\n", "loss.backward()" ] }, { "cell_type": "code", "execution_count": 250, "metadata": {}, "outputs": [], "source": [ "# take a step\n", "optimizer.step()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Train Logistic Regression\n", "\n", "Now that we have asserted our model works as should, it's time to train it" ] }, { "cell_type": "code", "execution_count": 251, "metadata": {}, "outputs": [], "source": [ "\n", "def train(model, iterator, optimizer, criterion):\n", " \n", " # hold avg loss and acc sum of all batches\n", " epoch_loss = 0\n", " epoch_acc = 0\n", " \n", " \n", " for batch in iterator:\n", " \n", " # zero-out all gradients (if any) from our model parameters\n", " model.zero_grad()\n", " \n", " \n", " \n", " # extract input and label\n", " \n", " # input.shape = (B, fetures)\n", " input = batch[0].cuda().float()\n", " # label.shape = (B)\n", " label = batch[1].cuda().float()\n", " \n", " \n", " # Start PyTorch's Dynamic Graph\n", " \n", " # predictions.shape = (B)\n", " predictions = model(input)\n", " \n", " # average batch loss \n", " loss = criterion(predictions, label)\n", " \n", " # calculate grad(loss) / grad(parameters)\n", " # \"clears\" PyTorch's dynamic graph\n", " loss.backward()\n", " \n", " \n", " # perform SGD \"step\" operation\n", " optimizer.step()\n", " \n", " \n", " # Given that PyTorch variables are \"contagious\" (they record all operations)\n", " # we need to \".detach()\" to stop them from recording any performance\n", " # statistics\n", " \n", " \n", " # average batch accuracy\n", " acc = binary_accuracy(predictions.detach(), label)\n", " \n", "\n", " \n", " # record our stats\n", " epoch_loss += loss.detach()\n", " epoch_acc += acc\n", " \n", " # NOTE: tense.item() unpacks Tensor item to a regular python object \n", " # tense.tensor([1]).item() == 1\n", " \n", " # return average loss and acc of epoch\n", " return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)\n" ] }, { "cell_type": "code", "execution_count": 252, "metadata": {}, "outputs": [], "source": [ "# compute average accuracy per batch\n", "\n", "def binary_accuracy(preds, y):\n", " # preds.shape = (B)\n", " # y.shape = (B)\n", "\n", " #round predictions to the closest integer\n", " rounded_preds = torch.round(preds)\n", " correct = (rounded_preds == y).sum()\n", " acc = correct.item() / len(y)\n", " return acc" ] }, { "cell_type": "code", "execution_count": 253, "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "def epoch_time(start_time, end_time):\n", " elapsed_time = end_time - start_time\n", " elapsed_mins = int(elapsed_time / 60)\n", " elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n", " return elapsed_mins, elapsed_secs\n", " \n", " " ] }, { "cell_type": "code", "execution_count": 254, "metadata": {}, "outputs": [], "source": [ "def evaluate(model, iterator, criterion):\n", " \n", " epoch_loss = 0\n", " epoch_acc = 0\n", " \n", " # turn off grad tracking as we are only evaluation performance\n", " with torch.no_grad():\n", " \n", " for batch in iterator:\n", "\n", " # extract input and label \n", " input = batch[0].cuda().float()\n", " label = batch[1].cuda().float()\n", "\n", "\n", " # predictions.shape = (B,1)\n", " predictions = model(input)\n", "\n", " # average batch loss \n", " loss = criterion(predictions, label)\n", "\n", " # average batch accuracy\n", " acc = binary_accuracy(predictions, label)\n", "\n", " epoch_loss += loss\n", " epoch_acc += acc\n", " \n", " return epoch_loss.item() / len(iterator), epoch_acc / len(iterator)" ] }, { "cell_type": "code", "execution_count": 255, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "---------------------------------------------------------------------------\n", "Epoch: 01 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.662 | Train Acc: 62.65%\n", "\t Val. Loss: 0.653 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 02 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.658 | Train Acc: 62.65%\n", "\t Val. Loss: 0.649 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 03 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.655 | Train Acc: 62.65%\n", "\t Val. Loss: 0.646 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 04 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.652 | Train Acc: 62.65%\n", "\t Val. Loss: 0.643 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 05 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.649 | Train Acc: 62.65%\n", "\t Val. Loss: 0.639 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 06 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.645 | Train Acc: 62.65%\n", "\t Val. Loss: 0.636 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 07 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.642 | Train Acc: 62.87%\n", "\t Val. Loss: 0.633 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 08 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.639 | Train Acc: 62.87%\n", "\t Val. Loss: 0.630 | Val. Acc: 63.28%\n", "---------------------------------------------------------------------------\n", "Epoch: 09 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.636 | Train Acc: 62.87%\n", "\t Val. Loss: 0.627 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 10 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.633 | Train Acc: 62.87%\n", "\t Val. Loss: 0.624 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 11 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.630 | Train Acc: 62.87%\n", "\t Val. Loss: 0.621 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 12 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.627 | Train Acc: 62.65%\n", "\t Val. Loss: 0.618 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 13 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.624 | Train Acc: 62.65%\n", "\t Val. Loss: 0.615 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 14 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.621 | Train Acc: 62.44%\n", "\t Val. Loss: 0.612 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 15 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.618 | Train Acc: 62.44%\n", "\t Val. Loss: 0.609 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 16 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.615 | Train Acc: 62.44%\n", "\t Val. Loss: 0.606 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 17 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.612 | Train Acc: 62.87%\n", "\t Val. Loss: 0.603 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 18 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.609 | Train Acc: 63.52%\n", "\t Val. Loss: 0.600 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 19 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.607 | Train Acc: 63.95%\n", "\t Val. Loss: 0.597 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 20 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.604 | Train Acc: 63.95%\n", "\t Val. Loss: 0.594 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 21 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.601 | Train Acc: 64.59%\n", "\t Val. Loss: 0.592 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 22 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.599 | Train Acc: 64.81%\n", "\t Val. Loss: 0.589 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 23 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.596 | Train Acc: 65.02%\n", "\t Val. Loss: 0.586 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 24 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.593 | Train Acc: 65.02%\n", "\t Val. Loss: 0.584 | Val. Acc: 64.06%\n", "---------------------------------------------------------------------------\n", "Epoch: 25 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.591 | Train Acc: 65.24%\n", "\t Val. Loss: 0.581 | Val. Acc: 64.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 26 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.588 | Train Acc: 65.46%\n", "\t Val. Loss: 0.578 | Val. Acc: 64.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 27 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.586 | Train Acc: 65.89%\n", "\t Val. Loss: 0.576 | Val. Acc: 64.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 28 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.583 | Train Acc: 66.32%\n", "\t Val. Loss: 0.573 | Val. Acc: 65.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 29 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.581 | Train Acc: 66.75%\n", "\t Val. Loss: 0.571 | Val. Acc: 65.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 30 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.578 | Train Acc: 66.96%\n", "\t Val. Loss: 0.568 | Val. Acc: 65.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 31 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.576 | Train Acc: 66.96%\n", "\t Val. Loss: 0.566 | Val. Acc: 66.41%\n", "---------------------------------------------------------------------------\n", "Epoch: 32 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.573 | Train Acc: 67.40%\n", "\t Val. Loss: 0.564 | Val. Acc: 68.75%\n", "---------------------------------------------------------------------------\n", "Epoch: 33 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.571 | Train Acc: 67.61%\n", "\t Val. Loss: 0.561 | Val. Acc: 68.75%\n", "---------------------------------------------------------------------------\n", "Epoch: 34 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.569 | Train Acc: 68.04%\n", "\t Val. Loss: 0.559 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 35 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.566 | Train Acc: 68.26%\n", "\t Val. Loss: 0.556 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 36 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.564 | Train Acc: 68.26%\n", "\t Val. Loss: 0.554 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 37 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.562 | Train Acc: 68.69%\n", "\t Val. Loss: 0.552 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 38 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.560 | Train Acc: 68.69%\n", "\t Val. Loss: 0.550 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 39 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.558 | Train Acc: 68.90%\n", "\t Val. Loss: 0.547 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 40 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.555 | Train Acc: 69.77%\n", "\t Val. Loss: 0.545 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 41 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.553 | Train Acc: 70.84%\n", "\t Val. Loss: 0.543 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 42 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.551 | Train Acc: 71.55%\n", "\t Val. Loss: 0.541 | Val. Acc: 69.53%\n", "---------------------------------------------------------------------------\n", "Epoch: 43 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.549 | Train Acc: 71.77%\n", "\t Val. Loss: 0.539 | Val. Acc: 70.31%\n", "---------------------------------------------------------------------------\n", "Epoch: 44 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.547 | Train Acc: 71.77%\n", "\t Val. Loss: 0.536 | Val. Acc: 70.31%\n", "---------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch: 45 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.545 | Train Acc: 71.98%\n", "\t Val. Loss: 0.534 | Val. Acc: 70.31%\n", "---------------------------------------------------------------------------\n", "Epoch: 46 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.543 | Train Acc: 72.20%\n", "\t Val. Loss: 0.532 | Val. Acc: 70.31%\n", "---------------------------------------------------------------------------\n", "Epoch: 47 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.541 | Train Acc: 73.06%\n", "\t Val. Loss: 0.530 | Val. Acc: 71.09%\n", "---------------------------------------------------------------------------\n", "Epoch: 48 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.539 | Train Acc: 73.06%\n", "\t Val. Loss: 0.528 | Val. Acc: 71.09%\n", "---------------------------------------------------------------------------\n", "Epoch: 49 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.537 | Train Acc: 73.06%\n", "\t Val. Loss: 0.526 | Val. Acc: 71.09%\n", "---------------------------------------------------------------------------\n", "Epoch: 50 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.535 | Train Acc: 73.49%\n", "\t Val. Loss: 0.524 | Val. Acc: 71.09%\n", "---------------------------------------------------------------------------\n", "Epoch: 51 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.533 | Train Acc: 73.71%\n", "\t Val. Loss: 0.522 | Val. Acc: 71.88%\n", "---------------------------------------------------------------------------\n", "Epoch: 52 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.531 | Train Acc: 73.92%\n", "\t Val. Loss: 0.520 | Val. Acc: 71.88%\n", "---------------------------------------------------------------------------\n", "Epoch: 53 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.529 | Train Acc: 74.14%\n", "\t Val. Loss: 0.518 | Val. Acc: 71.88%\n", "---------------------------------------------------------------------------\n", "Epoch: 54 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.527 | Train Acc: 74.14%\n", "\t Val. Loss: 0.516 | Val. Acc: 72.66%\n", "---------------------------------------------------------------------------\n", "Epoch: 55 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.525 | Train Acc: 74.57%\n", "\t Val. Loss: 0.514 | Val. Acc: 73.44%\n", "---------------------------------------------------------------------------\n", "Epoch: 56 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.524 | Train Acc: 74.57%\n", "\t Val. Loss: 0.512 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 57 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.522 | Train Acc: 74.78%\n", "\t Val. Loss: 0.511 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 58 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.520 | Train Acc: 75.00%\n", "\t Val. Loss: 0.509 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 59 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.518 | Train Acc: 75.43%\n", "\t Val. Loss: 0.507 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 60 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.516 | Train Acc: 76.08%\n", "\t Val. Loss: 0.505 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 61 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.515 | Train Acc: 76.51%\n", "\t Val. Loss: 0.503 | Val. Acc: 74.22%\n", "---------------------------------------------------------------------------\n", "Epoch: 62 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.513 | Train Acc: 76.72%\n", "\t Val. Loss: 0.502 | Val. Acc: 75.78%\n", "---------------------------------------------------------------------------\n", "Epoch: 63 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.511 | Train Acc: 77.16%\n", "\t Val. Loss: 0.500 | Val. Acc: 75.78%\n", "---------------------------------------------------------------------------\n", "Epoch: 64 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.510 | Train Acc: 77.37%\n", "\t Val. Loss: 0.498 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 65 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.508 | Train Acc: 77.37%\n", "\t Val. Loss: 0.496 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 66 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.506 | Train Acc: 77.37%\n", "\t Val. Loss: 0.495 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 67 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.505 | Train Acc: 77.59%\n", "\t Val. Loss: 0.493 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 68 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.503 | Train Acc: 77.59%\n", "\t Val. Loss: 0.491 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 69 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.501 | Train Acc: 78.02%\n", "\t Val. Loss: 0.490 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 70 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.500 | Train Acc: 78.02%\n", "\t Val. Loss: 0.488 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 71 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.498 | Train Acc: 78.45%\n", "\t Val. Loss: 0.486 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 72 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.497 | Train Acc: 78.45%\n", "\t Val. Loss: 0.485 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 73 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.495 | Train Acc: 78.66%\n", "\t Val. Loss: 0.483 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 74 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.494 | Train Acc: 79.09%\n", "\t Val. Loss: 0.482 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 75 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.492 | Train Acc: 79.31%\n", "\t Val. Loss: 0.480 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 76 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.491 | Train Acc: 79.31%\n", "\t Val. Loss: 0.478 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 77 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.489 | Train Acc: 79.53%\n", "\t Val. Loss: 0.477 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 78 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.488 | Train Acc: 79.74%\n", "\t Val. Loss: 0.475 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 79 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.486 | Train Acc: 80.17%\n", "\t Val. Loss: 0.474 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 80 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.485 | Train Acc: 80.17%\n", "\t Val. Loss: 0.472 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 81 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.483 | Train Acc: 80.60%\n", "\t Val. Loss: 0.471 | Val. Acc: 76.56%\n", "---------------------------------------------------------------------------\n", "Epoch: 82 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.482 | Train Acc: 81.03%\n", "\t Val. Loss: 0.469 | Val. Acc: 77.34%\n", "---------------------------------------------------------------------------\n", "Epoch: 83 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.480 | Train Acc: 81.47%\n", "\t Val. Loss: 0.468 | Val. Acc: 77.34%\n", "---------------------------------------------------------------------------\n", "Epoch: 84 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.479 | Train Acc: 81.68%\n", "\t Val. Loss: 0.466 | Val. Acc: 78.12%\n", "---------------------------------------------------------------------------\n", "Epoch: 85 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.478 | Train Acc: 81.68%\n", "\t Val. Loss: 0.465 | Val. Acc: 78.12%\n", "---------------------------------------------------------------------------\n", "Epoch: 86 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.476 | Train Acc: 81.68%\n", "\t Val. Loss: 0.464 | Val. Acc: 78.12%\n", "---------------------------------------------------------------------------\n", "Epoch: 87 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.475 | Train Acc: 81.68%\n", "\t Val. Loss: 0.462 | Val. Acc: 78.12%\n", "---------------------------------------------------------------------------\n", "Epoch: 88 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.474 | Train Acc: 82.11%\n", "\t Val. Loss: 0.461 | Val. Acc: 78.91%\n", "---------------------------------------------------------------------------\n", "Epoch: 89 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.472 | Train Acc: 82.11%\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\t Val. Loss: 0.459 | Val. Acc: 78.91%\n", "---------------------------------------------------------------------------\n", "Epoch: 90 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.471 | Train Acc: 82.11%\n", "\t Val. Loss: 0.458 | Val. Acc: 78.91%\n", "---------------------------------------------------------------------------\n", "Epoch: 91 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.470 | Train Acc: 82.33%\n", "\t Val. Loss: 0.457 | Val. Acc: 78.91%\n", "---------------------------------------------------------------------------\n", "Epoch: 92 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.468 | Train Acc: 82.76%\n", "\t Val. Loss: 0.455 | Val. Acc: 78.91%\n", "---------------------------------------------------------------------------\n", "Epoch: 93 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.467 | Train Acc: 83.41%\n", "\t Val. Loss: 0.454 | Val. Acc: 79.69%\n", "---------------------------------------------------------------------------\n", "Epoch: 94 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.466 | Train Acc: 83.62%\n", "\t Val. Loss: 0.453 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 95 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.465 | Train Acc: 83.84%\n", "\t Val. Loss: 0.451 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 96 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.463 | Train Acc: 84.27%\n", "\t Val. Loss: 0.450 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 97 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.462 | Train Acc: 84.48%\n", "\t Val. Loss: 0.449 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 98 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.461 | Train Acc: 84.48%\n", "\t Val. Loss: 0.447 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 99 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.460 | Train Acc: 84.48%\n", "\t Val. Loss: 0.446 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 100 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.458 | Train Acc: 84.48%\n", "\t Val. Loss: 0.445 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 101 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.457 | Train Acc: 84.48%\n", "\t Val. Loss: 0.444 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 102 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.456 | Train Acc: 84.48%\n", "\t Val. Loss: 0.442 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 103 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.455 | Train Acc: 84.91%\n", "\t Val. Loss: 0.441 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 104 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.454 | Train Acc: 84.91%\n", "\t Val. Loss: 0.440 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 105 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.452 | Train Acc: 84.91%\n", "\t Val. Loss: 0.439 | Val. Acc: 80.47%\n", "---------------------------------------------------------------------------\n", "Epoch: 106 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.451 | Train Acc: 84.91%\n", "\t Val. Loss: 0.438 | Val. Acc: 81.25%\n", "---------------------------------------------------------------------------\n", "Epoch: 107 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.450 | Train Acc: 84.91%\n", "\t Val. Loss: 0.436 | Val. Acc: 81.25%\n", "---------------------------------------------------------------------------\n", "Epoch: 108 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.449 | Train Acc: 84.91%\n", "\t Val. Loss: 0.435 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 109 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.448 | Train Acc: 85.13%\n", "\t Val. Loss: 0.434 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 110 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.447 | Train Acc: 85.13%\n", "\t Val. Loss: 0.433 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 111 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.446 | Train Acc: 85.13%\n", "\t Val. Loss: 0.432 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 112 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.445 | Train Acc: 85.34%\n", "\t Val. Loss: 0.430 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 113 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.444 | Train Acc: 85.34%\n", "\t Val. Loss: 0.429 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 114 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.443 | Train Acc: 85.99%\n", "\t Val. Loss: 0.428 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 115 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.441 | Train Acc: 85.99%\n", "\t Val. Loss: 0.427 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 116 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.440 | Train Acc: 85.99%\n", "\t Val. Loss: 0.426 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 117 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.439 | Train Acc: 85.99%\n", "\t Val. Loss: 0.425 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 118 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.438 | Train Acc: 86.21%\n", "\t Val. Loss: 0.424 | Val. Acc: 82.03%\n", "---------------------------------------------------------------------------\n", "Epoch: 119 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.437 | Train Acc: 86.42%\n", "\t Val. Loss: 0.423 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 120 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.436 | Train Acc: 86.42%\n", "\t Val. Loss: 0.422 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 121 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.435 | Train Acc: 86.42%\n", "\t Val. Loss: 0.421 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 122 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.434 | Train Acc: 86.64%\n", "\t Val. Loss: 0.419 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 123 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.433 | Train Acc: 86.85%\n", "\t Val. Loss: 0.418 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 124 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.432 | Train Acc: 86.85%\n", "\t Val. Loss: 0.417 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 125 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.431 | Train Acc: 86.85%\n", "\t Val. Loss: 0.416 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 126 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.430 | Train Acc: 86.85%\n", "\t Val. Loss: 0.415 | Val. Acc: 89.84%\n", "---------------------------------------------------------------------------\n", "Epoch: 127 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.429 | Train Acc: 86.85%\n", "\t Val. Loss: 0.414 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 128 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.428 | Train Acc: 86.85%\n", "\t Val. Loss: 0.413 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 129 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.427 | Train Acc: 86.85%\n", "\t Val. Loss: 0.412 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 130 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.426 | Train Acc: 86.85%\n", "\t Val. Loss: 0.411 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 131 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.425 | Train Acc: 86.85%\n", "\t Val. Loss: 0.410 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 132 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.425 | Train Acc: 86.85%\n", "\t Val. Loss: 0.409 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 133 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.424 | Train Acc: 86.85%\n", "\t Val. Loss: 0.408 | Val. Acc: 90.62%\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "---------------------------------------------------------------------------\n", "Epoch: 134 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.423 | Train Acc: 86.85%\n", "\t Val. Loss: 0.407 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 135 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.422 | Train Acc: 87.07%\n", "\t Val. Loss: 0.406 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 136 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.421 | Train Acc: 87.28%\n", "\t Val. Loss: 0.405 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 137 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.420 | Train Acc: 87.28%\n", "\t Val. Loss: 0.404 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 138 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.419 | Train Acc: 87.28%\n", "\t Val. Loss: 0.403 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 139 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.418 | Train Acc: 87.28%\n", "\t Val. Loss: 0.402 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 140 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.417 | Train Acc: 87.28%\n", "\t Val. Loss: 0.402 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 141 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.416 | Train Acc: 87.50%\n", "\t Val. Loss: 0.401 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 142 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.416 | Train Acc: 87.50%\n", "\t Val. Loss: 0.400 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 143 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.415 | Train Acc: 87.50%\n", "\t Val. Loss: 0.399 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 144 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.414 | Train Acc: 87.50%\n", "\t Val. Loss: 0.398 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 145 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.413 | Train Acc: 87.50%\n", "\t Val. Loss: 0.397 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 146 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.412 | Train Acc: 87.50%\n", "\t Val. Loss: 0.396 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 147 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.411 | Train Acc: 87.50%\n", "\t Val. Loss: 0.395 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 148 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.411 | Train Acc: 87.50%\n", "\t Val. Loss: 0.394 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 149 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.410 | Train Acc: 87.50%\n", "\t Val. Loss: 0.393 | Val. Acc: 90.62%\n", "---------------------------------------------------------------------------\n", "Epoch: 150 | Epoch Time: 0m 0s\n", "\tTrain Loss: 0.409 | Train Acc: 87.50%\n", "\t Val. Loss: 0.393 | Val. Acc: 90.62%\n" ] } ], "source": [ "N_EPOCHS = 150\n", "\n", "# track statistics\n", "track_stats = {'epoch': [],\n", " 'train_loss': [],\n", " 'train_acc': [],\n", " 'valid_loss':[],\n", " 'valid_acc':[]}\n", "\n", "\n", "best_valid_loss = float('inf')\n", "\n", "for epoch in range(N_EPOCHS):\n", "\n", " start_time = time.time()\n", " \n", " train_loss, train_acc = train(model, train_dl, optimizer, criterion)\n", " valid_loss, valid_acc = evaluate(model, test_dl, criterion)\n", " \n", " end_time = time.time()\n", " \n", " # record operations\n", " track_stats['epoch'].append(epoch + 1)\n", " track_stats['train_loss'].append(train_loss)\n", " track_stats['train_acc'].append(train_acc)\n", " track_stats['valid_loss'].append(valid_loss)\n", " track_stats['valid_acc'].append(valid_acc)\n", " \n", " \n", "\n", " epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n", " \n", " # if this was our best performance, record model parameters\n", " if valid_loss < best_valid_loss:\n", " best_valid_loss = valid_loss\n", " torch.save(model.state_dict(), 'best_log_regression.pt')\n", " \n", " # print out stats\n", " print('-'*75)\n", " print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n", " print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n", " print(f'\\t Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:.2f}%')\n" ] }, { "cell_type": "markdown", "metadata": { "scrolled": true }, "source": [ "# Visualization\n", "\n", "Our model performed very well! With a top validation accuracy of 90.62%\n", "\n", "Now, let us graph our results" ] }, { "cell_type": "code", "execution_count": 256, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain_losstrain_accvalid_lossvalid_acc
010.6618840.6265390.6528510.632812
120.6584710.6265390.6494580.632812
230.6551210.6265390.6461000.632812
340.6518150.6265390.6427800.632812
450.6485500.6265390.6394960.632812
560.6453230.6265390.6362490.632812
670.6421340.6286950.6330390.632812
780.6389830.6286950.6298650.632812
890.6358680.6286950.6267280.640625
9100.6327900.6286950.6236260.640625
10110.6297480.6286950.6205590.640625
11120.6267420.6265390.6175270.640625
12130.6237710.6265390.6145290.640625
13140.6208350.6243840.6115650.640625
14150.6179330.6243840.6086340.640625
15160.6150650.6243840.6057370.640625
16170.6122310.6286950.6028730.640625
17180.6094300.6351600.6000400.640625
18190.6066610.6394700.5972400.640625
19200.6039250.6394700.5944710.640625
20210.6012200.6459360.5917330.640625
21220.5985470.6480910.5890250.640625
22230.5959050.6502460.5863480.640625
23240.5932930.6502460.5837010.640625
24250.5907120.6524010.5810830.648438
25260.5881600.6545570.5784940.648438
26270.5856380.6588670.5759340.648438
27280.5831440.6631770.5734020.656250
28290.5806790.6674880.5708970.656250
29300.5782420.6696430.5684210.656250
..................
1201210.4352260.8642240.4205190.898438
1211220.4342180.8663790.4194550.898438
1221230.4332190.8685340.4183980.898438
1231240.4322260.8685340.4173490.898438
1241250.4312420.8685340.4163080.898438
1251260.4302650.8685340.4152750.898438
1261270.4292950.8685340.4142490.906250
1271280.4283330.8685340.4132310.906250
1281290.4273780.8685340.4122200.906250
1291300.4264300.8685340.4112160.906250
1301310.4254900.8685340.4102200.906250
1311320.4245560.8685340.4092310.906250
1321330.4236300.8685340.4082490.906250
1331340.4227100.8685340.4072730.906250
1341350.4217970.8706900.4063050.906250
1351360.4208910.8728450.4053440.906250
1361370.4199920.8728450.4043890.906250
1371380.4190990.8728450.4034410.906250
1381390.4182120.8728450.4024990.906250
1391400.4173320.8728450.4015640.906250
1401410.4164580.8750000.4006360.906250
1411420.4155910.8750000.3997140.906250
1421430.4147300.8750000.3987980.906250
1431440.4138740.8750000.3978880.906250
1441450.4130250.8750000.3969850.906250
1451460.4121820.8750000.3960880.906250
1461470.4113450.8750000.3951960.906250
1471480.4105140.8750000.3943110.906250
1481490.4096890.8750000.3934320.906250
1491500.4088690.8750000.3925580.906250
\n", "

150 rows × 5 columns

\n", "
" ], "text/plain": [ " epoch train_loss train_acc valid_loss valid_acc\n", "0 1 0.661884 0.626539 0.652851 0.632812\n", "1 2 0.658471 0.626539 0.649458 0.632812\n", "2 3 0.655121 0.626539 0.646100 0.632812\n", "3 4 0.651815 0.626539 0.642780 0.632812\n", "4 5 0.648550 0.626539 0.639496 0.632812\n", ".. ... ... ... ... ...\n", "145 146 0.412182 0.875000 0.396088 0.906250\n", "146 147 0.411345 0.875000 0.395196 0.906250\n", "147 148 0.410514 0.875000 0.394311 0.906250\n", "148 149 0.409689 0.875000 0.393432 0.906250\n", "149 150 0.408869 0.875000 0.392558 0.906250\n", "\n", "[150 rows x 5 columns]" ] }, "execution_count": 256, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# format data \n", "import pandas as pd\n", "\n", "stats = pd.DataFrame(track_stats)\n", "stats" ] }, { "cell_type": "code", "execution_count": 257, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'epoch': 1.0,\n", " 'train_loss': 0.6618835843842605,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6528506278991699,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 2.0,\n", " 'train_loss': 0.6584710745975889,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6494578719139099,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 3.0,\n", " 'train_loss': 0.6551205207561624,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6461004018783569,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 4.0,\n", " 'train_loss': 0.6518148882635708,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.642779529094696,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 5.0,\n", " 'train_loss': 0.6485495074041958,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6394957304000854,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 6.0,\n", " 'train_loss': 0.6453227339119747,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6362490057945251,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 7.0,\n", " 'train_loss': 0.6421338443098397,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.6330390572547913,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 8.0,\n", " 'train_loss': 0.6389825755152209,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.6298654079437256,\n", " 'valid_acc': 0.6328125},\n", " {'epoch': 9.0,\n", " 'train_loss': 0.6358680067391231,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.6267277598381042,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 10.0,\n", " 'train_loss': 0.6327900722109038,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.623625636100769,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 11.0,\n", " 'train_loss': 0.629748245765423,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.6205587387084961,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 12.0,\n", " 'train_loss': 0.6267420670081829,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6175265908241272,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 13.0,\n", " 'train_loss': 0.6237710097740436,\n", " 'train_acc': 0.6265394088669951,\n", " 'valid_loss': 0.6145287752151489,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 14.0,\n", " 'train_loss': 0.6208349425217201,\n", " 'train_acc': 0.624384236453202,\n", " 'valid_loss': 0.6115648746490479,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 15.0,\n", " 'train_loss': 0.6179331417741447,\n", " 'train_acc': 0.624384236453202,\n", " 'valid_loss': 0.6086344718933105,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 16.0,\n", " 'train_loss': 0.6150652129074623,\n", " 'train_acc': 0.624384236453202,\n", " 'valid_loss': 0.6057370901107788,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 17.0,\n", " 'train_loss': 0.6122307612978178,\n", " 'train_acc': 0.6286945812807881,\n", " 'valid_loss': 0.6028725504875183,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 18.0,\n", " 'train_loss': 0.609429721174569,\n", " 'train_acc': 0.6351600985221675,\n", " 'valid_loss': 0.6000401973724365,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 19.0,\n", " 'train_loss': 0.6066611059780779,\n", " 'train_acc': 0.6394704433497537,\n", " 'valid_loss': 0.5972397327423096,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 20.0,\n", " 'train_loss': 0.603924718396417,\n", " 'train_acc': 0.6394704433497537,\n", " 'valid_loss': 0.5944706201553345,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 21.0,\n", " 'train_loss': 0.6012202295763739,\n", " 'train_acc': 0.645935960591133,\n", " 'valid_loss': 0.5917326211929321,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 22.0,\n", " 'train_loss': 0.598547179123451,\n", " 'train_acc': 0.6480911330049262,\n", " 'valid_loss': 0.5890252590179443,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 23.0,\n", " 'train_loss': 0.5959050408725081,\n", " 'train_acc': 0.6502463054187192,\n", " 'valid_loss': 0.5863481163978577,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 24.0,\n", " 'train_loss': 0.5932934201996902,\n", " 'train_acc': 0.6502463054187192,\n", " 'valid_loss': 0.5837007761001587,\n", " 'valid_acc': 0.640625},\n", " {'epoch': 25.0,\n", " 'train_loss': 0.5907119882517847,\n", " 'train_acc': 0.6524014778325123,\n", " 'valid_loss': 0.581082820892334,\n", " 'valid_acc': 0.6484375},\n", " {'epoch': 26.0,\n", " 'train_loss': 0.5881601530930092,\n", " 'train_acc': 0.6545566502463054,\n", " 'valid_loss': 0.5784939527511597,\n", " 'valid_acc': 0.6484375},\n", " {'epoch': 27.0,\n", " 'train_loss': 0.5856377831820784,\n", " 'train_acc': 0.6588669950738917,\n", " 'valid_loss': 0.5759336352348328,\n", " 'valid_acc': 0.6484375},\n", " {'epoch': 28.0,\n", " 'train_loss': 0.5831442865832098,\n", " 'train_acc': 0.6631773399014779,\n", " 'valid_loss': 0.5734015703201294,\n", " 'valid_acc': 0.65625},\n", " {'epoch': 29.0,\n", " 'train_loss': 0.580679202901906,\n", " 'train_acc': 0.6674876847290641,\n", " 'valid_loss': 0.5708974003791809,\n", " 'valid_acc': 0.65625},\n", " {'epoch': 30.0,\n", " 'train_loss': 0.5782424005968817,\n", " 'train_acc': 0.6696428571428572,\n", " 'valid_loss': 0.5684206485748291,\n", " 'valid_acc': 0.65625},\n", " {'epoch': 31.0,\n", " 'train_loss': 0.5758331561910695,\n", " 'train_acc': 0.6696428571428572,\n", " 'valid_loss': 0.5659710168838501,\n", " 'valid_acc': 0.6640625},\n", " {'epoch': 32.0,\n", " 'train_loss': 0.5734513381431843,\n", " 'train_acc': 0.6739532019704434,\n", " 'valid_loss': 0.5635480284690857,\n", " 'valid_acc': 0.6875},\n", " {'epoch': 33.0,\n", " 'train_loss': 0.5710964202880859,\n", " 'train_acc': 0.6761083743842364,\n", " 'valid_loss': 0.561151385307312,\n", " 'valid_acc': 0.6875},\n", " {'epoch': 34.0,\n", " 'train_loss': 0.5687679422312769,\n", " 'train_acc': 0.6804187192118227,\n", " 'valid_loss': 0.5587806701660156,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 35.0,\n", " 'train_loss': 0.5664656408901872,\n", " 'train_acc': 0.6825738916256158,\n", " 'valid_loss': 0.5564355850219727,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 36.0,\n", " 'train_loss': 0.5641893847235318,\n", " 'train_acc': 0.6825738916256158,\n", " 'valid_loss': 0.5541156530380249,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 37.0,\n", " 'train_loss': 0.5619383844836005,\n", " 'train_acc': 0.686884236453202,\n", " 'valid_loss': 0.5518207550048828,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 38.0,\n", " 'train_loss': 0.5597124428584658,\n", " 'train_acc': 0.686884236453202,\n", " 'valid_loss': 0.5495502352714539,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 39.0,\n", " 'train_loss': 0.5575110994536301,\n", " 'train_acc': 0.6890394088669951,\n", " 'valid_loss': 0.5473039746284485,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 40.0,\n", " 'train_loss': 0.5553342227278084,\n", " 'train_acc': 0.6976600985221675,\n", " 'valid_loss': 0.5450814962387085,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 41.0,\n", " 'train_loss': 0.5531812865158607,\n", " 'train_acc': 0.708435960591133,\n", " 'valid_loss': 0.5428824424743652,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 42.0,\n", " 'train_loss': 0.551051929079253,\n", " 'train_acc': 0.7155172413793104,\n", " 'valid_loss': 0.5407066345214844,\n", " 'valid_acc': 0.6953125},\n", " {'epoch': 43.0,\n", " 'train_loss': 0.5489459531060581,\n", " 'train_acc': 0.7176724137931034,\n", " 'valid_loss': 0.5385535955429077,\n", " 'valid_acc': 0.703125},\n", " {'epoch': 44.0,\n", " 'train_loss': 0.5468627995458143,\n", " 'train_acc': 0.7176724137931034,\n", " 'valid_loss': 0.5364230871200562,\n", " 'valid_acc': 0.703125},\n", " {'epoch': 45.0,\n", " 'train_loss': 0.5448023697425579,\n", " 'train_acc': 0.7198275862068966,\n", " 'valid_loss': 0.5343146920204163,\n", " 'valid_acc': 0.703125},\n", " {'epoch': 46.0,\n", " 'train_loss': 0.5427641704164702,\n", " 'train_acc': 0.7219827586206896,\n", " 'valid_loss': 0.5322281718254089,\n", " 'valid_acc': 0.703125},\n", " {'epoch': 47.0,\n", " 'train_loss': 0.5407478398290174,\n", " 'train_acc': 0.7306034482758621,\n", " 'valid_loss': 0.5301631689071655,\n", " 'valid_acc': 0.7109375},\n", " {'epoch': 48.0,\n", " 'train_loss': 0.538753345094878,\n", " 'train_acc': 0.7306034482758621,\n", " 'valid_loss': 0.5281195044517517,\n", " 'valid_acc': 0.7109375},\n", " {'epoch': 49.0,\n", " 'train_loss': 0.5367800942782698,\n", " 'train_acc': 0.7306034482758621,\n", " 'valid_loss': 0.5260966420173645,\n", " 'valid_acc': 0.7109375},\n", " {'epoch': 50.0,\n", " 'train_loss': 0.5348278900672649,\n", " 'train_acc': 0.7349137931034483,\n", " 'valid_loss': 0.5240945219993591,\n", " 'valid_acc': 0.7109375},\n", " {'epoch': 51.0,\n", " 'train_loss': 0.5328963049526872,\n", " 'train_acc': 0.7370689655172413,\n", " 'valid_loss': 0.5221127271652222,\n", " 'valid_acc': 0.71875},\n", " {'epoch': 52.0,\n", " 'train_loss': 0.5309851745079304,\n", " 'train_acc': 0.7392241379310345,\n", " 'valid_loss': 0.520150899887085,\n", " 'valid_acc': 0.71875},\n", " {'epoch': 53.0,\n", " 'train_loss': 0.5290941369944605,\n", " 'train_acc': 0.7413793103448276,\n", " 'valid_loss': 0.5182088613510132,\n", " 'valid_acc': 0.71875},\n", " {'epoch': 54.0,\n", " 'train_loss': 0.5272229951003502,\n", " 'train_acc': 0.7413793103448276,\n", " 'valid_loss': 0.516286313533783,\n", " 'valid_acc': 0.7265625},\n", " {'epoch': 55.0,\n", " 'train_loss': 0.5253712555457806,\n", " 'train_acc': 0.7456896551724138,\n", " 'valid_loss': 0.5143830180168152,\n", " 'valid_acc': 0.734375},\n", " {'epoch': 56.0,\n", " 'train_loss': 0.5235388525601091,\n", " 'train_acc': 0.7456896551724138,\n", " 'valid_loss': 0.5124986171722412,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 57.0,\n", " 'train_loss': 0.5217253586341595,\n", " 'train_acc': 0.7478448275862069,\n", " 'valid_loss': 0.5106328129768372,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 58.0,\n", " 'train_loss': 0.5199305764560042,\n", " 'train_acc': 0.75,\n", " 'valid_loss': 0.5087854862213135,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 59.0,\n", " 'train_loss': 0.5181542429430731,\n", " 'train_acc': 0.7543103448275862,\n", " 'valid_loss': 0.5069563388824463,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 60.0,\n", " 'train_loss': 0.5163960950127964,\n", " 'train_acc': 0.7607758620689655,\n", " 'valid_loss': 0.5051449537277222,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 61.0,\n", " 'train_loss': 0.5146557380413187,\n", " 'train_acc': 0.7650862068965517,\n", " 'valid_loss': 0.5033513307571411,\n", " 'valid_acc': 0.7421875},\n", " {'epoch': 62.0,\n", " 'train_loss': 0.512933139143319,\n", " 'train_acc': 0.7672413793103449,\n", " 'valid_loss': 0.5015749931335449,\n", " 'valid_acc': 0.7578125},\n", " {'epoch': 63.0,\n", " 'train_loss': 0.5112278379242996,\n", " 'train_acc': 0.771551724137931,\n", " 'valid_loss': 0.49981582164764404,\n", " 'valid_acc': 0.7578125},\n", " {'epoch': 64.0,\n", " 'train_loss': 0.509539768613618,\n", " 'train_acc': 0.7737068965517241,\n", " 'valid_loss': 0.4980735778808594,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 65.0,\n", " 'train_loss': 0.5078684708167767,\n", " 'train_acc': 0.7737068965517241,\n", " 'valid_loss': 0.4963480234146118,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 66.0,\n", " 'train_loss': 0.5062139116484543,\n", " 'train_acc': 0.7737068965517241,\n", " 'valid_loss': 0.4946388006210327,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 67.0,\n", " 'train_loss': 0.5045756964847959,\n", " 'train_acc': 0.7758620689655172,\n", " 'valid_loss': 0.4929458796977997,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 68.0,\n", " 'train_loss': 0.5029537924404802,\n", " 'train_acc': 0.7758620689655172,\n", " 'valid_loss': 0.49126893281936646,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 69.0,\n", " 'train_loss': 0.501347673350367,\n", " 'train_acc': 0.7801724137931034,\n", " 'valid_loss': 0.48960769176483154,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 70.0,\n", " 'train_loss': 0.4997573720997778,\n", " 'train_acc': 0.7801724137931034,\n", " 'valid_loss': 0.487962007522583,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 71.0,\n", " 'train_loss': 0.49818255983549975,\n", " 'train_acc': 0.7844827586206896,\n", " 'valid_loss': 0.4863317012786865,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 72.0,\n", " 'train_loss': 0.4966230063602842,\n", " 'train_acc': 0.7844827586206896,\n", " 'valid_loss': 0.4847164750099182,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 73.0,\n", " 'train_loss': 0.4950785472475249,\n", " 'train_acc': 0.7866379310344828,\n", " 'valid_loss': 0.48311617970466614,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 74.0,\n", " 'train_loss': 0.4935490509559368,\n", " 'train_acc': 0.790948275862069,\n", " 'valid_loss': 0.4815305769443512,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 75.0,\n", " 'train_loss': 0.49203392554973735,\n", " 'train_acc': 0.7931034482758621,\n", " 'valid_loss': 0.47995948791503906,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 76.0,\n", " 'train_loss': 0.49053353276746026,\n", " 'train_acc': 0.7931034482758621,\n", " 'valid_loss': 0.4784027636051178,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 77.0,\n", " 'train_loss': 0.489047280673323,\n", " 'train_acc': 0.7952586206896551,\n", " 'valid_loss': 0.47686007618904114,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 78.0,\n", " 'train_loss': 0.4875750706113618,\n", " 'train_acc': 0.7974137931034483,\n", " 'valid_loss': 0.4753313660621643,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 79.0,\n", " 'train_loss': 0.4861166723843279,\n", " 'train_acc': 0.8017241379310345,\n", " 'valid_loss': 0.4738163650035858,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 80.0,\n", " 'train_loss': 0.4846720531068999,\n", " 'train_acc': 0.8017241379310345,\n", " 'valid_loss': 0.4723149538040161,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 81.0,\n", " 'train_loss': 0.4832408181552229,\n", " 'train_acc': 0.8060344827586207,\n", " 'valid_loss': 0.4708269238471985,\n", " 'valid_acc': 0.765625},\n", " {'epoch': 82.0,\n", " 'train_loss': 0.48182293464397563,\n", " 'train_acc': 0.8103448275862069,\n", " 'valid_loss': 0.4693520665168762,\n", " 'valid_acc': 0.7734375},\n", " {'epoch': 83.0,\n", " 'train_loss': 0.48041820526123047,\n", " 'train_acc': 0.8146551724137931,\n", " 'valid_loss': 0.4678902328014374,\n", " 'valid_acc': 0.7734375},\n", " {'epoch': 84.0,\n", " 'train_loss': 0.47902633403909617,\n", " 'train_acc': 0.8168103448275862,\n", " 'valid_loss': 0.46644127368927,\n", " 'valid_acc': 0.78125},\n", " {'epoch': 85.0,\n", " 'train_loss': 0.4776471894362877,\n", " 'train_acc': 0.8168103448275862,\n", " 'valid_loss': 0.4650050103664398,\n", " 'valid_acc': 0.78125},\n", " {'epoch': 86.0,\n", " 'train_loss': 0.4762807056821626,\n", " 'train_acc': 0.8168103448275862,\n", " 'valid_loss': 0.46358129382133484,\n", " 'valid_acc': 0.78125},\n", " {'epoch': 87.0,\n", " 'train_loss': 0.47492652103818694,\n", " 'train_acc': 0.8168103448275862,\n", " 'valid_loss': 0.4621698558330536,\n", " 'valid_acc': 0.78125},\n", " {'epoch': 88.0,\n", " 'train_loss': 0.47358470127500335,\n", " 'train_acc': 0.8211206896551724,\n", " 'valid_loss': 0.4607706367969513,\n", " 'valid_acc': 0.7890625},\n", " {'epoch': 89.0,\n", " 'train_loss': 0.4722549504247205,\n", " 'train_acc': 0.8211206896551724,\n", " 'valid_loss': 0.45938345789909363,\n", " 'valid_acc': 0.7890625},\n", " {'epoch': 90.0,\n", " 'train_loss': 0.4709371040607321,\n", " 'train_acc': 0.8211206896551724,\n", " 'valid_loss': 0.45800817012786865,\n", " 'valid_acc': 0.7890625},\n", " {'epoch': 91.0,\n", " 'train_loss': 0.46963099775643186,\n", " 'train_acc': 0.8232758620689655,\n", " 'valid_loss': 0.45664459466934204,\n", " 'valid_acc': 0.7890625},\n", " {'epoch': 92.0,\n", " 'train_loss': 0.468336532855856,\n", " 'train_acc': 0.8275862068965517,\n", " 'valid_loss': 0.45529258251190186,\n", " 'valid_acc': 0.7890625},\n", " {'epoch': 93.0,\n", " 'train_loss': 0.46705347916175577,\n", " 'train_acc': 0.834051724137931,\n", " 'valid_loss': 0.45395195484161377,\n", " 'valid_acc': 0.796875},\n", " {'epoch': 94.0,\n", " 'train_loss': 0.4657817709034887,\n", " 'train_acc': 0.8362068965517241,\n", " 'valid_loss': 0.452622652053833,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 95.0,\n", " 'train_loss': 0.4645211449984846,\n", " 'train_acc': 0.8383620689655172,\n", " 'valid_loss': 0.45130446553230286,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 96.0,\n", " 'train_loss': 0.4632716014467437,\n", " 'train_acc': 0.8426724137931034,\n", " 'valid_loss': 0.44999733567237854,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 97.0,\n", " 'train_loss': 0.46203294293633823,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.4487009644508362,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 98.0,\n", " 'train_loss': 0.4608049721553408,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.447415292263031,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 99.0,\n", " 'train_loss': 0.45958755756246633,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.4461402893066406,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 100.0,\n", " 'train_loss': 0.45838060050175106,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.44487571716308594,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 101.0,\n", " 'train_loss': 0.4571839036612675,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.443621426820755,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 102.0,\n", " 'train_loss': 0.4559974341556944,\n", " 'train_acc': 0.8448275862068966,\n", " 'valid_loss': 0.44237738847732544,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 103.0,\n", " 'train_loss': 0.45482102755842535,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.44114333391189575,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 104.0,\n", " 'train_loss': 0.45365445367221174,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.43991929292678833,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 105.0,\n", " 'train_loss': 0.45249781115301724,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.4387049674987793,\n", " 'valid_acc': 0.8046875},\n", " {'epoch': 106.0,\n", " 'train_loss': 0.45135073826230804,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.4375004470348358,\n", " 'valid_acc': 0.8125},\n", " {'epoch': 107.0,\n", " 'train_loss': 0.45021320211476296,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.4363054931163788,\n", " 'valid_acc': 0.8125},\n", " {'epoch': 108.0,\n", " 'train_loss': 0.4490851040544181,\n", " 'train_acc': 0.8491379310344828,\n", " 'valid_loss': 0.4351199269294739,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 109.0,\n", " 'train_loss': 0.4479663783106311,\n", " 'train_acc': 0.8512931034482759,\n", " 'valid_loss': 0.4339437484741211,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 110.0,\n", " 'train_loss': 0.4468568604567955,\n", " 'train_acc': 0.8512931034482759,\n", " 'valid_loss': 0.4327767789363861,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 111.0,\n", " 'train_loss': 0.4457563202956627,\n", " 'train_acc': 0.8512931034482759,\n", " 'valid_loss': 0.43161892890930176,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 112.0,\n", " 'train_loss': 0.4446647249419114,\n", " 'train_acc': 0.853448275862069,\n", " 'valid_loss': 0.4304701089859009,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 113.0,\n", " 'train_loss': 0.4435821072808627,\n", " 'train_acc': 0.853448275862069,\n", " 'valid_loss': 0.4293302297592163,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 114.0,\n", " 'train_loss': 0.442508105573983,\n", " 'train_acc': 0.8599137931034483,\n", " 'valid_loss': 0.42819908261299133,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 115.0,\n", " 'train_loss': 0.44144278559191474,\n", " 'train_acc': 0.8599137931034483,\n", " 'valid_loss': 0.42707669734954834,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 116.0,\n", " 'train_loss': 0.44038595002273034,\n", " 'train_acc': 0.8599137931034483,\n", " 'valid_loss': 0.4259628653526306,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 117.0,\n", " 'train_loss': 0.43933756598110857,\n", " 'train_acc': 0.8599137931034483,\n", " 'valid_loss': 0.424857497215271,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 118.0,\n", " 'train_loss': 0.43829746904044314,\n", " 'train_acc': 0.8620689655172413,\n", " 'valid_loss': 0.4237605333328247,\n", " 'valid_acc': 0.8203125},\n", " {'epoch': 119.0,\n", " 'train_loss': 0.4372657249713766,\n", " 'train_acc': 0.8642241379310345,\n", " 'valid_loss': 0.4226718544960022,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 120.0,\n", " 'train_loss': 0.43624200492069637,\n", " 'train_acc': 0.8642241379310345,\n", " 'valid_loss': 0.4215913414955139,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 121.0,\n", " 'train_loss': 0.43522624311776,\n", " 'train_acc': 0.8642241379310345,\n", " 'valid_loss': 0.4205189347267151,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 122.0,\n", " 'train_loss': 0.4342184724478886,\n", " 'train_acc': 0.8663793103448276,\n", " 'valid_loss': 0.41945451498031616,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 123.0,\n", " 'train_loss': 0.43321859425511855,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.41839802265167236,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 124.0,\n", " 'train_loss': 0.4322263454568797,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.41734933853149414,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 125.0,\n", " 'train_loss': 0.43124179182381467,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4163084030151367,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 126.0,\n", " 'train_loss': 0.43026470315867454,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.41527506709098816,\n", " 'valid_acc': 0.8984375},\n", " {'epoch': 127.0,\n", " 'train_loss': 0.42929527677338697,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.41424936056137085,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 128.0,\n", " 'train_loss': 0.42833305227345436,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4132310152053833,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 129.0,\n", " 'train_loss': 0.4273781940854829,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4122201204299927,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 130.0,\n", " 'train_loss': 0.4264304391269026,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.41121649742126465,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 131.0,\n", " 'train_loss': 0.4254899189389985,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4102201461791992,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 132.0,\n", " 'train_loss': 0.42455633755387934,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4092308580875397,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 133.0,\n", " 'train_loss': 0.42362982651283,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.4082486629486084,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 134.0,\n", " 'train_loss': 0.4227101556186018,\n", " 'train_acc': 0.8685344827586207,\n", " 'valid_loss': 0.40727338194847107,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 135.0,\n", " 'train_loss': 0.4217972919858735,\n", " 'train_acc': 0.8706896551724138,\n", " 'valid_loss': 0.4063050448894501,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 136.0,\n", " 'train_loss': 0.4208910711880388,\n", " 'train_acc': 0.8728448275862069,\n", " 'valid_loss': 0.40534353256225586,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 137.0,\n", " 'train_loss': 0.41999155899574014,\n", " 'train_acc': 0.8728448275862069,\n", " 'valid_loss': 0.40438878536224365,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 138.0,\n", " 'train_loss': 0.41909852521172886,\n", " 'train_acc': 0.8728448275862069,\n", " 'valid_loss': 0.4034406840801239,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 139.0,\n", " 'train_loss': 0.41821210137728987,\n", " 'train_acc': 0.8728448275862069,\n", " 'valid_loss': 0.4024991989135742,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 140.0,\n", " 'train_loss': 0.4173319586392107,\n", " 'train_acc': 0.8728448275862069,\n", " 'valid_loss': 0.40156421065330505,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 141.0,\n", " 'train_loss': 0.41645826142409753,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.4006357192993164,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 142.0,\n", " 'train_loss': 0.4155908781906654,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.39971357583999634,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 143.0,\n", " 'train_loss': 0.4147295458563443,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.39879775047302246,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 144.0,\n", " 'train_loss': 0.4138744617330617,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.39788818359375,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 145.0,\n", " 'train_loss': 0.41302542850889007,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.39698484539985657,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 146.0,\n", " 'train_loss': 0.4121824461838295,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.3960876166820526,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 147.0,\n", " 'train_loss': 0.4113452845606311,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.3951963782310486,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 148.0,\n", " 'train_loss': 0.4105140422952586,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.3943111300468445,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 149.0,\n", " 'train_loss': 0.40968858784642714,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.3934318423271179,\n", " 'valid_acc': 0.90625},\n", " {'epoch': 150.0,\n", " 'train_loss': 0.40886885544349405,\n", " 'train_acc': 0.875,\n", " 'valid_loss': 0.39255842566490173,\n", " 'valid_acc': 0.90625}]" ] }, "execution_count": 257, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = []\n", "for row in stats.iterrows():\n", " data.append(row[1].to_dict())\n", "data" ] }, { "cell_type": "code", "execution_count": 258, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n", " \n", " \n", " HiPlot\n", " \n", " \n", " \n", "
Loading HiPlot...
\n", " \n", "
\n", "\n", " \n", " \n", " \n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 258, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import hiplot as hip\n", "hip.Experiment.from_iterable(data).display(force_full_width = True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above graph gives us a very nice way to visualize our expected general patterns: \n", "as the number of epoch increases, train and validation loss decreases while train and validation accuracy increase" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Further, we can investigate the magnitude of each weight parameter to shed insight on the variables that had a higher level of influence on our prediction (assuming that higher magnitudes correlate with higher importance)\n", "\n", "**NOTE**: this is only possible as our model is just a one layer linear operation. If this was a \"deep\" model, interpretation by weight magnitude would not be possible" ] }, { "cell_type": "code", "execution_count": 259, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
radius_meantexture_meanperimeter_meanarea_meansmoothness_meancompactness_meanconcavity_meanpoints_meansymmetry_meandimension_mean...radius_worsttexture_worstperimeter_worstarea_worstsmoothness_worstcompactness_worstconcavity_worstpoints_worstsymmetry_worstdimension_worst
01.1617650.6772372.2550953.453925-0.3815822.0461125.1075393.2686450.2493270.61387...2.5254640.678428-0.5702255.1252370.9066723.3808831.8960724.8355142.8841022.432787
\n", "

1 rows × 30 columns

\n", "
" ], "text/plain": [ " radius_mean texture_mean perimeter_mean area_mean smoothness_mean \\\n", "0 1.161765 0.677237 2.255095 3.453925 -0.381582 \n", "\n", " compactness_mean concavity_mean points_mean symmetry_mean \\\n", "0 2.046112 5.107539 3.268645 0.249327 \n", "\n", " dimension_mean ... radius_worst texture_worst perimeter_worst \\\n", "0 0.61387 ... 2.525464 0.678428 -0.570225 \n", "\n", " area_worst smoothness_worst compactness_worst concavity_worst \\\n", "0 5.125237 0.906672 3.380883 1.896072 \n", "\n", " points_worst symmetry_worst dimension_worst \n", "0 4.835514 2.884102 2.432787 \n", "\n", "[1 rows x 30 columns]" ] }, "execution_count": 259, "metadata": {}, "output_type": "execute_result" } ], "source": [ "params = list(model.parameters())[0].detach().cpu().view(-1).numpy()\n", "param_df = pd.DataFrame(params).T\n", "param_df.columns = df.columns\n", "param_df" ] }, { "cell_type": "code", "execution_count": 260, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 260, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAHWCAYAAACxEqjTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzde1yUZf7/8RfDWQ4uiqgIhAoKeICE1KhQk3LTtq9rbbommquWi+G6ISq2KmqaStmWImV5aK00SywtPKHiCd0g8gSo/EISRRoBFWEcTjO/P/g6XxAGMeUwzOf5ePB46Hjd133NUB8urvu+35eJVqvVIoQQokVTNPcAhBBC3JsUayGEMABSrIUQwgBIsRZCCAMgxVoIIQyAFGshhDAAUqxbqJKSEvbs2dPcwxBCtBBGWaw1Gk1zD0GnsrKyztdLSkrYu3dvE49GCNFSmbTGh2JWrFhBQUEB5eXlDB8+nODgYEJCQnj++ec5deoU48ePx8LCgs8++wy1Wo29vT2hoaE4ODiQkJDA/v37qaiooGPHjoSFhWFpaVnrHBqNhunTp7Nq1SpUKhV/+9vfWLBgAT4+PsyfP5/Q0FBsbW1Zs2YNSqUSS0tLXnvtNR555BG2bt3K9evXuXbtGnZ2dowaNYo1a9ZQUVGBVqslPDycr776iuTkZJydnenbty8hISHN8EkKIVoMbSt069YtrVar1ZaWlmrffPNNbVFRkfYvf/mL9tixY1qtVqstLy/XvvXWW9qbN29qtVqt9tixY9qYmBitVqvVFhUV6frZvHmzNj4+Xu953n77be2lS5e0KSkp2jlz5mi3bdumLSsr04aGhmq1Wq123bp12q1bt2q1Wq32zJkz2pkzZ2q1Wq32q6++0s6aNUtbWlqqa3f48GHd2EpLS7W//fab9s0333xon4kQwrCZNfcPi8YQHx9PcnIyAPn5+Vy9ehWFQsHAgQMByM3NJScnh8WLFwNVs2QHBwcAcnJy2LJlCyUlJajVanx9ffWex9vbm4yMDJRKJSNHjmT//v34+PjQvXt3AM6dO0d4eDgAvXv3pri4GJVKBUBAQAAWFhYA9OjRg7i4OAoKChgwYACdO3e+53tMSEggISEBgMUzX7vvz0iIlsjcsVuD25bnZzX7GJpSqyvWaWlpnDlzhrfffhtLS0uioqIoLy/H3NwcheL/luhdXFxYsmRJreNjYmKIiIjA3d2dxMRE0tLS9J7Ly8uLffv2cf36dV5++WV27NhBWloaPj4+AGjrWWGqvrTy5JNP4uHhQWpqKkuWLGHq1Kk4OTnV+z6Dg4MJDg4GGu8/WiFEy9HqLjCqVCpsbGywtLTkypUrZGZm1mrj7OxMUVERFy5cAKCiooKcnBwA1Go1Dg4OVFRUcOTIkXrP5enpyYULFzAxMcHCwgJ3d3cSEhLw8vICqmbed/pIS0vDzs6ONm3a1Ornt99+o2PHjgwfPpyAgAB+/fVXrK2tuX379gN9FkK0aprKxvlqoVrdzNrPz499+/Yxc+ZMnJ2d8fT0rNXGzMyM8PBwNmzYgEqlorKykuHDh+Pq6sro0aOZO3cuHTp0wM3Nrd6CaW5uTvv27XXn8Pb25tixY7i5uQHw8ssvs2bNGmbOnImlpSXTpk2rs5+kpCSOHDmCqakpf/jDH3jppZewtbWlZ8+ehIeH4+fnJxcYhTByrfJuEGMjyyCitbivNevfzjfOGDr2bJR+H1SrWwYRQojWqNUtgzSGuLg4jh8/XuO1xx9/nFGjRjXTiIQQtKCH25qCLIO0ArIMIlqL+1kGKcvVf6fWg7Bw7tUo/T4oWQYRQggDIMsgQgjDZGTLIDKzFkIIAyAzayGEYdLKzLrV++qrrzh9+jQAP/zwA6Wlpc08IiHEfTOyJxiNsliPHj2avn37AlWhT1KshRAtXZMugxw6dIidO3diYmKCm5sbY8aMITY2lqKiIl2mtKOjIzExMVhbW5OVlcWNGzcYN26cLjHvu+++4/DhwygUCvz8/HjllVfqzKCurKwkIiKCVatWoVAoKC0tZcaMGaxatYqPP/4Yf39/CgsLKSwsZOHChdjb2/PUU09x6dIlXn31VaAq2e7KlStMmDCh1ntRKpUsXboULy8vMjMzeeSRRxg8eDBff/01N2/eZPr06Xh4eKBWq1m/fj05OTlUVlbyl7/8hcceewylUsnq1at1Pyj+9re/0bNnT9LS0vj666+xs7MjJyeHbt26ERYWhomJSZN9n4QwCEa2DNJkxTonJ4e4uDgWL16Mvb09xcXFrF69mqCgIAYPHsyBAwdYv349s2bNAuDGjRssWrSI3Nxcli9fzsCBA/n5559JTk5m6dKlWFpaUlxcDMCAAQN0CXRbtmzhwIEDPPfcczzyyCOkp6fTu3dvfvrpJ3x9fTEz+7+3PHz4cH744QcWLFiAvb09arWa7du3M27cOMzMzEhMTOS11/THj+bl5fHmm2/i4uJCZGQkR48eZdGiRaSkpBAXF8esWbOIi4ujd+/ehIaGUlJSwty5c+nTpw9t27blX//6FxYWFly9epUPPviAZcuWAXDx4kVWrlyJg4MD8+bN4/z587pwqDskIlUI49Jkxfrs2bMMHDgQe3t7AGxtbcnMzGTmzJkABAUF8cUXX+jaP/bYYygUClxcXLh58yYAZ86cYfDgwbp4UVtbW0B/BnVgYCBJSUn07t2bY8eOMWzYsHrHaGVlRa9evUhNTaVLly5UVlbqQpnq4uTkpPt3V1dX+vTpo/ut4dq1awCcPn2an376iZ07dwJQVlZGfn4+7dq1Y926dWRnZ6NQKLh69aquXw8PD9q3bw+Au7s7SqWyVrGWiFRh9Izs1r0mK9Zarfa+fpU3NzevcWx9fejLoA4ICODLL7+kuLiYrKwsevfufc/zDh06lO3bt+Ps7MzgwYMbPEYTExPd301MTHT7PGr/d5suZ2fnGsdu3bqVtm3bEh0djVar5ZVXXqmzX4VC0aL2jBSipdAa2TJIk11g7NOnD8ePH+fWrVsAFBcX06NHD5KSkgA4evRordnj3Xx9fTl48KBunffOMoi+DGorKys8PDzYsGED/v7+NTYfqN5GrVbr/u7p6UlBQQHHjh3jiSeeeLA3/b9j3rVrl+4HzsWLF4Gq3G0HBwcUCgWHDx+WgiyEqFeTzaxdXV3585//TFRUFAqFAnd3dyZOnEhsbCw7duzQXWCsj5+fH9nZ2cyZMwczMzMeffRRxo4dW28GdWBgICtXriQqKqrOPoODg1m6dCkODg4sWLAAqAppys7O1i2zPIiXXnqJjRs36pZ7OnTowJw5cxg2bBjvvfceJ06coFevXnVuyiuEqIeRTXAkyKkOy5YtY8SIEfTp06e5h9IgsmYtWov7CXIqzUxqlDFYegY2Sr8Pyijvs9anpKSEf/zjH1hYWBhMoRbCaGk1jfPVQsnM+h5u3brFokWLar0+f/587OzsmmFEtZX+cqK5hyDEQ2HZfWCD25ZeONo4Y+jxZKP0+6AkG+Qe7OzsiI6Obu5hCCHu1oIfDW8MUqyFEIapBS9ZNAZZsxZCCAMgM2shhGEyslv3Wv3M+qOPPuLy5cv1tvnxxx/v2aY5xMXFNfcQhBAtRKsv1lOnTsXFxaXeNsnJyc1WrCsr9V8k2b59exOORAgDY2S37hncMsidaFIPDw+ys7Pp3Lkzb7zxBhcuXGDTpk1UVlbSvXt3pkyZgrm5OVFRUYSEhNC9e3dCQkIYPnw4qampWFhYEBERwW+//UZKSgrp6els27aN8PBwUlNT2bdvH6ampri4uDBjxow6xxIeHs6iRYto06YNkyZNYsKECQwaNIhVq1YxaNAgvLy8+PTTT/nll18wNTVl/Pjx9O7dm8TERFJTUykrK6O0tJTp06fz73//G5VKhUajYfLkybp/j4iIwNXVlenTpzfxJy1EC2dkyyAGV6wBcnNzmTp1Kl5eXqxZs4bvv/+ehIQE5s2bh7OzM6tXr2bv3r2MGDGixnGlpaV4enry17/+lc8//5z9+/fz4osvEhAQgL+/f43M7NWrV2Nubk5JSYnecfTs2ZPz58/j6OhIx44dycjIYNCgQWRmZjJlyhT27NkDwHvvvceVK1d4++23+eCDDwC4cOEC7777Lra2tuzcuRNfX19GjRqFRqOhtLQUb29vdu/erfe2weoRqQunjHzgz1QI0bIZ5DJI+/btdaFPQUFBnD17FicnJ12y3aBBg8jIyKh1nJmZGf7+/gB069ZNF2N6Nzc3Nz788EMOHz6Mqamp3nF4e3uTnp5ORkYGzzzzDDk5ORQWFmJra4uVlRXnzp0jKCgIgC5dutChQwddFGrfvn112SPdu3fn4MGDbN26lUuXLmFtbX3PzyA4OJhly5bpMrCFMDZabWWjfLVUBlmsf++uKaamprpjFQqF3vXiyMhIhg0bRlZWFrNnz9bbztvbm3PnzpGRkUGvXr2ws7PjxIkTuh8k9T0cWj24ycfHh4ULF9KuXTtWrVrFoUOHftf7E0K0XgZZrPPz87lw4QJQFa3ap08flEoleXl5ABw+fBgfH58G92dtba1L6tNoNOTn59O7d2/GjRuHSqWqEaFanaOjI7du3SIvL4+OHTvi5eXFzp078fb2BqqK8J3I1tzcXPLz82vlWgNcu3aNtm3bEhwczNNPP62LUTUzM6OioqLB70MIoyIXGFu+Ll26kJiYyNq1a+nUqRMTJ07E09OTlStX6i4wPvPMMw3uLzAwkI8//phdu3YxY8YMYmNjUalUAIwYMQIbGxu9x3p4eOiyqL29vdm8ebNuZv3ss8/yySefEB4ejqmpKaGhoTU2FrgjLS2NnTt3YmpqipWVFW+88QZQtRFCREQEXbt2lQuMQtzNyC4wGlyQk1KpZPny5bz33nvNPZQWQ4KcRGtxP0FO6tQdjTIGq34vNEq/D8ogZ9ZCCNGcSxYajYY5c+bQrl075syZU+PfEhMT2bRpE+3atQPgj3/8I0OHDn3gcxpcsXZycmryWfXBgweJj4+v8VrPnj2ZPHlyk45DH82Pu5p7CEI8HPcxs25O8fHxdOnSpcauVNUFBgYyadKkh3pOgyvWzWHIkCEMGTKkuYchhKiumSJSCwoKSE1NZdSoUXz//fdNdl4p1kIIw9RMyyAbN25k3LhxemfVAP/973/JyMigc+fOTJgwAUdHxwc+rxRrIYSopvrTwVD1AFpwcDAAP/30E23btqVbt26kpaXVeby/vz9PPPEE5ubm7N27l5iYGN1m3A9CivXvoNFoUCgM8hZ1IVqPRrp1r3pxvtv58+dJSUnh559/pqysjNu3b/Phhx/WuLW2+nZ/wcHBfPHFFw9lXFKs67BixQoKCgooLy9n+PDhBAcHExISwvPPP8+pU6cYP348FhYWfPbZZ6jVauzt7QkNDcXBwYGEhAT2799PRUUFHTt2JCwsrMbTitUdP36cb775BoVCQZs2bVi4cCEajYYvvviC9PR0ysvLGTZs2H3dMy6EaDxjx45l7NixwP89H3H3MxDXr1/HwcEBgJSUlHumfjaUFOs6hIaGYmtrS1lZGZGRkQwYMIDS0lJcXV0ZPXo0FRUVREVFMWvWLOzt7UlKSmLz5s2EhoYyYMAA3U/lLVu2cODAAZ577rk6z/PNN9/w1ltv0a5dO11g1IEDB2jTpg3vvPMO5eXlzJs3D19fX5ycnJrs/QthEFrQ04ZfffUV3bt3JyAggF27dpGSkoKpqSm2traEhoY+lHNIsa5DfHw8ycnJQNWj7VevXkWhUOhS+XJzc8nJyWHx4sVA1bLInZ+kOTk5bNmyhZKSEtRqNb6+vnrP07NnT2JiYnj88ccZMGAAAKdOneLSpUucOFH1oItKpeLq1atSrIVoYXr16kWvXr0AGD16tO716rPvh0mK9V3S0tI4c+YMb7/9NpaWlkRFRVFeXo65uXmNdWoXFxeWLFlS6/iYmBgiIiJwd3cnMTFR70UIgNdee43MzExSU1OZNWsWK1asQKvVMnHiRPz8/OodZ/WLIAt8615mEaJVM7LHzeUq2V1UKhU2NjZYWlpy5coVMjMza7VxdnamqKhIFyZVUVFBTk4OAGq1GgcHByoqKnQhTvrk5eXh6enJ6NGjsbOzo6CgAD8/P/bu3asLcMrNza0zSEoiUoXR02ga56uFkpn1Xfz8/Ni3bx8zZ87E2dkZT0/PWm3MzMwIDw9nw4YNqFQqKisrGT58uG5Ne+7cuXTo0AE3N7d678X8/PPPdfnWvXv35pFHHsHNzQ2lUsns2bMBsLe3JyIionHerBDCYBhckJOo7fbmB7+HU4iWwPqvCxvc9vbhjY0zhqBXG6XfByXLIEIIYQBkGaQJxMXFcfz48RqvPf7444waNaqZRiREK9CC15cbgxTrJjBq1CgpzEI8bC3oPuumIMW6FdBezW3uIQghGpkUayGEYTKyZRC5wCiEEAZAZtZCCMMka9ZCCGEAZBmkddq7dy+HDh16KH3FxcU9lH6EEKKhjKJYV1ZW8uyzzzJo0KCH0t/27dvv+xiNkc0ChGh0Wk3jfLVQBrMMolQqWbp0KR4eHmRnZ9O5c2feeOMNrly5UucmAFFRUfTo0YPz588TEBDA7du3sbKy4oUXXiAqKgp3d3cuXrxIUVER06ZN49tvv+XSpUsEBgYyZswYAA4fPsyuXbuoqKjA09OTyZMns3nzZsrKyoiIiMDV1ZXp06fX2U6hUNTasMDLy6vW+/riiy902bd9+/Zl/PjxFBUVsXbtWgoKCgCYMGFCnccKIYyHwRRrqEqgmzp1Kl5eXqxZs4Y9e/bw448/1rkJAFQl6C1cWJU1sHXr1hp9mZmZsXDhQuLj44mOjmbZsmXY2toSFhbGiBEjuHnzJklJSSxevBgzMzM+/fRTjhw5wiuvvMLu3buJjo4G4PLly3W2GzRoUI0NC+pSXFzMjz/+yL///W9MTEx0GxBs2LCB559/Hi8vL/Lz81myZAnvv/9+jWOrR6TOl6hrYYyM7LdVgyrW7du3180wg4KC2L59u95NAAACAwP19hUQEACAm5sbLi4uuuM6duxIQUEB586d4+LFi0RGRgJQVlaGvb19rX7Onj2rt131DQvqYm1tjYWFBR999BH9+vXD398fgDNnznD58mVdO5VKxe3bt7G2tta9Vn2fONXKKXrPIYRoHQyqWJuYmNT4u5WVld5NAAC9ex8CmJub6/q88+c7f6+srESr1TJo0KB77vhQX7u7Nyy4m6mpKUuXLuXMmTMkJSWxe/duFixYgFarZcmSJVhYWNR7biGMmpHNrA3qAmN+fr4u8P/o0aN4enrq3QTgQfXp04cTJ05w8+ZNoGrJ4tq1a0DVEsqdzQHqa3cvarUalUpFv379ePXVV8nOzgagb9++7N69W9fuzutCiGrkAmPL1aVLFxITE1m7di2dOnXiueeew8/Pr85NAB6Ui4sLY8aM4e2330ar1WJqasqkSZPo0KEDQ4cOJSIigq5duzJ9+nS97e7l9u3brFixgvLycrRaLRMmTABg4sSJrFu3jpkzZ1JZWYm3tzevvfbaA78nIYThMpjNB5RKJcuXL+e9995r7qG0OLJmLVqLNm9+0uC2t3e82yhjsH5hZqP0+6AMahlECCGMlcEsgzg5ORn0rDo6OhqlUlnjtVdeeeWeu5g3SHnFg/chhKFpwevLjcFgirWhk01vhXjI5G4QIYQQLY3MrIUQhsnIlkGMbmYtgUpCCEPU6mbWK1asoKCggPLycoYPH05wcHCtQCULC4s6w58SEhLYv38/FRUVdOzYkbCwML1PQcbExGBhYUFubi7Xrl0jNDSUxMREMjMz8fDwYNq0aQCcOnWKrVu36voMDQ3FysqKb775hp9++omysjJ69OjBa6+9homJCVFRUXh4eJCWloZKpWLq1Kl4e3s35UcohGEwsolXqyvWoaGh2NraUlZWRmRkJAMGDKgRqFRRUUFUVFSd4U8DBgzQ5W1s2bKFAwcO8Nxzz+k9V0lJCfPnzyclJYXly5ezePFiXFxciIyMJDs7m3bt2hEXF8e8efOwsrLi22+/5fvvv+ell17ij3/8Iy+99BIAq1at4qefftLllWg0Gt555x1SU1P55ptvmDdvXuN/cEIYGinWhi0+Pp7k5GSg6vH0q1ev1ghUys3N1Rv+lJOTw5YtWygpKUGtVuPr61vvufz9/TExMcHNzY22bdvi5uYGgKurK0qlkoKCAi5fvqwrthUVFfTo0QOoCoDasWMHpaWlFBcX4+rqqivW/fv3B6Bbt261bvcTQhinVlWs09LSOHPmDG+//TaWlpZERUVRXl5eK1BJX/hTTEwMERERuLu7k5iYSFpaWr3nqy8MSqPRoFAo6NOnDzNmzKhxXFlZGevWreOdd97B0dGRrVu3UlZWVqtfhUKhd429RkSqQ51NhGjdDOPh64emVV1gVKlU2NjYYGlpyZUrV8jMzKzVxtnZWW/4k1qtxsHBgYqKCo4cOfLA47mz+UFeXh4ApaWl5ObmUl5eDoC9vT1qtZr//ve/9913cHAwy5YtY9myZQ88TiFEy9eqZtZ+fn7s27ePmTNn4uzsjKenZ602ZmZmhIeH1xn+NHr0aObOnUuHDh1wc3Pj9u3bDzQee3t7pk2bxgcffKAr0GPGjMHZ2ZmhQ4cSHh6Ok5MT3bt3f6DzCGGUjGzN2mCCnIR+quUTm3sIQjwUbWZvaHDb25sXNMoYrP+6sFH6fVCtamYthDAiRjazlmJ9D3FxcRw/frzGa48//jijRo1qphEJIQCje4JRivU9jBo1SgqzEKLZSbFuBbQVlc09BCGanpEtg7SqW/eEEKK1kpm1EMIwGdmNbFKshRCGSZZBDEdJSQl79uxp7mE0msTERAoLC5t7GEKIFsDgi/XevXubexgPRKvV6s3/SExM5Pr16008IiEMhEbTOF8tVIOXQQ4dOsTOnTt1KXNjxowhNjaWoqIiXSa0o6Njg3OeQ0JCeOaZZ0hLS8PGxoYZM2Zgb2+vN1P6xo0bfPLJJ7oUusmTJ7Nr1y7y8vKIiIigb9++9OvXj6+//ho7OztycnLo1q0bYWFhmJiYkJWVVWeGdXx8PPv27cPU1BQXFxdmzJhBeno6GzZUPUllYmLCwoULsba2rvWZfPrpp/j5+REQEEB0dDQ2NjaEhoZy4MABlEolY8aM4fvvv+fgwYMAPP3004wYMQKlUsk777xDr169uHDhAhEREWzdupWsrCwAhgwZgqOjI7/88gsffvghFhYWLFmyBAsLiwf7bgshDFaDinVOTg5xcXEsXrwYe3t7iouLWb16NUFBQQwePJgDBw6wfv16Zs2aBdw759nd3Z3S0lK6du3K+PHj+eabb/j666+ZNGmS3kzpDRs24OPjQ0REBBqNBrVazdixY8nJySE6OhqoSt27ePEiK1euxMHBgXnz5nH+/Hk8PDx047s7w/q7775j9erVmJubU1JSAsCOHTuYNGkSXl5eqNXqGol61Xl7e5ORkUFAQACFhYXcuHEDgHPnzhEYGEhWVhYHDx7UJfzNnTsXHx8fbGxsyM3N5e9//zuTJ08mKyuLwsJC3e7tJSUl2NjYsHv3bkJCQiQ7RIi6GNlDMQ1aBjl79iwDBw7E3t4eAFtbWzIzM3nyyScBCAoK4vz587r2deU8KxQKXc4zVM1YAwMDAXjqqac4d+4cUPWDYf78+YSHh3PkyBFdIt7Zs2d59tlnqwatUNCmTZs6x+rh4UH79u1RKBS4u7ujVCprZFhHRESwbds23Vqwm5sbH374IYcPH8bU1BQALy8v/vOf/xAfH09JSYnu9bt5e3tz7tw5Ll++jIuLC23btuX69etcuHCBnj17cu7cOfr374+VlRVWVlb079+fjIwMABwdHXXZ1k5OTiiVStavX8/JkyfrnMXfLSEhgTlz5jBnzpx7thWiNdJqtI3y1VI1aGat1WoxMTFpcKf3ynmuy53+7zdTWt+5oWYetL4M68jISNLT00lJSWHbtm2sXLmSkSNH0q9fP1JTU3nrrbeYN28eXbp0qXVsu3btKC4u5uTJk3h7e1NcXMzx48exsrLC2tqa+jKyrKysdH+2tbUlOjqakydPsnv3bpKSkggNDa33fQYHB+t+AylZMr7+D0UIYfAaNLPu06cPx48f59atWwAUFxfTo0cPkpKSADh69CheXl73dWKtVsuJEydqHa8vU7pPnz66i4kajQaVSoW1tXWDYkz1ZVhrNBry8/Pp3bs348aNQ6VSoVarycvLw83NjZEjR9KtWzeuXLmit+8ePXrwww8/4OPjg7e3Nzt37tS9F29vb5KTkyktLUWtVpOcnFznfopFRUVoNBoGDhzImDFjuHjxIlBV0B80plWIVksuMNbm6urKn//8Z6KionTLCxMnTiQ2NpYdO3boLtjdD0tLS3Jycpg9ezZt2rThn//8J4DeTOlXX32VtWvXcuDAARQKBVOmTKFHjx707NmT8PBw/Pz86NevX91vUk+GdefOnVm1ahUqlQqAESNGYGNjw1dffUVaWhoKhYIuXbrw6KOP6n0f3t7enD59mk6dOuHo6EhxcbGuIHfr1o3Bgwczd+5coOoCY9euXWtt1VVYWEhsbKzut4CxY8cCMHjwYD755BO5wCiEaL4865CQEDZt2tQcp251ZBlEtBY2b/2nwW1VsWGNMoY2f1/VKP0+KIO+z1oIIYxFsz1ubkiz6kuXLrFqVc2ftubm5ixdurSZRiSEoAXfudEYJBukAdzc3HT3crdIanVzj0CIpteCLwY2BlkGEUIIAyAzayGEYZKZtRBCiJZGZtZCCMNkZJsPtOqZ9UcffcTly5frbfPjjz/es40QogUysicYW3Wxnjp1Ki4uLvW2SU5OlmIthGjxDGoZRKlUsnTpUjw8PMjOzqZz58688cYbXLhwgU2bNlFZWUn37t2ZMmUK5ubmREVF6SJGQ0JCGD58OKmpqVhYWBAREcFvv/1GSkoK6enpbNu2jfDwcFJTU2vlW9dl69atKJVKbty4wdWrVxk/fjyZmZn8/PPPtGvXjtmzZ2NmZqY3R1tfbndMTAzW1tZkZWVx48YNxo0bx8CBA5v4kxbCAMh91i1bbm4uU6dOxRdUghUAACAASURBVMvLizVr1vD999+TkJDAvHnzcHZ2ZvXq1ezdu5cRI0bUOK60tBRPT0/++te/8vnnn7N//35efPFFAgIC8Pf31xXEuvKt9fntt99YsGABly9f5l//+hfh4eGMGzeO6OhoUlNT6devn94cbX253QA3btxg0aJF5Obmsnz58jqLdUJCAgkJCQDMqztuWwjRihhcsW7fvr0u1S4oKIht27bh5OSEs7MzAIMGDWLPnj21irWZmRn+/v5AVcDS6dOn6+z/Tr71Y489Rv/+/esdy6OPPoqZmRlubm5oNBr8/Px0fVy7dq1GjjZUpQU6ODgAVbndW7ZsoaSkBLVaja+vr67fxx57DIVCgYuLCzdv3qzz3DUiUue9XO84hWiVjGzzAYMr1veTq12dqamp7liFQkFlZWWd7erKt9a3+YCZmZmuv+r9m5iY6PrXl6NdX2539UzuZsrZEqLlM7JlEIO7wJifn6/LpT569Ch9+vRBqVSSl5cHwOHDh/Hx8Wlwf9UzsfXlW/9e+nK0QX9utxBC1MXgZtZdunQhMTGRtWvX0qlTJyZOnIinpycrV67UXWB85plnGtxfYGAgH3/8Mbt27WLGjBnExsbWyrf+vfTlaLu6uurN7RZCNIy2Bd9m1xiaLc/691AqlSxfvly3sayoImvWorWwWby1wW1L3pnQOGOI/KxR+n1QBjezFkIIwOjWrA2qWDs5OTX5rPrgwYPEx8fXeK1nz55Mnjy5ScdRn8rc6809BCFEIzOoYt0chgwZwpAhQ5p7GEKIu8mte0IIYQCMbBnE4G7dE0IIY2R0xdpQUvaUSiVHjx5t7mEI0XJJ6l7rVl/Knr6nGhuTvnNeu3ZNirUQQqfR16zVajXvv/8+hYWFaDQaXnzxRY4dO0ZERAQAp0+fZu/evcycOZOQkBCGDRvGmTNnsLW11YUu5efn8+qrrxIQEEBiYiI//vgjGo2GnJwc/vSnP1FRUcHhw4cxNzcnMjISW1tb8vLyWLduHUVFRVhaWvL6669TXFxcK2Xvo48+okePHpw/f57evXuTmJjIBx98gJmZGSqVioiICN3fq7t58yZLly5l+fLlZGdnM2vWLNasWYOjoyNhYWG8++67FBUVERsbS1FRkS5xz9HRkZiYGGxtbcnOzqZr164EBASwYcMGoOpR9YULF/Lll19y+fJlIiIiGDRoEM8//3xjf6uEMCxGtmbd6MX65MmTODg4EBkZCYBKpWLr1q26Anbw4EEGDx4MVCXj9erVS5dct2XLFv71r39x+fJlYmJiCAgIAKpCkFasWEF5eTlhYWG88sorrFixgo0bN3Lo0CFGjBjB2rVrmTJlCp07dyYzM5NPP/2UBQsW1ErZuzOmhQsXAlUz2tTUVPr3709SUhIDBgyoVagB2rZtS3l5OSqVinPnztG9e3cyMjLw8vLC3t4eS0tL1q1bR1BQEIMHD+bAgQO6BD6Aq1evMm/ePBQKBcuWLWPSpEl4eXmhVqsxNzdn7Nix7Ny5kzlz5jTmt0cIw2Vkd4M0+jKIm5sbZ86c4fPPPycjI4M2bdoQFBTE4cOHKSkp4cKFCzz66KNA1ePZ1ZPrfHx8dKl2165d0/XZq1cvrK2tsbe3p02bNroifqedWq3m/PnzrFy5koiICNauXcuNGzf0jjEwMFD356effprExESAGj9I6nJnRp6ens6f//xnMjIyyMjIwNvbG4DMzEyefPJJoCoh8Pz587pjBw4ciEJR9fF7eXnxn//8h/j4eEpKSvQGR1WXkJDAnDlzpJgLYSQafWbt7OzM8uXLSU1N5csvv8TX15enn36a5cuXY2FhweOPP64rTncn11VPtau+tls9lU6hUNRqp9FosLGxITo6ukFjtLS01P3Zy8uLdevWkZ6ejkajwc3NTe9x3t7eZGRkkJ+fT0BAAN999x2ALoq1PlZWVro/jxw5kn79+pGamspbb73FvHnz7nl89YjUokkNz0IRotUwsmWQRp9ZFxYWYmFhQVBQEH/605/IysqiXbt2ODg4sG3btnpnrr9XmzZtcHJy4vjx40BVzGh2djZQM2VPn6CgID744IN7Pgzj7e3NkSNH6NSpEwqFAltbW37++Wd69uwJVM28k5KSgKqEwDs53HfLy8vDzc2NkSNH0q1bN65cudKgcQohjEejz6wvXbrE559/rpsp33lM+6mnnuLWrVv33CPx95o+fTqffPIJcXFxVFRU8MQTT+Du7l4jZe/NN9+s89innnqKLVu28MQTT9R7DicnJwBdJGvPnj0pKCjA1tYWgIkTJxIbG8uOHTt0FxjrEh8fT1paGgqFgi5duvDoo49iYmKCqampXGAUQg9J3Wsi69ato2vXrjz99NPNcfp6nThxguTkZMLCwpp7KA0iyyCitbBft6/BbYtnj2qUMdguj2uUfh9Us9xnPXv2bC5dusRTTz3VHKev1/r16/niiy948cUXm3soQgih0yzZIMuXL2+O0zbI3/72t1qvffrppzXu5AAYPny4BDwJ0ZyM7AKjBDk1QEuKQ61LRX5Zcw9BCNHIpFgLIQyTPBQjhBCipZGZtRDCMMmatRBCtHxaIyvWsgzyO3z11VecPn0agB9++IHS0tJmHpEQorWTmfXvMHr0aN2f4+Pjeeqpp2rkiwghmkAzzazLyspYsGABFRUVVFZWMnDgQF5++eUabcrLy1m9ejVZWVnY2dkxY8YM3RPPv5dBFetDhw6xc+dOTExMcHNzY8yYMXrzoq2trcnKyuLGjRuMGzdOF4n63XffcfjwYRQKBX5+frzyyiskJCSwf/9+Kioq6NixI2FhYVRWVhIREcGqVatQKBSUlpYyY8YMVq1axccff4y/vz+FhYUUFhaycOFC7O3teeqpp7h06RKvvvoqUJWMd+XKFSZMmFDrvdSV8x0YGEhWVhafffYZarVa954cHBya8mMWQtTD3NycBQsWYGVlRUVFBfPnz8fPz48ePXro2hw4cAAbGxtWrVrFsWPH+OKLL/jnP//5QOc1mGKdk5NDXFwcixcvxt7enuLiYlavXq03L/rGjRssWrSI3Nxcli9fzsCBA/n5559JTk5m6dKlWFpaUlxcDMCAAQN0CXZbtmzhwIEDPPfcczzyyCOkp6fTu3dvfvrpJ3x9fWtkWw8fPpwffviBBQsWYG9vj1qtZvv27YwbNw4zMzMSExN57bXX6nw/deV8V1RU6N6Dvb09SUlJbN68uc5MkYSEBBISEgCY9fA+ZiEMRzNlg5iYmOhSMysrK6msrNSlhd6RkpLCX/7yF6AqDnn9+vVotdpa7e6HwRTrs2fPMnDgQOzt7QGwtbUlMzOTmTNnAlVJeV988YWu/WOPPYZCocDFxYWbN28CcObMGQYPHqxbsrgTuJSTk8OWLVsoKSlBrVbj6+sLVOVcJyUl0bt3b44dO8awYcPqHaOVlRW9evUiNTWVLl26UFlZqTdi1c3NjU2bNvH555/j7++Pt7c3ly5dIicnh8WLFwOg0Wj0zqqrR6QW/s+ge3+AQrQ2zXiBUaPRMHv2bPLy8hg2bBienp41/r2wsJD27dsDVdHPbdq04datW7r69XsYTLG+359K1TOv72RV6esjJiaGiIgI3N3dSUxMJC0tDYCAgAC+/PJLiouLycrKonfv3vc879ChQ9m+fTvOzs71xr/WlfPdv39/XFxcWLJkSYPfpxDi4ar+WyvUnBjdoVAoiI6OpqSkhHfffZdLly7VmJjVlY/3ILNqMKC7Qfr06cPx48e5desWAMXFxQ3Oi77D19eXgwcP6u7euLMMolarcXBwoKKigiNHjujaW1lZ4eHhwYYNG/D399ft7FKdlZUVarVa93dPT08KCgo4duxYvRGrdeV8Ozs7U1RUxIULFwCoqKggJyenIR+PEMZHo22Ur+DgYJYtW6b7urtQV2djY4OPjw8nT56s8Xr79u0pKCgAqpZKVCqV7jf538tgZtaurq78+c9/JioqCoVCgbu7e4Pzou/w8/MjOzubOXPmYGZmxqOPPsrYsWMZPXo0c+fOpUOHDri5udUI/Q8MDGTlypVERUXV2WdwcDBLly7FwcGBBQsWAPD444+TnZ1d7zenrpxvMzMzwsPD2bBhAyqVisrKSoYPH46rq+v9f2BCiEZRVFSEqakpNjY2lJWVcebMGf7nf/6nRht/f38SExPp0aMHJ06coFevXg88s262POvWbNmyZYwYMYI+ffo0yflkzVq0Fu2+O9TgtkWv138N6fey/3hPvf/+66+/EhMTg0ajQavV8vjjj/PSSy/x1Vdf0b17dwICAigrK2P16tVcvHgRW1tbZsyYQceOHR9oXFKsH6KSkhLmzp3LI488oncXmsYgxVq0FvdVrKc82yhjsP9kb6P0+6AMZhnEENjY2PDBBx/UeO3WrVssWrSoVtv58+djZ2f3UM6rUd+7jRDCsEmxbmR2dnYN3mVdCHEfJBtECCFESyMzayGEQZLUPSGEEC2OURTrtLQ0li1bBlQ9s//tt98284gaJi0trdZGvUKI/9VID8W0VAa9DKLVatFqtXU+WahPQEAAAQEBjTiq+6fRaOp8D2lpaVhZWdGzZ89mGJUQLZxxbcFoeMVaqVTyzjvv0KtXLy5cuIC7uzuXLl2irKysRq7syZMn2bhxI3Z2dnTt2lV3fGJiIr/88guTJk0iJiYGf39/XXxqSEgImzZt4vr16/z73/9GpVKh0WiYPHky3t7etcaSlJREZmYmEyZMID4+nvj4eFavXk1eXh4xMTEsXryYM2fOsGnTJiorK+nevTtTpkzB3NycadOmMWTIEE6dOsUf//hHbt68yb59+zA1NcXFxYWxY8eyb98+FAoFR44c4W9/+1udYxBCGAeDK9YAubm5/P3vf2fy5MkUFxdja2uLRqNh0aJF/Prrr3Tu3JmPP/6Y+fPn06lTJ95///376v/o0aP4+voyatQoNBqN3p1gfHx82LlzJwAZGRnY2dlRWFjIuXPn8Pb2pqysjDVr1jBv3jycnZ1ZvXo1e/fuZcSIEUBV2NSdhL3XX3+d1atXY25uTklJCTY2NjzzzDNYWVnxwgsv1Dp39bCZmff17oRoHYztAqNBFmtHR0dd0HdSUhL79++nsrKS69evc/nyZbRaLU5OTnTu3Bmoik+tnqJ1L927dyc2NpaKigr69++Pu7t7ne3+8Ic/oFaruX37NgUFBTzxxBOkp6dz7tw5+vfvT25uLk5OTjg7OwMwaNAg9uzZoyvWgYGBur7c3Nz48MMPeeyxx+jfv/89x1g9CSx/mDzBKERrZ5AXGO8EfyuVSnbu3Mm8efN499136devH+Xl5Q3ux9TUFM3/BphrtVoqKiqAqhnzwoULadeuHatWreLQIf2PwHp6enLw4EGcnZ3x9vbm3LlzXLhw4Z4JgECNrcAiIyMZNmwYWVlZzJ49m8rKyga/DyGMkpFdYDTIYn2HSqXCysqKNm3acOPGDV1MobOzM0qlkry8PKBqWaMuHTp0ICsrC4Dk5GRdgbx27Rpt27YlODiYp59+mosXL+odw52lEG9vb7p27UpaWhrm5ua0adOm1jgOHz6Mj49PrT40Gg35+fn07t2bcePGoVKpUKvVWFtb14hfFUJUo2mkrxbKIJdB7nB3d8fd3Z3w8HCcnJx0d01YWFjw+uuvs2zZMuzs7PDy8qozF3ro0KFER0cTGRlJnz59dDPdtLQ0du7ciampKVZWVrzxxht6x+Dl5UVBQQHe3t4oFArat2+vW/awsLAgNDSUlStX6i4wPvPMM7X60Gg0rFq1CpVKBcCIESOwsbHB39+flStXkpycLBcYhTBykrrXCsiatWgtHPc0PHXv+l8GN8oYHL5ObJR+H5RBL4MIIYSxMOhlkKY0d+7cWhcvw8LC9G6I25Qqy5p7BEI0gxa8vtwYpFg30NKlS5t7CEKIaoztPmtZBhFCCAMgM2shhGEysmUQmVkLIYQBaNKZ9datW7GysuL27dt4e3vTt2/fpjw9KSkpXL58mZEjRzbpeX+vxMRE+vbtS7t27Zp7KEK0OFojm1k3yzLI6NGjm+O0LTIetb6Y18TERFxdXaVYC1EXKdYPV1xcHIcOHcLR0RE7Ozu6detWI5p02rRpPPHEE6SlpVFZWclrr73G5s2bycvL409/+hPPPlu13fyOHTs4fvw45eXl9O/fn5dfflkXl9qzZ08uXLhAu3btmDVrFhYWFsTHx9eIHJ0xY0aNeNRr164RGxtLUVER9vb2hIaG4ujoSExMDNbW1mRlZXHjxg3GjRuni1C926effoqfnx8BAQFER0djY2NDaGgoBw4cQKlUMmbMGL7//nsOHjwIwNNPP82IESNqxbxGRESwdetW3aPvQ4YMwdHRkV9++YUPP/wQCwsLlixZgoWFRWN/u4QQLVSjFuusrCyOHTvGihUrqKysZPbs2XTr1q1WO0dHR5YsWcLGjRtZs2YNixcvpry8nDfffJNnn32WU6dOcfXqVZYuXYpWq2XFihWkp6fj6OjI1atX+cc//sHUqVNZuXIlJ06cICgoiO+++65G5Ojd1q1bR1BQEIMHD+bAgQOsX7+eWbNmAXDjxg0WLVpEbm4uy5cv11usvb29ycjIICAggMLCQm7cuAHAuXPnCAwMJCsri4MHD7JkyRKg6l5tHx8fbGxsasS8ZmVlUVhYyHvvvQegi0jdvXs3ISEhdO/evda5q0ek/vN3fG+EMHTGtgzSqBcYMzIy6N+/P5aWlrRp00bvEsSd193c3PDw8MDa2hp7e3tdoT116hSnT59m1qxZzJ49mytXrujCkZycnHQRpt26dePatWu6vj788EMOHz6MqalprXNmZmby5JNPAlURqtW3z3rsscdQKBS4uLhw8+ZNve/vTsre5cuXcXFxoW3btly/fp0LFy7Qs2dPXVSqlZUVVlZW9O/fn4yMDKBmzKuTkxNKpZL169dz8uRJrK2t7/nZBgcHs2zZMt12ZUKI1q3Rl0FMTEzuPQizqmEoFArMzc11rysUCl0S3siRI2uFICmVylrty8qqHueLjIwkPT2dlJQUtm3bxsqVKxs85up91hed0q5dO4qLizl58iTe3t4UFxdz/PhxrKyssLa2rvfYOzGvALa2tkRHR3Py5El2795NUlISoaGhDR6vEEZJZtYPj7e3Nz/++CNlZWXcvn2bn3766Xf14+vry8GDB3VxoYWFhfXOePVFjlbXo0cPkpKSgKoI1YbkT9elR48e/PDDD/j4+ODt7c3OnTt1fXl7e5OcnExpaSlqtZrk5OQ6k/OKiorQaDQMHDiQMWPG6CJZ79w5I4QQjTqz7tatG4GBgURERNChQ4ffXRB9fX25cuUKb731FlBVxMLCwvRulKsvcrS6iRMnEhsby44dO3QXGH8Pb29vTp8+TadOnXB0dKS4uFhXkLt168bgwYOZO3cuUHWBsWvXriiVyhp9FBYWEhsbq9sIYezYsQAMHjyYTz75RC4wClEHY1uzlojUVuC3IRKRKlqHjgcbHpGqHNo4/9077W/4GJqSPMEohBAGQLJBGuDSpUusWrWqxmvm5uYtJomvrFi+jcL4GNsyiPxf3gBubm5ER0c39zCEEEZMirUQwjBp731bcGsixVoIYZCMbRlELjAKIYQBkGLdAGlpabrHulNSUvj222+beURCCK3GpFG+WiqjXgapL55Un5YYsyqEaP2MrljfHU/q7u7OpUuXKCsrY+DAgbz88ssAnDx5ko0bN2JnZ0fXrl11x1ePWa0e9QoQEhLCpk2buH79Ov/+979RqVRoNBomT55c52PmGo2G2NjYGtGozz//PHl5eaxbt46ioiIsLS15/fXX6dKlSxN8OkIYDmNbsza6Yg3UiCctLi7G1tYWjUbDokWL+PXXX+ncuTMff/wx8+fPp1OnTrz//vv31f/Ro0fx9fVl1KhRaDQaSktL62yXnZ1dKxoVYO3atUyZMoXOnTuTmZnJp59+yoIFC2ocWz0iddr9fgBCtAJauRuk9aseT5qUlMT+/fuprKzk+vXrXL58Ga1Wi5OTE507dwaqIlTvFMaG6N69O7GxsVRUVNC/f39dhOvdqkej9uvXj759+6JWqzl//nyNlMCKiopaxwYHBxMcHAxAzmNDGzw2IYRhMspifSeeVKlUsnPnTt555x1sbW2JiYmhvLy8wf2Ymprqwpe0Wq2uqPr4+LBw4UJSU1NZtWoVL7zwAoMG1c4xqCsa9dVXX8XGxkYewhHiHoxtGcSo7wZRqVRYWVnRpk0bbty4wcmTJwFwdnZGqVTqNjg4evRoncd36NBBt96cnJysy96+du0abdu2JTg4mKeffloXeXq3uqJR27Rpg5OTE8ePHweqfghkZ2c/zLcthDBARjmzvsPd3R13d3fCw8NxcnKiZ8+eAFhYWPD666+zbNky7Ozs8PLyIicnp9bxQ4cOJTo6msjISPr06YOlpSVQdavfzp07MTU1xcrKijfeeKPO8+uLRp0+fTqffPIJcXFxVFRU8MQTT+hdShHCWLXk2+wag0SktgKyZi1aC9fk/Q1u21j/3d/PGJqSUc+shRCGy9immVKsm8jcuXNrXbwMCwvDzc2tmUYkhGEztmUQKdZNpKVkXwshDJMUayGEQTK2mbVR37onhBCGQmbWQgiDZGwXGA1iZr13714OHXo4Ow7HxcU9lH6EEM3L2CJSW3yxrqys5Nlnn63zce3fY/v27fd9zJ2HVoQQork0yTKIUqlk6dKleHh4kJ2dTefOnXnjjTe4cuUKn332GWq1Gnt7e0JDQ3FwcCAqKooePXpw/vx5AgICuH37NlZWVrzwwgtERUXh7u7OxYsXKSoqYtq0aXz77bdcunSJwMBAxowZA8Dhw4fZtWsXFRUVeHp6MnnyZDZv3kxZWRkRERG4uroyffr0OtspFApCQkJ4/vnnOXXqFOPHj8fLy6vW+5o2bRpPPPEEaWlpVFZW8tprr7F582by8vL405/+xLPPPgvAjh07OH78OOXl5fTv318Xw7pixQoKCgooLy9n+PDhumCmkJAQhg8fTmpqKhYWFkRERPCHP/yhKb5VQhgMSd1rJLm5uUydOhUvLy/WrFnDnj17+PHHH5k1axb29vYkJSWxefNmQkNDgarcjoULFwKwdevWmoM2M2PhwoXEx8cTHR3NsmXLsLW1JSwsjBEjRnDz5k2SkpJYvHgxZmZmfPrppxw5coRXXnmF3bt360KSLl++XGe7QYMGUVpaiqurK6NHj673fTk6OrJkyRI2btzImjVrWLx4MeXl5bz55ps8++yznDp1iqtXr7J06VK0Wi0rVqwgPT0dHx8fQkNDsbW1paysjMjISAYMGICdnR2lpaV4enry17/+lc8//5z9+/fz4osv1jivRKQKYVyarFi3b99eNzsNCgpi+/bt5OTksHjxYqBqqcHBwUHXPjAwUG9fd3ZqcXNzw8XFRXdcx44dKSgo4Ny5c1y8eJHIyEgAysrKsLe3r9XP2bNn9bZTKBS6TQXqU30sarUaa2trrK2tMTc3p6SkhFOnTnH69GlmzZoFgFqtJi8vDx8fH+Lj40lOTgYgPz+fq1evYmdnh5mZGf7+/gB069aN06dP1zqvRKQKY2dsqXtNVqxNTGr+ymJlZYWLiwtLliyps/2dUKS6mJub6/q88+c7f6+srESr1TJo0CBdMJI+9bUzNzdv0HZfZmZVH6FCoagxFoVCoUvhGzlyJM8880yN49LS0jhz5gxvv/02lpaWREVF6Z5wNDU11X1e1fsRQvwfjZEtgzTZBcb8/HwuXLgAVEWOenp6UlRUpHutoqKizmS736NPnz6cOHGCmzdvAlBcXMy1a9eAquJ6J3e6vnYPi6+vLwcPHkStVgNVSXs3b95EpVJhY2ODpaUlV65cITMz86GeVwjRujTZzLpLly4kJiaydu1aOnXqxHPPPYefnx8bNmxApVJRWVnJ8OHDcXV1feBzubi4MGbMGN5++220Wi2mpqZMmjSJDh06MHToUCIiIujatSvTp0/X2+5h8fX15cqVK7z11ltA1W8UYWFh+Pn5sW/fPmbOnImzszOenp4P7ZxCGANju8DYJBGpSqWS5cuX6/YaFA+XrFmL1uJ+4knPez3XKGPoeW5Xo/T7oOQJRiGEQWrJD7A0hiYp1k5OTgY9q46OjkapVNZ47ZVXXsHPz6+ZRlRT2W35mStEayf/lzdAREREcw9BCHEXY8sGkWIthDBIxrYM0uKzQYQQQsjMWghhoOShGFHDRx99xOXLl+tt8+OPP96zjRBCPAgp1vcwdepUXFxc6m2TnJwsxVqIJqbVmjTKV0tldMsg+uJaL1y4wKZNm6isrKR79+5MmTIFc3NzoqKiCAkJoXv37nVGl/7222+kpKSQnp7Otm3bCA8PJzU1lX379mFqaoqLiwszZsyocyzp6els2LABqMo1WbhwIdbW1nojVYUQ/0fuBjECd8e1fv/99yQkJDBv3jycnZ1ZvXo1e/fuZcSIETWO0xddGhAQgL+/vy6l77vvvmP16tW65D19duzYwaRJk/Dy8kKtVmNubl5vpGp11SNSpzzkz0cI0fIY5TLI3XGtZ8+excnJCWdnZwAGDRpERkZGrePuji7VF/rk5ubGhx9+yOHDhzE1NdU7Di8vL/7zn/8QHx9PSUkJpqamNSJVZ8+ezZUrV8jLy6t1bHBwMMuWLWPZsmX3/f6FaA00WpNG+WqpjHJmfXdca0M1NLo0MjKS9PR0UlJS2LZtGytXrqyzaI8cOZJ+/fqRmprKW2+9xbx583Sv3x2pKoQwbkY5s747rrVPnz4olUrdDPbw4cO1lh3qY21tze3bt4GqTRTy8/Pp3bs348aNQ6VS6eJR75aXl4ebmxsjR46kW7duXLlyRW+kqhCiJrnAaATujmudOHEinp6erFy5UneB8X5mtoGBgXz88cfs2rWLGTNmC/II0AAAIABJREFUEBsbi0qlAmDEiBHY2NjUeVx8fDxpaWkoFAq6dOnCo48+irm5eZ2Rqm3btn3wNy5EK2JsFxibJCK1JWmNca2/9B7W3EMQ4qHofnZPg9umuv5Po4yhX853jdLvgzLKmbUQwvC15IuBjcHoinVzxLUePHiQ+Pj4Gq/17NmTyZMnP5T+KyuN6z9aIYyR0RXr5jBkyBCGDBnS3MMQolVpyRcDG4NR3g0ihBCGRmbWQgiDZGxr1gY7sy4pKWHPnoZfORZCtC7aRvpqqQy6WO/du7e5hyGEEE2iQcsghw4dYufOnZiYmODm5saYMWOIjY2lqKgIe3t7QkNDcXR0JCYmBgsLC3Jzc7l27RqhoaEkJiaSmZmJh4cH06ZNAyAkJIRnnnmGtLQ0bGxsmDFjBvb29iQkJLB//34qKiro2LEjYWFhWFpacuPGDT755BPdprWTJ09m165d5OXlERERQd++fenXrx9ff/01dnZ25OTk0K1bN8LCwjAxMSErK4vPPvsMtVqtG6+DgwPx8fG10vH0JeHdLS0tja1bt9K2bVt+/fVX+vfvj5ubG/Hx8ZSVlREREUGnTp0oKipi7dq1FBQUADBhwgS8vLz4f//v/7Fx40bKysqwsLAgNDQUZ2dnEhMTSUlJobS0lN9++43+/fszbty4h/LNFqI1MbZlkHsW65ycHOLi4li8eDH29vYUFxezevVqgoKCGDx4MAcOHGD9+vXMmjULqJrxzp8/n5SUFJYvX87ixYtxcXEhMjKS7Oxs3N3dKS0tpWvXrowfP55vvvmGr7/+mkmTJjFgwACCg4MB2LJlCwcOHOC5555jw4YN+Pj4EBERgUajQa1WM3bsWHJycoiOjgaqiufFixdZuXIlDg4OzJs3j/Pnz+Ph4aEbn729PUlJSWzevJnQ0NA60/HqSsLT59dff+X999/H1taWN954g6FDh/LOO+8QHx/P7t27efXVV9mwYQPPP/88Xl5e5Ofns2TJEt5//32cnZ1ZuHAhpqamnD59mi+//JKZM2cCkJ2dzYoVKzAzM2PGjBn88Y9/xNHR8cG+00IIg3bPYn327FkGDhyIvb09ALa2tmRmZuoKS1BQEF988YWuvb+/v24G3rZtW9zc3ABwdXVFqVTi7u6OiYkJgYGB/7+9O4+Lstz/P/5iGBAQMRRRESfckHHDFNFccEmt49Kx7KQZllvWV7M8IiaaJ81QgY5WSqaJ2nHNc7SS0rJSckGPGrmBuxKgEiIqwThsM78/+HEfEFAUB2aYz9PHPB4y3fd1X/dgHy6u+77eNwC9evXiww8/BAp/MGzevJns7Gz0ej2+vr5KH958802gMEDJycmJrKysUn1t2bIl9evXB8DLy4u0tDScnJxITk5m/vz5QGF2h6urK/C/dLwuXbrg7+8P/C8Jr2fPnnTt2lVprywtWrRQ2mrUqBEdOnRQ2j116hQAJ0+eLPFgAp1Ox507d9DpdERGRip5JMVDodq1a4eTkxMAnp6epKenlyrWxSNSx5XbQyFqLmu7de++xdpoND5QSl3RSNTGxqbEqNTGxgaDwVDmPkXtR0ZGEhwcjJeXFzExMcTHx1f4uMWPDYVFveh4np6ehIaGltq+rHS8spLwmjRpct/jFT/f4udqNBoJDQ3F3t6+xL6rV6+mbdu2BAcHk5aWxrx588o9j7LS/fr376/8FnLum2fu/cEIUQOVXU1qrvteYGzfvj0HDx7kzz//BCArKwtvb29iY2OBwtS6omzoijIajRw6dKjU/nq9HldXV/Lz89m3b1+JPhRdTDQYDOh0uhJJd/fi4eFBZmamkrKXn59PcnJyuel4ZSXhVUaHDh34/vvvla8TExOBwhF2vXr1AIiJianUMYQQNd99R9ZNmzblueeeY+7cuahUKry8vBg7dizLly9n+/btygW7B1GrVi2Sk5N55513cHJy4u9//zsAI0aMYNasWTRo0ACNRqMU4zFjxrBy5Up2796NSqXitddew9vbm9atWxMUFETHjh3p1KlT2SeoVhMUFMSaNWvQ6XQUFBQwaNAgGjduzNKlS0ul43355ZelkvAqY+zYsURFRTF9+nQKCgrQarVMnDiRv/71r0RGRvLdd9/Rtm3bSh1DCGtkxLqmQaoldW/06NGsW7euqg9bY53TyjSIqBm8T39//43+v72N/maSPgSk/vue//3TTz8lLi6OunXrlpkzFB8fT3h4OO7u7gB07dqVF154odL9khWMQgiLZKimFSx9+vThmWeeITIystxttFotM2fOfKTHrZZibUmj6qSkJJYuXVriPTs7OxYsWFBNPRJCABiqaRqkTZs2ypqPqiQj6/vQaDTKvdzmKkdf/r3gQoiqd+7cOYKDg3F1dWX06NE0bdq00m1KsRZCWCRTXWAsvoYBSt4mWxHNmjXj008/xcHBgbi4OCIiIvjkk08q3S8p1kIIUcyDFue7FS1oA+jUqRNRUVFKNEdlWGyQkxDCuhlM9KqsW7duUXST3YULFzAYDNSpU6fS7crIWgghHsBHH31EQkICf/75J2+88QYvvvgi+fn5AAwcOJBDhw6xa9cubG1tsbe3Z+rUqQ+0Crw8Vvd0c4Avv/wSrVZLhw4d+O677+jfvz+1atWq7m6VcPjwYTw8PPD09LzvtiebDa2CHglheu0vR1d4210NR5qkDwP/2GySdivLKqdBRowYoYQu7dixg5ycnGrrS3l5KUeOHCkRACWEKMlcp0FMpUqnQR4kF9vR0ZFLly5x69YtAgMD6datGwDffPMNe/fuRaVS0bFjR15++eUyc7ALCgoIDg5m6dKlqFQqcnJymDp1KkuXLmXFihV07tyZjIwMMjIymDdvHi4uLvTq1YukpCTGjBkDFF4VvnLlCq+++mqpc/nmm2+ws7Nj0KBBrF27lt9//5333nuPkydPsmfPHt566y3279/PV199BcATTzyh5FKPHj2aIUOGcPz4cV555RV+/fVXjh49iq2tLR06dKBr164cPXqUhIQEtm7dSlBQEI0aNaqab5IQwixVWbF+0FzsW7du8f7773P16lXCwsLo1q0bv/32G0eOHGHBggXUqlVLiUktLwf78ccfJyEhgXbt2vHrr7/i6+uLWv2/Ux40aBDfffcd7733Hi4uLuj1er766isCAwNRq9XExMQwceLEMs9Hq9Xy7bffMmjQIC5dukReXh75+fmcOXMGrVZLRkYGGzZsICwsjNq1a/PBBx9w+PBh/P39ycnJoWnTpowYMYKsrCyWL1/ORx99hI2NDdnZ2dSuXRs/Pz86d+6s/JC6W/Hbi15+ZN8lISyHOY+CTaHKpkHKy8Xu2bMnUJiLffbsWWX7Ll26oFKp8PT05Pbt20BhNnSfPn2U+WVnZ2eg8AfBP/7xD4KCgti3bx/JyckAdO/eXUkHPHDggJKhXR4HBwfatm1LXFwcV65coaCgQMnjvlvz5s25dOkSd+7cwc7ODm9vby5duqQU64sXL9K2bVtcXFywtbWlV69enD59GiiMPS0qwo6Ojtjb2/PZZ5/x3//+t8Jz5/3792fRokUsWrSoQtsLISxblRXrh83FLtr3Xm1ERkYybtw4/vnPf/K3v/2NvLw8APz8/Dh27BhZWVlcunSJdu3a3fe4Tz31FDExMezZs4c+ffqUu51araZBgwbs2bMHb29vtFotp06dIjU1lSZNmnCv67Z2dnaoVIUfva2tLQsWLKBr164cOXKkzNxtIURpRmxM8jJXVVasH0Uutq+vL3v27FEuCBZNg5SXg+3g4EDLli1Zs2YNnTt3VgpkcQ4ODuj1euXrVq1acePGDQ4cOECPHj3u2R+tVkt0dDRarRYfHx9+/PFH5Uk4rVq1IiEhgczMTAwGAwcOHKBNmzal2tDr9eh0Ojp16sSYMWOUvOuK5nULYa0MNqZ5masqm7N+FLnYHTt2JDExkZkzZ6JWq3niiScYNWpUuTnYUDgVsnjxYubOnVtmm/3792fBggW4urry3nvvAfDkk0+SmJioTLOUR6vV8tVXX+Ht7Y2DgwP29vZotVoAXF1dGTVqlPIEmCeeeIIuXbqUauPOnTuEh4eTl5eH0WhULmZ2796dFStWsHPnTqZNmyYXGIWwclZ5n/X9LFq0iMGDB9O+ffvq7kqFyH3WoqZ4kPusv2k0yiR9+GvqRpO0W1lWeZ91ebKzs3n77bext7e3mEIthLAOsty8mNq1a/Pxxx+XeO/PP//k/fffL7XtP/7xj0ey3v9RSNBXLiBGCHPxIEMka5sSkGJ9H3Xq1DH7PGshrJHcZy2EEMLsyMhaCGGRDI8gyc6SyMhaCCEsQI0v1vHx8cqS7KNHj/L1119Xc4+EEI+C0UQvc2Wx0yBGoxGj0VjmqsTy+Pn54efnZ8JeCSGEaVhUsU5LS2PhwoW0bduWc+fO4eXlRVJSErm5uXTr1o0XX3wRgGPHjrF27Vrq1KlDs2bNlP1jYmK4ePEi48ePJzIyskSq3ejRo1m3bh03b97ko48+QqfTYTAYmDBhgrIq8W6jR4/m6aef5uTJkzg7O/PSSy+xfv160tPTGTNmDH5+fhgMBjZs2EBCQgJ5eXk8/fTTDBgwAL1eT3h4ONnZ2eTn5zNy5Ei6dOminGPr1q05d+4c9erVY8aMGdjb25v+AxbCgljb3SAWVawBrl69yv/93/8xYcIEsrKycHZ2xmAw8P777/P777/TuHFjVqxYwT/+8Q8aNWrEkiVLHqj9/fv34+vry/PPP4/BYLjngwlycnJo27YtgYGBREREsHnzZt59911SUlKIjIzEz8+P3bt34+TkxMKFC8nLy2POnDn4+vpSv359pk+fjpOTE5mZmcyePVsZ9V+7do23336bN954g8WLF3Po0CECAgJKHLt4ROoTD/gZClETmHOOhylYXLF2c3PD29sbgNjYWH7++WcKCgq4efMmKSkpGI1G3N3dady4MVAYvVr8sfL306JFC5YvX05+fj7+/v54eXmVu61araZjx44AaDQa7OzsUKvVaDQarl+/DsDx48dJSkri0KFDAOh0Oq5du0a9evXYtGkTp0+fxsbGhoyMDCUK1t3dXTlu8+bNlbaKK/4E5i+/kERrIWo6iyvWDg4OQOGUSHR0NAsXLsTZ2ZnIyEglGrUibG1tlUdqGY1G5YGXbdq0Yd68ecTFxbF06VKeffZZevfuXW4bRZGtNjY2yoMNVCoVBQUFSttjx45VinqRmJgYMjMzWbRoEWq1msmTJ5ObmwuUjIdVqVTK+0KI/zGYcZypKVjs3SA6nQ4HBwecnJy4desWx44dA8DDw4O0tDRSU1OBwmmNsjRo0IBLly4Bhc87LCqu169fp27duvTv359+/fpx+fLlSvWzY8eO7Nq1S/lhcPXqVSUWtW7duqjVak6dOlXm6FkIIYpY3Mi6iJeXF15eXgQFBeHu7k7r1q0BsLe35/XXX2fRokXUqVMHHx8f5ckxxT311FNEREQQEhJC+/btlSe0xMfHEx0dja2tLQ4ODrz55puV6me/fv1IS0vjnXfeAcDFxYXg4GB69uxJWFgYM2fOxMvLiyZNmlTqOEJYG3O+zc4UJCK1BviyscxZi5phxLUNFd72X00CTdKHV66sN0m7lWWx0yBCCGFNLHYapCrNmjWr1MXLKVOmlPsw3ap2U21dF1qEALnPWpRhwYIF1d0FIYSVk2IthLBI1naxTYq1EMIiWdsKRrnAKIQQFsDsi3VaWlqJhS0xMTFERUVVY4+qRnx8PGfPnq3ubghhtgwmepkrsy/W169fL3cVYk1QtOT9blKshRDFPfSctV6vZ8mSJWRkZGAwGBg+fDgbNmygR48exMfHU1BQwMSJE9m0aROpqakMHTqUgQMHYjQaWb9+vbI8fPjw4XTv3r3c9zdu3EhKSgrBwcH07t0bZ2dnbt68SWhoKH/88Qf+/v4EBhbeHD969GgGDRpEXFwc9vb2BAcH89hjj5GZmcnKlSu5ceMGAK+++io+Pj4kJCSwZs0aoDDbY968eej1+gpFpMbGxnL+/HleffVVduzYwY4dO1i2bBmpqalERkYyf/58Tp48ybp16ygoKKBFixa89tpr2NnZMXnyZPr27cvx48d55plnuH37Nj/++CO2trZ4enoyatQofvzxR1QqFfv27WPcuHHlxrQKYa3MeRRsCg9drI8dO4arqyshISFAYVbHhg0bcHNzIzQ0lLVr1/Lpp58yf/588vLymDZtGgMHDuS///0viYmJREREkJmZSUhICFqtlrNnz5b5/qhRo4iOjmbmzJlA4TRIYmIi4eHhqNVqpk6dyjPPPIObmxs5OTm0atVKyZX++eefGT58OGvWrGHIkCH4+PiQnp5OaGgoS5YsYfv27YwfPx4fHx/0ej12dnb89NNPFYpIbdOmDdHR0QCcPn2aOnXqkJGRwZkzZ9BqteTm5vLpp58yZ84cPDw8WLZsGbt27WLw4MFAYVjT/PnzAXj99ddZtmwZdnZ2ZGdnU7t2bQYMGICDgwPPPvtsmccvHpHq9bDfRCGExXjoYq3RaFi3bh3r16+nc+fOysivKJNZo9Gg1+txdHTE0dFRKURnzpyhR48eqFQqHnvsMdq0acPFixfLfd/R0bHUsdu1a4eTkxMAnp6epKen4+bmhlqtpnPnzkBhtOiJEycAOHnyJCkpKcr+Op2OO3fu4OPjw7/+9S969uxJ165dqV+/foUjUh977DH0ej137tzhxo0b9OjRg4SEBM6cOYO/vz9Xr17F3d0dDw8PAHr37s0PP/ygFOvu3buX+Cw/+eQTunTpgr+/f4U+/+IRqZ9tMM2yWyHMmdHK7gZ56GLt4eFBWFgYcXFxbNy4EV9f38IGi8WE3h31WZRsV1nltVs8svTumNLQ0NBST1sZNmwYnTp1Ii4ujtmzZzNnzpwHikht1aoVe/bswcPDA61Wy549ezh37hyvvPIKaWlp9zyHouAogJCQEBISEjh69Chbt25l8eLFD/6hCGFlrG0a5KEvMGZkZGBvb09AQABDhw5V4kbvR6vVcvDgQQwGA5mZmZw+fZqWLVuW+76joyN37tx52G4C0KFDB77//nvl68TERABSU1PRaDQMGzaM5s2bc+XKlQeKSC2aCtFqtTRr1oz4+Hjs7OxwcnIqFdW6d+9e2rRpU6oNg8FAeno67dq1IzAwEJ1Op/xGotfrK3XeQoia46FH1klJSaxfv14J3Z8wYUKFRoT+/v6cO3eO4OBgAAIDA3nsscfKfd/Z2RlbW9sSFxgf1NixY4mKimL69OkUFBSg1WqZOHEiO3bsID4+HpVKRZMmTXjiiSc4cOBAhSNSfXx8uHHjBlqtFpVKRf369ZVpD3t7eyZNmsTixYuVC4wDBgwo1YbBYGDp0qXodDoABg8eTO3atencuTOLFy/myJEjcoFRiDJY28haIlJrgM+aypy1qBneSK54POkyE/27f/MB+lCVZLm5EMIiWdsoU4p1BZh7ROots1/aJMSjZ23ZIFKsK0AiUoUQ1U2KtRDCIlnbBUb5BVoIISyA2Rbr7Oxsfvjhh4fa9+6kPktUE85BCFOS1D0zkZ2dza5dux5q34dN6isvAc+UylvVWdPTBoWoLKOJXubKbOesN27cSGpqKsHBwXTo0IG6dety8OBB8vLy8Pf358UXX+TChQt89tlnLFiwAIPBwKxZs5g6dWqZSX0XL15k/PjxACxatIihQ4fStm1bRo8ezZAhQzh+/DivvPIK9vb2fPHFF+j1elxcXJg0aRKurq6l+nf79m0WLFhAWFgYiYmJzJgxg08//RQ3NzemTJnChx9+SGZmJsuXLyczM1Npy83NjcjISJydnUlMTKRZs2b4+fmVSv+7+xyGDBlSpZ+/EMK8mG2xHjVqFMnJyURERHD8+HEOHTrEggULMBqNhIeHk5CQQJs2bfDz82Pz5s3k5ubSq1cvNBpNmUl95cnJyaFp06aMGDGC/Px85s6dy4wZM3BxcSE2NpZNmzYxadKkUvvVrVuXvLw8dDodZ86coUWLFpw+fRofHx9cXFyoVasWUVFRBAQE0KdPH3bv3s3q1auZMWMGANeuXWPOnDmoVCoWLVpUKv3v7nMQQpQkt+6ZoePHj3PixAml0On1elJTU2nTpg0vvPACISEh2NnZMW7cuAduW6VS0a1bNwCuXr1KcnKyEl1qMBjKHFUX8fb25uzZsyQkJPDcc89x7NgxjEajsjT8/PnzTJ8+HYCAgAA2bNig7NutWzdUqsJZqLLS/+6neETqYw981kIIS2MRxRoKE/LKytbIyspCr9eTn59Pbm4uDg4OpbZRqVQUX1VffIGLnZ2dUjShMHI1NDS0Qn3SarWcPn2a9PR0/Pz8+OabbwCUmNZ7Kd7PstL/7qd4ROqiTbLcXFgfc74YaApme4GxeNqer68ve/bsUVLoMjIyuH37NgArVqxgxIgR9OrVSxm53p3U5+7uTmJiopJwd+HChTKP6eHhQWZmJufOnQMgPz+f5OTkcvuo1WrZt28fjRo1QqVS4ezszG+//Ubr1q2BwpF3bGwsAPv378fHx6fMdspK/3sUaYNCiJrDbEfWderUoXXr1gQFBdGxY0d69uzJ7NmzgcJR6ZQpUzh27Bi2trb07NkTg8HAu+++y6lTp/Dx8SmR1Dd48GDc3d2ZPn06TZs2pVmzZmUeU61WExQUxJo1a9DpdBQUFDBo0CCaNm1a5vbu7u4ASvRp69atuXHjhpIMOHbsWJYvX8727duVC4xlKSv9z8bGpsQ5yAVGIUoy5zs3TEFS92qARY/LNIioGWb+XvHEu9DHXzZJH2b/vuH+G1UDs50GEUII8T9mOw1iTlatWsXZs2dLvDdo0CD69u1bTT0SQljbBUYp1hUwYcKE6u6CEMLKSbEWQlgka7vYJsVaCGGRrG0aRC4wCiGEBTDLYm3t8ahCiPsz2JjmZa7MtlhbQzyqEEJUlFnOWZt7PCrA3Llz8fLy4vLly2RmZjJ58mS+/vprkpKS6N69OyNHjgRg79697Ny5k/z8fFq1asWECRNQqVR8/vnnXLx4kdzcXLp168aLL74IwOTJk+nduze//vor+fn5TJs2jSZNmlTNBy+EBTFY2SVGsyzW5h6PWkStVjNv3jx27NhBREQEixYtwtnZmSlTpjB48GBu375NbGws8+fPR61Ws2rVKvbt20fv3r156aWXcHZ2xmAw8P777/P777/z+OOPA4VL7cPCwvjhhx+Ijo7mjTfeeKSfrxA1gXWVajMt1sWZazwqgJ+fHwAajQZPT09l+4YNG3Ljxg3OnDnD5cuXCQkJASA3NxcXFxcAYmNj+fnnnykoKODmzZukpKQoxbpr164ANG/enMOHD5d5bIlIFcK6mH2xBvOMRy3aHwqf7lL096KvCwoKMBqN9O7dm1GjRpXYLy0tjejoaBYuXIizszORkZEl+qVWq5W+l/fYL4lIFdbO2q4ymeUFRkuIR62I9u3bc+jQIaW/WVlZXL9+HZ1Oh4ODA05OTty6dYtjx45V6jhCiJrPLEfWlhCPWhGenp6MHDmSDz74AKPRiK2tLePHj8fb2xsvLy+CgoJwd3dX8q+FEBVnbRcYJSK1BpCIVFFTPEhE6gyvl0zSh/DETSZpt7LMchpECCFESWY5DWJOJB5VCPNkbRcYpVjfhyXEoxZY2dydENZIirUQwiJZ2wVGmbMWQggLICNrIYRFsq5xtRRrIYSFkguMJrJlyxYcHBy4c+cOWq2WDh06VNWhATh69CgpKSkMGzasSo8rhBCPQpWPrEeMGFHVhwQKQ5eKgpeEEJbPaGUTISYt1tu2beOXX37Bzc2NOnXq0Lx5cyIjI+ncuTPdunVj8uTJ9OjRg/j4eAoKCpg4cSKbNm0iNTWVoUOHMnDgQAC2b99eKs86LS2NhQsX0rp1a86dO0e9evWYMWMG9vb27Nixgx9//BFbW1s8PT2ZOnUqMTExSq719evXWb58OZmZmUputZubG5GRkTg6OnLp0iVu3bpFYGCgksp3t/j4eLZs2ULdunX5/fff8ff3R6PRsGPHDnJzcwkODqZRo0ZkZmaycuVKbty4AcCrr76Kj48PFy5cYO3ateTm5mJvb8+kSZPw8PAgJiaGo0ePkpOTwx9//IG/vz+BgbJCUQhrZ7JifenSJQ4cOEB4eDgFBQW88847NG/evNR2bm5uhIaGsnbtWj799FPmz59PXl4e06ZNY+DAgRw/fpxr166VyrN2c3Pj2rVrvP3227zxxhssXryYQ4cOERAQwDfffMOyZcuws7MjOzu71DGjoqIICAigT58+7N69m9WrVysRrLdu3eL999/n6tWrhIWFlVusAX7//XeWLFmCs7Mzb775Jk899RQLFy5kx44dfP/994wZM4Y1a9YwZMgQfHx8SE9PJzQ0lCVLluDh4cG8efOwtbXlxIkTbNy4kenTpwOQmJhIeHg4arWaqVOn8swzz+Dm5lbi2MUjUus89HdJCMslc9aPyOnTp/H396dWrVoA5U5BFM+E1uv1ODo64ujoqBTa8vKs3dzccHd3x8vLCyjMfr5+/brS1ieffEKXLl3w9/cvdczz588rhTEgIEBJ7APo0qULKpUKT09PJS2vPC1atFAyrBs1aqTMw2s0Gk6dOgXAyZMnSUlJUfbR6XTcuXMHnU5HZGQkqampACWiUNu1a4eTkxNQGAaVnp5eqlgXj0gN3fTyPfspRE1kbfdZm3QaxMbm/k+fLJ7dXDwTuniWc1l51mlpaaW2z83NBSAkJISEhASOHj3K1q1bWbx4cYX7XLzN+2Vc3Z1hXTzfuuiZjkajkdDQUOzt7Uvsu3r1atq2bUtwcDBpaWnMmzevzHbvlWkthLAeJlsUo9VqOXz4MLm5udy5c4dff/31odq5V551WYpyq9u1a0dgYCA6nU7Zt4i3tzexsbEA7N+/Hx8fn4fqW0V06NCB77+uiZ4PAAAgAElEQVT/Xvk6MTERKBxh16tXD7j3o8eEEGUzmuhlrkw2sm7evDndu3cnODiYBg0aPHRB9PX15cqVK6XyrIs/4aU4g8HA0qVL0el0AAwePJjatWuX2Gbs2LEsX76c7du3KxcYTWXs2LFERUUxffp0CgoK0Gq1TJw4kb/+9a9ERkby3Xff0bZtW5MdXwhRM0iedQ0Q+rjMWYuaYfbvG+6/0f/3utffTNKHFYn/Nkm7lSUrGIUQFknuBhElJCUlsXTp0hLv2dnZsWDBgmrqUWl/2ORXdxeEECYmxfo+NBoNERER1d0NIcRdrG0Fo0SkCiGEBZCRtRDCIlXXnPWxY8dYs2YNBoOBp556qlQ4XF5eHsuWLePSpUvUqVOHqVOn4u7uXunjWtXI+vDhwyVWEwohxIMwGAxERUUxa9YslixZwoEDB0rVlN27d1O7dm2WLl3K4MGDS6yQrgyrKtZHjhwpt1jLKkEhLIvRRH/u5cKFCzRq1IiGDRuiVqvp3r07R44cKbHN0aNH6dOnDwDdunXj1KlT910NXREmnQbR6/UsWbKEjIwMDAYDw4cP58CBAwQHBwNw4sQJdu3axfTp0xk9ejRPP/00J0+exNnZmZdeeon169eTnp7OmDFj8PPzIyYmhsOHD2MwGEhOTmbo0KHk5+ezd+9e7OzsCAkJwdnZmdTUVKKiosjMzKRWrVq8/vrrZGVlcfToURISEti6dStBQUF89tlneHt7c/bsWdq1a0dMTAwff/wxarUanU5HcHCw8vXd5s6di5eXF5cvXyYzM5PJkyfz9ddfk5SURPfu3Rk5ciQAe/fuZefOneTn59OqVSsmTJiASqXi888/5+LFi+Tm5tKtWzdefPFFACZPnkzv3r359ddfyc/PZ9q0aTRp0sSU3yYhLFJ1TINkZGRQv3595ev69etz/vz5crextbXFycmJP//8ExcXl0od26TF+tixY7i6uhISEgIULrHesmWLEk26Z88e5SdQTk4Obdu2JTAwkIiICDZv3sy7775LSkoKkZGRSuBTcnIy4eHh5OXlMWXKFF5++WXCw8NZu3Ytv/zyC4MHD2blypW89tprNG7cmPPnz7Nq1Sree+89/Pz8lHjWIjqdTsnluH79OnFxcfj7+xMbG0vXrl3LLNRF1Go18+bNY8eOHURERLBo0SKcnZ2ZMmUKgwcP5vbt28TGxjJ//nzUajWrVq1i37599O7dm5deeglnZ2cMBgPvv/8+v//+O48//jgAderUISwsjB9++IHo6GjeeOMNU3x7hBBlKJ5oCSVD08oaId+dgVSRbR6GSYu1RqNh3bp1rF+/ns6dO6PVagkICGDv3r307duXc+fO8eabbxZ2RK2mY8eOyn52dnao1Wo0Go2SpgfQtm1bJZnPycmpRGpfUlISer2es2fPlghvys8v/z7k7t27K3/v168f27dvx9/fnz179vD666/f8/yKH9vT01NJ4GvYsCE3btzgzJkzXL58WflhlZubq/x0jY2N5eeff6agoICbN2+SkpKiFOuuXbsChUv2Dx8+XOax7/4HJYS1MZho8XXx4ny3+vXrK9n0ADdu3FD+v797m/r161NQUIBOp8PZ2bnS/TJpsfbw8CAsLIy4uDg2btyIr68v/fr1IywsDHt7e5588klsbW2Bwl8Xin762NjYlEjjKz6ffHci3d3bGQwGateuXeF7o4siXAF8fHyIiooiISEBg8GARqO5577FU/buTuArKCjAaDTSu3dvRo0aVWK/tLQ0oqOjWbhwIc7OzkRGRpKXl6f89/LOvbji/6De2lw9T98Rwtq0aNGCa9eukZaWRr169YiNjeWtt94qsU3nzp2JiYnB29ubQ4cO0bZt20cysjbpBcaMjAzs7e0JCAhg6NChXLp0iXr16uHq6srWrVuVKZBHycnJCXd3dw4ePAgU/kpSlHTn6OjInTt37rl/QEAAH3/8MX379q10X9q3b8+hQ4eUlMCsrCyuX7+OTqfDwcEBJycnbt26xbFjxyp9LCGsTXWk7tna2jJu3DhCQ0P5+9//zpNPPknTpk358ssvOXr0KFD4G3pWVhZTpkzh22+/5eWXH012j0lH1klJSaxfv14ZKU+YMAGAXr168eeff+Lp6WmS47711lt8/vnnbNu2jfz8fHr06IGXlxfdu3dnxYoV7Ny5k2nTppW5b69evdi8eTM9evSodD88PT0ZOXIkH3zwAUajEVtbW8aPH4+3tzdeXl4EBQXh7u5O69atK30sIaxNdT18oFOnTnTq1KnEe8WfLWtvb19ufamMakndi4qKolmzZvTr16+qD31fhw4d4siRI0yZMqW6u1Jhb3nJNIioGT5J/LLC2456/DmT9GHj71+ZpN3KqvIVjO+88w4ODg688sorVX3o+1q9ejW//fabckFQCGG+rC0bpMqLdVhYWFUfssLGjRtX6r1Vq1Zx9uzZEu8NGjTokcxpCyFERUk2yH0UzbObsyxk9aWwPpJnLYQQFsDanm5uVdkgQghhqWRkLYSwSNZ2gdGqR9bZ2dn88MMP1d0NIYS4L6sv1rt27arubgghHoLBRC9zVWXTIL/88gvR0dHY2Nig0WgYOXIky5cvVxL4Jk2ahJubG5GRkdjb23P16lWuX7/OpEmTiImJ4fz587Rs2ZLJkycDMHr0aAYMGEB8fDy1a9dm6tSpuLi48NNPP/Hzzz+Tn59Pw4YNmTJlCrVq1eLWrVt8/vnnpKWlAYV3eezcuZPU1FSCg4Pp0KEDnTp14t///jd16tQhOTmZ5s2bM2XKFGxsbLh06RJffPEFer1e6a+rqys7duzgxx9/xNbWFk9PT6ZOnUpCQgJr1qwBCnNC5s2bh6OjY6nP5ObNm3z00UfodDoMBgMTJkxAq9Vy/PhxtmzZopzDpEmTcHBwqKpvlRDCDFVJsU5OTmbbtm3Mnz8fFxcXsrKyWLZsGQEBAfTp04fdu3ezevVqZsyYARSOeP/xj39w9OhRwsLCmD9/Pp6enoSEhJCYmIiXlxc5OTk0a9aMV155hf/85z/8+9//Zvz48XTt2lUJONq8eTO7d+/mL3/5C2vWrKFNmzYEBwdjMBjQ6/WMGjWK5ORkJfQpPj6ey5cvs3jxYlxdXZkzZw5nz56lZcuWSv9cXFyIjY1l06ZNTJo0iW+++YZly5ZhZ2dHdnY2ANu3b2f8+PH4+Pig1+tLhDwVt3//fnx9fXn++ecxGAzk5OSQmZnJtm3bmDNnDg4ODnz99dd8++23vPDCC1XwnRLCclTD4utqVSXF+tSpU3Tr1k2JB3V2dub8+fNMnz4dKAxPKv7om86dOysj8Lp16yrpd02bNiUtLQ0vLy9sbGyUeNNevXrx4YcfAoU/GDZv3kx2djZ6vR5fX1+lD0VxrCqVCicnJ7Kyskr1tWXLlkpwuJeXF2lpaTg5OZGcnMz8+fOBwkf7FMUiajQaPvnkE7p06YK/vz9QmN73r3/9i549e9K1a9cSYeXFtWjRguXLl5Ofn4+/vz9eXl4kJCSQkpLCnDlzgMJ4V29v71L7SkSqsHbWdutelRRro9H4QBGB94oeNRjKnlUqaj8yMpLg4GC8vLyIiYkhPj7+gfp6dwRr0fE8PT0JDQ0ttX1ISAgJCQkcPXqUrVu3snjxYoYNG0anTp2Ii4tj9uzZzJkzp8ynvbRp04Z58+YRFxfH0qVLefbZZ6lduzbt27dn6tSp9+xn8YjUcZtl1C1ETVclFxjbt2/PwYMH+fPPP4HCqFBvb29iY2OBwukAHx+fB2rTaDRy6NChUvvr9XpcXV3Jz89n3759JfpQdDHRYDCg0+kqFJkKhbncmZmZnDt3Digc7SYnJ2MwGEhPT6ddu3YEBgai0+nQ6/Wkpqai0WgYNmwYzZs358qVK2W2e/36derWrUv//v3p168fly9fVh4zlpqaChQ+Qefq1asP9NkIYQ3kAqMJNG3alOeee465c+eiUqnw8vJi7NixLF++nO3btysX7B5ErVq1SE5O5p133sHJyYm///3vQGFU4axZs2jQoAEajUYpxmPGjGHlypXs3r0blUrFa6+9hre3N61btyYoKIiOHTuWij0solarCQoKYs2aNeh0OgoKChg0aBCNGzdm6dKl6HQ6AAYPHkzt2rX58ssviY+PR6VS0aRJE5544oky242Pjyc6OhpbW1scHBx48803cXFxYfLkyXz88cfKAwlGjhyJh4fHA30+QoiapVoiUh+F0aNHs27duuruhlkY5yXTIKJmWJ34nwpvO0Qz2CR9+DbpO5O0W1myglEIYZHkAqOFsKRRdVJSEkuXLi3xnp2dHQsWLKimHgkhLI3FFmtLotFoKvwA34eRazTnyyJCmIaFzuA+NKtebi6EEJZCRtZCCItkbb9PSrEWQlgkiUgV93T48GFSUlKquxtCCCsjxfoBHTlypNxiXVAgz0IUoqoYMJrkZa7MfhpEr9ezZMkSMjIyMBgMDB8+nAMHDhAcHAzAiRMn2LVrF9OnT2f06NE8/fTTnDx5EmdnZ1566SXWr19Peno6Y8aMwc/Pj5iYGA4fPozBYCA5OZmhQ4eSn5/P3r17sbOzIyQkBGdnZ1JTU4mKiiIzM5NatWrx+uuvk5WVxdGjR0lISGDr1q0EBQXx2WefKUvE27VrR0xMDB9//DFqtRqdTkdwcLDy9d3KilfV6/WsXr2a5ORkCgoK+Nvf/kaXLl2q+mMXQpgZsy/Wx44dw9XVlZCQEAB0Oh1btmxRcrD37NlDnz59gMIcjbZt2xIYGEhERASbN2/m3XffJSUlhcjISPz8/IDCZL7w8HDy8vKYMmUKL7/8MuHh4axdu5ZffvmFwYMHs3LlSl577TUaN27M+fPnWbVqFe+99x5+fn507tyZbt26KX3U6XTMmzcPKMz7iIuLw9/fn9jYWLp27VpmoQbKjFfdtm0b7dq1Y9KkSWRnZzNr1izat28vedZC3MXabt0z+2Kt0WhYt24d69evp3Pnzmi1WgICAti7dy99+/bl3LlzSvSpWq2mY8eOyn52dnao1Wo0Gg3Xr19X2mzbti2Ojo44Ojri5OSkFHGNRkNSUhJ6vZ6zZ8+yePFiZZ/8/Pxy+1gU1QrQr18/tm/fjr+/P3v27OH111+/57ndHa964sQJfv31V6KjowHIzc0lPT0dT0/PEvtKRKoQ1sXsi7WHhwdhYWHExcWxceNGfH196devH2FhYdjb2/Pkk09ia2sLgK2trRKVamNjo4xoVSpVifnku2NQ797OYDBQu3btCi9kqVWrlvJ3Hx8foqKiSEhIwGAwKFncZSkrXtVoNBIUFHTf4KbiEamBm56vUD+FqEnMeX7ZFMz+AmNGRgb29vYEBAQwdOhQLl26RL169XB1dWXr1q3KFMij5OTkhLu7OwcPHgQKf91KTEwEqFCsakBAAB9//DF9+/Ytd5vy4lV9fX3ZuXOn8ive5cuXH81JCVHDGE30x1yZ/cg6KSmJ9evXKyPlCRMmAIVPh/nzzz9LTQ88Km+99Raff/4527ZtIz8/nx49euDl5UX37t1ZsWIFO3fuZNq0aWXu26tXLzZv3kyPHj3Kbd9gMJQZr/rCCy+wdu1a5Sk6DRo0YObMmY/+BIUQFsViI1KjoqJo1qwZ/fr1q+6ulHLo0CGOHDnClClTquR4gY/LNIioGdb/vq3C2wY0ecokfdh75WeTtFtZZj8NUpZ33nmHpKQkevXqVd1dKWX16tVs2LCB4cOHV3dXhBA1iNlPg5QlLCysurtQrnHjxpV6b9WqVZw9e7bEe4MGDbrnnLYQ4t4sckqgEiyyWFuaonl2U8kw6k3avhDmSO4GEUIIYXZkZC2EsEgyshZCCGF2ZGQthLBIFnrX8UOzqGK9ZcsWHBwcuHPnDlqtlg4dOlTp8Y8ePUpKSgrDhg2r0uMKIUqztmkQiyrWRUaMGFEtx/Xz81NCn4QQoiqZfbHetm0bv/zyC25ubtSpU4fmzZsTGRmpxJROnjyZHj16EB8fT0FBARMnTmTTpk2kpqYydOhQBg4cCMD27ds5ePAgeXl5+Pv78+KLL5KWlsbChQtp3bo1586do169esyYMQN7e/sys6ZjYmK4ePEi48eP5/r16yxfvlyJap00aRJubm5ERkbi6OjIpUuXuHXrFoGBgSXiVIu7efMmH330ETqdDoPBwIQJE9BqtRw/fpwtW7aQn59Pw4YNmTRpkkSkCnEXc87xMAWzLtaXLl3iwIEDhIeHU1BQwDvvvEPz5s1Lbefm5kZoaChr167l008/Zf78+eTl5TFt2jQGDhzI8ePHuXbtGgsWLMBoNBIeHk5CQgJubm5cu3aNt99+mzfeeIPFixdz6NAhAgICysyaLi4qKoqAgAD69OnD7t27Wb16NTNmzADg1q1bvP/++1y9epWwsLByi/X+/fvx9fXl+eefx2AwkJOTQ2ZmJtu2bWPOnDk4ODjw9ddf8+233/LCCy+U2FciUoWwLmZdrE+fPo2/v78SQVreFETxPGq9Xq9kVRcV2uPHj3PixAmlmOr1elJTU3Fzc8Pd3R0vLy8AmjdvruRel5U1Xdz58+eVsKWAgAA2bNig/LcuXbqgUqnw9PTk9u3b5Z5fixYtWL58Ofn5+fj7++Pl5UVCQgIpKSnMmTMHKMzR9vb2LrVv8YjUQRsHlf8hClFDyQVGM1OUT30vxfOo786qLsqxHjZsGAMGDCixX1paWqntc3NzgbKzpiuqeJv3+gfVpk0b5s2bR1xcHEuXLuXZZ5+ldu3atG/fnqlTp1b4eEKIms+s77PWarUcPnyY3Nxc7ty5w6+//vpQ7fj6+rJnzx70+sJl2RkZGfcc8ZaXNV2ct7c3sbGxQOF0ho+PzwP36/r169StW5f+/fvTr18/Ll++rDzPMTU1FSh8VNnVq1cfuG0hajp5YK4Zad68Od27dyc4OJgGDRo8VEGEwmJ95coVZs+eDYCDgwNTpkxBpSr7Z1V5WdPFjR07luXLl7N9+3blAuODio+PJzo6GltbWxwcHHjzzTdxcXFh8uTJfPzxx+Tl5QEwcuTI+z45RghrY23TIBabZy3+Z5BG5qxFzbAjaUeFt32iUfkP96iM31IPmKTdyjLrkbUQQpTHnKcsTEGKdRVISkpi6dKlJd6zs7NjwYIFj6T9AvnlSIgaT4p1FdBoNBV+UroQomJkUYwQQlgAg5X9RmnWt+4JIYQoJCNrIYRFsrZpELMfWaelpbF//37l65iYGKKioqqxR0IIUfXMvlhfv369RLEWQggonLM2xctcPfQ0iF6vZ8mSJWRkZGAwGBg+fDgbNmy4b1yp0Whk/fr1HDt2DIDhw4fTvXv3ct/fuHEjKSkpBAcH07t3b5ydnbl58yahoaH88ccf+Pv7ExgYCMDo0aMZNGgQcXFx2NvbExwczGOPPUZmZiYrV67kxo0bALz66qv4+PiQkJDAmjVrgMIMknnz5qHX68uMLS3L6NGjefrppzl58iTOzs689NJLrF+/nvT0dMaMGYOfnx8Gg4ENGzaQkJBAXl4eTz/9NAMGDECv1xMeHk52djb5+fmMHDmSLl263DO2VQjxP9Y2DfLQxfrYsWO4uroSEhICgE6nY8OGDfeNK/3vf/9LYmIiERERZGZmEhISglar5ezZs2W+P2rUKKKjo5k5cyZQOA2SmJhIeHg4arWaqVOn8swzz+Dm5kZOTg6tWrVSiubPP//M8OHDWbNmDUOGDMHHx4f09HRCQ0NZsmQJ27dvZ/z48fj4+KDX67Gzs+Onn34qFVtanpycHNq2bUtgYCARERFs3ryZd999l5SUFCIjI/Hz82P37t04OTmxcOFC8vLymDNnDr6+vtSvX5/p06fj5OREZmYms2fPVtIDy4ttLU4iUoWwLg9drDUaDevWrWP9+vV07txZGX3eL670zJkz9OjRA5VKxWOPPUabNm24ePFiue87OjqWOna7du1wcnICwNPTk/T0dNzc3FCr1XTu3BkozBU5ceIEACdPniQlJUXZX6fTcefOHXx8fPjXv/5Fz5496dq1K/Xr1y8ztrTcD0+tpmPHjsr52tnZoVar0Wg0StTq8ePHSUpK4tChQ8qxr127Rr169di0aROnT5/GxsamRLhUebGtxRWPSH16w18q8B0TomYx5ykLU3joYu3h4UFYWBhxcXFs3LgRX1/fwgYrEFdaWeW1a2trq0SqFn/faDQSGhpaaiph2LBhdOrUibi4OGbPns2cOXPKjC3t3bt3mf0ofjwbG5sS51782GPHjlWKepGYmBgyMzNZtGgRarWayZMnK/Gs5cW2CiGs10NfYMzIyMDe3p6AgACGDh3KpUuXKrSfVqvl4MGDGAwGMjMzOX36NC1btiz3fUdHR+7cufOw3QSgQ4cOfP/998rXiYmJAKSmpqLRaBg2bBjNmzfnypUrZcaWVkbHjh3ZtWsX+fn5AFy9ehW9Xo9Op6Nu3bqo1WpOnTpV5uhZCFE+o4n+mKuHHlknJSWxfv16ZUQ5YcKECgX0+/v7c+7cOYKDgwEIDAzkscceK/d9Z2dnbG1tS1xgfFBjx44lKiqK6dOnU1BQgFarZeLEiezYsYP4+HhUKhVNmjThiSee4MCBA6ViSyujX79+pKWl8c477wDg4uJCcHAwPXv2JCwsjJkzZ+Ll5UWTJk0qdRwhRM0mEak1wNNNZc5a1Aw/JO+s8LYt3DqZpA8X0+NM0m5lyQpGIYRFMucpC1OQYl0Bs2bNUp7aUmTKlCloNJpq6lFJBRiquwtCCBOTYl0Bjyp3Wgjx6BiN1jVIMfvl5kIIIWRkLYSwUPJYLyGEsADWdiNbjZ8GkYhVIURNUOOLtUSsClEzGTCa5GWuqm0axNojVg0GA8uXL1eW6fft25chQ4aQmppKVFQUmZmZ1KpVi9dff11WNwohqq9YW3vEamJiIhkZGfzzn/8EIDs7G4CVK1fy2muv0bhxY86fP8+qVat47733Su0vEanC2lnbnHW1FWtrj1h1d3cnLS2N1atX06lTJzp06IBer+fs2bMlMlaKAqDuVjwitf+Gpx/koxeiRpCI1Cpi7RGrzs7OREREcOzYMb7//ntiY2MZM2YMtWvXJiIi4pGcpxCi5qi2C4zWHrGamZmJwWCgW7dujBw5ksuXL+Pk5IS7uzsHDx4ECn9IFB1LCFGSRKRWEWuPWM3IyGD58uUYDIVLZkeNGgXAW2+9xeeff862bdvIz8+nR48e93xajRDCOkhEag3Qv6nMWYua4afkHyq8bcO6Pibpwx+3z5ik3cqq8fdZCyFETSDLzauAqSNWcwxl3zEiRE1mzgtYTEGKdRWQiFUhHj1rm8GVaRAhhLAAMrIWQlgka1sUIyNrIYSwABZTrHft2sUvv/zySNratm3bI2nH1BITE4mLM88nLQtR3YxGo0le5soiinVBQQEDBw4sc9n2w/jqq68eeJ+ixSumUN4y+sTERH777TeTHVcISyYRqSaSlpbGggULaNmyJYmJiTRu3Jg333yTK1eu8MUXX6DX63FxcWHSpEm4uroyd+5cvL29OXv2LH5+fty5cwcHBweeffZZ5s6di5eXF5cvXyYzM5PJkyfz9ddfk5SURPfu3Rk5ciQAe/fuZefOneTn59OqVSsmTJjApk2byM3NJTg4mKZNm/LWW2+VuZ1KpWL06NEMGTKE48eP88orr+DjU/Im/AsXLvD1118zffp0jhw5wkcffcQXX3yBwWBg2rRpLFu2jMTERD7//HNycnJo2LAh//d//4ezs3Op83Nzc+M///kPKpUKJycn5syZw5dffklubi5nzpzhueeeo3v37lX17RJCmJkqvcB49epV3njjDXx8fPj000/54YcfOHz4MDNmzMDFxYXY2Fg2bdrEpEmTgMJ0u3nz5gGwZcuWkh1Xq5k3bx47duwgIiKCRYsW4ezszJQpUxg8eDC3b98mNjaW+fPno1arWbVqFfv27ePll1/m+++/V8KSUlJSytyud+/e5OTk0LRpU0aMGFHm+TRr1kzJ/jh9+jQajYYLFy5gMBho2bIlAMuWLWPcuHG0adOGL7/8kv/85z+MGTOm1PkFBQUxe/Zs6tWrR3Z2Nmq1mhEjRnDx4kXGjx9f6tgSkSqsnTlPWZhClRbr+vXrK6PTgIAAvvrqK5KTk5k/fz5QONXg6uqqbH+vkWTxKFVPT09lv4YNG3Ljxg3OnDnD5cuXlbzs3NxcXFxcSrVz6tSpcrdTqVR069at3D7Y2trSqFEjUlJSuHjxIoMHD+b06dMYDAa0Wi06nY7s7GzatGkDQO/evVmyZEmZ59e6dWsiIyN58skn6dq1a7nHLFI8IrXXuqfuu70QwrJVabEuih8t4uDggKenJ6GhoWVuX6tWrXLbKoo5tbGxKRF5amNjQ0FBAUajkd69eysBSeW513Z2dnaoVPee1tdqtRw7dgxbW1s6dOhAZGQkBoOB0aNH33M/KHl+EydO5Pz588TFxTFjxgzCw8Pvu78Q1swcb907ePAg//73v7ly5QoLFiygRYsWZW43efJkHBwcUKlU2NrasmjRovu2XaUXGNPT0zl37hwA+/fvp1WrVmRmZirv5efnk5yc/EiO1b59ew4dOsTt27cByMrK4vr160DhFEpRqP+9tqsIrVbLd999h7e3Ny4uLmRlZXH16lWaNm2Kk5MTzs7OnD59GiicQy/rEV9QGLfaqlUrRowYQZ06dbhx4wYODg6VjncVoqYyx4jUpk2bMn369HL/Py/uvffeU6ZwK6JKR9ZNmjQhJiaGlStX0qhRI/7yl7/QsWNH1qxZg06no6CggEGDBtG0adNKH8vT05ORI0fywQcfYDQasbW1Zfz48TRo0ICnnnqK4OBgmjVrxltvvVXudhXRqlUrbt++rXxzNBoNLi4uym8RkydPVi4wuru7K/Pxd1u/fj3Xrl0DCp9k8/jjj+Pm5sY333xDcHCwXG3V5TMAAAWrSURBVGAUwgJ4enqarO0qi0hNS0sjLCxMeeageHR6NZE5a1Ez7Lvyc4W3dXR83CR9uHPn90q3MXfuXEaPHn3PaZCibP0BAwYo15/uRZabCyFEMXffaVX8Yj7A/PnzuXXrVqn9Ro4cSZcuXSp0jPnz51OvXj1u377NBx98gIeHh3IjQnnk4QMVFBERQVpaWon3Xn75ZTp27FhNPfoftX2T6u6CEI9Efu6VCm/r4PBoIobvptcnVbqN+42si9uyZYuyhuReZGRdQUWPCxNCiMrQ6/UYjUYcHR3R6/WcOHGCF1544b77yci6BpCRtagpHmRkXcuh8jcilCVH//B3pB0+fJjVq1eTmZlJ7dq18fLyYvbs2WRkZLBixQpCQkL4448/+PDDD4HCqImePXvy/PPP37ftGluss7Oz2b9/P08//eDPJ0xLS+PcuXP07NnTBD179KRYi5riQYq1fS3T3HmRm5NiknYryyKCnB5GdnY2u3bteqh9r1+/zv79+x94P1OGPQkhrFuNHVl/9NFHHDlyBA8PDzp06EDdunU5ePAgeXl5+Pv78+KLL3LhwgU+++wzFixYgMFgYNasWUydOpUVK1aQkpKCu7s7vXv3xtnZuURGx6JFixg6dCht27YtFfZkb29fZjBVWXbs2MGPP/6Ira0tnp6eTJ06Fb1ez+rVq0lOTqagoIC//e1v973CLCNrUVM8yMjazkT/7vMeoA9VqcZeYBw1ahTJyclERERw/PhxDh06xIIFCzAajYSHh5OQkECbNm3w8/Nj8+bN5Obm0qtXLzQaDaNGjSI6OpqZM2cCEBMTU+5xioc95efnM3fu3HKDqe72zTffsGzZMuzs7MjOzgYKs7bbtWvHpEmTyM7OZtasWbRv3x4HB4dH/hkJISxHjS3WxR0/fpwTJ04wY8YMoPBqbGpqKm3atOGFF14gJCQEOzs7xo0b98BtFw97unr16j2Dqe6m0Wj45JNP6NKlC/7+/gCcOHGCX3/9lejoaKAwWCo9Pd2kK6OEsEQ1ckrgHqyiWAMMGzaMAQMGlHo/KysLvV5Pfn4+ubm5ZY5gVSpViTjGvLw85e93hz3dK5jqbiEhISQkJHD06FG2bt3K4sWLMRqNBAUF4eHhcc99i9+4//2OLyq0AkpUzk8//SSfsxl5kCmTmqDGXmB0dHRUQpB8fX3Zs2cPer0egIyMDCW4acWKFYwYMYJevXqxYcOGUvsCuLu7k5iYiMFgID09nQsXLpR5TA8PjwoHUxW11a5dOwIDA9HpdOj1enx9fdm5c6fyw6EoL/tu/fv3Z9GiRSxatEgKSBWR/HBRnWrsyLpOnTq0bt2aoKAgOnbsSM+ePZk9ezZQGM06ZcoUJdq0Z8+eGAwG3n33XU6dOoWPjw+2trYEBwfTu3dvBg8ejLu7O9OnT6dp06Y0a9aszGOq1WqCgoIqFExlMBhYunQpOp0OgMGDB1O7dm1eeOEF1q5dy/Tp0wFo0KCBMncuhLBeNfZuECEetZkzZ1Y4zlKIR63GToMI8ajJdJOoTjKyrgKrVq3i7NmzJd4bNGgQffv2raYeCSEsjRRrIYSwADX2AqMQj8qxY8dYs2YNBoOBp556imHDhlV3l4QVkjlrIe7BYDAQFRXFrFmzWLJkCQcOHCAlxTyDfkTNJsVaiHu4cOECjRo1omHDhqjVarp3786RI0equ1vCCkmxFuIeMjIyqF+/vvJ1/fr1ycjIqMYeCWslxVqIeyjr+nvRk+uFqEpSrIW4h/r163Pjxg3l6xs3btwznEsIU5FiLcQ9tGjRgmvXrpGWlkZ+fj6xsbH4+flVd7eEFZL7rIW4j7i4OL744gsMBgN9+/at0PPyhHjUpFgLIYQFkGkQIYSwAFKshRDCAkixFkIICyDFWgghLIAUayGEsABSrIUQwgJIsRZCCAsgxVoIISzA/wPUVikkIt9f3AAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize = (4,8))\n", "sns.heatmap(param_df.T.sort_values(0,ascending = False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Assuming that weights with higher magnitudes equate to a higher level of importance, we can see that \n", "\n", "1. area_worst\n", "2. area_se and\n", "3. points_worst\n", "\n", "were the top 3 variables that helped us differentiate between Malign and Benign cells." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n", "\n", "Logistic Regression is a poweful method to analyze the relationship between quantitative and binary qualitative variables that uses the techniques of DL to present a meaninful relationship.\n", "\n", "Given its \"shallow\" architecture in comparison with alternative DL architectures, it does not necessitate much data to learn relationships.\n", "\n", "As such, Logistic Regression is an important concept to have on anyone's Data Science arsenal" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }