{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Networks\n", "> The previous chapters taught you how to build models in TensorFlow 2.0. In this chapter, you will apply those same tools to build, train, and make predictions with neural networks. You will learn how to define dense layers, apply activation functions, select an optimizer, and apply regularization to reduce overfitting. You will take advantage of TensorFlow's flexibility by using both low-level linear algebra and high-level Keras API operations to define and train models. This is the Summary of lecture \"Introduction to TensorFlow in Python\", via datacamp.\n", "\n", "- toc: true \n", "- badges: true\n", "- comments: true\n", "- author: Chanseok Kang\n", "- categories: [Python, Datacamp, Tensorflow-Keras, Deep_Learning]\n", "- image: images/10_7_3_1_network.png" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'2.2.0'" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import tensorflow as tf\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "plt.rcParams['figure.figsize'] = (8, 8)\n", "\n", "tf.__version__" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dense layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The linear algebra of dense layers\n", "There are two ways to define a dense layer in tensorflow. The first involves the use of low-level, linear algebraic operations. The second makes use of high-level keras operations. In this exercise, we will use the first method to construct the network shown in the image below.\n", "\n", "\n", "\n", "The input layer contains 3 features -- education, marital status, and age -- which are available as `borrower_features`. The hidden layer contains 2 nodes and the output layer contains a single node.\n", "\n", "For each layer, you will take the previous layer as an input, initialize a set of weights, compute the product of the inputs and weights, and then apply an activation function." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "borrower_features = np.array([[2., 2., 43.]], np.float32)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dense1's output shape: (1, 2)\n" ] } ], "source": [ "# Initialize bias1\n", "bias1 = tf.Variable(1.0, tf.float32)\n", "\n", "# Initialize weights1 as 3x2 variable of ones\n", "weights1 = tf.Variable(tf.ones((3, 2)))\n", "\n", "# Perform matrix multiplication of borrower_features and weights1\n", "product1 = tf.matmul(borrower_features, weights1)\n", "\n", "# Apply sigmoid activation function to product1 + bias1\n", "dense1 = tf.keras.activations.sigmoid(product1 + bias1)\n", "\n", "# Print shape of dense1\n", "print(\"dense1's output shape: {}\".format(dense1.shape))" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "prediction: 0.9525741338729858\n", "\n", " actual: 1\n" ] } ], "source": [ "# Initialize bias2 and weights2\n", "bias2 = tf.Variable(1.0)\n", "weights2 = tf.Variable(tf.ones((2, 1)))\n", "\n", "# Perform matrix multiplication of dense1 and weights2\n", "product2 = tf.matmul(dense1, weights2)\n", "\n", "# Apply activation to product2 + bias2 and print the prediction\n", "prediction = tf.keras.activations.sigmoid(product2 + bias2)\n", "print('prediction: {}'.format(prediction.numpy()[0, 0]))\n", "print('\\n actual: 1')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our model produces predicted values in the interval between 0 and 1. For the example we considered, the actual value was 1 and the predicted value was a probability between 0 and 1. This, of course, is not meaningful, since we have not yet trained our model's parameters." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The low-level approach with multiple examples\n", "In this exercise, we'll build further intuition for the low-level approach by constructing the first dense hidden layer for the case where we have multiple examples. We'll assume the model is trained and the first layer weights, `weights1`, and bias, `bias1`, are available. We'll then perform matrix multiplication of the `borrower_features` tensor by the `weights1` variable. Recall that the `borrower_features` tensor includes education, marital status, and age. Finally, we'll apply the sigmoid function to the elements of `products1 + bias1`, yielding `dense1`.\n", "\n", "$$ \\text{products1} = \\begin{bmatrix} 3 & 3 & 23 \\\\ 2 & 1 & 24 \\\\ 1 & 1 & 49 \\\\ 1 & 1 & 49 \\\\ 2 & 1 & 29 \\end{bmatrix} \\begin{bmatrix} -0.6 & 0.6 \\\\ 0.8 & -0.3 \\\\ -0.09 & -0.08 \\end{bmatrix} $$" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " shape of borrower_features: (1, 3)\n", "\n", " shape of weights1: (3, 2)\n", "\n", " shape of bias1: (1,)\n", "\n", " shape of dense1: (1, 2)\n" ] } ], "source": [ "bias1 = tf.Variable([0.1], tf.float32)\n", "\n", "# Compute the product of borrower_features and weights1\n", "products1 = tf.matmul(borrower_features, weights1)\n", "\n", "# Apply a sigmoid activation function to products1 + bias1\n", "dense1 = tf.keras.activations.sigmoid(products1 + bias1)\n", "\n", "# Print the shapes of borrower_features, weights1, bias1, and dense1\n", "print('\\n shape of borrower_features: ', borrower_features.shape)\n", "print('\\n shape of weights1: ', weights1.shape)\n", "print('\\n shape of bias1: ', bias1.shape)\n", "print('\\n shape of dense1: ', dense1.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that our input data, `borrower_features`, is 5x3 because it consists of 5 examples for 3 features. The shape of `weights1` is 3x2, as it was in the previous exercise, since it does not depend on the number of examples. Additionally, `bias1` is a scalar. Finally, `dense1` is 5x2, which means that we can multiply it by the following set of weights, `weights2`, which we defined to be 2x1 in the previous exercise." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using the dense layer operation\n", "We've now seen how to define dense layers in tensorflow using linear algebra. In this exercise, we'll skip the linear algebra and let keras work out the details. This will allow us to construct the network below, which has 2 hidden layers and 10 features, using less code than we needed for the network with 1 hidden layer and 3 features.\n", "\n", "\n", "\n", "To construct this network, we'll need to define three dense layers, each of which takes the previous layer as an input, multiplies it by weights, and applies an activation function." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | ID | \n", "LIMIT_BAL | \n", "SEX | \n", "EDUCATION | \n", "MARRIAGE | \n", "AGE | \n", "PAY_0 | \n", "PAY_2 | \n", "PAY_3 | \n", "PAY_4 | \n", "... | \n", "BILL_AMT4 | \n", "BILL_AMT5 | \n", "BILL_AMT6 | \n", "PAY_AMT1 | \n", "PAY_AMT2 | \n", "PAY_AMT3 | \n", "PAY_AMT4 | \n", "PAY_AMT5 | \n", "PAY_AMT6 | \n", "default.payment.next.month | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "1 | \n", "20000.0 | \n", "2 | \n", "2 | \n", "1 | \n", "24 | \n", "2 | \n", "2 | \n", "-1 | \n", "-1 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "689.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "1 | \n", "
1 | \n", "2 | \n", "120000.0 | \n", "2 | \n", "2 | \n", "2 | \n", "26 | \n", "-1 | \n", "2 | \n", "0 | \n", "0 | \n", "... | \n", "3272.0 | \n", "3455.0 | \n", "3261.0 | \n", "0.0 | \n", "1000.0 | \n", "1000.0 | \n", "1000.0 | \n", "0.0 | \n", "2000.0 | \n", "1 | \n", "
2 | \n", "3 | \n", "90000.0 | \n", "2 | \n", "2 | \n", "2 | \n", "34 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "14331.0 | \n", "14948.0 | \n", "15549.0 | \n", "1518.0 | \n", "1500.0 | \n", "1000.0 | \n", "1000.0 | \n", "1000.0 | \n", "5000.0 | \n", "0 | \n", "
3 | \n", "4 | \n", "50000.0 | \n", "2 | \n", "2 | \n", "1 | \n", "37 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "... | \n", "28314.0 | \n", "28959.0 | \n", "29547.0 | \n", "2000.0 | \n", "2019.0 | \n", "1200.0 | \n", "1100.0 | \n", "1069.0 | \n", "1000.0 | \n", "0 | \n", "
4 | \n", "5 | \n", "50000.0 | \n", "1 | \n", "2 | \n", "1 | \n", "57 | \n", "-1 | \n", "0 | \n", "-1 | \n", "0 | \n", "... | \n", "20940.0 | \n", "19146.0 | \n", "19131.0 | \n", "2000.0 | \n", "36681.0 | \n", "10000.0 | \n", "9000.0 | \n", "689.0 | \n", "679.0 | \n", "0 | \n", "
5 rows × 25 columns
\n", "