{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# How I Implemented The Most Simple Neural Network Using Python\n",
"> \"Training a neural network to output '42' when given '5'\"\n",
"\n",
"- toc: true \n",
"- badges: true\n",
"- comments: true\n",
"- categories: [Grokking Deep Learning, Deep Learning, Python]\n",
"- image: images/SimpleNN.png"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Context\n",
"\n",
"I've been reading the book [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning) by [Andrew W. Trask](https://twitter.com/iamtrask) and instead of summarizing concepts, I want to review them by building a simple neural network. This neural network will use the concepts in the first 4 chapters of the book."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# What I'm Building\n",
"\n",
"I'm going to build a neural network which outputs a target number given a specific input number. For example, given the number `5`, I want the neural network to output the number `42`.\n",
"\n",
"Now I can hear you think to yourself, \"That's stupid. How is that better than a function with the line `return 42` in the body?\"\n",
"\n",
"What's cool about this code is that I didn't type the number `5` or `42` anywhere in the body of the network. Instead, I told the network I wanted it to print `42` when it received `5` as an input and it figure out how to adjust itself to do that. \n",
"\n",
"In fact, I could train the network on any 2 numbers using the same code. Try changing the parameters yourself and test it out!\n",
"\n",
"With that context, let's see what the code looks like for this most simple neural network."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The Code"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# A simple neural network class\n",
"class SimpleNN:\n",
" def __init__(self):\n",
" self.weight = 1.0\n",
" self.alpha = 0.01\n",
" \n",
" def train(self, input, goal, epochs):\n",
" for i in range(epochs):\n",
" pred = input * self.weight\n",
" delta = pred - goal\n",
" error = delta ** 2\n",
" derivative = delta * input\n",
" self.weight = self.weight - (self.alpha * derivative)\n",
" print(\"Error: \" + str(error))\n",
" \n",
" def predict(self, input):\n",
" return input * self.weight"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Error: 1369.0\n",
"Error: 770.0625\n",
"Error: 433.16015625\n",
"Error: 243.6525878906251\n",
"Error: 137.05458068847665\n",
"Error: 77.09320163726807\n",
"Error: 43.36492592096329\n",
"Error: 24.39277083054185\n",
"Error: 13.72093359217979\n",
"Error: 7.718025145601132\n",
"Error: 4.341389144400637\n",
"Error: 2.442031393725358\n",
"Error: 1.373642658970514\n",
"Error: 0.7726739956709141\n",
"Error: 0.43462912256489855\n",
"Error: 0.24447888144275018\n",
"Error: 0.13751937081154697\n",
"Error: 0.07735464608149517\n",
"Error: 0.043511988420844\n",
"Error: 0.02447549348672308\n"
]
}
],
"source": [
"# Create a new SimpleNN\n",
"neural_network = SimpleNN()\n",
"# Train the SimpleNN \n",
"neural_network.train(input=5, goal=42, epochs=20)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"41.88266515825944"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"neural_network.predict(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After 20 rounds of training, the network's final prediction is off by about `0.02`. Not bad!\n",
"\n",
"Even in this barebones neural network, there's a lot going on. Let's take it line by line."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Neural Networks\n",
"\n",
"A neural network is a collection of weights being used to compute an error function. That's it. \n",
"\n",
"The interesting thing about this statement is that for any error function, no matter how complicated, you can compute the relationship between a weight and the final error of the network. Therefore, after each prediction, we can change each weight in the network to inch the final error towards 0.\n",
"\n",
"Let's take a look what a neural network needs to make a prediction.\n",
"\n",
"### The 2 Things A Neural Network Needs To Make A Prediction\n",
"\n",
"#### The Weight\n",
"\n",
"```python\n",
"self.weight = 1.0\n",
"```\n",
"\n",
"I mentioned before that a neural network is just \"a collection of weights\". So what are weights?\n",
"\n",
"`weight` is a number that the neural network stores and remembers. It can be thought of of the _\"memory\"_ of the network. After each round of training, the network updates the `weight` to make more accurate predictions.\n",
"\n",
"In our network, I set `weight=1.0`. I just used trial-and-error to figure out a good starting weight for this problem.\n",
"\n",
"#### The Input\n",
"\n",
"```python\n",
"def train(self, input, goal, epochs):\n",
"\n",
"def predict(self, input):\n",
"```\n",
"`input` is a number that the neural network accepts. This can be thought of as information from the outside world.\n",
"\n",
"In our network, I set `input=5` when I start training the network. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# So how does this thing learn?\n",
"\n",
"I use a method called **Stochasitc Gradient Descent** to get `SimpleNN` to learn the training data. \n",
"\n",
"At a high level, the 4 step process is:\n",
"\n",
"1. Make a prediction using a given input\n",
"1. Calculate the error\n",
"1. Calculate the derivative to tell us how much to adjust the weights by\n",
"1. Adjust the weight and go back to step 1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. The Prediction\n",
"\n",
"```python\n",
"pred = input * self.weight\n",
"```\n",
"\n",
"When the neural network has both an `input` and `weight`, it multiplies them together to make a prediction. Every single neural network, from the most simple to ones with 1000s of layers works this way.\n",
"\n",
"## 2. How much are we off by?\n",
"\n",
"```python\n",
"delta = pred - goal\n",
"error = delta ** 2\n",
"```\n",
"\n",
"So we've seen that the network make a prediction by multiplying `input` and `weight`. After it makes a prediction, the network is able to calculate how much it was off by.\n",
"\n",
"A neural network learning is all about error attribution. How much did each weight contribute to the overall error of the system and how can we change the weight so that error is minimized? In our example, it's easy to figure out since there is only 1 weight.\n",
"\n",
"How do we calculate the error? One thing we need to keep in mind is that we want the error to be a positive number. If the error is allowed to be negative, multiple errors might accidentally cancel each other out when averaged together.\n",
"\n",
"In our case, we square the amount we are off by. Why square instead of something straightforward like absolute value? Squaring gives us a sense of importance. Large errors are magnified while small errors are minimized. Therefore, we can prioritize large errors before small errors. Absolute value doesn't give us this additional sense of importance. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Adjusting the weights\n",
"\n",
"```python\n",
"derivative = delta * input\n",
"self.weight = self.weight - (self.alpha * derivative)\n",
"```\n",
"\n",
"The network figures out how much to adjust the weights by using a derivative. How does derivative play into this process? What a derivative tells us is the direction and amount one variable changes when you change a different variable. In our case, derivatives tell us much much error changes when you change the weight. Given that we want error to be 0, this is exactly what we need. \n",
"\n",
"The network calculates the derivative by multiplying the `delta` by the weight's input to get the `weight_delta`. `weight_delta` is the direction and the amount we're going to change the weight by.\n",
"\n",
"```python\n",
"self.alpha = 0.01\n",
"```\n",
"\n",
"One bit of nuance is the variable `alpha`. `alpha` is a throttle limiting how much we actually adjust the weights. Determining the appropriate rate of change for the weights of a neural network is a challenge. If the steps are too large, the network will overshoot the error getting to zero and start acting in unpredictable ways. If the steps are too small, the network will take a long time and need a very large number of training cycles.\n",
"\n",
"The solution to this problem is to multiply partial derivative by a single number between 0 and 1. This lets us control the rate of change and adjust the learning as needed.\n",
"\n",
"Finding the appropriate `alpha` is often done through trial and error so we're just going to hard code is here."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Training rounds\n",
"\n",
"```python\n",
"neural_network.train(input=5, goal=42, epochs=20)\n",
"\n",
"for i in range(epochs):\n",
"```\n",
"\n",
"Finally, there's the concept of `epochs`. This refers to the number of times the network will go through the entire data set. The appropriate number of epochs for a problem will often be found through trial and error. \n",
"\n",
"I'm using `20` in the example, which I found by running the training with different epochs and picking the lowest one with an acceptable error. Feel free to experiment with the number of epochs and see what happens at different numbers. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# So what did I accomplish?\n",
"\n",
"I'm able to give the neural network the number `5`, and have it output a number very close to our goal number `42` without putting the number `5` or `42` in the body of the function.\n",
"\n",
"I also learned the basic parts which make up all neural networks and we learned the process of how the network learns.\n",
"\n",
"As we start to move into networks with multiple inputs, multiple outputs, and multiple layers, it's going to get a lot more complicated. However, the mental model stays the same. The network makes a prediction by multiplying the received input with its stored weights. It measures the error, takes the derivative, and adjusts the weights so that error moves towards 0. Then it goes again."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# What's next?\n",
"\n",
"I'm going to tackle multiple inputs and multiple outputs. I'll see how matrices come into play and how we can build a simple library to do matrix math.\n",
"\n",
"See you then!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}