{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Network Initializer\n",
"\n",
"### What is neuron?\n",
"\n",
"Feed-forward neural networks are inspired by the information processing of one or more neural cells, called a neuron. A neuron accepts input signals via its dendrites, which pass the electrical signal down to the cell body. The axon carries the signal out to synapses, which are the connections of a cell’s axon to other cell’s dendrites."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from random import random, seed\n",
"\n",
"def initialize_network(n_inputs, n_hidden, n_outputs):\n",
" network = list()\n",
" # Creating hidden layers according to the number of inputs\n",
" hidden_layer = [{'weights': [random() for i in range(n_inputs + 1)]} for i in range(n_hidden)]\n",
" network.append(hidden_layer)\n",
" # Creating output layer according to the number of hidden layers\n",
" output_layer = [{'weights': [random() for i in range(n_hidden + 1)]} for i in range(n_outputs)]\n",
" network.append(output_layer)\n",
" return network"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# It is good practice to initialize the network weights to small random numbers. \n",
"# In this case, will we use random numbers in the range of 0 to 1.\n",
"# To achieve that we seed random with 1\n",
"seed(1)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[{'weights': [0.7887233511355132, 0.0938595867742349, 0.02834747652200631]}]\n",
"[{'weights': [0.8357651039198697, 0.43276706790505337]}, {'weights': [0.762280082457942, 0.0021060533511106927]}]\n"
]
}
],
"source": [
"# 2 input units, 1 hidden unit and 2 output units\n",
"network = initialize_network(2, 1, 2)\n",
"# You can see the hidden layer has one neuron with 2 input weights plus the bias.\n",
"# The output layer has 2 neurons, each with 1 weight plus the bias.\n",
"\n",
"for layer in network:\n",
" print(layer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Forward propagate\n",
"\n",
"We can calculate an output from a neural network by propagating an input signal through each layer until the output layer outputs its values.\n",
"\n",
"We can break forward propagation down into three parts:\n",
"\n",
"\n",
"1. Neuron Activation.\n",
"\n",
"2. Neuron Transfer.\n",
"\n",
"3. Forward Propagation.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Neuron Activation\n",
"\n",
"The first step is to calculate the activation of one neuron given an input.\n",
"\n",
"Neuron activation is calculated as the weighted sum of the inputs. Much like linear regression.\n",
"\n",
"\n",
"activation = sum(weight_i * input_i) + bias\n",
"\n",
"\n",
"Where weight is a network weight, input is an input, i is the index of a weight or an input and bias is a special weight that has no input to multiply with (or you can think of the input as always being 1.0).\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Implementation\n",
"def activate(weights, inputs):\n",
" activation = weights[-1]\n",
" for i in range(len(weights) - 1):\n",
" activation += weights[i] * inputs[i]\n",
" return activation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Neuron Transfer\n",
"\n",
"Once a neuron is activated, we need to transfer the activation to see what the neuron output actually is.\n",
"\n",
"Different transfer functions can be used. It is traditional to use the *sigmoid activation function*, but you can also use the *tanh* (hyperbolic tangent) function to transfer outputs. More recently, the *rectifier transfer function* has been popular with large deep learning networks.\n",
"\n",
"\n",
"Sigmoid formula\n",
"\n",
"output = 1 / (1 + e^(-activation))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from math import exp\n",
"\n",
"def transfer(activation):\n",
" return 1.0 / (1.0 + exp(-activation))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3. Forawrd propagate"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Foward propagate is self-explanatory\n",
"def forward_propagate(network, row):\n",
" inputs = row\n",
" for layer in network:\n",
" new_inputs = []\n",
" for neuron in layer:\n",
" activation = activate(neuron['weights'], inputs)\n",
" neuron['output'] = transfer(activation)\n",
" new_inputs.append(neuron['output'])\n",
" inputs = new_inputs\n",
" return inputs"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"inputs = [1, 0, None]\n",
"output = forward_propagate(network, inputs)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0.7335023968859138, 0.6296776889933221]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Running the example propagates the input pattern [1, 0] and produces an output value that is printed.\n",
"# Because the output layer has two neurons, we get a list of two numbers as output.\n",
"output"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Backpropagation\n",
"\n",
"### What is it?\n",
"\n",
"1. Error is calculated between the expected outputs and the outputs forward propagated from the network.\n",
"\n",
"2. These errors are then propagated backward through the network from the output layer to the hidden layer, assigning blame for the error and updating weights as they go.\n",
"\n",
"\n",
"### This part is broken down into two sections.\n",
"\n",
"- Transfer Derivative\n",
"- Error Backpropagation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Transfer Derivative\n",
"\n",
"Given an output value from a neuron, we need to calculate it’s *slope*.\n",
"\n",
"derivative = output * (1.0 - output)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Calulates the derivation from an neuron output\n",
"def transfer_derivative(output):\n",
" return output * (1.0 - output)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Error Backpropagation\n",
"\n",
"1. calculate the error for each output neuron, this will give us our error signal (input) to propagate backwards through the network.\n",
"\n",
"error = (expected - output) * transfer_derivative(output)\n",
"\n",
"\n",
"expected: expected output value for the neuron\n",
"\n",
"output: output value for the neuron and transfer_derivative()\n",
"\n",
"----\n",
"\n",
"The back-propagated error signal is accumulated and then used to determine the error for the neuron in the hidden layer, as follows:\n",
"\n",
"\n",
"error = (weight_k * error_j) * transfer_derivative(output)\n",
"\n",
"error_j: the error signal from the jth neuron in the output layer\n",
"\n",
"weight_k: the weight that connects the kth neuron to the current neuron and output is the output for the current neuron"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def backward_propagate_error(network, expected):\n",
" for i in reversed(range(len(network))):\n",
" layer = network[i]\n",
" errors = list()\n",
" if i != len(network) - 1:\n",
" for j in range(len(layer)):\n",
" error = 0.0\n",
" for neuron in network[i + 1]:\n",
" error += (neuron['weights'][j] * neuron['delta'])\n",
" errors.append(error)\n",
" else:\n",
" for j in range(len(layer)):\n",
" neuron = layer[j]\n",
" errors.append(expected[j] - neuron['output'])\n",
" for j in range(len(layer)):\n",
" neuron = layer[j]\n",
" neuron['delta'] = errors[j] * transfer_derivative(neuron['output'])"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[{'weights': [0.7887233511355132, 0.0938595867742349, 0.02834747652200631], 'output': 0.6936142046010635, 'delta': -0.011477619712406795}]\n",
"[{'weights': [0.8357651039198697, 0.43276706790505337], 'output': 0.7335023968859138, 'delta': -0.1433825771158816}, {'weights': [0.762280082457942, 0.0021060533511106927], 'output': 0.6296776889933221, 'delta': 0.08635312555373359}]\n"
]
}
],
"source": [
"expected = [0, 1]\n",
"backward_propagate_error(network, expected)\n",
"# delta: error value\n",
"for layer in network:\n",
" print(layer)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Train Network\n",
"\n",
"Two parts\n",
"\n",
"- Update Weights\n",
"\n",
"- Train Network"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Update weights\n",
"\n",
"Once errors are calculated for each neuron in the network via the back propagation method above, they can be used to update weights.\n",
"\n",
"weight = weight + learning_rate * error * input\n",
"\n",
"weight = given weight\n",
"\n",
"The learning_rate parameter is explicitly specified by the programmer. This controls how much the weights will be updated at each step.",
"\n",
"A learning rate that is very large may sound appealing, as the weights will be more dramatically updated, which could lead to faster learning.",
"\n",
"However, this causes the learning process to become very unstable. Ideally, we want a learning rate that is steady and reliable, but will find a solution in a reasonable amount of time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}