{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
""
],
"text/vnd.plotly.v1+html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"import math\n",
"import plotly.offline as py\n",
"import plotly.graph_objs as go\n",
"py.init_notebook_mode(connected=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Perceptron\n",
"Let us implement a perceptron. \n",
"\n",
"\\\\( F(x) = {\\sigma}\\biggl(\\sum_{i=0}^n W_i.x_i + b\\biggr) \\\\)\n",
"\n",
"\n",
" * This can take \\\\( n \\\\) input.\n",
" * Has \\\\( n \\\\) weights.\n",
" * \\\\( \\sigma \\\\) is the sigmoid activation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Let us create a perceptron class\n",
"\n",
"\n",
"h = []\n",
"class Perceptron():\n",
" \"\"\" The implementation of the perceptron model. \"\"\"\n",
" \n",
" def __init__(self,num_inputs,lr):\n",
" \"\"\" \n",
" Arguments\n",
" num_inputs: number of inputs to the perceptron\n",
" lr : Learning rate for the perceptron\n",
" \"\"\" \n",
" \n",
" \n",
" #Here we use num_inputs + 1, because this would take the bias into account if we pad the input with one. \n",
" #So x dot product of W gives us Wx + b\n",
" self.W = np.random.randn(num_inputs+1) \n",
" \n",
" self.lr = lr\n",
" \n",
" def step_function(self,x):\n",
" \"\"\"\n",
" Arguments\n",
" x : the input on which the step_function should be applied\n",
" \"\"\"\n",
" if (x>0):\n",
" return 1\n",
" else :\n",
" return 0 \n",
" \n",
" \n",
" def forward(self,x):\n",
" \"\"\"\n",
" Arguments\n",
" x : the input numpyt array on which the perceptron is trained.\n",
" \"\"\"\n",
" output = x.T.dot(self.W)\n",
" return self.step_function(output)\n",
" \n",
" def loss(self,predict,label):\n",
" \"\"\"\n",
" Arguments\n",
" predict : value prediced by the perceptron\n",
" label : original values to be predicted\n",
" \"\"\"\n",
" \n",
" l = label - predict\n",
" return l\n",
" \n",
" def back_propagate(self,loss_value,x):\n",
" \"\"\"\n",
" Arguments\n",
" loss_value : the calculated loss for a set of label and predicted value\n",
" x : the set of input values used for training\n",
" \"\"\"\n",
" \n",
" self.W += (self.lr*loss_value*x)\n",
" \n",
" def batch_train(self,x,label,epochs=2):\n",
" \"\"\"\n",
" Arguments\n",
" x : an array of data for training\n",
" label : the orignal label for the set of training data\n",
" epochs : total number of the times the data is used to train\n",
" \"\"\"\n",
" x = np.array(x)\n",
" n = x.shape[0]\n",
" bias_axis = np.ones([n,1])\n",
" x = np.concatenate((x,bias_axis),axis=1)\n",
" loss_hist = []\n",
" assert x.shape[1] == (self.W.shape[0]), \"Input shape does not match the specified length\"\n",
" for I in range(epochs):\n",
" avg_loss = 0\n",
" for J in range(x.shape[0]):\n",
" pred = self.forward(x[J])\n",
" l = self.loss(pred,label[J])\n",
" self.back_propagate(l,x[J])\n",
" avg_loss += abs(l)\n",
" h.append(self.W)\n",
" loss_hist.append(avg_loss)\n",
" return loss_hist"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Let us test it\n",
"Here we train the perceptron to approximate the function of an AND gate. \n",
"\n",
"#### AND gate \n",
"AND gate is a logical gate, which takes in one or more input and the resultant is one only if all of the input is one else it is zero\n",
"\n",
"Input1 | Input2 | Output\n",
"---|---|---\n",
"0 | 0 | 0\n",
"0 | 1 | 0\n",
"1 | 0 | 0\n",
"1 | 1 | 1"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Perceptron has been trained successfully\n"
]
}
],
"source": [
"p = Perceptron(2,0.5)\n",
"\n",
"# AND gate input\n",
"x = np.array([[0,0],\n",
" [0,1],\n",
" [1,0],\n",
" [1,1]])\n",
"\n",
"# Output of AND gate\n",
"y = np.array([0,0,0,1])\n",
"\n",
"#training\n",
"hist = p.batch_train(x,y,12)\n",
"\n",
"assert p.forward(np.array([0,0,1])) == 0, \"Try using more epochs\"\n",
"assert p.forward(np.array([0,1,1])) == 0, \"Try using more epochs\"\n",
"assert p.forward(np.array([1,0,1])) == 0, \"Try using more epochs\"\n",
"assert p.forward(np.array([1,1,1])) == 1, \"Try using more epochs\"\n",
"print (\"Perceptron has been trained successfully\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualization \n",
"\n",
"We can infer a lot about our perceptron by using an array of visualization techiniques. We will discuss some important visualization here.\n",
"\n",
"\n",
"First Let us plot the Loss to see the progress of the training. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"linkText": "Export to plot.ly",
"plotlyServerURL": "https://plot.ly",
"showLink": false
},
"data": [
{
"type": "scatter",
"uid": "ff5dd194-e7b6-4854-9491-a7573b23cdb0",
"x": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11
],
"y": [
2,
2,
3,
2,
1,
0,
0,
0,
0,
0,
0,
0
]
}
],
"layout": {
"title": {
"text": "Loss Value"
},
"xaxis": {
"title": {
"text": "Epochs"
}
},
"yaxis": {
"title": {
"text": "Loss"
}
}
}
},
"text/html": [
""
],
"text/vnd.plotly.v1+html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"trace = go.Scatter(\n",
" x = [I for I in range(len(hist))], \n",
" y = hist\n",
")\n",
"\n",
"layout = dict(title = 'Loss Value',\n",
" xaxis = dict(title = 'Epochs'),\n",
" yaxis = dict(title = 'Loss'),\n",
" )\n",
"\n",
"fig = dict(data=[trace], layout=layout)\n",
"py.iplot(fig)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From the above we can see that the model has converged after 5th epoch, this convergence epochs number may become longer or shorter if you run it. After training the model for 5 epochs, it learned the function of AND gate. If you see the training cell, we have used assert statements to verify the behaviour of the neural network. First two are the inputs and the one is padded to the input for the bias.\n",
"\n",
"## Decision plane\n",
"\n",
"Let us now visualize the Decision plane created by the perceptron. We have plotted the inputs in the XY-plane and Z-plane acts as the output. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"application/vnd.plotly.v1+json": {
"config": {
"linkText": "Export to plot.ly",
"plotlyServerURL": "https://plot.ly",
"showLink": false
},
"data": [
{
"colorscale": "Viridis",
"type": "surface",
"uid": "1b1ac50e-5c52-4559-9845-2428d96c085e",
"x": [
[
-0.2,
-0.2,
-0.2,
-0.2
],
[
0.2666666666666666,
0.2666666666666666,
0.2666666666666666,
0.2666666666666666
],
[
0.7333333333333332,
0.7333333333333332,
0.7333333333333332,
0.7333333333333332
],
[
1.2,
1.2,
1.2,
1.2
]
],
"y": [
[
-0.2,
0.2666666666666666,
0.7333333333333332,
1.2
],
[
-0.2,
0.2666666666666666,
0.7333333333333332,
1.2
],
[
-0.2,
0.2666666666666666,
0.7333333333333332,
1.2
],
[
-0.2,
0.2666666666666666,
0.7333333333333332,
1.2
]
],
"z": [
[
1.8178717089054577,
1.5618650100898448,
1.3058583112742321,
1.0498516124586192
],
[
1.3365404554653322,
1.0805337566497193,
0.8245270578341067,
0.568520359018494
],
[
0.8552092020252068,
0.5992025032095941,
0.34319580439398134,
0.08718910557836868
],
[
0.37387794858508117,
0.1178712497694685,
-0.13813544904614417,
-0.39414214786175705
]
]
},
{
"marker": {
"color": [
0,
0,
0,
1
],
"colorscale": "Viridis",
"line": {
"width": 0.5
},
"opacity": 0.8,
"size": 12
},
"mode": "markers",
"type": "scatter3d",
"uid": "32acd8d7-9985-43d3-ac43-e25c53e8d0e5",
"x": [
0,
0,
1,
1
],
"y": [
0,
1,
0,
1
],
"z": [
0,
0,
0,
1
]
}
],
"layout": {
"scene": {
"xaxis": {
"title": {
"text": "Input 1"
}
},
"yaxis": {
"title": {
"text": "Input 2"
}
},
"zaxis": {
"title": {
"text": "Output"
}
}
},
"title": {
"text": "Decision plane of And Gate"
}
}
},
"text/html": [
""
],
"text/vnd.plotly.v1+html": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def plot_decision_plane(w):\n",
" x_point = [0.0,0.0,1.0,1.0]\n",
" y_point = [0.0,1.0,0.0,1.0]\n",
" z_point = [0.0,0.0,0.0,1.0]\n",
" \n",
" x_plane = np.linspace(-0.2, 1.2, num=4)\n",
" y_plane = np.linspace(-0.2, 1.2, num=4)\n",
" Y_plane, X_plane = np.meshgrid(x_plane, y_plane)\n",
" Z_plane = -w[0]*X_plane - w[1]*Y_plane - w[2]\n",
" \n",
" data = [\n",
" go.Surface(x=X_plane,y=Y_plane,z=Z_plane,colorscale = 'Viridis'),\n",
" go.Scatter3d(\n",
" x=x_point,\n",
" y=y_point,\n",
" z=z_point,\n",
" mode='markers',\n",
" marker=dict(\n",
" size=12,\n",
" line=dict(\n",
" width=0.5\n",
" ),\n",
" colorscale='Viridis',\n",
" color=z_point,\n",
" opacity=0.8\n",
" )\n",
" )]\n",
" layout = go.Layout(\n",
" title= 'Decision plane of And Gate',\n",
" scene = dict(\n",
" xaxis = dict(title = 'Input 1'),\n",
" yaxis=dict(title = 'Input 2'),\n",
" zaxis=dict(title = 'Output')\n",
" )\n",
" )\n",
" \n",
" fig = go.Figure(data=data, layout=layout)\n",
" py.iplot(fig)\n",
" \n",
" \n",
"plot_decision_plane(p.W)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we see above only one point which yields 1 as output has been separated by the plane from the other three points which yields an output of 0. This plane is called as the decision plane. Every points(in XY-Plane) below the plane yields an output of 0 and above the plane yields 1.\n",
"\n",
"\n",
"### Things to try out\n",
" * You can try to approximate OR,NOR,NAND gates. \n",
" * Use different Activations and see if anything changes.\n",
" * Use a sigmoid activation and use it for regression instead of a classification."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}