{
"cells": [
{
"cell_type": "markdown",
"id": "4cb1c411-b550-4a0d-80f9-40846dc12f92",
"metadata": {},
"source": [
"# Notebook 3: Backpropagation [](https://colab.research.google.com/github/mattsankner/micrograd/blob/main/mg3_backpropogation.ipynb) [](https://nbviewer.jupyter.org/github/mattsankner/micrograd/blob/main/mg3_backpropogation.ipynb)\n"
]
},
{
"cell_type": "markdown",
"id": "36145db7-df04-4169-af61-2e1636139018",
"metadata": {},
"source": [
"## Welcome to the third mini-lecture!"
]
},
{
"cell_type": "markdown",
"id": "9c79f45c-e6d9-4d43-8c42-2a8353cb2065",
"metadata": {},
"source": [
"Now that we have some sense of what the derivative tells us about the function, for how to do backpropagation manually with the chain rule, and now that we also have our Value class partially built out, let's begin this lecture with by backpropogating through a neuron (which we will eventually use to build out neural networks). Run the code until you get to the next text block."
]
},
{
"cell_type": "code",
"execution_count": 106,
"id": "1ec7ba82-fc87-45b7-ab2a-36feaf1ee2f1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: graphviz in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (0.20.3)\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
]
}
],
"source": [
"!pip install graphviz\n",
"!export PATH=\"/usr/local/opt/graphviz/bin:$PATH\""
]
},
{
"cell_type": "code",
"execution_count": 107,
"id": "0d369653-eee5-48c2-8f36-c432fe364ab3",
"metadata": {},
"outputs": [],
"source": [
"from graphviz import Digraph #graphviz is an opensource vizualization software. We are building out this graph in graphviz API. \n",
"\n",
"def trace(root): #helper function that enumerates the ndoes and edges in the graph\n",
" # builds a set of all nodes and edges in a graph\n",
" nodes, edges = set(), set()\n",
" def build(v):\n",
" if v not in nodes:\n",
" nodes.add(v)\n",
" for child in v._prev:\n",
" edges.add((child, v))\n",
" build(child)\n",
" build(root)\n",
" return nodes, edges\n",
"\n",
"def draw_dot(root): #creating op nodes (not Value objects)\n",
" dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right\n",
" \n",
" nodes, edges = trace(root) #call trace\n",
" for n in nodes:\n",
" uid = str(id(n))\n",
" # for any value in the graph, create a rectangular ('record') node for it\n",
" dot.node(name = uid, label = \"{ %s | data %.4f | grad % .4f}\" % (n.label, n.data, n.grad), shape='record')\n",
" if n._op:\n",
" # if this value is a result of some operation, create an op node for it. Not a value object\n",
" dot.node(name = uid + n._op, label = n._op)\n",
" # and connect this node to it\n",
" dot.edge(uid + n._op, uid)\n",
"\n",
" for n1, n2 in edges:\n",
" # connect n1 to the op node of n2\n",
" dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
"\n",
" return dot"
]
},
{
"cell_type": "markdown",
"id": "b1d46bed-5938-4f60-8f10-7651daabb051",
"metadata": {},
"source": [
"### One more example of backpropogation..."
]
},
{
"cell_type": "markdown",
"id": "23506377-4008-4064-803a-170a1f8a0c8b",
"metadata": {},
"source": [
"We want to backpropogate through a neuron. In the simplest case, a neural network is made up of layers of neurons with an input layer, intermediate layers, and an output layer. These are called **Multi-Layer Perceptrons**. We can also mathmatically model the neurons we will use to contain the data and gradients for each variable in our neural netowrk. Observe the two pictures below:"
]
},
{
"cell_type": "markdown",
"id": "c4c290c0-016e-4977-acc8-fece7d3dde96",
"metadata": {},
"source": [
"Two Layer Neural Network:\n",
"\n",
"\n",
"Neuron:\n",
""
]
},
{
"cell_type": "markdown",
"id": "0fea1b85-0b64-4ad5-a3a8-6dc710c2596b",
"metadata": {},
"source": [
"In the neuron, we have inputs ($x$), and the synapses that have weights on them. The $w's$ are weights. The synapse enacts with the input to the neuron multiplicatively. What flows to the cell body of the neuron is $w*x$. There are many of these inputs flowing into the cell body. The cell body also has bias, $b$, which indicates the trigger-happiness of the neuron. \n",
"\n",
"Basically, we take all of the $w*x$ of the inputs, add it to the bias, and put it into the activation function."
]
},
{
"cell_type": "markdown",
"id": "16ed06d3-2d22-4fb0-8309-aa12e8ca27a9",
"metadata": {},
"source": [
"- ```x0, x1, x2``` = axen from a neuron\n",
"- ```w0, w1, w2``` = weights\n",
"- ```synapse``` = w0x0. Many synapses (```w1x1, $w2x2```)\n",
"- ```synapse``` travels on a dendrite to the cell body\n",
"- cell body contains the sum of the inputs plus a ```bias``` (which dictates the trigger happiness of the synapse/neuron -> bit more or bit less trigger-happy regardless of input)\n",
"- taking all of the ```wx``` of all the inputs + bias (```i``` iterations), taking it through an activation function (```sigmoid```, ```tanh```, etc.)\n",
"\n",
"The python ```numpy``` library has a ```tanh```, ```np.tanh```. We will use this in the example, and call it on a range and plot it:"
]
},
{
"cell_type": "code",
"execution_count": 108,
"id": "b6dae21e-e708-42aa-9d70-b858370f62c2",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 109,
"id": "f8668bf3-2764-4374-898c-9c1e894785ff",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABHYklEQVR4nO3de1xUdf4/8NeZYRgYucl1QFFATTEveEnCbpYIpLtp25Zu9lXZwl8Xao02i76pqaVd3LLMza3NtE3XvrWb1WboRItuieiqZJmaKIiCAyjCcJHhMHN+fwCTE6igzJyZM6/n48EDzpnPHN7nvSP76lw+R5AkSQIRERGRgqjkLoCIiIiopzHgEBERkeIw4BAREZHiMOAQERGR4jDgEBERkeIw4BAREZHiMOAQERGR4jDgEBERkeJ4yV2AHKxWK8rLy+Hv7w9BEOQuh4iIiLpAkiTU1dUhKioKKtWlj9F4ZMApLy9HdHS03GUQERHRFTh58iT69u17yTEeGXD8/f0BtDYoICBA5mpcgyiK2LZtG1JSUqDRaOQuR/HYb+div52L/XYuT+q3yWRCdHS07f/HL8UjA077aamAgAAGnDaiKEKn0yEgIEDx/0BcAfvtXOy3c7HfzuWJ/e7K5SW8yJiIiIgUhwGHiIiIFIcBh4iIiBSHAYeIiIgUhwGHiIiIFIcBh4iIiBSHAYeIiIgUhwGHiIiIFIcBh4iIiBTHoQFnx44d+PWvf42oqCgIgoDNmzdf9j15eXkYPXo0tFotBg4ciHXr1nUYs3r1asTExMDHxweJiYnYvXt3zxdPREREbsuhAaehoQEjR47E6tWruzS+uLgYU6ZMwa233orCwkLMmzcPDzzwALZu3Wob8+GHHyIrKwuLFi3Cvn37MHLkSKSmpqKystJRu0FERERuxqHPorr99ttx++23d3n8mjVrEBsbiz/96U8AgPj4eHzzzTd47bXXkJqaCgB49dVXkZGRgfT0dNt7vvjiC6xduxZPP/10z+8EERERuR2Xethmfn4+kpOT7dalpqZi3rx5AIDm5mbs3bsX2dnZttdVKhWSk5ORn59/0e2azWaYzWbbsslkAtD6gDJRFHtwD9xXex/YD+dgv52L/XYupfZbkiS0WCWIFitaLBJEq4QWixUtVgktFgnNbetbrK3rLBd+SRKs1tb3W61Ai9UKqwRY29bbfrZ9b/1ZumCd1LYs4cJloMViwdFTAo7lHoWgUkECAAmQ0DYGaPvetoCf17X+LF3w88/72nH/Lz5G6mTc6H5BmDJcf7Vtt9Odz5RLBRyj0YiIiAi7dRERETCZTDh//jzOnTsHi8XS6ZjDhw9fdLvLly/H4sWLO6zftm0bdDpdzxSvEAaDQe4SPAr77Vzst3PJ1e8WK9DYApy3tH5vbBHQZAHMbV/NVqDZIsBsBZrbllvXCxCtre9vkfDzzxcsS7j8U6zloQZOFstdhJ2i4hIIJ609us3GxsYuj3WpgOMo2dnZyMrKsi2bTCZER0cjJSUFAQEBMlbmOkRRhMFgwKRJk6DRaOQuR/HYb+div52rp/tttUo4d15EVZ0ZVXVmVLZ/r29GVZ0ZZ+rNMJ1vQW2TCNN5EefFnv0/1UtRqwR4qQR4qQVoVCpo1AK81CrbepXQ9r19WQV4qVRQCa3vVQsCBEGAWgUIggCVAKgEoe2rdZ1aEAABrctoX9/6miAAklXC6fIy9O3bByqVGm3D274LFyy3hrO2b/brLljf7sLXfl7X/l6hwzrbctv3EX0DkRwffrUtttN+BqYrXCrg6PV6VFRU2K2rqKhAQEAAfH19oVaroVarOx2j11/8MJhWq4VWq+2wXqPR8I/dL7AnzsV+Oxf77Vzd6XeDuQUlZxtQcqYRJWcbUHymASVnGlBWcx5VdWa0WDueMrkcfx8vBPpqEOirQYCPBr20XtB5q6HzVsPXW41e3l7wbVtu/9lHo4a3lwratq/Wn9V2y95eKnipVLbgIjdRFLFly0lMnjxc8Z/v7uyfSwWcpKQkbNmyxW6dwWBAUlISAMDb2xtjxoxBbm4upk2bBgCwWq3Izc1FZmams8slIqJuOt9swQ/ltfjuZA2OVtSj+GxrkKmsM1/2vSG9vBHmr0V4gA/C/bW2r1B/LYJ8vW1hJtBXAz8fL6hdIHyQfBwacOrr61FUVGRbLi4uRmFhIYKDg9GvXz9kZ2ejrKwM77//PgDgwQcfxJtvvon58+fj97//Pb7++mv83//9H7744gvbNrKysjB79myMHTsW48aNw8qVK9HQ0GC7q4qIiFyD1SrhaEUd9p+sQeHJGhSW1uBIRR0sFzkaE9zLGzEhOsSE9kJsSC/EhPZCv2AdwgO0CPXTQqPm3LTUdQ4NOP/9739x66232pbbr4OZPXs21q1bh9OnT6O0tNT2emxsLL744gs8/vjjeP3119G3b1/89a9/td0iDgDTp09HVVUVFi5cCKPRiISEBOTk5HS48JiIiJzLapXw3akaGA4aYTiowjP7vkaD2dJhXLi/FgnRQYiPDEBcWC/EtIWZQF9ln14h53JowJkwYUKnt5q162yW4gkTJmD//v2X3G5mZiZPSRERuYDmFisKis9i60EjDD9WoMLUfqpJBcACX40aw/sGYlR0EBKigzAyOgiRgT62C1iJHMWlrsEhIiLX12BuwfafqrDtoBG5hytR19Rie62Xtxo3DwpFwPlyzEy7EUP7BMGLp5ZIBgw4RER0WVarhLyfKrGxoBQ7jp5Bc8vPt2KH+nlj0tAIpAzVY/zAEKgkK7ZsKUN8pD/DDcmGAYeIiC6qucWKTwvL8M5/juOninrb+n7BOqReG4HUa/UY1a+33R1LohPnoSG6GAYcIiLqoK5JxN93l2LtNyUwmpoAAH5aL8y4Lhq/HdsXgyP8eR0NuTQGHCIisqkwNWHtt8XYuKsUdebWa2vC/bVIvyEW9yb2451O5DYYcIiICKVnG7Hq66PYXFgG0dJ69+vAcD/MvTkOUxOioPVSy1whUfcw4BAReTCrVcK6nSV4eethNLVdOzMuJhj/75Y43Do43CUeRUB0JRhwiIg8VPGZBsz/+DvsKTkHAEiKC8GTaYMxul9vmSsjunoMOEREHsZilfDet8V4ZesRmFus6OWtxjNT4nHvuH68cJgUgwGHiMiDHKuqx5MffYd9pTUAgBsHhuLFu4ajb2+dvIUR9TAGHCIiD2CxSnj3m+P407afYG6xwk/rhWenxGP6ddE8akOKxIBDRKRwRZX1ePLj77C/7ajNzdeEYflvhqNPkK+8hRE5EAMOEZGC7S6uxv3r9qDO3AJ/rRcW/Goo7h7bl0dtSPEYcIiIFGr7T1X4f3/7L5pEK8bFBOP13yUgMpBHbcgzMOAQESnQl9+fxmOb9kO0SLh1cBjeum8MfDScrI88BwMOEZHCfLz3FOZ//B2sEjBleCRem54Aby8+1Zs8CwMOEZGCrN9ZgkWfHQQA3DO2L5b/ZoTdk76JPAUDDhGRQqz+dxFe2XoEAPD7G2Lx7JR4PmqBPBYDDhGRm5MkCS/lHMGa7ccAAH+YOAjzkgfxTinyaAw4RERuzGqVsPCzH/DBrlIAwP9OjkfGzXEyV0UkPwYcIiI3ZbVK+ONH3+Gf+8sgCMCyO4fjd+P6yV0WkUtgwCEiclNvbT+Gf+4vg5dKwKvTE3DHyCi5SyJyGbxvkIjIDeUfO4s/bWu9oPiFO4cx3BD9AgMOEZGbqaxrwmOb9sMqAXeN7ot7xkbLXRKRy2HAISJyIxarhD/8vRBVdWYMjvDH89OG8W4pok4w4BARuZGVX/2E/ONn0ctbjdUzR8PXm49fIOoMAw4RkZvIO1KJVV8XAQCW/WY4Bob7yVwRketiwCEicgPlNefx+IeFAID7ru+HqQl95C2IyMUx4BARuTjRYkXmxn041yhieJ9ALPjVULlLInJ5DDhERC7uxS8PY19pDfx9vLD63tHQevG6G6LLYcAhInJhOT8Y8e43xQCAP909Ev1CdDJXROQenBJwVq9ejZiYGPj4+CAxMRG7d+++6NgJEyZAEIQOX1OmTLGNmTNnTofX09LSnLErREROc+JsA5786DsAQMZNsUi5Vi9zRUTuw+GPavjwww+RlZWFNWvWIDExEStXrkRqaiqOHDmC8PDwDuP/+c9/orm52bZ89uxZjBw5EnfffbfduLS0NLz33nu2Za1W67idICJysibRgoc37EOduQVj+vfG/LQhcpdE5FYcfgTn1VdfRUZGBtLT0zF06FCsWbMGOp0Oa9eu7XR8cHAw9Hq97ctgMECn03UIOFqt1m5c7969Hb0rRERO89f/HMfBchOCe3njzXtHQaPmFQVE3eHQIzjNzc3Yu3cvsrOzbetUKhWSk5ORn5/fpW28++67mDFjBnr16mW3Pi8vD+Hh4ejduzduu+02PP/88wgJCel0G2azGWaz2bZsMpkAAKIoQhTF7u6WIrX3gf1wDvbbudyt31V1Zvw57xgA4H9vH4xQnZfb1A64X7/dnSf1uzv7KEiSJDmqkPLycvTp0wc7d+5EUlKSbf38+fOxfft2FBQUXPL9u3fvRmJiIgoKCjBu3Djb+k2bNkGn0yE2NhbHjh3DM888Az8/P+Tn50Ot7nh3wXPPPYfFixd3WL9x40bodLxgj4hcy6ZjKuRXqtDfT8LjwyzgkxiIWjU2NuLee+9FbW0tAgICLjnW4dfgXI13330Xw4cPtws3ADBjxgzbz8OHD8eIESMwYMAA5OXlYeLEiR22k52djaysLNuyyWRCdHQ0UlJSLtsgTyGKIgwGAyZNmgSNRiN3OYrHfjuXO/X7sLEOu3a1HuF++XeJGN0vSN6CroA79VsJPKnf7WdgusKhASc0NBRqtRoVFRV26ysqKqDXX/pugIaGBmzatAlLliy57O+Ji4tDaGgoioqKOg04Wq2204uQNRqN4j8M3cWeOBf77Vyu3m9JkvDS1qOQJGDK8EgkDgiTu6Sr4ur9VhpP6Hd39s+hV615e3tjzJgxyM3Nta2zWq3Izc21O2XVmY8++ghmsxn33XffZX/PqVOncPbsWURGRl51zUREcsk7UoVvis7AW63CU7xriuiqOPyy/KysLLzzzjtYv349Dh06hIceeggNDQ1IT08HAMyaNcvuIuR27777LqZNm9bhwuH6+no8+eST2LVrF0pKSpCbm4upU6di4MCBSE1NdfTuEBE5RIvFihe2HAIAzLkhhhP6EV0lh1+DM336dFRVVWHhwoUwGo1ISEhATk4OIiIiAAClpaVQqexz1pEjR/DNN99g27ZtHbanVqtx4MABrF+/HjU1NYiKikJKSgqWLl3KuXCIyG39fc9JFFXWo7dOg0duHSh3OURuzykXGWdmZiIzM7PT1/Ly8jqsGzx4MC52c5evry+2bt3ak+UREcnK1CTiNcNPAIDHJ12DQF9lX0dB5AycOYqISGar/12E6oZmDAjrhd+N6yd3OUSKwIBDRCSjk9WNeO+bEgDAM5PjOWMxUQ/hvyQiIhm9lHMYzRYrbhgYgtuGdHw+HxFdGQYcIiKZ7D1xDv86cBqCAPzv5KEQOGUxUY9hwCEikoEkSXj+ix8BAPeMicbQKM6qTtSTGHCIiGTwrwOnsb+0BjpvNZ5IuUbucogUhwGHiMjJmkQLXvzyMADgwVsGIDzAR+aKiJSHAYeIyMk+2HUCZTXnoQ/wQcZNcXKXQ6RIDDhERE7UYrHivW9LAAB/SB4EX2+1vAURKRQDDhGRE311qBJlNefRW6fBnaP6yF0OkWIx4BAROdF73xYDAO5N7AcfDY/eEDkKAw4RkZP8WG5CQXE11CoB913fX+5yiBSNAYeIyEnW7Ww9enP7MD0iA31lroZI2RhwiIicoLqhGZsLywEA6TfEyFsMkQdgwCEicoK/7y5Fc4sVw/sEYnS/3nKXQ6R4DDhERA4mWqz4W/4JAK1Hb/jMKSLHY8AhInKwnB+MMJqaEOqnxZQRkXKXQ+QRGHCIiBxs3c4SAMDMxH7QevHWcCJnYMAhInKgA6dqsPfEOWjUAmYm9pO7HCKPwYBDRORA69oeyzBleCQfqknkRAw4REQOUlnXhM8PtN8aHitzNUSehQGHiMhBNhaUQrRIGNUvCCOjg+Quh8ijMOAQETlAc4sVH+wqBcCjN0RyYMAhInKAL74vx5l6MyICtLh9mF7ucog8DgMOEVEPkyQJ77VdXHxfYn9o1PxTS+Rs/FdHRNTD9pXW4MCpWnh7qXAvbw0nkgUDDhFRD2uf2O+OkVEI8dPKWwyRh2LAISLqQcbaJnz5/WkAwJzxMfIWQ+TBGHCIiHrQB7tOoMUqYVxMMIb1CZS7HCKPxYBDRNRDrFYJ/9h3CgAwa3x/mash8mwMOEREPWRPSTVO1zbBX+uF5PgIucsh8mhOCTirV69GTEwMfHx8kJiYiN27d1907Lp16yAIgt2Xj4/981skScLChQsRGRkJX19fJCcn4+jRo47eDSKiS/rsu9bHMqQO08NHw6eGE8nJ4QHnww8/RFZWFhYtWoR9+/Zh5MiRSE1NRWVl5UXfExAQgNOnT9u+Tpw4Yff6yy+/jDfeeANr1qxBQUEBevXqhdTUVDQ1NTl6d4iIOiVarNjSdnHxHSOjZK6GiBwecF599VVkZGQgPT0dQ4cOxZo1a6DT6bB27dqLvkcQBOj1ettXRMTPh3olScLKlSvx7LPPYurUqRgxYgTef/99lJeXY/PmzY7eHSKiTn1z9AzONYoI9fPG+AEhcpdD5PG8HLnx5uZm7N27F9nZ2bZ1KpUKycnJyM/Pv+j76uvr0b9/f1itVowePRrLli3DtddeCwAoLi6G0WhEcnKybXxgYCASExORn5+PGTNmdNie2WyG2Wy2LZtMJgCAKIoQRfGq91MJ2vvAfjgH++1czuj35v2tFxfffm0EJKsFotXisN/l6vj5di5P6nd39tGhAefMmTOwWCx2R2AAICIiAocPH+70PYMHD8batWsxYsQI1NbWYsWKFRg/fjwOHjyIvn37wmg02rbxy222v/ZLy5cvx+LFizus37ZtG3Q63ZXsmmIZDAa5S/Ao7LdzOarfzRYg53s1AAEhDcXYsqXYIb/H3fDz7Vye0O/GxsYuj3VowLkSSUlJSEpKsi2PHz8e8fHx+Mtf/oKlS5de0Tazs7ORlZVlWzaZTIiOjkZKSgoCAgKuumYlEEURBoMBkyZNgkajkbscxWO/ncvR/d7yvRHm3QfQJ8gHD99zEwRB6PHf4U74+XYuT+p3+xmYrnBowAkNDYVarUZFRYXd+oqKCuj1XXu6rkajwahRo1BUVAQAtvdVVFQgMjLSbpsJCQmdbkOr1UKr7ThdukajUfyHobvYE+div53LUf3+4ofWv3F3JPSBt7d3j2/fXfHz7Vye0O/u7J9DLzL29vbGmDFjkJuba1tntVqRm5trd5TmUiwWC77//ntbmImNjYVer7fbpslkQkFBQZe3SUTUU2rPi8g7UgWAd08RuRKHn6LKysrC7NmzMXbsWIwbNw4rV65EQ0MD0tPTAQCzZs1Cnz59sHz5cgDAkiVLcP3112PgwIGoqanBK6+8ghMnTuCBBx4A0HqH1bx58/D8889j0KBBiI2NxYIFCxAVFYVp06Y5eneIiOxsPWhEs8WKQeF+GKL3l7scImrj8IAzffp0VFVVYeHChTAajUhISEBOTo7tIuHS0lKoVD8fSDp37hwyMjJgNBrRu3dvjBkzBjt37sTQoUNtY+bPn4+GhgbMnTsXNTU1uPHGG5GTk9NhQkAiIkf7vG1yvztGRnn8tTdErsQpFxlnZmYiMzOz09fy8vLsll977TW89tprl9yeIAhYsmQJlixZ0lMlEhF1W2VdE74tOgMAuCOBp6eIXAmfRUVEdIW2HDgNqwSMjA5C/5BecpdDRBdgwCEiukKfXXB6iohcCwMOEdEVOFndiH2lNRAE4NcjIi//BiJyKgYcIqIr8PmB1qM3SXEhCA/gDQ5EroYBh4joCnxWyNNTRK6MAYeIqJt+qqjDYWMdNGoBtw/j6SkiV8SAQ0TUTe1Hb265JhyBOmVPjU/krhhwiIi6QZKkn++e4tw3RC6LAYeIqBsKT9agtLoRvho1kuPD5S6HiC6CAYeIqBvaj95MGhoBnbdTJoMnoivAgENE1EUWq4R/HTgNAJjK01NELo0Bh4ioiwqOn0VVnRmBvhrcNChM7nKI6BIYcIiIuqj99NTk4Xp4e/HPJ5Er479QIqIuaLFY8eUPRgDArzm5H5HLY8AhIuqC/544h9rzIoJ7eSMxNkTucojoMhhwiIi6IPdQBQBgwuAwqFWCzNUQ0eUw4BARdUHuoUoAwMQhETJXQkRdwYBDRHQZx6vqcfxMAzRqATdfEyp3OUTUBQw4RESX8fXh1qM3ibEh8Pfhs6eI3AEDDhHRZXzVdv3NRD6agchtMOAQEV1CbaOIPSXnAPD6GyJ3woBDRHQJeT9VwmKVMCjcD/1CdHKXQ0RdxIBDRHQJtrun4nn0hsidMOAQEV1Ei8WKvCOtASeZ198QuRUGHCKii/jviXMwNbWgt06DUf16y10OEXUDAw4R0UW0z1586+Bwzl5M5GYYcIiILoLX3xC5LwYcIqJOcPZiIvfGgENE1In2ozecvZjIPTHgEBF1IvcwZy8mcmcMOEREv8DZi4ncn1MCzurVqxETEwMfHx8kJiZi9+7dFx37zjvv4KabbkLv3r3Ru3dvJCcndxg/Z84cCIJg95WWlubo3SAiD8HZi4ncn8MDzocffoisrCwsWrQI+/btw8iRI5GamorKyspOx+fl5eF3v/sd/v3vfyM/Px/R0dFISUlBWVmZ3bi0tDScPn3a9vX3v//d0btCRB6Cd08RuT+HB5xXX30VGRkZSE9Px9ChQ7FmzRrodDqsXbu20/EbNmzAww8/jISEBAwZMgR//etfYbVakZubazdOq9VCr9fbvnr35iRcRHT1RM5eTKQIXo7ceHNzM/bu3Yvs7GzbOpVKheTkZOTn53dpG42NjRBFEcHBwXbr8/LyEB4ejt69e+O2227D888/j5CQkE63YTabYTabbcsmkwkAIIoiRFHs7m4pUnsf2A/nYL+dqzv9Liiuts1ePCzSj/8bXQF+vp3Lk/rdnX10aMA5c+YMLBYLIiLsD/NGRETg8OHDXdrGU089haioKCQnJ9vWpaWl4Te/+Q1iY2Nx7NgxPPPMM7j99tuRn58PtVrdYRvLly/H4sWLO6zftm0bdDqeX7+QwWCQuwSPwn47V1f6vblEBUCFgToztuZ86fiiFIyfb+fyhH43NjZ2eaxDA87VevHFF7Fp0ybk5eXBx8fHtn7GjBm2n4cPH44RI0ZgwIAByMvLw8SJEztsJzs7G1lZWbZlk8lku7YnICDAsTvhJkRRhMFgwKRJk6DRcM4PR2O/nas7/V658hsAjfifiQm4fZjeOQUqDD/fzuVJ/W4/A9MVDg04oaGhUKvVqKiosFtfUVEBvf7SfzhWrFiBF198EV999RVGjBhxybFxcXEIDQ1FUVFRpwFHq9VCq9V2WK/RaBT/Yegu9sS52G/nuly/j1fVo/hsIzRqAbfG6/m/zVXi59u5PKHf3dk/h15k7O3tjTFjxthdINx+wXBSUtJF3/fyyy9j6dKlyMnJwdixYy/7e06dOoWzZ88iMjKyR+omIs/E2YuJlMPhd1FlZWXhnXfewfr163Ho0CE89NBDaGhoQHp6OgBg1qxZdhchv/TSS1iwYAHWrl2LmJgYGI1GGI1G1NfXAwDq6+vx5JNPYteuXSgpKUFubi6mTp2KgQMHIjU11dG7Q0QK1j578W1DePcUkbtz+DU406dPR1VVFRYuXAij0YiEhATk5OTYLjwuLS2FSvVzznrrrbfQ3NyM3/72t3bbWbRoEZ577jmo1WocOHAA69evR01NDaKiopCSkoKlS5d2ehqKiKgrLpy9OJnz3xC5PadcZJyZmYnMzMxOX8vLy7NbLikpueS2fH19sXXr1h6qjIioFWcvJlIWPouKiAicvZhIaRhwiMjjcfZiIuVhwCEij7e/tAamphYE6TQY1Y+PfSFSAgYcIvJ4O36qAgDcNCgMapUgczVE1BMYcIjI4+042hpwbh4UKnMlRNRTGHCIyKNVNzTj+7JaAMDN14TJXA0R9RQGHCLyaP85WgVJAobo/RER4HP5NxCRW2DAISKPtuOnMwB49IZIaRhwiMhjSZKE/9iuv2HAIVISBhwi8liHjXWorDPDR6PC2BjeHk6kJAw4ROSx2m8Pvz4uBD4atczVEFFPYsAhIo+1g6eniBSLAYeIPFJjcwv2FLc+PZwXGBMpDwMOEXmkguPVaLZY0SfIFwPCesldDhH1MAYcIvJI29uuv7n5mlAIAh/PQKQ0DDhE5JF4/Q2RsjHgEJHHOXWuEcerGqBWCRg/kM+fIlIiBhwi8jjtsxcnRAch0FcjczVE5AgMOETkcdrnv+HpKSLlYsAhIo/SYrHi22Ptz5/i6SkipWLAISKPUniyBnVNLQjSaTCib5Dc5RCRgzDgEJFHaT89dcPAUKhVvD2cSKkYcIjIo2w/2np66hbOXkykaAw4ROQxzjU248CpGgC8wJhI6RhwiMhj7DxWDUkCBkf4Qx/oI3c5RORADDhE5DH+U8S7p4g8BQMOEXkESQK+OXoWAJ8eTuQJGHCIyCOcPg9U1Jnho1HhuphgucshIgdjwCEij3C4pvWW8MTYEPho1DJXQ0SOxoBDRB6hPeDw9BSRZ2DAISLFO99swTFTa8C5hRcYE3kEpwSc1atXIyYmBj4+PkhMTMTu3bsvOf6jjz7CkCFD4OPjg+HDh2PLli12r0uShIULFyIyMhK+vr5ITk7G0aNHHbkLROTGdpdUo0USEBnogwFhfnKXQ0RO4PCA8+GHHyIrKwuLFi3Cvn37MHLkSKSmpqKysrLT8Tt37sTvfvc73H///di/fz+mTZuGadOm4YcffrCNefnll/HGG29gzZo1KCgoQK9evZCamoqmpiZH7w4RuaH/FLXePXXTwBAIAh/PQOQJHB5wXn31VWRkZCA9PR1Dhw7FmjVroNPpsHbt2k7Hv/7660hLS8OTTz6J+Ph4LF26FKNHj8abb74JoPXozcqVK/Hss89i6tSpGDFiBN5//32Ul5dj8+bNjt4dInJD/2m7PfzGgSEyV0JEzuLlyI03Nzdj7969yM7Otq1TqVRITk5Gfn5+p+/Jz89HVlaW3brU1FRbeCkuLobRaERycrLt9cDAQCQmJiI/Px8zZszosE2z2Qyz2WxbNplMAABRFCGK4hXvn5K094H9cA7223nKa87j+JkGCJBwXb8A9twJ+Pl2Lk/qd3f20aEB58yZM7BYLIiIiLBbHxERgcOHD3f6HqPR2Ol4o9Foe7193cXG/NLy5cuxePHiDuu3bdsGnU7XtZ3xEAaDQe4SPAr77Xg7KwQAavT3A3Z/kyd3OR6Fn2/n8oR+NzY2dnmsQwOOq8jOzrY7KmQymRAdHY2UlBQEBATIWJnrEEURBoMBkyZNgkajkbscxWO/nWfL3wsBVGJIkJX9dhJ+vp3Lk/rdfgamKxwacEJDQ6FWq1FRUWG3vqKiAnq9vtP36PX6S45v/15RUYHIyEi7MQkJCZ1uU6vVQqvVdliv0WgU/2HoLvbEudhvx2qxWLHzeDUAID5IYr+djP12Lk/od3f2z6EXGXt7e2PMmDHIzc21rbNarcjNzUVSUlKn70lKSrIbD7QedmsfHxsbC71ebzfGZDKhoKDgotskIs/03aka1DW1INDXC/14dziRR3H4KaqsrCzMnj0bY8eOxbhx47By5Uo0NDQgPT0dADBr1iz06dMHy5cvBwD84Q9/wC233II//elPmDJlCjZt2oT//ve/ePvttwEAgiBg3rx5eP755zFo0CDExsZiwYIFiIqKwrRp0xy9O0TkRrb/1Pr08PFxIVAJZTJXQ0TO5PCAM336dFRVVWHhwoUwGo1ISEhATk6O7SLh0tJSqFQ/H0gaP348Nm7ciGeffRbPPPMMBg0ahM2bN2PYsGG2MfPnz0dDQwPmzp2Lmpoa3HjjjcjJyYGPj4+jd4eI3MiOn6oAADcNCgEqGHCIPIlTLjLOzMxEZmZmp6/l5eV1WHf33Xfj7rvvvuj2BEHAkiVLsGTJkp4qkYgUpqaxGQdO1QAAbhwYiv0Vlx5PRMrCZ1ERkSJ9U3QGVgkYFO6HyEAe3SXyNAw4RKRI7aen+PRwIs/EgENEiiNJEna0XWDMgEPkmRhwiEhxjlbWw2hqgtZLhcTYYLnLISIZMOAQkeK0n55KjAuBj0YtczVEJAcGHCJSnO3t198MCpW5EiKSCwMOESlKk2jB7uLWxzPcwutviDwWAw4RKUpBcTXMLVZEBvpgYDifz0DkqRhwiEhRbLeHDwqDIAgyV0NEcmHAISJF4fw3RAQw4BCRgpTXnMfRynqohNbHMxCR52LAISLF+M/R1qM3I6ODEKjTyFwNEcmJAYeIFMM2e/Egnp4i8nQMOESkCBarhG+K+HgGImrFgENEivDdqRrUnhcR4OOFkX0D5S6HiGTGgENEitB+99SNg0LhpeafNiJPx78CRKQIF85/Q0TEgENEbq+2UUThyRoAvP6GiFox4BCR2/um6AysEjAw3A9RQb5yl0NELoABh4jcHk9PEdEvMeAQkVuTJAk7jrY/noGzFxNRKwYcInJrRZX1OF3bBK2XCtfHhchdDhG5CAYcInJr29tOT42LDYaPRi1zNUTkKhhwiMit7TjaOnvxLbx7ioguwIBDRG6rSbSg4PhZALw9nIjsMeAQkdvaXVwNc4sV+gAfDAr3k7scInIhDDhE5Lbar7+5+ZpQCIIgczVE5EoYcIjIbX19uBIAcOvgcJkrISJXw4BDRG7pWFU9is80wFutwk28/oaIfoEBh4jcUu6hCgBAYlww/LReMldDRK6GAYeI3NJXh1pPTyXHR8hcCRG5IocGnOrqasycORMBAQEICgrC/fffj/r6+kuOf/TRRzF48GD4+vqiX79+eOyxx1BbW2s3ThCEDl+bNm1y5K4QkQupaWzG3hPnAAC3DeH1N0TUkUOP686cOROnT5+GwWCAKIpIT0/H3LlzsXHjxk7Hl5eXo7y8HCtWrMDQoUNx4sQJPPjggygvL8fHH39sN/a9995DWlqabTkoKMiRu0JELiTvSBUsVgmDI/wRHayTuxwickEOCziHDh1CTk4O9uzZg7FjxwIAVq1ahcmTJ2PFihWIiorq8J5hw4bhH//4h215wIABeOGFF3DfffehpaUFXl4/lxsUFAS9Xu+o8onIheW23T01MZ5Hb4iocw4LOPn5+QgKCrKFGwBITk6GSqVCQUEB7rzzzi5tp7a2FgEBAXbhBgAeeeQRPPDAA4iLi8ODDz6I9PT0i86DYTabYTabbcsmkwkAIIoiRFHs7q4pUnsf2A/nYL+vnGixIu9Ia8CZMCikSz1kv52L/XYuT+p3d/bRYQHHaDQiPNz+v668vLwQHBwMo9HYpW2cOXMGS5cuxdy5c+3WL1myBLfddht0Oh22bduGhx9+GPX19Xjsscc63c7y5cuxePHiDuu3bdsGnY6Hty9kMBjkLsGjsN/dd7RWQF2TGn5eEsq+34nTP3T9vey3c7HfzuUJ/W5sbOzy2G4HnKeffhovvfTSJcccOnSou5vtwGQyYcqUKRg6dCiee+45u9cWLFhg+3nUqFFoaGjAK6+8ctGAk52djaysLLttR0dHIyUlBQEBAVddqxKIogiDwYBJkyZBo9HIXY7isd9XbtmXRwCcwKThffCrKcO69B7227nYb+fypH63n4Hpim4HnCeeeAJz5sy55Ji4uDjo9XpUVlbarW9paUF1dfVlr52pq6tDWloa/P398cknn1z2f7DExEQsXboUZrMZWq22w+tarbbT9RqNRvEfhu5iT5yL/e4eSZLw7yOtj2dIGarvdu/Yb+div53LE/rdnf3rdsAJCwtDWNjlZw1NSkpCTU0N9u7dizFjxgAAvv76a1itViQmJl70fSaTCampqdBqtfjss8/g4+Nz2d9VWFiI3r17dxpiiEg5jp9pQMnZRs5eTESX5bBrcOLj45GWloaMjAysWbMGoigiMzMTM2bMsN1BVVZWhokTJ+L999/HuHHjYDKZkJKSgsbGRnzwwQcwmUy2w1FhYWFQq9X4/PPPUVFRgeuvvx4+Pj4wGAxYtmwZ/vjHPzpqV4jIRXD2YiLqKof+hdiwYQMyMzMxceJEqFQq3HXXXXjjjTdsr4uiiCNHjtguGtq3bx8KCgoAAAMHDrTbVnFxMWJiYqDRaLB69Wo8/vjjkCQJAwcOxKuvvoqMjAxH7goRuYD22YsncnI/IroMhwac4ODgi07qBwAxMTGQJMm2PGHCBLvlzqSlpdlN8EdEnuHC2Ysn8vEMRHQZfBYVEbkFzl5MRN3BgENEboGzFxNRdzDgEJHLu3D2Yp6eIqKuYMAhIpe3p6QadU0tCO7ljYToILnLISI3wIBDRC4vt+3uqVsHh0Ot6vyZc0REF2LAISKXJkmSbf6bZF5/Q0RdxIBDRC7tWBVnLyai7mPAISKX9vVhzl5MRN3HgENELo2zFxPRlWDAISKXxdmLiehKMeAQkcvi7MVEdKUYcIjIZX3VdvcUZy8mou5iwCEilyRarNj+UxUAnp4iou5jwCEil8TZi4noajDgEJFL4uzFRHQ1GHCIyOVIkoScH4wAOHsxEV0ZBhwicjn7Ss+hrOY8enmrcSvnvyGiK8CAQ0Qu59PCcgBA6rV6+GjUMldDRO6IAYeIXEqLxYot358GAPw6IUrmaojIXTHgEJFL2XnsLM7UN6O3ToMbB4bKXQ4RuSkGHCJyKZ9913p6asqISGjU/BNFRFeGfz2IyGU0iRZsbbt76o6RfWSuhojcGQMOEbmMvCOVqDO3IDLQB2P795a7HCJyYww4ROQy2k9P/XpkFFSc3I+IrgIDDhG5hLom0TZ78R0jefcUEV0dBhwicgmGHytgbrEiLqwXro0KkLscInJzDDhE5BLaJ/e7Y2QUBIGnp4jo6jDgEJHsztab8U3RGQA8PUVEPYMBh4hkt+UHIyxWCcP7BCIuzE/ucohIARhwiEh2n19weoqIqCcw4BCRrMprzmN3STUEAfjVyEi5yyEihXBowKmursbMmTMREBCAoKAg3H///aivr7/keyZMmABBEOy+HnzwQbsxpaWlmDJlCnQ6HcLDw/Hkk0+ipaXFkbtCRA7yedvcN9fFBCMy0FfmaohIKbwcufGZM2fi9OnTMBgMEEUR6enpmDt3LjZu3HjJ92VkZGDJkiW2ZZ1OZ/vZYrFgypQp0Ov12LlzJ06fPo1Zs2ZBo9Fg2bJlDtsXInKM9sn9pvLJ4UTUgxwWcA4dOoScnBzs2bMHY8eOBQCsWrUKkydPxooVKxAVdfE/ZjqdDnq9vtPXtm3bhh9//BFfffUVIiIikJCQgKVLl+Kpp57Cc889B29vb4fsDxH1vGNV9ThYboKXSsDkYTw9RUQ9x2EBJz8/H0FBQbZwAwDJyclQqVQoKCjAnXfeedH3btiwAR988AH0ej1+/etfY8GCBbajOPn5+Rg+fDgiIiJs41NTU/HQQw/h4MGDGDVqVIftmc1mmM1m27LJZAIAiKIIURSvel+VoL0P7IdzsN+tNu87CQC4YWAI/LwFh/WD/XYu9tu5PKnf3dlHhwUco9GI8PBw+1/m5YXg4GAYjcaLvu/ee+9F//79ERUVhQMHDuCpp57CkSNH8M9//tO23QvDDQDb8sW2u3z5cixevLjD+m3bttmd/iLAYDDIXYJH8eR+SxKwqVANQEC0tQJbtmxx+O/05H7Lgf12Lk/od2NjY5fHdjvgPP3003jppZcuOebQoUPd3azN3LlzbT8PHz4ckZGRmDhxIo4dO4YBAwZc0Tazs7ORlZVlWzaZTIiOjkZKSgoCAjglPNCaig0GAyZNmgSNRiN3OYrHfgM/lJlQtWsXfDQqPDHjNvhpHXdJIPvtXOy3c3lSv9vPwHRFt/+iPPHEE5gzZ84lx8TFxUGv16OystJufUtLC6qrqy96fU1nEhMTAQBFRUUYMGAA9Ho9du/ebTemoqICAC66Xa1WC61W22G9RqNR/Iehu9gT5/Lkfm852PrvdmJ8BHr7OefuKU/utxzYb+fyhH53Z/+6HXDCwsIQFhZ22XFJSUmoqanB3r17MWbMGADA119/DavVagstXVFYWAgAiIyMtG33hRdeQGVlpe0UmMFgQEBAAIYOHdrNvSEiOVitEv514DQATu5HRI7hsHlw4uPjkZaWhoyMDOzevRvffvstMjMzMWPGDNsdVGVlZRgyZIjtiMyxY8ewdOlS7N27FyUlJfjss88wa9Ys3HzzzRgxYgQAICUlBUOHDsX//M//4LvvvsPWrVvx7LPP4pFHHun0KA0RuZ49JdU4XdsEfx8vTBh8+f9gIiLqLodO9LdhwwYMGTIEEydOxOTJk3HjjTfi7bfftr0uiiKOHDliu2jI29sbX331FVJSUjBkyBA88cQTuOuuu/D555/b3qNWq/Gvf/0LarUaSUlJuO+++zBr1iy7eXOIyLV92jb3ze3D9NB6qWWuhoiUyKET/QUHB19yUr+YmBhIkmRbjo6Oxvbt2y+73f79+zvljgsi6nlNogVbvm8/PdVH5mqISKn4LCoicqpPC8tQ0yiiT5AvkgaEyF0OESkUAw4ROY0kSXjv2xIAwOzx/aFWCfIWRESKxYBDRE6z63g1Dhvr4KtRY/rYfnKXQ0QKxoBDRE6zbmcxAOA3o/sgUKfs+TqISF4MOETkFCerG2H4sXVyvznjY+QthogUjwGHiJzib7tOwCoBNw0KxaAIf7nLISKFY8AhIodrbG7Bpt2lAHj0hoicgwGHiBzun/vKYGpqQf8QHW4dHC53OUTkARhwiMihJEnCup0lAIDZSTFQ8dZwInICBhwicqhvis6gqLIevbzV+O3YvnKXQ0QeggGHiBxqXdvEfnePjUaAD28NJyLnYMAhIocpOdOAr49UAgBmJfWXuRoi8iQMOETkMOvzSyBJwK2DwxAX5id3OUTkQRhwiMgh6ppEfPTfUwCAOTfEylwNEXkaBhwicoh/7D2FenML4sJ64aaBoXKXQ0QehgGHiHqc1Sphff4JAED6eN4aTkTOx4BDRD1u+09VKD7TAH8fL/xmNG8NJyLnY8Ahoh73XtvEftPHRqOX1kveYojIIzHgEFGPKqqsx46fqiAIwKykGLnLISIPxYBDRD1qfdvRm4lDItAvRCdvMUTksRhwiKjHnGtoxj/2td4a/vsbYuQthog8GgMOEfWY13OPorHZgqGRAUgaECJ3OUTkwRhwiKhHHK+qxwe7Wm8Nf2ZyPASBt4YTkXwYcIioRyz/8jBarBJuGxKOGwdxYj8ikhcDDhFdtfxjZ2H4sQJqlYBnJg+RuxwiIgYcIro6VquE57/4EQBw77h+GBjuL3NFREQMOER0lf65vwwHy03w13phXvIgucshIgLAgENEV6GxuQWvbD0MAMi8bSBC/LQyV0RE1IoBh4iu2Ns7jqPCZEZ0sC9mj4+RuxwiIhsGHCK6IhWmJvxl+3EAwFNpQ+CjUctcERHRzxhwiOiKrNh6BOdFC0b3C8KU4ZFyl0NEZMehAae6uhozZ85EQEAAgoKCcP/996O+vv6i40tKSiAIQqdfH330kW1cZ69v2rTJkbtCRBf4oawWH7c9kuHZXw3lpH5E5HK8HLnxmTNn4vTp0zAYDBBFEenp6Zg7dy42btzY6fjo6GicPn3abt3bb7+NV155Bbfffrvd+vfeew9paWm25aCgoB6vn4g6kiQJL3xxCJIE3DEyCqP79Za7JCKiDhwWcA4dOoScnBzs2bMHY8eOBQCsWrUKkydPxooVKxAVFdXhPWq1Gnq93m7dJ598gnvuuQd+fn5264OCgjqMJSLHyz1UifzjZ+HtpcL8tMFyl0NE1CmHBZz8/HwEBQXZwg0AJCcnQ6VSoaCgAHfeeedlt7F3714UFhZi9erVHV575JFH8MADDyAuLg4PPvgg0tPTL3qY3Gw2w2w225ZNJhMAQBRFiKLY3V1TpPY+sB/O4a79Fi1WvNA2qV96Un9E+GncYh/ctd/uiv12Lk/qd3f20WEBx2g0Ijw83P6XeXkhODgYRqOxS9t49913ER8fj/Hjx9utX7JkCW677TbodDps27YNDz/8MOrr6/HYY491up3ly5dj8eLFHdZv27YNOp2ui3vkGQwGg9wleBR36/eO0wKKz6rh5yUhrukotmw5KndJ3eJu/XZ37LdzeUK/Gxsbuzy22wHn6aefxksvvXTJMYcOHeruZjs4f/48Nm7ciAULFnR47cJ1o0aNQkNDA1555ZWLBpzs7GxkZWXZlk0mE6Kjo5GSkoKAgICrrlUJRFGEwWDApEmToNFo5C5H8dyx37XnRTy38hsAIuZPHorfXBctd0ld5o79dmfst3N5Ur/bz8B0RbcDzhNPPIE5c+ZcckxcXBz0ej0qKyvt1re0tKC6urpL1858/PHHaGxsxKxZsy47NjExEUuXLoXZbIZW23EmVa1W2+l6jUaj+A9Dd7EnzuUu/ZYkCc9+egDnGkUMCvfDvYkx8FK73ywT7tJvpWC/ncsT+t2d/et2wAkLC0NYWNhlxyUlJaGmpgZ79+7FmDFjAABff/01rFYrEhMTL/v+d999F3fccUeXfldhYSF69+7daYghoqv33rclyDlohEYt4JW7R7pluCEiz+Kwa3Di4+ORlpaGjIwMrFmzBqIoIjMzEzNmzLDdQVVWVoaJEyfi/fffx7hx42zvLSoqwo4dO7Bly5YO2/38889RUVGB66+/Hj4+PjAYDFi2bBn++Mc/OmpXiDzavtJzWLal9bTz/06OR0J0kLwFERF1gUPnwdmwYQMyMzMxceJEqFQq3HXXXXjjjTdsr4uiiCNHjnS4aGjt2rXo27cvUlJSOmxTo9Fg9erVePzxxyFJEgYOHIhXX30VGRkZjtwVIo90rqEZmRv2ocUqYcrwSD5viojchkMDTnBw8EUn9QOAmJgYSJLUYf2yZcuwbNmyTt+TlpZmN8EfETmG1Srh8f8rRHltE2JDe+HFu4ZzxmIichs8kU5EnXpr+zHkHamC1kuF1feOhr+Psi9eJCJlYcAhog7yj53Fn7YdAQAsmXothkZxOgUici8MOERkp7KuCY/+fT+sEnDX6L64Z6z7zHdDRNSOAYeIbCxWCY/9fT/O1JtxTYQflk67ltfdEJFbYsAhIpvXDD9h1/Fq9PJW488zx0Dn7dD7EIiIHIYBh4gAAP8+Uok3/10EAFj2m+EYGO4nc0VERFeOAYeIUFZzHlkfFgIA7ru+H6Ym9JG3ICKiq8Tjz0Qe7mR1I2b+tQDnGkUM7xOIBb8aKndJRERXjQGHyIMVVdZh5l8LUGEyo1+wDm/dNxpaL7XcZRERXTUGHCIP9UNZLWat3Y3qhmYMCvfDBw8kIiLAR+6yiIh6BAMOkQfaU1KN37+3B3XmFozoG4h16eMQ3Mtb7rKIiHoMAw6Rh9nxUxXm/u2/aBKtGBcbjHdnj+VjGIhIcRhwiDxIzg9GPPb3/Wi2WHHLNWFYc98Y+HrzmhsiUh4GHCIP8c99p/DkxwdgsUqYPFyPldNHwduLM0UQkTIx4BB5gL/ll2DBpwcBAL8d0xcv/mY4vNQMN0SkXAw4RArW3GLFyq9+wp/zjgEA5oyPwcJfDYVKxedLEZGyMeAQKdQPZbX440ff4bCxDgDw6G0DkTXpGj48k4g8AgMOkcKYWyxYlVuEt7Yfg8UqIbiXN5ZMvRa/GhEld2lERE7DgEOkIAdO1eCPH32HnyrqAQBTRkRiyR3XIsRPK3NlRETOxYBDpABNogWv5x7F2zuOw2KVEOrnjaVTh+H24ZFyl0ZEJAsGHCI3t7/0HJ78+ACKKluP2twxMgrP3XEtZyYmIo/GgEPkpsprzuPtHcfxfn4JrBIQ6qfF89OGIW2YXu7SiIhkx4BD5GYOnTbhnR3H8dl35WixSgCAO0f1wcJfDUVvHrUhIgLAgEPkFiRJQv7xs/jL9uPY/lOVbX1SXAgevnUAbhoUJmN1RESuhwGHyIW1WKz48gcj3t5xHN+X1QIAVAJw+/BI/L+b4zCib5C8BRIRuSgGHCIXdLr2PLZ8b8S6ncU4WX0eAOCjUeGesdF44MY49AvRyVwhEZFrY8AhcgGSJKGosh7bfqzA1oNGHDhVa3utt06D2eNjMCsphndGERF1EQMOkUysErD/ZA1yj5yB4WAFjp9psL0mCMCYfr0xNSEKvx0TDV9vtYyVEhG5HwYcIiexWiUUn21AYWkNdhefxZffqWHatdv2urdahRsGhiDlWj2S4yMQ5s/Zh4mIrhQDDpGDnKk3o7C0Bt+dqkHhyRp8d7IGpqaWC0YI8NN64bYh4Ui5NgK3XBMGfx+NbPUSESkJAw7RVaptFFF8tgElZxpQfKYBRVX1+O5kDU6dO99hrNZLheF9AjG8TwC01ceROT0Zfr48UkNE1NMcFnBeeOEFfPHFFygsLIS3tzdqamou+x5JkrBo0SK88847qKmpwQ033IC33noLgwYNso2prq7Go48+is8//xwqlQp33XUXXn/9dfj5+TlqV8jDNYkWVNWZUVnXhPKaptYgc0GgOdcodvo+QQAGhvlhZHQQEtq+Buv9oVGrIIoitmw5Bq2Xysl7Q0TkGRwWcJqbm3H33XcjKSkJ7777bpfe8/LLL+ONN97A+vXrERsbiwULFiA1NRU//vgjfHx8AAAzZ87E6dOnYTAYIIoi0tPTMXfuXGzcuNFRu0IK0yRaYDovovYXX9UNzaisM6PS1NT6ve1n+9NKnQv31yImtBdiQ3ohNqxX61GavoEI4CknIiJZOCzgLF68GACwbt26Lo2XJAkrV67Es88+i6lTpwIA3n//fURERGDz5s2YMWMGDh06hJycHOzZswdjx44FAKxatQqTJ0/GihUrEBUV5ZB9IcezWCWIFitarBJaLFaIFgktVivMohXNltbv5hYLmlusMNu+LDC3WHG+2YLGZgvON7eg4Rc/t77WAlNTC2rPizCdF2FusXa7Pm8vFcL9tdAH+KB/SC/EhupaA01oL8SE9EIvLc/2EhG5Epf5q1xcXAyj0Yjk5GTbusDAQCQmJiI/Px8zZsxAfn4+goKCbOEGAJKTk6FSqVBQUIA777yz022bzWaYzWbbsslkAgCIoghR7Pz0wpXYV1qDL7432q2TOhsoSZ2+Lv1isNT2avt6qdNxEiTJ/jWpbZ3tPdLPYy58XWobIEGCxWpFhVGFf53bDwgCJKn1t1vb39s2ziq13g1kldp+bvtusUqQLvjZYpVgkSRYrRJa2sa3WFuXLVLr6y0WCWJbsPnlvjuaIAABPl4I8NEg0FeDAF8v9NZ5I9xfizB/b4T7aRHm3/oV7q9FgI8XBEG4yNakbn+O2sf35OePLo79di7227k8qd/d2UeXCThGY2swiIiIsFsfERFhe81oNCI8PNzudS8vLwQHB9vGdGb58uW2I0oX2rZtG3S6npsRdmeFgA+Pu/N8JSqguuryw5xELUjwUgEaAfBStX0JgMb2c+vrWhXgrf75u7dKglbd/nPrl68XoPOS4KsGdF6AVg2ohBYATfa/VAJgav2qQevXUQfuo8FgcODW6ZfYb+div53LE/rd2NjY5bHdCjhPP/00XnrppUuOOXToEIYMGdKdzTpcdnY2srKybMsmkwnR0dFISUlBQEBAj/2evqdqEXy4ssN6AR3/y//CgwHCRdbbvdf+m93RBOGC9wm/fE34+bsAoe1765gLf7ZaLDhy5DCGxsfDy0sNAQJUQtv7BMH2O9SCAEFofU2t+vlnlSBApfr5Z7VKgJdKgEpo+97ZslqARiVAo1bBSy3AS6WCRt36evu2lUoURRgMBkyaNAkaDa/TcTT227nYb+fypH63n4Hpim4FnCeeeAJz5sy55Ji4uLjubNJGr9cDACoqKhAZGWlbX1FRgYSEBNuYykr7ANHS0oLq6mrb+zuj1Wqh1Xa8FVej0fToh2FMbCjGxIb22PacSRRFbKk9hMlJMYr/B+JKevozSJfGfjsX++1cntDv7uxftwJOWFgYwsLCul1QV8TGxkKv1yM3N9cWaEwmEwoKCvDQQw8BAJKSklBTU4O9e/dizJgxAICvv/4aVqsViYmJDqmLiIiI3I/DJuEoLS1FYWEhSktLYbFYUFhYiMLCQtTX19vGDBkyBJ988gmA1tMg8+bNw/PPP4/PPvsM33//PWbNmoWoqChMmzYNABAfH4+0tDRkZGRg9+7d+Pbbb5GZmYkZM2bwDioiIiKycdhFxgsXLsT69etty6NGjQIA/Pvf/8aECRMAAEeOHEFt7c9PTZ4/fz4aGhowd+5c1NTU4MYbb0ROTo5tDhwA2LBhAzIzMzFx4kTbRH9vvPGGo3aDiIiI3JDDAs66desuOweO9It7gwVBwJIlS7BkyZKLvic4OJiT+hEREdElcZ54IiIiUhwGHCIiIlIcBhwiIiJSHAYcIiIiUhwGHCIiIlIcBhwiIiJSHAYcIiIiUhwGHCIiIlIcBhwiIiJSHIfNZOzK2mdQ7s5j15VOFEU0NjbCZDIp/mm0roD9di7227nYb+fypH63///2L5+E0BmPDDh1dXUAgOjoaJkrISIiou6qq6tDYGDgJccIUldikMJYrVaUl5fD398fgiDIXY5LMJlMiI6OxsmTJxEQECB3OYrHfjsX++1c7LdzeVK/JUlCXV0doqKioFJd+iobjzyCo1Kp0LdvX7nLcEkBAQGK/wfiSthv52K/nYv9di5P6ffljty040XGREREpDgMOERERKQ4DDgEANBqtVi0aBG0Wq3cpXgE9tu52G/nYr+di/3unEdeZExERETKxiM4REREpDgMOERERKQ4DDhERESkOAw4REREpDgMOHRRZrMZCQkJEAQBhYWFcpejSCUlJbj//vsRGxsLX19fDBgwAIsWLUJzc7PcpSnG6tWrERMTAx8fHyQmJmL37t1yl6RIy5cvx3XXXQd/f3+Eh4dj2rRpOHLkiNxleYwXX3wRgiBg3rx5cpfiMhhw6KLmz5+PqKgouctQtMOHD8NqteIvf/kLDh48iNdeew1r1qzBM888I3dpivDhhx8iKysLixYtwr59+zBy5EikpqaisrJS7tIUZ/v27XjkkUewa9cuGAwGiKKIlJQUNDQ0yF2a4u3Zswd/+ctfMGLECLlLcSm8TZw69eWXXyIrKwv/+Mc/cO2112L//v1ISEiQuyyP8Morr+Ctt97C8ePH5S7F7SUmJuK6667Dm2++CaD1OXTR0dF49NFH8fTTT8tcnbJVVVUhPDwc27dvx8033yx3OYpVX1+P0aNH489//jOef/55JCQkYOXKlXKX5RJ4BIc6qKioQEZGBv72t79Bp9PJXY7Hqa2tRXBwsNxluL3m5mbs3bsXycnJtnUqlQrJycnIz8+XsTLPUFtbCwD8LDvYI488gilTpth9zqmVRz5sky5OkiTMmTMHDz74IMaOHYuSkhK5S/IoRUVFWLVqFVasWCF3KW7vzJkzsFgsiIiIsFsfERGBw4cPy1SVZ7BarZg3bx5uuOEGDBs2TO5yFGvTpk3Yt28f9uzZI3cpLolHcDzE008/DUEQLvl1+PBhrFq1CnV1dcjOzpa7ZLfW1X5fqKysDGlpabj77ruRkZEhU+VEV++RRx7BDz/8gE2bNsldimKdPHkSf/jDH7Bhwwb4+PjIXY5L4jU4HqKqqgpnz5695Ji4uDjcc889+PzzzyEIgm29xWKBWq3GzJkzsX79ekeXqghd7be3tzcAoLy8HBMmTMD111+PdevWQaXif3tcrebmZuh0Onz88ceYNm2abf3s2bNRU1ODTz/9VL7iFCwzMxOffvopduzYgdjYWLnLUazNmzfjzjvvhFqttq2zWCwQBAEqlQpms9nuNU/EgEN2SktLYTKZbMvl5eVITU3Fxx9/jMTERPTt21fG6pSprKwMt956K8aMGYMPPvjA4/8o9aTExESMGzcOq1atAtB66qRfv37IzMzkRcY9TJIkPProo/jkk0+Ql5eHQYMGyV2SotXV1eHEiRN269LT0zFkyBA89dRTPDUIXoNDv9CvXz+7ZT8/PwDAgAEDGG4coKysDBMmTED//v2xYsUKVFVV2V7T6/UyVqYMWVlZmD17NsaOHYtx48Zh5cqVaGhoQHp6utylKc4jjzyCjRs34tNPP4W/vz+MRiMAIDAwEL6+vjJXpzz+/v4dQkyvXr0QEhLCcNOGAYdIRgaDAUVFRSgqKuoQIHlw9epNnz4dVVVVWLhwIYxGIxISEpCTk9PhwmO6em+99RYAYMKECXbr33vvPcyZM8f5BZHH4ykqIiIiUhxeyUhERESKw4BDREREisOAQ0RERIrDgENERESKw4BDREREisOAQ0RERIrDgENERESKw4BDREREisOAQ0RERIrDgENERESKw4BDREREisOAQ0RERIrz/wHoJ4OxMzmLfQAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(np.arange(-5,5,0.2), np.tanh(np.arange(-5,5,0.2))); plt.grid();\n",
"#called on a range from -5 to 5 with a slope of -2."
]
},
{
"cell_type": "markdown",
"id": "e2f42787-d0cd-46e3-bc19-d85e8ffedb3d",
"metadata": {},
"source": [
"- Inputs as they come in get squashed on the y coordinate, only goes up to 1 and then plateaus. \n",
"- We cap it smoothly to 1 and -1. This is this is an activation (squashing) function. \n",
"- What comes out of this neuron is the activation applied to the dot product of the weights and the inputs (see cell image)"
]
},
{
"cell_type": "code",
"execution_count": 110,
"id": "5b903cb3-027f-425d-8096-ffab56e166fa",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1257pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1256.75 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1252.75,-206 1252.75,4 -4,4\"/>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732342992+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732342992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4732342992+ -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4736242704->4732342992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4737919632 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4737919632</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-110.5 324,-146.5 542.25,-146.5 542.25,-110.5 324,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-111 376.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-111 461,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732347344+ -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732347344+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4737919632->4732347344+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4737919632->4732347344+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-110.69C551.12,-109.24 559.54,-107.86 567.19,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.72,-110.06 577.02,-104.98 566.58,-103.15 567.72,-110.06\"/>\n",
"</g>\n",
"<!-- 4737919632* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4737919632*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4737919632*->4737919632 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4737919632*->4737919632</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C295.29,-128.5 303.43,-128.5 312.17,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-132 322.01,-128.5 312.01,-125 312.01,-132\"/>\n",
"</g>\n",
"<!-- 4737348240 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4737348240</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734231440* -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4734231440*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4737348240->4734231440* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737348240->4734231440*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4732342992 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4732342992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732342992+->4732342992 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4732342992+->4732342992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4730404560 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4730404560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-0.5 2.25,-36.5 195.75,-36.5 195.75,-0.5 2.25,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-1 34.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-1 114.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730404560->4734231440* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4730404560->4734231440*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4734231440 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4734231440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-55.5 326.25,-91.5 540,-91.5 540,-55.5 326.25,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-56 379,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-56 458.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734231440->4732347344+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4734231440->4732347344+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-90.37C549.89,-91.87 559,-93.32 567.21,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.63,-98.07 577.05,-96.18 567.72,-91.16 566.63,-98.07\"/>\n",
"</g>\n",
"<!-- 4734231440*->4734231440 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734231440*->4734231440</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C296,-73.5 305.08,-73.5 314.82,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-77 324.55,-73.5 314.55,-70 314.55,-77\"/>\n",
"</g>\n",
"<!-- 4737726352 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4737726352</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-165.5 0,-201.5 198,-201.5 198,-165.5 0,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-166 32.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-166 116.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737726352->4737919632* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4737726352->4737919632*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4737717712 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737717712</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737717712->4737919632* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4737717712->4737919632*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"<!-- 4732347344 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4732347344</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732347344->4732342992+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4732347344->4732342992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4732347344+->4732347344 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4732347344+->4732347344</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a2a93d0>"
]
},
"execution_count": 110,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# inputs x1,x2; two dimensional neuron (two weights come in)\n",
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"\n",
"# weights w1,w2; the synaptec strengths of each input\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"\n",
"# bias of the neuron b\n",
"b = Value(6.8813735870195432, label='b')\n",
"\n",
"#according to the model above, we multiply x1 * w1 and x2 * w2 + b. \n",
"# x1*w1 + x2*w2 + b; doing it in small steps so we have pointers to the intermediate steps\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"\n",
"#n is now the cell body raw activation sum without the raw activation function\n",
"n = x1w1x2w2 + b; n.label = 'n' \n",
"draw_dot(n)"
]
},
{
"cell_type": "markdown",
"id": "69910132-7991-4d0a-a363-8b7cefa53785",
"metadata": {},
"source": [
"Now, we're going to take it through activation function tanh to get the output."
]
},
{
"cell_type": "code",
"execution_count": 111,
"id": "4fd999df-31e9-48bc-b735-d54456147b14",
"metadata": {},
"outputs": [],
"source": [
"import math"
]
},
{
"cell_type": "code",
"execution_count": 112,
"id": "d547d069-2849-4aaa-bafc-1521e29a662b",
"metadata": {},
"outputs": [],
"source": [
"# inputs x1,x2; two dimensional neuron (two weights come in)\n",
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"b = Value(6.8813735870195432, label='b')\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"n = x1w1x2w2 + b; n.label = 'n'\n",
"\n",
"\n",
"#o = n.tanh(); o.label = 'o'\n",
"\n",
"#draw_dot(o)\n",
"\n",
"#output would be an error because we haven't written the tanh activation function yet. Implement this tanh function because tanh is a hyperbolic function,\n",
"#and we only have so far implemented an addition function and a multiplication function. We can't make a tanh \n",
"#out of just addition and multiplication; we need to add in exponentiation and division functionalities"
]
},
{
"cell_type": "markdown",
"id": "e4e8343a-1d88-4542-812d-83aad3e78e66",
"metadata": {},
"source": [
"Here is the hyperbolic tangent function we will use for the activation:\n",
"\n",
"$$tanh(x)={\\frac {\\sinh x}{\\cosh x}}={\\frac {e^{x}-e^{-x}}{e^{x}+e^{-x}}}={\\frac {e^{2x}-1}{e^{2x}+1}}$$\n",
"\n",
"More on this is available at: https://en.wikipedia.org/wiki/Hyperbolic_functions\n",
"\n",
"Now let's implement ```tanh()``` in the ```Value``` class. Since we don't necessarily need the most atomic pieces for this function, we can create functions at arbitrary points of abstraction. The only thing that matters is that we know how to differentiate (create the local derivative (how the inputs impact the output)), that's all you need.\n",
"\n",
"We could create a ```exp(self)``` function. The line of thinking for this would be that we already have addition and multiplication, so all we would need to do is make an exponent function and then use them all in conjunction with each other for the hyperbolic tangent formula above. But since the above is true, we will directly implement ```tanh()``` instead of breaking it up into its atomic pieces. \n",
"\n",
"Notice the ```tanh()``` implementation in the Value class:\n"
]
},
{
"cell_type": "code",
"execution_count": 113,
"id": "bcc706d8-da0c-40a1-a67e-b87537d3354b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Value(data=-8.0)"
]
},
"execution_count": 113,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data\n",
" self._prev = set(_children)\n",
" self._op = _op\n",
" self.label = label\n",
" self.grad = 0.0\n",
" \n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data, (self, other), '+')\n",
" return out\n",
" \n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data, (self, other), '*')\n",
" return out\n",
"\n",
" #def exp(self):\n",
" #one option is to implement this exp function for the tanh, because this is the only thing we need for tanh besides add and multiply.\n",
"\n",
" def tanh(self):\n",
" x = self.data\n",
" t = (math.exp(2*x)-1)/(math.exp(2*x)+1)\n",
" \n",
" #children of node, one output. Self is a tuple:\n",
" out = Value(t, (self, ), 'tanh')\n",
" return out\n",
"\n",
"a = Value(2.0, label='a')\n",
"b = Value(-3.0, label='b')\n",
"c = Value(10.0, label='c') #added labels\n",
"e = a*b; e.label = 'e' #added e for a * b\n",
"d = e + c; d.label = 'd'\n",
"f = Value(-2.0, label = 'f')\n",
"L = d*f; label = 'L'\n",
"L"
]
},
{
"cell_type": "markdown",
"id": "826348a7-2693-45cc-8bd5-807975cbd67b",
"metadata": {},
"source": [
"Now, we should be able to do ```n.tanh()```:"
]
},
{
"cell_type": "code",
"execution_count": 114,
"id": "200d4fdc-ed53-4b8e-a4a3-7276f9d51384",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4732402704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4732402704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733586768+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733586768+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4732402704->4733586768+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4732402704->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4737397584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4737397584+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4736242704->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4736242704* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4736242704*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4736242704*->4736242704 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4736242704*->4736242704</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733444560 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733444560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733444560->4737397584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733444560->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733444560* -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733444560*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733444560*->4733444560 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733444560*->4733444560</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4732394256 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732394256</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732394256->4736242704* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4732394256->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4731908880 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4731908880</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731908880->4736242704* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4731908880->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"<!-- 4731903248 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731903248</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731903248->4733444560* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4731903248->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4733586768 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733586768</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733446096tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733586768->4733446096tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733586768->4733446096tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4733586768+->4733586768 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733586768+->4733586768</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4737397584 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737397584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584->4733586768+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737397584->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4737397584+->4737397584 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4737397584+->4737397584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4732403600 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732403600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732403600->4733444560* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4732403600->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4733446096 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4733446096</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh->4733446096 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4733446096tanh->4733446096</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a5ef8d0>"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"b = Value(6.8813735870195432, label='b') #weird number used for nice output... try with b = Value(8.0)\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"n = x1w1x2w2 + b; n.label = 'n'\n",
"\n",
"o = n.tanh(); o.label = 'o' #we've now implemented tanh(). As long as we know the derivative of tanh, we can backprop through it.\n",
"#let's visualize it. tanh is squashing our output n to a value between -1 and 1:\n",
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "d5e2b4dd-fa32-41ba-891e-186f51a08ad7",
"metadata": {},
"source": [
"What is the derivative of ```o``` with respect to all of the inputs?\n",
"\n",
"We care about the derivative of the weights of the neurons specifically ($w2$ and $w1$), because that's what we'll be changing for all the neurons in the optimization.\n",
"\n",
"Eventually there's a loss function we'll use that measures the accuracy of the neural net, which we'll try to improve with backpropogration.\n",
"\n",
"Let's start with backprop at the end. What is the derivative of ```o``` with respect to ```o``` (the base case)? It's always $1.0$\n"
]
},
{
"cell_type": "code",
"execution_count": 115,
"id": "0fcfd822-49bb-4715-98f6-8ea7dab0f8b5",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4732402704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4732402704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733586768+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733586768+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4732402704->4733586768+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4732402704->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4737397584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4737397584+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4736242704->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4736242704* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4736242704*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4736242704*->4736242704 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4736242704*->4736242704</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733444560 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733444560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733444560->4737397584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733444560->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733444560* -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733444560*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733444560*->4733444560 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733444560*->4733444560</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4732394256 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732394256</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732394256->4736242704* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4732394256->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4731908880 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4731908880</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731908880->4736242704* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4731908880->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"<!-- 4731903248 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731903248</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731903248->4733444560* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4731903248->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4733586768 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733586768</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733446096tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733586768->4733446096tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733586768->4733446096tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4733586768+->4733586768 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733586768+->4733586768</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4737397584 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737397584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584->4733586768+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737397584->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4737397584+->4737397584 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4737397584+->4737397584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4732403600 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732403600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732403600->4733444560* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4732403600->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4733446096 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4733446096</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh->4733446096 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4733446096tanh->4733446096</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a5c5c10>"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"o.grad = 1.0\n",
"#now, we see o.grad is 1\n",
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "a000cd7d-ce7e-4cc4-a226-810715bc1494",
"metadata": {},
"source": [
"Now, we backprop through ```tanh()```. To do this, we need the derivative of ```tanh()```.\n",
"If we go back to the wiki site and find the derivative function for this:\n",
"https://en.wikipedia.org/wiki/Hyperbolic_functions\n",
"\n",
"Then, in the third row of derivatives, we are greeted with:\n",
"\n",
"$${\\frac{d}{dx}tanh(x)={1- {tanh}^2(x)}}$$\n"
]
},
{
"cell_type": "code",
"execution_count": 116,
"id": "9c30e0bc-b3ba-435c-b8bd-50bcf9865282",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.4999999999999999"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#o = tanh(n)\n",
"#what is do/dn? -> the above is saying that:\n",
"#do/dn = 1 - o**2\n",
"1 - o.data**2 # this means the local derivative of the tanh() operation is 0.5. We set it and redraw:"
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "4fe0575f-cb4f-49e6-8325-28e356159d02",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4732402704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4732402704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733586768+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733586768+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4732402704->4733586768+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4732402704->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4737397584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4737397584+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4736242704->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4736242704* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4736242704*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4736242704*->4736242704 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4736242704*->4736242704</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733444560 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733444560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733444560->4737397584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733444560->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733444560* -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733444560*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733444560*->4733444560 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733444560*->4733444560</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4732394256 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732394256</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732394256->4736242704* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4732394256->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4731908880 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4731908880</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731908880->4736242704* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4731908880->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"<!-- 4731903248 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731903248</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731903248->4733444560* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4731903248->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4733586768 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733586768</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733446096tanh -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733446096tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733586768->4733446096tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733586768->4733446096tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4733586768+->4733586768 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733586768+->4733586768</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4737397584 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737397584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4737397584->4733586768+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737397584->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4737397584+->4737397584 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4737397584+->4737397584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4732403600 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732403600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732403600->4733444560* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4732403600->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4733446096 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4733446096</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh->4733446096 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4733446096tanh->4733446096</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a5799d0>"
]
},
"execution_count": 117,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"n.grad = 0.5\n",
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "e8af2221-6db1-4ab9-b2db-7a520eb6429c",
"metadata": {},
"source": [
"Let's continue with backpropogation from $n$ to $b + x1w1 + x2w2$. \n",
"\n",
"Remember that a $+$ operator is an even distributor of the gradient in backpropogation. The gradient flows to $b$ and $x$ evenly, so we will set those below. Then, we meet another $+$ operator that adds $x2w2$ and $x1w1$. This will also distribute $x1w1+x2w2$ evenly.\n",
"\n",
"We set them and redraw:"
]
},
{
"cell_type": "code",
"execution_count": 118,
"id": "8e4e162d-9523-4fc7-aea5-de3de695d8ac",
"metadata": {},
"outputs": [],
"source": [
"x1w1x2w2.grad = 0.5\n",
"b.grad = 0.5\n",
"x2w2.grad = 0.5\n",
"x1w1.grad = 0.5"
]
},
{
"cell_type": "code",
"execution_count": 119,
"id": "2a74fbbb-e3d3-4bea-8f6e-24bb718b275e",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4732402704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4732402704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733586768+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733586768+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4732402704->4733586768+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4732402704->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4737397584+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4737397584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4737397584+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4736242704->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4736242704* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4736242704*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4736242704*->4736242704 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4736242704*->4736242704</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733444560 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733444560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733444560->4737397584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733444560->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733444560* -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733444560*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733444560*->4733444560 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733444560*->4733444560</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4732394256 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732394256</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732394256->4736242704* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4732394256->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4731908880 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4731908880</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731908880->4736242704* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4731908880->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"<!-- 4731903248 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731903248</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731903248->4733444560* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4731903248->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4733586768 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733586768</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733446096tanh -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733446096tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733586768->4733446096tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733586768->4733446096tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4733586768+->4733586768 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733586768+->4733586768</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4737397584 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737397584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4737397584->4733586768+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737397584->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4737397584+->4737397584 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4737397584+->4737397584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4732403600 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732403600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4732403600->4733444560* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4732403600->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4733446096 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4733446096</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh->4733446096 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4733446096tanh->4733446096</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a645e10>"
]
},
"execution_count": 119,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "3a844fba-04f7-4c2f-9289-fef5bc4fe98f",
"metadata": {},
"source": [
"Now, we have $0.5$ flowing into the final multiplication operations. If we want to calculate ```x2.grad```, it will be ```x2.grad = w2.data * x2w2.grad```. Then, ```w2.grad = x2.data * x2w2.grad```. This is this the chain rule (refer back to the de/da example in the second lecture if help is needed)"
]
},
{
"cell_type": "code",
"execution_count": 122,
"id": "fe29794e-d62b-4e14-b496-9b0c87e0970e",
"metadata": {},
"outputs": [],
"source": [
"#derivative always tells us the influence of data on the final output.\n",
"#if we want the output of this neuron to increase, the influence of these expressions on the output is positive.\n",
"\n",
"#If I wiggle w2, how is the output changing? it's not, because it's getting multiplied by 0 (grad = 0). Because it's not changing,\n",
"#there is no derivative, thus w2 grad = 0.\n",
"\n",
"#the local derivative of * with respect to x1 is w1. \n",
"x1.grad = w1.data * x1w1.grad \n",
"w1.grad = x1.data * x1w1.grad\n",
"x2.grad = w2.data * x2w2.grad\n",
"w2.grad = x2.data * x2w2.grad"
]
},
{
"cell_type": "code",
"execution_count": 123,
"id": "8325199a-028f-427c-aaa1-397285240e09",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4732402704 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4732402704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733586768+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733586768+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4732402704->4733586768+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4732402704->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4736242704 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4736242704</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4737397584+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4737397584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4736242704->4737397584+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4736242704->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4736242704* -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4736242704*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4736242704*->4736242704 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4736242704*->4736242704</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733444560 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733444560</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733444560->4737397584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733444560->4737397584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733444560* -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733444560*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733444560*->4733444560 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733444560*->4733444560</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4732394256 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4732394256</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4732394256->4736242704* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4732394256->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4731908880 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4731908880</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731908880->4736242704* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4731908880->4736242704*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"<!-- 4731903248 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731903248</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4731903248->4733444560* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4731903248->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4733586768 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733586768</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733446096tanh -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733446096tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733586768->4733446096tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733586768->4733446096tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4733586768+->4733586768 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733586768+->4733586768</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4737397584 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4737397584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4737397584->4733586768+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4737397584->4733586768+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4737397584+->4737397584 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4737397584+->4737397584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4732403600 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4732403600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-0.5 1.5,-36.5 196.5,-36.5 196.5,-0.5 1.5,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"31,-1 31,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"70.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-1 110.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -1.5000</text>\n",
"</g>\n",
"<!-- 4732403600->4733444560* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4732403600->4733444560*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4733446096 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4733446096</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4733446096tanh->4733446096 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4733446096tanh->4733446096</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a242090>"
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "e655a65e-1c67-459d-ad73-d97112de5d7f",
"metadata": {},
"source": [
"Now, we will see how we can equip the backward pass automatically instead of manually! We revisit the Value object and making ```backward()``` functions that will do the chain rule at each local node. It will chain the output gradient into the input gradients. We will not implement this in the constructor, but rather in the ```__add__```, ```__mul__```, and ```tanh()``` functions themselves. Read the comments for explanations, and then reinitialize the Value class by running it again."
]
},
{
"cell_type": "code",
"execution_count": 273,
"id": "269d2fa4-c672-46fa-9313-c19ff973d70b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Value(data=-8.0)"
]
},
"execution_count": 273,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''): \n",
" self.data = data\n",
" self.grad = 0.0 \n",
" self._prev = set(_children)\n",
" self._backward = lambda : None\n",
" #^fucntion that does piece of chain rule at each little node that took inputs and produced outputs, store \n",
" #how we are going to chain the output gradient into the input gradients\n",
" #by default it doesn't do anything -> this way for a leaf node there's nothing to do. But, we will implement it in the other functions.\n",
" self._op = _op\n",
" self.label=label\n",
"\n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data, (self, other), '+') #feed in children of the value\n",
" \n",
" #out.grad will be propogated into self.grad and others.grad for addition\n",
" def _backward(): \n",
" self.grad + 1.0 * out.grad #addition -> local derivative of self * the global derivative\n",
" other.grad + 1.0 * out.grad #addition chain rule\n",
" \n",
" out._backward = _backward #don't call the function like out._backward = _backward(), because that returns none -> just store/save it\n",
" return out\n",
" \n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data, (self, other), '*')\n",
"\n",
" def _backward():\n",
" self.grad = other.data * out.grad #local derivative * out.grad(global derivative)\n",
" other.grad = self.data * out.grad \n",
" out._backward = _backward\n",
" return out\n",
" \n",
" def tanh(self):\n",
" x = self.data\n",
" t = (math.exp(2*x)-1)/(math.exp(2*x)+1)\n",
" \n",
" out = Value(t, (self, ), 'tanh')\n",
"\n",
" def _backward():\n",
" #we have out.grad, we want to chain it into self.grad\n",
" #self.grad will be the local derivative of this operation above (tanh).\n",
" #The gradient has to be multiplied because of the chain rule; out.grad is chained through the local grad into self.grad\n",
" #1-tanh(x**2) -> \n",
" #1 - t^2 is the local derivative (t is tanh).\n",
" self.grad = (1-t**2) * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
"a = Value(2.0, label='a')\n",
"b = Value(-3.0, label='b')\n",
"c = Value(10.0, label='c')\n",
"e = a * b; e.label='e'\n",
"d = e + c; d.label='d'\n",
"f = Value(-2.0, label='f')\n",
"L = d * f; L.label = 'L'\n",
"L"
]
},
{
"cell_type": "markdown",
"id": "d9ece6e0-f5fd-4303-8c60-de40f46a1ada",
"metadata": {},
"source": [
"Now, we reinitialize the variables for our neural net:"
]
},
{
"cell_type": "code",
"execution_count": 211,
"id": "a008a58f-1004-4ac5-90e4-2c3fdea55fc8",
"metadata": {},
"outputs": [],
"source": [
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"b = Value(6.8813735870195432, label='b') #weird number used for nice output... try with b = Value(8.0)\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"n = x1w1x2w2 + b; n.label = 'n'\n",
"\n",
"o = n.tanh(); o.label = 'o'"
]
},
{
"cell_type": "code",
"execution_count": 212,
"id": "fd8662e7-b03d-4695-82cf-c8d7f5e129d7",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aabc610>"
]
},
"execution_count": 212,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o) #make sure all the grads are 0 again"
]
},
{
"cell_type": "markdown",
"id": "845d1984-c4eb-40a7-923a-a88485fb5551",
"metadata": {},
"source": [
"Great! Now, we don't have to do the backward part manually anymore (i.e., ```x1.grad = w1.data * x1w1.grad```, etc.\n",
"\n",
"We initialize ```o.grad``` to 1 below because the default constructor initializes it to 0.0, which will cause self.grad to be 0 in the \n",
"```backward()``` ```tanh()``` function. "
]
},
{
"cell_type": "code",
"execution_count": 213,
"id": "196e4dc0-6e9f-4d9a-bd56-e31b9985cebb",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aaa9e10>"
]
},
"execution_count": 213,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"o.grad = 1.0\n",
"draw_dot(o)"
]
},
{
"cell_type": "code",
"execution_count": 214,
"id": "5cdb208b-c0dc-4951-adb4-31f10279a1e9",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aaabd50>"
]
},
"execution_count": 214,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"o._backward() #will propogate the o.grad through tanh(). The global derivative (init to 1) multiplied by the local derivative\n",
"draw_dot(o) #n.grad should be 0.5"
]
},
{
"cell_type": "markdown",
"id": "096bb4a4-bf17-471d-8d6b-ac7b33f6aa04",
"metadata": {},
"source": [
"Now, run the following and observe the changes in draw_dot(o) by re-running it after each block:"
]
},
{
"cell_type": "code",
"execution_count": 215,
"id": "f4d6449f-c68a-489c-9a9e-0a8f519b4089",
"metadata": {},
"outputs": [],
"source": [
"n._backward() #spreads gradient to both children (+)"
]
},
{
"cell_type": "code",
"execution_count": 216,
"id": "7d6469de-b7dc-41b3-95ba-6c6c1ba8dd51",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aa895d0>"
]
},
"execution_count": 216,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o)"
]
},
{
"cell_type": "code",
"execution_count": 218,
"id": "497c2212-e2c0-4600-acef-19d9c9b13b12",
"metadata": {},
"outputs": [],
"source": [
"b._backward() #b doesn't have a backward, because it's a leaf node. By init, it's the empty function"
]
},
{
"cell_type": "code",
"execution_count": 219,
"id": "bdd25a5f-f48e-4128-a67c-fa1b667311f2",
"metadata": {},
"outputs": [],
"source": [
"x1w1x2w2._backward() #0.5 is routed"
]
},
{
"cell_type": "code",
"execution_count": 220,
"id": "8d86f2c9-0211-4f48-af6e-3d0bdf5b00a5",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-55.5 3.75,-91.5 194.25,-91.5 194.25,-55.5 3.75,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-56 33.25,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-56 113,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-73.5C204.21,-73.5 213.66,-73.5 222.21,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-77 232.12,-73.5 222.12,-70 222.12,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aa8afd0>"
]
},
"execution_count": 220,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o)"
]
},
{
"cell_type": "code",
"execution_count": 221,
"id": "67edb666-5fdb-49bb-be87-cc1cec94d423",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4554886160 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4554886160</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-109.5 1374.75,-145.5 1558.5,-145.5 1558.5,-109.5 1374.75,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-110 1397.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-110 1477.25,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4554886160tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4554886160tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4554886160tanh->4554886160 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4554886160tanh->4554886160</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-127.5C1346.17,-127.5 1354.36,-127.5 1363.07,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-131 1372.81,-127.5 1362.81,-124 1362.81,-131\"/>\n",
"</g>\n",
"<!-- 4742494736 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4742494736</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742494992+ -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4742494992+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4742494736->4742494992+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4742494736->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4742494736* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4742494736*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742494736*->4742494736 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4742494736*->4742494736</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4734471760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4734471760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-109.5 1065,-145.5 1248.75,-145.5 1248.75,-109.5 1065,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-110 1087.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-110 1167.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734471760->4554886160tanh -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4734471760->4554886160tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-127.5C1257.39,-127.5 1265.52,-127.5 1272.98,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-131 1282.95,-127.5 1272.95,-124 1272.95,-131\"/>\n",
"</g>\n",
"<!-- 4734471760+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4734471760+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-127.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-122.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4734471760+->4734471760 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4734471760+->4734471760</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-127.5C1036.42,-127.5 1044.61,-127.5 1053.32,-127.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-131 1063.06,-127.5 1053.06,-124 1053.06,-131\"/>\n",
"</g>\n",
"<!-- 4742485584 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4742485584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-137.5 711.75,-173.5 895.5,-173.5 895.5,-137.5 711.75,-137.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-138 734.5,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-138 814.25,-173.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-150.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742485584->4734471760+ -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4742485584->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-142.48C919.97,-139.05 944.58,-135.55 963.93,-132.79\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.34,-136.26 973.75,-131.39 963.35,-129.33 964.34,-136.26\"/>\n",
"</g>\n",
"<!-- 4742489808 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4742489808</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-55.5 1.5,-91.5 196.5,-91.5 196.5,-55.5 1.5,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"31,-56 31,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"70.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-56 110.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -1.5000</text>\n",
"</g>\n",
"<!-- 4742484304* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4742484304*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4742489808->4742484304* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4742489808->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M196.76,-73.5C205.77,-73.5 214.47,-73.5 222.4,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.25,-77 232.25,-73.5 222.25,-70 222.25,-77\"/>\n",
"</g>\n",
"<!-- 4742494928 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4742494928</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 198,-36.5 198,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-1 32.5,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-1 116.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4742494928->4742484304* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4742494928->4742484304*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M168.91,-36.94C178.74,-39.93 188.67,-43.15 198,-46.5 207.96,-50.07 218.58,-54.47 228.18,-58.68\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.54,-61.78 237.1,-62.67 229.4,-55.39 226.54,-61.78\"/>\n",
"</g>\n",
"<!-- 4742481168 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4742481168</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-165.5 2.25,-201.5 195.75,-201.5 195.75,-165.5 2.25,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-166 34.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-166 114.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742481168->4742494736* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4742481168->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4742494992 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4742494992</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742494992->4734471760+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4742494992->4734471760+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M939.14,-118.99C947.95,-120.21 956.31,-121.36 963.87,-122.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.16,-125.83 973.55,-123.73 964.12,-118.9 963.16,-125.83\"/>\n",
"</g>\n",
"<!-- 4742494992+->4742494992 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4742494992+->4742494992</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4742484304 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4742484304</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742484304->4742494992+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4742484304->4742494992+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4742484304*->4742484304 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4742484304*->4742484304</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4742481360 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4742481360</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4742481360->4742494736* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4742481360->4742494736*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aabd510>"
]
},
"execution_count": 221,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x2w2._backward()\n",
"x1w1._backward()\n",
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "57a4b0c3-3223-46a4-8e36-382576685e04",
"metadata": {},
"source": [
"### Now we get rid of one more inefficiency... calling ._backward() manually\n",
"\n",
"We layed out a math expresssion, trying to go backwards thru expressiosn. Never want to call a .back for any node before we've done everything after it. We have to get all of its full dependencies, they have to propogate through it before we use it. \n",
"\n",
"We can do this through a topological sort, where we make a graph that has all of its edges layed out from left to right. Wikipedia says it's: \"a linear ordering of its vertices such that for every directed edge (u,v) from vertex u to vertex v, u comes before v in the ordering.\"\n",
"\n",
"\n",
"\n",
"Let's use some topological sort code that orders our Value objects from beginning to lastNotice how the output frames ```o``` last (```o.data = 0.707...```)"
]
},
{
"cell_type": "code",
"execution_count": 231,
"id": "f9163a2a-4d10-4787-a5c4-a34c4716d8be",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Value(data=6.881373587019543),\n",
" Value(data=0.0),\n",
" Value(data=1.0),\n",
" Value(data=0.0),\n",
" Value(data=2.0),\n",
" Value(data=-3.0),\n",
" Value(data=-6.0),\n",
" Value(data=-6.0),\n",
" Value(data=0.8813735870195432),\n",
" Value(data=0.7071067811865476)]"
]
},
"execution_count": 231,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"topo = [] #topological graph\n",
"visited = set() #maintain a set of visited nodes\n",
"def build_topo(v): #start at some root node; for us the root node is o\n",
" if v not in visited: #go through all of its children, lay out from left to right. If node is not visted, marks as visted\n",
" #iterates through all of its children, calls build_topo on them, and after it's gone through \n",
" #all of the children, it adds itself. This node (o), will only add itself to the topo list\n",
" #after all of the children are in the list.\n",
" visited.add(v)\n",
" for child in v._prev:\n",
" build_topo(child)\n",
" topo.append(v)\n",
"build_topo(o)\n",
"topo"
]
},
{
"cell_type": "markdown",
"id": "4349689c-cf86-4710-a0d7-7ab002b9e6d7",
"metadata": {},
"source": [
"Now, let's test it by reinitializing and re-running with the topology code. Let's reset the gradients again, make sure they are reset with ```draw_dot(o)```, and then reset ```o``` to $1.0$ so we can begin backpropagation. Note the slight change to ```build_topo```."
]
},
{
"cell_type": "code",
"execution_count": 232,
"id": "704fecf7-8b3d-47c0-9945-0aa3b460a0d3",
"metadata": {},
"outputs": [],
"source": [
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"b = Value(6.8813735870195432, label='b') #weird number used for nice output... try with b = Value(8.0)\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"n = x1w1x2w2 + b; n.label = 'n'\n",
"\n",
"o = n.tanh(); o.label = 'o'"
]
},
{
"cell_type": "code",
"execution_count": 233,
"id": "3fe6f710-244d-4d9a-8a80-1b4158f49ea4",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4742482960 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4742482960</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-54.5 1374.75,-90.5 1558.5,-90.5 1558.5,-54.5 1374.75,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-55 1397.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-55 1477.25,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4742482960tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4742482960tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4742482960tanh->4742482960 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4742482960tanh->4742482960</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-72.5C1346.17,-72.5 1354.36,-72.5 1363.07,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-76 1372.81,-72.5 1362.81,-69 1362.81,-76\"/>\n",
"</g>\n",
"<!-- 4730012688 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4730012688</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-110.5 324,-146.5 542.25,-146.5 542.25,-110.5 324,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-111 376.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-111 461,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734125328+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4734125328+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4730012688->4734125328+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4730012688->4734125328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-110.69C551.12,-109.24 559.54,-107.86 567.19,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.72,-110.06 577.02,-104.98 566.58,-103.15 567.72,-110.06\"/>\n",
"</g>\n",
"<!-- 4730012688* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4730012688*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4730012688*->4730012688 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4730012688*->4730012688</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C295.29,-128.5 303.43,-128.5 312.17,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-132 322.01,-128.5 312.01,-125 312.01,-132\"/>\n",
"</g>\n",
"<!-- 4730023440 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4730023440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-165.5 0,-201.5 198,-201.5 198,-165.5 0,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-166 32.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-166 116.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730023440->4730012688* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4730023440->4730012688*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4730013200 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4730013200</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-54.5 1065,-90.5 1248.75,-90.5 1248.75,-54.5 1065,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-55 1087.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-55 1167.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730013200->4742482960tanh -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4730013200->4742482960tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-72.5C1257.39,-72.5 1265.52,-72.5 1272.98,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-76 1282.95,-72.5 1272.95,-69 1272.95,-76\"/>\n",
"</g>\n",
"<!-- 4730013200+ -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4730013200+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4730013200+->4730013200 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4730013200+->4730013200</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-72.5C1036.42,-72.5 1044.61,-72.5 1053.32,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-76 1063.06,-72.5 1053.06,-69 1053.06,-76\"/>\n",
"</g>\n",
"<!-- 4730018384 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4730018384</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-55.5 2.25,-91.5 195.75,-91.5 195.75,-55.5 2.25,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-56 34.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-56 114.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730013584* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4730013584*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4730018384->4730013584* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4730018384->4730013584*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-73.5C205.14,-73.5 214.15,-73.5 222.32,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-77 232.13,-73.5 222.13,-70 222.13,-77\"/>\n",
"</g>\n",
"<!-- 4734125328 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4734125328</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4734125328->4730013200+ -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4734125328->4730013200+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M934.24,-82.02C944.99,-80.49 955.18,-79.03 964.21,-77.75\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.46,-81.25 973.86,-76.37 963.47,-74.32 964.46,-81.25\"/>\n",
"</g>\n",
"<!-- 4734125328+->4734125328 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4734125328+->4734125328</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4730019600 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4730019600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-27.5 711.75,-63.5 895.5,-63.5 895.5,-27.5 711.75,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-28 734.5,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-28 814.25,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730019600->4730013200+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4730019600->4730013200+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-58.05C919.97,-61.36 944.58,-64.74 963.93,-67.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.36,-70.86 973.75,-68.75 964.32,-63.92 963.36,-70.86\"/>\n",
"</g>\n",
"<!-- 4730027856 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4730027856</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-110.5 3.75,-146.5 194.25,-146.5 194.25,-110.5 3.75,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-111 33.25,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-111 113,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730027856->4730012688* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4730027856->4730012688*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M194.46,-128.5C204.21,-128.5 213.66,-128.5 222.21,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.12,-132 232.12,-128.5 222.12,-125 222.12,-132\"/>\n",
"</g>\n",
"<!-- 4730013584 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4730013584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-55.5 326.25,-91.5 540,-91.5 540,-55.5 326.25,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-56 379,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-56 458.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730013584->4734125328+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4730013584->4734125328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-90.37C549.89,-91.87 559,-93.32 567.21,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.63,-98.07 577.05,-96.18 567.72,-91.16 566.63,-98.07\"/>\n",
"</g>\n",
"<!-- 4730013584*->4730013584 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4730013584*->4730013584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C296,-73.5 305.08,-73.5 314.82,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-77 324.55,-73.5 314.55,-70 314.55,-77\"/>\n",
"</g>\n",
"<!-- 4730015184 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4730015184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730015184->4730013584* -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4730015184->4730013584*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11aaa9e90>"
]
},
"execution_count": 233,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o) #gradients are reset"
]
},
{
"cell_type": "code",
"execution_count": 234,
"id": "fe9b36ec-17a2-4d85-b5a5-ce8403954567",
"metadata": {},
"outputs": [],
"source": [
"o.grad = 1.0 #base case\n",
"topo = [] \n",
"visited = set()\n",
"def build_topo(v): \n",
" if v not in visited: \n",
" visited.add(v)\n",
" for child in v._prev:\n",
" build_topo(child)\n",
" topo.append(v)\n",
"build_topo(o)\n",
"\n",
"#o only adds itself to topo list after all of the children have been processed\n",
"#just calling ._backward() to all of the topological nodes in order\n",
"for node in reversed(topo): #reverse order, starting at o\n",
" node._backward()"
]
},
{
"cell_type": "code",
"execution_count": 235,
"id": "9229dbe3-22aa-41e5-931f-6c562fad5128",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4742482960 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4742482960</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-54.5 1374.75,-90.5 1558.5,-90.5 1558.5,-54.5 1374.75,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-55 1397.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-55 1477.25,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4742482960tanh -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4742482960tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4742482960tanh->4742482960 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4742482960tanh->4742482960</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-72.5C1346.17,-72.5 1354.36,-72.5 1363.07,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-76 1372.81,-72.5 1362.81,-69 1362.81,-76\"/>\n",
"</g>\n",
"<!-- 4730012688 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4730012688</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-110.5 324,-146.5 542.25,-146.5 542.25,-110.5 324,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-111 376.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-111 461,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734125328+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4734125328+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4730012688->4734125328+ -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4730012688->4734125328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-110.69C551.12,-109.24 559.54,-107.86 567.19,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.72,-110.06 577.02,-104.98 566.58,-103.15 567.72,-110.06\"/>\n",
"</g>\n",
"<!-- 4730012688* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4730012688*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4730012688*->4730012688 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4730012688*->4730012688</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C295.29,-128.5 303.43,-128.5 312.17,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-132 322.01,-128.5 312.01,-125 312.01,-132\"/>\n",
"</g>\n",
"<!-- 4730023440 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4730023440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-165.5 0,-201.5 198,-201.5 198,-165.5 0,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-166 32.5,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-166 116.75,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4730023440->4730012688* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4730023440->4730012688*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4730013200 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4730013200</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-54.5 1065,-90.5 1248.75,-90.5 1248.75,-54.5 1065,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-55 1087.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-55 1167.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4730013200->4742482960tanh -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4730013200->4742482960tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-72.5C1257.39,-72.5 1265.52,-72.5 1272.98,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-76 1282.95,-72.5 1272.95,-69 1272.95,-76\"/>\n",
"</g>\n",
"<!-- 4730013200+ -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4730013200+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4730013200+->4730013200 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4730013200+->4730013200</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-72.5C1036.42,-72.5 1044.61,-72.5 1053.32,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-76 1063.06,-72.5 1053.06,-69 1053.06,-76\"/>\n",
"</g>\n",
"<!-- 4730018384 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4730018384</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-55.5 2.25,-91.5 195.75,-91.5 195.75,-55.5 2.25,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-56 34.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-56 114.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730013584* -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4730013584*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4730018384->4730013584* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4730018384->4730013584*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-73.5C205.14,-73.5 214.15,-73.5 222.32,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-77 232.13,-73.5 222.13,-70 222.13,-77\"/>\n",
"</g>\n",
"<!-- 4734125328 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4734125328</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4734125328->4730013200+ -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4734125328->4730013200+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M934.24,-82.02C944.99,-80.49 955.18,-79.03 964.21,-77.75\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.46,-81.25 973.86,-76.37 963.47,-74.32 964.46,-81.25\"/>\n",
"</g>\n",
"<!-- 4734125328+->4734125328 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4734125328+->4734125328</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4730019600 -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4730019600</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-27.5 711.75,-63.5 895.5,-63.5 895.5,-27.5 711.75,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-28 734.5,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-28 814.25,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4730019600->4730013200+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4730019600->4730013200+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-58.05C919.97,-61.36 944.58,-64.74 963.93,-67.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.36,-70.86 973.75,-68.75 964.32,-63.92 963.36,-70.86\"/>\n",
"</g>\n",
"<!-- 4730027856 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4730027856</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-110.5 1.5,-146.5 196.5,-146.5 196.5,-110.5 1.5,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"31,-111 31,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"70.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-111 110.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -1.5000</text>\n",
"</g>\n",
"<!-- 4730027856->4730012688* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4730027856->4730012688*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M196.76,-128.5C205.77,-128.5 214.47,-128.5 222.4,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.25,-132 232.25,-128.5 222.25,-125 222.25,-132\"/>\n",
"</g>\n",
"<!-- 4730013584 -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4730013584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-55.5 326.25,-91.5 540,-91.5 540,-55.5 326.25,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-56 379,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-56 458.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4730013584->4734125328+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4730013584->4734125328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-90.37C549.89,-91.87 559,-93.32 567.21,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.63,-98.07 577.05,-96.18 567.72,-91.16 566.63,-98.07\"/>\n",
"</g>\n",
"<!-- 4730013584*->4730013584 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4730013584*->4730013584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C296,-73.5 305.08,-73.5 314.82,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-77 324.55,-73.5 314.55,-70 314.55,-77\"/>\n",
"</g>\n",
"<!-- 4730015184 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4730015184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4730015184->4730013584* -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4730015184->4730013584*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x119ee7cd0>"
]
},
"execution_count": 235,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "e2bf7364-7932-4feb-81bb-59fe6faee98c",
"metadata": {},
"source": [
"### Now, we will hide the topological backpropogation function in the Value class to make the code more seamless. Then, we will reinitialize values, make sure all gradients are set to 0, and then do the whole thing with one function : ```o.backward()```!"
]
},
{
"cell_type": "code",
"execution_count": 274,
"id": "f51b0db0-05fb-400a-93c8-cf53ac3bb150",
"metadata": {},
"outputs": [],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''): \n",
" self.data = data\n",
" self.grad = 0.0 \n",
" self._prev = set(_children)\n",
" self._backward = lambda : None\n",
" self._op = _op\n",
" self.label=label\n",
"\n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data, (self, other), '+') \n",
" \n",
" def _backward(): \n",
" self.grad = 1.0 * out.grad \n",
" other.grad = 1.0 * out.grad \n",
" out._backward = _backward\n",
" return out\n",
" \n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data, (self, other), '*')\n",
"\n",
" def _backward():\n",
" self.grad = other.data * out.grad \n",
" other.grad = self.data * out.grad\n",
" out._backward = _backward\n",
" return out\n",
" \n",
" def tanh(self):\n",
" x = self.data\n",
" t = (math.exp(2*x)-1)/(math.exp(2*x)+1)\n",
" \n",
" out = Value(t, (self, ), 'tanh')\n",
"\n",
" def _backward():\n",
" self.grad = (1-t**2) * out.grad\n",
" out._backward = _backward\n",
" return out\n",
"\n",
" #define backward() without the underscore. \n",
" def backward(self): #Build the topological graph starting at self\n",
" topo = [] #topological list where we populate the order\n",
" visited = set() #maintain a set of visited nodes\n",
" def build_topo(v): #start at the root node\n",
" if v not in visited: \n",
" visited.add(v)\n",
" for child in v._prev:\n",
" build_topo(child)\n",
" topo.append(v)\n",
" build_topo(self)\n",
"\n",
" self.grad = 1.0 #initalize root.grad\n",
" for node in reversed(topo):\n",
" node._backward() #call _backward() and do backpropogation on all of the children. "
]
},
{
"cell_type": "code",
"execution_count": 270,
"id": "629ce317-7b6f-480d-87a4-f1ff987f84c8",
"metadata": {},
"outputs": [],
"source": [
"#redefine variables:\n",
"x1 = Value(2.0, label='x1')\n",
"x2 = Value(0.0, label='x2')\n",
"w1 = Value(-3.0, label='w1')\n",
"w2 = Value(1.0, label='w2')\n",
"b = Value(6.8813735870195432, label='b')\n",
"x1w1 = x1*w1; x1w1.label = 'x1*w1'\n",
"x2w2 = x2*w2; x2w2.label = 'x2*w2'\n",
"x1w1x2w2 = x1w1 + x2w2; x1w1x2w2.label = 'x1*w1 + x2*w2'\n",
"n = x1w1x2w2 + b; n.label = 'n'\n",
"o = n.tanh(); o.label = 'o'"
]
},
{
"cell_type": "code",
"execution_count": 271,
"id": "732ce65c-dce7-48a5-8b4b-6deead69abac",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4733509584 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4733509584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-54.5 1065,-90.5 1248.75,-90.5 1248.75,-54.5 1065,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-55 1087.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-55 1167.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730296720tanh -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4730296720tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733509584->4730296720tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733509584->4730296720tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-72.5C1257.39,-72.5 1265.52,-72.5 1272.98,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-76 1282.95,-72.5 1272.95,-69 1272.95,-76\"/>\n",
"</g>\n",
"<!-- 4733509584+ -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4733509584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4733509584+->4733509584 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4733509584+->4733509584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-72.5C1036.42,-72.5 1044.61,-72.5 1053.32,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-76 1063.06,-72.5 1053.06,-69 1053.06,-76\"/>\n",
"</g>\n",
"<!-- 4733506128 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4733506128</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733512272+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4733512272+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4733506128->4733512272+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4733506128->4733512272+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4733506128* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733506128*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733506128*->4733506128 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733506128*->4733506128</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733512272 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733512272</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733512272->4733509584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733512272->4733509584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M934.24,-82.02C944.99,-80.49 955.18,-79.03 964.21,-77.75\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.46,-81.25 973.86,-76.37 963.47,-74.32 964.46,-81.25\"/>\n",
"</g>\n",
"<!-- 4733512272+->4733512272 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733512272+->4733512272</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4741194640 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4741194640</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733506512* -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4733506512*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4741194640->4733506512* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4741194640->4733506512*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4731880208 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731880208</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-0.5 3.75,-36.5 194.25,-36.5 194.25,-0.5 3.75,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-1 33.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-1 113,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4731880208->4733506512* -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4731880208->4733506512*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
"</g>\n",
"<!-- 4733504336 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733504336</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-27.5 711.75,-63.5 895.5,-63.5 895.5,-27.5 711.75,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-28 734.5,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-28 814.25,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733504336->4733509584+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4733504336->4733509584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-58.05C919.97,-61.36 944.58,-64.74 963.93,-67.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.36,-70.86 973.75,-68.75 964.32,-63.92 963.36,-70.86\"/>\n",
"</g>\n",
"<!-- 4733506512 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733506512</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733506512->4733512272+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4733506512->4733512272+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733506512*->4733506512 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4733506512*->4733506512</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4730296720 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4730296720</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-54.5 1374.75,-90.5 1558.5,-90.5 1558.5,-54.5 1374.75,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-55 1397.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-55 1477.25,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4730296720tanh->4730296720 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4730296720tanh->4730296720</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-72.5C1346.17,-72.5 1354.36,-72.5 1363.07,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-76 1372.81,-72.5 1362.81,-69 1362.81,-76\"/>\n",
"</g>\n",
"<!-- 4741190608 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4741190608</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4741190608->4733506128* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4741190608->4733506128*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4733514192 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733514192</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733514192->4733506128* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4733514192->4733506128*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11a6c4290>"
]
},
"execution_count": 271,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#make sure all grads are 0 once more\n",
"draw_dot(o)"
]
},
{
"cell_type": "code",
"execution_count": 272,
"id": "8996f017-5597-4eba-aac1-7306d1589483",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1567pt\" height=\"210pt\"\n",
" viewBox=\"0.00 0.00 1566.50 210.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-206 1562.5,-206 1562.5,4 -4,4\"/>\n",
"<!-- 4733509584 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4733509584</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1065,-54.5 1065,-90.5 1248.75,-90.5 1248.75,-54.5 1065,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1076.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">n</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1087.75,-55 1087.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1127.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1167.5,-55 1167.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1208.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4730296720tanh -->\n",
"<g id=\"node13\" class=\"node\">\n",
"<title>4730296720tanh</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1311.75\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1311.75\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">tanh</text>\n",
"</g>\n",
"<!-- 4733509584->4730296720tanh -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4733509584->4730296720tanh</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1249.01,-72.5C1257.39,-72.5 1265.52,-72.5 1272.98,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1272.95,-76 1282.95,-72.5 1272.95,-69 1272.95,-76\"/>\n",
"</g>\n",
"<!-- 4733509584+ -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4733509584+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"1002\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"1002\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4733509584+->4733509584 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4733509584+->4733509584</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1029.28,-72.5C1036.42,-72.5 1044.61,-72.5 1053.32,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1053.06,-76 1063.06,-72.5 1053.06,-69 1053.06,-76\"/>\n",
"</g>\n",
"<!-- 4733506128 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4733506128</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"326.25,-110.5 326.25,-146.5 540,-146.5 540,-110.5 326.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"352.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"379,-111 379,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"458.75,-111 458.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"499.38\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733512272+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4733512272+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"605.25\" cy=\"-100.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"605.25\" y=\"-95.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4733506128->4733512272+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4733506128->4733512272+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M540.42,-111.01C549.89,-109.45 559,-107.95 567.21,-106.6\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"567.75,-110.05 577.05,-104.98 566.62,-103.15 567.75,-110.05\"/>\n",
"</g>\n",
"<!-- 4733506128* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4733506128*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-128.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-123.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4733506128*->4733506128 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4733506128*->4733506128</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-128.5C296,-128.5 305.08,-128.5 314.82,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"314.55,-132 324.55,-128.5 314.55,-125 314.55,-132\"/>\n",
"</g>\n",
"<!-- 4733512272 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4733512272</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"668.25,-82.5 668.25,-118.5 939,-118.5 939,-82.5 668.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"720.88\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1 + x2*w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"773.5,-83 773.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"815.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"857.75,-83 857.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"898.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733512272->4733509584+ -->\n",
"<g id=\"edge12\" class=\"edge\">\n",
"<title>4733512272->4733509584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M934.24,-82.02C944.99,-80.49 955.18,-79.03 964.21,-77.75\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"964.46,-81.25 973.86,-76.37 963.47,-74.32 964.46,-81.25\"/>\n",
"</g>\n",
"<!-- 4733512272+->4733512272 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4733512272+->4733512272</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M632.73,-100.5C639.73,-100.5 647.79,-100.5 656.52,-100.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"656.39,-104 666.39,-100.5 656.39,-97 656.39,-104\"/>\n",
"</g>\n",
"<!-- 4741194640 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4741194640</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 198,-91.5 198,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"32.5,-56 32.5,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"116.75,-56 116.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"157.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4733506512* -->\n",
"<g id=\"node11\" class=\"node\">\n",
"<title>4733506512*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"261\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"261\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4741194640->4733506512* -->\n",
"<g id=\"edge10\" class=\"edge\">\n",
"<title>4741194640->4733506512*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M198.14,-73.5C206.61,-73.5 214.8,-73.5 222.29,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.29,-77 232.29,-73.5 222.29,-70 222.29,-77\"/>\n",
"</g>\n",
"<!-- 4731880208 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4731880208</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-0.5 1.5,-36.5 196.5,-36.5 196.5,-0.5 1.5,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"16.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"31,-1 31,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"70.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"110.75,-1 110.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -1.5000</text>\n",
"</g>\n",
"<!-- 4731880208->4733506512* -->\n",
"<g id=\"edge14\" class=\"edge\">\n",
"<title>4731880208->4733506512*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-36.99C180.9,-39.65 189.7,-42.5 198,-45.5 208.09,-49.14 218.82,-53.73 228.47,-58.12\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"226.9,-61.25 237.45,-62.31 229.86,-54.91 226.9,-61.25\"/>\n",
"</g>\n",
"<!-- 4733504336 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4733504336</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"711.75,-27.5 711.75,-63.5 895.5,-63.5 895.5,-27.5 711.75,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"723.12\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.5,-28 734.5,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 6.8814</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"814.25,-28 814.25,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"854.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733504336->4733509584+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4733504336->4733509584+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M895.91,-58.05C919.97,-61.36 944.58,-64.74 963.93,-67.4\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"963.36,-70.86 973.75,-68.75 964.32,-63.92 963.36,-70.86\"/>\n",
"</g>\n",
"<!-- 4733506512 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4733506512</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"324,-55.5 324,-91.5 542.25,-91.5 542.25,-55.5 324,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"350.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">x1*w1</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"376.75,-56 376.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"418.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"461,-56 461,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"501.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4733506512->4733512272+ -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4733506512->4733512272+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M542.35,-90.68C551.12,-92.07 559.54,-93.4 567.19,-94.62\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"566.59,-98.07 577.02,-96.18 567.69,-91.15 566.59,-98.07\"/>\n",
"</g>\n",
"<!-- 4733506512*->4733506512 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4733506512*->4733506512</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M288.21,-73.5C295.29,-73.5 303.43,-73.5 312.17,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"312.01,-77 322.01,-73.5 312.01,-70 312.01,-77\"/>\n",
"</g>\n",
"<!-- 4730296720 -->\n",
"<g id=\"node12\" class=\"node\">\n",
"<title>4730296720</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1374.75,-54.5 1374.75,-90.5 1558.5,-90.5 1558.5,-54.5 1374.75,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1386.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">o</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1397.5,-55 1397.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1437.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.7071</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1477.25,-55 1477.25,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1517.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4730296720tanh->4730296720 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4730296720tanh->4730296720</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M1339.03,-72.5C1346.17,-72.5 1354.36,-72.5 1363.07,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"1362.81,-76 1372.81,-72.5 1362.81,-69 1362.81,-76\"/>\n",
"</g>\n",
"<!-- 4741190608 -->\n",
"<g id=\"node14\" class=\"node\">\n",
"<title>4741190608</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"3.75,-165.5 3.75,-201.5 194.25,-201.5 194.25,-165.5 3.75,-165.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">x2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"33.25,-166 33.25,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"73.12\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 0.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"113,-166 113,-201.5\"/>\n",
"<text text-anchor=\"middle\" x=\"153.62\" y=\"-178.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.5000</text>\n",
"</g>\n",
"<!-- 4741190608->4733506128* -->\n",
"<g id=\"edge11\" class=\"edge\">\n",
"<title>4741190608->4733506128*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M172.12,-165.01C180.9,-162.35 189.7,-159.5 198,-156.5 208.09,-152.86 218.82,-148.27 228.47,-143.88\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"229.86,-147.09 237.45,-139.69 226.9,-140.75 229.86,-147.09\"/>\n",
"</g>\n",
"<!-- 4733514192 -->\n",
"<g id=\"node15\" class=\"node\">\n",
"<title>4733514192</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.25,-110.5 2.25,-146.5 195.75,-146.5 195.75,-110.5 2.25,-110.5\"/>\n",
"<text text-anchor=\"middle\" x=\"18.5\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">w2</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"34.75,-111 34.75,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"74.62\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 1.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"114.5,-111 114.5,-146.5\"/>\n",
"<text text-anchor=\"middle\" x=\"155.12\" y=\"-123.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4733514192->4733506128* -->\n",
"<g id=\"edge13\" class=\"edge\">\n",
"<title>4733514192->4733506128*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M195.84,-128.5C205.14,-128.5 214.15,-128.5 222.32,-128.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"222.13,-132 232.13,-128.5 222.13,-125 222.13,-132\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11ab7ba90>"
]
},
"execution_count": 272,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#FINISH IT!\n",
"o.backward()\n",
"draw_dot(o)"
]
},
{
"cell_type": "markdown",
"id": "20f0da93-7dce-49f5-89a9-2f2ee73f7a22",
"metadata": {},
"source": [
"### And that's Backpropogation for one neuron, which in practice, would be a small part of a large neural net :)\n",
"\n",
"### Before you move on to the next lecture, there is a bug in the backpropogation where one node is used multiple times. See if you can find the bug and fix it! If not, no worries. It will be done in the beginning of the next lecture."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}