{
"cells": [
{
"cell_type": "markdown",
"id": "71e476da-9da9-4574-9e65-02db8db03fd4",
"metadata": {},
"source": [
"# Notebook 2: Chain Rule [](https://colab.research.google.com/github/mattsankner/micrograd/blob/main/mg2_chain_rule.ipynb) [](https://nbviewer.jupyter.org/github/mattsankner/micrograd/blob/main/mg2_chain_rule.ipynb)"
]
},
{
"cell_type": "markdown",
"id": "36145db7-df04-4169-af61-2e1636139018",
"metadata": {},
"source": [
"## Welcome to the second mini-lecture!"
]
},
{
"cell_type": "markdown",
"id": "9c79f45c-e6d9-4d43-8c42-2a8353cb2065",
"metadata": {},
"source": [
"Now that we have some sense of what the derivative tells us about the function, let's start by building out a Value class for our previous values $a, b, c$. This way, we can ascribe more properties to them than just their value. \n",
"\n",
"As we move to neural networks, we need good data structures that can hold massive mathematical expressions. Here's the class that makes Value objects:"
]
},
{
"cell_type": "code",
"execution_count": 61,
"id": "e207cd4b-c277-4aa3-82e2-e471f24d25fc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Value(data=2.0)"
]
},
"execution_count": 61,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#class Value takes a single scalar value that it wraps and keeps track of.\n",
"class Value:\n",
"\n",
"#each Value object will have data, children Value nodes , an operator (+, -, *, etc.), a label, and a gradient. These are explained below.\n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data\n",
" self._prev = set(_children)\n",
" self._op = _op\n",
" self.label = label\n",
" self.grad = 0.0\n",
" \n",
"#returns a value wrapper with the data. Prints out a nice expression, instead of something crytpic\n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
" \n",
"a = Value(2.0)\n",
"a"
]
},
{
"cell_type": "markdown",
"id": "c28b781f-8aba-4da2-8a8c-558fb6e6e758",
"metadata": {},
"source": [
"Now, we want to be able to do $a+b$, but currently python doesn't know how to add two of our Value objects.\n",
"\n",
"In python, we need to use special double underscore operators to define the operators for the objects, using special, specific names like ```__add__```, etc.\n",
"Let's define the add and multiply function."
]
},
{
"cell_type": "code",
"execution_count": 63,
"id": "baba4e21-f79f-49a1-a5e6-fdb85b948322",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Value(data=4.0)"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data\n",
" self._prev = set(_children)\n",
" self._op = _op\n",
" self.label = label\n",
" self.grad = 0.0\n",
" \n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
"#to match built in python function .__add__()\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data)\n",
" return out\n",
" \n",
"#to match built in python function .__mul__()\n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data)\n",
" return out\n",
"\n",
"a = Value(2.0)\n",
"b = Value(-3.0)\n",
"c = Value(10.0)\n",
"\n",
"a * b + c #equivalent in python to a.__mul__(b).__add__(c). If it were addition, it would be a.__add__(b).__add__(c). Can call this manually"
]
},
{
"cell_type": "markdown",
"id": "6c0823b0-efe2-4687-9c5b-e1a3a1ce1dc2",
"metadata": {},
"source": [
"We want to create children Value nodes that show how we got to the value we're at. This will be stored in the \n",
"```_children=()``` tuple, which will be stored in a set of ```_children, self.prev.``` Notice self._prev below.\n",
"\n",
"\n",
"When we create a value through addition or multiplicatoin, we want to pass in these children of the value, which is ```(self, other)``` into the operator.\n",
"\n",
"Now, we know the children of the ```Value``` object , but not the operation. Notice we add the operation sign at the end of the ```Value``` object creation in the ```out```.\n"
]
},
{
"cell_type": "code",
"execution_count": 86,
"id": "f2a2932d-79ac-476b-bed4-08a683d7b96a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{Value(data=-6.0), Value(data=10.0)}\n",
"+\n"
]
}
],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data\n",
" self._prev = set(_children)\n",
" self._op = _op\n",
" self.label = label\n",
" self.grad = 0.0\n",
" \n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
"#to match built in python function .__add__()\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data, (self, other), '+')\n",
" return out\n",
" \n",
"#to match built in python function .__mul__()\n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data, (self, other), '*')\n",
" return out\n",
"\n",
"a = Value(2.0, label='a')\n",
"b = Value(-3.0, label='b')\n",
"c = Value(10.0, label='c') #added labels\n",
"e = a*b; e.label = 'e' #added e for a * b\n",
"d = e + c; d.label = 'd'\n",
"print(d._prev)\n",
"print(d._op)"
]
},
{
"cell_type": "markdown",
"id": "7e79873c-2b62-41a1-b943-88c423bf9230",
"metadata": {},
"source": [
"As we build out the expression, we want to build out something that can help us visualize the expressions. We will use the Graphviz api for this. Before we understand what the code does, visualize what it does below by running the defintion block and then ```draw_dot()``` on our previous value ```d```. Then, read the function comments to help you understand the code. Ultimately, it's not uber important to understand the finer details of ```trace()``` and ```draw_dot()```.\n",
"\n",
"We can call ```draw_dot()``` on a root node (```Value``` obj), and ```trace()``` will visualize it."
]
},
{
"cell_type": "code",
"execution_count": 76,
"id": "1ec7ba82-fc87-45b7-ab2a-36feaf1ee2f1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: graphviz in /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages (0.20.3)\n",
"\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.1.2\u001b[0m\n",
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
]
}
],
"source": [
"!pip install graphviz\n",
"!export PATH=\"/usr/local/opt/graphviz/bin:$PATH\""
]
},
{
"cell_type": "code",
"execution_count": 84,
"id": "0d369653-eee5-48c2-8f36-c432fe364ab3",
"metadata": {},
"outputs": [],
"source": [
"from graphviz import Digraph #graphviz is an opensource vizualization software. We are building out this graph in graphviz API. \n",
"\n",
"def trace(root): #helper function that enumerates the ndoes and edges in the graph\n",
" # builds a set of all nodes and edges in a graph\n",
" nodes, edges = set(), set()\n",
" def build(v):\n",
" if v not in nodes:\n",
" nodes.add(v)\n",
" for child in v._prev:\n",
" edges.add((child, v))\n",
" build(child)\n",
" build(root)\n",
" return nodes, edges\n",
"\n",
"def draw_dot(root): #creating op nodes (not Value objects)\n",
" dot = Digraph(format='svg', graph_attr={'rankdir': 'LR'}) # LR = left to right\n",
" \n",
" nodes, edges = trace(root) #call trace\n",
" for n in nodes:\n",
" uid = str(id(n))\n",
" # for any value in the graph, create a rectangular ('record') node for it\n",
" dot.node(name = uid, label = \"{ %s | data %.4f | grad % .4f}\" % (n.label, n.data, n.grad), shape='record')\n",
" if n._op:\n",
" # if this value is a result of some operation, create an op node for it. Not a value object\n",
" dot.node(name = uid + n._op, label = n._op)\n",
" # and connect this node to it\n",
" dot.edge(uid + n._op, uid)\n",
"\n",
" for n1, n2 in edges:\n",
" # connect n1 to the op node of n2\n",
" dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
"\n",
" return dot"
]
},
{
"cell_type": "code",
"execution_count": 91,
"id": "b4caa4ef-fcbf-4ac6-8456-319d9631ca09",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"822pt\" height=\"128pt\"\n",
" viewBox=\"0.00 0.00 821.75 128.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 124)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-124 817.75,-124 817.75,4 -4,4\"/>\n",
"<!-- 4809677328 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809677328</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"630,-27.5 630,-63.5 813.75,-63.5 813.75,-27.5 630,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"641.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"652.75,-28 652.75,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"692.62\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"732.5,-28 732.5,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"773.12\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809677328+ -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809677328+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"567\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"567\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4809677328+->4809677328 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809677328+->4809677328</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M594.28,-45.5C601.42,-45.5 609.61,-45.5 618.32,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"618.06,-49 628.06,-45.5 618.06,-42 618.06,-49\"/>\n",
"</g>\n",
"<!-- 4809671760 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809671760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.62,-83.5 2.62,-119.5 185.62,-119.5 185.62,-83.5 2.62,-83.5\"/>\n",
"<text text-anchor=\"middle\" x=\"13.62\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-84 24.62,-119.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.5\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-84 104.38,-119.5\"/>\n",
"<text text-anchor=\"middle\" x=\"145\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809671888* -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4809671888*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"251.25\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"251.25\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809671760->4809671888* -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809671760->4809671888*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M185.81,-85.13C195.66,-83.35 205.21,-81.63 213.82,-80.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"214.19,-83.57 223.41,-78.34 212.95,-76.68 214.19,-83.57\"/>\n",
"</g>\n",
"<!-- 4809674896 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4809674896</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-28.5 0,-64.5 188.25,-64.5 188.25,-28.5 0,-28.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-29 22.75,-64.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-29 107,-64.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.62\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809674896->4809671888* -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809674896->4809671888*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.49,-62.75C197.27,-64.28 205.77,-65.76 213.51,-67.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"212.66,-70.51 223.11,-68.78 213.86,-63.61 212.66,-70.51\"/>\n",
"</g>\n",
"<!-- 4809671888 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809671888</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"315.38,-55.5 315.38,-91.5 502.88,-91.5 502.88,-55.5 315.38,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"326.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"337.38,-56 337.38,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"421.62,-56 421.62,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"462.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809671888->4809677328+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4809671888->4809677328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M503.05,-56.81C512.25,-55.16 521.16,-53.56 529.24,-52.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"529.71,-55.58 538.93,-50.36 528.47,-48.69 529.71,-55.58\"/>\n",
"</g>\n",
"<!-- 4809671888*->4809671888 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809671888*->4809671888</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M278.69,-73.5C286.1,-73.5 294.64,-73.5 303.71,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"303.51,-77 313.51,-73.5 303.51,-70 303.51,-77\"/>\n",
"</g>\n",
"<!-- 4809676112 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809676112</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"314.25,-0.5 314.25,-36.5 504,-36.5 504,-0.5 314.25,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"325.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"336.25,-1 336.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"422.75,-1 422.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"463.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809676112->4809677328+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809676112->4809677328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M504.39,-34.83C513.16,-36.35 521.65,-37.82 529.37,-39.15\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"528.5,-42.56 538.95,-40.81 529.69,-35.66 528.5,-42.56\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11e79a690>"
]
},
"execution_count": 91,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(d) "
]
},
{
"cell_type": "markdown",
"id": "b45beae3-ad35-4f1d-b3cb-35146955f62b",
"metadata": {},
"source": [
"Above, we can visualize ```a * b + c```.\n",
"\n",
"Now, let's add variables ```f```, and a return variable ```L```, and re-run ```draw_dot(L)```."
]
},
{
"cell_type": "code",
"execution_count": 93,
"id": "a80edebb-1f2a-411e-adb8-cd8f7770ba9f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Value(data=-8.0)"
]
},
"execution_count": 93,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"class Value:\n",
" \n",
" def __init__(self, data, _children=(), _op='', label=''):\n",
" self.data = data\n",
" self._prev = set(_children)\n",
" self._op = _op\n",
" self.label = label\n",
" self.grad = 0.0\n",
" \n",
" def __repr__(self):\n",
" return f\"Value(data={self.data})\"\n",
"\n",
"#to match built in python function .__add__()\n",
" def __add__(self, other):\n",
" out = Value(self.data + other.data, (self, other), '+')\n",
" return out\n",
" \n",
"#to match built in python function .__mul__()\n",
" def __mul__(self, other):\n",
" out = Value(self.data * other.data, (self, other), '*')\n",
" return out\n",
"\n",
"a = Value(2.0, label='a')\n",
"b = Value(-3.0, label='b')\n",
"c = Value(10.0, label='c') #added labels\n",
"e = a*b; e.label = 'e' #added e for a * b\n",
"d = e + c; d.label = 'd'\n",
"f = Value(-2.0, label = 'f')\n",
"L = d*f; label = 'L'\n",
"L"
]
},
{
"cell_type": "code",
"execution_count": 92,
"id": "efd2f2e5-6748-4355-9a20-c87f82cd5475",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1135pt\" height=\"128pt\"\n",
" viewBox=\"0.00 0.00 1135.25 128.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 124)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-124 1131.25,-124 1131.25,4 -4,4\"/>\n",
"<!-- 4809677328 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809677328</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"631.12,-27.5 631.12,-63.5 814.88,-63.5 814.88,-27.5 631.12,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"642.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"653.88,-28 653.88,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"693.75\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"733.62,-28 733.62,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.25\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809677392* -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4809677392*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"879\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"879\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809677328->4809677392* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809677328->4809677392*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M815.36,-61.52C824.5,-63.12 833.36,-64.67 841.4,-66.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"840.58,-69.49 851.04,-67.77 841.79,-62.6 840.58,-69.49\"/>\n",
"</g>\n",
"<!-- 4809677328+ -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809677328+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"567\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"567\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4809677328+->4809677328 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809677328+->4809677328</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M594.47,-45.5C601.87,-45.5 610.38,-45.5 619.43,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"619.17,-49 629.17,-45.5 619.17,-42 619.17,-49\"/>\n",
"</g>\n",
"<!-- 4809677392 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809677392</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"942,-54.5 942,-90.5 1127.25,-90.5 1127.25,-54.5 942,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"951.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"961.75,-55 961.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1003.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -8.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1046,-55 1046,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1086.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809677392*->4809677392 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809677392*->4809677392</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M906.41,-72.5C913.45,-72.5 921.5,-72.5 930.05,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"930.04,-76 940.04,-72.5 930.04,-69 930.04,-76\"/>\n",
"</g>\n",
"<!-- 4809671760 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809671760</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.62,-83.5 2.62,-119.5 185.62,-119.5 185.62,-83.5 2.62,-83.5\"/>\n",
"<text text-anchor=\"middle\" x=\"13.62\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-84 24.62,-119.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.5\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-84 104.38,-119.5\"/>\n",
"<text text-anchor=\"middle\" x=\"145\" y=\"-96.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809671888* -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809671888*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"251.25\" cy=\"-73.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"251.25\" y=\"-68.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809671760->4809671888* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809671760->4809671888*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M185.81,-85.13C195.66,-83.35 205.21,-81.63 213.82,-80.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"214.19,-83.57 223.41,-78.34 212.95,-76.68 214.19,-83.57\"/>\n",
"</g>\n",
"<!-- 4809674896 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4809674896</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-28.5 0,-64.5 188.25,-64.5 188.25,-28.5 0,-28.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-29 22.75,-64.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-29 107,-64.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.62\" y=\"-41.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809674896->4809671888* -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4809674896->4809671888*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.49,-62.75C197.27,-64.28 205.77,-65.76 213.51,-67.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"212.66,-70.51 223.11,-68.78 213.86,-63.61 212.66,-70.51\"/>\n",
"</g>\n",
"<!-- 4809671888 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809671888</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"315.38,-55.5 315.38,-91.5 502.88,-91.5 502.88,-55.5 315.38,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"326.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"337.38,-56 337.38,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"421.62,-56 421.62,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"462.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809671888->4809677328+ -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809671888->4809677328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M503.05,-56.81C512.25,-55.16 521.16,-53.56 529.24,-52.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"529.71,-55.58 538.93,-50.36 528.47,-48.69 529.71,-55.58\"/>\n",
"</g>\n",
"<!-- 4809671888*->4809671888 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809671888*->4809671888</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M278.69,-73.5C286.1,-73.5 294.64,-73.5 303.71,-73.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"303.51,-77 313.51,-73.5 303.51,-70 303.51,-77\"/>\n",
"</g>\n",
"<!-- 4809676112 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4809676112</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"314.25,-0.5 314.25,-36.5 504,-36.5 504,-0.5 314.25,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"325.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"336.25,-1 336.25,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"422.75,-1 422.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"463.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809676112->4809677328+ -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4809676112->4809677328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M504.39,-34.83C513.16,-36.35 521.65,-37.82 529.37,-39.15\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"528.5,-42.56 538.95,-40.81 529.69,-35.66 528.5,-42.56\"/>\n",
"</g>\n",
"<!-- 4809677200 -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4809677200</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"630,-82.5 630,-118.5 816,-118.5 816,-82.5 630,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"640.25\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"650.5,-83 650.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"692.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.75,-83 734.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"775.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809677200->4809677392* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4809677200->4809677392*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M816.25,-83.73C825.1,-82.12 833.67,-80.56 841.46,-79.14\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"841.93,-82.62 851.15,-77.38 840.68,-75.73 841.93,-82.62\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11eae3950>"
]
},
"execution_count": 92,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(L)"
]
},
{
"cell_type": "markdown",
"id": "155a0fa9-48fb-431b-b552-01925e342a20",
"metadata": {},
"source": [
"### Let's recap. \n",
"\n",
"We've built out scalar valued mathematical expressions using $+$ and $*$. We can do a forward pass with multiple inputs going into the expression that produces the single output ```L```. ```Draw_dot(L)``` visualizes the forward pass. \n",
"\n",
"Next, we run backpropagation, where we reverse and calculate the gradient along all of the intermediate values. For every single value, we compute the derivative of that node with respect to ```L```. The derivative of ```L``` with respect to ```L``` is ```1```. Now, we derive ```L``` with respect to ```f, d, c, e, b,``` and then ```a```|. \n",
"\n",
"In a neural network setting, you would be very interested in the derivative of the loss function ```L``` with respect to the weights of the neural network. These variables will eventually represent the weights of the neural network. We need to know how those weights are impacting the loss function, so we would be interested in the derivative of the output with respect to some of its leaf nodes. The leaf nodes will be weigths of the neural net. And the other leaf nodes will be the data itself, but we won't take the derivative of these, because the data is fixed, but not the weights.\n",
"\n",
"\n",
"\n",
"In the next exercise, we will focus on the ```self.grad``` in the Value class above, which will maintain the derivative of ```L``` with respect to a ```Value```. It will be initialized to $0$, which means it will start with no impact on the output (changing the variable does not change the loss function)."
]
},
{
"cell_type": "markdown",
"id": "5df4905a-7660-41d9-83f1-b9da760a39ba",
"metadata": {},
"source": [
"### We will start to fill in the gradients.\n",
"\n",
"We'll implement a dummy function with local variables (not affecting the local scope). Read the commments and follow along."
]
},
{
"cell_type": "code",
"execution_count": 102,
"id": "c2708b1a-87ed-436e-9cdf-be77f47a85b7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6.000000000021544\n"
]
}
],
"source": [
"def dummyFunction():\n",
" h = 0.0001 #small a mount to add\n",
"\n",
" #same as last time\n",
" a = Value(2.0, label='a')\n",
" b = Value(-3.0, label='b')\n",
" c = Value(10.0, label='c')\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L1 = L.data #use.data because they're floats\n",
"\n",
" a = Value(2.0 + h, label='a') #we choose to bump a by h, so we can later get the derivative of L with respect to a\n",
" b = Value(-3.0, label='b')\n",
" c = Value(10.0, label='c')\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L2 = L.data # base case would be adding h to L -> L2 = L.data + h = 1\n",
"\n",
" print((L2 - L1)/h) #rise over run -> change in L over tiny change to a\n",
" \n",
" #should print derivative of L with respect to a because a is what we bumped by h. Answer is 6. Check output.\n",
"\n",
" \n",
" \n",
" #Adding a small amount (h) to a -> measures L with respect to a\n",
"\n",
"#when testing derivatives, plug in ( + h) to any of the value objects in L2. For example, if finding the derivative of L with respect to f,\n",
"#add h to -2.0 in f's value object\n",
"\n",
"dummyFunction()"
]
},
{
"cell_type": "markdown",
"id": "a6ee23a5-1b15-441b-b4e9-3e026be7892b",
"metadata": {},
"source": [
"Now, let's set a gradient:"
]
},
{
"cell_type": "code",
"execution_count": 104,
"id": "274cae34-245c-4401-98a8-ef6b18ed8c08",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1135pt\" height=\"154pt\"\n",
" viewBox=\"0.00 0.00 1135.25 154.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 150)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-150 1131.25,-150 1131.25,4 -4,4\"/>\n",
"<!-- 4809743376 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809743376</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"942,-81.5 942,-117.5 1127.25,-117.5 1127.25,-81.5 942,-81.5\"/>\n",
"<text text-anchor=\"middle\" x=\"951.88\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"961.75,-82 961.75,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1003.88\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -8.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1046,-82 1046,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1086.62\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4809743376* -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809743376*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"879\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"879\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809743376*->4809743376 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809743376*->4809743376</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M906.41,-99.5C913.45,-99.5 921.5,-99.5 930.05,-99.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"930.04,-103 940.04,-99.5 930.04,-96 930.04,-103\"/>\n",
"</g>\n",
"<!-- 4809743440 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809743440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"630,-109.5 630,-145.5 816,-145.5 816,-109.5 630,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"640.25\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"650.5,-110 650.5,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"692.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.75,-110 734.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"775.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809743440->4809743376* -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809743440->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M816.25,-110.73C825.1,-109.12 833.67,-107.56 841.46,-106.14\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"841.93,-109.62 851.15,-104.38 840.68,-102.73 841.93,-109.62\"/>\n",
"</g>\n",
"<!-- 4631919184 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4631919184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"314.25,-82.5 314.25,-118.5 504,-118.5 504,-82.5 314.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"325.25\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"336.25,-83 336.25,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"422.75,-83 422.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"463.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809743184+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4809743184+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"567\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"567\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4631919184->4809743184+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4631919184->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M504.39,-83.57C513.16,-81.99 521.65,-80.47 529.37,-79.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"529.73,-82.57 538.95,-77.36 528.49,-75.68 529.73,-82.57\"/>\n",
"</g>\n",
"<!-- 4809663696 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809663696</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 188.25,-91.5 188.25,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-56 22.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-56 107,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809741072* -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809741072*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"251.25\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"251.25\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809663696->4809741072* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809663696->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.49,-56.65C197.36,-55.05 205.93,-53.5 213.73,-52.09\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"214.2,-55.56 223.42,-50.34 212.95,-48.68 214.2,-55.56\"/>\n",
"</g>\n",
"<!-- 4806767312 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4806767312</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.62,-0.5 2.62,-36.5 185.62,-36.5 185.62,-0.5 2.62,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"13.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-1 24.62,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-1 104.38,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"145\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4806767312->4809741072* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4806767312->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M185.81,-34.28C195.45,-35.96 204.82,-37.59 213.29,-39.07\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"212.65,-42.51 223.1,-40.77 213.85,-35.61 212.65,-42.51\"/>\n",
"</g>\n",
"<!-- 4809741072 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809741072</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"315.38,-27.5 315.38,-63.5 502.88,-63.5 502.88,-27.5 315.38,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"326.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"337.38,-28 337.38,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"421.62,-28 421.62,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"462.25\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809741072->4809743184+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809741072->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M503.05,-61.6C512.25,-63.19 521.16,-64.73 529.24,-66.13\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"528.48,-69.55 538.93,-67.81 529.68,-62.66 528.48,-69.55\"/>\n",
"</g>\n",
"<!-- 4809741072*->4809741072 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809741072*->4809741072</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M278.69,-45.5C286.1,-45.5 294.64,-45.5 303.71,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"303.51,-49 313.51,-45.5 303.51,-42 303.51,-49\"/>\n",
"</g>\n",
"<!-- 4809743184 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4809743184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"631.12,-54.5 631.12,-90.5 814.88,-90.5 814.88,-54.5 631.12,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"642.5\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"653.88,-55 653.88,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"693.75\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"733.62,-55 733.62,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"774.25\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809743184->4809743376* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4809743184->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M815.36,-88.52C824.5,-90.12 833.36,-91.67 841.4,-93.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"840.58,-96.49 851.04,-94.77 841.79,-89.6 840.58,-96.49\"/>\n",
"</g>\n",
"<!-- 4809743184+->4809743184 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809743184+->4809743184</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M594.47,-72.5C601.87,-72.5 610.38,-72.5 619.43,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"619.17,-76 629.17,-72.5 619.17,-69 619.17,-76\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11eb229d0>"
]
},
"execution_count": 104,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"L.grad = 1.0 #manual backpropagation\n",
"draw_dot(L) #look at L.grad"
]
},
{
"cell_type": "markdown",
"id": "21622f28-4a5d-4ff7-b5ed-3a2a29730528",
"metadata": {},
"source": [
"Now, we want to get the value of ```d.grad``` and ```f.grad```. To do this, we derive $dL/dd$ and $dL/df$:\n",
"\n",
"$L = d * f$\n",
"\n",
"$dL/dd = ?$ (ends up being ```f```)\n",
"\n",
"Proof:\n",
"\n",
"$f(x + h) - f(x)) / h =$\n",
"\n",
"$((d + h) * f - d * f) / h =$ \n",
"\n",
"$(d+f + h * f - d * f) / h =$ \n",
"\n",
"$(h*f)/h =$ \n",
"\n",
"$f$\n",
"\n",
"\n",
"**The same is true for $dL/df$; it ends up deriving to $d$. Let's set the grads and redraw.**\n"
]
},
{
"cell_type": "code",
"execution_count": 108,
"id": "a68d396a-63a3-4982-b5f9-675831338120",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1138pt\" height=\"154pt\"\n",
" viewBox=\"0.00 0.00 1137.50 154.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 150)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-150 1133.5,-150 1133.5,4 -4,4\"/>\n",
"<!-- 4809743376 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809743376</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"944.25,-81.5 944.25,-117.5 1129.5,-117.5 1129.5,-81.5 944.25,-81.5\"/>\n",
"<text text-anchor=\"middle\" x=\"954.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"964,-82 964,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1006.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -8.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1048.25,-82 1048.25,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1088.88\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4809743376* -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809743376*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"881.25\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"881.25\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809743376*->4809743376 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809743376*->4809743376</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M908.66,-99.5C915.7,-99.5 923.75,-99.5 932.3,-99.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"932.29,-103 942.29,-99.5 932.29,-96 932.29,-103\"/>\n",
"</g>\n",
"<!-- 4809743440 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809743440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"631.12,-109.5 631.12,-145.5 817.12,-145.5 817.12,-109.5 631.12,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"641.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"651.62,-110 651.62,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"693.75\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"735.88,-110 735.88,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"776.5\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
"</g>\n",
"<!-- 4809743440->4809743376* -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809743440->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M817.6,-110.81C826.76,-109.16 835.62,-107.56 843.67,-106.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"844.09,-109.58 853.31,-104.36 842.84,-102.7 844.09,-109.58\"/>\n",
"</g>\n",
"<!-- 4631919184 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4631919184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"314.25,-82.5 314.25,-118.5 504,-118.5 504,-82.5 314.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"325.25\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"336.25,-83 336.25,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"422.75,-83 422.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"463.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809743184+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4809743184+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"567\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"567\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4631919184->4809743184+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4631919184->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M504.39,-83.57C513.16,-81.99 521.65,-80.47 529.37,-79.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"529.73,-82.57 538.95,-77.36 528.49,-75.68 529.73,-82.57\"/>\n",
"</g>\n",
"<!-- 4809663696 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809663696</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 188.25,-91.5 188.25,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-56 22.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-56 107,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809741072* -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809741072*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"251.25\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"251.25\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809663696->4809741072* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809663696->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.49,-56.65C197.36,-55.05 205.93,-53.5 213.73,-52.09\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"214.2,-55.56 223.42,-50.34 212.95,-48.68 214.2,-55.56\"/>\n",
"</g>\n",
"<!-- 4806767312 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4806767312</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.62,-0.5 2.62,-36.5 185.62,-36.5 185.62,-0.5 2.62,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"13.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-1 24.62,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-1 104.38,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"145\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4806767312->4809741072* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4806767312->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M185.81,-34.28C195.45,-35.96 204.82,-37.59 213.29,-39.07\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"212.65,-42.51 223.1,-40.77 213.85,-35.61 212.65,-42.51\"/>\n",
"</g>\n",
"<!-- 4809741072 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809741072</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"315.38,-27.5 315.38,-63.5 502.88,-63.5 502.88,-27.5 315.38,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"326.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"337.38,-28 337.38,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"421.62,-28 421.62,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"462.25\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809741072->4809743184+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809741072->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M503.05,-61.6C512.25,-63.19 521.16,-64.73 529.24,-66.13\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"528.48,-69.55 538.93,-67.81 529.68,-62.66 528.48,-69.55\"/>\n",
"</g>\n",
"<!-- 4809741072*->4809741072 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809741072*->4809741072</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M278.69,-45.5C286.1,-45.5 294.64,-45.5 303.71,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"303.51,-49 313.51,-45.5 303.51,-42 303.51,-49\"/>\n",
"</g>\n",
"<!-- 4809743184 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4809743184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"630,-54.5 630,-90.5 818.25,-90.5 818.25,-54.5 630,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"641.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"652.75,-55 652.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"692.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"732.5,-55 732.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"775.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809743184->4809743376* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4809743184->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M818.49,-88.75C827.27,-90.28 835.77,-91.76 843.51,-93.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"842.66,-96.51 853.11,-94.78 843.86,-89.61 842.66,-96.51\"/>\n",
"</g>\n",
"<!-- 4809743184+->4809743184 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809743184+->4809743184</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M594.31,-72.5C601.49,-72.5 609.72,-72.5 618.47,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"618.29,-76 628.29,-72.5 618.29,-69 618.29,-76\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11eb73ad0>"
]
},
"execution_count": 108,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"f.grad = 4.0 #value of d.data\n",
"d.grad = -2.0 #value of f.data\n",
"draw_dot(L)\n",
"#to test that these work, go back to dummyFunction() and increment d.data or f.data by h, and view the output. It should be the given grad."
]
},
{
"cell_type": "markdown",
"id": "cbf738fa-3a76-46ae-8db1-b56e057e9bc2",
"metadata": {},
"source": [
"### Now, we are continuing the backpropagation by calculating the derivative of L with respect to C. Also will derive dL by de (dL/de). \n",
"\n",
"dL/dc : derivative of L with respect to C. \n",
"\n",
"How do we derive dL by dC?\n",
"\n",
"\n",
"We already know derivative of ```L``` with respect to ```d```. So we know how ```L``` is sensitive to ```d```. But how is ```L``` sensitive to ```C```?\n",
"So if we adjust ```C```, how does that impact ```L``` through ```d```? if know impact ```c``` on ```d``` and ```d``` on ```L```, should be able to put it together to figure out how ```c``` impacts ```L```.\n",
"\n",
"```dd/dc```?\n",
"\n",
"$d = c + e$ -> differentiating $c+e$ with respect to $c$ gives you $1 + 0$. \n",
"\n",
"Can derive/prove this:\n",
"\n",
"$f(x+h) - f(x) / h$\n",
"\n",
"$dd/dc = ((c + e + h) - ( c + e )) / h$\n",
"\n",
"$(c + h + e -c -e) / h -> h/h = 1.0$\n",
"\n",
"$dd/dc = 1.0$\n",
"\n",
"Symetrically, $dd/de = 1.0$.\n",
"\n",
"The local derivative of the sum is very simple. All is knows is the local ```+```. The + node only nodes the derivatives of ```d``` with respect to ```e```, and ```d``` with respect to ```c```. And it knows that it took ```c``` and ```e``` and added them and made ```d```. This little ```+``` node can be embedded in a massive graph. \n",
"\n",
"Now we know how to derive ```c``` and ```e``` from ```d```, and ```d``` from ```c```, but how do we put it all together? We ultimately want not just the local derivative, but $dL/dc$.\n",
"\n",
"Answer: the chain rule in calculus: https://en.wikipedia.org/wiki/Chain_rule"
]
},
{
"cell_type": "markdown",
"id": "0db67463-cce9-4ba7-9eee-c2facf227261",
"metadata": {},
"source": [
"## Chain rule explained:\n",
"\n",
"- If a variable ```z``` depends on the variable ```y```, and ```y``` depends on the variable ```x```, (that is, ```y``` and ```z``` are dependent variables), then ```z``` depends on ```x``` as well, via the intermediate variable ```y```. In this case, the chain rule is expressed as $dz/dx = dz/dy * dy/dx$\n",
"\n",
"- For our use case, $dL/dc = dL/dd * dd/dc$\n",
"\n",
"- **This is how we chain these derivatives together, correctly. To differentiate through a function composition, we have to apply a multiplication of those derivatives.**\n",
"\n",
"- Knowing the instantaneous rate of change of ```z``` relative to ```y``` and that of ```y``` relative to ```x``` allows one to calculate the instantaenous rate of change of ```z``` relative to ```x``` as the product of the two rates of change.\n",
"\n",
"### Example: \n",
"\n",
"- If a ```car``` travels $2x$ as fast as a ```bicycle``` and the ```bicycle``` is $4x$ as fast as the ```walking man```, then the car travels $2 x 4 = 8x$ as fast as the man.\n",
"- If we use the chain rule: let ```z```, ```y```, and ```x``` be the variable positions of the ```car```, ```bicycle```, and the ```walking man```. The rate of change of the relative poisitons of the ```car``` and the ```bicycle``` is $dz / dy = 2$. Then, $dy / dx = 4$. So the rate of change of the relative positions of the ```car``` and the ```walking man``` is $dz/dx = dz/dy * dy/dx = 2 * 4 = 8$"
]
},
{
"cell_type": "markdown",
"id": "b86843ef-c098-4340-bfbf-db3e111cfc74",
"metadata": {},
"source": [
"### Now that we have an idea about the chain rule, let's find dL/dc.\n",
"\n",
"$dd/de = 1.0$\n",
"$dd/dc = 1.0$ -> local derivatives are 1\n",
"$d = c + e$\n",
"\n",
"KNOW: \n",
"$dL / dd = -2$\n",
"$dd / dc = 1$\n",
"\n",
"WANT:\n",
"Chain Rule: $dL / dc = (dL / dd) * (dd / dc)$\n",
"\n",
"A $(+)$ node is routes the gradient because the $(+)$ node's gradients are just $1$\n",
"So in the chain rule, $1 x dL/dd = dL/dd$, so the derivative just gets routed to $c$ and $e$\n",
"\n",
"Backpropogating signal that is flowing backwards through the graph, carrying the info of what the dL with respect to all the intermediate nodes, a $(+)$ node simply distributes a derivative to all its children\n"
]
},
{
"cell_type": "code",
"execution_count": 110,
"id": "18382f97-1ba1-49e1-b948-dff5e5516a36",
"metadata": {},
"outputs": [],
"source": [
"#Thus... \n",
"c.grad = -2.0 #-2 * 1 -> dL/dc\n",
"e.grad = -2.0 #-2 * 1 -> dL/de"
]
},
{
"cell_type": "code",
"execution_count": 114,
"id": "bdede23f-1b96-4e77-881b-f88db1e2628f",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1142pt\" height=\"154pt\"\n",
" viewBox=\"0.00 0.00 1142.00 154.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 150)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-150 1138,-150 1138,4 -4,4\"/>\n",
"<!-- 4809743376 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809743376</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"948.75,-81.5 948.75,-117.5 1134,-117.5 1134,-81.5 948.75,-81.5\"/>\n",
"<text text-anchor=\"middle\" x=\"958.62\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"968.5,-82 968.5,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1010.62\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -8.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1052.75,-82 1052.75,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1093.38\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4809743376* -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809743376*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"885.75\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"885.75\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809743376*->4809743376 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809743376*->4809743376</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M913.16,-99.5C920.2,-99.5 928.25,-99.5 936.8,-99.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"936.79,-103 946.79,-99.5 936.79,-96 936.79,-103\"/>\n",
"</g>\n",
"<!-- 4809743440 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809743440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"635.62,-109.5 635.62,-145.5 821.62,-145.5 821.62,-109.5 635.62,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"645.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"656.12,-110 656.12,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"698.25\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"740.38,-110 740.38,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"781\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
"</g>\n",
"<!-- 4809743440->4809743376* -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809743440->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M822.1,-110.81C831.26,-109.16 840.12,-107.56 848.17,-106.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"848.59,-109.58 857.81,-104.36 847.34,-102.7 848.59,-109.58\"/>\n",
"</g>\n",
"<!-- 4631919184 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4631919184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"314.25,-82.5 314.25,-118.5 508.5,-118.5 508.5,-82.5 314.25,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"325.25\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"336.25,-83 336.25,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"422.75,-83 422.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"465.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809743184+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4809743184+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"571.5\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"571.5\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4631919184->4809743184+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4631919184->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M508.91,-83.41C517.71,-81.85 526.2,-80.34 533.92,-78.98\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"534.27,-82.47 543.5,-77.28 533.05,-75.58 534.27,-82.47\"/>\n",
"</g>\n",
"<!-- 4809663696 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809663696</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 188.25,-91.5 188.25,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-56 22.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-56 107,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.62\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4809741072* -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809741072*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"251.25\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"251.25\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809663696->4809741072* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809663696->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.49,-56.65C197.36,-55.05 205.93,-53.5 213.73,-52.09\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"214.2,-55.56 223.42,-50.34 212.95,-48.68 214.2,-55.56\"/>\n",
"</g>\n",
"<!-- 4806767312 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4806767312</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"2.62,-0.5 2.62,-36.5 185.62,-36.5 185.62,-0.5 2.62,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"13.62\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"24.62,-1 24.62,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.5\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"104.38,-1 104.38,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"145\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4806767312->4809741072* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4806767312->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M185.81,-34.28C195.45,-35.96 204.82,-37.59 213.29,-39.07\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"212.65,-42.51 223.1,-40.77 213.85,-35.61 212.65,-42.51\"/>\n",
"</g>\n",
"<!-- 4809741072 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809741072</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"315.38,-27.5 315.38,-63.5 507.38,-63.5 507.38,-27.5 315.38,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"326.38\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"337.38,-28 337.38,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"379.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"421.62,-28 421.62,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"464.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809741072->4809743184+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809741072->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M507.55,-61.75C516.78,-63.33 525.7,-64.85 533.79,-66.23\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"533.03,-69.65 543.48,-67.89 534.21,-62.75 533.03,-69.65\"/>\n",
"</g>\n",
"<!-- 4809741072*->4809741072 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809741072*->4809741072</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M278.72,-45.5C286.09,-45.5 294.59,-45.5 303.63,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"303.39,-49 313.39,-45.5 303.39,-42 303.39,-49\"/>\n",
"</g>\n",
"<!-- 4809743184 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4809743184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"634.5,-54.5 634.5,-90.5 822.75,-90.5 822.75,-54.5 634.5,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"645.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"657.25,-55 657.25,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"697.12\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"737,-55 737,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"779.88\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809743184->4809743376* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4809743184->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M822.99,-88.75C831.77,-90.28 840.27,-91.76 848.01,-93.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"847.16,-96.51 857.61,-94.78 848.36,-89.61 847.16,-96.51\"/>\n",
"</g>\n",
"<!-- 4809743184+->4809743184 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809743184+->4809743184</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M598.81,-72.5C605.99,-72.5 614.22,-72.5 622.97,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"622.79,-76 632.79,-72.5 622.79,-69 622.79,-76\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11eaca6d0>"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(L) #now we have the derivative of L with respect to c and e (grad for c and e)\n",
"#This backpropogating signal is carrying information of the gradients from the end to the start through the intermediate nodes.\n",
"#The + node distributes the derivative to its children."
]
},
{
"cell_type": "markdown",
"id": "886bc950-e801-4160-b292-4670bf89e824",
"metadata": {},
"source": [
"Now, let's increment c.data instead in our dummy function. Can you guess what we'll get as our answer for $(L2-L1)/h$?"
]
},
{
"cell_type": "code",
"execution_count": 128,
"id": "ddbc0dd7-81fd-4bbf-8e63-5d5e2a098b87",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-1.9999999999953388\n"
]
}
],
"source": [
"def dummyFunction():\n",
" h = 0.0001 #small a mount to add\n",
"\n",
" a = Value(2.0, label='a')\n",
" b = Value(-3.0, label='b')\n",
" c = Value(10.0, label='c')\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L1 = L.data \n",
"\n",
" a = Value(2.0, label='a') \n",
" b = Value(-3.0, label='b')\n",
" c = Value(10.0, label='c')\n",
" c.data += h #increment c by h\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L2 = L.data \n",
"\n",
" print((L2 - L1)/h) \n",
" \n",
"dummyFunction()"
]
},
{
"cell_type": "markdown",
"id": "30690d53-386d-4096-9554-acf6d22d7f3a",
"metadata": {},
"source": [
"As expected! It's -2, the same as c.grad."
]
},
{
"cell_type": "markdown",
"id": "b453d5f5-0a57-44ff-b535-11a690fed9ee",
"metadata": {},
"source": [
"Now we are going to keep applying the chain rule recursively through the graph. Now, we have dL/de is -2:"
]
},
{
"cell_type": "markdown",
"id": "f70ef327-b8e7-4525-b387-26b9eb5d1431",
"metadata": {},
"source": [
"$dL / de = -2.0$\n",
"\n",
"$dL / da = (dL / de) * (de / da)$ (Chain Rule)\n",
"\n",
"$dL / db = (dL / de) * (de / db)$ (Chain Rule)\n",
"\n",
"So, we need to find out $de/da$ and $de/db$.\n",
"\n",
"$de/da = a * b; e = a * b$\n",
"\n",
"To get $de/da$, we differentiate it as we did before with two variables joined by a multiplication operator. That will give us $de/da = b.data$: ```b = -3.0```\n",
"\n",
"As a reminder, the formula to differentiate is: $f(x) = f(x + h) -f(x) / h$\n",
"\n",
"calculating the rate of change of $e$ with respect to $a$, and calculating the difference in the variable $e$ when $a$ is increased by small amount $h$ and then divide that difference by $h$.\n",
"\n",
"$de/da = lim h-> 0: e(a + h) - e(a) / h$ -> then incrememt by $h$. For the proof for $de/da$:\n",
"\n",
"$e(a + h) / h = e(a) / h$\n",
"\n",
"$e(a + h) / h = (a + h) * b / h$\n",
"\n",
"$e(a + h) = (a + h) * b$\n",
"\n",
"$e(a + h) - e(a) = (a + h) * b - a * b$\n",
"\n",
"$(a + h) * b - a * b = a * b + b * h - a * b$\n",
"\n",
"$= h * b$\n",
"\n",
"Divide by $h: (e(a+h)-e(a))/h = (h * b) / h$\n",
"\n",
"$= b$\n",
" \n",
"$dL / da = (dL / de) * (de / da) = (dL / de) * b$\n",
"\n",
"$dL / db = (dL / de) * (de / db) = (dL / de) * a$\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "02fc7b27-cb75-40a6-a3a7-3e7abdc06129",
"metadata": {},
"source": [
"### That's how we derive $de/da$ and $de/db$, and put it all together the chain rule to get $dL/da$ and $dL/db$. Let's set those gradients and redraw:"
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "0174ae54-8fe8-43ef-8276-0f748831d8f7",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1147pt\" height=\"154pt\"\n",
" viewBox=\"0.00 0.00 1146.50 154.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 150)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-150 1142.5,-150 1142.5,4 -4,4\"/>\n",
"<!-- 4809743376 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4809743376</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"953.25,-81.5 953.25,-117.5 1138.5,-117.5 1138.5,-81.5 953.25,-81.5\"/>\n",
"<text text-anchor=\"middle\" x=\"963.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"973,-82 973,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1015.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -8.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1057.25,-82 1057.25,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1097.88\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 1.0000</text>\n",
"</g>\n",
"<!-- 4809743376* -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4809743376*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"890.25\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"890.25\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809743376*->4809743376 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4809743376*->4809743376</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M917.66,-99.5C924.7,-99.5 932.75,-99.5 941.3,-99.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"941.29,-103 951.29,-99.5 941.29,-96 941.29,-103\"/>\n",
"</g>\n",
"<!-- 4809743440 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4809743440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"640.12,-109.5 640.12,-145.5 826.12,-145.5 826.12,-109.5 640.12,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"650.38\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"660.62,-110 660.62,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"702.75\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"744.88,-110 744.88,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"785.5\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
"</g>\n",
"<!-- 4809743440->4809743376* -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4809743440->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M826.6,-110.81C835.76,-109.16 844.62,-107.56 852.67,-106.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"853.09,-109.58 862.31,-104.36 851.84,-102.7 853.09,-109.58\"/>\n",
"</g>\n",
"<!-- 4631919184 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4631919184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"318.75,-82.5 318.75,-118.5 513,-118.5 513,-82.5 318.75,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"329.75\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"340.75,-83 340.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"384\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 10.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"427.25,-83 427.25,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"470.12\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809743184+ -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4809743184+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"576\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"576\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4631919184->4809743184+ -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4631919184->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M513.41,-83.41C522.21,-81.85 530.7,-80.34 538.42,-78.98\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"538.77,-82.47 548,-77.28 537.55,-75.58 538.77,-82.47\"/>\n",
"</g>\n",
"<!-- 4809663696 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4809663696</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-55.5 0,-91.5 192.75,-91.5 192.75,-55.5 0,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-56 22.75,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-56 107,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"149.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -4.0000</text>\n",
"</g>\n",
"<!-- 4809741072* -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809741072*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"255.75\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"255.75\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4809663696->4809741072* -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4809663696->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M193,-56.49C201.81,-54.92 210.32,-53.41 218.06,-52.03\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"218.43,-55.52 227.66,-50.32 217.21,-48.62 218.43,-55.52\"/>\n",
"</g>\n",
"<!-- 4806767312 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4806767312</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"4.88,-0.5 4.88,-36.5 187.88,-36.5 187.88,-0.5 4.88,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"15.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"26.88,-1 26.88,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"66.75\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"106.62,-1 106.62,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.25\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 6.0000</text>\n",
"</g>\n",
"<!-- 4806767312->4809741072* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4806767312->4809741072*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.01,-34.05C198.46,-35.84 208.62,-37.59 217.74,-39.15\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"217.11,-42.59 227.56,-40.84 218.3,-35.69 217.11,-42.59\"/>\n",
"</g>\n",
"<!-- 4809741072 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4809741072</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"319.88,-27.5 319.88,-63.5 511.88,-63.5 511.88,-27.5 319.88,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"330.88\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">e</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"341.88,-28 341.88,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"384\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"426.12,-28 426.12,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"469\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809741072->4809743184+ -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809741072->4809743184+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M512.05,-61.75C521.28,-63.33 530.2,-64.85 538.29,-66.23\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"537.53,-69.65 547.98,-67.89 538.71,-62.75 537.53,-69.65\"/>\n",
"</g>\n",
"<!-- 4809741072*->4809741072 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4809741072*->4809741072</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M283.22,-45.5C290.59,-45.5 299.09,-45.5 308.13,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"307.89,-49 317.89,-45.5 307.89,-42 307.89,-49\"/>\n",
"</g>\n",
"<!-- 4809743184 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4809743184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"639,-54.5 639,-90.5 827.25,-90.5 827.25,-54.5 639,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"650.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">d</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"661.75,-55 661.75,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"701.62\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 4.0000</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"741.5,-55 741.5,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"784.38\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4809743184->4809743376* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4809743184->4809743376*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M827.49,-88.75C836.27,-90.28 844.77,-91.76 852.51,-93.1\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"851.66,-96.51 862.11,-94.78 852.86,-89.61 851.66,-96.51\"/>\n",
"</g>\n",
"<!-- 4809743184+->4809743184 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4809743184+->4809743184</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M603.31,-72.5C610.49,-72.5 618.72,-72.5 627.47,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"627.29,-76 637.29,-72.5 627.29,-69 627.29,-76\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11e83c4d0>"
]
},
"execution_count": 117,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a.grad = -2.0 * -3.0 \n",
"b.grad = -2.0 * 2.0\n",
"draw_dot(L)"
]
},
{
"cell_type": "markdown",
"id": "1fa8fdb8-3c1a-44d0-93af-a0203614c731",
"metadata": {},
"source": [
"Now, let's make sure our claims of ```a.grad``` and ```b.grad``` are true. Below we increment $a$ by $h$ and expect to get ```a.grad```."
]
},
{
"cell_type": "code",
"execution_count": 123,
"id": "e642c1be-6bc8-480f-80e2-911443fc31d2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6.000000000021544\n"
]
}
],
"source": [
"def dummyFunction():\n",
" h = 0.0001\n",
"\n",
" a = Value(2.0, label='a')\n",
" b = Value(-3.0, label='b')\n",
" c = Value(10.0, label='c')\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L1 = L.data\n",
"\n",
" a = Value(2.0, label='a')\n",
" a.data += h #increment a by h, expect to see a.grad\n",
" b = Value(-3.0, label='b')\n",
" #b.data += h can do the same with b.data\n",
" c = Value(10.0, label='c')\n",
" e = a*b; e.label='e'\n",
" d = e + c; d.label='d'\n",
" f = Value(-2.0, label='f')\n",
" L = d*f; L.label = 'L'\n",
" L2 = L.data\n",
"\n",
" print((L2 - L1)/h)\n",
"\n",
"dummyFunction()"
]
},
{
"cell_type": "markdown",
"id": "1d0c17bf-c608-4c7f-857e-f0a48dcbc6d5",
"metadata": {},
"source": [
"### Manual Backpropogation Summary\n",
"\n",
"- iterated through all nodes 1 by 1 and locally applied the chain rule\n",
"- We always know what the derivative of L is with respect to the local output\n",
"- The output was produced through some operation, and we have the pointers to its children nodes of the operation\n",
"- In this operation, we know what the local derivatives are, and we just multiply them onto the derivative\n",
"- We just go through and recursively multiply on the local derivatives\n",
"- That's backpropatation!\n",
"- **Backpropagation is just a recursive application of the chain rule backwards through the computational graph**"
]
},
{
"cell_type": "markdown",
"id": "db350b13-171c-41e5-8d36-bbe7d6879ef7",
"metadata": {},
"source": [
"Let's see it in action! We'll nudge our inputs to make $L$ go up. If we want $L$ to go up, that means we have to go in the direction of the gradient.\n",
"$a, b, c,$ and $f$ should increase in the direction by a small step size. \n",
"These nodes are the leaf nodes which we have control over. $L$ should go up postively if we nudge in the direction of the gradient."
]
},
{
"cell_type": "code",
"execution_count": 126,
"id": "3afe5248-4d60-47c4-ab11-c5483233652a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-6.586368000000001\n"
]
}
],
"source": [
"#one step of optimization, gradients give us power because we know how to influence the final outcome.\n",
"\n",
"a.data += 0.01 * a.grad\n",
"b.data += 0.01 * b.grad\n",
"c.data += 0.01 * c.grad\n",
"f.data += 0.01 * f.grad\n",
"\n",
"#re-run forward pass\n",
"e = a * b\n",
"d = e + c\n",
"L = d * f\n",
"\n",
"print(L.data)\n",
"#because we nudge the leaf nodes in the direction of the gradient, we expect L.data to go up. Right now, it's -8.\n",
"#this is one step of an optimization of our neural network. The gradients tell us how to influence the final outcome L."
]
},
{
"cell_type": "code",
"execution_count": 127,
"id": "13783e56-8309-4d94-9c9e-81f66befea7a",
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 11.0.0 (20240428.1522)\n",
" -->\n",
"<!-- Pages: 1 -->\n",
"<svg width=\"1138pt\" height=\"154pt\"\n",
" viewBox=\"0.00 0.00 1137.50 154.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 150)\">\n",
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-150 1133.5,-150 1133.5,4 -4,4\"/>\n",
"<!-- 4811876944 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>4811876944</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"944.25,-81.5 944.25,-117.5 1129.5,-117.5 1129.5,-81.5 944.25,-81.5\"/>\n",
"<text text-anchor=\"middle\" x=\"954.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"964,-82 964,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1006.12\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.5864</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"1048.25,-82 1048.25,-117.5\"/>\n",
"<text text-anchor=\"middle\" x=\"1088.88\" y=\"-94.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4811876944* -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>4811876944*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"881.25\" cy=\"-99.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"881.25\" y=\"-94.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4811876944*->4811876944 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>4811876944*->4811876944</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M908.66,-99.5C915.7,-99.5 923.75,-99.5 932.3,-99.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"932.29,-103 942.29,-99.5 932.29,-96 932.29,-103\"/>\n",
"</g>\n",
"<!-- 4631919184 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>4631919184</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"318.75,-82.5 318.75,-118.5 506.25,-118.5 506.25,-82.5 318.75,-82.5\"/>\n",
"<text text-anchor=\"middle\" x=\"329.75\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">c</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"340.75,-83 340.75,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"380.62\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 9.9600</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"420.5,-83 420.5,-118.5\"/>\n",
"<text text-anchor=\"middle\" x=\"463.38\" y=\"-95.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -2.0000</text>\n",
"</g>\n",
"<!-- 4811869328+ -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>4811869328+</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"569.25\" cy=\"-72.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"569.25\" y=\"-67.45\" font-family=\"Times,serif\" font-size=\"14.00\">+</text>\n",
"</g>\n",
"<!-- 4631919184->4811869328+ -->\n",
"<g id=\"edge6\" class=\"edge\">\n",
"<title>4631919184->4811869328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M506.64,-83.65C515.4,-82.06 523.88,-80.53 531.59,-79.13\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"531.96,-82.62 541.18,-77.4 530.71,-75.73 531.96,-82.62\"/>\n",
"</g>\n",
"<!-- 4809743440 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>4809743440</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"632.25,-109.5 632.25,-145.5 818.25,-145.5 818.25,-109.5 632.25,-109.5\"/>\n",
"<text text-anchor=\"middle\" x=\"642.5\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">f</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"652.75,-110 652.75,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"694.88\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -1.9200</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"737,-110 737,-145.5\"/>\n",
"<text text-anchor=\"middle\" x=\"777.62\" y=\"-122.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 4.0000</text>\n",
"</g>\n",
"<!-- 4809743440->4811876944* -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>4809743440->4811876944*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M818.5,-110.73C827.35,-109.12 835.92,-107.56 843.71,-106.14\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"844.18,-109.62 853.4,-104.38 842.93,-102.73 844.18,-109.62\"/>\n",
"</g>\n",
"<!-- 4811869328 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>4811869328</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"634.88,-54.5 634.88,-90.5 815.62,-90.5 815.62,-54.5 634.88,-54.5\"/>\n",
"<text text-anchor=\"middle\" x=\"644.75\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"654.62,-55 654.62,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"694.5\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 3.4304</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"734.38,-55 734.38,-90.5\"/>\n",
"<text text-anchor=\"middle\" x=\"775\" y=\"-67.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4811869328->4811876944* -->\n",
"<g id=\"edge8\" class=\"edge\">\n",
"<title>4811869328->4811876944*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M815.83,-88.21C825.46,-89.89 834.81,-91.53 843.27,-93.02\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"842.63,-96.46 853.09,-94.74 843.84,-89.56 842.63,-96.46\"/>\n",
"</g>\n",
"<!-- 4811869328+->4811869328 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>4811869328+->4811869328</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M596.72,-72.5C604.51,-72.5 613.53,-72.5 623.11,-72.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"623.05,-76 633.05,-72.5 623.05,-69 623.05,-76\"/>\n",
"</g>\n",
"<!-- 4806767312 -->\n",
"<g id=\"node7\" class=\"node\">\n",
"<title>4806767312</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"4.88,-55.5 4.88,-91.5 187.88,-91.5 187.88,-55.5 4.88,-55.5\"/>\n",
"<text text-anchor=\"middle\" x=\"15.88\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">a</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"26.88,-56 26.88,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"66.75\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">data 2.1200</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"106.62,-56 106.62,-91.5\"/>\n",
"<text text-anchor=\"middle\" x=\"147.25\" y=\"-68.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 6.0000</text>\n",
"</g>\n",
"<!-- 4811016976* -->\n",
"<g id=\"node10\" class=\"node\">\n",
"<title>4811016976*</title>\n",
"<ellipse fill=\"none\" stroke=\"black\" cx=\"255.75\" cy=\"-45.5\" rx=\"27\" ry=\"18\"/>\n",
"<text text-anchor=\"middle\" x=\"255.75\" y=\"-40.45\" font-family=\"Times,serif\" font-size=\"14.00\">*</text>\n",
"</g>\n",
"<!-- 4806767312->4811016976* -->\n",
"<g id=\"edge9\" class=\"edge\">\n",
"<title>4806767312->4811016976*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M188.01,-57.37C198.67,-55.48 209.04,-53.63 218.3,-51.99\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"218.65,-55.48 227.88,-50.28 217.42,-48.59 218.65,-55.48\"/>\n",
"</g>\n",
"<!-- 4809663696 -->\n",
"<g id=\"node8\" class=\"node\">\n",
"<title>4809663696</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-36.5 192.75,-36.5 192.75,-0.5 0,-0.5\"/>\n",
"<text text-anchor=\"middle\" x=\"11.38\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">b</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"22.75,-1 22.75,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"64.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -3.0800</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"107,-1 107,-36.5\"/>\n",
"<text text-anchor=\"middle\" x=\"149.88\" y=\"-13.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad -4.0000</text>\n",
"</g>\n",
"<!-- 4809663696->4811016976* -->\n",
"<g id=\"edge7\" class=\"edge\">\n",
"<title>4809663696->4811016976*</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M193,-34.91C201.81,-36.42 210.32,-37.88 218.06,-39.2\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"217.21,-42.61 227.66,-40.85 218.4,-35.71 217.21,-42.61\"/>\n",
"</g>\n",
"<!-- 4811016976 -->\n",
"<g id=\"node9\" class=\"node\">\n",
"<title>4811016976</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"319.88,-27.5 319.88,-63.5 505.12,-63.5 505.12,-27.5 319.88,-27.5\"/>\n",
"<text text-anchor=\"middle\" x=\"329.75\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\"> </text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"339.62,-28 339.62,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"381.75\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">data -6.5296</text>\n",
"<polyline fill=\"none\" stroke=\"black\" points=\"423.88,-28 423.88,-63.5\"/>\n",
"<text text-anchor=\"middle\" x=\"464.5\" y=\"-40.7\" font-family=\"Times,serif\" font-size=\"14.00\">grad 0.0000</text>\n",
"</g>\n",
"<!-- 4811016976->4811869328+ -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>4811016976->4811869328+</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M505.31,-61.52C514.49,-63.12 523.39,-64.67 531.47,-66.08\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"530.71,-69.5 541.16,-67.77 531.91,-62.61 530.71,-69.5\"/>\n",
"</g>\n",
"<!-- 4811016976*->4811016976 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>4811016976*->4811016976</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M283,-45.5C290.42,-45.5 298.97,-45.5 308.07,-45.5\"/>\n",
"<polygon fill=\"black\" stroke=\"black\" points=\"307.88,-49 317.88,-45.5 307.88,-42 307.88,-49\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
],
"text/plain": [
"<graphviz.graphs.Digraph at 0x11ec31d50>"
]
},
"execution_count": 127,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"draw_dot(L)"
]
},
{
"cell_type": "markdown",
"id": "e3248e90-5a31-4063-9a80-505431aca221",
"metadata": {},
"source": [
"## In the next mini-lesson, we will do another, more complex example of backpropogating. We will backpropogate through a neuron..."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}