{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Mining Input Grammars\n",
    "\n",
    "So far, the grammars we have seen have been mostly specified manually – that is, you (or the person knowing the input format) had to design and write a grammar in the first place.  While the grammars we have seen so far have been rather simple, creating a grammar for complex inputs can involve quite some effort.  In this chapter, we therefore introduce techniques that _automatically mine grammars from programs_ – by executing the programs and observing how they process which parts of the input.  In conjunction with a grammar fuzzer, this allows us to \n",
    "1. take a program, \n",
    "2. extract its input grammar, and \n",
    "3. fuzz it with high efficiency and effectiveness, using the concepts in this book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:52.730324Z",
     "iopub.status.busy": "2024-01-18T17:19:52.729866Z",
     "iopub.status.idle": "2024-01-18T17:19:52.797017Z",
     "shell.execute_reply": "2024-01-18T17:19:52.796609Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"640\"\n",
       "            height=\"360\"\n",
       "            src=\"https://www.youtube-nocookie.com/embed/ddM1oL2LYDI\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x105bf71f0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bookutils import YouTubeVideo\n",
    "YouTubeVideo(\"ddM1oL2LYDI\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Prerequisites**\n",
    "\n",
    "* You should have read the [chapter on grammars](Grammars.ipynb).\n",
    "* The [chapter on configuration fuzzing](ConfigurationFuzzer.ipynb) introduces grammar mining for configuration options, as well as observing variables and values during execution.\n",
    "* We use the tracer from the [chapter on coverage](Coverage.ipynb).\n",
    "* The concept of parsing from the [chapter on parsers](Parser.ipynb) is also useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Synopsis\n",
    "<!-- Automatically generated. Do not edit. -->\n",
    "\n",
    "To [use the code provided in this chapter](Importing.ipynb), write\n",
    "\n",
    "```python\n",
    ">>> from fuzzingbook.GrammarMiner import <identifier>\n",
    "```\n",
    "\n",
    "and then make use of the following features.\n",
    "\n",
    "\n",
    "This chapter provides a number of classes to mine input grammars from existing programs.  The function `recover_grammar()` could be the easiest to use.  It takes a function and a set of inputs, and returns a grammar that describes its input language.\n",
    "\n",
    "We apply `recover_grammar()` on a `url_parse()` function that takes and decomposes URLs:\n",
    "\n",
    "```python\n",
    ">>> url_parse('https://www.fuzzingbook.org/')\n",
    ">>> URLS\n",
    "['http://user:pass@www.google.com:80/?q=path#ref',\n",
    " 'https://www.cispa.saarland:80/',\n",
    " 'http://www.fuzzingbook.org/#News']\n",
    "```\n",
    "We extract the input grammar for `url_parse()` using `recover_grammar()`:\n",
    "\n",
    "```python\n",
    ">>> grammar = recover_grammar(url_parse, URLS, files=['urllib/parse.py'])\n",
    ">>> grammar\n",
    "{'<start>': ['<urlsplit@437:url>'],\n",
    " '<urlsplit@437:url>': ['<urlparse@394:scheme>:<_splitnetloc@411:url>'],\n",
    " '<urlparse@394:scheme>': ['https', 'http'],\n",
    " '<_splitnetloc@411:url>': ['//<urlparse@394:netloc>/',\n",
    "  '//<urlparse@394:netloc><urlsplit@481:url>'],\n",
    " '<urlparse@394:netloc>': ['user:pass@www.google.com:80',\n",
    "  'www.fuzzingbook.org',\n",
    "  'www.cispa.saarland:80'],\n",
    " '<urlsplit@481:url>': ['<urlsplit@486:url>#<urlparse@394:fragment>',\n",
    "  '/#<urlparse@394:fragment>'],\n",
    " '<urlsplit@486:url>': ['/?<urlparse@394:query>'],\n",
    " '<urlparse@394:query>': ['q=path'],\n",
    " '<urlparse@394:fragment>': ['ref', 'News']}\n",
    "```\n",
    "The names of nonterminals are a bit technical; but the grammar nicely represents the structure of the input; for instance, the different schemes (`\"http\"`, `\"https\"`) are all identified:\n",
    "\n",
    "```python\n",
    ">>> syntax_diagram(grammar)\n",
    "start\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-1.svg)\n",
    "```\n",
    "urlsplit@437:url\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-2.svg)\n",
    "```\n",
    "urlparse@394:scheme\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-3.svg)\n",
    "```\n",
    "_splitnetloc@411:url\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-4.svg)\n",
    "```\n",
    "urlparse@394:netloc\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-5.svg)\n",
    "```\n",
    "urlsplit@481:url\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-6.svg)\n",
    "```\n",
    "urlsplit@486:url\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-7.svg)\n",
    "```\n",
    "urlparse@394:query\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-8.svg)\n",
    "```\n",
    "urlparse@394:fragment\n",
    "\n",
    "```\n",
    "![](PICS/GrammarMiner-synopsis-9.svg)\n",
    "\n",
    "The grammar can be immediately used for fuzzing, producing arbitrary combinations of input elements, which are all syntactically valid.\n",
    "\n",
    "```python\n",
    ">>> from GrammarCoverageFuzzer import GrammarCoverageFuzzer\n",
    ">>> fuzzer = GrammarCoverageFuzzer(grammar)\n",
    ">>> [fuzzer.fuzz() for i in range(5)]\n",
    "['https://www.fuzzingbook.org/',\n",
    " 'http://user:pass@www.google.com:80/#News',\n",
    " 'http://www.cispa.saarland:80/?q=path#ref',\n",
    " 'https://user:pass@www.google.com:80/#ref',\n",
    " 'https://www.cispa.saarland:80/']\n",
    "```\n",
    "Being able to automatically extract a grammar and to use this grammar for fuzzing makes for very effective test generation with a minimum of manual work.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A Grammar Challenge\n",
    "\n",
    "Consider the `process_inventory()` method from the [chapter on parsers](Parser.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:52.820727Z",
     "iopub.status.busy": "2024-01-18T17:19:52.820505Z",
     "iopub.status.idle": "2024-01-18T17:19:52.823026Z",
     "shell.execute_reply": "2024-01-18T17:19:52.822713Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import bookutils.setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:52.824965Z",
     "iopub.status.busy": "2024-01-18T17:19:52.824826Z",
     "iopub.status.idle": "2024-01-18T17:19:52.826647Z",
     "shell.execute_reply": "2024-01-18T17:19:52.826358Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from typing import List, Tuple, Callable, Any\n",
    "from collections.abc import Iterable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:52.828309Z",
     "iopub.status.busy": "2024-01-18T17:19:52.828171Z",
     "iopub.status.idle": "2024-01-18T17:19:53.379184Z",
     "shell.execute_reply": "2024-01-18T17:19:53.378754Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Parser import process_inventory, process_vehicle, process_car, process_van, lr_graph  # minor dependency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It takes inputs of the following form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.381966Z",
     "iopub.status.busy": "2024-01-18T17:19:53.381582Z",
     "iopub.status.idle": "2024-01-18T17:19:53.383611Z",
     "shell.execute_reply": "2024-01-18T17:19:53.383321Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "INVENTORY = \"\"\"\\\n",
    "1997,van,Ford,E350\n",
    "2000,car,Mercury,Cougar\n",
    "1999,car,Chevy,Venture\\\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.385287Z",
     "iopub.status.busy": "2024-01-18T17:19:53.385163Z",
     "iopub.status.idle": "2024-01-18T17:19:53.387090Z",
     "shell.execute_reply": "2024-01-18T17:19:53.386810Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have a Ford E350 van from 1997 vintage.\n",
      "It is an old but reliable model!\n",
      "We have a Mercury Cougar car from 2000 vintage.\n",
      "It is an old but reliable model!\n",
      "We have a Chevy Venture car from 1999 vintage.\n",
      "It is an old but reliable model!\n"
     ]
    }
   ],
   "source": [
    "print(process_inventory(INVENTORY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We found from the [chapter on parsers](Parser.ipynb) that coarse grammars do not work well for fuzzing when the input format includes details expressed only in code. That is, even though we have the formal specification of CSV files ([RFC 4180](https://tools.ietf.org/html/rfc4180)), the inventory system includes further rules as to what is expected at each index of the CSV file. The solution of simply recombining existing inputs, while practical, is incomplete. In particular, it relies on a formal input specification being available in the first place. However, we have no assurance that the program obeys the input specification given."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One of the ways out of this predicament is to interrogate the program under test as to what its input specification is. That is, if the program under test is written in a style such that specific methods are responsible for handling specific parts of the input, one can recover the parse tree by observing the process of parsing. Further, one can recover a reasonable approximation of the grammar by abstraction from multiple input trees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    " _We start with the assumption (1) that the program is written in such a fashion that specific methods are responsible for parsing specific fragments of the program -- This includes almost all ad hoc parsers._\n",
    "\n",
    "The idea is as follows:\n",
    "\n",
    "* Hook into the Python execution and observe the fragments of input string as they are produced and named in different methods.\n",
    "* Stitch the input fragments together in a tree structure to retrieve the **Parse Tree**.\n",
    "* Abstract common elements from multiple parse trees to produce the **Context Free Grammar** of the input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A Simple Grammar Miner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Say we want to obtain the input grammar for the function `process_vehicle()`. We first collect the sample inputs for this function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.388897Z",
     "iopub.status.busy": "2024-01-18T17:19:53.388784Z",
     "iopub.status.idle": "2024-01-18T17:19:53.390463Z",
     "shell.execute_reply": "2024-01-18T17:19:53.390235Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "VEHICLES = INVENTORY.split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The set of methods responsible for processing inventory are the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.392018Z",
     "iopub.status.busy": "2024-01-18T17:19:53.391906Z",
     "iopub.status.idle": "2024-01-18T17:19:53.393543Z",
     "shell.execute_reply": "2024-01-18T17:19:53.393228Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "INVENTORY_METHODS = {\n",
    "    'process_inventory',\n",
    "    'process_vehicle',\n",
    "    'process_van',\n",
    "    'process_car'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We have seen from the chapter on [configuration fuzzing](ConfigurationFuzzer.ipynb) that one can hook into the Python runtime to observe the arguments to a function and any local variables created. We have also seen that one can obtain the context of execution by inspecting the `frame` argument. Here is a simple tracer that can return the local variables and other contextual information in a traced function. We reuse the `Coverage` tracing class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tracer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.395228Z",
     "iopub.status.busy": "2024-01-18T17:19:53.395112Z",
     "iopub.status.idle": "2024-01-18T17:19:53.396981Z",
     "shell.execute_reply": "2024-01-18T17:19:53.396554Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Coverage import Coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.398958Z",
     "iopub.status.busy": "2024-01-18T17:19:53.398795Z",
     "iopub.status.idle": "2024-01-18T17:19:53.400549Z",
     "shell.execute_reply": "2024-01-18T17:19:53.400257Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import inspect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.402038Z",
     "iopub.status.busy": "2024-01-18T17:19:53.401930Z",
     "iopub.status.idle": "2024-01-18T17:19:53.404269Z",
     "shell.execute_reply": "2024-01-18T17:19:53.403977Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Coverage):\n",
    "    def traceit(self, frame, event, arg):\n",
    "        method_name = inspect.getframeinfo(frame).function\n",
    "        if method_name not in INVENTORY_METHODS:\n",
    "            return\n",
    "        file_name = inspect.getframeinfo(frame).filename\n",
    "\n",
    "        param_names = inspect.getargvalues(frame).args\n",
    "        lineno = inspect.getframeinfo(frame).lineno\n",
    "        local_vars = inspect.getargvalues(frame).locals\n",
    "        print(event, file_name, lineno, method_name, param_names, local_vars)\n",
    "        return self.traceit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We run the code under trace context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.405907Z",
     "iopub.status.busy": "2024-01-18T17:19:53.405804Z",
     "iopub.status.idle": "2024-01-18T17:19:53.456454Z",
     "shell.execute_reply": "2024-01-18T17:19:53.456165Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "call /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 29 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 30 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350'}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 31 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350', '_': []}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 32 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350', '_': []}\n",
      "call /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 40 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 41 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 42 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350', 'res': ['We have a Ford E350 van from 1997 vintage.']}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 43 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 46 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350', 'res': ['We have a Ford E350 van from 1997 vintage.'], 'iyear': 1997}\n",
      "line /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 47 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}\n",
      "return /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 47 process_van ['year', 'company', 'model'] {'year': '1997', 'company': 'Ford', 'model': 'E350', 'res': ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], 'iyear': 1997}\n",
      "return /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb 32 process_vehicle ['vehicle'] {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350', '_': []}\n"
     ]
    }
   ],
   "source": [
    "with Tracer() as tracer:\n",
    "    process_vehicle(VEHICLES[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The main thing that we want out of tracing is a list of assignments of input fragments to different variables. We can use the tracing facility `settrace()` to get that as we showed above.\n",
    "\n",
    "However, the `settrace()` function hooks into the Python debugging facility. When it is in operation, no debugger can hook into the program. That is, if there is a problem with our grammar miner, we will not be able to attach a debugger to it to understand what is happening. This is not ideal. Hence, we limit the tracer to the simplest implementation possible, and implement the core of grammar mining in later stages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `traceit()` function relies on information from the `frame` variable which exposes Python internals. We define a `context` class that encapsulates the information that we need from the `frame`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Context\n",
    "\n",
    "The `Context` class provides easy access to the information such as the current module, and parameter names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.458508Z",
     "iopub.status.busy": "2024-01-18T17:19:53.458395Z",
     "iopub.status.idle": "2024-01-18T17:19:53.461016Z",
     "shell.execute_reply": "2024-01-18T17:19:53.460709Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Context:\n",
    "    def __init__(self, frame, track_caller=True):\n",
    "        self.method = inspect.getframeinfo(frame).function\n",
    "        self.parameter_names = inspect.getargvalues(frame).args\n",
    "        self.file_name = inspect.getframeinfo(frame).filename\n",
    "        self.line_no = inspect.getframeinfo(frame).lineno\n",
    "\n",
    "    def _t(self):\n",
    "        return (self.file_name, self.line_no, self.method,\n",
    "                ','.join(self.parameter_names))\n",
    "\n",
    "    def __repr__(self):\n",
    "        return \"%s:%d:%s(%s)\" % self._t()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here we add a few convenience methods that operate on the `frame` to `Context`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.463101Z",
     "iopub.status.busy": "2024-01-18T17:19:53.462932Z",
     "iopub.status.idle": "2024-01-18T17:19:53.465678Z",
     "shell.execute_reply": "2024-01-18T17:19:53.465331Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Context(Context):\n",
    "    def extract_vars(self, frame):\n",
    "        return inspect.getargvalues(frame).locals\n",
    "\n",
    "    def parameters(self, all_vars):\n",
    "        return {k: v for k, v in all_vars.items() if k in self.parameter_names}\n",
    "\n",
    "    def qualified(self, all_vars):\n",
    "        return {\"%s:%s\" % (self.method, k): v for k, v in all_vars.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We hook printing the context to our `traceit()` to see it in action. First we define a `log_event()` for displaying events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.467730Z",
     "iopub.status.busy": "2024-01-18T17:19:53.467528Z",
     "iopub.status.idle": "2024-01-18T17:19:53.469528Z",
     "shell.execute_reply": "2024-01-18T17:19:53.469238Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def log_event(event, var):\n",
    "    print({'call': '->', 'return': '<-'}.get(event, '  '), var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "And use the `log_event()` in the `traceit()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.471544Z",
     "iopub.status.busy": "2024-01-18T17:19:53.471369Z",
     "iopub.status.idle": "2024-01-18T17:19:53.473421Z",
     "shell.execute_reply": "2024-01-18T17:19:53.473152Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def traceit(self, frame, event, arg):\n",
    "        log_event(event, Context(frame))\n",
    "        return self.traceit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Running `process_vehicle()` under trace prints the contexts encountered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.475038Z",
     "iopub.status.busy": "2024-01-18T17:19:53.474906Z",
     "iopub.status.idle": "2024-01-18T17:19:53.478637Z",
     "shell.execute_reply": "2024-01-18T17:19:53.478264Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Coverage.ipynb:102:__exit__(self,exc_type,exc_value,tb)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Coverage.ipynb:105:__exit__(self,exc_type,exc_value,tb)\n"
     ]
    }
   ],
   "source": [
    "with Tracer() as tracer:\n",
    "    process_vehicle(VEHICLES[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The trace produced by executing any function can get overwhelmingly large. Hence, we need to restrict our attention to specific modules. Further, we also restrict our attention exclusively to `str` variables since these variables are more likely to contain input fragments. (We will show how to deal with complex objects later in exercises.)\n",
    "\n",
    "The `Context` class we developed earlier is used to decide which modules to monitor, and which variables to trace.\n",
    "\n",
    "We store the current *input string* so that it can be used to determine if any particular string fragments came from the current input string. Any optional arguments are processed separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.481084Z",
     "iopub.status.busy": "2024-01-18T17:19:53.480976Z",
     "iopub.status.idle": "2024-01-18T17:19:53.483029Z",
     "shell.execute_reply": "2024-01-18T17:19:53.482777Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def __init__(self, my_input, **kwargs):\n",
    "        self.options(kwargs)\n",
    "        self.my_input, self.trace = my_input, []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We use an optional argument `files` to indicate the specific source files we are interested in, and `methods` to indicate which specific methods are of interest. Further, we also use `log` to specify whether verbose logging should be enabled during trace. We use the `log_event()` method we defined earlier for logging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The options processing is as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.484605Z",
     "iopub.status.busy": "2024-01-18T17:19:53.484521Z",
     "iopub.status.idle": "2024-01-18T17:19:53.486425Z",
     "shell.execute_reply": "2024-01-18T17:19:53.486202Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def options(self, kwargs):\n",
    "        self.files = kwargs.get('files', [])\n",
    "        self.methods = kwargs.get('methods', [])\n",
    "        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `files` and `methods` are checked to determine, if a particular event should be traced or not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.488025Z",
     "iopub.status.busy": "2024-01-18T17:19:53.487930Z",
     "iopub.status.idle": "2024-01-18T17:19:53.490006Z",
     "shell.execute_reply": "2024-01-18T17:19:53.489772Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def tracing_context(self, cxt, event, arg):\n",
    "        fres = not self.files or any(\n",
    "            cxt.file_name.endswith(f) for f in self.files)\n",
    "        mres = not self.methods or any(cxt.method == m for m in self.methods)\n",
    "        return fres and mres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Similar to the context of events, we also want to restrict our attention to specific variables. For now, we want to focus only on strings. (See the Exercises at the end of the chapter on how to extend it to other kinds of objects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.491503Z",
     "iopub.status.busy": "2024-01-18T17:19:53.491423Z",
     "iopub.status.idle": "2024-01-18T17:19:53.493259Z",
     "shell.execute_reply": "2024-01-18T17:19:53.492951Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def tracing_var(self, k, v):\n",
    "        return isinstance(v, str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We modify the `traceit()` to call an `on_event()` function with the context information only on the specific events we are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.494975Z",
     "iopub.status.busy": "2024-01-18T17:19:53.494863Z",
     "iopub.status.idle": "2024-01-18T17:19:53.497896Z",
     "shell.execute_reply": "2024-01-18T17:19:53.497584Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Tracer(Tracer):\n",
    "    def on_event(self, event, arg, cxt, my_vars):\n",
    "        self.trace.append((event, arg, cxt, my_vars))\n",
    "        \n",
    "    def create_context(self, frame):\n",
    "        return Context(frame)\n",
    "\n",
    "    def traceit(self, frame, event, arg):\n",
    "        cxt = self.create_context(frame)\n",
    "        if not self.tracing_context(cxt, event, arg):\n",
    "            return self.traceit\n",
    "        self.log(event, cxt)\n",
    "\n",
    "        my_vars = {\n",
    "            k: v\n",
    "            for k, v in cxt.extract_vars(frame).items()\n",
    "            if self.tracing_var(k, v)\n",
    "        }\n",
    "        self.on_event(event, arg, cxt, my_vars)\n",
    "        return self.traceit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `Tracer` class can now focus on specific kinds of events on specific files. Further, it provides a first level filter for variables that we find interesting. For example, we want to focus specifically on variables from `process_*` methods that contain input fragments. Here is how our updated `Tracer` can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.499521Z",
     "iopub.status.busy": "2024-01-18T17:19:53.499408Z",
     "iopub.status.idle": "2024-01-18T17:19:53.502685Z",
     "shell.execute_reply": "2024-01-18T17:19:53.502380Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n"
     ]
    }
   ],
   "source": [
    "with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:\n",
    "    process_vehicle(VEHICLES[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The execution produced the following trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.504267Z",
     "iopub.status.busy": "2024-01-18T17:19:53.504190Z",
     "iopub.status.idle": "2024-01-18T17:19:53.506541Z",
     "shell.execute_reply": "2024-01-18T17:19:53.506295Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "call process_vehicle {'vehicle': '1997,van,Ford,E350'}\n",
      "line process_vehicle {'vehicle': '1997,van,Ford,E350'}\n",
      "line process_vehicle {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_vehicle {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'}\n",
      "call process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "line process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "return process_van {'year': '1997', 'company': 'Ford', 'model': 'E350'}\n",
      "return process_vehicle {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'}\n"
     ]
    }
   ],
   "source": [
    "for t in tracer.trace:\n",
    "    print(t[0], t[2].method, dict(t[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Since we are saving the input already in `Tracer`, it is redundant to specify it separately again as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.508116Z",
     "iopub.status.busy": "2024-01-18T17:19:53.508036Z",
     "iopub.status.idle": "2024-01-18T17:19:53.511524Z",
     "shell.execute_reply": "2024-01-18T17:19:53.511241Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n"
     ]
    }
   ],
   "source": [
    "with Tracer(VEHICLES[0], methods=INVENTORY_METHODS, log=True) as tracer:\n",
    "    process_vehicle(tracer.my_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### DefineTracker"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We define a `DefineTracker` class that processes the trace from the `Tracer`. The idea is to store different variable definitions which are input fragments.\n",
    "\n",
    "The tracker identifies string fragments that are part of the input string, and stores them in a dictionary `my_assignments`. It saves the trace, and the corresponding input for processing. Finally, it calls `process()` to process the `trace` it was given. We will start with a simple tracker that relies on certain assumptions, and later see how these assumptions can be relaxed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.513781Z",
     "iopub.status.busy": "2024-01-18T17:19:53.513502Z",
     "iopub.status.idle": "2024-01-18T17:19:53.515584Z",
     "shell.execute_reply": "2024-01-18T17:19:53.515355Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker:\n",
    "    def __init__(self, my_input, trace, **kwargs):\n",
    "        self.options(kwargs)\n",
    "        self.my_input = my_input\n",
    "        self.trace = trace\n",
    "        self.my_assignments = {}\n",
    "        self.process()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One of the problems of using substring search is that short string sequences tend to be included in other string sequences even though they may not have come from the original string. That is, say the input fragment is `v`, it could have equally come from either `van` or `chevy`. We rely on being able to predict the exact place in the input where a given fragment occurred. Hence, we define a constant `FRAGMENT_LEN` such that we ignore strings up to that length. We also incorporate a logging facility as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.517382Z",
     "iopub.status.busy": "2024-01-18T17:19:53.517057Z",
     "iopub.status.idle": "2024-01-18T17:19:53.518851Z",
     "shell.execute_reply": "2024-01-18T17:19:53.518616Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "FRAGMENT_LEN = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.520531Z",
     "iopub.status.busy": "2024-01-18T17:19:53.520348Z",
     "iopub.status.idle": "2024-01-18T17:19:53.522130Z",
     "shell.execute_reply": "2024-01-18T17:19:53.521902Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker(DefineTracker):\n",
    "    def options(self, kwargs):\n",
    "        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None\n",
    "        self.fragment_len = kwargs.get('fragment_len', FRAGMENT_LEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Our tracer simply records the variable values as they occur. We next need to check if the variables contain values from the **input string**. Common ways to do this is to rely on symbolic execution or at least dynamic tainting, which are powerful, but also complex. However, one can obtain a reasonable approximation by simply relying on substring search. That is, we consider any value produced that is a substring of the original input string to have come from the original input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We define an `is_input_fragment()` method that relies on string inclusion to detect if the string came from the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.523651Z",
     "iopub.status.busy": "2024-01-18T17:19:53.523492Z",
     "iopub.status.idle": "2024-01-18T17:19:53.525342Z",
     "shell.execute_reply": "2024-01-18T17:19:53.525111Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker(DefineTracker):\n",
    "    def is_input_fragment(self, var, value):\n",
    "        return len(value) >= self.fragment_len and value in self.my_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can use `is_input_fragment()` to select only a subset of variables defined, as implemented below in `fragments()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.526915Z",
     "iopub.status.busy": "2024-01-18T17:19:53.526790Z",
     "iopub.status.idle": "2024-01-18T17:19:53.528726Z",
     "shell.execute_reply": "2024-01-18T17:19:53.528444Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker(DefineTracker):\n",
    "    def fragments(self, variables):\n",
    "        return {k: v for k, v in variables.items(\n",
    "        ) if self.is_input_fragment(k, v)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The tracker processes each event, and at each event, it updates the dictionary `my_assignments` with the current local variables that contain strings that are part of the input. Note that there is a choice here with respect to what happens during reassignment. We can either discard all the reassignments, or keep only the last assignment. Here, we choose the latter. If you want the former behavior, check whether the value exists in `my_assignments` before storing a fragment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.530332Z",
     "iopub.status.busy": "2024-01-18T17:19:53.530226Z",
     "iopub.status.idle": "2024-01-18T17:19:53.532384Z",
     "shell.execute_reply": "2024-01-18T17:19:53.532148Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker(DefineTracker):\n",
    "    def track_event(self, event, arg, cxt, my_vars):\n",
    "        self.log(event, (cxt.method, my_vars))\n",
    "        self.my_assignments.update(self.fragments(my_vars))\n",
    "\n",
    "    def process(self):\n",
    "        for event, arg, cxt, my_vars in self.trace:\n",
    "            self.track_event(event, arg, cxt, my_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Using the tracker, we can obtain the input fragments. For example, say we are only interested in strings that are at least `5` characters long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.533870Z",
     "iopub.status.busy": "2024-01-18T17:19:53.533742Z",
     "iopub.status.idle": "2024-01-18T17:19:53.535725Z",
     "shell.execute_reply": "2024-01-18T17:19:53.535437Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicle = '1997,van,Ford,E350'\n"
     ]
    }
   ],
   "source": [
    "tracker = DefineTracker(tracer.my_input, tracer.trace, fragment_len=5)\n",
    "for k, v in tracker.my_assignments.items():\n",
    "    print(k, '=', repr(v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Or strings that are `2` characters long (the default)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.537282Z",
     "iopub.status.busy": "2024-01-18T17:19:53.537179Z",
     "iopub.status.idle": "2024-01-18T17:19:53.539147Z",
     "shell.execute_reply": "2024-01-18T17:19:53.538874Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicle = '1997,van,Ford,E350'\n",
      "year = '1997'\n",
      "kind = 'van'\n",
      "company = 'Ford'\n",
      "model = 'E350'\n"
     ]
    }
   ],
   "source": [
    "tracker = DefineTracker(tracer.my_input, tracer.trace)\n",
    "for k, v in tracker.my_assignments.items():\n",
    "    print(k, '=', repr(v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.540654Z",
     "iopub.status.busy": "2024-01-18T17:19:53.540554Z",
     "iopub.status.idle": "2024-01-18T17:19:53.542109Z",
     "shell.execute_reply": "2024-01-18T17:19:53.541879Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class DefineTracker(DefineTracker):\n",
    "    def assignments(self):\n",
    "        return self.my_assignments.items()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Assembling a Derivation Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.543626Z",
     "iopub.status.busy": "2024-01-18T17:19:53.543530Z",
     "iopub.status.idle": "2024-01-18T17:19:53.545303Z",
     "shell.execute_reply": "2024-01-18T17:19:53.545019Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import START_SYMBOL, syntax_diagram, \\\n",
    "    is_nonterminal, Grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.546798Z",
     "iopub.status.busy": "2024-01-18T17:19:53.546693Z",
     "iopub.status.idle": "2024-01-18T17:19:53.548176Z",
     "shell.execute_reply": "2024-01-18T17:19:53.547968Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from GrammarFuzzer import GrammarFuzzer, display_tree, \\\n",
    "    DerivationTree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The input fragments from the `DefineTracker` only tell half the story. The fragments may be created at different stages of parsing. Hence, we need to assemble the fragments to a  derivation tree of the input. The basic idea is as follows:\n",
    "\n",
    "Our input from the previous step was:\n",
    "\n",
    "```python\n",
    "\"1997,van,Ford,E350\"\n",
    "```\n",
    "\n",
    "We start a derivation tree, and associate it with the start symbol in the grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.549513Z",
     "iopub.status.busy": "2024-01-18T17:19:53.549431Z",
     "iopub.status.idle": "2024-01-18T17:19:53.551278Z",
     "shell.execute_reply": "2024-01-18T17:19:53.551043Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree: DerivationTree = (START_SYMBOL, [(\"1997,van,Ford,E350\", [])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.552672Z",
     "iopub.status.busy": "2024-01-18T17:19:53.552592Z",
     "iopub.status.idle": "2024-01-18T17:19:53.941286Z",
     "shell.execute_reply": "2024-01-18T17:19:53.940918Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"120pt\" height=\"73pt\"\n",
       " viewBox=\"0.00 0.00 119.75 72.50\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 68.5)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-68.5 115.75,-68.5 115.75,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,Ford,E350</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-50.46C55.88,-44.24 55.88,-34.15 55.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-25.6 55.88,-15.6 52.38,-25.6 59.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x10500a0e0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The next input was:\n",
    "```python\n",
    "vehicle = \"1997,van,Ford,E350\"\n",
    "```\n",
    "Since vehicle covers the `<start>` node's value completely, we replace the value with the vehicle node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.943292Z",
     "iopub.status.busy": "2024-01-18T17:19:53.943133Z",
     "iopub.status.idle": "2024-01-18T17:19:53.945280Z",
     "shell.execute_reply": "2024-01-18T17:19:53.945017Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree: DerivationTree = (START_SYMBOL, \n",
    "                                   [('<vehicle>', [(\"1997,van,Ford,E350\", [])],\n",
    "                                                   [])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:53.946729Z",
     "iopub.status.busy": "2024-01-18T17:19:53.946595Z",
     "iopub.status.idle": "2024-01-18T17:19:54.324708Z",
     "shell.execute_reply": "2024-01-18T17:19:54.324347Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"120pt\" height=\"123pt\"\n",
       " viewBox=\"0.00 0.00 119.75 122.75\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 118.75)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-118.75 115.75,-118.75 115.75,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-100.71C55.88,-94.49 55.88,-84.4 55.88,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-75.85 55.88,-65.85 52.38,-75.85 59.38,-75.85\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,Ford,E350</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-50.46C55.88,-44.24 55.88,-34.15 55.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-25.6 55.88,-15.6 52.38,-25.6 59.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cd840>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The next input was:\n",
    "```python\n",
    "year = '1997'\n",
    "```\n",
    "Traversing the derivation tree from `<start>`, we see that it replaces a portion of the `<vehicle>` node's value. Hence we split the `<vehicle>` node's value to two children, where one corresponds to the value `\"1997\"` and the other to `\",van,Ford,E350\"`, and replace the first one with the node `<year>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:54.326610Z",
     "iopub.status.busy": "2024-01-18T17:19:54.326450Z",
     "iopub.status.idle": "2024-01-18T17:19:54.328785Z",
     "shell.execute_reply": "2024-01-18T17:19:54.328486Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree: DerivationTree = (START_SYMBOL, \n",
    "                                   [('<vehicle>', [('<year>', [('1997', [])]),\n",
    "                                                   (\",van,Ford,E350\", [])], [])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:54.330321Z",
     "iopub.status.busy": "2024-01-18T17:19:54.330188Z",
     "iopub.status.idle": "2024-01-18T17:19:54.705251Z",
     "shell.execute_reply": "2024-01-18T17:19:54.704909Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"150pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 150.25 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 146.25,-169 146.25,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"59.88\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"59.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M59.88,-150.96C59.88,-144.74 59.88,-134.65 59.88,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"63.38,-126.1 59.88,-116.1 56.38,-126.1 63.38,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M54.97,-100.71C49.27,-93.84 39.67,-82.25 31.94,-72.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"34.86,-70.97 25.79,-65.5 29.47,-75.44 34.86,-70.97\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"99.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">,van,Ford,E350</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M64.78,-100.71C70.48,-93.84 80.08,-82.25 87.81,-72.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"90.28,-75.44 93.96,-65.5 84.89,-70.97 90.28,-75.44\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M19.88,-50.46C19.88,-44.24 19.88,-34.15 19.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"23.38,-25.6 19.88,-15.6 16.38,-25.6 23.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072ceef0>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We perform similar operations for \n",
    "```python\n",
    "company = 'Ford'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:54.707066Z",
     "iopub.status.busy": "2024-01-18T17:19:54.706942Z",
     "iopub.status.idle": "2024-01-18T17:19:54.709234Z",
     "shell.execute_reply": "2024-01-18T17:19:54.708894Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree: DerivationTree = (START_SYMBOL, \n",
    "                                   [('<vehicle>', [('<year>', [('1997', [])]),\n",
    "                                                   (\",van,\", []),\n",
    "                                                   ('<company>', [('Ford', [])]),\n",
    "                                                   (\",E350\", [])], [])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:54.710840Z",
     "iopub.status.busy": "2024-01-18T17:19:54.710714Z",
     "iopub.status.idle": "2024-01-18T17:19:55.110981Z",
     "shell.execute_reply": "2024-01-18T17:19:55.110562Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"228pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 228.00 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 224,-169 224,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"102.88\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"102.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M102.88,-150.96C102.88,-144.74 102.88,-134.65 102.88,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"106.38,-126.1 102.88,-116.1 99.38,-126.1 106.38,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M92.36,-100.51C79.27,-92.9 56.6,-79.73 40.1,-70.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.01,-67.2 31.61,-65.2 38.5,-73.25 42.01,-67.2\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"70.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">,van,</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M98.95,-100.71C94.54,-94.05 87.19,-82.98 81.12,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"84.11,-72 75.67,-65.6 78.28,-75.87 84.11,-72\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"135.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M106.92,-100.71C111.47,-94.05 119.05,-82.98 125.31,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"128.18,-75.82 130.94,-65.59 122.41,-71.87 128.18,-75.82\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"203.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">,E350</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>1&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M115.67,-100.51C132.06,-92.68 160.77,-78.97 180.97,-69.32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.31,-72.56 189.82,-65.09 179.29,-66.24 182.31,-72.56\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M19.88,-50.46C19.88,-44.24 19.88,-34.15 19.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"23.38,-25.6 19.88,-15.6 16.38,-25.6 23.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"135.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M135.88,-50.46C135.88,-44.24 135.88,-34.15 135.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"139.38,-25.6 135.88,-15.6 132.38,-25.6 139.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cef50>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similarly for\n",
    "```python\n",
    "kind = 'van'\n",
    "```\n",
    "and\n",
    "```python\n",
    "model = 'E350'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.113465Z",
     "iopub.status.busy": "2024-01-18T17:19:55.113132Z",
     "iopub.status.idle": "2024-01-18T17:19:55.115850Z",
     "shell.execute_reply": "2024-01-18T17:19:55.115580Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree: DerivationTree = (START_SYMBOL, \n",
    "                                   [('<vehicle>', [('<year>', [('1997', [])]),\n",
    "                                                   (\",\", []),\n",
    "                                                   (\"<kind>\", [('van', [])]),\n",
    "                                                   (\",\", []),\n",
    "                                                   ('<company>', [('Ford', [])]),\n",
    "                                                   (\",\", []),\n",
    "                                                   (\"<model>\", [('E350', [])])\n",
    "                                                   ], [])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.117413Z",
     "iopub.status.busy": "2024-01-18T17:19:55.117312Z",
     "iopub.status.idle": "2024-01-18T17:19:55.494644Z",
     "shell.execute_reply": "2024-01-18T17:19:55.494281Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"403pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 403.38 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 399.38,-169 399.38,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-150.96C178.88,-144.74 178.88,-134.65 178.88,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-126.1 178.88,-116.1 175.38,-126.1 182.38,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M158.74,-100.51C131.42,-92.23 82.39,-77.35 50.5,-67.67\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"51.63,-64.35 41.05,-64.8 49.6,-71.05 51.63,-64.35\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"72.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M165.45,-100.51C148.17,-92.65 117.84,-78.84 96.62,-69.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"98.12,-66.02 87.57,-65.06 95.22,-72.39 98.12,-66.02\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.38,-100.71C164.59,-93.62 151.29,-81.51 140.9,-72.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"143.3,-69.5 133.55,-65.36 138.58,-74.68 143.3,-69.5\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>1&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-100.71C178.88,-94.49 178.88,-84.4 178.88,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-75.85 178.88,-65.85 175.38,-75.85 182.38,-75.85\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>1&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M187.23,-100.51C197.41,-93.08 214.83,-80.34 227.87,-70.81\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.56,-73.91 235.57,-65.18 225.43,-68.25 229.56,-73.91\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"310.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>1&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.59,-100.51C218.19,-92.25 258.71,-77.44 285.19,-67.76\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.31,-71.08 294.5,-64.36 283.9,-64.51 286.31,-71.08\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>1&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M203.07,-100.51C236.36,-92.1 296.52,-76.9 334.77,-67.24\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"335.58,-70.65 344.41,-64.81 333.86,-63.86 335.58,-70.65\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M19.88,-50.46C19.88,-44.24 19.88,-34.15 19.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"23.38,-25.6 19.88,-15.6 16.38,-25.6 23.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.88,-50.46C125.88,-44.24 125.88,-34.15 125.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"129.38,-25.6 125.88,-15.6 122.38,-25.6 129.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.88,-50.46C244.88,-44.24 244.88,-34.15 244.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"248.38,-25.6 244.88,-15.6 241.38,-25.6 248.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>11&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M369.88,-50.46C369.88,-44.24 369.88,-34.15 369.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"373.38,-25.6 369.88,-15.6 366.38,-25.6 373.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cfc40>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We now develop the complete algorithm with the above described steps.\n",
    "The derivation tree `TreeMiner` is initialized with the input string, and the variable assignments, and it converts the assignments to the corresponding derivation tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.496327Z",
     "iopub.status.busy": "2024-01-18T17:19:55.496217Z",
     "iopub.status.idle": "2024-01-18T17:19:55.498847Z",
     "shell.execute_reply": "2024-01-18T17:19:55.498489Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner:\n",
    "    def __init__(self, my_input, my_assignments, **kwargs):\n",
    "        self.options(kwargs)\n",
    "        self.my_input = my_input\n",
    "        self.my_assignments = my_assignments\n",
    "        self.tree = self.get_derivation_tree()\n",
    "\n",
    "    def options(self, kwargs):\n",
    "        self.log = log_call if kwargs.get('log') else lambda _i, _v: None\n",
    "\n",
    "    def get_derivation_tree(self):\n",
    "        return (START_SYMBOL, [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `log_call()` is as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.500651Z",
     "iopub.status.busy": "2024-01-18T17:19:55.500531Z",
     "iopub.status.idle": "2024-01-18T17:19:55.502233Z",
     "shell.execute_reply": "2024-01-18T17:19:55.501934Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def log_call(indent, var):\n",
    "    print('\\t' * indent, var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The basic idea is as follows:\n",
    "* **For now, we assume that the value assigned to a variable is stable. That is, it is never reassigned. In particular, there are no recursive calls, or multiple calls to the same function from different parts.** (We will show how to overcome this limitation later).\n",
    "* For each pair _var_, _value_ found in `my_assignments`:\n",
    "    1. We search for occurrences of _value_ `val` in the derivation tree recursively.\n",
    "    2. If an occurrence was found as a value `V1` of a node `P1`, we partition the value of the node `P1` into three parts, with the central part matching the _value_ `val`, and the first and last part, the corresponding prefix and suffix in `V1`.\n",
    "    3. Reconstitute the node `P1` with three children, where prefix and suffix mentioned earlier are string values, and the matching value `val` is replaced by a node `var` with a single value `val`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "First, we define a wrapper to generate a nonterminal from a variable name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.503812Z",
     "iopub.status.busy": "2024-01-18T17:19:55.503690Z",
     "iopub.status.idle": "2024-01-18T17:19:55.505406Z",
     "shell.execute_reply": "2024-01-18T17:19:55.505138Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def to_nonterminal(var):\n",
    "    return \"<\" + var.lower() + \">\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `string_part_of_value()` method checks whether the given `part` value was part of the whole."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.506958Z",
     "iopub.status.busy": "2024-01-18T17:19:55.506850Z",
     "iopub.status.idle": "2024-01-18T17:19:55.508542Z",
     "shell.execute_reply": "2024-01-18T17:19:55.508304Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def string_part_of_value(self, part, value):\n",
    "        return (part in value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `partition_by_part()` splits the `value` by the given part if it matches, and returns a list containing the first part, the part that was replaced, and the last part. This is a format that can be used as a part of the list of children."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.510130Z",
     "iopub.status.busy": "2024-01-18T17:19:55.510026Z",
     "iopub.status.idle": "2024-01-18T17:19:55.511794Z",
     "shell.execute_reply": "2024-01-18T17:19:55.511571Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def partition(self, part, value):\n",
    "        return value.partition(part)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.513175Z",
     "iopub.status.busy": "2024-01-18T17:19:55.513056Z",
     "iopub.status.idle": "2024-01-18T17:19:55.515093Z",
     "shell.execute_reply": "2024-01-18T17:19:55.514843Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def partition_by_part(self, pair, value):\n",
    "        k, part = pair\n",
    "        prefix_k_suffix = [\n",
    "                    (k, [[part, []]]) if i == 1 else (e, [])\n",
    "                    for i, e in enumerate(self.partition(part, value))\n",
    "                    if e]\n",
    "        return prefix_k_suffix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `insert_into_tree()` method accepts a given tree `tree` and a `(k,v)` pair.  It recursively checks whether the given pair can be applied. If the pair can be applied, it applies the pair and returns `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.516820Z",
     "iopub.status.busy": "2024-01-18T17:19:55.516708Z",
     "iopub.status.idle": "2024-01-18T17:19:55.520155Z",
     "shell.execute_reply": "2024-01-18T17:19:55.519840Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def insert_into_tree(self, my_tree, pair):\n",
    "        var, values = my_tree\n",
    "        k, v = pair\n",
    "        self.log(1, \"- Node: %s\\t\\t? (%s:%s)\" % (var, k, repr(v)))\n",
    "        applied = False\n",
    "        for i, value_ in enumerate(values):\n",
    "            value, arr = value_\n",
    "            self.log(2, \"-> [%d] %s\" % (i, repr(value)))\n",
    "            if is_nonterminal(value):\n",
    "                applied = self.insert_into_tree(value_, pair)\n",
    "                if applied:\n",
    "                    break\n",
    "            elif self.string_part_of_value(v, value):\n",
    "                prefix_k_suffix = self.partition_by_part(pair, value)\n",
    "                del values[i]\n",
    "                for j, rep in enumerate(prefix_k_suffix):\n",
    "                    values.insert(j + i, rep)\n",
    "                applied = True\n",
    "\n",
    "                self.log(2, \" > %s\" % (repr([i[0] for i in prefix_k_suffix])))\n",
    "                break\n",
    "            else:\n",
    "                continue\n",
    "        return applied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here is how `insert_into_tree()` is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.521880Z",
     "iopub.status.busy": "2024-01-18T17:19:55.521767Z",
     "iopub.status.idle": "2024-01-18T17:19:55.523694Z",
     "shell.execute_reply": "2024-01-18T17:19:55.523393Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tree: DerivationTree = (START_SYMBOL, [(\"1997,van,Ford,E350\", [])])\n",
    "m = TreeMiner('', {}, log=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "First, we have our input string as the only node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.525237Z",
     "iopub.status.busy": "2024-01-18T17:19:55.525145Z",
     "iopub.status.idle": "2024-01-18T17:19:55.929214Z",
     "shell.execute_reply": "2024-01-18T17:19:55.928633Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"120pt\" height=\"73pt\"\n",
       " viewBox=\"0.00 0.00 119.75 72.50\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 68.5)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-68.5 115.75,-68.5 115.75,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,Ford,E350</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-50.46C55.88,-44.24 55.88,-34.15 55.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-25.6 55.88,-15.6 52.38,-25.6 59.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cf2b0>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inserting the `<vehicle>` node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.931407Z",
     "iopub.status.busy": "2024-01-18T17:19:55.931256Z",
     "iopub.status.idle": "2024-01-18T17:19:55.933600Z",
     "shell.execute_reply": "2024-01-18T17:19:55.933270Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t - Node: <start>\t\t? (<vehicle>:'1997,van,Ford,E350')\n",
      "\t\t -> [0] '1997,van,Ford,E350'\n",
      "\t\t  > ['<vehicle>']\n"
     ]
    }
   ],
   "source": [
    "v = m.insert_into_tree(tree, ('<vehicle>', \"1997,van,Ford,E350\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:55.935265Z",
     "iopub.status.busy": "2024-01-18T17:19:55.935138Z",
     "iopub.status.idle": "2024-01-18T17:19:56.321766Z",
     "shell.execute_reply": "2024-01-18T17:19:56.321407Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"120pt\" height=\"123pt\"\n",
       " viewBox=\"0.00 0.00 119.75 122.75\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 118.75)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-118.75 115.75,-118.75 115.75,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-100.71C55.88,-94.49 55.88,-84.4 55.88,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-75.85 55.88,-65.85 52.38,-75.85 59.38,-75.85\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"55.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,Ford,E350</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M55.88,-50.46C55.88,-44.24 55.88,-34.15 55.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"59.38,-25.6 55.88,-15.6 52.38,-25.6 59.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072ce6e0>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inserting `<model>` node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:56.323685Z",
     "iopub.status.busy": "2024-01-18T17:19:56.323562Z",
     "iopub.status.idle": "2024-01-18T17:19:56.325563Z",
     "shell.execute_reply": "2024-01-18T17:19:56.325311Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t - Node: <start>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] '1997,van,Ford,E350'\n",
      "\t\t  > ['1997,van,Ford,', '<model>']\n"
     ]
    }
   ],
   "source": [
    "v = m.insert_into_tree(tree, ('<model>', 'E350'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:56.327475Z",
     "iopub.status.busy": "2024-01-18T17:19:56.327245Z",
     "iopub.status.idle": "2024-01-18T17:19:56.692057Z",
     "shell.execute_reply": "2024-01-18T17:19:56.691671Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"160pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 160.12 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 156.12,-169 156.12,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"83.62\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"83.62\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M83.62,-150.96C83.62,-144.74 83.62,-134.65 83.62,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"87.13,-126.1 83.63,-116.1 80.13,-126.1 87.13,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"41.62\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,Ford,</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M78.47,-100.71C72.49,-93.84 62.41,-82.25 54.3,-72.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"57.02,-70.73 47.81,-65.48 51.74,-75.32 57.02,-70.73\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"126.62\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M88.9,-100.71C95.02,-93.84 105.35,-82.25 113.65,-72.93\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"116.26,-75.26 120.3,-65.47 111.04,-70.61 116.26,-75.26\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"126.62\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M126.62,-50.46C126.62,-44.24 126.62,-34.15 126.62,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"130.13,-25.6 126.63,-15.6 123.13,-25.6 130.13,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cfe80>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree((tree))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inserting `<company>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:56.694084Z",
     "iopub.status.busy": "2024-01-18T17:19:56.693931Z",
     "iopub.status.idle": "2024-01-18T17:19:56.696562Z",
     "shell.execute_reply": "2024-01-18T17:19:56.696205Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t - Node: <start>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] '1997,van,Ford,'\n",
      "\t\t  > ['1997,van,', '<company>', ',']\n"
     ]
    }
   ],
   "source": [
    "v = m.insert_into_tree(tree, ('<company>', 'Ford'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:56.698920Z",
     "iopub.status.busy": "2024-01-18T17:19:56.698723Z",
     "iopub.status.idle": "2024-01-18T17:19:57.088624Z",
     "shell.execute_reply": "2024-01-18T17:19:57.088213Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"264pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 263.50 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 259.5,-169 259.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"138\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"138\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M138,-150.96C138,-144.74 138,-134.65 138,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"141.5,-126.1 138,-116.1 134.5,-126.1 141.5,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">1997,van,</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.94,-100.51C105.68,-92.57 73.5,-78.59 51.25,-68.92\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"52.9,-65.82 42.33,-65.04 50.11,-72.24 52.9,-65.82\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"105\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M133.95,-100.71C129.4,-94.05 121.83,-82.98 115.56,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"118.47,-71.87 109.93,-65.59 112.69,-75.82 118.47,-71.87\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"171\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M142.05,-100.71C146.6,-94.05 154.17,-82.98 160.44,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"163.31,-75.82 166.07,-65.59 157.53,-71.87 163.31,-75.82\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"230\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>1&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M149.65,-100.51C164.38,-92.79 190.01,-79.35 208.36,-69.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"209.87,-72.88 217.1,-65.14 206.62,-66.68 209.87,-72.88\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"105\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M105,-50.46C105,-44.24 105,-34.15 105,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"108.5,-25.6 105,-15.6 101.5,-25.6 108.5,-25.6\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"230\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>6&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M230,-50.46C230,-44.24 230,-34.15 230,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"233.5,-25.6 230,-15.6 226.5,-25.6 233.5,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cfcd0>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inserting `<kind>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.090562Z",
     "iopub.status.busy": "2024-01-18T17:19:57.090408Z",
     "iopub.status.idle": "2024-01-18T17:19:57.092920Z",
     "shell.execute_reply": "2024-01-18T17:19:57.092553Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t - Node: <start>\t\t? (<kind>:'van')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<kind>:'van')\n",
      "\t\t -> [0] '1997,van,'\n",
      "\t\t  > ['1997,', '<kind>', ',']\n"
     ]
    }
   ],
   "source": [
    "v = m.insert_into_tree(tree, ('<kind>', 'van'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.094892Z",
     "iopub.status.busy": "2024-01-18T17:19:57.094753Z",
     "iopub.status.idle": "2024-01-18T17:19:57.511885Z",
     "shell.execute_reply": "2024-01-18T17:19:57.511454Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"347pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 346.88 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 342.88,-169 342.88,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"155.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"155.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M155.38,-150.96C155.38,-144.74 155.38,-134.65 155.38,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"158.88,-126.1 155.38,-116.1 151.88,-126.1 158.88,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"15.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">1997,</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M137.64,-100.51C113.2,-92.09 69,-76.86 40.97,-67.2\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"42.43,-64 31.83,-64.05 40.15,-70.61 42.43,-64\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"69.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M144.48,-100.51C130.78,-92.83 106.98,-79.47 89.84,-69.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"91.93,-67.02 81.5,-65.18 88.51,-73.12 91.93,-67.02\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"122.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M151.33,-100.71C146.78,-94.05 139.2,-82.98 132.94,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"135.84,-71.87 127.31,-65.59 130.07,-75.82 135.84,-71.87\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>1&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M159.42,-100.71C163.97,-94.05 171.55,-82.98 177.81,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"180.68,-75.82 183.44,-65.59 174.91,-71.87 180.68,-75.82\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"254.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>1&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M167.91,-100.51C183.91,-92.72 211.86,-79.1 231.65,-69.45\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"233.12,-72.63 240.58,-65.1 230.06,-66.33 233.12,-72.63\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"313.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>1&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M175.39,-100.51C202.45,-92.25 250.99,-77.43 282.68,-67.75\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"283.54,-71.15 292.08,-64.88 281.49,-64.45 283.54,-71.15\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"69.38\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M69.38,-50.46C69.38,-44.24 69.38,-34.15 69.38,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"72.88,-25.6 69.38,-15.6 65.88,-25.6 72.88,-25.6\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>6&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.38,-50.46C188.38,-44.24 188.38,-34.15 188.38,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"191.88,-25.6 188.38,-15.6 184.88,-25.6 191.88,-25.6\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"313.38\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M313.38,-50.46C313.38,-44.24 313.38,-34.15 313.38,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"316.88,-25.6 313.38,-15.6 309.88,-25.6 316.88,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072cefe0>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inserting `<year>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.513812Z",
     "iopub.status.busy": "2024-01-18T17:19:57.513695Z",
     "iopub.status.idle": "2024-01-18T17:19:57.515867Z",
     "shell.execute_reply": "2024-01-18T17:19:57.515632Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t - Node: <start>\t\t? (<year>:'1997')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<year>:'1997')\n",
      "\t\t -> [0] '1997,'\n",
      "\t\t  > ['<year>', ',']\n"
     ]
    }
   ],
   "source": [
    "v = m.insert_into_tree(tree, ('<year>', '1997'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.517530Z",
     "iopub.status.busy": "2024-01-18T17:19:57.517302Z",
     "iopub.status.idle": "2024-01-18T17:19:57.919123Z",
     "shell.execute_reply": "2024-01-18T17:19:57.918792Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"403pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 403.38 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 399.38,-169 399.38,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-150.96C178.88,-144.74 178.88,-134.65 178.88,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-126.1 178.88,-116.1 175.38,-126.1 182.38,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M158.74,-100.51C131.42,-92.23 82.39,-77.35 50.5,-67.67\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"51.63,-64.35 41.05,-64.8 49.6,-71.05 51.63,-64.35\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"72.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M165.45,-100.51C148.17,-92.65 117.84,-78.84 96.62,-69.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"98.12,-66.02 87.57,-65.06 95.22,-72.39 98.12,-66.02\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.38,-100.71C164.59,-93.62 151.29,-81.51 140.9,-72.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"143.3,-69.5 133.55,-65.36 138.58,-74.68 143.3,-69.5\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>1&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-100.71C178.88,-94.49 178.88,-84.4 178.88,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-75.85 178.88,-65.85 175.38,-75.85 182.38,-75.85\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>1&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M187.23,-100.51C197.41,-93.08 214.83,-80.34 227.87,-70.81\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.56,-73.91 235.57,-65.18 225.43,-68.25 229.56,-73.91\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"310.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>1&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.59,-100.51C218.19,-92.25 258.71,-77.44 285.19,-67.76\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.31,-71.08 294.5,-64.36 283.9,-64.51 286.31,-71.08\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>1&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M203.07,-100.51C236.36,-92.1 296.52,-76.9 334.77,-67.24\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"335.58,-70.65 344.41,-64.81 333.86,-63.86 335.58,-70.65\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M19.88,-50.46C19.88,-44.24 19.88,-34.15 19.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"23.38,-25.6 19.88,-15.6 16.38,-25.6 23.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.88,-50.46C125.88,-44.24 125.88,-34.15 125.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"129.38,-25.6 125.88,-15.6 122.38,-25.6 129.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.88,-50.46C244.88,-44.24 244.88,-34.15 244.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"248.38,-25.6 244.88,-15.6 241.38,-25.6 248.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>11&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M369.88,-50.46C369.88,-44.24 369.88,-34.15 369.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"373.38,-25.6 369.88,-15.6 366.38,-25.6 373.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072ceb90>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To make life simple, we define a wrapper function `nt_var()` that will convert a token to its corresponding nonterminal symbol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.920905Z",
     "iopub.status.busy": "2024-01-18T17:19:57.920783Z",
     "iopub.status.idle": "2024-01-18T17:19:57.922806Z",
     "shell.execute_reply": "2024-01-18T17:19:57.922527Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def nt_var(self, var):\n",
    "        return var if is_nonterminal(var) else to_nonterminal(var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now, we need to apply a new definition to an entire grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.924261Z",
     "iopub.status.busy": "2024-01-18T17:19:57.924152Z",
     "iopub.status.idle": "2024-01-18T17:19:57.925967Z",
     "shell.execute_reply": "2024-01-18T17:19:57.925682Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def apply_new_definition(self, tree, var, value):\n",
    "        nt_var = self.nt_var(var)\n",
    "        return self.insert_into_tree(tree, (nt_var, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This algorithm is implemented as `get_derivation_tree()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.927558Z",
     "iopub.status.busy": "2024-01-18T17:19:57.927412Z",
     "iopub.status.idle": "2024-01-18T17:19:57.930267Z",
     "shell.execute_reply": "2024-01-18T17:19:57.929608Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def get_derivation_tree(self):\n",
    "        tree = (START_SYMBOL, [(self.my_input, [])])\n",
    "\n",
    "        for var, value in self.my_assignments:\n",
    "            self.log(0, \"%s=%s\" % (var, repr(value)))\n",
    "            self.apply_new_definition(tree, var, value)\n",
    "        return tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `TreeMiner` is used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.932420Z",
     "iopub.status.busy": "2024-01-18T17:19:57.932277Z",
     "iopub.status.idle": "2024-01-18T17:19:57.937492Z",
     "shell.execute_reply": "2024-01-18T17:19:57.937256Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " vehicle='1997,van,Ford,E350'\n",
      "\t - Node: <start>\t\t? (<vehicle>:'1997,van,Ford,E350')\n",
      "\t\t -> [0] '1997,van,Ford,E350'\n",
      "\t\t  > ['<vehicle>']\n",
      " year='1997'\n",
      "\t - Node: <start>\t\t? (<year>:'1997')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<year>:'1997')\n",
      "\t\t -> [0] '1997,van,Ford,E350'\n",
      "\t\t  > ['<year>', ',van,Ford,E350']\n",
      " kind='van'\n",
      "\t - Node: <start>\t\t? (<kind>:'van')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<kind>:'van')\n",
      "\t\t -> [0] '<year>'\n",
      "\t - Node: <year>\t\t? (<kind>:'van')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ',van,Ford,E350'\n",
      "\t\t  > [',', '<kind>', ',Ford,E350']\n",
      " company='Ford'\n",
      "\t - Node: <start>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] '<year>'\n",
      "\t - Node: <year>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind>'\n",
      "\t - Node: <kind>\t\t? (<company>:'Ford')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ',Ford,E350'\n",
      "\t\t  > [',', '<company>', ',E350']\n",
      " model='E350'\n",
      "\t - Node: <start>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] '<vehicle>'\n",
      "\t - Node: <vehicle>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] '<year>'\n",
      "\t - Node: <year>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind>'\n",
      "\t - Node: <kind>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company>'\n",
      "\t - Node: <company>\t\t? (<model>:'E350')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ',E350'\n",
      "\t\t  > [',', '<model>']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('<start>',\n",
       " [('<vehicle>',\n",
       "   [('<year>', [['1997', []]]),\n",
       "    (',', []),\n",
       "    ('<kind>', [['van', []]]),\n",
       "    (',', []),\n",
       "    ('<company>', [['Ford', []]]),\n",
       "    (',', []),\n",
       "    ('<model>', [['E350', []]])])])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with Tracer(VEHICLES[0]) as tracer:\n",
    "    process_vehicle(tracer.my_input)\n",
    "assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()\n",
    "dt = TreeMiner(tracer.my_input, assignments, log=True)\n",
    "dt.tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The obtained derivation tree is as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:57.939192Z",
     "iopub.status.busy": "2024-01-18T17:19:57.939041Z",
     "iopub.status.idle": "2024-01-18T17:19:58.357396Z",
     "shell.execute_reply": "2024-01-18T17:19:58.357041Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"403pt\" height=\"173pt\"\n",
       " viewBox=\"0.00 0.00 403.38 173.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 169)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-169 399.38,-169 399.38,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-150.96C178.88,-144.74 178.88,-134.65 178.88,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-126.1 178.88,-116.1 175.38,-126.1 182.38,-126.1\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M158.74,-100.51C131.42,-92.23 82.39,-77.35 50.5,-67.67\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"51.63,-64.35 41.05,-64.8 49.6,-71.05 51.63,-64.35\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"72.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M165.45,-100.51C148.17,-92.65 117.84,-78.84 96.62,-69.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"98.12,-66.02 87.57,-65.06 95.22,-72.39 98.12,-66.02\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172.38,-100.71C164.59,-93.62 151.29,-81.51 140.9,-72.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"143.3,-69.5 133.55,-65.36 138.58,-74.68 143.3,-69.5\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"178.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>1&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M178.88,-100.71C178.88,-94.49 178.88,-84.4 178.88,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"182.38,-75.85 178.88,-65.85 175.38,-75.85 182.38,-75.85\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>1&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M187.23,-100.51C197.41,-93.08 214.83,-80.34 227.87,-70.81\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"229.56,-73.91 235.57,-65.18 225.43,-68.25 229.56,-73.91\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"310.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>1&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.59,-100.51C218.19,-92.25 258.71,-77.44 285.19,-67.76\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"286.31,-71.08 294.5,-64.36 283.9,-64.51 286.31,-71.08\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>1&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M203.07,-100.51C236.36,-92.1 296.52,-76.9 334.77,-67.24\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"335.58,-70.65 344.41,-64.81 333.86,-63.86 335.58,-70.65\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M19.88,-50.46C19.88,-44.24 19.88,-34.15 19.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"23.38,-25.6 19.88,-15.6 16.38,-25.6 23.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"125.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.88,-50.46C125.88,-44.24 125.88,-34.15 125.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"129.38,-25.6 125.88,-15.6 122.38,-25.6 129.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"244.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.88,-50.46C244.88,-44.24 244.88,-34.15 244.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"248.38,-25.6 244.88,-15.6 241.38,-25.6 248.38,-25.6\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"369.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>11&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M369.88,-50.46C369.88,-44.24 369.88,-34.15 369.88,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"373.38,-25.6 369.88,-15.6 366.38,-25.6 373.38,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1072ce020>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(TreeMiner(tracer.my_input, assignments).tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Combining all the pieces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.359814Z",
     "iopub.status.busy": "2024-01-18T17:19:58.359663Z",
     "iopub.status.idle": "2024-01-18T17:19:58.367669Z",
     "shell.execute_reply": "2024-01-18T17:19:58.367340Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1997,van,Ford,E350\n",
      "vehicle = '1997,van,Ford,E350'\n",
      "year = '1997'\n",
      "kind = 'van'\n",
      "company = 'Ford'\n",
      "model = 'E350'\n",
      "\n",
      "2000,car,Mercury,Cougar\n",
      "vehicle = '2000,car,Mercury,Cougar'\n",
      "year = '2000'\n",
      "kind = 'car'\n",
      "company = 'Mercury'\n",
      "model = 'Cougar'\n",
      "\n",
      "1999,car,Chevy,Venture\n",
      "vehicle = '1999,car,Chevy,Venture'\n",
      "year = '1999'\n",
      "kind = 'car'\n",
      "company = 'Chevy'\n",
      "model = 'Venture'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "trees = []\n",
    "for vehicle in VEHICLES:\n",
    "    print(vehicle)\n",
    "    with Tracer(vehicle) as tracer:\n",
    "        process_vehicle(tracer.my_input)\n",
    "    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()\n",
    "    trees.append((tracer.my_input, assignments))\n",
    "    for var, val in assignments:\n",
    "        print(var + \" = \" + repr(val))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The corresponding derivation trees are below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.369516Z",
     "iopub.status.busy": "2024-01-18T17:19:58.369276Z",
     "iopub.status.idle": "2024-01-18T17:19:58.372150Z",
     "shell.execute_reply": "2024-01-18T17:19:58.371926Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1997,van,Ford,E350\n",
      "2000,car,Mercury,Cougar\n",
      "1999,car,Chevy,Venture\n"
     ]
    }
   ],
   "source": [
    "csv_dt = []\n",
    "for inputstr, assignments in trees:\n",
    "    print(inputstr)\n",
    "    dt = TreeMiner(inputstr, assignments)\n",
    "    csv_dt.append(dt)\n",
    "    display_tree(dt.tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recovering Grammars from Derivation Trees\n",
    "\n",
    "We define a class `Miner` that can combine multiple derivation trees to produce the grammar. The initial grammar is empty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.373740Z",
     "iopub.status.busy": "2024-01-18T17:19:58.373608Z",
     "iopub.status.idle": "2024-01-18T17:19:58.375407Z",
     "shell.execute_reply": "2024-01-18T17:19:58.375121Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarMiner:\n",
    "    def __init__(self):\n",
    "        self.grammar = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `tree_to_grammar()` method converts our derivation tree to a grammar by picking one node at a time, and adding it to the grammar. The node name becomes the key, and any list of children it has becomes another alternative for that key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.377024Z",
     "iopub.status.busy": "2024-01-18T17:19:58.376896Z",
     "iopub.status.idle": "2024-01-18T17:19:58.379838Z",
     "shell.execute_reply": "2024-01-18T17:19:58.379474Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarMiner(GrammarMiner):\n",
    "    def tree_to_grammar(self, tree):\n",
    "        node, children = tree\n",
    "        one_alt = [ck for ck, gc in children]\n",
    "        hsh = {node: [one_alt] if one_alt else []}\n",
    "        for child in children:\n",
    "            if not is_nonterminal(child[0]):\n",
    "                continue\n",
    "            chsh = self.tree_to_grammar(child)\n",
    "            for k in chsh:\n",
    "                if k not in hsh:\n",
    "                    hsh[k] = chsh[k]\n",
    "                else:\n",
    "                    hsh[k].extend(chsh[k])\n",
    "        return hsh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.381919Z",
     "iopub.status.busy": "2024-01-18T17:19:58.381737Z",
     "iopub.status.idle": "2024-01-18T17:19:58.384699Z",
     "shell.execute_reply": "2024-01-18T17:19:58.384364Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': [['<vehicle>']],\n",
       " '<vehicle>': [['<year>', ',', '<kind>', ',', '<company>', ',', '<model>']],\n",
       " '<year>': [['1997']],\n",
       " '<kind>': [['van']],\n",
       " '<company>': [['Ford']],\n",
       " '<model>': [['E350']]}"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm = GrammarMiner()\n",
    "gm.tree_to_grammar(csv_dt[0].tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The grammar being generated here is `canonical`. We define a function `readable()` that takes in a canonical grammar and returns it in a readable form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.386567Z",
     "iopub.status.busy": "2024-01-18T17:19:58.386432Z",
     "iopub.status.idle": "2024-01-18T17:19:58.388532Z",
     "shell.execute_reply": "2024-01-18T17:19:58.388258Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def readable(grammar):\n",
    "    def readable_rule(rule):\n",
    "        return ''.join(rule)\n",
    "\n",
    "    return {k: list(set(readable_rule(a) for a in grammar[k]))\n",
    "            for k in grammar}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.389987Z",
     "iopub.status.busy": "2024-01-18T17:19:58.389888Z",
     "iopub.status.idle": "2024-01-18T17:19:58.400955Z",
     "shell.execute_reply": "2024-01-18T17:19:58.400691Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 199.5 62\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M159.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">vehicle</text></g></g><path d=\"M139.5 31h20\"/></g><path d=\"M 159.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 575.5 62\" width=\"575.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M535.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M515.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">year</text></g><path d=\"M114.0 31h10\"/><path d=\"M124.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M134.0 31h0.0\"/><path d=\"M162.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"134.0\" y=\"20\"/><text x=\"148.25\" y=\"35\">,</text></g><path d=\"M162.5 31h10\"/><path d=\"M172.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M182.5 31h0.0\"/><path d=\"M236.5 31h0.0\"/><rect height=\"22\" width=\"54.0\" x=\"182.5\" y=\"20\"/><text x=\"209.5\" y=\"35\">kind</text></g><path d=\"M236.5 31h10\"/><path d=\"M246.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M256.5 31h0.0\"/><path d=\"M285.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"256.5\" y=\"20\"/><text x=\"270.75\" y=\"35\">,</text></g><path d=\"M285.0 31h10\"/><path d=\"M295.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M305.0 31h0.0\"/><path d=\"M384.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"305.0\" y=\"20\"/><text x=\"344.75\" y=\"35\">company</text></g><path d=\"M384.5 31h10\"/><path d=\"M394.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M404.5 31h0.0\"/><path d=\"M433.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"404.5\" y=\"20\"/><text x=\"418.75\" y=\"35\">,</text></g><path d=\"M433.0 31h10\"/><path d=\"M443.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M453.0 31h0.0\"/><path d=\"M515.5 31h0.0\"/><rect height=\"22\" width=\"62.5\" x=\"453.0\" y=\"20\"/><text x=\"484.25\" y=\"35\">model</text></g></g><path d=\"M515.5 31h20\"/></g><path d=\"M 535.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 165.5 62\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M125.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">van</text></g></g><path d=\"M105.5 31h20\"/></g><path d=\"M 125.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(readable(gm.tree_to_grammar(csv_dt[0].tree)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `add_tree()` method gets a combined list of non-terminals from current grammar, and the tree to be added to the grammar, and updates the definitions of each non-terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.402661Z",
     "iopub.status.busy": "2024-01-18T17:19:58.402507Z",
     "iopub.status.idle": "2024-01-18T17:19:58.404217Z",
     "shell.execute_reply": "2024-01-18T17:19:58.403948Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.405650Z",
     "iopub.status.busy": "2024-01-18T17:19:58.405542Z",
     "iopub.status.idle": "2024-01-18T17:19:58.407777Z",
     "shell.execute_reply": "2024-01-18T17:19:58.407511Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarMiner(GrammarMiner):\n",
    "    def add_tree(self, t):\n",
    "        t_grammar = self.tree_to_grammar(t.tree)\n",
    "        self.grammar = {\n",
    "            key: self.grammar.get(key, []) + t_grammar.get(key, [])\n",
    "            for key in itertools.chain(self.grammar.keys(), t_grammar.keys())\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `add_tree()` is used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.409505Z",
     "iopub.status.busy": "2024-01-18T17:19:58.409382Z",
     "iopub.status.idle": "2024-01-18T17:19:58.411301Z",
     "shell.execute_reply": "2024-01-18T17:19:58.410980Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "inventory_grammar_miner = GrammarMiner()\n",
    "for dt in csv_dt:\n",
    "    inventory_grammar_miner.add_tree(dt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.413262Z",
     "iopub.status.busy": "2024-01-18T17:19:58.413133Z",
     "iopub.status.idle": "2024-01-18T17:19:58.421756Z",
     "shell.execute_reply": "2024-01-18T17:19:58.421511Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 199.5 62\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M159.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">vehicle</text></g></g><path d=\"M139.5 31h20\"/></g><path d=\"M 159.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 575.5 62\" width=\"575.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M535.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M515.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">year</text></g><path d=\"M114.0 31h10\"/><path d=\"M124.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M134.0 31h0.0\"/><path d=\"M162.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"134.0\" y=\"20\"/><text x=\"148.25\" y=\"35\">,</text></g><path d=\"M162.5 31h10\"/><path d=\"M172.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M182.5 31h0.0\"/><path d=\"M236.5 31h0.0\"/><rect height=\"22\" width=\"54.0\" x=\"182.5\" y=\"20\"/><text x=\"209.5\" y=\"35\">kind</text></g><path d=\"M236.5 31h10\"/><path d=\"M246.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M256.5 31h0.0\"/><path d=\"M285.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"256.5\" y=\"20\"/><text x=\"270.75\" y=\"35\">,</text></g><path d=\"M285.0 31h10\"/><path d=\"M295.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M305.0 31h0.0\"/><path d=\"M384.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"305.0\" y=\"20\"/><text x=\"344.75\" y=\"35\">company</text></g><path d=\"M384.5 31h10\"/><path d=\"M394.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M404.5 31h0.0\"/><path d=\"M433.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"404.5\" y=\"20\"/><text x=\"418.75\" y=\"35\">,</text></g><path d=\"M433.0 31h10\"/><path d=\"M443.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M453.0 31h0.0\"/><path d=\"M515.5 31h0.0\"/><rect height=\"22\" width=\"62.5\" x=\"453.0\" y=\"20\"/><text x=\"484.25\" y=\"35\">model</text></g></g><path d=\"M515.5 31h20\"/></g><path d=\"M 535.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 174.0 122\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1997</text></g></g><path d=\"M114.0 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M114.0 91h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"80\"/><text x=\"87.0\" y=\"95\">1999</text></g></g><path d=\"M114.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 199.5 122\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h12.75\"/><path d=\"M126.75 91h12.75\"/><g class=\"terminal\">\n",
       "<path d=\"M72.75 91h0.0\"/><path d=\"M126.75 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"72.75\" y=\"80\"/><text x=\"99.75\" y=\"95\">Ford</text></g></g><path d=\"M139.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 199.5 122\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h12.75\"/><path d=\"M126.75 31h12.75\"/><g class=\"terminal\">\n",
       "<path d=\"M72.75 31h0.0\"/><path d=\"M126.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"72.75\" y=\"20\"/><text x=\"99.75\" y=\"35\">E350</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M135.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M135.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"50\"/><text x=\"99.75\" y=\"65\">Cougar</text></g></g><path d=\"M139.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M139.5 91h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M139.5 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"80\"/><text x=\"99.75\" y=\"95\">Venture</text></g></g><path d=\"M139.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(readable(inventory_grammar_miner.grammar))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Given execution traces from various inputs, one can define `update_grammar()` to obtain the complete grammar from the traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.423498Z",
     "iopub.status.busy": "2024-01-18T17:19:58.423256Z",
     "iopub.status.idle": "2024-01-18T17:19:58.425929Z",
     "shell.execute_reply": "2024-01-18T17:19:58.425619Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarMiner(GrammarMiner):\n",
    "    def update_grammar(self, inputstr, trace):\n",
    "        at = self.create_tracker(inputstr, trace)\n",
    "        dt = self.create_tree_miner(inputstr, at.assignments())\n",
    "        self.add_tree(dt)\n",
    "        return self.grammar\n",
    "\n",
    "    def create_tracker(self, *args):\n",
    "        return DefineTracker(*args)\n",
    "\n",
    "    def create_tree_miner(self, *args):\n",
    "        return TreeMiner(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The complete grammar recovery is implemented in `recover_grammar()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.427487Z",
     "iopub.status.busy": "2024-01-18T17:19:58.427366Z",
     "iopub.status.idle": "2024-01-18T17:19:58.429935Z",
     "shell.execute_reply": "2024-01-18T17:19:58.429639Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def recover_grammar(fn: Callable, inputs: Iterable[str], \n",
    "                    **kwargs: Any) -> Grammar:\n",
    "    miner = GrammarMiner()\n",
    "\n",
    "    for inputstr in inputs:\n",
    "        with Tracer(inputstr, **kwargs) as tracer:\n",
    "            fn(tracer.my_input)\n",
    "        miner.update_grammar(tracer.my_input, tracer.trace)\n",
    "\n",
    "    return readable(miner.grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that the grammar could have been retrieved directly from the tracker, without the intermediate derivation tree stage. However, going through the derivation tree allows one to inspect the inputs being fragmented and verify that it happens correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 1. Recovering the Inventory Grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.431570Z",
     "iopub.status.busy": "2024-01-18T17:19:58.431467Z",
     "iopub.status.idle": "2024-01-18T17:19:58.435496Z",
     "shell.execute_reply": "2024-01-18T17:19:58.435197Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "inventory_grammar = recover_grammar(process_vehicle, VEHICLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.436921Z",
     "iopub.status.busy": "2024-01-18T17:19:58.436837Z",
     "iopub.status.idle": "2024-01-18T17:19:58.439032Z",
     "shell.execute_reply": "2024-01-18T17:19:58.438778Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': ['<vehicle>'],\n",
       " '<vehicle>': ['<year>,<kind>,<company>,<model>'],\n",
       " '<year>': ['2000', '1997', '1999'],\n",
       " '<kind>': ['car', 'van'],\n",
       " '<company>': ['Mercury', 'Chevy', 'Ford'],\n",
       " '<model>': ['E350', 'Cougar', 'Venture']}"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inventory_grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 2. Recovering URL Grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Our algorithm is robust enough to recover grammar from real world programs. For example, the `urlparse` function in the Python `urlib` module accepts the following sample URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.440492Z",
     "iopub.status.busy": "2024-01-18T17:19:58.440415Z",
     "iopub.status.idle": "2024-01-18T17:19:58.442141Z",
     "shell.execute_reply": "2024-01-18T17:19:58.441876Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "URLS = [\n",
    "    'http://user:pass@www.google.com:80/?q=path#ref',\n",
    "    'https://www.cispa.saarland:80/',\n",
    "    'http://www.fuzzingbook.org/#News',\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The urllib caches its intermediate results for faster access. Hence, we need to disable it using `clear_cache()` after every invocation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.443803Z",
     "iopub.status.busy": "2024-01-18T17:19:58.443698Z",
     "iopub.status.idle": "2024-01-18T17:19:58.445659Z",
     "shell.execute_reply": "2024-01-18T17:19:58.445195Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from urllib.parse import urlparse, clear_cache  # type: ignore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We use the sample URLs to recover grammar as follows. The `urlparse` function tends to cache its previous parsing results. Hence, we define a new method `url_parse()` that clears the cache before each call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.447293Z",
     "iopub.status.busy": "2024-01-18T17:19:58.447203Z",
     "iopub.status.idle": "2024-01-18T17:19:58.449050Z",
     "shell.execute_reply": "2024-01-18T17:19:58.448762Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def url_parse(url):\n",
    "    clear_cache()\n",
    "    urlparse(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.450416Z",
     "iopub.status.busy": "2024-01-18T17:19:58.450309Z",
     "iopub.status.idle": "2024-01-18T17:19:58.571121Z",
     "shell.execute_reply": "2024-01-18T17:19:58.570852Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://user:pass@www.google.com:80/?q=path#ref\n",
      "url = 'http://user:pass@www.google.com:80/?q=path#ref'\n",
      "scheme = 'http'\n",
      "netloc = 'user:pass@www.google.com:80'\n",
      "fragment = 'ref'\n",
      "query = 'q=path'\n",
      "\n",
      "https://www.cispa.saarland:80/\n",
      "url = 'https://www.cispa.saarland:80/'\n",
      "scheme = 'https'\n",
      "netloc = 'www.cispa.saarland:80'\n",
      "\n",
      "http://www.fuzzingbook.org/#News\n",
      "url = 'http://www.fuzzingbook.org/#News'\n",
      "scheme = 'http'\n",
      "netloc = 'www.fuzzingbook.org'\n",
      "fragment = 'News'\n",
      "\n",
      "http://user:pass@www.google.com:80/?q=path#ref\n",
      "https://www.cispa.saarland:80/\n",
      "http://www.fuzzingbook.org/#News\n"
     ]
    }
   ],
   "source": [
    "trees = []\n",
    "for url in URLS:\n",
    "    print(url)\n",
    "    with Tracer(url) as tracer:\n",
    "        url_parse(tracer.my_input)\n",
    "    assignments = DefineTracker(tracer.my_input, tracer.trace).assignments()\n",
    "    trees.append((tracer.my_input, assignments))\n",
    "    for var, val in assignments:\n",
    "        print(var + \" = \" + repr(val))\n",
    "    print()\n",
    "\n",
    "\n",
    "url_dt = []\n",
    "for inputstr, assignments in trees:\n",
    "    print(inputstr)\n",
    "    dt = TreeMiner(inputstr, assignments)\n",
    "    url_dt.append(dt)\n",
    "    display_tree(dt.tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us use `url_parse()` to recover the grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.572786Z",
     "iopub.status.busy": "2024-01-18T17:19:58.572672Z",
     "iopub.status.idle": "2024-01-18T17:19:58.680929Z",
     "shell.execute_reply": "2024-01-18T17:19:58.680621Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "url_grammar = recover_grammar(url_parse, URLS, files=['urllib/parse.py'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.682698Z",
     "iopub.status.busy": "2024-01-18T17:19:58.682602Z",
     "iopub.status.idle": "2024-01-18T17:19:58.691799Z",
     "shell.execute_reply": "2024-01-18T17:19:58.691473Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 165.5 62\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M125.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">url</text></g></g><path d=\"M105.5 31h20\"/></g><path d=\"M 125.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 643.5 122\" width=\"643.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M603.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M583.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">scheme</text></g><path d=\"M131.0 31h10\"/><path d=\"M141.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M151.0 31h0.0\"/><path d=\"M196.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"151.0\" y=\"20\"/><text x=\"173.75\" y=\"35\">://</text></g><path d=\"M196.5 31h10\"/><path d=\"M206.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M216.5 31h0.0\"/><path d=\"M287.5 31h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"216.5\" y=\"20\"/><text x=\"252.0\" y=\"35\">netloc</text></g><path d=\"M287.5 31h10\"/><path d=\"M297.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M307.5 31h0.0\"/><path d=\"M344.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"307.5\" y=\"20\"/><text x=\"326.0\" y=\"35\">/?</text></g><path d=\"M344.5 31h10\"/><path d=\"M354.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M364.5 31h0.0\"/><path d=\"M427.0 31h0.0\"/><rect height=\"22\" width=\"62.5\" x=\"364.5\" y=\"20\"/><text x=\"395.75\" y=\"35\">query</text></g><path d=\"M427.0 31h10\"/><path d=\"M437.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M447.0 31h0.0\"/><path d=\"M475.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"447.0\" y=\"20\"/><text x=\"461.25\" y=\"35\">#</text></g><path d=\"M475.5 31h10\"/><path d=\"M485.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M495.5 31h0.0\"/><path d=\"M583.5 31h0.0\"/><rect height=\"22\" width=\"88.0\" x=\"495.5\" y=\"20\"/><text x=\"539.5\" y=\"35\">fragment</text></g></g><path d=\"M583.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h65.5\"/><path d=\"M518.0 61h65.5\"/><g class=\"non-terminal\">\n",
       "<path d=\"M125.5 61h0.0\"/><path d=\"M196.5 61h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"125.5\" y=\"50\"/><text x=\"161.0\" y=\"65\">scheme</text></g><path d=\"M196.5 61h10\"/><path d=\"M206.5 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M216.5 61h0.0\"/><path d=\"M262.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"216.5\" y=\"50\"/><text x=\"239.25\" y=\"65\">://</text></g><path d=\"M262.0 61h10\"/><path d=\"M272.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M282.0 61h0.0\"/><path d=\"M353.0 61h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"282.0\" y=\"50\"/><text x=\"317.5\" y=\"65\">netloc</text></g><path d=\"M353.0 61h10\"/><path d=\"M363.0 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M373.0 61h0.0\"/><path d=\"M410.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"373.0\" y=\"50\"/><text x=\"391.5\" y=\"65\">/#</text></g><path d=\"M410.0 61h10\"/><path d=\"M420.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M430.0 61h0.0\"/><path d=\"M518.0 61h0.0\"/><rect height=\"22\" width=\"88.0\" x=\"430.0\" y=\"50\"/><text x=\"474.0\" y=\"65\">fragment</text></g></g><path d=\"M583.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h123.75\"/><path d=\"M459.75 91h123.75\"/><g class=\"non-terminal\">\n",
       "<path d=\"M183.75 91h0.0\"/><path d=\"M254.75 91h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"183.75\" y=\"80\"/><text x=\"219.25\" y=\"95\">scheme</text></g><path d=\"M254.75 91h10\"/><path d=\"M264.75 91h10\"/><g class=\"terminal\">\n",
       "<path d=\"M274.75 91h0.0\"/><path d=\"M320.25 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"274.75\" y=\"80\"/><text x=\"297.5\" y=\"95\">://</text></g><path d=\"M320.25 91h10\"/><path d=\"M330.25 91h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M340.25 91h0.0\"/><path d=\"M411.25 91h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"340.25\" y=\"80\"/><text x=\"375.75\" y=\"95\">netloc</text></g><path d=\"M411.25 91h10\"/><path d=\"M421.25 91h10\"/><g class=\"terminal\">\n",
       "<path d=\"M431.25 91h0.0\"/><path d=\"M459.75 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"431.25\" y=\"80\"/><text x=\"445.5\" y=\"95\">/</text></g></g><path d=\"M583.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 603.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 182.5 92\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 369.5 122\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M329.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(url_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The recovered grammar describes the URL format reasonably well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Fuzzing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can now use our recovered grammar for fuzzing as follows.\n",
    "\n",
    "First, the inventory grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.693698Z",
     "iopub.status.busy": "2024-01-18T17:19:58.693484Z",
     "iopub.status.idle": "2024-01-18T17:19:58.696580Z",
     "shell.execute_reply": "2024-01-18T17:19:58.696313Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1999,car,Mercury,E350\n",
      "1997,car,Chevy,Cougar\n",
      "1999,van,Mercury,Venture\n",
      "2000,car,Ford,Venture\n",
      "1997,car,Mercury,E350\n",
      "1997,car,Mercury,Cougar\n",
      "1999,car,Chevy,E350\n",
      "1999,car,Chevy,E350\n",
      "1999,car,Mercury,Cougar\n",
      "2000,car,Chevy,E350\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(inventory_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next, the URL grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.698282Z",
     "iopub.status.busy": "2024-01-18T17:19:58.698193Z",
     "iopub.status.idle": "2024-01-18T17:19:58.700866Z",
     "shell.execute_reply": "2024-01-18T17:19:58.700634Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://user:pass@www.google.com:80/#News\n",
      "http://user:pass@www.google.com:80/\n",
      "http://www.cispa.saarland:80/\n",
      "https://www.fuzzingbook.org/\n",
      "https://www.cispa.saarland:80/?q=path#ref\n",
      "http://www.cispa.saarland:80/\n",
      "http://www.cispa.saarland:80/?q=path#News\n",
      "https://user:pass@www.google.com:80/#ref\n",
      "https://user:pass@www.google.com:80/\n",
      "https://www.cispa.saarland:80/#News\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(url_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "What this means is that we can now take a program and a few samples, extract its grammar, and then use this very grammar for fuzzing.  Now that's quite an opportunity!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Problems with the Simple Miner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One of the problems with our simple grammar miner is the assumption that the values assigned to variables are stable. Unfortunately, that may not hold true in all cases. For example, here is a URL with a slightly different format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.702569Z",
     "iopub.status.busy": "2024-01-18T17:19:58.702490Z",
     "iopub.status.idle": "2024-01-18T17:19:58.704073Z",
     "shell.execute_reply": "2024-01-18T17:19:58.703829Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "URLS_X = URLS + ['ftp://freebsd.org/releases/5.8']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The grammar generated from this set of samples is not as nice as what we got earlier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.705512Z",
     "iopub.status.busy": "2024-01-18T17:19:58.705433Z",
     "iopub.status.idle": "2024-01-18T17:19:58.842925Z",
     "shell.execute_reply": "2024-01-18T17:19:58.842579Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.845059Z",
     "iopub.status.busy": "2024-01-18T17:19:58.844906Z",
     "iopub.status.idle": "2024-01-18T17:19:58.856361Z",
     "shell.execute_reply": "2024-01-18T17:19:58.856067Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 413.0 92\" width=\"413.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M373.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M353.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">scheme</text></g><path d=\"M131.0 31h10\"/><path d=\"M141.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M151.0 31h0.0\"/><path d=\"M196.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"151.0\" y=\"20\"/><text x=\"173.75\" y=\"35\">://</text></g><path d=\"M196.5 31h10\"/><path d=\"M206.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M216.5 31h0.0\"/><path d=\"M287.5 31h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"216.5\" y=\"20\"/><text x=\"252.0\" y=\"35\">netloc</text></g><path d=\"M287.5 31h10\"/><path d=\"M297.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M307.5 31h0.0\"/><path d=\"M353.0 31h0.0\"/><rect height=\"22\" width=\"45.5\" x=\"307.5\" y=\"20\"/><text x=\"330.25\" y=\"35\">url</text></g></g><path d=\"M353.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h123.75\"/><path d=\"M229.25 61h123.75\"/><g class=\"non-terminal\">\n",
       "<path d=\"M183.75 61h0.0\"/><path d=\"M229.25 61h0.0\"/><rect height=\"22\" width=\"45.5\" x=\"183.75\" y=\"50\"/><text x=\"206.5\" y=\"65\">url</text></g></g><path d=\"M353.0 61h20\"/></g><path d=\"M 373.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"152\" viewBox=\"0 0 643.5 152\" width=\"643.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 81v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 91h0.0\"/><path d=\"M603.5 91h0.0\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M583.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"60.0\" y=\"50\"/><text x=\"95.5\" y=\"65\">scheme</text></g><path d=\"M131.0 61h10\"/><path d=\"M141.0 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M151.0 61h0.0\"/><path d=\"M196.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"151.0\" y=\"50\"/><text x=\"173.75\" y=\"65\">://</text></g><path d=\"M196.5 61h10\"/><path d=\"M206.5 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M216.5 61h0.0\"/><path d=\"M287.5 61h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"216.5\" y=\"50\"/><text x=\"252.0\" y=\"65\">netloc</text></g><path d=\"M287.5 61h10\"/><path d=\"M297.5 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M307.5 61h0.0\"/><path d=\"M344.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"307.5\" y=\"50\"/><text x=\"326.0\" y=\"65\">/?</text></g><path d=\"M344.5 61h10\"/><path d=\"M354.5 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M364.5 61h0.0\"/><path d=\"M427.0 61h0.0\"/><rect height=\"22\" width=\"62.5\" x=\"364.5\" y=\"50\"/><text x=\"395.75\" y=\"65\">query</text></g><path d=\"M427.0 61h10\"/><path d=\"M437.0 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M447.0 61h0.0\"/><path d=\"M475.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"447.0\" y=\"50\"/><text x=\"461.25\" y=\"65\">#</text></g><path d=\"M475.5 61h10\"/><path d=\"M485.5 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M495.5 61h0.0\"/><path d=\"M583.5 61h0.0\"/><rect height=\"22\" width=\"88.0\" x=\"495.5\" y=\"50\"/><text x=\"539.5\" y=\"65\">fragment</text></g></g><path d=\"M583.5 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-40a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h196.5\"/><path d=\"M387.0 31h196.5\"/><g class=\"terminal\">\n",
       "<path d=\"M256.5 31h0.0\"/><path d=\"M387.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"130.5\" x=\"256.5\" y=\"20\"/><text x=\"321.75\" y=\"35\">/releases/5.8</text></g></g><path d=\"M583.5 31a10 10 0 0 1 10 10v40a10 10 0 0 0 10 10\"/><path d=\"M40.0 91h20\"/><g>\n",
       "<path d=\"M60.0 91h65.5\"/><path d=\"M518.0 91h65.5\"/><g class=\"non-terminal\">\n",
       "<path d=\"M125.5 91h0.0\"/><path d=\"M196.5 91h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"125.5\" y=\"80\"/><text x=\"161.0\" y=\"95\">scheme</text></g><path d=\"M196.5 91h10\"/><path d=\"M206.5 91h10\"/><g class=\"terminal\">\n",
       "<path d=\"M216.5 91h0.0\"/><path d=\"M262.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"216.5\" y=\"80\"/><text x=\"239.25\" y=\"95\">://</text></g><path d=\"M262.0 91h10\"/><path d=\"M272.0 91h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M282.0 91h0.0\"/><path d=\"M353.0 91h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"282.0\" y=\"80\"/><text x=\"317.5\" y=\"95\">netloc</text></g><path d=\"M353.0 91h10\"/><path d=\"M363.0 91h10\"/><g class=\"terminal\">\n",
       "<path d=\"M373.0 91h0.0\"/><path d=\"M410.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"373.0\" y=\"80\"/><text x=\"391.5\" y=\"95\">/#</text></g><path d=\"M410.0 91h10\"/><path d=\"M420.0 91h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M430.0 91h0.0\"/><path d=\"M518.0 91h0.0\"/><rect height=\"22\" width=\"88.0\" x=\"430.0\" y=\"80\"/><text x=\"474.0\" y=\"95\">fragment</text></g></g><path d=\"M583.5 91h20\"/><path d=\"M40.0 91a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 121h123.75\"/><path d=\"M459.75 121h123.75\"/><g class=\"non-terminal\">\n",
       "<path d=\"M183.75 121h0.0\"/><path d=\"M254.75 121h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"183.75\" y=\"110\"/><text x=\"219.25\" y=\"125\">scheme</text></g><path d=\"M254.75 121h10\"/><path d=\"M264.75 121h10\"/><g class=\"terminal\">\n",
       "<path d=\"M274.75 121h0.0\"/><path d=\"M320.25 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"274.75\" y=\"110\"/><text x=\"297.5\" y=\"125\">://</text></g><path d=\"M320.25 121h10\"/><path d=\"M330.25 121h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M340.25 121h0.0\"/><path d=\"M411.25 121h0.0\"/><rect height=\"22\" width=\"71.0\" x=\"340.25\" y=\"110\"/><text x=\"375.75\" y=\"125\">netloc</text></g><path d=\"M411.25 121h10\"/><path d=\"M421.25 121h10\"/><g class=\"terminal\">\n",
       "<path d=\"M431.25 121h0.0\"/><path d=\"M459.75 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"431.25\" y=\"110\"/><text x=\"445.5\" y=\"125\">/</text></g></g><path d=\"M583.5 121a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 603.5 91 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 182.5 122\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h8.5\"/><path d=\"M114.0 91h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"68.5\" y=\"80\"/><text x=\"91.25\" y=\"95\">ftp</text></g></g><path d=\"M122.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"152\" viewBox=\"0 0 369.5 152\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 81v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 91h0.0\"/><path d=\"M329.5 91h0.0\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-40a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v40a10 10 0 0 0 10 10\"/><path d=\"M40.0 91h20\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91h20\"/><path d=\"M40.0 91a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 121h68.0\"/><path d=\"M241.5 121h68.0\"/><g class=\"terminal\">\n",
       "<path d=\"M128.0 121h0.0\"/><path d=\"M241.5 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"113.5\" x=\"128.0\" y=\"110\"/><text x=\"184.75\" y=\"125\">freebsd.org</text></g></g><path d=\"M309.5 121a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 91 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(url_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Clearly, something has gone wrong.\n",
    "\n",
    "To investigate why the `url` definition has gone wrong, let us inspect the trace for the URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.858126Z",
     "iopub.status.busy": "2024-01-18T17:19:58.858044Z",
     "iopub.status.idle": "2024-01-18T17:19:58.886184Z",
     "shell.execute_reply": "2024-01-18T17:19:58.885767Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 372 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "1 392 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "5 124 ({'arg': ''},)\n",
      "6 121 ({'arg': ''},)\n",
      "7 126 ({'arg': ''},)\n",
      "8 127 ({'arg': ''},)\n",
      "10 393 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "11 437 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "12 458 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "16 124 ({'arg': ''},)\n",
      "17 121 ({'arg': ''},)\n",
      "18 126 ({'arg': ''},)\n",
      "19 127 ({'arg': ''},)\n",
      "21 460 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "22 461 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\t'},)\n",
      "23 462 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\t'},)\n",
      "24 460 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\t'},)\n",
      "25 461 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\r'},)\n",
      "26 462 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\r'},)\n",
      "27 460 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\r'},)\n",
      "28 461 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "29 462 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "30 460 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "31 464 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "32 465 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "33 466 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "34 467 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "35 469 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "36 471 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n'},)\n",
      "37 472 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': ''},)\n",
      "38 473 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': ''},)\n",
      "39 474 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': ''},)\n",
      "40 475 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'h'},)\n",
      "41 474 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'h'},)\n",
      "42 475 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 't'},)\n",
      "43 474 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 't'},)\n",
      "44 475 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 't'},)\n",
      "45 474 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 't'},)\n",
      "46 475 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "47 474 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "48 478 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': '', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "49 480 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "50 481 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': '', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "51 411 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)\n",
      "52 412 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)\n",
      "53 413 ({'url': '//user:pass@www.google.com:80/?q=path#ref'},)\n",
      "54 414 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)\n",
      "55 415 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)\n",
      "56 416 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)\n",
      "57 413 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '/'},)\n",
      "58 414 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)\n",
      "59 415 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)\n",
      "60 416 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)\n",
      "61 413 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '?'},)\n",
      "62 414 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)\n",
      "63 415 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)\n",
      "64 416 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)\n",
      "65 413 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)\n",
      "66 417 ({'url': '//user:pass@www.google.com:80/?q=path#ref', 'c': '#'},)\n",
      "68 482 ({'url': '/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "69 483 ({'url': '/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "70 482 ({'url': '/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "71 485 ({'url': '/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "72 486 ({'url': '/?q=path#ref', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': '', 'c': 'p'},)\n",
      "73 487 ({'url': '/?q=path', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': 'ref', 'c': 'p'},)\n",
      "74 488 ({'url': '/?q=path', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': '', 'fragment': 'ref', 'c': 'p'},)\n",
      "75 489 ({'url': '/', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'c': 'p'},)\n",
      "76 419 ({'netloc': 'user:pass@www.google.com:80'},)\n",
      "77 420 ({'netloc': 'user:pass@www.google.com:80'},)\n",
      "78 421 ({'netloc': 'user:pass@www.google.com:80'},)\n",
      "80 490 ({'url': '/', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'c': 'p'},)\n",
      "84 491 ({'url': '/', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'c': 'p'},)\n",
      "85 492 ({'url': '/', 'scheme': 'http', 'b': '\\n', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'c': 'p'},)\n",
      "90 394 ({'url': 'http://user:pass@www.google.com:80/?q=path#ref', 'scheme': ''},)\n",
      "91 395 ({'url': '/', 'scheme': 'http', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref'},)\n",
      "92 398 ({'url': '/', 'scheme': 'http', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref'},)\n",
      "93 399 ({'url': '/', 'scheme': 'http', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'params': ''},)\n",
      "97 400 ({'url': '/', 'scheme': 'http', 'netloc': 'user:pass@www.google.com:80', 'query': 'q=path', 'fragment': 'ref', 'params': ''},)\n"
     ]
    }
   ],
   "source": [
    "clear_cache()\n",
    "with Tracer(URLS_X[0]) as tracer:\n",
    "    urlparse(tracer.my_input)\n",
    "for i, t in enumerate(tracer.trace):\n",
    "    if t[0] in {'call', 'line'} and 'parse.py' in str(t[2]) and t[3]:\n",
    "        print(i, t[2]._t()[1], t[3:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Notice how the value of `url` changes as the parsing progresses? This violates our assumption that the value assigned to a variable is stable. We next look at how this limitation can be removed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Grammar Miner with Reassignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One way to uniquely identify different variables is to annotate them with *line numbers* both when they are defined and also when their value changes. Consider the code fragment below"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tracking variable assignment locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.888121Z",
     "iopub.status.busy": "2024-01-18T17:19:58.888031Z",
     "iopub.status.idle": "2024-01-18T17:19:58.889792Z",
     "shell.execute_reply": "2024-01-18T17:19:58.889422Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def C(cp_1):\n",
    "    c_2 = cp_1 + '@2'\n",
    "    c_3 = c_2 + '@3'\n",
    "    return c_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.891450Z",
     "iopub.status.busy": "2024-01-18T17:19:58.891362Z",
     "iopub.status.idle": "2024-01-18T17:19:58.893276Z",
     "shell.execute_reply": "2024-01-18T17:19:58.892870Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def B(bp_7):\n",
    "    b_8 = bp_7 + '@8'\n",
    "    return C(b_8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.895100Z",
     "iopub.status.busy": "2024-01-18T17:19:58.895015Z",
     "iopub.status.idle": "2024-01-18T17:19:58.897070Z",
     "shell.execute_reply": "2024-01-18T17:19:58.896671Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def A(ap_12):\n",
    "    a_13 = ap_12 + '@13'\n",
    "    a_14 = B(a_13) + '@14'\n",
    "    a_14 = a_14 + '@15'\n",
    "    a_13 = a_14 + '@16'\n",
    "    a_14 = B(a_13) + '@17'\n",
    "    a_14 = B(a_13) + '@18'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Notice how all variables are either named corresponding to either where they are defined, or the value is annotated to indicate that it was changed.\n",
    "\n",
    "Let us run this under the trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:58.898636Z",
     "iopub.status.busy": "2024-01-18T17:19:58.898562Z",
     "iopub.status.idle": "2024-01-18T17:19:59.004165Z",
     "shell.execute_reply": "2024-01-18T17:19:59.003805Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "call 1:A {'ap_12': '____'}\n",
      "line 2:A {'ap_12': '____'}\n",
      "line 3:A {'ap_12': '____', 'a_13': '____@13'}\n",
      "call 1:B {'bp_7': '____@13'}\n",
      "line 2:B {'bp_7': '____@13'}\n",
      "line 3:B {'bp_7': '____@13', 'b_8': '____@13@8'}\n",
      "call 1:C {'cp_1': '____@13@8'}\n",
      "line 2:C {'cp_1': '____@13@8'}\n",
      "line 3:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2'}\n",
      "line 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}\n",
      "return 4:C {'cp_1': '____@13@8', 'c_2': '____@13@8@2', 'c_3': '____@13@8@2@3'}\n",
      "return 3:B {'bp_7': '____@13', 'b_8': '____@13@8'}\n",
      "line 4:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14'}\n",
      "line 5:A {'ap_12': '____', 'a_13': '____@13', 'a_14': '____@13@8@2@3@14@15'}\n",
      "line 6:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15'}\n",
      "call 1:B {'bp_7': '____@13@8@2@3@14@15@16'}\n",
      "line 2:B {'bp_7': '____@13@8@2@3@14@15@16'}\n",
      "line 3:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}\n",
      "call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}\n",
      "line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}\n",
      "line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}\n",
      "line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}\n",
      "return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}\n",
      "return 3:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}\n",
      "line 7:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@17'}\n",
      "call 1:B {'bp_7': '____@13@8@2@3@14@15@16'}\n",
      "line 2:B {'bp_7': '____@13@8@2@3@14@15@16'}\n",
      "line 3:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}\n",
      "call 1:C {'cp_1': '____@13@8@2@3@14@15@16@8'}\n",
      "line 2:C {'cp_1': '____@13@8@2@3@14@15@16@8'}\n",
      "line 3:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2'}\n",
      "line 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}\n",
      "return 4:C {'cp_1': '____@13@8@2@3@14@15@16@8', 'c_2': '____@13@8@2@3@14@15@16@8@2', 'c_3': '____@13@8@2@3@14@15@16@8@2@3'}\n",
      "return 3:B {'bp_7': '____@13@8@2@3@14@15@16', 'b_8': '____@13@8@2@3@14@15@16@8'}\n",
      "return 7:A {'ap_12': '____', 'a_13': '____@13@8@2@3@14@15@16', 'a_14': '____@13@8@2@3@14@15@16@8@2@3@18'}\n",
      "call 102:__exit__ {}\n",
      "line 105:__exit__ {}\n"
     ]
    }
   ],
   "source": [
    "with Tracer('____') as tracer:\n",
    "    A(tracer.my_input)\n",
    "\n",
    "for t in tracer.trace:\n",
    "    print(t[0], \"%d:%s\" % (t[2].line_no, t[2].method), t[3])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Each variable was referenced first as follows:\n",
    "\n",
    "* `cp_1` -- *call* `1:C`\n",
    "* `c_2` -- *line* `3:C` (but the previous event was *line* `2:C`)\n",
    "* `c_3` -- *line* `4:C` (but the previous event was *line* `3:C`)\n",
    "* `bp_7` -- *call* `7:B`\n",
    "* `b_8` -- *line* `9:B` (but the previous event was *line* `8:B`)\n",
    "* `ap_12` -- *call* `12:A`\n",
    "* `a_13` -- *line* `14:A` (but the previous event was *line* `13:A`)\n",
    "* `a_14` -- *line* `15:A` (the previous event was *return* `9:B`. However, the previous event in `A()` was *line* `14:A`)\n",
    "* reassign `a_14` at *15* -- *line* `16:A` (the previous event was *line* `15:A`)\n",
    "* reassign `a_13` at *16* -- *line* `17:A` (the previous event was *line* `16:A`)\n",
    "* reassign `a_14` at *17* -- *return* `17:A` (the previous event in `A()` was *line* `17:A`)\n",
    "* reassign `a_14` at *18* -- *return* `18:A` (the previous event in `A()` was *line* `18:A`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So, our observations are that, if it is a call, the current location is the right one for any new variables being defined. On the other hand, if the variable being referenced for the first time (or reassigned a new value), then the  right location to consider is the previous location *in the same method invocation*. Next, let us see how we can incorporate this information into variable naming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next, we need a way to track the individual method calls as they are being made. For this we define the class `CallStack`. Each method invocation gets a separate identifier, and when the method call is over, the identifier is reset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### CallStack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.006198Z",
     "iopub.status.busy": "2024-01-18T17:19:59.006073Z",
     "iopub.status.idle": "2024-01-18T17:19:59.008910Z",
     "shell.execute_reply": "2024-01-18T17:19:59.008593Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class CallStack:\n",
    "    def __init__(self, **kwargs):\n",
    "        self.options(kwargs)\n",
    "        self.method_id = (START_SYMBOL, 0)\n",
    "        self.method_register = 0\n",
    "        self.mstack = [self.method_id]\n",
    "\n",
    "    def enter(self, method):\n",
    "        self.method_register += 1\n",
    "        self.method_id = (method, self.method_register)\n",
    "        self.log('call', \"%s%s\" % (self.indent(), str(self)))\n",
    "        self.mstack.append(self.method_id)\n",
    "\n",
    "    def leave(self):\n",
    "        self.mstack.pop()\n",
    "        self.log('return', \"%s%s\" % (self.indent(), str(self)))\n",
    "        self.method_id = self.mstack[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A few extra functions to make life simpler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.010983Z",
     "iopub.status.busy": "2024-01-18T17:19:59.010667Z",
     "iopub.status.idle": "2024-01-18T17:19:59.014023Z",
     "shell.execute_reply": "2024-01-18T17:19:59.013489Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class CallStack(CallStack):\n",
    "    def options(self, kwargs):\n",
    "        self.log = log_event if kwargs.get('log') else lambda _evt, _var: None\n",
    "\n",
    "    def indent(self):\n",
    "        return len(self.mstack) * \"\\t\"\n",
    "\n",
    "    def at(self, n):\n",
    "        return self.mstack[n]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(mstack) - 1\n",
    "\n",
    "    def __str__(self):\n",
    "        return \"%s:%d\" % self.method_id\n",
    "\n",
    "    def __repr__(self):\n",
    "        return repr(self.method_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We also define a convenience method to display a given stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.015903Z",
     "iopub.status.busy": "2024-01-18T17:19:59.015814Z",
     "iopub.status.idle": "2024-01-18T17:19:59.018205Z",
     "shell.execute_reply": "2024-01-18T17:19:59.017908Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def display_stack(istack):\n",
    "    def stack_to_tree(stack):\n",
    "        current, *rest = stack\n",
    "        if not rest:\n",
    "            return (repr(current), [])\n",
    "        return (repr(current), [stack_to_tree(rest)])\n",
    "    display_tree(stack_to_tree(istack.mstack), graph_attr=lr_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here is how we can use the `CallStack`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.020128Z",
     "iopub.status.busy": "2024-01-18T17:19:59.019900Z",
     "iopub.status.idle": "2024-01-18T17:19:59.022689Z",
     "shell.execute_reply": "2024-01-18T17:19:59.022358Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('<start>', 0)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs = CallStack()\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.024625Z",
     "iopub.status.busy": "2024-01-18T17:19:59.024468Z",
     "iopub.status.idle": "2024-01-18T17:19:59.027057Z",
     "shell.execute_reply": "2024-01-18T17:19:59.026699Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('hello', 1)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs.enter('hello')\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.029109Z",
     "iopub.status.busy": "2024-01-18T17:19:59.028912Z",
     "iopub.status.idle": "2024-01-18T17:19:59.031760Z",
     "shell.execute_reply": "2024-01-18T17:19:59.031451Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('world', 2)"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs.enter('world')\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.033438Z",
     "iopub.status.busy": "2024-01-18T17:19:59.033307Z",
     "iopub.status.idle": "2024-01-18T17:19:59.035811Z",
     "shell.execute_reply": "2024-01-18T17:19:59.035468Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('hello', 1)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs.leave()\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.037668Z",
     "iopub.status.busy": "2024-01-18T17:19:59.037525Z",
     "iopub.status.idle": "2024-01-18T17:19:59.040053Z",
     "shell.execute_reply": "2024-01-18T17:19:59.039707Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('world', 3)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs.enter('world')\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.041766Z",
     "iopub.status.busy": "2024-01-18T17:19:59.041654Z",
     "iopub.status.idle": "2024-01-18T17:19:59.044026Z",
     "shell.execute_reply": "2024-01-18T17:19:59.043680Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('hello', 1)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cs.leave()\n",
    "display_stack(cs)\n",
    "cs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In order to account for variable reassignments, we need to have a more intelligent data structure than a dictionary for storing variables. We first define a simple interface `Vars`. It acts as a container for variables, and is instantiated at `my_assignments`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Vars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `Vars` stores references to variables as they occur during parsing in its internal dictionary `defs`. We initialize the dictionary with the original string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.045922Z",
     "iopub.status.busy": "2024-01-18T17:19:59.045805Z",
     "iopub.status.idle": "2024-01-18T17:19:59.047812Z",
     "shell.execute_reply": "2024-01-18T17:19:59.047456Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Vars:\n",
    "    def __init__(self, original):\n",
    "        self.defs = {}\n",
    "        self.my_input = original"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The dictionary needs two methods: `update()` that takes a set of key-value pairs to update itself, and `_set_kv()` that updates a particular key-value pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.049564Z",
     "iopub.status.busy": "2024-01-18T17:19:59.049442Z",
     "iopub.status.idle": "2024-01-18T17:19:59.051636Z",
     "shell.execute_reply": "2024-01-18T17:19:59.051335Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Vars(Vars):\n",
    "    def _set_kv(self, k, v):\n",
    "        self.defs[k] = v\n",
    "\n",
    "    def __setitem__(self, k, v):\n",
    "        self._set_kv(k, v)\n",
    "\n",
    "    def update(self, v):\n",
    "        for k, v in v.items():\n",
    "            self._set_kv(k, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `Vars` is a proxy for the internal dictionary. For example, here is how one can use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.053792Z",
     "iopub.status.busy": "2024-01-18T17:19:59.053672Z",
     "iopub.status.idle": "2024-01-18T17:19:59.055951Z",
     "shell.execute_reply": "2024-01-18T17:19:59.055624Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v = Vars('')\n",
    "v.defs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.057629Z",
     "iopub.status.busy": "2024-01-18T17:19:59.057529Z",
     "iopub.status.idle": "2024-01-18T17:19:59.059718Z",
     "shell.execute_reply": "2024-01-18T17:19:59.059370Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': 'X'}"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v['x'] = 'X'\n",
    "v.defs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.061604Z",
     "iopub.status.busy": "2024-01-18T17:19:59.061457Z",
     "iopub.status.idle": "2024-01-18T17:19:59.063929Z",
     "shell.execute_reply": "2024-01-18T17:19:59.063610Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x': 'x', 'y': 'y'}"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.update({'x': 'x', 'y': 'y'})\n",
    "v.defs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### AssignmentVars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We now extend the simple `Vars` to account for variable reassignments. For this, we define `AssignmentVars`.\n",
    "\n",
    "The idea for detecting reassignments and renaming variables is as follows: We keep track of the previous reassignments to particular variables using `accessed_seq_var`. It contains the last rename of any particular variable as its corresponding value. The `new_vars` contains a list of all new variables that were added on this iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.065781Z",
     "iopub.status.busy": "2024-01-18T17:19:59.065667Z",
     "iopub.status.idle": "2024-01-18T17:19:59.067923Z",
     "shell.execute_reply": "2024-01-18T17:19:59.067571Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(Vars):\n",
    "    def __init__(self, original):\n",
    "        super().__init__(original)\n",
    "        self.accessed_seq_var = {}\n",
    "        self.var_def_lines = {}\n",
    "        self.current_event = None\n",
    "        self.new_vars = set()\n",
    "        self.method_init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `method_init()` method takes care of keeping track of method invocations using records saved in the `call_stack`. `event_locations` is for keeping track of the locations accessed *within this method*. This is used for line number tracking of variable definitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.070258Z",
     "iopub.status.busy": "2024-01-18T17:19:59.069903Z",
     "iopub.status.idle": "2024-01-18T17:19:59.072414Z",
     "shell.execute_reply": "2024-01-18T17:19:59.072059Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def method_init(self):\n",
    "        self.call_stack = CallStack()\n",
    "        self.event_locations = {self.call_stack.method_id: []}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `update()` is now modified to track the changed line numbers if any, using `var_location_register()`. We reinitialize the `new_vars` after use for the next event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.074006Z",
     "iopub.status.busy": "2024-01-18T17:19:59.073900Z",
     "iopub.status.idle": "2024-01-18T17:19:59.075789Z",
     "shell.execute_reply": "2024-01-18T17:19:59.075533Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def update(self, v):\n",
    "        for k, v in v.items():\n",
    "            self._set_kv(k, v)\n",
    "        self.var_location_register(self.new_vars)\n",
    "        self.new_vars = set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The variable name now incorporates an index of how many reassignments it has gone through, effectively making each reassignment a unique variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.077351Z",
     "iopub.status.busy": "2024-01-18T17:19:59.077219Z",
     "iopub.status.idle": "2024-01-18T17:19:59.079143Z",
     "shell.execute_reply": "2024-01-18T17:19:59.078834Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def var_name(self, var):\n",
    "        return (var, self.accessed_seq_var[var])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "While storing variables, we need to first check whether it was previously known. If it is not, we need to initialize the rename count. This is accomplished by `var_access`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.080816Z",
     "iopub.status.busy": "2024-01-18T17:19:59.080683Z",
     "iopub.status.idle": "2024-01-18T17:19:59.082639Z",
     "shell.execute_reply": "2024-01-18T17:19:59.082359Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def var_access(self, var):\n",
    "        if var not in self.accessed_seq_var:\n",
    "            self.accessed_seq_var[var] = 0\n",
    "        return self.var_name(var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "During a variable reassignment, we update the `accessed_seq_var` to reflect the new count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.084238Z",
     "iopub.status.busy": "2024-01-18T17:19:59.084067Z",
     "iopub.status.idle": "2024-01-18T17:19:59.086170Z",
     "shell.execute_reply": "2024-01-18T17:19:59.085893Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def var_assign(self, var):\n",
    "        self.accessed_seq_var[var] += 1\n",
    "        self.new_vars.add(self.var_name(var))\n",
    "        return self.var_name(var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "These methods can be used as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.087676Z",
     "iopub.status.busy": "2024-01-18T17:19:59.087560Z",
     "iopub.status.idle": "2024-01-18T17:19:59.089556Z",
     "shell.execute_reply": "2024-01-18T17:19:59.089293Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav = AssignmentVars('')\n",
    "sav.defs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.090958Z",
     "iopub.status.busy": "2024-01-18T17:19:59.090864Z",
     "iopub.status.idle": "2024-01-18T17:19:59.093004Z",
     "shell.execute_reply": "2024-01-18T17:19:59.092665Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('v1', 0)"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav.var_access('v1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.094865Z",
     "iopub.status.busy": "2024-01-18T17:19:59.094733Z",
     "iopub.status.idle": "2024-01-18T17:19:59.096994Z",
     "shell.execute_reply": "2024-01-18T17:19:59.096684Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('v1', 1)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav.var_assign('v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Assigning to it again increments the counter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.098825Z",
     "iopub.status.busy": "2024-01-18T17:19:59.098701Z",
     "iopub.status.idle": "2024-01-18T17:19:59.100624Z",
     "shell.execute_reply": "2024-01-18T17:19:59.100358Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('v1', 2)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav.var_assign('v1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The core of the logic is in `_set_kv()`. When a variable is being assigned, we get the sequenced variable name `s_var`. If the sequenced variable name was previously unknown in `defs`, then we have no further concerns. We add the sequenced variable to `defs`.\n",
    "\n",
    "If the variable is previously known, then it is an indication of a possible reassignment. In this case, we look at the value the variable is holding. We check if the value changed. If it has not, then it is not.\n",
    "\n",
    "If the value has changed, it is a reassignment. We first increment the variable usage sequence using `var_assign`, retrieve the new name, update the new name in `defs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.102143Z",
     "iopub.status.busy": "2024-01-18T17:19:59.102044Z",
     "iopub.status.idle": "2024-01-18T17:19:59.104075Z",
     "shell.execute_reply": "2024-01-18T17:19:59.103761Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def _set_kv(self, var, val):\n",
    "        s_var = self.var_access(var)\n",
    "        if s_var in self.defs and self.defs[s_var] == val:\n",
    "            return\n",
    "        self.defs[self.var_assign(var)] = val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here is how it can be used. Assigning a variable the first time initializes its counter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.105627Z",
     "iopub.status.busy": "2024-01-18T17:19:59.105515Z",
     "iopub.status.idle": "2024-01-18T17:19:59.107525Z",
     "shell.execute_reply": "2024-01-18T17:19:59.107283Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('x', 1): 'X'}"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav = AssignmentVars('')\n",
    "sav['x'] = 'X'\n",
    "sav.defs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If the variable is assigned again with the same value, it is probably not a reassignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.109051Z",
     "iopub.status.busy": "2024-01-18T17:19:59.108933Z",
     "iopub.status.idle": "2024-01-18T17:19:59.110988Z",
     "shell.execute_reply": "2024-01-18T17:19:59.110628Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('x', 1): 'X'}"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav['x'] = 'X'\n",
    "sav.defs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "However, if the value changed, it is a reassignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.112688Z",
     "iopub.status.busy": "2024-01-18T17:19:59.112558Z",
     "iopub.status.idle": "2024-01-18T17:19:59.115194Z",
     "shell.execute_reply": "2024-01-18T17:19:59.114789Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('x', 1): 'X', ('x', 2): 'Y'}"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sav['x'] = 'Y'\n",
    "sav.defs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There is a subtlety here. It is possible for a child method to be called from the middle of a parent method, and for both to use the same variable name with different values. In this case, when the child returns, parent will have the old variable with old value in context. With our implementation, we consider this as a reassignment. However, this is OK because adding a new reassignment is harmless, but missing one is not. Further, we will discuss later how this can be avoided."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We also define bookkeeping codes for `register_event()` `method_enter()` and `method_exit()` which are the methods responsible for keeping track of the method stack. The basic idea is that, each `method_enter()` represents a new method invocation. Hence, it merits a new method id, which is generated from the `method_register`, and saved in the `method_id`. Since this is a new method, the method stack is extended by one element with this id. In the case of `method_exit()`, we pop the method stack, and reset the current `method_id` to what was below the current one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.116832Z",
     "iopub.status.busy": "2024-01-18T17:19:59.116721Z",
     "iopub.status.idle": "2024-01-18T17:19:59.119219Z",
     "shell.execute_reply": "2024-01-18T17:19:59.118988Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def method_enter(self, cxt, my_vars):\n",
    "        self.current_event = 'call'\n",
    "        self.call_stack.enter(cxt.method)\n",
    "        self.event_locations[self.call_stack.method_id] = []\n",
    "        self.register_event(cxt)\n",
    "        self.update(my_vars)\n",
    "\n",
    "    def method_exit(self, cxt, my_vars):\n",
    "        self.current_event = 'return'\n",
    "        self.register_event(cxt)\n",
    "        self.update(my_vars)\n",
    "        self.call_stack.leave()\n",
    "\n",
    "    def method_statement(self, cxt, my_vars):\n",
    "        self.current_event = 'line'\n",
    "        self.register_event(cxt)\n",
    "        self.update(my_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For each of the method events, we also register the event using `register_event()` which keeps track of the line numbers that were referenced in *this* method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.120875Z",
     "iopub.status.busy": "2024-01-18T17:19:59.120769Z",
     "iopub.status.idle": "2024-01-18T17:19:59.122468Z",
     "shell.execute_reply": "2024-01-18T17:19:59.122245Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def register_event(self, cxt):\n",
    "        self.event_locations[self.call_stack.method_id].append(cxt.line_no)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `var_location_register()` keeps the locations of newly added variables. The definition location of variables in a `call` is the *current*  location. However, for a `line`, it would be the previous event in the current method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.123998Z",
     "iopub.status.busy": "2024-01-18T17:19:59.123895Z",
     "iopub.status.idle": "2024-01-18T17:19:59.126215Z",
     "shell.execute_reply": "2024-01-18T17:19:59.125957Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def var_location_register(self, my_vars):\n",
    "        def loc(mid):\n",
    "            if self.current_event == 'call':\n",
    "                return self.event_locations[mid][-1]\n",
    "            elif self.current_event == 'line':\n",
    "                return self.event_locations[mid][-2]\n",
    "            elif self.current_event == 'return':\n",
    "                return self.event_locations[mid][-2]\n",
    "            else:\n",
    "                assert False\n",
    "\n",
    "        my_loc = loc(self.call_stack.method_id)\n",
    "        for var in my_vars:\n",
    "            self.var_def_lines[var] = my_loc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We define `defined_vars()` which returns the names of variables annotated with the line numbers as below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.127926Z",
     "iopub.status.busy": "2024-01-18T17:19:59.127808Z",
     "iopub.status.idle": "2024-01-18T17:19:59.130022Z",
     "shell.execute_reply": "2024-01-18T17:19:59.129760Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def defined_vars(self, formatted=True):\n",
    "        def fmt(k):\n",
    "            v = (k[0], self.var_def_lines[k])\n",
    "            return \"%s@%s\" % v if formatted else v\n",
    "\n",
    "        return [(fmt(k), v) for k, v in self.defs.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Similar to `defined_vars()` we define `seq_vars()` which annotates different variables with the number of times they were used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.131716Z",
     "iopub.status.busy": "2024-01-18T17:19:59.131582Z",
     "iopub.status.idle": "2024-01-18T17:19:59.133876Z",
     "shell.execute_reply": "2024-01-18T17:19:59.133515Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentVars(AssignmentVars):\n",
    "    def seq_vars(self, formatted=True):\n",
    "        def fmt(k):\n",
    "            v = (k[0], self.var_def_lines[k], k[1])\n",
    "            return \"%s@%s:%s\" % v if formatted else v\n",
    "\n",
    "        return {fmt(k): v for k, v in self.defs.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### AssignmentTracker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `AssignmentTracker` keeps the assignment definitions using the `AssignmentVars` we defined previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.136037Z",
     "iopub.status.busy": "2024-01-18T17:19:59.135858Z",
     "iopub.status.idle": "2024-01-18T17:19:59.138301Z",
     "shell.execute_reply": "2024-01-18T17:19:59.137957Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentTracker(DefineTracker):\n",
    "    def __init__(self, my_input, trace, **kwargs):\n",
    "        self.options(kwargs)\n",
    "        self.my_input = my_input\n",
    "\n",
    "        self.my_assignments = self.create_assignments(my_input)\n",
    "\n",
    "        self.trace = trace\n",
    "        self.process()\n",
    "\n",
    "    def create_assignments(self, *args):\n",
    "        return AssignmentVars(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To fine-tune the process, we define an optional parameter called `track_return`. During tracing a method return, Python produces a virtual variable that contains the result of the returned value. If the `track_return` is set, we capture this value as a variable.\n",
    "\n",
    "* `track_return` -- if true, add a *virtual variable* to the Vars representing the return value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.139926Z",
     "iopub.status.busy": "2024-01-18T17:19:59.139813Z",
     "iopub.status.idle": "2024-01-18T17:19:59.141684Z",
     "shell.execute_reply": "2024-01-18T17:19:59.141437Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentTracker(AssignmentTracker):\n",
    "    def options(self, kwargs):\n",
    "        self.track_return = kwargs.get('track_return', False)\n",
    "        super().options(kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There can be different kinds of events during a trace, which includes `call` when a function is entered, `return` when the function returns, `exception` when an exception is thrown and `line` when a statement is executed.\n",
    "\n",
    "The previous `Tracker` was too simplistic in that it did not distinguish between the different events. We rectify that and define `on_call()`, `on_return()`, and `on_line()` respectively, which get called on their corresponding events.\n",
    "\n",
    "Note that `on_line()` is called also for `on_return()`. The reason is, that Python invokes the trace function *before* the corresponding line is executed. Hence, effectively, the `on_return()` is called with the binding produced by the execution of the previous statement in the environment. Our processing in effect is done on values that were bound by the previous statement. Hence, calling `on_line()` here is appropriate as it provides the event handler a chance to work on the previous binding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.143326Z",
     "iopub.status.busy": "2024-01-18T17:19:59.143205Z",
     "iopub.status.idle": "2024-01-18T17:19:59.146632Z",
     "shell.execute_reply": "2024-01-18T17:19:59.146202Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class AssignmentTracker(AssignmentTracker):\n",
    "    def on_call(self, arg, cxt, my_vars):\n",
    "        my_vars = cxt.parameters(my_vars)\n",
    "        self.my_assignments.method_enter(cxt, self.fragments(my_vars))\n",
    "\n",
    "    def on_line(self, arg, cxt, my_vars):\n",
    "        self.my_assignments.method_statement(cxt, self.fragments(my_vars))\n",
    "\n",
    "    def on_return(self, arg, cxt, my_vars):\n",
    "        self.on_line(arg, cxt, my_vars)\n",
    "        my_vars = {'<-%s' % cxt.method: arg} if self.track_return else {}\n",
    "        self.my_assignments.method_exit(cxt, my_vars)\n",
    "\n",
    "    def on_exception(self, arg, cxt, my_vara):\n",
    "        return\n",
    "\n",
    "    def track_event(self, event, arg, cxt, my_vars):\n",
    "        self.current_event = event\n",
    "        dispatch = {\n",
    "            'call': self.on_call,\n",
    "            'return': self.on_return,\n",
    "            'line': self.on_line,\n",
    "            'exception': self.on_exception\n",
    "        }\n",
    "        dispatch[event](arg, cxt, my_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can now use `AssignmentTracker` to track the different variables. To verify that our variable line number inference works, we recover definitions from the functions `A()`, `B()` and `C()` (with data annotations removed so that the input fragments are correctly identified). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.148548Z",
     "iopub.status.busy": "2024-01-18T17:19:59.148448Z",
     "iopub.status.idle": "2024-01-18T17:19:59.150191Z",
     "shell.execute_reply": "2024-01-18T17:19:59.149885Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def C(cp_1):  # type: ignore\n",
    "    c_2 = cp_1\n",
    "    c_3 = c_2\n",
    "    return c_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.151764Z",
     "iopub.status.busy": "2024-01-18T17:19:59.151666Z",
     "iopub.status.idle": "2024-01-18T17:19:59.153361Z",
     "shell.execute_reply": "2024-01-18T17:19:59.153063Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def B(bp_7):  # type: ignore\n",
    "    b_8 = bp_7\n",
    "    return C(b_8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.154837Z",
     "iopub.status.busy": "2024-01-18T17:19:59.154754Z",
     "iopub.status.idle": "2024-01-18T17:19:59.156577Z",
     "shell.execute_reply": "2024-01-18T17:19:59.156331Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def A(ap_12):  # type: ignore\n",
    "    a_13 = ap_12\n",
    "    a_14 = B(a_13)\n",
    "    a_14 = a_14\n",
    "    a_13 = a_14\n",
    "    a_14 = B(a_13)\n",
    "    a_14 = B(a_14)[3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Running `A()` with sufficient input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.158064Z",
     "iopub.status.busy": "2024-01-18T17:19:59.157977Z",
     "iopub.status.idle": "2024-01-18T17:19:59.262249Z",
     "shell.execute_reply": "2024-01-18T17:19:59.261944Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ap_12@1:1 = '---xxx'\n",
      "a_13@2:1 = '---xxx'\n",
      "bp_7@1:1 = '---xxx'\n",
      "b_8@2:1 = '---xxx'\n",
      "cp_1@1:1 = '---xxx'\n",
      "c_2@2:1 = '---xxx'\n",
      "c_3@3:1 = '---xxx'\n",
      "a_14@3:1 = '---xxx'\n",
      "a_14@7:2 = 'xxx'\n",
      "\n",
      "ap_12@1 = '---xxx'\n",
      "a_13@2 = '---xxx'\n",
      "bp_7@1 = '---xxx'\n",
      "b_8@2 = '---xxx'\n",
      "cp_1@1 = '---xxx'\n",
      "c_2@2 = '---xxx'\n",
      "c_3@3 = '---xxx'\n",
      "a_14@3 = '---xxx'\n",
      "a_14@7 = 'xxx'\n"
     ]
    }
   ],
   "source": [
    "with Tracer('---xxx') as tracer:\n",
    "    A(tracer.my_input)\n",
    "tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)\n",
    "for k, v in tracker.my_assignments.seq_vars().items():\n",
    "    print(k, '=', repr(v))\n",
    "print()\n",
    "for k, v in tracker.my_assignments.defined_vars(formatted=True):\n",
    "    print(k, '=', repr(v))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As can be seen, the line numbers are now correctly identified for each variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " Let us try retrieving the assignments for a real world example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.264366Z",
     "iopub.status.busy": "2024-01-18T17:19:59.264204Z",
     "iopub.status.idle": "2024-01-18T17:19:59.386053Z",
     "shell.execute_reply": "2024-01-18T17:19:59.385683Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@372 = 'http://user:pass@www.google.com:80/?q=path#ref'\n",
      "url@478 = '//user:pass@www.google.com:80/?q=path#ref'\n",
      "scheme@478 = 'http'\n",
      "url@481 = '/?q=path#ref'\n",
      "netloc@481 = 'user:pass@www.google.com:80'\n",
      "url@486 = '/?q=path'\n",
      "fragment@486 = 'ref'\n",
      "query@488 = 'q=path'\n",
      "url@393 = 'http://user:pass@www.google.com:80/?q=path#ref'\n",
      "\n",
      "url@372 = 'https://www.cispa.saarland:80/'\n",
      "url@478 = '//www.cispa.saarland:80/'\n",
      "scheme@478 = 'https'\n",
      "netloc@481 = 'www.cispa.saarland:80'\n",
      "url@393 = 'https://www.cispa.saarland:80/'\n",
      "\n",
      "url@372 = 'http://www.fuzzingbook.org/#News'\n",
      "url@478 = '//www.fuzzingbook.org/#News'\n",
      "scheme@478 = 'http'\n",
      "url@481 = '/#News'\n",
      "netloc@481 = 'www.fuzzingbook.org'\n",
      "fragment@486 = 'News'\n",
      "url@393 = 'http://www.fuzzingbook.org/#News'\n",
      "\n",
      "url@372 = 'ftp://freebsd.org/releases/5.8'\n",
      "url@478 = '//freebsd.org/releases/5.8'\n",
      "scheme@478 = 'ftp'\n",
      "url@481 = '/releases/5.8'\n",
      "netloc@481 = 'freebsd.org'\n",
      "url@393 = 'ftp://freebsd.org/releases/5.8'\n",
      "url@394 = '/releases/5.8'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "traces = []\n",
    "for inputstr in URLS_X:\n",
    "    clear_cache()\n",
    "    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:\n",
    "        urlparse(tracer.my_input)\n",
    "    traces.append((tracer.my_input, tracer.trace))\n",
    "\n",
    "    tracker = AssignmentTracker(tracer.my_input, tracer.trace, log=True)\n",
    "    for k, v in tracker.my_assignments.defined_vars():\n",
    "        print(k, '=', repr(v))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The line numbers of variables can be verified from the source code of [urllib/parse.py](https://github.com/python/cpython/blob/3.6/Lib/urllib/parse.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recovering a Derivation Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Does handling variable reassignments help with our URL examples? We look at these next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.388488Z",
     "iopub.status.busy": "2024-01-18T17:19:59.388294Z",
     "iopub.status.idle": "2024-01-18T17:19:59.390501Z",
     "shell.execute_reply": "2024-01-18T17:19:59.390226Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class TreeMiner(TreeMiner):\n",
    "    def get_derivation_tree(self):\n",
    "        tree = (START_SYMBOL, [(self.my_input, [])])\n",
    "        for var, value in self.my_assignments:\n",
    "            self.log(0, \"%s=%s\" % (var, repr(value)))\n",
    "            self.apply_new_definition(tree, var, value)\n",
    "        return tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 1: Recovering URL Derivation Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "First we obtain the derivation tree of the URL 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### URL 1 derivation tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.392176Z",
     "iopub.status.busy": "2024-01-18T17:19:59.392067Z",
     "iopub.status.idle": "2024-01-18T17:19:59.818202Z",
     "shell.execute_reply": "2024-01-18T17:19:59.817862Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"512pt\" height=\"324pt\"\n",
       " viewBox=\"0.00 0.00 511.88 323.75\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 319.75)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-319.75 507.88,-319.75 507.88,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-302.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-252.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@372&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.38,-301.71C123.38,-295.49 123.38,-285.4 123.38,-276.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"126.88,-276.85 123.38,-266.85 119.88,-276.85 126.88,-276.85\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"45.38\" y=\"-201.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;scheme@478&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M113.5,-251.26C101.24,-243.68 80.07,-230.58 64.57,-221\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"66.56,-218.12 56.22,-215.83 62.88,-224.07 66.56,-218.12\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-201.95\" font-family=\"Times,serif\" font-size=\"14.00\">: (58)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.38,-251.46C123.38,-245.24 123.38,-235.15 123.38,-226.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"126.88,-226.6 123.38,-216.6 119.88,-226.6 126.88,-226.6\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-201.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@478&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M131.61,-251.26C141.53,-243.9 158.45,-231.34 171.25,-221.84\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"173.25,-224.71 179.2,-215.94 169.08,-219.09 173.25,-224.71\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"45.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">http</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M45.38,-201.21C45.38,-194.99 45.38,-184.9 45.38,-176.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"48.88,-176.35 45.38,-166.35 41.88,-176.35 48.88,-176.35\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"125.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">//</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M180.4,-201.01C169.74,-192.86 150.75,-178.31 138.11,-168.63\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"140.27,-165.87 130.2,-162.57 136.01,-171.43 140.27,-165.87\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;netloc@481&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.38,-201.21C188.38,-194.99 188.38,-184.9 188.38,-176.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"191.88,-176.35 188.38,-166.35 184.88,-176.35 191.88,-176.35\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"336.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@481&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>5&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M207.12,-201.01C232.25,-192.82 277.14,-178.19 306.84,-168.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"307.92,-171.83 316.34,-165.41 305.75,-165.18 307.92,-171.83\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"165.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">user:pass@www.google.com:80</text>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>7&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M185.55,-150.96C182.49,-144.52 177.44,-133.94 173.17,-124.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"176.33,-123.48 168.87,-115.96 170.01,-126.49 176.33,-123.48\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"303.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@486&gt;</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M332.33,-150.96C327.78,-144.3 320.2,-133.23 313.94,-124.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"316.84,-122.12 308.31,-115.84 311.07,-126.07 316.84,-122.12\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<text text-anchor=\"middle\" x=\"370.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\"># (35)</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>9&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M340.54,-150.96C345.29,-144.23 353.21,-132.98 359.7,-123.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"362.4,-126.02 365.3,-115.83 356.68,-121.99 362.4,-126.02\"/>\n",
       "</g>\n",
       "<!-- 15 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>15</title>\n",
       "<text text-anchor=\"middle\" x=\"454.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;fragment@486&gt;</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;15 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>9&#45;&gt;15</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M351.32,-150.76C370.83,-142.79 405.26,-128.71 428.92,-119.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"430.22,-122.28 438.15,-115.26 427.57,-115.8 430.22,-122.28\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"271.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">/?</text>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>10&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M299.45,-100.71C295.04,-94.05 287.69,-82.98 281.62,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"284.61,-72 276.17,-65.6 278.78,-75.87 284.61,-72\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"334.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;query@488&gt;</text>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>10&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M307.18,-100.71C311.45,-94.05 318.57,-82.98 324.45,-73.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"327.26,-75.92 329.72,-65.61 321.37,-72.13 327.26,-75.92\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<text text-anchor=\"middle\" x=\"334.38\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">q=path</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M334.38,-50.46C334.38,-44.24 334.38,-34.15 334.38,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"337.88,-25.6 334.38,-15.6 330.88,-25.6 337.88,-25.6\"/>\n",
       "</g>\n",
       "<!-- 16 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>16</title>\n",
       "<text text-anchor=\"middle\" x=\"454.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">ref</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;16 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>15&#45;&gt;16</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M454.38,-100.71C454.38,-94.49 454.38,-84.4 454.38,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"457.88,-75.85 454.38,-65.85 450.88,-75.85 457.88,-75.85\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x112f4f790>"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clear_cache()\n",
    "with Tracer(URLS_X[0], files=['urllib/parse.py']) as tracer:\n",
    "    urlparse(tracer.my_input)\n",
    "sm = AssignmentTracker(tracer.my_input, tracer.trace)\n",
    "dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())\n",
    "display_tree(dt.tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next, we obtain the derivation tree of URL 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### URL 4 derivation tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:19:59.819966Z",
     "iopub.status.busy": "2024-01-18T17:19:59.819850Z",
     "iopub.status.idle": "2024-01-18T17:20:00.242062Z",
     "shell.execute_reply": "2024-01-18T17:20:00.241702Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"322pt\" height=\"274pt\"\n",
       " viewBox=\"0.00 0.00 322.12 273.50\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 269.5)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-269.5 318.12,-269.5 318.12,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-252.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-201.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@372&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.38,-251.46C123.38,-245.24 123.38,-235.15 123.38,-226.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"126.88,-226.6 123.38,-216.6 119.88,-226.6 126.88,-226.6\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"45.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;scheme@478&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M113.5,-201.01C101.24,-193.43 80.07,-180.33 64.57,-170.75\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"66.56,-167.87 56.22,-165.58 62.88,-173.82 66.56,-167.87\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"123.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">: (58)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.38,-201.21C123.38,-194.99 123.38,-184.9 123.38,-176.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"126.88,-176.35 123.38,-166.35 119.88,-176.35 126.88,-176.35\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-151.7\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@478&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M131.61,-201.01C141.53,-193.65 158.45,-181.09 171.25,-171.59\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"173.25,-174.46 179.2,-165.69 169.08,-168.84 173.25,-174.46\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"45.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">ftp</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M45.38,-150.96C45.38,-144.74 45.38,-134.65 45.38,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"48.88,-126.1 45.38,-116.1 41.88,-126.1 48.88,-126.1\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"125.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">//</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M180.4,-150.76C169.74,-142.61 150.75,-128.06 138.11,-118.38\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"140.27,-115.62 130.2,-112.32 136.01,-121.18 140.27,-115.62\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;netloc@481&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.38,-150.96C188.38,-144.74 188.38,-134.65 188.38,-125.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"191.88,-126.1 188.38,-116.1 184.88,-126.1 191.88,-126.1\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"280.38\" y=\"-101.45\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@481&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>5&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M200.03,-150.76C214.75,-143.04 240.38,-129.6 258.74,-119.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"260.25,-123.13 267.48,-115.39 257,-116.93 260.25,-123.13\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"188.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">freebsd.org</text>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>7&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M188.38,-100.71C188.38,-94.49 188.38,-84.4 188.38,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"191.88,-75.85 188.38,-65.85 184.88,-75.85 191.88,-75.85\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"280.38\" y=\"-51.2\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;url@394&gt;</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M280.38,-100.71C280.38,-94.49 280.38,-84.4 280.38,-75.64\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"283.88,-75.85 280.38,-65.85 276.88,-75.85 283.88,-75.85\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"280.38\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">/releases/5.8</text>\n",
       "</g>\n",
       "<!-- 10&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>10&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M280.38,-50.46C280.38,-44.24 280.38,-34.15 280.38,-25.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"283.88,-25.6 280.38,-15.6 276.88,-25.6 283.88,-25.6\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1075c7df0>"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clear_cache()\n",
    "with Tracer(URLS_X[-1], files=['urllib/parse.py']) as tracer:\n",
    "    urlparse(tracer.my_input)\n",
    "sm = AssignmentTracker(tracer.my_input, tracer.trace)\n",
    "dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())\n",
    "display_tree(dt.tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The derivation trees seem to belong to the same grammar. Hence, we obtain the grammar for the complete set. First, we update the `recover_grammar()` to use `AssignTracker`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recover Grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.244118Z",
     "iopub.status.busy": "2024-01-18T17:20:00.243980Z",
     "iopub.status.idle": "2024-01-18T17:20:00.246644Z",
     "shell.execute_reply": "2024-01-18T17:20:00.246307Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarMiner(GrammarMiner):\n",
    "    def update_grammar(self, inputstr, trace):\n",
    "        at = self.create_tracker(inputstr, trace)\n",
    "        dt = self.create_tree_miner(inputstr, at.my_assignments.defined_vars())\n",
    "        self.add_tree(dt)\n",
    "        return self.grammar\n",
    "\n",
    "    def create_tracker(self, *args):\n",
    "        return AssignmentTracker(*args)\n",
    "\n",
    "    def create_tree_miner(self, *args):\n",
    "        return TreeMiner(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next, we use the modified `recover_grammar()` on derivation trees obtained from URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.248146Z",
     "iopub.status.busy": "2024-01-18T17:20:00.248037Z",
     "iopub.status.idle": "2024-01-18T17:20:00.389875Z",
     "shell.execute_reply": "2024-01-18T17:20:00.389589Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The recovered grammar is below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.391737Z",
     "iopub.status.busy": "2024-01-18T17:20:00.391645Z",
     "iopub.status.idle": "2024-01-18T17:20:00.406113Z",
     "shell.execute_reply": "2024-01-18T17:20:00.405812Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 199.5 62\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M159.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">url@372</text></g></g><path d=\"M139.5 31h20\"/></g><path d=\"M 159.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@372\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 373.0 62\" width=\"373.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M333.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M313.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M165.0 31h0.0\"/><rect height=\"22\" width=\"105.0\" x=\"60.0\" y=\"20\"/><text x=\"112.5\" y=\"35\">scheme@478</text></g><path d=\"M165.0 31h10\"/><path d=\"M175.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M185.0 31h0.0\"/><path d=\"M213.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"185.0\" y=\"20\"/><text x=\"199.25\" y=\"35\">:</text></g><path d=\"M213.5 31h10\"/><path d=\"M223.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M233.5 31h0.0\"/><path d=\"M313.0 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"233.5\" y=\"20\"/><text x=\"273.25\" y=\"35\">url@478</text></g></g><path d=\"M313.0 31h20\"/></g><path d=\"M 333.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme@478\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 182.5 122\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h8.5\"/><path d=\"M114.0 91h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"68.5\" y=\"80\"/><text x=\"91.25\" y=\"95\">ftp</text></g></g><path d=\"M122.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@478\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 381.5 92\" width=\"381.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M341.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h25.5\"/><path d=\"M296.0 31h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"85.5\" y=\"20\"/><text x=\"104.0\" y=\"35\">//</text></g><path d=\"M122.5 31h10\"/><path d=\"M132.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M142.5 31h0.0\"/><path d=\"M247.5 31h0.0\"/><rect height=\"22\" width=\"105.0\" x=\"142.5\" y=\"20\"/><text x=\"195.0\" y=\"35\">netloc@481</text></g><path d=\"M247.5 31h10\"/><path d=\"M257.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M267.5 31h0.0\"/><path d=\"M296.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"267.5\" y=\"20\"/><text x=\"281.75\" y=\"35\">/</text></g></g><path d=\"M321.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M321.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M97.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"50\"/><text x=\"78.5\" y=\"65\">//</text></g><path d=\"M97.0 61h10\"/><path d=\"M107.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 61h0.0\"/><path d=\"M222.0 61h0.0\"/><rect height=\"22\" width=\"105.0\" x=\"117.0\" y=\"50\"/><text x=\"169.5\" y=\"65\">netloc@481</text></g><path d=\"M222.0 61h10\"/><path d=\"M232.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M242.0 61h0.0\"/><path d=\"M321.5 61h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"242.0\" y=\"50\"/><text x=\"281.75\" y=\"65\">url@481</text></g></g><path d=\"M321.5 61h20\"/></g><path d=\"M 341.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netloc@481\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"152\" viewBox=\"0 0 369.5 152\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 81v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 91h0.0\"/><path d=\"M329.5 91h0.0\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-40a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v40a10 10 0 0 0 10 10\"/><path d=\"M40.0 91h20\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91h20\"/><path d=\"M40.0 91a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 121h68.0\"/><path d=\"M241.5 121h68.0\"/><g class=\"terminal\">\n",
       "<path d=\"M128.0 121h0.0\"/><path d=\"M241.5 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"113.5\" x=\"128.0\" y=\"110\"/><text x=\"184.75\" y=\"125\">freebsd.org</text></g></g><path d=\"M309.5 121a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 91 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@481\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 390.0 122\" width=\"390.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M350.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M330.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">url@486</text></g><path d=\"M139.5 31h10\"/><path d=\"M149.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M159.5 31h0.0\"/><path d=\"M188.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"159.5\" y=\"20\"/><text x=\"173.75\" y=\"35\">#</text></g><path d=\"M188.0 31h10\"/><path d=\"M198.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M208.0 31h0.0\"/><path d=\"M330.0 31h0.0\"/><rect height=\"22\" width=\"122.0\" x=\"208.0\" y=\"20\"/><text x=\"269.0\" y=\"35\">fragment@486</text></g></g><path d=\"M330.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h95.25\"/><path d=\"M234.75 61h95.25\"/><g class=\"non-terminal\">\n",
       "<path d=\"M155.25 61h0.0\"/><path d=\"M234.75 61h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"155.25\" y=\"50\"/><text x=\"195.0\" y=\"65\">url@394</text></g></g><path d=\"M330.0 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h45.5\"/><path d=\"M284.5 91h45.5\"/><g class=\"terminal\">\n",
       "<path d=\"M105.5 91h0.0\"/><path d=\"M142.5 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"105.5\" y=\"80\"/><text x=\"124.0\" y=\"95\">/#</text></g><path d=\"M142.5 91h10\"/><path d=\"M152.5 91h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M162.5 91h0.0\"/><path d=\"M284.5 91h0.0\"/><rect height=\"22\" width=\"122.0\" x=\"162.5\" y=\"80\"/><text x=\"223.5\" y=\"95\">fragment@486</text></g></g><path d=\"M330.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 350.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@486\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 273.5 62\" width=\"273.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M233.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M213.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M97.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"20\"/><text x=\"78.5\" y=\"35\">/?</text></g><path d=\"M97.0 31h10\"/><path d=\"M107.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 31h0.0\"/><path d=\"M213.5 31h0.0\"/><rect height=\"22\" width=\"96.5\" x=\"117.0\" y=\"20\"/><text x=\"165.25\" y=\"35\">query@488</text></g></g><path d=\"M213.5 31h20\"/></g><path d=\"M 233.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query@488\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fragment@486\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url@394\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 250.5 62\" width=\"250.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M210.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"130.5\" x=\"60.0\" y=\"20\"/><text x=\"125.25\" y=\"35\">/releases/5.8</text></g></g><path d=\"M190.5 31h20\"/></g><path d=\"M 210.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(url_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us fuzz a little to see if the produced values are sane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.407735Z",
     "iopub.status.busy": "2024-01-18T17:20:00.407646Z",
     "iopub.status.idle": "2024-01-18T17:20:00.410896Z",
     "shell.execute_reply": "2024-01-18T17:20:00.410636Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://user:pass@www.google.com:80/#News\n",
      "https://freebsd.org/\n",
      "ftp://www.fuzzingbook.org/\n",
      "https://freebsd.org/\n",
      "https://user:pass@www.google.com:80/#ref\n",
      "http://user:pass@www.google.com:80/?q=path#News\n",
      "http://www.cispa.saarland:80/#ref\n",
      "https://www.fuzzingbook.org/#News\n",
      "ftp://www.fuzzingbook.org/\n",
      "ftp://user:pass@www.google.com:80/#ref\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(url_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Our modifications do seem to help. Next, we check whether we can still retrieve the grammar for inventory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 2: Recovering Inventory Grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.412640Z",
     "iopub.status.busy": "2024-01-18T17:20:00.412511Z",
     "iopub.status.idle": "2024-01-18T17:20:00.417833Z",
     "shell.execute_reply": "2024-01-18T17:20:00.417601Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "inventory_grammar = recover_grammar(process_vehicle, VEHICLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.419316Z",
     "iopub.status.busy": "2024-01-18T17:20:00.419226Z",
     "iopub.status.idle": "2024-01-18T17:20:00.428370Z",
     "shell.execute_reply": "2024-01-18T17:20:00.428092Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 225.0 62\" width=\"225.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M185.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M165.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M165.0 31h0.0\"/><rect height=\"22\" width=\"105.0\" x=\"60.0\" y=\"20\"/><text x=\"112.5\" y=\"35\">vehicle@29</text></g></g><path d=\"M165.0 31h20\"/></g><path d=\"M 185.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicle@29\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 677.5 62\" width=\"677.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M637.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M617.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">year@30</text></g><path d=\"M139.5 31h10\"/><path d=\"M149.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M159.5 31h0.0\"/><path d=\"M188.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"159.5\" y=\"20\"/><text x=\"173.75\" y=\"35\">,</text></g><path d=\"M188.0 31h10\"/><path d=\"M198.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M208.0 31h0.0\"/><path d=\"M287.5 31h0.0\"/><rect height=\"22\" width=\"79.5\" x=\"208.0\" y=\"20\"/><text x=\"247.75\" y=\"35\">kind@30</text></g><path d=\"M287.5 31h10\"/><path d=\"M297.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M307.5 31h0.0\"/><path d=\"M336.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"307.5\" y=\"20\"/><text x=\"321.75\" y=\"35\">,</text></g><path d=\"M336.0 31h10\"/><path d=\"M346.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M356.0 31h0.0\"/><path d=\"M461.0 31h0.0\"/><rect height=\"22\" width=\"105.0\" x=\"356.0\" y=\"20\"/><text x=\"408.5\" y=\"35\">company@30</text></g><path d=\"M461.0 31h10\"/><path d=\"M471.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M481.0 31h0.0\"/><path d=\"M509.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"481.0\" y=\"20\"/><text x=\"495.25\" y=\"35\">,</text></g><path d=\"M509.5 31h10\"/><path d=\"M519.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M529.5 31h0.0\"/><path d=\"M617.5 31h0.0\"/><rect height=\"22\" width=\"88.0\" x=\"529.5\" y=\"20\"/><text x=\"573.5\" y=\"35\">model@30</text></g></g><path d=\"M617.5 31h20\"/></g><path d=\"M 637.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "year@30\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 174.0 122\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1997</text></g></g><path d=\"M114.0 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M114.0 91h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"80\"/><text x=\"87.0\" y=\"95\">1999</text></g></g><path d=\"M114.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind@30\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "company@30\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 199.5 122\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h12.75\"/><path d=\"M126.75 91h12.75\"/><g class=\"terminal\">\n",
       "<path d=\"M72.75 91h0.0\"/><path d=\"M126.75 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"72.75\" y=\"80\"/><text x=\"99.75\" y=\"95\">Ford</text></g></g><path d=\"M139.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model@30\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 199.5 122\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h12.75\"/><path d=\"M126.75 31h12.75\"/><g class=\"terminal\">\n",
       "<path d=\"M72.75 31h0.0\"/><path d=\"M126.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"72.75\" y=\"20\"/><text x=\"99.75\" y=\"35\">E350</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M135.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M135.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"50\"/><text x=\"99.75\" y=\"65\">Cougar</text></g></g><path d=\"M139.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M139.5 91h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 91h0.0\"/><path d=\"M139.5 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"80\"/><text x=\"99.75\" y=\"95\">Venture</text></g></g><path d=\"M139.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(inventory_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Using fuzzing to produce values from the grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.430113Z",
     "iopub.status.busy": "2024-01-18T17:20:00.429882Z",
     "iopub.status.idle": "2024-01-18T17:20:00.432937Z",
     "shell.execute_reply": "2024-01-18T17:20:00.432700Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000,van,Chevy,Cougar\n",
      "2000,van,Ford,Cougar\n",
      "1999,van,Chevy,E350\n",
      "1997,car,Ford,E350\n",
      "2000,van,Mercury,Venture\n",
      "1999,car,Ford,E350\n",
      "1999,car,Mercury,Venture\n",
      "1999,car,Mercury,E350\n",
      "1997,van,Mercury,Cougar\n",
      "1999,car,Chevy,Venture\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(inventory_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Problems with the Grammar Miner with Reassignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "One of the problems with our grammar miner is that it doesn't yet account for the current context. That is, when replacing, a variable can replace tokens that it does not have access to (and hence, it is not a fragment of). Consider this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.434672Z",
     "iopub.status.busy": "2024-01-18T17:20:00.434585Z",
     "iopub.status.idle": "2024-01-18T17:20:00.869062Z",
     "shell.execute_reply": "2024-01-18T17:20:00.868626Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"508pt\" height=\"598pt\"\n",
       " viewBox=\"0.00 0.00 508.25 598.25\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 594.25)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-594.25 504.25,-594.25 504.25,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-256.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"123\" y=\"-256.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;inventory@22&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M39.42,-263.12C46.64,-263.12 55.33,-263.12 64.33,-263.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"64.3,-266.63 74.3,-263.13 64.3,-259.63 64.3,-266.63\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"247.12\" y=\"-464.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle@24&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M128.06,-270.05C145.12,-299.1 211.24,-411.71 236.36,-454.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"233.32,-456.23 241.4,-463.08 239.36,-452.69 233.32,-456.23\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<text text-anchor=\"middle\" x=\"247.12\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">\\n (10)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>1&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M151.1,-270.23C170.96,-275.43 197.84,-282.47 217.94,-287.74\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"216.87,-291.08 227.43,-290.23 218.64,-284.3 216.87,-291.08\"/>\n",
       "</g>\n",
       "<!-- 15 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>15</title>\n",
       "<text text-anchor=\"middle\" x=\"247.12\" y=\"-256.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle@24&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;15 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>1&#45;&gt;15</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M170.17,-263.12C178.25,-263.12 186.7,-263.12 194.91,-263.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"194.77,-266.63 204.77,-263.13 194.77,-259.63 194.77,-266.63\"/>\n",
       "</g>\n",
       "<!-- 27 -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>27</title>\n",
       "<text text-anchor=\"middle\" x=\"247.12\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">\\n (10)</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;27 -->\n",
       "<g id=\"edge27\" class=\"edge\">\n",
       "<title>1&#45;&gt;27</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M151.1,-256.02C170.96,-250.82 197.84,-243.78 217.94,-238.51\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"218.64,-241.95 227.43,-236.02 216.87,-235.17 218.64,-241.95\"/>\n",
       "</g>\n",
       "<!-- 28 -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>28</title>\n",
       "<text text-anchor=\"middle\" x=\"247.12\" y=\"-80.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;vehicle@24&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;28 -->\n",
       "<g id=\"edge28\" class=\"edge\">\n",
       "<title>1&#45;&gt;28</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M128.94,-256.01C146.89,-230.14 208.98,-140.66 234.76,-103.5\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"237.57,-105.59 240.4,-95.38 231.82,-101.6 237.57,-105.59\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-576.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year@30&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M251.74,-478.02C261.22,-495.48 288.56,-541.82 324,-567.12 326.18,-568.69 328.54,-570.11 330.98,-571.4\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"329.39,-574.52 339.95,-575.39 332.23,-568.12 329.39,-574.52\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-544.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>2&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M254.59,-478.05C266.96,-490.91 295.28,-518.7 324,-535.12 330.5,-538.84 338.02,-541.92 345.05,-544.34\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"343.77,-547.61 354.36,-547.26 345.87,-540.93 343.77,-547.61\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind@30&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>2&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M264,-478.15C279.27,-484.83 303.06,-495.04 324,-503.12 328.48,-504.86 333.24,-506.62 337.93,-508.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"336.54,-511.53 347.14,-511.56 338.87,-504.93 336.54,-511.53\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-480.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>2&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M287.87,-476.36C306.49,-478.82 328.11,-481.67 344.42,-483.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"343.85,-487.27 354.22,-485.11 344.76,-480.33 343.85,-487.27\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company@30&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>2&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M287.87,-465.89C295.85,-464.83 304.39,-463.71 312.82,-462.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"313.24,-466.07 322.7,-461.29 312.32,-459.13 313.24,-466.07\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-416.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>2&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M264,-464.1C279.27,-457.42 303.06,-447.21 324,-439.12 330.81,-436.5 338.25,-433.8 345.11,-431.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"346.14,-434.74 354.45,-428.17 343.86,-428.12 346.14,-434.74\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-384.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model@30&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>2&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M254.59,-464.2C266.96,-451.34 295.28,-423.55 324,-407.12 327.03,-405.39 330.28,-403.8 333.59,-402.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"334.78,-405.64 342.82,-398.74 332.23,-399.13 334.78,-405.64\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-576.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M403.31,-583.12C418.94,-583.12 437.46,-583.12 451.85,-583.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"451.74,-586.63 461.74,-583.13 451.74,-579.63 451.74,-586.63\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>6&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M403.59,-519.12C420.6,-519.12 440.96,-519.12 455.69,-519.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"455.44,-522.63 465.44,-519.13 455.44,-515.63 455.44,-522.63\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M416.87,-455.12C429.18,-455.12 442,-455.12 452.51,-455.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"452.49,-458.63 462.49,-455.13 452.49,-451.63 452.49,-458.63\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-384.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.09,-391.12C423.07,-391.12 438.57,-391.12 451.08,-391.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"450.92,-394.63 460.92,-391.13 450.92,-387.63 450.92,-394.63\"/>\n",
       "</g>\n",
       "<!-- 16 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>16</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-352.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year@30&gt;</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;16 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>15&#45;&gt;16</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M273.22,-270.19C278.49,-272.46 283.74,-275.39 288,-279.12 312.54,-300.64 299,-322.14 324,-343.12 325.68,-344.54 327.5,-345.83 329.41,-347.01\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"327.59,-350.02 338.1,-351.35 330.72,-343.75 327.59,-350.02\"/>\n",
       "</g>\n",
       "<!-- 18 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>18</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;18 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>15&#45;&gt;18</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M269.48,-270.13C275.67,-272.57 282.29,-275.59 288,-279.12 306.2,-290.39 305.56,-300.26 324,-311.12 330.46,-314.93 337.96,-318.03 344.99,-320.45\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"343.72,-323.72 354.31,-323.35 345.8,-317.04 343.72,-323.72\"/>\n",
       "</g>\n",
       "<!-- 19 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>19</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;kind@30&gt;</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;19 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>15&#45;&gt;19</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M275.06,-270.23C291.72,-274.62 313.35,-280.32 331.71,-285.16\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"330.61,-288.49 341.17,-287.66 332.4,-281.72 330.61,-288.49\"/>\n",
       "</g>\n",
       "<!-- 21 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>21</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-256.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;21 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>15&#45;&gt;21</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M287.87,-263.12C306.49,-263.12 328.11,-263.12 344.42,-263.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"344.21,-266.63 354.21,-263.13 344.21,-259.63 344.21,-266.63\"/>\n",
       "</g>\n",
       "<!-- 22 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>22</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company@30&gt;</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;22 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>15&#45;&gt;22</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M275.06,-256.02C291.72,-251.63 313.35,-245.93 331.71,-241.09\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"332.4,-244.53 341.17,-238.59 330.61,-237.76 332.4,-244.53\"/>\n",
       "</g>\n",
       "<!-- 24 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>24</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-192.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;24 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>15&#45;&gt;24</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M269.48,-256.12C275.67,-253.68 282.29,-250.66 288,-247.12 306.2,-235.86 305.56,-225.99 324,-215.12 330.46,-211.32 337.96,-208.22 344.99,-205.8\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"345.8,-209.21 354.31,-202.9 343.72,-202.53 345.8,-209.21\"/>\n",
       "</g>\n",
       "<!-- 25 -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>25</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-160.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model@30&gt;</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;25 -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>15&#45;&gt;25</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M273.22,-256.06C278.49,-253.79 283.74,-250.86 288,-247.12 312.54,-225.61 299,-204.11 324,-183.12 325.68,-181.71 327.5,-180.42 329.41,-179.24\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"330.72,-182.5 338.1,-174.9 327.59,-176.23 330.72,-182.5\"/>\n",
       "</g>\n",
       "<!-- 17 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>17</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-352.95\" font-family=\"Times,serif\" font-size=\"14.00\">2000</text>\n",
       "</g>\n",
       "<!-- 16&#45;&gt;17 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>16&#45;&gt;17</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M403.31,-359.12C418.94,-359.12 437.46,-359.12 451.85,-359.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"451.74,-362.63 461.74,-359.13 451.74,-355.63 451.74,-362.63\"/>\n",
       "</g>\n",
       "<!-- 20 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>20</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">car</text>\n",
       "</g>\n",
       "<!-- 19&#45;&gt;20 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>19&#45;&gt;20</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M403.59,-295.12C421.34,-295.12 442.73,-295.12 457.57,-295.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"457.16,-298.63 467.16,-295.13 457.16,-291.63 457.16,-298.63\"/>\n",
       "</g>\n",
       "<!-- 23 -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>23</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">Mercury</text>\n",
       "</g>\n",
       "<!-- 22&#45;&gt;23 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>22&#45;&gt;23</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M416.87,-231.12C425.24,-231.12 433.84,-231.12 441.8,-231.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"441.74,-234.63 451.74,-231.13 441.74,-227.63 441.74,-234.63\"/>\n",
       "</g>\n",
       "<!-- 26 -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>26</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-160.95\" font-family=\"Times,serif\" font-size=\"14.00\">Cougar</text>\n",
       "</g>\n",
       "<!-- 25&#45;&gt;26 -->\n",
       "<g id=\"edge26\" class=\"edge\">\n",
       "<title>25&#45;&gt;26</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.09,-167.12C420.98,-167.12 433.97,-167.12 445.29,-167.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"445.27,-170.63 455.27,-167.13 445.27,-163.63 445.27,-170.63\"/>\n",
       "</g>\n",
       "<!-- 29 -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>29</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-128.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;year@30&gt;</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;29 -->\n",
       "<g id=\"edge29\" class=\"edge\">\n",
       "<title>28&#45;&gt;29</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M264,-94.15C279.27,-100.83 303.06,-111.04 324,-119.12 328.48,-120.86 333.24,-122.62 337.93,-124.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"336.54,-127.53 347.14,-127.56 338.87,-120.93 336.54,-127.53\"/>\n",
       "</g>\n",
       "<!-- 31 -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>31</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-96.95\" font-family=\"Times,serif\" font-size=\"14.00\">,car,</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;31 -->\n",
       "<g id=\"edge31\" class=\"edge\">\n",
       "<title>28&#45;&gt;31</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M287.87,-92.36C307.81,-94.99 331.2,-98.08 347.81,-100.27\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"347.16,-103.71 357.53,-101.55 348.07,-96.77 347.16,-103.71\"/>\n",
       "</g>\n",
       "<!-- 32 -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>32</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;company@30&gt;</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;32 -->\n",
       "<g id=\"edge32\" class=\"edge\">\n",
       "<title>28&#45;&gt;32</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M287.87,-81.89C295.85,-80.83 304.39,-79.71 312.82,-78.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"313.24,-82.07 322.7,-77.29 312.32,-75.13 313.24,-82.07\"/>\n",
       "</g>\n",
       "<!-- 34 -->\n",
       "<g id=\"node35\" class=\"node\">\n",
       "<title>34</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-32.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;34 -->\n",
       "<g id=\"edge34\" class=\"edge\">\n",
       "<title>28&#45;&gt;34</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M264,-80.1C279.27,-73.42 303.06,-63.21 324,-55.12 330.81,-52.5 338.25,-49.8 345.11,-47.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"346.14,-50.74 354.45,-44.17 343.86,-44.12 346.14,-50.74\"/>\n",
       "</g>\n",
       "<!-- 35 -->\n",
       "<g id=\"node36\" class=\"node\">\n",
       "<title>35</title>\n",
       "<text text-anchor=\"middle\" x=\"370.5\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;model@30&gt;</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;35 -->\n",
       "<g id=\"edge35\" class=\"edge\">\n",
       "<title>28&#45;&gt;35</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M254.59,-80.2C266.96,-67.34 295.28,-39.55 324,-23.12 327.03,-21.39 330.28,-19.8 333.59,-18.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"334.78,-21.64 342.82,-14.74 332.23,-15.13 334.78,-21.64\"/>\n",
       "</g>\n",
       "<!-- 30 -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>30</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-128.95\" font-family=\"Times,serif\" font-size=\"14.00\">1999</text>\n",
       "</g>\n",
       "<!-- 29&#45;&gt;30 -->\n",
       "<g id=\"edge30\" class=\"edge\">\n",
       "<title>29&#45;&gt;30</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M403.31,-135.12C418.94,-135.12 437.46,-135.12 451.85,-135.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"451.74,-138.63 461.74,-135.13 451.74,-131.63 451.74,-138.63\"/>\n",
       "</g>\n",
       "<!-- 33 -->\n",
       "<g id=\"node34\" class=\"node\">\n",
       "<title>33</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">Chevy</text>\n",
       "</g>\n",
       "<!-- 32&#45;&gt;33 -->\n",
       "<g id=\"edge33\" class=\"edge\">\n",
       "<title>32&#45;&gt;33</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M416.87,-71.12C427.5,-71.12 438.52,-71.12 448.1,-71.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"447.81,-74.63 457.81,-71.13 447.81,-67.63 447.81,-74.63\"/>\n",
       "</g>\n",
       "<!-- 36 -->\n",
       "<g id=\"node37\" class=\"node\">\n",
       "<title>36</title>\n",
       "<text text-anchor=\"middle\" x=\"476.62\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Venture</text>\n",
       "</g>\n",
       "<!-- 35&#45;&gt;36 -->\n",
       "<g id=\"edge36\" class=\"edge\">\n",
       "<title>35&#45;&gt;36</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M409.09,-7.12C420.64,-7.12 433.22,-7.12 444.31,-7.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"444.11,-10.63 454.11,-7.13 444.11,-3.63 444.11,-10.63\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x10770b340>"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with Tracer(INVENTORY) as tracer:\n",
    "    process_inventory(tracer.my_input)\n",
    "sm = AssignmentTracker(tracer.my_input, tracer.trace)\n",
    "dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars())\n",
    "display_tree(dt.tree, graph_attr=lr_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "As can be seen, the derivation tree obtained is not quite what we expected. The issue is easily seen if we enable logging in the `TreeMiner`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.870958Z",
     "iopub.status.busy": "2024-01-18T17:20:00.870822Z",
     "iopub.status.idle": "2024-01-18T17:20:00.878668Z",
     "shell.execute_reply": "2024-01-18T17:20:00.878303Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " inventory@22='1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'\n",
      "\t - Node: <start>\t\t? (<inventory@22>:'1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'\n",
      "\t\t  > ['<inventory@22>']\n",
      " vehicle@24='1997,van,Ford,E350'\n",
      "\t - Node: <start>\t\t? (<vehicle@24>:'1997,van,Ford,E350')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<vehicle@24>:'1997,van,Ford,E350')\n",
      "\t\t -> [0] '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'\n",
      "\t\t  > ['<vehicle@24>', '\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture']\n",
      " year@30='1997'\n",
      "\t - Node: <start>\t\t? (<year@30>:'1997')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<year@30>:'1997')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'1997')\n",
      "\t\t -> [0] '1997,van,Ford,E350'\n",
      "\t\t  > ['<year@30>', ',van,Ford,E350']\n",
      " kind@30='van'\n",
      "\t - Node: <start>\t\t? (<kind@30>:'van')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<kind@30>:'van')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<kind@30>:'van')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<kind@30>:'van')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ',van,Ford,E350'\n",
      "\t\t  > [',', '<kind@30>', ',Ford,E350']\n",
      " company@30='Ford'\n",
      "\t - Node: <start>\t\t? (<company@30>:'Ford')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<company@30>:'Ford')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Ford')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Ford')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<company@30>:'Ford')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ',Ford,E350'\n",
      "\t\t  > [',', '<company@30>', ',E350']\n",
      " model@30='E350'\n",
      "\t - Node: <start>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'E350')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ',E350'\n",
      "\t\t  > [',', '<model@30>']\n",
      " vehicle@24='2000,car,Mercury,Cougar'\n",
      "\t - Node: <start>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<vehicle@24>:'2000,car,Mercury,Cougar')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'\n",
      "\t\t  > ['\\n', '<vehicle@24>', '\\n1999,car,Chevy,Venture']\n",
      " year@30='2000'\n",
      "\t - Node: <start>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'2000')\n",
      "\t\t -> [0] '2000,car,Mercury,Cougar'\n",
      "\t\t  > ['<year@30>', ',car,Mercury,Cougar']\n",
      " kind@30='car'\n",
      "\t - Node: <start>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<kind@30>:'car')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ',car,Mercury,Cougar'\n",
      "\t\t  > [',', '<kind@30>', ',Mercury,Cougar']\n",
      " company@30='Mercury'\n",
      "\t - Node: <start>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<company@30>:'Mercury')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ',Mercury,Cougar'\n",
      "\t\t  > [',', '<company@30>', ',Cougar']\n",
      " model@30='Cougar'\n",
      "\t - Node: <start>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'Cougar')\n",
      "\t\t -> [0] 'Mercury'\n",
      "\t\t -> [5] ',Cougar'\n",
      "\t\t  > [',', '<model@30>']\n",
      " vehicle@24='1999,car,Chevy,Venture'\n",
      "\t - Node: <start>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'Mercury'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<vehicle@24>:'1999,car,Chevy,Venture')\n",
      "\t\t -> [0] 'Cougar'\n",
      "\t\t -> [3] '\\n1999,car,Chevy,Venture'\n",
      "\t\t  > ['\\n', '<vehicle@24>']\n",
      " year@30='1999'\n",
      "\t - Node: <start>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'Mercury'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] 'Cougar'\n",
      "\t\t -> [3] '\\n'\n",
      "\t\t -> [4] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<year@30>:'1999')\n",
      "\t\t -> [0] '1999,car,Chevy,Venture'\n",
      "\t\t  > ['<year@30>', ',car,Chevy,Venture']\n",
      " company@30='Chevy'\n",
      "\t - Node: <start>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'Mercury'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] 'Cougar'\n",
      "\t\t -> [3] '\\n'\n",
      "\t\t -> [4] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<company@30>:'Chevy')\n",
      "\t\t -> [0] '1999'\n",
      "\t\t -> [1] ',car,Chevy,Venture'\n",
      "\t\t  > [',car,', '<company@30>', ',Venture']\n",
      " model@30='Venture'\n",
      "\t - Node: <start>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '<inventory@22>'\n",
      "\t - Node: <inventory@22>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '1997'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'van'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'Ford'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'E350'\n",
      "\t\t -> [1] '\\n'\n",
      "\t\t -> [2] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '2000'\n",
      "\t\t -> [1] ','\n",
      "\t\t -> [2] '<kind@30>'\n",
      "\t - Node: <kind@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'car'\n",
      "\t\t -> [3] ','\n",
      "\t\t -> [4] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'Mercury'\n",
      "\t\t -> [5] ','\n",
      "\t\t -> [6] '<model@30>'\n",
      "\t - Node: <model@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'Cougar'\n",
      "\t\t -> [3] '\\n'\n",
      "\t\t -> [4] '<vehicle@24>'\n",
      "\t - Node: <vehicle@24>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '<year@30>'\n",
      "\t - Node: <year@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] '1999'\n",
      "\t\t -> [1] ',car,'\n",
      "\t\t -> [2] '<company@30>'\n",
      "\t - Node: <company@30>\t\t? (<model@30>:'Venture')\n",
      "\t\t -> [0] 'Chevy'\n",
      "\t\t -> [3] ',Venture'\n",
      "\t\t  > [',', '<model@30>']\n"
     ]
    }
   ],
   "source": [
    "dt = TreeMiner(tracer.my_input, sm.my_assignments.defined_vars(), log=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Look at the last statement. We have a value `1999,car,` where only the `year` got replaced. We no longer have a `'car'` variable to continue the replacement here. This happens because the `'car'` value in `'1999,car,Chevy,Venture'` is not treated as a new value because the value `'car'` had occurred for `'vehicle'` variable in the exact same location for a *different* method call (for `'2000,car,Mercury,Cougar'`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A Grammar Miner with Scope\n",
    "\n",
    "We need to incorporate inspection of the variables in the current context. We already have a stack of method calls so that we can obtain the current method at any point. We need to do the same for variables.\n",
    "\n",
    "For that, we extend the `CallStack` to a new class `InputStack` which holds the method invoked as well as the parameters observed. It is essentially the record of activation of the method. We start with the original input at the base of the stack, and for each new method-call, we push the parameters of that call into the stack as a new record."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Input Stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.880974Z",
     "iopub.status.busy": "2024-01-18T17:20:00.880757Z",
     "iopub.status.idle": "2024-01-18T17:20:00.883823Z",
     "shell.execute_reply": "2024-01-18T17:20:00.883368Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(CallStack):\n",
    "    def __init__(self, i, fragment_len=FRAGMENT_LEN):\n",
    "        self.inputs = [{START_SYMBOL: i}]\n",
    "        self.fragment_len = fragment_len\n",
    "        super().__init__()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In order to check if a particular variable be saved, we define `in_current_record()` which checks only the variables in the current scope for inclusion (rather than the original input string)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.885942Z",
     "iopub.status.busy": "2024-01-18T17:20:00.885837Z",
     "iopub.status.idle": "2024-01-18T17:20:00.887886Z",
     "shell.execute_reply": "2024-01-18T17:20:00.887595Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(InputStack):\n",
    "    def in_current_record(self, val):\n",
    "        return any(val in var for var in self.inputs[-1].values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.889479Z",
     "iopub.status.busy": "2024-01-18T17:20:00.889393Z",
     "iopub.status.idle": "2024-01-18T17:20:00.890942Z",
     "shell.execute_reply": "2024-01-18T17:20:00.890668Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_istack = InputStack('hello my world')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.892406Z",
     "iopub.status.busy": "2024-01-18T17:20:00.892317Z",
     "iopub.status.idle": "2024-01-18T17:20:00.894653Z",
     "shell.execute_reply": "2024-01-18T17:20:00.894320Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.in_current_record('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.896485Z",
     "iopub.status.busy": "2024-01-18T17:20:00.896384Z",
     "iopub.status.idle": "2024-01-18T17:20:00.899133Z",
     "shell.execute_reply": "2024-01-18T17:20:00.898767Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.in_current_record('bye')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.901357Z",
     "iopub.status.busy": "2024-01-18T17:20:00.901064Z",
     "iopub.status.idle": "2024-01-18T17:20:00.903188Z",
     "shell.execute_reply": "2024-01-18T17:20:00.902934Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_istack.inputs.append({'greeting': 'hello', 'location': 'world'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.904994Z",
     "iopub.status.busy": "2024-01-18T17:20:00.904722Z",
     "iopub.status.idle": "2024-01-18T17:20:00.906871Z",
     "shell.execute_reply": "2024-01-18T17:20:00.906547Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.in_current_record('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.908489Z",
     "iopub.status.busy": "2024-01-18T17:20:00.908315Z",
     "iopub.status.idle": "2024-01-18T17:20:00.910685Z",
     "shell.execute_reply": "2024-01-18T17:20:00.910412Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.in_current_record('my')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We define the method `ignored()` that returns true if either the variable is not a string, or the variable length is less than the defined `fragment_len`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.912996Z",
     "iopub.status.busy": "2024-01-18T17:20:00.912629Z",
     "iopub.status.idle": "2024-01-18T17:20:00.914964Z",
     "shell.execute_reply": "2024-01-18T17:20:00.914596Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(InputStack):\n",
    "    def ignored(self, val):\n",
    "        return not (isinstance(val, str) and len(val) >= self.fragment_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.917280Z",
     "iopub.status.busy": "2024-01-18T17:20:00.917041Z",
     "iopub.status.idle": "2024-01-18T17:20:00.919591Z",
     "shell.execute_reply": "2024-01-18T17:20:00.919314Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack = InputStack('hello world')\n",
    "my_istack.ignored(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.921204Z",
     "iopub.status.busy": "2024-01-18T17:20:00.921062Z",
     "iopub.status.idle": "2024-01-18T17:20:00.923206Z",
     "shell.execute_reply": "2024-01-18T17:20:00.922952Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.ignored('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.924714Z",
     "iopub.status.busy": "2024-01-18T17:20:00.924584Z",
     "iopub.status.idle": "2024-01-18T17:20:00.926738Z",
     "shell.execute_reply": "2024-01-18T17:20:00.926475Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_istack.ignored('help')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can now define the `in_scope()` method that checks whether the variable needs to be ignored, and if it is not to be ignored, whether the variable value is present in the current scope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.928680Z",
     "iopub.status.busy": "2024-01-18T17:20:00.928495Z",
     "iopub.status.idle": "2024-01-18T17:20:00.930874Z",
     "shell.execute_reply": "2024-01-18T17:20:00.930508Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(InputStack):\n",
    "    def in_scope(self, k, val):\n",
    "        if self.ignored(val):\n",
    "            return False\n",
    "        return self.in_current_record(val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Finally, we update `enter()` that pushes relevant variables in the current context to the stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.933343Z",
     "iopub.status.busy": "2024-01-18T17:20:00.933080Z",
     "iopub.status.idle": "2024-01-18T17:20:00.935467Z",
     "shell.execute_reply": "2024-01-18T17:20:00.935174Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(InputStack):\n",
    "    def enter(self, method, inputs):\n",
    "        my_inputs = {k: v for k, v in inputs.items() if self.in_scope(k, v)}\n",
    "        self.inputs.append(my_inputs)\n",
    "        super().enter(method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When a method returns, we also need a corresponding `leave()` to pop out the inputs and unwind the stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.937314Z",
     "iopub.status.busy": "2024-01-18T17:20:00.937143Z",
     "iopub.status.idle": "2024-01-18T17:20:00.939071Z",
     "shell.execute_reply": "2024-01-18T17:20:00.938843Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class InputStack(InputStack):\n",
    "    def leave(self):\n",
    "        self.inputs.pop()\n",
    "        super().leave()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### ScopedVars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We need to update our `AssignmentVars` to include information about which scope the variable was defined in. We start by updating `method_init()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.940814Z",
     "iopub.status.busy": "2024-01-18T17:20:00.940581Z",
     "iopub.status.idle": "2024-01-18T17:20:00.942724Z",
     "shell.execute_reply": "2024-01-18T17:20:00.942430Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(AssignmentVars):\n",
    "    def method_init(self):\n",
    "        self.call_stack = self.create_call_stack(self.my_input)\n",
    "        self.event_locations = {self.call_stack.method_id: []}\n",
    "\n",
    "    def create_call_stack(self, i):\n",
    "        return InputStack(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Similarly, the `method_enter()` now initializes the `accessed_seq_var` for the current method call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.944383Z",
     "iopub.status.busy": "2024-01-18T17:20:00.944233Z",
     "iopub.status.idle": "2024-01-18T17:20:00.946889Z",
     "shell.execute_reply": "2024-01-18T17:20:00.946467Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def method_enter(self, cxt, my_vars):\n",
    "        self.current_event = 'call'\n",
    "        self.call_stack.enter(cxt.method, my_vars)\n",
    "        self.accessed_seq_var[self.call_stack.method_id] = {}\n",
    "        self.event_locations[self.call_stack.method_id] = []\n",
    "        self.register_event(cxt)\n",
    "        self.update(my_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `update()` method now saves the context in which the value is defined. In the case of a parameter to a function, the context should be the context in which the function was called. On the other hand, a value defined during a statement execution would have the current context.\n",
    "\n",
    "Further, we annotate on value rather than key because we do not want to duplicate variables when parameters are in context in the next line. They will have same value, but different context because they are present in a statement execution.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.949110Z",
     "iopub.status.busy": "2024-01-18T17:20:00.948877Z",
     "iopub.status.idle": "2024-01-18T17:20:00.951476Z",
     "shell.execute_reply": "2024-01-18T17:20:00.951202Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def update(self, v):\n",
    "        if self.current_event == 'call':\n",
    "            context = -2\n",
    "        elif self.current_event == 'line':\n",
    "            context = -1\n",
    "        else:\n",
    "            context = -1\n",
    "        for k, v in v.items():\n",
    "            self._set_kv(k, (v, self.call_stack.at(context)))\n",
    "        self.var_location_register(self.new_vars)\n",
    "        self.new_vars = set()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We also need to save the current method invocation to determine which variables are in scope. This information is now incorporated in the variable name as `accessed_seq_var[method_id][var]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.953003Z",
     "iopub.status.busy": "2024-01-18T17:20:00.952876Z",
     "iopub.status.idle": "2024-01-18T17:20:00.954679Z",
     "shell.execute_reply": "2024-01-18T17:20:00.954427Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def var_name(self, var):\n",
    "        return (var, self.call_stack.method_id,\n",
    "                self.accessed_seq_var[self.call_stack.method_id][var])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "As before, `var_access` simply initializes the corresponding counter, this time in the context of `method_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.956314Z",
     "iopub.status.busy": "2024-01-18T17:20:00.956177Z",
     "iopub.status.idle": "2024-01-18T17:20:00.958197Z",
     "shell.execute_reply": "2024-01-18T17:20:00.957965Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def var_access(self, var):\n",
    "        if var not in self.accessed_seq_var[self.call_stack.method_id]:\n",
    "            self.accessed_seq_var[self.call_stack.method_id][var] = 0\n",
    "        return self.var_name(var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "During a variable reassignment, we update the `accessed_seq_var` to reflect the new count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.959909Z",
     "iopub.status.busy": "2024-01-18T17:20:00.959772Z",
     "iopub.status.idle": "2024-01-18T17:20:00.961847Z",
     "shell.execute_reply": "2024-01-18T17:20:00.961570Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def var_assign(self, var):\n",
    "        self.accessed_seq_var[self.call_stack.method_id][var] += 1\n",
    "        self.new_vars.add(self.var_name(var))\n",
    "        return self.var_name(var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We now update `defined_vars()` to account for the new information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.964225Z",
     "iopub.status.busy": "2024-01-18T17:20:00.964055Z",
     "iopub.status.idle": "2024-01-18T17:20:00.966404Z",
     "shell.execute_reply": "2024-01-18T17:20:00.966160Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def defined_vars(self, formatted=True):\n",
    "        def fmt(k):\n",
    "            method, i = k[1]\n",
    "            v = (method, i, k[0], self.var_def_lines[k])\n",
    "            return \"%s[%d]:%s@%s\" % v if formatted else v\n",
    "\n",
    "        return [(fmt(k), v) for k, v in self.defs.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Updating `seq_vars()` to account for new information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.968105Z",
     "iopub.status.busy": "2024-01-18T17:20:00.967904Z",
     "iopub.status.idle": "2024-01-18T17:20:00.970325Z",
     "shell.execute_reply": "2024-01-18T17:20:00.970066Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedVars(ScopedVars):\n",
    "    def seq_vars(self, formatted=True):\n",
    "        def fmt(k):\n",
    "            method, i = k[1]\n",
    "            v = (method, i, k[0], self.var_def_lines[k], k[2])\n",
    "            return \"%s[%d]:%s@%s:%s\" % v if formatted else v\n",
    "\n",
    "        return {fmt(k): v for k, v in self.defs.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Scope Tracker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "With the `InputStack` and `Vars` defined, we can now define the `ScopeTracker`. The `ScopeTracker` only saves variables if the value is present in the current scope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.972042Z",
     "iopub.status.busy": "2024-01-18T17:20:00.971906Z",
     "iopub.status.idle": "2024-01-18T17:20:00.973935Z",
     "shell.execute_reply": "2024-01-18T17:20:00.973686Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTracker(AssignmentTracker):\n",
    "    def __init__(self, my_input, trace, **kwargs):\n",
    "        self.current_event = None\n",
    "        super().__init__(my_input, trace, **kwargs)\n",
    "\n",
    "    def create_assignments(self, *args):\n",
    "        return ScopedVars(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We define a wrapper for checking whether a variable is present in the scope."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.975421Z",
     "iopub.status.busy": "2024-01-18T17:20:00.975284Z",
     "iopub.status.idle": "2024-01-18T17:20:00.977249Z",
     "shell.execute_reply": "2024-01-18T17:20:00.976997Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTracker(ScopeTracker):\n",
    "    def is_input_fragment(self, var, value):\n",
    "        return self.my_assignments.call_stack.in_scope(var, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can use the `ScopeTracker` as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.979376Z",
     "iopub.status.busy": "2024-01-18T17:20:00.979209Z",
     "iopub.status.idle": "2024-01-18T17:20:00.989792Z",
     "shell.execute_reply": "2024-01-18T17:20:00.989457Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory[1]:inventory@22:1 = ('1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', ('<start>', 0))\n",
      "process_inventory[1]:inventory@22:2 = ('1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_inventory[1]:vehicle@24:1 = ('1997,van,Ford,E350', ('process_inventory', 1))\n",
      "process_vehicle[2]:vehicle@29:1 = ('1997,van,Ford,E350', ('process_inventory', 1))\n",
      "process_vehicle[2]:vehicle@29:2 = ('1997,van,Ford,E350', ('process_vehicle', 2))\n",
      "process_vehicle[2]:year@30:1 = ('1997', ('process_vehicle', 2))\n",
      "process_vehicle[2]:kind@30:1 = ('van', ('process_vehicle', 2))\n",
      "process_vehicle[2]:company@30:1 = ('Ford', ('process_vehicle', 2))\n",
      "process_vehicle[2]:model@30:1 = ('E350', ('process_vehicle', 2))\n",
      "process_van[3]:year@40:1 = ('1997', ('process_vehicle', 2))\n",
      "process_van[3]:company@40:1 = ('Ford', ('process_vehicle', 2))\n",
      "process_van[3]:model@40:1 = ('E350', ('process_vehicle', 2))\n",
      "process_van[3]:year@40:2 = ('1997', ('process_van', 3))\n",
      "process_van[3]:company@40:2 = ('Ford', ('process_van', 3))\n",
      "process_van[3]:model@40:2 = ('E350', ('process_van', 3))\n",
      "process_inventory[1]:vehicle@24:2 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))\n",
      "process_vehicle[4]:vehicle@29:1 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))\n",
      "process_vehicle[4]:vehicle@29:2 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))\n",
      "process_vehicle[4]:year@30:1 = ('2000', ('process_vehicle', 4))\n",
      "process_vehicle[4]:kind@30:1 = ('car', ('process_vehicle', 4))\n",
      "process_vehicle[4]:company@30:1 = ('Mercury', ('process_vehicle', 4))\n",
      "process_vehicle[4]:model@30:1 = ('Cougar', ('process_vehicle', 4))\n",
      "process_car[5]:year@49:1 = ('2000', ('process_vehicle', 4))\n",
      "process_car[5]:company@49:1 = ('Mercury', ('process_vehicle', 4))\n",
      "process_car[5]:model@49:1 = ('Cougar', ('process_vehicle', 4))\n",
      "process_car[5]:year@49:2 = ('2000', ('process_car', 5))\n",
      "process_car[5]:company@49:2 = ('Mercury', ('process_car', 5))\n",
      "process_car[5]:model@49:2 = ('Cougar', ('process_car', 5))\n",
      "process_inventory[1]:vehicle@24:3 = ('1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_vehicle[6]:vehicle@29:1 = ('1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_vehicle[6]:vehicle@29:2 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))\n",
      "process_vehicle[6]:year@30:1 = ('1999', ('process_vehicle', 6))\n",
      "process_vehicle[6]:kind@30:1 = ('car', ('process_vehicle', 6))\n",
      "process_vehicle[6]:company@30:1 = ('Chevy', ('process_vehicle', 6))\n",
      "process_vehicle[6]:model@30:1 = ('Venture', ('process_vehicle', 6))\n",
      "process_car[7]:year@49:1 = ('1999', ('process_vehicle', 6))\n",
      "process_car[7]:company@49:1 = ('Chevy', ('process_vehicle', 6))\n",
      "process_car[7]:model@49:1 = ('Venture', ('process_vehicle', 6))\n",
      "process_car[7]:year@49:2 = ('1999', ('process_car', 7))\n",
      "process_car[7]:company@49:2 = ('Chevy', ('process_car', 7))\n",
      "process_car[7]:model@49:2 = ('Venture', ('process_car', 7))\n"
     ]
    }
   ],
   "source": [
    "vehicle_traces = []\n",
    "with Tracer(INVENTORY) as tracer:\n",
    "    process_inventory(tracer.my_input)\n",
    "sm = ScopeTracker(tracer.my_input, tracer.trace)\n",
    "vehicle_traces.append((tracer.my_input, sm))\n",
    "for k, v in sm.my_assignments.seq_vars().items():\n",
    "    print(k, '=', repr(v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recovering a Derivation Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The main difference in `apply_new_definition()` is that we add a second condition that checks for scope. In particular, variables are only allowed to replace portions of string fragments that were in scope.\n",
    "The variable scope is indicated by `scope`. However, merely accounting for scope is not sufficient. For example, consider the fragment below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "def my_fn(stringval):\n",
    "    partA, partB = stringval.split('/')\n",
    "    return partA, partB\n",
    "\n",
    "svalue = ...\n",
    "v1, v2 = my_fn(svalue)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here, `v1` and `v2` get their values from a previous function call. Not from their current context. That is, we have to provide an exception for cases where an internal child method call may have generated a large fragment as we showed above. To account for that, we define `mseq()` that retrieves the method call sequence. In the above case, the `mseq()` of the internal child method call would be larger than the current `mseq()`. If so, we allow the replacement to proceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.992115Z",
     "iopub.status.busy": "2024-01-18T17:20:00.991908Z",
     "iopub.status.idle": "2024-01-18T17:20:00.993975Z",
     "shell.execute_reply": "2024-01-18T17:20:00.993658Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTreeMiner(TreeMiner):\n",
    "    def mseq(self, key):\n",
    "        method, seq, var, lno = key\n",
    "        return seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `nt_var()` method needs to take the tuple and generate a non-terminal symbol out of it. We skip the method sequence because it is not relevant for the grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.995841Z",
     "iopub.status.busy": "2024-01-18T17:20:00.995688Z",
     "iopub.status.idle": "2024-01-18T17:20:00.997589Z",
     "shell.execute_reply": "2024-01-18T17:20:00.997337Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTreeMiner(ScopeTreeMiner):\n",
    "    def nt_var(self, key):\n",
    "        method, seq, var, lno = key\n",
    "        return to_nonterminal(\"%s@%d:%s\" % (method, lno, var))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We now redefine the `apply_new_definition()` to account for context and scope. In particular, a variable is allowed to replace a part of a value only if the variable is in *scope* -- that is, it's scope (method sequence number of either its calling context in case it is a parameter or the current context in case it is a statement) is same as that of the value's method sequence number. An exception is made when the value's method sequence number is greater than the variable's method sequence number. In that case, the value may have come from an internal call. We allow the replacement to proceed in that case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:00.999201Z",
     "iopub.status.busy": "2024-01-18T17:20:00.999083Z",
     "iopub.status.idle": "2024-01-18T17:20:01.003405Z",
     "shell.execute_reply": "2024-01-18T17:20:01.003112Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTreeMiner(ScopeTreeMiner):\n",
    "    def partition(self, part, value):\n",
    "        return value.partition(part)\n",
    "    def partition_by_part(self, pair, value):\n",
    "        (nt_var, nt_seq), (v, v_scope) = pair\n",
    "        prefix_k_suffix = [\n",
    "                    (nt_var, [(v, [], nt_seq)]) if i == 1 else (e, [])\n",
    "                    for i, e in enumerate(self.partition(v, value))\n",
    "                    if e]\n",
    "        return prefix_k_suffix\n",
    "    \n",
    "    def insert_into_tree(self, my_tree, pair):\n",
    "        var, values, my_scope = my_tree\n",
    "        (nt_var, nt_seq), (v, v_scope) = pair\n",
    "        applied = False\n",
    "        for i, value_ in enumerate(values):\n",
    "            key, arr, scope = value_\n",
    "            self.log(2, \"-> [%d] %s\" % (i, repr(value_)))\n",
    "            if is_nonterminal(key):\n",
    "                applied = self.insert_into_tree(value_, pair)\n",
    "                if applied:\n",
    "                    break\n",
    "            else:\n",
    "                if v_scope != scope:\n",
    "                    if nt_seq > scope:\n",
    "                        continue\n",
    "                if not v or not self.string_part_of_value(v, key):\n",
    "                    continue\n",
    "                prefix_k_suffix = [(k, children, scope) for k, children\n",
    "                                   in self.partition_by_part(pair, key)]\n",
    "                del values[i]\n",
    "                for j, rep in enumerate(prefix_k_suffix):\n",
    "                    values.insert(j + i, rep)\n",
    "\n",
    "                applied = True\n",
    "                self.log(2, \" > %s\" % (repr([i[0] for i in prefix_k_suffix])))\n",
    "                break\n",
    "        return applied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `apply_new_definition()` is now modified to carry additional contextual information `mseq`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.005014Z",
     "iopub.status.busy": "2024-01-18T17:20:01.004925Z",
     "iopub.status.idle": "2024-01-18T17:20:01.006918Z",
     "shell.execute_reply": "2024-01-18T17:20:01.006701Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTreeMiner(ScopeTreeMiner):\n",
    "    def apply_new_definition(self, tree, var, value_):\n",
    "        nt_var = self.nt_var(var)\n",
    "        seq = self.mseq(var)\n",
    "        val, (smethod, mseq) = value_\n",
    "        return self.insert_into_tree(tree, ((nt_var, seq), (val, mseq)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We also modify `get_derivation_tree()` so that the initial node carries the context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.008460Z",
     "iopub.status.busy": "2024-01-18T17:20:01.008376Z",
     "iopub.status.idle": "2024-01-18T17:20:01.010865Z",
     "shell.execute_reply": "2024-01-18T17:20:01.010547Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopeTreeMiner(ScopeTreeMiner):\n",
    "    def get_derivation_tree(self):\n",
    "        tree = (START_SYMBOL, [(self.my_input, [], 0)], 0)\n",
    "        for var, value in self.my_assignments:\n",
    "            self.log(0, \"%s=%s\" % (var, repr(value)))\n",
    "            self.apply_new_definition(tree, var, value)\n",
    "        return tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 1: Recovering URL Parse Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We verify that our URL parse tree recovery still works as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.012798Z",
     "iopub.status.busy": "2024-01-18T17:20:01.012641Z",
     "iopub.status.idle": "2024-01-18T17:20:01.146944Z",
     "shell.execute_reply": "2024-01-18T17:20:01.146608Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('urlparse', 1, 'url', 372) = ('http://user:pass@www.google.com:80/?q=path#ref', ('<start>', 0))\n",
      "('urlparse', 1, 'url', 372) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('http://user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'url', 478) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'scheme', 478) = ('http', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//user:pass@www.google.com:80/?q=path#ref', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//user:pass@www.google.com:80/?q=path#ref', ('_splitnetloc', 5))\n",
      "('urlsplit', 3, 'url', 481) = ('/?q=path#ref', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'netloc', 481) = ('user:pass@www.google.com:80', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'url', 486) = ('/?q=path', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'fragment', 486) = ('ref', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'query', 488) = ('q=path', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('user:pass@www.google.com:80', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('user:pass@www.google.com:80', ('_checknetloc', 6))\n",
      "('urlparse', 1, 'scheme', 394) = ('http', ('urlparse', 1))\n",
      "('urlparse', 1, 'netloc', 394) = ('user:pass@www.google.com:80', ('urlparse', 1))\n",
      "('urlparse', 1, 'query', 394) = ('q=path', ('urlparse', 1))\n",
      "('urlparse', 1, 'fragment', 394) = ('ref', ('urlparse', 1))\n",
      "('urlparse', 1, 'url', 372) = ('https://www.cispa.saarland:80/', ('<start>', 0))\n",
      "('urlparse', 1, 'url', 372) = ('https://www.cispa.saarland:80/', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('https://www.cispa.saarland:80/', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('https://www.cispa.saarland:80/', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'url', 478) = ('//www.cispa.saarland:80/', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'scheme', 478) = ('https', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//www.cispa.saarland:80/', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//www.cispa.saarland:80/', ('_splitnetloc', 5))\n",
      "('urlsplit', 3, 'netloc', 481) = ('www.cispa.saarland:80', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('www.cispa.saarland:80', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('www.cispa.saarland:80', ('_checknetloc', 6))\n",
      "('urlparse', 1, 'scheme', 394) = ('https', ('urlparse', 1))\n",
      "('urlparse', 1, 'netloc', 394) = ('www.cispa.saarland:80', ('urlparse', 1))\n",
      "('urlparse', 1, 'url', 372) = ('http://www.fuzzingbook.org/#News', ('<start>', 0))\n",
      "('urlparse', 1, 'url', 372) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('http://www.fuzzingbook.org/#News', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('http://www.fuzzingbook.org/#News', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'url', 478) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'scheme', 478) = ('http', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//www.fuzzingbook.org/#News', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//www.fuzzingbook.org/#News', ('_splitnetloc', 5))\n",
      "('urlsplit', 3, 'url', 481) = ('/#News', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'netloc', 481) = ('www.fuzzingbook.org', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'fragment', 486) = ('News', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('www.fuzzingbook.org', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('www.fuzzingbook.org', ('_checknetloc', 6))\n",
      "('urlparse', 1, 'scheme', 394) = ('http', ('urlparse', 1))\n",
      "('urlparse', 1, 'netloc', 394) = ('www.fuzzingbook.org', ('urlparse', 1))\n",
      "('urlparse', 1, 'fragment', 394) = ('News', ('urlparse', 1))\n",
      "('urlparse', 1, 'url', 372) = ('ftp://freebsd.org/releases/5.8', ('<start>', 0))\n",
      "('urlparse', 1, 'url', 372) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('ftp://freebsd.org/releases/5.8', ('urlparse', 1))\n",
      "('urlsplit', 3, 'url', 437) = ('ftp://freebsd.org/releases/5.8', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'url', 478) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'scheme', 478) = ('ftp', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//freebsd.org/releases/5.8', ('urlsplit', 3))\n",
      "('_splitnetloc', 5, 'url', 411) = ('//freebsd.org/releases/5.8', ('_splitnetloc', 5))\n",
      "('urlsplit', 3, 'url', 481) = ('/releases/5.8', ('urlsplit', 3))\n",
      "('urlsplit', 3, 'netloc', 481) = ('freebsd.org', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('freebsd.org', ('urlsplit', 3))\n",
      "('_checknetloc', 6, 'netloc', 419) = ('freebsd.org', ('_checknetloc', 6))\n",
      "('urlparse', 1, 'url', 394) = ('/releases/5.8', ('urlparse', 1))\n",
      "('urlparse', 1, 'scheme', 394) = ('ftp', ('urlparse', 1))\n",
      "('urlparse', 1, 'netloc', 394) = ('freebsd.org', ('urlparse', 1))\n"
     ]
    }
   ],
   "source": [
    "url_dts = []\n",
    "for inputstr in URLS_X:\n",
    "    clear_cache()\n",
    "    with Tracer(inputstr, files=['urllib/parse.py']) as tracer:\n",
    "        urlparse(tracer.my_input)\n",
    "    sm = ScopeTracker(tracer.my_input, tracer.trace)\n",
    "    for k, v in sm.my_assignments.defined_vars(formatted=False):\n",
    "        print(k, '=', repr(v))\n",
    "    dt = ScopeTreeMiner(\n",
    "        tracer.my_input,\n",
    "        sm.my_assignments.defined_vars(\n",
    "            formatted=False))\n",
    "    display_tree(dt.tree, graph_attr=lr_graph)\n",
    "    url_dts.append(dt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example 2: Recovering Inventory Parse Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Next, we look at recovering the parse tree from `process_inventory()` which failed last time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.148970Z",
     "iopub.status.busy": "2024-01-18T17:20:01.148828Z",
     "iopub.status.idle": "2024-01-18T17:20:01.573620Z",
     "shell.execute_reply": "2024-01-18T17:20:01.573255Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory[1]:inventory@22 = ('1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', ('<start>', 0))\n",
      "process_inventory[1]:inventory@22 = ('1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_inventory[1]:vehicle@24 = ('1997,van,Ford,E350', ('process_inventory', 1))\n",
      "process_vehicle[2]:vehicle@29 = ('1997,van,Ford,E350', ('process_inventory', 1))\n",
      "process_vehicle[2]:vehicle@29 = ('1997,van,Ford,E350', ('process_vehicle', 2))\n",
      "process_vehicle[2]:year@30 = ('1997', ('process_vehicle', 2))\n",
      "process_vehicle[2]:kind@30 = ('van', ('process_vehicle', 2))\n",
      "process_vehicle[2]:company@30 = ('Ford', ('process_vehicle', 2))\n",
      "process_vehicle[2]:model@30 = ('E350', ('process_vehicle', 2))\n",
      "process_van[3]:year@40 = ('1997', ('process_vehicle', 2))\n",
      "process_van[3]:company@40 = ('Ford', ('process_vehicle', 2))\n",
      "process_van[3]:model@40 = ('E350', ('process_vehicle', 2))\n",
      "process_van[3]:year@40 = ('1997', ('process_van', 3))\n",
      "process_van[3]:company@40 = ('Ford', ('process_van', 3))\n",
      "process_van[3]:model@40 = ('E350', ('process_van', 3))\n",
      "process_inventory[1]:vehicle@24 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))\n",
      "process_vehicle[4]:vehicle@29 = ('2000,car,Mercury,Cougar', ('process_inventory', 1))\n",
      "process_vehicle[4]:vehicle@29 = ('2000,car,Mercury,Cougar', ('process_vehicle', 4))\n",
      "process_vehicle[4]:year@30 = ('2000', ('process_vehicle', 4))\n",
      "process_vehicle[4]:kind@30 = ('car', ('process_vehicle', 4))\n",
      "process_vehicle[4]:company@30 = ('Mercury', ('process_vehicle', 4))\n",
      "process_vehicle[4]:model@30 = ('Cougar', ('process_vehicle', 4))\n",
      "process_car[5]:year@49 = ('2000', ('process_vehicle', 4))\n",
      "process_car[5]:company@49 = ('Mercury', ('process_vehicle', 4))\n",
      "process_car[5]:model@49 = ('Cougar', ('process_vehicle', 4))\n",
      "process_car[5]:year@49 = ('2000', ('process_car', 5))\n",
      "process_car[5]:company@49 = ('Mercury', ('process_car', 5))\n",
      "process_car[5]:model@49 = ('Cougar', ('process_car', 5))\n",
      "process_inventory[1]:vehicle@24 = ('1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_vehicle[6]:vehicle@29 = ('1999,car,Chevy,Venture', ('process_inventory', 1))\n",
      "process_vehicle[6]:vehicle@29 = ('1999,car,Chevy,Venture', ('process_vehicle', 6))\n",
      "process_vehicle[6]:year@30 = ('1999', ('process_vehicle', 6))\n",
      "process_vehicle[6]:kind@30 = ('car', ('process_vehicle', 6))\n",
      "process_vehicle[6]:company@30 = ('Chevy', ('process_vehicle', 6))\n",
      "process_vehicle[6]:model@30 = ('Venture', ('process_vehicle', 6))\n",
      "process_car[7]:year@49 = ('1999', ('process_vehicle', 6))\n",
      "process_car[7]:company@49 = ('Chevy', ('process_vehicle', 6))\n",
      "process_car[7]:model@49 = ('Venture', ('process_vehicle', 6))\n",
      "process_car[7]:year@49 = ('1999', ('process_car', 7))\n",
      "process_car[7]:company@49 = ('Chevy', ('process_car', 7))\n",
      "process_car[7]:model@49 = ('Venture', ('process_car', 7))\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 9.0.0 (20230911.1827)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"1852pt\" height=\"662pt\"\n",
       " viewBox=\"0.00 0.00 1851.50 662.25\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 658.25)\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-658.25 1847.5,-658.25 1847.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<text text-anchor=\"middle\" x=\"19.88\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;start&gt;</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<text text-anchor=\"middle\" x=\"174.38\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_inventory@22:inventory&gt;</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M39.57,-327.12C46.61,-327.12 55.2,-327.12 64.6,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"64.46,-330.63 74.46,-327.13 64.46,-323.63 64.46,-330.63\"/>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<text text-anchor=\"middle\" x=\"407.62\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_inventory@22:inventory&gt;</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M272.78,-327.12C281.09,-327.12 289.54,-327.12 297.93,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"297.82,-330.63 307.82,-327.13 297.82,-323.63 297.82,-330.63\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<text text-anchor=\"middle\" x=\"634.5\" y=\"-432.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_inventory@24:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M422.69,-334.13C460.08,-352.75 562.26,-403.64 609.26,-427.05\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"607.57,-430.12 618.08,-431.45 610.69,-423.86 607.57,-430.12\"/>\n",
       "</g>\n",
       "<!-- 23 -->\n",
       "<g id=\"node24\" class=\"node\">\n",
       "<title>23</title>\n",
       "<text text-anchor=\"middle\" x=\"634.5\" y=\"-352.95\" font-family=\"Times,serif\" font-size=\"14.00\">\\n (10)</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;23 -->\n",
       "<g id=\"edge23\" class=\"edge\">\n",
       "<title>2&#45;&gt;23</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M458.53,-334.23C503.52,-340.63 568.1,-349.82 604.91,-355.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"604.37,-358.51 614.76,-356.46 605.35,-351.58 604.37,-358.51\"/>\n",
       "</g>\n",
       "<!-- 24 -->\n",
       "<g id=\"node25\" class=\"node\">\n",
       "<title>24</title>\n",
       "<text text-anchor=\"middle\" x=\"634.5\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_inventory@24:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;24 -->\n",
       "<g id=\"edge24\" class=\"edge\">\n",
       "<title>2&#45;&gt;24</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M505.95,-327.12C514.27,-327.12 522.71,-327.12 531.07,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"530.9,-330.63 540.9,-327.13 530.9,-323.63 530.9,-330.63\"/>\n",
       "</g>\n",
       "<!-- 44 -->\n",
       "<g id=\"node45\" class=\"node\">\n",
       "<title>44</title>\n",
       "<text text-anchor=\"middle\" x=\"634.5\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">\\n (10)</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;44 -->\n",
       "<g id=\"edge44\" class=\"edge\">\n",
       "<title>2&#45;&gt;44</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M458.53,-320.02C503.52,-313.62 568.1,-304.43 604.91,-299.19\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"605.35,-302.67 614.76,-297.79 604.37,-295.74 605.35,-302.67\"/>\n",
       "</g>\n",
       "<!-- 45 -->\n",
       "<g id=\"node46\" class=\"node\">\n",
       "<title>45</title>\n",
       "<text text-anchor=\"middle\" x=\"634.5\" y=\"-216.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_inventory@24:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;45 -->\n",
       "<g id=\"edge45\" class=\"edge\">\n",
       "<title>2&#45;&gt;45</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M423.73,-320.14C461.56,-302.64 560.74,-256.78 607.89,-234.97\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"609.18,-238.23 616.79,-230.85 606.25,-231.87 609.18,-238.23\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<text text-anchor=\"middle\" x=\"848.62\" y=\"-464.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 3&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>3&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M682.58,-446.23C714.16,-450.99 755.97,-457.3 789.61,-462.37\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"788.77,-465.79 799.18,-463.82 789.82,-458.86 788.77,-465.79\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<text text-anchor=\"middle\" x=\"1056.38\" y=\"-528.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M872.43,-478.22C908.75,-489.52 979.93,-511.66 1022,-524.74\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1020.75,-528.02 1031.34,-527.65 1022.83,-521.34 1020.75,-528.02\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-640.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:year&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1063.65,-542.09C1080.32,-560.11 1127.91,-608.46 1178.25,-631.12 1183.39,-633.44 1188.84,-635.42 1194.41,-637.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1193.2,-640.41 1203.76,-639.64 1195.02,-633.66 1193.2,-640.41\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-608.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>5&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1067.86,-542.14C1088.22,-555.42 1134.91,-584.23 1178.25,-599.12 1199.69,-606.49 1225.2,-610.61 1243.66,-612.81\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.1,-616.27 1253.42,-613.87 1243.85,-609.31 1243.1,-616.27\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-576.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:kind&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>5&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1081.77,-542.23C1106.08,-549.15 1144.54,-559.7 1178.25,-567.12 1189.47,-569.6 1201.54,-571.94 1213.09,-574.04\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1212.47,-577.49 1222.92,-575.79 1213.69,-570.59 1212.47,-577.49\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-544.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>5&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1142.13,-541.54C1178.68,-544.31 1218.71,-547.34 1243.78,-549.23\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.36,-552.71 1253.6,-549.98 1243.89,-545.73 1243.36,-552.71\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:company&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>5&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1142.13,-528.71C1150.34,-528.09 1158.73,-527.45 1167.09,-526.82\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1167.22,-530.32 1176.93,-526.08 1166.69,-523.34 1167.22,-530.32\"/>\n",
       "</g>\n",
       "<!-- 18 -->\n",
       "<g id=\"node19\" class=\"node\">\n",
       "<title>18</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-480.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;18 -->\n",
       "<g id=\"edge18\" class=\"edge\">\n",
       "<title>5&#45;&gt;18</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1081.77,-528.02C1106.08,-521.1 1144.54,-510.55 1178.25,-503.12 1200.26,-498.28 1225.53,-493.91 1243.76,-490.96\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1244.05,-494.46 1253.38,-489.44 1242.96,-487.55 1244.05,-494.46\"/>\n",
       "</g>\n",
       "<!-- 19 -->\n",
       "<g id=\"node20\" class=\"node\">\n",
       "<title>19</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:model&gt;</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;19 -->\n",
       "<g id=\"edge19\" class=\"edge\">\n",
       "<title>5&#45;&gt;19</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1067.86,-528.11C1088.22,-514.83 1134.91,-486.02 1178.25,-471.12 1185.82,-468.52 1193.91,-466.33 1202.01,-464.48\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1202.47,-467.95 1211.54,-462.47 1201.03,-461.1 1202.47,-467.95\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-640.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:year&gt;</text>\n",
       "</g>\n",
       "<!-- 6&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>6&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.63,-647.12C1364.64,-647.12 1382.68,-647.12 1399.74,-647.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1399.28,-650.63 1409.28,-647.13 1399.28,-643.63 1399.28,-650.63\"/>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-640.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:year&gt;</text>\n",
       "</g>\n",
       "<!-- 7&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>7&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1547.14,-647.12C1563.93,-647.12 1582.08,-647.12 1599.35,-647.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1599.03,-650.63 1609.03,-647.13 1599.03,-643.63 1599.03,-650.63\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-640.95\" font-family=\"Times,serif\" font-size=\"14.00\">1997</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1746.74,-647.12C1764.04,-647.12 1781.54,-647.12 1794.98,-647.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1794.95,-650.63 1804.95,-647.13 1794.95,-643.63 1794.95,-650.63\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-576.95\" font-family=\"Times,serif\" font-size=\"14.00\">van</text>\n",
       "</g>\n",
       "<!-- 11&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>11&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.92,-583.12C1387.52,-583.12 1432.63,-583.12 1457.97,-583.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1457.77,-586.63 1467.77,-583.13 1457.77,-579.63 1457.77,-586.63\"/>\n",
       "</g>\n",
       "<!-- 15 -->\n",
       "<g id=\"node16\" class=\"node\">\n",
       "<title>15</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:company&gt;</text>\n",
       "</g>\n",
       "<!-- 14&#45;&gt;15 -->\n",
       "<g id=\"edge15\" class=\"edge\">\n",
       "<title>14&#45;&gt;15</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1361.05,-519.12C1369.37,-519.12 1377.82,-519.12 1386.15,-519.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1385.92,-522.63 1395.92,-519.13 1385.92,-515.63 1385.92,-522.63\"/>\n",
       "</g>\n",
       "<!-- 16 -->\n",
       "<g id=\"node17\" class=\"node\">\n",
       "<title>16</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:company&gt;</text>\n",
       "</g>\n",
       "<!-- 15&#45;&gt;16 -->\n",
       "<g id=\"edge16\" class=\"edge\">\n",
       "<title>15&#45;&gt;16</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1560.62,-519.12C1568.83,-519.12 1577.21,-519.12 1585.54,-519.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1585.31,-522.63 1595.31,-519.13 1585.31,-515.63 1585.31,-522.63\"/>\n",
       "</g>\n",
       "<!-- 17 -->\n",
       "<g id=\"node18\" class=\"node\">\n",
       "<title>17</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-512.95\" font-family=\"Times,serif\" font-size=\"14.00\">Ford</text>\n",
       "</g>\n",
       "<!-- 16&#45;&gt;17 -->\n",
       "<g id=\"edge17\" class=\"edge\">\n",
       "<title>16&#45;&gt;17</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1760.17,-519.12C1773.21,-519.12 1785.72,-519.12 1795.82,-519.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1795.66,-522.63 1805.66,-519.13 1795.66,-515.63 1795.66,-522.63\"/>\n",
       "</g>\n",
       "<!-- 20 -->\n",
       "<g id=\"node21\" class=\"node\">\n",
       "<title>20</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:model&gt;</text>\n",
       "</g>\n",
       "<!-- 19&#45;&gt;20 -->\n",
       "<g id=\"edge20\" class=\"edge\">\n",
       "<title>19&#45;&gt;20</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1353.26,-455.12C1366.66,-455.12 1380.56,-455.12 1394,-455.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1393.65,-458.63 1403.65,-455.13 1393.65,-451.63 1393.65,-458.63\"/>\n",
       "</g>\n",
       "<!-- 21 -->\n",
       "<g id=\"node22\" class=\"node\">\n",
       "<title>21</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_van@40:model&gt;</text>\n",
       "</g>\n",
       "<!-- 20&#45;&gt;21 -->\n",
       "<g id=\"edge21\" class=\"edge\">\n",
       "<title>20&#45;&gt;21</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1552.7,-455.12C1566.02,-455.12 1580.02,-455.12 1593.63,-455.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1593.44,-458.63 1603.44,-455.13 1593.44,-451.63 1593.44,-458.63\"/>\n",
       "</g>\n",
       "<!-- 22 -->\n",
       "<g id=\"node23\" class=\"node\">\n",
       "<title>22</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-448.95\" font-family=\"Times,serif\" font-size=\"14.00\">E350</text>\n",
       "</g>\n",
       "<!-- 21&#45;&gt;22 -->\n",
       "<g id=\"edge22\" class=\"edge\">\n",
       "<title>21&#45;&gt;22</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1752.05,-455.12C1767.36,-455.12 1782.49,-455.12 1794.48,-455.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1794.18,-458.63 1804.18,-455.13 1794.18,-451.63 1794.18,-458.63\"/>\n",
       "</g>\n",
       "<!-- 25 -->\n",
       "<g id=\"node26\" class=\"node\">\n",
       "<title>25</title>\n",
       "<text text-anchor=\"middle\" x=\"848.62\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 24&#45;&gt;25 -->\n",
       "<g id=\"edge25\" class=\"edge\">\n",
       "<title>24&#45;&gt;25</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M726.69,-327.12C734.95,-327.12 743.33,-327.12 751.63,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"751.37,-330.63 761.37,-327.13 751.37,-323.63 751.37,-330.63\"/>\n",
       "</g>\n",
       "<!-- 26 -->\n",
       "<g id=\"node27\" class=\"node\">\n",
       "<title>26</title>\n",
       "<text text-anchor=\"middle\" x=\"1056.38\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 25&#45;&gt;26 -->\n",
       "<g id=\"edge26\" class=\"edge\">\n",
       "<title>25&#45;&gt;26</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M934.5,-327.12C942.72,-327.12 951.1,-327.12 959.41,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"959.18,-330.63 969.18,-327.13 959.18,-323.63 959.18,-330.63\"/>\n",
       "</g>\n",
       "<!-- 27 -->\n",
       "<g id=\"node28\" class=\"node\">\n",
       "<title>27</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-416.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:year&gt;</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;27 -->\n",
       "<g id=\"edge27\" class=\"edge\">\n",
       "<title>26&#45;&gt;27</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1065.21,-334.02C1083.55,-349.75 1131.27,-388.37 1178.25,-407.12 1184.33,-409.55 1190.78,-411.61 1197.33,-413.36\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1196.38,-416.73 1206.92,-415.66 1198.02,-409.92 1196.38,-416.73\"/>\n",
       "</g>\n",
       "<!-- 31 -->\n",
       "<g id=\"node32\" class=\"node\">\n",
       "<title>31</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-384.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;31 -->\n",
       "<g id=\"edge31\" class=\"edge\">\n",
       "<title>26&#45;&gt;31</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1072.3,-334.2C1094.79,-344.59 1138.93,-363.97 1178.25,-375.12 1199.93,-381.28 1225.23,-385.51 1243.57,-388.07\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1242.88,-391.51 1253.25,-389.35 1243.8,-384.57 1242.88,-391.51\"/>\n",
       "</g>\n",
       "<!-- 32 -->\n",
       "<g id=\"node33\" class=\"node\">\n",
       "<title>32</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-352.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:kind&gt;</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;32 -->\n",
       "<g id=\"edge32\" class=\"edge\">\n",
       "<title>26&#45;&gt;32</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1104.29,-334.23C1135.76,-338.99 1177.42,-345.3 1210.94,-350.37\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1210.06,-353.78 1220.47,-351.82 1211.11,-346.86 1210.06,-353.78\"/>\n",
       "</g>\n",
       "<!-- 34 -->\n",
       "<g id=\"node35\" class=\"node\">\n",
       "<title>34</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-320.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;34 -->\n",
       "<g id=\"edge34\" class=\"edge\">\n",
       "<title>26&#45;&gt;34</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1142.13,-327.12C1178.68,-327.12 1218.71,-327.12 1243.78,-327.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.6,-330.63 1253.6,-327.13 1243.6,-323.63 1243.6,-330.63\"/>\n",
       "</g>\n",
       "<!-- 35 -->\n",
       "<g id=\"node36\" class=\"node\">\n",
       "<title>35</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:company&gt;</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;35 -->\n",
       "<g id=\"edge35\" class=\"edge\">\n",
       "<title>26&#45;&gt;35</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1104.29,-320.02C1135.76,-315.26 1177.42,-308.95 1210.94,-303.88\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1211.11,-307.39 1220.47,-302.43 1210.06,-300.47 1211.11,-307.39\"/>\n",
       "</g>\n",
       "<!-- 39 -->\n",
       "<g id=\"node40\" class=\"node\">\n",
       "<title>39</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-256.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;39 -->\n",
       "<g id=\"edge39\" class=\"edge\">\n",
       "<title>26&#45;&gt;39</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1072.3,-320.05C1094.79,-309.66 1138.93,-290.28 1178.25,-279.12 1199.93,-272.97 1225.23,-268.74 1243.57,-266.18\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.8,-269.68 1253.25,-264.9 1242.88,-262.74 1243.8,-269.68\"/>\n",
       "</g>\n",
       "<!-- 40 -->\n",
       "<g id=\"node41\" class=\"node\">\n",
       "<title>40</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:model&gt;</text>\n",
       "</g>\n",
       "<!-- 26&#45;&gt;40 -->\n",
       "<g id=\"edge40\" class=\"edge\">\n",
       "<title>26&#45;&gt;40</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1065.21,-320.23C1083.55,-304.5 1131.27,-265.88 1178.25,-247.12 1184.33,-244.7 1190.78,-242.64 1197.33,-240.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1198.02,-244.33 1206.92,-238.59 1196.38,-237.52 1198.02,-244.33\"/>\n",
       "</g>\n",
       "<!-- 28 -->\n",
       "<g id=\"node29\" class=\"node\">\n",
       "<title>28</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-416.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:year&gt;</text>\n",
       "</g>\n",
       "<!-- 27&#45;&gt;28 -->\n",
       "<g id=\"edge28\" class=\"edge\">\n",
       "<title>27&#45;&gt;28</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.63,-423.12C1365.11,-423.12 1383.68,-423.12 1401.16,-423.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1400.96,-426.63 1410.96,-423.13 1400.96,-419.63 1400.96,-426.63\"/>\n",
       "</g>\n",
       "<!-- 29 -->\n",
       "<g id=\"node30\" class=\"node\">\n",
       "<title>29</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-416.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:year&gt;</text>\n",
       "</g>\n",
       "<!-- 28&#45;&gt;29 -->\n",
       "<g id=\"edge29\" class=\"edge\">\n",
       "<title>28&#45;&gt;29</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1545.49,-423.12C1563.07,-423.12 1582.25,-423.12 1600.4,-423.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1600.34,-426.63 1610.34,-423.13 1600.34,-419.63 1600.34,-426.63\"/>\n",
       "</g>\n",
       "<!-- 30 -->\n",
       "<g id=\"node31\" class=\"node\">\n",
       "<title>30</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-416.95\" font-family=\"Times,serif\" font-size=\"14.00\">2000</text>\n",
       "</g>\n",
       "<!-- 29&#45;&gt;30 -->\n",
       "<g id=\"edge30\" class=\"edge\">\n",
       "<title>29&#45;&gt;30</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1745.11,-423.12C1763.05,-423.12 1781.36,-423.12 1795.25,-423.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1795.11,-426.63 1805.11,-423.13 1795.11,-419.63 1795.11,-426.63\"/>\n",
       "</g>\n",
       "<!-- 33 -->\n",
       "<g id=\"node34\" class=\"node\">\n",
       "<title>33</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-352.95\" font-family=\"Times,serif\" font-size=\"14.00\">car</text>\n",
       "</g>\n",
       "<!-- 32&#45;&gt;33 -->\n",
       "<g id=\"edge33\" class=\"edge\">\n",
       "<title>32&#45;&gt;33</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.92,-359.12C1388.31,-359.12 1434.44,-359.12 1459.47,-359.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1459.25,-362.63 1469.25,-359.13 1459.25,-355.63 1459.25,-362.63\"/>\n",
       "</g>\n",
       "<!-- 36 -->\n",
       "<g id=\"node37\" class=\"node\">\n",
       "<title>36</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:company&gt;</text>\n",
       "</g>\n",
       "<!-- 35&#45;&gt;36 -->\n",
       "<g id=\"edge36\" class=\"edge\">\n",
       "<title>35&#45;&gt;36</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1361.05,-295.12C1369.85,-295.12 1378.78,-295.12 1387.58,-295.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1387.53,-298.63 1397.53,-295.13 1387.53,-291.63 1387.53,-298.63\"/>\n",
       "</g>\n",
       "<!-- 37 -->\n",
       "<g id=\"node38\" class=\"node\">\n",
       "<title>37</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:company&gt;</text>\n",
       "</g>\n",
       "<!-- 36&#45;&gt;37 -->\n",
       "<g id=\"edge37\" class=\"edge\">\n",
       "<title>36&#45;&gt;37</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1559.19,-295.12C1568.35,-295.12 1577.74,-295.12 1587.03,-295.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1586.78,-298.63 1596.78,-295.13 1586.78,-291.63 1586.78,-298.63\"/>\n",
       "</g>\n",
       "<!-- 38 -->\n",
       "<g id=\"node39\" class=\"node\">\n",
       "<title>38</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-288.95\" font-family=\"Times,serif\" font-size=\"14.00\">Mercury</text>\n",
       "</g>\n",
       "<!-- 37&#45;&gt;38 -->\n",
       "<g id=\"edge38\" class=\"edge\">\n",
       "<title>37&#45;&gt;38</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1758.55,-295.12C1767.88,-295.12 1776.98,-295.12 1785.17,-295.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1784.92,-298.63 1794.92,-295.13 1784.92,-291.63 1784.92,-298.63\"/>\n",
       "</g>\n",
       "<!-- 41 -->\n",
       "<g id=\"node42\" class=\"node\">\n",
       "<title>41</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:model&gt;</text>\n",
       "</g>\n",
       "<!-- 40&#45;&gt;41 -->\n",
       "<g id=\"edge41\" class=\"edge\">\n",
       "<title>40&#45;&gt;41</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1353.26,-231.12C1367.1,-231.12 1381.47,-231.12 1395.32,-231.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1395.3,-234.63 1405.3,-231.13 1395.3,-227.63 1395.3,-234.63\"/>\n",
       "</g>\n",
       "<!-- 42 -->\n",
       "<g id=\"node43\" class=\"node\">\n",
       "<title>42</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:model&gt;</text>\n",
       "</g>\n",
       "<!-- 41&#45;&gt;42 -->\n",
       "<g id=\"edge42\" class=\"edge\">\n",
       "<title>41&#45;&gt;42</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1551.3,-231.12C1565.43,-231.12 1580.36,-231.12 1594.83,-231.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1594.79,-234.63 1604.79,-231.13 1594.79,-227.63 1594.79,-234.63\"/>\n",
       "</g>\n",
       "<!-- 43 -->\n",
       "<g id=\"node44\" class=\"node\">\n",
       "<title>43</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-224.95\" font-family=\"Times,serif\" font-size=\"14.00\">Cougar</text>\n",
       "</g>\n",
       "<!-- 42&#45;&gt;43 -->\n",
       "<g id=\"edge43\" class=\"edge\">\n",
       "<title>42&#45;&gt;43</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1750.82,-231.12C1764.2,-231.12 1777.5,-231.12 1788.74,-231.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1788.53,-234.63 1798.53,-231.13 1788.53,-227.63 1788.53,-234.63\"/>\n",
       "</g>\n",
       "<!-- 46 -->\n",
       "<g id=\"node47\" class=\"node\">\n",
       "<title>46</title>\n",
       "<text text-anchor=\"middle\" x=\"848.62\" y=\"-136.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 45&#45;&gt;46 -->\n",
       "<g id=\"edge46\" class=\"edge\">\n",
       "<title>45&#45;&gt;46</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M654.32,-216.03C691.36,-202.06 773.91,-170.93 818.2,-154.22\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"819.3,-157.55 827.43,-150.74 816.83,-151 819.3,-157.55\"/>\n",
       "</g>\n",
       "<!-- 47 -->\n",
       "<g id=\"node48\" class=\"node\">\n",
       "<title>47</title>\n",
       "<text text-anchor=\"middle\" x=\"1056.38\" y=\"-112.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@29:vehicle&gt;</text>\n",
       "</g>\n",
       "<!-- 46&#45;&gt;47 -->\n",
       "<g id=\"edge47\" class=\"edge\">\n",
       "<title>46&#45;&gt;47</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M910.52,-136.02C933.33,-133.36 959.47,-130.31 983.2,-127.54\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"983.6,-131.02 993.13,-126.39 982.79,-124.07 983.6,-131.02\"/>\n",
       "</g>\n",
       "<!-- 48 -->\n",
       "<g id=\"node49\" class=\"node\">\n",
       "<title>48</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-192.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:year&gt;</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;48 -->\n",
       "<g id=\"edge48\" class=\"edge\">\n",
       "<title>47&#45;&gt;48</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1067.86,-126.14C1088.22,-139.42 1134.91,-168.23 1178.25,-183.12 1185.82,-185.73 1193.91,-187.92 1202.01,-189.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1201.03,-193.15 1211.54,-191.78 1202.47,-186.3 1201.03,-193.15\"/>\n",
       "</g>\n",
       "<!-- 52 -->\n",
       "<g id=\"node53\" class=\"node\">\n",
       "<title>52</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-160.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;52 -->\n",
       "<g id=\"edge52\" class=\"edge\">\n",
       "<title>47&#45;&gt;52</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1081.77,-126.23C1106.08,-133.15 1144.54,-143.7 1178.25,-151.12 1200.26,-155.97 1225.53,-160.34 1243.76,-163.29\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1242.96,-166.7 1253.38,-164.81 1244.05,-159.79 1242.96,-166.7\"/>\n",
       "</g>\n",
       "<!-- 53 -->\n",
       "<g id=\"node54\" class=\"node\">\n",
       "<title>53</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-128.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:kind&gt;</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;53 -->\n",
       "<g id=\"edge53\" class=\"edge\">\n",
       "<title>47&#45;&gt;53</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1142.13,-125.54C1154.62,-126.49 1167.51,-127.46 1180.08,-128.41\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1179.75,-131.9 1189.99,-129.16 1180.28,-124.92 1179.75,-131.9\"/>\n",
       "</g>\n",
       "<!-- 55 -->\n",
       "<g id=\"node56\" class=\"node\">\n",
       "<title>55</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-96.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;55 -->\n",
       "<g id=\"edge55\" class=\"edge\">\n",
       "<title>47&#45;&gt;55</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1142.13,-112.71C1178.68,-109.94 1218.71,-106.91 1243.78,-105.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.89,-108.52 1253.6,-104.27 1243.36,-101.54 1243.89,-108.52\"/>\n",
       "</g>\n",
       "<!-- 56 -->\n",
       "<g id=\"node57\" class=\"node\">\n",
       "<title>56</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:company&gt;</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;56 -->\n",
       "<g id=\"edge56\" class=\"edge\">\n",
       "<title>47&#45;&gt;56</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1081.77,-112.02C1106.08,-105.1 1144.54,-94.55 1178.25,-87.12 1189.47,-84.65 1201.54,-82.31 1213.09,-80.21\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1213.69,-83.66 1222.92,-78.46 1212.47,-76.76 1213.69,-83.66\"/>\n",
       "</g>\n",
       "<!-- 60 -->\n",
       "<g id=\"node61\" class=\"node\">\n",
       "<title>60</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-32.95\" font-family=\"Times,serif\" font-size=\"14.00\">, (44)</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;60 -->\n",
       "<g id=\"edge60\" class=\"edge\">\n",
       "<title>47&#45;&gt;60</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1067.86,-112.11C1088.22,-98.83 1134.91,-70.02 1178.25,-55.12 1199.69,-47.76 1225.2,-43.64 1243.66,-41.44\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1243.85,-44.94 1253.42,-40.38 1243.1,-37.98 1243.85,-44.94\"/>\n",
       "</g>\n",
       "<!-- 61 -->\n",
       "<g id=\"node62\" class=\"node\">\n",
       "<title>61</title>\n",
       "<text text-anchor=\"middle\" x=\"1269.75\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_vehicle@30:model&gt;</text>\n",
       "</g>\n",
       "<!-- 47&#45;&gt;61 -->\n",
       "<g id=\"edge61\" class=\"edge\">\n",
       "<title>47&#45;&gt;61</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1063.65,-112.16C1080.32,-94.14 1127.91,-45.79 1178.25,-23.12 1183.39,-20.81 1188.84,-18.83 1194.41,-17.13\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1195.02,-20.59 1203.76,-14.61 1193.2,-13.84 1195.02,-20.59\"/>\n",
       "</g>\n",
       "<!-- 49 -->\n",
       "<g id=\"node50\" class=\"node\">\n",
       "<title>49</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-192.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:year&gt;</text>\n",
       "</g>\n",
       "<!-- 48&#45;&gt;49 -->\n",
       "<g id=\"edge49\" class=\"edge\">\n",
       "<title>48&#45;&gt;49</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.63,-199.12C1365.11,-199.12 1383.68,-199.12 1401.16,-199.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1400.96,-202.63 1410.96,-199.13 1400.96,-195.63 1400.96,-202.63\"/>\n",
       "</g>\n",
       "<!-- 50 -->\n",
       "<g id=\"node51\" class=\"node\">\n",
       "<title>50</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-192.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:year&gt;</text>\n",
       "</g>\n",
       "<!-- 49&#45;&gt;50 -->\n",
       "<g id=\"edge50\" class=\"edge\">\n",
       "<title>49&#45;&gt;50</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1545.49,-199.12C1563.07,-199.12 1582.25,-199.12 1600.4,-199.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1600.34,-202.63 1610.34,-199.13 1600.34,-195.63 1600.34,-202.63\"/>\n",
       "</g>\n",
       "<!-- 51 -->\n",
       "<g id=\"node52\" class=\"node\">\n",
       "<title>51</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-192.95\" font-family=\"Times,serif\" font-size=\"14.00\">1999</text>\n",
       "</g>\n",
       "<!-- 50&#45;&gt;51 -->\n",
       "<g id=\"edge51\" class=\"edge\">\n",
       "<title>50&#45;&gt;51</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1745.11,-199.12C1763.05,-199.12 1781.36,-199.12 1795.25,-199.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1795.11,-202.63 1805.11,-199.13 1795.11,-195.63 1795.11,-202.63\"/>\n",
       "</g>\n",
       "<!-- 54 -->\n",
       "<g id=\"node55\" class=\"node\">\n",
       "<title>54</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-128.95\" font-family=\"Times,serif\" font-size=\"14.00\">car</text>\n",
       "</g>\n",
       "<!-- 53&#45;&gt;54 -->\n",
       "<g id=\"edge54\" class=\"edge\">\n",
       "<title>53&#45;&gt;54</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1347.92,-135.12C1388.31,-135.12 1434.44,-135.12 1459.47,-135.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1459.25,-138.63 1469.25,-135.13 1459.25,-131.63 1459.25,-138.63\"/>\n",
       "</g>\n",
       "<!-- 57 -->\n",
       "<g id=\"node58\" class=\"node\">\n",
       "<title>57</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:company&gt;</text>\n",
       "</g>\n",
       "<!-- 56&#45;&gt;57 -->\n",
       "<g id=\"edge57\" class=\"edge\">\n",
       "<title>56&#45;&gt;57</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1361.05,-71.12C1369.85,-71.12 1378.78,-71.12 1387.58,-71.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1387.53,-74.63 1397.53,-71.13 1387.53,-67.63 1387.53,-74.63\"/>\n",
       "</g>\n",
       "<!-- 58 -->\n",
       "<g id=\"node59\" class=\"node\">\n",
       "<title>58</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:company&gt;</text>\n",
       "</g>\n",
       "<!-- 57&#45;&gt;58 -->\n",
       "<g id=\"edge58\" class=\"edge\">\n",
       "<title>57&#45;&gt;58</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1559.19,-71.12C1568.35,-71.12 1577.74,-71.12 1587.03,-71.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1586.78,-74.63 1596.78,-71.13 1586.78,-67.63 1586.78,-74.63\"/>\n",
       "</g>\n",
       "<!-- 59 -->\n",
       "<g id=\"node60\" class=\"node\">\n",
       "<title>59</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-64.95\" font-family=\"Times,serif\" font-size=\"14.00\">Chevy</text>\n",
       "</g>\n",
       "<!-- 58&#45;&gt;59 -->\n",
       "<g id=\"edge59\" class=\"edge\">\n",
       "<title>58&#45;&gt;59</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1758.55,-71.12C1770.09,-71.12 1781.27,-71.12 1790.82,-71.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1790.79,-74.63 1800.79,-71.13 1790.79,-67.63 1790.79,-74.63\"/>\n",
       "</g>\n",
       "<!-- 62 -->\n",
       "<g id=\"node63\" class=\"node\">\n",
       "<title>62</title>\n",
       "<text text-anchor=\"middle\" x=\"1479\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:model&gt;</text>\n",
       "</g>\n",
       "<!-- 61&#45;&gt;62 -->\n",
       "<g id=\"edge62\" class=\"edge\">\n",
       "<title>61&#45;&gt;62</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1353.26,-7.12C1367.1,-7.12 1381.47,-7.12 1395.32,-7.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1395.3,-10.63 1405.3,-7.13 1395.3,-3.63 1395.3,-10.63\"/>\n",
       "</g>\n",
       "<!-- 63 -->\n",
       "<g id=\"node64\" class=\"node\">\n",
       "<title>63</title>\n",
       "<text text-anchor=\"middle\" x=\"1678.5\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">&lt;process_car@49:model&gt;</text>\n",
       "</g>\n",
       "<!-- 62&#45;&gt;63 -->\n",
       "<g id=\"edge63\" class=\"edge\">\n",
       "<title>62&#45;&gt;63</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1551.3,-7.12C1565.43,-7.12 1580.36,-7.12 1594.83,-7.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1594.79,-10.63 1604.79,-7.13 1594.79,-3.63 1594.79,-10.63\"/>\n",
       "</g>\n",
       "<!-- 64 -->\n",
       "<g id=\"node65\" class=\"node\">\n",
       "<title>64</title>\n",
       "<text text-anchor=\"middle\" x=\"1819.88\" y=\"-0.95\" font-family=\"Times,serif\" font-size=\"14.00\">Venture</text>\n",
       "</g>\n",
       "<!-- 63&#45;&gt;64 -->\n",
       "<g id=\"edge64\" class=\"edge\">\n",
       "<title>63&#45;&gt;64</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M1750.82,-7.12C1763.67,-7.12 1776.44,-7.12 1787.39,-7.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"1787.37,-10.63 1797.37,-7.13 1787.37,-3.63 1787.37,-10.63\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1167ca080>"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with Tracer(INVENTORY) as tracer:\n",
    "    process_inventory(tracer.my_input)\n",
    "\n",
    "sm = ScopeTracker(tracer.my_input, tracer.trace)\n",
    "for k, v in sm.my_assignments.defined_vars():\n",
    "    print(k, '=', repr(v))\n",
    "inventory_dt = ScopeTreeMiner(\n",
    "    tracer.my_input,\n",
    "    sm.my_assignments.defined_vars(\n",
    "        formatted=False))\n",
    "display_tree(inventory_dt.tree, graph_attr=lr_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The recovered parse tree seems reasonable.\n",
    "\n",
    "One of the things that one might notice from our Example (2) is that the three subtrees -- `vehicle[2:1]`, `vehicle[4:1]` and `vehicle[6:1]` are quite alike. We will examine how this can be exploited to generate a grammar directly, next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Grammar Mining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `tree_to_grammar()` is now redefined as follows, to account for the extra scope in nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.575522Z",
     "iopub.status.busy": "2024-01-18T17:20:01.575379Z",
     "iopub.status.idle": "2024-01-18T17:20:01.578572Z",
     "shell.execute_reply": "2024-01-18T17:20:01.578261Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedGrammarMiner(GrammarMiner):\n",
    "    def tree_to_grammar(self, tree):\n",
    "        key, children, scope = tree\n",
    "        one_alt = [ckey for ckey, gchildren, cscope in children if ckey != key]\n",
    "        hsh = {key: [one_alt] if one_alt else []}\n",
    "        for child in children:\n",
    "            (ckey, _gc, _cscope) = child\n",
    "            if not is_nonterminal(ckey):\n",
    "                continue\n",
    "            chsh = self.tree_to_grammar(child)\n",
    "            for k in chsh:\n",
    "                if k not in hsh:\n",
    "                    hsh[k] = chsh[k]\n",
    "                else:\n",
    "                    hsh[k].extend(chsh[k])\n",
    "        return hsh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar is in canonical form, which needs to be massaged to display. First, the recovered grammar for inventory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.580850Z",
     "iopub.status.busy": "2024-01-18T17:20:01.580697Z",
     "iopub.status.idle": "2024-01-18T17:20:01.599454Z",
     "shell.execute_reply": "2024-01-18T17:20:01.599143Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 395.0 62\" width=\"395.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M355.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><rect height=\"22\" width=\"275.0\" x=\"60.0\" y=\"20\"/><text x=\"197.5\" y=\"35\">process_inventory@22:inventory</text></g></g><path d=\"M335.0 31h20\"/></g><path d=\"M 355.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@22:inventory\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1031.0 62\" width=\"1031.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M991.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M971.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M318.0 31h0.0\"/><rect height=\"22\" width=\"258.0\" x=\"60.0\" y=\"20\"/><text x=\"189.0\" y=\"35\">process_inventory@24:vehicle</text></g><path d=\"M318.0 31h10\"/><path d=\"M328.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M338.0 31h0.0\"/><path d=\"M366.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"338.0\" y=\"20\"/><text x=\"352.25\" y=\"35\">\n",
       "</text></g><path d=\"M366.5 31h10\"/><path d=\"M376.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M386.5 31h0.0\"/><path d=\"M644.5 31h0.0\"/><rect height=\"22\" width=\"258.0\" x=\"386.5\" y=\"20\"/><text x=\"515.5\" y=\"35\">process_inventory@24:vehicle</text></g><path d=\"M644.5 31h10\"/><path d=\"M654.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M664.5 31h0.0\"/><path d=\"M693.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"664.5\" y=\"20\"/><text x=\"678.75\" y=\"35\">\n",
       "</text></g><path d=\"M693.0 31h10\"/><path d=\"M703.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M713.0 31h0.0\"/><path d=\"M971.0 31h0.0\"/><rect height=\"22\" width=\"258.0\" x=\"713.0\" y=\"20\"/><text x=\"842.0\" y=\"35\">process_inventory@24:vehicle</text></g></g><path d=\"M971.0 31h20\"/></g><path d=\"M 991.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@24:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 361.0 62\" width=\"361.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M321.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M301.0 31h20\"/></g><path d=\"M 321.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "si = ScopedGrammarMiner()\n",
    "si.add_tree(inventory_dt)\n",
    "syntax_diagram(readable(si.grammar))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The recovered grammar for URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.601255Z",
     "iopub.status.busy": "2024-01-18T17:20:01.601084Z",
     "iopub.status.idle": "2024-01-18T17:20:01.621381Z",
     "shell.execute_reply": "2024-01-18T17:20:01.620996Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 276.0 62\" width=\"276.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M236.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlparse@372:url</text></g></g><path d=\"M216.0 31h20\"/></g><path d=\"M 236.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@372:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 276.0 62\" width=\"276.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M236.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@437:url</text></g></g><path d=\"M216.0 31h20\"/></g><path d=\"M 236.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@437:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 526.0 62\" width=\"526.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M486.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M466.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">urlsplit@478:scheme</text></g><path d=\"M241.5 31h10\"/><path d=\"M251.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M261.5 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"261.5\" y=\"20\"/><text x=\"275.75\" y=\"35\">:</text></g><path d=\"M290.0 31h10\"/><path d=\"M300.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M310.0 31h0.0\"/><path d=\"M466.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"310.0\" y=\"20\"/><text x=\"388.0\" y=\"35\">urlsplit@478:url</text></g></g><path d=\"M466.0 31h20\"/></g><path d=\"M 486.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@478:scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 301.5 62\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M261.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">urlparse@394:scheme</text></g></g><path d=\"M241.5 31h20\"/></g><path d=\"M 261.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 182.5 122\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h8.5\"/><path d=\"M114.0 91h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"68.5\" y=\"80\"/><text x=\"91.25\" y=\"95\">ftp</text></g></g><path d=\"M122.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@478:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 310.0 62\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M270.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">_splitnetloc@411:url</text></g></g><path d=\"M250.0 31h20\"/></g><path d=\"M 270.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_splitnetloc@411:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 534.5 92\" width=\"534.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M494.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M474.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M97.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"20\"/><text x=\"78.5\" y=\"35\">//</text></g><path d=\"M97.0 31h10\"/><path d=\"M107.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 31h0.0\"/><path d=\"M298.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"117.0\" y=\"20\"/><text x=\"207.75\" y=\"35\">urlsplit@481:netloc</text></g><path d=\"M298.5 31h10\"/><path d=\"M308.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M318.5 31h0.0\"/><path d=\"M474.5 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"318.5\" y=\"20\"/><text x=\"396.5\" y=\"35\">urlsplit@481:url</text></g></g><path d=\"M474.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h63.75\"/><path d=\"M410.75 61h63.75\"/><g class=\"terminal\">\n",
       "<path d=\"M123.75 61h0.0\"/><path d=\"M160.75 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"123.75\" y=\"50\"/><text x=\"142.25\" y=\"65\">//</text></g><path d=\"M160.75 61h10\"/><path d=\"M170.75 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M180.75 61h0.0\"/><path d=\"M362.25 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"180.75\" y=\"50\"/><text x=\"271.5\" y=\"65\">urlsplit@481:netloc</text></g><path d=\"M362.25 61h10\"/><path d=\"M372.25 61h10\"/><g class=\"terminal\">\n",
       "<path d=\"M382.25 61h0.0\"/><path d=\"M410.75 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"382.25\" y=\"50\"/><text x=\"396.5\" y=\"65\">/</text></g></g><path d=\"M474.5 61h20\"/></g><path d=\"M 494.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@481:netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 335.5 62\" width=\"335.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M295.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">_checknetloc@419:netloc</text></g></g><path d=\"M275.5 31h20\"/></g><path d=\"M 295.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_checknetloc@419:netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 301.5 62\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M261.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">urlparse@394:netloc</text></g></g><path d=\"M241.5 31h20\"/></g><path d=\"M 261.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"152\" viewBox=\"0 0 369.5 152\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 81v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 91h0.0\"/><path d=\"M329.5 91h0.0\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-40a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v40a10 10 0 0 0 10 10\"/><path d=\"M40.0 91h20\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91h20\"/><path d=\"M40.0 91a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 121h68.0\"/><path d=\"M241.5 121h68.0\"/><g class=\"terminal\">\n",
       "<path d=\"M128.0 121h0.0\"/><path d=\"M241.5 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"113.5\" x=\"128.0\" y=\"110\"/><text x=\"184.75\" y=\"125\">freebsd.org</text></g></g><path d=\"M309.5 121a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 91 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@481:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 543.0 122\" width=\"543.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M503.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M483.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@486:url</text></g><path d=\"M216.0 31h10\"/><path d=\"M226.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M236.0 31h0.0\"/><path d=\"M264.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"236.0\" y=\"20\"/><text x=\"250.25\" y=\"35\">#</text></g><path d=\"M264.5 31h10\"/><path d=\"M274.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M284.5 31h0.0\"/><path d=\"M483.0 31h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"284.5\" y=\"20\"/><text x=\"383.75\" y=\"35\">urlsplit@486:fragment</text></g></g><path d=\"M483.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h83.75\"/><path d=\"M399.25 61h83.75\"/><g class=\"terminal\">\n",
       "<path d=\"M143.75 61h0.0\"/><path d=\"M180.75 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"143.75\" y=\"50\"/><text x=\"162.25\" y=\"65\">/#</text></g><path d=\"M180.75 61h10\"/><path d=\"M190.75 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M200.75 61h0.0\"/><path d=\"M399.25 61h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"200.75\" y=\"50\"/><text x=\"300.0\" y=\"65\">urlsplit@486:fragment</text></g></g><path d=\"M483.0 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h133.5\"/><path d=\"M349.5 91h133.5\"/><g class=\"non-terminal\">\n",
       "<path d=\"M193.5 91h0.0\"/><path d=\"M349.5 91h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"193.5\" y=\"80\"/><text x=\"271.5\" y=\"95\">urlparse@394:url</text></g></g><path d=\"M483.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 503.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@486:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 350.0 62\" width=\"350.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M310.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M97.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"20\"/><text x=\"78.5\" y=\"35\">/?</text></g><path d=\"M97.0 31h10\"/><path d=\"M107.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" width=\"173.0\" x=\"117.0\" y=\"20\"/><text x=\"203.5\" y=\"35\">urlsplit@488:query</text></g></g><path d=\"M290.0 31h20\"/></g><path d=\"M 310.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@488:query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 293.0 62\" width=\"293.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M253.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M233.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M233.0 31h0.0\"/><rect height=\"22\" width=\"173.0\" x=\"60.0\" y=\"20\"/><text x=\"146.5\" y=\"35\">urlparse@394:query</text></g></g><path d=\"M233.0 31h20\"/></g><path d=\"M 253.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@486:fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 318.5 62\" width=\"318.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M278.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M258.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M258.5 31h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"60.0\" y=\"20\"/><text x=\"159.25\" y=\"35\">urlparse@394:fragment</text></g></g><path d=\"M258.5 31h20\"/></g><path d=\"M 278.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 250.5 62\" width=\"250.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M210.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"130.5\" x=\"60.0\" y=\"20\"/><text x=\"125.25\" y=\"35\">/releases/5.8</text></g></g><path d=\"M190.5 31h20\"/></g><path d=\"M 210.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "su = ScopedGrammarMiner()\n",
    "for t in url_dts:\n",
    "    su.add_tree(t)\n",
    "syntax_diagram(readable(su.grammar))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One might notice that the grammar is not entirely human-readable, with a number of single token definitions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Hence, the last piece of the puzzle is the cleanup method `clean_grammar()`, which cleans up such definitions. The idea is to look for single token definitions such that a key is defined exactly by another key (single alternative, single token, nonterminal)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.623217Z",
     "iopub.status.busy": "2024-01-18T17:20:01.623126Z",
     "iopub.status.idle": "2024-01-18T17:20:01.625705Z",
     "shell.execute_reply": "2024-01-18T17:20:01.625454Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedGrammarMiner(ScopedGrammarMiner):\n",
    "    def get_replacements(self, grammar):\n",
    "        replacements = {}\n",
    "        for k in grammar:\n",
    "            if k == START_SYMBOL:\n",
    "                continue\n",
    "            alts = grammar[k]\n",
    "            if len(set([str(i) for i in alts])) != 1:\n",
    "                continue\n",
    "            rule = alts[0]\n",
    "            if len(rule) != 1:\n",
    "                continue\n",
    "            tok = rule[0]\n",
    "            if not is_nonterminal(tok):\n",
    "                continue\n",
    "            replacements[k] = tok\n",
    "        return replacements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Once we have such a list, iteratively replace the original key where ever it is used with the token we found earlier. Repeat until none is left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.627691Z",
     "iopub.status.busy": "2024-01-18T17:20:01.627462Z",
     "iopub.status.idle": "2024-01-18T17:20:01.630708Z",
     "shell.execute_reply": "2024-01-18T17:20:01.630338Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedGrammarMiner(ScopedGrammarMiner):\n",
    "    def clean_grammar(self):\n",
    "        replacements = self.get_replacements(self.grammar)\n",
    "\n",
    "        while True:\n",
    "            changed = set()\n",
    "            for k in self.grammar:\n",
    "                if k in replacements:\n",
    "                    continue\n",
    "                new_alts = []\n",
    "                for alt in self.grammar[k]:\n",
    "                    new_alt = []\n",
    "                    for t in alt:\n",
    "                        if t in replacements:\n",
    "                            new_alt.append(replacements[t])\n",
    "                            changed.add(t)\n",
    "                        else:\n",
    "                            new_alt.append(t)\n",
    "                    new_alts.append(new_alt)\n",
    "                self.grammar[k] = new_alts\n",
    "            if not changed:\n",
    "                break\n",
    "            for k in changed:\n",
    "                self.grammar.pop(k, None)\n",
    "        return readable(self.grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `clean_grammar()` is used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.632261Z",
     "iopub.status.busy": "2024-01-18T17:20:01.632172Z",
     "iopub.status.idle": "2024-01-18T17:20:01.649136Z",
     "shell.execute_reply": "2024-01-18T17:20:01.648851Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 395.0 62\" width=\"395.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M355.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><rect height=\"22\" width=\"275.0\" x=\"60.0\" y=\"20\"/><text x=\"197.5\" y=\"35\">process_inventory@22:inventory</text></g></g><path d=\"M335.0 31h20\"/></g><path d=\"M 355.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@22:inventory\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 980.0 62\" width=\"980.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M940.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M301.0 31h10\"/><path d=\"M311.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M321.0 31h0.0\"/><path d=\"M349.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"321.0\" y=\"20\"/><text x=\"335.25\" y=\"35\">\n",
       "</text></g><path d=\"M349.5 31h10\"/><path d=\"M359.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M369.5 31h0.0\"/><path d=\"M610.5 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"369.5\" y=\"20\"/><text x=\"490.0\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M610.5 31h10\"/><path d=\"M620.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M630.5 31h0.0\"/><path d=\"M659.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"630.5\" y=\"20\"/><text x=\"644.75\" y=\"35\">\n",
       "</text></g><path d=\"M659.0 31h10\"/><path d=\"M669.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M679.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"679.0\" y=\"20\"/><text x=\"799.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M920.0 31h20\"/></g><path d=\"M 940.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "si = ScopedGrammarMiner()\n",
    "si.add_tree(inventory_dt)\n",
    "syntax_diagram(readable(si.clean_grammar()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We update the `update_grammar()` to use the right tracker and miner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.650959Z",
     "iopub.status.busy": "2024-01-18T17:20:01.650695Z",
     "iopub.status.idle": "2024-01-18T17:20:01.653095Z",
     "shell.execute_reply": "2024-01-18T17:20:01.652850Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ScopedGrammarMiner(ScopedGrammarMiner):\n",
    "    def update_grammar(self, inputstr, trace):\n",
    "        at = self.create_tracker(inputstr, trace)\n",
    "        dt = self.create_tree_miner(\n",
    "            inputstr, at.my_assignments.defined_vars(\n",
    "                formatted=False))\n",
    "        self.add_tree(dt)\n",
    "        return self.grammar\n",
    "\n",
    "    def create_tracker(self, *args):\n",
    "        return ScopeTracker(*args)\n",
    "\n",
    "    def create_tree_miner(self, *args):\n",
    "        return ScopeTreeMiner(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `recover_grammar()` uses the right miner, and returns a cleaned grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.654929Z",
     "iopub.status.busy": "2024-01-18T17:20:01.654719Z",
     "iopub.status.idle": "2024-01-18T17:20:01.656855Z",
     "shell.execute_reply": "2024-01-18T17:20:01.656582Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def recover_grammar(fn, inputs, **kwargs):  # type: ignore\n",
    "    miner = ScopedGrammarMiner()\n",
    "    for inputstr in inputs:\n",
    "        with Tracer(inputstr, **kwargs) as tracer:\n",
    "            fn(tracer.my_input)\n",
    "        miner.update_grammar(tracer.my_input, tracer.trace)\n",
    "    return readable(miner.clean_grammar())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.658473Z",
     "iopub.status.busy": "2024-01-18T17:20:01.658288Z",
     "iopub.status.idle": "2024-01-18T17:20:01.816373Z",
     "shell.execute_reply": "2024-01-18T17:20:01.816072Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "url_grammar = recover_grammar(url_parse, URLS_X, files=['urllib/parse.py'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.818117Z",
     "iopub.status.busy": "2024-01-18T17:20:01.818022Z",
     "iopub.status.idle": "2024-01-18T17:20:01.832052Z",
     "shell.execute_reply": "2024-01-18T17:20:01.831600Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 276.0 62\" width=\"276.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M236.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@437:url</text></g></g><path d=\"M216.0 31h20\"/></g><path d=\"M 236.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@437:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 560.0 62\" width=\"560.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M520.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M500.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">urlparse@394:scheme</text></g><path d=\"M241.5 31h10\"/><path d=\"M251.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M261.5 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"261.5\" y=\"20\"/><text x=\"275.75\" y=\"35\">:</text></g><path d=\"M290.0 31h10\"/><path d=\"M300.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M310.0 31h0.0\"/><path d=\"M500.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"310.0\" y=\"20\"/><text x=\"405.0\" y=\"35\">_splitnetloc@411:url</text></g></g><path d=\"M500.0 31h20\"/></g><path d=\"M 520.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 182.5 122\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h8.5\"/><path d=\"M114.0 91h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 91h0.0\"/><path d=\"M114.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"68.5\" y=\"80\"/><text x=\"91.25\" y=\"95\">ftp</text></g></g><path d=\"M122.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_splitnetloc@411:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 534.5 92\" width=\"534.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M494.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h63.75\"/><path d=\"M410.75 31h63.75\"/><g class=\"terminal\">\n",
       "<path d=\"M123.75 31h0.0\"/><path d=\"M160.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"123.75\" y=\"20\"/><text x=\"142.25\" y=\"35\">//</text></g><path d=\"M160.75 31h10\"/><path d=\"M170.75 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M180.75 31h0.0\"/><path d=\"M362.25 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"180.75\" y=\"20\"/><text x=\"271.5\" y=\"35\">urlparse@394:netloc</text></g><path d=\"M362.25 31h10\"/><path d=\"M372.25 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M382.25 31h0.0\"/><path d=\"M410.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"382.25\" y=\"20\"/><text x=\"396.5\" y=\"35\">/</text></g></g><path d=\"M474.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M474.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M97.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"50\"/><text x=\"78.5\" y=\"65\">//</text></g><path d=\"M97.0 61h10\"/><path d=\"M107.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 61h0.0\"/><path d=\"M298.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"117.0\" y=\"50\"/><text x=\"207.75\" y=\"65\">urlparse@394:netloc</text></g><path d=\"M298.5 61h10\"/><path d=\"M308.5 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M318.5 61h0.0\"/><path d=\"M474.5 61h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"318.5\" y=\"50\"/><text x=\"396.5\" y=\"65\">urlsplit@481:url</text></g></g><path d=\"M474.5 61h20\"/></g><path d=\"M 494.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"152\" viewBox=\"0 0 369.5 152\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 81v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 91h0.0\"/><path d=\"M329.5 91h0.0\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 91a10 10 0 0 0 10 -10v-40a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v40a10 10 0 0 0 10 10\"/><path d=\"M40.0 91h20\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91h20\"/><path d=\"M40.0 91a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 121h68.0\"/><path d=\"M241.5 121h68.0\"/><g class=\"terminal\">\n",
       "<path d=\"M128.0 121h0.0\"/><path d=\"M241.5 121h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"113.5\" x=\"128.0\" y=\"110\"/><text x=\"184.75\" y=\"125\">freebsd.org</text></g></g><path d=\"M309.5 121a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 91 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@481:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 543.0 122\" width=\"543.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M503.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M483.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@486:url</text></g><path d=\"M216.0 31h10\"/><path d=\"M226.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M236.0 31h0.0\"/><path d=\"M264.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"236.0\" y=\"20\"/><text x=\"250.25\" y=\"35\">#</text></g><path d=\"M264.5 31h10\"/><path d=\"M274.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M284.5 31h0.0\"/><path d=\"M483.0 31h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"284.5\" y=\"20\"/><text x=\"383.75\" y=\"35\">urlparse@394:fragment</text></g></g><path d=\"M483.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h83.75\"/><path d=\"M399.25 61h83.75\"/><g class=\"terminal\">\n",
       "<path d=\"M143.75 61h0.0\"/><path d=\"M180.75 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"143.75\" y=\"50\"/><text x=\"162.25\" y=\"65\">/#</text></g><path d=\"M180.75 61h10\"/><path d=\"M190.75 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M200.75 61h0.0\"/><path d=\"M399.25 61h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"200.75\" y=\"50\"/><text x=\"300.0\" y=\"65\">urlparse@394:fragment</text></g></g><path d=\"M483.0 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h133.5\"/><path d=\"M349.5 91h133.5\"/><g class=\"non-terminal\">\n",
       "<path d=\"M193.5 91h0.0\"/><path d=\"M349.5 91h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"193.5\" y=\"80\"/><text x=\"271.5\" y=\"95\">urlparse@394:url</text></g></g><path d=\"M483.0 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 503.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@486:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 350.0 62\" width=\"350.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M310.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M97.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"20\"/><text x=\"78.5\" y=\"35\">/?</text></g><path d=\"M97.0 31h10\"/><path d=\"M107.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" width=\"173.0\" x=\"117.0\" y=\"20\"/><text x=\"203.5\" y=\"35\">urlparse@394:query</text></g></g><path d=\"M290.0 31h20\"/></g><path d=\"M 310.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 250.5 62\" width=\"250.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M210.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M190.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"130.5\" x=\"60.0\" y=\"20\"/><text x=\"125.25\" y=\"35\">/releases/5.8</text></g></g><path d=\"M190.5 31h20\"/></g><path d=\"M 210.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(url_grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.833816Z",
     "iopub.status.busy": "2024-01-18T17:20:01.833737Z",
     "iopub.status.idle": "2024-01-18T17:20:01.836723Z",
     "shell.execute_reply": "2024-01-18T17:20:01.836440Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://freebsd.org/\n",
      "http://www.cispa.saarland:80/#News\n",
      "http://user:pass@www.google.com:80/\n",
      "ftp://www.cispa.saarland:80/#News\n",
      "http://www.fuzzingbook.org/\n",
      "https://freebsd.org/\n",
      "https://user:pass@www.google.com:80/\n",
      "ftp://www.fuzzingbook.org/\n",
      "https://www.cispa.saarland:80/#News\n",
      "https://user:pass@www.google.com:80/\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(url_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.838383Z",
     "iopub.status.busy": "2024-01-18T17:20:01.838236Z",
     "iopub.status.idle": "2024-01-18T17:20:01.847272Z",
     "shell.execute_reply": "2024-01-18T17:20:01.846961Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "inventory_grammar = recover_grammar(process_inventory, [INVENTORY])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.848934Z",
     "iopub.status.busy": "2024-01-18T17:20:01.848844Z",
     "iopub.status.idle": "2024-01-18T17:20:01.864494Z",
     "shell.execute_reply": "2024-01-18T17:20:01.864237Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 395.0 62\" width=\"395.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M355.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><rect height=\"22\" width=\"275.0\" x=\"60.0\" y=\"20\"/><text x=\"197.5\" y=\"35\">process_inventory@22:inventory</text></g></g><path d=\"M335.0 31h20\"/></g><path d=\"M 355.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@22:inventory\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 980.0 62\" width=\"980.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M940.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M301.0 31h10\"/><path d=\"M311.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M321.0 31h0.0\"/><path d=\"M349.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"321.0\" y=\"20\"/><text x=\"335.25\" y=\"35\">\n",
       "</text></g><path d=\"M349.5 31h10\"/><path d=\"M359.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M369.5 31h0.0\"/><path d=\"M610.5 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"369.5\" y=\"20\"/><text x=\"490.0\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M610.5 31h10\"/><path d=\"M620.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M630.5 31h0.0\"/><path d=\"M659.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"630.5\" y=\"20\"/><text x=\"644.75\" y=\"35\">\n",
       "</text></g><path d=\"M659.0 31h10\"/><path d=\"M669.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M679.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"679.0\" y=\"20\"/><text x=\"799.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M920.0 31h20\"/></g><path d=\"M 940.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(inventory_grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.866188Z",
     "iopub.status.busy": "2024-01-18T17:20:01.866074Z",
     "iopub.status.idle": "2024-01-18T17:20:01.874464Z",
     "shell.execute_reply": "2024-01-18T17:20:01.874243Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000,car,Ford,E350\n",
      "2000,van,Ford,E350\n",
      "1997,car,Chevy,Cougar\n",
      "1997,van,Ford,E350\n",
      "1997,car,Ford,E350\n",
      "1999,car,Mercury,Venture\n",
      "2000,car,Ford,E350\n",
      "2000,van,Chevy,Venture\n",
      "1997,van,Mercury,Venture\n",
      "2000,car,Ford,E350\n",
      "1997,van,Ford,Venture\n",
      "2000,van,Mercury,E350\n",
      "1997,van,Chevy,Venture\n",
      "2000,van,Ford,Venture\n",
      "1999,car,Ford,E350\n",
      "1997,car,Chevy,Venture\n",
      "1997,car,Mercury,E350\n",
      "1997,van,Ford,E350\n",
      "1997,van,Ford,Cougar\n",
      "1999,van,Chevy,E350\n",
      "2000,car,Ford,E350\n",
      "1997,van,Chevy,Venture\n",
      "2000,van,Chevy,Venture\n",
      "2000,van,Ford,E350\n",
      "1997,van,Chevy,Cougar\n",
      "1999,van,Chevy,E350\n",
      "1999,car,Chevy,E350\n",
      "1999,car,Ford,E350\n",
      "2000,van,Ford,Venture\n",
      "1997,van,Ford,Venture\n"
     ]
    }
   ],
   "source": [
    "f = GrammarFuzzer(inventory_grammar)\n",
    "for _ in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We see how tracking scope helps us to extract an even more precise grammar."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notice that we use *String* inclusion testing as a way of determining whether a particular string fragment came from the original input string. While this may seem rather error-prone compared to dynamic tainting, we note that numerous tracing tools such as `dtrace()` and `ptrace()` allow one to obtain the information we seek from execution of binaries directly in different platforms. However, methods for obtaining dynamic taints almost always involve instrumenting the binaries before they can be used. Hence, this method of string inclusion can be more generally applied than dynamic tainting approaches. Further, dynamic taints are often lost due to implicit transmission, or at the boundary between *Python* and *C* code. String inclusion has not such problems. Hence, our approach can often obtain better results than relying on dynamic tainting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Synopsis\n",
    "\n",
    "This chapter provides a number of classes to mine input grammars from existing programs.  The function `recover_grammar()` could be the easiest to use.  It takes a function and a set of inputs, and returns a grammar that describes its input language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We apply `recover_grammar()` on a `url_parse()` function that takes and decomposes URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.876191Z",
     "iopub.status.busy": "2024-01-18T17:20:01.876046Z",
     "iopub.status.idle": "2024-01-18T17:20:01.877886Z",
     "shell.execute_reply": "2024-01-18T17:20:01.877587Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "url_parse('https://www.fuzzingbook.org/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.879689Z",
     "iopub.status.busy": "2024-01-18T17:20:01.879541Z",
     "iopub.status.idle": "2024-01-18T17:20:01.881714Z",
     "shell.execute_reply": "2024-01-18T17:20:01.881470Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://user:pass@www.google.com:80/?q=path#ref',\n",
       " 'https://www.cispa.saarland:80/',\n",
       " 'http://www.fuzzingbook.org/#News']"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "URLS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We extract the input grammar for `url_parse()` using `recover_grammar()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.883350Z",
     "iopub.status.busy": "2024-01-18T17:20:01.883221Z",
     "iopub.status.idle": "2024-01-18T17:20:01.993056Z",
     "shell.execute_reply": "2024-01-18T17:20:01.992759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': ['<urlsplit@437:url>'],\n",
       " '<urlsplit@437:url>': ['<urlparse@394:scheme>:<_splitnetloc@411:url>'],\n",
       " '<urlparse@394:scheme>': ['https', 'http'],\n",
       " '<_splitnetloc@411:url>': ['//<urlparse@394:netloc>/',\n",
       "  '//<urlparse@394:netloc><urlsplit@481:url>'],\n",
       " '<urlparse@394:netloc>': ['user:pass@www.google.com:80',\n",
       "  'www.fuzzingbook.org',\n",
       "  'www.cispa.saarland:80'],\n",
       " '<urlsplit@481:url>': ['<urlsplit@486:url>#<urlparse@394:fragment>',\n",
       "  '/#<urlparse@394:fragment>'],\n",
       " '<urlsplit@486:url>': ['/?<urlparse@394:query>'],\n",
       " '<urlparse@394:query>': ['q=path'],\n",
       " '<urlparse@394:fragment>': ['ref', 'News']}"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grammar = recover_grammar(url_parse, URLS, files=['urllib/parse.py'])\n",
    "grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The names of nonterminals are a bit technical; but the grammar nicely represents the structure of the input; for instance, the different schemes (`\"http\"`, `\"https\"`) are all identified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:01.994762Z",
     "iopub.status.busy": "2024-01-18T17:20:01.994647Z",
     "iopub.status.idle": "2024-01-18T17:20:02.005887Z",
     "shell.execute_reply": "2024-01-18T17:20:02.005551Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 276.0 62\" width=\"276.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M236.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@437:url</text></g></g><path d=\"M216.0 31h20\"/></g><path d=\"M 236.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@437:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 560.0 62\" width=\"560.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M520.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M500.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">urlparse@394:scheme</text></g><path d=\"M241.5 31h10\"/><path d=\"M251.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M261.5 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"261.5\" y=\"20\"/><text x=\"275.75\" y=\"35\">:</text></g><path d=\"M290.0 31h10\"/><path d=\"M300.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M310.0 31h0.0\"/><path d=\"M500.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"310.0\" y=\"20\"/><text x=\"405.0\" y=\"35\">_splitnetloc@411:url</text></g></g><path d=\"M500.0 31h20\"/></g><path d=\"M 520.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:scheme\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 182.5 92\" width=\"182.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M142.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M122.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"60.0\" y=\"20\"/><text x=\"91.25\" y=\"35\">https</text></g></g><path d=\"M122.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h4.25\"/><path d=\"M118.25 61h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 61h0.0\"/><path d=\"M118.25 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"64.25\" y=\"50\"/><text x=\"91.25\" y=\"65\">http</text></g></g><path d=\"M122.5 61h20\"/></g><path d=\"M 142.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_splitnetloc@411:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 534.5 92\" width=\"534.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M494.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h63.75\"/><path d=\"M410.75 31h63.75\"/><g class=\"terminal\">\n",
       "<path d=\"M123.75 31h0.0\"/><path d=\"M160.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"123.75\" y=\"20\"/><text x=\"142.25\" y=\"35\">//</text></g><path d=\"M160.75 31h10\"/><path d=\"M170.75 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M180.75 31h0.0\"/><path d=\"M362.25 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"180.75\" y=\"20\"/><text x=\"271.5\" y=\"35\">urlparse@394:netloc</text></g><path d=\"M362.25 31h10\"/><path d=\"M372.25 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M382.25 31h0.0\"/><path d=\"M410.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"382.25\" y=\"20\"/><text x=\"396.5\" y=\"35\">/</text></g></g><path d=\"M474.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M474.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M97.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"50\"/><text x=\"78.5\" y=\"65\">//</text></g><path d=\"M97.0 61h10\"/><path d=\"M107.0 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 61h0.0\"/><path d=\"M298.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"117.0\" y=\"50\"/><text x=\"207.75\" y=\"65\">urlparse@394:netloc</text></g><path d=\"M298.5 61h10\"/><path d=\"M308.5 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M318.5 61h0.0\"/><path d=\"M474.5 61h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"318.5\" y=\"50\"/><text x=\"396.5\" y=\"65\">urlsplit@481:url</text></g></g><path d=\"M474.5 61h20\"/></g><path d=\"M 494.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:netloc\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"122\" viewBox=\"0 0 369.5 122\" width=\"369.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M329.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M309.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"249.5\" x=\"60.0\" y=\"20\"/><text x=\"184.75\" y=\"35\">user:pass@www.google.com:80</text></g></g><path d=\"M309.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h34.0\"/><path d=\"M275.5 61h34.0\"/><g class=\"terminal\">\n",
       "<path d=\"M94.0 61h0.0\"/><path d=\"M275.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"181.5\" x=\"94.0\" y=\"50\"/><text x=\"184.75\" y=\"65\">www.fuzzingbook.org</text></g></g><path d=\"M309.5 61h20\"/><path d=\"M40.0 61a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><g>\n",
       "<path d=\"M60.0 91h25.5\"/><path d=\"M284.0 91h25.5\"/><g class=\"terminal\">\n",
       "<path d=\"M85.5 91h0.0\"/><path d=\"M284.0 91h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"198.5\" x=\"85.5\" y=\"80\"/><text x=\"184.75\" y=\"95\">www.cispa.saarland:80</text></g></g><path d=\"M309.5 91a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/></g><path d=\"M 329.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@481:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 543.0 92\" width=\"543.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M503.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M483.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M216.0 31h0.0\"/><rect height=\"22\" width=\"156.0\" x=\"60.0\" y=\"20\"/><text x=\"138.0\" y=\"35\">urlsplit@486:url</text></g><path d=\"M216.0 31h10\"/><path d=\"M226.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M236.0 31h0.0\"/><path d=\"M264.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"236.0\" y=\"20\"/><text x=\"250.25\" y=\"35\">#</text></g><path d=\"M264.5 31h10\"/><path d=\"M274.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M284.5 31h0.0\"/><path d=\"M483.0 31h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"284.5\" y=\"20\"/><text x=\"383.75\" y=\"35\">urlparse@394:fragment</text></g></g><path d=\"M483.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h83.75\"/><path d=\"M399.25 61h83.75\"/><g class=\"terminal\">\n",
       "<path d=\"M143.75 61h0.0\"/><path d=\"M180.75 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"143.75\" y=\"50\"/><text x=\"162.25\" y=\"65\">/#</text></g><path d=\"M180.75 61h10\"/><path d=\"M190.75 61h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M200.75 61h0.0\"/><path d=\"M399.25 61h0.0\"/><rect height=\"22\" width=\"198.5\" x=\"200.75\" y=\"50\"/><text x=\"300.0\" y=\"65\">urlparse@394:fragment</text></g></g><path d=\"M483.0 61h20\"/></g><path d=\"M 503.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlsplit@486:url\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 350.0 62\" width=\"350.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M310.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M97.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"37.0\" x=\"60.0\" y=\"20\"/><text x=\"78.5\" y=\"35\">/?</text></g><path d=\"M97.0 31h10\"/><path d=\"M107.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M117.0 31h0.0\"/><path d=\"M290.0 31h0.0\"/><rect height=\"22\" width=\"173.0\" x=\"117.0\" y=\"20\"/><text x=\"203.5\" y=\"35\">urlparse@394:query</text></g></g><path d=\"M290.0 31h20\"/></g><path d=\"M 310.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:query\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 191.0 62\" width=\"191.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M151.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M131.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"60.0\" y=\"20\"/><text x=\"95.5\" y=\"35\">q=path</text></g></g><path d=\"M131.0 31h20\"/></g><path d=\"M 151.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "urlparse@394:fragment\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M109.75 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M109.75 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"64.25\" y=\"20\"/><text x=\"87.0\" y=\"35\">ref</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">News</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar can be immediately used for fuzzing, producing arbitrary combinations of input elements, which are all syntactically valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.007570Z",
     "iopub.status.busy": "2024-01-18T17:20:02.007486Z",
     "iopub.status.idle": "2024-01-18T17:20:02.314819Z",
     "shell.execute_reply": "2024-01-18T17:20:02.314516Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from GrammarCoverageFuzzer import GrammarCoverageFuzzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.316741Z",
     "iopub.status.busy": "2024-01-18T17:20:02.316571Z",
     "iopub.status.idle": "2024-01-18T17:20:02.320896Z",
     "shell.execute_reply": "2024-01-18T17:20:02.320514Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://www.fuzzingbook.org/',\n",
       " 'http://user:pass@www.google.com:80/#News',\n",
       " 'http://www.cispa.saarland:80/?q=path#ref',\n",
       " 'https://user:pass@www.google.com:80/#ref',\n",
       " 'https://www.cispa.saarland:80/']"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fuzzer = GrammarCoverageFuzzer(grammar)\n",
    "[fuzzer.fuzz() for i in range(5)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Being able to automatically extract a grammar and to use this grammar for fuzzing makes for very effective test generation with a minimum of manual work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lessons Learned\n",
    "\n",
    "* Given a set of sample inputs for a program, we can learn an input grammar by examining variable values during execution if the program relies on handwritten parsers.\n",
    "* Simple string inclusion checks are sufficient to obtain reasonably accurate grammars from real world programs.\n",
    "* The resulting grammars can be directly used for fuzzing, and can have a multiplier effect on any samples you have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Next Steps\n",
    "\n",
    "* Learn how to use [information flow](InformationFlow.ipynb) to further improve mapping inputs to states."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Background\n",
    "\n",
    "Recovering the language from a _set of samples_ (i.e., not taking into account a possible program that might process them) is a well researched topic.  The excellent reference by Higuera \\cite{higuera2010grammatical} covers all the classical approaches. The current state of the art in black box grammar mining is described by Clark \\cite{clark2013learning}.\n",
    "\n",
    "Learning an input language from a _program_, with or without samples, is yet an emerging topic, despite its potential for fuzzing.  The pioneering work in this area was done by Lin et al. \\cite{Lin2008} who invented a way to retrieve the parse trees from top down and bottom up parsers. The approach described in this chapter is based directly on the AUTOGRAM work of Hoschele et al. \\cite{Hoschele2017}."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise 1: Flattening complex objects\n",
    "\n",
    "Our grammar miners only check for string fragments. However, programs may often pass containers or custom objects containing input fragments. For example, consider the plausible modification for our inventory processor, where we use a custom object `Vehicle` to carry fragments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.322715Z",
     "iopub.status.busy": "2024-01-18T17:20:02.322619Z",
     "iopub.status.idle": "2024-01-18T17:20:02.324772Z",
     "shell.execute_reply": "2024-01-18T17:20:02.324481Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class Vehicle:\n",
    "    def __init__(self, vehicle: str):\n",
    "        year, kind, company, model, *_ = vehicle.split(',')\n",
    "        self.year, self.kind, self.company, self.model = year, kind, company, model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.326281Z",
     "iopub.status.busy": "2024-01-18T17:20:02.326162Z",
     "iopub.status.idle": "2024-01-18T17:20:02.328268Z",
     "shell.execute_reply": "2024-01-18T17:20:02.327985Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_inventory_with_obj(inventory: str) -> str:\n",
    "    res = []\n",
    "    for vehicle in inventory.split('\\n'):\n",
    "        ret = process_vehicle(vehicle)\n",
    "        res.extend(ret)\n",
    "\n",
    "    return '\\n'.join(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.329826Z",
     "iopub.status.busy": "2024-01-18T17:20:02.329716Z",
     "iopub.status.idle": "2024-01-18T17:20:02.331698Z",
     "shell.execute_reply": "2024-01-18T17:20:02.331460Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_vehicle_with_obj(vehicle: str) -> List[str]:\n",
    "    v = Vehicle(vehicle)\n",
    "    if v.kind == 'van':\n",
    "        return process_van_with_obj(v)\n",
    "\n",
    "    elif v.kind == 'car':\n",
    "        return process_car_with_obj(v)\n",
    "\n",
    "    else:\n",
    "        raise Exception('Invalid entry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.333278Z",
     "iopub.status.busy": "2024-01-18T17:20:02.333172Z",
     "iopub.status.idle": "2024-01-18T17:20:02.335165Z",
     "shell.execute_reply": "2024-01-18T17:20:02.334931Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_van_with_obj(vehicle: Vehicle) -> List[str]:\n",
    "    res = [\n",
    "        \"We have a %s %s van from %s vintage.\" % (vehicle.company,\n",
    "                                                  vehicle.model, vehicle.year)\n",
    "    ]\n",
    "    iyear = int(vehicle.year)\n",
    "    if iyear > 2010:\n",
    "        res.append(\"It is a recent model!\")\n",
    "    else:\n",
    "        res.append(\"It is an old but reliable model!\")\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.336583Z",
     "iopub.status.busy": "2024-01-18T17:20:02.336484Z",
     "iopub.status.idle": "2024-01-18T17:20:02.338431Z",
     "shell.execute_reply": "2024-01-18T17:20:02.338176Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_car_with_obj(vehicle: Vehicle) -> List[str]:\n",
    "    res = [\n",
    "        \"We have a %s %s car from %s vintage.\" % (vehicle.company,\n",
    "                                                  vehicle.model, vehicle.year)\n",
    "    ]\n",
    "    iyear = int(vehicle.year)\n",
    "    if iyear > 2016:\n",
    "        res.append(\"It is a recent model!\")\n",
    "    else:\n",
    "        res.append(\"It is an old but reliable model!\")\n",
    "    return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We recover the grammar as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.339911Z",
     "iopub.status.busy": "2024-01-18T17:20:02.339812Z",
     "iopub.status.idle": "2024-01-18T17:20:02.414973Z",
     "shell.execute_reply": "2024-01-18T17:20:02.414643Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "vehicle_grammar = recover_grammar(\n",
    "    process_inventory_with_obj,\n",
    "    [INVENTORY],\n",
    "    methods=INVENTORY_METHODS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The new vehicle grammar is missing in details, especially as to the different models and company for a van and car."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.416973Z",
     "iopub.status.busy": "2024-01-18T17:20:02.416853Z",
     "iopub.status.idle": "2024-01-18T17:20:02.431874Z",
     "shell.execute_reply": "2024-01-18T17:20:02.431613Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 980.0 62\" width=\"980.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M940.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M301.0 31h10\"/><path d=\"M311.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M321.0 31h0.0\"/><path d=\"M349.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"321.0\" y=\"20\"/><text x=\"335.25\" y=\"35\">\n",
       "</text></g><path d=\"M349.5 31h10\"/><path d=\"M359.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M369.5 31h0.0\"/><path d=\"M610.5 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"369.5\" y=\"20\"/><text x=\"490.0\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M610.5 31h10\"/><path d=\"M620.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M630.5 31h0.0\"/><path d=\"M659.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"630.5\" y=\"20\"/><text x=\"644.75\" y=\"35\">\n",
       "</text></g><path d=\"M659.0 31h10\"/><path d=\"M669.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M679.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"679.0\" y=\"20\"/><text x=\"799.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M920.0 31h20\"/></g><path d=\"M 940.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(vehicle_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "The problem is that, we are looking specifically for string objects that contain fragments of the input string during tracing. Can you modify our grammar miner to correctly account for the complex objects too?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "The problem can be understood if we execute the tracer under verbose logging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.433872Z",
     "iopub.status.busy": "2024-01-18T17:20:02.433718Z",
     "iopub.status.idle": "2024-01-18T17:20:02.441104Z",
     "shell.execute_reply": "2024-01-18T17:20:02.440892Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:22:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:23:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory)\n",
      "\n",
      "Traced values:\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:22:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:23:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('return', ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('return', ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('return', ['We have a Mercury Cougar car from 2000 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('return', ['We have a Mercury Cougar car from 2000 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('return', ['We have a Chevy Venture car from 1999 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('return', ['We have a Chevy Venture car from 1999 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('return', 'We have a Ford E350 van from 1997 vintage.\\nIt is an old but reliable model!\\nWe have a Mercury Cougar car from 2000 vintage.\\nIt is an old but reliable model!\\nWe have a Chevy Venture car from 1999 vintage.\\nIt is an old but reliable model!', /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n"
     ]
    }
   ],
   "source": [
    "with Tracer(INVENTORY, methods=INVENTORY_METHODS, log=True) as tracer:\n",
    "    process_inventory(tracer.my_input)\n",
    "print()\n",
    "print('Traced values:')\n",
    "for t in tracer.trace:\n",
    "    print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "You can see that we lose track of string fragments as soon as they are incorporated into the `Vehicle` object. The way out is to trace these variables separately."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "For that, we develop the `flatten()` method that given any custom complex object and its key, returns a list of flattened (_key_, _value_) pairs that correspond to the object passed in.\n",
    "\n",
    "The `MAX_DEPTH` parameter controls the maximum flattening limit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.442940Z",
     "iopub.status.busy": "2024-01-18T17:20:02.442764Z",
     "iopub.status.idle": "2024-01-18T17:20:02.444523Z",
     "shell.execute_reply": "2024-01-18T17:20:02.444240Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "MAX_DEPTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.446242Z",
     "iopub.status.busy": "2024-01-18T17:20:02.446035Z",
     "iopub.status.idle": "2024-01-18T17:20:02.447775Z",
     "shell.execute_reply": "2024-01-18T17:20:02.447550Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "def set_flatten_depth(depth):\n",
    "    global MAX_DEPTH\n",
    "    MAX_DEPTH = depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.449358Z",
     "iopub.status.busy": "2024-01-18T17:20:02.449163Z",
     "iopub.status.idle": "2024-01-18T17:20:02.452622Z",
     "shell.execute_reply": "2024-01-18T17:20:02.452371Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "def flatten(key, val, depth=MAX_DEPTH):\n",
    "    tv = type(val)\n",
    "    if depth <= 0:\n",
    "        return [(key, val)]\n",
    "    if isinstance(val, (int, float, complex, str, bytes, bytearray)):\n",
    "        return [(key, val)]\n",
    "    elif isinstance(val, (set, frozenset, list, tuple, range)):\n",
    "        values = [(i, e) for i, elt in enumerate(val) for e in flatten(i, elt, depth-1)]\n",
    "        return [(\"%s.%d\" % (key, i), v) for i, v in values]\n",
    "    elif isinstance(val, dict):\n",
    "        values = [e for k, elt in val.items() for e in flatten(k, elt, depth-1)]\n",
    "        return [(\"%s.%s\" % (key, k), v) for k, v in values]\n",
    "    elif isinstance(val, str):\n",
    "        return [(key, val)]\n",
    "    elif hasattr(val, '__dict__'):\n",
    "        values = [e for k, elt in val.__dict__.items()\n",
    "                  for e in flatten(k, elt, depth-1)]\n",
    "        return [(\"%s.%s\" % (key, k), v) for k, v in values]\n",
    "    else:\n",
    "        return [(key, val)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Next, we hook the `flatten()` into the `Context` class so that the parameters we obtain are flattened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.454150Z",
     "iopub.status.busy": "2024-01-18T17:20:02.454075Z",
     "iopub.status.idle": "2024-01-18T17:20:02.456516Z",
     "shell.execute_reply": "2024-01-18T17:20:02.456274Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class Context(Context):\n",
    "    def extract_vars(self, frame):\n",
    "        vals = inspect.getargvalues(frame).locals\n",
    "        return {k1: v1 for k, v in vals.items() for k1, v1 in flatten(k, v)}\n",
    "\n",
    "    def parameters(self, all_vars):\n",
    "        def check_param(k):\n",
    "            return any(k.startswith(p) for p in self.parameter_names)\n",
    "        return {k: v for k, v in all_vars.items() if check_param(k)}\n",
    "\n",
    "    def qualified(self, all_vars):\n",
    "        return {\"%s:%s\" % (self.method, k): v for k, v in all_vars.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "With this change, we have the following trace output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.458070Z",
     "iopub.status.busy": "2024-01-18T17:20:02.457988Z",
     "iopub.status.idle": "2024-01-18T17:20:02.465862Z",
     "shell.execute_reply": "2024-01-18T17:20:02.465627Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:22:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:23:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "-> /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory)\n",
      "   /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory)\n",
      "<- /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory)\n",
      "\n",
      "Traced values:\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:22:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:23:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:40:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:41:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:42:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:43:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:46:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('return', ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:47:process_van(year,company,model), {'year': '1997', 'company': 'Ford', 'model': 'E350'})\n",
      "('return', ['We have a Ford E350 van from 1997 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:32:process_vehicle(vehicle), {'vehicle': '1997,van,Ford,E350', 'year': '1997', 'kind': 'van', 'company': 'Ford', 'model': 'E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1997,van,Ford,E350'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('return', ['We have a Mercury Cougar car from 2000 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '2000', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('return', ['We have a Mercury Cougar car from 2000 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '2000,car,Mercury,Cougar', 'year': '2000', 'kind': 'car', 'company': 'Mercury', 'model': 'Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '2000,car,Mercury,Cougar'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:25:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:29:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:30:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:31:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:34:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('call', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:49:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:50:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:51:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:52:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:55:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('return', ['We have a Chevy Venture car from 1999 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:56:process_car(year,company,model), {'year': '1999', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('return', ['We have a Chevy Venture car from 1999 vintage.', 'It is an old but reliable model!'], /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:35:process_vehicle(vehicle), {'vehicle': '1999,car,Chevy,Venture', 'year': '1999', 'kind': 'car', 'company': 'Chevy', 'model': 'Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:26:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:24:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('line', None, /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n",
      "('return', 'We have a Ford E350 van from 1997 vintage.\\nIt is an old but reliable model!\\nWe have a Mercury Cougar car from 2000 vintage.\\nIt is an old but reliable model!\\nWe have a Chevy Venture car from 1999 vintage.\\nIt is an old but reliable model!', /Users/zeller/Projects/fuzzingbook/notebooks/Parser.ipynb:27:process_inventory(inventory), {'inventory': '1997,van,Ford,E350\\n2000,car,Mercury,Cougar\\n1999,car,Chevy,Venture', 'vehicle': '1999,car,Chevy,Venture'})\n"
     ]
    }
   ],
   "source": [
    "with Tracer(INVENTORY, methods=INVENTORY_METHODS, log=True) as tracer:\n",
    "    process_inventory(tracer.my_input)\n",
    "print()\n",
    "print('Traced values:')\n",
    "for t in tracer.trace:\n",
    "    print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Our change seems to have worked. Let us derive the grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.467451Z",
     "iopub.status.busy": "2024-01-18T17:20:02.467372Z",
     "iopub.status.idle": "2024-01-18T17:20:02.477155Z",
     "shell.execute_reply": "2024-01-18T17:20:02.476770Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "vehicle_grammar = recover_grammar(\n",
    "    process_inventory,\n",
    "    [INVENTORY],\n",
    "    methods=INVENTORY_METHODS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.478825Z",
     "iopub.status.busy": "2024-01-18T17:20:02.478735Z",
     "iopub.status.idle": "2024-01-18T17:20:02.493103Z",
     "shell.execute_reply": "2024-01-18T17:20:02.492869Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 395.0 62\" width=\"395.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M355.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><rect height=\"22\" width=\"275.0\" x=\"60.0\" y=\"20\"/><text x=\"197.5\" y=\"35\">process_inventory@22:inventory</text></g></g><path d=\"M335.0 31h20\"/></g><path d=\"M 355.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@22:inventory\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 980.0 62\" width=\"980.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M940.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M301.0 31h10\"/><path d=\"M311.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M321.0 31h0.0\"/><path d=\"M349.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"321.0\" y=\"20\"/><text x=\"335.25\" y=\"35\">\n",
       "</text></g><path d=\"M349.5 31h10\"/><path d=\"M359.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M369.5 31h0.0\"/><path d=\"M610.5 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"369.5\" y=\"20\"/><text x=\"490.0\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M610.5 31h10\"/><path d=\"M620.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M630.5 31h0.0\"/><path d=\"M659.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"630.5\" y=\"20\"/><text x=\"644.75\" y=\"35\">\n",
       "</text></g><path d=\"M659.0 31h10\"/><path d=\"M669.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M679.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"679.0\" y=\"20\"/><text x=\"799.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M920.0 31h20\"/></g><path d=\"M 940.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(vehicle_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "The recovered grammar contains all the details that we were able to recover before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise 2: Incorporating Taints from InformationFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "We have been using *string inclusion* to check whether a particular fragment came from the input string. This is unsatisfactory as it required us to compromise on the size of the strings tracked, which was limited to those greater than `FRAGMENT_LEN`. Further, it is possible that a single method could process a string where a fragment repeats, but is part of different tokens. For example, an embedded comma in the CSV file would cause our parser to fail. One way to avoid this is to rely on *dynamic taints*, and check for taint inclusion rather than string inclusion.\n",
    "\n",
    "The chapter on [information flow](InformationFlow.ipynb) details how to incorporate dynamic taints. Can you update our grammar miner based on scope to use *dynamic taints* instead?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "First, we import `ostr` to track the origins of string fragments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.495192Z",
     "iopub.status.busy": "2024-01-18T17:20:02.495027Z",
     "iopub.status.idle": "2024-01-18T17:20:02.557982Z",
     "shell.execute_reply": "2024-01-18T17:20:02.557719Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "from InformationFlow import ostr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Next, we define `is_fragment()` to verify that a fragment is from a given input string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.560152Z",
     "iopub.status.busy": "2024-01-18T17:20:02.559930Z",
     "iopub.status.idle": "2024-01-18T17:20:02.561957Z",
     "shell.execute_reply": "2024-01-18T17:20:02.561697Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "def is_fragment(fragment, original):\n",
    "    assert isinstance(original, ostr)\n",
    "    if not isinstance(fragment, ostr):\n",
    "        return False\n",
    "    return set(fragment.origin) <= set(original.origin)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Now, all that remains is to hook the tainted fragment check to our grammar miner. This is accomplished by modifying `in_current_record()` and `ignored()` methods in the `InputStack`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.563641Z",
     "iopub.status.busy": "2024-01-18T17:20:02.563513Z",
     "iopub.status.idle": "2024-01-18T17:20:02.565331Z",
     "shell.execute_reply": "2024-01-18T17:20:02.565089Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedInputStack(InputStack):\n",
    "    def in_current_record(self, val):\n",
    "        return any(is_fragment(val, var) for var in self.inputs[-1].values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.567170Z",
     "iopub.status.busy": "2024-01-18T17:20:02.566876Z",
     "iopub.status.idle": "2024-01-18T17:20:02.568548Z",
     "shell.execute_reply": "2024-01-18T17:20:02.568342Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedInputStack(TaintedInputStack):\n",
    "    def ignored(self, val):\n",
    "        return not isinstance(val, ostr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "We then hook in the `TaintedInputStack` to the grammar mining infrastructure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.570451Z",
     "iopub.status.busy": "2024-01-18T17:20:02.570116Z",
     "iopub.status.idle": "2024-01-18T17:20:02.571912Z",
     "shell.execute_reply": "2024-01-18T17:20:02.571674Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedScopedVars(ScopedVars):\n",
    "    def create_call_stack(self, i):\n",
    "        return TaintedInputStack(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.573538Z",
     "iopub.status.busy": "2024-01-18T17:20:02.573359Z",
     "iopub.status.idle": "2024-01-18T17:20:02.575167Z",
     "shell.execute_reply": "2024-01-18T17:20:02.574955Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedScopeTracker(ScopeTracker):\n",
    "    def create_assignments(self, *args):\n",
    "        return TaintedScopedVars(*args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.577217Z",
     "iopub.status.busy": "2024-01-18T17:20:02.576934Z",
     "iopub.status.idle": "2024-01-18T17:20:02.579197Z",
     "shell.execute_reply": "2024-01-18T17:20:02.578949Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedScopeTreeMiner(ScopeTreeMiner):\n",
    "    def string_part_of_value(self, part, value):\n",
    "        return is_fragment(part, value)\n",
    "    \n",
    "    def partition(self, part, value):\n",
    "        begin = value.origin.index(part.origin[0])\n",
    "        end = value.origin.index(part.origin[-1])+1\n",
    "        return value[:begin], value[begin:end], value[end:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "<!-- **Advanced.** The *dynamic taint* approach is limited in that it can not observe implicit flows. For example, consider the fragment below.\n",
    "\n",
    "```python\n",
    "if my_fragment == 'begin':\n",
    "    return 'begin'\n",
    "```\n",
    "\n",
    "In this case, we lose track of the string `begin` that is returned even though it is dependent on the value of `my_fragment`. For such cases, a better (but costly) alternative is to rely on concolic execution and capture the constraints as it relates to input characters on each variable.\n",
    "\n",
    "The chapter on [concolic fuzzing](ConcolicFuzzer.ipynb) details how to incorporate concolic symbolic execution to program execution. Can you update our grammar miner to use *concolic execution* to track taints instead?\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.580848Z",
     "iopub.status.busy": "2024-01-18T17:20:02.580618Z",
     "iopub.status.idle": "2024-01-18T17:20:02.582636Z",
     "shell.execute_reply": "2024-01-18T17:20:02.582391Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class TaintedScopedGrammarMiner(ScopedGrammarMiner):\n",
    "    def create_tracker(self, *args):\n",
    "        return TaintedScopeTracker(*args)\n",
    "\n",
    "    def create_tree_miner(self, *args):\n",
    "        return TaintedScopeTreeMiner(*args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Finally, we define `recover_grammar_with_taints()` to recover the grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.584308Z",
     "iopub.status.busy": "2024-01-18T17:20:02.584152Z",
     "iopub.status.idle": "2024-01-18T17:20:02.586410Z",
     "shell.execute_reply": "2024-01-18T17:20:02.586158Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "def recover_grammar_with_taints(fn, inputs, **kwargs):\n",
    "    miner = TaintedScopedGrammarMiner()\n",
    "    for inputstr in inputs:\n",
    "        with Tracer(ostr(inputstr), **kwargs) as tracer:\n",
    "            fn(tracer.my_input)\n",
    "        miner.update_grammar(tracer.my_input, tracer.trace)\n",
    "    return readable(miner.clean_grammar())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Here is how one can use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.588229Z",
     "iopub.status.busy": "2024-01-18T17:20:02.587918Z",
     "iopub.status.idle": "2024-01-18T17:20:02.632823Z",
     "shell.execute_reply": "2024-01-18T17:20:02.632549Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "inventory_grammar = recover_grammar_with_taints(\n",
    "    process_inventory, [INVENTORY],\n",
    "    methods=[\n",
    "        'process_inventory', 'process_vehicle', 'process_car', 'process_van'\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.634678Z",
     "iopub.status.busy": "2024-01-18T17:20:02.634592Z",
     "iopub.status.idle": "2024-01-18T17:20:02.650063Z",
     "shell.execute_reply": "2024-01-18T17:20:02.649814Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 395.0 62\" width=\"395.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M355.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M335.0 31h0.0\"/><rect height=\"22\" width=\"275.0\" x=\"60.0\" y=\"20\"/><text x=\"197.5\" y=\"35\">process_inventory@22:inventory</text></g></g><path d=\"M335.0 31h20\"/></g><path d=\"M 355.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_inventory@22:inventory\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 980.0 62\" width=\"980.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M940.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M301.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"60.0\" y=\"20\"/><text x=\"180.5\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M301.0 31h10\"/><path d=\"M311.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M321.0 31h0.0\"/><path d=\"M349.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"321.0\" y=\"20\"/><text x=\"335.25\" y=\"35\">\n",
       "</text></g><path d=\"M349.5 31h10\"/><path d=\"M359.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M369.5 31h0.0\"/><path d=\"M610.5 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"369.5\" y=\"20\"/><text x=\"490.0\" y=\"35\">process_vehicle@29:vehicle</text></g><path d=\"M610.5 31h10\"/><path d=\"M620.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M630.5 31h0.0\"/><path d=\"M659.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"630.5\" y=\"20\"/><text x=\"644.75\" y=\"35\">\n",
       "</text></g><path d=\"M659.0 31h10\"/><path d=\"M669.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M679.0 31h0.0\"/><path d=\"M920.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"679.0\" y=\"20\"/><text x=\"799.5\" y=\"35\">process_vehicle@29:vehicle</text></g></g><path d=\"M920.0 31h20\"/></g><path d=\"M 940.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@29:vehicle\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 1221.5 62\" width=\"1221.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M1181.5 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M275.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"60.0\" y=\"20\"/><text x=\"167.75\" y=\"35\">process_vehicle@30:year</text></g><path d=\"M275.5 31h10\"/><path d=\"M285.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M295.5 31h0.0\"/><path d=\"M324.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"295.5\" y=\"20\"/><text x=\"309.75\" y=\"35\">,</text></g><path d=\"M324.0 31h10\"/><path d=\"M334.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M344.0 31h0.0\"/><path d=\"M559.5 31h0.0\"/><rect height=\"22\" width=\"215.5\" x=\"344.0\" y=\"20\"/><text x=\"451.75\" y=\"35\">process_vehicle@30:kind</text></g><path d=\"M559.5 31h10\"/><path d=\"M569.5 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M579.5 31h0.0\"/><path d=\"M608.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"579.5\" y=\"20\"/><text x=\"593.75\" y=\"35\">,</text></g><path d=\"M608.0 31h10\"/><path d=\"M618.0 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M628.0 31h0.0\"/><path d=\"M869.0 31h0.0\"/><rect height=\"22\" width=\"241.0\" x=\"628.0\" y=\"20\"/><text x=\"748.5\" y=\"35\">process_vehicle@30:company</text></g><path d=\"M869.0 31h10\"/><path d=\"M879.0 31h10\"/><g class=\"terminal\">\n",
       "<path d=\"M889.0 31h0.0\"/><path d=\"M917.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"28.5\" x=\"889.0\" y=\"20\"/><text x=\"903.25\" y=\"35\">,</text></g><path d=\"M917.5 31h10\"/><path d=\"M927.5 31h10\"/><g class=\"non-terminal\">\n",
       "<path d=\"M937.5 31h0.0\"/><path d=\"M1161.5 31h0.0\"/><rect height=\"22\" width=\"224.0\" x=\"937.5\" y=\"20\"/><text x=\"1049.5\" y=\"35\">process_vehicle@30:model</text></g></g><path d=\"M1161.5 31h20\"/></g><path d=\"M 1181.5 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 301.5 92\" width=\"301.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M261.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M241.5 31h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"20\"/><text x=\"150.75\" y=\"35\">process_car@49:year</text></g></g><path d=\"M241.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M241.5 61h0.0\"/><rect height=\"22\" width=\"181.5\" x=\"60.0\" y=\"50\"/><text x=\"150.75\" y=\"65\">process_van@40:year</text></g></g><path d=\"M241.5 61h20\"/></g><path d=\"M 261.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">1997</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:kind\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 165.5 92\" width=\"165.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M125.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M105.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"20\"/><text x=\"82.75\" y=\"35\">car</text></g></g><path d=\"M105.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M105.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"45.5\" x=\"60.0\" y=\"50\"/><text x=\"82.75\" y=\"65\">van</text></g></g><path d=\"M105.5 61h20\"/></g><path d=\"M 125.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 327.0 92\" width=\"327.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M287.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M267.0 31h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"20\"/><text x=\"163.5\" y=\"35\">process_van@40:company</text></g></g><path d=\"M267.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M267.0 61h0.0\"/><rect height=\"22\" width=\"207.0\" x=\"60.0\" y=\"50\"/><text x=\"163.5\" y=\"65\">process_car@49:company</text></g></g><path d=\"M267.0 61h20\"/></g><path d=\"M 287.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">Ford</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_vehicle@30:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 310.0 92\" width=\"310.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M270.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M250.0 31h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"20\"/><text x=\"155.0\" y=\"35\">process_van@40:model</text></g></g><path d=\"M250.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><g class=\"non-terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M250.0 61h0.0\"/><rect height=\"22\" width=\"190.0\" x=\"60.0\" y=\"50\"/><text x=\"155.0\" y=\"65\">process_car@49:model</text></g></g><path d=\"M250.0 61h20\"/></g><path d=\"M 270.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_van@40:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"62\" viewBox=\"0 0 174.0 62\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 21v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 31h0.0\"/><path d=\"M134.0 31h0.0\"/><path d=\"M40.0 31h20\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">E350</text></g></g><path d=\"M114.0 31h20\"/></g><path d=\"M 134.0 31 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:year\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 174.0 92\" width=\"174.0\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M134.0 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M114.0 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"20\"/><text x=\"87.0\" y=\"35\">2000</text></g></g><path d=\"M114.0 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M114.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"54.0\" x=\"60.0\" y=\"50\"/><text x=\"87.0\" y=\"65\">1999</text></g></g><path d=\"M114.0 61h20\"/></g><path d=\"M 134.0 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:company\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 31h0.0\"/><path d=\"M139.5 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"20\"/><text x=\"99.75\" y=\"35\">Mercury</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h8.5\"/><path d=\"M131.0 61h8.5\"/><g class=\"terminal\">\n",
       "<path d=\"M68.5 61h0.0\"/><path d=\"M131.0 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"62.5\" x=\"68.5\" y=\"50\"/><text x=\"99.75\" y=\"65\">Chevy</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_car@49:model\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" class=\"railroad-diagram\" height=\"92\" viewBox=\"0 0 199.5 92\" width=\"199.5\">\n",
       "<g transform=\"translate(.5 .5)\">\n",
       "<style>/* <![CDATA[ */\n",
       "    svg.railroad-diagram {\n",
       "    }\n",
       "    svg.railroad-diagram path {\n",
       "        stroke-width:3;\n",
       "        stroke:black;\n",
       "        fill:white;\n",
       "    }\n",
       "    svg.railroad-diagram text {\n",
       "        font:14px \"Fira Mono\", monospace;\n",
       "        text-anchor:middle;\n",
       "    }\n",
       "    svg.railroad-diagram text.label{\n",
       "        text-anchor:start;\n",
       "    }\n",
       "    svg.railroad-diagram text.comment{\n",
       "        font:italic 12px \"Fira Mono\", monospace;\n",
       "    }\n",
       "    svg.railroad-diagram rect{\n",
       "        stroke-width:2;\n",
       "        stroke:black;\n",
       "        fill:mistyrose;\n",
       "    }\n",
       "\n",
       "/* ]]> */\n",
       "</style><g>\n",
       "<path d=\"M20 51v20m10 -20v20m-10 -10h20\"/></g><g>\n",
       "<path d=\"M40 61h0.0\"/><path d=\"M159.5 61h0.0\"/><path d=\"M40.0 61a10 10 0 0 0 10 -10v-10a10 10 0 0 1 10 -10\"/><g>\n",
       "<path d=\"M60.0 31h4.25\"/><path d=\"M135.25 31h4.25\"/><g class=\"terminal\">\n",
       "<path d=\"M64.25 31h0.0\"/><path d=\"M135.25 31h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"71.0\" x=\"64.25\" y=\"20\"/><text x=\"99.75\" y=\"35\">Cougar</text></g></g><path d=\"M139.5 31a10 10 0 0 1 10 10v10a10 10 0 0 0 10 10\"/><path d=\"M40.0 61h20\"/><g>\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><g class=\"terminal\">\n",
       "<path d=\"M60.0 61h0.0\"/><path d=\"M139.5 61h0.0\"/><rect height=\"22\" rx=\"10\" ry=\"10\" width=\"79.5\" x=\"60.0\" y=\"50\"/><text x=\"99.75\" y=\"65\">Venture</text></g></g><path d=\"M139.5 61h20\"/></g><path d=\"M 159.5 61 h 20 m -10 -10 v 20 m 10 -20 v 20\"/></g></svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "syntax_diagram(inventory_grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-18T17:20:02.651723Z",
     "iopub.status.busy": "2024-01-18T17:20:02.651591Z",
     "iopub.status.idle": "2024-01-18T17:20:02.990854Z",
     "shell.execute_reply": "2024-01-18T17:20:02.990553Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "url_grammar = recover_grammar_with_taints(\n",
    "    url_parse, URLS_X + ['ftp://user4:pass1@host4/?key4=value3'],\n",
    "    methods=['urlsplit', 'urlparse', '_splitnetloc'])"
   ]
  }
 ],
 "metadata": {
  "ipub": {
   "bibliography": "fuzzingbook.bib"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}