{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "toc-hr-collapsed": false
   },
   "source": [
    "# Testing Configurations\n",
    "\n",
    "The behavior of a program is not only governed by its data.  The _configuration_ of a program – that is, the settings that govern the execution of a program on its (regular) input data, as set by options or configuration files – just as well influences behavior, and thus can and should be tested.  In this chapter, we explore how to systematically _test_ and _cover_ software configurations.  By _automatically inferring configuration options_, we can apply these techniques out of the box, with no need for writing a grammar.  Finally, we show how to systematically cover _combinations_ of configuration options, quickly detecting unwanted interferences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:08.892731Z",
     "iopub.status.busy": "2023-01-07T14:21:08.892346Z",
     "iopub.status.idle": "2023-01-07T14:21:08.949393Z",
     "shell.execute_reply": "2023-01-07T14:21:08.949672Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"640\"\n",
       "            height=\"360\"\n",
       "            src=\"https://www.youtube-nocookie.com/embed/L0ztoXVru2U\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x1052a35b0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bookutils import YouTubeVideo\n",
    "YouTubeVideo('L0ztoXVru2U')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Prerequisites**\n",
    "\n",
    "* You should have read the [chapter on grammars](Grammars.ipynb).\n",
    "* You should have read the [chapter on grammar coverage](GrammarCoverageFuzzer.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:08.951919Z",
     "iopub.status.busy": "2023-01-07T14:21:08.951609Z",
     "iopub.status.idle": "2023-01-07T14:21:08.953036Z",
     "shell.execute_reply": "2023-01-07T14:21:08.953398Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import bookutils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:08.955750Z",
     "iopub.status.busy": "2023-01-07T14:21:08.954932Z",
     "iopub.status.idle": "2023-01-07T14:21:08.957139Z",
     "shell.execute_reply": "2023-01-07T14:21:08.957337Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from typing import List, Union, Optional, Callable, Type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "## Synopsis\n",
    "<!-- Automatically generated. Do not edit. -->\n",
    "\n",
    "To [use the code provided in this chapter](Importing.ipynb), write\n",
    "\n",
    "```python\n",
    ">>> from fuzzingbook.ConfigurationFuzzer import <identifier>\n",
    "```\n",
    "\n",
    "and then make use of the following features.\n",
    "\n",
    "\n",
    "This chapter provides two classes:\n",
    "\n",
    "* `OptionRunner` automatically extract command-line options from a Python program;\n",
    "* `OptionFuzzer` uses these to automatically test a Python program with a large variety of options.\n",
    "\n",
    "`OptionRunner` runs a program up to the point where it parses its arguments, and then extracts a grammar that describes its invocations:\n",
    "\n",
    "```python\n",
    ">>> autopep8_runner = OptionRunner(\"autopep8\", \"foo.py\")\n",
    "```\n",
    "The grammar can be extracted via the method `ebnf_grammar()`:\n",
    "\n",
    "```python\n",
    ">>> option_ebnf_grammar = autopep8_runner.ebnf_grammar()\n",
    ">>> option_ebnf_grammar\n",
    "{'<start>': ['(<option>)*<arguments>'],\n",
    " '<option>': [' -h',\n",
    "  ' --help',\n",
    "  ' --version',\n",
    "  ' -v',\n",
    "  ' --verbose',\n",
    "  ' -d',\n",
    "  ' --diff',\n",
    "  ' -i',\n",
    "  ' --in-place',\n",
    "  ' --global-config <filename>',\n",
    "  ' --ignore-local-config',\n",
    "  ' -r',\n",
    "  ' --recursive',\n",
    "  ' -j <n>',\n",
    "  ' --jobs <n>',\n",
    "  ' -p <n>',\n",
    "  ' --pep8-passes <n>',\n",
    "  ' -a',\n",
    "  ' --aggressive',\n",
    "  ' --experimental',\n",
    "  ' --exclude <globs>',\n",
    "  ' --list-fixes',\n",
    "  ' --ignore <errors>',\n",
    "  ' --select <errors>',\n",
    "  ' --max-line-length <n>',\n",
    "  ' --line-range <line> <line>',\n",
    "  ' --range <line> <line>',\n",
    "  ' --indent-size <int>',\n",
    "  ' --hang-closing',\n",
    "  ' --exit-code'],\n",
    " '<arguments>': [' foo.py'],\n",
    " '<str>': ['<char>+'],\n",
    " '<char>': ['0',\n",
    "  '1',\n",
    "  '2',\n",
    "  '3',\n",
    "  '4',\n",
    "  '5',\n",
    "  '6',\n",
    "  '7',\n",
    "  '8',\n",
    "  '9',\n",
    "  'a',\n",
    "  'b',\n",
    "  'c',\n",
    "  'd',\n",
    "  'e',\n",
    "  'f',\n",
    "  'g',\n",
    "  'h',\n",
    "  'i',\n",
    "  'j',\n",
    "  'k',\n",
    "  'l',\n",
    "  'm',\n",
    "  'n',\n",
    "  'o',\n",
    "  'p',\n",
    "  'q',\n",
    "  'r',\n",
    "  's',\n",
    "  't',\n",
    "  'u',\n",
    "  'v',\n",
    "  'w',\n",
    "  'x',\n",
    "  'y',\n",
    "  'z',\n",
    "  'A',\n",
    "  'B',\n",
    "  'C',\n",
    "  'D',\n",
    "  'E',\n",
    "  'F',\n",
    "  'G',\n",
    "  'H',\n",
    "  'I',\n",
    "  'J',\n",
    "  'K',\n",
    "  'L',\n",
    "  'M',\n",
    "  'N',\n",
    "  'O',\n",
    "  'P',\n",
    "  'Q',\n",
    "  'R',\n",
    "  'S',\n",
    "  'T',\n",
    "  'U',\n",
    "  'V',\n",
    "  'W',\n",
    "  'X',\n",
    "  'Y',\n",
    "  'Z',\n",
    "  '!',\n",
    "  '\"',\n",
    "  '#',\n",
    "  '$',\n",
    "  '%',\n",
    "  '&',\n",
    "  \"'\",\n",
    "  '(',\n",
    "  ')',\n",
    "  '*',\n",
    "  '+',\n",
    "  ',',\n",
    "  '-',\n",
    "  '.',\n",
    "  '/',\n",
    "  ':',\n",
    "  ';',\n",
    "  '<',\n",
    "  '=',\n",
    "  '>',\n",
    "  '?',\n",
    "  '@',\n",
    "  '[',\n",
    "  '\\\\',\n",
    "  ']',\n",
    "  '^',\n",
    "  '_',\n",
    "  '`',\n",
    "  '{',\n",
    "  '|',\n",
    "  '}',\n",
    "  '~'],\n",
    " '<filename>': ['<str>'],\n",
    " '<int>': ['(-)?<digit>+'],\n",
    " '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],\n",
    " '<n>': ['<int>'],\n",
    " '<globs>': ['<str>'],\n",
    " '<errors>': ['<str>'],\n",
    " '<line>': ['<int>']}\n",
    "```\n",
    "The grammar can be immediately used for fuzzing. A `GrammarCoverageFuzzer` will ensure all options are covered:\n",
    "\n",
    "```python\n",
    ">>> from Grammars import convert_ebnf_grammar\n",
    ">>> fuzzer = GrammarCoverageFuzzer(convert_ebnf_grammar(option_ebnf_grammar))\n",
    ">>> [fuzzer.fuzz() for i in range(3)]\n",
    "[' foo.py',\n",
    " ' --verbose --ignore-local-config --line-range -1 78 --jobs -43 --pep8-passes 6 -i --indent-size 2 --range -59330 -0 -r --diff --exit-code --global-config KC --aggressive --recursive --ignore 37~ --help --experimental -v -p 7 --hang-closing --exclude t -d -h --in-place --version -a --list-fixes --max-line-length 4 --select e* -j 635 --select 5ulP --ignore G --global-config -E --ignore q --exclude 2^ --global-config \"r --select z --ignore F6 --exclude Xd --exclude ,8 --global-config A --ignore L --ignore Q --global-config 4 --ignore O= --ignore /.m[ --ignore Dy --ignore :s --ignore v --exclude R --ignore B< --exclude $ --select ko --global-config N --global-config @ --ignore hZU% --global-config 0) --exclude n!JT --ignore ? --exclude #&M --select ; --exclude \\\\ --ignore 1 --global-config |I --select > --global-config pb --select V --global-config 9 --global-config Hcwg{Sj --global-config ` --global-config i}W --ignore x --select _ --select ](\\'f+6 --select a]Y --exclude * --global-config z --ignore-local-config --aggressive foo.py',\n",
    " ' foo.py']\n",
    "```\n",
    "The `OptionFuzzer` class summarizes these steps.  Its constructor takes an `OptionRunner` to automatically extract the grammar; it does the necessary steps to extract the grammar and fuzz with it.\n",
    "\n",
    "```python\n",
    ">>> autopep8_runner = OptionRunner(\"autopep8\", \"foo.py\")\n",
    ">>> autopep8_fuzzer = OptionFuzzer(autopep8_runner)\n",
    ">>> [autopep8_fuzzer.fuzz() for i in range(3)]\n",
    "[' foo.py',\n",
    " \" -a -d -i --help --verbose --exclude + --in-place --max-line-length -29 --line-range -7 3 -v --aggressive --indent-size -01 --jobs 6489550 --global-config !KGD'V -r -j 106 --list-fixes --hang-closing foo.py\",\n",
    " ' --exit-code -h --recursive --pep8-passes -0 --ignore ) --ignore-local-config --diff --experimental --version --range 3 -542 -p 56 --select k --diff -h --diff foo.py']\n",
    "```\n",
    "The final step in testing would now to invoke the program with these arguments.\n",
    "\n",
    "Note that `OptionRunner` is experimental: It assumes that the Python program in question uses the `argparse` module; and not all `argparse` features are supported.  Still, it does a pretty good job even on nontrivial programs.\n",
    "\n",
    "The `OptionRunner` constructor accepts an additional `miner` keyword parameter, which takes the class of the argument grammar miner to be used. By default, this is `OptionGrammarMiner` – a helper class that can be used (and extended) to create own option grammar miners.\n",
    "\n",
    "![](PICS/ConfigurationFuzzer-synopsis-1.svg)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Configuration Options\n",
    "\n",
    "When we talk about the input to a program, we usually think of the _data_ it processes. This is also what we have been fuzzing in the past chapters – be it with [random input](Fuzzer.ipynb), [mutation-based fuzzing](MutationFuzzer.ipynb), or [grammar-based fuzzing](GrammarFuzzer.ipynb).  However, programs typically have several input sources, all of which can and should be tested – and included in test generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One important source of input is the program's _configuration_ – that is, a set of inputs that typically is set once when beginning to process data and then stays constant while processing data, while the program is running, or even while the program is deployed.  Such a configuration is frequently set in _configuration files_ (for instance, as key/value pairs); the most ubiquitous method for command-line tools, though, are _configuration options_ on the command line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As an example, consider the `grep` utility to find textual patterns in files.  The exact mode by which `grep` works is governed by a multitude of options, which can be listed by providing a `--help` option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:08.959743Z",
     "iopub.status.busy": "2023-01-07T14:21:08.959420Z",
     "iopub.status.idle": "2023-01-07T14:21:09.073687Z",
     "shell.execute_reply": "2023-01-07T14:21:09.073925Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]\r\n",
      "\t[-e pattern] [-f file] [--binary-files=value] [--color=when]\r\n",
      "\t[--context[=num]] [--directories=action] [--label] [--line-buffered]\r\n",
      "\t[--null] [pattern] [file ...]\r\n"
     ]
    }
   ],
   "source": [
    "!grep --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "All these options need to be tested for whether they operate correctly.  In security testing, any such option may also trigger a yet unknown vulnerability.  Hence, such options can become _fuzz targets_ on their own.  In this chapter, we analyze how to systematically test such options – and better yet, how to extract possible configurations right out of given program files, such that we do not have to specify anything."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Options in Python\n",
    "\n",
    "Let us stick to our common programming language here and examine how options are processed in Python.  The `argparse` module provides a parser for command-line arguments (and options) with great functionality – and great complexity.  You start by defining a parser (`argparse.ArgumentParser()`) to which individual arguments with various features are added, one after another.  Additional parameters for each argument can specify the type (`type`) of the argument (say, integers or strings), or the number of arguments (`nargs`)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "By default, arguments are stored under their name in the `args` object coming from `parse_args()` – thus, `args.integers` holds the `integer` arguments added earlier.  Special actions (`actions`) allow storing specific values in given variables; the `store_const` action stores the given `const` in the attribute named by `dest`.  The following example takes a number of integer arguments (`integers`) as well as an operator (`--sum`, `--min`, or `--max`) to be applied on these integers.  The operators all store a function reference in the `accumulate` attribute, which is finally invoked on the integers parsed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.077133Z",
     "iopub.status.busy": "2023-01-07T14:21:09.076537Z",
     "iopub.status.idle": "2023-01-07T14:21:09.078084Z",
     "shell.execute_reply": "2023-01-07T14:21:09.078437Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import argparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.082505Z",
     "iopub.status.busy": "2023-01-07T14:21:09.082018Z",
     "iopub.status.idle": "2023-01-07T14:21:09.083609Z",
     "shell.execute_reply": "2023-01-07T14:21:09.083891Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def process_numbers(args=[]):\n",
    "    parser = argparse.ArgumentParser(description='Process some integers.')\n",
    "    parser.add_argument('integers', metavar='N', type=int, nargs='+',\n",
    "                        help='an integer for the accumulator')\n",
    "    group = parser.add_mutually_exclusive_group(required=True)\n",
    "    group.add_argument('--sum', dest='accumulate', action='store_const',\n",
    "                       const=sum,\n",
    "                       help='sum the integers')\n",
    "    group.add_argument('--min', dest='accumulate', action='store_const',\n",
    "                       const=min,\n",
    "                       help='compute the minimum')\n",
    "    group.add_argument('--max', dest='accumulate', action='store_const',\n",
    "                       const=max,\n",
    "                       help='compute the maximum')\n",
    "\n",
    "    args = parser.parse_args(args)\n",
    "    print(args.accumulate(args.integers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here's how `process_numbers()` works.  We can, for instance, invoke the `--min` option on the given arguments to compute the minimum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.086934Z",
     "iopub.status.busy": "2023-01-07T14:21:09.086439Z",
     "iopub.status.idle": "2023-01-07T14:21:09.088355Z",
     "shell.execute_reply": "2023-01-07T14:21:09.088578Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n"
     ]
    }
   ],
   "source": [
    "process_numbers([\"--min\", \"100\", \"200\", \"300\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Or compute the sum of three numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.091252Z",
     "iopub.status.busy": "2023-01-07T14:21:09.090875Z",
     "iopub.status.idle": "2023-01-07T14:21:09.092274Z",
     "shell.execute_reply": "2023-01-07T14:21:09.092528Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n"
     ]
    }
   ],
   "source": [
    "process_numbers([\"--sum\", \"1\", \"2\", \"3\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "When defined via `add_mutually_exclusive_group()` (as above), options are mutually exclusive.  Consequently, we can have only one operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.094791Z",
     "iopub.status.busy": "2023-01-07T14:21:09.094471Z",
     "iopub.status.idle": "2023-01-07T14:21:09.095964Z",
     "shell.execute_reply": "2023-01-07T14:21:09.096198Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import bookutils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.124889Z",
     "iopub.status.busy": "2023-01-07T14:21:09.124388Z",
     "iopub.status.idle": "2023-01-07T14:21:09.130334Z",
     "shell.execute_reply": "2023-01-07T14:21:09.130671Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from ExpectError import ExpectError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.133199Z",
     "iopub.status.busy": "2023-01-07T14:21:09.132767Z",
     "iopub.status.idle": "2023-01-07T14:21:09.137414Z",
     "shell.execute_reply": "2023-01-07T14:21:09.137616Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "usage: ipykernel_launcher.py [-h] (--sum | --min | --max) N [N ...]\n",
      "ipykernel_launcher.py: error: argument --max: not allowed with argument --sum\n",
      "SystemExit: 2 (expected)\n"
     ]
    }
   ],
   "source": [
    "with ExpectError(SystemExit, print_traceback=False):\n",
    "    process_numbers([\"--sum\", \"--max\", \"1\", \"2\", \"3\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A Grammar for Configurations\n",
    "\n",
    "How can we test a system with several options?  The easiest answer is to write a grammar for it.  The grammar `PROCESS_NUMBERS_EBNF_GRAMMAR` reflects the possible combinations of options and arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.139703Z",
     "iopub.status.busy": "2023-01-07T14:21:09.139413Z",
     "iopub.status.idle": "2023-01-07T14:21:09.602302Z",
     "shell.execute_reply": "2023-01-07T14:21:09.602520Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import crange, srange, convert_ebnf_grammar, extend_grammar, is_valid_grammar\n",
    "from Grammars import START_SYMBOL, new_symbol, Grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.605372Z",
     "iopub.status.busy": "2023-01-07T14:21:09.605040Z",
     "iopub.status.idle": "2023-01-07T14:21:09.606408Z",
     "shell.execute_reply": "2023-01-07T14:21:09.606613Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "PROCESS_NUMBERS_EBNF_GRAMMAR: Grammar = {\n",
    "    \"<start>\": [\"<operator> <integers>\"],\n",
    "    \"<operator>\": [\"--sum\", \"--min\", \"--max\"],\n",
    "    \"<integers>\": [\"<integer>\", \"<integers> <integer>\"],\n",
    "    \"<integer>\": [\"<digit>+\"],\n",
    "    \"<digit>\": crange('0', '9')\n",
    "}\n",
    "\n",
    "assert is_valid_grammar(PROCESS_NUMBERS_EBNF_GRAMMAR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.608614Z",
     "iopub.status.busy": "2023-01-07T14:21:09.608307Z",
     "iopub.status.idle": "2023-01-07T14:21:09.609789Z",
     "shell.execute_reply": "2023-01-07T14:21:09.609546Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "PROCESS_NUMBERS_GRAMMAR = convert_ebnf_grammar(PROCESS_NUMBERS_EBNF_GRAMMAR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can feed this grammar into our [grammar coverage fuzzer](GrammarCoverageFuzzer.ipynb) and have it cover one option after another:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.611477Z",
     "iopub.status.busy": "2023-01-07T14:21:09.611194Z",
     "iopub.status.idle": "2023-01-07T14:21:09.820652Z",
     "shell.execute_reply": "2023-01-07T14:21:09.820903Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from GrammarCoverageFuzzer import GrammarCoverageFuzzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.836052Z",
     "iopub.status.busy": "2023-01-07T14:21:09.835710Z",
     "iopub.status.idle": "2023-01-07T14:21:09.837148Z",
     "shell.execute_reply": "2023-01-07T14:21:09.837347Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--max 9 5 8 210 80 9756431\n",
      "--sum 9 4 99 1245 612370\n",
      "--min 2 3 0 46 15798 7570926\n"
     ]
    }
   ],
   "source": [
    "f = GrammarCoverageFuzzer(PROCESS_NUMBERS_GRAMMAR, min_nonterminals=10)\n",
    "for i in range(3):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Of course, we can also invoke `process_numbers()` with these very arguments. To this end, we need to convert the string produced by the grammar back into a list of individual arguments, using `split()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.858405Z",
     "iopub.status.busy": "2023-01-07T14:21:09.850919Z",
     "iopub.status.idle": "2023-01-07T14:21:09.860041Z",
     "shell.execute_reply": "2023-01-07T14:21:09.860240Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['--max', '8', '9', '3067', '44', '13852967057']\n",
      "13852967057\n",
      "['--sum', '9', '8', '63', '9278111', '59206197798']\n",
      "59215475989\n",
      "['--min', '4', '1', '4864', '33342', '7827970808951']\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "f = GrammarCoverageFuzzer(PROCESS_NUMBERS_GRAMMAR, min_nonterminals=10)\n",
    "for i in range(3):\n",
    "    args = f.fuzz().split()\n",
    "    print(args)\n",
    "    process_numbers(args)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Similarly, we can define grammars for any program to be tested; as well as define grammars for, say, configuration files.  Yet, the grammar has to be updated with every change to the program, which creates a maintenance burden.  Given that the information required for the grammar is already all encoded in the program, the question arises: _Can't we go and extract configuration options right out of the program in the first place?_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Mining Configuration Options\n",
    "\n",
    "In this section, we try to extract option and argument information right out of a program, such that we do not have to specify a configuration grammar.  The aim is to have a configuration fuzzer that works on the options and arguments of an arbitrary program, as long as it follows specific conventions for processing its arguments.  In the case of Python programs, this means using the `argparse` module.\n",
    "\n",
    "Our idea is as follows: We execute the given program up to the point where the arguments are actually parsed – that is, `argparse.parse_args()` is invoked.  Up to this point, we track all calls into the argument parser, notably those calls that define arguments and options (`add_argument()`).  From these, we construct the grammar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Tracking Arguments\n",
    "\n",
    "Let us illustrate this approach with a simple experiment: We define a trace function (see [our chapter on coverage](Coverage.ipynb) for details) that is active while `process_numbers` is invoked.  If we have a call to a method `add_argument`, we access and print out the local variables (which at this point are the arguments to the method)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.862504Z",
     "iopub.status.busy": "2023-01-07T14:21:09.862188Z",
     "iopub.status.idle": "2023-01-07T14:21:09.863534Z",
     "shell.execute_reply": "2023-01-07T14:21:09.863783Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.865778Z",
     "iopub.status.busy": "2023-01-07T14:21:09.865437Z",
     "iopub.status.idle": "2023-01-07T14:21:09.866639Z",
     "shell.execute_reply": "2023-01-07T14:21:09.866878Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.869287Z",
     "iopub.status.busy": "2023-01-07T14:21:09.868893Z",
     "iopub.status.idle": "2023-01-07T14:21:09.870150Z",
     "shell.execute_reply": "2023-01-07T14:21:09.870401Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def trace_locals(frame, event, arg):\n",
    "    if event != \"call\":\n",
    "        return\n",
    "    method_name = frame.f_code.co_name\n",
    "    if method_name != \"add_argument\":\n",
    "        return\n",
    "    locals = frame.f_locals\n",
    "    print(method_name, locals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "What we get is a list of all calls to `add_argument()`, together with the method arguments passed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.873061Z",
     "iopub.status.busy": "2023-01-07T14:21:09.872750Z",
     "iopub.status.idle": "2023-01-07T14:21:09.874394Z",
     "shell.execute_reply": "2023-01-07T14:21:09.874195Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "add_argument {'self': ArgumentParser(prog='ipykernel_launcher.py', usage=None, description='Process some integers.', formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True), 'args': ('-h', '--help'), 'kwargs': {'action': 'help', 'default': '==SUPPRESS==', 'help': 'show this help message and exit'}}\n",
      "add_argument {'self': ArgumentParser(prog='ipykernel_launcher.py', usage=None, description='Process some integers.', formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True), 'args': ('integers',), 'kwargs': {'metavar': 'N', 'type': <class 'int'>, 'nargs': '+', 'help': 'an integer for the accumulator'}}\n",
      "add_argument {'self': <argparse._MutuallyExclusiveGroup object at 0x1058b7280>, 'args': ('--sum',), 'kwargs': {'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function sum>, 'help': 'sum the integers'}}\n",
      "add_argument {'self': <argparse._MutuallyExclusiveGroup object at 0x1058b7280>, 'args': ('--min',), 'kwargs': {'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function min>, 'help': 'compute the minimum'}}\n",
      "add_argument {'self': <argparse._MutuallyExclusiveGroup object at 0x1058b7280>, 'args': ('--max',), 'kwargs': {'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function max>, 'help': 'compute the maximum'}}\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "sys.settrace(trace_locals)\n",
    "process_numbers([\"--sum\", \"1\", \"2\", \"3\"])\n",
    "sys.settrace(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "From the `args` argument, we can access the individual options and arguments to be defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.876656Z",
     "iopub.status.busy": "2023-01-07T14:21:09.876367Z",
     "iopub.status.idle": "2023-01-07T14:21:09.877487Z",
     "shell.execute_reply": "2023-01-07T14:21:09.877712Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def trace_options(frame, event, arg):\n",
    "    if event != \"call\":\n",
    "        return\n",
    "    method_name = frame.f_code.co_name\n",
    "    if method_name != \"add_argument\":\n",
    "        return\n",
    "    locals = frame.f_locals\n",
    "    print(locals['args'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.879972Z",
     "iopub.status.busy": "2023-01-07T14:21:09.879449Z",
     "iopub.status.idle": "2023-01-07T14:21:09.881661Z",
     "shell.execute_reply": "2023-01-07T14:21:09.882004Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('-h', '--help')\n",
      "('integers',)\n",
      "('--sum',)\n",
      "('--min',)\n",
      "('--max',)\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "sys.settrace(trace_options)\n",
    "process_numbers([\"--sum\", \"1\", \"2\", \"3\"])\n",
    "sys.settrace(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We see that each argument comes as a tuple with one (say, `integers` or `--sum`) or two members (`-h` and `--help`), which denote alternate forms for the same option.  Our job will be to go through the arguments of `add_arguments()` and detect not only the names of options and arguments, but also whether they accept additional parameters, as well as the type of the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### A Grammar Miner for Options and Arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let us now build a class that gathers all this information to create a grammar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We use the `ParseInterrupt` exception to interrupt program execution after gathering all arguments and options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.884766Z",
     "iopub.status.busy": "2023-01-07T14:21:09.884346Z",
     "iopub.status.idle": "2023-01-07T14:21:09.885651Z",
     "shell.execute_reply": "2023-01-07T14:21:09.885910Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ParseInterrupt(Exception):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The class `OptionGrammarMiner` takes an executable function for which the grammar of options and arguments is to be mined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.888242Z",
     "iopub.status.busy": "2023-01-07T14:21:09.887943Z",
     "iopub.status.idle": "2023-01-07T14:21:09.889058Z",
     "shell.execute_reply": "2023-01-07T14:21:09.889330Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner:\n",
    "    \"\"\"Helper class for extracting option grammars\"\"\"\n",
    "\n",
    "    def __init__(self, function: Callable, log: bool = False):\n",
    "        \"\"\"Constructor.\n",
    "        `function` - a function processing arguments using argparse()\n",
    "        `log` - output diagnostics if True\n",
    "        \"\"\"\n",
    "        self.function = function\n",
    "        self.log = log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The method `mine_ebnf_grammar()` is where everything happens.  It creates a grammar of the form\n",
    "\n",
    "```\n",
    "<start> ::= <option>* <arguments>\n",
    "<option> ::= <empty>\n",
    "<arguments> ::= <empty>\n",
    "```\n",
    "\n",
    "in which the options and arguments will be collected.  It then sets a trace function (see [our chapter on coverage](Coverage.ipynb) for details) that is active while the previously defined `function` is invoked.  Raising `ParseInterrupt` (when `parse_args()` is invoked) ends execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.892366Z",
     "iopub.status.busy": "2023-01-07T14:21:09.892068Z",
     "iopub.status.idle": "2023-01-07T14:21:09.893179Z",
     "shell.execute_reply": "2023-01-07T14:21:09.893427Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    OPTION_SYMBOL = \"<option>\"\n",
    "    ARGUMENTS_SYMBOL = \"<arguments>\"\n",
    "\n",
    "    def mine_ebnf_grammar(self):\n",
    "        \"\"\"Extract EBNF option grammar\"\"\"\n",
    "        self.grammar = {\n",
    "            START_SYMBOL: [\"(\" + self.OPTION_SYMBOL + \")*\" + self.ARGUMENTS_SYMBOL],\n",
    "            self.OPTION_SYMBOL: [],\n",
    "            self.ARGUMENTS_SYMBOL: []\n",
    "        }\n",
    "        self.current_group = self.OPTION_SYMBOL\n",
    "\n",
    "        old_trace = sys.gettrace()\n",
    "        sys.settrace(self.traceit)\n",
    "        try:\n",
    "            self.function()\n",
    "        except ParseInterrupt:\n",
    "            pass\n",
    "        sys.settrace(old_trace)\n",
    "\n",
    "        return self.grammar\n",
    "\n",
    "    def mine_grammar(self):\n",
    "        \"\"\"Extract BNF option grammar\"\"\"\n",
    "        return convert_ebnf_grammar(self.mine_ebnf_grammar())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The trace function checks for four methods: `add_argument()` is the most important function, resulting in processing arguments; `frame.f_locals` again is the set of local variables, which at this point is mostly the arguments to `add_argument()`.  Since mutually exclusive groups also have a method `add_argument()`, we set the flag `in_group` to differentiate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that we make no specific efforts to differentiate between multiple parsers or groups; we simply assume that there is one parser, and at any point at most one mutually exclusive group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.896489Z",
     "iopub.status.busy": "2023-01-07T14:21:09.896153Z",
     "iopub.status.idle": "2023-01-07T14:21:09.897413Z",
     "shell.execute_reply": "2023-01-07T14:21:09.897606Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def traceit(self, frame, event, arg):\n",
    "        if event != \"call\":\n",
    "            return\n",
    "\n",
    "        if \"self\" not in frame.f_locals:\n",
    "            return\n",
    "\n",
    "        self_var = frame.f_locals[\"self\"]\n",
    "        method_name = frame.f_code.co_name\n",
    "\n",
    "        if method_name == \"add_argument\":\n",
    "            in_group = repr(type(self_var)).find(\"Group\") >= 0\n",
    "            self.process_argument(frame.f_locals, in_group)\n",
    "        elif method_name == \"add_mutually_exclusive_group\":\n",
    "            self.add_group(frame.f_locals, exclusive=True)\n",
    "        elif method_name == \"add_argument_group\":\n",
    "            # self.add_group(frame.f_locals, exclusive=False)\n",
    "            pass\n",
    "        elif method_name == \"parse_args\":\n",
    "            raise ParseInterrupt\n",
    "\n",
    "        return self.traceit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The `process_arguments()` now analyzes the arguments passed and adds them to the grammar:\n",
    "\n",
    "* If the argument starts with `-`, it gets added as an optional element to the `<option>` list\n",
    "* Otherwise, it gets added to the `<argument>` list.\n",
    "\n",
    "The optional `nargs` argument specifies how many arguments can follow.  If it is a number, we add the appropriate number of elements to the grammar; if it is an abstract specifier (say, `+` or `*`), we use it directly as EBNF operator.\n",
    "\n",
    "Given the large number of parameters and optional behavior, this is a somewhat messy function, but it does the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.900088Z",
     "iopub.status.busy": "2023-01-07T14:21:09.899774Z",
     "iopub.status.idle": "2023-01-07T14:21:09.900996Z",
     "shell.execute_reply": "2023-01-07T14:21:09.901317Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def process_argument(self, locals, in_group):\n",
    "        args = locals[\"args\"]\n",
    "        kwargs = locals[\"kwargs\"]\n",
    "\n",
    "        if self.log:\n",
    "            print(args)\n",
    "            print(kwargs)\n",
    "            print()\n",
    "\n",
    "        for arg in args:\n",
    "            self.process_arg(arg, in_group, kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.904934Z",
     "iopub.status.busy": "2023-01-07T14:21:09.904609Z",
     "iopub.status.idle": "2023-01-07T14:21:09.905822Z",
     "shell.execute_reply": "2023-01-07T14:21:09.906020Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def process_arg(self, arg, in_group, kwargs):\n",
    "        if arg.startswith('-'):\n",
    "            if not in_group:\n",
    "                target = self.OPTION_SYMBOL\n",
    "            else:\n",
    "                target = self.current_group\n",
    "            metavar = None\n",
    "            arg = \" \" + arg\n",
    "        else:\n",
    "            target = self.ARGUMENTS_SYMBOL\n",
    "            metavar = arg\n",
    "            arg = \"\"\n",
    "\n",
    "        if \"nargs\" in kwargs:\n",
    "            nargs = kwargs[\"nargs\"]\n",
    "        else:\n",
    "            nargs = 1\n",
    "\n",
    "        param = self.add_parameter(kwargs, metavar)\n",
    "        if param == \"\":\n",
    "            nargs = 0\n",
    "\n",
    "        if isinstance(nargs, int):\n",
    "            for i in range(nargs):\n",
    "                arg += param\n",
    "        else:\n",
    "            assert nargs in \"?+*\"\n",
    "            arg += '(' + param + ')' + nargs\n",
    "\n",
    "        if target == self.OPTION_SYMBOL:\n",
    "            self.grammar[target].append(arg)\n",
    "        else:\n",
    "            self.grammar[target].append(arg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The method `add_parameter()` handles possible parameters of options.  If the argument has an `action` defined, it takes no parameter.  Otherwise, we identify the type of the parameter (as `int` or `str`) and augment the grammar with an appropriate rule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.907900Z",
     "iopub.status.busy": "2023-01-07T14:21:09.907591Z",
     "iopub.status.idle": "2023-01-07T14:21:09.908872Z",
     "shell.execute_reply": "2023-01-07T14:21:09.909070Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import inspect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.911786Z",
     "iopub.status.busy": "2023-01-07T14:21:09.911404Z",
     "iopub.status.idle": "2023-01-07T14:21:09.912865Z",
     "shell.execute_reply": "2023-01-07T14:21:09.913183Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def add_parameter(self, kwargs, metavar):\n",
    "        if \"action\" in kwargs:\n",
    "            # No parameter\n",
    "            return \"\"\n",
    "\n",
    "        type_ = \"str\"\n",
    "        if \"type\" in kwargs:\n",
    "            given_type = kwargs[\"type\"]\n",
    "            # int types come as '<class int>'\n",
    "            if inspect.isclass(given_type) and issubclass(given_type, int):\n",
    "                type_ = \"int\"\n",
    "\n",
    "        if metavar is None:\n",
    "            if \"metavar\" in kwargs:\n",
    "                metavar = kwargs[\"metavar\"]\n",
    "            else:\n",
    "                metavar = type_\n",
    "\n",
    "        self.add_type_rule(type_)\n",
    "        if metavar != type_:\n",
    "            self.add_metavar_rule(metavar, type_)\n",
    "\n",
    "        param = \" <\" + metavar + \">\"\n",
    "\n",
    "        return param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The method `add_type_rule()` adds a rule for parameter types to the grammar.  If the parameter is identified by a meta-variable (say, `N`), we add a rule for this as well to improve legibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.916134Z",
     "iopub.status.busy": "2023-01-07T14:21:09.915818Z",
     "iopub.status.idle": "2023-01-07T14:21:09.917002Z",
     "shell.execute_reply": "2023-01-07T14:21:09.917246Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def add_type_rule(self, type_):\n",
    "        if type_ == \"int\":\n",
    "            self.add_int_rule()\n",
    "        else:\n",
    "            self.add_str_rule()\n",
    "\n",
    "    def add_int_rule(self):\n",
    "        self.grammar[\"<int>\"] = [\"(-)?<digit>+\"]\n",
    "        self.grammar[\"<digit>\"] = crange('0', '9')\n",
    "\n",
    "    def add_str_rule(self):\n",
    "        self.grammar[\"<str>\"] = [\"<char>+\"]\n",
    "        self.grammar[\"<char>\"] = srange(\n",
    "            string.digits\n",
    "            + string.ascii_letters\n",
    "            + string.punctuation)\n",
    "\n",
    "    def add_metavar_rule(self, metavar, type_):\n",
    "        self.grammar[\"<\" + metavar + \">\"] = [\"<\" + type_ + \">\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The method `add_group()` adds a new mutually exclusive group to the grammar.  We define a new symbol (say, `<group>`) for the options added to the group, and use the `required` and `exclusive` flags to define an appropriate expansion operator.  The group is then prefixed to the grammar, as in\n",
    "\n",
    "```\n",
    "<start> ::= <group><option>* <arguments>\n",
    "<group> ::= <empty>\n",
    "```\n",
    "\n",
    "and filled with the next calls to `add_argument()` within the group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.920030Z",
     "iopub.status.busy": "2023-01-07T14:21:09.919730Z",
     "iopub.status.idle": "2023-01-07T14:21:09.921014Z",
     "shell.execute_reply": "2023-01-07T14:21:09.921194Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class OptionGrammarMiner(OptionGrammarMiner):\n",
    "    def add_group(self, locals, exclusive):\n",
    "        kwargs = locals[\"kwargs\"]\n",
    "        if self.log:\n",
    "            print(kwargs)\n",
    "\n",
    "        required = kwargs.get(\"required\", False)\n",
    "        group = new_symbol(self.grammar, \"<group>\")\n",
    "\n",
    "        if required and exclusive:\n",
    "            group_expansion = group\n",
    "        if required and not exclusive:\n",
    "            group_expansion = group + \"+\"\n",
    "        if not required and exclusive:\n",
    "            group_expansion = group + \"?\"\n",
    "        if not required and not exclusive:\n",
    "            group_expansion = group + \"*\"\n",
    "\n",
    "        self.grammar[START_SYMBOL][0] = group_expansion + \\\n",
    "            self.grammar[START_SYMBOL][0]\n",
    "        self.grammar[group] = []\n",
    "        self.current_group = group"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "That's it!  With this, we can now extract the grammar from our `process_numbers()` program.  Turning on logging again reveals the variables we draw upon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.923968Z",
     "iopub.status.busy": "2023-01-07T14:21:09.923647Z",
     "iopub.status.idle": "2023-01-07T14:21:09.925030Z",
     "shell.execute_reply": "2023-01-07T14:21:09.925300Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('-h', '--help')\n",
      "{'action': 'help', 'default': '==SUPPRESS==', 'help': 'show this help message and exit'}\n",
      "\n",
      "('integers',)\n",
      "{'metavar': 'N', 'type': <class 'int'>, 'nargs': '+', 'help': 'an integer for the accumulator'}\n",
      "\n",
      "{'required': True}\n",
      "('--sum',)\n",
      "{'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function sum>, 'help': 'sum the integers'}\n",
      "\n",
      "('--min',)\n",
      "{'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function min>, 'help': 'compute the minimum'}\n",
      "\n",
      "('--max',)\n",
      "{'dest': 'accumulate', 'action': 'store_const', 'const': <built-in function max>, 'help': 'compute the maximum'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "miner = OptionGrammarMiner(process_numbers, log=True)\n",
    "process_numbers_grammar = miner.mine_ebnf_grammar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here is the extracted grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.927576Z",
     "iopub.status.busy": "2023-01-07T14:21:09.927253Z",
     "iopub.status.idle": "2023-01-07T14:21:09.928891Z",
     "shell.execute_reply": "2023-01-07T14:21:09.929226Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': ['<group>(<option>)*<arguments>'],\n",
       " '<option>': [' -h', ' --help'],\n",
       " '<arguments>': ['( <integers>)+'],\n",
       " '<int>': ['(-)?<digit>+'],\n",
       " '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],\n",
       " '<integers>': ['<int>'],\n",
       " '<group>': [' --sum', ' --min', ' --max']}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar properly identifies the group found:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.931310Z",
     "iopub.status.busy": "2023-01-07T14:21:09.930984Z",
     "iopub.status.idle": "2023-01-07T14:21:09.932284Z",
     "shell.execute_reply": "2023-01-07T14:21:09.932597Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<group>(<option>)*<arguments>']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<start>\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.934631Z",
     "iopub.status.busy": "2023-01-07T14:21:09.934334Z",
     "iopub.status.idle": "2023-01-07T14:21:09.935737Z",
     "shell.execute_reply": "2023-01-07T14:21:09.936048Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' --sum', ' --min', ' --max']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<group>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It also identifies a `--help` option provided not by us, but by the `argparse` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.938120Z",
     "iopub.status.busy": "2023-01-07T14:21:09.937811Z",
     "iopub.status.idle": "2023-01-07T14:21:09.939311Z",
     "shell.execute_reply": "2023-01-07T14:21:09.939478Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' -h', ' --help']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<option>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar also correctly identifies the types of the arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.941650Z",
     "iopub.status.busy": "2023-01-07T14:21:09.941220Z",
     "iopub.status.idle": "2023-01-07T14:21:09.943023Z",
     "shell.execute_reply": "2023-01-07T14:21:09.943450Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['( <integers>)+']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<arguments>\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.946318Z",
     "iopub.status.busy": "2023-01-07T14:21:09.945847Z",
     "iopub.status.idle": "2023-01-07T14:21:09.947537Z",
     "shell.execute_reply": "2023-01-07T14:21:09.947783Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<int>']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<integers>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The rules for `int` are set as defined by `add_int_rule()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.950087Z",
     "iopub.status.busy": "2023-01-07T14:21:09.949755Z",
     "iopub.status.idle": "2023-01-07T14:21:09.951163Z",
     "shell.execute_reply": "2023-01-07T14:21:09.951406Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['(-)?<digit>+']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "process_numbers_grammar[\"<int>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can take this grammar and convert it to BNF, such that we can fuzz with it right away:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.953545Z",
     "iopub.status.busy": "2023-01-07T14:21:09.953198Z",
     "iopub.status.idle": "2023-01-07T14:21:09.954495Z",
     "shell.execute_reply": "2023-01-07T14:21:09.954703Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "assert is_valid_grammar(process_numbers_grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:09.956977Z",
     "iopub.status.busy": "2023-01-07T14:21:09.956639Z",
     "iopub.status.idle": "2023-01-07T14:21:09.957724Z",
     "shell.execute_reply": "2023-01-07T14:21:09.957994Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grammar = convert_ebnf_grammar(process_numbers_grammar)\n",
    "assert is_valid_grammar(grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.013850Z",
     "iopub.status.busy": "2023-01-07T14:21:10.002610Z",
     "iopub.status.idle": "2023-01-07T14:21:10.015402Z",
     "shell.execute_reply": "2023-01-07T14:21:10.015665Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " --sum 9\n",
      " --max -h --help --help -16 -0\n",
      " --min --help 2745341 8\n",
      " --min 1 27\n",
      " --sum --help --help -2\n",
      " --sum --help 0 3 -77\n",
      " --sum -3\n",
      " --sum --help 429 8 10 0295 -694 1\n",
      " --max -h 91 -1425 99\n",
      " --sum -795 -94 8 -44\n"
     ]
    }
   ],
   "source": [
    "f = GrammarCoverageFuzzer(grammar)\n",
    "for i in range(10):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Each and every invocation adheres to the rules as set forth in the `argparse` calls.  By mining options and arguments from existing programs, we can now fuzz these options out of the box – without having to specify a grammar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "toc-hr-collapsed": true
   },
   "source": [
    "## Testing Autopep8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let us try out the option grammar miner on real-world Python programs.  `autopep8` is a tool that automatically converts Python code to the [PEP 8 Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/).  (Actually, all Python code in this book runs through `autopep8` during production.)  `autopep8` offers a wide range of options, as can be seen by invoking it with `--help`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.018468Z",
     "iopub.status.busy": "2023-01-07T14:21:10.018007Z",
     "iopub.status.idle": "2023-01-07T14:21:10.203346Z",
     "shell.execute_reply": "2023-01-07T14:21:10.203567Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: autopep8 [-h] [--version] [-v] [-d] [-i] [--global-config filename]\r\n",
      "                [--ignore-local-config] [-r] [-j n] [-p n] [-a]\r\n",
      "                [--experimental] [--exclude globs] [--list-fixes]\r\n",
      "                [--ignore errors] [--select errors] [--max-line-length n]\r\n",
      "                [--line-range line line] [--hang-closing] [--exit-code]\r\n",
      "                [files ...]\r\n",
      "\r\n",
      "Automatically formats Python code to conform to the PEP 8 style guide.\r\n",
      "\r\n",
      "positional arguments:\r\n",
      "  files                 files to format or '-' for standard in\r\n",
      "\r\n",
      "options:\r\n",
      "  -h, --help            show this help message and exit\r\n",
      "  --version             show program's version number and exit\r\n",
      "  -v, --verbose         print verbose messages; multiple -v result in more\r\n",
      "                        verbose messages\r\n",
      "  -d, --diff            print the diff for the fixed source\r\n",
      "  -i, --in-place        make changes to files in place\r\n",
      "  --global-config filename\r\n",
      "                        path to a global pep8 config file; if this file does\r\n",
      "                        not exist then this is ignored (default:\r\n",
      "                        /Users/zeller/.config/pep8)\r\n",
      "  --ignore-local-config\r\n",
      "                        don't look for and apply local config files; if not\r\n",
      "                        passed, defaults are updated with any config files in\r\n",
      "                        the project's root directory\r\n",
      "  -r, --recursive       run recursively over directories; must be used with\r\n",
      "                        --in-place or --diff\r\n",
      "  -j n, --jobs n        number of parallel jobs; match CPU count if value is\r\n",
      "                        less than 1\r\n",
      "  -p n, --pep8-passes n\r\n",
      "                        maximum number of additional pep8 passes (default:\r\n",
      "                        infinite)\r\n",
      "  -a, --aggressive      enable non-whitespace changes; multiple -a result in\r\n",
      "                        more aggressive changes\r\n",
      "  --experimental        enable experimental fixes\r\n",
      "  --exclude globs       exclude file/directory names that match these comma-\r\n",
      "                        separated globs\r\n",
      "  --list-fixes          list codes for fixes; used by --ignore and --select\r\n",
      "  --ignore errors       do not fix these errors/warnings (default:\r\n",
      "                        E226,E24,W50,W690)\r\n",
      "  --select errors       fix only these errors/warnings (e.g. E4,W)\r\n",
      "  --max-line-length n   set maximum allowed line length (default: 79)\r\n",
      "  --line-range line line, --range line line\r\n",
      "                        only fix errors found within this inclusive range of\r\n",
      "                        line numbers (e.g. 1 99); line numbers are indexed at\r\n",
      "                        1\r\n",
      "  --hang-closing        hang-closing option passed to pycodestyle\r\n",
      "  --exit-code           change to behavior of exit code. default behavior of\r\n",
      "                        return value, 0 is no differences, 1 is error exit.\r\n",
      "                        return 2 when add this option. 2 is exists\r\n",
      "                        differences.\r\n"
     ]
    }
   ],
   "source": [
    "!autopep8 --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Autopep8 Setup\n",
    "\n",
    "We want to systematically test these options.  In order to deploy our configuration grammar miner, we need to find the source code of the executable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.205822Z",
     "iopub.status.busy": "2023-01-07T14:21:10.205456Z",
     "iopub.status.idle": "2023-01-07T14:21:10.206686Z",
     "shell.execute_reply": "2023-01-07T14:21:10.206878Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.209020Z",
     "iopub.status.busy": "2023-01-07T14:21:10.208732Z",
     "iopub.status.idle": "2023-01-07T14:21:10.209998Z",
     "shell.execute_reply": "2023-01-07T14:21:10.210196Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def find_executable(name):\n",
    "    for path in os.get_exec_path():\n",
    "        qualified_name = os.path.join(path, name)\n",
    "        if os.path.exists(qualified_name):\n",
    "            return qualified_name\n",
    "    return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.212510Z",
     "iopub.status.busy": "2023-01-07T14:21:10.212174Z",
     "iopub.status.idle": "2023-01-07T14:21:10.213832Z",
     "shell.execute_reply": "2023-01-07T14:21:10.214044Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/zeller/.pyenv/versions/3.10.2/bin/autopep8'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_executable = find_executable(\"autopep8\")\n",
    "assert autopep8_executable is not None\n",
    "autopep8_executable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Next, we build a function that reads the contents of the file and executes it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.216292Z",
     "iopub.status.busy": "2023-01-07T14:21:10.215960Z",
     "iopub.status.idle": "2023-01-07T14:21:10.217419Z",
     "shell.execute_reply": "2023-01-07T14:21:10.217601Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def autopep8():\n",
    "    executable = find_executable(\"autopep8\")\n",
    "\n",
    "    # First line has to contain \"/usr/bin/env python\" or like\n",
    "    first_line = open(executable).readline()\n",
    "    assert first_line.find(\"python\") >= 0\n",
    "\n",
    "    contents = open(executable).read()\n",
    "    exec(contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Mining an Autopep8 Grammar\n",
    "\n",
    "We can use the `autopep8()` function in our grammar miner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.219654Z",
     "iopub.status.busy": "2023-01-07T14:21:10.219296Z",
     "iopub.status.idle": "2023-01-07T14:21:10.220654Z",
     "shell.execute_reply": "2023-01-07T14:21:10.220447Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_miner = OptionGrammarMiner(autopep8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "and extract a grammar for it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.222674Z",
     "iopub.status.busy": "2023-01-07T14:21:10.222363Z",
     "iopub.status.idle": "2023-01-07T14:21:10.246894Z",
     "shell.execute_reply": "2023-01-07T14:21:10.247069Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_ebnf_grammar = autopep8_miner.mine_ebnf_grammar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This works because here, `autopep8` is not a separate process (and a separate Python interpreter), but we run the `autopep8()` function (and the `autopep8` code) in our current Python interpreter – up to the call to `parse_args()`, where we interrupt execution again.  At this point, the `autopep8` code has done nothing but setting up the argument parser – which is what we are interested in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar options mined reflect precisely the options seen when providing `--help`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.249067Z",
     "iopub.status.busy": "2023-01-07T14:21:10.248692Z",
     "iopub.status.idle": "2023-01-07T14:21:10.250204Z",
     "shell.execute_reply": "2023-01-07T14:21:10.250434Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' -h', ' --help', ' --version', ' -v', ' --verbose', ' -d', ' --diff', ' -i', ' --in-place', ' --global-config <filename>', ' --ignore-local-config', ' -r', ' --recursive', ' -j <n>', ' --jobs <n>', ' -p <n>', ' --pep8-passes <n>', ' -a', ' --aggressive', ' --experimental', ' --exclude <globs>', ' --list-fixes', ' --ignore <errors>', ' --select <errors>', ' --max-line-length <n>', ' --line-range <line> <line>', ' --range <line> <line>', ' --indent-size <int>', ' --hang-closing', ' --exit-code']\n"
     ]
    }
   ],
   "source": [
    "print(autopep8_ebnf_grammar[\"<option>\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Metavariables like `<n>` or `<line>` are placeholders for integers.  We assume all metavariables of the same name have the same type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.252386Z",
     "iopub.status.busy": "2023-01-07T14:21:10.252083Z",
     "iopub.status.idle": "2023-01-07T14:21:10.254067Z",
     "shell.execute_reply": "2023-01-07T14:21:10.253855Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<int>']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_ebnf_grammar[\"<line>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The grammar miner has inferred that the argument to `autopep8` is a list of files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.256061Z",
     "iopub.status.busy": "2023-01-07T14:21:10.255745Z",
     "iopub.status.idle": "2023-01-07T14:21:10.257569Z",
     "shell.execute_reply": "2023-01-07T14:21:10.257882Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['( <files>)*']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_ebnf_grammar[\"<arguments>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "which in turn all are strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.260300Z",
     "iopub.status.busy": "2023-01-07T14:21:10.259984Z",
     "iopub.status.idle": "2023-01-07T14:21:10.261663Z",
     "shell.execute_reply": "2023-01-07T14:21:10.261927Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<str>']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_ebnf_grammar[\"<files>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As we are only interested in testing options, not arguments, we fix the arguments to a single mandatory input.  (Otherwise, we'd have plenty of random file names generated.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.264087Z",
     "iopub.status.busy": "2023-01-07T14:21:10.263197Z",
     "iopub.status.idle": "2023-01-07T14:21:10.265235Z",
     "shell.execute_reply": "2023-01-07T14:21:10.265578Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_ebnf_grammar[\"<arguments>\"] = [\" <files>\"]\n",
    "autopep8_ebnf_grammar[\"<files>\"] = [\"foo.py\"]\n",
    "assert is_valid_grammar(autopep8_ebnf_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Creating Autopep8 Options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let us now use the inferred grammar for fuzzing.  Again, we convert the EBNF grammar into a regular BNF grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.267949Z",
     "iopub.status.busy": "2023-01-07T14:21:10.267642Z",
     "iopub.status.idle": "2023-01-07T14:21:10.268810Z",
     "shell.execute_reply": "2023-01-07T14:21:10.269053Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_grammar = convert_ebnf_grammar(autopep8_ebnf_grammar)\n",
    "assert is_valid_grammar(autopep8_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "And we can use the grammar for fuzzing all options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.296800Z",
     "iopub.status.busy": "2023-01-07T14:21:10.296312Z",
     "iopub.status.idle": "2023-01-07T14:21:10.312446Z",
     "shell.execute_reply": "2023-01-07T14:21:10.312818Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " -r foo.py\n",
      " -h --experimental --hang-closing foo.py\n",
      " --list-fixes -v foo.py\n",
      " --aggressive -d foo.py\n",
      " --indent-size 9 --help foo.py\n",
      " --exit-code --recursive foo.py\n",
      " --diff --version -i foo.py\n",
      " --max-line-length 0 --in-place --verbose foo.py\n",
      " --ignore-local-config -a foo.py\n",
      " --select x -i --exit-code foo.py\n",
      " -j 8 --diff foo.py\n",
      " -d -v -d foo.py\n",
      " -p 6 -i foo.py\n",
      " -v --diff foo.py\n",
      " --ignore uA --recursive foo.py\n",
      " --jobs 5 -r foo.py\n",
      " --range 4 1 foo.py\n",
      " --ignore-local-config -i foo.py\n",
      " -r --exit-code foo.py\n",
      " -v -r foo.py\n"
     ]
    }
   ],
   "source": [
    "f = GrammarCoverageFuzzer(autopep8_grammar, max_nonterminals=4)\n",
    "for i in range(20):\n",
    "    print(f.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us apply these options on the actual program.  We need a file `foo.py` that will serve as input: (Note that the following commands will overwrite the file `foo.py`, if it already exists in the current working directory. Be aware of this, if you downloaded the notebooks and are working locally.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.315396Z",
     "iopub.status.busy": "2023-01-07T14:21:10.315063Z",
     "iopub.status.idle": "2023-01-07T14:21:10.316310Z",
     "shell.execute_reply": "2023-01-07T14:21:10.316558Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def create_foo_py():\n",
    "    open(\"foo.py\", \"w\").write(\"\"\"\n",
    "def twice(x = 2):\n",
    "    return  x  +  x\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.318569Z",
     "iopub.status.busy": "2023-01-07T14:21:10.318199Z",
     "iopub.status.idle": "2023-01-07T14:21:10.319700Z",
     "shell.execute_reply": "2023-01-07T14:21:10.319979Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "create_foo_py()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.321992Z",
     "iopub.status.busy": "2023-01-07T14:21:10.321656Z",
     "iopub.status.idle": "2023-01-07T14:21:10.322894Z",
     "shell.execute_reply": "2023-01-07T14:21:10.323128Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "def twice(x = 2):\n",
      "    return  x  +  x\n"
     ]
    }
   ],
   "source": [
    "print(open(\"foo.py\").read(), end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see how `autopep8` fixes the spacing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.325170Z",
     "iopub.status.busy": "2023-01-07T14:21:10.324863Z",
     "iopub.status.idle": "2023-01-07T14:21:10.490109Z",
     "shell.execute_reply": "2023-01-07T14:21:10.490334Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "def twice(x=2):\r\n",
      "    return x + x\r\n"
     ]
    }
   ],
   "source": [
    "!autopep8 foo.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us now put things together.  We define a `ProgramRunner` that will run the `autopep8` executable with arguments coming from the mined `autopep8` grammar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.492912Z",
     "iopub.status.busy": "2023-01-07T14:21:10.492592Z",
     "iopub.status.idle": "2023-01-07T14:21:10.494044Z",
     "shell.execute_reply": "2023-01-07T14:21:10.494233Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Fuzzer import ProgramRunner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Running `autopep8` with the mined options reveals a surprisingly high number of passing runs.  (We see that some options depend on each other or are mutually exclusive, but this is handled by the program logic, not the argument parser, and hence out of our scope.)  The `GrammarCoverageFuzzer` ensures that each option is tested at least once.  (Digits and letters, too, by the way.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:10.502034Z",
     "iopub.status.busy": "2023-01-07T14:21:10.501599Z",
     "iopub.status.idle": "2023-01-07T14:21:11.472536Z",
     "shell.execute_reply": "2023-01-07T14:21:11.472759Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ autopep8 foo.py\n",
      "$ autopep8 --diff --max-line-length 4 --exit-code --range 5 8 -p 2 foo.py\n",
      "$ autopep8 --ignore z --verbose -r --list-fixes foo.py\n",
      "--recursive must be used with --in-place or --diff$ autopep8 --exclude 5 -h -i --aggressive --in-place foo.py\n",
      "$ autopep8 --select a --help --experimental foo.py\n",
      "$ autopep8 --indent-size -30 --recursive foo.py\n",
      "--recursive must be used with --in-place or --diff$ autopep8 --global-config < -j 9 -v -a foo.py\n",
      "parallel jobs requires --in-place$ autopep8 --line-range 7 1 --hang-closing -d foo.py\n",
      "First value of --range should be less than or equal to the second$ autopep8 --pep8-passes 6 --hang-closing --version --ignore-local-config foo.py\n",
      "$ autopep8 --jobs -2 --experimental --version foo.py\n",
      "$ autopep8 --ignore Y: --select ! --global-config e foo.py\n",
      "$ autopep8 --select 1 -a --recursive --aggressive foo.py\n",
      "--recursive must be used with --in-place or --diff$ autopep8 --ignore * --ignore `0 --global-config _ --verbose foo.py\n",
      "[file:foo.py]\n",
      "--->  Applying global fix for E265\n",
      "--->  5 issue(s) to fix {'E251': {2}, 'E271': {3}, 'E221': {3}, 'E222': {3}}\n",
      "--->  3 issue(s) to fix {'E251': {2}, 'E221': {3}, 'E222': {3}}\n",
      "--->  1 issue(s) to fix {'E222': {3}}\n",
      "--->  0 issue(s) to fix {}\n",
      "$ autopep8 --global-config ,\\ --exclude r -v foo.py\n",
      "[file:foo.py]\n",
      "--->  Applying global fix for E265\n",
      "--->  5 issue(s) to fix {'E251': {2}, 'E271': {3}, 'E221': {3}, 'E222': {3}}\n",
      "--->  3 issue(s) to fix {'E251': {2}, 'E221': {3}, 'E222': {3}}\n",
      "--->  1 issue(s) to fix {'E222': {3}}\n",
      "--->  0 issue(s) to fix {}\n",
      "$ autopep8 --global-config xd6M --recursive foo.py\n",
      "--recursive must be used with --in-place or --diff$ autopep8 --select R --exclude L --version --ignore-local-config foo.py\n",
      "$ autopep8 --select \" --verbose -h -d foo.py\n",
      "$ autopep8 --diff -i -h foo.py\n",
      "$ autopep8 --in-place --select w --version -i foo.py\n",
      "$ autopep8 --ignore 49 --exclude lI -i foo.py\n"
     ]
    }
   ],
   "source": [
    "f = GrammarCoverageFuzzer(autopep8_grammar, max_nonterminals=5)\n",
    "for i in range(20):\n",
    "    invocation = \"autopep8\" + f.fuzz()\n",
    "    print(\"$ \" + invocation)\n",
    "    args = invocation.split()\n",
    "    autopep8_runner = ProgramRunner(args)\n",
    "    result, outcome = autopep8_runner.run()\n",
    "    if result.stderr != \"\":\n",
    "        print(result.stderr, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Our `foo.py` file now has been formatted in place a number of times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.475306Z",
     "iopub.status.busy": "2023-01-07T14:21:11.474986Z",
     "iopub.status.idle": "2023-01-07T14:21:11.476535Z",
     "shell.execute_reply": "2023-01-07T14:21:11.476869Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "def twice(x=2):\n",
      "    return x + x\n"
     ]
    }
   ],
   "source": [
    "print(open(\"foo.py\").read(), end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We don't need it anymore, so we clean up things:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.478860Z",
     "iopub.status.busy": "2023-01-07T14:21:11.478449Z",
     "iopub.status.idle": "2023-01-07T14:21:11.480092Z",
     "shell.execute_reply": "2023-01-07T14:21:11.480468Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.482720Z",
     "iopub.status.busy": "2023-01-07T14:21:11.482342Z",
     "iopub.status.idle": "2023-01-07T14:21:11.483915Z",
     "shell.execute_reply": "2023-01-07T14:21:11.484180Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "os.remove(\"foo.py\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Classes for Fuzzing Configuration Options\n",
    "\n",
    "Let us now create reusable classes that we can use for testing arbitrary programs.  (Okay, make that \"arbitrary programs that are written in Python and use the `argparse` module to process command-line arguments.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The class `OptionRunner` is a subclass of `ProgramRunner` that takes care of automatically determining the grammar, using the same steps we used for `autopep8`, above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.486286Z",
     "iopub.status.busy": "2023-01-07T14:21:11.485631Z",
     "iopub.status.idle": "2023-01-07T14:21:11.487614Z",
     "shell.execute_reply": "2023-01-07T14:21:11.487843Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import unreachable_nonterminals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.491752Z",
     "iopub.status.busy": "2023-01-07T14:21:11.491406Z",
     "iopub.status.idle": "2023-01-07T14:21:11.492728Z",
     "shell.execute_reply": "2023-01-07T14:21:11.493157Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionRunner(ProgramRunner):\n",
    "    \"\"\"Run a program while determining its option grammar\"\"\"\n",
    "\n",
    "    def __init__(self, program: Union[str, List[str]],\n",
    "                 arguments: Optional[str] = None, *,\n",
    "                 log: bool = False,\n",
    "                 miner_class: Optional[Type[OptionGrammarMiner]] = None):\n",
    "        \"\"\"Constructor.\n",
    "        `program` - the (Python) program to be executed\n",
    "        `arguments` - an (optional) string with arguments for `program`\n",
    "        `log` - if True, enable logging in miner\n",
    "        `miner_class` - the `OptionGrammarMiner` class to be used\n",
    "                  (default: `OptionGrammarMiner`)\n",
    "        \"\"\"\n",
    "        if isinstance(program, str):\n",
    "            self.base_executable = program\n",
    "        else:\n",
    "            self.base_executable = program[0]\n",
    "\n",
    "        if miner_class is None:\n",
    "            miner_class = OptionGrammarMiner\n",
    "        self.miner_class = miner_class\n",
    "        self.log = log\n",
    "\n",
    "        self.find_contents()\n",
    "        self.find_grammar()\n",
    "        if arguments is not None:\n",
    "            self.set_arguments(arguments)\n",
    "        super().__init__(program)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "First, we find the contents of the Python executable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.496772Z",
     "iopub.status.busy": "2023-01-07T14:21:11.496188Z",
     "iopub.status.idle": "2023-01-07T14:21:11.497655Z",
     "shell.execute_reply": "2023-01-07T14:21:11.497919Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionRunner(OptionRunner):\n",
    "    def find_contents(self):\n",
    "        self._executable = find_executable(self.base_executable)\n",
    "        if self._executable is None:\n",
    "            raise IOError(self.base_executable + \": not found\")\n",
    "\n",
    "        first_line = open(self._executable).readline()\n",
    "        if first_line.find(\"python\") < 0:\n",
    "            raise IOError(self.base_executable + \": not a Python executable\")\n",
    "\n",
    "        self.contents = open(self._executable).read()\n",
    "\n",
    "    def invoker(self):\n",
    "        # We are passing the local variables as is, such that we can access `self`\n",
    "        # We set __name__ to '__main__' to invoke the script as an executable\n",
    "        exec(self.contents, {'__name__': '__main__'})\n",
    "\n",
    "    def executable(self):\n",
    "        return self._executable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Next, we determine the grammar using the `OptionGrammarMiner` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.500912Z",
     "iopub.status.busy": "2023-01-07T14:21:11.500571Z",
     "iopub.status.idle": "2023-01-07T14:21:11.502097Z",
     "shell.execute_reply": "2023-01-07T14:21:11.502463Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionRunner(OptionRunner):\n",
    "    def find_grammar(self):\n",
    "        miner = self.miner_class(self.invoker, log=self.log)\n",
    "        self._ebnf_grammar = miner.mine_ebnf_grammar()\n",
    "\n",
    "    def ebnf_grammar(self):\n",
    "        \"\"\"Return extracted grammar in EBNF form\"\"\"\n",
    "        return self._ebnf_grammar\n",
    "\n",
    "    def grammar(self):\n",
    "        \"\"\"Return extracted grammar in BNF form\"\"\"\n",
    "        return convert_ebnf_grammar(self._ebnf_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The two service methods `set_arguments()` and `set_invocation()` help us to change the arguments and program, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.506508Z",
     "iopub.status.busy": "2023-01-07T14:21:11.506045Z",
     "iopub.status.idle": "2023-01-07T14:21:11.507476Z",
     "shell.execute_reply": "2023-01-07T14:21:11.507746Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionRunner(OptionRunner):\n",
    "    def set_arguments(self, args):\n",
    "        self._ebnf_grammar[\"<arguments>\"] = [\" \" + args]\n",
    "        # Delete rules for previous arguments\n",
    "        for nonterminal in unreachable_nonterminals(self._ebnf_grammar):\n",
    "            del self._ebnf_grammar[nonterminal]\n",
    "\n",
    "    def set_invocation(self, program):\n",
    "        self.program = program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can instantiate the class on `autopep8` and immediately get the grammar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.510208Z",
     "iopub.status.busy": "2023-01-07T14:21:11.509802Z",
     "iopub.status.idle": "2023-01-07T14:21:11.514399Z",
     "shell.execute_reply": "2023-01-07T14:21:11.514616Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_runner = OptionRunner(\"autopep8\", \"foo.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.517135Z",
     "iopub.status.busy": "2023-01-07T14:21:11.516778Z",
     "iopub.status.idle": "2023-01-07T14:21:11.518260Z",
     "shell.execute_reply": "2023-01-07T14:21:11.518473Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' -h', ' --help', ' --version', ' -v', ' --verbose', ' -d', ' --diff', ' -i', ' --in-place', ' --global-config <filename>', ' --ignore-local-config', ' -r', ' --recursive', ' -j <n>', ' --jobs <n>', ' -p <n>', ' --pep8-passes <n>', ' -a', ' --aggressive', ' --experimental', ' --exclude <globs>', ' --list-fixes', ' --ignore <errors>', ' --select <errors>', ' --max-line-length <n>', ' --line-range <line> <line>', ' --range <line> <line>', ' --indent-size <int>', ' --hang-closing', ' --exit-code']\n"
     ]
    }
   ],
   "source": [
    "print(autopep8_runner.ebnf_grammar()[\"<option>\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "An `OptionFuzzer` interacts with the given `OptionRunner` to obtain its grammar, which is then passed to its `GrammarCoverageFuzzer` superclass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.521388Z",
     "iopub.status.busy": "2023-01-07T14:21:11.521048Z",
     "iopub.status.idle": "2023-01-07T14:21:11.522267Z",
     "shell.execute_reply": "2023-01-07T14:21:11.522628Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionFuzzer(GrammarCoverageFuzzer):\n",
    "    \"\"\"Fuzz a (Python) program using its arguments\"\"\"\n",
    "\n",
    "    def __init__(self, runner: OptionRunner, *args, **kwargs):\n",
    "        \"\"\"Constructor. `runner` is an OptionRunner.\"\"\"\n",
    "        assert issubclass(type(runner), OptionRunner)\n",
    "        self.runner = runner\n",
    "        grammar = runner.grammar()\n",
    "        super().__init__(grammar, *args, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "When invoking `run()`, the `OptionFuzzer` creates a new invocation (using `fuzz()` from its grammar) and runs the now given (or previously set) runner with the arguments from the grammar.  Note that the runner specified in `run()` can differ from the one set during initialization; this allows for mining options from one program and applying it in another context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.525660Z",
     "iopub.status.busy": "2023-01-07T14:21:11.525240Z",
     "iopub.status.idle": "2023-01-07T14:21:11.526483Z",
     "shell.execute_reply": "2023-01-07T14:21:11.526678Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class OptionFuzzer(OptionFuzzer):\n",
    "    def run(self, runner=None, inp=\"\"):\n",
    "        if runner is None:\n",
    "            runner = self.runner\n",
    "        assert issubclass(type(runner), OptionRunner)\n",
    "        invocation = runner.executable() + \" \" + self.fuzz()\n",
    "        runner.set_invocation(invocation.split())\n",
    "        return runner.run(inp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Example: Autopep8\n",
    "\n",
    "Let us apply our newly defined classes on the `autopep8` runner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.529868Z",
     "iopub.status.busy": "2023-01-07T14:21:11.529221Z",
     "iopub.status.idle": "2023-01-07T14:21:11.531018Z",
     "shell.execute_reply": "2023-01-07T14:21:11.531620Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_fuzzer = OptionFuzzer(autopep8_runner, max_nonterminals=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.541193Z",
     "iopub.status.busy": "2023-01-07T14:21:11.540654Z",
     "iopub.status.idle": "2023-01-07T14:21:11.542136Z",
     "shell.execute_reply": "2023-01-07T14:21:11.542338Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " foo.py\n",
      " --in-place --ignore-local-config --jobs 6 --recursive -i foo.py\n",
      " --help -a --indent-size -95 --pep8-passes 3 --exclude = -r foo.py\n"
     ]
    }
   ],
   "source": [
    "for i in range(3):\n",
    "    print(autopep8_fuzzer.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can now systematically test `autopep8` with these classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.547632Z",
     "iopub.status.busy": "2023-01-07T14:21:11.547257Z",
     "iopub.status.idle": "2023-01-07T14:21:11.592874Z",
     "shell.execute_reply": "2023-01-07T14:21:11.593136Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(CompletedProcess(args=['/Users/zeller/.pyenv/versions/3.10.2/bin/autopep8', '--hang-closing', '--exit-code', '-d', '--version', 'foo.py'], returncode=0, stdout='autopep8 1.6.0 (pycodestyle: 2.5.0)\\n', stderr=''),\n",
       " 'PASS')"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_fuzzer.run(autopep8_runner)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example: MyPy\n",
    "\n",
    "We can extract options for the `mypy` static type checker for Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.595998Z",
     "iopub.status.busy": "2023-01-07T14:21:11.595571Z",
     "iopub.status.idle": "2023-01-07T14:21:11.597090Z",
     "shell.execute_reply": "2023-01-07T14:21:11.597299Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "assert find_executable(\"mypy\") is not None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.599755Z",
     "iopub.status.busy": "2023-01-07T14:21:11.599397Z",
     "iopub.status.idle": "2023-01-07T14:21:11.732375Z",
     "shell.execute_reply": "2023-01-07T14:21:11.732740Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' -h', ' --help', ' -v', ' --verbose', ' -V', ' --version', ' --config-file <str>', ' --warn-unused-configs', ' --no-warn-unused-configs', ' --no-namespace-packages', ' --namespace-packages', ' --ignore-missing-imports', ' --follow-imports <str>', ' --python-executable', ' --no-site-packages', ' --no-silence-site-packages', ' --python-version <x.y>', ' -2', ' --py2', ' --platform', ' --always-true', ' --always-false', ' --disallow-any-unimported', ' --disallow-any-expr', ' --disallow-any-decorated', ' --disallow-any-explicit', ' --disallow-any-generics', ' --allow-any-generics', ' --disallow-subclassing-any', ' --allow-subclassing-any', ' --disallow-untyped-calls', ' --allow-untyped-calls', ' --disallow-untyped-defs', ' --allow-untyped-defs', ' --disallow-incomplete-defs', ' --allow-incomplete-defs', ' --check-untyped-defs', ' --no-check-untyped-defs', ' --disallow-untyped-decorators', ' --allow-untyped-decorators', ' --implicit-optional', ' --no-implicit-optional', ' --strict-optional', ' --no-strict-optional', ' --warn-redundant-casts', ' --no-warn-redundant-casts', ' --warn-unused-ignores', ' --no-warn-unused-ignores', ' --no-warn-no-return', ' --warn-no-return', ' --warn-return-any', ' --no-warn-return-any', ' --warn-unreachable', ' --no-warn-unreachable', ' --allow-untyped-globals', ' --disallow-untyped-globals', ' --allow-redefinition', ' --disallow-redefinition', ' --no-implicit-reexport', ' --implicit-reexport', ' --strict-equality', ' --no-strict-equality', ' --strict-concatenate', ' --no-strict-concatenate', ' --strict', ' --disable-error-code', ' --enable-error-code', ' --show-error-context', ' --hide-error-context', ' --show-column-numbers', ' --hide-column-numbers', ' --show-error-end', ' --hide-error-end', ' --hide-error-codes', ' --show-error-codes', ' --pretty', ' --no-pretty', ' --no-color-output', ' --color-output', ' --no-error-summary', ' --error-summary', ' --show-absolute-path', ' --hide-absolute-path', ' --soft-error-limit <int>', ' -i', ' --incremental', ' --no-incremental', ' --cache-dir', ' --sqlite-cache', ' --no-sqlite-cache', ' --cache-fine-grained', ' --skip-version-check', ' --skip-cache-mtime-checks', ' --pdb', ' --show-traceback', ' --tb', ' --raise-exceptions', ' --custom-typing-module <MODULE>', ' --disable-recursive-aliases', ' --enable-recursive-aliases', ' --enable-incomplete-feature', ' --custom-typeshed-dir <DIR>', ' --warn-incomplete-stub', ' --no-warn-incomplete-stub', ' --shadow-file', ' --fast-exit', ' --no-fast-exit', ' --allow-empty-bodies', ' --disallow-empty-bodies', ' --any-exprs-report <DIR>', ' --cobertura-xml-report <DIR>', ' --html-report <DIR>', ' --linecount-report <DIR>', ' --linecoverage-report <DIR>', ' --lineprecision-report <DIR>', ' --txt-report <DIR>', ' --xml-report <DIR>', ' --xslt-html-report <DIR>', ' --xslt-txt-report <DIR>', ' --quickstart-file <str>', ' --junit-xml <str>', ' --find-occurrences <CLASS.MEMBER>', ' --scripts-are-modules', ' --install-types', ' --no-install-types', ' --non-interactive', ' --interactive', ' --stats', ' --inferstats', ' --dump-build-stats', ' --timing-stats <str>', ' --debug-cache', ' --dump-deps', ' --dump-graph', ' --semantic-analysis-only', ' --local-partial-types', ' --logical-deps', ' --bazel', ' --package-root', ' --cache-map( <str>)+', ' --enable-incomplete-features', ' --disable-bytearray-promotion', ' --disable-memoryview-promotion', ' --explicit-package-bases', ' --no-explicit-package-bases', ' --fast-module-lookup', ' --no-fast-module-lookup', ' --exclude', ' -m', ' --module', ' -p', ' --package', ' -c', ' --command']\n"
     ]
    }
   ],
   "source": [
    "mypy_runner = OptionRunner(\"mypy\", \"foo.py\")\n",
    "print(mypy_runner.ebnf_grammar()[\"<option>\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.749705Z",
     "iopub.status.busy": "2023-01-07T14:21:11.749199Z",
     "iopub.status.idle": "2023-01-07T14:21:11.934214Z",
     "shell.execute_reply": "2023-01-07T14:21:11.934500Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " --follow-imports l foo.py\n",
      " --stats --allow-any-generics --custom-typeshed-dir yB --enable-recursive-aliases --enable-error-code --disable-memoryview-promotion --version --disallow-redefinition --no-warn-redundant-casts --warn-unreachable --no-warn-unused-configs --config-file Fq --hide-absolute-path --show-traceback --no-strict-optional foo.py\n",
      " --show-error-end --bazel --py2 --non-interactive foo.py\n",
      " --ignore-missing-imports --no-silence-site-packages --no-fast-exit foo.py\n",
      " --no-warn-incomplete-stub --error-summary --warn-unused-configs --allow-empty-bodies --debug-cache --scripts-are-modules --xslt-txt-report c --dump-deps --warn-no-return --implicit-reexport --disallow-untyped-decorators --skip-cache-mtime-checks -i -V --no-color-output --no-strict-concatenate --explicit-package-bases --disallow-empty-bodies --allow-untyped-globals --enable-incomplete-feature --no-install-types --python-version f --always-true --disallow-any-generics --show-error-context --allow-incomplete-defs --skip-version-check --help --disallow-any-unimported --fast-exit --dump-build-stats --no-error-summary --fast-module-lookup --show-column-numbers foo.py\n",
      " --cache-fine-grained --always-false --allow-untyped-calls --no-pretty --local-partial-types foo.py\n",
      " --linecount-report ( --hide-error-codes -p --check-untyped-defs foo.py\n",
      " --cobertura-xml-report W --hide-error-end --warn-redundant-casts --strict-concatenate --disallow-untyped-defs --no-warn-unused-ignores --implicit-optional --xslt-html-report 4% --cache-map X C foo.py\n",
      " --no-implicit-optional --soft-error-limit 0 --find-occurrences 9 --warn-unused-ignores --quickstart-file ] -h --no-namespace-packages --pdb foo.py\n",
      " --module --no-fast-module-lookup --disallow-any-explicit --disable-recursive-aliases --no-warn-no-return --no-incremental --no-site-packages --cache-dir foo.py\n"
     ]
    }
   ],
   "source": [
    "mypy_fuzzer = OptionFuzzer(mypy_runner, max_nonterminals=5)\n",
    "for i in range(10):\n",
    "    print(mypy_fuzzer.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example: Notedown\n",
    "\n",
    "Here's the configuration options for the `notedown` Notebook to Markdown converter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.937345Z",
     "iopub.status.busy": "2023-01-07T14:21:11.936942Z",
     "iopub.status.idle": "2023-01-07T14:21:11.938481Z",
     "shell.execute_reply": "2023-01-07T14:21:11.938689Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "assert find_executable(\"notedown\") is not None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.941233Z",
     "iopub.status.busy": "2023-01-07T14:21:11.940571Z",
     "iopub.status.idle": "2023-01-07T14:21:11.942003Z",
     "shell.execute_reply": "2023-01-07T14:21:11.942387Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:11.948588Z",
     "iopub.status.busy": "2023-01-07T14:21:11.948058Z",
     "iopub.status.idle": "2023-01-07T14:21:14.095819Z",
     "shell.execute_reply": "2023-01-07T14:21:14.096126Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with warnings.catch_warnings():\n",
    "    # Workaround: `notedown` can issue a `DeprecationWarning`\n",
    "    warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "    notedown_runner = OptionRunner(\"notedown\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.098646Z",
     "iopub.status.busy": "2023-01-07T14:21:14.098278Z",
     "iopub.status.idle": "2023-01-07T14:21:14.099760Z",
     "shell.execute_reply": "2023-01-07T14:21:14.099960Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' -h', ' --help', ' -o( <str>)?', ' --output( <str>)?', ' --from <str>', ' --to <str>', ' --run', ' --execute', ' --timeout <int>', ' --strip', ' --precode( <str>)+', ' --knit( <str>)?', ' --rmagic', ' --nomagic', ' --render', ' --template <str>', ' --match <str>', ' --examples', ' --version', ' --debug']\n"
     ]
    }
   ],
   "source": [
    "print(notedown_runner.ebnf_grammar()[\"<option>\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.138803Z",
     "iopub.status.busy": "2023-01-07T14:21:14.138427Z",
     "iopub.status.idle": "2023-01-07T14:21:14.140094Z",
     "shell.execute_reply": "2023-01-07T14:21:14.140578Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " -o :Y\n",
      " --from x --precode S ! --debug --rmagic\n",
      " --output --execute --run w\n",
      " -h --version --examples +\n",
      " --match J --help --strip b\n",
      " --nomagic --render --examples -\n",
      " --rmagic --nomagic --strip ^\n",
      " --timeout -6 --knit --to 5[ --template R --version _\n",
      " -o --debug --examples =\n",
      " --output } -h --debug --render o\n"
     ]
    }
   ],
   "source": [
    "notedown_fuzzer = OptionFuzzer(notedown_runner, max_nonterminals=5)\n",
    "for i in range(10):\n",
    "    print(notedown_fuzzer.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Combinatorial Testing\n",
    "\n",
    "Our `CoverageGrammarFuzzer` does a good job in covering each and every option at least once, which is great for systematic testing.  However, as we also can see in our examples above, some options require each other, while others interfere with each other.  What we should do as good testers is not only to cover every option individually, but also _combinations_ of options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The Python `itertools` module gives us means to create combinations from lists.  We can, for instance, take the `notedown` options and create a list of all pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.143061Z",
     "iopub.status.busy": "2023-01-07T14:21:14.142745Z",
     "iopub.status.idle": "2023-01-07T14:21:14.144063Z",
     "shell.execute_reply": "2023-01-07T14:21:14.144391Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from itertools import combinations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.146654Z",
     "iopub.status.busy": "2023-01-07T14:21:14.146294Z",
     "iopub.status.idle": "2023-01-07T14:21:14.147621Z",
     "shell.execute_reply": "2023-01-07T14:21:14.147854Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "option_list = notedown_runner.ebnf_grammar()[\"<option>\"]\n",
    "pairs = list(combinations(option_list, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There's quite a number of pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.150883Z",
     "iopub.status.busy": "2023-01-07T14:21:14.150339Z",
     "iopub.status.idle": "2023-01-07T14:21:14.152659Z",
     "shell.execute_reply": "2023-01-07T14:21:14.153080Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "190"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.155880Z",
     "iopub.status.busy": "2023-01-07T14:21:14.155420Z",
     "iopub.status.idle": "2023-01-07T14:21:14.156859Z",
     "shell.execute_reply": "2023-01-07T14:21:14.157223Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(' -h', ' --help'), (' -h', ' -o( <str>)?'), (' -h', ' --output( <str>)?'), (' -h', ' --from <str>'), (' -h', ' --to <str>'), (' -h', ' --run'), (' -h', ' --execute'), (' -h', ' --timeout <int>'), (' -h', ' --strip'), (' -h', ' --precode( <str>)+'), (' -h', ' --knit( <str>)?'), (' -h', ' --rmagic'), (' -h', ' --nomagic'), (' -h', ' --render'), (' -h', ' --template <str>'), (' -h', ' --match <str>'), (' -h', ' --examples'), (' -h', ' --version'), (' -h', ' --debug'), (' --help', ' -o( <str>)?')]\n"
     ]
    }
   ],
   "source": [
    "print(pairs[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Testing every such pair of options frequently suffices to cover all interferences between options.  (Programs rarely have conditions involving three or more configuration settings.)  To this end, we _change_ the grammar from having a list of options to having a list of _option pairs_, such that covering these will automatically cover all pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We create a function `pairwise()` that takes a list of options as occurring in our grammar and returns a list of _pairwise options_ – that is, our original options, but concatenated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.159686Z",
     "iopub.status.busy": "2023-01-07T14:21:14.159337Z",
     "iopub.status.idle": "2023-01-07T14:21:14.160540Z",
     "shell.execute_reply": "2023-01-07T14:21:14.160719Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def pairwise(option_list):\n",
    "    return [option_1 + option_2\n",
    "            for (option_1, option_2) in combinations(option_list, 2)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here's the first 20 pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.162971Z",
     "iopub.status.busy": "2023-01-07T14:21:14.162464Z",
     "iopub.status.idle": "2023-01-07T14:21:14.164299Z",
     "shell.execute_reply": "2023-01-07T14:21:14.164117Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' -h --help', ' -h -o( <str>)?', ' -h --output( <str>)?', ' -h --from <str>', ' -h --to <str>', ' -h --run', ' -h --execute', ' -h --timeout <int>', ' -h --strip', ' -h --precode( <str>)+', ' -h --knit( <str>)?', ' -h --rmagic', ' -h --nomagic', ' -h --render', ' -h --template <str>', ' -h --match <str>', ' -h --examples', ' -h --version', ' -h --debug', ' --help -o( <str>)?']\n"
     ]
    }
   ],
   "source": [
    "print(pairwise(option_list)[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The new grammar `pairwise_notedown_grammar` is a copy of the `notedown` grammar, but with the list of options replaced with the above pairwise option list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.167173Z",
     "iopub.status.busy": "2023-01-07T14:21:14.166820Z",
     "iopub.status.idle": "2023-01-07T14:21:14.168200Z",
     "shell.execute_reply": "2023-01-07T14:21:14.168592Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "notedown_grammar = notedown_runner.grammar()\n",
    "pairwise_notedown_grammar = extend_grammar(notedown_grammar)\n",
    "pairwise_notedown_grammar[\"<option>\"] = pairwise(notedown_grammar[\"<option>\"])\n",
    "assert is_valid_grammar(pairwise_notedown_grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Using the \"pairwise\" grammar to fuzz now covers one pair after another:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.171283Z",
     "iopub.status.busy": "2023-01-07T14:21:14.170966Z",
     "iopub.status.idle": "2023-01-07T14:21:14.172447Z",
     "shell.execute_reply": "2023-01-07T14:21:14.172784Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "notedown_pairwise_fuzzer = GrammarCoverageFuzzer(\n",
    "    pairwise_notedown_grammar, max_nonterminals=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.199027Z",
     "iopub.status.busy": "2023-01-07T14:21:14.195254Z",
     "iopub.status.idle": "2023-01-07T14:21:14.350686Z",
     "shell.execute_reply": "2023-01-07T14:21:14.350903Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " -h --debug 5M\n",
      " --help --debug --execute --render\n",
      " -h --timeout 1 /~\n",
      " --nomagic --debug -h --help }\n",
      " -h --nomagic --execute --examples\n",
      " --knit --version --help --strip\n",
      " --strip --examples CD\n",
      " -h --examples --examples --debug --help --version *\n",
      " --timeout 6 --match i .\n",
      " --help --rmagic --strip --version (\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    print(notedown_pairwise_fuzzer.fuzz())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Can we actually test all combinations of options?  Not in practice, as the number of combinations quickly grows as the length increases.  It decreases again as the number of options reaches the maximum (with 20 options, there is only 1 combination involving _all_ options), but the absolute numbers are still staggering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.363499Z",
     "iopub.status.busy": "2023-01-07T14:21:14.363022Z",
     "iopub.status.idle": "2023-01-07T14:21:14.461288Z",
     "shell.execute_reply": "2023-01-07T14:21:14.461853Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 20\n",
      "2 190\n",
      "3 1140\n",
      "4 4845\n",
      "5 15504\n",
      "6 38760\n",
      "7 77520\n",
      "8 125970\n",
      "9 167960\n",
      "10 184756\n",
      "11 167960\n",
      "12 125970\n",
      "13 77520\n",
      "14 38760\n",
      "15 15504\n",
      "16 4845\n",
      "17 1140\n",
      "18 190\n",
      "19 20\n"
     ]
    }
   ],
   "source": [
    "for combination_length in range(1, 20):\n",
    "    tuples = list(combinations(option_list, combination_length))\n",
    "    print(combination_length, len(tuples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Formally, the number of combinations of length $k$ in a set of options of length $n$ is the binomial coefficient\n",
    "$$\n",
    "{n \\choose k} = \\frac{n!}{k!(n - k)!}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "which for $k = 2$ (all pairs) gives us\n",
    "\n",
    "$$\n",
    "{n \\choose 2} = \\frac{n!}{2(n - 2)!} = \\frac{n (n - 1)}{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For `autopep8` with its 30 options..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.465506Z",
     "iopub.status.busy": "2023-01-07T14:21:14.464876Z",
     "iopub.status.idle": "2023-01-07T14:21:14.468261Z",
     "shell.execute_reply": "2023-01-07T14:21:14.468499Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(autopep8_runner.ebnf_grammar()[\"<option>\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.470376Z",
     "iopub.status.busy": "2023-01-07T14:21:14.470004Z",
     "iopub.status.idle": "2023-01-07T14:21:14.471548Z",
     "shell.execute_reply": "2023-01-07T14:21:14.471763Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# docassert\n",
    "assert len(autopep8_runner.ebnf_grammar()[\"<option>\"]) == 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "... we thus need 870 tests to cover all pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.474061Z",
     "iopub.status.busy": "2023-01-07T14:21:14.473668Z",
     "iopub.status.idle": "2023-01-07T14:21:14.475504Z",
     "shell.execute_reply": "2023-01-07T14:21:14.475750Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "870"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(autopep8_runner.ebnf_grammar()[\"<option>\"]) * \\\n",
    "    (len(autopep8_runner.ebnf_grammar()[\"<option>\"]) - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "For `mypy` with its 140+ options, though, we already end up with 20,000+ tests to be conducted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.477593Z",
     "iopub.status.busy": "2023-01-07T14:21:14.477293Z",
     "iopub.status.idle": "2023-01-07T14:21:14.479838Z",
     "shell.execute_reply": "2023-01-07T14:21:14.480039Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "154"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mypy_runner.ebnf_grammar()[\"<option>\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.482589Z",
     "iopub.status.busy": "2023-01-07T14:21:14.482167Z",
     "iopub.status.idle": "2023-01-07T14:21:14.483519Z",
     "shell.execute_reply": "2023-01-07T14:21:14.483862Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# docassert\n",
    "assert len(mypy_runner.ebnf_grammar()[\"<option>\"]) >= 140"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.486479Z",
     "iopub.status.busy": "2023-01-07T14:21:14.486055Z",
     "iopub.status.idle": "2023-01-07T14:21:14.487897Z",
     "shell.execute_reply": "2023-01-07T14:21:14.488150Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23562"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(mypy_runner.ebnf_grammar()[\"<option>\"]) * \\\n",
    "    (len(mypy_runner.ebnf_grammar()[\"<option>\"]) - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Even if each pair takes a second to run, we'd still be done in three hours of testing, though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If your program has more options that you all want to get covered in combinations, it is advisable that you limit the number of configurations further – for instance by limiting combinatorial testing to those combinations that possibly can interact with each other; and covering all other (presumably orthogonal) options individually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This mechanism of creating configurations by extending grammars can be easily extended to other configuration targets.  One may want to explore a greater number of configurations, or expansions in specific contexts.  The [exercises](#Exercises), below, have a number of options ready for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Synopsis\n",
    "\n",
    "This chapter provides two classes:\n",
    "\n",
    "* `OptionRunner` automatically extract command-line options from a Python program;\n",
    "* `OptionFuzzer` uses these to automatically test a Python program with a large variety of options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`OptionRunner` runs a program up to the point where it parses its arguments, and then extracts a grammar that describes its invocations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.490597Z",
     "iopub.status.busy": "2023-01-07T14:21:14.490277Z",
     "iopub.status.idle": "2023-01-07T14:21:14.494020Z",
     "shell.execute_reply": "2023-01-07T14:21:14.494210Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_runner = OptionRunner(\"autopep8\", \"foo.py\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The grammar can be extracted via the method `ebnf_grammar()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.503503Z",
     "iopub.status.busy": "2023-01-07T14:21:14.502736Z",
     "iopub.status.idle": "2023-01-07T14:21:14.507543Z",
     "shell.execute_reply": "2023-01-07T14:21:14.514682Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': ['(<option>)*<arguments>'],\n",
       " '<option>': [' -h',\n",
       "  ' --help',\n",
       "  ' --version',\n",
       "  ' -v',\n",
       "  ' --verbose',\n",
       "  ' -d',\n",
       "  ' --diff',\n",
       "  ' -i',\n",
       "  ' --in-place',\n",
       "  ' --global-config <filename>',\n",
       "  ' --ignore-local-config',\n",
       "  ' -r',\n",
       "  ' --recursive',\n",
       "  ' -j <n>',\n",
       "  ' --jobs <n>',\n",
       "  ' -p <n>',\n",
       "  ' --pep8-passes <n>',\n",
       "  ' -a',\n",
       "  ' --aggressive',\n",
       "  ' --experimental',\n",
       "  ' --exclude <globs>',\n",
       "  ' --list-fixes',\n",
       "  ' --ignore <errors>',\n",
       "  ' --select <errors>',\n",
       "  ' --max-line-length <n>',\n",
       "  ' --line-range <line> <line>',\n",
       "  ' --range <line> <line>',\n",
       "  ' --indent-size <int>',\n",
       "  ' --hang-closing',\n",
       "  ' --exit-code'],\n",
       " '<arguments>': [' foo.py'],\n",
       " '<str>': ['<char>+'],\n",
       " '<char>': ['0',\n",
       "  '1',\n",
       "  '2',\n",
       "  '3',\n",
       "  '4',\n",
       "  '5',\n",
       "  '6',\n",
       "  '7',\n",
       "  '8',\n",
       "  '9',\n",
       "  'a',\n",
       "  'b',\n",
       "  'c',\n",
       "  'd',\n",
       "  'e',\n",
       "  'f',\n",
       "  'g',\n",
       "  'h',\n",
       "  'i',\n",
       "  'j',\n",
       "  'k',\n",
       "  'l',\n",
       "  'm',\n",
       "  'n',\n",
       "  'o',\n",
       "  'p',\n",
       "  'q',\n",
       "  'r',\n",
       "  's',\n",
       "  't',\n",
       "  'u',\n",
       "  'v',\n",
       "  'w',\n",
       "  'x',\n",
       "  'y',\n",
       "  'z',\n",
       "  'A',\n",
       "  'B',\n",
       "  'C',\n",
       "  'D',\n",
       "  'E',\n",
       "  'F',\n",
       "  'G',\n",
       "  'H',\n",
       "  'I',\n",
       "  'J',\n",
       "  'K',\n",
       "  'L',\n",
       "  'M',\n",
       "  'N',\n",
       "  'O',\n",
       "  'P',\n",
       "  'Q',\n",
       "  'R',\n",
       "  'S',\n",
       "  'T',\n",
       "  'U',\n",
       "  'V',\n",
       "  'W',\n",
       "  'X',\n",
       "  'Y',\n",
       "  'Z',\n",
       "  '!',\n",
       "  '\"',\n",
       "  '#',\n",
       "  '$',\n",
       "  '%',\n",
       "  '&',\n",
       "  \"'\",\n",
       "  '(',\n",
       "  ')',\n",
       "  '*',\n",
       "  '+',\n",
       "  ',',\n",
       "  '-',\n",
       "  '.',\n",
       "  '/',\n",
       "  ':',\n",
       "  ';',\n",
       "  '<',\n",
       "  '=',\n",
       "  '>',\n",
       "  '?',\n",
       "  '@',\n",
       "  '[',\n",
       "  '\\\\',\n",
       "  ']',\n",
       "  '^',\n",
       "  '_',\n",
       "  '`',\n",
       "  '{',\n",
       "  '|',\n",
       "  '}',\n",
       "  '~'],\n",
       " '<filename>': ['<str>'],\n",
       " '<int>': ['(-)?<digit>+'],\n",
       " '<digit>': ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'],\n",
       " '<n>': ['<int>'],\n",
       " '<globs>': ['<str>'],\n",
       " '<errors>': ['<str>'],\n",
       " '<line>': ['<int>']}"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "option_ebnf_grammar = autopep8_runner.ebnf_grammar()\n",
    "option_ebnf_grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar can be immediately used for fuzzing. A `GrammarCoverageFuzzer` will ensure all options are covered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.521909Z",
     "iopub.status.busy": "2023-01-07T14:21:14.520054Z",
     "iopub.status.idle": "2023-01-07T14:21:14.523723Z",
     "shell.execute_reply": "2023-01-07T14:21:14.524143Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import convert_ebnf_grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:14.534271Z",
     "iopub.status.busy": "2023-01-07T14:21:14.526913Z",
     "iopub.status.idle": "2023-01-07T14:21:17.078928Z",
     "shell.execute_reply": "2023-01-07T14:21:17.079275Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' foo.py',\n",
       " ' --verbose --ignore-local-config --line-range -1 78 --jobs -43 --pep8-passes 6 -i --indent-size 2 --range -59330 -0 -r --diff --exit-code --global-config KC --aggressive --recursive --ignore 37~ --help --experimental -v -p 7 --hang-closing --exclude t -d -h --in-place --version -a --list-fixes --max-line-length 4 --select e* -j 635 --select 5ulP --ignore G --global-config -E --ignore q --exclude 2^ --global-config \"r --select z --ignore F6 --exclude Xd --exclude ,8 --global-config A --ignore L --ignore Q --global-config 4 --ignore O= --ignore /.m[ --ignore Dy --ignore :s --ignore v --exclude R --ignore B< --exclude $ --select ko --global-config N --global-config @ --ignore hZU% --global-config 0) --exclude n!JT --ignore ? --exclude #&M --select ; --exclude \\\\ --ignore 1 --global-config |I --select > --global-config pb --select V --global-config 9 --global-config Hcwg{Sj --global-config ` --global-config i}W --ignore x --select _ --select ](\\'f+6 --select a]Y --exclude * --global-config z --ignore-local-config --aggressive foo.py',\n",
       " ' foo.py']"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fuzzer = GrammarCoverageFuzzer(convert_ebnf_grammar(option_ebnf_grammar))\n",
    "[fuzzer.fuzz() for i in range(3)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `OptionFuzzer` class summarizes these steps.  Its constructor takes an `OptionRunner` to automatically extract the grammar; it does the necessary steps to extract the grammar and fuzz with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.084001Z",
     "iopub.status.busy": "2023-01-07T14:21:17.081215Z",
     "iopub.status.idle": "2023-01-07T14:21:17.085385Z",
     "shell.execute_reply": "2023-01-07T14:21:17.085600Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "autopep8_runner = OptionRunner(\"autopep8\", \"foo.py\")\n",
    "autopep8_fuzzer = OptionFuzzer(autopep8_runner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.149611Z",
     "iopub.status.busy": "2023-01-07T14:21:17.105920Z",
     "iopub.status.idle": "2023-01-07T14:21:17.157554Z",
     "shell.execute_reply": "2023-01-07T14:21:17.157909Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' foo.py',\n",
       " \" -a -d -i --help --verbose --exclude + --in-place --max-line-length -29 --line-range -7 3 -v --aggressive --indent-size -01 --jobs 6489550 --global-config !KGD'V -r -j 106 --list-fixes --hang-closing foo.py\",\n",
       " ' --exit-code -h --recursive --pep8-passes -0 --ignore ) --ignore-local-config --diff --experimental --version --range 3 -542 -p 56 --select k --diff -h --diff foo.py']"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[autopep8_fuzzer.fuzz() for i in range(3)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The final step in testing would now to invoke the program with these arguments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that `OptionRunner` is experimental: It assumes that the Python program in question uses the `argparse` module; and not all `argparse` features are supported.  Still, it does a pretty good job even on nontrivial programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `OptionRunner` constructor accepts an additional `miner` keyword parameter, which takes the class of the argument grammar miner to be used. By default, this is `OptionGrammarMiner` – a helper class that can be used (and extended) to create own option grammar miners."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.160231Z",
     "iopub.status.busy": "2023-01-07T14:21:17.159933Z",
     "iopub.status.idle": "2023-01-07T14:21:17.161045Z",
     "shell.execute_reply": "2023-01-07T14:21:17.161281Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# ignore\n",
    "from ClassDiagram import display_class_hierarchy\n",
    "from Fuzzer import Fuzzer, Runner, ProgramRunner\n",
    "from Grammars import Expansion\n",
    "from GrammarFuzzer import GrammarFuzzer, DerivationTree\n",
    "from GrammarCoverageFuzzer import TrackingGrammarCoverageFuzzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.165603Z",
     "iopub.status.busy": "2023-01-07T14:21:17.165290Z",
     "iopub.status.idle": "2023-01-07T14:21:17.634681Z",
     "shell.execute_reply": "2023-01-07T14:21:17.634925Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 7.0.2 (20221119.0110)\n",
       " -->\n",
       "<!-- Pages: 1 -->\n",
       "<svg width=\"589pt\" height=\"704pt\"\n",
       " viewBox=\"0.00 0.00 589.00 704.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 700)\">\n",
       "<g id=\"a_graph0\"><a xlink:title=\"OptionRunner class hierarchy\">\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-700 585,-700 585,4 -4,4\"/>\n",
       "</a>\n",
       "</g>\n",
       "<!-- OptionRunner -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>OptionRunner</title>\n",
       "<g id=\"a_node1\"><a xlink:href=\"#\" xlink:title=\"class OptionRunner:&#10;Run a program while determining its option grammar\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"1.5,-32 1.5,-162 114.5,-162 114.5,-32 1.5,-32\"/>\n",
       "<text text-anchor=\"start\" x=\"14\" y=\"-147.8\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">OptionRunner</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"1.5,-139 114.5,-139\"/>\n",
       "<g id=\"a_node1_0\"><a xlink:href=\"#\" xlink:title=\"OptionRunner\">\n",
       "<g id=\"a_node1_1\"><a xlink:href=\"#\" xlink:title=\"__init__(self, program: Union[str, List[str]], arguments: Optional[str] = None, *, log: bool = False, miner_class: Optional[Type[OptionGrammarMiner]] = None):&#10;Constructor.&#10;`program` &#45; the (Python) program to be executed&#10;`arguments` &#45; an (optional) string with arguments for `program`&#10;`log` &#45; if True, enable logging in miner&#10;`miner_class` &#45; the `OptionGrammarMiner` class to be used&#10;(default: `OptionGrammarMiner`)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-127.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_2\"><a xlink:href=\"#\" xlink:title=\"ebnf_grammar(self):&#10;Return extracted grammar in EBNF form\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-116.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">ebnf_grammar()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_3\"><a xlink:href=\"#\" xlink:title=\"grammar(self):&#10;Return extracted grammar in BNF form\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-105.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">grammar()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_4\"><a xlink:href=\"#\" xlink:title=\"executable(self)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-93.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">executable()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_5\"><a xlink:href=\"#\" xlink:title=\"find_contents(self)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-82.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">find_contents()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_6\"><a xlink:href=\"#\" xlink:title=\"find_grammar(self)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-71.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">find_grammar()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_7\"><a xlink:href=\"#\" xlink:title=\"invoker(self)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-60.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">invoker()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_8\"><a xlink:href=\"#\" xlink:title=\"set_arguments(self, args)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-49.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">set_arguments()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node1_9\"><a xlink:href=\"#\" xlink:title=\"set_invocation(self, program)\">\n",
       "<text text-anchor=\"start\" x=\"10\" y=\"-38.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">set_invocation()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- ProgramRunner -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>ProgramRunner</title>\n",
       "<g id=\"a_node2\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"class ProgramRunner:&#10;Test a program with inputs.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-230.5 0,-272.5 116,-272.5 116,-230.5 0,-230.5\"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-258.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">ProgramRunner</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"0,-249.5 116,-249.5\"/>\n",
       "<g id=\"a_node2_10\"><a xlink:href=\"#\" xlink:title=\"ProgramRunner\">\n",
       "<g id=\"a_node2_11\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"__init__(self, program: Union[str, List[str]]) &#45;&gt; None:&#10;Initialize.&#10;`program` is a program spec as passed to `subprocess.run()`\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-238\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- OptionRunner&#45;&gt;ProgramRunner -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>OptionRunner&#45;&gt;ProgramRunner</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M58,-161.79C58,-181.76 58,-202.89 58,-219.53\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"54.5,-219.25 58,-229.25 61.5,-219.25 54.5,-219.25\"/>\n",
       "</g>\n",
       "<!-- Runner -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>Runner</title>\n",
       "<g id=\"a_node3\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"class Runner:&#10;Base class for testing inputs.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"19.5,-309.5 19.5,-403.5 96.5,-403.5 96.5,-309.5 19.5,-309.5\"/>\n",
       "<text text-anchor=\"start\" x=\"35\" y=\"-389.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">Runner</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"19.5,-380.5 96.5,-380.5\"/>\n",
       "<g id=\"a_node3_12\"><a xlink:href=\"#\" xlink:title=\"Runner\">\n",
       "<g id=\"a_node3_13\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"FAIL = &#39;FAIL&#39;\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-368\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">FAIL</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node3_14\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"PASS = &#39;PASS&#39;\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-357\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">PASS</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node3_15\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"UNRESOLVED = &#39;UNRESOLVED&#39;\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-346\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">UNRESOLVED</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"19.5,-339.5 96.5,-339.5\"/>\n",
       "<g id=\"a_node3_16\"><a xlink:href=\"#\" xlink:title=\"Runner\">\n",
       "<g id=\"a_node3_17\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"__init__(self) &#45;&gt; None:&#10;Initialize\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-328.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node3_18\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"run(self, inp: str) &#45;&gt; Any:&#10;Run the runner with the given input\">\n",
       "<text text-anchor=\"start\" x=\"28\" y=\"-317.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">run()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- ProgramRunner&#45;&gt;Runner -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>ProgramRunner&#45;&gt;Runner</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M58,-272.36C58,-279.92 58,-288.97 58,-298.28\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"54.5,-298.15 58,-308.15 61.5,-298.15 54.5,-298.15\"/>\n",
       "</g>\n",
       "<!-- OptionFuzzer -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>OptionFuzzer</title>\n",
       "<g id=\"a_node4\"><a xlink:href=\"#\" xlink:title=\"class OptionFuzzer:&#10;Fuzz a (Python) program using its arguments\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"172.5,-70.5 172.5,-123.5 271.5,-123.5 271.5,-70.5 172.5,-70.5\"/>\n",
       "<text text-anchor=\"start\" x=\"180.5\" y=\"-109.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">OptionFuzzer</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"172.5,-100.5 271.5,-100.5\"/>\n",
       "<g id=\"a_node4_19\"><a xlink:href=\"#\" xlink:title=\"OptionFuzzer\">\n",
       "<g id=\"a_node4_20\"><a xlink:href=\"#\" xlink:title=\"__init__(self, runner: OptionRunner, *args, **kwargs):&#10;Constructor. `runner` is an OptionRunner.\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-89.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node4_21\"><a xlink:href=\"#\" xlink:title=\"run(self, runner=None, inp=&#39;&#39;):&#10;Run `runner` with fuzz input\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-78.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">run()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- GrammarCoverageFuzzer -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>GrammarCoverageFuzzer</title>\n",
       "<g id=\"a_node5\"><a xlink:href=\"GrammarCoverageFuzzer.ipynb\" xlink:title=\"class GrammarCoverageFuzzer:&#10;Produce from grammars, aiming for coverage of all expansions.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"135,-233.5 135,-269.5 309,-269.5 309,-233.5 135,-233.5\"/>\n",
       "<text text-anchor=\"start\" x=\"143\" y=\"-248.8\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">GrammarCoverageFuzzer</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- OptionFuzzer&#45;&gt;GrammarCoverageFuzzer -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>OptionFuzzer&#45;&gt;GrammarCoverageFuzzer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222,-123.18C222,-150.57 222,-194.23 222,-222.63\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"218.5,-222.26 222,-232.26 225.5,-222.26 218.5,-222.26\"/>\n",
       "</g>\n",
       "<!-- SimpleGrammarCoverageFuzzer -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>SimpleGrammarCoverageFuzzer</title>\n",
       "<g id=\"a_node6\"><a xlink:href=\"GrammarCoverageFuzzer.ipynb\" xlink:title=\"class SimpleGrammarCoverageFuzzer:&#10;When choosing expansions, prefer expansions not covered.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"114.5,-338.5 114.5,-374.5 329.5,-374.5 329.5,-338.5 114.5,-338.5\"/>\n",
       "<text text-anchor=\"start\" x=\"122.5\" y=\"-353.8\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">SimpleGrammarCoverageFuzzer</text>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- GrammarCoverageFuzzer&#45;&gt;SimpleGrammarCoverageFuzzer -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>GrammarCoverageFuzzer&#45;&gt;SimpleGrammarCoverageFuzzer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222,-269.05C222,-284.72 222,-308.57 222,-327.26\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"218.5,-327.14 222,-337.14 225.5,-327.14 218.5,-327.14\"/>\n",
       "</g>\n",
       "<!-- TrackingGrammarCoverageFuzzer -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>TrackingGrammarCoverageFuzzer</title>\n",
       "<g id=\"a_node7\"><a xlink:href=\"GrammarCoverageFuzzer.ipynb\" xlink:title=\"class TrackingGrammarCoverageFuzzer:&#10;Track grammar coverage during production\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"107.5,-440.5 107.5,-482.5 336.5,-482.5 336.5,-440.5 107.5,-440.5\"/>\n",
       "<text text-anchor=\"start\" x=\"115.5\" y=\"-468.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">TrackingGrammarCoverageFuzzer</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107.5,-459.5 336.5,-459.5\"/>\n",
       "<g id=\"a_node7_22\"><a xlink:href=\"#\" xlink:title=\"TrackingGrammarCoverageFuzzer\">\n",
       "<g id=\"a_node7_23\"><a xlink:href=\"GrammarCoverageFuzzer.ipynb\" xlink:title=\"__init__(self, *args, **kwargs) &#45;&gt; None:&#10;Produce strings from `grammar`, starting with `start_symbol`.&#10;If `min_nonterminals` or `max_nonterminals` is given, use them as limits&#10;for the number of nonterminals produced.&#10;If `disp` is set, display the intermediate derivation trees.&#10;If `log` is set, show intermediate steps as text on standard output.\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-448\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- SimpleGrammarCoverageFuzzer&#45;&gt;TrackingGrammarCoverageFuzzer -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>SimpleGrammarCoverageFuzzer&#45;&gt;TrackingGrammarCoverageFuzzer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222,-374.05C222,-388.98 222,-411.32 222,-429.55\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"218.5,-429.31 222,-439.31 225.5,-429.31 218.5,-429.31\"/>\n",
       "</g>\n",
       "<!-- GrammarFuzzer -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>GrammarFuzzer</title>\n",
       "<g id=\"a_node8\"><a xlink:href=\"GrammarFuzzer.ipynb\" xlink:title=\"class GrammarFuzzer:&#10;Produce strings from grammars efficiently, using derivation trees.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"164,-519.5 164,-583.5 280,-583.5 280,-519.5 164,-519.5\"/>\n",
       "<text text-anchor=\"start\" x=\"172\" y=\"-569.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">GrammarFuzzer</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"164,-560.5 280,-560.5\"/>\n",
       "<g id=\"a_node8_24\"><a xlink:href=\"#\" xlink:title=\"GrammarFuzzer\">\n",
       "<g id=\"a_node8_25\"><a xlink:href=\"GrammarFuzzer.ipynb\" xlink:title=\"__init__(self, grammar: Dict[str, List[Expansion]], start_symbol: str = &#39;&lt;start&gt;&#39;, min_nonterminals: int = 0, max_nonterminals: int = 10, disp: bool = False, log: Union[bool, int] = False) &#45;&gt; None:&#10;Produce strings from `grammar`, starting with `start_symbol`.&#10;If `min_nonterminals` or `max_nonterminals` is given, use them as limits&#10;for the number of nonterminals produced.&#10;If `disp` is set, display the intermediate derivation trees.&#10;If `log` is set, show intermediate steps as text on standard output.\">\n",
       "<text text-anchor=\"start\" x=\"189\" y=\"-549\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node8_26\"><a xlink:href=\"GrammarFuzzer.ipynb\" xlink:title=\"fuzz(self) &#45;&gt; str:&#10;Produce a string from the grammar.\">\n",
       "<text text-anchor=\"start\" x=\"189\" y=\"-538\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">fuzz()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node8_27\"><a xlink:href=\"GrammarFuzzer.ipynb\" xlink:title=\"fuzz_tree(self) &#45;&gt; DerivationTree:&#10;Produce a derivation tree from the grammar.\">\n",
       "<text text-anchor=\"start\" x=\"189\" y=\"-527\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">fuzz_tree()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- TrackingGrammarCoverageFuzzer&#45;&gt;GrammarFuzzer -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>TrackingGrammarCoverageFuzzer&#45;&gt;GrammarFuzzer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222,-482.47C222,-490.17 222,-499.29 222,-508.31\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"218.5,-508.05 222,-518.05 225.5,-508.05 218.5,-508.05\"/>\n",
       "</g>\n",
       "<!-- Fuzzer -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>Fuzzer</title>\n",
       "<g id=\"a_node9\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"class Fuzzer:&#10;Base class for fuzzers.\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"183.5,-620.5 183.5,-695.5 260.5,-695.5 260.5,-620.5 183.5,-620.5\"/>\n",
       "<text text-anchor=\"start\" x=\"201\" y=\"-681.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">Fuzzer</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"183.5,-672.5 260.5,-672.5\"/>\n",
       "<g id=\"a_node9_28\"><a xlink:href=\"#\" xlink:title=\"Fuzzer\">\n",
       "<g id=\"a_node9_29\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"__init__(self) &#45;&gt; None:&#10;Constructor\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-661.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node9_30\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"fuzz(self) &#45;&gt; str:&#10;Return fuzz input\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-650.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">fuzz()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node9_31\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"run(self, runner: Fuzzer.Runner = &lt;Fuzzer.Runner object at 0x1057ea680&gt;) &#45;&gt; Tuple[subprocess.CompletedProcess, str]:&#10;Run `runner` with fuzz input\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-639.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-style=\"italic\" font-size=\"10.00\">run()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node9_32\"><a xlink:href=\"Fuzzer.ipynb\" xlink:title=\"runs(self, runner: Fuzzer.Runner = &lt;Fuzzer.PrintRunner object at 0x1057ea5f0&gt;, trials: int = 10) &#45;&gt; List[Tuple[subprocess.CompletedProcess, str]]:&#10;Run `runner` with fuzz input, `trials` times\">\n",
       "<text text-anchor=\"start\" x=\"192\" y=\"-628.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">runs()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- GrammarFuzzer&#45;&gt;Fuzzer -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>GrammarFuzzer&#45;&gt;Fuzzer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222,-583.29C222,-591.41 222,-600.37 222,-609.16\"/>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"218.5,-609.02 222,-619.02 225.5,-609.02 218.5,-609.02\"/>\n",
       "</g>\n",
       "<!-- OptionGrammarMiner -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>OptionGrammarMiner</title>\n",
       "<g id=\"a_node10\"><a xlink:href=\"#\" xlink:title=\"class OptionGrammarMiner:&#10;Helper class for extracting option grammars\">\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"289.5,-0.5 289.5,-193.5 442.5,-193.5 442.5,-0.5 289.5,-0.5\"/>\n",
       "<text text-anchor=\"start\" x=\"297.5\" y=\"-179.3\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"14.00\" fill=\"#b03a2e\">OptionGrammarMiner</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"289.5,-170.5 442.5,-170.5\"/>\n",
       "<g id=\"a_node10_33\"><a xlink:href=\"#\" xlink:title=\"OptionGrammarMiner\">\n",
       "<g id=\"a_node10_34\"><a xlink:href=\"#\" xlink:title=\"ARGUMENTS_SYMBOL = &#39;&lt;arguments&gt;&#39;\">\n",
       "<text text-anchor=\"start\" x=\"318\" y=\"-158.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">ARGUMENTS_SYMBOL</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_35\"><a xlink:href=\"#\" xlink:title=\"OPTION_SYMBOL = &#39;&lt;option&gt;&#39;\">\n",
       "<text text-anchor=\"start\" x=\"318\" y=\"-147.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">OPTION_SYMBOL</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"289.5,-140.5 442.5,-140.5\"/>\n",
       "<g id=\"a_node10_36\"><a xlink:href=\"#\" xlink:title=\"OptionGrammarMiner\">\n",
       "<g id=\"a_node10_37\"><a xlink:href=\"#\" xlink:title=\"__init__(self, function: Callable, log: bool = False):&#10;Constructor.&#10;`function` &#45; a function processing arguments using argparse()&#10;`log` &#45; output diagnostics if True\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-129.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">__init__()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_38\"><a xlink:href=\"#\" xlink:title=\"mine_ebnf_grammar(self):&#10;Extract EBNF option grammar\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-118.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">mine_ebnf_grammar()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_39\"><a xlink:href=\"#\" xlink:title=\"mine_grammar(self):&#10;Extract BNF option grammar\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-107.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"10.00\">mine_grammar()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_40\"><a xlink:href=\"#\" xlink:title=\"add_group(self, locals, exclusive)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-95.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_group()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_41\"><a xlink:href=\"#\" xlink:title=\"add_int_rule(self)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-84.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_int_rule()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_42\"><a xlink:href=\"#\" xlink:title=\"add_metavar_rule(self, metavar, type_)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-73.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_metavar_rule()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_43\"><a xlink:href=\"#\" xlink:title=\"add_parameter(self, kwargs, metavar)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-62.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_parameter()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_44\"><a xlink:href=\"#\" xlink:title=\"add_str_rule(self)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-51.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_str_rule()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_45\"><a xlink:href=\"#\" xlink:title=\"add_type_rule(self, type_)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-40.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">add_type_rule()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_46\"><a xlink:href=\"#\" xlink:title=\"process_arg(self, arg, in_group, kwargs)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-29.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">process_arg()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_47\"><a xlink:href=\"#\" xlink:title=\"process_argument(self, locals, in_group)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-18.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">process_argument()</text>\n",
       "</a>\n",
       "</g>\n",
       "<g id=\"a_node10_48\"><a xlink:href=\"#\" xlink:title=\"traceit(self, frame, event, arg)\">\n",
       "<text text-anchor=\"start\" x=\"309\" y=\"-7.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"10.00\">traceit()</text>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- Legend -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>Legend</title>\n",
       "<text text-anchor=\"start\" x=\"461\" y=\"-114.5\" font-family=\"Patua One, Helvetica, sans-serif\" font-weight=\"bold\" font-size=\"10.00\" fill=\"#b03a2e\">Legend</text>\n",
       "<text text-anchor=\"start\" x=\"461\" y=\"-104.5\" font-family=\"Patua One, Helvetica, sans-serif\" font-size=\"10.00\">• </text>\n",
       "<text text-anchor=\"start\" x=\"468\" y=\"-104.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-weight=\"bold\" font-size=\"8.00\">public_method()</text>\n",
       "<text text-anchor=\"start\" x=\"461\" y=\"-94.5\" font-family=\"Patua One, Helvetica, sans-serif\" font-size=\"10.00\">• </text>\n",
       "<text text-anchor=\"start\" x=\"468\" y=\"-94.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-size=\"8.00\">private_method()</text>\n",
       "<text text-anchor=\"start\" x=\"461\" y=\"-84.5\" font-family=\"Patua One, Helvetica, sans-serif\" font-size=\"10.00\">• </text>\n",
       "<text text-anchor=\"start\" x=\"468\" y=\"-84.5\" font-family=\"'Fira Mono', 'Source Code Pro', 'Courier', monospace\" font-style=\"italic\" font-size=\"8.00\">overloaded_method()</text>\n",
       "<text text-anchor=\"start\" x=\"461\" y=\"-75.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"9.00\">Hover over names to see doc</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x13fde47f0>"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ignore\n",
    "display_class_hierarchy([OptionRunner, OptionFuzzer, OptionGrammarMiner],\n",
    "                        public_methods=[\n",
    "                            Fuzzer.__init__,\n",
    "                            Fuzzer.fuzz,\n",
    "                            Fuzzer.run,\n",
    "                            Fuzzer.runs,\n",
    "                            GrammarFuzzer.__init__,\n",
    "                            GrammarFuzzer.fuzz,\n",
    "                            GrammarFuzzer.fuzz_tree,\n",
    "                            TrackingGrammarCoverageFuzzer.__init__,\n",
    "                            OptionFuzzer.__init__,\n",
    "                            OptionFuzzer.run,\n",
    "                            Runner.__init__,\n",
    "                            Runner.run,\n",
    "                            ProgramRunner.__init__,\n",
    "                            ProgramRunner.__init__,\n",
    "                            OptionRunner.__init__,\n",
    "                            OptionRunner.ebnf_grammar,\n",
    "                            OptionRunner.grammar,\n",
    "                            OptionGrammarMiner.__init__,\n",
    "                            OptionGrammarMiner.mine_ebnf_grammar,\n",
    "                            OptionGrammarMiner.mine_grammar,\n",
    "                        ],\n",
    "                        types={\n",
    "                            'DerivationTree': DerivationTree,\n",
    "                            'Expansion': Expansion,\n",
    "                            'Grammar': Grammar\n",
    "                        },\n",
    "                        project='fuzzingbook')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lessons Learned\n",
    "\n",
    "* Besides regular input data, program _configurations_ make an important testing target.\n",
    "* For a given program using a standard library to parse command-line options and arguments, one can automatically extract these and convert them into a grammar.\n",
    "* To cover not only single options, but combinations of options, one can expand the grammar to cover all pairs, or come up with even more ambitious targets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Next Steps\n",
    "\n",
    "If you liked the idea of mining a grammar from a program, do not miss:\n",
    "\n",
    "* [how to mine grammars for input data](GrammarMiner.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Our next steps in the book focus on:\n",
    "\n",
    "* [how to parse and recombine inputs](Parser.ipynb)\n",
    "* [how to assign weights and probabilities to specific productions](ProbabilisticGrammarFuzzer.ipynb)\n",
    "* [how to simplify inputs that cause a failure](Reducer.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Background\n",
    "\n",
    "Although configuration data is just as likely to cause failures as other input data, it has received relatively little attention in test generation – possibly because, unlike \"regular\" input data, configuration data is not so much under control of external parties, and because, again unlike regular data, there is little variance in configurations.  Creating models for software configurations and using these models for testing is commonplace, as is the idea of pairwise testing.  For an overview, see \\cite{Pezze2008}; for a discussion and comparison of state-of-the-art techniques, see \\cite{Petke2015}.\n",
    "\n",
    "More specifically, \\cite{Sutton2007} also discuss techniques to systematically cover command-line options.  Dai et al. \\cite{Dai2010} apply configuration fuzzing by changing variables associated with configuration files.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "toc-hr-collapsed": false
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "toc-hr-collapsed": true
   },
   "source": [
    "### Exercise 1: #ifdef Configuration Fuzzing\n",
    "\n",
    "In C programs, the *C preprocessor* can be used to choose which code parts should be compiled and which ones should not.  As an example, in the C code\n",
    "\n",
    "```C\n",
    "#ifdef LONG_FOO\n",
    "long foo() { ... }\n",
    "#else\n",
    "int foo() { ... }\n",
    "#endif\n",
    "```\n",
    "\n",
    "the compiler will compile the function `foo()` with return type`long` if the _preprocessor variable_ `LONG_FOO` is defined, and with return type `int` if not.  Such preprocessor variables are either set in the source files (using `#define`, as in `#define LONG_FOO`) or on the C compiler command line (using `-D<variable>` or `-D<variable>=<value>`, as in `-DLONG_FOO`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Such *conditional compilation* is used to configure C programs towards their environment.  System-specific code can contain lots of conditional compilation.  As an example, consider this excerpt of `xmlparse.c`, the XML parser that is part of the Python runtime library:\n",
    "\n",
    "```c\n",
    "#if defined(_WIN32) && !defined(LOAD_LIBRARY_SEARCH_SYSTEM32)\n",
    "# define LOAD_LIBRARY_SEARCH_SYSTEM32  0x00000800\n",
    "#endif\n",
    "\n",
    "#if !defined(HAVE_GETRANDOM) && !defined(HAVE_SYSCALL_GETRANDOM) \\\n",
    "    && !defined(HAVE_ARC4RANDOM_BUF) && !defined(HAVE_ARC4RANDOM) \\\n",
    "    && !defined(XML_DEV_URANDOM) \\\n",
    "    && !defined(_WIN32) \\\n",
    "    && !defined(XML_POOR_ENTROPY)\n",
    "# error\n",
    "#endif\n",
    "\n",
    "#if !defined(TIOCSWINSZ) || defined(__SCO__) || defined(__UNIXWARE__)\n",
    "#define USE_SYSV_ENVVARS\t/* COLUMNS/LINES vs. TERMCAP */\n",
    "#endif\n",
    "\n",
    "#ifdef XML_UNICODE_WCHAR_T\n",
    "#define XML_T(x) (const wchar_t)x\n",
    "#define XML_L(x) L ## x\n",
    "#else\n",
    "#define XML_T(x) (const unsigned short)x\n",
    "#define XML_L(x) x\n",
    "#endif\n",
    "\n",
    "int fun(int x) { return XML_T(x); }\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A typical configuration for the C preprocessor on the above code could be `cc -c -D_WIN32 -DXML_POOR_ENTROPY -DXML_UNICODE_WCHAR_T xmlparse.c`, defining the given preprocessor variables and selecting the appropriate code fragments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Since the compiler can only compile one configuration at a time (implying that we can also only _test_ one resulting executable at a time), your task is to find out which of these configurations actually compile.  To this end, proceed in three steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Part 1: Extract Preprocessor Variables\n",
    "\n",
    "Write a _function_ `cpp_identifiers()` that, given a set of lines (say, from `open(filename).readlines()`), extracts all preprocessor variables referenced in `#if` or `#ifdef` preprocessor instructions.  Apply `ifdef_identifiers()` on the sample C input above, such that\n",
    "\n",
    "```python\n",
    "cpp_identifiers(open(\"xmlparse.c\").readlines()) \n",
    "```\n",
    "\n",
    "returns the set\n",
    "\n",
    "```python\n",
    "{'_WIN32', 'LOAD_LIBRARY_SEARCH_SYSTEM32', 'HAVE_GETRANDOM', 'HAVE_SYSCALL_GETRANDOM', 'HAVE_ARC4RANDOM_BUF', ...}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.**  Let us start with creating a sample input file, `xmlparse.c`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.637945Z",
     "iopub.status.busy": "2023-01-07T14:21:17.637489Z",
     "iopub.status.idle": "2023-01-07T14:21:17.638909Z",
     "shell.execute_reply": "2023-01-07T14:21:17.639161Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "filename = \"xmlparse.c\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.641348Z",
     "iopub.status.busy": "2023-01-07T14:21:17.640984Z",
     "iopub.status.idle": "2023-01-07T14:21:17.642582Z",
     "shell.execute_reply": "2023-01-07T14:21:17.642909Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "open(filename, \"w\").write(\n",
    "    \"\"\"\n",
    "#if defined(_WIN32) && !defined(LOAD_LIBRARY_SEARCH_SYSTEM32)\n",
    "# define LOAD_LIBRARY_SEARCH_SYSTEM32  0x00000800\n",
    "#endif\n",
    "\n",
    "#if !defined(HAVE_GETRANDOM) && !defined(HAVE_SYSCALL_GETRANDOM) \\\n",
    "    && !defined(HAVE_ARC4RANDOM_BUF) && !defined(HAVE_ARC4RANDOM) \\\n",
    "    && !defined(XML_DEV_URANDOM) \\\n",
    "    && !defined(_WIN32) \\\n",
    "    && !defined(XML_POOR_ENTROPY)\n",
    "# error\n",
    "#endif\n",
    "\n",
    "#if !defined(TIOCSWINSZ) || defined(__SCO__) || defined(__UNIXWARE__)\n",
    "#define USE_SYSV_ENVVARS\t/* COLUMNS/LINES vs. TERMCAP */\n",
    "#endif\n",
    "\n",
    "#ifdef XML_UNICODE_WCHAR_T\n",
    "#define XML_T(x) (const wchar_t)x\n",
    "#define XML_L(x) L ## x\n",
    "#else\n",
    "#define XML_T(x) (const unsigned short)x\n",
    "#define XML_L(x) x\n",
    "#endif\n",
    "\n",
    "int fun(int x) { return XML_T(x); }\n",
    "\"\"\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "To find C preprocessor `#if` directives and preprocessor variables, we use regular expressions matching them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.644640Z",
     "iopub.status.busy": "2023-01-07T14:21:17.644358Z",
     "iopub.status.idle": "2023-01-07T14:21:17.645721Z",
     "shell.execute_reply": "2023-01-07T14:21:17.646183Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.648821Z",
     "iopub.status.busy": "2023-01-07T14:21:17.648445Z",
     "iopub.status.idle": "2023-01-07T14:21:17.649587Z",
     "shell.execute_reply": "2023-01-07T14:21:17.650025Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "re_cpp_if_directive = re.compile(r\"\\s*#\\s*(el)?if\")\n",
    "re_cpp_identifier = re.compile(r\"[a-zA-Z_$]+\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.652448Z",
     "iopub.status.busy": "2023-01-07T14:21:17.652109Z",
     "iopub.status.idle": "2023-01-07T14:21:17.653289Z",
     "shell.execute_reply": "2023-01-07T14:21:17.653536Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "def cpp_identifiers(lines):\n",
    "    identifiers = set()\n",
    "    for line in lines:\n",
    "        if re_cpp_if_directive.match(line):\n",
    "            identifiers |= set(re_cpp_identifier.findall(line))\n",
    "\n",
    "    # These are preprocessor keywords\n",
    "    identifiers -= {\"if\", \"ifdef\", \"ifndef\", \"defined\"}\n",
    "    return identifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.655334Z",
     "iopub.status.busy": "2023-01-07T14:21:17.655034Z",
     "iopub.status.idle": "2023-01-07T14:21:17.657670Z",
     "shell.execute_reply": "2023-01-07T14:21:17.657894Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'HAVE_ARC',\n",
       " 'HAVE_GETRANDOM',\n",
       " 'HAVE_SYSCALL_GETRANDOM',\n",
       " 'LOAD_LIBRARY_SEARCH_SYSTEM',\n",
       " 'RANDOM',\n",
       " 'RANDOM_BUF',\n",
       " 'TIOCSWINSZ',\n",
       " 'XML_DEV_URANDOM',\n",
       " 'XML_POOR_ENTROPY',\n",
       " 'XML_UNICODE_WCHAR_T',\n",
       " '_WIN',\n",
       " '__SCO__',\n",
       " '__UNIXWARE__'}"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cpp_ids = cpp_identifiers(open(\"xmlparse.c\").readlines())\n",
    "cpp_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Part 2: Derive an Option Grammar\n",
    "\n",
    "With the help of `cpp_identifiers()`, create a grammar which has C compiler invocations with a list of options, where each option takes the form `-D<variable>` for a preprocessor variable `<variable>`.  Using this grammar `cpp_grammar`, a fuzzer \n",
    "\n",
    "```python\n",
    "g = GrammarCoverageFuzzer(cpp_grammar)\n",
    "```\n",
    "\n",
    "would create C compiler invocations such as\n",
    "\n",
    "```python\n",
    "[g.fuzz() for i in range(10)]\n",
    "['cc -DHAVE_SYSCALL_GETRANDOM xmlparse.c',\n",
    " 'cc -D__SCO__ -DRANDOM_BUF -DXML_UNICODE_WCHAR_T -D__UNIXWARE__ xmlparse.c',\n",
    " 'cc -DXML_POOR_ENTROPY xmlparse.c',\n",
    " 'cc -DRANDOM xmlparse.c',\n",
    " 'cc -D_WIN xmlparse.c',\n",
    " 'cc -DHAVE_ARC xmlparse.c', ...]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** This is not very difficult:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.659897Z",
     "iopub.status.busy": "2023-01-07T14:21:17.659595Z",
     "iopub.status.idle": "2023-01-07T14:21:17.660839Z",
     "shell.execute_reply": "2023-01-07T14:21:17.661071Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "from Grammars import Grammar, is_valid_grammar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.663941Z",
     "iopub.status.busy": "2023-01-07T14:21:17.663508Z",
     "iopub.status.idle": "2023-01-07T14:21:17.664648Z",
     "shell.execute_reply": "2023-01-07T14:21:17.664877Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "cpp_grammar: Grammar = {\n",
    "    \"<start>\": [\"cc -c<options> \" + filename],\n",
    "    \"<options>\": [\"<option>\", \"<options><option>\"],\n",
    "    \"<option>\": []\n",
    "}\n",
    "\n",
    "for id in cpp_ids:\n",
    "    s = new_symbol(cpp_grammar, \"<\" + id + \">\")\n",
    "    cpp_grammar[\"<option>\"].append(s)\n",
    "    cpp_grammar[s] = [\" -D\" + id]\n",
    "\n",
    "assert is_valid_grammar(cpp_grammar)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.667198Z",
     "iopub.status.busy": "2023-01-07T14:21:17.666906Z",
     "iopub.status.idle": "2023-01-07T14:21:17.668454Z",
     "shell.execute_reply": "2023-01-07T14:21:17.668833Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'<start>': ['cc -c<options> xmlparse.c'],\n",
       " '<options>': ['<option>', '<options><option>'],\n",
       " '<option>': ['<XML_UNICODE_WCHAR_T>',\n",
       "  '<HAVE_GETRANDOM>',\n",
       "  '<TIOCSWINSZ>',\n",
       "  '<__UNIXWARE__>',\n",
       "  '<LOAD_LIBRARY_SEARCH_SYSTEM>',\n",
       "  '<RANDOM_BUF>',\n",
       "  '<XML_POOR_ENTROPY>',\n",
       "  '<HAVE_SYSCALL_GETRANDOM>',\n",
       "  '<__SCO__>',\n",
       "  '<HAVE_ARC>',\n",
       "  '<XML_DEV_URANDOM>',\n",
       "  '<_WIN>',\n",
       "  '<RANDOM>'],\n",
       " '<XML_UNICODE_WCHAR_T>': [' -DXML_UNICODE_WCHAR_T'],\n",
       " '<HAVE_GETRANDOM>': [' -DHAVE_GETRANDOM'],\n",
       " '<TIOCSWINSZ>': [' -DTIOCSWINSZ'],\n",
       " '<__UNIXWARE__>': [' -D__UNIXWARE__'],\n",
       " '<LOAD_LIBRARY_SEARCH_SYSTEM>': [' -DLOAD_LIBRARY_SEARCH_SYSTEM'],\n",
       " '<RANDOM_BUF>': [' -DRANDOM_BUF'],\n",
       " '<XML_POOR_ENTROPY>': [' -DXML_POOR_ENTROPY'],\n",
       " '<HAVE_SYSCALL_GETRANDOM>': [' -DHAVE_SYSCALL_GETRANDOM'],\n",
       " '<__SCO__>': [' -D__SCO__'],\n",
       " '<HAVE_ARC>': [' -DHAVE_ARC'],\n",
       " '<XML_DEV_URANDOM>': [' -DXML_DEV_URANDOM'],\n",
       " '<_WIN>': [' -D_WIN'],\n",
       " '<RANDOM>': [' -DRANDOM']}"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cpp_grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Part 3: C Preprocessor Configuration Fuzzing\n",
    "\n",
    "Using the grammar just produced, use a `GrammarCoverageFuzzer` to\n",
    "\n",
    "1. Test each processor variable individually\n",
    "2. Test each pair of processor variables, using `pairwise()`.\n",
    "\n",
    "What happens if you actually run the invocations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** We can simply run the coverage fuzzer, as described above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.671639Z",
     "iopub.status.busy": "2023-01-07T14:21:17.671213Z",
     "iopub.status.idle": "2023-01-07T14:21:17.673117Z",
     "shell.execute_reply": "2023-01-07T14:21:17.673526Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cc -c -DRANDOM_BUF xmlparse.c'"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g = GrammarCoverageFuzzer(cpp_grammar)\n",
    "g.fuzz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.676227Z",
     "iopub.status.busy": "2023-01-07T14:21:17.675636Z",
     "iopub.status.idle": "2023-01-07T14:21:17.677214Z",
     "shell.execute_reply": "2023-01-07T14:21:17.677423Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "from Fuzzer import ProgramRunner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.683586Z",
     "iopub.status.busy": "2023-01-07T14:21:17.683212Z",
     "iopub.status.idle": "2023-01-07T14:21:17.955965Z",
     "shell.execute_reply": "2023-01-07T14:21:17.956228Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ cc -c -D__UNIXWARE__ -DTIOCSWINSZ -DXML_POOR_ENTROPY -DHAVE_GETRANDOM xmlparse.c\n",
      "$ cc -c -D__SCO__ xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DHAVE_SYSCALL_GETRANDOM -DXML_UNICODE_WCHAR_T -DRANDOM xmlparse.c\n",
      "xmlparse.c:22:25: error: expected ')'\n",
      "int fun(int x) { return XML_T(x); }\n",
      "                        ^\n",
      "xmlparse.c:15:25: note: expanded from macro 'XML_T'\n",
      "#define XML_T(x) (const wchar_t)x\n",
      "                        ^\n",
      "xmlparse.c:22:25: note: to match this '('\n",
      "xmlparse.c:15:18: note: expanded from macro 'XML_T'\n",
      "#define XML_T(x) (const wchar_t)x\n",
      "                 ^\n",
      "xmlparse.c:22:25: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]\n",
      "int fun(int x) { return XML_T(x); }\n",
      "                        ^~~~~~~~\n",
      "xmlparse.c:15:25: note: expanded from macro 'XML_T'\n",
      "#define XML_T(x) (const wchar_t)x\n",
      "                  ~~~~~ ^\n",
      "1 warning and 1 error generated.\n",
      "$ cc -c -DLOAD_LIBRARY_SEARCH_SYSTEM -DHAVE_ARC -D_WIN xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DXML_DEV_URANDOM xmlparse.c\n",
      "$ cc -c -DHAVE_GETRANDOM -DTIOCSWINSZ xmlparse.c\n",
      "$ cc -c -DXML_DEV_URANDOM xmlparse.c\n",
      "$ cc -c -DRANDOM xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DLOAD_LIBRARY_SEARCH_SYSTEM xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DXML_DEV_URANDOM -DLOAD_LIBRARY_SEARCH_SYSTEM xmlparse.c\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    invocation = g.fuzz()\n",
    "    print(\"$\", invocation)\n",
    "    # subprocess.call(invocation, shell=True)\n",
    "    cc_runner = ProgramRunner(invocation.split(' '))\n",
    "    (result, outcome) = cc_runner.run()\n",
    "    print(result.stderr, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "To test all pairs, we can use `pairwise()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.959306Z",
     "iopub.status.busy": "2023-01-07T14:21:17.958930Z",
     "iopub.status.idle": "2023-01-07T14:21:17.960494Z",
     "shell.execute_reply": "2023-01-07T14:21:17.960853Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<XML_UNICODE_WCHAR_T><HAVE_GETRANDOM>',\n",
       " '<XML_UNICODE_WCHAR_T><TIOCSWINSZ>',\n",
       " '<XML_UNICODE_WCHAR_T><__UNIXWARE__>',\n",
       " '<XML_UNICODE_WCHAR_T><LOAD_LIBRARY_SEARCH_SYSTEM>',\n",
       " '<XML_UNICODE_WCHAR_T><RANDOM_BUF>',\n",
       " '<XML_UNICODE_WCHAR_T><XML_POOR_ENTROPY>',\n",
       " '<XML_UNICODE_WCHAR_T><HAVE_SYSCALL_GETRANDOM>',\n",
       " '<XML_UNICODE_WCHAR_T><__SCO__>',\n",
       " '<XML_UNICODE_WCHAR_T><HAVE_ARC>',\n",
       " '<XML_UNICODE_WCHAR_T><XML_DEV_URANDOM>']"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairwise_cpp_grammar = extend_grammar(cpp_grammar)\n",
    "pairwise_cpp_grammar[\"<option>\"] = pairwise(cpp_grammar[\"<option>\"])\n",
    "pairwise_cpp_grammar[\"<option>\"][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:17.968586Z",
     "iopub.status.busy": "2023-01-07T14:21:17.968212Z",
     "iopub.status.idle": "2023-01-07T14:21:18.231980Z",
     "shell.execute_reply": "2023-01-07T14:21:18.232516Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$ cc -c -DXML_POOR_ENTROPY xmlparse.c\n",
      "$ cc -c -DRANDOM_BUF -DRANDOM_BUF xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DHAVE_GETRANDOM -D__UNIXWARE__ xmlparse.c\n",
      "$ cc -c -D_WIN xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DHAVE_SYSCALL_GETRANDOM xmlparse.c\n",
      "$ cc -c -D__UNIXWARE__ -DLOAD_LIBRARY_SEARCH_SYSTEM xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DHAVE_SYSCALL_GETRANDOM xmlparse.c\n",
      "$ cc -c -DHAVE_SYSCALL_GETRANDOM xmlparse.c\n",
      "$ cc -c -DTIOCSWINSZ xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n",
      "$ cc -c -DRANDOM xmlparse.c\n",
      "xmlparse.c:7:3: error: \n",
      "# error\n",
      "  ^\n",
      "1 error generated.\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    invocation = g.fuzz()\n",
    "    print(\"$\", invocation)\n",
    "    # subprocess.call(invocation, shell=True)\n",
    "    cc_runner = ProgramRunner(invocation.split(' '))\n",
    "    (result, outcome) = cc_runner.run()\n",
    "    print(result.stderr, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Some of the compilation errors we get could be expected – for instance, defining `XML_UNICODE_WCHAR_T` when actually, the type is not supported in our environment.  Other errors may not be expected – and it is these errors we would find through systematic configuration fuzzing, as described above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "At the end, don't forget to clean up:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.235233Z",
     "iopub.status.busy": "2023-01-07T14:21:18.234849Z",
     "iopub.status.idle": "2023-01-07T14:21:18.236495Z",
     "shell.execute_reply": "2023-01-07T14:21:18.236801Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "os.remove(\"xmlparse.c\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.239159Z",
     "iopub.status.busy": "2023-01-07T14:21:18.238853Z",
     "iopub.status.idle": "2023-01-07T14:21:18.240218Z",
     "shell.execute_reply": "2023-01-07T14:21:18.240500Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "if os.path.exists(\"xmlparse.o\"):\n",
    "    os.remove(\"xmlparse.o\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "toc-hr-collapsed": true
   },
   "source": [
    "### Exercise 2: .ini Configuration Fuzzing\n",
    "\n",
    "Besides command-line options, _configuration files_ are another important source of configurations.  In this exercise, we will consider the very simple configuration language provided by the Python `ConfigParser` module, which is very similar to what is found in Microsoft Windows _.ini_ files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The following example for a `ConfigParser` input file stems right from [the ConfigParser documentation](https://docs.python.org/3/library/configparser.html):\n",
    "```\n",
    "[DEFAULT]\n",
    "ServerAliveInterval = 45\n",
    "Compression = yes\n",
    "CompressionLevel = 9\n",
    "ForwardX11 = yes\n",
    "\n",
    "[bitbucket.org]\n",
    "User = hg\n",
    "\n",
    "[topsecret.server.com]\n",
    "Port = 50022\n",
    "ForwardX11 = no\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The above `ConfigParser` file can be created programmatically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.242476Z",
     "iopub.status.busy": "2023-01-07T14:21:18.242196Z",
     "iopub.status.idle": "2023-01-07T14:21:18.243337Z",
     "shell.execute_reply": "2023-01-07T14:21:18.243681Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import configparser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.247046Z",
     "iopub.status.busy": "2023-01-07T14:21:18.246679Z",
     "iopub.status.idle": "2023-01-07T14:21:18.249146Z",
     "shell.execute_reply": "2023-01-07T14:21:18.249368Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[DEFAULT]\n",
      "serveraliveinterval = 45\n",
      "compression = yes\n",
      "compressionlevel = 9\n",
      "forwardx11 = yes\n",
      "\n",
      "[bitbucket.org]\n",
      "user = hg\n",
      "\n",
      "[topsecret.server.com]\n",
      "port = 50022\n",
      "forwardx11 = no\n",
      "\n"
     ]
    }
   ],
   "source": [
    "config = configparser.ConfigParser()\n",
    "config['DEFAULT'] = {'ServerAliveInterval': '45',\n",
    "                     'Compression': 'yes',\n",
    "                     'CompressionLevel': '9'}\n",
    "config['bitbucket.org'] = {}\n",
    "config['bitbucket.org']['User'] = 'hg'\n",
    "config['topsecret.server.com'] = {}\n",
    "topsecret = config['topsecret.server.com']\n",
    "topsecret['Port'] = '50022'     # mutates the parser\n",
    "topsecret['ForwardX11'] = 'no'  # same here\n",
    "config['DEFAULT']['ForwardX11'] = 'yes'\n",
    "with open('example.ini', 'w') as configfile:\n",
    "    config.write(configfile)\n",
    "\n",
    "with open('example.ini') as configfile:\n",
    "    print(configfile.read(), end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and be read in again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "cell_style": "center",
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.252269Z",
     "iopub.status.busy": "2023-01-07T14:21:18.251912Z",
     "iopub.status.idle": "2023-01-07T14:21:18.253831Z",
     "shell.execute_reply": "2023-01-07T14:21:18.254064Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'50022'"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config = configparser.ConfigParser()\n",
    "config.read('example.ini')\n",
    "topsecret = config['topsecret.server.com']\n",
    "topsecret['Port']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Part 1: Read Configuration\n",
    "\n",
    "Using `configparser`, create a program reading in the above configuration file and accessing the individual elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Part 2: Create a Configuration Grammar\n",
    "\n",
    "Design a grammar that will automatically create configuration files suitable for your above program.  Fuzz your program with it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Part 3: Mine a Configuration Grammar\n",
    "\n",
    "By dynamically tracking the individual accesses to configuration elements, you can again extract a basic grammar from the execution.  To this end, create a subclass of `ConfigParser` with a special method `__getitem__`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.256550Z",
     "iopub.status.busy": "2023-01-07T14:21:18.256249Z",
     "iopub.status.idle": "2023-01-07T14:21:18.257534Z",
     "shell.execute_reply": "2023-01-07T14:21:18.257722Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "class TrackingConfigParser(configparser.ConfigParser):\n",
    "    def __getitem__(self, key):\n",
    "        print(\"Accessing\", repr(key))\n",
    "        return super().__getitem__(key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For a `TrackingConfigParser` object `p`, `p.__getitem__(key)` will be invoked whenever `p[key]` is accessed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.259850Z",
     "iopub.status.busy": "2023-01-07T14:21:18.259492Z",
     "iopub.status.idle": "2023-01-07T14:21:18.261314Z",
     "shell.execute_reply": "2023-01-07T14:21:18.261587Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accessing 'topsecret.server.com'\n"
     ]
    }
   ],
   "source": [
    "tracking_config_parser = TrackingConfigParser()\n",
    "tracking_config_parser.read('example.ini')\n",
    "section = tracking_config_parser['topsecret.server.com']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Using `__getitem__()`, as above, implement a tracking mechanism that, while your program accesses the read configuration, automatically saves options accessed and values read.  Create a prototype grammar from these values; use it for fuzzing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "At the end, don't forget to clean up:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.264049Z",
     "iopub.status.busy": "2023-01-07T14:21:18.263634Z",
     "iopub.status.idle": "2023-01-07T14:21:18.265058Z",
     "shell.execute_reply": "2023-01-07T14:21:18.265598Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.267933Z",
     "iopub.status.busy": "2023-01-07T14:21:18.267632Z",
     "iopub.status.idle": "2023-01-07T14:21:18.268888Z",
     "shell.execute_reply": "2023-01-07T14:21:18.269214Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "outputs": [],
   "source": [
    "os.remove(\"example.ini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Left to the reader.  Enjoy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "toc-hr-collapsed": true
   },
   "source": [
    "### Exercise 3: Extracting and Fuzzing C Command-Line Options\n",
    "\n",
    "In C programs, the `getopt()` function are frequently used to process configuration options.  A call\n",
    "\n",
    "```\n",
    "getopt(argc, argv, \"bf:\")\n",
    "```\n",
    "\n",
    "indicates that the program accepts two options `-b` and `-f`, with `-f` taking an argument (as indicated by the following colon)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Part 1: Getopt Fuzzing\n",
    "\n",
    "Write a framework which, for a given C program, automatically extracts the argument to `getopt()` and derives a fuzzing grammar for it.  There are multiple ways to achieve this:\n",
    "\n",
    "1. Scan the program source code for occurrences of `getopt()` and return the string passed.  (Crude, but should frequently work.)\n",
    "2. Insert your own implementation of `getopt()` into the source code (effectively replacing `getopt()` from the runtime library), which outputs the `getopt()` argument and exits the program.  Recompile and run.\n",
    "3. (Advanced.) As above, but instead of changing the source code, hook into the _dynamic linker_ which at runtime links the program with the C runtime library.  Set the library loading path (on Linux and Unix, this is the `LD_LIBRARY_PATH` environment variable) such that your own version of `getopt()` is linked first, and the regular libraries later.  Executing the program (without recompiling) should yield the desired result.\n",
    "\n",
    "Apply this on `grep` and `ls`; report the resulting grammars and results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Left to the reader.  Enjoy hacking!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "#### Part 2: Fuzzing Long Options in C\n",
    "\n",
    "Same as Part 1, but also hook into the GNU variant `getopt_long()`, which accepts \"long\" arguments with double dashes such as `--help`.  Note that method 1, above, will not work here, since the \"long\" options are defined in a separately defined structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Left to the reader.  Enjoy hacking!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise 4: Expansions in Context\n",
    "\n",
    "In our above option configurations, we have multiple symbols which all expand to the same integer.  For instance, the `--line-range` option of `autopep8` takes two `<line>` parameters which both expand into the same `<int>` symbol:\n",
    "\n",
    "```\n",
    "<option> ::= ... | --line-range <line> <line> | ...\n",
    "<line> ::= <int>\n",
    "<int> ::= (-)?<digit>+\n",
    "<digit> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "cell_style": "center",
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.271658Z",
     "iopub.status.busy": "2023-01-07T14:21:18.271318Z",
     "iopub.status.idle": "2023-01-07T14:21:18.273008Z",
     "shell.execute_reply": "2023-01-07T14:21:18.273527Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<int>']"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_runner.ebnf_grammar()[\"<line>\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "cell_style": "center",
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.276063Z",
     "iopub.status.busy": "2023-01-07T14:21:18.275693Z",
     "iopub.status.idle": "2023-01-07T14:21:18.277053Z",
     "shell.execute_reply": "2023-01-07T14:21:18.277410Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['(-)?<digit>+']"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_runner.ebnf_grammar()[\"<int>\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-01-07T14:21:18.280007Z",
     "iopub.status.busy": "2023-01-07T14:21:18.279569Z",
     "iopub.status.idle": "2023-01-07T14:21:18.281361Z",
     "shell.execute_reply": "2023-01-07T14:21:18.281602Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopep8_runner.ebnf_grammar()[\"<digit>\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Once the `GrammarCoverageFuzzer` has covered all variations of `<int>` (especially by covering all digits) for _one_ option, though, it will no longer strive to achieve such coverage for the next option.  Yet, it could be desirable to achieve such coverage for each option separately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One way to achieve this with our existing `GrammarCoverageFuzzer` is again to change the grammar accordingly.  The idea is to _duplicate_ expansions – that is, to replace an expansion of a symbol $s$ with a new symbol $s'$ whose definition is duplicated from $s$.  This way, $s'$ and $s$ are separate symbols from a coverage point of view and would be independently covered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As an example, consider again the above `--line-range` option.  If we want our tests to independently cover all elements of the two `<line>` parameters, we can duplicate the second `<line>` expansion into a new symbol `<line'>` with subsequent duplicated expansions:\n",
    "```\n",
    "<option> ::= ... | --line-range <line> <line'> | ...\n",
    "<line> ::= <int>\n",
    "<line'> ::= <int'>\n",
    "<int> ::= (-)?<digit>+\n",
    "<int'> ::= (-)?<digit'>+\n",
    "<digit> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9\n",
    "<digit'> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Design a function `inline(grammar, symbol)` that returns a duplicate of `grammar` in which every occurrence of `<symbol>` and its expansions become separate copies.  The above grammar could be a result of `inline(autopep8_runner.ebnf_grammar(), \"<line>\")`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "When copying, expansions in the copy should also refer to symbols in the copy.  Hence, when expanding `<int>` in\n",
    "\n",
    "```<int> ::= <int><digit>```\n",
    "\n",
    "make that\n",
    "\n",
    "```<int> ::= <int><digit>\n",
    "<int'> ::= <int'><digit'>\n",
    "```\n",
    "\n",
    "(and not `<int'> ::= <int><digit'>` or `<int'> ::= <int><digit>`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "Be sure to add precisely one new set of symbols for each occurrence in the original grammar, and not to expand further in the presence of recursion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Again, left to the reader.  Enjoy!"
   ]
  }
 ],
 "metadata": {
  "ipub": {
   "bibliography": "fuzzingbook.bib",
   "toc": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}