{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracking inconsistencies in Jupyter notebooks\n",
    "\n",
    "This is the first in a series of multiple posts covering different aspects of Jupyter notebooks, specifically focused the issues that arise from the interaction between and invisible runtime (the \"*kernel*\") and a source that can arbitrarily be executed out of order.\n",
    "\n",
    "One of the major criticism of notebooks (see for example Joel Grus' [talk](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/preview?slide=id.g362da58057_0_1) at JupyterCon 2018 and this follow up [article](https://yihui.name/en/2018/09/notebook-war/)) by Yihui Xie is that they do not enforce execution order, ie. you can run the first cell, then the third, and finally the second, and end up in an inconsistent state.\n",
    "\n",
    "There have been several ideas surfaced to alleviate this concern, for example [turning notebooks into a dataflow programming model](https://dataflownb.github.io) which adds named identifiers to the outputs, in effect creating a DAG of cells. Alternatively, [nodebook](https://github.com/stitchfix/nodebook) serializes the entire state after the execution of each cell and enforces that re-executing a prior cell picks up from that serialized states and invalidates anything under it.\n",
    "\n",
    "The latter is interesting, in that it brings notebooks closer to the way people think about Python scripts in general, but the constraint that the entire state needs to be serialized is far too big, especially considering notebooks are mostly used to analyze large amounts of data, which would be prohibitively costly to store multiple copies of.\n",
    "\n",
    "This however gave me an idea. Instead of strictly enforcing that everything be executed from the top, what if we could use a similar technique to detect which past executions end up being inconsistent with the current state of a kernel. Conceptually, we can treat each execution as a function of the variables it reads from the environment, and mark executions as inconsistent if they operate on variables that have changed. This notebook investigates how this idea might work in practice, and what limitations it might have, using a toy example. As we'll see, this isn't a silver bullet and comes with several caveats which I'll get into towards the end.\n",
    "\n",
    "## Tracking changes\n",
    "\n",
    "The first step to implement the above is to have a solid strategy for checking whether values have changed in between executions. To that effect, I'm using `dill` because as a complex hashing function because it supports throwing a bunch of stuff at it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dill\n",
    "from typing import Any, NewType, Optional\n",
    "\n",
    "DillHash = NewType(\"DillHash\", Optional[int])\n",
    "\n",
    "\n",
    "def dill_hash(value: Any) -> DillHash:\n",
    "    try:\n",
    "        return hash(dill.dumps(value))\n",
    "    except TypeError:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will handle numbers, strings, lists, functions, etc and is stable across calls:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = [123, \"foo\", lambda x: x]\n",
    "\n",
    "for value in values:\n",
    "    assert dill_hash(value) == dill_hash(value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And will fallback on returning `None` for objects that `dill` cannot pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert dill_hash(y for y in [1, 2, 3]) == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A [spherical](https://en.wikipedia.org/wiki/Spherical_cow) kernel\n",
    "\n",
    "Since this is not meant to be a full blown implementation of the concept, rather an exploration of what is possible with this technique, we'll implement something that resembles an iPython kernel for illustration purposes. At its simplest level, this boils down to the following interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import abc\n",
    "\n",
    "\n",
    "class AbstractKernel(metaclass=abc.ABCMeta):\n",
    "    execution_count: int\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def execute(self, source: str) -> \"ExecutedCode\":\n",
    "        \"\"\"\n",
    "        Execute `source` within the context of this kernel.\n",
    "        \"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Consistency Tracking Kernel\n",
    "\n",
    "In order to track which variables are required during code execution, we'll need several building blocks:\n",
    "\n",
    "  1. A way of describing what state variables and cells are in.\n",
    "  1. An environment to store global variables and a way to check what state they are in\n",
    "  1. An abstraction to represent code that has been executed within the kernel\n",
    "  1. Some mechanism to capture which global variables a piece of code has read\n",
    "  1. Finally, a kernel that ties all of the above together.\n",
    "\n",
    "### State \n",
    "This is a ternary value because as stated above, in some cases where variables aren't hashable we might not know whether they have changed or not. We'll also want to pretty print those so let's abuse the `Enum` to store some ANSI colors in there too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30mState.Consistent\u001b[0m\n",
      "\u001b[33mState.Unknown\u001b[0m\n",
      "\u001b[31mState.Inconsistent\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from enum import Enum\n",
    "\n",
    "\n",
    "class State(Enum):\n",
    "    Consistent = \"\\033[30m\"  # black\n",
    "    Unknown = \"\\033[33m\"  # yellow\n",
    "    Inconsistent = \"\\033[31m\"  # red\n",
    "\n",
    "\n",
    "RESET = \"\\033[0m\"\n",
    "\n",
    "for state in State:\n",
    "    print(f\"{state.value}{state}{RESET}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment\n",
    "Assuming we store the environment in a `namespace` dictionary, the following method will be useful to check which state a variable is in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Namespace(dict):\n",
    "    def get_state(self, name: str, previous_hash: int) -> State:\n",
    "        \"\"\"\n",
    "        Compare the current hash of a given item name with the one provided.\n",
    "\n",
    "        If either hashes is None, we can't establish whether the variable is \n",
    "        consistent and return Unknown. Otherwise we return Consistent if both\n",
    "        hashes are identical.\n",
    "        \"\"\"\n",
    "        if name not in self:\n",
    "            return State.Inconsistent\n",
    "\n",
    "        current_hash = dill_hash(self[name])\n",
    "        if None in [current_hash, previous_hash]:\n",
    "            return State.Unknown\n",
    "\n",
    "        if current_hash == previous_hash:\n",
    "            return State.Consistent\n",
    "        else:\n",
    "            return State.Inconsistent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Executed Code\n",
    "Next we'll need something representing a piece of code we executed within the context of a kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textwrap import dedent\n",
    "from typing import Dict\n",
    "\n",
    "\n",
    "class ExecutedCode:\n",
    "    \"\"\"\n",
    "    Abstraction capturing the code that was run in a given call to \n",
    "    alongside its dependencies and at which index it was executed\n",
    "    \"\"\"\n",
    "\n",
    "    source: str\n",
    "    dependencies: Dict[str, DillHash]\n",
    "    execution_order: int\n",
    "\n",
    "    def __init__(self, source: str, execution_order: int) -> None:\n",
    "        self.source = dedent(source).strip()\n",
    "        self.execution_order = execution_order\n",
    "        self.dependencies = {}\n",
    "\n",
    "    def add_dependency(self, name: str, value: Any) -> None:\n",
    "        self.dependencies[name] = dill_hash(value)\n",
    "\n",
    "    def get_state(self, namespace: Namespace) -> State:\n",
    "        \"\"\"\n",
    "        Aggregates variable level state into code level state:\n",
    "        Inconsistent wins over Unknown wins over Consistent.\n",
    "        \"\"\"\n",
    "        if not self.dependencies:\n",
    "            return State.Consistent\n",
    "\n",
    "        return max(\n",
    "            (\n",
    "                namespace.get_state(name, previous_hash=value_hash)\n",
    "                for name, value_hash in self.dependencies.items()\n",
    "            ),\n",
    "            key=list(State).index,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Namespace Proxy\n",
    "\n",
    "In order to know which variables are accessed when code is being executed, we have to wrap our `Namespace` in some kind of observer that can keep track of reads and accordingly update the relevant input code's dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NamespaceProxy(dict):\n",
    "    \"\"\"\n",
    "    A dictionary subclass that tracks when variables are read and updates the relevant `Code`\n",
    "    \"\"\"\n",
    "\n",
    "    namespace: Namespace\n",
    "    executed_code: ExecutedCode\n",
    "\n",
    "    def __init__(self, namespace: Namespace, executed_code: ExecutedCode) -> None:\n",
    "        self.namespace = namespace\n",
    "        self.executed_code = executed_code\n",
    "\n",
    "    def __getitem__(self, name: str) -> Any:\n",
    "        value = self.namespace[name]\n",
    "        self.executed_code.add_dependency(name, value)\n",
    "        return value\n",
    "\n",
    "    def __setitem__(self, name: str, value: Any) -> None:\n",
    "        self.namespace[name] = value\n",
    "\n",
    "    def __delitem__(self, name: str) -> None:\n",
    "        del self.namespace[name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kernel\n",
    "\n",
    "Finally we have all the ingredients to define our kernel as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Dict, List, NamedTuple, Tuple\n",
    "\n",
    "\n",
    "class Kernel(AbstractKernel):\n",
    "    \"\"\"\n",
    "    A Kernel implementation that keeps track of inconsistent code executions.\n",
    "    \"\"\"\n",
    "\n",
    "    execution_count: int\n",
    "    namespace: Namespace\n",
    "\n",
    "    def __init__(self) -> None:\n",
    "        self.execution_count = 0\n",
    "        self.namespace = Namespace()\n",
    "\n",
    "    def execute(self, source: str) -> ExecutedCode:\n",
    "        executed_code = ExecutedCode(source, execution_order=self.execution_count)\n",
    "\n",
    "        exec(executed_code.source, NamespaceProxy(self.namespace, executed_code))\n",
    "        self.execution_count += 1\n",
    "\n",
    "        return executed_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example\n",
    "\n",
    "Let's consider the following very simple example: we first assign `x`, assign twice its value to `y`, and then update `x`. As one would expect, once the third line has been executed, the second one becomes inconsistent, because at that point `x == 2` and `y == 2` which is different from `2 * x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30mx = 1\u001b[0m\n",
      "\u001b[31my = 2 * x\u001b[0m\n",
      "\u001b[30mx = 2\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "kernel = Kernel()\n",
    "\n",
    "for executed_code in [\n",
    "    kernel.execute(\"x = 1\"),\n",
    "    kernel.execute(\"y = 2 * x\"),\n",
    "    kernel.execute(\"x = 2\"),\n",
    "]:\n",
    "    print(f\"{executed_code.get_state(kernel.namespace).value}{executed_code.source}{RESET}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tying things into a \"Notebook\"\n",
    "\n",
    "The confusing part of executing isn't so much the fact that the line above might become inconsistent, after all the above *is* idiomatic Python. What usually trips users up is when cells are executed out of order, for example:\n",
    "\n",
    "    [0] x = 1\n",
    "    [1] print(x)\n",
    "    1\n",
    "\n",
    "And the user goes back and edits/executes a previous cell, such as:\n",
    "\n",
    "    [2] x = 2\n",
    "    [1] print(x)\n",
    "    1\n",
    "    \n",
    "In order to illustrate this, let's come up with a very simple model of a notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textwrap import indent\n",
    "\n",
    "\n",
    "class Cell:\n",
    "    executed_code: ExecutedCode\n",
    "    state: State\n",
    "\n",
    "    def __init__(self, executed_code: ExecutedCode) -> None:\n",
    "        self.executed_code = executed_code\n",
    "        self.state = State.Consistent\n",
    "\n",
    "    def display(self) -> None:\n",
    "        execution_order_prefix = f\"[{self.executed_code.execution_order}] \"\n",
    "        formatted_source = indent(self.executed_code.source, len(execution_order_prefix) * \" \").strip()\n",
    "        print(f\"{self.state.value}{execution_order_prefix}{formatted_source}{RESET}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which we can use to render a cell containing a code block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[2] def square(x: int) -> int:\n",
      "        return x * x\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "Cell(\n",
    "    ExecutedCode(\n",
    "        \"\"\"\n",
    "        def square(x: int) -> int:\n",
    "            return x * x\n",
    "        \"\"\",\n",
    "        execution_order=2,\n",
    "    )\n",
    ").display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can assemble several such cells in a notebook which has an associated kernel where we'll execute their code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Notebook:\n",
    "    \"\"\"\n",
    "    A notebook is a collection of cells and a kernel\n",
    "    \"\"\"\n",
    "\n",
    "    cells: List[Cell]\n",
    "    kernel: Kernel\n",
    "\n",
    "    def __init__(self, *sources: str) -> None:\n",
    "        self.cells = []\n",
    "        self.kernel = Kernel()\n",
    "\n",
    "        for source in sources:\n",
    "            self.add_cell(source)\n",
    "\n",
    "    def add_cell(self, source: str) -> None:\n",
    "        self.cells.append(Cell(executed_code=self.kernel.execute(source)))\n",
    "\n",
    "    def update_cell(self, index: int, source: str) -> None:\n",
    "        self.cells[index] = Cell(executed_code=self.kernel.execute(source))\n",
    "\n",
    "        for cell in self.cells[index + 1 :]:\n",
    "            cell.state = cell.executed_code.get_state(self.kernel.namespace)\n",
    "\n",
    "    def reexecute_cell(self, index: int) -> None:\n",
    "        self.update_cell(index, self.cells[index].executed_code.source)\n",
    "\n",
    "    def display(self):\n",
    "        for cell in self.cells:\n",
    "            cell.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going back to the example above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[0] x = 1\u001b[0m\n",
      "\u001b[30m[1] y = x\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb = Notebook(\n",
    "    \"x = 1\", \n",
    "    \"y = x\",\n",
    ")\n",
    "nb.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we update the first cell to `x = 2`, then we end up with an inconsistent second cell, because `y != x` after the execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[2] x = 2\u001b[0m\n",
      "\u001b[31m[1] y = x\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb.update_cell(0, \"x = 2\")\n",
    "nb.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, if instead of starting with a number we start with `x` being a generator, which we can't serialize, we can surface that we're now in an unknown state (ie. it might be consistent, but we can't ensure it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[2] x = (y for y in [1, 2, 3])\u001b[0m\n",
      "\u001b[33m[1] z = x\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb2 = Notebook(\n",
    "    \"x = (y for y in [1, 2, 3])\", \n",
    "    \"z = x\",\n",
    ")\n",
    "nb2.reexecute_cell(0)\n",
    "nb2.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a slightly more convoluted example, where the variable itself doesn't change, but its value does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[0] counters = {'a': 0, 'b': 1}\u001b[0m\n",
      "\u001b[30m[1] counters['a'] += 1\u001b[0m\n",
      "\u001b[30m[2] x = counters['a']\u001b[0m\n",
      "\u001b[30m[3] y = 1 + 1\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb3 = Notebook(\n",
    "    \"counters = {'a': 0, 'b': 1}\",\n",
    "    \"counters['a'] += 1\",\n",
    "    \"x = counters['a']\",\n",
    "    \"y = 1 + 1\",\n",
    ")\n",
    "nb3.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we now execute cell `1` again, then cell `2` becomes inconsistent, because `x == 1`, whereas `counters['a'] == 2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[0] counters = {'a': 0, 'b': 1}\u001b[0m\n",
      "\u001b[30m[4] counters['a'] += 1\u001b[0m\n",
      "\u001b[31m[2] x = counters['a']\u001b[0m\n",
      "\u001b[30m[3] y = 1 + 1\u001b[0m\n",
      "\n",
      "x: 1\n",
      "counters['a']: 2\n"
     ]
    }
   ],
   "source": [
    "nb3.reexecute_cell(1)\n",
    "nb3.display()\n",
    "\n",
    "print(\"\")\n",
    "print(\"x:\", nb3.kernel.namespace[\"x\"])\n",
    "print(\"counters['a']:\", nb3.kernel.namespace[\"counters\"][\"a\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caveats and limitations\n",
    "\n",
    "Like I mentioned earlier, there are quite a few limitations to this approach:\n",
    "\n",
    "- It requires objects to be picklable to reliably assess whether cells are consistent or not. Introducing a concept of `Unknown` consistency somehow mitigates that, in that there is at least a fallback behavior even if not totally satisfactory. \n",
    "\n",
    "- It assumes that cells are deterministic, which excludes a large swath of use cases where randomness is involved. \n",
    "- This method will err on the side of inconsistency. Consider the following example, because the dictionary has changed in between calls, we're now marking the last cell as inconsistent, even though the output is the same. I don't claim to have a solution for this, but I would still argue in favor of at least warning the user that that cell **might** be in an inconsistent state because `d` changed.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[0] d = {1: 2}\u001b[0m\n",
      "\u001b[30m[3] d[2] = 4\u001b[0m\n",
      "\u001b[31m[2] x = d[1]\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb4 = Notebook(\n",
    "    \"d = {1: 2}\", \n",
    "    \"d[2] = 3\", \n",
    "    \"x = d[1]\",\n",
    ")\n",
    "nb4.update_cell(1, \"d[2] = 4\")\n",
    "\n",
    "nb4.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Because we are using `dill.dumps` to compute the value hash, the overhead on larger data structures might be prohibitively expensive (think gigabyte sized dataframes)\n",
    "- Some might prefer a different definition of consistency, for example that inconsistencies should propagate, that is in the following case, both cells `1` and `2` should be inconsistent. This is slightly out of scope as this notebook is getting rather long, but it can be mitigated by having `Code` track both its inputs and outputs, and propagate state along those edges. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[30m[3] x = 2\u001b[0m\n",
      "\u001b[31m[1] y = x + 1\u001b[0m\n",
      "\u001b[30m[2] z = y + 1\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "nb5 = Notebook(\n",
    "    \"x = 1\", \n",
    "    \"y = x + 1\", \n",
    "    \"z = y + 1\",\n",
    ")\n",
    "nb5.update_cell(0, \"x = 2\")\n",
    "nb5.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We've explored using hashing to keep track of dependencies between cells in a notebook and its associated kernel, and we've shown this method to be effective on toy examples. While implementing this in Jupyter could provide value to the user - one could imagine building a plugin surfacing the likely state of a cell right next to it, we've also surfaced a number of limitations that might prevent it from being the most efficient solution to the problem of keeping track of out of order executions in a notebook. \n",
    "\n",
    "In a subsequent note, we'll take a look at a different strategy, leveraging the OS itself to take care of some of the grunt work for us.\n",
    "\n",
    " \n",
    "\n",
    "*Are you excited about this kind of work? Then you should [reach out](mailto:adrien@mlon.com), I'm starting [mlon](https://mlon.com) to build better tools for data science and AI.*"
   ]
  }
 ],
 "metadata": {
  "_draft": {
   "nbviewer_url": "https://gist.github.com/e8eebdbb8544d71c85634f053a6258ed"
  },
  "gist": {
   "data": {
    "description": "notebooks/tracking_inconsistencies_in_notebooks.ipynb",
    "public": false
   },
   "id": "e8eebdbb8544d71c85634f053a6258ed"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}