{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`importnb` and [`nbval`](https://github.com/computationalmodelling/nbval)\n",
    "\n",
    "`importnb, pidgin` are complimented nicely by `nbval`.  `nbval` + `importnb` performs\n",
    "normal test discovery AND compares the out results.  Doctests may be included also. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "    with __import__('IPython').utils.capture.capture_output(stdout=True):\n",
    "        if __name__ == '__main__':\n",
    "            no_nbval = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --collect-only\n",
    "            with_nbval = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --nbval --collect-only\n",
    "            with_nbval_doctests = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --doctest-modules --nbval --collect-only\n",
    "            no_nbval_doctests = !ipython -m pytest -- 2018-11-23-importnb-and-nbval.ipynb --doctest-modules --collect-only\n",
    "\n",
    "            for report in (no_nbval, with_nbval, with_nbval_doctests, no_nbval_doctests):\n",
    "                while not report[0].startswith('collected'): report.pop(0)\n",
    "                while report[-1].startswith('==='): report.pop()\n",
    "                while not report[-1].strip(): report.pop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Include a function that is found with normal `pytest` discovery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def test_function(): \"\"\"This will not be tested.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Include a `doctest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def a_function_with_a_doctest(): \"\"\">>> assert True\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget about the underutilized [`__test__` `object` recognized by `doctest`](https://docs.python.org/2/library/doctest.html#which-docstrings-are-examined)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "    __test__ = {'another doctest': \">>> assert 100\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assert that __test__ is found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "    if __name__ == '__main__': \n",
    "        assert '.__test__.' in ''.join(with_nbval_doctests)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'There are 8 code cells.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    with open('2018-11-23-importnb-and-nbval.ipynb') as file: cells = len(\n",
    "        list(object for object in __import__('nbformat').read(file, 4)['cells']\n",
    "             if object['cell_type'] == 'code' and ''.join(object['source']).strip()))\n",
    "        \n",
    "    F\"\"\"There are {cells} code cells.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DoctestItem</th>\n",
       "      <th>Function</th>\n",
       "      <th>IPyNbCell</th>\n",
       "      <th>IPyNbFile</th>\n",
       "      <th>NotebookModule</th>\n",
       "      <th>total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>(all, collected 11 items)</th>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(doctests, collected 3 items)</th>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(importnb, collected 1 item)</th>\n",
       "      <td></td>\n",
       "      <td>1.0</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(nbval, collected 9 items)</th>\n",
       "      <td></td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              DoctestItem  Function IPyNbCell IPyNbFile  \\\n",
       "(all, collected 11 items)               2       1.0         8         1   \n",
       "(doctests, collected 3 items)           2       1.0                       \n",
       "(importnb, collected 1 item)                    1.0                       \n",
       "(nbval, collected 9 items)                      1.0         8         1   \n",
       "\n",
       "                               NotebookModule  total  \n",
       "(all, collected 11 items)                 1.0   13.0  \n",
       "(doctests, collected 3 items)             1.0    4.0  \n",
       "(importnb, collected 1 item)              1.0    2.0  \n",
       "(nbval, collected 9 items)                1.0   11.0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "    if __name__ == '__main__':\n",
    "        import pandas\n",
    "\n",
    "        df = pandas.DataFrame(list(map(list, \"\"\"100\\n110\\n111\\n101\"\"\".splitlines())), columns=[\n",
    "            'importnb', 'nbval', 'doctests' \n",
    "        ], index=[\n",
    "            'importnb', 'nbval', 'all', 'doctests'\n",
    "        ]).pipe(lambda df: df.set_index(list(df.columns), append=True))\n",
    "\n",
    "        collected = pandas.concat(dict(zip(df.index.get_level_values(0), [\n",
    "            pandas.Series(object)\n",
    "            for object in (no_nbval, with_nbval, with_nbval_doctests, no_nbval_doctests)\n",
    "        ]))).unstack().rename(columns={0:'collected'}).set_index('collected', append=True)\n",
    "\n",
    "        collected = collected.stack(\n",
    "        ).str.split(expand=True)[0].str.lstrip('<').to_frame(\n",
    "            'object'\n",
    "        ).reset_index(-1, drop=True)['object'].pipe(\n",
    "            lambda df: df.groupby([df.index, df.values]).count()\n",
    "        ).unstack().pipe(\n",
    "            lambda df: df.assign(total=df.sum(axis=1))\n",
    "        ).fillna('')\n",
    "        collected.pipe(display)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The difference in the numbers of tests is the 8 code cells plus the number of modules or files. '"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    F\"The difference in the numbers of tests is the {cells} code cells plus the number of modules or files. \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>--nbval --doctest-modules</code> & `importnb` ensure that a notebook can be tested in a quite a few ways.  An author should not be limited in their mode of testing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be great ideal to focus on notebooks as tests rather than experimentation.  Early in the ideation process there is a little difference between source code and test code.  There are tools in the notebook ecosystem that seamlessly work with a stack of existing notebooks.  Using the suite of `pytest` notebook testing tools `nbval` and `importnb` will assist authors in writing more durable notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}