{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "#default_exp sync\n",
    "#default_cls_lvl 3\n",
    "from nbdev.showdoc import show_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "#export\n",
    "from nbdev.imports import *\n",
    "from nbdev.export import *\n",
    "from fastcore.script import *\n",
    "import nbformat\n",
    "from nbformat.sign import NotebookNotary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Synchronize and diff\n",
    "\n",
    "> The functions that propagates small changes in the library back to notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The library is primarily developed in notebooks so any big changes should be made there. But sometimes, it's easier to fix small bugs or typos in the modules directly. `nbdev_update_lib` is the function that will propagate those changes back to the corresponding notebooks. Note that you can't create new cells with that functionality, so your corrections should remain limited."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the way back to notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to get the name of the object we are looking for, and then we'll try to find it in our index file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def _get_property_name(p):\n",
    "    \"Get the name of property `p`\"\n",
    "    if hasattr(p, 'fget'):\n",
    "        return p.fget.func.__qualname__ if hasattr(p.fget, 'func') else p.fget.__qualname__\n",
    "    else: return next(iter(re.findall(r'\\'(.*)\\'', str(p)))).split('.')[-1]\n",
    "\n",
    "def get_name(obj):\n",
    "    \"Get the name of `obj`\"\n",
    "    if hasattr(obj, '__name__'):       return obj.__name__\n",
    "    elif getattr(obj, '_name', False): return obj._name\n",
    "    elif hasattr(obj,'__origin__'):    return str(obj.__origin__).split('.')[-1] #for types\n",
    "    elif type(obj)==property:          return _get_property_name(obj)\n",
    "    else:                              return str(obj).split('.')[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nbdev.export import DocsTestClass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(get_name(in_ipython), 'in_ipython')\n",
    "test_eq(get_name(DocsTestClass.test), 'test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def qual_name(obj):\n",
    "    \"Get the qualified name of `obj`\"\n",
    "    if hasattr(obj,'__qualname__'): return obj.__qualname__\n",
    "    if inspect.ismethod(obj):       return f\"{get_name(obj.__self__)}.{get_name(fn)}\"\n",
    "    return get_name(obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Qualified name is different from name in python for methods and properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(qual_name(DocsTestClass.test), 'DocsTestClass.test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "class _PropertyClass:\n",
    "    p_lambda = property(lambda x: x)\n",
    "    def some_getter(self): return 7\n",
    "    p_getter = property(some_getter)\n",
    "\n",
    "test_eq(get_name(_PropertyClass.p_lambda), '_PropertyClass.<lambda>')\n",
    "test_eq(get_name(_PropertyClass.p_getter), '_PropertyClass.some_getter')\n",
    "test_eq(get_name(_PropertyClass), '_PropertyClass')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def source_nb(func, is_name=None, return_all=False, mod=None):\n",
    "    \"Return the name of the notebook where `func` was defined\"\n",
    "    is_name = is_name or isinstance(func, str)\n",
    "    if mod is None: mod = get_nbdev_module()\n",
    "    index = mod.index\n",
    "    name = func if is_name else qual_name(func)\n",
    "    while len(name) > 0:\n",
    "        if name in index: return (name,index[name]) if return_all else index[name]\n",
    "        name = '.'.join(name.split('.')[:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can either pass an object or its name (by default `is_name` will look if `func` is a string or not to determine if it should be `True` or `False`, but you can override if there is some inconsistent behavior). \n",
    "\n",
    "If passed a method of a class, the function will return the notebook in which the largest part of the function name was defined in case there is a monkey-matching that defines `class.method` in a different notebook than `class`. If `return_all=True`, the function will return a tuple with the name by which the function was found and the notebook.\n",
    "\n",
    "For properties defined using `property` or our own `add_props` helper, we approximate the name by looking at their getter functions, since we don't seem to have access to the property name itself. If everything fails (a getter cannot be found), we return the name of the object that contains the property. This suffices for `source_nb` to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(source_nb(notebook2script), '00_export.ipynb')\n",
    "test_eq(source_nb(DocsTestClass), '00_export.ipynb')\n",
    "test_eq(source_nb(DocsTestClass.test), '00_export.ipynb')\n",
    "assert source_nb(int) is None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If someone decides to change a module instead of the notebooks, the following functions help update the notebooks accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "_re_cell = re.compile(r'^# Cell|^# Internal Cell|^# Comes from\\s+(\\S+), cell')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def _split(code):\n",
    "    lines = code.split('\\n')\n",
    "    nbs_path = Config().path(\"nbs_path\").relative_to(Config().config_file.parent)\n",
    "    prefix = '' if nbs_path == Path('.') else f'{nbs_path}/'\n",
    "    default_nb = re.search(f'File to edit: {prefix}(\\\\S+)\\\\s+', lines[0]).groups()[0]\n",
    "    s,res = 1,[]\n",
    "    while _re_cell.search(lines[s]) is None: s += 1\n",
    "    e = s+1\n",
    "    while e < len(lines):\n",
    "        while e < len(lines) and _re_cell.search(lines[e]) is None: e += 1\n",
    "        grps = _re_cell.search(lines[s]).groups()\n",
    "        nb = grps[0] or default_nb\n",
    "        content = lines[s+1:e]\n",
    "        while len(content) > 1 and content[-1] == '': content = content[:-1]\n",
    "        res.append((nb, '\\n'.join(content)))\n",
    "        s,e = e,e+1\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def relimport2name(name, mod_name):\n",
    "    \"Unwarps a relative import in `name` according to `mod_name`\"\n",
    "    mod_name = str(Path(mod_name))\n",
    "    if mod_name.endswith('.py'): mod_name = mod_name[:-3]\n",
    "    mods = mod_name.split(os.path.sep)\n",
    "    i = last_index(Config().lib_name, mods)\n",
    "    mods = mods[i:]\n",
    "    if name=='.': return '.'.join(mods[:-1])\n",
    "    i = 0\n",
    "    while name[i] == '.': i += 1\n",
    "    return '.'.join(mods[:-i] + [name[i:]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we say from \n",
    "``` python\n",
    "from .submodule import bla\n",
    "``` \n",
    "in a module, it needs to be converted to something like \n",
    "``` python\n",
    "from module.submodule import bla\n",
    "```\n",
    "or \n",
    "``` python\n",
    "from module1.module2.submodule import bla\n",
    "``` \n",
    "depending on where we are. This function deals with those imports renaming."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eq(relimport2name('.core', 'nbdev/data.py'), 'nbdev.core')\n",
    "test_eq(relimport2name('.core', 'home/sgugger/fastai_dev/nbdev/nbdev/data.py'), 'nbdev.core')\n",
    "test_eq(relimport2name('..core', 'nbdev/vision/data.py'), 'nbdev.core')\n",
    "test_eq(relimport2name('.transform', 'nbdev/vision/data.py'), 'nbdev.vision.transform')\n",
    "test_eq(relimport2name('..notebook.core', 'nbdev/data/external.py'), 'nbdev.notebook.core')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "#Catches any from .bla import something and catches .bla in group 1, the imported thing(s) in group 2.\n",
    "_re_loc_import = re.compile(r'(\\s*)from(\\s+)(\\.\\S*)(\\s+)import(\\s+)(.*)$')\n",
    "_re_loc_import1 = re.compile(r'(\\s*)import(\\s+)(\\.\\S*)(.*)$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def _deal_loc_import(code, fname):\n",
    "    def _replace(m):\n",
    "        s1,s2,mod,s3,s4,obj = m.groups()\n",
    "        return f\"{s1}from{s2}{relimport2name(mod, fname)}{s3}import{s4}{obj}\"\n",
    "    def _replace1(m):\n",
    "        s1,s2,mod,end = m.groups()\n",
    "        return f\"{s1}import{s2}{relimport2name(mod, fname)}{end}\"\n",
    "    return '\\n'.join([_re_loc_import1.sub(_replace1, _re_loc_import.sub(_replace,line)) for line in code.split('\\n')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "code = \\\n",
    "\"\"\"from  .core   import *\n",
    "nothing to see\n",
    "  \\t from \\t .vision \\t import \\t bla1, \\nbla2\n",
    "import .vision\n",
    "import .utils as u\"\"\"\n",
    "test_eq(_deal_loc_import(code, 'nbdev/data.py'), \n",
    "\"\"\"from  nbdev.core   import *\n",
    "nothing to see\n",
    "  \\t from \\t nbdev.vision \\t import \\t bla1, \\nbla2\n",
    "import nbdev.vision\n",
    "import nbdev.utils as u\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def _script2notebook(fname, dic, silent=False):\n",
    "    \"Put the content of `fname` back in the notebooks it came from.\"\n",
    "    if os.environ.get('IN_TEST',0): return  # don't export if running tests\n",
    "    fname = Path(fname)\n",
    "    with open(fname, encoding='utf8') as f: code = f.read()\n",
    "    splits = _split(code)\n",
    "    rel_name = fname.absolute().resolve().relative_to(Config().path(\"lib_path\"))\n",
    "    key = str(rel_name.with_suffix(''))\n",
    "    assert len(splits)==len(dic[key]), f'\"{rel_name}\" exported from notebooks should have {len(dic[key])} cells but has {len(splits)}.'\n",
    "    assert all([c1[0]==c2[1]] for c1,c2 in zip(splits, dic[key]))\n",
    "    splits = [(c2[0],c1[0],c1[1]) for c1,c2 in zip(splits, dic[key])]\n",
    "    nb_fnames = {Config().path(\"nbs_path\")/s[1] for s in splits}\n",
    "    for nb_fname in nb_fnames:\n",
    "        nb = read_nb(nb_fname)\n",
    "        for i,f,c in splits:\n",
    "            c = _deal_loc_import(c, str(fname))\n",
    "            if f == nb_fname.name:\n",
    "                flags = split_flags_and_code(nb['cells'][i], str)[0]\n",
    "                nb['cells'][i]['source'] = flags + '\\n' + c.replace('', '')\n",
    "        NotebookNotary().sign(nb)\n",
    "        nbformat.write(nb, str(nb_fname), version=4)\n",
    "\n",
    "    if not silent: print(f\"Converted {rel_name}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#hide\n",
    "\n",
    "Some things are written to the library with a `#nbdev_comment ` comment prefix. e.g. The following cell:\n",
    "\n",
    "```python\n",
    "#export\n",
    "def _not_included_by_default(): pass\n",
    "_all_=[_not_included_by_default]\n",
    "```\n",
    "\n",
    "would get written to `{module}.py` as:\n",
    "\n",
    "```python\n",
    "def _not_included_by_default(): pass\n",
    "#nbdev_comment _all_=[_not_included_by_default]\n",
    "```\n",
    "\n",
    "In this case\n",
    "- `_all_=[_not_included_by_default]` should not be part of the module\n",
    "- but we can't just remove it, or we would lose code when writing back to notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted export.py.\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "dic = notebook2script(silent=True, to_dict=True)\n",
    "_script2notebook(Config().path(\"lib_path\")/'export.py', dic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "exported = get_nbdev_module().modules\n",
    "files = nbglob(extension='.py', config_key='lib_path')\n",
    "files = files.filter(lambda x: str(x.relative_to(Config().path(\"lib_path\"))) in exported)\n",
    "test_eq(len(files) > 0, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "@call_parse\n",
    "def nbdev_update_lib(fname:Param(\"A python filename or glob to convert\", str)=None,\n",
    "                     silent:Param(\"Don't print results\", bool_arg)=False):\n",
    "    \"Propagates any change in the modules matching `fname` to the notebooks that created them\"\n",
    "    if fname and fname.endswith('.ipynb'): raise ValueError(\"`nbdev_update_lib` operates on .py files.  If you wish to convert notebooks instead, see `nbdev_build_lib`.\")\n",
    "    if os.environ.get('IN_TEST',0): return\n",
    "    dic = notebook2script(silent=True, to_dict=True)\n",
    "    exported = get_nbdev_module().modules\n",
    "    \n",
    "    files = nbglob(fname, extension='.py', config_key='lib_path')\n",
    "    files = files.filter(lambda x: str(x.relative_to(Config().path(\"lib_path\"))) in exported)\n",
    "    files.map(partial(_script2notebook, dic=dic, silent=silent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `fname` is not specified, this will convert all modules and submodules in the `lib_folder` defined in `setting.ini`. Otherwise `fname` can be a single filename or a glob expression.\n",
    "\n",
    "`silent` makes the command not print any statement. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "#nbdev_update_lib()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Diff & trust notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before making a commit, you may want to check there is no diff between the exported library and the notebooks. You may also want to make this part of your CI, so that you don't accidentally merge a PR that introduces some changes between the two. This function is there to print this diff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import subprocess\n",
    "from distutils.dir_util import copy_tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "@call_parse\n",
    "def nbdev_diff_nbs():\n",
    "    \"Prints the diff between an export of the library in notebooks and the actual modules\"\n",
    "    cfg = Config()\n",
    "    lib_folder = cfg.path(\"lib_path\")\n",
    "    with tempfile.TemporaryDirectory() as d1, tempfile.TemporaryDirectory() as d2:\n",
    "        copy_tree(cfg.path(\"lib_path\"), d1)\n",
    "        notebook2script(silent=True)\n",
    "        copy_tree(cfg.path(\"lib_path\"), d2)\n",
    "        shutil.rmtree(cfg.path(\"lib_path\"))\n",
    "        shutil.copytree(d1, str(cfg.path(\"lib_path\")))\n",
    "        for d in [d1, d2]:\n",
    "            if (Path(d)/'__pycache__').exists(): shutil.rmtree(Path(d)/'__pycache__')\n",
    "        res = subprocess.run(['diff', '-ru', d1, d2], stdout=subprocess.PIPE)\n",
    "        print(res.stdout.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nbdev_diff_nbs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you receive an output, you'll need to either run `notebook2script()` or `nbdev_update_lib()` to fix the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "@call_parse\n",
    "def nbdev_trust_nbs(fname:Param(\"A notebook name or glob to convert\", str)=None,\n",
    "                    force_all:Param(\"Trust even notebooks that haven't changed\", bool)=False):\n",
    "    \"Trust notebooks matching `fname`\"\n",
    "    check_fname = Config().path(\"nbs_path\")/\".last_checked\"\n",
    "    last_checked = os.path.getmtime(check_fname) if check_fname.exists() else None\n",
    "    files = nbglob(fname)\n",
    "    for fn in files:\n",
    "        if last_checked and not force_all:\n",
    "            last_changed = os.path.getmtime(fn)\n",
    "            if last_changed < last_checked: continue\n",
    "        nb = read_nb(fn)\n",
    "        if not NotebookNotary().check_signature(nb): NotebookNotary().sign(nb)\n",
    "    check_fname.touch(exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted 00_export.ipynb.\n",
      "Converted 01_sync.ipynb.\n",
      "Converted 02_showdoc.ipynb.\n",
      "Converted 03_export2html.ipynb.\n",
      "Converted 04_test.ipynb.\n",
      "Converted 05_merge.ipynb.\n",
      "Converted 06_cli.ipynb.\n",
      "Converted 07_clean.ipynb.\n",
      "Converted 99_search.ipynb.\n",
      "Converted example.ipynb.\n",
      "Converted index.ipynb.\n",
      "Converted nbdev_comments.ipynb.\n",
      "Converted tutorial.ipynb.\n",
      "Converted tutorial_colab.ipynb.\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "from nbdev.export import *\n",
    "notebook2script()"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}