{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "086cc19a-639d-441f-8489-9b822e3d5e45",
   "metadata": {
    "tags": []
   },
   "source": [
    "# CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ff8b805-53e7-40b0-a069-e2b2445ae9ff",
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import os, pathlib, tempfile, shutil, atexit, hashlib, pprint\n",
    "from IPython.display import *\n",
    "from IPython import get_ipython # needed for `jupyter_execute` because magics?\n",
    "RTD = os.environ.get(\"READTHEDOCS\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b7c1556-a706-41fc-b858-2d3b55907003",
   "metadata": {
    "tags": [
     "output_scroll"
    ]
   },
   "source": [
    "The `jupyter lite` (or `jupyter-lite`) CLI provides tools for lifecycle of combining...\n",
    "- **user-authored content** like _Notebooks_ and _Lab Extensions_.\n",
    "- the core JupyterLite **static assets** \n",
    "\n",
    "... into a **ready-to-[deploy](./deploying.md)** (and optionally **reproducible**) Jupyter sites which require no server."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "716098de-aacc-44ab-a78e-136a1a394bdb",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "674839c9-ab5c-43d1-9dda-d94554ee790d",
   "metadata": {
    "tags": [
     "output_scroll"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install jupyterlite\n",
    "!jupyter lite --version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344d0c39-0f36-493d-ba08-6c8814c05086",
   "metadata": {},
   "source": [
    "### Extras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a22683e-3c78-4271-b30a-60de50d2b5af",
   "metadata": {},
   "source": [
    "Some extra features of different _addons_ have additional dependencies.\n",
    "\n",
    "```bash\n",
    "pip install jupyterlite[contents]  # jupyter_server for better contents API\n",
    "pip install jupyterlite[serve]  # tornado for better local preview\n",
    "pip install jupyterlite[lab]  # a known-compatible jupyterlab (entails `contents` and `serve`)\n",
    "# TODO: [archive]  # use libarchive\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dce75b4-2588-40d5-a0b3-2335806d7be4",
   "metadata": {},
   "source": [
    "## The Lite Dir\n",
    "\n",
    "When you run `jupyter lite` commands, it assumes your current working directory is the partial contents of a JupyterLite site. You can override this with `--lite-dir`. By default, the built site will be created in `_output`, but can be overridden with `--output-dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fefc0961-21dd-42ad-9ec7-56b55b0ec50f",
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "if \"TMP_DIR\" not in globals():\n",
    "    TMP_DIR = pathlib.Path(tempfile.mkdtemp(prefix=\"_my_lite_dir_\"))\n",
    "    def clean():\n",
    "        shutil.rmtree(TMP_DIR)\n",
    "    atexit.register(clean)\n",
    "os.chdir(TMP_DIR)\n",
    "print(pathlib.Path.cwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "833eafef-c5fd-4e4e-b3b6-6ed5aed622d6",
   "metadata": {},
   "source": [
    "### Well-known Files\n",
    "\n",
    "Some files in your `--lite-dir` that have special meaning:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c315615d-023c-4d75-b3c8-3a51abc838bf",
   "metadata": {},
   "source": [
    "| paths                       | file                | if found                                       |\n",
    "|-----------------------------|---------------------|------------------------------------------------|\n",
    "| `.`<br/>`./lab`<br/>`./retro` | `jupyter-lite.{json,ipynb}` | merge contents with static in `_output/{path}/jupyter-lite.{json,ipynb}` |\n",
    "| `.`<br/>`./lab`<br/>`./retro` | `overrides.json`    | merge with static `_output/{*}/jupyter-lite.json`          ||\n",
    "| `./files/`                    | `*`                 | copy verbatim to `_output/files/*` and index in `/api/contents` |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37bcecbd-af66-4700-a10f-36056d2a10d5",
   "metadata": {},
   "source": [
    "## Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c11085fe-e463-4ef5-b0bf-a468e28f5913",
   "metadata": {},
   "source": [
    "### Common Parameters\n",
    "\n",
    "| parameter                | description                                                                         | default                       | environment variable        |\n",
    "| ------------------------ | ----------------------------------------------------------------------------------- | ----------------------------- | --------------------------- |\n",
    "| `--lite-dir`             | configuration and content for the site                                              | current working directory     | `JUPYTERLITE_DIR`           |\n",
    "| `--output-dir`           | where the hostable site will be created                                             | `_output`                     | `JUPYTERLITE_OUTPUT_DIR`    |\n",
    "| `--cache-dir`            | a cache directory for downloads                                                     | `<lite_dir>/.cache`           | `JUPYTERLITE_CACHE_DIR`     |\n",
    "| `--app-archive`          | an alternate site to base off of                                                    | bundled                       |                             |\n",
    "| `--files`                | directory to copy to `_output/files/` and available as _Contents_                   | `./files`                     |                             |\n",
    "| `--ignore-files`         | patterns that should _never_ be included in `/files/` (even if found in `lite-dir`) | various                       |                             |\n",
    "| `--output-archive`       | the path to the archive                                                             | `<directory>-jupyterlite.tgz` | `JUPYTERLAB_OUTPUT_ARCHIVE` |\n",
    "| `--port`                 | port on `127.0.0.1` to serve the test server                                        | `8000`                        | `JUPYTERLITE_PORT`          |\n",
    "| `--base-url`             | the URL prefix to include before the site                                           | `/`                           | `JUPYTERLITE_BASE_URL`      |\n",
    "| `--source-date-epoch`    | optionally enable additional reproducible build measures (best-effort!)             |                               | `SOURCE_DATE_EPOCH`         |\n",
    "| `--federated-extensions` | paths to folders, `pip`/`conda` packages with extensions [see note](#conda-packages)|                               |                             |\n",
    "| `--ignore-sys-prefix`    | don't copy any contents, such as install labextensions, from `sys.prefix`           | `False`                       |                             |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86cf31a3-1dc3-4c7f-a620-3655d02b548a",
   "metadata": {},
   "source": [
    "All parameters may be configured via a `jupyter_lite_config.json` in the directory where `jupyter lite` is launched, or given via `--config`. \n",
    "\n",
    "```{hint}\n",
    "For an advanced example, see the [configuration](https://github.com/jupyterlite/jupyterlite/tree/main/examples) used for this documentation.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2bf35ba-3be6-4cf0-a0b1-abb00ccbc093",
   "metadata": {},
   "source": [
    "### Help\n",
    "\n",
    "The CLI provides its own documentation, under `--help` (or `-h`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44bf9602-35ae-4a89-bc01-f8bd65567133",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdfdc1ad-a690-4dfd-b4e2-fe4cfda05443",
   "metadata": {},
   "source": [
    "### Status\n",
    "\n",
    "Always safe to run, this command provides an overview of what JupyterLite has been doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83cc3606-a0e1-4f88-bd77-12c6d2d74785",
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71523b95-096e-4ca9-b383-f453e591df6b",
   "metadata": {},
   "source": [
    "### List\n",
    "\n",
    "Always safe to run, this command provides an overview of what JupyterLite _might_ do.\n",
    "\n",
    "> _TODO: improve on default output_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e2da6c4-d1cc-4a78-a22c-ce88c9d0c46a",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fac0352-03c8-4342-aa1e-bbca4d6b9706",
   "metadata": {},
   "source": [
    "### Init\n",
    "\n",
    "Copy all the static data to the `--output-dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bff9693b-08a8-4be7-a248-dc3ee8b1055d",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite init"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deb8643d-adb7-42fa-8408-3a6d7ecd2e84",
   "metadata": {},
   "source": [
    "### Build\n",
    "\n",
    "Copy all the **user-authored content** to the `--output-dir`, and applies appropriate changes to e.g. generated Contents API responses.\n",
    "\n",
    "Special well-known files will be _merged_ appropriately, but generally, files that exist in the user directory will overwrite any existing content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a234a07b-07d7-4320-b260-f59882d456e6",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite build"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212aa2c4-6a73-494c-a872-9d75693f87e8",
   "metadata": {},
   "source": [
    "### Serve\n",
    "\n",
    "Serve the `--output-dir` on `http://127.0.0.1:{--port=8000}{--base-url=/}`.\n",
    "\n",
    "```{warning}\n",
    "This is _not_ a production server. Please consider _any_ of the [deployment](./deploying.md) options\n",
    "before trying to make this something it isn't.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afb00066-9d95-4e2d-9f43-0a350586e49d",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite serve --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b437ed9-4a8b-42cf-a8da-a1282daed39c",
   "metadata": {},
   "source": [
    "### Check\n",
    "\n",
    "Use all available mechanisms to verify that the build folder conforms to schema, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bf76f11-6a45-4c4f-bc02-3a49f39ab799",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b554754d-9c38-4f2f-9633-659c83fc8867",
   "metadata": {},
   "source": [
    "### Archive\n",
    "\n",
    "Turn the _output directory_ into a `.tgz` file. This is usually easier to move around than (sometimes) hundreds of files, and can be used as the baseline for future sites.\n",
    "\n",
    "> This command is _relatively_ expensive, and is skipped for documentation purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05fb9e31-2f5e-497c-9a17-5523da383582",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!jupyter lite archive --help\n",
    "if not RTD:\n",
    "    !jupyter lite archive"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc782c85-716b-40b6-b1bd-b4075fe41424",
   "metadata": {},
   "source": [
    "But let's talk about a more _reproducible_ asset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f933e5a-a9c5-4f09-824e-b70cc6f2fcdb",
   "metadata": {
    "tags": [
     "hide-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# we clean out the TMP_DIR for the reproducibility examples\n",
    "shutil.rmtree(TMP_DIR / \"_output\", ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6376cc0-94bf-4558-8808-824364a03cd7",
   "metadata": {},
   "source": [
    "<a id=\"reproducibility\"></a>\n",
    "\n",
    "#### Reproducible Archives\n",
    "\n",
    "> _🛠️ This feature is a **work-in-progress**, and should not be relied upon by any production workflows **Just Yet**._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5206b427-96a1-4e11-9d35-636b166a122e",
   "metadata": {},
   "source": [
    "If `--source-date-epoch` is given, a number of measures will be taken to *try to ensure* that the output of `jupyter lite archive`, an npm-compatible `tgz` package, always returns a bit-for-bit [reproducible build](https://reproducible-builds.org).\n",
    "\n",
    "The most obvious change is that the modified time of each file \"clamped\" to that time. Some other changes:\n",
    "- file ownership will be reset\n",
    "- predictable sorting will be used\n",
    "- additional checks will be applied\n",
    "\n",
    "```{note}\n",
    "This is a shortcut for setting the environment variable `SOURCE_DATE_EPOCH`:\n",
    "\n",
    "| platform         | command                                               |\n",
    "|------------------|-------------------------------------------------------|\n",
    "| Linux<br/>MacOS  | `export SOURCE_DATE_EPOCH=<a timestamp>`              |\n",
    "| Windows          | `set SOURCE_DATE_EPOCH=<a timestamp>`                 |\n",
    "| Python           | `os.environ.update(SOURCE_DATE_EPOCH, <a timestamp>)` |\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c83b952-793a-4def-9266-4371ce8a1134",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "if not \"source_date_epoch\" in globals():\n",
    "    from datetime import datetime\n",
    "    source_date_epoch = int(datetime.utcnow().timestamp())\n",
    "\n",
    "print(\"SOURCE_DATE_EPOCH is\", source_date_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "692f442b-2def-40b3-a7d4-3503c57b56a5",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "if not RTD:\n",
    "    !jupyter lite archive --source-date-epoch {source_date_epoch} --output-archive ./a.tgz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0071cee2-6ce3-4b63-969b-b35842b30e96",
   "metadata": {},
   "source": [
    "If we clear out our `_output`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdd37b3d-5018-4b38-aa50-d82fcfdc3001",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "if not RTD:\n",
    "    shutil.rmtree(TMP_DIR / \"_output\", ignore_errors=True)\n",
    "    pprint.pprint([*TMP_DIR.rglob(\"*\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "785b23e2-84e6-4dcf-8e22-ef73dae059c1",
   "metadata": {},
   "source": [
    "...and rebuild, we should always get the same file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f9027d9-ac28-4e9d-a90a-8db2835809a0",
   "metadata": {
    "tags": [
     "output_scroll",
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "if not RTD:\n",
    "    !jupyter lite archive --source-date-epoch {source_date_epoch} --output-archive ./b.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dc180b1-72ae-489d-9271-49337066062f",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "if not RTD:\n",
    "    a, b = [\n",
    "        hashlib.sha256((TMP_DIR / f\"{x}.tgz\").read_bytes()).hexdigest() \n",
    "        for x in \"ab\"\n",
    "    ]\n",
    "    print(\"We built app archives with the SHA256SUMS of:\\n\", a, \"\\n\", b)\n",
    "    try:\n",
    "        assert a == b, \"We did not reproducibly build today.\\n- {}\\n- {}\\n\\n\".format(a, b)\n",
    "    except AssertionError as err:\n",
    "        if shutil.which(\"diffoscope\"):\n",
    "            print(\"We did NOT reproducibly build today, checking in with `diffoscope`...\")\n",
    "            !diffoscope a.tgz b.tgz\n",
    "        print(\"...but at least we tried REALLY hard!\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8cc9e98-a8af-41d0-8d3b-08db7e5153be",
   "metadata": {},
   "source": [
    "## Miscellaneous"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b081fe0-7720-4327-a087-c9555ff14fb9",
   "metadata": {},
   "source": [
    "### conda packages\n",
    "\n",
    "While `--federated-extensions` support the `.tar.bz2` created by most `conda` packages, there are some issues:\n",
    "\n",
    "- `anaconda.org` uses non-standard HTTP headers to S3 buckets to provide packages\n",
    "- the `conda-forge` channel provides all of its builds as GitHub releases, and can be predictably transformed, e.g.\n",
    "\n",
    "```\n",
    "https://anaconda.org/conda-forge/jupyterlab_widgets/1.0.0/download/noarch/jupyterlab_widgets-1.0.0-pyhd8ed1ab_1.tar.bz2\n",
    "                                                                   |      |\n",
    "                                                                   |      +---------------------------------------------+\n",
    "                                                                   v      v                                             v\n",
    "         https://github.com/conda-forge/releases/releases/download/noarch/jupyterlab_widgets-1.0.0-pyhd8ed1ab_1.tar.bz2/jupyterlab_widgets-1.0.0-pyhd8ed1ab_1.tar.bz2\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-autonumbering": true,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}