{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Banner](../media/banner2.png)\n",
"\n",
"---\n",
"# Workshop 2.1: Jupyter Notebooks Advanced\n",
"\n",
"* **Contributors**:\n",
" * Ashwin Patil (@ashwinpatil)\n",
" * Luis Francisco Monge Martinez (@LuckyLuke)\n",
" * Ian Hellen (@ianhellen)\n",
"
\n",
"* **Agenda**:\n",
" * [Jupyter is not just Python](#notjustpython)\n",
" * [Jupyter Kernels & Python environments](#kernels)\n",
" * [Magics](#magics)\n",
" * [Widgets introduction](#widgets)[\n",
" * [Jupyter Extensions](#extensions)\n",
" * [Export and create notebooks](#nbconvert)\n",
" * [Dev topics - Debugging and testing notebook code](#debugging)\n",
"
\n",
"* **Notebook**: [https://aka.ms/Jupyterthon-ws-2-1](https://aka.ms/Jupyterthon-ws-2-1)\n",
"* **License**: [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/)\n",
"\n",
"* **Q&A** - OTR Discord **#Jupyterthon #WORKSHOP DAY 2 - JUPYTER ADVANCED**"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"---\n",
"\n",
"# Jupyter is not just Python [Ashwin]\n",
"- Powershell kernel\n",
"- R kernel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Jupyter Kernels & Python environments\n",
"\n",
"Python environments let you create \"isolated\" installations with independent versions of packages.\n",
"\n",
"This is usually **A VERY GOOD IDEA**!\n",
"\n",
"Linux\n",
"\n",
"```bash\n",
"python -m venv MyNewEnv\n",
"source ./MyNewEnv/Scripts/activate\n",
"pip install msticpy\n",
"```\n",
"\n",
"Windows\n",
"\n",
"```cmd\n",
"python -m venv MyNewEnv\n",
".\\MyNewEnv\\Scripts\\activate\n",
"pip install msticpy\n",
"```\n",
"\n",
"Conda\n",
"\n",
"```bash\n",
"conda create -n MyNewCondaEnv\n",
"conda activate MyNewCondaEnv\n",
"conda install pip\n",
"pip install msticpy\n",
"```\n",
"\n",
"## Using different Python Kernels with Jupyter\n",
"\n",
"Note: VSCode seems to be able to use Python or Conda environments anyway but installing a dedicated ipykernel is needed for debugging.\n",
"\n",
"```bash\n",
"python -m ipykernel install --user --name MyNewCondaEnv --display-name \"Python3 (MyNewCondaEnv)\"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Kernels1](../media/JLab_kernels1.png)\n",
"![Kernels2](../media/JLab_kernels2.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### To remove unwanted kernels\n",
"\n",
"```\n",
"jupyter kernelspec remove KERNELNAME\n",
"\n",
"```\n",
"\n",
"Example\n",
"\n",
"```\n",
"(base) e:\\src\\test>jupyter kernelspec list\n",
"[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.\n",
"Available kernels:\n",
" bhconda C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\bhconda\n",
" bluehound C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\bluehound\n",
" condadev C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\condadev\n",
" mynewcondaenv C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n",
" python3 C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\python3\n",
" xpython F:\\anaconda\\share\\jupyter\\kernels\\xpython\n",
"\n",
"\n",
"(base) e:\\src\\test>jupyter kernelspec remove mynewcondaenv\n",
"[RemoveKernelSpec] WARNING | Config option `kernel_spec_manager_class` not recognized by `RemoveKernelSpec`.\n",
"Kernel specs to remove:\n",
" mynewcondaenv C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n",
"Remove 1 kernel specs [y/N]: y\n",
"[RemoveKernelSpec] Removed C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n",
"```\n",
"\n",
"Remove the environment if you don't need it\n",
"\n",
"Python venv - just delete the venv folder\n",
"\n",
"Conda\n",
"```\n",
"conda remove --all -n MyNewCondaEnv\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Magics [Ian]\n",
"\n",
"[https://ipython.readthedocs.io/en/stable/interactive/magics.html](https://ipython.readthedocs.io/en/stable/interactive/magics.html)\n",
"\n",
"## What are they?\n",
"\n",
"Magics are a kind of macro/function that allows you to invoke functionality\n",
"of the notebook or OS independent of the kernel language.\n",
"\n",
"### Line magics - single %\n",
"- Only operate on the arguments on the remainder of the line\n",
"- Can be mixed with other code\n",
"\n",
"### Cell magics - double %%\n",
"- Operate on whole cell contents\n",
"- Must be in their own cell and at the start of the cell (even comments!)\n",
"\n",
"## Popular magics - \n",
"
\n", "%magic %env %writefile %js %hmtl %pip %logstart \n", "
\n", "\n", "%magic - lists all magic functions (LONG!)\n", "\n", "%logstart log_file - very useful if you are prone to deleting/overwriting your code and then regret it\n", "\n", "%pdb, %tb and %xmode covered in later section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get or set environment variables\n", "\n", "%env\n", "
" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'C:\\\\Users\\\\Ian'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%env HOME" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# %load ./test_mod.py\n", "import sys\n", "print(sys.version_info)\n", "\n", "print(sys.platform)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Run pip\n", "\n", "%pip \n", "
\n", "\n", "Always use this rather than !pip" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: pandas\n", "Version: 1.2.1\n", "Summary: Powerful data structures for data analysis, time series, and statistics\n", "Home-page: https://pandas.pydata.org\n", "Author: None\n", "Author-email: None\n", "License: BSD\n", "Location: c:\\users\\ian\\appdata\\roaming\\python\\python37\\site-packages\n", "Requires: python-dateutil, pytz, numpy\n", "Required-by: statsmodels, seaborn, qgrid, pandasgui, pandas-profiling, Kqlmagic, hvplot, holoviews, msticnb, msticpy\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip show pandas" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sys.version_info(major=3, minor=7, micro=11, releaselevel='final', serial=0)\n", "win32\n" ] } ], "source": [ "%run test_mod.py" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "29809.0993334462" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import math\n", "max((math.pow(math.pi, x) for x in range(10)))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.78 µs ± 44 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" ] } ], "source": [ "%timeit max((math.pow(math.pi, x) for x in range(10)))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "Hello Jupyterthon!\n", "
\n" ], "text/plain": [ "\n", "Hello Jupyterthon!\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Write (or append) the contents of a cell to a file\n", "\n", "** Note - cell magic! **\n", "\n",
"%%writefile file_name
\n",
"%%writefile -a file_name\n",
"
\n", "%run py_file_name\n", "
\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sys.version_info(major=3, minor=7, micro=11, releaselevel='final', serial=0)\n", "win32\n", "win32\n" ] } ], "source": [ "%run test_mod.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Invoking shell commands\n", "\n", "Prefix with !\n", "\n", "These are not magics - they directly invoke underlying OS commands.\n", "\n", "Like line magics, can use these mixed with other code" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Volume in drive E has no label.\n", " Volume Serial Number is 7E50-19F7\n", "\n", " Directory of e:\\src\\infosec-jupyterthon\\workshops\\2021\\day2\n", "\n", "11/24/2021 05:33 PM\n", "jupyter nbconvert --to FORMAT input_notebook.ipynb\n", "
\n" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[NbConvertApp] Converting notebook day2-1-Jupyter-advanced-topics.ipynb to RST\n", "[NbConvertApp] Writing 37699 bytes to day2-1-Jupyter-advanced-topics.rst\n" ] } ], "source": [ "!jupyter nbconvert --to RST day2-1-Jupyter-advanced-topics.ipynb" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## In code" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cell_type': 'markdown',\n", " 'metadata': {},\n", " 'source': '## InfoSec Jupyterthon Day 2\\n\\n---\\n\\n# 1. Jupyter Notebooks Advanced\\n\\nContents\\n\\n- [Jupyter is not just Python](#notjustpython)\\n- [Jupyter Kernels & Python environments](#kernels)\\n- [Magics](#magics)\\n- [Widgets introduction](#widgets)[\\n- [Jupyter Extensions](#extensions)\\n- [Using NBConvert to export and create notebooks](#nbconvert)\\n- [Dev topics - Debugging and testing notebook code](#debugging)'}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import nbformat\n", "\n", "# Import notebook into structured format with nbformat\n", "our_notebook = nbformat.read(\"day2-1-Jupyter-advanced-topics.ipynb\", as_version=4)\n", "our_notebook.cells[0]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Convert a notebook to HTML" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", "\n", "\n", "