{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Banner](../media/banner2.png)\n", "\n", "---\n", "# Workshop 2.1: Jupyter Notebooks Advanced\n", "\n", "* **Contributors**:\n", " * Ashwin Patil (@ashwinpatil)\n", " * Luis Francisco Monge Martinez (@LuckyLuke)\n", " * Ian Hellen (@ianhellen)\n", "

\n", "* **Agenda**:\n", " * [Jupyter is not just Python](#notjustpython)\n", " * [Jupyter Kernels & Python environments](#kernels)\n", " * [Magics](#magics)\n", " * [Widgets introduction](#widgets)[\n", " * [Jupyter Extensions](#extensions)\n", " * [Export and create notebooks](#nbconvert)\n", " * [Dev topics - Debugging and testing notebook code](#debugging)\n", "

\n", "* **Notebook**: [https://aka.ms/Jupyterthon-ws-2-1](https://aka.ms/Jupyterthon-ws-2-1)\n", "* **License**: [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/)\n", "\n", "* **Q&A** - OTR Discord **#Jupyterthon #WORKSHOP DAY 2 - JUPYTER ADVANCED**" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "---\n", "\n", "# Jupyter is not just Python [Ashwin]\n", "- Powershell kernel\n", "- R kernel" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Jupyter Kernels & Python environments\n", "\n", "Python environments let you create \"isolated\" installations with independent versions of packages.\n", "\n", "This is usually **A VERY GOOD IDEA**!\n", "\n", "Linux\n", "\n", "```bash\n", "python -m venv MyNewEnv\n", "source ./MyNewEnv/Scripts/activate\n", "pip install msticpy\n", "```\n", "\n", "Windows\n", "\n", "```cmd\n", "python -m venv MyNewEnv\n", ".\\MyNewEnv\\Scripts\\activate\n", "pip install msticpy\n", "```\n", "\n", "Conda\n", "\n", "```bash\n", "conda create -n MyNewCondaEnv\n", "conda activate MyNewCondaEnv\n", "conda install pip\n", "pip install msticpy\n", "```\n", "\n", "## Using different Python Kernels with Jupyter\n", "\n", "Note: VSCode seems to be able to use Python or Conda environments anyway but installing a dedicated ipykernel is needed for debugging.\n", "\n", "```bash\n", "python -m ipykernel install --user --name MyNewCondaEnv --display-name \"Python3 (MyNewCondaEnv)\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Kernels1](../media/JLab_kernels1.png)\n", "![Kernels2](../media/JLab_kernels2.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### To remove unwanted kernels\n", "\n", "```\n", "jupyter kernelspec remove KERNELNAME\n", "\n", "```\n", "\n", "Example\n", "\n", "```\n", "(base) e:\\src\\test>jupyter kernelspec list\n", "[ListKernelSpecs] WARNING | Config option `kernel_spec_manager_class` not recognized by `ListKernelSpecs`.\n", "Available kernels:\n", " bhconda C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\bhconda\n", " bluehound C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\bluehound\n", " condadev C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\condadev\n", " mynewcondaenv C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n", " python3 C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\python3\n", " xpython F:\\anaconda\\share\\jupyter\\kernels\\xpython\n", "\n", "\n", "(base) e:\\src\\test>jupyter kernelspec remove mynewcondaenv\n", "[RemoveKernelSpec] WARNING | Config option `kernel_spec_manager_class` not recognized by `RemoveKernelSpec`.\n", "Kernel specs to remove:\n", " mynewcondaenv C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n", "Remove 1 kernel specs [y/N]: y\n", "[RemoveKernelSpec] Removed C:\\Users\\Ian\\AppData\\Roaming\\jupyter\\kernels\\mynewcondaenv\n", "```\n", "\n", "Remove the environment if you don't need it\n", "\n", "Python venv - just delete the venv folder\n", "\n", "Conda\n", "```\n", "conda remove --all -n MyNewCondaEnv\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Magics [Ian]\n", "\n", "[https://ipython.readthedocs.io/en/stable/interactive/magics.html](https://ipython.readthedocs.io/en/stable/interactive/magics.html)\n", "\n", "## What are they?\n", "\n", "Magics are a kind of macro/function that allows you to invoke functionality\n", "of the notebook or OS independent of the kernel language.\n", "\n", "### Line magics - single %\n", "- Only operate on the arguments on the remainder of the line\n", "- Can be mixed with other code\n", "\n", "### Cell magics - double %%\n", "- Operate on whole cell contents\n", "- Must be in their own cell and at the start of the cell (even comments!)\n", "\n", "## Popular magics - \n", "

\n", "%magic %env %writefile %js %hmtl %pip %logstart \n", "

\n", "\n", "%magic - lists all magic functions (LONG!)\n", "\n", "%logstart log_file - very useful if you are prone to deleting/overwriting your code and then regret it\n", "\n", "%pdb, %tb and %xmode covered in later section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get or set environment variables\n", "

\n", "%env\n", "

" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'C:\\\\Users\\\\Ian'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%env HOME" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# %load ./test_mod.py\n", "import sys\n", "print(sys.version_info)\n", "\n", "print(sys.platform)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Run pip\n", "

\n", "%pip \n", "

\n", "\n", "Always use this rather than !pip" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: pandas\n", "Version: 1.2.1\n", "Summary: Powerful data structures for data analysis, time series, and statistics\n", "Home-page: https://pandas.pydata.org\n", "Author: None\n", "Author-email: None\n", "License: BSD\n", "Location: c:\\users\\ian\\appdata\\roaming\\python\\python37\\site-packages\n", "Requires: python-dateutil, pytz, numpy\n", "Required-by: statsmodels, seaborn, qgrid, pandasgui, pandas-profiling, Kqlmagic, hvplot, holoviews, msticnb, msticpy\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip show pandas" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sys.version_info(major=3, minor=7, micro=11, releaselevel='final', serial=0)\n", "win32\n" ] } ], "source": [ "%run test_mod.py" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "29809.0993334462" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import math\n", "max((math.pow(math.pi, x) for x in range(10)))" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.78 µs ± 44 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" ] } ], "source": [ "%timeit max((math.pow(math.pi, x) for x in range(10)))" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "Hello Jupyterthon!\n", "

\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%html\n", "

\n", "Hello Jupyterthon!\n", "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Write (or append) the contents of a cell to a file\n", "\n", "** Note - cell magic! **\n", "

\n", "%%writefile file_name
\n", "%%writefile -a file_name\n", "

\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Appending to test_mod.py\n" ] } ], "source": [ "%%writefile -a test_mod.py\n", "\n", "print(sys.platform)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Run a Python script\n", "

\n", "%run py_file_name\n", "

\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sys.version_info(major=3, minor=7, micro=11, releaselevel='final', serial=0)\n", "win32\n", "win32\n" ] } ], "source": [ "%run test_mod.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Invoking shell commands\n", "\n", "Prefix with !\n", "\n", "These are not magics - they directly invoke underlying OS commands.\n", "\n", "Like line magics, can use these mixed with other code" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Volume in drive E has no label.\n", " Volume Serial Number is 7E50-19F7\n", "\n", " Directory of e:\\src\\infosec-jupyterthon\\workshops\\2021\\day2\n", "\n", "11/24/2021 05:33 PM .\n", "11/24/2021 01:00 PM ..\n", "11/24/2021 05:34 PM 69,310 day2-1-Jupyter-advanced-topics.ipynb\n", "11/23/2021 10:34 AM 4,246,504 day2-2-Visualization.ipynb\n", "11/24/2021 04:20 PM 178,240 day2-3-Advanced-pandas.ipynb\n", "11/24/2021 11:22 AM 39,858 day2-4-MSTICPy.ipynb\n", "11/23/2021 10:34 AM 208,453 Holoviews.png\n", "11/23/2021 10:34 AM 50,124 JLab_kernels1.png\n", "11/23/2021 10:34 AM 88,887 JLab_kernels2.png\n", "11/24/2021 05:33 PM 60 test_mod.py\n", "11/24/2021 05:33 PM __pycache__\n", " 8 File(s) 4,881,436 bytes\n", " 3 Dir(s) 229,351,170,048 bytes free\n" ] } ], "source": [ "!dir" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Captured 18 lines:\n", " [' Volume in drive E has no label.', ' Volume Serial Number is 7E50-19F7', '', ' Directory of e:\\\\src\\\\infosec-jupyterthon\\\\workshops\\\\2021\\\\day2', '', '11/24/2021 05:33 PM .', '11/24/2021 01:00 PM ..', '11/24/2021 05:34 PM 70,452 day2-1-Jupyter-advanced-topics.ipynb', '11/23/2021 10:34 AM 4,246,504 day2-2-Visualization.ipynb', '11/24/2021 04:20 PM 178,240 day2-3-Advanced-pandas.ipynb', '11/24/2021 11:22 AM 39,858 day2-4-MSTICPy.ipynb', '11/23/2021 10:34 AM 208,453 Holoviews.png', '11/23/2021 10:34 AM 50,124 JLab_kernels1.png', '11/23/2021 10:34 AM 88,887 JLab_kernels2.png', '11/24/2021 05:33 PM 60 test_mod.py', '11/24/2021 05:33 PM __pycache__', ' 8 File(s) 4,882,578 bytes', ' 3 Dir(s) 229,351,165,952 bytes free']\n" ] } ], "source": [ "my_folder = !dir\n", "\n", "print(f\"Captured {len(my_folder)} lines:\\n\", my_folder)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating Magics" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "from IPython.core.magic import register_line_magic\n", "## also register_cell_magic for cell magics\n", "# register_line_cell_magic for a magic that works with both\n", "\n", "@register_line_magic\n", "def ian_is(line):\n", " \"my line magic\"\n", " return f\"Ian is {' '.join(word.capitalize() for word in line.split())}\"\n", "\n", "del ian_is" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Ian is A Fan Of Python'" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%ian_is a fan of Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Magic example" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "import msticpy" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[('ipv4', ['92.207.181.106', '81.0.236.93', '51.75.33.120', '94.177.248.64']),\n", " ('ipv6', ['11:20:26']),\n", " ('dns', ['av-quiz.tk']),\n", " ('url', ['http://av-quiz.tk/wp-content/k6K/']),\n", " ('linux_path', ['//av-quiz.tk/wp-content/k6K/\\t\\t\\tNov']),\n", " ('sha256_hash',\n", " ['f7a4da96129e9c9708a005ee28e4a46af092275af36e3afd63ff201633c70285',\n", " 'd95125b9b82df0734b6bc27c426d42dea895c642f2f6516132c80f896be6cf32',\n", " 'b95a6218777e110578fa017ac14b33bf968ca9c57af7e99bd5843b78813f46e0',\n", " '9c345ee65032ec38e1a29bf6b645cde468e3ded2e87b0c9c4a93c517d465e70d',\n", " 'bd9b8fe173935ad51f14abc16ed6a5bf6ee92ec4f45fd2ae1154dd2f727fb245'])]" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%ioc\n", "\n", "TYPE\n", "INDICATOR\n", "ROLE\n", "TITLE\n", "ADDED\n", "ACTIVE\n", "RELATED PULSES\n", "URL\thttp://av-quiz.tk/wp-content/k6K/\t\t\tNov 16, 2021, 11:20:26 AM\t\t2\t\n", "IPv4\t94.177.248.64\t\t\tNov 16, 2021, 11:20:26 AM\t\t8\t\n", "IPv4\t92.207.181.106\t\t\tNov 16, 2021, 11:20:26 AM\t\t2\t\n", "IPv4\t81.0.236.93\t\t\tNov 16, 2021, 11:20:26 AM\t\t126\t\n", "IPv4\t51.75.33.120\t\t\tNov 16, 2021, 11:20:26 AM\t\t265\t\n", "FileHash-SHA256\tf7a4da96129e9c9708a005ee28e4a46af092275af36e3afd63ff201633c70285\t\t\tNov 16, 2021, 11:20:26 AM\t\t3\t\n", "FileHash-SHA256\td95125b9b82df0734b6bc27c426d42dea895c642f2f6516132c80f896be6cf32\t\t\tNov 16, 2021, 11:20:26 AM\t\t3\t\n", "FileHash-SHA256\tbd9b8fe173935ad51f14abc16ed6a5bf6ee92ec4f45fd2ae1154dd2f727fb245\t\t\tNov 16, 2021, 11:20:26 AM\t\t3\t\n", "FileHash-SHA256\tb95a6218777e110578fa017ac14b33bf968ca9c57af7e99bd5843b78813f46e0\t\t\tNov 16, 2021, 11:20:26 AM\t\t2\t\n", "FileHash-SHA256\t9c345ee65032ec38e1a29bf6b645cde468e3ded2e87b0c9c4a93c517d465e70d\t\t\tNov 16, 2021, 11:20:26 AM\t\t2\t\n" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "----\n", "\n", "# Widgets introduction [Luis]\n", "Interactive HTML widgets for Jupyter Notebooks and IPython kernel. \n", "Easy way to avoid input errors, types mismatch, date fortmat errors...\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Name: ipykernel\n", "Version: 6.0.1\n", "Summary: IPython Kernel for Jupyter\n", "Home-page: https://ipython.org\n", "Author: IPython Development Team\n", "Author-email: ipython-dev@scipy.org\n", "License: BSD\n", "Location: c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages\n", "Requires: traitlets, ipython, tornado, debugpy, jupyter-client\n", "Required-by: qtconsole, notebook, jupyter, jupyter-console, ipywidgets\n" ] } ], "source": [ "!pip show ipykernel\n", "#It's neccessary to select an ipykernel to work with ipywidgets" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import ipywidgets as widgets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Integer Slider" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "8039dc289b9f4ffc85ee8d833060dd7f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "IntSlider(value=0)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "w = widgets.IntSlider()\n", "display(w)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "w.value = 89" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Intenger Range Slider" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ea2473ab24d54c0fbbb6a5c38b68c16e", "version_major": 2, "version_minor": 0 }, "text/plain": [ "IntRangeSlider(value=(5, 7), max=10)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "widgets.IntRangeSlider(value=[5, 7], min=0, max=10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Integer Progress Bar" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "p = widgets.IntProgress(\n", " value=0,\n", " min=0,\n", " max=9,\n", " description='Loading:',\n", " bar_style='', # 'success', 'info', 'warning', 'danger' or ''\n", " style={'bar_color': 'maroon'},\n", " orientation='horizontal'\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "150bf939604f4a368c7fb73b404f0c81", "version_major": 2, "version_minor": 0 }, "text/plain": [ "IntProgress(value=0, description='Loading:', max=9, style=ProgressStyle(bar_color='maroon'))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "***Finished!***" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import time\n", "from IPython.display import Markdown\n", "display(p)\n", "for x in range(10):\n", " p.value = x\n", " time.sleep(1)\n", " if x>4:\n", " p.style.bar_color = 'green'\n", "p.close()\n", "display(Markdown('***Finished!***'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dropdown" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "7e5b58a3a5fd49799b066ebfa4961709", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Dropdown(description='Number:', index=1, options=(('One', 1), ('Two', 2), ('Three', 3)), value=2)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "widgets.Dropdown(\n", " options=[('One', 1), ('Two', 2), ('Three', 3)],\n", " value=2,\n", " description='Number:',\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multiselector" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "64f91bd72cb94f03aa637bef5fd6d84f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "SelectMultiple(description='Modules', options=('Option1', 'Option2', 'Option3'), value=())" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "sm = widgets.SelectMultiple(\n", " options=['Option1', 'Option2', 'Option3'],\n", " #rows=10,\n", " description='Modules',\n", " disabled=False\n", ")\n", "display(sm)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('Option1', 'Option2')" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sm.value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data Picker" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d85315d8cdc74171abfff61ea68ae936", "version_major": 2, "version_minor": 0 }, "text/plain": [ "DatePicker(value=None, description='Pick a Date')" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "widgets.DatePicker(\n", " description='Pick a Date',\n", " disabled=False\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### File Uploader" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9df96c89a0b34677b24febbc08155fd5", "version_major": 2, "version_minor": 0 }, "text/plain": [ "FileUpload(value={}, description='Upload')" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fu = widgets.FileUpload(\n", " accept='', # Accepted file extension e.g. '.txt', '.pdf', 'image/*', 'image/*,.pdf'\n", " multiple=False # True to accept multiple files upload else False\n", ")\n", "display(fu)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### More sophisticated file/folder chooser \n", "[ipyfilechooser Project](https://github.com/crahan/ipyfilechooser)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting ipyfilechooser\n", " Downloading ipyfilechooser-0.6.0-py3-none-any.whl (11 kB)\n", "Requirement already satisfied: ipywidgets in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipyfilechooser) (7.6.3)\n", "Requirement already satisfied: ipykernel>=4.5.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (6.0.1)\n", "Requirement already satisfied: widgetsnbextension~=3.5.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (3.5.1)\n", "Requirement already satisfied: ipython>=4.0.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (7.25.0)\n", "Requirement already satisfied: traitlets>=4.3.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (5.0.5)\n", "Requirement already satisfied: nbformat>=4.2.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (5.1.3)\n", "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipywidgets->ipyfilechooser) (1.0.0)\n", "Requirement already satisfied: jupyter-client in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets->ipyfilechooser) (6.1.12)\n", "Requirement already satisfied: tornado>=4.2 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets->ipyfilechooser) (6.1)\n", "Requirement already satisfied: debugpy>=1.0.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets->ipyfilechooser) (1.3.0)\n", "Requirement already satisfied: decorator in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (4.4.2)\n", "Requirement already satisfied: colorama in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.4.4)\n", "Requirement already satisfied: setuptools>=18.5 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (52.0.0.post20210125)\n", "Requirement already satisfied: jedi>=0.16 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.18.0)\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (3.0.19)\n", "Requirement already satisfied: pygments in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (2.9.0)\n", "Requirement already satisfied: matplotlib-inline in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.1.2)\n", "Requirement already satisfied: backcall in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.2.0)\n", "Requirement already satisfied: pickleshare in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.7.5)\n", "Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.8.2)\n", "Requirement already satisfied: jupyter-core in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets->ipyfilechooser) (4.7.1)\n", "Requirement already satisfied: ipython-genutils in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets->ipyfilechooser) (0.2.0)\n", "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets->ipyfilechooser) (3.2.0)\n", "Requirement already satisfied: six>=1.11.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->ipyfilechooser) (1.16.0)\n", "Requirement already satisfied: pyrsistent>=0.14.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->ipyfilechooser) (0.18.0)\n", "Requirement already satisfied: attrs>=17.4.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets->ipyfilechooser) (21.2.0)\n", "Requirement already satisfied: wcwidth in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets->ipyfilechooser) (0.2.5)\n", "Requirement already satisfied: notebook>=4.4.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (6.4.0)\n", "Requirement already satisfied: nbconvert in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (6.1.0)\n", "Requirement already satisfied: argon2-cffi in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (20.1.0)\n", "Requirement already satisfied: terminado>=0.8.3 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.10.1)\n", "Requirement already satisfied: pyzmq>=17 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (22.1.0)\n", "Requirement already satisfied: jinja2 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (3.0.1)\n", "Requirement already satisfied: Send2Trash>=1.5.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.7.1)\n", "Requirement already satisfied: prometheus-client in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.11.0)\n", "Requirement already satisfied: python-dateutil>=2.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->ipyfilechooser) (2.8.2)\n", "Requirement already satisfied: pywin32>=1.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jupyter-core->nbformat>=4.2.0->ipywidgets->ipyfilechooser) (301)\n", "Requirement already satisfied: pywinpty>=1.1.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.1.3)\n", "Requirement already satisfied: cffi>=1.0.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.14.6)\n", "Requirement already satisfied: pycparser in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (2.20)\n", "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (2.0.1)\n", "Requirement already satisfied: entrypoints>=0.2.2 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.3)\n", "Requirement already satisfied: mistune<2,>=0.8.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.8.4)\n", "Requirement already satisfied: bleach in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (3.3.1)\n", "Requirement already satisfied: testpath in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.5.0)\n", "Requirement already satisfied: pandocfilters>=1.4.1 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.4.3)\n", "Requirement already satisfied: jupyterlab-pygments in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.1.2)\n", "Requirement already satisfied: defusedxml in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.7.1)\n", "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.5.3)\n", "Requirement already satisfied: async-generator in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.10)\n", "Requirement already satisfied: nest-asyncio in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (1.5.1)\n", "Requirement already satisfied: packaging in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (21.0)\n", "Requirement already satisfied: webencodings in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (0.5.1)\n", "Requirement already satisfied: pyparsing>=2.0.2 in c:\\users\\pebryan\\appdata\\local\\continuum\\anaconda3\\envs\\dev38\\lib\\site-packages (from packaging->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->ipyfilechooser) (2.4.7)\n", "Installing collected packages: ipyfilechooser\n", "Successfully installed ipyfilechooser-0.6.0\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "%pip install ipyfilechooser" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4ac9d78f969947c5bf2492965a66fea0", "version_major": 2, "version_minor": 0 }, "text/plain": [ "FileChooser(path='C:\\source\\infosec-jupyterthon\\workshops\\2021\\day2', filename='', title='Input folder Path…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ipyfilechooser import FileChooser\n", "fc = FileChooser()\n", "#fc.show_only_dirs = True\n", "fc.show_hidden = True\n", "fc.use_dir_icons = True\n", "fc.title = 'Input folder Path'\n", "display(fc)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "MSTICPy also includes a number of advanced widgets. You can find out more about them in the workshop session on MSTICPy later today." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "# Jupyter Extensions [Luis]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Extension are client-specific, most only Jupyter classic. In this section we will talk about JupyterLab extensions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fundamentally, JupyterLab is designed as an extensible environment. JupyterLab extensions can customize or enhance any part of JupyterLab. They can provide new themes, file viewers and editors, or renderers for rich outputs in notebooks. Extensions can add items to the menu or command palette, keyboard shortcuts, or settings in the settings system. Extensions can provide an API for other extensions to use and can depend on other extensions. In fact, the whole of JupyterLab itself is simply a collection of extensions that are no more powerful or privileged than any custom extension." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating config file\n", "This file will be used to keep extensions configurations. \n", "File will be created in '~/.jupyter/jupyter_lab_config.py'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!jupyter lab --generate-config" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### JupyterLab System Monitor\n", "JupyterLab extension to display system information (memory and cpu usage). [Project](https://github.com/jtpio/jupyterlab-system-monitor)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install jupyterlab-system-monitor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add this lines to config file. \n", "```\n", "# amount of memory expressed in bytes\n", "c.ResourceUseDisplay.mem_limit = 8564768768\n", "c.ResourceUseDisplay.track_cpu_percent = True\n", "c.ResourceUseDisplay.cpu_limit = 8\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Sysmonitor](https://github.com/jtpio/jupyterlab-system-monitor/raw/main/doc/screencast.gif)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Git\n", "A JupyterLab extension for version control using Git. [Project](https://github.com/jupyterlab/jupyterlab-git)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install jupyterlab-git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Git](https://raw.githubusercontent.com/jupyterlab/jupyterlab-git/master/docs/figs/preview.gif)" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "### JupyterLab Templates\n", "Support for jupyter notebook templates in jupyterlab. [Project](https://github.com/jpmorganchase/jupyterlab_templates)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install jupyterlab_templates\n", "!jupyter labextension install jupyterlab_templates\n", "!jupyter serverextension enable --py jupyterlab_templates" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Add this lines to config file. \n", "```\n", "c.JupyterLabTemplates.template_dirs = ['list', 'of', 'template', 'directories']\n", "c.JupyterLabTemplates.include_default = True\n", "c.JupyterLabTemplates.include_core_paths = True\n", "```\n", "**Tip**: It's necessary to put the templates inside a folder inside indicated folder." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "![Templates](https://raw.githubusercontent.com/jpmorganchase/jupyterlab_templates/main/docs/example1.gif)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Code Snippets (Elyra)\n", "The ability to reuse pieces of code allows users to avoid doing repetitive work, making the programming workflow more simple and productive. Elyra supports custom code snippets that can be added to the file editor. [Project](https://elyra.readthedocs.io/en/latest/getting_started/overview.html#reusable-code-snippets)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install elyra-code-snippet-extension\n", "!pip install -U \"nbclassic>=0.2.8\"\n", "!jupyter lab build" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![Snippets example](https://elyra.readthedocs.io/en/latest/_images/code-snippet-expanded.png)" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Export and create notebooks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NBFormat - Create a notebook programmatically [Roberto]\n", "\n", "* Jupyter notebook files are simple JSON documents, containing text, source code, rich media output, and metadata.\n", "* Each segment of the document is stored in a cell.\n", "* We can use the [nbformat](https://nbformat.readthedocs.io/en/latest/api.html) Python APIs to create notebook markdown and code cells.\n", "\n", "**Create a Notebook Object**\n", "* Import nbformat library\n", "* Create a new notebook object\n", "* Initialize notebook cells as an empty list" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "import nbformat as nbf\n", "nb = nbf.v4.new_notebook()\n", "nb['cells'] = []" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Create a Markdown Cell**\n", "* Use the [nbformat.v4.new_markdown_cell API](https://nbformat.readthedocs.io/en/latest/api.html#nbformat.v4.new_markdown_cell) to create a new markdown cell\n", "* Append the results to the notebooks cells list" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': '95abf033',\n", " 'cell_type': 'markdown',\n", " 'source': '# Remote Service Creation',\n", " 'metadata': {}}]" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nb['cells'].append(nbf.v4.new_markdown_cell(\"# Remote Service Creation\"))\n", "nb['cells']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Create a Code Cell**\n", "* Use the [nbformat.v4.new_code_cell API](https://nbformat.readthedocs.io/en/latest/api.html#nbformat.v4.new_code_cell) to create a new code cell\n", "* Append the results to the notebooks cells list" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'id': '95abf033',\n", " 'cell_type': 'markdown',\n", " 'source': '# Remote Service Creation',\n", " 'metadata': {}},\n", " {'id': 'bb39946a',\n", " 'cell_type': 'code',\n", " 'metadata': {},\n", " 'execution_count': None,\n", " 'source': 'from openhunt.mordorutils import *\\nspark = get_spark()',\n", " 'outputs': []}]" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nb['cells'].append(nbf.v4.new_code_cell(\"\"\"from openhunt.mordorutils import *\n", "spark = get_spark()\"\"\"\n", "))\n", "nb['cells']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Write Noteook File**\n", "Use the [nbformat.write API](https://nbformat.readthedocs.io/en/latest/api.html#nbformat.write) to write the notebook object to a file." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nbf.write(nb, \"test.ipynb\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](../media/day2/nbformat-write-notebook.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Examples: Document research and detection logic in notebooks programmatically**\n", "* [An interactive Book over the Threat Hunter Playbook](https://medium.com/threat-hunters-forge/writing-an-interactive-book-over-the-threat-hunter-playbook-with-the-help-of-the-jupyter-book-3ff37a3123c7)\n", "* [Jupyter Notebooks 📓 from SIGMA Rules 🛡⚔️ to Query Elasticsearch 🏹](https://medium.com/threat-hunters-forge/jupyter-notebooks-from-sigma-rules-%EF%B8%8F-to-query-elasticsearch-31a74cc59b99)\n", "* [Seurity Datasets project: YAML -> Notebooks](https://github.com/OTRF/Security-Datasets)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## NBConvert - Exporting and converting to other formats [Ian] \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### From the command line\n", "\n", "

\n", "jupyter nbconvert --to FORMAT input_notebook.ipynb\n", "

\n" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[NbConvertApp] Converting notebook day2-1-Jupyter-advanced-topics.ipynb to RST\n", "[NbConvertApp] Writing 37699 bytes to day2-1-Jupyter-advanced-topics.rst\n" ] } ], "source": [ "!jupyter nbconvert --to RST day2-1-Jupyter-advanced-topics.ipynb" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## In code" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'cell_type': 'markdown',\n", " 'metadata': {},\n", " 'source': '## InfoSec Jupyterthon Day 2\\n\\n---\\n\\n# 1. Jupyter Notebooks Advanced\\n\\nContents\\n\\n- [Jupyter is not just Python](#notjustpython)\\n- [Jupyter Kernels & Python environments](#kernels)\\n- [Magics](#magics)\\n- [Widgets introduction](#widgets)[\\n- [Jupyter Extensions](#extensions)\\n- [Using NBConvert to export and create notebooks](#nbconvert)\\n- [Dev topics - Debugging and testing notebook code](#debugging)'}" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import nbformat\n", "\n", "# Import notebook into structured format with nbformat\n", "our_notebook = nbformat.read(\"day2-1-Jupyter-advanced-topics.ipynb\", as_version=4)\n", "our_notebook.cells[0]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Convert a notebook to HTML" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n", "\n", "\n", "Notebook