{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Getting Started with Azure ML Notebooks and Azure Sentinel\n",
"**Notebook Version:** 1.0
\n",
" **Python Version:** Python 3.6 (including Python 3.6 - AzureML)
\n",
" **Required Packages**:
\n",
" **Platforms Supported**:\n",
" - Azure Notebooks Free Compute\n",
" - Azure Notebooks DSVM\n",
" - OS Independent\n",
"\n",
"**Data Sources Required**:\n",
" - Log Analytics - SiginLogs (Optional)\n",
" - VirusTotal\n",
" - MaxMind\n",
" \n",
" \n",
"This notebook takes you through the basics needed to get started with Azure Notebooks and Azure Sentinel, and how to perform the basic actions of data acquisition, data enrichment, data analysis, and data visualization. These actions are the building blocks of threat hunting with notebooks and are useful to understand before running more complex notebooks. This notebook only lightly covers each topic but includes 'learn more' sections to provide you with the resource to deep dive into each of these topics. \n",
"\n",
"This notebook assumes that you are running this in an Azure Notebooks environment, however it will work in other Jupyter environments.\n",
"\n",
"**Note:**\n",
"This notebooks uses SigninLogs from your Azure Sentinel Workspace. If you are not yet collecting SigninLogs configure this connector in the Azure Sentinel portal before running this notebook.\n",
"This notebook also uses the VirusTotal API for data enrichment, for this you will require an API key which can be obtained by signing up for a free [VirusTotal community account](https://www.virustotal.com/gui/join-us)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## What is a Jupyter notebook?\n",
"You are currently reading a Jupyter notebook. [Jupyter](http://jupyter.org/) is an interactive development and data manipulation environment presented in a browser. Using Jupyter you can create documents, called Notebooks. These documents are made up of cells that contain interactive code, alongside that code's output, and other items such as text and images (what you are looking at now is a cell of Markdown text).\n",
"\n",
"The name, Jupyter, comes from the core supported programming languages that it supports: Julia, Python, and R. Whilst you can use any of these languages we are going to use Python in this notebook, in addition the notebooks that come with Azure Sentinel are all written in Python. Whilst there are pros, and cons to each language Python is a well-established language that has a large number of materials and libraries well suited for data analysis and security investigation, making it ideal for our needs.\n",
"\n",
"### Learn more:\n",
" - The [Infosec Jupyter Book](https://infosecjupyterbook.com/introduction.html) has more details on the technical working of Jupyter.\n",
" - [The Jupyter Project documentation](https://jupyter.org/documentation)\n",
"\n",
"---\n",
"## How to use a Jupyter notebook?\n",
"To use a Jupyter notebook you need a Jupyter server that will render the notebook and execute the code within it. This can take the form of a local [Jupyter installation](https://pypi.org/project/jupyter/), or a remotely hosted version such as [Azure Notebooks](https://notebooks.azure.com/). If you are reading this it is highly likely that you already have a Jupyter server that this notebook is using.\n",
"You can learn more about installing and running your own Jupyter server [here](https://realpython.com/jupyter-notebook-introduction/).\n",
"\n",
"### Using Azure Notebooks\n",
"If you accessed this notebook from Azure Sentinel, you are probably using Azure Notebooks to run this notebook. Azure Notebooks runs in the same way that a local Jupyter server with, except with the additional feature of integrated project management and file storage. When you open a notebook in Azure Notebooks the user interface is nearly identical to a standard Jupyter notebook experience.\n",
"\n",
"Before you can start running code in a notebook you need to make sure that it is connected to a Jupyter server and you have the correct type of kernel configured. For this notebook we are going to be using Python 3.6, hopefully Azure Notebooks has already loaded this kernel for you - you can check this by looking at the top left corner of the screen where you should see the currently connected kernel. \n",
"\n",
"![KernelIssue](https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/images/nb_img1.png)\n",
"\n",
"If this does not read Python 3.6 you can select the correct kernel by selecting Kernel > Change kernel from the top menu and clicking Python 3.6.\n",
"\n",
"> **Note**: the notebook works with Python 3.6, 3.7 or later. If you are using this notebook in Azure ML or another Jupyter environment you can choose any kernel that supports Python 3.6 or later\n",
"\n",
"![KernelPicker](https://github.com/Azure/Azure-Sentinel-Notebooks/raw/master/images/nb_img2.png)\n",
"\n",
"Once you have done this you should be ready to move onto a code cell.\n",
"> **Tip**: You can identify which cells are code by selecting them and looking at the drop down box at the center of the top menu. It will either read 'Code' (for interactive code cells), 'Markdown' (for Markdown text cells like this one), or RawNBConvert (these are just raw data and not interpreted by Jupyter - they can be used by tools that process notebook files, such as *nbconvert* to render the data into HTML or LaTeX). \n",
"\n",
"If you click on the cell below you should see this box change to 'Code'.\n",
"\n",
"### Learn More:\n",
"More details on Azure Notebooks can be found in the [Azure Notebooks documentation](https://docs.microsoft.com/en-us/azure/notebooks/) and the [Azure Sentinel documentation](https://docs.microsoft.com/en-us/azure/sentinel/notebooks).\n",
"\n",
"---\n",
"## Running code\n",
"Once you have selected a code cell you can run it by clicking the run button at the menu bar at the top, or by pressing Ctrl+Enter.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# This is our first code cell, it contains basic Python code.\n",
"# You can run a code cell by selecting it and clicking the Run button in the top menu, or by pressing Shift + Enter.\n",
"# Once you run a code cell any output from that code will be displayed directly below it.\n",
"print(\"Congratulations you just ran this code cell\")\n",
"y = 2+2\n",
"print(\"2 + 2 =\", y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Variables set within a code cell persist between cells meaning you can chain cells together"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"y + 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Learn More : \n",
" - The [Infosec Jupyter Book](https://infosecjupyterbook.com/) provides an infosec specific intro to Python.\n",
" - [Real Python](https://realpython.com/) is a comprehensive set of Python learnings and tutorials.\n",
"
\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that you understand the basics we can move onto more complex code.\n",
"\n",
"---\n",
"## Setting up the environment\n",
"Code cells behave in the same way your code would in other environments, so you need to remember about common coding practices such as variable initialization and library imports. \n",
"Before we execute more complex code we need to make sure the required packages are installed and libraries imported. At the top of many of the Azure Sentinel notebooks you will see large cells that will check kernel versions and then install and import all the libraries we are going to be using in the notebook, make sure you run this before running other cells in the notebook.\n",
"If you are running notebooks locally or via dedicated compute in Azure Notebooks library installs will persist but this is not the case with Azure Notebooks free tier, so you will need to install each time you run. Even if running in a static environment imports are required for each run so make sure you run this cell regardless."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\r\n",
"from IPython.display import display, HTML\r\n",
"\r\n",
"REQ_PYTHON_VER=(3, 6)\r\n",
"REQ_MSTICPY_VER=(1, 0, 0)\r\n",
"REQ_MP_EXTRAS=[\"Azure\"]\r\n",
"\r\n",
"display(HTML(\"