{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"# A Getting Started Guide for Microsoft Sentinel notebooks with PowerShell\r\n",
"**Notebook Version:** 2.0
\r\n",
"\r\n",
"**Data Sources Required**:\r\n",
" - Log Analytics - SecurityEvent (Optional)\r\n",
"\r\n",
"**.Net Interactive installation is required! **:\r\n",
" - To use this notebook, you will first need to install .Net Interactive. \r\n",
" Please follow the instructions in the section of \"Installing the required PowerShell modules\". \r\n",
" Current Ubuntu version is 18.04, and .NET SDK 5.0 is required.\r\n",
" Details can be found in this article -> [Microsoft Sentinel Notebooks + Powershell](https://aka.ms/sentinel/pwsh-notebooks).\r\n",
"\r\n",
"** About this notebook **: \r\n",
"\r\n",
"This notebook takes you through the basics needed to get started with PowerShell notebooks that leverage Microsoft Sentinel data and APIs. \r\n",
"\r\n",
"This notebook assumes that you are running this in an Azure Machine Learning notebooks environment created via the Microsoft Sentinel UI as this notebook has not yet been tested in other environments. Check the [official documentation](https://docs.microsoft.com/azure/sentinel/notebooks) on creating a Microsoft Sentinel AML workspace/environment to learn more.\r\n",
"\r\n",
"For a notebook that provides more definitive guidance to the notebook experience, launch the [A Getting Started Guide for Microsoft Sentinel ML Notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Getting%20Started%20Guide%20For%20Azure%20Sentinel%20ML%20Notebooks.ipynb) notebook from the Microsoft Sentinel notebook UI. This notebook provides a step-by-step overview of the notebook experience as well as some tips and tricks on how to get the most out of your Jupyter notebook experience.\r\n",
"\r\n",
"For more information as to why Juypter for security investigations, check out this excellent article [Why Use Jupyter for Security Investigations](https://techcommunity.microsoft.com/t5/azure-sentinel/why-use-jupyter-for-security-investigations/ba-p/475729)\r\n",
"\r\n",
"Lastly, don't forget to install .Net Interactive to use this notebook!"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Installing .NET SDK and Interactive\r\n",
"Please execute the following cells to install .NET SDK and Interactive. \r\n",
"You need install them once on a compute instance, and only once. \r\n",
"If you have already install them, you may safely ignore these following 6 cells in this block. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# If you are not sure, you may check what kernels you already have on the compute instance:\r\n",
"!jupyter kernelspec list"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# For Ubuntu 18.04\r\n",
"!wget https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb\r\n",
"!sudo dpkg -i packages-microsoft-prod.deb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# Install .NET SDK 5.0\r\n",
"!sudo apt-get update; \\\r\n",
"sudo apt-get install -y apt-transport-https && \\\r\n",
"sudo apt-get update && \\\r\n",
"sudo apt-get install -y dotnet-sdk-5.0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# Install the dotnet interactive global tool\r\n",
"!dotnet tool install --global Microsoft.dotnet-interactive"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# Create�symlink�between the dotnet interactive and thelocal bin directory \r\n",
"!sudo ln -s /home/azureuser/.dotnet/tools/dotnet-interactive /usr/local/bin/dotnet-interactive"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"# Kernel: Python 3.8 - AzureML\r\n",
"# Finally install the .NET kernel: C#, F#, and PowerShell\r\n",
"!dotnet interactive jupyter install"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"### Once successfully complete the above steps, please:
\r\n",
"** 1. Reload the page,
**\r\n",
"** 2. Select .NET (PowerShell) kernel
**\r\n",
"** 3. Then restart kernel **"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"\r\n",
"### Using Azure Notebooks\r\n",
"\r\n",
"For this notebook we are going to be using PowerShell, so you will need to select the **\".NET (PowerShell)\"** kernel in the dropdown on the top right corner of the notebook UI.\r\n",
"\r\n",
"Once you have selected the right kernel, you are ready to move onto the next code cell.\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Installing the required PowerShell modules\r\n",
"Code cells behave in the same way your code would in other environments, so you need to remember about common coding practices such as variable initialization and module/library imports. For this notebook you only need to make sure to install the required PowerShell modules since those are not installed by default."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"gather": {
"logged": 1619476521403
}
},
"outputs": [],
"source": [
"##Installs modules necessary to run notebook\r\n",
"Install-Module Az.Compute,Az.Resources,Az.OperationalInsights,powershell-yaml -Force"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Working with PowerShell within a Jupyter notebook - The Basics \r\n",
"In this section we added a few tips and tricks to using PowerShell in a notebook!\r\n",
"- While there are differences between running PowerShell in a notebook environment vs a local machine, most features are support.\r\n",
"- If you plan on porting your existing scripts, there are some modifications might need to be made to account for these differences.\r\n",
"- Due to an additional UI+kernel intecepting your PowerShell commands, be sure to not overload the output as this can cause a chokepoint. \r\n",
"- Also, since the output goes to a white (or black if darkmode is set) UI, some output colors might need to be modified to be visible.\r\n",
"
\r\n",
"
\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Work with the display\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619478290725
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Get more details on your PowerShell environment\r\n",
"$PSVersionTable"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601310999049
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Output Markdown\r\n",
"Out-Display \"**THIS IS BOLD** _ITALICS_\" -MimeType text/markdown\r\n",
"\r\n",
"##Set foreground color\r\n",
"$host.UI.RawUI.ForegroundColor = [System.ConsoleColor]::Blue\r\n",
"\r\n",
"##Default output colors can be changed \r\n",
"$Host.PrivateData.WarningBackgroundColor = \"White\"\r\n",
"$Host.PrivateData.WarningForegroundColor = \"Black\"\r\n",
"\r\n",
"##Change output color inline\r\n",
"Write-Host \"Lets get started ...\" -ForegroundColor Blue -BackgroundColor White\r\n",
"\r\n",
"##You can write-host with the -nonewlinke flag\r\n",
"Write-Host \"Hello \" -NoNewline -ForegroundColor Red\r\n",
"Write-Host \"World!\" -ForegroundColor Blue"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Output to HTML or markdown. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619476558458
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Output HTML\r\n",
"#!html\r\n",
"Hello in HTML!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619476560691
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"#!markdown\r\n",
"\r\n",
"Write a **list** ...\r\n",
"* first item\r\n",
"* second item\r\n",
"\r\n",
"...or a _table_...\r\n",
"\r\n",
"|CallerIP |PrincipalString |\r\n",
"|---------|--------|\r\n",
"|10.1.1.1 |joe@contoso.com |\r\n",
"|10.1.1.2 |joanne@contoso.com |"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Output from C# and visa versa. Yes, switching between DotNet languages is possible too"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619476565849
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"#!csharp\r\n",
"var x=\"Hello using C#!\";\r\n",
"Console.WriteLine(x);\r\n",
"\r\n",
"#!pwsh\r\n",
"$x = \"Hello using PowerShell!\"\r\n",
"Write-Host $x\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Download content"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619476574649
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Download IOCs from the internet and use them in your investigation/hunts\r\n",
"$ips = (Invoke-WebRequest 'https://raw.githubusercontent.com/parthdmaniar/coronavirus-covid-19-SARS-CoV-2-IoCs/master/IPs').content\r\n",
"$ips"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Prompt for information"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311146113
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##You can ask for user input\r\n",
"Write-Host \"Don't forget that execution of cells will block on prompts until you submit!\"\r\n",
"$name = Read-Host -Prompt \"What is the server name you would like to investigate? \"\r\n",
"$name"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"#### Use progress bars or run commands in parallel"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311158225
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##You can use a progress bar\r\n",
"For ($i=0; $i -le 100; $i++) {\r\n",
" Write-Progress -Id 1 -Activity \"Parent work progress\" -Status \"Current Count: $i\" -PercentComplete $i -CurrentOperation \"Counting ...\"\r\n",
" \r\n",
" For ($j=0; $j -le 10; $j++) {\r\n",
" Start-Sleep -Milliseconds 5\r\n",
" Write-Progress -Parent 1 -Id 2 -Activity \"Child work progress\" -Status \"Current Count: $j\" -PercentComplete ($j*10) -CurrentOperation \"Working ...\"\r\n",
" }\r\n",
" \r\n",
" if ($i -eq 50) {\r\n",
" Write-Host \"Doing the work around here...\" -Foreground DarkBlue\r\n",
" \"Output goes here...\"\r\n",
" }\r\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311176143
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##If you have long running task, that prints output to the screen\r\n",
"##use the -Parallel flag to run them in parallel, vastly improving performance.\r\n",
"##Example below runs one loop sequentially while the second example runs them in parallel\r\n",
"\r\n",
"Write-Host \"Number of seconds running commands sequentially: \" -nonewline\r\n",
"(Measure-Command { \r\n",
" 1..5 | ForEach-Object -Process {write-output \"This is number $_\"; sleep 2}\r\n",
"}).Seconds\r\n",
"\r\n",
"Write-Host \"Number of seconds running commands in parallel: \" -nonewline\r\n",
"(Measure-Command { \r\n",
" 1..5 | ForEach-Object -Parallel {write-output \"This is number $_\"; sleep 2}\r\n",
"}).Seconds"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Microsoft Sentinel Configuration\r\n",
"Once we have set up our Jupyter environment with the libraries that we'll use in the notebook, we need to make sure we have some configuration in place. Some of the notebook components need addtional configuration to connect to external services (e.g. API keys to retrieve Threat Intelligence data). This includes configuration for connection to our Microsoft Sentinel workspace.\r\n",
"For this notebook, we simply import the configuration from the config.json file that is created in your notebook explorer folder when you launch the notebook from the Microsoft Sentinel UI.\r\n",
"
\r\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619478152392
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Get your configuration file settings\r\n",
"$nbcontentpath = \"config.json\"\r\n",
"if(!(test-path $nbcontentpath)){\r\n",
" write-host \"INFO: Your configuration path ($nbcontentpath) could not be located.\"\r\n",
" write-host \"INFO: Attempting to build the file path explicitly. If this continues to be a problem, run 'dir' within the cell to find the current working directory and update the `$nbcontentpath variable accordingly.\" \r\n",
" $username = read-host \"Enter the user name used for the notebook file explorer (the name of the top level folder):\"\r\n",
" $nbcontentpath = \"users\\$username\\config.json\"\r\n",
"}\r\n",
"\r\n",
"##Path fix in case you picked up the cookie cutter configuration file (if you cloned repo from GitHub in terminal)\r\n",
"if(test-path $nbcontentpath){\r\n",
" $content = gc $nbcontentpath | ?{$_ -match \"cookiecutter\"}\r\n",
" if($content.Length -gt 0) {\r\n",
" $nbcontentpath = \"..\\\" + $nbcontentpath\r\n",
" } \r\n",
"}\r\n",
"\r\n",
"try {\r\n",
" $nbconfigcontent = Get-Content $nbcontentpath -ErrorAction Stop \r\n",
"}\r\n",
"catch {\r\n",
" write-host \"ERROR: Your configuration path ($nbcontentpath) could not be located. Please fix before continuing further.\" \r\n",
"}\r\n",
"\r\n",
"##Set variables you will use throughout the notebook\r\n",
"$tenantId = ($nbconfigcontent | ConvertFrom-Json).tenant_id\r\n",
"$subscriptionId = ($nbconfigcontent | ConvertFrom-Json).subscription_id\r\n",
"$resourceGroup = ($nbconfigcontent | ConvertFrom-Json).resource_group\r\n",
"$workspaceName = ($nbconfigcontent | ConvertFrom-Json).workspace_name\r\n",
"$workspaceId = ($nbconfigcontent | ConvertFrom-Json).workspace_id\r\n",
"\r\n",
"Write-Host \"SubscriptionId: \" $subscriptionId \r\n",
"Write-Host \"TenantId: \" $tenantId\r\n",
"Write-Host \"WorkspaceId: \" $workspaceId\r\n",
"Write-Host \"workspaceName: \" $workspaceName "
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Connect to your Microsoft Sentinel workspace\r\n",
"Once you have configured your notebook, now you can connect to your workspace.\r\n",
"\r\n",
"> **Note**:
\r\n",
"- We changed the default foreground colors in case you are using the \"Light\" notebook UI theme, since the yellow output will be hard to see. Feel free to modify.\r\n",
"
\r\n",
"
\r\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1619478238641
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"#Change the default colors used for PowerShell warnings as they make the Connect-AzAccount output difficult to see \r\n",
"$Host.PrivateData.WarningBackgroundColor = \"White\"\r\n",
"$Host.PrivateData.WarningForegroundColor = \"Black\"\r\n",
"\r\n",
"##Connect to selected subscription\r\n",
"Connect-AzAccount -UseDeviceAuthentication\r\n",
"Select-AzSubscription -SubscriptionId $subscriptionId -TenantId $tenantId"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"gather": {
"logged": 1601311424514
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Configure the Log Analytics workspace\r\n",
"$workspace = $null\r\n",
"$workspaces = Get-AzOperationalInsightsWorkspace -ResourceGroupName $resourceGroup\r\n",
"if($workspaces.Length -gt 1) {\r\n",
" Write-Host \"INFO: Multiple workspaces detected.\" \r\n",
" foreach($wksp in $workspaces){\r\n",
" if($wksp.Name -eq $workspaceName) {\r\n",
" $workspace = $wksp\r\n",
" } \r\n",
" } \r\n",
"}\r\n",
"else {\r\n",
" $workspace = $workspaces \r\n",
"}\r\n",
"Write-Host \"INFO: Ensure that the workspace -- {\"$workspace.Name\"} is the intended target workspace before continuing to the next cell.\" \r\n",
"$workspace\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Access your hunting queries \r\n",
"Utilize the savedsearch API to download and run your hunting queries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311442748
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Query your workspace using the savedsearches API\r\n",
"$savedSearchQueries = (Get-AzOperationalInsightsSavedSearch -ResourceGroupName $resourceGroup -WorkspaceName $workspaceName).value\r\n",
"$huntingQueries = $savedSearchQueries | %{$_.properties } | ? {$_.Category -match \"Hunting Queries\"}\r\n",
"Write-Host \"Displaying the first 5 hunting queries...\"\r\n",
"0..4 | foreach {Write-Host \"Hunting Query Name: \" -nonewline;$huntingQueries[$_].DisplayName}"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Access your Microsoft Sentinel incidents \r\n",
"Utilize the Microsoft Sentinel API to download metadata regarding your incidents\r\n",
"\r\n",
"> **Note**: It could take a few seconds to download all of your incidents!\r\n",
"
\r\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311538992
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Build resource id\r\n",
"$worksapceId = \"subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.OperationalInsights/workspaces/${workspaceName}\"\r\n",
"$incidentsResource = $worksapceId + \"/\" + \"providers/Microsoft.SecurityInsights/incidents\"\r\n",
"\r\n",
"##Get incidents\r\n",
"$incidents = Get-AzResource -ResourceId $incidentsResource -ApiVersion \"2019-01-01-preview\"\r\n",
"\r\n",
"##Only display a few incidents as the notebook has to translate the PowerShell output into the Jupyter UI\r\n",
"0..4 | foreach {Write-Host \"Incident Number {$_}: \" -nonewline;$incidents[$_].Properties}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311570671
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Retrieve all of your incident counts and day they were created\r\n",
"$incidentsforgraph = $incidents | % {$_.properties} | select Title, Description, Severity, Status, Owner, createdTimeUtc, relatedAnalyticRuleIds, incidentUrl | ? {(get-date $_.createdTimeUtc) -gt (get-date).AddDays(-31d)} \r\n",
"\r\n",
"##Add formated property for the date they were created to make it easier to create graphs\r\n",
"foreach($incident in $incidentsforgraph)\r\n",
"{\r\n",
" $incident | Add-Member -MemberType NoteProperty -Name NewDate -Value (get-date $incident.createdTimeUtc -format \"yyyy-MM-dd\") -Force\r\n",
"}\r\n",
"\r\n",
"##Retrieve all of your incident counts and day they were created for the last 30 days\r\n",
"$openIncidents = $incidentsforgraph | ? {$_.Status -ne \"Closed\"} \r\n",
"$closedIncidents = $incidentsforgraph | ? {$_.Status -eq \"Closed\"} \r\n",
"Write-Host \"Total incidents (last 30 days) : \" $incidentsforgraph.count\r\n",
"Write-Host \"Total Open Incidents (last 30 days) : \" $openIncidents.count\r\n",
"Write-Host \"Total Closed Incidents (last 30 days) : \" $closedIncidents.count\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Chart your incidents using XPlot\r\n",
"Charts can be rendered using [Xplot.Plotly](https://fslab.org/XPlot/). \r\n",
"Here is a simple example on how to combine your incident data with XPlot.\r\n",
"
\r\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"gather": {
"logged": 1601311603107
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##At least one of each incident type (open or closed) must exists to run this cell\r\n",
"if(($closedIncidents -eq $null) -or ($openIncidents -eq $null)){\r\n",
" Write-Host \"You need at least one instance of each incident type (open or closed) to render the chart\"\r\n",
"}\r\n",
"else {\r\n",
" ##Create open incident plots\r\n",
" $openSeries = [Graph.Scatter]::new()\r\n",
" $openSeries.name = \"Open Incidents\"\r\n",
" $openSeries.x = @(($openIncidents | group-object -Property NewDate).Name | % {$_.ToString()})\r\n",
" $openSeries.y = (($openIncidents | group-object -Property NewDate | Select Count | %{$_.Count})) \r\n",
"\r\n",
" ##Create closed incident plots\r\n",
" $closeSeries = [Graph.Scatter]::new()\r\n",
" $closeSeries.name = \"Closed Incidents\"\r\n",
" $closeSeries.x = @(($closedIncidents | group-object -Property NewDate).Name | % {$_.ToString()})\r\n",
" $closeSeries.y = (($closedIncidents | group-object -Property NewDate | Select Count | %{$_.Count})) \r\n",
"\r\n",
" ##Display chart\r\n",
" $chart = @($openSeries, $closeSeries) | New-PlotlyChart -Title \"Open vs Closed Incidents\"\r\n",
" Out-Display $chart\r\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Query your Microsoft Sentinel Data\r\n",
"Data within your Microsoft Sentinel workspace can be manipulated.\r\n",
"My favorite part about working with notebooks is that I can extract values from one query or API call and use them as inputs to another query and/or API.\r\n",
"\r\n",
"> **Note**: \r\n",
"- The query below requires the Heartbeat table. This was chosen as an example since it will reside in all Microsoft Sentinel workspaces.\r\n",
"- For a more 'real world' example, pick another table or add your own query.\r\n",
"
\r\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311660434
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Add a timeframe variable\r\n",
"$timeframe = $null\r\n",
"do {\r\n",
" $timeframe = Read-Host \"How many days back would you like to query the data? (you must enter an integer for number of days):\"\r\n",
" $timeframe = $timeframe -as [int]\r\n",
" if ($timeframe -eq $null) { write-host \"You must enter a numeric value\" }\r\n",
"}\r\n",
"until ($timeframe -ne $null)\r\n",
"write-host \"You entered: $timeframe days as the input timespan.\"\r\n",
"\r\n",
"\r\n",
"##Query Heartbeat table\r\n",
"$query = \"Heartbeat | where TimeGenerated >= ago($timeframe\" + \"d\" + \") | take 10\"\r\n",
"Write-Host \"Query to run: \" $query\r\n",
"##Run query and add results to object. Now you can use object to display data or graph\r\n",
"$queryResults = Invoke-AzOperationalInsightsQuery -Workspace $Workspace -query $query\r\n",
"#0..1 | foreach {Write-Host \"Result Number {$_}: \";$queryResults.results[$_] }\r\n",
"$queryResults.Results | select TimeGenerated, ComputerIP, Computer, `\r\n",
" OSType, RemoteIPLongitude, RemoteIPLatitude, RemoteIPCountry, `\r\n",
" Resource, ResourceType, ComputerEnvironment | Format-Table \r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Match Microsoft Sentinel data with IOCs\r\n",
"You can also join data from external sources...\r\n",
"\r\n",
"
\r\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601314580432
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Using the example from an earlier cell, collect a list of IOCs and join them with IPs from a query\r\n",
"##Download IOCs from the internet and use them in your investigation/hunts\r\n",
"$ips = ((Invoke-WebRequest 'https://raw.githubusercontent.com/parthdmaniar/coronavirus-covid-19-SARS-CoV-2-IoCs/master/IPs').content).ToString() -Split \"`n\" \r\n",
"\r\n",
"\r\n",
"##Query the Log Analytics table\r\n",
"$query = @\"\r\n",
" Heartbeat \r\n",
" | where TimeGenerated >= ago(1d) \r\n",
" | summarize RecordCount=count() by ComputerIPs=ComputerIP \r\n",
"\"@\r\n",
"\r\n",
"##Run query and add results to object. Now you can use object to display data or graph\r\n",
"$queryResults = Invoke-AzOperationalInsightsQuery -Workspace $Workspace -query $query\r\n",
"$computerips=($queryResults.Results | Group-Object ComputerIPs).Group.ComputerIPs\r\n",
"\r\n",
"##Compare IOC IPs to IPs from your logs\r\n",
"Write-Host \"Example of comparing IPs to IOCs...\"\r\n",
"foreach($computerip in ($computerips | select -first 10 )) {\r\n",
" for($i=0;$i -lt $ips.Length;$i++) {\r\n",
" write-host \"IOC-IP:\" $ips[$i] \"does not match IP:\" $ips[$i] \"from logs!\"\r\n",
" if($i -gt 3) {break}\r\n",
" }\r\n",
"}\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Enriching data\r\n",
"\r\n",
"Now that we have seen how to query for data, we can see how you can enrich data with additional data sources. \r\n",
"- For this we are going to use an external threat intelligence provider to give us some more details about a URL.\r\n",
"- The example cell below assuming you have a VirusTotal (VT) key in your yaml configuration file\r\n",
"- If you don't already have a VirusTotal API key, signup [here](https://www.virustotal.com/gui/join-us).\r\n",
"- If not, you can either hardcode the VT key in the cell or run the [A Getting Started Guide for Microsoft Sentinel ML Notebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/A%20Getting%20Started%20Guide%20For%20Azure%20Sentinel%20ML%20Notebooks.ipynb) notebook for instructions on how to create the configuration file with your VirusTotal key included. \r\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311693429
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Get your configuration file settings\r\n",
"$configFileSuccess=$false\r\n",
"$yamlcontentpath = \"msticpyconfig.yaml\"\r\n",
"$yaml = $null\r\n",
"if(!(test-path $yamlcontentpath)) {\r\n",
" write-host \"INFO: Your configuration path ($yamlcontentpath) could not be located.\"\r\n",
" write-host \"INFO: Attempting to build the file path explicitly. If this continues to be a problem, run 'dir' within the cell to find the current working directory and update the `$nbcontentpath variable accordingly.\" \r\n",
" $username = read-host \"Enter the user name used for the notebook file explorer:\"\r\n",
" $yamlcontentpath = \"users\\$username\\msticpyconfig.yaml\"\r\n",
"}\r\n",
"\r\n",
"##Path fix in case you picked up the cookie cutter configuration file (if you cloned repo from GitHub in terminal)\r\n",
"if(test-path $yamlcontentpath){\r\n",
" $content = gc $yamlcontentpath #| ? {$_ -match \"your-workspace-id\"}\r\n",
" if($content.Length -gt 0) {\r\n",
" $yamlcontentpath = \"..\\\" + $yamlcontentpath\r\n",
" } \r\n",
"}\r\n",
"\r\n",
"##Set Yaml content\r\n",
"try {\r\n",
" $configFileSuccess=$true\r\n",
" $yamlcontent = Get-Content $yamlcontentpath -ErrorAction Stop -Raw\r\n",
" $yaml = ConvertFrom-Yaml $yamlcontent\r\n",
"}\r\n",
"catch {\r\n",
" $configFileSuccess=$false\r\n",
" write-host \"ERROR: Your configuration path ($yamlcontentpath) could not be located. Please fix or hardcode the key before continuing further.\" \r\n",
"}\r\n",
"\r\n",
"\r\n",
"##Harcode your key here if you haven't configured the yaml configuration file\r\n",
"##$APIKey = \"<>\"\r\n",
"\r\n",
"##Set your API key and you are good to go\r\n",
"$APIKey = $yaml.TIProviders.VirusTotal.Args.AuthKey\r\n",
"if($APIKey -eq $null){\r\n",
" $configFileSuccess = $false\r\n",
"}\r\n",
"else {\r\n",
" $configFileSuccess = $true\r\n",
"}\r\n",
"\r\n",
"if($configFileSuccess) {\r\n",
" write-host \"INFO: Your VT key was correctly configured. \" \r\n",
" $yaml.TIProviders.VirusTotal\r\n",
"}\r\n",
"else {\r\n",
" write-host \"ERROR: Your VT key was not correctly configured. Please fix before continuing further.\" \r\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601311710827
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Input VT URL and Key\r\n",
"##Ideally, it would be better to retrieve the key from msticpyconfig.yaml\r\n",
"##$APIKey = ''\r\n",
"\r\n",
"$Resource = Read-Host \"Enter the URL you would like to submit (example: support.btcsupports.com):\"\r\n",
"\r\n",
"##Test URL\r\n",
"##$Resource = 'http://support.btcsupports.com/'\r\n",
"\r\n",
"##Setup VT URI\r\n",
"$URI = 'https://www.virustotal.com/vtapi/v2/url/report'\r\n",
"$QueryResources = $Resource -join ','\r\n",
"$OldEAP = $ErrorActionPreference\r\n",
"$ErrorActionPreference = 'SilentlyContinue'\r\n",
"$Body = @{'resource'= $QueryResources; 'apikey'= $APIKey; 'scan'=$scanurl}\r\n",
"\r\n",
"\r\n",
"# Start building parameters for REST Method invokation.\r\n",
"$Params = @{}\r\n",
"$Params.add('Body', $Body)\r\n",
"$Params.add('Method', 'Get')\r\n",
"$Params.add('Uri',$URI)\r\n",
"$Params.Add('ErrorVariable', 'RESTError')\r\n",
"$ReportResult = Invoke-RestMethod @Params\r\n",
"\r\n",
"$ErrorActionPreference = $OldEAP\r\n",
"\r\n",
"if ($RESTError)\r\n",
"{\r\n",
" if ($RESTError.Message.Contains('403'))\r\n",
" {\r\n",
"throw 'API key is not valid.'\r\n",
" }\r\n",
" elseif ($RESTError.Message -like '*204*')\r\n",
" {\r\n",
"throw 'API key rate has been reached.'\r\n",
" }\r\n",
" else\r\n",
" {\r\n",
"throw $RESTError\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"foreach ($URLReport in $ReportResult)\r\n",
"{\r\n",
" $URLReport.pstypenames.insert(0,'VirusTotal.URL.Report')\r\n",
" Write-host \"Resource:\" $URLReport.resource\r\n",
" Write-host \"Last Scan:\" $URLReport.scan_date\r\n",
" Write-host \"VT Link:\" $URLReport.permalink\r\n",
" Write-host \"Positive Scans:\" $URLReport.positives\r\n",
" Write-host \"Total Scans:\" $URLReport.total \r\n",
" $URLReport\r\n",
"}\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
},
"source": [
"---\r\n",
"## Get your watchlist aliases and data \r\n",
"Retrieve your watchlist aliases and data by running the code below\r\n",
"> **Note**: You must be part of the private preview program to use this feature. Sign-up at [www.aka.ms/SecurityPrP](www.aka.ms/SecurityPrP) to get started!\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1601314697387
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
},
"outputs": [],
"source": [
"##Retrieve watchlist results\r\n",
"$queryResults = $null\r\n",
"$watchlistalias = read-host \"Enter your watchlist alias to get the results:\"\r\n",
"$query = \"_GetWatchlist('$watchlistalias')\"\r\n",
"$queryResults = Invoke-AzOperationalInsightsQuery -Workspace $Workspace -query $query\r\n",
"$queryResults.Results | Select Watchlistitem | select -first 100"
]
}
],
"metadata": {
"kernel_info": {
"name": ".net-powershell"
},
"kernelspec": {
"display_name": "Python 3.8 - AzureML",
"language": "python",
"name": "python38-azureml"
},
"language_info": {
"file_extension": ".ps1",
"mimetype": "text/x-powershell",
"name": "PowerShell",
"pygments_lexer": "powershell",
"version": "7.0"
},
"microsoft": {
"host": {
"AzureML": {
"notebookHasBeenCompleted": true
}
}
},
"nteract": {
"version": "nteract-front-end@1.0.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}