{
"cells": [
{
"cell_type": "markdown",
"source": [
"# Hands on - Data Discovery using Azure REST API\n",
"\n",
"__Notebook Version:__ 1.0
\n",
"__Python Version:__ Python 3.8 - AzureML
\n",
"__Required Packages:__ No
\n",
"__Platforms Supported:__ Azure Machine Learning Notebooks\n",
" \n",
"__Data Source Required:__ Log Analytics tables \n",
" \n",
"### Description\n",
"This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Microsoft Sentinel data discovery by using Azure REST API.
\n",
"*** No need to download and install any other Python modules. ***
\n",
"*** Please run the cells sequentially to avoid errors. ***
\n",
"\n",
"## Table of Contents\n",
"1. Warm-up\n",
"2. Azure Authentication\n",
"3. List Microsoft Sentinel Watchlists Using API\n",
"4. List Microsoft Sentinel Incidents Using API"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"## 1. Warm-up"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# If you need to know what Python modules are available, you may run this:\n",
"# help(\"modules\")"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1627596066714
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Loading Python libraries\n",
"from azure.identity import AzureCliCredential\n",
"\n",
"import requests\n",
"import json\n",
"import pandas\n",
"from IPython.display import display, HTML, Markdown"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434870178
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Functions will be used in this notebook\n",
"def read_config_values(file_path):\n",
" \"This loads pre-generated parameters for Microsoft Sentinel Workspace\"\n",
" with open(file_path) as json_file:\n",
" if json_file:\n",
" json_config = json.load(json_file)\n",
" return (json_config[\"tenant_id\"],\n",
" json_config[\"subscription_id\"],\n",
" json_config[\"resource_group\"],\n",
" json_config[\"workspace_id\"],\n",
" json_config[\"workspace_name\"],\n",
" json_config[\"user_alias\"],\n",
" json_config[\"user_object_id\"])\n",
" return None\n",
"\n",
"def has_valid_token():\n",
" \"Check to see if there is a valid AAD token\"\n",
" try:\n",
" error = \"ERROR: Please run 'az login' to setup account.\"\n",
" expired = \"ERROR: AADSTS70043: The refresh token has expired or is invalid\"\n",
" validator = !az account get-access-token\n",
" \n",
" if any(expired in item for item in validator.get_list()):\n",
" return '**The refresh token has expired.
Please continue your login process. Then:
1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\";
2. Otherwise, you may just restart the kernel from top menu.
Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n",
" elif any(error in item for item in validator.get_list()):\n",
" return \"Please run 'az login' to setup account\"\n",
" else:\n",
" return None\n",
" except:\n",
" return \"Please login\"\n",
" \n",
"# Calling Microsoft Sentinel API for List, the same template can be used for calling other Azure REST APIs with different parameters.\n",
"# For different environments, such as national clouds, you may need to use different root_url, please contact with your admins.\n",
"# It can be ---.azure.us, ---.azure.microsoft.scloud, ---.azure.eaglex.ic.gov, etc.\n",
"def call_azure_rest_api_for_list(token, resource_name, api_version):\n",
" \"Calling Microsoft Sentinel REST API\"\n",
" headers = {\"Authorization\": token, \"content-type\":\"application/json\" }\n",
" provider_name = \"Microsoft.OperationalInsights\"\n",
" provider2_name = \"Microsoft.SecurityInsights\"\n",
" target_resource_name = resource_name\n",
" api_version = api_version\n",
" root_url = \"https://management.azure.com\"\n",
" arm_rest_url_template_for_list = \"{0}/subscriptions/{1}/resourceGroups/{2}/providers/{3}/workspaces/{4}/providers/{5}/{6}?api-version={7}\"\n",
" arm_rest_url = arm_rest_url_template_for_list.format(root_url, subscription_id, resource_group, provider_name, workspace_name, provider2_name, target_resource_name, api_version)\n",
" response = requests.get(arm_rest_url, headers=headers, verify=True)\n",
" return response\n",
"\n",
"def display_result_name(response):\n",
" \"Default to display column - name, you may change it to other columns\"\n",
" column_name = \"name\"\n",
" if response != None:\n",
" entries = [item[column_name] for item in response.json()[\"value\"]] \n",
" display(entries)\n",
"\n",
"def display_result(response):\n",
" \"Display the result set as pandas.DataFrame\"\n",
" if response != None:\n",
" df = pandas.DataFrame(response.json()[\"value\"])\n",
" display(df)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434872530
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Calling the above function to populate Microsoft Sentinel workspace parameters\n",
"# The file, config.json, was generated by the system, however, you may modify the values, or manually set the variables\n",
"tenant_id, subscription_id, resource_group, workspace_id, workspace_name, user_alias, user_object_id = read_config_values('config.json');"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434875964
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"## 2. Azure Authentication"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Azure CLI is used to get device code to login into Azure, you need to copy the code and open the DeviceLogin site.\n",
"# You may add [--tenant $tenant_id] to the command\n",
"if has_valid_token() != None:\n",
" message = '**The refresh token has expired.
Please continue your login process. Then:
1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\";
2. Otherwise, you may just restart the kernel from top menu.
Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n",
" display(Markdown(message))\n",
" !echo -e '\\e[42m'\n",
" !az login --tenant $tenant_id --use-device-code"
],
"outputs": [],
"execution_count": null,
"metadata": {
"gather": {
"logged": 1632434877884
}
}
},
{
"cell_type": "code",
"source": [
"# Extract access token, which will be used to access Microsoft Sentinel Watchlist API for your Watchlist data. \n",
"token = AzureCliCredential().get_token('https://management.azure.com')\n",
"access_token = token.token\n",
"header_token_value = \"Bearer {}\".format(access_token)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434887274
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"## 3. List Microsoft Sentinel Watchlists Using API"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Calling Microsoft Sentinel Watchlist API\n",
"# If you don't have Watchlist, you may create one, or try to access different features, such as Bookmarks.\n",
"response_watchlist = call_azure_rest_api_for_list(header_token_value, \"watchlists\", \"2019-01-01-preview\")"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434893904
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Display the result\n",
"display_result_name(response_watchlist)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434896691
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"## 4. List Microsoft Sentinel Incidents Using API"
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Calling Microsoft Sentinel Incident API\n",
"# If you don't have incidents, you may create one through Azure Portal.\n",
"response_incident = call_azure_rest_api_for_list(header_token_value, \"incidents\", \"2020-01-01\")"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434901252
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "code",
"source": [
"# Display the result\n",
"display_result(response_incident)"
],
"outputs": [],
"execution_count": null,
"metadata": {
"collapsed": true,
"gather": {
"logged": 1632434904269
},
"jupyter": {
"outputs_hidden": false,
"source_hidden": false
},
"nteract": {
"transient": {
"deleting": false
}
}
}
},
{
"cell_type": "markdown",
"source": [
"Thanks for coming along all the way to the end. In the next Hands-on notebook, I will show you how to access data using Azure SDK for Python. And keep one of the watchlist name, it will be used in the next notebook. A la prochaine."
],
"metadata": {
"nteract": {
"transient": {
"deleting": false
}
}
}
}
],
"metadata": {
"celltoolbar": "Tags",
"kernel_info": {
"name": "python38-azureml"
},
"kernelspec": {
"name": "python38-azureml",
"language": "python",
"display_name": "Python 3.8 - AzureML"
},
"language_info": {
"name": "python",
"version": "3.8.5",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"microsoft": {
"host": {
"AzureML": {
"notebookHasBeenCompleted": true
}
}
},
"nteract": {
"version": "nteract-front-end@1.0.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}