{ "cells": [ { "cell_type": "markdown", "source": [ "# Hands on - Data Discovery using Azure REST API\n", "\n", "__Notebook Version:__ 1.0
\n", "__Python Version:__ Python 3.8 - AzureML
\n", "__Required Packages:__ No
\n", "__Platforms Supported:__ Azure Machine Learning Notebooks\n", " \n", "__Data Source Required:__ Log Analytics tables \n", " \n", "### Description\n", "This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Microsoft Sentinel data discovery by using Azure REST API.
\n", "*** No need to download and install any other Python modules. ***
\n", "*** Please run the cells sequentially to avoid errors. ***
\n", "\n", "## Table of Contents\n", "1. Warm-up\n", "2. Azure Authentication\n", "3. List Microsoft Sentinel Watchlists Using API\n", "4. List Microsoft Sentinel Incidents Using API" ], "metadata": { "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "markdown", "source": [ "## 1. Warm-up" ], "metadata": { "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# If you need to know what Python modules are available, you may run this:\n", "# help(\"modules\")" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1627596066714 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Loading Python libraries\n", "from azure.identity import AzureCliCredential\n", "\n", "import requests\n", "import json\n", "import pandas\n", "from IPython.display import display, HTML, Markdown" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434870178 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Functions will be used in this notebook\n", "def read_config_values(file_path):\n", " \"This loads pre-generated parameters for Microsoft Sentinel Workspace\"\n", " with open(file_path) as json_file:\n", " if json_file:\n", " json_config = json.load(json_file)\n", " return (json_config[\"tenant_id\"],\n", " json_config[\"subscription_id\"],\n", " json_config[\"resource_group\"],\n", " json_config[\"workspace_id\"],\n", " json_config[\"workspace_name\"],\n", " json_config[\"user_alias\"],\n", " json_config[\"user_object_id\"])\n", " return None\n", "\n", "def has_valid_token():\n", " \"Check to see if there is a valid AAD token\"\n", " try:\n", " error = \"ERROR: Please run 'az login' to setup account.\"\n", " expired = \"ERROR: AADSTS70043: The refresh token has expired or is invalid\"\n", " validator = !az account get-access-token\n", " \n", " if any(expired in item for item in validator.get_list()):\n", " return '**The refresh token has expired.
Please continue your login process. Then:
1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\";
2. Otherwise, you may just restart the kernel from top menu.
Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n", " elif any(error in item for item in validator.get_list()):\n", " return \"Please run 'az login' to setup account\"\n", " else:\n", " return None\n", " except:\n", " return \"Please login\"\n", " \n", "# Calling Microsoft Sentinel API for List, the same template can be used for calling other Azure REST APIs with different parameters.\n", "# For different environments, such as national clouds, you may need to use different root_url, please contact with your admins.\n", "# It can be ---.azure.us, ---.azure.microsoft.scloud, ---.azure.eaglex.ic.gov, etc.\n", "def call_azure_rest_api_for_list(token, resource_name, api_version):\n", " \"Calling Microsoft Sentinel REST API\"\n", " headers = {\"Authorization\": token, \"content-type\":\"application/json\" }\n", " provider_name = \"Microsoft.OperationalInsights\"\n", " provider2_name = \"Microsoft.SecurityInsights\"\n", " target_resource_name = resource_name\n", " api_version = api_version\n", " root_url = \"https://management.azure.com\"\n", " arm_rest_url_template_for_list = \"{0}/subscriptions/{1}/resourceGroups/{2}/providers/{3}/workspaces/{4}/providers/{5}/{6}?api-version={7}\"\n", " arm_rest_url = arm_rest_url_template_for_list.format(root_url, subscription_id, resource_group, provider_name, workspace_name, provider2_name, target_resource_name, api_version)\n", " response = requests.get(arm_rest_url, headers=headers, verify=True)\n", " return response\n", "\n", "def display_result_name(response):\n", " \"Default to display column - name, you may change it to other columns\"\n", " column_name = \"name\"\n", " if response != None:\n", " entries = [item[column_name] for item in response.json()[\"value\"]] \n", " display(entries)\n", "\n", "def display_result(response):\n", " \"Display the result set as pandas.DataFrame\"\n", " if response != None:\n", " df = pandas.DataFrame(response.json()[\"value\"])\n", " display(df)" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434872530 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Calling the above function to populate Microsoft Sentinel workspace parameters\n", "# The file, config.json, was generated by the system, however, you may modify the values, or manually set the variables\n", "tenant_id, subscription_id, resource_group, workspace_id, workspace_name, user_alias, user_object_id = read_config_values('config.json');" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434875964 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "markdown", "source": [ "## 2. Azure Authentication" ], "metadata": { "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Azure CLI is used to get device code to login into Azure, you need to copy the code and open the DeviceLogin site.\n", "# You may add [--tenant $tenant_id] to the command\n", "if has_valid_token() != None:\n", " message = '**The refresh token has expired.
Please continue your login process. Then:
1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\";
2. Otherwise, you may just restart the kernel from top menu.
Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n", " display(Markdown(message))\n", " !echo -e '\\e[42m'\n", " !az login --tenant $tenant_id --use-device-code" ], "outputs": [], "execution_count": null, "metadata": { "gather": { "logged": 1632434877884 } } }, { "cell_type": "code", "source": [ "# Extract access token, which will be used to access Microsoft Sentinel Watchlist API for your Watchlist data. \n", "token = AzureCliCredential().get_token('https://management.azure.com')\n", "access_token = token.token\n", "header_token_value = \"Bearer {}\".format(access_token)" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434887274 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "markdown", "source": [ "## 3. List Microsoft Sentinel Watchlists Using API" ], "metadata": { "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Calling Microsoft Sentinel Watchlist API\n", "# If you don't have Watchlist, you may create one, or try to access different features, such as Bookmarks.\n", "response_watchlist = call_azure_rest_api_for_list(header_token_value, \"watchlists\", \"2019-01-01-preview\")" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434893904 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Display the result\n", "display_result_name(response_watchlist)" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434896691 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "markdown", "source": [ "## 4. List Microsoft Sentinel Incidents Using API" ], "metadata": { "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Calling Microsoft Sentinel Incident API\n", "# If you don't have incidents, you may create one through Azure Portal.\n", "response_incident = call_azure_rest_api_for_list(header_token_value, \"incidents\", \"2020-01-01\")" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434901252 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "code", "source": [ "# Display the result\n", "display_result(response_incident)" ], "outputs": [], "execution_count": null, "metadata": { "collapsed": true, "gather": { "logged": 1632434904269 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } } }, { "cell_type": "markdown", "source": [ "Thanks for coming along all the way to the end. In the next Hands-on notebook, I will show you how to access data using Azure SDK for Python. And keep one of the watchlist name, it will be used in the next notebook. A la prochaine." ], "metadata": { "nteract": { "transient": { "deleting": false } } } } ], "metadata": { "celltoolbar": "Tags", "kernel_info": { "name": "python38-azureml" }, "kernelspec": { "name": "python38-azureml", "language": "python", "display_name": "Python 3.8 - AzureML" }, "language_info": { "name": "python", "version": "3.8.5", "mimetype": "text/x-python", "codemirror_mode": { "name": "ipython", "version": 3 }, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py" }, "microsoft": { "host": { "AzureML": { "notebookHasBeenCompleted": true } } }, "nteract": { "version": "nteract-front-end@1.0.0" } }, "nbformat": 4, "nbformat_minor": 0 }