{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Troubleshooting Azure Sentinel Notebooks\n", "\n", "If you are having trouble with Jupyter notebooks run this notebook to help\n", "identify where the problem might be.\n", "\n", "Select the notebook menu item `Cell->Run All` - check for any warnings or errors.\n", "\n", "Read the text above the cell(s) that produce errors - the text\n", "contains links to resources that describe how to fix the error." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Python Version Check\n", "\n", "> **Note**\n", "> You can set the default Python version in Azure Notebooks project settings.\n", "> \n", "> For details on how to do this see [AzureNotebooks-ConfigurePythonVersion](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/AzureNotebooks-ConfigurePythonVersion.ipynb)\n", ">\n", "> If you are using a Data Science Virtual Machine as your\n", "> Azure Notebooks compute you should read [Provisioning a DSVM](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/Provisioning%20DSVM.ipynb)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:21.030512Z", "start_time": "2020-02-27T00:34:21.016520Z" } }, "outputs": [], "source": [ "import sys\n", "from IPython.display import display, HTML, Markdown\n", "MIN_REQ_PYTHON = (3, 6)\n", "\n", "errors = []\n", "warns = []\n", "info = []\n", "def setup_err(mssg):\n", " display(Markdown(\"

Setup Error

\"))\n", " display(Markdown(\"

%s

\" % mssg))\n", " errors.append(mssg)\n", "\n", "def setup_ok(mssg):\n", " display(Markdown(\"

%s Ok

\" % mssg))\n", " info.append(mssg)\n", "\n", "def setup_warn(mssg):\n", " display(Markdown(\"

%s

\" % mssg))\n", " warns.append(mssg)\n", " \n", "display(Markdown(\"#### Checking Python version...\"))\n", "if sys.version_info < MIN_REQ_PYTHON:\n", " setup_err(\"Python version\")\n", " display(Markdown('Check the Kernel->Change Kernel menu and ensure that Python 3.6'))\n", " display(Markdown('or later is selected as the active kernel.'))\n", "else:\n", " setup_ok(\n", " \"Python version {}.{}.{}\".format(\n", " sys.version_info[0], sys.version_info[1], sys.version_info[2]\n", " )\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Package Import Check\n", "\n", "This section checks the import of `msticpy` and its dependent packages.\n", "\n", "> **Note**\n", "> If you are repeatedly seeing packages going missing when working in Azure Notebooks\n", "> this may be because the docker containers running the Python kernel are\n", "> recycled after a few hours when not in use. This causes the environments\n", "> to reset to defaults.\n", "> \n", "> To prevent this you should configure you Azure Notebooks project with a\n", "> requirements.txt file that is automatically run (and packages installed)\n", "> when the contain is initialized.\n", ">\n", "> For details on how to do this see [AzureNotebooks-ConfigurePythonVersion](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/HowTos/AzureNotebooks-ConfigurePythonVersion.ipynb)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:26.670210Z", "start_time": "2020-02-27T00:34:21.032510Z" }, "scrolled": true }, "outputs": [], "source": [ "import importlib\n", "import sys\n", "import warnings\n", "from IPython.display import display, HTML, Markdown\n", "\n", "MSTICPY_REQ_VERSION = (0, 2, 7)\n", "display(Markdown(\"#### Checking msticpy...\"))\n", "warn_mssg = []\n", "err_mssg = []\n", "restart_req = False\n", "\n", "MISSING_PKG_ERR = \"\"\"\n", "

Warning {package} is not installed or has an unsupported version

\n", " \"\"\"\n", "need_update = False\n", "try:\n", " import msticpy\n", " mp_version = tuple([int(v) for v in msticpy.__version__.split(\".\")])\n", " if mp_version < MSTICPY_REQ_VERSION:\n", " setup_err(\"msticpy %s.%s.%s or later is required.\" % MSTICPY_REQ_VERSION)\n", " need_update = True\n", "except ImportError:\n", " display(HTML(MISSING_PKG_ERR.format(package=\"msticpy\")))\n", " need_update = True\n", " \n", "else:\n", " setup_ok(f\"msticpy version {msticpy.__version__}\")\n", "\n", "if need_update:\n", " resp = input(\"Install the package now? (y/n)\")\n", " if resp.casefold().startswith(\"y\"):\n", " !pip install --user --upgrade msticpy\n", " if \"msticpy\" in sys.modules:\n", " importlib.reload(msticpy)\n", " else:\n", " import msticpy\n", " print(f\"msticpy installed - version {msticpy.__version__}\")\n", " \n", " else:\n", " setup_warn(\"msticpy missing or out-of-date.\")\n", " display(Markdown(\"Please run `pip install --user --upgrade msticpy` to upgrade/install msticpy\"))\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas Version Check\n", "\n", "Many of the notebooks and msticpy features require a mininum\n", "pandas version of 0.25.0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:26.683205Z", "start_time": "2020-02-27T00:34:26.671186Z" } }, "outputs": [], "source": [ "display(Markdown(\"#### Checking pandas...\"))\n", "PANDAS_REQ_VERSION = (0, 25, 0)\n", "need_update = False\n", "try:\n", " import pandas as pd\n", " pd_version = tuple([int(v) for v in pd.__version__.split(\".\")])\n", " if pd_version < PANDAS_REQ_VERSION:\n", " setup_err(\"pandas %s.%s.%s or later is required.\" % PANDAS_REQ_VERSION)\n", " need_update = True\n", "except ImportError:\n", " display(HTML(MISSING_PKG_ERR.format(package=\"pandas\")))\n", " need_update = True\n", "else:\n", " setup_ok(f\"Pandas version {pd.__version__}\")\n", " \n", "if need_update:\n", " resp = input(\"Install the package now? (y/n)\")\n", " if resp.casefold().startswith(\"y\"):\n", " !pip install --user --upgrade pandas\n", " if \"pandas\" in sys.modules:\n", " importlib.reload(pd)\n", " else:\n", " import pandas as pd\n", " print(f\"pandas installed - version {pandas.__version__}\")\n", " \n", " else:\n", " setup_warn(\"pandas missing or out-of-date.\")\n", " display(Markdown(\"Please run `pip install --user --upgrade pandas` to upgrade/install pandas\"))" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-02-24T19:41:28.539607Z", "start_time": "2020-02-24T19:41:28.536637Z" } }, "source": [ "## Workspace Configuration Check\n", "\n", "This section checks for presence of configuration files `config.json`\n", "and `msticpyconfig.yaml`\n", "\n", "The `msticpyconfig.yaml` can store the workspace and tenant information\n", "for your Azure Sentinel workspace. It can also store values for multiple\n", "workspaces. If you have the values configured in this file you do not\n", "need to worry about the values in `config.json`.\n", "\n", "You can specify the location of your `msticpyconfig.yaml` in the\n", "environment variable `MSTICPYCONFIG`. This will make the file \n", "accessible to all notebooks running on the system. For\n", "more information on configuring `msticpyconfig.yaml` see the next\n", "cell [mstipcy Configuration](#msticpy-Configuration)\n", "\n", "If you want to transfer your workspace settings to `msticpyconfig.yaml`\n", "from `config.json`, simply copy the value of the `tenant_id` and \n", "`workspace_id` settings to the relevant section. \n", "\n", "> **Note** the value names in msticpyconfig.yaml use slightly different naming\n", "> conventions:\n", "```\n", " WorkspaceId: 0cd830ff-60dc-40d1-8045-11d2b7b277e1\n", " TenantId: aff2102d-1d6c-4501-9efb-6053ab7efb19\n", "```\n", "\n", "### Workspace Configuration - config.json\n", "Creating an Azure Notebooks project from Azure Sentinel \n", "will automatically create a `config.json` file in the root of\n", "your Azure Notebooks project and populate values\n", "for your Azure Sentinel workspace.\n", "\n", "If you have copied the notebooks elsewhere (e.g. to run them locally,\n", "or you are running them on a Data Science Virtual machine) you \n", "should copy this original config.json to the folder from which\n", "you are running notebooks.\n", "\n", "> **Note** if you are using a `msticpyconfig.yaml` to store your\n", "> workspace settings, most notebooks will take values from that.\n", "> As with `config.json` - you must have a locally accessible copy\n", "> of this file, so you will need to copy it to other systems if\n", "> you are running notebooks from there.\n", "\n", "\n", "If you are using the config.json (default config for Azure Sentinel\n", "with Azure Notebooks), your config.json should look something like this\n", "```json\n", "{\n", " \"tenant_id\": \"aff2102d-1d6c-4501-9efb-6053ab7efb19\",\n", " \"subscription_id\": \"9ce7caeb-1f42-4141-b076-7f448a00aceb\",\n", " \"resource_group\": \"MyResourceGroup\",\n", " \"workspace_id\": \"0cd830ff-60dc-40d1-8045-11d2b7b277e1\",\n", " \"workspace_name\": \"MyResourceSubscription\"\n", "}\n", "```\n", "\n", "The tenant_id and workspace_id values must be configured, other values\n", "are optional but recommended." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:26.714184Z", "start_time": "2020-02-27T00:34:26.684179Z" }, "scrolled": true }, "outputs": [], "source": [ "import os\n", "\n", "import json\n", "from pathlib import Path\n", "import uuid\n", "import yaml\n", "\n", "\n", "def valid_uuid(uuid_str):\n", " try:\n", " uuid.UUID(uuid_str)\n", " except (ValueError, TypeError):\n", " return False\n", " return True\n", "\n", "def check_mp_config_ws(mp_path):\n", " with open(mp_path, \"r\") as mp_yml:\n", " mp_config = yaml.safe_load(mp_yml)\n", " mp_errors = []\n", " as_settings = mp_config.get(\"AzureSentinel\", {})\n", " if not as_settings:\n", " mp_errors.append(f\"Missing or empty 'AzureSentinel' section in {mp_path}\")\n", " ws_settings = as_settings.get(\"Workspaces\", {})\n", " if not ws_settings:\n", " mp_errors.append(f\"Missing or empty 'Workspaces' section in {mp_path}\")\n", " no_default = True\n", " for ws, ws_settings in ws_settings.items():\n", " if ws == \"Default\":\n", " no_default = False\n", " ws_id = ws_settings.get(\"WorkspaceId\")\n", " if not ws_id and not valid_uuid(ws_id):\n", " mp_errors.append(f\"Invalid GUID for WorkspaceId in {ws} section\")\n", " ten_id = ws_settings.get(\"TenantId\")\n", " if not ten_id and not valid_uuid(ten_id):\n", " mp_errors.append(f\"Invalid GUID for TenantId in {ws} section\")\n", " warnings = [\"No default workspace set\"] if no_default else []\n", " return mp_errors, warnings\n", "\n", "def check_json_config(json_path):\n", " j_conf_errs = []\n", " with open(json_path, \"r\") as json_file:\n", " conf_json = json.load(json_file)\n", " conf_tenant = conf_json.get(\"tenant_id\")\n", " if conf_tenant == \"{{cookiecutter.tenant_id}}\":\n", " j_conf_errs.append(\"Tenant Id is set to default value\")\n", " elif not valid_uuid(conf_tenant):\n", " j_conf_errs.append(\"Tenant ID is not a valid GUID.\")\n", " conf_ws = conf_json.get(\"workspace_id\")\n", " if conf_ws == \"{{cookiecutter.workspace_id}}\":\n", " j_conf_errs.append(\"Workspace Id is set to default value\")\n", " elif not valid_uuid(conf_ws):\n", " j_conf_errs.append(\"Workspace ID is not a valid GUID.\")\n", " return j_conf_errs\n", "\n", "\n", "mp_warnings = []\n", "display(Markdown(\"#### Checking Azure Sentinel Workspace config...\"))\n", "mp_path = os.environ.get(\"MSTICPYCONFIG\", \"./msticpyconfig.yaml\")\n", "if Path(mp_path).exists():\n", " mp_errs, mp_warnings = check_mp_config_ws(mp_path)\n", "else:\n", " mp_errs = [f\"{mp_path} not found\"]\n", " \n", "DEF_CONF_JSON = \"./config.json\"\n", "if Path(DEF_CONF_JSON).exists():\n", " jc_errs = check_json_config(DEF_CONF_JSON)\n", "\n", "if jc_errs and mp_errs:\n", " setup_err(\"No valid workspace configuration found.\")\n", " if jc_errs:\n", " print(jc_errs)\n", " if mp_errs:\n", " print(mp_errs)\n", "else:\n", " if not jc_errs:\n", " setup_ok(f\"Workspace configuration found in '{DEF_CONF_JSON}'\")\n", " if not mp_errs:\n", " setup_ok(f\"Workspace configuration found in '{mp_path}'\")\n", " else:\n", " setup_warn(f\"Workspace configuration: Cannot find msticpy config {mp_path}\")\n", " if mp_warnings:\n", " display(Markdown(f\"
{', '.join(mp_warnings)}
\"))" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-02-24T19:41:42.199685Z", "start_time": "2020-02-24T19:41:42.196662Z" } }, "source": [ "# msticpy Configuration\n", "\n", "The msticpy configuration file `msticpyconfig.yaml` holds setting\n", "required by TI Providers and other data providers such as GeoIP.\n", "\n", "These features will not work unless you valid API Keys from your\n", "accounts with these providers stored in `msticpyconfig.yaml`.\n", "\n", "> **Note** you can store the actual values of the keys in \n", "> environment variables\n", "> and use the settings in `msticpyconfig.yaml` to reference these. \n", "\n", "For more information on `msticpy` configuration file settings,\n", "please refer to the following items:\n", "\n", "- [Configuration guide notebook](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb)\n", "- [msticpy configuration documentation](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:26.742145Z", "start_time": "2020-02-27T00:34:26.715162Z" }, "scrolled": true }, "outputs": [], "source": [ "import os\n", "import json\n", "from pathlib import Path\n", "import uuid\n", "import yaml\n", "\n", "def check_mp_config_providers(mp_path):\n", " mp_errors = []\n", " if not Path(mp_path).exists():\n", " mp_errors.append(f\"No msticpyconfig.yaml found.\")\n", " return mp_errors\n", " \n", " with open(mp_path, \"r\") as mp_yml:\n", " mp_config = yaml.safe_load(mp_yml)\n", " \n", " for conf_section in [\"TIProviders\", \"OtherProviders\"]:\n", " display(Markdown(f\"Checking {conf_section}...\"))\n", " settings = mp_config.get(conf_section, {})\n", " mp_errors += check_provider_settings(settings, conf_section)\n", " \n", "def check_provider_settings(settings, section):\n", " mp_errors = []\n", " if not settings:\n", " mp_errors.append(f\"Missing or empty '{section}' section in {mp_path}\")\n", " for p_name, p_setting in settings.items():\n", " print(p_name, end=\" \")\n", " ti_args = p_setting.get(\"Args\")\n", " if p_name in [\"OTX\", \"VirusTotal\", \"XForce\", \"OpenPageRank\"]:\n", " if (\n", " \"AuthKey\" not in ti_args\n", " or not ti_args[\"AuthKey\"]\n", " ):\n", " mp_errors.append(f\"{section}: Missing or invalid AuthKey for {p_name} section\")\n", " if p_name == \"XForce\":\n", " if (\n", " \"ApiKey\" not in ti_args\n", " or not ti_args[\"ApiKey\"]\n", " ):\n", " mp_errors.append(f\"{section}: Missing or invalid ApiKey for {p_name} section\")\n", " if p_name == \"AzureSentinel\":\n", " ws_id = p_setting.get(\"WorkspaceID\")\n", " if not ws_id or not valid_uuid(ws_id):\n", " mp_errors.append(f\"{section}: Invalid GUID for WorkspaceID in {p_name} section\")\n", " ten_id = p_setting.get(\"TenantID\")\n", " if not ten_id or not valid_uuid(ten_id):\n", " mp_errors.append(f\"{section}: Invalid GUID for TenantID in {p_name} section\")\n", " print()\n", " return mp_errors\n", "\n", "\n", "display(Markdown(\"#### Checking msticpy config...\"))\n", "mp_path = os.environ.get(\"MSTICPYCONFIG\", \"./msticpyconfig.yaml\")\n", "if not Path(mp_path).exists():\n", " setup_err(f\"Cannot find msticpy config {mp_path}\")\n", " if \"MSTICPYCONFIG\" in os.environ:\n", " setup_warn(f\"'MSTICPYCONFIG' points to non-existent file\")\n", "else:\n", " mp_errs = check_mp_config_providers(mp_path)\n", " if mp_errs:\n", " setup_err(\"Invalid msticpy configuration found.\")\n", " print(mp_errs)\n", " else:\n", " setup_ok(f\"msticpy configuration in '{mp_path}'\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Summary" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-02-27T00:34:26.758139Z", "start_time": "2020-02-27T00:34:26.745144Z" } }, "outputs": [], "source": [ "if errors:\n", " display(Markdown(f\"

{len(errors)} errors:\"))\n", " for mssg in errors:\n", " display(Markdown(f\"{mssg}\"))\n", "if warns:\n", " display(Markdown(f\"

{len(warns)} warnings:\"))\n", " for mssg in warns:\n", " display(Markdown(f\"{mssg}\"))\n", "display(Markdown(f\"

Info/Success:\"))\n", "for mssg in info:\n", " display(Markdown(f\"{mssg}\"))" ] } ], "metadata": { "hide_input": false, "kernelspec": { "display_name": "Python 3.6", "language": "python", "name": "python36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }