{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ " # Windows Host Explorer\n", " <details>\n", "  Details...\n", "\n", " **Notebook Version:** 1.0
\n", " **Python Version:** Python 3.6 (including Python 3.6 - AzureML)
\n", " **Required Packages**: kqlmagic, msticpy, pandas, numpy, matplotlib, bokeh, networkx, ipywidgets, ipython, scikit_learn, dnspython, ipwhois, folium, maxminddb_geolite2
\n", " **Platforms Supported**:\n", " - Azure Notebooks Free Compute\n", " - Azure Notebooks DSVM\n", " - OS Independent\n", "\n", " **Data Sources Required**:\n", " - Log Analytics - SecurityAlert, SecurityEvent (EventIDs 4688 and 4624/25), AzureNetworkAnalytics_CL, Heartbeat\n", " - (Optional) - VirusTotal, AlienVault OTX, IBM XForce, Open Page Rank, (all require accounts and API keys)\n", " </details>\n", "\n", " Brings together a series of queries and visualizations to help you determine the security state of the Windows host or virtual machine that you are investigating.\n" ] }, { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "### Notebook initialization\n", "The next cell:\n", "- Checks for the correct Python version\n", "- Checks versions and optionally installs required packages\n", "- Imports the required packages into the notebook\n", "- Sets a number of configuration options.\n", "\n", "This should complete without errors. If you encounter errors or warnings look at the following two notebooks:\n", "- [TroubleShootingNotebooks](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/TroubleShootingNotebooks.ipynb)\n", "- [ConfiguringNotebookEnvironment](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb)\n", "\n", "If you are running in the Azure Sentinel Notebooks environment (Azure Notebooks or Azure ML) you can run live versions of these notebooks:\n", "- [Run TroubleShootingNotebooks](./TroubleShootingNotebooks.ipynb)\n", "- [Run ConfiguringNotebookEnvironment](./ConfiguringNotebookEnvironment.ipynb)\n", "\n", "You may also need to do some additional configuration to successfully use functions such as Threat Intelligence service lookup and Geo IP lookup. \n", "There are more details about this in the `ConfiguringNotebookEnvironment` notebook and in these documents:\n", "- [msticpy configuration](https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html)\n", "- [Threat intelligence provider configuration](https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html#configuration-file)\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:27:18.623464Z", "start_time": "2020-05-15T23:27:15.156160Z" } }, "outputs": [ { "data": { "text/html": [ "Environment setup has completed." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Continuing wth notebook setup." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "Checking msticpy version..." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "msticpy version 0.5.1 OK" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Processing imports....\n", "Checking configuration....\n", "Setting options....\n" ] }, { "data": { "text/html": [ "

Notebook setup complete

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from pathlib import Path\n", "import os\n", "import sys\n", "import warnings\n", "from IPython.display import display, HTML, Markdown\n", "\n", "REQ_PYTHON_VER=(3, 6)\n", "REQ_MSTICPY_VER=(0, 5, 0)\n", "\n", "display(HTML(\"

Starting Notebook setup...

\"))\n", "if Path(\"./utils/nb_check.py\").is_file():\n", " from utils.nb_check import check_python_ver, check_mp_ver\n", "\n", " check_python_ver(min_py_ver=REQ_PYTHON_VER)\n", " try:\n", " check_mp_ver(min_msticpy_ver=REQ_MSTICPY_VER)\n", " except ImportError:\n", " !pip install --upgrade msticpy\n", " if \"msticpy\" in sys.modules:\n", " importlib.reload(msticpy)\n", " else:\n", " import msticpy\n", " check_mp_ver(REQ_PYTHON_VER)\n", " \n", "\n", "# If not using Azure Notebooks, install msticpy with\n", "# !pip install msticpy\n", "from msticpy.nbtools import nbinit\n", "nbinit.init_notebook(\n", " namespace=globals(),\n", " extra_imports=[\"ipwhois, IPWhois\"]\n", ");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## Get WorkspaceId and Authenticate to Azure Sentinel\n", " <details>\n", " Details...\n", " If you are using user/device authentication, run the following cell.\n", " - Click the 'Copy code to clipboard and authenticate' button.\n", " - This will pop up an Azure Active Directory authentication dialog (in a new tab or browser window). The device code will have been copied to the clipboard.\n", " - Select the text box and paste (Ctrl-V/Cmd-V) the copied value.\n", " - You should then be redirected to a user authentication page where you should authenticate with a user account that has permission to query your Log Analytics workspace.\n", "\n", " Use the following syntax if you are authenticating using an Azure Active Directory AppId and Secret:\n", " ```\n", " %kql loganalytics://tenant(aad_tenant).workspace(WORKSPACE_ID).clientid(client_id).clientsecret(client_secret)\n", " ```\n", " instead of\n", " ```\n", " %kql loganalytics://code().workspace(WORKSPACE_ID)\n", " ```\n", "\n", " Note: you may occasionally see a JavaScript error displayed at the end of the authentication - you can safely ignore this.
\n", " On successful authentication you should see a ```popup schema``` button.\n", " To find your Workspace Id go to [Log Analytics](https://ms.portal.azure.com/#blade/HubsExtension/Resources/resourceType/Microsoft.OperationalInsights%2Fworkspaces). Look at the workspace properties to find the ID.\n", " </details>" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:27:22.847608Z", "start_time": "2020-05-15T23:27:22.839609Z" } }, "outputs": [ { "data": { "text/html": [ "

Workspace details collected from config file

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "#See if we have an Azure Sentinel Workspace defined in our config file, if not let the user specify Workspace and Tenant IDs\n", "from msticpy.nbtools.wsconfig import WorkspaceConfig\n", "# WorkspaceConfig.list_workspaces()\n", "# ws_config = WorkspaceConfig(workspace=\"My_Workspace_Name\")\n", "# calling WorkspaceConfig with no parameters will load the default workspace from msticpyconfig.yaml\n", "# or fall back on a config.json file.\n", "ws_config = WorkspaceConfig()\n", "try:\n", " ws_id = ws_config['workspace_id']\n", " ten_id = ws_config['tenant_id']\n", " config = True\n", " md(\"Workspace details collected from config file\")\n", "except KeyError:\n", " md(('Please go to your Log Analytics workspace, copy the workspace ID'\n", " ' and/or tenant Id and paste here to enable connection to the workspace and querying of it..
'))\n", " ws_id_wgt = nbwidgets.GetEnvironmentKey(env_var='WORKSPACE_ID',\n", " prompt='Please enter your Log Analytics Workspace Id:', auto_display=True)\n", " ten_id_wgt = nbwidgets.GetEnvironmentKey(env_var='TENANT_ID',\n", " prompt='Please enter your Log Analytics Tenant Id:', auto_display=True)\n", " config = False\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:28:39.796803Z", "start_time": "2020-05-15T23:27:27.080209Z" } }, "outputs": [ { "data": { "application/javascript": [ "try {IPython.notebook.kernel.execute(\"NOTEBOOK_URL = '\" + window.location + \"'\");} catch(err) {;}" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " \n", "\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", "
\n", " \n", "\n", " \n", " \n", "
\n", "\n", " \n", "\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "if config is False:\n", " ws_id = ws_id_wgt.value\n", " ten_id = ten_id_wgt.value\n", "# Establish a query provider for Azure Sentinel and connect to it\n", "qry_prov = QueryProvider('LogAnalytics')\n", "qry_prov.connect(connection_str=ws_config.code_connect_str)\n", "table_index = qry_prov.schema_tables" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2019-10-31T23:37:18.211230Z", "start_time": "2019-10-31T23:37:18.204259Z" } }, "source": [ "### Authentication and Configuration Problems\n", "\n", "
\n", "
\n", " Click for details about configuring your authentication parameters\n", " \n", " \n", "The notebook is expecting your Azure Sentinel Tenant ID and Workspace ID to be configured in one of the following places:\n", "- `config.json` in the current folder\n", "- `msticpyconfig.yaml` in the current folder or location specified by `MSTICPYCONFIG` environment variable.\n", " \n", "For help with setting up your `config.json` file (if this hasn't been done automatically) see the [`ConfiguringNotebookEnvironment`](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb) notebook in the root folder of your Azure-Sentinel-Notebooks project. This shows you how to obtain your Workspace and Subscription IDs from the Azure Sentinel Portal. You can use the SubscriptionID to find your Tenant ID). To view the current `config.json` run the following in a code cell.\n", "\n", "```%pfile config.json```\n", "\n", "For help with setting up your `msticpyconfig.yaml` see the [Setup](#Setup) section at the end of this notebook and the [ConfigureNotebookEnvironment notebook](https://github.com/Azure/Azure-Sentinel-Notebooks/blob/master/ConfiguringNotebookEnvironment.ipynb)\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Contents](#Contents)\n", " # Search for a Host name and query host properties" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:28:41.610484Z", "start_time": "2020-05-15T23:28:41.598485Z" } }, "outputs": [], "source": [ "host_text = widgets.Text(\n", " description=\"Enter the Host name to search for:\", **WIDGET_DEFAULTS\n", ")\n", "display(host_text)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:28:46.198826Z", "start_time": "2020-05-15T23:28:46.144827Z" } }, "outputs": [], "source": [ "query_times = nbwidgets.QueryTime(units=\"day\", max_before=20, before=5, max_after=1)\n", "query_times.display()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:28:58.158859Z", "start_time": "2020-05-15T23:28:55.922817Z" } }, "outputs": [], "source": [ "# Get single event - try process creation\n", "if \"SecurityEvent\" not in table_index:\n", " raise ValueError(\"No Windows event log data available in the workspace\")\n", "host_name = None\n", "matching_hosts_df = qry_prov.WindowsSecurity.list_host_processes(\n", " query_times, host_name=host_text.value.strip(), add_query_items=\"| distinct Computer\"\n", ")\n", "if len(matching_hosts_df) > 1:\n", " print(f\"Multiple matches for '{host_text.value}'. Please select a host from the list.\")\n", " choose_host = nbwidgets.SelectString(\n", " item_list=list(matching_hosts_df[\"Computer\"].values),\n", " description=\"Select the host.\",\n", " auto_display=True,\n", " )\n", "elif not matching_hosts_df.empty:\n", " host_name = matching_hosts_df[\"Computer\"].iloc[0]\n", " print(f\"Unique host found: {host_name}\")\n", "else:\n", " print(f\"Host not found: {host_text.value}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:12.506439Z", "start_time": "2020-05-15T23:29:01.493356Z" } }, "outputs": [], "source": [ "if not host_name:\n", " host_name = choose_host.value\n", "\n", "host_entity = None\n", "if not matching_hosts_df.empty:\n", " host_entity = entities.Host(src_event=matching_hosts_df[matching_hosts_df[\"Computer\"] == host_name].iloc[0])\n", "if not host_entity:\n", " raise LookupError(f\"Could not find Windows events the name {host_name}\")\n", "\n", "def populate_heartbeat_details(host_hb_df, host_entity=None):\n", " if not host_hb_df.empty:\n", " host_hb = host_hb_df.iloc[0]\n", " if not host_entity:\n", " host_entity = entities.Host(host_hb[\"Computer\"])\n", " host_entity.SourceComputerId = host_hb[\"SourceComputerId\"]\n", " host_entity.OSType = host_hb[\"OSType\"]\n", " host_entity.OSMajorVersion = host_hb[\"OSMajorVersion\"]\n", " host_entity.OSMinorVersion = host_hb[\"OSMinorVersion\"]\n", " host_entity.ComputerEnvironment = host_hb[\"ComputerEnvironment\"]\n", " host_entity.ResourceId = host_hb[\"ResourceId\"]\n", " host_entity.OmsSolutions = [\n", " sol.strip() for sol in host_hb[\"Solutions\"].split(\",\")\n", " ]\n", " host_entity.VMUUID = host_hb[\"VMUUID\"]\n", "\n", " ip_entity = entities.IpAddress()\n", " ip_entity.Address = host_hb[\"ComputerIP\"]\n", " geoloc_entity = entities.GeoLocation()\n", " geoloc_entity.CountryName = host_hb[\"RemoteIPCountry\"]\n", " geoloc_entity.Longitude = host_hb[\"RemoteIPLongitude\"]\n", " geoloc_entity.Latitude = host_hb[\"RemoteIPLatitude\"]\n", " ip_entity.Location = geoloc_entity\n", " host_entity.IPAddress = ip_entity # TODO change to graph edge\n", " return host_entity\n", "\n", "def convert_to_ip_entities(ip_str):\n", " iplocation = GeoLiteLookup()\n", " ip_entities = []\n", " if ip_str:\n", " if \",\" in ip_str:\n", " addrs = ip_str.split(\",\")\n", " elif \" \" in ip_str:\n", " addrs = ip_str.split(\" \")\n", " else:\n", " addrs = [ip_str]\n", " for addr in addrs:\n", " ip_entity = entities.IpAddress()\n", " ip_entity.Address = addr.strip()\n", " iplocation.lookup_ip(ip_entity=ip_entity)\n", " ip_entities.append(ip_entity)\n", " return ip_entities\n", "\n", "# Add this information to our inv_host_entity\n", "def populate_host_aznet_ips(az_net_df, host_entity):\n", " retrieved_address = []\n", " if len(az_net_df) == 1:\n", " host_entity.private_ips = convert_to_ip_entities(\n", " az_net_df[\"PrivateIPAddresses\"].iloc[0]\n", " )\n", " host_entity.public_ips = convert_to_ip_entities(\n", " az_net_df[\"PublicIPAddresses\"].iloc[0]\n", " )\n", " retrieved_address = [ip.Address for ip in host_entity.public_ips]\n", " else:\n", " if \"private_ips\" not in host_entity:\n", " host_entity.private_ips = []\n", " if \"public_ips\" not in host_entity:\n", " host_entity.public_ips = []\n", "\n", "\n", "iplocation = GeoLiteLookup()\n", "\n", "# Try to get an OMS Heartbeat for this computer\n", "if \"Heartbeat\" in table_index:\n", " print(f\"Looking for {host_name} in OMS Heartbeat data...\")\n", " host_hb_df = qry_prov.Network.get_heartbeat_for_host(host_name=host_name)\n", " host_entity = populate_heartbeat_details(host_hb_df, host_entity)\n", "\n", "if \"AzureNetworkAnalytics_CL\" in table_index:\n", " print(f\"Looking for {host_name} IP addresses in network flows...\")\n", " az_net_df = qry_prov.Network.get_ips_for_host(host_name=host_name)\n", " populate_host_aznet_ips(az_net_df, host_entity)\n", "\n", "md(\"Host Details\", \"bold\")\n", "print(host_entity)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Contents](#Contents)\n", " # Related Alerts\n", " Look for any related alerts around this time." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:12.712441Z", "start_time": "2020-05-15T23:29:12.667444Z" } }, "outputs": [], "source": [ "ra_query_times = nbwidgets.QueryTime(\n", " units=\"day\",\n", " origin_time=query_times.origin_time,\n", " max_before=28,\n", " max_after=5,\n", " before=5,\n", " auto_display=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:15.961693Z", "start_time": "2020-05-15T23:29:14.578094Z" } }, "outputs": [], "source": [ "\n", "related_alerts = qry_prov.SecurityAlert.list_related_alerts(\n", " ra_query_times, host_name=host_entity.HostName\n", ")\n", "\n", "def print_related_alerts(alertDict, entityType, entityName):\n", " if len(alertDict) > 0:\n", " display(\n", " Markdown(\n", " f\"### Found {len(alertDict)} different alert types related to this {entityType} (`{entityName}`)\"\n", " )\n", " )\n", " for (k, v) in alertDict.items():\n", " print(f\"- {k}, # Alerts: {v}\")\n", " else:\n", " print(f\"No alerts for {entityType} entity `{entityName}`\")\n", "\n", "\n", "if isinstance(related_alerts, pd.DataFrame) and not related_alerts.empty:\n", " host_alert_items = (\n", " related_alerts[[\"AlertName\", \"TimeGenerated\"]]\n", " .groupby(\"AlertName\")\n", " .TimeGenerated.agg(\"count\")\n", " .to_dict()\n", " )\n", " print_related_alerts(host_alert_items, \"host\", host_entity.HostName)\n", " if len(host_alert_items) > 1:\n", " nbdisplay.display_timeline(\n", " data=related_alerts, title=\"Alerts\", source_columns=[\"AlertName\"], height=200\n", " )\n", "else:\n", " display(Markdown(\"No related alerts found.\"))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## Browse List of Related Alerts\n", " Select an Alert to view details" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:19.615661Z", "start_time": "2020-05-15T23:29:19.546661Z" } }, "outputs": [], "source": [ "def disp_full_alert(alert):\n", " global related_alert\n", " related_alert = SecurityAlert(alert)\n", " nbdisplay.display_alert(related_alert, show_entities=True)\n", "\n", "recenter_wgt = widgets.Checkbox(\n", " value=True,\n", " description='Center subsequent query times round selected Alert?',\n", " disabled=False,\n", " **WIDGET_DEFAULTS\n", ")\n", "if related_alerts is not None and not related_alerts.empty:\n", " related_alerts[\"CompromisedEntity\"] = related_alerts[\"Computer\"]\n", " display(Markdown(\"### Click on alert to view details.\"))\n", " display(recenter_wgt)\n", " rel_alert_select = nbwidgets.AlertSelector(\n", " alerts=related_alerts,\n", " action=disp_full_alert,\n", " )\n", " rel_alert_select.display()\n", " \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Contents](#toc)\n", " # Host Logons\n", " This section looks at successful and failed logons on the host." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:25.409590Z", "start_time": "2020-05-15T23:29:25.359585Z" } }, "outputs": [], "source": [ "# set the origin time to the time of our alert\n", "origin_time = (related_alert.TimeGenerated \n", " if recenter_wgt.value \n", " else query_times.origin_time)\n", "logon_query_times = nbwidgets.QueryTime(\n", " units=\"day\",\n", " origin_time=origin_time,\n", " before=5,\n", " after=1,\n", " max_before=20,\n", " max_after=20,\n", ")\n", "logon_query_times.display()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Successful Logons - Timeline and LogonType breakdown" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:29.150066Z", "start_time": "2020-05-15T23:29:27.337759Z" } }, "outputs": [], "source": [ "host_logons = qry_prov.WindowsSecurity.list_host_logons(\n", " logon_query_times, host_name=host_entity.HostName\n", ")\n", "\n", "if host_logons is not None and not host_logons.empty:\n", " display(Markdown(\"### Logon timeline.\"))\n", " tooltip_cols = [\n", " \"TargetUserName\",\n", " \"TargetDomainName\",\n", " \"SubjectUserName\",\n", " \"SubjectDomainName\",\n", " \"LogonType\",\n", " \"IpAddress\",\n", " ]\n", " nbdisplay.display_timeline(\n", " data=host_logons,\n", " group_by=\"TargetUserName\",\n", " source_columns=tooltip_cols,\n", " legend=\"right\", yaxis=True\n", " )\n", "\n", " display(Markdown(\"### Counts of logon events by logon type.\"))\n", " display(Markdown(\"Min counts for each logon type highlighted.\"))\n", " logon_by_type = (\n", " host_logons[[\"Account\", \"LogonType\", \"EventID\"]]\n", " .astype({'LogonType': 'int32'})\n", " .merge(right=pd.Series(data=nbdisplay._WIN_LOGON_TYPE_MAP, name=\"LogonTypeDesc\"),\n", " left_on=\"LogonType\", right_index=True)\n", " .drop(columns=\"LogonType\")\n", " .groupby([\"Account\", \"LogonTypeDesc\"])\n", " .count()\n", " .unstack()\n", " .rename(columns={\"EventID\": \"LogonCount\"})\n", " .fillna(0)\n", " .style\n", " .background_gradient(cmap=\"viridis\", low=0.5, high=0)\n", " .format(\"{0:0>3.0f}\")\n", " )\n", " display(logon_by_type)\n", "else:\n", " display(Markdown(\"No logon events found for host.\"))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " [Contents](#toc)\n", " ## Failed Logons" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:32.494051Z", "start_time": "2020-05-15T23:29:31.042487Z" } }, "outputs": [], "source": [ "failedLogons = qry_prov.WindowsSecurity.list_host_logon_failures(\n", " logon_query_times, host_name=host_entity.HostName\n", ")\n", "if failedLogons.empty:\n", " print(\"No logon failures recorded for this host between \",\n", " f\" {logon_query_times.start} and {logon_query_times.end}\"\n", " )\n", "else:\n", " nbdisplay.display_timeline(\n", " data=host_logons.query('TargetLogonId != \"0x3e7\"'),\n", " overlay_data=failedLogons,\n", " alert=related_alert,\n", " title=\"Logons (blue=user-success, green=failed)\",\n", " source_columns=tooltip_cols,\n", " height=200,\n", " )\n", " display(failedLogons\n", " .astype({'LogonType': 'int32'})\n", " .merge(right=pd.Series(data=nbdisplay._WIN_LOGON_TYPE_MAP, name=\"LogonTypeDesc\"),\n", " left_on=\"LogonType\", right_index=True)\n", " [['Account', 'EventID', 'TimeGenerated',\n", " 'Computer', 'SubjectUserName', 'SubjectDomainName',\n", " 'TargetUserName', 'TargetDomainName',\n", " 'LogonTypeDesc','IpAddress', 'WorkstationName'\n", " ]])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accounts With Failed And Successful Logons\n", "This query joins failed and successful logons for the same account name. Multiple logon failures followed by a sucessful logon might indicate attempts to guess or probe the user password." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:35.995808Z", "start_time": "2020-05-15T23:29:35.834809Z" } }, "outputs": [], "source": [ "if not failedLogons.empty:\n", " combined = pd.concat([failedLogons,\n", " host_logons[host_logons[\"TargetUserName\"]\n", " .isin(failedLogons[\"TargetUserName\"]\n", " .drop_duplicates())]])\n", " display(combined.head())\n", " combined[\"LogonStatus\"] = combined.apply(lambda x: \"Failed\" if x.EventID == 4625 else \"Success\", axis=1)\n", " nbdisplay.display_timeline(data=combined,\n", " group_by=\"LogonStatus\",\n", " source_columns=[\"TargetUserName\", \"LogonType\", \"SubjectUserName\", \"TargetLogonId\"],\n", " legend=\"inline\",\n", " yaxis=True,\n", " height=200)\n", " display(combined.sort_values(\"TimeGenerated\"))\n", "else:\n", " md(f\"No logon failures recorded for this host between {logon_query_times.start} and {logon_query_times.end}\") " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Contents](#Contents)\n", "# Other Security Events\n", " It's often useful to look at what other events were being logged\n", " at the time of the attack.\n", " \n", " We show events here grouped by Account. Things to look for are:\n", " \n", " - Unexpected events that change system security such as the addition of accounts or services\n", " - Event types that occur for only a single account - especially if there are a lot of event types only executed by a single account." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:47.340949Z", "start_time": "2020-05-15T23:29:38.308340Z" } }, "outputs": [], "source": [ "md(f\"Collecting Windows Event Logs for {host_entity.HostName}, this may take a few minutes...\")\n", "\n", "all_events_df = qry_prov.WindowsSecurity.list_host_events(\n", " logon_query_times,\n", " host_name=host_entity.HostName,\n", " add_query_items=\"| where EventID != 4688 and EventID != 4624\",\n", ")\n", "\n", "# Create a pivot of Event vs. Account\n", "win_events_acc = all_events_df[[\"Account\", \"Activity\", \"TimeGenerated\"]].copy()\n", "win_events_acc = win_events_acc.replace(\"-\\\\-\", \"No Account\").replace(\n", " {\"Account\": \"\"}, value=\"No Account\"\n", ")\n", "win_events_acc[\"Account\"] = win_events_acc.apply(lambda x: x.Account.split(\"\\\\\")[-1], axis=1)\n", "event_pivot = (\n", " pd.pivot_table(\n", " win_events_acc,\n", " values=\"TimeGenerated\",\n", " index=[\"Activity\"],\n", " columns=[\"Account\"],\n", " aggfunc=\"count\",\n", " )\n", " .fillna(0)\n", " .reset_index()\n", ")\n", "display(Markdown(\"Yellow highlights indicate account with highest event count\"))\n", "(\n", " event_pivot.style\n", " .applymap(lambda x: \"color: white\" if x == 0 else \"\")\n", " .applymap(\n", " lambda x: \"background-color: lightblue\"\n", " if not isinstance(x, str) and x > 0\n", " else \"\"\n", " )\n", " .set_properties(subset=[\"Activity\"], **{\"width\": \"400px\", \"text-align\": \"left\"})\n", " .highlight_max(axis=1)\n", " .hide_index()\n", ")\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parse Event Data for Selected Events\n", "For events that you want to look at in more detail you can parse out the full EventData field (containing all fields of the original event). The `parse_event_data` function below does that - transforming the EventData XML into a dictionary of property/value pairs). The `expand_event_properties` function takes this dictionary and transforms into columns in the output DataFrame.\n", "\n", "
\n", "<details>\n", "  More details...\n", "You can do this for multiple event types in a single pass but, dependng on the schema of each event you may end up with a lot of sparsely populated columns. E.g. suppose EventID 1 has EventData fields A, B and C and EventID 2 has fields A, D, E. If you parse both IDs you'll will end up with a DataFrame with columns A, B, C, D and E with contents populated only for the rows that with corresponding data.\n", "\n", "We recommend that you process batches of related event types (e.g. all user account change events) that have similar sets of fields to keep the output DataFrame manageable.\n", "</details>" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:29:58.968325Z", "start_time": "2020-05-15T23:29:58.080325Z" } }, "outputs": [], "source": [ "# Function to convert EventData XML into dictionary and\n", "# populate columns into DataFrame from previous query result\n", "import xml.etree.ElementTree as ET\n", "from xml.etree.ElementTree import ParseError\n", "\n", "SCHEMA = \"http://schemas.microsoft.com/win/2004/08/events/event\"\n", "\n", "\n", "def parse_event_data(row):\n", " try:\n", " xdoc = ET.fromstring(row.EventData)\n", " col_dict = {\n", " elem.attrib[\"Name\"]: elem.text for elem in xdoc.findall(f\"{{{SCHEMA}}}Data\")\n", " }\n", " reassigned = set()\n", " for k, v in col_dict.items():\n", " if k in row and not row[k]:\n", " row[k] = v\n", " reassigned.add(k)\n", " if reassigned:\n", " # print('Reassigned: ', ', '.join(reassigned))\n", " for k in reassigned:\n", " col_dict.pop(k)\n", " return col_dict\n", " except (ParseError, TypeError):\n", " return None\n", "\n", "\n", "# Parse event properties into a dictionary\n", "all_events_df[\"EventProperties\"] = all_events_df.apply(parse_event_data, axis=1)\n", "\n", "# For a specific event ID you can explode the EventProperties values into their own columns\n", "# using this function. You can do this for the whole data set but it will result\n", "# in a lot of sparse columns in the output data frame\n", "def expand_event_properties(input_df):\n", " exp_df = input_df.apply(lambda x: pd.Series(x.EventProperties), axis=1)\n", " return (\n", " exp_df.drop(set(input_df.columns).intersection(exp_df.columns), axis=1)\n", " .merge(\n", " input_df.drop(\"EventProperties\", axis=1),\n", " how=\"inner\",\n", " left_index=True,\n", " right_index=True,\n", " )\n", " .replace(\"\", np.nan) # these 3 lines get rid of blank columns\n", " .dropna(axis=1, how=\"all\")\n", " .fillna(\"\")\n", " )\n", "\n", "\n", "expand_event_properties(all_events_df[all_events_df[\"EventID\"] == 4724]).head(2)\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Account Change Events - Timeline\n", "Here we want to focus on a some specific subcategories of events. Attackers commonly try to add or change user accounts and group memberships. We also include events related to addition or change of scheduled tasks and Windows services. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:30:02.077311Z", "start_time": "2020-05-15T23:30:01.842315Z" } }, "outputs": [], "source": [ "# Get a full list of Windows Security Events\n", "import pkgutil\n", "import os\n", "w_evt = pkgutil.get_data(\"msticpy\", f\"resources{os.sep}WinSecurityEvent.json\")\n", "win_event_df = pd.read_json(w_evt)\n", "\n", "# Create criteria for events that we're interested in\n", "acct_sel = win_event_df[\"subcategory\"] == \"User Account Management\"\n", "group_sel = win_event_df[\"subcategory\"] == \"Security Group Management\"\n", "schtask_sel = (win_event_df[\"subcategory\"] == \"Other Object Access Events\") & (\n", " win_event_df[\"description\"].str.contains(\"scheduled task\")\n", ")\n", "\n", "event_list = win_event_df[acct_sel | group_sel | schtask_sel][\"event_id\"].to_list()\n", "# Add Service install event\n", "event_list.append(7045)\n", "\n", "# Plot events on a timeline\n", "p = nbdisplay.display_timeline(\n", " data=all_events_df[all_events_df[\"EventID\"].isin(event_list)],\n", " group_by=\"EventID\",\n", " source_columns=[\"Activity\", \"Account\"],\n", " legend=\"right\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Show Details of Selected Events\n", "From the above data - pick which event types you want to view (by default, all are selected).\n", "The second cell will display the event types selected." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2019-11-01T19:38:42.155265Z", "start_time": "2019-11-01T19:38:42.104295Z" } }, "outputs": [], "source": [ "# populate actual events IDs to select from\n", "recorded_events = (all_events_df['EventID']\n", " [all_events_df[\"EventID\"]\n", " .isin(event_list)].drop_duplicates().values)\n", "event_subset = win_event_df[win_event_df[\"event_id\"].isin(event_list)\n", " & win_event_df[\"event_id\"].isin(recorded_events)]\n", "items = list(event_subset.apply(lambda x: (x.full_desc, x.event_id), axis=1).values)\n", "ss = nbwidgets.SelectSubset(\n", " items,\n", " default_selected=items\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2019-11-01T19:38:48.430726Z", "start_time": "2019-11-01T19:38:48.412764Z" } }, "outputs": [], "source": [ "col_names = ['TimeGenerated', 'Account', 'AccountType',\n", " 'Computer', 'EventID', 'Activity', 'SubjectAccount',\n", " 'SubjectDomainName', 'SubjectLogonId', 'SubjectUserName',\n", " 'TargetAccount', 'TargetDomainName', 'TargetSid', 'TargetUserName']\n", "display(all_events_df[all_events_df[\"EventID\"].isin(ss.selected_values)]\n", " [col_names]\n", " .replace(to_replace=\"\", value=np.NAN)\n", " .dropna(axis=1, how=\"all\"))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Contents](#Contents)\n", "# Examine Logon Sessions\n", "Looking at characteristics and activity of individual logon sessions is an effective way of spottting clusters of attacker activity.\n", "\n", "The biggest problem is deciding which logon sessions are the ones to look at. We may already have some indicators of sessions that we want to examine from earlier sections:\n", "\n", "- Accounts that experienced a series of failed logons followed by successful logons [see](#Accounts With Failed And Successful Logons)\n", "- Accounts that triggered unexpected events [see](#Show-Timeline-of-Account-Change-Events)\n", "\n", "In this section we use clustering to collapse repetive logons and show details of the distinct logon patterns\n", "\n", " ## Browse logon account details" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:30:15.514844Z", "start_time": "2020-05-15T23:30:14.697818Z" } }, "outputs": [], "source": [ "from msticpy.sectools.eventcluster import (\n", " dbcluster_events,\n", " add_process_features,\n", " _string_score,\n", ")\n", "\n", "if host_logons is None or host_logons.empty:\n", " display(Markdown(\"No host logons recorded. This section cannot be run.\"))\n", " raise ValueError(\"aborted\")\n", "\n", "# Set up clustering features and run DBScan clustering\n", "logon_features = host_logons.copy()\n", "logon_features[\"AccountNum\"] = host_logons.apply(\n", " lambda x: _string_score(x.Account), axis=1\n", ")\n", "logon_features[\"TargetUserNum\"] = host_logons.apply(\n", " lambda x: _string_score(x.TargetUserName), axis=1\n", ")\n", "logon_features[\"LogonHour\"] = host_logons.apply(lambda x: x.TimeGenerated.hour, axis=1)\n", "\n", "# you might need to play around with the max_cluster_distance parameter.\n", "# decreasing this gives more clusters.\n", "(clus_logons, _, _) = dbcluster_events(\n", " data=logon_features,\n", " time_column=\"TimeGenerated\",\n", " cluster_columns=[\"AccountNum\", \"LogonType\", \"TargetUserNum\"],\n", " max_cluster_distance=0.0001,\n", ")\n", "display(Markdown(f\"Number of input events: {len(host_logons)}\"))\n", "display(Markdown(f\"Number of clustered events: {len(clus_logons)}\"))\n", "\n", "display(Markdown(\"### Relative frequencies by account pattern\"))\n", "plt.rcParams[\"figure.figsize\"] = (12, 4)\n", "clus_logons.sort_values(\"Account\").plot.barh(x=\"Account\", y=\"ClusterSize\");\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## View distinct host logon patterns" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:30:17.480349Z", "start_time": "2020-05-15T23:30:17.426340Z" } }, "outputs": [], "source": [ "import re\n", "\n", "# Build a list of distinct logon patterns from the clustered data\n", "dist_logons = clus_logons.sort_values(\"TimeGenerated\")[\n", " [\"TargetUserName\", \"TimeGenerated\", \"LastEventTime\", \"LogonType\", \"ClusterSize\"]\n", "]\n", "dist_logons = dist_logons.apply(\n", " lambda x: (\n", " f\"{x.TargetUserName}: \"\n", " f\"(logontype {x.LogonType}) \"\n", " f\"timerange: {x.TimeGenerated} - {x.LastEventTime} \"\n", " f\"count: {x.ClusterSize}\"\n", " ),\n", " axis=1,\n", ")\n", "# Convert to dict, flipping keys/values\n", "dist_logons = {v: k for k, v in dist_logons.to_dict().items()}\n", "\n", "\n", "def get_selected_logon_cluster(selected_item):\n", " acct = clus_logons.loc[selected_item][\"TargetUserName\"]\n", " logon_type = clus_logons.loc[selected_item][\"LogonType\"]\n", " return host_logons.query(\"TargetUserName == @acct and LogonType == @logon_type\")\n", "\n", "\n", "# Create an Output widget to show the Logon Details\n", "w_output = widgets.Output(layout={\"border\": \"1px solid black\"})\n", "\n", "\n", "def show_logon(idx):\n", " w_output.clear_output()\n", " with w_output:\n", " nbdisplay.display_logon_data(pd.DataFrame(clus_logons.loc[idx]).T)\n", "\n", "\n", "logon_wgt = nbwidgets.SelectString(\n", " description=\"Select logon cluster to examine\",\n", " item_dict=dist_logons,\n", " action=show_logon,\n", " height=\"200px\",\n", " width=\"100%\",\n", " auto_display=True,\n", ")\n", "display(w_output)\n", "# Display the first item on first view\n", "top_item = next(iter(dist_logons.values()))\n", "with w_output:\n", " nbdisplay.display_logon_data(pd.DataFrame(clus_logons.loc[top_item]).T)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ## Analyze Processes Patterns for logon sessions\n", " \n", "In this section we look at the types of processes run in each logon session. For each process (and process characteristics such as command line structure) we measure its rarity compared to other processes on the same host. We then calculate the mean rarity of all processes in a logon session and display the results ordered by rarity. One is the highest possible score and would indicate all processes in the session have a unique execution pattern.\n", " \n", "Note: The next section retrieves processes for time period around the logons for the user ID selected in the previous session. If you want to view a broader time boundary please adjust the query time boundaries in below." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:30:31.264572Z", "start_time": "2020-05-15T23:30:31.213572Z" } }, "outputs": [], "source": [ "# set the origin time to start at the first logon in our set\n", "# end end 2hrs after the last\n", "start_time = host_logons[\"TimeGenerated\"].min()\n", "end_time = host_logons[\"TimeGenerated\"].max()\n", "time_diff = int((end_time - start_time).total_seconds() / (60 * 60) + 2)\n", "proc_query_times = nbwidgets.QueryTime(\n", " units=\"hours\",\n", " origin_time=start_time,\n", " before=1,\n", " after=time_diff + 1,\n", " max_before=20,\n", " max_after=time_diff + 20,\n", ")\n", "proc_query_times.display()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Compute the relative rarity of processes in each session\n", " This should be a good guide to which sessions are the more interesting to look at.\n", " \n", " **Note** Clustering lots (1000s) of events will take a little time to compute." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:30:56.847458Z", "start_time": "2020-05-15T23:30:36.409635Z" } }, "outputs": [], "source": [ "from msticpy.sectools.eventcluster import dbcluster_events, add_process_features\n", "from collections import Counter\n", "\n", "print(\"Getting process events...\", end=\"\")\n", "processes_on_host = qry_prov.WindowsSecurity.list_host_processes(\n", " proc_query_times, host_name=host_entity.HostName\n", ")\n", "print(f\"done. {len(processes_on_host)} events\")\n", "print(\"Clustering. Please wait...\", end=\"\")\n", "feature_procs = add_process_features(input_frame=processes_on_host, path_separator=\"\\\\\")\n", "\n", "feature_procs[\"accountNum\"] = feature_procs.apply(\n", " lambda x: _string_score(x.Account), axis=1\n", ")\n", "# you might need to play around with the max_cluster_distance parameter.\n", "# decreasing this gives more clusters.\n", "(clus_events, dbcluster, x_data) = dbcluster_events(\n", " data=feature_procs,\n", " cluster_columns=[\n", " \"commandlineTokensFull\",\n", " \"pathScore\",\n", " \"accountNum\",\n", " \"isSystemSession\",\n", " ],\n", " max_cluster_distance=0.0001,\n", ")\n", "print(\"done\")\n", "print(\"Number of input events:\", len(feature_procs))\n", "print(\"Number of clustered events:\", len(clus_events))\n", "\n", "# Join the clustered results back to the original process frame\n", "procs_with_cluster = feature_procs.merge(\n", " clus_events[\n", " [\n", " \"commandlineTokensFull\",\n", " \"accountNum\",\n", " \"pathScore\",\n", " \"isSystemSession\",\n", " \"ClusterSize\",\n", " ]\n", " ],\n", " on=[\"commandlineTokensFull\", \"accountNum\", \"pathScore\", \"isSystemSession\"],\n", ")\n", "# Rarity = inverse of cluster size\n", "procs_with_cluster[\"Rarity\"] = 1 / procs_with_cluster[\"ClusterSize\"]\n", "# count the number of processes for each logon ID\n", "lgn_proc_count = (\n", " pd.concat(\n", " [\n", " processes_on_host.groupby(\"TargetLogonId\")[\"TargetLogonId\"].count(),\n", " processes_on_host.groupby(\"SubjectLogonId\")[\"SubjectLogonId\"].count(),\n", " ]\n", " ).sum(level=0)\n", ").to_dict()\n", "\n", "# Display the results\n", "md(\"Sessions ordered by process rarity\", 'bold')\n", "md(\"Higher score indicates higher number of unusual processes\")\n", "process_rarity = (procs_with_cluster.groupby([\"SubjectUserName\", \"SubjectLogonId\"])\n", " .agg({\"Rarity\": \"mean\", \"TimeGenerated\": \"count\"})\n", " .rename(columns={\"TimeGenerated\": \"ProcessCount\"})\n", " .reset_index())\n", "display(\n", " process_rarity\n", " .sort_values(\"Rarity\", ascending=False)\n", " .style.bar(subset=[\"Rarity\"], color=\"#d65f5f\")\n", ")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Overview of session timelines for sessions with higher rarity score" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:31:02.979872Z", "start_time": "2020-05-15T23:31:02.675871Z" } }, "outputs": [], "source": [ "# Display process timeline for 75% percentile rarest scores\n", "rare_sess = process_rarity[process_rarity[\"Rarity\"]\n", " > process_rarity[\"Rarity\"].quantile(.25)]\n", "rare_sessions = processes_on_host[(processes_on_host[\"SubjectLogonId\"]\n", " .isin(rare_sess[\"SubjectLogonId\"]))\n", " & (processes_on_host[\"SubjectUserName\"]\n", " .isin(rare_sess[\"SubjectUserName\"]))]\n", "\n", "md(\"Timeline of sessions with higher process rarity\", \"large\")\n", "md(\"Multiple sessions (y-axis) may be are shown for each account.\")\n", "md(\"You will likely need to zoom in to see the individual session processes.\")\n", "\n", "nbdisplay.display_timeline(\n", " data=rare_sessions,\n", " group_by=\"SubjectLogonId\",\n", " source_columns=[\"SubjectUserName\", \"SubjectLogonId\", \"NewProcessName\", \"CommandLine\"],\n", " legend=\"right\",\n", " yaxis=True\n", ");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### View the processes for these Sessions" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2020-05-15T23:31:09.344452Z", "start_time": "2020-05-15T23:31:09.280062Z" } }, "outputs": [], "source": [ "def view_logon_sess(logon_id=\"\"):\n", " global selected_logon\n", " selected_logon = host_logons[host_logons[\"TargetLogonId\"] == logon_id]\n", "\n", " if all_procs.value:\n", " sess_procs = processes_on_host.query(\n", " \"TargetLogonId == @logon_id | SubjectLogonId == @logon_id\"\n", " )\n", " else:\n", " sess_procs = procs_with_cluster.query(\"SubjectLogonId == @logon_id\")[\n", " [\"NewProcessName\", \"CommandLine\", \"SubjectLogonId\", \"ClusterSize\"]\n", " ].drop_duplicates()\n", " display(sess_procs)\n", "\n", "sessions = list(process_rarity\n", " .sort_values(\"Rarity\", ascending=False)\n", " .apply(lambda x: (f\"{x.SubjectLogonId} {x.SubjectUserName} Rarity={x.Rarity}\",\n", " x.SubjectLogonId), \n", " axis=1))\n", "all_procs = widgets.Checkbox(\n", " value=False,\n", " description=\"View All Processes (Show clustered only if not checked)\",\n", " **WIDGET_DEFAULTS,\n", ")\n", "display(all_procs)\n", "logon_wgt = nbwidgets.SelectString(\n", " description=\"Select logon session to examine\",\n", " item_dict={label: val for label, val in sessions},\n", " height=\"300px\",\n", " width=\"100%\",\n", " auto_display=True,\n", " action=view_logon_sess,\n", ")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Browse All Sessions (Optional)\n", "\n", "**If the previous section did not reveal anything interesting you can opt to browse all logon sessions.**\n", "\n", "**Otherwise, skip to the [Check Commandline for IoCs section](#Check-for-IOCs-in-Commandline-for-selected-session)**\n", "\n", "To do this you need to first pick an account + logon type (in the following cell) then pick a particular session that you want to view in the subsequent cell. Use the rarity score from the previous graph to guide you.\n", "\n", " ### Step 1 - Select a logon ID and Type" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2019-10-19T01:18:41.70951Z", "start_time": "2019-10-19T01:18:41.686523Z" } }, "outputs": [], "source": [ "logon_wgt2 = nbwidgets.SelectString(\n", " description=\"Select logon cluster to examine\",\n", " item_dict=dist_logons,\n", " height=\"200px\",\n", " width=\"100%\",\n", " auto_display=True,\n", ")\n", "all_procs = widgets.Checkbox(\n", " value=False,\n", " description=\"View All Processes (Clustered only if not checked)\",\n", " **WIDGET_DEFAULTS,\n", ")\n", "display(all_procs)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " ### Step 2 - Pick a logon session to view its processes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "ExecuteTime": { "end_time": "2019-10-19T01:18:55.502417Z", "start_time": "2019-10-19T01:18:55.450467Z" } }, "outputs": [], "source": [ "selected_logon_cluster = get_selected_logon_cluster(logon_wgt2.value)\n", "\n", "selected_tgt_logon = selected_logon_cluster[\"TargetUserName\"].iat[0]\n", "system_logon = selected_tgt_logon.lower() == \"system\" or selected_tgt_logon.endswith(\n", " \"$\"\n", ")\n", "\n", "if system_logon:\n", " display(\n", " Markdown(\n", " '

Warning: the selected '\n", " \"account name appears to be a system account.


\"\n", " \"It is difficult to accurately associate processes \"\n", " \"with the specific logon sessions.
\"\n", " \"Showing clustered events for entire time selection.\"\n", " )\n", " )\n", " display(\n", " clus_events.sort_values(\"TimeGenerated\")[\n", " [\n", " \"TimeGenerated\",\n", " \"LastEventTime\",\n", " \"NewProcessName\",\n", " \"CommandLine\",\n", " \"ClusterSize\",\n", " \"commandlineTokensFull\",\n", " \"pathScore\",\n", " \"isSystemSession\",\n", " ]\n", " ]\n", " )\n", "\n", "# Display a pick list for logon instances\n", "sel_1 = host_logons[\"TargetLogonId\"].isin(lgn_proc_count)\n", "sel_2 = host_logons[\"TargetUserName\"] == selected_tgt_logon\n", "items = (\n", " host_logons[sel_1 & sel_2]\n", " .sort_values(\"TimeGenerated\")\n", " .apply(\n", " lambda x: (\n", " f\"{x.TargetUserName}: \"\n", " f\"(logontype={x.LogonType}) \"\n", " f\"(timestamp={x.TimeGenerated}) \"\n", " f\"logonid={x.TargetLogonId}\"\n", " ),\n", " axis=1,\n", " )\n", " .values.tolist()\n", ")\n", "if not items:\n", " items = [\"No processes for logon\"]\n", "sess_w = widgets.Select(\n", " options=items, description=\"Select logon instance to examine\", **WIDGET_DEFAULTS\n", ")\n", "\n", "import re\n", "\n", "logon_list_regex = r\"\"\"\n", "(?P[^:]+):\\s+\n", "\\(logontype=(?P[^)]+)\\)\\s+\n", "\\(timestamp=(?P