{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# MSTICpy - Mordor data provider and browser\n", "\n", "### Description\n", "This notebook provides a guided example of using the Mordor data provider and browser included with MSTICpy.\n", "\n", "For more information on the Mordor data sets see the [Open Threat Research Forge Mordor GitHub repo](https://github.com/OTRF/mordor)\n", "\n", "You must have msticpy installed to run this notebook:\n", "```\n", "%pip install --upgrade msticpy\n", "```\n", "\n", "MSTICpy versions >= 0.8.5\n", "\n", "### Contents:\n", "- Using the Mordor data provider to retrieve data sets\n", " - Listing queries\n", " - Running a query to retrieve data\n", " - Optional parameters\n", " - Searching for queries by Mordor property\n", "- Mordor Browser\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the Data Provider to download datasets\n", "\n", "Using the data provider you can download and render event data as a pandas DataFrame.\n", "\n", "> **Note** - Mordor includes both host event data and network capture data.
\n", "> Although Capture files can be downloaded and unpacked
\n", "> they currently cannot be populated into a pandas DataFrame.\n", "> This is the case for most `network` datasets.
\n", "> `Host` event data is retrieved and populated into DataFrames.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieving Mitre data...\n", "Retrieving Mordor data...\n" ] } ], "source": [ "from msticpy.data import QueryProvider\n", "\n", "CACHE_FOLDER = \"~/.msticpy/mordor\"\n", "mdr_data = QueryProvider(\"Mordor\", save_folder=CACHE_FOLDER)\n", "mdr_data.connect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### List Queries\n", "\n", "> Note: Many Mordor data entries have multiple data sets, so we see more queries than Mordor entries.\n", "\n", "(Only first 15 shown)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['atomic.aws.collection.ec2_proxy_s3_exfiltration',\n", " 'atomic.linux.defense_evasion.host.sh_binary_padding_dd',\n", " 'atomic.linux.discovery.host.sh_arp_cache',\n", " 'atomic.linux.initial_access.network.log4jshell_reversheshell_netcat',\n", " 'atomic.windows.collection.host.msf_record_mic',\n", " 'atomic.windows.credential_access.host.cmd_lsass_memory_dumpert_syscalls',\n", " 'atomic.windows.credential_access.host.cmd_psexec_lsa_secrets_dump',\n", " 'atomic.windows.credential_access.host.cmd_sam_copy_esentutl',\n", " 'atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges',\n", " 'atomic.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_backupkeys_dcerpc_smb_lsarpc',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_extract_keys',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_logonpasswords',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_lsadump_patch',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_sam_access']" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.list_queries()[:15]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Retrieving/querying a data set" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/atomic/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip\n", "Extracting covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges_2020-08-05020926.json\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
@versionKeywordsThreadIDVersionDestAddresshostLayerRTIDMessageSourceModuleNameSourceName...PropertiesOperationTypeQueryNameQueryResultsQueryStatusPipeNameDisabledPrivilegeListEnabledPrivilegeListShareLocalPathRelativeTargetName
01-921436483760003481648881239.255.255.250wec.internal.cloudapp.net44.0The Windows Filtering Platform has permitted a...eventlogMicrosoft-Windows-Security-Auditing...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
11-922337203685477580844522NaNwec.internal.cloudapp.netNaNFile created:\\r\\nRuleName: -\\r\\nUtcTime: 2020-...eventlogMicrosoft-Windows-Sysmon...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
21-922337203685477580844522NaNwec.internal.cloudapp.netNaNRawAccessRead detected:\\r\\nRuleName: -\\r\\nUtcT...eventlogMicrosoft-Windows-Sysmon...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

3 rows × 145 columns

\n", "
" ], "text/plain": [ " @version Keywords ThreadID Version DestAddress \\\n", "0 1 -9214364837600034816 4888 1 239.255.255.250 \n", "1 1 -9223372036854775808 4452 2 NaN \n", "2 1 -9223372036854775808 4452 2 NaN \n", "\n", " host LayerRTID \\\n", "0 wec.internal.cloudapp.net 44.0 \n", "1 wec.internal.cloudapp.net NaN \n", "2 wec.internal.cloudapp.net NaN \n", "\n", " Message SourceModuleName \\\n", "0 The Windows Filtering Platform has permitted a... eventlog \n", "1 File created:\\r\\nRuleName: -\\r\\nUtcTime: 2020-... eventlog \n", "2 RawAccessRead detected:\\r\\nRuleName: -\\r\\nUtcT... eventlog \n", "\n", " SourceName ... Properties OperationType \\\n", "0 Microsoft-Windows-Security-Auditing ... NaN NaN \n", "1 Microsoft-Windows-Sysmon ... NaN NaN \n", "2 Microsoft-Windows-Sysmon ... NaN NaN \n", "\n", " QueryName QueryResults QueryStatus PipeName DisabledPrivilegeList \\\n", "0 NaN NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN NaN \n", "2 NaN NaN NaN NaN NaN \n", "\n", " EnabledPrivilegeList ShareLocalPath RelativeTargetName \n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "\n", "[3 rows x 145 columns]" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges().head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Optional parameters\n", "\n", "The data provider and the query functions support some parameters to control\n", "aspects of the query operation.\n", "\n", "- **use_cached** : bool, optional
\n", " Try to use locally saved file first,\n", " by default True. If you’ve previously downloaded a file, it will use\n", " this rather than downloading a new copy.\n", "- **save_folder** : str, optional
\n", " Path to output folder, by default\n", " \".\". The path that downloaded and extracted files are saved to.\n", "- **silent** : bool
\n", " If True, suppress feedback. By default, False.\n", "\n", "If you specify these when you initialize the data provider, the settings\n", "will apply to all queries." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieving Mitre data...\n", "Retrieving Mordor data...\n" ] } ], "source": [ "mdr_data = QueryProvider(\"Mordor\", save_folder=\"./mordor\")\n", "mdr_data.connect()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using these parameters in the query will override the provider settings\n", "and defaults for that query." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
@versionKeywordsThreadIDVersionDestAddresshostLayerRTIDMessageSourceModuleNameSourceName...PropertiesOperationTypeQueryNameQueryResultsQueryStatusPipeNameDisabledPrivilegeListEnabledPrivilegeListShareLocalPathRelativeTargetName
01-921436483760003481648881239.255.255.250wec.internal.cloudapp.net44.0The Windows Filtering Platform has permitted a...eventlogMicrosoft-Windows-Security-Auditing...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
11-922337203685477580844522NaNwec.internal.cloudapp.netNaNFile created:\\r\\nRuleName: -\\r\\nUtcTime: 2020-...eventlogMicrosoft-Windows-Sysmon...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
\n", "

2 rows × 145 columns

\n", "
" ], "text/plain": [ " @version Keywords ThreadID Version DestAddress \\\n", "0 1 -9214364837600034816 4888 1 239.255.255.250 \n", "1 1 -9223372036854775808 4452 2 NaN \n", "\n", " host LayerRTID \\\n", "0 wec.internal.cloudapp.net 44.0 \n", "1 wec.internal.cloudapp.net NaN \n", "\n", " Message SourceModuleName \\\n", "0 The Windows Filtering Platform has permitted a... eventlog \n", "1 File created:\\r\\nRuleName: -\\r\\nUtcTime: 2020-... eventlog \n", "\n", " SourceName ... Properties OperationType \\\n", "0 Microsoft-Windows-Security-Auditing ... NaN NaN \n", "1 Microsoft-Windows-Sysmon ... NaN NaN \n", "\n", " QueryName QueryResults QueryStatus PipeName DisabledPrivilegeList \\\n", "0 NaN NaN NaN NaN NaN \n", "1 NaN NaN NaN NaN NaN \n", "\n", " EnabledPrivilegeList ShareLocalPath RelativeTargetName \n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "\n", "[2 rows x 145 columns]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges(silent=True, save_folder=\"./mordor\").head(2)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Volume in drive E has no label.\n", " Volume Serial Number is 7E50-19F7\n", "\n", " Directory of e:\\src\\microsoft\\msticpy\\docs\\notebooks\\mordor\n", "\n", "01/06/2022 05:15 PM .\n", "01/06/2022 05:15 PM ..\n", "01/06/2022 05:15 PM 76,924 atomic-windows-credential_access-host-covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip\n", "01/06/2022 05:15 PM 1,651,230 covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges_2020-08-05020926.json\n", " 2 File(s) 1,728,154 bytes\n", " 2 Dir(s) 227,291,512,832 bytes free\n" ] } ], "source": [ "!dir mordor" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting summary data about a query\n", "\n", "Call the query function with a single \"?\" parameter." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query: covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges\n", "Data source: Mordor\n", "Covenant DCSync\n", "\n", "Notes\n", "-----\n", "Mordor ID: SDWIN-200805020926\n", "This dataset represents adversaries abusing Active Directory Replication services to retrieve secret domain data (i.e. NTLM hashes) from domain accounts.\n", "\n", "Mitre Techniques: T1003: OS Credential Dumping\n", "Mitre Tactics: TA0006: Credential Access\n", "\n", "Parameters\n", "----------\n", "Query:\n", "https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/atomic/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip\n" ] } ], "source": [ "mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges(\"?\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Searching for Queries with QueryProvider.search_queries()\n", "Search queries for matching attributes.\n", "\n", "#### Parameters\n", "\n", "**search** : str Search string. \n", "\n", "Substrings separated by commas will be treated as OR terms - e.g. \"a, b\" == \"a\" or \"b\".
\n", "Substrings separated by \"+\" will be treated as AND terms - e.g. \"a + b\" == \"a\" and \"b\"\n", "\n", "#### Returns\n", "List of matching query names." ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['atomic.aws.collection.ec2_proxy_s3_exfiltration (AWS Cloud Bank Breach S3)']" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.search_queries(\"AWS\")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['atomic.windows.defense_evasion.host.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)',\n", " 'atomic.windows.defense_evasion.network.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)']" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.search_queries(\"Empire + T1222\")" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['atomic.windows.credential_access.host.empire_mimikatz_sam_access (Empire Mimikatz SAM Extract Hashes)',\n", " 'atomic.windows.defense_evasion.host.empire_wdigest_downgrade.tar (Empire WDigest Downgrade)',\n", " 'atomic.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)',\n", " 'atomic.windows.credential_access.network.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_lsadump_patch (Empire Mimikatz Lsadump LSA Patch)',\n", " 'atomic.windows.credential_access.host.empire_mimikatz_logonpasswords (Empire Mimikatz LogonPasswords)']" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdr_data.search_queries(\"Empire + Credential\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mordor Browser\n", "\n", "We've also built a more specialized browser for Mordor data. This uses the metadata in the repository to let you view full details of the dataset.\n", "\n", "You can also preview the dataset (if it is convertible to a DataFrame).\n", "\n", "For details of the data shown please see the [Mordor GitHub repo](https://github.com/OTRF/mordor)
and the [Threat Hunter Playbook](https://threathunterplaybook.com/introduction.html)\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieving Mitre data...\n", "Retrieving Mordor data...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "06471b702a984bc9a3c519a103e93174", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(VBox(children=(HTML(value='

Mordor dataset browser

'), Select(description='Data sets', l…" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from msticpy.vis.mordor_browser import MordorBrowser\n", "\n", "mdr_browser = MordorBrowser()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Mordor Browser Details\n", "The top scrollable list is a list of the Mordor datasets. Selecting one of these updates the data in the lower half of the browser.\n", "\n", "#### Filter Drop-down\n", "To narrow your search you can filter using a text search or filter by Mitre Attack Techniques or Tactics.\n", "- The Filter text box uses the same syntax as the provider `search_queries()` function.\n", " - Simple text string will find matches for datasets that contain this string\n", " - Strings separated by \",\" are treated as OR terms - i.e. it will match items that contain ANY of the substrings\n", " - Strings separated by \"+\" are treated as AND terms - i.e. it will match items that contain ALL of the substrings\n", "- The Mitre Techniques and Tactics lists are multi-select lists. Only items that have techniques and tactics matching\n", " the selected items will be show.\n", "- Reset Filter button will clear any filtering.\n", "\n", "#### Main Details Window\n", "- title, ID, author, creation date, modification date and description are self-explanatory.\n", "- tags can be used for searching\n", "- file_paths (see below)\n", "- attacks - lists related Mitre Technique and Tactics. The item title is a link to the Mitre page describing the technique or tactic.\n", "- notebooks - if there is a notebook in the Threat Hunter Playbook site, a link to it is shown here. (multiple notebooks might be shown)\n", "- simulation - raw data listing the steps in the attack (and useful for replaying the attack in a demo environment).\n", "- references - links to any external data about the attack.\n", "\n", "#### File_paths\n", "This section allows you to select, download and (in most cases) display the event data relating to the attack.\n", "\n", "Select a file and click on the Download button.\n", "\n", "The zipped file is downloaded and extracted. If it is event data, this is converted to a\n", "pandas DataFrame and displayed below the rest of the data.\n", "\n", "The current dataset is available as an attribute of the browser:\n", "```\n", " mdr_browser.current_dataset\n", "```\n", "\n", "Datasets that you've downloaded and displayed in this session are also cached in the browser and available in the \n", "`mdr_browser.datasets` attribute.\n", "\n", "#### Downloaded files\n", "By default files are downloaded and extracted to the current folder. You can change this with the\n", "`save_folder` parameter when creating the `MordorBrowser` object.\n", "\n", "You can also specify the `use_cached` parameter. By default, this is `True`, which causes downloaded files not\n", "to be deleted after extraction. These local copies are used if you try to view the same data set again.\n", "This also works across sessions.\n", "\n", "If `use_cache` is set to False, files are deleted immediately after downloading, extracting and populating the\n", "DataFrame." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using the standard query browser\n", "\n", "> **Note** - In the `Example` section, ignore the examples of parameters
\n", "> passed to the query - these are not needed and ignored." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "cb323a2e318048f398fdad41a831eeb2", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "

AWS Cloud Bank Breach S3

Notes
Mordor ID: SDAWS-200914011940
This dataset represents adversaries abusing a misconfigured EC2 reverse proxy to obtain instance profile keys and eventually exfiltrate files from an S3 bucket.
Mitre Techniques: T1078: Valid Accounts, T1530: Data from Cloud Storage Object
Mitre Tactics: TA0001: Initial Access, TA0003: Persistence, TA0004: Privilege Escalation, TA0005: Defense Evasion, TA0009: Collection

Parameters


Query

https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datase\n",
                            "  ts/atomic/aws/collection/ec2_proxy_s3_exfiltration.zip

\n", "

Example

\n", "

{QueryProvider}[.QueryPath].QueryName(params...)

\n", " 
qry_prov.atomic.aws.collection.ec2_proxy_s3_exfiltration(start=start, end=end, hostname=host)
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "mdr_data.browse_queries()" ] } ], "metadata": { "interpreter": { "hash": "b736adfe05d9ae282eea4c01a733d58a0215ef3399d39339e6557e4c515b0f48" }, "kernelspec": { "display_name": "Python 3.7.10 64-bit (conda)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }