{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Splunk - Data Connector" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Description\n", "The data provider module of msticpy provides functions to allow for the defining of data sources, connectors to them and queries for them as well as the ability to return query result from the defined data sources. \n", "\n", "For more information on Data Propviders, check documentation\n", "- Data Provider: https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html\n", "\n", "In this notebooks we will demonstrate Splunk data connector feature of msticpy. \n", "This feature is built on-top of the [Splunk Enterprise SDK for Python] (https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/) with some customizations and enhancements." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:14:41.244954Z", "start_time": "2020-08-07T19:14:41.241519Z" } }, "outputs": [], "source": [ "# Only run first time to install/upgrade msticpy to latest version\n", "#!pip install --upgrade msticpy[splunk]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Authentication\n", "\n", "Authentication for the Splunk data provider is handled by specifying credentials directly in the connect call or specifying the credentials in msticpy config file.\n", "\n", "For more information on how to create new user with approapriate roles and permissions, follow Splunk Docs [Addandeditusers](https://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Addandeditusers) and [Aboutusersandroles](https://docs.splunk.com/Documentation/Splunk/8.0.5/Security/Aboutusersandroles). The user should have permission to at least run its own searches or more depending upon the actions to be performed by user.\n", "\n", "Once you created user account with the appropriate roles, you will require the following details to specify while connecting\n", "- host = \"localhost\"(Splunk server FQDN hostname to connect, for locally installed splunk, you can specify localhost)\n", "- port = 8089 (Splunk REST API )\n", "- username = \"admin\" (username to connect to Splunk instance)\n", "- password = \"yourpassword\" (password of the userspecified in username)\n", "\n", "Once you have details, you can specify it in `msticpyconfig.yaml` as shown in below example" ] }, { "cell_type": "markdown", "metadata": { "ExecuteTime": { "end_time": "2020-08-07T17:50:18.361039Z", "start_time": "2020-08-07T17:50:18.349006Z" } }, "source": [ "```\n", "SplunkApp:\n", " Args:\n", " host: \"{Splunk server FQDN or localhost}\"\n", " port: \"{default 8089}\"\n", " username: \"{username with search permissions to connect}\"\n", " password: \"{password of the user specified}\"\n", "```" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:14:46.801520Z", "start_time": "2020-08-07T19:14:44.889959Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Imports Complete\n" ] } ], "source": [ "#Check we are running Python 3.6\n", "import sys\n", "MIN_REQ_PYTHON = (3,6)\n", "if sys.version_info < MIN_REQ_PYTHON:\n", " print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')\n", " print('or later is selected as the active kernel.')\n", " sys.exit(\"Python %s.%s or later is required.\\n\" % MIN_REQ_PYTHON)\n", "\n", "#imports\n", "import pandas as pd\n", "import msticpy.nbtools as nbtools\n", "\n", "#data library imports\n", "from msticpy.data.data_providers import QueryProvider\n", "\n", "print('Imports Complete')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Instantiating a query provider\n", "\n", "You can instantiate a data provider for Splunk by specifying the credentials in connect or in msticpy config file. \n", "
If the details are correct and authentication is successful, it will show connected." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:09.229632Z", "start_time": "2020-08-07T19:15:08.536426Z" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "connected\n" ] } ], "source": [ "splunk_prov = QueryProvider('Splunk')\n", "splunk_prov.connect(host=, username=, password=)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Listing available queries\n", "Upon connecting to the Splunk data environment, we can take a look what query options available to us by running `QUERY_PROVIDER.list_queries()`\n", "\n", "For more information, refer documentation : [Listing available queries](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#listing-available-queries).\n", "\n", "This will display all the saved searches from the connected splunk instance and also pre-built custom queries to do common operations such as list datatypes, list saved searches, alerts, audittrail informaion. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:41.581783Z", "start_time": "2020-08-07T19:15:41.576649Z" } }, "outputs": [ { "data": { "text/plain": [ "['Alerts.list_all_alerts',\n", " 'SavedSearches.Errors_in_the_last_24_hours',\n", " 'SavedSearches.Errors_in_the_last_hour',\n", " 'SavedSearches.License_Usage_Data_Cube',\n", " 'SavedSearches.Load_sample_User_Agreements',\n", " 'SavedSearches.Messages_by_minute_last_3_hours',\n", " 'SavedSearches.Orphaned_scheduled_searches',\n", " 'SavedSearches.Score-Base',\n", " 'SavedSearches.Splunk_errors_last_24_hours',\n", " 'SavedSearches.Website_Performance_Problem',\n", " 'SavedSearches.inoperable_sites_rangemap',\n", " 'SavedSearches.slow_sites_avg_rangemap',\n", " 'SavedSearches.slow_sites_rangemap',\n", " 'SavedSearches.web_ping_inputs_lookup_gen',\n", " 'SavedSearches.website_availability_overview',\n", " 'SavedSearches.website_performance_problems',\n", " 'SplunkGeneral.get_events_parameterized',\n", " 'SplunkGeneral.list_all_datatypes',\n", " 'SplunkGeneral.list_all_savedsearches',\n", " 'audittrail.list_all_audittrail']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_prov.list_queries()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to get help for specific query , you can execute `QUERY_PROVIDER.('?')` .\n", "\n", "For more information , refer documentation - [Getting Help for a query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#getting-help-for-a-query)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:45.274273Z", "start_time": "2020-08-07T19:15:45.270895Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query: get_events_parameterized\n", "Data source: Splunk\n", "Generic parameterized query from index/source\n", "\n", "Parameters\n", "----------\n", "add_query_items: str (optional)\n", " Additional query clauses\n", " (default value is: | head 100)\n", "end: datetime (optional)\n", " Query end time\n", " (default value is: 08/26/2017:00:00:00)\n", "index: str (optional)\n", " Splunk index name\n", " (default value is: *)\n", "project_fields: str (optional)\n", " Project Field names\n", " (default value is: | table TimeCreated, host, EventID, EventDescripti...)\n", "source: str (optional)\n", " Splunk source type\n", " (default value is: *)\n", "start: datetime (optional)\n", " Query start time\n", " (default value is: 08/25/2017:00:00:00)\n", "timeformat: str (optional)\n", " Datetime format to use in Splunk query\n", " (default value is: \"%Y-%m-%d %H:%M:%S.%6N\")\n", "Query:\n", " search index={index} source={source} timeformat={timeformat} earliest={start} latest={end} {project_fields} {add_query_items}\n" ] } ], "source": [ "splunk_prov.SplunkGeneral.get_events_parameterized('?')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to print the query prior to executing, pass 'print' as an argument" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:47.472139Z", "start_time": "2020-08-07T19:15:47.464802Z" } }, "outputs": [ { "data": { "text/plain": [ "' search index=* source=* timeformat=\"%Y-%m-%d %H:%M:%S.%6N\" earliest=\"2020-08-15 19:15:47.466710\" latest=\"2020-08-15 19:15:47.466938\" | table TimeCreated, host, EventID, EventDescription, User, process, cmdline, Image, parent_process, ParentCommandLine, dest, Hashes | head 100'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_prov.SplunkGeneral.get_events_parameterized('print')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have set the arguments and then would like to validate the query, use below example" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:48.918900Z", "start_time": "2020-08-07T19:15:48.910871Z" } }, "outputs": [ { "data": { "text/plain": [ "' search index=botsv2 source=WinEventLog:Microsoft-Windows-Sysmon/Operational timeformat=%Y-%m-%d %H:%M:%S earliest=\"2017-08-25 00:00:00\" latest=\"2017-08-25 10:00:00\" | table TimeCreated, host, EventID, EventDescription, User, process, cmdline, Image, parent_process, ParentCommandLine, dest, Hashes | head 100'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_prov.SplunkGeneral.get_events_parameterized('print',\n", " index=\"botsv2\",\n", " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", " timeformat=\"%Y-%m-%d %H:%M:%S\",\n", " start=\"2017-08-25 00:00:00\",\n", " end=\"2017-08-25 10:00:00\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running pre-defined query\n", "In order to run pre-defined query , execute with the name either by setting values for arguments if available or run with default arguments.\n", "\n", "For more information , refer documentation - [Running an pre-definedfined query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#running-an-pre-defined-query)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:15:53.467435Z", "start_time": "2020-08-07T19:15:52.097203Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeCreatedhostEventIDEventDescriptionUserprocessImagedestcmdlineparent_processParentCommandLineHashes
02017-08-25T04:57:45.512440700Zvenus3Network ConnectNT AUTHORITY\\SYSTEMpowershell.exeC:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow...45.77.65.211.vultr.comNaNNaNNaNNaN
12017-08-25T04:57:45.213738500Zwrk-aturing5Process TerminateNaNconhost.exeC:\\Windows\\System32\\conhost.exeNaNNaNNaNNaNNaN
22017-08-25T04:57:45.213738500Zwrk-aturing5Process TerminateNaNcscript.exeC:\\Windows\\System32\\cscript.exeNaNNaNNaNNaNNaN
32017-08-25T04:57:45.088941700Zwrk-aturing1Process CreateNT AUTHORITY\\SYSTEMconhost.exeC:\\Windows\\System32\\conhost.exewrk-aturing.frothly.local\\??\\C:\\Windows\\system32\\conhost.exeC:\\Windows\\System32\\csrss.exe%SystemRoot%\\system32\\csrss.exe ObjectDirector...SHA1=680DEC0F8907F4B8911FBE2AA5F2FD25425BE0B0
42017-08-25T04:57:45.088941700Zwrk-aturing1Process CreateNT AUTHORITY\\SYSTEMcscript.exeC:\\Windows\\System32\\cscript.exewrk-aturing.frothly.localC:\\Windows\\system32\\cscript.exe //Job:AgentHI...C:\\Program Files (x86)\\Symantec\\Symantec Endpo...\"C:\\Program Files (x86)\\Symantec\\Symantec Endp...SHA1=70096A77E202CF9F30C064956F36D14BCBD8F7BB
.......................................
952017-08-25T04:57:02.003800000Zwrk-ghoppy1Process CreateNT AUTHORITY\\SYSTEMsplunk-powershell.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...wrk-ghoppy.frothly.local\"C:\\Program Files\\SplunkUniversalForwarder\\bin...C:\\Program Files\\SplunkUniversalForwarder\\bin\\...\"C:\\Program Files\\SplunkUniversalForwarder\\bin...SHA1=50A428905F5BA8808464F8A8183DD3662D8157F6
962017-08-25T04:57:01.170335100Zvenus3Network ConnectNT AUTHORITY\\SYSTEMpowershell.exeC:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow...45.77.65.211.vultr.comNaNNaNNaNNaN
972017-08-25T04:57:01.941402000Zwrk-ghoppy5Process TerminateNaNsplunk-winprintmon.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...NaNNaNNaNNaNNaN
982017-08-25T04:57:01.863404500Zwrk-ghoppy1Process CreateNT AUTHORITY\\SYSTEMsplunk-netmon.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...wrk-ghoppy.frothly.local\"C:\\Program Files\\SplunkUniversalForwarder\\bin...C:\\Program Files\\SplunkUniversalForwarder\\bin\\...\"C:\\Program Files\\SplunkUniversalForwarder\\bin...SHA1=0644F98A9874414C738A0B8841BB997FB9BFC274
992017-08-25T04:57:01.754208000Zwrk-ghoppy5Process TerminateNaNsplunk-powershell.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...NaNNaNNaNNaNNaN
\n", "

100 rows × 12 columns

\n", "
" ], "text/plain": [ " TimeCreated host EventID EventDescription \\\n", "0 2017-08-25T04:57:45.512440700Z venus 3 Network Connect \n", "1 2017-08-25T04:57:45.213738500Z wrk-aturing 5 Process Terminate \n", "2 2017-08-25T04:57:45.213738500Z wrk-aturing 5 Process Terminate \n", "3 2017-08-25T04:57:45.088941700Z wrk-aturing 1 Process Create \n", "4 2017-08-25T04:57:45.088941700Z wrk-aturing 1 Process Create \n", ".. ... ... ... ... \n", "95 2017-08-25T04:57:02.003800000Z wrk-ghoppy 1 Process Create \n", "96 2017-08-25T04:57:01.170335100Z venus 3 Network Connect \n", "97 2017-08-25T04:57:01.941402000Z wrk-ghoppy 5 Process Terminate \n", "98 2017-08-25T04:57:01.863404500Z wrk-ghoppy 1 Process Create \n", "99 2017-08-25T04:57:01.754208000Z wrk-ghoppy 5 Process Terminate \n", "\n", " User process \\\n", "0 NT AUTHORITY\\SYSTEM powershell.exe \n", "1 NaN conhost.exe \n", "2 NaN cscript.exe \n", "3 NT AUTHORITY\\SYSTEM conhost.exe \n", "4 NT AUTHORITY\\SYSTEM cscript.exe \n", ".. ... ... \n", "95 NT AUTHORITY\\SYSTEM splunk-powershell.exe \n", "96 NT AUTHORITY\\SYSTEM powershell.exe \n", "97 NaN splunk-winprintmon.exe \n", "98 NT AUTHORITY\\SYSTEM splunk-netmon.exe \n", "99 NaN splunk-powershell.exe \n", "\n", " Image \\\n", "0 C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow... \n", "1 C:\\Windows\\System32\\conhost.exe \n", "2 C:\\Windows\\System32\\cscript.exe \n", "3 C:\\Windows\\System32\\conhost.exe \n", "4 C:\\Windows\\System32\\cscript.exe \n", ".. ... \n", "95 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "96 C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow... \n", "97 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "98 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "99 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "\n", " dest \\\n", "0 45.77.65.211.vultr.com \n", "1 NaN \n", "2 NaN \n", "3 wrk-aturing.frothly.local \n", "4 wrk-aturing.frothly.local \n", ".. ... \n", "95 wrk-ghoppy.frothly.local \n", "96 45.77.65.211.vultr.com \n", "97 NaN \n", "98 wrk-ghoppy.frothly.local \n", "99 NaN \n", "\n", " cmdline \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 \\??\\C:\\Windows\\system32\\conhost.exe \n", "4 C:\\Windows\\system32\\cscript.exe //Job:AgentHI... \n", ".. ... \n", "95 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "96 NaN \n", "97 NaN \n", "98 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "99 NaN \n", "\n", " parent_process \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 C:\\Windows\\System32\\csrss.exe \n", "4 C:\\Program Files (x86)\\Symantec\\Symantec Endpo... \n", ".. ... \n", "95 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "96 NaN \n", "97 NaN \n", "98 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "99 NaN \n", "\n", " ParentCommandLine \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 %SystemRoot%\\system32\\csrss.exe ObjectDirector... \n", "4 \"C:\\Program Files (x86)\\Symantec\\Symantec Endp... \n", ".. ... \n", "95 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "96 NaN \n", "97 NaN \n", "98 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "99 NaN \n", "\n", " Hashes \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 SHA1=680DEC0F8907F4B8911FBE2AA5F2FD25425BE0B0 \n", "4 SHA1=70096A77E202CF9F30C064956F36D14BCBD8F7BB \n", ".. ... \n", "95 SHA1=50A428905F5BA8808464F8A8183DD3662D8157F6 \n", "96 NaN \n", "97 NaN \n", "98 SHA1=0644F98A9874414C738A0B8841BB997FB9BFC274 \n", "99 NaN \n", "\n", "[100 rows x 12 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_prov.SplunkGeneral.get_events_parameterized(\n", " index=\"botsv2\",\n", " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", " start=\"2017-08-25 00:00:00.000000\",\n", " end=\"2017-08-25 10:00:00.000000\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By-default, splunk query results are limited to 100. you can specify `count=0` argument to return all the results. \n", "Deafult value for `add_query_items` argument is set to `| head 100` which you can reset as shown in below example while retrieving all results. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:16:25.861893Z", "start_time": "2020-08-07T19:15:55.084994Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeCreatedhostEventIDEventDescriptionUserprocessImagedestcmdlineparent_processParentCommandLineHashes
02017-08-25T04:57:45.512440700Zvenus3Network ConnectNT AUTHORITY\\SYSTEMpowershell.exeC:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow...45.77.65.211.vultr.comNaNNaNNaNNaN
12017-08-25T04:57:45.213738500Zwrk-aturing5Process TerminateNaNconhost.exeC:\\Windows\\System32\\conhost.exeNaNNaNNaNNaNNaN
22017-08-25T04:57:45.213738500Zwrk-aturing5Process TerminateNaNcscript.exeC:\\Windows\\System32\\cscript.exeNaNNaNNaNNaNNaN
32017-08-25T04:57:45.088941700Zwrk-aturing1Process CreateNT AUTHORITY\\SYSTEMconhost.exeC:\\Windows\\System32\\conhost.exewrk-aturing.frothly.local\\??\\C:\\Windows\\system32\\conhost.exeC:\\Windows\\System32\\csrss.exe%SystemRoot%\\system32\\csrss.exe ObjectDirector...SHA1=680DEC0F8907F4B8911FBE2AA5F2FD25425BE0B0
42017-08-25T04:57:45.088941700Zwrk-aturing1Process CreateNT AUTHORITY\\SYSTEMcscript.exeC:\\Windows\\System32\\cscript.exewrk-aturing.frothly.localC:\\Windows\\system32\\cscript.exe //Job:AgentHI...C:\\Program Files (x86)\\Symantec\\Symantec Endpo...\"C:\\Program Files (x86)\\Symantec\\Symantec Endp...SHA1=70096A77E202CF9F30C064956F36D14BCBD8F7BB
.......................................
79232017-08-25T04:57:46.758125600Zwrk-klagerf1Process CreateNT AUTHORITY\\SYSTEMsplunk-admon.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...wrk-klagerf.frothly.local\"C:\\Program Files\\SplunkUniversalForwarder\\bin...C:\\Program Files\\SplunkUniversalForwarder\\bin\\...\"C:\\Program Files\\SplunkUniversalForwarder\\bin...SHA1=1C0C7368C8B7B688CCF77D1062708E60D581B0AF
79242017-08-25T04:57:46.695728800Zwrk-klagerf5Process TerminateNaNsplunk-MonitorNoHandle.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...NaNNaNNaNNaNNaN
79252017-08-25T04:57:46.570935200Zwrk-klagerf1Process CreateNT AUTHORITY\\SYSTEMsplunk-MonitorNoHandle.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...wrk-klagerf.frothly.local\"C:\\Program Files\\SplunkUniversalForwarder\\bin...C:\\Program Files\\SplunkUniversalForwarder\\bin\\...\"C:\\Program Files\\SplunkUniversalForwarder\\bin...SHA1=F48EDD0FE4D013D690196572EA96A4FA6EB04E77
79262017-08-25T04:57:46.539736800Zwrk-klagerf5Process TerminateNaNsplunk-powershell.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...NaNNaNNaNNaNNaN
79272017-08-25T04:57:46.430542400Zwrk-klagerf1Process CreateNT AUTHORITY\\SYSTEMsplunk-powershell.exeC:\\Program Files\\SplunkUniversalForwarder\\bin\\...wrk-klagerf.frothly.local\"C:\\Program Files\\SplunkUniversalForwarder\\bin...C:\\Program Files\\SplunkUniversalForwarder\\bin\\...\"C:\\Program Files\\SplunkUniversalForwarder\\bin...SHA1=50A428905F5BA8808464F8A8183DD3662D8157F6
\n", "

7928 rows × 12 columns

\n", "
" ], "text/plain": [ " TimeCreated host EventID EventDescription \\\n", "0 2017-08-25T04:57:45.512440700Z venus 3 Network Connect \n", "1 2017-08-25T04:57:45.213738500Z wrk-aturing 5 Process Terminate \n", "2 2017-08-25T04:57:45.213738500Z wrk-aturing 5 Process Terminate \n", "3 2017-08-25T04:57:45.088941700Z wrk-aturing 1 Process Create \n", "4 2017-08-25T04:57:45.088941700Z wrk-aturing 1 Process Create \n", "... ... ... ... ... \n", "7923 2017-08-25T04:57:46.758125600Z wrk-klagerf 1 Process Create \n", "7924 2017-08-25T04:57:46.695728800Z wrk-klagerf 5 Process Terminate \n", "7925 2017-08-25T04:57:46.570935200Z wrk-klagerf 1 Process Create \n", "7926 2017-08-25T04:57:46.539736800Z wrk-klagerf 5 Process Terminate \n", "7927 2017-08-25T04:57:46.430542400Z wrk-klagerf 1 Process Create \n", "\n", " User process \\\n", "0 NT AUTHORITY\\SYSTEM powershell.exe \n", "1 NaN conhost.exe \n", "2 NaN cscript.exe \n", "3 NT AUTHORITY\\SYSTEM conhost.exe \n", "4 NT AUTHORITY\\SYSTEM cscript.exe \n", "... ... ... \n", "7923 NT AUTHORITY\\SYSTEM splunk-admon.exe \n", "7924 NaN splunk-MonitorNoHandle.exe \n", "7925 NT AUTHORITY\\SYSTEM splunk-MonitorNoHandle.exe \n", "7926 NaN splunk-powershell.exe \n", "7927 NT AUTHORITY\\SYSTEM splunk-powershell.exe \n", "\n", " Image \\\n", "0 C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\pow... \n", "1 C:\\Windows\\System32\\conhost.exe \n", "2 C:\\Windows\\System32\\cscript.exe \n", "3 C:\\Windows\\System32\\conhost.exe \n", "4 C:\\Windows\\System32\\cscript.exe \n", "... ... \n", "7923 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7924 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7925 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7926 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7927 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "\n", " dest \\\n", "0 45.77.65.211.vultr.com \n", "1 NaN \n", "2 NaN \n", "3 wrk-aturing.frothly.local \n", "4 wrk-aturing.frothly.local \n", "... ... \n", "7923 wrk-klagerf.frothly.local \n", "7924 NaN \n", "7925 wrk-klagerf.frothly.local \n", "7926 NaN \n", "7927 wrk-klagerf.frothly.local \n", "\n", " cmdline \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 \\??\\C:\\Windows\\system32\\conhost.exe \n", "4 C:\\Windows\\system32\\cscript.exe //Job:AgentHI... \n", "... ... \n", "7923 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "7924 NaN \n", "7925 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "7926 NaN \n", "7927 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "\n", " parent_process \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 C:\\Windows\\System32\\csrss.exe \n", "4 C:\\Program Files (x86)\\Symantec\\Symantec Endpo... \n", "... ... \n", "7923 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7924 NaN \n", "7925 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "7926 NaN \n", "7927 C:\\Program Files\\SplunkUniversalForwarder\\bin\\... \n", "\n", " ParentCommandLine \\\n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 %SystemRoot%\\system32\\csrss.exe ObjectDirector... \n", "4 \"C:\\Program Files (x86)\\Symantec\\Symantec Endp... \n", "... ... \n", "7923 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "7924 NaN \n", "7925 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "7926 NaN \n", "7927 \"C:\\Program Files\\SplunkUniversalForwarder\\bin... \n", "\n", " Hashes \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 SHA1=680DEC0F8907F4B8911FBE2AA5F2FD25425BE0B0 \n", "4 SHA1=70096A77E202CF9F30C064956F36D14BCBD8F7BB \n", "... ... \n", "7923 SHA1=1C0C7368C8B7B688CCF77D1062708E60D581B0AF \n", "7924 NaN \n", "7925 SHA1=F48EDD0FE4D013D690196572EA96A4FA6EB04E77 \n", "7926 NaN \n", "7927 SHA1=50A428905F5BA8808464F8A8183DD3662D8157F6 \n", "\n", "[7928 rows x 12 columns]" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_prov.SplunkGeneral.get_events_parameterized(\n", " index=\"botsv2\",\n", " source=\"WinEventLog:Microsoft-Windows-Sysmon/Operational\",\n", " start=\"2017-08-25 00:00:00.000000\",\n", " end=\"2017-08-25 10:00:00.000000\",\n", " add_query_items='',\n", " count=0\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running a Ad-hoc Splunk query\n", "You can also define a your own splunk query and run it via splunk provider via `QUERY_PROVIDER.exec_query()`\n", "\n", "For more information, check documentation [Running and Ad-hoc Query](https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProviders.html#running-an-ad-hoc-query)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2020-08-07T19:16:35.628634Z", "start_time": "2020-08-07T19:16:35.294749Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TimeGeneratedTotalBytesSent
02020-07-02T10:00:00Z27055
12020-07-02T09:00:00Z33777
22020-07-02T08:00:00Z27355
32020-07-02T07:00:00Z25544
42020-07-02T06:00:00Z11771
\n", "
" ], "text/plain": [ " TimeGenerated TotalBytesSent\n", "0 2020-07-02T10:00:00Z 27055\n", "1 2020-07-02T09:00:00Z 33777\n", "2 2020-07-02T08:00:00Z 27355\n", "3 2020-07-02T07:00:00Z 25544\n", "4 2020-07-02T06:00:00Z 11771" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "splunk_query = '''\n", "search index=\"blackhat\" sourcetype=\"network\" earliest=0 \n", "| table TimeGenerated, TotalBytesSent\n", "'''\n", "df = splunk_prov.exec_query(splunk_query)\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## References\n", "\n", "- Splunk Enterprise SDK for Python: https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/\n", "- Splunk Community : https://community.splunk.com/t5/Community/ct-p/en-us\n", "- Splunk Documentation: https://docs.splunk.com/Documentation" ] } ], "metadata": { "hide_input": false, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "185.554px" }, "toc_section_display": true, "toc_window_display": true }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 4 }