{ "metadata": { "name": "", "signature": "sha256:63cd3717f97f61613cf08bec451e491fc65d856c9b41a57647ba11b53252aedd" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "# WIP: Rekall to Pandas Dataframe\n", "This notebook demonstrates a particularily kewl feature of workbench. Quickly and efficiently going from raw data to a Pandas Dataframe. \n", "\n", "Here we're using the workbench server to look at a forensic memory image that workbench processes with the Rekall python module https://github.com/google/rekall. Any thing that is kewl in this notebook is because of Rekall, anything that is lame is probably Workbench (our Rekall integration is days old).\n", "\n", "**Super Big Thanks**\n", "\n", "- JPH Security: This notebook utilitizes the 'Baseline Approach' outlined in this [JPH Security Blog](http://jphsecurity.blogspot.com/2012/01/developing-baseline-approach-to.html). Resources like this are terrific and greatly appreciated by us and the community.\n", "- Michael Cohen (scudette): Main developer of the Google Rekall project and amazingly patient with my dumb questions.\n", "\n", "**Tools in this Notebook:**\n", "\n", "- Workbench: Open Source Security Framework [Workbench GitHub](https://github.com/SuperCowPowers/workbench)\n", "- Rekall Memory Forensic Framework (http://www.rekall-forensic.com)\n", "- Pandas: Python Data Analysis Library (http://pandas.pydata.org)\n", "\n", "**More Info:** \n", "\n", "- See [PCAP_to_Dataframe](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Dataframe.ipynb) for a notebook on turning a PCAP into a Pandas Dataframe.\n", "- See [PCAP_to_Graph](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Graph.ipynb) for a short notebook on turning a PCAP into a Neo4j graph.\n", "\n", "- See [Workbench Demo Notebook](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/Workbench_Demo.ipynb) for a lot more info on using workbench.\n", "

\n", "\n", "## Lets start up the workbench server...\n", "Run the workbench server (from somewhere, for the demo we're just going to start a local one)\n", "
\n",
      "$ workbench_server\n",
      "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets start to interact with workbench, please note there is NO specific client to workbench,\n", "# Just use the ZeroRPC Python, Node.js, or CLI interfaces.\n", "import zerorpc\n", "c = zerorpc.Client(timeout=120)\n", "c.connect(\"tcp://127.0.0.1:4242\")" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 1, "text": [ "[None]" ] } ], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "# Read in the Data\n", " The data is pulled from a popular publically available memory image dataset called exemplar4.vmem.\n", "" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Load in the Memory Image file\n", "with open('../data/mem_images/exemplar4.vmem','rb') as f:\n", " mem_md5 = c.store_sample(f.read(), 'exemplar4.vmem', 'mem')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets look at the workers that we might invoke\n", "print c.help_workers()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Workbench Workers:\n", "\tjson_meta ['sample', 'meta']\n", "\tlog_meta ['sample', 'meta']\n", "\tmem_base ['sample']\n", "\tmem_connscan ['sample']\n", "\tmem_dlllist ['sample']\n", "\tmem_meta ['sample']\n", "\tmem_procdump ['sample']\n", "\tmem_pslist ['sample']\n", "\tmeta ['sample']\n", "\tmeta_deep ['sample', 'meta']\n", "\tpcap_bro ['sample']\n", "\tpcap_graph ['pcap_bro']\n", "\tpcap_graph_0_1 ['pcap_bro']\n", "\tpcap_http_graph ['pcap_bro']\n", "\tpe_classifier ['pe_features', 'pe_indicators']\n", "\tpe_deep_sim ['meta_deep']\n", "\tpe_features ['sample']\n", "\tpe_indicators ['sample']\n", "\tpe_peid ['sample']\n", "\tstrings ['sample']\n", "\tswf_meta ['sample', 'meta']\n", "\tunzip ['sample']\n", "\turl ['strings']\n", "\tview ['meta']\n", "\tview_customer ['meta']\n", "\tview_log_meta ['log_meta']\n", "\tview_meta ['meta']\n", "\tview_pcap ['pcap_bro']\n", "\tview_pcap_details ['view_pcap']\n", "\tview_pdf ['meta', 'strings']\n", "\tview_pe ['meta', 'strings', 'pe_peid', 'pe_indicators', 'pe_classifier', 'pe_disass']\n", "\tview_zip ['meta', 'unzip']\n", "\tvt_query ['meta']\n", "\tyara_sigs ['sample']\n" ] } ], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now we invoke the mem_meta worker (all memory workers start with mem_)\n", "output = c.work_request('mem_meta', mem_md5)['mem_meta']\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 4, "text": [ "{'md5': '359df4feb25af47cfef228f393d07c10',\n", " 'plugin_name': 'imageinfo',\n", " 'sections': {'Info': [{'Fact': 'Kernel DTB', 'Value': '0x7d0000'},\n", " {'Fact': 'NT Build', 'Value': '2600.xpsp_sp2_rtm.040803-2158'},\n", " {'Fact': 'NT Build Ex', 'Value': '-'},\n", " {'Fact': 'Signed Drivers', 'Value': '-'},\n", " {'Fact': 'Time (UTC)', 'Value': '2009-01-08 02:02:18+0000'},\n", " {'Fact': 'Time (Local)', 'Value': '2009-01-08 07:02:18+0000'},\n", " {'Fact': 'Sec Since Boot', 'Value': 937.34375},\n", " {'Fact': 'NtSystemRoot', 'Value': 'C:\\\\WINDOWS'}],\n", " 'Physical Layout': [{'Number of Pages': 158,\n", " 'Phys End': 651264,\n", " 'Phys Start': 4096},\n", " {'Number of Pages': 3839, 'Phys End': 16773120, 'Phys Start': 1048576},\n", " {'Number of Pages': 61168, 'Phys End': 267321344, 'Phys Start': 16777216},\n", " {'Number of Pages': 256, 'Phys End': 268435456, 'Phys Start': 267386880}]}}" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now we look at the pslist worker (which is just a big blog of python data)\n", "output = c.work_request('mem_pslist', mem_md5)['mem_pslist']\n", "str(output)[:50]" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 5, "text": [ "\"{'sections': {'Info': [{'Sess': '-', 'Hnds': 244, \"" ] } ], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "# Lets look at the Data\n", "We're going to use some nice functionality in the Pandas dataframe to look at our memory image data, specifically we're going to group by Parent Process IDs (PPIDs) and see which processes came from which parents.\n", "

\n", "This type of operation is really just scratching the surface when it comes to dataframes, so quickly and efficiently populating a dataframe is super awesome." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Okay that didn't seem very useful, just a gigantic ugly blob of python.\n", "# Lets push the pslist info section into a pandas dataframe\n", "import pandas as pd\n", "df = pd.DataFrame(output['sections']['Info'])\n", "df.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ExitHndsNameOffset (V)PIDPPIDSessStartThdsWow64
0 - 244 System [_EPROCESS _EPROCESS] @ 0x817CC7F8 (pid=4)\\n ... 4 0 - - 53 False
1 - 98 alg.exe [_EPROCESS _EPROCESS] @ 0x8163D020 (pid=408)\\n... 408 656 0 2009-01-08T01:48:23Z 5 False
2 - 21 smss.exe [_EPROCESS _EPROCESS] @ 0x8140F600 (pid=516)\\n... 516 4 - 2009-01-08T01:46:50Z 3 False
3 - 303 csrss.exe [_EPROCESS _EPROCESS] @ 0x81712170 (pid=588)\\n... 588 516 0 2009-01-08T01:46:56Z 9 False
4 - 599 winlogon.exe [_EPROCESS _EPROCESS] @ 0x8172D2D8 (pid=612)\\n... 612 516 0 2009-01-08T01:46:56Z 20 False
\n", "

5 rows \u00d7 10 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 6, "text": [ " Exit Hnds Name Offset (V) \\\n", "0 - 244 System [_EPROCESS _EPROCESS] @ 0x817CC7F8 (pid=4)\\n ... \n", "1 - 98 alg.exe [_EPROCESS _EPROCESS] @ 0x8163D020 (pid=408)\\n... \n", "2 - 21 smss.exe [_EPROCESS _EPROCESS] @ 0x8140F600 (pid=516)\\n... \n", "3 - 303 csrss.exe [_EPROCESS _EPROCESS] @ 0x81712170 (pid=588)\\n... \n", "4 - 599 winlogon.exe [_EPROCESS _EPROCESS] @ 0x8172D2D8 (pid=612)\\n... \n", "\n", " PID PPID Sess Start Thds Wow64 \n", "0 4 0 - - 53 False \n", "1 408 656 0 2009-01-08T01:48:23Z 5 False \n", "2 516 4 - 2009-01-08T01:46:50Z 3 False \n", "3 588 516 0 2009-01-08T01:46:56Z 9 False \n", "4 612 516 0 2009-01-08T01:46:56Z 20 False \n", "\n", "[5 rows x 10 columns]" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now lets use the Pandas groupby methods\n", "df['count'] = 1\n", "df.groupby(['PPID','Name','PID']).sum()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
HndsThdscount
PPIDNamePID
0 System4 244 53 1
4 smss.exe516 21 3 1
516 csrss.exe588 303 9 1
winlogon.exe612 599 20 1
612 lsass.exe668 321 20 1
services.exe656 249 15 1
656 alg.exe408 98 5 1
spoolsv.exe1516 109 11 1
svchost.exe888 222 9 1
984 1491 69 1
1020 197 18 1
1232 79 5 1
1304 202 13 1
984 wscntfy.exe1048 27 1 1
1888svhost.exe1936 83 7 1
2000explorer.exe1928 311 12 1
\n", "

16 rows \u00d7 3 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 7, "text": [ " Hnds Thds count\n", "PPID Name PID \n", "0 System 4 244 53 1\n", "4 smss.exe 516 21 3 1\n", "516 csrss.exe 588 303 9 1\n", " winlogon.exe 612 599 20 1\n", "612 lsass.exe 668 321 20 1\n", " services.exe 656 249 15 1\n", "656 alg.exe 408 98 5 1\n", " spoolsv.exe 1516 109 11 1\n", " svchost.exe 888 222 9 1\n", " 984 1491 69 1\n", " 1020 197 18 1\n", " 1232 79 5 1\n", " 1304 202 13 1\n", "984 wscntfy.exe 1048 27 1 1\n", "1888 svhost.exe 1936 83 7 1\n", "2000 explorer.exe 1928 311 12 1\n", "\n", "[16 rows x 3 columns]" ] } ], "prompt_number": 7 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "## The groupby is well organized, as we break it down we get nervous...\n", "- **smss.exe[516] spawned:**\n", " - csrss.exe\n", " - winlogon.exe[612]\n", "- **winlogon.exe[612] spawned:**\n", " - lsass.exe\n", " - services.exe[656]\n", "- **services.exe[656] spawned:**\n", " - alg.exe\n", " - spoolsv.exe\n", " - svchost.exe (5 of them)\n", "\n", "### Why is 1888 spawning svhost.exe[1936], not quite svchost.exe and more importantly not spawned by services.exe?!" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Now we look at the connscan worker\n", "output = c.work_request('mem_connscan', mem_md5)['mem_connscan']\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 8, "text": [ "{'md5': '359df4feb25af47cfef228f393d07c10',\n", " 'plugin_name': 'connscan',\n", " 'sections': {'Info': [{'Local Address': '192.168.30.128:1052',\n", " 'Offset(P)': 23629832,\n", " 'Pid': 1644,\n", " 'Remote Address': '204.160.104.126:80'},\n", " {'Local Address': '192.168.30.128:1051',\n", " 'Offset(P)': 23665408,\n", " 'Pid': 1644,\n", " 'Remote Address': '204.160.104.126:80'},\n", " {'Local Address': '192.168.30.128:1049',\n", " 'Offset(P)': 25150656,\n", " 'Pid': 1644,\n", " 'Remote Address': '65.54.152.225:80'},\n", " {'Local Address': '192.168.30.128:1048',\n", " 'Offset(P)': 25151712,\n", " 'Pid': 1644,\n", " 'Remote Address': '65.54.77.76:80'},\n", " {'Local Address': '192.168.30.128:1050',\n", " 'Offset(P)': 25152768,\n", " 'Pid': 1644,\n", " 'Remote Address': '192.221.98.124:80'},\n", " {'Local Address': '192.168.30.128:1056',\n", " 'Offset(P)': 25727888,\n", " 'Pid': 1936,\n", " 'Remote Address': '66.249.128.230:9899'},\n", " {'Local Address': '192.168.30.128:1055',\n", " 'Offset(P)': 26176800,\n", " 'Pid': 1644,\n", " 'Remote Address': '192.168.30.129:80'},\n", " {'Local Address': '3.0.48.2:18776',\n", " 'Offset(P)': 27477112,\n", " 'Pid': 2168284584,\n", " 'Remote Address': '192.221.114.126:19277'}]}}" ] } ], "prompt_number": 8 }, { "cell_type": "code", "collapsed": false, "input": [ "# Same as above we'll throw it into a Dataframe and do a group by\n", "conn_df = pd.DataFrame(output['sections']['Info'])\n", "conn_df['count'] = 1\n", "conn_df.groupby(['Pid','Remote Address']).sum()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Offset(P)count
PidRemote Address
1644 192.168.30.129:80 26176800 1
192.221.98.124:80 25152768 1
204.160.104.126:80 47295240 2
65.54.152.225:80 25150656 1
65.54.77.76:80 25151712 1
1936 66.249.128.230:9899 25727888 1
2168284584192.221.114.126:19277 27477112 1
\n", "

7 rows \u00d7 2 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 9, "text": [ " Offset(P) count\n", "Pid Remote Address \n", "1644 192.168.30.129:80 26176800 1\n", " 192.221.98.124:80 25152768 1\n", " 204.160.104.126:80 47295240 2\n", " 65.54.152.225:80 25150656 1\n", " 65.54.77.76:80 25151712 1\n", "1936 66.249.128.230:9899 25727888 1\n", "2168284584 192.221.114.126:19277 27477112 1\n", "\n", "[7 rows x 2 columns]" ] } ], "prompt_number": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "## Okay we see that our 'weird' svhost.exe[1936] is communicating to the outside world...\n", "- \"Port 9989/TCP is associated with the Ini-Killer remote access Trojan\" (quote from JPH Security Blog)\n", "- The 66.249.128.230 host is now clean, but at one point is was probably naughty. The important part here is that we can very quickly 'pivot' on the data from Rekall using Workbench." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Now lets look at the DLL for the various processes\n", "output = c.work_request('mem_dlllist', mem_md5)['mem_dlllist']" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 10 }, { "cell_type": "code", "collapsed": false, "input": [ "# Each process has it's own section\n", "output['sections'].keys()" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 11, "text": [ "['Info',\n", " 'lsass_exe pid: 668',\n", " 'services_exe pid: 656',\n", " 'winlogon_exe pid: 612',\n", " 'svchost_exe pid: 1020',\n", " 'svchost_exe pid: 1232',\n", " 'svchost_exe pid: 1304',\n", " 'System pid: 4',\n", " 'spoolsv_exe pid: 1516',\n", " 'svhost_exe pid: 1936',\n", " 'csrss_exe pid: 588',\n", " 'explorer_exe pid: 1928',\n", " 'svchost_exe pid: 888',\n", " 'alg_exe pid: 408',\n", " 'smss_exe pid: 516',\n", " 'svchost_exe pid: 984',\n", " 'wscntfy_exe pid: 1048']" ] } ], "prompt_number": 11 }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets look at the process of interest\n", "dll_df = pd.DataFrame(output['sections']['svhost_exe pid: 1936'])\n", "dll_df" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
BaseLoad Reason/CountPathSize
0 Pointer to Pointer to - 65535 C:\\Windows\\msagent\\svhost.exe 430080
1 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\ntdll.dll 720896
2 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\kernel32.dll 999424
3 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\advapi32.dll 634880
4 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\RPCRT4.dll 593920
5 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\shell32.dll 8470528
6 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\msvcrt.dll 360448
7 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\GDI32.dll 286720
8 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\USER32.dll 589824
9 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\SHLWAPI.dll 483328
10 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\ws2_32.dll 94208
11 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\WS2HELP.dll 32768
12 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\ole32.dll 1294336
13 Pointer to Pointer to - 65535 C:\\WINDOWS\\system32\\oleaut32.dll 573440
14 Pointer to Pointer to - 3 C:\\WINDOWS\\WinSxS\\x86_Microsoft.Windows.Common... 1056768
15 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\comctl32.dll 618496
16 Pointer to Pointer to - 2 C:\\WINDOWS\\system32\\version.dll 32768
17 Pointer to Pointer to - 2 C:\\WINDOWS\\system32\\wsock32.dll 36864
18 Pointer to Pointer to - 2 C:\\WINDOWS\\system32\\uxtheme.dll 229376
19 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\wininet.dll 679936
20 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\CRYPT32.dll 606208
21 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\MSASN1.dll 73728
22 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\Secur32.dll 69632
23 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\urlmon.dll 638976
24 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\icmp.dll 16384
25 Pointer to Pointer to - 3 C:\\WINDOWS\\system32\\iphlpapi.dll 102400
26 Pointer to Pointer to - 4 C:\\WINDOWS\\system32\\mswsock.dll 258048
27 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\hnetcfg.dll 360448
28 Pointer to Pointer to - 1 C:\\WINDOWS\\System32\\wshtcpip.dll 32768
29 Pointer to Pointer to - 2 C:\\WINDOWS\\system32\\DNSAPI.dll 159744
30 Pointer to Pointer to - 1 C:\\WINDOWS\\System32\\winrnr.dll 32768
31 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\WLDAP32.dll 180224
32 Pointer to Pointer to - 1 C:\\WINDOWS\\system32\\rasadhlp.dll 24576
\n", "

33 rows \u00d7 4 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 12, "text": [ " Base Load Reason/Count \\\n", "0 Pointer to Pointer to - 65535 \n", "1 Pointer to Pointer to - 65535 \n", "2 Pointer to Pointer to - 65535 \n", "3 Pointer to Pointer to - 65535 \n", "4 Pointer to Pointer to - 65535 \n", "5 Pointer to Pointer to - 65535 \n", "6 Pointer to Pointer to - 65535 \n", "7 Pointer to Pointer to - 65535 \n", "8 Pointer to Pointer to - 65535 \n", "9 Pointer to Pointer to - 65535 \n", "10 Pointer to Pointer to - 65535 \n", "11 Pointer to Pointer to - 65535 \n", "12 Pointer to Pointer to - 65535 \n", "13 Pointer to Pointer to - 65535 \n", "14 Pointer to Pointer to - 3 \n", "15 Pointer to Pointer to - 1 \n", "16 Pointer to Pointer to - 2 \n", "17 Pointer to Pointer to - 2 \n", "18 Pointer to Pointer to - 2 \n", "19 Pointer to Pointer to - 1 \n", "20 Pointer to Pointer to - 1 \n", "21 Pointer to Pointer to - 1 \n", "22 Pointer to Pointer to - 1 \n", "23 Pointer to Pointer to - 1 \n", "24 Pointer to Pointer to - 1 \n", "25 Pointer to Pointer to - 3 \n", "26 Pointer to Pointer to - 4 \n", "27 Pointer to Pointer to - 1 \n", "28 Pointer to Pointer to - 1 \n", "29 Pointer to Pointer to - 2 \n", "30 Pointer to Pointer to - 1 \n", "31 Pointer to Pointer to - 1 \n", "32 Pointer to Pointer to - 1 \n", "\n", " Path Size \n", "0 C:\\Windows\\msagent\\svhost.exe 430080 \n", "1 C:\\WINDOWS\\system32\\ntdll.dll 720896 \n", "2 C:\\WINDOWS\\system32\\kernel32.dll 999424 \n", "3 C:\\WINDOWS\\system32\\advapi32.dll 634880 \n", "4 C:\\WINDOWS\\system32\\RPCRT4.dll 593920 \n", "5 C:\\WINDOWS\\system32\\shell32.dll 8470528 \n", "6 C:\\WINDOWS\\system32\\msvcrt.dll 360448 \n", "7 C:\\WINDOWS\\system32\\GDI32.dll 286720 \n", "8 C:\\WINDOWS\\system32\\USER32.dll 589824 \n", "9 C:\\WINDOWS\\system32\\SHLWAPI.dll 483328 \n", "10 C:\\WINDOWS\\system32\\ws2_32.dll 94208 \n", "11 C:\\WINDOWS\\system32\\WS2HELP.dll 32768 \n", "12 C:\\WINDOWS\\system32\\ole32.dll 1294336 \n", "13 C:\\WINDOWS\\system32\\oleaut32.dll 573440 \n", "14 C:\\WINDOWS\\WinSxS\\x86_Microsoft.Windows.Common... 1056768 \n", "15 C:\\WINDOWS\\system32\\comctl32.dll 618496 \n", "16 C:\\WINDOWS\\system32\\version.dll 32768 \n", "17 C:\\WINDOWS\\system32\\wsock32.dll 36864 \n", "18 C:\\WINDOWS\\system32\\uxtheme.dll 229376 \n", "19 C:\\WINDOWS\\system32\\wininet.dll 679936 \n", "20 C:\\WINDOWS\\system32\\CRYPT32.dll 606208 \n", "21 C:\\WINDOWS\\system32\\MSASN1.dll 73728 \n", "22 C:\\WINDOWS\\system32\\Secur32.dll 69632 \n", "23 C:\\WINDOWS\\system32\\urlmon.dll 638976 \n", "24 C:\\WINDOWS\\system32\\icmp.dll 16384 \n", "25 C:\\WINDOWS\\system32\\iphlpapi.dll 102400 \n", "26 C:\\WINDOWS\\system32\\mswsock.dll 258048 \n", "27 C:\\WINDOWS\\system32\\hnetcfg.dll 360448 \n", "28 C:\\WINDOWS\\System32\\wshtcpip.dll 32768 \n", "29 C:\\WINDOWS\\system32\\DNSAPI.dll 159744 \n", "30 C:\\WINDOWS\\System32\\winrnr.dll 32768 \n", "31 C:\\WINDOWS\\system32\\WLDAP32.dll 180224 \n", "32 C:\\WINDOWS\\system32\\rasadhlp.dll 24576 \n", "\n", "[33 rows x 4 columns]" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "## We see from dll disk locations that our 'weird' svhost.exe[1936] is still weird...\n", "- C:\\Windows\\msagent\\svhost.exe doesn't look like a standard windows system location\n", "- We see communication dlls being loaded (wsock32.dll, ws2_32.dll)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Dump PE Files from all the processes\n", "output = c.work_request('mem_procdump', mem_md5)['mem_procdump']\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 13, "text": [ "{'dumped_files': [{'md5': '3dfee6c10e85fa73af431541ae57af1b',\n", " 'name': 'alg_exe_408.exe'},\n", " {'md5': 'f79c50b2a6e7b1a532c72d7e2e99a1b4', 'name': 'csrss_exe_588.exe'},\n", " {'md5': '4c4e5b2aed0a2d0776cd1bb640d94f10', 'name': 'explorer_exe_1928.exe'},\n", " {'md5': 'e5dd6fbc3580b523005e16e76677d942', 'name': 'lsass_exe_668.exe'},\n", " {'md5': '0a02acdfb8c10511aee002805eb324e4', 'name': 'services_exe_656.exe'},\n", " {'md5': 'a11279e7f15a0a9f342e0809d3460e26', 'name': 'smss_exe_516.exe'},\n", " {'md5': '01979d4ac9b6b0ac7a7f92dce6c1a5a7', 'name': 'spoolsv_exe_1516.exe'},\n", " {'md5': '5e4ab0ef8720a60cd986dfbc9673cfad', 'name': 'svchost_exe_1020.exe'},\n", " {'md5': '3936b4cc812630cc484096da94ca20b0', 'name': 'svchost_exe_1232.exe'},\n", " {'md5': '3d0857eaadc6f6f2281ace126a153906', 'name': 'svchost_exe_1304.exe'},\n", " {'md5': '919c1bfa6a544cf585c9e42462b61841', 'name': 'svchost_exe_888.exe'},\n", " {'md5': 'a128462ec37cffe7e57a1bc7defbc178', 'name': 'svchost_exe_984.exe'},\n", " {'md5': '0374a3a1689771e93432f8803cc2a09c', 'name': 'svhost_exe_1936.exe'},\n", " {'md5': 'd41d8cd98f00b204e9800998ecf8427e', 'name': 'System_4.exe'},\n", " {'md5': 'f5804b6e198a633c3e47703e8792e3d7', 'name': 'winlogon_exe_612.exe'},\n", " {'md5': 'eb891ddaffbae5e69a28363220ac79ee', 'name': 'wscntfy_exe_1048.exe'}],\n", " 'md5': '359df4feb25af47cfef228f393d07c10'}" ] } ], "prompt_number": 13 }, { "cell_type": "code", "collapsed": false, "input": [ "# Okay nice, now let look deeper out the files with Workbench\n", "# First the file that we're pretty sure is naughty\n", "c.work_request('view', '0374a3a1689771e93432f8803cc2a09c')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 14, "text": [ "{'view': {'md5': '0374a3a1689771e93432f8803cc2a09c',\n", " 'view_pe': {'classification': 'Evil!',\n", " 'customer': 'Huge Inc',\n", " 'disass': 'plugin_failed',\n", " 'encoding': 'binary',\n", " 'file_size': 126464,\n", " 'file_type': 'PE32 executable (GUI) Intel 80386, for MS Windows',\n", " 'filename': 'svhost_exe_1936.exe',\n", " 'import_time': '2014-06-16T21:54:21.990000Z',\n", " 'indicators': [{'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 0. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 1. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 2. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 3. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 4. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 5. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': \"Error parsing the import directory. Invalid Import data at RVA: 0x4ea64 ('Invalid entries in the Import Table. Aborting parsing.')\",\n", " 'severity': 2},\n", " {'category': 'MALFORMED',\n", " 'description': 'Reported Checksum does not match actual checksum',\n", " 'severity': 2},\n", " {'attributes': True,\n", " 'category': 'MALFORMED',\n", " 'description': 'Corrupted import table',\n", " 'severity': 3},\n", " {'category': 'MALFORMED',\n", " 'description': 'Image size does not match reported size',\n", " 'severity': 3},\n", " {'attributes': ['', '', '', '.xxx', '.adata'],\n", " 'category': 'MALFORMED',\n", " 'description': 'Section(s) with a non-standard name, tamper indication',\n", " 'severity': 3}],\n", " 'length': 126464,\n", " 'md5': '0374a3a1689771e93432f8803cc2a09c',\n", " 'mime_type': 'application/x-dosexec',\n", " 'peid_Matches': [],\n", " 'type_tag': 'exe'}}}" ] } ], "prompt_number": 14 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now smss_exe_516.exe\n", "c.work_request('view', 'a11279e7f15a0a9f342e0809d3460e26')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 20, "text": [ "{'view': {'md5': 'a11279e7f15a0a9f342e0809d3460e26',\n", " 'view_pe': {'classification': 'Evil!',\n", " 'customer': 'Mega Corp',\n", " 'disass': 'plugin_failed',\n", " 'encoding': 'binary',\n", " 'file_size': 50688,\n", " 'file_type': 'PE32 executable (native) Intel 80386, for MS Windows',\n", " 'filename': 'smss_exe_516.exe',\n", " 'import_time': '2014-06-16T21:54:21.959000Z',\n", " 'indicators': [{'category': 'MALFORMED',\n", " 'description': 'Reported Checksum does not match actual checksum',\n", " 'severity': 2}],\n", " 'length': 50688,\n", " 'md5': 'a11279e7f15a0a9f342e0809d3460e26',\n", " 'mime_type': 'application/x-dosexec',\n", " 'peid_Matches': [],\n", " 'type_tag': 'exe'}}}" ] } ], "prompt_number": 20 }, { "cell_type": "code", "collapsed": false, "input": [ "# Virus Total Query (on svhost.exe)\n", "c.work_request('vt_query', '0374a3a1689771e93432f8803cc2a09c')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 16, "text": [ "{'vt_query': {'file_type': 'PE32 executable (GUI) Intel 80386, for MS Windows',\n", " 'md5': '0374a3a1689771e93432f8803cc2a09c',\n", " 'positives': 17,\n", " 'scan_date': '2013-04-09 02:20:01',\n", " 'scan_results': [['Mal_Otorun7', 2],\n", " ['Heuristic.LooksLike.Win32.Suspicious.C', 1],\n", " ['Backdoor.Rbot!2E9D', 1],\n", " ['Worm:Win32/Pushbot.gen!C', 1],\n", " ['TR/Autorun.126464', 1]],\n", " 'total': 46}}" ] } ], "prompt_number": 16 }, { "cell_type": "code", "collapsed": false, "input": [ "# Virus Total Query (on smss_exe_516.exe)\n", "c.work_request('vt_query', 'a11279e7f15a0a9f342e0809d3460e26')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 17, "text": [ "{'vt_query': {'file_type': 'PE32 executable (native) Intel 80386, for MS Windows',\n", " 'md5': 'a11279e7f15a0a9f342e0809d3460e26',\n", " 'not_found': True}}" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#Wrap Up\n", "Well for this notebook we went from a forensic memory image to a Pandas Dataframe with the Power of Rekall (http://www.rekall-forensic.com). We hope this exercise showed some neato functionality using [Workbench](https://github.com/SuperCowPowers/workbench), we encourage you to check out the GitHub repository and our other notebooks:\n", "- [PCAP_to_Graph](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Graph.ipynb) for a short notebook on turning a PCAP into a Neo4j graph.\n", "- [Workbench Demo](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/Workbench_Demo.ipynb) general introduction to Workbench.\n", "- [PCAP_DriveBy](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/PCAP_DriveBy.ipynb) a detail look at a Web DriveBy from the [ThreatGlass](http://www.threatglass.com) repository.\n", "- [PE File Sim Graph](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/PE_SimGraph.ipynb) using Neo4j to generate a similarity graph using PE File features.\n", "- [Generator Pipelines](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/Generator_Pipelines.ipynb) using the client/server streaming generators to demonstrate 'chaining' generators." ] } ], "metadata": {} } ] }