{ "metadata": { "name": "", "signature": "sha256:63cd3717f97f61613cf08bec451e491fc65d856c9b41a57647ba11b53252aedd" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "# WIP: Rekall to Pandas Dataframe\n", "This notebook demonstrates a particularily kewl feature of workbench. Quickly and efficiently going from raw data to a Pandas Dataframe. \n", "\n", "Here we're using the workbench server to look at a forensic memory image that workbench processes with the Rekall python module https://github.com/google/rekall. Any thing that is kewl in this notebook is because of Rekall, anything that is lame is probably Workbench (our Rekall integration is days old).\n", "\n", "**Super Big Thanks**\n", "\n", "- JPH Security: This notebook utilitizes the 'Baseline Approach' outlined in this [JPH Security Blog](http://jphsecurity.blogspot.com/2012/01/developing-baseline-approach-to.html). Resources like this are terrific and greatly appreciated by us and the community.\n", "- Michael Cohen (scudette): Main developer of the Google Rekall project and amazingly patient with my dumb questions.\n", "\n", "**Tools in this Notebook:**\n", "\n", "- Workbench: Open Source Security Framework [Workbench GitHub](https://github.com/SuperCowPowers/workbench)\n", "- Rekall Memory Forensic Framework (http://www.rekall-forensic.com)\n", "- Pandas: Python Data Analysis Library (http://pandas.pydata.org)\n", "\n", "**More Info:** \n", "\n", "- See [PCAP_to_Dataframe](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Dataframe.ipynb) for a notebook on turning a PCAP into a Pandas Dataframe.\n", "- See [PCAP_to_Graph](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Graph.ipynb) for a short notebook on turning a PCAP into a Neo4j graph.\n", "\n", "- See [Workbench Demo Notebook](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/Workbench_Demo.ipynb) for a lot more info on using workbench.\n", "\n", "$ workbench_server\n", "" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets start to interact with workbench, please note there is NO specific client to workbench,\n", "# Just use the ZeroRPC Python, Node.js, or CLI interfaces.\n", "import zerorpc\n", "c = zerorpc.Client(timeout=120)\n", "c.connect(\"tcp://127.0.0.1:4242\")" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 1, "text": [ "[None]" ] } ], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# Read in the Data\n", " The data is pulled from a popular publically available memory image dataset called exemplar4.vmem.\n", "" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Load in the Memory Image file\n", "with open('../data/mem_images/exemplar4.vmem','rb') as f:\n", " mem_md5 = c.store_sample(f.read(), 'exemplar4.vmem', 'mem')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets look at the workers that we might invoke\n", "print c.help_workers()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Workbench Workers:\n", "\tjson_meta ['sample', 'meta']\n", "\tlog_meta ['sample', 'meta']\n", "\tmem_base ['sample']\n", "\tmem_connscan ['sample']\n", "\tmem_dlllist ['sample']\n", "\tmem_meta ['sample']\n", "\tmem_procdump ['sample']\n", "\tmem_pslist ['sample']\n", "\tmeta ['sample']\n", "\tmeta_deep ['sample', 'meta']\n", "\tpcap_bro ['sample']\n", "\tpcap_graph ['pcap_bro']\n", "\tpcap_graph_0_1 ['pcap_bro']\n", "\tpcap_http_graph ['pcap_bro']\n", "\tpe_classifier ['pe_features', 'pe_indicators']\n", "\tpe_deep_sim ['meta_deep']\n", "\tpe_features ['sample']\n", "\tpe_indicators ['sample']\n", "\tpe_peid ['sample']\n", "\tstrings ['sample']\n", "\tswf_meta ['sample', 'meta']\n", "\tunzip ['sample']\n", "\turl ['strings']\n", "\tview ['meta']\n", "\tview_customer ['meta']\n", "\tview_log_meta ['log_meta']\n", "\tview_meta ['meta']\n", "\tview_pcap ['pcap_bro']\n", "\tview_pcap_details ['view_pcap']\n", "\tview_pdf ['meta', 'strings']\n", "\tview_pe ['meta', 'strings', 'pe_peid', 'pe_indicators', 'pe_classifier', 'pe_disass']\n", "\tview_zip ['meta', 'unzip']\n", "\tvt_query ['meta']\n", "\tyara_sigs ['sample']\n" ] } ], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now we invoke the mem_meta worker (all memory workers start with mem_)\n", "output = c.work_request('mem_meta', mem_md5)['mem_meta']\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 4, "text": [ "{'md5': '359df4feb25af47cfef228f393d07c10',\n", " 'plugin_name': 'imageinfo',\n", " 'sections': {'Info': [{'Fact': 'Kernel DTB', 'Value': '0x7d0000'},\n", " {'Fact': 'NT Build', 'Value': '2600.xpsp_sp2_rtm.040803-2158'},\n", " {'Fact': 'NT Build Ex', 'Value': '-'},\n", " {'Fact': 'Signed Drivers', 'Value': '-'},\n", " {'Fact': 'Time (UTC)', 'Value': '2009-01-08 02:02:18+0000'},\n", " {'Fact': 'Time (Local)', 'Value': '2009-01-08 07:02:18+0000'},\n", " {'Fact': 'Sec Since Boot', 'Value': 937.34375},\n", " {'Fact': 'NtSystemRoot', 'Value': 'C:\\\\WINDOWS'}],\n", " 'Physical Layout': [{'Number of Pages': 158,\n", " 'Phys End': 651264,\n", " 'Phys Start': 4096},\n", " {'Number of Pages': 3839, 'Phys End': 16773120, 'Phys Start': 1048576},\n", " {'Number of Pages': 61168, 'Phys End': 267321344, 'Phys Start': 16777216},\n", " {'Number of Pages': 256, 'Phys End': 268435456, 'Phys Start': 267386880}]}}" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now we look at the pslist worker (which is just a big blog of python data)\n", "output = c.work_request('mem_pslist', mem_md5)['mem_pslist']\n", "str(output)[:50]" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 5, "text": [ "\"{'sections': {'Info': [{'Sess': '-', 'Hnds': 244, \"" ] } ], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "# Lets look at the Data\n", "We're going to use some nice functionality in the Pandas dataframe to look at our memory image data, specifically we're going to group by Parent Process IDs (PPIDs) and see which processes came from which parents.\n", "
\n", " | Exit | \n", "Hnds | \n", "Name | \n", "Offset (V) | \n", "PID | \n", "PPID | \n", "Sess | \n", "Start | \n", "Thds | \n", "Wow64 | \n", "
---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "- | \n", "244 | \n", "System | \n", "[_EPROCESS _EPROCESS] @ 0x817CC7F8 (pid=4)\\n ... | \n", "4 | \n", "0 | \n", "- | \n", "- | \n", "53 | \n", "False | \n", "
1 | \n", "- | \n", "98 | \n", "alg.exe | \n", "[_EPROCESS _EPROCESS] @ 0x8163D020 (pid=408)\\n... | \n", "408 | \n", "656 | \n", "0 | \n", "2009-01-08T01:48:23Z | \n", "5 | \n", "False | \n", "
2 | \n", "- | \n", "21 | \n", "smss.exe | \n", "[_EPROCESS _EPROCESS] @ 0x8140F600 (pid=516)\\n... | \n", "516 | \n", "4 | \n", "- | \n", "2009-01-08T01:46:50Z | \n", "3 | \n", "False | \n", "
3 | \n", "- | \n", "303 | \n", "csrss.exe | \n", "[_EPROCESS _EPROCESS] @ 0x81712170 (pid=588)\\n... | \n", "588 | \n", "516 | \n", "0 | \n", "2009-01-08T01:46:56Z | \n", "9 | \n", "False | \n", "
4 | \n", "- | \n", "599 | \n", "winlogon.exe | \n", "[_EPROCESS _EPROCESS] @ 0x8172D2D8 (pid=612)\\n... | \n", "612 | \n", "516 | \n", "0 | \n", "2009-01-08T01:46:56Z | \n", "20 | \n", "False | \n", "
5 rows \u00d7 10 columns
\n", "\n", " | \n", " | \n", " | Hnds | \n", "Thds | \n", "count | \n", "
---|---|---|---|---|---|
PPID | \n", "Name | \n", "PID | \n", "\n", " | \n", " | \n", " |
0 | \n", "System | \n", "4 | \n", "244 | \n", "53 | \n", "1 | \n", "
4 | \n", "smss.exe | \n", "516 | \n", "21 | \n", "3 | \n", "1 | \n", "
516 | \n", "csrss.exe | \n", "588 | \n", "303 | \n", "9 | \n", "1 | \n", "
winlogon.exe | \n", "612 | \n", "599 | \n", "20 | \n", "1 | \n", "|
612 | \n", "lsass.exe | \n", "668 | \n", "321 | \n", "20 | \n", "1 | \n", "
services.exe | \n", "656 | \n", "249 | \n", "15 | \n", "1 | \n", "|
656 | \n", "alg.exe | \n", "408 | \n", "98 | \n", "5 | \n", "1 | \n", "
spoolsv.exe | \n", "1516 | \n", "109 | \n", "11 | \n", "1 | \n", "|
svchost.exe | \n", "888 | \n", "222 | \n", "9 | \n", "1 | \n", "|
984 | \n", "1491 | \n", "69 | \n", "1 | \n", "||
1020 | \n", "197 | \n", "18 | \n", "1 | \n", "||
1232 | \n", "79 | \n", "5 | \n", "1 | \n", "||
1304 | \n", "202 | \n", "13 | \n", "1 | \n", "||
984 | \n", "wscntfy.exe | \n", "1048 | \n", "27 | \n", "1 | \n", "1 | \n", "
1888 | \n", "svhost.exe | \n", "1936 | \n", "83 | \n", "7 | \n", "1 | \n", "
2000 | \n", "explorer.exe | \n", "1928 | \n", "311 | \n", "12 | \n", "1 | \n", "
16 rows \u00d7 3 columns
\n", "\n", " | \n", " | Offset(P) | \n", "count | \n", "
---|---|---|---|
Pid | \n", "Remote Address | \n", "\n", " | \n", " |
1644 | \n", "192.168.30.129:80 | \n", "26176800 | \n", "1 | \n", "
192.221.98.124:80 | \n", "25152768 | \n", "1 | \n", "|
204.160.104.126:80 | \n", "47295240 | \n", "2 | \n", "|
65.54.152.225:80 | \n", "25150656 | \n", "1 | \n", "|
65.54.77.76:80 | \n", "25151712 | \n", "1 | \n", "|
1936 | \n", "66.249.128.230:9899 | \n", "25727888 | \n", "1 | \n", "
2168284584 | \n", "192.221.114.126:19277 | \n", "27477112 | \n", "1 | \n", "
7 rows \u00d7 2 columns
\n", "\n", " | Base | \n", "Load Reason/Count | \n", "Path | \n", "Size | \n", "
---|---|---|---|---|
0 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\Windows\\msagent\\svhost.exe | \n", "430080 | \n", "
1 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\ntdll.dll | \n", "720896 | \n", "
2 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\kernel32.dll | \n", "999424 | \n", "
3 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\advapi32.dll | \n", "634880 | \n", "
4 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\RPCRT4.dll | \n", "593920 | \n", "
5 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\shell32.dll | \n", "8470528 | \n", "
6 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\msvcrt.dll | \n", "360448 | \n", "
7 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\GDI32.dll | \n", "286720 | \n", "
8 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\USER32.dll | \n", "589824 | \n", "
9 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\SHLWAPI.dll | \n", "483328 | \n", "
10 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\ws2_32.dll | \n", "94208 | \n", "
11 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\WS2HELP.dll | \n", "32768 | \n", "
12 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\ole32.dll | \n", "1294336 | \n", "
13 | \n", "Pointer to Pointer to - | \n", "65535 | \n", "C:\\WINDOWS\\system32\\oleaut32.dll | \n", "573440 | \n", "
14 | \n", "Pointer to Pointer to - | \n", "3 | \n", "C:\\WINDOWS\\WinSxS\\x86_Microsoft.Windows.Common... | \n", "1056768 | \n", "
15 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\comctl32.dll | \n", "618496 | \n", "
16 | \n", "Pointer to Pointer to - | \n", "2 | \n", "C:\\WINDOWS\\system32\\version.dll | \n", "32768 | \n", "
17 | \n", "Pointer to Pointer to - | \n", "2 | \n", "C:\\WINDOWS\\system32\\wsock32.dll | \n", "36864 | \n", "
18 | \n", "Pointer to Pointer to - | \n", "2 | \n", "C:\\WINDOWS\\system32\\uxtheme.dll | \n", "229376 | \n", "
19 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\wininet.dll | \n", "679936 | \n", "
20 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\CRYPT32.dll | \n", "606208 | \n", "
21 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\MSASN1.dll | \n", "73728 | \n", "
22 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\Secur32.dll | \n", "69632 | \n", "
23 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\urlmon.dll | \n", "638976 | \n", "
24 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\icmp.dll | \n", "16384 | \n", "
25 | \n", "Pointer to Pointer to - | \n", "3 | \n", "C:\\WINDOWS\\system32\\iphlpapi.dll | \n", "102400 | \n", "
26 | \n", "Pointer to Pointer to - | \n", "4 | \n", "C:\\WINDOWS\\system32\\mswsock.dll | \n", "258048 | \n", "
27 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\hnetcfg.dll | \n", "360448 | \n", "
28 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\System32\\wshtcpip.dll | \n", "32768 | \n", "
29 | \n", "Pointer to Pointer to - | \n", "2 | \n", "C:\\WINDOWS\\system32\\DNSAPI.dll | \n", "159744 | \n", "
30 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\System32\\winrnr.dll | \n", "32768 | \n", "
31 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\WLDAP32.dll | \n", "180224 | \n", "
32 | \n", "Pointer to Pointer to - | \n", "1 | \n", "C:\\WINDOWS\\system32\\rasadhlp.dll | \n", "24576 | \n", "
33 rows \u00d7 4 columns
\n", "