{ "metadata": { "name": "", "signature": "sha256:3d611d2e47fe19b82f234319492f2c5a54c840555b581d4b3330e6df506ba867" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "# Workbench: Adding a worker \n", "We believe that transparency, flexibility and on-site customization are critical to an agile security framework. In this notebook we illustrate how easy it is to add workers to Workbench and more importantly how to codify your teams domain knowledge for on-site customization and agility.\n", "\n", "**Tools in this Notebook:**\n", "- Workbench: Open Source Security Framework [Workbench GitHub](https://github.com/SuperCowPowers/workbench)\n", "- Bro Network Security Monitor (http://www.bro.org)\n", "- Pandas: Python Data Analysis Library (http://pandas.pydata.org)\n", "\n", "**More Info:** \n", "- See [Workbench Demo Notebook](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/Workbench_Demo.ipynb) for a lot more info on using workbench.\n", "

\n", "\n", "## Lets start up the workbench server...\n", "Run the workbench server (from somewhere, for the demo we're just going to start a local one)\n", "
\n",
      "$ workbench_server\n",
      "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets start to interact with workbench, please note there is NO specific client to workbench,\n", "# Just use the ZeroRPC Python, Node.js, or CLI interfaces.\n", "import zerorpc\n", "c = zerorpc.Client()\n", "c.connect(\"tcp://127.0.0.1:4242\")" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 1, "text": [ "[None]" ] } ], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "## So I'm confused what am I suppose to do with workbench? \n", "
\n", " Workbench is often confusing for new users (we're trying to work on that). Please see our github repository https://github.com/SuperCowPowers/workbench for the latest documentation and notebooks examples (the notebook examples can really help). New users can start by typing **c.help()** after they connect to workbench." ] }, { "cell_type": "code", "collapsed": false, "input": [ "# I forgot what stuff I can do with workbench\n", "print c.help()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "Welcome to Workbench: Here's a list of help commands:\n", "\t - Run c.help_basic() for beginner help\n", "\t - Run c.help_commands() for command help\n", "\t - Run c.help_workers() for a list of workers\n", "\t - Run c.help_advanced() for advanced help\n", "\n", "See https://github.com/SuperCowPowers/workbench for more information\n" ] } ], "prompt_number": 2 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now lets get infomation about the dynamically loaded workers (your site may have many more!)\n", "# Next to each worker name is the list of dependences that worker has declared\n", "print c.help_workers()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Workbench Workers:\n", "\tjson_meta ['sample', 'meta']\n", "\tlog_meta ['sample', 'meta']\n", "\tmeta ['sample']\n", "\tmeta_deep ['sample', 'meta']\n", "\tmy_meta ['sample', 'meta']\n", "\tpcap_bro ['sample']\n", "\tpcap_graph ['pcap_bro']\n", "\tpcap_http_graph ['pcap_bro']\n", "\tpe_classifier ['pe_features', 'pe_indicators']\n", "\tpe_deep_sim ['meta_deep']\n", "\tpe_features ['sample']\n", "\tpe_indicators ['sample']\n", "\tpe_peid ['sample']\n", "\tstrings ['sample']\n", "\tswf_meta ['sample', 'meta']\n", "\tunzip ['sample']\n", "\turl ['strings']\n", "\tview ['meta']\n", "\tview_customer ['meta']\n", "\tview_log_meta ['log_meta']\n", "\tview_meta ['meta']\n", "\tview_pcap ['pcap_bro']\n", "\tview_pcap_details ['view_pcap']\n", "\tview_pdf ['meta', 'strings']\n", "\tview_pe ['meta', 'strings', 'pe_peid', 'pe_indicators', 'pe_classifier']\n", "\tview_zip ['meta', 'unzip']\n", "\tvt_query ['meta']\n", "\tyara_sigs ['sample']\n" ] } ], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "# Lets gets the infomation about the meta worker\n", "print c.help_worker('meta')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", " Worker: meta ['sample']\n", "\t This worker computes meta data for any file type. \n" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "# Okay lets load up a file, and see what this silly meta thing gives back\n", "filename = '../data/pe/bad/9e42ff1e6f75ae3e60b24e48367c8f26'\n", "with open(filename,'rb') as f:\n", " my_md5 = c.store_sample(f.read(), filename, 'exe')\n", "output = c.work_request('meta', my_md5)\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 5, "text": [ "{'meta': {'customer': 'Huge Inc',\n", " 'encoding': 'binary',\n", " 'file_size': 51200,\n", " 'file_type': 'PE32 executable (console) Intel 80386, for MS Windows',\n", " 'filename': '../../data/pe/bad/9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'import_time': '2014-06-21T23:51:49.122000Z',\n", " 'length': 51200,\n", " 'md5': '9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'mime_type': 'application/x-dosexec',\n", " 'type_tag': 'exe'}}" ] } ], "prompt_number": 5 }, { "cell_type": "code", "collapsed": false, "input": [ "# Pfff... my meta data worker will be WAY better!\n", "# Err.. okay I'll just copy the meta worker file and see what happens.\n", "# Note: obviously you'd just go to the shell and cp meta.py my_meta.py\n", "# but since we're in IPython...\n", "%cd /Users/briford/work/workbench/server/workers\n", "%cp meta.py my_meta.py\n", "%cd /Users/briford/work/workbench/notebooks" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "/Users/briford/work/workbench/server/workers\n" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "# Okay just cause I'm feeling crazy lets look at help_workers again\n", "print c.help_workers()" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Workbench Workers:\n", "\tjson_meta ['sample', 'meta']\n", "\tlog_meta ['sample', 'meta']\n", "\tmeta ['sample']\n", "\tmeta_deep ['sample', 'meta']\n", "\tmy_meta ['sample', 'meta']\n", "\tpcap_bro ['sample']\n", "\tpcap_graph ['pcap_bro']\n", "\tpcap_http_graph ['pcap_bro']\n", "\tpe_classifier ['pe_features', 'pe_indicators']\n", "\tpe_deep_sim ['meta_deep']\n", "\tpe_features ['sample']\n", "\tpe_indicators ['sample']\n", "\tpe_peid ['sample']\n", "\tstrings ['sample']\n", "\tswf_meta ['sample', 'meta']\n", "\tunzip ['sample']\n", "\turl ['strings']\n", "\tview ['meta']\n", "\tview_customer ['meta']\n", "\tview_log_meta ['log_meta']\n", "\tview_meta ['meta']\n", "\tview_pcap ['pcap_bro']\n", "\tview_pcap_details ['view_pcap']\n", "\tview_pdf ['meta', 'strings']\n", "\tview_pe ['meta', 'strings', 'pe_peid', 'pe_indicators', 'pe_classifier']\n", "\tview_zip ['meta', 'unzip']\n", "\tvt_query ['meta']\n", "\tyara_sigs ['sample']\n" ] } ], "prompt_number": 7 }, { "cell_type": "code", "collapsed": false, "input": [ "# My mind must be playing tricks, lets see if I can run my worker\n", "output = c.work_request('my_meta', my_md5)\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 8, "text": [ "{'my_meta': {'entropy': 7.250194413754419,\n", " 'md5': '9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'sha1': 'e0a6d12499ed16b33c71ddec42ca8aa7bcecaaf9',\n", " 'sha256': '88eea1726a149ac5c08b74547a05177398757f328c0faf821b822789d76863b7',\n", " 'ssdeep': '1536:pTrBy35F8qNwtqKiE/n5zTY+LK9lqB9HtZeV0D:hrEpF8q6qKiE/npi9UDHtZeV4'}}" ] } ], "prompt_number": 8 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "## Errr... wha?... But I'm hitting the Workbench server!? So WTF just happened?\n", "### Workbench has a dynamic plugin loader/validator, when a new file is detected in the worker directory the following steps happen automatically:\n", "\n", "- The plugin goes through several validation checks\n", "- If the validation succeeds the plugin is dynamically loaded\n", "- Your new plugins in now running on the local server\n", "- Also all of the CI build/test/coverage/docs now include your plug in!\n", "\n", "## Okay I'm going to call BS... lets run the tests and see what happens!" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# I've been around software... testing, server integration, test coverage all that stuff is\n", "# a complete PITA, heck I spend half my time doing that.. there's no way all that just happened.\n", "!./runtests" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\r\n", "<<< Note: Most of these tests require a local server running >>>\r\n", "\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ ".." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ ".." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ "." ] }, { "output_type": "stream", "stream": "stdout", "text": [ ".\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "Name Stmts Miss Cover Missing\r\n", "-------------------------------------------------------\r\n", "__init__ NoSource: No source for code: '/Users/briford/work/workbench/server/workers/__init__.py'\r\n", "json_meta 33 2 94% 22, 57\r\n", "log_meta 30 1 97% 49\r\n", "meta 40 1 98% 61\r\n", "meta_deep 38 1 97% 58\r\n", "my_meta 40 1 98% 61\r\n", "pcap_bro 122 9 93% 23-26, 115-116, 119, 121, 123, 197\r\n", "pcap_graph 112 6 95% 90, 117, 177, 181, 188, 231\r\n", "pcap_http_graph 90 5 94% 65, 121, 125, 134, 177\r\n", "pe_classifier 30 1 97% 54\r\n", "pe_deep_sim 39 1 97% 64\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "pe_features 208 21 90% 98-100, 148-149, 167-171, 179, 200, 222, 233, 244, 283, 290, 301, 304-305, 349\r\n", "pe_indicators 240 23 90% 52-53, 80, 103, 151, 159, 163, 173, 181, 193, 205, 217, 250, 260, 283, 301, 331, 349, 358, 386, 396-397, 439\r\n", "pe_peid 38 3 92% 23-24, 64\r\n", "strings 27 1 96% 44\r\n", "swf_meta 23 1 96% 44\r\n", "unzip 38 1 97% 60\r\n", "url 28 1 96% 46\r\n", "view 51 3 94% 25, 27, 86\r\n", "view_customer 23 1 96% 42\r\n", "view_log_meta 23 1 96% 40\r\n", "view_meta 23 1 96% 41\r\n", "view_pcap 31 1 97% 57\r\n", "view_pcap_details 32 1 97% 78\r\n", "view_pdf 28 2 93% 12, 46\r\n", "view_pe 39 2 95% 13, 64\r\n", "view_zip 36 2 94% 17, 61\r\n", "vt_query 55 7 87% 18, 22, 36-37, 50-51, 98\r\n", "workbench_keys/__init__ 1 0 100% \r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "yara_sigs 42 2 95% 20, 75\r\n", "-------------------------------------------------------\r\n", "TOTAL 1560 102 93% \r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "----------------------------------------------------------------------\r\n", "Ran 28 tests in 29.193s\r\n", "\r\n", "OK\r\n" ] } ], "prompt_number": 12 }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "# My new plugin has 98% test coverage...\n", "# Alright, are we feeling awesome yet? \n", " Welcome to a higher dimensional plane of awesomeness. Now that we're 'locked and loaded' lets focus on the the fun stuff, making the new plugin rock my co-workers minds!\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Okay my_meta.py is just a copy of meta.py (lets look at it)\n", "
\n", "
\n", "\n", "## We use our favorite editor, make changes to my_meta.py and hit save.\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "# Alright, now it's time to get fabulous!\n", "### The server plugin manager has a file system watchdog that has detected changes in your plugin. \n", "\n", "### The manager 'looks' at the new plugin and as long as it passes all the validation tests, it's automatically reloaded!!!" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# You sir are on some sort of needle drug... so you're saying that all the new functionality\n", "# that I just typed in is already available on the server? Help too? \n", "print c.help_worker('my_meta')\n", "output = c.work_request('my_meta', my_md5)\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", " Worker: my_meta ['sample']\n", "\tThis worker computes my more super awesome meta-data\n", " Seriously:\n", " 1) All the sha hashes\n", " 2) SSDeep (oh yeah)\n", " 3) Entropy (science!)\n", " \n" ] }, { "metadata": {}, "output_type": "pyout", "prompt_number": 36, "text": [ "{'my_meta': {'entropy': 2.440069216444288,\n", " 'md5': '0cb9aa6fb9c4aa3afad7a303e21ac0f3',\n", " 'sha1': '96e85768a12b2f319f2a4f0c048460e1b73aa573',\n", " 'sha256': '4ecf79302ba0439f62e15d0526a297975e6bb32ea25c8c70a608916a609e5a9c',\n", " 'ssdeep': '192:a8jJIFYrq9ATskBTp2jLDL3P1oynldvSo71nF:oFpNnnX1Tn'}}" ] } ], "prompt_number": 36 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Okay that was spiffy I'll give you that\n", "But I want my new worker to have access to the output of other workers, like I want to look the mime_type that the 'meta' worker has and then do some cool stuff based on that.\n", "\n", "## Lets look at the changes we made to my_meta.py\n", "We changed the dependency line and added the 'meta' worker (could be ANY worker)\n", "We also pulled the data from the meta worker and added the line about packed file.\n", "

\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Run my new code\n", "output = c.work_request('my_meta', my_md5)\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 16, "text": [ "{'my_meta': {'entropy': 7.250194413754419,\n", " 'md5': '9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'packed': 'probably',\n", " 'sha1': 'e0a6d12499ed16b33c71ddec42ca8aa7bcecaaf9',\n", " 'sha256': '88eea1726a149ac5c08b74547a05177398757f328c0faf821b822789d76863b7',\n", " 'ssdeep': '1536:pTrBy35F8qNwtqKiE/n5zTY+LK9lqB9HtZeV0D:hrEpF8q6qKiE/npi9UDHtZeV4'}}" ] } ], "prompt_number": 16 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Let enumerate all the neat things that just happened\n", "\n", "- I changed my worker, the plugin manager saw the change, validated my worker and dynamically loaded it\n", "- Although Workbench utilizes caching (no work is ever recomputed unless it needs to be) in this case it recognized that the 'modification time' of the worker was newer than the work results so it recomputes the results.\n", "- Lets look at my new plugin output in MongoDB\n", "

\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# So lets do a more complicated worker just for hammering home what's happening..\n", "# Workbench uses Directed Acyclic Graphs to pipeline workers together, it recursively\n", "# satisfies dependencies with agressive caching, shallow memory copies and gevent based\n", "# co-operative processes on the server side. Basicaly six slices of awesome...\n", "output = c.work_request('view', my_md5)\n", "output" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 17, "text": [ "{'view': {'md5': '9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'view_pe': {'classification': 'Evil!',\n", " 'customer': 'Huge Inc',\n", " 'disass': 'plugin_failed',\n", " 'encoding': 'binary',\n", " 'file_size': 51200,\n", " 'file_type': 'PE32 executable (console) Intel 80386, for MS Windows',\n", " 'filename': '../../data/pe/bad/9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'import_time': '2014-06-21T23:51:49.122000Z',\n", " 'indicators': [{'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 0. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'category': 'PE_WARN',\n", " 'description': 'Suspicious flags set for section 2. Both IMAGE_SCN_MEM_WRITE and IMAGE_SCN_MEM_EXECUTE are set. This might indicate a packed executable.',\n", " 'severity': 2},\n", " {'attributes': ['queryperformancecounter', 'gettickcount'],\n", " 'category': 'ANTI_DEBUG',\n", " 'description': 'Imported symbols related to anti-debugging',\n", " 'severity': 3},\n", " {'category': 'MALFORMED',\n", " 'description': 'Checksum of Zero',\n", " 'severity': 1},\n", " {'category': 'MALFORMED',\n", " 'description': 'Reported Checksum does not match actual checksum',\n", " 'severity': 2},\n", " {'category': 'MALFORMED',\n", " 'description': 'Image size does not match reported size',\n", " 'severity': 3},\n", " {'attributes': ['lsicbkg'],\n", " 'category': 'MALFORMED',\n", " 'description': 'Section(s) with a non-standard name, tamper indication',\n", " 'severity': 3},\n", " {'attributes': ['getmodulehandlea'],\n", " 'category': 'PROCESS_MANIPULATION',\n", " 'description': 'Imported symbols related to process manipulation/injection',\n", " 'severity': 3},\n", " {'attributes': ['getsystemtimeasfiletime'],\n", " 'category': 'PROCESS_SPAWN',\n", " 'description': 'Imported symbols related to spawning a new process',\n", " 'severity': 2},\n", " {'attributes': ['findfirstfilew', 'findnextfilew'],\n", " 'category': 'SYSTEM_PROBE',\n", " 'description': 'Imported symbols related to probing the system',\n", " 'severity': 2}],\n", " 'length': 51200,\n", " 'md5': '9e42ff1e6f75ae3e60b24e48367c8f26',\n", " 'mime_type': 'application/x-dosexec',\n", " 'peid_Matches': ['Microsoft Visual C++ v7.0'],\n", " 'type_tag': 'exe'}}}" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "# View is a magic unicorn, it pulls different DAGs based on mime-type\n", "
\n", "
" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Yeah but I want to run my new worker on a LOTS of samples and I\n", "# want to put the results into a Pandas dataframes and run some \n", "# statistics, and do some Machine Learning and kewl plots!\n", "\n", "# This is just throwing files at Workbench (could be pdfs, swfs, pcap, memory_images, etc)\n", "import os\n", "file_list = [os.path.join('../data/pe/bad', child) for child in os.listdir('../data/pe/bad')]\n", "working_set = []\n", "for filename in file_list:\n", " with open(filename,'rb') as f:\n", " md5 = c.store_sample(f.read(), filename, 'exe')\n", " working_set.append(md5)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 21 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now just run a batch request against all the sample we just threw in\n", "results = c.batch_work_request('my_meta', {'md5_list':working_set})\n", "results" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 27, "text": [ "" ] } ], "prompt_number": 27 }, { "cell_type": "code", "collapsed": false, "input": [ "# Now toss that client-server generator into a dataframe (zero-copy and efficient)\n", "import pandas as pd\n", "df_meta = pd.DataFrame(results)\n", "df_meta.head()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
entropymd5packedsha1sha256ssdeep
0 7.894680 033d91aae8ad29ed9fbb858179271232 probably 83ab10907b254752f312c89125957f10d35cb9d4 eb107c004e6e1bbd3b32ad7961661bbe28a577b0cb5dac... 1536:h6+LbfPbI5dzmJu9Tgj5aOItvEqRCHW9pjVrs2ryr...
1 2.440069 0cb9aa6fb9c4aa3afad7a303e21ac0f3 probably not 96e85768a12b2f319f2a4f0c048460e1b73aa573 4ecf79302ba0439f62e15d0526a297975e6bb32ea25c8c... 192:a8jJIFYrq9ATskBTp2jLDL3P1oynldvSo71nF:oFpN...
2 5.125292 0e882ec9b485979ea84c7843d41ba36f probably not 12fb0a1b7d9c2b2a41f4da9ce5bbfb140fb16939 616cf9e729c883d979212eb55178b7aac80dd9f58cb449... 768:5HyLMqtEM1Htz8kDmP9l+nZZYp41oj7EZmJxl/N9j6...
3 6.303055 0e8b030fb6ae48ffd29e520fc16b5641 probably not 82d57b8302b7497b2f6943f18e2d2687b9b0f5eb feaf72bdad035e198d297bfb0b8d891645f1dacd78f0db... 1536:1uNqjqzs1hQHhInEeJMzcmGqyF7Jwe9pvUo+5TDU4...
4 7.593283 0eb9e990c521b30428a379700ec5ab3e probably b778fc55f0538de865d4853099a3faa0b29f311d dc5e8176a5f012ebdb4835f9b570a12c045d059f6f5bdc... 1536:KcE4iMgXjTJpdGaaJG6Mhawv7r9ZaobsLBq+h5ttB...
\n", "

5 rows \u00d7 6 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 28, "text": [ " entropy md5 packed \\\n", "0 7.894680 033d91aae8ad29ed9fbb858179271232 probably \n", "1 2.440069 0cb9aa6fb9c4aa3afad7a303e21ac0f3 probably not \n", "2 5.125292 0e882ec9b485979ea84c7843d41ba36f probably not \n", "3 6.303055 0e8b030fb6ae48ffd29e520fc16b5641 probably not \n", "4 7.593283 0eb9e990c521b30428a379700ec5ab3e probably \n", "\n", " sha1 \\\n", "0 83ab10907b254752f312c89125957f10d35cb9d4 \n", "1 96e85768a12b2f319f2a4f0c048460e1b73aa573 \n", "2 12fb0a1b7d9c2b2a41f4da9ce5bbfb140fb16939 \n", "3 82d57b8302b7497b2f6943f18e2d2687b9b0f5eb \n", "4 b778fc55f0538de865d4853099a3faa0b29f311d \n", "\n", " sha256 \\\n", "0 eb107c004e6e1bbd3b32ad7961661bbe28a577b0cb5dac... \n", "1 4ecf79302ba0439f62e15d0526a297975e6bb32ea25c8c... \n", "2 616cf9e729c883d979212eb55178b7aac80dd9f58cb449... \n", "3 feaf72bdad035e198d297bfb0b8d891645f1dacd78f0db... \n", "4 dc5e8176a5f012ebdb4835f9b570a12c045d059f6f5bdc... \n", "\n", " ssdeep \n", "0 1536:h6+LbfPbI5dzmJu9Tgj5aOItvEqRCHW9pjVrs2ryr... \n", "1 192:a8jJIFYrq9ATskBTp2jLDL3P1oynldvSo71nF:oFpN... \n", "2 768:5HyLMqtEM1Htz8kDmP9l+nZZYp41oj7EZmJxl/N9j6... \n", "3 1536:1uNqjqzs1hQHhInEeJMzcmGqyF7Jwe9pvUo+5TDU4... \n", "4 1536:KcE4iMgXjTJpdGaaJG6Mhawv7r9ZaobsLBq+h5ttB... \n", "\n", "[5 rows x 6 columns]" ] } ], "prompt_number": 28 }, { "cell_type": "code", "collapsed": false, "input": [ "# Plotting defaults\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "plt.rcParams['font.size'] = 12.0\n", "plt.rcParams['figure.figsize'] = 18.0, 8.0" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 29 }, { "cell_type": "code", "collapsed": false, "input": [ "# Plot stuff (yes this is a silly plot but it's just an example :)\n", "df_meta.boxplot('entropy','packed')\n", "plt.xlabel('Packed')\n", "plt.ylabel('Entropy')\n", "plt.title('Entropy of Sample')\n", "plt.suptitle('')" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 31, "text": [ "" ] }, { "metadata": {}, "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAABDcAAAHxCAYAAAB5xFfZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm0lPV9P/D3UARRWQUUUESCppFWsWmMWqJXYxqNtTam\nrjFyFa3RWKtZqsYFUDTaxFOM1UQTCUiMnqQm8RhbtSZelyStScy1JmhwCWJlUVkE3IX5/eFvrlyB\nO8gy88zwep0zHr4zc+f5zOM5zPe+eT6fKZXL5XIAAAAAGlS3ehcAAAAAsCGEGwAAAEBDE24AAAAA\nDU24AQAAADQ04QYAAADQ0IQbAAAAQEMTbgAADeOtt97KSSedlIEDB6Zbt265//77613Semlra0u3\nbt0yd+7cepcCAE1BuAEABdXa2ppu3bqtduvTp897ep2TTz45BxxwwCaqsrZuvfXW3HzzzfnJT36S\n+fPnZ5999lnj8x599NEcccQRGTp0aLbccsvssMMOOeyww9Le3l7jigGAWuhe7wIAgLXbb7/98v3v\nf7/Tfd26bZp/m3jjjTfSo0ePTfLaG8sTTzyRYcOGZe+9917rc1544YUceOCB+ehHP5rbb789gwcP\nzrPPPpu77747ixYtqmG1AECtuHIDAAqsR48eGTx4cKfbwIEDOx5vaWnJKaeckksuuSRDhgzJtttu\nm3HjxuXll19OkkycODFTp07Nfffd13Hlx4033pjk7ZDk6quvznHHHZd+/fpl3LhxSZLp06dnt912\nS8+ePbPjjjvmwgsvzIoVKzodc/z48Tn33HMzaNCg9O3bN6eeempef/31JMm0adPSv3//vPrqq53e\ny8UXX5xdd921y/f7ta99LSNHjkzPnj0zatSoXHXVVZ2Oe9FFF+Xpp59Ot27dMnLkyDW+xs9//vMs\nXLgwU6dOzQc/+MHsuOOO2XfffTNx4sQceOCBHc+76qqrsueee6Z3794ZMmRIjj322MyfP7/j8Urr\nyH/+539mn332yVZbbZUPfehDeeyxx/K///u/+au/+qtsvfXW+fCHP5zHHnus4+emTZuWLbbYIj/9\n6U8zevTo9OrVK3vvvXceeeSRLt/7k08+mU996lPp379/BgwYkI9//OP53e9+1+XPAABvE24AQIGV\ny+Wqz/n3f//3LFmyJPfdd19uueWW/OQnP8kVV1yRJPnSl76U4447Lvvuu2/mz5+f+fPn5+ijj+74\n2UmTJmXs2LH57W9/m0suuSR33HFHxo8fn3HjxuX3v/99rrzyylxzzTWZNGnSasdcvHhxHnzwwdx0\n00358Y9/nPPOOy9Jcswxx6RUKuUHP/hBx/NXrlyZqVOn5pRTTlnr+7jmmmty0UUX5ctf/nJmzpyZ\nL33pSzn33HMzderUJMmPfvSjfOELX8iIESMyf/78/OpXv1rj6wwdOjRJcvPNN2flypVrPV6pVMqV\nV16Z3/3ud/nRj36UOXPm5JhjjlnteRdccEG+8pWv5De/+U222GKLHHPMMTn99NMzefLk/OY3v0mP\nHj1y4okndvqZlStX5pxzzsk3v/nNPPTQQxk0aFAOPfTQvPbaa2usZcGCBRk7dmy23377PPjgg/mf\n//mfvP/9709LS0tefPHFtb4HAOD/KwMAhTRu3Lhy9+7dy9tss02n29/+7d92PGf//fcvjxkzptPP\nnXbaaeV99tmnYz1+/PhyS0vLaq9fKpXKJ598cqf7xo4dWz766KM73XfVVVeVe/XqVX7zzTc7jrnz\nzjuXV65c2fGc66+/vrzllluWX3nllXK5XC6feeaZ5bFjx3Y8fuedd5Z79OhRfuGFF9b6fnfYYYfy\nOeec0+m+s88+uzxy5MiO9YQJE8qjRo1a62tUXHTRReUePXqU+/TpUz7ggAPKEydOLD/22GNd/szD\nDz9cLpVK5blz55bL5XL53nvvLZdKpfJtt93W8Zwf/OAH5VKpVP7hD3/Ycd+PfvSjcqlUKr/88svl\ncrlc/s53vlMulUrln/3sZx3PWbx4cXmbbbYp33DDDZ1e+7nnnut4X3vvvXenelauXFl+3/veV54y\nZUrV9wsAmztXbgBAgVXaGVa9XXfddR2Pl0ql7LHHHp1+ZsiQIVmwYME6vf5ee+3VaT1z5szst99+\nne7bb7/98tprr+Wpp57q9HOlUqljve++++b111/veM6pp56an//85/nDH/6QJPnWt76Vww8/vFNL\nzaqWLl2a5557bo3Hnj179lqveFibSZMmZcGCBZk2bVr23nvv3Hrrrdl9991z8803dzynra0tH//4\nxzN8+PD06dMnH/nIR5IkzzzzTKfXWvX8brfddkmS3XfffbX7nn/++U4/t+qw0379+uUDH/hAZs6c\nucZ6f/WrX+U3v/lNevfu3XHr06dPnnnmmTz55JPv6b0DwObIQFEAKLAtt9xyrbMlKt49BLRUKnXZ\njrGqrbfeer3qKldpl9ltt90yduzYXH/99TnnnHNy++2354477livY62vfv365ZOf/GQ++clP5rLL\nLsvHP/7xnH/++Tn22GMzZ86cfOITn8i4ceMyceLEDBw4MM8++2wOOuigvPHGG51eZ4sttuj4cyXQ\nWdN91c55V+esXC7noIMOyr/927+t9ljfvn2rv1kA2My5cgMACmzVqyPWV48ePToNBO3K6NGjc999\n93W677777stWW22V973vfR33/epXv+r0y/wvfvGL9OzZs9NzTj311Nx44425/vrrs8MOO+Sggw5a\n63H79OmTHXbYYY3HHjlyZLbccst1qr8ru+66a1544YWO+l977bVMmTIl++yzT3bZZZdOw0Q3hl/+\n8pcdf16yZEkef/zx7Lbbbmt87l/+5V/md7/7XYYNG5aRI0d2um277bYbtS4AaEbCDQAosNdffz0L\nFizoGAZauVWUy+WqV1GMHDkyjz/+eGbOnJkXX3xxtSsTVnXeeefl1ltvzRVXXJFZs2bl+9//fiZN\nmpQvfOEL6d79nQs+Fy5cmM997nN5/PHHc8cdd+Siiy7KZz/72fTq1avjOX//93+fJJk8eXJOPvnk\nqu/1vPPOy9VXX51vf/vbeeKJJ3Ldddflm9/8Zr785S9X/dlV3X777fn0pz+d22+/PX/4wx/yxBNP\n5Fvf+la+853v5JOf/GSSZJdddkmpVMrXvva1/PGPf8yPf/zjXHLJJe/pOF0plUo555xz8sADD+TR\nRx/NCSeckD59+uS4445b4/PPOOOMrFixIocffngefPDBzJ49Ow8++GDOP//8TiEJALBm2lIAoKBK\npVIeeOCBDBkyZLX7X3jhhQwYMCClUmm1qzvefd/48eNz7733Zt99983SpUszbdq0nHDCCWs85iGH\nHJKpU6fm8ssvz0UXXZRBgwblc5/7XCZMmNDp9Y888sj07t07Y8eOzRtvvJFjjjkml19+eafX6tmz\nZ44//vhce+21Oemkk6q+39NOOy0vv/xyLrvsspx++ukZPnx4rrjiik7fRLKm9/tuf/Znf5a+ffvm\n3HPPzZw5c1IqlTJixIicf/75+fznP5/k7ZkZV199dS6//PJceuml+cu//MtMmTIln/jEJ1Y7l++2\nLvd169Ytl112WU499dQ8/fTTGTNmTO64445OV6Cs+jODBw/OL3/5y3z5y1/OEUcckaVLl2b77bfP\nfvvt1/HtLwDA2pXK1f65ZyP6v//7v5x22mn5xS9+kR49euTv//7vM2XKlPzJn/xJrUoAADbQAQcc\nkF122SXXX3991eceddRRWbFiRW699dYaVFYM06ZNyymnnJI333yz3qUAwGajpm0pZ555ZgYOHJh5\n8+alvb099913X6699tpalgAAbKB1aYVZvHhx7rrrrvz4xz/O2WefXaPKAIDNVU3bUn7/+9/nqquu\nSo8ePbLddtvl4IMPzu9///talgAAbKB1aQ3Zc889s2jRopxzzjkZO3ZsjSorjo0xCBYAWHc1bUs5\n88wzs2TJklx33XVZtGhRDj744EyePDmHH354rUoAAAAAmkxNw41FixbloIMOyqOPPpoVK1aktbU1\nU6dOrdXhAQAAgCZUs3CjXC5nr732yhFHHJEvfvGLWbZsWU466aS8//3vzxVXXNHxvFGjRuWpp56q\nRUkAAABAg9hjjz3S3t6+xsdqFm688MIL2W677fLSSy+ld+/eSZIf//jHufDCC/Poo4++U1CpVHVI\nGdD8Jk6cmIkTJ9a7DACgAOwLgKTrvKBm35YycODADBkyJN/4xjeyYsWKLFmyJNOnT88ee+xRqxIA\nAACAJlSzcKNUKuWHP/xhbr/99gwcODC77LJLevbsmX/913+tVQlAA5k9e3a9SwAACsK+AKimpl8F\n++EPfzgPPPBALQ8JNKgxY8bUuwQAoCDsC4BqavptKevCzA0AAADg3brKC2p65QbQvEqlUr1L6JLQ\nFAAAmlfNZm4Aza1cLm/UW3LvRn49AKBRtbW11bsEoOCEGwAAAEBDM3MDKKRSKfFXAQAAUGHmBrBG\nAwYkixfXu4q1K+oYj/79k0WL6l0FAABQoS0FNmOLF799dUQRb/fe21b3GtZ2K3IgBADNyMwNoBrh\nBgAAANDQzNyAzZi5FuvHeQMAgNrrKi9w5QYAAADQ0IQbQCHprQUAKuwLgGp8WwpsxsopJQX9RpIi\nK6/yXwAAoP7M3IDNmNkR68d5AwCA2jNzAwAAAGhawg2gkPTWAgAV9gVANcINAAAAoKGZuQGbMbMj\n1o/zBgAAtWfmBgAAANC0hBtAIemtBQAq7AuAaoQbAAAAQEMzcwM2Y2ZHrB/nDQAAas/MDQAAAKBp\nCTeAQtJbCwBU2BcA1Qg3AAAAgIZm5gZsxsyOWD/OGwAA1J6ZGwAAAEDTEm4AhaS3FgCosC8AqhFu\nAAAAAA3NzA3YjJkdsX6cNwAAqD0zNwAAAICmJdwACklvLQBQYV8AVCPcAAAAABqamRuwGTM7Yv04\nbwAAUHtmbgAAAABNS7gBFJLeWgCgwr4AqEa4AQAAADQ0MzdgM2Z2xPpx3gAAoPbM3AAAAACalnAD\nKCS9tQBAhX0BUI1wAwAAAGhoZm7AZszsiPXjvAEAQO2ZuQEAAAA0LeEGbOZKpaLe2gpQw5pv/fvX\n+/8aAGxezNwAqule7wKA+ilya4XWDwAAYF2ZuQEUknADAABYlZkbAAAAQNMSbgAF1VbvAgCAgjBz\nA6hGuAEAAAA0NOEGUEgTJrTUuwQAoCBaWlrqXQJQcAaKAgAAAIVnoCjQcPTWAgAV9gVANcINAAAA\noKFpSwEAAAAKT1sKAAAA0LSEG0Ahtba21bsEAKAgzNwAqhFuAIU0fXq9KwAAABqFmRtAIZVKib8K\nAACACjM3AAAAgKYl3AAKqq3eBQAABWHmBlCNcAMAAABoaMINoJAmTGipdwkAQEG0tLTUuwSg4AwU\nBQAAAAqvEANFt9lmm/Tu3bvj1r1795x55pm1OjywiZVKpULfAIDGZeYGUE3Nwo3ly5dn2bJlWbZs\nWebPn59evXrlqKOOqtXhgU2sXC5v1Nu99967UV8PAABoXnVpS5k+fXouueSSPPnkk6sXpC0FAAAA\neJdCtKWsavr06TnhhBPqcWgAAACgydQ83HjmmWdy//33Z9y4cbU+NNBA9NYCABX2BUA13Wt9wBkz\nZuQjH/lIdtppp7U+p7W1NSNGjEiS9OvXL2PGjOn4+qfKX2zW1tbW1tbW1tbW1sVdH3DAASmyyqXt\nRTlf1tbWq6+nTJmS9vb2jnygKzWfubHrrrvmy1/+clpbW9dckJkbAAAAwLt0lRfUNNz4xS9+kb/+\n67/OggULsvXWW6+5IOEGAAAA8C6FGSh644035lOf+tRagw2AisolaQAAra1t9S4BKLi6fBVsV1y5\nASRvhxuVXjsAYPNWKrWlXG6pdxlAnRWmLWVdCDcAAIBVlUqJXxGAwrSlAAAAAGxswg2gkMzcAADe\n0VbvAoCCE24AAAAADU24ARSSYaIAQMWECS31LgEoOANFAQAAgMIzUBRoOGZuAAAV9gVANcINAAAA\noKFpSwEAAAAKT1sKAAAA0LSEG0Ah6a0FACpaW9vqXQJQcMINAACg0KZPr3cFQNGZuQEAABRaqZT4\nFQEwcwMAAABoWsINoJDM3AAA3tFW7wKAghNuAAAAAA1NuAEUUktLS71LAAAKYsKElnqXABScgaIA\nAABA4RkoCjQcMzcAgAr7AqAa4QYAAADQ0LSlAAAAAIWnLQUAAABoWsINoJD01gIAFa2tbfUuASg4\n4QYAAFBo06fXuwKg6MzcAAAACq1USvyKAJi5AQAAADQt4QZQSGZuAADvaKt3AUDBCTcAAACAhmbm\nBgAAkAEDksWL611F4+nfP1m0qN5VwOahq7xAuAEAABjauZ6cN6gdA0WBhmPmBgBQYV8AVCPcAAAA\nABqathQAAEB7xXpy3qB2tKUAAAAATUu4ARSS3loAoMK+AKhGuAEAAAA0NDM3AAAAsyPWk/MGtWPm\nBgAAANC0hBtAIemtBQAq7AuAaoQbAAAAQEMzcwMAADA7Yj05b1A7Zm4AAAAATUu4ARSS3loAoMK+\nAKhGuAEAAAA0NDM3AAAAsyPWk/MGtWPmBgAAANC0hBtAIemtBQAq7AuAaoQbAAAAQEMzcwMAADA7\nYj05b1A7Zm4AAAAATUu4ARSS3loAoMK+AKhGuAEAAAA0NDM3AAAAsyPWk/MGtWPmBgAAANC0hBtA\nIemtBQAq7AuAaoQbAAAAQEMzcwMAADA7Yj05b1A7Zm4AAAAATUu4ARSS3loAoMK+AKhGuAEAAAA0\nNDM3AAAAsyPWk/MGtWPmBgAAANC0ah5u3HLLLfnABz6QbbbZJqNGjcqDDz5Y6xKABqC3FgCosC8A\nquley4P913/9V84999x8//vfz1577ZV58+ZpQQEAAAA2SE1nbuy777455ZRTcuKJJ669IDM3AACg\n5syOWD/OG9ROIWZurFixIr/5zW/y/PPPZ5dddsmOO+6Yf/zHf8xrr71WqxIAAACAJlSzcGPBggV5\n8803c+utt+bBBx9Me3t7fvvb32by5Mm1KgFoIHprAYAK+wKgmprN3OjVq1eS5B//8R+z3XbbJUk+\n//nPZ/LkyasFHK2trRkxYkSSpF+/fhkzZkxaWlqSvPMXm7W1tbW1tbW1tbX1xlsnxapn1XV7e3uh\n6ll1nbSlra049VhbN9N6ypQpaW9v78gHulLTmRvDhw/PpZdems985jNJkh/+8IeZPHlyHn744XcK\nMnMDAABqzuyI9eO8Qe10lRfU9NtSTjzxxFx99dU5+OCD07179/zrv/5rDjvssFqWAAAArEE5paRU\n7yoaT3mV/wL1062WB7vwwgvzoQ99KLvuumt22223fPCDH8z5559fyxKABlG5JA0AqI1Sym9fglDA\nW9u999a9hrXdSoINKISaXrnRvXv3XHPNNbnmmmtqeVgAAACgidV05sa6MHMDAABqz+yI9eO8Qe10\nlRfUtC0FAAAAYGMTbgCFZOYGAFBhXwBUI9wAAAAAGpqZGwAAgNkR68l5g9oxcwMAAABoWsINoJD0\n1gIAFfYFQDXCDQAAAKChmbkBAACYHbGenDeoHTM3AAAAgKYl3AAKSW8tAFBhXwBUI9wAAAAAGpqZ\nGwAAgNkR68l5g9oxcwMAAABoWsINoJD01gIAFfYFQDXCDQAAAKChmbkBAACYHbGenDeoHTM3AAAA\ngKYl3AAKSW8tAFBhXwBUI9wAAAAAGpqZGwAAgNkR68l5g9oxcwMAAABoWsINoJD01gIAFfYFQDXC\nDQAAAKChmbkBAACYHbGenDeoHTM3AAAAgKYl3AAKSW8tANReqVTUW1sBaljzrX//ev9fA5Kke70L\nAAAA6q/IrRVaP4BqzNwAAAAKTbgBJGZuAAAAAE1MuAEUkpkbAMA72updAFBwwg0AAACgoQk3gEJq\naWmpdwkAQEFMmNBS7xKAgjNQFAAAACg8A0WBhmPmBgBQYV8AVCPcAAAAABqathQAAACg8LSlAAAA\nAE1LuAEUkt5aAKCitbWt3iUABSfcAAAACm369HpXABSdmRsAAEChlUqJXxEAMzcAAACApiXcAArJ\nzA0A4B1t9S4AKDjhBgAAANDQhBtAIbW0tNS7BACgICZMaKl3CUDBGSgKAAAAFN4GDxRduHDhRi0I\noBozNwCACvsCoJp1CjeGDx+eww8/PP/+7/+eN954Y1PXBAAAALDO1qkt5fnnn8/NN9+cGTNm5Kmn\nnsqRRx6ZE044IWPHjt34BWlLAQAAAN6lq7zgPc/cePzxxzNjxozcdNNN6datW44//viMHz8+O+20\n0yYvFgAAANg8bfDMjVXNnz8/CxYsyNKlSzNy5Mg899xzGTNmTL7yla9scKEAFXprAYCK1ta2epcA\nFNw6Xbnxu9/9Lt/97ndz8803p1evXhk3blyOP/747LjjjkmS2bNn58///M+zbNmyDS/IlRtA3g43\nfB0sAJAkpVJbyuWWepcB1NkGt6UMGDAgxx57bE444YR8+MMfXuNzLrzwwlxyySUbVmmEGwAAQGel\nUuJXBGCDw4033ngjPXr02OiFrYlwAwAAWJVwA0g2wsyNLbbYIjfccEMOOuig7LbbbvnYxz6Wb3/7\n21m5cuVGLRSgwswNAOAdbfUuACi47uvypHPOOSe33XZbzjrrrAwfPjxz5szJlVdemT/84Q/56le/\nuqlrBAAAAFirdWpLGTRoUB5++OGOAaJJ8uyzz2bPPffMiy++uHEL0pYCAACsYuLEt2/A5m2D21L6\n9OmT3r17d7qvd+/e6du374ZXBwAA0AXBBlDNOoUbZ511Vj71qU/l7rvvzmOPPZa77rorRx55ZM4+\n++w8/fTTHTeAjcXMDQCgwr4AqGad2lK6dauegZRKpaxYsWLDC9KWAuTtTUxLS0u9ywAACsC+AEg2\nwlfB1pJwAwAAAHi3rvKCdfq2lIo5c+bkueeey7BhwzJ8+PCNUhwAAADAhlinmRvz5s3L/vvvn1Gj\nRuWII47IqFGjst9++2Xu3Lnv6WAtLS3p1atXevfund69e+cDH/jAehUNND+9tQBARWtrW71LAApu\nncKNz372s9ljjz2yePHizJs3L4sXL86ee+6Zz372s+/pYKVSKddcc02WLVuWZcuW5bHHHluvogEA\ngM3H9On1rgAounWaubHttttm3rx56dGjR8d9r7/+eoYOHZqFCxeu88EOOOCAHH/88Rk/fvzaCzJz\nAwAAWEWplPgVAegqL1inKzcGDBiQmTNndrrv8ccfT//+/d9zMeedd14GDRqUsWPH5r777nvPPw8A\nAACwqnUaKPrP//zP+djHPpbx48dnp512yuzZs/Od73wnl1xyyXs62BVXXJHRo0enR48eufnmm3PY\nYYelvb09I0eOXK/igeblK98AgHe0JWmpcw1Aka3zV8H+7Gc/y0033ZR58+Zl6NChOfbYY/PRj350\ngw5+yCGH5NBDD80ZZ5zxTkGlUsaNG5cRI0YkSfr165cxY8Z0/JJTGTJobW3d3OvKn4tSj7W1tbW1\ntXX91gcc0J5y+azC1GNtbV2b9ZQpU9Le3t6RD0yaNGmtbSlVw4233nor73//+zNz5sz07Nmzq6e+\nZ2sLN8zcAAAAKiZOfPsGbN42aOZG9+7d061bt7z66qsbVMRLL72Uu+66K6+99lreeuut3HTTTXng\ngQdy8MEHb9DrAgAAzU2wAVRTNdxIkrPPPjtHH3102tra8tRTT+Xpp5/uuK2rN998MxdeeGEGDx6c\nQYMG5Zprrsltt92WUaNGrXfxQPOqXJIGAGBfAFSzTgNFK20j//Vf/7XaYytXrlynAw0cODAPPfTQ\neygNAAAAoLp1HihaK2ZuAAAAAO+2QTM3kuTMM89c4/1nnXXW+lcFAAAAsBGsU7jxne98Z43333jj\njRu1GIAKvbUAQEVra1u9SwAKrsuZGzfccEOSt78OdurUqSmXyymVSkmSp556KoMGDdr0FQIAAJu1\n6dOTadPqXQVQZF3O3GhpaUmpVMoDDzyQj3zkI+/8UKmU7bbbLv/0T/+Uvffee+MWZOYGAACwilIp\n8SsC0FVesE4DRc8///xceumlG72wNRFuAAAAqxJuAMlGCDcqnn/++SxfvrzTfSNHjtyw6t5dkHAD\nyNszN1paWupdBgBQAKVSW8rllnqXAdRZV3lBlzM3Ku68886MHz8+8+bNW+2FV6xYseEVAgAAAKyn\ndfq2lNNPPz0XXnhhli9fnpUrV3bcBBvApuKqDQCgYsKElnqXABTcOrWlDBgwIAsXLuz4ppRNWpC2\nFAAAAOBdusoL1unKjfHjx2fq1KkbtSiArrS1tdW7BACgIOwLgGrWaebGL3/5y1x11VW5/PLLs/32\n23fcXyqVcv/992+y4gAAAACqWae2lGnTpq35h0uljBs3buMWpC0FAAAAeJf1/irYM888M1//+tc7\n1t/+9rdz8sknd6yPOOKI/PCHP9yIpQo3AAAAgNWtd7jRu3fvLFu2rGPdv3//LF68eK2PbwzCDSB5\nu7fWN6YAAEnS2tqWadNa6l0GUGcbPFAUAACgXqZPr3cFQNEJN4BCctUGAPCOlnoXABRcl9+WsmLF\nivzsZz9LkpTL5bz11lud1itWrNj0FQIAAAB0ocuZGyNGjEipVOpYl8vlTusk+eMf/7hxCzJzA4iZ\nGwDAO0qltpTLLfUuA6izrvKCLq/cmD179qaoBwAAAGCjMXMDKCRXbQAAFRMmtNS7BKDgumxLqQdt\nKQAAAMC7+SpYoOG0tbXVuwQAoCDsC4BqhBsAAABAQ9OWAgAAABSethQAAACgaQk3gELSWwsAVLS2\nttW7BKDghBsAAEChTZ9e7wqAojNzAwAAKLRSKfErAmDmBgAAANC0hBtAIZm5AQC8o63eBQAFJ9wA\nAAAAGppwAyiklpaWepcAABTEhAkt9S4BKDgDRQEAAIDCM1AUaDhmbgAAFfYFQDXCDQAAAKChaUsB\nAAAACk9bCgAAANC0hBtAIemtBQAqWlvb6l0CUHDCDQAAoNCmT693BUDRmbkBAAAUWqmU+BUBMHMD\nAAAAaFqluGjLAAAUAElEQVTCDaCQzNwAAN7RVu8CgIITbgAAAAANTbgBFFJLS0u9SwAACmLChJZ6\nlwAUnIGiAAAAQOEZKAo0HDM3AIAK+wKgGuEGAAAA0NC0pQAAABtdqVSqdwld8jsHNJ6u8oLuNa4F\nAADYDAgPgFrSlgIUkt5aAKDCvgCoRrgBAAAANDQzNwAAAIDC81WwAAAAQNMSbgCFpLcWAKiwLwCq\nEW4AAAAADc3MDQAAAKDwzNwAAAAAmpZwAygkvbUAQIV9AVCNcAMAAABoaGZuAAAAAIVn5gYAAADQ\ntGoebjzxxBPZcsst85nPfKbWhwYaiN5aAKBiypS2epcAFFzNw43Pfe5z2WuvvVIqlWp9aAAAoAG1\nt9e7AqDoahpu3HLLLenfv38++tGPmqsBdKmlpaXeJQAABTFiREu9SwAKrnutDrR06dJMmDAh9957\nb66//vpaHRYAAGhAbW1v35Jk0qR37m9pefsGsKqahRsXXnhhTj755AwdOlRLClBVW1ubqzcAYDO2\naogxe3ZbJk5sqWM1QNHVJNxob2/PT3/60/z2t79NkqotKa2trRkxYkSSpF+/fhkzZkzHLzmVIYPW\n1tbW1tbW1tbW1pvHev789iTFqcfa2ro26ylTpqS9vb0jH+hKqVyD4RdXXXVVzj///PTu3TtJsnz5\n8qxYsSK77bZbfv3rX3cuqIvvrQUAADY/bW3J//9dB9iMdZUX1CTcePXVV7Ns2bIkb1+18bWvfS2z\nZ8/ON7/5zWy77bbrXCwAAACweeoqL+hWiwJ69eqVwYMHZ/Dgwdluu+2yzTbbpFevXqsFGwAVlUvS\nAADsC4BqajZQdFUTJkyox2EBAACAJlSTtpT3QlsKAAAA8G51b0sBAAAA2FSEG0Ah6a0FACrsC4Bq\nhBsAAABAQzNzAwAAACg8MzcAAACApiXcAApJby0AUGFfAFQj3AAAAAAampkbAAAAQOGZuQEAAAA0\nLeEGUEh6awGACvsCoBrhBgAAANDQzNwAAAAACs/MDQAAAKBpCTeAQtJbCwBU2BcA1Qg3AAAAgIZm\n5gYAAABQeGZuAAAAAE1LuAEUkt5aAKDCvgCoRrgBAAAANDQzNwAAAIDCM3MDAAAAaFrCDaCQ9NYC\nABX2BUA1wg0AAACgoZm5AQAAABSemRsAAABA0xJuAIWktxYAqLAvAKoRbgAAAAANzcwNAAAAoPDM\n3AAAAACalnADKCS9tQBAhX0BUI1wAwAAAGhoZm4AAAAAhWfmBgAAANC0hBtAIemtBQAq7AuAaoQb\nAAAAQEMzcwMAAAAoPDM3AAAAgKYl3AAKSW8tAFBhXwBUI9wAAAAAGpqZGwAAAEDhmbkBAAAANC3h\nBlBIemsBgAr7AqAa4QYAAADQ0MzcAAAAAArPzA0AAACgaQk3gELSWwsAVNgXANUINwAAAICGZuYG\nAAAAUHhmbgAAAABNS7gBFJLeWgCgwr4AqEa4AQAAADQ0MzcAAACAwjNzAwAAAGhawg2gkPTWAgAV\n9gVANcINAAAAoKGZuQEAAAAUnpkbAAAAQNMSbgCFpLcWAKiwLwCqEW4AAAAADc3MDQAAAKDwzNwA\nAAAAmlZNw43jjz8+Q4YMSZ8+fTJy5MhceumltTw80ED01gIAFfYFQDU1DTfOO++8/PGPf8zSpUvz\nn//5n7n66qtz55131rIEAAAAoMl0r+XBRo8e3fng3btn8ODBtSwBaBAtLS31LgEAKAj7AqCams/c\nOP3007P11ltn9OjRueCCC/IXf/EXtS4BAAAAaCI1DzeuvfbaLF++PPfcc08uuOCCPPTQQ7UuAWgA\nemsBgAr7AqCamralVJRKpbS0tOTII4/MzTffnL322qvT462trRkxYkSSpF+/fhkzZkzHpWiVv9is\nra2tra2tra2trTePdXt7e6Hqsba2rs16ypQpaW9v78gHulIqr+1LYmvg5JNPzvbbb5/Jkye/U1AX\n31sLAAAAbJ66ygu61aqIF154IbfccktefvnlrFixInfddVd+8IMf5PDDD69VCQAAAEATqlm4USqV\n8s1vfjM77LBDtt1221x44YWZMWNGPvShD9WqBKCBVC5JAwCwLwCqqdnMjYEDB/pLCQAAANjo6jpz\nY03M3AAAAADerRAzNwAAAAA2BeEGUEja2ACACvsCoBrhBgAAANDQzNwAAAAACs/MDQAAAKBpCTeA\nQtJbCwBU2BcA1Qg3AAAAgIZm5gYAAABQeGZuAAAAAE1LuAEUkt5aAKDCvgCoRrgBAAAANDQzNwAA\nAIDCM3MDAAAAaFrCDaCQ9NYCABX2BUA1wg0AAACgoZm5AQAAABSemRsAAABA0xJuAIWktxYAqLAv\nAKoRbgAAAAANzcwNAAAAoPDM3AAAAACalnADKCS9tQBAhX0BUI1wAwAAAGhoZm4AAAAAhWfmBgAA\nANC0hBtAIemtBQAq7AuAaoQbAAAAQEMzcwMAAAAoPDM3AAAAgKYl3AAKSW8tAFBhXwBUI9wAAAAA\nGppwAyiklpaWepcAABRGS70LAApOuAEAABSarhSgGuEGUEh6awGAitmz2+pdAlBw3etdAAAAwLu1\ntb1zxcb06cmIEW//uaXl7RvAqkrltX1JbJ109b21AADA5mfixLdvwOatq7xAWwoAAADQ0IQbQCGZ\nuQEAVPTr11bvEoCCE24AAACFNmZMvSsAis7MDQAAAKDwzNwAAAAAmpZwAygkMzcAgAr7AqAa4QYA\nAADQ0MzcAAAAAArPzA0AAACgaQk3gELSWwsAVNgXANUINwAAAICGZuYGAAAAUHhmbgAAAABNS7gB\nFJLeWgCgwr4AqEa4AQAAADQ0MzcAAACAwjNzAwAAAGhawg2gkPTWAgAV9gVANcINAAAAoKGZuQEA\nAAAUnpkbAAAAQNMSbgCFpLcWAKiwLwCqEW4AAAAADc3MDQAAAKDwzNwAAAAAmlbNwo033ngj48eP\nz4gRI9KnT5/sueeeufPOO2t1eKDB6K0FACrsC4BqahZuvPXWWxk+fHjuv//+LF26NJMnT85RRx2V\nZ555plYlAA2kvb293iUAAAVhXwBUU7NwY6uttsqECRMyfPjwJMmhhx6anXfeOQ8//HCtSgAayJIl\nS+pdAgBQEPYFQDV1m7mxYMGCzJo1K6NHj65XCQAAAEATqEu48eabb+bTn/50Wltbs+uuu9ajBKDg\nZs+eXe8SAICCsC8Aqqn5V8GuXLkyxx13XJYvX57bbrstf/Inf9Lp8TFjxuSRRx6pZUkAAABAwe2x\nxx5rncFT03CjXC7npJNOypw5c/If//Ef6dmzZ60ODQAAADSp7rU82GmnnZbHH38899xzj2ADAAAA\n2ChqduXGM888k5133jlbbrllp1aU66+/Pscee2wtSgAAAACaUM0Giu60005ZuXJlXnnllSxbtqzj\nJtiAzVNbW1t23HHH9frZ2bNnp1u3blm5cuUaH584cWI+85nPbEh5AMBG4PMeqJW6fRUswKZSKpXq\nXQIAsInV4/N+Q8IaYNMSbgCbxFtvvVW3Y9f4S6AAYLPl8x4oCuEG8J6MGDEil19+eUaPHp0BAwbk\npJNOyuuvv562trbssMMO+Zd/+ZcMGTIk48ePzxtvvJGzzjorw4YNy7Bhw3L22WfnjTfe6PR6X/nK\nVzJo0KDsvPPO+d73vtdx/x133JE999wzffv2zfDhwzNp0qTVarnhhhsybNiwDB06NFdeeWWnxyr/\nmnPooYfm3/7t3zo9tvvuu+e2227bWKcEAJrK2j7rkzTF532l3eXGG2/MTjvtlEGDBuWyyy7rePz1\n119f4/t5+eWXc8ghh2Tu3Lnp3bt3+vTpk/nz57/HswtsKsIN4D373ve+l7vvvjtPPfVUZs2alcmT\nJ6dUKmXBggVZvHhx5syZk+uuuy6TJ0/OQw89lEceeSSPPPJIHnrooUyePLnjdebPn5+FCxdm7ty5\nmT59ev7hH/4hs2bNSpJss802+e53v5uXXnopd9xxR77xjW+stkFpa2vLk08+mbvvvjtXXHFFfvrT\nn3Y8VvnXnNbW1nz3u9/tuP+RRx7J3Llzc+ihh27KUwQADW1Nn/UVzfJ5//Of/zyzZs3KT3/601x8\n8cX5wx/+kCS59NJL1/h+tt5669x5550ZOnRoli1blqVLl2b77bffgLMMbEzCDeA9KZVKOeOMMzJs\n2LD0798/559/fm6++eYkSbdu3TJp0qRsscUW2XLLLfO9730vF110UQYOHJiBAwdmwoQJmTFjRqfX\nu+SSS7LFFltkv/32y6GHHprvf//7SZL9998/o0ePTpL8+Z//eY455pjcd999nX52woQJ6dWrV/7s\nz/4sJ554YkcdqzrssMMya9asPPXUU0mSGTNm5Jhjjkn37jX9JmwAaBhdfdYnzfN5P2HChPTs2TO7\n77579thjjzzyyCNJ0uX70QoDxSXcAN6zVQdpDR8+PHPnzk2SDBo0KD169Oh4bO7cudlpp53W+Nwk\n6d+/f3r16tWx3mmnnToe/5//+Z8ccMABGTx4cPr165frrrsuCxcuXKc6VrXlllvmqKOOyowZM1Iu\nl3PLLbeYrA4AVXT1Gdssn/erXnWx1VZbZfny5ev0foBiEm4A79mcOXM6/Xno0KFJVp9aPnTo0Mye\nPXuNz02SxYsX55VXXulYP/PMMxk2bFiS5Ljjjsvf/d3f5f/+7/+yZMmSfPazn13tq+DeXUflZ99t\n3Lhxuemmm3LPPfdkq622yoc//OH3+I4BYPOyts/6pPk/77t6P76RDYpLuAG8J+VyOddee22ee+65\nLFq0KJdeemmOOeaYNT732GOPzeTJk/Piiy/mxRdfzMUXX7zav6JMmDAhb775Zh544IHccccdOfLI\nI5Mky5cvT//+/dOjR4889NBD+d73vrfahmLy5Ml59dVX8/vf/z7Tpk3L0UcfvcY69tlnn5RKpXzx\ni1/MCSecsBHOAgA0r/fyWZ803+d9V+9nu+22y8KFC7N06dL1fn1g09B0DrwnpVIpxx13XP76r/86\nc+fOzd/93d/lggsuyH//93+vthm54IILsnTp0uy+++5JkqOOOioXXHBBx+sMGTIk/fv3z9ChQ7P1\n1lvnuuuuy6677pokufbaa/OFL3whZ5xxRvbff/8cffTRWbJkSac69t9//4waNSorV67Ml770pRx0\n0EEdj727lhNOOCEXXXSRb0kBgCrW9lm/6uOrasTP+66uwOjq/fzpn/5pjj322IwcOTIrV67MzJkz\nDRWFgiiVTcUB3oOdd945N9xwQw488MB6l/KezJgxI9/61rdy//3317sUACi0Rv2sT3zew+ZMWwrQ\n9F555ZVcc801+Yd/+Id6lwIAbCI+72HzJtwAmtpdd92VwYMHZ8iQITnuuOPqXQ4AsAn4vAe0pQAA\nAAANzZUbAAAAQEMTbgAAAAANTbgBAAAANDThBgAAANDQhBsAQCG1tbVlxx133Civ1a1btzz99NMb\n5bUAgOIRbgAAG82IESOy1VZbpXfv3tl+++1z4okn5uWXX653WQBAkxNuAAAbTalUyk9+8pMsW7Ys\nDz/8cH79619n8uTJ9S4LAGhywg0AYJMYOnRoDjnkkDz66KM57LDDMnjw4AwYMCCHHXZYnnvuuY7n\nLVq0KCeeeGKGDRuWAQMG5JOf/OQaX+/rX/96Ro8enblz5+b111/PF7/4xey0007Zfvvtc9ppp+W1\n117reO5Xv/rVDB06NDvssEOmTp26yd8rAFBfwg0AYKMql8tJkmeffTb/8R//kfe973056aSTMmfO\nnMyZMye9evXKGWec0fH8z3zmM3nttdcyc+bMPP/88/n85z+/2mtefPHFufHGG3P//fdn6NChOffc\nc/Pkk0/mkUceyZNPPpnnnnsuF198cZLkzjvvzJVXXpl77rkns2bNyj333FObNw4A1E2pXNmBAABs\noBEjRmThwoXp3r17+vbtm7/5m7/JlVdemZ49e3Y8p729PQceeGAWLVqUefPmZYcddsiiRYvSt2/f\nTq/V1taW448/PkcddVR+/etf54477kjv3r1TLpfTu3fv/O///m9GjhyZJPnlL3+ZT3/603n66adz\n0kknZfvtt89ll12WJHniiSfy/ve/P08++WTH8wGA5tK93gUAAM2jVCrltttuy4EHHthx3yuvvJJT\nTz01d911VxYvXpwkWb58ecrlcp599tkMGDBgtWCjYsmSJfn2t7+dW265Jb17906SvPDCC3nllVfy\nwQ9+sON55XI5K1euTJLMmzcvH/rQhzoeGz58+EZ/nwBAsWhLAQA2qSuvvDKzZs3KQw89lJdeein3\n3XdfyuVyyuVydtxxxyxatCgvvfTSGn+2f//++clPfpITTzwxv/jFL5IkAwcOTK9evTJz5swsXrw4\nixcvzpIlS7J06dIkyZAhQzJnzpyO11j1zwBAcxJuAACb1PLly9OrV6/07ds3ixYtyqRJkzoeGzJk\nSA455JCcfvrpWbJkSd58883cf//9nX5+v/32y0033ZQjjjgiv/rVr9KtW7eccsopOeuss/LCCy8k\nSZ577rncfffdSZKjjjoq06ZNy2OPPZZXXnml0/EAgOYk3AAANqmzzjorr776agYOHJh99903hxxy\nSEqlUsfjM2bMyBZbbJE//dM/zXbbbZevf/3rHY9VnnfQQQdl6tSpOeyww9Le3p4rrrgio0aNyt57\n752+ffvmYx/7WGbNmpUkOfjgg3PWWWflwAMPzK677pqPfvSjnY4HADQfA0UBAACAhubKDQAAAKCh\nCTcAAACAhibcAAAAABqacAMAAABoaMINAAAAoKEJNwAAAICGJtwAAAAAGppwAwAAAGhowg0AAACg\nof0/DUnSv1yiSlIAAAAASUVORK5CYII=\n", "text": [ "" ] } ], "prompt_number": 31 }, { "cell_type": "code", "collapsed": false, "input": [ "# Groupby and Statistics (yes silly again but just an example)\n", "df_meta.groupby('packed').describe()" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
entropy
packed
probablycount 23.000000
mean 7.612472
std 0.290949
min 7.066162
25% 7.339412
50% 7.778263
75% 7.852313
max 7.919686
probably notcount 27.000000
mean 5.832247
std 1.017639
min 2.440069
25% 5.564519
50% 5.967721
75% 6.537146
max 6.998708
\n", "

16 rows \u00d7 1 columns

\n", "
" ], "metadata": {}, "output_type": "pyout", "prompt_number": 33, "text": [ " entropy\n", "packed \n", "probably count 23.000000\n", " mean 7.612472\n", " std 0.290949\n", " min 7.066162\n", " 25% 7.339412\n", " 50% 7.778263\n", " 75% 7.852313\n", " max 7.919686\n", "probably not count 27.000000\n", " mean 5.832247\n", " std 1.017639\n", " min 2.440069\n", " 25% 5.564519\n", " 50% 5.967721\n", " 75% 6.537146\n", " max 6.998708\n", "\n", "[16 rows x 1 columns]" ] } ], "prompt_number": 33 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#Wrap Up\n", "Well for this notebook we illustrated how simple it is to add a worker to the Workbench project. We hope this exercise showed some neato functionality using [Workbench](https://github.com/SuperCowPowers/workbench), we encourage you to check out the GitHub repository and our other notebooks:\n", "- [PCAP_to_Graph](http://nbviewer.ipython.org/github/SuperCowPowers/workbench/blob/master/workbench/notebooks/PCAP_to_Graph.ipynb) for a short notebook on turning this PCAP into a Neo4j graph.\n", "- [Workbench Demo](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/Workbench_Demo.ipynb) general introduction to Workbench.\n", "- [PCAP_DriveBy](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/PCAP_DriveBy.ipynb) a detail look at a Web DriveBy from the [ThreatGlass](http://www.threatglass.com) repository.\n", "- [PE File Sim Graph](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/PE_SimGraph.ipynb) using Neo4j to generate a similarity graph using PE File features.\n", "- [Generator Pipelines](http://nbviewer.ipython.org/url/raw.github.com/SuperCowPowers/workbench/master/workbench/notebooks/Generator_Pipelines.ipynb) using the client/server streaming generators to demonstrate 'chaining' generators." ] } ], "metadata": {} } ] }