{ "metadata": { "name": "Day_21_CommonCrawl_Completed" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Goals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For us to learn:\n", "\n", "* the basics of how to process CommonCrawl data by counting files and tallying file sizes in the CC crawl\n", "* **how to do this processing in parallel fashion using PiCloud, Amazon AWS (specifically S3), the [`boto` library](http://boto.readthedocs.org/en/latest/)** \n", "\n", "This notebook duplicates some of [Day_20_CommonCrawl_Starter](http://nbviewer.ipython.org/urls/raw.github.com/rdhyee/working-open-data/master/notebooks/Day_20_CommonCrawl_Starter.ipynb).\n", "\n", "For moving files between your computer and PiCloud, look at [Day_20_Moving_files_to_PiCloud.ipynb](http://nbviewer.ipython.org/urls/raw.github.com/rdhyee/working-open-data/master/notebooks/Day_20_Moving_files_to_PiCloud.ipynb).\n", "\n", "For understanding the actual content of the files in Common Crawl, we'll look at [Day_21_CommonCrawl_Content.ipynb](http://nbviewer.ipython.org/urls/raw.github.com/rdhyee/working-open-data/master/notebooks/Day_21_CommonCrawl_Content.ipynb)" ] }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Learning about Common Crawl structure" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Good to review Dave Lester's talk: http://www.slideshare.net/davelester/introduction-to-common-crawl \n", "\n", "If you need general intro to Common Crawl, watch the [Common Crawl Video](https://www.youtube.com/watch?v=ozX4GvUWDm4)." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Common Crawl data stored in Amazon S3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Common Crawl data structure is documented at https://commoncrawl.atlassian.net/wiki/display/CRWL/About+the+Data+Set. To quote the docs:\n", "\n", "The entire Common Crawl data set is stored on Amazon S3 as a Public Data Set:\n", "\n", " http://aws.amazon.com/datasets/41740\n", "\n", "The data set is divided into three major subsets:\n", "\n", "* Archived Crawl #1 - s3://aws-publicdatasets/common-crawl/crawl-001/ - crawl data from 2008/2010\n", "* Archived Crawl #2 - s3://aws-publicdatasets/common-crawl/crawl-002/ - crawl data from 2009/2010\n", "* Current Crawl - s3://aws-publicdatasets/common-crawl/parse-output/ - crawl data from 2012\n", "\n", "The two archived crawl data sets are stored in folders organized by the year, month, date, and hour the content was crawled. For example:\n", "\n", " s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\n", "\n", "The current crawl data set is stored in the \"parse-output\" folder in a similar manner to how Nutch stores archives. Crawl data is stored in a \"segments\" subfolder, then in a folder that starts with the UNIX timestamp of crawl start time. For example:\n", "\n", " s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz\n" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Using s3cmd and boto to confirm the examples from the documentation" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# this key, secret access to aws-publicdatasets only -- created for WwOD 13 student usage\n", "\n", "# turns out there is an anonymous mode in boto for public data sets:\n", "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n", "# conn=S3Connection(anon=True) will work instead of conn= S3Connection(KEY, SECRET) -- but there seems to be \n", "# a bug in how S3Connection gets pickled for anon=True -- so for now, just use the KEY, SECRET\n", "\n", "KEY = 'AKIAJH2FD7572FCTVSSQ'\n", "SECRET = '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'" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 1 }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use this key/secret pair to configure both `boto` and `s3cmd`" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# s3cmd installed in custom PiCloud environment -- and maybe in your local environment too\n", "\n", "# confirm s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\n", "# doc for s3cmd: http://s3tools.org/s3cmd\n", "\n", "!s3cmd ls s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "2012-01-05 19:19 100001092 s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\r\n" ] } ], "prompt_number": 2 }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "EXERCISE: use s3cmd to confirm existence of `s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz`" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "2012-07-09 10:43 100001274 s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz\r\n" ] } ], "prompt_number": 3 }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "using s3cmd to look at parse-output and valid_segments.txt in current crawl" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# looking at parse-output itself\n", "\n", "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ " DIR s3://aws-publicdatasets/common-crawl/parse-output-test/\r\n", " DIR s3://aws-publicdatasets/common-crawl/parse-output/\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "2012-09-04 05:03 0 s3://aws-publicdatasets/common-crawl/parse-output-test_$folder$\r\n", "2012-11-09 11:28 0 s3://aws-publicdatasets/common-crawl/parse-output_$folder$\r\n" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "# looking at what is contained by parse-output \"folder\"\n", "\n", "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ " DIR s3://aws-publicdatasets/common-crawl/parse-output/checkpoint_staging/\r\n", " DIR s3://aws-publicdatasets/common-crawl/parse-output/checkpoints/\r\n", " DIR s3://aws-publicdatasets/common-crawl/parse-output/segment/\r\n", " DIR s3://aws-publicdatasets/common-crawl/parse-output/valid_segments2/\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "2012-10-17 00:11 0 s3://aws-publicdatasets/common-crawl/parse-output/checkpoint_staging_$folder$\r\n", "2012-11-09 00:10 0 s3://aws-publicdatasets/common-crawl/parse-output/checkpoints_$folder$\r\n", "2012-09-05 05:13 0 s3://aws-publicdatasets/common-crawl/parse-output/segment_$folder$\r\n", "2012-11-09 11:28 2478 s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\r\n", "2012-09-05 05:13 0 s3://aws-publicdatasets/common-crawl/parse-output/valid_segments2_$folder$\r\n", "2012-07-09 15:07 0 s3://aws-publicdatasets/common-crawl/parse-output/valid_segments_$folder$\r\n" ] } ], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "There is a list of \"valid segments\" in \n", "\n", " s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\n", "\n", "-- a list of segments that are part of the current crawl. Let's download it and study it.\n", "\n", "See [discussion about valid segments](https://groups.google.com/forum/#!msg/common-crawl/QYTmnttZZyo/NPiXvK8ZeiMJ)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "2012-11-09 11:28 2478 s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\r\n" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "# we can download it:\n", "\n", "!s3cmd get --force s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt -> ./valid_segments.txt [1 of 1]\r\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\r", " 2478 of 2478 100% in 0s 4.48 kB/s\r", " 2478 of 2478 100% in 0s 4.48 kB/s done\r\n" ] } ], "prompt_number": 7 }, { "cell_type": "code", "collapsed": false, "input": [ "!head valid_segments.txt" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "1346823845675\r\n", "1346823846036\r\n", "1346823846039\r\n", "1346823846110\r\n", "1346823846125\r\n", "1346823846150\r\n", "1346823846176\r\n", "1346876860445\r\n", "1346876860454\r\n", "1346876860467\r\n" ] } ], "prompt_number": 8 }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "using boto to study parse-output and valid_segments.txt" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# http://boto.s3.amazonaws.com/s3_tut.html\n", "\n", "import boto\n", "from boto.s3.connection import S3Connection\n", "\n", "from itertools import islice\n", "\n", "conn = S3Connection(KEY,SECRET)\n", "\n", "# turns out there is an anonymous mode in boto for public data sets:\n", "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n", "#conn=S3Connection(anon=True)\n", "\n", "bucket = conn.get_bucket('aws-publicdatasets')\n", "for key in islice(bucket.list(prefix=\"common-crawl/parse-output/\", delimiter=\"/\"),None):\n", " print key.name.encode('utf-8')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "common-crawl/parse-output/checkpoint_staging_$folder$" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "common-crawl/parse-output/checkpoints_$folder$\n", "common-crawl/parse-output/segment_$folder$\n", "common-crawl/parse-output/valid_segments.txt\n", "common-crawl/parse-output/valid_segments2_$folder$\n", "common-crawl/parse-output/valid_segments_$folder$\n", "common-crawl/parse-output/checkpoint_staging/\n", "common-crawl/parse-output/checkpoints/\n", "common-crawl/parse-output/segment/\n", "common-crawl/parse-output/valid_segments2/\n" ] } ], "prompt_number": 9 }, { "cell_type": "code", "collapsed": false, "input": [ "# get valid_segments\n", "# https://commoncrawl.atlassian.net/wiki/display/CRWL/About+the+Data+Set\n", "\n", "import boto\n", "from boto.s3.connection import S3Connection\n", "\n", "conn = S3Connection(KEY, SECRET)\n", "bucket = conn.get_bucket('aws-publicdatasets')\n", "\n", "k = bucket.get_key(\"common-crawl/parse-output/valid_segments.txt\")\n", "s = k.get_contents_as_string()\n", "\n", "valid_segments = filter(None, s.split(\"\\n\"))\n", "\n", "print len(valid_segments), valid_segments[0]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "177 1346823845675\n" ] } ], "prompt_number": 10 }, { "cell_type": "code", "collapsed": false, "input": [ "# valid_segments are Unix timestamps (in ms) -- confirm current crawl is from 2012\n", "\n", "import datetime\n", "datetime.datetime.fromtimestamp(float(valid_segments[0])/1000.)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 11, "text": [ "datetime.datetime(2012, 9, 4, 22, 44, 5, 675000)" ] } ], "prompt_number": 11 }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Using boto to compile stats on each valid segment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As of the time of this writing (April 4, 2013), there are 177 valid segments in the current crawl. Now, it's time to figure out how to write a Python function called `segment_stats` that takes a segment id and an optional `stop` parameter (for the max number of keys to iterate through) of the form\n", "\n", " def segment_stats(seg_id, stop=None):\n", " pass\n", " # YOUR EXERCISE TO FILL IN\n", "\n", "and returns a `dict` with 2 keys: \n", "\n", "* `count` holding the number of keys inside the given valid segment\n", "* `size` holding the total number of bytes held in the keys\n", "\n", "broken down by file type (there are 3 major types):\n", "\n", "* `arg.gz` for the \n", "* 'metadata' for the metadata files\n", "* 'textData' for the textdata files\n", "* 'success' for success files\n", "\n", "For example:\n", "\n", " segment_stats('1346823845675', None)\n", "\n", "should return:\n", "\n", " {\n", " 'count': {'arc.gz': 11904, 'metadata': 4377, 'success': 1, 'textData': 4377},\n", " 'size': {'arc.gz': 967409519222,\n", " 'metadata': 187079951008,\n", " 'success': 0,\n", " 'textData': 129994977292}\n", " }" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Start by looking at a small subset of keys from valid_segments[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since it can take 10-50 seconds or so to retrieve all the keys in a valid segment, it's worth limiting to say first 10 to get a feel for what you can do with a key. Run the following:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "from itertools import islice\n", "\n", "import boto\n", "from boto.s3.connection import S3Connection\n", "\n", "conn = S3Connection(KEY, SECRET)\n", "bucket = conn.get_bucket('aws-publicdatasets')\n", "for key in islice(bucket.list(prefix=\"common-crawl/parse-output/segment/1346823845675/\", delimiter=\"/\"),10):\n", " print key.name.encode('utf-8')" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "common-crawl/parse-output/segment/1346823845675/1346864466526_10.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864469604_0.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864469638_1.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864471290_4.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864477152_29.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864479613_6.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864480261_2.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864480936_5.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864484063_39.arc.gz\n", "common-crawl/parse-output/segment/1346823845675/1346864484163_3.arc.gz\n" ] } ], "prompt_number": 12 }, { "cell_type": "code", "collapsed": false, "input": [ "# WARNING -- this might take a bit of time to run -- run it to see how long it takes you to get all the keys in this\n", "# segment. time depends on where you are running this code\n", "\n", "%time all_files = list(islice(bucket.list(prefix=\"common-crawl/parse-output/segment/1346823845675/\", delimiter=\"/\"),None))\n", "print len(all_files), all_files[0]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "CPU times: user 4.77 s, sys: 0.29 s, total: 5.06 s\n", "Wall time: 44.76 s\n", "20659 \n" ] } ], "prompt_number": 13 }, { "cell_type": "markdown", "metadata": {}, "source": [ "But it's useful now to have `all_files` to hold all the keys under the segment `1346823845675` Note, for example, you can get the size of the file and the name -- and the type of file (boto.s3.key.Key)" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# http://boto.readthedocs.org/en/latest/ref/s3.html#module-boto.s3.key\n", "\n", "file0 = all_files[0]\n", "type(file0), file0.name, file0.size" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 14, "text": [ "(boto.s3.key.Key,\n", " u'common-crawl/parse-output/segment/1346823845675/1346864466526_10.arc.gz',\n", " 100011998)" ] } ], "prompt_number": 14 }, { "cell_type": "code", "collapsed": false, "input": [ "import boto\n", "from boto.s3.connection import S3Connection\n", "\n", "# this key, secret access to aws-publicdatasets only -- createdd for WwOD 13 student usage\n", "KEY = 'AKIAJH2FD7572FCTVSSQ'\n", "SECRET = '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'\n", "\n", "from itertools import islice\n", "from pandas import DataFrame\n", "\n", "conn= S3Connection(KEY, SECRET)\n", "bucket = conn.get_bucket('aws-publicdatasets')\n", "\n", "# you might find this conversion function between DataFrame and a list of a regular dict useful\n", "#https://gist.github.com/mikedewar/1486027#comment-804797\n", "def df_to_dictlist(df):\n", " return [{k:df.values[i][v] for v,k in enumerate(df.columns)} for i in range(len(df))]\n", "\n", "def cc_file_type(path):\n", "\n", " fname = path.split(\"/\")[-1]\n", " \n", " if fname[-7:] == '.arc.gz':\n", " return 'arc.gz'\n", " elif fname[:9] == 'textData-':\n", " return 'textData'\n", " elif fname[:9] == 'metadata-':\n", " return 'metadata'\n", " elif fname == '_SUCCESS':\n", " return 'success'\n", " else:\n", " return 'other'\n", " \n", "# a first pass, using DataFrame. Might not be so efficient considering we are returning only totals\n", "def segment_stats(seg_id, stop=None):\n", " all_files = islice(bucket.list(prefix=\"common-crawl/parse-output/segment/{0}/\".format(seg_id), delimiter=\"/\"),stop)\n", " df = DataFrame([{'size': f.size if hasattr(f, 'size') else 0, 'name':f.name, 'type':cc_file_type(f.name)} for f in all_files])\n", " return {'count': df_to_dictlist(df[['size','type']].groupby('type').count()[['size']].T)[0],\n", " 'size': df_to_dictlist(df[['size', 'type']].groupby('type').sum().astype('int64').T)[0]}\n", " " ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 15 }, { "cell_type": "code", "collapsed": false, "input": [ "# another version of segment_stats that doesn't use DataFrame; probably easier to comprehend what's going on too -- and possibly\n", "# faster\n", "\n", "def segment_stats2(seg_id, stop=None):\n", " from collections import Counter\n", " file_count = Counter()\n", " byte_count = Counter()\n", " \n", " all_files = islice(bucket.list(prefix=\"common-crawl/parse-output/segment/{0}/\".format(seg_id), delimiter=\"/\"),stop)\n", " for f in all_files:\n", " file_type = cc_file_type(f.name)\n", " file_count.update({file_type: 1})\n", " byte_count.update({file_type: f.size if hasattr(f, 'size') else 0})\n", " \n", " return {'count': dict(file_count),\n", " 'size': dict(byte_count)}\n" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 16 }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Running segment_status locally and on PiCloud" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# recall the first segment -- let's work on that segment\n", "valid_segments[0]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 17, "text": [ "'1346823845675'" ] } ], "prompt_number": 17 }, { "cell_type": "code", "collapsed": false, "input": [ "# look at how long it takes to run locally\n", "\n", "%time segment_stats(valid_segments[0], None)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "CPU times: user 4.18 s, sys: 0.15 s, total: 4.33 s\n", "Wall time: 28.97 s\n" ] }, { "output_type": "pyout", "prompt_number": 18, "text": [ "{'count': {'arc.gz': 11904, 'metadata': 4377, 'success': 1, 'textData': 4377},\n", " 'size': {'arc.gz': 967409519222,\n", " 'metadata': 187079951008,\n", " 'success': 0,\n", " 'textData': 129994977292}}" ] } ], "prompt_number": 18 }, { "cell_type": "code", "collapsed": false, "input": [ "# here's how to run it on PiCloud\n", "# Prerequisite: http://docs.picloud.com/primer.html <--- READ THIS AND STUDY TO REFRESH YOUR MEMORY\n", "\n", "import cloud\n", "jid = cloud.call(segment_stats, '1346823845675', None, _env='/rdhyee/Working_with_Open_Data')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 19 }, { "cell_type": "code", "collapsed": false, "input": [ "# pull up status -- refresh until done\n", "cloud.status(jid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 25, "text": [ "'processing'" ] } ], "prompt_number": 25 }, { "cell_type": "code", "collapsed": false, "input": [ "# this will block until job is done or errors out\n", "\n", "cloud.join(jid)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 27 }, { "cell_type": "code", "collapsed": false, "input": [ "# get your result\n", "cloud.result(jid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 28, "text": [ "{'count': {'arc.gz': 11904, 'metadata': 4377, 'success': 1, 'textData': 4377},\n", " 'size': {'arc.gz': 967409519222,\n", " 'metadata': 187079951008,\n", " 'success': 0,\n", " 'textData': 129994977292}}" ] } ], "prompt_number": 28 }, { "cell_type": "code", "collapsed": false, "input": [ "# get some basic info\n", "cloud.info(jid)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 29, "text": [ "{1788: {'runtime': 16.1878, 'status': 'done', 'stderr': None, 'stdout': None}}" ] } ], "prompt_number": 29 }, { "cell_type": "code", "collapsed": false, "input": [ "# get some specific info\n", "cloud.info(jid, info_requested=['created', 'finished', 'runtime', 'cputime'])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 30, "text": [ "{1788: {'cputime.system': 2.55,\n", " 'cputime.user': 8.0,\n", " 'created': datetime.datetime(2013, 4, 16, 15, 59, 32),\n", " 'finished': datetime.datetime(2013, 4, 16, 15, 59, 52),\n", " 'runtime': 16.1878}}" ] } ], "prompt_number": 30 }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "What I got the first time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I had to retry 2 jobs\n", "\n", "* https://www.picloud.com/accounts/jobs/#/?ujid=344 -> read timed out\n", "* 375 -> AttributeError: 'Prefix' object has no attribute 'size'" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# now tally everything noting the retries -- might be worth writing this generally\n", "# THIS CODE REFERS SPECIFICALLY TO RAYMOND YEE'S JOBS -- REPLACE WITH YOUR OWN IDS\n", "\n", "from pandas import DataFrame\n", "\n", "import cloud\n", "from itertools import izip, ifilter, chain, islice\n", "\n", "from matplotlib import pyplot as plt\n", "\n", "valid_segments\n", "segment_jids = xrange(319, 496)\n", "retries_seg_ids = ['1346876860789', '1350433106986']\n", "retries_jids = xrange(496, 498)\n", "\n", "tally = list(ifilter(lambda x: x[2] == 'done', \n", " izip(chain(valid_segments, retries_seg_ids), chain(segment_jids, retries_jids), \n", " cloud.status(list(chain(segment_jids, retries_jids))))))\n", "\n", "result = cloud.result([jid for (seg_id, jid, status) in tally])\n", "\n", "# http://docs.picloud.com/moduledoc.html#module-cloud\n", "\n", "jobs_info = cloud.info(list(islice(chain(segment_jids, retries_jids),None)),\n", " info_requested=['created', 'finished', 'runtime', 'cputime', 'core']\n", " )\n", "\n", "started = [{'jid':k, 'time':v['finished'] - datetime.timedelta(seconds=v['runtime']), 'count': 1} for (k,v) in jobs_info.items()]\n", "finished = [{'jid':k, 'time':v['finished'], 'count': -1} for (k,v) in jobs_info.items()]\n", "\n", "df = DataFrame(started + finished)\n", "\n", "exclude_n = 4\n", "\n", "plot(df.sort_index(by='time')['time'][:-exclude_n], df.sort_index(by='time')['count'].cumsum()[:-exclude_n])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 31, "text": [ "[]" ] }, { "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVOX+B/DPIAqYyyUXzAU1RUFFRQXNosbSbCOXFsXS\nEqykcsmyxeyKllmZlXXNvBZat7TMW+otxav1G9FbVzQzUFHSJJfMFG+CbAqc3x9fDzPDLMxyhhmG\nz/v18gWznDMPB/zMM9/znOfRKYqigIiI/EaAtxtARETaYrATEfkZBjsRkZ9hsBMR+RkGOxGRn2Gw\nExH5GbvBnpSUhLCwMERHR1fdd+DAAdxxxx3o27cvEhISkJOT4/FGEhGR4+wG+8SJE5Genm5237x5\n8zBhwgTs3bsX48aNw7x58zzaQCIico7dYI+Pj0doaKjZfc2bN0d+fj4qKyuRn59v8TgREXmXrqYr\nT/Py8pCQkIDs7GwAQEFBAeLi4nDq1Cm0bdsWmZmZaNq0qflOdTrPtZiIyI9pMRmA0ydPk5KSMGXK\nFOTn52Py5MlITk622Th7/+bMmVPjc7z5z5fbx7b5X9t8vX11oW2A/Kv++G23Wb/fF4+bVpwO9h07\ndiApKQmBgYFITk5GRkaGZo0hItJaUJC3W1D7nA72IUOGYMOGDQCA9evXY9iwYZo3iohIKwz2ahIT\nEzF48GAcOnQIHTp0wIoVKzB79mysW7cOffr0wcaNG/H888+79MJ6vd6l7WqLL7ePbXONL7cN8O32\n1eW2eTPYvXXcajx56tJOdTpN60VERDVRx2xUj57Jk4Flyyzv90VaZSevPCUiv8ZSDBFRHVdRYX6b\nwU5EVMddumR+m8FORFTHMdgZ7ETkZy5eNL/NYCciquOq99iDg73TDm8K9HYDiIi0lJoKhIYCHTsC\njRsDjRrJ/cuXA0lJQIMGXm1ereA4diLyC7bmHly+HHjoIfk+MxOIja29NjmL49iJiBxgmpP1pb/J\nYCeieqOy0tstqB0MdiLya6a9dAY7EZGfYbATEfkB9tiJiPwYg52IyA+wx05E5McY7JCFq8PCwhAd\nHW12/4oVKxAVFYWePXvimWee8WgDiYjcwR57NRMnTkR6errZffv27cPf//53bNiwAfv378dTTz3l\n0QYSEbnDNNirz9Xur+zOFRMfH4+8vDyz+zZt2oTk5GREREQAAFq1amV129TU1Krv9Xq9T6+ZSER1\njxrSlZX2e+JFRcbvS0s92yZnGQwGGAwGzfdb41wxeXl5SEhIQHZ2NgBg2LBh6NmzJ3bs2IG+ffti\nxowZ6NGjh/lOOVcMEXnYyJHA0aNAVpZz2/lyNHltrpjS0lKcO3cO27dvx4gRI/D444+73QgiImd9\n843zoV5fOB3sgwYNwpgxYxASEoKEhAQcPHgQpb72+YaI/F5Jibdb4LucDvZrrrkGmzZtgqIo2Llz\nJ7p06YLg+jiTPRF5VUVF/Zhb3RV2gz0xMRGDBw9Gbm4uOnTogBUrVmDEiBEoLy9Hjx498Morr+CN\nN96orbYSEZkJCfF2C3wTF9ogojpJpwNatQLOnHFuO1+OJi60QUT1XuPG3m6Bb2KwE1GdxVKMdQx2\nIqqzGOzWMdiJqM5isFvHYCeiOovBbh2DnYjqLAa7dRzuSER1kk7n2na+HE0c7khERFYx2ImI/AyD\nnYjIzzDYiYj8DIOdiMjPMNiJiPwMg52IyM8w2ImI/IzdYE9KSkJYWBiio6MtHlu0aBECAgJw7tw5\njzWOiIicZzfYJ06ciPT0dIv7jx8/ji1btqBjx44eaxgREbnGbrDHx8cjNDTU4v4ZM2bgtdde81ij\niIjIdYHObrB+/Xq0b98evXv3tvu81NTUqu/1ej30er2zL0VE5NcMBgMMBoPm+61xErC8vDwkJCQg\nOzsbxcXFGDJkCLZs2YJmzZqhc+fO2L17N1q0aGG+U04CRkQexknAbHNqVMyRI0eQl5eHPn36oHPn\nzjhx4gT69++PP/74w+2GEBF5WlCQt1tQO5wqxURHR+P06dNVtzt37owffvgBV155peYNIyIi19jt\nsScmJmLw4MHIzc1Fhw4dsGLFCrPHda5+FiIiIo/hQhtEVCe50q8MCgJKS7Vvi1a40AYREVnFYCci\n8jMMdiIiP8NgJyLyMwx2IiI/w2AnIvIzDHYiIj/j9CRgRETewmsiHcMeOxGRn2GwExH5GQY7EZGf\nYbATEfkZBjsRkZ9hsBMR+RkGOxGRn2GwExH5GbvBnpSUhLCwMERHR1fdN3PmTERFRaFfv36YPn06\nSkpKPN5IIiJynN1gnzhxItLT083uu/nmm7F//37s3r0bRUVFWLVqlUcbSEREzrEb7PHx8QgNDTW7\nb9iwYQgICEBAQACGDx+Obdu2ebSBRETkHLfmilm+fDkmTZpk9bHU1NSq7/V6PfR6vTsvRUTkdwwG\nAwwGg+b7rXEx67y8PCQkJCA7O9vs/nnz5iErKwtr16613CkXsyYiD6g+CdjQocDWrcbba9cCd99t\nvP3ee8DkycbbXMzajpUrV2Lz5s34+OOP3W4AEZEjrOXdc8+Z377rLvPbjzziufb4MqdLMenp6Vi4\ncCEyMjIQHBzsiTYREVmorPR2C+oOuz32xMREDB48GIcOHUKHDh2QlpaGKVOm4MKFCxg6dChiYmLw\n6KOP1lZbiageq6iwvI8VX+tqrLG7tFPW2IlIY6WlQEiI+X1bt0qdXaUo5nX46rdZYyci8iHWeuxk\nHYOdiOoEazV2FgasY7ATUZ3Ak6eOY7ATUZ3Ak6eOY7ATUZ3AHrvjGOxEVCcw2B3HYCeiOoGlGMcx\n2ImoTnC0xz5uHHDDDUDXrnL7xhs91yZfxWAnojqhosL8AqWnnzZ+ryjG3vsnnwAGA/Dzz3L7m2+A\niRNrrZk+gcFORHVCZSXQoIHxtk7neCmmvtXnGexEVCdUVgIBLiZWfavFM9iJqE6oqDAPdvbYbWOw\nE1GdYK0U4yj22ImIfJC1Hruj2GMnIvJB1XvsgOM9cfbYiYh8UPWTp+yx22Y32JOSkhAWFobo6Oiq\n+woLCzFixAiEh4dj5MiRuHDhgscbSUTkTimGPXYTEydORHp6utl9S5cuRXh4OH7++We0b98e7733\nnkcbSEQEsBTjDLvBHh8fj9DQULP7MjMzkZycjKCgICQlJWHnzp0ebSAREcCTp84IdHaDXbt2ITIy\nEgAQGRmJzMxMq89LTU2t+l6v10Ov17vUQCKq31aulLVKO3c2D3OdDmjc2LF9tGjhkaa5zWAwwGAw\naL7fGhezzsvLQ0JCArKzswEA4eHhyM3NRXBwMIqLixEVFYVff/3VfKdczJqINKIG+JYtwIQJwJEj\ncv+cOfLv55+Bbt3s76O4GDh5EoiO5mLWVsXGxiInJwcAkJOTg9jYWLcbQURkj1qCqZ55Ol3NoQ7I\nG0OHDtq3y1c5HewDBw5EWloaSkpKkJaWhkGDBnmiXUREVRo0qH8nQN1hN9gTExMxePBg5ObmokOH\nDlixYgVSUlJw7NgxdO/eHSdPnsTkyZNrq61EVE9ZC3ZnTp7WN3ZPnq5evdrq/evXr/dIY4iIrFGD\nnb12x/DKUyLyeQEB7LE7g8FORD6PNXbnMNiJyOcx2J3DYCcin6eWYkzDnaUY2xjsROTz2GN3DoOd\niHyeOvkXe+yOYbATkc/Kz5evigJs3OjdttQlTk8CRkRUW5Yska+HDwOLFgFhYUBCAhAeDtx7r3P7\natAAKC+XNwl/7+0z2InIZ/35p/ntsjJgwwbX9tWwofwrLQVCQtxvmy9jKYaIfFZBgfntkhL39te0\nqeU+/RGDnYh8VmGh+e2yMvf216yZ5T79EYOdiHyW1r1r9tiJiLxM6xBu1ozBTkTkVVqXTZo2ZSmG\niMir2GN3DYOdiHyWJ2rs7LHbsXz5cgwePBj9+/fH9OnTtWwTEREURfsQZo/djnPnzuHll1/Gli1b\nsGvXLuTm5mLz5s1at42I6rHSUrlSVEvssdsREhICRVFw/vx5lJSUoLi4GKGhoVq3jYjqsYICoEkT\nbfdZX3rsLk0pEBISgqVLl6JTp04ICgrC1KlTERcXZ/ac1NTUqu/1ej30er077SSieuK774AxY4AJ\nE4BWrYALF7Tbt6/12A0GAwwGg+b71SmK87McnzlzBrGxsdi6dStCQ0Nxzz334Mknn8Ttt98uO9Xp\n4MJuiYiwdCnw6KPA9dcDp08D2dlAo0bGx92Jls8/Bz77DFi71v12eoJW2elSKSYzMxODBg1C165d\n0aJFC9xzzz3IyMhwuzFERKdOydeTJ2VGxoYNtdu3r/XYPcWlYI+Pj8fu3btx7tw5lJWVYdOmTbj5\n5pu1bhsR1UNqsJ84YVxgQyussdvRrFkzzJ49G6NGjUJxcTFuueUWDBkyROu2EVE9pAZ7WZn2wV5f\neuwu1dhr3Clr7ETkov79gT17jN/v3m2+MIY70fLrr0B8PHDsmHtt9BSv1tiJiDxF7bEDQKDGSwHV\nlx47g52IfEZFBXDmjPG2p0ox/l5QYLATkc84exb4y1+Mt7UOdnV5PHdXYvJ1DHYi8hmnTgFXXWW8\nrXWwA/VjZAyDnYh8Rm0Ee32oszPYichnsMeuDQY7EXlERQXwxRfApk2W872kpwPnzxtvf/89cPw4\n8PvvtdNjZ7ATEblg507grruAmTOBn34yf2zKFOCHH4y3331X3gTy84EWLWTbH34AnnpKHt+2Tbt2\n9egB7Nql3f58EYOdiDyirEy+njplPrywsBA4cgS4eNF4n6JIj/38eaB5cyAuDujXD7jxRnn8+uuB\noUO1adfIkcCXX2qzL1/FYCcij1CD+9w5oLLSeH92tgS5GvyAMdgLCiTYPWnIEODgQeC33zz7Ot7E\nYCcij7h0yfi9abCrZRlbPfZmzazvT6uLiho1Am67DdiwQZv9+SIGOxF5hGlwmwb73r3WH1d77LaC\nXUujRvl3OYbBTkQeYSvYf/oJ6NrVssf+229StrFVitFyGoBbbpGROH/+qd0+fQmDnYg8wlqwV1QA\n+/YBAwZY1tgrK4FffqmdHnuTJoBeD3z9tedfyxsY7ETkEdaC/fBhoHVrWcvU2uPl5Z6vsav8uRzj\ncrAXFRXhgQceQLdu3dCjRw/897//1bJdRFTHWQvuvXuBvn3lBGb1Uox6MVLTptb3p3WwJyQAW7b4\n54RgLgf7nDlzEB4ejqysLGRlZSEqKkrLdhFRHacGd0CAMdh/+gno00eCvXoppn17CfWAWqojtGwp\nY+W3bKmd16tNLh/CrVu3YtasWQgODkZgYCCae3rwKRHVKWqwh4VZ9tiDgix77OHh9sewe2IOdX8t\nx7i0PsmJEydQWlqKlJQU5OTkYPTo0Zg2bRqCg4OrnpOamlr1vV6vh16vd7etRORj1LHqgYESvBUV\n8jUgwNgjb9fOssd+4ID5fC2VlUDHjjKlQG0aORKYN09q+1qv1uQIg8EAg8Gg+X5d+lFKS0uRm5uL\nhQsXYujQoXjkkUewZs0aTJgwoeo5psFORP5HUaSkMnQoMH++XM25bRuQliYljltuked17iwhXlQk\nwxk7dpSSy6+/GvdVWWnsydsyZgxg0nfURHg4EBEhFyuNHq3tvh1RvdM7d+5cTfbrUimma9eu6N69\nOxISEhASEoLExERs2rRJkwYRUd2g9taPHJErRnfvNob1nj3Se1+wQIY2qrcbNpSFqWNizCcBKysD\noqOB99+3/XopKTIrpNZeeEH+VVRov29vcbnGHhERgZ07d6KyshJff/01hmo1Qw8R1QlqsJ86Jd/v\n2wecPm18vKJCSjJxcUBmpvm2MTHA/v1AaancLi3VvjfuqFtvBa68EvjkE++8vie4HOyvv/46pk2b\nhn79+iE4OBhjx47Vsl1E5OPUYC8ttR7sZ87IEMb+/aW2bjp3TOPGQPfuxnljSkvtl2E8SaeTTxZz\n5pif0K3LXD5d0K1bN45dJ6rHTIP65Em5bbrkXG6unCht2lTq7FlZ5turPfmBA6UU460eOwBcd53M\n0/73vwOPP+69dmiFV54SkUtMg/3HHyXEQ0ON9+XmGi86UkNcpzM+blqi8WYpRjV/PvDyy3KSt65j\nsBORS6oHe69eMmZdlZ9vvNho4EDLOruvBXvfvsANNwCLF3u3HVpgsBORS0yDPTtbRrWYBjtg2WM3\n1aOHzOj4v/95t8Zuat484M03ZVhmXcZgJyKXmAb7xYuWPXbA2GPv1Qs4e9b8sQYNZLz77t3er7Gr\nIiJkPPtrr3m7Je5hsBORS0yDHTAP9nbt5KvaY2/YUELctMYOGHvyvlCKUf31r8Dy5TKMs65isBOR\nS9Rgb94c6NBBvqrB3qYNEBJiPqFXXJzlPuLigJ07pcfeqJHn2+yIdu2AiROBF1/0dktcx2AnIpeo\nwd6undTXAWOw63RAly7GHjtgO9i3b5cefW3N6uiIZ58F1qyRq2rrIi9Me0NEviojQ2rh6rwpP/0k\n09oGBsoJxW3bgH/9C3j7bRlBAsjcL2qwX3WVfA0Lkx646cRacXGW4R0eLvOhVy/reFvLlsDUqcC0\nacAXX/jOpwlH6RRF+8kwdTodPLBbIvKwuXOBTZsA9drDDz4AnnxSeuUHDsh92dnAtdfKdLfjx8tQ\nxwYNgBYtJKB37gR695aJv0JDgSuuMO7/0CG54tRUp04yx4yvRUZJCXDffTJq54svzMfoe4pW2elD\nH36IyNsKCyWYf/9dbpeVyQRf6m0AKC6W0P79d6BnT1nqrkULeaxhQ7mKs1kzWTjDNNQBy1AHLJ/j\nK0JCgM8/l5O+11wj67HWFQx2IqpSUCDlk6++ktvq3Cmm47pzc+XrwYMS5O7y1WAH5JPIokVSlrn2\nWuMnGV/HYCeiKoWFMo/6hg1y29qkWPv3y9dDh/w/2FWPPipTCickSC/e1zHYiahKQQFw772AwSAl\nF9N1SVX79snX+tBjN3X77XIiecYMuYDJ184JmGKwE1GVwkIZ5TJgALB1q/0ee25u/Qp2QOaT+f57\nYNUqYPJk3xvNo2KwE1GVggI58XnnnVKOsRbsR4/KVLylpdoEe5Mm7u+jNrVvL2Pvjx8H7rjDfO1W\nX8FgJ6IqhYUS2gkJMl69pMT68yIj5Wt967GrmjaVN74uXWQU0PHj3m6ROZeDvaKiAjExMUhISNCy\nPUTkRWqPvUsXoFUrYMcOy+cEBgJXXy1f62uwA/LzL1kCPPigDIc0XcPV21wO9sWLF6NHjx7QVZ/V\nh4jqLLXHDkg55scfJbxNA7xzZ1narlOn+h3sgEydMGOGXIl7yy3yKccXuBTsJ06cwMaNGzFp0iRe\nYUrkJy5eBCorjfOi33mnfG3b1jhVAAB06yaX2F99NYNdNXo08PXXwCOPAO+84+3WuDhXzBNPPIGF\nCxeiwM5Zg9TU1Krv9Xo99Hq9Ky9FRG6qqJCrJiMibD/n0CGZH6VZM+PUunFxclWpOgXvsWPyNSJC\n9tmlC4PdVFwc8N13wG23AYcPA2+9ZTlNcXUGgwEGg0Hztjgd7F999RVat26NmJgYuw0yDXYi8p5v\nvpE5xm1dNXnpkkwNkJ0tl9GrAgJk6trmzWUETGmpzA3Trx/w559SjqmocL99t98OJCe7vx9f0KmT\nhHtcHHDPPXJi1Z7qnd65c+dq0g6ng/27777Dhg0bsHHjRpSWlqKgoAATJkzARx99pEmDiEhbP/1k\nf7x1SYkE9KFD5rMxAsDDDxu/f+ABz7SvXTu5qtNf/OUvwLhxMklaTcHuKU7X2F9++WUcP34cR48e\nxaeffoobb7yRoU7kw7Ky7Pes1SGNhw6Zz59Orhs1SoLdW6cg3R7HzlExRL4tKwsoL7f9eGmpfD10\nyLcWu6jLeveWr1lZ3nl9t36NN9xwAzaoswURkc+5eBHIyXGsx37wIHvsWtHpjL12b+D7M5EfO3RI\nQsaRYM/NZbBricFORB6RlQX06lVzsF91FZCfz2DX0jXXAKdPe2eBDgY7kR/LygJiYuzX2EtKjBcd\nMdi106CBXOTljV47g53Ij2VnS7DX1GO/4gq5kpQnT7XlrXIMf41EfkztsdcU7CEhQNeu7LFr7cYb\nZf7606dr93UZ7ER+Kj9fJvW6+mrHgj0igsGutaAgmRxs/frafV0GO5Gfys4GoqNlPpeaauzssXuO\nN8oxDHaiWrRtGzBhguvbP/GETBF79901L8t24ADQo4f0GktLgfPnrT+vtFSCvVcv45S9pJ1bb5UV\nl2wtWuIJDHaiWnL4MHDXXUBGhmvbb9kiMwauXQv885/A00/bf35RkczW2KQJkJgIvPSS9eepPfb4\neGDdOtfaRrY1bSqLlpw8WXuvyWAnqgX/+5+sj/nYY0BZmfPbl5QAKSnA0KESENddJ3XbNWtsb3Pp\nknFa3fnzgRUr5CIka/sODpYLmRo1cr5tVLN27RjsRH7l0iXg3nvlJNrUqa4F+/z5Ml3ugw8CJ07I\nPOlr18obRU6O7ddVg71NG+DZZ2W1n+rUHjt5DoOdyI8oioR5w4bAokVS77540bl9HDgALFsmZZg2\nbWT7wEAJ+ldekfLOhQuW25kGOyDtyM0FNm0yfx6D3fMY7ER+5J135MTZp5/KiJNGjZzrsVdWApMn\nA6mpskRdWJjcr86bnpwsl65PmmQ5RWz1YG/UCHjjDTkBa/rmwmD3PAY7kZ/YtAlYsAD46is5iQlI\n0FZUSGA7YuVKGbUyebLcbtNGvpouiPG3v0lPvPpam+Xllgtn3H67LEa9ZInxPga759V2sLu05ikR\n2bdvn6w4tG6dLJemUk9QlpXVHKZnzgDPPQds3mwcX37llfK9aWCHhEi9/ZprgAEDgMGD5f7qPXb1\n9d98U0bA3Hef1OoZ7J7Xrh3w22+193rssRNp7I8/gIQECVA1ZE05Wmd/6ing/vuBvn2N9wUESDmm\nemBffTXwwQfAmDHy+oD1YAeAyEhZu3T2bLnNYPe8OlGKOX78OIYMGYKePXtCr9dj1apVWreLqE4q\nKwNGj5ZAvu8+689xpM7+7beAwQBYW9s4LMyyxALIcMoHHgDGjpUyjK1gB2Rx6w0bgD17jBcokee0\nbQucOuV4Cc5dLgV7w4YN8eabb2L//v1Yu3YtZs+ejcLCQq3bRlSnKArw0EMyt7m9xeaDguwHe2mp\njFl/5x25uKi6Nm2sBzsgr9uggQS3vWD/y1+AF18Epk0DiosZ7J4WHCwXKp09Wzuv51Kwt2nTBn0v\nfz5s2bIlevbsid27d2vaMKK6ZsECGVP+4Yf2p7+tKdhfeQXo2VPm8rbGVo8dkFBftQr4+GPp8dsK\ndgBISpKrU3/4gcFeG2qzHOP2ydPDhw9j//79iIuLM7s/NTW16nu9Xg+9Xu/uSzksPx/48Ue5Ss8d\nigJ88QUwcqTvTo6UlSX12gEDvN0SbaxbJ/VpLY73H38Ae/cCN9/s2vaVlTJMcdy4mp/7r38B770H\n7NwJNG5s/7nq3C3WHD4so1z27rW9vTrk0ZZWreSK1Ouvtx/sDRoAixfL84KD7e+T3KcGe0yM8T6D\nwQCDwaD9iyluKCgoUPr166esW7fO7H43d+u2FSsU5eab3d/P0qWKAijK8ePu78sTPv9cUUJCFOWh\nh7zdEm38619yvI8e1WZ/jz7q3t/BmjWKEhDg2HMfekhRlixx7Ln3368oCxZYf2zJEkWZNMn+9gcO\nKEpWVs2vs3Wropw5U/PzVq9WlLKymp9H7nn0UUV56y37z9EqO10eFXPp0iXcddddGD9+PEaMGKHd\nO40GcnLcP0mxe7fUKa3VOL1NUYB58+Ty8IkTLS9MqYsKC+Xy+EaN7M8d7qhTp2SUiL3pau2prJR6\ndWWlY+0pLXX8b+WZZ+QqUmuz/eXlAV262N8+Kkqm463JTTcBLVvW/LyxYzlHTG2IjAQOHqyd13Ip\n2BVFQXJyMnr16oXp06dr3Sa3uRvs+fkyLep778lJJl8KzpISmalv40b52B8T41vtc9ULLwBDhsjF\nM66Gsak33pAAdHVfa9dKUAcF1Tw9LmCcSMsRvXoBAwcCaWmWjx09KseA/E9UlO15fbTmUrD/5z//\nwccff4xvv/0WMTExiImJQXp6utZtc5k7wV5ZKWN877pLhq35kt9+k3pogwZyYuyqq+SCk7oe7JmZ\nUst+/XX52dwN9vx8Cc1nn3VtX2pvPTVVerKOjDl3dsjgc88BCxdavmkw2P1XbfbYXTp5et1116Gy\ntgZkOqm0FPjlFwk9V7z8spQFXnnFeJ8vBOfu3bISS0qKhIJOJ/frdLU3NtYTLl0CHn5YJshq2VJG\ne7gb7G+/LW/KnTq5tq+1a2Vo2vDhzgW7MycgBw2Si4pWrzZfeIPB7r/atZNRSP/7HxAa6tnX8rsr\nT3/+WYLOlbDbuhV4913gs8+MownUAPWmNWtkFZa33wZmzTJvU0CAb7zxuOqNN2SUhzryxN1gLyiQ\n3+Ezz8jv0JEyiinT3rp6+b8ngh2QN+hXXjH+rRYUyDBIR+riVPfodLXXa/e7YM/JkaBwNthPnJAS\nzCefyFVivqCyUgJm5kxZPWfUKMvn1OVSzJEjUo5YutT4ZuVusC9dKsMbu3Z1bV+mvXXA8WB3psau\nGjoUuOIK40LHam/dFzoT5Bm1VWf3u0nAcnLk4o6iIse3uXhRFkKYOlVO4JnyVnAWF8uiCsePy0lS\ndVa/6upqKUZRZMbCZ5+VkoTKnWAvLpb5WbZudW1fam990SJjuHqqxg7Iazz3nFzYNHKkBLvphGHk\nf6Ki2GN3iRrszoTd008DLVrIx3dfcPKknCQNCgL+7/9shzpQd3vsH38sl1dXH1TlzsnT99+XGQ57\n9ZLbzgZ79d46IOUcT5ViAAn0wkLgm29YX68PIiNrp8fut8Hu6FjoNWtkMqSPPrJ9GXhtBueuXTIU\n7u67pU01hUVdrLGfPSvlpffft7w03tUe+8WLUtZ5/nnX9lVRYV5bV3myFAPI7+/ZZ6XXzmD3f+yx\nu6CiQk6e9ujhWI/94EG5KGbtWttnqWuz3vnZZ7IQwpIl8p/dkdeui6WYGTNk5sP+/S0fczXYP/pI\nfu+mUyugyFCAAAAS7ElEQVQ4sy9rvXVAgt2RE7DuzJA4bpxMJbBhA4Pd33XpIuVVW1NKaMVjNfbf\nf7dfQgCsr/DijEuXZHs1APPyZJ6Mpk1rDruiIukVq4sEu0pR5LXszW1S03PUk6QffSQnSfv0cfz1\nHS3FXLxoeXVhRUXNc7I48hz1WNub+Eq1ZQuQkSELUVgTGGj905a19qvKy2V0yYoV5vc7OiqmokKu\n5DWtras8OSrGtJ0zZwJTpjDY/V3DhvI7PnzYWDL0BI/12PV6OZlly88/S137yy9df40HHwT+8Q/j\n7V9+kdEQV14po1zUBQes+cc/5KTdQw/Zf42WLWX2O1u2bpVeor3Z+jZvBuLibIfMt9/KBTo7dzoX\n6oC8ke3da/3ydFO9e1tOLDVihEw4Zc/UqbLggz1r1kj5yN7vW7VwoYSwrcvvr7xSRstUN2qU9Ss1\nATluTZrIqkCmmjaV41JTTXPXLvlavbcOyPUQ27fb315R3At2QNYuvfVW+fsl/9avn+WC4lrzWLD3\n6yflBFvOnJHL9adPl5EBrnz8Pn0a+Pxz4+3CQllbMjxcAvvhh233Zi9cACIiai53vPmm9KTOnLH+\n+J9/SmDOmWN7H7/9JgsavP669ceLiqSMUNOsfdbcdJNMK/DEE/afl58P/P3vlvc99RTw00+2tzt3\nTnqyGzfa3/dPP8k0sDV9ejh/3n6vNCVFjnn1v4eiIvk7KSiw3Kaw0PqnwyZN5I1kzBj7b3z5+UDH\njtb/FhYulBkQ//tf29sXF8uJbndmpAwJkWN8xRWu74PqhhdeAF591X7H010eC/YlS6Q3vmWL9ceL\niiRYd++WHtMtt9gOT1sKCmT/Fy4Y96n+x5g7V3rwH31kfVt7ixCYio+X1XAmT7YeWooiI1g+/BD4\n7jvr+ygqktr5okWy6HB1lZWu1/J1OmDZMvnkYPomV93FizJG33QYaHm5vPmNHWu7t11RATz6qPQo\nT52y/pxLlyTUjx6VK3ftqWlRh2uvBdq3l/MNpiorpTf/0kuW29jrLU+aJG+aM2bYfs3z54Hmza0/\n1rGjHN/ERHkTt+bCBd+cLI58U2SkXG1seqJfax4L9tBQ+eiclCSX0FanhnCrVlKqiI2Vkob6sdgR\n589LT23zZrldXGwM9qAgKbc89RRw7Jjlto4GOyD110OHJBirUxRZEHjJElmWzNr4+aIiGakze7YE\nafX6v6I4Vp+2pVkzKeU89pi8mVlz8aIcY9PALC+X2SH797cdfOXl8qlg0iT5+aydu7h4Udrw5Zdy\ngdC6dbbb6sj6mrNmmV+RCRhntExLk/qkqdJS+X1bo9PJJ5UtW2y/8RUUSPttGTlSlp2bNMn6m3th\noZR9iBz117/KHP579nhm/x4dFTNsmNRGH3vM8jHT3nWDBjLc6623pGf7/vuO7b+gQK4WVYOkqMh8\nkYM+fYxT21YPJGeCPThY3iRmzJDavSlFkfAYPVrm/7BWflJ/1ilTJNg++MD8cXd67KoBAyQQx461\nfrLv4kXg8cfNyzHqyed335Xg++ILy+3Uk6dz5kjP9I03LJ+jHsu2bSXcH3oIyM623s6SkpoXohg+\nXPb31VfG+yorpd791FOWNf+yMvv1bdM3vqNHLR+312NXLVwotf9lyywfY4+dnKUuTTh1qmeGK3t8\nuOMrr8i70qefmt9vGuyqUaNkxMQbb0jvqKYhQefPS7B//bWEi7V9zpwpPfklS8zvv3TJuTmoY2Lk\nl1C9jqwGOyBzuaxbJxebmLpwQdrVoIG8ac2aJXV3032402NXTZsmn2BmzTK/v6JC2njHHfLGlJUl\n96ujkpo1k+XUUlJkKJYp9TmBgfKc116T8pkp0zfJ2FipSY8YYb205kiP3fSKTPVYq8do+nRpv+kx\nduTE5YAB8qZr7Y2voKDmYA8Olk87L7xgPH6qwkIGOzkvKUmyqXrZUQseD/bGjeUqw6lTzdf7sxbC\ngNSfMjPlP1t8PPDrr9b3e+mS/AeNiJB/GRnmpRhVYKDU2efOlXKK6faO9thVzz4rbyZLlxrvMw32\n0FAJ7qQkeZ7pz6r+x4+Olnr9448bH9eixw7IPlaskFEqX39tvL+sTN7EAgOlVr58udxfXm48BgMH\nSmiOH28+3LC83HhSsFMnWWA5MVHCTFV9KOK4cXLC8u67LUPU0YWTR4+WE7fbtslt9RgFB8tJ6OnT\njSdYHR2R8sQTUvqrXts8f95+KUbVrZt0OsaMMS+5XbjAUgw5T12a8OmnnZsCxRG1coHSgAESZKYl\nEVvBDkgIfvaZ9K4GDjTO/WFKHQGj00kNdN06y1KMKiJCgn3CBGMYXLzofLCrbxJz5shwTcA82AEp\nI9x6q/koleo/6/PPAwcOGEsfWvXYARlC+sknEuDqG6lp8CYnS8+7uNjyOoKnn5afZcEC430VFebP\nGTNGThZPmWK8z9qb5Pz50gs2/ahZWSltcSSEGzSQKR7Uk7Gmx2jUKBmGqr5B2auxm9LpgJUr5dOj\n6XAzR3rsqvHj5W/S9I2ZpRhyVXw8MHiwfBLWUq1deTprlvSM3n1XbufkGOwO7dLpgCeflP+EEyZI\nSce0BGLay1KDXS15WJOSIs9X51mvqcdua4HZ7t3l4/gDD0joVQ92QHqU27bJyRHAsl3BwdKznzJF\nRlo422OvafHb+HgJnvvukzaaBnt4uJwL+Pxzy2Bv0EA+Xb3zDvD993KfaY9d9fbb8viqVXLb9E1S\nbVtAgOxrxw7jJxy1Z+3oz3r//TIGffdu82Ok08n5mNRUOTFfU41dZTAY0LKlvPFNnGh843O0x676\n29/k5//4Y7mt1clTjyxqrBG2zTWOtO211+RvylZ1whUuB3tGRgaioqIQERGBd955p8bnBwbKCcjU\nVCmJHD5sP9hVer2UZtavl1WN1HHMpiMZIiOlp759u+1gDwiQERVvvw38+KPrwQ5IaIaEyAk1a8He\npIn0DCdPlnlRrH06ue46qUM//bTzPXZH/liee04C+cUXLUslDz8svV1rV/62aycnWMeNk8Cr3mMH\n5Gf59FOp6f/yi/n5CtO2NWsmv7d58+QiLEfLMKpGjeREqVprNz1GffrIG/rcuY6XYtS2XX+9DOG8\n/375+ZzpsQPy+12zRj6V5eZq12Ov6wHlLXW9beHh8sl25kztXtflYJ82bRqWLVuGrVu3YsmSJTh7\n9myN23TrJv/Jx4+XXpajF2O0by9LwbVpIyfnDhwwH8mglmMOH7Y/4qJDBxlLPn68/Gd0thSjCgiQ\nWvaiRXJhjrUeaHy81KIfe8x22WnBAikJfPut9nPSqL3vZcuAf//bPNhvv11Gh5w9a31KhxEjpJw0\nebL1HjsgJ5NnzZI3gOJi28eySxfp2ScmyjQCzs6nMmmS9Prz8iyP0Ysvys+4d6/zV30+/7zsb/58\n53vsgFzJO2+elKbOnmWNndwzc6ZcQa0Vl4L9/OUzg9dffz06duyIm2++GTsdbFVKilxosm+fc1fZ\nBQVJGWfWLOnFf/SR+X/GkSPla037vP9+Kaf885+uBzsg77Kvvy5XSdoK5ZdekmF/e/ZYb1fz5jJa\nJy3NM5ONXXWVfHKYOtU82AMD5QSvtd64atEiY9ttXVE5bZr8Lj/91P6xvPFGOS8xZozzwa4OE83P\ntzxGrVvL38OmTY7V2E2pb3xLl8onSGd67KrJk+WN6623WGMn9zRuLBUAregUxflRlFu3bsUHH3yA\n1atXAwDee+89nDx5Ei+++KLslEvAEBG5xIVItuCR2R21aBgREbnGpVJMbGwsDprMFr9//34MGjRI\ns0YREZHrXAr25pcLkhkZGcjLy8OWLVswcOBATRtGRESucbkU89Zbb+GRRx7BpUuXMHXqVLRs2VLL\ndhERkYsc7rFXH7d+ww034O6778YVV1yBtLQ0jB8/Hvn5+Q5tq5o5cyaioqLQr18/TJ8+HSU2Js22\ntX1hYSFGjBiB1q1bo2nTpujatWvV4y+88AL69OmDvn37utQ2d7dX2xYeHo7rrrsO3bt3N3uOu/tX\nLVq0CAEBATh37pzPHLvU1FS0b98eMTExiImJQXp6us+0DQBWrFiBqKgo9OzZE8/YWMHcG20bO3Zs\n1THr3LkzYmJiXGqbp/7mDhw4gDvuuAN9+/ZFQkICcmysYOKNY3fo0CHcd9996NGjB8aOHeuVLElK\nSkJYWBiio6PN7jf9vYwcORIX1HnGnWxbTdubURzUt29fZdu2bUpeXp7SvXt35cyZM0pBQUHV43Pn\nzlVeeOEFh7Y9e/asoiiK8u9//1upqKhQKioqlEmTJinvv/++U9u/+uqryuOPP6706dNHGTVqlDJr\n1iy323bmzBlFURS3fza1baWlpUrLli2VlJQUTY6d2j5FUZRjx44pw4cPVzp16qTk5+d7/dip+05N\nTVUWLVpkdRtvtU09btnZ2cqgQYOU3NxcRVEU5Y8//vCZtpl68sknlRdffNGltmn9N6fuf8yYMcpn\nn32mKIqirFq1Shk7dqzXj52678TERGXNmjWKoijKggULlLfffrvW25aRkaHs2bNH6dWrl9n9pr+X\nxx57TFm4cKFLbatpe1MO9dhtjVtvevmqjPLychQVFSHYylUi1rb97+XlaIYNG4aAgAAEBARg+PDh\n2KbO+OTg9pmZmbj33nuh0+kwe/Zs5Obmut02dTy+uz9bZmYmkpOTUVpaihYtWuDMmTOaHDvT6wVm\nzJiB1+xMMlHbx+6/JssMKTWMjPLW73XTpk1ITk5GREQEAKBVq1Y+0zbTY7dmzRokJiY63TZP/M2p\n+2/evDny8/NRWVmJ/Px8hFpZAd5bf3MGgwEJCQkAgDvvvBP/+c9/arVtABAfH2/1mKi/l6CgICQl\nJVm95seR36u97atzKNh37dqFyMjIqts9evSoetHnn38ebdq0wY4dO/DU5Ymyf/vtN9x+++01bmtq\n+fLlVb8YR7fftWsXCgsLERkZicjISGRmZmraNne2Vx/btWsXevfujczMTE2P3fr169G+fXv07t3b\n7Dj6yrF75513MGjQILz66qsovDwVpC+0bfPmzdi3bx8GDBiASZMm4cCBAz7TNtX27dsRFhaGLl26\nON02T/7NLVy4EIsXL0ZoaCiWLFmCV1991WeO3bBhw7By5UqUlZXhww8/xHeXlzOrrbbZY/q66r6d\nbZu17e1xexKw+fPn49ixY4iLi6uqV7Zt2xZfm84bW4N58+ahadOmuOeeexzaXr0AqqZeobttc2V7\nR9vmzv5LSkrw8ssvY+7cuVX3q6/nC8cuJSUFR48exebNm3HkyBEsu7w6hTfbpu67rKwM586dw/bt\n2zFixAg8fnmaRl84bqrVq1dj3LhxVbd94W8OkBrylClTkJ+fj8mTJyM5OVnT9rlz7ObOnYt9+/Zh\n0KBBqKioQMjlS5x9oW229q3l77U6h4K9pnHrjRs3RlJSEr5XpwR0YtuVK1di8+bN+FidKs+B7dWh\nlbGxsWjSpAkOHjyInJwcxMbGato2V7Y3bZvapqysLMTGxlp9DVf2f+TIEeTl5aFPnz7o3LkzTpw4\ngf79++OPaqvjeuvYtW7dGjqdDs2bN8djjz2GL7/80uljp3Xb1H0PGjQIY8aMQUhICBISEnDw4EGU\nVlvRxZt/c+Xl5fjyyy8xZswYi20daZun/uYAYMeOHUhKSkJgYCCSk5ORkZHhdPs8dew6deqEv/3t\nb/jxxx9x0003Yfjw4bXaNnvU3wuAqn0727aatq/OoWC3NW7958uTkpeXl2P16tUYPXq0w9sCQHp6\nOhYuXIgNGzbYrFvZ237gwIH4/PPPoSgKXnrpJXTr1k2ztrm7/cCBA5GWloZGjRohPz8frVu31uzY\n9erVC6dPn8bRo0dx9OhRtG/fHnv27EHr1q194tidurzqdXl5OVatWoXbbrvN6WPnqbZdc8012LRp\nExRFwc6dO9GlSxeLvz1vtQ2Q6TqioqLQtm1bi20daZun/uYAYMiQIdiwYQMAKQUOGzbM6fZ56tid\nubxc18mTJ/Huu+9aDXZPts0e9fdSUlKCtLQ0qxdzOvJ7tbe9hRpPr15mMBiUyMhIpUuXLsrixYsV\nRVGUu+66S+nVq5cSGxurzJw5Uzl37pyiKIpy8uRJ5bbbbrO7raIoSteuXZXw8HClb9++St++fZWU\nlBSnti8oKFDuvPNOpVWrVkqTJk2Uq6++WrO2ubu92rYOHToo1157rdKtWzdNj52pzp07V42K8YVj\nN378eCU6Olrp37+/8sQTT/hU28rLy5VHHnlEiYyMVEaOHKlkZmb6TNsURVEefPBBZdmyZWb3+crf\n3L59+5SxY8cqvXv3VsaNG6fk5OT4zLFbvHix0q1bNyUiIkKZP3++y8fOnbaNHTtWueqqq5RGjRop\n7du3V9LS0ix+LyNGjFAKCwtd/r2abm+PS5OAERGR76q1FZSIiKh2MNiJiPwMg52IyM8w2ImI/AyD\nnYjIzzDYiYj8zP8DtDOLhZZTxfEAAAAASUVORK5CYII=\n" } ], "prompt_number": 31 }, { "cell_type": "code", "collapsed": false, "input": [ "from collections import Counter\n", "\n", "file_counter = Counter()\n", "byte_counter = Counter()\n", "\n", "result = cloud.result([jid for (seg_id, jid, status) in tally])\n", "\n", "for r in result:\n", " file_counter.update(r['count'])\n", " byte_counter.update(r['size'])\n", " \n", "file_counter, byte_counter" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 32, "text": [ "(Counter({'arc.gz': 856589, 'textData': 341525, 'metadata': 341517, 'success': 71, 'other': 17}),\n", " Counter({'arc.gz': 71106384571350, 'metadata': 11010558690874, 'textData': 6978342039325, 'other': 1626219, 'success': 0}))" ] } ], "prompt_number": 32 }, { "cell_type": "code", "collapsed": false, "input": [ "jobs_info" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 33, "text": [ "{319L: {'cputime.system': 2.25,\n", " 'cputime.user': 6.4,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 1, 59),\n", " 'runtime': 13.1426},\n", " 320L: {'cputime.system': 0.625,\n", " 'cputime.user': 11.200000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 2, 24),\n", " 'runtime': 24.1863},\n", " 321L: {'cputime.system': 1.925,\n", " 'cputime.user': 3.5,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 2, 31),\n", " 'runtime': 5.93496},\n", " 322L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 2, 35),\n", " 'runtime': 3.2074},\n", " 323L: {'cputime.system': 0,\n", " 'cputime.user': 6.4,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 2, 48),\n", " 'runtime': 13.1659},\n", " 324L: {'cputime.system': 0.32499999999999973,\n", " 'cputime.user': 7.350000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 3, 3),\n", " 'runtime': 14.6188},\n", " 325L: {'cputime.system': 0,\n", " 'cputime.user': 4.474999999999998,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 3, 16),\n", " 'runtime': 13.0314},\n", " 326L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 5.449999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 3, 29),\n", " 'runtime': 12.9266},\n", " 327L: {'cputime.system': 0,\n", " 'cputime.user': 4.800000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 3, 42),\n", " 'runtime': 12.1657},\n", " 328L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 10.25,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 4, 5),\n", " 'runtime': 22.8888},\n", " 329L: {'cputime.system': 0.2999999999999998,\n", " 'cputime.user': 5.099999999999994,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 4, 19),\n", " 'runtime': 14.095},\n", " 330L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 3.200000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 4, 27),\n", " 'runtime': 7.54293},\n", " 331L: {'cputime.system': 0,\n", " 'cputime.user': 5.125,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 4, 41),\n", " 'runtime': 13.6969},\n", " 332L: {'cputime.system': 0,\n", " 'cputime.user': 5.125,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 4, 55),\n", " 'runtime': 13.7375},\n", " 333L: {'cputime.system': 0.6499999999999995,\n", " 'cputime.user': 5.125,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 5, 11),\n", " 'runtime': 15.6019},\n", " 334L: {'cputime.system': 0,\n", " 'cputime.user': 5.424999999999997,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 5, 24),\n", " 'runtime': 12.3113},\n", " 335L: {'cputime.system': 0,\n", " 'cputime.user': 4.5,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 5, 33),\n", " 'runtime': 9.28734},\n", " 336L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 3.825000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 5, 44),\n", " 'runtime': 10.8933},\n", " 337L: {'cputime.system': 0,\n", " 'cputime.user': 3.8499999999999943,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 5, 54),\n", " 'runtime': 9.42668},\n", " 338L: {'cputime.system': 0,\n", " 'cputime.user': 5.75,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 6, 8),\n", " 'runtime': 14.3095},\n", " 339L: {'cputime.system': 0.625,\n", " 'cputime.user': 3.8500000000000085,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 6, 19),\n", " 'runtime': 9.92344},\n", " 340L: {'cputime.system': 0,\n", " 'cputime.user': 6.075000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 6, 33),\n", " 'runtime': 14.4308},\n", " 341L: {'cputime.system': 1.9249999999999998,\n", " 'cputime.user': 9.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7),\n", " 'runtime': 25.4636},\n", " 342L: {'cputime.system': 0,\n", " 'cputime.user': 7.375,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 6, 51),\n", " 'runtime': 17.0747},\n", " 343L: {'cputime.system': 0,\n", " 'cputime.user': 9.274999999999991,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 16),\n", " 'runtime': 25.5032},\n", " 344L: {'cputime.system': 1.6,\n", " 'cputime.user': 3.825,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 11),\n", " 'runtime': 68.5595},\n", " 345L: {'cputime.system': 1.9249999999999998,\n", " 'cputime.user': 7.675,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 35),\n", " 'runtime': 20.742},\n", " 346L: {'cputime.system': 0,\n", " 'cputime.user': 5.75,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 31),\n", " 'runtime': 14.6856},\n", " 347L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 8.325000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 50),\n", " 'runtime': 18.6423},\n", " 348L: {'cputime.system': 0,\n", " 'cputime.user': 4.4750000000000005,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 46),\n", " 'runtime': 11.2397},\n", " 349L: {'cputime.system': 1.9249999999999998,\n", " 'cputime.user': 5.1,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 59),\n", " 'runtime': 11.6796},\n", " 350L: {'cputime.system': 0.6500000000000004,\n", " 'cputime.user': 4.15,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 7, 57),\n", " 'runtime': 10.263},\n", " 351L: {'cputime.system': 2.25,\n", " 'cputime.user': 5.425,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 2),\n", " 'runtime': 10.7821},\n", " 352L: {'cputime.system': 0,\n", " 'cputime.user': 4.800000000000011,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 1),\n", " 'runtime': 10.786},\n", " 353L: {'cputime.system': 0.625,\n", " 'cputime.user': 5.449999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 9),\n", " 'runtime': 11.5266},\n", " 354L: {'cputime.system': 0,\n", " 'cputime.user': 2.25,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 5),\n", " 'runtime': 5.7994},\n", " 355L: {'cputime.system': 0,\n", " 'cputime.user': 3.5249999999999773,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 8),\n", " 'runtime': 6.19568},\n", " 356L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 6.3999999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 16),\n", " 'runtime': 13.7778},\n", " 357L: {'cputime.system': 1.6,\n", " 'cputime.user': 4.15,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 24),\n", " 'runtime': 69.7021},\n", " 358L: {'cputime.system': 0,\n", " 'cputime.user': 5.125,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 17),\n", " 'runtime': 11.5543},\n", " 359L: {'cputime.system': 0,\n", " 'cputime.user': 5.125,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 20),\n", " 'runtime': 11.5575},\n", " 360L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 5.75,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 21),\n", " 'runtime': 12.2515},\n", " 361L: {'cputime.system': 1.9249999999999998,\n", " 'cputime.user': 10.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 34),\n", " 'runtime': 83.3468},\n", " 362L: {'cputime.system': 0.2999999999999998,\n", " 'cputime.user': 5.449999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 27),\n", " 'runtime': 10.4084},\n", " 363L: {'cputime.system': 0,\n", " 'cputime.user': 2.2249999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 23),\n", " 'runtime': 5.62689},\n", " 364L: {'cputime.system': 0.6500000000000004,\n", " 'cputime.user': 4.475000000000023,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 31),\n", " 'runtime': 10.8136},\n", " 365L: {'cputime.system': 0,\n", " 'cputime.user': 5.450000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 35),\n", " 'runtime': 14.1835},\n", " 366L: {'cputime.system': 0,\n", " 'cputime.user': 7.050000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 39),\n", " 'runtime': 16.1453},\n", " 367L: {'cputime.system': 0,\n", " 'cputime.user': 8.325000000000003,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 43),\n", " 'runtime': 15.9796},\n", " 368L: {'cputime.system': 0,\n", " 'cputime.user': 1.5999999999999943,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 36),\n", " 'runtime': 5.46459},\n", " 369L: {'cputime.system': 1.6,\n", " 'cputime.user': 4.475,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 48),\n", " 'runtime': 11.1548},\n", " 370L: {'cputime.system': 0.3249999999999993,\n", " 'cputime.user': 6.724999999999994,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 50),\n", " 'runtime': 13.8263},\n", " 371L: {'cputime.system': 0,\n", " 'cputime.user': 7.674999999999983,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 53),\n", " 'runtime': 16.2544},\n", " 372L: {'cputime.system': 0,\n", " 'cputime.user': 7.349999999999998,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 14.9538},\n", " 373L: {'cputime.system': 1.275,\n", " 'cputime.user': 4.15,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 51),\n", " 'runtime': 9.09589},\n", " 374L: {'cputime.system': 0,\n", " 'cputime.user': 4.800000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 50),\n", " 'runtime': 9.93375},\n", " 375L: {'cputime.system': 1.6,\n", " 'cputime.user': 1.6,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 10, 2),\n", " 'runtime': 72.9493},\n", " 376L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 45),\n", " 'runtime': 1.83439},\n", " 377L: {'cputime.system': 1.5999999999999999,\n", " 'cputime.user': 1.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 49),\n", " 'runtime': 2.75356},\n", " 378L: {'cputime.system': 1.3000000000000003,\n", " 'cputime.user': 2.55,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 51),\n", " 'runtime': 3.27721},\n", " 379L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.9499999999999993,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 47),\n", " 'runtime': 1.76929},\n", " 380L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 49),\n", " 'runtime': 1.75147},\n", " 381L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 57),\n", " 'runtime': 1.89922},\n", " 382L: {'cputime.system': 1.5999999999999999,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 54),\n", " 'runtime': 2.92445},\n", " 383L: {'cputime.system': 1.5999999999999999,\n", " 'cputime.user': 1.6,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 55),\n", " 'runtime': 3.24286},\n", " 384L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.9750000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 51),\n", " 'runtime': 1.32308},\n", " 385L: {'cputime.system': 2.25,\n", " 'cputime.user': 1.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 54),\n", " 'runtime': 2.90845},\n", " 386L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 52),\n", " 'runtime': 1.84549},\n", " 387L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 52),\n", " 'runtime': 1.88334},\n", " 388L: {'cputime.system': 1.9250000000000003,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 55),\n", " 'runtime': 3.2299},\n", " 389L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000028,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 53),\n", " 'runtime': 1.96082},\n", " 390L: {'cputime.system': 0.2999999999999998,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 53),\n", " 'runtime': 1.7653},\n", " 391L: {'cputime.system': 0.32499999999999973,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 53),\n", " 'runtime': 1.54679},\n", " 392L: {'cputime.system': 1.275,\n", " 'cputime.user': 1.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 10, 7),\n", " 'runtime': 74.0247},\n", " 393L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 54),\n", " 'runtime': 1.82901},\n", " 394L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000057,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 55),\n", " 'runtime': 1.81283},\n", " 395L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 55),\n", " 'runtime': 1.59021},\n", " 396L: {'cputime.system': 1.925,\n", " 'cputime.user': 2.575,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 58),\n", " 'runtime': 3.13537},\n", " 397L: {'cputime.system': 0.9500000000000002,\n", " 'cputime.user': 1.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 59),\n", " 'runtime': 2.8934},\n", " 398L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 56),\n", " 'runtime': 1.70866},\n", " 399L: {'cputime.system': 0,\n", " 'cputime.user': 0.32499999999999973,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 56),\n", " 'runtime': 62.2221},\n", " 400L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 56),\n", " 'runtime': 1.8694},\n", " 401L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 57),\n", " 'runtime': 1.82157},\n", " 402L: {'cputime.system': 1.6,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 59),\n", " 'runtime': 2.67548},\n", " 403L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 57),\n", " 'runtime': 1.97104},\n", " 404L: {'cputime.system': 0,\n", " 'cputime.user': 0.950000000000017,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 57),\n", " 'runtime': 1.80539},\n", " 405L: {'cputime.system': 2.225,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9),\n", " 'runtime': 3.00357},\n", " 406L: {'cputime.system': 0,\n", " 'cputime.user': 1.6000000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 58),\n", " 'runtime': 1.8707},\n", " 407L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999979,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 58),\n", " 'runtime': 1.71175},\n", " 408L: {'cputime.system': 0,\n", " 'cputime.user': 0.32499999999999973,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 59),\n", " 'runtime': 1.68704},\n", " 409L: {'cputime.system': 1.5999999999999999,\n", " 'cputime.user': 1.9,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 2),\n", " 'runtime': 2.92198},\n", " 410L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999773,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 59),\n", " 'runtime': 1.90711},\n", " 411L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000005,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 8, 59),\n", " 'runtime': 1.51018},\n", " 412L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9),\n", " 'runtime': 1.86641},\n", " 413L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9),\n", " 'runtime': 1.76413},\n", " 414L: {'cputime.system': 0,\n", " 'cputime.user': 1.2750000000000021,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9),\n", " 'runtime': 1.90701},\n", " 415L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 1),\n", " 'runtime': 1.91548},\n", " 416L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 1),\n", " 'runtime': 1.94664},\n", " 417L: {'cputime.system': 1.5999999999999999,\n", " 'cputime.user': 2.55,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 2.99209},\n", " 418L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 1),\n", " 'runtime': 1.60673},\n", " 419L: {'cputime.system': 0,\n", " 'cputime.user': 1.3000000000000114,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 1),\n", " 'runtime': 1.75869},\n", " 420L: {'cputime.system': 0.32499999999999973,\n", " 'cputime.user': 0.9750000000000001,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 1),\n", " 'runtime': 1.71851},\n", " 421L: {'cputime.system': 0,\n", " 'cputime.user': 1.5999999999999943,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 2),\n", " 'runtime': 1.77275},\n", " 422L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.6500000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 2),\n", " 'runtime': 1.66671},\n", " 423L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 2),\n", " 'runtime': 1.74193},\n", " 424L: {'cputime.system': 0,\n", " 'cputime.user': 1.2750000000000021,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 2),\n", " 'runtime': 1.66118},\n", " 425L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.6499999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 3),\n", " 'runtime': 1.65309},\n", " 426L: {'cputime.system': 1.6,\n", " 'cputime.user': 1.6,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 3.05715},\n", " 427L: {'cputime.system': 0.2999999999999998,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 3),\n", " 'runtime': 1.60067},\n", " 428L: {'cputime.system': 0,\n", " 'cputime.user': 1.2999999999999998,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 3),\n", " 'runtime': 1.44499},\n", " 429L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 3),\n", " 'runtime': 1.68051},\n", " 430L: {'cputime.system': 0,\n", " 'cputime.user': 0.32500000000000284,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 1.58001},\n", " 431L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 1.68855},\n", " 432L: {'cputime.system': 0,\n", " 'cputime.user': 1.2750000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 1.67802},\n", " 433L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 10, 4),\n", " 'runtime': 61.6087},\n", " 434L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 1.46981},\n", " 435L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 5),\n", " 'runtime': 1.8635},\n", " 436L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 4),\n", " 'runtime': 1.40582},\n", " 437L: {'cputime.system': 0,\n", " 'cputime.user': 0.32499999999998863,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 5),\n", " 'runtime': 1.82435},\n", " 438L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 5),\n", " 'runtime': 1.70683},\n", " 439L: {'cputime.system': 0,\n", " 'cputime.user': 0.32500000000000284,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 5),\n", " 'runtime': 1.39514},\n", " 440L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 1.75967},\n", " 441L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000005,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 1.65567},\n", " 442L: {'cputime.system': 0,\n", " 'cputime.user': 0.9499999999999993,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 1.79096},\n", " 443L: {'cputime.system': 0,\n", " 'cputime.user': 0.9499999999999957,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 1.79765},\n", " 444L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 6),\n", " 'runtime': 1.52851},\n", " 445L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000011,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 7),\n", " 'runtime': 1.63952},\n", " 446L: {'cputime.system': 0,\n", " 'cputime.user': 0.950000000000017,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 7),\n", " 'runtime': 1.44924},\n", " 447L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 1.2750000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 7),\n", " 'runtime': 1.7404},\n", " 448L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 7),\n", " 'runtime': 1.46547},\n", " 449L: {'cputime.system': 0,\n", " 'cputime.user': 0.2999999999999998,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 1.60016},\n", " 450L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 1.64608},\n", " 451L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 1.67322},\n", " 452L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 10, 18),\n", " 'runtime': 71.9488},\n", " 453L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000057,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 1.49604},\n", " 454L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 8),\n", " 'runtime': 1.41834},\n", " 455L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 9),\n", " 'runtime': 1.78469},\n", " 456L: {'cputime.system': 1.2750000000000001,\n", " 'cputime.user': 2.55,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 12),\n", " 'runtime': 3.06034},\n", " 457L: {'cputime.system': 0,\n", " 'cputime.user': 1.299999999999983,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 9),\n", " 'runtime': 1.59818},\n", " 458L: {'cputime.system': 0,\n", " 'cputime.user': 0.9499999999999957,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 9),\n", " 'runtime': 1.67393},\n", " 459L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 9),\n", " 'runtime': 1.54002},\n", " 460L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000057,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 10),\n", " 'runtime': 1.76928},\n", " 461L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 10),\n", " 'runtime': 1.8821},\n", " 462L: {'cputime.system': 0,\n", " 'cputime.user': 1.2750000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 10),\n", " 'runtime': 1.57101},\n", " 463L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 10),\n", " 'runtime': 1.85145},\n", " 464L: {'cputime.system': 0,\n", " 'cputime.user': 0.9499999999999957,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 10),\n", " 'runtime': 1.57632},\n", " 465L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 11),\n", " 'runtime': 1.60546},\n", " 466L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 11),\n", " 'runtime': 1.47648},\n", " 467L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 11),\n", " 'runtime': 1.71701},\n", " 468L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 11),\n", " 'runtime': 1.49436},\n", " 469L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 12),\n", " 'runtime': 1.66813},\n", " 470L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 12),\n", " 'runtime': 1.70157},\n", " 471L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999999,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 12),\n", " 'runtime': 1.50946},\n", " 472L: {'cputime.system': 0,\n", " 'cputime.user': 0.9749999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 12),\n", " 'runtime': 1.78406},\n", " 473L: {'cputime.system': 0,\n", " 'cputime.user': 0.6499999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 13),\n", " 'runtime': 1.98973},\n", " 474L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000227,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 13),\n", " 'runtime': 1.41618},\n", " 475L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000011,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 13),\n", " 'runtime': 1.61929},\n", " 476L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 13),\n", " 'runtime': 1.43119},\n", " 477L: {'cputime.system': 0,\n", " 'cputime.user': 1.2999999999999998,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.567},\n", " 478L: {'cputime.system': 0,\n", " 'cputime.user': 0.9499999999999993,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.6739},\n", " 479L: {'cputime.system': 0,\n", " 'cputime.user': 1.2749999999999986,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.79159},\n", " 480L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.6499999999999995,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.70314},\n", " 481L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000002,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.70992},\n", " 482L: {'cputime.system': 0.2999999999999998,\n", " 'cputime.user': 1.5999999999999996,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.76887},\n", " 483L: {'cputime.system': 2.225,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 18),\n", " 'runtime': 3.32801},\n", " 484L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.6500000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 14),\n", " 'runtime': 1.6893},\n", " 485L: {'cputime.system': 0.3250000000000002,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 15),\n", " 'runtime': 1.56225},\n", " 486L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 15),\n", " 'runtime': 1.71709},\n", " 487L: {'cputime.system': 0,\n", " 'cputime.user': 0.6500000000000004,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.89705},\n", " 488L: {'cputime.system': 0,\n", " 'cputime.user': 0.625,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 15),\n", " 'runtime': 1.64788},\n", " 489L: {'cputime.system': 0,\n", " 'cputime.user': 1.3000000000000043,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.59969},\n", " 490L: {'cputime.system': 0,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.64547},\n", " 491L: {'cputime.system': 0,\n", " 'cputime.user': 0.32499999999999973,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.51186},\n", " 492L: {'cputime.system': 0.3249999999999993,\n", " 'cputime.user': 0.9750000000000014,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.69827},\n", " 493L: {'cputime.system': 0.6500000000000004,\n", " 'cputime.user': 0.9500000000000011,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 1.68349},\n", " 494L: {'cputime.system': 0,\n", " 'cputime.user': 0.9500000000000011,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 17),\n", " 'runtime': 1.74946},\n", " 495L: {'cputime.system': 0,\n", " 'cputime.user': 0,\n", " 'created': datetime.datetime(2013, 4, 4, 3, 1, 44),\n", " 'finished': datetime.datetime(2013, 4, 4, 3, 9, 16),\n", " 'runtime': 0.195143},\n", " 496L: {'cputime.system': 2.55,\n", " 'cputime.user': 11.2,\n", " 'created': datetime.datetime(2013, 4, 4, 5, 21, 18),\n", " 'finished': datetime.datetime(2013, 4, 4, 5, 22, 39),\n", " 'runtime': 42.0444},\n", " 497L: {'cputime.system': 1.9249999999999998,\n", " 'cputime.user': 2.225,\n", " 'created': datetime.datetime(2013, 4, 4, 5, 21, 18),\n", " 'finished': datetime.datetime(2013, 4, 4, 5, 22, 46),\n", " 'runtime': 4.60588}}" ] } ], "prompt_number": 33 }, { "cell_type": "code", "collapsed": false, "input": [ "from collections import Counter\n", "jobs_counter= Counter()\n", "[jobs_counter.update(dict([(k, v[k]) for k in ('cputime.system', 'cputime.user', 'runtime')])) for v in jobs_info.values()]\n", "jobs_counter\n", "\n", "#print (jobs_counter['cputime.user'] + jobs_counter['cputime.system']), (jobs_counter['cputime.user'] + jobs_counter['cputime.system'])/3600. * 0.05\n", "print jobs_counter['runtime'], (jobs_counter['runtime'])/3600. * 0.05" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "1509.430293 0.020964309625\n" ] } ], "prompt_number": 34 }, { "cell_type": "code", "collapsed": false, "input": [ "# maybe use pickle to serialize results\n", "import pickle\n", "s = pickle.loads(pickle.dumps(dict(zip([seg_id for (seg_id, jid, status) in tally], result))))" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 35 }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "picloud job infos" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# http://docs.picloud.com/moduledoc.html#module-cloud\n", "\n", "jobs_info = cloud.info(list(islice(chain(segment_jids, retries_jids),None)),\n", " info_requested=['created', 'finished', 'runtime', 'cputime']\n", " )" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 36 }, { "cell_type": "code", "collapsed": false, "input": [ "from matplotlib import pyplot as plt" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 37 }, { "cell_type": "code", "collapsed": false, "input": [ "started = [{'jid':k, 'time':v['finished'] - datetime.timedelta(seconds=v['runtime']), 'count': 1} for (k,v) in jobs_info.items()]\n", "finished = [{'jid':k, 'time':v['finished'], 'count': -1} for (k,v) in jobs_info.items()]\n", "\n", "df = DataFrame(started + finished)\n", "\n", "exclude_n = 4\n", "\n", "plot(df.sort_index(by='time')['time'][:-exclude_n], df.sort_index(by='time')['count'].cumsum()[:-exclude_n])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 38, "text": [ "[]" ] }, { "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVOX+B/DPIAqYyyUXzAU1RUFFRQXNosbSbCOXFsXS\nEqykcsmyxeyKllmZlXXNvBZat7TMW+otxav1G9FbVzQzUFHSJJfMFG+CbAqc3x9fDzPDLMxyhhmG\nz/v18gWznDMPB/zMM9/znOfRKYqigIiI/EaAtxtARETaYrATEfkZBjsRkZ9hsBMR+RkGOxGRn2Gw\nExH5GbvBnpSUhLCwMERHR1fdd+DAAdxxxx3o27cvEhISkJOT4/FGEhGR4+wG+8SJE5Genm5237x5\n8zBhwgTs3bsX48aNw7x58zzaQCIico7dYI+Pj0doaKjZfc2bN0d+fj4qKyuRn59v8TgREXmXrqYr\nT/Py8pCQkIDs7GwAQEFBAeLi4nDq1Cm0bdsWmZmZaNq0qflOdTrPtZiIyI9pMRmA0ydPk5KSMGXK\nFOTn52Py5MlITk622Th7/+bMmVPjc7z5z5fbx7b5X9t8vX11oW2A/Kv++G23Wb/fF4+bVpwO9h07\ndiApKQmBgYFITk5GRkaGZo0hItJaUJC3W1D7nA72IUOGYMOGDQCA9evXY9iwYZo3iohIKwz2ahIT\nEzF48GAcOnQIHTp0wIoVKzB79mysW7cOffr0wcaNG/H888+79MJ6vd6l7WqLL7ePbXONL7cN8O32\n1eW2eTPYvXXcajx56tJOdTpN60VERDVRx2xUj57Jk4Flyyzv90VaZSevPCUiv8ZSDBFRHVdRYX6b\nwU5EVMddumR+m8FORFTHMdgZ7ETkZy5eNL/NYCciquOq99iDg73TDm8K9HYDiIi0lJoKhIYCHTsC\njRsDjRrJ/cuXA0lJQIMGXm1ereA4diLyC7bmHly+HHjoIfk+MxOIja29NjmL49iJiBxgmpP1pb/J\nYCeieqOy0tstqB0MdiLya6a9dAY7EZGfYbATEfkB9tiJiPwYg52IyA+wx05E5McY7JCFq8PCwhAd\nHW12/4oVKxAVFYWePXvimWee8WgDiYjcwR57NRMnTkR6errZffv27cPf//53bNiwAfv378dTTz3l\n0QYSEbnDNNirz9Xur+zOFRMfH4+8vDyz+zZt2oTk5GREREQAAFq1amV129TU1Krv9Xq9T6+ZSER1\njxrSlZX2e+JFRcbvS0s92yZnGQwGGAwGzfdb41wxeXl5SEhIQHZ2NgBg2LBh6NmzJ3bs2IG+ffti\nxowZ6NGjh/lOOVcMEXnYyJHA0aNAVpZz2/lyNHltrpjS0lKcO3cO27dvx4gRI/D444+73QgiImd9\n843zoV5fOB3sgwYNwpgxYxASEoKEhAQcPHgQpb72+YaI/F5Jibdb4LucDvZrrrkGmzZtgqIo2Llz\nJ7p06YLg+jiTPRF5VUVF/Zhb3RV2gz0xMRGDBw9Gbm4uOnTogBUrVmDEiBEoLy9Hjx498Morr+CN\nN96orbYSEZkJCfF2C3wTF9ogojpJpwNatQLOnHFuO1+OJi60QUT1XuPG3m6Bb2KwE1GdxVKMdQx2\nIqqzGOzWMdiJqM5isFvHYCeiOovBbh2DnYjqLAa7dRzuSER1kk7n2na+HE0c7khERFYx2ImI/AyD\nnYjIzzDYiYj8DIOdiMjPMNiJiPwMg52IyM8w2ImI/IzdYE9KSkJYWBiio6MtHlu0aBECAgJw7tw5\njzWOiIicZzfYJ06ciPT0dIv7jx8/ji1btqBjx44eaxgREbnGbrDHx8cjNDTU4v4ZM2bgtdde81ij\niIjIdYHObrB+/Xq0b98evXv3tvu81NTUqu/1ej30er2zL0VE5NcMBgMMBoPm+61xErC8vDwkJCQg\nOzsbxcXFGDJkCLZs2YJmzZqhc+fO2L17N1q0aGG+U04CRkQexknAbHNqVMyRI0eQl5eHPn36oHPn\nzjhx4gT69++PP/74w+2GEBF5WlCQt1tQO5wqxURHR+P06dNVtzt37owffvgBV155peYNIyIi19jt\nsScmJmLw4MHIzc1Fhw4dsGLFCrPHda5+FiIiIo/hQhtEVCe50q8MCgJKS7Vvi1a40AYREVnFYCci\n8jMMdiIiP8NgJyLyMwx2IiI/w2AnIvIzDHYiIj/j9CRgRETewmsiHcMeOxGRn2GwExH5GQY7EZGf\nYbATEfkZBjsRkZ9hsBMR+RkGOxGRn2GwExH5GbvBnpSUhLCwMERHR1fdN3PmTERFRaFfv36YPn06\nSkpKPN5IIiJynN1gnzhxItLT083uu/nmm7F//37s3r0bRUVFWLVqlUcbSEREzrEb7PHx8QgNDTW7\nb9iwYQgICEBAQACGDx+Obdu2ebSBRETkHLfmilm+fDkmTZpk9bHU1NSq7/V6PfR6vTsvRUTkdwwG\nAwwGg+b7rXEx67y8PCQkJCA7O9vs/nnz5iErKwtr16613CkXsyYiD6g+CdjQocDWrcbba9cCd99t\nvP3ee8DkycbbXMzajpUrV2Lz5s34+OOP3W4AEZEjrOXdc8+Z377rLvPbjzziufb4MqdLMenp6Vi4\ncCEyMjIQHBzsiTYREVmorPR2C+oOuz32xMREDB48GIcOHUKHDh2QlpaGKVOm4MKFCxg6dChiYmLw\n6KOP1lZbiageq6iwvI8VX+tqrLG7tFPW2IlIY6WlQEiI+X1bt0qdXaUo5nX46rdZYyci8iHWeuxk\nHYOdiOoEazV2FgasY7ATUZ3Ak6eOY7ATUZ3Ak6eOY7ATUZ3AHrvjGOxEVCcw2B3HYCeiOoGlGMcx\n2ImoTnC0xz5uHHDDDUDXrnL7xhs91yZfxWAnojqhosL8AqWnnzZ+ryjG3vsnnwAGA/Dzz3L7m2+A\niRNrrZk+gcFORHVCZSXQoIHxtk7neCmmvtXnGexEVCdUVgIBLiZWfavFM9iJqE6oqDAPdvbYbWOw\nE1GdYK0U4yj22ImIfJC1Hruj2GMnIvJB1XvsgOM9cfbYiYh8UPWTp+yx22Y32JOSkhAWFobo6Oiq\n+woLCzFixAiEh4dj5MiRuHDhgscbSUTkTimGPXYTEydORHp6utl9S5cuRXh4OH7++We0b98e7733\nnkcbSEQEsBTjDLvBHh8fj9DQULP7MjMzkZycjKCgICQlJWHnzp0ebSAREcCTp84IdHaDXbt2ITIy\nEgAQGRmJzMxMq89LTU2t+l6v10Ov17vUQCKq31aulLVKO3c2D3OdDmjc2LF9tGjhkaa5zWAwwGAw\naL7fGhezzsvLQ0JCArKzswEA4eHhyM3NRXBwMIqLixEVFYVff/3VfKdczJqINKIG+JYtwIQJwJEj\ncv+cOfLv55+Bbt3s76O4GDh5EoiO5mLWVsXGxiInJwcAkJOTg9jYWLcbQURkj1qCqZ55Ol3NoQ7I\nG0OHDtq3y1c5HewDBw5EWloaSkpKkJaWhkGDBnmiXUREVRo0qH8nQN1hN9gTExMxePBg5ObmokOH\nDlixYgVSUlJw7NgxdO/eHSdPnsTkyZNrq61EVE9ZC3ZnTp7WN3ZPnq5evdrq/evXr/dIY4iIrFGD\nnb12x/DKUyLyeQEB7LE7g8FORD6PNXbnMNiJyOcx2J3DYCcin6eWYkzDnaUY2xjsROTz2GN3DoOd\niHyeOvkXe+yOYbATkc/Kz5evigJs3OjdttQlTk8CRkRUW5Yska+HDwOLFgFhYUBCAhAeDtx7r3P7\natAAKC+XNwl/7+0z2InIZ/35p/ntsjJgwwbX9tWwofwrLQVCQtxvmy9jKYaIfFZBgfntkhL39te0\nqeU+/RGDnYh8VmGh+e2yMvf216yZ5T79EYOdiHyW1r1r9tiJiLxM6xBu1ozBTkTkVVqXTZo2ZSmG\niMir2GN3DYOdiHyWJ2rs7LHbsXz5cgwePBj9+/fH9OnTtWwTEREURfsQZo/djnPnzuHll1/Gli1b\nsGvXLuTm5mLz5s1at42I6rHSUrlSVEvssdsREhICRVFw/vx5lJSUoLi4GKGhoVq3jYjqsYICoEkT\nbfdZX3rsLk0pEBISgqVLl6JTp04ICgrC1KlTERcXZ/ac1NTUqu/1ej30er077SSieuK774AxY4AJ\nE4BWrYALF7Tbt6/12A0GAwwGg+b71SmK87McnzlzBrGxsdi6dStCQ0Nxzz334Mknn8Ttt98uO9Xp\n4MJuiYiwdCnw6KPA9dcDp08D2dlAo0bGx92Jls8/Bz77DFi71v12eoJW2elSKSYzMxODBg1C165d\n0aJFC9xzzz3IyMhwuzFERKdOydeTJ2VGxoYNtdu3r/XYPcWlYI+Pj8fu3btx7tw5lJWVYdOmTbj5\n5pu1bhsR1UNqsJ84YVxgQyussdvRrFkzzJ49G6NGjUJxcTFuueUWDBkyROu2EVE9pAZ7WZn2wV5f\neuwu1dhr3Clr7ETkov79gT17jN/v3m2+MIY70fLrr0B8PHDsmHtt9BSv1tiJiDxF7bEDQKDGSwHV\nlx47g52IfEZFBXDmjPG2p0ox/l5QYLATkc84exb4y1+Mt7UOdnV5PHdXYvJ1DHYi8hmnTgFXXWW8\nrXWwA/VjZAyDnYh8Rm0Ee32oszPYichnsMeuDQY7EXlERQXwxRfApk2W872kpwPnzxtvf/89cPw4\n8PvvtdNjZ7ATEblg507grruAmTOBn34yf2zKFOCHH4y3331X3gTy84EWLWTbH34AnnpKHt+2Tbt2\n9egB7Nql3f58EYOdiDyirEy+njplPrywsBA4cgS4eNF4n6JIj/38eaB5cyAuDujXD7jxRnn8+uuB\noUO1adfIkcCXX2qzL1/FYCcij1CD+9w5oLLSeH92tgS5GvyAMdgLCiTYPWnIEODgQeC33zz7Ot7E\nYCcij7h0yfi9abCrZRlbPfZmzazvT6uLiho1Am67DdiwQZv9+SIGOxF5hGlwmwb73r3WH1d77LaC\nXUujRvl3OYbBTkQeYSvYf/oJ6NrVssf+229StrFVitFyGoBbbpGROH/+qd0+fQmDnYg8wlqwV1QA\n+/YBAwZY1tgrK4FffqmdHnuTJoBeD3z9tedfyxsY7ETkEdaC/fBhoHVrWcvU2uPl5Z6vsav8uRzj\ncrAXFRXhgQceQLdu3dCjRw/897//1bJdRFTHWQvuvXuBvn3lBGb1Uox6MVLTptb3p3WwJyQAW7b4\n54RgLgf7nDlzEB4ejqysLGRlZSEqKkrLdhFRHacGd0CAMdh/+gno00eCvXoppn17CfWAWqojtGwp\nY+W3bKmd16tNLh/CrVu3YtasWQgODkZgYCCae3rwKRHVKWqwh4VZ9tiDgix77OHh9sewe2IOdX8t\nx7i0PsmJEydQWlqKlJQU5OTkYPTo0Zg2bRqCg4OrnpOamlr1vV6vh16vd7etRORj1LHqgYESvBUV\n8jUgwNgjb9fOssd+4ID5fC2VlUDHjjKlQG0aORKYN09q+1qv1uQIg8EAg8Gg+X5d+lFKS0uRm5uL\nhQsXYujQoXjkkUewZs0aTJgwoeo5psFORP5HUaSkMnQoMH++XM25bRuQliYljltuked17iwhXlQk\nwxk7dpSSy6+/GvdVWWnsydsyZgxg0nfURHg4EBEhFyuNHq3tvh1RvdM7d+5cTfbrUimma9eu6N69\nOxISEhASEoLExERs2rRJkwYRUd2g9taPHJErRnfvNob1nj3Se1+wQIY2qrcbNpSFqWNizCcBKysD\noqOB99+3/XopKTIrpNZeeEH+VVRov29vcbnGHhERgZ07d6KyshJff/01hmo1Qw8R1QlqsJ86Jd/v\n2wecPm18vKJCSjJxcUBmpvm2MTHA/v1AaancLi3VvjfuqFtvBa68EvjkE++8vie4HOyvv/46pk2b\nhn79+iE4OBhjx47Vsl1E5OPUYC8ttR7sZ87IEMb+/aW2bjp3TOPGQPfuxnljSkvtl2E8SaeTTxZz\n5pif0K3LXD5d0K1bN45dJ6rHTIP65Em5bbrkXG6unCht2lTq7FlZ5turPfmBA6UU460eOwBcd53M\n0/73vwOPP+69dmiFV54SkUtMg/3HHyXEQ0ON9+XmGi86UkNcpzM+blqi8WYpRjV/PvDyy3KSt65j\nsBORS6oHe69eMmZdlZ9vvNho4EDLOruvBXvfvsANNwCLF3u3HVpgsBORS0yDPTtbRrWYBjtg2WM3\n1aOHzOj4v/95t8Zuat484M03ZVhmXcZgJyKXmAb7xYuWPXbA2GPv1Qs4e9b8sQYNZLz77t3er7Gr\nIiJkPPtrr3m7Je5hsBORS0yDHTAP9nbt5KvaY2/YUELctMYOGHvyvlCKUf31r8Dy5TKMs65isBOR\nS9Rgb94c6NBBvqrB3qYNEBJiPqFXXJzlPuLigJ07pcfeqJHn2+yIdu2AiROBF1/0dktcx2AnIpeo\nwd6undTXAWOw63RAly7GHjtgO9i3b5cefW3N6uiIZ58F1qyRq2rrIi9Me0NEviojQ2rh6rwpP/0k\n09oGBsoJxW3bgH/9C3j7bRlBAsjcL2qwX3WVfA0Lkx646cRacXGW4R0eLvOhVy/reFvLlsDUqcC0\nacAXX/jOpwlH6RRF+8kwdTodPLBbIvKwuXOBTZsA9drDDz4AnnxSeuUHDsh92dnAtdfKdLfjx8tQ\nxwYNgBYtJKB37gR695aJv0JDgSuuMO7/0CG54tRUp04yx4yvRUZJCXDffTJq54svzMfoe4pW2elD\nH36IyNsKCyWYf/9dbpeVyQRf6m0AKC6W0P79d6BnT1nqrkULeaxhQ7mKs1kzWTjDNNQBy1AHLJ/j\nK0JCgM8/l5O+11wj67HWFQx2IqpSUCDlk6++ktvq3Cmm47pzc+XrwYMS5O7y1WAH5JPIokVSlrn2\nWuMnGV/HYCeiKoWFMo/6hg1y29qkWPv3y9dDh/w/2FWPPipTCickSC/e1zHYiahKQQFw772AwSAl\nF9N1SVX79snX+tBjN3X77XIiecYMuYDJ184JmGKwE1GVwkIZ5TJgALB1q/0ee25u/Qp2QOaT+f57\nYNUqYPJk3xvNo2KwE1GVggI58XnnnVKOsRbsR4/KVLylpdoEe5Mm7u+jNrVvL2Pvjx8H7rjDfO1W\nX8FgJ6IqhYUS2gkJMl69pMT68yIj5Wt967GrmjaVN74uXWQU0PHj3m6ROZeDvaKiAjExMUhISNCy\nPUTkRWqPvUsXoFUrYMcOy+cEBgJXXy1f62uwA/LzL1kCPPigDIc0XcPV21wO9sWLF6NHjx7QVZ/V\nh4jqLLXHDkg55scfJbxNA7xzZ1narlOn+h3sgEydMGOGXIl7yy3yKccXuBTsJ06cwMaNGzFp0iRe\nYUrkJy5eBCorjfOi33mnfG3b1jhVAAB06yaX2F99NYNdNXo08PXXwCOPAO+84+3WuDhXzBNPPIGF\nCxeiwM5Zg9TU1Krv9Xo99Hq9Ky9FRG6qqJCrJiMibD/n0CGZH6VZM+PUunFxclWpOgXvsWPyNSJC\n9tmlC4PdVFwc8N13wG23AYcPA2+9ZTlNcXUGgwEGg0Hztjgd7F999RVat26NmJgYuw0yDXYi8p5v\nvpE5xm1dNXnpkkwNkJ0tl9GrAgJk6trmzWUETGmpzA3Trx/w559SjqmocL99t98OJCe7vx9f0KmT\nhHtcHHDPPXJi1Z7qnd65c+dq0g6ng/27777Dhg0bsHHjRpSWlqKgoAATJkzARx99pEmDiEhbP/1k\nf7x1SYkE9KFD5rMxAsDDDxu/f+ABz7SvXTu5qtNf/OUvwLhxMklaTcHuKU7X2F9++WUcP34cR48e\nxaeffoobb7yRoU7kw7Ky7Pes1SGNhw6Zz59Orhs1SoLdW6cg3R7HzlExRL4tKwsoL7f9eGmpfD10\nyLcWu6jLeveWr1lZ3nl9t36NN9xwAzaoswURkc+5eBHIyXGsx37wIHvsWtHpjL12b+D7M5EfO3RI\nQsaRYM/NZbBricFORB6RlQX06lVzsF91FZCfz2DX0jXXAKdPe2eBDgY7kR/LygJiYuzX2EtKjBcd\nMdi106CBXOTljV47g53Ij2VnS7DX1GO/4gq5kpQnT7XlrXIMf41EfkztsdcU7CEhQNeu7LFr7cYb\nZf7606dr93UZ7ER+Kj9fJvW6+mrHgj0igsGutaAgmRxs/frafV0GO5Gfys4GoqNlPpeaauzssXuO\nN8oxDHaiWrRtGzBhguvbP/GETBF79901L8t24ADQo4f0GktLgfPnrT+vtFSCvVcv45S9pJ1bb5UV\nl2wtWuIJDHaiWnL4MHDXXUBGhmvbb9kiMwauXQv885/A00/bf35RkczW2KQJkJgIvPSS9eepPfb4\neGDdOtfaRrY1bSqLlpw8WXuvyWAnqgX/+5+sj/nYY0BZmfPbl5QAKSnA0KESENddJ3XbNWtsb3Pp\nknFa3fnzgRUr5CIka/sODpYLmRo1cr5tVLN27RjsRH7l0iXg3nvlJNrUqa4F+/z5Ml3ugw8CJ07I\nPOlr18obRU6O7ddVg71NG+DZZ2W1n+rUHjt5DoOdyI8oioR5w4bAokVS77540bl9HDgALFsmZZg2\nbWT7wEAJ+ldekfLOhQuW25kGOyDtyM0FNm0yfx6D3fMY7ER+5J135MTZp5/KiJNGjZzrsVdWApMn\nA6mpskRdWJjcr86bnpwsl65PmmQ5RWz1YG/UCHjjDTkBa/rmwmD3PAY7kZ/YtAlYsAD46is5iQlI\n0FZUSGA7YuVKGbUyebLcbtNGvpouiPG3v0lPvPpam+Xllgtn3H67LEa9ZInxPga759V2sLu05ikR\n2bdvn6w4tG6dLJemUk9QlpXVHKZnzgDPPQds3mwcX37llfK9aWCHhEi9/ZprgAEDgMGD5f7qPXb1\n9d98U0bA3Hef1OoZ7J7Xrh3w22+193rssRNp7I8/gIQECVA1ZE05Wmd/6ing/vuBvn2N9wUESDmm\nemBffTXwwQfAmDHy+oD1YAeAyEhZu3T2bLnNYPe8OlGKOX78OIYMGYKePXtCr9dj1apVWreLqE4q\nKwNGj5ZAvu8+689xpM7+7beAwQBYW9s4LMyyxALIcMoHHgDGjpUyjK1gB2Rx6w0bgD17jBcokee0\nbQucOuV4Cc5dLgV7w4YN8eabb2L//v1Yu3YtZs+ejcLCQq3bRlSnKArw0EMyt7m9xeaDguwHe2mp\njFl/5x25uKi6Nm2sBzsgr9uggQS3vWD/y1+AF18Epk0DiosZ7J4WHCwXKp09Wzuv51Kwt2nTBn0v\nfz5s2bIlevbsid27d2vaMKK6ZsECGVP+4Yf2p7+tKdhfeQXo2VPm8rbGVo8dkFBftQr4+GPp8dsK\ndgBISpKrU3/4gcFeG2qzHOP2ydPDhw9j//79iIuLM7s/NTW16nu9Xg+9Xu/uSzksPx/48Ue5Ss8d\nigJ88QUwcqTvTo6UlSX12gEDvN0SbaxbJ/VpLY73H38Ae/cCN9/s2vaVlTJMcdy4mp/7r38B770H\n7NwJNG5s/7nq3C3WHD4so1z27rW9vTrk0ZZWreSK1Ouvtx/sDRoAixfL84KD7e+T3KcGe0yM8T6D\nwQCDwaD9iyluKCgoUPr166esW7fO7H43d+u2FSsU5eab3d/P0qWKAijK8ePu78sTPv9cUUJCFOWh\nh7zdEm38619yvI8e1WZ/jz7q3t/BmjWKEhDg2HMfekhRlixx7Ln3368oCxZYf2zJEkWZNMn+9gcO\nKEpWVs2vs3Wropw5U/PzVq9WlLKymp9H7nn0UUV56y37z9EqO10eFXPp0iXcddddGD9+PEaMGKHd\nO40GcnLcP0mxe7fUKa3VOL1NUYB58+Ty8IkTLS9MqYsKC+Xy+EaN7M8d7qhTp2SUiL3pau2prJR6\ndWWlY+0pLXX8b+WZZ+QqUmuz/eXlAV262N8+Kkqm463JTTcBLVvW/LyxYzlHTG2IjAQOHqyd13Ip\n2BVFQXJyMnr16oXp06dr3Sa3uRvs+fkyLep778lJJl8KzpISmalv40b52B8T41vtc9ULLwBDhsjF\nM66Gsak33pAAdHVfa9dKUAcF1Tw9LmCcSMsRvXoBAwcCaWmWjx09KseA/E9UlO15fbTmUrD/5z//\nwccff4xvv/0WMTExiImJQXp6utZtc5k7wV5ZKWN877pLhq35kt9+k3pogwZyYuyqq+SCk7oe7JmZ\nUst+/XX52dwN9vx8Cc1nn3VtX2pvPTVVerKOjDl3dsjgc88BCxdavmkw2P1XbfbYXTp5et1116Gy\ntgZkOqm0FPjlFwk9V7z8spQFXnnFeJ8vBOfu3bISS0qKhIJOJ/frdLU3NtYTLl0CHn5YJshq2VJG\ne7gb7G+/LW/KnTq5tq+1a2Vo2vDhzgW7MycgBw2Si4pWrzZfeIPB7r/atZNRSP/7HxAa6tnX8rsr\nT3/+WYLOlbDbuhV4913gs8+MownUAPWmNWtkFZa33wZmzTJvU0CAb7zxuOqNN2SUhzryxN1gLyiQ\n3+Ezz8jv0JEyiinT3rp6+b8ngh2QN+hXXjH+rRYUyDBIR+riVPfodLXXa/e7YM/JkaBwNthPnJAS\nzCefyFVivqCyUgJm5kxZPWfUKMvn1OVSzJEjUo5YutT4ZuVusC9dKsMbu3Z1bV+mvXXA8WB3psau\nGjoUuOIK40LHam/dFzoT5Bm1VWf3u0nAcnLk4o6iIse3uXhRFkKYOlVO4JnyVnAWF8uiCsePy0lS\ndVa/6upqKUZRZMbCZ5+VkoTKnWAvLpb5WbZudW1fam990SJjuHqqxg7Iazz3nFzYNHKkBLvphGHk\nf6Ki2GN3iRrszoTd008DLVrIx3dfcPKknCQNCgL+7/9shzpQd3vsH38sl1dXH1TlzsnT99+XGQ57\n9ZLbzgZ79d46IOUcT5ViAAn0wkLgm29YX68PIiNrp8fut8Hu6FjoNWtkMqSPPrJ9GXhtBueuXTIU\n7u67pU01hUVdrLGfPSvlpffft7w03tUe+8WLUtZ5/nnX9lVRYV5bV3myFAPI7+/ZZ6XXzmD3f+yx\nu6CiQk6e9ujhWI/94EG5KGbtWttnqWuz3vnZZ7IQwpIl8p/dkdeui6WYGTNk5sP+/S0fczXYP/pI\nfu+mUyugyFCAAAAS7ElEQVQ4sy9rvXVAgt2RE7DuzJA4bpxMJbBhA4Pd33XpIuVVW1NKaMVjNfbf\nf7dfQgCsr/DijEuXZHs1APPyZJ6Mpk1rDruiIukVq4sEu0pR5LXszW1S03PUk6QffSQnSfv0cfz1\nHS3FXLxoeXVhRUXNc7I48hz1WNub+Eq1ZQuQkSELUVgTGGj905a19qvKy2V0yYoV5vc7OiqmokKu\n5DWtras8OSrGtJ0zZwJTpjDY/V3DhvI7PnzYWDL0BI/12PV6OZlly88/S137yy9df40HHwT+8Q/j\n7V9+kdEQV14po1zUBQes+cc/5KTdQw/Zf42WLWX2O1u2bpVeor3Z+jZvBuLibIfMt9/KBTo7dzoX\n6oC8ke3da/3ydFO9e1tOLDVihEw4Zc/UqbLggz1r1kj5yN7vW7VwoYSwrcvvr7xSRstUN2qU9Ss1\nATluTZrIqkCmmjaV41JTTXPXLvlavbcOyPUQ27fb315R3At2QNYuvfVW+fsl/9avn+WC4lrzWLD3\n6yflBFvOnJHL9adPl5EBrnz8Pn0a+Pxz4+3CQllbMjxcAvvhh233Zi9cACIiai53vPmm9KTOnLH+\n+J9/SmDOmWN7H7/9JgsavP669ceLiqSMUNOsfdbcdJNMK/DEE/afl58P/P3vlvc99RTw00+2tzt3\nTnqyGzfa3/dPP8k0sDV9ejh/3n6vNCVFjnn1v4eiIvk7KSiw3Kaw0PqnwyZN5I1kzBj7b3z5+UDH\njtb/FhYulBkQ//tf29sXF8uJbndmpAwJkWN8xRWu74PqhhdeAF591X7H010eC/YlS6Q3vmWL9ceL\niiRYd++WHtMtt9gOT1sKCmT/Fy4Y96n+x5g7V3rwH31kfVt7ixCYio+X1XAmT7YeWooiI1g+/BD4\n7jvr+ygqktr5okWy6HB1lZWu1/J1OmDZMvnkYPomV93FizJG33QYaHm5vPmNHWu7t11RATz6qPQo\nT52y/pxLlyTUjx6VK3ftqWlRh2uvBdq3l/MNpiorpTf/0kuW29jrLU+aJG+aM2bYfs3z54Hmza0/\n1rGjHN/ERHkTt+bCBd+cLI58U2SkXG1seqJfax4L9tBQ+eiclCSX0FanhnCrVlKqiI2Vkob6sdgR\n589LT23zZrldXGwM9qAgKbc89RRw7Jjlto4GOyD110OHJBirUxRZEHjJElmWzNr4+aIiGakze7YE\nafX6v6I4Vp+2pVkzKeU89pi8mVlz8aIcY9PALC+X2SH797cdfOXl8qlg0iT5+aydu7h4Udrw5Zdy\ngdC6dbbb6sj6mrNmmV+RCRhntExLk/qkqdJS+X1bo9PJJ5UtW2y/8RUUSPttGTlSlp2bNMn6m3th\noZR9iBz117/KHP579nhm/x4dFTNsmNRGH3vM8jHT3nWDBjLc6623pGf7/vuO7b+gQK4WVYOkqMh8\nkYM+fYxT21YPJGeCPThY3iRmzJDavSlFkfAYPVrm/7BWflJ/1ilTJNg++MD8cXd67KoBAyQQx461\nfrLv4kXg8cfNyzHqyed335Xg++ILy+3Uk6dz5kjP9I03LJ+jHsu2bSXcH3oIyM623s6SkpoXohg+\nXPb31VfG+yorpd791FOWNf+yMvv1bdM3vqNHLR+312NXLVwotf9lyywfY4+dnKUuTTh1qmeGK3t8\nuOMrr8i70qefmt9vGuyqUaNkxMQbb0jvqKYhQefPS7B//bWEi7V9zpwpPfklS8zvv3TJuTmoY2Lk\nl1C9jqwGOyBzuaxbJxebmLpwQdrVoIG8ac2aJXV3032402NXTZsmn2BmzTK/v6JC2njHHfLGlJUl\n96ujkpo1k+XUUlJkKJYp9TmBgfKc116T8pkp0zfJ2FipSY8YYb205kiP3fSKTPVYq8do+nRpv+kx\nduTE5YAB8qZr7Y2voKDmYA8Olk87L7xgPH6qwkIGOzkvKUmyqXrZUQseD/bGjeUqw6lTzdf7sxbC\ngNSfMjPlP1t8PPDrr9b3e+mS/AeNiJB/GRnmpRhVYKDU2efOlXKK6faO9thVzz4rbyZLlxrvMw32\n0FAJ7qQkeZ7pz6r+x4+Olnr9448bH9eixw7IPlaskFEqX39tvL+sTN7EAgOlVr58udxfXm48BgMH\nSmiOH28+3LC83HhSsFMnWWA5MVHCTFV9KOK4cXLC8u67LUPU0YWTR4+WE7fbtslt9RgFB8tJ6OnT\njSdYHR2R8sQTUvqrXts8f95+KUbVrZt0OsaMMS+5XbjAUgw5T12a8OmnnZsCxRG1coHSgAESZKYl\nEVvBDkgIfvaZ9K4GDjTO/WFKHQGj00kNdN06y1KMKiJCgn3CBGMYXLzofLCrbxJz5shwTcA82AEp\nI9x6q/koleo/6/PPAwcOGEsfWvXYARlC+sknEuDqG6lp8CYnS8+7uNjyOoKnn5afZcEC430VFebP\nGTNGThZPmWK8z9qb5Pz50gs2/ahZWSltcSSEGzSQKR7Uk7Gmx2jUKBmGqr5B2auxm9LpgJUr5dOj\n6XAzR3rsqvHj5W/S9I2ZpRhyVXw8MHiwfBLWUq1deTprlvSM3n1XbufkGOwO7dLpgCeflP+EEyZI\nSce0BGLay1KDXS15WJOSIs9X51mvqcdua4HZ7t3l4/gDD0joVQ92QHqU27bJyRHAsl3BwdKznzJF\nRlo422OvafHb+HgJnvvukzaaBnt4uJwL+Pxzy2Bv0EA+Xb3zDvD993KfaY9d9fbb8viqVXLb9E1S\nbVtAgOxrxw7jJxy1Z+3oz3r//TIGffdu82Ok08n5mNRUOTFfU41dZTAY0LKlvPFNnGh843O0x676\n29/k5//4Y7mt1clTjyxqrBG2zTWOtO211+RvylZ1whUuB3tGRgaioqIQERGBd955p8bnBwbKCcjU\nVCmJHD5sP9hVer2UZtavl1WN1HHMpiMZIiOlp759u+1gDwiQERVvvw38+KPrwQ5IaIaEyAk1a8He\npIn0DCdPlnlRrH06ue46qUM//bTzPXZH/liee04C+cUXLUslDz8svV1rV/62aycnWMeNk8Cr3mMH\n5Gf59FOp6f/yi/n5CtO2NWsmv7d58+QiLEfLMKpGjeREqVprNz1GffrIG/rcuY6XYtS2XX+9DOG8\n/375+ZzpsQPy+12zRj6V5eZq12Ov6wHlLXW9beHh8sl25kztXtflYJ82bRqWLVuGrVu3YsmSJTh7\n9myN23TrJv/Jx4+XXpajF2O0by9LwbVpIyfnDhwwH8mglmMOH7Y/4qJDBxlLPn68/Gd0thSjCgiQ\nWvaiRXJhjrUeaHy81KIfe8x22WnBAikJfPut9nPSqL3vZcuAf//bPNhvv11Gh5w9a31KhxEjpJw0\nebL1HjsgJ5NnzZI3gOJi28eySxfp2ScmyjQCzs6nMmmS9Prz8iyP0Ysvys+4d6/zV30+/7zsb/58\n53vsgFzJO2+elKbOnmWNndwzc6ZcQa0Vl4L9/OUzg9dffz06duyIm2++GTsdbFVKilxosm+fc1fZ\nBQVJGWfWLOnFf/SR+X/GkSPla037vP9+Kaf885+uBzsg77Kvvy5XSdoK5ZdekmF/e/ZYb1fz5jJa\nJy3NM5ONXXWVfHKYOtU82AMD5QSvtd64atEiY9ttXVE5bZr8Lj/91P6xvPFGOS8xZozzwa4OE83P\ntzxGrVvL38OmTY7V2E2pb3xLl8onSGd67KrJk+WN6623WGMn9zRuLBUAregUxflRlFu3bsUHH3yA\n1atXAwDee+89nDx5Ei+++KLslEvAEBG5xIVItuCR2R21aBgREbnGpVJMbGwsDprMFr9//34MGjRI\ns0YREZHrXAr25pcLkhkZGcjLy8OWLVswcOBATRtGRESucbkU89Zbb+GRRx7BpUuXMHXqVLRs2VLL\ndhERkYsc7rFXH7d+ww034O6778YVV1yBtLQ0jB8/Hvn5+Q5tq5o5cyaioqLQr18/TJ8+HSU2Js22\ntX1hYSFGjBiB1q1bo2nTpujatWvV4y+88AL69OmDvn37utQ2d7dX2xYeHo7rrrsO3bt3N3uOu/tX\nLVq0CAEBATh37pzPHLvU1FS0b98eMTExiImJQXp6us+0DQBWrFiBqKgo9OzZE8/YWMHcG20bO3Zs\n1THr3LkzYmJiXGqbp/7mDhw4gDvuuAN9+/ZFQkICcmysYOKNY3fo0CHcd9996NGjB8aOHeuVLElK\nSkJYWBiio6PN7jf9vYwcORIX1HnGnWxbTdubURzUt29fZdu2bUpeXp7SvXt35cyZM0pBQUHV43Pn\nzlVeeOEFh7Y9e/asoiiK8u9//1upqKhQKioqlEmTJinvv/++U9u/+uqryuOPP6706dNHGTVqlDJr\n1iy323bmzBlFURS3fza1baWlpUrLli2VlJQUTY6d2j5FUZRjx44pw4cPVzp16qTk5+d7/dip+05N\nTVUWLVpkdRtvtU09btnZ2cqgQYOU3NxcRVEU5Y8//vCZtpl68sknlRdffNGltmn9N6fuf8yYMcpn\nn32mKIqirFq1Shk7dqzXj52678TERGXNmjWKoijKggULlLfffrvW25aRkaHs2bNH6dWrl9n9pr+X\nxx57TFm4cKFLbatpe1MO9dhtjVtvevmqjPLychQVFSHYylUi1rb97+XlaIYNG4aAgAAEBARg+PDh\n2KbO+OTg9pmZmbj33nuh0+kwe/Zs5Obmut02dTy+uz9bZmYmkpOTUVpaihYtWuDMmTOaHDvT6wVm\nzJiB1+xMMlHbx+6/JssMKTWMjPLW73XTpk1ITk5GREQEAKBVq1Y+0zbTY7dmzRokJiY63TZP/M2p\n+2/evDny8/NRWVmJ/Px8hFpZAd5bf3MGgwEJCQkAgDvvvBP/+c9/arVtABAfH2/1mKi/l6CgICQl\nJVm95seR36u97atzKNh37dqFyMjIqts9evSoetHnn38ebdq0wY4dO/DU5Ymyf/vtN9x+++01bmtq\n+fLlVb8YR7fftWsXCgsLERkZicjISGRmZmraNne2Vx/btWsXevfujczMTE2P3fr169G+fXv07t3b\n7Dj6yrF75513MGjQILz66qsovDwVpC+0bfPmzdi3bx8GDBiASZMm4cCBAz7TNtX27dsRFhaGLl26\nON02T/7NLVy4EIsXL0ZoaCiWLFmCV1991WeO3bBhw7By5UqUlZXhww8/xHeXlzOrrbbZY/q66r6d\nbZu17e1xexKw+fPn49ixY4iLi6uqV7Zt2xZfm84bW4N58+ahadOmuOeeexzaXr0AqqZeobttc2V7\nR9vmzv5LSkrw8ssvY+7cuVX3q6/nC8cuJSUFR48exebNm3HkyBEsu7w6hTfbpu67rKwM586dw/bt\n2zFixAg8fnmaRl84bqrVq1dj3LhxVbd94W8OkBrylClTkJ+fj8mTJyM5OVnT9rlz7ObOnYt9+/Zh\n0KBBqKioQMjlS5x9oW229q3l77U6h4K9pnHrjRs3RlJSEr5XpwR0YtuVK1di8+bN+FidKs+B7dWh\nlbGxsWjSpAkOHjyInJwcxMbGato2V7Y3bZvapqysLMTGxlp9DVf2f+TIEeTl5aFPnz7o3LkzTpw4\ngf79++OPaqvjeuvYtW7dGjqdDs2bN8djjz2GL7/80uljp3Xb1H0PGjQIY8aMQUhICBISEnDw4EGU\nVlvRxZt/c+Xl5fjyyy8xZswYi20daZun/uYAYMeOHUhKSkJgYCCSk5ORkZHhdPs8dew6deqEv/3t\nb/jxxx9x0003Yfjw4bXaNnvU3wuAqn0727aatq/OoWC3NW7958uTkpeXl2P16tUYPXq0w9sCQHp6\nOhYuXIgNGzbYrFvZ237gwIH4/PPPoSgKXnrpJXTr1k2ztrm7/cCBA5GWloZGjRohPz8frVu31uzY\n9erVC6dPn8bRo0dx9OhRtG/fHnv27EHr1q194tidurzqdXl5OVatWoXbbrvN6WPnqbZdc8012LRp\nExRFwc6dO9GlSxeLvz1vtQ2Q6TqioqLQtm1bi20daZun/uYAYMiQIdiwYQMAKQUOGzbM6fZ56tid\nubxc18mTJ/Huu+9aDXZPts0e9fdSUlKCtLQ0qxdzOvJ7tbe9hRpPr15mMBiUyMhIpUuXLsrixYsV\nRVGUu+66S+nVq5cSGxurzJw5Uzl37pyiKIpy8uRJ5bbbbrO7raIoSteuXZXw8HClb9++St++fZWU\nlBSnti8oKFDuvPNOpVWrVkqTJk2Uq6++WrO2ubu92rYOHToo1157rdKtWzdNj52pzp07V42K8YVj\nN378eCU6Olrp37+/8sQTT/hU28rLy5VHHnlEiYyMVEaOHKlkZmb6TNsURVEefPBBZdmyZWb3+crf\n3L59+5SxY8cqvXv3VsaNG6fk5OT4zLFbvHix0q1bNyUiIkKZP3++y8fOnbaNHTtWueqqq5RGjRop\n7du3V9LS0ix+LyNGjFAKCwtd/r2abm+PS5OAERGR76q1FZSIiKh2MNiJiPwMg52IyM8w2ImI/AyD\nnYjIzzDYiYj8zP8DtDOLhZZTxfEAAAAASUVORK5CYII=\n" } ], "prompt_number": 38 }, { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Try to do this better with automated retry and using cloud.iresult" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "run jobs locally using cloud.mp" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# http://docs.picloud.com/cloud_cloudmp.html \n", "\n", "USE_LOCAL = False\n", "\n", "if USE_LOCAL:\n", " CLOUD = cloud.mp\n", "else:\n", " CLOUD = cloud\n", "\n", "# try setting n_tasks to something less than # of all segments to test out code\n", "\n", "n_tasks = len(valid_segments)\n", "\n", "jids = CLOUD.map(segment_stats2, valid_segments[:n_tasks], [None]*n_tasks, _env='Working_with_Open_Data')" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 39 }, { "cell_type": "code", "collapsed": false, "input": [ "jids" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 40, "text": [ "xrange(1789, 1966)" ] } ], "prompt_number": 40 }, { "cell_type": "code", "collapsed": false, "input": [ "CLOUD.status(jids)[:5]" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 42, "text": [ "['done', 'processing', 'done', 'done', 'done']" ] } ], "prompt_number": 42 }, { "cell_type": "code", "collapsed": false, "input": [ "jobs_info = CLOUD.info(jids,\n", " info_requested=['created', 'finished', 'runtime', 'cputime']\n", " )\n", "\n", "from collections import Counter\n", "jobs_counter= Counter()\n", "[jobs_counter.update(dict([(k, v[k]) for k in ('cputime.system', 'cputime.user', 'runtime')])) for v in jobs_info.values()]\n", "jobs_counter\n", "\n", "#print (jobs_counter['cputime.user'] + jobs_counter['cputime.system']), (jobs_counter['cputime.user'] + jobs_counter['cputime.system'])/3600. * 0.05\n", "print \"total runtime (s): \", jobs_counter['runtime'], \"estimated cost: \", (jobs_counter['runtime'])/3600. * 0.05" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "total runtime (s): 1336.6622 estimated cost: 0.0185647527778\n" ] } ], "prompt_number": 44 }, { "cell_type": "code", "collapsed": false, "input": [ "# plot # cores running vs time\n", "\n", "started = [{'jid':k, 'time':v['finished'] - datetime.timedelta(seconds=v['runtime']), 'count': 1} for (k,v) in jobs_info.items()]\n", "finished = [{'jid':k, 'time':v['finished'], 'count': -1} for (k,v) in jobs_info.items()]\n", "\n", "df = DataFrame(started + finished)\n", "\n", "plot(df.sort_index(by='time')['time'], df.sort_index(by='time')['count'].cumsum())" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 45, "text": [ "[]" ] }, { "output_type": "display_data", "png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAD9CAYAAACY0k3rAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVOX+B/DPuGGmWWqohYSZiprKuEFminRt8UpgyzXr\n2oL2U9QW0fbuvVj3Vjc1Mm+5XcnKi6aVu6EVjmuCuYeYGwQiomwKCsoMz++PrwMDM+fMmeEMZ5bv\n+/XyNTAz58yXI3znmef5Ps+jE0IIMMYYc3uNtA6AMcaYMpywGWPMQ3DCZowxD8EJmzHGPAQnbMYY\n8xCcsBljzEMoStgmkwl6vR6RkZEAgPj4eAQEBECv10Ov1yM5OdmlQTLGGAOaKHnS3Llz0bNnT5SW\nlgIAdDod4uLiEBcX59LgGGOM1bDbwj5z5gw2bdqECRMmwDzHRggBnm/DGGMNy24Le9q0aZg1axYu\nXbpUfZ9Op8O8efOwatUqjB49GpMnT0arVq1qHafT6dSPljHGfIBUg1i2hb1hwwb4+/tDr9fXOkFs\nbCwyMzOxefNmnDp1CgsXLpR8UU/+949//EPzGLT+x9eArwFfg4a9BnJkE/bu3buxbt06dO7cGWPH\njkVKSgqeeeYZ+Pv7Q6fToXXr1pgyZQpWr14t+yKMMcbqTzZhv//++8jJyUFmZiZWrFiBiIgIfPXV\nV8jLywMAGI1GJCUlYeTIkQ0SLGOM+TLFddhCiOp+6ddeew19+vRBWFgYKisrERsb67IAtRQeHq51\nCJrja8DXAOBrALjHNdAJe50mzp5Yp7PbH8MYY6w2udzJMx0ZY8xDcMJmjDEPwQmbMcY8BCdsxhjz\nEJywGWPMQ3DCZowxD8EJmzHGPAQnbMYY8xCcsBljzENwwmaMMQ/BCZsxxjwEJ2zGGPMQnLAZY8xD\ncMJmsnjBRcbcBydsJqtPHyArS+soGGOAwoRtMpmg1+sRGRkJACgtLUVUVBQCAwMRHR2NsrIylwbJ\ntHH+PPDbb0BBgdaRMMYAhQl77ty56NmzZ/WOM/Pnz0dgYCBOnDiBgIAALFiwwKVBMm3s2UO3V65o\nGwdjjNhN2GfOnMGmTZswYcKE6l0Q0tLSMH78ePj5+SEmJgapqakuD5Q1vF9+oVtO2Iy5hyb2njBt\n2jTMmjULly5dqr5v7969CA4OBgAEBwcjLS3N5rHx8fHVX4eHh7vFnmjewGgEmtj9n6u/X34BWrVy\nbcKuqgJMJqBpU8eOu3oV8PNzTUyA/DUWAqisBJo1s38ekwlo1Ai4/uHULVRWOn69mesYDAYYDAZl\nTxYy1q9fLyZPniyEEGLr1q1i1KhRQgghOnXqJMrLy4UQQly+fFkEBgZaHWvn1MxJ334rRL9+rn+d\noiIhWrcWYupUIQIChPjb34Q4dUrd18jMFCIsTIjnnlN+zNGjQowbJ4SfnxB//KFuPGbFxUK0ayfE\nsWO2H9+0SYh771V2rg8/FOLhh4WorFQvvvoChNixQ+somBS53CnbJbJ7926sW7cOnTt3xtixY5GS\nkoJx48Zh4MCByMjIAABkZGRg4MCB9XuLYYqYTMDf/gYcPQpcv/wus2QJEBkJzJsHbNgAXLoEhIYC\nERHA11/Xv9X9/ffAoEHAAw8Aq1cDly/LP//wYeAvfwGGDQO6dwfuvx/YurV+MUj54w+gqAh4/nm6\n5nWdPQvs2gXs32//XIWFFOfUqe5VIsndXJ5JNmG///77yMnJQWZmJlasWIGIiAh8/fXXCA0NRWJi\nIsrLy5GYmIiwsLCGitenffMNcMstwIQJwHffue51TCbgP/8BXnqJvu/bF/jkE+DMGWDyZGDFCiAg\nAJg4EUhNdSwRVVQAL74IzJgBrF8PzJwJhIXR17bs3QtERQEPPkgJ/vRp4O23gZEjge3b6/+z2pKT\nA4wYQV0uCQnWjxcXAzfeCHz+uf1zXbsGvPkmsHs38PHH6sfqqMpKum3bVts4mHMcqsM2V4nExsYi\nOzsb3bt3R25uLiZNmuSS4FgNo5GS27vvAo8/7tqEvX490LEjUPeDk58fvfbGjcCRI0BQEPDXvwJ3\n3w3MmQPk58uf9/hx4J57gLw8ap2GhtL9Tz0FJCXVfu6uXcBDDwGPPkrJ8/RpSvItW9Ljw4YB27ap\n8uNaOXMGCAykTxn//jdw7Fjtx4uKgPHj6f+guFj+XNeuAe3a0aeUjz+mTxNaunCBbhs31jYO5iQt\n+mGY4778UoihQ4WoqhLCaBTC31+IkyftH1dRIYTJZPsxo1GI60MRtQwfLkRSkrK4qqqoP/T554W4\n+WYhoqKEWLvW+jX/9z/qF/7sMzrG0sWLQtx0kxAFBUL8/LMQ4eFCdO4sxKJFFL8tJpMQbdsKceaM\nsjjNysrsP+ett4R49136+vPPhQgNpWtlNnEi3f/000LMmWN9vOU1nTBBiMWL6eu9e+kapKU5FrOa\n9u2jPuwDB7SLgcmTy50809FDJCYCr75K1QaNG1P/6lNPAdnZ8se98AJ1Y9RVVUWt5TFjat+/bRtw\n8iTw2GPK4tLpgCFDKL7sbOCRR6jLY9OmmufMm0d97z/+SLHUrZi46SZqeX/2GfDkk/SzHT9OsUtV\ngjRqBAwYABw8qCxOgFq7LVvSrZyDB4GuXenriROB8nJgx46axzMzgdtuA55+2ror59QpoH176lYx\nv6a5ImPAAOCDD+haaOX8ee1em9UfJ2wPkZUF9OpV8/0HH1DCHTQISE62fUxpKbB2LX0M//XX2o+9\n8QZ9tE9NpUFMgErlJk0C5s5VVrJWV6tWQEwM9UmbBxGTkoCPPgJ+/hkICZE+tkULYOVK4LnngGee\nUVa22KyZ7UFBKX/8QbcWFapWzp2j/uZRo+j7Ro0Af/+avt8rV+jx8HBK/kZjzbFCUL+/yVTTv37t\nWu1reeed9t8wXOniRe1em9UfJ2wPYDJRv+/tt9fcp9NRi3vVKhqE/NvfrJPXmjXA0KGU3KdOpVY1\nACxeTI99/z3dP3s23T97NrUso6PViTs5GZg2DfjhB+rvtic9Xb3XtuXUKbqVS9jLlgGjR9f0ldeV\nkkIt5datrR9bt4762v/+95oWed2ErTVO2J6NE7YHyMujUX1bf/j33Qfs20etvgcfrD3wt2wZDQo+\n9xy1/r76ilq677xDg2Bt21IXxZo1lGASEqj7Qo1JHnv2UEt59WoalFSifXtqnbuKvYQtBLB0KV0v\nKRs3UoVKXVeuAC+/TN0699/PCZu5RgPMl2N1mUz0h9OmjbLnZ2dT1YKU9u2BLVuA+Higf38qu7vr\nLiAtjRJmo0ZUphcZSUnpm2+Abt3o2DZtKLE+8ABVoNxxR71/PAD0euvWAYMHKz8mKopidRVzwi4t\nBUpKgJtvrv34r79Sf/V999k+Xgjqm//hB+vH3n+f+uEjIqib5MwZqsFWOiPSWUVFyn+PAOcTdnEx\nlZQybXELWwNbtgAdOtCg2unT9p9/4ABNFpHTuDHw3ntUL/3cc5S0o6KobxigEr2JE6m0rO4KAXFx\nNMj4yivO/DTWBg+m1v3DDys/ZtAgGmx0JXMf9vbtVEdet447OZmug9QnjLQ0Sr49elg/tmAB8K9/\n0ddNmgC9e9NKh65sYV+5Qn3ieXnKj3FmYc1r1+hNwZHxAuYanLA1cOkSMHw4Je1Bg6iFW7fW19L/\n/mddzSFl0CD6AzN3h1iaOZMqG+oKDKTnq7W+xMsvK4/X7I03XNsdAtAysc2b0wBux46UnC2T9oUL\ntccJ6vrsM3rTs5XQL16kNwGzpk1pzMCySkRta9fS61ZUKD/GPHjqiHPn6La83PFjmbo4YWugooK6\nMd57j0rounWjwcExY2gKtqVTp+jfiBHKz5+fT9Onhw9XN25PV1hI/fZ5ecCf/gQsX05J2zwBx/y4\nLRcuUAlfTIz1Y9euURK31ZJ2ZQv7q6/o1pGWr2VVi1KcsN0HJ2wNVFRQSw+gftR33qGukQEDaOAw\nOrqmDM/cunaklXbtGtVo82y22iwTdseOlLRXrKDySIOBWuBSCXvJEqoesdVfXFYmXVXiqoSdl0cD\nu7ffXlP9owS3sD0bJ2wNWCZss5YtqUzv9GmqMhg9mvqAExOtuzaUsNX14cuEqEnY585RwgboWn/z\nDfDEEzS2YCthm0xUzjdliu1zyyVspYOOFy861lJOSqI39pYt6TjzlHN7LFvYpaXKulPMfeS8YJT2\nOGFrYOdO6brkG26gmYInT1LSHjbMek0POW3a0GCm3CQVb9GrF1W2KNlzcskSoEsXmjl57lztqpuI\nCCpnBGwnbPPKiP372z53SQnN1rSluJgmFMkRgrqvli6Vf56lNWvok1fjxjR78bbbaOKRPeaELQRd\nuxEjaMKUHPPsSHvPY67HCbuBGQy0Ap299bL8/ID/+z/gyy8dq4tu2RJYtMi9Fsx3lfffp08SoaFU\nVy4lOZm6ndato++rqqyTr7mcsV076+PttSzz8ihh1nXxIv2TG8gEKPYDB+ifEpWVtHjWPfdQGWRZ\nGX1ie+klGoi0d6xZaSlNVoqJkV9xsahIWVzM9ThhNyCjkf6oZs2qKbdjztPpaCbl6tVAbCwtY1p3\nUO3gQarC+f77mvVBOne2Tszm6eK2Wsr2+m7PnrWdsH//ncox5WrLhaDB59hY6wFnKYcP08/QujW1\nsMvLafr8xo306cpWnbiZ5fW5do1a2adOUQWRFHsrErKGwwm7AS1aRB+5H39c60i8y+DB1OLct48G\nEs2DZDk5NFnos89qT+Cx1cVkbkXa+mRib6Du7NmaPnFLGRm2a7YtbdlC6678/e+0ZK2StcVTU2tK\nIBs3pn7opk3pU8PatcCzz9KMVlssf5arV2nQe+1a+iT39de2j+EWtvvghN1AiopoJuLcub7RXdHQ\nbr2VWpbDh1Pi2rgR+POfqSb8iSdqP9dWwi4slD//DTdIPybVwraXsM2t67ffppp8Pz8gN1c+DoCq\nQ8xriTdqRAnbPLB5zz3At9/Sqoc7d1ofW7eF3awZlZhu3AhMn257UwhO2O6DE3YD+d//qGSvTx+t\nI/FejRsD//gH8OGHtNpely6UhCw9+GDNSnyWhg0Dxo6VPveNN0o/duIEdVHUlZMjP9U/P5+SunmS\n0Z13KhtAPXgQ6NePvm7UiCZiWXaxDR1K1+DDD62PtUzYltVKPXvSmEDdHXYqKmhSlyPT35nryCbs\niooKhIaGIiQkBGFhYUi4/r8ZHx+PgIAA6PV66PV6JEut78mqbdtGyYK5nrlVO2yY9aeZV14BgoOt\nj+na1XrXG0u2BiMBGsBMS6tp8Vq6dk1+Z/ezZ6laxVwvr2QdFSEoqd95J32v01EXUIcOtZ/XsaPt\n+mzLLpG65Yi33mp9zOrV9OZgOYuTaUd28afmzZtj69ataNGiBa5evYr+/ftj1KhR0Ol0iIuLQ1xc\nXEPF6dGEoI+atvYHZK6jdJVAJW691fb9R4/SY7Yer6yUn/Ak1ZUip6iIErvlQkzmiUBKWLawy8rs\nlxwuXkwVTeZ1Upi27L6nt7j+WausrAxGoxF+15sMwp22gHZzv/9OH6k7ddI6Et9g/tVsiIRdUkL9\nxra4ImFnZVnX8NtqYUuRa2HXdfIkLWAVFeVYjMx17C6vWlVVBb1ej/T0dHzyyScIvD7jYN68eVi1\nahVGjx6NyZMno5WNt+r4+Pjqr8PDwxFed5k4H7FtG308Zw3DvCZ4+/bqnVOqSwRwfcI+d466Vm65\nxXbCrk8LWy5h//e/VBIp163D6s9gMMBgMCh6rt2E3ahRIxw6dAhZWVkYOXIk7r33XsTGxuLvf/87\nLl26hFdffRULFy7EjBkzrI61TNi+LCXFsaVGWf3o9dKr6jnj1VeBceOkH5eaVWo0Wi9BYOn4cVqH\n3J6JE2mN7hkzak+rN8vNVZb4haDnmmdzXrkiX/3y/fc0bZ+5Vt3G7EyZonjFVSJBQUEYOXIkUlNT\n4e/vD51Oh9atW2PKlClYvXp1vQL2ZiYT8NNPyv4wmTpuv53Wp1bLRx/R+tZyrydFbtH/X36xv6Rs\ncTGVK5aU0Pfl5daTrkpKbFep1HX0KLWozV1zRqP0J4Dz5+kfVzW5F9mEXVBQgJLrvymFhYXYsmUL\noqKikHd9NRij0YikpCSMtLVnEgNA09ADAhzvq2Tuz1xRIdf1IlUOd+4cTVs37/wj5bvvanYoAihh\nW7aKzTEo2TNz69baS+4ajdKbHe/aRV09vOKje5HtEsnLy8Ozzz4Lk8mEDh06YMaMGejYsSOeeeYZ\nHDx4EM2aNcPQoUMRGxvbUPF6nORkLufzVgUFdCvXxyvVwjZPfrFXyrd8OU0AskzYljXh5v56uTpx\ns5QUmmW7bx99by9h33uv/XOyhiWbsHv37o39+/db3f+VeeV0ZldyMk1IYN5Hbmsu88p2UmvG/PKL\n9GCl5fn37wc+/7ymL7m8vPYAaHa2slhNJlp47LPPaC0bwH7CVvJ7e+YMzZb091cWB6sfnunoQiUl\ntBoat1S8U2QkrQJoi7lFLDXw+fvv9ssOFy2iFnGbNjUrBhYUWLfalUxq2buXuuXMA5YXL9KAqFQL\n//hxZWWRb79N1SSsYXDCdqHt22lQicuivNMdd9BaILZcvix/bEmJ/IDk5cvUGp4+vXbSP3HCut9b\nSUnfunXAI4/Q1zodnadrV+k3lMuX5Uv+zHbudG5jX+YcTtgutHUrLY7PfI+ShH3zzdKPf/EFfTKr\nO43++PH6J2zzeczLzdZlMtG0ermSRIC6bE6ftv+zMvVwwnahlBTeCNdX1SdhG43AnDnA66/Xvr+w\nkGqp607isZewT52irpRBg2ruM7ewbblyhfre7dWx79pV83zWMDhhu0hBAc1Kk9pWink3e5seXLxI\nGxDYsnIlLQpVt0bb3Lqum0jtTUtfv5762y37q211rZiZE7Y9O3dSfTq3sBsOJ2wXOXqUBm0c2e2c\neRephZWqqmhJVKl9IH/+2fZSr2fOWK9HM3s2zYSUc/iw9WqC589LJ/orV5SVCR46RBtD8F6PDYcT\ntosUF/Mawr5OqquitJQSolRJ3cWLtjcDtjVQOX26/S6RurXbAL1pSFWIXL6srIWdmUlrjrOGwwnb\nRYqL5asAmPeTasHKdYcAlJhtPS51vz3l5dYDiEJI91Er6RKprKRBR16BsmFxwnYRTthMquVrr0Lk\n4kXbj0vdb0/d6eyAfMK+fNl+l0h2NtV1c5dfw+KE7SKcsJlUC1tJwla7he1IwlbSwj59umbXG0uV\nlTXVI0x9nLBdJDdX+RrFzPt8+aV1WZ6ZySQ/O7G01PakFVuJVwlbx8mt1V1cbL8l/+23wJAh1vev\nXUulrGfOOB4ns48Ttov8/jvQvbvWUTCtPPOM/Bu2XLmnrSVUAflWsRzLzXbN5FrRZ8/KLxmbmwus\nWgW8+KL1Y999Rz83b4fnGpywXeT4cU7YTJq9hC3VknYmYUudTy5hyy0H/PHHwHPPWU/gqaigtbtX\nr6aZmoWFjsfK5HHCdoHiYvrlVbrPHvMdJhPd6vXSj1dW2l5/xtltVNVM2AUFlIynT7d+7KefaMOD\nfv2A0aNpLRSmLk7YLiA1I42xU6foVqqP2FyCZ+t3x9kuEamELdWKl0vY8+bRCoK2uky+/x547DH6\n+rXXgP/8h6etq40TtgtkZSnbson5nieeoMQmRar/GqBdzJUspWrrnI60sOVmYe7ZA0RH237s2LGa\nrp7u3WnyT2am4/EyabIJu6KiAqGhoQgJCUFYWBgSro8klJaWIioqCoGBgYiOjkYZr69Yi71BG+a7\n2ral7gIpUsn14kWaCm6rMkOOEDR13NbKe3IVJ1IteanuGoC6cyxnb9rbTYc5TvaSNm/eHFu3bsXB\ngwexbds2LFmyBCdOnMD8+fMRGBiIEydOICAgAAvU3PHUCyjdxZqxuqQSdloaLQblaFnf1atUvmeZ\nPK9do1upqfFy5MoBTSbeA9LV7L4Htrj+uamsrAxGoxF+fn5IS0vD+PHj4efnh5iYGKSmpro8UE9i\nb5SdMSlXrkgn5QcecPx8tt4A7K0kKIcTtrbsvsdWVVVBr9cjPT0dn3zyCQIDA7F3714EX19ZPTg4\nGGlpaTaPjY+Pr/46PDwc4eHhqgTt7rhLhDlLrqRvxAh1zmdvIFCuGoUTtvoMBgMMBoOi59pN2I0a\nNcKhQ4eQlZWFkSNH4t5774VQWF9kmbB9idygDWNycnKsP51VVdFt796On89Wwj53Tv4Yk0m6/9lo\nlE7YZWXKlmVltdVtzM6cOVPyuYqHBYKCgjBy5EikpqZi4MCByMjIAABkZGRg4MCBzkfrhbilwZx1\n+DDVMlsqLaVbZ36nbK3UB9AGCVIqK2kndKnHpBL2+fNA+/aOx8iUk03YBQUFKCkpAQAUFhZiy5Yt\niIqKQmhoKBITE1FeXo7ExESE1d0aw8dxwmbOspWwr/8JOqWiwnYXi9w61nJJWeqxy5fp917Jxr3M\nebIJOy8vDxEREejbty+eeuopzJgxAx07dkRsbCyys7PRvXt35ObmYtKkSQ0Vr0fghM2cVVJinbCL\ni50/n1SfuNoJ+8IFwN+fJ4u5mmwfdu/evbF//36r+1u1aoW1a9e6LChPxwmbOePiRbqtu2xpffZM\nlErYUvs5AlT252iXSH4+cOutzsXIlHOiEpPJEYJ24uBfXuaorCy6rTvgFxfnXIUIQAPgdfeW/PFH\n+Qk4Ukm5spIWdLK19V16uvybAFMHJ2yV5eRQPx7v58gcVVZmexZh69aOz3A0s7UP5J/+JH+MVMLO\nzKRyVVst9l9+oQ15mWvx5FGVpacDvXppHQXzVGpXWTiz85FUws7IAK5Pv7Cyezdwzz2Ox8ccwwlb\nZUePcsJmztM6YQshnbCPHQN69LC+v6SE9nisO1jK1McJW2Xcwmb1oXXCrqykAXNbE2eOHbO9KUda\nGq3Sxxvyuh4nbJWdOgV07ap1FMxTObMruhxb24PJSU8H7rrL9mM5OUBQkPX9ZWXyZYJMPZywVfbH\nH/KzyBiTY2+3ckcJ4dgypzk5wP33236soIBqrW3hCTMNgxO2ikwmKunr1EnrSJinUjthV1U5vi51\nRIT0Y1JdNnVLB5lrcMJW0dmztDGp1KQDxuzROmHrdICtRTXNe1HW3XjXjFvYDYMTtoqys7k7hNWP\noyV49phMjk0XDwmhXXHqMu+ALrXpgVwL++OPgaIi5TEwaTxxRkVyfXyM2bN/v/oVRufPK591GxFh\nPS3ezN4CVFIt7JIS2mF95UraVZ1b4vXDLWwV2ZoGzJhSer363WmZmco3hG7VSnrN7YoK+a4Vqd/7\nI0eA0FB6I3r0UdqyjDmPE7aKeOMC5k6MRtpfVK1uOrnWsdRjhw8DffsCCxfS5gbjxtX0hzPHccJW\nESds5k7OnKEuOqldzh3lTMI+dIhmQDZpAixfTn3hkyfLb0PGpHHCVlFBASds5j4c6Q5RQi5hy3WJ\nmLtZmjcH1qyhvvo5c9SLy5dwwlZJYSHw9dfAI49oHQljpKLCdsWHs6RK+gDpZF5aWnvQs1UrWi72\n11/Vi8uXyCbsnJwcDB8+HL169UJ4eDiSkpIA0Oa6AQEB0Ov10Ov1SE5ObpBg3dnMmcDjjwN33611\nJIzVcGRauj0dOkg/JjfYXnczD96VxnmyZX1NmzZFQkICQkJCUFBQgEGDBiEyMhI6nQ5xcXGIi4tr\nqDjdWkYG9c9d35eYMbdha+1qZ8klbLnuEt59ST2yCbtDhw7ocP1/qV27dujVqxf27t0LABA8alAt\nLg546y35j4yMacHVCfvaNbqVG9jkhK0exRNnTp48ifT0dISGhmLHjh2YN28eVq1ahdGjR2Py5Mlo\nZeMzUXx8fPXX4eHhCLc159XDpaQAp08DU6ZoHQlj1tTsErG1Il9ZGd3KdXM4krAffZQ+rVq+AZw7\nByxbBsyYofw8nsRgMMBgMCh7slDg0qVLol+/fmLNmjVCCCHy8/NFVVWVKCkpES+88IKYNWuW1TEK\nT+3x3nhDiJkztY6Csdo2bhQCEOLjj9U53++/C2EyWd8/Zw69ji09e9Jjubm171++XIgxY6yfbzTS\n8/Pza9//889CtG/vXNyeSC532q0SqaysxGOPPYZx48YhKioKAODv7w+dTofWrVtjypQpWL16tfNv\nLx5u/36gXz+to2DMNlsbDjijWzfbMx3NLWw5SlvYFy7Qbd3XuXKFdmU3P+7LZBO2EALjx4/H3Xff\njVdeeaX6/ry8PACA0WhEUlISRo4c6doo3ZQQlLD1eq0jYcw2V+9kXlpq/zlKE3Z+Pt1WVdW+//Jl\nuv3tN+VxeSvZhL1r1y4sW7YMKSkp1SV8P/zwA15//XX06dMHYWFhqKysRGxsbEPF61Zyc6k1cNtt\nWkfCWG3l5XRra4cYNanZwj53jm45YUuTHXQcMmQIqupePQAPP/ywywLyJL/+Sq1rritl7ub0abqV\nWg5VLXIze82FZEpjMCfsumuNnD9Pg5C//UabXC9dCnz4oeMbM3gDH/yR1bNpE/DAA1pHwZg1JV0V\nanj3XdplyRZzF4fSJVVtdYkUFwNz5wL/93/U0v7oI2DRIu+tGLGHE7aTTCZg7VogOlrrSBizZjQ2\nzOv4+UlPqCkqomnpSj+B2uoSeestKvUbNIgGHdeuBfbtAzZvBhIS6he7J+INDJy0Zw/9okot+M6Y\nliortY6ASK2vbUvdFnZqKiXoo0eBDRtoA4Rnn6V68ORkYPBgGj8aM0b9uN0VJ2wnrV7NrWvmvhqq\nhW1Pnz7Kn2vZwjYagUmTgFmzgJtvrrl/6lT6ulMnYONG4E9/oo2BvXBOnk3cJeKkjRuB62XpjLkd\nqd3NG9qQIcqfm51NtyYTbXjQpg3w1FN0X5MmwD331J7z0KcPsGIF8Je/2N/CzFtwC9tJ+fnAHXdo\nHQVjts2YAUyYoG0MeXnK3zh+/ZVa1V260PokH35In2LN/d+jRwMjRlgfFxFBg5pFRTUtcW/GLWwn\nXb2q7joNjKmpSRPtFyPr0EH5gOOCBcDEiVSq9/XXQM+ewIABNY/7+am7tren4ha2kyoq1Nt6iTFf\nVlICfPdvb6GiAAAVDElEQVQdcOwYsHgxle5t3ap1VO6JW9hOMA/ouHpSAmO+YNky4MEHqfvk9Glq\nSQ8bpnVU7okTthO4dc2YepYuBV54oeb711/n2cNSOGE74dIloHVrraNgzDvk5wNdu9Z8r9YKg96I\nE7YTiouBW27ROgrGvMPVq/yJVSlO2E4oKfGNEiLGGkLdhM3dIdI4YTuBEzZj6uEWtnKcsJ1QUECz\nsBhj9XP1KlVdWSZsd5ml6Y64MM0JBw86tqgNY8y2nTtpgox5beuyMuDGG7WNyZ3JtrBzcnIwfPhw\n9OrVC+Hh4UhKSgIAlJaWIioqCoGBgYiOjkaZkm0nvEhqKi33yBirnx9+ACz3Q+FkLU82YTdt2hQJ\nCQlIT0/Ht99+i3feeQelpaWYP38+AgMDceLECQQEBGDBggUNFa/mKiuBQ4eA/v21joQxz1c3YTN5\nsgm7Q4cOCAkJAQC0a9cOvXr1wt69e5GWlobx48fDz88PMTExSE1NbZBg3cHhw0DnzvJbIzHG7MvO\npu2/LNcMYfIU92GfPHkS6enpGDRoEJ5//nkEBwcDAIKDg5GWlmbzmPj4+Oqvw8PDEe4Fi9YePFh7\niUfGmHO2bgXuv98392a0ZDAYYDAYFD1XUcIuLS3FmDFjkJCQgJYtW0KYd9e0wzJhe4uKCqBVK62j\nYMzzFRUBHTtqHYX26jZmZ86cKflcu+9tlZWVeOyxxzBu3DhEXV+xf+DAgcjIyAAAZGRkYODAgfUM\n2XMofK9ijNlx5Qpwww1aR+FZZBO2EALjx4/H3XffjVdeeaX6/tDQUCQmJqK8vByJiYkICwtzeaDu\nQgieicWYGsrLgRYttI7Cs8gm7F27dmHZsmVISUmBXq+HXq9HcnIyYmNjkZ2dje7duyM3NxeTJk1q\nqHjdAidsxuqvvJxb2I6S7cMeMmQIqiz3nLewdu1alwTk7goLeRotY2ooLeW6a0f5+PisYyoqgEWL\ngL/+VetIGPN8J07QHo5MOU7YDvjySyAkhP4xxuonIwPo0UPrKDwLryWikNFIe80tXap1JIx5vuJi\n4PJl4PbbtY7Es3ALW6FVq6hm9L77tI6EMc+XkQEEB/MAvqM4YSu0aBEwfbrWUTDmHXJygG7dtI7C\n83DCVujMGaBnT62jYMx7tG2rdQSehxO2QgUFQLt2WkfBmPfgBdQcxwlbAaORakZ5WzDG1MMJ23Gc\nsBUoKqJd0hs31joSxrwHJ2zHccJWYPNm4I47tI6CMe+i1ifWNm2AL74AJCZlexVO2HakpQFxccB/\n/6t1JIx5l86d1TnPunWAwQD8+c801uTNOGHLyM4GRo8Glizh2Y2MqcW8RPFdd6lzvttuA1JSaGPs\n/v2BPXvUOa874oQtobQUiIwEpk0DHnlE62gY8x6FhXTbpo1652zalGYif/op/b1++ql3rl3PCdsG\nkwl4+mnaa44nyzCmrlOn6NYVsxyjoqiFvXQpMGYMcOmS+q+hJU7YNsyeDZSUAPPn89RZxtSm1wN9\n+rju/HfeCezeTX+7b7/tutfRgt2EHRMTg/bt26N3797V98XHxyMgIKDWpgbe4uRJYNYsWpmvWTOt\no2HM+zzzDHDokGtfo3lz4KGHgLIy175OQ7ObsJ9//nmrhKzT6RAXF4cDBw7gwIEDeOihh1wWYEMS\nApg4EXjrLfVGsBljTC12E/Z9992HW265xep+pTune5KlS4GLF4GXXtI6EsYYs+b0etjz5s3DqlWr\nMHr0aEyePBmtWrWyek58fHz113W3cnc3+fnAG2/QJJkmvEo4Y6yBGAwGGAwGRc/VCQVN5aysLERG\nRuLIkSMAgPPnz+PWW2/FpUuX8Oqrr6Jbt26YMWNG7RPrdB7VCn/vPSAvD/j8c60jYYyp4YsvgO3b\n6daTyOVOp6pE/P39odPp0Lp1a0yZMgWrV6+uV4Du4MwZKrxnjDF35VTCzsvLAwAYjUYkJSVh5MiR\nqgalhfx8oH17raNgjDFpdntrx44di23btqGgoACdOnXCzJkzYTAYcPDgQTRr1gxDhw5FbGxsQ8Tq\nUpywGWPuzm7CXr58udV9MTExLglGS5ywGWPujmc6gqainzsHdOigdSSMMSaNEzZodmPHjkDLllpH\nwhhj0jhhAzh82LVrGzDGmBo4YYMTNmPMM3DCBnDkCCdsxpj78/mEXVUFpKYC/fppHQljTE133gn8\n8APw009aR6Ien0/Yv/5KO6Lz6nyMeZdhw4Dly2k513ff9Y5Nen0+Ya9fz1uAMeathg8H9u0Dfv4Z\nGDnS8zfp9fmEvW4dJ2zGvFnHjpSwQ0Ko6/OXX7SOyHmKVutz6sQesFpfVhYwaBCt0te4sdbRMMZc\nbf16YPx42jrs5Ze1jsY2udzp0wn7gw+A06eBxYu1joQx1lAyM6m1fewYtb7djVzu9Nml+o1G2mR3\nzRqtI2GMNaTOnYGbbqIc4Gl8tg977VqgUycu52OMeQ6fTdj/+Q/w4otaR8EYY8r5ZMI+cgQ4fhx4\n7DGtI2GMacXNh9hs8smEvWQJjRQ3bap1JIwxLfTrB8TGAoWFWkfiGNmEHRMTg/bt26O3xWaHpaWl\niIqKQmBgIKKjo1FWVubyINVkMgErVwJPPaV1JIwxrXz7LdCjB9C/P5CWpnU0yskm7Oeffx7Jycm1\n7ps/fz4CAwNx4sQJBAQEYMGCBS4NUG3bt9POMsHBWkfCGNNK06bA7NlAQgIwahSNaXlCF4lswr7v\nvvtwyy231LovLS0N48ePh5+fH2JiYpCamurSANW2YgUwdqzWUTDG3MHo0TTzcckS4MkngdJSrSOS\n53Ad9t69exF8vXkaHByMNJnPE/Hx8dVfh4eHIzw83OEA1VRZCXz3HS34xBhjANClC7B7N818HDCA\n5mb06NFwr28wGGAwGBQ91+5Mx6ysLERGRuLIkSMAgMDAQBw/fhzNmzfHlStX0KNHD/zxxx/WJ3bD\nmY6//AJMngwcOKB1JIwxd/Tmm0BREbBwoXYxyOVOh6tEBg4ciIyMDABARkYGBg4cWL/oGtCBA/QO\nyhhjttx5p3svw+pwwg4NDUViYiLKy8uRmJiIsLAwV8TlEvv3A3q91lEwxphzZBP22LFjMXjwYBw/\nfhydOnXCF198gdjYWGRnZ6N79+7Izc3FpEmTGirWejtwgBM2Y8xz+cxqfZWVQOvWwIULwI03ah0N\nY8wdLV5MddlaruCpah+2p8rLA9q04WTNGPNcPpOwKyqAFi20joIxxpznUwm7eXOto2CMMedxwmaM\nMQ/BCZsxxiyYTFpHIM1nEvbFi7QtEGOMSRk0CNi4EZgzxz0Xg/KZhH32LHDbbVpHwRhzZ337Aqmp\nwDff0AYnJSVaR1SbzyTsvDz33CGZMeZegoKAHTuogTdgAHDwoNYR1fCZhM0tbMaYUn5+tEb2P/8J\njBgB/Pe/7tFF4lMJm1vYjDFHPPkkbXqSkADExGiftH0mYeflcQubMea4Hj1ouvqaNcC5c9rG4jMJ\nm7tEGGPOuvFG9ygL9omEbTTS7sj+/lpHwhhjzvOJhH3+PNC2LdDE4Q3RGGPMffhEws7K4gFHxlj9\nVVRo+/pen7D37gXGjKERXsYYc9bTTwMPPQRc395WE/XawCAoKAg33XQTGjdujKZNm9baQd0dNjBI\nSgJeeYUWI4+K0jQUxpgX+PprIC4OmD0bePZZ17yGXO6sV8Lu3Lkz9u3bhzZt2jj0oq5WVQW88w6w\nYgWwdi3Qu7cmYTDGvFB6Ok1bHzIEmDcPuOEGdc/v0h1ntG5F11VaCkRHA7t20ZoAnKwZY2rq1Yu6\nWi9fBu65Bzh5suFeu151EzqdDhEREejcuTNiYmLwyCOP1Ho8Pj6++uvw8HCEh4fX5+VkXbsG/Pwz\n8OqrwL33At9+CzRr5rKXY4z5sFatqMv188+BwYOBBQuARx917lwGgwEGg0HRc+vVJZKXl4eOHTsi\nIyMDkZGR2LlzJzp06EAnboAukcpKYOtWYOVKmoXUvTswcSIwbhyg07n0pRljDADNgvzLX6ib5MMP\ngaZN63c+l/VhW4qLi0OPHj3wwgsv2H3R+jAagW3bKEl//z3QpQtdrCeeADp1Uv3lGGPMrsJC4Jln\naDnWb74BAgKcP5dL+rCvXLmC0tJSAMCFCxewefNmPPTQQ86eTpbJBBgMwOTJwO23A2+8Adx1F/Uj\n7dlDo7acrBljWmnbFli/Hhg1Chg4EPjxR9e8jtN92Pn5+Rg9ejQAoG3btpg+fTo6qZg1q6po4HDl\nSuqP7tCB6ql376ZWNWOMuZNGjYA33wTCwqhme+JEqlZr3Fi911CtS8TqxPXoEpk5E1i0iN61xoyh\n7o5u3VQOkDHGXCQvj5ZmbdEC2LDBsaTdIH3YjryonGvXaGWsw4dpWUPGGPNERiPQrh2V/bVrp/w4\nl9Zhq62wkFrWnKwZY56sSRP1F5xz24TNGGOsNrdL2AUFjn18YIwxX+F2CZtb2IwxZpvbJezcXN7K\nizHGbHG7hP3HH8Add2gdBWOMuR+3S9jZ2UBgoNZRMMaY+3G7hM0tbMYYs81tErbRCGzZAhw/zi1s\nxpj3yM9X71yaJmwhgP37axZveucd4KOPeNCRMeYd4uOBiAhg0yZ1zqfJ1PSsLFr8e9ky2oX4r3+l\nxVK6d3dFJIwxpp2dO4GxY2n51Zkz7c9+dIu1RIqLgVWrKEkfPUoLOo0bR1vs8GYDjDFvdv48NUpN\nJmqsXt/nxSbN1hK5epU2GXj0USAoiNaInT4dOHsWmD+fttbhZM0Y83b+/kByMnDffcCAAcD27c6d\nx6Ut7DZtBPr0oS6Pxx4Dbr7ZFa/EGGOeY/Nm4NlnaexuxgxaR9uSZi3sAwdoz8Xx4z0zWSvdGNOb\n8TXgawDwNQDUuwYPPki7Za1eDURHU3exUk4n7O3bt6NHjx7o2rUr5s2bZ/M5nl6ex7+kfA0AvgYA\nXwNA3WvQqRPtTdulC9C/P7Bvn7LjnE7YL7/8MhYuXIiffvoJn332GQoKCpw9FWOM+ZxmzYCEBCpl\nfvhhYMECKnWW49Ty2hcvXgQADB06FADwwAMPIDU1FX/+85+dOR1jjPmsxx8H+vShyrmdO+Wf69Sg\n408//YQlS5Zg+fLlAIAFCxYgNzcX7733Xs2JufyDMcacIpWWVd7Axv4LMsYYc45TfdgDBw7EsWPH\nqr9PT09HWFiYakExxhiz5lTCbt26NQCqFMnKysKPP/6I0NBQVQNjjDFWm9NdIp988gkmTpyIyspK\nvPTSS2jHGzEyxphLOV3WN2zYMGRkZODkyZN46aWX1IzJKTExMWjfvj169+5d6/4vvvgCPXr0QK9e\nvfD666/bPNZeTfmcOXPQqFEjFBUVWT2Wk5OD4cOHo1evXggPD0dSUlL1Y7///juefvpp9OzZE08+\n+STKy8vr+VPK0+oaVFRUIDQ0FCEhIQgLC0NCQkL1Y75yDcxMJhP0ej0iIyOr7/OlaxAUFIQ+ffpA\nr9dj0KBB1fd7wzWIj49HQEAA9Ho99Ho9kpOTHTpelWsgvMT27dvF/v37xd13311935EjR0RYWJg4\nfvy4EEKI8+fP2zw2JCREbNu2TWRlZYnu3buLCxcuVD+WnZ0tHnzwQREUFCQKCwutjs3LyxMHDhwQ\nQghx4cIF0blzZ1FaWiqEEGLs2LFi5cqVQgghPvjgA/Hpp5+q88NK0OoaCCHE5cuXhRBCVFRUiF69\neokTJ04IIXzrGgghxJw5c8RTTz0lIiMjq+/zpWsg9ZgnX4OCggIhhBDx8fFizpw5dl9b6ng1roHX\nJGwhhMjMzKz1H/TRRx+JxYsXyx5TUlIiQkJCqr9/8cUXxYYNG6q/f/zxx8WhQ4fs/qGajRo1SqSk\npAghhOjYsaMoLy8XQgiRnp4uxowZ49DP4wytr0FBQYHo3r27yM7OFkL41jXIyckR999/v0hJSRGj\nRo2qvt+XrkFQUFB1grLkDdcgPj5ezJ492+nj1bgGbrPjjCts2bIFv/32GwYMGIAJEybg6NGjAICz\nZ89WT/LZu3cvgoODq4/p2bMn9uzZAwBYu3YtAgIC0KdPn1rntTze0smTJ5Genl79UXDEiBFYunQp\nrl69ii+//BK7d+92yc8pp6GuQVVVFfr27Yv27dtj6tSp6NSpEwDfugbTpk3DrFmz0KjOaj6+dA10\nOh0iIiIQHR2NdevWVd/vDdcAAObNm4ewsDD8+9//RmlpqUPHq3ENvDphV1RUoKioCDt27EBUVBSm\nTp0KALjtttuwceNGyeN0Oh3Ky8vx/vvvY+bMmdX3i+u15baOLy0txZgxY5CQkIAbb7wRADBz5kz8\n9ttvCAsLg8lkwg033KD2j2hXQ12DRo0a4dChQzh58iQ+//xzHDhwAIDvXIMNGzbA398fer3eag6C\nr1wDANi1axcOHTqEDz74AHFxcTh37hwAz74GZrGxscjMzMTmzZtx6tQpLFy40KHjVbkGDrfJ3Vjd\nj0AzZsyo9ZHO8iOJWd2PMFOnThUbNmwQR44cEf7+/iIoKEgEBQWJJk2aiDvuuEPk5+dbve61a9fE\niBEjREJCgmRsmzZtEi+++GJ9fjxFtLoGlqZPny7mz59vdb83X4M333xTBAQEiKCgINGhQwfRokUL\nMW7cOKvYvPka1DVt2jSxaNEiq/s98RrUdfDgQTF48GCr+5Ue7+w18OqE/d1334kpU6aIqqoqsWfP\nHjFkyBCbx5kHCTIzM60GWsyk+u2qqqrEuHHjxLRp06weMw9qnDlzRowaNcrmf5zatLgGFy5cEMXF\nxUII6sPu3bu3OHv2rBDCd66BJYPBUKsP21euweXLl8WlS5eEEPQz9+zZs3oswxuugfl3urKyUrz2\n2mvin//8p0PHq3ENvCZhP/nkk6Jjx46iWbNmIiAgQCQmJgqj0SgmTpwogoODRXR0tEhLSxNCCJGb\nmytGjhxZfazBYBDBwcGiS5cuYu7cuTbP37lz5+pfUsvjd+zYIXQ6nejbt68ICQkRISEh4ocffhBC\nCDF37lzRrVs30bVrV/Gvf/3LlT++EEK7a3Do0CGh1+tFnz59xAMPPCC+/PLL6mN85RpYMhgMtapE\nfOUanDp1SvTt21f07dtXREREiCVLllQf4w3XYNy4caJ3796if//+Ytq0aZK/B1LHq3ENXLbjDGOM\nMXV59aAjY4x5E07YjDHmIThhM8aYh+CEzRhjHoITNmOMeQhO2Iwx5iH+HxfpdMwwfGC3AAAAAElF\nTkSuQmCC\n" } ], "prompt_number": 45 }, { "cell_type": "code", "collapsed": false, "input": [ "byte_counter, file_counter" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 46, "text": [ "(Counter({'arc.gz': 71106384571350L, 'metadata': 11010558690874L, 'textData': 6978342039325L, 'other': 1626219, 'success': 0}),\n", " Counter({'arc.gz': 856589, 'textData': 341525, 'metadata': 341517, 'success': 71, 'other': 17}))" ] } ], "prompt_number": 46 }, { "cell_type": "code", "collapsed": false, "input": [ "# http://stackoverflow.com/a/1823101/7782\n", "\n", "import locale\n", "locale.setlocale(locale.LC_ALL, 'en_US')\n", "\n", "locale.format(\"%d\", byte_counter['arc.gz'], grouping=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "pyout", "prompt_number": 47, "text": [ "'71,106,384,571,350'" ] } ], "prompt_number": 47 }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] } ], "metadata": {} } ] }