{
 "metadata": {
  "name": "",
  "signature": "sha256:4a51afdcc72d6c3700e33a0f3fa5a5570e51f803ccfcaeea43ab06c775c03da7"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Pyxplorer - interactive data set exploration\n",
      "\n",
      "The goal of pyxplorer is to provide a simple tool that allows interactive\n",
      "profiling of datasets that are accessible via a SQL like interface. The only\n",
      "requirement to run data profiling is that you are able to provide a Python\n",
      "DBAPI like interface to your data source and the data source is able to\n",
      "understand simplistic SQL queries.\n",
      "\n",
      "I built this piece of software while trying to get a better understanding of\n",
      "data distribution in a massive several hundred million record large dataset.\n",
      "Depending on the size of the dataset and the query engine the resposne time\n",
      "can ranging from seconds (Impala) to minutes (Hive) or even hourse (MySQL)\n",
      "\n",
      "The typical use case is to use `pyxplorer` interactively from an iPython\n",
      "Notebook or iPython shell to incrementally extract information about your data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "    $> pip install pyxplorer pympala"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Questions, Ideas, Comments:\n",
      "\n",
      "https://github.com/grundprinzip/pyxplorer"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Example using Impala\n",
      "\n",
      "Basically `pyexplorer` works with all DBAPI like interfaces, but to show the advantages of running a high-performance data analysis on large amounts of data we will use Impala to store our data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from impala.dbapi import connect\n",
      "conn = connect(host='diufpc57', port=21050)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Database Operations"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Imagine that you are provided with access to a huge Hive/Impala database on\n",
      "your very own Hadoop cluster and you're asked to profile the data to get a\n",
      "better understanding for performing more specific data science later on. \n",
      "Based on this connection, we can now instantiate a new explorer object."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pyxplorer as pxp\n",
      "data = pxp.Database(\"tpcc3\", conn)\n",
      "data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<table><tr><th>Name</th><th>Size</th></tr><tr><td>customerp</td><td>30000000</td></tr><tr><td>districtp</td><td>10000</td></tr><tr><td>historyp</td><td>30000000</td></tr><tr><td>itemp</td><td>100000</td></tr><tr><td>new_orderp</td><td>9000000</td></tr><tr><td>oorderp</td><td>30000000</td></tr><tr><td>order_linep</td><td>299991280</td></tr><tr><td>stockp</td><td>100000000</td></tr><tr><td>warehousep</td><td>1000</td></tr></table><p>Rows: 9 / Columns: 2</p>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "<pyxplorer.manager.Database instance at 0x1033a5bd8>"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This simple code gives you access to all the tables in this database. Let's further investigate how many tables and columns exist in the database."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "9"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The above is the idiomatic python way, but sometimes, it might not be as easy to gasp what is meant, which gives you the chance to use as well"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.num_tables()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "9"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Get the total number of columns:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sum([len(x.columns()) for x in data.tables()])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "92"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Or we can directly use the number of columns method on the database object"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.num_columns()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "92"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It seems like we have a better understanding of the dataset, but how many tuples are we talking about?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.num_tuples()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "499102280"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Singel Table Operation"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Using the above operations, we can perform simple operations on all tables, but let's have a further look at single table operations to extract more information from instances.\n",
      "\n",
      "In this example, we want to investigate the `order_line` table."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab = data['order_linep']\n",
      "tab"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "<Table: \"tpcc3.order_linep\">"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's start by doing some basic inspection of the table, like extracting the number of rows and the number of columns"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.size()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "299991280"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(tab.columns())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "10"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.columns()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "[ol_w_id,\n",
        " ol_d_id,\n",
        " ol_o_id,\n",
        " ol_number,\n",
        " ol_i_id,\n",
        " ol_delivery_d,\n",
        " ol_amount,\n",
        " ol_supply_w_id,\n",
        " ol_quantity,\n",
        " ol_dist_info]"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Columns are special objects that can be easily and interactively inspected in iPython Notebooks, the default information per column are the `min` and `max` value, the most frequent and least frequent value and the total number of distinct values. Based on these measrues we provide information about the column.\n",
      "\n",
      "$uniqueness = \\frac{distinct}{rows}$\n",
      "\n",
      "$constancy = \\frac{count_{mf}}{rows}$"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab['ol_w_id']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<table><tr><th>Name</th><th>Value</th></tr><tr><td>Min</td><td>1</td></tr><tr><td>Max</td><td>1000</td></tr><tr><td>#Distinct Values</td><td>1000</td></tr><tr><td>Most Frequent</td><td>109 (301593)</td></tr><tr><td>Least Frequent</td><td>212 (298395)</td></tr><tr><td>Top 10 MF</td><td>109,676,117,460,19,877,165,764,340,689</td></tr><tr><td>Top 10 LF</td><td>212, 405, 52, 284, 304, 769, 727, 665, 90, 163</td></tr><tr><td>Uniqueness</td><td>3.33343022504e-06</td></tr><tr><td>Constancy</td><td>0.00100533922186</td></tr></table><p>Rows: 9 / Columns: 2</p>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "ol_w_id"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Its possible to access the column either using subscript notation or directly as an attribute of the object"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.ol_w_id"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<table><tr><th>Name</th><th>Value</th></tr><tr><td>Min</td><td>1</td></tr><tr><td>Max</td><td>1000</td></tr><tr><td>#Distinct Values</td><td>1000</td></tr><tr><td>Most Frequent</td><td>109 (301593)</td></tr><tr><td>Least Frequent</td><td>212 (298395)</td></tr><tr><td>Top 10 MF</td><td>109,676,117,460,19,877,165,764,340,689</td></tr><tr><td>Top 10 LF</td><td>212, 405, 52, 284, 304, 769, 727, 665, 90, 163</td></tr><tr><td>Uniqueness</td><td>3.33343022504e-06</td></tr><tr><td>Constancy</td><td>0.00100533922186</td></tr></table><p>Rows: 9 / Columns: 2</p>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 21,
       "text": [
        "ol_w_id"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Based on this information we can further deduct what role this column might have in the overall schema. For example, based on the uniqueness we can say that the column is not suitable to uniquely identify every row. In additiona, based on the constancy of the most frequent value and the spread between the most and least frequent value we can deduct that the data is almost uniformly distributed."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Distinct Values and Distinct Value Distribution\n",
      "\n",
      "One important feature ist to look at the distinct values and their distribution"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.ol_w_id.dcount()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 22,
       "text": [
        "1000"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "dist = tab.ol_w_id.distribution(limit=10000)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline\n",
      "dist.fraction.hist()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "<matplotlib.axes.AxesSubplot at 0x10c550ad0>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAD9CAYAAABN7FvjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHJBJREFUeJzt3X9Qk/cdB/B3BG79Qy1SS7QJWyyEYgQJ0wF1s8YT7LSW\n2brpqF1j1d3WTrfOXR27XW/tHxP8o+1ZW9tut270vPPHzhXcbTJs6+O2bor1x11PPMUZlJ9ZXaDi\nuorKd38gkVTliUmefJ/ny/t1x5UnIXw+nybf50O+n8TYhBACREREIxgjOwEiIjI/NgsiItLFZkFE\nRLrYLIiISBebBRER6WKzICIiXbrNoq2tDXPnzsW0adOQn5+PV155BQDw/PPPw+l0oqioCEVFRdiz\nZ0/4NtXV1XC73cjLy0NjY6Nx2RMRUVLY9N5n0d3dje7ubni9Xly8eBEzZsxAXV0ddu7ciXHjxmHd\nunURP9/c3IzHHnsMhw4dQkdHB8rKynDq1CmMGcMnMUREVqV7Bp80aRK8Xi8AYOzYsZg6dSo6OjoA\nADfrM/X19aisrERaWhpcLhdycnLQ1NSU4LSJiCiZUm/nh1tbW3H06FGUlpbigw8+wObNm/H2229j\n5syZePHFF5Geno7Ozk6UlpaGb+N0OsPNZTibzRZ/9kREo4ysf3Qj6r2hixcv4pvf/CY2bdqEsWPH\n4qmnnkIgEMCxY8cwefJk/OQnP7nlbW/VGIQQSn794he/kJ4D62N9rE+9L5miahaXL1/GkiVL8Pjj\nj2Px4sUAgMzMTNhsNthsNqxevTq81eRwONDW1ha+bXt7OxwOhwGpm1dra6vsFAzF+qyN9VEsdJuF\nEAKrVq2Cx+PBM888E768q6sr/P0777yDgoICAEBFRQW2b9+O/v5+BAIBtLS0oLi42IDUiYgoWXRn\nFh988AG2bt2K6dOno6ioCACwYcMGbNu2DceOHYPNZsOUKVPw5ptvAgA8Hg+WLl0Kj8eD1NRUbNmy\nZdTNJ1asWCE7BUOxPmtjfRQL3ZfOGhbYZpO+B0dEZCUyz5t884MBNE2TnYKhWJ+1sT6KBZsFERHp\n4jYUEZFFcBuKiIhMjc3CAKrvmbI+a2N9FAs2CyIi0sWZBRGRRXBmQUREpsZmYQDV90xZn7WxPooF\nmwUREenizIKIyCI4syAiIlNjszCA6numrM/aWB/Fgs2CiIh0cWZBRGQRnFkQEZGpsVkYQPU9U9Zn\nbayPYsFmQUREujizINMaPz4DfX09SY87btwEXLgQSnpcIj0yz5tsFmRaNpsNgIzHCB+bZE4ccCtG\n9T1T1etTner3n+r1ycJmQUREurgNRaYlbxsqDcCVpEflrIT0cGZBdBMyZxaclZAZcWahGNX3TFWv\nT3Wq33+q1ycLmwUREeniNhSZFrehiCJxG4qIiEyNzcIAqu+Zql6f6lS//1SvTxY2CyIi0sWZBZkW\nZxZEkTizICIiU2OzMIDqe6aq16c61e8/1euThc2CiIh0cWZBpsWZBVEkU88s2traMHfuXEybNg35\n+fl45ZVXAAChUAjl5eXIzc3F/Pnz0dvbG75NdXU13G438vLy0NjYaFz2RESUFLrNIi0tDS+//DKO\nHz+OAwcO4LXXXsOJEydQU1OD8vJynDp1CvPmzUNNTQ0AoLm5GTt27EBzczMaGhrw9NNPY2BgwPBC\nzET1PVPV61Od6vef6vXJotssJk2aBK/XCwAYO3Yspk6dio6ODuzevRt+vx8A4Pf7UVdXBwCor69H\nZWUl0tLS4HK5kJOTg6amJgNLICIio6Xezg+3trbi6NGjKCkpQTAYhN1uBwDY7XYEg0EAQGdnJ0pL\nS8O3cTqd6OjouOnvW7FiBVwuFwAgPT0dXq8XPp8PwPW/Dqx47PP5TJWPleu7bujYp/jxtSNF7j/V\nH5/JePxrmobW1lbIFvWA++LFi5gzZw6ee+45LF68GBMmTEBPT0/4+oyMDIRCIaxduxalpaVYvnw5\nAGD16tVYuHAhHn300cjAHHCTDg64iSKZesANAJcvX8aSJUvwne98B4sXLwYw+Gyiu7sbANDV1YXM\nzEwAgMPhQFtbW/i27e3tcDgcic7b1G78q1gtqtenOtXvP9Xrk0W3WQghsGrVKng8HjzzzDPhyysq\nKlBbWwsAqK2tDTeRiooKbN++Hf39/QgEAmhpaUFxcbFB6RMRUTLobkP9/e9/xwMPPIDp06df2xYY\nfGlscXExli5dinPnzsHlcmHnzp1IT08HAGzYsAFvvfUWUlNTsWnTJjz44IM3BuY2FOngNhRRJH4G\nN9FNsFkQRTL9zIJuj+p7pqrXpzrV7z/V65OFzYKIiHRxG4pMi9tQRJG4DUVERKbGZmEA1fdMVa9P\ndarff6rXJwubBRER6eLMgkyLMwuiSJxZEBGRqbFZGED1PVPV61Od6vef6vXJwmZBRES6OLMg0+LM\ngigSZxZERGRqbBYGUH3PVPX6VKf6/ad6fbKwWRARkS7OLMi0OLMgisSZBRERmRqbhQFU3zNVvT7V\nqX7/qV6fLGwWRESkizMLMi3OLIgiyTxvpkqJSpYzfnwG+vp6ZKdBRJJwG8oAKu6ZDjYKce1r37Dv\njfwiI6j4+BxO9fpkYbMgIiJdnFlQVOTMDzizIBqO77MgIiJTY7MwgPp7pprsBCgOqj8+Va9PFjYL\nIiLSxZkFRYUzi+TE5ZqgkXBmQUREpsZmYQD190w12QlQHFR/fKpenyxsFkREpIszC4oKZxbJics1\nQSPhzIKIiEyNzcIA6u+ZarIToDio/vhUvT5Z2CyIiEgXZxYUFc4skhOXa4JGwpkFERGZWlTNYuXK\nlbDb7SgoKAhf9vzzz8PpdKKoqAhFRUXYs2dP+Lrq6mq43W7k5eWhsbEx8VmbnPp7pprsBCgOqj8+\nVa9PlqiaxZNPPomGhoaIy2w2G9atW4ejR4/i6NGjWLBgAQCgubkZO3bsQHNzMxoaGvD0009jYGAg\n8ZkTEVHSRNUsZs+ejQkTJtxw+c32zurr61FZWYm0tDS4XC7k5OSgqakp/kwtxOfzyU7BYD7ZCVAc\nVH98ql6fLHF9BvfmzZvx9ttvY+bMmXjxxReRnp6Ozs5OlJaWhn/G6XSio6PjprdfsWIFXC4XACA9\nPR1erzd8Rw89leSxOY4HabjeKLRr/zX6GDrXq3Z87chk9z+P5RwPfd/a2grpRJQCgYDIz88PHweD\nQTEwMCAGBgbEz3/+c7Fy5UohhBBr1qwRW7duDf/cqlWrxK5du274fbcR2nL27dsnO4WEAyAAce1r\n37DvjfxCkuKYJ24yqPj4HE7l+mSeN2N+ZpGZmRn+fvXq1Xj44YcBAA6HA21tbeHr2tvb4XA4Yg1D\nNIqkXnuJcnKNGzcBFy6Ekh6XrCXml852dXWFv3/nnXfCr5SqqKjA9u3b0d/fj0AggJaWFhQXF8ef\nqYWov2fqk52Aoq5g8P0dyf3q6+tJSnXJov76kyOqZxaVlZXYv38/zp8/j6ysLLzwwgvQNA3Hjh2D\nzWbDlClT8OabbwIAPB4Pli5dCo/Hg9TUVGzZskXKX0tERJQ4fAe3ATRNU+6vm8h3cGtIzrOL0fcO\n7uTE1RB5/6m1FlVcf0P4Dm4iIjI1PrOgqPDfhlI7LteiNfCZBRERmRqbhQHU/7dpNNkJUFw02QkY\nSv31JwebBRER6eLMgqLCmYXacbkWrYEzCyIiMjU2CwOov2eqyU6A4qLJTsBQ6q8/OdgsiIhIF2cW\nFBXOLNSOy7VoDZxZEBGRqbFZGED9PVNNdgIUF012AoZSf/3JwWZBRES6OLOgqHBmoXZcrkVr4MyC\niIhMjc3CAOrvmWqyE6C4aLITMJT6608ONgsiItLFmQVFhTMLteNyLVoDZxZERGRqbBYGUH/PVJOd\nAMVFk52AodRff3KwWRARkS7OLCgqnFmoHZdr0Ro4syAiIlNjszCA+nummuwEKC6a7AQMpf76k4PN\ngoiIdHFmQVHhzELtuFyL1sCZBRERmRqbhQHU3zPVZCdAcdFkJ2Ao9defHGwWRESkizMLigpnFmrH\n5Vq0Bs4siIjI1NgsDKD+nqkmOwGKiyY7AUOpv/7kYLMgIiJdnFlQVDizUDsu16I1cGZBRESmxmZh\nAPX3TDXZCVBcNNkJGEr99SdHVM1i5cqVsNvtKCgoCF8WCoVQXl6O3NxczJ8/H729veHrqqur4Xa7\nkZeXh8bGxsRnTURESRXVzOJvf/sbxo4diyeeeAIfffQRAGD9+vWYOHEi1q9fj40bN6Knpwc1NTVo\nbm7GY489hkOHDqGjowNlZWU4deoUxoyJ7EucWVgLZxZqx+VatAbTzyxmz56NCRMmRFy2e/du+P1+\nAIDf70ddXR0AoL6+HpWVlUhLS4PL5UJOTg6ampoSnDYRESVTaqw3DAaDsNvtAAC73Y5gMAgA6Ozs\nRGlpafjnnE4nOjo6bvo7VqxYAZfLBQBIT0+H1+uFz+cDcH3f0YrHw/dMzZBPIo4HaQB8iNzz9g27\nzohj6Fyv2jF0rk/E8fBYg9fLfnxx/d38eOj71tZWyBb1S2dbW1vx8MMPh7ehJkyYgJ6envD1GRkZ\nCIVCWLt2LUpLS7F8+XIAwOrVq7Fw4UI8+uijkYEV3obSNO1zJ1nri9yG0nD9JGRoVIy27aDkxNUQ\nef+ptRZVXH9DTL8NdTN2ux3d3d0AgK6uLmRmZgIAHA4H2trawj/X3t4Oh8MRZ5rWouoD9Tqf7AQo\nLj7ZCRhK/fUnR8zNoqKiArW1tQCA2tpaLF68OHz59u3b0d/fj0AggJaWFhQXFycmWyIikiKqZlFZ\nWYlZs2bh5MmTyMrKwm9/+1tUVVVh7969yM3Nxfvvv4+qqioAgMfjwdKlS+HxeLBgwQJs2bLl2hbG\n6KH+67w12QlQXDTZCRhK/fUnB/+5DwOouGfKmYVKcTVwZmFNMs+bbBYUFb7PQu24XIvWYMkBNxER\njR5sFgZQf89Uk50AxUWTnYCh1F9/crBZEBGRLs4sKCqcWagdl2vRGjizICIiU2OzMID6e6aa7AQo\nLprsBAyl/vqTg82CiIh0cWZBUeHMQu24XIvWwJkFERGZGpuFAdTfM9VkJ0Bx0T53nAqbzZbUr/Hj\nM4yrTvn1J0fMH35ERKq4gmRvf/X1ja5/XFQFnFlQVDizYNxEx+T6v32cWRARkamxWRjAyD3T8eMz\nkr6/fOPnkRhXHyWDJjsBQ3FmYQzOLCymr68H8rYqiGi04szCYuTMDgBZ+9qjp9bRFpfrPxacWRAR\nkamxWRhA/T1TTXYCFBdNdgKGUn/9ycFmQUREujizsBjOLBhXjbhc/7HgzIKIiEyNzcIA6u+ZarIT\noLhoshMwlPrrTw42CyIi0sWZhcVwZsG4asTl+o8FZxZERGRqbBYGUH/PVJOdAMVFk52AodRff3Kw\nWRARkS7OLCyGMwvGVSMu138sOLMgIiJTY7MwgPp7pprsBCgumuwEDKX++pODzYKIiHRxZmExnFkw\nrhpxuf5jwZkFERGZGpuFAdTfM9VkJ0Bx0WQnYCj1158ccX8Gt8vlwvjx45GSkoK0tDQ0NTUhFAph\n2bJlOHv2LFwuF3bu3In09PRE5EtERBLEPbOYMmUKDh8+jIyMjPBl69evx8SJE7F+/Xps3LgRPT09\nqKmpiQzMmUVMOLNgXDXicv3HwvIzi88nv3v3bvj9fgCA3+9HXV1dIsIQEZEkcW9D2Ww2lJWVISUl\nBd/73vfw3e9+F8FgEHa7HQBgt9sRDAZvetsVK1bA5XIBANLT0+H1euHz+QBc33e04vHwPdNE//7r\nho59SToeusyHyD1vo+ND53rVjqFzfSKOh8cy4vdHczz4mLba+kv28dD3ra2tkC3ubaiuri5MnjwZ\nH3/8McrLy7F582ZUVFSgp6cn/DMZGRkIhUKRgRXehhq+CBLNHNtQGiKbSDJiJpPqcTVE3n9qbUMZ\nuf5ks/Q21OTJkwEAd999Nx555BE0NTXBbreju7sbwGAzyczMjDeMpaj6QL3OJzsBiotPdgKGUn/9\nyRFXs/j000/R19cHAPjvf/+LxsZGFBQUoKKiArW1tQCA2tpaLF68OP5MiYhImriaRTAYxOzZs+H1\nelFSUoJFixZh/vz5qKqqwt69e5Gbm4v3338fVVVVicrXEtR/nbcmOwGKiyY7AUOpv/7kiGvAPWXK\nFBw7duyGyzMyMvDuu+/G86uJiMhE+G9DWYw5Btwqx2TcZMXk+r99lh5wExGR+tgsDKD+nqkmOwGK\niyY7AUOpv/7kYLMgIiJdnFlYDGcWjKtGXK7/WHBmQUREpsZmYQD190w12QlQXDTZCRhK/fUnB5sF\nERHp4szCYjizYFw14nL9x4IzCyIiMjU2CwOov2eqyU6A4qLJTgBAKmw2W9K/xo/P0E+NborNgogk\nuILBrS8jvvbd8rq+vuufs0O3hzMLi+HMgnHViCuvViufd2SeN+P+WNXRavz4DP6VQkSjBrehYjTY\nKG7/aXD8X2agyU6A4qLJTsBgmuwElMRmQUREujiziNHomh3Iijuaah1tcTmziAXfZ0FERKbGZmEI\nTXYCBtNkJ0Bx0WQnYDBNdgJKYrMgIiJdnFnEiDMLVWMyrroxB+Na/bzDmQUREZkWm4UhNNkJGEyT\nnQDFRZOdgME02Qkoic2CiIh0cWYRI84sVI3JuOrGHIxr9fMOZxZERGRabBaG0GQnYDBNdgIUF012\nAgbTZCegJDYLIiLSxZlFjDizUDUm46obczCu1c87/DwLIiLDpV77Qy+5xo2bgAsXQkmPm0jchjKE\nJjsBg2myE6C4aLITMJg2wnVGfpzrrb9U+KA0yz+zuHz5Mh566FsIhS7KToWISFmWn1lcuHABd901\nCVeu7E5AVrejHKNnf1lW3NFU62iLO5pqHYybiPMdZxZxGjMmFUCZ7DSIiJTFmYUhNNkJGEyTnQDF\nRZOdgME02Qkoic3CEMdkJ2Aw1etTner3n+r1yWFYs2hoaEBeXh7cbjc2btxoVBiT6pWdgMFUr091\nqt9/qtcnhyHN4urVq1izZg0aGhrQ3NyMbdu24cSJE0aEIiKiJDCkWTQ1NSEnJwculwtpaWn49re/\njfr6eiNCmVSr7AQM1io7AYpLq+wEDNYqOwElGfJqqI6ODmRlZYWPnU4nDh48eMPPJfadlMl/V+bI\nMWslxTXS8LhG1nermMmketzP339mWz/xGunxKee+lfHO8UQypFlE8z/Fyv8+CxHRaGPINpTD4UBb\nW1v4uK2tDU6n04hQRESUBIY0i5kzZ6KlpQWtra3o7+/Hjh07UFFRYUQoIiJKAkO2oVJTU/Hqq6/i\nwQcfxNWrV7Fq1SpMnTrViFBERJQEus8sonm/xA9/+EO43W4UFhbi6NGjAIAFCxZg06ZNSE1NxVtv\nvRVx21AohPLycuTm5mL+/Pno7R18XXR/fz+efPJJTJ8+HV6vF/v37w/fZseOHSgsLER+fj6qqqpu\nyGHXrl0YM2YMjhw5EnXxsdY20m0TXdvOnTsxbdo05OfnY/ny5VHXZoX6Tp8+jdmzZ6OoqAiFhYXY\ns2eP9Pp+//vfY9q0aUhJSbnhsVRdXQ232428vDw0NjaGLz98+DAKCgrgdrvxox/9KHz5pUuXsGzZ\nMrjdbpSWluLs2bNK1ffSSy9h2rRpKCwsRFlZGc6dO6dUfUPMcm5JdG23fW4RI7hy5YrIzs4WgUBA\n9Pf3i8LCQtHc3BzxM3/605/EggULhBBCHDhwQJSUlOje9tlnnxUbN24UQghRU1MjfvrTnwohhHj1\n1VfFypUrhRBC/Pvf/xYzZswQQghx/vx58cUvflGcP39eCCGE3+8X7733XjiHCxcuiNmzZ4v7779f\nHD58eKSSLFXbqVOnRFFRkejt7RVCCPHxxx9HVZtV6vP7/eKNN94QQgjR3NwsXC6X9PpOnDghTp48\nKXw+X8Rj6fjx46KwsFD09/eLQCAgsrOzxcDAgBBCiK985Svi4MGDQgghFixYIPbs2SOEEOK1114T\nTz31lBBCiO3bt4tly5YpVd++ffvE//73PyGEEK+//rpy9QlhrnNLImuL5dwy4jOLaN4vsXv3bvj9\nfgBASUkJent70d3dPeJth9/G7/ejrq4OAHDixAnMnTsXAHD33XcjPT0dhw4dwpkzZ+B2u3HXXXcB\nAObNm4ddu3aFc3juuedQVVWFL3zhC1G/ysoKtf3617/GmjVrcOeddwIAJk6cGFVtVqlv8uTJ+OST\nTwAAvb29cDgc0uvLy8tDbm7uDfHq6+tRWVmJtLQ0uFwu5OTk4ODBg+jq6kJfXx+Ki4sBAE888UT4\n/8nw+EuWLMF7772nVH0+nw933HFHOH57e7tS9QHmOrcksrZYzi0jNoubvV+io6Mjqp/p7Oy85W2D\nwSDsdjsAwG63IxgMAgAKCwuxe/duXL16FYFAAIcPH0Z7ezvcbjdOnjyJs2fP4sqVK6irqws/MI8c\nOYKOjg4sXLgQQPSvZbZCbS0tLTh58iS+9rWv4f7778df/vKXqGoze31Dr5T72c9+htraWmRlZeGh\nhx7C5s2bpdd3K52dnRGv6Bv+u4Zf7nA4wr9rePzU1FTceeedCIWi+7Q0K9Q33G9+85vwGlSlPrOd\nWxJZWyznlhEH3NH+z4mm4wohbvr7bDZb+PKVK1fixIkTmDlzJr70pS9h1qxZSElJQXp6Ol5//XUs\nW7YMY8aMwaxZs3DmzBkIIbBu3TrU1tZGxImG2WsDBj/Y6fTp09i/fz/a2trwwAMP4KOPPgr/NWDV\n+v71r38BANatW4fVq1fjxz/+MQ4cOIDHH38cx48fjyrvRNZnRlaqb+vWrThy5AhefvnlqG9j9vrM\ncm4xSiznlhGbRTTvl/j8z7S3t8PpdOLy5cs3XD60zWC329Hd3Y1Jkyahq6sLmZmZAICUlBS89NJL\n4dt89atfDT/tWrRoERYtWgQA+NWvfoXU1FT09fXh+PHj8Pl8AIDu7m5UVFTgj3/8I7785S+PVJrp\nawOArKwslJSUICUlBS6XC7m5uTh9+jRmzJgxYm1Wqe8f//gHXnjhBQBAaWkpPvvsM5w/fz6qp8SJ\nrC+a9wHd6nc5HI6I7Zehy4duc+7cOdxzzz24cuUKPvnkE2RkZOjWZvb6hm8Xvvvuu9iwYQP++te/\nIi0tLarazF6f0+k0zbnFqMdmTOeWkQYaly9fFvfee68IBALi0qVLuoOaf/7zn+FBzUi3ffbZZ0VN\nTY0QQojq6urwkPTTTz8VFy9eFEII0djYKObMmROOEwwGhRBChEIh4fV6RUtLyw35fn7wY/XaGhoa\nhN/vF0IMDqCysrJEKBRSpr5HHnlE/O53vxNCDA6477nnnqhqM7K+IT6fT3z44Yfh46Eh4qVLl8SZ\nM2fEvffeGx4iFhcXiwMHDoiBgYEbBtzf//73hRBCbNu27bYGwFao78iRIyI7O1ucPn066rqsVN/n\nf5/sc0sia4vl3DJisxBCiD//+c8iNzdXZGdniw0bNgghhHjjjTfCr2IRQogf/OAHIjs7W0yfPj3i\nf+jNbiuEEP/5z3/EvHnzhNvtFuXl5aKnp0cIIUQgEBD33XefmDp1qigvLxfnzp0L36ayslJ4PB7h\n8XjEjh07bprr7dyhVqlt3bp1wuPxiIKCglvWbdX6Tp8+LebMmSMKCwuF1+sVe/fulV7fH/7wB+F0\nOsUdd9wh7Ha7+PrXvx6+7pe//KXIzs4W9913n2hoaAhf/uGHH4r8/HyRnZ0t1q5dG778s88+E9/6\n1rdETk6OKCkpEYFAQKn6ysrKxKRJk4TX6xVer1d84xvfUKq+4cxwbkl0bbd7bpH2GdxERGQd/KQ8\nIiLSxWZBRES62CyIiEgXmwUREelisyAiIl1sFkREpOv/gLsNdbFXPv4AAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x10c5506d0>"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The above example validates our assumption of unifrom distribution as the histograms spreads basically from $0.0009$ to $0.001006$.\n",
      "\n",
      "In addition to performing such analysis on a column level, we can have a look at the distributions as well from a higher level. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.dcounts().fraction.hist()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "<matplotlib.axes.AxesSubplot at 0x109958510>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD9CAYAAABOd5eOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFZNJREFUeJzt3XFs1PX9x/FXtfwy5qZtlX5x0PzKkKaUQnsbk7lNOVav\nKs6uTMI0hpzo+BnNsuF/zZbFbVnkxv4YTvfH/mBwMZsb2RbsMiSj0kMdKhulkCBS4worrL3JjrNM\nlJb2+/uDth8Y9Hq9b+/7/fZ7z0fS0M/12nv3lfpu+7rrWWTbti0AwLR2jdcDAACcY5kDQACwzAEg\nAFjmABAALHMACACWOQAEwITLPJ1Oa/Xq1Vq4cKFqamr05ptvKpVKKRKJqKqqSo2NjUqn027MCgAY\nx4TL/Nvf/rZWrlypo0eP6vDhw6qurlYsFlMkElFXV5caGhoUi8XcmBUAMI6iTH809P777ysUCunv\nf//7ZZdXV1dr7969sixLfX19CofDevvtt/M+LADg6oozvbG7u1uzZs3SunXrdOjQIX32s5/V5s2b\nlUwmZVmWJMmyLCWTySvet6ioKD8TA0DA5fKH+RlrlgsXLqijo0NPPPGEOjo6dN11111RqRQVFY27\nuG3b5sW29dRTT3k+g19eyIIsyCLzS64yLvO5c+dq7ty5+tznPidJWr16tTo6OjR79mz19fVJknp7\ne1VeXp7zAIXg+PHjXo/gG2RhkIVBFs5lXOazZ89WRUWFurq6JEltbW1atGiR7rvvPsXjcUlSPB5X\nc3Nz/icFAIwrY2cuSc8++6weeughDQwMaP78+dq6dauGhoa0Zs0abdmyRZWVldq+fbsbs05bDz/8\nsNcj+AZZGGRhkIVzGR/N4ugDFxU56n8AoBDlujv5C1AXJBIJr0fwDbIwyMIgC+cmrFmc2LRpUz4/\nfFZCoZAikYjXYwBAXuW1Zrn22jUqKvrffHz4rNj2CdXVvacDB/Z4NgMATEauNUtefzIfGvo/SQ35\nvIkJ7JFt/8jD2wcAd9CZu4A+0CALgywMsnCOZQ4AAZDXzlxqk9c1Syj0I3V00JkDmB54aCIAFDCW\nuQvoAw2yMMjCIAvnWOYAEAB05gDgI3TmAFDAWOYuoA80yMIgC4MsnGOZA0AA0JkDgI/QmQNAAWOZ\nu4A+0CALgywMsnCOZQ4AAUBnDgA+QmcOAAWMZe4C+kCDLAyyMMjCOZY5AAQAnTkA+AidOQAUMJa5\nC+gDDbIwyMIgC+dY5gAQAHTmAOAjdOYAUMAmXOaVlZVasmSJQqGQbr31VklSKpVSJBJRVVWVGhsb\nlU6n8z7odEYfaJCFQRYGWTg34TIvKipSIpHQwYMHtX//fklSLBZTJBJRV1eXGhoaFIvF8j4oAGB8\nE3bm8+bN09/+9jfdeOONY5dVV1dr7969sixLfX19CofDevvtty//wHTmADBpuXbmxdl84DvvvFPX\nXnutHnvsMa1fv17JZFKWZUmSLMtSMpkc5703SXp15PUSSfWSwiPnxMi/+Tx3jk0y+mtcOBzmzJkz\nZ9+cE4mEtm3bJulirZ0zewL//Oc/bdu27X/96192XV2d/corr9glJSWXXae0tPSK95NkS222ZHv4\n8rIdCq2Y6FPMu/b2dq9H8A2yMMjCIAsji7V8VRN25jfffLMkadasWVq1apX2798/Vq9IUm9vr8rL\ny3P/bgIAcCzjMj937pzOnj0rSfrggw/05z//WYsXL1ZTU5Pi8bgkKR6Pq7m5Of+TTmOjv1qBLC5F\nFgZZOJexM08mk1q1apUk6cKFC3rooYfU2NiopUuXas2aNdqyZYsqKyu1fft2V4YFAFxdxmU+b948\ndXZ2XnF5WVmZ2tra8jZU0CQSCX7yGEEWBlkYZOEcfwEKAAHAc7MAgI/w3CwAUMBY5i4Y/QMBkMWl\nyMIgC+dY5gAQAHTmAOAjdOYAUMBY5i6gDzTIwiALgyycY5kDQADQmQOAj9CZA0ABY5m7gD7QIAuD\nLAyycI5lDgABQGcOAD5CZw4ABYxl7gL6QIMsDLIwyMI5ljkABACdOQD4CJ05ABQwlrkL6AMNsjDI\nwiAL51jmABAAdOYA4CN05gBQwFjmLqAPNMjCIAuDLJxjmQNAANCZA4CP0JkDQAFjmbuAPtAgC4Ms\nDLJwLqtlPjQ0pFAopPvuu0+SlEqlFIlEVFVVpcbGRqXT6bwOCQDILKtl/swzz6impmakB5disZgi\nkYi6urrU0NCgWCyW1yGnu3A47PUIvkEWBlkYZOHchMv85MmT2rlzp77xjW+MlfKtra2KRqOSpGg0\nqh07duR3SgBARsUTXeHJJ5/UT37yE/X3949dlkwmZVmWJMmyLCWTyXHee5OkV0deL5FULyk8ck6M\n/JvPc+fYJKOd3OhPAG6eL+0Dvbh9P51HL/PLPF6eOzs7tWHDBt/M4+V58+bNqq+v9808bp4TiYS2\nbdsmSaqsrFTO7Az++Mc/2k888YRt27bd3t5uf+UrX7Ft27ZLSkouu15paekV7yvJltpsyfbw5WU7\nFFqR6VN0RXt7u9cj+AZZGGRhkIUxwVoeV8afzPft26fW1lbt3LlTH330kfr7+7V27VpZlqW+vj7N\nnj1bvb29Ki8vz/27SQEY/W4MsrgUWRhk4VzGzvzpp59WT0+Puru79Zvf/EZf/vKX9fzzz6upqUnx\neFySFI/H1dzc7MqwAICrm9TjzEcfzdLS0qLdu3erqqpKe/bsUUtLS16GC4pL++JCRxYGWRhk4dyE\nd4COWr58uZYvXy5JKisrU1tbW96GAgBMDs/NAgA+wnOzAEABY5m7gD7QIAuDLAyycI5lDgABQGcO\nAD5CZw4ABYxl7gL6QIMsDLIwyMI5ljkABACdOQD4CJ05ABQwlrkL6AMNsjDIwiAL51jmABAAdOYA\n4CN05gBQwFjmLqAPNMjCIAuDLJxjmQNAANCZA4CP0JkDQAFjmbuAPtAgC4MsDLJwjmUOAAFAZw4A\nPkJnDgAFjGXuAvpAgywMsjDIwjmWOQAEAJ05APgInTkAFDCWuQvoAw2yMMjCIAvnir0eAAD84Prr\ny3T27Bmvx8hZxs78o48+0vLly3X+/HkNDAzoq1/9qjZu3KhUKqWvf/3rOnHihCorK7V9+3aVlJRc\n/oHpzAFMIxd3Vl7uQpykPHTmH/vYx9Te3q7Ozk4dPnxY7e3teu211xSLxRSJRNTV1aWGhgbFYrGc\nxwYAODdhZ/7xj39ckjQwMKChoSGVlpaqtbVV0WhUkhSNRrVjx478TjnN0QcaZGGQhUEWzk3YmQ8P\nD+szn/mM3n33XT3++ONatGiRksmkLMuSJFmWpWQyOc57b5L06sjrJZLqJYVHzomRf/N57hybZPSL\nJRwOc/bwPMov83h57uzs9NU8Xp47Ozt9MY8xeg67cE5I2jZyrlSusn6c+fvvv6+77rpLGzdu1Ne+\n9jWdOWPuKCgrK1Mqlbr8A9OZA5hGAt2ZX+qGG27QvffeqwMHDsiyLPX19UmSent7VV5ePukbBgBM\nnYzL/PTp00qn05KkDz/8ULt371YoFFJTU5Pi8bgkKR6Pq7m5Of+TTmNX/gpXuMjCIAuDLJzL2Jn3\n9vYqGo1qeHhYw8PDWrt2rRoaGhQKhbRmzRpt2bJl7KGJAADv8NwsAKAC6swBAP7FMncBfaBBFgZZ\nGGThHMscAAKAzhwARGcOAPABlrkL6AMNsjDIwiAL51jmABAAdOYAIDpzAIAPsMxdQB9okIVBFgZZ\nOMcyB4AAoDMHANGZAwB8gGXuAvpAgywMsjDIwjmWOQAEAJ05AIjOHADgAyxzF9AHGmRhkIVBFs6x\nzAEgAOjMAUB05gAAH2CZu4A+0CALgywMsnCOZQ4AAUBnDgCiMwcA+ADL3AX0gQZZGGRhkIVzLHMA\nCAA6cwBQwDvznp4erVixQosWLVJtba1+9rOfSZJSqZQikYiqqqrU2NiodDqd28wAgCmRcZnPmDFD\nP/3pT3XkyBG98cYb+vnPf66jR48qFospEomoq6tLDQ0NisVibs07LdEHGmRhkIVBFs5lXOazZ89W\nfX29JOkTn/iEFi5cqFOnTqm1tVXRaFSSFI1GtWPHjvxPCgAYV3G2Vzx+/LgOHjyoZcuWKZlMyrIs\nSZJlWUomk+O81yZJr468XiKpXlJ45JwY+Tef586xSUa/84fDYdfP4XDY09vn7N/zKL/M49V59DKv\n5zFGz2EXzglJ20bOlcpVVneA/uc//9Hy5cv1ve99T83NzSotLdWZM2fG3l5WVqZUKnX5B+YOUADT\nSKDvAJWkwcFB3X///Vq7dq2am5slXfxpvK+vT5LU29ur8vLySd9wIbnyu37hIguDLAyycC7jMrdt\nW48++qhqamq0YcOGscubmpoUj8clSfF4fGzJAwC8kbFmee2113THHXdoyZIlI7+CSBs3btStt96q\nNWvW6B//+IcqKyu1fft2lZSUXP6BqVkATCPTvWbJeAfol770JQ0PD1/1bW1tbZO+MQBAfvDn/C6g\nDzTIwiALgyycY5kDQADw3CwAoOnfmfOTOQAEAMvcBfSBBlkYZGGQhXMscwAIADpzABCdOQDAB1jm\nLqAPNMjCIAuDLJxjmQNAANCZA4DozAEAPsAydwF9oEEWBlkYZOEcyxwAAoDOHABEZw4A8AGWuQvo\nAw2yMMjCIAvnWOYAEAB05gAgOnMAgA+wzF1AH2iQhUEWBlk4xzIHgACgMwcA0ZkDAHyAZe4C+kCD\nLAyyMMjCOZY5AAQAnTkAiM4cAOADLHMX0AcaZGGQhUEWzmVc5o888ogsy9LixYvHLkulUopEIqqq\nqlJjY6PS6XTehwQAZJZxma9bt067du267LJYLKZIJKKuri41NDQoFovldcAgCIfDXo/gG2RhkIVB\nFs5lXOa33367SktLL7ustbVV0WhUkhSNRrVjx478TQcAyErxZN8hmUzKsixJkmVZSiaTGa69SdKr\nI6+XSKqXFB45J0b+zee5c2yS0U5u9CcAN8+X9oFe3L6fzqOX+WUeL8+dnZ3asGGDb+bx8rx582bV\n19d7Po8xeg67cE5I2jZyrlTO7Al0d3fbtbW1Y+eSkpLL3l5aWnrV95NkS222ZHv48rIdCq2Y6FPM\nu/b2dq9H8A2yMMjC8EMWF3eWl/tq9GXCtXxVk340i2VZ6uvrkyT19vaqvLw89+8kBWL0uz/I4lJk\nYZCFc5Ne5k1NTYrH45KkeDyu5ubmKR8KADA5GZf5gw8+qC984Qs6duyYKioqtHXrVrW0tGj37t2q\nqqrSnj171NLS4tas09aVfVzhIguDLAyycC7jHaAvvPDCVS9va2vLyzAAgNzw3CwAIJ6bBQDgAyxz\nF9AHGmRhkIVBFs6xzAEgAOjMAUB05gAAH2CZu4A+0CALgywMsnCOZQ4AAUBnDgCiMwcA+ADL3AX0\ngQZZGGRhkIVzLHMACAA6cwAQnTkAwAdY5i6gDzTIwiALgyycY5kDQADQmQOA6MwBAD7AMncBfaBB\nFgZZGGThHMscAAKAzhwARGcOAPABlrkL6AMNsjDIwiAL51jmABAAdOYAIDpzAIAPsMxdQB9okIVB\nFgZZOFfs9QD5tUoHD/aP/Prknf/5n5k6f/6cpzP4RWdnp8LhsKczXH99mc6ePePpDJI/vi7IIjhy\n/sl8165dqq6u1oIFC/TjH/94KmeaQv262IF5+zIw8GHeP9PpIp1Oez3CyPLi60IiiyDJaZkPDQ3p\nm9/8pnbt2qW33npLL7zwgo4ePTrVswEAspTTMt+/f79uueUWVVZWasaMGXrggQf04osvTvVsCKDj\nx497PQIQSDl15qdOnVJFRcXYee7cuXrzzTevcs07c51rCnnbl4/yurf3k3g87vUI4uviUn6YgSyc\nymmZZxN6nh6+DgC4ipxqljlz5qinp2fs3NPTo7lz507ZUACAyclpmS9dulTvvPOOjh8/roGBAf32\nt79VU1PTVM8GAMhSTjVLcXGxnnvuOd11110aGhrSo48+qoULF071bACALOX8OPN77rlHx44d03PP\nPad4PJ7x8ebf+ta3tGDBAtXV1engwYM5D+t3Ez32/le/+pXq6uq0ZMkSffGLX9Thw4c9mNId2f4d\nwl//+lcVFxfrD3/4g4vTuSubLBKJhEKhkGpraz3/o6p8miiL06dP6+6771Z9fb1qa2u1bds294d0\nwSOPPCLLsrR48eJxrzPpvWk7cOHCBXv+/Pl2d3e3PTAwYNfV1dlvvfXWZdf505/+ZN9zzz22bdv2\nG2+8YS9btszJTfpWNlns27fPTqfTtm3b9ksvvVTQWYxeb8WKFfa9995r/+53v/Ng0vzLJoszZ87Y\nNTU1dk9Pj23btv3ee+95MWreZZPFU089Zbe0tNi2fTGHsrIye3Bw0Itx8+qVV16xOzo67Nra2qu+\nPZe96ei5WbJ5vHlra6ui0agkadmyZUqn00omk05u1peyyeK2227TDTfcIOliFidPnvRi1LzL9u8Q\nnn32Wa1evVqzZs3yYEp3ZJPFr3/9a91///1jDyK46aabvBg177LJ4uabb1Z/f78kqb+/XzfeeKOK\ni4P3rCO33367SktLx317LnvT0TK/2uPNT506NeF1grjEssniUlu2bNHKlSvdGM112X5dvPjii3r8\n8ccl+eUxxlMvmyzeeecdpVIprVixQkuXLtXzzz/v9piuyCaL9evX68iRI/rUpz6luro6PfPMM26P\n6Qu57E1H3/Ky/Q/Q/q/HnAfxP9zJfE7t7e365S9/qb/85S95nMg72WSxYcMGxWIxFRVdfO7m//4a\nCYpsshgcHFRHR4defvllnTt3Trfddps+//nPa8GCBS5M6J5ssnj66adVX1+vRCKhd999V5FIRIcO\nHdInP/lJFyb0l8nuTUfLPJvHm//3dU6ePKk5c+Y4uVlfyvax94cPH9b69eu1a9eujL9mTWfZZHHg\nwAE98MADki7e6fXSSy9pxowZgXuIazZZVFRU6KabbtLMmTM1c+ZM3XHHHTp06FDglnk2Wezbt0/f\n/e53JUnz58/XvHnzdOzYMS1dutTVWb2W0950UuIPDg7an/70p+3u7m77/PnzE94B+vrrrwf2Tr9s\nsjhx4oQ9f/58+/XXX/doSndkk8WlHn74Yfv3v/+9ixO6J5ssjh49ajc0NNgXLlywP/jgA7u2ttY+\ncuSIRxPnTzZZPPnkk/b3v/9927Ztu6+vz54zZ47973//24tx8667uzurO0Cz3ZuOfjIf7/Hmv/jF\nLyRJjz32mFauXKmdO3fqlltu0XXXXaetW7c6uUnfyiaLH/7whzpz5sxYTzxjxgzt37/fy7HzIpss\nCkU2WVRXV+vuu+/WkiVLdM0112j9+vWqqanxePKpl00W3/nOd7Ru3TrV1dVpeHhYmzZtUllZmceT\nT70HH3xQe/fu1enTp1VRUaEf/OAHGhwclJT73szb/wMUAOAe/rdxABAALHMACACWOQAEAMscAAKA\nZQ4AAcAyB4AA+H9PIHvJ8uT8wwAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x10995a0d0>"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(data.dcounts())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 102,
       "text": [
        "98"
       ]
      }
     ],
     "prompt_number": 102
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data.dcounts().head(10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>table</th>\n",
        "      <th>column</th>\n",
        "      <th>distinct</th>\n",
        "      <th>size</th>\n",
        "      <th>fraction</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> customerp</td>\n",
        "      <td>        c_w_id</td>\n",
        "      <td>     1000</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-05</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> customerp</td>\n",
        "      <td>        c_d_id</td>\n",
        "      <td>       10</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-07</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> customerp</td>\n",
        "      <td>          c_id</td>\n",
        "      <td>     3000</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 1.000000e-04</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> customerp</td>\n",
        "      <td>    c_discount</td>\n",
        "      <td>     5000</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 1.666667e-04</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> customerp</td>\n",
        "      <td>      c_credit</td>\n",
        "      <td>        2</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 6.666667e-08</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> customerp</td>\n",
        "      <td>        c_last</td>\n",
        "      <td>     1000</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-05</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> customerp</td>\n",
        "      <td>       c_first</td>\n",
        "      <td> 29999284</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 9.999761e-01</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> customerp</td>\n",
        "      <td>  c_credit_lim</td>\n",
        "      <td>        1</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-08</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8</th>\n",
        "      <td> customerp</td>\n",
        "      <td>     c_balance</td>\n",
        "      <td>        1</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-08</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9</th>\n",
        "      <td> customerp</td>\n",
        "      <td> c_ytd_payment</td>\n",
        "      <td>        1</td>\n",
        "      <td> 30000000</td>\n",
        "      <td> 3.333333e-08</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "       table         column  distinct      size      fraction\n",
        "0  customerp         c_w_id      1000  30000000  3.333333e-05\n",
        "1  customerp         c_d_id        10  30000000  3.333333e-07\n",
        "2  customerp           c_id      3000  30000000  1.000000e-04\n",
        "3  customerp     c_discount      5000  30000000  1.666667e-04\n",
        "4  customerp       c_credit         2  30000000  6.666667e-08\n",
        "5  customerp         c_last      1000  30000000  3.333333e-05\n",
        "6  customerp        c_first  29999284  30000000  9.999761e-01\n",
        "7  customerp   c_credit_lim         1  30000000  3.333333e-08\n",
        "8  customerp      c_balance         1  30000000  3.333333e-08\n",
        "9  customerp  c_ytd_payment         1  30000000  3.333333e-08"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Herew, we see that `dcounts` refers to a Pandas data frame object with 96 rows."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Data Loading (Beta)\n",
      "\n",
      "While in most cases the data to profile is already available as a Hive or Impala or even MySQL table, there might exist the case that we simply have a directory full of CSV files following the same format and want to perform the same analysis. \n",
      "\n",
      "To facilitate this process, `pyxplorer` provides a simple interface to generate an external table based on a sample from the CSV files in a given directory.\n",
      "\n",
      "The idea is that a directory in HDFS already contains a set of files that all follow the same specification and the user now wants to perform some data profiling. We further assume that the user has access to Hive / Impala to run queries on the data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pyxplorer.loader import Loader\n",
      "loader = Loader(\"/user/martin/test1\", name_node=\"diufpc56\", hive_server=\"diufpc301\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It will check the first file it finds in the directory, extract the data of the first few hundred lines. It will then try to guess the correct separator and build an external table. The rest of the operations follows the normal explorative schedule described before."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "loader.load()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "('default', 'pyxplorer_data')"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pyxdb = m.Database(\"default\", conn)\n",
      "pyxdb"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<table><tr><th>Name</th><th>Size</th></tr><tr><td>pyxplorer_data</td><td>7</td></tr></table><p>Rows: 1 / Columns: 2</p>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 33,
       "text": [
        "<pyxplorer.manager.Database instance at 0x10cf79d88>"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab = pyxdb[\"pyxplorer_data\"]\n",
      "tab"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 37,
       "text": [
        "<Table: \"default.pyxplorer_data\">"
       ]
      }
     ],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.columns()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 38,
       "text": [
        "[col_0, col_1, col_2, col_3, col_4, col_5, col_6]"
       ]
      }
     ],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tab.col_0"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<table><tr><th>Name</th><th>Value</th></tr><tr><td>Min</td><td>100</td></tr><tr><td>Max</td><td>distance</td></tr><tr><td>#Distinct Values</td><td>7</td></tr><tr><td>Most Frequent</td><td>distance (1)</td></tr><tr><td>Least Frequent</td><td>distance (1)</td></tr><tr><td>Top 10 MF</td><td>100,129,148,distance,192,113,168</td></tr><tr><td>Top 10 LF</td><td>100, 129, 148, distance, 192, 113, 168</td></tr><tr><td>Uniqueness</td><td>1.0</td></tr><tr><td>Constancy</td><td>0.142857142857</td></tr></table><p>Rows: 9 / Columns: 2</p>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 39,
       "text": [
        "col_0"
       ]
      }
     ],
     "prompt_number": 39
    }
   ],
   "metadata": {}
  }
 ]
}