{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Usage of Daru::DataFrame\n",
    "\n",
    "Daru::DataFrame is a two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and vectors). \n",
    "\n",
    "Arithmetic operations align on both row and vector labels. Can be thought of as a container for Daru::Vector objects. This is primary data structure used by daru and gems that depend on it (like statsample).\n",
    "\n",
    "You should use DataFrame because it allows you to easily store, access and manipulate labelled data, plot it using an interactive graph library and perform various statistics operations by ignoring missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "if(window['d3'] === undefined ||\n",
       "   window['Nyaplot'] === undefined){\n",
       "    var path = {\"d3\":\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\",\"downloadable\":\"https://cdn.rawgit.com/domitry/d3-downloadable/master/d3-downloadable\"};\n",
       "\n",
       "\n",
       "\n",
       "    var shim = {\"d3\":{\"exports\":\"d3\"},\"downloadable\":{\"exports\":\"downloadable\"}};\n",
       "\n",
       "    require.config({paths: path, shim:shim});\n",
       "\n",
       "\n",
       "require(['d3'], function(d3){window['d3']=d3;console.log('finished loading d3');require(['downloadable'], function(downloadable){window['downloadable']=downloadable;console.log('finished loading downloadable');\n",
       "\n",
       "\tvar script = d3.select(\"head\")\n",
       "\t    .append(\"script\")\n",
       "\t    .attr(\"src\", \"https://cdn.rawgit.com/domitry/Nyaplotjs/master/release/nyaplot.js\")\n",
       "\t    .attr(\"async\", true);\n",
       "\n",
       "\tscript[0][0].onload = script[0][0].onreadystatechange = function(){\n",
       "\n",
       "\n",
       "\t    var event = document.createEvent(\"HTMLEvents\");\n",
       "\t    event.initEvent(\"load_nyaplot\",false,false);\n",
       "\t    window.dispatchEvent(event);\n",
       "\t    console.log('Finished loading Nyaplotjs');\n",
       "\n",
       "\t};\n",
       "\n",
       "\n",
       "});});\n",
       "}\n"
      ],
      "text/plain": [
       "\"if(window['d3'] === undefined ||\\n   window['Nyaplot'] === undefined){\\n    var path = {\\\"d3\\\":\\\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min\\\",\\\"downloadable\\\":\\\"https://cdn.rawgit.com/domitry/d3-downloadable/master/d3-downloadable\\\"};\\n\\n\\n\\n    var shim = {\\\"d3\\\":{\\\"exports\\\":\\\"d3\\\"},\\\"downloadable\\\":{\\\"exports\\\":\\\"downloadable\\\"}};\\n\\n    require.config({paths: path, shim:shim});\\n\\n\\nrequire(['d3'], function(d3){window['d3']=d3;console.log('finished loading d3');require(['downloadable'], function(downloadable){window['downloadable']=downloadable;console.log('finished loading downloadable');\\n\\n\\tvar script = d3.select(\\\"head\\\")\\n\\t    .append(\\\"script\\\")\\n\\t    .attr(\\\"src\\\", \\\"https://cdn.rawgit.com/domitry/Nyaplotjs/master/release/nyaplot.js\\\")\\n\\t    .attr(\\\"async\\\", true);\\n\\n\\tscript[0][0].onload = script[0][0].onreadystatechange = function(){\\n\\n\\n\\t    var event = document.createEvent(\\\"HTMLEvents\\\");\\n\\t    event.initEvent(\\\"load_nyaplot\\\",false,false);\\n\\t    window.dispatchEvent(event);\\n\\t    console.log('Finished loading Nyaplotjs');\\n\\n\\t};\\n\\n\\n});});\\n}\\n\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "require 'daru'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Creation and Access\n",
    "\n",
    "Daru offers many options for creating DataFrames. You can create it from Hashes, Arrays, Daru::Vectors or even load it from CSV files, Excel spreadsheets or SQL databases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From Array of Arrays**\n",
    "\n",
    "In the example below, I'm specifying the vertical _Vectors_ of the DataFrame as an Array of Arrays and I specify their names in the `:order` option, by supplying an Array of names that the vectors should be called by. \n",
    "\n",
    "In the `:index` option, we'll specify the names of the rows of the DataFrame. If the `:index` is not given, DataFrame will assign numerical indexes starting from 0 to each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:22605100  rows: 4  cols: 2<tr><th></th><th>a</th><th>b</th></tr><tr><td>one</td><td>1</td><td>1</td></tr><tr><td>two</td><td>2</td><td>2</td></tr><tr><td>three</td><td>3</td><td>3</td></tr><tr><td>four</td><td>4</td><td>4</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:22605100 @name = 7f3cee3e-c498-4b4f-93a1-5ff0dfef3999 @size = 4>\n",
       "                    a          b \n",
       "       one          1          1 \n",
       "       two          2          2 \n",
       "     three          3          3 \n",
       "      four          4          4 \n"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new([[1,2,3,4], [1,2,3,4]],order: [:a, :b], index: [:one, :two, :three, :four])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From Hash of Arrays**\n",
    "\n",
    "A similar DataFrame can be created from a Hash. In this case the keys of the Hash are the names of the vectors in the DataFrame. The `:order` option, if specified, will only serve to decide the orientation of the Vectors in the DataFrame. Not specfiying `:order` in this case will align the vectors alphabetically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:22188400  rows: 4  cols: 2<tr><th></th><th>b</th><th>a</th></tr><tr><td>0</td><td>1</td><td>1</td></tr><tr><td>1</td><td>2</td><td>2</td></tr><tr><td>2</td><td>3</td><td>3</td></tr><tr><td>3</td><td>4</td><td>4</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:22188400 @name = 13ffb01b-8b25-41e4-a565-fe9596dec62d @size = 4>\n",
       "                    b          a \n",
       "         0          1          1 \n",
       "         1          2          2 \n",
       "         2          3          3 \n",
       "         3          4          4 \n"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({a: [1,2,3,4], b: [1,2,3,4]},order: [:b, :a])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From Hash of Vectors**\n",
    "\n",
    "A DataFrame can be created from a Hash of Daru::Vectors and their names. The name of the vector will be the key and the corresponding value, a Daru::Vector.\n",
    "\n",
    "The values of the DataFrame are aligned according to the index of each Daru::Vector. A *nil* is assigned whenever a particular index is not available for one Vector but is present in any of the other Vectors, and the resulting index of the DataFrame is a union of the indexes of all the Vectors in alphabetical order.\n",
    "\n",
    "The sizes or indexes of the supplied Vectors don't matter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:21716520  rows: 6  cols: 2<tr><th></th><th>v1</th><th>v2</th></tr><tr><td>a</td><td>1</td><td>33</td></tr><tr><td>absent</td><td></td><td>44</td></tr><tr><td>b</td><td>2</td><td>11</td></tr><tr><td>c</td><td>3</td><td></td></tr><tr><td>d</td><td>4</td><td></td></tr><tr><td>e</td><td>5</td><td>22</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:21716520 @name = 56d8b0a4-5efa-4e34-bc9c-2701dae0449a @size = 6>\n",
       "                   v1         v2 \n",
       "         a          1         33 \n",
       "    absent        nil         44 \n",
       "         b          2         11 \n",
       "         c          3        nil \n",
       "         d          4        nil \n",
       "         e          5         22 \n"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v1 = Daru::Vector.new([1,2,3,4,5], index: [:a, :b, :c, :d, :e])\n",
    "v2 = Daru::Vector.new([11,22,33,44], index: [:b, :e, :a, :absent])\n",
    "\n",
    "Daru::DataFrame.new({v1: v1, v2: v2})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The 'clone' option**\n",
    "\n",
    "If you have Vectors that have _exactly_ the same index, you can specify the `:clone` option to DataFrame. Setting `:clone` to *false* will direct daru to utilize the same Vector objects in creating the DataFrame, that you have specified in the Hash and will prevent their cloning when being stored in the DataFrame. Thus the object IDs of the Vectors will remain the same.\n",
    "\n",
    "Be wary of making changes in the DataFrame or the supplied vectors if you set `:clone` to *false*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "equalness a : true\n",
      "equalness b : true\n"
     ]
    }
   ],
   "source": [
    "v1 = Daru::Vector.new([1,2,3,4,5])\n",
    "v2 = Daru::Vector.new([11,22,33,44,55])\n",
    "\n",
    "df = Daru::DataFrame.new({a: v1, b: v2}, clone: false)\n",
    "puts \"equalness a : #{v1.object_id == df[:a].object_id}\\nequalness b : #{v2.object_id == df[:b].object_id}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating with rows**\n",
    "\n",
    "If you want to create a DataFrame by specifying the rows, you can do so by specifying an _Array of Arrays_ or _Array of Vectors_ to the `.rows` method.\n",
    "\n",
    "Lets first see creating DataFrames from an Array of Arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:20876660  rows: 4  cols: 4<tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th></tr><tr><td>0</td><td>1</td><td>11</td><td>10</td><td>a</td></tr><tr><td>1</td><td>2</td><td>22</td><td>20</td><td>4</td></tr><tr><td>2</td><td>3</td><td>33</td><td>30</td><td>g</td></tr><tr><td>3</td><td>4</td><td>44</td><td>40</td><td>3</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:20876660 @name = 56f52f77-e2dc-40b4-92bc-13770f4ad989 @size = 4>\n",
       "                    a          b          c          d \n",
       "         0          1         11         10          a \n",
       "         1          2         22         20          4 \n",
       "         2          3         33         30          g \n",
       "         3          4         44         40          3 \n"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Daru::DataFrame.rows([\n",
    "  [1,11,10,'a'],\n",
    "  [2,22,20 ,4 ],\n",
    "  [3,33,30,'g'],\n",
    "  [4,44,40, 3 ]\n",
    "  ], order: [:a, :b, :c, :d])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you supply an Array of Vectors to the `.rows` method, the index of the Vectors will be automatically assigned as the names of the vectors of the DataFrame. Moreover, elements will be aligned by their indexes in the completed DataFrame.\n",
    "\n",
    "If a Vector does not have a particular index that is present in other Vectors, a **nil** will be placed in that position.\n",
    "\n",
    "The `:order` option should be set in this case to whatever values you want to keep in your DataFrame to avoid unexpected behaviour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"6\">Daru::DataFrame:20467260  rows: 2  cols: 5<tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th><th>odd</th></tr><tr><td>0</td><td>1</td><td>2</td><td>3</td><td>4</td><td></td></tr><tr><td>1</td><td>11</td><td>44</td><td>22</td><td></td><td>55</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:20467260 @name = d0b7c255-1d2c-45ff-96eb-42842e4032ca @size = 2>\n",
       "                    a          b          c          d        odd \n",
       "         0          1          2          3          4        nil \n",
       "         1         11         44         22        nil         55 \n"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1 = Daru::Vector.new([1,2,3,4,5], index: [:a, :b, :c, :d, :e])\n",
    "r2 = Daru::Vector.new([11,22,33,44,55], index: [:a, :c, :e, :b, :odd])\n",
    "\n",
    "Daru::DataFrame.rows([r1,r2], order: [:a, :b, :c, :d, :odd])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data from different data sources\n",
    "\n",
    "Daru::DataFrame currently supports loading data from CSV files, Excel spreadsheets and SQL databases. You can also write your DataFrames to these kinds of files using some simple functions. Daru also supports saving and loading data by Marshalling. Lets go through them one by one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**CSV (Comma Separated Values) files**\n",
    "\n",
    "To demonstrate loading and writing to CSV files, we'll read some sales data from [this CSV file](https://github.com/v0dro/daru/blob/master/spec/fixtures/sales-funnel.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"9\">Daru::DataFrame:18079560  rows: 17  cols: 8<tr><th></th><th>Account</th><th>Manager</th><th>Name</th><th>Price</th><th>Product</th><th>Quantity</th><th>Rep</th><th>Status</th></tr><tr><td>0</td><td>714466</td><td>Debra Henley</td><td>Trantow-Barrows</td><td>30000</td><td>CPU</td><td>1</td><td>Craig Booker</td><td>presented</td></tr><tr><td>1</td><td>714466</td><td>Debra Henley</td><td>Trantow-Barrows</td><td>10000</td><td>Software</td><td>1</td><td>Craig Booker</td><td>presented</td></tr><tr><td>2</td><td>714466</td><td>Debra Henley</td><td>Trantow-Barrows</td><td>5000</td><td>Maintenance</td><td>2</td><td>Craig Booker</td><td>pending</td></tr><tr><td>3</td><td>737550</td><td>Debra Henley</td><td>Fritsch, Russel and Anderson</td><td>35000</td><td>CPU</td><td>1</td><td>Craig Booker</td><td>declined</td></tr><tr><td>4</td><td>146832</td><td>Debra Henley</td><td>Kiehn-Spinka</td><td>65000</td><td>CPU</td><td>2</td><td>Daniel Hilton</td><td>won</td></tr><tr><td>5</td><td>218895</td><td>Debra Henley</td><td>Kulas Inc</td><td>40000</td><td>CPU</td><td>2</td><td>Daniel Hilton</td><td>pending</td></tr><tr><td>6</td><td>218895</td><td>Debra Henley</td><td>Kulas Inc</td><td>10000</td><td>Software</td><td>1</td><td>Daniel Hilton</td><td>presented</td></tr><tr><td>7</td><td>412290</td><td>Debra Henley</td><td>Jerde-Hilpert</td><td>5000</td><td>Maintenance</td><td>2</td><td>John Smith</td><td>pending</td></tr><tr><td>8</td><td>740150</td><td>Debra Henley</td><td>Barton LLC</td><td>35000</td><td>CPU</td><td>1</td><td>John Smith</td><td>declined</td></tr><tr><td>9</td><td>141962</td><td>Fred Anderson</td><td>Herman LLC</td><td>65000</td><td>CPU</td><td>2</td><td>Cedric Moss</td><td>won</td></tr><tr><td>10</td><td>163416</td><td>Fred Anderson</td><td>Purdy-Kunde</td><td>30000</td><td>CPU</td><td>1</td><td>Cedric Moss</td><td>presented</td></tr><tr><td>11</td><td>239344</td><td>Fred Anderson</td><td>Stokes LLC</td><td>5000</td><td>Maintenance</td><td>1</td><td>Cedric Moss</td><td>pending</td></tr><tr><td>12</td><td>239344</td><td>Fred Anderson</td><td>Stokes LLC</td><td>10000</td><td>Software</td><td>1</td><td>Cedric Moss</td><td>presented</td></tr><tr><td>13</td><td>307599</td><td>Fred Anderson</td><td>Kassulke, Ondricka and Metz</td><td>7000</td><td>Maintenance</td><td>3</td><td>Wendy Yule</td><td>won</td></tr><tr><td>14</td><td>688981</td><td>Fred Anderson</td><td>Keeling LLC</td><td>100000</td><td>CPU</td><td>5</td><td>Wendy Yule</td><td>won</td></tr><tr><td>15</td><td>729833</td><td>Fred Anderson</td><td>Koepp Ltd</td><td>65000</td><td>CPU</td><td>2</td><td>Wendy Yule</td><td>declined</td></tr><tr><td>16</td><td>729833</td><td>Fred Anderson</td><td>Koepp Ltd</td><td>5000</td><td>Monitor</td><td>2</td><td>Wendy Yule</td><td>presented</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:18079560 @name = 80362321-c889-498d-a450-2d05a1dc968d @size = 17>\n",
       "              Account    Manager       Name      Price    Product   Quantity        Rep     Status \n",
       "         0     714466 Debra Henl Trantow-Ba      30000        CPU          1 Craig Book  presented \n",
       "         1     714466 Debra Henl Trantow-Ba      10000   Software          1 Craig Book  presented \n",
       "         2     714466 Debra Henl Trantow-Ba       5000 Maintenanc          2 Craig Book    pending \n",
       "         3     737550 Debra Henl Fritsch, R      35000        CPU          1 Craig Book   declined \n",
       "         4     146832 Debra Henl Kiehn-Spin      65000        CPU          2 Daniel Hil        won \n",
       "         5     218895 Debra Henl  Kulas Inc      40000        CPU          2 Daniel Hil    pending \n",
       "         6     218895 Debra Henl  Kulas Inc      10000   Software          1 Daniel Hil  presented \n",
       "         7     412290 Debra Henl Jerde-Hilp       5000 Maintenanc          2 John Smith    pending \n",
       "         8     740150 Debra Henl Barton LLC      35000        CPU          1 John Smith   declined \n",
       "         9     141962 Fred Ander Herman LLC      65000        CPU          2 Cedric Mos        won \n",
       "        10     163416 Fred Ander Purdy-Kund      30000        CPU          1 Cedric Mos  presented \n",
       "        11     239344 Fred Ander Stokes LLC       5000 Maintenanc          1 Cedric Mos    pending \n",
       "        12     239344 Fred Ander Stokes LLC      10000   Software          1 Cedric Mos  presented \n",
       "        13     307599 Fred Ander Kassulke,        7000 Maintenanc          3 Wendy Yule        won \n",
       "        14     688981 Fred Ander Keeling LL     100000        CPU          5 Wendy Yule        won \n",
       "       ...        ...        ...        ...        ...        ...        ...        ...        ... \n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Daru::DataFrame.from_csv 'data/sales-funnel.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify all the options to the `.from_csv` function that you do to the Ruby `CSV.read()` function, since this is what is used internally.\n",
    "\n",
    "For example, if the columns in your CSV file are separated by something other that commas, you can use the `:col_sep` option. If you want to convert numeric values to numbers and not keep them as strings, you can use the `:converters` option and set it to `:numeric`.\n",
    "\n",
    "The `.from_csv` function uses the following defaults for reading CSV files (that are passed into the `CSV.read()` function):\n",
    "``` ruby\n",
    "\n",
    "{\n",
    "  :col_sep           => ',',\n",
    "  :converters        => :numeric\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `#write_csv` function is used for writing the contents of a DataFrame to a CSV file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Excel Files**\n",
    "\n",
    "The `::from_excel` method can be used for loading Excel files. The [spreadsheet](https://github.com/zdavatz/spreadsheet) gem is used in the background in this case, so whatever variants of Excel compatible files can be loaded by spreadsheet should be easily loadable in this case too.\n",
    "\n",
    "Let me demonstrate this using [this Excel file](https://github.com/v0dro/daru/blob/master/spec/fixtures/test_xls.xls)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"6\">Daru::DataFrame:16647660  rows: 6  cols: 5<tr><th></th><th>id</th><th>name</th><th>age</th><th>city</th><th>a1</th></tr><tr><td>0</td><td>1</td><td>Alex</td><td>20</td><td>New York</td><td>a,b</td></tr><tr><td>1</td><td>2</td><td>Claude</td><td>23</td><td>London</td><td>b,c</td></tr><tr><td>2</td><td>3</td><td>Peter</td><td>25</td><td>London</td><td>a</td></tr><tr><td>3</td><td>4</td><td>Franz</td><td></td><td>Paris</td><td></td></tr><tr><td>4</td><td>5</td><td>George</td><td>5.5</td><td>Tome</td><td>a,b,c</td></tr><tr><td>5</td><td>6</td><td>Fernand</td><td></td><td></td><td></td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:16647660 @name = dd26cbc7-93a2-4e25-a85c-859c3a13d58e @size = 6>\n",
       "                   id       name        age       city         a1 \n",
       "         0          1       Alex         20   New York        a,b \n",
       "         1          2     Claude         23     London        b,c \n",
       "         2          3      Peter         25     London          a \n",
       "         3          4      Franz        nil      Paris        nil \n",
       "         4          5     George        5.5       Tome      a,b,c \n",
       "         5          6    Fernand        nil        nil        nil \n"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.from_excel 'data/test_xls.xls'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, the `#write_excel` method can be used for writing data stored in the DataFrame to an Excel file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**SQL Databases**\n",
    "\n",
    "Similar to the examples above you can use the `::from_sql` and `#write_sql` methods for interacting with SQL databases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Plaintext Files**\n",
    "\n",
    "In case your data is stored as columns in plaintext (for example [this](https://github.com/v0dro/daru/blob/master/spec/fixtures/bank2.dat) file), you can use the `::from_plaintext` method for loading data from the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying and accessing data\n",
    "\n",
    "Daru::DataFrame consists of rows and vectors, both of which can be accessed by their labels using an intuitive syntax.\n",
    "\n",
    "Consider the following DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:14984040  rows: 7  cols: 3<tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><td>a</td><td>1</td><td>a</td><td>11</td></tr><tr><td>b</td><td>2</td><td>b</td><td>22</td></tr><tr><td>c</td><td>3</td><td>c</td><td>33</td></tr><tr><td>d</td><td>4</td><td>d</td><td>44</td></tr><tr><td>e</td><td>5</td><td>e</td><td>55</td></tr><tr><td>f</td><td>6</td><td>f</td><td>66</td></tr><tr><td>g</td><td>7</td><td>g</td><td>77</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14984040 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 7>\n",
       "                    a          b          c \n",
       "         a          1          a         11 \n",
       "         b          2          b         22 \n",
       "         c          3          c         33 \n",
       "         d          4          d         44 \n",
       "         e          5          e         55 \n",
       "         f          6          f         66 \n",
       "         g          7          g         77 \n"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({\n",
    "  a: [1,2,3,4,5,6,7], \n",
    "  b: ['a','b','c','d','e','f','g'], \n",
    "  c: [11,22,33,44,55,66,77]\n",
    "  }, index: [:a,:b,:c,:d,:e,:f,:g])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access any Vector using the `#[]` operator. The resultant Vector is returned as a Daru::Vector which preserves the index of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:14980940  size: 7</th></tr><tr><th> </th><th>b</th></tr><tr><td>a</td><td>a</td></tr><tr><td>b</td><td>b</td></tr><tr><td>c</td><td>c</td></tr><tr><td>d</td><td>d</td></tr><tr><td>e</td><td>e</td></tr><tr><td>f</td><td>f</td></tr><tr><td>g</td><td>g</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:14980940 @name = b @size = 7 >\n",
       "      b\n",
       "  a   a\n",
       "  b   b\n",
       "  c   c\n",
       "  d   d\n",
       "  e   e\n",
       "  f   f\n",
       "  g   g\n"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify a Range inside `#[]` to return a DataFrame which contains the columns within the Range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:14029820  rows: 7  cols: 2<tr><th></th><th>b</th><th>c</th></tr><tr><td>a</td><td>a</td><td>11</td></tr><tr><td>b</td><td>b</td><td>22</td></tr><tr><td>c</td><td>c</td><td>33</td></tr><tr><td>d</td><td>d</td><td>44</td></tr><tr><td>e</td><td>e</td><td>55</td></tr><tr><td>f</td><td>f</td><td>66</td></tr><tr><td>g</td><td>g</td><td>77</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14029820 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 7>\n",
       "                    b          c \n",
       "         a          a         11 \n",
       "         b          b         22 \n",
       "         c          c         33 \n",
       "         d          d         44 \n",
       "         e          e         55 \n",
       "         f          f         66 \n",
       "         g          g         77 \n"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:b..:c]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A row can be accessed using the `#row[]` method. The row is also returned as a Daru::Vector and any operations so any operations on a Daru::Vector will be valid on the row too.\n",
    "\n",
    "The index of the returned row corresponds to the names of the Vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:13588820  size: 3</th></tr><tr><th> </th><th>c</th></tr><tr><td>a</td><td>3</td></tr><tr><td>b</td><td>c</td></tr><tr><td>c</td><td>33</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:13588820 @name = c @size = 3 >\n",
       "      c\n",
       "  a   3\n",
       "  b   c\n",
       "  c  33\n"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.row[:c]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here too, you can specify a Range, and you will receive a Daru::DataFrame instead of a Daru::Vector containing the relevant rows specified by the Range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:24490780  rows: 3  cols: 3<tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><td>d</td><td>4</td><td>d</td><td>44</td></tr><tr><td>e</td><td>5</td><td>e</td><td>55</td></tr><tr><td>f</td><td>6</td><td>f</td><td>66</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:24490780 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 3>\n",
       "                    a          b          c \n",
       "         d          4          d         44 \n",
       "         e          5          e         55 \n",
       "         f          6          f         66 \n"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.row[:d..:f]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rows can be accessed using numerical indices too (this works for columns too)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:24061940  size: 3</th></tr><tr><th> </th><th>3</th></tr><tr><td>a</td><td>4</td></tr><tr><td>b</td><td>d</td></tr><tr><td>c</td><td>44</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:24061940 @name = 3 @size = 3 >\n",
       "      3\n",
       "  a   4\n",
       "  b   d\n",
       "  c  44\n"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.row[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get the top 3 rows by passing an argument to the `#head` method (or the bottom 3 using `#tail`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:23701640  rows: 3  cols: 3<tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><td>a</td><td>1</td><td>a</td><td>11</td></tr><tr><td>b</td><td>2</td><td>b</td><td>22</td></tr><tr><td>c</td><td>3</td><td>c</td><td>33</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:23701640 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 3>\n",
       "                    a          b          c \n",
       "         a          1          a         11 \n",
       "         b          2          b         22 \n",
       "         c          3          c         33 \n"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering, selecting, adding and deleting data\n",
    "\n",
    "A column can be added by simply specifying it's name and value using the `#[]=` operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:14984040  rows: 7  cols: 4<tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th></tr><tr><td>a</td><td>1</td><td>a</td><td>11</td><td>11</td></tr><tr><td>b</td><td>2</td><td>b</td><td>22</td><td>44</td></tr><tr><td>c</td><td>3</td><td>c</td><td>33</td><td>99</td></tr><tr><td>d</td><td>4</td><td>d</td><td>44</td><td>176</td></tr><tr><td>e</td><td>5</td><td>e</td><td>55</td><td>275</td></tr><tr><td>f</td><td>6</td><td>f</td><td>66</td><td>396</td></tr><tr><td>g</td><td>7</td><td>g</td><td>77</td><td>539</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14984040 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 7>\n",
       "                    a          b          c          d \n",
       "         a          1          a         11         11 \n",
       "         b          2          b         22         44 \n",
       "         c          3          c         33         99 \n",
       "         d          4          d         44        176 \n",
       "         e          5          e         55        275 \n",
       "         f          6          f         66        396 \n",
       "         g          7          g         77        539 \n"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:d] = df[:a] * df[:c]\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can delete a vector with the `#delete_vector` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:14984040  rows: 7  cols: 3<tr><th></th><th>a</th><th>c</th><th>d</th></tr><tr><td>a</td><td>1</td><td>11</td><td>11</td></tr><tr><td>b</td><td>2</td><td>22</td><td>44</td></tr><tr><td>c</td><td>3</td><td>33</td><td>99</td></tr><tr><td>d</td><td>4</td><td>44</td><td>176</td></tr><tr><td>e</td><td>5</td><td>55</td><td>275</td></tr><tr><td>f</td><td>6</td><td>66</td><td>396</td></tr><tr><td>g</td><td>7</td><td>77</td><td>539</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14984040 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 7>\n",
       "                    a          c          d \n",
       "         a          1         11         11 \n",
       "         b          2         22         44 \n",
       "         c          3         33         99 \n",
       "         d          4         44        176 \n",
       "         e          5         55        275 \n",
       "         f          6         66        396 \n",
       "         g          7         77        539 \n"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.delete_vector :b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you try to insert a Daru::Vector that does not conform to the index of the DataFrame, the values will be appropriately placed such that they conform to the DataFrame's index.\n",
    "\n",
    "*nil* is inserted wherever a similar index cannot be found on the DataFrame.\n",
    "\n",
    "Inserting an Array will require the Array to be of the same length as that of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:14984040  rows: 7  cols: 4<tr><th></th><th>a</th><th>c</th><th>d</th><th>b</th></tr><tr><td>a</td><td>1</td><td>11</td><td>11</td><td>a</td></tr><tr><td>b</td><td>2</td><td>22</td><td>44</td><td>c</td></tr><tr><td>c</td><td>3</td><td>33</td><td>99</td><td>33</td></tr><tr><td>d</td><td>4</td><td>44</td><td>176</td><td>b</td></tr><tr><td>e</td><td>5</td><td>55</td><td>275</td><td>d</td></tr><tr><td>f</td><td>6</td><td>66</td><td>396</td><td>88</td></tr><tr><td>g</td><td>7</td><td>77</td><td>539</td><td></td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14984040 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 7>\n",
       "                    a          c          d          b \n",
       "         a          1         11         11          a \n",
       "         b          2         22         44          c \n",
       "         c          3         33         99         33 \n",
       "         d          4         44        176          b \n",
       "         e          5         55        275          d \n",
       "         f          6         66        396         88 \n",
       "         g          7         77        539        nil \n"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:b] = Daru::Vector.new(['a',33,'b','c','d',88,'e'], index: [:a,:c,:d,:b,:e,:f,:extra])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inserting a row also works similarly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:14984040  rows: 8  cols: 4<tr><th></th><th>a</th><th>c</th><th>d</th><th>b</th></tr><tr><td>a</td><td>1</td><td>11</td><td>11</td><td>a</td></tr><tr><td>b</td><td>2</td><td>22</td><td>44</td><td>c</td></tr><tr><td>c</td><td>3</td><td>33</td><td>99</td><td>33</td></tr><tr><td>d</td><td>4</td><td>44</td><td>176</td><td>b</td></tr><tr><td>e</td><td>5</td><td>55</td><td>275</td><td>d</td></tr><tr><td>f</td><td>6</td><td>66</td><td>396</td><td>88</td></tr><tr><td>g</td><td>7</td><td>77</td><td>539</td><td></td></tr><tr><td>latest</td><td>30</td><td>10</td><td>40</td><td>20</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14984040 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 8>\n",
       "                    a          c          d          b \n",
       "         a          1         11         11          a \n",
       "         b          2         22         44          c \n",
       "         c          3         33         99         33 \n",
       "         d          4         44        176          b \n",
       "         e          5         55        275          d \n",
       "         f          6         66        396         88 \n",
       "         g          7         77        539        nil \n",
       "    latest         30         10         40         20 \n"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.row[:latest] = Daru::Vector.new([10,20,30,40], index: [:c,:b,:a,:d])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In both row and vector insertion, if the index specified is not present in the DataFrame, a new index is created and appended or if it is present then the existing index will be over-ridden."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For filtering out certain rows/vectors based on their values, use the `#filter` method. By default it iterates over vectors and keeps those vectors for which the block returns **true**. It accepts an optional _axis_ argument which lets you specify whether you want to iterate over vectors or rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:20876140  rows: 8  cols: 2<tr><th></th><th>a</th><th>c</th></tr><tr><td>a</td><td>1</td><td>11</td></tr><tr><td>b</td><td>2</td><td>22</td></tr><tr><td>c</td><td>3</td><td>33</td></tr><tr><td>d</td><td>4</td><td>44</td></tr><tr><td>e</td><td>5</td><td>55</td></tr><tr><td>f</td><td>6</td><td>66</td></tr><tr><td>g</td><td>7</td><td>77</td></tr><tr><td>latest</td><td>30</td><td>10</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:20876140 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 8>\n",
       "                    a          c \n",
       "         a          1         11 \n",
       "         b          2         22 \n",
       "         c          3         33 \n",
       "         d          4         44 \n",
       "         e          5         55 \n",
       "         f          6         66 \n",
       "         g          7         77 \n",
       "    latest         30         10 \n"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter vectors.\n",
    "\n",
    "# The `type` method returns either :numeric or :object. The :numeric type states\n",
    "# that the Vector consists only of numerical data (combined with missing data).\n",
    "# If the type happens to be :object, it contains non-numerical data like strings\n",
    "# or symbols. Statistical operations will not be possible on Vectors of type :object.\n",
    "\n",
    "df.filter do |vector|\n",
    "  vector.type == :numeric and vector.median < 50\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:20409180  rows: 3  cols: 4<tr><th></th><th>a</th><th>c</th><th>d</th><th>b</th></tr><tr><td>a</td><td>1</td><td>11</td><td>11</td><td>a</td></tr><tr><td>b</td><td>2</td><td>22</td><td>44</td><td>c</td></tr><tr><td>latest</td><td>30</td><td>10</td><td>40</td><td>20</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:20409180 @name = 9d700c98-537c-455e-87c7-8c7ff6a46472 @size = 3>\n",
       "                    a          c          d          b \n",
       "         a          1         11         11          a \n",
       "         b          2         22         44          c \n",
       "    latest         30         10         40         20 \n"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Filter rows\n",
    "\n",
    "df.filter(:row) do |row|\n",
    "  row[:a] + row[:d] < 100\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A DataFrame can be transposed using the `#transpose` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"9\">Daru::DataFrame:18063520  rows: 4  cols: 8<tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th><th>e</th><th>f</th><th>g</th><th>latest</th></tr><tr><td>a</td><td>1</td><td>2</td><td>3</td><td>4</td><td>5</td><td>6</td><td>7</td><td>30</td></tr><tr><td>c</td><td>11</td><td>22</td><td>33</td><td>44</td><td>55</td><td>66</td><td>77</td><td>10</td></tr><tr><td>d</td><td>11</td><td>44</td><td>99</td><td>176</td><td>275</td><td>396</td><td>539</td><td>40</td></tr><tr><td>b</td><td>a</td><td>c</td><td>33</td><td>b</td><td>d</td><td>88</td><td></td><td>20</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:18063520 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 4>\n",
       "                    a          b          c          d          e          f          g     latest \n",
       "         a          1          2          3          4          5          6          7         30 \n",
       "         c         11         22         33         44         55         66         77         10 \n",
       "         d         11         44         99        176        275        396        539         40 \n",
       "         b          a          c         33          b          d         88        nil         20 \n"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.transpose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arithmetic\n",
    "\n",
    "All arithmetic operations can be performed on a Daru::DataFrame and you can a DataFrame with another DataFrame, a Vector or a scalar.\n",
    "\n",
    "Indexes are aligned appropriately whenever an operation is performed with a non-scalar quantity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**With a Scalar**\n",
    "\n",
    "Adding a scalar quantity will add that number to all the numeric type vectors, keeping :object type Vectors the way they originally were."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:17731620  rows: 8  cols: 4<tr><th></th><th>a</th><th>c</th><th>d</th><th>b</th></tr><tr><td>a</td><td>11</td><td>21</td><td>21</td><td>a</td></tr><tr><td>b</td><td>12</td><td>32</td><td>54</td><td>c</td></tr><tr><td>c</td><td>13</td><td>43</td><td>109</td><td>33</td></tr><tr><td>d</td><td>14</td><td>54</td><td>186</td><td>b</td></tr><tr><td>e</td><td>15</td><td>65</td><td>285</td><td>d</td></tr><tr><td>f</td><td>16</td><td>76</td><td>406</td><td>88</td></tr><tr><td>g</td><td>17</td><td>87</td><td>549</td><td></td></tr><tr><td>latest</td><td>40</td><td>20</td><td>50</td><td>20</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:17731620 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 8>\n",
       "                    a          c          d          b \n",
       "         a         11         21         21          a \n",
       "         b         12         32         54          c \n",
       "         c         13         43        109         33 \n",
       "         d         14         54        186          b \n",
       "         e         15         65        285          d \n",
       "         f         16         76        406         88 \n",
       "         g         17         87        549        nil \n",
       "    latest         40         20         50         20 \n"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df + 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**With another DataFrame**\n",
    "\n",
    "Performing arithmetic between two data frames will align the elements by row and column indexes of either dataframe. \n",
    "\n",
    "If a column is present in one dataframe but not in the other, the resultant dataframe will be populated with a column full of *nils* of that name.\n",
    "\n",
    "DataFrames need not be of the same size for this operation to succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"6\">Daru::DataFrame:16665280  rows: 9  cols: 5<tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th><th>f</th></tr><tr><td>a</td><td>69</td><td></td><td>32</td><td></td><td></td></tr><tr><td>b</td><td>72</td><td></td><td>56</td><td></td><td></td></tr><tr><td>c</td><td>38</td><td></td><td>108</td><td></td><td></td></tr><tr><td>d</td><td>26</td><td></td><td>47</td><td></td><td></td></tr><tr><td>e</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>f</td><td>84</td><td></td><td>101</td><td></td><td></td></tr><tr><td>g</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>latest</td><td>73</td><td></td><td>31</td><td></td><td></td></tr><tr><td>older</td><td></td><td></td><td></td><td></td><td></td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:16665280 @name = 6b06722c-f7ce-4485-8e9b-a169488f63ae @size = 9>\n",
       "                    a          b          c          d          f \n",
       "         a         69        nil         32        nil        nil \n",
       "         b         72        nil         56        nil        nil \n",
       "         c         38        nil        108        nil        nil \n",
       "         d         26        nil         47        nil        nil \n",
       "         e        nil        nil        nil        nil        nil \n",
       "         f         84        nil        101        nil        nil \n",
       "         g        nil        nil        nil        nil        nil \n",
       "    latest         73        nil         31        nil        nil \n",
       "     older        nil        nil        nil        nil        nil \n"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = Daru::DataFrame.new({\n",
    "  a: 7.times.map { rand(100) },\n",
    "  f: 7.times.map { rand(100) },\n",
    "  c: 7.times.map { rand(100) }\n",
    "  }, index: [:a,:b,:c,:d,:latest,:older,:f])\n",
    "\n",
    "df1 + df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics\n",
    "\n",
    "Statistical methods perform basic statistics on numerical Vectors only.\n",
    "\n",
    "For a whole list of methods see the Daru::Maths::Statistics::DataFrame module in the [docs](https://rubygems.org/gems/daru).\n",
    "\n",
    "To demonstrate, the `#mean` method calculates the mean of each numeric vector and returns a Daru::Vector with the vector's name as the index alongwith the corresponding value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:14533320  size: 3</th></tr><tr><th> </th><th>mean</th></tr><tr><td>a</td><td>7.25</td></tr><tr><td>c</td><td>39.75</td></tr><tr><td>d</td><td>197.5</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:14533320 @name = mean @size = 3 >\n",
       "       mean\n",
       "    a  7.25\n",
       "    c 39.75\n",
       "    d 197.5\n"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `#describe` method can be used for knowing various statistics in one shot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:14352440  rows: 5  cols: 3<tr><th></th><th>a</th><th>c</th><th>d</th></tr><tr><td>count</td><td>8</td><td>8</td><td>8</td></tr><tr><td>mean</td><td>7.25</td><td>39.75</td><td>197.5</td></tr><tr><td>std</td><td>9.40744386111339</td><td>25.06990227344335</td><td>190.99214643539665</td></tr><tr><td>min</td><td>1</td><td>10</td><td>11</td></tr><tr><td>max</td><td>30</td><td>77</td><td>539</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14352440 @name = faf98540-fbb0-481e-b44b-b321a253c2f7 @size = 5>\n",
       "                    a          c          d \n",
       "     count          8          8          8 \n",
       "      mean       7.25      39.75      197.5 \n",
       "       std 9.40744386 25.0699022 190.992146 \n",
       "       min          1         10         11 \n",
       "       max         30         77        539 \n"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`#cov` will return a covariance matrix of the DataFrame, and it will be properly indexed so you can see the data clearly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:13991820  rows: 3  cols: 3<tr><th></th><th>a</th><th>c</th><th>d</th></tr><tr><td>a</td><td>88.5</td><td>-66.5</td><td>-233.0</td></tr><tr><td>c</td><td>-66.5</td><td>628.5</td><td>4637.0</td></tr><tr><td>d</td><td>-233.0</td><td>4637.0</td><td>36478.0</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:13991820 @name = 11a2179b-136b-45df-9e71-765e9fdce958 @size = 3>\n",
       "                    a          c          d \n",
       "         a       88.5      -66.5     -233.0 \n",
       "         c      -66.5      628.5     4637.0 \n",
       "         d     -233.0     4637.0    36478.0 \n"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.cov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise `#corr` computes the correlation matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:12502180  rows: 3  cols: 3<tr><th></th><th>a</th><th>c</th><th>d</th></tr><tr><td>a</td><td>1.0</td><td>-0.28196640612394586</td><td>-0.12967873822641748</td></tr><tr><td>c</td><td>-0.28196640612394586</td><td>0.9999999999999998</td><td>0.9684315851062977</td></tr><tr><td>d</td><td>-0.12967873822641748</td><td>0.9684315851062977</td><td>1.0</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:12502180 @name = 2c35c2ae-0580-42d9-88ac-26348b6a22d3 @size = 3>\n",
       "                    a          c          d \n",
       "         a        1.0 -0.2819664 -0.1296787 \n",
       "         c -0.2819664 0.99999999 0.96843158 \n",
       "         d -0.1296787 0.96843158        1.0 \n"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.corr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use [report builder](https://rubygems.org/gems/reportbuilder) to create a quick summary of the DataFrame using the `#summary` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "= 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77\n",
      "  Number of rows: 8\n",
      "  Element:[a]\n",
      "  == a\n",
      "    n :8\n",
      "    n valid:8\n",
      "    median: 4.5\n",
      "    mean: 7.2500\n",
      "    std.dev.: 9.4074\n",
      "    std.err.: 3.3260\n",
      "    skew: 1.6908\n",
      "    kurtosis: 1.3190\n",
      "  Element:[c]\n",
      "  == c\n",
      "    n :8\n",
      "    n valid:8\n",
      "    median: 38.5\n",
      "    mean: 39.7500\n",
      "    std.dev.: 25.0699\n",
      "    std.err.: 8.8635\n",
      "    skew: 0.1381\n",
      "    kurtosis: -1.7271\n",
      "  Element:[d]\n",
      "  == a\n",
      "    n :8\n",
      "    n valid:8\n",
      "    median: 137.5\n",
      "    mean: 197.5000\n",
      "    std.dev.: 190.9921\n",
      "    std.err.: 67.5259\n",
      "    skew: 0.5945\n",
      "    kurtosis: -1.3406\n",
      "  Element:[b]\n",
      "  == b\n",
      "    n :8\n",
      "    n valid:7\n",
      "    factors: a,c,33,b,d,88,20\n",
      "    mode: a\n",
      "    Distribution\n",
      "+----+---+--------+\n",
      "| a  | 1 | 14.29% |\n",
      "| b  | 1 | 14.29% |\n",
      "| c  | 1 | 14.29% |\n",
      "| d  | 1 | 14.29% |\n",
      "| 20 | 1 | 14.29% |\n",
      "| 33 | 1 | 14.29% |\n",
      "| 88 | 1 | 14.29% |\n",
      "+----+---+--------+\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "puts df.summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping and iterators\n",
    "\n",
    "Daru::DataFrame offers many iterators to loop over either rows or columns. \n",
    "\n",
    "**#each**\n",
    "\n",
    "`#each` works exactly like Array#each. The default mode for `each` is to iterate over the columns of the DataFrame. To iterate over rows you must pass the axis, i.e `:row` as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"130\", \"1110\", \"1140\", \"a20\"]\n"
     ]
    }
   ],
   "source": [
    "# Iterate over vectors\n",
    "\n",
    "e = []\n",
    "df.each do |vector|\n",
    "  e << vector[:a].to_s + vector[:latest].to_s\n",
    "end\n",
    "\n",
    "puts e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11, 44, 99, 176, 275, 396, 539, 300]\n"
     ]
    }
   ],
   "source": [
    "# Iterate over rows\n",
    "\n",
    "r = []\n",
    "df.each(:row) do |row|\n",
    "  r << row[:a] * row[:c]\n",
    "end\n",
    "\n",
    "puts r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**#map**\n",
    "\n",
    "The #map iterator works like Array#map. The value returned by each run of the block is added to an Array and the Array is returned.\n",
    "\n",
    "This method also accepts an `axis` argument, like `#each`. The default is `:vector`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7.25, 39.75, 197.5]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Map over vectors. \n",
    "\n",
    "# The `only_numerics` method returns a DataFrame which contains vectors \n",
    "# with only numerical values. Setting the `:clone` option to false will \n",
    "# return the same Vector objects that are contained in the original DataFrame.\n",
    "\n",
    "df.only_numerics(clone: false).map do |vector| \n",
    "  vector.mean\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7.666666666666667, 22.666666666666668, 42.0, 74.66666666666667, 111.66666666666667, 139.0, 207.66666666666666, 25.0]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Map over rows.\n",
    "\n",
    "# Calling `only_numerics` on a Daru::Vector will return a Vector with only numeric and\n",
    "# missing data. Data marked as 'missing' is not considered during statistical computation.\n",
    "\n",
    "df.map(:row) do |row|\n",
    "  row.only_numerics.mean\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**#recode**\n",
    "\n",
    "Recode works similarly to `#map`, but an important difference between the two is that recode returns a modified _Daru::DataFrame_ instead of an Array. For this reason, `#recode`expects that every run of the block to return a `Daru::Vector`.\n",
    "\n",
    "Just like map and each, recode also accepts an optional _axis_ argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:22133080  rows: 8  cols: 3<tr><th></th><th>a</th><th>c</th><th>d</th></tr><tr><td>a</td><td>7</td><td>77</td><td>275</td></tr><tr><td>b</td><td>15.0</td><td>125.0</td><td>505.5</td></tr><tr><td>c</td><td>3</td><td>33</td><td>99</td></tr><tr><td>d</td><td>4</td><td>44</td><td>176</td></tr><tr><td>e</td><td>5</td><td>55</td><td>275</td></tr><tr><td>f</td><td>6</td><td>66</td><td>396</td></tr><tr><td>g</td><td>7</td><td>77</td><td>539</td></tr><tr><td>latest</td><td>30</td><td>10</td><td>40</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:22133080 @name = a77a166f-9c49-48ea-9dcb-bdb32606e830 @size = 8>\n",
       "                    a          c          d \n",
       "         a          7         77        275 \n",
       "         b       15.0      125.0      505.5 \n",
       "         c          3         33         99 \n",
       "         d          4         44        176 \n",
       "         e          5         55        275 \n",
       "         f          6         66        396 \n",
       "         g          7         77        539 \n",
       "    latest         30         10         40 \n"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recode vectors\n",
    "\n",
    "df.only_numerics(clone: false).recode do |vector|\n",
    "  vector[:a] = vector[:d]  + vector[:c]\n",
    "  vector[:b] = vector.mean + vector[:a]\n",
    "  vector # <- return the vector to the block\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"5\">Daru::DataFrame:21467720  rows: 8  cols: 4<tr><th></th><th>a</th><th>c</th><th>d</th><th>b</th></tr><tr><td>a</td><td>0</td><td>11</td><td>11</td><td>0</td></tr><tr><td>b</td><td>-22</td><td>22</td><td>44</td><td>0</td></tr><tr><td>c</td><td>-66</td><td>33</td><td>99</td><td>33</td></tr><tr><td>d</td><td>-132</td><td>44</td><td>176</td><td>0</td></tr><tr><td>e</td><td>-220</td><td>55</td><td>275</td><td>0</td></tr><tr><td>f</td><td>-330</td><td>66</td><td>396</td><td>88</td></tr><tr><td>g</td><td>-462</td><td>77</td><td>539</td><td></td></tr><tr><td>latest</td><td>-30</td><td>10</td><td>40</td><td>20</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:21467720 @name = 7ebe63b4-aa3b-42f4-a0d1-c5b7d6813b77 @size = 8>\n",
       "                    a          c          d          b \n",
       "         a          0         11         11          0 \n",
       "         b        -22         22         44          0 \n",
       "         c        -66         33         99         33 \n",
       "         d       -132         44        176          0 \n",
       "         e       -220         55        275          0 \n",
       "         f       -330         66        396         88 \n",
       "         g       -462         77        539        nil \n",
       "    latest        -30         10         40         20 \n"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Recode rows\n",
    "\n",
    "df.recode(:row) do |row|\n",
    "  row[:a] = row[:c] - row[:d]\n",
    "  row[:b] = row[:b].to_i if row[:b].is_a?(String)\n",
    "  row\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**#collect**\n",
    "\n",
    "The `#collect` iterator works similar to `#map`, the only difference being that it returns a Daru::Vector comprising of the results of each block run. The resultant Vector has the same index as that of the axis over which `collect` has iterated.\n",
    "\n",
    "It also accepts the optional _axis_ argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:20466840  size: 4</th></tr><tr><th> </th><th>nil</th></tr><tr><td>a</td><td>9</td></tr><tr><td>c</td><td>99</td></tr><tr><td>d</td><td>495</td></tr><tr><td>b</td><td>121</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:20466840 @name = nil @size = 4 >\n",
       "    nil\n",
       "  a   9\n",
       "  c  99\n",
       "  d 495\n",
       "  b 121\n"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Collect Vectors\n",
    "\n",
    "df.collect do |vector|\n",
    "  vector[:c] + vector[:f]\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:20062900  size: 8</th></tr><tr><th> </th><th>nil</th></tr><tr><td>a</td><td>1</td></tr><tr><td>b</td><td>24</td></tr><tr><td>c</td><td>69</td></tr><tr><td>d</td><td>136</td></tr><tr><td>e</td><td>225</td></tr><tr><td>f</td><td>336</td></tr><tr><td>g</td><td>469</td></tr><tr><td>latest</td><td>60</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:20062900 @name = nil @size = 8 >\n",
       "          nil\n",
       "     a      1\n",
       "     b     24\n",
       "     c     69\n",
       "     d    136\n",
       "     e    225\n",
       "     f    336\n",
       "     g    469\n",
       "latest     60\n"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Collect Rows\n",
    "\n",
    "df.collect(:row) do |row|\n",
    "  row[:a] + row[:d] - row[:c]\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**#vector_by_calculation**\n",
    "\n",
    "`#vector_by_calculation` is a DSL that can be used for generating a Daru::Vector based on the results returned by the block.\n",
    "\n",
    "This DSL lets you refer to elements directly as methods inside the block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"2\">Daru::Vector:17919800  size: 8</th></tr><tr><th> </th><th>nil</th></tr><tr><td>a</td><td>23</td></tr><tr><td>b</td><td>68</td></tr><tr><td>c</td><td>135</td></tr><tr><td>d</td><td>224</td></tr><tr><td>e</td><td>335</td></tr><tr><td>f</td><td>468</td></tr><tr><td>g</td><td>623</td></tr><tr><td>latest</td><td>80</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::Vector:17919800 @name = nil @size = 8 >\n",
       "          nil\n",
       "     a     23\n",
       "     b     68\n",
       "     c    135\n",
       "     d    224\n",
       "     e    335\n",
       "     f    468\n",
       "     g    623\n",
       "latest     80\n"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.vector_by_calculation { a + c + d }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting\n",
    "\n",
    "Daru::DataFrame offers a robust `#sort` function which can be used for hierarchically sorting the Vectors in the DataFrame.\n",
    "\n",
    "Here are couple of examples to demonstrate a lot of the options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:17606280  rows: 5  cols: 3<tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><td>0</td><td>g</td><td>4</td><td>This</td></tr><tr><td>1</td><td>g</td><td>4</td><td>dataframe</td></tr><tr><td>2</td><td>g</td><td>335</td><td>is</td></tr><tr><td>3</td><td>sort</td><td>32</td><td>for</td></tr><tr><td>4</td><td>this</td><td>11</td><td>sorting</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:17606280 @name = 8bfa42b2-d724-4b78-9e13-7b0bfc156236 @size = 5>\n",
       "                    a          b          c \n",
       "         0          g          4       This \n",
       "         1          g          4  dataframe \n",
       "         2          g        335         is \n",
       "         3       sort         32        for \n",
       "         4       this         11    sorting \n"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({\n",
    "  a: ['g', 'g','g','sort', 'this'],\n",
    "  b: [4,4,335,32,11],\n",
    "  c: ['This', 'dataframe','is','for','sorting']\n",
    "  })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Array passed as an argument to 'sort' tells the method the order\n",
    "in which preference of sorting should be given to each Vector.\n",
    "\n",
    "The **:ascending** option will tell DataFrame the order in which you want\n",
    "the Vectors to be sorted. *true* for ascending sort and *false* for \n",
    "descending sort.\n",
    "\n",
    "The **:by** option lets you define a custom attribute for each vector to sort by.\n",
    "This works similarly to passing a block to Array#sort_by."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"4\">Daru::DataFrame:17102340  rows: 5  cols: 3<tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><td>2</td><td>g</td><td>335</td><td>is</td></tr><tr><td>0</td><td>g</td><td>4</td><td>This</td></tr><tr><td>1</td><td>g</td><td>4</td><td>dataframe</td></tr><tr><td>3</td><td>sort</td><td>32</td><td>for</td></tr><tr><td>4</td><td>this</td><td>11</td><td>sorting</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:17102340 @name = 8bfa42b2-d724-4b78-9e13-7b0bfc156236 @size = 5>\n",
       "                    a          b          c \n",
       "         2          g        335         is \n",
       "         0          g          4       This \n",
       "         1          g          4  dataframe \n",
       "         3       sort         32        for \n",
       "         4       this         11    sorting \n"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort([:a,:b,:c], ascending: [true, false, true], by: {c: lambda { |a| a.size }})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Additional examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort a dataframe with a vector sequence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:15834560  rows: 5  cols: 2<tr><th></th><th>a</th><th>b</th></tr><tr><td>2</td><td>1</td><td>3</td></tr><tr><td>0</td><td>1</td><td>5</td></tr><tr><td>3</td><td>2</td><td>2</td></tr><tr><td>1</td><td>2</td><td>4</td></tr><tr><td>4</td><td>3</td><td>1</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:15834560 @name = 2f0ecfcc-7a96-41c4-8cb6-dd674bdcb8a6 @size = 5>\n",
       "                    a          b \n",
       "         2          1          3 \n",
       "         0          1          5 \n",
       "         3          2          2 \n",
       "         1          2          4 \n",
       "         4          3          1 \n"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({a: [1,2,1,2,3], b: [5,4,3,2,1]})\n",
    "\n",
    "df.sort [:a, :b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort a dataframe without a block. Here nils will be handled automatically and appear at top. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:15003920  rows: 5  cols: 2<tr><th></th><th>a</th><th>b</th></tr><tr><td>1</td><td></td><td>3</td></tr><tr><td>3</td><td></td><td>1</td></tr><tr><td>0</td><td>-3</td><td>4</td></tr><tr><td>2</td><td>-1</td><td>2</td></tr><tr><td>4</td><td>5</td><td>4</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:15003920 @name = 9f505204-91a6-4eee-870f-0863a83c4cb8 @size = 5>\n",
       "                    a          b \n",
       "         1        nil          3 \n",
       "         3        nil          1 \n",
       "         0         -3          4 \n",
       "         2         -1          2 \n",
       "         4          5          4 \n"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({a: [-3,nil,-1,nil,5], b: [4,3,2,1,4]})\n",
    "\n",
    "df.sort([:a])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort a dataframe with a block with nils handled automatically. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:14432560  rows: 6  cols: 2<tr><th></th><th>a</th><th>b</th></tr><tr><td>2</td><td>1</td><td></td></tr><tr><td>5</td><td>1</td><td></td></tr><tr><td>4</td><td>-1</td><td>x</td></tr><tr><td>1</td><td>-1</td><td>aa</td></tr><tr><td>0</td><td></td><td>aaa</td></tr><tr><td>3</td><td></td><td>baaa</td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14432560 @name = 14c60806-0313-4f71-a4e0-94c0578ae21e @size = 6>\n",
       "                    a          b \n",
       "         2          1        nil \n",
       "         5          1        nil \n",
       "         4         -1          x \n",
       "         1         -1         aa \n",
       "         0        nil        aaa \n",
       "         3        nil       baaa \n"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({a: [nil,-1,1,nil,-1,1], b: ['aaa','aa',nil,'baaa','x',nil] })\n",
    "\n",
    "# df.sort [:b], by: {b: lambda { |a| a.length } }\n",
    "# This would give \"NoMethodError: undefined method `length' for nil:NilClass\"\n",
    "\n",
    "# Instead you could do the following if you want the nils to be handled automatically\n",
    "df.sort [:b], by: {b: lambda { |a| a.length } }, handle_nils: true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort a dataframe with a block with nils handled manually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><th colspan=\"3\">Daru::DataFrame:14040080  rows: 6  cols: 2<tr><th></th><th>a</th><th>b</th></tr><tr><td>4</td><td>-1</td><td>x</td></tr><tr><td>1</td><td>-1</td><td>aa</td></tr><tr><td>0</td><td></td><td>aaa</td></tr><tr><td>3</td><td></td><td>baaa</td></tr><tr><td>2</td><td>1</td><td></td></tr><tr><td>5</td><td>1</td><td></td></tr></table>"
      ],
      "text/plain": [
       "\n",
       "#<Daru::DataFrame:14040080 @name = d68ab08a-0688-4d64-989f-c895f0cf60db @size = 6>\n",
       "                    a          b \n",
       "         4         -1          x \n",
       "         1         -1         aa \n",
       "         0        nil        aaa \n",
       "         3        nil       baaa \n",
       "         2          1        nil \n",
       "         5          1        nil \n"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = Daru::DataFrame.new({a: [nil,-1,1,nil,-1,1], b: ['aaa','aa',nil,'baaa','x',nil] })\n",
    "\n",
    "# To print nils at the bottom one can use lambda { |a| (a.nil?)[1]:[0,a.length] }\n",
    "df.sort [:b], by: {b: lambda { |a| (a.nil?)?[1]:[0,a.length] } }, handle_nils: true"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Ruby 2.2.1",
   "language": "ruby",
   "name": "ruby"
  },
  "language_info": {
   "file_extension": ".rb",
   "mimetype": "application/x-ruby",
   "name": "ruby",
   "version": "2.2.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}