{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://xnd.io/images/xndlogo%20transparentbg.png\" align=\"center\" width=\"auto\">\n",
    "\n",
    "<h1 align=\"center\">What is xnd?</h1>\n",
    "\n",
    "xnd is a module that implements a container type for mapping all python values relevant for scientific computing directly to memory.\n",
    "xnd has a superset of features for typed memory found in similar libraries like numpy and apache arrow. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xnd import xnd\n",
    "import numpy as np\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python 3.7.3 | packaged by conda-forge | (default, Mar 27 2019, 23:01:00) \n",
      "[GCC 7.3.0]\n"
     ]
    }
   ],
   "source": [
    "print('Python %s' % sys.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Xnd container creation routines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating xnd container from Python data types\n",
    "\n",
    "- xnd supports type inference. This allows the user to create xnd container from Python data types. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([1, 2, 3, 4, 5], type='5 * int64')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xnd([1, 2, 3, 4, 5]) # xnd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3, 4, 5])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([1, 2, 3, 4, 5])  # numpy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([[1.0, 1.5], [-1.5, 1.0]], type='2 * 2 * float64')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xnd([[1., 1.5], [-1.5, 1.]]) # xnd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1. ,  1.5],\n",
       "       [-1.5,  1. ]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([[1, 1.5], [-1.5, 1]]) # numpy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You can see some differences with numpy at this level already, such as the array dimensionality being included in the type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The default string is variable-length in xnd. In numpy, you either choose a maximum size, or use object arrays with lower performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd(['this', 'is', 'a', 'test', 'notebook'], type='5 * string')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xnd(['this', 'is', 'a', 'test', 'notebook']) # xnd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['this', 'is', 'a', 'test', 'notebook'], dtype='<U8')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(['this', 'is', 'a', 'test', 'notebook']) # numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- xnd has a variable-length dimension type, which supports ragged arrays. [Ragged arrays](https://en.wikipedia.org/wiki/Jagged_array) also known as jagged arrays are array of arrays which can be of different sizes and producing rows of jagged edges when visualized as output. In contrast, two dimensional arrays are always rectangular. If you give this kind of data to numpy, it uses arrays which are slower and the array programming functionality in the ragged dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([[1, 5, 2], [1], [7, 9, 10, 20, 13]], type='var * var * int64')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xnd([[1, 5, 2], [1], [7, 9, 10, 20, 13]]) # xnd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([list([1, 5, 2]), list([1]), list([7, 9, 10, 20, 13])], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([[1, 5, 2], [1], [7, 9, 10, 20, 13]]) # numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Categorical Type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd(['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet', 'red', 'green'],\n",
       "    type='9 * categorical('red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet')')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "levels = ['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet']\n",
    "rainbow = xnd(['red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet', 'red', 'green'], levels=levels)\n",
    "rainbow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ndt(\"9 * categorical('red', 'orange', 'yellow', 'green', 'blue', 'indigo', 'violet')\")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rainbow.type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['red',\n",
       " 'orange',\n",
       " 'yellow',\n",
       " 'green',\n",
       " 'blue',\n",
       " 'indigo',\n",
       " 'violet',\n",
       " 'red',\n",
       " 'green']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rainbow.value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Structs: xnd provides a convenient way to create arrays of structs.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([{'title': 'Introduction to Digital Signal Processing', 'speaker': 'Allen Downey', 'room': 10},\n",
       "     {'title': 'Making Art with Python', 'speaker': 'Emily Xie', 'room': 16},\n",
       "     {'title': 'Foundations of Numerical Computing in Python', 'speaker': 'Scott Sanderson', 'room': 20},\n",
       "     {'title': 'Exploratory Data Visualization with Vega, Vega-Lite, and Altair',\n",
       "      'speaker': 'Jake VanderPlas',\n",
       "      'room': 21}],\n",
       "    type='4 * {title : string, speaker : string, room : int64}')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [{'title': 'Introduction to Digital Signal Processing', \n",
    "         'speaker': 'Allen Downey', \n",
    "          'room': 10},\n",
    "        {'title': 'Making Art with Python', \n",
    "         'speaker':'Emily Xie', \n",
    "         'room': 16},\n",
    "        {'title': 'Foundations of Numerical Computing in Python', \n",
    "         'speaker': 'Scott Sanderson', \n",
    "         'room': 20},\n",
    "        {'title':'Exploratory Data Visualization with Vega, Vega-Lite, and Altair', \n",
    "         'speaker':'Jake VanderPlas', \n",
    "         'room': 21}]\n",
    "\n",
    "x = xnd(data)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd({'title': 'Introduction to Digital Signal Processing', 'speaker': 'Allen Downey', 'room': 10},\n",
       "    type='{title : string, speaker : string, room : int64}')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd('Making Art with Python', type='string')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating xnd container from numpy arrays and records arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([[[0.11817033651791786, 0.6839453361584004, 0.2536092535496538, 0.5055059760011743, 0.5067576432750356],\n",
       "      [0.2407058584752727, 0.917665583250738, 0.8704926424782194, 0.7343222993465005, 0.5910180874913052],\n",
       "      [0.5784732883146575, 0.24073382655422526, 0.4994625009464535, 0.9768528225906218, 0.6219891558285271],\n",
       "      [0.6283109904322957, 0.6100928008157002, 0.09671910142133089, 0.9730523719507609, 0.40202043302233015]],\n",
       "     [[0.8123631051139746, 0.2764564198782935, 0.5577962501643701, 0.34262584837570087, 0.6934284288379087],\n",
       "      [0.8883322974424327, 0.34710528373845473, 0.5859331417671697, 0.9287901094069656, 0.7268583870243034],\n",
       "      [0.717240914249425, 0.023725909478674367, 0.8919150596382612, 0.9975249827738114, 0.6919546876794813],\n",
       "      [0.11544921395164576, 0.8551688052855212, 0.9800429375894515, 0.0999515741040673, 0.0964583576381497]],\n",
       "     [[0.24356407513913303, 0.5536004544475481, 0.3720522474923057, 0.630410357936431, 0.849146997258151],\n",
       "      [0.299330177070638, 0.4173690236773434, 0.8213706636491637, 0.268687077192604, 0.059057608602548384],\n",
       "      [0.6632819719439912, 0.09713680970610383, 0.1583002524139403, 0.4984218536455166, 0.6383858635378117],\n",
       "      [0.4485071912623374, 0.2522084331523653, 0.7343169569711295, 0.6571516222532012, 0.42948442645729346]]],\n",
       "    type='3 * 4 * 5 * float64')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.random.random(size=(3, 4, 5))\n",
    "xnd.from_buffer(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "xnd([{'foo': b'Hello\\x00', 'bar': {'A': 1, 'B': 2}}, {'foo': b'World\\x00', 'bar': {'A': 3, 'B': 4}}],\n",
       "    type='2 * {foo : fixed_bytes(size=6), bar : {A : int64, B : int64}}')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record_array = np.rec.array([('Hello', (1, 2)),\n",
    "                             ('World', (3, 4))], \n",
    "                            dtype=[('foo', 'S6'),\n",
    "                                   ('bar', [('A', int), ('B', int)])])\n",
    "xnd.from_buffer(record_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating xnd container with explicity types\n",
    "\n",
    "Creating an xnd container with explicit types has significant performance advantages for large arrays. This is because xnd does not have to infer the type for each element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 1_000_000\n",
    "a_list = [1] * N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22.5 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "xnd(a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.3 ms ± 266 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "types = f'{N} * int64'\n",
    "xnd(a_list, type=types)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}