{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Histogramming, grouping, and binning\n",
    "\n",
    "## Overview\n",
    "\n",
    "Histogramming (see [sc.hist](../../generated/functions/scipp.hist.rst#scipp.hist)), grouping (using [sc.groupby](../groupby.ipynb)), and binning (see [Binned data](binned-data.ipynb)) all serve similar but slightly different purposes.\n",
    "Picking the optimal one of the three for a particular application may yield more natural code and better performance.\n",
    "Let us start by an example.\n",
    "Consider a table of scattered measurements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipp as sc\n",
    "\n",
    "N = 5000\n",
    "values = 10 * np.random.rand(N)\n",
    "table = sc.DataArray(\n",
    "    data=sc.array(\n",
    "        dims=['position'], unit=sc.units.counts, values=values, variances=values\n",
    "    ),\n",
    "    coords={\n",
    "        'x': sc.array(dims=['position'], unit='m', values=np.random.rand(N)),\n",
    "        'y': sc.array(dims=['position'], unit='m', values=np.random.rand(N)),\n",
    "    },\n",
    ")\n",
    "table.values *= 1.0 / np.exp(5.0 * table.coords['x'].values)\n",
    "sc.table(table['position', :5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We may now be interested in the total intensity (counts) as a function of `'x'`.\n",
    "There are three ways to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "xbins = sc.linspace('x', 0, 1, num=40, unit='m')\n",
    "ds = sc.Dataset(\n",
    "    {\n",
    "        'histogram': table.hist(x=xbins),\n",
    "        'groupby': table.groupby('x', bins=xbins).sum('position'),\n",
    "        'bin': table.bin(x=xbins).bins.sum(),\n",
    "    }\n",
    ")\n",
    "ds.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above plot we can only see a single line, since the three solutions yield exactly the same result (neglecting floating-point rounding errors):\n",
    "\n",
    "- `hist` sorts data points into 'x' bins, summing immediately.\n",
    "- `groupby` groups by 'x' and then sums (on-the-fly) all data points falling in the same 'x' bin.\n",
    "- `bin` sorts data points into 'x' bins.\n",
    "  Summing all rows in a bin yields the same result as grouping and summing directly.\n",
    "\n",
    "So in this case we get equivalent results, but the application areas differ, as described in more detail in the following sections.\n",
    "\n",
    "## Histogramming\n",
    "\n",
    "[scipp.hist](../../generated/functions/scipp.hist.rst) directly sums the data and is efficient.\n",
    "Limitations are:\n",
    "\n",
    "- When histogramming in more than one dimension, the implementation uses `sc.bin` internally, which may be less efficient and uses more memory.\n",
    "- Can only apply \"sum\" or \"nansum\" to accumulate into a bin.\n",
    "  [scipp.nanhist](../../generated/functions/scipp.nanhist.rst) is currently implemented differently and uses `sc.bin` internally.\n",
    "  It therefore uses more memory and may be less efficient.\n",
    "  \n",
    "We can also histogram binned data (since [binning](#Binning) preserves the `'y'` coord), to create 2-D (or N-D) histograms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned = table.bin(x=xbins)\n",
    "hist = binned.hist(y=30)\n",
    "hist.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another capability of `hist` is to histogram a dimension that has previously been binned with a different or higher resolution, i.e. different bin edges.\n",
    "Compare to the plot of the initial example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned = table.bin(x=xbins)\n",
    "binned.hist(x=100).plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping\n",
    "\n",
    "`groupby` is more flexible in terms of operations than can be applied and may be the go-to solution when a quick one-liner is required.\n",
    "Limitations are:\n",
    "\n",
    "- Can only group along a single dimension.\n",
    "- Works best for small to medium-sized data, or if data is already mostly sorted along the grouping dimension.\n",
    "  Slow if millions of small input slices contribute to each group.\n",
    "  \n",
    "`groupby` can also operate on binned data, combining bin contents by concatenation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned = table.bin(x=xbins)\n",
    "binned.coords['param'] = sc.array(\n",
    "    dims=['x'], values=(np.random.random(39) * 4).astype(np.int32)\n",
    ")\n",
    "grouped = binned.groupby('param').bins.concat('x')\n",
    "grouped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each output bin is a combination of multiple input bins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "grouped.values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binning\n",
    "  \n",
    "[scipp.bin](../../generated/functions/scipp.bin.rst) actually reorders data and meta data such that all data contributing to a bin is in a contiguous block.\n",
    "Binning along multiple dimensions is supported.\n",
    "Of the three options it is the only solution that supports *modifying* data in the grouped/binned layout.\n",
    "A variety of operations on such binned data is available.\n",
    "Limitations are:\n",
    "\n",
    "- Requires copying and reordering the input data and can thus become expensive.\n",
    "  \n",
    "In the above example the `'y'` information is dropped by `hist` and `groupby`, but `bin` preserves it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned = table.bin(x=xbins)\n",
    "binned.values[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we omit the call to `bins.sum` in the original example, we can subsequently apply another [histogramming](#Histogramming) or binning operation to the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned = binned.bin(y=100)\n",
    "binned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the 1-D example above, summing the bins is equivalent to histogramming binned data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binned.bins.sum().plot()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}