{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.1 Advanced Indexing\n",
    "\n",
    "## Indexing files\n",
    "\n",
    "As was shown earlier, we can create an index of the data space using the `index()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import signac\n",
    "\n",
    "project = signac.get_project(root='projects/tutorial')\n",
    "index = list(project.index())\n",
    "\n",
    "for doc in index[:3]:\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the `Collection` class to manage the index directly in-memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "index = signac.Collection(project.index())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This enables us for example, to quickly search for all indexes related to a specific state point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in index.find({'statepoint.p': 0.1}):\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point the index contains information about the statepoint and all data stored in the job document.\n",
    "If we want to include the `V.txt` text files we used to store data in, with the index, we need to tell **signac** the filename pattern and optionally the file format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = signac.Collection(project.index('.*\\.txt'))\n",
    "for doc in index.find(limit=2):\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index contains basic information about the files within our data space, such as the path and the *MD5* hash sum.\n",
    "The ``format`` field currently says ``File``, which is the default value.\n",
    "\n",
    "We can specify that all files ending with ``.txt`` are to be defined to be of ``TextFile`` format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = signac.Collection(project.index({'.*\\.txt': 'TextFile'}))\n",
    "print(index.find_one({'format': 'TextFile'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating a Master Index\n",
    "\n",
    "A *master index* is compiled from multiple other indexes, which is useful when operating on data compiled from multiple sources, such as multiple **signac** projects.\n",
    "\n",
    "To make a data space part of *master index*, we need to create a ``signac_access.py`` module.\n",
    "We use the access module to define how the index for the particular space is to be generated.\n",
    "We can create a basic access module using the `Project.create_access_module()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make sure to remoe any remnants from previous runs...\n",
    "% rm -f projects/tutorial/signac_access.py\n",
    "\n",
    "# This will generate a minimal access module:\n",
    "project.create_access_module(master=False)\n",
    "\n",
    "% cat projects/tutorial/signac_access.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When compiling a *master index*, **signac** will search for access modules named ``signac_access.py``.\n",
    "Whenever it finds a file with that name, it will import the module and compile all indeces yielded from a function called ``get_indeces()`` into the master index.\n",
    "\n",
    "Let's try that!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "master_index = signac.Collection(signac.index())\n",
    "for doc in master_index.find(limit=2):\n",
    "    print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note, that we executed the ``index()`` function without specifying the project directory.\n",
    "The function *crawled* through all sub-directories below the root directory in an attempt to find *acccess modules*.\n",
    "\n",
    "We can use the *access module* to control how exactly the index is generated, for example by adding filename and format definitions.\n",
    "Usually we could edit the file directly, here we will just overwrite the old one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "access_module = \\\n",
    "\"\"\"import signac\n",
    "\n",
    "def get_indeces(root):\n",
    "    yield signac.get_project(root).index({'.*\\.txt': 'TextFile'})\n",
    "\"\"\"\n",
    "\n",
    "with open('projects/tutorial/signac_access.py', 'w') as file:\n",
    "    file.write(access_module)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now files will also be part of the master index!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "master_index = signac.Collection(signac.index())\n",
    "print(master_index.find_one({'format': 'TextFile'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the ``signac.fetch()`` function to directly open files associated with a particular index document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in master_index.find({'format': 'TextFile'}, limit=3):\n",
    "    with signac.fetch(doc) as file:\n",
    "        p = doc['statepoint']['p']\n",
    "        V = [float(v) for v in file.read().strip().split(',')]\n",
    "        print(p, V)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think of `fetch()` like the built-in `open()` function. It allows us to retrieve and open files based on the index document (file id) instead of an absolute file path. This makes it easier to operate on data agnostic to its actual physical location."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that we can specify *access modules* for any kind of data space, it does not have to be a *signac project*!\n",
    "\n",
    "In the [next section](signac_202_Integration_with_pandas.ipynb), we will learn how to use indeces in combination with pandas dataframes."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}