{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial on advanced topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the environment\n",
    "It is assumed you are familiar with basics of the library. If not - see _\"Tutorial on basic topics\"_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pymice as pm\n",
    "import glob\n",
    "\n",
    "pm.getTutorialData(quiet=True)\n",
    "\n",
    "dataFiles = glob.glob('C57_AB/*.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visits order\n",
    "Visit objects returned by _`.getVisits()`_ method are not ordered by default, which means they might be in a random order. In an extremal case the order may be different for every call of the method!\n",
    "\n",
    "Of course it is possible to sort the returned list, however it is also possible to have it already ordered by attribute(s) indicated by the _`order`_ parameter of the method.\n",
    "\n",
    "### Visits ordered by one attribute\n",
    "To have the list ordered by one attribute pass the name of the attribute as the _`order`_ parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ml = pm.Loader('C57_AB/2012-08-28 15.33.58.zip')\n",
    "visits = ml.getVisits(order='Start')\n",
    "for visit in visits[:5]:\n",
    "    print visit.Start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visits ordered by a nested attribute\n",
    "Since some attributes (e.g. _`.Animal`_) are objects with their own attributes, it is also possible to have the list sorted by such \"nested\" attributes (e.g. _`.Animal.Name`_)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visits = ml.getVisits(order='Animal.Name')\n",
    "for visit in visits[::500]:\n",
    "    print visit.Animal.Name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visits ordered by multiple attributes\n",
    "It is possible to have visits ordered firstly by one attibute, then by another one (and so on) by passing a sequence of attribute names as _`order`_ parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visits = ml.getVisits(order=('Cage', 'Corner'))\n",
    "for visit in visits[::500]:\n",
    "    print visit.Cage, visit.Corner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visit selection\n",
    "Usually only a subset of recorded visits is meaningful for certain analysis. The library provides methods for selection of such subsets.\n",
    "\n",
    "### Visits of a particular mouse\n",
    "It is common to analyse visits of cetrain animal.\n",
    "\n",
    "To select visits of a particular mouse pass its name as the _`mice`_ parameter of _`.getVisits()`_ method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visits = ml.getVisits(mice='C57 A 1')\n",
    "print len(visits)\n",
    "print visits[321].Animal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visits of a group of mice\n",
    "The _`mice`_ parameter accepts also multiple names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "visits = ml.getVisits(mice=['C57 A 1', 'C57 A 2'])\n",
    "visitorNames = [v.Animal.Name for v in visits]\n",
    "for mouse in set(visitorNames):\n",
    "    print '%s: %d visits' % (mouse, visitorNames.count(mouse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visits performed at particular time\n",
    "Analyses are very often focused on a certain time period (e.g. phase of experiment phase).\n",
    "\n",
    "To select visits before or after particular time you can use (respectively) _`start`_ and _`end`_ parameters of _`getVisits()`_ method.\n",
    "\n",
    "In the example visits started between 15:00 (inclusive) and 16:00 (exclusive) 28th August 2012 are being selected.\n",
    "**The time is given as _Eastern European Time_ (EET).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import pytz\n",
    "\n",
    "EET = pytz.timezone('Etc/GMT-2')\n",
    "\n",
    "timeFrom = datetime(2012, 8, 28, 15, 0, tzinfo=EET)\n",
    "timeTo = datetime(2012, 8, 28, 16, 0, tzinfo=EET)\n",
    "\n",
    "visits = ml.getVisits(start=timeFrom, end=timeTo)\n",
    "print min(v.Start for v in visits)\n",
    "print max(v.Start for v in visits)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment timeline configuration file\n",
    "It is rather inconvenient to have starts and ends of time periods hardcoded in the source code. It would be much more convenient to have the time periods defined in a configuration file.\n",
    "\n",
    "To aid that purpose library provides _`ExperimentTimeline`_ class for loading such files. The class inherits from both [_`ConfigParser.RawConfigParser`_ class of The Python Standard Library](https://docs.python.org/2/library/configparser.html#rawconfigparser-objects) and [_`matplotlib.ticker.Formatter`_ class of matplotlib library](http://matplotlib.org/api/ticker_api.html#matplotlib.ticker.Formatter).\n",
    "\n",
    "### File format\n",
    "The format of the file is an adaptation of [INI format](https://en.wikipedia.org/wiki/INI_file). To be more specific, it is the format supported by the [_`ConfigParser`_ module of The Python Standard Library](https://docs.python.org/2/library/configparser.html).\n",
    "\n",
    "Every section of the file describes a time period with following properties:\n",
    "* `start` - a start time of the period in `YYYY-MM-DD HH:MM` or `YYYY-MM-DD HH:MM:SS` format,\n",
    "* `end` - an end time of the period in `YYYY-MM-DD HH:MM` or `YYYY-MM-DD HH:MM:SS` format,\n",
    "* `tzinfo` - a name of timezone used by `start` and `end` property (it has to be recognizable by [_`timezone`_ function of the _`pytz`_ module](http://pytz.sourceforge.net/)).\n",
    "\n",
    "A configuration file might look like this:\n",
    "```\n",
    "[SA 1 dark]\n",
    "start = 2012-08-28 13:00\n",
    "end = 2012-08-29 01:00\n",
    "tzinfo = Etc/GMT-2\n",
    "\n",
    "[SA 1 light]\n",
    "start = 2012-08-29 01:00\n",
    "end = 2012-08-29 13:00\n",
    "tzinfo = Etc/GMT-2\n",
    "```\n",
    "\n",
    "### Loading a configuration file\n",
    "To load a configuration file pass a path to it to the class constructor. Then you will be able to list defined time period (with _`.sections()`_ method) and get their boundaries (with _`.getTime()`_ method)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "timeline = pm.Timeline('C57_AB/timeline.ini')\n",
    "\n",
    "for timePeriod in timeline.sections():\n",
    "    start, end = timeline.getTimeBounds(timePeriod)\n",
    "    print \"%s:\\t%s - %s\" % (timePeriod, start, end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boundaries of many time periods\n",
    "To get boundaries of a minimal time period covering several time periods defined in the config file pass a collection of their names as a parameter of _`.getTime()`_ method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "start, end = timeline.getTimeBounds(['Place Pref 1 dark',\n",
    "                                     'Place Pref 3 light'])\n",
    "print '%s - %s' % (start, end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Debugging a configuration file\n",
    "You might want to verify whether every time period defined in the file is defined as it was supposed to be. A simplest way for that is screening the file with the _`plotPhases()`_ function of the _`pymice.debug`_ module.\n",
    "\n",
    "IPython Notebook notes:\n",
    "* `%matplotlib inline` is a directive of [IPython Notebook](http://ipython.org/notebook.html) necessary to display the image,\n",
    "* `_` intercepts a [matplotlib.figure.Figure](http://matplotlib.org/api/figure_api.html#matplotlib.figure.Figure) instance returned by the function in order not to display the image twice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pymice.debug as pmd\n",
    "_ = pmd.plotPhases(timeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customising loaded data\n",
    "### Omitting nosepoke data loading\n",
    "In order to speed up the loading process you might disable loading of nosepoke data (if they are not necessary) by setting getNp option to False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ml = pm.Loader('C57_AB/2012-08-28 15.33.58.zip', getNp=False)\n",
    "visits = ml.getVisits(order=('Start', 'End'))\n",
    "visit = visits[4]\n",
    "print visit.Nosepokes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading additional data\n",
    "No log, environment nor hardware events data are loaded unless the loader is explicitely told to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print ml.getLog()\n",
    "print ml.getEnvironment()\n",
    "print ml.getHardwareEvents()\n",
    "ml = pm.Loader('C57_AB/2012-08-28 15.33.58.zip', getLog=True, getHw=True, getEnv=True)\n",
    "print len(ml.getLog())\n",
    "print len(ml.getEnvironment())\n",
    "print len(ml.getHardwareEvents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data validity tests\n",
    "\n",
    "It is important to be sure the data you are going to analyse are valid. For that purpose log analysis is necessary.\n",
    "\n",
    "### Data validator and log analyzers\n",
    "Log analysis is being performed by a `DataValidator` object with a collection of log analyzers given as arguments of its constructor.\n",
    "\n",
    "Instances of _`LickometerLogAnalyzer`_ class checks log for lickometer problems while _`PresenceLogAnalyzer`_ - for presence of unidentified animal.\n",
    "\n",
    "Remember that for log analysis log data must be loaded (_`getLog`_ flag of constructors od _`Loader`_ and _`Merger`_ objects must be set to _`True`_)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "loaders = [pm.Loader(filename, getLog=True) for filename in dataFiles]\n",
    "mm = pm.Merger(*loaders, getLog=True)\n",
    "\n",
    "dataValidator = pm.DataValidator(pm.LickometerLogAnalyzer(),\n",
    "                                 pm.PresenceLogAnalyzer())\n",
    "validatorReport = dataValidator(mm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results of log analysis\n",
    "After the analysis is performed, you can use _`TestMiceData`_ objects to check, if there is no problem with data in a period of interest (POI).\n",
    "\n",
    "The POI in the example is defined as a period from first to last registration (obtained with _`getStart()`_ and _`getEnd()`_ methods)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "start = mm.getStart()\n",
    "end = mm.getEnd()\n",
    "interval = (start, end)\n",
    "\n",
    "strInterval = '(from %s to %s)' % interval\n",
    "\n",
    "noLickometerProblems = pm.FailureInspector('Lickometer')\n",
    "noPresenceProblems = pm.FailureInspector('Presence')\n",
    "\n",
    "if noLickometerProblems(validatorReport, interval):\n",
    "    print \"lickometer ok \" + strInterval\n",
    "    \n",
    "else:\n",
    "    print \"lickometer problem \" + strInterval\n",
    "    \n",
    "if noPresenceProblems(validatorReport, interval):\n",
    "    print \"presences ok \" + strInterval\n",
    "    \n",
    "else:\n",
    "    print \"unknown mice presence problem \" + strInterval\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}