{ "metadata": { "name": "" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "[PyBroMo](http://tritemio.github.io/PyBroMo/) - Reference - Data format and internals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "*This notebook is part of [PyBroMo](http://tritemio.github.io/PyBroMo/) a \n", "python-based single-molecule Brownian motion diffusion simulator \n", "that simulates confocal [smFRET](http://en.wikipedia.org/wiki/Single-molecule_FRET)\n", "experiments. You can find the full list of notebooks in \n", "[Usage Examples](http://tritemio.github.io/PyBroMo/#usage-examples).*\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Summary.** *This notebook describes the HDF5 file format used to store simulation results. \n", "> It is also an example on how to use the pytables API to navigate through the HDF5 file.*" ] }, { "cell_type": "code", "collapsed": false, "input": [ "%run -i load_bromo.py" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "C:\\Data\\Antonio\\software\\src\\pybromo\n", "\n", "Current software revision:\n", " 4265047 Fix timestamps storage\n" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "\n", "WARNING -> Uncommitted changes:\n", "M brownian.py\n", "MM \"notebooks/PyBroMo - Reference - Data format and internals.ipynb\"\n", "M storage.py\n", "\n" ] } ], "prompt_number": 1 }, { "cell_type": "code", "collapsed": false, "input": [ "# Simulation time step\n", "t_step = 0.5e-6 # seconds\n", "\n", "# Time duration of the simulation\n", "t_max = .3 # seconds\n", "\n", "# Particles definition\n", "P = gen_particles(15, box)\n", "\n", "# Particle simulation definition\n", "S = ParticlesSimulation(D=D, t_step=t_step, t_max=t_max, particles=P, box=box, psf=psf)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 2 }, { "cell_type": "markdown", "metadata": {}, "source": [ "#HDF5 File format\n", "\n", "The simulation is saved in a HDF5 file, one for each running engine. The file has the following content:\n", "\n", "- **/parameters**\n", " \n", " * Numeric parameters (storead as scalar arrays)\n", " - `D`\n", " - `t_step`\n", " - `t_max`\n", " - `EID`\n", " - `ID`\n", " - `chunksize`: used for `emission` and `position` arrays\n", " - `np`: number of particles\n", " - `pMol`: particles concentration in pico-Molar\n", "\n", " * Non-numeric parameters (stored as group attributes)\n", " - `box`: the `Box()` object (stores boundaries and size)\n", " - `particles`: the `Particles()` object (stores a list of particles' starting positions)\n", "\n", "\n", "- **/psf**\n", " * Arrays of PSF used in the simulation. This is the raw array as saved from PSFLab. The name of the array is the same as the origina file name. The PSF array has the following attributes:\n", " - 'fname', 'dir_', 'x_step', 'z_step'\n", " * The array and its attributes allow to recreate the `NumericPSF()` object on a simulation reload.\n", " - **TODO**: 'wavelength', 'NA', and other PSFLab parameters\n", " * `default_psf`: hard link to the PSF used in the simualation, used as persistent name\n", "\n", "\n", "- **/trajectories**\n", " * `emission`: 2D array of emission traces: one row per particle. Shape: (num_particles, time)\n", " * `emission_tot`: 1D array of emission trace: total emission from all the particles: Shape: (time)\n", " * `position`: 3D array of positions. Shape (num_particles, 3, time)\n", "\n", "\n", "- **/timestamps**\n", " * Arrays of timestamps for different `rate_max`, `bg_rate` and `seed`.\n", "\n", "## How to access\n", "\n", "The HDF5 file handle is in `S.store.data_file` after you run `S.open_store()`:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "S.open_store(chunksize=2**20)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 3 }, { "cell_type": "code", "collapsed": false, "input": [ "# Simulate 3D diffusion and emission\n", "S.sim_motion_em_chunk(total_emission=False, delete_pos=True)" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "[4428] Simulating particle 0 \n", "[4428] Simulating particle 1 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 2 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 3 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 4 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 5 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 6 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 7 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 8 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 9 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 10 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 11 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 12 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 13 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "[4428] Simulating particle 14 " ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n" ] } ], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "# Generate timestamps\n", "S.sim_timestamps_em_store(max_rate=3e5, bg_rate=2e3, seed=1)" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 5 }, { "cell_type": "code", "collapsed": false, "input": [ "S.compact_name_core()" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 6, "text": [ "'7798be_D1.2e-11_15P_32pM_step0.5us'" ] } ], "prompt_number": 6 }, { "cell_type": "code", "collapsed": false, "input": [ "S.compact_name_core(t_max=True)" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 7, "text": [ "'7798be_D1.2e-11_15P_32pM_step0.5us_t_max0.3s'" ] } ], "prompt_number": 7 }, { "cell_type": "code", "collapsed": false, "input": [ "S.compact_name()" ], "language": "python", "metadata": {}, "outputs": [ { "metadata": {}, "output_type": "pyout", "prompt_number": 8, "text": [ "'7798be_D1.2e-11_15P_32pM_step0.5us_t_max0.3s_ID0-0'" ] } ], "prompt_number": 8 }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 9 }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#HDF5 File inspection" ] }, { "cell_type": "code", "collapsed": false, "input": [ "print 'Main groups:\\n'\n", "for node in S.store.data_file.root:\n", " print node\n", " for n in node:\n", " print '\\t%s' % n.name\n", " print '\\t %s' % n.title" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Main groups:\n", "\n", "/parameters (Group) 'Simulation parameters'\n", "\tD\n", "\t Diffusion coefficient (m^2/s)\n", "\tEID\n", "\t IPython Engine ID (int)\n", "\tID\n", "\t Simulation ID (int)\n", "\tchunksize\n", "\t Chunksize for arrays\n", "\tnp\n", "\t Number of simulated particles\n", "\tpico_mol\n", "\t Particles concentration (pM)\n", "\tt_max\n", "\t Simulation total time (s)\n", "\tt_step\n", "\t Simulation time-step (s)\n", "/psf (Group) 'PSFs used in the simulation'\n", "\tdefault_psf\n", "\t PSF x-z slice (PSFLab array)\n", "\txz_realistic_z50_150_160_580nm_n1335_HR2\n", "\t PSF x-z slice (PSFLab array)\n", "/timestamps (Group) 'Timestamps of emitted photons'\n", "\tts_max_rate300kcps_bg2000cps_seed1\n", "\t Simulated photon timestamps\n", "\tts_max_rate300kcps_bg2000cps_seed1_par\n", "\t Particle number for each timestamp\n", "/trajectories (Group) 'Simulated trajectories'\n", "\temission\n", "\t Emission trace of each particle\n", "\temission_tot\n", "\t Summed emission trace of all the particles\n" ] } ], "prompt_number": 9 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Group: /parameters" ] }, { "cell_type": "code", "collapsed": false, "input": [ "group = '/parameters'\n", "\n", "print 'Numeric attributes (nodes) in %s:\\n' % group\n", "\n", "print S.store.data_file.get_node(group)\n", "for node in S.store.data_file.get_node(group)._f_list_nodes():\n", " print '\\t%s' % node.name\n", " print '\\t %s' % node.title" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Numeric attributes (nodes) in /parameters:\n", "\n", "/parameters (Group) 'Simulation parameters'\n", "\tD\n", "\t Diffusion coefficient (m^2/s)\n", "\tEID\n", "\t IPython Engine ID (int)\n", "\tID\n", "\t Simulation ID (int)\n", "\tchunksize\n", "\t Chunksize for arrays\n", "\tnp\n", "\t Number of simulated particles\n", "\tpico_mol\n", "\t Particles concentration (pM)\n", "\tt_max\n", "\t Simulation total time (s)\n", "\tt_step\n", "\t Simulation time-step (s)\n" ] } ], "prompt_number": 10 }, { "cell_type": "code", "collapsed": false, "input": [ "group = '/parameters'\n", "\n", "print 'Attributes in %s:\\n' % group\n", "\n", "print S.store.data_file.get_node(group)\n", "for attr in S.store.data_file.get_node(group)._v_attrs._f_list():\n", " print '\\t%s' % attr\n", " print '\\t %s' % type(S.store.data_file.get_node(group)._v_attrs[attr])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Attributes in /parameters:\n", "\n", "/parameters (Group) 'Simulation parameters'\n", "\tbox\n", "\t \n", "\tparticles\n", "\t \n" ] } ], "prompt_number": 11 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Group /psf" ] }, { "cell_type": "code", "collapsed": false, "input": [ "group = '/psf'\n", "\n", "print 'Nodes in in %s:\\n' % group\n", "\n", "print S.store.data_file.get_node(group)\n", "for node in S.store.data_file.get_node(group)._f_list_nodes():\n", " print '\\t%s %s' % (node.name, node.shape)\n", " print '\\t %s' % node.title" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Nodes in in /psf:\n", "\n", "/psf (Group) 'PSFs used in the simulation'\n", "\tdefault_psf (193, 129)\n", "\t PSF x-z slice (PSFLab array)\n", "\txz_realistic_z50_150_160_580nm_n1335_HR2 (193L, 129L)\n", "\t PSF x-z slice (PSFLab array)\n" ] } ], "prompt_number": 20 }, { "cell_type": "markdown", "metadata": {}, "source": [ "###`default_psf` attributes" ] }, { "cell_type": "code", "collapsed": false, "input": [ "node_name = '/psf/default_psf'\n", "node = S.store.data_file.get_node(node_name)\n", "\n", "print \"\\n%s %s: '%s'\" % (node.name, node.shape, node.title)\n", "print '\\n List of attributes:'\n", "for attr in node.attrs._f_list():\n", " print '\\t%s' % attr\n", " print \"\\t %s\" % repr(node._v_attrs[attr])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "default_psf (193, 129): 'PSF x-z slice (PSFLab array)'\n", "\n", " List of attributes:\n", "\tdir_\n", "\t 'psf_data/'\n", "\tfname\n", "\t 'xz_realistic_z50_150_160_580nm_n1335_HR2'\n", "\tx_step\n", "\t 0.0625\n", "\tz_step\n", "\t 0.0625\n" ] } ], "prompt_number": 18 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Group /trajectories" ] }, { "cell_type": "code", "collapsed": false, "input": [ "group = '/trajectories'\n", "\n", "print 'Nodes in in %s:\\n' % group\n", "\n", "print S.store.data_file.get_node(group)\n", "for node in S.store.data_file.get_node(group)._f_list_nodes():\n", " print '\\t%s %s' % (node.name, node.shape)\n", " print '\\t %s' % node.title" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Nodes in in /trajectories:\n", "\n", "/trajectories (Group) 'Simulated trajectories'\n", "\temission (15, 600000)\n", "\t Emission trace of each particle\n", "\temission_tot (0,)\n", "\t Summed emission trace of all the particles\n" ] } ], "prompt_number": 19 }, { "cell_type": "markdown", "metadata": {}, "source": [ "###`emission` attributes" ] }, { "cell_type": "code", "collapsed": false, "input": [ "node_name = '/trajectories/emission'\n", "node = S.store.data_file.get_node(node_name)\n", "\n", "print \"\\n%s %s: '%s'\" % (node.name, node.shape, node.title)\n", "print '\\n List of attributes:'\n", "for attr in node.attrs._f_list():\n", " print '\\t%s' % attr\n", " print \"\\t %s\" % repr(node._v_attrs[attr])" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "\n", "emission (15, 600000): 'Emission trace of each particle'\n", "\n", " List of attributes:\n", "\tpsf_fname\n", "\t 'xz_realistic_z50_150_160_580nm_n1335_HR2'\n", "\tseed\n", "\t 1\n" ] } ], "prompt_number": 17 }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Group /timestamps" ] }, { "cell_type": "code", "collapsed": false, "input": [ "group = '/timestamps'\n", "\n", "print 'Nodes in in %s:\\n' % group\n", "\n", "print S.store.data_file.get_node(group)\n", "for node in S.store.data_file.get_node(group)._f_list_nodes():\n", " print '\\t%s' % node.name\n", " print '\\t %s' % node.title" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Nodes in in /timestamps:\n", "\n", "/timestamps (Group) 'Timestamps of emitted photons'\n", "\tts_max_rate300kcps_bg2000cps_seed1\n", "\t Simulated photon timestamps\n", "\tts_max_rate300kcps_bg2000cps_seed1_par\n", "\t Particle number for each timestamp\n" ] } ], "prompt_number": 16 }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "from IPython.core.display import HTML\n", "HTML(open(\"./styles/custom2.css\", \"r\").read())" ], "language": "python", "metadata": {}, "outputs": [ { "html": [ "\n" ], "metadata": {}, "output_type": "pyout", "prompt_number": 3, "text": [ "" ] } ], "prompt_number": 3 } ], "metadata": {} } ] }