{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Images, Darks, & Flats with EPICS area detector, ophyd, and Bluesky\n", "\n", "The HDF5 File Writer can be configured to save image frames into separate datasets within the same HDF5 data file. The selection of frame type (image frame, background/dark frame, white/flat frame) is made by use of an existing PV in area detector: `$(P):cam1:FrameType` which is an *mbbo* record. In ophyd, the readback version of this PV: `$(P):cam1:FrameType_RBV` is used to define operational values for the ophyd device. Be sure to configure the readback PV with the same values.\n", "\n", "Both these PVs (`$(P):cam1:FrameType` and `$(P):cam1:FrameType_RBV`) should be configured so the values on the RHS are the HDF5 address in the file where the image frame should be stored:\n", "\n", " PV.ZRST = /exchange/data\n", " PV.ONST = /exchange/data_dark\n", " PV.TWST = /exchange/data_white\n", " # make sure to put these in autosave so they are restored when the IOC is restarted!\n", "\n", "The HDF5 addresses shown are for the Data Exchange format. In NeXus, the detector data is stored within the instrument group, traditionally at `/entry/instrument/detector/`. We then hard link that into the `/entry/data` group. But, there is a big, fat, stinkin problem. The EPICS mbbo fields for the various string values (ZRST, ONST, TWST, ...) are only 25 characters long, not long enough to hold this path. So we write the data into the `/entry/data` group and hard link it (in the layout.xml file) to the instrument group.\n", "\n", " PV.ZRST = /entry/data/data\n", " PV.ONST = /entry/data/dark\n", " PV.TWST = /entry/data/white\n", " # make sure to put these in autosave so they are restored when the IOC is restarted!\n", "\n", "The area detector attributes XML file needs the selection PV included in its list. We'll call it `SaveDest` so we can use the same name in the layout file:\n", "\n", " \n", "\n", "Then, the HDF5 layout XML file refers to this `SaveDest` attribute in the setup (add this to the XML file just after the opening `hdf5_layout` and before the first `group` element)\n", "\n", " \n", "\n", "The name `detector_data_destination` is hard-coded in the source code of the HDF5 file writer.\n", "\n", "In the BlueSky plan, write the frame type with 0: image, 1: dark, 2: flat before acquiring the image frame of that type. Then the HDF5 file writer will direct the image frame to the correct dataset as specified by the ZRST, ONST, or TWST field, respectively." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Try it with a NeXus file and the AD SimDetector" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, configure an instance of the sim detector." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from ophyd import Component, Device, EpicsSignal, EpicsSignalRO, EpicsSignalWithRBV\n", "from ophyd import SingleTrigger, AreaDetector, SimDetector\n", "from ophyd import HDF5Plugin, ImagePlugin\n", "from ophyd.areadetector import ADComponent\n", "from ophyd.areadetector.filestore_mixins import FileStoreHDF5IterativeWrite\n", "\n", "image_file_path = \"/tmp/simdet/%Y/%m/%d/\"\n", "_ad_prefix = \"13SIM1:\"\n", "\n", "\n", "class MyHDF5Plugin(HDF5Plugin, FileStoreHDF5IterativeWrite):\n", " \"\"\"\n", " \"\"\"\n", " layout_filename = ADComponent(EpicsSignal, \"XMLFileName\")\n", " layout_filename_valid = ADComponent(EpicsSignal, \"XMLValid_RBV\", string=True)\n", " \n", "\n", "class MySingleTriggerHdf5SimDetector(SingleTrigger, SimDetector): \n", " \n", " image = Component(ImagePlugin, suffix=\"image1:\")\n", " hdf1 = Component(\n", " MyHDF5Plugin,\n", " suffix='HDF1:', \n", " root='/', # for databroker\n", " write_path_template=image_file_path, # for EPICS AD\n", " )" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "adsimdet = MySingleTriggerHdf5SimDetector(_ad_prefix, name='adsimdet')\n", "adsimdet.read_attrs.append(\"hdf1\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Setup Bluesky" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loading metadata history from /home/mintadmin/.config/bluesky/bluesky_history.db\n" ] } ], "source": [ "from bluesky import RunEngine\n", "from bluesky.utils import get_history\n", "import bluesky.plans as bp\n", "\n", "\n", "RE = RunEngine(get_history())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set the counting time per frame to something short." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "adsimdet.cam.stage_sigs[\"acquire_time\"] = 0.02" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, count `adsimdet` using default settings." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('4e811b82-c19c-4e85-b5c9-1a0fdf3d4b71',)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "RE(bp.count([adsimdet]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking inside the HDF5 file, did not find the FrameType PV." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Configure the attributes file (for the IOC)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'simDetectorAttributes.xml'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adsimdet.cam.nd_attributes_file.value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll add the PV to this file (which is located in the IOC's startup directory) and then reload that file into the EPICS AD." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "adsimdet.cam.nd_attributes_file.put(\"simDetectorAttributes.xml\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Count again and check if the PV is now in the HDF5 file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check that the attributes XML file was read and everything was OK with it. This is not in the ophyd device so we'll make a local signal to check this. Fix any errors before proceeding." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Attributes file OK'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "att_status = EpicsSignal(\"13SIM1:cam1:NDAttributesStatus\", name=\"att_status\", string=True)\n", "att_status.value" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('ababddb6-adaf-40bf-8d81-60c4f4aa080c',)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "RE(bp.count([adsimdet]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "note: took two image captures to see the attributes ... hmmm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've been using the default layout. We'll need our own so that we can customize it.\n", "\n", "In the IOC directory, copy the `hdf5_layout_nexus.xml` file (this is the file we have been using as a default) to a new file called `layout.xml` and tell the HDF5 file write to use it. Then, check that it was read and the file content was acceptable." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "adsimdet.hdf1.layout_filename.put(\"layout.xml\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Yes'" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adsimdet.hdf1.layout_filename_valid.value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try it and verify `SaveDest` is present in the output file." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('b0d92bb3-bf3b-48ff-9ebf-cffe0560254c',)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "RE(bp.count([adsimdet]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Configure our PV(s) for the *NeXus* addresses we want to use. We'll use PyEpics here since that will look more obvious. Still, we must reconnect our ophyd object after our change to the EPICS PVs to pick up this change." ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import epics\n", "\n", "# this is the PV we use as the `SaveDest` attribute\n", "epics.caput(\"13SIM1:cam1:FrameType.ZRST\", \"/entry/data/data\")\n", "epics.caput(\"13SIM1:cam1:FrameType.ONST\", \"/entry/data/dark\")\n", "epics.caput(\"13SIM1:cam1:FrameType.TWST\", \"/entry/data/flat\")\n", "\n", "# ophyd needs this configuration\n", "epics.caput(\"13SIM1:cam1:FrameType_RBV.ZRST\", \"/entry/data/data\")\n", "epics.caput(\"13SIM1:cam1:FrameType_RBV.ONST\", \"/entry/data/dark\")\n", "epics.caput(\"13SIM1:cam1:FrameType_RBV.TWST\", \"/entry/data/flat\")\n", "\n", "# re-connect the detector object to pick upthese changes\n", "adsimdet = MySingleTriggerHdf5SimDetector(_ad_prefix, name='adsimdet')\n", "adsimdet.read_attrs.append(\"hdf1\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also need to modify our layout file, adding datasets in the right places for the additional image types. Here's `layout.xml` after those edits are complete:\n", "\n", "```\n", "\n", "\n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t\n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t\n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t \n", "\t\t\n", " \n", "\n", " \n", " \n", " \n", " \n", "\n", "\t\t\n", "\t\t\n", "\t\t\n", "\t\t\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", "```\n", "\n", "Then, reload (and test) the XML layout file." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'Yes'" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "adsimdet.hdf1.layout_filename.put(\"layout.xml\")\n", "adsimdet.hdf1.layout_filename_valid.value" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Try it and check the output file. This won't be a great test since we are only writing one image type. Just check that it seems to work." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('75293b34-b3f2-482e-8852-6434dfc04da9',)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "RE(bp.count([adsimdet]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Series of images, darks, and flats\n", "\n", "Try this series: 4 images, 3 darks, 2 flats" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import bluesky.plan_stubs as bps\n", "import time\n", "\n", "def frame_set(det, frame_type=0, num_frames=1):\n", " print(\"type {}, frames {}\".format(frame_type, num_frames))\n", " yield from bps.mv(det.cam.frame_type, frame_type)\n", " for frame_num in range(num_frames):\n", " print(\"acquire {} frame {} of {}\".format(frame_type, frame_num+1, num_frames))\n", " yield from bps.mv(det.cam.acquire, 1)\n", " while det.cam.acquire.value != 0:\n", " # wait for acquisition to finish\n", " yield from bps.sleep(0.01)\n", " print(\"acquired\")\n", "\n", "\n", "def series(det, num_images=4, num_darks=3, num_flats=2):\n", " num_frames = [num_images, num_darks, num_flats]\n", " total = sum(num_frames)\n", " print(\"total frames:\", total)\n", "\n", " print(\"setup\")\n", " yield from bps.mv(\n", " det.hdf1.num_capture, total,\n", " det.hdf1.file_write_mode, 'Capture',\n", " det.cam.image_mode, \"Multiple\",\n", " )\n", " yield from bps.abs_set(\n", " det.hdf1.capture, 1,\n", " )\n", "\n", " for i, num in enumerate(num_frames):\n", " yield from frame_set(det, frame_type=i, num_frames=num)\n", "\n", " print(\"restore\")\n", " yield from bps.mv(\n", " det.hdf1.num_capture, 1,\n", " det.hdf1.file_write_mode, 'Single',\n", " det.cam.image_mode, \"Single\",\n", " det.cam.num_exposures, 1,\n", " det.cam.frame_type, 0,\n", " )\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total frames: 9\n", "setup\n", "type 0, frames 4\n", "acquire 0 frame 1 of 4\n", "acquired\n", "acquire 0 frame 2 of 4\n", "acquired\n", "acquire 0 frame 3 of 4\n", "acquired\n", "acquire 0 frame 4 of 4\n", "acquired\n", "type 1, frames 3\n", "acquire 1 frame 1 of 3\n", "acquired\n", "acquire 1 frame 2 of 3\n", "acquired\n", "acquire 1 frame 3 of 3\n", "acquired\n", "type 2, frames 2\n", "acquire 2 frame 1 of 2\n", "acquired\n", "acquire 2 frame 2 of 2\n", "acquired\n", "restore\n" ] }, { "data": { "text/plain": [ "()" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "RE(series(adsimdet))" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "#RE.abort()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#adsimdet.cam.acquire.put(0)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#adsimdet.hdf1.capture.put(0)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "#adsimdet.cam.frame_type.put(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Final HDF5 data file looks like this (for clarity, no attributes or array data shown):\n", "\n", "```\n", " entry:NXentry\n", " data:NXdata\n", " dark:NX_UINT8[3,19,33] = [ ... ]\n", " data:NX_UINT8[4,19,33] = [ ... ]\n", " flat:NX_UINT8[2,19,33] = [ ... ]\n", " instrument:NXinstrument\n", " NDAttributes:NXcollection\n", " AcquireTime:NX_FLOAT64[9] = [ ... ]\n", " AttributesFileNative:NX_INT8[9] = [ ... ]\n", " AttributesFileParam:NX_CHAR[256] = simDetectorAttributes.xml\n", " AttributesFileString:NX_CHAR[256] = simDetectorAttributes.xml\n", " CameraManufacturer:NX_CHAR[256] = Simulated detector\n", " CameraModel:NX_CHAR[256] = Basic simulator\n", " E:NX_FLOAT64[9] = [ ... ]\n", " Gettysburg:NX_CHAR[256] = Four score and seven years ago our fathers\n", " ID_Energy:NX_FLOAT32[9] = [ ... ]\n", " ID_Energy_EGU:NX_FLOAT32[9] = [ ... ]\n", " ImageCounter:NX_INT32[9] = [ ... ]\n", " MaxSizeX:NX_INT32[9] = [ ... ]\n", " MaxSizeY:NX_INT32[9] = [ ... ]\n", " NDArrayEpicsTSSec:NX_UINT32[9] = [ ... ]\n", " NDArrayEpicsTSnSec:NX_UINT32[9] = [ ... ]\n", " NDArrayTimeStamp:NX_FLOAT64[9] = [ ... ]\n", " NDArrayUniqueId:NX_INT32[9] = [ ... ]\n", " Pi:NX_FLOAT64[9] = [ ... ]\n", " RingCurrent:NX_FLOAT32[9] = [ ... ]\n", " RingCurrent_EGU:NX_FLOAT32[9] = [ ... ]\n", " SaveDest:NX_CHAR[256] = /entry/data/data\n", " Ten:NX_INT32[9] = [ ... ]\n", " timestamp:NX_FLOAT64[9,5] = [ ... ]\n", " detector:NXdetector\n", " dark --> /entry/data/dark\n", " data --> /entry/data/data\n", " flat --> /entry/data/flat\n", " NDAttributes:NXcollection\n", " ColorMode:NX_INT32[9] = [ ... ]\n", " performance\n", "```\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }