{ "cells": [ { "cell_type": "markdown", "id": "6377d707-491b-4bd6-985c-72820709afb1", "metadata": {}, "source": [ "# Make an MTH5 from ZEN data\n", "\n", "This notebook provides an example of how to read in ZEN (.Z3D) files into an MTH5. " ] }, { "cell_type": "code", "execution_count": 1, "id": "1208c257-8064-4ed3-a7db-eebe14b248d2", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-09-07 18:20:17,973 [line 135] mth5.setup_logger - INFO: Logging file can be found C:\\Users\\jpeacock\\OneDrive - DOI\\Documents\\GitHub\\mth5\\logs\\mth5_debug.log\n" ] } ], "source": [ "from mth5.mth5 import MTH5\n", "from mth5.io.zen import Z3DCollection\n", "from mth5 import read_file" ] }, { "cell_type": "markdown", "id": "c0fe5fce-0345-4b25-b1bb-64268a7e5861", "metadata": {}, "source": [ "### Z3D Collection\n", "\n", "We will use the `Z3DCollection` to assemble the *.z3d* files into a logical order by schedule action or run. \n", "\n", "**Note**: `n_samples` is an estimate based on file size not the data. To get an accurate number you should read in the full file. Same with `start` and `end`. `start` is based on the schedule start time which is usually 2 seconds earlier than the data start because of instrument buffer while chaning sampling rates. `end` is based on file size and sample rate.\n", "\n", "The `Z3DCollection.get_runs()` will return a two level ordered dictionary (`OrderedDict`). The first level is keyed by station ID. These objects are in turn ordered dictionaries by run ID. Therefore you can loop over stations and runs. " ] }, { "cell_type": "code", "execution_count": 2, "id": "2e0dc972-54eb-43d3-af4f-0abc2c4834e1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 1 station with 2 runs\n" ] } ], "source": [ "zc = Z3DCollection(r\"c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3d_data\")\n", "runs = zc.get_runs(sample_rates=[4096, 256])\n", "print(f\"Found {len(runs)} station with {len(runs[list(runs.keys())[0]])} runs\")" ] }, { "cell_type": "code", "execution_count": 3, "id": "039570b6-e1c1-46ce-bdec-d92554e30497", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
surveystationrunstartendchannel_idcomponentfnsample_ratefile_sizen_samplessequence_numberinstrument_idcalibration_fn
5100sr4096_00012022-05-17 12:59:57+00:002022-05-17 13:09:53.349854+00:004exc:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3...4096.0964157224426491ZEN_024None
6100sr4096_00012022-05-17 12:59:57+00:002022-05-17 13:09:53.351807+00:005eyc:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3...4096.0964160424426571ZEN_024None
7100sr4096_00012022-05-17 12:59:57+00:002022-05-17 13:09:53.348877+00:001hxc:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3...4096.0964462824426451ZEN_024None
8100sr4096_00012022-05-17 12:59:57+00:002022-05-17 13:09:53.351318+00:002hyc:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3...4096.0964415624426551ZEN_024None
9100sr4096_00012022-05-17 12:59:57+00:002022-05-17 13:09:53.351562+00:003hzc:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3...4096.0964416024426561ZEN_024None
\n", "
" ], "text/plain": [ " survey station run start \\\n", "5 100 sr4096_0001 2022-05-17 12:59:57+00:00 \n", "6 100 sr4096_0001 2022-05-17 12:59:57+00:00 \n", "7 100 sr4096_0001 2022-05-17 12:59:57+00:00 \n", "8 100 sr4096_0001 2022-05-17 12:59:57+00:00 \n", "9 100 sr4096_0001 2022-05-17 12:59:57+00:00 \n", "\n", " end channel_id component \\\n", "5 2022-05-17 13:09:53.349854+00:00 4 ex \n", "6 2022-05-17 13:09:53.351807+00:00 5 ey \n", "7 2022-05-17 13:09:53.348877+00:00 1 hx \n", "8 2022-05-17 13:09:53.351318+00:00 2 hy \n", "9 2022-05-17 13:09:53.351562+00:00 3 hz \n", "\n", " fn sample_rate file_size \\\n", "5 c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3... 4096.0 9641572 \n", "6 c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3... 4096.0 9641604 \n", "7 c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3... 4096.0 9644628 \n", "8 c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3... 4096.0 9644156 \n", "9 c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3... 4096.0 9644160 \n", "\n", " n_samples sequence_number instrument_id calibration_fn \n", "5 2442649 1 ZEN_024 None \n", "6 2442657 1 ZEN_024 None \n", "7 2442645 1 ZEN_024 None \n", "8 2442655 1 ZEN_024 None \n", "9 2442656 1 ZEN_024 None " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "runs[\"100\"][\"sr4096_0001\"]" ] }, { "cell_type": "markdown", "id": "2ea10bfb-89ca-466b-895e-47840ada4569", "metadata": {}, "source": [ "## Build MTH5\n", "\n", "Now that we have a logical collection of files, lets load them into an MTH5. We will simply loop of the stations, runs, and channels in the ordered dictionary.\n", "\n", "There are a few things that we need to keep track of. \n", "\n", "- The station metadata pulled directly from the Z3D files can be input into the station metadata, be sure to use the `write_metadata` method to write the metadata to the MTH5.\n", "- The Z3D files have the coil response and zen response embedded in the file, so we can put those into the appropriate filter container in MTH5. This is important for calibrating later. \n", "- Since this is a MTH5 file version 0.2.0 the filters are in the `survey_group` so add them there.\n", "- If you want to calibrate the data set calibrate to `True`. " ] }, { "cell_type": "code", "execution_count": 4, "id": "77321550-53cc-4849-965f-e0a25fc579e9", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-09-07 18:20:18,860 [line 663] mth5.mth5.MTH5._initialize_file - INFO: Initialized MTH5 0.2.0 file c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3d_data\\from_z3d.h5 in mode a\n" ] } ], "source": [ "calibrate = True\n", "m = MTH5()\n", "if calibrate:\n", " m.data_level = 2\n", "m.open_mth5(zc.file_path.joinpath(\"from_z3d.h5\"))" ] }, { "cell_type": "code", "execution_count": 5, "id": "2957f307-ee67-4c61-90a3-cfdbc6e961f9", "metadata": {}, "outputs": [], "source": [ "survey_group = m.add_survey(\"test\")" ] }, { "cell_type": "code", "execution_count": 6, "id": "4b8c1f73-45d2-448f-a2af-a7d3ff77b438", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-09-07 18:20:22,178 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n", "2022-09-07 18:20:23,720 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n", "2022-09-07 18:20:25,140 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n", "2022-09-07 18:20:29,176 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n", "2022-09-07 18:20:30,853 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n", "2022-09-07 18:20:32,481 [line 221] mt_metadata.base.metadata.frequency_response_table_filter.complex_response - WARNING: Extrapolating, use values outside calibration frequencies with caution\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Wall time: 14 s\n" ] } ], "source": [ "%%time\n", "for station_id in runs.keys():\n", " station_group = survey_group.stations_group.add_station(station_id)\n", " station_group.metadata.update(zc.station_metadata_dict[station_id])\n", " station_group.write_metadata()\n", " for run_id, run_df in runs[station_id].items():\n", " run_group = station_group.add_run(run_id)\n", " for row in run_df.itertuples():\n", " ch_ts = read_file(row.fn)\n", " # NOTE: this is where the calibration occurs\n", " if calibrate:\n", " ch_ts = ch_ts.remove_instrument_response()\n", " run_group.from_channel_ts(ch_ts)\n", " " ] }, { "cell_type": "code", "execution_count": 7, "id": "9654c7b8-a25f-4769-98db-6cdbf6f5dab6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Wall time: 5.76 s\n" ] } ], "source": [ "%%time\n", "station_group.validate_station_metadata()\n", "station_group.write_metadata()\n", "\n", "survey_group.update_survey_metadata()\n", "survey_group.write_metadata()" ] }, { "cell_type": "markdown", "id": "4c715640-c4bf-4dc9-99c9-f3d2d133b164", "metadata": { "tags": [] }, "source": [ "#### MTH5 Structure\n", "\n", "Have a look at the MTH5 structure and make sure it looks correct." ] }, { "cell_type": "code", "execution_count": 8, "id": "a52dacb5-4b0b-4f5e-bcdf-e058c6f29094", "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true }, "tags": [] }, "outputs": [ { "data": { "text/plain": [ "/:\n", "====================\n", " |- Group: Experiment\n", " --------------------\n", " |- Group: Reports\n", " -----------------\n", " |- Group: Standards\n", " -------------------\n", " --> Dataset: summary\n", " ......................\n", " |- Group: Surveys\n", " -----------------\n", " |- Group: test\n", " --------------\n", " |- Group: Filters\n", " -----------------\n", " |- Group: coefficient\n", " ---------------------\n", " |- Group: dipole_55.00m\n", " -----------------------\n", " |- Group: dipole_56.00m\n", " -----------------------\n", " |- Group: zen_counts2mv\n", " -----------------------\n", " |- Group: fap\n", " -------------\n", " |- Group: ant4_2314_response\n", " ----------------------------\n", " --> Dataset: fap_table\n", " ........................\n", " |- Group: ant4_2324_response\n", " ----------------------------\n", " --> Dataset: fap_table\n", " ........................\n", " |- Group: ant4_2334_response\n", " ----------------------------\n", " --> Dataset: fap_table\n", " ........................\n", " |- Group: fir\n", " -------------\n", " |- Group: time_delay\n", " --------------------\n", " |- Group: zpk\n", " -------------\n", " |- Group: Reports\n", " -----------------\n", " |- Group: Standards\n", " -------------------\n", " --> Dataset: summary\n", " ......................\n", " |- Group: Stations\n", " ------------------\n", " |- Group: 100\n", " -------------\n", " |- Group: Transfer_Functions\n", " ----------------------------\n", " |- Group: sr256_0002\n", " --------------------\n", " --> Dataset: ex\n", " .................\n", " --> Dataset: ey\n", " .................\n", " --> Dataset: hx\n", " .................\n", " --> Dataset: hy\n", " .................\n", " --> Dataset: hz\n", " .................\n", " |- Group: sr4096_0001\n", " ---------------------\n", " --> Dataset: ex\n", " .................\n", " --> Dataset: ey\n", " .................\n", " --> Dataset: hx\n", " .................\n", " --> Dataset: hy\n", " .................\n", " --> Dataset: hz\n", " .................\n", " --> Dataset: channel_summary\n", " ..............................\n", " --> Dataset: tf_summary\n", " ........................." ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m" ] }, { "cell_type": "markdown", "id": "1d0c040c-560d-4208-932a-a9e1f261f371", "metadata": { "jupyter": { "source_hidden": true }, "tags": [] }, "source": [ "### Channel Summary\n", "\n", "Have a look at the channel summary and make sure everything looks good." ] }, { "cell_type": "code", "execution_count": 9, "id": "413436b3-2986-4dda-a3d9-3dc921adbce2", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
surveystationrunlatitudelongitudeelevationcomponentstartendn_samplessample_ratemeasurement_typeazimuthtiltunitshdf5_referencerun_hdf5_referencestation_hdf5_reference
0test100sr256_000240.497576-116.8211881456.7ex2022-05-17 13:09:58+00:002022-05-17 15:54:42+00:002530304256.0electric0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
1test100sr256_000240.497576-116.8211881456.7ey2022-05-17 13:09:58+00:002022-05-17 15:54:42+00:002530304256.0electric0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
2test100sr256_000240.497576-116.8211881456.7hx2022-05-17 13:09:58+00:002022-05-17 15:54:42+00:002530304256.0magnetic0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
3test100sr256_000240.497576-116.8211881456.7hy2022-05-17 13:09:58+00:002022-05-17 15:54:42+00:002530304256.0magnetic90.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
4test100sr256_000240.497576-116.8211881456.7hz2022-05-17 13:09:58+00:002022-05-17 15:54:42+00:002530304256.0magnetic0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
5test100sr4096_000140.497576-116.8211881456.7ex2022-05-17 12:59:58+00:002022-05-17 13:09:41.997559+00:0023920544096.0electric0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
6test100sr4096_000140.497576-116.8211881456.7ey2022-05-17 12:59:58+00:002022-05-17 13:09:41.999023+00:0023920604096.0electric0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
7test100sr4096_000140.497576-116.8211881456.7hx2022-05-17 12:59:58+00:002022-05-17 13:09:41.996094+00:0023920484096.0magnetic0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
8test100sr4096_000140.497576-116.8211881456.7hy2022-05-17 12:59:58+00:002022-05-17 13:09:41.997070+00:0023920524096.0magnetic90.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
9test100sr4096_000140.497576-116.8211881456.7hz2022-05-17 12:59:58+00:002022-05-17 13:09:41.998779+00:0023920594096.0magnetic0.00.0count<HDF5 object reference><HDF5 object reference><HDF5 object reference>
\n", "
" ], "text/plain": [ " survey station run latitude longitude elevation component \\\n", "0 test 100 sr256_0002 40.497576 -116.821188 1456.7 ex \n", "1 test 100 sr256_0002 40.497576 -116.821188 1456.7 ey \n", "2 test 100 sr256_0002 40.497576 -116.821188 1456.7 hx \n", "3 test 100 sr256_0002 40.497576 -116.821188 1456.7 hy \n", "4 test 100 sr256_0002 40.497576 -116.821188 1456.7 hz \n", "5 test 100 sr4096_0001 40.497576 -116.821188 1456.7 ex \n", "6 test 100 sr4096_0001 40.497576 -116.821188 1456.7 ey \n", "7 test 100 sr4096_0001 40.497576 -116.821188 1456.7 hx \n", "8 test 100 sr4096_0001 40.497576 -116.821188 1456.7 hy \n", "9 test 100 sr4096_0001 40.497576 -116.821188 1456.7 hz \n", "\n", " start end n_samples \\\n", "0 2022-05-17 13:09:58+00:00 2022-05-17 15:54:42+00:00 2530304 \n", "1 2022-05-17 13:09:58+00:00 2022-05-17 15:54:42+00:00 2530304 \n", "2 2022-05-17 13:09:58+00:00 2022-05-17 15:54:42+00:00 2530304 \n", "3 2022-05-17 13:09:58+00:00 2022-05-17 15:54:42+00:00 2530304 \n", "4 2022-05-17 13:09:58+00:00 2022-05-17 15:54:42+00:00 2530304 \n", "5 2022-05-17 12:59:58+00:00 2022-05-17 13:09:41.997559+00:00 2392054 \n", "6 2022-05-17 12:59:58+00:00 2022-05-17 13:09:41.999023+00:00 2392060 \n", "7 2022-05-17 12:59:58+00:00 2022-05-17 13:09:41.996094+00:00 2392048 \n", "8 2022-05-17 12:59:58+00:00 2022-05-17 13:09:41.997070+00:00 2392052 \n", "9 2022-05-17 12:59:58+00:00 2022-05-17 13:09:41.998779+00:00 2392059 \n", "\n", " sample_rate measurement_type azimuth tilt units \\\n", "0 256.0 electric 0.0 0.0 count \n", "1 256.0 electric 0.0 0.0 count \n", "2 256.0 magnetic 0.0 0.0 count \n", "3 256.0 magnetic 90.0 0.0 count \n", "4 256.0 magnetic 0.0 0.0 count \n", "5 4096.0 electric 0.0 0.0 count \n", "6 4096.0 electric 0.0 0.0 count \n", "7 4096.0 magnetic 0.0 0.0 count \n", "8 4096.0 magnetic 90.0 0.0 count \n", "9 4096.0 magnetic 0.0 0.0 count \n", "\n", " hdf5_reference run_hdf5_reference station_hdf5_reference \n", "0 \n", "1 \n", "2 \n", "3 \n", "4 \n", "5 \n", "6 \n", "7 \n", "8 \n", "9 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m.channel_summary.summarize()\n", "m.channel_summary.to_dataframe()" ] }, { "cell_type": "markdown", "id": "0960c550-fd15-4b0c-89e9-dc7c11f95b05", "metadata": {}, "source": [ "## Close the MTH5\n", "\n", "This is important, you should close the file after you are done using it. Otherwise bad things can happen if you try to open it with another program or Python interpreter." ] }, { "cell_type": "code", "execution_count": 10, "id": "e99299cb-0165-4b10-9c5a-1ee5dcbc82bf", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-09-07 18:20:39,445 [line 744] mth5.mth5.MTH5.close_mth5 - INFO: Flushing and closing c:\\Users\\jpeacock\\OneDrive - DOI\\mt\\example_z3d_data\\from_z3d.h5\n" ] } ], "source": [ "m.close_mth5()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" } }, "nbformat": 4, "nbformat_minor": 5 }