{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Parallel GST using MPI\n", "The purpose of this tutorial is to demonstrate how to compute GST estimates in parallel (using multiple CPUs or \"processors\"). The core PyGSTi computational routines are written to take advantage of multiple processors via the MPI communication framework, and so one must have a version of MPI and the `mpi4py` python package installed in order use run pyGSTi calculations in parallel. \n", "\n", "Since `mpi4py` doesn't play nicely with Jupyter notebooks, this tutorial is a bit more clunky than the others. In it, we will create a standalone Python script that imports `mpi4py` and execute it.\n", "\n", "We will use as an example the same \"standard\" single-qubit model of the first tutorial. We'll first create a dataset, and then a script to be run in parallel which loads the data. The creation of a simulated data is performed in the same way as the first tutorial. Since *random* numbers are generated and used as simulated counts within the call to `generate_fake_data`, it is important that this is *not* done in a parallel environment, or different CPUs may get different data sets. (This isn't an issue in the typical situation when the data is obtained experimentally.)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "#Import pyGSTi and the \"stardard 1-qubit quantities for a model with X(pi/2), Y(pi/2), and idle gates\"\n", "import pygsti\n", "from pygsti.construction import std1Q_XYI\n", "\n", "#Create a data set\n", "target_model = std1Q_XYI.target_model()\n", "fiducials = std1Q_XYI.fiducials\n", "germs = std1Q_XYI.germs\n", "maxLengths = [1,2,4,8,16,32]\n", "\n", "mdl_datagen = target_model.depolarize(op_noise=0.1, spam_noise=0.001)\n", "listOfExperiments = pygsti.construction.make_lsgst_experiment_list(target_model.operations.keys(), fiducials, fiducials, germs, maxLengths)\n", "ds = pygsti.construction.generate_fake_data(mdl_datagen, listOfExperiments, nSamples=1000,\n", " sampleError=\"binomial\", seed=1234)\n", "pygsti.io.write_dataset(\"example_files/mpi_example_dataset.txt\", ds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we'll write a Python script that will load in the just-created `DataSet`, run GST on it, and write the output to a file. The only major difference between the contents of this script and previous examples is that the script imports `mpi4py` and passes a MPI comm object (`comm`) to the `do_long_sequence_gst` function. Since parallel computing is best used for computationaly intensive GST calculations, we also demonstrate how to set a per-processor memory limit to tell pyGSTi to partition its computations so as to not exceed this memory usage. Lastly, note the use of the `gaugeOptParams` argument of `do_long_sequence_gst`, which can be used to weight different model members differently during gauge optimization." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "mpiScript = \"\"\"\n", "import time\n", "import pygsti\n", "from pygsti.construction import std1Q_XYI\n", "\n", "#get MPI comm\n", "from mpi4py import MPI\n", "comm = MPI.COMM_WORLD\n", "\n", "print(\"Rank %d started\" % comm.Get_rank())\n", "\n", "#define target model, fiducials, and germs as before\n", "target_model = std1Q_XYI.target_model()\n", "fiducials = std1Q_XYI.fiducials\n", "germs = std1Q_XYI.germs\n", "maxLengths = [1,2,4,8,16,32]\n", "\n", "#tell gauge optimization to weight the operation matrix\n", "# elements 100x more heavily than the SPAM vector elements, and\n", "# to specifically weight the Gx gate twice as heavily as the other\n", "# gates.\n", "goParams = {'itemWeights':{'spam': 0.01, 'gates': 1.0, 'Gx': 2.0} }\n", "\n", "#Specify a per-core memory limit (useful for larger GST calculations)\n", "memLim = 2.1*(1024)**3 # 2.1 GB\n", "\n", "#Perform TP-constrained GST\n", "target_model.set_all_parameterizations(\"TP\")\n", " \n", "#load the dataset\n", "ds = pygsti.io.load_dataset(\"example_files/mpi_example_dataset.txt\")\n", "\n", "start = time.time()\n", "results = pygsti.do_long_sequence_gst(ds, target_model, fiducials, fiducials,\n", " germs, maxLengths,memLimit=memLim,\n", " gaugeOptParams=goParams, comm=comm,\n", " verbosity=2)\n", "end = time.time()\n", "print(\"Rank %d finished in %.1fs\" % (comm.Get_rank(), end-start))\n", "if comm.Get_rank() == 0:\n", " import pickle\n", " pickle.dump(results, open(\"example_files/mpi_example_results.pkl\",\"wb\"))\n", "\"\"\"\n", "with open(\"example_files/mpi_example_script.py\",\"w\") as f:\n", " f.write(mpiScript)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we run the script with 3 processors using `mpiexec`. The `mpiexec` executable should have been installed with your MPI distribution -- if it doesn't exist, try replacing `mpiexec` with `mpirun`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rank 0 started\r\n", "Rank 1 started\r\n", "Rank 2 started\r\n", "--- Circuit Creation ---\r\n", " 1702 sequences created\r\n", " Dataset has 1702 entries: 1702 utilized, 0 requested sequences were missing\r\n", "--- LGST ---\r\n", " Singular values of I_tilde (truncating to first 4 of 6) = \r\n", " 4.244089943192679\r\n", " 1.1594632778409208\r\n", " 0.9651516670737965\r\n", " 0.9297628363691268\r\n", " 0.049256811347238104\r\n", " 0.025150658372136828\r\n", " \r\n", " Singular values of target I_tilde (truncating to first 4 of 6) = \r\n", " 4.242640687119286\r\n", " 1.414213562373096\r\n", " 1.4142135623730956\r\n", " 1.4142135623730954\r\n", " 2.5038933168948026e-16\r\n", " 2.023452063009528e-16\r\n", " \r\n", "--- Iterative MLGST: Iter 1 of 6 92 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.13, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 95.7297 (92 data params - 43 model params = expected mean of 49; p-value = 7.43161e-05)\r\n", " Completed in 0.3s\r\n", " 2*Delta(log(L)) = 96.1163\r\n", " Iteration 1 took 0.3s\r\n", " \r\n", "--- Iterative MLGST: Iter 2 of 6 168 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.13, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 162.192 (168 data params - 43 model params = expected mean of 125; p-value = 0.0141238)\r\n", " Completed in 0.2s\r\n", " 2*Delta(log(L)) = 162.56\r\n", " Iteration 2 took 0.2s\r\n", " \r\n", "--- Iterative MLGST: Iter 3 of 6 450 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.13, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 484.676 (450 data params - 43 model params = expected mean of 407; p-value = 0.00480708)\r\n", " Completed in 0.5s\r\n", " 2*Delta(log(L)) = 485.572\r\n", " Iteration 3 took 0.5s\r\n", " \r\n", "--- Iterative MLGST: Iter 4 of 6 862 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.14, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 895.303 (862 data params - 43 model params = expected mean of 819; p-value = 0.0324382)\r\n", " Completed in 0.9s\r\n", " 2*Delta(log(L)) = 896.23\r\n", " Iteration 4 took 1.0s\r\n", " \r\n", "--- Iterative MLGST: Iter 5 of 6 1282 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.14, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 1350.86 (1282 data params - 43 model params = expected mean of 1239; p-value = 0.0140423)\r\n", " Completed in 1.5s\r\n", " 2*Delta(log(L)) = 1351.9\r\n", " Iteration 5 took 1.6s\r\n", " \r\n", "--- Iterative MLGST: Iter 6 of 6 1702 operation sequences ---: \r\n", " --- Minimum Chi^2 GST ---\r\n", " Memory limit = 2.10GB\r\n", " Cur, Persist, Gather = 0.15, 0.00, 0.21 GB\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Sum of Chi^2 = 1800.55 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.0081481)\r\n", " Completed in 2.3s\r\n", " 2*Delta(log(L)) = 1801.81\r\n", " Iteration 6 took 2.4s\r\n", " \r\n", " Switching to ML objective (last iteration)\r\n", " --- MLGST ---\r\n", " Memory: limit = 2.10GB(cur, persist, gthr = 0.15, 0.00, 0.21 GB)\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Maximum log(L) = 900.852 below upper bound of -2.84686e+06\r\n", " 2*Delta(log(L)) = 1801.7 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.00773372)\r\n", " Completed in 0.7s\r\n", " 2*Delta(log(L)) = 1801.7\r\n", " Final MLGST took 0.7s\r\n", " \r\n", "Iterative MLGST Total Time: 6.6s\r\n", " -- Adding Gauge Optimized (go0) --\r\n", "--- Re-optimizing logl after robust data scaling ---\r\n", " --- MLGST ---\r\n", " Memory: limit = 2.10GB(cur, persist, gthr = 0.15, 0.00, 0.21 GB)\r\n", " Finding num_nongauge_params is too expensive: using total params.\r\n", " Maximum log(L) = 900.852 below upper bound of -2.84686e+06\r\n", " 2*Delta(log(L)) = 1801.7 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.00773372)\r\n", " Completed in 0.5s\r\n", "Rank 2 finished in 9.4s\r\n", "Rank 1 finished in 9.4s\r\n", " -- Adding Gauge Optimized (go0) --\r\n", "Rank 0 finished in 9.4s\r\n" ] } ], "source": [ "! mpiexec -n 3 python3 \"example_files/mpi_example_script.py\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice in the above that output within `do_long_sequence_gst` is not duplicated (only the first processor outputs to stdout) so that the output looks identical to running on a single processor. Finally, we just need to read the pickled `Results` object from file and proceed with any post-processing analysis. In this case, we'll just create a report. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*** Creating workspace ***\n", "*** Generating switchboard ***\n", "Found standard clifford compilation from std1Q_XYI\n", "*** Generating tables ***\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/enielse/research/pyGSTi/packages/pygsti/report/factory.py:785: UserWarning:\n", "\n", "Idle tomography failed:\n", "Label{layers}\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " targetSpamBriefTable took 0.741819 seconds\n", " targetGatesBoxTable took 0.171605 seconds\n", " datasetOverviewTable took 0.119944 seconds\n", " bestGatesetSpamParametersTable took 0.00061 seconds\n", " bestGatesetSpamBriefTable took 0.160337 seconds\n", " bestGatesetSpamVsTargetTable took 0.10074 seconds\n", " bestGatesetGaugeOptParamsTable took 0.000563 seconds\n", " bestGatesetGatesBoxTable took 0.159274 seconds\n", " bestGatesetChoiEvalTable took 0.467779 seconds\n", " bestGatesetDecompTable took 0.217626 seconds\n", " bestGatesetEvalTable took 0.003944 seconds\n", " bestGermsEvalTable took 0.030369 seconds\n", " bestGatesetVsTargetTable took 0.064669 seconds\n", " bestGatesVsTargetTable_gv took 0.272684 seconds\n", " bestGatesVsTargetTable_gvgerms took 0.120676 seconds\n", " bestGatesVsTargetTable_gi took 0.011457 seconds\n", " bestGatesVsTargetTable_gigerms took 0.045032 seconds\n", " bestGatesVsTargetTable_sum took 0.2168 seconds\n", " bestGatesetErrGenBoxTable took 0.757526 seconds\n", " metadataTable took 0.08499 seconds\n", " stdoutBlock took 0.001135 seconds\n", " profilerTable took 0.001579 seconds\n", " softwareEnvTable took 0.039033 seconds\n", " exampleTable took 0.055899 seconds\n", " singleMetricTable_gv took 0.212782 seconds\n", " singleMetricTable_gi took 0.013763 seconds\n", " fiducialListTable took 0.00094 seconds\n", " prepStrListTable took 0.000207 seconds\n", " effectStrListTable took 0.000215 seconds\n", " colorBoxPlotKeyPlot took 0.075305 seconds\n", " germList2ColTable took 0.000329 seconds\n", " progressTable took 4.463347 seconds\n", "*** Generating plots ***\n", " gramBarPlot took 0.125494 seconds\n", " progressBarPlot took 0.08425 seconds\n", " progressBarPlot_sum took 0.000674 seconds\n", " finalFitComparePlot took 0.072736 seconds\n", " bestEstimateColorBoxPlot took 16.191067 seconds\n", " bestEstimateTVDColorBoxPlot took 14.47499 seconds\n", " bestEstimateColorScatterPlot took 17.164604 seconds\n", " bestEstimateColorHistogram took 14.966767 seconds\n", " progressTable_scl took 3.376393 seconds\n", " progressBarPlot_scl took 0.044108 seconds\n", " bestEstimateColorBoxPlot_scl took 11.72345 seconds\n", " bestEstimateColorScatterPlot_scl took 12.955372 seconds\n", " bestEstimateColorHistogram_scl took 11.788572 seconds\n", " dataScalingColorBoxPlot took 0.12912 seconds\n", "*** Merging into template file ***\n", " Rendering topSwitchboard took 0.000115 seconds\n", " Rendering maxLSwitchboard1 took 7.8e-05 seconds\n", " Rendering targetSpamBriefTable took 0.071255 seconds\n", " Rendering targetGatesBoxTable took 0.061378 seconds\n", " Rendering datasetOverviewTable took 0.001056 seconds\n", " Rendering bestGatesetSpamParametersTable took 0.001418 seconds\n", " Rendering bestGatesetSpamBriefTable took 0.135043 seconds\n", " Rendering bestGatesetSpamVsTargetTable took 0.001596 seconds\n", " Rendering bestGatesetGaugeOptParamsTable took 0.000922 seconds\n", " Rendering bestGatesetGatesBoxTable took 0.119739 seconds\n", " Rendering bestGatesetChoiEvalTable took 0.115429 seconds\n", " Rendering bestGatesetDecompTable took 0.070942 seconds\n", " Rendering bestGatesetEvalTable took 0.011932 seconds\n", " Rendering bestGermsEvalTable took 0.0441 seconds\n", " Rendering bestGatesetVsTargetTable took 0.001135 seconds\n", " Rendering bestGatesVsTargetTable_gv took 0.002487 seconds\n", " Rendering bestGatesVsTargetTable_gvgerms took 0.003616 seconds\n", " Rendering bestGatesVsTargetTable_gi took 0.00255 seconds\n", " Rendering bestGatesVsTargetTable_gigerms took 0.002752 seconds\n", " Rendering bestGatesVsTargetTable_sum took 0.002272 seconds\n", " Rendering bestGatesetErrGenBoxTable took 0.261762 seconds\n", " Rendering metadataTable took 0.002315 seconds\n", " Rendering stdoutBlock took 0.000737 seconds\n", " Rendering profilerTable took 0.001567 seconds\n", " Rendering softwareEnvTable took 0.002251 seconds\n", " Rendering exampleTable took 0.022291 seconds\n", " Rendering metricSwitchboard_gv took 4.1e-05 seconds\n", " Rendering metricSwitchboard_gi took 2.6e-05 seconds\n", " Rendering singleMetricTable_gv took 0.005312 seconds\n", " Rendering singleMetricTable_gi took 0.004573 seconds\n", " Rendering fiducialListTable took 0.002853 seconds\n", " Rendering prepStrListTable took 0.00201 seconds\n", " Rendering effectStrListTable took 0.001991 seconds\n", " Rendering colorBoxPlotKeyPlot took 0.02336 seconds\n", " Rendering germList2ColTable took 0.003847 seconds\n", " Rendering progressTable took 0.00421 seconds\n", " Rendering gramBarPlot took 0.023216 seconds\n", " Rendering progressBarPlot took 0.020221 seconds\n", " Rendering progressBarPlot_sum took 0.020006 seconds\n", " Rendering finalFitComparePlot took 0.020415 seconds\n", " Rendering bestEstimateColorBoxPlot took 0.192293 seconds\n", " Rendering bestEstimateTVDColorBoxPlot took 0.187625 seconds\n", " Rendering bestEstimateColorScatterPlot took 0.35193 seconds\n", " Rendering bestEstimateColorHistogram took 0.204445 seconds\n", " Rendering progressTable_scl took 0.004331 seconds\n", " Rendering progressBarPlot_scl took 0.019671 seconds\n", " Rendering bestEstimateColorBoxPlot_scl took 0.189318 seconds\n", " Rendering bestEstimateColorScatterPlot_scl took 0.348455 seconds\n", " Rendering bestEstimateColorHistogram_scl took 0.210257 seconds\n", " Rendering dataScalingColorBoxPlot took 0.043448 seconds\n", "Output written to example_files/mpi_example_brief directory\n", "Opening example_files/mpi_example_brief/main.html...\n", "*** Report Generation Complete! Total time 115.088s ***\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pickle\n", "results = pickle.load(open(\"example_files/mpi_example_results.pkl\",\"rb\"))\n", "pygsti.report.create_standard_report(results, \"example_files/mpi_example_brief\",\n", " title=\"MPI Example Report\", verbosity=2, auto_open=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.0" } }, "nbformat": 4, "nbformat_minor": 1 }