{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Parallel GST using MPI\n", "The purpose of this tutorial is to demonstrate how to compute GST estimates in parallel (using multiple CPUs or \"processors\"). The core PyGSTi computational routines are written to take advantage of multiple processors via the MPI communication framework, and so one must have a version of MPI and the `mpi4py` python package installed in order use run pyGSTi calculations in parallel. \n", "\n", "Since `mpi4py` doesn't play nicely with Jupyter notebooks, this tutorial is a bit more clunky than the others. In it, we will create a standalone Python script that imports `mpi4py` and execute it.\n", "\n", "We will use as an example the same \"standard\" single-qubit gate set of the first tutorial. We'll first create a dataset, and then a script to be run in parallel which loads the data. The creation of a simulated data is performed in the same way as the first tutorial. Since *random* numbers are generated and used as simulated counts within the call to `generate_fake_data`, it is important that this is *not* done in a parallel environment, or different CPUs may get different data sets. (This isn't an issue in the typical situation when the data is obtained experimentally.)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "#Import pyGSTi and the \"stardard 1-qubit quantities for a gateset with X(pi/2), Y(pi/2), and idle gates\"\n", "import pygsti\n", "from pygsti.construction import std1Q_XYI\n", "\n", "#Create a data set\n", "gs_target = std1Q_XYI.gs_target\n", "fiducials = std1Q_XYI.fiducials\n", "germs = std1Q_XYI.germs\n", "maxLengths = [1,2,4,8,16,32]\n", "\n", "gs_datagen = gs_target.depolarize(gate_noise=0.1, spam_noise=0.001)\n", "listOfExperiments = pygsti.construction.make_lsgst_experiment_list(gs_target.gates.keys(), fiducials, fiducials, germs, maxLengths)\n", "ds = pygsti.construction.generate_fake_data(gs_datagen, listOfExperiments, nSamples=1000,\n", " sampleError=\"binomial\", seed=1234)\n", "pygsti.io.write_dataset(\"example_files/mpi_example_dataset.txt\", ds)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Next, we'll write a Python script that will load in the just-created `DataSet`, run GST on it, and write the output to a file. The only major difference between the contents of this script and previous examples is that the script imports `mpi4py` and passes a MPI comm object (`comm`) to the `do_long_sequence_gst` function. Since parallel computing is best used for computationaly intensive GST calculations, we also demonstrate how to set a per-processor memory limit to tell pyGSTi to partition its computations so as to not exceed this memory usage. Lastly, note the use of the `gaugeOptParams` argument of `do_long_sequence_gst`, which can be used to weight different gate set members differently during gauge optimization." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "mpiScript = \"\"\"\n", "import time\n", "import pygsti\n", "from pygsti.construction import std1Q_XYI\n", "\n", "#get MPI comm\n", "from mpi4py import MPI\n", "comm = MPI.COMM_WORLD\n", "\n", "print(\"Rank %d started\" % comm.Get_rank())\n", "\n", "#define target gateset, fiducials, and germs as before\n", "gs_target = std1Q_XYI.gs_target\n", "fiducials = std1Q_XYI.fiducials\n", "germs = std1Q_XYI.germs\n", "maxLengths = [1,2,4,8,16,32]\n", "\n", "#tell gauge optimization to weight the gate matrix\n", "# elements 100x more heavily than the SPAM vector elements, and\n", "# to specifically weight the Gx gate twice as heavily as the other\n", "# gates.\n", "goParams = {'itemWeights':{'spam': 0.01, 'gates': 1.0, 'Gx': 2.0} }\n", "\n", "#Specify a per-core memory limit (useful for larger GST calculations)\n", "memLim = 2.1*(1024)**3 # 2.1 GB\n", "\n", "#Perform TP-constrained GST\n", "gs_target.set_all_parameterizations(\"TP\")\n", " \n", "#load the dataset\n", "ds = pygsti.io.load_dataset(\"example_files/mpi_example_dataset.txt\")\n", "\n", "start = time.time()\n", "results = pygsti.do_long_sequence_gst(ds, gs_target, fiducials, fiducials,\n", " germs, maxLengths,memLimit=memLim,\n", " gaugeOptParams=goParams, comm=comm,\n", " verbosity=2)\n", "end = time.time()\n", "print(\"Rank %d finished in %.1fs\" % (comm.Get_rank(), end-start))\n", "if comm.Get_rank() == 0:\n", " import pickle\n", " pickle.dump(results, open(\"example_files/mpi_example_results.pkl\",\"wb\"))\n", "\"\"\"\n", "with open(\"example_files/mpi_example_script.py\",\"w\") as f:\n", " f.write(mpiScript)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Next, we run the script with 3 processors using `mpiexec`. The `mpiexec` executable should have been installed with your MPI distribution -- if it doesn't exist, try replacing `mpiexec` with `mpirun`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rank 1 started\n", "Rank 2 started\n", "Rank 0 started\n", "--- Gate Sequence Creation ---\n", " 1702 sequences created\n", " Dataset has 1702 entries: 1702 utilized, 0 requested sequences were missing\n", "--- LGST ---\n", " Singular values of I_tilde (truncating to first 4 of 6) = \n", " 4.245058951635965\n", " 1.1585845663892917\n", " 0.9677945789630839\n", " 0.9223871580426515\n", " 0.07048465733093594\n", " 0.0213024962973271\n", " \n", " Singular values of target I_tilde (truncating to first 4 of 6) = \n", " 4.242640687119285\n", " 1.4142135623730954\n", " 1.4142135623730947\n", " 1.4142135623730945\n", " 3.1723744950054595e-16\n", " 1.0852733691121267e-16\n", " \n", "--- Iterative MLGST: Iter 1 of 6 92 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.11, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 86.4516 (92 data params - 43 model params = expected mean of 49; p-value = 0.000769292)\n", " Completed in 0.3s\n", " 2*Delta(log(L)) = 86.7954\n", " Iteration 1 took 0.4s\n", " \n", "--- Iterative MLGST: Iter 2 of 6 168 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.11, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 167.265 (168 data params - 43 model params = expected mean of 125; p-value = 0.00692365)\n", " Completed in 0.3s\n", " 2*Delta(log(L)) = 167.759\n", " Iteration 2 took 0.5s\n", " \n", "--- Iterative MLGST: Iter 3 of 6 450 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.11, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 477.858 (450 data params - 43 model params = expected mean of 407; p-value = 0.00876085)\n", " Completed in 0.6s\n", " 2*Delta(log(L)) = 478.93\n", " Iteration 3 took 1.2s\n", " \n", "--- Iterative MLGST: Iter 4 of 6 862 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.11, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 890.263 (862 data params - 43 model params = expected mean of 819; p-value = 0.0419469)\n", " Completed in 1.1s\n", " 2*Delta(log(L)) = 891.425\n", " Iteration 4 took 2.0s\n", " \n", "--- Iterative MLGST: Iter 5 of 6 1282 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.12, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 1343.35 (1282 data params - 43 model params = expected mean of 1239; p-value = 0.0200193)\n", " Completed in 1.5s\n", " 2*Delta(log(L)) = 1344.71\n", " Iteration 5 took 2.9s\n", " \n", "--- Iterative MLGST: Iter 6 of 6 1702 gate strings ---: \n", " --- Minimum Chi^2 GST ---\n", " Memory limit = 2.10GB\n", " Cur, Persist, Gather = 0.13, 0.00, 0.21 GB\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Sum of Chi^2 = 1791.49 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.0121291)\n", " Completed in 1.9s\n", " 2*Delta(log(L)) = 1793.12\n", " Iteration 6 took 3.8s\n", " \n", " Switching to ML objective (last iteration)\n", " --- MLGST ---\n", " Memory: limit = 2.10GB(cur, persist, gthr = 0.13, 0.00, 0.21 GB)\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Maximum log(L) = 896.511 below upper bound of -2.84686e+06\n", " 2*Delta(log(L)) = 1793.02 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.011354)\n", " Completed in 4.4s\n", " 2*Delta(log(L)) = 1793.02\n", " Final MLGST took 4.4s\n", " \n", "Iterative MLGST Total Time: 15.2s\n", " -- Adding Gauge Optimized (go0) --\n", "--- Re-optimizing logl after robust data scaling ---\n", " --- MLGST ---\n", " Memory: limit = 2.10GB(cur, persist, gthr = 0.13, 0.00, 0.21 GB)\n", " Finding num_nongauge_params is too expensive: using total params.\n", " Maximum log(L) = 847.34 below upper bound of -2.84686e+06\n", " 2*Delta(log(L)) = 1694.68 (1702 data params - 43 model params = expected mean of 1659; p-value = 0.265464)\n", " Completed in 4.5s\n", "Rank 1 finished in 41.4s\n", "Rank 2 finished in 41.5s\n", " -- Adding Gauge Optimized (go0) --\n", "Rank 0 finished in 41.5s\n" ] } ], "source": [ "! mpiexec -n 3 python3 \"example_files/mpi_example_script.py\"" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "Notice in the above that output within `do_long_sequence_gst` is not duplicated (only the first processor outputs to stdout) so that the output looks identical to running on a single processor. Finally, we just need to read the pickled `Results` object from file and proceed with any post-processing analysis. In this case, we'll just create a report. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "*** Creating workspace ***\n", "*** Generating switchboard ***\n", "Found standard clifford compilation from std1Q_XYI\n", "*** Generating tables ***\n", " targetSpamBriefTable took 0.036339 seconds\n", " targetGatesBoxTable took 0.042545 seconds\n", " datasetOverviewTable took 0.074418 seconds\n", " bestGatesetSpamParametersTable took 0.000744 seconds\n", " bestGatesetSpamBriefTable took 0.044855 seconds\n", " bestGatesetSpamVsTargetTable took 0.10052 seconds\n", " bestGatesetGaugeOptParamsTable took 0.000615 seconds\n", " bestGatesetGatesBoxTable took 0.043628 seconds\n", " bestGatesetChoiEvalTable took 0.064883 seconds\n", " bestGatesetDecompTable took 0.053313 seconds\n", " bestGatesetEvalTable took 0.004118 seconds\n", " bestGermsEvalTable took 0.02304 seconds\n", " bestGatesetVsTargetTable took 1.045753 seconds\n", " bestGatesVsTargetTable_gv took 0.19339 seconds\n", " bestGatesVsTargetTable_gvgerms took 0.069318 seconds\n", " bestGatesVsTargetTable_gi took 0.012074 seconds\n", " bestGatesVsTargetTable_gigerms took 0.027005 seconds\n", " bestGatesVsTargetTable_sum took 0.18262 seconds\n", " bestGatesetErrGenBoxTable took 0.2096 seconds\n", " metadataTable took 0.098872 seconds\n", " stdoutBlock took 0.001928 seconds\n", " profilerTable took 0.00188 seconds\n", " softwareEnvTable took 0.068032 seconds\n", " exampleTable took 0.017096 seconds\n", " singleMetricTable_gv took 0.174502 seconds\n", " singleMetricTable_gi took 0.015208 seconds\n", " fiducialListTable took 0.001194 seconds\n", " prepStrListTable took 0.000576 seconds\n", " effectStrListTable took 0.000275 seconds\n", " colorBoxPlotKeyPlot took 0.019118 seconds\n", " germList2ColTable took 0.000623 seconds\n", " progressTable took 5.78273 seconds\n", "*** Generating plots ***\n", " gramBarPlot took 0.052711 seconds\n", " progressBarPlot took 0.381747 seconds\n", " progressBarPlot_sum took 0.000671 seconds\n", " finalFitComparePlot took 0.134434 seconds\n", " bestEstimateColorBoxPlot took 21.550004 seconds\n", " bestEstimateTVDColorBoxPlot took 18.409911 seconds\n", " bestEstimateColorScatterPlot took 21.664449 seconds\n", " bestEstimateColorHistogram took 19.050734 seconds\n", " progressTable_scl took 4.790609 seconds\n", " progressBarPlot_scl took 0.295817 seconds\n", " bestEstimateColorBoxPlot_scl took 18.492381 seconds\n", " bestEstimateColorScatterPlot_scl took 21.518567 seconds\n", " bestEstimateColorHistogram_scl took 18.695508 seconds\n", " dataScalingColorBoxPlot took 0.143529 seconds\n", "*** Merging into template file ***\n", " Rendering bestGatesetChoiEvalTable took 0.033745 seconds\n", " Rendering stdoutBlock took 0.001235 seconds\n", " Rendering dataScalingColorBoxPlot took 0.019759 seconds\n", " Rendering bestGatesetGatesBoxTable took 0.04592 seconds\n", " Rendering targetSpamBriefTable took 0.022889 seconds\n", " Rendering bestGatesetVsTargetTable took 0.001444 seconds\n", " Rendering singleMetricTable_gi took 0.007747 seconds\n", " Rendering profilerTable took 0.002771 seconds\n", " Rendering softwareEnvTable took 0.00474 seconds\n", " Rendering targetGatesBoxTable took 0.02136 seconds\n", " Rendering bestEstimateColorHistogram took 0.048994 seconds\n", " Rendering bestEstimateColorBoxPlot_scl took 0.073053 seconds\n", " Rendering metricSwitchboard_gv took 0.000124 seconds\n", " Rendering bestGatesetDecompTable took 0.02705 seconds\n", " Rendering germList2ColTable took 0.003683 seconds\n", " Rendering bestGatesetSpamVsTargetTable took 0.003372 seconds\n", " Rendering fiducialListTable took 0.003846 seconds\n", " Rendering exampleTable took 0.009154 seconds\n", " Rendering bestGatesetErrGenBoxTable took 0.088269 seconds\n", " Rendering metricSwitchboard_gi took 9.4e-05 seconds\n", " Rendering metadataTable took 0.005963 seconds\n", " Rendering bestEstimateColorHistogram_scl took 0.045217 seconds\n", " Rendering bestGatesVsTargetTable_gigerms took 0.005504 seconds\n", " Rendering singleMetricTable_gv took 0.008137 seconds\n", " Rendering progressBarPlot_sum took 0.003947 seconds\n", " Rendering bestGatesVsTargetTable_gvgerms took 0.007567 seconds\n", " Rendering effectStrListTable took 0.002663 seconds\n", " Rendering maxLSwitchboard1 took 0.000205 seconds\n", " Rendering bestEstimateColorBoxPlot took 0.0746 seconds\n", " Rendering progressTable_scl took 0.008232 seconds\n", " Rendering gramBarPlot took 0.00424 seconds\n", " Rendering bestGatesetSpamParametersTable took 0.002144 seconds\n", " Rendering colorBoxPlotKeyPlot took 0.012292 seconds\n", " Rendering bestEstimateColorScatterPlot took 0.058579 seconds\n", " Rendering bestGermsEvalTable took 0.083469 seconds\n", " Rendering finalFitComparePlot took 0.004606 seconds\n", " Rendering bestGatesVsTargetTable_sum took 0.004716 seconds\n", " Rendering datasetOverviewTable took 0.001223 seconds\n", " Rendering bestGatesVsTargetTable_gv took 0.004931 seconds\n", " Rendering progressTable took 0.008913 seconds\n", " Rendering progressBarPlot_scl took 0.003431 seconds\n", " Rendering progressBarPlot took 0.00421 seconds\n", " Rendering bestGatesVsTargetTable_gi took 0.00529 seconds\n", " Rendering prepStrListTable took 0.003234 seconds\n", " Rendering topSwitchboard took 0.000173 seconds\n", " Rendering bestGatesetSpamBriefTable took 0.054296 seconds\n", " Rendering bestEstimateTVDColorBoxPlot took 0.076042 seconds\n", " Rendering bestGatesetGaugeOptParamsTable took 0.001321 seconds\n", " Rendering bestEstimateColorScatterPlot_scl took 0.069485 seconds\n", " Rendering bestGatesetEvalTable took 0.027865 seconds\n", "Output written to example_files/mpi_example_brief directory\n", "Opening example_files/mpi_example_brief/main.html...\n", "*** Report Generation Complete! Total time 155.457s ***\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pickle\n", "results = pickle.load(open(\"example_files/mpi_example_results.pkl\",\"rb\"))\n", "pygsti.report.create_standard_report(results, \"example_files/mpi_example_brief\",\n", " title=\"MPI Example Report\", verbosity=2, auto_open=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 1 }