{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Bempp OpenCL performance benchmarks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is a test notebook to benchmark the Bempp performance on different devices. The default figures reported here\n", "are obtained under Ubuntu 20.04 Linux on an Intel Core i9-9980HK 8 Core CPU with a base clock of 2.4GHz and a maximum turbo\n", "frequency of 5GHz. The GPU device is an NVIDIA Quadro RTX 3000 GPU with 6GB Ram.\n", "\n", "As OpenCL CPU driver we test both POCL (in Version 1.5) and the Intel OpenCL CPU driver, both with default vectorization options.\n", "\n", "We are benchmarking the following operator types\n", "\n", "* Boundary Operators:\n", "\n", " * Laplace single and double layer boundary operator\n", " * Helmholtz single and double layer boundary operator\n", " * Maxwell electric and magnetic field boundary operator\n", " \n", " \n", "* Domain Potential Operators:\n", "\n", " * Laplace single and double layer potential operator\n", " * Helmholtz single and double layer potential operator\n", " * Maxwell electric and magnetic field domain potential operator\n", "\n", "\n", "We are testing all operators in single and double precision. For the GPU we only perform single precision tests as it is significantly\n", "slower in double precision.\n", "\n", "As mesh we use a uniform sphere with 8192 elements. As wavenumber in the Helmholtz and Maxwell tests we use the value $k=1.0$. This has no effect on the performance. As scalar function spaces we use spaces of continuous $P1$ functions. For Maxwell we use RWG functions of order 0." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## General Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section we define the general objects that we need in all benchmarks." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import bempp.api\n", "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "grid = bempp.api.shapes.regular_sphere(5)\n", "p1_space = bempp.api.function_space(grid, \"P\", 1)\n", "rwg_space = bempp.api.function_space(grid, \"RWG\", 0)\n", "snc_space = bempp.api.function_space(grid, \"SNC\", 0)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def benchmark_boundary_operator(operator, precision):\n", " \"\"\"Benchmark an operator with given precision\"\"\"\n", " result = %timeit -o -r 2 -n 2 operator(precision).weak_form()\n", " return result.best\n", "\n", "def benchmark_potential_operator(operator, fun, precision):\n", " \"\"\"Benchmark an operator with given precision\"\"\"\n", " result = %timeit -o -r 2 -n 2 res = operator(precision) @ fun\n", " return result.best" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Boundary Operators" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We first define the boundary operators that we want to test. We make them dependent only on a *precision* argument. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from bempp.api.operators import boundary\n", "\n", "k = 1.0\n", "\n", "operators = [\n", " lambda precision: boundary.laplace.single_layer(p1_space, p1_space, p1_space, precision=precision),\n", " lambda precision: boundary.laplace.double_layer(p1_space, p1_space, p1_space, precision=precision),\n", " lambda precision: boundary.helmholtz.single_layer(p1_space, p1_space, p1_space, k, precision=precision),\n", " lambda precision: boundary.helmholtz.double_layer(p1_space, p1_space, p1_space, k, precision=precision),\n", " lambda precision: boundary.maxwell.electric_field(rwg_space, rwg_space, snc_space, k, precision=precision),\n", " lambda precision: boundary.maxwell.magnetic_field(rwg_space, rwg_space, snc_space, k, precision=precision),\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We setup a Pandas data frame to conveniently store the different timings." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "driver_labels = ['Portable Computing Language', 'Intel(R) OpenCL']\n", "precision_labels = ['single', 'double']\n", "\n", "operator_labels = [\n", " \"laplace single layer bnd\",\n", " \"laplace double layer bnd\",\n", " \"helmholtz single layer bnd\",\n", " \"helmholtz double layer bnd\",\n", " \"maxwell electric bnd\",\n", " \"maxwell magnetic bnd\",\n", "]\n", "df = pd.DataFrame(index=operator_labels, columns=pd.MultiIndex.from_product([driver_labels, precision_labels]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We assemble each operator once to make sure that all Numba functions are compiled." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/betcke/miniconda3/envs/dev/lib/python3.8/site-packages/pyopencl/__init__.py:267: CompilerWarning: Non-empty compiler output encountered. Set the environment variable PYOPENCL_COMPILER_OUTPUT=1 to see more.\n", " warn(\"Non-empty compiler output encountered. Set the \"\n" ] } ], "source": [ "for precision in ['single', 'double']:\n", " if precision == 'single':\n", " bempp.api.VECTORIZATION_MODE = 'vec16'\n", " else:\n", " bempp.api.VECTORIZATION_MODE = 'vec8'\n", " for driver_name in driver_labels:\n", " bempp.api.set_default_cpu_device_by_name(driver_name)\n", " for op in operators:\n", " op(precision).weak_form()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's run the actual benchmarks." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Driver: Portable Computing Language\n", "748 ms ± 23 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "784 ms ± 615 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.8 s ± 36.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.96 s ± 20.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "3.87 s ± 6 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "4.37 s ± 30.5 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Intel(R) OpenCL\n", "619 ms ± 2.25 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "660 ms ± 1.56 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.26 s ± 7.74 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.3 s ± 3.02 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.59 s ± 25.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.95 s ± 47.6 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Portable Computing Language\n", "1.35 s ± 8.83 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.39 s ± 1.76 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "4.75 s ± 2.76 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "4.91 s ± 1.59 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "6.96 s ± 51.1 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "7.7 s ± 15 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Intel(R) OpenCL\n", "1.31 s ± 33.4 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.35 s ± 869 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.19 s ± 15.1 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.36 s ± 31.3 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "5.15 s ± 11.8 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "5.86 s ± 64.3 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n" ] } ], "source": [ "for precision in ['single', 'double']:\n", " if precision == 'single':\n", " bempp.api.VECTORIZATION_MODE = 'vec16'\n", " else:\n", " bempp.api.VECTORIZATION_MODE = 'vec8'\n", " for driver_name in driver_labels:\n", " print(f\"Driver: {driver_name}\")\n", " bempp.api.set_default_cpu_device_by_name(driver_name)\n", " for label, op in zip(operator_labels, operators):\n", " df.loc[label, (driver_name, precision)] = benchmark_boundary_operator(op, precision)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "results_boundary_operators = df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Potential Operators" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are going to evaluate the potentials at 10000 evaluation points. The points are normalised to lie on a sphere with radius .5 As evaluation function we use a simple constant function." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "npoints = 10000\n", "rand = np.random.RandomState(0)\n", "points = rand.randn(3, npoints)\n", "points /= np.linalg.norm(points, axis=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let us define the operators and functions." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from bempp.api.operators import potential\n", "\n", "k = 1.0\n", "\n", "operators = [\n", " lambda precision: potential.laplace.single_layer(p1_space, points, precision=precision),\n", " lambda precision: potential.laplace.double_layer(p1_space, points, precision=precision),\n", " lambda precision: potential.helmholtz.single_layer(p1_space, points, k, precision=precision),\n", " lambda precision: potential.helmholtz.double_layer(p1_space, points, k, precision=precision),\n", " lambda precision: potential.maxwell.electric_field(rwg_space, points, k, precision=precision),\n", " lambda precision: potential.maxwell.magnetic_field(rwg_space, points, k, precision=precision),\n", "]\n", "\n", "functions = [\n", " bempp.api.GridFunction.from_ones(p1_space),\n", " bempp.api.GridFunction.from_ones(p1_space),\n", " bempp.api.GridFunction.from_ones(p1_space),\n", " bempp.api.GridFunction.from_ones(p1_space),\n", " bempp.api.GridFunction.from_ones(rwg_space),\n", " bempp.api.GridFunction.from_ones(rwg_space),\n", "]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We assemble each operator once to compile all functions." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "for precision in ['single', 'double']:\n", " if precision == 'single':\n", " bempp.api.VECTORIZATION_MODE = 'vec16'\n", " else:\n", " bempp.api.VECTORIZATION_MODE = 'vec8'\n", " for driver_name in driver_labels:\n", " bempp.api.set_default_cpu_device_by_name(driver_name)\n", " for op, fun in zip(operators, functions):\n", " res = op(precision) @ fun" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create the data structure to store the results." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "driver_labels = ['Portable Computing Language', 'Intel(R) OpenCL', 'NVIDIA CUDA']\n", "precision_labels = ['single', 'double']\n", "\n", "operator_labels = [\n", " \"laplace single layer pot\",\n", " \"laplace double layer pot\",\n", " \"helmholtz single layer pot\",\n", " \"helmholtz double layer pot\",\n", " \"maxwell electric pot\",\n", " \"maxwell magnetic pot\",\n", "]\n", "df = pd.DataFrame(index=operator_labels, columns=pd.MultiIndex.from_product([driver_labels, precision_labels]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally, we run the actual tests." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Driver: Portable Computing Language\n", "91.9 ms ± 3.32 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "114 ms ± 619 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "435 ms ± 1.26 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "455 ms ± 734 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.03 s ± 961 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "999 ms ± 7.38 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Intel(R) OpenCL\n", "95.2 ms ± 4.22 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "106 ms ± 2.82 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "172 ms ± 1.16 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "198 ms ± 5.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "382 ms ± 5.03 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "351 ms ± 2.34 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: NVIDIA CUDA\n", "25.9 ms ± 4.28 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "22.4 ms ± 255 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "34.9 ms ± 1.67 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "36.2 ms ± 340 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "71.5 ms ± 394 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "71 ms ± 18.2 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Portable Computing Language\n", "149 ms ± 28.2 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "170 ms ± 7.11 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "797 ms ± 916 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "841 ms ± 195 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.3 s ± 7.4 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "2.22 s ± 2.97 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: Intel(R) OpenCL\n", "190 ms ± 2.53 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "218 ms ± 64.4 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "319 ms ± 1.77 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "358 ms ± 541 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "754 ms ± 6.19 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "655 ms ± 1.32 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "Driver: NVIDIA CUDA\n", "449 ms ± 27.5 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "468 ms ± 45.5 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "750 ms ± 67.6 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "818 ms ± 1.53 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.53 s ± 64.7 µs per loop (mean ± std. dev. of 2 runs, 2 loops each)\n", "1.45 s ± 1.42 ms per loop (mean ± std. dev. of 2 runs, 2 loops each)\n" ] } ], "source": [ "bempp.api.set_default_gpu_device_by_name('NVIDIA CUDA')\n", "\n", "for precision in ['single', 'double']:\n", " if precision == 'single':\n", " bempp.api.VECTORIZATION_MODE = 'vec16'\n", " else:\n", " bempp.api.VECTORIZATION_MODE = 'vec8'\n", " for driver_name in driver_labels:\n", " print(f\"Driver: {driver_name}\")\n", " if driver_name == 'NVIDIA CUDA':\n", " bempp.api.POTENTIAL_OPERATOR_DEVICE_TYPE = 'gpu'\n", " bempp.api.VECTORIZATION_MODE = 'novec'\n", " else:\n", " bempp.api.POTENTIAL_OPERATOR_DEVICE_TYPE = 'cpu'\n", " bempp.api.set_default_cpu_device_by_name(driver_name)\n", " for label, op, fun in zip(operator_labels, operators, functions):\n", " df.loc[label, (driver_name, precision)] = benchmark_potential_operator(op, fun, precision)" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "results_potential_operators = df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Output" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Portable Computing Language \\\n", " single double \n", "laplace single layer bnd 0.725336 1.344582 \n", "laplace double layer bnd 0.783209 1.390851 \n", "helmholtz single layer bnd 2.760039 4.743433 \n", "helmholtz double layer bnd 2.943111 4.907221 \n", "maxwell electric bnd 3.861363 6.908861 \n", "maxwell magnetic bnd 4.338312 7.688866 \n", "\n", " Intel(R) OpenCL \n", " single double \n", "laplace single layer bnd 0.617227 1.276647 \n", "laplace double layer bnd 0.658904 1.353705 \n", "helmholtz single layer bnd 1.249008 2.174586 \n", "helmholtz double layer bnd 1.296146 2.32856 \n", "maxwell electric bnd 2.567333 5.136648 \n", "maxwell magnetic bnd 2.901476 5.794071 \n", " Portable Computing Language \\\n", " single double \n", "laplace single layer pot 0.088567 0.121174 \n", "laplace double layer pot 0.113249 0.162764 \n", "helmholtz single layer pot 0.433934 0.795617 \n", "helmholtz double layer pot 0.453831 0.840373 \n", "maxwell electric pot 1.029883 2.297465 \n", "maxwell magnetic pot 0.991923 2.220875 \n", "\n", " Intel(R) OpenCL NVIDIA CUDA \n", " single double single double \n", "laplace single layer pot 0.091016 0.186997 0.02161 0.421118 \n", "laplace double layer pot 0.102928 0.218029 0.022098 0.467704 \n", "helmholtz single layer pot 0.170673 0.316907 0.033239 0.74997 \n", "helmholtz double layer pot 0.192861 0.357478 0.035876 0.816843 \n", "maxwell electric pot 0.37675 0.747511 0.07108 1.533835 \n", "maxwell magnetic pot 0.34823 0.653525 0.07103 1.450303 \n" ] } ], "source": [ "print(results_boundary_operators)\n", "print(results_potential_operators)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }