{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<script async src=\"https://www.googletagmanager.com/gtag/js?id=UA-59152712-8\"></script>\n",
"<script>\n",
" window.dataLayer = window.dataLayer || [];\n",
" function gtag(){dataLayer.push(arguments);}\n",
" gtag('js', new Date());\n",
"\n",
" gtag('config', 'UA-59152712-8');\n",
"</script>\n",
"\n",
"# BlackHoles@Home Tutorial: Creating `BOINC` native applications\n",
"\n",
"## Author: Leo Werneck\n",
"\n",
"## This tutorial notebook demonstrates how to write native programs for the `BOINC` infrastructure, as well as how to convert `NRPy+` code into a `BOINC` application\n",
"\n",
"## <font color=red>**WARNING**:</font> this tutorial notebook is currently incompatible with Windows\n",
"\n",
"## Introduction:\n",
"\n",
"The [BlackHoles@Home](http://blackholesathome.net/) project allows users to volunteer CPU time so a large number of binary black holes simulations can be performed. The objective is to create a large catalog of [gravitational waveforms](https://en.wikipedia.org/wiki/Gravitational_wave), which can be used by observatories such as [LIGO](https://www.ligo.org), [VIRGO](https://www.virgo-gw.eu), and, in the future, [LISA](https://lisa.nasa.gov) in order to infer what was the source of a detected gravitational wave.\n",
"\n",
"BlackHoles@Home is destined to run on the [BOINC](https://boinc.berkeley.edu) infrastructure (alongside [Einstein@Home](https://einsteinathome.org/) and [many other great projects](https://boinc.berkeley.edu/projects.php)), enabling anyone with a computer to contribute to the construction of the largest numerical relativity gravitational wave catalogs ever produced.\n",
"\n",
"### Additional Reading Material:\n",
"\n",
"* [BOINC's Wiki page](https://boinc.berkeley.edu/trac/wiki)\n",
"* [BOINC's Basic API Wiki page](https://boinc.berkeley.edu/trac/wiki/BasicApi)\n",
"* [Tutorial notebook on how to compile the `BOINC` libraries](Tutorial-BlackHolesAtHome-Compiling_the_BOINC_libraries.ipynb)\n",
"* [Tutorial notebook on creating a `BOINC` application using the `BOINC` WrapperApp](Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='toc'></a>\n",
"\n",
"# Table of Contents\n",
"$$\\label{toc}$$\n",
"\n",
"This tutorial explains how to use the `BOINC` wrapper application to run a simple program. The structture of this notebook is as follows:\n",
"\n",
"1. [Step 1](#introduction): Introduction\n",
"1. [Step 2](#loading_python_nrpy_modules): Loading needed Python/NRPy+ modules\n",
"1. [Step 3](#creating_native_boinc_app): Creating a `BOINC` native application\n",
" 1. [Step 3.a](#simplest_boinc_app): A very simple `BOINC` native application\n",
" 1. [Step 3.b](#nrpy_to_boinc): Converting any `NRPy+` code into a `BOINC` native app\n",
"1. [Step 4](#latex_pdf_output): Output this notebook to $\\LaTeX$-formatted PDF file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='introduction'></a>\n",
"\n",
"# Step 1: Introduction \\[Back to [top](#toc)\\]\n",
"$$\\label{introduction}$$\n",
"\n",
"A native `BOINC` application is a program which directly interfaces with the `BOINC` API. During compilation, we link the executable with the `BOINC` libraries, thus creating an executable which can run in the `BOINC` infrastructure. If you have not yet compiled the `BOINC` libraries, please read the [tutorial notebook on how to do so](Tutorial-BlackHolesAtHome-Compiling_the_BOINC_libraries.ipynb).\n",
"\n",
"This tutorial notebook aims at teaching you two key concepts:\n",
"\n",
"1. How to write simple `BOINC` applications by hand\n",
"1. How to convert `NRPy+` code into a `BOINC` application\n",
"\n",
"We will be using the `NRPy+` code generated by the [Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb) NRPy+ tutorial notebook as an example."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='loading_python_nrpy_modules'></a>\n",
"\n",
"# Step 2: Loading needed Python/NRPy+ modules \\[Back to [top](#toc)\\]\n",
"$$\\label{loading_python_nrpy_modules}$$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Step 2: Load Python/NRPy+ modules and perform basic setup\n",
"# Step 2.a: Load needed Python modules\n",
"import os,sys\n",
"\n",
"# Step 2.b: Add NRPy's root directory to the sys.path()\n",
"sys.path.append(\"..\")\n",
"\n",
"# Step 2.c: Load NRPy+'s command line helper module\n",
"import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface\n",
"\n",
"# Step 2.d: Set the path to the BOINC source code\n",
"path_to_boinc = \"/Users/werneck/bhah/boinc\"\n",
"boinc_api_dir = os.path.join(path_to_boinc,\"api\")\n",
"boinc_lib_dir = os.path.join(path_to_boinc,\"lib\")\n",
"boinc_zip_dir = os.path.join(path_to_boinc,\"zip\")\n",
"current_path = os.getcwd()\n",
"\n",
"# Step 2.e: Adjust the compiler and compilation flags based on the system\n",
"# Step 2.e.i: Set the C++ compiler flags\n",
"global CXX_compiler,CXXFLAGS,LDFLAGS\n",
"CXXFLAGS = \"-fopenmp -march=native -Ofast -funroll-loops \"\n",
"CXXFLAGS += \"-I%s -I%s -I%s \"%(boinc_api_dir,boinc_lib_dir,boinc_zip_dir)\n",
"LDFLAGS = \"-L%s -L%s -L%s -lboinc_api -lboinc -lboinc_zip \"%(boinc_api_dir,boinc_lib_dir,boinc_zip_dir)\n",
"\n",
"# Step 2.e.ii: Set the C++ compiler\n",
"if sys.platform == 'linux':\n",
" CXX_compiler = \"g++ \"\n",
"elif sys.platform == 'darwin':\n",
" # Set path to Clang compiler installed with homebrew\n",
" path_to_llvm = \"/usr/local/opt/llvm/\"\n",
" path_to_clangpp = os.path.join(path_to_llvm,\"bin\",\"clang++\")\n",
" path_to_clang_include = os.path.join(path_to_llvm,\"include\")\n",
" path_to_clang_library = os.path.join(path_to_llvm,\"lib\")\n",
" CXX_compiler = path_to_clangpp+\" \"\n",
" CXXFLAGS += \"-I%s \"%(path_to_clang_include)\n",
" LDFLAGS += \"-L%s \"%(path_to_clang_library)\n",
"else:\n",
" print(\"Error: platform %s is currently not supported.\"%sys.platform)\n",
" sys.exit(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='creating_native_boinc_app'></a>\n",
"\n",
"# Step 3: Creating a `BOINC` native application \\[Back to [top](#toc)\\]\n",
"$$\\label{creating_native_boinc_app}$$\n",
"\n",
"A native `BOINC` application can be created by:\n",
"\n",
"1. Including the `BOINC` api header file by adding `#include \"boinc_api.h\"` to your code\n",
"1. Calling the `boinc_init()` function at the beginning of the main function\n",
"1. Using `boinc_finish(0)` instead of `return 0` at the end of the main function\n",
"\n",
"The `boinc_finish(err_code)` function should also be used instead of the `exit(err_code)` function in case you needed to program to stop running return an error code."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='simplest_boinc_app'></a>\n",
"\n",
"## Step 3.a: A very simple `BOINC` native application \\[Back to [top](#toc)\\]\n",
"$$\\label{simplest_boinc_app}$$\n",
"\n",
"We now provide one of the simplest possible examples of a `BOINC` application, with minimal error handling included. This application:\n",
"\n",
"1. Initializes the `BOINC` environment\n",
"1. Checks that the `BOINC` environment was initialized correctly\n",
"1. Prints a message to the user\n",
"1. Finalizes the `BOINC` environment and terminates"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%writefile simplest_boinc_app.cpp\n",
"// Step 0: Basic includes\n",
"// Step 0.a: Basic C++ header files\n",
"#include <iostream>\n",
"// Step 0.b: BOINC api header file\n",
"#include \"boinc_api.h\"\n",
"\n",
"// Program description: this is one of the simplest BOINC\n",
"// applications that can be written.\n",
"// We start the BOINC environment\n",
"// by calling the boinc_init() function,\n",
"// check everything is OK (erroring out\n",
"// if it isn't), print a message to the\n",
"// user, and terminate using a call to\n",
"// the boinc_finish() function.\n",
"int main() {\n",
"\n",
" // Step 1: Initialize the BOINC environment with boinc_init()\n",
" int status = boinc_init();\n",
"\n",
" // Step 2: Check everything is OK, error out if not\n",
" if( status != 0 ) {\n",
" fprintf(stderr,\"ERROR: boinc_init() returned a non-zero value: %d\\n\",status);\n",
" boinc_finish(status);\n",
" }\n",
"\n",
" // Step 3: Print a message to the user\n",
" printf(\"Hello BOINC!\\n\");\n",
"\n",
" // Step 4: Terminate the program with boinc_finish()\n",
" boinc_finish(0);\n",
"\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let us now compile and run the application:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"compile_string = CXX_compiler+CXXFLAGS+\"simplest_boinc_app.cpp -o simplest_boinc_app \"+LDFLAGS\n",
"!rm -rf simplest_boinc_app_test_dir\n",
"cmd.mkdir(\"simplest_boinc_app_test_dir\")\n",
"!mv simplest_boinc_app.cpp simplest_boinc_app_test_dir\n",
"!cd simplest_boinc_app_test_dir && $compile_string && ./simplest_boinc_app && ls"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that [just like when using the `BOINC` WrapperApp](Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp.ipynb), we have produced the output files `boinc_finish_called` and `stderr.txt`, even though we did not explicitly generate them in our program. This is because the `BOINC` api generates these files automatically for us. If we take a look at the contents of the files, we see that the `boinc_finish_called` simply contains the integer argument of the `boinc_finish()` function, while the `stderr.txt` contains some basic information stating that we are running the application outside of the `BOINC` infrastructure and that the `boinc_finish()` function was called:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!cd simplest_boinc_app_test_dir && cat boinc_finish_called stderr.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='nrpy_to_boinc'></a>\n",
"\n",
"## Step 3.b: Converting any `NRPy+` code into a `BOINC` native app \\[Back to [top](#toc)\\]\n",
"$$\\label{nrpy_to_boinc}$$\n",
"\n",
"We now provide a script for converting an existing `NRPy+` code into a `BOINC` application. Note that it is relatively easy to convert an existing `C` or `C++` application into a native `BOINC` application. Unless you want to manually create a wrapper function that calls your `C` code, it is recommended to compile your code using a `C++` compiler instead. In the case of `NRPy+` applications, this can be achieved by simply adding:\n",
"\n",
"```cpp\n",
"#ifdef __cplusplus\n",
"# define restrict __restrict__\n",
"#endif\n",
"```\n",
"\n",
"to the very top of the main application source code file, changing the file extension from `.c` to `.cpp`/`.cc`/`.C`, and then compiling the code using the flag `-std=c++11`. We also need to replace all calls to the `exit()` function with calls to the `boinc_finish()` function.\n",
"\n",
"The following script takes care of that:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Converting NRPy+ code into a BOINC app\n",
"# Description: This function reads a NRPy+ source code\n",
"# one line at a time and copies them into\n",
"# a new file which is compatible with the\n",
"# BOINC infrastructure.\n",
"def NRPy_to_BOINC(input_file,output_file):\n",
" # Step 1: Open the NRPy+ input file\n",
" with open(input_file,\"r\") as file:\n",
"\n",
" # Step 2: Create the BOINC application\n",
" # Step 2.a: Print a message to the user describing\n",
" # some basic changes. Add the \"restrict\"\n",
" # keyword so that it is compatible with\n",
" # C++, which is required by BOINC.\n",
" output_string = \"\"\"\n",
"//****************************************************************\n",
"// This NRPy+ code has been converted to work with the\n",
"// BOINC infrastructure. Please compile it with a C++\n",
"// compiler. Don't forget to add the -std=c++11 flag.\n",
"#ifdef __cplusplus\n",
"# define restrict __restrict__\n",
"#endif\n",
"\n",
"// .--------------------.\n",
"// | BOINC HEADER FILES |\n",
"// .--------------------.\n",
"// Note: You can comment out (or remove) the boinc_zip.h header\n",
"// if you do not plan on using the BOINC zip functions.\n",
"#include \\\"boinc_api.h\\\"\n",
"#include \\\"boinc_zip.h\\\"\n",
"//****************************************************************\n",
"\n",
"\"\"\"\n",
" # Step 2.b: Loop over the file, adding calls to\n",
" # the BOINC API functions as needed.\n",
" indent = \" \"\n",
" for line in file:\n",
" # Step 2.b.i: After the main() function, add a call to the boinc_init() function\n",
" if \"int main\" in line:\n",
" output_string += \"\\n\"+line+\"\\n\"+indent+\"boinc_init();\\n\"\n",
" # Step 2.b.ii: Replace return 0; with boinc_finish(0);\n",
" elif \"return 0\" in line:\n",
" output_string += indent+\"boinc_finish(0);\\n\"\n",
" # Step 2.b.iii: Replace exit(err_code) function calls with boinc_finish(err_code)\n",
" elif \"exit(\" in line:\n",
" output_string += line.replace(\"exit\",\"boinc_finish\")\n",
" else:\n",
" # Step 2.b.iv: Otherwise, just copy the original source code\n",
" output_string += line\n",
"\n",
" # Step 3: Write the output file\n",
" with open(output_file,\"w\") as file:\n",
" file.write(output_string)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's convert a `NRPy+` generated code into a `BOINC` code, compile it, and run it. We will take as an example the files obtained after running the [Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb). Running the cell below will perform the following tasks:\n",
"\n",
"1. Run the [Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb) NRPy+ tutorial notebook.\n",
"1. Move the folder containing the source files into our current working directory (`nrpytutorial/BHAH`)\n",
"1. Convert the main program, which is defined in the `BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.c` file, into a `BOINC` compatible application\n",
"1. Compile the source code, linking to the `BOINC` libraries\n",
"1. Execute the code\n",
"\n",
"*WARNING*: because this step involves generating the source code for the BSSN equations, running the cell below will take a few minutes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Run the Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb tutorial notebook\n",
"!pip install runipy > /dev/null\n",
"!rm -rf BSSN_Two_BHs_Collide_Ccodes out96*.txt out96*.png\n",
"!cd .. && runipy Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb && mv BSSN_Two_BHs_Collide_Ccodes BHAH\n",
"\n",
"# Compute\n",
"NRPy_to_BOINC(\"BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.c\",\"BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.cpp\")\n",
"compile_string = CXX_compiler+CXXFLAGS+\"BrillLindquist_Playground.cpp -o ../BrillLindquist_Playground \"+LDFLAGS\n",
"!cd BSSN_Two_BHs_Collide_Ccodes/ && $compile_string\n",
"!./BrillLindquist_Playground 96 16 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now visualize the solution, just like with the regular NRPy+ code (the cell below contains code that was extracted from the [Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb](../Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb) NRPy+ tutorial notebook):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"## VISUALIZATION ANIMATION, PART 1: Generate PNGs, one per frame of movie ##\n",
"\n",
"import numpy as np\n",
"from scipy.interpolate import griddata\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib.pyplot import savefig\n",
"from IPython.display import HTML\n",
"import matplotlib.image as mgimg\n",
"\n",
"import glob\n",
"import sys\n",
"from matplotlib import animation\n",
"\n",
"outdir = \"./\"\n",
"globby = glob.glob(os.path.join(outdir,'out96-00*.txt'))\n",
"file_list = []\n",
"for x in sorted(globby):\n",
" file_list.append(x)\n",
"\n",
"bound=1.4\n",
"pl_xmin = -bound\n",
"pl_xmax = +bound\n",
"pl_ymin = -bound\n",
"pl_ymax = +bound\n",
"\n",
"for filename in file_list:\n",
" fig = plt.figure()\n",
" x,y,cf,Ham = np.loadtxt(filename).T #Transposed for easier unpacking\n",
"\n",
" plotquantity = cf\n",
" plotdescription = \"Numerical Soln.\"\n",
" plt.title(\"Black Hole Head-on Collision (conf factor)\")\n",
" plt.xlabel(\"y/M\")\n",
" plt.ylabel(\"z/M\")\n",
"\n",
" grid_x, grid_y = np.mgrid[pl_xmin:pl_xmax:300j, pl_ymin:pl_ymax:300j]\n",
" points = np.zeros((len(x), 2))\n",
" for i in range(len(x)):\n",
" # Zach says: No idea why x and y get flipped...\n",
" points[i][0] = y[i]\n",
" points[i][1] = x[i]\n",
"\n",
" grid = griddata(points, plotquantity, (grid_x, grid_y), method='nearest')\n",
" gridcub = griddata(points, plotquantity, (grid_x, grid_y), method='cubic')\n",
" im = plt.imshow(gridcub, extent=(pl_xmin,pl_xmax, pl_ymin,pl_ymax))\n",
" ax = plt.colorbar()\n",
" ax.set_label(plotdescription)\n",
" savefig(os.path.join(filename+\".png\"),dpi=150)\n",
" plt.close(fig)\n",
" sys.stdout.write(\"%c[2K\" % 27)\n",
" sys.stdout.write(\"Processing file \"+filename+\"\\r\")\n",
" sys.stdout.flush()\n",
"\n",
"## VISUALIZATION ANIMATION, PART 2: Combine PNGs to generate movie ##\n",
"\n",
"# https://stackoverflow.com/questions/14908576/how-to-remove-frame-from-matplotlib-pyplot-figure-vs-matplotlib-figure-frame\n",
"# https://stackoverflow.com/questions/23176161/animating-pngs-in-matplotlib-using-artistanimation\n",
"\n",
"fig = plt.figure(frameon=False)\n",
"ax = fig.add_axes([0, 0, 1, 1])\n",
"ax.axis('off')\n",
"\n",
"myimages = []\n",
"\n",
"for i in range(len(file_list)):\n",
" img = mgimg.imread(file_list[i]+\".png\")\n",
" imgplot = plt.imshow(img)\n",
" myimages.append([imgplot])\n",
"\n",
"ani = animation.ArtistAnimation(fig, myimages, interval=100, repeat_delay=1000)\n",
"ani.save(os.path.join(outdir,'BH_Head-on_Collision.mp4'), fps=5,dpi=150)\n",
"plt.close()\n",
"\n",
"# Embed video based on suggestion:\n",
"# https://stackoverflow.com/questions/39900173/jupyter-notebook-html-cell-magic-with-python-variable\n",
"HTML(\"\"\"\n",
"<video width=\"480\" height=\"360\" controls>\n",
" <source src=\\\"\"\"\"+os.path.join(outdir,\"BH_Head-on_Collision.mp4\")+\"\"\"\\\" type=\"video/mp4\">\n",
"</video>\n",
"\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='latex_pdf_output'></a>\n",
"\n",
"# Step 4: Output this notebook to $\\LaTeX$-formatted PDF file \\[Back to [top](#toc)\\]\n",
"$$\\label{latex_pdf_output}$$\n",
"\n",
"The following code cell converts this Jupyter notebook into a proper, clickable $\\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename\n",
"[Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications.pdf](Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-03-07T17:21:21.309338Z",
"iopub.status.busy": "2021-03-07T17:21:21.308582Z",
"iopub.status.idle": "2021-03-07T17:21:24.791779Z",
"shell.execute_reply": "2021-03-07T17:21:24.792580Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"!cp ../latex_nrpy_style.tplx .\n",
"cmd.output_Jupyter_notebook_to_LaTeXed_PDF(\"Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications\")\n",
"!rm -f latex_nrpy_style.tplx"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}