{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<script async src=\"https://www.googletagmanager.com/gtag/js?id=UA-59152712-8\"></script>\n",
    "<script>\n",
    "  window.dataLayer = window.dataLayer || [];\n",
    "  function gtag(){dataLayer.push(arguments);}\n",
    "  gtag('js', new Date());\n",
    "\n",
    "  gtag('config', 'UA-59152712-8');\n",
    "</script>\n",
    "\n",
    "# BlackHoles@Home Tutorial: Creating a `BOINC` app using the `WrapperApp`\n",
    "\n",
    "## Author: Leo Werneck\n",
    "\n",
    "## This tutorial notebook demonstrates how to write a program that runs in the `BOINC` infrastructure using the `WrapperApp`\n",
    "\n",
    "## <font color=red>**WARNING**:</font> this tutorial notebook is currently incompatible with Windows\n",
    "\n",
    "## Introduction:\n",
    "\n",
    "The [BlackHoles@Home](http://blackholesathome.net/) project allows users to volunteer CPU time so a large number of binary black holes simulations can be performed. The objective is to create a large catalog of [gravitational waveforms](https://en.wikipedia.org/wiki/Gravitational_wave), which can be used by observatories such as [LIGO](https://www.ligo.org), [VIRGO](https://www.virgo-gw.eu), and, in the future, [LISA](https://lisa.nasa.gov) in order to infer what was the source of a detected gravitational wave.\n",
    "\n",
    "BlackHoles@Home is destined to run on the [BOINC](https://boinc.berkeley.edu) infrastructure (alongside [Einstein@Home](https://einsteinathome.org/) and [many other great projects](https://boinc.berkeley.edu/projects.php)), enabling anyone with a computer to contribute to the construction of the largest numerical relativity gravitational wave catalogs ever produced.\n",
    "\n",
    "### Additional Reading Material:\n",
    "\n",
    "* [BOINC's Wiki page](https://boinc.berkeley.edu/trac/wiki)\n",
    "* [BOINC's WrapperApp Wiki page](https://boinc.berkeley.edu/trac/wiki/WrapperApp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='toc'></a>\n",
    "\n",
    "# Table of Contents\n",
    "$$\\label{toc}$$\n",
    "\n",
    "This tutorial explains how to use the `BOINC` wrapper application to run a simple program. The structture of this notebook is as follows:\n",
    "\n",
    "1. [Step 1](#introduction): Introduction\n",
    "1. [Step 2](#compiling_wrapper_app): Compiling the `BOINC` wrapper app for your platform\n",
    "1. [Step 3](#using_wrapper_app): Using the `BOINC` wrapper app\n",
    "    1. [Step 3.a](#the_main_application): The main application\n",
    "    1. [Step 3.b](#compiling_the_main_application): Compiling the main application\n",
    "    1. [Step 3.c](#job_xml): The `job.xml` file\n",
    "        1. [Step 3.c.i](#simple_job_xml): A very simple `job.xml` file\n",
    "        1. [Step 3.c.ii](#job_xml_output_redirect_and_zip): Redirecting and zipping output files\n",
    "1. [Step 4](#latex_pdf_output): Output this notebook to $\\LaTeX$-formatted PDF file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='introduction'></a>\n",
    "\n",
    "# Step 1: Introduction \\[Back to [top](#toc)\\]\n",
    "$$\\label{introduction}$$\n",
    "\n",
    "The [`WrapperApp`](https://boinc.berkeley.edu/trac/wiki/WrapperApp) is the simplest way of converting an existing program into a `BOINC` compatible application. The program that will be actually running is the `WrapperApp` and it will take care of:\n",
    "\n",
    "* Interfacing with the `BOINC` libraries\n",
    "* Running the original program\n",
    "* Handling input/output files\n",
    "\n",
    "Let us assume a simple `BOINC` application, which is made out of only one program, `bhah_test_app`. The directory of this application should then contain the following files:\n",
    "\n",
    "* The application file `bhah_test_app` with the name format `appname_version_platform`.\n",
    "* The `WrapperApp` file with the name format `WrapperAppname_version_platform`.\n",
    "* The `WrapperApp` configuration file, which we will typically call `appname_version_job.xml`.\n",
    "* The appication version file, which is called `version.xml`.\n",
    "\n",
    "We note that the application we will create in this tutorial notebook is analogous to the native `BOINC` application we create in [this tutorial notebook](Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications.ipynb), and thus reading that tutorial notebook is also recommended."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='compiling_wrapper_app'></a>\n",
    "\n",
    "# Step 2: Compiling the `BOINC` wrapper app for your platform \\[Back to [top](#toc)\\]\n",
    "$$\\label{compiling_wrapper_app}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2: Compiling the BOINC wrapper app\n",
    "# Step 2.a: Load needed Python modules\n",
    "import os,sys\n",
    "\n",
    "# Step 2.b: Add NRPy's root directory to the sys.path()\n",
    "sys.path.append(\"..\")\n",
    "\n",
    "# Step 2.c: Load NRPy+'s command line helper module\n",
    "import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface\n",
    "\n",
    "# Step 2.d: Set the path to the BOINC source code\n",
    "path_to_boinc = \"~/bhah/boinc\"\n",
    "current_path  = os.getcwd()\n",
    "\n",
    "# Step 2.e: Check the platform and adjust the compilation command accordingly\n",
    "if sys.platform == \"linux\":\n",
    "    wrapper_compile = \"make\"\n",
    "\n",
    "elif sys.platform == \"darwin\":\n",
    "    wrapper_compile = \"source BuildMacWrapper.sh\"\n",
    "    \n",
    "else:\n",
    "    print(\"Unsupported platform: \"+sys.platform)\n",
    "    sys.exit(1)\n",
    "\n",
    "# Step 2.f: Compile the wrapper app\n",
    "!cd $path_to_boinc/samples/wrapper && $wrapper_compile\n",
    "\n",
    "# Step 2.g: Copy the wrapper app to the current working directory\n",
    "!cp $path_to_boinc/samples/wrapper/wrapper $current_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='using_wrapper_app'></a>\n",
    "\n",
    "# Step 3: Using the `BOINC` wrapper app \\[Back to [top](#toc)\\]\n",
    "$$\\label{using_wrapper_app}$$\n",
    "\n",
    "Once we have everything set up, using the wrapper app is as simple as running\n",
    "\n",
    "```bash\n",
    "$: ./wrapper\n",
    "```\n",
    "\n",
    "The following steps will describe how to set the configuration files so that the wrapper app works as you expect it to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='the_main_application'></a>\n",
    "\n",
    "## Step 3.a: The main application \\[Back to [top](#toc)\\]\n",
    "$$\\label{the_main_application}$$\n",
    "\n",
    "Let us start by writing a simple application which we will run using the `BOINC` wrapper app. In order for us to be able to see some additional configuration features of the wrapper app, we will make our main application slightly more complicated than a simple \"Hello World!\" program.\n",
    "\n",
    "This application takes any number of command line arguments and then prints them to `stdout`, `stderr`, and an output text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile simple_app.c\n",
    "// Step 0: Load all the necessary C header files\n",
    "#include <stdio.h>\n",
    "#include <stdlib.h>\n",
    "\n",
    "// Program description: this program is just a slightly\n",
    "//                      more complicated version of the\n",
    "//                      \"Hello World!\" program, where\n",
    "//                      we will be taking some command\n",
    "//                      line inputs and printing them to\n",
    "//                      stdout, stderr, and an output file.\n",
    "int main( int argc, char** argv ) {\n",
    "\n",
    "  // Step 1: Check correct usage\n",
    "  if( argc == 1 ) {\n",
    "    fprintf(stderr,\"(ERROR) Correct usage is ./simple_app <command_line_arguments>\\n\");\n",
    "    exit(1);\n",
    "  }\n",
    "\n",
    "  // Step 2: Print all command line arguments to\n",
    "  //         stdout, stderr, and an output file\n",
    "  //\n",
    "  // Step 2.a: Open the output file\n",
    "  // Step 2.a.i: Set the output file name\n",
    "  char filename[100] = \"output_file.txt\";\n",
    "  // Step 2.a.ii: Open the file\n",
    "  FILE* filept = fopen(filename,\"w\");\n",
    "  // Step 2.a.iii: Check everything is OK\n",
    "  if( !filept ) {\n",
    "    fprintf(stderr,\"(ERROR) Could not open file %s\\n.\",filename);\n",
    "    exit(1);\n",
    "  }\n",
    "\n",
    "  // Step 2.b: Print an information line\n",
    "  fprintf(stdout,\"(INFO) Got the following command line arguments:\");\n",
    "  fprintf(stderr,\"(INFO) Got the following command line arguments:\");\n",
    "  fprintf(filept,\"(INFO) Got the following command line arguments:\");\n",
    "  \n",
    "  // Step 2.c: Loop over the command line arguments, printing\n",
    "  //           them to stdout, stderr, and our output file\n",
    "  for(int i=1;i<argc;i++) {\n",
    "    fprintf(stdout,\" %s\",argv[i]);\n",
    "    fprintf(stderr,\" %s\",argv[i]);\n",
    "    fprintf(filept,\" %s\",argv[i]);\n",
    "  }\n",
    "\n",
    "  // Step 2.d: Add a line break to the output\n",
    "  fprintf(stdout,\"\\n\");\n",
    "  fprintf(stderr,\"\\n\");\n",
    "  fprintf(filept,\"\\n\");\n",
    "\n",
    "  // Step 2.d: Close the output file\n",
    "  fclose(filept);\n",
    "\n",
    "  // All done!\n",
    "  return 0;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='compiling_the_main_application'></a>\n",
    "\n",
    "## Step 3.b: Compiling the main application \\[Back to [top](#toc)\\]\n",
    "$$\\label{compiling_the_main_application}$$\n",
    "\n",
    "We now compile the main application using NRPy+'s `cmdline_helper` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cmd.C_compile(\"simple_app.c\",\"simple_app\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='job_xml'></a>\n",
    "\n",
    "## Step 3.c: The `job.xml` file \\[Back to [top](#toc)\\]\n",
    "$$\\label{job_xml}$$\n",
    "\n",
    "Let's see what happens if we try running the wrapper app:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -f job.xml\n",
    "cmd.Execute(\"wrapper\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen above, the `BOINC` wrapper application requests an input file, `job.xml`, to be present in the current working directory. We will now set up a `job.xml` file for the wrapper app in a way that it works correctly with our `simple_app`. A `job.xml` has the following syntax:\n",
    "\n",
    "```xml\n",
    "<job_desc>\n",
    "    <task>\n",
    "        ...task_options...\n",
    "    </task>\n",
    "    ...additional_options...\n",
    "</job_desc>\n",
    "```\n",
    "\n",
    "All the configurations for the wrapper application are enclosed by the `job_desc` environment. To configure the wrapper to work with our specific application, we provide the `task_options`, while `additional_options` can be provided for additional configuration, as we will see."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='simple_job_xml'></a>\n",
    "\n",
    "### Step 3.c.i: A very simple `job.xml` file \\[Back to [top](#toc)\\]\n",
    "$$\\label{simple_job_xml}$$\n",
    "\n",
    "First, let us start with a very basic configuration: let us ask the wrapper to run our simple application using the command line arguments `1 2 3 4 testing hello world 4 3 2 1`. This is achieved with the following `job.xml` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile job.xml\n",
    "<job_desc>\n",
    "    <task>\n",
    "      <application>simple_app</application>\n",
    "      <command_line>1 2 3 4 testing hello world 4 3 2 1</command_line>\n",
    "    </task>\n",
    "</job_desc>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now copy everything into a new, fresh directory and run our wrapper application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf wrapper_app_test\n",
    "cmd.mkdir(\"wrapper_app_test\")\n",
    "!cp wrapper simple_app job.xml wrapper_app_test && cd wrapper_app_test && ./wrapper && ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that after execution, we see the output `(INFO) Got the following command line arguments: 1 2 3 4 testing hello world 4 3 2 1` printed to `stdout`. If we examine the file `output_file.txt`, we will see the same output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat output_file.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `stderr.txt` file is automatically generated by the `BOINC` wrapper app, and it contains all the output which was sent to `stderr`. We see that while we also have the expected output in it, there is also some additional information which was generated by the wrapper app:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat stderr.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aditionally, we see that the wrapper application has created two additional files: the `boinc_finish_called` and the `wrapper_checkpoint.txt`. For our purposes, the `wrapper_checkpoint.txt` file is irrelevant, so we will ignore it for now. The `boinc_finish_called` file contains the numerical value returned by our program, `simple_app`. As is usual in `C`, if the return value is `0`, then the execution was successful, while a non-zero value indicates an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat boinc_finish_called"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='job_xml_output_redirect_and_zip'></a>\n",
    "\n",
    "### Step 3.c.ii: Redirecting and zipping output files  \\[Back to [top](#toc)\\]\n",
    "$$\\label{job_xml_output_redirect_and_zip}$$\n",
    "\n",
    "Now that we have seen the simplest possible case, let us look at something slightly more complicated. The following `job.xml` file asks the wrapper app to perform the following tasks:\n",
    "\n",
    "1. Run the `simple_app` application with command line arguments `1 2 3 4 testing hello world 4 3 2 1`\n",
    "1. Redirect all `stdout` output to the file `simple_app.out`\n",
    "1. Redirect all `stderr` output to the file `simple_app.err`\n",
    "1. Zip all the output files we have seen before into a single file: `output.zip`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile job.xml\n",
    "<job_desc>\n",
    "    <task>\n",
    "      <application>simple_app</application>\n",
    "      <command_line>1 2 3 4 testing hello world 4 3 2 1</command_line>\n",
    "      <stdout_filename>simple_app.out</stdout_filename>\n",
    "      <stderr_filename>simple_app.err</stderr_filename>\n",
    "    </task>\n",
    "    <zip_output>\n",
    "      <zipfilename>output.zip</zipfilename>\n",
    "      <filename>simple_app.out</filename>\n",
    "      <filename>simple_app.err</filename>\n",
    "      <filename>output_file.txt</filename>\n",
    "      <filename>boinc_finish_called</filename>\n",
    "      <filename>wrapper_checkpoint.txt</filename>\n",
    "    </zip_output>\n",
    "</job_desc>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us see what happens when we run the wrapper app:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf wrapper_app_test\n",
    "cmd.mkdir(\"wrapper_app_test\")\n",
    "!cp wrapper simple_app job.xml wrapper_app_test && cd wrapper_app_test && ./wrapper && ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that we now have the output files `simple_app.out` and `simple_app.err`, as expected. The file `stderr.txt` is still present, by default. We also have all our output files neatly collected into a single zip file, `output.zip`. Note that zipping the output is not done with the goal of reducing the overall size of the output, but because it easier to communicate the output files back to the `BOINC` server."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='latex_pdf_output'></a>\n",
    "\n",
    "# Step 4: Output this notebook to $\\LaTeX$-formatted PDF file \\[Back to [top](#toc)\\]\n",
    "$$\\label{latex_pdf_output}$$\n",
    "\n",
    "The following code cell converts this Jupyter notebook into a proper, clickable $\\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename\n",
    "[Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp.pdf](Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-03-07T17:21:21.309338Z",
     "iopub.status.busy": "2021-03-07T17:21:21.308582Z",
     "iopub.status.idle": "2021-03-07T17:21:24.791779Z",
     "shell.execute_reply": "2021-03-07T17:21:24.792580Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cp ../latex_nrpy_style.tplx .\n",
    "cmd.output_Jupyter_notebook_to_LaTeXed_PDF(\"Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp\")\n",
    "!rm -f latex_nrpy_style.tplx"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}