{ "cells": [ { "cell_type": "markdown", "id": "e62fd042", "metadata": {}, "source": [ "
\n", " \n", " April 15$^\\mathrm{th}$, 2025\n", " \n", "
\n", " \n", " FLOWVPM Setup\n", " \n", "
\n", " \n", " Linux Instructions\n", " \n", "
" ] }, { "cell_type": "markdown", "id": "f1f43f2e", "metadata": {}, "source": [ "Tested on Ubuntu 22.04.4 LTS with Julia v1.10.2" ] }, { "cell_type": "markdown", "id": "fa59dea7", "metadata": { "heading_collapsed": true }, "source": [ "# Compiling ExaFMM: Automatic" ] }, { "cell_type": "markdown", "id": "20e57a9d", "metadata": { "hidden": true }, "source": [ "Here we show how to automatically compile ExaFMM with the [`build.sh`](https://github.com/byuflowlab/FLOWExaFMM.jl/blob/master/build.sh) script." ] }, { "cell_type": "markdown", "id": "90c301e3", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 1) Cloning FLOWExaFMM" ] }, { "cell_type": "markdown", "id": "37ba7d17", "metadata": { "hidden": true }, "source": [ "Since we will need to compile the c++ part of ExaFMM, first clone FLOWExaFMM somewhere in your machine:\n", "\n", "```bash\n", "git clone https://github.com/byuflowlab/FLOWExaFMM\n", "```" ] }, { "cell_type": "markdown", "id": "945c1643", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 2) Compiling ExaFMM" ] }, { "cell_type": "markdown", "id": "af3d2df3", "metadata": { "hidden": true }, "source": [ "Run the script [`build.sh`](https://github.com/byuflowlab/FLOWExaFMM.jl/blob/master/build.sh) that is under FLOWExaFMM:\n", "```bash\n", "cd path/to/your/FLOWExaFMM\n", "sh build.sh\n", "```\n", "\n", "This should have generated the file `fmm.so` under `src/`, which is a binary library containing ExaFMM." ] }, { "cell_type": "markdown", "id": "968ad33f", "metadata": { "heading_collapsed": true }, "source": [ "# Compiling ExaFMM: Manual" ] }, { "cell_type": "markdown", "id": "ffb7dd5a", "metadata": { "hidden": true }, "source": [ "If `build.sh` fails to automatically compile ExaFMM, the following steps will help you debug the source of the error and compile the code manually.\n", "\n", "These instruction where tested on Ubuntu 22.04 LTS with Julia v1.8.5 on a Dell 7760 laptop." ] }, { "cell_type": "markdown", "id": "9a7c3da1", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 1) Setting `CxxWrap` up" ] }, { "cell_type": "markdown", "id": "c97a1892", "metadata": { "hidden": true }, "source": [ "First, we will test that CxxWrap runs correctly in your machine. This package is a Julia wrapper for c++ code. \n", "\n", "Start by adding CxxWrap to Julia:\n", "\n", "```julia\n", "julia> ] add CxxWrap\n", "```\n", "\n", "You might get an error complaining that you don't have CMake installed on your system. If so go ahead and get that set up. In a Linux machine it's done with\n", "\n", "```bash\n", "sudo apt-get install cmake\n", "```\n", "\n", "After installing CMake, make sure your `CxxWrap` package is getting built:\n", "\n", "```julia\n", "julia> ] test CxxWrap\n", "```" ] }, { "cell_type": "markdown", "id": "04e42eb3", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 2) Testing `CxxWrap`" ] }, { "cell_type": "markdown", "id": "74f43573", "metadata": { "hidden": true }, "source": [ "Test that `CxxWrap` is working properly as follows. \n", "\n", "First, create a file named `hello.cpp` with the following C++ code:\n", "\n", "```cxx\n", "#include \n", "#include \"jlcxx/jlcxx.hpp\"\n", "\n", "// Test function\n", "std::string greet()\n", "{\n", " return \"hello, world\";\n", "}\n", "\n", "// Exposing the function to Julia\n", "JLCXX_MODULE define_julia_module(jlcxx::Module& mod)\n", "{\n", " mod.method(\"greet\", &greet);\n", "}\n", "```" ] }, { "cell_type": "markdown", "id": "8283a561", "metadata": { "hidden": true }, "source": [ "In order to compile the code, we need to point the compiler to wherever the CxxWrap include files are. Most likely, they are under the path returned by \n", "```julia\n", "import CxxWrap\n", "CxxWrap.prefix_path()\n", "```\n", "\n", "In my case, this is what I get:" ] }, { "cell_type": "code", "execution_count": 1, "id": "9e4ae21d", "metadata": { "hidden": true }, "outputs": [ { "data": { "text/plain": [ "\"/home/edoalvar/.julia/artifacts/5209ca23f516fb3391b885eef717e49b4ee0a268\"" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import CxxWrap\n", "CxxWrap.prefix_path()" ] }, { "cell_type": "markdown", "id": "46f52986", "metadata": { "hidden": true }, "source": [ "You will also have to find out where the Julia include files are. This can be done with the following command:" ] }, { "cell_type": "code", "execution_count": 2, "id": "cf286db0", "metadata": { "hidden": true }, "outputs": [ { "data": { "text/plain": [ "\"/home/edoalvar/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/include/julia\"" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "abspath(Sys.BINDIR, Base.INCLUDEDIR, \"julia\")" ] }, { "cell_type": "markdown", "id": "a24c91b5", "metadata": { "hidden": true }, "source": [ "Then, we generate the C++ object (called `hello.cpp.o`) with the following command:\n", "\n", "```bash\n", "JLCXX_H=/home/edoalvar/.julia/artifacts/5209ca23f516fb3391b885eef717e49b4ee0a268/include\n", "JULIA_H=/home/edoalvar/.julia/juliaup/julia-1.10.2+0.x64.linux.gnu/include/julia\n", "\n", "# Compile object hello.cpp.o\n", "c++ -DJULIA_ENABLE_THREADING -Dhello_EXPORTS -I$JLCXX_H -I$JULIA_H \\\n", "-march=native -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -fPIC \\\n", "-o hello.cpp.o -c hello.cpp\n", "\n", "```" ] }, { "cell_type": "markdown", "id": "9a3b1e9e", "metadata": { "hidden": true }, "source": [ "> **NOTE:** Make sure you are using an updated version of `gcc` (`gcc --version` must show 7.3 or newer).\n", "\n", "> **NOTE 2:** `JLCXX_H` and `JULIA_H` can be automatically defined in the command line as follows\n", "> ```bash\n", "> JLCXX_H=$(julia --print \"import CxxWrap; CxxWrap.prefix_path()\")\n", "JLCXX_H=${JLCXX_H%\\\"}; JLCXX_H=${JLCXX_H#\\\"}; JLCXX_H=$JLCXX_H/include\n", "> \n", "> JULIA_H=$(julia --print \"abspath(Sys.BINDIR, Base.INCLUDEDIR)\")\n", "JULIA_H=${JULIA_H%\\\"}; JULIA_H=${JULIA_H#\\\"}; JULIA_H=$JULIA_H/julia\n", "> ```" ] }, { "cell_type": "markdown", "id": "257f3889", "metadata": { "hidden": true }, "source": [ "In order to convert the object into a shared library, we will have to point the compiler to where both `libcxxwrap_julia.so` and `libjulia.so` are. We then generate the shared library `libhello.so` through the following command:\n", "\n", "```bash\n", "JLCXX_LIB=${JLCXX_H}/../lib/\n", "JULIA_LIB=${JULIA_H}/../../lib/\n", "\n", "# Creates shared library libhello.so\n", "c++ -fPIC -march=native -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG \\\n", "-shared -Wl,-soname,libhello.so -o libhello.so hello.cpp.o \\\n", "-Wl,-rpath,: -L${JLCXX_LIB} -lcxxwrap_julia -L${JULIA_LIB} -ljulia\n", "```\n", "\n", "In summary,\n", "* `JULIA_H` must point to the directory that contains `julia.h`.\n", "* `JLCXX_H` must point to the directory that contains `jlcxx/jlcxx.hpp`.\n", "* `JULIA_LIB` must point to the directory that contains `libjulia.so.1` (or whatever version of libjulia you have there).\n", "* `JLCXX_LIB` must point to the directory that contains `libcxxwrap_julia.so.0.12.2` (or whatever version of libcxxwrap you found)." ] }, { "cell_type": "markdown", "id": "8f6f60d2", "metadata": { "hidden": true }, "source": [ "Now, we test that the C++ code was successfully compiled by importing the `libhello` library into Julia and calling its `greet()` function. Open the Julia REPL and paste the following lines:\n", "\n", "```julia\n", "# Load the module and generate the functions\n", "module CppHello\n", " using CxxWrap\n", "\n", " @wrapmodule( () -> \"./libhello\" )\n", "\n", " function __init__()\n", " @initcxx\n", " end\n", "end\n", "\n", "# Call greet and show the result\n", "@show CppHello.greet()\n", "```\n", "\n", "This should have returned a heart-warming Hello World." ] }, { "cell_type": "markdown", "id": "d5624e74", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 3) Cloning FLOWExaFMM" ] }, { "cell_type": "markdown", "id": "0538804d", "metadata": { "hidden": true }, "source": [ "Since we will need to compile the c++ part of ExaFMM, first clone FLOWExaFMM somewhere in your machine:\n", "\n", "```bash\n", "git clone https://github.com/byuflowlab/FLOWExaFMM\n", "```" ] }, { "cell_type": "markdown", "id": "c991c8b1", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### 4) Compiling ExaFMM" ] }, { "cell_type": "markdown", "id": "64a8a99f", "metadata": { "hidden": true }, "source": [ "Before compiling `ExaFMM`, make sure you have an mpi library in your system for parallel processing. In Ubuntu you can install the development tools of `OpenMPI` with the following command:\n", "```bash\n", "sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev\n", "```\n", "\n", "Now, go to wherever you cloned FLOWExaFMM. In order to compile FLOWExaFMM wrapped with `libcxxwrap`, first we need to find the flags `JULIA_H`, `JLCXX_H`, `JULIA_LIB`, and `JLCXX_LIB` inside the build script file `build.sh`, and point them to the paths that we determined in the \"Hello, World\" example (see previous section).\n", "\n", "Now go to the root level of the `FLOWExaFMM` folder and run the command `sh build.sh`. If everything went well, this script will compile and generate a shared library `fmm.so` under `src/` in the `FLOWExaFMM` package." ] }, { "cell_type": "markdown", "id": "1ec6e631", "metadata": { "heading_collapsed": true }, "source": [ "# Adding FLOWExaFMM and FLOWVPM to Julia" ] }, { "cell_type": "markdown", "id": "61cbbb5f", "metadata": { "hidden": true }, "source": [ "Now that ExaFMM is compiled, you can add FLOWExaFMM to your Julia environment as a development package pointing directly to wherever you compiled the package:\n", "\n", "```julia\n", "julia> ] develop path/to/your/flowexafmm/FLOWExaFMM\n", "```\n", "\n", "You can add FLOWVPM to Julia directly from the repo:\n", "```julia\n", "julia> ] add https://github.com/byuflowlab/FLOWVPM.jl\n", "```\n", "\n", "For sanity, check that FLOWExaFMM and FLOWVPM are running correctly by running their unit tests:\n", "\n", "```julia\n", "julia> ] test FLOWExaFMM\n", "julia> ] test FLOWVPM\n", "```" ] }, { "cell_type": "markdown", "id": "18f24096", "metadata": { "heading_collapsed": true }, "source": [ "# Troubleshooting" ] }, { "cell_type": "markdown", "id": "46bee0c9", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### BYU Supercomputer" ] }, { "cell_type": "markdown", "id": "a21efabc", "metadata": { "hidden": true }, "source": [ "Some common problems that may come up when compiling and running FLOWVPM on BYU's [FSL supercomputer](https://rc.byu.edu/documentation/resources)." ] }, { "cell_type": "markdown", "id": "9de6a678", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### `libmpi.so.40: cannot open shared object file`" ] }, { "cell_type": "markdown", "id": "e1178c81", "metadata": { "hidden": true }, "source": [ "Importing FLOWExaFMM I'm running into the following problem:\n", "\n", "```julia\n", "julia> import FLOWExaFMM\n", "ERROR: InitError: could not load library \"/fslhome/edoalvar/Codes/FLOWExaFMM/src/fmm\"\n", "libmpi.so.40: cannot open shared object file: No such file or directory\n", "\n", "```\n", "\n", "It seems like the login node doesn't load the lib folder of openmpi to the system level, so we will have to bundle it up into the shared library manually. This is found under `/apps/openmpi/4.1.1/gcc-10.2.0_cuda-11.2.1/lib`. I ended up taking the last command that is run in `make` and added that path after the `-rpath` flag, then re-run the command manually. This looks as follows:\n", "\n", "```bash\n", "cd build/3d\n", "rm -f fmm; rm ../../src/fmm.so\n", "\n", "mpicxx -ffast-math -funroll-loops -fabi-version=6 -Wfatal-errors -fopenmp -g -O2 -o fmm fmm-fmm.o -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -fPIC -march=native -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -shared -Wl,-rpath,/apps/openmpi/4.1.1/gcc-10.2.0_cuda-11.2.1/lib: -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -L/apps/julia/1.6.1/gcc-10.2.0/include/julia/../../lib -ljulia\n", "\n", "cp fmm ../../src/fmm.so\n", "```\n", "\n", "That should do the trick." ] }, { "cell_type": "markdown", "id": "5a6457ea", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "#### Not Working on Node" ] }, { "cell_type": "markdown", "id": "04e0a7af", "metadata": { "hidden": true }, "source": [ "When using FLOWExaFMM in a node for batch work, the code needs to have been compiled inside the node, so you probably want to recompile the code when you launch each batch. Alternatively, DG suggested using the flag `-march=broadwell` when compiling for the m9 nodes---this way you only need to compile it once in the login node:\n", "\n", "```bash\n", "cd build/3d\n", "rm -f fmm; rm ../../src/fmm.so\n", "\n", "mpicxx -ffast-math -funroll-loops -fabi-version=6 -Wfatal-errors -fopenmp -g -O2 -o fmm fmm-fmm.o -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -fPIC -march=broadwell -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -shared -Wl,-rpath,/apps/openmpi/4.1.1/gcc-10.2.0_cuda-11.2.1/lib: -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -L/apps/julia/1.6.1/gcc-10.2.0/include/julia/../../lib -ljulia\n", "\n", "cp fmm ../../src/fmm.so\n", "```" ] }, { "cell_type": "markdown", "id": "5e872171", "metadata": { "hidden": true }, "source": [ "If useful, here is the compilation instruction that worked for me:\n", "```bash\n", "mpicxx -DHAVE_CONFIG_H -DJULIA_ENABLE_THREADING -Dhello_EXPORTS -I/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include -I/apps/julia/1.6.1/gcc-10.2.0/include/julia -march=broadwell -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -fPIC -I. -I.. -DEXAFMM_WITH_OPENMP -msse3 -mavx -mavx2 -DNDEBUG -DEXAFMM_EAGER -ffast-math -funroll-loops -fabi-version=6 -Wfatal-errors -fopenmp -g -O2 -MT fmm-fmm.o -MD -MP -MF .deps/fmm-fmm.Tpo -c -o fmm-fmm.o `test -f 'fmm.cxx' || echo './'`fmm.cxx\n", "\n", "mpicxx -ffast-math -funroll-loops -fabi-version=6 -Wfatal-errors -fopenmp -g -O2 -o fmm fmm-fmm.o -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -fPIC -march=native -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -shared -Wl,-rpath,: -L/fslhome/edoalvar/.julia/artifacts/16e1de4679fb8520a8af4e6831c7c8e9893d18b4/include/../lib -lcxxwrap_julia -L/apps/julia/1.6.1/gcc-10.2.0/include/julia/../../lib -ljulia\n", "\n", "\n", "```" ] }, { "cell_type": "markdown", "id": "990ef506", "metadata": { "heading_collapsed": true, "hidden": true }, "source": [ "### FLOWExaFMM BinaryBuilder" ] }, { "cell_type": "markdown", "id": "adcd54c8", "metadata": { "hidden": true }, "source": [ "A first stab at generating binary files with BinaryBuilder.\n", "\n", "`https://github.com/byuflowlab/FLOWExaFMM` commit `43c5eecf454f73b828e2536702f8f7d3c6c5889e`\n", "* Binary dependencies: `libcxxwrap_julia`, `libjulia`, `OpenMPI`, `MPICH_jll`, `MicrosoftMPI_jll`, `LLVMOpenMP_jll`\n", "* Project name: `FLOWExaFMM`\n", "* Version: `2.1.0`\n", "* Customize compilers: GCC `v10.2.0`, LLVM `v12.0.0`" ] }, { "cell_type": "markdown", "id": "522c1ec8", "metadata": { "hidden": true }, "source": [ "```bash\n", "cd FLOWExaFMM\n", "\n", "JULIA_H=${WORKSPACE}/destdir/include/julia\n", "JLCXX_H=${WORKSPACE}/destdir/include/jlcxx\n", "\n", "JULIA_LIB=${WORKSPACE}/destdir/lib\n", "JLCXX_LIB=${WORKSPACE}/destdir/lib\n", "\n", "cp -r deps build && cd build\n", "./configure --prefix=${prefix} --build=${MACHTYPE} --host=${target}\n", "cd 3d\n", "make JULIA_H=$JULIA_H JLCXX_H=$JLCXX_H JULIA_LIB=$JULIA_LIB JLCXX_LIB=$JLCXX_LIB\n", "make install\n", "```" ] }, { "cell_type": "markdown", "id": "419ac000", "metadata": { "hidden": true }, "source": [ "TO KEEP IN MIND\n", "\n", "* Use `libcxxwrap_julia-v0.8.3+0` libcxxwrap_julia_jll v0.9.1\n", "* Use `libjulia-v1.6.0+0` instead of `libjulia_jll v1.8.0+2`\n", "\n", "* ```bash\n", "Warning: /tmp/jl_Ju0DSQ/KDG9HsBs/x86_64-linux-gnu-libgfortran5-cxx11/destdir/bin/fmm contains std::string values! This causes incompatibilities across the GCC 4/5 version boundary. To remedy this, you must build a tarball for both GCC 4 and GCC 5. To do this, immediately after your `platforms` definition in your `build_tarballs.jl` file, add the line:\n", "│ \n", "│ platforms = expand_cxxstring_abis(platforms)\n", "└ @ BinaryBuilder.Auditor ~/.julia/packages/BinaryBuilder/CKu9k/src/auditor/compiler_abi.jl:247\n", "```\n", "* Is it possible that the different MPI binaries are conflicting with each other?" ] }, { "cell_type": "markdown", "id": "10ebebb5", "metadata": { "hidden": true }, "source": [ "```bash\n", "mpicxx -DHAVE_CONFIG_H -DJULIA_ENABLE_THREADING -Dhello_EXPORTS -I/workspace/destdir/include/jlcxx -I/workspace/destdir/include/julia -Wunused-parameter -Wextra -Wreorder -std=gnu++1z -O3 -DNDEBUG -fPIC -I. -I.. -DEXAFMM_WITH_OPENMP -DNDEBUG -DEXAFMM_EAGER -ffast-math -funroll-loops -fabi-version=6 -Wfatal-errors -fopenmp -g -O2 -MT fmm-fmm.o -MD -MP -MF .deps/fmm-fmm.Tpo -c -o fmm-fmm.o `test -f 'fmm.cxx' || echo './'`fmm.cxx\n", "```" ] } ], "metadata": { "hide_input": false, "kernelspec": { "display_name": "Julia 1.10.2 (14 threads) 1.10.2", "language": "julia", "name": "julia-1.10.2-_14-threads_-1.10" }, "language_info": { "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", "version": "1.10.2" } }, "nbformat": 4, "nbformat_minor": 5 }