Finite Element Discretization Library __ _ __ ___ / _| ___ _ __ ___ | '_ ` _ \ | |_ / _ \| '_ ` _ \ | | | | | || _|| __/| | | | | | |_| |_| |_||_| \___||_| |_| |_| https://mfem.org This file provides a detailed description of how to build and install the MFEM library. For a simple build, see the step-by-step instructions on the website at https://mfem.org/building. The MFEM library has a serial and an MPI-based parallel version, which largely share the same code base. The only prerequisite for building the serial version of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM requires an MPI C++ compiler, as well as the following external libraries: - hypre (a library of high-performance preconditioners) https://github.com/hypre-space/hypre - METIS (a family of multilevel partitioning algorithms) https://github.com/mfem/tpls Note: We recommend our mirror of metis-4.0.3/5.1.0 above because the METIS webpage, http://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down and we don't support yet the new repo https://github.com/KarypisLab/METIS. The hypre dependency can be downloaded as a tarball from GitHub or from the project webpage https://www.llnl.gov/casc/hypre. For example, the 2.24.0 release of hypre is available at https://github.com/hypre-space/hypre/archive/v2.24.0.tar.gz The METIS dependency can be disabled but that is not generally recommended, see the option MFEM_USE_METIS. MFEM also includes support for devices such as GPUs, and programming models such as CUDA, HIP, OCCA, OpenMP and RAJA. - Starting with version 4.0, MFEM requires a C++11 compiler. We recommend using a newer compiler, e.g. GCC version 4.9 or higher. - CUDA support requires an NVIDIA GPU and an installation of the CUDA Toolkit https://developer.nvidia.com/cuda-toolkit - HIP support requires an AMD GPU and an installation of the ROCm software stack https://rocmdocs.amd.com - OCCA support requires the OCCA library https://libocca.org - OpenMP support requires a compiler implementing the OpenMP API https://www.openmp.org - RAJA support requires installation of the RAJA performance portability layer with (optionally) support for CUDA and OpenMP https://github.com/LLNL/RAJA The library supports two build systems: one based on GNU make, and a second one based on CMake. Both build systems are described below. Some hints for building without GNU make or CMake can be found at the end of this file. In addition to the native build systems, MFEM packages are also available in the following package managers: - Spack, https://github.com/spack/spack - OpenHPC, http://openhpc.community - Conda-forge, https://conda-forge.org (pre-built binaries linked with OpenMPI/MPICH, hypre, and METIS) We also recommend downloading and building the MFEM-based GLVis visualization tool which can be used to visualize the meshes and solution in MFEM's examples and miniapps. See https://glvis.org and https://mfem.org/building. Quick start with GNU make ========================= Serial build: make serial -j 4 Parallel build: (download hypre and METIS 4 from above URLs) (build METIS 4 in ../metis-4.0 relative to mfem/) (build hypre in ../hypre relative to mfem/) make parallel -j 4 CUDA build: make cuda -j 4 (build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_70') HIP build: make hip -j 4 (build for a specific AMD GPU chip: 'make hip -j 4 HIP_ARCH=gfx900') Example codes (serial/parallel, depending on the build): cd examples make -j 4 Build everything (library, examples and miniapps) with current configuration: make all -j 4 Quick-check the build by running Example 1/1p (optional): make check Quick start with CMake ====================== Serial build: mkdir ; cd cmake make -j 4 (assuming "UNIX Makefiles" generator) Parallel build: (download hypre and METIS 4 from above URLs) (build METIS 4 in ../metis-4.0 relative to mfem/) (build hypre in ../hypre relative to mfem/) mkdir ; cd cmake -DMFEM_USE_MPI=YES make -j 4 CUDA build: (this build requires CMake 3.8 or newer) mkdir ; cd cmake -DMFEM_USE_CUDA=YES make -j 4 Example codes (serial/parallel, depending on the build): make examples -j 4 Build everything (library, examples and miniapps) with current configuration: make exec -j 4 Quick-check the build by running Example 1/1p (optional): make check Building with GNU make ====================== The MFEM build system consists of two steps: configuration and compilation. The configuration step can be used to adjust paths to external libraries, compilers, flags, etc, similar to "./configure". It is performed by running make config [OPTIONS] ... The OPTIONS are of the form VARIABLE=VALUE. Detailed description of the configuration options is given below. Alternatively, the options can be specified with an input file: cp config/defaults.mk config/user.mk (edit config/user.mk) make config Note that config/user.mk, if present, is loaded after config/defaults.mk and its path/name can be changed with make config USER_CONFIG= The build system can be configured to use a separate build directory, for an out-of-source build. There are two ways to do that: the first one is mkdir ; cd make -f /makefile config [OPTIONS] ... The second one is cd make BUILD_DIR= config [OPTIONS] ... Note that in both cases the default location for the (optional) user configuration file is /config/user.mk. Once configured, the library can be built simply with cd (if building out-of-source) make Note that re-configuration is only needed to change the currently configured options. Several shortcut targets combining (re-)configuration and compilation are also defined: make serial -> Builds serial optimized version of the library make parallel -> Builds parallel optimized version of the library make debug -> Builds serial debug version of the library make pdebug -> Builds parallel debug version of the library make cuda -> Builds serial cuda optimized version of the library make pcuda -> Builds parallel cuda optimized version of the library make cudebug -> Builds serial cuda debug version of the library make pcudebug -> Builds parallel cuda debug version of the library make hip -> Builds serial hip optimized version of the library make phip -> Builds parallel hip optimized version of the library make hipdebug -> Builds serial hip debug version of the library make phipdebug -> Builds parallel hip debug version of the library Note that any of the above shortcuts accept configuration options, either at the command line or through a user configuration file. The build can be quick-tested by running make check which will simply compile and run Example 1/1p. For more extensive tests that check the results from all the serial/parallel MFEM examples and miniapps use: make test Note that by default MFEM uses "mpirun -np" in its test runs (this is also what is used in the sample runs of its examples and miniapps). The MPI launcher can be changed by the user as described in the "Specifying an MPI job launcher" section at the end of this file. Running all the tests may take a while. Implementation details about the check and test targets can be found in the top-level makefile and the config/test.mk file. An optional installation of the library and the headers can be performed with make install [PREFIX=] The library will be installed in $(PREFIX)/lib, the headers in $(PREFIX)/include, and the configuration makefile (config.mk) in $(PREFIX)/share/mfem. The PREFIX option can also be set during configuration. Information about the current build configuration can be viewed using make status make info To clean the library and object files, but keep the current configuration, use make clean To clean everything, including the current configuration, use make distclean For a short help message, use make help The build process creates the MFEM library (libmfem.a) and the include file (mfem.hpp) needed in MFEM-based applications, see e.g. the example codes in the examples/ directory or the miniapps in the miniapps/ directory. A selected subset of configuration options and derived makefile variables are also written to the file config/config.mk. This file can be included by other makefiles to obtain information about the MFEM configuration, see e.g. the makefile in the examples/ directory. Configuration options (GNU make) ================================ See the configuration file config/defaults.mk for the default settings. Compilers: CXX - C++ compiler, serial build MPICXX - MPI C++ compiler, parallel build CUDA_CXX - The CUDA compiler, 'nvcc' Compiler options: OPTIM_FLAGS - Options for optimized build DEBUG_FLAGS - Options for debug build CXXFLAGS - If not set, defined based on the above optimized/debug flags CPPFLAGS - Additional compiler options Build options: STATIC - Build a static version of the library (YES/NO), default = YES SHARED - Build a shared version of the library (YES/NO), default = NO Installation options: PREFIX - Specify the installation directory. The library (libmfem.a) will be installed in $(PREFIX)/lib, the headers in $(PREFIX)/include, and the configuration makefile (config.mk) in $(PREFIX)/share/mfem. INSTALL - Specify the install program, e.g /usr/bin/install MFEM library features/options (GNU make) ---------------------------------------- MFEM_USE_MPI = YES/NO Choose parallel/serial build. The parallel build requires proper setup of the HYPRE_* and METIS_* library options, see below. MFEM_USE_METIS = YES/NO Enable/disable the use of the METIS library. By default, this option is set to the value of MFEM_USE_MPI. If this option is explicitly disabled in a parallel build, then the only parallel partitioning (domain decomposition) option in the library will be Cartesian partitioning with box meshes, and thus most of the parallel examples and miniapps will fail. MFEM_PRECISION = double/Double/DOUBLE/single/Single/SINGLE Use single (float type) or double floating-point precision. In the configuration header 'config/_config.hpp' this option is represented by defining exactly one of the macros: MFEM_USE_DOUBLE, or MFEM_USE_SINGLE. In the exported config files 'config.mk' and 'MFEMConfig.cmake', the option is represented by the variables MFEM_USE_DOUBLE and MFEM_USE_SINGLE defined as YES/NO (make) or ON/OFF (cmake). For more details see https://github.com/orgs/mfem/discussions/4207 MFEM_DEBUG = YES/NO Choose debug/optimized build. The debug build enables a number of messages and consistency checks that may simplify bug-hunting. MFEM_USE_EXCEPTIONS = YES/NO Enable the use of exceptions. In particular, modifies the default behavior when errors are encountered: throw an exception, instead of aborting. MFEM_USE_LIBUNWIND = YES/NO Use libunwind to print a stacktrace whenever mfem_error is raised. The information printed is enough to determine the line numbers where the error originated, provided MFEM_DEBUG=YES or build flags include `-g'. MFEM_USE_METIS_5 = YES/NO Specify the version of the METIS library - 5 (YES) or 4 (NO). MFEM_USE_LAPACK = YES/NO Use LAPACK routines for various dense linear algebra operations. When enabled, this option uses the LAPACK_* library options, see below. (When not enabled MFEM provides simple internal implementations where appropriate.) MFEM_THREAD_SAFE = YES/NO Use thread-safe implementation for some classes/methods. This comes at the cost of extra memory allocation and de-allocation. MFEM_USE_LEGACY_OPENMP = YES/NO Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE. This option is deprecated. MFEM_USE_OPENMP = YES/NO Enable the OpenMP backend. MFEM_USE_MEMALLOC = YES/NO Internal MFEM option: enable batch allocation for some small objects. Recommended value is YES. MFEM_TIMER_TYPE = 0/1/2/3/4/5/6/NO Specify which library functions to use in the class StopWatch used for measuring time. The available options are: 0 - use std::clock from , standard C++ 1 - use times from 2 - use high-resolution POSIX clocks (see option POSIX_CLOCKS_LIB) 3 - use QueryPerformanceCounter from 4 - use mach_absolute_time from + std::clock (Mac) 5 - use gettimeofday from 6 - use MPI_Wtime from NO - use option 3 if the compiler macro _WIN32 is defined, 0 otherwise MFEM_USE_SUNDIALS = YES/NO Enable MFEM time integrators and non-linear solvers based on the SUNDIALS library. When enabled, this option uses the SUNDIALS_* library options, see below. MFEM_USE_SUITESPARSE = YES/NO Enable MFEM functionality based on the SuiteSparse library. Currently, this option adds the classes UMFPackSolver and KLUSolver (both sparse serial direct solvers). When enabled, this option uses the SUITESPARSE_* library options, see below. MFEM_USE_SUPERLU = YES/NO Enable MFEM functionality based on the SuperLU_DIST library. Currently, this option adds the classes SuperLUSolver (a parallel sparse direct solver) and SuperLURowLocMatrix a distributed CSR matrix class needed by SuperLU. When enabled, this option uses the SUPERLU_* library options, see below. MFEM_USE_SUPERLU5 = YES/NO If SuperLU functionality is enabled, use the older 5.1.0 version rather than the more recent 6+ versions. MFEM_USE_MUMPS = YES/NO Enable MFEM functionality based on the MUMPS library. Currently, this option adds the class MUMPSSolver (a parallel sparse direct solver). When enabled, this option uses the MUMPS_* library options, see below. MFEM_USE_STRUMPACK = YES/NO Enable MFEM functionality based on the STRUMPACK sparse direct solver and preconditioner through the STRUMPACKSolver and STRUMPACKRowLocMatrix classes. When enabled, this option uses the STRUMPACK_* library options, see below. MFEM_USE_GINKGO = YES/NO Enable MFEM functionality based on the Ginkgo library, which provides iterative linear solvers and preconditioners with OpenMP, CUDA backends, see https://github.com/ginkgo-project/ginkgo. When enabled, the user can use Ginkgo's solvers and preconditioners as shown in examples/ginkgo/. MFEM_USE_AMGX = YES/NO Enable MFEM functionality based on the AmgX multigrid library from NVIDIA. Allows the user to use SparseMatrices and HypreParMatrices to solve linear systems with the routines from the AmgX library. MFEM_USE_GNUTLS = YES/NO Enable secure socket support in class socketstream, using the auxiliary GnuTLS_* classes, based on the GnuTLS library. This option may be useful in multi-user environment to prevent users from sending/receiving visualization data to/from other users. When this option is enabled, the default behavior in class socketstream is to use secure sockets, e.g. when connecting to a GLVis visualization server. In order for this to work, one needs to generate GLVis server/client key pairs (in ~/.config/glvis), similar to ssh keys -- the script 'glvis-keygen.sh' in the main GLVis directory can be used to do that: bash glvis-keygen.sh ["Your Name"] ["Your Email"] In MFEM v3.3.2 and earlier, the secure authentication is based on OpenPGP keys, while later versions use X.509 certificates. The latest version of the script 'glvis-keygen.sh' can be used to generate both types of keys. When MFEM_USE_GNUTLS is enabled, the additional build options, GNUTLS_*, are also used, see below. MFEM_USE_NETCDF = YES/NO NetCDF is the library that is used by the SNL Cubit mesh generator to create Genesis mesh files. This option enables a reader for these files, which requires that NetCDF be installed, see the NETCDF_* build options below. MFEM_USE_PETSC = YES/NO Enable MFEM linear and non-linear solvers, preconditioners, time integrators and other features based on the PETSc package. When enabled, this option uses the PETSC_* library options, see below. MFEM_USE_SLEPC = YES/NO Enable MFEM eigensolvers based on the SLEPc package. When enabled, this option uses the SLEPC_* library options, see below. MFEM_USE_MPFR = YES/NO MPFR is a library for multiple-precision floating-point computations. This option enables the use of MPFR in MFEM, e.g. for precise computation of 1D quadrature rules. When enabled, this option uses the MPFR_* library options, see below. MFEM_USE_SIDRE = YES/NO Sidre is a component of LLNL's axom project, https://github.com/LLNL/axom, that provides an HDF5-based file format for visualization or restart capability following the Conduit (https://github.com/LLNL/conduit) mesh blueprint specification. When enabled, this option requires installation of HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project. MFEM_USE_SIMD = YES/NO Enables the high performance templated classes to use architecture dependent SIMD intrinsics instead of the generic implementation of class AutoSIMD in linalg/simd/auto.hpp. This option should be combined with suitable compiler options, such as -march=native, to enable optimal vectorization. MFEM_USE_CONDUIT = YES/NO Enables support for converting MFEM Mesh and Grid Function objects to and from Conduit Mesh Blueprint Descriptions (https://github.com/LLNL/conduit/) and support for JSON and Binary I/O via Conduit Relay. This option requires an installation of Conduit. If Conduit was built with HDF5 support, it also requires an installation of HDF5 (see also MFEM_USE_NETCDF). MFEM_USE_ADIOS2 = YES/NO Enables support for ADIOS2, version 2 of the adaptable input output system for scientific data management. In MFEM, ADIOS2 provides parallel I/O with ParaView visualization. MFEM_USE_ZLIB = YES/NO Enables use of on-the-fly gzip compressed streams. With this feature enabled (YES), MFEM can compress its output files on-the-fly. In addition, it can read back files compressed with zlib (or any compression utility capable of creating a gzip-compatible output such as gzip). MFEM will write compressed files if the mode argument in the constructor includes a 'z' character. With this feature disabled (NO), MFEM will not be able to properly read an input file if it is gzip compressed. In that case, the solution is to uncompress the file with an external tool (such as gunzip) before attempting to use it with MFEM. When enabled, this option uses the ZLIB_* library options, see below. MFEM_USE_PUMI = YES/NO Enable the usage of PUMI (https://scorec.rpi.edu/pumi/) in MFEM. The Parallel Unstructured Mesh Infrastructure (PUMI) is an unstructured, distributed mesh data management system that is capable of handling general non-manifold models and effectively supports automated adaptive analysis. PUMI enables support for parallel unstructured mesh modifications in MFEM. The develop branch of PUMI repository (https://github.com/SCOREC/core) should be used for most updated features. MFEM_USE_UMPIRE = YES/NO Enables support for Umpire, a resource management library that allows the discovery, provision, and management of memory on machines with multiple memory devices like NUMA and GPUs. MFEM_USE_BENCHMARK = YES/NO Enables support for Google Benchmark, a library to support the benchmarking of functions, in the tests/benchmarks directory. MFEM_USE_HIOP = YES/NO Enable the usage of HiOp (https://github.com/LLNL/hiop) in MFEM. HiOp is an HPC solver for nonlinear optimization problems. MFEM_USE_CODIPACK = YES/NO Enable automatic differentiation using the CoDiPack library. www.scicomp.uni-kl.de/codi/ MFEM_USE_ALGOIM = YES/NO Enable the usage of Algoim - a collection of high-order accurate numerical methods and C++ algorithms for working with implicitly-defined geometry and level set methods. The Algoim library requires the Blitz++ library. The MFEM provides interface to Algoim v1. Thus, to check out the specific state use: git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a https://algoim.github.io MFEM_USE_ADFORWARD = YES/NO Enable forward mode for AD packages. This option is valid only if the AD package supports two modes (backward/forward). MFEM_USE_CUDA = YES/NO Enables support for CUDA devices in MFEM. CUDA is a parallel computing platform and programming model for general computing on graphical processing units (GPUs). The variable CUDA_ARCH is used to specify the CUDA compute capability used during compilation (by default, CUDA_ARCH=sm_60). When enabled, this option uses the CUDA_* build options, see below. MFEM_USE_HIP = YES/NO Enables support for AMD devices in MFEM. HIP is a heterogeneous-compute interface for portability developed by AMD that can target both AMD and NVIDIA GPUs. The variable HIP_ARCH is used to specify the AMD GPU processor used during compilation (by default, HIP_ARCH=gfx900). When enabled, this option uses the HIP_* build options, see below. MFEM_USE_RAJA = YES/NO Enable support for the RAJA performance portability layer in MFEM. RAJA provides a portable abstraction for loops, supporting different programming model backends. When using RAJA built with CUDA support, CUDA support must be also enabled in MFEM, i.e. MFEM_USE_CUDA=YES must be set. MFEM_USE_OCCA = YES/NO Enables support for the OCCA library in MFEM. OCCA is an open-source library which aims to make it easy to program different types of devices (e.g. CPU, GPU, FPGA) by providing an unified API for interacting with JIT-compiled backends. In order to use the OCCA CUDA backend, CUDA support must be enabled in MFEM as well, i.e. MFEM_USE_CUDA=YES must be set. MFEM_USE_GSLIB = YES/NO Enables MFEM functionality based on the GSLIB library, and specifically its FindPoints component, which provides a robust algorithms to evaluate finite element functions in a collection of points in physical space. When enabled, the user can use the GSLIB-FindPoints methods as shown in miniapps/gslib. MFEM_USE_CEED = YES/NO Enables support for the libCEED library in MFEM. libCEED is a portable library for performant high-order operator evaluation developed by the Center for Efficient Exascale Discretizations in the Exascale Computing Project. MFEM_USE_MKL_CPARDISO = YES/NO Enables the interface to MKL CPardiso: the Intel MKL Parallel Direct Sparse Solver for Clusters. Make sure to set the correct values for MKL_MPI_WRAPPER and MKL_LIBRARY_SUBDIR as shown in defaults.mk. If you configure MFEM with MFEM_USE_LAPACK=YES, verify that the MKL LAPACK libraries are used. The OpenMP capabilities are disabled at link time. MFEM_USE_MOONOLITH = YES/NO Enables the ParMoonolith interface for parallel non-conforming, non-matching, variational, volumetric mesh information transfer. It requires the variable MOONOLITH_DIR= to be defined in the environment in order to be used with the Makefile. Makefile users are also required to install moonolith using the command `make install_all`, see https://bitbucket.org/zulianp/par_moonolith for details. Although Moonolith is an MPI-based library, both serial (MFEM_USE_MPI=NO) and parallel (MFEM_USE_MPI=YES) versions of MFEM are supported. MFEM_USE_CALIPER = YES/NO Enables the interface to Caliper. Caliper is a library to integrate performance profiling capabilities into applications. To use Caliper, developers mark code regions of interest using either Caliper's annotation API or their equivalent in MFEM. Applications can then enable performance profiling at runtime with Caliper's configuration API. Alternatively, one can configure Caliper through environment variables or config files. MFEM_USE_FMS = YES/NO Enables support for the FMS library which consists of the DataCollection sub-class mfem::FMSDataCollection for I/O in FMS formats, see the header file fem/fmsdatacollection.hpp. In addition, this option enables in-memory conversion routines between FMS's FmsDataCollection structure and MFEM's DataCollection class, see the header file fem/fmsconvert.hpp. MFEM_USE_PARELAG = YES/NO Enables the miniapps that use the ParELAG library. MFEM does not currently use ParELAG. In fact, ParELAG is dependent on MFEM. Therefore, this option currently only concerns the miniapps. MFEM_USE_ENZYME = YES/NO Enables automatic differentiation support through the LLVM plugin Enzyme. This requires the compiler to be set to clang (>=14.0.0). We also advise to use the link time optimization (LTO) plugin, to enable functions that you define over multiple files (compilation units) and want to be differentiated automatically, to work. This requires to also use LLVM/LLD for linking. Recommended options are in config/defaults.mk. MFEM_BUILD_TAG = (any value) An optional tag to characterize the build. Exported to config/config.mk. Can be used to identify the MFEM build from other makefiles. VERBOSE = YES/NO Print some informational messages when building. External libraries (GNU make): ------------------------------ Two types of library configuration options are used: _OPT - for compiler options which usually specify an include path, e.g.: -I/home/user/hypre/include _LIB - for link options which usually specify link path and library name, e.g.: -L/home/user/hypre/lib -lHYPRE If specifying relative paths, they should be relative to the top-level MFEM directory and use the string @MFEM_DIR@, e.g. HYPRE_OPT = -I@MFEM_DIR@/../hypre. The specific libraries and their options are: - HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES. See also the "Specific options for hypre" section at the end of this file. URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre Options: HYPRE_OPT, HYPRE_LIB. Versions: HYPRE >= 2.10.0b (HYPRE built without CUDA) HYPRE >= 2.20.0 (HYPRE built with '--enable-mixedint') HYPRE >= 2.22.1 (HYPRE built with CUDA) HYPRE >= 2.23.0 (HYPRE built with HIP) - METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set MFEM_USE_METIS_5 = YES (default is to use METIS 4). URL: https://github.com/mfem/tpls (MFEM mirror, see above) Options: METIS_OPT, METIS_LIB. Versions: METIS 4.0.3 or 5.1.0. - LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized implementations can also be used, e.g. the ATLAS project. URL: http://www.netlib.org/lapack (LAPACK) http://math-atlas.sourceforge.net (ATLAS) Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB. - OpenMP (optional), usually part of compiler, used when either MFEM_USE_OPENMP or MFEM_USE_LEGACY_OPENMP is set to YES. Options: OPENMP_OPT, OPENMP_LIB. - High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be necessary to link with a system library (e.g. librt.so). Option: POSIX_CLOCKS_LIB (default = -lrt). - SUNDIALS (optional), used when MFEM_USE_SUNDIALS = YES. Beginning with MFEM v3.3, SUNDIALS v2.7.0 is supported. Beginning with MFEM v3.3.2, SUNDIALS v3.0.0 is also supported. Beginning with MFEM v4.1, only SUNDIALS v5.0.0+ is supported. When MFEM_USE_CUDA is enabled, only SUNDIALS v5.4.0+ is supported. If MFEM_USE_MPI is enabled, we expect that SUNDIALS is built with support for both MPI and hypre. If MFEM_USE_CUDA is enabled, we expect that SUNDIALS is built with support for CUDA. If MFEM_USE_HIP is enabled, we expect that SUNDIALS is built with support for HIP. URL: http://computing.llnl.gov/projects/sundials/sundials-software Options: SUNDIALS_OPT, SUNDIALS_LIB. Versions: SUNDIALS >= 5.0.0, SUNDIALS >= 5.4.0 for CUDA support, and SUNDIALS >= 5.7.0 for HIP support. - SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES. URL: http://faculty.cse.tamu.edu/davis/suitesparse.html Options: SUITESPARSE_OPT, SUITESPARSE_LIB. Versions: SuiteSparse >= 4.5.4, older versions may work too. - SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution. Both ParMETIS and the included METIS 5 should be built and installed in the same location. If using SuperLU_Dist v5, set MFEM_USE_SUPERLU5=YES. URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU Options: SUPERLU_OPT, SUPERLU_LIB. Versions: SuperLU_DIST >= 5.1.0. - MUMPS (optional), used when MFEM_USE_MUMPS = YES. Note that MUMPS requires LAPACK, SCALAPACK and a reordering package such as PORD or METIS. URL: http://mumps.enseeiht.fr Options: MUMPS_OPT, MUMPS_LIB. Versions: MUMPS >= 5.1.1 - STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS and PT-Scotch are optional dependencies. The support for STRUMPACK was added in MFEM v3.3.2. URL: http://portal.nersc.gov/project/sparse/strumpack Options: STRUMPACK_OPT, STRUMPACK_LIB. Versions: STRUMPACK >= 3.0.0. - Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a C++ compiler that supports the C++-14 standard. For additional requirements and dependencies of specific modules, see the Ginkgo webpage below. URL: https://ginkgo-project.github.io Options: GINKGO_OPT, GINKGO_LIB, GINKGO_DIR, GINKGO_BUILD_TYPE (Release or Debug). Versions: Ginkgo >= 1.4.0. - AmgX (optional), used when MFEM_USE_AMGX = YES. URL: https://github.com/NVIDIA/AMGX Options: AMGX_OPT, AMGX_LIB. Versions: AmgX >= 2.1, older versions may work too. - GnuTLS (optional), used when MFEM_USE_GNUTLS = YES. On most Linux systems, GnuTLS is available as a development package, e.g. gnutls-devel. On Mac OS X, one can get the library through the Homebrew package manager (http://brew.sh). URL: http://gnutls.org Options: GNUTLS_OPT, GNUTLS_LIB. Versions: GnuTLS >= 2.12.0, older versions may work too. - NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit mesh files. Also requires installation of HDF5 and ZLIB, as explained at the NetCDF web site. Note that we use the plain vanilla "C" version of NetCDF, you don't need the C++ or parallel versions. URL: www.unidata.ucar.edu/software/netcdf Options: NETCDF_OPT, NETCDF_LIB. Versions: NetCDF >= 4.4.0. - PETSc (optional), used when MFEM_USE_PETSC = YES. Version 3.8 or higher of the PETSC dev branch is required. The MFEM and PETSc builds can share common libraries, e.g., hypre and SUNDIALS. Here's an example configuration, assuming PETSc has been cloned on the same level as mfem and hypre: ./configure --download-fblaslapack=yes --download-scalapack=yes \ --download-mumps=yes --download-suitesparse=yes \ --with-hypre-dir=../hypre/src/hypre \ --with-shared-libraries=0 When building PETSc with HIP, one may need to add a flag like -std=c2x to CFLAGS to allow proper parsing of the hipsparse header under C. URL: https://www.mcs.anl.gov/petsc Options: PETSC_OPT, PETSC_LIB. Versions: PETSc >= 3.8.0 (PETSc build without CUDA/HIP) PETSc >= 3.15.0 (PETSc built with CUDA) PETSc >= 3.19.0 (PETSc built with HIP, older versions may work too) - SLEPc (optional), used when MFEM_USE_SLEPC = YES. SLEPc depends on PETSc and uses some of the PETSc options when compiled. URL: https://slepc.upv.es/ Options: SLEPC_OPT, SLEPC_LIB. Versions: SLEPc >= 3.8.0. - Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES. Starting with MFEM v4.1, Axom version 0.3.1 or later is required. URL: https://github.com/LLNL/axom https://github.com/LLNL/conduit (Conduit) https://support.hdfgroup.org/HDF5 (HDF5) Options: SIDRE_OPT, SIDRE_LIB. Versions: Axom >= 0.3.1. - Conduit (optional), used when MFEM_USE_CONDUIT = YES. Conduit Mesh Blueprint support requires Conduit >= v0.3.1 and VisIt >= v2.13.1 to read the output. URL: https://github.com/LLNL/conduit (Conduit) https://support.hdfgroup.org/HDF5 (HDF5) Options: CONDUIT_OPT, CONDUIT_LIB. Versions: Conduit >= 0.3.1. - ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES. URL: https://adios2.readthedocs.io/ Versions: ADIOS >= 2.5.0. - PUMI (optional), used when MFEM_USE_PUMI = YES. URL: https://scorec.rpi.edu/pumi https://github.com/SCOREC/core Options: PUMI_OPT, PUMI_LIB. Versions: PUMI >= 2.2.6. - HiOp (optional), used when MFEM_USE_HIOP = YES. URL: https://github.com/LLNL/hiop Options: HIOP_OPT, HIOP_LIB. Versions: HIOP >= 0.4.6. - CoDiPack (optional), used with MFEM_USE_CODIPACK = YES URL: https://www.scicomp.uni-kl.de/codi/ Options: CODIPACK_OPT Versions: 1.9.3 - GSLIB (optional), used when MFEM_USE_GSLIB = YES. The gslib library must be built prior to the MFEM build, as follows: download gslib-1.0.7, untar it at the same level as MFEM and create a symbolic link: "ln -s gslib-1.0.7 gslib". Build gslib in parallel or in serial based on the desired MFEM build: "make clean; make CC=mpicc" or "make clean; make CC=gcc MPI=0". Build MFEM with MFEM_USE_GSLIB=YES. URL: https://github.com/gslib/gslib/archive/v1.0.7.tar.gz Options: GSLIB_OPT, GSLIB_LIB. Versions: GSLIB >= 1.0.7. - ALGOIM (optional), used when MFEM_USE_ALGOIM=YES. The library provides only headers so it just needs to be downloaded at the same level as MFEM. Download the specific version we use as: "git clone https://github.com/algoim/algoim.git; git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a" ALGOIM depends on BLITZ and the library must be built prior to the MFEM build. Download v1.0.2, untar it at the same level as MFEM and create a symbolic link: "ln -s blitz-1.0.2 blitz". Build Blitz using CMake as: "cmake . -DCMAKE_INSTALL_PREFIX=.; make lib; make install" URL: https://github.com/blitzpp/blitz/archive/refs/tags/1.0.2.tar.gz Options: BLITZ_OPT, BLITZ_LIB Versions: BLITZ = 1.0.2 - MKL CPardiso (optional), used when MFEM_USE_MKL_CPARDISO = YES. URL: https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html Options: MKL_CPARDISO_OPT, MKL_CPARDISO_LIB. Versions: Intel MKL >= 2020. - CUDA (optional), used when MFEM_USE_CUDA = YES. URL: https://developer.nvidia.com/cuda-toolkit Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB. Versions: CUDA >= 10.1.168. - HIP (optional), used when MFEM_USE_HIP = YES. URL: https://rocmdocs.amd.com Options: HIP_CXX, HIP_ARCH, HIP_OPT, HIP_LIB. - OCCA (optional), used when MFEM_USE_OCCA = YES. URL: https://libocca.org Options: OCCA_DIR, OCCA_OPT, OCCA_LIB. Versions: OCCA >= 1.1.0. - libCEED (optional), used when MFEM_USE_CEED = YES. URL: https://github.com/CEED/libCEED https://ceed.exascaleproject.org/libceed Options: CEED_DIR, CEED_OPT, CEED_LIB. Versions: libCEED >= 0.12. - RAJA (optional), used when MFEM_USE_RAJA = YES. Beginning with MFEM v4.5.1, only RAJA v2022.10.3+ is supported. URL: https://github.com/LLNL/RAJA Options: RAJA_DIR, RAJA_OPT, RAJA_LIB. Versions: RAJA >= 2022.10.3. - Moonolith (optional), use when MFEM_USE_MOONOLITH = YES. URL: https://bitbucket.org/zulianp/par_moonolith Options: MOONOLITH_DIR Versions: MOONOLITH >= 1.1.0. - Caliper (optional), used when MFEM_USE_CALIPER = YES. URL: https://github.com/LLNL/Caliper Options: CALIPER_DIR Versions: CALIPER >= 2.5.0, older versions may work too. - Umpire, used when MFEM_USE_UMPIRE = YES. Umpire requires camp when the Umpire version is >= 3.0.0. URL: https://github.com/LLNL/Umpire Options: UMPIRE_DIR, UMPIRE_OPT, UMPIRE_LIB. Versions: Umpire >= 3.0.0. - Benchmark, used when MFEM_USE_BENCHMARK = YES. URL: https://github.com/google/benchmark Options: BENCHMARK_DIR, BENCHMARK_LIB. Versions: Benchmark >= 1.5.6. - MPFR (optional), used when MFEM_USE_MPFR = YES. URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org Options: MPFR_OPT, MPFR_LIB. - Libunwind (optional), used when MFEM_USE_LIBUNWIND = YES. The library is included with OS X (as of version 10.11). On Linux it could be installed with the libunwind-devel package. URL: http://www.nongnu.org/libunwind Options: LIBUNWIND_OPT, LIBUNWIND_LIB. - ZLIB (optional), used when MFEM_USE_ZLIB = YES, or when MFEM_USE_NETCDF = YES (in the default settings for NETCDF_OPT and NETCDF_LIB). URL: https://zlib.net Options: ZLIB_OPT, ZLIB_LIB. - FMS (optional), used when MFEM_USE_FMS = YES. URL: https://github.com/CEED/FMS Options: FMS_OPT, FMS_LIB. Versions: FMS >= 0.2. - ParELAG, used when MFEM_USE_PARELAG = YES. URL: https://github.com/LLNL/parelag Options: PARELAG_DIR, PARELAG_OPT, PARELAG_LIB. - Enzyme, used when MFEM_USE_ENZYME = YES. Requires LLVM/Clang >= 14.0.0. URL: https://github.com/EnzymeAD/Enzyme Options: ENZYME_DIR, ENZYME_OPT, ENZYME_LIB. Versions: Enzyme >= v0.0.33. Building with CMake =================== The MFEM build system consists of two steps: configuration and compilation. The configuration step can be used to adjust paths to external libraries, compilers, flags, etc, similar to any CMake build system. It is performed by running mkdir ; cd cmake [OPTIONS] ... The OPTIONS are of the form -D=, e.g. -DMFEM_USE_MPI=YES. Detailed description of the configuration options is given below. Alternatively, the options can be specified with an input file: cd /config cp defaults.cmake user.cmake (edit user.cmake) cd cmake Note that user.cmake, if present, is loaded before defaults.cmake (and thus the former takes precedence over the latter) and its path/name can be changed with cmake -DUSER_CONFIG= Debug and optimization options are controlled through the CMake variable CMAKE_BUILD_TYPE which can be set to standard values like "Debug", and "Release" (default). To use a specific generator use the "-G " option of cmake: cmake -G "Xcode" cmake -G "Visual Studio 12 2013" cmake -G "MinGW Makefiles" With CMake it is possible to build MFEM as a shared library using the standard CMake option -DBUILD_SHARED_LIBS=1. Once configured, the library can be built simply with (assuming a UNIX type system, where the default is to generate "UNIX Makefiles") make -j 4 or cmake --build . or cmake --build . --config Release [Visual Studio, Xcode] The build can be quick-tested by running make check or cmake --build . --target check or cmake --build . --config Release --target check [Visual Studio, Xcode] which will simply compile and run Example 1/1p. For more extensive tests that check the results from all the serial/parallel MFEM examples and miniapps use: make exec -j 4 make test or cmake --build . --target exec cmake --build . --target test or cmake --build . --config Release --target exec [Visual Studio, Xcode] cmake --build . --config Release --target RUN_TESTS [Visual Studio, Xcode] Note that running all the tests may take a while. Installation prefix can be configured by setting the standard CMake variable CMAKE_INSTALL_PREFIX. To install the library, use make install or cmake --build . --target install or cmake --build . --config Release --target install [Xcode] cmake --build . --config Release --target INSTALL [Visual Studio] The library will be installed in /lib, the headers in /include, and the configuration CMake files in /lib/cmake/mfem. Configuration variables (CMake) =============================== See the configuration file config/defaults.cmake for the default settings. Note: the option MFEM_USE_CUDA requires CMake version 3.8 or newer! Non-standard CMake variables for compilers: CXX - If set, overwrite the auto-detected C++ compiler, serial build MPICXX - If set, overwrite the auto-detected MPI C++ compiler, parallel build The compiler options for the various build types can be controlled using standard CMake variables like CMAKE_CXX_FLAGS_RELEASE and CMAKE_CXX_FLAGS_DEBUG. MFEM library features/options (CMake) ------------------------------------- The following options are equivalent to the GNU make options with the same name: [see "MFEM library features/options (GNU make)" above] MFEM_USE_MPI MFEM_USE_METIS - Set to ${MFEM_USE_MPI}, can be overwritten. MFEM_PRECISION MFEM_USE_LIBUNWIND MFEM_USE_LAPACK MFEM_THREAD_SAFE MFEM_USE_LEGACY_OPENMP MFEM_USE_OPENMP MFEM_USE_MEMALLOC MFEM_TIMER_TYPE - Set automatically, can be overwritten. MFEM_USE_SUITESPARSE MFEM_USE_SUPERLU MFEM_USE_MUMPS MFEM_USE_STRUMPACK MFEM_USE_GINKGO MFEM_USE_AMGX MFEM_USE_GNUTLS MFEM_USE_NETCDF MFEM_USE_MPFR MFEM_USE_ZLIB MFEM_USE_PUMI MFEM_USE_HIOP MFEM_USE_CODIPACK MFEM_USE_ADFORWARD MFEM_USE_CUDA MFEM_USE_HIP MFEM_USE_OCCA MFEM_USE_CEED MFEM_USE_RAJA MFEM_USE_UMPIRE MFEM_USE_SIDRE MFEM_USE_MOONOLITH MFEM_USE_CALIPER MFEM_USE_FMS MFEM_USE_BENCHMARK MFEM_USE_PARELAG MFEM_USE_ENZYME The following options are CMake specific: MFEM_ENABLE_TESTING - Enable the ctest framework for testing. MFEM_ENABLE_EXAMPLES - Build all of the examples by default. MFEM_ENABLE_MINIAPPS - Build all of the miniapps by default. External libraries (CMake): --------------------------- For details about the external libraries, see the "External libraries (GNU make)" section above. The MFEM CMake build system provides auto-detection for some packages/libraries, as listed below. The following configuration options are used/defined: _DIR Directory to search for first. The exact subdirectories searched, for headers and libraries, are chosen based on . If the library is not found in this location, then standard locations are searched. _REQUIRED_PACKAGES Specifies a list of package names that have to be explicitly added when linking with in addition to its main library, e.g. ParMETIS requires METIS, so we set ParMETIS_REQUIRED_PACKAGES to "METIS", see defaults.cmake. _INCLUDE_DIRS Location of the headers. Set by auto-detection, if successful. Can be set explicitly, e.g. if auto-detection fails. _LIBRARIES List of the library files/names/link-options. Set by auto-detection, if successful. Can be set explicitly, e.g. if auto-detection fails. The CMake build system adds auto-detection for the following packages/libraries: - HYPRE - METIS - The option MFEM_USE_METIS_5 is auto-detected. - ParMETIS - SuiteSparse - SuperLUDist, STRUMPACK - Ginkgo - AMGX - GNUTLS - Extends the built-in CMake support, to search GNUTLS_DIR as well. - NETCDF - MPFR - LIBUNWIND - POSIXCLOCKS - PUMI - HIOP - CoDiPack - OCCA - RAJA - UMPIRE - AXOM - Used when MFEM_USE_SIDRE is enabled - MOONOLITH - CALIPER - FMS - BENCHMARK - ParELAG - Enzyme The following built-in CMake packages are also used: - MPI, OpenMP, ZLIB - LAPACK, BLAS - Both are enabled via MFEM_USE_LAPACK. If auto-detection fails, set the _LIBRARIES option directly; the configuration option _DIR is not supported. Building without GNU make or CMake ================================== Before using another build system (e.g. Visual Studio) it is necessary to create a proper configuration header file, config/config.hpp, using the template from config/config.hpp.in: cp config/config.hpp.in config/_config.hpp The file config/_config.hpp can then be edited to enable desired options. The MFEM library is simply a combination of all object files obtained by compiling the .cpp source files in the source directories: general, linalg, mesh, and fem. Specifying an MPI job launcher ============================== By default, MFEM will use 'mpirun -np #' to launch any of its parallel tests or miniapps, where # is the number of MPI tasks. An alternate MPI launcher can be provided by setting the MFEM_MPIEXEC and MFEM_MPIEXEC_NP config variables. MFEM will expect the launcher command, plus the command line option to allow it to specify a number of MPI tasks. MFEM_MPIEXEC = mpirun # default MFEM_MPIEXEC_NP = -np # default MFEM_MPIEXEC = srun # example for platforms using SLURM MFEM_MPIEXEC_NP = -n # example for platforms using SLURM Specific options for hypre ========================== The hypre library has multiple options to define local and global index storage sizes. By default, all indices are stored as an architecture aware integer. For most platforms, this will be 32-bit. This limits the maximum number of global degrees of freedom in a vector or matrix to about 2 billion. In order to solve larger problems, there are two options: 1. Building hypre with '--enable-bigint' defines the local and global indices to be 64-bit. This is convenient, but requires more memory than necessary. 2. Building hypre with '--enable-mixedint' defines the local indiced to be 32-bit, while using a 64-bit storage for global indices. This option is currently tested only in ex1p, and may not work in more general settings.