{ "cells": [ { "cell_type": "markdown", "id": "4cf645cf-a403-40f7-b9c7-c4dd5eed647b", "metadata": {}, "source": [ "# Brief Intro to Lmod on CyberGISX\n", "\n", "**Author:** Alexander Michels\n", "\n", "This notebook is a very brief explanation of what Lmod is and how to use it on CyberGISX." ] }, { "cell_type": "markdown", "id": "895b9258-e807-4c5b-a711-b69c51a1f962", "metadata": {}, "source": [ "## What is Lmod?\n", "\n", ">Lmod is an implementation of Environment Modules, much of what is said here is true for any environment modules system but there are many features which are unique to Lmod. Environment Modules provide a convenient way to dynamically change the users’ environment through modulefiles. This includes easily adding or removing directories to the PATH environment variable.\n", "\n", "- [https://lmod.readthedocs.io/en/latest/](https://lmod.readthedocs.io/en/latest/)\n", "\n", "Basically, Lmod is software that allows us have multiple co-existing versions of software and simply \"load\" and \"unload\" them at will!\n", "\n", "While Lmod does provide a command line interface (CLI), we recommend you DO NOT use Jupyter `!` commands to interact with Lmod because it won't display outputs correctly and often will not accomplish what you are hoping to. Lmod works by manipulating environmental variables like PATH and `!` commands spawn subprocesses so your environment will not \"remember\" what happened in them. \n", "\n", "As an example of this \"forgetfulness\" of `!` let's try to set out name as an environmental variable and then in the same cell try to print the variable:" ] }, { "cell_type": "code", "execution_count": 1, "id": "275b20c7-7dea-4ac0-9218-45a2b9dc5f4d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "!export MYNAME=\"Alex\"\n", "!echo $MYNAME" ] }, { "cell_type": "markdown", "id": "536fb052-ac92-4d6e-826e-1262c410d697", "metadata": {}, "source": [ "Similarly, a `module load xxx` won't be remembered if run through an `!` command.\n", "\n", "Luckily, Lmod comes with a very helpful Python package to interact with it! Some of the [details can be found in our Knowledge Base](https://cybergisxhub.cigi.illinois.edu/knowledge-base/working-with-notebooks/integrating-python-and-the-jupyter-notebook-environment/software-packages-and-libraries-available-within-cybergis-jupyter/) and [the implementation is very simple if you want to check it out](https://github.com/TACC/Lmod/blob/main/init/env_modules_python.py.in).\n", "\n", "Basically after we load it, instead of running `!module list` we will run `module(\"list\")`. [More details on the module command syntax can be found here](https://lmod.readthedocs.io/en/latest/010_user.html). Let's load the Python package and try it out:" ] }, { "cell_type": "code", "execution_count": 2, "id": "67a3c6ba-21d6-4089-8467-113423dd7b9b", "metadata": {}, "outputs": [], "source": [ "import sys, os # we need this to add the $MODULESHOME path to our Python path\n", "sys.path.insert(0, os.path.join(os.environ['MODULESHOME'], \"init\")) # add to the python path\n", "from env_modules_python import module # import package" ] }, { "cell_type": "code", "execution_count": 3, "id": "9ac3c6cb-fc85-4195-8071-68962d083c84", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n", "Currently Loaded Modules:\n", " 1) GCCcore/8.3.0 43) LAME/3.100-GCCcore-8.3.0\n", " 2) zlib/1.2.11-GCCcore-8.3.0 44) x265/3.2-GCCcore-8.3.0\n", " 3) binutils/2.32-GCCcore-8.3.0 45) util-linux/2.34-GCCcore-8.3.0\n", " 4) GCC/8.3.0 46) fontconfig/2.13.1-GCCcore-8.3.0\n", " 5) numactl/2.0.12-GCCcore-8.3.0 47) xorg-macros/1.19.2-GCCcore-8.3.0\n", " 6) XZ/5.2.4-GCCcore-8.3.0 48) X11/20190717-GCCcore-8.3.0\n", " 7) libxml2/2.9.9-GCCcore-8.3.0 49) FriBidi/1.0.5-GCCcore-8.3.0\n", " 8) libpciaccess/0.14-GCCcore-8.3.0 50) FFmpeg/4.2.1-GCCcore-8.3.0\n", " 9) hwloc/1.11.12-GCCcore-8.3.0 51) pixman/0.38.4-GCCcore-8.3.0\n", " 10) OpenMPI/3.1.4-GCC-8.3.0 52) libffi/3.2.1-GCCcore-8.3.0\n", " 11) OpenBLAS/0.3.7-GCC-8.3.0 53) GLib/2.62.0-GCCcore-8.3.0\n", " 12) gompi/2019b 54) cairo/1.16.0-GCCcore-8.3.0\n", " 13) FFTW/3.3.8-gompi-2019b 55) GMP/6.1.2-GCCcore-8.3.0\n", " 14) ScaLAPACK/2.0.2-gompi-2019b 56) nettle/3.5.1-GCCcore-8.3.0\n", " 15) foss/2019b 57) libdrm/2.4.99-GCCcore-8.3.0\n", " 16) bzip2/1.0.8-GCCcore-8.3.0 58) LLVM/9.0.0-GCCcore-8.3.0\n", " 17) ncurses/6.1-GCCcore-8.3.0 59) libunwind/1.3.1-GCCcore-8.3.0\n", " 18) gettext/0.20.1-GCCcore-8.3.0 60) Mesa/19.1.7-GCCcore-8.3.0\n", " 19) libpng/1.6.37-GCCcore-8.3.0 61) libGLU/9.0.1-GCCcore-8.3.0\n", " 20) libreadline/8.0-GCCcore-8.3.0 62) gzip/1.10-GCCcore-8.3.0\n", " 21) Szip/2.1.1-GCCcore-8.3.0 63) lz4/1.9.2-GCCcore-8.3.0\n", " 22) HDF5/1.10.5-gompi-2019b 64) zstd/1.4.4-GCCcore-8.3.0\n", " 23) cURL/7.66.0-GCCcore-8.3.0 65) GRASS/7.8.3-foss-2019b\n", " 24) netCDF/4.7.1-gompi-2019b 66) MPICH/3.3.2-GCC-8.3.0\n", " 25) expat/2.2.7-GCCcore-8.3.0 67) RHESSysEastCoast/7.2.0-foss-2019b\n", " 26) GEOS/3.8.0-GCC-8.3.0 68) netCDF-Fortran/4.5.2-gompi-2019b\n", " 27) Tcl/8.6.9-GCCcore-8.3.0 69) SUMMA/3.0.3-foss-2019b\n", " 28) SQLite/3.29.0-GCCcore-8.3.0 70) TauDEM/5.3.8-foss-2019b\n", " 29) NASM/2.14.02-GCCcore-8.3.0 71) WRF/4.2.1-foss-2019b-dmpar\n", " 30) libjpeg-turbo/2.0.3-GCCcore-8.3.0 72) WPS/4.2-foss-2019b-dmpar\n", " 31) JasPer/2.0.14-GCCcore-8.3.0 73) find_inlets/20191210-foss-2019b\n", " 32) LibTIFF/4.0.10-GCCcore-8.3.0 74) Boost/1.71.0-gompi-2019b\n", " 33) PCRE/8.43-GCCcore-8.3.0 75) Xvfb/1.20.8-GCCcore-8.3.0\n", " 34) PROJ/6.2.1-GCCcore-8.3.0 76) protozero/1.7.0-GCCcore-8.3.0\n", " 35) libgeotiff/1.5.1-GCCcore-8.3.0 77) sparsehash/2.0.3-GCCcore-8.3.0\n", " 36) libtirpc/1.2.6-GCCcore-8.3.0 78) libosmium/2.15.6-foss-2019b\n", " 37) HDF/4.2.14-GCCcore-8.3.0 79) SoPlex/4.0.1-foss-2019b\n", " 38) GDAL/3.0.2-foss-2019b 80) PostgreSQL/12.4-GCCcore-8.3.0\n", " 39) FreeXL/1.0.5-GCCcore-8.3.0 81) protobuf/3.10.0-GCCcore-8.3.0\n", " 40) libspatialite/4.3.0a-GCC-8.3.0 82) protobuf-c/1.3.3-GCCcore-8.3.0\n", " 41) freetype/2.10.1-GCCcore-8.3.0 83) PostGIS/3.1.2-foss-2019b\n", " 42) x264/20190925-GCCcore-8.3.0 84) cybergisx/0.8.0\n", "\n", " \n", "\n", "\n" ] } ], "source": [ "output = module(\"list\") # the output is the return code and stderr output, we catch it to make the notebook cleaner" ] }, { "cell_type": "markdown", "id": "39d379d7-f899-40ea-97b4-8359fbedfd1e", "metadata": {}, "source": [ "The `list` command displays the currently loaded modules. To view the available modules, we can use `avail`:" ] }, { "cell_type": "code", "execution_count": 4, "id": "2dc0d35f-24a0-4e38-858c-d7ce2021a519", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n", "-------------- /cvmfs/cybergis.illinois.edu/software/metamodules ---------------\n", " cjw/2021-09 cjw/2022-03 (D) cybergisx/0.8.0 (L) iguide/0.1.0\n", "\n", "--------- /cvmfs/cybergis.illinois.edu/software/easybuild/modules/all ----------\n", " ANTLR/2.7.7-GCCcore-8.3.0-Java-11\n", " Anaconda3/2020.11\n", " Anaconda3/2022.05 (D)\n", " Armadillo/11.4.3-foss-2022b\n", " Autoconf/2.69-GCCcore-8.3.0\n", " Autoconf/2.71-GCCcore-12.2.0 (D)\n", " Automake/1.16.1-GCCcore-8.3.0\n", " Automake/1.16.5-GCCcore-12.2.0 (D)\n", " Autotools/20180311-GCCcore-8.3.0\n", " Autotools/20220317-GCCcore-12.2.0 (D)\n", " BLIS/0.9.0-GCC-12.2.0\n", " Bison/3.3.2-GCCcore-8.3.0\n", " Bison/3.3.2\n", " Bison/3.5.3-GCCcore-9.3.0\n", " Bison/3.8.2-GCCcore-12.2.0\n", " Bison/3.8.2 (D)\n", " Boost/1.71.0-gompi-2019b (L)\n", " Boost/1.81.0-GCC-12.2.0 (D)\n", " Brotli/1.0.9-GCCcore-12.2.0\n", " Brunsli/0.1-GCCcore-12.2.0\n", " CDO/1.9.10-gompi-2019b\n", " CFITSIO/4.2.0-GCCcore-12.2.0\n", " CMake/3.15.3-GCCcore-8.3.0\n", " CMake/3.24.3-GCCcore-12.2.0 (D)\n", " Clang/9.0.1-GCCcore-8.3.0\n", " DB/18.1.32-GCCcore-8.3.0\n", " DB/18.1.40-GCCcore-12.2.0 (D)\n", " DBus/1.13.12-GCCcore-8.3.0\n", " Doxygen/1.8.16-GCCcore-8.3.0\n", " Doxygen/1.9.5-GCCcore-12.2.0 (D)\n", " ESMF/8.0.0-foss-2019b\n", " EasyBuild/4.7.1\n", " Eigen/3.4.0-GCCcore-12.2.0\n", " FFTW.MPI/3.3.10-gompi-2022b\n", " FFTW/3.3.8-gompi-2019b (L)\n", " FFTW/3.3.10-GCC-12.2.0 (D)\n", " FFmpeg/4.2.1-GCCcore-8.3.0 (L)\n", " FlexiBLAS/3.2.1-GCC-12.2.0\n", " FreeXL/1.0.5-GCCcore-8.3.0 (L)\n", " FriBidi/1.0.5-GCCcore-8.3.0 (L)\n", " GCC/8.3.0 (L)\n", " GCC/9.3.0\n", " GCC/12.2.0 (D)\n", " GCCcore/8.3.0 (L)\n", " GCCcore/9.3.0\n", " GCCcore/12.2.0 (D)\n", " GDAL/3.0.2-foss-2019b (L)\n", " GDAL/3.6.2-foss-2022b (D)\n", " GEOS/3.8.0-GCC-8.3.0 (L)\n", " GEOS/3.11.1-GCC-12.2.0 (D)\n", " GLib/2.62.0-GCCcore-8.3.0 (L)\n", " GMP/6.1.2-GCCcore-8.3.0 (L)\n", " GMP/6.2.1-GCCcore-12.2.0 (D)\n", " GObject-Introspection/1.63.1-GCCcore-8.3.0-Python-3.7.4\n", " GRASS/7.8.3-foss-2019b (L)\n", " GSL/2.6-GCC-8.3.0\n", " GTS/0.7.6-foss-2019b\n", " Gdk-Pixbuf/2.38.2-GCCcore-8.3.0\n", " Ghostscript/9.50-GCCcore-8.3.0\n", " Graphviz/2.42.2-foss-2019b\n", " HDF/4.2.14-GCCcore-8.3.0 (L)\n", " HDF/4.2.15-GCCcore-12.2.0 (D)\n", " HDF5/1.10.5-gompi-2019b (L)\n", " HDF5/1.14.0-gompi-2022b (D)\n", " HarfBuzz/2.6.4-GCCcore-8.3.0\n", " Highway/1.0.3-GCCcore-12.2.0\n", " ICU/64.2-GCCcore-8.3.0\n", " ICU/72.1-GCCcore-12.2.0 (D)\n", " Imath/3.1.6-GCCcore-12.2.0\n", " JasPer/2.0.14-GCCcore-8.3.0 (L)\n", " JasPer/4.0.0-GCCcore-12.2.0 (D)\n", " Java/11.0.2 (11)\n", " LAME/3.100-GCCcore-8.3.0 (L)\n", " LERC/4.0.0-GCCcore-12.2.0\n", " LLVM/9.0.0-GCCcore-8.3.0 (L)\n", " LibTIFF/4.0.10-GCCcore-8.3.0 (L)\n", " LibTIFF/4.4.0-GCCcore-12.2.0 (D)\n", " M4/1.4.18-GCCcore-8.3.0\n", " M4/1.4.18-GCCcore-9.3.0\n", " M4/1.4.18\n", " M4/1.4.19-GCCcore-12.2.0\n", " M4/1.4.19 (D)\n", " MPICH/3.3.2-GCC-8.3.0 (L)\n", " MPICH/3.3.2-GCC-9.3.0 (D)\n", " Mako/1.1.0-GCCcore-8.3.0\n", " Mesa/19.1.7-GCCcore-8.3.0 (L)\n", " Meson/0.51.2-GCCcore-8.3.0-Python-3.7.4\n", " Meson/0.59.1-GCCcore-8.3.0-Python-3.7.4\n", " Meson/0.64.0-GCCcore-12.2.0 (D)\n", " NASM/2.14.02-GCCcore-8.3.0 (L)\n", " NASM/2.15.05-GCCcore-12.2.0 (D)\n", " NCL/6.6.2-foss-2019b\n", " NCO/4.9.3-foss-2019b\n", " NSPR/4.21-GCCcore-8.3.0\n", " NSS/3.45-GCCcore-8.3.0\n", " Ninja/1.9.0-GCCcore-8.3.0\n", " Ninja/1.11.1-GCCcore-12.2.0 (D)\n", " OpenBLAS/0.3.7-GCC-8.3.0 (L)\n", " OpenBLAS/0.3.21-GCC-12.2.0 (D)\n", " OpenEXR/3.1.5-GCCcore-12.2.0\n", " OpenMPI/3.1.4-GCC-8.3.0 (L)\n", " OpenMPI/4.1.4-GCC-12.2.0 (D)\n", " OpenSSL/1.1\n", " PCRE/8.43-GCCcore-8.3.0 (L)\n", " PCRE/8.45-GCCcore-12.2.0 (D)\n", " PCRE2/10.33-GCCcore-8.3.0\n", " PCRE2/10.40-GCCcore-12.2.0 (D)\n", " PMIx/4.2.2-GCCcore-12.2.0\n", " PROJ/6.2.1-GCCcore-8.3.0 (L)\n", " PROJ/9.1.1-GCCcore-12.2.0 (D)\n", " Pango/1.44.7-GCCcore-8.3.0\n", " Perl/5.30.0-GCCcore-8.3.0-minimal\n", " Perl/5.30.0-GCCcore-8.3.0\n", " Perl/5.36.0-GCCcore-12.2.0 (D)\n", " PostGIS/3.1.2-foss-2019b (L)\n", " PostgreSQL/12.4-GCCcore-8.3.0 (L)\n", " Python/2.7.16-GCCcore-8.3.0\n", " Python/3.7.4-GCCcore-8.3.0\n", " Python/3.10.8-GCCcore-12.2.0-bare\n", " Python/3.10.8-GCCcore-12.2.0 (D)\n", " Qhull/2020.2-GCCcore-12.2.0\n", " Qt5/5.13.1-GCCcore-8.3.0\n", " RHESSysEastCoast/7.2.0-foss-2019b (L)\n", " Rust/1.65.0-GCCcore-12.2.0\n", " SQLite/3.29.0-GCCcore-8.3.0 (L)\n", " SQLite/3.39.4-GCCcore-12.2.0 (D)\n", " SUMMA/3.0.3-foss-2019b (L)\n", " SWIG/4.0.1-GCCcore-8.3.0\n", " ScaLAPACK/2.0.2-gompi-2019b (L)\n", " ScaLAPACK/2.2.0-gompi-2022b-fb (D)\n", " SciPy-bundle/2023.02-gfbf-2022b\n", " SoPlex/4.0.1-foss-2019b (L)\n", " Szip/2.1.1-GCCcore-8.3.0 (L)\n", " Szip/2.1.1-GCCcore-12.2.0 (D)\n", " TauDEM/5.3.8-foss-2019b (L)\n", " TauDEM/5.3.9-dev-foss-2019b (D)\n", " Tcl/8.6.9-GCCcore-8.3.0 (L)\n", " Tcl/8.6.12-GCCcore-12.2.0 (D)\n", " UCC/1.1.0-GCCcore-12.2.0\n", " UCX/1.13.1-GCCcore-12.2.0\n", " UDUNITS/2.2.26-GCCcore-8.3.0\n", " UnZip/6.0-GCCcore-12.2.0\n", " WPS/4.2-foss-2019b-dmpar (L)\n", " WRF/4.2.1-foss-2019b-dmpar (L)\n", " X11/20190717-GCCcore-8.3.0 (L)\n", " XZ/5.2.4-GCCcore-8.3.0 (L)\n", " XZ/5.2.7-GCCcore-12.2.0 (D)\n", " Xerces-C++/3.2.4-GCCcore-12.2.0\n", " Xvfb/1.20.8-GCCcore-8.3.0 (L)\n", " YAXT/0.6.2-gompi-2019b\n", " Yasm/1.3.0-GCCcore-8.3.0\n", " arpack-ng/3.8.0-foss-2022b\n", " binutils/2.32-GCCcore-8.3.0 (L)\n", " binutils/2.32\n", " binutils/2.34-GCCcore-9.3.0\n", " binutils/2.34\n", " binutils/2.39-GCCcore-12.2.0\n", " binutils/2.39 (D)\n", " bzip2/1.0.8-GCCcore-8.3.0 (L)\n", " bzip2/1.0.8-GCCcore-12.2.0 (D)\n", " cURL/7.66.0-GCCcore-8.3.0 (L)\n", " cURL/7.86.0-GCCcore-12.2.0 (D)\n", " cairo/1.16.0-GCCcore-8.3.0 (L)\n", " double-conversion/3.1.4-GCCcore-8.3.0\n", " ecCodes/2.17.0-gompi-2019b\n", " expat/2.2.7-GCCcore-8.3.0 (L)\n", " expat/2.4.9-GCCcore-12.2.0 (D)\n", " find_inlets/20191210-foss-2019b (L)\n", " flex/2.6.4-GCCcore-8.3.0\n", " flex/2.6.4-GCCcore-9.3.0\n", " flex/2.6.4-GCCcore-12.2.0\n", " flex/2.6.4 (D)\n", " fontconfig/2.13.1-GCCcore-8.3.0 (L)\n", " foss/2019b (L)\n", " foss/2022b (D)\n", " freetype/2.10.1-GCCcore-8.3.0 (L)\n", " g2clib/1.6.0-GCCcore-8.3.0\n", " g2lib/3.1.0-GCCcore-8.3.0\n", " gettext/0.19.8.1\n", " gettext/0.20.1-GCCcore-8.3.0 (L)\n", " gettext/0.21.1-GCCcore-12.2.0\n", " gettext/0.21.1 (D)\n", " gfbf/2022b\n", " giflib/5.2.1-GCCcore-12.2.0\n", " git/2.38.1-GCCcore-12.2.0-nodocs\n", " gompi/2019b (L)\n", " gompi/2022b (D)\n", " googletest/1.12.1-GCCcore-12.2.0\n", " gperf/3.1-GCCcore-8.3.0\n", " groff/1.22.4-GCCcore-8.3.0\n", " groff/1.22.4-GCCcore-12.2.0 (D)\n", " gzip/1.10-GCCcore-8.3.0 (L)\n", " gzip/1.12-GCCcore-12.2.0 (D)\n", " help2man/1.47.8-GCCcore-8.3.0\n", " help2man/1.47.12-GCCcore-9.3.0\n", " help2man/1.49.2-GCCcore-12.2.0 (D)\n", " hwloc/1.11.12-GCCcore-8.3.0 (L)\n", " hwloc/2.8.0-GCCcore-12.2.0 (D)\n", " hypothesis/6.68.2-GCCcore-12.2.0\n", " intltool/0.51.0-GCCcore-8.3.0\n", " jbigkit/2.1-GCCcore-12.2.0\n", " json-c/0.16-GCCcore-12.2.0\n", " libGLU/9.0.1-GCCcore-8.3.0 (L)\n", " libarchive/3.6.1-GCCcore-12.2.0\n", " libdap/3.20.6-GCCcore-8.3.0\n", " libdeflate/1.15-GCCcore-12.2.0\n", " libdrm/2.4.99-GCCcore-8.3.0 (L)\n", " libevent/2.1.11-GCCcore-8.3.0\n", " libevent/2.1.12-GCCcore-12.2.0 (D)\n", " libfabric/1.16.1-GCCcore-12.2.0\n", " libffi/3.2.1-GCCcore-8.3.0 (L)\n", " libffi/3.4.4-GCCcore-12.2.0 (D)\n", " libgd/2.2.5-GCCcore-8.3.0\n", " libgeotiff/1.5.1-GCCcore-8.3.0 (L)\n", " libgeotiff/1.7.1-GCCcore-12.2.0 (D)\n", " libiconv/1.16-GCCcore-8.3.0\n", " libiconv/1.17-GCCcore-12.2.0 (D)\n", " libjpeg-turbo/2.0.3-GCCcore-8.3.0 (L)\n", " libjpeg-turbo/2.1.4-GCCcore-12.2.0 (D)\n", " libosmium/2.15.6-foss-2019b (L)\n", " libpciaccess/0.14-GCCcore-8.3.0 (L)\n", " libpciaccess/0.17-GCCcore-12.2.0 (D)\n", " libpng/1.6.37-GCCcore-8.3.0 (L)\n", " libpng/1.6.38-GCCcore-12.2.0 (D)\n", " libreadline/8.0-GCCcore-8.3.0 (L)\n", " libreadline/8.2-GCCcore-12.2.0 (D)\n", " libspatialite/4.3.0a-GCC-8.3.0 (L)\n", " libtirpc/1.2.6-GCCcore-8.3.0 (L)\n", " libtirpc/1.3.3-GCCcore-12.2.0 (D)\n", " libtool/2.4.6-GCCcore-8.3.0\n", " libtool/2.4.7-GCCcore-12.2.0 (D)\n", " libunwind/1.3.1-GCCcore-8.3.0 (L)\n", " libxml2/2.9.9-GCCcore-8.3.0 (L)\n", " libxml2/2.10.3-GCCcore-12.2.0 (D)\n", " lz4/1.9.2-GCCcore-8.3.0 (L)\n", " lz4/1.9.4-GCCcore-12.2.0 (D)\n", " makedepend/1.0.6-GCCcore-8.3.0\n", " makeinfo/6.7-GCCcore-8.3.0-minimal\n", " ncurses/6.0\n", " ncurses/6.1-GCCcore-8.3.0 (L)\n", " ncurses/6.3-GCCcore-12.2.0\n", " ncurses/6.3 (D)\n", " netCDF-C++4/4.3.1-gompi-2019b\n", " netCDF-Fortran/4.5.2-gompi-2019b (L)\n", " netCDF/4.7.1-gompi-2019b (L)\n", " netCDF/4.9.0-gompi-2022b (D)\n", " nettle/3.5.1-GCCcore-8.3.0 (L)\n", " nlohmann_json/3.11.2-GCCcore-12.2.0\n", " numactl/2.0.12-GCCcore-8.3.0 (L)\n", " numactl/2.0.16-GCCcore-12.2.0 (D)\n", " pixman/0.38.4-GCCcore-8.3.0 (L)\n", " pkg-config/0.29.2-GCCcore-8.3.0\n", " pkgconf/1.8.0\n", " pkgconf/1.9.3-GCCcore-12.2.0 (D)\n", " protobuf-c/1.3.3-GCCcore-8.3.0 (L)\n", " protobuf/3.10.0-GCCcore-8.3.0 (L)\n", " protozero/1.7.0-GCCcore-8.3.0 (L)\n", " pybind11/2.10.3-GCCcore-12.2.0\n", " re2c/1.2.1-GCCcore-8.3.0\n", " snappy/1.1.7-GCCcore-8.3.0\n", " sparsehash/2.0.3-GCCcore-8.3.0 (L)\n", " tcsh/6.22.02-GCCcore-8.3.0\n", " time/1.9-GCCcore-8.3.0\n", " util-linux/2.34-GCCcore-8.3.0 (L)\n", " x264/20190925-GCCcore-8.3.0 (L)\n", " x265/3.2-GCCcore-8.3.0 (L)\n", " xorg-macros/1.19.2-GCCcore-8.3.0 (L)\n", " xorg-macros/1.19.3-GCCcore-12.2.0 (D)\n", " xproto/7.0.31-GCCcore-8.3.0\n", " zlib/1.2.11-GCCcore-8.3.0 (L)\n", " zlib/1.2.11-GCCcore-9.3.0\n", " zlib/1.2.11\n", " zlib/1.2.12-GCCcore-12.2.0\n", " zlib/1.2.12 (D)\n", " zstd/1.4.4-GCCcore-8.3.0 (L)\n", " zstd/1.5.2-GCCcore-12.2.0 (D)\n", "\n", "-------------------- /usr/share/lmod/lmod/modulefiles/Core ---------------------\n", " lmod settarg\n", "\n", " Where:\n", " Aliases: Aliases exist: foo/1.2.3 (1.2) means that \"module load foo/1.2\" will load foo/1.2.3\n", " D: Default Module\n", " L: Module is loaded\n", "\n", "If the avail list is too long consider trying:\n", "\n", "\"module --default avail\" or \"ml -d av\" to just list the default modules.\n", "\"module overview\" or \"ml ov\" to display the number of modules for each name.\n", "\n", "Use \"module spider\" to find all possible modules and extensions.\n", "Use \"module keyword key1 key2 ...\" to search for all possible modules matching\n", "any of the \"keys\".\n", "\n", "\n", "\n" ] } ], "source": [ "output = module(\"avail\") # the output is the return code and stderr output, we catch it to make the notebook cleaner" ] }, { "cell_type": "markdown", "id": "7f56259e-5dda-4b71-8bae-0acaa1bf1c9f", "metadata": {}, "source": [ "Lastly, we are going to try to load a different version of GDAL and demonstrate that it is in fact a new version. First, let's check the version we have:" ] }, { "cell_type": "code", "execution_count": 5, "id": "cd9dfd7f-7032-4247-888e-b6680b08b3b6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GDAL 3.3.1, released 2021/06/28\n" ] } ], "source": [ "!gdalinfo --version" ] }, { "cell_type": "markdown", "id": "a58eeab5-aa01-4742-b00f-5f734fa2550d", "metadata": {}, "source": [ "Currently, we are using the GDAL version 3.3.1 from the Anaconda environment, but suppose we need to use another version available through Lmod like 3.6.2 (see `GDAL/3.6.2-foss-2022b` in the `avail` output). To use that, we simply:" ] }, { "cell_type": "code", "execution_count": 6, "id": "39d66d03-4513-41bc-a5c3-a72b8532b3e6", "metadata": { "scrolled": true, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n", "The following have been reloaded with a version change:\n", " 1) Boost/1.71.0-gompi-2019b => Boost/1.81.0-GCC-12.2.0\n", " 2) FFTW/3.3.8-gompi-2019b => FFTW/3.3.10-GCC-12.2.0\n", " 3) GCC/8.3.0 => GCC/12.2.0\n", " 4) GCCcore/8.3.0 => GCCcore/12.2.0\n", " 5) GDAL/3.0.2-foss-2019b => GDAL/3.6.2-foss-2022b\n", " 6) GEOS/3.8.0-GCC-8.3.0 => GEOS/3.11.1-GCC-12.2.0\n", " 7) GMP/6.1.2-GCCcore-8.3.0 => GMP/6.2.1-GCCcore-12.2.0\n", " 8) HDF/4.2.14-GCCcore-8.3.0 => HDF/4.2.15-GCCcore-12.2.0\n", " 9) HDF5/1.10.5-gompi-2019b => HDF5/1.14.0-gompi-2022b\n", " 10) JasPer/2.0.14-GCCcore-8.3.0 => JasPer/4.0.0-GCCcore-12.2.0\n", " 11) LibTIFF/4.0.10-GCCcore-8.3.0 => LibTIFF/4.4.0-GCCcore-12.2.0\n", " 12) NASM/2.14.02-GCCcore-8.3.0 => NASM/2.15.05-GCCcore-12.2.0\n", " 13) OpenBLAS/0.3.7-GCC-8.3.0 => OpenBLAS/0.3.21-GCC-12.2.0\n", " 14) OpenMPI/3.1.4-GCC-8.3.0 => OpenMPI/4.1.4-GCC-12.2.0\n", " 15) PCRE/8.43-GCCcore-8.3.0 => PCRE/8.45-GCCcore-12.2.0\n", " 16) PROJ/6.2.1-GCCcore-8.3.0 => PROJ/9.1.1-GCCcore-12.2.0\n", " 17) SQLite/3.29.0-GCCcore-8.3.0 => SQLite/3.39.4-GCCcore-12.2.0\n", " 18) ScaLAPACK/2.0.2-gompi-2019b => ScaLAPACK/2.2.0-gompi-2022b-fb\n", " 19) Szip/2.1.1-GCCcore-8.3.0 => Szip/2.1.1-GCCcore-12.2.0\n", " 20) Tcl/8.6.9-GCCcore-8.3.0 => Tcl/8.6.12-GCCcore-12.2.0\n", " 21) XZ/5.2.4-GCCcore-8.3.0 => XZ/5.2.7-GCCcore-12.2.0\n", " 22) binutils/2.32-GCCcore-8.3.0 => binutils/2.39-GCCcore-12.2.0\n", " 23) bzip2/1.0.8-GCCcore-8.3.0 => bzip2/1.0.8-GCCcore-12.2.0\n", " 24) cURL/7.66.0-GCCcore-8.3.0 => cURL/7.86.0-GCCcore-12.2.0\n", " 25) expat/2.2.7-GCCcore-8.3.0 => expat/2.4.9-GCCcore-12.2.0\n", " 26) foss/2019b => foss/2022b\n", " 27) gompi/2019b => gompi/2022b\n", " 28) gzip/1.10-GCCcore-8.3.0 => gzip/1.12-GCCcore-12.2.0\n", " 29) hwloc/1.11.12-GCCcore-8.3.0 => hwloc/2.8.0-GCCcore-12.2.0\n", " 30) libffi/3.2.1-GCCcore-8.3.0 => libffi/3.4.4-GCCcore-12.2.0\n", " 31) libgeotiff/1.5.1-GCCcore-8.3.0 => libgeotiff/1.7.1-GCCcore-12.2.0\n", " 32) libjpeg-turbo/2.0.3-GCCcore-8.3.0 => libjpeg-turbo/2.1.4-GCCcore-12.2.0\n", " 33) libpciaccess/0.14-GCCcore-8.3.0 => libpciaccess/0.17-GCCcore-12.2.0\n", " 34) libpng/1.6.37-GCCcore-8.3.0 => libpng/1.6.38-GCCcore-12.2.0\n", " 35) libreadline/8.0-GCCcore-8.3.0 => libreadline/8.2-GCCcore-12.2.0\n", " 36) libtirpc/1.2.6-GCCcore-8.3.0 => libtirpc/1.3.3-GCCcore-12.2.0\n", " 37) libxml2/2.9.9-GCCcore-8.3.0 => libxml2/2.10.3-GCCcore-12.2.0\n", " 38) lz4/1.9.2-GCCcore-8.3.0 => lz4/1.9.4-GCCcore-12.2.0\n", " 39) ncurses/6.1-GCCcore-8.3.0 => ncurses/6.3-GCCcore-12.2.0\n", " 40) netCDF/4.7.1-gompi-2019b => netCDF/4.9.0-gompi-2022b\n", " 41) numactl/2.0.12-GCCcore-8.3.0 => numactl/2.0.16-GCCcore-12.2.0\n", " 42) zlib/1.2.11-GCCcore-8.3.0 => zlib/1.2.12-GCCcore-12.2.0\n", " 43) zstd/1.4.4-GCCcore-8.3.0 => zstd/1.5.2-GCCcore-12.2.0\n", "\n", "\n" ] } ], "source": [ "output = module(\"load\", \"GDAL/3.6.2-foss-2022b\")" ] }, { "cell_type": "markdown", "id": "7d1ab592-7c9b-4c7b-9a58-06ce622ea31c", "metadata": {}, "source": [ "The output tells us that many packages have changed versions to accomdate this switch!\n", "\n", "To verify, let's check the version again:" ] }, { "cell_type": "code", "execution_count": 7, "id": "64b8908d-d63c-4f4f-adc8-5e0f3cbaa67d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GDAL 3.6.2, released 2023/01/02\n" ] } ], "source": [ "!gdalinfo --version" ] }, { "cell_type": "markdown", "id": "101f3077-c4c3-4695-825a-6f319db8f7ad", "metadata": {}, "source": [ "## Learn More\n", "\n", "We have a paper under submission about the architecture of CyberGISX, but in the meantime check out:\n", "\n", "* [Lmod: Read the Docs](https://lmod.readthedocs.io/en/latest/index.html)\n", "* EasyBuild (how the listed software is installed):\n", " * [Documentation](https://docs.easybuild.io/)\n", " * [List of supported software](https://docs.easybuild.io/version-specific/supported-software/)" ] }, { "cell_type": "code", "execution_count": null, "id": "aacfe2be-0bf7-43e9-a954-dae8067aa873", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" } }, "nbformat": 4, "nbformat_minor": 5 }