Skip to content

Installation ​

ADAM is provided as source files and must be compiled to produce executables.

ADAM has two types of dependencies: source libraries fetched by FoBiS (into src/third_party) and external libraries that must be installed on the system.

Supported Compilers ​

CompilerStatus
NVIDIA HPC SDK, nvfortran 26.9 (HPC-X Open MPI bundled)GPU (OpenACC, *-fnl-nvf) and CPU builds; see Install NVIDIA HPC SDK
GNU gfortran 16.2.0 + Open MPI 5.0.11CPU builds (*-cpu-gnu); CI uses Ubuntu 24.04's gfortran 14 (varset ci_gnu)
AMD flang (ROCm)OpenMP offload builds (*-omp-amd)
Intel oneAPI (ifx)OpenMP offload templates, experimental

These are the versions the regression suites are run with. Fortran .mod files work only with the compiler that wrote them, so HDF5 must be built with the same compiler and MPI as ADAM, and a build must start clean after a compiler switch.

Obtain ADAM ​

Clone the repository, then fetch source dependencies with FoBiS:

bash
git clone https://github.com/szaghi/adam
cd adam
fobis fetch

fobis fetch reads src/third_party/.deps_config.ini and downloads all required source libraries into src/third_party/.

ADAM tracks the newest main/master of every dependency: no commit is pinned, and the fobos.lock that fobis fetch writes stays local (it is gitignored). A fresh clone therefore gets the current heads, but an existing checkout does not move by itself. Bring it to the current heads with:

bash
fobis fetch --update

then rebuild clean, since the dependencies' module files change.

Alternatively, use the provided install.sh script which automates both steps:

bash
# clone + fetch deps + build
scripts/install.sh --download git --build fobis --mode <mode>

# download a release tarball instead of cloning
scripts/install.sh --download wget --build fobis --mode <mode> --tag v1.0.0

Run scripts/install.sh --help for the full option list.

Compile ADAM ​

The preferred build system is FoBiS — the CLI binary is fobis (version 3.8+); flags use double-dash long form. Install it from PyPI:

bash
pip install FoBiS.py

(FoBiS.py is the PyPI package name; it installs the fobis command.)

The repository root structure:

.
├── exe/
├── fobos
├── LICENSE.lgpl3.md
├── LICENSE.mit.md
├── README.md
├── scripts/
└── src/

List all available build modes:

bash
fobis build --lmodes

Modes are named <app>-<backend>-<compiler>[-debug], e.g. prism-cpu-gnu, flume-fnl-nvf, flume-cpu-gnu-debug. Every mode needs a varset that points it at the HDF5 build of its toolchain (see Varsets); the *-nvf modes also take the GPU compute capability from it:

bash
fobis build --mode prism-cpu-gnu                       # the default varset, local_gnu
fobis build --mode flume-fnl-nvf --varset local_nvf    # an NVF mode always needs an NVF varset

Install NVIDIA HPC SDK ​

Download NVIDIA HPC SDK 26.9 from the NVIDIA HPC SDK downloads page, unpack it and run its install script; the default installation path is /opt/nvidia/hpc_sdk/Linux_x86_64/26.9.

Pick the CUDA bundle your GPU driver can run. A bundle whose CUDA is newer than the driver supports fails at the first kernel launch: on a node limited to CUDA 12.9, use the cuda_multi bundle and build with -gpu=cuda12.9.

Then put the compilers and the bundled MPI on PATH, with the SDK's module files:

bash
module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/26.9

or directly:

bash
NVHPC=/opt/nvidia/hpc_sdk/Linux_x86_64/26.9
export PATH=$NVHPC/compilers/bin:$NVHPC/comm_libs/mpi/bin:$PATH
export MANPATH=$NVHPC/compilers/man:$MANPATH

In a non-interactive shell (a batch job, a script) module load may not populate PATH; export it directly there.

Known issues: NVHPC 26.9 with HPC-X 2.50 ​

Both were reported on Ubuntu 24.04 under WSL2 with the SDK's bundled HPC-X 2.50 (Open MPI 5) in #96.

  • prterun needs libnl-route-3. Without it every mpirun stops with libnl-route-3.so.200: cannot open shared object file. Install it once:

    bash
    sudo apt-get install libnl-route-3-200
  • Debug modes die in MPI_Init. The *-fnl-nvf-debug modes (template-nvf-debug) compile and link with -Ktrap=fp, and a floating-point exception raised inside UCX's ucp_worker_create during MPI_Init traps with signal 8, before any ADAM code runs. Until that changes, debug with a local copy of the mode without -Ktrap=fp; the bounds, pointer and stack checks (-Mbounds -Mchkptr -Mchkstk) still work.

Install External Dependencies ​

ADAM needs parallel HDF5, with zlib and szip, built with the same compiler and MPI as ADAM itself. scripts/hdf5_build.sh fetches the HDF5 develop branch with szip and zlib and builds all three with the mpicc and mpif90 found on PATH, so load the toolchain first. The repository's own builds use HDF5 2.3.0 (develop at ab5df552).

bash
# with the toolchain on PATH: -sdk NVF for nvfortran, -sdk GNU for gfortran
scripts/hdf5_build.sh -build -sdk NVF -abs-paths -lsrc /tmp/hdf5-src -hdf5 $PWD/lib/hdf5/develop/nvf/26.9

On Rocky Linux or RHEL, HDF5 2.3 installs into lib64/ while ADAM expects lib/: add -DCMAKE_INSTALL_LIBDIR=lib to the script's cmake line. An HDF5 built on a newer glibc than the machine that links ADAM fails with undefined reference to __isoc23_strtol; build it on that machine.

Varsets: one per machine and toolchain ​

The HDF5 path is the fobos variable $HDF5_PREFIX, defined per [varset:*] in the repository fobos. It is a fobos variable, not a shell environment variable: exporting it has no effect. NVF varsets also set $NVF_CC, the -gpu=ccXX compute capability. Selecting a varset selects the HDF5 build:

VarsetMachine and toolchain$HDF5_PREFIX$NVF_CC$NVF_LTO
local_gnu (default)workstation: gfortran 16.2.0 + Open MPI 5.0.11lib/hdf5/develop/gnu/16.2.0——
local_nvfworkstation: nvfortran 26.9lib/hdf5/develop/nvf/26.9cc89,lto
ci_gnuGitHub Actions: Ubuntu 24.04 gfortran 14 + Open MPIlib/hdf5/develop/gnu/14.2.0——
iac_gnumedooza cluster: gfortran 16.2.0 + Open MPI 5.0.11lib/hdf5/develop/iac/gnu/16.2.0/openmpi-5.0.11——
iac_nvfmedooza cluster: nvfortran 26.9, CUDA 12.9lib/hdf5/develop/iac/nvf/26.9cc80,cuda12.9,lto
leonardoLeonardo, Spack NVHPC 24.5Spack pathcc80(off)
spacehpcCray PE/opt/cray/pe/hdf5/1.14.3.3/nvidia/23.3cc90(off)

$NVF_CC can also pin the CUDA toolkit, as iac_nvf does (cc80,cuda12.9, issue #102). nvfortran otherwise targets the CUDA of the build machine's driver, and on a node with no driver (a login node) its newest bundled CUDA: a binary built there for a newer CUDA than the compute nodes' driver supports fails at the first GPU call ("Invalid image").

$NVF_LTO turns on the device link-time optimisation of the optimized OpenACC modes (-gpu=$NVF_CC$NVF_LTO): ,lto lets the device link inline the device routines across modules, which removes calls that kept the kernels' private arrays in local memory. On an A30 it takes 35% (uniform grid) and 27% (refined walls) off the FLUME FNL loop and 40% off its face-flux kernel (profiling results), at the cost of a longer build. It is empty (off) by default and in the varsets whose nvfortran has not been checked with it; the debug modes never use it. The overlay varset nolto turns it off (--varset "iac_nvf nolto"), and a change of it recompiles every file (cflags_heritage).

The AMD builds use local_amd or thera_amd together with a GPU-architecture varset (gfx90a, gfx942, ...), e.g. --varset "thera_amd gfx942".

The toolchain must match the varset exactly. A mismatch stops the build at use hdf5 ("created by a different version of GNU Fortran") or, under nvfortran, with an internal compiler error on a module symbol. Bring up a new machine by adding its own [varset:*] block (New machine).

New machine ​

A new workstation, cluster, compiler or GPU architecture needs one thing from ADAM: a [varset:*] block in fobos. The rest is the toolchain. A failure in a lower layer usually shows up as a confusing error in the layer above:

4. regression harness  run.sh -> mpirun -np 2 -> digest.py (h5py venv)
3. ADAM build          fobis build --mode prism-cpu-gnu --varset <name>
2. varset (fobos)      $HDF5_PREFIX, $NVF_CC        <- the only machine-specific layer
1. toolchain           gfortran|nvfortran + MPI + parallel HDF5 + FFTW + python3

Prerequisites ​

ToolRequired forCheck
gfortran (16.2.0; CI uses 14)CPU backendgfortran --version
nvfortran (26.9)FNL backendnvfortran --version
MPIbothmpirun --version
parallel HDF5 built with that MPIbothh5pfc -show
FFTW 3CPU backend: every PRISM mode and the FLUME -fft modes link -lfftw3; the NVIDIA FNL modes link cuFFT, part of NVHPCls /usr/lib/x86_64-linux-gnu/libfftw3.so (apt install libfftw3-dev)
fobis 3.8+bothfobis --version
python3 with venvthe regression harnesspython3 -m venv --help

fobis fetch is mandatory on a fresh clone: src/third_party/ is not tracked, and the FNL backend reaches the GPU through the FUNDAL library it fetches. The legacy FoBiS.py build -mode X form is deprecated in FoBiS 3.8+ (it is translated to the long form); write fobis build --mode X --varset Y.

Steps ​

  1. Load the compiler and MPI first, before building HDF5 and before fobis: fobis calls mpif90, and a missing module shows up as FileNotFoundError: [Errno 2] No such file or directory: 'mpif90'.

    bash
    module load gcc/16.2.0 openmpi/5.0.11-gnu16.2.0   # CPU: gfortran 16.2.0 + Open MPI 5.0.11
    module load nvhpc/26.9                           # GPU: nvfortran 26.9 + bundled HPC-X Open MPI
  2. Find or build a parallel HDF5. Look for one first (h5pfc -show, module avail hdf5). Otherwise build it with scripts/hdf5_build.sh (Install External Dependencies): it compiles with the mpicc and mpif90 on PATH, so the MPI loaded at step 1 is the MPI ADAM must use. scripts/hdf5_build.sh -h lists its options (-use-autot swaps CMake for autotools). The in-tree convention for the prefix is lib/hdf5/develop/<sdk>/<version>; any path works.

  3. Add a varset to fobos:

    ini
    [varset:mymachine_gnu]
    $HDF5_PREFIX = /path/to/parallel/hdf5
    
    [varset:mymachine_nvf]
    $HDF5_PREFIX = /path/to/nvf/parallel/hdf5
    $NVF_CC      = cc90                        # the GPU's compute capability

    Read the compute capability off the device and map X.Y to ccXY (A100 cc80, Ada/RTX 40 cc89, H100 cc90, Blackwell cc100):

    bash
    nvidia-smi --query-gpu=name,compute_cap --format=csv    # "NVIDIA GeForce RTX 4090, 8.9" -> cc89

    $NVF_CC is defined only in the NVF varsets: an *-fnl-nvf mode built without --varset <an nvf varset> leaves -gpu=$NVF_CC unexpanded and dies on the first nvfortran call.

  4. Build with the new varset:

    bash
    fobis build --mode prism-cpu-gnu --varset mymachine_gnu   # exe/adam_prism_cpu

    A requires executable stack linker warning on adam_tree_bucket_object.o is harmless. On a Cray, or wherever HDF5 is a system module that must be named rather than found under a prefix, prism-fnl-nvf-spacehpc shows the pattern (ext_libs = hdf5_fortran hdf5 z): copy it in fobos.d/prism.fobos and adapt ext_libs.

  5. Run the first gate, the CPU regression suite with the committed goldens:

    bash
    ./src/tests/prism/regression/run.sh cpu --varset mymachine_gnu   # 15 PASS, 11 SKIP, 0 FAIL

    A port is correct when the committed goldens pass within tolerance; a new machine does not mint goldens. The GPU gate, the oracles and what to do with a failure are in the regression cookbook.

SymptomLayerCause
FileNotFoundError: ... 'mpif90'1MPI module not loaded before fobis
-gpu=$NVF_CC unexpanded; nvfortran dies at once2NVF mode built without --varset <nvf varset>
HDF5 link errors (h5open_f undefined)1-2wrong $HDF5_PREFIX, or HDF5 built with another MPI
use hdf5: "created by a different version of GNU Fortran"1-2the toolchain does not match the varset; build clean
run hangs at the first I/O1serial HDF5, or HDF5's MPI is not the application's

Checklist:

[ ] fobis 3.8+; `fobis fetch` run on the fresh clone
[ ] compiler + MPI modules loaded BEFORE building HDF5 and BEFORE fobis
[ ] parallel HDF5 built against THAT MPI
[ ] [varset:<machine>] in fobos with $HDF5_PREFIX (+ $NVF_CC for the GPU)
[ ] $NVF_CC matches `nvidia-smi --query-gpu=compute_cap`
[ ] fobis build --mode prism-cpu-gnu --varset <machine> succeeds
[ ] run.sh cpu --varset <machine> passes every goldened case
[ ] (GPU) GPU-aware MPI; run.sh fnl --varset <machine>_nvf passes (cookbook T7)
[ ] (GPU) the WSL2 settings NOT copied into a cluster job script

Next ​

Build FLUME and run a Sod shock tube: first run. Then the tutorials.