Skip to content

FNL library (GPU) ​

The FNL backend is ADAM's GPU layer, built on the FUNDAL device-memory library. The same sources compile as OpenACC with nvfortran (*-fnl-nvf modes) or as OpenMP target offloading with amdflang (*-fnl-omp-amd modes, DEV_OMP): every kernel carries an !$acc directive and its !$omp twin.

Every subsystem has two parts: a wrapper object holding the device arrays, and a kernel module with the device routines. The wrappers are plain types, not extensions of the common classes (the Poisson solvers excepted). Each is a component of the application's FNL type, initialised from the realm's common objects passed as arguments. Equations, coefficients and data structures are defined once in src/lib/common (library objects) and mirrored to the device.

use adam_fnl_library exposes the FNL API together with adam_common_library and adam_fnl_mpih_global (the exceptions are listed in the module summary).

Program-scope singletons ​

The only FNL singleton is mpih_fnl (adam_fnl_mpih_global, type mpih_fnl_object): the GPU-aware MPI handler, which also owns the device context and is initialised once per process. The other FNL objects are components of the application's FNL type, one set per realm, initialised after the common initialisation from the realm's own objects (PRISM, adam_prism_fnl_object.F90):

fortran
if (.not.mpih_fnl_is_initialized) then                                           ! once per process
   call mpih_fnl%initialize(do_mpi_init=..., do_device_init=.true., verbose=.true.)
   mpih_fnl_is_initialized = .true.
endif
call self%prism_common_object%initialize(filename=filename, memory_avail=memory_avail_, verbose=.true.)
call self%field_fnl%initialize(grid=self%adam%grid, field=self%adam%field, maps=self%adam%maps, verbose=.true.)
call self%ib_fnl%initialize(grid=self%adam%grid, field=self%adam%field, ib=self%ib)
call self%rk_fnl%initialize(grid=self%adam%grid, field=self%adam%field, rk=self%rk)

Application FNL objects follow the same pattern (PRISM's coil_fnl, pic_fnl, pml_fnl). An application that reconstructs with the library WENO coefficients on the device also calls self%weno_fnl%initialize(weno=self%weno); PRISM does not (its FNL backend refuses scheme_space = weno).

Design and memory model ​

CPU and GPU array layout ​

The common library stores fields with the variable index fastest; on the device the block index moves to the front, so threads mapped to adjacent blocks read adjacent memory:

CPU:  q_cpu(nv, ni, nj, nk, nb)   — stride-1 on nv
GPU:  q_gpu(nb, ni, nj, nk, nv)   — blocks contiguous

The device state arrays (q_gpu, dq_gpu, …) belong to the application's FNL type (e.g. flume_fnl_object%q_gpu); field_fnl%initialize(..., q_gpu=) can allocate one in this layout. The application transposes between layouts with FUNDAL's buffered transposed copies (dev_memcpy_to_device(..., ij=[1,5], buf=)) or with the device kernels pack_blocks_transposed_dev / unpack_blocks_transposed_dev of adam_fnl_field_kernels. The staging buffer of a buffered copy must have the destination's extent; otherwise nvfortran makes a host copy-in temporary and the copy fails (use dev_assign_to_device(..., ij=[1,5]) instead). The transposition happens only at I/O and AMR updates; time integration uses the device layout alone.

FUNDAL integration ​

UtilityPurpose
dev_allocAllocate a device array
dev_assign_to_deviceCopy a host array to the device
dev_memcpy_to_deviceRaw copy to the device
DEVICEVAR(array) macroMark an array as device-resident (no implicit copy)

DEVICEVAR is defined in fundal.H and used at the top of every kernel file: it tells the compiler that the array lives on the device, so the OpenACC runtime inserts no host copy.

IB solid masking ​

Kernels that modify field variables skip the cells inside a solid (phi_gpu(b,i,j,k,all_solids) < 0), tested in the innermost loop.

Field ​

adam_fnl_field_object ​

A plain type, initialised from the realm's grid, field and maps. It holds the device geometry and maps (its maps component is a maps_fnl_object) and the device ghost exchange.

Device arrayShapePurpose
x_cell_gpu, y_cell_gpu, z_cell_gpu(nb, 1-ngc:ni+ngc) per axisCell centroid coordinates
dxyz_gpu(nb, 3)Block spacing
fec_1_6_array_gpu(26)Face/edge/corner codes 1–26 → boundary faces 1–6
MethodPurpose
initialize(grid, field, maps[, nv_aux][, q_gpu][, verbose])Allocate the device arrays (and q_gpu(nb,…,nv) when passed)
copy_cpu_gpuCopy the coordinates and maps to the device
reduce_ghost_local_gpu, reduce_ghost_mpi_gpuFold deposited ghosts back onto their owner cells, locally and across ranks
update_ghost_local_gpuIntra-rank ghost update on the device
update_ghost_mpi_gpuPack on the device, MPI exchange, unpack on the device
compute_q_gradient(b, ivar, dx, dy, dz, q_gpu, gradient)AMR criterion `max

initialize, copy_cpu_gpu, update_ghost_mpi_gpu and compute_q_gradient take the realm's CPU grid/field/maps (and the per-rank comm_map_*_ptr_ghost) as arguments, so a multi-realm caller always exchanges with its own decomposition.

GPU-direct ghost exchange ​

update_ghost_mpi_gpu posts MPI_Isend/MPI_Irecv directly on the device ghost buffers (send_buffer_ghost_gpu, recv_buffer_ghost_gpu), which lets CUDA-aware MPI use GPUDirect RDMA. There is no host-staging fallback. On the WSL2 development box this path aborts with two or more ranks unless UCX is forced eager (UCX_RNDV_THRESH=inf); that setting is WSL-only and a regression on a cluster. Details: GPU5.

adam_fnl_field_kernels ​

All routines carry !$acc parallel loop independent and DEVICEVAR on every device argument.

KernelPurpose
compute_q_gradient_devCentred gradient magnitude, reduction(max:)
compute_normL2_residuals_devL2 norm √(Σ dq²) per variable, reduction(+:)
copy_transpose_gpu_cpu_devTranspose (nb,ni,nj,nk,nv) → (nv,ni,nj,nk,nb) on the device
pack_blocks_transposed_dev, unpack_blocks_transposed_devLayout transposition of the state
populate_send_buffer_ghost_gpu_devPack ghosts into the MPI send buffer (1-cell and 8-cell AMR averaging)
receive_recv_buffer_ghost_gpu_devUnpack the MPI receive buffer into ghosts
update_ghost_local_gpu_devIntra-rank block-to-block ghost update, with AMR coarse↔fine averaging

WENO reconstruction ​

adam_fnl_weno_object ​

A plain type, initialised from the realm's weno (initialize(weno)). The CPU object computes the coefficients; the FNL object mirrors them to the device. PRISM's FNL backend does not initialise it.

Device arrayShapePurpose
a_gpu(2, 0:S-1, S)Optimal weights per sub-stencil and face
p_gpu(2, 0:S-1, 0:S-1, S)Polynomial reconstruction coefficients
d_gpu(0:S-1, 0:S-1, 0:S-1, S)Smoothness-indicator coefficients
ror_schemes_gpu, ror_ivar_gpu, cell_scheme_gpu(:), (:), (nb, ni, nj, nk)Reduced-order reconstruction near solids; used by no maintained application

adam_fnl_weno_kernels ​

KernelDirectivePurpose
weno_reconstruct_upwind_dev(S, a, p, d, zeps, V, VR)!$acc routine seqLeft (VR(1)) and right (VR(2)) face values from stencil V, half-width S a run-time value
weno_reconstruct_upwind_wratio_dev!$acc routine seqThe same, also returning the smoothness ratio of each face, WRATIO(f) = min_k w(f,k) / a(f,k,S)

The algorithm: S polynomial reconstructions on the sub-stencils, the smoothness indicators, then VR(f) = Σ_k w(f,k) · VP(f,k). !$acc routine seq makes the routine callable inside a parallel region, one thread per stencil.

adam_fnl_weno_width_kernels ​

One routine per stencil width, weno_reconstruct_upwind_s<N>_dev and weno_reconstruct_upwind_wratio_s<N>_dev for S = N = 1 … 5 (WENO-1 … WENO-9), with S a compile-time constant. FLUME's per-width face kernels (..._s<N>_dev) call them. The body is one include, adam_fnl_weno_width_body.INC, shared with the generic routines, so both are bitwise the same. The generic routines stay for callers whose width is a run-time value.

Runge-Kutta integration ​

adam_fnl_rk_object ​

A plain type, initialised from the realm's grid, field and rk. It holds the stage storage on the device and runs the stage updates.

Device arrayShapePurpose
q_rk_gpu(nb, ni, nj, nk, nv, nrk)Stage values (1 slot for low-storage, nrk for SSP)
alph_gpu(nrk, nrk)SSP alpha coefficients
beta_gpu, gamm_gpu(nrk)SSP beta and gamma coefficients
SchemeStorageStages
RK_1, RK_2, RK_3Low-storage, overwrites q_gpu1 slot
RK_SSP_11, RK_SSP_22, RK_SSP_33Multi-stage1 / 2 / 3
RK_SSP_54Multi-stage5
MethodPurpose
initialize(grid, field, rk)Allocate q_rk_gpu for the scheme
initialize_stages(q_gpu)Copy q_gpu into every stage slot
assign_stage(s, q_gpu, phi_gpu)Copy q_gpu into stage s, skipping solid cells
compute_stage(s, dt, phi_gpu)Accumulate stages 1…s−1 into stage s (SSP)
compute_stage_ls(s, dt, phi_gpu, dq_gpu, q_gpu)Low-storage update q = ark·q_n + brk·q + dt·crk·dq
update_q(s, dt, phi_gpu)Final assembly q += dt·beta(s)·q_rk(:,:,:,:,:,s)

adam_fnl_rk_kernels ​

All kernels carry !$acc parallel loop independent and skip solid cells.

KernelPurpose
rk_assign_stage_devq_rk(:,s) ← q_gpu (fluid cells)
rk_initialize_stages_devq_rk(:,all_s) ← q_gpu
rk_compute_stage_devq_rk(:,s) += dt·α(s,ss)·q_rk(:,ss), ss = 1…s−1
rk_compute_stage_ls_devq = ark·q_n + brk·q + dt·crk·dq
rk_update_q_devq += dt·β(s)·q_rk(:,s), s = 1…nrk

Immersed boundary method ​

adam_fnl_ib_object ​

A plain type, initialised from the realm's grid, field and ib. It holds the signed distance phi_gpu and runs the eikonal wall enforcement. Sign convention: phi < 0 inside the solid, phi > 0 in the fluid.

Device arrayShapePurpose
phi_gpu(nb, ni, nj, nk, n_solids+1)Signed distance; the last slice is the union of all solids
q_bcs_vars_gpu(:,:)Wall state variables per solid
MethodPurpose
initialize(grid, field, ib)Allocate phi_gpu and q_bcs_vars_gpu; copy the wall data to the device
evolve_eikonal(dxyz_gpu, dq_gpu, q_gpu)Advance the eikonal extrapolation inside the solid; dxyz_gpu passed by the caller (field_fnl%dxyz_gpu)
invert_eikonal(q_gpu)Wall condition at the surface: BCS_VISCOUS reverses the velocity, BCS_EULER reflects its normal component u → u − 2(u·n̂)n̂

adam_fnl_ib_kernels ​

KernelPurpose
compute_phi_analytical_sphere_devφ = −(‖x − xc‖ − R)
compute_phi_all_solids_devφ_all = max(φ₁, …, φ_ns)
compute_eikonal_dq_phi_dev`dq =
evolve_eikonal_q_phi_devq -= dq inside the solid
invert_eikonal_q_phi_devMomentum reflection at the surface
move_phi_dev, reduce_cell_order_phi_devLevel-set advection and order reduction near solids; used by no maintained application

Communication maps ​

adam_fnl_maps_object ​

A plain type, the maps component of field_fnl_object, initialised from the realm's maps. It mirrors the communication tables to the device, so ghost packing and unpacking need no CPU staging.

Device arrayColumnsContent
local_map_ghost_cell_gpu9(b_src, b_dst, i_src, j_src, k_src, i_dst, j_dst, k_dst, mode)
comm_map_send_ghost_cell_gpu7(b_src, i, j, k, v_offset, buf_idx, mode)
comm_map_recv_ghost_cell_gpu6(buf_idx, b_dst, i, j, k, v_offset)
send_buffer_ghost_gpu, recv_buffer_ghost_gpu—Packed MPI buffers
local_map_bc_crown_gpu—Boundary-condition crown map

mode = 1: one-to-one cells (same level); mode = 8: eight-cell average (fine → coarse at an AMR interface).

MethodPurpose
initialize(maps)Initialise and call copy_cpu_gpu
copy_cpu_gpu(verbose)Copy every map to the device (dev_assign_to_device)

Finite difference/volume operators ​

adam_fnl_fdv_operators_library ​

The operators of adam_fdv_operators_library, callable inside parallel regions (!$acc routine seq, one thread per cell). Each takes the half-width s and the spacing dxyz, so the order is chosen at call time.

OperatorFD centredFV centred
Gradientcompute_gradient_fd_centered_devcompute_gradient_fv_centered_dev
Divergencecompute_divergence_fd_centered_devcompute_divergence_fv_centered_dev
Curlcompute_curl_fd_centered_devcompute_curl_fv_centered_dev
Laplaciancompute_laplacian_fd_centered_devcompute_laplacian_fv_centered_dev

Turbulence ​

adam_fnl_turbulence_kernels ​

Device twins of adam_turbulence_kernels (library objects): compute_mu_t_vreman_dev, compute_mu_t_wale_dev, compute_mu_t_sigma_dev, on transposed arrays (b, i, j, k, v). They call the pointwise models of adam_turbulence_library; the body is adam_fnl_turbulence_body.INC.

Poisson solvers ​

The device twins of the FLAIL solvers. Both extend their common parent.

adam_fnl_poisson_mg_object ​

Extends poisson_mg_object. The leaves, the hierarchy and every CSR operator are assembled on the host; work_place_fnl then puts the operators, the level diagonals, colours and leaf maps, the cell volumes and the work vectors on the device until destroy, with halo buffers sized for the largest operator.

PartImplementation
apply_fnlPacks the halo on the device, exchanges it by MPI_ALLTOALLV on device buffers (GPU-aware MPI; none on one rank), one row per thread in the host's order
smooth_fnlOne kernel per colour after its halo exchange. It equals the host's sequential Gauss-Seidel: the colour is the global cell parity and the level operators are 7-point, so no row couples to its own colour (no such entry in 5704 smoother calls over the AMR, isolated, boundary-kind and sub-block suites)
Vector hooksDevice kernels; dots and max-norm as device reductions
RootThrough the host to the FFT solver, cuFFT by default (initialize_fnl passes poisson_fft_fnl_object when no coarse is given), at most 32^3 cells (root_cells_max_fnl)
On the hostRight-hand side, James step, ghost-fill placement

initialize_fnl releases and places the device copies itself, since the parent's initialize dispatches to the host's no-op hooks. Results equal the host's to round-off (summation order of the dots, FMA contraction). FLUME's FNL backend and bench_poisson_fnl set flail%mg_mold to it; test-poisson-mg-fnl-nvf builds test_poisson_mg on it.

Root size on cuFFT, 3.9 M AMR cellsCost per solve
32^3 (default)1×
128^32.4× to 6.8×

adam_fnl_poisson_fft_object ​

Extends poisson_fft_object with cuFFT. In the distributed slab transform the block planes are redistributed on the host; the transform runs on each rank's GPU with its arrays resident (slab_r, slab_c, pen, khat_pen, the transpose buffers, created with the plans):

HookDevice work
slab_update_fnlslab_r up and down
slab_planes_fnlOne D2Z/Z2D plan for one plane, executed on every plane
slab_transpose_fnlPack and unpack on the device, MPI_ALLTOALLV on device buffers (none on one rank)
slab_lines_fnlOne Z2Z plan for one row of z-lines, executed on every row
slab_spectral_fnlDivision by the eigenvalues, or product with the kernel spectrum

The result does not depend on the rank count. The rank-0 path remains for PRISM's CG solves and the multigrid's small root: the global source goes to the device, D2Z, a kernel divides by the eigenvalues (host-computed tables), Z2D, and the potential comes back. cuFFT is bound to the stream of the synchronous OpenACC queue (acc_get_cuda_stream(acc_async_sync)), so transforms and kernels are ordered. Isolated: the padded transforms, with the kernel spectrum khat made once on the device (enter data, released by destroy_plans).

ItemDetail
Build-D_CUFFT -cudalib=cufft (mode test-poisson-fft-fnl-nvf); cufft and openacc are intrinsic modules in fobos.d/templates.fobos. NVIDIA only: without _CUFFT (AMD) making the plans stops the run
MemoryNot FUNDAL-agnostic: the arrays live in an OpenACC data region around the cuFFT calls (host_data use_device)
VerificationSV-2 against the discrete solution at round-off, SV-6 against the exact potentials of two spheres; both against the FFTW solver (selfgravity/poisson_oracle.py compare, isolated: 1.6e-15)

MPI handler ​

adam_fnl_mpih_object ​

fortran
use :: fundal_mpih_object, only : mpih_fnl_object => mpih_object

FUNDAL's MPI handler under the alias mpih_fnl_object: rank and size, rank-prefixed output, timing, GPU-aware MPI.

Module summary ​

ModuleRoleExtends
adam_fnl_libraryAggregate entry point—
adam_fnl_field_objectDevice geometry, maps, ghost exchange—
adam_fnl_field_kernelsGradient, L2 norm, ghost pack/unpack, transposition—
adam_fnl_weno_objectWENO coefficient mirror (+ unused ROR tables)—
adam_fnl_weno_kernelsUpwind WENO reconstruction, run-time width—
adam_fnl_weno_width_kernelsUpwind WENO reconstruction, one routine per width (not re-exported)—
adam_fnl_rk_objectRK stage storage and updates—
adam_fnl_rk_kernelsStage assign, accumulate, low-storage, final update—
adam_fnl_ib_objectSigned distance and eikonal wall condition—
adam_fnl_ib_kernelsEikonal evolution, sphere distance, momentum inversion—
adam_fnl_maps_objectCommunication tables and MPI buffers—
adam_fnl_turbulence_kernelsSubgrid-scale eddy viscosity—
adam_fnl_fdv_operators_libraryDevice-callable FD/FV operators—
adam_fnl_poisson_fft_objectFFT Poisson solver on cuFFT (not re-exported)poisson_fft_object
adam_fnl_poisson_mg_objectMultigrid Poisson solver on the device (not re-exported)poisson_mg_object
adam_fnl_mpih_objectFUNDAL MPI handler alias—
adam_fnl_mpih_globalSingleton mpih_fnl—