adam_flume_fnl_object

ADAM, FLUME FNL (OpenACC / OpenMP offload) backend object.

Implements the forest contract on the device. The state lives on the device for the whole run in the transposed layout (b, i, j, k, v); full-field device-to-host copies happen only on save steps. The FNL helpers (field, IB, RK, WENO) are per-realm components initialized after the common initialization from the realm's own objects; mpih_fnl is the only FNL singleton and is initialized once per process. Space operator: characteristic (or conservative) WENO flux splitting, face fluxes then flux difference (issue #35, section 3.4); the per-face physics is the shared adam_flume_euler_library, so only the kernels are FNL-specific.

Source: src/app/flume/fnl/adam_flume_fnl_object.F90

Dependencies

Contents

• equilibrium_seam_face
• flume_fnl_object
• accumulate_seam_fluxes
• flag_troubled_seam_cells
• add_dissipative_fluxes_dev
• apply_floors
• allocate_gpu
• check_nonfinite
• check_glm_ch
• compute_conservation
• apply_forcing
• update_hybrid_sensor
• overwrite_hybrid_fluxes
• update_wall_model_faces
• close_noslip_fluxes
• overwrite_wall_model_fluxes
• update_hybrid_faces
• compute_mu_t
• update_sgs_lambda
• compute_turbulence_history
• compute_divb_history
• compute_dflux_norms
• compute_q_aux
• copy_cpu_gpu
• wm_filter_to_host
• wm_filter_to_device
• copy_gpu_cpu
• copy_phi_gpu
• refresh_potential
• copy_potential_gpu
• solve_potential_gpu
• destroy
• initialize_flume
• limit_positivity_dev
• seam_lists
• seam_sync_fluxes_dev
• seam_sync_factors_dev
• seam_sync_theta_dev
• seam_sync_blend_dev
• save_residuals
• save_simulation_data
• set_boundary_conditions
• build_bc_pass_rows
• inflow_df_to_device
• wall_history_to_host
• free_inflow_df_rows
• update_ghost
• advance_one_step_forest
• after_topology_build_forest
• apply_reflux_to_stage_forest
• begin_stage_forest
• close_step_forest
• compute_local_dt_forest
• end_stage_forest
• reflux_stage
• fill_seam_from_peer_forest
• pack_seam_cells_forest
• unpack_seam_cells_forest
• finalize_forest
• finalize_mpi_forest
• initialize_forest
• is_done_forest
• open_step_forest
• seam_ghosts_to_stages
• post_step_forest
• sync_dual_energy
• apply_cooling
• accrete_sinks_gpu
• cooling_tables
• regrid_forest
• regrid_begin_forest
• regrid_mark_forest
• regrid_apply_forest
• regrid_end_forest
• copy_blocks_transposed
• refresh_equilibrium_residual
• compute_gravity_correction
• compute_residuals_weno_dev
• compute_residuals_riemann_dev
• integrate_rk_ls_dev
• integrate_rk_ssp_dev
• rk_assign_stage
• rk_compute_stage
• rk_compute_stage_ls
• rk_update_q
• seam_gather_cells
• seam_gather_stencils
• seam_gather_faces
• seam_scatter_faces
• seam_scatter_cells
• cooling_time_min
• stages_per_step_forest

Derived Types

equilibrium_seam_face

The equilibrium's flux mismatch of one register face, F_coarse - F_fine_sum per skin cell (issue #73, M7-G2b).

Components

Name	Type	Attributes	Description
d	real(kind=R8P)	allocatable	Mismatch [nv, nface_cells].

flume_fnl_object

FLUME FNL backend object.

Inheritance

Extends: flume_common_object

Components

Name	Type	Attributes	Description
io	type(io_object)		IO handler.
amr	type(amr_object)		AMR marker handler.
slices	type(slices_object)		Slices handler.
blanesmoan	type(blanesmoan_object)		Blanes-Moan integrator.
cfm	type(cfm_object)		Commutator-Free Magnus integrator.
leapfrog	type(leapfrog_object)		Leapfrog integrator.
flail	type(flail_object)		FLAIL, the elliptic solver library (issue #84).
weno	type(weno_object)		WENO reconstructor.
ib	type(ib_object)		Immersed boundary.
rk	type(rk_object)		Runge-Kutta integrator.
adam	type(adam_object)		ADAM, grid + tree + field + maps orchestrator.
fdv_scheme	character(len=:)	allocatable	FDV scheme, fd/fv.
fdv_order	integer(kind=I4P)		Order of finite difference/volume schemes, general order.
fdv_half_stencil	integer(kind=I4P)		Half stencil length of finite difference/volume schemes.
fdv_half_stencils	integer(kind=I4P)		Half stencil length of fdv schemes for each derivative up to 6.
ngc	integer(kind=I4P)	pointer	Number of ghost cells.
ni	integer(kind=I4P)	pointer	Number of cells in i direction.
nj	integer(kind=I4P)	pointer	Number of cells in j direction.
nk	integer(kind=I4P)	pointer	Number of cells in k direction.
nb	integer(kind=I4P)	pointer	Total blocks number for MPI.
blocks_number	integer(kind=I4P)	pointer	Actual blocks number.
nv	integer(kind=I4P)	pointer	Number of variables in q vector.
realm_index	integer(kind=I4P)		Inter-realm seam coupling — currently-published stage buffer key.
stage_active	integer(kind=I4P)		A copy of the forest's flux register (its faces and lookup, the accumulators unused), refreshed by the forest each
flux_register_copy	type(flux_register_object)		Procedure pointer TBPs for FDV operators (set at initialization by backend).
compute_block_total_variation	procedure(compute_block_total_variation_interface)	pass(self), pointer	Compute TV.
compute_curl	procedure(compute_curl_interface)	pass(self), pointer	Compute curl.
compute_curl_extended	procedure(compute_curl_interface)	pass(self), pointer	Compute curl on extended support.
compute_derivative1	procedure(compute_derivative1_interface)	pass(self), pointer	Compute deriv1.
compute_derivative2	procedure(compute_derivative2_interface)	pass(self), pointer	Compute deriv2.
compute_derivative4	procedure(compute_derivative4_interface)	pass(self), pointer	Compute deriv4.
compute_divergence	procedure(compute_divergence_interface)	pass(self), pointer	Compute dive.
compute_gradient	procedure(compute_gradient_interface)	pass(self), pointer	Compute grad.
compute_gradient_extended	procedure(compute_gradient_interface)	pass(self), pointer	Compute gradient on extended support.
compute_laplacian	procedure(compute_laplacian_interface)	pass(self), pointer	Compute laplac.
amr_locked_	logical		Runtime AMR locked after initialization.
regrid_phase	type(regrid_phases_t)		The state of the current runtime regrid (issue #110).
amr_criteria	type(amr_criterion_pointer)	allocatable	Registered refinement criteria (issue #110, P4).
amr_marker_list	type(amr_criterion_pointer)	allocatable	The [amr_marker_N] criteria, in input order:
save_auxiliary_fields	logical		Save the auxiliary variables with the fields.
glm_speed_reported	real(kind=R8P)		Largest wave speed above c_h reported so far.
divb_reported	real(kind=R8P)		Largest monitored measure above divb_tol reported so far.
divb_seam	integer(kind=I4P)	allocatable	Seam faces flags of the div(B) history [nb, 6].
q	real(kind=R8P)	allocatable	Conservative variables [nv, 1-ngc:ni+ngc, ..., nb].
dq	real(kind=R8P)	allocatable	Residuals [nv, 1-ngc:ni+ngc, ..., nb].
q_aux	real(kind=R8P)	allocatable	Auxiliary variables [nv_aux, 1-ngc:ni+ngc, ..., nb].
turbulence	type(turbulence_object)		Subgrid-scale model ([turbulence], issue #119, T1).
forcing	type(flume_forcing_object)		Mass-flow forcing ([forcing], issue #119, T3.1).
wm_faces	integer(kind=I4P)	allocatable	Wall model: block faces it holds [nb, 6] (issue #119, T3.3).
has_wall_model	logical		Some face of the domain is modelled.
ns_faces	integer(kind=I4P)	allocatable	No-slip walls: block faces on them [nb, 6] (issue #119, the #65 mass leak).
has_noslip	logical		Some face of the domain is a no-slip wall of a hydrodynamic model.
wm_avg_x	real(kind=R8P)	allocatable	Wall model input filter: the averaged u, v, w, p, rho on the x faces [5, 2, nj, nk, nb], side 1 low.
wm_avg_y	real(kind=R8P)	allocatable	The same on the y faces [5, ni, 2, nk, nb].
wm_avg_z	real(kind=R8P)	allocatable	The same on the z faces [5, ni, nj, 2, nb].
has_wm_filter	logical		Some modelled face filters its samples (wall_model_filter > 0).
has_wm_viscosity	logical		Some modelled face puts its eddy viscosity below the matching height.
wm_filter_fresh	logical		The averages hold no history (start, regrid, a checkpoint without them): the next commit takes the sample.
wm_filter_it	integer(kind=I4P)		The iteration whose sample the averages hold.
hybrid_faces	integer(kind=I4P)	allocatable	Hybrid scheme: block faces it must not read across [nb, 6].
hybrid_has_faces	logical		Hybrid scheme: some block of the realm has such a face.
sgs_lambda	real(kind=R8P)		Local time step limit of the eddy viscosity.
phi_grav	real(kind=R8P)	allocatable	Gravitational potential [1-ngc:ni+ngc, ..., nb] (issue #81);
poisson_wall	real(kind=R8P)		Wall time of the last Poisson solve (s).
q_name	type(string)	allocatable	Conservative variables names.
dq_name	type(string)	allocatable	Residuals names.
q_aux_name	type(string)	allocatable	Auxiliary variables names.
bc	type(flume_bc_object)		Boundary conditions.
diagnostics	type(flume_diagnostics_object)		Diagnostics.
ic	type(flume_ic_object)		Initial conditions.
numerics	type(flume_numerics_object)		Numerics.
physics	type(flume_physics_object)		Physics.
units	type(flume_reference_object)		Reference layer (dimensional input, issue #49).
sinks	type(flume_sinks_object)		Sink particles (issue #94).
time	type(flume_time_object)		Time handler.
field_fnl	type(field_fnl_object)		Field helper (coordinates, maps, ghost exchange).
ib_fnl	type(ib_fnl_object)		Immersed boundary helper.
rk_fnl	type(rk_fnl_object)		Runge-Kutta helper.
weno_fnl	type(weno_fnl_object)		WENO helper.
q_gpu	real(kind=R8P)	pointer	Conservative variables [nb, i, j, k, nv].
dq_gpu	real(kind=R8P)	pointer	Residuals [nb, i, j, k, nv].
q_aux_gpu	real(kind=R8P)	pointer	Auxiliary variables [nb, i, j, k, nv_aux].
hybrid_faces_gpu	integer(kind=I4P)	pointer	Hybrid scheme: block faces it must not read across [nb, 6].
wm_faces_gpu	integer(kind=I4P)	pointer	Wall model: block faces [nb, 6].
ns_faces_gpu	integer(kind=I4P)	pointer	No-slip walls: block faces [nb, 6].
wm_avg_x_gpu	real(kind=R8P)	pointer	Wall model input filter, x faces [nb, 2, nj, nk, 5] (one element without a filter).
wm_avg_y_gpu	real(kind=R8P)	pointer	The same, y faces [nb, ni, 2, nk, 5].
wm_avg_z_gpu	real(kind=R8P)	pointer	The same, z faces [nb, ni, nj, 2, 5].
psi_gpu	real(kind=R8P)	pointer	Hybrid sensor [nb, i, j, k, nv], component 1, q-shaped.
mu_t_gpu	real(kind=R8P)	pointer	Eddy viscosity [nb, i, j, k, 1], with a subgrid-scale model.
flx_f_gpu	real(kind=R8P)	pointer	X-face fluxes [nb, 0:ni, 1:nj, 1:nk, nv].
fly_f_gpu	real(kind=R8P)	pointer	Y-face fluxes [nb, 1:ni, 0:nj, 1:nk, nv].
flz_f_gpu	real(kind=R8P)	pointer	Z-face fluxes [nb, 1:ni, 1:nj, 0:nk, nv].
gcor_gpu	real(kind=R8P)	pointer	Fourth-order correction of the gravity energy source
phi_grav_gpu	real(kind=R8P)	pointer	Gravitational potential (issue #81) [nb, i, j, k, 1]; a
dq_e_gpu	real(kind=R8P)	pointer	Residual of the hydrostatic equilibrium (issue #73, M7-G2b)
seam_e	type(equilibrium_seam_face)	allocatable	The equilibrium's 2:1 flux mismatch per register face.
equilibrium_stale	logical		dq_e_gpu describes another grid (init, regrid).
equilibrium_computing	logical		dq_e_gpu is being computed: no subtraction.
seam	type(flume_seam_sync_object)		Per-stage seam flux synchronisation of the limiter (issue #50).
lam_gpu	real(kind=R8P)	pointer	Positivity limiter cell factors (component 1, q-shaped for
q_inflow_gpu	real(kind=R8P)	pointer	Conservative inflow state of each face [nv, 6].
df_rows_gpu	integer(kind=I4P)	pointer	Digital-filter inflow: block and cell of each ghost row [4, rows] (#119).
df_q_gpu	real(kind=R8P)	pointer	Digital-filter inflow: state of each ghost row [5, rows] (#119).
wall_sign_gpu	real(kind=R8P)	pointer	Wall mirror sign per variable and direction [nv, 3].
divb_seam_gpu	integer(kind=I4P)	pointer	Seam faces flags of the div(B) history [nb, 6].
bc_rows_gpu	integer(kind=I4P)	pointer	BC crown map rows grouped by pass [rows, ngc] (#90 F-A').
bc_pass_range	integer(kind=I4P)	allocatable	First, last entry of each pass in bc_rows_gpu [2, 3, ngc].
skin_faces	integer(kind=I4P)	allocatable	Register faces [3, n]: block, fec, column - 1 (#90 F-B).
skin_faces_gpu	integer(kind=I4P)	pointer	Device copy of skin_faces.
skins_gpu	real(kind=R8P)	pointer	Packed skins of the register faces [nv, columns].
buf_5D_R8P	real(kind=R8P)	allocatable	Transposed copy buffer, extent identical to q_gpu.
db5	integer(kind=I4P)		Device bounds of the transposed copies.
hb5	integer(kind=I4P)		Host bounds of the transposed copies.
compute_residuals_dev	procedure(compute_residuals_dev_interface)	pass(self), pointer	Space operator.
integrate_dev	procedure(integrate_dev_interface)	pass(self), pointer	Time operator.

Type-Bound Procedures

Name	Attributes	Description
load_fdv_from_file	pass(self)	Load FDV config from file.
close_block_xh5f	nopass	Close XH5F file block.
close_file_xh5f	nopass	Close XH5F file.
open_block_xh5f	pass(self)	Open block file XH5F.
open_file_xh5f	pass(self)	Open file XH5F.
save_q_xh5f	pass(self)	Save in XH5F (XDMF/HDF5) format.
compute_curl_fd_extended	pass(self)	Compute curl of vector field, finite difference, even for half gcs.
compute_gradient_fd_extended	pass(self)	Compute gradient of scalar field, finite difference, even for half
amr_update	pass(self)	Do AMR update (initialization-time only).
runtime_amr_forest	pass(self)	Whether this realm regrids during the run (issue #74).
mark_by_geometry	pass(self)	Mark blocks to be refined by a primitive geometric box.
mark_by_gradient	pass(self)	Mark blocks by the gradient of a conservative or auxiliary variable.
mark_by_solid	pass(self)	Mark blocks crossed by the surface of an immersed solid.
mark_by_lohner	pass(self)	Mark blocks by the Loehner estimator of a variable (issue #74).
mark_by_jeans	pass(self)	Mark blocks by the cells per Jeans length (issue #91).
mark_runtime	pass(self)	Combine the markers into the flags of a runtime regrid (issue #74).
register_amr_criterion	pass(self)	Register a refinement criterion by name (issue #110, P4).
check_regridded_state	pass(self)	Stop on an inadmissible state after a regrid (issue #74).
is_regrid_step	pass(self)	Whether the step just closed is a runtime regrid step (issue #74).
accumulate_seam_skin	pass(self)	Route one weighted seam face skin to the forest's flux register.
add_seam_gravity_work	pass(self)	Add the seam gravity work of self-gravity to a seam skin.
allocate_common	pass(self)	Allocate common data.
compute_fields_number	pass(self)	Compute the block-sized fields allocated per block.
compute_phi	pass(self)	Compute the immersed solids distance function (host).
destroy_common	pass(self)	Free common data.
glm_lambda	pass(self)	Return the GLM bound of the local dt, c_h max sum_d 1/dx_d.
initialize	pass(self)	Initialize the common data.
log_dt_limit	pass(self)	Log which limit, hyperbolic or diffusive, sets dt.
log_gravity_regime	pass(self)	Log the velocity gravity adds in a step (issue #73).
log_self_gravity	pass(self)	Log the self-gravity regime (issue #81).
poisson_iterations	pass(self)	FCG iterations of the last multigrid solve.
wall_temperature_bound	pass(self)	Return the hottest isothermal wall temperature of the dt bound.
null_freeze	pass(self)	Return the variable each null direction freezes.
output_factors	pass(self)	Return the output factors of written variables.
output_names	pass(self)	Return the names of every written variable.
report_divb	pass(self)	Reduce and save the div(B) norms, apply the divb_tol monitor.
report_glm_speed	pass(self)	Check c_h against the fastest wave (warning or stop).
nonfinite_total	pass(self)	Sum the non-finite values count over the ranks.
stop_nonfinite	pass(self)	Stop on a non-finite state, locating it on every rank.
load_restart_files	pass(self)	Load restart files.
save_restart_files	pass(self)	Save restart files.
load_wm_filter	pass(self)	Load the wall-model input filter averages of a restart (issue #119).
save_wm_filter	pass(self)	Save the wall-model input filter averages (issue #119).
advance_inflow_df	pass(self)	Move the digital-filter inflow on by the step (issue #119, T3.7).
save_wall_history	pass(self)	Save the wall-model history on its cadence (issue #119, T3.7).
save_slices	pass(self)	Save the slices on their cadence.
save_xh5f	pass(self)	Save fields in XH5F format.
set_divb_seam	pass(self)	Set the seam faces flags of the div(B) history.
set_glm_damping	pass(self)	Set the GLM damping once the grid exists.
set_potential	pass(self)	Set the gravitational potential of the current grid.
solve_potential	pass(self)	Solve the self-gravity potential of a host state.
solve_potential_packed	pass(self)	Solve the self-gravity potential of a packed density.
compute_sink_accelerations	pass(self)	Mesh gravity at the sinks (issue #94).
deposit_sinks	pass(self)	Add the sinks' CIC mass to a packed density (issue #94).
drift_sinks	pass(self)	Drift the sinks by a step (issue #94).
kick_sinks	pass(self)	Kick the sinks by half a step (issue #94).
sink_stencil	pass(self)	CIC stencil of a point: target points and weights (issue #94).
sinks_dt_limit	pass(self)	Time step limit of the sinks (issue #94).
accrete_sinks	pass(self)	Threshold accretion of the committed host state (issue #94).
accrete_cells	pass(self)	Threshold accretion on gathered cells (issue #94).
sink_accretion_cells	pass(self)	The cells within an accretion radius (issue #94).
create_sinks	pass(self)	Sink creation, Federrath et al. 2010 (issue #94).
merge_sinks	pass(self)	Merge bound, converging sinks (issue #94).
sink_accretion_radius	pass(self)	Accretion radius of a point (issue #94).
coupling_descriptor_forest	pass(self)	Return the realm coupling descriptor.
set_wall_model_faces	pass(self)	Flag the block faces of the wall model.
set_hybrid_faces	pass(self)	Flag the block faces the central flux must not read across.
accumulate_seam_fluxes	pass(self)	Accumulate the weighted seam face fluxes of one stage.
reflux_stage	pass(self)	The per-stage reflux (issue #81, P5c).
flag_troubled_seam_cells	pass(self)	Flag the troubled seam face cells of one stage (#68 E1).
add_dissipative_fluxes_dev	pass(self)	Add the dissipative face fluxes on the device (issue #65).
apply_floors	pass(self)	Apply the MHD positivity floors of a stage.
allocate_gpu	pass(self)	Allocate device data.
check_glm_ch	pass(self)	Check the GLM c_h against the fastest wave.
compute_conservation	pass(self)	Compute and save the conservation integrals.
compute_divb_history	pass(self)	Compute and save the div(B) norms (MHD).
compute_turbulence_history	pass(self)	Compute and save the turbulence history (issue #119).
apply_forcing	pass(self)	Hold the bulk momentum of a channel (issue #119, T3.1).
update_hybrid_sensor	pass(self)	Compute the sensor, single-valued on the ghost cells (issue #119, T2).
overwrite_hybrid_fluxes	pass(self)	Central flux where the sensor allows it (issue #119, T2).
update_wall_model_faces	pass(self)	Flag the block faces of the wall model (issue #119, T3.3).
overwrite_wall_model_fluxes	pass(self)	The wall-face fluxes of the wall model (issue #119, T3.3).
close_noslip_fluxes	pass(self)	Close the inviscid fluxes through no-slip wall faces (the #65 mass leak).
update_hybrid_faces	pass(self)	Flag the block faces of the sensor (issue #119, T2.4).
compute_mu_t	pass(self)	Compute the eddy viscosity of the model (issue #119).
update_sgs_lambda	pass(self)	Update the time step limit of the eddy viscosity (issue #119).
compute_dflux_norms	pass(self)	Compute the D_flux norms of the committed state (#68 A2).
compute_q_aux	pass(self)	Compute the auxiliary variables.
compute_gravity_correction	pass(self)	Compute the fourth-order gravity energy correction.
refresh_equilibrium_residual	pass(self)	Recompute the equilibrium residual if stale.
check_nonfinite	pass(self)	Stop on a non-finite committed state.
sync_dual_energy	pass(self)	Synchronise the total energy and the entropy (issue #91).
apply_cooling	pass(self)	Radiative cooling of the committed state (issue #93).
cooling_time_min	pass(self)	Shortest cooling time of the committed state (issue #93).
accrete_sinks_gpu	pass(self)	Threshold accretion of the committed device state (issue #94).
copy_cpu_gpu	pass(self)	Copy state and topology from host to device.
copy_gpu_cpu	pass(self)	Copy state from device to host.
wm_filter_to_host	pass(self)	Copy the wall-model filter averages from the device to the host arrays (issue #119).
wm_filter_to_device	pass(self)	Copy the host wall-model filter averages to the device (issue #119).
inflow_df_to_device	pass(self)	Copy the digital-filter inflow ghost rows to the device (issue #119).
wall_history_to_host	pass(self)	Copy the state to the host for the wall-model history (issue #119).
copy_phi_gpu	pass(self)	Copy the immersed solids distance function to the device.
refresh_potential	pass(self)	Set the gravitational potential and copy it to the device.
copy_potential_gpu	pass(self)	Copy the gravitational potential to the device.
solve_potential_gpu	pass(self)	Solve the self-gravity potential of a device state.
destroy	pass(self)	Free device and host data.
initialize_flume	pass(self)	Initialize the FNL backend.
limit_positivity_dev	pass(self)	Apply the positivity limiter to the stage face fluxes.
seam_ghosts_to_stages	pass(self)	Copy the end_of_step seam ghosts into the stages (#113).
seam_lists	pass(self)	Host lists of the seam skin cells and faces (issue #50).
seam_sync_blend_dev	pass(self)	Set the seam face fluxes with the seam factors (issue #50).
seam_sync_factors_dev	pass(self)	Recompute the seam cells' factors (issue #50).
seam_sync_fluxes_dev	pass(self)	Publish the donor states and the fine means (issue #50).
seam_sync_theta_dev	pass(self)	Compute the seam factors (issue #50).
save_residuals	pass(self)	Save residuals history.
save_simulation_data	pass(self)	Save fields, restart and diagnostics on their cadence.
set_boundary_conditions	pass(self)	Set boundary conditions on the device crown maps.
build_bc_pass_rows	pass(self)	Group the BC crown map rows by pass (issue #90).
update_ghost	pass(self)	Update ghost cells: local, MPI, boundary conditions.
advance_one_step_forest	pass(self)	Advance one full step (fast path).
after_topology_build_forest	pass(self)	Copy the forest-built seam and BC maps to the device.
apply_reflux_to_stage_forest	pass(self)	Apply the reflux correction.
begin_stage_forest	pass(self)	Begin an integrator stage (staged path).
close_step_forest	pass(self)	Close a step (staged path).
compute_local_dt_forest	pass(self)	Compute the local stability-limited time step.
end_stage_forest	pass(self)	End an integrator stage (staged path).
fill_seam_from_peer_forest	pass(self)	Fill inter-realm seam ghosts from a peer.
pack_seam_cells_forest	pass(self)	Pack own cells of the cross-rank seam send rows.
unpack_seam_cells_forest	pass(self)	Unpack the cross-rank seam receive rows into own ghosts.
finalize_forest	pass(self)	Finalize the realm.
finalize_mpi_forest	pass(self)	Finalize the FNL MPI handler.
initialize_forest	pass(self)	Initialize the realm.
is_done_forest	pass(self)	Return true if the realm is done.
open_step_forest	pass(self)	Open a step (staged path).
post_step_forest	pass(self)	Post-step IO and diagnostics.
regrid_forest	pass(self)	Regrid on the [amr] frequency cadence (issue #74).
regrid_begin_forest	pass(self)	Regrid phase: begin (issue #110).
regrid_mark_forest	pass(self)	Regrid phase: mark one sweep (issue #110).
regrid_apply_forest	pass(self)	Regrid phase: adapt to the tree marks (issue #110).
regrid_end_forest	pass(self)	Regrid phase: refresh the new state (issue #110).
stages_per_step_forest	pass(self)	Return the integrator stages per step.

Subroutines

accumulate_seam_fluxes

Accumulate the seam face fluxes of stage s, weighted by its SSP coefficient, into the forest's flux register.

The register is host-side: the skins of all the seam faces of the rank are packed on the device in one launch and copied to the host in one transfer, then each is routed by the shared accumulate_seam_skin, as on the CPU.

subroutine accumulate_seam_fluxes(self, s, flux_register, weight, with_gravity_work)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
s	integer(kind=I4P)	in		Runge-Kutta stage.
flux_register	class(flux_register_object)	inout		Forest's flux register.
weight	real(kind=R8P)	in	optional	Weight replacing beta(s) (D_flux, #68 A2).
with_gravity_work	logical	in	optional	Add the seam gravity work (default:

Call graph

flag_troubled_seam_cells

Flag the troubled seam face cells of one stage for the reflux policy troubled (issue #68 E1b), the FNL twin of the CPU flagging: the 6 cells across every owned register face cell (the coarse skin cell, two inside it, three ghosts outside) are gathered from q_aux_gpu to the host (density, velocity, pressure), where the same seam_weno5_sensor and the limiter's host seam factors (issue #50) flag them. Seam cells only: a small gather.

subroutine flag_troubled_seam_cells(self, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
flux_register	class(flux_register_object)	inout		Forest's flux register.

Call graph

add_dissipative_fluxes_dev

Add the dissipative face fluxes of the active directions to the inviscid ones (issue #65), from the auxiliary variables of the stage, on the device: the viscous and heat fluxes when a viscosity or a conductivity is on, the Ohmic ones (MHD) when the resistivity is; the kernel of the order is selected here, never inside the loops.

subroutine add_dissipative_fluxes_dev(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

apply_floors

Apply the MHD positivity floors to the interior of a stage state, before its ghost exchange (issue #41, 3.8).

Euler has no floors (return before any work). MHD: the floored cells of the stage are logged by rank 0 when any; a non-positive density or pressure with the floors disabled (both zero) is fatal, reported with the global minimum density and pressure.

subroutine apply_floors(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	inout		Conservative variables.

Call graph

allocate_gpu

Allocate device data (every dev_alloc checked) and the host staging buffer.

subroutine allocate_gpu(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

check_nonfinite

Stop the run when the committed state holds a non-finite (NaN or infinite) value (issue #45): a run could otherwise finish with NaN fields and exit 0. One device pass over the interior cells, the model selected outside the kernels; the state is copied to the host only to locate a failure.

subroutine check_nonfinite(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

check_glm_ch

Check the GLM cleaning speed against the fastest wave of the committed state (issue #41, section 3.5); a no-op without GLM.

subroutine check_glm_ch(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

compute_conservation

Compute the volume integrals of the conservative variables on the device and save them on their cadence; with self-gravity (issue #81) the row gains W = 1/2 sum rho Phi dV and E + W, Phi solved from the committed state.

subroutine compute_conservation(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

apply_forcing

Hold the bulk momentum on the committed state, once per step (see adam_flume_forcing_object).

subroutine apply_forcing(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

update_hybrid_sensor

Compute the sensor; exchanged where the realm has flagged faces, its ghost cells then exact copies.

subroutine update_hybrid_sensor(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

overwrite_hybrid_fluxes

Compute the sensor and replace the WENO flux by the central one where it allows it (issue #119, T2).

subroutine overwrite_hybrid_fluxes(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

update_wall_model_faces

Flag the block faces of the wall model (issue #119, T3.3), at the start and after each step, so after a regrid.

subroutine update_wall_model_faces(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

close_noslip_fluxes

Zero the inviscid fluxes through no-slip wall faces but the normal momentum, before the dissipative ones are added: the wall is impermeable.

subroutine close_noslip_fluxes(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

overwrite_wall_model_fluxes

The wall-face fluxes of the wall model, after every other dissipative flux (issue #119, T3.3).

subroutine overwrite_wall_model_fluxes(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

update_hybrid_faces

Flag the block faces for the sensor (issue #119, T2.4), at the start and after each step, so after any regrid.

subroutine update_hybrid_faces(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

compute_mu_t

Compute the eddy viscosity from the auxiliary variables, ghost cells included in them (issue #119, T1): the kernel of the model is selected here, never inside the loops.

subroutine compute_mu_t(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

update_sgs_lambda

Update the local time step limit of the eddy viscosity from the committed state (issue #119, T1): compute_local_dt_forest cannot refresh the ghost cells, and adds this limit to its own, which over-estimates the sum of the two per cell.

subroutine update_sgs_lambda(self, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
realm	type(realm_pointer)	inout	target, optional	Sibling realms.

Call graph

compute_turbulence_history

Compute and save the turbulence history of the committed state on its cadence (issue #119, T0.1): the ghost cells are refreshed first and, with sibling realms, the inter-realm seam ghosts are refilled from the peers, as for the div(B) history; the auxiliary variables are recomputed from them.

subroutine compute_turbulence_history(self, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
realm	type(realm_pointer)	inout	target, optional	Sibling realms.

Call graph

compute_divb_history

Compute and save the div(B) norms of the committed state on the diagnostics cadence (issue #41, D-10); a no-op for Euler. The ghost cells are refreshed first and, with sibling realms, the inter-realm seam ghosts are refilled from the peers (update_ghost does not fill them, issue #31): the stencils of the seam-local cells read them. The seam faces flags are rebuilt and copied to the device at every call (6 nb integers): at step 0 the forest may not have registered the seams yet.

subroutine compute_divb_history(self, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
realm	type(realm_pointer)	inout	target, optional	Sibling realms.

Call graph

compute_dflux_norms

Compute the D_flux norms of the committed state (issue #68 A2), the device twin of the CPU procedure: the committed state is copied into the idle first RK stage buffer and its residual evaluated into the second, the seam face fluxes accumulated with weight 1 into the realm's copy of the forest's flux register (host side) and completed across the ranks; the reflux correction of the coarse skin cells of the intra-realm 2:1 faces this rank owns is built on the host over the blocks in use and copied to the device, where compute_dflux_norms_dev reduces the norms. The caller has refreshed the ghost cells of q_gpu and copied the seam faces flags to the device.

subroutine compute_dflux_norms(self, norms)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
norms	real(kind=R8P)	out		The five norms of this rank.

Call graph

compute_q_aux

Compute the auxiliary variables on the device, ghost cells included.

subroutine compute_q_aux(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Conservative variables.

Call graph

copy_cpu_gpu

Copy state and topology (coordinates, maps) from host to device.

bounded (the runtime regrid, issue #75) copies the state of the blocks in use only: the device blocks beyond blocks_number hold nothing the scheme reads. Otherwise the whole block capacity nb is copied (initialization: it also gives the unused device blocks finite values).

subroutine copy_cpu_gpu(self, verbose, bounded)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
verbose	logical	in	optional	Trigger verbose output.
bounded	logical	in	optional	Copy the blocks in use only.

Call graph

wm_filter_to_host

Copy the wall-model filter averages from the device (b, i, j, k, v) to the host arrays (v, i, j, k, b) of the checkpoint (issue #119): a whole-array copy into a host buffer of the device shape (no staging buffer of another extent, issue #31), transposed on the host.

subroutine wm_filter_to_host(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

wm_filter_to_device

Copy the host wall-model filter averages (v, i, j, k, b) to the device (b, i, j, k, v) (issue #119): transposed on the host into a buffer of the device shape, then a whole-array copy.

subroutine wm_filter_to_device(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

copy_gpu_cpu

Copy state and residuals from device to host, the blocks in use only (issue #75: the outputs, the restart files and the non-finite report read nothing beyond blocks_number).

subroutine copy_gpu_cpu(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

copy_phi_gpu

Copy the immersed solids distance function (computed on the host, the solids are static) to the device, transposed from (s, i, j, k, b) to (b, i, j, k, s); the staging buffer has the extent of the device array (issue #31). bounded (the runtime regrid, issue #75) copies the blocks in use only.

subroutine copy_phi_gpu(self, bounded)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
bounded	logical	in	optional	Copy the blocks in use only.

Call graph

refresh_potential

Set the gravitational potential of the current grid on the host (set_potential, issue #81) and copy it to the device, transposed from (i, j, k, b) to (b, i, j, k, 1); the device array is reallocated to the host extent (the block capacity may change at a regrid). Without a potential both are one-element placeholders, never zero bytes (issue #82). Call it after the state is on the device and after every regrid.

subroutine refresh_potential(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

copy_potential_gpu

Copy the host gravitational potential to the device, transposed from (i, j, k, b) to (b, i, j, k, 1); the device array is reallocated to the host extent.

subroutine copy_potential_gpu(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

solve_potential_gpu

Solve the self-gravity potential of the device state q_gpu (issue #81, M8-P3): a device kernel packs the interior density, the host gathers it to rank 0 through the library solver (cuFFT on rank 0's GPU), and the potential, every ghost layer filled, comes back to phi_grav and, for the active blocks only, to the device (unpack_potential_dev); on an AMR grid the multigrid solver runs on the host, its coarsest level on cuFFT (P5b). Optionally the energy sum_cells rho Phi dV (every rank). The density and the potential cross the host every stage (gather to rank 0).

Only the active blocks cross the bus: phi_grav is sized to the block capacity (23270 blocks against 16 in use on an A30 at 32^2), and copying it whole every stage cost 0.73 s per solve, 99.6% of the solve.

subroutine solve_potential_gpu(self, q_gpu, energy)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Device state (b, i, j, k, v).
energy	real(kind=R8P)	out	optional	sum_cells rho Phi dV.

Call graph

destroy

Free device and host data: own device buffers, the FNL helpers (library teardown) and the common data.

subroutine destroy(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

initialize_flume

Initialize the FNL backend: MPI and device (once per process), device budget, common data, FNL helpers, device data, dispatch.

subroutine initialize_flume(self, filename, realms_number)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout	target	The equation.
filename	character(len=*)	in		Input file name.
realms_number	integer(kind=I4P)	in	optional	Realm count; divides the device budget.

Call graph

limit_positivity_dev

Apply the positivity limiter to the device face fluxes of the stage (issue #47, D-9; device twin of the CPU limit_positivity): the model's factors kernel, the ghost exchange of the factors on the device (intra-realm copies and GPU-direct MPI only: a physical-boundary ghost keeps the factor 1 it was allocated with), the blending of the active directions; the counters are reduced over the ranks and logged. With 2:1 AMR seams the seam flux is synchronised as on the CPU (issue #50, adam_flume_seam_sync_object): the seam passes run on the host over the seam cells, fed by device gathers, and scatter their factors and face fluxes back.

subroutine limit_positivity_dev(self, q_gpu, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Conservative variables of the stage.
flux_register	class(flux_register_object)	in	optional	Forest's flux register.

Call graph

seam_lists

Host lists of this rank's seam skin cells (issue #50): coarse (or fine) selects the coarse (fine) seam faces; each entry has its interior cell (i, j, k, b), its face (axis, i, j, k, b) in the flux arrays' indexing and its skin destination (cdst: skin cell of the coarse skins; fdst: compact fine store, fcc: the coarse skin cell covering it, fw: its weight in the coarse mean, 1/4, or 1/2 along the unrefined z of a quadtree, issue #46). The order is block, face, skin cell, as on the CPU.

subroutine seam_lists(self, coarse, ccell, cface, cdst, fine, fcell, fface, fdst, fcc, fw)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.
coarse	logical	in		Lists to build.
ccell	integer(kind=I4P)	out	allocatable	Coarse cells [4, n], faces [5, n].
cface	integer(kind=I4P)	out	allocatable	Coarse cells [4, n], faces [5, n].
cdst	integer(kind=I4P)	out	allocatable	Coarse skin destinations.
fine	logical	in		Lists to build.
fcell	integer(kind=I4P)	out	allocatable	Fine cells [4, n], faces [5, n].
fface	integer(kind=I4P)	out	allocatable	Fine cells [4, n], faces [5, n].
fdst	integer(kind=I4P)	out	allocatable	Fine store slots, covering coarse skin cells.
fcc	integer(kind=I4P)	out	allocatable	Fine store slots, covering coarse skin cells.
fw	real(kind=R8P)	out	allocatable, optional	Fine weights in the coarse means.

Call graph

seam_sync_fluxes_dev

Seam synchronisation, first phase (issue #50; host twin of the CPU seam_sync_fluxes, fed by device gathers).

subroutine seam_sync_fluxes_dev(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Conservative variables of the stage.

Call graph

seam_sync_factors_dev

Seam synchronisation, second phase (issue #50; host twin of the CPU seam_sync_factors): gather the seam cells' stencils, auxiliary variables and face fluxes, recompute their factors with the seam faces overridden on the host, scatter them to the device factors.

subroutine seam_sync_factors_dev(self, q_gpu, dbad, dnonfinite)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Conservative variables of the stage.
dbad	integer(kind=I4P)	inout		Inadmissible backbones count.
dnonfinite	integer(kind=I4P)	inout		Non-finite cells count.

Call graph

seam_sync_theta_dev

Seam synchronisation, third phase (issue #50; host twin of the CPU seam_sync_theta).

subroutine seam_sync_theta_dev(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

seam_sync_blend_dev

Seam synchronisation, last phase (issue #50; host twin of the CPU seam_sync_blend): the seam face fluxes with the seam factor, scattered to the device face fluxes.

subroutine seam_sync_blend_dev(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

save_residuals

Save residuals history (L2 norm of dq on the device, MPI-reduced, rank 0 writes).

subroutine save_residuals(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

save_simulation_data

Save fields, restart, slices and conservation history, each on its own cadence; state copied to host only when saved.

subroutine save_simulation_data(self, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
realm	type(realm_pointer)	inout	target, optional	Sibling realms.

Call graph

set_boundary_conditions

Set boundary conditions on the device crown maps: three passes (rows beyond one, two, three realm faces), crown by crown within each (see the CPU set_boundary_conditions, issue #65 P0).

subroutine set_boundary_conditions(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	inout		Conservative variables.

Call graph

build_bc_pass_rows

Group the rows of the BC crown map by pass, per crown, for set_boundary_conditions (issue #90, F-A'): pass p (rows beyond p realm faces: 1 faces, 2 edges, 3 corners) of crown c is bc_rows_gpu(r1:r2,c), with [r1, r2] = bc_pass_range(:,p,c) (empty when r2 < r1), so each launch visits its own rows only. The rows are the live ones (block > 0) in map order; built from the host map, after every copy of the device map.

subroutine build_bc_pass_rows(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

inflow_df_to_device

List the digital-filter inflow ghost rows from the host map and copy them and their states to the device (issue #119, T3.7): once per step (the states change) and after every copy of the map. The device arrays hold at least one column, so a rank without rows copies no zero-size array (issue #82).

subroutine inflow_df_to_device(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

wall_history_to_host

Copy the committed state to the host for the wall-model history (issue #119, T3.7): the blocks in use only, the state only.

subroutine wall_history_to_host(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

free_inflow_df_rows

Free the device digital-filter inflow rows.

subroutine free_inflow_df_rows(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

update_ghost

Update ghost cells on the device: intra-realm local copies, GPU-direct MPI exchange, boundary conditions, then the positivity blend of the inadmissible face ghosts (issue #50, D2; see the CPU update_ghost).

subroutine update_ghost(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	inout		Conservative variables.

Call graph

advance_one_step_forest

Advance one full step of size dt (fast path: single realm, no AMR seam faces).

subroutine advance_one_step_forest(self, dt)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
dt	real(kind=R8P)	in		Time step from the forest's global reduction.

Call graph

after_topology_build_forest

Copy the host maps the forest built after the realm initialization to the device: the inter-realm seam map and buffers (maps%seam_local_*) and the BC crown map, whose seam rows the forest rewrote to BC_SEAM (issue #37; without it the seam fill kernel reads an unset device map and the BC kernel extrapolates over the seam ghosts).

subroutine after_topology_build_forest(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

apply_reflux_to_stage_forest

Apply the Berger-Colella reflux correction to the committed q_gpu (the forest calls it once per step, after close_step_forest, with the final stage); device twin of the CPU apply.

For every register face whose coarse side this realm and this rank own, the host mismatch slab F_coarse - F_fine_sum (step fluxes, sum_s beta_s F_s) is copied to the device and added, scaled by sgn dt / dx_coarse, to the coarse skin cells.

The scale uses the step the update used, time%dt: on the last step of a time-driven run the realm caps it to land on time_max, and the forest's dt argument is the uncapped value (with it the correction was off by the ratio of the two, 1.3e-9 of the mass on sod-amr; issue #37).

subroutine apply_reflux_to_stage_forest(self, stage, dt, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
stage	integer(kind=I4P)	in		Integrator stage.
dt	real(kind=R8P)	in		Forest time step (unused: see the scale below).
flux_register	class(flux_register_object)	in		Forest's flux register.

Call graph

begin_stage_forest

Begin integrator stage k (staged path): publish the stage and compute its state.

subroutine begin_stage_forest(self, k, K_total, dt, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
k	integer(kind=I4P)	in		Stage index (1..K_total).
K_total	integer(kind=I4P)	in		Forest-wide stage count for this step.
dt	real(kind=R8P)	in		Time step from the forest.
realm	type(realm_pointer)	inout	target, optional	Sibling realms (contract parity).

Call graph

close_step_forest

Close a step (staged path): assemble q, save residuals, advance time, clear the active stage.

subroutine close_step_forest(self, dt)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
dt	real(kind=R8P)	in		Time step from the forest (the local capped value is time%dt).

Call graph

compute_local_dt_forest

Compute the local stability-limited time step on the device, dt = CFL / max(sum_d (|u_d| + a) / dx_d); with GLM, also dt <= CFL / (c_h max sum_d 1 / dx_d) (glm_lambda, host data, issue #41, section 3.5); with dissipative terms the diffusive part 2 nu sum_d 1 / dx_d^2 joins the cell sum (issue #65, D-M4-3, as on the CPU).

subroutine compute_local_dt_forest(self, dt_local)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.
dt_local	real(kind=R8P)	out		Local stability-limited time step.

Call graph

end_stage_forest

End integrator stage k (staged path): residuals on the stage state, then stage assignment.

subroutine end_stage_forest(self, k, K_total, dt, realm, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
k	integer(kind=I4P)	in		Stage index (1..K_total).
K_total	integer(kind=I4P)	in		Forest-wide stage count.
dt	real(kind=R8P)	in		Time step from the forest.
realm	type(realm_pointer)	inout	target, optional	Sibling realms (parity only).
flux_register	class(flux_register_object)	inout	optional	Forest's flux register.

Call graph

reflux_stage

The per-stage reflux ([numerics] reflux_cadence = stage, issue #81 P5c; FNL twin of the CPU reflux_stage): the seam fluxes of stage s (with the seam gravity work) into the realm's register copy with weight 1, completed across the ranks; each coarse seam face's mismatch F_coarse - F_fine goes to the device and corrects the stage residual dq_gpu by sgn / H (apply_reflux_face_dev). The end-of-step reflux is then skipped.

subroutine reflux_stage(self, s)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
s	integer(kind=I4P)	in		Runge-Kutta stage.

Call graph

fill_seam_from_peer_forest

Fill this realm's seam ghosts for peer slot p_idx from the peer's interior, on the active device buffers (q_gpu when stage_active == 0, else the active stage of q_rk_gpu) of both realms.

subroutine fill_seam_from_peer_forest(self, peer, p_idx)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
peer	class(realm_object)	in	target	Peer realm.
p_idx	integer(kind=I4P)	in		Peer slot.

Call graph

pack_seam_cells_forest

Pack the ghost values of the seam send rows of peer slot p_idx from the active device buffer (q_gpu when stage_active == 0, else the active stage of q_rk_gpu) into the host buf, nv values per row: cell copies, or the 2:1 interpolations and restrictions of a refined seam (issues #40, #52).

subroutine pack_seam_cells_forest(self, p_idx, buf)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.
p_idx	integer(kind=I4P)	in		Peer slot (the realm owning the ghosts).
buf	real(kind=R8P)	out		Packed values.

Call graph

unpack_seam_cells_forest

Unpack the host buf into this realm's seam ghosts of the cross-rank receive rows of peer slot p_idx, on the active device buffer, nv values per row (issue #40).

subroutine unpack_seam_cells_forest(self, p_idx, buf)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
p_idx	integer(kind=I4P)	in		Peer slot (the realm owning the cells).
buf	real(kind=R8P)	in		Packed values.

Call graph

finalize_forest

Finalize the realm: close the output files and free device and host data (MPI is finalized once by the forest).

subroutine finalize_forest(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

finalize_mpi_forest

Finalize the FNL MPI handler (called once by the forest after every realm is finalized).

subroutine finalize_mpi_forest(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation (carries no MPI state).

Call graph

initialize_forest

Initialize the realm (issue #35, section 6.1): backend init, IC (or restart) and initial AMR on the host, device topology and state sync, ghost update, initial output, output files open, AMR lock.

subroutine initialize_forest(self, filename, realms_number, memory_avail, nv, verbose)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
filename	character(len=*)	in		Input file name.
realms_number	integer(kind=I4P)	in	optional	Realm count; divides the device budget.
memory_avail	real(kind=R8P)	in	optional	Unused: the budget comes from the device.
nv	integer(kind=I4P)	in	optional	Unused: nv is decided by the physics.
verbose	logical	in	optional	Unused: initialization is always verbose.

Call graph

is_done_forest

Return true if the realm has reached its end.

subroutine is_done_forest(self, done)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.
done	logical	out		True if the realm is done.

Call graph

open_step_forest

Open a step (staged path): time bookkeeping and Runge-Kutta stages initialization.

subroutine open_step_forest(self, dt)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
dt	real(kind=R8P)	in		Time step from the forest.

Call graph

seam_ghosts_to_stages

Copy the ghosts of q_gpu of the end_of_step seams into every stage of q_rk_gpu (issue #113): every stage of the step reads the peer state of the last exchange (the device twin of adam_seam_exchange%seam_ghosts_to_stages).

subroutine seam_ghosts_to_stages(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

post_step_forest

Post-step work: the non-finite state check, the GLM c_h check, the div(B) history (MHD), fields, restart and conservation history on their cadence.

subroutine post_step_forest(self, dt, t, it, do_save_state, do_save_residuals, do_save_restart, do_amr, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
dt	real(kind=R8P)	in		Time step just advanced.
t	real(kind=R8P)	in		Time after the advance.
it	integer(kind=I4P)	in		Iteration after the advance.
do_save_state	logical	in	optional	Unused: cadence is internal.
do_save_residuals	logical	in	optional	Unused: cadence is internal.
do_save_restart	logical	in	optional	Unused: cadence is internal.
do_amr	logical	in	optional	Unused: the forest regrids (regrid_forest).
realm	type(realm_pointer)	inout	target, optional	Sibling realms.

Call graph

sync_dual_energy

Synchronise the total energy and the advected entropy of the committed device state, once per step (issue #91): the device twin of the CPU sync_dual_energy (sync_dual_energy_dev, the same rule), the lifted cells and the energy added summed over the ranks and logged when nonzero.

subroutine sync_dual_energy(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

apply_cooling

Cool the committed device state over the step just advanced, once per step (issue #93): the device twin of the CPU apply_cooling (apply_cooling_dev, or apply_cooling_entropy_dev with the advected entropy, the same rule and the same library arithmetic); the radiated energy summed over the ranks into cooling%radiated.

subroutine apply_cooling(self, dt)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
dt	real(kind=R8P)	in		Time step just advanced.

Call graph

accrete_sinks_gpu

Threshold accretion of the committed device state (issue #94): the cells within an accretion radius of a sink are gathered from the device, accreted on the host by accrete_cells (the arithmetic of the CPU backend) and scattered back; a handful of cells per sink cross the bus.

subroutine accrete_sinks_gpu(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

cooling_tables

Return the cooling tables padded to the fixed size of the device kernels (COOLING_NODES_MAX, issue #93): node temperatures, Lambda, exponents and evolution function in the columns, zero beyond the n_nodes used.

Attributes: pure

subroutine cooling_tables(cooling, tables)

Arguments

Name	Type	Intent	Attributes	Description
cooling	type(flume_cooling_object)	in		Cooling.
tables	real(kind=R8P)	out		Padded tables.

Call graph

regrid_forest

Regrid the realm on the cadence of [amr] frequency (issue #74, M5-P3); the forest calls it after every step, before post_step, on the committed state (single realm: a multi-realm forest calls the phases itself, #110).

A host round trip (D-M5-7): the regrid machinery (markers, tree, transfers, redistribution, maps) is the host library's, so each sweep refreshes the ghost cells on the device (the markers and the prolongation read them), copies q to the host, marks (mark_runtime) and regrids there (adam_object%amr_update with [amr] regrid_prolongation), and, on a changed grid, recomputes the solids' distance function and copies the state, the block coordinates, the maps (a map the new grid no longer has is freed on the device, issue #74 G6) and the distance function back to the device. Then, as on the CPU, the new state is checked and its ghosts and auxiliary variables refreshed. The device arrays are sized by the block capacity nb, so nothing is reallocated; the state and distance function copies move the blocks in use only, packed and transposed on the device (issue #75: the whole capacity cost 2.3-3.1 s per regrid on the development box, whatever the grid). The wall time of the regrid (largest over the ranks) is logged with it. regridded tells the forest to rebuild the 2:1 flux register (host side); the grid is replicated, so every rank returns the same value. The phases are regrid_begin_forest, regrid_mark_forest and regrid_apply_forest per sweep, regrid_end_forest.

subroutine regrid_forest(self, regridded)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
regridded	logical	out		True if the grid changed.

Call graph

regrid_begin_forest

Start a regrid: active on a regrid step of [amr] frequency (issue #110).

subroutine regrid_begin_forest(self, active)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
active	logical	out		The realm marks its blocks at this step.

Call graph

regrid_mark_forest

Sweep sweep of a regrid step: refresh the ghosts on the device, copy q to the host, mark there and gather the marks on the replicated tree (issue #110).

subroutine regrid_mark_forest(self, sweep, marked)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
sweep	integer(kind=I4P)	in		Sweep, from 1.
marked	logical	out		The realm marked its blocks.

Call graph

regrid_apply_forest

Adapt to the marks on the replicated tree, on the host (adam_object%amr_update_apply); on a changed grid, recompute the solids' distance function and copy the state, the block coordinates, the maps and the distance function back to the device, then the potential and the sinks' gravity (issue #110).

subroutine regrid_apply_forest(self, changed)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
changed	logical	out		The tree changed.

Call graph

regrid_end_forest

End a regrid: on a changed grid, check the new state, refresh its ghosts and auxiliary variables and log the wall time of the regrid (issue #110).

subroutine regrid_end_forest(self, regridded)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
regridded	logical	out		The grid changed in some sweep.

Call graph

copy_blocks_transposed

Copy blocks 1:blocks_number of a field between the host a(v, i, j, k, b) and the device a_gpu(b, i, j, k, v), ghost cells included (issue #75). The blocks in use are a strided section of a_gpu (the block is its leading index) but a contiguous slab of a: a device kernel packs (or unpacks) them, transposed, into a scratch buffer of the slab's shape, moved by one contiguous copy. The buffer has the extent of the slab, the host side is the slab itself: no staging buffer of another extent (issue #31). A rank without blocks copies nothing.

subroutine copy_blocks_transposed(blocks_number, ni, nj, nk, ngc, nv, a, a_gpu, to_device)

Arguments

Name	Type	Intent	Attributes	Description
blocks_number	integer(kind=I4P)	in		Blocks to copy.
ni	integer(kind=I4P)	in		Block dimensions, ghosts.
nj	integer(kind=I4P)	in		Block dimensions, ghosts.
nk	integer(kind=I4P)	in		Block dimensions, ghosts.
ngc	integer(kind=I4P)	in		Block dimensions, ghosts.
nv	integer(kind=I4P)	in		Variables number.
a	real(kind=R8P)	inout	target, contiguous	Host field (v, i, j, k, b).
a_gpu	real(kind=R8P)	in	pointer	Device field (b, i, j, k, v).
to_device	logical	in		Host to device, else back.

Call graph

refresh_equilibrium_residual

Recompute the residual of the hydrostatic equilibrium when the grid it describes is gone (issue #73, M7-G2b; device twin of the CPU procedure): the equilibrium is set on the host (ic%set_equilibrium) and copied into a scratch device field, its residual evaluated into dq_e_gpu; at AMR 2:1 seams the equilibrium's seam fluxes are accumulated with weight 1 into the realm's copy of the register, completed across the ranks, and their mismatch stored per face (seam_e) for the reflux to subtract.

subroutine refresh_equilibrium_residual(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

compute_gravity_correction

Compute the fourth-order correction of the gravity energy source into gcor_gpu (issue #73, M7-G2a; device twin of the CPU compute_gravity_correction): the faces of the block, then the face beyond each end of every direction with gravity, by the edge kernel of the space operator's face, selected here outside the kernels.

subroutine compute_gravity_correction(self, q_gpu)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	in		Conservative variables.

Call graph

compute_residuals_weno_dev

Compute the residuals with the WENO space operator on the device: ghost update, auxiliary variables, face fluxes of the active directions, flux difference.

The face fluxes of a null direction are never computed: they keep their zero initialization (dev_alloc). On the staged path (AMR seam faces), the seam face fluxes of every stage are accumulated into the forest's flux register.

subroutine compute_residuals_weno_dev(self, q_gpu, dq_gpu, s, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	inout		Conservative variables.
dq_gpu	real(kind=R8P)	inout		Residuals.
s	integer(kind=I4P)	in	optional	Runge-Kutta stage.
flux_register	class(flux_register_object)	inout	optional	Forest's flux register for reflux.

Call graph

compute_residuals_riemann_dev

Compute the residuals with the weno-riemann space operator on the device (issue #47): as compute_residuals_weno_dev, with the face fluxes of the WENO interpolation, the Riemann solver and the high-order correction. The host selects the kernel of the (model, solver) pair; a pair without kernels is fatal.

subroutine compute_residuals_riemann_dev(self, q_gpu, dq_gpu, s, flux_register)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
q_gpu	real(kind=R8P)	inout		Conservative variables.
dq_gpu	real(kind=R8P)	inout		Residuals.
s	integer(kind=I4P)	in	optional	Runge-Kutta stage.
flux_register	class(flux_register_object)	inout	optional	Forest's flux register for reflux.

Call graph

integrate_rk_ls_dev

Integrate one time step with a low-storage Runge-Kutta scheme on the device.

subroutine integrate_rk_ls_dev(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

integrate_rk_ssp_dev

Integrate one time step with a strong stability preserving Runge-Kutta scheme on the device.

subroutine integrate_rk_ssp_dev(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

rk_assign_stage

Assign the residual of stage s to the Runge-Kutta stage buffer; the solid cells are masked with immersed solids.

subroutine rk_assign_stage(self, s)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
s	integer(kind=I4P)	in		Stage.

Call graph

rk_compute_stage

Compute the state of stage s; the solid cells are masked with immersed solids.

subroutine rk_compute_stage(self, s)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
s	integer(kind=I4P)	in		Stage.

Call graph

rk_compute_stage_ls

Advance low-storage stage s; the solid cells are masked with immersed solids.

subroutine rk_compute_stage_ls(self, s)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.
s	integer(kind=I4P)	in		Stage.

Call graph

rk_update_q

Assemble the committed state of a strong stability preserving step; the solid cells are masked with immersed solids.

subroutine rk_update_q(self)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	inout		The equation.

Call graph

seam_gather_cells

Gather on the host out(v, m) = the first nv components of the device field a_gpu at cells(:, m) (issue #50).

subroutine seam_gather_cells(cells, nv, ngc, a_gpu, out)

Arguments

Name	Type	Intent	Attributes	Description
cells	integer(kind=I4P)	in		Cells (i, j, k, b) [4, n].
nv	integer(kind=I4P)	in		Components.
ngc	integer(kind=I4P)	in		Ghost cells number.
a_gpu	real(kind=R8P)	in		Device field.
out	real(kind=R8P)	inout		Gathered values [nv, n].

Call graph

seam_gather_stencils

Gather on the host the axis stencils of half width s around cells(:, m) (issue #50).

subroutine seam_gather_stencils(cells, nv, ngc, s, q_gpu, out)

Arguments

Name	Type	Intent	Attributes	Description
cells	integer(kind=I4P)	in		Cells (i, j, k, b) [4, n].
nv	integer(kind=I4P)	in		Variables number.
ngc	integer(kind=I4P)	in		Ghost cells number.
s	integer(kind=I4P)	in		Stencil half width.
q_gpu	real(kind=R8P)	in		Device conservative variables.
out	real(kind=R8P)	inout		Stencils [nv, 2 s + 1, 3, n].

Call graph

seam_gather_faces

Gather on the host the device face fluxes at faces(:, m) = (axis, i, j, k, b) (issue #50).

subroutine seam_gather_faces(faces, nv, flx_f_gpu, fly_f_gpu, flz_f_gpu, out)

Arguments

Name	Type	Intent	Attributes	Description
faces	integer(kind=I4P)	in		Faces [5, n].
nv	integer(kind=I4P)	in		Variables number.
flx_f_gpu	real(kind=R8P)	in		X-face fluxes.
fly_f_gpu	real(kind=R8P)	in		Y-face fluxes.
flz_f_gpu	real(kind=R8P)	in		Z-face fluxes.
out	real(kind=R8P)	inout		Gathered fluxes [nv, n].

Call graph

seam_scatter_faces

Set the device face fluxes at faces(:, m) = (axis, i, j, k, b) to the host in(:, m) (issue #50).

subroutine seam_scatter_faces(faces, nv, in, flx_f_gpu, fly_f_gpu, flz_f_gpu)

Arguments

Name	Type	Intent	Attributes	Description
faces	integer(kind=I4P)	in		Faces [5, n].
nv	integer(kind=I4P)	in		Variables number.
in	real(kind=R8P)	in		Fluxes [nv, n].
flx_f_gpu	real(kind=R8P)	inout		X-face fluxes.
fly_f_gpu	real(kind=R8P)	inout		Y-face fluxes.
flz_f_gpu	real(kind=R8P)	inout		Z-face fluxes.

Call graph

seam_scatter_cells

Set the first nv components of the device field a_gpu at cells(:, m) to the host in(:, m) (issue #50).

subroutine seam_scatter_cells(cells, nv, ngc, in, a_gpu)

Arguments

Name	Type	Intent	Attributes	Description
cells	integer(kind=I4P)	in		Cells (i, j, k, b) [4, n].
nv	integer(kind=I4P)	in		Components.
ngc	integer(kind=I4P)	in		Ghost cells number.
in	real(kind=R8P)	in		Values [nv, n].
a_gpu	real(kind=R8P)	inout		Device field.

Call graph

Functions

cooling_time_min

Return the shortest cooling time of the committed device state on this rank (compute_cooling_time_min_dev), for the [cooling].(dt_factor) limiter (issue #93).

Returns: real(kind=R8P)

function cooling_time_min(self) result(t_cool)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.

Call graph

stages_per_step_forest

Return the integrator stages per step: only SSP schemes are stage-splittable (staged path).

Returns: integer(kind=I4P)

function stages_per_step_forest(self) result(K)

Arguments

Name	Type	Intent	Attributes	Description
self	class(flume_fnl_object)	in		The equation.

Call graph