Forest cookbook

This page is the task-oriented companion of the forest reference. Each recipe is a configuration ADAM runs today:

• a picture of what it builds;
• the manifest and realm keys it needs;
• what the machinery does with them;
• what it refuses;
• how to tell from the log that it did the right thing;
• the test that keeps it working.

The figures are drawn to scale by scripts/make_forest_figures.py (standard library only; run it to regenerate docs/public/forest/*.svg).

What "verified" means here

Every recipe names its tests. Unless stated otherwise the oracle is bitwise: the forest run must reproduce the single-realm run on the same cells to the last bit, on the CPU and the FNL (OpenACC) backends. Where a test holds only to round-off or to a tolerance, the recipe says so and why.

Vocabulary

Term	Meaning
realm	One simulation domain with its own INI (grid, numerics, physics), its own block tree and its own partition over the MPI ranks.
forest	A set of realms advanced together with one global time step. A plain INI is a one-realm forest.
manifest	The small INI with a [forest] section that lists the realms and the seams between them.
seam	A face where two realms touch (an inter-realm seam, declared in the manifest), or a 2:1 face inside one realm's tree (an intra-realm AMR seam, created by refinement).
face pair	One [forest.topology.face_N] entry: realm_a/face_a glued to realm_b/face_b.
ghost slab	The ngc layers of ghost cells beyond a realm's seam face, filled from the peer realm.
row	One seam ghost to fill: a copy (same cell size), an interpolate (a fine ghost from coarse cells) or a restrict (a coarse ghost from fine cells).
flux register	The per-face store of the coarse-side flux and the sum of the fine-side fluxes; after the step the coarse cells get the Berger-Colella correction.
cadence	When the seam ghosts are refreshed: before every RK stage (β, stage_coincident) or once per step (α, end_of_step).

Which recipe?

You want	Recipe
write the program that drives the realms	R0
one domain, uniform grid	R1
one domain, finer cells in a region	R2
two domains, same cells, glued	R3
choose how the seam is synchronised, or mix RK schemes	R4
two domains, same cells, different block sizes	R5
two domains, one twice as fine as the other	R6
a seam and refinement inside a realm	R7
any of the above on many ranks	R8
restart a forest	R9
build a split from an existing single-realm input	R10
run any of the above on the GPU	R11
let the realms regrid at runtime, across their seams	R12

---

R0. The driver pattern

Problem. Write the program of an application so that the same executable runs a plain input and a manifest.

Code (src/app/flume/cpu/adam_flume_cpu.F90; the FNL driver adam_flume_fnl.F90 is the same with flume_fnl_object):

type(flume_cpu_object), allocatable, target :: realm(:)          ! 1 for a single input, N for a manifest
type(realm_pointer),    allocatable         :: realm_pointers(:) ! what the forest takes (#96)
type(forest_object)                         :: forest
type(forest_manifest_t)                     :: manifest
...
if (is_forest_manifest(trim(input_file_name))) then
   call read_forest_manifest(filename=trim(input_file_name), manifest=manifest)
   allocate(realm(manifest%realms_number))
   call point_realms
   call forest%simulate_from_manifest(realm=realm_pointers, manifest=manifest)
else
   allocate(realm(1))
   call point_realms
   call forest%simulate(realm=realm_pointers, filename=trim(input_file_name))
endif
contains
   subroutine point_realms
   integer :: i
   allocate(realm_pointers(size(realm)))
   do i = 1, size(realm)
      realm_pointers(i)%ptr => realm(i)
   enddo
   endsubroutine point_realms

What it does. is_forest_manifest reports whether the file has a [forest] section with realms_number >= 1. The realms live in a monomorphic array of the concrete type, and the forest receives an array of realm_pointer: it never indexes a polymorphic array, which nvfortran addresses wrongly (#96). simulate and simulate_from_manifest initialize the realms, run the time loop (fast path or staged path, R1 and R2) and finalize them.

A single-realm application can be smaller: RIPPLE's driver (src/app/ripple/cpu/adam_ripple_cpu.F90) declares one realm and one pointer and calls forest%simulate only.

Reference. Forest concepts; RIPPLE.

R1. One realm, uniform

A plain INI, no manifest: the forest has one realm and takes the fused fast path (advance_one_step_forest); no seam machinery runs.

[amr]
max_level      = 2
ratio          = 8        ; octree
iu_ref_levels  = 2        ; 4 x 4 x 4 blocks
markers_number = 0

• Blocks. iu_ref_levels = L gives 2^L blocks per refined axis (x, y, z on an octree; x, y on a quadtree, ratio = 4), each with ni x nj x nk cells.
• Verified by every single-realm regression case (FLUME sod-x, PRISM rmf, ...).

R2. One realm with AMR: intra-realm 2:1 faces

A refinement marker creates blocks one level finer; where a level-l block meets a level-l+1 block the tree has a 2:1 face. No manifest is involved.

What happens at a 2:1 face:

• Ghosts. The fine block's ghosts that overlap the coarse block are interpolated with the [amr] seam_ghost_fill regime; the coarse block's ghosts that overlap fine cells take the mean of the 2x2x2 fine cells.
• Reflux. The face is registered (SEAM_KIND_INTRA_REALM_AMR); after the step the coarse cells get the Berger-Colella correction (FLUME: [numerics] reflux = .true.).

seam_ghost_fill	Order	Footprint	Use
tricubic (default)	q = 4	4x4x4 coarse cells	production
restriction-compatible	q = 2	3x3x3	exact restriction of the interpolant (R∘P = I); diagnostics
injection	0	the anchor cell	legacy, comparisons

Requirements. The first two are checked at initialisation:

1. On an octree (ratio = 8) with a null z axis, nk >= 4 (the tricubic footprint is 4 cells along every refined axis). A quadtree (ratio = 4) never refines z, so any nk works, nk = 1 included (#46; the NVF and GMP backends still refuse quadtree AMR).
2. Even ni/nj/nk along every refined, non-null axis when 2:1 refinement is possible (markers and max_level > iu_ref_levels): an odd count is refused (#39; it used to give NaN silently).
3. At least 2 ngc cells (6 with WENO5) along the same axes. The coarse ghosts beside a finer block are restricted from 2 ngc fine cells; a thinner block makes the outer layers read its own ghosts, which the same exchange is still filling (stale, silently, depending on the exchange order: 0.2 in density between 1 and 2 ranks on the step forest). Refused at initialisation (#66).
4. At least 4 cells per block on every axis, null axes included, for the coarse-fine interpolation footprint.

Log. forest: registered intra-realm AMR seam faces: +N.

PRISM. A 2:1 face injects an O(h^p) div(B) source that refinement reduces but time does not (#29, accepted). The run-time monitor [IO] seam_divB_tol turns it into a warning or a stop; see the reference.

Verified by FLUME sod-amr (a 2:1 face crossed by a shock), amr-periodic-reflux (V3: conservation to round-off, the negative control without reflux, and the odd-cell refusal), MHD MV-11; PRISM rmf-amr, rmf-amr-fd, rmf-amr-fd-pulse.

R3. Two realms, mirror seam, lined-up blocks

The manifest (any file name; the apps recognise it by its [forest] section):

[forest]
realms_number = 2

[realm.1]
ini = sod-2realm-r1.ini        ; relative to the working directory, not to the manifest

[realm.2]
ini = sod-2realm-r2.ini

[forest.topology]
inter_realm_faces_number = 1

[forest.topology.face_1]
realm_a          = 1
face_a           = +x          ; +x | -x | +y | -y | +z | -z
realm_b          = 2
face_b           = -x
coupling         = mirror      ; the default
coupling_cadence = stage_coincident

The realm INIs are complete inputs. For a split of a single-realm domain, each realm keeps the cell size and the block layout and takes its part of the extent:

; realm 1, x in [0, 0.5]                     ; realm 2, x in [0.5, 1]
[grid]                                         [grid]
ni     = 25          ; 50 / 2                  ni     = 25
emin_x = 0.0                                   emin_x = 0.5
emax_x = 0.5                                   emax_x = 1.0
[IO]                                           [IO]
output_basename  = sod-2realm-r1               output_basename  = sod-2realm-r2
restart_basename = sod-2realm-r1-restart       restart_basename = sod-2realm-r2-restart

What happens:

• Ghosts. Every seam ghost is a copy of the peer cell with the same centre. The forest builds the rows at initialisation from the replicated trees; it fills the realm 1 slab from realm 2 and the realm 2 slab from realm 1, edge and corner ghosts included.
• Register. One register face per realm_a seam block. On a mirror seam both sides compute the same flux, so the correction is exactly zero: the forest prints forest: reflux max|F_coarse-F_fine_sum| only when it is not.
• Physical faces. A realm's faces that are not on a seam keep their [bc_*] boundary conditions.

Requirements.

1. The same cell size on both sides, with cell centres that coincide across the seam.
2. The two seam faces cover each other.
3. One cell size along each realm's seam face (no refinement touching the seam face).

Log. forest: realm R seam rows: L local, R received, S sent (per rank).

Verified by FLUME multirealm/check.sh leg 1 (Sod) and the regression cases sod-2realm (equivalent_to sod-x) and rj2a-2realm (MHD, equivalent_to rj2a-x); PRISM rmf-2realm* and rmf-2realm-fd-pulse (div(B) = div(D) = 0 across the seam).

R4. Seam cadence: α or β

coupling_cadence	Ghosts refreshed	Requires	Gives
stage_coincident (β)	before every RK stage's residual, from the peer's stage buffer	the same scheme_time, RK scheme, stage count and nv on both sides (checked)	the seam has the time order of the interior; the split reproduces the single realm (bitwise in every FLUME split)
end_of_step (α, default)	once, at the top of the step, from the peer's committed state, for every stage of the step	nothing	first-order seam coupling in time; the realms may use different RK schemes (asymmetric K)

Choose β whenever both realms use the same time integrator. It is the cadence under which a split reproduces the single-realm run, and the only one that keeps a PRISM 1:1 seam divergence-free (#31: under α the substage ghosts are stale and div(B) leaks).

Choose α for different RK schemes per realm (for example SSP-33 next to SSP-54): the realm with fewer stages idles the trailing ones.

Rules.

1. β on a pair whose realms differ in scheme, stages or nv stops at initialisation, naming the field; it is never silently downgraded.
2. All face pairs joining the same two realms must declare the same cadence.

Verified by PRISM rmf-2realm (α), rmf-2realm-asymK (α, K = 3 and 5), rmf-2realm-stagesync (β, the single-realm golden within the digest tolerances), rmf-2realm-fd-pulse (β div-free, with α as the leaking negative control). FLUME multirealm/check.sh leg 14 runs α against β; every other FLUME forest test uses β.

What α costs in FLUME (multirealm/check.sh leg 14, #113): a vortex moving along a 1:1 seam, 120 steps at CFL 0.4. The α run differs from the β run by $4.5\cdot {10}^{-2}$ at most (relative to each field's largest value, in the cells next to the seam), $1.5\cdot {10}^{-2}$ at CFL 0.2: the gap shrinks with the time step (ratio 3.05 for a halved step), the signature of the lagged coupling. Its restart is bitwise.

α before #113

Until #113 the α fill reached the committed state only, never the Runge–Kutta stage buffers the residuals are computed on: the stages read seam ghosts no fill wrote, and α and β differed by 18–45 % next to the seam after four steps. FLUME copies the α seam ghosts into every stage since #113; PRISM does not yet, so in PRISM keep β whenever both realms use the same integrator.

R5. Mirror seam between blocks that do not line up

The two realms keep the same cells but use different block sizes along the seam: for example realm 2 with one block level fewer and twice the cells per block. Nothing changes in the manifest (#51).

; realm 1: 4 blocks per axis                ; realm 2: 2 blocks per axis, the same cells
ni = 25   nj = 4   nk = 4                      ni = 50   nj = 8   nk = 8
iu_ref_levels = 2                              iu_ref_levels = 1

What happens:

• Ghosts. They are rows per cell, so block boundaries do not matter.
• Register. At initialisation the forest intersects each register face with the realm_b seam leaves, in cells. Each realm_b block face gets one overlap row per register face it touches (maps%seam_overlap); at every accumulation its skin is scattered into those faces (flux_register%accumulate_fine_overlaps). A lined-up block has one row covering its whole skin, which is exactly R3.

Requirements. The overlaps must be cell-aligned and cover each register face exactly; otherwise the forest stops ("the cells of realm A and realm B do not line up across the seam", or "the seam block ... faces N of its M seam cells").

Verified by:

• FLUME multirealm/check.sh leg 5: Sod split along x and along z with misaligned blocks, bitwise, 2 and 4 ranks.
• MV-14 leg 5: the MHD rotor on an octree, misaligned both ways, bitwise on all 9 fields. This is the test with teeth: with the overlap offsets forced to zero the density goes negative at step 2.
• Regression sod-2realm-blocks (equivalent_to sod-x).
• PRISM rmf-2realm-fv-pulse-blocks: on the FV path the register closes exactly (mismatch 0). The fields agree to 1e-13 only, because PRISM FV changes at round-off with the block layout alone.

R6. Refined seam: a 2:1 jump between realms

One realm is exactly twice as fine as the other along every axis (#52):

[forest.topology.face_1]
realm_a          = 1
face_a           = +x
realm_b          = 2
face_b           = -x
coupling         = refined           ; the coarse side is found from the cell sizes
coupling_cadence = stage_coincident

; realm 1 (coarse), x in [0, 0.75]          ; realm 2 (fine), x in [0.75, 1]
ni = 36   nj = 4   nk = 4                      ni = 12   nj = 4   nk = 4
iu_ref_levels = 2      ; dx = 1/192            iu_ref_levels = 3      ; dx = 1/384
seam_ghost_fill = tricubic ; [amr]             seam_ghost_fill = tricubic ; [amr]

What happens:

• Rows. The seam reuses the intra-realm 2:1 formulas. A fine-side ghost is an interpolate row: the owner of the coarse cells evaluates the seam_ghost_fill interpolant around the coarse cell containing the ghost. A coarse-side ghost is a restrict row: the mean of the 2x2x2 fine cells under it. Rows whose cells live on the same rank travel as messages to self.
• Register. One register face per coarse seam block. Its four fine blocks are each restricted 2:1 into their quadrant (SEAM_KIND_INTER_REALM_REFINED).
• Time. No subcycling: both realms take the global time step.

Requirements. Each is checked, with a named error:

1. A cell-size ratio of exactly 2 along every axis.
2. The same block cell counts along the seam (the tangential n).
3. The same [amr] seam_ghost_fill in both realms.
4. Even ni, nj, nk in the fine realm.
5. Nested blocks: every fine seam block covers one quadrant of one coarse seam block face.
6. Declaring such a pair mirror is refused ("coupling = mirror joins cells of the same size").

Log. forest: face_pair 1 is a 2:1 seam, realm 1 coarse, realm 2 fine (issue #52).

PRISM. The 2:1 seam carries the same O(h^p) div(B) source as an intra-realm 2:1 face (#29), and the [IO] seam_divB_tol monitor covers it.

Verified by:

• FLUME multirealm/check.sh leg 4: sod-amr split at its 2:1 face, along x and along z, bitwise, CPU np 2-4 and FNL.
• MV-14 leg 4: RJ2a on the same cells, all 9 fields bitwise.
• Regression sod-amr-refined (equivalent_to sod-amr).
• PRISM rmf-2realm-fd-pulse-refined: bitwise on all 27 fields, the #29 source unchanged, the monitor fires.

R7. Both families: a mirror seam and AMR inside a realm

A realm of a forest may carry its own refinement, as in R2, away from its seam face. The register then holds both kinds of face, the intra-realm 2:1 faces first, and one reduction completes them.

Requirement. Refinement must not reach the seam face: each realm's seam face keeps one cell size ("realm R has seam cells of different sizes (refined blocks along the seam)"). To put a resolution jump on the seam, use R6.

Verified by FLUME multirealm/check.sh leg 2 and the regression case sod-amr-2realm (equivalent_to sod-amr). Before the register composed the two kinds, the 2:1 face got no reflux.

R8. Many ranks

Nothing to configure. Each realm is partitioned over all ranks on its own (Morton order), so the two sides of a seam may live on different ranks (#40):

• Rows. Every rank enumerates every seam row from the replicated trees, in one canonical order. It keeps local rows (a direct copy, device to device on FNL), receive rows and send rows. The sends and receives match one to one with no index exchange (point-to-point MPI, tag 4040).
• Register. It is replicated and completed by a reduction per face.

Check. The seam rows log line: on an x split the rows are local, while on a z split at 2 ranks they are all received and sent.

Cost. Set ADAM_SEAM_TIMING=1 (mpirun -x ADAM_SEAM_TIMING) to print, at the end, the time per phase of the seam fill and of the reflux (#53). On FNL the cross-rank rows go through host buffers; the fills of all pairs are posted together and completed by one wait. On medooza (A30, 2-4 ranks) the transfer is at most about 3.5% of the step; a large seam imbalance means one rank reaches the fill late (load imbalance), not a slow exchange.

Verified by FLUME multirealm/check.sh legs 1-5 at np 2, 3 and 4 (leg 3: the z split equals sod-z bitwise on 1 and N ranks).

A rank may own no block of a realm. Every collective in the step must then still be entered by that rank, so a code change that skips work on empty ranks must keep the collectives. The PRISM FNL residuals once hung this way; rmf-2realm-fd-pulse-refined at np 2 covers it.

R9. Restart

Each realm restarts from its own files, with the usual keys in its own INI:

[IO]
restart_save     = 100                      ; write every 100 steps
restart          = .false.                  ; .true. to resume
restart_basename = sod-2realm-r1-restart    ; DIFFERENT in every realm

Give every realm its own restart_basename (and output_basename). Two realms writing to the same name overwrite each other's files.

The restarted realms load their trees and fields, and the forest builds the seams from the restored trees, so a forest that regridded at runtime restarts on its current grid, mixed levels along a seam included. Restart on the rank count the run was saved with (fewer ranks stop the run, #109). The restart is bitwise under either cadence (α since #113: the forest fills α seams at the top of every step, from the state the restart files hold).

Verified by MV-14 legs 2 and 3: a 2-realm run stopped at half its steps and restarted equals the straight run bitwise, with byte-identical histories; leg 3 does the same on an AMR box with a non-zero psi. multirealm/check.sh leg 13 (#110 P5) restarts a forest whose realms regrid, at a step that coarsens and leaves 2:1 faces on the seam, and runs past a later regrid that changes the grid: bitwise, histories identical, on CPU and FNL. Leg 14 (#113) restarts an α forest: bitwise, histories identical, on CPU and FNL.

R10. Splitting an existing input

src/tests/flume/verification/mhd/multirealm/make_split.py splits a single-realm input in two at the centre of x, halving ni and writing the manifest:

make_split.py single.ini out-dir name                       # R3: lined-up blocks
make_split.py single.ini out-dir name --coarse-blocks 2     # R5: realm 2 on blocks twice as large
make_split.py single.ini out-dir name --set time.it_max=50  # override a key in both realms

For a refined split (R6), start from a single-realm run with a box marker. Give the coarse realm the coarse part, unrefined; give the fine realm the refined part, built uniform one level finer (iu_ref_levels + 1, cell counts chosen so that dx halves). Drop the markers in both. verification/multirealm/sod-amr-refined*.ini and src/tests/prism/regression/rmf-2realm-fd-pulse-refined/ are worked examples.

R11. Forest on FNL

Problem. Run a forest on the GPU backend.

Input. The same manifest and realm inputs as on the CPU, run with the FNL executable (mpirun -np 2 exe/adam_flume_fnl forest.ini). The forest machinery is the same on both backends: the multi-realm verification legs run on CPU and FNL, and the regression suites hold FNL goldens for the forest cases (sod-2realm, sod-2realm-blocks, sod-amr-2realm, sod-amr-refined; PRISM rmf-2realm, rmf-2realm-asymK, rmf-2realm-stagesync).

What changes on FNL (the realm-contract hooks an FNL backend overrides, src/app/flume/fnl/adam_flume_fnl_object.F90):

• after_topology_build_forest: the forest builds the inter-realm seam maps on the host after the realms are initialized; this hook copies them, and the boundary-crown map whose seam rows the forest rewrote, to the device. Without it the seam-fill kernel reads an unset device map.
• fill_seam_from_peer_forest, pack_seam_cells_forest, unpack_seam_cells_forest: the seam ghosts are filled on the device buffers of both realms (q_gpu, or the active Runge-Kutta stage), and the cross-rank rows are packed on the device and exchanged through a host buffer.
• apply_reflux_to_stage_forest: the register's mismatch is copied to the device and applied to q_gpu.
• finalize_mpi_forest: finalizes mpih_fnl instead of mpih; the forest calls it once, after every realm is finalized.
• The device budget is divided among the realms of the forest and the ranks sharing the device.

Verified by the FNL regression goldens of the forest cases and the multi-realm legs on FNL (multi-realm runs).

Reference. GPU; forest concepts.

---

R12. Runtime AMR on a forest

Problem. Let the realms of a forest refine and coarsen as the flow moves, a refined region crossing or running along a seam (#110).

Input. Each realm keeps its own [amr] section with frequency > 0 and its markers, exactly as a single realm (the FLUME runtime-AMR guide); the realms may use different markers, levels and cadences. The seam declares how its two sides may differ:

[forest.topology.face_1]
realm_a          = 1
face_a           = +x
realm_b          = 2
face_b           = -x
coupling         = auto              ; mirror, refined, or auto: decided from the cells at every topology build
coupling_cadence = stage_coincident  ; beta (R4)

What the machinery does. At a regrid step the forest runs the realm phases itself: every realm marks, the forest balances the marks across the seams on the replicated trees (the same on every rank, checked by a checksum), every realm adapts, and the inter-realm topology (register faces, seam ghost rows, boundary crown, device maps) is rebuilt from the new trees and every seam refilled from its peer, sweep after sweep. The balance rules compare cell sizes:

coupling	Along each seam side	Across the seam
mirror	one level (a refine vote on one seam block refines the whole side)	equal cells
refined	one level	the declared 2:1 jump
auto	levels may differ face by face, as inside a realm	every pair of neighbouring blocks (edge and corner neighbours included): equal cells or 2:1

An auto seam whose two sides hold one cell size each resolves to mirror or refined at every topology build, so a seam may turn 2:1 and back; with several sizes on a side it stays auto and every structure is built face by face (a 2:1 face may have either realm coarse).

Requirements. A 2:1 face across realms needs nested blocks (the fine blocks quadrants of the coarse one, halves on a quadtree), even block cell counts along the axes the trees refine, the same tree kind ([amr] ratio) in both realms and a seam normal to a refined axis: on a quadtree any nk works, and a 2:1 seam across $z$ is refused (#71). The 6-point seam fills and restrictions need ni, nj, nk >= 6 on the realm they read.

Log. forest: regrid sweep S, marks (refine/derefine) per realm: ... ; balanced: ... for every sweep (rank 0), flume: regrid at step N: blocks a -> b, R refined, C coarsened, S sweeps per realm, and at every topology build forest: ...: mixed levels along the seam, E faces of equal cells and F 2:1 faces for an auto seam with mixed levels. With ADAM_SEAM_TIMING=1 the timing report adds forest regrid timing (the regrid, the balance, the topology rebuilds and the seam refills, with the count of rebuilds) and forest rebuild phases (the coupling checks, the flux register, the seam ghost rows with the boundary crown, the backend hook, and the register face count).

Verified by multirealm/check.sh legs 7-12 on CPU, legs 8, 9, 11 and 12 on FNL (legs 7 and 10 on CPU only):

• leg 7: a realm whose regrids keep its grid (the forest still fills the seams before marking), bitwise to the single-realm run;
• leg 8: Sod split at the diaphragm into two auto realms regridding by a Löhner marker, against the single realm with the same marker: the seam turns 2:1 and back, bitwise with seam_ghost_fill = injection;
• leg 9: a mirror seam between realms at different levels (equal cells), mass conserved to round-off;
• leg 10: a registered refinement criterion against the built-in box marker it reproduces, on one realm and on a forest, bitwise;
• leg 11: a box next to the seam in each realm, so the seam holds 2:1 faces in both directions, against the single realm, bitwise with injection, and with the topology rebuilt every step;
• leg 12: a vortex moving along an auto seam in a closed box, both realms regridding: the seam turns mixed, the mass is conserved to round-off, bitwise on 1 and 2 ranks and with the topology rebuilt every step;
• leg 13 (R9): a restart across these regrids.

Reference. Forest concepts (runtime regridding, cross-realm balance, mixed levels along a seam); writing a refinement criterion.

---

Diagnostics

Log line	Meaning
forest: realm R seam rows: L local, R received, S sent	rows this rank fills locally, receives and sends for realm R
forest: face_pair N is a 2:1 seam, realm C coarse, realm F fine	a refined pair was accepted
forest: registered intra-realm AMR seam faces: +N	2:1 faces inside the realms (R2, R7)
forest: reflux max|F_coarse-F_fine_sum| = ...	printed only when non-zero: expected on 2:1 faces, never on a mirror seam
forest timing (max over ranks, s): ..., forest regrid timing ..., forest rebuild phases ..., forest seam timing realm R ...	with ADAM_SEAM_TIMING=1; the regrid lines price the runtime regrid of a forest (R12)
forest: regrid sweep S, marks (refine/derefine) per realm: ...; balanced: ...	a runtime regrid sweep, the marks before and after the cross-realm balance (R12)
forest: ...: mixed levels along the seam, E faces of equal cells and F 2:1 faces	an auto seam with several levels on a side, built face by face (R12)
WARNING: seam div(B) ... exceeds [IO].seam_divB_tol	PRISM, with the monitor armed (R2, R6)

Refusals and fixes

Message (abridged)	Cause	Fix
only coupling = mirror, refined or auto is implemented	periodic / interpolate	reserved names; before #52 they silently ran as mirror
coupling = mirror joins cells of the same size, the seam cells differ	different cell sizes on a mirror pair	equal cells, or coupling = refined for a 2:1 jump
coupling = refined needs ... (a ratio of exactly 2, the same block cell counts along the seam, the same seam_ghost_fill, even counts, nested blocks)	an R6 requirement	see R6
realm R has seam cells of different sizes	refinement reaches the face of a mirror or refined seam	keep refinement away from the seam (R7), put the jump on the seam (R6), or coupling = auto (R12)
the seam block of realm R at ... faces cells of realm P that are neither one size equal to, half or twice its own / is not covered by nested blocks	an auto seam with a jump other than 1:1 or 2:1, or 2:1 faces that do not nest	let the forest balance it (runtime AMR), or align the realms' blocks
the seam marks did not settle in 64 passes: a refinement next to a seam needs the peer realm finer than its [amr] max_level allows	the cross-realm balance hit a peer's max_level	raise the peer's max_level, or lower this realm's
the cells of realm A and realm B do not line up / faces N of its M seam cells	misaligned cells, or faces that do not cover each other	align the extents to the cell size; make the seam faces coincide
the seam ghost (i,j,k) of block b of realm R does not match the cells of realm P ...; ghost centre ..., peer cell ...	a peer cell of the wrong size or position	as above
stage_coincident requires equal scheme_time / rk_scheme / physics nv / K	β on unlike realms	end_of_step, or align the integrators
conflicting coupling_cadence between realm A and realm B	two pairs between the same realms with different cadences	one cadence per realm pair
[amr] ratio = 4 (quadtree) with AMR markers ... on the NVF and GMP backends	#46: their ghost kernels are octree-only	ratio = 8, or the CPU/FNL backend
[grid].(ni)=+7 is odd: 2:1 refinement ...	#39	even counts along refined axes
[grid].(nj)=+4 is less than 2*ngc = +6 ...	#66: thin blocks read stale ghosts at 2:1 seams	at least 2 ngc cells along refined axes
(positivity_limiter)=cell is not supported on multi-realm runs	the FLUME limiter	no limiter in a forest

Current limits

• Topologies. The schema takes any number of realms and face pairs; the verified forests have two realms and one face pair, and the three-realm forward-facing step (static refinement).
• Runtime AMR across seams. Inter-realm 2:1 faces work on every tree kind since #71, but the runtime-AMR legs (balance, mixed levels) run on octrees; a seam cannot wrap a periodic axis (two faces of the same two realms at opposite ends) is not supported, and not refused either; α seams are not verified in FLUME (R4).
• Time stepping. There is no subcycling: one global time step, set by the most restrictive realm.
• FLUME limiter. The positivity limiter is refused in a forest.
• FNL cost. On FNL the cross-rank seam rows go through the host on every fill (#53).
• Divergence at 2:1 seams. A 2:1 seam, inter- or intra-realm, is not divergence-free for collocated B in PRISM (#29, accepted, monitored).
• Reflux by app. FLUME accumulates the seam fluxes of every stage weighted by its SSP coefficient. PRISM accumulates its final stage (α.r1), and only on the FV path: the FD path does not use the register.

Verification map

Scenario	FLUME	PRISM	Oracle
R2 intra-realm 2:1	sod-amr, amr-periodic-reflux (V3), MV-11	rmf-amr*	goldens; conservation to round-off
R3 mirror, lined up	leg 1, sod-2realm, rj2a-2realm	rmf-2realm, rmf-2realm-fd-pulse	bitwise; div-free
R4 α / β / asymmetric K	leg 14 (α vs β); the others β	rmf-2realm, -asymK, -stagesync, -fd-pulse	goldens; β matches the single realm; α within a lag-sized bound of β, halving with the step
R5 misaligned blocks	leg 5, MV-14 leg 5, sod-2realm-blocks	rmf-2realm-fv-pulse-blocks	bitwise; register exact
R6 refined 2:1	leg 4, MV-14 leg 4, sod-amr-refined	rmf-2realm-fd-pulse-refined	bitwise
R7 both families	leg 2, sod-amr-2realm	—	bitwise
R8 many ranks	legs 1-5 at np 2-4, leg 3 (z split)	np 2 in every case	bitwise
R9 restart	MV-14 legs 2-3, leg 13 (across regrids), leg 14 (α)	—	bitwise, histories identical
R12 runtime AMR on a forest	legs 7-12	—	legs 7, 8, 11 bitwise vs the single realm (8, 11 with injection); 9 conservation; 10 the registered criterion bitwise vs the built-in marker; 12 conservation, ranks and hook bitwise

"leg N" is src/tests/flume/verification/multirealm/check.sh (--legs legN; leg 4 needs leg 2, leg 5 needs legs 1 and 3); "MV-14 leg N" is src/tests/flume/verification/mhd/multirealm/check.sh (--legs mv14-N; leg 2 needs leg 1). FLUME case names are regression cases under src/tests/flume/regression/, PRISM ones under src/tests/prism/regression/.