Skip to content

Forest (multi-realm) ​

ADAM's forest is the orchestrator that drives more than one independent simulation domain — a realm — through a synchronized time loop. Realms share a global timestep but may differ in spatial grid, integrator stride, and spatial operator. Inter-realm seams (faces where two realms touch) are coupled via ghost-cell exchanges whose cadence is selected per seam in the manifest.

Two flagship use cases motivate the multi-realm machinery:

  • Asymmetric integrator stride — one realm advances on SSP-RK-3 for a stiff region while another uses SSP-RK-5 for a region requiring higher temporal accuracy, both with the same global dt.
  • Accuracy-driven spatial decomposition — one realm uses WENO-7 near shocks while another uses FD-centered in smooth regions; both run the same RK scheme but with operators tuned to their local physics regime.

The first scenario requires the α (end-of-step) seam-coupling cadence, the AMReX-aligned default. The second is best served by β (stage-coincident), the opt-in alternative that recovers single-realm temporal accuracy at the seam.

Looking for a recipe?

The forest cookbook walks through every configuration available today (mirror and 2:1 seams, misaligned blocks, AMR inside a realm, cadences, many ranks, restart), with figures, input snippets, refusals and the tests that cover each one. This page is the reference.

The manifest ​

A forest is declared by a small INI file. PRISM and other apps auto-detect a manifest by looking for a [forest] section; a plain single-realm INI runs as a forest of one realm. Without 2:1 AMR faces each step takes the fused fast path (advance_one_step_forest); with a 2:1 face it takes the staged path, so the seam fluxes reach the flux register and the reflux fires (evolve_one_step, adam_forest_object.F90).

ini
; forest.ini — minimal two-realm manifest

[forest]
realms_number = 2

[realm.1]
ini = realm_1.ini       ; opened as given: relative to the working directory, not to forest.ini

[realm.2]
ini = realm_2.ini

[forest.topology]
inter_realm_faces_number = 1

[forest.topology.face_1]
realm_a          = 1
face_a           = +x                  ; realm 1's +x face is glued to realm 2's -x face
realm_b          = 2
face_b           = -x
coupling         = mirror              ; mirror | refined (2:1, issue #52)
coupling_cadence = end_of_step         ; end_of_step (default, α) | stage_coincident (β)

Schema summary:

SectionKeyRequiredDescription
[forest]realms_numberyesNumber of realms in the forest (≥ 1).
[realm.N]iniyesPer-realm INI path, opened as given: relative to the working directory of the run, not to the manifest's directory.
[forest.topology]inter_realm_faces_numbernoCount of inter-realm seams; absent means no inter-realm coupling.
[forest.topology.face_N]realm_a, realm_byes1-based realm indices on each side of the seam.
face_a, face_byesFace codes: +x/-x/+y/-y/+z/-z.
couplingnomirror (default; same cell size, pass-through copy), refined (a 2:1 resolution jump, issue #52) or auto (decided again at every topology build from the cells of the two sides: mirror or refined when each side holds one cell size, else face by face, the levels free to differ along the seam; the seam of a forest whose realms regrid at runtime, issue #110). periodic and interpolate are reserved and refused at initialization.
coupling_cadencenoend_of_step (default, α) or stage_coincident (β).

Runtime regridding (issue #74). After every step, before the per-step diagnostics and output, the time loop calls forest%regrid (so the fields, histories and restart files of a step describe the grid the next step runs on, and a run restarted at a regrid step continues exactly): every realm's regrid_forest hook may regrid it (default: never), and when a tree changed the forest rebuilds the intra-realm 2:1 faces of the flux register (the fused single-realm fast path is chosen per step from the register, so a run that gains or loses 2:1 faces switches path). On a multi-realm forest (issue #110) the forest keeps the manifest and, after any realm regridded, re-runs populate_inter_realm_topology, which is idempotent: the inter-realm register faces, the seam ghost rows (local, MPI send and receive), the BC_SEAM rows of the boundary crown and their device copies (after_topology_build_forest) are rebuilt from the new trees. make_local_maps_bc keeps BC_SEAM on the face rows of every seam face (maps%seam_face), so the crown a realm rebuilds during its own regrid never lets a boundary condition overwrite the seam ghosts. Every seam is then filled from its peer on the committed state, both cadences; before the realms mark, the seams are filled as well, so ghost-reading markers see the peer across a seam (both fills only when a realm regrids at runtime: a checkpoint holds the ghost cells). The environment variable ADAM_FOREST_REBUILD_TOPOLOGY=1 rebuilds the topology after every step whether or not a realm regridded, a test hook: src/tests/flume/verification/multirealm/check.sh passes all its legs with it, bit for bit. Cross-realm balance (#110 P2). regrid_forest marks and adapts a realm in one call, so on a forest whose realms regrid at runtime the forest runs the realm phases itself (regrid_realms): regrid_begin_forest, then per sweep regrid_mark_forest (each realm's markers, the marks gathered on its replicated tree, adam_object%mpi_gather_refinement_needed), balance_seam_marks, regrid_apply_forest (adam_object%amr_update_apply) for every realm whose tree carries a mark, and finally regrid_end_forest. The balance works on the replicated trees, the same on every rank, to a fixed point with each realm's own 2:1 sanitize. A mirror or refined seam keeps one level per seam side (a refine vote on a leaf along the seam refines the whole side; the side derefines only if every leaf along it agrees), and across the seam equal cells (mirror) or the declared 2:1 jump (refined). An auto seam is balanced face by face (#110 P3, balance_mixed_pair): every seam leaf against the leaves of the peer realm facing it, their predicted cells (after the marks) equal or in a ratio of 2, so the levels may differ along the seam as they do inside a realm. The rules compare cell sizes, not levels: each realm counts levels from its own root, so equal cells may sit at different levels (leg 9 of multirealm/check.sh: a mirror seam between realms at levels 2 and 1). A side below what the rule asks is raised (its derefinement dropped, or refined below max_level, a realm that does not regrid at that step included), else the refinement of the side above is dropped. Every rank balances its own copy of the replicated trees; a checksum of the marks of every realm is compared over the ranks after each balance (check_marks_agree) and stops the run if they differ, and rank 0 logs each sweep's refine/derefine marks per realm before and after the balance. multirealm/check.sh leg 8 runs Sod split at the diaphragm into two auto realms, both regridding by a Löhner marker, against the single-realm Sod with the same marker: the seam turns 2:1 and back, the regrids happen on the same steps, and with seam_ghost_fill = injection the union is bitwise the single run (with the default tricubic fill, within 2e-13: the realms' coarse blocks are larger in y and z, which moves the 4-point footprints of their 2:1 faces). Mixed levels along a seam (#110 P3). When a side of an auto seam holds several cell sizes, the pair stays auto and every structure is built face by face: check_mixed_seam checks that every seam leaf faces cells of one size, equal to its own (covering its face), half (its 2:1 quadrants) or twice (it is a 2:1 quadrant); seam_register_faces gives each register face its own coarse and fine sides (a mirror face for equal cells, realm_a coarse by convention; a 2:1 face whose coarse side is either realm); the seam ghost rows were already per ghost. multirealm/check.sh leg 11 puts a box next to the seam in each realm, over half of it, so the seam holds 2:1 faces in both directions: against the single realm with the same boxes, bitwise with seam_ghost_fill = injection (with tricubic within 1.5e-13), also with the topology rebuilt every step. Leg 12 starts a vortex across a seam in a closed box (the flow along the seam, y periodic, inviscid walls at the x ends), both realms regridding by a Löhner marker: the refined patch moves along the seam, which turns mixed, the mass is conserved to round-off, and the run is bitwise on 1 and 2 ranks and with the topology rebuilt every step. Restart (#110 P5). A realm loads its restart files (its tree and fields) inside its own initialisation, and the forest builds the inter-realm topology afterwards, from the restored trees; the checks accept every level layout a regrid produces, so a forest whose seams are stage_coincident restarts from any step, a mixed seam included, and continues bitwise. An end_of_step seam does not: the end-of-step fill writes the committed state only, so the stage buffers keep the seam ghosts of the previous step, which the restart files do not hold, and the restarted run differs at the seam from its first step. Restart on the rank count the run was saved with: fewer ranks stop the run (issue #109). Leg 13 restarts leg 12's case at step 155, a regrid that coarsens and leaves 2:1 faces on the seam, and runs past a regrid that changes the grid again (step 190): the last fields bitwise and every history file identical to the uninterrupted run, and a restart on fewer ranks stops. A 2:1 seam is per axis (#71): ratio 2 along the axes the trees refine and 1 along a null one (a quadtree's z), even block cell counts along the refined axes, the same tree kind in both realms, and a seam normal to a refined axis; leg 15 of multirealm/check.sh splits a quadtree along x and y bitwise. A limit: the balance only drops the marks of the seam leaves, so a refinement inside a realm that the 2:1 rule carries to its seam side, against a peer already at its [amr] max_level, cannot settle and stops the run with that diagnosis (raise the peer's max_level).

Each realm's INI is a complete per-app input file. Sections like [grid], [numerics], [physics], [runge_kutta] are populated as usual; the manifest contributes only the inter-realm topology.

Seams across ranks (issue #40) ​

The realms are partitioned over the ranks independently, each in its own Morton order, so the two sides of a seam can sit on different ranks: a domain split in two halves normal to z puts, on two ranks, the top seam blocks of the lower realm and the bottom seam blocks of the upper realm on different ranks. The forest therefore never assumes a seam to be rank-local:

  • Seam ghosts. At topology time every rank enumerates every seam ghost of every realm, from the replicated trees and in one canonical order (face pair, side, leaf, ghost cell), and looks up the peer cell in the peer realm's tree (its owner rank and owner-local block are known on every rank). Each rank keeps the rows it takes part in: local rows (ghost and peer cell on this rank, copied directly), receive rows (ghost here, cell elsewhere) and send rows (cell here, ghost elsewhere). Because the order is shared, the rows a rank sends to another and the rows that rank receives match one to one, with no index exchange. adam_seam_exchange fills a seam: the local copy, then the owner of the cells packs them from its active buffer, the buffers travel point to point, and the owner of the ghosts unpacks them. The edge and corner ghosts of the seam slab are filled too. Past a tangential boundary of the realm, a slab ghost lies outside the peer domain: at a physical boundary it is left to the boundary condition, and along an axis periodic on both realms its centre is wrapped by one period and filled from that peer cell (issue #65 P4: before, nothing filled those corners, which only stencils with tangential reach read; the dissipative fluxes of FLUME turned them into NaN at the first step).
  • Reflux register. The inter-realm register faces are registered from the replicated tree, like the intra-realm 2:1 faces: every rank holds the same faces under the same cursors, and reduce_fine_sums completes the fine sums of both kinds across ranks.

The peer slots and their cadence come from the manifest, so every rank calls the seam fill for the same (realm, peer) pairs in the same order; the exchange is point to point, and a rank with no rows of a pair skips it.

What a seam still requires, checked at initialization (error_stop naming the face pair, block or ghost):

  1. coupling = mirror: both sides have the same cell size, with cell centres that coincide across the seam;
  2. the two seam faces cover each other, with cells that line up across the seam; the blocks need not line up (issue #51): realms with different block sizes along a mirror seam are coupled, and each realm_b block face is scattered into every register face it overlaps (maps%seam_overlap, flux_register%accumulate_fine_overlaps);
  3. coupling = refined (a 2:1 resolution jump, issue #52) and every 2:1 face of an auto seam with mixed levels: one cell size per side (per face, for auto) with ratio exactly 2 along every axis, the same block cell counts along the seam, the same [amr] seam_ghost_fill, even block cell counts in the fine realm, and nested blocks (each fine seam block covers one 2:1 quadrant of a coarse seam block face).

A refined seam reuses the intra-realm 2:1 machinery. A fine seam ghost takes the coarse->fine interpolant of the selected seam_ghost_fill regime around the coarse cell containing it (interp_seam_ghost, footprint inside the coarse block interior); a coarse seam ghost takes the restriction of the fine cells under it, the mean of the 2x2x2 or, with [amr] seam_restriction = lagrange6 or weno6 (the same in both realms), the 6-point restriction of the intra-realm seams; the fine owner packs the codes and the regime in the row's meta (#68 C2, C3). The owner of the cells computes these values and sends them (interpolate and restrict rows; same-rank ones are messages to self). The register holds one face per coarse seam block, its four fine blocks restricted 2:1 into their quadrants as for an intra-realm 2:1 face (SEAM_KIND_INTER_REALM_REFINED). Time stepping is not subcycled: one global time step. src/tests/flume/verification/multirealm/check.sh (leg 4, --legs leg2,leg4: it compares against the sod-amr run of leg 2) splits sod-amr at its 2:1 face into a coarse and a fine realm, along x and along z: the union reproduces the single-realm run bit for bit. The same holds on the FNL backend, where the owner of the cells evaluates them on the device. For MHD, mhd/multirealm/check.sh (leg 4, --legs mv14-4) runs RJ2a on the same split, and for PRISM src/tests/prism/regression/rmf-2realm-fd-pulse-refined/check.sh splits rmf-amr-fd-pulse at its 2:1 face: both bitwise, the PRISM one with its seam div(B) source (issue #29) unchanged.

Before issue #52, periodic and interpolate were accepted and silently run as mirror; they are now refused.

src/tests/flume/verification/multirealm/check.sh (leg 3, --legs leg3) splits Sod along z: on one rank and on N ranks the union reproduces the single-realm sod-z bit for bit (measured on 2, 3 and 4 ranks; on 2 every seam ghost crosses ranks, 3468 rows each way). Before issue #40 the two-rank run ended normally with a wrong solution (205 steps instead of 174, realm 1 mass frozen at 0.5, reflux mismatch 69).

Timing the exchange (issue #53). With ADAM_SEAM_TIMING=1 in the environment (pass it to the ranks, e.g. mpirun -x ADAM_SEAM_TIMING), any forest run (single realm included) prints at the end:

  • one line with the time loop, the steps, the reflux fine-sum reduction, a barrier wait before it (the load imbalance the collective would otherwise absorb; the barrier exists only when timing), the reflux correction, and the two timers of the batched seam fills: seam imbalance, a barrier wait before their posts (the partners' arrival skew, timing only), and seam wait, their single wait;
  • one line per realm with the phases of the seam fills of its ghosts: the local copy, buffers, pack (on the owner of the cells), MPI (the posts) and unpack, plus the fills and rows moved.

Times are the maximum over the ranks; counts are summed. Without the variable nothing is timed and nothing is printed.

Batched fills (issue #53, F3). The forest posts the fills of every (realm, slot) pair of a cadence (β at every stage, α at the end of the step) in manifest order, each pair with its own message tag, and completes them with one wait, instead of one round trip per pair. Every pack precedes every unpack, which is exact because a pack reads real cells only, never a seam ghost. Measured on medooza (A30, FNL, 2 and 4 ranks): the transfer itself (seam wait) is 0.005-0.14 s, at most about 3.5% of the step; what the earlier single MPI timer showed on the z splits (up to 27% of the step) was mostly a partner arriving late, 0.06-0.72 s of load imbalance. The batch gains 1-4% of the step on the z splits.

Coupling cadence: α vs β ​

Each inter-realm seam carries a coupling_cadence selected independently in the manifest. The forest's evolve_one_step orchestrator iterates seams (not realms) when filling ghost cells and gates per-seam.

α — end_of_step (default) ​

Mid-step peer ghosts are intentionally frozen at the start of the step. At the top of every global timestep, before any realm opens it, the forest fires one synchronized inter-realm exchange across all α seams, into each realm's committed state q; the realm's open_step_forest then copies those seam ghosts into every RK stage buffer, so all the stages of the step read the peers' state at t^n. Filling at the top of the step, not after close_step_forest, makes the stages read the peers' final committed state, after their post-step work (dual-energy sync, cooling, sinks): the result does not depend on the output cadence, and a restart reproduces the uninterrupted run bitwise (#113).

α before #113, and α in PRISM

Before #113 the α fill reached the committed state only, never the stage buffers the residuals are computed on, so the α stages read seam ghosts that no fill wrote (zero). FLUME copies the ghosts into its stages since #113. PRISM does not yet: its α stages still read stale seam ghosts, which is part of why an α 1:1 seam leaks div(B) there (#31).

This is the AMReX FillCoarsePatch convention (Berger-Oliger 1984; AMReX Amr.cpp::timeStep). It is well-understood numerically and admits asymmetric per-realm K: a forest may mix SSP-RK-3 and SSP-RK-5 realms without restructuring.

The cost is structural: first-order seam coupling in time, while the per-realm interior keeps the full RK order. Far from steep gradients the effect is benign. Full-order coupling under asymmetric K would need dense (continuous) Runge-Kutta output for the peer reads; that design (γ) is recorded in #17 and is not planned.

When to use α:

  • Realms with different RK strides (asymmetric K).
  • Heterogeneous physics regions where seam coupling order is not the dominant accuracy concern.
  • As the default choice when in doubt.

β — stage_coincident (opt-in) ​

Peer ghosts are refreshed once per RK substage, inside the K loop, before end_stage_forest computes residuals. Each realm reads peer's stage-k interior at substage k, giving the seam the same temporal order as the per-realm interior.

When admissibility holds (see below), β recovers bit-equivalence to a monolithic single-realm run on the union grid. This is the strong oracle the rmf-2realm-stagesync regression case enforces continuously on every CI run.

When to use β:

  • Realms share the same ODE solver and stage count, but differ in spatial operator (the spatial-accuracy decomposition use case).
  • Seam-coupling time order matters: e.g., wave propagation crossing the seam, where first-order coupling would visibly drift.
  • Production runs where the marginal cost of K extra exchanges per step is worth the gained order.

Admissibility (β) ​

β is admissible on a seam iff both endpoint realms agree on:

  1. ODE solver scheme: numerics%scheme_time AND rk%scheme (e.g., both runge-kutta-ssp-54).
  2. Stage count: rk%nrk — the K each realm reports through stages_per_step_forest().
  3. Physics layout: physics%nv (and the per-variable index assignments).

Spatial operator (numerics%scheme_space) and grid resolution MAY differ. That is precisely the use case β exists to serve.

The forest enforces admissibility at init time via check_beta_admissibility. Any disagreement triggers an immediate error_stop naming the offending face_pair and the specific descriptor field that mismatched. No silent downgrade to α. β is opt-in — if you asked for it and the manifest is inadmissible, you want to know loudly.

Decision table ​

Your realms have...Choose
Different RK schemes / strides (K)α — β is not admissible
Different physics%nv (one CT-divergence-corrected, one not)α — β is not admissible
Same RK, same physics, same spatial operatorβ — full equivalence to single-realm
Same RK, same physics, different spatial operatorsβ — recovers seam temporal order
You're prototyping a new manifest and just want it to runα — works in every case

The orchestrator step cycle ​

forest_object%evolve_one_step drives a synchronized timestep across all realms. The structure of one timestep:

Per-seam gating: Phase 2 fires only on seams declared stage_coincident (β), once per substage; Phase 0 fires only on seams declared end_of_step (α), once per step.

Load-bearing invariant under β: Phase 2 must complete on ALL realms before Phase 3 starts on ANY realm. Otherwise the read-after-overwrite race between fill_seam_from_peer_forest writes and compute_residuals reads returns. The serial inner loops within a rank give this for free under the current Phase-A replicated-forest layout.

K-gating in Phases 1 and 3: realm is participates only when k ≤ K_realm(is) = realm(is)%stages_per_step_forest(). A realm with K < K_max no-ops the trailing stages, which is what makes asymmetric K work under α. Under β the admissibility check requires equal K, so the gate is vacuous.

Reflux at α.r1 ​

flux_register's third axis is collapsed to size 1 (α.r1; PRD #16 M2). Realms gate their reflux body on each realm's final RK substage:

fortran
if (stage /= self%rk%nrk) return    ! α.r1 end-of-step gate

The forest reduces the register once after the stage loop and applies the correction once per step, after close_step_forest has committed q. This is independent of α/β: β does not restore Wang 2018 per-stage RK-weighted reflux — that refinement is deferred to a future milestone.

Admissibility flow (β) ​

The check runs once, at forest init, after every realm's initialize_forest has populated its components. The cached seam_local_cadence(p) is then consumed by evolve_one_step at every step without further per-step manifest reads.

The other seam family: intra-realm AMR seams ​

Everything above concerns inter-realm seams — faces where two realms declared in a manifest touch. There is a second, independent seam family: the intra-realm AMR seam, a 2:1 resolution jump inside a single realm's Morton octree, where a level-ℓ block abuts a level-ℓ+1 block. This is not manifest-declared; it arises from AMR refinement markers ([amr] markers_number ≥ 1).

The two families are structurally disjoint and use different machinery:

Inter-realm seamIntra-realm AMR seam
Declared by[forest] manifest face-pairsAMR refinement markers
Resolutionsame (mirror, blocks lined up or not) or 2:1 (refined)2:1 coarse↔fine jump
Ghost filladam_seam_exchange%seam_fill_batch (seam_fill_post / seam_fill_complete per pair; seam_fill for one pair): local rows and cross-rank rows (copy, interpolate, restrict)update_ghost_local flag-4 path → interp_seam_ghost
Fill regimecopy (mirror); the seam_ghost_fill regime and the 2x2x2 mean (refined)selectable: injection / restriction-compatible / tricubic
Cadenceα / β per seamintrinsic to update_ghost (every stage)
Registered asSEAM_KIND_INTER_REALM (with maps%seam_overlap for misaligned blocks), SEAM_KIND_INTER_REALM_REFINEDSEAM_KIND_INTRA_REALM_AMR
Refluxinter-realm flux registerBerger-Colella reflux at the 2:1 face

A single-realm run with no manifest never touches the inter-realm path; a forest of same-resolution realms never touches the AMR-seam path. Both can coexist (a forest of realms, each internally AMR-refined).

Coarse→fine ghost fill regime ([amr] seam_ghost_fill) ​

At an intra-realm 2:1 seam, the fine block's ghost cells overlapping the coarse neighbour must be filled by interpolation. The regime is selected in the [amr] section:

ini
[amr]
seam_ghost_fill = tricubic   ; injection | restriction-compatible | tricubic
ValueEnumOrderFootprintNotes
injectionSEAM_FILL_INJECTION0 (legacy)1straight copy of the coarse value
restriction-compatibleSEAM_FILL_COMPATIBLEq=23exact R∘P = I restriction-compatible stencil
tricubicSEAM_FILL_TRICUBICq=44high-order tensor-product cubic

Effective default is tricubic: when the [amr] seam_ghost_fill key is absent, adam_object%initialize sets maps%seam_ghost_fill = SEAM_FILL_TRICUBIC (adam_adam_object.F90:236). (The adam_maps_object struct initializer is SEAM_FILL_INJECTION — a latent fallback only reachable if the parse block is bypassed; the two literals disagree by design, with the runtime override winning.) The FNL device path carries the regime through update_ghost_local_gpu(..., seam_ghost_fill=self%maps%seam_ghost_fill, ...).

The map-row flag (column 9 of the local ghost map) is a separate selector: value 4 routes a ghost cell to interp_seam_ghost (the coarse→fine path); 1 is plain injection copy, 8 is coarsening mean. The seam_ghost_fill regime (0/1/2) is orthogonal — it chooses how the flag-4 cells interpolate.

The 2:1 seam is not div-free (accept-truncation) ​

On a uniform grid PRISM holds div(B) = 0 to machine precision — compute_curl_fd_centered and compute_divergence_fd_centered share the antisymmetric FD1_CC primitive, so div_h(curl_h) ≡ 0 by operator commutation (a mimetic identity, Ranocha 2019 Remark 2.8). At a 2:1 AMR seam this identity breaks: coarse (Δx) and fine (Δx/2) sides difference with different-resolution stencils, commutation fails, and the seam injects an O(h^p) div(B) source. It is a divergence-constraint violation, not an energy instability (‖B‖ stays flat while div(B) runs away). The source converges under refinement (p_obs ≈ +1.15 → +2) but is unbounded in t at fixed h. Five fix classes (matched ghost values, single-valued flux, matched-difference operator, full Dedner GLM, SBP+SAT) were ruled out with evidence; the resolution is accept-truncation (issue #29). The rmf-amr-fd-pulse case is the source-free convergence instrument.

Seam div(B) guard-rail ([IO]) ​

Because the AMR-seam div(B) growth is otherwise silent, an opt-in run-time monitor lives in save_divergence_history:

ini
[IO]
seam_divB_tol   = 1.0E-06   ; monitor threshold; <= 0 disables (default -1.0, off)
seam_divB_error = .false.   ; .true. = error_stop on exceedance; else warn-only

The monitor arms whenever seam_divB_tol > 0: its second condition, allocated(maps%amr_seam_quadrant), holds on every run, because register_intra_realm_amr_seams allocates the quadrant table for every realm with blocks. It therefore covers intra-realm 2:1 faces and inter-realm refined seams alike; on a run without 2:1 seams div(B) stays at round-off and it stays silent. On max|div(B)| > seam_divB_tol it either error_stops (seam_divB_error = .true.) or prints a warning and continues. It is off by default (save_divergence_history in adam_prism_common_object.F90). To delay (not cure) the runaway on long AMR runs, enable divergence_correction = hyperbolic (it requires constrained_transport = D, B or DB; without one the run stops at initialization, issue #61) and tune the damping rate [physics].alpha (1/s, default 0). alpha replaced the dimensionless ratio c_r in 68457200; the delay measured in issue #29 was obtained with c_r and has not been re-measured with alpha.

Regression coverage ​

The cadence × K matrix and both seam families are covered by cases under src/tests/prism/regression/, each run on both CPU and FNL backends at mpirun -np 2. See the PRISM regression suite page for the full harness design and per-case goals.

Inter-realm cases:

CaseCadenceK_realmOracle
rmf-2realmα5 / 5own α golden (single-realm rmf split at x=0)
rmf-2realm-asymKα3 / 5own α golden — the asymmetric-K validation (SSP-33 ∥ SSP-54)
rmf-2realm-stagesyncβ5 / 5own β golden + continuous match against rmf/golden/<backend>/digest.txt
rmf-2realm-fd-pulseβ5 / 5the inter-realm 1:1 div-free reproducer (issue #31) — manual only: no golden, its check.sh is run by hand, skipped by run.sh
rmf-2realm-fv-pulse-blocksβ5 / 5the 1:1 pulse on the FV path with the seam blocks not lined up, each way (issue #51): the register closes exactly (`max
rmf-2realm-fd-pulse-refinedβ5 / 5rmf-amr-fd-pulse split at its 2:1 face, coupling = refined (issue #52): bitwise equal to the single-realm run, #29 seam div(B) source included — manual only: check.sh, no golden

rmf-2realm-stagesync is the load-bearing β oracle: under same-K, same-physics, same-ODE decomposition the multi-realm digest matches the single-realm rmf digest within (rtol=1e-06, atol=1e-3) (per-block metadata auto-downgraded to SKIP_METADATA). The harness fires this cross-config check on every run.

rmf-2realm-fd-pulse documents issue #31 (as a manual reproducer — it carries no golden, so run.sh skips it and only its hand-run check.sh checks this): a 1:1 same-resolution inter-realm mirror seam must be div-free like a 1:1 intra-block interface. It holds div(B) = div(D) = 0 — but only under β. Under α the seam ghosts go unfilled during RK substages (fill_seam_from_peer_forest runs once per step, so substages 2..N read stale/zero seam ghosts), which leaks div(B). For a 1:1 same-dt seam, β is the correct and required cadence.

Intra-realm AMR-seam cases (digest-goldened on both backends since d15fed4c, and driven by a bespoke check.sh div-oracle that run.sh does not invoke):

CaseSchemeOracle (check.sh)
rmf-amrfv_centeredrefinement + registration + reflux fires; np1≡np2 register parity (#28)
rmf-amr-fdfd_centeredcontrol `max
rmf-amr-fd-pulsefd_centeredsource-free two-invariant oracle + --convergence p_obs ≥ 0.8 ladder + RK-contract legs (#25)

Further reading ​

  • Manifest parser: src/lib/common/adam_forest_manifest.F90 — INI schema, parser, the forest_manifest_t and forest_face_pair_t structs; the COUPLING_*/CADENCE_* string→code mapping.
  • Orchestrator: src/lib/common/adam_forest_object.F90 — forest_object, evolve_one_step's phase outline, register_inter_realm_seams, register_intra_realm_amr_seams, build_seam_local_map, check_beta_admissibility.
  • Realm contract: src/lib/common/adam_realm_object.F90 — the _forest-suffixed TBP family every app extension overrides (fill_seam_from_peer_forest, coupling_descriptor_forest, apply_reflux_to_stage_forest, after_topology_build_forest, ...). See also the library page, section "Forest orchestration", for the library-developer view.
  • Seam constants & maps: src/lib/common/adam_maps_object.F90 — FACE_*, COUPLING_*, CADENCE_*, the seam_local_* arrays; src/lib/common/adam_seam_interpolation_library.F90 — SEAM_FILL_* regimes and stencils; BC_SEAM in adam_parameters.f90 (the forest rewrites the BC crown rows of a seam face to it, and sets maps%seam_face(face), which the apps read to fill the ghosts at a realm edge that lies beyond a seam and a physical face, issue #65); SEAM_KIND_* in adam_flux_register_object.F90.
  • PRISM backend seam TBPs: src/app/prism/cpu/adam_prism_cpu_object.F90 (fill_seam_from_peer_forest, the issue-#31 diagnostic seam refill in post_step_forest) and its FNL twin src/app/prism/fnl/adam_prism_fnl_object.F90 (the 4-kernel device seam fill, after_topology_build_forest device-map refresh).
  • Sibling case READMEs: src/tests/prism/regression/rmf-2realm/README.md (α), .../rmf-2realm-stagesync/README.md (β), and the suite README.md.
  • Design history: GitHub issues #10 (Phase D inception), #13 (interface machinery), #16 (α end-of-step barrier), #18 (β stage-coincident recovery), #21 (tricubic seam ghost fill), #29 (2:1 seam div(B) accept-truncation), #31 (inter-realm 1:1 seam + fWLayer HtoD).