Skip to content

Slices output ​

CasesIssuesTypeBackendsFigureRegression
slices—CCPU, FNLno—

Case and scope ​

The [slices] output writes the fields interpolated on lines, planes and boxes into MPI-IO .mat files. No run reads them back, so a wrong value goes unnoticed. The case recomputes every written value from the checkpoint of the slice step and checks the decomposition independence, the non-dimensional scaling, the accuracy against an analytic wave and the input validation of the [slice_N] sections.

What the case catches:

  • a slice value that is not the trilinear (or inverse-distance) combination of the right eight cells, a point written by no rank, a point in the wrong place;
  • stale auxiliary ghosts: the curl and the divergence are computed on the interior cells only, and save_slices must refresh their intra-realm ghosts before interpolating;
  • an output that depends on the rank count (each point is written by the one rank owning it);
  • slice extents left dimensional under a non-dimensional model: the points were then in the wrong place, or outside the domain and the run aborted at the first save;
  • a missing or invalid [slice_N] option that left the slice undefined (n_save = 0 crashed on mod(it, 0));
  • a stale, longer slice file not truncated by the save.

Models and numerics ​

ItemChoiceCode
Casethe source-free pulse of rmf-amr-fd-pulse (2:1 seam at x=0, 36 blocks), curl and divergence savedseam div(B)
Interpolationtrilinear or inverse_distance over the eight cells around the point, ghost cells includedinterpolate_at_point, save_slice, adam_adam_object.F90; adam_slices_object.F90
Ghost refreshauxiliary fields refreshed before the savesave_slices, adam_prism_common_object.F90
.mat layout4 int32 (ni, nj, nk, 3+nq), then per point 3 coordinates and nq values in float64, i fastest—

Initial and boundary conditions ​

ItemValue
Domain, grid[−0.06,0.06]3 m; 8 blocks of 163 at level 1, x<0 refined to level 2
Initial conditionGaussian pulse along +y, sigma = 0.01, B0 = 10
Steps5; slices saved every 5 steps
slice_1trilinear line, 97 points along x∈[−0.06,0.06] at y=z=0.0013 (through the seam)
slice_2inverse_distance plane, 33×29, x∈[−0.05,0.05], y∈[−0.04,0.04], z=0.0007
slice_3trilinear box, 73, [−0.004,0.004]3, around the block corner at the origin
slice_4inverse_distance plane, 16×32, x∈[0,0.06], y∈[−0.06,0.06], z=0.001875: exactly on cell centres
Ranks1, 2, 4 (--np)
Non-dimensional legAdimEM, L0 = 0.01, B0 = 10, a plane wave along y with λ=0.04; the last rank count
Convergence legthe same plane wave, no AMR, 16 and 32 cells per block side, 1 step, one line slice off the cell centres

Input files ​

FileRole
src/tests/prism/regression/slices/slices.inithe case and its four slices
src/tests/prism/regression/slices/check.shthe runs and the legs (--build, --np "N ...", --convergence; PRISM_EXE)
src/tests/prism/regression/slices/slices_oracle.pyrecomputes every slice value; checks the face ghosts
src/tests/prism/regression/slices/slices_convergence.pythe slice error against the analytic plane wave
bash
cd src/tests/prism/regression/slices && ./check.sh --convergence
PRISM_EXE=$PWD/../../../../../exe/adam_prism_fnl ./check.sh --np "1 2"   # FNL

Results ​

LegAssertsBoundCPUFNL
oracle, every slice and field set (q, curl, div), per rank countcoordinates against the [slice_N] box; values against the recomputed interpolation, relative to the largest of the eight stencil valuescoordinates 10−14; values 10−12coordinates ≤1.4⋅10−17, values ≤1.7⋅10−14 (36 files, np 1, 2, 4)coordinates ≤1.7⋅10−17, values ≤1.6⋅10−14 (24 files, np 1, 2)
oracle, face ghostsghost slab equal to the same-level neighbour's interior slab, every field set (faces across the 2:1 seam counted, not checked)bitwise0 (68 faces, 16 across levels skipped)0 (68 faces, 16 across levels skipped)
decompositionevery .mat file at 2 and 4 ranks identical to the 1-rank filebitwise24 files identical12 files identical (np 2)
truncationthe pre-placed 100000-byte file replaced by a file of the right sizefile sizepasspass
non-dimensionalthe oracle, with the extents divided by L0as the oraclecoordinates 1.8⋅10−15, values 4.3⋅10−16coordinates 8.9⋅10−16, values 5.5⋅10−16
--convergenceobserved order of the slice error, 16 → 32 cells≥1.81.981.98

Input validation (1 rank): each leg must stop with the message and write no slice.

LegChangeMessage
missing-n_saven_save deletedfailed to load [slice_1].(n_save)
n_save-zeron_save = 0[slice_1].(n_save) = 0 must be >= 1
itype-unknownitype = bilinearunknown [slice_1].(itype) "bilinear"
nj-zeronj = 0[slice_1].(ni, nj, nk)
emin-above-maxemin_y = 0.002[slice_1] emin
number-negativeslices_number = -1[slices].(slices_number) = -1 must be >= 0
outside-domainemax_x = 0.07outside the realm domain
itype-crlfitype = trilinear with a trailing CRaccepted: the run completes and saves the slice

Related pages: refusals for the input validation of the other sections.

Findings ​

  • The slices are recomputed point by point from the checkpoint, so the oracle covers the interpolation weights, the stencil choice, the owner of each point and the ghost refresh at once.
  • A point on a block face (on the seam, x=0) is owned by one side or the other by its last bits (gfortran writes 0, nvfortran −2.7⋅10−18): the oracle recomputes each value at the written coordinates.
  • After one step the solver error is far below the interpolation error of a line slice off the cell centres, so the convergence leg measures the trilinear error, O(h2).