Skip to content

Restart round trip and output channels ​

IDGroupMilestoneTypeBackendsFigureRegression
V7EulerM1 (#35) · P7; #42, #109 (follow-up)CCPU, FNLno—

Case and scope ​

Restart files, slices and auxiliary fields are outputs that no run reads back in normal operation, so a defect in them is silent. The case runs each case twice: run A goes N steps; run B goes N/2 steps saving a restart, restarts and completes N. The two final states must be bitwise identical on the interior cells, and the residual and conservation histories byte-identical: the restarted run appends to them and does not re-save its starting step.

Four cases cover the three integration paths of FLUME and an empty rank:

  • sod-x, the single-realm fast path;
  • amr-periodic, the staged path with static 2:1 AMR and the reflux;
  • shock-cylinder, the immersed boundary with AMR;
  • uniform, the skeleton case whose 8 blocks all sit on rank 0 at np 2, so rank 1 owns no block when the restart is saved.

What the case catches:

  • a restart channel that misses part of the state: any field, time, step counter or history line not saved or not read back breaks the bitwise identity;
  • an empty rank that writes no blocks file, so the restart aborts (#42);
  • a restart snapshot placed off its cells: the XH5F snapshot written with every restart holds the interior cells only, and its block origin must be the first stored cell, not the ghost corner (#109);
  • a restart on fewer ranks than saved, which must stop instead of dropping the missing ranks' blocks (#109);
  • wrong derived outputs: the auxiliary fields against the conservative ones, a slice against the cell values.

Models and numerics ​

ItemChoiceCode
Restart files<restart_basename>.time, .tnd, -procNNNNNN.fbd per rank, and the XH5F snapshot <restart_basename>.xdmf + .h5 (interior cells)save_restart_files, load_restart_files, adam_adam_object.F90; input
Rank checka restart whose blocks file was saved on more ranks than the run has stopsadam_tree_object.F90
Auxiliary fields[IO] save_auxiliary_fields: u,v,w,p,H,a in every checkpointinput
Slices[slices], one trilinear slicelibrary slices_object, save_slice, adam_adam_object.F90
Casesthe numerics of Sod (V1), conservation across AMR seams (V3) and shock over a cylinder (V6)—

Initial and boundary conditions ​

LegCaseBlocksRun ARun BChecks
v7sod/sod-x.ini6440 steps20 + restart + 20interior fields bitwise, histories byte-identical; the snapshot of step 20 against the regular save of step 20; a restart on np − 1 ranks
v7conservation/amr-periodic.ini1520 steps10 + restart + 10interior fields bitwise, histories byte-identical
v7shock-cylinder/shock-cylinder.ini17640 steps20 + restart + 20interior fields bitwise, histories byte-identical
v7uniform/input.ini8, all on rank 010 steps5 + restart + 5interior fields bitwise, histories byte-identical (the state is uniform: the leg checks that an empty rank saves and restarts)
slicessod/sod-x.ini, save_auxiliary_fields = .true., one trilinear slice of 200×1×1 points on the x cell centres6440 steps—auxiliary fields against the conservative ones; slice against the cell values

Every run on 2 ranks (--np N). Interior cells only: the edge and corner ghost cells that no map fills keep stale values the directional stencils never read.

Input files ​

FileRole
src/tests/flume/verification/io/check.shruns the legs (--legs v7,slices, default both; --np N; --build builds flume-cpu-gnu; FLUME_EXE selects the backend)
src/tests/flume/verification/conservation/conservation_oracle.py --compare A B --tol 0 --ngc 3runs A and B cell by cell, tolerance 0
src/tests/flume/verification/io/snapshot_oracle.pythe snapshot block origins against the regular save of the same step
src/tests/flume/verification/io/io_oracle.pythe auxiliary fields and the slice (relative tolerance 10−12)
bash
cd src/tests/flume/verification/io && ./check.sh                       # CPU
FLUME_EXE=$PWD/../../../../../exe/adam_flume_fnl ./check.sh            # FNL

No figure: the legs are a bitwise identity and an output contract.

Results ​

LegCheckBoundCPUFNL
v7sod-x: interior fields, A against Bbitwisebitwisebitwise
v7amr-periodic: interior fieldsbitwisebitwisebitwise
v7shock-cylinder: interior fieldsbitwisebitwisebitwise
v7uniform, an empty rank at the save: interior fieldsbitwisebitwisebitwise
v7residual and conservation histories, every casebyte-identicalidenticalidentical
v7restart snapshot origin against the first stored cell (64 blocks)0 cells (<10−9)0—
v7restart on 1 of 2 ranksstops with the rank messagestops—
slicesauxiliary fields u,v,w,p,H,a against the recomputed ones≤10−12 relative01.9⋅10−16
slicesslice points on the cell centres; slice against the cell values0 off; ≤10−120; 5.6⋅10−160; 5.6⋅10−16

The FNL column of the two #109 rows has not been run since the fix.

Related cases: restart across runtime regrids is runtime regrids (rg3–rg6); the MHD restart round trips are MV-14 legs 2 and 3 on forest seams; the forest restart is runtime AMR on a forest.

Findings ​

  • A restart is invisible: on every integration path, and with a rank that owns no block, the restarted run ends bitwise on the straight one, and its histories continue byte for byte.
  • The restart snapshot is a regular XH5F file of the interior cells: a reader places it on the cells of the field saves of the same step.
  • A restart needs the rank count it was saved with; fewer ranks stop the run.
  • The auxiliary fields and the slices reproduce the state they are derived from to round-off.