adam_seam_exchange

ADAM, inter-realm seam ghost fill across ranks (issue #40).

A seam ghost of realm is is filled from the interior cell of its peer realm that contains the ghost centre. The realms are partitioned over the ranks independently, so that cell may be owned by another rank. The forest splits the seam rows of every realm at topology time:

• local rows (maps%seam_local_map_ghost_cell): ghost and peer cell on this rank, copied by the realm's own fill_seam_from_peer_forest;
• cross-rank rows (maps%seam_mpi_recv_cell on the realm owning the ghosts, maps%seam_mpi_send_cell on the realm owning the cells), enumerated in one canonical order on every rank, so the rows rank A sends to rank B for a seam and the rows B receives from A match one to one.

Across a 2:1 seam (coupling = refined, issue #52) a send row carries a kind: copy, interpolate (the coarse anchor cell and the packed octant and anchor shifts of seam_meta_pack, evaluated by interp_seam_ghost, the intra-realm coarse->fine fill) or restrict (the base fine cell, mean of its 2x2x2 block, the intra-realm fine->coarse fill), so the owner of the cells sends the ghost values. Same-rank copy rows keep the local path; same-rank interpolate and restrict rows travel as messages to self (rank = this rank in the send/receive rows), without MPI.

seam_fill moves both kinds for one (realm, peer slot) pair: the local copy, then the owner of the cells packs (pack_seam_cells_forest), the buffers travel point to point, the owner of the ghosts unpacks (unpack_seam_cells_forest). Each rank calls it for the same pairs in the same order (the peer slots come from the manifest, not from the rows a rank holds); a rank with no rows of a pair returns at once, so the exchange is not collective, but a rank that skipped a pair it has rows for would leave its partners waiting.

Source: src/lib/common/adam_seam_exchange.F90

Dependencies

Contents

• seam_post_object
• seam_fill
• seam_fill_batch
• seam_fill_post
• seam_fill_complete
• seam_fill_all
• pack_seam_rows
• unpack_seam_rows
• seam_peer_slot

Variables

Name	Type	Attributes	Description
SEAM_TAG	integer(kind=I4P)	parameter	MPI tag of the cross-rank seam messages.
SEAM_ROW_COPY	integer(kind=I4P)	parameter	Same cell size: copy the cell.
SEAM_ROW_INTERPOLATE	integer(kind=I4P)	parameter	Coarse cell to a fine ghost: interpolate (interp_seam_ghost).
SEAM_ROW_RESTRICT	integer(kind=I4P)	parameter	Fine cells to a coarse ghost: mean of the 2x2x2 fine cells.

Derived Types

seam_post_object

A posted seam fill of one (realm, peer slot) pair (issue #53, F3): its buffers and requests live until the wait.

Components

Name	Type	Attributes	Description
is	integer(kind=I4P)		Realm owning the ghosts.
p	integer(kind=I4P)		Peer slot.
n_recv	integer(kind=I4P)		Cross-rank receive rows.
n_req	integer(kind=I4P)		Posted requests.
active	logical		Cross-rank (or to-self) rows were posted.
timing	logical		Accumulate the phase timers.
req	integer(kind=I4P)	allocatable	MPI requests.
send_buf	real(kind=R8P)	allocatable	Packed values sent (alive until the wait).
recv_buf	real(kind=R8P)	allocatable	Values received.

Subroutines

seam_fill

Fill the seam ghosts of self (the forest realm self%realm_index) from its peer slot p: local rows, then cross-rank rows. Only self is written; realm(:) (which contains self) is read for the peer.

subroutine seam_fill(self, realm, p)

Arguments

Name	Type	Intent	Attributes	Description
self	class(realm_object)	inout	target	Realm owning the ghosts.
realm	class(realm_object)	in	target	Forest realms.
p	integer(kind=I4P)	in		Peer slot of self.

Call graph

seam_fill_batch

Fill the seam ghosts of every (realm, peer slot) pair of seam cadence cadence with one wait (issue #53, F3): post every pair (local rows, pack, receives and sends), wait once on all the requests, then unpack every pair in pair order. Every rank reaches the same pairs in the same order (the slots come from the manifest), and each pair has its own tag. Batching moves every pack before every unpack: the packs read real (interior) cells only, never a seam ghost, so the values sent are the ones the pair-by-pair exchange sends. wtime (timing only) accumulates the wait at a barrier put before the posts (the partners' arrival skew, load imbalance) and the single wait.

subroutine seam_fill_batch(realm, cadence, wtime)

Arguments

Name	Type	Intent	Attributes	Description
realm	class(realm_object)	inout	target	Forest realms.
cadence	integer(kind=I4P)	in		Seam cadence of the pairs to fill.
wtime	real(kind=R8P)	inout		Timing: imbalance before the posts, wait.

Call graph

seam_fill_post

Post the fill of the seam ghosts of self from its peer slot p: the local rows (copied now), the pack of the rows self's ghosts receive from the cells this rank owns, the receives and the sends (one message per partner rank, tagged by the pair) and the message to self. seam_fill_complete unpacks once the requests are complete.

subroutine seam_fill_post(self, realm, p, post)

Arguments

Name	Type	Intent	Attributes	Description
self	class(realm_object)	inout	target	Realm owning the ghosts.
realm	class(realm_object)	in	target	Forest realms.
p	integer(kind=I4P)	in		Peer slot of self.
post	type(seam_post_object)	out		The posted exchange.

Call graph

seam_fill_complete

Complete a posted fill (seam_fill_post), its requests complete: unpack the received rows into self's ghosts.

subroutine seam_fill_complete(self, post)

Arguments

Name	Type	Intent	Attributes	Description
self	class(realm_object)	inout		Realm owning the ghosts.
post	type(seam_post_object)	in		The posted exchange.

Call graph

seam_fill_all

Fill the seam ghosts of self from every peer slot, whatever the cadence (diagnostics on the committed state).

subroutine seam_fill_all(self, realm)

Arguments

Name	Type	Intent	Attributes	Description
self	class(realm_object)	inout	target	Realm owning the ghosts.
realm	class(realm_object)	in	target	Forest realms.

Call graph

pack_seam_rows

Pack the ghost values of the send rows [rank, b, i, j, k, kind, meta] row_start..row_start+row_count-1 from the field q (the cells' owner active buffer), nv values per row in row order (issue #40, #52): a copy of the cell, the coarse->fine interpolant of interp_seam_ghost (the intra-realm fill; injection copies the anchor) or the restriction from the base cell: the mean of the 2x2x2 fine cells (summed in the intra-realm order, then scaled) when the meta is 0, otherwise the 6-point Lagrange or the WENO restriction of its packed codes and regime ([amr] seam_restriction, #68 C2, C3).

subroutine pack_seam_rows(rows, row_start, row_count, regime, ngc, q, buf)

Arguments

Name	Type	Intent	Attributes	Description
rows	integer(kind=I4P)	in		Send rows.
row_start	integer(kind=I4P)	in		First row.
row_count	integer(kind=I4P)	in		Number of rows.
regime	integer(kind=I4P)	in		Coarse->fine fill regime (maps%seam_ghost_fill).
ngc	integer(kind=I4P)	in		Ghost cells number.
q	real(kind=R8P)	in		Field (v, i, j, k, b).
buf	real(kind=R8P)	out		Packed values.

Call graph

unpack_seam_rows

Unpack buf (nv values per row in row order) into the ghost cells of the receive rows [rank, b, i, j, k]row_start..row_start+row_count-1 of the field q (issue #40).

subroutine unpack_seam_rows(rows, row_start, row_count, ngc, buf, q)

Arguments

Name	Type	Intent	Attributes	Description
rows	integer(kind=I4P)	in		Receive rows.
row_start	integer(kind=I4P)	in		First row.
row_count	integer(kind=I4P)	in		Number of rows.
ngc	integer(kind=I4P)	in		Ghost cells number.
buf	real(kind=R8P)	in		Packed values.
q	real(kind=R8P)	inout		Field (v, i, j, k, b).

Call graph

Functions

seam_peer_slot

Return the slot of realm is in the peer slots of peer (0 if is is not a peer of it).

Attributes: pure

Returns: integer(kind=I4P)

function seam_peer_slot(peer, is) result(q)

Arguments

Name	Type	Intent	Attributes	Description
peer	class(realm_object)	in		Realm whose slots are searched.
is	integer(kind=I4P)	in		Realm index looked up.

Call graph