FoBiS.py

Appearance

ACC009 mpi-on-device-data-without-update ​

warning opt-in

MPI sends or receives a device-resident array with no host update around the call and no host_data.

Why ​

When an array lives on the GPU, the host copy is stale until !$acc update self(buf) and the device copy is stale after a receive until !$acc update device(buf). An MPI call on the host copy without those updates sends old data or leaves the received data on the host only: results look right on one rank and diverge as the decomposition is refined. Either stage through the host (update self before sending, update device after receiving) or give MPI the device address with !$acc host_data use_device(buf) (GPU-aware MPI). Reported in the unit of the call only: updates done in another procedure are not seen.

Reported ​

fortran
!$acc data copy(halo)
call mpi_sendrecv(halo, n, MPI_REAL8, dst, 0, halo_in, n, MPI_REAL8, src, 0, comm, st, ierr)

Preferred ​

fortran
!$acc update self(halo)
call mpi_sendrecv(halo, n, MPI_REAL8, dst, 0, halo_in, n, MPI_REAL8, src, 0, comm, st, ierr)
!$acc update device(halo_in)

Suppressing ​

For one occurrence, with the reason on the same line (or alone on the line before the statement):

fortran
! fobis: ignore[ACC009] <why this occurrence is fine>

For the whole project, ignore = ACC009 in the fobos [lint] section.

References ​

Sections of the shipped knowledge packs behind this rule:

  • hpc-practice:gpu-practice#gpu-mpi-coherence (fobis kb show hpc-practice:gpu-practice#gpu-mpi-coherence)
  • openacc:cheatsheet#mpi-openacc (fobis kb show openacc:cheatsheet#mpi-openacc)
  • openacc:patterns#host-data-for-library-device-aware-mpi-interop (fobis kb show openacc:patterns#host-data-for-library-device-aware-mpi-interop)