FoBiS.py

Appearance

9. On a cluster ​

On a cluster the programs are built on the login node and run on compute nodes, through a scheduler. The commands of this track take --runner slurm: each run becomes a Slurm job, and its results are read exactly as a local run's, with the same probes and verdicts.

Recorded on a cluster

This chapter ran on a Slurm 23.11 cluster with NVIDIA A30 GPUs and nvfortran 25.5 (NVIDIA HPC SDK), from the project as chapter 8 left it. Its outputs were recorded there once and are not regenerated by the CI. The host names, the Slurm account and the partition are replaced by generic ones.

The scheduler ​

Where to submit is a property of the machine, not of the project: it goes in the user configuration (~/.config/fobis/config.ini), and a project can add to it in its fobos.

ini
[scheduler]
kind        = slurm
partition   = gpu
account     = myproject
; run in every job before the program
setup       = module load nvhpc
; where MPI and GPU runs are refused
login_nodes = login*
; more #SBATCH lines
options     = --mem=32G

A GPU version ​

The trapezoidal rule on 100 million samples of sin(x), computed and summed on the GPU with OpenACC:

fortran
!< The integral of sin(x) over [0, pi], exact value 2, by the trapezoidal rule on a GPU (OpenACC).
program area_acc
!< The integral of sin(x) over [0, pi], exact value 2, by the trapezoidal rule on a GPU (OpenACC).
use, intrinsic :: iso_fortran_env, only : R8P => real64
implicit none
integer,   parameter  :: n = 100000000        !< Number of intervals.
real(R8P), parameter  :: pi = acos(-1._R8P)   !< Pi.
real(R8P), allocatable :: y(:)                !< Samples of sin(x) at the n + 1 nodes.
real(R8P)             :: h                    !< Interval width.
real(R8P)             :: s                    !< Integral estimate.
integer               :: i                    !< Counter.

allocate(y(0:n))
h = pi / n
!$acc data create(y)
!$acc parallel loop
do i = 0, n
   y(i) = sin(i * h)
enddo
s = 0._R8P
!$acc parallel loop reduction(+:s)
do i = 1, n - 1
   s = s + y(i)
enddo
!$acc update self(y)
!$acc end data
s = (s + 0.5_R8P * (y(0) + y(n))) * h
print '(A, I0)', 'n = ', n
print '(A, ES24.17)', 'trapezoid = ', s
endprogram area_acc

A mode for nvfortran, and a case that needs one GPU:

ini
[gpu]
help      = The OpenACC program, for an NVIDIA A30 (compute capability 8.0)
compiler  = nvfortran
cflags    = -c -O2 -acc -gpu=cc80
lflags    = -acc -gpu=cc80
src       = ./src/
build_dir = ./build/gpu/
target    = src/area_acc.F90
ini
[case:gpu]
description = sin(x) over [0, pi] with 100 million intervals, on one GPU
target      = src/area_acc.F90
mode        = gpu
gpus        = 1
timeout     = 300
quantities  = stdout
tolerance   = trapezoid: rel 1e-12

Running through the scheduler ​

On the login node, a case with gpus (or ranks) does not run locally:

console
$ fobis case run gpu
Running case gpu...
  error
    refusing to run an MPI or GPU case on the login node login01.cluster ([scheduler] login_nodes): run it through the scheduler with --runner slurm
[exit status 1]

With --runner slurm, the program is built on the login node, then submitted. The job's resources come from the case (gpus = 1; timeout, in seconds, rounded up to whole minutes) and the [scheduler] settings:

console
$ fobis case run gpu --runner slurm
Running case gpu...
slurm: job 1356000 submitted (gpu)
slurm: job 1356000: pending
  ok (exit 0, 0.57 s) in .fobis/runs/gpu/default
    n = 100000000
    trapezoid = 1.9999999999999998
console
$ fobis verify gpu --update --runner slurm
Verifying case gpu...
slurm: job 1356001 submitted (gpu)
slurm: job 1356001: pending
gpu: UPDATED (reference written to verification/references/gpu.json (from uncommitted code))

1 of 1 case(s) verified.

The reference of area, written on a workstation in chapter 2, checked on a compute node:

console
$ fobis verify area --runner slurm
Verifying case area...
slurm: job 1356002 submitted (area)
slurm: job 1356002: pending
area: PASS
  n: equal (1 value(s), ulp 4)
  trapezoid: equal (1 value(s), ulp 4)
  simpson: within tolerance (ulp 4): max abs 4.44089e-16, max rel 2.22043e-16, L2 4.44089e-16
    got 2.0000165910479355, reference 2.000016591047936
  changed since the reference (results unaffected):
    compiler.path: '~/.local/bin/gfortran' -> '~/.local/bin/gfortran'
    compiler.version: 'GNU Fortran (Ubuntu 13.4.0-6ubuntu1~24~ppa1) 13.4.0' -> 'GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20)'
    environment.modules: [] -> ['intel/nvidia/cuda-12.3.2', 'intel/gcc-8.5.0/hwloc-2.10.0', 'intel/gcc-8.5.0/pmix-5.0.1', 'intel/gcc-8.5.0/ucx-1.15.0', 'intel/slurm', 'hpcx', 'intel/nvidia_hpc_sdk/nvhpc-hpcx-cuda12/25.5']
    git.commit: 'b15eb6be644457b2e64c9cc90d3fb60599820baa' -> '8389043c4d874efcfa02f9ffc72ad038da4cc9b6'
    git.diff_sha256: None -> '9a8afd220189bb3431217a4094d3d94e43a3e1e4112bf2b22173f3760402fc76'
    git.dirty: False -> True
    git.untracked_sources: None -> ['src/area_acc.F90']
    machine.cores: 16 -> 1
    machine.cpu: '13th Gen Intel(R) Core(TM) i9-13900KF' -> 'Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz'
    machine.os: 'Linux 6.18.33.2-microsoft-standard-WSL2' -> 'Linux 4.18.0-513.24.1.el8_9.x86_64'
    machine.python: '3.12.6' -> '3.12.13'
    scheduler.account: None -> 'myproject'
    scheduler.kind: None -> 'slurm'
    scheduler.partition: None -> 'gpu'

1 of 1 case(s) verified.

Another compiler (gfortran 8.5 instead of 13), another CPU, a job given one core: the result is within the tolerance of chapter 5, and the provenance lists what differs. The machine facts are the compute node's, recorded by the job itself, not the login node's. (compiler.path differs although both read ~/.local/bin/gfortran: the two machines' home directories are different directories.)

Where the GPU time goes ​

fobis profile --tool nsys runs the case under Nsight Systems, in a job, and maps the kernels and the transfers to the source:

console
$ fobis profile gpu --tool nsys --runner slurm 2>/dev/null
profile gpu (nsys): 0.002779 s in GPU kernels, 100.0% in the mode's sources
  kernels:
     55.9%  0.001555 s  area_acc_16_gpu = area_acc (src/area_acc.F90:16), 1 launches
     37.8%  0.00105 s  area_acc_21_gpu = area_acc (src/area_acc.F90:21), 1 launches
      6.3%  0.0001742 s  area_acc_21_gpu__red = area_acc (src/area_acc.F90:21), 1 launches
  host-device transfers:
    Device-to-Host: 2 copies, 800 MB, 0.1956 s
  transfers by source line (OpenACC):
    download 0.1959 s  2 calls  src/area_acc.F90:25  !$acc update self(y)
    upload   7.187e-06 s  1 calls  src/area_acc.F90:21  !$acc parallel loop reduction(+:s)
  note: host-device transfers 0.1956 s = 70.4 x the kernel time: the data movement dominates, keep the data on the device
  note: kernel and transfer times measured by Nsight Systems; no hardware counters (ncu not used)

The kernels take under 3 ms. Copying all the samples back to the host, at line 25, takes 0.19 s, 68 times as long, and the host needs only the first and the last sample:

console
$ grep -n "update self" src/area_acc.F90
25:!$acc update self(y(0:0), y(n:n))
console
$ fobis profile gpu --tool nsys --runner slurm 2>/dev/null
profile gpu (nsys): 0.002778 s in GPU kernels, 100.0% in the mode's sources
  kernels:
     56.0%  0.001555 s  area_acc_16_gpu = area_acc (src/area_acc.F90:16), 1 launches
     37.8%  0.001049 s  area_acc_21_gpu = area_acc (src/area_acc.F90:21), 1 launches
      6.3%  0.0001737 s  area_acc_21_gpu__red = area_acc (src/area_acc.F90:21), 1 launches
  host-device transfers:
    Device-to-Host: 3 copies, 0 MB, 8.704e-06 s
  transfers by source line (OpenACC):
    download 0.0003179 s  3 calls  src/area_acc.F90:25  !$acc update self(y(0:0), y(n:n))
    upload   7.745e-06 s  1 calls  src/area_acc.F90:21  !$acc parallel loop reduction(+:s)
  note: host-device transfers 8.704e-06 s = 0.00313 x the kernel time
  note: kernel and transfer times measured by Nsight Systems; no hardware counters (ncu not used)
console
$ fobis verify gpu --runner slurm
Verifying case gpu...
slurm: job 1356005 submitted (gpu)
slurm: job 1356005: pending
gpu: PASS
  n: equal (1 value(s), exact)
  trapezoid: equal (1 value(s), rel 1e-12)
  changed since the reference (results unaffected):
    git.diff_sha256: '9a8afd220189bb3431217a4094d3d94e43a3e1e4112bf2b22173f3760402fc76' -> '9b0139474b4acf4888c489fab8043a74bfdd86315d55aae9b25aea6c7c5d31b8'

1 of 1 case(s) verified.

Transfers down from 0.19 s to 9 µs, and the result unchanged.

nsys and TMPDIR

With TMPDIR on a network file system, Nsight Systems records no CUDA or OpenACC event, without an error: FoBiS runs it with TMPDIR=/tmp, local to the node. Found while recording this chapter.

What you learned

[scheduler], --runner slurm on case run, verify, test, assess, bench, diff-run and profile; the login-node guard; the compute node's facts in the provenance; --tool nsys. Reference: running on a cluster, profiling.

This is the end of the track. Track 6 puts a model to work under these checks.