FoBiS.py

Appearance

2. References ​

fobis verify runs the cases and compares their quantities with references: files recording the values a correct version produced. The first time, write them:

console
$ fobis verify --update
Verifying case area...
area: UPDATED (reference written to verification/references/area.json)

1 of 1 case(s) verified.

A reference is written only with --update, never as a side effect: it is the claim "these numbers are right", made by you. It holds the quantities and the provenance of the run that produced them, and belongs in the repository (verification/references/area.json; the chapter commits it).

console
$ fobis verify
Verifying case area...
area: PASS
  n: equal (1 value(s), exact)
  trapezoid: equal (1 value(s), exact)
  simpson: equal (1 value(s), exact)

1 of 1 case(s) verified.

A wrong change ​

Now a slip in Simpson's rule: the weights 4 and 2 swapped (mod(i, 2) == 0 for == 1). It compiles, and the program still prints a number close to 2:

console
$ git diff --stat
 src/quad_rules.F90 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
console
$ fobis verify
Verifying case area...
area: FAIL
  n: equal (1 value(s), exact)
  trapezoid: equal (1 value(s), exact)
  simpson: 1/1 out of tolerance (exact): max abs 0.0128925, max rel 0.00644619, L2 0.0128925
    got 1.987124096496743, reference 2.000016591047936
  changed since the reference:
    git.diff_sha256: None -> '60db68284b8518a333f57e6eb3b297ec37d0570429844c4cb372d3fea5c01a5b'
    git.dirty: False -> True

0 of 1 case(s) verified.
[exit status 1]

The failure names the quantity, by how much it is off, and what changed since the reference: here only the working tree (dirty, and the hash of the uncommitted diff). A different compiler version or another dependency commit would be listed too, which is how a last-digit drift after a toolchain upgrade is told apart from a bug.

The default comparison is exact. Chapter 5 declares a tolerance, when a refactoring is allowed to move the last bits.

What you learned

fobis verify, --update, references in the repository, a failure explained by its provenance. Reference: verifying results.

Next: 3. Defects only some builds see.