FoBiS.py

Appearance

6. Is it faster? ​

Removing merge from the loop could make Simpson's rule faster. A measurement says, a single timing does not: run times vary from run to run, and only an interval tells whether two medians differ.

The case fine runs area with four million intervals, and a [bench:fine] section says how precisely to measure it: at least 10 runs, until the median is known within ±2% (95% interval):

ini
[bench:fine]
min_runs = 10
target   = 0.02

--compare REF builds the revision and the working tree in scratch copies and measures them interleaved, one run of each in turn, so that a change of the machine's load affects both alike (the progress lines go to the standard error, discarded here):

console
$ fobis bench fine --compare HEAD~1 2>/dev/null
bench fine (mode release, wall time)
  HEAD~1: median 0.03192 s [0.03117, 0.03237] 95%, MAD 0.0007874 s, min 0.03094 s, 35 runs, CPU/wall 1.08
  working tree: median 0.0309 s [0.03055, 0.03165] 95%, MAD 0.0005326 s, min 0.03018 s, 23 runs, CPU/wall 1.08
  vs HEAD~1: speedup 1.033 [0.985, 1.059]: no measurable difference (the intervals overlap)

The verdict comes from the intervals, not from the medians: they overlap, so the refactoring is not measurably faster. The speedup's interval says the same, it contains 1.

Measured, not generated

Timings depend on the machine: the outputs of this page were recorded once and are not compared by the CI. Yours will differ.

--record appends a measurement to verification/bench/fine.jsonl with the machine's identity (CPU model, cores, GPUs, compiler and version). A plain fobis bench then compares with the last record of the same machine and mode, and exits with 1 when the case is measurably slower:

console
$ fobis bench fine --record 2>/dev/null
bench fine (mode release, wall time)
  measured: median 0.03143 s [0.03095, 0.0324] 95%, MAD 0.0009789 s, min 0.0302 s, 50 runs, CPU/wall 1.08
  note: the interval is +-2.3%, wider than the target 2.0%: the runs stopped at max_runs or max_time
  recorded in verification/bench/fine.jsonl
console
$ wc -l verification/bench/fine.jsonl
1 verification/bench/fine.jsonl

--feedback adds what the compiler did differently on both sides: loops that lost vectorisation, new implicit host-device copies.

What you learned

[bench:NAME], fobis bench, --compare, --record, verdicts from intervals. Reference: benchmarks.

Next: 7. One command for all of it.