Continuous integration and machine output

A CI job runs the commands you run, and reads their results. With --json a command writes one JSON document on its standard output and its messages on the standard error, so a script, a later job step or an agent reads facts, not log lines. The exit status carries the verdict on its own: a step fails when the command fails.

How it works:

1. One document on stdout. With --json (or --format json/sarif, or --print on the commands that generate files), FoBiS's progress lines, warnings and errors go to stderr; stdout holds the document and nothing else.
2. The facts of a failure in the document. A failed build carries the compiler's messages parsed and FoBiS's diagnosis; a failed test, the crash located in the sources; a failed verification, the worst point.
3. The exit status does not change with --json. 0 when the command succeeds, 1 when it fails (a build, a test, a finding, a result out of tolerance), 2 when the command is declared wrongly (an unknown case, mode key or stage). The document is printed before FoBiS exits, whatever the status.

Tutorial: continuous integration, one command for all of it. Reference: build, test, gate, introspect.

A GitHub Actions workflow

Install, fetch, build, test, coverage:

name: CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Tools
        run: |
          sudo apt-get install -y gfortran
          pip install FoBiS.py gcovr
      - name: Dependencies, at the tags the fobos pins
        run: fobis fetch
      - name: Build
        run: fobis build --mode release
      - name: Tests
        run: fobis test --json > tests.json
      - name: Coverage
        run: |
          fobis test --mode coverage
          fobis coverage --mode coverage --format xml --fail-under 90

• fobis fetch clones the dependencies at the tags the fobos pins. The lock file lives in .fobis_deps/, which is not committed: a fresh checkout has none, so fetch --frozen belongs on machines that keep their fetched dependencies, not on a fresh CI runner. Pin with tag= or rev= in the fobos instead. More in Dependencies.
• The coverage step fails the job below 90% of the lines (--fail-under). gcovr 7.2 with gcov 13 reports Fortran module procedures; gcovr 8.6 reports 0 lines for them. The upload to Codecov is in Testing.
• tests.json is a file a later step reads, or an artifact to upload: fobis test --json still exits 1 when a test fails, so the step fails too.
• Several compilers: a job matrix over --mode (one mode per compiler) or over --compiler; for the compilers of a workstation or a cluster loaded by environment modules, see HPC and clusters.

A first step that fails fast, before compiling anything: fobis check resolves every file's dependencies.

$ fobis check --json
{
  "status": "ok",
  "files": 241,
  "unresolved": [],
  "cycles": [],
  "warnings": []
}

Cache between runs

Cache the fetched dependencies, keyed on the fobos that pins them:

- uses: actions/cache@v4
  with:
    path: .fobis_deps
    key: fobis-deps-${{ runner.os }}-${{ hashFiles('fobos') }}
- run: fobis fetch            # an existing clone is checked out at the pin, not cloned again

• The key must change with the pins. fobis fetch without --update does not fetch new commits into an existing clone: a tag added upstream after the cache was saved is not in it. Keying on fobos makes a new pin a new cache.
• Caching the build directory does not save compile time. FoBiS rebuilds an object older than its source, and a fresh checkout makes every source newer than any restored object.
• fobis cache does not help here: fobis build never fills its store in this version.

One command for a pull request

fobis gate runs every applicable check, each as a nested fobis command with --json, and gives one verdict: build, review of the changed lines, tests, verification against the references, differential execution against the base, benchmarks, numerical assessment, the matrix of compilers and shadows. A stage that does not apply (no case, no benchmark, no change) is skipped with its reason; a failed build skips the stages that need it.

[gate]
skip           = bench
matrix_modes   = fast
matrix_shadows = none checks

$ fobis gate
gate: build: fobis build --json
gate: review: fobis review --format json --exit-zero
gate: test: fobis test --json
gate: verify: fobis verify --json
gate: diff-run: fobis diff-run HEAD --json
gate: assess: fobis assess area --json
gate: matrix: fobis matrix --json --modes fast --shadows none,checks --verify
gate: FAIL, 12.3 s
  build     pass [1.3 s]
  review    pass [0.2 s]
  test      pass [0.8 s]
  verify    FAIL [0.7 s]: fine: no reference (fobis verify fine --update, once right)
  diff-run  pass [1.9 s]
  bench     skip: by [gate] skip
  assess    pass [3.4 s]
  matrix    pass [4.0 s]
[exit status 1]

- uses: actions/checkout@v4
  with:
    fetch-depth: 0            # the merge base must be in the clone
- run: pip install FoBiS.py
- run: fobis fetch
- run: fobis gate --base origin/main --json > gate.json

• --base REF compares with the merge base of HEAD and REF: review sees the branch's changes, uncommitted ones included, and diff-run runs the cases on both sides. Without --base, the working tree is compared with HEAD.
• Skip what is noise on a shared runner: benchmark timings (skip = bench). advisory = ... keeps a stage in the report without letting it fail the gate; --fail-fast stops at the first failing stage.
• Exit status: 0 pass, 1 a non-advisory stage failed or gave no JSON, 2 a declaration error or an unknown base.
• The gate runs locally. On a cluster its nested commands do not take --runner slurm: see HPC and clusters.

Every stage and key: Verifying results. Reference: gate.

Lint and review on pull requests

Annotations on the changed lines, and the job fails on a finding:

fobis lint --format github                            # every finding of the project
fobis review --base origin/main --format github       # only the lines the branch changed

$ fobis lint --format github --select COR040
::warning file=src/integrate_mpi.F90,line=4,endLine=4,title=COR040 use-all::'use mpi' has no 'only:' list
[exit status 1]

The report in the job summary, or as a comment with your usual action:

- run: fobis review --base origin/${{ github.base_ref }} --format markdown --exit-zero >> "$GITHUB_STEP_SUMMARY"

$ fobis review --format markdown
### fobis review

1 finding(s) on the changed lines (1 warning), 4 changed line(s) in 1 Fortran file(s).

| Severity | Rule | Location | Message |
|---|---|---|---|
| warning | `OMP001` shared-scalar-written-in-parallel-region | `src/sweep.F90:16` | 'n' is shared and written by every thread (missing private?) |

<details><summary>Rules</summary>

- `OMP001` shared-scalar-written-in-parallel-region: A shared scalar is assigned inside an OpenMP parallel region: a data race.

Explain a rule with `fobis lint --explain <ID>`.

</details>
[exit status 1]

GitHub code scanning (findings tracked across commits in the repository's Security tab):

- run: fobis lint --format sarif --exit-zero > fobis-lint.sarif
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: fobis-lint.sarif

• Exit status: 1 when a warning- or error-level finding remains; --exit-zero always 0, for a step that only uploads a report.
• review needs the merge base in the clone: fetch-depth: 0 on the checkout.
• Old code: review judges only the changed lines; for lint, a baseline keeps the existing findings out of the verdict, so only new ones fail the job.

Rules, baselines, suppressions and fixes: Code analysis.

Keep the agents' instructions current

fobis agents-md writes the project's facts (modes, commands, kinds, parallel models, cases) into a marked block of AGENTS.md. In CI, check that the committed block matches the fobos and the sources:

- run: fobis agents-md --check

Exit 0 when every file's block is current; 1 when one is stale (the unified diff is printed), has no block, or the file does not exist; 2 for broken markers or a fobos error. The block holds no date, version or machine state, so it changes only when the project does. --check --json gives each file's status and diff. More in Code analysis.

Read the results in a script

fobis build --json            # status, objects compiled, errors; on failure diagnostics and diagnoses
fobis check --json            # unresolved dependencies, cycles, warnings
fobis test --json -j 4        # every test's status; with -j the document adds jobs and elapsed
fobis tree --json             # the dependency tree, each dependency with its own version
fobis introspect --buildoptions --mode release   # JSON always: a mode's effective settings
fobis verify --json           # each case against its reference, with the worst point
fobis gate --json             # every stage's verdict and its first failing fact

A successful build:

$ fobis build --mode release --json
{
  "status": "ok",
  "target": "src/integrate.F90",
  "objects_compiled": 0,
  "errors": []
}

A file using a module nobody defines: the compiler's message parsed (diagnostics), and what FoBiS establishes from the project's facts (diagnoses, the same as build --explain without a model):

$ fobis build --mode release --json
{
  "status": "error",
  "target": "src/integrate.F90",
  "objects_compiled": 0,
  "errors": [
    "warning: the file 'src/residual.F90' depends on 'quad_errors' that is unreachable",
    "src/residual.F90:4:5:\n\n    4 | use quad_errors, only : R8P\n      |     1\nFatal Error: Cannot open module file 'quad_errors.mod' for reading at (1): No such file or directory\ncompilation terminated.\n"
  ],
  "diagnostics": [
    {
      "tool": "gnu",
      "severity": "fatal",
      "message": "Cannot open module file 'quad_errors.mod' for reading at (1): No such file or directory",
      "file": "src/residual.F90",
      "line": 4,
      "column": 5
    }
  ],
  "diagnoses": [
    {
      "kind": "missing-module",
      "summary": "module quad_errors is not available when src/residual.F90 is compiled",
      "diagnostic": {
        "tool": "gnu",
        "severity": "fatal",
        "message": "Cannot open module file 'quad_errors.mod' for reading at (1): No such file or directory",
        "file": "src/residual.F90",
        "line": 4,
        "column": 5
      },
      "facts": [
        "no project source defines module quad_errors, and it is not a known library module."
      ],
      "advice": [
        "declare the project that provides it in fobos `[dependencies]` and run `fobis fetch`, or add the directory of its .mod files to `cflags` (`-I...`)"
      ],
      "consequences": 0
    }
  ]
}
[exit status 1]

The tests, with standard error thrown away to show that stdout is the document alone:

$ fobis test --json 2> /dev/null
{
  "status": "ok",
  "test_dir": "test",
  "passed": 2,
  "failed": 0,
  "results": [
    {
      "name": "test_exactness",
      "suite": "",
      "exit_code": 0,
      "duration": 0.001,
      "stdout": "exact on polynomials of its degree",
      "stderr": "",
      "status": "passed"
    },
    {
      "name": "test_order",
      "suite": "",
      "exit_code": 0,
      "duration": 0.001,
      "stdout": "observed orders: trapezoid  2.001, simpson  4.005",
      "stderr": "",
      "status": "passed"
    }
  ]
}

In a script:

result=$(fobis build --json)
if [ "$(echo "$result" | jq -r .status)" != ok ]; then
  echo "$result" | jq -r '.diagnoses[].summary'
  exit 1
fi

• Which commands: --json on build, clean, fetch, check, tree, test, case list, case run, provenance, verify, matrix, assess, diff-run, bench, gate, profile, query, agents-md, and commit --show-facts, ask --show-context. lint, review, merge-assist, testgen and ecosystem impact take --format json. introspect always prints JSON.
• Statuses differ by command: ok/error for build, check and test; pass/fail for verify and gate; per test passed, failed, build-failed, timeout, error. Read the exit status for the verdict, the document for the facts.
• Exit 2 means the declaration is wrong, not the code: an unknown case, stage or [scheduler] key, a launcher not on the PATH. Several of these print Error: ... on stderr and no document.
• Progress lines of long commands (gate: ..., slurm: job ..., the matrix's cells) go to stderr too.
• Each document's fields are listed on the command's reference page.

Agents read the same documents through the MCP server, fobis mcp: see AI assistance.

Review before every commit

The rules on the staged lines only, with no model: a hook as fast as fobis lint. With pre-commit, in .pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: fobis-review
        name: fobis review
        entry: fobis review --staged
        language: system
        pass_filenames: false
        files: '\.(f|F|for|f90|F90|f95|f03|f08)$'

Or as a plain git hook, .git/hooks/pre-commit (executable):

#!/bin/sh
exec fobis review --staged

The commit is refused (exit 1) when a staged line has a warning- or error-level finding. When a reviewed file has unstaged changes too, FoBiS warns: the line numbers are the staged version's. Tutorial: review what changed.

The same CI in every project

fobis scaffold keeps a project's CI workflows, release script and documentation boilerplate in line with a template (status lists the files OK, OUTDATED or MISSING, sync updates them after showing the diff), and fobis ecosystem does it for every project of a group. Tutorial: shared boilerplate. More in Ecosystem.