Processes and tasks
A program built with FIGURE draws a frame, waits for input, and draws the next one. Work that lasts longer than a frame must run beside the frame loop, or the window stops answering: in a window it greys out, in remote mode the browser stops receiving frames and looks dead. FIGURE runs two kinds of work beside the loop:
- processes: other programs (a grid tool,
sbatchandsqueue, a local solver run), started from the program, with their output shown in a log view; - tasks: Fortran procedures of the program run on a thread of their own (reading a large file, a long computation), with their progress and a cancel.
Processes
| Procedure | Notes |
|---|---|
p%start(args [, cwd] [, env] [, merge_stderr] [, detached] [, log_file] [, stat]) | Start the program args(1) with the arguments args(2:). |
p%poll(lines [, stat]) | The complete lines printed since the last poll, oldest first; no element when there is nothing new. |
p%running() → logical | .true. while the process runs (captured: until FIGURE has read all its output). |
p%exit_code([stat]) → int32 | The exit code of a captured process that has ended (after kill: the negative signal number on POSIX, 1 on Windows). |
p%killed() → logical | .true. when the process ended after p%kill. |
p%pid() → int64 | The process id the system gave it (0 before start). |
p%kill([force] [, stat]) | Ask the process to end; with force, at once. |
p is a process_object. One object runs one process at a time: start again once the previous one has ended. Every frame the program polls the lines and adds them to a log view:
call ig_begin_disabled(tool%running()) ! one run at a time
if (ig_button(ICON_PLAY//' Run')) then
call output%clear()
call tool%start([character(len=512) :: me, '--tool']) ! the program and its arguments: no shell
started = .true.
endif
call ig_end_disabled()
call ig_same_line()
call ig_begin_disabled(.not. tool%running())
if (ig_button(ICON_SQUARE//' Stop')) call tool%kill()
call ig_end_disabled()
call ig_same_line()
if (started) then
call tool%poll(lines) ! the complete lines since the last frame
call output%add(lines)
if (tool%running()) then
write(status, '(A,I0)') 'running, pid ', tool%pid()
elseif (tool%killed()) then
status = 'stopped'
else
write(status, '(A,I0)') 'ended, exit code ', tool%exit_code()
endif
call ig_text(trim(status))
else
call ig_text_disabled('not started')
endif
call output%draw('output') ! only the lines in view; follows the end
The two modes
| Captured (default) | Detached (detached=.true.) | |
|---|---|---|
| Output | piped to the program: p%poll | written to log_file: p%poll follows the file |
| Standard error | with the output (merge_stderr, default); else the program's own | with the output (merge_stderr); else discarded |
| When the program ends | the child is killed | the child goes on |
| Exit code | p%exit_code() | not known (the child's own log tells) |
| For | short tools: a grid check, squeue, a converter | long runs: a solver run that must outlive the GUI |
A captured child is killed when its object ends (a local variable going out of scope, a start after its end releases it), at app%destroy, and when the program exits normally (stop, error stop, the end of the main program): first politely, then by force after 0.1 s. A program killed by a signal cannot kill its children: they go on until they end or write to the closed pipe.
A detached child is started in a session of its own (POSIX: setsid, after a double fork, so that it is not the program's child: the system's init process, or a subreaper, reaps it); its output goes to log_file, created or truncated at start. Its end is seen in every case on Linux: the program reaps it when it is itself the reaper (PID 1 of a container without an init), and a zombie that another process holds unreaped counts as ended (read from /proc). Elsewhere such a zombie keeps p%running() true (Known issues). The program follows it with p%poll while it runs, and can start a new GUI later that reads the same file.
p%pid() is the started program's process id, a detached one's included (0 before start): for a line in the program's own log, or a kill from another tool. p%exit_code() tells how a captured process ended; a detached one's is not known (its own log tells).
Arguments: no shell
The arguments are an array, passed to the program as they are: no shell parses them, so spaces, quotes, $ and * in a file name need no escaping, and a name typed by the user cannot run a command. Each element is trimmed of its trailing blanks. When a pipeline or a redirection is wanted, the shell is written explicitly: ['sh', '-c', 'squeue -u $USER | tail -n +2'] (on Windows ['cmd', '/c', '...']). The program is searched in PATH when its name has no /.
cwd sets the working directory of the child (default: the program's). env entries are NAME=value and are added to the program's environment, replacing a variable of the same name.
The drain
A pipe holds 64 KB. A child that fills it stops until someone reads, and a program whose frame loop sleeps (lazy redraw) or is busy would never read: the child would hang. FIGURE reads every captured pipe on a native thread of its own, into a buffer, as the child writes, and wakes the frame loop so that the new lines are drawn at once. The program polls when it likes: a child printing a megabyte while the program does nothing still ends.
Lines are cut at the newline, and a carriage return is dropped (CRLF endings, Windows' tools). A last line without a newline waits until it is completed; once the process has ended, it is given as it is. lines is a character(len=:), allocatable array as long as the longest line: trim the shorter ones.
Statuses
| Failure | Status |
|---|---|
start while this object's process still runs | FIG_ERR_STATE |
start with an empty args, an env entry without =, detached without log_file (or log_file without detached) | FIG_ERR_ARGUMENT |
the program cannot be started (not found, not executable), or log_file cannot be written | FIG_ERR_FAILED |
poll, kill or exit_code before start; exit_code while it runs or of a detached process | FIG_ERR_STATE |
kill refused by the system | FIG_ERR_FAILED |
stat is optional: absent, a failure stops the program (error stop) with the message, which figure_last_error() gives otherwise (Error handling). kill on a process that has ended does nothing.
Modes
- Window: as above; the reader wakes the frame loop with an event.
- Headless: the same (the loop does not wait, nothing to wake).
- Remote: the same: the processes run on the machine the program runs on (the server, a cluster node), not on the browser's. The reader ends the remote loop's wait on the network at once (a datagram to a socket the loop waits on), so that new output is sent at once.
Threads
p%start,p%poll,p%running,p%exit_code,p%killed,p%pidandp%killare not refused on a task's thread: the native layer keeps each process's state behind a lock of its own, so a task may run a tool with aprocess_objectof its own and read its output.- One
process_objectbelongs to one thread at a time: two threads calling the same object race on the object itself, which is the program's to protect (What code is safe on a worker). - A failure's message,
figure_last_error(), is kept per thread: a task reads its own.
Limits
killstops the program started, not the programs it started itself:sh -c 'a; b'killed leavesarunning. Writeexecin the script for its last command, or kill the job by its own means (scancel).- On Windows a polite kill is a close request, which a console program may ignore: use
force=.true.. - The output of a captured process is kept until polled, without a limit: poll it every frame.
- See also Known issues.
The log view
| Procedure | Notes |
|---|---|
log%add(line), log%add(lines) | Add one line or an array of lines (as p%poll gives them); trailing blanks are removed. |
log%clear() | Remove every line. |
log%set_cap(cap [, stat]) | Keep at most cap lines (default 10000): the oldest go first. The cap counts lines, not bytes. |
log%draw(id [, height]) | Draw the lines in a region of their own; height in pixels (absent or 0: the rest of the window, negative: leave that much below). |
log%count() → int32, log%line(i) → character | The lines kept, and line i (1 is the oldest kept; '' outside 1 to count()). |
log%following() → logical | .true. while the view follows the end. |
log%in_view(first, last) | The kept lines drawn by the last draw (last < first when none was). |
log%copy([all]) → character | The chosen lines (with all, every line), joined by newlines. |
log is a log_view_object: a ring buffer of lines with a cap, drawn with a list clipper, so a view of 10000 lines costs what the 20 in view cost. 1200 lines into a view that keeps 1000, drawn for three frames, then read back (docs/examples/src/cb_log_view.F90, run headless by the documentation's build):
call log%set_cap(1000_int32) ! keep the last 1000 lines
do i = 1_int32, 1200_int32
write(line, '(A,I0,A,ES9.2)') 'step ', i, ' residual ', 10.0_real64**(-real(i, real64) / 200.0_real64)
call log%add(trim(line)) ! or log%add(lines), as p%poll gives them
enddo
do i = 1_int32, 3_int32 ! every frame
if (.not. app%next_frame()) exit
call ig_set_next_window(0.0_real64, 0.0_real64, 500.0_real64, 300.0_real64, always=.true.)
if (ig_begin('Log')) call log%draw('lines') ! only the lines in view; follows the end
call ig_end()
enddocall log%in_view(first, last)
write(output_unit, '(A,I0,A,A)') 'kept: ', log%count(), ', the oldest: ', log%line(1_int32)
write(output_unit, '(A,L1,A,I0,A,I0,A,A)') 'following: ', log%following(), ', in view: ', first, ' to ', last, &
', the newest: ', log%line(log%count())
write(output_unit, '(A,I0,A)') 'copied, every line kept: ', len(log%copy(all=.true.)), ' characters' ! without all: the chosenA recipe for this: the cookbook.
$ ./cb_log_view
kept: 1000, the oldest: step 201 residual 9.89E-02
following: T, in view: 987 to 1000, the newest: step 1200 residual 1.00E-06
copied, every line kept: 28200 characters- Following the end: while the view is scrolled to the bottom, new lines scroll it; scrolled up (wheel, scrollbar), it stays where it is, and following starts again when it is scrolled back to the bottom.
- Choosing and copying: a click chooses a line, shift+click extends the choice to another; ctrl+c (cmd+c on macOS) copies the chosen lines, and the right-click menu has Copy and Copy all. The clipboard is the system's in a window; in headless and remote modes it is Dear ImGui's own, inside the program (the browser's clipboard is not reached).
- Reading it back: the example's second part reads what a test checks:
log%linethe kept lines,log%in_viewthe ones on screen,log%followingwhether new lines will scroll the view;log%copygives the text ctrl+c copies. set_capwith fewer lines than kept drops the oldest;capbelow 1 is refused withFIG_ERR_ARGUMENT.- The same in window, headless and remote modes.
- Threads:
log%drawdraws, and is refused on a task's thread like every drawing procedure (the threading rule); the other procedures work on the object alone, so a log the frame loop draws is filled and read on the frame loop's thread.
Tasks
| Procedure | Notes |
|---|---|
t%start(work, data [, stat]) | Run work(h, data) on a worker thread; data (any type) is copied into the task. A finished task whose result was not taken is discarded. |
t%running() → logical, t%done() → logical | It runs; it has ended and its result waits for t%result. |
t%progress(fraction [, message]) | The last progress the task reported: fraction in [0, 1] (-1 before its first report) (Reports and failures). |
t%cancel([stat]) | Ask the task to stop at its next safe point. |
t%result(data [, stat]) | Once done, the task's data, moved out (no copy); the object is free for another start. |
h%report(fraction [, message]) | In the task: post its progress, fraction in [0, 1]; the frame loop is woken to show it. |
h%cancelled() → logical | In the task: .true. once the program asked it to stop. |
h%fail(message) | In the task: report a failure, which t%result gives; the task still returns by itself. |
t is a task_object, h the task_handle_object FIGURE gives to work, a subroutine with the interface task_work: subroutine work(h, data), with type(task_handle_object), intent(inout) :: h and class(*), intent(inout) :: data. work is a module procedure (it runs on another thread: an internal procedure reaches its host's variables through the host's stack frame, which is the main thread's), and it is recursive, as is every procedure of yours that it calls: two tasks run the same code at the same time, which a procedure not declared recursive does not allow (gfortran's -fcheck=recursion, part of -fcheck=all, stops the program with "Recursive call to nonrecursive procedure"). data is the task's own copy of what start was given: a file name, or an extended type holding both the input and the arrays the task fills. work selects its type, works on it, reports its progress, and checks h%cancelled() at its safe points; t%result(data) hands it back without a copy:
recursive subroutine read_field(h, data)
!< Read the field block by block: report the progress after each block, stop at a cancel.
type(task_handle_object), intent(inout) :: h !< The task's handle.
class(*), intent(inout) :: data !< A field_stats.
integer(int64), parameter :: BLOCK = 65536_int64 !< Values per block.
real(real64) :: chunk(BLOCK) !< One block.
integer(int64) :: bytes, n, k, blocks, m !< File size, values, block, blocks, values in it.
integer :: u, ios !< Unit, I/O status.
character(len=40) :: msg !< Progress message.
real(real64) :: total !< Sum of the values.
select type (data)
type is (field_stats)
open(newunit=u, file=trim(data%file), access='stream', form='unformatted', action='read', status='old', &
iostat=ios) ! a unit of its own: never shared with the main thread
if (ios /= 0) then
call h%fail('cannot open '//trim(data%file))
return
endif
inquire(unit=u, size=bytes)
n = bytes / 8_int64
blocks = (n + BLOCK - 1_int64) / BLOCK
total = 0.0_real64
do k = 1_int64, blocks
if (h%cancelled()) exit ! a safe point: the statistics so far are consistent
m = min(BLOCK, n - (k - 1_int64) * BLOCK)
read(u, iostat=ios) chunk(1:m)
if (ios /= 0) then
call h%fail('cannot read '//trim(data%file))
exit
endif
data%values = data%values + m
data%vmin = min(data%vmin, minval(chunk(1:m)))
data%vmax = max(data%vmax, maxval(chunk(1:m)))
total = total + sum(chunk(1:m))
write(msg, '(A,I0,A,I0)') 'block ', k, ' of ', blocks
call h%report(real(k, real64) / real(blocks, real64), trim(msg))
call slow_disk()
enddo
close(u)
if (data%values > 0_int64) data%mean = total / real(data%values, real64)
endselect
endsubroutine read_fieldThe program starts the task from a button and, every frame, shows its progress until it is done, then takes its result once:
call ig_begin_disabled(reader%running())
if (ig_button(ICON_FOLDER_OPEN//' Read')) call reader%start(read_field, field_stats(file=FILE))
call ig_end_disabled()
call ig_same_line()
call ig_begin_disabled(.not. reader%running())
if (ig_button(ICON_X//' Cancel')) call reader%cancel() ! the task stops at its next block
call ig_end_disabled()
if (reader%running()) then
call reader%progress(fraction, msg)
write(line, '(F5.1,A,A)') 100.0_real64 * max(fraction, 0.0_real64), ' % ', msg
call ig_text(trim(line))
elseif (reader%done()) then ! ended: take the result once
call reader%result(data)
select type (data)
type is (field_stats)
stats = data
have = .true.
endselect
endif
call ig_separator()
if (have) then
write(line, '(A,I0,A)') 'values ', stats%values, ' read'
call ig_text(trim(line))
write(line, '(A,F8.4,A,F8.4,A,F8.4)') 'min ', stats%vmin, ' max ', stats%vmax, ' mean ', stats%mean
call ig_text(trim(line))
else
call ig_text_disabled('no field read yet')
endif
The threading rule
A task computes and reads its own files; it never calls FIGURE's drawing procedures: Dear ImGui is single-threaded, and only the main thread draws. FIGURE refuses such a call from a worker: the call returns at once (.false. for a begin), the task's status becomes FIG_ERR_STATE, given by t%result with a message naming the procedure, and the task is asked to stop (h%cancelled() becomes .true.).
Every drawing procedure of FIGURE checks: app%next_frame, the containers (ig_begin, ig_begin_child, menus, popups, tab bars, tables, trees, ip_begin_plot, ...), the widgets (ig_text, ig_button, ig_slider, ig_input_text, ...), the plot items (ip_line, ip_heatmap, ip_contour, ...), the layout calls (ig_same_line, ig_set_next_window, ig_push_id, ...), the style calls (ig_push_style_color, ig_set_style_var, ig_push_font, ...), the list clipper, the textures, log%draw, every procedure of a view3d_object, and the calls that read or change the GUI's state (app%dropped, app%notify, the file browsers, the clipboard, the colormaps). A widget called from a task outside any container is refused like a container: it returns at once (.false., its values unchanged), with FIG_ERR_STATE in stat where it has one. The check costs the main thread one call, about 1.5 ns. The application's procedures that change or read the GUI's state refuse too: app%init* and app%destroy, the style (set_style, set_font, load_style, save_style, palette_color), set_docking, set_icon, set_title and title, the layouts, save_image and image_result, file_dialog_result and notifications. A task may call the ones that only set or read a flag: below. Not checked: the raw bindings (figure_cimgui, figure_cimplot), Dear ImGui's and ImPlot's own functions, which a task must not call (known issues).
Calling the application from a task
These calls of the application are thread-safe: a task, an OpenMP thread or an MPI progress thread may make them while the main thread runs the frame loop. Each sets or reads one flag of the native layer, which is atomic; none is refused.
| Call | From another thread than the loop's |
|---|---|
app%wake() | the next frame starts without waiting for input or max_wait |
app%request_frames(n) | at least n more frames, the first at once; concurrent requests keep the largest |
app%quit() | the next next_frame returns .false., at once |
app%quit_requested() → logical | the user asked to quit; one caller gets each request |
app%set_lazy(enabled [, max_wait]) | lazy redraw on or off, from the next frame |
app%set_confirm_quit(enabled) | the window's close button reported instead of ending the loop |
app%content_scale() → real64 | the display's content scale |
wake, request_frames and quit called from another thread than the loop's wake a lazy loop that is waiting: in window mode with an event, in remote mode with a datagram to a socket the loop waits on; headless mode never waits. Called from the loop's own thread they do not wake anything, there is nothing to wake: the next next_frame does not wait. A task's progress report (h%report) and its end wake the loop too: wake is for news that does not go through the report, such as a file the task has written (cookbook).
Before v0.3.1 these flags were plain fields: quit and request_frames from a task raced with the main thread (undefined behaviour in C++: scripts/tsan_check.sh gave 16 ThreadSanitizer reports, at 8 lines of the native layer; none since), and a request from a task waited for the loop's next input or max_wait (0.5 s by default) before its frame.
Reports and failures
h%report clamps fraction to [0, 1] and, without message, keeps the previous message; each report wakes the frame loop, so a few reports a second are plenty. t%progress reads the last one from the program's side: its message is cut at 512 characters ('' before the first message).
h%fail(message) records a failure and does not stop the task: work returns by itself, at once or at its next safe point, and t%result then gives FIG_ERR_FAILED with the message. Only the first failure is kept; a later h%fail is ignored.
Cancelling
Cancellation is cooperative: t%cancel() sets a flag, and the task returns at its next safe point, where data is consistent (between two blocks of a file, two passes of a smoother). Nothing is stopped in the middle of a write. A cancelled task is done like a finished one: t%result gives its data as it was left, with FIG_OK. A task_object that ends while its task runs (a local variable, the end of the program's scope) asks it to stop and waits for it, up to its next safe point: a task with no safe point is waited for until its end.
What code is safe on a worker
- Its own data and its own files. The task works on its copy of
data; the program does not reach it untilt%result. Variables shared with the main thread (module variables, pointers into the program's arrays) are the program's to protect: a data race there is the program's, not FIGURE's. - Fortran I/O on units of its own. Open and close units inside the task (
newunit=), never read or write a unit the main thread also uses. gfortran's I/O is thread-safe per unit; nvfortran's only when compiled with-mp(OpenMP): without it, concurrent formatted I/O, internalwriteto a string included, fails at random (Known issues). - Images read from files.
fig_read_imagedraws nothing: a task decodes a PNG or a JPEG and gives the pixels back; the main thread uploads them (tex%init,tex%update;tex%loadis refused). - A large stack. Each worker has 64 MiB of stack (a secondary thread's default is 512 KiB on macOS): automatic arrays and temporaries of a few MiB fit.
- No drawing: see the rule above.
Statuses
| Failure | Status |
|---|---|
start while this object's task runs, or from a task | FIG_ERR_STATE |
| no thread can be created | FIG_ERR_FAILED |
cancel or result before start; result while the task runs | FIG_ERR_STATE |
result of a task that called a drawing procedure | FIG_ERR_STATE (the message names it) |
result of a task that reported a failure with h%fail | FIG_ERR_FAILED (the message is the task's) |
stat is optional: absent, a failure stops the program (error stop) with the message (Error handling).
Modes
- Window: the task's reports and its end wake the frame loop with an event: the progress is drawn at once, even with lazy redraw.
- Headless: the same (the loop does not wait).
- Remote: the same: the task runs on the server; its reports and its end end the remote loop's wait on the network at once (a datagram to a socket the loop waits on), so that its progress is sent at once. A long read no longer stops the streaming, so the browser stays alive.
Threads
h%report,h%cancelledandh%failare the task's own: called insidework, on its thread (the handle exists only there).t%startis refused on a task's thread (FIG_ERR_STATE): tasks do not start tasks. Several tasks may run at once, each with its own object.t%running,t%done,t%progress,t%cancelandt%resultare the program's side, called where the object lives (the frame loop's thread, in practice): the native layer keeps the task's state behind a lock, so they read it while the task runs.- A task may use a process and the thread-safe calls of the application; nothing that draws.
Feedback
The program shows what runs with a progress bar (ig_progress_bar(fraction), indeterminate for work with no known length, such as a tool that prints no progress), often in a status bar along the bottom (cookbook), and tells the user that something ended with a notification (app%notify): the program notifies when it sees a process no longer running or a task done, since a task does not draw.