Skip to content

stringifor_encodings ​

StringiFor, text encodings: byte order marks, encoding names and conversion to UTF-8.

StringiFor works on UTF-8. Text in another encoding is converted on input by to_utf8, which reads it in pieces (an incomplete character at the end of a piece is carried to the next one), so a file can be converted in chunks. The supported encodings are UTF-8, US-ASCII, ISO-8859-1 (Latin-1), windows-1252, UTF-16 and UTF-32 (little or big endian); canonical_encoding folds their IANA names and aliases. bom_encoding and bom_length read a byte order mark. An invalid sequence becomes a replacement (U+FFFD by default) and is reported by its position, as Python bytes.decode(encoding, errors='replace') does.

Source: src/lib/stringifor_encodings.F90

Dependencies

Contents ​

Variables ​

NameTypeAttributesDescription
ENC_UNKNOWNintegerparameterNot supported.
ENC_UTF8integerparameterUTF-8.
ENC_ASCIIintegerparameterUS-ASCII.
ENC_LATIN1integerparameterISO-8859-1.
ENC_CP1252integerparameterwindows-1252.
ENC_UTF16LEintegerparameterUTF-16, little endian.
ENC_UTF16BEintegerparameterUTF-16, big endian.
ENC_UTF32LEintegerparameterUTF-32, little endian.
ENC_UTF32BEintegerparameterUTF-32, big endian.
UNDEFINEDintegerparameterA windows-1252 byte without a character.
CP1252_HIGHintegerparameterwindows-1252 0x80..0x9F.
LOWERcharacter(len=*)parameterLowercase letters.
UPPERcharacter(len=*)parameterUppercase letters.

Subroutines ​

to_utf8 ​

Convert bytes in encoding (a name or an alias of canonical_encoding) to UTF-8.

  • A source read in pieces passes carry: it is prepended to bytes and, unless is_last, receives the bytes of an incomplete character at the end (a UTF-16 unit split, a surrogate pair split, a UTF-32 unit split). Without carry (or with is_last, the default), an incomplete character at the end is invalid.
  • An invalid sequence becomes replacement (default U+FFFD): a US-ASCII byte above 127, one of the 5 windows-1252 bytes without a character, an unpaired UTF-16 surrogate (each unit), a UTF-32 unit that is not a Unicode scalar value, an incomplete character at the end (as a whole). iostat is then the position of the first one in carry followed by bytes, 0 if there is none.
  • UTF-16 and UTF-32 without an endianness are big endian (RFC 2781); a byte order mark is not removed (it becomes U+FEFF): read it first with bom_encoding and bom_length. UTF-8 is returned as it is (check it with string%utf8_error).
  • An encoding not supported gives an empty utf8, iostat=-1 and iomsg saying so.
fortran
 character(len=:), allocatable :: utf8
 character(len=:), allocatable :: carry
 integer                       :: iostat
 logical                       :: test_passed(5)
 call to_utf8(bytes='caf'//char(233), encoding='latin1', utf8=utf8)
 test_passed(1) = utf8=='caf'//char(195)//char(169)
 call to_utf8(bytes=char(128)//'5', encoding='windows-1252', utf8=utf8)
 test_passed(2) = utf8==char(226)//char(130)//char(172)//'5'
 carry = ''
 call to_utf8(bytes='a'//char(0)//char(61), encoding='UTF-16LE', utf8=utf8, carry=carry, is_last=.false.)
 test_passed(3) = utf8=='a'.and.len(carry)==1
 call to_utf8(bytes=char(216)//char(0)//char(222), encoding='UTF-16LE', utf8=utf8, carry=carry, is_last=.true.)
 test_passed(4) = utf8==char(240)//char(159)//char(152)//char(128)
 call to_utf8(bytes='a'//char(0)//char(0)//char(220), encoding='UTF-16LE', utf8=utf8, iostat=iostat)
 test_passed(5) = utf8=='a'//char(239)//char(191)//char(189).and.iostat==3
 print '(L1)', all(test_passed)

Attributes: pure

fortran
subroutine to_utf8(bytes, encoding, utf8, carry, is_last, replacement, iostat, iomsg)

Arguments

NameTypeIntentAttributesDescription
bytescharacter(len=*)inBytes in the encoding.
encodingcharacter(len=*)inEncoding name.
utf8character(len=:)outallocatableBytes in UTF-8.
carrycharacter(len=:)inoutallocatable, optionalIncomplete character between pieces.
is_lastlogicalinoptionalThe source ends with these bytes.
replacementcharacter(len=*)inoptionalReplacement of an invalid sequence.
iostatintegeroutoptionalPosition of the first invalid sequence.
iomsgcharacter(len=*)inoutoptionalMessage of an error.

Call graph

close_tail ​

Carry the incomplete character at the end of the input (is_carried), or replace it.

Attributes: pure

fortran
subroutine close_tail(input, tail, is_carried, rep, buffer, n, first_error, carried)

Arguments

NameTypeIntentAttributesDescription
inputcharacter(len=*)inBytes.
tailintegerinFirst byte of the incomplete character.
is_carriedlogicalinThe incomplete character is carried.
repcharacter(len=*)inReplacement.
buffercharacter(len=*)inoutOutput buffer.
nintegerinoutOutput length.
first_errorintegerinoutPosition of the first invalid sequence.
carriedcharacter(len=:)inoutallocatableIncomplete character carried.

Call graph

put ​

Append the UTF-8 encoding of a codepoint to the buffer.

Attributes: pure

fortran
subroutine put(buffer, n, codepoint)

Arguments

NameTypeIntentAttributesDescription
buffercharacter(len=*)inoutOutput buffer.
nintegerinoutOutput length.
codepointintegerinCodepoint (a Unicode scalar value).

Call graph

single_byte ​

Convert a single byte encoding (US-ASCII, ISO-8859-1, windows-1252): ASCII runs are copied as they are.

Attributes: pure

fortran
subroutine single_byte(input, kind_, rep, buffer, n, first_error)

Arguments

NameTypeIntentAttributesDescription
inputcharacter(len=*)inBytes.
kind_integerinEncoding.
repcharacter(len=*)inReplacement.
buffercharacter(len=*)inoutOutput buffer.
nintegeroutOutput length.
first_errorintegerinoutPosition of the first invalid byte.

Call graph

utf16 ​

Convert UTF-16; tail is the first byte of an incomplete character at the end, len(input)+1 if there is none.

Attributes: pure

fortran
subroutine utf16(input, is_big, rep, buffer, n, first_error, tail)

Arguments

NameTypeIntentAttributesDescription
inputcharacter(len=*)inBytes.
is_biglogicalinBig endian.
repcharacter(len=*)inReplacement.
buffercharacter(len=*)inoutOutput buffer.
nintegeroutOutput length.
first_errorintegerinoutPosition of the first invalid sequence.
tailintegeroutFirst byte of the incomplete character at the end.

Call graph

replace_unit ​

Append the replacement of an invalid sequence at byte at.

Attributes: pure

fortran
subroutine replace_unit(rep, buffer, n, first_error, at)

Arguments

NameTypeIntentAttributesDescription
repcharacter(len=*)inReplacement.
buffercharacter(len=*)inoutOutput buffer.
nintegerinoutOutput length.
first_errorintegerinoutPosition of the first invalid sequence.
atintegerinPosition of the sequence.

Call graph

utf32 ​

Convert UTF-32; tail is the first byte of an incomplete unit at the end, len(input)+1 if there is none.

Attributes: pure

fortran
subroutine utf32(input, is_big, rep, buffer, n, first_error, tail)

Arguments

NameTypeIntentAttributesDescription
inputcharacter(len=*)inBytes.
is_biglogicalinBig endian.
repcharacter(len=*)inReplacement.
buffercharacter(len=*)inoutOutput buffer.
nintegeroutOutput length.
first_errorintegerinoutPosition of the first invalid unit.
tailintegeroutFirst byte of the incomplete unit at the end.

Call graph

Functions ​

bom_encoding ​

Return the encoding named by the byte order mark that starts bytes: UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, UTF-32BE, or an empty string if there is none.

@note FF FE 00 00 is read as the UTF-32LE mark (not as the UTF-16LE mark followed by U+0000).

fortran
 logical :: test_passed(4)
 test_passed(1) = bom_encoding(char(239)//char(187)//char(191)//'<a/>')=='UTF-8'
 test_passed(2) = bom_encoding(char(255)//char(254)//'<'//char(0))=='UTF-16LE'
 test_passed(3) = bom_encoding(char(0)//char(0)//char(254)//char(255))=='UTF-32BE'
 test_passed(4) = bom_encoding('<a/>')==''
 print '(L1)', all(test_passed)

Attributes: pure

Returns: character(len=:)

fortran
function bom_encoding(bytes) result(encoding)

Arguments

NameTypeIntentAttributesDescription
bytescharacter(len=*)inBytes, the first ones at least.

Call graph

bom_length ​

Return the length in bytes of the byte order mark that starts bytes (3 for UTF-8, 2 for UTF-16, 4 for UTF-32), 0 if there is none.

fortran
 logical :: test_passed(3)
 test_passed(1) = bom_length(char(239)//char(187)//char(191)//'x')==3
 test_passed(2) = bom_length(char(254)//char(255)//char(0)//'x')==2
 test_passed(3) = bom_length('x')==0
 print '(L1)', all(test_passed)

Attributes: pure

Returns: integer

fortran
function bom_length(bytes) result(n)

Arguments

NameTypeIntentAttributesDescription
bytescharacter(len=*)inBytes, the first ones at least.

Call graph

canonical_encoding ​

Return the canonical name of a supported encoding, given its name or one of its aliases (IANA character sets, case ignored, blanks around ignored), or an empty string if it is not supported.

The canonical names are UTF-8, US-ASCII, ISO-8859-1, windows-1252, UTF-16, UTF-16LE, UTF-16BE, UTF-32, UTF-32LE and UTF-32BE.

fortran
 logical :: test_passed(5)
 test_passed(1) = canonical_encoding('utf8')=='UTF-8'
 test_passed(2) = canonical_encoding(' Latin1 ')=='ISO-8859-1'
 test_passed(3) = canonical_encoding('CP1252')=='windows-1252'
 test_passed(4) = canonical_encoding('ascii')=='US-ASCII'
 test_passed(5) = canonical_encoding('Shift_JIS')==''
 print '(L1)', all(test_passed)

Attributes: pure

Returns: character(len=:)

fortran
function canonical_encoding(name) result(canonical)

Arguments

NameTypeIntentAttributesDescription
namecharacter(len=*)inEncoding name.

Call graph

bom_kind ​

Return the encoding of the byte order mark that starts bytes, ENC_UNKNOWN if there is none.

Attributes: pure

Returns: integer

fortran
function bom_kind(bytes) result(kind_)

Arguments

NameTypeIntentAttributesDescription
bytescharacter(len=*)inBytes.

Call graph

byte_value ​

Return the value 0..255 of a byte (ichar may be signed above 127 on some processors).

Attributes: elemental

Returns: integer

fortran
function byte_value(c) result(value)

Arguments

NameTypeIntentAttributesDescription
ccharacter(len=1)inByte.

Call graph

encoding_kind ​

Return the encoding of a canonical name (see canonical_encoding), ENC_UNKNOWN for an empty one.

Attributes: pure

Returns: integer

fortran
function encoding_kind(canonical) result(kind_)

Arguments

NameTypeIntentAttributesDescription
canonicalcharacter(len=*)inCanonical name.

Call graph

itoa ​

Return the decimal digits of a non negative integer.

Attributes: pure

Returns: character(len=:)

fortran
function itoa(i) result(text)

Arguments

NameTypeIntentAttributesDescription
iintegerinInteger.

Call graph

lowered ​

Return the text with its ASCII letters lowercased.

Attributes: pure

Returns: character(len=len)

fortran
function lowered(text) result(lower_text)

Arguments

NameTypeIntentAttributesDescription
textcharacter(len=*)inText.

Call graph

len_trim_bytes ​

Return the UTF-8 length of a Unicode scalar value.

Attributes: elemental

Returns: integer

fortran
function len_trim_bytes(codepoint) result(m)

Arguments

NameTypeIntentAttributesDescription
codepointintegerinCodepoint.

Call graph

unit16 ​

Return the value of a 16-bit unit.

Attributes: pure

Returns: integer

fortran
function unit16(pair, is_big) result(value)

Arguments

NameTypeIntentAttributesDescription
paircharacter(len=2)inTwo bytes.
is_biglogicalinBig endian.

Call graph