8. Unicode text

A string holds bytes. Text in UTF-8 is stored as it is, so everything so far works on it unchanged, but one question now has three answers: how long is this text? This chapter builds a small table of names in several scripts and meets each of them.

Writing non-ASCII text

names(1) = 'Ren\u00e9e'
names(2) = 'Ada'
names(3) = '\u65e5\u672c\u8a9e'
names = names%unescape_c()
text = names(1)
print '(A)', text%chars()
print '(A)', text%escape_c()//''

Keep Fortran sources in ASCII and write the other characters with unescape_c, which expands \uXXXX, \n, \t and the other C escapes; it is elemental, so it works on the whole array. escape_c goes the other way: it writes any bytes as printable ASCII, which is what you want in a log. escapes_error() gives the byte of the first malformed escape, 0 if there is none.

Bytes, characters, columns

do i = 1, size(names)
  print '(A,": bytes ",I0,", characters ",I0,", columns ",I0)', names(i)%chars(), names(i)%len(), &
                                                                 names(i)%len(by=SF_CHARS), names(i)%width()
enddo

len() counts bytes: é is 2 bytes, 日 is 3. It is the default because a byte position is what raw(i:j) takes. len(by=SF_CHARS) counts characters (codepoints), and width() the columns a terminal uses: 日 takes two of them.

Aligning a column

do i = 1, size(names)
  print '(3A)', '|', names(i)%ljust(8, '.')//'', '| bytes'
enddo
do i = 1, size(names)
  print '(3A)', '|', names(i)%ljust(8, '.', by=SF_COLUMNS)//'', '| columns'
enddo

The first table is wrong: a width in bytes leaves Renée one dot short, and 日本語 has no padding at all. ljust, rjust, center, fill, justify and expand_tabs take by=SF_COLUMNS to count columns, or by=SF_CHARS to count characters. Columns are what aligns on a terminal.

Positions

b = text%index('e', back=.true.)
print '(A,I0,A,I0)', 'last e: byte ', b, ', character ', text%index('e', back=.true., by=SF_CHARS)
print '(A,I0)', 'character 5 starts at byte ', text%byte_pos(5)
print '(A,I0)', 'byte 5 is in character ', text%char_pos(5)
print '(2A)', 'characters 1 to 4: ', text%slice(first=1, last=4, by=SF_CHARS)//''
bad = text%slice(first=1, last=4)
print '(2A)', 'bytes 1 to 4:      ', bad%escape_c()//''

index, scan, verify, slice, insert and len_last_word also take by=SF_CHARS. byte_pos and char_pos convert between the two, to mix them with raw(i:j). The last line shows why the unit matters: 4 bytes end in the middle of é, and the half that is left is an ill-formed byte.

Codepoints

codes = text%codepoints()
print '(*(Z0,:,1X))', codes
print '(A)', from_codepoints(codes(5:1:-1))//''

codepoints returns the value of each character, and from_codepoints builds a string from such values. Methods such as reverse already work on whole characters, so you rarely need these two.

Text that is not UTF-8

bad = 'caf'//char(233)
print '(A,L2,A,I0)', 'well-formed:', bad%is_utf8(), ', first bad byte: ', bad%utf8_error()
print '(A)', bad%escape_c()//''
print '(A)', bad%to_valid_utf8()//''
print '(A)', bad%to_valid_utf8(replacement='?')//''

A file in Latin-1 is not valid UTF-8: its é is the single byte E9. is_utf8 and utf8_error find this. to_valid_utf8 replaces each ill-formed sequence by U+FFFD, or by replacement. Nothing else ever drops or changes such a byte: it counts as one character and goes through unchanged.

Whitespace

text = '\u3000 two\u00a0 words \u2003'
text = text%unescape_c()
print '(3A)', '[', text%strip(whitespace=.true., unicode=.true.)//'', ']'
print '(3A)', '[', text%compact(unicode=.true.)//'', ']'
print '(L1)', text%is_space(unicode=.true.)

Text copied from a web page holds no-break spaces (U+00A0) that the default whitespace, ASCII as in C, does not see. unicode=.true. on is_space, compact and, with whitespace=.true., on strip, lstrip and rstrip uses the Unicode set.

Running it

$ unicode_text
Renée
Ren\u00e9e
Renée: bytes 6, characters 5, columns 5
Ada: bytes 3, characters 3, columns 3
日本語: bytes 9, characters 3, columns 6
|Renée..| bytes
|Ada.....| bytes
|日本語| bytes
|Renée...| columns
|Ada.....| columns
|日本語..| columns
last e: byte 6, character 5
character 5 starts at byte 6
byte 5 is in character 4
characters 1 to 4: René
bytes 1 to 4:      Ren\xc3
52 65 6E E9 65
eéneR
well-formed: F, first bad byte: 4
caf\xe9
caf�
caf?
[two  words]
[two words]
F

What you learned

unescape_c, escape_c, len(by=), width, ljust(by=), byte_pos, char_pos, codepoints, from_codepoints, is_utf8, utf8_error, to_valid_utf8, unicode=. Reference: Unicode and UTF-8.

Next: the cookbook, short recipes for everyday tasks.