Appends normalized UTF-8. Invalid input or a limit error leaves the builder unchanged.
Example: error := normalize("é", .NFC, *builder); appends "é".
#import "Extensions/Unicode_Text";
Unicode normalization and default extended grapheme, word and sentence boundaries (UCD 17.0.0). Offsets are bytes; input strings stay owned by the caller. Errors contain static messages.
Procedures#
General#
Uses quick-check tables; Maybe cases fall back to exact normalization and comparison.
Example: yes, error := is_normalized("é", .NFC); gives true and .NONE.
Segmentation#
Creates a byte-offset iterator. Words include punctuation and spaces; no dictionary tailoring.
Example: iter := segments("hello world", .WORD); then call next_segment(*iter).
Returns [start, end) and true, or false at EOF/error. Each successful call advances.
Example: start, end, more := next_segment(*iter); slices the next segment by byte offsets.
Finds the cluster end from a known boundary; decodes only through that cluster and its lookahead.
Example: end, error := next_grapheme("éx", 0); gives 3.
Finds the cluster start before index, backing up to a guaranteed boundary for context.
Example: start, error := previous_grapheme("éx", 3); gives 0.
Counts extended grapheme clusters without allocation.
Example: count, error := grapheme_count("éx"); gives 2.
Types#
A Unicode normalization form: canonical composition (NFC) or decomposition (NFD), or the
compatibility variants (NFKC, NFKD).
Normalization_Form :: enum u8 {
NFC;
NFD;
NFKC;
NFKD;
}Why a text operation failed.
Unicode_Text_Error_Kind :: enum u8 {
NONE;
INVALID_UTF8;
LIMIT;
INVALID_ARGUMENT;
ALLOCATION;
}The outcome of a text operation; kind is .NONE on success. message is static and offset is
the byte offset of the problem.
Unicode_Text_Error :: struct {
kind: Unicode_Text_Error_Kind;
message: string;
offset: int;
}Limits for normalization and segmentation. Exceeding one fails with .LIMIT; a negative one with
.INVALID_ARGUMENT.
Unicode_Text_Options :: struct {
max_size := 16 * 1024 * 1024;
max_output := 64 * 1024 * 1024;
stream_safe := false;
}Which Unicode boundaries segments finds: extended grapheme clusters, words or sentences.
Segmentation :: enum u8 {
GRAPHEME;
WORD;
SENTENCE;
}Borrows its text and allocates nothing. Check error after iteration, including empty input.
Segment_Iterator :: struct {
text: string;
kind: Segmentation;
offset: int;
error: Unicode_Text_Error;
lookahead := -1;
lookahead_class: S;
}