Esc

Unicode_Text

moduleextensionSource
  • macOS: Everything works on macOS
  • Linux: Everything works on Linux
  • Windows: Everything works on Windows
  • Wasm: Everything works on Wasm
·since 0.36.1

#import "Extensions/Unicode_Text";

Unicode normalization and default extended grapheme, word and sentence boundaries (UCD 17.0.0). Offsets are bytes; input strings stay owned by the caller. Errors contain static messages.

Procedures#

General#

Segmentation#

next_segmentsince 0.36.1Source
(iter: *Segment_Iterator) -> (start: int, end: int, more: bool)

Returns [start, end) and true, or false at EOF/error. Each successful call advances. Example: start, end, more := next_segment(*iter); slices the next segment by byte offsets.

Types#

Normalization_Formenumsince 0.36.1Source

A Unicode normalization form: canonical composition (NFC) or decomposition (NFD), or the compatibility variants (NFKC, NFKD).

Normalization_Form :: enum u8 {
    NFC;
    NFD;
    NFKC;
    NFKD;
}

Why a text operation failed.

Unicode_Text_Error_Kind :: enum u8 {
    NONE;
    INVALID_UTF8;
    LIMIT;
    INVALID_ARGUMENT;
    ALLOCATION;
}
Unicode_Text_Errorstructsince 0.36.1Source

The outcome of a text operation; kind is .NONE on success. message is static and offset is the byte offset of the problem.

Unicode_Text_Error :: struct {
    kind: Unicode_Text_Error_Kind;
    message: string;
    offset: int;
}
Unicode_Text_Optionsstructsince 0.36.1Source

Limits for normalization and segmentation. Exceeding one fails with .LIMIT; a negative one with .INVALID_ARGUMENT.

Unicode_Text_Options :: struct {
    max_size := 16 * 1024 * 1024;
    max_output := 64 * 1024 * 1024;
    stream_safe := false;
}
Segmentationenumsince 0.36.1Source

Which Unicode boundaries segments finds: extended grapheme clusters, words or sentences.

Segmentation :: enum u8 {
    GRAPHEME;
    WORD;
    SENTENCE;
}
Segment_Iteratorstructsince 0.36.1Source

Borrows its text and allocates nothing. Check error after iteration, including empty input.

Segment_Iterator :: struct {
    text: string;
    kind: Segmentation;
    offset: int;
    error: Unicode_Text_Error;
    lookahead := -1;
    lookahead_class: S;
}