Esc

Protobuf

moduleextensionSource
  • macOS: Everything works on macOS
  • Linux: Everything works on Linux
  • Windows: Everything works on Windows
  • Wasm: Everything works on Wasm
ยทsince 0.36.1

#import "Extensions/Protobuf";

Protobuf messages generated by Extensions/Protobuf/Schema, in binary and proto3 JSON.

Import with a name: Proto :: #import "Extensions/Protobuf";. Decoded allocations belong to context.allocator and are released by Proto.free. Copy decode owns shared payload/container blocks at the top-level message. Free that owner, and treat its nested values as views. Use set_string/set_bytes to copy a field from another decoded owner. Directly assigned strings/bytes must be individually owned allocations (or use a no-op-free arena for the whole message). to_json/from_json speak the proto3 JSON mapping; give an Any's types with the schema's proto_registry in Json_Options.

Procedures#

General#

encodesince 0.36.1Source
(value: $T) -> ([] u8, Error)

Encodes into one exact-sized allocation; TOO_LARGE returns no bytes before allocation.

sizesince 0.36.1Source
(value: $T) -> s64

Computes the encoded byte length without allocating output, saturating at 2 GiB.

decodesince 0.36.1Source
($T: Type, bytes: [] u8, options := Decode_Options.{}) -> (T, Error)

Decodes a generated message. A failure releases all allocations and returns a zero value.

mergesince 0.36.1Source
(value: *$T, bytes: [] u8, options := Decode_Options.{}) -> Error

Merges wire data into a generated message. On failure the successfully merged prefix remains owned by value and must be freed. Strings and scalar values replace; message values merge.

free3 overloadssince 0.36.1Source
(value: *$T)
(error: *Error)
(values: *[..] $T)

Releases a decoded message recursively and resets it; safe to call twice.

equalsince 0.36.1Source
(a: $T, b: T) -> bool

Deep equality of generated messages, including preserved unknown bytes.

besidesince 0.36.1Source
(path: string, loc := #caller_location) -> string

A path relative to the calling Jai source, allocated with context.allocator.

Wire#

failsince 0.36.1Source
(r: *Reader, kind: Error_Kind) -> bool

Records the first failure; generated code adds path segments while unwinding.

field_errorsince 0.36.1Source
(r: *Reader, name: string, index: s64 = -1) -> bool

Prepends a field path to a nested error. Called only on failure.

remainingsince 0.36.1Source
(r: *Reader) -> s64

Remaining bytes within the active message or packed field.

read_varintsince 0.36.1Source
(r: *Reader) -> u64

Reads a bounded unsigned varint. One-byte values stay inline at the call site.

varint_sizesince 0.36.1Source
(value: u64) -> s64

Number of bytes in an unsigned varint.

write_varintsince 0.36.1Source
(w: *Writer, value: u64)

Writes a varint into the exact-sized output buffer. One-byte values (tags, short lengths, small numbers) stay inline at the call site.

zigzagsince 0.36.1Source
(value: s64) -> u64

Zigzag maps the signed 64-bit extremes to unsigned varints without signed overflow.

unzigzagsince 0.36.1Source
(value: u64) -> s64

Inverse zigzag. For sint32 callers truncate to u32 before calling.

write_fixedsince 0.36.1Source
(w: *Writer, value: u64, $width: s64)

Writes the low width bytes in little-endian order. Every supported target (x64, arm64, wasm) is little endian, so this is one store.

read_lengthsince 0.36.1Source
(r: *Reader) -> s64

Reads and checks a length before any slice or allocation. A one-byte length that fits stays inline; anything else, including every failure, takes read_long_length.

read_tagsince 0.36.1Source
(r: *Reader) -> u64

Validates a protobuf tag before narrowing it. A one-byte tag can't be too long or too large, so it only needs a nonzero field and a known wire type.

skipsince 0.36.1Source
(r: *Reader, tag: u64) -> bool

Skips an unknown wire field, including a matching group as one opaque unit.

(count: s64) -> [] u8

Allocates byte storage whose caller must completely initialize before exposing it.

write_bytessince 0.36.1Source
(w: *Writer, bytes: [] u8)

Appends bytes to a sized output buffer. Up to 16 bytes are copied with two overlapping fixed-size moves, which compile to plain loads and stores instead of a memcpy call.

(w: *Writer, value: *$T)

Writes a message using its preorder size entry (no back-patching).

read_messagesince 0.36.1Source
(r: *Reader, value: *$T) -> bool

Decodes a length-delimited child into an existing message, restoring parent bounds.

bytes_equalsince 0.36.1Source
(a: [] u8, b: [] u8) -> bool

Compares byte sequences by content.

valid_utf8since 0.36.1Source
(bytes: [] u8) -> bool

Strict UTF-8 validation: rejects overlong encodings, surrogate code points and > U+10FFFF.

scalar_bitssince 0.36.1Source
(value: $T, $kind: Scalar) -> u64

Maps a scalar value to its unsigned wire bits; kind is specialized at compile time.

scalar_sizesince 0.36.1Source
(value: $T, $kind: Scalar) -> s64

Encoded scalar payload length, without its field tag.

free_scalarsince 0.36.1Source
(value: *$T, $kind: Scalar)

Releases an owned scalar payload; numeric values need no allocation.

read_scalarsince 0.36.1Source
(r: *Reader, value: *$T, $kind: Scalar) -> bool

Reads a scalar payload, replacing an owned string or byte buffer only after validation.

enum_knownsince 0.36.1Source
(value: $T) -> bool

Tests declared enum numbers; specialization makes the membership table constant.

read_groupsince 0.36.1Source
(r: *Reader, value: *$T) -> bool

Reads a typed group with a matching end tag and the same recursion budget as messages.

(value: *$T, r: *Reader) -> bool

Recursively checks required presence after merging all occurrences.

(negative: bool, nan: bool) -> float64

Constructs special floating schema defaults during generation, preserving IEEE values.

end_groupsince 0.36.1Source
(r: *Reader, tag: u64) -> bool

Completes a typed group only when its end tag matches the active field.

skip_unknownsince 0.36.1Source
(r: *Reader, tag: u64) -> bool

Skips and retains an unmatched complete field from the current message.

Fields#

write_tagsince 0.36.1Source
(w: *Writer, $tag: s64)

Writes a constant tag as literal bytes, with no runtime varint loop.

tag_sizesince 0.36.1Source
(tag: s64) -> s64

Size of a tag, evaluated at generation/specialization time.

has_fieldsince 0.36.1Source
(value: $T, $kind: Scalar, force := false) -> bool

Tests implicit field presence; forced presence also encodes scalar defaults.

size_fieldsince 0.36.1Source
(value: $T, $kind: Scalar, $tag: s64, cache: *[..] u32, force := false) -> s64

Sizes one singular field and appends nested message sizes in preorder.

size_messagesince 0.36.1Source
(value: *$T, cache: *[..] u32) -> s64

Saturates oversized message sizes before storing the u32 cache entry.

encode_fieldsince 0.36.1Source
(w: *Writer, value: $T, $kind: Scalar, $tag: s64, force := false)

Writes one singular field using the preorder size cache.

(values: [] $T, $kind: Scalar, $tag: s64, $packed: bool, cache: *[..] u32) -> s64

Sizes repeated fields, with packed scalars sharing one length prefix.

read_fieldsince 0.36.1Source
(r: *Reader, value: *$T, $kind: Scalar, wire: u64, path: string) -> bool

Reads a singular scalar/message occurrence; wrong wire types remain unknown.

(r: *Reader, values: *[..] $T, $kind: Scalar, wire: u64, path: string) -> bool

Reads packed or expanded repeated occurrences, preserving bounds and index paths.

equal_fieldsince 0.36.1Source
(a: $T, b: T, $kind: Scalar) -> bool

Compares scalar values or optional message pointers by content.

size_mapsince 0.36.1Source
(map: *$T, $E: Type, $mode: Map_Storage, $tag: s64, cache: *[..] u32) -> s64

Sizes all map entries using one shared codec specialization.

equal_mapsince 0.36.1Source
(a: *$T, b: *T, $E: Type, $mode: Map_Storage) -> bool

Map equality uses lookup for HASH_MAP and TABLE, or order for ARRAY.

read_mapsince 0.36.1Source
(r: *Reader, map: *$T, $E: Type, $mode: Map_Storage, wire: u64, path: string) -> bool

Reads a map entry, replacing owned duplicate keys and values after a successful read.

free_oneofsince 0.36.1Source
(v: *$V)

Frees the selected protobuf oneof case and resets its presence.

read_oneofsince 0.36.1Source
(r: *Reader, v: *$V, $selected: V.Which, $kind: Scalar, wire: u64, path: string) -> bool

Reads a oneof occurrence without changing the selected case for a rejected closed enum.

equal_oneofsince 0.36.1Source
(a: *$V, b: *V) -> bool

Compares the selected case and its payload, with no field-name reflection at runtime.

JSON#

to_jsonsince 0.36.1Source
(value: $T, options := Json_Options.{}) -> (string, Error)

The proto3 JSON of a generated message, allocated with context.allocator. Fails with .BAD_VALUE for a value JSON can't express (a Timestamp out of range, a NaN in a Value) and .UNKNOWN_TYPE for an Any whose type isn't in options.registry.

write_jsonsince 0.36.1Source
(builder: *String_Builder, value: $T, options := Json_Options.{}) -> Error

Appends the proto3 JSON of a message to a builder. On failure part of it may be written.

from_jsonsince 0.36.1Source
($T: Type, text: string, options := Json_Options.{}) -> (T, Error)

Decodes a message from its proto3 JSON. A failure releases everything and returns a zero value; the error's message says what was wrong and path where.

merge_jsonsince 0.36.1Source
(value: *$T, text: string, options := Json_Options.{}) -> Error

Merges proto3 JSON into a message: present scalars replace, repeated fields append, messages merge and map keys replace. On failure the merged prefix stays owned by value.

Extensions#

get2 overloadssince 0.36.1Source
(message: *$E, extension: Extension(E, $T, $kind)) -> (T, bool)
(message: *$E, extension: Repeated_Extension(E, $T, $kind)) -> [..] T

Reads a singular extension. The last occurrence of a scalar wins, and message occurrences merge. Returns the default and false when absent. Strings, bytes and the strings inside a message point into the extendee's unknown fields, so they stay valid until it is changed, freed or decoded into. A message value owns blocks: release it with free.

has2 overloadssince 0.36.1Source
(message: *$E, extension: Extension(E, $T, $kind)) -> bool
(message: *$E, extension: Repeated_Extension(E, $T, $kind)) -> bool

Tests whether a singular extension is present.

set2 overloadssince 0.36.1Source
(message: *$E, extension: Extension(E, $T, $kind), value: T) -> Error
(message: *$E, extension: Repeated_Extension(E, $T, $kind), values: [] T) -> Error

Sets a singular extension, replacing every earlier occurrence. A message value is encoded (copied) and stays the caller's; it fails like encode when a required field is missing, leaving the extendee unchanged.

(r: *Reader, tag: u64, $kind: Scalar, $T: Type, name: string) -> bool

Validates an extension value met while decoding, as a normal field of that type would be: a string must be UTF-8, and a message must decode. Called from the extendee's generated decoder only for extensions that need it; skip_utf8 turns the check off.

Storage#

(mode: Allocator_Mode, size: s64, old_size: s64, old: *void, data: *void) -> *void

Ignores individual frees into owned blocks, delegating every other operation.

add_storagesince 0.36.1Source
(r: *Reader, count: s64) -> [] u8

Adds one bounded payload allocation to the current top-level message's storage.

add_borrowedsince 0.36.1Source
(r: *Reader)

Records a borrowed input: its strings and bytes are views that free must skip.

holdssince 0.36.1Source
(block: *Storage_Block, address: u64) -> bool

Tests whether a pointer lies in a block's payload or in its borrowed input.

(r: *Reader, count: s64) -> [] u8

Places one field in a geometrically grown chunk bounded by unread input bytes.

free_messagesince 0.36.1Source
(value: *$T)

Shared generated-message free wrapper; nested messages with no independent blocks use the enclosing scope. Hand-built values still release their ordinary allocations.

(r: *Reader, value: *$T, $kind: Scalar)

Wrap only frees, never allocations: resizable arrays retain their original allocator.

(r: *Reader, value: *$T)

Releases a temporary decoded map entry under its top-level storage scope.

(r: *Reader, value: *$T)

Releases the previous oneof selection without freeing block-backed payloads.

owns_payloadsince 0.36.1Source
(block: *Storage_Block, pointer: *void) -> bool

Tests whether a field aliases any payload block or borrowed input of this owner.

(mode: Allocator_Mode, size: s64, old_size: s64, old: *void, data: *void) -> *void

Allocates arrays and message objects from a stable per-decode arena. Array allocators retain this heap header, never a Reader or another stack variable. Growing the newest allocation extends it in place; any other outgrown buffer becomes scrap that later allocations are carved from, so growth leaves no dead space.

grow_blocksince 0.36.1Source
(root: *Storage_Block, needed: s64) -> *void

Resizes the current block, which holds exactly one array, and relinks it.

arena_scrapsince 0.36.1Source
(root: *Storage_Block, memory: *void, size: s64)

Records a dead region; regions too small to hold the link are simply lost.

arena_takesince 0.36.1Source
(root: *Storage_Block, needed: s64) -> *void

Bump-allocates from the current region, moving to a scrap region or a new block when full.

(r: *Reader) -> Allocator

Produces the stable allocator only when a structural allocation is needed. Borrowed decodes use it too: only string and bytes payloads differ between the two modes.

(r: *Reader, values: *[..] $T, value: T)

Grows a repeated array with a stable arena allocator captured by its container.

add_decodedsince 0.36.1Source
(r: *Reader, values: *[..] $T) -> *T

Appends a zeroed element for the decoder to fill, with the same allocator as append_decoded.

(r: *Reader, child: **Storage_Block)

Moves an independently decoded child's storage into the enclosing message before merging it. Stable arena headers remain alive until the enclosing owner is freed.

(r: *Reader, bytes: [] u8)

Unknown wire copies use the same stable arena as repeated containers.

set_stringsince 0.36.1Source
(owner: *$T, field: *string, replacement: string)

Replaces a string with an owned copy. This accepts a slice from another decoded message without sharing its lifetime. A directly assigned non-block string must be an individually owned allocation from the message's allocator.

set_bytessince 0.36.1Source
(owner: *$T, field: *[] u8, replacement: [] u8)

Replaces bytes with an owned copy, including bytes from another decoded message.

(pointer: *void)

Avoids routing block-backed pointers through the general heap free machinery.

(scope: *Storage_Allocator, pointer: *void) -> bool

Consecutive field releases usually hit the same chunk. Cache that interval so payload frees do not walk the structural-block list for every string.

Gen#

mainsince 0.36.1Source
()

Prints the generated source for the schema path given on the command line.

Types#

Error_Kindenumsince 0.36.1Source

Why encoding or decoding stopped. Decode offsets are measured from the start of input.

Error_Kind :: enum u8 {
    NONE;
    TRUNCATED;
    BAD_TAG;
    BAD_WIRE_TYPE;
    BAD_GROUP;
    VARINT_TOO_LONG;
    TOO_DEEP;
    TOO_LARGE;
    BAD_UTF8;
    MISSING_REQUIRED;
    FIELD_ZERO;
    BAD_JSON;
    BAD_VALUE;
    UNKNOWN_FIELD;
    DUPLICATE_FIELD;
    UNKNOWN_TYPE;
}
Errorstructsince 0.36.1Source

An encoding or decoding failure. The owned path names a field, including repeated indices.

Error :: struct {
    kind: Error_Kind;
    offset: s64;
    path: string;
    message: string;
}
Decode_Optionsstructsince 0.36.1Source

Limits for decode and merge. With borrow, strings and bytes point into the input, which must outlive the result; free knows not to release them.

Decode_Options :: struct {
    max_depth: s64 = 100;
    max_size: s64 = 2147483647;
    borrow := false;
    discard_unknown := false;
    skip_utf8 := false;
}
Optionalstructsince 0.36.1Source

Explicit scalar presence, including a present default value.

Optional :: struct(T: Type) {
    has: bool;
    value: T;
}
Map_Entrystructsince 0.36.1Source

A map entry for MAP=.ARRAY; duplicates and wire order are retained.

Map_Entry :: struct(K: Type, V: Type) {
    key: K;
    value: V;
}
Scalarenumsince 0.36.1Source

Wire kinds specialized by generated field operations.

Scalar :: enum {
    INT32;
    INT64;
    UINT32;
    UINT64;
    SINT32;
    SINT64;
    FIXED32;
    FIXED64;
    SFIXED32;
    SFIXED64;
    FLOAT;
    DOUBLE;
    BOOL;
    STRING;
    BYTES;
    ENUM;
    MESSAGE;
    GROUP;
    CLOSED_ENUM;
    RAW_STRING;
}
Readerstructsince 0.36.1Source

Decoder state shared by nested generated messages; end bounds only decrease for a child.

Reader :: struct {
    bytes: [] u8;
    pos: s64;
    end: s64 = -1;
    depth: s64;
    options: Decode_Options;
    error: Error;
    group: u64;
    group_ended: bool;
    unknown: *[..] u8;
    tag_start: s64;
    enum_tag: u64;
    enum_rejected: bool;
    suppress_enum_unknown: bool;
    storage: **Storage_Block;
    payload: [] u8;
    chunk_size: s64;
    borrow_recorded: bool;
}
Writerstructsince 0.36.1Source

Exact output buffer and preorder nested-message sizes used by generated encoders.

Writer :: struct {
    bytes: [] u8;
    pos: s64;
    cache: [] u32;
    next: s64;
}
Entry_Codecstructsince 0.36.1Source

Shared map-entry codec; it never stores unknown fields or emits synthetic schema types.

Entry_Codec :: struct(K: Type, V: Type, key_kind: Scalar, value_kind: Scalar) {
    key: K;
    value: V = #run entry_default(V);
    invalid_enum: bool;
    proto_storage: *Storage_Block;
    proto_has_required :: false;

    proto_check :: (entry: *Entry_Codec, r: *Reader) -> bool {
        #if value_kind == .MESSAGE return check_message(entry.value, r);
        else return true;
    }

    proto_size :: (entry: *Entry_Codec, cache: *[..] u32) -> s64 {
        slot := cache.count;

        array_add(cache, 0);
        n := size_field(entry.key, key_kind, 8, cache, true) + size_field(entry.value, value_kind, 16 + cast(s64) scalar_wire(value_kind), cache, true);

        cache.data[slot] = cast(u32) min(n, 2147483648);

        return min(n, 2147483648);
    }

    proto_encode :: (entry: *Entry_Codec, w: *Writer) {
        encode_field(w, entry.key, key_kind, 8 + cast(s64) scalar_wire(key_kind), true);
        encode_field(w, entry.value, value_kind, 16 + cast(s64) scalar_wire(value_kind), true);
    }

    proto_free :: (entry: *Entry_Codec) {
        free_field(*entry.key, key_kind);
        free_field(*entry.value, value_kind);
    }

    proto_merge :: (entry: *Entry_Codec, r: *Reader) -> bool {
        suppressed := r.suppress_enum_unknown;

        #if value_kind == .CLOSED_ENUM r.suppress_enum_unknown = true;
        defer r.suppress_enum_unknown = suppressed;

        while remaining(r) > 0 {
            tag := read_tag(r);

            if r.error.kind != .NONE return false;

            if tag == 8 + scalar_wire(key_kind) {
                if !read_scalar(r, *entry.key, key_kind) return field_error(r, "key");
            } else if tag == 16 + scalar_wire(value_kind) {
                #if value_kind == .MESSAGE {
                    if !entry.value entry.value = new_message(r, type_of(entry.value.*));
                    if !read_message(r, entry.value) return field_error(r, "value");
                } else {
                    if !read_scalar(r, *entry.value, value_kind) return field_error(r, "value");
                    #if value_kind == .CLOSED_ENUM if r.enum_rejected entry.invalid_enum = true;
                }
            } else if !skip(r, tag) return false;
        }

        return true;
    }
}
Map_Storageenumsince 0.36.1Source

Map container selection for the shared map operations.

Map_Storage :: enum {
    HASH_MAP;
    TABLE;
    ARRAY;
}
Json_Optionsstructsince 0.36.1Source

How to_json writes and from_json reads.

Json_Options :: struct {
    emit_defaults := false;
    proto_names := false;
    enum_numbers := false;
    pretty := false;
    ignore_unknown_fields := false;
    max_depth: s64 = 100;
    registry: [] Json_Type;
}
Json_Typestructsince 0.36.1Source

One message type of an Any registry: its full .proto name and its JSON codec.

Json_Type :: struct {
    name: string;
    run: Json_Any_Proc;
}
Json_Any_Proctypesince 0.36.1Source
:: #type (out: *Json_Out, input: *Json_In, bytes: [] u8, result: *[] u8) -> bool

Writes the fields of bytes (with out) or reads them into result (with input).

Json_Anystructsince 0.36.1Source

The JSON codec of one message type, for Any. Generated schemas list one per message in proto_registry.

Json_Any :: struct(T: Type) {
    run :: (out: *Json_Out, input: *Json_In, bytes: [] u8, result: *[] u8) -> bool {
        if out {
            value, error := decode(T, bytes);

            defer T.proto_free(*value);
            defer json_free_error(*error);

            if error.kind != .NONE return json_write_fail(out, .BAD_VALUE, "an Any holds bytes that don't decode as its type");

            first := false;

            return json_write_any_body(out, *value, *first);
        }

        value: T;
        saved := input.pr;

        input.pr = Reader.{ bytes = saved.bytes, pos = saved.pos, storage = *value.proto_storage };
        ok := json_read_any_body(input, *value);

        input.pr = saved;

        if ok {
            encoded, error := encode(value);

            if error.kind != .NONE {
                json_free_error(*error);
                ok = Json.json_fail(*input.r, .INVALID_VALUE, "an Any's message doesn't encode");
            } else result.* = encoded;
        }

        T.proto_free(*value);

        return ok;
    }
}
Well_Knownenumsince 0.36.1Source

Which special JSON form a well-known type has. Generated well-known messages say theirs.

Well_Known :: enum u8 {
    NONE;
    TIMESTAMP;
    DURATION;
    FIELD_MASK;
    STRUCT;
    VALUE;
    LIST_VALUE;
    ANY;
    EMPTY;
    WRAPPER;
}
Extensionstructsince 0.36.1Source

A singular extension of message Extendee holding a T. The Schema module generates one constant per extend field; kind selects the wire form. Without a value get returns default.

Extension :: struct(Extendee: Type, T: Type, kind: Scalar) {
    number: s64;
    name: string;
    default: T;
}
Repeated_Extensionstructsince 0.36.1Source

A repeated extension of message Extendee with elements of type T.

Repeated_Extension :: struct(Extendee: Type, T: Type, kind: Scalar) {
    number: s64;
    name: string;
    packed: bool;
}
Storage_Blockstructsince 0.36.1Source

One top-level decode owns the blocks used by all its children. A header and its payload share one allocation; only actual field bytes are copied into payload chunks. A borrowed decode adds a block that records its input, so fields pointing there are never freed either.

Storage_Block :: struct {
    next: *Storage_Block;
    count: s64;
    borrowed: [] u8;
    parent: Allocator;
    arena: [] u8;
    arena_limit: s64;
    arena_block: *Storage_Block;
    scrap: *Scrap;
}
Scrapstructsince 0.36.1Source

A dead region of the arena (an outgrown array buffer or a block tail) that later small allocations are served from. The link lives inside the dead memory itself.

Scrap :: struct {
    next: *Scrap;
    size: s64;
}
Storage_Allocatorstructsince 0.36.1Source

Allocator scope used while replacing/freeing fields: interior payload pointers are retained until their owner releases the entire block. Other allocations free normally.

Storage_Allocator :: struct {
    parent: Allocator;
    storage: **Storage_Block;
    hot: *Storage_Block;
}

Constants#

:: 524288

Larger blocks would strand too much unused tail when a decode ends.