Encodes into one exact-sized allocation; TOO_LARGE returns no bytes before allocation.
#import "Extensions/Protobuf";
Protobuf messages generated by Extensions/Protobuf/Schema, in binary and proto3 JSON.
Import with a name: Proto :: #import "Extensions/Protobuf";.
Decoded allocations belong to context.allocator and are released by Proto.free.
Copy decode owns shared payload/container blocks at the top-level message. Free that
owner, and treat its nested values as views. Use set_string/set_bytes to copy a field
from another decoded owner. Directly assigned strings/bytes must be individually owned
allocations (or use a no-op-free arena for the whole message).
to_json/from_json speak the proto3 JSON mapping; give an Any's types with the schema's
proto_registry in Json_Options.
Procedures#
General#
Appends a generated message to a resizable byte array.
Computes the encoded byte length without allocating output, saturating at 2 GiB.
Decodes a generated message. A failure releases all allocations and returns a zero value.
Merges wire data into a generated message. On failure the successfully merged prefix remains owned by value and must be freed. Strings and scalar values replace; message values merge.
Releases a decoded message recursively and resets it; safe to call twice.
Deep equality of generated messages, including preserved unknown bytes.
A path relative to the calling Jai source, allocated with context.allocator.
Wire#
Records the first failure; generated code adds path segments while unwinding.
Prepends a field path to a nested error. Called only on failure.
Remaining bytes within the active message or packed field.
Reads a bounded unsigned varint. One-byte values stay inline at the call site.
The multi-byte path of read_varint, rejecting overflow in the tenth byte.
Number of bytes in an unsigned varint.
Writes a varint into the exact-sized output buffer. One-byte values (tags, short lengths, small numbers) stay inline at the call site.
The multi-byte path of write_varint.
Zigzag maps the signed 64-bit extremes to unsigned varints without signed overflow.
Inverse zigzag. For sint32 callers truncate to u32 before calling.
Reads a little-endian fixed-width integer.
Writes the low width bytes in little-endian order. Every supported target (x64, arm64, wasm) is little endian, so this is one store.
Reads and checks a length before any slice or allocation. A one-byte length that fits stays inline; anything else, including every failure, takes read_long_length.
Size limits precede truncation.
Validates a protobuf tag before narrowing it. A one-byte tag can't be too long or too large, so it only needs a nonzero field and a known wire type.
The multi-byte path of read_tag, and every failure.
Skips an unknown wire field, including a matching group as one opaque unit.
Allocates byte storage whose caller must completely initialize before exposing it.
Appends bytes to a sized output buffer. Up to 16 bytes are copied with two overlapping fixed-size moves, which compile to plain loads and stores instead of a memcpy call.
Writes a message using its preorder size entry (no back-patching).
Decodes a length-delimited child into an existing message, restoring parent bounds.
Compares byte sequences by content.
Strict UTF-8 validation: rejects overlong encodings, surrogate code points and > U+10FFFF.
Maps a scalar value to its unsigned wire bits; kind is specialized at compile time.
Wire type for a scalar kind.
Encoded scalar payload length, without its field tag.
Writes a scalar payload without its tag.
Releases an owned scalar payload; numeric values need no allocation.
Reads a scalar payload, replacing an owned string or byte buffer only after validation.
Tests declared enum numbers; specialization makes the membership table constant.
Reads a typed group with a matching end tag and the same recursion budget as messages.
Recursively checks required presence after merging all occurrences.
Constructs special floating schema defaults during generation, preserving IEEE values.
Completes a typed group only when its end tag matches the active field.
Skips and retains an unmatched complete field from the current message.
Fields#
Writes a constant tag as literal bytes, with no runtime varint loop.
Size of a tag, evaluated at generation/specialization time.
Tests implicit field presence; forced presence also encodes scalar defaults.
Sizes one singular field and appends nested message sizes in preorder.
Saturates oversized message sizes before storing the u32 cache entry.
Writes one singular field using the preorder size cache.
Sizes an explicitly present scalar.
Writes an explicitly present scalar.
Sizes repeated fields, with packed scalars sharing one length prefix.
Writes repeated fields, including packed scalars.
Releases a scalar or optional message pointer.
Reads a singular scalar/message occurrence; wrong wire types remain unknown.
Reads an explicitly present scalar, setting presence only after success.
Reads packed or expanded repeated occurrences, preserving bounds and index paths.
Compares scalar values or optional message pointers by content.
Compares explicit presence, ignoring the payload when absent.
Compares repeated field values in order.
Releases every repeated element and its container.
Sizes all map entries using one shared codec specialization.
Writes all map entries in the container's iteration order.
Releases map keys, values and container storage.
Map equality uses lookup for HASH_MAP and TABLE, or order for ARRAY.
Reads a map entry, replacing owned duplicate keys and values after a successful read.
Frees the selected protobuf oneof case and resets its presence.
Reads a oneof occurrence without changing the selected case for a rejected closed enum.
Compares the selected case and its payload, with no field-name reflection at runtime.
JSON#
The proto3 JSON of a generated message, allocated with context.allocator. Fails with
.BAD_VALUE for a value JSON can't express (a Timestamp out of range, a NaN in a Value) and
.UNKNOWN_TYPE for an Any whose type isn't in options.registry.
Appends the proto3 JSON of a message to a builder. On failure part of it may be written.
Decodes a message from its proto3 JSON. A failure releases everything and returns a zero
value; the error's message says what was wrong and path where.
Merges proto3 JSON into a message: present scalars replace, repeated fields append, messages merge and map keys replace. On failure the merged prefix stays owned by value.
Extensions#
Reads a singular extension. The last occurrence of a scalar wins, and message occurrences
merge. Returns the default and false when absent. Strings, bytes and the strings inside a
message point into the extendee's unknown fields, so they stay valid until it is changed,
freed or decoded into. A message value owns blocks: release it with free.
Tests whether a singular extension is present.
Sets a singular extension, replacing every earlier occurrence. A message value is encoded
(copied) and stays the caller's; it fails like encode when a required field is missing,
leaving the extendee unchanged.
Appends one element to a repeated extension.
Removes a singular extension.
Validates an extension value met while decoding, as a normal field of that type would be:
a string must be UTF-8, and a message must decode. Called from the extendee's generated
decoder only for extensions that need it; skip_utf8 turns the check off.
Storage#
Ignores individual frees into owned blocks, delegating every other operation.
Links a new block of count payload bytes into the current top-level message's storage.
Adds one bounded payload allocation to the current top-level message's storage.
Records a borrowed input: its strings and bytes are views that free must skip.
Tests whether a pointer lies in a block's payload or in its borrowed input.
Places one field in a geometrically grown chunk bounded by unread input bytes.
Shared generated-message free wrapper; nested messages with no independent blocks use the enclosing scope. Hand-built values still release their ordinary allocations.
Releases a replaced scalar while retaining interior pointers in decode storage.
Wrap only frees, never allocations: resizable arrays retain their original allocator.
Releases a temporary decoded map entry under its top-level storage scope.
Releases the previous oneof selection without freeing block-backed payloads.
Tests whether a field aliases any payload block or borrowed input of this owner.
Allocates arrays and message objects from a stable per-decode arena. Array allocators retain this heap header, never a Reader or another stack variable. Growing the newest allocation extends it in place; any other outgrown buffer becomes scrap that later allocations are carved from, so growth leaves no dead space.
Resizes the current block, which holds exactly one array, and relinks it.
Records a dead region; regions too small to hold the link are simply lost.
Bump-allocates from the current region, moving to a scrap region or a new block when full.
Produces the stable allocator only when a structural allocation is needed. Borrowed decodes use it too: only string and bytes payloads differ between the two modes.
Constructs a singular message in decode storage.
Grows a repeated array with a stable arena allocator captured by its container.
Appends a zeroed element for the decoder to fill, with the same allocator as append_decoded.
Moves an independently decoded child's storage into the enclosing message before merging it. Stable arena headers remain alive until the enclosing owner is freed.
Unknown wire copies use the same stable arena as repeated containers.
Appends one re-encoded closed-enum tag byte to unknown storage.
Replaces a string with an owned copy. This accepts a slice from another decoded message without sharing its lifetime. A directly assigned non-block string must be an individually owned allocation from the message's allocator.
Replaces bytes with an owned copy, including bytes from another decoded message.
Avoids routing block-backed pointers through the general heap free machinery.
Consecutive field releases usually hit the same chunk. Cache that interval so payload frees do not walk the structural-block list for every string.
Gen#
Prints the generated source for the schema path given on the command line.
Types#
Why encoding or decoding stopped. Decode offsets are measured from the start of input.
Error_Kind :: enum u8 {
NONE;
TRUNCATED;
BAD_TAG;
BAD_WIRE_TYPE;
BAD_GROUP;
VARINT_TOO_LONG;
TOO_DEEP;
TOO_LARGE;
BAD_UTF8;
MISSING_REQUIRED;
FIELD_ZERO;
BAD_JSON;
BAD_VALUE;
UNKNOWN_FIELD;
DUPLICATE_FIELD;
UNKNOWN_TYPE;
}An encoding or decoding failure. The owned path names a field, including repeated indices.
Error :: struct {
kind: Error_Kind;
offset: s64;
path: string;
message: string;
}Limits for decode and merge. With borrow, strings and bytes point into the input, which
must outlive the result; free knows not to release them.
Decode_Options :: struct {
max_depth: s64 = 100;
max_size: s64 = 2147483647;
borrow := false;
discard_unknown := false;
skip_utf8 := false;
}Explicit scalar presence, including a present default value.
Optional :: struct(T: Type) {
has: bool;
value: T;
}A map entry for MAP=.ARRAY; duplicates and wire order are retained.
Map_Entry :: struct(K: Type, V: Type) {
key: K;
value: V;
}Wire kinds specialized by generated field operations.
Scalar :: enum {
INT32;
INT64;
UINT32;
UINT64;
SINT32;
SINT64;
FIXED32;
FIXED64;
SFIXED32;
SFIXED64;
FLOAT;
DOUBLE;
BOOL;
STRING;
BYTES;
ENUM;
MESSAGE;
GROUP;
CLOSED_ENUM;
RAW_STRING;
}Decoder state shared by nested generated messages; end bounds only decrease for a child.
Reader :: struct {
bytes: [] u8;
pos: s64;
end: s64 = -1;
depth: s64;
options: Decode_Options;
error: Error;
group: u64;
group_ended: bool;
unknown: *[..] u8;
tag_start: s64;
enum_tag: u64;
enum_rejected: bool;
suppress_enum_unknown: bool;
storage: **Storage_Block;
payload: [] u8;
chunk_size: s64;
borrow_recorded: bool;
}Exact output buffer and preorder nested-message sizes used by generated encoders.
Writer :: struct {
bytes: [] u8;
pos: s64;
cache: [] u32;
next: s64;
}Shared map-entry codec; it never stores unknown fields or emits synthetic schema types.
Entry_Codec :: struct(K: Type, V: Type, key_kind: Scalar, value_kind: Scalar) {
key: K;
value: V = #run entry_default(V);
invalid_enum: bool;
proto_storage: *Storage_Block;
proto_has_required :: false;
proto_check :: (entry: *Entry_Codec, r: *Reader) -> bool {
#if value_kind == .MESSAGE return check_message(entry.value, r);
else return true;
}
proto_size :: (entry: *Entry_Codec, cache: *[..] u32) -> s64 {
slot := cache.count;
array_add(cache, 0);
n := size_field(entry.key, key_kind, 8, cache, true) + size_field(entry.value, value_kind, 16 + cast(s64) scalar_wire(value_kind), cache, true);
cache.data[slot] = cast(u32) min(n, 2147483648);
return min(n, 2147483648);
}
proto_encode :: (entry: *Entry_Codec, w: *Writer) {
encode_field(w, entry.key, key_kind, 8 + cast(s64) scalar_wire(key_kind), true);
encode_field(w, entry.value, value_kind, 16 + cast(s64) scalar_wire(value_kind), true);
}
proto_free :: (entry: *Entry_Codec) {
free_field(*entry.key, key_kind);
free_field(*entry.value, value_kind);
}
proto_merge :: (entry: *Entry_Codec, r: *Reader) -> bool {
suppressed := r.suppress_enum_unknown;
#if value_kind == .CLOSED_ENUM r.suppress_enum_unknown = true;
defer r.suppress_enum_unknown = suppressed;
while remaining(r) > 0 {
tag := read_tag(r);
if r.error.kind != .NONE return false;
if tag == 8 + scalar_wire(key_kind) {
if !read_scalar(r, *entry.key, key_kind) return field_error(r, "key");
} else if tag == 16 + scalar_wire(value_kind) {
#if value_kind == .MESSAGE {
if !entry.value entry.value = new_message(r, type_of(entry.value.*));
if !read_message(r, entry.value) return field_error(r, "value");
} else {
if !read_scalar(r, *entry.value, value_kind) return field_error(r, "value");
#if value_kind == .CLOSED_ENUM if r.enum_rejected entry.invalid_enum = true;
}
} else if !skip(r, tag) return false;
}
return true;
}
}Map container selection for the shared map operations.
Map_Storage :: enum {
HASH_MAP;
TABLE;
ARRAY;
}How to_json writes and from_json reads.
Json_Options :: struct {
emit_defaults := false;
proto_names := false;
enum_numbers := false;
pretty := false;
ignore_unknown_fields := false;
max_depth: s64 = 100;
registry: [] Json_Type;
}One message type of an Any registry: its full .proto name and its JSON codec.
Json_Type :: struct {
name: string;
run: Json_Any_Proc;
}Writes the fields of bytes (with out) or reads them into result (with input).
The JSON codec of one message type, for Any. Generated schemas list one per message in
proto_registry.
Json_Any :: struct(T: Type) {
run :: (out: *Json_Out, input: *Json_In, bytes: [] u8, result: *[] u8) -> bool {
if out {
value, error := decode(T, bytes);
defer T.proto_free(*value);
defer json_free_error(*error);
if error.kind != .NONE return json_write_fail(out, .BAD_VALUE, "an Any holds bytes that don't decode as its type");
first := false;
return json_write_any_body(out, *value, *first);
}
value: T;
saved := input.pr;
input.pr = Reader.{ bytes = saved.bytes, pos = saved.pos, storage = *value.proto_storage };
ok := json_read_any_body(input, *value);
input.pr = saved;
if ok {
encoded, error := encode(value);
if error.kind != .NONE {
json_free_error(*error);
ok = Json.json_fail(*input.r, .INVALID_VALUE, "an Any's message doesn't encode");
} else result.* = encoded;
}
T.proto_free(*value);
return ok;
}
}Which special JSON form a well-known type has. Generated well-known messages say theirs.
Well_Known :: enum u8 {
NONE;
TIMESTAMP;
DURATION;
FIELD_MASK;
STRUCT;
VALUE;
LIST_VALUE;
ANY;
EMPTY;
WRAPPER;
}The state of a running to_json, for Any codecs.
Json_Out :: struct {
w: Json.Json_Writer;
options: Json_Options;
error: Error;
}The state of a running from_json, for Any codecs.
Json_In :: struct {
r: Json.Json_Reader;
pr: Reader;
options: Json_Options;
skipped: bool;
}A singular extension of message Extendee holding a T. The Schema module generates one
constant per extend field; kind selects the wire form. Without a value get returns
default.
Extension :: struct(Extendee: Type, T: Type, kind: Scalar) {
number: s64;
name: string;
default: T;
}A repeated extension of message Extendee with elements of type T.
Repeated_Extension :: struct(Extendee: Type, T: Type, kind: Scalar) {
number: s64;
name: string;
packed: bool;
}One top-level decode owns the blocks used by all its children. A header and its payload share one allocation; only actual field bytes are copied into payload chunks. A borrowed decode adds a block that records its input, so fields pointing there are never freed either.
A dead region of the arena (an outgrown array buffer or a block tail) that later small allocations are served from. The link lives inside the dead memory itself.
Scrap :: struct {
next: *Scrap;
size: s64;
}Allocator scope used while replacing/freeing fields: interior payload pointers are retained until their owner releases the entire block. Other allocations free normally.
Storage_Allocator :: struct {
parent: Allocator;
storage: **Storage_Block;
hot: *Storage_Block;
}Constants#
Larger blocks would strand too much unused tail when a decode ends.