Jai_Lexer

Prints an error at the lexer's current position, formatted like print, and marks the lexer as having reported one.

True for a character that can begin an identifier: a letter or _.

assert(starts_identifier(#char "_") && !starts_identifier(#char "7"));

Consumes the next token.

lexer: Lexer;
set_input_from_string(*lexer, "a b");
eat_token(*lexer);
assert(peek_next_token(*lexer).ident_value.name == "b");

Returns a copy of the token most recently consumed with eat_token, for reporting its position after you have moved on.

Returns the next token without consuming it. Same as peek_token(lexer, 0).

Returns the token lookahead_index places ahead without consuming it; 0 is the next token. The index must be below MAX_CONCURRENT_TOKENS, and the pointer stays valid until you eat past it.

lexer: Lexer;
set_input_from_string(*lexer, "a + b");
assert(peek_token(*lexer, 1).type == #char "+");
assert(peek_token(*lexer, 0).ident_value.name == "a");

Reads the file at file_path and lexes it. Returns false if it cannot be read. The lexer frees the text when it is given new input.

lexer: Lexer;
assert(!set_input_from_file(*lexer, "no/such/file.jai"));

Points lexer at input and resets its position. The lexer reads input in place, so keep it alive while you lex.

lexer: Lexer;
set_input_from_string(*lexer, "x := 1;");
assert(peek_next_token(*lexer).type == .IDENT);

Sets an identifier token's type to the keyword it spells, if any, and returns the type.

lexer: Lexer;
set_input_from_string(*lexer, "struct");
token := peek_next_token(*lexer);
assert(check_for_keyword(token) == .KEYWORD_STRUCT);

Which hex escape a string is reading: \x (two digits), \u (four) or \U (eight).

Flags on a @note.

One token: its type, where it starts (l0, c0, line and column) and ends (l1, c1), and its value. Read ident_value for an identifier, integer_value or float64_value for a number (check value_flags), string_value for a string.

What kind of token this is. Single-character tokens use their character as the value (#char "+"); everything else is named.

How a string or number token was written: as a here-string, in hex or binary, as a float, a float that needs 64-bit precision, or an integer that overflowed.

The state of a lex in progress: the input, the position, and the tokens read ahead. Start from a zeroed Lexer and call set_input_from_string or set_input_from_file.