The state of a lex in progress: the input, the position, and the tokens read ahead. Start from a zeroed Lexer and call set_input_from_string or set_input_from_file.
Fields
- current_line_number: s32
- current_external_file_error_report_line_number: s32
- current_character_index: s32
- atom_pool: P.Pool
- tokens: [MAX_CONCURRENT_TOKENS] Token
- token_array_cursor: s32
- num_incoming_tokens: s32
- input: string
- input_cursor: s64
- should_free_input: := false
- total_lines_processed: s64
- nonblank_lines_processed: s64
- previous_token_line_number: s64
- eof_token: Token
- user_data: *void
- scratch_buffer: [MAX_IDENTIFIER_LENGTH] u8
- reported_error: := false