Expand description
SQL Tokenizer
The tokenizer (a.k.a. lexer) converts a string into a sequence of tokens.
The tokens then form the input for the parser, which outputs an Abstract Syntax Tree (AST).
- Location
- Location in input string
- Span
- A span represents a linear portion of the input string (start, end)
- Token
With Span - A Token with Span attached to it
- Tokenizer
- SQL Tokenizer
- Tokenizer
Error - An error reported by the tokenizer, with a human-readable
messageand alocation. - Word
- A keyword (like SELECT) or an optionally quoted SQL identifier
- Token
- SQL Token enumeration
- Whitespace
- Represents whitespace in the input: spaces, newlines, tabs and comments.
- Token
With Location Deprecated - Backwards compatibility struct for
TokenWithSpan