Various improvements to the BQN lexer by slotThe ยท Pull Request #2789 ยท pygments/pygments
added 5 commits
October 7, 2024 08:39\w matches all alphanumeric Unicode characters, including ones (e.g., ๐)
that BQN treats special. This is especially troublesome for variables;
previously, something like
๐i
would have returned
(Token.Operator, '๐'),
(Token.Error, 'i'),
instead of
(Token.Operator, '๐'),
(Token.Name.Variable, 'i').
This extends to special sequences like \b, which care about the
difference between \w and \W.
BQN does not actually use * (ASTERISK) anywhere, but there is a primitive function โ (STAR OPERATOR).
slotThe
changed the title
Fix BQN lexer treating special characters as word chars
Various improvements to the BQN lexer
slotThe
deleted the
bqn/inter-word-chars
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters