func SplitTokensIntoLines(tokens []Token) (out [][]Token)

func Stringify(tokens ...Token) string

func TokenTypeStrings() []string

type AnalyseConfig struct {
	Regexes []RegexConfig `xml:"regex,omitempty"`
	
	First bool `xml:"first,attr"`
}

func MustParseColour(colour string) Colour

MustParseColour is like ParseColour except it panics if the colour is invalid.

Will panic if colour is in an invalid format.

func NewColour(r, g, b uint8) Colour

NewColour creates a Colour directly from RGB values.

ParseColour in the forms #rgb, #rrggbb, #ansi<colour>, or #<colour>. Will return an "unset" colour if invalid.

Blue component of colour.

func (c Colour) BrightenOrDarken(factor float64) Colour

BrightenOrDarken brightens a colour if it is < 0.5 brightness or darkens if > 0.5 brightness.

Brightness of the colour (roughly) in the range 0.0 to 1.0.

func (c Colour) ClampBrightness(min, max float64) Colour

ClampBrightness returns a copy of this colour with its brightness adjusted such that it falls within the range [min, max] (or very close to it due to rounding errors). The supplied values use the same [0.0, 1.0] range as Brightness.

Distance between this colour and another.

This uses the approach described here (https://www.compuphase.com/cmetric.htm). This is not as accurate as LAB, et. al. but is *vastly* simpler and sufficient for our needs.

Green component of colour.

IsSet returns true if the colour is set.

func (c Colours) Len() int

func (c Colours) Swap(i, j int)

type Config struct {
	
	Name string `xml:"name,omitempty"`

	
	Aliases []string `xml:"alias,omitempty"`

	
	Filenames []string `xml:"filename,omitempty"`

	
	AliasFilenames []string `xml:"alias_filename,omitempty"`

	
	MimeTypes []string `xml:"mime_type,omitempty"`

	
	CaseInsensitive bool `xml:"case_insensitive,omitempty"`

	
	DotAll bool `xml:"dot_all,omitempty"`

	
	
	
	NotMultiline bool `xml:"not_multiline,omitempty"`

	
	
	EnsureNL bool `xml:"ensure_nl,omitempty"`

	
	
	
	Priority float32 `xml:"priority,omitempty"`

	
	
	
	
	
	Analyse *AnalyseConfig `xml:"analyse,omitempty"`
}

type Emitter interface {
	
	Emit(groups []string, state *LexerState) Iterator
}

func ByGroupNames(emitters map[string]Emitter) Emitter

ByGroupNames emits a token for each named matching group in the rule's regex.

func ByGroups(emitters ...Emitter) Emitter

ByGroups emits a token for each matching group in the rule's regex.

Using returns an Emitter that uses a given Lexer reference for parsing and emitting.

The referenced lexer must be stored in the same LexerRegistry.

func UsingByGroup(sublexerNameGroup, codeGroup int, emitters ...Emitter) Emitter

UsingByGroup emits tokens for the matched groups in the regex using a sublexer. Used when lexing code blocks where the name of a sublexer is contained within the block, for example on a Markdown text block or SQL language block.

An attempt to load the sublexer will be made using the captured value from the text of the matched sublexerNameGroup. If a sublexer matching the sublexerNameGroup is available, then tokens for the matched codeGroup will be emitted using the sublexer. Otherwise, if no sublexer is available, then tokens will be emitted from the passed emitter.

Example:

var Markdown = internal.Register(MustNewLexer(
	&Config{
		Name:      "markdown",
		Aliases:   []string{"md", "mkd"},
		Filenames: []string{"*.md", "*.mkd", "*.markdown"},
		MimeTypes: []string{"text/x-markdown"},
	},
	Rules{
		"root": {
			{"^(```)(\\w+)(\\n)([\\w\\W]*?)(^```$)",
				UsingByGroup(
					2, 4,
					String, String, String, Text, String,
				),
				nil,
			},
		},
	},
))

See the lexers/markdown.go for the complete example.

Note: panic's if the number of emitters does not equal the number of matched groups in the regex.

func UsingLexer(lexer Lexer) Emitter

UsingLexer returns an Emitter that uses a given Lexer for parsing and emitting.

This Emitter is not serialisable.

UsingSelf is like Using, but uses the current Lexer.

type EmitterFunc func(groups []string, state *LexerState) Iterator

type Formatter interface {
	
	
	
	Format(w io.Writer, style *Style, iterator Iterator) error
}

func RecoveringFormatter(formatter Formatter) Formatter

RecoveringFormatter wraps a formatter with panic recovery.

type Iterator func() Token

func Concaterator(iterators ...Iterator) Iterator

Concaterator concatenates tokens from a series of iterators.

func Literator(tokens ...Token) Iterator

Literator converts a sequence of literal Tokens into an Iterator.

func (i Iterator) Stdlib() func(yield func(Token) bool)

Stdlib converts a Chroma iterator to a Go 1.23-compatible iterator.

func (i Iterator) Tokens() []Token

Tokens consumes all tokens from the iterator and returns them as a slice.

func Coalesce(lexer Lexer) Lexer

Coalesce is a Lexer interceptor that collapses runs of common types into a single token.

func DelegatingLexer(root Lexer, language Lexer) Lexer

DelegatingLexer combines two lexers to handle the common case of a language embedded inside another, such as PHP inside HTML or PHP inside plain text.

It takes two lexer as arguments: a root lexer and a language lexer. First everything is scanned using the language lexer, which must return "Other" for unrecognised tokens. Then all "Other" tokens are lexed using the root lexer. Finally, these two sets of tokens are merged.

The lexers from the template lexer package use this base lexer.

func RemappingLexer(lexer Lexer, mapper func(Token) []Token) Lexer

RemappingLexer remaps a token to a set of, potentially empty, tokens.

func TypeRemappingLexer(lexer Lexer, mapping TypeMapping) Lexer

TypeRemappingLexer remaps types of tokens coming from a parent Lexer.

eg. Map "defvaralias" tokens of type NameVariable to NameFunction:

mapping := TypeMapping{
	{NameVariable, NameFunction, []string{"defvaralias"},
}
lexer = TypeRemappingLexer(lexer, mapping)

type LexerMutator interface {
	
	
	
	MutateLexer(rules CompiledRules, state string, rule int) error
}

type LexerRegistry struct {
	Lexers Lexers
	
}

func NewLexerRegistry() *LexerRegistry

NewLexerRegistry creates a new LexerRegistry of Lexers.

Aliases of all the lexers, and skip those lexers who do not have any aliases, or show their name instead

Analyse text content and return the "best" lexer..

Get a Lexer by name, alias or file extension.

Match returns the first lexer matching filename.

Note that this iterates over all file patterns in all lexers, so is not fast.

func (l *LexerRegistry) MatchMimeType(mimeType string) Lexer

MatchMimeType attempts to find a lexer for the given MIME type.

Names of all lexers, optionally including aliases.

func (l *LexerRegistry) Register(lexer Lexer) Lexer

Register a Lexer with the LexerRegistry. If the lexer is already registered it will be replaced.

func (l *LexerState) Get(key interface{}) interface{}

Get mutator context.

func (l *LexerState) Iterator() Token

Iterator returns the next Token from the lexer.

func (l *LexerState) Set(key interface{}, value interface{})

Set mutator context.

func (l Lexers) Len() int

func (l Lexers) Swap(i, j int)

type Mutator interface {
	
	Mutate(state *LexerState) error
}

Combined creates a new anonymous state from the given states, and pushes that state.

func Mutators(modifiers ...Mutator) Mutator

Mutators applies a set of Mutators in order.

Pop state from the stack when rule matches.

Push states onto the stack.

type MutatorFunc func(state *LexerState) error

type PrioritisedLexers []Lexer

func (l PrioritisedLexers) Swap(i, j int)

type RegexConfig struct {
	Pattern string  `xml:"pattern,attr"`
	Score   float32 `xml:"score,attr"`
}

type RegexLexer struct {
	
}

func MustNewLexer(config *Config, rules func() Rules) *RegexLexer

MustNewLexer creates a new Lexer with deferred rules generation or panics.

MustNewXMLLexer constructs a new RegexLexer from an XML file or panics.

func NewLexer(config *Config, rulesFunc func() Rules) (*RegexLexer, error)

NewLexer creates a new regex-based Lexer.

"rules" is a state machine transition map. Each key is a state. Values are sets of rules that match input, optionally modify lexer state, and output tokens.

NewXMLLexer creates a new RegexLexer from a serialised RegexLexer.

Unmarshal a RegexLexer from XML.

AnalyseText scores how likely a fragment of text is to match this lexer, between 0.0 and 1.0.

func (r *RegexLexer) Config() *Config

Config returns the Config for this Lexer.

func (r *RegexLexer) MustRules() Rules

MustRules is like Rules() but will panic on error.

SetAnalyser sets the analyser function used to perform content inspection.

func (r *RegexLexer) SetConfig(config *Config) *RegexLexer

SetConfig replaces the Config for this Lexer.

func (r *RegexLexer) SetRegistry(registry *LexerRegistry) Lexer

SetRegistry the lexer will use to lookup other lexers if necessary.

func (r *RegexLexer) SetTracing(trace bool)

SetTracing enables debug tracing.

This complies with the TracingLexer interface.

Tokenise text using lexer, returning an iterator.

type Rule struct {
	Pattern string
	Type    Emitter
	Mutator Mutator
}

func Default(mutators ...Mutator) Rule

Default returns a Rule that applies a set of Mutators.

func (r Rules) Clone() Rules

Clone returns a clone of the Rules.

func (r Rules) Merge(rules Rules) Rules

Merge creates a clone of "r" then merges "rules" into the clone.

func (r Rules) Rename(oldRule, newRule string) Rules

Rename clones rules then a rule.

type SerialisableEmitter interface {
	Emitter
	EmitterKind() string
}

type SerialisableMutator interface {
	Mutator
	MutatorKind() string
}

type Style struct {
	Name string
	
}

func MustNewStyle(name string, entries StyleEntries) *Style

MustNewStyle creates a new style or panics.

MustNewXMLStyle is like NewXMLStyle but panics on error.

NewStyle creates a new style definition.

NewXMLStyle parses an XML style definition.

func (s *Style) Builder() *StyleBuilder

Builder creates a mutable builder from this Style.

The builder can then be safely modified. This is a cheap operation.

func (s *Style) Get(ttype TokenType) StyleEntry

Get a style entry. Will try sub-category or category if an exact match is not found, and finally return the Background.

func (s *Style) Has(ttype TokenType) bool

Has checks if an exact style entry match exists for a token type.

This is distinct from Get() which will merge parent tokens.

func (s *Style) Types() []TokenType

Types that are styled.

type StyleBuilder struct {
	
}

func (s *StyleBuilder) AddAll(entries StyleEntries) *StyleBuilder

func (s *StyleBuilder) AddEntry(ttype TokenType, entry StyleEntry) *StyleBuilder

func (s *StyleBuilder) Get(ttype TokenType) StyleEntry

func (s *StyleBuilder) Transform(transform func(StyleEntry) StyleEntry) *StyleBuilder

Transform passes each style entry currently defined in the builder to the supplied function and saves the returned value. This can be used to adjust a style's colours; see Colour's ClampBrightness function, for example.

type StyleEntry struct {
	
	Colour     Colour
	Background Colour
	Border     Colour

	Bold      Trilean
	Italic    Trilean
	Underline Trilean
	NoInherit bool
}

func MustParseStyleEntry(entry string) StyleEntry

MustParseStyleEntry parses a Pygments style entry or panics.

ParseStyleEntry parses a Pygments style entry.

func (s StyleEntry) Inherit(ancestors ...StyleEntry) StyleEntry

Inherit styles from ancestors.

Ancestors should be provided from oldest to newest.

func (s StyleEntry) Sub(e StyleEntry) StyleEntry

Sub subtracts e from s where elements match.

type Token struct {
	Type  TokenType `json:"type"`
	Value string    `json:"value"`
}

EOF is returned by lexers at the end of input.

Tokenise text using lexer, returning tokens as a slice.

func (t *Token) Clone() Token

Clone returns a clone of the Token.

const (
	
	Background TokenType = -1 - iota
	
	PreWrapper
	
	Line
	
	LineNumbers
	
	LineNumbersTable
	
	LineHighlight
	
	LineTable
	
	LineTableTD
	
	LineLink
	
	CodeLine
	
	Error
	
	Other
	
	None
	
	Ignore
	
	EOFType TokenType = 0
)

Meta token types.

const (
	Keyword TokenType = 1000 + iota
	KeywordConstant
	KeywordDeclaration
	KeywordNamespace
	KeywordPseudo
	KeywordReserved
	KeywordType
)

Keywords.

const (
	Name TokenType = 2000 + iota
	NameAttribute
	NameClass
	NameConstant
	NameDecorator
	NameEntity
	NameException
	NameKeyword
	NameLabel
	NameNamespace
	NameOperator
	NameOther
	NamePseudo
	NameProperty
	NameTag
)

Names.

const (
	NameBuiltin TokenType = 2100 + iota
	NameBuiltinPseudo
)

Builtin names.

const (
	NameVariable TokenType = 2200 + iota
	NameVariableAnonymous
	NameVariableClass
	NameVariableGlobal
	NameVariableInstance
	NameVariableMagic
)

Variable names.

const (
	NameFunction TokenType = 2300 + iota
	NameFunctionMagic
)

Function names.

const (
	Literal TokenType = 3000 + iota
	LiteralDate
	LiteralOther
)

Literals.

const (
	LiteralString TokenType = 3100 + iota
	LiteralStringAffix
	LiteralStringAtom
	LiteralStringBacktick
	LiteralStringBoolean
	LiteralStringChar
	LiteralStringDelimiter
	LiteralStringDoc
	LiteralStringDouble
	LiteralStringEscape
	LiteralStringHeredoc
	LiteralStringInterpol
	LiteralStringName
	LiteralStringOther
	LiteralStringRegex
	LiteralStringSingle
	LiteralStringSymbol
)

Strings.

const (
	LiteralNumber TokenType = 3200 + iota
	LiteralNumberBin
	LiteralNumberFloat
	LiteralNumberHex
	LiteralNumberInteger
	LiteralNumberIntegerLong
	LiteralNumberOct
	LiteralNumberByte
)

Literals.

const (
	Operator TokenType = 4000 + iota
	OperatorWord
)

Operators.

const (
	Generic TokenType = 7000 + iota
	GenericDeleted
	GenericEmph
	GenericError
	GenericHeading
	GenericInserted
	GenericOutput
	GenericPrompt
	GenericStrong
	GenericSubheading
	GenericTraceback
	GenericUnderline
)

Generic tokens.

const (
	Text TokenType = 8000 + iota
	TextWhitespace
	TextSymbol
	TextPunctuation
)

Text.

const (
	Punctuation TokenType = 5000 + iota
)

Punctuation.

TokenTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func TokenTypeValues() []TokenType

TokenTypeValues returns all values of the enum

func (t TokenType) Category() TokenType

func (t TokenType) InCategory(other TokenType) bool

func (t TokenType) InSubCategory(other TokenType) bool

func (i TokenType) IsATokenType() bool

IsATokenType returns "true" if the value is listed in the enum definition. "false" otherwise

MarshalText implements the encoding.TextMarshaler interface for TokenType

func (t TokenType) Parent() TokenType

func (t TokenType) SubCategory() TokenType

UnmarshalText implements the encoding.TextUnmarshaler interface for TokenType

type TokeniseOptions struct {
	
	State string
	
	Nested bool

	
	
	EnsureLF bool
}

type Trace struct {
	Lexer   string  `json:"lexer"`
	State   string  `json:"state"`
	Rule    int     `json:"rule"`
	Pattern string  `json:"pattern"`
	Pos     int     `json:"pos"`
	Length  int     `json:"length"`
	Elapsed float64 `json:"elapsedMs"` 
}

type TracingLexer interface {
	Lexer
	SetTracing(enable bool)
}

const (
	Pass Trilean = iota
	Yes
	No
)

Trilean states.

Prefix returns s with "no" as a prefix if Trilean is no.

type TypeMapping []struct {
	From, To TokenType
	Words    []string
}

type ValidatingEmitter interface {
	Emitter
	ValidateEmitter(rule *CompiledRule) error
}