bpo-25324: Move the description of tokenize tokens to token.rst. by serhiy-storchaka · Pull Request #1911

bpo-25324: Move the description of tokenize tokens to token.rst. by serhiy-storchaka · Pull Request #1911 · python/cpython

Merged

Jun 6, 2017

Merged

bpo-25324: Move the description of tokenize tokens to token.rst.#1911
serhiy-storchaka merged 2 commits intopython:masterfrom
serhiy-storchaka:token-docs

Copy link Copy Markdown

Member

Also tweaks the formatting.

Documentation in the Doc dir

label

Copy link Copy Markdown

@serhiy-storchaka, thanks for your PR! By analyzing the history of the files in this pull request, we identified @birkenfeld, @benjaminp and @vadmium to be potential reviewers.

vadmium reviewed Jun 3, 2017

Comment thread

		Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` to bring
		the tokens in the C code in line with the tokens needed in
		:mod:`tokenize` module. These tokens aren't used by the C tokenizer. No newline at end of file
		Following token type values aren't used by the C tokenizer but are needed for