Here is preliminary patch that refactors the lowest level of Python tokenizer, reading and decoding. It splits the code on smaller simpler functions, decreases the source size by 37 lines, and fixes bugs: issue14811, issue18961, and a number of others. Added tests for most of fixed bugs (except leaks and others hardly reproducible). But the fix for other bugs can be harder, especially for issues with null byte (issue1105770, issue20115).

Many bug easily can be fixed if read all Python file in memory instead of reading it line by line. I don't know if it is acceptable.

Hi Serhiy,

Just of your information but I think you know that, the tests pass ;-)

[398/399] test_multiprocessing_spawn (138 sec) -- running: test_tools
(108 sec)
[399/399] test_tools (121 sec)
385 tests OK.
3 tests altered the execution environment:
    test___all__ test_site test_warnings
11 tests skipped:
    test_devpoll test_kqueue test_msilib test_ossaudiodev
    test_startfile test_tix test_tk test_ttk_guionly test_winreg
    test_winsound test_zipfile64

But I am interested by this part of CPython, I am not an expert in
lexing and parsing but how can I help you ? I am a novice in this
domain.

Stephane

"especially for issues with null byte"

I don't think that we should put to much energy in handling correctly NUL bytes. I see NUL bytes in code as bugs in the code, not in the Python parser. We *might* try to give warnings or better error messages to the user, that's all.

New changeset 23a7481eafd4 by Serhiy Storchaka in branch 'default':
Issues #25643, #26581: Added new tests for detecting Python source code encoding.
https://hg.python.org/cpython/rev/23a7481eafd4

@serhiy: did you still want to commit this?

New changeset 261a452a1300eeeae1428ffd6e6623329c085e2c by Pablo Galindo in branch 'master':
bpo-25643: Refactor the C tokenizer into smaller, logical units (GH-25050)
https://github.com/python/cpython/commit/261a452a1300eeeae1428ffd6e6623329c085e2c

Oh, 6 years to fix this bug. Better late than never ;-) Thanks for reporting and for fixing it!

pull_requests: + pull_request23799
stage: patch review