Issue 35968: lib2to3 cannot parse rf''

Issue35968

Created on 2019-02-11 16:39 by majuscule, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (4)
msg335238 - (view) Author: Dylan Lloyd (majuscule) Date: 2019-02-11 16:39
```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

regex_format = rf''
```

```
yapf poc.py
```

```
Traceback (most recent call last):
  File "lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="''", context=('', (4, 17))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "lib/python3.6/site-packages/yapf/__init__.py", line 326, in run_main
    sys.exit(main(sys.argv))
  File "lib/python3.6/site-packages/yapf/__init__.py", line 213, in main
    verbose=args.verbose)
  File "lib/python3.6/site-packages/yapf/__init__.py", line 263, in FormatFiles
    in_place, print_diff, verify, verbose)
  File "lib/python3.6/site-packages/yapf/__init__.py", line 289, in _FormatFile
    logger=logging.warning)
  File "lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 91, in FormatFile
    verify=verify)
  File "lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="''", context=('', (4, 17))
```
msg335240 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-02-11 17:02
Seems related : issue33266
msg335241 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-02-11 17:18
The reported example works fine on master and 3.7 . The related issue's fix was not backported to 3.6 . 3.6 is in security fixes only mode. So upgrading to 3.7 will help and I propose closing this as a duplicate of issue33266 .

➜  cpython git:(master) ✗ cat foo.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

regex_format = rf''
➜  cpython git:(master) ✗ ./python.exe ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to foo.py
RefactoringTool: Files that need to be modified:
RefactoringTool: foo.py
➜  cpython git:(master) ✗ python3.7 ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to foo.py
RefactoringTool: Files that need to be modified:
RefactoringTool: foo.py
➜  cpython git:(master) ✗ python3.6 ./Tools/scripts/2to3 foo.py
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't parse foo.py: ParseError: bad input: type=3, value="''", context=('', (4, 17))
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't parse foo.py: ParseError: bad input: type=3, value="''", context=('', (4, 17))
msg335250 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-02-11 18:41
@xtreak, I concur.  While it would have better to originally fix this in the release where f strings first appeared, the window to do that has closed.  Sorry!
History
Date User Action Args
2022-04-11 14:59:11adminsetgithub: 80149
2019-02-11 18:41:07ned.deilysetstatus: open -> closed

superseder: 2to3 doesn't parse all valid string literals

nosy: + ned.deily
messages: + msg335250
resolution: duplicate
stage: resolved

2019-02-11 17:18:39xtreaksetnosy: + benjamin.peterson
messages: + msg335241
2019-02-11 17:02:42xtreaksetnosy: + xtreak
messages: + msg335240
2019-02-11 16:39:57majusculecreate