While running the 2to3 script on the scons codebase, I ran into an
UnicodeDecodeError.

Attached is just the portion of the script that causes the error.

2to3 throws an error on the string regardless of whether the unicode
string literal is prepended on the front.

RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: ws_comma
Traceback (most recent call last):
  File "/usr/local/bin/2to3", line 5, in <module>
    sys.exit(refactor.main())
  File "/usr/local/lib/python3.0/lib2to3/refactor.py", line 81, in main
    rt.refactor_args(args)
  File "/usr/local/lib/python3.0/lib2to3/refactor.py", line 188, in
refactor_args
    self.refactor_file(arg)
  File "/usr/local/lib/python3.0/lib2to3/refactor.py", line 217, in
refactor_file
    input = f.read() + "\n" # Silence certain parse errors
  File "/usr/local/lib/python3.0/io.py", line 1611, in read
    decoder.decode(self.buffer.read(), final=True))
  File "/usr/local/lib/python3.0/io.py", line 1199, in decode
    output = self.decoder.decode(input, final=final)
  File "/usr/local/lib/python3.0/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 59-60:
invalid data

2to3 running under Python 2.5.1 handles this file just fine. 2to3
running under 3.0a4+ (r62404) fails as detailed below. However, that
file doesn't run correctly under Python itself:

collinwinter@Silves:~/src/python/py3k$ ./python
/home/collinwinter/Desktop/2to3bug.py 
  File "/home/collinwinter/Desktop/2to3bug.py", line 3
    collinwinter@Silves:~/src/python/py3k

This suggests this problem isn't 2to3-specific. Refiling this issue
against py3k's Unicode support.

Someone on the #python IRC channel suggested that the default for python
3.0 for unicode string literals is reversed from python 2.5.

If you remove the unicode string literal (u'') from the front of the
string, it runs fine under python 3.0 and fails under 2.5 and 2.6 instead.

Also, I can confirm that running 2to3 with Python 2.6 correctly converts
the script but running 2to3 with Python 3.0 results in a
UnicodeDecodeError exception.

Confirmed in py3k on rev71995.

The problem is that 2to3 just reads the file with whatever
locale.getpreferredencoding() returns. It should use
tokenize.detect_encoding() to discover the correct encoding to open it with.

Patch using tokenize.detect_encoding() to read the encoding of Python 
scripts instead of using default io.open() encoding (utf-8).

We might write unit test.

See also related issue: #5093

Fixed in r72491.

nosy: + vstinner
messages: + msg87175

keywords: + patch

messages: + msg86641
stage: test needed