Home Original page

Issue 19020: Regression: Windows-tkinter-idle, unicode, and 0xxx filename

Current 3.3.2+ repository build (32 bit) and 3.4.0a2 repository build (32 bit) and installation (64 bit) have a problem that did not exist in my Win6 3.3.2 installation (64 bit). (Bruce Sherwood discovered the symption with some installed version of 3.4.0a?) Adding to recent files a path with a directory or file whose name begins with '0' causes Idle to stop with a traceback such as given below. Currently, opening *any* file causes the recent files list to be rebuilt (seen in traceback). So the problem happens whether one opens a new file with a leading 0 somewhere or if there merely is one such already on the list.

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Python34\lib\tkinter\__init__.py", line 1475, in __call__
    return self.func(*args)
  File "C:\Python34\lib\idlelib\IOBinding.py", line 179, in open
    flist.open(filename, self.loadfile)
  File "C:\Python34\lib\idlelib\FileList.py", line 34, in open
    return action(filename)
  File "C:\Python34\lib\idlelib\IOBinding.py", line 240, in loadfile
    self.updaterecentfileslist(filename)
  File "C:\Python34\lib\idlelib\IOBinding.py", line 521, in updaterecentfileslist
    self.editwin.update_recent_files_list(filename)
  File "C:\Python34\lib\idlelib\EditorWindow.py", line 915, in update_recent_files_list
    menu.delete(0, END)  # clear, and rebuild:
  File "C:\Python34\lib\tkinter\__init__.py", line 2739, in delete
    if 'command' in self.entryconfig(i):
  File "C:\Python34\lib\tkinter\__init__.py", line 2749, in entryconfigure
    return self._configure(('entryconfigure', index), cnf, kw)
  File "C:\Python34\lib\tkinter\__init__.py", line 1247, in _configure
    self.tk.call(_flatten((self._w, cmd)))):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 14: invalid start byte

'invalid start byte' 0xc0 has appeared before in other issues.

Bruce got this with
  H:\HP_Documents\0PythonWork\AirplaneKinematics\accel2.py
I get 12' instead of '14' with
  F:\Python\mypy\0...' (file or directory, must be 0, not 1)
It appears that the string is being improperly 're-cooked' at some point, so that '\x' sequences are 1 position rather than 2.

At the point of failure, I believe tk is checking whether a particular menuitem has an associated callback that should be deleted before the menuitem itself is deleted (before later being rebuilt). (It is a separate issue of whether we could just adjust the recent file list (sub sub menu) rather than deleting and rebuilding it, possibly the same as it was. Even if we did, the issue would still arise when opening a new '0' file, or when closing Idle.)
Tests added in issue19085 have a special case for tuple values because widget[name] and widget.configure(name) return different results in such cases. When remove this special case, following tests fails:

======================================================================
FAIL: test_text (tkinter.test.test_ttk.test_widgets.ButtonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 381, in test_text
    self.checkParams(widget, 'text', '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_offvalue (tkinter.test.test_ttk.test_widgets.CheckbuttonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/test_ttk/test_widgets.py", line 248, in test_offvalue
    self.checkParams(widget, 'offvalue', 1, 2.3, '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_onvalue (tkinter.test.test_ttk.test_widgets.CheckbuttonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/test_ttk/test_widgets.py", line 252, in test_onvalue
    self.checkParams(widget, 'onvalue', 1, 2.3, '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_text (tkinter.test.test_ttk.test_widgets.CheckbuttonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 381, in test_text
    self.checkParams(widget, 'text', '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_values (tkinter.test.test_ttk.test_widgets.ComboboxTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/test_ttk/test_widgets.py", line 363, in test_values
    self.checkParam(self.combo, 'values', (42, 3.14, '', 'any string'))
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: Tuples differ: (42, 3.14, '', ('any', 'string')) != (42, 3.14, '', 'any string')

First differing element 3:
('any', 'string')
any string

- (42, 3.14, '', ('any', 'string'))
?                -    ^^^^        -

+ (42, 3.14, '', 'any string')
?                    ^


======================================================================
FAIL: test_text (tkinter.test.test_ttk.test_widgets.LabelFrameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 381, in test_text
    self.checkParams(widget, 'text', '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_text (tkinter.test.test_ttk.test_widgets.LabelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 381, in test_text
    self.checkParams(widget, 'text', '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_text (tkinter.test.test_ttk.test_widgets.RadiobuttonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 381, in test_text
    self.checkParams(widget, 'text', '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

======================================================================
FAIL: test_value (tkinter.test.test_ttk.test_widgets.RadiobuttonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tkinter/test/test_ttk/test_widgets.py", line 701, in test_value
    self.checkParams(widget, 'value', 1, 2.3, '', 'any string')
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 89, in checkParams
    self.checkParam(widget, name, value, **kwargs)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 63, in checkParam
    self.assertEqual2(t[4], expected, eq=eq)
  File "/home/serhiy/py/cpython/Lib/tkinter/test/widget_tests.py", line 41, in assertEqual2
    self.assertEqual(actual, expected, msg)
AssertionError: ('any', 'string') != 'any string'

With the tkinter_configure_splitlist.patch patch they are passed again.
> I am also not clear on the relation between the UnicodeDecodeError and tuple splitting. Does '_flatten((self._w, cmd)))' call split or splitlist on the tuple arg? Is so, do you know why a problem with that would lead to the UDError? Does your patch fix the leading '0' regression?

The traceback is misleading. Full statement is:

            for x in self.tk.split(
                    self.tk.call(_flatten((self._w, cmd)))):

Where cmd is ('entryconfigure', index). The UnicodeDecodeError error was raised neither by _flatten() nor call(), but by split().

When run `./python -m idlelib.idle \\0.py` call() returns and split() gets a tuple of tuples: (('-activebackground', '', '', '', ''), ('-activeforeground', '', '', '', ''), ('-accelerator', '', '', '', ''), ('-background', '', '', '', ''), ('-bitmap', '', '', '', ''), ('-columnbreak', '', '', 0, 0), ('-command', '', '', '', '3067328620open_recent_file'), ('-compound', 'compound', 'Compound', <index object: 'none'>, 'none'), ('-font', '', '', '', ''), ('-foreground', '', '', '', ''), ('-hidemargin', '', '', 0, 0), ('-image', '', '', '', ''), ('-label', '', '', '', '1 /home/serhiy/py/cpython/\\0.py'), ('-state', '', '', <index object: 'normal'>, 'normal'), ('-underline', '', '', -1, 0)). When set wantobjects in Lib/tkinter/__init__.py to 0, it will get a string r"{-activebackground {} {} {} {}} {-activeforeground {} {} {} {}} {-accelerator {} {} {} {}} {-background {} {} {} {}} {-bitmap {} {} {} {}} {-columnbreak {} {} 0 0} {-command {} {} {} 3067013228open_recent_file} {-compound compound Compound none none} {-font {} {} {} {}} {-foreground {} {} {} {}} {-hidemargin {} {} 0 0} {-image {} {} {} {}} {-label {} {} {} {1 /home/serhiy/py/cpython/\0.py}} {-state {} {} normal normal} {-underline {} {} -1 0}".  Then split() try recursively split its argument. When it splits '1 /home/serhiy/py/cpython/\\0.py' it interprets '\\0' as backslash substitution of octal code 0 which means a character with code 0. Tcl uses modified UTF-8 encoding in which null code is encoded as b'\xC0\x80'. This bytes sequence is invalid UTF-8. That is why UnicodeDecodeError was raised (patch for issue13153 handles b'\xC0\x80' more correctly). When you will try '\101.py', it will be translated by split() to 'A.py'.