Home Original page

[Python-Dev] Support of UTF-16 and UTF-32 source encodings

Victor Stinner victor.stinner at gmail.com
Sat Nov 14 17:56:43 EST 2015 
These encodings are rarely used. I don't think that any text editor use
them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
space.

Ok, even if it exists, Python already accepts a very wide range of
encoding. It is not worth to make the parser much more complex just to
support encodings which are also never used (for .py files).

Victor
Le 14 nov. 2015 20:20, "Serhiy Storchaka" <storchaka at gmail.com> a écrit :

> For now UTF-16 and UTF-32 source encodings are not supported. There is a
> comment in Parser/tokenizer.c:
>
>     /* Disable support for UTF-16 BOMs until a decision
>        is made whether this needs to be supported.  */
>
> Can we make a decision whether this support will be added in foreseeable
> future (say in near 10 years), or no?
>
> Removing commented out and related code will help to refactor the
> tokenizer, and that can help to fix some existing bugs (e.g. issue14811,
> issue18961, issue20115 and may be others). Current tokenizing code is too
> tangled.
>
> If the support of UTF-16 and UTF-32 is planned, I'll take this to
> attention during refactoring. But in many places besides the tokenizer the
> ASCII compatible encoding of source files is expected.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151114/ac0a9077/attachment.html>
More information about the Python-Dev mailing list