Python 3.6 doesn't use the right encoding in os.strerror(), time.stftime(), locale.localeconv(), time.tzname, etc. on macOS, FreeBSD and other platforms.

See my fix locale encodings in bpo-29240: commit 7ed7aead9503102d2ed316175f198104e0cd674c, and test_all_locales.py attached to bpo-29240.

See also the bpo-31900 for locale.localeconv() encoding issue when LC_NUMERIC encoding is different than the LC_CTYPE encoding.

I'm not sure that locale.bindtextdomain() uses the right encoding neither. I propose the following fix:

diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
index 324b694b83..1de17d3620 100644
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -555,7 +555,7 @@ PyIntl_bindtextdomain(PyObject* self,PyObject*args)
         PyErr_SetFromErrno(PyExc_OSError);
         return NULL;
     }
-    result = PyUnicode_DecodeLocale(current_dirname, NULL);
+    result = PyUnicode_DecodeFSDefault(current_dirname);
     Py_XDECREF(dirname_bytes);
     return result;
 }

Another issue: _Py_DecodeUTF8Ex() creates surrogate pairs with 16-bit wchar_t (on Windows), whereas input bytes should be escaped. I'm quite sure that it's a bug.

Example of bug on FreeBSD 11:

haypo@freebsd$ LC_ALL=C ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, "fr_FR.ISO8859-1"); print(ascii(os.strerror(2)))'

'Fichier ou r\udce9pertoire inexistant'

Expected result:

haypo@freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))'

'Fichier ou r\xe9pertoire inexistant'

New changeset b92c159efada05b3a5ff9d0dbce3fcb2334631f6 by Victor Stinner in branch '3.6':
[3.6] bpo-32555: Fix locale encodings (#5193)
https://github.com/python/cpython/commit/b92c159efada05b3a5ff9d0dbce3fcb2334631f6

freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))'
'Fichier ou r\xe9pertoire inexistant'

I ran manually this test on FreeBSD: it pass on Python 3.6, 3.7 and master. I close the issue.


> Another issue: _Py_DecodeUTF8Ex() creates surrogate pairs with 16-bit wchar_t (on Windows), whereas input bytes should be escaped. I'm quite sure that it's a bug.

I created bpo-33928 for that one.

stage: patch review -> resolved