One test of test_site fails on an android emulator running an x86 system image at API level 21.

See the attached test_output.txt file.

The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set, hence in the _bootlocale module, the statement '_locale.CODESET' raises AttributeError and the locale module is imported upon interpreter startup. The locale module imports re.

See issue #19205 for why we would rather not import re and locale on startup.

This seems difficult to fix without either skipping most part of the test as it is done with Mac OS X, or having a specific sys.platform for android to handle the AttributeError in _bootlocale by having the getpreferredencoding() fuction returning 'UTF-8' ('UTF-8' is the file system encoding on android).

BTW the test runs fine on android when the AttributeError in _bootlocale is hard-coded with a getpreferredencoding() fuction returning 'UTF-8' and not importing locale.

Sorry for the confusion, the file system encoding is not the locale encoding.

In issue #9548, Antoine proposed a patch that avoids the import of the re, collections and functools modules by the _io module on startup, by refactoring and moving code from locale to _bootlocale. The attached refactor_locale.patch does that for python 3.6. The reasons for why Antoine patch has not been pushed still apply to this patch :(

The patch does fix the problem for Android though.

An improvement to Python startup time on Android (Android does not have nl_langinfo()) is to have _bootlocale.getpreferredencoding() return 'ascii' without importing locale, when none of the locale environment variables is set. With patch no-locale-envvar.patch, test_site runs ok on android-21-x86 emulator when the locale environment variables are not set.

Committing this patch while leaving the current issue open would also allow removing this issue from the dependencies of the Android meta-issue #26865.

This patch fixes test_startup_imports when the platform does not have langinfo.h.

Entered new issue 28596: "on Android _bootlocale on startup relies on too many library modules".

Patch that follows closely the conditionals in the __bootlocale module.

> The problem is caused by the fact that android does not have HAVE_LANGINFO_H and CODESET set

Hum, is it possible to get the locale encoding by another way?

If not, what is the locale encoding?

Does Android provide mbstowcs() and wcstombs() functions?

Seems Android/BioniC always uses UTF-8: https://android.googlesource.com/platform/bionic/+/master/libc/bionic/mbrtoc32.cpp#83

If it is not possible to change the locale, it makes sense to hardcode utf8.

Note: to avoid mojibake, it's better if sys.getfilesystemencoding() and
locale.getpreferredencoding(False) are equal. I understand that both must
be utf8.

There are some locale strings supported in setlocale(): https://android.googlesource.com/platform/bionic/+/master/libc/bionic/locale.cpp#104. However, seems mbstowcs just ignores such a setting on Android. Here's an example:

#include <locale.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>

#define BUFFER_SIZE 10

void test_mbstowcs()
{
    wchar_t dest[BUFFER_SIZE];
    memset(dest, 0, sizeof(dest));
    printf("mbstowcs: %ld\n", mbstowcs(dest, "中文", BUFFER_SIZE));
    printf("dest: %x %x\n", dest[0], dest[1]);
}

int main()
{
    printf("setlocale: %d\n",  setlocale(LC_ALL, "en_US.UTF-8") != NULL);
    test_mbstowcs();
    printf("setlocale: %d\n",  setlocale(LC_ALL, "C") != NULL);
    test_mbstowcs();
    return 0;
}

On Linux (glibc 2.24) the result is:

$ ./a.out 
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587
setlocale: 1
mbstowcs: -1
dest: 0 0

On Android (6.0 Marshmallow) the result is:
shell@ASUS_Z00E_2:/ $ /data/local/tmp/a.out
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587
setlocale: 1
mbstowcs: 2
dest: 4e2d 6587

A quick search indicates setlocale() affects *scanf functions only, so I guess it's safe to force UTF-8 in CPython.

In Python, the most important functions are Py_DecodeLocale() and
Py_EncodeLocale() which use mbstowcs() and wvstombs().

Submitted a patch to issue28596

Closing as invalid, it is useful to have the test failing on platforms that do not have CODESET and detect that too many modules are imported on startup. For Android, this problem is fixed in issue 28596.

stage: commit review -> resolved

messages: + msg281090
stage: patch review -> commit review

assignee: xdegaye
components: + Tests, - Library (Lib), Cross-Build
stage: patch review

messages: + msg266008

title: android: test_site fails -> _bootlocale imports locale at startup on Android