Expand Up @@ -21,11 +21,24 @@ CPython's startup behaviour when creating an alternate executable or embedding it as a Python execution engine inside a larger application.
When implementation of this proposal is completed, interpreter startup will consist of two clearly distinct and independently configurable phases: consist of three clearly distinct and independently configurable phases:
* Python core runtime preconfiguration
* setting up memory management * determining the encodings used for system interfaces (including settings passed in for later configuration phase)
* Python core runtime initialization
* ensuring C API is ready for use * ensuring builtin and frozen modules are accessible
* Python runtime initialization * Main interpreter configuration
* ensuring external modules are accessible * (Note: the name of this phase is quite likely to change)
Changes are also proposed that impact main module execution and subinterpreter initialization.
Expand All @@ -38,11 +51,16 @@ Proposal ========
This PEP proposes that initialization of the CPython runtime be split into two clearly distinct phases: three clearly distinct phases:
* core runtime preconfiguration * core runtime initialization * main interpreter configuration
(Earlier versions proposed only two phases, but experience with attempting to implement the PEP as an internal CPython refactoring showed that at least 3 phases are needed to get clear separation of concerns)
The proposed design also has significant implications for:
* main module execution Expand Down Expand Up @@ -157,14 +175,14 @@ dominated by IO operations. However, to monitor the impact of any changes, a simple benchmark can be used to check how long it takes to start and then tear down the interpreter::
python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" python3 -m timeit -s "from subprocess import call" "call(['./python', '-Sc', 'pass'])"
Current numbers on my system for Python 3.7 (using the 3.5 subprocess and timeit modules to execute the check, all with non-debug builds):: Current numbers on my system for Python 3.7 (as built by the Fedora project)::
$ python3 -m timeit -s "from subprocess import call" "call(['./python', '-c', 'pass'])" 100 loops, best of 3: 13.9 msec per loop $ python3 -m timeit -s "from subprocess import call" "call(['python3', '-Sc', 'pass'])" 50 loops, best of 5: 6.48 msec per loop
(TODO: run this microbenchmark with perf rather than the stdlib timeit)
This PEP is not expected to have any significant effect on the startup time, as it is aimed primarily at *reordering* the existing initialization Expand Down Expand Up @@ -278,47 +296,37 @@ implementation of the import system in Python 3.3, there is no clear way to structure a draft implementation that won't be prone to the kinds of merge conflicts that afflicted the original attempt.
Accordingly, the implementation strategy now being proposed is to implement this refactoring as a private API for CPython 3.7, before exposing the new functions and structures as public API elements in CPython 3.8.
The affected APIs with a leading underscore, as they would be named in CPython 3.7:
* ``_Py_IsRuntimeInitialized`` * ``_Py_InitializeRuntime`` * ``_PyRuntimeConfig`` * ``_PyRuntimeConfig_INIT`` * ``_Py_ReadHashSeed`` * ``_Py_ConfigureMainInterpreter`` * ``_PyMainInterpreterConfig`` * ``_PyInterpreterConfig`` * ``_Py_ReadMainInterpreterConfig`` * ``_PyRun_PrepareMain`` * ``_PyRun_ExecMain`` * ``_Py_InterpreterState_Main``
New APIs described in the rest of the PEP with a leading underscore are intended to be retained permanently as private CPython implementation details.
The principle benefit of this approach is allowing the refactoring to adopt the new configuration structures to be handled on a setting by setting basis over the course of the 3.7 and 3.8 development cycles, rather than having to migrate them in a single monolithic change. It also means any new settings introduced for these releases can be handled using the new approach from the start, even if some existing settings have yet to be migrated.
If all existing settings are successfully migrated to the new initialization model in time for the 3.7.0a4 release in December 2017, then the proposal would be to make the APIs public for the 3.7.0b1 release in January 2018, rather than waiting for 3.8. Accordingly, the implementation strategy was revised to instead first implement this refactoring as a private API for CPython 3.7, and then review the viability of exposing the new functions and structures as public API elements in CPython 3.8.
After the initial merge, Victor Stinner then proceeded to actually migrate settings to the new structure in order to successfully implement the PEP 540 UTF-8 mode changes (which required the ability to track all settings that had previously been decoded with the locale encoding, and decode them again using UTF-8 instead). Eric Snow also migrated a number of internal subsystems over as part of making the subinterpreter feature more robust.
That work showed that the detailed design currently proposed in this PEP has a range of practical issues, so it's currently expected to remain a private API for CPython 3.8, with the possibility of making it public and stable in CPython 3.9.

Design Details ==============
(Note: details here are still very much in flux, but preliminary feedback is appreciated anyway) .. note::
The API details here are still very much in flux, as the private refactoring work has shown that these specific API designs aren't really viable in practice. The header files that show the current state of the private API are mainly:
* https://github.com/python/cpython/blob/master/Include/cpython/coreconfig.h * https://github.com/python/cpython/blob/master/Include/cpython/pystate.h * https://github.com/python/cpython/blob/master/Include/cpython/pylifecycle.h
The main theme of this proposal is to initialize the core language runtime and create a partially initialized interpreter state for the main interpreter Expand Down Expand Up @@ -1035,37 +1043,20 @@ Open Questions Implementation ==============
A reference implementation is available as a feature branch in Nick Coghlan's CPython sandbox on BitBucket [2_].
It implements the ``_Py_InitializeRuntime`` and ``_Py_ConfigureMainInterpreter`` aspects of the refactoring as a private API. All ``_PyRuntimeConfig`` settings are included, but currently only the "install signal handlers" setting is implemented for the main interpreter configuration. The reference implementation is being developed as a private API refactoring within the CPython reference interpreter (as attempting to maintain it as an independent project proved impractical).
The feature branch is a couple of iterations behind the API design in the PEP so the exact interfaces currently available are:
* ``_Py_IsCoreInitialized`` (rename pending: ``_Py_IsRuntimeInitialized``) * ``_Py_InitializeCore`` (rename pending: ``_Py_InitializeRuntime``) * ``_PyCoreConfig`` (rename pending: ``_Py_RuntimeConfig``) * ``_PyCoreConfig_INIT`` (rename pending: ``_Py_RuntimeConfig_INIT``) * ``_Py_ReadHashSeed`` * ``_Py_InitializeMainInterpreter`` (rename pending: ``_Py_ConfigureMainInterpreter``) * ``_PyMainInterpreterConfig`` * ``_PyMainInterpreterConfig_INIT`` * ``_Py_ReadMainInterpreterConfig``

The Status Quo ============== The Status Quo (as of Python 3.6) =================================
The current mechanisms for configuring the interpreter have accumulated in a fairly ad hoc fashion over the past 20+ years, leading to a rather inconsistent interface with varying levels of documentation.
(Note: some of the info below could probably be cleaned up and added to the C API documentation for 3.6 - it's all CPython specific, so it C API documentation for 3.x - it's all CPython specific, so it doesn't belong in the language reference)

Expand Down