bpo-1635741: Port _collections extension module to multiphase initialization(PEP 489) by shihai1991 · Pull Request #18066 · python/cpython

The key benefits that multi-phase initialisation brings to a relatively straightforward extension module like collections is that it gains support for true reloading, and ceases to be a backchannel for potential communication between subinterpreters. The latter aspect is one of the pre-requisites for the long term goal of potentially moving away from sharing the GIL between subinterpreters (no shared GIL -> no shared refcounts -> no shared Python objects).

If we look at one of the types that collections defines, we see that we normally can't persuade the interpreter to re-create it, even if we're accessing the module from a subinterpreter:

>>> import collections
>>> print(hex(id(collections.deque)))
0x751a00
>>> del sys.modules["collections"]
>>> import collections
>>> print(hex(id(collections.deque)))
0x751a00
>>> from importlib import reload
>>> reload(collections)
<module 'collections' from '/home/ncoghlan/devel/cpython/Lib/collections/__init__.py'>
>>> print(hex(id(collections.deque)))
0x751a00
>>> import _xxsubinterpreters as interp
>>> si = interp.create()
>>> interp.run_string(si, "import collections; print(hex(id(collections.deque)))")
0x751a00

Even trashing the module to the point where reload doesn't work any more, still doesn't clear the hidden module state cache (the object ID only changed from above because this is a different REPL session):

>>> import collections
>>> from importlib import reload
>>> print(hex(id(collections.deque)))
0x7fc37355d080
>>> collections.__dict__.clear()
>>> vars(collections)
{}
>>> reload(collections)
Traceback (most recent call last):
 ...
AttributeError: module has no attribute '__name__'
>>> import sys
>>> del sys.modules["collections"]
>>> import collections
>>> print(hex(id(collections.deque)))
0x7fc37355d080

By contrast, once it has migrated to multi-phase initialisation, then reloading it (or loading it in a subinterpreter) will actually recreate the contents, not just the containing dictionary and module metadata.

(cc @ericsnowcurrently @encukou in case I missed something above).