gh-60074: add new stable API function PyType_FromMetaclass by wjakob · Pull Request #93012 · python/cpython
TL;DR: Numerous flagship Python packages internally rely on pybind11 to bridge C++ and Python. This includes SciPy, PyTorch, Tensorflow, JAX, and many others. This PR adds critical functionality needed to eventually move more of them onto the Py_LIMITED_API and thereby simplify deployment of binary wheels.
Context: C++ <-> Python binding tools like pybind11 and nanobind require the ability to instantiate types using a custom metaclass. pybind11 uses them to install custom handling for some type-related operations, and nanobind goes even further by storing binding-related data structures in an enlarged PyHeapTypeObject allocated by a metaclass.
Unfortunately is not currently possible to dynamically create suitable heap types using the Py_LIMITED_API, which means that each extension library using these tools must be compiled many times for redistribution. I believe that it would be useful to the Python community to at least optionally support Py_LIMITED_API in these tools, but this requires a small change in CPython.
This pull request is another attempt at addressing the issue pointed out in issue #60074. It adds a new stable API function PyType_FromMetaclass that mirrors the behavior of PyType_FromModuleAndSpec except that it takes an additional metaclass argument.
I used this PR to create a proof-of-concept version of nanobind (limited_api branch) that dynamically create types using the limited API. It works 🎉 .
I have a somewhat audacious request, which is that this change is considered for Python 3.11 despite having recently entered feature freeze. I believe that this change is small enough and of significant utility for the wider Python community (potentially even for other groups working on bindings like SWIG or Cython)