At Anatoly's prompting, Andrey Ponomarenko set up ABI/API compatibility checks for CPython on the Linux upstream tracker (http://upstream-tracker.org/)

Everything: http://upstream-tracker.org/versions/python.html
Public API/ABI (no leading underscores): http://upstream-tracker.org/versions/python_public_api.html
3.2 limited API/stable ABI (no leading underscores): http://upstream-tracker.org/versions/python_stable_api.html

Andrey's analysis in the python-ideas thread [1] indicates we've likely made some mistakes in exposing struct internals, as well as exposing new APIs unintentionally under older Py_LIMITED_API definitions.

There's probably not too much (if anything) we can do to rectify the past mistakes, but we could set up a daily or weekly check (akin to the existing refleak scan) that monitored for such mistakes to prevent the introduction of any new errors along these lines.

[1] https://mail.python.org/pipermail/python-dev/2014-April/133754.html

There are just three simple steps to setup ABI checker:

1. Create ABI dump of the _reference_ version (see instructions [1]) and commit it to the source tree (ABI-ref.dump):

$> abi-compliance-checker -l python -dump descriptor.xml -dump-path ABI-ref.dump

2. Create ABI dump of the _current_ version (ABI-cur.dump)
3. Compare these ABI dumps:

$> abi-compliance-checker -l python -old ABI-ref.dump -new ABI-cur.dump -report-path abi-report.html

To skip analysis of symbols with leading underscore pass this additional option to the tool when comparing ABI dumps: --skip-internal="\A_"

For more information see [2].

[1] https://mail.python.org/pipermail/python-dev/2014-April/133754.html
[2] http://ispras.linuxbase.org/index.php/ABI_compliance_checker

Updated link to the ABI tracker report: https://abi-laboratory.pro/tracker/timeline/python/

(The stable ABI report still has a separate entry on http://upstream.rosalinux.ru/ but it links to the same report as I've linked above)

I'm bumping this based on the 3.7.0 stable ABI break that Christian Tismer reported for PyIndex_Check - it's clear that we *need* some form of automated testing for the stable ABI, or we're going to continue to see issues like this crop up at release time.

(Folks freshly added to the nosy list: I added you either as release managers for 3.7+, or else because we've previously talked about the stable ABI and its maintainability)

FTR, the issue of ABI breakage has come up quite a few times on the mailing list.  Most recently:

  https://mail.python.org/pipermail/python-dev/2018-June/153775.html

That email thread contains multiple references to scripts that may be used to identify ABI issues, among other good points.  FWIW, when consolidating the C globals last year I added a tool to track the remaining globals:

  Tools/c-globals

It is similar (at least in intent) to any solution we might pursue relative to the stable ABI.

Tools/c-globals moved under Tools/c-analyzer