TST: instability with setuptools 71

Opening this for visibility: we're affected by an upstream issue in setuptools 71: pypa/setuptools#4480
example logs

Note that it seems related but I'm not positive it's the exact same bug: between CI and local runs I'm actually seeing different failure modes, but they all seem related to this one bug so I don't think it's worth the effort of disentangling everything yet (hopefully the situation will be much clearer after another prompt bugfix release)