Great observation!

tl;dr +1, but add an option to disable --no-binary.

The kinds of benchmarks where dependencies matter are typically workload-oriented (macro benchmarks). We want to be able to communicate a performance delta to users relative to how they should expect their Python workload to perform. So with that in mind I'd lean toward avoiding --no-binary if possible. Performance difference between binary and source distributions should be communicated by the relevant projects (e.g. mypy).

That said, I get what you are saying about apples-to-apples comparisons for Python implementations (or distros). We also run into this with running the benchmarks against CPython main: few dependencies are available on PyPI. So I agree we don't want benchmark dependencies to dictate any difference in performance between result sets.

Weighing the two sides, I'm in favor of the latter, which you are proposing. So +1. There is still a use for using pre-built extensions, though, so it would be good for pyperformance to allow us to opt out of --no-binary.