I don't think it's installing into the same venv though. I was under the assumption that it's installing into separate venvs.

For each benchmark, it tries to install in the "common" venv, and if that fails (usually due to a version conflict) it creates a new benchmark-specific venv.

When I have it build for all of pyperformance + python_macrobenchmarks, I get the following under the venv directory (meaning it created 2 venvs, 1 common and 1 just for pytorch_alexnet_inference):

cpython3.14-f90d3bc7a4bc-compat-cb6a104a6688
cpython3.14-f90d3bc7a4bc-compat-cb6a104a6688-bm-pytorch_alexnet_inference

How many do you get?

Oh no. Can you please give the version of Python you're using and your OS? For context, a lot of things don't fit into the common venv for me.

I'm using Python 3.11 as the "driver" of pyperformance, but benchmarking with CPython main. I see the same timings across all of the OS's -- Windows about 6 minutes rather than 3, but that's not surprising on something I/O bound. These are the OS details:

linux: Intel® Xeon® W-2255 CPU @ 3.70GHz, running Ubuntu 20.04 LTS, gcc 9.4.0
linux2: 12th Gen Intel® Core™ i9-12900 @ 2.40 GHz, running Ubuntu 22.04 LTS, gcc 11.3.0
linux-aarch64: ARM Neoverse N1, running Ubuntu 22.04 LTS, gcc 11.4.0
macos: M1 arm64 Mac® Mini, running macOS 13.2.1, clang 1400.0.29.202
windows: 12th Gen Intel® Core™ i9-12900 @ 2.40 GHz, running Windows 11 Pro (21H2, 22000.1696), MSVC v143