> Ok, I ran a subset of the benchmarks to record their memory footprint and got these results:

I'm not sure that the code tracking the memory usage in performance works :-) It may be worth to double check the code. By the way, perf has a --tracemalloc option, but performance doesn't have it.

perf has two options: --track-memory and --tracemalloc, see the doc:

http://perf.readthedocs.io/en/latest/runner.html#misc

perf has different implementations to track the memory usage:

* resource.getrusage(resource.RUSAGE_SELF) * 1024
* Windows: GetProcessMemoryInfo()
* tracemalloc: tracemalloc.get_traced_memory()[1]

In the 3 cases, perf saves the *peak* of the memory usage.