the difference in performance behavior with and without MAP_POPULATE is the following: with MAP_POPULATE perf degradation occurs (and is tangible) only when we run out of space to place a new object and, thus, need to mmap a new file. Without MAP_POPULATE, we distribute degraded performance (due to page faulting on the server side) uniformly across most object puts. As stated, the advantage of the latter is throughput predictability.

It is important to note that, in steady state, when the object store has been fully warmed up, presence/absence of MAP_POPULATE doesn't matter. It only matters during the object store ramp up.

My favorite longer-term solution continues to be preallocating the configured capacity on object store startup as a single mmapped file and pre-faulting it in the background.