Home Original page

[Python-Dev] The untuned tunable parameter ARENA_SIZE

Victor Stinner victor.stinner at gmail.com
Thu Jun 1 10:58:17 EDT 2017 
It seems very complex and not portable at all to "free" a part of an
arena. We already support freeing a whole arena using munmap(). I was
a huge enhancement in our memory allocator. Change made in Python 2.5?
I don't recall, ask Evan Jones:
http://www.evanjones.ca/memoryallocator/ :-)

I'm not sure that it's worth it to increase the arena size and try to
implement the MADV_DONTNEED / MADV_FREE thing.

Victor

2017-06-01 11:21 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:
> Thanks for detailed info.
>
> But I don't think it's a big problem.
> Arenas are returned to system by chance.  So other processes
> shouldn't relying to it.
>
> And I don't propose to stop returning arena to system.
> I just mean per pool (part of arena) MADV_DONTNEED can reduce RSS.
>
> If we use very large arena, or stop returning arena to system,
> it can be problem.
>
> Regards,
>
> On Thu, Jun 1, 2017 at 6:05 PM, Siddhesh Poyarekar <siddhesh at gotplt.org> wrote:
>> On Thursday 01 June 2017 01:53 PM, INADA Naoki wrote:
>>> * On Linux, madvice(..., MADV_DONTNEED) can be used.
>>
>> madvise does not reduce the commit charge in the Linux kernel, so in
>> high consumption scenarios (and where memory overcommit is disabled or
>> throttled) you'll see programs dying with OOM despite the MADV_DONTNEED.
>>  The way we solved it in glibc was to use mprotect to drop PROT_READ and
>> PROT_WRITE in blocks that we don't need when we detect that the system
>> is not configured to overcommit (using /proc/sys/vm/overcommit_memory).
>> You'll need to fix the protection again though if you want to reuse the
>> block.
>>
>> Siddhesh
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
More information about the Python-Dev mailing list