@edgan8 Thanks for bringing this up!

The reason I set a memory limit is to prevent one memory-demanding parallel execution kills another (it happened before).

For example, let's say we only have 16G RAM and employ 2 parallel workers:

• Worker 1 allocates 15GB
• Worker 2 tries to allocate >1 GB and it will just die :(

But I think we should make it user-configurable (and also document it) so that people can use it in their preferred ways.

@edgan8 Thanks for bringing this up!

The reason I set a memory limit is to prevent one memory-demanding parallel execution kills another (it happened before).

For example, let's say we only have 16G RAM and employ 2 parallel workers:

• Worker 1 allocates 15GB
• Worker 2 tries to allocate >1 GB and it will just die :(

But I think we should make it user-configurable (and also document it) so that people can use it in their preferred ways.

Got it, thanks for the context! The specific issue I'm running into is I run this in a docker environment that already has a hard limit on stack size that is lower than 4GB. In that case, this code will crash trying to set the stack size higher than the os hard limit. If we make it configurable, that would solve the issue, or we could modify the resource limit code to first check if the existing limits are already below the requested ones before trying to set them.

@edgan8 Thanks for bringing this up!

I think we can make the following updates:

1. Make maximum_memory_bytes configurable through environment variables
2. The default value is min(4GB, system_max)
3. Users can set it to some bytes or do -1 which turns it to None

Let me know if you want to implement this. I can also take it over in the incoming days. Thanks again for the report!

@edgan8 Thanks for bringing this up!

I think we can make the following updates:

1. Make maximum_memory_bytes configurable through environment variables
2. The default value is min(4GB, system_max)
3. Users can set it to some bytes or do -1 which turns it to None

Let me know if you want to implement this. I can also take it over in the incoming days. Thanks again for the report!

Yes those updates make sense, for (2) I think we can use resources to check the current hard limit for a resource type to set an upper bound well. If you have the bandwidth would trust you to make the changes better! Thanks for the fast response, this code has been very useful.

Let me know if f0897c4 fixed the issue. :)

Let me know if f0897c4 fixed the issue. :)

Got it thanks! In that case I can just set the environment variable to -1. It doesn't fix the issue though because the stack size could have hard limits that are smaller than the total memory size.