When I first wrote and started using ThreadPoolExecutor, I had a lot of code like this:
with ThreadPoolExecutor(max_workers=500) as e:
e.map(download, images)
I didn't expect that `images` would be a large list but, if it was, I wanted all of the downloads to happen in parallel.
I didn't want to have to explicitly take into account the list size when starting the executor (e.g. max_works=min(500, len(images))) but I also didn't want to create 500 threads up front when I only needed a few.
My use case involved transient ThreadPoolExecutors so I didn't have to worry about idle threads.
In principle, I'd be OK with trying to avoid unnecessary thread creation if the implementation can be simple and efficient enough.
https://github.com/python/cpython/pull/6375 seems simple enough but I haven't convinced myself that it works yet ;-) |