The join() method of multiprocessing.Pool calls self._worker_handler.join(): it's a thread running _handle_workers(). The core of this thread function is:

        while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
            pool._maintain_pool()
            time.sleep(0.1)

I understand that the delay of 100 ms is used to check regularly the stop condition changed. This sleep causes a mandatory delay of 100 ms on Pool.join().

Attached PR 11136 modify _worker_handler() loop to wait on threading.Event events, so Pool.join() completes as soon as possible.

Example:
---
import multiprocessing
import time

def the_test():
    start_time = time.monotonic()
    pool = multiprocessing.Pool(1)
    res = pool.apply_async(int, ("1",))
    pool.close()
    #pool.terminate()
    pool.join()
    dt = time.monotonic() - start_time
    print("%.3f sec" % dt)

the_test()
---

Minimum timing with _handle_results() using:

* current code (time.sleep(0.1)): min 0.132 sec
* time.sleep(1.0): min 1.033 sec
* my PR using events (wait(0.1)): min 0.033 sec

Currently, join() minimum timing depends on _handle_results() sleep() duration (100 ms).

With my PR, it completes as soon as possible: when state change and/or when a result is set.

My PR still requires an hardcoded delay of 100 ms to workaround bpo-35478 bug: results are never set if the pool is terminated.

My PR 11136 doesn't work: _maintain_pool() should be called frequently to check when a worker completed. Polling worker exit status seems inefficient :-(

asyncio uses SIGCHLD signal to be notified when a child process completes. SafeChildWatcher calls os.waitpid(pid, os.WNOHANG) on each child process, whereas FastChildWatcher() uses os.waitpid(-1, os.WNOHANG).

> My PR 11136 doesn't work: _maintain_pool() should be called frequently to check when a worker completed. Polling worker exit status seems inefficient :-(

I created bpo-35493: "multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit".

_worker_handler has two issues:

* It polls the worker status every status every 100 ms: I created bpo-35493 to investigate how to avoid that
* After close() or terminate() has been called, it loops until self._cache is empty. I would like to use result.wait(), but a result never completes after terminate(): I created bpo-35478 to see if tasks can be unblocked in that case (to ensure that result.wait() completes after terminate().

Nobody managed to find a solution in 3 years. I close the issue.

dependencies: - multiprocessing: ApplyResult.get() hangs if the pool is terminated, multiprocessing.Pool._worker_handler(): use SIGCHLD to be notified on worker exit
resolution: out of date
stage: patch review -> resolved