Home Original page

bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure by tomMoral · Pull Request #17670 · python/cpython

added 2 commits

December 20, 2019 07:39

rad-pat

@rad-pat @tomMoral 

…tdown(wait=False)

When a ProcessPoolExecutor was created, a job added and then shutdown with
wait=False, an OSError: handle is closed error was raised by the ThreadWakeup
class. The threadwakeup should not be closed on shutdown if wait is False,
but in the shutdown_worker method.

@pitrou pitrou changed the title bpo-39104 Fix hanging ProcessPoolExcutor on shutdown nowait with pickling failure bpo-39104: Fix hanging ProcessPoolExcutor on shutdown nowait with pickling failure

Feb 13, 2020

pitrou

@tomMoral

pitrou

pitrou

pitrou

pitrou

@pitrou

@pitrou pitrou changed the title bpo-39104: Fix hanging ProcessPoolExcutor on shutdown nowait with pickling failure bpo-39104: Fix hanging ProcessPoolExecutor on shutdown nowait with pickling failure

Feb 16, 2020

@pitrou

This was referenced

Apr 29, 2020

colesbury referenced this pull request in colesbury/nogil

Oct 6, 2021 
…ckling failure (GH-17670)

As reported initially by @rad-pat in #6084, the following script causes a deadlock.

```
from concurrent.futures import ProcessPoolExecutor


class ObjectWithPickleError():
    """Triggers a RuntimeError when sending job to the workers"""

    def __reduce__(self):
        raise RuntimeError()


if __name__ == "__main__":
    e = ProcessPoolExecutor()
    f = e.submit(id, ObjectWithPickleError())
    e.shutdown(wait=False)
    f.result()  # Deadlock on get
```

This is caused by the fact that the main process is closing communication channels that might be necessary to the `queue_management_thread` later. To avoid this, this PR let the `queue_management_thread` manage all the closing.



https://bugs.python.org/issue39104



Automerge-Triggered-By: @pitrou