Issue43944
Created on 2021-04-26 20:59 by Genarito, last changed 2022-04-11 14:59 by admin.
| Messages (8) | |||
|---|---|---|---|
| msg391985 - (view) | Author: Genaro Camele (Genarito) | Date: 2021-04-26 20:59 | |
I've a piece of code that submits a task to a [ThreadPoolExecutor][1] which starts a [Process][2]. I've realised that in Python 3.8 that Process finished with exit code `0`. But I've updated Python to the 3.9 version and this started to finishing with exit code `1`! Even when the Process executes an empty task.
Here's a minimal example:
```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor
def some_task():
pass
def execute_error():
p = Process(target=some_task)
p.start()
p.join()
print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!!
executor = ThreadPoolExecutor(max_workers=4)
executor.submit(execute_error)
# execute_error() # IMPORTANT: this works correctly (exit 0)
```
My versions:
```
Ubuntu 21.04
Python 3.9.4
```
**Note** that if `__execute_error` is called outside the ThreadPoolExecutor it works correctly.
Running on Python 3.8.6 exitcode = 0 too.
[1]: https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
[2]: https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.Process
|
|||
| msg396522 - (view) | Author: Alexandre Sicard (sicard_elda) | Date: 2021-06-25 10:16 | |
Thank you very much for this report, Genaro. I encountered the same bug with a Process running in the context of a Django view. Downgrading to Python 3.8 also fixed the issue for me. Versions: python:3.9-alpine Docker image, running Python 3.9.5 and Alpine 3.13. Can also reproduce on the standard (Debian Buster based) python:3.9 image, and on Arch Linux (bare metal), all also running Python 3.9.5. |
|||
| msg398157 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021-07-24 14:11 | |
I am working on a fix for this bug. I'm a beginner cpython contributor, so if anyone recognizes this as a fool's errand, please let me know! |
|||
| msg398159 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021-07-24 14:30 | |
Ah never mind. @Genarito, the ThreadPoolExecutor is supposed to be used as a context manager. In your current code, the script ends and Python starts tearing itself down while `execute_error` is still running in a subprocess.
If you simply use the ThreadPoolExecutor to a context manager, the error goes away::
```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor
def some_task():
pass
def execute_error():
p = Process(target=some_task)
p.start()
p.join()
print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!!
# THIS IS THE IMPORTANT CHANGE
with ThreadPoolExecutor(max_workers=4) as executor:
executor.submit(execute_error)
```
|
|||
| msg398175 - (view) | Author: Genaro Camele (Genarito) | Date: 2021-07-24 21:55 | |
Hi @jack__d, thanks for your answer and time. Unfortunately, It's still a regression, as in Python < 3.9 my example works as expected |
|||
| msg398202 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021-07-26 00:24 | |
I've identified the first bad commit with git-bisect: commit b61b818d916942aad1f8f3e33181801c4a1ed14b Author: Kyle Stanley <aeros167@gmail.com> Date: Fri Mar 27 15:31:22 2020 -0400 bpo-39812: Remove daemon threads in concurrent.futures (GH-19149) Remove daemon threads from :mod:`concurrent.futures` by adding an internal `threading._register_atexit()`, which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads. |
|||
| msg398203 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021-07-26 00:25 | |
The first bad commit was a fix for bpo-39812. |
|||
| msg416966 - (view) | Author: Thomas Grainger (graingert) * | Date: 2022-04-08 09:35 | |
the problem is multiprocessing/process is calling threading._shutdown which tries to join its own thread, because concurrent.futures.thread._threads_queues contains the main thread in the subprocess File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/multiprocessing/process.py", line 333, in _bootstrap threading._shutdown() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1530, in _shutdown atexit_call() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit t.join() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1086, in join raise RuntimeError("cannot join current thread") |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:59:44 | admin | set | github: 88110 |
| 2022-04-08 09:35:24 | graingert | set | nosy:
+ graingert messages: + msg416966 |
| 2021-07-26 00:25:22 | jack__d | set | messages: + msg398203 |
| 2021-07-26 00:24:24 | jack__d | set | messages: + msg398202 |
| 2021-07-24 21:55:55 | Genarito | set | messages: + msg398175 |
| 2021-07-24 14:30:25 | jack__d | set | messages: + msg398159 |
| 2021-07-24 14:11:03 | jack__d | set | nosy:
+ jack__d messages: + msg398157 |
| 2021-06-25 10:16:24 | sicard_elda | set | nosy:
+ sicard_elda messages: + msg396522 |
| 2021-04-30 22:15:35 | terry.reedy | set | nosy:
+ pitrou, davin |
| 2021-04-26 20:59:45 | Genarito | create | |