Message 321842 - Python tracker

Message321842

Author pitrou
Recipients davin, fhstoica, pitrou, xiang.zhang
Date 2018-07-17.16:51:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1531846277.67.0.56676864532.issue34140@psf.upfronthosting.co.za>
In-reply-to
Content
The problem is you're joining the child processes before draining the queue in the parent.

Generally, instead of building your own kind of synchronization like this, I would recommend you use the higher-level abstractions provided by multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor.

By the way, this issue is mentioned precisely in the documentation:

"""
As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.

This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.
"""

(from https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues)
History
Date User Action Args
2018-07-17 16:51:17pitrousetrecipients: + pitrou, davin, xiang.zhang, fhstoica
2018-07-17 16:51:17pitrousetmessageid: <1531846277.67.0.56676864532.issue34140@psf.upfronthosting.co.za>
2018-07-17 16:51:17pitroulinkissue34140 messages
2018-07-17 16:51:17pitroucreate