## Test code ##
## Modified a bit from the original written by Doug Hellmann
## https://pymotw.com/3/multiprocessing/communication.html

import multiprocessing
import time


class Consumer(multiprocessing.Process):
    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue

    def run(self):
        proc_name = self.name
        while True:
            print('Getting task')
            next_task = self.task_queue.get()
            print(f'task got: {next_task}')
            if next_task is None:
                print('{}: Exiting'.format(proc_name))
                self.task_queue.task_done()
                break
            print('{}: {}'.format(proc_name, next_task))
            answer = next_task()
            self.task_queue.task_done()
            self.result_queue.put(answer)


class Task:
    def __init__(self, a, b):
        self.a = a
        self.b = b

    def __call__(self):
        time.sleep(0.1)
        return '{self.a} * {self.b} = {product}'.format(
            self=self, product=self.a * self.b)

    def __str__(self):
        return '{self.a} * {self.b}'.format(self=self)


def test():
    tasks = multiprocessing.JoinableQueue()
    results = multiprocessing.Queue()
    num_consumers = multiprocessing.cpu_count() * 2
    print('Creating {} consumers'.format(num_consumers))
    consumers = [Consumer(tasks, results) for i in range(num_consumers)]
    [w.start() for w in consumers]
    num_jobs = 10
    print('Putting')
    [tasks.put(Task(i, i)) for i in range(num_jobs)]
    print('Poisoning')
    [tasks.put(None) for i in range(num_consumers)]
    print('Joining')
    tasks.join()
    while num_jobs:
        result = results.get()
        print('Result:', result)
        num_jobs -= 1

###
1. This code works perfectly in 3.7.1 but halts the main process in 3.7.2
2. It seems the JoinableQueue is empty when it is accessed by processes.
3. IMHO, resource sharing mechanism in multiprocessing.queue seems not working properly.

I just tried your script in "9a3ffc" (3.7.2final) and "260ec2c36a" (3.7.1final) and it worked on both without halting the main process.

I'm on MacOS Sierra, can you give more details about your environment?

Here is my environment

---system
CPU: Intel i5 @2.67GHz
RAM: 8G
OS: Windows 10 Home (64bit)
OS version: 1803
OS build: 17134.472

---python
version1: 3.7.1 AMD64 on win32
version2: 3.7.2 AMD64 on win32
Python path: (venv)/Scripts/python.exe
IDE: VS Code(1.30.1)
Terminal: Git Bash

I couldn't reproduce on Ubuntu either.  I tried the "fork", "forkserver" and "spawn" methods (all with 3.7.2).

Terry, if you are on Windows, can you try the script?  Be sure to enclose the test() call in a "if __name__ == '__main__'" guard.

mp_hang.py: I created the example into a script, I added the __main__ section described by Antoine. I cannot reproduce the bug on the Python master branch on Linux.

@June Kim: What is the output when it hangs? Can you try your example without VS Code? For example, try to run it in a cmd.exe terminal?

I ran mp_hang.py on Windows 10 with Python 3.7.2: the script completes (it doesn't hang). The issue might be specific to VS Code (on Windows?).

Hi there. I get this behavior as well, although only in a venv.

                      Main       Virtual      
v3.7.1:260ec2c36a  Completes    Completes
v3.7.2:9a3ffc0492  Completes      Hangs

Some other details of my setup:
 - Windows 10 Pro, Version 1803 (OS Build 17134.472)
 - Python platform is AMD64 on win32
 - Running from the command line (cmd.exe)
 - The virtual environment was created from the command line like this: 
     .\python -m venv c:\temp\Py-3.7.2b-Venv

In case this is a clue - the attached script "mp_hang2.py" adds a call to qsize() and uses only a single consumer. When I run it from the command line it does one of two things:


Option 1:

C:\TEMP\Py-3.7.2b-Venv\Scripts>.\python.exe "C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py"
Creating 1 consumers
Putting
Poisoning
Joining
Process Consumer-1:
Traceback (most recent call last):
  File "C:\Users\Tom.Wilson\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py", line 18, in run
    print(f'Queue size: {self.task_queue.qsize()}')
  File "C:\Users\Tom.Wilson\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 117, in qsize
    return self._maxsize - self._sem._semlock._get_value()
PermissionError: [WinError 5] Access is denied


Option 2:

C:\TEMP\Py-3.7.2b-Venv\Scripts>.\python.exe "C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py"
Creating 1 consumers
Putting
Poisoning
Joining
Queue size: 2147483647
Getting task
 <<< Hangs here >>>


If I can provide anything else please let me know.

> Hi there. I get this behavior as well, although only in a venv.

Ah, thanks for the clarification.  Then it's a duplicate of issue35797.

resolution: duplicate
stage: resolved

messages: + msg334194

messages: + msg333008