v.3.2a3

If the maxtasksperchild argument is used, the program will just hang after whatever that value is rather than working as expected. Tested in Windows XP 32-bit

test code:

import multiprocessing

def f(x):
    return 0;

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=2,maxtasksperchild=1);
    results = list();
    for i in range(10):
        results.append(pool.apply_async(f, (i)));
    pool.close();
    pool.join();
    for r in results:
        print(r);
    print("Done");

I can duplicate this using python-2.7-8.fc14.1 on Fedora 14, and using map_async with the pool.

This problem arises because the pool's close method is called before all the tasks have completed. Putting a sleep(1) before pool.close() won't exhibit this lockup.
The root cause is that close makes the workers handler thread exit: since the maxtasksperchild argument is used, workers exit when they've processed their max number of tasks. But since the workers handler thread exited, it doesn't maintain the pool of workers anymore, and thus the remaining tasks are not treated anymore, and the task handler thread waits indefinitely (since it waits until the cache is empty).
The solution is to prevent the worker handler thread from exiting until the cache has been drained (unless the pool is terminated in which case it must exit right away).
Attached is a patch and relevant test.

Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.

> Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.
>

Yes. We should fix those.

Here's an updated patch.
I'll open a separate issue for the thread-safety.

The patch looks good to me, thanks.

New changeset 3465a9b2d25c by Charles-François Natali in branch '2.7':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/3465a9b2d25c

New changeset 52c98a729a71 by Charles-François Natali in branch '3.2':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/52c98a729a71

New changeset c2cdabc44665 by Charles-François Natali in branch 'default':
Issue #10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/c2cdabc44665

James, thanks for the report!

stage: patch review -> resolved

nosy: + pitrou
messages: + msg146119

keywords: + needs review
stage: patch review

nosy: + neologix
messages: + msg133648

keywords: + patch

type: behavior
versions: + Python 3.1, Python 2.7