Home Original page

70% [* SPAM *] Re: multiprocessing.Queue blocks when sending large object

DPalao dpalao.python at gmail.com
Mon Dec 5 03:02:08 EST 2011 
El Martes Noviembre 29 2011, DPalao escribió:
> Hello,
> I'm trying to use multiprocessing to parallelize a code. There is a number
> of tasks (usually 12) that can be run independently. Each task produces a
> numpy array, and at the end, those arrays must be combined.
> I implemented this using Queues (multiprocessing.Queue): one for input and
> another for output.
> But the code blocks. And it must be related to the size of the item I put
> on the Queue: if I put a small array, the code works well; if the array is
> realistically large (in my case if can vary from 160kB to 1MB), the code
> blocks apparently forever.
> I have tried this:
> http://www.bryceboe.com/2011/01/28/the-python-multiprocessing-queue-and-lar
> ge- objects/
> but it didn't work (especifically I put a None sentinel at the end for each
> worker).
> 
> Before I change the implementation,
> is there a way to bypass this problem with  multiprocessing.Queue?
> Should I post the code (or a sketchy version of it)?
> 
> TIA,
> 
> David

Just for reference. The other day I found the explanation by "ryles" on 
his/her mail of 27th aug 2009, with title "Re: Q: multiprocessing.Queue size 
limitations or bug...". It is very clarifying.
After having read that I arranged the program such that the main process did 
not need to know when the others have finished, so I changed the process join 
call with a queue get call, until a None (one per process) is returned.

Best,

David
More information about the Python-list mailing list