I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file) 

I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383)

    if not hasattr(iterable, '__len__'):
        iterable = list(iterable)

    if chunksize is None:
        chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
        if extra:
            chunksize += 1

When we convert an iterable to list(iterable), we lost the advantage of using the generator. I'm not sure how to fix it, maybe we can set a default value for an object don't have '__len__' attr, any ideas?

Do you imap or imap_unorderd?
They are intended for use with iterator, including generator.

Thank you for the hint, INADA. I think we should add something like "if you are using generator, consider use imap instead" in https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.pool.Pool.map

> I think we should add something like "if you are using generator, consider use imap instead" 

I think it's not good hint.
There are short generator.  And there are long (or infinite) iterator other than generator too.
Maybe, "if iterator is not sequence (e.g. generator) and can be very big, consider using `imap` or `imap_unorderd` with explicit `chunksize` option for better efficiency."

But I'm not good at writing English.
Someone other than me can write better paragraph.

Thank you, I will try to make a pull request and let other to edit it.

One thing worth a try here maybe turn `len` to `operator.length_hint`. But I am not sure it's a good idea and just a mention here.

Thank you Xiang Zhang, I found the code keeps hanging when I use imap, I will try to figure out tomorrow.

The code didn't work with imap because imap create a generator, so we can't access result outside the with statement.

    with Pool(os.cpu_count()) as p:
        result = p.imap(clean_up, k, 50)
    for r in result:
        print(r)

In https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#using-a-pool-of-workers I found the correct example. I'm not sure should me add example or warning in imap function.

Why accessing the result outside of with block? The pool is terminated while exiting the block before the work is done.

Yes, we should not. But we can do this when use map function. the document gives a good example but doesn't say much about real differences between map and imap. Maybe we should add some notes like INADA suggest. 

map function will convert iterable to list if it doesn't implement __len__ function, so if you are using a generator, you should consider use imap. As well as add a warning about don't try to access the result outside the with statement. 

But if you guys think the docs are good enough, please close this issue.

I'm +1 for INADA's change, but not more examples trying to distinguish every detail difference.

New changeset 3bab40db96efda2e127ef84e6501fda0cdc4f5b8 by Antoine Pitrou (Windson yang) in branch 'master':
bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324)
https://github.com/python/cpython/commit/3bab40db96efda2e127ef84e6501fda0cdc4f5b8

New changeset c2674bf11036af1e06c1be739f0eebcc72dfbf7a by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) (gh-11673)
https://github.com/python/cpython/commit/c2674bf11036af1e06c1be739f0eebcc72dfbf7a

This is basically fixed, except that I'll let the Release Manager choose whether 3.6 gets the fix as well. Thanks!

resolution: fixed
stage: patch review -> resolved

messages: + msg321836
versions: - Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7