concurrent.futures uses polling in its worker threads and processes
(with a timeout of 0.1).
It means that:
1) this prevents CPUs to enter low power states durably
2) it incurs latency when calling shutdown() on an executor (this seems to be the main source of slowness in test_concurrent_futures under Linux)

Ok, here is a patch. Summary:
- leave a minimal amount of polling (every 600 seconds) to avoid blocking forever if there's a bug (shouldn't happen of course, but who knows? especially with multiprocessing)
- when wanting to wakeup a worker, put None in its receiving queue
- remove periodic cleanup of thread references by using a weak dict instead
- in tests, compute runtime and make the test fail if the runtime exceeds 60 seconds (to report aforementioned synchronization bugs)

Tested under Linux (extensively) and Windows 7 (briefly).

(patch is for 3.2, by the way. Perhaps this should only be fixed in default?)

I would suggest that you base your patch on 3.3/default.

> I would suggest that you base your patch on 3.3/default.

Well, since the module is new, I think it would be nice to fix such
quirks in the bugfix branch as well. So, following the recommended
workflow, I've started with a 3.2 patch.

Your approach seems workable but your patch allows the interpreter to exit while work items are still being processed. See the comment at the top of concurrent/futures/thread.py.

> Your approach seems workable but your patch allows the interpreter to
> exit while work items are still being processed. See the comment at
> the top of concurrent/futures/thread.py.

Why are you saying that? In my patch, _python_exit() still takes care of
joining worker threads.

Sorry, I didn't read an error message very carefully. When I apply your patch I see:

>>> from concurrent.futures import *
>>> from time import *
>>> t = ThreadPoolExecutor(5)
>>> t.submit(sleep, 100)
<Future at 0x8acc94c state=running>
>>> <ctrl-D>
Error in atexit._run_exitfuncs:
NameError: global name 'thread' is not defined

Does that not happen in your environment?

> Sorry, I didn't read an error message very carefully. When I apply your patch I see:
> 
> >>> from concurrent.futures import *
> >>> from time import *
> >>> t = ThreadPoolExecutor(5)
> >>> t.submit(sleep, 100)
> <Future at 0x8acc94c state=running>
> >>> <ctrl-D>
> Error in atexit._run_exitfuncs:
> NameError: global name 'thread' is not defined
> 
> Does that not happen in your environment?

Ah, thank you, it does happen here too. I should fix the error (a typo)
and add tests for this too.

Ok, here is a new patch with an additional test for the atexit hook. If you don't object, I would like to start committing the test changes, and then the code changes themselves.

Oops, test didn't work under Windows. Here is a new patch.

New changeset 76a898433a02 by Antoine Pitrou in branch '3.2':
Add tests for the atexit hook in concurrent.futures (part of #11635)
http://hg.python.org/cpython/rev/76a898433a02

New changeset d6bbde982c1c by Antoine Pitrou in branch 'default':
Add tests for the atexit hook in concurrent.futures (part of #11635)
http://hg.python.org/cpython/rev/d6bbde982c1c

Tests now committed, here is a patch without them.

After studying the multiprocessing code, it turns out that Queue.get() with a timeout does its own rather high-frequency polling under Windows (see Modules/_multiprocessing/pipe_connection.c). Therefore, here is an updated patch which doesn't have a security timeout at all.

New changeset 4390d6939a56 by Antoine Pitrou in branch '3.2':
Issue #11635: Don't use polling in worker threads and processes launched by
http://hg.python.org/cpython/rev/4390d6939a56

New changeset a76257a99636 by Antoine Pitrou in branch 'default':
Issue #11635: Don't use polling in worker threads and processes launched by
http://hg.python.org/cpython/rev/a76257a99636

I've now pushed the patch. I hope this won't break anything, closing.

stage: patch review -> resolved

messages: + msg132012

messages: + msg131986

messages: + msg131920

stage: needs patch -> patch review
messages: + msg131918
versions: + Python 3.2

messages: + msg131831
keywords: + patch
nosy: gregory.p.smith, bquinlan, pitrou, jyasskin, stutzbach