Graceful loop shutdown with running executor tasks

This repository was archived by the owner on Nov 23, 2017. It is now read-only.

This repository was archived by the owner on Nov 23, 2017. It is now read-only.

@asvetlov

Description

When I send a long task to tread-pool executor and immediately close the loop without waiting future returned by .run_in_executor() call I get exception report in logs:

ERROR:concurrent.futures:exception calling callback for <Future at 0x7ff1ae393630 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.4/concurrent/futures/_base.py", line 297, in _invoke_callbacks
    callback(self)
  File "/home/andrew/projects/asyncio/asyncio/futures.py", line 410, in <lambda>
    new_future._copy_state, future))
  File "/home/andrew/projects/asyncio/asyncio/base_events.py", line 487, in call_soon_threadsafe
    handle = self._call_soon(callback, args)
  File "/home/andrew/projects/asyncio/asyncio/base_events.py", line 461, in _call_soon
    self._check_closed()
  File "/home/andrew/projects/asyncio/asyncio/base_events.py", line 288, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

That's because task executed in thread has finished after loop closing. The task calls call_soon_threadsafe() (see futures.wrap_future() for details) and it fails on check for loop health.

Due threaded nature it's hard to avoid situations like this: threads cannot be cancelled, we need cope with these even after loop closing. That's pretty common situation, especially for client-side jobs like web page fetching: DNS resolver is executed in thread pool also.

I see two options:

  1. Ignore loop closing in wrap_future call, skip call_soon_threadsafe() for closed loops
  2. Do the same but only if special boolean flag passed into run_in_executor() (while I don't see situations when I don't like the behavior, maybe flag should be True by default).