s390x RHEL 3.x buildbot:
https://buildbot.python.org/all/#/builders/21/builds/3150

test_cancel_gather_1 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
Ensure that a gathering future refuses to be cancelled once all ... ok
test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ...
Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 4396153272064)>
ok

test_cancel_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_task_catching (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok

--


AMD64 FreeBSD 10-STABLE Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/167/builds/1181

test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

Another similar error on "ARMv7 Debian buster 3.x" buildbot:
https://buildbot.python.org/all/#/builders/176/builds/664

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

AMD64 Ubuntu Shared 3.x:
https://buildbot.python.org/all/#/builders/141/builds/1912

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
test_close (test.test_asyncio.test_unix_events.SafeChildWatcherTests) ... /srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/asyncio/base_events.py:646: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

https://buildbot.python.org/all/#/builders/167/builds/1188

Variant:

test_cancel_both_task_and_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_current_task (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_gather_1 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
Ensure that a gathering future refuses to be cancelled once all ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
Dangling thread: <_MainThread(MainThread, started 34389124096)>
ok
test_cancel_gather_2 (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_inner_future (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok
test_cancel_task_catching (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... ok

See also bpo-34037: "asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads".

New changeset 49a7e347976c9b39149ac7505b11ad6e9e2bdeec by Victor Stinner in branch 'master':
bpo-37137: Fix test_asyncio: use TestCase.set_event_loop() (GH-13779)
https://github.com/python/cpython/commit/49a7e347976c9b39149ac7505b11ad6e9e2bdeec

My change didn't fix the issue:

https://buildbot.python.org/all/#/builders/176/builds/680

test_proc_exited (test.test_asyncio.test_subprocess.SubprocessTransportTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

I push a fix in asyncio, instead of trying to fix individual asyncio tests:

New changeset 0f0a30f4da4b529e0f7df857b9f575b231b32758 by Victor Stinner in branch 'master':
bpo-34037, asyncio: add BaseEventLoop.wait_executor_on_close (GH-13786)
https://github.com/python/cpython/commit/0f0a30f4da4b529e0f7df857b9f575b231b32758

See https://bugs.python.org/issue34037#msg344476

My second attempt isn't fully successful :-( test_asyncio still fails with ENV_CHANGED :-(

https://buildbot.python.org/all/#/builders/185/builds/488

test_current_task_no_running_loop_implicit (test.test_asyncio.test_tasks.CCurrentLoopTests) ... Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)

I suspect that I've added this dangling thread by https://github.com/python/cpython/blob/master/Lib/asyncio/unix_events.py#L1248-L1251 :(

Working on PR

> I suspect that I've added this dangling thread by https://github.com/python/cpython/blob/master/Lib/asyncio/unix_events.py#L1248-L1251 :(

Can you please try to revert commit 13ed07998ad93dbdd94991ba0451b9b559f07972?

We have two options:
1. Apply https://github.com/python/cpython/pull/13792 to wait for dangling threads
2. Revert added watchers entirely

What do you prefer?

> 2. Revert added watchers entirely

I prefer this option just because we must fix all bugs as soon as possible to get the beta1 released, and this bug is just one bug in a list of bugs.

I might try to help you *later* to fix the change to prevent this bug.

Got you. https://github.com/python/cpython/pull/13793

Thanks Andrew! Sorry that you had to revert your change :-(

We can reconsider to merge again your change after beta1 (if our release manager is fine with that).

The initial issue has been fixed.

Let's continue the discussion on https://bugs.python.org/issue35621 to maybe reapply commit 13ed07998ad93dbdd94991ba0451b9b559f07972.