vstinner@apu$ ./python -W error -m test -v test_warnings

======================================================================
ERROR: test_exec_filename (test.test_warnings.CWarnTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/test/test_warnings/__init__.py", line 450, in test_exec_filename
    exec(codeobj)
  File "<warnings-test>", line 2, in <module>
UserWarning: hello

======================================================================
ERROR: test_exec_filename (test.test_warnings.PyWarnTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/Lib/test/test_warnings/__init__.py", line 450, in test_exec_filename
    exec(codeobj)
  File "<warnings-test>", line 2, in <module>
  File "/home/vstinner/prog/python/master/Lib/warnings.py", line 318, in warn
    globals, source)
  File "/home/vstinner/prog/python/master/Lib/warnings.py", line 363, in warn_explicit
    raise message
UserWarning: hello


It seems to be a regression introduced by bpo-33375: commit 11a896652ee98aa44e59ed25237f9efb56635dcf.

I consider that this issue is easy, core developers/long time contriubtors: please don't fix it, but explain how to fix it and let a newcomer to fix it!
https://mail.python.org/pipermail/python-committers/2017-June/004564.html

Yes, this should be easy. I misunderstood how to use the catch_warnings context manager. I thought that catch_warnings itself set up the warnings filters you need. You actually need to do that with a separate call inside the with block, as shown here:

https://docs.python.org/3/library/warnings.html#testing-warnings

New changeset 419e88d18ccc83e182afca124d42f46794022aab by Victor Stinner (Christopher Frederickson) in branch 'master':
bpo-33912: Fix test_warnings when run with -Werror (GH-7839)
https://github.com/python/cpython/commit/419e88d18ccc83e182afca124d42f46794022aab

Thank you Christopher Frederickson for your contribution! I merged your PR. I confirm that it fixes the issue.

Thanks Thomas Kluyver for the hints ;-)