The core problem is that `importlib.invalidate_caches()` is full of
lies and does not actually invalidate all of the relevant
caches. Python maintains sys.path_importer_cache, which maps from
directory names on the sys.path to "importer" objects for that path
entry. `importlib.invalidate_caches()` invalidates the caches inside
each of the per-directory importer objects, but there is an additional
negative cache that is *not* cleared: sys.path entries whose directory
doesn't exist have their importer set to None, which prevents that
path entry from ever being searched unless it is manually cleared.

This failure comes up rarely, because it only occurs if the following
events occur in sequence:

 1. 'stubgen-test-path' is added to sys.path, but no new import is
    done while there is a 'stubgen-test-path' subdirectory in the working
    directory. This is done by all the stubgen tests that don't end with
    `_import`.
 2. Some non-stubgen test does an import of a previously unimported
    module. This will cause
    sys.path_importer_cache['stubgen-test-path'] to be set to None
    when the directory doesn't exist during the module search.
    This can happen the *first* time that a `_python2` test is run
    since `parse.parse()` dynamically imports `mypy.fastparse2`
    when asked to parse python 2.
 3. A stubgen test tries to use importlib to import a module in
    'stubgen-test-path'. All of the `_import` tests do this.

We fix this by clearing out the relevant entry from
`sys.path_importer_cache` ourselves.

msullivan added a commit that referenced this pull request

While trying to test mypy_mypyc wheels in travis on OS X, I pretty
consistently got failures trying to import a plugin module. I had
trouble reproducing it anywhere else, but the issue turned out to be a
cousin of the dreaded stubgen flake (#4811), where caching in
importlib causes a file to be missed even though it is present now.

Here we solve it by using absolute paths in the `load_plugins`
manipulations of sys.path.

gvanrossum pushed a commit that referenced this pull request

…5937)

While trying to test mypy_mypyc wheels in travis on OS X, I pretty
consistently got failures trying to import a plugin module. I had
trouble reproducing it anywhere else, but the issue turned out to be a
cousin of the dreaded stubgen flake (#4811), where caching in
importlib causes a file to be missed even though it is present now.

Here we solve it by using absolute paths in the `load_plugins`
manipulations of sys.path.