The behavior is odd, I don't really know how to characterize it other than something serious has gone wrong.  Memory corruption thus differing failures perhaps?  it seems like maybe a race condition?

on the main branch (i cannot reproduce this on 3.10 thankfully):

1) git checkout https://github.com/python/cpython.git upstream
2) mkdir b-u && cd b-u
3) ../upstream/configure --with-pydebug && make -j3 && ./python -m test.regrtest test_importlib

- sometimes it hangs forever.
- sometimes it crashes with a long list of error messages coming out of regrtest itself.  The crashes appear to happen in a child process so regrtest can continue and run other tests if you tell it to run more.  Sometimes it'll be an error about "import _frozenimport_lib as bootstrap" failing with sys.modules containing None. (Linux does that for me)  On macOS when it doesn't hang, I get a blowup from test_importlib/test_threaded_import.py.

I've reproduced this on Linux and macOS.  macOS alternates between a traceback blowup and hang with an occasional pass.  On Linux it is quite consistently a huge chain of stacktraces.

Linux:

```
0:00:00 load avg: 2.48 Run tests sequentially
0:00:00 load avg: 2.48 [1/1] test_importlib
Failed to import test module: test.test_importlib.builtin.test_finder
Traceback (most recent call last):
  File "/home/greg/oss/python/cpython/gpshead/Lib/importlib/__init__.py", line 16, in <module>
    import _frozen_importlib as _bootstrap
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: import of _frozen_importlib halted; None in sys.modules

During handling of the above exception, another exception occurred:

...  tons more  ...
```

Our buildbot fleet is not bloody looking.  Which is why I tried it on two different systems and OSes before reporting.  When I do a ./configure and run everything within the source tree it does not fail.  This is only for proper out-of-tree builds (which are what I always use - as should everybody).

Do our buildbots only do in-tree builds?

for easy pasting:

git clone https://github.com/python/cpython.git upstream && \
mkdir b-u && cd b-u && \
../upstream/configure --with-pydebug && make -j3 && \
./python -m test.regrtest test_importlib

[change python to python.exe for macos, adjust -j as appropriate]

(if I can figure out git bisect command i'll start one before going to bed...)

(oh hey, i've got macOS producing the same error as Linux now somehow)

further down in the tracebacks in that situation:

```
  File "/Users/greg/oss/python/upstream/Lib/importlib/_bootstrap.py", line 887, in _fix_up_module
    assert module.__file__ == __file__, (module.__file__, __file__)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: ('/Users/greg/oss/python/upstream/Lib/zipimport.py', '/Users/greg/oss/python/b-u/../upstream/Lib/zipimport.py')
```

git bisect on Linux yielded this culprit change:


```
79cf20e48d0b5d69d9fac2a0204b5ac2c366066a is the first bad commit
commit 79cf20e48d0b5d69d9fac2a0204b5ac2c366066a
Author: Eric Snow <ericsnowcurrently@gmail.com>
Date:   Thu Oct 14 15:32:18 2021 -0600

    bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656)
```

...
Running a git bisect ritual on macOS wasn't helpful, test_importlib out of tree too often simply hangs there (going much further back).  Apparently more than one possible issue at play here.  Lets let this issue be for the Linux one.

I'm looking into it.

> Do our buildbots only do in-tree builds?

From what I understand, we do not have any (stable?) buildbots that check out-of-tree builds.

FWIW, in the last month I started testing my changes out-of-tree before merging, after Victor once pointed out that I had broken such builds.  Clearly there must be a gap in what I'm doing.

@gps, on macOS, you may need to bump the open file descriptors limit for the test process, i.e. ulimit -n 1000. I build and run out-of-tree builds all the time on macOS.

The PR I put up should fix the problem.  If it doesn't then I'll probably drop the checks in the second half of FrozenImporter._fix_module().

FWIW, Greg's test case does not fail for me with 6a533a4238

New changeset f4b12440cf24d7636755aac5b2645e47713557c7 by Eric Snow in branch 'main':
bpo-45506: Go back to not running most of test_embed in out-of-tree builds. (gh-29063)
https://github.com/python/cpython/commit/f4b12440cf24d7636755aac5b2645e47713557c7

New changeset 17c61045c51512add61a9e75e9c7343cf4e4fb82 by Eric Snow in branch 'main':
bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040)
https://github.com/python/cpython/commit/17c61045c51512add61a9e75e9c7343cf4e4fb82

i'm considering this fixed based on my testing.  we still lack buildbots using out of tree setup.  i'll ponder that.  But this issue need not track that infrastructure.

Thanks for bringing this up, Greg!

New changeset 7b438282d39e742b2628a32e7c6699a140ff4cfb by Eric Snow in branch 'main':
bpo-45506: Stop skipping test_embed. (gh-29300)
https://github.com/python/cpython/commit/7b438282d39e742b2628a32e7c6699a140ff4cfb

Hijacking this issue number for a related test failure I found: if you "make install" Python and then run the test_embed tests from the build directory, some will successfully find the stdlib under PREFIX and fail because they expect to never find it.

PR incoming.

New changeset 4c616911b69ce07fb35da1721506bfaba0998c30 by Steve Dower in branch 'main':
bpo-45506: Fix test_embed expecting to not find stdlib in source tree build when stdlib has been installed. (GH-29649)
https://github.com/python/cpython/commit/4c616911b69ce07fb35da1721506bfaba0998c30

assignee: eric.snow
resolution: fixed
stage: patch review -> resolved