The lib2to3 library doesn't detect/apply any fixes if the source files aren't included with the distribution. See get_all_fix_names at https://github.com/python/cpython/blob/master/Lib/lib2to3/refactor.py#L30

This affects Azure's official Python site extensions [1], which only include bytecode pyc's for the fixes [2].

This results in bad installs when using setuptools to install packages that leverage the use_2to3 flag during the install [3]

[1] https://github.com/Azure/azure-python-siteextensions
[2] https://github.com/Azure/azure-python-siteextensions/issues/14
[3] https://github.com/pypa/setuptools/issues/1120#issuecomment-320769664

What are you reporting as the bug here?  2to3 obviously can't work without the source, so based just on what you have written here this sounds like an Azure bug.

I'm reporting that lib2to3 doesn't work without having source .py files for the fixes that are packaged with the lib2to3 library. Perhaps the get_all_fix_names method in lib2to3/refactor.py should check for either .py or .pyc files?

The source files are included with the package that I'm installing (anyjson 0.3.3 in this case).

Ah, I see.  We don't really support .pyc-only distribution, though we try not to break it.

Do you want to propose a fix?

It also breaks .zip file distribution, which I'm fairly sure we do explicitly support by virtue of having "python36.zip" in sys.path by default. And the ".pyc-only in a zip file" distribution is *totally* busted :) (and also officially released for Windows...)

I'm dragging in an importlib expert to the discussion, probably against his will, since this is really a problem of "how do I programmatically discover what I can import without using os.listdir()". Hopefully Brett can answer that question and we can come up with a patch.

Simplest way is to do https://docs.python.org/3/library/importlib.html#importlib.util.find_spec and see if a spec can be found for the module in question. That will do the search for the module but it won't load it. This does dictate that you know the name of the module upfront, though. If you want more of a "list all modules since I don't know what I want" then you're out of luck until I develop the API (maybe 3.7?).

I'm sorry about entering the discussion without knowing any connection between the original problem and Brett's question, but isn't the latter answered by pkgutil.iter_modules (and pkgutil.walk_packages if needed)?

You're technically right, Verdan, but pkgutils is kind of a hack as it isn't always using standard APIs to achieve its goals, so I personally never recommend using the module unless absolutely necessary and you know exactly how brittle it can be.

Perhaps the more important question is whether 2to3 really needs to dynamically generate a list of fixers or if that should have been stopped due to YAGNI. There are only a few modules in the stdlib, and it doesn't seem to support proper namespace packages, so unless there are plans to suddenly add many more we could just make it a hard coded list.