The current implementation of `pkgutil.walk_packages()` is confusing. Users may be given incomplete results while no exception is raised if they don't explicitly provide the `prefix` parameter. The doc says:

>   prefix is a string to output on the front of every module name on output.

But the fact is, `prefix` is not merely an output formatter at all. This function cannot handle an empty prefix (which is the default) and will often encounter import errors which are then simply ignored without an `onerror` function (which is default again).

See test program for details.

Doc:

https://docs.python.org/3.6/library/pkgutil.html#pkgutil.walk_packages

Source:

https://github.com/python/cpython/blob/3.6/Lib/pkgutil.py

Update test program.

I think this is actually two distinct problems, one documentation one (which should be addressed in the online docs for all currently maintained versions), and one actual functional issue.

The documentation issue is the one you've reported: in order for the recursive descent to work in walk_packages given the current algorithm, then the combination of the given prefix, and the current global import configuration must allow that package to actually be imported. While there is a note about that limitation, it's currently thoroughly unclear.

The functional issue is two-fold:

1. pkgutil.iter_modules() doesn't identify PEP 420 namespace packages correctly (it ignores them as not being potential packages)

2. The recursive import to check pkg.__path__ uses a name based global __import__ rather than the more state independent https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly technique

It's that second problem that introduces the "prefix must be set to get useful output" behaviour that you're currently seeing.

https://bugs.python.org/issue29258 is an existing issue for the PEP 420 limitation (it also notes why fixing that implicitly is a potential problem)

I've retitled this issue to be specifically about the misleading docs for the "prefix" option (and updated the metadata accordingly).

"Recursive descent in pkgutil.walk_packages depends on sys.path" would be a reasonable title for an issue to actually fix the operation to use a better algorithm based on newer importlib APIs, but I haven't filed that myself.

components: + Documentation, - Library (Lib)
title: pkgutil.walk_packages gives incomplete results -> pkgutil.walk_packages "prefix" option docs are misleading
nosy: + docs@python
versions: + Python 3.7
messages: + msg315018

messages: + msg314960