In Python 3.10, it seems that top-level frames generated by running exec() have their f_lineno attribute set to None.

inspect.getframeinfo() tries to build context lines and fails on this line in such a case:

        start = lineno - 1 - context//2

because lineno is None.

It's not clear whether this is a bug in inspect or the way such frames get their f_lineno attribute initialized.

The same code works just fine in Python 3.9.

To add some more context:

This came up while porting eGenix PyRun to Python 3.10. While installing setuptools 58.2.0 via "pyrun setup.py install", an exception was raised in getframeinfo().

PyRun uses exec() to run Python code:

    def pyrun_exec_code_file(filename, globals_dict, locals_dict=None):
        with open(filename, 'r', encoding='utf-8') as file:
            source = file.read()
        code = compile(source, filename, 'exec', optimize=pyrun_optimized)
        exec(code, globals_dict, locals_dict)

Using pdb, I then found that the top frame does not have f_lineno set in Python 3.10.

What is `source`?

In the case of setuptools, this would be the file setup.py, but I think the specific file is not relevant. I can try to come up with a shorter example.

Update: I've been trying hard to find a short version which triggers the issue, but so far it seems to only trigger when using exec() from a frozen Python module.

There don't appear to be many ways frame->f_lineno can end up being -1 (which then gets translated into None in Python). _PyCode_CheckLineNumber() is one source I found.

Any hints where to look for the cause of this weird effect ?

If I knew where to look, I would be looking myself :)

Is the frozen module one built into CPython or one you have generated?

I see this in modules frozen by the Tools/freeze/ tool.

The line numbers shown for such frozen modules in the frameinfo stack are a bit off as well, compared normal Python. Could this be an indication that there's something not working quite right, which then leads to _PyCode_CheckLineNumber() returning -1 ?

The Tools/freeze/ doesn't do anything special, BTW. All it does is load the module and then store the marshal'ed code objects in C arrays. The information read from those C arrays should be the same as what Python reads from PYC files.

I've looked at how the importlib freeze logic works, compared to Tools/freeze/freeze.py.

The only difference I can spot is that importlib uses C to build the C array and does a compile followed by a marshal.dumps, whereas freeze.py loads all modules into memory and then runs marshal on module.__code__.

Could this cause issues with the line number calculations in 3.10 ?

Turns out this was a bug in the freeze.py script I was using. I had added a bug work-around for the modulefinder module and even though it should work as advertised, it seems to be missing some code object attributes when recreating the objects which fixed file paths.

The stdlib version uses the .replace() method which was added in Python 3.8 and that appears to work better.

Is it possible that code objects now have some extra attributes in 3.10 which aren't exposed ? E.g. things placed into ce_extras ?

Hmm, perhaps I should reopen the ticket, even though I now found the cause.

After all, it is possible that lineno is None and inspect.getframeinfo() cannot handle it :-)

And it may be worthwhile investigating why recreation of a code object using:

            return types.CodeType(co.co_argcount,
                                  co.co_posonlyargcount,
                                  co.co_kwonlyargcount,
                                  co.co_nlocals, co.co_stacksize,
                                  co.co_flags, co.co_code, co.co_consts,
                                  co.co_names, co.co_varnames,
                                  co.co_filename, co.co_name,
                                  co.co_firstlineno, co.co_lnotab,
                                  co.co_freevars, co.co_cellvars)

does not necessarily create a valid copy of a code object co.

You are on own if you create code objects by calling `types.CodeType`.
The docs could be a lot clearer about that, though.

pull_requests: + pull_request30134
stage: resolved -> patch review