Home Original page

[Python-Dev] Lazy initialization of module global state

Neil Schemenauer neil at python.ca
Fri Sep 8 11:50:12 EDT 2017 
This is an idea that came out of the lazy module loading (via AST
analysis), posted to python-ideas.  The essential idea is to split
the marshal data stored in the .pyc into smaller pieces and only
load the parts as they are accessed.  E.g. use a __getattr__ hook on
the module to unmarshal+exec the code.

I have a very early prototype:

https://github.com/warsaw/lazyimport/blob/master/lazy_compile.py

Would work like a "compile_all.py" tool.  It writes standard .pyc
files right now.  It is not there yet but I should use the AST
analysis, like the lazy module load stuff, to determine if things
have potential side-effects on module import.  Those things will get
loaded eagerly, like they do now.

Initially I was thinking of class definitions and functions but now
I realize any global state could get this treatment.  E.g. if you
have a large dictionary global, don't unmarshal it until someone
accesses the module attribute.

This should be pretty safe to do and should give a significant
benefit in startup time and memory usage.
More information about the Python-Dev mailing list