Home Original page

[Python-Dev] python process creation overhead

Stephen J. Turnbull stephen at xemacs.org
Wed May 14 04:33:42 CEST 2014 
Gregory Szorc writes:

 > But the great many of us still on 2.7 likely won't see a benefit,
 > correct? How insane would it be for people to do things like pass -S
 > in the shebang and manually implement the parts of site.py that are
 > actually needed?

Well, since it probably won't work ....<wink/>  That is to say,
site.py typically provides different facilities to different programs
-- that's why some parts of it show up as unneeded.  So you need to
carefully analyze *each* *subprocess* that you propose to invoke with
-S and determine which parts of site.py it needs.

In most cases I suspect you would better look at alternatives that
avoid invoking a subprocess per task, but instead maintains a pool of
worker processes (or threads).  You might even be able to save network
traffic or IPC by caching replies to common requests in the worker
processes, which may save more per task than process invocation.

Even where -S makes sense, I would suggest invoking "python -S"
explicitly from the parent process rather than munging the shebang in
the children.  (You do need to be careful to audit changes in the
child programs to be sure they aren't changed in ways that change
site.py requirements.  Putting -S in the shebang may help catch such
problems early.)
More information about the Python-Dev mailing list