Home Original page

[Python-ideas] Python 3000 TIOBE -3%

M.-A. Lemburg mal at egenix.com
Fri Feb 10 19:36:52 CET 2012 
Massimo Di Pierro wrote:
> Forking is a solution only for simple toy cases and in trivially parallel cases. People use processes to parallelize web serves and task queues where the tasks do not need to talk to each other (except with the parent/master process). If you have 100 cores even with a small 50MB program, in order to parallelize it you go from 50MB to 5GB. Memory and memory access become a major bottle neck.

By the time we 100 core CPUs, we'll be measuring RAM in TB, so that
shouldn't be a problem ;-)

Many Python use cases are indeed easy to scale using multiple processes
which then each run on a separate core, so that approach is a very
workable way forward.

If you need to share data across processes, you can use a shared
memory mechanism. In many cases, the data to be shared will already
be stored in a database and those can easily be accessed from all
processes (again using shared memory).

I often find these GIL discussion a bit theoretical. In practice
I've so far never run into any issues with Python scalability. It's
other components that cause a lot more trouble, like e.g. database
query scalability, network congestion or disk access being too slow.

In cases where the GIL does cause problems, it's usually better to
consider changing the application design and use asynchronous processing
with a single threaded design or a multi-process design where each of
the processes only uses a low number of threads (20-50 per process).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 10 2012)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
More information about the Python-ideas mailing list