Home Original page

[Python-ideas] PyParallel update (was: solving multi-core Python)

Trent Nelson trent at snakebite.org
Tue Jun 23 16:03:55 CEST 2015 
On Tue, Jun 23, 2015 at 09:53:01AM -0400, Trent Nelson wrote:
> On Sat, Jun 20, 2015 at 03:42:33PM -0600, Eric Snow wrote:
> > Furthermore, removing the GIL is perhaps an obvious solution but not
> > the only one.  Others include Trent Nelson's PyParallels, STM, and
> > other Python implementations..
> 
> So, I've been sprinting relentlessly on PyParallel since Christmas, and
> recently reached my v0.0 milestone of being able to handle all the TEFB
> tests, plus get the "instantaneous wiki search" thing working too.
> 
> The TEFB (Techempower Framework Benchmarks) implementation is here:
>     https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/tefb/tefb.py?at=3.3-px
>     (The aim was to have it compete in this: https://www.techempower.com/benchmarks/#section=data-r10, but unfortunately they broke their Windows support after round 9, so there's no way to get PyParallel into the official results without fixing that first.)
> 
> The wiki thing is here:
> 
> https://bitbucket.org/tpn/pyparallel/src/8528b11ba51003a9821ceb75683ee96ed33db28a/examples/wiki/wiki.py?at=3.3-px
> 
> I particularly like the wiki example as it leverages a lot of benefits
> afforded by PyParallel's approach to parallelism, concurrency and
> asynchronous I/O:
>     - Load a digital search trie (datrie.Trie) that contains every
>       Wikipedia title and the byte-offset within the wiki.xml where
>       the title was found.  (Once loaded the RSS of python.exe is about
>       11GB; the trie itself has about 16 million items in it.)

Oops, I was off by about 12 million:

    C:\PyParallel33>python.exe
    PyParallel 3.3.5 (3.3-px:829ae345012e+, Jun 15 2015, 16:54:16) [MSC v.1600 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> os.chdir('examples\\wiki')
    >>> import wiki as w
    About to load titles trie, this will take a while...
    >>> len(w.titles)
    27962169
More information about the Python-ideas mailing list