Home Original page

Python speed-up

Bengt Richter bokr at oz.net
Wed Sep 22 16:34:21 EDT 2004 
On Wed, 22 Sep 2004 21:39:25 +0200, Gerrit <gerrit at nl.linux.org> wrote:

>Alex Martelli wrote:
>> Putting together a big string with a loop of 'bigstring+=piece' is
>> perhaps the best-known performance trap in Python.  Python 2.4 has
>> turned somersaults to lessen the penalty you pay for this, but it's
>> still pretty slow compared to "accumulate pieces in a list, ''.join the
>> list when it's done".  There's really nothing better than this _in
>> general_.
>
>Why isn't cStringIO faster than concatenating strings?
You are timing extra overheads besides the actual concatenation of the data, I beleive.

>
>Using python2.4:
>
>$ python timeit.py -s 'from cStringIO import StringIO; s=StringIO()' "s.write('foo')"
>1000000 loops, best of 3: 0.555 usec per loop
It should improve a little if you hoist out the s.write attribute lookup handicap
from the loop, e.g., with swrite = s.write and then use swrite('foo')
but you are still timing a function call in the mix.

>
>$ python timeit.py -s 's=""' "s+='foo'"
>1000000 loops, best of 3: 0.383 usec per loop
It should slow down a bit if you give it the same function-call handicap as swrite
one way or another.
>
>$ python timeit.py -s 'L=[]' "L.append('foo')"
>1000000 loops, best of 3: 0.32 usec per loop
>
>$ python timeit.py -s 'from StringIO import StringIO; s=StringIO()' "s.write('foo')"
>100000 loops, best of 3: 3.19 usec per loop
>
>I was about to advise cStringIO as being much faster than concatenating
>strings, but just before I was about to send the e-mail, I spied with my
>little eye that the numbers were swapped - concatenating strings is
>actually *faster* than using cStringIO! What's happening?
>
Apples vs oranges, partly, and Skip mentions s+='xxx' optimization for 2.4.

Regards,
Bengt Richter
More information about the Python-list mailing list