Home Original page

[Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

Victor Stinner victor.stinner at gmail.com
Tue Jan 28 11:22:40 CET 2014 
2014-01-28 "Martin v. Löwis" <martin at v.loewis.de>:
> Debugging reveals that it is actually the many integer objects which
> trigger the sharing code. So a much simplified example of Victor's
> benchmarking code can use
>
> data = [0]*10000000
>
> The difference between version 2 and version 3 here is that v2 marshals
> a lot of "0" integers, whereas version 3 marshals a single one, and then
> a lot of references to this integer.

Since the output size looks to be the same, it may be interesting to
special-case small integers, or even integers and floats in general.
Handling references to these numbers takes probably more CPU, whereas
the gain on the file size is probably minor.

I wrote a short patch:
http://bugs.python.org/issue20416

"dumps v3 is 60% faster, loads v3 is also 14% *faster*."

"dumps v4 is 66% faster, loads v4 is 16% faster."

"file size (on version 3 and 4) is unchanged with my patch."

"So with the patch, the Python 3.4 default version (4) is *faster*
(dump 20% faster, load 16% faster) and produces *smaller files* (10%
smaller)."

It looks like a win-win patch :-)

The drawback is that files storing many duplicated huge numbers will
not be smaller with marshal version >= 3.

Victor
More information about the Python-Dev mailing list