Home Original page

pickle->zlib->shelve

Thomas Weholt thomas at bibsyst.no
Wed Oct 20 06:28:31 EDT 1999 
Hi,

I got an idea ( and I`m a newbie, so bare with me ) :

I got an database in which I want to store some customized objects.
These objects are in my opinion not very big, but they seem to take up a
lot of space anyhow, when pushed into a database. On solution I`ve used
in a similar situation using a different programming language, was to
use plain strings, and use some sort of delimiter between each
attribute. That will of course not include any methods defined in any
object. Then the string is stored in a database in the "d[id] = str"
fashion.

These strings can also be be some length, and often lots of repetitive
info. Then it struck me that I could use zlib to compress the string
before inserting it into the database, and decompress it when it was
taken out. The number of items to be processed couldn`t be huge due to
the decompression time of course, but with few items this could possibly
work, and result in a smaller database.

I was thinking maybe I could use pickle and compress the pickled object
too, before storing it into the database and save some space. Pickled
objects also seem to have lots of repetitive data.

Now my question is :

1. has this been tested and found ineffective in relation to a) speed of
decompression or b) size of objects not big enough to gain any
significant size advantage by compression
2. if not tested, why? Maybe a subclass of pickle could implement
compression and decompression of objects?

Since these scripts are to be run on 400Mhz computers and the number of
items will not exceed 20-30, objects being relativly small, how will the
speed be? Any clues??

Awaiting flames and harsh words of discouragement,
Thomas Weholt
More information about the Python-list mailing list