Home Original page

Size limit on compiling?

Bengt Richter bokr at oz.net
Thu Feb 26 13:27:30 EST 2004 
On Thu, 26 Feb 2004 09:31:02 -0500, Peter Hansen <peter at engcorp.com> wrote:

>Leif K-Brooks wrote:
>> 
>> I have a Python file of 1.2MB (it's a static database, not code, so it
>> really does have to be that big). It doesn't seem to get compiled when
>> imported (there's no .pyc file created), so importing it takes a few
>> seconds which I would really like to avoid. Is Python not compiling it
>> because of the large size? Is it because the whole file is on one really
>> long line? How can I fix this?
>
>If it's the main file, it won't get compiled anyway.  Only files
>that are imported are compiled.  In any case, see Aahz' answer for the
>best approach.
>
If your file is highly structured (or you can separately prepare a version that is),
you might consider accessing it via mmap and struct (or even without struct,
if it's e.g., a flat array of fixed-length strings). You could enhance this
by pre-computing an index and storing it at the end or beginning of the file,
maybe data for a dict of (offset,length) values, similarly packed/retrieved
from the raw binary file (byte string) format. If you open the file read-only,
an efficient OS will not even page in or allocate swap file space for anything
you don't access (though 1.2 mb these days is not a big worry ;-).
But if your data is complicated, pretty soon you'd be reinventing pickle
and various db things, and you might as well use what's available already.

Regards,
Bengt Richter
More information about the Python-list mailing list