Home Original page

Exponential time increase in garbage collection

Sean McGrath sean.mcgrath at propylon.com
Sat May 18 13:08:43 EDT 2002 
[Tim Peters]
 >Sorry, this is too vague to work with.  Start with which verion of Python
 >you're using,

Python 1.5.2 but I've tried with 2.1.3 as well and get the same result.

 >and try to come up with a self-contained small test case
 >exhibiting the symptom.

System in question runs to many thousands of lines of code
so getting a standalone test case is proving interesting but
work continues apace to so do.

 >> The behavior is the same on Windows and Unix.

 >At the start, you said this happens "depending on the machine".  So what
 >varies besides Windows-vs-Unix that *does* make a difference?  Also be
 >specific aobut which flavors of Windows and Unix you're using (different
 >flavors of WIndows have radically different memory behavior, and ditto
 >Unix).

Sorry, I meant that the absolute numbers vary from machine to machine but the
underlying problem is the same. Windows 2000, NT and Solaris 2.6.

I have spent the day investigating this and the problem seems to
be to do with cPickle!

I have pickled versions of my, reasonably complex objects.
If I load, say 200 of these from their pickles, deleting them
takes quite a long time. If however, I create 200 duplicate
objects using copy.deepcopy(), the deleting time
is significantly faster.

It does not matter how I store references to the objects - dictionaries, 
lists, variable names.
It seems that whenever they are finalised - even if that is triggered by 
dropping off
the end of the program, the long wait starts...

In my last test, 140 objects read from pickles took 323 seconds to delete.
By switching to deepcopy() it dropped to 45 seconds - about 7 times faster.

Does this scenario ring any bells with anyone?

regards,
Sean
More information about the Python-list mailing list