Home Original page

Profiling memory usage?

Michael James Barber mjbarber at ascc.artsci.wustl.edu
Wed Feb 27 13:22:02 EST 2002 
In article <a5j1vb$7mq9u$1 at ID-11957.news.dfncis.de>,
Emile van Sebille <emile at fenx.com> wrote:
>"Michael James Barber" <mjbarber at ascc.artsci.wustl.edu> wrote in message
>news:a5iogb$laa at ascc.artsci.wustl.edu...
>>[are there memory profiling tools?]
>
>I also looked for a tool and found none.  One of my projects requires a
>
Drat.  Ah well, at least it supports the "no such tool" explanation over 
the "I'm incompetent at using search tools" explanation.  (Either that, or 
I have some bad news for you Emile... ;->)

>there weren't good internal profiling tools, I looked at memory usage
>using pstat.exe on windows, but you could probably use top in a similar
>kludgey output parsing way.  (Now someone can chime in with the *easy*
>way to do this ;-)  
>
Using 'top' is exactly what I'm doing, and it is quite kludgey. Seems like 
there must be an easy way, so I'll second this request!  For me, 
determining the memory used by an object would let me do the needed
profiling quite easily.

>
>The kinds of changes I made included:
[selectively snipping and rearranging a list of nice changes]
>  -- not holding on to references to large structures when I was done
>with them (doh)
>
This is what's killing me.  I need to hold the references somewhere, since 
it takes too long to generate the data not to preserve it all.  Could 
store to disk, I suppose.

>  -- forcing commonly used string parts of keys to be interned
>
What I'm doing is most analogous to this, I think.  I'll present my 
solution, in case others run into the same, and - in a totally 
self-serving fashion - in the the hopes that others will suggest 
improvements.

I think the approach I used is called the "Flyweight" design pattern - but 
I would need to reread the description of the pattern to be sure.  The 
class name could be wrong.  At any rate, here's the class:

class FlyweightFactory:
    """Simple class for working with the flyweight design pattern.  Call 
as a function to use.
    
    If fwf is a FlyweightFactory instance, use as: fw = fwf(obj).  If obj 
is not hashable, fw 
    will be obj itself (i.e., fw is obj). When obj is hashable, fw will be 
equal (fw == obj) to 
    obj, but not the same object (fw is not obj)."""
    
    def __init__(self):
        self._flyweights = {}
        
    def __call__(self,  obj):
        try:
            fw = self._flyweights.setdefault(obj, obj)
        except TypeError:
            # TypeError is raised when obj is mutable
            fw = obj
        return fw

Possibly, the try should be omitted, just letting the TypeError be raised.  
It may also be a better idea to make this a new style class by subclassing 
from dict.  And the docstring isn't very good. 

I was surprised by how easy this was to do.  I certainly hadn't expected 
that identifying where to use it would be the hard part!
-- 
					Michael J. Barber
More information about the Python-list mailing list