Home Original page

Slow output

alex23 wuwei23 at gmail.com
Wed Jun 27 22:05:47 EDT 2012 
On Jun 28, 3:33 am, subhabangal... at gmail.com wrote:
> May any one suggest me what may be the likely issue?

In situations like this, it always helps to see your code, especially
if you can reduce it down to only the part doing the loading.

One thing that can help reduce memory usage is to replace lists/list
comprehensions with generators. For example, this loads the entire
file into memory:

    for line in open('big.txt').readlines():
         # do something to line

While this only loads one line at a time:

    for line in open('big.txt'):
        # do something to line

You can also do the same with multiple files by creating a generator
to return their content a line at a time:

    filenames = ['a.txt', 'b.txt', 'c.txt']
    files = (open(name) for name in filenames)
    lines = (line for file in files for line in file)

    for line in lines:
        # do something to line

I highly recommend David Beazley's "Generator Tricks for Systems
Programmers" for more techniques like this.
More information about the Python-list mailing list