Home Original page

Reading a large csv file

skip at pobox.com skip at pobox.com
Wed Jun 24 12:21:57 EDT 2009 
    Mag> s=0

    Mag> #Takes the longest here
    Mag> for y in fs:
    Mag>      continue
    Mag>   a=y.split(',')
    Mag>   s=s+1
    Mag>   dset.resize(s,axis=0)
    Mag> fs.close()

    Mag> f.close()

    Mag> This works but just takes a VERY long time.

    Mag> Any way to optimize this?

I sort of suspect you're missing something there.  Is there nothing between
the for loop and the overly indented continue statement?

At any rate, try using the csv module to read in your records:

    import csv

    reader = csv.reader(fs)
    ...

    for s, row in enumerate(reader):
        dset.resize(s, axis=0)

-- 
Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/
    when i wake up with a heart rate below 40, i head right for the espresso
    machine. -- chaos @ forums.usms.org
More information about the Python-list mailing list