Home Original page

How to efficiently read binary files?

Robert Kern robert.kern at gmail.com
Sun Apr 30 18:03:34 EDT 2006 
Grant Edwards wrote:
> On 2006-04-30, David Lees <debl2NoSpam at verizon.net> wrote:
> 
>>I want to process large binary files (>2GB) in Python.  I have played 
>>around with prototypes in pure Python and profiled the code.  Most of 
>>the time seems to be spent converting back and forth to and from strings 
>>using the struct module.  Is there a way to directly read into an array 
>>of integers in Python?
> 
> Perhaps the numarray module?

numpy for new code, please. In particular, numarray is limited by 32-bit APIs
even on 64-bit platforms, so a >2GB file will be difficult to process even when
using mmap. numpy removed this restriction on 64-bit platforms. 32-bit users
will still have to split up the file into <2GB chunks, though.

  http://numeric.scipy.org/
  https://lists.sourceforge.net/lists/listinfo/numpy-discussion

-- 
Robert Kern
robert.kern at gmail.com

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco
More information about the Python-list mailing list