Home Original page

File size limitations

Thomas Wouters thomas at xs4all.net
Fri Feb 4 06:19:37 EST 2000 
On Thu, Feb 03, 2000 at 04:10:07PM -0700, Bjorn Pettersen wrote:
> gmc333 at my-deja.com wrote:

> > I'd like to use Python for some large-scale file processing on Windows
> > NT. There is a possibility that the size of the files I'll be working
> > with will exceed 2^32 bytes.
> >
> > Can Python handle files that large? Random I/O is a specific concern,
> > since an implementation based on fseek/lseek system calls will fail
> > (these calls use 32-bit math, and bad things can happen with seeks that
> > go too far).

> I'm pretty sure Python uses the fseek/lseek etc. functions for file IO.  It
> would be trivial to create an extension module with Swig
<snip>

Actually, The Code does this:

#if defined(HAVE_FSEEKO)
        ret = fseeko(f->f_fp, offset, whence);
#elif defined(HAVE_FSEEK64)
        ret = fseek64(f->f_fp, offset, whence);
#else
        ret = fseek(f->f_fp, offset, whence);
#endif

(see fileobject.c) which at least suggests python can handle 64bit
filesystems if they exist. Another hint was a posting a few weeks back by
someone who hit a bug because tell() returns a long object instead of an int
object, for very large files (on filesystems that support it.) Wether it
works on Windows NT is something else, but I'd hazard a 'yes' on that,
myself. It's easy to try, though, start the interpreter, open a file, seek()
to a very large number, and write some characters.

-- 
Thomas Wouters <thomas at xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
More information about the Python-list mailing list