Home Original page

Out of memory generating using SHA

Tim Peters tim_one at email.msn.com
Sun Aug 27 16:33:30 EDT 2000 
[Tw]
> I need to generate SHA CRC-codes for large files, larger than
> available memory. The default implementation I've seen seem to read
> the entire file into memory. Is there a way to read chunks of the file
> instead?

>>> import sha
>>> s = sha.sha()
>>> s.update('a')
>>> s.update('bcd')
>>> s.update('ef')
>>> print s.hexdigest()
1f8ac10f23c5b5bc1167bda84b833e5c057a77d2
>>> print sha.sha('abcdef').hexdigest()
1f8ac10f23c5b5bc1167bda84b833e5c057a77d2
>>>

That is, an sha object can be updated incrementally, and it returns the same
digest as doing it all in one shot.

To read a file in chunks, see the docs for file objects.  In particular,
file.read(N) reads (at most) N bytes.  Stick that in a loop; call .update
for each chunk.
More information about the Python-list mailing list