Home Original page

getting fileinput to do errors='ignore' or 'replace'?

Oscar Benjamin oscar.j.benjamin at gmail.com
Fri Dec 4 04:00:32 EST 2015 
On 4 Dec 2015 08:36, "Serhiy Storchaka" <storchaka at gmail.com> wrote:
>
> On 04.12.15 00:26, Oscar Benjamin wrote:
>>
>> On 3 Dec 2015 16:50, "Terry Reedy" <tjreedy at udel.edu> wrote:
>>>
>>> fileinput is an ancient module that predates iterators (and generators)
>>
>> and context managers. Since by 2.7 open files are both context managers
and
>> line iterators, you can easily write your own multi-file line iteration
>> that does exactly what you want.  At minimum:
>>>
>>>
>>> for file in files:
>>>      with codecs.open(file, errors='ignore') as f
>>>      # did not look up signature,
>>>          for line in f:
>>>              do_stuff(line)
>>
>>
>> The above is fine but...
>>
>>> To make this reusable, wrap in 'def filelines(files):' and replace
>>
>> 'do_stuff(line)' with 'yield line'.
>>
>> That doesn't work entirely correctly as you end up yielding from inside a
>> with statement. If the user of your generator function doesn't fully
>> consume the generator then whichever file is currently open is not
>> guaranteed to be closed.
>
>
> You can convert the generator to context manager and use it in the with
statement to guarantee closing.
>
> with contextlib.closing(filelines(files)) as f:
>     for line in f:
>         ...

Or you can use fileinput which is designed to be exactly this kind of
context manager and to be used in this way. Although fileinput is slightly
awkward in defaulting to reading stdin.

--
Oscar
More information about the Python-list mailing list