As mentioned in Issue 20132, iterencode() and iterdecode() only work on text-to-byte codecs, because they assume particular data types when finalizing the incremental codecs. This patch changes the signature of the IncrementalEncoder and IncrementalDecoder methods from

IncrementalEncoder.encode(object[, final])
IncrementalEncoder.decode(object[, final])

to

IncrementalEncoder.encode([object,] [final])
IncrementalEncoder.decode([object,] [final])

so that iteren/decode(), and perhaps in the future, StreamWriter/Reader, can operate the incremental codec without knowing what kind of data should be processed.

Original patch has lots of whitespace changes, probably due to generated codec code not being regenerated for a long time. This diff ignores the space changes, so should be easier to review.

Another idea that doesn’t involve changing the incremental codec APIs is kind of described in <https://bugs.python.org/issue7475#msg145986>: to add format parameters to iterencode() and iterdecode(), which would allow it to determine the right data type to finalize the codecs with.

The patch changes public interface. This breaks compatibility with third-party codecs implementing it.

We have found other solution to iterencode/iterdecode problem. For example we can buffer iterated values and encode with one step delay:

    prev = sentinel = object()
    for input in iterator:
        if prev is not sentinel:
            output = encoder.encode(prev)
            if output:
                yield output
        prev = input
    if prev is not sentinel:
        output = encoder.encode(prev, True)
        if output:
            yield output

Or remember the previous value and use it to calculate the empty value at the end (works only if input type supports slicing):

    prev = sentinel = object()
    for input in iterator:
        output = encoder.encode(input)
        if output:
            yield output
        prev = input
    if prev is not sentinel:
        output = encoder.encode(prev[:0], True)
        if output:
            yield output

Serhiy’s two proposals won’t work for codecs that include non-zero output for zero input:

>>> tuple(iterencode((), "utf-8-sig"))
(b'\xef\xbb\xbf',)
>>> encode(b"", "uu")
b'begin 666 <data>\n \nend\n'
>>> encode(b"", "zlib")
b'x\x9c\x03\x00\x00\x00\x00\x01'

However I agree that changing the incremental codec APIs is not ideal. Since nobody seems to care that much, it might be simpler to document that:

* iterencode() only works where text str objects can be encoded, so base64-codec is not supported, but rot13-codec is supported
* iterdecode() only works where bytes objects can be decoded, so rot13-codec is not supported, but base64-codec should be supported (pending other aspects of Issue 20132)

Here is my documentation proposal.

> it might be simpler to document that

Agreed.

New changeset 402eba63650c by Martin Panter in branch '3.5':
Issue #23231: Document codecs.iterencode(), iterdecode() shortcomings
https://hg.python.org/cpython/rev/402eba63650c

New changeset 0837940bcb9f by Martin Panter in branch '3.6':
Issue #23231: Merge codecs doc from 3.5 into 3.6
https://hg.python.org/cpython/rev/0837940bcb9f

New changeset 1955dcc27332 by Martin Panter in branch 'default':
Issue #23231: Merge codecs doc from 3.6
https://hg.python.org/cpython/rev/1955dcc27332

messages: + msg273203
nosy: + r.david.murray

components: + Documentation, - Library (Lib)
stage: patch review

messages: + msg256746
versions: + Python 3.6, - Python 3.5

nosy: + serhiy.storchaka

messages: + msg233933