Expected behavior of string.Formatter() is to return unicode strings for unicode templates, and "byte" strings for str templates. Which is exactly what it does, with one frustrating exception: for empty unicode string it returns byte str. Test follows:

import string
template = u""
result = string.Formatter().format(template)
assert isinstance(result, unicode)
# AssertionError

Adding failing test.  Patch coming next.

Here are some related failing cases that I found:

>>> f = string.Formatter()
>>> f.format(u"{0}", "")
''
>>> f.format(u"{0}", 1)
'1'
>>> f.format(u"{0}", "a")
'a'
>>> f.format(u"{0}{1}", "a", "b")
'ab'
>>> f.format("{0}", u"a") 
u'a'

Note that PEP 3101 says the following:

"In all cases, the type of the format string dominates - that
is, the result of the conversion will always result in an object
that contains the same representation of characters as the
input format string."

Actually, I'm going to defer on creating a patch because this covers more scenarios than I originally thought and so may require more time.

Format with unicode is a bit of a mess in 2.7.  It would be consistent with the rest of python2 for

  >>> f.format("{0}", u"a")
  u'a'

to be correct.

See also issue 7300 and issue 15276.

What about cases like this?

>>> f.format(u'{0}', '\xe9')
'\xe9'

It seems fixing this issue for non-empty strings would cause formerly running cases like this to raise UnicodeDecodeError.

>>> unicode('\xe9')
 ...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

Would that be acceptable?

Note that I didn't say it was correct, I just said it was consistent :)

And no, breaking stuff that current works is a non-starter for 2.7.

I filed issue 15952 for the behavior difference between format(value) and value.__format__() and the related lack of documentation re: unicode format strings.

Given that the expected behavior for the current issue doesn't seem to be documented (aside from PEP 3101, which is probably too late to follow), we should probably agree on what the behavior should be (as well as documenting it) before or while addressing this issue.

Attached is a proposed patch.

Some explanation behind the patch that stems from the above comments:

The following is an example of Formatter.format() returning str in the current implementation that would break if we made Formatter.format() return unicode whenever format_string is unicode:

>>> f.format(u"{0}", "\xc3\xa9")  # UTF-8 encoded "e-acute".
'\xc3\xa9'

(It would break with a UnicodeDecodeError because 'ascii' is the default encoding.)

Since we can't change Formatter.format(format_string) to return unicode whenever format_string is unicode without breaking existing code, I believe the best we can do is to document the departure from PEP 3101.  Since the caller has to handle return values of type str anyways, I don't think it helps to ensure that more return values are unicode.

messages: + msg170693

nosy: + chris.jerdonek
messages: + msg170552

keywords: + patch
stage: needs patch