Message 128786 - Python tracker

Message128786

Author	vstinner
Recipients	eric.smith, ezio.melotti, lemburg, mark.dickinson, ron_adam, vstinner, ysj.ray
Date	2011-02-18.15:05:33
SpamBayes Score	3.2290056e-05
Marked as misclassified	No
Message-id	<1298041533.87.0.990211455284.issue7330@psf.upfronthosting.co.za>
In-reply-to

Content
> Oh, what if the trunked char* cannot be decoded correctly? > e.g. a tow-bytes character is divided in the middle? Yes, but PyUnicode_FromFormatV() uses UTF-8 decoder with replace error handler, and so the incomplete byte sequence will be replaced by � (it doesn't fail with an error). Example: >>> "abc€".encode("utf-8")[:-1].decode("utf-8", "replace") 'abc�'

Content

> Oh, what if the trunked char* cannot be decoded correctly?
> e.g. a tow-bytes character is divided in the middle? 

Yes, but PyUnicode_FromFormatV() uses UTF-8 decoder with replace error handler, and so the incomplete byte sequence will be replaced by � (it doesn't fail with an error). Example:

>>> "abc€".encode("utf-8")[:-1].decode("utf-8", "replace")
'abc�'

History
Date	User	Action	Args
2011-02-18 15:05:33	vstinner	set	recipients: + vstinner, lemburg, mark.dickinson, eric.smith, ron_adam, ezio.melotti, ysj.ray
2011-02-18 15:05:33	vstinner	set	messageid: <1298041533.87.0.990211455284.issue7330@psf.upfronthosting.co.za>
2011-02-18 15:05:33	vstinner	link	issue7330 messages
2011-02-18 15:05:33	vstinner	create