This bug applies to 6.x to

[2007-04-12 18:12 UTC] jp at df5ea dot net

Description:
------------
When decoding a string with surrogate pairs in it, JSON_decode() produces incorrect UTF-8. Instead of encoding the two surrogate characters as one UTF-8 sequence it encodes it as two sequences wich represent the two surrogate code points.

The decoded string is actually CESU-8. The JSON_encode() function can not encode such a string.

I have a patch to JSON_parse.c that transcodes the UTF-16 properly to UTF-8.

Reproduce code:
---------------
<?php
$single_barline = "\360\235\204\200";
$array = array($single_barline);
print bin2hex($single_barline) . "\n";
// print $single_barline . "\n\n";
$json = json_encode($array);
print $json . "\n\n";
$json_decoded = json_decode($json, true);
// print $json_decoded[0] . "\n";
print bin2hex($json_decoded[0]) . "\n";
print "END\n";
?>


Expected result:
----------------
The output form the two bin2hex functions should be the same:

f09d8480

["\ud834\udd00"]

f09d8480
END


Actual result:
--------------
The second string is different from the input string and illegal UTF-8.

f09d8480

["\ud834\udd00"]

eda0b4edb480
END

[2007-04-12 19:41 UTC] iliaa@php.net

Can you post a link to the patch?

[2007-04-12 20:07 UTC] jp at df5ea dot net

http://anna.df5ea.net/~jp/JSON_parser.c.patch

An extra parameter is added to utf16_to_utf8(): prev_utf16. This parameter is used to store the previously decoded UTF-16 code unit. When the function encounters an high surrogate this value is used to look for a low surrogate. From this pair it builds the correct UTF-8 sequence.

When it encounters an surrogate code point not in a pair it is ignored. The prev_utf16 variable in JSON_parser() is reset between different strings.

If there is a speed concern regarding the parser it is also possible to drop the prev_utf16 part. The decoder function could then look in the decoding buffer to look for the low surrogate. If needed I can submit a patch to get the function operating in this way.

[2007-04-15 14:39 UTC] iliaa@php.net

Can you please provide an optimized version of the patch?

[2007-04-16 21:15 UTC] jp at df5ea dot net

http://anna.df5ea.net/~jp/JSON_parser.c.2.patch

This patch only adds code to utf16_to_utf8(). When it encounters a high surrogate it looks in the string buffer for a low surrogate. If it has found a pair then it replaces the pair with the proper UTF-8 sequence.

utf16_to_utf8() will still emit incorrect UTF-8 when you encode surrogate characters outside of pairs. But UTF-16 containg such non-paired surrogate code units is incorrect too.

[2007-04-16 22:31 UTC] iliaa@php.net

This bug has been fixed in CVS.

Snapshots of the sources are packaged every three hours; this change
will be in the next snapshot. You can grab the snapshot at
http://snaps.php.net/.
 
Thank you for the report, and for helping us make PHP better.

[2007-04-17 08:25 UTC] jp at df5ea dot net

I failed to check it in advance, but from looking at the sources of the latest CVS snapshot this bug does also apply to PHP 6.x.

Maybe it should be fixed there too.