[2016-06-14 13:21 UTC] cmb@php.net

 [2016-06-15 00:08 UTC] cmb@php.net

 [2016-06-26 04:22 UTC] php-bugs at lists dot php dot net

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.

 [2016-07-17 11:07 UTC] cronfy at gmail dot com

 [2016-07-17 11:07 UTC] cronfy at gmail dot com

Yes, it it locale dependent. If I prepend script with 'setlocale(LC_ALL, "C")', everything works correctly. But if I don't (my default locale is ru_RU.UTF-8) or if I explicitly use 'setlocale(LC_ALL, "ru_RU")', result is incorrect.

I can't demonstrate it at 3v4l.org as it does not support required locales.

 [2016-07-17 11:08 UTC] cronfy at gmail dot com

I can't change this bug status to Reopened, because I do not have enough privileges.

 [2016-07-17 11:45 UTC] cmb@php.net

 [2016-07-21 16:13 UTC] cmb@php.net

 [2016-07-21 16:13 UTC] cmb@php.net

This issue does not only affect str_getcsv(), but also other CSV
reading functions, so I've changed the bug title.

 [2016-07-21 17:13 UTC] cmb@php.net

 [2018-04-09 18:21 UTC] ganlvtech at qq dot com

str_getcsv not correctly work with qouted multibyte character

PHP version: 7.2.2
Operating system: Windows 10 zh-CN

Description:
------------
str_getcsv not correctly work with qouted multibyte characters.

When the multibyte characters are simply seperated by comma, everything seems ok.

If the value contains a quotation mark("), I need to escape quotation mark by doubled quotation mark(""), and quote the value with a pair of quotation mark. And when I try to decode the csv string by str_getcsv, this value will combined with next value (I lost a column and got two value together in one column).

There is not just one type of wrong result. But I think every type of wrong result be caused by the escaped quotation mark.

Bug #72330: CSV fields incorrectly split if escape char followed by UTF chars


Test script:
---------------
<?php
// Test 1
$data = [
    "\xE4\xBD\xA0\xE5\xA5\xBD", // 你好
    "\xE4\xB8\x96\xE7\x95\x8C", // 世界
];
$encoded = implode(',', array_map(function ($value) {
    return '"' . str_replace('"', '""', $value) . '"';
}, $data));
var_dump(str_getcsv($encoded) === $data);

// Test 2
$data = [
    "\"\xE5\x95\x8A", // "啊
];
$encoded = str_putcsv($data);
var_dump(str_getcsv($encoded) === $data);

/** @link https://bugs.php.net/bug.php?id=64183 */
function str_putcsv($fields, $delimiter = ',', $enclosure = '"', $escape_char = '\\') {
    $stream = fopen('php://memory', 'w+');
    fputcsv($stream, $fields, $delimiter, $enclosure, $escape_char);
    rewind($stream);
    return stream_get_contents($stream);
}


Expected result:
----------------
bool(true)
bool(true)


Actual result:
--------------
bool(false)
bool(false)

 [2018-04-09 18:21 UTC] ganlvtech at qq dot com

Related To: Bug #72330

 [2018-04-09 21:30 UTC] cmb@php.net

 [2018-04-10 10:56 UTC] cmb@php.net

 [2018-04-10 11:44 UTC] ganlvtech at qq dot com

Thank you very much.

I've also tested the script on Ubuntu 16.04 with php 7.2.2. Test passed.

So the problem may only be reproduced on Windows platform with code page set to a code page with multi-byte chars (e.g. Chinese Simplified cp936).

After searching on the web for half an hour, I found that 'setlocale to utf8 on Windows' is imposible.

There might be a hack for the problem. Set locale to 'en_US' or any locale without multi-byte characters. This works fine, but I don't think it's a good way.

In a word, this function is not binary-safe.

 [2018-04-10 16:48 UTC] cmb@php.net

 [2018-04-10 16:48 UTC] cmb@php.net

Thanks for further investigating!

> Set locale to 'en_US' or any locale without multi-byte
> characters. This works fine, but I don't think it's a good way.

It appears to be a viable workaround, as long as you are dealing
with valid UTF-8, and the delimiter and enclosing characters are
single-byte.

A cleaner, but less efficient solution would be to convert the
string to a supported encoding, so str_getcsv() works as expected,
and to convert the resulting array elements back to UTF-8.

Either way, the general caveat regarding setlocale() in
multi-threaded environments[1] applies.

So the best solution might be to use a userland implementation
which does not rely on the locale at all, but rather supports
specifying the character encoding of the CSV input.  Adding
something like this to PHP would require the RFC process[2]. 

> In a word, this function is not binary-safe.

In PHP context "binary-safe" usually means that a function
correctly processes strings containing NUL bytes.  str_getcsv()
does this[3], so the function is "binary-safe".

Obviously, you are referring to another defintion of
"binary-safe"[4]. In this sense, str_getcsv() is not
"binary-safe", and actually it can't be, because there are
character encodings where the binary representation of the
delimiter and escape character may be *part* of the binary
representation of other characters.

I'm going to improve the relevant documentation, and will close
this ticket afterwards, since there's not much else which could be
done, unfortunately.

[1] <http://www.php.net/manual/en/function.setlocale.php#refsect1-function.setlocale-notes>
[2] <https://wiki.php.net/rfc/howto>
[3] <https://3v4l.org/qMBEG>
[4] <https://en.wikipedia.org/wiki/Binary-safe>

 [2018-04-10 17:01 UTC] cmb@php.net

 [2018-04-10 17:20 UTC] ganlvtech at qq dot com

Thanks for your systematical interperation. This might not be a bug, and it can be closed now. Thank you.