Issue 32827: Fix incorrect usage of _PyUnicodeWriter_Prepare()

classification
Title: Fix incorrect usage of _PyUnicodeWriter_Prepare()
Type: resource usage Stage: resolved
Components: Interpreter Core, Unicode Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, miss-islington, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords: patch

Created on 2018-02-12 11:47 by serhiy.storchaka, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5636 merged serhiy.storchaka, 2018-02-12 11:51
PR 5650 merged miss-islington, 2018-02-13 06:28
PR 5651 closed miss-islington, 2018-02-13 06:29
PR 5656 merged miss-islington, 2018-02-13 09:02
Messages (6)
msg312038 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-12 11:47
Based on discussion in PR 660.

_PyUnicodeWriter_Prepare() is used incorrectly in unicode_decode_call_errorhandler_writer(), _PyUnicode_DecodeUnicodeEscape() and PyUnicode_DecodeRawUnicodeEscape() in Objects/unicodeobject.c. The second argument is the number of characters that should be reserved after the current position. But in these places the total minimal size is passed to _PyUnicodeWriter_Prepare(). This can lead to allocating more memory than necessary.
msg312094 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2018-02-13 02:26
Nice. The only question I have is this is a bugfix or enhancement? Current code seems won't cause any problem but just allocates more memory than needed.
msg312102 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-13 06:26
I think this is a bugfix and should be backported. This is why I made a separate PR rather of suggesting changes to PR 660.
msg312103 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-13 06:27
New changeset b7e2d67f7c035f09c921ca4e7a36529cd502ccf7 by Serhiy Storchaka in branch 'master':
bpo-32827: Fix usage of _PyUnicodeWriter_Prepare() in decoding errors handler. (GH-5636)
https://github.com/python/cpython/commit/b7e2d67f7c035f09c921ca4e7a36529cd502ccf7
msg312105 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-13 07:15
New changeset 09819ef05a9a1b13321b56c09f48ca45b46e8656 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
bpo-32827: Fix usage of _PyUnicodeWriter_Prepare() in decoding errors handler. (GH-5636) (GH-5650)
https://github.com/python/cpython/commit/09819ef05a9a1b13321b56c09f48ca45b46e8656
msg312112 - (view) Author: miss-islington (miss-islington) Date: 2018-02-13 09:32
New changeset 310b424c476e3de3370990e5b9a10acc64ed17f8 by Miss Islington (bot) in branch '3.6':
bpo-32827: Fix usage of _PyUnicodeWriter_Prepare() in decoding errors handler. (GH-5636)
https://github.com/python/cpython/commit/310b424c476e3de3370990e5b9a10acc64ed17f8
History
Date User Action Args
2022-04-11 14:58:57adminsetgithub: 77008
2018-02-13 09:33:43serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-02-13 09:32:53miss-islingtonsetnosy: + miss-islington
messages: + msg312112
2018-02-13 09:02:24miss-islingtonsetpull_requests: + pull_request5458
2018-02-13 07:15:59serhiy.storchakasetmessages: + msg312105
2018-02-13 06:29:45miss-islingtonsetpull_requests: + pull_request5453
2018-02-13 06:28:48miss-islingtonsetstage: commit review -> patch review
pull_requests: + pull_request5452
2018-02-13 06:27:36serhiy.storchakasetmessages: + msg312103
2018-02-13 06:26:43serhiy.storchakasetmessages: + msg312102
2018-02-13 02:26:33xiang.zhangsetmessages: + msg312094
stage: patch review -> commit review
2018-02-12 11:51:38serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request5436
2018-02-12 11:47:03serhiy.storchakacreate