Issue29659
Created on 2017-02-27 00:57 by goodboy, last changed 2022-04-11 14:58 by admin. This issue is now closed.
| Pull Requests | |||
|---|---|---|---|
| URL | Status | Linked | Edit |
| PR 328 | closed | goodboy, 2017-02-27 01:19 | |
| Messages (5) | |||
|---|---|---|---|
| msg288616 - (view) | Author: Tyler Goodlet (goodboy) * | Date: 2017-02-27 00:57 | |
It would be handy to be able to pass the `length` default argument defined by `shutil.copyfileobj()` to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which call `shutil.copyfileobj()`. I propose a simple non-invasive change where the following functions will also expose the `length` kwarg and pass it downwards to `copyfileobj`: - copyfile - copy - copy2 |
|||
| msg288962 - (view) | Author: Raymond Hettinger (rhettinger) * ![]() |
Date: 2017-03-04 05:26 | |
This seems reasonable to me. |
|||
| msg288974 - (view) | Author: Inada Naoki (methane) * ![]() |
Date: 2017-03-04 09:06 | |
How about increasing default value to 32KiB or 64KiB too? binutils's cp works well for most cases, while it doesn't have option to specify blocksize. I want Python's copy functions works nice enough for common cases too. binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h But copyfileobj can be used other file-likes. On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB. So block size larger than 64KiB may cause unneeded blocking. That's why I suggested 32KiB or 64KiB. |
|||
| msg305625 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2017-11-06 10:11 | |
Could you please provide any benchmarks Tyler? |
|||
| msg336504 - (view) | Author: Inada Naoki (methane) * ![]() |
Date: 2019-02-25 09:27 | |
bpo-33671 fixed this. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:58:43 | admin | set | github: 73845 |
| 2019-02-25 09:27:42 | methane | set | status: open -> closed versions: + Python 3.8, - Python 3.7 messages: + msg336504 resolution: fixed |
| 2017-11-06 10:11:21 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages: + msg305625 |
| 2017-03-04 09:06:50 | methane | set | nosy:
+ methane messages: + msg288974 |
| 2017-03-04 05:26:00 | rhettinger | set | nosy:
+ rhettinger messages: + msg288962 |
| 2017-03-03 21:55:28 | terry.reedy | set | nosy:
+ terry.reedy type: enhancement |
| 2017-02-27 19:16:23 | Mariatta | set | components: + Library (Lib) |
| 2017-02-27 01:19:32 | goodboy | set | pull_requests: + pull_request288 |
| 2017-02-27 00:57:24 | goodboy | create | |
