This is a follow up of #25063 and similar to socket.sendfile() (#17552). It provides a 20/25% speedup when copying files with shutil.copyfile(), shutil.copy() and shutil.copy2(). Differently from #25063 this is used for filesystem files only and copyfileobj() is left alone.

Unmerged #26826 is also related to this. I applied #26826 patch and built a wrapper around copy_file_range() and the speedup is basically the same. Nevertheless, even when #26826 gets merged it probably makes sense to rely on sendfile() in case copy_file_range() is not available (it was introduced in 2016) or the UNIX platform supports file-to-file copy via sendfile(2) (even though I'm not aware of any, so this would basically be Linux-only).

Some benchmarks:

    $ dd if=/dev/urandom of=/tmp/f1 bs=1K count=128
    $ time ./python -m timeit -s 'import shutil; p1 = "/tmp/f1"; p2 = "/tmp/f2"' 'shutil.copyfile(p1, p2)'

128K copy
=========

--- without patch:

    2000 loops, best of 5: 160 usec per loop

    real    0m2.353s
    user    0m0.454s
    sys     0m1.435s

--- with patch:

    2000 loops, best of 5: 187 usec per loop

    real    0m2.724s
    user    0m0.627s
    sys     0m1.634s

8MB copy
========

$ dd if=/dev/urandom of=/tmp/f1 bs=1M count=8

--- without patch:

    50 loops, best of 5: 9.51 msec per loop

    real    0m3.392s
    user    0m0.343s
    sys     0m2.478s

--- with patch:

    50 loops, best of 5: 7.75 msec per loop

    real    0m2.878s
    user    0m0.105s
    sys     0m2.187s

512MB copy
==========

--- without patch:

    1 loop, best of 5: 872 msec per loop

    real    0m5.574s
    user    0m0.402s
    sys     0m3.115s

--- with patch:

    1 loop, best of 5: 646 msec per loop

    real    0m5.475s
    user    0m0.037s
    sys     0m2.959s

Duplicate of https://bugs.python.org/issue25156 ?

Oh! I got confused by the fact that #25063 was rejected due to concerns about copyfileobj() otherwise I would have commented on your patch which I totally missed. Yes, this overlaps with #25156 patch but it uses the same logic of socket.sendfile() which IMO is more robust. If you agree I'd close #25156 and continue in here, and I'm more than happy to co-author the contribution if that's OK with you.

Honestly, whatever gets this thing moving forward is good with me.

I'm a bit depressed by the number of issues here that have a good patch waiting to be merged, or even read, and that languish for years.

I haven't read your patch in detail, but if others agree that it is better than mine and can be merged, let's do it.

> I'm a bit depressed by the number of issues here that have a good patch waiting to be merged, or even read, and that languish for years.

I'm really sorry about that. We, core developers, are doing our best, but we are 34 active core developers who merged 5000 pull requests in 1 year. We don't scale well :-(

We are working on getting more people onboard.

I updated the code to rely on sendfile(2) on Linux only, which apparently is the only one supporting copy between regular files and added a check to fail immediately in case the filesystem is full. 

Can somebody review the patch?

Closing as duplicate of #33671.

Closing as duplicate of #33671.

resolution: duplicate
stage: patch review -> resolved