HTTPConnection.send() should use os.sendfile when possible to avoid copying data into userspace and back.

This is not possible for two reasons:

- on most POSIX systems, sendfile() works with mmap-like ("regular") files only, while HTTPConnection.send() accepts any file-like object as long as it provides a read() method

- after read()ing a chunk of data from the file and before send()ing it over the socket, the data can be subject to an intermediate conversion (datablock.encode("iso-8859-1")):
http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
...whereas sendfile() can only be used to send a binary file "as-is"

I think we can use sendfile() in ftplib.py though .
I'll open a ticket for that.

2011/12/8 Giampaolo Rodola' <report@bugs.python.org>:
>
> Giampaolo Rodola' <g.rodola@gmail.com> added the comment:
>
> This is not possible for two reasons:
>
> - on most POSIX systems, sendfile() works with mmap-like ("regular") files only, while HTTPConnection.send() accepts any file-like object as long as it provides a read() method
>
> - after read()ing a chunk of data from the file and before send()ing it over the socket, the data can be subject to an intermediate conversion (datablock.encode("iso-8859-1")):
> http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
> ...whereas sendfile() can only be used to send a binary file "as-is"

I presume you could check for a binary mode, though? Also, you can
catch EINVAl on invalid fds.

ftplib's sendfile support is not tracked as issue13559.
Considerations I made there should apply here as well.

Ops! I meant issue13564.

Patch in attachment uses the newly added socket.sendfile() method (issue 17552).

The multiple personalities of HTTPConnection.send() and friends is a bit of a can of worms. I suggest working on Issue 23740 to get an idea of what kinds of file objects are meant to be supported, and what things may work by accident and be used in the real world.

For instance, is it possible to manually set Content-Length, and say supply a GzipFile reader, or file object positioned halfway through the file? How does this interact with the socket.sendfile() call?

This issue is no newcomer friendly, I remove the "easy" keyword.

I would like to take a stab at this. Giampaolo, would it be okay if I made a pull request updated from your patch? With the appropriate "Co-authored-by: Author Name <email_address>" line.

Alex, https://bugs.python.org/issue23740 is identified as a dependency on this issue. We will have to resolve that first, and come back to this. And yes, if you contribute on other's patch, both the contributions will be included and appropriately credited.

To check my understanding

Is the motivation for the closer to

1. using sendfile() will break $X, and we know X
2. there's high probability sendfile() will break something
3. there's unknown probability sendfile() will break something
4. there's low probability sendfile() will break something, but it is still too high
5. any non-trivial change here is too risky, regardless of sendfile()
6. something else?

My guess is 5.

Yes, the point number 5. We will have to evaluate if sendfile side-steps and avoids the issues noted in issue23740

sendfile() only works for plain HTTP. For technical reasons it does not work for HTTPS (*). These days majority of services use HTTPS. Therefore the usefulness of sendfile() patch is minimal.

(*) It is possible to use sendfile() for TLS connections, but the feature requires a Kernel module that provides kTLS offloading feature, https://www.kernel.org/doc/html/latest/networking/tls-offload.html . In user space it requires OpenSSL 3.0.0 with kTLS support. 3.0.0 is currently under development.

messages: + msg388346
versions: + Python 3.10, - Python 3.5

dependencies: + http.client request and send method have some datatype issues
stage: needs patch -> patch review

keywords: + patch