tl;dr TCP_NODELAY should be set by default and/or there should be a
      proper way to set it.

I've ported ZEO, ZODB's client-server networking layer to asyncio.
Things were going pretty well.  I've been developing on a
Mac. Yesterday, I ran some performance measurements on Linux and found
some tests ran 30x slower.

After some digging, I came across this:

https://github.com/python/asyncio/issues/311

Then led me to try setting TCP_NODELAY, which provided Linux
performance comparable to Mac OS performance.

Issue 311 suggested that this was a kernal-version issue.  I think
this is a failure to set, or at least provide a way to set a "don't be
stupid" option.

I originally tried this on Ubuntu 14.04, with kernal
3.13.0-74-generic.  I then tried Ubuntu 16.04, in a docker image, with
kernal 4.4.12-boot2docker. For both of these, performance for the
write tests were 30x slower unless I set TCP_NODELAY.

Finally, I tried an Amazon standard AMI, which runs some version of
RedHat, with kernal 4.4.11-23.53.amzn1.x86_64.  On that system,
performance of the tests was similar to Mac OS X without setting
TCP_NODELAY.

Note that the non-slow kernal version was lower than the slow (Ubuntu)
one. I don't think this is mearly a kernal version issue, nor do I
think this should have been dismissed as one.

I couldn't find a way to set TCP_NODELAY cleanly. Did I miss something?
https://github.com/python/asyncio/issues/286 suggests there isn't one.

I ended up having to set the option on _sock transport attributes,
which is dirty and, I assume, won't work with uvloop. (BTW, uvloop was
only ~15x slower on linux systems with this problem.)

I think TCP_NODELAY should be set by default.  Perhaps it shouldn't be
set on mobile, by everywhere else, I think it's a "don't be stupid"
option.

I also think there should be a way to set it cleanly.

IMO, this is extremely important. Linux is a wildly important platform
for networking applications and Python, and, for better or worse,
Ubuntu is a very commonly used distribution.  Having asyncio, perform
so poorly on these platforms is a big deal.

Gaaa, forgot to set meta data.

See also https://github.com/python/asyncio/issues/286

I also wanted to raise this question.  I think transports should set NODELAY by default (as Golang, for instance).  I've been hit by this thing a few times already -- performance of protocols over TCP is terrible because somebody forgot to set this flag.

I forgot to switch to the asyncio branch of ZEO when testing on the Amazon/RedHat linux. When I do, the tests run slow there too. Asyncio is dog slow on every linux I've tried.

Fine with me -- I have no idea what this flag does (nor the desire to learn :-).

Jim, I can make a PR, unless you want to do that.

Yury, I'd be fine with you making a PR. :)

Is there a similar update that can be made to uvloop?

> Is there a similar update that can be made to uvloop?

Yes.  Once it's committed to asyncio, I'll add it to uvloop immediately.  That's one benefit of using uvloop -- it gets updates faster ;)

PR: https://github.com/python/asyncio/pull/373

I missed the point that you can get a transport's socket using get_extra_info. IMO, this provides an adequate escape from the default and qualifies as a "proper way to set it".

I still think the default should be to enable TCP_NODELAY.

vote +10 for that

New changeset 3f7e4ae9eba3 by Yury Selivanov in branch '3.5':
Issue #27456: asyncio: Set TCP_NODELAY by default.
https://hg.python.org/cpython/rev/3f7e4ae9eba3

New changeset 10384c5c18f5 by Yury Selivanov in branch 'default':
Merge 3.5 (issue #27456)
https://hg.python.org/cpython/rev/10384c5c18f5

Committed, should be in 3.6 b1.

Seems like this fix is incomplete. It contains this check:

    sock.type == socket.SOCK_STREAM

But sock.type is not only a type (at least in Linux and FreeBSD), it also may contain SOCK_NONBLOCK and SOCK_CLOEXEC flags. So I'm hitting the same problem: on the Linux in asyncio I have:

    > sock.type == socket.SOCK_STREAM | socket.SOCK_NONBLOCK == 2049
    True

So this check isn't working and TCP_NODELAY still disabled by default.

Links:
- http://man7.org/linux/man-pages/man2/socket.2.html
- https://github.com/torvalds/linux/blob/v4.13/include/linux/net.h#L77
- https://github.com/freebsd/freebsd/blob/stable/11/sys/sys/socket.h#L110

Linux has SOCK_TYPE_MASK definition equal to 0xf, but I can't find such definition in the FreeBSD sources. And I don't know how to reliably and with forward compatibility check sock.type without calling getsockopt() syscall.

Currently I have a fix in my project, where:

    _sock_type_mask = 0xf if hasattr(socket, 'SOCK_NONBLOCK') else 0xffffffff

And then in my own _set_nodelay(sock) function:

    sock.type & _sock_type_mask == socket.SOCK_STREAM

Should I make a pull request or someone knows more reliable check? Or it is ok to add one more syscall?

    sock.getsockopt(socket.SOL_SOCKET, socket.SO_TYPE) == socket.SOCK_STREAM

> Seems like this fix is incomplete. It contains this check:
>
>    sock.type == socket.SOCK_STREAM
>
> But sock.type is not only a type 

Good catch.  I made a PR to fix this.

> sock.type == socket.SOCK_STREAM

Oh, right. I wrote PR 4233 to fix the code.

Yury: "Good catch.  I made a PR to fix this."

Oops. If you wrote a PR, would it mind to compare it with mine, please?

_ipaddr_info() also uses "type == socket.SOCK_STREAM". Is it ok?

New changeset e796b2fe26f220107ac50667de6cc86c82b465e3 by Yury Selivanov in branch 'master':
bpo-27456: Ensure TCP_NODELAY is set on linux (#4231)
https://github.com/python/cpython/commit/e796b2fe26f220107ac50667de6cc86c82b465e3

New changeset 572636d42566da8eb6e85d5b8164e9ed8860a255 by Yury Selivanov in branch '3.6':
bpo-27456: Ensure TCP_NODELAY is set on linux (#4231) (#4898)
https://github.com/python/cpython/commit/572636d42566da8eb6e85d5b8164e9ed8860a255

New changeset a7bd64c0c01379e9b82e86ad41a301329a0775d9 by Yury Selivanov in branch 'master':
bpo-27456: Simplify sock type checks (#4922)
https://github.com/python/cpython/commit/a7bd64c0c01379e9b82e86ad41a301329a0775d9

Yury: thanks for fixing socket.socket.type. asyncio code now looks simpler and easier to follow without the helper functions just to test the socket type for equality.

Note, TCP_NODELAY can not be set on UNIX streaming socket. Do you have corresponding tests for UNIX sockets ?

stage: resolved -> patch review

stage: resolved

messages: + msg269831

components: + asyncio
type: performance