HTTP/2 POST loop fails with ENHANCE_YOUR_CALM after 40701 iterations

  • Version: v10.10.0
  • Platform: Linux david-Latitude-E6440 4.15.0-34-generic logo ideas #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Subsystem: http2

Here's a simple server app which reads HTTP/2 POST request bodies to their end and then responds:

const http2 = require('http2');
const fs = require('fs');
const path = require('path');
const { Writable } = require('stream');

const server = http2.createSecureServer({
    key: fs.readFileSync(path.join(__dirname, 'server.key')),
    cert: fs.readFileSync(path.join(__dirname, 'server.crt'))
});

server.on('session', function (session) {
    session.on('stream', function (stream, headers) {
        stream.on('end', function () {
            this.respond({
                ':status': 200,
                'Access-Control-Allow-Origin': 'http://localhost:8000'
            }, {
                endStream: true
            });
        });
        stream.pipe(new Writable({
            write: (chunk, encoding, cb) => cb()
        }));
    });
});

server.listen(7000, function () {
    console.log('READY.');
});

For this test, the cert and key are self-issued and my browser trusts them.

Here's a Web page which makes requests in series to the server:

<html>
<head>
<script>
async function test() {
  while (true) {
    response = await fetch('https://localhost:7000', { method: 'POST' });
    await response.arrayBuffer();
  }
}
</script>
</head>
<body onload='test()'>
</body>
</html>

I expect this to continue indefinitely.

However, what happens is I get the following error after 40701 iterations:

memtest.html:6 POST https://localhost:7000/ net::ERR_SPDY_PROTOCOL_ERROR

I did some debugging using Wireshark and found the error was ENHANCE_YOUR_CALM.

After adding some tracing to node_http2.cc, I found the error being produced here: https://github.com/nodejs/node/blob/v10.10.0/src/node_http2.cc#L863

I've tracked this down to current_nghttp2_memory_ continually growing.

The cause for this is nghttp2 allocation of 232 bytes for each stream (https://github.com/nodejs/node/blob/v10.10.0/deps/nghttp2/lib/nghttp2_session.c#L1029) is never being released.

This is because nghttp2 keeps closed streams around (up to the concurrent connection limit).

If I add the following line to Http2Options::Http2Options in src/node_http2.cc:

nghttp2_option_set_no_closed_streams(options_, 1);

then the test works as expected and doesn't fail at 40701 iterations.