Allow HTTP client to send other requests if no other clients are waiting

Basic Infos

This issue complies with the issue POLICY doc.
I have read the documentation at readthedocs and the issue is not addressed there.
I have tested that the issue is present in current master branch (aka latest git).
I have searched the issue tracker for a similar issue.
If there is a stack dump, I have decoded it.
I have filled out all fields below.

Platform

Hardware: ESP8266MOD (ESP-12E I think)
Core Version: 52f4cc8 (June 29th 2020)
Development Env: Arduino IDE (v1.8.12)
Operating System: Arch Linux

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E Module)
Flash Size: 4MB
lwip Variant: v2 Lower Memory
CPU Frequency: 160MHz
Upload Speed: 921600

Problem Description

The HTTP server always closes connections to it after 2s (HTTP_MAX_CLOSE_WAIT) and never accepts many requests from the same socket. It always returns the header Connection: close to tell the client to close the connection.

This is extremely inneficient because the clients must establish a new TCP, and possibly a new TLS session, for each request. It is understandable considering that the server cannot handle multiple requests in parallel, so you don't want a single client to connect to the server and send requests to it forever and block all the other clients.

The image below shows just how inneficient this is by showing a connection to an HTTPS server on a NodeMCU running at 160MHz with an encryption key of 512bits.

This gets even worse when the encryption key is 2048bits as you can see below.

These two images show how most of the delay is caused by the TLS setup is. This delay can get worse because the TLS setup is also highly dependent on the latency because there's a lot of back and forth.

By reusing the same connection for multiple subsequent requests, we can get rid of the TLS setup for all the subsequent requests and see a great performance boost. There will also be a performance boost for unencrypted requests, but it won't be as significant.

To prevent a client to block all the other clients, I suggest to only allow the current client to stay connected if no other clients are waiting.

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WebServerSecure.h>
#include <ESP8266WiFi.h>

static const uint8_t PRIVATE_KEY[] = R"EOF(
-----BEGIN RSA PRIVATE KEY-----
MIIBPAIBAAJBAMaLGUw9UMkni86+fipZS3zoJwza4/nDJkOQeC8M31yb35fISva6
4d2K1HLMIBl4ViaNSd1RElzRJifSy2bIdcMCAwEAAQJBAIaD44Xl3QAMTQqrwWsL
yLs9xodNHjwv3ZLVJLgr7oEc3yUeCv4q28AwlYDOO04OoT53GAS3m1qYv4FG7jox
VaECIQD6/9e2s4WqUpOrmFdRz3AMZx5LbzMrfwYO/yEztNoLCQIhAMp/t8DxqVK9
Cu/h03x/gIal9alpv6uD3MRU4MfC6FFrAiEA9/01NQcUJl8mFaETjPoF+8saTG+W
v//ljYWXWU3zLHkCIHe3RBJsjHcezg19i8Npublg+jhbDXa/8U+dAnr27tPbAiEA
uysakbOSKKk6NyF7zujFEu4yhgnoqrVwYb78RunsVSM=
-----END RSA PRIVATE KEY-----
)EOF";

static const uint8_t CERTIFICATE[] = R"EOF(
-----BEGIN CERTIFICATE-----
MIICBTCCAa+gAwIBAgIUPtU58ibKekgnRomdBcIJIdYUSZ4wDQYJKoZIhvcNAQEF
BQAwVzELMAkGA1UEBhMCQ0ExEjAQBgNVBAgMCVF1w4PCqWJlYzERMA8GA1UEBwwI
R2F0aW5lYXUxITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0x
OTA4MjMwMjMzNDhaFw00OTA4MTUwMjMzNDhaMFcxCzAJBgNVBAYTAkNBMRIwEAYD
VQQIDAlRdcODwqliZWMxETAPBgNVBAcMCEdhdGluZWF1MSEwHwYDVQQKDBhJbnRl
cm5ldCBXaWRnaXRzIFB0eSBMdGQwXDANBgkqhkiG9w0BAQEFAANLADBIAkEAxosZ
TD1QySeLzr5+KllLfOgnDNrj+cMmQ5B4LwzfXJvfl8hK9rrh3YrUcswgGXhWJo1J
3VESXNEmJ9LLZsh1wwIDAQABo1MwUTAdBgNVHQ4EFgQUTNUSlxtCEYth3de5ciL6
qiDdpZcwHwYDVR0jBBgwFoAUTNUSlxtCEYth3de5ciL6qiDdpZcwDwYDVR0TAQH/
BAUwAwEB/zANBgkqhkiG9w0BAQUFAANBAEqW+lzyWOT8cA+dVRwW+BkHguxR1ev6
zYHQwup2cIEwXeArBptlX0wkdjb4bGtwWM1NiqtCHCeCXyQhuPdMCLE=
-----END CERTIFICATE-----
)EOF";

#define WIFI_SSID "your-ssid"
#define WIFI_PASSWORD "your-password"

ESP8266WebServerSecure server(443);

void setup() {
    Serial.begin(115200);
    Serial.setDebugOutput(true);
    Serial.println();

    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
    uint8_t i = 0;
    while (WiFi.status() != WL_CONNECTED) {
        i++;
        if (i == 21) {
            WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
            Serial.println("Could not connect to " WIFI_SSID);
        }
        delay(500);
    }
    Serial.print("Connected! IP address: ");
    Serial.println(WiFi.localIP());

    server.getServer().setServerKeyAndCert(PRIVATE_KEY, sizeof(PRIVATE_KEY),
                                           CERTIFICATE, sizeof(CERTIFICATE));
    server.on("/", []() {
        server.send(200, "text/plain", "Hey!\r\n");
    });
    server.begin();
}

void loop() {
    server.handleClient();
}

Benchmarking the solution

I've created a script in ruby that should be helpful to benchmark the solution:

require 'net/http'
require 'benchmark'

DOMAIN = "<your controller's IP>"
TIMES = 100

uri = URI("https://#{DOMAIN}/")

request = Net::HTTP::Get.new(uri)

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# Allow self signed certificates
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

Benchmark.bm(12) do |bm|
  request["Connection"] = "keep-alive"
  bm.report("keep-alive:") {
    http.start do |http|
      TIMES.times do |_|
        response = http.request(request)
      end
    end
  }

  request["Connection"] = "close"
  bm.report("close:") {
    TIMES.times do |_|
      http.start do |http|
        http.get(uri.path)
      end
    end
  }
end

If you change the DOMAIN variable to google.com, the result is:

                   user     system      total        real
keep-alive:    0.087922   0.011206   0.099128 (  3.915354)
close:         0.292852   0.128215   0.421067 (  8.163557)

We can see the performance gained by keeping the connection alive.

If you change the DOMAIN variable to the IP of your controller, the result is:

                   user     system      total        real
keep-alive:    0.323017   0.090638   0.413655 ( 29.205106)
close:         0.274748   0.070732   0.345480 ( 28.794096)

We can see that there is currently no performance gain by keeping the connection alive.

Debug Messages

n/a