What

After #14335 primary and replica clients are now handled by IO-threads. That frees up the main thread during replication and allows us to introduce compression over the replication stream.

This feature introduces zstd compression for replication. It only works when IO-threads are enabled so not to bog-down the main thread with compressing/decompressing data.

Why

One can enable compression when latency is not that important for replication (i.e replica will
become eventually consistent with the primary, just that "eventually" will be later), but the network traffic will be massively lowered, especially in a multi-replica scenario in high-write environment. Although not tested - another possible benefit is the lowered consumption of kernel buffer space.

How

We introduce two new configs:

• repl-compression
• compression-max-latency
  Explained below.

---

Compression is enabled via the repl-compression <N> config where N is a compression level. This value is 1:1 with zstd's API which expected level 1-22. Both primary and replica clients must have the same value for repl-compression and both must have enabled IO-threads for compression to be initialized.

Implementation-wise we create compressionConnection which wraps the current connect and "intercepts" the read/write kernel calls. When primary sends data from the replication buffer the compressionConnection actually first compresses data in zstd's internal user-space buffers before actually sending it on the socket. Similarly replica client reads the compressed data, decompresses it via zstd and only then returns it to the client's querybuffer.

Since zstd may buffer data in its internal buffers (both in compress and decompress cases) we have the other config compression-max-latency - this is a period in milliseconds on which the primary flushes the internal zstd buffers. This lowers latency but also decreases compression ratio. The default value of 100ms shows a good trade-off but users might want to experiment with different settings. Note that replica client doesn't use this config, since after zstd flushes its buffers it appends the compressed data with end frame message, so the replica automatically knows it doesn't need to wait for more data before decompressing.

Notes on the implementation

• Currently we don't use zstd's dictionary feature. For small data it may be beneficial as the replication stream uses the RESP protocol (i.e a lot of repeating character patterns), although for big random data we may not see such benefit as the resp bytes can be spaced out a lot. This may be added in the future but it might take time to experiment with it so I decided to go ahead with the PR.
• Compression functionality is carefully abstracted (both zlib and zstd were supported but we left only zstd for the superior functionality). Thus we may introduce other compression algorithms in the future with ease.
• The implementation is mostly independent of the type of clients so we might enable client compression for all client traffic in the future. That would, of course, mean support from client libraries and the benefits are questionable.
• Compression is almost transparent in the code. F.e connRead and connWrite completely hide if we read/write compressed data. Sadly we need to track both un/decompressed and compressed data size (mostly for stats) so there are a couple code smells like connCheckLastRead which returns the number of compressed bytes read just after connRead. I'm open to better ideas.

---