TLDR: We reduced Corrosion’s broadcast bandwidth from ~10 GiB/s to ~900 MiB/s by introducing message compression and replacing our gossip-based broadcast protocol with Plumtree.
Compression
Previously, Corrosion sent out every change uncompressed to other nodes, but changes a fair amount of repeated data like column names associated with each row in a change and site_id for a change. Introducing compression with custom dictionaries trained on change data took the bandwith down from ~10GiB/s to ~5Gib/s
(Thanks to Adam for suggesting zstd custom dictionaries that really made a difference)
Broadcast Protocol
Corrosion previously used a gossip-based broadcast protocol, we’d select a number of random nodes and send them changes, repeating this process several times. This ensured that changes eventually reached almost every node, but it also meant repeatedly sending changes to nodes that had already received them..
We’ve introduced a new broadcast protocol - Plumtree which maintains a set of eager peers that a node uses to broadcast messages, along with a lazy set. Eager peers get the full message payload while we’d only send a smaller message ID. If a lazy peer doesn’t receive the corresponding message within a timeout, it can request the message from the sender. Once a peer requests a message, it gets promoted to the eager set.Turning on plumtree took the broadcast bandwith from ~5GiB/s to ~900MiBs.
(Thanks to @gorbak25 for his reviews and suggestions!)
Latency also improved as nodes spend less time receiving more changes and then discarding changes they’ve already seem.
