A short note on congestion control
Congestion control answers one question: how fast should I send, given that I do not know the capacity of the path in front of me. The two algorithms most likely to be running on your servers give different answers, and the difference shows up in specific situations.
CUBIC reacts
CUBIC, the Linux default for years, treats packet loss as the signal of congestion. It grows the send window until something is dropped, then backs off and grows again along a cubic curve. On a clean path with shallow buffers this works well and is fair between competing flows of similar round-trip time.
Its weakness is that loss is not always congestion. On a link with deep buffers, loss arrives late, after a queue has already formed, so the algorithm spends its time keeping that queue full. On lossy wireless, it mistakes random loss for congestion and backs off when it did not need to.
BBR measures
BBR, from Google, tries to estimate the bottleneck bandwidth and the round-trip time directly, then sends at a rate near the estimate. It does not require loss as its primary signal, so it keeps queues short and behaves better on paths where loss is not congestion.
The tradeoff is that it is more aggressive toward competing flows, particularly flows running a loss-based algorithm, and it can be unfair in that mixed case. It also depends on its estimates being accurate; a path with heavy aggregation can mislead it.
When the choice matters
- High-latency paths. Loss-based algorithms recover slowly when round-trip time is large, because feedback arrives late. BBR's direct estimation helps most here.
- Lossy links. If loss is not congestion, using it as a signal is a mistake. BBR avoids the confusion.
- Shallow buffers. With small queues, CUBIC is already fine; there is not much headroom left to recover.
- Mixed fleets. If your traffic competes with loss-based flows on shared paths, the fairness question is real and worth measuring rather than assuming.
A practical default
On a server serving interactive traffic over long paths, BBR with a fair queue is a reasonable default and is easy to reverse. The measurement to make before and after is not throughput but the tail of the latency distribution under load, which is exactly what changes.
If you change nothing else, change that, and measure the tail.