Measuring tail latency in a small cluster
Almost every dashboard shows an average, and almost every average is useless for latency. A service with a 12 ms mean can still feel terrible, because the number people actually experience is the slow one they happened to hit, not the mean of everyone's requests.
I spent a week last spring chasing a complaint that a service was "slow" while every graph said it was fine. The mean was flat. The p99 was flat. The problem was in the p99.9, which nobody had plotted, and it was concentrated in one code path.
Why percentiles compress badly
A percentile is a summary, and summaries discard the shape of the distribution. Two services can share an identical p99 and behave completely differently: one has a smooth right tail, the other has a step where a subset of requests waits on a lock. The percentile does not tell you which you have.
What you want instead is the tail itself, at several resolutions. Plot a histogram on a log scale, and plot the maximum alongside the percentiles. When the maximum and the p99.9 diverge, something is periodically stalling.
Coordination omission
The other trap is measuring only where you can. If a request fans out to twelve backends and waits for all of them, the latency of the frontend is the maximum of twelve draws, not the average. Each backend can look healthy while the composite is slow.
The fix is to measure at the edge, where the user's request actually completes, and to record the full duration rather than the internal segments. Segment timings are useful for attribution, but they cannot tell you the composite latency, because they are recorded on the wrong side of the fan-out.
What to record
- The full request duration at the boundary, before any internal queueing.
- A histogram with enough resolution in the tail. Linear buckets are wrong here; use log-spaced ones.
- The count of in-flight requests at the moment of measurement, so you can separate queueing from service time.
- The maximum, not just percentiles. It is the cheapest signal that something changed.
Reading the result
Once you have the histogram, the shape tells the story. A tail that rises with load is queueing. A tail that is flat regardless of load is a fixed cost, a synchronous call to something slow, or a periodic stall like a garbage collection pause or a cache refresh.
If the max is more than an order of magnitude above the p99, you do not have a latency problem, you have an occasional stall. Find the period, not the path.
That one observation, plotting the max next to the percentiles, is usually enough to name the culprit within an afternoon.