Why You Cannot Average p95 Latency Across Servers

The average of server p95 values is generally not the p95 of all requests. Each server percentile has already discarded most of the distribution information. Even weighting those percentiles by request count does not reconstruct the missing distribution.

To calculate a fleet percentile, combine compatible observations or an appropriate aggregate histogram first, then estimate the percentile. Prometheus’s histogram and summary guidance explains the relevant aggregation distinctions. Keep the request population, time window and bucket schema consistent before trusting the result.

Reproduce the problem with two servers

Use this invented nearest-rank example. Server A handles 100 requests, each taking 100 milliseconds. Server B handles ten requests, each taking 1,000 milliseconds.

Their individual p95 values are 100 ms and 1,000 ms. The simple average is 550 ms. A request-count-weighted average is about 181.82 ms. Neither is the fleet p95.

Across all 110 requests, the nearest-rank p95 position is ceiling(0.95 × 110), or position 105. The first 100 sorted values are 100 ms and positions 101 through 110 are 1,000 ms, so the combined p95 is 1,000 ms.

State the percentile convention when comparing exact small samples. Different interpolation conventions can differ on some datasets, but that does not make averaging precomputed quantiles a general solution.

Understand what a percentile loses

A p95 is one point in a distribution. It does not describe how many observations lie near it or how the remaining values are spread. Many very different datasets can share the same p95.

Request counts add one useful quantity but do not restore all the missing values. That is why weighting a per-server quantile by traffic still fails in the example.

The same issue appears when averaging hourly p95 values into a daily p95 or combining regional percentiles into a global one. Changing the grouping label does not change the underlying mathematical limitation.

Combine the right data before the quantile

If raw observations are available and appropriate to retain, calculate the percentile over the combined population. For operational metrics at scale, histograms offer an aggregate representation of the distribution without storing every individual request.

For classic Prometheus histograms, an illustrative fleet query is:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

This assumes a compatible classic histogram metric and the intended population of series. Add the necessary job, route or environment filters for your actual scope. Do not copy a fleet-wide query into a dashboard that claims to describe only one service.

Keep bucket schemas compatible

Classic histogram aggregation depends on meaningful common bucket boundaries. If instances expose different schemas, summing identically named boundaries may produce a distribution that does not represent the full population consistently.

Review instrumentation and rollout behavior before changing buckets. During a mixed-version deployment, a dashboard can look continuous while combining incompatible measurements.

Native histograms have different representation and query considerations. Use the current documentation for the type you actually ingest rather than applying classic _bucket syntax to every metric with “histogram” in its description.

Interpret the estimate near important thresholds

A histogram-derived quantile is an estimate whose accuracy depends on the representation and distribution. A wide bucket around an important latency threshold can limit what you can conclude.

If the decision is whether requests meet a particular objective, a direct fraction of observations below that threshold may be more useful than a single estimated percentile, provided the relevant boundary and population are represented correctly.

Do not display many decimal places as though they establish measurement precision. Explain the bucket or resolution limitations and inspect the surrounding distribution when the result is close to a decision boundary.

Define the population honestly

Choose whether latency includes failures, cancellations, retries and queue time. A server-only completion metric can exclude the requests users abandoned or the time spent before the server began work.

Keep units and windows explicit. Mixing milliseconds and seconds, or combining a five-minute rate with a daily label, can create a more serious error than the percentile aggregation itself.

Separate low-traffic results from missing data. A percentile based on very few observations can jump sharply, while no observations do not mean zero latency. The dashboard should preserve that distinction.

Validate the dashboard with a fixture

Retain the two-server dataset as a small test explaining why the old averaging query is wrong. Then check the actual metric query against a controlled workload whose distribution is known.

Review the result after adding an instance, changing traffic balance and deploying an instrumentation update. These changes can expose an accidental grouping or population mismatch even when the quantile function itself is correct.

Confirm that the label shown in the dashboard matches the query grouping. A panel titled global p95 should not quietly calculate one percentile per instance and let the visualization average the returned series.

Conclusion

Aggregate distributions before calculating a percentile. Keep compatible measurements, explicit populations and realistic precision so a fleet p95 describes requests rather than an average of already-compressed summaries.