LLM Latency Metrics: TTFT, Token Throughput, and the User's Wait

A model can generate tokens quickly and still feel slow. The user may wait several seconds before the first visible output, or receive a fast opening followed by long pauses. A single tokens-per-second figure hides those differences.

Decide the acceptable experience for each feature before testing. A background classification job and an interactive writing assistant do not need identical response pacing. Record the user's practical deadline so optimisation serves the feature rather than a dashboard number. If the user needs a short decision quickly, a faster first useful answer may matter more than the completion speed of an unnecessarily long response.

Useful LLM latency measurement follows the request from the user's action to the completed answer. Separate startup delay, output pacing, and total duration, then test the workload your product actually serves. This guide uses synthetic arithmetic examples rather than claiming benchmark results for any model.

Define the timestamps first

Record when the client sends the request, when it receives the first meaningful output, and when the response completes. Decide whether connection setup and queueing are included. A server-side metric and a browser-side metric may use different starting points.

Time to first token, often shortened to TTFT, describes the initial wait according to the measurement boundary. Total latency describes completion time. Output pacing can be measured between streamed events, but an event may contain more than one token.

vLLM's production metrics documentation lists latency and request measurements for that serving system. Check the installed version's definitions instead of assuming identically named metrics mean the same thing across runtimes.

Separate token counts from stream events

A stream parser receives bytes and events, not a guaranteed one-token-per-message sequence. If you time every event and call the result “inter-token latency,” you may be measuring transport batching instead of model token generation.

Use tokenizer or server usage information where available, and document the counting method. If the provider doesn't expose reliable token counts, report event pacing honestly. Don't create a precise throughput claim from an approximate character count.

The vLLM metrics design documentation distinguishes streamed-event intervals from per-request time-per-output-token calculations. That distinction is useful when comparing dashboards that appear to disagree but actually describe different boundaries.

Use a worked example carefully

Suppose a fictional request starts at zero, receives its first output at 1.2 seconds, and finishes at 5.2 seconds with 101 output tokens. A simplified average decode interval is (5.2 - 1.2) / (101 - 1), or 0.04 seconds per token.

That is an arithmetic illustration, not an observed model result. It doesn't reveal whether output arrived evenly or in bursts. It also doesn't include any application work after the stream finished. Preserve the timeline alongside the summary metric.

For sanitised benchmark records, JSON Formatter can help inspect timestamp and count fields. It won't measure a model or verify the timing boundary. Keep clocks and units explicit in your test harness.

Build representative request groups

Test short prompts, long prompts, short answers, and long answers separately. Include the tools, retrieval, and preprocessing that the real product uses. A bare inference benchmark won't describe a workflow that performs several network calls before generation.

Record concurrency and arrival pattern. One request on an idle server can perform differently from many requests arriving together. Test a normal load and a realistic busy period, with bounded resource use in an environment you control.

Include cache state when relevant. A warm model and a cold startup are different user experiences. If prompt caching affects performance, record whether the request qualified rather than assuming every repeated prompt receives the same benefit.

Look at distributions, not only averages

Report median and slower-end observations for each workload group. A good average can hide a small number of very slow requests that users notice most. Keep enough sample context to distinguish a stable pattern from a one-off network problem.

When aggregating histogram metrics, use the monitoring system's supported calculation rather than averaging percentiles from separate machines. A percentile describes a distribution; averaging two percentile values doesn't reconstruct the combined distribution.

Use CSV Viewer for a sanitised summary containing group, concurrency, sample count, TTFT, and completion duration. Keep the raw measurements in an approved system so someone can reproduce the calculation.

Check quality alongside speed

A faster deployment isn't an improvement if it truncates useful answers or changes task accuracy. Preserve the same model settings and evaluation tasks when comparing performance, then record any deliberate differences such as quantisation or output limits.

Also inspect cancellation and failures. A request that never completes shouldn't disappear from the benchmark. Report rejected, cancelled, and failed requests separately, with clear reasons where available. Otherwise the fastest successful subset can make an unreliable service look excellent.

For the transport layer, see JSON Lines versus SSE parsing. For input selection, context engineering helps explain why unnecessary context can affect the work the system must perform.

Conclusion

Measure the wait users experience, not just the generation rate a server advertises. Define boundaries, distinguish tokens from events, use representative load, and preserve quality checks. TTFT, pacing, and completion time together produce a much more useful performance picture.