The first token is a separate wait.
Time to first token (TTFT) measures the delay from a request until its first output token arrives. It can include network delay, queueing, and prompt processing, depending on the measurement boundary. Always check whether a result was measured at the client or inside the server.
Generation speed describes the next part.
Inter-token latency describes the spacing between generated tokens. Time per output token is usually an average over generation after the first token; it should not be confused with the tail of the inter-token latency distribution. Averages can hide stalls that a person notices immediately.
Throughput belongs to the whole system.
System output throughput counts the output tokens generated across all requests per unit time. Increasing concurrent work can improve utilization, but it can also increase queueing and slow individual requests. Keep per-user output speed and aggregate server throughput in separate columns.
Compare inside an acceptable experience.
Define the latency targets first, then compare the capacity achieved while staying within them. For a conversational workload, that might include P95 TTFT and an output-speed threshold. The right targets depend on the application; a batch pipeline can reasonably choose different constraints.
Cost needs the same boundary.
Dividing an hourly instance price by measured output throughput gives a test-specific compute cost. It assumes that workload and utilization. Real production cost also includes idle capacity, redundancy, failed requests, storage, and traffic. State whether tokens mean input, output, or a combination.
Put the context to work.
Explore the models, compare the fundamentals, and start with a clearer picture.
Compare models