Time to first token#
TTFT includes queuing and prefill. It determines how long a user waits before seeing output.
Tokens per second#
Decode throughput determines how quickly the answer continues. Aggregate server throughput is not the same as per-user generation speed.
Publish the workload#
Input length, output length, concurrency, GPU and region belong beside every number. A fast short-prompt result does not predict a long-context RAG request.