Know exactly what was running.
Look for the model revision, weight precision and quantization format, runtime version, driver, GPU model and count, instance type, and parallelism settings. The same model name on the same GPU family is not enough to identify a reproducible setup.
Ask what the requests looked like.
Input and output lengths, sampling parameters, concurrency, and arrival rate all affect the result. Fixed-length synthetic prompts are useful for controlled comparisons, but they do not capture a production distribution by themselves. Reasoning output and multi-turn histories can change the load substantially.
Check the cache policy.
Repeated prompts can reuse prefix state, so later runs may do less work than earlier runs. A realistic shared-prefix workload is valid if it is intentional and labeled. An accidental warm cache should not be presented as an uncached result. Record cache settings and the reset or seeding procedure.
Read failures alongside successes.
A high throughput number is less helpful if a meaningful fraction of requests failed or exceeded the latency budget. Record attempted and completed requests, errors, timeouts, warmup, measurement duration, and repeat variability. Tail latency needs enough samples to be informative.
Look for the raw evidence.
A useful report includes the configuration, invocation, machine-readable results, and enough detail to rerun the test. Keep measured performance separate from publisher claims and extrapolated prices. When a result is not available yet, leaving the field empty is more honest than filling in an attractive estimate.
Put the context to work.
Explore the models, compare the fundamentals, and start with a clearer picture.
Compare models