Name the part of inference that benefits.
vLLM's automatic prefix caching reuses cached attention state for shared prompt prefixes. It can reduce repeated prompt processing, such as multiple questions about the same long document. The documented benefit concerns prefill; it does not by itself accelerate the generation of new output tokens. Keep time to first token separate from output-generation speed when explaining a result.
Design three workloads instead of one repeated prompt.
Our suggested comparison includes unique prefixes, deliberately shared prefixes, and a replay of an application-like mix. Keep the lengths and generation settings comparable, and report the actual cache-hit behavior where the runtime exposes it. The shared-prefix case is useful evidence for a document workflow; the unique-prefix case answers a different question. Neither should stand in for the other without explanation.
Make the cache state reproducible.
Document whether a process was restarted or cache state was reset, how warmup was performed, and which requests populated the cache. Model warmup and prefix warmup are distinct: preparing kernels need not mean priming the exact evaluation prompts. In a proposed experiment, we would save the prompt ordering and run each scenario several times. Another operator should be able to reconstruct what state existed before measurement began.
Interpret a speedup within its workload.
Suppose a synthetic test repeatedly asks questions about one shared document. A warm run may be representative of that document being popular, but says little about a service receiving unrelated uploads. Report both cases and retain failed requests, timeouts, measurement duration, and latency percentiles. BenchGrid has not yet measured a cache speedup; this note describes how we plan to avoid confusing reuse with raw inference performance.
Put the context to work.
Explore the models, compare the fundamentals, and start with a clearer picture.
Compare models