Use the arithmetic as a lower bound.
For an illustrative 12-billion-parameter model, raw weight storage is 24 GB at 16 bits, 12 GB at 8 bits, and 6 GB at 4 bits. Divide parameters × bits by eight to obtain bytes. This example uses decimal GB and assumes every parameter uses the same precision. Actual files can include scales, unquantized layers, and additional modules; serving also needs working memory and attention state. A 6 GB arithmetic result is not a recommendation to buy a 6 GB GPU.
Name the format, not just the bit width.
INT8 and FP8 use the same number of bits but represent values differently. A checkpoint's quantization scheme and the inference engine's implementation determine which hardware can execute it. vLLM publishes a compatibility table by implementation and GPU architecture. Check the exact combination, including your installed version, before comparing prices. Record the checkpoint revision, format, GPU, runtime, and cache precision together so a later test refers to the same configuration.
Set a quality gate before measuring speed.
Create a small evaluation set from the job your application performs: valid structured responses, correct answers to document questions, or passing code tests. Run the same set on the reference and candidate checkpoints with the same generation budget. Review failures by category instead of accepting a single average score. Our suggested decision rule is to choose an acceptable error threshold before seeing the results, then reject configurations that fail it even if they use less memory.
Measure the configuration you will serve.
Keep prompt lengths, output limits, cache policy, and arrival rate consistent. Report latency and successful output throughput alongside memory. A format that makes a model fit may still miss your latency target. Conversely, extra memory headroom can be useful for more concurrent requests. BenchGrid's precision tabs show weight-storage arithmetic only; they do not assert that a compatible checkpoint exists or that we have tested its quality.
Put the context to work.
Explore the models, compare the fundamentals, and start with a clearer picture.
Compare models