Start with the weights.
A simple lower-bound estimate is total parameters × bytes per parameter. At 16-bit precision, a nominal 8-billion-parameter model stores roughly 16 GB of weights. At 8-bit, the arithmetic gives 8 GB; at 4-bit, 4 GB. These are decimal gigabytes, not GiB, and they do not include quantization scales, extra modules, or runtime allocations.
Leave room for the conversation.
The KV cache stores attention state for the sequences being served. Its footprint depends on the architecture, cache precision, sequence lengths, and concurrent requests. Increasing context or concurrency can consume the memory that appeared to be spare. Hybrid and sliding-window attention also change the calculation, so one generic percentage is not a reliable fit test.
Active parameters are not resident parameters.
Mixture-of-experts models activate only a subset of their weights for each token. That reduces some computation; it does not remove the other experts from a fully resident deployment. Use total parameters when estimating model weight storage. Offloading changes the memory and transfer trade-off and needs a separate measurement.
A smaller format is a different configuration.
An arithmetic 4-bit estimate does not establish that a compatible, accurate, and fast 4-bit checkpoint exists. Verify the quantization format and runtime kernels. Then check the output quality on representative tasks before choosing it solely for its smaller memory footprint.
Turn an estimate into a deployment test.
Choose a supported checkpoint, set a representative context limit, and start at low concurrency. Record allocated memory and failures, then increase the load while watching latency. Only a successful test of that exact configuration can turn a memory estimate into evidence that the deployment works.
Put the context to work.
Explore the models, compare the fundamentals, and start with a clearer picture.
Compare models