Compact models to evaluate
A compact open-model shortlist spanning Gemma, MiMo, Qwen, and Llama, with checkpoint-size caveats and practical deployment evaluation questions.
- What is included
- Six selected compact or edge-focused profiles: four current candidates and two older baselines. Inclusion reflects the recorded model size or publisher positioning, not a verified single-GPU fit.
- How this list is ordered
- Alphabetical by model name. This is an editorial shortlist, not a quality, latency, or GPU-memory ranking.
Publisher sources are linked for every model. No independent performance scores have been measured. On smaller screens, scroll the table horizontally.
| Model | Published size | Context | Deployment note |
|---|---|---|---|
| 11.95B | 256K | A compact unified multimodal candidate; review each input modality separately.Compare model | |
| 8B+ | 128K | Edge-focused; effective parameters omit part of the full weight scope.Compare model | |
| ~8B | 128K | An established baseline; inspect the checkpoint's access and license terms.Compare model | |
| ~9B | Under review | The distilled 9B checkpoint, distinct from the large Flash and Pro releases.Compare model | |
| 8.2B | 32K native | An older baseline with a recorded parameter count and weight arithmetic.Compare model | |
| ~9B | Under review | A compact vision-language candidate for a controlled application test.Compare model |
Define compact in terms of the job
A smaller nominal model can be a useful starting point, but the relevant budget includes serving state and the inputs your application accepts. Decide whether the first experiment needs text only, images, or audio. A model outside that modality scope is not a substitute merely because its name contains a smaller number.
Do not equate effective size with total storage
Some model names describe effective parameters or a rounded backbone count. Gemma E4B is an example where the full scope requires additional attention. Our profile notes explain the published boundary; pending memory estimates remain pending here.
Use the older entries as controls
Keep a reproducible baseline while testing newer candidates. Use the same prompt set, quality threshold, and output budget. Only after the checkpoint runs successfully under representative load should it receive a minimum or recommended GPU configuration.
Before you choose hardware
4-bit vs. 8-bit LLM inference: what actually changes? explains the next planning step. For the distinction between publisher specifications, calculations, and measurements, read the BenchGrid methodology.