How we got this number
Each generated token reads the whole model from memory once: 7.1 GB. The NVIDIA DGX Spark (128GB) moves 273 GB/s, so the ceiling is 38 tokens/sec. Real machines reach a fraction of that ceiling; we use 70%, an assumption until we get measured runs for this kind of machine. Send us your real numbers and we'll replace the estimate.
Faster machines for Mistral nemo 12B
Other models on the NVIDIA DGX Spark (128GB)
| lfm 2.5 8B (A1B) | Fast | 120+ tok/s |
| minicpm v4.6 1B | Fast | 119 tok/s |
| Gemma 2 2B | Fast | 119 tok/s |
| codeGemma 2B | Fast | 119 tok/s |
| Gemma 2B | Fast | 112 tok/s |
| smollm 2 1.7B | Fast | 106 tok/s |