Fast (20+ tok/s)

Qwen3.5 0.8B 80DeepSeek-R1 1.5B 73Qwen 1.8B 73lfm 2.5 8B (A1B) 65Llama 3.2 1B 62Qwen 3 1.7B 57Granite 3.1 moe 1B 57minicpm v4.6 1B 50Gemma 2 2B 50codeGemma 2B 50Gemma 2B 47smollm 2 1.7B 44starCoder 2 3B 44falcon 3 1B 44Qwen2.5 3B 42Qwen2.5 Coder 3B 42Qwen3.5 2B 42Qwen 3-VL 2B 42Llama 3.2 3B 40orca mini 3B 40Granite 3.1 moe 3B 40falcon 3 3B 40Granite 4.1 3B 38Granite 4.2 3B 36Phi 3 3.8B 36Qwen 4B 35Qwen 3 4B 31Qwen2.5-VL 3B 25Gemma 3 4B 24Qwen 3-VL 4B 24translateGemma 4B 24Qwen3.5 4B 24Llama 2 7B 21Code Llama 7B 21DeepSeek Coder 6.7B 21Llama 2 uncensored 7B 21

Comfortable (10-20 tok/s)

orca mini 7B 20Mistral 7B 18Qwen2 7B 18starCoder 2 7B 18OLMo 2 7B 18falcon 3 7B 17DeepSeek-R1 7B 17Qwen2.5 7B 17Llama 3 8B 17Qwen2.5 Coder 7B 17LLaVA 7B 17Llama 3.1 8B 16Qwen 7B 16dolPhin 3 8B 16Gemma 7B 16codeGemma 7B 16Qwen 3 8B 16DeepSeek-R1 8B 15Granite 4.2 8B 15Granite 4.1 8B 15Gemma 2 9B 15minicpm v 8B 15LLaVA Llama3 8B 15ornith 9B 14Gemma 3n E2B 14Qwen2.5-VL 7B 13minicpm v4.5 8B 13Qwen 3-VL 8B 13falcon 3 10B 13ornith 1.5 9B 12Qwen3.5 9B 12Mistral nemo 12B 11Gemma 4 E2B 11Llama 2 13B 11Code Llama 13B 11Gemma 3n E4B 11Gemma 4 12B 11Llama 3.2 Vision 11B 10Phi 3 14B 10orca mini 13B 10LLaVA 13B 10

Slow but usable (5-10 tok/s)

Gemma 3 12B 9.9translateGemma 12B 9.9Qwen 14B 9.8OLMo 2 13B 9.5DeepSeek Coder v2 16B 9.0DeepSeek-R1 14B 8.9Qwen2.5 14B 8.9Qwen2.5 Coder 14B 8.9Phi 4 14B 8.8Qwen 3 14B 5.2Gemma 4 E4B 5.0

Too slow (<5 tok/s)

starCoder 2 15B 4.8

Won't fit

Mistral small 22Bgpt-oss 20BMistral small 24BMistral small3.2 24BGemma 2 27BGranite 4.1 30BGemma 3 27BQwen3.5 27BQwen3.6 27BtranslateGemma 27BGranite 4.2 30BQwen3.8 27Bmuse glimmer 30BQwen 32BQwen 3 30B (A3B)Gemma 4 26B (A4B)Qwen3 Coder 30B (A3B)Code Llama 34BDeepSeek Coder 33BDeepSeek-R1 32BQwen2.5 32BQwen 3 32BGemma 4 31BQwen2.5 Coder 32BLLaVA 34BQwen 3-VL 30B (A3B)qwq 32Bornith 35BQwen 3-VL 32BQwen2.5-VL 32Bornith 1.5 35BQwen3.5 35B (A3B)Qwen3.6 35B (A3B)Nemotron 3.5 lightning 30B (A3B)mixtral 8X7BNemotron 3 33BLlama 2 70BCode Llama 70Borca mini 70BLlama 2 uncensored 70BLlama 3 70BQwen 72BQwen2 72BLlama 3.1 70BDeepSeek-R1 70BLlama 3.3 70BQwen2.5 72BQwen2.5-VL 72BLlama 3.2 Vision 90BQwen 110Bgpt-oss 120BMistral medium 3.5 128Bmixtral 8X22BQwen3.5 122B (A10B)Nemotron 3 super 120B (A12B)Qwen3.8 flash next 125B (A6B)