Fast (20+ tok/s)

Qwen3.5 0.8B 46DeepSeek-R1 1.5B 42Qwen 1.8B 42lfm 2.5 8B (A1B) 38Llama 3.2 1B 35Qwen 3 1.7B 33Granite 3.1 moe 1B 33minicpm v4.6 1B 29Gemma 2 2B 29codeGemma 2B 29Gemma 2B 27smollm 2 1.7B 26starCoder 2 3B 26falcon 3 1B 26Qwen2.5 3B 24Qwen2.5 Coder 3B 24Qwen3.5 2B 24Qwen 3-VL 2B 24Llama 3.2 3B 23orca mini 3B 23Granite 3.1 moe 3B 23falcon 3 3B 23Granite 4.1 3B 22Granite 4.2 3B 21Phi 3 3.8B 21Qwen 4B 20

Comfortable (10-20 tok/s)

Qwen 3 4B 18Qwen2.5-VL 3B 14Gemma 3 4B 14Qwen 3-VL 4B 14translateGemma 4B 14Qwen3.5 4B 14Qwen 3 30B (A3B) 14Qwen3 Coder 30B (A3B) 14Qwen 3-VL 30B (A3B) 13Llama 2 7B 12Code Llama 7B 12DeepSeek Coder 6.7B 12Llama 2 uncensored 7B 12Qwen3.5 35B (A3B) 12Qwen3.6 35B (A3B) 12orca mini 7B 11gpt-oss 20B 11Mistral 7B 11Qwen2 7B 11starCoder 2 7B 11OLMo 2 7B 10Nemotron 3.5 lightning 30B (A3B) 10falcon 3 7B 10

Slow but usable (5-10 tok/s)

DeepSeek-R1 7B 9.8Qwen2.5 7B 9.8Llama 3 8B 9.8Qwen2.5 Coder 7B 9.8LLaVA 7B 9.8Llama 3.1 8B 9.4Qwen 7B 9.4dolPhin 3 8B 9.4Gemma 7B 9.2codeGemma 7B 9.2Gemma 4 26B (A4B) 9.2DeepSeek-R1 8B 8.9Qwen 3 8B 8.9Granite 4.2 8B 8.7Granite 4.1 8B 8.7Gemma 2 9B 8.5minicpm v 8B 8.4LLaVA Llama3 8B 8.4ornith 9B 8.2Gemma 3n E2B 8.2Qwen2.5-VL 7B 7.7minicpm v4.5 8B 7.6Qwen 3-VL 8B 7.6falcon 3 10B 7.3ornith 1.5 9B 7.0Qwen3.5 9B 7.0Mistral nemo 12B 6.5Gemma 4 E2B 6.4Llama 2 13B 6.2Code Llama 13B 6.2Gemma 3n E4B 6.1Gemma 4 12B 6.1Llama 3.2 Vision 11B 5.9Phi 3 14B 5.8orca mini 13B 5.8LLaVA 13B 5.8Gemma 3 12B 5.7translateGemma 12B 5.7Qwen 14B 5.6OLMo 2 13B 5.5DeepSeek Coder v2 16B 5.2DeepSeek-R1 14B 5.1Qwen2.5 14B 5.1Qwen2.5 Coder 14B 5.1Phi 4 14B 5.1Qwen 3 14B 5.0

Too slow (<5 tok/s)

Gemma 4 E4B 4.8starCoder 2 15B 4.7Mistral small 22B 3.5Mistral small 24B 3.3Mistral small3.2 24B 3.1Gemma 2 27B 2.9Granite 4.1 30B 2.7Gemma 3 27B 2.7Qwen3.5 27B 2.7Qwen3.6 27B 2.7translateGemma 27B 2.7Granite 4.2 30B 2.6Qwen3.8 27B 2.6muse glimmer 30B 2.6Qwen 32B 2.6Code Llama 34B 2.4DeepSeek Coder 33B 2.4DeepSeek-R1 32B 2.3Qwen2.5 32B 2.3Qwen 3 32B 2.3Gemma 4 31B 2.3Qwen2.5 Coder 32B 2.3LLaVA 34B 2.3qwq 32B 2.3mixtral 8X7B 2.3ornith 35B 2.2Qwen 3-VL 32B 2.2Qwen2.5-VL 32B 2.2ornith 1.5 35B 2.0

Won't fit

Nemotron 3 33BLlama 2 70BCode Llama 70Borca mini 70BLlama 2 uncensored 70BLlama 3 70BQwen 72BQwen2 72BLlama 3.1 70BDeepSeek-R1 70BLlama 3.3 70BQwen2.5 72BQwen2.5-VL 72BLlama 3.2 Vision 90BQwen 110Bgpt-oss 120BMistral medium 3.5 128Bmixtral 8X22BQwen3.5 122B (A10B)Nemotron 3 super 120B (A12B)Qwen3.8 flash next 125B (A6B)