Fast (20+ tok/s)

Qwen3.5 0.8B 54DeepSeek-R1 1.5B 49Qwen 1.8B 49lfm 2.5 8B (A1B) 44Llama 3.2 1B 41Qwen 3 1.7B 38Granite 3.1 moe 1B 38minicpm v4.6 1B 34Gemma 2 2B 34codeGemma 2B 34Gemma 2B 32smollm 2 1.7B 30starCoder 2 3B 30falcon 3 1B 30Qwen2.5 3B 28Qwen2.5 Coder 3B 28Qwen3.5 2B 28Qwen 3-VL 2B 28Llama 3.2 3B 27orca mini 3B 27Granite 3.1 moe 3B 27falcon 3 3B 27Granite 4.1 3B 26Granite 4.2 3B 24Phi 3 3.8B 24Qwen 4B 23Qwen 3 4B 21

Comfortable (10-20 tok/s)

Qwen2.5-VL 3B 17Gemma 3 4B 16Qwen 3-VL 4B 16translateGemma 4B 16Qwen3.5 4B 16Qwen 3 30B (A3B) 16Qwen3 Coder 30B (A3B) 16Qwen 3-VL 30B (A3B) 15Llama 2 7B 14Code Llama 7B 14DeepSeek Coder 6.7B 14Llama 2 uncensored 7B 14Qwen3.5 35B (A3B) 14Qwen3.6 35B (A3B) 14orca mini 7B 13gpt-oss 20B 13Mistral 7B 12Qwen2 7B 12starCoder 2 7B 12OLMo 2 7B 12Nemotron 3.5 lightning 30B (A3B) 12falcon 3 7B 12DeepSeek-R1 7B 11Qwen2.5 7B 11Llama 3 8B 11Qwen2.5 Coder 7B 11LLaVA 7B 11Llama 3.1 8B 11Qwen 7B 11dolPhin 3 8B 11Gemma 7B 11codeGemma 7B 11Gemma 4 26B (A4B) 11DeepSeek-R1 8B 10Qwen 3 8B 10Granite 4.2 8B 10Granite 4.1 8B 10Gemma 2 9B 10

Slow but usable (5-10 tok/s)

minicpm v 8B 9.8LLaVA Llama3 8B 9.8ornith 9B 9.6Gemma 3n E2B 9.6Qwen2.5-VL 7B 9.0minicpm v4.5 8B 8.8Qwen 3-VL 8B 8.8falcon 3 10B 8.5ornith 1.5 9B 8.1Qwen3.5 9B 8.1Mistral nemo 12B 7.6Gemma 4 E2B 7.5Llama 2 13B 7.3Code Llama 13B 7.3Gemma 3n E4B 7.2Gemma 4 12B 7.1Llama 3.2 Vision 11B 6.9Phi 3 14B 6.8orca mini 13B 6.8LLaVA 13B 6.7Gemma 3 12B 6.6translateGemma 12B 6.6Qwen 14B 6.6OLMo 2 13B 6.4DeepSeek Coder v2 16B 6.0DeepSeek-R1 14B 6.0Qwen2.5 14B 6.0Qwen2.5 Coder 14B 6.0Phi 4 14B 5.9Qwen 3 14B 5.8Gemma 4 E4B 5.6starCoder 2 15B 5.4

Too slow (<5 tok/s)

Mistral small 22B 4.1Mistral small 24B 3.8Mistral small3.2 24B 3.6Gemma 2 27B 3.4Granite 4.1 30B 3.2Gemma 3 27B 3.2Qwen3.5 27B 3.2Qwen3.6 27B 3.2translateGemma 27B 3.2Granite 4.2 30B 3.0Qwen3.8 27B 3.0muse glimmer 30B 3.0Qwen 32B 3.0Code Llama 34B 2.8DeepSeek Coder 33B 2.8DeepSeek-R1 32B 2.7Qwen2.5 32B 2.7Qwen 3 32B 2.7Gemma 4 31B 2.7Qwen2.5 Coder 32B 2.7LLaVA 34B 2.7qwq 32B 2.7mixtral 8X7B 2.7ornith 35B 2.6Qwen 3-VL 32B 2.6Qwen2.5-VL 32B 2.6ornith 1.5 35B 2.3

Won't fit

Nemotron 3 33BLlama 2 70BCode Llama 70Borca mini 70BLlama 2 uncensored 70BLlama 3 70BQwen 72BQwen2 72BLlama 3.1 70BDeepSeek-R1 70BLlama 3.3 70BQwen2.5 72BQwen2.5-VL 72BLlama 3.2 Vision 90BQwen 110Bgpt-oss 120BMistral medium 3.5 128Bmixtral 8X22BQwen3.5 122B (A10B)Nemotron 3 super 120B (A12B)Qwen3.8 flash next 125B (A6B)