Fast (20+ tok/s)

Qwen3.5 0.8B 120+DeepSeek-R1 1.5B 120+Qwen 1.8B 120+Llama 3.2 1B 120+Qwen 3 1.7B 120+Granite 3.1 moe 1B 120+minicpm v4.6 1B 120+Gemma 2 2B 120+codeGemma 2B 120+Gemma 2B 120+smollm 2 1.7B 120+starCoder 2 3B 120+falcon 3 1B 120+Qwen2.5 3B 120+Qwen2.5 Coder 3B 120+Qwen3.5 2B 120+Qwen 3-VL 2B 120+Llama 3.2 3B 120+orca mini 3B 120+Granite 3.1 moe 3B 120+falcon 3 3B 120+lfm 2.5 8B (A1B) 120+Granite 4.1 3B 117Granite 4.2 3B 112Phi 3 3.8B 112Qwen 4B 107Qwen 3 4B 95Qwen2.5-VL 3B 77Gemma 3 4B 74Qwen 3-VL 4B 74translateGemma 4B 74Qwen3.5 4B 72Qwen 3 30B (A3B) 72Qwen3 Coder 30B (A3B) 72Qwen 3-VL 30B (A3B) 68Llama 2 7B 65Code Llama 7B 65DeepSeek Coder 6.7B 65Llama 2 uncensored 7B 65Qwen3.5 35B (A3B) 64Qwen3.6 35B (A3B) 64orca mini 7B 60gpt-oss 20B 60Mistral 7B 56Qwen2 7B 56starCoder 2 7B 56OLMo 2 7B 55Nemotron 3.5 lightning 30B (A3B) 55falcon 3 7B 53DeepSeek-R1 7B 52Qwen2.5 7B 52Llama 3 8B 52Qwen2.5 Coder 7B 52LLaVA 7B 52Llama 3.1 8B 50Qwen 7B 50dolPhin 3 8B 50Gemma 7B 49codeGemma 7B 49Gemma 4 26B (A4B) 49DeepSeek-R1 8B 47Qwen 3 8B 47Granite 4.2 8B 46Granite 4.1 8B 46Gemma 2 9B 46minicpm v 8B 45LLaVA Llama3 8B 45ornith 9B 44Gemma 3n E2B 44Qwen2.5-VL 7B 41minicpm v4.5 8B 40Qwen 3-VL 8B 40falcon 3 10B 39ornith 1.5 9B 37Qwen3.5 9B 37Mistral nemo 12B 35Gemma 4 E2B 34Llama 2 13B 33Code Llama 13B 33Gemma 3n E4B 33Gemma 4 12B 32Llama 3.2 Vision 11B 32Phi 3 14B 31orca mini 13B 31LLaVA 13B 31Gemma 3 12B 30translateGemma 12B 30Qwen 14B 30OLMo 2 13B 29DeepSeek Coder v2 16B 28DeepSeek-R1 14B 27Qwen2.5 14B 27Qwen2.5 Coder 14B 27Phi 4 14B 27Qwen 3 14B 26Gemma 4 E4B 26starCoder 2 15B 25mixtral 8X7B 20

Comfortable (10-20 tok/s)

Mistral small 22B 19Mistral small 24B 18Mistral small3.2 24B 16Gemma 2 27B 15Granite 4.1 30B 14Gemma 3 27B 14Qwen3.5 27B 14Qwen3.6 27B 14translateGemma 27B 14Granite 4.2 30B 14Qwen3.8 27B 14muse glimmer 30B 14Qwen 32B 14Code Llama 34B 13DeepSeek Coder 33B 13DeepSeek-R1 32B 12Qwen2.5 32B 12Qwen 3 32B 12Gemma 4 31B 12Qwen2.5 Coder 32B 12LLaVA 34B 12qwq 32B 12ornith 35B 12Qwen 3-VL 32B 12Qwen2.5-VL 32B 12ornith 1.5 35B 11

Slow but usable (5-10 tok/s)

Nemotron 3 33B 8.8Llama 2 70B 6.3Code Llama 70B 6.3orca mini 70B 6.3Llama 2 uncensored 70B 6.3Llama 3 70B 6.1Qwen 72B 6.0Qwen2 72B 6.0Llama 3.1 70B 5.7DeepSeek-R1 70B 5.7Llama 3.3 70B 5.7

Too slow (<5 tok/s)

Qwen2.5 72B 3.1Qwen2.5-VL 72B 3.0Llama 3.2 Vision 90B 2.7

Won't fit

Qwen 110Bgpt-oss 120BMistral medium 3.5 128Bmixtral 8X22BQwen3.5 122B (A10B)Nemotron 3 super 120B (A12B)Qwen3.8 flash next 125B (A6B)