Fast (20+ tok/s)

Qwen3.5 0.8B 120+DeepSeek-R1 1.5B 120+Qwen 1.8B 120+Llama 3.2 1B 120+Qwen 3 1.7B 120+Granite 3.1 moe 1B 120+minicpm v4.6 1B 120+Gemma 2 2B 120+codeGemma 2B 120+Gemma 2B 120+smollm 2 1.7B 120+starCoder 2 3B 120+falcon 3 1B 120+lfm 2.5 8B (A1B) 120+Qwen2.5 3B 115Qwen2.5 Coder 3B 115Qwen3.5 2B 115Qwen 3-VL 2B 115Llama 3.2 3B 109orca mini 3B 109Granite 3.1 moe 3B 109falcon 3 3B 109Granite 4.1 3B 104Granite 4.2 3B 99Phi 3 3.8B 99Qwen 4B 95Qwen 3 4B 84Qwen2.5-VL 3B 68Gemma 3 4B 66Qwen 3-VL 4B 66translateGemma 4B 66Qwen3.5 4B 64Qwen 3 30B (A3B) 64Qwen3 Coder 30B (A3B) 64Qwen 3-VL 30B (A3B) 61Llama 2 7B 58Code Llama 7B 58DeepSeek Coder 6.7B 58Llama 2 uncensored 7B 58Qwen3.5 35B (A3B) 57Qwen3.6 35B (A3B) 57orca mini 7B 53gpt-oss 20B 53Mistral 7B 50Qwen2 7B 50starCoder 2 7B 50OLMo 2 7B 49Nemotron 3.5 lightning 30B (A3B) 49falcon 3 7B 48DeepSeek-R1 7B 47Qwen2.5 7B 47Llama 3 8B 47Qwen2.5 Coder 7B 47LLaVA 7B 47Llama 3.1 8B 45Qwen 7B 45dolPhin 3 8B 45Gemma 7B 44codeGemma 7B 44Gemma 4 26B (A4B) 44DeepSeek-R1 8B 42Qwen 3 8B 42Granite 4.2 8B 41Granite 4.1 8B 41Gemma 2 9B 40minicpm v 8B 40LLaVA Llama3 8B 40ornith 9B 39Gemma 3n E2B 39Qwen2.5-VL 7B 36minicpm v4.5 8B 36Qwen 3-VL 8B 36falcon 3 10B 35ornith 1.5 9B 33Qwen3.5 9B 33Mistral nemo 12B 31Gemma 4 E2B 30Llama 2 13B 30Code Llama 13B 30Gemma 3n E4B 29Gemma 4 12B 29Llama 3.2 Vision 11B 28Phi 3 14B 28orca mini 13B 28LLaVA 13B 27Gemma 3 12B 27translateGemma 12B 27Qwen 14B 27OLMo 2 13B 26DeepSeek Coder v2 16B 25DeepSeek-R1 14B 24Qwen2.5 14B 24Qwen2.5 Coder 14B 24Phi 4 14B 24Qwen 3 14B 24Gemma 4 E4B 23starCoder 2 15B 22

Comfortable (10-20 tok/s)

mixtral 8X7B 18Mistral small 22B 17Mistral small 24B 16Mistral small3.2 24B 15Gemma 2 27B 14Granite 4.1 30B 13Gemma 3 27B 13Qwen3.5 27B 13Qwen3.6 27B 13translateGemma 27B 13Granite 4.2 30B 12Qwen3.8 27B 12muse glimmer 30B 12Qwen 32B 12Code Llama 34B 12DeepSeek Coder 33B 12DeepSeek-R1 32B 11Qwen2.5 32B 11Qwen 3 32B 11Gemma 4 31B 11Qwen2.5 Coder 32B 11LLaVA 34B 11qwq 32B 11ornith 35B 10Qwen 3-VL 32B 10Qwen2.5-VL 32B 10

Slow but usable (5-10 tok/s)

ornith 1.5 35B 9.5Nemotron 3 33B 7.8Llama 2 70B 5.6Code Llama 70B 5.6orca mini 70B 5.6Llama 2 uncensored 70B 5.6Llama 3 70B 5.5Qwen 72B 5.3Qwen2 72B 5.3Llama 3.1 70B 5.1DeepSeek-R1 70B 5.1Llama 3.3 70B 5.1

Too slow (<5 tok/s)

Qwen2.5 72B 2.8Qwen2.5-VL 72B 2.7Llama 3.2 Vision 90B 2.4

Won't fit

Qwen 110Bgpt-oss 120BMistral medium 3.5 128Bmixtral 8X22BQwen3.5 122B (A10B)Nemotron 3 super 120B (A12B)Qwen3.8 flash next 125B (A6B)