Fast (20+ tok/s)

Qwen3.5 0.8B 120+DeepSeek-R1 1.5B 120+Qwen 1.8B 120+Llama 3.2 1B 120+Qwen 3 1.7B 120+Granite 3.1 moe 1B 120+lfm 2.5 8B (A1B) 120+minicpm v4.6 1B 119Gemma 2 2B 119codeGemma 2B 119Gemma 2B 112smollm 2 1.7B 106starCoder 2 3B 106falcon 3 1B 106Qwen2.5 3B 101Qwen2.5 Coder 3B 101Qwen3.5 2B 101Qwen 3-VL 2B 101Llama 3.2 3B 96orca mini 3B 96Granite 3.1 moe 3B 96falcon 3 3B 96Granite 4.1 3B 91Granite 4.2 3B 87Phi 3 3.8B 87Qwen 4B 83Qwen 3 4B 74Qwen2.5-VL 3B 60Gemma 3 4B 58Qwen 3-VL 4B 58translateGemma 4B 58Qwen3.5 4B 56Qwen 3 30B (A3B) 56Qwen3 Coder 30B (A3B) 56Qwen 3-VL 30B (A3B) 53Llama 2 7B 50Code Llama 7B 50DeepSeek Coder 6.7B 50Llama 2 uncensored 7B 50Qwen3.5 35B (A3B) 50Qwen3.6 35B (A3B) 50orca mini 7B 47gpt-oss 20B 46Mistral 7B 43Qwen2 7B 43starCoder 2 7B 43OLMo 2 7B 43Nemotron 3.5 lightning 30B (A3B) 43falcon 3 7B 42DeepSeek-R1 7B 41Qwen2.5 7B 41Llama 3 8B 41Qwen2.5 Coder 7B 41LLaVA 7B 41Llama 3.1 8B 39Qwen 7B 39dolPhin 3 8B 39Gemma 7B 38codeGemma 7B 38Gemma 4 26B (A4B) 38DeepSeek-R1 8B 37Qwen 3 8B 37Granite 4.2 8B 36Granite 4.1 8B 36Gemma 2 9B 35minicpm v 8B 35LLaVA Llama3 8B 35ornith 9B 34Gemma 3n E2B 34gpt-oss 120B 33Qwen2.5-VL 7B 32minicpm v4.5 8B 31Qwen 3-VL 8B 31falcon 3 10B 30ornith 1.5 9B 29Qwen3.5 9B 29Mistral nemo 12B 27Gemma 4 E2B 27Llama 2 13B 26Code Llama 13B 26Gemma 3n E4B 26Gemma 4 12B 25Llama 3.2 Vision 11B 25Phi 3 14B 24orca mini 13B 24LLaVA 13B 24Gemma 3 12B 24translateGemma 12B 24Qwen 14B 23OLMo 2 13B 23DeepSeek Coder v2 16B 22DeepSeek-R1 14B 21Qwen2.5 14B 21Qwen2.5 Coder 14B 21Phi 4 14B 21Qwen 3 14B 21

Comfortable (10-20 tok/s)

Gemma 4 E4B 20starCoder 2 15B 19mixtral 8X7B 16Qwen3.5 122B (A10B) 16Mistral small 22B 15Mistral small 24B 14Mistral small3.2 24B 13Nemotron 3 super 120B (A12B) 12Gemma 2 27B 12Granite 4.1 30B 11Gemma 3 27B 11Qwen3.5 27B 11Qwen3.6 27B 11translateGemma 27B 11Granite 4.2 30B 11Qwen3.8 27B 11muse glimmer 30B 11Qwen 32B 11Code Llama 34B 10DeepSeek Coder 33B 10

Slow but usable (5-10 tok/s)

Qwen3.8 flash next 125B (A6B) 9.8DeepSeek-R1 32B 9.6Qwen2.5 32B 9.6Qwen 3 32B 9.6Gemma 4 31B 9.6Qwen2.5 Coder 32B 9.6LLaVA 34B 9.6qwq 32B 9.6ornith 35B 9.1Qwen 3-VL 32B 9.1Qwen2.5-VL 32B 9.1ornith 1.5 35B 8.3Nemotron 3 33B 6.8

Too slow (<5 tok/s)

Llama 2 70B 4.9Code Llama 70B 4.9orca mini 70B 4.9Llama 2 uncensored 70B 4.9Llama 3 70B 4.8Qwen 72B 4.7Qwen2 72B 4.7Llama 3.1 70B 4.4DeepSeek-R1 70B 4.4Llama 3.3 70B 4.4Qwen2.5 72B 4.1Qwen2.5-VL 72B 3.9Llama 3.2 Vision 90B 3.5Qwen 110B 3.0Mistral medium 3.5 128B 2.4mixtral 8X22B 2.4