Benchmarks
62 updates on Benchmarks.
Benchmark of 68 small language models finds architecture outweighs size on device
A study of 68 models from 100M to 5B puts Phi-3 near 70 percent accuracy and finds first-token time and memory tracking architecture, not parameter count.
Apple puts the cost of 2-bit compression at 3.4 MMLU points
Apple measures its on-device model at 67.8 MMLU in 16 bits and 64.4 after compression to 2 bits per weight, and details the distillation pipeline behind it.
Liquid AI releases LFM2, three CPU-first models from 350M to 1.2B
Liquid AI released open-weight models of 350M, 700M and 1.2B parameters and reports 2x faster decode and prefill than Qwen3 on CPU.
Hugging Face's SmolLM3 is a 3B model with 128k context and two reasoning modes
Hugging Face released a 3B model with a 128k context window, six languages and a switchable reasoning mode, along with the full training recipe.
Alibaba gives Qwen3's 0.6B and 1.7B models a reasoning switch
Qwen3-0.6B and Qwen3-1.7B carry the family's switch between a reasoning mode and a fast mode, with 32K context and Apache 2.0 weights.
Google's ML Drift runs Llama 3.1 8B on a phone GPU at 12.7 tokens per second
The paper measures 37.1 decode tokens per second for Gemma2 2B and 12.7 for Llama 3.1 8B on the Adreno 750 GPU of a Samsung S24.
Gemma 3 adds a 1B size that fits in 0.5 GB as an int4 checkpoint
Google released Gemma 3 at 1B, 4B, 12B and 27B with quantisation-aware int4 checkpoints of 0.5 GB and 2.6 GB for the two smallest sizes.
Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini
Helsinki and EURECOM researchers measured 11 models on a Raspberry Pi 5 and a Jetson Orin Nano, where memory and battery gave out before speed did.
Meta's ParetoQ compares five bit widths and puts the accuracy cliff at 1 bit
Meta trained the same models at 1, 1.58, 2, 3 and 4 bits under one recipe and reports 1.58-bit to 3-bit beating 4-bit at matched memory.
Amazon survey puts some small models at 10 to 100 times their parameter count
A survey of about 160 papers finds 1B to 8B models matching much larger ones on single benchmarks, and shows where models below 1B fall apart.
EXO Labs benchmarks put Llama 3.1 8B at 14 tok/s on an iPhone 15 Pro
EXO Labs published automated inference benchmarks from real devices, covering iPhone 15 Pro, Galaxy S24 Ultra, Mac mini M4 Pro clusters and desktop GPUs.
BlueLM-V-3B runs a multimodal model on a Dimensity 9300 NPU in 2.2 GB
CUHK and vivo AI Lab report 24.4 tok/s and 2.2 GB peak memory for a 3B vision-language model on a MediaTek Dimensity 9300.