Benchmarks
62 updates on Benchmarks.
Liquid AI releases LFM2.5-24B-A2B, a 2B-active MoE for local agents
Liquid AI reports 184 tok/s on an Apple M5 Max for its 24B-parameter mixture-of-experts model, which activates 2.45B parameters per token.
Nvidia releases Nemotron-3-Diarization, a 100M speaker model with 0.32 second latency
Nemotron-3-Diarization labels up to eight speakers in streaming or offline audio and ranks first on Voice Arena’s Diarization-Bench v1 with 14.7% DER.
Liquid AI releases a DSpark drafter that speeds up LFM2.5-VL-3B decoding
Liquid AI’s 280M-parameter draft model speeds up decoding of its 3B vision-language model by up to 3.13x on Apple Silicon, for 8.9% more parameters.
Hugging Face adds native GGUF inference to Transformers, starting with Qwen3.5 on Macs
Transformers now loads GGUF models in packed form and runs them on Apple Silicon GPUs via Metal kernels, starting with Qwen3.5.
Desert Ant Labs releases 18 on-device models for speech, audio and privacy tasks
Desert Ant Labs released 18 single-task models that run locally on phones and laptops and are free up to 100k monthly active devices.
Artificial Analysis benchmarks 33 local models on an iPhone 17 Pro
A benchmark of quantised small models on an iPhone 17 Pro reports intelligence scores, generation times and peak memory between 0.4 and 6.9 GB.
Liquid AI releases LFM2.5-2.6B for on-device agents
Liquid AI reports 30 tokens per second on a phone and under 2.5 GB of memory for its 2.6-billion-parameter model, released with open weights.
MobiBench paper benchmarks small LLMs in llama.cpp on two laptops and an RTX 3060
MobiBench is a llama.cpp benchmark suite for speed, memory and accuracy of small LLMs, but the reported tests ran on two laptops and not on phones.
Prism ML releases Bonsai 27B, a 3.9 GB 1-bit model that fits an iPhone 17 Pro
Prism ML’s 1-bit Bonsai 27B compresses Qwen3.6 27B to 3.9 GB and runs on iPhone, iPad, Mac and NVIDIA GPUs under Apache 2.0.
NPUs lead LLM prefill, CPUs lead decoding on Snapdragon phones
A measurement study of five mobile LLM frameworks finds NPUs excel at prefill, CPUs at decoding, and framework gaps of up to 10x on NPUs.
MLPerf Mobile v6.0 adds Llama tests in 1B, 3B and 8B sizes
MLCommons added Llama tests in 1B, 3B and 8B sizes to its mobile benchmark app, reporting token throughput next to the existing vision and image tests.
CAPED redacts phone screenshots before a cloud GUI agent sees them
Researchers at CUHK, Southeast University and Shandong University report a phone-side screenshot filter that cuts seeded leakage from 0.766 to 0.268.