Benchmarks
52 updates on Benchmarks.
Artificial Analysis benchmarks 33 local models on an iPhone 17 Pro
A benchmark of quantised small models on an iPhone 17 Pro reports intelligence scores, generation times and peak memory between 0.4 and 6.9 GB.
Liquid AI releases LFM2.5-2.6B for on-device agents
Liquid AI reports 30 tokens per second on a phone and under 2.5 GB of memory for its 2.6-billion-parameter model, released with open weights.
MLPerf Mobile v6.0 adds Llama tests in 1B, 3B and 8B sizes
MLCommons added Llama tests in 1B, 3B and 8B sizes to its mobile benchmark app, reporting token throughput next to the existing vision and image tests.
CAPED redacts phone screenshots before a cloud GUI agent sees them
Researchers at CUHK, Southeast University and Shandong University report a phone-side screenshot filter that cuts seeded leakage from 0.766 to 0.268.
LiteRT-LM reports 52 decode tokens per second for Gemma 4 E2B on an Android GPU
Google publishes prefill and decode figures for its on-device runtime, adds Swift and JavaScript APIs, and reports a 2.2x speedup from multi-token prediction.
Tempo compresses hour-long video with a 2B vision model so a 4B LLM can answer
Meta AI and KAUST report 52.7 on LVBench for a 6B system in which a 2B vision-language model squeezes long video down to about 3 tokens per frame.
iPhone 16 Pro settles 41.5 percent below peak over 20 back-to-back prompts
Four authors ran Qwen 2.5 1.5B 20 times in a row on four edge platforms and report the iPhone 16 Pro settling 41.5 percent below its peak throughput.
Meta's MobileLLM-Flash goes shallow and wide for 1.8 times faster prefill
Meta designed 350M, 650M and 1.4B models by measuring latency on a Galaxy S25, reversing the deep-and-thin rule of the first MobileLLM.
Alibaba adds 0.8B and 2B sizes to Qwen3.5, with 262K context and a vision encoder
Alibaba released Qwen3.5-0.8B and Qwen3.5-2B, dense vision-language models with a 262,144-token context, Apache 2.0 weights and 4-bit MNN builds.
Nemotron-Flash-1B decodes 1.9 times faster than Qwen3-0.6B on an H100
Nvidia designed a hybrid 1B and 3B model family around measured decoding latency on an H100 rather than parameter count, and released three checkpoints.
Intelligence per watt puts local model coverage at 88.7% of real queries
Stanford and Together AI measured 20+ local models on 1M real queries and propose accuracy per watt as the metric for local inference.
lm-Meter times on-device inference and finds prefill, not decode, is the bottleneck
A profiler that runs inside the phone measures each inference phase and reports prefill slowing 158 times as models grow, against 10 times for decode.