On-device
Raspberry Pi 5 beats human reading speed only up to Phi 3.5 mini
Helsinki and EURECOM researchers measured 11 models on a Raspberry Pi 5 and a Jetson Orin Nano, where memory and battery gave out before speed did.
MobiLLM fine-tunes OPT-1.3B in 4.5 GB on device by moving backpropagation to a server
Researchers fine-tuned OPT-1.3B on a Jetson Xavier NX in 4.5 GB by keeping the frozen model on the device and the trainable side network on a server.
GenAI at the edge survey lists 12 accelerators, 8 of them only simulated
A Johns Hopkins and Duke survey of generative AI on edge devices puts peak accelerator efficiency at 74.34 TOPS/W, with 8 of 12 designs only simulated.
Meta's ParetoQ compares five bit widths and puts the accuracy cliff at 1 bit
Meta trained the same models at 1, 1.58, 2, 3 and 4 bits under one recipe and reports 1.58-bit to 3-bit beating 4-bit at matched memory.
Alibaba MNN runs 4-bit LLMs on phone CPUs and GPUs, with a multimodal Android app
MNN-LLM converts PyTorch checkpoints into a 4-bit MNN format for phones, and Alibaba reports prefill 8.6 times faster than llama.cpp on an Android CPU.
Amazon survey puts some small models at 10 to 100 times their parameter count
A survey of about 160 papers finds 1B to 8B models matching much larger ones on single benchmarks, and shows where models below 1B fall apart.
EXO Labs benchmarks put Llama 3.1 8B at 14 tok/s on an iPhone 15 Pro
EXO Labs published automated inference benchmarks from real devices, covering iPhone 15 Pro, Galaxy S24 Ultra, Mac mini M4 Pro clusters and desktop GPUs.
BlueLM-V-3B runs a multimodal model on a Dimensity 9300 NPU in 2.2 GB
CUHK and vivo AI Lab report 24.4 tok/s and 2.2 GB peak memory for a 3B vision-language model on a MediaTek Dimensity 9300.
PhoneLM searches for a fast architecture before training it and hits 58 tok/s
BUPT researchers picked their 0.5B and 1.5B transformer shapes by measuring speed on a Snapdragon 8 Gen 3 first, then pre-training the winner.
Hugging Face trains SmolLM2 at 135M, 360M and 1.7B on up to 11T tokens
Hugging Face released SmolLM2 in three sizes trained on up to 11 trillion tokens, with 4-bit builds from 118 MB for on-device runtimes.
Mistral puts Ministral 3B and 8B on devices with 128k context
Ministral 3B and 8B handle up to 128k tokens for on-device work, but only the 8B Instruct weights were published, and for research use.
PalmBench finds iPhones running local LLMs about three times faster than Android phones
A benchmark of quantised LLMs on eight phones and boards reports throughput, memory, power and heat, with iPhones ahead and 4-bit drawing more power than 3-bit.