On-device
Show HN: Off Grid runs text, image, vision and speech models offline on phones
An MIT-licensed Android and iOS app bundles llama.cpp, Stable Diffusion and Whisper for offline use. The Show HN thread reached 124 points and 66 comments.
Intelligence per watt puts local model coverage at 88.7% of real queries
Stanford and Together AI measured 20+ local models on 1M real queries and propose accuracy per watt as the metric for local inference.
Meta's MobileLLM-Pro runs a 128k context from a 590 MB 4-bit build
Meta Reality Labs released a 1.08B on-device model with a 128k context window, measured at 33.6 tok/s decode on a Galaxy S25 CPU.
Confidant fine-tunes Phi2-2.7B across a phone and two laptops in 40.1 hours
Zhejiang University researchers split a language model across three everyday devices and fine-tuned Phi2-2.7B on Alpaca in 40.1 hours without moving raw data.
ExecuTorch 1.0 reaches general availability for on-device PyTorch models
The PyTorch edge runtime promotes Core ML, Qualcomm Hexagon, Arm Ethos-U, Vulkan and XNNPACK backends to production status.
lm-Meter times on-device inference and finds prefill, not decode, is the bottleneck
A profiler that runs inside the phone measures each inference phase and reports prefill slowing 158 times as models grow, against 10 times for decode.
Meta trains 140M to 950M reasoning models on 4.2T tokens
MobileLLM-R1 spans 140M to 950M parameters, trained on 4.2T tokens, and Meta scores the 950M model at 74.0 on MATH500.
A19 Pro puts Neural Accelerators in every GPU core
Apple says the iPhone 17 Pro chip pairs Neural Accelerators in each of six GPU cores with a 16-core Neural Engine to run large local language models.
ShadowNPU scores attention on the Snapdragon NPU and reports 4.5 times faster inference
Peking University and BUPT move attention token scoring onto a Snapdragon NPU and report up to 4.5 times faster inference using a single CPU core.
P/D-Device prefills in the cloud and decodes on the phone, cutting TTFT 60%
Huawei researchers split prefill and decoding between cloud and phone, reporting time to first token down at least 60% and cloud throughput up to 15x.
Benchmark of 68 small language models finds architecture outweighs size on device
A study of 68 models from 100M to 5B puts Phi-3 near 70 percent accuracy and finds first-token time and memory tracking architecture, not parameter count.
Apple puts the cost of 2-bit compression at 3.4 MMLU points
Apple measures its on-device model at 67.8 MMLU in 16 bits and 64.4 after compression to 2 bits per weight, and details the distillation pipeline behind it.