On-device
One shared on-device LLM keeps a context per app and switches in 0.27 seconds
Peking University, BUPT and Tsinghua researchers compress and swap each app conversation context in 16-token chunks, switching up to 20 times faster.
Google ships an experimental MediaPipe LLM Inference API for web, Android and iOS
The experimental API runs Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B fully on device, with int8 and int4 weights.
MeRino designs sub-100M language models that run 4.9 times faster than OPT-350M
Researchers design 52M to 64M parameter transformers by maximising entropy under a compute budget, matching OPT-350M accuracy on an NVIDIA Jetson Nano.
Stability AI trains Stable LM 2 1.6B on seven languages and 2 trillion tokens
The technical report details a 1.6B model pre-trained on seven languages and measures 127 tok/s for a 4-bit build on an M2 Mac mini.
BitNet b1.58 gives every weight three values and runs 3B in 2.22 GB
Microsoft trained models whose every weight is -1, 0 or 1, which replaces multiplication with addition, and reports parity with full-precision Llama from 3B.
MobiLlama is a fully transparent 0.5B model that runs in 770 MB on a phone
MBZUAI published a 0.5B model that shares one feed-forward block across all layers and reports 7.02 tok/s in 770 MB on a Snapdragon 685 phone.
Qualcomm AI Hub opens 75 pre-optimised models and profiling on hosted Snapdragon phones
Qualcomm opened a library of more than 75 models tuned for Snapdragon, with compilation and profiling on real phones in its cloud and two 7B chat models listed.
Meta's MobileLLM trades width for depth and gains 2.7 and 4.3 points
Meta Reality Labs built 125M and 350M models around deep and thin layers and profiled them on an iPhone 13 through ExecuTorch.
Gemma 2B and 7B open the Gemma line, built on Gemini research
Google released Gemma 2B and 7B with an 8192-token context, weights on Kaggle and Hugging Face under a custom Gemma licence, not an open source one.
MobileVLM V2 runs 1.7B, 3B and 7B vision models on 144 image tokens
Meituan and Zhejiang University report a 1.7B vision language model at 64.2 on six benchmarks and 51.63 tok/s on an NVIDIA Jetson Orin.
Galaxy S24 becomes the second phone line to run Gemini Nano
Google brought Gemini Nano to the Galaxy S24 for on-device Magic Compose in Messages, while the rest of the Galaxy AI features run on Gemini Pro.
Arm runs Llama 2 7B at 9.6 tokens per second on three mobile CPU cores
Arm demonstrated Llama 2 7B generating 9.6 tokens per second on three Cortex-A700 CPU cores, using int4 kernels it says beat llama.cpp by 20 percent.