On-device
MLC LLM brings local language models to iPhone, browsers and consumer GPUs
A compiler stack built on Apache TVM deploys chat models natively to iOS, browsers and consumer GPUs, with an iPhone build handed out through TestFlight.
Sherpa runs LLaMA on an Android phone through a Flutter chat app
The Flutter app loads a ggml model file off the phone and chats with it offline, and development stopped three weeks after it started.
Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++
The dependency-free C and C++ inference engine supports 1.5-bit to 8-bit quantisation and treats Apple silicon as a first-class target.
Qualcomm runs Stable Diffusion on an Android phone for the first time
Qualcomm AI Research generated 512x512 images in under 15 seconds on a Snapdragon 8 Gen 2 phone after quantising the model from FP32 to INT8.
George Hotz starts tinygrad, a framework that ports to an accelerator in about 25 ops
The tiny corp framework caps its repository at 26,500 lines, ships Metal, Adreno and WebGPU backends, and runs openpilot on a Snapdragon 845 GPU.
ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses
The release introduced a reduced-size runtime for phones and the ORT model format, the base for the NNAPI, Core ML, QNN and XNNPACK mobile packages.
MobileBERT runs in 62 ms on a Pixel 4 with 25.3M parameters
Carnegie Mellon and Google Brain distilled BERT into a 25.3M-parameter model that answers in 62 ms on a Pixel 4 and scores 77.7 on GLUE.
Hugging Face's DistilBERT is 40% smaller than BERT and 71% faster on an iPhone
The 66M-parameter distilled BERT keeps 97% of its teacher on GLUE and weighs 207 MB inside a question-answering app on an iPhone 7 Plus.