On-device
TinyLlama pretrains a 1.1B model on 3 trillion tokens
Singapore University of Technology and Design trained a 1.1B model on 3 trillion tokens with 16 A100-40G GPUs and released it under Apache 2.0.
Apple researchers run models twice the size of available DRAM from flash
The LLM in a flash paper loads parameters from flash on demand and reports 4 to 5 times faster CPU and 20 to 25 times faster GPU inference.
Stability AI releases StableLM Zephyr 3B for edge devices
Stability AI tuned a 3B chat model with direct preference optimisation for edge devices and reports an MT-Bench score of 6.64.
Gemini Nano ships on the Pixel 8 Pro and Android gets AICore
Google put Gemini Nano on the Pixel 8 Pro for Recorder summaries and Gboard Smart Reply, and introduced AICore as the Android service behind it.
Apple publishes MLX, where CPU and GPU share arrays without copies
Apple machine learning research released an array framework for Apple silicon with a unified memory model, lazy evaluation and Swift bindings for iOS.
LLM.swift wraps llama.cpp for on-device text generation in Swift apps
A Swift package that wraps llama.cpp so iOS, macOS, watchOS, tvOS and visionOS apps can load a GGUF file and generate text on the device.
Snapdragon 8 Gen 3 targets 10-billion-parameter models on device
Qualcomm says the new flagship runs generative models with up to 10 billion parameters on device and reaches up to 20 tokens per second for LLMs.
MIT HAN Lab publishes TinyChatEngine for 4-bit LLMs on laptops and Raspberry Pi
The from-scratch C and C++ engine from Song Han's group at MIT runs 4-bit LLaMA2-7B on x86, Apple silicon, Raspberry Pi and Nvidia GPUs.
Hugging Face publishes swift-transformers for Core ML models in Swift apps
A Swift package with tokenizers, Hub downloads and a Core ML generation loop, demonstrated with Llama 2, Falcon, StarCoder and GPT models.
LLMFarm runs llama.cpp models offline on iOS and macOS
The iOS and macOS app loads a quantised model file off the device and chats with it offline, with 3B the practical size on an iPhone.
RWKV trains like a transformer and runs with constant memory per token
The paper releases pretrained RNN weights from 169M to 14B parameters and reports constant time and memory per token during inference.
Qualcomm argues for hybrid AI, citing 10 times the cost per generative AI search query
Qualcomm argues that cloud-only inference cannot scale, and puts models of 1B to 10B parameters on phones and laptops at INT4.