Quantisation
46 updates on Quantisation.
GenAI at the edge survey lists 12 accelerators, 8 of them only simulated
A Johns Hopkins and Duke survey of generative AI on edge devices puts peak accelerator efficiency at 74.34 TOPS/W, with 8 of 12 designs only simulated.
Meta's ParetoQ compares five bit widths and puts the accuracy cliff at 1 bit
Meta trained the same models at 1, 1.58, 2, 3 and 4 bits under one recipe and reports 1.58-bit to 3-bit beating 4-bit at matched memory.
Alibaba MNN runs 4-bit LLMs on phone CPUs and GPUs, with a multimodal Android app
MNN-LLM converts PyTorch checkpoints into a 4-bit MNN format for phones, and Alibaba reports prefill 8.6 times faster than llama.cpp on an Android CPU.
Amazon survey puts some small models at 10 to 100 times their parameter count
A survey of about 160 papers finds 1B to 8B models matching much larger ones on single benchmarks, and shows where models below 1B fall apart.
BlueLM-V-3B runs a multimodal model on a Dimensity 9300 NPU in 2.2 GB
CUHK and vivo AI Lab report 24.4 tok/s and 2.2 GB peak memory for a 3B vision-language model on a MediaTek Dimensity 9300.
PalmBench finds iPhones running local LLMs about three times faster than Android phones
A benchmark of quantised LLMs on eight phones and boards reports throughput, memory, power and heat, with iPhones ahead and 4-bit drawing more power than 3-bit.
torchchat runs Llama 3 8B on a Galaxy S23 and iPhone at more than 8 tok/s
PyTorch's demonstration codebase exports Llama models to ExecuTorch and ships iOS and Android sample apps for running them on a phone.
Spectra ships ternary 3.9B models that match 4-bit quantisation at half the bits
Nolano AI trained 54 models from 99M to 3.9B parameters on the same 300B tokens in ternary, quantised and half-precision form to compare them by bit size.
llm.npu prefills 1,106 tok/s for Qwen1.5-1.8B on a Snapdragon 8 Gen 3
Peking University and BUPT move prompt processing onto the Hexagon NPU and report up to 43.6x faster prefill and up to 59.5x lower energy use.
Apple Intelligence pairs a 3-billion-parameter on-device model with a server model
Apple reports 0.6 ms per prompt token and 30 tokens per second on an iPhone 15 Pro for a model compressed to an average of 3.7 bits per weight.
ExecuTorch alpha runs Llama 2 7B on iPhone 15 Pro and Galaxy phones
PyTorch's edge runtime brought 4-bit Llama 2 7B to iPhone and Galaxy handsets, added early Llama 3 8B support and leaned on Apple, Arm and Qualcomm.
Microsoft runs Phi-3-mini offline on an iPhone 14 at over 12 tokens per second
The 3.8-billion-parameter model takes about 1.8 GB at 4-bit and scores 69 percent on MMLU, which Microsoft compares to Mixtral 8x7B and GPT-3.5.