Quantisation
46 updates on Quantisation.
Hugging Face adds native GGUF inference to Transformers, starting with Qwen3.5 on Macs
Transformers now loads GGUF models in packed form and runs them on Apple Silicon GPUs via Metal kernels, starting with Qwen3.5.
Cactus releases Needle 3, a 121M-parameter tool-calling model in an 8 to 29 MB binary
Needle 3 is a 121M-parameter model for tool calling, structured extraction and embeddings that ships as a single 8 to 29 MB binary.
OpenBMB releases MiniCPM5-2B for local deployment
A 2.5-billion-parameter dense model with a 131,072-token context, released under Apache-2.0 with GGUF, MLX and LiteRT-LM builds.
Artificial Analysis benchmarks 33 local models on an iPhone 17 Pro
A benchmark of quantised small models on an iPhone 17 Pro reports intelligence scores, generation times and peak memory between 0.4 and 6.9 GB.
Graphcore runs Llama 3.2 11B Vision on a Pixel 8a with 2.7-bit quantisation
Graphcore Research fits Llama-3.2-11B-Vision-Instruct into 3.6 GB with a 2.7-bit format and runs it on a Pixel 8a at 3.8 tok/s.
MobiBench paper benchmarks small LLMs in llama.cpp on two laptops and an RTX 3060
MobiBench is a llama.cpp benchmark suite for speed, memory and accuracy of small LLMs, but the reported tests ran on two laptops and not on phones.
Prism ML releases Bonsai 27B, a 3.9 GB 1-bit model that fits an iPhone 17 Pro
Prism ML’s 1-bit Bonsai 27B compresses Qwen3.6 27B to 3.9 GB and runs on iPhone, iPad, Mac and NVIDIA GPUs under Apache 2.0.
Apertus Mini distils an open-data 8B into 0.5B, 1.5B and 4B models
Swiss AI distilled its fully open Apertus 8B into 0.5B, 1.5B and 4B models on 1.7T tokens, with 3-bit to 6-bit MLX builds for Apple devices.
Meta AI builds MobileMoE, 5.3B parameters with 0.9B active per token
Meta AI trained three on-device mixture-of-experts models that store 1.3B to 5.3B parameters and run 272M to 922M of them per token.
Prism ML releases Bonsai Image 4B, a 1-bit image model that runs on iPhone
Bonsai Image 4B generates a 512x512 image in 9.4 seconds on an iPhone 17 Pro Max, and its 1-bit diffusion transformer needs 0.93 GB.
Quant.npu runs 4-bit LLMs up to 15.1 percent faster on a Qualcomm NPU
Eight authors report a fully static integer quantisation method that runs 4-bit models up to 15.1 percent faster on a Qualcomm SM8650 NPU.
Tencent open-sources a 440 MB offline translation model for phones
Hy-MT1.5-1.8B-1.25bit compresses a 1.8-billion-parameter translation model from 3.3 GB to 440 MB and runs offline on a phone. Weights and an Android demo are public.