Meta's MobileLLM-Pro runs a 128k context from a 590 MB 4-bit build
Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights.
Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB.
The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents.
Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30.
Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License.