Meta AI builds MobileMoE, 5.3B parameters with 0.9B active per token
Meta AI published MobileMoE on May 26, 2026, a family of three language models for phones that hold 1.3B, 2.8B and 5.3B parameters but run only 272M, 528M and 922M of them on any given token. Each one is a sparse mixture of experts, meaning the feedforward block in every layer is split into 60 small networks plus one always-on shared network, and a router picks 4 of the 60 for each token while the rest sit idle. Per-token compute therefore follows the active count, and the memory footprint follows the total.
That split is what makes the models awkward to host on a phone. The authors quantise all three to 4 bits with quantisation-aware training, which puts the weights at 0.68 GB, 1.48 GB and 2.75 GB, because every expert has to be reachable even though only a few run per token. On a Samsung Galaxy S25 they measure peak resident memory during inference at 1.49 GB for the smallest model at an 8k context against 1.91 GB for the dense MobileLLM-Pro, and at 4.71 GB for the largest. Sizing the whole expert set to fit phone DRAM is the opposite of the route Edge0 takes, which streams expert weights off storage to keep a 35B model inside a couple of gigabytes.
The paper reports that peak memory under real prompts runs 1.2 to 2.1 times higher than under repeated dummy tokens for the mixture-of-experts models, while staying flat for the dense baseline, because varied input routes to more experts and pulls more of them into RAM. Meta AI therefore profiled with real code, knowledge and maths prompts and calls dummy-prompt figures a lower bound rather than a measurement.
The authors ran the models through ExecuTorch on the Galaxy S25 with a Snapdragon 8 Elite on 4 CPU threads and on an iPhone 16 Pro with an A18 Pro on 2 CPU threads, after writing a fused mixture-of-experts operator because mobile CPU backends ship none. On a 512-token prompt they measure the smallest model decoding at 130.0 tok/s on the Galaxy S25 against 56.5 for MobileLLM-Pro, and at 180.5 tok/s on the iPhone 16 Pro against 55.1. Time to first token on a 1k prompt drops from 4.26 s to 2.01 s on the Galaxy and from 6.03 s to 2.14 s on the iPhone. Across both phones, CPU and GPU, and the XNNPACK and MLX backends, Meta AI reports the smallest model running 1.8 to 3.8 times faster at prefill and 2.2 to 3.4 times faster at decode than the dense baseline at comparable 4-bit weight memory.
Across 14 benchmarks the authors score the instruction-tuned models at 46.7, 55.3 and 60.1 on average at 16-bit precision, against 42.7 for the dense Qwen3.5 0.8B, 44.0 for Gemma 3 1B, 46.9 for Llama 3.2 1B and 50.8 for Qwen3.5 2B, the last of which carries 1.9B active parameters. The largest model reaches 60.1 against 55.6 for Ai2’s OLMoE-1B-7B, with 30 percent fewer active and 23 percent fewer total parameters. After 4-bit quantisation those averages fall to 44.0, 52.5 and 57.8, which leaves the smallest model 1.5 points behind MobileLLM-Pro at 45.5 while scoring 9.4 points higher on MMLU. Meta AI states that the training data across all four stages comes from openly licensed public datasets, names no weight release, and lists mobile NPU deployment, expert pruning and mixed-precision quantisation as further work.
