Alibaba MNN runs 4-bit LLMs on phone CPUs and GPUs, with a multimodal Android app
Alibaba published the first public build of its MNN-LLM Android app on January 23, 2025, a client that downloads open models and runs them entirely on the phone, covering text-to-text, image-to-text, audio-to-text and text-to-image generation through diffusion models. It sits on MNN, the inference and training framework Alibaba has kept open source since 2019 and uses in more than 30 of its own apps, among them Taobao, Youku and DingTalk. Alibaba reports the runtime prefilling 8.6 times faster than llama.cpp on an Android CPU, with decoding 2.3 times faster.
Models reach the device through llmexport, which converts a PyTorch checkpoint to ONNX and then to MNN’s own format, splitting the weights into a separate file and quantising them to 4-bit or 8-bit by default, with block size configurable and 5-bit and 6-bit available through the MNNConvert tool. The same README lists Qwen, Gemma, Llama including TinyLlama and MobileLLM, Baichuan, Yi, DeepSeek, InternLM, Phi, ReaderLM and SmolLM as supported families. LoRA adapters can be merged into the base model at export or kept as a separate file that the runtime loads and swaps while running.
The MNN-LLM paper, presented at the ACM Multimedia Asia 2024 workshops, describes a DRAM-Flash split that leaves the embedding table, roughly 15% of the weights, in bfloat16 on flash storage and reads only the single token needed for each decode step. The authors put the saving for Qwen2 7B at about 2.18 GB of DRAM. Layers and the LM head stay in memory at int4 or int8 with asymmetric quantisation, computed as W4A8 or W8A8 on CPU and W4A16 or W8A16 on GPU, while the KV cache quantises keys to int4 or int8 and values to fp8 before spilling to flash with prefetching. The team measured the speed comparisons on a Xiaomi 14 against llama.cpp, MLC-LLM and fastllm, using Qwen2 1.5B, Qwen2 7B and Llama 3 8B.
Android builds run on the CPU with ARMv8.2 fp16 instructions and add an OpenCL path for the GPU, and the same runtime compiles for iOS with Metal, for macOS, Linux and Windows, and for browsers through WebAssembly. Alibaba distributes the app as a downloadable APK and warns that the first version was tested only on a OnePlus 13 and a Xiaomi 14 Ultra, so low-spec phones may run slowly or not at all. MNN is published under the Apache 2.0 license.
