NPUs lead LLM prefill, CPUs lead decoding on Snapdragon phones

A team of researchers has published a cross-layer measurement study of mobile LLM inference titled “Is Your NPU Ready for LLMs?”. It covers five frameworks, llama.cpp, MNN, MLC-LLM, MLLM and Qualcomm’s GENIE, on CPU, GPU and NPU backends. The study finds that framework-induced performance gaps grow on NPUs and reach up to 10x.

The tests ran on four Snapdragon phones, the Xiaomi 17, OnePlus 15, Xiaomi 15 and Xiaomi 14, with Llama 3.2 1B and 3B, Qwen 2.5 1.5B and 7B and Phi 3.5 3.8B, mostly with 4-bit weights. At a 256-token context, GENIE on the NPU reaches 1,463.7 tok/s in prefill against 115.1 tok/s for llama.cpp. In decoding the order flips, with the CPU at about 50 to 76 tok/s and the NPU at about 10 to 34 tok/s. The authors attribute this to NPUs preferring large, fixed-shape workloads, which conflicts with the small, dynamic kernels of decoding.

For energy, the team built PowerBench, a lightweight C++ library that maps PMIC power zones to CPU, GPU and NPU for backend-specific attribution. They report that suboptimal configurations waste up to 40% of energy and that a tuned setup cuts NPU energy by up to 54.8%. The recommended settings are a 20 μs RPC polling interval, an NPU sleep latency of 65535 μs, the CPU at its lowest frequency and full-graph offloading with QNN.