S2-MoE speeds up MoE decoding by up to 5.3x in llama.cpp

Researchers at Peking University have published S2-MoE, a self-speculative decoding framework for mixture-of-experts (MoE) models on edge devices. MoE models activate only a few experts per token, which makes verifying several drafted tokens at once expensive. The method reports a speedup of up to 5.3x, about 2.0x on average, over standard autoregressive decoding and is implemented in llama.cpp.

The paper tests DeepSeek-V2-Lite-Chat (15.3B parameters, 1.35B active), OLMoE-1B-7B (6.7B, 1.1B active), Qwen3-30B-A3B (30.5B, 3.3B active) and GPT-OSS-120B (116.8B, 5.7B active). Hardware is a Jetson Orin NX 16GB, Jetson AGX Orin with 32 GB or 64 GB, and an RTX 4090 with constrained GPU memory. On the Orin NX, OLMoE rises from 2.30 to 12.08 tok/s and DeepSeek from 0.55 to 2.07 tok/s. Speedups on that board range from 2.10x to 3.17x for Qwen3 and from 1.57x to 2.53x for GPT-OSS.

The draft pass uses fewer experts per token, and the draft and target share one KV cache. Routing-aware expansion uses the draft’s expert routing to estimate verification cost and adds candidates only where the expected gain is higher. Reuse-aware gating nudges the router toward frequently activated experts, and expert-level offloading handles models that do not fit in memory.

The authors measure a perplexity ratio of 1.012 to 1.013 and top-1 agreement above 89%, with comparable results on LongBench. They compare against standard autoregressive decoding, layer-sparsity and expert-sparsity self-speculation, EAGLE-3 and Cascade, and say other MoE offloading work lacks public llama.cpp implementations. The code is on GitHub.