https://llmobile.news/tags/gpu/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/s2-moe-self-speculative-decoding/https://llmobile.news/ticker/s2-moe-self-speculative-decoding/Wed, 19 Aug 2026 00:00:00 +0200Researchers at Peking University have published S2-MoE, a self-speculative decoding framework for mixture-of-experts (MoE) models on edge devices. MoE models activate only a few experts per token, which makes verifying several drafted tokens at once expensive. The method reports a speedup of up to 5.3x, about 2.0x on average, over standard autoregressive decoding and is implemented in llama.cpp. The paper tests DeepSeek-V2-Lite-Chat (15.3B parameters, 1.35B active), OLMoE-1B-7B (6.7B, 1.1B active), Qwen3-30B-A3B (30.5B, 3.3B active) and GPT-OSS-120B (116.8B, 5.7B active). Hardware is a Jetson Orin NX 16GB, Jetson AGX Orin with 32 GB or 64 GB, and an RTX 4090 with constrained GPU memory. On the Orin NX, OLMoE rises from 2.30 to 12.08 tok/s and DeepSeek from 0.55 to 2.07 tok/s. Speedups on that board range from 2.10x to 3.17x for Qwen3 and from 1.57x to 2.53x for GPT-OSS. The draft pass uses fewer experts per token, and the draft and target share one KV cache. Routing-aware expansion uses the draft’s expert routing to estimate verification cost and adds candidates only where the expected gain is higher. Reuse-aware gating nudges the router toward frequently activated experts, and expert-level offloading handles models that do not fit in memory. The authors measure a perplexity ratio of 1.012 to 1.013 and top-1 agreement above 89%, with comparable results on LongBench. They compare against standard autoregressive decoding, layer-sparsity and expert-sparsity self-speculation, EAGLE-3 and Cascade, and say other MoE offloading work lacks public llama.cpp implementations. The code is on GitHub. Source: https://arxiv.org/abs/2608.15018v2 Read the article: https://llmobile.news/ticker/s2-moe-self-speculative-decoding/ResearchOpen sourceGPURuntimeshttps://llmobile.news/ticker/mobibench-llamacpp/https://llmobile.news/ticker/mobibench-llamacpp/Wed, 15 Jul 2026 00:00:00 +0200Researchers have published MobiBench, a benchmarking suite built on llama.cpp that measures how small language models behave under limited resources. It covers prefill speed, decode speed, time-to-first-token, memory use and task accuracy. The paper tests Gemma 1B, TinyLlama, Llama 3B and Phi-2, most of them released in 2023 and 2024, plus Q2_K and Q3_K_M quantized variants of Phi-2. A diagram in the paper shows the llama.cpp execution pipeline from runtime initialization and model loading to compute graph construction and iterative decoding. Figure 1 of the paper, titled high-level execution pipeline of llama.cpp.MobiBench authors, arXiv Despite the mobile framing, the authors ran the tests on a gaming laptop with an Intel Core i7 and an RTX 3060 GPU and on a standard laptop with an Intel Core i5 and no dedicated graphics. The paper reports no phone, Android or ARM measurements and no battery or energy data. On the GPU, Llama 3B reaches 225 to 243 tok/s in decode, Phi-2 reaches 102 to 122 tok/s and Gemma 1B reaches 150 to 167 tok/s. The Gemma 1B figure is about 7 to 8 times the 20 to 23 tok/s the authors measured on the CPU. For TinyLlama, GPU inference cut average wall-clock time per sample from 65.81 s to 1.39 s on MMLU, a 12 to 47 times speedup across the four tasks. On the GPU, Phi-2 used about 2,070 to 2,091 MB of memory, the Q3_K_M variant about 1,982 to 2,000 MB and the Q2_K variant about 1,694 to 1,710 MB. The Q2_K variant decoded at 46 to 56 tok/s and the Q3_K_M variant at 58 to 88 tok/s. On MMLU, Phi-2 scored 0.243, the Q3_K_M variant 0.263 and the Q2_K variant 0.233. The accuracy tests used 1,000 samples each from RepLiQA, CNN/DailyMail summarization, SciQ and MMLU. Summarization BERTScore F1 stayed between 0.8388 and 0.8458 across all models, and an LLM-based semantic evaluation gave mean scores from 0.4554 for TinyLlama on the GPU to 0.6961 for Gemma 1B on the CPU. The authors list several limitations, namely the restriction to llama.cpp and consumer-grade Intel and RTX 3060 hardware, modest scores on domain-specific reasoning tasks and varying output formats that affect multiple-choice scoring. Source: https://arxiv.org/html/2609.13159v1 Read the article: https://llmobile.news/ticker/mobibench-llamacpp/BenchmarksResearchQuantisationGPURuntimes