https://llmobile.news/tags/android/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/mediatek-dimensity-9600-pro/https://llmobile.news/ticker/mediatek-dimensity-9600-pro/Tue, 15 Sep 2026 00:00:00 +0200MediaTek announced the Dimensity 9600 Pro, a flagship smartphone chip built on a 2nm process. According to the company, its NPU 1090 supports on-device applications with models of up to 30B parameters. MediaTek reports 51% higher LLM prefill performance, the phase where the model reads the prompt, and 55% higher token generation per watt, measured on demo devices in its own labs. The chip pairs the NPU 1090 with a second-generation Super Efficient NPU, which MediaTek says cuts power consumption for always-on AI by 40%. The platform supports LPDDR6 memory and UFS 5.0 storage. In September 2025, MediaTek announced that it had completed the tape-out, the final design handoff to the fab, of a flagship chip on TSMC’s N2P 2nm process, with volume production expected in late 2026. The CPU uses a 2+3+3 layout of eight big cores, with two C2-Ultra cores at up to 4.55 GHz. MediaTek states up to 17% higher single-core and up to 15% higher multi-core performance over the previous generation, and up to 61% lower multi-core power consumption. The first smartphones with the Dimensity 9600 Pro and the related Dimensity 9600M are expected to launch this quarter, according to MediaTek. Source: https://www.mediatek.com/press-room/mediatek-dimensity-9600-pro-sets-new-standard-for-flagship-smartphone-chips Read the article: https://llmobile.news/ticker/mediatek-dimensity-9600-pro/ChipsAndroidSmartphoneMediaTekhttps://llmobile.news/ticker/arm-create-china-2026/https://llmobile.news/ticker/arm-create-china-2026/Fri, 11 Sep 2026 00:00:00 +0200Arm has published five developer takeaways from Arm Create, its developer events in Shanghai and Shenzhen. Two of them concern on-device AI. Arm says model choice starts with the workload and not with model size alone, and that developers should decide which parts of an application stay on the device, which run on nearby edge infrastructure and which need the cloud. The Shenzhen panel included Alibaba Qwen, ModelBest, Tencent Hunyuan and Ultralytics. In the Shanghai keynote, Shantu Roy, Arm’s VP of Developer Relations, discussed the Arm AI Portal. Arm says the portal lists models validated and optimized for Arm-based platforms, together with performance data for specific targets, code and deployment workflows. Coding agents can reach the same information through the Arm MCP Server. The recap shows the portal’s evaluation of Qwen3-TTS 0.6B Custom Voice, a multilingual streaming text-to-speech model from Alibaba, on a mobile CPU. The entry lists a vivo X300 with 8 CPU cores and 16 GB of memory, SME2, the XNNPACK and KleidiAI optimizations, FP16 weights and the LiteRT runtime. It reports a real-time factor of 1.2x against a baseline of 0.28x and a median end-to-end latency of 3,878 ms against 16,877 ms. Peak memory is 4,727 MB against 6,718 MB, and the evaluation uses the English subset of the MiniMaxAI TTS-Multilingual-Test-Set. Evaluation results in the Arm AI Portal, as shown in Arm's recap. Source: Arm. Arm also points to Arm CSS for Mobile 2, which combines the Arm C2 CPU Cluster with SME2 and the Mali G2-Ultra NX GPU. Arm says the platform supports new on-device AI experiences on mobile. The next Arm Create event moves to the US, and Arm has not given a date. Source: https://newsroom.arm.com/blog/takeaways-from-arm-create-china-2026 Read the article: https://llmobile.news/ticker/arm-create-china-2026/ChipsModelsAndroidTTShttps://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/https://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/Thu, 10 Sep 2026 11:00:00 +0200A user has reported results from llama.cpp’s Hexagon NPU backend on a OnePlus 12 with a Snapdragon 8 Gen 3. They cross-compiled llama.cpp on Ubuntu following the project’s Snapdragon documentation and ran it from Termux on the phone. The reported figures are 8 tokens per second of prompt processing and 4.5 of generation with gemma-3-12b-it-qat at Q4_0, and 20 tokens per second of prompt processing and 12.5 of generation with gemma-3-4b-it-qat at Q4_0. According to the report the speed is about the same as running on the same chip’s CPU, but the phone does not get hot. The report lists several limitations. The backend accepts only Q4_0, IQ4_NL, MXFP4, Q8_0 and F32 files, and does not support KV cache quantisation. It runs on chips as old as the Snapdragon 888, but the poster notes that only the 8 Gen 2 and newer carry the tensor module used for language models. The Hexagon NPU can address 4 GB, so a model plus KV cache beyond that has to be split across more than one NPU device: LD_LIBRARY_PATH=./lib:/vendor/lib64 \ ADSP_LIBRARY_PATH=./lib \ GGML_HEXAGON_NDEV=2 \ ./bin/llama-completion \ -m /sdcard/gguf/gemma-3-12b-it-qat-Q4_0.gguf \ -ngl 99 --device HTP0,HTP1 \ -sys 'You are a helpful AI assistant' For context the poster cites 34 INT8 TOPS and 76.8 GB/s of memory bandwidth for the 8 Gen 3 NPU, against 80 TOPS and 228 GB/s for Qualcomm’s X2 Elite Extreme and 248 TOPS and 936 GB/s for an Nvidia RTX 3090. They ask readers with a Snapdragon 8 Elite Gen 5 phone or an X2 Elite Extreme laptop to post their own numbers, since those parts are said to address more than 4 GB and would not need multiple devices. Source: https://www.reddit.com/r/LocalLLaMA/comments/1t0k6fj/running_llamacpp_on_snapdragon_hexagon_npu_seems/ Read the article: https://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/QualcommAndroidNPUllama.cppGemmahttps://llmobile.news/ticker/adk-for-kotlin-1-0/https://llmobile.news/ticker/adk-for-kotlin-1-0/Wed, 09 Sep 2026 00:00:00 +0200Google has released ADK for Kotlin 1.0, the general-availability version of its Agent Development Kit for Kotlin, Java and Android. The kit is built on Kotlin Multiplatform and has full feature parity with ADK 1.0 Core, according to Google. It adds a set of Android extensions for agents that run on the device. The LiteRT-LM module runs open models such as Gemma on the device with tool calling, on Android and on the JVM, and needs JDK 21 or newer according to the project README. A separate ML Kit module runs the built-in Gemini Nano model through the ML Kit GenAI Prompt API. It is Android-only, ships as a 1.0.0-beta and does not support tool calling yet. The documentation says agents can run without an API key and without a network connection at inference time, and developers can combine on-device and cloud models in one multi-agent system, with cloud Gemini models reached through Firebase AI Logic. Further Android modules store chat sessions in a local Room database, keep long-term memory in an on-device AppSearch index and save agent files in app-private storage. Skills, which are bundles of domain-specific prompts and tools, can ship inside the APK’s assets folder. Tools are declared with @Tool and @Param annotations, and a Kotlin Symbol Processing step generates their schemas at compile time with no runtime reflection. ADK for Kotlin is available under the Apache 2.0 license on GitHub, and the documentation covers setup. The repository includes runnable examples, among them a Java interoperability sample. Read the article: https://llmobile.news/ticker/adk-for-kotlin-1-0/RuntimesAndroidhttps://llmobile.news/ticker/arm-mali-g2-ultra-nx/https://llmobile.news/ticker/arm-mali-g2-ultra-nx/Tue, 08 Sep 2026 00:00:00 +0200Arm has introduced the Mali G2-Ultra NX, a smartphone GPU that places dedicated neural accelerators inside each shader core. The accelerators reuse the GPU’s memory system, coherent caches and control structures, support INT8 and INT16 processing and include hardware-accelerated optical flow for motion estimation. Arm calls it the first AI-native Mali GPU and positions the accelerators for neural graphics at 1 W. The GPU is part of the Arm CSS for Mobile 2 platform. The GPU reaches up to 24% higher benchmark performance than the previous generation and 14% higher performance in non-AI gaming workloads, Arm says. Its neural graphics features are Neural Super Sampling, which reconstructs a higher-resolution image from a lower-resolution render, Neural Frame Rate Upscaling, which generates intermediate frames, and Neural Super Sampling and Denoising, which combines upscaling with denoising for ray-traced scenes. Arm says frame rate upscaling supports up to 120 FPS for longer gaming sessions. In its Neural Dawn demo, Arm reports up to 4x higher performance efficiency and up to 70% lower external memory traffic. Arm's overview slide for the Mali G2-Ultra NX. Source: Arm. Arm's diagram of how Neural Frame Rate Upscaling builds an intermediate frame. Source: Arm. The new execution engine is the largest update to the Mali instruction set architecture in seven generations, Arm says, with up to 2x more registers per warp. The third-generation hardware ray tracing unit adds support for Opacity Micromaps, which handle complex transparent geometry. Arm reports up to 13% lower DRAM traffic on ray tracing benchmarks, a 30% higher frame rate and up to 70% less ray tracing work in a scene from Moku’s Central Garden. For developers, Arm offers the Arm Neural Graphics Development Kit with machine learning extensions for Vulkan, plug-ins for Unreal Engine, an SDK for custom engines and tools for profiling, training and model optimization. Arm names integrations with Tencent Games Central Tech’s Magic Dawn engine, Unity China’s Tuanjie Engine and Unreal Engine MegaLights. Keli Zhou, engine lead for Where Winds Meet, says the game will be among the first to bring Arm Neural Technology to players. Arm’s post names no launch dates for devices with the GPU. Source: https://newsroom.arm.com/blog/arm-mali-g2-ultra-nx-ai-native-mobile-graphics Read the article: https://llmobile.news/ticker/arm-mali-g2-ultra-nx/ChipsAndroidhttps://llmobile.news/ticker/arm-css-for-mobile-2/https://llmobile.news/ticker/arm-css-for-mobile-2/Tue, 08 Sep 2026 00:00:00 +0200Arm has introduced Arm CSS for Mobile 2, a compute platform for smartphone chips that combines the C2 CPU cluster, the Mali G2-Ultra NX GPU and the SI L2 system interconnect. The C2 cluster pairs C2-Ultra and C2-Pro CPUs with two SME2 units, the Scalable Matrix Extension 2 that speeds up matrix math for AI on the CPU. Arm says this doubles the SME2 capability of the previous-generation configuration and reports up to 1.7x performance across the latest AI models. The cluster delivers up to 15% higher single-thread performance, 15% faster web browsing, 12% faster app launch and 12% higher multi-thread performance, Arm reports. For AI, it cites a peak uplift of up to 70% in selected tasks. Its slide compares speech, personal memory retrieval and prefill, the phase where a model reads the prompt, against the C1-Ultra with SME2. In a representative agentic workflow covering speech processing, memory retrieval, reasoning, app execution and web browsing, the C2-Ultra with two SME2 units finishes 24% faster than the previous generation, according to Arm. Arm's own comparison of the C2-Ultra with the C1-Ultra, with the AI tasks measured against the C1-Ultra with SME2. Source: Arm. The example flagship configuration in Arm’s slides has two C2-Ultra and six C2-Pro cores. The Mali G2-Ultra NX GPU integrates neural accelerators into its shader cores and adds a new execution engine and a third-generation ray tracing unit for neural graphics. Arm says the SI L2 interconnect provides lower-latency access, higher bandwidth, coherency and quality-of-service controls for CPU, GPU and other resources working at the same time. Partners can use each component on its own or combine them with custom and third-party IP. Arm's slide for the C2-Ultra CPU with an example flagship cluster layout. Source: Arm. On the software side, Arm lists KleidiAI, its optimized libraries for Arm CPUs including SME2 paths, plus integrations with common AI frameworks. The Arm AI Portal offers validated models with performance and accuracy data, code examples and deployment resources, and the Arm MCP Server connects them to agentic development tools. vivo says it is bringing Arm Neural Technology to its latest flagship smartphones built on the platform, aimed at mobile gaming. Arm’s post names no launch dates for devices with CSS for Mobile 2. Source: https://newsroom.arm.com/blog/arm-css-for-mobile-2-and-c2-cpu-cluster Read the article: https://llmobile.news/ticker/arm-css-for-mobile-2/ChipsAndroidhttps://llmobile.news/ticker/gemini-nano-4-first-devices/https://llmobile.news/ticker/gemini-nano-4-first-devices/Thu, 27 Aug 2026 16:00:00 +0200Google’s ML Kit GenAI documentation now lists the first devices running nano-v4, 9to5Google reports. The list covers the Pixel 11, Pixel 11 Pro, Pixel 11 Pro XL and Pixel 11 Pro Fold, plus Samsung’s Galaxy Z Flip8, Galaxy Z Fold8 and Galaxy Z Fold8 Ultra. The same documentation sets Nano v3 or greater as the requirement for Gemini Intelligence, Google’s on-device feature set. According to the report, that requirement first appeared in May 2026, was removed, and has now been reinstated. Listed hardware requirements include 12 GB or more of RAM, a qualified flagship system-on-chip, five or more OS upgrades and six years of security support. Gemini Intelligence features named in the report include Rambler and Proactive Assistance on Pixel 11, and task automation across more than 40 apps on Samsung’s foldables. Source: https://9to5google.com/2026/08/27/gemini-intelligence-nano-4/ Read the article: https://llmobile.news/ticker/gemini-nano-4-first-devices/GoogleAndroidPixelSamsungGemini Nanohttps://llmobile.news/ticker/ornith-1-5/https://llmobile.news/ticker/ornith-1-5/Wed, 19 Aug 2026 00:00:00 +0200Ornith has released Ornith-1.5, a model family with a 9B dense model, a 35B mixture-of-experts model that activates about 3B parameters per token and a 397B mixture-of-experts model. The 9B model also comes as Ornith-1.5-9B-Mobile, a quantized variant (about 1,5 GB on disk) designed for iPhone and Android. The official weights are on Hugging Face. The 9B model scores 47.0 on Terminal-Bench 2.1 with the Claude Code harness and 70.6 on SWE-bench Verified in Ornith’s tests. The company says the model matches or exceeds much larger models such as Gemma 4-31B and Qwen 3.6-35B. In the company’s chart, Qwen3.6-35B-A3B leads on SWE-bench Verified with 73.4 and on Terminal-Bench 2.1 with 52.5, while the 9B model scores 86.4 on GPQA Diamond and 54.2 on MCP-Atlas. The previous Ornith-1.0-9B reaches 43.1 on Terminal-Bench 2.1 in the same chart. Ornith's own benchmark figures for the 9B model. According to Ornith, its training loop lets the model propose new tasks, generate task-specific scaffolds and produce solution rollouts, with the reward from the rollouts propagated across all three stages. The company reports that the 35B model scores 67.8 on Terminal-Bench 2.1 with the Terminus-2 harness, against 52.5 for Qwen 3.6-35B. The models are on Hugging Face, with GGUF builds of all three sizes and MLX builds of the 9B and 35B models. Source: https://ornith.ai/ornith_1_5.html Read the article: https://llmobile.news/ticker/ornith-1-5/ModelsAppleAndroidhttps://llmobile.news/ticker/online-sdft/https://llmobile.news/ticker/online-sdft/Tue, 18 Aug 2026 16:57:46 +0200I-Ju Lin and Zhang-Wei Hong published Online-SDFT, a method that keeps fine-tuning a small language model on a phone from the user’s own stream of decisions. On a synthetic notification-routing benchmark with Liquid AI’s LFM2.5-230M, the authors report 70.28% of decisions matching a hidden user preference against 52.78% for the strongest baseline they tested, and cumulative regret of 44.76 against 105.26, over 720 decisions in three paired streams. They state that this comparison ran on Apple MPS rather than on a phone, and that the confidence intervals are nominal across only three streams. Self-distillation here means the model teaches itself. The model picks an action first, the phone later observes what happened, and the same model is then run again with that outcome added to its input. Its better-informed answer becomes the training target for the version that saw only the original context, so no second teacher model, no human label and no reward score is needed. The authors keep the 230M base frozen and train a LoRA adapter of 172,032 parameters across 48 tensors in six attention layers, with the teacher pass running with the adapter switched off so student updates cannot move the teacher. Feeding one interaction at a time into so small an adapter makes it noisy and prone to forgetting, so the authors keep a 64-row buffer of recent lessons and sample each batch from it. Their ablation puts that buffer ahead of the exploration schedule as the load-bearing part, with accuracy dropping from 70.28% to 39.58% and regret climbing from 44.76 to 134.90 once replay is removed, while dropping exploration instead leaves accuracy at 69.44% and regret at 62.00. Diagram: I-Ju Lin and Zhang-Wei Hong. The repository carries an Android proof of concept in which LFM2.5-230M decides and ONNX Runtime Training performs the adapter update on the device, with replay and checkpoints in app-private storage that survive a restart. Exporting the model still happens on a Linux host, and the deployment guide asks for Android 12 or newer on ARM64 with 12 GB of RAM at minimum, 16 GB preferred, and 8 GB of free storage, because the training graph is FP32. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: I-Ju Lin and Zhang-Wei Hong. Watch on the project page The authors call that graph a correctness implementation rather than a claim of practical background training on low-memory phones, and state that they did not measure Android latency, peak memory, battery use or thermal behaviour, and that the benchmark uses synthetic streams rather than real users. The code, the Android app and a Colab notebook are on GitHub under the MIT license. No paper or preprint accompanies the release. Source: https://lin826.github.io/SLM-Online-SDFT/ Read the article: https://llmobile.news/ticker/online-sdft/ResearchDistillationAndroidLFMOpen sourcehttps://llmobile.news/ticker/rikkahub-agent-local-llm/https://llmobile.news/ticker/rikkahub-agent-local-llm/Sat, 08 Aug 2026 00:00:00 +0200Adam Conway at XDA has tested RikkaHub Agent, a fork by ExTV of the open-source Android LLM client RikkaHub, which adds an agent layer with more than 80 device tools. The phone runs the agent loop and executes the tools; the model itself runs elsewhere on the local network over an OpenAI-compatible endpoint. The test device was an Oppo Find N5 with a Snapdragon 8 Elite and 16 GB of RAM. Inference ran on a Radeon RX 7900 XTX with Qwen 3.6 27B at Q4_K_M, roughly 16 GB, at 40 to 50 tokens per second. A second configuration used Qwen 3.6 35B on a Lenovo ThinkStation PGX with Nvidia’s GB10 Grace Blackwell. Screenshot: XDA. The tool set covers tapping, swiping, scrolling and typing on screen, reading notifications, sending SMS, file management, browser control, SSH, scheduled jobs, a JavaScript runtime and Telegram. All tools are off by default and enabled per assistant, actions with side effects require approval, and a deterministic blocklist rejects destructive commands. The default context is 7,000 input tokens, rising to 32,000 with every tool enabled. Asked whether whisper.cpp was installed, the agent determined that neither the command-line tool nor a model was present, requested permission, then installed the dependencies, cloned the repository and compiled it inside Termux. The build took seven minutes, and the agent chose the 75 MB small model on speed and accuracy grounds. It then transcribed a Telegram voice message, installing ffmpeg, converting the OGG/Opus file to 16 kHz mono WAV and returning a transcript in about two seconds. Running the model on the phone itself was slower. With Gemma 4 E2B, a 2.6 GB build accelerated by LiteRT-LM, Conway reports chat worked but agentic tasks were painful, and tool calls were emitted as plain text until an update fixed the behaviour. Frontier intelligence is never what I’d want from a phone agent, which is why a local LLM works perfectly. Source: https://www.xda-developers.com/turned-phone-ai-agent-powered-local-llm-installed-software/ Read the article: https://llmobile.news/ticker/rikkahub-agent-local-llm/AndroidAgentsQwenOpen sourceSpeechhttps://llmobile.news/ticker/gemini-nano-4-mlkit-prompt-api/https://llmobile.news/ticker/gemini-nano-4-mlkit-prompt-api/Wed, 22 Jul 2026 18:00:00 +0200Google’s Android developer blog states that Samsung’s new foldable devices come with Gemini Nano 4, which it calls its latest on-device model. The post credits Nano 4 with support for more than 140 languages and better multimodal understanding. Apps reach the model through ML Kit’s Prompt API, which sends natural language requests on-device to Gemini Nano. It takes text, or a combination of image and text, and returns text or structured output. Google names structured output and thinking mode as the features to use for on-device intelligence. The ML Kit release notes dated 14 July 2026 record the structured output API, system instructions and thinking mode arriving in the Prompt API, along with multi-image support and an output token limit raised to 4,096 tokens. A note dated 21 July records a fix for Gemini Nano v4 compatibility in the Prompt API on non-Pixel devices. The Prompt API moved from alpha to beta in January 2026 and carries no service level agreement or deprecation policy. The post also points developers to app functions, which share an app’s capabilities with the Gemini Intelligence system. Its remaining sections cover adaptive layouts, fold-aware design, CameraX and Wear OS widgets. Source: https://android-developers.googleblog.com/2026/07/optimize-galaxy-screen-sizes.html Read the article: https://llmobile.news/ticker/gemini-nano-4-mlkit-prompt-api/GoogleAndroidSamsungGemini NanoDeveloper toolshttps://llmobile.news/ticker/fblayout/https://llmobile.news/ticker/fblayout/Tue, 07 Jul 2026 12:18:25 +0200Researchers at the University of Macau, the University of Georgia and the Hong Kong University of Science and Technology published FBLayout on July 7, 2026, a system that fine-tunes transformer models on a phone’s own GPU, accepted at ACM MobiSys 2026. On a OnePlus Ace 5 Pro with a Snapdragon 8 Elite and its Adreno 830 GPU, the authors measure training on Llama 3.2 1B, Qwen2.5 1.5B and Gemma 2 2B at 3.9 to 4.1 times the speed of MNN, 4.3 to 4.9 times TFLite and 5.4 to 5.7 times TVM. Over the same three baselines they count 3.5 times fewer global memory reads and 4.2 times fewer cache misses. The authors time inference on the Snapdragon 8 Elite at roughly 6.5 times faster on the GPU than on the CPU, while fine-tuning models of 0.5B to 2B parameters on the GPU comes out level with the CPU or up to 43 percent worse. Their explanation is memory layout, meaning the order in which a tensor’s numbers physically sit in memory. Mobile GPUs hold tensors in a 2D texture grid whose cache rewards reading neighbouring cells, and training reuses forward-pass activations and weights in the backward pass, which sums along a different dimension and so walks the same tensor in strides instead of in rows. Each stride that misses the cache becomes a fetch from DRAM at under 100 GB/s of mobile memory bandwidth, and the GPU stalls waiting for it. Existing frameworks take one of two ways out, and the paper prices both. MNN reuses the forward layout in the backward pass, which the authors put at up to a 5.3 times slowdown on key operations and at under 21 percent arithmetic-unit utilisation in a matrix-multiply benchmark. TFLite instead inserts explicit transpose and reshape operators, and the paper puts that route at 40.8 to 50.0 percent of total runtime across the three LLMs and at roughly a third of all memory allocations. FBLayout defines one tile-based layout, called R-Tile, that keeps both directions contiguous, then rewrites the remaining reshapes as coordinate arithmetic so no data moves, which they report covers over 86 percent of all layout transformations and over 95 percent of those in the LLMs. The evaluation covers seven models, adding BERT-Large, ViT-Large, Whisper-Large and Stable Diffusion 1.5 to the three LLMs, with LoRA adapters in the attention and MLP blocks for everything except ViT. Beyond the Snapdragon 8 Elite phone the authors test a OnePlus Ace 10 Pro with a Snapdragon 8 Gen 1 and an Adreno 730 and a OnePlus Ace 5 Ultra with a Dimensity 9400+ and an Arm Mali Immortalis-G925, and report gains on both. Preparing the Llama 3.2 1B training graph takes 6.1 seconds in their measurement, against 723 seconds in MNN, 862 in TFLite and over 4,800 in TVM, and a Monsoon power monitor puts energy use over five training rounds at 3.5 to 6.3 times below the baselines. FBLayout is built on top of Alibaba’s MNN as about 9.2K lines of C++ and OpenCL, extending it with automatic differentiation so the training graphs exist in the first place. The paper names no code release and points to no repository. It appeared at MobiSys 2026 in Cambridge in June 2026 and is on arXiv under a CC BY-NC-ND 4.0 license. Source: https://arxiv.org/abs/2607.21624 Read the article: https://llmobile.news/ticker/fblayout/ResearchMemoryQualcommMediaTekAndroidhttps://llmobile.news/ticker/mlperf-inference-mobile/https://llmobile.news/ticker/mlperf-inference-mobile/Mon, 15 Jun 2026 17:00:00 +0200MLCommons announced MLPerf Mobile v6.0 on June 15, 2026, adding generative AI tests that run Llama on an Android device and report token throughput. The new workload comes in 1B, 3B and 8B parameter sizes, using Llama 3.2 1B Instruct, Llama 3.2 3B Instruct and Llama 3.1 8B Instruct. Each size runs twice, once against TinyMMLU, a small multiple-choice knowledge set, and once against IFEval, which checks whether a model obeys the instructions in a prompt, according to the task configuration in the app repository. MLCommons says the tests run on devices with enough memory through the CPU without tailored acceleration, and that v6.0 adds NPU-accelerated execution of Llama 3.1 8B Instruct on Qualcomm Snapdragon 8 Elite Gen 5. The v6.0 app release of June 9, 2026 also brings support for MediaTek Dimensity 9500 chips and updates support for the Samsung Exynos 2600. The rest of the suite covers image classification with MobileNetV4 on ImageNet, object detection on COCO 2017, segmentation with MOSAIC on ADE20K, language understanding with MobileBERT on SQuAD 1.1, super resolution with EDSR, and Stable Diffusion 1.5 on COCO captions, as the benchmark page lists them. Almost every test runs in the single stream scenario, where the score is the 90th percentile latency for one query at a time, and image classification also runs an offline pass scored as throughput. Stable Diffusion, the first generative test in the suite, arrived in the v4.1 app release of February 11, 2025. Mostafa El-Khamy, co-chair of the MLPerf Mobile working group, had set out the plan in May 2024. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite. Results come from chip and device makers that run the app and submit their logs. The published v6.0 results hold one submission in the closed division, from Samsung on an Exynos 2600 phone, and it covers the vision tests, language understanding, super resolution and Stable Diffusion rather than the new Llama ones. Samsung’s logs give a 90th percentile single stream latency of 0.56 ms for image classification, 0.83 ms for MobileBERT and 1.87 s for one Stable Diffusion image. The v5.0 round carried submissions from Samsung and Qualcomm Innovation Center. Screenshots: Samsung's submission to MLPerf Mobile v6.0, published by MLCommons. Submissions are held back until review closes under the MLCommons submission rules, which keep every result confidential inside a review committee after the deadline and let submitters file objections that the committee then decides. MLCommons publishes the accepted runs in the results table on the MLPerf Inference Mobile page, with the logs and calibration notes in a per-round repository on GitHub. The app is available through Google Play, the Apple App Store and the MLPerf Mobile repository under the Apache 2.0 license. Source: https://mlcommons.org/2026/06/mlperf-mobile-v6/ Read the article: https://llmobile.news/ticker/mlperf-inference-mobile/BenchmarksAndroidLlamaNPUQualcommhttps://llmobile.news/ticker/on-device-diffusion-llm-npu/https://llmobile.news/ticker/on-device-diffusion-llm-npu/Thu, 11 Jun 2026 14:44:00 +0200Tuowei Wang, Yanfan Sun and Ju Ren of Tsinghua University and Beihang University published llada.cpp on June 11, 2026, an inference framework that runs diffusion language models on a phone’s NPU. On an OnePlus Ace5 Pro with a Snapdragon 8 Elite, the authors measure LLaDA-8B-Instruct generating 128 tokens 38 times faster than the same model on the phone’s CPU for GSM8K prompts and 42 times faster on BoolQ, and they report a range of 17 times to 42 times across their phones and tasks. The same figure puts Llama-3-8B-Instruct on the NPU at 19 to 21 times that CPU baseline, and the authors state that llada.cpp runs LLaDA-8B up to 3.9 times faster than an autoregressive model of the same size. Chart: Wang, Sun and Ren, CC BY 4.0. A diffusion language model does not emit one token after another. It starts from a block of masked positions, 32 tokens by default here, and refines all of them together over repeated denoising steps, committing the positions it is most confident about and carrying the rest into the next step. That turns decoding into a run of large matrix operations, which is the work a mobile NPU is built for, while autoregressive decoding hands the NPU one token’s worth of work per step. The cost the paper identifies is that every denoising step recomputes the full sequence, so the parallel work gets repeated many times over. llada.cpp is built on llama.cpp and targets Qualcomm’s Hexagon NPU with 4-bit weights. Once a block is nearly done only a few masked tokens are left, so the framework pulls tokens from the next block into the same NPU pass to keep it full, keeps already committed tokens open to revision and repairs the unstable ones on the CPU while the NPU carries on, and repacks the limited address space the NPU can see so weights and cache do not have to be remapped between steps. The authors’ breakdown of a 128-token GSM8K request on the Snapdragon 8 Elite puts it at 2996.2 s on the CPU, 607.0 s once the prefix KV cache is reused, and 16.1 s with the full framework. Accuracy moves in both directions. Measured on 200-sample subsets against LLaDA-8B-Instruct on the CPU, the authors’ table has GSM8K rising from 39.0 to 43.5 and ARC-C from 84.0 to 85.0, while BoolQ falls from 82.5 to 80.5 and HellaSwag from 51.0 to 49.5, and their ablation shows the borrowed future-block tokens costing accuracy on all four tasks before the CPU-side revision path recovers it. The work covers three 16 GB phones, an OnePlus 12 with a Snapdragon 8 Gen 3, the Ace5 Pro, and an OnePlus 15 with a Snapdragon 8 Elite Gen 5, and the authors state that on the Snapdragon 8 Gen 3 their framework is still slower than Llama-3-8B at 128 tokens and only pulls ahead on the two newer chips. The paper is on arXiv under CC BY 4.0 and names no code release. Source: https://arxiv.org/abs/2606.13740 Read the article: https://llmobile.news/ticker/on-device-diffusion-llm-npu/QualcommNPUAndroidllama.cppResearchhttps://llmobile.news/ticker/caped-privacy-defence/https://llmobile.news/ticker/caped-privacy-defence/Wed, 10 Jun 2026 22:48:00 +0200Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames. A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all. Figure: Shen et al., licensed CC BY 4.0. The mock SunShop task asks the agent to buy sunglasses, so the personalised rows around it count as incidental exposure. CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow. The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements. The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version. Source: https://arxiv.org/abs/2606.12666 Read the article: https://llmobile.news/ticker/caped-privacy-defence/AgentsAndroidPrivacyResearchBenchmarkshttps://llmobile.news/ticker/litert-lm-on-device-genai/https://llmobile.news/ticker/litert-lm-on-device-genai/Tue, 19 May 2026 18:00:00 +0200Google has detailed LiteRT-LM, the runtime it uses to run Gemma 4 on-device in Chrome, ChromeOS, the Pixel Watch and the Google AI Edge Gallery app. It sits on top of LiteRT, formerly TensorFlow Lite, and adds GenAI libraries over the XNNPACK and ML Drift kernels. Running Gemma 4 E2B without multi-token prediction, Google measures 52 decode tokens per second on the Android GPU through OpenCL, 56 on iOS through Metal, and up to 76 in the browser on WebGPU. The Android figures come from a Samsung S26 Ultra, iOS from an iPhone 17 Pro, and web from Chrome on a 2024 MacBook Pro with an M4 Max. Chart: Google. The published comparison puts LiteRT-LM prefill at 3,808 tokens per second against 292 for llama.cpp on the Android GPU, 2,878 against 1,493 for MLX on iOS, and 4,182 against 1,127 for ONNX on the web. Decode figures in the same chart are 52 against 19, 56 against 32 and 76 against 29. Multi-token prediction raises decode further. Google reports up to a 2.2x speedup, taking Gemma 4 E2B from 52 to 85 tokens per second and E4B from 21 to 47, measured on a Samsung S26 Ultra GPU. The drafter and the primary model run on the same hardware block so the shared KV cache and activations stay in local memory. Chart: Google. On memory, LiteRT-LM keeps per-layer embeddings out of memory and loads image and audio encoders only when a task needs them. Google states that the 2.58 GB Gemma 4 E2B model runs with a physical footprint of 607 MB on Apple mobile CPUs using XNNPACK weight caching. For agentic use the runtime supports Thinking Mode, which developers can stream to the interface or strip to save KV cache space: By dedicating a scratchpad for step-by-step reasoning before the model commits to an action, LiteRT-LM can significantly improve the output quality. Constrained decoding enforces JSON schemas on tool payloads, and native function calling pauses execution, returns the tool-call request to the app and resumes on the tool’s output. Session save and restore serialise KV cache state so long conversations skip the prefill phase on return. Beyond the existing Kotlin and C++ interfaces on Android, Google is adding an open-source Swift API for Apple platforms and a JavaScript API for the web through WebAssembly and WebGPU. Source: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/ Read the article: https://llmobile.news/ticker/litert-lm-on-device-genai/GoogleLiteRTGemmaBenchmarksAndroidhttps://llmobile.news/ticker/airgap/https://llmobile.news/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.news/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.news/ticker/mobilellm-flash/https://llmobile.news/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.news/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.news/ticker/clawmobile/https://llmobile.news/ticker/clawmobile/Thu, 26 Feb 2026 13:34:00 +0100Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against. What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning. Architecture diagram: Du et al., CC BY 4.0. The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks. The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license. Source: https://arxiv.org/abs/2602.22942 Read the article: https://llmobile.news/ticker/clawmobile/AgentsAndroidPixelResearchOpen sourcehttps://llmobile.news/ticker/mobile-agent-v3-5/https://llmobile.news/ticker/mobile-agent-v3-5/Sun, 15 Feb 2026 02:52:00 +0100Alibaba’s Tongyi Lab published Mobile-Agent-v3.5 on February 15, 2026, describing GUI-Owl-1.5, a family of agent models that read a screenshot and emit the next tap, type or scroll. The family spans 2B, 4B, 8B, 32B and 235B-A22B parameters in instruct and reasoning variants, all built on Qwen3-VL. Every benchmark in the paper was run server-side against cloud sandboxes and emulators, and the authors give no on-device deployment, latency or hardware figures anywhere, so the small sizes are the only thing that points at a phone. The authors write that the smaller instruct variants, which skip the reasoning step, are fast enough to be deployed on edge devices and to work alongside a larger cloud model. Across platforms, the authors report their 32B-Instruct model reaching 56.5 on OSWorld-Verified for desktop tasks, 47.6 on OSWorld-MCP and 46.8 on Mobile-World, the last two of which also require calling external tools. On AndroidWorld, which scores an agent driving real Android apps in an emulator, their 8B reasoning model reaches 71.6, which the paper puts just behind UI-TARS-2 at 73.3. For browsers, the paper’s table has the 32B reasoning model at 48.4 on WebArena, 46.6 on VisualWebArena and 82.1 on WebVoyager. The small checkpoints carry a large part of that. The authors report 2B-Instruct at 43.5 on OSWorld-Verified and 67.9 on AndroidWorld, and 4B-Instruct at 48.2 and 69.8, which puts the 4B model within 2 points of their own 32B on Android tasks. On ScreenSpot-Pro, which tests whether a model can point at the right element in a high-resolution screenshot, they report 72.9 for 32B-Instruct and 80.3 when the model first crops a region and looks again. The paper’s prose and its tables disagree on which variant owns several headline numbers, and the figures above follow the tables. Charts: Alibaba Tongyi Lab, the authors' own benchmark figures. Six checkpoints are on Hugging Face under the MIT license, covering 2B, 4B and 8B instruct, 8B reasoning, and 32B in both variants. The 235B-A22B model named in the paper is not among them. The code is MIT as well, and the repository points at a hosted cloud-sandbox demo rather than a local runtime. Source: https://arxiv.org/abs/2602.16855 Read the article: https://llmobile.news/ticker/mobile-agent-v3-5/AlibabaAgentsQwenOpen weightsAndroidhttps://llmobile.news/ticker/off-grid-offline-ai-app/https://llmobile.news/ticker/off-grid-offline-ai-app/Sat, 14 Feb 2026 23:39:00 +0100Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT. The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite. The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU. Comments disputed whether models of that size are usable. One reader wrote: Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations. Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”. Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day. Source: https://news.ycombinator.com/item?id=47019133 Read the article: https://llmobile.news/ticker/off-grid-offline-ai-app/AndroidiOSOpen sourcellama.cppImage generationhttps://llmobile.news/ticker/mobilellm-pro/https://llmobile.news/ticker/mobilellm-pro/Mon, 10 Nov 2025 06:28:00 +0100Meta Reality Labs published the MobileLLM-Pro technical report on November 10, 2025, describing a 1.08B-parameter language model built for phones and wearables. The model handles a context window of 128,000 tokens, four times what Gemma 3 1B and Qwen 3 0.6B accept. Quantising weights and embeddings to 4 bits takes the file from 2.2 GB in bf16 down to 590 MB for the CPU build and 720 MB for the accelerator build, which cannot share embedding weights. Meta exported the model through ExecuTorch and measured it on a Samsung Galaxy S25 CPU with the XNNPACK backend and on a Samsung Galaxy S24 Hexagon Tensor Processor. On a 2,000-token prompt the paper reports 8.9 s of prefill on the CPU against 2.0 s on the Hexagon unit, and decode speeds of 33.6 tok/s and 31.6 tok/s while generating 1,000 tokens. At 8,000 tokens prefill climbs to 63.5 s on the CPU and 9.8 s on the Hexagon unit, decode falls to 19.7 and 22.8 tok/s, and the key-value cache grows from 14 MB to 40 MB. The architecture puts 30 layers of width 1280 behind 20 attention heads and 4 key-value heads, and shares one weight matrix between the input embedding and the output head, which the authors say saves 260M parameters, about a quarter of the model. Three out of every four layers use local attention, comparing each token only against the 512 most recent ones, with a global layer every fourth one and at both ends. Meta’s model card credits that interleaving with cutting the key-value cache at 8,000 tokens from 117 MB to 40 MB and with 1.8 times lower prefill latency than global attention in every layer. Meta trained the model by distilling the logits of Llama 4 Scout under a KL divergence loss across three pretraining phases on roughly 1.64T tokens of open data, the second of them a step the authors call implicit positional distillation, which stretches the context window to 128k by having the teacher pass on the ability rather than feeding the student long documents. Meta puts the base model 5.7 and 7.9 points above Gemma 3 1B and Llama 3.2 1B on average across the benchmarks its model card lists, and the paper’s table has it at 76.24 on BoolQ against 63.20 and 62.51, at 52.62 on ARC-Challenge against 38.40 and 38.28, and at 100% needle-in-a-haystack retrieval against 96.80 for Llama 3.2 1B. The instruction-tuned version scores 59.8 on HumanEval against 41.5 and 37.8, 44.8 on MMLU against Gemma’s 29.9 and Llama’s 49.3, and 62.0 on IFEval against Gemma’s 80.2. In Meta’s own human evaluation over 100 prompts per task, raters preferred Gemma 3 1B for summarisation 51 to 47 and for rewriting 49 to 45, while MobileLLM-Pro led on recall 47 to 30. Meta released four checkpoints on Hugging Face, the base model, the instruction-tuned one, and two quantisation-ready base checkpoints produced by quantisation-aware training with self-distillation from the full-precision model. One uses 4-bit group-wise weights at group size 32 with 8-bit dynamic activations and an 8-bit key-value cache for CPU backends, the other 4-bit channel-wise weights for the Apple Neural Engine and Qualcomm’s Hexagon processor, which handle group-wise quantisation poorly or not at all. Needle-in-a-haystack retrieval holds at 96.44% and 98.67% against 100% at full precision. The weights sit behind a request form and carry the FAIR Noncommercial Research License. Source: https://arxiv.org/abs/2511.06719 Read the article: https://llmobile.news/ticker/mobilellm-pro/MetaResearchDistillationQuantisationAndroidhttps://llmobile.news/ticker/confidant-collaborative-training/https://llmobile.news/ticker/confidant-collaborative-training/Mon, 03 Nov 2025 10:00:00 +0100Researchers at Zhejiang University, Zhejiang University of Technology and Hangzhou Dianzi University published Confidant in the ACM MobiCom 2025 proceedings on November 3, 2025, a day before the conference opened in Hong Kong. The framework fine-tunes a language model across several off-the-shelf devices instead of one, and the authors fine-tuned Phi2-2.7B on the Alpaca instruction dataset in 40.1 hours using three of them. Their write-up of that run names a Redmi K50 phone, a Core i7-13650HX laptop and a MacBook Pro with an M1 Pro, while the paper’s abstract and conclusion describe the same result as three mobile devices. The same trio fine-tuned BERT-Base in 2.2 hours. Confidant cuts a model into consecutive slices, gives each device one slice, and passes activations from one device to the next, so no device ever holds the whole model. Batches flow through that chain as a pipeline, with each device starting a new batch while the devices behind it are still working on earlier ones. The device that holds the training data acts as coordinator and recalculates where the cuts fall during training, using each participant’s measured compute time, free memory and bandwidth, and it discards any split whose slice would not fit in a device’s memory. Inside a device, a second scheduler profiles the CPU and the GPU on varying numbers of attention heads and gives each processor as many heads as it can finish in roughly the same time, which is what the authors credit for running up to 1.94 times faster than a capacity-aware pipeline baseline they implemented themselves, and up to 3.82 times faster than the best single device, both on GPT-2 Medium. Diagram: Chen et al., Figure 3 of the paper, licensed CC BY 4.0. Splitting the model is what brings the footprint per device down. The paper’s memory table puts fine-tuning BERT-Base at a batch size of 8 at 5.5 GB on a single Redmi K50, against a per-device average of 3.17 GB across three devices and 2.43 GB across four. Larger models leave no single-device figure to compare with, because a Redmi K50 fits only 1 of Phi2-2.7B’s 32 transformer blocks at a batch size of 4, and the per-device averages of 15.31 GB for Phi2-2.7B and 32.07 GB for LLaMA3-8B across three devices are carried by the two laptops rather than the phone. A separate table puts the memory PyTorch needs to fine-tune Phi2-2.7B at a batch size of 8 at 46.5 GB, next to the 8 GB to 12 GB of the four phones it lists. The paper’s prose says Confidant almost does not cause any accuracy drop, while its own model performance table puts the three-device runs at a test loss of 0.66 on Phi2-2.7B and 3.55 on LLaMA3-8B, against 0.52 and 2.86 for fine-tuning the same models the conventional way on a single machine, where a lower loss is better. The four-device runs land at 0.76 and 2.85 on that same comparison, and BERT-Base token classification accuracy holds at 97.66% against 97.70%. Keeping data on the device is the stated motivation. The authors write that training with Confidant does not require sharing raw data and that devices exchange only intermediate outputs and model gradients, which they present as the privacy argument for their two scenarios, a household assistant fine-tuned on family data during idle hours and a conference assistant fine-tuned on attendees’ phones. They also state that fine-tuning on a single cloud server would be faster and more straightforward, and that their aim is to use idle mobile hardware instead. The Android side is built on Alibaba’s MNN framework with laptops running PyTorch, about 10,900 lines of code in total, the paper is published under a Creative Commons Attribution 4.0 license, and the authors wrote that they planned to release the code in late 2025. Source: https://dl.acm.org/doi/10.1145/3680207.3723487 Read the article: https://llmobile.news/ticker/confidant-collaborative-training/ResearchMemoryPhiLlamaAndroidhttps://llmobile.news/ticker/mobilellm-r1/https://llmobile.news/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.news/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.news/ticker/shadownpu/https://llmobile.news/ticker/shadownpu/Fri, 22 Aug 2025 09:41:00 +0200Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026. The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100. On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use. Diagram: Yin et al. The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat. Source: https://arxiv.org/abs/2508.16703 Read the article: https://llmobile.news/ticker/shadownpu/QualcommNPUResearchQwenAndroidhttps://llmobile.news/ticker/demystifying-slms-edge/https://llmobile.news/ticker/demystifying-slms-edge/Sun, 27 Jul 2025 09:00:00 +0200Ten researchers at Beijing University of Posts and Telecommunications, Peng Cheng Laboratory, the University of Cambridge, Xiaomi, Helixon Research and Flower Labs collected 68 small language models between 100M and 5B parameters, released by 24 organisations between OPT in May 2022 and Llama 3.2 in September 2024, and measured both their accuracy and their cost on edge hardware. In Demystifying Small Language Models for Edge Deployment, presented at ACL 2025, Microsoft’s Phi-3 and Phi-3.5-mini sit highest on the capability chart at about 70 percent average accuracy over 10 reasoning and knowledge datasets, level with the Llama 3.1 line the authors draw for comparison. The runtime half of the study covers the 20 models that llama.cpp supports, quantised to 4-bit and run with a 50-token prompt and 50 generated tokens. Chart: Lu et al., licensed CC BY 4.0. Accuracy is averaged over the 10 datasets with math excluded, and circle size is proportional to model size. Parameter count predicts on-device latency poorly. The measurements put OpenELM-3B at 9.4 percent more parameters than Phi-2 but 9.9 percent faster to the first token, and Qwen1.5-0.5B at 25.4 percent more parameters than Qwen2-0.5B yet 31.9 percent faster to the first token and 11.8 percent faster per decoded token. Qwen1.5-1.8B carries 30.5 percent more parameters than Pythia-1.4B and still reaches the first token 18.5 percent sooner. The authors trace the gap to how a model spends its parameter budget, with Qwen2 sharing its input embedding and output layer and putting the saved weights into wider attention and feed-forward layers, which costs compute in the prefill stage. Memory ran from 275 MB to 2456 MB across the 20 models, measured with llama.cpp on a Jetson Orin NX at a fixed 2048-token context. Vocabulary size moves that figure as much as parameter count does, and the paper reports Bloom-560M needing 492 MB of compute buffer for its 250,880-token vocabulary, 3.5 times what OpenELM-1.1B needs with 32,000 tokens. OpenELM-1.1B uses 23.8 percent less memory than Bloom-560M despite 32.3 percent more parameters, and OpenELM-3B 13.8 percent less than Gemma-2B despite 21.1 percent more, which the authors put down to the small vocabulary and to grouped-query attention, where several attention heads share one set of cached keys and values. Raising each model to its own maximum context window pushes Qwen2-0.5B past 6 GB, with compute buffer and KV cache making up 85 percent of the total for the Qwen2 models. Charts: Lu et al., licensed CC BY 4.0. Measured with llama.cpp at 4-bit, and memory taken on a Jetson Orin NX at a 2048-token context. Quantisation paid off in decoding rather than in prefill. Testing five settings on Phi-1.5, the authors found Q4_K_M the strongest, roughly halving per-token decode latency against FP16, while at a 50-token prompt it cut the time to the first token by only 13 percent. Q6_K and Q3_K matched or exceeded FP16 on longer prompts, which the paper puts down to the alignment and padding their irregular bit widths force. The hardware gap was wider than the quantisation gap, with the Jetson’s GPU 40 times faster than the Meizu 18 Pro’s CPU at prefill but only 1.84 times faster at decode, and the Xiaomi 12S on a Snapdragon 8 Gen 1+ the fastest of the three phones, ahead of the Pixel 7 Pro and the Meizu on its Snapdragon 888. The authors name two limits on their own work. They left math datasets out of the capability evaluation because of the gap between small models and larger ones on mathematical reasoning, and they confined the cost analysis to the 20 models llama.cpp supports so that the inference engine stayed constant, excluding every model it cannot load. The paper is free to read at the ACL Anthology under a Creative Commons Attribution 4.0 licence, and the same group had earlier measured 22 models from 0.5B to 7B on four Android phones, three of which reappear in this test bed. PalmBench adds power draw and surface temperature on comparable hardware, and Artificial Analysis has since reported quality, speed and memory together for 33 quantised models on one iPhone 17 Pro. Source: https://aclanthology.org/2025.acl-long.718/ Read the article: https://llmobile.news/ticker/demystifying-slms-edge/BenchmarksResearchllama.cppMemoryAndroidhttps://llmobile.news/ticker/gemma-3n/https://llmobile.news/ticker/gemma-3n/Tue, 20 May 2025 19:00:00 +0200Google previewed Gemma 3n on May 20, 2025, a model built for phones, tablets and laptops. Its 5B and 8B versions run with dynamic memory footprints of 2 GB and 3 GB, which Google says matches what 2B and 4B models normally need. The saving comes from Per-Layer Embeddings, a Google DeepMind technique that keeps embedding parameters off the accelerator. For E2B that cuts what has to be loaded into accelerator memory from 5.44B parameters to 1.91B, with the 2.55B embedding parameters cached to fast storage instead. A MatFormer setup nests a smaller but fully functional 2B model inside the 4B one, so developers can trade quality against footprint. Google reports the model starting to respond about 1.5 times faster on mobile than Gemma 3 4B. Gemma 3n takes audio, text, images and video as input, including interleaved inputs across modalities, and handles speech recognition and translation. Google states that the same architecture powers the next generation of Gemini Nano, which was to reach Android and Chrome later in 2025. Qualcomm Technologies, MediaTek and Samsung System LSI worked on the optimisation. Diagram: Google. ▶Announcing Gemma 3n Preview: Powerful, Efficient, Mobile-First AILoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Google for Developers. Update, June 26, 2025. Google released Gemma 3n as E2B and E4B, naming them for 2B and 4B effective parameters out of 5B and 8B raw ones. Google reports E4B scoring 1303 on LMArena and calls it the first model under 10 billion parameters to pass 1300, ahead of Llama 4 Maverick 17B 128E at 1292 and GPT 4.1-nano at 1288. The release adds a MobileNet-V5-300M vision encoder, which Google says handles up to 60 frames per second on a Pixel, and KV cache sharing that doubles prefill performance. Text covers 140 languages, multimodal understanding 35. Weights are on Hugging Face and Kaggle, with support in Ollama, llama.cpp, MLX, LiteRT and the Google AI Edge Gallery. Chart: Google. Source: https://developers.googleblog.com/en/introducing-gemma-3n/ Read the article: https://llmobile.news/ticker/gemma-3n/GoogleGemmaGemini NanoAndroidOpen weightshttps://llmobile.news/ticker/googleai-edge-gallery/https://llmobile.news/ticker/googleai-edge-gallery/Sun, 18 May 2025 00:21:00 +0200Google’s AI Edge team has published Google AI Edge Gallery, an app that runs generative models entirely on the device. The home screen offers three tasks, AI Chat for multi-turn conversation, Ask Image for questions about a photo from the camera or the gallery, and Prompt Lab for single-turn work, which covers free-form prompts, rewriting text in a chosen tone, summarising and generating code. Once a model file is on the phone, the app needs no connection and Google states that all processing happens on the device. Screenshot: Google. The third screen shows the per-model settings, including the GPU and CPU toggle. Models come from the LiteRT Community organisation on Hugging Face, which the app’s welcome text links to directly. The first release listed three downloads, Gemma 3 1B IT at 4-bit in 554.7 MB, Hammer 2.1 1.5B at 8-bit and Qwen2.5 1.5B Instruct at 8-bit, the latter two at roughly 1.6 GB each. Gated repositories ask for a Hugging Face access token, which the settings dialog stores and can clear again. Anything outside that list has to arrive as a LiteRT .task bundle picked from device storage, and the app tells the user that no other file type is supported. Each model then carries a settings dialog with an accelerator switch between GPU and CPU, next to max tokens, top-k, top-p and temperature. The shipped list sets a different default per model, GPU first for Gemma 3 1B and CPU only for Qwen2.5 1.5B Instruct. After every reply the app prints four of its own measurements, time to first token in seconds, prefill speed and decode speed in tokens per second, and total latency in seconds. It derives them from the session itself, dividing the prompt’s token count by the time the first token took and counting decoded tokens against the clock that starts with that first token, so the numbers describe whatever hardware the app happens to be running on. Google calls this an “experimental Alpha release” and ships it under the Apache 2.0 licence. There is no store listing at this point, so installing means taking the APK from the project’s releases page on a device running Android 8.0 or newer, with a project wiki covering the corporate-device case. The README lists Android as available now and iOS as coming soon. Underneath, the app builds on LiteRT and Google’s MediaPipe LLM inference API for Android. Source: https://github.com/google-ai-edge/gallery Read the article: https://llmobile.news/ticker/googleai-edge-gallery/GoogleAndroidLiteRTHugging FaceOpen sourcehttps://llmobile.news/ticker/scaling-on-device-gpu-inference/https://llmobile.news/ticker/scaling-on-device-gpu-inference/Thu, 01 May 2025 02:44:00 +0200Eight researchers at Google published ML Drift on May 1, 2025, a GPU inference framework for running large generative models on the graphics chip of a phone, a laptop or a Mac. On the Adreno 750 GPU of a Samsung S24, the authors measure 1370 tokens per second of prefill and 37.1 tokens per second of decode for Gemma2 2B, and 412 and 12.7 tokens per second for Llama 3.1 8B. They report a 5 times to 11 times prefill speedup over llama.cpp, MLC LLM and the Qualcomm AI Hub benchmark on Qualcomm GPUs, and the work is due at the CVPR 2025 Workshop on Efficient and On-Device Generation (EDGE). The team benchmarked five mobile GPUs, the Adreno 830 in a Xiaomi 15 Pro with 16 GB of RAM, the Adreno 750 in a Samsung S24 with 8 GB, the Adreno 740 in a Samsung S23 Ultra with 8 GB, the Arm Immortalis-G720 in a vivo X100 Pro with 16 GB and the Arm Mali-G715 in a Pixel 9 with 12 GB, each at a fixed 1024 prefill and 256 generated tokens. Llama 3.1 8B with every weight at 8-bit ran out of memory on the S24, the S23 Ultra and the Pixel 9 and finished only on the two 16 GB phones, at 7.70 and 4.72 tokens per second of decode. A mixed scheme that holds attention weights at 8-bit and drops embedding and feed-forward weights to 4-bit ran the 8B model on all five devices and lifted decode by up to 1.9 times. ML Drift also treats the two phases of generation separately, since filling the prompt is limited by arithmetic while producing each further token is limited by memory bandwidth. Chart: Google. Solid bars are prefill on the left axis, cross-hatched bars decode on the right axis. Gaps mark runs the authors could not measure. A mobile GPU constrains inference in ways the paper works through one at a time. Memory comes first, and the authors report that Stable Diffusion 1.4 would need 4.31 GB for its intermediate 16-bit activations, which handing the same buffers to operations whose lifetimes do not overlap cuts to 387 MB. Weight layout comes second, and rearranging weight tensors into slices of four channels that match the 4-element units a GPU computes with buys up to a 20% speedup in matrix multiplication, which the authors call the primary driver of the framework’s advantage. Shader code comes third, and ML Drift writes its kernels once at startup from hand-tuned templates, so the translation between a tensor’s logical indices and the buffer or texture that actually holds it costs nothing while tokens are being produced. The same engine renders Stable Diffusion 1.4 at 512 by 512 pixels over 20 iterations in 10.96 seconds on the S23 Ultra, under 9 seconds on the S24 and in 3.4 seconds on a laptop with an Intel Lunar Lake Ultra 7 258V. On an NVIDIA RTX 4090 the OpenCL path runs 5% to 25% slower than llama.cpp on CUDA, which the authors put down to NVIDIA’s Tensor Cores being unreachable through OpenCL and WebGPU. The paper names sub-channel quantisation, sparsity and vendor extensions for matrix multiplication as the next steps, and points to no public release of the framework. Source: https://arxiv.org/abs/2505.00232 Read the article: https://llmobile.news/ticker/scaling-on-device-gpu-inference/GoogleQualcommAndroidBenchmarksResearchhttps://llmobile.news/ticker/alibaba-mnn/https://llmobile.news/ticker/alibaba-mnn/Thu, 23 Jan 2025 13:30:00 +0100Alibaba published the first public build of its MNN-LLM Android app on January 23, 2025, a client that downloads open models and runs them entirely on the phone, covering text-to-text, image-to-text, audio-to-text and text-to-image generation through diffusion models. It sits on MNN, the inference and training framework Alibaba has kept open source since 2019 and uses in more than 30 of its own apps, among them Taobao, Youku and DingTalk. Alibaba reports the runtime prefilling 8.6 times faster than llama.cpp on an Android CPU, with decoding 2.3 times faster. Models reach the device through llmexport, which converts a PyTorch checkpoint to ONNX and then to MNN’s own format, splitting the weights into a separate file and quantising them to 4-bit or 8-bit by default, with block size configurable and 5-bit and 6-bit available through the MNNConvert tool. The same README lists Qwen, Gemma, Llama including TinyLlama and MobileLLM, Baichuan, Yi, DeepSeek, InternLM, Phi, ReaderLM and SmolLM as supported families. LoRA adapters can be merged into the base model at export or kept as a separate file that the runtime loads and swaps while running. The MNN-LLM paper, presented at the ACM Multimedia Asia 2024 workshops, describes a DRAM-Flash split that leaves the embedding table, roughly 15% of the weights, in bfloat16 on flash storage and reads only the single token needed for each decode step. The authors put the saving for Qwen2 7B at about 2.18 GB of DRAM. Layers and the LM head stay in memory at int4 or int8 with asymmetric quantisation, computed as W4A8 or W8A8 on CPU and W4A16 or W8A16 on GPU, while the KV cache quantises keys to int4 or int8 and values to fp8 before spilling to flash with prefetching. The team measured the speed comparisons on a Xiaomi 14 against llama.cpp, MLC-LLM and fastllm, using Qwen2 1.5B, Qwen2 7B and Llama 3 8B. Android builds run on the CPU with ARMv8.2 fp16 instructions and add an OpenCL path for the GPU, and the same runtime compiles for iOS with Metal, for macOS, Linux and Windows, and for browsers through WebAssembly. Alibaba distributes the app as a downloadable APK and warns that the first version was tested only on a OnePlus 13 and a Xiaomi 14 Ultra, so low-spec phones may run slowly or not at all. MNN is published under the Apache 2.0 license. Diagram: Alibaba. The architecture overview from the MNN repository. Source: https://github.com/alibaba/MNN Read the article: https://llmobile.news/ticker/alibaba-mnn/AlibabaAndroidQuantisationQwenOpen sourcehttps://llmobile.news/ticker/exolabs-benchmarks/https://llmobile.news/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.news/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.news/ticker/phonelm/https://llmobile.news/ticker/phonelm/Thu, 07 Nov 2024 03:19:00 +0100Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3. The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search. Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma. The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure. An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage. Source: https://arxiv.org/abs/2411.05046 Read the article: https://llmobile.news/ticker/phonelm/ResearchOpen weightsAndroidNPUAgentshttps://llmobile.news/ticker/palmbench/https://llmobile.news/ticker/palmbench/Sat, 05 Oct 2024 05:37:00 +0200Researchers at the University of Wisconsin-Madison, Amazon Web Services AI and Uber ran quantised language models across eight phones and edge boards and measured the widest gap between Apple and everything else. In PalmBench, published as a conference paper at ICLR 2025, TinyLlama-1.1B at 4-bit reached close to 100 tok/s on an iPhone 15 Pro against roughly 30 tok/s on the fastest Android phone in the set, a Pixel 7. The authors report that the three-year-old iPhone 12 Pro also stayed ahead of every Android device and of Nvidia’s Jetson Orin Nano. Chart: Li et al., licensed CC BY 4.0. Throughput measured with MLC LLM, which was the only framework that ran on all eight platforms. The test bed covers a Pixel 4, Pixel 5a, Pixel 7, Galaxy S22 Ultra, iPhone 12 Pro, iPhone 15 Pro, an Orange Pi 5 and a Jetson Orin Nano, each with 6 GB or 8 GB of memory. Models include Llama-2-7B, Llama-3 and 3.1-8B, Llama-3.2-3B, Mistral-7B, Vicuna, Qwen2, Phi-2, Gemma-2-2B, RedPajama-INCITE-3B and TinyLlama-1.1B, quantised from 2-bit to 8-bit and additionally with GPTQ, AWQ and FasterTransformer. The team used two runtimes, MLC LLM and llama.cpp, and states that llama.cpp does not run on iPhone, so the cross-platform numbers come from MLC. Quantisation costs more answer quality than the bit count alone suggests. Scoring each quantised build’s exact match against the unquantised model’s own answers on SQuAD and Natural Questions, the paper’s figure puts 4-bit builds between about 50 and 61 percent, 6-bit between roughly 62 and 78 percent, and 2-bit and 3-bit builds as low as 15 percent. The authors note that 3-bit did not clearly beat 2-bit, and that 5-bit landed close to 4-bit while consuming more resources and degrading more than any 4-bit variant. Chart: Li et al., licensed CC BY 4.0. Exact match is measured against the answers of the same model before quantisation, not against ground truth. The 4-bit builds drew more power than the 3-bit ones rather than less. The paper’s table records 4-bit consuming more energy on all eight devices for the same 3B Llama model, rising from 10.13 to 13.05 mWh on the iPhone 15 Pro and from 14.54 to 14.68 mWh on the Pixel 7, which the authors put down to the higher-precision build occupying more GPU duty cycles. Surface temperature measured with a FLIR C5 thermal camera peaked at 75.4 °C on the Orange Pi 5 and 46.3 °C on the iPhone 15 Pro at 4-bit. Harmful output tracked the bit count more cleanly, with the hallucination rate on Llama-3-8B at 34.7 percent for 2-bit against 9.1 percent for 4-bit GPTQ and 7.9 percent at 8-bit. The authors name several limits on their own work. They cannot compare the two runtimes on iOS because llama.cpp does not support the iPhone, they profiled total memory only at 3-bit and 4-bit because higher-bit MLC builds exceeded the devices’ capacity, and they dropped models that would not fit at all, such as the unquantised Llama-2-7B at 13.11 GB. They also note that both runtimes expose few decoding options, with beam search unavailable in llama.cpp. Artificial Analysis has since run an independent benchmark of 33 quantised models on a single iPhone 17 Pro. Source: https://arxiv.org/abs/2410.05315 Read the article: https://llmobile.news/ticker/palmbench/BenchmarksQuantisationiPhoneAndroidMLC LLMhttps://llmobile.news/ticker/llama-stack/https://llmobile.news/ticker/llama-stack/Wed, 25 Sep 2024 19:30:00 +0200Meta announced Llama Stack at Connect on September 25, 2024, a set of REST APIs plus client SDKs for Python, Node, Swift and Kotlin that cover the same Llama features wherever the model runs. The repository defines APIs for Inference, Safety, Memory, Agentic System, Evaluation, Post Training, Synthetic Data Generation and Reward Scoring, and lists PyTorch ExecuTorch as the on-device provider, covering inference and agents on iOS. The models announced at the same event are covered separately. Llama Stack separates providers from distributions. A provider is the implementation behind an API, which can be local code or a pointer to a remote service, and a distribution assembles providers behind one endpoint. Meta’s README states that providers can be mixed, that a hobbyist can serve a small model locally and use a cloud provider for a large one, and that the higher level APIs the app works with do not change, including across the server and mobile-device boundary. Diagram: Meta. From the Llama Stack repository documentation. On iOS the local provider is a Swift class called LocalInference, which the documentation describes as powered by ExecuTorch. It implements the same Inference interface the Swift client uses for remote servers, so an app that already streams chat completions or agent turns keeps its calling code and swaps the service it was given. Developers export the model to an ExecuTorch .pte file, bundle it with tokenizer.model in the app, and link the XNNPACK, Core ML and MPS backends plus the quantized and optimised kernels, with Meta’s own example loading a SpinQuant build of Llama 3.2 1B. The Swift package is added through Xcode’s package manager and generates its types from the Llama Stack OpenAPI spec, and Meta listed its first release on October 10, 2024. The same iOS documentation stated that on-device inference for Android was coming soon, and pointed developers to ExecuTorch’s own Android demo app in the meantime. Llama Stack, the Swift client and the Kotlin client are published under the MIT license. Update, December 6, 2024. Meta released version 0.0.54 of the Kotlin client, which adds on-device inference on Android. The README names ExecuTorch as the local inference distributor, has developers run a script that fetches a prebuilt executorch.aar with the XNNPACK delegate into the app’s libs directory, and shows the only difference between the two modes as the client builder, LlamaStackClientLocalClient with a model and tokenizer path against LlamaStackClientOkHttpClient with a base URL. The chatCompletion call that follows is identical. Source: https://github.com/meta-llama/llama-stack Read the article: https://llmobile.news/ticker/llama-stack/MetaLlamaiOSAndroidDeveloper toolshttps://llmobile.news/ticker/elastic-on-device-llm-service/https://llmobile.news/ticker/elastic-on-device-llm-service/Sun, 08 Sep 2024 08:32:08 +0200Researchers at Peking University and Beijing University of Posts and Telecommunications published ElastiLM on September 8, 2024, a phone-side language model service that resizes itself for every request. Running LLaMA-7B on a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2, the authors measure 0.31 seconds to switch from one sub-model to another, against 8.3 seconds for swapping in a separately pretrained model of the right size and 6.2 seconds for the pruning method LLMPruner. The setup the paper assumes is one model held by the operating system that every app calls, the way Android AICore serves apps like Gboard smart reply. Those callers want different things from it, and the authors put an always-on voice assistant that needs an answer almost immediately next to a background screen-event recorder that can wait. ElastiLM takes a latency budget along with each request and then picks how much of the model to run and how much of the prompt to keep, in steps of 10 percent, rather than running one fixed model at one fixed speed. Accuracy is what pays for the speed. Cutting the prompt drops words the model might have needed and pruning removes whole attention heads and MLP neurons, and the split matters as much as the total, since the authors show one science question that half the prompt with an 80 percent sub-model answers correctly while 80 percent of the prompt with a half-size sub-model gets wrong. Preparing a model costs 68.3 GPU hours on a server with NVIDIA A40s, which the authors price at about $100 of rented time and put at 21.7 to 68.2 hours more than the pruning baselines. The service runs on four big CPU cores, and the authors state that as of August 2024 no mobile GPU or NPU could run ElastiLM or its baselines end to end, so their accelerator figures are projected from profiled latency rather than measured on a device. Across the Redmi K60 Champion, a Mi 14 and a Redmi K70 Pro, the last two on Snapdragon 8 Gen 3, the authors report ElastiLM answering more requests correctly than seven baselines by 6.60 to 14.83 percent in absolute accuracy, 10.45 percent on average, on traces of 600 requests drawn from six datasets. Peak memory for LLaMA-7B sits at 15 to 17 GB, level with the baselines, while keeping a dedicated model per latency budget would take 29.3 GB or 76.3 GB and runs out of memory on all three phones, which carry 19 GB to 24 GB of RAM. Weight-only quantisation to 8 bits costs almost no accuracy and 4 bits about 3 percent absolute, by the authors’ measurement. The paper appeared at MobiCom 2025 in Hong Kong in November 2025 and is on arXiv under a CC BY 4.0 license. Charts: Yin et al. Online overhead measured on a Redmi K60 Champion Edition, with the dashed line marking 1 percent of average time to first token. Source: https://arxiv.org/abs/2409.09071 Read the article: https://llmobile.news/ticker/elastic-on-device-llm-service/ResearchAndroidQualcommLlamahttps://llmobile.news/ticker/torchchat/https://llmobile.news/ticker/torchchat/Tue, 30 Jul 2024 18:00:00 +0200PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch. Diagram: PyTorch. The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark. Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster. Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb. The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference. Source: https://github.com/pytorch/torchchat Read the article: https://llmobile.news/ticker/torchchat/PyTorchLlamaAndroidiOSQuantisationhttps://llmobile.news/ticker/fast-on-device-llm-inference-with-npus/https://llmobile.news/ticker/fast-on-device-llm-inference-with-npus/Mon, 08 Jul 2024 14:20:00 +0200Researchers at Peking University and the Beijing University of Posts and Telecommunications published llm.npu on July 8, 2024, an inference engine that runs the prefill stage of a language model on a phone’s NPU. They tested it on a Redmi K70 Pro with a Snapdragon 8 Gen 3 and a Redmi K60 Pro with a Snapdragon 8 Gen 2, using Qwen1.5-1.8B, Gemma-2B, Phi-2 2.7B, LLaMA2-7B and Mistral-7B. On 1024-token prompts on the K70 Pro the authors report prefill 7.3x to 38.4x faster than the CPU engines MNN and llama.cpp and 1.27x to 43.6x faster than the GPU engines TFLite and MLC-LLM, and for Qwen1.5-1.8B they measure 1,106 tok/s against 151 tok/s for MNN on the CPU and 57 tok/s for llama.cpp on the CPU. Energy was measured on the K60 Pro, the only device they could root, where prefill used 1.85x to 59.5x less energy than the same baselines. Chart: Xu et al., published under CC BY 4.0. The prompts come from the LongBench 2WikiMultihopQA dataset. Prefill is the stage where the model reads the whole prompt before it writes its first token, and the paper puts it at 88.3% to 98.8% of total latency on a mobile CPU for UI automation, chat summaries and context-aware generation, and at 54.2% to 91.7% on a mobile GPU. A mobile NPU is fast at integer matrix multiplication but expects fixed tensor shapes, while prompts arrive at any length, so every new length means rebuilding the graph. Quantised activations also contain outliers, single values far outside the usual range, and the accurate remedy is per-group quantisation, which mobile NPUs cannot execute directly and which the authors measure at up to 10.7x overhead when it is split into sub-tensor multiplications. llm.npu answers both by cutting a prompt into fixed 256-token chunks that run on pre-built graphs, sharing the operators that do not depend on prompt length so the memory cost drops by up to 4x, pulling the outlier values out to the CPU or GPU to run in parallel, and scheduling transformer blocks across the processors out of order. Accuracy stays within 1% of FP16 on LAMBADA, HellaSwag, WinoGrande, OpenBookQA and MMLU. Across three application workloads, UI automation, context-aware email replies and chat summaries, the authors measure end-to-end latency 1.4x to 32.8x lower than the baselines, with decoding still running unoptimised on the CPU. The system is built on the group’s own mllm inference framework and Qualcomm’s QNN SDK, in about 10,000 lines of C, C++ and assembly, and the code is public in the mllm repository under an MIT license. The paper appeared at ASPLOS 2025 in Rotterdam in spring 2025. PowerInfer-v2, the one comparison system that also uses the NPU for prefill, is not open source, so the authors took its figures from its paper and report a 3.28x to 5.6x speedup over it. Source: https://arxiv.org/abs/2407.05858 Read the article: https://llmobile.news/ticker/fast-on-device-llm-inference-with-npus/NPUQualcommAndroidQuantisationResearchhttps://llmobile.news/ticker/llms-on-mobile-measurements/https://llmobile.news/ticker/llms-on-mobile-measurements/Tue, 11 Jun 2024 15:00:00 +0200Five researchers at Beijing University of Posts and Telecommunications measured language model inference on four Android phones and presented the results at the EdgeFM workshop of MobiSys 2024, deploying 22 models from 0.5B to 7B parameters through llama.cpp. For the 18 models they charted at 4-bit quantisation, accuracy for most of them sat between 55 and 60 percent, averaged across Winogrande, HellaSwag and MMLU, and latency came out at about 200 ms per token averaged over the four devices. The phones were a Xiaomi 14 with a Snapdragon 8 Gen 3 and 16 GB of RAM, a Pixel 7 Pro with a Tensor G2 and 12 GB, a Xiaomi 12S with a Snapdragon 8 Gen 1+ and 12 GB, and a Meizu 18 Pro with a Snapdragon 888 and 8 GB. Most of the 7B models needed about 4 GB of memory, which the authors put at half or a third of a current phone’s RAM. Their memory leaderboard has Bloom-7B highest, at 1.27 times the footprint of MPT-7B at the same parameter count, and Qwen1.5 0.5B lowest at under 1 GB. Memory use did not move with input or output length in their runs, with Baichuan holding at 4.83 GB whether the prompt was 10 or 50 tokens long and whether 1 or 1000 tokens were generated, which they trace to llama.cpp allocating one large pool at startup rather than sizing it per phase. Newer chips helped the two inference phases unevenly. Against the Snapdragon 888, the authors measured the Snapdragon 8 Gen 3 raising prefill throughput on LLaMA 2 by 81% but decode throughput by only 43%, with the Snapdragon 8 Gen 1+ at 45% and 34%. Decode is the phase that emits one token at a time and so governs how fast text appears on screen, and there the gap between the 8 Gen 3 and the 8 Gen 1+ was 7%. Continuous prefill runs also pushed the Xiaomi 12S, Xiaomi 14 and Meizu 18 Pro into thermal throttling, with per-token times drifting during a test series. The processor comparison went against the GPU in most cases. In the engine test llama.cpp on the 8 Gen 3 CPU prefilled 1.63 times faster than MLC LLM on the same phone’s GPU, 10.3 tokens per second against 6.3, while MLC on the GPU decoded faster at 9.8 against 6.6. On the mid-range Snapdragon 7 Gen 2 the CPU was ahead of the GPU in both phases, 4.59 and 4.65 against 3.3 and 3.5. The authors put this down to mobile GPUs being built for rendering rather than general-purpose compute, and to the extra cost of moving data. Quantisation cost less than the hardware choices did. Testing eight bit-widths on LLaMA 2, the authors report 4-bit as the best compromise, giving up 0.6% average accuracy against the 32-bit model while cutting model size 8 times, where 2-bit halves the size again for four times the accuracy loss. Code and results are published as TinyLLMLeaderBoard on GitHub, and the six-page paper is free to read in the ACM Digital Library. Source: https://dl.acm.org/doi/10.1145/3662006.3662059 Read the article: https://llmobile.news/ticker/llms-on-mobile-measurements/BenchmarksResearchllama.cppQualcommAndroidhttps://llmobile.news/ticker/powerinfer-2/https://llmobile.news/ticker/powerinfer-2/Mon, 10 Jun 2024 16:01:21 +0200Researchers at the IPADS lab of Shanghai Jiao Tong University published PowerInfer-2 on June 10, 2024, an inference framework for running language models that exceed a phone’s memory. On a OnePlus 12, a Snapdragon 8 Gen 3 device with 24 GB of DRAM of which 19 GB can be occupied by an app, the authors measure TurboSparse-Mixtral-47B decoding at 11.68 tokens per second, and describe it as the first system to serve a 47B model on a smartphone. In the same configuration they report 3.12 times the speed of LLMFlash and 21.2 times the speed of llama.cpp. LLMFlash here is their own reimplementation of Apple’s LLM in a flash techniques inside llama.cpp, which they built because that work was never released as code. PowerInfer-2 splits each feedforward layer into neuron clusters, small groups of neurons that tend to activate together, and treats a cluster rather than a whole matrix as the unit of work. Frequently activated clusters go to the phone’s NPU for dense arithmetic, while rarely activated ones go to the CPU, which copes better with irregular sparse work. Weights that are not already in an in-memory cache are read from UFS flash on demand, and those reads are overlapped with computation so that a core can work on one cluster while the next is still arriving. For prompt processing the system loads whole layers with sequential reads, which the authors measure at 3 times the throughput of random reads, and prefetches the next layer while the current one computes. The authors vary how much memory the app may use from 7 GB to 19 GB with the 47B model. At 7 GB the cache holds only 1.8 percent of the feedforward weights, so nearly every neuron has to be fetched from flash for every token and decoding falls to 2.13 tokens per second, which is still 1.84 times LLMFlash. The 47B model activates only about 3B parameters per token thanks to its mixture-of-experts structure and added sparsity, which is why it decodes at roughly the pace of a 7B model in the same test. Streaming weights off storage to fit a large model into a small budget is the same bet Edge0 makes at the level of whole experts rather than individual neurons. Chart: Xue et al. LLMFlash is the authors' own reimplementation of Apple's method. On OpenCompass at 4-bit weights, the authors put PowerInfer-2 at an average of 78.38 across Arc-Challenge, Arc-Easy, MMLU and GSM8K for Qwen2 7B against 79.25 for llama.cpp, and at 68.35 against 70.12 for Bamboo-7B, with Qualcomm’s QNN engine at 56.93 and 63.26. Token latency is uneven, and for the 47B model with half the feedforward weights offloaded they report a mean of 99.76 ms per token and a 99th percentile of 140.56 ms, tracking a cache miss rate that averages 3.5 percent but reaches 18.9 percent at the 99th percentile. Prompt processing for Bamboo-7B runs at 404.6 tokens per second with half the weights offloaded, against over 700 tokens per second with everything in memory. They measure energy at 0.257 joules per token against 0.373 for QNN and 0.672 for llama.cpp. PowerInfer-2 uses privileged system calls such as memory locking, so the authors built and ran it on rooted Android phones, and they state that nothing in the design requires kernel changes and that it should port to iOS. It also needs models with predictable sparsity, which is why the group trained sparsified TurboSparse versions of Mistral and Mixtral, and on an unmodified SiLU-based Mistral-7B the gain over LLMFlash drops to 2.4 times against 4.6 times for the ReLU-based Bamboo-7B. The system is 12,000 lines of C++ added to the group’s PowerInfer engine for PCs, which is MIT-licensed, but the announcement there links the paper and the model weights rather than a release of the PowerInfer-2 code. The paper was last revised in December 2024. Source: https://arxiv.org/abs/2406.06282 Read the article: https://llmobile.news/ticker/powerinfer-2/MemoryNPUAndroidQualcommResearchhttps://llmobile.news/ticker/octopus-v2/https://llmobile.news/ticker/octopus-v2/Tue, 02 Apr 2024 11:01:00 +0200Wei Chen and Zhiyuan Li published Octopus v2 on April 2, 2024, a 2B function-calling model fine-tuned from Google’s Gemma 2B that turns a spoken or typed request into an Android API call. On their own evaluation set built from 20 Android APIs, the authors report 99.524% accuracy and 0.38 seconds per call for Octopus v2, against 98.571% and 1.02 seconds for GPT-4. Both authors give Stanford University as their affiliation on the paper and publish the weights under the name Nexa AI. The distinguishing idea is what the authors call a functional token. Each of the 20 APIs gets its own special token added to the tokenizer and to the model’s output head, so picking a function becomes a single choice among 20 token candidates instead of writing the function name out as text and hoping every piece of it lands correctly. Since the model learned during fine-tuning what each of those tokens stands for, the function descriptions no longer have to sit in the prompt, which the paper puts at over 95% less context. Retrieval-augmented function calling processes roughly 1,000 tokens per call in their comparison, while Octopus v2 sees the query alone, typically under 30 tokens. Shorter context is also where the speed comes from. The authors measure Llama-7B with retrieval-augmented function calling at 68.095% accuracy and 13.46 seconds per call on a single NVIDIA A100, and Octopus v2 at 0.38 seconds under the same settings on the same A100, a 35-fold difference. The phone figure is a separate measurement. With the model quantised and the state for the fixed prompt prefix precomputed, they report a function call finishing in 1.1 to 1.7 seconds for queries of 20 to 30 tokens on what the paper calls a standard Android phone, without naming the device, and they estimate 37 times more calls per iPhone battery charge than a 7B model with retrieval. The authors name limits of their own result. Training with LoRA instead of the full model drops accuracy to 99.048%, and cutting the training set from 1,000 to 100 generated examples per API drops it to 98.095%, which they still put forward as the low end of a usable range. The set of callable functions is fixed at training time, since every API has to exist as a token before the model can emit it, and they price the data generation at 0.0224 USD per 1,000 examples per API. They also note that the GPT-3.5 and GPT-4 latencies were taken through OpenAI’s API on March 18, 2024, and may reflect API traffic or hardware allocation at that moment. Nexa AI released the weights on Hugging Face under a non-commercial CC BY-NC 4.0 license, on top of Google’s Gemma 2B. The model card said at the time that the team was still preparing the code for an open-source release. Source: https://arxiv.org/abs/2404.01744 Read the article: https://llmobile.news/ticker/octopus-v2/Nexa AIAgentsAndroidGemmaResearchhttps://llmobile.news/ticker/on-device-llmaas-context/https://llmobile.news/ticker/on-device-llmaas-context/Mon, 18 Mar 2024 15:03:23 +0100Researchers at Peking University, the Beijing University of Posts and Telecommunications and Tsinghua University published LLMS on March 18, 2024, an operating system service that keeps a separate conversation context for every app calling one shared on-device language model. Serving eight active contexts of Llama2-7B on a Jetson Orin NX, the authors measure 0.27 seconds to switch to another app’s context. They report switches up to 20 times and on average 9.7 times faster than a baseline that swaps the same chunks but quantises every one to 8 bits, modelled on the server system vLLM. The premise is one model held by the operating system instead of a copy inside each app, the way Android AICore serves Google apps, with every caller keeping its own KV cache, the stored attention state that lets a model continue a conversation without rereading it. On a Xiaomi 14 running llama.cpp, the authors measure one Llama2-7B context at the full 4k-token window at 2.02 GB, against 3.92 GB for the 4-bit weights, so a few apps holding contexts outweigh the model they all share. Letting the phone’s low-memory killer reclaim that memory means rebuilding the context by running the text through the model again, which the authors put at 22.92 seconds and 94.57 J on the same phone, close to the 90.04 J they measure for one minute of YouTube. What moves to storage is the KV cache, cut into chunks of 16 tokens, while the prompt and output text stay resident. The service quantises each chunk to 8, 4 or 2 bits according to how much attention its tokens draw from the rest of the context, holding a 50 percent average set by the system, then refills a context by reading some chunks from disk while recomputing others from the stored text and overlapping the two. The authors report twice the compression ratio of static quantisation at negligible accuracy loss, where compressing every chunk alike to 4 bits or 2 bits costs up to 59 percent and 99 percent of accuracy. Each of the three techniques carries part of that 0.27 seconds, which the authors break down as 0.42 seconds without the per-chunk compression, 0.62 seconds without the eviction order and 1.62 seconds without the read-and-recompute pipeline. The prototype is 3.5k lines of Python and C++ on Hugging Face Transformers and mllm, run on a Jetson Orin NX, a Jetson TX2 and a Xiaomi 14 with a Snapdragon 8 Gen 3, each with 8 GB of RAM, using Llama2-7B and OPT-6.7B over 72-hour traces synthesised from six datasets, and the authors state that it leaves token generation speed within 5 percent of running without it. The paper is on arXiv under a CC BY 4.0 license, and the same four authors published the work at SenSys 2026 in Saint-Malo in May 2026 as An Efficient Context Management System for On-Device LLMaaS. Charts: Yin et al. Measured on a Xiaomi 14, with the memory breakdown taken on llama.cpp. Source: https://arxiv.org/abs/2403.11805 Read the article: https://llmobile.news/ticker/on-device-llmaas-context/ResearchMemoryQuantisationLlamaAndroidhttps://llmobile.news/ticker/mediapipe-llm-inference-api/https://llmobile.news/ticker/mediapipe-llm-inference-api/Thu, 07 Mar 2024 17:00:00 +0100Google released the MediaPipe LLM Inference API on March 7, 2024, an experimental way to run language models fully on device from web, Android and iOS apps. Four open models are supported at launch, Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B. The API quantises weights to int8, with Gemma 2B using mixed 4-bit and 8-bit weights. Google measured throughput on unnamed high-end devices with a 1024-token input prompt and a maximum of 1280 tokens. In its charts, Gemma 2B at int4 prefills at roughly 680 tokens per second on WebGPU and on an Android GPU, and decodes at about 57 tokens per second on WebGPU against 31 on an Android GPU and 27 on iOS. Chart: Google. Chart: Google. Gemma 2B at int4 was the only model that ran on iOS. Google marks the Android version as intended for experimental and research use only, and points production apps to the Gemini API or to Gemini Nano through Android AICore instead. On iOS, Gemma 2B at int4 was the only model the team could run, which Google attributes to the memory available on the platform. Source: https://developers.googleblog.com/en/large-language-models-on-device-with-mediapipe-and-tensorflow-lite/ Read the article: https://llmobile.news/ticker/mediapipe-llm-inference-api/GoogleMediaPipeDeveloper toolsAndroidiOShttps://llmobile.news/ticker/mobillama/https://llmobile.news/ticker/mobillama/Mon, 26 Feb 2024 19:59:00 +0100Researchers at MBZUAI published MobiLlama on February 26, 2024, a language model of 0.5B parameters aimed at devices that cannot host a large one. The stated aim is a fully transparent small model rather than open weights alone, so the authors release the complete pre-training data pipeline, the training and evaluation code, the weights and more than 300 intermediate checkpoints. They also derive a 0.8B version from the same design. The design point is a single feed-forward block shared by every transformer layer. Each transformer block normally carries its own feed-forward network, the part that reworks each token’s representation after attention, and the authors measure those layers as 65% of all trainable parameters in their 1.2B reference model. Reusing one block across all 22 layers cuts trainable parameters by 60%, from 1.2B to 0.52B, while keeping the 22 layers and the hidden size of 2048 that the reference model has. The authors report pre-training the shared version in 7 days and 26.6K A100 GPU hours, against 12 days and 46.1K hours for the unshared 1.2B model. Diagram: Thawakar et al. For deployment the authors measured a smartphone with a Snapdragon 685, a laptop with an i7 CPU and a desktop with an RTX 2080 Ti, using 4-bit GGUF builds on the phone and the laptop. On the phone they put MobiLlama 0.5B at 770 MB of RAM and 7.02 tok/s, drawing 5.32 mAh per 1,000 tokens at 13.02% CPU utilisation. They measured Llama 2 7B on the same phone at 4287 MB, 1.19 tok/s and 10.07 mAh per 1,000 tokens with 77.41% CPU utilisation, and Phi-2 2.7B at 1893 MB and 2.88 tok/s. On the laptop CPU the 0.5B model reaches 36.32 tok/s in 799 MB. MobiLlama 0.5B averages 46.00 across nine benchmarks from the Open LLM Leaderboard in the authors’ own evaluation, which they compare with 43.57 for pythia-410m, and the 0.8B version, made by widening the shared block, averages 46.67. The models were pre-trained on 1.2T tokens from the Amber dataset of LLM360, where the two largest parts are a curated web crawl and code. Weights and chat-tuned variants are on Hugging Face under Apache 2.0, and the repository carries the training code and an Android app package for running the model on a phone. Source: https://arxiv.org/abs/2402.16840 Read the article: https://llmobile.news/ticker/mobillama/ResearchOpen weightsOpen sourceAndroidQuantisationhttps://llmobile.news/ticker/qualcomm-ai-hub/https://llmobile.news/ticker/qualcomm-ai-hub/Mon, 26 Feb 2024 09:00:00 +0100Qualcomm launched Qualcomm AI Hub at Mobile World Congress in Barcelona on February 26, 2024, a developer service built around a model library of more than 75 pre-optimised AI and generative AI models for Snapdragon and Qualcomm platforms. Qualcomm states that spreading each model across the NPU, CPU and GPU of its AI Engine results in 4X faster inferencing times. The second half of the service is measurement. A developer uploads a trained model, the hub compiles it for a chosen chip and runtime, then runs it on a physical Snapdragon device in Qualcomm’s cloud and returns latency, memory use and a numerical check against the original, as the documentation describes. The same measurements are published for every catalogue entry, which is why each model page names the handset the figures came from. Diagram: Qualcomm. At launch the export recipes in Qualcomm’s repository covered two runtimes, TensorFlow Lite and Qualcomm AI Engine Direct, on Android 11 and newer, in FP16, INT8 and INT4. The supported chips ran from the Snapdragon 845 through the Snapdragon 888 to the Snapdragon 8 Gen 3, with Samsung Galaxy S21 to S24, Xiaomi 12 and 13 and Google Pixel 3 to 5 handsets named as targets. Everything is driven from a Python package called qai_hub. Two of the listed models generate text, Llama-v2-7B-Chat and Baichuan-7B, both quantised to 4-bit weights and 16-bit activations. Qualcomm’s model card puts Llama-v2-7B-Chat at 3.6 GB with a maximum context of 1024 tokens, and reports it measured on a Samsung Galaxy S23 Ultra running Android 13 on a Snapdragon 8 Gen 2, on the NPU, at 117.8 ms for each additional token and 2,578.5 ms to push a full 1024-token prompt through before the first token comes back. Peak memory for the token generator is 66 to 238 MB. Qualcomm publishes the models on the AI Hub site, on GitHub and on Hugging Face, and names no price for the service, asking only for a Qualcomm ID to sign in and an API token to submit jobs. The licensing is layered. Qualcomm’s export and optimisation code is BSD 3-Clause, each model keeps the licence of the original implementation, so Llama-v2-7B-Chat stays under Meta’s terms, and the compiled assets prepared for on-device deployment carry a separate Qualcomm AI Hub Proprietary License. Source: https://aihub.qualcomm.com/mobile/models Read the article: https://llmobile.news/ticker/qualcomm-ai-hub/QualcommDeveloper toolsAndroidNPULlamahttps://llmobile.news/ticker/galaxy-s24-gemini-nano/https://llmobile.news/ticker/galaxy-s24-gemini-nano/Wed, 17 Jan 2024 20:00:00 +0100Google announced on January 17, 2024 that the Galaxy S24 series runs Gemini Nano on device, which made it the first phone line outside the Pixel 8 Pro to do so. Google names Magic Compose in Google Messages as the feature that runs locally, and states that the data does not leave the phone. Image: Google. The rest of the announced features run in the cloud. Google lists Gemini Pro behind summarisation in Samsung Notes and Voice Recorder as well as keyboard features, with Generative Edit in the Gallery app built on Imagen 2. Gemini Ultra was still in testing at the time. The launch also introduced Circle to Search, a gesture that searches whatever is circled or highlighted on screen without switching apps. Samsung published its own account of the launch on the Samsung Newsroom. Source: https://blog.google/products/android/google-ai-samsung-galaxy-s24/ Read the article: https://llmobile.news/ticker/galaxy-s24-gemini-nano/SamsungGoogleGemini NanoAndroidhttps://llmobile.news/ticker/arm-generative-ai-on-mobile/https://llmobile.news/ticker/arm-generative-ai-on-mobile/Wed, 17 Jan 2024 15:00:00 +0100Arm showed a chat assistant running Meta’s Llama 2 7B on existing Android phones using three Cortex-A700 series CPU cores and LPDDR5x memory, at a text generation rate of 9.6 tokens per second. Gian Marco Iodice and Adnan AlSinan published the demo video and the measurement on January 17, 2024, and state that the video runs at actual speed and that everything runs locally on the device. Arm says the rate is faster than average human reading speed. Integer quantisation is what brought a 7B model into phone memory, according to Arm’s write-up. Arm puts the RAM needed for Llama 2 7B with 16-bit floating point weights at about 14 GB, and says quantising those weights to 4-bit shrinks the model four times, to roughly 4 GB. Arm warns that going down to 3-bit or 2-bit costs accuracy. Pinning each worker thread to a specific core with a thread affinity mask, rather than letting Android’s scheduler place it, improved latency by over 10 percent in Arm’s tests. Arm then wrote its own 4-bit matrix-by-vector and matrix-by-matrix routines, the two functions that dominate the work in a language model, tuned for Cortex-A700 series cores. The routines are built on two instructions the cores already carry, SDOT for dot products and SMMLA for 8-bit integer matrix multiply and accumulate. Arm reports that they cut the time to the first token by over 50 percent and raised text generation by 20 percent against the native implementation in llama.cpp, and said at the time that the routines would be available soon. Arm makes the case for the CPU on the shape of the workload rather than on peak throughput, arguing that the typical batch size and the balance of compute and bandwidth in language model inference suit a CPU, and that its programmability lets developers try new compression and quantisation schemes. The post names no NPU or GPU and offers no comparison against one. Arm puts the reach of the approach at around 99 percent of smartphones worldwide and says 70 percent of AI in third-party applications already runs on Arm CPUs. Arm later extended the same post with a second demo, a teaching assistant chatbot built on Microsoft’s Phi-3 3.8B, which Arm measures at just over 15 tokens per second on the same software optimisations. Arm names the Scalable Matrix Extension for Armv9-A as the next hardware step for these workloads. Source: https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/generative-ai-on-mobile-on-arm-cpu Read the article: https://llmobile.news/ticker/arm-generative-ai-on-mobile/ArmAndroidllama.cppLlamaQuantisationhttps://llmobile.news/ticker/gemini-nano-pixel-8-pro/https://llmobile.news/ticker/gemini-nano-pixel-8-pro/Wed, 06 Dec 2023 18:00:00 +0100Google brought Gemini Nano to the Pixel 8 Pro in its December 2023 feature drop, where it powers Summarize in Recorder and Smart Reply in Gboard. Google calls the Pixel 8 Pro the first smartphone engineered for Gemini Nano and runs the model on the Tensor G3. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Google. Gemini Nano summarising a recording in the Recorder app. Open the Android Developers post According to Google, running the model locally helps prevent sensitive data from leaving the phone and lets the features work without a network connection. Summarize in Recorder launched in English. Smart Reply in Gboard launched globally on the United States English keyboard layout, starting with WhatsApp, Line and KakaoTalk. On the same day Google introduced AICore, a system service in Android 14 that handles model management, runtimes and safety features for Gemini Nano. It supports Low Rank Adaptation, so developers can build small adapters trained on their own data, and it targets the Google Tensor TPU as well as NPUs from Qualcomm, Samsung and MediaTek. Google describes the service as isolated from the network by design and opened access through an early access programme. Diagram: Google. Source: https://blog.google/products/pixel/pixel-feature-drop-december-2023/ Read the article: https://llmobile.news/ticker/gemini-nano-pixel-8-pro/GoogleGemini NanoPixelAndroidDeveloper toolshttps://llmobile.news/ticker/snapdragon-8-gen-3/https://llmobile.news/ticker/snapdragon-8-gen-3/Tue, 24 Oct 2023 19:00:00 +0200Qualcomm launched the Snapdragon 8 Gen 3 at its Snapdragon Summit in Hawaii on October 24, 2023, calling it its first mobile platform designed with generative AI in mind. On the product page the company states support for generative models with up to 10 billion parameters on device and up to 20 tokens per second for large language models. Qualcomm puts the Hexagon NPU at 98 percent faster than the previous generation and claims 40 percent better performance per watt for sustained AI inference. It lists the Kryo CPU at 30 percent more performance and 20 percent better power efficiency, and the Adreno GPU at 25 percent faster rendering with 25 percent better efficiency. Among the workloads Qualcomm names for local execution are Stable Diffusion, virtual assistants and chatbots. Devices were announced by ASUS, Honor, iQOO, MEIZU, Nubia, OnePlus, OPPO, realme, Redmi, RedMagic, Sony, vivo, Xiaomi and ZTE. Image: Qualcomm. Source: https://www.qualcomm.com/news/releases/2023/10/qualcomm-launches-premium-snapdragon-8-gen-3-to-bring-generative Read the article: https://llmobile.news/ticker/snapdragon-8-gen-3/QualcommChipsNPUAndroidhttps://llmobile.news/ticker/sherpa/https://llmobile.news/ticker/sherpa/Mon, 27 Mar 2023 20:46:00 +0200Bip-Rep published Sherpa on March 27, 2023, a Flutter app that runs llama.cpp on the phone itself, so the chat works with no network connection. The README asks users to bring their own weights, Meta’s 7B LLaMA converted to a ggml model file, which the app loads through a file picker, and version 2.1.0 added Vicuna support and macOS builds next to the Android and Windows ones. The project publishes no tokens per second figure and states no minimum memory, and the only device it names is a OnePlus 7 with 8 GB of RAM in the demo video below, which the README says was recorded at real speed. The code is MIT licensed. Development stopped three weeks after it started. All 43 commits fall between March 27 and April 14, 2023, the last of the seven releases went out on April 14, 2023, and the llama.cpp fork the app builds against was last touched on April 12, 2023, four months before llama.cpp moved to the GGUF format. Sherpa reads the older ggml files rather than the GGUF models that have been standard since, and requests for GGUF support have gone unanswered, with 23 of the 29 issues ever filed still open. A contributor answered one of them by writing that the original developers had abandoned the project, and pointed to his own fork, Maid. ▶A working chatGPT running on a mobile using llama.cppLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: tibzejoker. Source: https://github.com/Bip-Rep/sherpa Read the article: https://llmobile.news/ticker/sherpa/llama.cppAndroidLlamaOpen sourcehttps://llmobile.news/ticker/qualcomm-stable-diffusion-android/https://llmobile.news/ticker/qualcomm-stable-diffusion-android/Thu, 23 Feb 2023 17:00:00 +0100Qualcomm AI Research has demonstrated Stable Diffusion v1.5, a model with more than 1 billion parameters, running entirely on an Android phone powered by a Snapdragon 8 Gen 2. The company reports image generation in under 15 seconds for 20 inference steps at 512x512 pixels, a workload that until then ran in the cloud. Images: Qualcomm. Sample outputs from the demo, generated on the phone. To fit the model on the phone, Qualcomm quantised it from FP32 to INT8 using its AI Model Efficiency Toolkit and a post-training method called AdaRound, which adapts how individual weights are rounded. According to the company, the model needed no retraining after the conversion. Inference runs through the Qualcomm AI Engine direct framework on the Hexagon processor, with the model prepared in Qualcomm AI Studio. The company sums up the result in its post. This is the fastest inference on a smartphone and comparable to cloud latency. Source: https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android Read the article: https://llmobile.news/ticker/qualcomm-stable-diffusion-android/QualcommImage generationQuantisationAndroidNPU