https://llmobile.news/tags/mlc-llm/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/lm-meter/https://llmobile.news/ticker/lm-meter/Tue, 07 Oct 2025 19:05:00 +0200Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference. Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16. Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7. The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks. Source: https://arxiv.org/abs/2510.06126 Read the article: https://llmobile.news/ticker/lm-meter/ResearchBenchmarksDeveloper toolsMLC LLMPixelhttps://llmobile.news/ticker/palmbench/https://llmobile.news/ticker/palmbench/Sat, 05 Oct 2024 05:37:00 +0200Researchers at the University of Wisconsin-Madison, Amazon Web Services AI and Uber ran quantised language models across eight phones and edge boards and measured the widest gap between Apple and everything else. In PalmBench, published as a conference paper at ICLR 2025, TinyLlama-1.1B at 4-bit reached close to 100 tok/s on an iPhone 15 Pro against roughly 30 tok/s on the fastest Android phone in the set, a Pixel 7. The authors report that the three-year-old iPhone 12 Pro also stayed ahead of every Android device and of Nvidia’s Jetson Orin Nano. Chart: Li et al., licensed CC BY 4.0. Throughput measured with MLC LLM, which was the only framework that ran on all eight platforms. The test bed covers a Pixel 4, Pixel 5a, Pixel 7, Galaxy S22 Ultra, iPhone 12 Pro, iPhone 15 Pro, an Orange Pi 5 and a Jetson Orin Nano, each with 6 GB or 8 GB of memory. Models include Llama-2-7B, Llama-3 and 3.1-8B, Llama-3.2-3B, Mistral-7B, Vicuna, Qwen2, Phi-2, Gemma-2-2B, RedPajama-INCITE-3B and TinyLlama-1.1B, quantised from 2-bit to 8-bit and additionally with GPTQ, AWQ and FasterTransformer. The team used two runtimes, MLC LLM and llama.cpp, and states that llama.cpp does not run on iPhone, so the cross-platform numbers come from MLC. Quantisation costs more answer quality than the bit count alone suggests. Scoring each quantised build’s exact match against the unquantised model’s own answers on SQuAD and Natural Questions, the paper’s figure puts 4-bit builds between about 50 and 61 percent, 6-bit between roughly 62 and 78 percent, and 2-bit and 3-bit builds as low as 15 percent. The authors note that 3-bit did not clearly beat 2-bit, and that 5-bit landed close to 4-bit while consuming more resources and degrading more than any 4-bit variant. Chart: Li et al., licensed CC BY 4.0. Exact match is measured against the answers of the same model before quantisation, not against ground truth. The 4-bit builds drew more power than the 3-bit ones rather than less. The paper’s table records 4-bit consuming more energy on all eight devices for the same 3B Llama model, rising from 10.13 to 13.05 mWh on the iPhone 15 Pro and from 14.54 to 14.68 mWh on the Pixel 7, which the authors put down to the higher-precision build occupying more GPU duty cycles. Surface temperature measured with a FLIR C5 thermal camera peaked at 75.4 °C on the Orange Pi 5 and 46.3 °C on the iPhone 15 Pro at 4-bit. Harmful output tracked the bit count more cleanly, with the hallucination rate on Llama-3-8B at 34.7 percent for 2-bit against 9.1 percent for 4-bit GPTQ and 7.9 percent at 8-bit. The authors name several limits on their own work. They cannot compare the two runtimes on iOS because llama.cpp does not support the iPhone, they profiled total memory only at 3-bit and 4-bit because higher-bit MLC builds exceeded the devices’ capacity, and they dropped models that would not fit at all, such as the unquantised Llama-2-7B at 13.11 GB. They also note that both runtimes expose few decoding options, with beam search unavailable in llama.cpp. Artificial Analysis has since run an independent benchmark of 33 quantised models on a single iPhone 17 Pro. Source: https://arxiv.org/abs/2410.05315 Read the article: https://llmobile.news/ticker/palmbench/BenchmarksQuantisationiPhoneAndroidMLC LLMhttps://llmobile.news/ticker/mlc-llm-consumer-devices/https://llmobile.news/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.news/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPU